2022-09-10T03:57:53.162 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-10T03:57:53.163 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023624 --name pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} -- /home/teuthworker/archive/pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023624/orig.config.yaml 2022-09-10T03:57:53.230 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-09-10T03:57:53.317 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023624 branch: wip-pdonnell-testing-20220909.143836 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} email: pdonnell@redhat.com first_in_suite: false job_id: '7023624' kernel: client: branch: testing kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20220909.143836 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 244418605927bb7842e73fc5d2a53889b760937f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 244418605927bb7842e73fc5d2a53889b760937f kclient: mntopts: - ms_mode=secure - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20220909.143836 sha1: 244418605927bb7842e73fc5d2a53889b760937f owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: 244418605927bb7842e73fc5d2a53889b760937f sleep_before_teardown: 0 suite: fs:workload suite_branch: wip-pdonnell-testing-20220909.143836 suite_path: /home/teuthworker/src/github.com_batrick_ceph_244418605927bb7842e73fc5d2a53889b760937f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 244418605927bb7842e73fc5d2a53889b760937f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCTsaYh7BTrUjWJ+iTEhZWHV4J+1l7Oo8bf3WBbtDw487cqyEE6vbvdyKF1xWRjSmN1q3VFHLiRGnZdZ0ab3AD2vpkFucfXsu9iULDYa+oa5XMPcp/b3KHt0NEV463HP/f5riu7iiQhB9gwnp3EEj0OK1jgsgWxRW9fZnFO1Mv3BqH3GPFp3a5tmgvEv4scTeREeWf+l4os8I2zKKnYgbghe0KHMJbKpGHOFY5s87qRDfLmvC0nu2ZT5bywoTLP7xUL5Ptes0zYyCuqL776ZqUWlw0+tAdns0rp/bQpK1tMnO3wpu9NoAJVrB5EHrO1VeyB7nVh22WWmeeCYLRMN7X8C2CsArUMp3cxZJzssEL5k7uT8Zhb5znDrIx0KT6tLXtQ9BCH/mXFmgz5CBpkQtWB1meyodceUoaqTMJm7cjBy5zlLIx1NfPB+Lh5LNR5yNhHUdC1sDTY6Cx3blLsuZ6Jmgg7yExC0RWWSCWKedGh+eQhM59jzXkFZC3sF7ZJ/I8= smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxjC3RYFi5JcT12aNErZaKqOdK8hpEKresMocdZ8xvlcEw7bT73iHkQziBk/EwmkG7jH2YouOp0EoF/+e8Bzkv7a4C/NmscYwi+2MCc8RHReFX7xtQ3SMGwbMYeJkix1J2fRUegz34o4/claltQn2D0VQJVR+X0yngJHs2ZHJUOZ6TwTqB02y8Mzc8rCj4ijMPLInlRf/Ge8PK5N9e0t0PPSpKAqcOFP1+qKVGLoRV71Gq6jYbrYCZYzFgfcBihmql5rT+IrfDfuxChk3hEeq6c+bqt6+s0/tgGFvCnJIuYaulvX6sycrGgRNQEeSixc7tZ7ghabXijK/Ts80dKsadLBb8QgjqRUh6U6l7Kp+vwdhcaB0Uo6ytZcXAhCkzJtHytW6qm6ffMiRKcO8TN/6FgCz5TB4lDowBYjcwXSGHWrrLK8vmPcH+zmJCQqsVAfbn9/m5CiPzHnrkQ4cfkWwcCV7AixXjVeS11R3GHE1/91U2j5YdYOaMA8Guv9i3xg8= smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnqe+dUPVWr93XFBadjU78COBpl2U+wUS3fZxHd/iz+2NZ9SmXcRgO6pbZHBStWkcNGuZSnIgQgYdCfGpvRajbCfAr+zthyazXlPUzM5b7K/K62A67KmzQAqh3n2UPyGQw/uzFGk1zTtuqFBsQJbmPXGEblVUn6i2EOfhIA4ZDm2G+UFpX6un2K1XaeND9eGRmjk3tEQFkMvE7t41egbE7419VRDQfoivClqTAfMzNaX7Ve+6MzWh0a/uv6fVMjmRbGzCZOiiMKnaxyYijMVsorV2uB0gcDYSFSjykPCNorqhpard38uNeRP2EmEqsBVvYQwNjSm9Yls8wwmKPpFz0/6tg33rGOoiPKG1+rOFS1lT6qp5/mYEhsQMNB2N1KHzu6rZ8wtzTkJHXzZ+LPyg05uik2I4sECD2GCRRb4OOyh1PQo8joGzN2tKOtaR2IT1h7OkOUr/Ry3XbkabMCA/xI50Qx/ZTSdUpMyrO7h8Xskso4FMhmQ2b8mUoG9nwebs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - direct_io teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b58f2c18636eb10faa77ed3614abd00cb85dfc2c timestamp: 2022-09-09_18:47:12 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-09-10T03:57:53.317 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_244418605927bb7842e73fc5d2a53889b760937f/qa; will attempt to use it 2022-09-10T03:57:53.318 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_244418605927bb7842e73fc5d2a53889b760937f/qa/tasks 2022-09-10T03:57:53.319 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-09-10T03:57:53.321 INFO:teuthology.task.internal:Checking packages... 2022-09-10T03:57:53.346 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '244418605927bb7842e73fc5d2a53889b760937f' 2022-09-10T03:57:53.346 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-09-10T03:57:53.347 INFO:teuthology.packaging:ref: None 2022-09-10T03:57:53.347 INFO:teuthology.packaging:tag: None 2022-09-10T03:57:53.347 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20220909.143836 2022-09-10T03:57:53.348 INFO:teuthology.packaging:sha1: 244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T03:57:53.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20220909.143836 2022-09-10T03:57:53.504 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-14898.g24441860 2022-09-10T03:57:53.504 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-09-10T03:57:53.512 INFO:teuthology.task.internal:no buildpackages task found 2022-09-10T03:57:53.512 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-09-10T03:57:53.539 INFO:teuthology.task.internal:Saving configuration 2022-09-10T03:57:53.562 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-09-10T03:57:53.568 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-10T03:57:53.595 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023624', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-10 03:52:59.632434', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPx7djnPhbRBzf3uDZL+IOuUHNw3ByhjmNoiFAMXxi+hMAzqHh+mFJG0OT4WoDiRAJmrEt0QbDpLpH6er9Y21gI='} 2022-09-10T03:57:53.622 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023624', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-10 03:52:59.631365', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdHZGjZ7Ljcpb5K+hg///3xcRpUlYaG+NUMqofZfYBWrfx0eSAyoYMUOGMV9ERRmT6QsJizQN3jfYFKslCffiY='} 2022-09-10T03:57:53.646 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2022-09-09_18:47:12-fs:workload-wip-pdonnell-testing-20220909.143836-distro-default-smithi/7023624', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-09-10 03:52:59.629701', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLbzzb6kbxttP5dJnHF/fMBdppMqYUB9Cc4K9OPP/1TszD5+Oz/ha5t0JbJyQRKf9BPYIKIGdcdFUuBih2NAM1g='} 2022-09-10T03:57:53.647 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-09-10T03:57:53.653 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2022-09-10T03:57:53.654 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2022-09-10T03:57:53.654 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2022-09-10T03:57:53.654 INFO:teuthology.run_tasks:Running task console_log... 2022-09-10T03:57:53.767 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d7d7147b8>, signals=[15]) 2022-09-10T03:57:53.769 INFO:teuthology.run_tasks:Running task internal.connect... 2022-09-10T03:57:53.777 INFO:teuthology.task.internal:Opening connections... 2022-09-10T03:57:53.777 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T03:57:53.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T03:57:53.880 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2022-09-10T03:57:53.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T03:57:53.958 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2022-09-10T03:57:53.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T03:57:54.032 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-09-10T03:57:54.040 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2022-09-10T03:57:54.061 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2022-09-10T03:57:54.061 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2022-09-10T03:57:54.121 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Red Hat Enterprise Linux" 2022-09-10T03:57:54.121 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="8.6 (Ootpa)" 2022-09-10T03:57:54.122 INFO:teuthology.orchestra.run.smithi081.stdout:ID="rhel" 2022-09-10T03:57:54.122 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="fedora" 2022-09-10T03:57:54.122 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="8.6" 2022-09-10T03:57:54.123 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el8" 2022-09-10T03:57:54.123 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-10T03:57:54.123 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2022-09-10T03:57:54.123 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-10T03:57:54.124 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.redhat.com/" 2022-09-10T03:57:54.124 INFO:teuthology.orchestra.run.smithi081.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-10T03:57:54.124 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-10T03:57:54.125 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T03:57:54.125 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-10T03:57:54.125 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-10T03:57:54.125 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-10T03:57:54.126 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-10T03:57:54.127 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2022-09-10T03:57:54.153 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2022-09-10T03:57:54.173 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2022-09-10T03:57:54.174 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2022-09-10T03:57:54.233 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Red Hat Enterprise Linux" 2022-09-10T03:57:54.233 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8.6 (Ootpa)" 2022-09-10T03:57:54.234 INFO:teuthology.orchestra.run.smithi104.stdout:ID="rhel" 2022-09-10T03:57:54.234 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="fedora" 2022-09-10T03:57:54.234 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8.6" 2022-09-10T03:57:54.234 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2022-09-10T03:57:54.235 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-10T03:57:54.235 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2022-09-10T03:57:54.235 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-10T03:57:54.235 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.redhat.com/" 2022-09-10T03:57:54.235 INFO:teuthology.orchestra.run.smithi104.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-10T03:57:54.236 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-10T03:57:54.236 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T03:57:54.236 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-10T03:57:54.237 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-10T03:57:54.237 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-10T03:57:54.237 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-10T03:57:54.237 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2022-09-10T03:57:54.259 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2022-09-10T03:57:54.281 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2022-09-10T03:57:54.281 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2022-09-10T03:57:54.340 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Red Hat Enterprise Linux" 2022-09-10T03:57:54.341 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8.6 (Ootpa)" 2022-09-10T03:57:54.341 INFO:teuthology.orchestra.run.smithi110.stdout:ID="rhel" 2022-09-10T03:57:54.341 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="fedora" 2022-09-10T03:57:54.342 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8.6" 2022-09-10T03:57:54.342 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8" 2022-09-10T03:57:54.342 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-10T03:57:54.342 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2022-09-10T03:57:54.343 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-10T03:57:54.343 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.redhat.com/" 2022-09-10T03:57:54.343 INFO:teuthology.orchestra.run.smithi110.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-10T03:57:54.343 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-10T03:57:54.344 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T03:57:54.344 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-10T03:57:54.344 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-10T03:57:54.344 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-10T03:57:54.344 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-10T03:57:54.345 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2022-09-10T03:57:54.366 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-09-10T03:57:54.381 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-09-10T03:57:54.389 INFO:teuthology.task.internal:Checking for old test directory... 2022-09-10T03:57:54.390 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2022-09-10T03:57:54.394 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2022-09-10T03:57:54.398 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2022-09-10T03:57:54.418 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-09-10T03:57:54.424 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-09-10T03:57:54.424 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2022-09-10T03:57:54.454 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2022-09-10T03:57:54.457 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2022-09-10T03:57:54.492 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-09-10T03:57:54.571 INFO:teuthology.run_tasks:Running task kernel... 2022-09-10T03:57:54.585 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2022-09-10T03:57:54.585 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2022-09-10T03:57:54.586 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2022-09-10T03:57:54.586 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-10T03:57:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T03:57:54.606 DEBUG:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2022-09-10T03:57:54.709 INFO:teuthology.orchestra.run.smithi081.stdout:RedHatEnterprise 2022-09-10T03:57:54.711 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T03:57:54.712 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2022-09-10T03:57:54.904 DEBUG:teuthology.task.kernel:sha1 for client.0 is a127ffb7df2a46d9c720771310f2b324a2af81f2 2022-09-10T03:57:54.904 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "a127ffb7df2a46d9c720771310f2b324a2af81f2"... 2022-09-10T03:57:54.905 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2022-09-10T03:57:54.924 INFO:teuthology.orchestra.run.smithi081.stdout:4.18.0-372.9.1.el8.x86_64 2022-09-10T03:57:54.924 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs a127ffb7df2a46d9c720771310f2b324a2af81f2 2022-09-10T03:57:54.925 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2022-09-10T03:57:54.925 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum upgrade -y linux-firmware 2022-09-10T03:57:55.947 INFO:teuthology.orchestra.run.smithi081.stdout:Updating Subscription Management repositories. 2022-09-10T03:57:55.947 INFO:teuthology.orchestra.run.smithi081.stdout:Unable to read consumer identity 2022-09-10T03:57:56.029 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T03:57:56.029 INFO:teuthology.orchestra.run.smithi081.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2022-09-10T03:57:56.030 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T03:57:57.559 INFO:teuthology.orchestra.run.smithi081.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2022-09-10T03:57:58.501 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2022-09-10T03:58:01.335 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 424 kB/s | 24 kB 00:00 2022-09-10T03:58:02.003 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2022-09-10T03:58:02.004 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2022-09-10T03:58:02.004 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2022-09-10T03:58:02.134 INFO:teuthology.task.kernel:Downloading kernel a127ffb7df2a46d9c720771310f2b324a2af81f2 on client.0... 2022-09-10T03:58:02.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2022-09-10T03:58:02.178 INFO:teuthology.orchestra.run.smithi081.stdout:RedHatEnterprise 2022-09-10T03:58:02.180 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T03:58:02.181 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a127ffb7df2a46d9c720771310f2b324a2af81f2 2022-09-10T03:58:02.337 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/a127ffb7df2a46d9c720771310f2b324a2af81f2/centos/8/flavors/default/x86_64/ 2022-09-10T03:58:02.338 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.0.0_rc4_ceph_ga127ffb7df2a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/a127ffb7df2a46d9c720771310f2b324a2af81f2/centos/8/flavors/default/x86_64/ --input-file=- 2022-09-10T03:58:02.341 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi081.front.sepia.ceph.com to complete... 2022-09-10T03:58:24.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10 03:58:24 URL:https://4.chacra.ceph.com/r/kernel/testing/a127ffb7df2a46d9c720771310f2b324a2af81f2/centos/8/flavors/default/x86_64/kernel-6.0.0_rc4_ceph_ga127ffb7df2a-1.x86_64.rpm [528623720/528623720] -> "/tmp/kernel.x86_64.rpm" [1] 2022-09-10T03:58:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:FINISHED --2022-09-10 03:58:24-- 2022-09-10T03:58:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:Total wall clock time: 22s 2022-09-10T03:58:24.807 INFO:teuthology.orchestra.run.smithi081.stderr:Downloaded: 1 files, 504M in 22s (22.9 MB/s) 2022-09-10T03:58:24.809 INFO:teuthology.task.kernel:Installing kernel a127ffb7df2a46d9c720771310f2b324a2af81f2 on client.0... 2022-09-10T03:58:24.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2022-09-10T03:58:26.893 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying... ######################################## 2022-09-10T03:58:27.016 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing... ######################################## 2022-09-10T03:58:27.113 INFO:teuthology.orchestra.run.smithi081.stdout:Updating / installing... 2022-09-10T03:59:06.901 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-6.0.0_rc4_ceph_ga127ffb7df2a-1 ######################################## 2022-09-10T03:59:31.458 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2022-09-10T03:59:31.458 DEBUG:teuthology.orchestra.run.smithi081:> rpm -qlp /tmp/kernel.x86_64.rpm 2022-09-10T03:59:31.528 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:31.528 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/config-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:31.528 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/vmlinux-6.0.0-rc4-ceph-ga127ffb7df2a.bz2 2022-09-10T03:59:31.529 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:31.529 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:31.529 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel 2022-09-10T03:59:31.530 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch 2022-09-10T03:59:31.530 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86 2022-09-10T03:59:31.530 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto 2022-09-10T03:59:31.531 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2022-09-10T03:59:31.531 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2022-09-10T03:59:31.531 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2022-09-10T03:59:31.531 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/camellia-x86_64.ko 2022-09-10T03:59:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2022-09-10T03:59:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2022-09-10T03:59:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/crc32-pclmul.ko 2022-09-10T03:59:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/crc32c-intel.ko 2022-09-10T03:59:31.532 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2022-09-10T03:59:31.533 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2022-09-10T03:59:31.533 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2022-09-10T03:59:31.533 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2022-09-10T03:59:31.533 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/serpent-avx2.ko 2022-09-10T03:59:31.534 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2022-09-10T03:59:31.534 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/sha1-ssse3.ko 2022-09-10T03:59:31.534 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/sha256-ssse3.ko 2022-09-10T03:59:31.534 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/sha512-ssse3.ko 2022-09-10T03:59:31.534 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2022-09-10T03:59:31.535 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2022-09-10T03:59:31.535 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/crypto/twofish-x86_64.ko 2022-09-10T03:59:31.535 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel 2022-09-10T03:59:31.536 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel/cpu 2022-09-10T03:59:31.536 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel/cpu/mce 2022-09-10T03:59:31.536 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2022-09-10T03:59:31.536 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm 2022-09-10T03:59:31.536 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm/kvm-amd.ko 2022-09-10T03:59:31.537 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm/kvm-intel.ko 2022-09-10T03:59:31.537 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/arch/x86/kvm/kvm.ko 2022-09-10T03:59:31.537 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto 2022-09-10T03:59:31.537 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/ansi_cprng.ko 2022-09-10T03:59:31.538 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/anubis.ko 2022-09-10T03:59:31.538 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/arc4.ko 2022-09-10T03:59:31.538 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx 2022-09-10T03:59:31.538 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_memcpy.ko 2022-09-10T03:59:31.538 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_pq.ko 2022-09-10T03:59:31.539 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_raid6_recov.ko 2022-09-10T03:59:31.539 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_tx.ko 2022-09-10T03:59:31.539 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/async_xor.ko 2022-09-10T03:59:31.539 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/async_tx/raid6test.ko 2022-09-10T03:59:31.540 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/authenc.ko 2022-09-10T03:59:31.540 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/authencesn.ko 2022-09-10T03:59:31.540 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/blake2b_generic.ko 2022-09-10T03:59:31.541 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/blowfish_common.ko 2022-09-10T03:59:31.541 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/blowfish_generic.ko 2022-09-10T03:59:31.541 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/camellia_generic.ko 2022-09-10T03:59:31.542 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cast5_generic.ko 2022-09-10T03:59:31.542 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cast6_generic.ko 2022-09-10T03:59:31.543 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cast_common.ko 2022-09-10T03:59:31.543 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/ccm.ko 2022-09-10T03:59:31.543 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cmac.ko 2022-09-10T03:59:31.543 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/crc32_generic.ko 2022-09-10T03:59:31.544 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/crypto_engine.ko 2022-09-10T03:59:31.544 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/crypto_user.ko 2022-09-10T03:59:31.544 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/cts.ko 2022-09-10T03:59:31.544 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/des_generic.ko 2022-09-10T03:59:31.544 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/dh_generic.ko 2022-09-10T03:59:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/echainiv.ko 2022-09-10T03:59:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/essiv.ko 2022-09-10T03:59:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/fcrypt.ko 2022-09-10T03:59:31.545 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/khazad.ko 2022-09-10T03:59:31.546 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/lz4.ko 2022-09-10T03:59:31.546 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/lz4hc.ko 2022-09-10T03:59:31.546 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/md4.ko 2022-09-10T03:59:31.546 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/michael_mic.ko 2022-09-10T03:59:31.547 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/pcbc.ko 2022-09-10T03:59:31.547 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/pcrypt.ko 2022-09-10T03:59:31.547 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/rmd160.ko 2022-09-10T03:59:31.547 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/seed.ko 2022-09-10T03:59:31.547 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/serpent_generic.ko 2022-09-10T03:59:31.548 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/tcrypt.ko 2022-09-10T03:59:31.548 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/tea.ko 2022-09-10T03:59:31.548 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/twofish_common.ko 2022-09-10T03:59:31.549 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/twofish_generic.ko 2022-09-10T03:59:31.549 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/vmac.ko 2022-09-10T03:59:31.549 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/wp512.ko 2022-09-10T03:59:31.549 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/xcbc.ko 2022-09-10T03:59:31.549 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/xor.ko 2022-09-10T03:59:31.550 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/crypto/xxhash_generic.ko 2022-09-10T03:59:31.550 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers 2022-09-10T03:59:31.550 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi 2022-09-10T03:59:31.550 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/acpi_ipmi.ko 2022-09-10T03:59:31.551 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/acpi_pad.ko 2022-09-10T03:59:31.551 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/custom_method.ko 2022-09-10T03:59:31.551 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/ec_sys.ko 2022-09-10T03:59:31.552 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/platform_profile.ko 2022-09-10T03:59:31.552 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/sbs.ko 2022-09-10T03:59:31.552 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/sbshc.ko 2022-09-10T03:59:31.552 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/acpi/video.ko 2022-09-10T03:59:31.553 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata 2022-09-10T03:59:31.553 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/acard-ahci.ko 2022-09-10T03:59:31.553 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/ahci_platform.ko 2022-09-10T03:59:31.553 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/ata_generic.ko 2022-09-10T03:59:31.554 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/libahci_platform.ko 2022-09-10T03:59:31.554 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_acpi.ko 2022-09-10T03:59:31.554 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ali.ko 2022-09-10T03:59:31.554 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_amd.ko 2022-09-10T03:59:31.555 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_artop.ko 2022-09-10T03:59:31.555 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_atiixp.ko 2022-09-10T03:59:31.555 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_atp867x.ko 2022-09-10T03:59:31.555 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_cmd640.ko 2022-09-10T03:59:31.556 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_cmd64x.ko 2022-09-10T03:59:31.556 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_cypress.ko 2022-09-10T03:59:31.556 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_efar.ko 2022-09-10T03:59:31.556 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt366.ko 2022-09-10T03:59:31.556 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt37x.ko 2022-09-10T03:59:31.557 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt3x2n.ko 2022-09-10T03:59:31.557 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_hpt3x3.ko 2022-09-10T03:59:31.557 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_it8213.ko 2022-09-10T03:59:31.557 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_it821x.ko 2022-09-10T03:59:31.558 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_jmicron.ko 2022-09-10T03:59:31.558 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_marvell.ko 2022-09-10T03:59:31.558 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_mpiix.ko 2022-09-10T03:59:31.559 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_netcell.ko 2022-09-10T03:59:31.559 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ninja32.ko 2022-09-10T03:59:31.559 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ns87410.ko 2022-09-10T03:59:31.559 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_ns87415.ko 2022-09-10T03:59:31.560 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_oldpiix.ko 2022-09-10T03:59:31.560 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_opti.ko 2022-09-10T03:59:31.560 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_optidma.ko 2022-09-10T03:59:31.561 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_pcmcia.ko 2022-09-10T03:59:31.561 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_pdc2027x.ko 2022-09-10T03:59:31.561 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_pdc202xx_old.ko 2022-09-10T03:59:31.561 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_piccolo.ko 2022-09-10T03:59:31.562 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_rdc.ko 2022-09-10T03:59:31.562 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sch.ko 2022-09-10T03:59:31.562 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_serverworks.ko 2022-09-10T03:59:31.563 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sil680.ko 2022-09-10T03:59:31.563 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sis.ko 2022-09-10T03:59:31.563 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_sl82c105.ko 2022-09-10T03:59:31.563 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_triflex.ko 2022-09-10T03:59:31.564 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pata_via.ko 2022-09-10T03:59:31.564 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/pdc_adma.ko 2022-09-10T03:59:31.564 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_inic162x.ko 2022-09-10T03:59:31.564 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_mv.ko 2022-09-10T03:59:31.564 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_nv.ko 2022-09-10T03:59:31.565 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_promise.ko 2022-09-10T03:59:31.565 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_qstor.ko 2022-09-10T03:59:31.565 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sil.ko 2022-09-10T03:59:31.565 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sil24.ko 2022-09-10T03:59:31.566 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sis.ko 2022-09-10T03:59:31.566 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_svw.ko 2022-09-10T03:59:31.566 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_sx4.ko 2022-09-10T03:59:31.566 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_uli.ko 2022-09-10T03:59:31.567 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_via.ko 2022-09-10T03:59:31.567 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ata/sata_vsc.ko 2022-09-10T03:59:31.567 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/auxdisplay 2022-09-10T03:59:31.567 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/auxdisplay/ks0108.ko 2022-09-10T03:59:31.568 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/base 2022-09-10T03:59:31.568 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/base/regmap 2022-09-10T03:59:31.568 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/base/regmap/regmap-i2c.ko 2022-09-10T03:59:31.569 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/bcma 2022-09-10T03:59:31.569 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/bcma/bcma.ko 2022-09-10T03:59:31.569 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block 2022-09-10T03:59:31.570 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/aoe 2022-09-10T03:59:31.570 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/aoe/aoe.ko 2022-09-10T03:59:31.570 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/brd.ko 2022-09-10T03:59:31.571 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/drbd 2022-09-10T03:59:31.571 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/drbd/drbd.ko 2022-09-10T03:59:31.571 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/floppy.ko 2022-09-10T03:59:31.571 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/loop.ko 2022-09-10T03:59:31.571 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/mtip32xx 2022-09-10T03:59:31.572 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2022-09-10T03:59:31.572 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/nbd.ko 2022-09-10T03:59:31.572 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/null_blk 2022-09-10T03:59:31.573 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/null_blk/null_blk.ko 2022-09-10T03:59:31.573 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/pktcdvd.ko 2022-09-10T03:59:31.573 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/rbd.ko 2022-09-10T03:59:31.574 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/virtio_blk.ko 2022-09-10T03:59:31.574 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/xen-blkback 2022-09-10T03:59:31.574 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2022-09-10T03:59:31.574 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/xen-blkfront.ko 2022-09-10T03:59:31.575 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/zram 2022-09-10T03:59:31.575 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/block/zram/zram.ko 2022-09-10T03:59:31.575 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char 2022-09-10T03:59:31.575 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hangcheck-timer.ko 2022-09-10T03:59:31.575 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random 2022-09-10T03:59:31.576 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/amd-rng.ko 2022-09-10T03:59:31.576 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/intel-rng.ko 2022-09-10T03:59:31.576 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2022-09-10T03:59:31.576 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/via-rng.ko 2022-09-10T03:59:31.577 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/hw_random/virtio-rng.ko 2022-09-10T03:59:31.577 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi 2022-09-10T03:59:31.577 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2022-09-10T03:59:31.578 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2022-09-10T03:59:31.578 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2022-09-10T03:59:31.578 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_si.ko 2022-09-10T03:59:31.579 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2022-09-10T03:59:31.579 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2022-09-10T03:59:31.580 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/lp.ko 2022-09-10T03:59:31.580 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/mwave 2022-09-10T03:59:31.580 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/mwave/mwave.ko 2022-09-10T03:59:31.581 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/pcmcia 2022-09-10T03:59:31.581 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/pcmcia/cm4000_cs.ko 2022-09-10T03:59:31.581 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/pcmcia/cm4040_cs.ko 2022-09-10T03:59:31.581 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/ppdev.ko 2022-09-10T03:59:31.582 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tlclk.ko 2022-09-10T03:59:31.582 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm 2022-09-10T03:59:31.582 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm.ko 2022-09-10T03:59:31.583 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_atmel.ko 2022-09-10T03:59:31.583 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_infineon.ko 2022-09-10T03:59:31.583 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_nsc.ko 2022-09-10T03:59:31.583 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_tis.ko 2022-09-10T03:59:31.584 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/tpm/tpm_tis_core.ko 2022-09-10T03:59:31.584 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/uv_mmtimer.ko 2022-09-10T03:59:31.584 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/char/virtio_console.ko 2022-09-10T03:59:31.585 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq 2022-09-10T03:59:31.585 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2022-09-10T03:59:31.586 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2022-09-10T03:59:31.586 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/p4-clockmod.ko 2022-09-10T03:59:31.586 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2022-09-10T03:59:31.587 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/powernow-k8.ko 2022-09-10T03:59:31.587 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/cpufreq/speedstep-lib.ko 2022-09-10T03:59:31.588 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto 2022-09-10T03:59:31.588 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/ccp 2022-09-10T03:59:31.588 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2022-09-10T03:59:31.589 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/ccp/ccp.ko 2022-09-10T03:59:31.589 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/padlock-aes.ko 2022-09-10T03:59:31.589 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/padlock-sha.ko 2022-09-10T03:59:31.589 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat 2022-09-10T03:59:31.589 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_common 2022-09-10T03:59:31.590 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2022-09-10T03:59:31.590 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_dh895xcc 2022-09-10T03:59:31.590 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2022-09-10T03:59:31.590 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/virtio 2022-09-10T03:59:31.591 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2022-09-10T03:59:31.591 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dca 2022-09-10T03:59:31.591 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dca/dca.ko 2022-09-10T03:59:31.591 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/devfreq 2022-09-10T03:59:31.591 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/devfreq/governor_simpleondemand.ko 2022-09-10T03:59:31.592 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma 2022-09-10T03:59:31.592 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/dw 2022-09-10T03:59:31.592 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/dw/dw_dmac.ko 2022-09-10T03:59:31.592 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/ioat 2022-09-10T03:59:31.593 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/dma/ioat/ioatdma.ko 2022-09-10T03:59:31.593 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac 2022-09-10T03:59:31.593 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/amd64_edac.ko 2022-09-10T03:59:31.593 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/e752x_edac.ko 2022-09-10T03:59:31.593 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/edac_mce_amd.ko 2022-09-10T03:59:31.594 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i3000_edac.ko 2022-09-10T03:59:31.594 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i3200_edac.ko 2022-09-10T03:59:31.594 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i5000_edac.ko 2022-09-10T03:59:31.594 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i5100_edac.ko 2022-09-10T03:59:31.595 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i5400_edac.ko 2022-09-10T03:59:31.595 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i7300_edac.ko 2022-09-10T03:59:31.595 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i7core_edac.ko 2022-09-10T03:59:31.595 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/i82975x_edac.ko 2022-09-10T03:59:31.595 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/ie31200_edac.ko 2022-09-10T03:59:31.596 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/sb_edac.ko 2022-09-10T03:59:31.596 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/edac/x38_edac.ko 2022-09-10T03:59:31.596 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/firmware 2022-09-10T03:59:31.596 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/firmware/edd.ko 2022-09-10T03:59:31.596 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/firmware/iscsi_ibft.ko 2022-09-10T03:59:31.597 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/gpio 2022-09-10T03:59:31.597 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/gpio/gpio-ich.ko 2022-09-10T03:59:31.597 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/gpio/gpio-viperboard.ko 2022-09-10T03:59:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid 2022-09-10T03:59:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-appleir.ko 2022-09-10T03:59:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-aureal.ko 2022-09-10T03:59:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-axff.ko 2022-09-10T03:59:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-dr.ko 2022-09-10T03:59:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-elecom.ko 2022-09-10T03:59:31.600 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-elo.ko 2022-09-10T03:59:31.600 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-emsff.ko 2022-09-10T03:59:31.601 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-gaff.ko 2022-09-10T03:59:31.601 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-gt683r.ko 2022-09-10T03:59:31.601 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-gyration.ko 2022-09-10T03:59:31.602 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-holtek-kbd.ko 2022-09-10T03:59:31.602 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-holtek-mouse.ko 2022-09-10T03:59:31.603 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-holtekff.ko 2022-09-10T03:59:31.603 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-hyperv.ko 2022-09-10T03:59:31.603 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-icade.ko 2022-09-10T03:59:31.603 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-keytouch.ko 2022-09-10T03:59:31.604 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-kye.ko 2022-09-10T03:59:31.604 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-lcpower.ko 2022-09-10T03:59:31.604 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-led.ko 2022-09-10T03:59:31.604 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-lenovo.ko 2022-09-10T03:59:31.605 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-logitech-dj.ko 2022-09-10T03:59:31.605 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-logitech-hidpp.ko 2022-09-10T03:59:31.605 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-multitouch.ko 2022-09-10T03:59:31.605 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-ortek.ko 2022-09-10T03:59:31.605 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-penmount.ko 2022-09-10T03:59:31.606 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-petalynx.ko 2022-09-10T03:59:31.606 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-picolcd.ko 2022-09-10T03:59:31.606 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-pl.ko 2022-09-10T03:59:31.606 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-plantronics.ko 2022-09-10T03:59:31.607 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-primax.ko 2022-09-10T03:59:31.607 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-rmi.ko 2022-09-10T03:59:31.607 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-arvo.ko 2022-09-10T03:59:31.607 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-common.ko 2022-09-10T03:59:31.607 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-isku.ko 2022-09-10T03:59:31.608 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-kone.ko 2022-09-10T03:59:31.608 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-koneplus.ko 2022-09-10T03:59:31.608 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-konepure.ko 2022-09-10T03:59:31.608 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2022-09-10T03:59:31.609 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-lua.ko 2022-09-10T03:59:31.609 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-pyra.ko 2022-09-10T03:59:31.609 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-ryos.ko 2022-09-10T03:59:31.609 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat-savu.ko 2022-09-10T03:59:31.610 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-roccat.ko 2022-09-10T03:59:31.610 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-saitek.ko 2022-09-10T03:59:31.610 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-samsung.ko 2022-09-10T03:59:31.610 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sensor-hub.ko 2022-09-10T03:59:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sjoy.ko 2022-09-10T03:59:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sony.ko 2022-09-10T03:59:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-speedlink.ko 2022-09-10T03:59:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-steelseries.ko 2022-09-10T03:59:31.611 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-sunplus.ko 2022-09-10T03:59:31.612 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-thrustmaster.ko 2022-09-10T03:59:31.612 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-tivo.ko 2022-09-10T03:59:31.612 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-tmff.ko 2022-09-10T03:59:31.612 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-topseed.ko 2022-09-10T03:59:31.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-twinhan.ko 2022-09-10T03:59:31.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-uclogic.ko 2022-09-10T03:59:31.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-waltop.ko 2022-09-10T03:59:31.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-wiimote.ko 2022-09-10T03:59:31.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-xinmo.ko 2022-09-10T03:59:31.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-zpff.ko 2022-09-10T03:59:31.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/hid-zydacron.ko 2022-09-10T03:59:31.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/uhid.ko 2022-09-10T03:59:31.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hid/wacom.ko 2022-09-10T03:59:31.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv 2022-09-10T03:59:31.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv/hv_balloon.ko 2022-09-10T03:59:31.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv/hv_utils.ko 2022-09-10T03:59:31.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hv/hv_vmbus.ko 2022-09-10T03:59:31.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon 2022-09-10T03:59:31.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/abituguru.ko 2022-09-10T03:59:31.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/abituguru3.ko 2022-09-10T03:59:31.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/acpi_power_meter.ko 2022-09-10T03:59:31.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ad7414.ko 2022-09-10T03:59:31.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ad7418.ko 2022-09-10T03:59:31.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adc128d818.ko 2022-09-10T03:59:31.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1025.ko 2022-09-10T03:59:31.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1026.ko 2022-09-10T03:59:31.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1029.ko 2022-09-10T03:59:31.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm1031.ko 2022-09-10T03:59:31.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adm9240.ko 2022-09-10T03:59:31.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ads7828.ko 2022-09-10T03:59:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7410.ko 2022-09-10T03:59:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7411.ko 2022-09-10T03:59:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7462.ko 2022-09-10T03:59:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7470.ko 2022-09-10T03:59:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7475.ko 2022-09-10T03:59:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/adt7x10.ko 2022-09-10T03:59:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/amc6821.ko 2022-09-10T03:59:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/applesmc.ko 2022-09-10T03:59:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/asb100.ko 2022-09-10T03:59:31.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/asc7621.ko 2022-09-10T03:59:31.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/asus_atk0110.ko 2022-09-10T03:59:31.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/atxp1.ko 2022-09-10T03:59:31.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/coretemp.ko 2022-09-10T03:59:31.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2022-09-10T03:59:31.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/dme1737.ko 2022-09-10T03:59:31.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ds1621.ko 2022-09-10T03:59:31.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ds620.ko 2022-09-10T03:59:31.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/emc1403.ko 2022-09-10T03:59:31.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/emc6w201.ko 2022-09-10T03:59:31.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/f71805f.ko 2022-09-10T03:59:31.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/f71882fg.ko 2022-09-10T03:59:31.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/f75375s.ko 2022-09-10T03:59:31.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/fam15h_power.ko 2022-09-10T03:59:31.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/fschmd.ko 2022-09-10T03:59:31.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/g760a.ko 2022-09-10T03:59:31.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/g762.ko 2022-09-10T03:59:31.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/gl518sm.ko 2022-09-10T03:59:31.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/gl520sm.ko 2022-09-10T03:59:31.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/hwmon-vid.ko 2022-09-10T03:59:31.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/i5500_temp.ko 2022-09-10T03:59:31.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/i5k_amb.ko 2022-09-10T03:59:31.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ibmaem.ko 2022-09-10T03:59:31.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ibmpex.ko 2022-09-10T03:59:31.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ina209.ko 2022-09-10T03:59:31.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ina2xx.ko 2022-09-10T03:59:31.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/it87.ko 2022-09-10T03:59:31.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/k10temp.ko 2022-09-10T03:59:31.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/k8temp.ko 2022-09-10T03:59:31.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lineage-pem.ko 2022-09-10T03:59:31.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm63.ko 2022-09-10T03:59:31.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm73.ko 2022-09-10T03:59:31.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm75.ko 2022-09-10T03:59:31.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm77.ko 2022-09-10T03:59:31.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm78.ko 2022-09-10T03:59:31.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm80.ko 2022-09-10T03:59:31.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm83.ko 2022-09-10T03:59:31.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm85.ko 2022-09-10T03:59:31.644 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm87.ko 2022-09-10T03:59:31.644 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm90.ko 2022-09-10T03:59:31.644 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm92.ko 2022-09-10T03:59:31.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm93.ko 2022-09-10T03:59:31.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm95234.ko 2022-09-10T03:59:31.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm95241.ko 2022-09-10T03:59:31.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/lm95245.ko 2022-09-10T03:59:31.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc2945.ko 2022-09-10T03:59:31.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4151.ko 2022-09-10T03:59:31.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4215.ko 2022-09-10T03:59:31.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4222.ko 2022-09-10T03:59:31.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4245.ko 2022-09-10T03:59:31.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4260.ko 2022-09-10T03:59:31.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/ltc4261.ko 2022-09-10T03:59:31.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max16065.ko 2022-09-10T03:59:31.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max1619.ko 2022-09-10T03:59:31.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max1668.ko 2022-09-10T03:59:31.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max197.ko 2022-09-10T03:59:31.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max6639.ko 2022-09-10T03:59:31.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max6650.ko 2022-09-10T03:59:31.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/max6697.ko 2022-09-10T03:59:31.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/mcp3021.ko 2022-09-10T03:59:31.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct6683.ko 2022-09-10T03:59:31.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct6775-core.ko 2022-09-10T03:59:31.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct6775.ko 2022-09-10T03:59:31.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/nct7802.ko 2022-09-10T03:59:31.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pc87360.ko 2022-09-10T03:59:31.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pc87427.ko 2022-09-10T03:59:31.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pcf8591.ko 2022-09-10T03:59:31.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus 2022-09-10T03:59:31.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/adm1275.ko 2022-09-10T03:59:31.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/lm25066.ko 2022-09-10T03:59:31.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2022-09-10T03:59:31.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/max16064.ko 2022-09-10T03:59:31.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/max34440.ko 2022-09-10T03:59:31.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/max8688.ko 2022-09-10T03:59:31.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/pmbus.ko 2022-09-10T03:59:31.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2022-09-10T03:59:31.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/tps40422.ko 2022-09-10T03:59:31.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2022-09-10T03:59:31.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2022-09-10T03:59:31.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/pmbus/zl6100.ko 2022-09-10T03:59:31.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/powr1220.ko 2022-09-10T03:59:31.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sch5627.ko 2022-09-10T03:59:31.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sch5636.ko 2022-09-10T03:59:31.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sch56xx-common.ko 2022-09-10T03:59:31.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sht15.ko 2022-09-10T03:59:31.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sht21.ko 2022-09-10T03:59:31.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/shtc1.ko 2022-09-10T03:59:31.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/sis5595.ko 2022-09-10T03:59:31.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/smsc47b397.ko 2022-09-10T03:59:31.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/smsc47m1.ko 2022-09-10T03:59:31.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/smsc47m192.ko 2022-09-10T03:59:31.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/thmc50.ko 2022-09-10T03:59:31.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp102.ko 2022-09-10T03:59:31.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp103.ko 2022-09-10T03:59:31.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp401.ko 2022-09-10T03:59:31.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/tmp421.ko 2022-09-10T03:59:31.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/via-cputemp.ko 2022-09-10T03:59:31.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/via686a.ko 2022-09-10T03:59:31.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/vt1211.ko 2022-09-10T03:59:31.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/vt8231.ko 2022-09-10T03:59:31.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83627ehf.ko 2022-09-10T03:59:31.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83627hf.ko 2022-09-10T03:59:31.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83781d.ko 2022-09-10T03:59:31.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83791d.ko 2022-09-10T03:59:31.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83792d.ko 2022-09-10T03:59:31.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83793.ko 2022-09-10T03:59:31.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83795.ko 2022-09-10T03:59:31.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83l785ts.ko 2022-09-10T03:59:31.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/hwmon/w83l786ng.ko 2022-09-10T03:59:31.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c 2022-09-10T03:59:31.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/algos 2022-09-10T03:59:31.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2022-09-10T03:59:31.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2022-09-10T03:59:31.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses 2022-09-10T03:59:31.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2022-09-10T03:59:31.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-amd756.ko 2022-09-10T03:59:31.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2022-09-10T03:59:31.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2022-09-10T03:59:31.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2022-09-10T03:59:31.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2022-09-10T03:59:31.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2022-09-10T03:59:31.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2022-09-10T03:59:31.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-i801.ko 2022-09-10T03:59:31.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-isch.ko 2022-09-10T03:59:31.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-ismt.ko 2022-09-10T03:59:31.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2022-09-10T03:59:31.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2022-09-10T03:59:31.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-parport.ko 2022-09-10T03:59:31.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2022-09-10T03:59:31.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-piix4.ko 2022-09-10T03:59:31.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-scmi.ko 2022-09-10T03:59:31.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-simtec.ko 2022-09-10T03:59:31.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2022-09-10T03:59:31.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2022-09-10T03:59:31.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-via.ko 2022-09-10T03:59:31.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-viapro.ko 2022-09-10T03:59:31.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2022-09-10T03:59:31.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-dev.ko 2022-09-10T03:59:31.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-mux.ko 2022-09-10T03:59:31.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2022-09-10T03:59:31.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-smbus.ko 2022-09-10T03:59:31.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/i2c/i2c-stub.ko 2022-09-10T03:59:31.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband 2022-09-10T03:59:31.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core 2022-09-10T03:59:31.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_cm.ko 2022-09-10T03:59:31.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_core.ko 2022-09-10T03:59:31.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_umad.ko 2022-09-10T03:59:31.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/ib_uverbs.ko 2022-09-10T03:59:31.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/iw_cm.ko 2022-09-10T03:59:31.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/rdma_cm.ko 2022-09-10T03:59:31.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/core/rdma_ucm.ko 2022-09-10T03:59:31.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw 2022-09-10T03:59:31.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/cxgb4 2022-09-10T03:59:31.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2022-09-10T03:59:31.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx4 2022-09-10T03:59:31.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2022-09-10T03:59:31.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx5 2022-09-10T03:59:31.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2022-09-10T03:59:31.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mthca 2022-09-10T03:59:31.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2022-09-10T03:59:31.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp 2022-09-10T03:59:31.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/ipoib 2022-09-10T03:59:31.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2022-09-10T03:59:31.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/iser 2022-09-10T03:59:31.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2022-09-10T03:59:31.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/isert 2022-09-10T03:59:31.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2022-09-10T03:59:31.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srp 2022-09-10T03:59:31.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2022-09-10T03:59:31.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srpt 2022-09-10T03:59:31.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2022-09-10T03:59:31.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input 2022-09-10T03:59:31.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport 2022-09-10T03:59:31.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/emu10k1-gp.ko 2022-09-10T03:59:31.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/fm801-gp.ko 2022-09-10T03:59:31.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/gameport.ko 2022-09-10T03:59:31.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/lightning.ko 2022-09-10T03:59:31.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/gameport/ns558.ko 2022-09-10T03:59:31.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/joydev.ko 2022-09-10T03:59:31.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/keyboard 2022-09-10T03:59:31.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/keyboard/gpio_keys.ko 2022-09-10T03:59:31.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse 2022-09-10T03:59:31.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/appletouch.ko 2022-09-10T03:59:31.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/bcm5974.ko 2022-09-10T03:59:31.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/cyapatp.ko 2022-09-10T03:59:31.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/elan_i2c.ko 2022-09-10T03:59:31.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/sermouse.ko 2022-09-10T03:59:31.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/synaptics_i2c.ko 2022-09-10T03:59:31.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/synaptics_usb.ko 2022-09-10T03:59:31.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/mouse/vsxxxaa.ko 2022-09-10T03:59:31.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/rmi4 2022-09-10T03:59:31.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/rmi4/rmi_core.ko 2022-09-10T03:59:31.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio 2022-09-10T03:59:31.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/altera_ps2.ko 2022-09-10T03:59:31.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/arc_ps2.ko 2022-09-10T03:59:31.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/hyperv-keyboard.ko 2022-09-10T03:59:31.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/serio/serio_raw.ko 2022-09-10T03:59:31.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/input/sparse-keymap.ko 2022-09-10T03:59:31.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/iommu 2022-09-10T03:59:31.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/iommu/amd 2022-09-10T03:59:31.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/iommu/amd/iommu_v2.ko 2022-09-10T03:59:31.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds 2022-09-10T03:59:31.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-blinkm.ko 2022-09-10T03:59:31.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-lm3530.ko 2022-09-10T03:59:31.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-lp3944.ko 2022-09-10T03:59:31.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/leds-ss4200.ko 2022-09-10T03:59:31.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger 2022-09-10T03:59:31.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2022-09-10T03:59:31.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2022-09-10T03:59:31.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2022-09-10T03:59:31.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2022-09-10T03:59:31.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2022-09-10T03:59:31.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2022-09-10T03:59:31.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2022-09-10T03:59:31.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2022-09-10T03:59:31.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2022-09-10T03:59:31.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md 2022-09-10T03:59:31.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/bcache 2022-09-10T03:59:31.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/bcache/bcache.ko 2022-09-10T03:59:31.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-bio-prison.ko 2022-09-10T03:59:31.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-cache-smq.ko 2022-09-10T03:59:31.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-cache.ko 2022-09-10T03:59:31.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-crypt.ko 2022-09-10T03:59:31.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-delay.ko 2022-09-10T03:59:31.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-flakey.ko 2022-09-10T03:59:31.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-log-userspace.ko 2022-09-10T03:59:31.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-multipath.ko 2022-09-10T03:59:31.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-queue-length.ko 2022-09-10T03:59:31.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-raid.ko 2022-09-10T03:59:31.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-round-robin.ko 2022-09-10T03:59:31.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-service-time.ko 2022-09-10T03:59:31.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-switch.ko 2022-09-10T03:59:31.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-thin-pool.ko 2022-09-10T03:59:31.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/dm-verity.ko 2022-09-10T03:59:31.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/faulty.ko 2022-09-10T03:59:31.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/linear.ko 2022-09-10T03:59:31.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/multipath.ko 2022-09-10T03:59:31.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/persistent-data 2022-09-10T03:59:31.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2022-09-10T03:59:31.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid0.ko 2022-09-10T03:59:31.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid1.ko 2022-09-10T03:59:31.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid10.ko 2022-09-10T03:59:31.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/md/raid456.ko 2022-09-10T03:59:31.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message 2022-09-10T03:59:31.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion 2022-09-10T03:59:31.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptbase.ko 2022-09-10T03:59:31.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptctl.ko 2022-09-10T03:59:31.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptfc.ko 2022-09-10T03:59:31.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptlan.ko 2022-09-10T03:59:31.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptsas.ko 2022-09-10T03:59:31.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptscsih.ko 2022-09-10T03:59:31.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/message/fusion/mptspi.ko 2022-09-10T03:59:31.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd 2022-09-10T03:59:31.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/lpc_ich.ko 2022-09-10T03:59:31.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/lpc_sch.ko 2022-09-10T03:59:31.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/mfd-core.ko 2022-09-10T03:59:31.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/sm501.ko 2022-09-10T03:59:31.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/viperboard.ko 2022-09-10T03:59:31.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/vx855.ko 2022-09-10T03:59:31.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/mfd/wl1273-core.ko 2022-09-10T03:59:31.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc 2022-09-10T03:59:31.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/altera-stapl 2022-09-10T03:59:31.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2022-09-10T03:59:31.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/apds9802als.ko 2022-09-10T03:59:31.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/apds990x.ko 2022-09-10T03:59:31.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/bh1770glc.ko 2022-09-10T03:59:31.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/cb710 2022-09-10T03:59:31.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/cb710/cb710.ko 2022-09-10T03:59:31.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom 2022-09-10T03:59:31.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/at24.ko 2022-09-10T03:59:31.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/eeprom.ko 2022-09-10T03:59:31.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2022-09-10T03:59:31.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/eeprom/max6875.ko 2022-09-10T03:59:31.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/enclosure.ko 2022-09-10T03:59:31.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/hpilo.ko 2022-09-10T03:59:31.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/isl29003.ko 2022-09-10T03:59:31.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/isl29020.ko 2022-09-10T03:59:31.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/lis3lv02d 2022-09-10T03:59:31.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2022-09-10T03:59:31.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2022-09-10T03:59:31.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei 2022-09-10T03:59:31.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei/mei-me.ko 2022-09-10T03:59:31.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei/mei-txe.ko 2022-09-10T03:59:31.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/mei/mei.ko 2022-09-10T03:59:31.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-gru 2022-09-10T03:59:31.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-gru/gru.ko 2022-09-10T03:59:31.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp 2022-09-10T03:59:31.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp/xp.ko 2022-09-10T03:59:31.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp/xpc.ko 2022-09-10T03:59:31.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/sgi-xp/xpnet.ko 2022-09-10T03:59:31.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/tifm_7xx1.ko 2022-09-10T03:59:31.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/tifm_core.ko 2022-09-10T03:59:31.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/tsl2550.ko 2022-09-10T03:59:31.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/vmw_balloon.ko 2022-09-10T03:59:31.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/vmw_vmci 2022-09-10T03:59:31.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2022-09-10T03:59:31.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net 2022-09-10T03:59:31.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/bonding 2022-09-10T03:59:31.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/bonding/bonding.ko 2022-09-10T03:59:31.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa 2022-09-10T03:59:31.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/b53 2022-09-10T03:59:31.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/b53/b53_common.ko 2022-09-10T03:59:31.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/bcm-sf2.ko 2022-09-10T03:59:31.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/mv88e6060.ko 2022-09-10T03:59:31.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/mv88e6xxx 2022-09-10T03:59:31.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2022-09-10T03:59:31.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/dummy.ko 2022-09-10T03:59:31.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/eql.ko 2022-09-10T03:59:31.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet 2022-09-10T03:59:31.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com 2022-09-10T03:59:31.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2022-09-10T03:59:31.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2022-09-10T03:59:31.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/3c59x.ko 2022-09-10T03:59:31.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/3com/typhoon.ko 2022-09-10T03:59:31.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390 2022-09-10T03:59:31.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/8390.ko 2022-09-10T03:59:31.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2022-09-10T03:59:31.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2022-09-10T03:59:31.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2022-09-10T03:59:31.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/adaptec 2022-09-10T03:59:31.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2022-09-10T03:59:31.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/agere 2022-09-10T03:59:31.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/agere/et131x.ko 2022-09-10T03:59:31.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/alteon 2022-09-10T03:59:31.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/alteon/acenic.ko 2022-09-10T03:59:31.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/altera 2022-09-10T03:59:31.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2022-09-10T03:59:31.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd 2022-09-10T03:59:31.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2022-09-10T03:59:31.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2022-09-10T03:59:31.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2022-09-10T03:59:31.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros 2022-09-10T03:59:31.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/alx 2022-09-10T03:59:31.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2022-09-10T03:59:31.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1c 2022-09-10T03:59:31.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2022-09-10T03:59:31.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1e 2022-09-10T03:59:31.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2022-09-10T03:59:31.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atlx 2022-09-10T03:59:31.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2022-09-10T03:59:31.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2022-09-10T03:59:31.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom 2022-09-10T03:59:31.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/b44.ko 2022-09-10T03:59:31.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2022-09-10T03:59:31.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/bnx2x 2022-09-10T03:59:31.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2022-09-10T03:59:31.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2022-09-10T03:59:31.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/genet 2022-09-10T03:59:31.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2022-09-10T03:59:31.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2022-09-10T03:59:31.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/brocade 2022-09-10T03:59:31.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/brocade/bna 2022-09-10T03:59:31.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2022-09-10T03:59:31.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio 2022-09-10T03:59:31.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb 2022-09-10T03:59:31.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2022-09-10T03:59:31.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb3 2022-09-10T03:59:31.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2022-09-10T03:59:31.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4 2022-09-10T03:59:31.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2022-09-10T03:59:31.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2022-09-10T03:59:31.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2022-09-10T03:59:31.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/libcxgb 2022-09-10T03:59:31.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2022-09-10T03:59:31.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/cisco 2022-09-10T03:59:31.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/cisco/enic 2022-09-10T03:59:31.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2022-09-10T03:59:31.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec 2022-09-10T03:59:31.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip 2022-09-10T03:59:31.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2022-09-10T03:59:31.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2022-09-10T03:59:31.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2022-09-10T03:59:31.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2022-09-10T03:59:31.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2022-09-10T03:59:31.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2022-09-10T03:59:31.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dlink 2022-09-10T03:59:31.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2022-09-10T03:59:31.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dlink/sundance.ko 2022-09-10T03:59:31.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/dnet.ko 2022-09-10T03:59:31.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/emulex 2022-09-10T03:59:31.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/emulex/benet 2022-09-10T03:59:31.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2022-09-10T03:59:31.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/ethoc.ko 2022-09-10T03:59:31.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/fealnx.ko 2022-09-10T03:59:31.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel 2022-09-10T03:59:31.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e100.ko 2022-09-10T03:59:31.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000 2022-09-10T03:59:31.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2022-09-10T03:59:31.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000e 2022-09-10T03:59:31.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2022-09-10T03:59:31.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/fm10k 2022-09-10T03:59:31.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2022-09-10T03:59:31.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/i40e 2022-09-10T03:59:31.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2022-09-10T03:59:31.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/iavf 2022-09-10T03:59:31.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2022-09-10T03:59:31.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igb 2022-09-10T03:59:31.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2022-09-10T03:59:31.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igbvf 2022-09-10T03:59:31.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2022-09-10T03:59:31.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgb 2022-09-10T03:59:31.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2022-09-10T03:59:31.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbe 2022-09-10T03:59:31.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2022-09-10T03:59:31.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbevf 2022-09-10T03:59:31.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2022-09-10T03:59:31.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/jme.ko 2022-09-10T03:59:31.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell 2022-09-10T03:59:31.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2022-09-10T03:59:31.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell/skge.ko 2022-09-10T03:59:31.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/marvell/sky2.ko 2022-09-10T03:59:31.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox 2022-09-10T03:59:31.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx4 2022-09-10T03:59:31.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2022-09-10T03:59:31.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2022-09-10T03:59:31.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx5 2022-09-10T03:59:31.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2022-09-10T03:59:31.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2022-09-10T03:59:31.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/micrel 2022-09-10T03:59:31.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2022-09-10T03:59:31.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/myricom 2022-09-10T03:59:31.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/myricom/myri10ge 2022-09-10T03:59:31.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2022-09-10T03:59:31.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/natsemi 2022-09-10T03:59:31.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2022-09-10T03:59:31.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2022-09-10T03:59:31.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/neterion 2022-09-10T03:59:31.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/neterion/s2io.ko 2022-09-10T03:59:31.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/nvidia 2022-09-10T03:59:31.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2022-09-10T03:59:31.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/packetengines 2022-09-10T03:59:31.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2022-09-10T03:59:31.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2022-09-10T03:59:31.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic 2022-09-10T03:59:31.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/netxen 2022-09-10T03:59:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2022-09-10T03:59:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2022-09-10T03:59:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/qlcnic 2022-09-10T03:59:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2022-09-10T03:59:31.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rdc 2022-09-10T03:59:31.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rdc/r6040.ko 2022-09-10T03:59:31.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek 2022-09-10T03:59:31.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2022-09-10T03:59:31.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/8139too.ko 2022-09-10T03:59:31.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/atp.ko 2022-09-10T03:59:31.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/realtek/r8169.ko 2022-09-10T03:59:31.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rocker 2022-09-10T03:59:31.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/rocker/rocker.ko 2022-09-10T03:59:31.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sfc 2022-09-10T03:59:31.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sfc/sfc.ko 2022-09-10T03:59:31.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/silan 2022-09-10T03:59:31.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/silan/sc92031.ko 2022-09-10T03:59:31.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sis 2022-09-10T03:59:31.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sis/sis190.ko 2022-09-10T03:59:31.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sis/sis900.ko 2022-09-10T03:59:31.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc 2022-09-10T03:59:31.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/epic100.ko 2022-09-10T03:59:31.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2022-09-10T03:59:31.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2022-09-10T03:59:31.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2022-09-10T03:59:31.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro 2022-09-10T03:59:31.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro/stmmac 2022-09-10T03:59:31.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2022-09-10T03:59:31.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2022-09-10T03:59:31.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun 2022-09-10T03:59:31.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/cassini.ko 2022-09-10T03:59:31.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/niu.ko 2022-09-10T03:59:31.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/sungem.ko 2022-09-10T03:59:31.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/sun/sunhme.ko 2022-09-10T03:59:31.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/tehuti 2022-09-10T03:59:31.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2022-09-10T03:59:31.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/ti 2022-09-10T03:59:31.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/ti/tlan.ko 2022-09-10T03:59:31.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/via 2022-09-10T03:59:31.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/via/via-rhine.ko 2022-09-10T03:59:31.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/via/via-velocity.ko 2022-09-10T03:59:31.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/wiznet 2022-09-10T03:59:31.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2022-09-10T03:59:31.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2022-09-10T03:59:31.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/xircom 2022-09-10T03:59:31.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2022-09-10T03:59:31.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/geneve.ko 2022-09-10T03:59:31.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/hyperv 2022-09-10T03:59:31.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/hyperv/hv_netvsc.ko 2022-09-10T03:59:31.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ieee802154 2022-09-10T03:59:31.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ieee802154/fakelb.ko 2022-09-10T03:59:31.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ifb.ko 2022-09-10T03:59:31.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ipvlan 2022-09-10T03:59:31.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ipvlan/ipvlan.ko 2022-09-10T03:59:31.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/macvlan.ko 2022-09-10T03:59:31.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/macvtap.ko 2022-09-10T03:59:31.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio 2022-09-10T03:59:31.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio.ko 2022-09-10T03:59:31.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2022-09-10T03:59:31.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mdio/mdio-bitbang.ko 2022-09-10T03:59:31.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/mii.ko 2022-09-10T03:59:31.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/net_failover.ko 2022-09-10T03:59:31.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/netconsole.ko 2022-09-10T03:59:31.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/nlmon.ko 2022-09-10T03:59:31.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/pcs 2022-09-10T03:59:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/pcs/pcs_xpcs.ko 2022-09-10T03:59:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy 2022-09-10T03:59:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/amd.ko 2022-09-10T03:59:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm-phy-lib.ko 2022-09-10T03:59:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2022-09-10T03:59:31.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm7xxx.ko 2022-09-10T03:59:31.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/bcm87xx.ko 2022-09-10T03:59:31.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/broadcom.ko 2022-09-10T03:59:31.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/cicada.ko 2022-09-10T03:59:31.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/davicom.ko 2022-09-10T03:59:31.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/dp83640.ko 2022-09-10T03:59:31.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/et1011c.ko 2022-09-10T03:59:31.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/icplus.ko 2022-09-10T03:59:31.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/lxt.ko 2022-09-10T03:59:31.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/marvell.ko 2022-09-10T03:59:31.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/micrel.ko 2022-09-10T03:59:31.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/national.ko 2022-09-10T03:59:31.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/phylink.ko 2022-09-10T03:59:31.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/qsemi.ko 2022-09-10T03:59:31.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/realtek.ko 2022-09-10T03:59:31.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/smsc.ko 2022-09-10T03:59:31.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/ste10Xp.ko 2022-09-10T03:59:31.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/phy/vitesse.ko 2022-09-10T03:59:31.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp 2022-09-10T03:59:31.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/bsd_comp.ko 2022-09-10T03:59:31.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_async.ko 2022-09-10T03:59:31.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_deflate.ko 2022-09-10T03:59:31.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_generic.ko 2022-09-10T03:59:31.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_mppe.ko 2022-09-10T03:59:31.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/ppp_synctty.ko 2022-09-10T03:59:31.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/pppoe.ko 2022-09-10T03:59:31.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/pppox.ko 2022-09-10T03:59:31.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/ppp/pptp.ko 2022-09-10T03:59:31.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/slip 2022-09-10T03:59:31.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/slip/slhc.ko 2022-09-10T03:59:31.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/slip/slip.ko 2022-09-10T03:59:31.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/sungem_phy.ko 2022-09-10T03:59:31.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/tap.ko 2022-09-10T03:59:31.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team 2022-09-10T03:59:31.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team.ko 2022-09-10T03:59:31.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_activebackup.ko 2022-09-10T03:59:31.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_broadcast.ko 2022-09-10T03:59:31.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_loadbalance.ko 2022-09-10T03:59:31.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_random.ko 2022-09-10T03:59:31.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/team/team_mode_roundrobin.ko 2022-09-10T03:59:31.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/tun.ko 2022-09-10T03:59:31.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/veth.ko 2022-09-10T03:59:31.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/virtio_net.ko 2022-09-10T03:59:31.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vmxnet3 2022-09-10T03:59:31.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2022-09-10T03:59:31.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vxlan 2022-09-10T03:59:31.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/vxlan/vxlan.ko 2022-09-10T03:59:31.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/xen-netback 2022-09-10T03:59:31.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/xen-netback/xen-netback.ko 2022-09-10T03:59:31.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/net/xen-netfront.ko 2022-09-10T03:59:31.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ntb 2022-09-10T03:59:31.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ntb/ntb.ko 2022-09-10T03:59:31.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme 2022-09-10T03:59:31.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme/host 2022-09-10T03:59:31.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme/host/nvme-core.ko 2022-09-10T03:59:31.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/nvme/host/nvme.ko 2022-09-10T03:59:31.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport 2022-09-10T03:59:31.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport.ko 2022-09-10T03:59:31.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport_cs.ko 2022-09-10T03:59:31.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport_pc.ko 2022-09-10T03:59:31.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/parport/parport_serial.ko 2022-09-10T03:59:31.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci 2022-09-10T03:59:31.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/hotplug 2022-09-10T03:59:31.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2022-09-10T03:59:31.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/pcie 2022-09-10T03:59:31.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/pcie/aer_inject.ko 2022-09-10T03:59:31.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pci/xen-pcifront.ko 2022-09-10T03:59:31.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia 2022-09-10T03:59:31.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia/i82092.ko 2022-09-10T03:59:31.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia/pd6729.ko 2022-09-10T03:59:31.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pcmcia/yenta_socket.ko 2022-09-10T03:59:31.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pinctrl 2022-09-10T03:59:31.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pinctrl/intel 2022-09-10T03:59:31.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2022-09-10T03:59:31.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform 2022-09-10T03:59:31.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86 2022-09-10T03:59:31.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/acer-wmi.ko 2022-09-10T03:59:31.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/acerhdf.ko 2022-09-10T03:59:31.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/apple-gmux.ko 2022-09-10T03:59:31.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/asus-laptop.ko 2022-09-10T03:59:31.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2022-09-10T03:59:31.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/asus-wmi.ko 2022-09-10T03:59:31.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/classmate-laptop.ko 2022-09-10T03:59:31.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/eeepc-laptop.ko 2022-09-10T03:59:31.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/eeepc-wmi.ko 2022-09-10T03:59:31.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2022-09-10T03:59:31.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2022-09-10T03:59:31.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/hdaps.ko 2022-09-10T03:59:31.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/hp-wmi.ko 2022-09-10T03:59:31.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/hp_accel.ko 2022-09-10T03:59:31.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/intel 2022-09-10T03:59:31.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/intel/intel-rst.ko 2022-09-10T03:59:31.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/intel_ips.ko 2022-09-10T03:59:31.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/msi-wmi.ko 2022-09-10T03:59:31.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/mxm-wmi.ko 2022-09-10T03:59:31.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/panasonic-laptop.ko 2022-09-10T03:59:31.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/samsung-laptop.ko 2022-09-10T03:59:31.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/samsung-q10.ko 2022-09-10T03:59:31.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/topstar-laptop.ko 2022-09-10T03:59:31.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2022-09-10T03:59:31.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/toshiba_haps.ko 2022-09-10T03:59:31.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/wmi-bmof.ko 2022-09-10T03:59:31.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/platform/x86/wmi.ko 2022-09-10T03:59:31.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/powercap 2022-09-10T03:59:31.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/powercap/intel_rapl_common.ko 2022-09-10T03:59:31.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/powercap/intel_rapl_msr.ko 2022-09-10T03:59:31.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps 2022-09-10T03:59:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients 2022-09-10T03:59:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients/pps-gpio.ko 2022-09-10T03:59:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients/pps-ldisc.ko 2022-09-10T03:59:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/clients/pps_parport.ko 2022-09-10T03:59:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/pps/pps_core.ko 2022-09-10T03:59:31.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ptp 2022-09-10T03:59:31.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ptp/ptp.ko 2022-09-10T03:59:31.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ptp/ptp_kvm.ko 2022-09-10T03:59:31.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc 2022-09-10T03:59:31.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-bq32k.ko 2022-09-10T03:59:31.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-bq4802.ko 2022-09-10T03:59:31.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1286.ko 2022-09-10T03:59:31.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1307.ko 2022-09-10T03:59:31.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1374.ko 2022-09-10T03:59:31.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1511.ko 2022-09-10T03:59:31.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1553.ko 2022-09-10T03:59:31.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1672.ko 2022-09-10T03:59:31.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds1742.ko 2022-09-10T03:59:31.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds2404.ko 2022-09-10T03:59:31.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-ds3232.ko 2022-09-10T03:59:31.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-em3027.ko 2022-09-10T03:59:31.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-fm3130.ko 2022-09-10T03:59:31.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-isl12022.ko 2022-09-10T03:59:31.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-isl1208.ko 2022-09-10T03:59:31.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-m41t80.ko 2022-09-10T03:59:31.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-m48t35.ko 2022-09-10T03:59:31.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-m48t59.ko 2022-09-10T03:59:31.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-max6900.ko 2022-09-10T03:59:31.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-msm6242.ko 2022-09-10T03:59:31.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf2127.ko 2022-09-10T03:59:31.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf85063.ko 2022-09-10T03:59:31.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf8523.ko 2022-09-10T03:59:31.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf8563.ko 2022-09-10T03:59:31.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-pcf8583.ko 2022-09-10T03:59:31.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rp5c01.ko 2022-09-10T03:59:31.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rs5c372.ko 2022-09-10T03:59:31.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rv3029c2.ko 2022-09-10T03:59:31.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rx8025.ko 2022-09-10T03:59:31.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-rx8581.ko 2022-09-10T03:59:31.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-stk17ta8.ko 2022-09-10T03:59:31.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-v3020.ko 2022-09-10T03:59:31.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/rtc/rtc-x1205.ko 2022-09-10T03:59:31.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi 2022-09-10T03:59:31.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/3w-9xxx.ko 2022-09-10T03:59:31.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/3w-sas.ko 2022-09-10T03:59:31.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/3w-xxxx.ko 2022-09-10T03:59:31.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/BusLogic.ko 2022-09-10T03:59:31.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/a100u2w.ko 2022-09-10T03:59:31.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aacraid 2022-09-10T03:59:31.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aacraid/aacraid.ko 2022-09-10T03:59:31.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/advansys.ko 2022-09-10T03:59:31.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aic7xxx 2022-09-10T03:59:31.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2022-09-10T03:59:31.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2022-09-10T03:59:31.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/am53c974.ko 2022-09-10T03:59:31.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/arcmsr 2022-09-10T03:59:31.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2022-09-10T03:59:31.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/atp870u.ko 2022-09-10T03:59:31.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/be2iscsi 2022-09-10T03:59:31.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2022-09-10T03:59:31.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bfa 2022-09-10T03:59:31.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bfa/bfa.ko 2022-09-10T03:59:31.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2fc 2022-09-10T03:59:31.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2022-09-10T03:59:31.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2i 2022-09-10T03:59:31.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2022-09-10T03:59:31.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ch.ko 2022-09-10T03:59:31.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/csiostor 2022-09-10T03:59:31.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/csiostor/csiostor.ko 2022-09-10T03:59:31.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi 2022-09-10T03:59:31.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb3i 2022-09-10T03:59:31.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2022-09-10T03:59:31.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb4i 2022-09-10T03:59:31.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2022-09-10T03:59:31.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2022-09-10T03:59:31.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/dc395x.ko 2022-09-10T03:59:31.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler 2022-09-10T03:59:31.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2022-09-10T03:59:31.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2022-09-10T03:59:31.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2022-09-10T03:59:31.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2022-09-10T03:59:31.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/dmx3191d.ko 2022-09-10T03:59:31.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/esas2r 2022-09-10T03:59:31.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/esas2r/esas2r.ko 2022-09-10T03:59:31.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/esp_scsi.ko 2022-09-10T03:59:31.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fcoe 2022-09-10T03:59:31.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fcoe/fcoe.ko 2022-09-10T03:59:31.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fcoe/libfcoe.ko 2022-09-10T03:59:31.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fnic 2022-09-10T03:59:31.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/fnic/fnic.ko 2022-09-10T03:59:31.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/hpsa.ko 2022-09-10T03:59:31.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/hptiop.ko 2022-09-10T03:59:31.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/hv_storvsc.ko 2022-09-10T03:59:31.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/initio.ko 2022-09-10T03:59:31.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ipr.ko 2022-09-10T03:59:31.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ips.ko 2022-09-10T03:59:31.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/isci 2022-09-10T03:59:31.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/isci/isci.ko 2022-09-10T03:59:31.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2022-09-10T03:59:31.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/iscsi_tcp.ko 2022-09-10T03:59:31.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libfc 2022-09-10T03:59:31.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libfc/libfc.ko 2022-09-10T03:59:31.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libiscsi.ko 2022-09-10T03:59:31.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libiscsi_tcp.ko 2022-09-10T03:59:31.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libsas 2022-09-10T03:59:31.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/libsas/libsas.ko 2022-09-10T03:59:31.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/lpfc 2022-09-10T03:59:31.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/lpfc/lpfc.ko 2022-09-10T03:59:31.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid 2022-09-10T03:59:31.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid.ko 2022-09-10T03:59:31.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2022-09-10T03:59:31.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2022-09-10T03:59:31.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2022-09-10T03:59:31.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mpt3sas 2022-09-10T03:59:31.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2022-09-10T03:59:31.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mvsas 2022-09-10T03:59:31.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mvsas/mvsas.ko 2022-09-10T03:59:31.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/mvumi.ko 2022-09-10T03:59:31.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/pm8001 2022-09-10T03:59:31.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/pm8001/pm80xx.ko 2022-09-10T03:59:31.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/pmcraid.ko 2022-09-10T03:59:31.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla1280.ko 2022-09-10T03:59:31.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla2xxx 2022-09-10T03:59:31.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2022-09-10T03:59:31.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2022-09-10T03:59:31.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla4xxx 2022-09-10T03:59:31.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2022-09-10T03:59:31.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/raid_class.ko 2022-09-10T03:59:31.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_debug.ko 2022-09-10T03:59:31.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_fc.ko 2022-09-10T03:59:31.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2022-09-10T03:59:31.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_sas.ko 2022-09-10T03:59:31.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_spi.ko 2022-09-10T03:59:31.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/scsi_transport_srp.ko 2022-09-10T03:59:31.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/ses.ko 2022-09-10T03:59:31.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/st.ko 2022-09-10T03:59:31.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/stex.ko 2022-09-10T03:59:31.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/sym53c8xx_2 2022-09-10T03:59:31.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2022-09-10T03:59:31.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/virtio_scsi.ko 2022-09-10T03:59:31.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/vmw_pvscsi.ko 2022-09-10T03:59:31.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/scsi/wd719x.ko 2022-09-10T03:59:31.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ssb 2022-09-10T03:59:31.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ssb/ssb.ko 2022-09-10T03:59:31.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target 2022-09-10T03:59:31.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/iscsi 2022-09-10T03:59:31.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2022-09-10T03:59:31.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/loopback 2022-09-10T03:59:31.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/loopback/tcm_loop.ko 2022-09-10T03:59:31.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_file.ko 2022-09-10T03:59:31.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_iblock.ko 2022-09-10T03:59:31.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_mod.ko 2022-09-10T03:59:31.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_pscsi.ko 2022-09-10T03:59:31.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/target_core_user.ko 2022-09-10T03:59:31.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/tcm_fc 2022-09-10T03:59:31.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2022-09-10T03:59:31.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal 2022-09-10T03:59:31.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel 2022-09-10T03:59:31.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal 2022-09-10T03:59:31.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2022-09-10T03:59:31.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2022-09-10T03:59:31.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2022-09-10T03:59:31.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2022-09-10T03:59:31.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2022-09-10T03:59:31.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2022-09-10T03:59:31.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2022-09-10T03:59:31.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2022-09-10T03:59:31.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2022-09-10T03:59:31.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2022-09-10T03:59:31.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2022-09-10T03:59:31.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2022-09-10T03:59:31.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2022-09-10T03:59:31.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2022-09-10T03:59:31.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2022-09-10T03:59:31.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty 2022-09-10T03:59:31.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/ipwireless 2022-09-10T03:59:31.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/ipwireless/ipwireless.ko 2022-09-10T03:59:31.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/n_gsm.ko 2022-09-10T03:59:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/n_hdlc.ko 2022-09-10T03:59:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/nozomi.ko 2022-09-10T03:59:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial 2022-09-10T03:59:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/8250 2022-09-10T03:59:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/8250/serial_cs.ko 2022-09-10T03:59:31.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/arc_uart.ko 2022-09-10T03:59:31.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/jsm 2022-09-10T03:59:31.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/serial/jsm/jsm.ko 2022-09-10T03:59:31.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/tty/synclink_gt.ko 2022-09-10T03:59:31.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs 2022-09-10T03:59:31.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/core 2022-09-10T03:59:31.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/core/ufshcd-core.ko 2022-09-10T03:59:31.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/host 2022-09-10T03:59:31.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/ufs/host/ufshcd-pci.ko 2022-09-10T03:59:31.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio 2022-09-10T03:59:31.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio.ko 2022-09-10T03:59:31.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_aec.ko 2022-09-10T03:59:31.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_cif.ko 2022-09-10T03:59:31.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_pci_generic.ko 2022-09-10T03:59:31.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/uio/uio_sercos3.ko 2022-09-10T03:59:31.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb 2022-09-10T03:59:31.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class 2022-09-10T03:59:31.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/cdc-acm.ko 2022-09-10T03:59:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/cdc-wdm.ko 2022-09-10T03:59:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/usblp.ko 2022-09-10T03:59:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/class/usbtmc.ko 2022-09-10T03:59:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/host 2022-09-10T03:59:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/host/sl811-hcd.ko 2022-09-10T03:59:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/host/u132-hcd.ko 2022-09-10T03:59:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/image 2022-09-10T03:59:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/image/mdc800.ko 2022-09-10T03:59:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/image/microtek.ko 2022-09-10T03:59:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc 2022-09-10T03:59:31.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/adutux.ko 2022-09-10T03:59:31.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/appledisplay.ko 2022-09-10T03:59:31.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/emi26.ko 2022-09-10T03:59:31.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/emi62.ko 2022-09-10T03:59:31.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/ezusb.ko 2022-09-10T03:59:31.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/ftdi-elan.ko 2022-09-10T03:59:31.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/idmouse.ko 2022-09-10T03:59:31.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/iowarrior.ko 2022-09-10T03:59:31.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/isight_firmware.ko 2022-09-10T03:59:31.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/ldusb.ko 2022-09-10T03:59:31.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/legousbtower.ko 2022-09-10T03:59:31.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/sisusbvga 2022-09-10T03:59:31.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2022-09-10T03:59:31.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/trancevibrator.ko 2022-09-10T03:59:31.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/usb3503.ko 2022-09-10T03:59:31.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/usblcd.ko 2022-09-10T03:59:31.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/usbsevseg.ko 2022-09-10T03:59:31.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/uss720.ko 2022-09-10T03:59:31.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/misc/yurex.ko 2022-09-10T03:59:31.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/phy 2022-09-10T03:59:31.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/phy/phy-generic.ko 2022-09-10T03:59:31.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial 2022-09-10T03:59:31.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/aircable.ko 2022-09-10T03:59:31.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ark3116.ko 2022-09-10T03:59:31.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/belkin_sa.ko 2022-09-10T03:59:31.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ch341.ko 2022-09-10T03:59:31.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/cp210x.ko 2022-09-10T03:59:31.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/cyberjack.ko 2022-09-10T03:59:31.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/cypress_m8.ko 2022-09-10T03:59:31.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/digi_acceleport.ko 2022-09-10T03:59:31.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/empeg.ko 2022-09-10T03:59:31.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ftdi_sio.ko 2022-09-10T03:59:31.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/garmin_gps.ko 2022-09-10T03:59:31.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/io_edgeport.ko 2022-09-10T03:59:31.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/io_ti.ko 2022-09-10T03:59:31.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ipaq.ko 2022-09-10T03:59:31.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ipw.ko 2022-09-10T03:59:31.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ir-usb.ko 2022-09-10T03:59:31.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/iuu_phoenix.ko 2022-09-10T03:59:31.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/keyspan.ko 2022-09-10T03:59:31.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/keyspan_pda.ko 2022-09-10T03:59:31.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/kl5kusb105.ko 2022-09-10T03:59:31.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/kobil_sct.ko 2022-09-10T03:59:31.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/mct_u232.ko 2022-09-10T03:59:31.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/mos7720.ko 2022-09-10T03:59:31.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/mos7840.ko 2022-09-10T03:59:31.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/navman.ko 2022-09-10T03:59:31.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/omninet.ko 2022-09-10T03:59:31.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/opticon.ko 2022-09-10T03:59:31.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/option.ko 2022-09-10T03:59:31.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/oti6858.ko 2022-09-10T03:59:31.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/pl2303.ko 2022-09-10T03:59:31.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/qcaux.ko 2022-09-10T03:59:31.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/qcserial.ko 2022-09-10T03:59:31.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/quatech2.ko 2022-09-10T03:59:31.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/safe_serial.ko 2022-09-10T03:59:31.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/sierra.ko 2022-09-10T03:59:31.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/spcp8x5.ko 2022-09-10T03:59:31.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ssu100.ko 2022-09-10T03:59:31.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/symbolserial.ko 2022-09-10T03:59:31.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2022-09-10T03:59:31.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/usb-serial-simple.ko 2022-09-10T03:59:31.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/usb_debug.ko 2022-09-10T03:59:31.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/usb_wwan.ko 2022-09-10T03:59:31.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/visor.ko 2022-09-10T03:59:31.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/whiteheat.ko 2022-09-10T03:59:31.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/serial/xsens_mt.ko 2022-09-10T03:59:31.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage 2022-09-10T03:59:31.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/uas.ko 2022-09-10T03:59:31.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-alauda.ko 2022-09-10T03:59:31.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-cypress.ko 2022-09-10T03:59:31.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-datafab.ko 2022-09-10T03:59:31.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-eneub6250.ko 2022-09-10T03:59:31.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-freecom.ko 2022-09-10T03:59:31.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-isd200.ko 2022-09-10T03:59:31.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-jumpshot.ko 2022-09-10T03:59:31.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-karma.ko 2022-09-10T03:59:31.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-onetouch.ko 2022-09-10T03:59:31.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-realtek.ko 2022-09-10T03:59:31.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-sddr09.ko 2022-09-10T03:59:31.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-sddr55.ko 2022-09-10T03:59:31.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/ums-usbat.ko 2022-09-10T03:59:31.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/storage/usb-storage.ko 2022-09-10T03:59:31.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip 2022-09-10T03:59:31.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip/usbip-core.ko 2022-09-10T03:59:31.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip/usbip-host.ko 2022-09-10T03:59:31.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/usb/usbip/vhci-hcd.ko 2022-09-10T03:59:31.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio 2022-09-10T03:59:31.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/pci 2022-09-10T03:59:31.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2022-09-10T03:59:31.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/pci/vfio-pci.ko 2022-09-10T03:59:31.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/vfio.ko 2022-09-10T03:59:31.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/vfio_iommu_type1.ko 2022-09-10T03:59:31.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vfio/vfio_virqfd.ko 2022-09-10T03:59:31.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost 2022-09-10T03:59:31.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost.ko 2022-09-10T03:59:31.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost_iotlb.ko 2022-09-10T03:59:31.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost_net.ko 2022-09-10T03:59:31.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/vhost/vhost_scsi.ko 2022-09-10T03:59:31.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video 2022-09-10T03:59:31.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight 2022-09-10T03:59:31.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight/apple_bl.ko 2022-09-10T03:59:31.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight/lcd.ko 2022-09-10T03:59:31.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/video/backlight/platform_lcd.ko 2022-09-10T03:59:31.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio 2022-09-10T03:59:31.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio.ko 2022-09-10T03:59:31.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_balloon.ko 2022-09-10T03:59:31.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_mmio.ko 2022-09-10T03:59:31.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_pci.ko 2022-09-10T03:59:31.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2022-09-10T03:59:31.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2022-09-10T03:59:31.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/virtio/virtio_ring.ko 2022-09-10T03:59:31.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1 2022-09-10T03:59:31.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters 2022-09-10T03:59:31.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters/ds1wm.ko 2022-09-10T03:59:31.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters/ds2482.ko 2022-09-10T03:59:31.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/masters/ds2490.ko 2022-09-10T03:59:31.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves 2022-09-10T03:59:31.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2406.ko 2022-09-10T03:59:31.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2408.ko 2022-09-10T03:59:31.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2413.ko 2022-09-10T03:59:31.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2423.ko 2022-09-10T03:59:31.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2431.ko 2022-09-10T03:59:31.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2433.ko 2022-09-10T03:59:31.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2780.ko 2022-09-10T03:59:31.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds2781.ko 2022-09-10T03:59:31.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2022-09-10T03:59:31.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_smem.ko 2022-09-10T03:59:31.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/slaves/w1_therm.ko 2022-09-10T03:59:31.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/w1/wire.ko 2022-09-10T03:59:31.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog 2022-09-10T03:59:31.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/alim1535_wdt.ko 2022-09-10T03:59:31.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/alim7101_wdt.ko 2022-09-10T03:59:31.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/f71808e_wdt.ko 2022-09-10T03:59:31.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/hpwdt.ko 2022-09-10T03:59:31.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/i6300esb.ko 2022-09-10T03:59:31.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2022-09-10T03:59:31.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/iTCO_wdt.ko 2022-09-10T03:59:31.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/ib700wdt.ko 2022-09-10T03:59:31.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/ibmasr.ko 2022-09-10T03:59:31.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/ie6xx_wdt.ko 2022-09-10T03:59:31.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/it8712f_wdt.ko 2022-09-10T03:59:31.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/it87_wdt.ko 2022-09-10T03:59:31.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/machzwd.ko 2022-09-10T03:59:31.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/nv_tco.ko 2022-09-10T03:59:31.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/pcwd_pci.ko 2022-09-10T03:59:31.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/pcwd_usb.ko 2022-09-10T03:59:31.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2022-09-10T03:59:31.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/sch311x_wdt.ko 2022-09-10T03:59:31.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/softdog.ko 2022-09-10T03:59:31.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/sp5100_tco.ko 2022-09-10T03:59:31.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/via_wdt.ko 2022-09-10T03:59:31.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/w83627hf_wdt.ko 2022-09-10T03:59:31.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/w83877f_wdt.ko 2022-09-10T03:59:31.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/w83977f_wdt.ko 2022-09-10T03:59:31.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/wdt_pci.ko 2022-09-10T03:59:31.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/watchdog/xen_wdt.ko 2022-09-10T03:59:31.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen 2022-09-10T03:59:31.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-acpi-processor.ko 2022-09-10T03:59:31.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-evtchn.ko 2022-09-10T03:59:31.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-gntalloc.ko 2022-09-10T03:59:31.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-gntdev.ko 2022-09-10T03:59:31.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-pciback 2022-09-10T03:59:31.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2022-09-10T03:59:31.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xen-privcmd.ko 2022-09-10T03:59:31.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xenfs 2022-09-10T03:59:31.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/drivers/xen/xenfs/xenfs.ko 2022-09-10T03:59:31.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs 2022-09-10T03:59:31.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/9p 2022-09-10T03:59:31.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/9p/9p.ko 2022-09-10T03:59:31.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/affs 2022-09-10T03:59:31.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/affs/affs.ko 2022-09-10T03:59:31.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/befs 2022-09-10T03:59:31.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/befs/befs.ko 2022-09-10T03:59:31.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/binfmt_misc.ko 2022-09-10T03:59:31.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/btrfs 2022-09-10T03:59:31.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/btrfs/btrfs.ko 2022-09-10T03:59:31.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cachefiles 2022-09-10T03:59:31.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cachefiles/cachefiles.ko 2022-09-10T03:59:31.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ceph 2022-09-10T03:59:31.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ceph/ceph.ko 2022-09-10T03:59:31.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cifs 2022-09-10T03:59:31.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cifs/cifs.ko 2022-09-10T03:59:31.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/coda 2022-09-10T03:59:31.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/coda/coda.ko 2022-09-10T03:59:31.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cramfs 2022-09-10T03:59:31.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/cramfs/cramfs.ko 2022-09-10T03:59:31.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/dlm 2022-09-10T03:59:31.856 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/dlm/dlm.ko 2022-09-10T03:59:31.856 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ecryptfs 2022-09-10T03:59:31.856 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ecryptfs/ecryptfs.ko 2022-09-10T03:59:31.856 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/f2fs 2022-09-10T03:59:31.856 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/f2fs/f2fs.ko 2022-09-10T03:59:31.857 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat 2022-09-10T03:59:31.857 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat/fat.ko 2022-09-10T03:59:31.857 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat/msdos.ko 2022-09-10T03:59:31.857 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fat/vfat.ko 2022-09-10T03:59:31.858 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fscache 2022-09-10T03:59:31.858 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fscache/fscache.ko 2022-09-10T03:59:31.858 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fuse 2022-09-10T03:59:31.858 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fuse/cuse.ko 2022-09-10T03:59:31.858 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/fuse/fuse.ko 2022-09-10T03:59:31.859 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/gfs2 2022-09-10T03:59:31.859 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/gfs2/gfs2.ko 2022-09-10T03:59:31.859 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfs 2022-09-10T03:59:31.859 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfs/hfs.ko 2022-09-10T03:59:31.859 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfsplus 2022-09-10T03:59:31.860 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/hfsplus/hfsplus.ko 2022-09-10T03:59:31.860 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/isofs 2022-09-10T03:59:31.860 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/isofs/isofs.ko 2022-09-10T03:59:31.860 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/jfs 2022-09-10T03:59:31.861 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/jfs/jfs.ko 2022-09-10T03:59:31.861 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/lockd 2022-09-10T03:59:31.861 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/lockd/lockd.ko 2022-09-10T03:59:31.861 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/minix 2022-09-10T03:59:31.861 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/minix/minix.ko 2022-09-10T03:59:31.862 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/netfs 2022-09-10T03:59:31.862 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/netfs/netfs.ko 2022-09-10T03:59:31.862 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs 2022-09-10T03:59:31.862 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/blocklayout 2022-09-10T03:59:31.862 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2022-09-10T03:59:31.863 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/filelayout 2022-09-10T03:59:31.863 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2022-09-10T03:59:31.863 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/flexfilelayout 2022-09-10T03:59:31.863 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2022-09-10T03:59:31.864 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/nfs.ko 2022-09-10T03:59:31.864 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/nfsv3.ko 2022-09-10T03:59:31.864 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs/nfsv4.ko 2022-09-10T03:59:31.864 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs_common 2022-09-10T03:59:31.864 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs_common/grace.ko 2022-09-10T03:59:31.865 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfs_common/nfs_acl.ko 2022-09-10T03:59:31.865 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfsd 2022-09-10T03:59:31.865 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nfsd/nfsd.ko 2022-09-10T03:59:31.865 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nilfs2 2022-09-10T03:59:31.865 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nilfs2/nilfs2.ko 2022-09-10T03:59:31.866 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls 2022-09-10T03:59:31.866 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-celtic.ko 2022-09-10T03:59:31.866 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-centeuro.ko 2022-09-10T03:59:31.866 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-croatian.ko 2022-09-10T03:59:31.867 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-cyrillic.ko 2022-09-10T03:59:31.867 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-gaelic.ko 2022-09-10T03:59:31.867 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-greek.ko 2022-09-10T03:59:31.867 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-iceland.ko 2022-09-10T03:59:31.867 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-inuit.ko 2022-09-10T03:59:31.868 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-roman.ko 2022-09-10T03:59:31.868 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-romanian.ko 2022-09-10T03:59:31.868 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/mac-turkish.ko 2022-09-10T03:59:31.868 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp1250.ko 2022-09-10T03:59:31.868 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp1251.ko 2022-09-10T03:59:31.869 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp1255.ko 2022-09-10T03:59:31.869 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp737.ko 2022-09-10T03:59:31.869 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp775.ko 2022-09-10T03:59:31.869 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp850.ko 2022-09-10T03:59:31.869 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp852.ko 2022-09-10T03:59:31.870 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp855.ko 2022-09-10T03:59:31.870 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp857.ko 2022-09-10T03:59:31.870 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp860.ko 2022-09-10T03:59:31.870 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp861.ko 2022-09-10T03:59:31.871 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp862.ko 2022-09-10T03:59:31.871 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp863.ko 2022-09-10T03:59:31.871 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp864.ko 2022-09-10T03:59:31.871 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp865.ko 2022-09-10T03:59:31.871 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp866.ko 2022-09-10T03:59:31.872 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp869.ko 2022-09-10T03:59:31.872 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp874.ko 2022-09-10T03:59:31.872 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp932.ko 2022-09-10T03:59:31.872 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp936.ko 2022-09-10T03:59:31.872 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp949.ko 2022-09-10T03:59:31.873 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_cp950.ko 2022-09-10T03:59:31.873 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_euc-jp.ko 2022-09-10T03:59:31.873 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-1.ko 2022-09-10T03:59:31.873 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-13.ko 2022-09-10T03:59:31.874 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-14.ko 2022-09-10T03:59:31.874 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-15.ko 2022-09-10T03:59:31.874 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-2.ko 2022-09-10T03:59:31.874 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-3.ko 2022-09-10T03:59:31.874 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-4.ko 2022-09-10T03:59:31.875 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-5.ko 2022-09-10T03:59:31.875 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-6.ko 2022-09-10T03:59:31.875 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-7.ko 2022-09-10T03:59:31.875 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_iso8859-9.ko 2022-09-10T03:59:31.875 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_koi8-r.ko 2022-09-10T03:59:31.876 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_koi8-ru.ko 2022-09-10T03:59:31.876 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_koi8-u.ko 2022-09-10T03:59:31.876 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/nls/nls_utf8.ko 2022-09-10T03:59:31.876 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2 2022-09-10T03:59:31.876 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/cluster 2022-09-10T03:59:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2022-09-10T03:59:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlm 2022-09-10T03:59:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2022-09-10T03:59:31.877 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlmfs 2022-09-10T03:59:31.878 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2022-09-10T03:59:31.878 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2.ko 2022-09-10T03:59:31.878 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2022-09-10T03:59:31.878 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2022-09-10T03:59:31.878 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2022-09-10T03:59:31.879 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/overlayfs 2022-09-10T03:59:31.879 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/overlayfs/overlay.ko 2022-09-10T03:59:31.879 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/pstore 2022-09-10T03:59:31.879 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/pstore/ramoops.ko 2022-09-10T03:59:31.879 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/reiserfs 2022-09-10T03:59:31.880 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/reiserfs/reiserfs.ko 2022-09-10T03:59:31.880 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/romfs 2022-09-10T03:59:31.880 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/romfs/romfs.ko 2022-09-10T03:59:31.880 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/smbfs_common 2022-09-10T03:59:31.881 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/smbfs_common/cifs_arc4.ko 2022-09-10T03:59:31.881 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/smbfs_common/cifs_md4.ko 2022-09-10T03:59:31.881 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/squashfs 2022-09-10T03:59:31.881 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/squashfs/squashfs.ko 2022-09-10T03:59:31.881 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/sysv 2022-09-10T03:59:31.882 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/sysv/sysv.ko 2022-09-10T03:59:31.882 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/udf 2022-09-10T03:59:31.882 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/udf/udf.ko 2022-09-10T03:59:31.882 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ufs 2022-09-10T03:59:31.883 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/ufs/ufs.ko 2022-09-10T03:59:31.883 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/xfs 2022-09-10T03:59:31.883 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/fs/xfs/xfs.ko 2022-09-10T03:59:31.883 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel 2022-09-10T03:59:31.883 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/rcu 2022-09-10T03:59:31.884 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/rcu/rcutorture.ko 2022-09-10T03:59:31.884 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/torture.ko 2022-09-10T03:59:31.884 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/trace 2022-09-10T03:59:31.884 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/kernel/trace/ring_buffer_benchmark.ko 2022-09-10T03:59:31.884 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib 2022-09-10T03:59:31.885 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/asn1_encoder.ko 2022-09-10T03:59:31.885 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crc-itu-t.ko 2022-09-10T03:59:31.885 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crc8.ko 2022-09-10T03:59:31.885 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crypto 2022-09-10T03:59:31.886 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crypto/libarc4.ko 2022-09-10T03:59:31.886 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/crypto/libdes.ko 2022-09-10T03:59:31.886 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/libcrc32c.ko 2022-09-10T03:59:31.886 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lru_cache.ko 2022-09-10T03:59:31.886 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lz4 2022-09-10T03:59:31.887 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lz4/lz4_compress.ko 2022-09-10T03:59:31.887 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/lz4/lz4hc_compress.ko 2022-09-10T03:59:31.887 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/math 2022-09-10T03:59:31.887 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/math/cordic.ko 2022-09-10T03:59:31.888 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/raid6 2022-09-10T03:59:31.888 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/raid6/raid6_pq.ko 2022-09-10T03:59:31.888 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/reed_solomon 2022-09-10T03:59:31.888 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/reed_solomon/reed_solomon.ko 2022-09-10T03:59:31.888 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/ts_bm.ko 2022-09-10T03:59:31.889 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/ts_fsm.ko 2022-09-10T03:59:31.889 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/ts_kmp.ko 2022-09-10T03:59:31.889 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/zstd 2022-09-10T03:59:31.889 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/lib/zstd/zstd_compress.ko 2022-09-10T03:59:31.889 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/mm 2022-09-10T03:59:31.890 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/mm/hwpoison-inject.ko 2022-09-10T03:59:31.890 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net 2022-09-10T03:59:31.890 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan 2022-09-10T03:59:31.890 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/6lowpan.ko 2022-09-10T03:59:31.890 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_dest.ko 2022-09-10T03:59:31.891 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_fragment.ko 2022-09-10T03:59:31.891 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_hop.ko 2022-09-10T03:59:31.891 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_ipv6.ko 2022-09-10T03:59:31.891 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_mobility.ko 2022-09-10T03:59:31.892 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_routing.ko 2022-09-10T03:59:31.892 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/6lowpan/nhc_udp.ko 2022-09-10T03:59:31.892 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802 2022-09-10T03:59:31.892 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/garp.ko 2022-09-10T03:59:31.892 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/mrp.ko 2022-09-10T03:59:31.893 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/p8022.ko 2022-09-10T03:59:31.893 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/psnap.ko 2022-09-10T03:59:31.893 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/802/stp.ko 2022-09-10T03:59:31.893 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/8021q 2022-09-10T03:59:31.893 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/8021q/8021q.ko 2022-09-10T03:59:31.894 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p 2022-09-10T03:59:31.894 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet.ko 2022-09-10T03:59:31.894 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet_fd.ko 2022-09-10T03:59:31.894 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet_rdma.ko 2022-09-10T03:59:31.895 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/9p/9pnet_virtio.ko 2022-09-10T03:59:31.895 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/batman-adv 2022-09-10T03:59:31.895 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/batman-adv/batman-adv.ko 2022-09-10T03:59:31.895 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge 2022-09-10T03:59:31.895 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/br_netfilter.ko 2022-09-10T03:59:31.896 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/bridge.ko 2022-09-10T03:59:31.896 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter 2022-09-10T03:59:31.896 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_802_3.ko 2022-09-10T03:59:31.896 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_among.ko 2022-09-10T03:59:31.897 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_arp.ko 2022-09-10T03:59:31.897 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2022-09-10T03:59:31.897 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_dnat.ko 2022-09-10T03:59:31.897 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_ip.ko 2022-09-10T03:59:31.897 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_ip6.ko 2022-09-10T03:59:31.898 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_limit.ko 2022-09-10T03:59:31.898 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_log.ko 2022-09-10T03:59:31.898 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_mark.ko 2022-09-10T03:59:31.898 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2022-09-10T03:59:31.898 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_nflog.ko 2022-09-10T03:59:31.899 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2022-09-10T03:59:31.899 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_redirect.ko 2022-09-10T03:59:31.899 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_snat.ko 2022-09-10T03:59:31.899 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_stp.ko 2022-09-10T03:59:31.900 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebt_vlan.ko 2022-09-10T03:59:31.900 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtable_broute.ko 2022-09-10T03:59:31.900 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtable_filter.ko 2022-09-10T03:59:31.900 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtable_nat.ko 2022-09-10T03:59:31.900 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/ebtables.ko 2022-09-10T03:59:31.901 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2022-09-10T03:59:31.901 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2022-09-10T03:59:31.901 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2022-09-10T03:59:31.901 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ceph 2022-09-10T03:59:31.901 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ceph/libceph.ko 2022-09-10T03:59:31.902 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/core 2022-09-10T03:59:31.902 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/core/failover.ko 2022-09-10T03:59:31.902 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/core/pktgen.ko 2022-09-10T03:59:31.902 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp 2022-09-10T03:59:31.903 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp.ko 2022-09-10T03:59:31.903 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp_diag.ko 2022-09-10T03:59:31.903 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp_ipv4.ko 2022-09-10T03:59:31.903 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dccp/dccp_ipv6.ko 2022-09-10T03:59:31.903 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dns_resolver 2022-09-10T03:59:31.904 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dns_resolver/dns_resolver.ko 2022-09-10T03:59:31.904 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa 2022-09-10T03:59:31.904 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/dsa_core.ko 2022-09-10T03:59:31.904 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/tag_brcm.ko 2022-09-10T03:59:31.904 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/tag_dsa.ko 2022-09-10T03:59:31.905 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/dsa/tag_trailer.ko 2022-09-10T03:59:31.905 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154 2022-09-10T03:59:31.905 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/6lowpan 2022-09-10T03:59:31.905 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2022-09-10T03:59:31.905 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/ieee802154.ko 2022-09-10T03:59:31.906 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ieee802154/ieee802154_socket.ko 2022-09-10T03:59:31.906 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4 2022-09-10T03:59:31.906 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ah4.ko 2022-09-10T03:59:31.906 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/esp4.ko 2022-09-10T03:59:31.907 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/fou.ko 2022-09-10T03:59:31.907 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/gre.ko 2022-09-10T03:59:31.907 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/inet_diag.ko 2022-09-10T03:59:31.907 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ip_gre.ko 2022-09-10T03:59:31.907 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ip_tunnel.ko 2022-09-10T03:59:31.908 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ip_vti.ko 2022-09-10T03:59:31.908 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ipcomp.ko 2022-09-10T03:59:31.908 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/ipip.ko 2022-09-10T03:59:31.908 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter 2022-09-10T03:59:31.908 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/arp_tables.ko 2022-09-10T03:59:31.909 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2022-09-10T03:59:31.909 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/arptable_filter.ko 2022-09-10T03:59:31.909 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2022-09-10T03:59:31.909 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2022-09-10T03:59:31.909 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2022-09-10T03:59:31.910 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_ah.ko 2022-09-10T03:59:31.910 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2022-09-10T03:59:31.910 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2022-09-10T03:59:31.910 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_nat.ko 2022-09-10T03:59:31.911 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_raw.ko 2022-09-10T03:59:31.911 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/iptable_security.ko 2022-09-10T03:59:31.911 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2022-09-10T03:59:31.911 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2022-09-10T03:59:31.911 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2022-09-10T03:59:31.912 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2022-09-10T03:59:31.912 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2022-09-10T03:59:31.912 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2022-09-10T03:59:31.912 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2022-09-10T03:59:31.912 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2022-09-10T03:59:31.913 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2022-09-10T03:59:31.913 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2022-09-10T03:59:31.913 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_bic.ko 2022-09-10T03:59:31.913 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_dctcp.ko 2022-09-10T03:59:31.913 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_diag.ko 2022-09-10T03:59:31.914 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_highspeed.ko 2022-09-10T03:59:31.914 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_htcp.ko 2022-09-10T03:59:31.914 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_hybla.ko 2022-09-10T03:59:31.914 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_illinois.ko 2022-09-10T03:59:31.915 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_lp.ko 2022-09-10T03:59:31.915 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_scalable.ko 2022-09-10T03:59:31.915 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_vegas.ko 2022-09-10T03:59:31.915 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_veno.ko 2022-09-10T03:59:31.915 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_westwood.ko 2022-09-10T03:59:31.916 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tcp_yeah.ko 2022-09-10T03:59:31.916 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/tunnel4.ko 2022-09-10T03:59:31.916 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/udp_diag.ko 2022-09-10T03:59:31.916 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/udp_tunnel.ko 2022-09-10T03:59:31.916 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv4/xfrm4_tunnel.ko 2022-09-10T03:59:31.917 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6 2022-09-10T03:59:31.917 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ah6.ko 2022-09-10T03:59:31.917 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/esp6.ko 2022-09-10T03:59:31.917 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/fou6.ko 2022-09-10T03:59:31.917 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ip6_tunnel.ko 2022-09-10T03:59:31.918 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ip6_udp_tunnel.ko 2022-09-10T03:59:31.918 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ip6_vti.ko 2022-09-10T03:59:31.918 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/ipcomp6.ko 2022-09-10T03:59:31.918 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter 2022-09-10T03:59:31.919 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6_tables.ko 2022-09-10T03:59:31.919 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2022-09-10T03:59:31.919 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2022-09-10T03:59:31.919 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2022-09-10T03:59:31.919 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2022-09-10T03:59:31.920 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2022-09-10T03:59:31.920 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2022-09-10T03:59:31.920 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2022-09-10T03:59:31.920 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2022-09-10T03:59:31.920 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2022-09-10T03:59:31.921 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2022-09-10T03:59:31.921 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2022-09-10T03:59:31.921 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2022-09-10T03:59:31.921 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2022-09-10T03:59:31.922 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2022-09-10T03:59:31.922 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/ip6table_security.ko 2022-09-10T03:59:31.922 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2022-09-10T03:59:31.922 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2022-09-10T03:59:31.922 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2022-09-10T03:59:31.923 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2022-09-10T03:59:31.923 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2022-09-10T03:59:31.923 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2022-09-10T03:59:31.923 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2022-09-10T03:59:31.923 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2022-09-10T03:59:31.924 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/sit.ko 2022-09-10T03:59:31.924 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/tunnel6.ko 2022-09-10T03:59:31.924 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/ipv6/xfrm6_tunnel.ko 2022-09-10T03:59:31.924 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/key 2022-09-10T03:59:31.924 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/key/af_key.ko 2022-09-10T03:59:31.925 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp 2022-09-10T03:59:31.925 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_core.ko 2022-09-10T03:59:31.925 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_debugfs.ko 2022-09-10T03:59:31.925 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_eth.ko 2022-09-10T03:59:31.926 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_ip.ko 2022-09-10T03:59:31.926 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_ip6.ko 2022-09-10T03:59:31.926 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_netlink.ko 2022-09-10T03:59:31.926 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/l2tp/l2tp_ppp.ko 2022-09-10T03:59:31.926 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/llc 2022-09-10T03:59:31.927 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/llc/llc.ko 2022-09-10T03:59:31.927 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mac802154 2022-09-10T03:59:31.927 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mac802154/mac802154.ko 2022-09-10T03:59:31.927 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mpls 2022-09-10T03:59:31.927 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/mpls/mpls_gso.ko 2022-09-10T03:59:31.928 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter 2022-09-10T03:59:31.928 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset 2022-09-10T03:59:31.928 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set.ko 2022-09-10T03:59:31.928 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2022-09-10T03:59:31.929 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2022-09-10T03:59:31.929 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2022-09-10T03:59:31.929 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2022-09-10T03:59:31.929 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2022-09-10T03:59:31.929 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2022-09-10T03:59:31.930 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2022-09-10T03:59:31.930 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2022-09-10T03:59:31.930 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2022-09-10T03:59:31.930 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2022-09-10T03:59:31.930 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2022-09-10T03:59:31.931 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2022-09-10T03:59:31.931 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2022-09-10T03:59:31.931 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2022-09-10T03:59:31.931 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2022-09-10T03:59:31.932 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs 2022-09-10T03:59:31.932 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs.ko 2022-09-10T03:59:31.932 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2022-09-10T03:59:31.932 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2022-09-10T03:59:31.933 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2022-09-10T03:59:31.933 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2022-09-10T03:59:31.933 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2022-09-10T03:59:31.933 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2022-09-10T03:59:31.933 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2022-09-10T03:59:31.934 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2022-09-10T03:59:31.934 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2022-09-10T03:59:31.934 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2022-09-10T03:59:31.934 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2022-09-10T03:59:31.935 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2022-09-10T03:59:31.935 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2022-09-10T03:59:31.935 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conncount.ko 2022-09-10T03:59:31.935 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack.ko 2022-09-10T03:59:31.936 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_amanda.ko 2022-09-10T03:59:31.936 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2022-09-10T03:59:31.936 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_ftp.ko 2022-09-10T03:59:31.936 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_h323.ko 2022-09-10T03:59:31.936 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_irc.ko 2022-09-10T03:59:31.937 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2022-09-10T03:59:31.937 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_netlink.ko 2022-09-10T03:59:31.937 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_pptp.ko 2022-09-10T03:59:31.937 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_sane.ko 2022-09-10T03:59:31.937 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_sip.ko 2022-09-10T03:59:31.938 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_snmp.ko 2022-09-10T03:59:31.938 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_conntrack_tftp.ko 2022-09-10T03:59:31.938 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_dup_netdev.ko 2022-09-10T03:59:31.938 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_flow_table.ko 2022-09-10T03:59:31.938 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_flow_table_inet.ko 2022-09-10T03:59:31.939 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_log_syslog.ko 2022-09-10T03:59:31.939 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat.ko 2022-09-10T03:59:31.939 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_amanda.ko 2022-09-10T03:59:31.939 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_ftp.ko 2022-09-10T03:59:31.940 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_irc.ko 2022-09-10T03:59:31.940 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_sip.ko 2022-09-10T03:59:31.940 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_nat_tftp.ko 2022-09-10T03:59:31.940 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_synproxy_core.ko 2022-09-10T03:59:31.941 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nf_tables.ko 2022-09-10T03:59:31.941 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink.ko 2022-09-10T03:59:31.941 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_acct.ko 2022-09-10T03:59:31.941 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_log.ko 2022-09-10T03:59:31.941 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_osf.ko 2022-09-10T03:59:31.942 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nfnetlink_queue.ko 2022-09-10T03:59:31.942 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_chain_nat.ko 2022-09-10T03:59:31.942 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_compat.ko 2022-09-10T03:59:31.942 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_ct.ko 2022-09-10T03:59:31.942 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_dup_netdev.ko 2022-09-10T03:59:31.943 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_fib.ko 2022-09-10T03:59:31.943 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_fwd_netdev.ko 2022-09-10T03:59:31.943 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_hash.ko 2022-09-10T03:59:31.943 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_limit.ko 2022-09-10T03:59:31.944 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_log.ko 2022-09-10T03:59:31.944 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_masq.ko 2022-09-10T03:59:31.944 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_nat.ko 2022-09-10T03:59:31.944 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_numgen.ko 2022-09-10T03:59:31.944 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_objref.ko 2022-09-10T03:59:31.945 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_queue.ko 2022-09-10T03:59:31.945 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_quota.ko 2022-09-10T03:59:31.945 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_redir.ko 2022-09-10T03:59:31.945 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_reject.ko 2022-09-10T03:59:31.945 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_reject_inet.ko 2022-09-10T03:59:31.946 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_socket.ko 2022-09-10T03:59:31.946 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/nft_tunnel.ko 2022-09-10T03:59:31.946 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_AUDIT.ko 2022-09-10T03:59:31.946 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CHECKSUM.ko 2022-09-10T03:59:31.947 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CLASSIFY.ko 2022-09-10T03:59:31.947 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CONNSECMARK.ko 2022-09-10T03:59:31.947 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_CT.ko 2022-09-10T03:59:31.947 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_DSCP.ko 2022-09-10T03:59:31.947 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_HL.ko 2022-09-10T03:59:31.948 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_HMARK.ko 2022-09-10T03:59:31.948 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_IDLETIMER.ko 2022-09-10T03:59:31.948 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_LED.ko 2022-09-10T03:59:31.948 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_LOG.ko 2022-09-10T03:59:31.948 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_MASQUERADE.ko 2022-09-10T03:59:31.949 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_NETMAP.ko 2022-09-10T03:59:31.949 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_NFLOG.ko 2022-09-10T03:59:31.949 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_NFQUEUE.ko 2022-09-10T03:59:31.949 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_RATEEST.ko 2022-09-10T03:59:31.949 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_REDIRECT.ko 2022-09-10T03:59:31.950 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_SECMARK.ko 2022-09-10T03:59:31.950 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TCPMSS.ko 2022-09-10T03:59:31.950 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2022-09-10T03:59:31.950 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TEE.ko 2022-09-10T03:59:31.950 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TPROXY.ko 2022-09-10T03:59:31.951 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_TRACE.ko 2022-09-10T03:59:31.951 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_addrtype.ko 2022-09-10T03:59:31.951 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_bpf.ko 2022-09-10T03:59:31.952 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_cgroup.ko 2022-09-10T03:59:31.952 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_cluster.ko 2022-09-10T03:59:31.952 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_comment.ko 2022-09-10T03:59:31.952 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connbytes.ko 2022-09-10T03:59:31.953 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connlabel.ko 2022-09-10T03:59:31.953 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connlimit.ko 2022-09-10T03:59:31.953 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_connmark.ko 2022-09-10T03:59:31.953 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_conntrack.ko 2022-09-10T03:59:31.953 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_cpu.ko 2022-09-10T03:59:31.954 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_dccp.ko 2022-09-10T03:59:31.954 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_devgroup.ko 2022-09-10T03:59:31.954 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_dscp.ko 2022-09-10T03:59:31.954 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_ecn.ko 2022-09-10T03:59:31.955 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_esp.ko 2022-09-10T03:59:31.955 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_hashlimit.ko 2022-09-10T03:59:31.955 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_helper.ko 2022-09-10T03:59:31.955 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_hl.ko 2022-09-10T03:59:31.955 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_ipcomp.ko 2022-09-10T03:59:31.956 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_iprange.ko 2022-09-10T03:59:31.956 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_ipvs.ko 2022-09-10T03:59:31.956 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_l2tp.ko 2022-09-10T03:59:31.956 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_length.ko 2022-09-10T03:59:31.957 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_limit.ko 2022-09-10T03:59:31.957 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_mac.ko 2022-09-10T03:59:31.957 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_mark.ko 2022-09-10T03:59:31.957 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_multiport.ko 2022-09-10T03:59:31.957 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_nat.ko 2022-09-10T03:59:31.958 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_nfacct.ko 2022-09-10T03:59:31.958 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_osf.ko 2022-09-10T03:59:31.958 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_owner.ko 2022-09-10T03:59:31.958 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_physdev.ko 2022-09-10T03:59:31.958 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_pkttype.ko 2022-09-10T03:59:31.959 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_policy.ko 2022-09-10T03:59:31.959 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_quota.ko 2022-09-10T03:59:31.959 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_rateest.ko 2022-09-10T03:59:31.959 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_realm.ko 2022-09-10T03:59:31.959 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_recent.ko 2022-09-10T03:59:31.960 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_sctp.ko 2022-09-10T03:59:31.960 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_set.ko 2022-09-10T03:59:31.960 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_socket.ko 2022-09-10T03:59:31.960 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_state.ko 2022-09-10T03:59:31.960 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_statistic.ko 2022-09-10T03:59:31.961 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_string.ko 2022-09-10T03:59:31.961 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_tcpmss.ko 2022-09-10T03:59:31.961 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_time.ko 2022-09-10T03:59:31.961 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netfilter/xt_u32.ko 2022-09-10T03:59:31.962 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netlink 2022-09-10T03:59:31.962 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/netlink/netlink_diag.ko 2022-09-10T03:59:31.962 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/nsh 2022-09-10T03:59:31.962 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/nsh/nsh.ko 2022-09-10T03:59:31.963 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch 2022-09-10T03:59:31.963 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/openvswitch.ko 2022-09-10T03:59:31.963 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/vport-geneve.ko 2022-09-10T03:59:31.963 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/vport-gre.ko 2022-09-10T03:59:31.963 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/openvswitch/vport-vxlan.ko 2022-09-10T03:59:31.964 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/packet 2022-09-10T03:59:31.964 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/packet/af_packet_diag.ko 2022-09-10T03:59:31.964 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds 2022-09-10T03:59:31.964 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds/rds.ko 2022-09-10T03:59:31.965 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds/rds_rdma.ko 2022-09-10T03:59:31.965 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/rds/rds_tcp.ko 2022-09-10T03:59:31.965 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched 2022-09-10T03:59:31.965 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_csum.ko 2022-09-10T03:59:31.965 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_gact.ko 2022-09-10T03:59:31.966 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_ipt.ko 2022-09-10T03:59:31.966 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_mirred.ko 2022-09-10T03:59:31.966 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_nat.ko 2022-09-10T03:59:31.966 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_pedit.ko 2022-09-10T03:59:31.966 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_police.ko 2022-09-10T03:59:31.967 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_simple.ko 2022-09-10T03:59:31.967 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_skbedit.ko 2022-09-10T03:59:31.967 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/act_vlan.ko 2022-09-10T03:59:31.967 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_basic.ko 2022-09-10T03:59:31.968 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_bpf.ko 2022-09-10T03:59:31.968 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_flow.ko 2022-09-10T03:59:31.968 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_fw.ko 2022-09-10T03:59:31.968 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_route.ko 2022-09-10T03:59:31.968 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_rsvp.ko 2022-09-10T03:59:31.969 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_rsvp6.ko 2022-09-10T03:59:31.969 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_tcindex.ko 2022-09-10T03:59:31.969 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/cls_u32.ko 2022-09-10T03:59:31.969 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_cmp.ko 2022-09-10T03:59:31.970 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_ipset.ko 2022-09-10T03:59:31.970 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_meta.ko 2022-09-10T03:59:31.970 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_nbyte.ko 2022-09-10T03:59:31.970 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_text.ko 2022-09-10T03:59:31.970 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/em_u32.ko 2022-09-10T03:59:31.971 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_cbq.ko 2022-09-10T03:59:31.971 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_choke.ko 2022-09-10T03:59:31.971 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_codel.ko 2022-09-10T03:59:31.971 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_drr.ko 2022-09-10T03:59:31.971 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_dsmark.ko 2022-09-10T03:59:31.972 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_fq.ko 2022-09-10T03:59:31.972 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_gred.ko 2022-09-10T03:59:31.972 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_hfsc.ko 2022-09-10T03:59:31.972 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_hhf.ko 2022-09-10T03:59:31.972 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_htb.ko 2022-09-10T03:59:31.973 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_ingress.ko 2022-09-10T03:59:31.973 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_mqprio.ko 2022-09-10T03:59:31.973 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_multiq.ko 2022-09-10T03:59:31.973 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_netem.ko 2022-09-10T03:59:31.974 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_pie.ko 2022-09-10T03:59:31.974 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_plug.ko 2022-09-10T03:59:31.974 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_prio.ko 2022-09-10T03:59:31.974 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_qfq.ko 2022-09-10T03:59:31.975 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_red.ko 2022-09-10T03:59:31.975 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_sfb.ko 2022-09-10T03:59:31.975 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_sfq.ko 2022-09-10T03:59:31.975 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_tbf.ko 2022-09-10T03:59:31.975 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sched/sch_teql.ko 2022-09-10T03:59:31.976 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sctp 2022-09-10T03:59:31.976 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sctp/sctp.ko 2022-09-10T03:59:31.976 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sctp/sctp_diag.ko 2022-09-10T03:59:31.976 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc 2022-09-10T03:59:31.976 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/auth_gss 2022-09-10T03:59:31.977 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2022-09-10T03:59:31.977 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2022-09-10T03:59:31.977 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/sunrpc.ko 2022-09-10T03:59:31.977 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/xprtrdma 2022-09-10T03:59:31.978 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2022-09-10T03:59:31.978 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/tipc 2022-09-10T03:59:31.978 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/tipc/diag.ko 2022-09-10T03:59:31.978 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/tipc/tipc.ko 2022-09-10T03:59:31.978 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/unix 2022-09-10T03:59:31.979 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/unix/unix_diag.ko 2022-09-10T03:59:31.979 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock 2022-09-10T03:59:31.979 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2022-09-10T03:59:31.979 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2022-09-10T03:59:31.980 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vsock.ko 2022-09-10T03:59:31.980 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vsock_diag.ko 2022-09-10T03:59:31.980 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/vmw_vsock/vsock_loopback.ko 2022-09-10T03:59:31.980 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/xfrm 2022-09-10T03:59:31.980 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/net/xfrm/xfrm_ipcomp.ko 2022-09-10T03:59:31.981 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security 2022-09-10T03:59:31.981 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys 2022-09-10T03:59:31.981 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/encrypted-keys 2022-09-10T03:59:31.981 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2022-09-10T03:59:31.982 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/trusted-keys 2022-09-10T03:59:31.982 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/security/keys/trusted-keys/trusted.ko 2022-09-10T03:59:31.982 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/virt 2022-09-10T03:59:31.982 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/virt/lib 2022-09-10T03:59:31.982 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/kernel/virt/lib/irqbypass.ko 2022-09-10T03:59:31.983 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.alias 2022-09-10T03:59:31.983 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.alias.bin 2022-09-10T03:59:31.983 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.builtin 2022-09-10T03:59:31.983 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.builtin.bin 2022-09-10T03:59:31.984 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.builtin.modinfo 2022-09-10T03:59:31.984 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.dep 2022-09-10T03:59:31.984 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.dep.bin 2022-09-10T03:59:31.984 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.devname 2022-09-10T03:59:31.985 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.order 2022-09-10T03:59:31.985 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.softdep 2022-09-10T03:59:31.985 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.symbols 2022-09-10T03:59:31.985 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.0.0-rc4-ceph-ga127ffb7df2a/modules.symbols.bin 2022-09-10T03:59:31.985 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/lib/.build-id 2022-09-10T03:59:31.986 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/lib/.build-id/26 2022-09-10T03:59:31.986 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/lib/.build-id/26/f3c15f7f2668ff3dc963ef43e786eb787aa4cb 2022-09-10T03:59:31.987 DEBUG:teuthology.task.kernel:get_image_version: 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:31.987 DEBUG:teuthology.orchestra.run.smithi081:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2022-09-10T03:59:32.015 INFO:teuthology.orchestra.run.smithi081.stdout:postinstall scriptlet (using /bin/sh): 2022-09-10T03:59:32.015 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a -a -r /boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a ]; then 2022-09-10T03:59:32.016 INFO:teuthology.orchestra.run.smithi081.stdout:cp /boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a /boot/.vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-10T03:59:32.016 INFO:teuthology.orchestra.run.smithi081.stdout:cp /boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a /boot/.System.map-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-10T03:59:32.016 INFO:teuthology.orchestra.run.smithi081.stdout:rm -f /boot/vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a /boot/System.map-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:32.016 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/installkernel 6.0.0-rc4-ceph-ga127ffb7df2a /boot/.vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a-rpm /boot/.System.map-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-10T03:59:32.017 INFO:teuthology.orchestra.run.smithi081.stdout:rm -f /boot/.vmlinuz-6.0.0-rc4-ceph-ga127ffb7df2a-rpm /boot/.System.map-6.0.0-rc4-ceph-ga127ffb7df2a-rpm 2022-09-10T03:59:32.017 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2022-09-10T03:59:32.017 INFO:teuthology.orchestra.run.smithi081.stdout:preuninstall scriptlet (using /bin/sh): 2022-09-10T03:59:32.017 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2022-09-10T03:59:32.018 INFO:teuthology.orchestra.run.smithi081.stdout:new-kernel-pkg --remove 6.0.0-rc4-ceph-ga127ffb7df2a --rminitrd --initrdfile=/boot/initramfs-6.0.0-rc4-ceph-ga127ffb7df2a.img 2022-09-10T03:59:32.018 INFO:teuthology.orchestra.run.smithi081.stdout:elif [ -x /usr/bin/kernel-install ]; then 2022-09-10T03:59:32.018 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-install remove 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:32.019 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2022-09-10T03:59:32.019 INFO:teuthology.orchestra.run.smithi081.stdout:postuninstall scriptlet (using /bin/sh): 2022-09-10T03:59:32.019 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /sbin/update-bootloader ]; then 2022-09-10T03:59:32.019 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/update-bootloader --remove 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:32.019 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2022-09-10T03:59:32.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2022-09-10T03:59:32.098 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2022-09-10T03:59:32.099 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2022-09-10T03:59:32.099 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.02 2022-09-10T03:59:32.099 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 123.el8 2022-09-10T03:59:32.099 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2022-09-10T03:59:32.100 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2022-09-10T03:59:32.100 INFO:teuthology.orchestra.run.smithi081.stdout:Group : System Environment/Base 2022-09-10T03:59:32.100 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 9265450 2022-09-10T03:59:32.101 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2022-09-10T03:59:32.101 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2022-09-10T03:59:32.101 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2022-09-10T03:59:32.101 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2022-09-10T03:59:32.102 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2022-09-10T03:59:32.102 INFO:teuthology.orchestra.run.smithi081.stdout:Relocations : (not relocatable) 2022-09-10T03:59:32.102 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : Red Hat, Inc. 2022-09-10T03:59:32.103 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : Red Hat, Inc. 2022-09-10T03:59:32.103 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2022-09-10T03:59:32.103 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2022-09-10T03:59:32.103 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2022-09-10T03:59:32.104 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T03:59:32.104 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2022-09-10T03:59:32.104 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2022-09-10T03:59:32.104 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2022-09-10T03:59:32.104 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2022-09-10T03:59:32.105 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T03:59:32.105 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2022-09-10T03:59:32.106 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2022-09-10T03:59:32.106 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:32.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2022-09-10T03:59:35.210 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2022-09-10T03:59:37.112 INFO:teuthology.orchestra.run.smithi081.stderr:done 2022-09-10T03:59:37.114 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2022-09-10T03:59:37.131 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.cfsGOPbQ6R 2022-09-10T03:59:37.131 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cfsGOPbQ6R 2022-09-10T03:59:37.213 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.cfsGOPbQ6R 2022-09-10T03:59:37.377 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.cfsGOPbQ6R is 6KB 2022-09-10T03:59:37.426 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.cfsGOPbQ6R 2022-09-10T03:59:37.442 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2022-09-10T03:59:37.506 INFO:teuthology.orchestra.run.smithi081.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2022-09-10T03:59:37.507 INFO:teuthology.orchestra.run.smithi081.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2022-09-10T03:59:37.507 INFO:teuthology.orchestra.run.smithi081.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.0.0-rc4-ceph-ga127ffb7df2a.conf 2022-09-10T03:59:37.508 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T03:59:37.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2022-09-10T04:00:07.659 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-09-10T04:00:07.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-09-10T04:00:07.662 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T04:01:07.664 DEBUG:teuthology.orchestra.remote:timed out 2022-09-10T04:01:37.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-09-10T04:01:37.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T04:01:38.028 DEBUG:teuthology.orchestra.run.smithi081:> true 2022-09-10T04:01:38.496 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-09-10T04:01:38.497 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.0.0-rc4-ceph-ga127ffb7df2a"... 2022-09-10T04:01:38.497 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2022-09-10T04:01:38.554 INFO:teuthology.orchestra.run.smithi081.stdout:6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T04:01:38.554 DEBUG:teuthology.task.kernel:current kernel version is 6.0.0-rc4-ceph-ga127ffb7df2a vs 6.0.0-rc4-ceph-ga127ffb7df2a 2022-09-10T04:01:38.555 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-09-10T04:01:38.555 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2022-09-10T04:01:39.557 DEBUG:teuthology.parallel:result is None 2022-09-10T04:01:39.558 INFO:teuthology.run_tasks:Running task internal.base... 2022-09-10T04:01:39.566 INFO:teuthology.task.internal:Creating test directory... 2022-09-10T04:01:39.567 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-10T04:01:39.572 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-10T04:01:39.576 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-09-10T04:01:39.628 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-09-10T04:01:39.635 INFO:teuthology.run_tasks:Running task internal.archive... 2022-09-10T04:01:39.646 INFO:teuthology.task.internal:Creating archive directory... 2022-09-10T04:01:39.646 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-10T04:01:39.651 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-10T04:01:39.654 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-09-10T04:01:39.727 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-09-10T04:01:39.734 INFO:teuthology.task.internal:Enabling coredump saving... 2022-09-10T04:01:39.735 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-10T04:01:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T04:01:39.755 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2022-09-10T04:01:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T04:01:39.775 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-10T04:01:39.781 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-10T04:01:39.797 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-09-10T04:01:39.839 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-10T04:01:39.857 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-10T04:01:39.865 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-10T04:01:39.882 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-10T04:01:39.927 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-10T04:01:39.963 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-09-10T04:01:39.968 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-09-10T04:01:39.976 INFO:teuthology.task.internal:Configuring sudo... 2022-09-10T04:01:39.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-10T04:01:39.982 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-10T04:01:39.984 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-09-10T04:01:40.064 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-09-10T04:01:40.076 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-09-10T04:01:40.077 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-10T04:01:40.082 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-10T04:01:40.084 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-09-10T04:01:40.134 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-10T04:01:40.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-10T04:01:40.320 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-10T04:01:40.408 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-10T04:01:40.495 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:01:40.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-10T04:01:40.587 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-10T04:01:40.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-10T04:01:40.700 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-10T04:01:40.746 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-10T04:01:40.790 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:01:40.791 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-10T04:01:40.859 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-10T04:01:40.907 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2022-09-10T04:01:40.967 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-10T04:01:41.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2022-09-10T04:01:41.058 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:01:41.059 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-09-10T04:01:41.128 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2022-09-10T04:01:41.134 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2022-09-10T04:01:41.138 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2022-09-10T04:01:41.196 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-10T04:01:41.228 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-10T04:01:41.267 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2022-09-10T04:01:41.740 INFO:teuthology.run_tasks:Running task internal.timer... 2022-09-10T04:01:41.748 INFO:teuthology.task.internal:Starting timer... 2022-09-10T04:01:41.749 INFO:teuthology.run_tasks:Running task pcp... 2022-09-10T04:01:41.761 INFO:teuthology.run_tasks:Running task selinux... 2022-09-10T04:01:41.770 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2022-09-10T04:01:41.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2022-09-10T04:01:41.919 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2022-09-10T04:01:41.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2022-09-10T04:01:41.988 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2022-09-10T04:01:41.990 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2022-09-10T04:01:42.053 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2022-09-10T04:01:42.055 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-09-10T04:01:42.056 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2022-09-10T04:01:42.123 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2022-09-10T04:01:42.125 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2022-09-10T04:01:42.145 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2022-09-10T04:01:42.146 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2022-09-10T04:01:42.170 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2022-09-10T04:01:42.171 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive'} 2022-09-10T04:01:42.171 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-10T04:01:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T04:01:42.264 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-10T04:01:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T04:01:42.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2022-09-10T04:01:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T04:01:42.323 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-09-10T04:01:42.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2022-09-10T04:01:42.416 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2022-09-10T04:01:42.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2022-09-10T04:01:42.472 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-09-10T04:01:42.483 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2022-09-10T04:01:42.484 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-09-10T04:01:42.515 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-09-10T04:01:42.517 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi104.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-09-10T04:13:27.249 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2022-09-10T04:13:27.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-09-10T04:13:27.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T04:13:27.396 DEBUG:teuthology.orchestra.run.smithi081:> true 2022-09-10T04:13:27.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-09-10T04:13:27.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T04:13:27.608 DEBUG:teuthology.orchestra.run.smithi104:> true 2022-09-10T04:13:27.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2022-09-10T04:13:27.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-09-10T04:13:27.773 DEBUG:teuthology.orchestra.run.smithi110:> true 2022-09-10T04:13:27.850 INFO:teuthology.run_tasks:Running task clock... 2022-09-10T04:13:27.867 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-09-10T04:13:27.868 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-10T04:13:27.871 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-10T04:13:27.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-10T04:13:27.877 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-10T04:13:27.881 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-09-10T04:13:27.881 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-09-10T04:13:27.915 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-10T04:13:27.927 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-10T04:13:27.931 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-10T04:13:27.942 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-10T04:13:27.962 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2022-09-10T04:13:27.996 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2022-09-10T04:13:28.003 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2022-09-10T04:13:28.008 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2022-09-10T04:13:28.012 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2022-09-10T04:13:28.024 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-10T04:13:28.024 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2022-09-10T04:13:28.038 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-10T04:13:28.039 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-10T04:13:28.053 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-10T04:13:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2022-09-10T04:13:28.116 INFO:teuthology.orchestra.run.smithi104.stderr:bash: ntpq: command not found 2022-09-10T04:13:28.120 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-10T04:13:28.120 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2022-09-10T04:13:28.120 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.121 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.121 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.121 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.138 INFO:teuthology.orchestra.run.smithi110.stderr:bash: ntpq: command not found 2022-09-10T04:13:28.141 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-10T04:13:28.141 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2022-09-10T04:13:28.142 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.142 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.142 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.142 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.149 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2022-09-10T04:13:28.189 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2022-09-10T04:13:28.229 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2022-09-10T04:13:28.357 INFO:teuthology.orchestra.run.smithi081.stderr:bash: ntpq: command not found 2022-09-10T04:13:28.365 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2022-09-10T04:13:28.365 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2022-09-10T04:13:28.366 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.366 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.367 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.367 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2022-09-10T04:13:28.370 INFO:teuthology.run_tasks:Running task pexec... 2022-09-10T04:13:28.381 INFO:teuthology.task.pexec:Executing custom commands... 2022-09-10T04:13:28.382 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:13:28.382 DEBUG:teuthology.orchestra.run.smithi081:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-09-10T04:13:28.383 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi104.front.sepia.ceph.com 2022-09-10T04:13:28.383 DEBUG:teuthology.orchestra.run.smithi104:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-09-10T04:13:28.384 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi110.front.sepia.ceph.com 2022-09-10T04:13:28.384 DEBUG:teuthology.orchestra.run.smithi110:> TESTDIR=/home/ubuntu/cephtest bash -s 2022-09-10T04:13:28.755 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2022-09-10T04:13:28.756 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2022-09-10T04:13:28.904 INFO:teuthology.orchestra.run.smithi081.stdout:Updating Subscription Management repositories. 2022-09-10T04:13:29.217 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 156 kB/s | 2.8 kB 00:00 2022-09-10T04:13:29.259 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 138 kB/s | 2.4 kB 00:00 2022-09-10T04:13:29.444 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 153 kB/s | 2.8 kB 00:00 2022-09-10T04:13:29.485 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 123 kB/s | 2.4 kB 00:00 2022-09-10T04:13:29.501 INFO:teuthology.orchestra.run.smithi081.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 156 kB/s | 2.8 kB 00:00 2022-09-10T04:13:29.836 INFO:teuthology.orchestra.run.smithi081.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 128 kB/s | 2.4 kB 00:00 2022-09-10T04:13:31.434 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2022-09-10T04:13:31.435 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2022-09-10T04:13:31.436 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2022-09-10T04:13:31.436 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2022-09-10T04:13:31.436 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2022-09-10T04:13:31.436 INFO:teuthology.orchestra.run.smithi104.stdout:Resetting modules: 2022-09-10T04:13:31.437 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools 2022-09-10T04:13:31.437 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:13:31.437 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2022-09-10T04:13:31.437 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2022-09-10T04:13:31.437 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:13:31.438 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2022-09-10T04:13:31.438 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2022-09-10T04:13:31.438 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2022-09-10T04:13:31.439 INFO:teuthology.orchestra.run.smithi110.stdout:Resetting modules: 2022-09-10T04:13:31.439 INFO:teuthology.orchestra.run.smithi110.stdout: container-tools 2022-09-10T04:13:31.439 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:13:31.439 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2022-09-10T04:13:31.440 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2022-09-10T04:13:31.440 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:13:31.627 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2022-09-10T04:13:31.631 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2022-09-10T04:13:31.807 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2022-09-10T04:13:31.808 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2022-09-10T04:13:31.808 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2022-09-10T04:13:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2022-09-10T04:13:31.809 INFO:teuthology.orchestra.run.smithi081.stdout:Resetting modules: 2022-09-10T04:13:31.809 INFO:teuthology.orchestra.run.smithi081.stdout: container-tools 2022-09-10T04:13:31.810 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:13:31.810 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2022-09-10T04:13:31.810 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2022-09-10T04:13:31.810 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:13:32.058 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2022-09-10T04:13:32.092 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2022-09-10T04:13:32.098 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2022-09-10T04:13:32.536 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 149 kB/s | 2.8 kB 00:00 2022-09-10T04:13:32.596 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 122 kB/s | 2.4 kB 00:00 2022-09-10T04:13:32.703 INFO:teuthology.orchestra.run.smithi081.stdout:Updating Subscription Management repositories. 2022-09-10T04:13:32.771 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 145 kB/s | 2.8 kB 00:00 2022-09-10T04:13:32.810 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 136 kB/s | 2.4 kB 00:00 2022-09-10T04:13:33.312 INFO:teuthology.orchestra.run.smithi081.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 141 kB/s | 2.8 kB 00:00 2022-09-10T04:13:33.652 INFO:teuthology.orchestra.run.smithi081.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 131 kB/s | 2.4 kB 00:00 2022-09-10T04:13:34.622 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2022-09-10T04:13:34.625 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================================================= 2022-09-10T04:13:34.625 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2022-09-10T04:13:34.626 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================================================= 2022-09-10T04:13:34.626 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2022-09-10T04:13:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: conmon x86_64 2:2.1.2-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 55 k 2022-09-10T04:13:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: container-selinux noarch 2:2.188.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 59 k 2022-09-10T04:13:34.627 INFO:teuthology.orchestra.run.smithi104.stdout: containernetworking-plugins x86_64 1:1.1.1-3.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 18 M 2022-09-10T04:13:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: containers-common x86_64 2:1-35.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 109 k 2022-09-10T04:13:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: criu x86_64 3.15-3.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 518 k 2022-09-10T04:13:34.628 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-overlayfs x86_64 1.9-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 73 k 2022-09-10T04:13:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: libslirp x86_64 4.4.0-1.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 70 k 2022-09-10T04:13:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: runc x86_64 1:1.1.3-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 3.0 M 2022-09-10T04:13:34.629 INFO:teuthology.orchestra.run.smithi104.stdout: slirp4netns x86_64 1.2.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-10T04:13:34.630 INFO:teuthology.orchestra.run.smithi104.stdout:Installing group/module packages: 2022-09-10T04:13:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 8.0 M 2022-09-10T04:13:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 524 k 2022-09-10T04:13:34.630 INFO:teuthology.orchestra.run.smithi104.stdout: crun x86_64 1.4.5-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 209 k 2022-09-10T04:13:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 149 k 2022-09-10T04:13:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: skopeo x86_64 2:1.8.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 6.9 M 2022-09-10T04:13:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-09-10T04:13:34.631 INFO:teuthology.orchestra.run.smithi104.stdout: udica noarch 0.2.6-4.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 49 k 2022-09-10T04:13:34.631 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2022-09-10T04:13:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-09-10T04:13:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-09-10T04:13:34.632 INFO:teuthology.orchestra.run.smithi104.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-09-10T04:13:34.632 INFO:teuthology.orchestra.run.smithi104.stdout:Installing module profiles: 2022-09-10T04:13:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools/common 2022-09-10T04:13:34.633 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling module streams: 2022-09-10T04:13:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: container-tools rhel8 2022-09-10T04:13:34.633 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:13:34.633 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2022-09-10T04:13:34.634 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================================================= 2022-09-10T04:13:34.634 INFO:teuthology.orchestra.run.smithi104.stdout:Install 10 Packages 2022-09-10T04:13:34.634 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 9 Packages 2022-09-10T04:13:34.634 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:13:34.635 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 40 M 2022-09-10T04:13:34.635 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2022-09-10T04:13:34.684 INFO:teuthology.orchestra.run.smithi104.stdout:(1/19): python3-pyxdg-0.25-16.el8.noarch.rpm 1.7 MB/s | 94 kB 00:00 2022-09-10T04:13:34.701 INFO:teuthology.orchestra.run.smithi104.stdout:(2/19): python3-pytoml-0.1.14-5.git7dea353.el8. 345 kB/s | 25 kB 00:00 2022-09-10T04:13:34.718 INFO:teuthology.orchestra.run.smithi104.stdout:(3/19): yajl-2.1.0-10.el8.x86_64.rpm 459 kB/s | 41 kB 00:00 2022-09-10T04:13:34.741 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2022-09-10T04:13:34.745 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================================================= 2022-09-10T04:13:34.745 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2022-09-10T04:13:34.746 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================================================= 2022-09-10T04:13:34.746 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2022-09-10T04:13:34.746 INFO:teuthology.orchestra.run.smithi110.stdout: conmon x86_64 2:2.1.2-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 55 k 2022-09-10T04:13:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: container-selinux noarch 2:2.188.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 59 k 2022-09-10T04:13:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: containernetworking-plugins x86_64 1:1.1.1-3.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 18 M 2022-09-10T04:13:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: containers-common x86_64 2:1-35.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 109 k 2022-09-10T04:13:34.747 INFO:teuthology.orchestra.run.smithi110.stdout: criu x86_64 3.15-3.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 518 k 2022-09-10T04:13:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-overlayfs x86_64 1.9-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 73 k 2022-09-10T04:13:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: libslirp x86_64 4.4.0-1.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 70 k 2022-09-10T04:13:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: runc x86_64 1:1.1.3-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 3.0 M 2022-09-10T04:13:34.748 INFO:teuthology.orchestra.run.smithi110.stdout: slirp4netns x86_64 1.2.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-10T04:13:34.749 INFO:teuthology.orchestra.run.smithi110.stdout:Installing group/module packages: 2022-09-10T04:13:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 8.0 M 2022-09-10T04:13:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 524 k 2022-09-10T04:13:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: crun x86_64 1.4.5-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 209 k 2022-09-10T04:13:34.749 INFO:teuthology.orchestra.run.smithi110.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 149 k 2022-09-10T04:13:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: skopeo x86_64 2:1.8.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 6.9 M 2022-09-10T04:13:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-09-10T04:13:34.750 INFO:teuthology.orchestra.run.smithi110.stdout: udica noarch 0.2.6-4.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 49 k 2022-09-10T04:13:34.751 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2022-09-10T04:13:34.751 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-09-10T04:13:34.752 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-09-10T04:13:34.752 INFO:teuthology.orchestra.run.smithi110.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-09-10T04:13:34.753 INFO:teuthology.orchestra.run.smithi110.stdout:Installing module profiles: 2022-09-10T04:13:34.753 INFO:teuthology.orchestra.run.smithi110.stdout: container-tools/common 2022-09-10T04:13:34.754 INFO:teuthology.orchestra.run.smithi110.stdout:Enabling module streams: 2022-09-10T04:13:34.754 INFO:teuthology.orchestra.run.smithi110.stdout: container-tools rhel8 2022-09-10T04:13:34.755 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:13:34.755 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2022-09-10T04:13:34.755 INFO:teuthology.orchestra.run.smithi110.stdout:========================================================================================================================= 2022-09-10T04:13:34.756 INFO:teuthology.orchestra.run.smithi110.stdout:Install 10 Packages 2022-09-10T04:13:34.756 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 9 Packages 2022-09-10T04:13:34.757 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:13:34.757 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 40 M 2022-09-10T04:13:34.759 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2022-09-10T04:13:34.777 INFO:teuthology.orchestra.run.smithi104.stdout:(4/19): cockpit-podman-49.1-1.module+el8.6.0+15 6.8 MB/s | 524 kB 00:00 2022-09-10T04:13:34.802 INFO:teuthology.orchestra.run.smithi104.stdout:(5/19): udica-0.2.6-4.module+el8.6.0+15917+093c 598 kB/s | 49 kB 00:00 2022-09-10T04:13:34.802 INFO:teuthology.orchestra.run.smithi110.stdout:(1/19): python3-pytoml-0.1.14-5.git7dea353.el8. 457 kB/s | 25 kB 00:00 2022-09-10T04:13:34.827 INFO:teuthology.orchestra.run.smithi110.stdout:(2/19): yajl-2.1.0-10.el8.x86_64.rpm 512 kB/s | 41 kB 00:00 2022-09-10T04:13:34.852 INFO:teuthology.orchestra.run.smithi104.stdout:(6/19): toolbox-0.0.99.3-0.6.module+el8.6.0+159 13 MB/s | 2.2 MB 00:00 2022-09-10T04:13:34.853 INFO:teuthology.orchestra.run.smithi110.stdout:(3/19): python3-pyxdg-0.25-16.el8.noarch.rpm 891 kB/s | 94 kB 00:00 2022-09-10T04:13:34.877 INFO:teuthology.orchestra.run.smithi104.stdout:(7/19): crun-1.4.5-2.module+el8.6.0+15917+093ca 2.7 MB/s | 209 kB 00:00 2022-09-10T04:13:34.895 INFO:teuthology.orchestra.run.smithi110.stdout:(4/19): udica-0.2.6-4.module+el8.6.0+15917+093c 1.2 MB/s | 49 kB 00:00 2022-09-10T04:13:34.911 INFO:teuthology.orchestra.run.smithi104.stdout:(8/19): python3-podman-4.0.0-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2022-09-10T04:13:34.920 INFO:teuthology.orchestra.run.smithi110.stdout:(5/19): cockpit-podman-49.1-1.module+el8.6.0+15 5.6 MB/s | 524 kB 00:00 2022-09-10T04:13:34.945 INFO:teuthology.orchestra.run.smithi104.stdout:(9/19): container-selinux-2.188.0-1.module+el8. 1.7 MB/s | 59 kB 00:00 2022-09-10T04:13:34.970 INFO:teuthology.orchestra.run.smithi110.stdout:(6/19): toolbox-0.0.99.3-0.6.module+el8.6.0+159 13 MB/s | 2.2 MB 00:00 2022-09-10T04:13:35.004 INFO:teuthology.orchestra.run.smithi110.stdout:(7/19): crun-1.4.5-2.module+el8.6.0+15917+093ca 2.5 MB/s | 209 kB 00:00 2022-09-10T04:13:35.029 INFO:teuthology.orchestra.run.smithi110.stdout:(8/19): python3-podman-4.0.0-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2022-09-10T04:13:35.053 INFO:teuthology.orchestra.run.smithi104.stdout:(10/19): buildah-1.26.2-1.module+el8.6.0+15917+ 29 MB/s | 8.0 MB 00:00 2022-09-10T04:13:35.071 INFO:teuthology.orchestra.run.smithi110.stdout:(9/19): container-selinux-2.188.0-1.module+el8. 1.4 MB/s | 59 kB 00:00 2022-09-10T04:13:35.137 INFO:teuthology.orchestra.run.smithi104.stdout:(11/19): skopeo-1.8.0-2.module+el8.6.0+15917+09 27 MB/s | 6.9 MB 00:00 2022-09-10T04:13:35.187 INFO:teuthology.orchestra.run.smithi104.stdout:(12/19): runc-1.1.3-2.module+el8.6.0+15917+093c 12 MB/s | 3.0 MB 00:00 2022-09-10T04:13:35.205 INFO:teuthology.orchestra.run.smithi110.stdout:(10/19): buildah-1.26.2-1.module+el8.6.0+15917+ 26 MB/s | 8.0 MB 00:00 2022-09-10T04:13:35.220 INFO:teuthology.orchestra.run.smithi104.stdout:(13/19): conmon-2.1.2-2.module+el8.6.0+15917+09 332 kB/s | 55 kB 00:00 2022-09-10T04:13:35.239 INFO:teuthology.orchestra.run.smithi110.stdout:(11/19): conmon-2.1.2-2.module+el8.6.0+15917+09 1.6 MB/s | 55 kB 00:00 2022-09-10T04:13:35.262 INFO:teuthology.orchestra.run.smithi104.stdout:(14/19): fuse-overlayfs-1.9-1.module+el8.6.0+15 589 kB/s | 73 kB 00:00 2022-09-10T04:13:35.297 INFO:teuthology.orchestra.run.smithi110.stdout:(12/19): runc-1.1.3-2.module+el8.6.0+15917+093c 13 MB/s | 3.0 MB 00:00 2022-09-10T04:13:35.342 INFO:teuthology.orchestra.run.smithi104.stdout:(15/19): slirp4netns-1.2.0-2.module+el8.6.0+159 356 kB/s | 54 kB 00:00 2022-09-10T04:13:35.380 INFO:teuthology.orchestra.run.smithi104.stdout:(16/19): libslirp-4.4.0-1.module+el8.6.0+15875+ 446 kB/s | 70 kB 00:00 2022-09-10T04:13:35.389 INFO:teuthology.orchestra.run.smithi110.stdout:(13/19): skopeo-1.8.0-2.module+el8.6.0+15917+09 18 MB/s | 6.9 MB 00:00 2022-09-10T04:13:35.405 INFO:teuthology.orchestra.run.smithi104.stdout:(17/19): containers-common-1-35.module+el8.6.0+ 1.7 MB/s | 109 kB 00:00 2022-09-10T04:13:35.423 INFO:teuthology.orchestra.run.smithi110.stdout:(14/19): fuse-overlayfs-1.9-1.module+el8.6.0+15 399 kB/s | 73 kB 00:00 2022-09-10T04:13:35.439 INFO:teuthology.orchestra.run.smithi110.stdout:(15/19): slirp4netns-1.2.0-2.module+el8.6.0+159 385 kB/s | 54 kB 00:00 2022-09-10T04:13:35.447 INFO:teuthology.orchestra.run.smithi104.stdout:(18/19): criu-3.15-3.module+el8.6.0+15875+dc9a2 7.6 MB/s | 518 kB 00:00 2022-09-10T04:13:35.465 INFO:teuthology.orchestra.run.smithi110.stdout:(16/19): libslirp-4.4.0-1.module+el8.6.0+15875+ 948 kB/s | 70 kB 00:00 2022-09-10T04:13:35.507 INFO:teuthology.orchestra.run.smithi110.stdout:(17/19): containers-common-1-35.module+el8.6.0+ 1.6 MB/s | 109 kB 00:00 2022-09-10T04:13:35.532 INFO:teuthology.orchestra.run.smithi110.stdout:(18/19): criu-3.15-3.module+el8.6.0+15875+dc9a2 7.7 MB/s | 518 kB 00:00 2022-09-10T04:13:35.560 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2022-09-10T04:13:35.563 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================================================= 2022-09-10T04:13:35.563 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2022-09-10T04:13:35.563 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================================================= 2022-09-10T04:13:35.564 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2022-09-10T04:13:35.564 INFO:teuthology.orchestra.run.smithi081.stdout: conmon x86_64 2:2.1.2-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 55 k 2022-09-10T04:13:35.564 INFO:teuthology.orchestra.run.smithi081.stdout: container-selinux noarch 2:2.188.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 59 k 2022-09-10T04:13:35.565 INFO:teuthology.orchestra.run.smithi081.stdout: containernetworking-plugins x86_64 1:1.1.1-3.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 18 M 2022-09-10T04:13:35.565 INFO:teuthology.orchestra.run.smithi081.stdout: containers-common x86_64 2:1-35.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 109 k 2022-09-10T04:13:35.565 INFO:teuthology.orchestra.run.smithi081.stdout: criu x86_64 3.15-3.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 518 k 2022-09-10T04:13:35.566 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-overlayfs x86_64 1.9-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 73 k 2022-09-10T04:13:35.566 INFO:teuthology.orchestra.run.smithi081.stdout: libslirp x86_64 4.4.0-1.module+el8.6.0+15875+dc9a2b96 rhel-8-for-x86_64-appstream-rpms 70 k 2022-09-10T04:13:35.567 INFO:teuthology.orchestra.run.smithi081.stdout: runc x86_64 1:1.1.3-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 3.0 M 2022-09-10T04:13:35.567 INFO:teuthology.orchestra.run.smithi081.stdout: slirp4netns x86_64 1.2.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-10T04:13:35.567 INFO:teuthology.orchestra.run.smithi081.stdout:Installing group/module packages: 2022-09-10T04:13:35.568 INFO:teuthology.orchestra.run.smithi081.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 8.0 M 2022-09-10T04:13:35.568 INFO:teuthology.orchestra.run.smithi081.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 524 k 2022-09-10T04:13:35.568 INFO:teuthology.orchestra.run.smithi081.stdout: crun x86_64 1.4.5-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 209 k 2022-09-10T04:13:35.569 INFO:teuthology.orchestra.run.smithi081.stdout: python3-podman noarch 4.0.0-1.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 149 k 2022-09-10T04:13:35.569 INFO:teuthology.orchestra.run.smithi081.stdout: skopeo x86_64 2:1.8.0-2.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 6.9 M 2022-09-10T04:13:35.569 INFO:teuthology.orchestra.run.smithi081.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 2.2 M 2022-09-10T04:13:35.569 INFO:teuthology.orchestra.run.smithi081.stdout: udica noarch 0.2.6-4.module+el8.6.0+15917+093ca6f8 rhel-8-for-x86_64-appstream-rpms 49 k 2022-09-10T04:13:35.570 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2022-09-10T04:13:35.570 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2022-09-10T04:13:35.570 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2022-09-10T04:13:35.570 INFO:teuthology.orchestra.run.smithi081.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2022-09-10T04:13:35.571 INFO:teuthology.orchestra.run.smithi081.stdout:Installing module profiles: 2022-09-10T04:13:35.571 INFO:teuthology.orchestra.run.smithi081.stdout: container-tools/common 2022-09-10T04:13:35.571 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling module streams: 2022-09-10T04:13:35.571 INFO:teuthology.orchestra.run.smithi081.stdout: container-tools rhel8 2022-09-10T04:13:35.571 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:13:35.572 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2022-09-10T04:13:35.572 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================================================= 2022-09-10T04:13:35.572 INFO:teuthology.orchestra.run.smithi081.stdout:Install 10 Packages 2022-09-10T04:13:35.572 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 9 Packages 2022-09-10T04:13:35.573 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:13:35.573 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 40 M 2022-09-10T04:13:35.573 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2022-09-10T04:13:35.625 INFO:teuthology.orchestra.run.smithi081.stdout:(1/19): yajl-2.1.0-10.el8.x86_64.rpm 711 kB/s | 41 kB 00:00 2022-09-10T04:13:35.651 INFO:teuthology.orchestra.run.smithi081.stdout:(2/19): python3-pytoml-0.1.14-5.git7dea353.el8. 299 kB/s | 25 kB 00:00 2022-09-10T04:13:35.676 INFO:teuthology.orchestra.run.smithi081.stdout:(3/19): python3-pyxdg-0.25-16.el8.noarch.rpm 867 kB/s | 94 kB 00:00 2022-09-10T04:13:35.680 INFO:teuthology.orchestra.run.smithi104.stdout:(19/19): containernetworking-plugins-1.1.1-3.mo 44 MB/s | 18 MB 00:00 2022-09-10T04:13:35.681 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:13:35.681 INFO:teuthology.orchestra.run.smithi104.stdout:Total 38 MB/s | 40 MB 00:01 2022-09-10T04:13:35.727 INFO:teuthology.orchestra.run.smithi081.stdout:(4/19): udica-0.2.6-4.module+el8.6.0+15917+093c 1.0 MB/s | 49 kB 00:00 2022-09-10T04:13:35.760 INFO:teuthology.orchestra.run.smithi081.stdout:(5/19): cockpit-podman-49.1-1.module+el8.6.0+15 4.7 MB/s | 524 kB 00:00 2022-09-10T04:13:35.790 INFO:teuthology.orchestra.run.smithi110.stdout:(19/19): containernetworking-plugins-1.1.1-3.mo 50 MB/s | 18 MB 00:00 2022-09-10T04:13:35.791 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:13:35.792 INFO:teuthology.orchestra.run.smithi110.stdout:Total 39 MB/s | 40 MB 00:01 2022-09-10T04:13:35.802 INFO:teuthology.orchestra.run.smithi081.stdout:(6/19): crun-1.4.5-2.module+el8.6.0+15917+093ca 4.9 MB/s | 209 kB 00:00 2022-09-10T04:13:35.852 INFO:teuthology.orchestra.run.smithi081.stdout:(7/19): toolbox-0.0.99.3-0.6.module+el8.6.0+159 9.7 MB/s | 2.2 MB 00:00 2022-09-10T04:13:35.895 INFO:teuthology.orchestra.run.smithi081.stdout:(8/19): python3-podman-4.0.0-1.module+el8.6.0+1 1.6 MB/s | 149 kB 00:00 2022-09-10T04:13:35.936 INFO:teuthology.orchestra.run.smithi081.stdout:(9/19): container-selinux-2.188.0-1.module+el8. 1.4 MB/s | 59 kB 00:00 2022-09-10T04:13:36.070 INFO:teuthology.orchestra.run.smithi081.stdout:(10/19): runc-1.1.3-2.module+el8.6.0+15917+093c 22 MB/s | 3.0 MB 00:00 2022-09-10T04:13:36.093 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2022-09-10T04:13:36.112 INFO:teuthology.orchestra.run.smithi081.stdout:(11/19): conmon-2.1.2-2.module+el8.6.0+15917+09 1.3 MB/s | 55 kB 00:00 2022-09-10T04:13:36.137 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2022-09-10T04:13:36.137 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2022-09-10T04:13:36.229 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2022-09-10T04:13:36.230 INFO:teuthology.orchestra.run.smithi081.stdout:(12/19): buildah-1.26.2-1.module+el8.6.0+15917+ 16 MB/s | 8.0 MB 00:00 2022-09-10T04:13:36.254 INFO:teuthology.orchestra.run.smithi081.stdout:(13/19): fuse-overlayfs-1.9-1.module+el8.6.0+15 518 kB/s | 73 kB 00:00 2022-09-10T04:13:36.274 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2022-09-10T04:13:36.275 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2022-09-10T04:13:36.354 INFO:teuthology.orchestra.run.smithi081.stdout:(14/19): skopeo-1.8.0-2.module+el8.6.0+15917+09 14 MB/s | 6.9 MB 00:00 2022-09-10T04:13:36.380 INFO:teuthology.orchestra.run.smithi081.stdout:(15/19): slirp4netns-1.2.0-2.module+el8.6.0+159 360 kB/s | 54 kB 00:00 2022-09-10T04:13:36.405 INFO:teuthology.orchestra.run.smithi081.stdout:(16/19): libslirp-4.4.0-1.module+el8.6.0+15875+ 468 kB/s | 70 kB 00:00 2022-09-10T04:13:36.449 INFO:teuthology.orchestra.run.smithi081.stdout:(17/19): containers-common-1-35.module+el8.6.0+ 1.6 MB/s | 109 kB 00:00 2022-09-10T04:13:36.481 INFO:teuthology.orchestra.run.smithi081.stdout:(18/19): criu-3.15-3.module+el8.6.0+15875+dc9a2 6.7 MB/s | 518 kB 00:00 2022-09-10T04:13:36.487 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2022-09-10T04:13:36.491 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2022-09-10T04:13:36.620 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2022-09-10T04:13:36.623 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2022-09-10T04:13:36.798 INFO:teuthology.orchestra.run.smithi081.stdout:(19/19): containernetworking-plugins-1.1.1-3.mo 41 MB/s | 18 MB 00:00 2022-09-10T04:13:36.799 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:13:36.800 INFO:teuthology.orchestra.run.smithi081.stdout:Total 33 MB/s | 40 MB 00:01 2022-09-10T04:13:37.501 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2022-09-10T04:13:37.513 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917+0 1/1 2022-09-10T04:13:37.533 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:37.645 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2022-09-10T04:13:37.657 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917+0 1/1 2022-09-10T04:13:37.678 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:37.747 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:37.836 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2022-09-10T04:13:37.888 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2022-09-10T04:13:37.889 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2022-09-10T04:13:37.890 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:38.505 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2022-09-10T04:13:38.510 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2022-09-10T04:13:39.586 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2022-09-10T04:13:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917+0 1/1 2022-09-10T04:13:39.702 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:39.928 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:54.716 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:54.863 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:55.247 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2/28 2022-09-10T04:13:55.460 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2/28 2022-09-10T04:13:57.018 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 3/28 2022-09-10T04:13:57.170 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 3/28 2022-09-10T04:13:57.245 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : containernetworking-plugins-1:1.1.1-3.module+el8.6 4/28 2022-09-10T04:13:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : containernetworking-plugins-1:1.1.1-3.module+el8.6 4/28 2022-09-10T04:13:57.417 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 5/28 2022-09-10T04:13:57.535 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 5/28 2022-09-10T04:13:57.606 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 6/28 2022-09-10T04:13:57.753 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 6/28 2022-09-10T04:13:57.779 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-10T04:13:57.796 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-10T04:13:57.913 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-10T04:13:57.930 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-10T04:13:57.938 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : yajl-2.1.0-10.el8.x86_64 8/28 2022-09-10T04:13:58.071 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : yajl-2.1.0-10.el8.x86_64 8/28 2022-09-10T04:13:58.098 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 9/28 2022-09-10T04:13:58.098 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : containers-common-2:1-35.module+el8.6.0+15917+093c 10/28 2022-09-10T04:13:58.098 INFO:teuthology.orchestra.run.smithi104.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2022-09-10T04:13:58.099 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:13:58.232 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 9/28 2022-09-10T04:13:58.232 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : containers-common-2:1-35.module+el8.6.0+15917+093c 10/28 2022-09-10T04:13:58.232 INFO:teuthology.orchestra.run.smithi110.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2022-09-10T04:13:58.233 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:13:58.533 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 11/28 2022-09-10T04:13:58.702 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 11/28 2022-09-10T04:13:58.720 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/28 2022-09-10T04:13:58.872 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/28 2022-09-10T04:13:59.094 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 13/28 2022-09-10T04:13:59.263 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 13/28 2022-09-10T04:13:59.742 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 1/28 2022-09-10T04:13:59.956 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 14/28 2022-09-10T04:14:00.149 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 14/28 2022-09-10T04:14:00.393 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2/28 2022-09-10T04:14:00.752 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 15/28 2022-09-10T04:14:00.915 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 16/28 2022-09-10T04:14:00.965 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 15/28 2022-09-10T04:14:01.094 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 17/28 2022-09-10T04:14:01.144 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 16/28 2022-09-10T04:14:01.320 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 18/28 2022-09-10T04:14:01.332 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 17/28 2022-09-10T04:14:01.471 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 19/28 2022-09-10T04:14:01.553 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 18/28 2022-09-10T04:14:01.652 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 20/28 2022-09-10T04:14:01.697 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 19/28 2022-09-10T04:14:01.845 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 21/28 2022-09-10T04:14:01.923 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 20/28 2022-09-10T04:14:01.979 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 22/28 2022-09-10T04:14:01.979 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-10T04:14:02.088 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 21/28 2022-09-10T04:14:02.129 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-10T04:14:02.246 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 22/28 2022-09-10T04:14:02.247 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-10T04:14:02.335 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 24/28 2022-09-10T04:14:02.366 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 3/28 2022-09-10T04:14:02.454 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 25/28 2022-09-10T04:14:02.463 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-10T04:14:02.572 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 26/28 2022-09-10T04:14:02.629 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : containernetworking-plugins-1:1.1.1-3.module+el8.6 4/28 2022-09-10T04:14:02.640 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 24/28 2022-09-10T04:14:02.762 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 27/28 2022-09-10T04:14:02.788 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 25/28 2022-09-10T04:14:02.809 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 5/28 2022-09-10T04:14:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-10T04:14:02.940 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 26/28 2022-09-10T04:14:03.006 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 6/28 2022-09-10T04:14:03.105 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 27/28 2022-09-10T04:14:03.202 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-10T04:14:03.259 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 7/28 2022-09-10T04:14:03.296 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-10T04:14:03.422 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : yajl-2.1.0-10.el8.x86_64 8/28 2022-09-10T04:14:03.608 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 9/28 2022-09-10T04:14:03.608 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : containers-common-2:1-35.module+el8.6.0+15917+093c 10/28 2022-09-10T04:14:03.608 INFO:teuthology.orchestra.run.smithi081.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2022-09-10T04:14:03.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:14:04.082 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 11/28 2022-09-10T04:14:04.324 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 12/28 2022-09-10T04:14:04.729 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 13/28 2022-09-10T04:14:05.714 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 14/28 2022-09-10T04:14:06.607 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 15/28 2022-09-10T04:14:06.800 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 16/28 2022-09-10T04:14:06.995 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 17/28 2022-09-10T04:14:07.240 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 18/28 2022-09-10T04:14:07.377 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 19/28 2022-09-10T04:14:07.567 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 20/28 2022-09-10T04:14:07.786 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 21/28 2022-09-10T04:14:07.959 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 22/28 2022-09-10T04:14:07.960 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-10T04:14:08.170 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/28 2022-09-10T04:14:08.392 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 24/28 2022-09-10T04:14:08.544 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 25/28 2022-09-10T04:14:08.720 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 26/28 2022-09-10T04:14:08.943 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 27/28 2022-09-10T04:14:09.100 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-10T04:14:18.472 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 28/28 2022-09-10T04:14:18.541 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 28/28 2022-09-10T04:14:18.880 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-10T04:14:18.880 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/28 2022-09-10T04:14:18.880 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/28 2022-09-10T04:14:18.881 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 3/28 2022-09-10T04:14:18.881 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 4/28 2022-09-10T04:14:18.881 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 5/28 2022-09-10T04:14:18.881 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 6/28 2022-09-10T04:14:18.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 7/28 2022-09-10T04:14:18.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 8/28 2022-09-10T04:14:18.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 9/28 2022-09-10T04:14:18.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 10/28 2022-09-10T04:14:18.882 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : container-selinux-2:2.188.0-1.module+el8.6.0+15917 11/28 2022-09-10T04:14:18.883 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 12/28 2022-09-10T04:14:18.883 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 13/28 2022-09-10T04:14:18.883 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 14/28 2022-09-10T04:14:18.883 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 15/28 2022-09-10T04:14:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 16/28 2022-09-10T04:14:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 17/28 2022-09-10T04:14:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 18/28 2022-09-10T04:14:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 19/28 2022-09-10T04:14:18.884 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 20/28 2022-09-10T04:14:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 21/28 2022-09-10T04:14:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 22/28 2022-09-10T04:14:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module+el8.6 23/28 2022-09-10T04:14:18.885 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 24/28 2022-09-10T04:14:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : containers-common-2:1-35.module+el8.6.0+15917+093c 25/28 2022-09-10T04:14:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 26/28 2022-09-10T04:14:18.886 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 27/28 2022-09-10T04:14:18.934 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-10T04:14:18.935 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/28 2022-09-10T04:14:18.935 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/28 2022-09-10T04:14:18.935 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 3/28 2022-09-10T04:14:18.935 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 4/28 2022-09-10T04:14:18.935 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 5/28 2022-09-10T04:14:18.936 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 6/28 2022-09-10T04:14:18.936 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 7/28 2022-09-10T04:14:18.936 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 8/28 2022-09-10T04:14:18.936 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 9/28 2022-09-10T04:14:18.937 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 10/28 2022-09-10T04:14:18.937 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : container-selinux-2:2.188.0-1.module+el8.6.0+15917 11/28 2022-09-10T04:14:18.937 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 12/28 2022-09-10T04:14:18.937 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 13/28 2022-09-10T04:14:18.938 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 14/28 2022-09-10T04:14:18.938 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 15/28 2022-09-10T04:14:18.939 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 16/28 2022-09-10T04:14:18.940 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 17/28 2022-09-10T04:14:18.940 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 18/28 2022-09-10T04:14:18.940 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 19/28 2022-09-10T04:14:18.940 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 20/28 2022-09-10T04:14:18.941 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 21/28 2022-09-10T04:14:18.941 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 22/28 2022-09-10T04:14:18.941 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module+el8.6 23/28 2022-09-10T04:14:18.941 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 24/28 2022-09-10T04:14:18.941 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : containers-common-2:1-35.module+el8.6.0+15917+093c 25/28 2022-09-10T04:14:18.942 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 26/28 2022-09-10T04:14:18.942 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 27/28 2022-09-10T04:14:19.620 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 28/28 2022-09-10T04:14:19.621 INFO:teuthology.orchestra.run.smithi104.stdout:Installed products updated. 2022-09-10T04:14:19.715 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 28/28 2022-09-10T04:14:19.715 INFO:teuthology.orchestra.run.smithi110.stdout:Installed products updated. 2022-09-10T04:14:19.844 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:14:19.845 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2022-09-10T04:14:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.845 INFO:teuthology.orchestra.run.smithi104.stdout: container-selinux-2:2.188.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: containernetworking-plugins-1:1.1.1-3.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: containers-common-2:1-35.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-10T04:14:19.846 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-10T04:14:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.847 INFO:teuthology.orchestra.run.smithi104.stdout: slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.848 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2022-09-10T04:14:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.848 INFO:teuthology.orchestra.run.smithi104.stdout: cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-09-10T04:14:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-09-10T04:14:19.849 INFO:teuthology.orchestra.run.smithi104.stdout: skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: yajl-2.1.0-10.el8.x86_64 2022-09-10T04:14:19.850 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:14:19.851 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2022-09-10T04:14:19.945 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:14:19.946 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2022-09-10T04:14:19.946 INFO:teuthology.orchestra.run.smithi110.stdout: conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.946 INFO:teuthology.orchestra.run.smithi110.stdout: container-selinux-2:2.188.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.946 INFO:teuthology.orchestra.run.smithi110.stdout: containernetworking-plugins-1:1.1.1-3.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.947 INFO:teuthology.orchestra.run.smithi110.stdout: containers-common-2:1-35.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.947 INFO:teuthology.orchestra.run.smithi110.stdout: criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-10T04:14:19.947 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.947 INFO:teuthology.orchestra.run.smithi110.stdout: libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-10T04:14:19.947 INFO:teuthology.orchestra.run.smithi110.stdout: runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.948 INFO:teuthology.orchestra.run.smithi110.stdout: slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.948 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2022-09-10T04:14:19.948 INFO:teuthology.orchestra.run.smithi110.stdout: buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.948 INFO:teuthology.orchestra.run.smithi110.stdout: cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.949 INFO:teuthology.orchestra.run.smithi110.stdout: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.949 INFO:teuthology.orchestra.run.smithi110.stdout: python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.949 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-09-10T04:14:19.949 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-09-10T04:14:19.949 INFO:teuthology.orchestra.run.smithi110.stdout: skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.950 INFO:teuthology.orchestra.run.smithi110.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:19.950 INFO:teuthology.orchestra.run.smithi110.stdout: udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:19.950 INFO:teuthology.orchestra.run.smithi110.stdout: yajl-2.1.0-10.el8.x86_64 2022-09-10T04:14:19.951 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:14:19.951 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2022-09-10T04:14:20.097 DEBUG:teuthology.parallel:result is None 2022-09-10T04:14:20.242 DEBUG:teuthology.parallel:result is None 2022-09-10T04:14:31.381 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: container-selinux-2:2.188.0-1.module+el8.6.0+15917 28/28 2022-09-10T04:14:32.283 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 28/28 2022-09-10T04:14:32.284 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/28 2022-09-10T04:14:32.284 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2/28 2022-09-10T04:14:32.284 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 3/28 2022-09-10T04:14:32.285 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8 4/28 2022-09-10T04:14:32.285 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f 5/28 2022-09-10T04:14:32.286 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 6/28 2022-09-10T04:14:32.286 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x 7/28 2022-09-10T04:14:32.287 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 8/28 2022-09-10T04:14:32.287 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6 9/28 2022-09-10T04:14:32.288 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86 10/28 2022-09-10T04:14:32.288 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : container-selinux-2:2.188.0-1.module+el8.6.0+15917 11/28 2022-09-10T04:14:32.288 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 12/28 2022-09-10T04:14:32.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_6 13/28 2022-09-10T04:14:32.289 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 14/28 2022-09-10T04:14:32.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86 15/28 2022-09-10T04:14:32.290 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 16/28 2022-09-10T04:14:32.291 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8 17/28 2022-09-10T04:14:32.291 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 18/28 2022-09-10T04:14:32.291 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8. 19/28 2022-09-10T04:14:32.292 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 20/28 2022-09-10T04:14:32.292 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86 21/28 2022-09-10T04:14:32.293 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 22/28 2022-09-10T04:14:32.293 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : containernetworking-plugins-1:1.1.1-3.module+el8.6 23/28 2022-09-10T04:14:32.294 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 24/28 2022-09-10T04:14:32.294 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : containers-common-2:1-35.module+el8.6.0+15917+093c 25/28 2022-09-10T04:14:32.295 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 26/28 2022-09-10T04:14:32.295 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 27/28 2022-09-10T04:14:33.222 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 28/28 2022-09-10T04:14:33.223 INFO:teuthology.orchestra.run.smithi081.stdout:Installed products updated. 2022-09-10T04:14:33.419 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:14:33.420 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2022-09-10T04:14:33.421 INFO:teuthology.orchestra.run.smithi081.stdout: conmon-2:2.1.2-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.422 INFO:teuthology.orchestra.run.smithi081.stdout: container-selinux-2:2.188.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:33.422 INFO:teuthology.orchestra.run.smithi081.stdout: containernetworking-plugins-1:1.1.1-3.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.422 INFO:teuthology.orchestra.run.smithi081.stdout: containers-common-2:1-35.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.422 INFO:teuthology.orchestra.run.smithi081.stdout: criu-3.15-3.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-10T04:14:33.423 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-overlayfs-1.9-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.423 INFO:teuthology.orchestra.run.smithi081.stdout: libslirp-4.4.0-1.module+el8.6.0+15875+dc9a2b96.x86_64 2022-09-10T04:14:33.423 INFO:teuthology.orchestra.run.smithi081.stdout: runc-1:1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.423 INFO:teuthology.orchestra.run.smithi081.stdout: slirp4netns-1.2.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.424 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2022-09-10T04:14:33.424 INFO:teuthology.orchestra.run.smithi081.stdout: buildah-1:1.26.2-1.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.424 INFO:teuthology.orchestra.run.smithi081.stdout: cockpit-podman-49.1-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:33.424 INFO:teuthology.orchestra.run.smithi081.stdout: crun-1.4.5-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.425 INFO:teuthology.orchestra.run.smithi081.stdout: python3-podman-4.0.0-1.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:33.425 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2022-09-10T04:14:33.425 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyxdg-0.25-16.el8.noarch 2022-09-10T04:14:33.425 INFO:teuthology.orchestra.run.smithi081.stdout: skopeo-2:1.8.0-2.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.425 INFO:teuthology.orchestra.run.smithi081.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+15917+093ca6f8.x86_64 2022-09-10T04:14:33.426 INFO:teuthology.orchestra.run.smithi081.stdout: udica-0.2.6-4.module+el8.6.0+15917+093ca6f8.noarch 2022-09-10T04:14:33.426 INFO:teuthology.orchestra.run.smithi081.stdout: yajl-2.1.0-10.el8.x86_64 2022-09-10T04:14:33.426 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:14:33.427 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2022-09-10T04:14:34.042 DEBUG:teuthology.parallel:result is None 2022-09-10T04:14:34.043 INFO:teuthology.run_tasks:Running task install... 2022-09-10T04:14:34.054 DEBUG:teuthology.task.install:project ceph 2022-09-10T04:14:34.055 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '244418605927bb7842e73fc5d2a53889b760937f'}} 2022-09-10T04:14:34.056 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '244418605927bb7842e73fc5d2a53889b760937f'} 2022-09-10T04:14:34.056 INFO:teuthology.task.install:Using flavor: default 2022-09-10T04:14:34.065 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-09-10T04:14:34.065 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2022-09-10T04:14:34.066 DEBUG:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2022-09-10T04:14:34.212 INFO:teuthology.orchestra.run.smithi081.stdout:RedHatEnterprise 2022-09-10T04:14:34.217 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:14:34.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo lsb_release -is 2022-09-10T04:14:34.219 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '244418605927bb7842e73fc5d2a53889b760937f', 'tag': None, 'wait_for_package': False} 2022-09-10T04:14:34.219 DEBUG:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2022-09-10T04:14:34.271 INFO:teuthology.orchestra.run.smithi104.stdout:RedHatEnterprise 2022-09-10T04:14:34.273 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:14:34.273 DEBUG:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2022-09-10T04:14:34.274 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '244418605927bb7842e73fc5d2a53889b760937f', 'tag': None, 'wait_for_package': False} 2022-09-10T04:14:34.274 DEBUG:teuthology.orchestra.run.smithi104:> sudo lsb_release -is 2022-09-10T04:14:34.319 INFO:teuthology.orchestra.run.smithi104.stdout:RedHatEnterprise 2022-09-10T04:14:34.321 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:14:34.323 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:14:34.334 INFO:teuthology.orchestra.run.smithi110.stdout:RedHatEnterprise 2022-09-10T04:14:34.334 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:14:34.335 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '244418605927bb7842e73fc5d2a53889b760937f', 'tag': None, 'wait_for_package': False} 2022-09-10T04:14:34.335 DEBUG:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2022-09-10T04:14:34.371 INFO:teuthology.orchestra.run.smithi081.stdout:RedHatEnterprise 2022-09-10T04:14:34.372 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:14:34.372 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:14:34.380 INFO:teuthology.orchestra.run.smithi110.stdout:RedHatEnterprise 2022-09-10T04:14:34.382 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:14:34.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:14:34.492 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/ 2022-09-10T04:14:34.493 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14898.g24441860 2022-09-10T04:14:34.533 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/ 2022-09-10T04:14:34.533 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14898.g24441860 2022-09-10T04:14:34.549 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/ 2022-09-10T04:14:34.550 INFO:teuthology.task.install.rpm:Package version is 17.0.0-14898.g24441860 2022-09-10T04:14:34.629 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-10T04:14:34.630 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:14:34.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-10T04:14:34.649 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-10T04:14:34.649 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:14:34.650 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-10T04:14:34.660 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-09-10T04:14:34.661 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/244418605927bb7842e73fc5d2a53889b760937f/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-10T04:14:34.669 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20220909.143836/244418605927bb7842e73fc5d2a53889b760937f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2022-09-10T04:14:34.670 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:14:34.671 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2022-09-10T04:14:34.702 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-09-10T04:14:34.702 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/244418605927bb7842e73fc5d2a53889b760937f/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-10T04:14:34.733 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-10T04:14:34.741 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2022-09-10T04:14:34.742 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/244418605927bb7842e73fc5d2a53889b760937f/;g' /etc/yum.repos.d/ceph.repo ; fi 2022-09-10T04:14:34.772 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-10T04:14:34.815 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-10T04:14:34.855 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-10T04:14:34.870 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2022-09-10T04:14:34.883 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2022-09-10T04:14:34.884 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2022-09-10T04:14:34.923 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2022-09-10T04:14:34.925 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2022-09-10T04:14:34.992 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2022-09-10T04:14:35.078 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2022-09-10T04:14:35.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2022-09-10T04:14:35.262 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2022-09-10T04:14:35.298 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2022-09-10T04:14:35.468 INFO:teuthology.orchestra.run.smithi104.stdout:52 files removed 2022-09-10T04:14:35.508 INFO:teuthology.orchestra.run.smithi110.stdout:52 files removed 2022-09-10T04:14:35.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-09-10T04:14:35.568 INFO:teuthology.orchestra.run.smithi081.stdout:Updating Subscription Management repositories. 2022-09-10T04:14:35.596 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-09-10T04:14:35.848 INFO:teuthology.orchestra.run.smithi104.stdout:Updating Subscription Management repositories. 2022-09-10T04:14:35.865 INFO:teuthology.orchestra.run.smithi081.stdout:52 files removed 2022-09-10T04:14:35.927 INFO:teuthology.orchestra.run.smithi110.stdout:Updating Subscription Management repositories. 2022-09-10T04:14:35.930 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2022-09-10T04:14:36.414 INFO:teuthology.orchestra.run.smithi081.stdout:Updating Subscription Management repositories. 2022-09-10T04:14:36.426 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 295 kB/s | 81 kB 00:00 2022-09-10T04:14:36.504 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 297 kB/s | 81 kB 00:00 2022-09-10T04:14:36.735 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 34 kB/s | 9.4 kB 00:00 2022-09-10T04:14:36.787 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 37 kB/s | 9.4 kB 00:00 2022-09-10T04:14:37.002 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2022-09-10T04:14:37.055 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 6.1 kB/s | 1.5 kB 00:00 2022-09-10T04:14:37.150 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 285 kB/s | 81 kB 00:00 2022-09-10T04:14:37.245 INFO:teuthology.orchestra.run.smithi104.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2022-09-10T04:14:37.248 INFO:teuthology.orchestra.run.smithi110.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2022-09-10T04:14:37.425 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 39 kB/s | 9.4 kB 00:00 2022-09-10T04:14:37.710 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 5.8 kB/s | 1.5 kB 00:00 2022-09-10T04:14:37.902 INFO:teuthology.orchestra.run.smithi081.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2022-09-10T04:14:38.206 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2022-09-10T04:14:38.236 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:00 2022-09-10T04:14:39.433 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 13 MB 00:01 2022-09-10T04:14:41.020 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 422 kB/s | 24 kB 00:00 2022-09-10T04:14:41.023 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 309 kB/s | 24 kB 00:00 2022-09-10T04:14:41.797 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 MB/s | 46 MB 00:00 2022-09-10T04:14:42.050 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 53 MB/s | 51 MB 00:00 2022-09-10T04:14:42.795 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 422 kB/s | 24 kB 00:00 2022-09-10T04:14:43.747 INFO:teuthology.orchestra.run.smithi081.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 52 MB/s | 46 MB 00:00 2022-09-10T04:14:49.497 INFO:teuthology.orchestra.run.smithi104.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 71 MB/s | 51 MB 00:00 2022-09-10T04:14:49.701 INFO:teuthology.orchestra.run.smithi110.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 67 MB/s | 46 MB 00:00 2022-09-10T04:14:52.423 INFO:teuthology.orchestra.run.smithi081.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 51 MB 00:00 2022-09-10T04:15:00.375 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-10T04:15:00.376 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-10T04:15:00.390 INFO:teuthology.orchestra.run.smithi104.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-09-10T04:15:00.391 INFO:teuthology.orchestra.run.smithi104.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-09-10T04:15:00.393 INFO:teuthology.orchestra.run.smithi104.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-09-10T04:15:00.394 INFO:teuthology.orchestra.run.smithi104.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-09-10T04:15:00.396 INFO:teuthology.orchestra.run.smithi104.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-09-10T04:15:00.400 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-09-10T04:15:00.401 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-09-10T04:15:00.425 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-10T04:15:00.426 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-10T04:15:00.441 INFO:teuthology.orchestra.run.smithi110.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-09-10T04:15:00.442 INFO:teuthology.orchestra.run.smithi110.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-09-10T04:15:00.443 INFO:teuthology.orchestra.run.smithi110.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-09-10T04:15:00.444 INFO:teuthology.orchestra.run.smithi110.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-09-10T04:15:00.447 INFO:teuthology.orchestra.run.smithi110.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-09-10T04:15:00.450 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-09-10T04:15:00.451 INFO:teuthology.orchestra.run.smithi110.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-09-10T04:15:00.528 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2022-09-10T04:15:00.541 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2022-09-10T04:15:00.542 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2022-09-10T04:15:00.542 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2022-09-10T04:15:00.542 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2022-09-10T04:15:00.543 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms 137 k 2022-09-10T04:15:00.543 INFO:teuthology.orchestra.run.smithi104.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-09-10T04:15:00.543 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:17.0.0-14898.g24441860.el8 ceph 6.4 k 2022-09-10T04:15:00.544 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:17.0.0-14898.g24441860.el8 ceph 5.2 M 2022-09-10T04:15:00.544 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:17.0.0-14898.g24441860.el8 ceph 844 k 2022-09-10T04:15:00.544 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14898.g24441860.el8 ceph 165 k 2022-09-10T04:15:00.545 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:17.0.0-14898.g24441860.el8 ceph 1.5 M 2022-09-10T04:15:00.545 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 115 k 2022-09-10T04:15:00.545 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 1.6 M 2022-09-10T04:15:00.546 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 7.4 M 2022-09-10T04:15:00.546 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 47 k 2022-09-10T04:15:00.547 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:17.0.0-14898.g24441860.el8 ceph 18 M 2022-09-10T04:15:00.547 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:17.0.0-14898.g24441860.el8 ceph 54 M 2022-09-10T04:15:00.547 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 263 k 2022-09-10T04:15:00.548 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 81 k 2022-09-10T04:15:00.548 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror x86_64 2:17.0.0-14898.g24441860.el8 ceph 258 k 2022-09-10T04:15:00.549 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 15 k 2022-09-10T04:15:00.549 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-09-10T04:15:00.549 INFO:teuthology.orchestra.run.smithi104.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-09-10T04:15:00.550 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-09-10T04:15:00.550 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:17.0.0-14898.g24441860.el8 ceph 30 k 2022-09-10T04:15:00.551 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 766 k 2022-09-10T04:15:00.551 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:17.0.0-14898.g24441860.el8 ceph 139 k 2022-09-10T04:15:00.552 INFO:teuthology.orchestra.run.smithi104.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-09-10T04:15:00.552 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-09-10T04:15:00.552 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-09-10T04:15:00.553 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-09-10T04:15:00.553 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-09-10T04:15:00.553 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-09-10T04:15:00.554 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:17.0.0-14898.g24441860.el8 ceph 190 k 2022-09-10T04:15:00.554 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:17.0.0-14898.g24441860.el8 ceph 371 k 2022-09-10T04:15:00.555 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:17.0.0-14898.g24441860.el8 ceph 358 k 2022-09-10T04:15:00.555 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:17.0.0-14898.g24441860.el8 ceph 107 k 2022-09-10T04:15:00.555 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:17.0.0-14898.g24441860.el8 ceph 94 k 2022-09-10T04:15:00.555 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.5 M 2022-09-10T04:15:00.556 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:17.0.0-14898.g24441860.el8 ceph 175 k 2022-09-10T04:15:00.556 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-09-10T04:15:00.556 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2022-09-10T04:15:00.556 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.6 M 2022-09-10T04:15:00.557 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.6 M 2022-09-10T04:15:00.557 INFO:teuthology.orchestra.run.smithi104.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-09-10T04:15:00.557 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-09-10T04:15:00.557 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2022-09-10T04:15:00.557 INFO:teuthology.orchestra.run.smithi104.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-09-10T04:15:00.558 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:17.0.0-14898.g24441860.el8 ceph 24 M 2022-09-10T04:15:00.558 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 23 k 2022-09-10T04:15:00.558 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:17.0.0-14898.g24441860.el8 ceph 2.3 M 2022-09-10T04:15:00.558 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 233 k 2022-09-10T04:15:00.559 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.9 M 2022-09-10T04:15:00.559 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:17.0.0-14898.g24441860.el8 ceph 17 M 2022-09-10T04:15:00.559 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 14 k 2022-09-10T04:15:00.559 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:17.0.0-14898.g24441860.el8 ceph 24 k 2022-09-10T04:15:00.559 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-09-10T04:15:00.560 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-10T04:15:00.560 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-09-10T04:15:00.560 INFO:teuthology.orchestra.run.smithi104.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-09-10T04:15:00.560 INFO:teuthology.orchestra.run.smithi104.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-09-10T04:15:00.561 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:17.0.0-14898.g24441860.el8 ceph 137 k 2022-09-10T04:15:00.561 INFO:teuthology.orchestra.run.smithi104.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-09-10T04:15:00.561 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-09-10T04:15:00.561 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-10T04:15:00.561 INFO:teuthology.orchestra.run.smithi104.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-09-10T04:15:00.562 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-09-10T04:15:00.562 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-09-10T04:15:00.562 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:17.0.0-14898.g24441860.el8 ceph 476 k 2022-09-10T04:15:00.562 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-09-10T04:15:00.563 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 5.8 M 2022-09-10T04:15:00.563 INFO:teuthology.orchestra.run.smithi104.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-09-10T04:15:00.563 INFO:teuthology.orchestra.run.smithi104.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-09-10T04:15:00.563 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-10T04:15:00.563 INFO:teuthology.orchestra.run.smithi104.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-09-10T04:15:00.564 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-09-10T04:15:00.564 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-09-10T04:15:00.564 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-09-10T04:15:00.564 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-09-10T04:15:00.565 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-09-10T04:15:00.565 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-09-10T04:15:00.565 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-09-10T04:15:00.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-10T04:15:00.565 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-09-10T04:15:00.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-10T04:15:00.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-10T04:15:00.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-10T04:15:00.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:17.0.0-14898.g24441860.el8 ceph 45 k 2022-09-10T04:15:00.566 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:17.0.0-14898.g24441860.el8 ceph 109 k 2022-09-10T04:15:00.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-10T04:15:00.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-09-10T04:15:00.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-10T04:15:00.567 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-10T04:15:00.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-09-10T04:15:00.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-10T04:15:00.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-10T04:15:00.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-10T04:15:00.568 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-09-10T04:15:00.569 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-09-10T04:15:00.569 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-09-10T04:15:00.569 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-10T04:15:00.569 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-10T04:15:00.570 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-10T04:15:00.570 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-09-10T04:15:00.570 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-09-10T04:15:00.571 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-10T04:15:00.571 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-10T04:15:00.571 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-09-10T04:15:00.571 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-09-10T04:15:00.571 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-09-10T04:15:00.572 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-10T04:15:00.572 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-10T04:15:00.572 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-09-10T04:15:00.572 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-09-10T04:15:00.572 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-09-10T04:15:00.573 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-09-10T04:15:00.573 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-09-10T04:15:00.573 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-10T04:15:00.573 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-10T04:15:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-10T04:15:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-10T04:15:00.574 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-10T04:15:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-09-10T04:15:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-10T04:15:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-10T04:15:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-10T04:15:00.575 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-10T04:15:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-10T04:15:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-10T04:15:00.576 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-10T04:15:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-10T04:15:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-09-10T04:15:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-10T04:15:00.577 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-09-10T04:15:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-10T04:15:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-09-10T04:15:00.578 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-09-10T04:15:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-09-10T04:15:00.579 INFO:teuthology.orchestra.run.smithi104.stdout:Enabling module streams: 2022-09-10T04:15:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql 10 2022-09-10T04:15:00.579 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:15:00.579 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2022-09-10T04:15:00.580 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================================================== 2022-09-10T04:15:00.580 INFO:teuthology.orchestra.run.smithi104.stdout:Install 128 Packages 2022-09-10T04:15:00.580 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 4 Packages 2022-09-10T04:15:00.580 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:15:00.582 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2022-09-10T04:15:00.582 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 216 M 2022-09-10T04:15:00.583 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2022-09-10T04:15:00.593 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================================================== 2022-09-10T04:15:00.593 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2022-09-10T04:15:00.594 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================================================== 2022-09-10T04:15:00.594 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2022-09-10T04:15:00.594 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms 137 k 2022-09-10T04:15:00.595 INFO:teuthology.orchestra.run.smithi110.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-09-10T04:15:00.595 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:17.0.0-14898.g24441860.el8 ceph 6.4 k 2022-09-10T04:15:00.595 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:17.0.0-14898.g24441860.el8 ceph 5.2 M 2022-09-10T04:15:00.595 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:17.0.0-14898.g24441860.el8 ceph 844 k 2022-09-10T04:15:00.596 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14898.g24441860.el8 ceph 165 k 2022-09-10T04:15:00.596 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:17.0.0-14898.g24441860.el8 ceph 1.5 M 2022-09-10T04:15:00.596 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 115 k 2022-09-10T04:15:00.596 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 1.6 M 2022-09-10T04:15:00.597 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 7.4 M 2022-09-10T04:15:00.597 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 47 k 2022-09-10T04:15:00.597 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:17.0.0-14898.g24441860.el8 ceph 18 M 2022-09-10T04:15:00.597 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:17.0.0-14898.g24441860.el8 ceph 54 M 2022-09-10T04:15:00.598 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 263 k 2022-09-10T04:15:00.598 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 81 k 2022-09-10T04:15:00.598 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror x86_64 2:17.0.0-14898.g24441860.el8 ceph 258 k 2022-09-10T04:15:00.599 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 15 k 2022-09-10T04:15:00.599 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-09-10T04:15:00.599 INFO:teuthology.orchestra.run.smithi110.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-09-10T04:15:00.600 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-09-10T04:15:00.600 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:17.0.0-14898.g24441860.el8 ceph 30 k 2022-09-10T04:15:00.600 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 766 k 2022-09-10T04:15:00.600 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:17.0.0-14898.g24441860.el8 ceph 139 k 2022-09-10T04:15:00.600 INFO:teuthology.orchestra.run.smithi110.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-09-10T04:15:00.601 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-09-10T04:15:00.601 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-09-10T04:15:00.601 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-09-10T04:15:00.601 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-09-10T04:15:00.602 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-09-10T04:15:00.602 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:17.0.0-14898.g24441860.el8 ceph 190 k 2022-09-10T04:15:00.602 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:17.0.0-14898.g24441860.el8 ceph 371 k 2022-09-10T04:15:00.602 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:17.0.0-14898.g24441860.el8 ceph 358 k 2022-09-10T04:15:00.603 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:17.0.0-14898.g24441860.el8 ceph 107 k 2022-09-10T04:15:00.603 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:17.0.0-14898.g24441860.el8 ceph 94 k 2022-09-10T04:15:00.603 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.5 M 2022-09-10T04:15:00.603 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:17.0.0-14898.g24441860.el8 ceph 175 k 2022-09-10T04:15:00.603 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-09-10T04:15:00.604 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2022-09-10T04:15:00.604 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.6 M 2022-09-10T04:15:00.604 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.6 M 2022-09-10T04:15:00.604 INFO:teuthology.orchestra.run.smithi110.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-09-10T04:15:00.604 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-09-10T04:15:00.605 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2022-09-10T04:15:00.605 INFO:teuthology.orchestra.run.smithi110.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-09-10T04:15:00.605 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:17.0.0-14898.g24441860.el8 ceph 24 M 2022-09-10T04:15:00.605 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 23 k 2022-09-10T04:15:00.605 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:17.0.0-14898.g24441860.el8 ceph 2.3 M 2022-09-10T04:15:00.606 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 233 k 2022-09-10T04:15:00.606 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.9 M 2022-09-10T04:15:00.606 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:17.0.0-14898.g24441860.el8 ceph 17 M 2022-09-10T04:15:00.606 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 14 k 2022-09-10T04:15:00.607 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:17.0.0-14898.g24441860.el8 ceph 24 k 2022-09-10T04:15:00.607 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-09-10T04:15:00.607 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-10T04:15:00.607 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-09-10T04:15:00.607 INFO:teuthology.orchestra.run.smithi110.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-09-10T04:15:00.608 INFO:teuthology.orchestra.run.smithi110.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-09-10T04:15:00.608 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:17.0.0-14898.g24441860.el8 ceph 137 k 2022-09-10T04:15:00.608 INFO:teuthology.orchestra.run.smithi110.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-09-10T04:15:00.608 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-09-10T04:15:00.609 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-10T04:15:00.609 INFO:teuthology.orchestra.run.smithi110.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-09-10T04:15:00.609 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-09-10T04:15:00.609 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-09-10T04:15:00.609 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:17.0.0-14898.g24441860.el8 ceph 476 k 2022-09-10T04:15:00.610 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-09-10T04:15:00.610 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 5.8 M 2022-09-10T04:15:00.610 INFO:teuthology.orchestra.run.smithi110.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-09-10T04:15:00.610 INFO:teuthology.orchestra.run.smithi110.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-09-10T04:15:00.611 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-10T04:15:00.611 INFO:teuthology.orchestra.run.smithi110.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-09-10T04:15:00.611 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-09-10T04:15:00.611 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-09-10T04:15:00.611 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-09-10T04:15:00.612 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-09-10T04:15:00.612 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-09-10T04:15:00.612 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-09-10T04:15:00.612 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-09-10T04:15:00.612 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-10T04:15:00.613 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-09-10T04:15:00.613 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-10T04:15:00.613 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-10T04:15:00.613 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-10T04:15:00.614 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:17.0.0-14898.g24441860.el8 ceph 45 k 2022-09-10T04:15:00.614 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:17.0.0-14898.g24441860.el8 ceph 109 k 2022-09-10T04:15:00.614 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-10T04:15:00.614 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-09-10T04:15:00.615 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-10T04:15:00.615 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-10T04:15:00.615 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-09-10T04:15:00.615 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-10T04:15:00.615 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-10T04:15:00.616 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-10T04:15:00.616 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-09-10T04:15:00.616 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-09-10T04:15:00.616 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-09-10T04:15:00.616 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-10T04:15:00.617 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-10T04:15:00.617 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-10T04:15:00.617 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-09-10T04:15:00.617 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-09-10T04:15:00.617 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-10T04:15:00.618 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-10T04:15:00.618 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-09-10T04:15:00.618 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-09-10T04:15:00.618 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-09-10T04:15:00.619 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-10T04:15:00.619 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-10T04:15:00.619 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-09-10T04:15:00.619 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-09-10T04:15:00.619 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-09-10T04:15:00.620 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-09-10T04:15:00.620 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-09-10T04:15:00.620 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-10T04:15:00.620 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-10T04:15:00.621 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-10T04:15:00.621 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-10T04:15:00.621 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-10T04:15:00.621 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-09-10T04:15:00.621 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-10T04:15:00.622 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-10T04:15:00.622 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-10T04:15:00.622 INFO:teuthology.orchestra.run.smithi110.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-10T04:15:00.622 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-10T04:15:00.622 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-10T04:15:00.623 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-10T04:15:00.623 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-10T04:15:00.623 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-09-10T04:15:00.623 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-10T04:15:00.623 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-09-10T04:15:00.624 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-10T04:15:00.624 INFO:teuthology.orchestra.run.smithi110.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-09-10T04:15:00.624 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-09-10T04:15:00.624 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-09-10T04:15:00.625 INFO:teuthology.orchestra.run.smithi110.stdout:Enabling module streams: 2022-09-10T04:15:00.625 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql 10 2022-09-10T04:15:00.625 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:15:00.625 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2022-09-10T04:15:00.625 INFO:teuthology.orchestra.run.smithi110.stdout:=========================================================================================================================== 2022-09-10T04:15:00.626 INFO:teuthology.orchestra.run.smithi110.stdout:Install 128 Packages 2022-09-10T04:15:00.626 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 4 Packages 2022-09-10T04:15:00.626 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:15:00.627 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 216 M 2022-09-10T04:15:00.627 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2022-09-10T04:15:00.728 INFO:teuthology.orchestra.run.smithi104.stdout:(1/132): ceph-17.0.0-14898.g24441860.el8.x86_64 37 kB/s | 6.4 kB 00:00 2022-09-10T04:15:00.751 INFO:teuthology.orchestra.run.smithi110.stdout:(1/132): ceph-17.0.0-14898.g24441860.el8.x86_64 45 kB/s | 6.4 kB 00:00 2022-09-10T04:15:01.304 INFO:teuthology.orchestra.run.smithi104.stdout:(2/132): ceph-fuse-17.0.0-14898.g24441860.el8.x 1.4 MB/s | 844 kB 00:00 2022-09-10T04:15:01.360 INFO:teuthology.orchestra.run.smithi110.stdout:(2/132): ceph-fuse-17.0.0-14898.g24441860.el8.x 1.4 MB/s | 844 kB 00:00 2022-09-10T04:15:01.463 INFO:teuthology.orchestra.run.smithi104.stdout:(3/132): ceph-immutable-object-cache-17.0.0-148 1.0 MB/s | 165 kB 00:00 2022-09-10T04:15:01.510 INFO:teuthology.orchestra.run.smithi110.stdout:(3/132): ceph-immutable-object-cache-17.0.0-148 1.1 MB/s | 165 kB 00:00 2022-09-10T04:15:01.530 INFO:teuthology.orchestra.run.smithi104.stdout:(4/132): ceph-base-17.0.0-14898.g24441860.el8.x 5.3 MB/s | 5.2 MB 00:00 2022-09-10T04:15:01.627 INFO:teuthology.orchestra.run.smithi110.stdout:(4/132): ceph-base-17.0.0-14898.g24441860.el8.x 5.1 MB/s | 5.2 MB 00:01 2022-09-10T04:15:01.952 INFO:teuthology.orchestra.run.smithi110.stdout:(5/132): ceph-mgr-17.0.0-14898.g24441860.el8.x8 4.7 MB/s | 1.5 MB 00:00 2022-09-10T04:15:02.013 INFO:teuthology.orchestra.run.smithi104.stdout:(5/132): ceph-mgr-17.0.0-14898.g24441860.el8.x8 3.2 MB/s | 1.5 MB 00:00 2022-09-10T04:15:02.527 INFO:teuthology.orchestra.run.smithi110.stdout:(6/132): ceph-mds-17.0.0-14898.g24441860.el8.x8 2.2 MB/s | 2.3 MB 00:01 2022-09-10T04:15:02.756 INFO:teuthology.orchestra.run.smithi104.stdout:(6/132): ceph-mon-17.0.0-14898.g24441860.el8.x8 5.2 MB/s | 3.9 MB 00:00 2022-09-10T04:15:03.123 INFO:teuthology.orchestra.run.smithi104.stdout:(7/132): ceph-mds-17.0.0-14898.g24441860.el8.x8 1.6 MB/s | 2.3 MB 00:01 2022-09-10T04:15:03.661 INFO:teuthology.orchestra.run.smithi110.stdout:(7/132): ceph-common-17.0.0-14898.g24441860.el8 7.9 MB/s | 24 MB 00:03 2022-09-10T04:15:03.711 INFO:teuthology.orchestra.run.smithi110.stdout:(8/132): ceph-mon-17.0.0-14898.g24441860.el8.x8 2.2 MB/s | 3.9 MB 00:01 2022-09-10T04:15:03.792 INFO:teuthology.orchestra.run.smithi110.stdout:(9/132): ceph-selinux-17.0.0-14898.g24441860.el 323 kB/s | 24 kB 00:00 2022-09-10T04:15:03.793 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-10T04:15:03.794 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2022-09-10T04:15:03.794 INFO:teuthology.orchestra.run.smithi081.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2022-09-10T04:15:03.794 INFO:teuthology.orchestra.run.smithi081.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2022-09-10T04:15:03.794 INFO:teuthology.orchestra.run.smithi081.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2022-09-10T04:15:03.795 INFO:teuthology.orchestra.run.smithi081.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2022-09-10T04:15:03.797 INFO:teuthology.orchestra.run.smithi081.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2022-09-10T04:15:03.800 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2022-09-10T04:15:03.801 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2022-09-10T04:15:03.942 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2022-09-10T04:15:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:=========================================================================================================================== 2022-09-10T04:15:03.956 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2022-09-10T04:15:03.956 INFO:teuthology.orchestra.run.smithi081.stdout:=========================================================================================================================== 2022-09-10T04:15:03.956 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2022-09-10T04:15:03.956 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-7.el8_6 rhel-8-for-x86_64-baseos-rpms 137 k 2022-09-10T04:15:03.957 INFO:teuthology.orchestra.run.smithi081.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2022-09-10T04:15:03.957 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:17.0.0-14898.g24441860.el8 ceph 6.4 k 2022-09-10T04:15:03.957 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:17.0.0-14898.g24441860.el8 ceph 5.2 M 2022-09-10T04:15:03.957 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:17.0.0-14898.g24441860.el8 ceph 844 k 2022-09-10T04:15:03.957 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-14898.g24441860.el8 ceph 165 k 2022-09-10T04:15:03.958 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:17.0.0-14898.g24441860.el8 ceph 1.5 M 2022-09-10T04:15:03.958 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 115 k 2022-09-10T04:15:03.958 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 1.6 M 2022-09-10T04:15:03.959 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 7.4 M 2022-09-10T04:15:03.959 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 47 k 2022-09-10T04:15:03.959 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:17.0.0-14898.g24441860.el8 ceph 18 M 2022-09-10T04:15:03.959 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:17.0.0-14898.g24441860.el8 ceph 54 M 2022-09-10T04:15:03.960 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 263 k 2022-09-10T04:15:03.960 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 81 k 2022-09-10T04:15:03.960 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror x86_64 2:17.0.0-14898.g24441860.el8 ceph 258 k 2022-09-10T04:15:03.960 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 15 k 2022-09-10T04:15:03.960 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2022-09-10T04:15:03.961 INFO:teuthology.orchestra.run.smithi081.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2022-09-10T04:15:03.961 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2022-09-10T04:15:03.961 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:17.0.0-14898.g24441860.el8 ceph 30 k 2022-09-10T04:15:03.962 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 766 k 2022-09-10T04:15:03.962 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:17.0.0-14898.g24441860.el8 ceph 139 k 2022-09-10T04:15:03.962 INFO:teuthology.orchestra.run.smithi081.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2022-09-10T04:15:03.963 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2022-09-10T04:15:03.963 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2022-09-10T04:15:03.963 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2022-09-10T04:15:03.963 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2022-09-10T04:15:03.964 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2022-09-10T04:15:03.964 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:17.0.0-14898.g24441860.el8 ceph 190 k 2022-09-10T04:15:03.964 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:17.0.0-14898.g24441860.el8 ceph 371 k 2022-09-10T04:15:03.965 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:17.0.0-14898.g24441860.el8 ceph 358 k 2022-09-10T04:15:03.965 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:17.0.0-14898.g24441860.el8 ceph 107 k 2022-09-10T04:15:03.965 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:17.0.0-14898.g24441860.el8 ceph 94 k 2022-09-10T04:15:03.965 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.5 M 2022-09-10T04:15:03.965 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:17.0.0-14898.g24441860.el8 ceph 175 k 2022-09-10T04:15:03.966 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2022-09-10T04:15:03.966 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2022-09-10T04:15:03.966 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.6 M 2022-09-10T04:15:03.966 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.6 M 2022-09-10T04:15:03.967 INFO:teuthology.orchestra.run.smithi081.stdout: openssl x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 709 k 2022-09-10T04:15:03.967 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 1.5 M 2022-09-10T04:15:03.967 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2022-09-10T04:15:03.967 INFO:teuthology.orchestra.run.smithi081.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2022-09-10T04:15:03.968 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:17.0.0-14898.g24441860.el8 ceph 24 M 2022-09-10T04:15:03.968 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 23 k 2022-09-10T04:15:03.968 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:17.0.0-14898.g24441860.el8 ceph 2.3 M 2022-09-10T04:15:03.969 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 233 k 2022-09-10T04:15:03.969 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:17.0.0-14898.g24441860.el8 ceph 3.9 M 2022-09-10T04:15:03.969 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:17.0.0-14898.g24441860.el8 ceph 17 M 2022-09-10T04:15:03.969 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:17.0.0-14898.g24441860.el8 ceph-noarch 14 k 2022-09-10T04:15:03.969 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:17.0.0-14898.g24441860.el8 ceph 24 k 2022-09-10T04:15:03.970 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2022-09-10T04:15:03.970 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2022-09-10T04:15:03.970 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2022-09-10T04:15:03.971 INFO:teuthology.orchestra.run.smithi081.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2022-09-10T04:15:03.971 INFO:teuthology.orchestra.run.smithi081.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2022-09-10T04:15:03.971 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:17.0.0-14898.g24441860.el8 ceph 137 k 2022-09-10T04:15:03.971 INFO:teuthology.orchestra.run.smithi081.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2022-09-10T04:15:03.971 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2022-09-10T04:15:03.972 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2022-09-10T04:15:03.972 INFO:teuthology.orchestra.run.smithi081.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2022-09-10T04:15:03.972 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2022-09-10T04:15:03.972 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2022-09-10T04:15:03.973 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:17.0.0-14898.g24441860.el8 ceph 476 k 2022-09-10T04:15:03.973 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2022-09-10T04:15:03.973 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:17.0.0-14898.g24441860.el8 ceph 5.8 M 2022-09-10T04:15:03.973 INFO:teuthology.orchestra.run.smithi081.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2022-09-10T04:15:03.973 INFO:teuthology.orchestra.run.smithi081.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2022-09-10T04:15:03.974 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2022-09-10T04:15:03.974 INFO:teuthology.orchestra.run.smithi081.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2022-09-10T04:15:03.974 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2022-09-10T04:15:03.974 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2022-09-10T04:15:03.975 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2022-09-10T04:15:03.975 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2022-09-10T04:15:03.975 INFO:teuthology.orchestra.run.smithi081.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2022-09-10T04:15:03.975 INFO:teuthology.orchestra.run.smithi081.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2022-09-10T04:15:03.975 INFO:teuthology.orchestra.run.smithi081.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2022-09-10T04:15:03.976 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2022-09-10T04:15:03.976 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2022-09-10T04:15:03.976 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2022-09-10T04:15:03.976 INFO:teuthology.orchestra.run.smithi081.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2022-09-10T04:15:03.977 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2022-09-10T04:15:03.977 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:17.0.0-14898.g24441860.el8 ceph 45 k 2022-09-10T04:15:03.977 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:17.0.0-14898.g24441860.el8 ceph 109 k 2022-09-10T04:15:03.977 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2022-09-10T04:15:03.977 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2022-09-10T04:15:03.978 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2022-09-10T04:15:03.978 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2022-09-10T04:15:03.978 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2022-09-10T04:15:03.978 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2022-09-10T04:15:03.979 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2022-09-10T04:15:03.979 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2022-09-10T04:15:03.979 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2022-09-10T04:15:03.979 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2022-09-10T04:15:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2022-09-10T04:15:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-10T04:15:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2022-09-10T04:15:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2022-09-10T04:15:03.980 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2022-09-10T04:15:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2022-09-10T04:15:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2022-09-10T04:15:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2022-09-10T04:15:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2022-09-10T04:15:03.981 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2022-09-10T04:15:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2022-09-10T04:15:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2022-09-10T04:15:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2022-09-10T04:15:03.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2022-09-10T04:15:03.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2022-09-10T04:15:03.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2022-09-10T04:15:03.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2022-09-10T04:15:03.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2022-09-10T04:15:03.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2022-09-10T04:15:03.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2022-09-10T04:15:03.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2022-09-10T04:15:03.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2022-09-10T04:15:03.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2022-09-10T04:15:03.986 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2022-09-10T04:15:03.986 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2022-09-10T04:15:03.986 INFO:teuthology.orchestra.run.smithi081.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2022-09-10T04:15:03.987 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2022-09-10T04:15:03.987 INFO:teuthology.orchestra.run.smithi081.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2022-09-10T04:15:03.987 INFO:teuthology.orchestra.run.smithi081.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2022-09-10T04:15:03.987 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2022-09-10T04:15:03.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2022-09-10T04:15:03.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2022-09-10T04:15:03.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2022-09-10T04:15:03.989 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2022-09-10T04:15:03.989 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2022-09-10T04:15:03.989 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2022-09-10T04:15:03.990 INFO:teuthology.orchestra.run.smithi081.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2022-09-10T04:15:03.990 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2022-09-10T04:15:03.990 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2022-09-10T04:15:03.991 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling module streams: 2022-09-10T04:15:03.991 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql 10 2022-09-10T04:15:03.991 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:15:03.992 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2022-09-10T04:15:03.992 INFO:teuthology.orchestra.run.smithi081.stdout:=========================================================================================================================== 2022-09-10T04:15:03.992 INFO:teuthology.orchestra.run.smithi081.stdout:Install 128 Packages 2022-09-10T04:15:03.993 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 4 Packages 2022-09-10T04:15:03.993 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:15:03.994 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 216 M 2022-09-10T04:15:03.994 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2022-09-10T04:15:04.241 INFO:teuthology.orchestra.run.smithi081.stdout:(1/132): ceph-17.0.0-14898.g24441860.el8.x86_64 25 kB/s | 6.4 kB 00:00 2022-09-10T04:15:04.256 INFO:teuthology.orchestra.run.smithi104.stdout:(8/132): ceph-common-17.0.0-14898.g24441860.el8 6.5 MB/s | 24 MB 00:03 2022-09-10T04:15:04.339 INFO:teuthology.orchestra.run.smithi104.stdout:(9/132): ceph-selinux-17.0.0-14898.g24441860.el 291 kB/s | 24 kB 00:00 2022-09-10T04:15:05.097 INFO:teuthology.orchestra.run.smithi104.stdout:(10/132): ceph-osd-17.0.0-14898.g24441860.el8.x 7.2 MB/s | 17 MB 00:02 2022-09-10T04:15:05.132 INFO:teuthology.orchestra.run.smithi081.stdout:(2/132): ceph-fuse-17.0.0-14898.g24441860.el8.x 946 kB/s | 844 kB 00:00 2022-09-10T04:15:05.189 INFO:teuthology.orchestra.run.smithi104.stdout:(11/132): cephfs-mirror-17.0.0-14898.g24441860. 2.8 MB/s | 258 kB 00:00 2022-09-10T04:15:05.281 INFO:teuthology.orchestra.run.smithi104.stdout:(12/132): libcephfs-devel-17.0.0-14898.g2444186 332 kB/s | 30 kB 00:00 2022-09-10T04:15:05.282 INFO:teuthology.orchestra.run.smithi081.stdout:(3/132): ceph-immutable-object-cache-17.0.0-148 1.1 MB/s | 165 kB 00:00 2022-09-10T04:15:05.448 INFO:teuthology.orchestra.run.smithi104.stdout:(13/132): libcephfs2-17.0.0-14898.g24441860.el8 4.5 MB/s | 766 kB 00:00 2022-09-10T04:15:05.548 INFO:teuthology.orchestra.run.smithi104.stdout:(14/132): libcephsqlite-17.0.0-14898.g24441860. 1.3 MB/s | 137 kB 00:00 2022-09-10T04:15:05.657 INFO:teuthology.orchestra.run.smithi104.stdout:(15/132): librados-devel-17.0.0-14898.g24441860 1.3 MB/s | 139 kB 00:00 2022-09-10T04:15:05.774 INFO:teuthology.orchestra.run.smithi104.stdout:(16/132): libradosstriper1-17.0.0-14898.g244418 4.0 MB/s | 476 kB 00:00 2022-09-10T04:15:06.453 INFO:teuthology.orchestra.run.smithi110.stdout:(10/132): ceph-radosgw-17.0.0-14898.g24441860.e 6.4 MB/s | 18 MB 00:02 2022-09-10T04:15:06.507 INFO:teuthology.orchestra.run.smithi081.stdout:(4/132): ceph-mds-17.0.0-14898.g24441860.el8.x8 1.8 MB/s | 2.3 MB 00:01 2022-09-10T04:15:06.603 INFO:teuthology.orchestra.run.smithi110.stdout:(11/132): cephfs-mirror-17.0.0-14898.g24441860. 1.7 MB/s | 258 kB 00:00 2022-09-10T04:15:06.687 INFO:teuthology.orchestra.run.smithi110.stdout:(12/132): libcephfs-devel-17.0.0-14898.g2444186 367 kB/s | 30 kB 00:00 2022-09-10T04:15:06.782 INFO:teuthology.orchestra.run.smithi104.stdout:(17/132): librgw2-17.0.0-14898.g24441860.el8.x8 5.8 MB/s | 5.8 MB 00:01 2022-09-10T04:15:06.854 INFO:teuthology.orchestra.run.smithi110.stdout:(13/132): libcephfs2-17.0.0-14898.g24441860.el8 4.5 MB/s | 766 kB 00:00 2022-09-10T04:15:06.857 INFO:teuthology.orchestra.run.smithi104.stdout:(18/132): python3-ceph-argparse-17.0.0-14898.g2 596 kB/s | 45 kB 00:00 2022-09-10T04:15:06.929 INFO:teuthology.orchestra.run.smithi110.stdout:(14/132): libcephsqlite-17.0.0-14898.g24441860. 1.8 MB/s | 137 kB 00:00 2022-09-10T04:15:06.932 INFO:teuthology.orchestra.run.smithi104.stdout:(19/132): python3-ceph-common-17.0.0-14898.g244 1.4 MB/s | 109 kB 00:00 2022-09-10T04:15:07.008 INFO:teuthology.orchestra.run.smithi104.stdout:(20/132): python3-cephfs-17.0.0-14898.g24441860 2.5 MB/s | 190 kB 00:00 2022-09-10T04:15:07.091 INFO:teuthology.orchestra.run.smithi104.stdout:(21/132): python3-rados-17.0.0-14898.g24441860. 4.3 MB/s | 371 kB 00:00 2022-09-10T04:15:07.116 INFO:teuthology.orchestra.run.smithi081.stdout:(5/132): ceph-mgr-17.0.0-14898.g24441860.el8.x8 2.5 MB/s | 1.5 MB 00:00 2022-09-10T04:15:07.138 INFO:teuthology.orchestra.run.smithi110.stdout:(15/132): ceph-osd-17.0.0-14898.g24441860.el8.x 3.6 MB/s | 17 MB 00:04 2022-09-10T04:15:07.159 INFO:teuthology.orchestra.run.smithi104.stdout:(22/132): python3-rbd-17.0.0-14898.g24441860.el 5.2 MB/s | 358 kB 00:00 2022-09-10T04:15:07.198 INFO:teuthology.orchestra.run.smithi110.stdout:(16/132): librados-devel-17.0.0-14898.g24441860 518 kB/s | 139 kB 00:00 2022-09-10T04:15:07.276 INFO:teuthology.orchestra.run.smithi104.stdout:(23/132): python3-rgw-17.0.0-14898.g24441860.el 917 kB/s | 107 kB 00:00 2022-09-10T04:15:07.343 INFO:teuthology.orchestra.run.smithi104.stdout:(24/132): rbd-fuse-17.0.0-14898.g24441860.el8.x 1.4 MB/s | 94 kB 00:00 2022-09-10T04:15:07.347 INFO:teuthology.orchestra.run.smithi110.stdout:(17/132): libradosstriper1-17.0.0-14898.g244418 2.2 MB/s | 476 kB 00:00 2022-09-10T04:15:07.430 INFO:teuthology.orchestra.run.smithi110.stdout:(18/132): python3-ceph-argparse-17.0.0-14898.g2 535 kB/s | 45 kB 00:00 2022-09-10T04:15:07.523 INFO:teuthology.orchestra.run.smithi110.stdout:(19/132): python3-ceph-common-17.0.0-14898.g244 1.2 MB/s | 109 kB 00:00 2022-09-10T04:15:07.623 INFO:teuthology.orchestra.run.smithi110.stdout:(20/132): python3-cephfs-17.0.0-14898.g24441860 1.9 MB/s | 190 kB 00:00 2022-09-10T04:15:07.650 INFO:teuthology.orchestra.run.smithi081.stdout:(6/132): ceph-base-17.0.0-14898.g24441860.el8.x 1.4 MB/s | 5.2 MB 00:03 2022-09-10T04:15:07.790 INFO:teuthology.orchestra.run.smithi110.stdout:(21/132): python3-rados-17.0.0-14898.g24441860. 2.2 MB/s | 371 kB 00:00 2022-09-10T04:15:07.943 INFO:teuthology.orchestra.run.smithi104.stdout:(25/132): rbd-mirror-17.0.0-14898.g24441860.el8 5.8 MB/s | 3.5 MB 00:00 2022-09-10T04:15:07.957 INFO:teuthology.orchestra.run.smithi110.stdout:(22/132): python3-rbd-17.0.0-14898.g24441860.el 2.1 MB/s | 358 kB 00:00 2022-09-10T04:15:08.027 INFO:teuthology.orchestra.run.smithi104.stdout:(26/132): rbd-nbd-17.0.0-14898.g24441860.el8.x8 2.1 MB/s | 175 kB 00:00 2022-09-10T04:15:08.050 INFO:teuthology.orchestra.run.smithi110.stdout:(23/132): python3-rgw-17.0.0-14898.g24441860.el 1.1 MB/s | 107 kB 00:00 2022-09-10T04:15:08.103 INFO:teuthology.orchestra.run.smithi104.stdout:(27/132): ceph-grafana-dashboards-17.0.0-14898. 305 kB/s | 23 kB 00:00 2022-09-10T04:15:08.160 INFO:teuthology.orchestra.run.smithi110.stdout:(24/132): rbd-fuse-17.0.0-14898.g24441860.el8.x 872 kB/s | 94 kB 00:00 2022-09-10T04:15:08.177 INFO:teuthology.orchestra.run.smithi104.stdout:(28/132): ceph-mgr-cephadm-17.0.0-14898.g244418 1.5 MB/s | 115 kB 00:00 2022-09-10T04:15:08.528 INFO:teuthology.orchestra.run.smithi104.stdout:(29/132): ceph-mgr-dashboard-17.0.0-14898.g2444 4.7 MB/s | 1.6 MB 00:00 2022-09-10T04:15:08.534 INFO:teuthology.orchestra.run.smithi110.stdout:(25/132): librgw2-17.0.0-14898.g24441860.el8.x8 4.4 MB/s | 5.8 MB 00:01 2022-09-10T04:15:08.636 INFO:teuthology.orchestra.run.smithi110.stdout:(26/132): rbd-nbd-17.0.0-14898.g24441860.el8.x8 1.7 MB/s | 175 kB 00:00 2022-09-10T04:15:08.750 INFO:teuthology.orchestra.run.smithi110.stdout:(27/132): ceph-grafana-dashboards-17.0.0-14898. 196 kB/s | 23 kB 00:00 2022-09-10T04:15:08.751 INFO:teuthology.orchestra.run.smithi081.stdout:(7/132): ceph-mon-17.0.0-14898.g24441860.el8.x8 2.4 MB/s | 3.9 MB 00:01 2022-09-10T04:15:08.843 INFO:teuthology.orchestra.run.smithi110.stdout:(28/132): ceph-mgr-cephadm-17.0.0-14898.g244418 1.2 MB/s | 115 kB 00:00 2022-09-10T04:15:09.314 INFO:teuthology.orchestra.run.smithi110.stdout:(29/132): ceph-mgr-dashboard-17.0.0-14898.g2444 3.5 MB/s | 1.6 MB 00:00 2022-09-10T04:15:09.392 INFO:teuthology.orchestra.run.smithi110.stdout:(30/132): rbd-mirror-17.0.0-14898.g24441860.el8 2.8 MB/s | 3.5 MB 00:01 2022-09-10T04:15:09.503 INFO:teuthology.orchestra.run.smithi110.stdout:(31/132): ceph-mgr-modules-core-17.0.0-14898.g2 2.1 MB/s | 233 kB 00:00 2022-09-10T04:15:09.601 INFO:teuthology.orchestra.run.smithi110.stdout:(32/132): ceph-mgr-rook-17.0.0-14898.g24441860. 470 kB/s | 47 kB 00:00 2022-09-10T04:15:09.710 INFO:teuthology.orchestra.run.smithi110.stdout:(33/132): ceph-prometheus-alerts-17.0.0-14898.g 133 kB/s | 14 kB 00:00 2022-09-10T04:15:09.786 INFO:teuthology.orchestra.run.smithi104.stdout:(30/132): ceph-mgr-diskprediction-local-17.0.0- 5.9 MB/s | 7.4 MB 00:01 2022-09-10T04:15:09.876 INFO:teuthology.orchestra.run.smithi110.stdout:(34/132): ceph-volume-17.0.0-14898.g24441860.el 1.5 MB/s | 263 kB 00:00 2022-09-10T04:15:09.877 INFO:teuthology.orchestra.run.smithi104.stdout:(31/132): ceph-mgr-modules-core-17.0.0-14898.g2 2.5 MB/s | 233 kB 00:00 2022-09-10T04:15:09.969 INFO:teuthology.orchestra.run.smithi110.stdout:(35/132): cephadm-17.0.0-14898.g24441860.el8.no 878 kB/s | 81 kB 00:00 2022-09-10T04:15:09.969 INFO:teuthology.orchestra.run.smithi104.stdout:(32/132): ceph-mgr-rook-17.0.0-14898.g24441860. 512 kB/s | 47 kB 00:00 2022-09-10T04:15:10.044 INFO:teuthology.orchestra.run.smithi110.stdout:(36/132): cephfs-top-17.0.0-14898.g24441860.el8 205 kB/s | 15 kB 00:00 2022-09-10T04:15:10.061 INFO:teuthology.orchestra.run.smithi104.stdout:(33/132): ceph-prometheus-alerts-17.0.0-14898.g 157 kB/s | 14 kB 00:00 2022-09-10T04:15:10.153 INFO:teuthology.orchestra.run.smithi104.stdout:(34/132): ceph-volume-17.0.0-14898.g24441860.el 2.8 MB/s | 263 kB 00:00 2022-09-10T04:15:10.253 INFO:teuthology.orchestra.run.smithi110.stdout:(37/132): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2022-09-10T04:15:10.262 INFO:teuthology.orchestra.run.smithi104.stdout:(35/132): cephadm-17.0.0-14898.g24441860.el8.no 747 kB/s | 81 kB 00:00 2022-09-10T04:15:10.303 INFO:teuthology.orchestra.run.smithi110.stdout:(38/132): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2022-09-10T04:15:10.346 INFO:teuthology.orchestra.run.smithi104.stdout:(36/132): cephfs-top-17.0.0-14898.g24441860.el8 185 kB/s | 15 kB 00:00 2022-09-10T04:15:10.354 INFO:teuthology.orchestra.run.smithi110.stdout:(39/132): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2022-09-10T04:15:10.470 INFO:teuthology.orchestra.run.smithi110.stdout:(40/132): python3-asyncssh-2.7.0-2.el8.noarch.r 4.0 MB/s | 476 kB 00:00 2022-09-10T04:15:10.521 INFO:teuthology.orchestra.run.smithi110.stdout:(41/132): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 867 kB/s | 44 kB 00:00 2022-09-10T04:15:10.537 INFO:teuthology.orchestra.run.smithi104.stdout:(37/132): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2022-09-10T04:15:10.571 INFO:teuthology.orchestra.run.smithi110.stdout:(42/132): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2022-09-10T04:15:10.588 INFO:teuthology.orchestra.run.smithi104.stdout:(38/132): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2022-09-10T04:15:10.613 INFO:teuthology.orchestra.run.smithi110.stdout:(43/132): python3-cachetools-3.1.1-4.el8.noarch 797 kB/s | 33 kB 00:00 2022-09-10T04:15:10.638 INFO:teuthology.orchestra.run.smithi104.stdout:(39/132): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2022-09-10T04:15:10.655 INFO:teuthology.orchestra.run.smithi110.stdout:(44/132): python3-certifi-2018.10.15-7.el8.noar 390 kB/s | 16 kB 00:00 2022-09-10T04:15:10.753 INFO:teuthology.orchestra.run.smithi104.stdout:(40/132): python3-asyncssh-2.7.0-2.el8.noarch.r 4.1 MB/s | 476 kB 00:00 2022-09-10T04:15:10.772 INFO:teuthology.orchestra.run.smithi110.stdout:(45/132): ceph-mgr-diskprediction-local-17.0.0- 5.1 MB/s | 7.4 MB 00:01 2022-09-10T04:15:10.798 INFO:teuthology.orchestra.run.smithi110.stdout:(46/132): python3-cheroot-8.5.2-1.el8.noarch.rp 1.2 MB/s | 173 kB 00:00 2022-09-10T04:15:10.822 INFO:teuthology.orchestra.run.smithi104.stdout:(41/132): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 635 kB/s | 44 kB 00:00 2022-09-10T04:15:10.857 INFO:teuthology.orchestra.run.smithi110.stdout:(47/132): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2022-09-10T04:15:10.864 INFO:teuthology.orchestra.run.smithi104.stdout:(42/132): python3-beautifulsoup4-4.6.3-2.el8.1. 4.4 MB/s | 185 kB 00:00 2022-09-10T04:15:10.899 INFO:teuthology.orchestra.run.smithi110.stdout:(48/132): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2022-09-10T04:15:10.914 INFO:teuthology.orchestra.run.smithi104.stdout:(43/132): python3-cachetools-3.1.1-4.el8.noarch 660 kB/s | 33 kB 00:00 2022-09-10T04:15:10.950 INFO:teuthology.orchestra.run.smithi110.stdout:(49/132): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2022-09-10T04:15:10.964 INFO:teuthology.orchestra.run.smithi104.stdout:(44/132): python3-certifi-2018.10.15-7.el8.noar 326 kB/s | 16 kB 00:00 2022-09-10T04:15:10.974 INFO:teuthology.orchestra.run.smithi110.stdout:(50/132): python3-cherrypy-18.4.0-1.el8.noarch. 1.9 MB/s | 384 kB 00:00 2022-09-10T04:15:11.015 INFO:teuthology.orchestra.run.smithi104.stdout:(45/132): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2022-09-10T04:15:11.041 INFO:teuthology.orchestra.run.smithi110.stdout:(51/132): python3-logutils-0.3.5-11.el8.noarch. 731 kB/s | 49 kB 00:00 2022-09-10T04:15:11.091 INFO:teuthology.orchestra.run.smithi110.stdout:(52/132): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2022-09-10T04:15:11.099 INFO:teuthology.orchestra.run.smithi104.stdout:(46/132): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2022-09-10T04:15:11.150 INFO:teuthology.orchestra.run.smithi110.stdout:(53/132): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2022-09-10T04:15:11.157 INFO:teuthology.orchestra.run.smithi104.stdout:(47/132): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2022-09-10T04:15:11.183 INFO:teuthology.orchestra.run.smithi110.stdout:(54/132): python3-kubernetes-11.0.0-6.el8.noarc 5.0 MB/s | 1.2 MB 00:00 2022-09-10T04:15:11.207 INFO:teuthology.orchestra.run.smithi104.stdout:(48/132): python3-jaraco-6.2-6.el8.noarch.rpm 223 kB/s | 11 kB 00:00 2022-09-10T04:15:11.208 INFO:teuthology.orchestra.run.smithi110.stdout:(55/132): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2022-09-10T04:15:11.234 INFO:teuthology.orchestra.run.smithi110.stdout:(56/132): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2022-09-10T04:15:11.259 INFO:teuthology.orchestra.run.smithi110.stdout:(57/132): python3-repoze-lru-0.7-6.el8.noarch.r 669 kB/s | 34 kB 00:00 2022-09-10T04:15:11.274 INFO:teuthology.orchestra.run.smithi104.stdout:(49/132): python3-jaraco-functools-2.0-4.el8.no 263 kB/s | 18 kB 00:00 2022-09-10T04:15:11.285 INFO:teuthology.orchestra.run.smithi110.stdout:(58/132): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2022-09-10T04:15:11.310 INFO:teuthology.orchestra.run.smithi110.stdout:(59/132): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2022-09-10T04:15:11.341 INFO:teuthology.orchestra.run.smithi110.stdout:(60/132): python3-simplegeneric-0.8.1-17.el8.no 373 kB/s | 19 kB 00:00 2022-09-10T04:15:11.360 INFO:teuthology.orchestra.run.smithi110.stdout:(61/132): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2022-09-10T04:15:11.385 INFO:teuthology.orchestra.run.smithi110.stdout:(62/132): python3-tempora-1.14.1-5.el8.noarch.r 583 kB/s | 29 kB 00:00 2022-09-10T04:15:11.418 INFO:teuthology.orchestra.run.smithi110.stdout:(63/132): python3-trustme-0.6.0-4.el8.noarch.rp 547 kB/s | 27 kB 00:00 2022-09-10T04:15:11.424 INFO:teuthology.orchestra.run.smithi104.stdout:(50/132): python3-kubernetes-11.0.0-6.el8.noarc 7.8 MB/s | 1.2 MB 00:00 2022-09-10T04:15:11.445 INFO:teuthology.orchestra.run.smithi110.stdout:(64/132): python3-waitress-1.2.1-2.el8.1.noarch 4.0 MB/s | 241 kB 00:00 2022-09-10T04:15:11.470 INFO:teuthology.orchestra.run.smithi110.stdout:(65/132): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2022-09-10T04:15:11.475 INFO:teuthology.orchestra.run.smithi104.stdout:(51/132): python3-logutils-0.3.5-11.el8.noarch. 976 kB/s | 49 kB 00:00 2022-09-10T04:15:11.496 INFO:teuthology.orchestra.run.smithi110.stdout:(66/132): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-09-10T04:15:11.520 INFO:teuthology.orchestra.run.smithi110.stdout:(67/132): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2022-09-10T04:15:11.526 INFO:teuthology.orchestra.run.smithi104.stdout:(52/132): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2022-09-10T04:15:11.545 INFO:teuthology.orchestra.run.smithi110.stdout:(68/132): python3-zc-lockfile-2.0-2.el8.noarch. 456 kB/s | 23 kB 00:00 2022-09-10T04:15:11.575 INFO:teuthology.orchestra.run.smithi104.stdout:(53/132): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2022-09-10T04:15:11.613 INFO:teuthology.orchestra.run.smithi110.stdout:(69/132): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 70 kB 00:00 2022-09-10T04:15:11.626 INFO:teuthology.orchestra.run.smithi104.stdout:(54/132): python3-pecan-1.3.2-9.el8.noarch.rpm 5.6 MB/s | 283 kB 00:00 2022-09-10T04:15:11.663 INFO:teuthology.orchestra.run.smithi110.stdout:(70/132): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2022-09-10T04:15:11.676 INFO:teuthology.orchestra.run.smithi104.stdout:(55/132): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2022-09-10T04:15:11.705 INFO:teuthology.orchestra.run.smithi110.stdout:(71/132): fuse-2.9.7-15.el8.x86_64.rpm 2.0 MB/s | 83 kB 00:00 2022-09-10T04:15:11.728 INFO:teuthology.orchestra.run.smithi081.stdout:(8/132): ceph-common-17.0.0-14898.g24441860.el8 3.1 MB/s | 24 MB 00:07 2022-09-10T04:15:11.743 INFO:teuthology.orchestra.run.smithi110.stdout:(72/132): libsepol-devel-2.9-3.el8.x86_64.rpm 2.5 MB/s | 87 kB 00:00 2022-09-10T04:15:11.751 INFO:teuthology.orchestra.run.smithi104.stdout:(56/132): python3-repoze-lru-0.7-6.el8.noarch.r 451 kB/s | 34 kB 00:00 2022-09-10T04:15:11.775 INFO:teuthology.orchestra.run.smithi081.stdout:(9/132): ceph-selinux-17.0.0-14898.g24441860.el 489 kB/s | 24 kB 00:00 2022-09-10T04:15:11.781 INFO:teuthology.orchestra.run.smithi110.stdout:(73/132): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2022-09-10T04:15:11.810 INFO:teuthology.orchestra.run.smithi104.stdout:(57/132): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2022-09-10T04:15:11.814 INFO:teuthology.orchestra.run.smithi110.stdout:(74/132): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2022-09-10T04:15:11.848 INFO:teuthology.orchestra.run.smithi110.stdout:(75/132): libselinux-devel-2.9-5.el8.x86_64.rpm 2.9 MB/s | 200 kB 00:00 2022-09-10T04:15:11.851 INFO:teuthology.orchestra.run.smithi104.stdout:(58/132): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2022-09-10T04:15:11.873 INFO:teuthology.orchestra.run.smithi110.stdout:(76/132): libverto-devel-0.3.0-5.el8.x86_64.rpm 310 kB/s | 18 kB 00:00 2022-09-10T04:15:11.894 INFO:teuthology.orchestra.run.smithi104.stdout:(59/132): python3-simplegeneric-0.8.1-17.el8.no 448 kB/s | 19 kB 00:00 2022-09-10T04:15:11.915 INFO:teuthology.orchestra.run.smithi110.stdout:(77/132): librabbitmq-0.9.0-3.el8.x86_64.rpm 702 kB/s | 47 kB 00:00 2022-09-10T04:15:11.942 INFO:teuthology.orchestra.run.smithi104.stdout:(60/132): python3-singledispatch-3.4.0.3-18.el8 577 kB/s | 24 kB 00:00 2022-09-10T04:15:11.949 INFO:teuthology.orchestra.run.smithi110.stdout:(78/132): python3-cryptography-3.2.1-5.el8.x86_ 7.3 MB/s | 559 kB 00:00 2022-09-10T04:15:11.983 INFO:teuthology.orchestra.run.smithi110.stdout:(79/132): python3-oauthlib-2.1.0-1.el8.noarch.r 2.3 MB/s | 155 kB 00:00 2022-09-10T04:15:12.008 INFO:teuthology.orchestra.run.smithi110.stdout:(80/132): python3-pycparser-2.14-14.el8.noarch. 1.9 MB/s | 109 kB 00:00 2022-09-10T04:15:12.041 INFO:teuthology.orchestra.run.smithi110.stdout:(81/132): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.5 MB/s | 560 kB 00:00 2022-09-10T04:15:12.075 INFO:teuthology.orchestra.run.smithi110.stdout:(82/132): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.5 MB/s | 171 kB 00:00 2022-09-10T04:15:12.100 INFO:teuthology.orchestra.run.smithi110.stdout:(83/132): python3-jwt-1.6.1-2.el8.noarch.rpm 743 kB/s | 43 kB 00:00 2022-09-10T04:15:12.134 INFO:teuthology.orchestra.run.smithi110.stdout:(84/132): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 987 kB/s | 58 kB 00:00 2022-09-10T04:15:12.183 INFO:teuthology.orchestra.run.smithi110.stdout:(85/132): elfutils-libelf-devel-0.186-1.el8.x86 727 kB/s | 60 kB 00:00 2022-09-10T04:15:12.239 INFO:teuthology.orchestra.run.smithi110.stdout:(86/132): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.3 MB/s | 238 kB 00:00 2022-09-10T04:15:12.252 INFO:teuthology.orchestra.run.smithi110.stdout:(87/132): libaio-devel-0.3.112-1.el8.x86_64.rpm 275 kB/s | 19 kB 00:00 2022-09-10T04:15:12.471 INFO:teuthology.orchestra.run.smithi110.stdout:(88/132): libuuid-devel-2.32.1-35.el8.x86_64.rp 420 kB/s | 98 kB 00:00 2022-09-10T04:15:12.486 INFO:teuthology.orchestra.run.smithi104.stdout:(61/132): ceph-test-17.0.0-14898.g24441860.el8. 6.7 MB/s | 54 MB 00:08 2022-09-10T04:15:12.503 INFO:teuthology.orchestra.run.smithi104.stdout:(62/132): python3-tempora-1.14.1-5.el8.noarch.r 52 kB/s | 29 kB 00:00 2022-09-10T04:15:12.619 INFO:teuthology.orchestra.run.smithi110.stdout:(89/132): NetworkManager-initscripts-updown-1.3 374 kB/s | 137 kB 00:00 2022-09-10T04:15:12.670 INFO:teuthology.orchestra.run.smithi110.stdout:(90/132): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.1 MB/s | 229 kB 00:00 2022-09-10T04:15:12.695 INFO:teuthology.orchestra.run.smithi110.stdout:(91/132): python3-requests-oauthlib-1.0.0-1.el8 576 kB/s | 43 kB 00:00 2022-09-10T04:15:12.770 INFO:teuthology.orchestra.run.smithi110.stdout:(92/132): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 722 kB/s | 72 kB 00:00 2022-09-10T04:15:12.787 INFO:teuthology.orchestra.run.smithi110.stdout:(93/132): libcom_err-devel-1.45.6-4.el8.x86_64. 420 kB/s | 39 kB 00:00 2022-09-10T04:15:12.821 INFO:teuthology.orchestra.run.smithi110.stdout:(94/132): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2022-09-10T04:15:12.879 INFO:teuthology.orchestra.run.smithi110.stdout:(95/132): openssl-devel-1.1.1k-7.el8_6.x86_64.r 21 MB/s | 2.3 MB 00:00 2022-09-10T04:15:12.921 INFO:teuthology.orchestra.run.smithi110.stdout:(96/132): libgfortran-8.5.0-10.1.el8_6.x86_64.r 6.3 MB/s | 644 kB 00:00 2022-09-10T04:15:12.928 INFO:teuthology.orchestra.run.smithi104.stdout:(63/132): ceph-radosgw-17.0.0-14898.g24441860.e 1.8 MB/s | 18 MB 00:10 2022-09-10T04:15:12.953 INFO:teuthology.orchestra.run.smithi104.stdout:(64/132): python3-trustme-0.6.0-4.el8.noarch.rp 59 kB/s | 27 kB 00:00 2022-09-10T04:15:12.963 INFO:teuthology.orchestra.run.smithi110.stdout:(97/132): postgresql-contrib-10.21-2.module+el8 9.5 MB/s | 808 kB 00:00 2022-09-10T04:15:12.979 INFO:teuthology.orchestra.run.smithi104.stdout:(65/132): python3-waitress-1.2.1-2.el8.1.noarch 506 kB/s | 241 kB 00:00 2022-09-10T04:15:12.997 INFO:teuthology.orchestra.run.smithi110.stdout:(98/132): bison-3.0.4-10.el8.x86_64.rpm 9.1 MB/s | 688 kB 00:00 2022-09-10T04:15:13.029 INFO:teuthology.orchestra.run.smithi104.stdout:(66/132): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2022-09-10T04:15:13.057 INFO:teuthology.orchestra.run.smithi110.stdout:(99/132): postgresql-10.21-2.module+el8.6.0+153 16 MB/s | 1.5 MB 00:00 2022-09-10T04:15:13.063 INFO:teuthology.orchestra.run.smithi104.stdout:(67/132): python3-websocket-client-0.56.0-5.el8 562 kB/s | 61 kB 00:00 2022-09-10T04:15:13.088 INFO:teuthology.orchestra.run.smithi104.stdout:(68/132): python3-zc-lockfile-2.0-2.el8.noarch. 392 kB/s | 23 kB 00:00 2022-09-10T04:15:13.100 INFO:teuthology.orchestra.run.smithi081.stdout:(10/132): ceph-osd-17.0.0-14898.g24441860.el8.x 3.1 MB/s | 17 MB 00:05 2022-09-10T04:15:13.114 INFO:teuthology.orchestra.run.smithi104.stdout:(69/132): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2022-09-10T04:15:13.131 INFO:teuthology.orchestra.run.smithi110.stdout:(100/132): python3-numpy-1.14.3-10.el8.x86_64.r 28 MB/s | 3.7 MB 00:00 2022-09-10T04:15:13.138 INFO:teuthology.orchestra.run.smithi104.stdout:(70/132): python3-webob-1.8.5-1.el8.1.noarch.rp 1.2 MB/s | 251 kB 00:00 2022-09-10T04:15:13.173 INFO:teuthology.orchestra.run.smithi110.stdout:(101/132): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2022-09-10T04:15:13.189 INFO:teuthology.orchestra.run.smithi104.stdout:(71/132): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2022-09-10T04:15:13.222 INFO:teuthology.orchestra.run.smithi104.stdout:(72/132): postgresql-contrib-10.21-2.module+el8 7.3 MB/s | 808 kB 00:00 2022-09-10T04:15:13.265 INFO:teuthology.orchestra.run.smithi110.stdout:(102/132): postgresql-server-10.21-2.module+el8 24 MB/s | 5.1 MB 00:00 2022-09-10T04:15:13.281 INFO:teuthology.orchestra.run.smithi104.stdout:(73/132): postgresql-10.21-2.module+el8.6.0+153 16 MB/s | 1.5 MB 00:00 2022-09-10T04:15:13.290 INFO:teuthology.orchestra.run.smithi110.stdout:(103/132): python3-markupsafe-0.23-19.el8.x86_6 333 kB/s | 39 kB 00:00 2022-09-10T04:15:13.292 INFO:teuthology.orchestra.run.smithi081.stdout:(11/132): ceph-radosgw-17.0.0-14898.g24441860.e 3.9 MB/s | 18 MB 00:04 2022-09-10T04:15:13.326 INFO:teuthology.orchestra.run.smithi081.stdout:(12/132): cephfs-mirror-17.0.0-14898.g24441860. 1.1 MB/s | 258 kB 00:00 2022-09-10T04:15:13.341 INFO:teuthology.orchestra.run.smithi110.stdout:(104/132): python3-prettytable-0.7.2-14.el8.noa 585 kB/s | 44 kB 00:00 2022-09-10T04:15:13.351 INFO:teuthology.orchestra.run.smithi081.stdout:(13/132): libcephfs-devel-17.0.0-14898.g2444186 523 kB/s | 30 kB 00:00 2022-09-10T04:15:13.356 INFO:teuthology.orchestra.run.smithi104.stdout:(74/132): python3-numpy-1.14.3-10.el8.x86_64.rp 28 MB/s | 3.7 MB 00:00 2022-09-10T04:15:13.366 INFO:teuthology.orchestra.run.smithi110.stdout:(105/132): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2022-09-10T04:15:13.408 INFO:teuthology.orchestra.run.smithi110.stdout:(106/132): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2022-09-10T04:15:13.422 INFO:teuthology.orchestra.run.smithi104.stdout:(75/132): thrift-0.13.0-2.el8.x86_64.rpm 4.8 MB/s | 1.7 MB 00:00 2022-09-10T04:15:13.426 INFO:teuthology.orchestra.run.smithi081.stdout:(14/132): libcephsqlite-17.0.0-14898.g24441860. 1.8 MB/s | 137 kB 00:00 2022-09-10T04:15:13.441 INFO:teuthology.orchestra.run.smithi110.stdout:(107/132): libtool-2.4.6-25.el8.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2022-09-10T04:15:13.465 INFO:teuthology.orchestra.run.smithi104.stdout:(76/132): python3-werkzeug-0.12.2-4.el8.noarch. 4.1 MB/s | 457 kB 00:00 2022-09-10T04:15:13.475 INFO:teuthology.orchestra.run.smithi110.stdout:(108/132): python3-numpy-f2py-1.14.3-10.el8.x86 3.3 MB/s | 225 kB 00:00 2022-09-10T04:15:13.491 INFO:teuthology.orchestra.run.smithi104.stdout:(77/132): python3-markupsafe-0.23-19.el8.x86_64 584 kB/s | 39 kB 00:00 2022-09-10T04:15:13.493 INFO:teuthology.orchestra.run.smithi081.stdout:(15/132): libcephfs2-17.0.0-14898.g24441860.el8 4.5 MB/s | 766 kB 00:00 2022-09-10T04:15:13.509 INFO:teuthology.orchestra.run.smithi110.stdout:(109/132): python3-mako-1.0.6-13.el8.noarch.rpm 4.6 MB/s | 157 kB 00:00 2022-09-10T04:15:13.519 INFO:teuthology.orchestra.run.smithi081.stdout:(16/132): librados-devel-17.0.0-14898.g24441860 1.5 MB/s | 139 kB 00:00 2022-09-10T04:15:13.532 INFO:teuthology.orchestra.run.smithi104.stdout:(78/132): python3-prettytable-0.7.2-14.el8.noar 658 kB/s | 44 kB 00:00 2022-09-10T04:15:13.558 INFO:teuthology.orchestra.run.smithi104.stdout:(79/132): librdkafka-0.11.4-3.el8.x86_64.rpm 5.2 MB/s | 354 kB 00:00 2022-09-10T04:15:13.584 INFO:teuthology.orchestra.run.smithi110.stdout:(110/132): openblas-threads-0.3.15-3.el8.x86_64 34 MB/s | 4.9 MB 00:00 2022-09-10T04:15:13.611 INFO:teuthology.orchestra.run.smithi081.stdout:(17/132): libradosstriper1-17.0.0-14898.g244418 4.0 MB/s | 476 kB 00:00 2022-09-10T04:15:13.617 INFO:teuthology.orchestra.run.smithi110.stdout:(111/132): python3-pyOpenSSL-19.0.0-1.el8.noarc 3.0 MB/s | 103 kB 00:00 2022-09-10T04:15:13.641 INFO:teuthology.orchestra.run.smithi104.stdout:(80/132): postgresql-server-10.21-2.module+el8. 14 MB/s | 5.1 MB 00:00 2022-09-10T04:15:13.669 INFO:teuthology.orchestra.run.smithi081.stdout:(18/132): python3-ceph-argparse-17.0.0-14898.g2 768 kB/s | 45 kB 00:00 2022-09-10T04:15:13.683 INFO:teuthology.orchestra.run.smithi104.stdout:(81/132): python3-jinja2-2.10.1-3.el8.noarch.rp 3.5 MB/s | 538 kB 00:00 2022-09-10T04:15:13.701 INFO:teuthology.orchestra.run.smithi110.stdout:(112/132): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2022-09-10T04:15:13.717 INFO:teuthology.orchestra.run.smithi104.stdout:(82/132): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2022-09-10T04:15:13.726 INFO:teuthology.orchestra.run.smithi110.stdout:(113/132): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2022-09-10T04:15:13.728 INFO:teuthology.orchestra.run.smithi081.stdout:(19/132): python3-ceph-common-17.0.0-14898.g244 1.8 MB/s | 109 kB 00:00 2022-09-10T04:15:13.742 INFO:teuthology.orchestra.run.smithi104.stdout:(83/132): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.2 MB/s | 225 kB 00:00 2022-09-10T04:15:13.760 INFO:teuthology.orchestra.run.smithi110.stdout:(114/132): flex-2.6.1-9.el8.x86_64.rpm 5.3 MB/s | 320 kB 00:00 2022-09-10T04:15:13.776 INFO:teuthology.orchestra.run.smithi104.stdout:(84/132): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2022-09-10T04:15:13.794 INFO:teuthology.orchestra.run.smithi110.stdout:(115/132): jq-1.6-3.el8.x86_64.rpm 5.9 MB/s | 202 kB 00:00 2022-09-10T04:15:13.795 INFO:teuthology.orchestra.run.smithi081.stdout:(20/132): python3-cephfs-17.0.0-14898.g24441860 2.8 MB/s | 190 kB 00:00 2022-09-10T04:15:13.818 INFO:teuthology.orchestra.run.smithi104.stdout:(85/132): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2022-09-10T04:15:13.887 INFO:teuthology.orchestra.run.smithi081.stdout:(21/132): python3-rados-17.0.0-14898.g24441860. 4.0 MB/s | 371 kB 00:00 2022-09-10T04:15:13.892 INFO:teuthology.orchestra.run.smithi104.stdout:(86/132): openblas-threads-0.3.15-3.el8.x86_64. 23 MB/s | 4.9 MB 00:00 2022-09-10T04:15:13.902 INFO:teuthology.orchestra.run.smithi110.stdout:(116/132): openblas-0.3.15-3.el8.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2022-09-10T04:15:13.943 INFO:teuthology.orchestra.run.smithi104.stdout:(87/132): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2022-09-10T04:15:13.958 INFO:teuthology.orchestra.run.smithi110.stdout:(117/132): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2022-09-10T04:15:13.979 INFO:teuthology.orchestra.run.smithi081.stdout:(22/132): python3-rbd-17.0.0-14898.g24441860.el 3.8 MB/s | 358 kB 00:00 2022-09-10T04:15:13.995 INFO:teuthology.orchestra.run.smithi110.stdout:(118/132): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2022-09-10T04:15:14.018 INFO:teuthology.orchestra.run.smithi104.stdout:(88/132): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2022-09-10T04:15:14.036 INFO:teuthology.orchestra.run.smithi110.stdout:(119/132): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.0 MB/s | 126 kB 00:00 2022-09-10T04:15:14.037 INFO:teuthology.orchestra.run.smithi081.stdout:(23/132): python3-rgw-17.0.0-14898.g24441860.el 1.8 MB/s | 107 kB 00:00 2022-09-10T04:15:14.043 INFO:teuthology.orchestra.run.smithi104.stdout:(89/132): flex-2.6.1-9.el8.x86_64.rpm 2.1 MB/s | 320 kB 00:00 2022-09-10T04:15:14.062 INFO:teuthology.orchestra.run.smithi110.stdout:(120/132): python3-jsonpointer-1.10-11.el8.noar 804 kB/s | 20 kB 00:00 2022-09-10T04:15:14.077 INFO:teuthology.orchestra.run.smithi104.stdout:(90/132): jq-1.6-3.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2022-09-10T04:15:14.104 INFO:teuthology.orchestra.run.smithi110.stdout:(121/132): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2022-09-10T04:15:14.124 INFO:teuthology.orchestra.run.smithi081.stdout:(24/132): rbd-fuse-17.0.0-14898.g24441860.el8.x 1.1 MB/s | 94 kB 00:00 2022-09-10T04:15:14.136 INFO:teuthology.orchestra.run.smithi104.stdout:(91/132): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2022-09-10T04:15:14.211 INFO:teuthology.orchestra.run.smithi104.stdout:(92/132): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2022-09-10T04:15:14.236 INFO:teuthology.orchestra.run.smithi104.stdout:(93/132): python3-pytz-2017.2-9.el8.noarch.rpm 541 kB/s | 54 kB 00:00 2022-09-10T04:15:14.246 INFO:teuthology.orchestra.run.smithi110.stdout:(122/132): python3-scipy-1.0.0-21.module+el8.5. 30 MB/s | 14 MB 00:00 2022-09-10T04:15:14.253 INFO:teuthology.orchestra.run.smithi104.stdout:(94/132): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2022-09-10T04:15:14.278 INFO:teuthology.orchestra.run.smithi104.stdout:(95/132): python3-jsonpointer-1.10-11.el8.noarc 478 kB/s | 20 kB 00:00 2022-09-10T04:15:14.288 INFO:teuthology.orchestra.run.smithi110.stdout:(123/132): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2022-09-10T04:15:14.304 INFO:teuthology.orchestra.run.smithi104.stdout:(96/132): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2022-09-10T04:15:14.314 INFO:teuthology.orchestra.run.smithi110.stdout:(124/132): oniguruma-6.8.2-2.el8.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2022-09-10T04:15:14.346 INFO:teuthology.orchestra.run.smithi104.stdout:(97/132): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.8 MB/s | 259 kB 00:00 2022-09-10T04:15:14.347 INFO:teuthology.orchestra.run.smithi110.stdout:(125/132): python3-jsonpatch-1.21-2.el8.noarch. 455 kB/s | 27 kB 00:00 2022-09-10T04:15:14.371 INFO:teuthology.orchestra.run.smithi104.stdout:(98/132): oniguruma-6.8.2-2.el8.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2022-09-10T04:15:14.372 INFO:teuthology.orchestra.run.smithi110.stdout:(126/132): uuid-1.6.2-43.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2022-09-10T04:15:14.406 INFO:teuthology.orchestra.run.smithi110.stdout:(127/132): python3-pyasn1-modules-0.3.7-6.el8.n 1.8 MB/s | 110 kB 00:00 2022-09-10T04:15:14.415 INFO:teuthology.orchestra.run.smithi081.stdout:(25/132): librgw2-17.0.0-14898.g24441860.el8.x8 6.5 MB/s | 5.8 MB 00:00 2022-09-10T04:15:14.427 INFO:teuthology.orchestra.run.smithi104.stdout:(99/132): python3-jsonpatch-1.21-2.el8.noarch.r 324 kB/s | 27 kB 00:00 2022-09-10T04:15:14.454 INFO:teuthology.orchestra.run.smithi104.stdout:(100/132): uuid-1.6.2-43.el8.x86_64.rpm 776 kB/s | 64 kB 00:00 2022-09-10T04:15:14.474 INFO:teuthology.orchestra.run.smithi081.stdout:(26/132): rbd-nbd-17.0.0-14898.g24441860.el8.x8 3.0 MB/s | 175 kB 00:00 2022-09-10T04:15:14.488 INFO:teuthology.orchestra.run.smithi104.stdout:(101/132): python3-pyasn1-modules-0.3.7-6.el8.n 1.8 MB/s | 110 kB 00:00 2022-09-10T04:15:14.522 INFO:teuthology.orchestra.run.smithi104.stdout:(102/132): pcre2-devel-10.32-2.el8.x86_64.rpm 8.8 MB/s | 605 kB 00:00 2022-09-10T04:15:14.532 INFO:teuthology.orchestra.run.smithi081.stdout:(27/132): ceph-grafana-dashboards-17.0.0-14898. 393 kB/s | 23 kB 00:00 2022-09-10T04:15:14.599 INFO:teuthology.orchestra.run.smithi081.stdout:(28/132): ceph-mgr-cephadm-17.0.0-14898.g244418 1.7 MB/s | 115 kB 00:00 2022-09-10T04:15:14.656 INFO:teuthology.orchestra.run.smithi104.stdout:(103/132): python3-scipy-1.0.0-21.module+el8.5. 22 MB/s | 14 MB 00:00 2022-09-10T04:15:14.682 INFO:teuthology.orchestra.run.smithi104.stdout:(104/132): fuse-2.9.7-15.el8.x86_64.rpm 431 kB/s | 83 kB 00:00 2022-09-10T04:15:14.706 INFO:teuthology.orchestra.run.smithi104.stdout:(105/132): libsepol-devel-2.9-3.el8.x86_64.rpm 476 kB/s | 87 kB 00:00 2022-09-10T04:15:14.731 INFO:teuthology.orchestra.run.smithi104.stdout:(106/132): libselinux-devel-2.9-5.el8.x86_64.rp 4.0 MB/s | 200 kB 00:00 2022-09-10T04:15:14.757 INFO:teuthology.orchestra.run.smithi104.stdout:(107/132): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.1 MB/s | 220 kB 00:00 2022-09-10T04:15:14.791 INFO:teuthology.orchestra.run.smithi104.stdout:(108/132): libverto-devel-0.3.0-5.el8.x86_64.rp 213 kB/s | 18 kB 00:00 2022-09-10T04:15:14.816 INFO:teuthology.orchestra.run.smithi104.stdout:(109/132): librabbitmq-0.9.0-3.el8.x86_64.rpm 565 kB/s | 47 kB 00:00 2022-09-10T04:15:14.849 INFO:teuthology.orchestra.run.smithi104.stdout:(110/132): python3-cryptography-3.2.1-5.el8.x86 5.9 MB/s | 559 kB 00:00 2022-09-10T04:15:14.882 INFO:teuthology.orchestra.run.smithi104.stdout:(111/132): python3-pycparser-2.14-14.el8.noarch 1.6 MB/s | 109 kB 00:00 2022-09-10T04:15:14.891 INFO:teuthology.orchestra.run.smithi081.stdout:(29/132): rbd-mirror-17.0.0-14898.g24441860.el8 4.5 MB/s | 3.5 MB 00:00 2022-09-10T04:15:14.939 INFO:teuthology.orchestra.run.smithi104.stdout:(112/132): python3-oauthlib-2.1.0-1.el8.noarch. 1.0 MB/s | 155 kB 00:00 2022-09-10T04:15:14.983 INFO:teuthology.orchestra.run.smithi104.stdout:(113/132): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.1 MB/s | 560 kB 00:00 2022-09-10T04:15:15.009 INFO:teuthology.orchestra.run.smithi104.stdout:(114/132): python3-jwt-1.6.1-2.el8.noarch.rpm 616 kB/s | 43 kB 00:00 2022-09-10T04:15:15.034 INFO:teuthology.orchestra.run.smithi104.stdout:(115/132): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.1 MB/s | 171 kB 00:00 2022-09-10T04:15:15.041 INFO:teuthology.orchestra.run.smithi081.stdout:(30/132): ceph-mgr-dashboard-17.0.0-14898.g2444 3.7 MB/s | 1.6 MB 00:00 2022-09-10T04:15:15.075 INFO:teuthology.orchestra.run.smithi104.stdout:(116/132): zlib-devel-1.2.11-18.el8_5.x86_64.rp 628 kB/s | 58 kB 00:00 2022-09-10T04:15:15.092 INFO:teuthology.orchestra.run.smithi104.stdout:(117/132): elfutils-libelf-devel-0.186-1.el8.x8 717 kB/s | 60 kB 00:00 2022-09-10T04:15:15.118 INFO:teuthology.orchestra.run.smithi104.stdout:(118/132): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2022-09-10T04:15:15.123 INFO:teuthology.orchestra.run.smithi110.stdout:(128/132): librbd1-17.0.0-14898.g24441860.el8.x 5.0 MB/s | 3.6 MB 00:00 2022-09-10T04:15:15.134 INFO:teuthology.orchestra.run.smithi081.stdout:(31/132): ceph-mgr-modules-core-17.0.0-14898.g2 2.5 MB/s | 233 kB 00:00 2022-09-10T04:15:15.151 INFO:teuthology.orchestra.run.smithi104.stdout:(119/132): libaio-devel-0.3.112-1.el8.x86_64.rp 248 kB/s | 19 kB 00:00 2022-09-10T04:15:15.165 INFO:teuthology.orchestra.run.smithi110.stdout:(129/132): openssl-1.1.1k-7.el8_6.x86_64.rpm 17 MB/s | 709 kB 00:00 2022-09-10T04:15:15.177 INFO:teuthology.orchestra.run.smithi104.stdout:(120/132): libuuid-devel-2.32.1-35.el8.x86_64.r 1.1 MB/s | 98 kB 00:00 2022-09-10T04:15:15.193 INFO:teuthology.orchestra.run.smithi104.stdout:(121/132): NetworkManager-initscripts-updown-1. 1.8 MB/s | 137 kB 00:00 2022-09-10T04:15:15.215 INFO:teuthology.orchestra.run.smithi110.stdout:(130/132): openssl-libs-1.1.1k-7.el8_6.x86_64.r 30 MB/s | 1.5 MB 00:00 2022-09-10T04:15:15.235 INFO:teuthology.orchestra.run.smithi104.stdout:(122/132): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.7 MB/s | 229 kB 00:00 2022-09-10T04:15:15.261 INFO:teuthology.orchestra.run.smithi104.stdout:(123/132): python3-requests-oauthlib-1.0.0-1.el 511 kB/s | 43 kB 00:00 2022-09-10T04:15:15.265 INFO:teuthology.orchestra.run.smithi110.stdout:(131/132): librados2-17.0.0-14898.g24441860.el8 4.0 MB/s | 3.6 MB 00:00 2022-09-10T04:15:15.286 INFO:teuthology.orchestra.run.smithi104.stdout:(124/132): xfsprogs-devel-5.0.0-10.el8.x86_64.r 794 kB/s | 72 kB 00:00 2022-09-10T04:15:15.328 INFO:teuthology.orchestra.run.smithi104.stdout:(125/132): libcom_err-devel-1.45.6-4.el8.x86_64 419 kB/s | 39 kB 00:00 2022-09-10T04:15:15.361 INFO:teuthology.orchestra.run.smithi104.stdout:(126/132): keyutils-libs-devel-1.5.10-9.el8.x86 649 kB/s | 48 kB 00:00 2022-09-10T04:15:15.403 INFO:teuthology.orchestra.run.smithi104.stdout:(127/132): libgfortran-8.5.0-10.1.el8_6.x86_64. 8.5 MB/s | 644 kB 00:00 2022-09-10T04:15:15.426 INFO:teuthology.orchestra.run.smithi081.stdout:(32/132): ceph-mgr-rook-17.0.0-14898.g24441860. 161 kB/s | 47 kB 00:00 2022-09-10T04:15:15.476 INFO:teuthology.orchestra.run.smithi081.stdout:(33/132): ceph-prometheus-alerts-17.0.0-14898.g 289 kB/s | 14 kB 00:00 2022-09-10T04:15:15.478 INFO:teuthology.orchestra.run.smithi104.stdout:(128/132): openssl-devel-1.1.1k-7.el8_6.x86_64. 11 MB/s | 2.3 MB 00:00 2022-09-10T04:15:15.521 INFO:teuthology.orchestra.run.smithi104.stdout:(129/132): openssl-1.1.1k-7.el8_6.x86_64.rpm 17 MB/s | 709 kB 00:00 2022-09-10T04:15:15.579 INFO:teuthology.orchestra.run.smithi104.stdout:(130/132): openssl-libs-1.1.1k-7.el8_6.x86_64.r 25 MB/s | 1.5 MB 00:00 2022-09-10T04:15:15.599 INFO:teuthology.orchestra.run.smithi081.stdout:(34/132): ceph-volume-17.0.0-14898.g24441860.el 2.2 MB/s | 263 kB 00:00 2022-09-10T04:15:15.660 INFO:teuthology.orchestra.run.smithi081.stdout:(35/132): cephadm-17.0.0-14898.g24441860.el8.no 1.2 MB/s | 81 kB 00:00 2022-09-10T04:15:15.727 INFO:teuthology.orchestra.run.smithi081.stdout:(36/132): cephfs-top-17.0.0-14898.g24441860.el8 232 kB/s | 15 kB 00:00 2022-09-10T04:15:15.952 INFO:teuthology.orchestra.run.smithi081.stdout:(37/132): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2022-09-10T04:15:15.996 INFO:teuthology.orchestra.run.smithi104.stdout:(131/132): librbd1-17.0.0-14898.g24441860.el8.x 6.0 MB/s | 3.6 MB 00:00 2022-09-10T04:15:16.003 INFO:teuthology.orchestra.run.smithi081.stdout:(38/132): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2022-09-10T04:15:16.061 INFO:teuthology.orchestra.run.smithi081.stdout:(39/132): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2022-09-10T04:15:16.153 INFO:teuthology.orchestra.run.smithi081.stdout:(40/132): python3-asyncssh-2.7.0-2.el8.noarch.r 5.1 MB/s | 476 kB 00:00 2022-09-10T04:15:16.212 INFO:teuthology.orchestra.run.smithi081.stdout:(41/132): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 748 kB/s | 44 kB 00:00 2022-09-10T04:15:16.279 INFO:teuthology.orchestra.run.smithi081.stdout:(42/132): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2022-09-10T04:15:16.338 INFO:teuthology.orchestra.run.smithi081.stdout:(43/132): python3-cachetools-3.1.1-4.el8.noarch 568 kB/s | 33 kB 00:00 2022-09-10T04:15:16.363 INFO:teuthology.orchestra.run.smithi104.stdout:(132/132): librados2-17.0.0-14898.g24441860.el8 3.6 MB/s | 3.6 MB 00:01 2022-09-10T04:15:16.366 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:15:16.367 INFO:teuthology.orchestra.run.smithi104.stdout:Total 14 MB/s | 216 MB 00:15 2022-09-10T04:15:16.396 INFO:teuthology.orchestra.run.smithi081.stdout:(44/132): python3-certifi-2018.10.15-7.el8.noar 280 kB/s | 16 kB 00:00 2022-09-10T04:15:16.455 INFO:teuthology.orchestra.run.smithi081.stdout:(45/132): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2022-09-10T04:15:16.555 INFO:teuthology.orchestra.run.smithi081.stdout:(46/132): ceph-mgr-diskprediction-local-17.0.0- 4.4 MB/s | 7.4 MB 00:01 2022-09-10T04:15:16.589 INFO:teuthology.orchestra.run.smithi081.stdout:(47/132): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2022-09-10T04:15:16.639 INFO:teuthology.orchestra.run.smithi081.stdout:(48/132): python3-jaraco-6.2-6.el8.noarch.rpm 223 kB/s | 11 kB 00:00 2022-09-10T04:15:16.690 INFO:teuthology.orchestra.run.smithi081.stdout:(49/132): python3-jaraco-functools-2.0-4.el8.no 348 kB/s | 18 kB 00:00 2022-09-10T04:15:16.724 INFO:teuthology.orchestra.run.smithi081.stdout:(50/132): python3-google-auth-1.1.1-10.el8.noar 485 kB/s | 81 kB 00:00 2022-09-10T04:15:16.765 INFO:teuthology.orchestra.run.smithi081.stdout:(51/132): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2022-09-10T04:15:16.807 INFO:teuthology.orchestra.run.smithi081.stdout:(52/132): python3-more-itertools-7.2.0-3.el8.no 1.4 MB/s | 59 kB 00:00 2022-09-10T04:15:16.849 INFO:teuthology.orchestra.run.smithi081.stdout:(53/132): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2022-09-10T04:15:16.891 INFO:teuthology.orchestra.run.smithi081.stdout:(54/132): python3-kubernetes-11.0.0-6.el8.noarc 5.8 MB/s | 1.2 MB 00:00 2022-09-10T04:15:16.941 INFO:teuthology.orchestra.run.smithi081.stdout:(55/132): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2022-09-10T04:15:16.958 INFO:teuthology.orchestra.run.smithi081.stdout:(56/132): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2022-09-10T04:15:16.984 INFO:teuthology.orchestra.run.smithi081.stdout:(57/132): python3-repoze-lru-0.7-6.el8.noarch.r 802 kB/s | 34 kB 00:00 2022-09-10T04:15:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:(58/132): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2022-09-10T04:15:17.034 INFO:teuthology.orchestra.run.smithi081.stdout:(59/132): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2022-09-10T04:15:17.085 INFO:teuthology.orchestra.run.smithi081.stdout:(60/132): python3-simplegeneric-0.8.1-17.el8.no 279 kB/s | 19 kB 00:00 2022-09-10T04:15:17.110 INFO:teuthology.orchestra.run.smithi081.stdout:(61/132): python3-singledispatch-3.4.0.3-18.el8 318 kB/s | 24 kB 00:00 2022-09-10T04:15:17.135 INFO:teuthology.orchestra.run.smithi081.stdout:(62/132): python3-tempora-1.14.1-5.el8.noarch.r 584 kB/s | 29 kB 00:00 2022-09-10T04:15:17.161 INFO:teuthology.orchestra.run.smithi081.stdout:(63/132): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2022-09-10T04:15:17.194 INFO:teuthology.orchestra.run.smithi081.stdout:(64/132): python3-waitress-1.2.1-2.el8.1.noarch 4.0 MB/s | 241 kB 00:00 2022-09-10T04:15:17.219 INFO:teuthology.orchestra.run.smithi081.stdout:(65/132): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2022-09-10T04:15:17.224 INFO:teuthology.orchestra.run.smithi110.stdout:(132/132): ceph-test-17.0.0-14898.g24441860.el8 4.0 MB/s | 54 MB 00:13 2022-09-10T04:15:17.227 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:15:17.227 INFO:teuthology.orchestra.run.smithi110.stdout:Total 13 MB/s | 216 MB 00:16 2022-09-10T04:15:17.236 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2022-09-10T04:15:17.245 INFO:teuthology.orchestra.run.smithi081.stdout:(66/132): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2022-09-10T04:15:17.270 INFO:teuthology.orchestra.run.smithi081.stdout:(67/132): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2022-09-10T04:15:17.296 INFO:teuthology.orchestra.run.smithi081.stdout:(68/132): python3-zc-lockfile-2.0-2.el8.noarch. 455 kB/s | 23 kB 00:00 2022-09-10T04:15:17.321 INFO:teuthology.orchestra.run.smithi081.stdout:(69/132): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2022-09-10T04:15:17.380 INFO:teuthology.orchestra.run.smithi081.stdout:(70/132): postgresql-contrib-10.21-2.module+el8 14 MB/s | 808 kB 00:00 2022-09-10T04:15:17.393 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2022-09-10T04:15:17.394 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2022-09-10T04:15:17.430 INFO:teuthology.orchestra.run.smithi081.stdout:(71/132): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2022-09-10T04:15:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:(72/132): postgresql-10.21-2.module+el8.6.0+153 26 MB/s | 1.5 MB 00:00 2022-09-10T04:15:17.555 INFO:teuthology.orchestra.run.smithi081.stdout:(73/132): thrift-0.13.0-2.el8.x86_64.rpm 6.1 MB/s | 1.7 MB 00:00 2022-09-10T04:15:17.664 INFO:teuthology.orchestra.run.smithi081.stdout:(74/132): python3-numpy-1.14.3-10.el8.x86_64.rp 21 MB/s | 3.7 MB 00:00 2022-09-10T04:15:17.706 INFO:teuthology.orchestra.run.smithi081.stdout:(75/132): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2022-09-10T04:15:17.757 INFO:teuthology.orchestra.run.smithi081.stdout:(76/132): python3-markupsafe-0.23-19.el8.x86_64 777 kB/s | 39 kB 00:00 2022-09-10T04:15:17.799 INFO:teuthology.orchestra.run.smithi081.stdout:(77/132): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2022-09-10T04:15:17.882 INFO:teuthology.orchestra.run.smithi081.stdout:(78/132): postgresql-server-10.21-2.module+el8. 16 MB/s | 5.1 MB 00:00 2022-09-10T04:15:17.916 INFO:teuthology.orchestra.run.smithi081.stdout:(79/132): librdkafka-0.11.4-3.el8.x86_64.rpm 3.0 MB/s | 354 kB 00:00 2022-09-10T04:15:17.958 INFO:teuthology.orchestra.run.smithi081.stdout:(80/132): python3-jinja2-2.10.1-3.el8.noarch.rp 7.0 MB/s | 538 kB 00:00 2022-09-10T04:15:18.000 INFO:teuthology.orchestra.run.smithi081.stdout:(81/132): libtool-2.4.6-25.el8.x86_64.rpm 8.4 MB/s | 709 kB 00:00 2022-09-10T04:15:18.034 INFO:teuthology.orchestra.run.smithi081.stdout:(82/132): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.9 MB/s | 225 kB 00:00 2022-09-10T04:15:18.075 INFO:teuthology.orchestra.run.smithi081.stdout:(83/132): python3-mako-1.0.6-13.el8.noarch.rpm 3.8 MB/s | 157 kB 00:00 2022-09-10T04:15:18.101 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2022-09-10T04:15:18.235 INFO:teuthology.orchestra.run.smithi081.stdout:(84/132): openblas-threads-0.3.15-3.el8.x86_64. 21 MB/s | 4.9 MB 00:00 2022-09-10T04:15:18.264 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2022-09-10T04:15:18.264 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2022-09-10T04:15:18.277 INFO:teuthology.orchestra.run.smithi081.stdout:(85/132): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2022-09-10T04:15:18.351 INFO:teuthology.orchestra.run.smithi081.stdout:(86/132): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2022-09-10T04:15:18.385 INFO:teuthology.orchestra.run.smithi081.stdout:(87/132): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2022-09-10T04:15:18.419 INFO:teuthology.orchestra.run.smithi081.stdout:(88/132): flex-2.6.1-9.el8.x86_64.rpm 4.7 MB/s | 320 kB 00:00 2022-09-10T04:15:18.461 INFO:teuthology.orchestra.run.smithi081.stdout:(89/132): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2022-09-10T04:15:18.541 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2022-09-10T04:15:18.545 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2022-09-10T04:15:18.569 INFO:teuthology.orchestra.run.smithi081.stdout:(90/132): openblas-0.3.15-3.el8.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2022-09-10T04:15:19.111 INFO:teuthology.orchestra.run.smithi081.stdout:(91/132): ceph-test-17.0.0-14898.g24441860.el8. 7.4 MB/s | 54 MB 00:07 2022-09-10T04:15:19.153 INFO:teuthology.orchestra.run.smithi081.stdout:(92/132): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2022-09-10T04:15:19.187 INFO:teuthology.orchestra.run.smithi081.stdout:(93/132): automake-1.16.1-7.el8.noarch.rpm 1.1 MB/s | 713 kB 00:00 2022-09-10T04:15:19.220 INFO:teuthology.orchestra.run.smithi081.stdout:(94/132): python3-jsonpointer-1.10-11.el8.noarc 628 kB/s | 20 kB 00:00 2022-09-10T04:15:19.246 INFO:teuthology.orchestra.run.smithi081.stdout:(95/132): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.3 MB/s | 126 kB 00:00 2022-09-10T04:15:19.288 INFO:teuthology.orchestra.run.smithi081.stdout:(96/132): libpq-13.5-1.el8.x86_64.rpm 2.9 MB/s | 198 kB 00:00 2022-09-10T04:15:19.321 INFO:teuthology.orchestra.run.smithi081.stdout:(97/132): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.4 MB/s | 259 kB 00:00 2022-09-10T04:15:19.355 INFO:teuthology.orchestra.run.smithi081.stdout:(98/132): oniguruma-6.8.2-2.el8.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2022-09-10T04:15:19.381 INFO:teuthology.orchestra.run.smithi081.stdout:(99/132): python3-jsonpatch-1.21-2.el8.noarch.r 463 kB/s | 27 kB 00:00 2022-09-10T04:15:19.389 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2022-09-10T04:15:19.393 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2022-09-10T04:15:19.422 INFO:teuthology.orchestra.run.smithi081.stdout:(100/132): uuid-1.6.2-43.el8.x86_64.rpm 961 kB/s | 64 kB 00:00 2022-09-10T04:15:19.448 INFO:teuthology.orchestra.run.smithi081.stdout:(101/132): python3-pyasn1-modules-0.3.7-6.el8.n 1.6 MB/s | 110 kB 00:00 2022-09-10T04:15:19.490 INFO:teuthology.orchestra.run.smithi081.stdout:(102/132): fuse-2.9.7-15.el8.x86_64.rpm 2.0 MB/s | 83 kB 00:00 2022-09-10T04:15:19.524 INFO:teuthology.orchestra.run.smithi081.stdout:(103/132): pcre2-devel-10.32-2.el8.x86_64.rpm 5.9 MB/s | 605 kB 00:00 2022-09-10T04:15:19.557 INFO:teuthology.orchestra.run.smithi081.stdout:(104/132): libsepol-devel-2.9-3.el8.x86_64.rpm 1.3 MB/s | 87 kB 00:00 2022-09-10T04:15:19.591 INFO:teuthology.orchestra.run.smithi081.stdout:(105/132): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.3 MB/s | 220 kB 00:00 2022-09-10T04:15:19.624 INFO:teuthology.orchestra.run.smithi081.stdout:(106/132): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2022-09-10T04:15:19.649 INFO:teuthology.orchestra.run.smithi081.stdout:(107/132): libverto-devel-0.3.0-5.el8.x86_64.rp 312 kB/s | 18 kB 00:00 2022-09-10T04:15:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:(108/132): librabbitmq-0.9.0-3.el8.x86_64.rpm 510 kB/s | 47 kB 00:00 2022-09-10T04:15:19.759 INFO:teuthology.orchestra.run.smithi081.stdout:(109/132): python3-cryptography-3.2.1-5.el8.x86 5.0 MB/s | 559 kB 00:00 2022-09-10T04:15:19.793 INFO:teuthology.orchestra.run.smithi081.stdout:(110/132): python3-oauthlib-2.1.0-1.el8.noarch. 2.0 MB/s | 155 kB 00:00 2022-09-10T04:15:19.826 INFO:teuthology.orchestra.run.smithi081.stdout:(111/132): python3-pycparser-2.14-14.el8.noarch 1.6 MB/s | 109 kB 00:00 2022-09-10T04:15:19.868 INFO:teuthology.orchestra.run.smithi081.stdout:(112/132): krb5-devel-1.18.2-14.el8.x86_64.rpm 7.3 MB/s | 560 kB 00:00 2022-09-10T04:15:19.893 INFO:teuthology.orchestra.run.smithi081.stdout:(113/132): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.5 MB/s | 171 kB 00:00 2022-09-10T04:15:19.927 INFO:teuthology.orchestra.run.smithi081.stdout:(114/132): python3-jwt-1.6.1-2.el8.noarch.rpm 727 kB/s | 43 kB 00:00 2022-09-10T04:15:20.069 INFO:teuthology.orchestra.run.smithi081.stdout:(115/132): python3-scipy-1.0.0-21.module+el8.5. 8.5 MB/s | 14 MB 00:01 2022-09-10T04:15:20.094 INFO:teuthology.orchestra.run.smithi081.stdout:(116/132): zlib-devel-1.2.11-18.el8_5.x86_64.rp 288 kB/s | 58 kB 00:00 2022-09-10T04:15:20.136 INFO:teuthology.orchestra.run.smithi081.stdout:(117/132): elfutils-libelf-devel-0.186-1.el8.x8 287 kB/s | 60 kB 00:00 2022-09-10T04:15:20.161 INFO:teuthology.orchestra.run.smithi081.stdout:(118/132): libaio-devel-0.3.112-1.el8.x86_64.rp 284 kB/s | 19 kB 00:00 2022-09-10T04:15:20.187 INFO:teuthology.orchestra.run.smithi081.stdout:(119/132): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.0 MB/s | 238 kB 00:00 2022-09-10T04:15:20.220 INFO:teuthology.orchestra.run.smithi081.stdout:(120/132): libuuid-devel-2.32.1-35.el8.x86_64.r 1.1 MB/s | 98 kB 00:00 2022-09-10T04:15:20.246 INFO:teuthology.orchestra.run.smithi081.stdout:(121/132): NetworkManager-initscripts-updown-1. 1.6 MB/s | 137 kB 00:00 2022-09-10T04:15:20.296 INFO:teuthology.orchestra.run.smithi081.stdout:(122/132): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.1 MB/s | 229 kB 00:00 2022-09-10T04:15:20.671 INFO:teuthology.orchestra.run.smithi081.stdout:(123/132): python3-requests-oauthlib-1.0.0-1.el 95 kB/s | 43 kB 00:00 2022-09-10T04:15:20.688 INFO:teuthology.orchestra.run.smithi081.stdout:(124/132): libcom_err-devel-1.45.6-4.el8.x86_64 99 kB/s | 39 kB 00:00 2022-09-10T04:15:20.714 INFO:teuthology.orchestra.run.smithi081.stdout:(125/132): xfsprogs-devel-5.0.0-10.el8.x86_64.r 155 kB/s | 72 kB 00:00 2022-09-10T04:15:20.747 INFO:teuthology.orchestra.run.smithi081.stdout:(126/132): keyutils-libs-devel-1.5.10-9.el8.x86 840 kB/s | 48 kB 00:00 2022-09-10T04:15:20.797 INFO:teuthology.orchestra.run.smithi081.stdout:(127/132): libgfortran-8.5.0-10.1.el8_6.x86_64. 7.6 MB/s | 644 kB 00:00 2022-09-10T04:15:20.881 INFO:teuthology.orchestra.run.smithi081.stdout:(128/132): openssl-devel-1.1.1k-7.el8_6.x86_64. 11 MB/s | 2.3 MB 00:00 2022-09-10T04:15:20.932 INFO:teuthology.orchestra.run.smithi081.stdout:(129/132): openssl-1.1.1k-7.el8_6.x86_64.rpm 14 MB/s | 709 kB 00:00 2022-09-10T04:15:20.998 INFO:teuthology.orchestra.run.smithi081.stdout:(130/132): openssl-libs-1.1.1k-7.el8_6.x86_64.r 22 MB/s | 1.5 MB 00:00 2022-09-10T04:15:21.324 INFO:teuthology.orchestra.run.smithi081.stdout:(131/132): librados2-17.0.0-14898.g24441860.el8 6.3 MB/s | 3.6 MB 00:00 2022-09-10T04:15:21.391 INFO:teuthology.orchestra.run.smithi081.stdout:(132/132): librbd1-17.0.0-14898.g24441860.el8.x 6.0 MB/s | 3.6 MB 00:00 2022-09-10T04:15:21.394 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:15:21.394 INFO:teuthology.orchestra.run.smithi081.stdout:Total 12 MB/s | 216 MB 00:17 2022-09-10T04:15:23.557 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2022-09-10T04:15:23.800 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-10T04:15:24.135 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-10T04:15:24.299 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/136 2022-09-10T04:15:24.387 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2022-09-10T04:15:24.465 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/136 2022-09-10T04:15:24.568 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2022-09-10T04:15:24.569 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2022-09-10T04:15:24.626 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/136 2022-09-10T04:15:24.697 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/136 2022-09-10T04:15:24.738 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2022-09-10T04:15:24.885 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/136 2022-09-10T04:15:25.003 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-10T04:15:25.109 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-10T04:15:25.220 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-10T04:15:25.432 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/136 2022-09-10T04:15:25.505 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-10T04:15:25.580 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/136 2022-09-10T04:15:25.701 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:15:25.774 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 4/136 2022-09-10T04:15:25.822 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2022-09-10T04:15:25.827 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2022-09-10T04:15:25.918 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 5/136 2022-09-10T04:15:26.056 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 5/136 2022-09-10T04:15:26.075 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:15:26.217 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:15:26.259 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-10T04:15:26.329 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:15:26.501 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:15:26.600 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:15:26.660 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-10T04:15:26.757 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:17.0.0-14898.g24441860.el8.x86_6 10/136 2022-09-10T04:15:26.862 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:15:26.979 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpq-13.5-1.el8.x86_64 11/136 2022-09-10T04:15:27.170 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/136 2022-09-10T04:15:27.254 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:15:27.434 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:15:27.503 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/136 2022-09-10T04:15:27.558 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:15:27.704 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/136 2022-09-10T04:15:27.776 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:15:27.850 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 15/136 2022-09-10T04:15:27.871 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:15:27.955 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 15/136 2022-09-10T04:15:28.032 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:17.0.0-14898.g24441860.el8.x86_6 10/136 2022-09-10T04:15:28.109 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 16/136 2022-09-10T04:15:28.177 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 16/136 2022-09-10T04:15:28.236 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpq-13.5-1.el8.x86_64 11/136 2022-09-10T04:15:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 17/136 2022-09-10T04:15:28.462 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/136 2022-09-10T04:15:28.500 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 18/136 2022-09-10T04:15:28.642 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 18/136 2022-09-10T04:15:28.794 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/136 2022-09-10T04:15:28.800 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 19/136 2022-09-10T04:15:29.196 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/136 2022-09-10T04:15:29.343 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 15/136 2022-09-10T04:15:29.420 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 19/136 2022-09-10T04:15:29.453 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 15/136 2022-09-10T04:15:29.625 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 16/136 2022-09-10T04:15:29.644 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 20/136 2022-09-10T04:15:29.709 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 16/136 2022-09-10T04:15:29.880 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 17/136 2022-09-10T04:15:30.189 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 18/136 2022-09-10T04:15:30.235 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 20/136 2022-09-10T04:15:30.420 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.15-3.el8.x86_64 21/136 2022-09-10T04:15:30.432 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/136 2022-09-10T04:15:30.573 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 20/136 2022-09-10T04:15:30.758 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 21/136 2022-09-10T04:15:30.899 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 21/136 2022-09-10T04:15:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2022-09-10T04:15:31.059 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 22/136 2022-09-10T04:15:31.125 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 21/136 2022-09-10T04:15:31.230 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-10T04:15:31.372 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/136 2022-09-10T04:15:31.520 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: openssl-libs-1:1.1.1k-7.el8_6.x86_64 1/136 2022-09-10T04:15:31.594 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 23/136 2022-09-10T04:15:31.680 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 22/136 2022-09-10T04:15:31.704 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/136 2022-09-10T04:15:31.891 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/136 2022-09-10T04:15:31.900 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 23/136 2022-09-10T04:15:31.925 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/136 2022-09-10T04:15:32.089 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/136 2022-09-10T04:15:32.180 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 25/136 2022-09-10T04:15:32.212 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/136 2022-09-10T04:15:32.398 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/136 2022-09-10T04:15:32.450 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 5/136 2022-09-10T04:15:32.501 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 23/136 2022-09-10T04:15:32.597 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:17.0.0-14898.g24441860.el8 27/136 2022-09-10T04:15:32.647 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-10T04:15:32.683 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.15-3.el8.x86_64 24/136 2022-09-10T04:15:32.743 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:17.0.0-14898.g24441860.e 28/136 2022-09-10T04:15:32.993 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_ 29/136 2022-09-10T04:15:33.143 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 6/136 2022-09-10T04:15:33.355 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:15:33.356 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 24/136 2022-09-10T04:15:33.681 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/136 2022-09-10T04:15:33.863 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:15:33.913 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/136 2022-09-10T04:15:33.916 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/136 2022-09-10T04:15:34.081 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:17.0.0-14898.g24441860.el8 27/136 2022-09-10T04:15:34.105 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:15:34.227 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:17.0.0-14898.g24441860.e 28/136 2022-09-10T04:15:34.292 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:15:34.292 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/136 2022-09-10T04:15:34.464 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:15:34.478 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_ 29/136 2022-09-10T04:15:34.505 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/136 2022-09-10T04:15:34.605 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:15:34.658 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/136 2022-09-10T04:15:34.835 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:17.0.0-14898.g24441860.el8.x86_6 10/136 2022-09-10T04:15:34.975 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/136 2022-09-10T04:15:35.076 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpq-13.5-1.el8.x86_64 11/136 2022-09-10T04:15:35.339 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/136 2022-09-10T04:15:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 13/136 2022-09-10T04:15:36.019 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 14/136 2022-09-10T04:15:36.180 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 15/136 2022-09-10T04:15:36.344 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 15/136 2022-09-10T04:15:36.378 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/136 2022-09-10T04:15:36.546 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 16/136 2022-09-10T04:15:36.645 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 16/136 2022-09-10T04:15:36.698 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/136 2022-09-10T04:15:36.767 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 31/136 2022-09-10T04:15:36.818 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 17/136 2022-09-10T04:15:36.988 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 18/136 2022-09-10T04:15:37.024 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 34/136 2022-09-10T04:15:37.178 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 18/136 2022-09-10T04:15:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/136 2022-09-10T04:15:37.338 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 19/136 2022-09-10T04:15:37.483 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/136 2022-09-10T04:15:37.692 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 32/136 2022-09-10T04:15:37.695 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 37/136 2022-09-10T04:15:37.893 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:17.0.0-14898.g24441860.el8.x86_ 38/136 2022-09-10T04:15:38.010 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/136 2022-09-10T04:15:38.076 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/136 2022-09-10T04:15:38.115 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 19/136 2022-09-10T04:15:38.235 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 40/136 2022-09-10T04:15:38.283 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/136 2022-09-10T04:15:38.304 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 20/136 2022-09-10T04:15:38.378 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 41/136 2022-09-10T04:15:38.527 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/136 2022-09-10T04:15:38.563 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/136 2022-09-10T04:15:38.726 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/136 2022-09-10T04:15:38.801 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 43/136 2022-09-10T04:15:38.921 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 37/136 2022-09-10T04:15:39.016 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 44/136 2022-09-10T04:15:39.078 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 20/136 2022-09-10T04:15:39.120 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:17.0.0-14898.g24441860.el8.x86_ 38/136 2022-09-10T04:15:39.295 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.15-3.el8.x86_64 21/136 2022-09-10T04:15:39.333 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/136 2022-09-10T04:15:39.429 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 45/136 2022-09-10T04:15:39.534 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/136 2022-09-10T04:15:39.600 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/136 2022-09-10T04:15:39.761 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 46/136 2022-09-10T04:15:39.804 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 41/136 2022-09-10T04:15:39.919 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 41/136 2022-09-10T04:15:39.925 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 47/136 2022-09-10T04:15:40.098 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.6-3.el8.x86_64 42/136 2022-09-10T04:15:40.190 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 21/136 2022-09-10T04:15:40.192 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 48/136 2022-09-10T04:15:40.250 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 43/136 2022-09-10T04:15:40.435 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 49/136 2022-09-10T04:15:40.485 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 44/136 2022-09-10T04:15:40.491 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 22/136 2022-09-10T04:15:40.696 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 23/136 2022-09-10T04:15:40.722 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : automake-1.16.1-7.el8.noarch 45/136 2022-09-10T04:15:40.762 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 50/136 2022-09-10T04:15:40.870 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 46/136 2022-09-10T04:15:40.974 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/136 2022-09-10T04:15:41.033 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 51/136 2022-09-10T04:15:41.036 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 47/136 2022-09-10T04:15:41.219 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 48/136 2022-09-10T04:15:41.232 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 52/136 2022-09-10T04:15:41.304 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 25/136 2022-09-10T04:15:41.415 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 49/136 2022-09-10T04:15:41.434 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 53/136 2022-09-10T04:15:41.575 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 54/136 2022-09-10T04:15:41.586 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 26/136 2022-09-10T04:15:41.663 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 50/136 2022-09-10T04:15:41.814 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:17.0.0-14898.g24441860.el8 27/136 2022-09-10T04:15:41.878 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/136 2022-09-10T04:15:41.929 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/136 2022-09-10T04:15:41.934 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:17.0.0-14898.g24441860.e 28/136 2022-09-10T04:15:42.101 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/136 2022-09-10T04:15:42.191 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_ 29/136 2022-09-10T04:15:42.279 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/136 2022-09-10T04:15:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 57/136 2022-09-10T04:15:42.575 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/136 2022-09-10T04:15:42.586 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/136 2022-09-10T04:15:42.845 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/136 2022-09-10T04:15:42.870 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 59/136 2022-09-10T04:15:43.056 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.7-15.el8.x86_64 60/136 2022-09-10T04:15:43.094 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/136 2022-09-10T04:15:43.272 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/136 2022-09-10T04:15:43.320 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 30/136 2022-09-10T04:15:43.335 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/136 2022-09-10T04:15:43.403 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/136 2022-09-10T04:15:43.505 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 62/136 2022-09-10T04:15:43.584 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 62/136 2022-09-10T04:15:43.705 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/136 2022-09-10T04:15:43.740 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/136 2022-09-10T04:15:43.779 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-3.el8.x86_64 63/136 2022-09-10T04:15:43.921 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 64/136 2022-09-10T04:15:43.981 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/136 2022-09-10T04:15:44.053 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/136 2022-09-10T04:15:44.142 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 65/136 2022-09-10T04:15:44.150 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/136 2022-09-10T04:15:44.263 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/136 2022-09-10T04:15:44.355 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : automake-1.16.1-7.el8.noarch 66/136 2022-09-10T04:15:44.376 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/136 2022-09-10T04:15:44.523 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 67/136 2022-09-10T04:15:44.539 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 61/136 2022-09-10T04:15:44.730 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/136 2022-09-10T04:15:44.926 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-10T04:15:45.190 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 62/136 2022-09-10T04:15:45.257 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/136 2022-09-10T04:15:45.393 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/136 2022-09-10T04:15:45.446 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/136 2022-09-10T04:15:45.601 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/136 2022-09-10T04:15:45.634 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 64/136 2022-09-10T04:15:45.783 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/136 2022-09-10T04:15:45.877 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/136 2022-09-10T04:15:45.972 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/136 2022-09-10T04:15:46.126 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/136 2022-09-10T04:15:46.168 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/136 2022-09-10T04:15:46.349 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.7-15.el8.x86_64 67/136 2022-09-10T04:15:46.358 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/136 2022-09-10T04:15:46.478 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 33/136 2022-09-10T04:15:46.514 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/136 2022-09-10T04:15:46.541 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/136 2022-09-10T04:15:46.710 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-10T04:15:46.720 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/136 2022-09-10T04:15:46.899 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 34/136 2022-09-10T04:15:46.925 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/136 2022-09-10T04:15:47.040 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/136 2022-09-10T04:15:47.117 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/136 2022-09-10T04:15:47.119 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.8-1.el8.noarch 35/136 2022-09-10T04:15:47.232 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/136 2022-09-10T04:15:47.319 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/136 2022-09-10T04:15:47.340 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 36/136 2022-09-10T04:15:47.411 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/136 2022-09-10T04:15:47.557 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/136 2022-09-10T04:15:47.559 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/136 2022-09-10T04:15:47.561 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 37/136 2022-09-10T04:15:47.748 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/136 2022-09-10T04:15:47.779 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/136 2022-09-10T04:15:47.794 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:17.0.0-14898.g24441860.el8.x86_ 38/136 2022-09-10T04:15:47.964 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/136 2022-09-10T04:15:47.996 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/136 2022-09-10T04:15:48.020 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 39/136 2022-09-10T04:15:48.150 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 40/136 2022-09-10T04:15:48.170 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/136 2022-09-10T04:15:48.184 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/136 2022-09-10T04:15:48.351 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 41/136 2022-09-10T04:15:48.368 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/136 2022-09-10T04:15:48.547 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/136 2022-09-10T04:15:48.579 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/136 2022-09-10T04:15:48.742 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 43/136 2022-09-10T04:15:48.750 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/136 2022-09-10T04:15:48.968 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/136 2022-09-10T04:15:48.985 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 44/136 2022-09-10T04:15:49.154 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/136 2022-09-10T04:15:49.279 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/136 2022-09-10T04:15:49.383 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/136 2022-09-10T04:15:49.470 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 45/136 2022-09-10T04:15:49.558 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/136 2022-09-10T04:15:49.604 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/136 2022-09-10T04:15:49.804 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/136 2022-09-10T04:15:49.829 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 46/136 2022-09-10T04:15:49.849 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/136 2022-09-10T04:15:50.037 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 47/136 2022-09-10T04:15:50.079 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/136 2022-09-10T04:15:50.086 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/136 2022-09-10T04:15:50.357 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/136 2022-09-10T04:15:50.365 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 48/136 2022-09-10T04:15:50.578 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/136 2022-09-10T04:15:50.607 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 49/136 2022-09-10T04:15:51.024 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 50/136 2022-09-10T04:15:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/136 2022-09-10T04:15:51.294 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/136 2022-09-10T04:15:51.428 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:17.0.0-14898.g24441860.el8.x86_64 93/136 2022-09-10T04:15:51.503 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:17.0.0-14898.g24441860.el8.x86_64 93/136 2022-09-10T04:15:51.576 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/136 2022-09-10T04:15:51.647 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 51/136 2022-09-10T04:15:51.663 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 94/136 2022-09-10T04:15:51.846 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/136 2022-09-10T04:15:51.850 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 52/136 2022-09-10T04:15:52.123 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 53/136 2022-09-10T04:15:52.172 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/136 2022-09-10T04:15:52.294 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 54/136 2022-09-10T04:15:52.423 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/136 2022-09-10T04:15:52.663 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/136 2022-09-10T04:15:52.669 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/136 2022-09-10T04:15:52.898 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/136 2022-09-10T04:15:53.201 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 57/136 2022-09-10T04:15:53.362 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/136 2022-09-10T04:15:53.480 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 58/136 2022-09-10T04:15:53.497 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:17.0.0-14898.g24441860.el8.x86_64 93/136 2022-09-10T04:15:53.566 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:17.0.0-14898.g24441860.el8.x86_64 93/136 2022-09-10T04:15:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 59/136 2022-09-10T04:15:53.772 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 94/136 2022-09-10T04:15:53.965 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.7-15.el8.x86_64 60/136 2022-09-10T04:15:54.177 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/136 2022-09-10T04:15:54.312 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/136 2022-09-10T04:15:54.494 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 62/136 2022-09-10T04:15:54.626 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 62/136 2022-09-10T04:15:54.794 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-3.el8.x86_64 63/136 2022-09-10T04:15:54.987 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 64/136 2022-09-10T04:15:55.263 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 65/136 2022-09-10T04:15:55.429 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:15:55.547 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : automake-1.16.1-7.el8.noarch 66/136 2022-09-10T04:15:55.764 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 67/136 2022-09-10T04:15:55.922 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 68/136 2022-09-10T04:15:55.978 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:15:56.147 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-10T04:15:56.463 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 70/136 2022-09-10T04:15:56.536 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:15:56.787 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 71/136 2022-09-10T04:15:56.828 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 96/136 2022-09-10T04:15:56.904 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 96/136 2022-09-10T04:15:56.950 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 72/136 2022-09-10T04:15:57.028 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 97/136 2022-09-10T04:15:57.117 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 73/136 2022-09-10T04:15:57.355 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 74/136 2022-09-10T04:15:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:15:57.588 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 75/136 2022-09-10T04:15:57.812 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 76/136 2022-09-10T04:15:57.822 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:15:58.015 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 77/136 2022-09-10T04:15:58.177 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 78/136 2022-09-10T04:15:58.374 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:15:58.432 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 79/136 2022-09-10T04:15:58.633 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 96/136 2022-09-10T04:15:58.634 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/136 2022-09-10T04:15:58.702 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 96/136 2022-09-10T04:15:58.810 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 81/136 2022-09-10T04:15:58.830 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 97/136 2022-09-10T04:15:59.126 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-2.6-1.el8.noarch 82/136 2022-09-10T04:15:59.379 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 83/136 2022-09-10T04:15:59.622 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 84/136 2022-09-10T04:15:59.881 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 85/136 2022-09-10T04:16:01.339 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 86/136 2022-09-10T04:16:01.633 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 87/136 2022-09-10T04:16:02.035 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 88/136 2022-09-10T04:16:02.375 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 89/136 2022-09-10T04:16:02.630 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 90/136 2022-09-10T04:16:02.843 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 91/136 2022-09-10T04:16:03.654 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 92/136 2022-09-10T04:16:03.829 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:17.0.0-14898.g24441860.el8.x86_64 93/136 2022-09-10T04:16:03.955 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:17.0.0-14898.g24441860.el8.x86_64 93/136 2022-09-10T04:16:04.186 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 94/136 2022-09-10T04:16:10.225 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:16:10.501 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:16:11.210 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 95/136 2022-09-10T04:16:11.475 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 96/136 2022-09-10T04:16:11.678 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 96/136 2022-09-10T04:16:11.800 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 97/136 2022-09-10T04:16:15.645 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 97/136 2022-09-10T04:16:15.646 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2022-09-10T04:16:15.646 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2022-09-10T04:16:15.646 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2022-09-10T04:16:15.646 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /run 2022-09-10T04:16:15.647 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2022-09-10T04:16:15.647 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2022-09-10T04:16:15.647 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2022-09-10T04:16:15.647 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2022-09-10T04:16:15.647 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /dev 2022-09-10T04:16:15.648 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:16:15.982 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 98/136 2022-09-10T04:16:17.467 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 98/136 2022-09-10T04:16:17.595 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 97/136 2022-09-10T04:16:17.596 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2022-09-10T04:16:17.596 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2022-09-10T04:16:17.596 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /dev 2022-09-10T04:16:17.597 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /run 2022-09-10T04:16:17.597 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2022-09-10T04:16:17.598 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2022-09-10T04:16:17.602 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2022-09-10T04:16:17.602 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2022-09-10T04:16:17.602 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /dev 2022-09-10T04:16:17.603 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:16:17.647 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 99/136 2022-09-10T04:16:17.864 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 99/136 2022-09-10T04:16:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 98/136 2022-09-10T04:16:18.016 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 100/136 2022-09-10T04:16:18.395 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 100/136 2022-09-10T04:16:18.548 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 101/136 2022-09-10T04:16:18.566 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 101/136 2022-09-10T04:16:19.476 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 98/136 2022-09-10T04:16:19.705 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 99/136 2022-09-10T04:16:19.921 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 99/136 2022-09-10T04:16:20.078 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 100/136 2022-09-10T04:16:20.113 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.0.0-14898.g24441860.el8.noarch 102/136 2022-09-10T04:16:20.299 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:17.0.0-14898.g24441860.el8.noarch 102/136 2022-09-10T04:16:20.428 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14898.g24441860. 103/136 2022-09-10T04:16:20.457 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 100/136 2022-09-10T04:16:20.595 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 101/136 2022-09-10T04:16:20.601 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14898.g24441860 104/136 2022-09-10T04:16:20.611 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 101/136 2022-09-10T04:16:20.789 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 105/136 2022-09-10T04:16:20.994 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 105/136 2022-09-10T04:16:21.315 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 106/136 2022-09-10T04:16:22.160 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.0.0-14898.g24441860.el8.noarch 102/136 2022-09-10T04:16:22.273 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 106/136 2022-09-10T04:16:22.340 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:17.0.0-14898.g24441860.el8.noarch 102/136 2022-09-10T04:16:22.430 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 107/136 2022-09-10T04:16:22.499 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14898.g24441860. 103/136 2022-09-10T04:16:22.520 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 107/136 2022-09-10T04:16:22.647 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14898.g24441860 104/136 2022-09-10T04:16:22.793 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 105/136 2022-09-10T04:16:22.888 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14898.g24441860.e 108/136 2022-09-10T04:16:22.999 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 105/136 2022-09-10T04:16:23.077 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 109/136 2022-09-10T04:16:23.113 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 109/136 2022-09-10T04:16:23.280 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 110/136 2022-09-10T04:16:23.288 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 106/136 2022-09-10T04:16:23.291 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 110/136 2022-09-10T04:16:24.246 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 106/136 2022-09-10T04:16:24.435 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 107/136 2022-09-10T04:16:24.524 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 107/136 2022-09-10T04:16:24.907 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14898.g24441860.e 108/136 2022-09-10T04:16:24.949 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:17.0.0-14898.g24441860.el8.x86_64 111/136 2022-09-10T04:16:25.088 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 109/136 2022-09-10T04:16:25.121 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 112/136 2022-09-10T04:16:25.123 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 109/136 2022-09-10T04:16:25.154 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 112/136 2022-09-10T04:16:25.260 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 110/136 2022-09-10T04:16:25.271 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 110/136 2022-09-10T04:16:25.330 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14898.g2444 113/136 2022-09-10T04:16:25.372 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14898.g2444 113/136 2022-09-10T04:16:25.530 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 114/136 2022-09-10T04:16:25.877 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 114/136 2022-09-10T04:16:26.022 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 115/136 2022-09-10T04:16:26.952 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:17.0.0-14898.g24441860.el8.x86_64 111/136 2022-09-10T04:16:27.166 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 112/136 2022-09-10T04:16:27.201 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 112/136 2022-09-10T04:16:27.327 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14898.g2444 113/136 2022-09-10T04:16:27.369 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14898.g2444 113/136 2022-09-10T04:16:27.502 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 114/136 2022-09-10T04:16:27.848 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 114/136 2022-09-10T04:16:27.993 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 115/136 2022-09-10T04:16:31.065 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 115/136 2022-09-10T04:16:31.372 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 116/136 2022-09-10T04:16:31.537 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libtool-2.4.6-25.el8.x86_64 117/136 2022-09-10T04:16:31.726 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 117/136 2022-09-10T04:16:32.017 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 118/136 2022-09-10T04:16:32.815 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 119/136 2022-09-10T04:16:33.034 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 115/136 2022-09-10T04:16:33.057 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 120/136 2022-09-10T04:16:33.249 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 121/136 2022-09-10T04:16:33.323 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 116/136 2022-09-10T04:16:33.390 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86 122/136 2022-09-10T04:16:33.498 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 97/136 2022-09-10T04:16:33.498 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2022-09-10T04:16:33.498 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2022-09-10T04:16:33.499 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /dev 2022-09-10T04:16:33.499 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /run 2022-09-10T04:16:33.499 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2022-09-10T04:16:33.499 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2022-09-10T04:16:33.500 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2022-09-10T04:16:33.500 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2022-09-10T04:16:33.500 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /dev 2022-09-10T04:16:33.500 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:16:33.529 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/136 2022-09-10T04:16:33.964 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 98/136 2022-09-10T04:16:34.173 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 117/136 2022-09-10T04:16:34.435 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/136 2022-09-10T04:16:34.710 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 119/136 2022-09-10T04:16:34.859 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libtool-2.4.6-25.el8.x86_64 120/136 2022-09-10T04:16:35.047 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 120/136 2022-09-10T04:16:35.288 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 121/136 2022-09-10T04:16:35.446 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86 122/136 2022-09-10T04:16:35.541 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:16:35.593 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/136 2022-09-10T04:16:35.696 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 98/136 2022-09-10T04:16:35.786 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:16:35.803 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:16:35.914 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 99/136 2022-09-10T04:16:35.936 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 125/136 2022-09-10T04:16:36.222 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 99/136 2022-09-10T04:16:36.235 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 126/136 2022-09-10T04:16:36.415 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 100/136 2022-09-10T04:16:36.530 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 127/136 2022-09-10T04:16:36.780 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/136 2022-09-10T04:16:36.911 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 100/136 2022-09-10T04:16:36.927 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-7.el8 129/136 2022-09-10T04:16:36.935 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-7.el8 129/136 2022-09-10T04:16:37.064 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 101/136 2022-09-10T04:16:37.112 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 130/136 2022-09-10T04:16:37.135 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 101/136 2022-09-10T04:16:37.377 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flex-2.6.1-9.el8.x86_64 131/136 2022-09-10T04:16:37.452 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 131/136 2022-09-10T04:16:37.622 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:16:37.669 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : bison-3.0.4-10.el8.x86_64 132/136 2022-09-10T04:16:37.683 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 132/136 2022-09-10T04:16:37.859 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:16:37.878 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:16:37.879 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 133/136 2022-09-10T04:16:37.880 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-10T04:16:38.055 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-10T04:16:38.055 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-10T04:16:38.092 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 125/136 2022-09-10T04:16:38.321 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-10T04:16:38.322 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:16:38.332 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 126/136 2022-09-10T04:16:38.591 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 127/136 2022-09-10T04:16:38.926 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/136 2022-09-10T04:16:39.104 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:16:39.105 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:17.0.0-14898.g24441860.el8.x86_64 1/136 2022-09-10T04:16:39.105 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 2/136 2022-09-10T04:16:39.105 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 3/136 2022-09-10T04:16:39.106 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 4/136 2022-09-10T04:16:39.106 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14898.g2444 5/136 2022-09-10T04:16:39.106 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 6/136 2022-09-10T04:16:39.106 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:16:39.107 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:16:39.107 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:16:39.107 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 10/136 2022-09-10T04:16:39.107 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 11/136 2022-09-10T04:16:39.107 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 12/136 2022-09-10T04:16:39.108 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 13/136 2022-09-10T04:16:39.108 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86 14/136 2022-09-10T04:16:39.108 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 15/136 2022-09-10T04:16:39.108 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 16/136 2022-09-10T04:16:39.109 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:17.0.0-14898.g24441860.el8.x86_ 17/136 2022-09-10T04:16:39.109 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 18/136 2022-09-10T04:16:39.109 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:17.0.0-14898.g24441860.el8.x86_64 19/136 2022-09-10T04:16:39.109 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14898.g24441860.e 20/136 2022-09-10T04:16:39.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:17.0.0-14898.g24441860.el8 21/136 2022-09-10T04:16:39.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_ 22/136 2022-09-10T04:16:39.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:17.0.0-14898.g24441860.el8.x86_6 23/136 2022-09-10T04:16:39.110 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 24/136 2022-09-10T04:16:39.111 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 25/136 2022-09-10T04:16:39.123 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 26/136 2022-09-10T04:16:39.124 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 27/136 2022-09-10T04:16:39.124 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 28/136 2022-09-10T04:16:39.125 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14898.g24441860 29/136 2022-09-10T04:16:39.126 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 30/136 2022-09-10T04:16:39.126 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 31/136 2022-09-10T04:16:39.126 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 32/136 2022-09-10T04:16:39.126 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14898.g24441860.e 33/136 2022-09-10T04:16:39.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 34/136 2022-09-10T04:16:39.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14898.g24441860. 35/136 2022-09-10T04:16:39.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 36/136 2022-09-10T04:16:39.127 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:17.0.0-14898.g24441860.el8.noarch 37/136 2022-09-10T04:16:39.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 38/136 2022-09-10T04:16:39.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/136 2022-09-10T04:16:39.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/136 2022-09-10T04:16:39.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/136 2022-09-10T04:16:39.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/136 2022-09-10T04:16:39.129 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/136 2022-09-10T04:16:39.130 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/136 2022-09-10T04:16:39.130 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/136 2022-09-10T04:16:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/136 2022-09-10T04:16:39.132 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/136 2022-09-10T04:16:39.132 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/136 2022-09-10T04:16:39.132 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/136 2022-09-10T04:16:39.133 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/136 2022-09-10T04:16:39.133 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/136 2022-09-10T04:16:39.134 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/136 2022-09-10T04:16:39.134 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/136 2022-09-10T04:16:39.135 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/136 2022-09-10T04:16:39.135 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/136 2022-09-10T04:16:39.135 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/136 2022-09-10T04:16:39.136 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/136 2022-09-10T04:16:39.136 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/136 2022-09-10T04:16:39.137 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/136 2022-09-10T04:16:39.137 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/136 2022-09-10T04:16:39.138 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/136 2022-09-10T04:16:39.138 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/136 2022-09-10T04:16:39.139 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/136 2022-09-10T04:16:39.139 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/136 2022-09-10T04:16:39.139 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/136 2022-09-10T04:16:39.140 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/136 2022-09-10T04:16:39.140 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/136 2022-09-10T04:16:39.141 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/136 2022-09-10T04:16:39.141 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-10T04:16:39.142 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/136 2022-09-10T04:16:39.142 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/136 2022-09-10T04:16:39.142 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 72/136 2022-09-10T04:16:39.143 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : bison-3.0.4-10.el8.x86_64 73/136 2022-09-10T04:16:39.143 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 74/136 2022-09-10T04:16:39.143 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/136 2022-09-10T04:16:39.143 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 76/136 2022-09-10T04:16:39.144 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/136 2022-09-10T04:16:39.144 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 78/136 2022-09-10T04:16:39.144 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 79/136 2022-09-10T04:16:39.145 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 80/136 2022-09-10T04:16:39.145 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 81/136 2022-09-10T04:16:39.145 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 82/136 2022-09-10T04:16:39.145 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 83/136 2022-09-10T04:16:39.146 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 84/136 2022-09-10T04:16:39.146 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 85/136 2022-09-10T04:16:39.146 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/136 2022-09-10T04:16:39.147 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/136 2022-09-10T04:16:39.147 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/136 2022-09-10T04:16:39.147 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flex-2.6.1-9.el8.x86_64 89/136 2022-09-10T04:16:39.147 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 90/136 2022-09-10T04:16:39.148 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-3.el8.x86_64 91/136 2022-09-10T04:16:39.148 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 92/136 2022-09-10T04:16:39.148 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : automake-1.16.1-7.el8.noarch 93/136 2022-09-10T04:16:39.148 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 94/136 2022-09-10T04:16:39.148 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 95/136 2022-09-10T04:16:39.149 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 96/136 2022-09-10T04:16:39.149 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpq-13.5-1.el8.x86_64 97/136 2022-09-10T04:16:39.149 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 98/136 2022-09-10T04:16:39.149 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 99/136 2022-09-10T04:16:39.150 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/136 2022-09-10T04:16:39.150 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 101/136 2022-09-10T04:16:39.150 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/136 2022-09-10T04:16:39.150 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 103/136 2022-09-10T04:16:39.151 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 104/136 2022-09-10T04:16:39.151 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/136 2022-09-10T04:16:39.151 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/136 2022-09-10T04:16:39.151 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 107/136 2022-09-10T04:16:39.151 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 108/136 2022-09-10T04:16:39.152 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 109/136 2022-09-10T04:16:39.152 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 110/136 2022-09-10T04:16:39.152 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 111/136 2022-09-10T04:16:39.153 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 112/136 2022-09-10T04:16:39.153 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 113/136 2022-09-10T04:16:39.153 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 114/136 2022-09-10T04:16:39.153 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 115/136 2022-09-10T04:16:39.154 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/136 2022-09-10T04:16:39.154 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 117/136 2022-09-10T04:16:39.154 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/136 2022-09-10T04:16:39.154 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 119/136 2022-09-10T04:16:39.155 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 120/136 2022-09-10T04:16:39.155 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-7.el8 121/136 2022-09-10T04:16:39.155 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 122/136 2022-09-10T04:16:39.156 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 123/136 2022-09-10T04:16:39.156 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 124/136 2022-09-10T04:16:39.156 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/136 2022-09-10T04:16:39.156 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 126/136 2022-09-10T04:16:39.157 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 127/136 2022-09-10T04:16:39.157 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 128/136 2022-09-10T04:16:39.157 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:17.0.0-14898.g24441860.el8.x86_64 129/136 2022-09-10T04:16:39.157 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/136 2022-09-10T04:16:39.158 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:17.0.0-14898.g24441860.el8.x86_64 131/136 2022-09-10T04:16:39.158 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/136 2022-09-10T04:16:39.158 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/136 2022-09-10T04:16:39.158 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/136 2022-09-10T04:16:39.159 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/136 2022-09-10T04:16:39.166 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flex-2.6.1-9.el8.x86_64 129/136 2022-09-10T04:16:39.241 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 129/136 2022-09-10T04:16:39.433 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : bison-3.0.4-10.el8.x86_64 130/136 2022-09-10T04:16:39.446 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 130/136 2022-09-10T04:16:39.616 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-7.el8 131/136 2022-09-10T04:16:39.625 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-7.el8 131/136 2022-09-10T04:16:39.804 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 132/136 2022-09-10T04:16:39.968 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 133/136 2022-09-10T04:16:39.969 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-10T04:16:40.153 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-10T04:16:40.154 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-10T04:16:40.198 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:17.0.0-14898.g24441860.el8.noarch 102/136 2022-09-10T04:16:40.366 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:17.0.0-14898.g24441860.el8.noarch 102/136 2022-09-10T04:16:40.420 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-10T04:16:40.420 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:16:40.497 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-14898.g24441860. 103/136 2022-09-10T04:16:40.669 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-14898.g24441860 104/136 2022-09-10T04:16:40.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 105/136 2022-09-10T04:16:41.049 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:16:41.050 INFO:teuthology.orchestra.run.smithi104.stdout:Installed products updated. 2022-09-10T04:16:41.190 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 105/136 2022-09-10T04:16:41.227 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:16:41.227 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:17.0.0-14898.g24441860.el8.x86_64 1/136 2022-09-10T04:16:41.228 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 2/136 2022-09-10T04:16:41.228 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 3/136 2022-09-10T04:16:41.229 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 4/136 2022-09-10T04:16:41.229 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14898.g2444 5/136 2022-09-10T04:16:41.230 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 6/136 2022-09-10T04:16:41.231 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:16:41.231 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:16:41.231 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:16:41.233 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 10/136 2022-09-10T04:16:41.233 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 11/136 2022-09-10T04:16:41.234 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 12/136 2022-09-10T04:16:41.234 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 13/136 2022-09-10T04:16:41.235 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86 14/136 2022-09-10T04:16:41.235 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 15/136 2022-09-10T04:16:41.236 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 16/136 2022-09-10T04:16:41.236 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:17.0.0-14898.g24441860.el8.x86_ 17/136 2022-09-10T04:16:41.237 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 18/136 2022-09-10T04:16:41.237 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:17.0.0-14898.g24441860.el8.x86_64 19/136 2022-09-10T04:16:41.238 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14898.g24441860.e 20/136 2022-09-10T04:16:41.238 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:17.0.0-14898.g24441860.el8 21/136 2022-09-10T04:16:41.239 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_ 22/136 2022-09-10T04:16:41.239 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:17.0.0-14898.g24441860.el8.x86_6 23/136 2022-09-10T04:16:41.240 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 24/136 2022-09-10T04:16:41.240 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 25/136 2022-09-10T04:16:41.241 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 26/136 2022-09-10T04:16:41.241 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 27/136 2022-09-10T04:16:41.242 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 28/136 2022-09-10T04:16:41.242 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14898.g24441860 29/136 2022-09-10T04:16:41.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 30/136 2022-09-10T04:16:41.243 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 31/136 2022-09-10T04:16:41.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 32/136 2022-09-10T04:16:41.244 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14898.g24441860.e 33/136 2022-09-10T04:16:41.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 34/136 2022-09-10T04:16:41.245 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14898.g24441860. 35/136 2022-09-10T04:16:41.246 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 36/136 2022-09-10T04:16:41.247 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:17.0.0-14898.g24441860.el8.noarch 37/136 2022-09-10T04:16:41.247 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 38/136 2022-09-10T04:16:41.247 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/136 2022-09-10T04:16:41.248 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/136 2022-09-10T04:16:41.249 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/136 2022-09-10T04:16:41.249 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/136 2022-09-10T04:16:41.250 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/136 2022-09-10T04:16:41.251 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/136 2022-09-10T04:16:41.251 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/136 2022-09-10T04:16:41.251 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/136 2022-09-10T04:16:41.252 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/136 2022-09-10T04:16:41.252 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/136 2022-09-10T04:16:41.253 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/136 2022-09-10T04:16:41.253 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/136 2022-09-10T04:16:41.254 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/136 2022-09-10T04:16:41.254 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/136 2022-09-10T04:16:41.255 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/136 2022-09-10T04:16:41.255 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/136 2022-09-10T04:16:41.255 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/136 2022-09-10T04:16:41.256 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/136 2022-09-10T04:16:41.256 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/136 2022-09-10T04:16:41.257 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/136 2022-09-10T04:16:41.257 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/136 2022-09-10T04:16:41.258 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/136 2022-09-10T04:16:41.258 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/136 2022-09-10T04:16:41.259 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/136 2022-09-10T04:16:41.259 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/136 2022-09-10T04:16:41.259 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/136 2022-09-10T04:16:41.260 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/136 2022-09-10T04:16:41.260 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/136 2022-09-10T04:16:41.260 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/136 2022-09-10T04:16:41.260 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/136 2022-09-10T04:16:41.261 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-10T04:16:41.261 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/136 2022-09-10T04:16:41.261 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/136 2022-09-10T04:16:41.261 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 72/136 2022-09-10T04:16:41.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 73/136 2022-09-10T04:16:41.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 74/136 2022-09-10T04:16:41.262 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/136 2022-09-10T04:16:41.263 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 76/136 2022-09-10T04:16:41.263 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 77/136 2022-09-10T04:16:41.263 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 78/136 2022-09-10T04:16:41.264 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 79/136 2022-09-10T04:16:41.264 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 80/136 2022-09-10T04:16:41.265 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 81/136 2022-09-10T04:16:41.265 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 82/136 2022-09-10T04:16:41.265 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 83/136 2022-09-10T04:16:41.265 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 84/136 2022-09-10T04:16:41.266 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/136 2022-09-10T04:16:41.266 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 86/136 2022-09-10T04:16:41.266 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/136 2022-09-10T04:16:41.266 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 88/136 2022-09-10T04:16:41.267 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 89/136 2022-09-10T04:16:41.267 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-7.el8 90/136 2022-09-10T04:16:41.267 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 91/136 2022-09-10T04:16:41.268 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 92/136 2022-09-10T04:16:41.268 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 93/136 2022-09-10T04:16:41.269 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/136 2022-09-10T04:16:41.269 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 95/136 2022-09-10T04:16:41.269 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 96/136 2022-09-10T04:16:41.269 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 97/136 2022-09-10T04:16:41.270 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 98/136 2022-09-10T04:16:41.270 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : bison-3.0.4-10.el8.x86_64 99/136 2022-09-10T04:16:41.270 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 100/136 2022-09-10T04:16:41.270 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 101/136 2022-09-10T04:16:41.271 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 102/136 2022-09-10T04:16:41.271 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 103/136 2022-09-10T04:16:41.271 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 104/136 2022-09-10T04:16:41.271 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 105/136 2022-09-10T04:16:41.272 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 106/136 2022-09-10T04:16:41.272 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 107/136 2022-09-10T04:16:41.272 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/136 2022-09-10T04:16:41.272 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 109/136 2022-09-10T04:16:41.272 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 110/136 2022-09-10T04:16:41.273 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 111/136 2022-09-10T04:16:41.273 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/136 2022-09-10T04:16:41.273 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 113/136 2022-09-10T04:16:41.273 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 114/136 2022-09-10T04:16:41.274 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flex-2.6.1-9.el8.x86_64 115/136 2022-09-10T04:16:41.274 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 116/136 2022-09-10T04:16:41.274 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.6-3.el8.x86_64 117/136 2022-09-10T04:16:41.274 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 118/136 2022-09-10T04:16:41.275 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : automake-1.16.1-7.el8.noarch 119/136 2022-09-10T04:16:41.275 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 120/136 2022-09-10T04:16:41.275 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 121/136 2022-09-10T04:16:41.275 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 122/136 2022-09-10T04:16:41.275 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpq-13.5-1.el8.x86_64 123/136 2022-09-10T04:16:41.276 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 124/136 2022-09-10T04:16:41.276 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 125/136 2022-09-10T04:16:41.276 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 126/136 2022-09-10T04:16:41.276 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 127/136 2022-09-10T04:16:41.277 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 128/136 2022-09-10T04:16:41.277 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:17.0.0-14898.g24441860.el8.x86_64 129/136 2022-09-10T04:16:41.277 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/136 2022-09-10T04:16:41.277 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:17.0.0-14898.g24441860.el8.x86_64 131/136 2022-09-10T04:16:41.277 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/136 2022-09-10T04:16:41.278 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/136 2022-09-10T04:16:41.278 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/136 2022-09-10T04:16:41.278 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/136 2022-09-10T04:16:41.288 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:16:41.289 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2022-09-10T04:16:41.289 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.289 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.290 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:16:41.290 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:16:41.290 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2022-09-10T04:16:41.290 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-initscripts-updown-1:1.36.0-7.el8_6.noarch 2022-09-10T04:16:41.291 INFO:teuthology.orchestra.run.smithi104.stdout: automake-1.16.1-7.el8.noarch 2022-09-10T04:16:41.291 INFO:teuthology.orchestra.run.smithi104.stdout: bison-3.0.4-10.el8.x86_64 2022-09-10T04:16:41.291 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.291 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.291 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.292 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.292 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.292 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.292 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.294 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.295 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.295 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.296 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.296 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.297 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.297 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.297 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.297 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.301 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.301 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.302 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.302 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.303 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.303 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.304 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:41.304 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-09-10T04:16:41.304 INFO:teuthology.orchestra.run.smithi104.stdout: flex-2.6.1-9.el8.x86_64 2022-09-10T04:16:41.305 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-2.9.7-15.el8.x86_64 2022-09-10T04:16:41.305 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-10T04:16:41.306 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-3.el8.x86_64 2022-09-10T04:16:41.307 INFO:teuthology.orchestra.run.smithi104.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-09-10T04:16:41.307 INFO:teuthology.orchestra.run.smithi104.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-09-10T04:16:41.308 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-09-10T04:16:41.308 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.309 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.309 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.309 INFO:teuthology.orchestra.run.smithi104.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-09-10T04:16:41.310 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-09-10T04:16:41.311 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-10T04:16:41.311 INFO:teuthology.orchestra.run.smithi104.stdout: libpq-13.5-1.el8.x86_64 2022-09-10T04:16:41.311 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-09-10T04:16:41.312 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-10T04:16:41.312 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.312 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.313 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-10T04:16:41.313 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.314 INFO:teuthology.orchestra.run.smithi104.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-09-10T04:16:41.314 INFO:teuthology.orchestra.run.smithi104.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-09-10T04:16:41.314 INFO:teuthology.orchestra.run.smithi104.stdout: libtool-2.4.6-25.el8.x86_64 2022-09-10T04:16:41.315 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-10T04:16:41.315 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-09-10T04:16:41.316 INFO:teuthology.orchestra.run.smithi104.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-09-10T04:16:41.316 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-10T04:16:41.316 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-10T04:16:41.316 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.15-3.el8.x86_64 2022-09-10T04:16:41.317 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-09-10T04:16:41.317 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:16:41.317 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-09-10T04:16:41.317 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-09-10T04:16:41.318 INFO:teuthology.orchestra.run.smithi104.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-09-10T04:16:41.319 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:16:41.319 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:16:41.320 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:16:41.320 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-10T04:16:41.320 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-10T04:16:41.321 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-10T04:16:41.321 INFO:teuthology.orchestra.run.smithi104.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-10T04:16:41.321 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-10T04:16:41.321 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-10T04:16:41.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-10T04:16:41.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-10T04:16:41.324 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-10T04:16:41.324 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-10T04:16:41.324 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-10T04:16:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-10T04:16:41.325 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-10T04:16:41.326 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-10T04:16:41.326 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-10T04:16:41.326 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-10T04:16:41.327 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-10T04:16:41.327 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-10T04:16:41.327 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-10T04:16:41.327 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-10T04:16:41.328 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-10T04:16:41.328 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-10T04:16:41.328 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-10T04:16:41.329 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-10T04:16:41.329 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-10T04:16:41.329 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-10T04:16:41.330 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-10T04:16:41.330 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-2.6-1.el8.noarch 2022-09-10T04:16:41.330 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-10T04:16:41.330 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-10T04:16:41.331 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-10T04:16:41.331 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-10T04:16:41.331 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-10T04:16:41.332 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-10T04:16:41.332 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.332 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.332 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-10T04:16:41.333 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-10T04:16:41.333 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.333 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-10T04:16:41.333 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-10T04:16:41.333 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-09-10T04:16:41.334 INFO:teuthology.orchestra.run.smithi104.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-10T04:16:41.334 INFO:teuthology.orchestra.run.smithi104.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-10T04:16:41.334 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-10T04:16:41.334 INFO:teuthology.orchestra.run.smithi104.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-10T04:16:41.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-10T04:16:41.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-10T04:16:41.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-10T04:16:41.335 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-10T04:16:41.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-10T04:16:41.336 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-10T04:16:41.337 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.337 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.337 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:41.337 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-10T04:16:41.338 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-10T04:16:41.338 INFO:teuthology.orchestra.run.smithi104.stdout: uuid-1.6.2-43.el8.x86_64 2022-09-10T04:16:41.338 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-09-10T04:16:41.338 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-10T04:16:41.340 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-09-10T04:16:41.340 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-10T04:16:41.340 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2022-09-10T04:16:41.433 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 106/136 2022-09-10T04:16:41.478 DEBUG:teuthology.parallel:result is None 2022-09-10T04:16:42.654 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 106/136 2022-09-10T04:16:42.838 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 107/136 2022-09-10T04:16:43.002 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 107/136 2022-09-10T04:16:43.173 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:16:43.174 INFO:teuthology.orchestra.run.smithi110.stdout:Installed products updated. 2022-09-10T04:16:43.406 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:16:43.407 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2022-09-10T04:16:43.407 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.407 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.407 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:16:43.407 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:16:43.408 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2022-09-10T04:16:43.408 INFO:teuthology.orchestra.run.smithi110.stdout: NetworkManager-initscripts-updown-1:1.36.0-7.el8_6.noarch 2022-09-10T04:16:43.408 INFO:teuthology.orchestra.run.smithi110.stdout: automake-1.16.1-7.el8.noarch 2022-09-10T04:16:43.408 INFO:teuthology.orchestra.run.smithi110.stdout: bison-3.0.4-10.el8.x86_64 2022-09-10T04:16:43.409 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.409 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.409 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.409 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.410 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.410 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.410 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.411 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.412 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.412 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.412 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.412 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.413 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.413 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.413 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.414 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.414 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.414 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.414 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.414 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:16:43.415 INFO:teuthology.orchestra.run.smithi110.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-09-10T04:16:43.415 INFO:teuthology.orchestra.run.smithi110.stdout: flex-2.6.1-9.el8.x86_64 2022-09-10T04:16:43.415 INFO:teuthology.orchestra.run.smithi110.stdout: fuse-2.9.7-15.el8.x86_64 2022-09-10T04:16:43.415 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-10T04:16:43.416 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.6-3.el8.x86_64 2022-09-10T04:16:43.416 INFO:teuthology.orchestra.run.smithi110.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-09-10T04:16:43.416 INFO:teuthology.orchestra.run.smithi110.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-09-10T04:16:43.416 INFO:teuthology.orchestra.run.smithi110.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-09-10T04:16:43.416 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.417 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.417 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.417 INFO:teuthology.orchestra.run.smithi110.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-09-10T04:16:43.417 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-09-10T04:16:43.417 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-10T04:16:43.418 INFO:teuthology.orchestra.run.smithi110.stdout: libpq-13.5-1.el8.x86_64 2022-09-10T04:16:43.418 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-09-10T04:16:43.418 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-10T04:16:43.418 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.419 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.419 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-10T04:16:43.419 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.419 INFO:teuthology.orchestra.run.smithi110.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-09-10T04:16:43.419 INFO:teuthology.orchestra.run.smithi110.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-09-10T04:16:43.420 INFO:teuthology.orchestra.run.smithi110.stdout: libtool-2.4.6-25.el8.x86_64 2022-09-10T04:16:43.420 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-10T04:16:43.420 INFO:teuthology.orchestra.run.smithi110.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-09-10T04:16:43.420 INFO:teuthology.orchestra.run.smithi110.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-09-10T04:16:43.421 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-10T04:16:43.421 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-10T04:16:43.421 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.15-3.el8.x86_64 2022-09-10T04:16:43.422 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-09-10T04:16:43.422 INFO:teuthology.orchestra.run.smithi110.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:16:43.422 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-09-10T04:16:43.422 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-09-10T04:16:43.422 INFO:teuthology.orchestra.run.smithi110.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-09-10T04:16:43.423 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:16:43.423 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:16:43.423 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:16:43.423 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-10T04:16:43.424 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-10T04:16:43.424 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-10T04:16:43.424 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-10T04:16:43.424 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-10T04:16:43.424 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.425 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.425 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.425 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-10T04:16:43.425 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-10T04:16:43.425 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-10T04:16:43.426 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-10T04:16:43.426 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-10T04:16:43.426 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-10T04:16:43.426 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-10T04:16:43.427 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-10T04:16:43.427 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-10T04:16:43.427 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-10T04:16:43.427 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-10T04:16:43.428 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-10T04:16:43.428 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-10T04:16:43.428 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-10T04:16:43.428 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-10T04:16:43.429 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-10T04:16:43.429 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-10T04:16:43.429 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-10T04:16:43.430 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-10T04:16:43.430 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-10T04:16:43.430 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-10T04:16:43.430 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-10T04:16:43.431 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-2.6-1.el8.noarch 2022-09-10T04:16:43.431 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-10T04:16:43.431 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-10T04:16:43.431 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-10T04:16:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-10T04:16:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-10T04:16:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-10T04:16:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.433 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-10T04:16:43.433 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-10T04:16:43.433 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.433 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-10T04:16:43.433 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-10T04:16:43.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-09-10T04:16:43.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-10T04:16:43.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-10T04:16:43.434 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-10T04:16:43.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-10T04:16:43.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-10T04:16:43.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-10T04:16:43.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-10T04:16:43.435 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-10T04:16:43.436 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-10T04:16:43.436 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-10T04:16:43.436 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.436 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.436 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:16:43.437 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-10T04:16:43.437 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-10T04:16:43.437 INFO:teuthology.orchestra.run.smithi110.stdout: uuid-1.6.2-43.el8.x86_64 2022-09-10T04:16:43.438 INFO:teuthology.orchestra.run.smithi110.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-09-10T04:16:43.438 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-10T04:16:43.439 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-14898.g24441860.e 108/136 2022-09-10T04:16:43.439 INFO:teuthology.orchestra.run.smithi110.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-09-10T04:16:43.440 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:16:43.440 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2022-09-10T04:16:43.615 DEBUG:teuthology.parallel:result is None 2022-09-10T04:16:43.628 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 109/136 2022-09-10T04:16:43.726 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 109/136 2022-09-10T04:16:43.931 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 110/136 2022-09-10T04:16:43.983 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 110/136 2022-09-10T04:16:45.849 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:17.0.0-14898.g24441860.el8.x86_64 111/136 2022-09-10T04:16:46.053 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 112/136 2022-09-10T04:16:46.148 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 112/136 2022-09-10T04:16:46.288 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-14898.g2444 113/136 2022-09-10T04:16:46.392 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-14898.g2444 113/136 2022-09-10T04:16:46.538 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 114/136 2022-09-10T04:16:46.996 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 114/136 2022-09-10T04:16:47.146 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 115/136 2022-09-10T04:16:53.021 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 115/136 2022-09-10T04:16:53.378 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 116/136 2022-09-10T04:16:53.554 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libtool-2.4.6-25.el8.x86_64 117/136 2022-09-10T04:16:53.850 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 117/136 2022-09-10T04:16:54.195 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 118/136 2022-09-10T04:16:55.582 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 119/136 2022-09-10T04:16:55.919 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 120/136 2022-09-10T04:16:56.083 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 121/136 2022-09-10T04:16:56.275 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86 122/136 2022-09-10T04:16:56.438 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 123/136 2022-09-10T04:17:00.192 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:17:00.603 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:17:00.671 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 124/136 2022-09-10T04:17:00.854 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 125/136 2022-09-10T04:17:01.102 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 126/136 2022-09-10T04:17:01.391 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 127/136 2022-09-10T04:17:01.696 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-1:1.1.1k-7.el8_6.x86_64 128/136 2022-09-10T04:17:01.870 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-7.el8 129/136 2022-09-10T04:17:01.917 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-7.el8 129/136 2022-09-10T04:17:02.088 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 130/136 2022-09-10T04:17:02.360 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flex-2.6.1-9.el8.x86_64 131/136 2022-09-10T04:17:02.491 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 131/136 2022-09-10T04:17:02.718 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : bison-3.0.4-10.el8.x86_64 132/136 2022-09-10T04:17:02.773 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 132/136 2022-09-10T04:17:02.997 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 133/136 2022-09-10T04:17:02.998 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-10T04:17:03.260 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 134/136 2022-09-10T04:17:03.261 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-10T04:17:03.598 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 135/136 2022-09-10T04:17:03.599 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:17:04.327 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:17:04.328 INFO:teuthology.orchestra.run.smithi081.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2022-09-10T04:17:04.328 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:17:05.153 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:17.0.0-14898.g24441860.el8.x86_64 1/136 2022-09-10T04:17:05.153 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 2/136 2022-09-10T04:17:05.153 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 3/136 2022-09-10T04:17:05.153 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 4/136 2022-09-10T04:17:05.154 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-14898.g2444 5/136 2022-09-10T04:17:05.154 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 6/136 2022-09-10T04:17:05.154 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 7/136 2022-09-10T04:17:05.154 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 8/136 2022-09-10T04:17:05.155 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 9/136 2022-09-10T04:17:05.155 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 10/136 2022-09-10T04:17:05.155 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 11/136 2022-09-10T04:17:05.155 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 12/136 2022-09-10T04:17:05.155 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_6 13/136 2022-09-10T04:17:05.156 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86 14/136 2022-09-10T04:17:05.156 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 15/136 2022-09-10T04:17:05.156 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_6 16/136 2022-09-10T04:17:05.156 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:17.0.0-14898.g24441860.el8.x86_ 17/136 2022-09-10T04:17:05.157 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:17.0.0-14898.g24441860.el8.x8 18/136 2022-09-10T04:17:05.157 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:17.0.0-14898.g24441860.el8.x86_64 19/136 2022-09-10T04:17:05.157 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:17.0.0-14898.g24441860.e 20/136 2022-09-10T04:17:05.157 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:17.0.0-14898.g24441860.el8 21/136 2022-09-10T04:17:05.157 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_ 22/136 2022-09-10T04:17:05.158 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:17.0.0-14898.g24441860.el8.x86_6 23/136 2022-09-10T04:17:05.158 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 24/136 2022-09-10T04:17:05.158 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 25/136 2022-09-10T04:17:05.158 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 26/136 2022-09-10T04:17:05.159 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 27/136 2022-09-10T04:17:05.170 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 28/136 2022-09-10T04:17:05.170 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-14898.g24441860 29/136 2022-09-10T04:17:05.171 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.no 30/136 2022-09-10T04:17:05.171 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8. 31/136 2022-09-10T04:17:05.171 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-14898.g24 32/136 2022-09-10T04:17:05.171 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-14898.g24441860.e 33/136 2022-09-10T04:17:05.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarc 34/136 2022-09-10T04:17:05.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-14898.g24441860. 35/136 2022-09-10T04:17:05.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 36/136 2022-09-10T04:17:05.172 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:17.0.0-14898.g24441860.el8.noarch 37/136 2022-09-10T04:17:05.174 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 38/136 2022-09-10T04:17:05.174 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/136 2022-09-10T04:17:05.174 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/136 2022-09-10T04:17:05.175 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/136 2022-09-10T04:17:05.175 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/136 2022-09-10T04:17:05.175 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/136 2022-09-10T04:17:05.175 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/136 2022-09-10T04:17:05.176 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/136 2022-09-10T04:17:05.176 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/136 2022-09-10T04:17:05.176 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/136 2022-09-10T04:17:05.176 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/136 2022-09-10T04:17:05.177 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/136 2022-09-10T04:17:05.177 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/136 2022-09-10T04:17:05.177 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/136 2022-09-10T04:17:05.178 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/136 2022-09-10T04:17:05.178 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/136 2022-09-10T04:17:05.178 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/136 2022-09-10T04:17:05.178 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/136 2022-09-10T04:17:05.179 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/136 2022-09-10T04:17:05.179 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/136 2022-09-10T04:17:05.179 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/136 2022-09-10T04:17:05.179 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/136 2022-09-10T04:17:05.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/136 2022-09-10T04:17:05.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/136 2022-09-10T04:17:05.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/136 2022-09-10T04:17:05.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/136 2022-09-10T04:17:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/136 2022-09-10T04:17:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/136 2022-09-10T04:17:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/136 2022-09-10T04:17:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/136 2022-09-10T04:17:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/136 2022-09-10T04:17:05.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/136 2022-09-10T04:17:05.182 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/136 2022-09-10T04:17:05.183 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/136 2022-09-10T04:17:05.183 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 72/136 2022-09-10T04:17:05.183 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : bison-3.0.4-10.el8.x86_64 73/136 2022-09-10T04:17:05.183 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 74/136 2022-09-10T04:17:05.183 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 75/136 2022-09-10T04:17:05.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 76/136 2022-09-10T04:17:05.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/136 2022-09-10T04:17:05.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 78/136 2022-09-10T04:17:05.184 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 79/136 2022-09-10T04:17:05.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 80/136 2022-09-10T04:17:05.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 81/136 2022-09-10T04:17:05.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 82/136 2022-09-10T04:17:05.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 83/136 2022-09-10T04:17:05.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 84/136 2022-09-10T04:17:05.188 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 85/136 2022-09-10T04:17:05.188 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 86/136 2022-09-10T04:17:05.189 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 87/136 2022-09-10T04:17:05.189 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/136 2022-09-10T04:17:05.189 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flex-2.6.1-9.el8.x86_64 89/136 2022-09-10T04:17:05.190 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 90/136 2022-09-10T04:17:05.190 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-3.el8.x86_64 91/136 2022-09-10T04:17:05.190 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 92/136 2022-09-10T04:17:05.191 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : automake-1.16.1-7.el8.noarch 93/136 2022-09-10T04:17:05.191 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 94/136 2022-09-10T04:17:05.191 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 95/136 2022-09-10T04:17:05.191 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 96/136 2022-09-10T04:17:05.192 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpq-13.5-1.el8.x86_64 97/136 2022-09-10T04:17:05.192 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 98/136 2022-09-10T04:17:05.192 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 99/136 2022-09-10T04:17:05.192 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 100/136 2022-09-10T04:17:05.193 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 101/136 2022-09-10T04:17:05.193 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/136 2022-09-10T04:17:05.193 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 103/136 2022-09-10T04:17:05.194 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 104/136 2022-09-10T04:17:05.194 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 105/136 2022-09-10T04:17:05.194 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/136 2022-09-10T04:17:05.195 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 107/136 2022-09-10T04:17:05.195 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 108/136 2022-09-10T04:17:05.195 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 109/136 2022-09-10T04:17:05.195 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 110/136 2022-09-10T04:17:05.196 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 111/136 2022-09-10T04:17:05.196 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 112/136 2022-09-10T04:17:05.196 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 113/136 2022-09-10T04:17:05.196 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 114/136 2022-09-10T04:17:05.197 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 115/136 2022-09-10T04:17:05.197 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 116/136 2022-09-10T04:17:05.198 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 117/136 2022-09-10T04:17:05.198 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/136 2022-09-10T04:17:05.199 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 119/136 2022-09-10T04:17:05.199 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 120/136 2022-09-10T04:17:05.199 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-7.el8 121/136 2022-09-10T04:17:05.200 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 122/136 2022-09-10T04:17:05.200 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 123/136 2022-09-10T04:17:05.201 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 124/136 2022-09-10T04:17:05.201 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/136 2022-09-10T04:17:05.202 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 126/136 2022-09-10T04:17:05.202 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 127/136 2022-09-10T04:17:05.202 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 128/136 2022-09-10T04:17:05.203 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:17.0.0-14898.g24441860.el8.x86_64 129/136 2022-09-10T04:17:05.203 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/136 2022-09-10T04:17:05.204 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:17.0.0-14898.g24441860.el8.x86_64 131/136 2022-09-10T04:17:05.204 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/136 2022-09-10T04:17:05.204 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:1.1.1k-7.el8_6.x86_64 133/136 2022-09-10T04:17:05.205 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 134/136 2022-09-10T04:17:05.205 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:1.1.1k-7.el8_6.x86_64 135/136 2022-09-10T04:17:08.128 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 136/136 2022-09-10T04:17:08.129 INFO:teuthology.orchestra.run.smithi081.stdout:Installed products updated. 2022-09-10T04:17:08.362 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:17:08.362 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2022-09-10T04:17:08.362 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.363 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.363 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:17:08.363 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:17:08.363 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2022-09-10T04:17:08.364 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-initscripts-updown-1:1.36.0-7.el8_6.noarch 2022-09-10T04:17:08.364 INFO:teuthology.orchestra.run.smithi081.stdout: automake-1.16.1-7.el8.noarch 2022-09-10T04:17:08.364 INFO:teuthology.orchestra.run.smithi081.stdout: bison-3.0.4-10.el8.x86_64 2022-09-10T04:17:08.364 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.365 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.365 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.365 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.365 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.366 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.366 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.366 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.367 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.367 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.367 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.367 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.368 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.368 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.369 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.369 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.369 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.370 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.370 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.370 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.370 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.371 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.371 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top-2:17.0.0-14898.g24441860.el8.noarch 2022-09-10T04:17:08.371 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2022-09-10T04:17:08.371 INFO:teuthology.orchestra.run.smithi081.stdout: flex-2.6.1-9.el8.x86_64 2022-09-10T04:17:08.372 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-2.9.7-15.el8.x86_64 2022-09-10T04:17:08.372 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2022-09-10T04:17:08.372 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-3.el8.x86_64 2022-09-10T04:17:08.372 INFO:teuthology.orchestra.run.smithi081.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2022-09-10T04:17:08.372 INFO:teuthology.orchestra.run.smithi081.stdout: krb5-devel-1.18.2-14.el8.x86_64 2022-09-10T04:17:08.373 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel-0.3.112-1.el8.x86_64 2022-09-10T04:17:08.373 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.373 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.373 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.374 INFO:teuthology.orchestra.run.smithi081.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2022-09-10T04:17:08.374 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2022-09-10T04:17:08.374 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.2-3.el8.x86_64 2022-09-10T04:17:08.374 INFO:teuthology.orchestra.run.smithi081.stdout: libpq-13.5-1.el8.x86_64 2022-09-10T04:17:08.375 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2022-09-10T04:17:08.375 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.9.0-3.el8.x86_64 2022-09-10T04:17:08.375 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.376 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.376 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-0.11.4-3.el8.x86_64 2022-09-10T04:17:08.376 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.377 INFO:teuthology.orchestra.run.smithi081.stdout: libselinux-devel-2.9-5.el8.x86_64 2022-09-10T04:17:08.377 INFO:teuthology.orchestra.run.smithi081.stdout: libsepol-devel-2.9-3.el8.x86_64 2022-09-10T04:17:08.377 INFO:teuthology.orchestra.run.smithi081.stdout: libtool-2.4.6-25.el8.x86_64 2022-09-10T04:17:08.377 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.3.1-3.el8.x86_64 2022-09-10T04:17:08.378 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2022-09-10T04:17:08.378 INFO:teuthology.orchestra.run.smithi081.stdout: libverto-devel-0.3.0-5.el8.x86_64 2022-09-10T04:17:08.378 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.8.1-11.el8.x86_64 2022-09-10T04:17:08.378 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.8.2-2.el8.x86_64 2022-09-10T04:17:08.379 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.15-3.el8.x86_64 2022-09-10T04:17:08.379 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-threads-0.3.15-3.el8.x86_64 2022-09-10T04:17:08.379 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2022-09-10T04:17:08.380 INFO:teuthology.orchestra.run.smithi081.stdout: pcre2-devel-10.32-2.el8.x86_64 2022-09-10T04:17:08.380 INFO:teuthology.orchestra.run.smithi081.stdout: pcre2-utf16-10.32-2.el8.x86_64 2022-09-10T04:17:08.380 INFO:teuthology.orchestra.run.smithi081.stdout: pcre2-utf32-10.32-2.el8.x86_64 2022-09-10T04:17:08.380 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:17:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:17:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2022-09-10T04:17:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2022-09-10T04:17:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.5.1-7.el8.noarch 2022-09-10T04:17:08.381 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2022-09-10T04:17:08.382 INFO:teuthology.orchestra.run.smithi081.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2022-09-10T04:17:08.382 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-3.1.1-4.el8.noarch 2022-09-10T04:17:08.382 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.382 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.383 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.383 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2018.10.15-7.el8.noarch 2022-09-10T04:17:08.384 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.11.5-5.el8.x86_64 2022-09-10T04:17:08.384 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.5.2-1.el8.noarch 2022-09-10T04:17:08.384 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2022-09-10T04:17:08.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2022-09-10T04:17:08.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2022-09-10T04:17:08.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-6.2-6.el8.noarch 2022-09-10T04:17:08.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2022-09-10T04:17:08.385 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.10.1-3.el8.noarch 2022-09-10T04:17:08.386 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-2.el8.noarch 2022-09-10T04:17:08.386 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-1.10-11.el8.noarch 2022-09-10T04:17:08.387 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jwt-1.6.1-2.el8.noarch 2022-09-10T04:17:08.387 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2022-09-10T04:17:08.387 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-11.el8.noarch 2022-09-10T04:17:08.388 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.0.6-13.el8.noarch 2022-09-10T04:17:08.388 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-0.23-19.el8.x86_64 2022-09-10T04:17:08.389 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2022-09-10T04:17:08.389 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-2.el8.noarch 2022-09-10T04:17:08.389 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2022-09-10T04:17:08.390 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2022-09-10T04:17:08.390 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2022-09-10T04:17:08.391 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.3.2-9.el8.noarch 2022-09-10T04:17:08.392 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-2.6-1.el8.noarch 2022-09-10T04:17:08.392 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-14.el8.noarch 2022-09-10T04:17:08.393 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2022-09-10T04:17:08.393 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2022-09-10T04:17:08.395 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2022-09-10T04:17:08.395 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.14-14.el8.noarch 2022-09-10T04:17:08.396 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2017.2-9.el8.noarch 2022-09-10T04:17:08.396 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.397 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.397 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-6.el8.noarch 2022-09-10T04:17:08.398 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2022-09-10T04:17:08.398 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.398 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.4.1-12.el8.noarch 2022-09-10T04:17:08.398 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.8-1.el8.noarch 2022-09-10T04:17:08.399 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2022-09-10T04:17:08.399 INFO:teuthology.orchestra.run.smithi081.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2022-09-10T04:17:08.399 INFO:teuthology.orchestra.run.smithi081.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2022-09-10T04:17:08.399 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-1.14.1-5.el8.noarch 2022-09-10T04:17:08.400 INFO:teuthology.orchestra.run.smithi081.stdout: python3-trustme-0.6.0-4.el8.noarch 2022-09-10T04:17:08.400 INFO:teuthology.orchestra.run.smithi081.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2022-09-10T04:17:08.400 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.5-1.el8.1.noarch 2022-09-10T04:17:08.401 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2022-09-10T04:17:08.401 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webtest-2.0.33-1.el8.noarch 2022-09-10T04:17:08.401 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2022-09-10T04:17:08.401 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2022-09-10T04:17:08.402 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.402 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.402 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:17.0.0-14898.g24441860.el8.x86_64 2022-09-10T04:17:08.402 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-1.el8.x86_64 2022-09-10T04:17:08.402 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.13.0-2.el8.x86_64 2022-09-10T04:17:08.403 INFO:teuthology.orchestra.run.smithi081.stdout: uuid-1.6.2-43.el8.x86_64 2022-09-10T04:17:08.403 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2022-09-10T04:17:08.403 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2022-09-10T04:17:08.404 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2022-09-10T04:17:08.404 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:17:08.404 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2022-09-10T04:17:08.767 DEBUG:teuthology.parallel:result is None 2022-09-10T04:17:08.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2022-09-10T04:17:08.953 INFO:teuthology.orchestra.run.smithi081.stdout:RedHatEnterprise 2022-09-10T04:17:08.960 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:17:08.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:17:09.122 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-10T04:17:09.195 INFO:teuthology.orchestra.run.smithi081.stdout:17.0.0-14898.g24441860.el8 2022-09-10T04:17:09.196 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14898.g24441860.el8 2022-09-10T04:17:09.197 INFO:teuthology.task.install:The correct ceph version 17.0.0-14898.g24441860 is installed. 2022-09-10T04:17:09.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo lsb_release -is 2022-09-10T04:17:09.264 INFO:teuthology.orchestra.run.smithi104.stdout:RedHatEnterprise 2022-09-10T04:17:09.267 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:17:09.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:17:09.420 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-10T04:17:09.447 INFO:teuthology.orchestra.run.smithi104.stdout:17.0.0-14898.g24441860.el8 2022-09-10T04:17:09.448 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14898.g24441860.el8 2022-09-10T04:17:09.448 INFO:teuthology.task.install:The correct ceph version 17.0.0-14898.g24441860 is installed. 2022-09-10T04:17:09.449 DEBUG:teuthology.orchestra.run.smithi110:> sudo lsb_release -is 2022-09-10T04:17:09.514 INFO:teuthology.orchestra.run.smithi110.stdout:RedHatEnterprise 2022-09-10T04:17:09.516 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2022-09-10T04:17:09.517 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:17:09.669 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2022-09-10T04:17:09.695 INFO:teuthology.orchestra.run.smithi110.stdout:17.0.0-14898.g24441860.el8 2022-09-10T04:17:09.696 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-14898.g24441860.el8 2022-09-10T04:17:09.697 INFO:teuthology.task.install:The correct ceph version 17.0.0-14898.g24441860 is installed. 2022-09-10T04:17:09.698 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-09-10T04:17:09.699 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:17:09.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-10T04:17:09.789 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:17:09.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-10T04:17:09.822 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:17:09.822 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-09-10T04:17:09.853 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-09-10T04:17:09.854 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:17:09.854 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2022-09-10T04:17:09.943 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-10T04:17:10.032 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:17:10.033 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2022-09-10T04:17:10.066 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-10T04:17:10.140 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:17:10.140 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2022-09-10T04:17:10.173 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-09-10T04:17:10.244 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-09-10T04:17:10.244 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:17:10.245 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-10T04:17:10.339 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-10T04:17:10.469 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:17:10.470 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-10T04:17:10.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-10T04:17:10.573 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:17:10.574 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2022-09-10T04:17:10.607 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-09-10T04:17:10.678 INFO:teuthology.run_tasks:Running task cephadm... 2022-09-10T04:17:10.787 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '244418605927bb7842e73fc5d2a53889b760937f'} 2022-09-10T04:17:10.788 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:17:10.789 INFO:tasks.cephadm:Cluster fsid is 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:17:10.789 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2022-09-10T04:17:10.789 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.81', 'mon.b': '172.21.15.104', 'mon.c': '172.21.15.110'} 2022-09-10T04:17:10.790 INFO:tasks.cephadm:First mon is mon.a on smithi081 2022-09-10T04:17:10.790 INFO:tasks.cephadm:First mgr is x 2022-09-10T04:17:10.790 INFO:tasks.cephadm:Normalizing hostnames... 2022-09-10T04:17:10.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo hostname $(hostname -s) 2022-09-10T04:17:10.885 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2022-09-10T04:17:10.918 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2022-09-10T04:17:10.951 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref 244418605927bb7842e73fc5d2a53889b760937f)... 2022-09-10T04:17:10.952 DEBUG:teuthology.orchestra.run.smithi081:> git archive --remote=git://git.ceph.com/ceph-ci.git 244418605927bb7842e73fc5d2a53889b760937f src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-09-10T04:17:11.071 DEBUG:teuthology.orchestra.run.smithi104:> git archive --remote=git://git.ceph.com/ceph-ci.git 244418605927bb7842e73fc5d2a53889b760937f src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-09-10T04:17:11.137 DEBUG:teuthology.orchestra.run.smithi110:> git archive --remote=git://git.ceph.com/ceph-ci.git 244418605927bb7842e73fc5d2a53889b760937f src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2022-09-10T04:17:11.204 DEBUG:teuthology.orchestra.run.smithi081:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-09-10T04:17:11.268 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-09-10T04:17:11.290 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2022-09-10T04:17:11.334 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f on all hosts... 2022-09-10T04:17:11.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f pull 2022-09-10T04:17:11.341 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f pull 2022-09-10T04:17:11.344 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f pull 2022-09-10T04:17:12.556 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f... 2022-09-10T04:17:12.773 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f... 2022-09-10T04:17:17.704 INFO:teuthology.orchestra.run.smithi081.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f... 2022-09-10T04:17:42.141 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2022-09-10T04:17:42.141 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev)", 2022-09-10T04:17:42.142 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "99abe8ecf349e26717ff98ecadc0eb52a10e136a556f660580abb29e949a9315", 2022-09-10T04:17:42.142 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2022-09-10T04:17:42.142 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ef7943fd189c4c1c78042f6a65ad2f9f5a8ed5762896e1b11df242b88d2216d2" 2022-09-10T04:17:42.143 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2022-09-10T04:17:42.143 INFO:teuthology.orchestra.run.smithi104.stdout:} 2022-09-10T04:17:52.085 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2022-09-10T04:17:52.085 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev)", 2022-09-10T04:17:52.086 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "99abe8ecf349e26717ff98ecadc0eb52a10e136a556f660580abb29e949a9315", 2022-09-10T04:17:52.086 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2022-09-10T04:17:52.086 INFO:teuthology.orchestra.run.smithi110.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ef7943fd189c4c1c78042f6a65ad2f9f5a8ed5762896e1b11df242b88d2216d2" 2022-09-10T04:17:52.086 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2022-09-10T04:17:52.087 INFO:teuthology.orchestra.run.smithi110.stdout:} 2022-09-10T04:17:54.250 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:17:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev)", 2022-09-10T04:17:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "image_id": "99abe8ecf349e26717ff98ecadc0eb52a10e136a556f660580abb29e949a9315", 2022-09-10T04:17:54.251 INFO:teuthology.orchestra.run.smithi081.stdout: "repo_digests": [ 2022-09-10T04:17:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ef7943fd189c4c1c78042f6a65ad2f9f5a8ed5762896e1b11df242b88d2216d2" 2022-09-10T04:17:54.252 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2022-09-10T04:17:54.252 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:17:54.281 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2022-09-10T04:17:54.374 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2022-09-10T04:17:54.413 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2022-09-10T04:17:54.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /etc/ceph 2022-09-10T04:17:54.537 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2022-09-10T04:17:54.569 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2022-09-10T04:17:54.601 INFO:tasks.cephadm:Writing seed config... 2022-09-10T04:17:54.602 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2022-09-10T04:17:54.603 INFO:tasks.cephadm: override: [client] debug client = 20 2022-09-10T04:17:54.603 INFO:tasks.cephadm: override: [client] debug ms = 1 2022-09-10T04:17:54.603 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2022-09-10T04:17:54.604 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2022-09-10T04:17:54.604 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2022-09-10T04:17:54.604 INFO:tasks.cephadm: override: [mds] debug mds = 20 2022-09-10T04:17:54.605 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2022-09-10T04:17:54.605 INFO:tasks.cephadm: override: [mds] debug ms = 1 2022-09-10T04:17:54.605 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2022-09-10T04:17:54.606 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2022-09-10T04:17:54.606 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2022-09-10T04:17:54.606 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2022-09-10T04:17:54.607 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2022-09-10T04:17:54.607 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2022-09-10T04:17:54.607 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2022-09-10T04:17:54.608 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2022-09-10T04:17:54.608 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2022-09-10T04:17:54.609 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2022-09-10T04:17:54.609 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2022-09-10T04:17:54.610 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2022-09-10T04:17:54.610 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2022-09-10T04:17:54.611 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2022-09-10T04:17:54.611 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2022-09-10T04:17:54.611 INFO:tasks.cephadm: override: [mon] debug mon = 20 2022-09-10T04:17:54.612 INFO:tasks.cephadm: override: [mon] debug ms = 1 2022-09-10T04:17:54.612 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2022-09-10T04:17:54.613 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2022-09-10T04:17:54.613 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2022-09-10T04:17:54.614 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2022-09-10T04:17:54.615 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2022-09-10T04:17:54.615 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2022-09-10T04:17:54.616 INFO:tasks.cephadm: override: [osd] debug ms = 1 2022-09-10T04:17:54.616 INFO:tasks.cephadm: override: [osd] debug osd = 20 2022-09-10T04:17:54.616 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2022-09-10T04:17:54.617 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2022-09-10T04:17:54.617 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2022-09-10T04:17:54.618 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2022-09-10T04:17:54.618 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2022-09-10T04:17:54.619 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2022-09-10T04:17:54.619 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2022-09-10T04:17:54.620 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2022-09-10T04:17:54.620 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2022-09-10T04:17:54.621 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:17:54.622 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2022-09-10T04:17:54.679 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 70cf895c-30bf-11ed-8431-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2022-09-10T04:17:54.680 DEBUG:teuthology.orchestra.run.smithi081:mon.a> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.a.service 2022-09-10T04:17:54.686 DEBUG:teuthology.orchestra.run.smithi081:mgr.x> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.x.service 2022-09-10T04:17:54.732 INFO:tasks.cephadm:Bootstrapping... 2022-09-10T04:17:54.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f -v bootstrap --fsid 70cf895c-30bf-11ed-8431-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.81 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2022-09-10T04:17:54.772 INFO:journalctl@ceph.mon.a.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:17:54.820 INFO:journalctl@ceph.mgr.x.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:17:55.057 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2022-09-10T04:17:55.057 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f', '-v', 'bootstrap', '--fsid', '70cf895c-30bf-11ed-8431-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.81', '--skip-admin-label'] 2022-09-10T04:17:55.249 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 4.1.1 2022-09-10T04:17:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2022-09-10T04:17:55.250 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying podman|docker is present... 2022-09-10T04:17:55.440 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 4.1.1 2022-09-10T04:17:55.440 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying lvm2 is present... 2022-09-10T04:17:55.441 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying time synchronization is in place... 2022-09-10T04:17:55.458 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2022-09-10T04:17:55.459 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2022-09-10T04:17:55.475 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2022-09-10T04:17:55.476 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2022-09-10T04:17:55.493 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout enabled 2022-09-10T04:17:55.510 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout active 2022-09-10T04:17:55.511 INFO:teuthology.orchestra.run.smithi081.stdout:Unit chronyd.service is enabled and running 2022-09-10T04:17:55.511 INFO:teuthology.orchestra.run.smithi081.stdout:Repeating the final host check... 2022-09-10T04:17:55.699 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 4.1.1 2022-09-10T04:17:55.700 INFO:teuthology.orchestra.run.smithi081.stdout:podman (/bin/podman) version 4.1.1 is present 2022-09-10T04:17:55.701 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl is present 2022-09-10T04:17:55.701 INFO:teuthology.orchestra.run.smithi081.stdout:lvcreate is present 2022-09-10T04:17:55.716 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2022-09-10T04:17:55.717 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2022-09-10T04:17:55.732 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2022-09-10T04:17:55.733 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2022-09-10T04:17:55.750 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout enabled 2022-09-10T04:17:55.767 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout active 2022-09-10T04:17:55.768 INFO:teuthology.orchestra.run.smithi081.stdout:Unit chronyd.service is enabled and running 2022-09-10T04:17:55.768 INFO:teuthology.orchestra.run.smithi081.stdout:Host looks OK 2022-09-10T04:17:55.768 INFO:teuthology.orchestra.run.smithi081.stdout:Cluster fsid: 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:17:55.768 INFO:teuthology.orchestra.run.smithi081.stdout:Acquiring lock 140042914503984 on /run/cephadm/70cf895c-30bf-11ed-8431-001a4aab830c.lock 2022-09-10T04:17:55.769 INFO:teuthology.orchestra.run.smithi081.stdout:Lock 140042914503984 acquired on /run/cephadm/70cf895c-30bf-11ed-8431-001a4aab830c.lock 2022-09-10T04:17:55.769 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying IP 172.21.15.81 port 3300 ... 2022-09-10T04:17:55.770 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying IP 172.21.15.81 port 6789 ... 2022-09-10T04:17:55.770 INFO:teuthology.orchestra.run.smithi081.stdout:Base mon IP(s) is [172.21.15.81:3300, 172.21.15.81:6789], mon addrv is [v2:172.21.15.81:3300,v1:172.21.15.81:6789] 2022-09-10T04:17:55.779 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2022-09-10T04:17:55.779 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.81 metric 100 2022-09-10T04:17:55.789 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2022-09-10T04:17:55.789 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2022-09-10T04:17:55.789 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1777sec hoplimit 64 pref medium 2022-09-10T04:17:55.798 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2022-09-10T04:17:55.798 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2022-09-10T04:17:55.799 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2022-09-10T04:17:55.799 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2022-09-10T04:17:55.799 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2022-09-10T04:17:55.799 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2022-09-10T04:17:55.800 INFO:teuthology.orchestra.run.smithi081.stdout:Mon IP `172.21.15.81` is in CIDR network `172.21.0.0/20` 2022-09-10T04:17:55.800 INFO:teuthology.orchestra.run.smithi081.stdout:Mon IP `172.21.15.81` is in CIDR network `172.21.0.0/20` 2022-09-10T04:17:55.801 INFO:teuthology.orchestra.run.smithi081.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2022-09-10T04:17:55.801 INFO:teuthology.orchestra.run.smithi081.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2022-09-10T04:17:55.802 INFO:teuthology.orchestra.run.smithi081.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f... 2022-09-10T04:17:57.001 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 99abe8ecf349e26717ff98ecadc0eb52a10e136a556f660580abb29e949a9315 2022-09-10T04:17:57.002 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f... 2022-09-10T04:17:57.002 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Getting image source signatures 2022-09-10T04:17:57.003 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:99a39ffc60545d3551e6b2d1fa8868206fd1c0517c86346f930cd3902f2e17c6 2022-09-10T04:17:57.003 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:5d1b43028e8e0373631944ff9016c99bdf03efc12705b9a423bc758fccf55806 2022-09-10T04:17:57.004 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:fb592cc51013ec12603a9147465efd45758ade1ae8577f001cb29668b77b1b99 2022-09-10T04:17:57.004 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:bb868b3138d9139f452065e8d61f4c081341a6caa61b0ac6f12814f3a0dc362e 2022-09-10T04:17:57.005 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:2252a7414fd07c781b1ed7bd7af688bb98409db8ecbad4aec3bacaa72fdc5aa7 2022-09-10T04:17:57.005 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying config sha256:99abe8ecf349e26717ff98ecadc0eb52a10e136a556f660580abb29e949a9315 2022-09-10T04:17:57.006 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Writing manifest to image destination 2022-09-10T04:17:57.006 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Storing signatures 2022-09-10T04:17:57.896 INFO:teuthology.orchestra.run.smithi081.stdout:ceph: stdout ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev) 2022-09-10T04:17:57.897 INFO:teuthology.orchestra.run.smithi081.stdout:Ceph version: ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev) 2022-09-10T04:17:57.897 INFO:teuthology.orchestra.run.smithi081.stdout:Extracting ceph user uid/gid from container image... 2022-09-10T04:17:58.689 INFO:teuthology.orchestra.run.smithi081.stdout:stat: stdout 167 167 2022-09-10T04:17:58.689 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial keys... 2022-09-10T04:17:59.488 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQD3Dxxj47vnBhAA9Cu5zVUmx1IVjqtayX8eoQ== 2022-09-10T04:18:00.284 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQD3DxxjS/ENORAAkbJNMKLu3LVZbzzOBOdu0g== 2022-09-10T04:18:01.083 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQD4DxxjV+mgKxAAbtlqEenB2QveQfS1eHO/Lg== 2022-09-10T04:18:01.085 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial monmap... 2022-09-10T04:18:04.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2022-09-10T04:18:04.492 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2022-09-10T04:18:04.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:04.493 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-09-10T04:18:04.493 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool for a [v2:172.21.15.81:3300,v1:172.21.15.81:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2022-09-10T04:18:04.494 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = octopus 2022-09-10T04:18:04.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: set fsid to 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:04.494 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2022-09-10T04:18:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:18:04.495 INFO:teuthology.orchestra.run.smithi081.stdout:Creating mon... 2022-09-10T04:18:05.395 INFO:teuthology.orchestra.run.smithi081.stdout:create mon.a on 2022-09-10T04:18:05.684 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2022-09-10T04:18:05.946 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2022-09-10T04:18:06.229 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-70cf895c-30bf-11ed-8431-001a4aab830c.target → /etc/systemd/system/ceph-70cf895c-30bf-11ed-8431-001a4aab830c.target. 2022-09-10T04:18:06.230 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-70cf895c-30bf-11ed-8431-001a4aab830c.target → /etc/systemd/system/ceph-70cf895c-30bf-11ed-8431-001a4aab830c.target. 2022-09-10T04:18:06.665 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.a 2022-09-10T04:18:06.665 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to reset failed state of unit ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.a.service: Unit ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.a.service not loaded. 2022-09-10T04:18:06.932 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-70cf895c-30bf-11ed-8431-001a4aab830c.target.wants/ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-70cf895c-30bf-11ed-8431-001a4aab830c@.service. 2022-09-10T04:18:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:06 smithi081 systemd[1]: Starting Ceph mon.a for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:18:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:07 smithi081 podman[89615]: 2022-09-10T04:18:07.956 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-10T04:18:07.956 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2022-09-10T04:18:07.976 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-10T04:18:07.977 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2022-09-10T04:18:07.978 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2022-09-10T04:18:07.978 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to enable service . firewalld.service is not available 2022-09-10T04:18:07.978 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mon to start... 2022-09-10T04:18:07.979 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mon... 2022-09-10T04:18:08.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:07 smithi081 bash[89615]: 4a64813cf6712b0793b31b8cb6c0cfe71280c3a1d9027f9afb65bc6588a56b49 2022-09-10T04:18:08.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:07 smithi081 ceph-mon[89656]: mkfs 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:08.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:07 smithi081 ceph-mon[89656]: mon.a is new leader, mons a in quorum (ranks 0) 2022-09-10T04:18:08.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:07 smithi081 systemd[1]: Started Ceph mon.a for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:18:08.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:08 smithi081 ceph-mon[89656]: mon.a is new leader, mons a in quorum (ranks 0) 2022-09-10T04:18:08.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:08 smithi081 ceph-mon[89656]: monmap e1: 1 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0]} 2022-09-10T04:18:08.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:08 smithi081 ceph-mon[89656]: fsmap 2022-09-10T04:18:08.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:08 smithi081 ceph-mon[89656]: osdmap e1: 0 total, 0 up, 0 in 2022-09-10T04:18:08.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:08 smithi081 ceph-mon[89656]: mgrmap e1: no daemons active 2022-09-10T04:18:08.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:08 smithi081 ceph-mon[89656]: from='client.? 172.21.15.81:0/574108133' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2022-09-10T04:18:08.986 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout cluster: 2022-09-10T04:18:08.987 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout id: 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:08.987 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2022-09-10T04:18:08.987 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:08.988 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout services: 2022-09-10T04:18:08.988 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.763517s) 2022-09-10T04:18:08.988 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2022-09-10T04:18:08.988 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2022-09-10T04:18:08.989 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:08.989 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout data: 2022-09-10T04:18:08.989 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2022-09-10T04:18:08.990 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2022-09-10T04:18:08.990 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2022-09-10T04:18:08.990 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout pgs: 2022-09-10T04:18:08.990 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:08.991 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.536+0000 7fb00935c700 1 Processor -- start 2022-09-10T04:18:08.991 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.536+0000 7fb00935c700 1 -- start start 2022-09-10T04:18:08.991 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.537+0000 7fb00935c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb0040782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:08.992 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.537+0000 7fb00935c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004078800 con 0x7fb004079de0 2022-09-10T04:18:08.992 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.538+0000 7fb002ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb0040782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:08.992 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.538+0000 7fb002ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb0040782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40766/0 (socket says 172.21.15.81:40766) 2022-09-10T04:18:08.993 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.538+0000 7fb002ffd700 1 -- 172.21.15.81:0/3655556471 learned_addr learned my addr 172.21.15.81:0/3655556471 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:08.993 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.538+0000 7fb002ffd700 1 -- 172.21.15.81:0/3655556471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb004078940 con 0x7fb004079de0 2022-09-10T04:18:08.993 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.539+0000 7fb002ffd700 1 --2- 172.21.15.81:0/3655556471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb0040782c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faff4009700 tx=0x7faff4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7816c3e25d0c386 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:08.993 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.539+0000 7fb001ffb700 1 -- 172.21.15.81:0/3655556471 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4004160 con 0x7fb004079de0 2022-09-10T04:18:08.994 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.539+0000 7fb001ffb700 1 -- 172.21.15.81:0/3655556471 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff40042c0 con 0x7fb004079de0 2022-09-10T04:18:08.994 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.539+0000 7fb001ffb700 1 -- 172.21.15.81:0/3655556471 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4011630 con 0x7fb004079de0 2022-09-10T04:18:08.994 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 -- 172.21.15.81:0/3655556471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 msgr2=0x7fb0040782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:08.994 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 --2- 172.21.15.81:0/3655556471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb0040782c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7faff4009700 tx=0x7faff4009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:08.995 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 -- 172.21.15.81:0/3655556471 shutdown_connections 2022-09-10T04:18:08.995 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 --2- 172.21.15.81:0/3655556471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb0040782c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:08.995 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 -- 172.21.15.81:0/3655556471 >> 172.21.15.81:0/3655556471 conn(0x7fb0040fe9f0 msgr2=0x7fb004100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:08.995 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 -- 172.21.15.81:0/3655556471 shutdown_connections 2022-09-10T04:18:08.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.540+0000 7fb00935c700 1 -- 172.21.15.81:0/3655556471 wait complete. 2022-09-10T04:18:08.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.541+0000 7fb00935c700 1 Processor -- start 2022-09-10T04:18:08.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.541+0000 7fb00935c700 1 -- start start 2022-09-10T04:18:08.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.542+0000 7fb00935c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb004198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:08.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.542+0000 7fb00935c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb004198ed0 con 0x7fb004079de0 2022-09-10T04:18:08.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.542+0000 7fb002ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb004198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:08.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.542+0000 7fb002ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb004198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40774/0 (socket says 172.21.15.81:40774) 2022-09-10T04:18:08.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.542+0000 7fb002ffd700 1 -- 172.21.15.81:0/574108133 learned_addr learned my addr 172.21.15.81:0/574108133 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:08.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.542+0000 7fb002ffd700 1 -- 172.21.15.81:0/574108133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40093e0 con 0x7fb004079de0 2022-09-10T04:18:08.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.543+0000 7fb002ffd700 1 --2- 172.21.15.81:0/574108133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb004198990 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faff4004000 tx=0x7faff40045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:08.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.543+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4003520 con 0x7fb004079de0 2022-09-10T04:18:08.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.543+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4003680 con 0x7fb004079de0 2022-09-10T04:18:08.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.543+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0041990d0 con 0x7fb004079de0 2022-09-10T04:18:08.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.543+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faff4003920 con 0x7fb004079de0 2022-09-10T04:18:08.999 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.543+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb004199500 con 0x7fb004079de0 2022-09-10T04:18:08.999 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.544+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7faff402c020 con 0x7fb004079de0 2022-09-10T04:18:08.999 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.544+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7faff401cbf0 con 0x7fb004079de0 2022-09-10T04:18:08.999 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.544+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb004060310 con 0x7fb004079de0 2022-09-10T04:18:09.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.548+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7faff401c450 con 0x7fb004079de0 2022-09-10T04:18:09.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.620+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb00405f990 con 0x7fb004079de0 2022-09-10T04:18:09.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.621+0000 7fafebfff700 1 -- 172.21.15.81:0/574108133 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7faff401cdb0 con 0x7fb004079de0 2022-09-10T04:18:09.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 msgr2=0x7fb004198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:09.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 --2- 172.21.15.81:0/574108133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb004198990 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7faff4004000 tx=0x7faff40045d0 comp rx=0 tx=0).stop 2022-09-10T04:18:09.001 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 shutdown_connections 2022-09-10T04:18:09.001 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 --2- 172.21.15.81:0/574108133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb004079de0 0x7fb004198990 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:09.001 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 >> 172.21.15.81:0/574108133 conn(0x7fb0040fe9f0 msgr2=0x7fb004100390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:09.001 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 shutdown_connections 2022-09-10T04:18:09.002 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:08.622+0000 7fb00935c700 1 -- 172.21.15.81:0/574108133 wait complete. 2022-09-10T04:18:09.002 INFO:teuthology.orchestra.run.smithi081.stdout:mon is available 2022-09-10T04:18:09.002 INFO:teuthology.orchestra.run.smithi081.stdout:Assimilating anything we can from ceph.conf... 2022-09-10T04:18:10.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:09 smithi081 ceph-mon[89656]: from='client.? 172.21.15.81:0/916879342' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-09-10T04:18:10.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:09 smithi081 ceph-mon[89656]: from='client.? 172.21.15.81:0/916879342' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2022-09-10T04:18:10.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:10.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [global] 2022-09-10T04:18:10.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout fsid = 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:10.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.81:3300,v1:172.21.15.81:6789] 2022-09-10T04:18:10.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2022-09-10T04:18:10.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2022-09-10T04:18:10.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2022-09-10T04:18:10.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2022-09-10T04:18:10.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:10.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [mgr] 2022-09-10T04:18:10.090 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2022-09-10T04:18:10.090 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:10.090 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [osd] 2022-09-10T04:18:10.090 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2022-09-10T04:18:10.091 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2022-09-10T04:18:10.091 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2022-09-10T04:18:10.091 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2022-09-10T04:18:10.091 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2022-09-10T04:18:10.091 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.584+0000 7f950894a700 1 Processor -- start 2022-09-10T04:18:10.092 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.584+0000 7f950894a700 1 -- start start 2022-09-10T04:18:10.092 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.584+0000 7f950894a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 0x7f9504108360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:10.092 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.584+0000 7f950894a700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504103de0 con 0x7f9504077ce0 2022-09-10T04:18:10.093 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.584+0000 7f950259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 0x7f9504108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:10.093 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.585+0000 7f950259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 0x7f9504108360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40786/0 (socket says 172.21.15.81:40786) 2022-09-10T04:18:10.093 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.585+0000 7f950259c700 1 -- 172.21.15.81:0/3505209885 learned_addr learned my addr 172.21.15.81:0/3505209885 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:10.094 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.585+0000 7f950259c700 1 -- 172.21.15.81:0/3505209885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95041088a0 con 0x7f9504077ce0 2022-09-10T04:18:10.094 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.585+0000 7f950259c700 1 --2- 172.21.15.81:0/3505209885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 0x7f9504108360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009700 tx=0x7f94f4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6c48f054a0d9f0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:10.094 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950159a700 1 -- 172.21.15.81:0/3505209885 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f4004160 con 0x7f9504077ce0 2022-09-10T04:18:10.095 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950159a700 1 -- 172.21.15.81:0/3505209885 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f40042c0 con 0x7f9504077ce0 2022-09-10T04:18:10.095 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950159a700 1 -- 172.21.15.81:0/3505209885 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f4011630 con 0x7f9504077ce0 2022-09-10T04:18:10.095 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 -- 172.21.15.81:0/3505209885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 msgr2=0x7f9504108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:10.095 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 --2- 172.21.15.81:0/3505209885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 0x7f9504108360 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009700 tx=0x7f94f4009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:10.095 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 -- 172.21.15.81:0/3505209885 shutdown_connections 2022-09-10T04:18:10.096 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 --2- 172.21.15.81:0/3505209885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504077ce0 0x7f9504108360 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:10.096 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 -- 172.21.15.81:0/3505209885 >> 172.21.15.81:0/3505209885 conn(0x7f95040fd8f0 msgr2=0x7f95040ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:10.096 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 -- 172.21.15.81:0/3505209885 shutdown_connections 2022-09-10T04:18:10.096 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.586+0000 7f950894a700 1 -- 172.21.15.81:0/3505209885 wait complete. 2022-09-10T04:18:10.097 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.587+0000 7f950894a700 1 Processor -- start 2022-09-10T04:18:10.097 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.588+0000 7f950894a700 1 -- start start 2022-09-10T04:18:10.097 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.588+0000 7f950894a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 0x7f9504198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:10.097 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.588+0000 7f950894a700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504199270 con 0x7f9504198920 2022-09-10T04:18:10.097 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.588+0000 7f950259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 0x7f9504198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:10.098 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.588+0000 7f950259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 0x7f9504198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40800/0 (socket says 172.21.15.81:40800) 2022-09-10T04:18:10.098 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.588+0000 7f950259c700 1 -- 172.21.15.81:0/916879342 learned_addr learned my addr 172.21.15.81:0/916879342 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:10.098 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.589+0000 7f950259c700 1 -- 172.21.15.81:0/916879342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40093e0 con 0x7f9504198920 2022-09-10T04:18:10.098 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.589+0000 7f950259c700 1 --2- 172.21.15.81:0/916879342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 0x7f9504198d30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004000 tx=0x7f94f40045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:10.098 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.589+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f4003520 con 0x7f9504198920 2022-09-10T04:18:10.099 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.589+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4003680 con 0x7f9504198920 2022-09-10T04:18:10.099 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.589+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94f4003920 con 0x7f9504198920 2022-09-10T04:18:10.099 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.589+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9504199470 con 0x7f9504198920 2022-09-10T04:18:10.099 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.590+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950419c0d0 con 0x7f9504198920 2022-09-10T04:18:10.100 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.590+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f94f402c020 con 0x7f9504198920 2022-09-10T04:18:10.100 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.590+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f94f401ccd0 con 0x7f9504198920 2022-09-10T04:18:10.100 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.591+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9504060310 con 0x7f9504198920 2022-09-10T04:18:10.100 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.594+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f94f401c450 con 0x7f9504198920 2022-09-10T04:18:10.100 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.662+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f9504077ce0 con 0x7f9504198920 2022-09-10T04:18:10.101 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.667+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 7 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f94f401ce90 con 0x7f9504198920 2022-09-10T04:18:10.101 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.667+0000 7f94f37fe700 1 -- 172.21.15.81:0/916879342 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f94f4019580 con 0x7f9504198920 2022-09-10T04:18:10.101 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 msgr2=0x7f9504198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:10.101 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 --2- 172.21.15.81:0/916879342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 0x7f9504198d30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004000 tx=0x7f94f40045d0 comp rx=0 tx=0).stop 2022-09-10T04:18:10.102 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 shutdown_connections 2022-09-10T04:18:10.102 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 --2- 172.21.15.81:0/916879342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9504198920 0x7f9504198d30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:10.102 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 >> 172.21.15.81:0/916879342 conn(0x7f95040fd8f0 msgr2=0x7f95040ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:10.102 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 shutdown_connections 2022-09-10T04:18:10.102 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:09.668+0000 7f950894a700 1 -- 172.21.15.81:0/916879342 wait complete. 2022-09-10T04:18:10.103 INFO:teuthology.orchestra.run.smithi081.stdout:Generating new minimal ceph.conf... 2022-09-10T04:18:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:10 smithi081 ceph-mon[89656]: from='client.? 172.21.15.81:0/2443403796' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:18:11.188 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.685+0000 7f8993909700 1 Processor -- start 2022-09-10T04:18:11.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.686+0000 7f8993909700 1 -- start start 2022-09-10T04:18:11.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.686+0000 7f8993909700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c1037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:11.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.686+0000 7f8993909700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c103da0 con 0x7f898c1033c0 2022-09-10T04:18:11.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.686+0000 7f89916a5700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c1037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:11.190 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.686+0000 7f89916a5700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c1037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48888/0 (socket says 172.21.15.81:48888) 2022-09-10T04:18:11.190 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.686+0000 7f89916a5700 1 -- 172.21.15.81:0/1788924936 learned_addr learned my addr 172.21.15.81:0/1788924936 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:11.191 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.687+0000 7f89916a5700 1 -- 172.21.15.81:0/1788924936 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c103ee0 con 0x7f898c1033c0 2022-09-10T04:18:11.191 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.687+0000 7f89916a5700 1 --2- 172.21.15.81:0/1788924936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c1037d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8988009700 tx=0x7f8988009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cacc1d1b143044 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:11.192 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.687+0000 7f897bfff700 1 -- 172.21.15.81:0/1788924936 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8988004160 con 0x7f898c1033c0 2022-09-10T04:18:11.192 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.687+0000 7f897bfff700 1 -- 172.21.15.81:0/1788924936 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f89880042c0 con 0x7f898c1033c0 2022-09-10T04:18:11.192 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.687+0000 7f897bfff700 1 -- 172.21.15.81:0/1788924936 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f89880039d0 con 0x7f898c1033c0 2022-09-10T04:18:11.193 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 -- 172.21.15.81:0/1788924936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 msgr2=0x7f898c1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:11.193 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 --2- 172.21.15.81:0/1788924936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c1037d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8988009700 tx=0x7f8988009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:11.193 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 -- 172.21.15.81:0/1788924936 shutdown_connections 2022-09-10T04:18:11.194 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 --2- 172.21.15.81:0/1788924936 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c1037d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:11.194 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 -- 172.21.15.81:0/1788924936 >> 172.21.15.81:0/1788924936 conn(0x7f898c0fe9f0 msgr2=0x7f898c100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:11.194 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 -- 172.21.15.81:0/1788924936 shutdown_connections 2022-09-10T04:18:11.195 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.688+0000 7f8993909700 1 -- 172.21.15.81:0/1788924936 wait complete. 2022-09-10T04:18:11.195 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.690+0000 7f8993909700 1 Processor -- start 2022-09-10T04:18:11.195 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.690+0000 7f8993909700 1 -- start start 2022-09-10T04:18:11.196 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.691+0000 7f8993909700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:11.196 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.691+0000 7f8993909700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c197410 con 0x7f898c1033c0 2022-09-10T04:18:11.196 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.691+0000 7f89916a5700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:11.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.691+0000 7f89916a5700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48904/0 (socket says 172.21.15.81:48904) 2022-09-10T04:18:11.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.691+0000 7f89916a5700 1 -- 172.21.15.81:0/2443403796 learned_addr learned my addr 172.21.15.81:0/2443403796 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:11.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.691+0000 7f89916a5700 1 -- 172.21.15.81:0/2443403796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89880093e0 con 0x7f898c1033c0 2022-09-10T04:18:11.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f89916a5700 1 --2- 172.21.15.81:0/2443403796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c196ed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8988009e90 tx=0x7f8988004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:11.198 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8988011840 con 0x7f898c1033c0 2022-09-10T04:18:11.198 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f898801a430 con 0x7f898c1033c0 2022-09-10T04:18:11.198 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8988019550 con 0x7f898c1033c0 2022-09-10T04:18:11.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c197610 con 0x7f898c1033c0 2022-09-10T04:18:11.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c197a40 con 0x7f898c1033c0 2022-09-10T04:18:11.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.692+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f898802a020 con 0x7f898c1033c0 2022-09-10T04:18:11.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.693+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f898802d480 con 0x7f898c1033c0 2022-09-10T04:18:11.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.693+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898c060310 con 0x7f898c1033c0 2022-09-10T04:18:11.200 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.696+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f898801a5a0 con 0x7f898c1033c0 2022-09-10T04:18:11.200 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.765+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f898c19a880 con 0x7f898c1033c0 2022-09-10T04:18:11.200 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.766+0000 7f897a7fc700 1 -- 172.21.15.81:0/2443403796 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f898801a740 con 0x7f898c1033c0 2022-09-10T04:18:11.201 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 msgr2=0x7f898c196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:11.201 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 --2- 172.21.15.81:0/2443403796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c196ed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8988009e90 tx=0x7f8988004040 comp rx=0 tx=0).stop 2022-09-10T04:18:11.201 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 shutdown_connections 2022-09-10T04:18:11.202 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 --2- 172.21.15.81:0/2443403796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f898c1033c0 0x7f898c196ed0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:11.202 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 >> 172.21.15.81:0/2443403796 conn(0x7f898c0fe9f0 msgr2=0x7f898c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:11.202 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 shutdown_connections 2022-09-10T04:18:11.203 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:10.767+0000 7f8993909700 1 -- 172.21.15.81:0/2443403796 wait complete. 2022-09-10T04:18:11.203 INFO:teuthology.orchestra.run.smithi081.stdout:Restarting the monitor... 2022-09-10T04:18:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:11 smithi081 systemd[1]: Stopping Ceph mon.a for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:18:11.944 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[89632]: 2022-09-10T04:18:11.690+0000 7f7b80169700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2022-09-10T04:18:11.944 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[89632]: 2022-09-10T04:18:11.690+0000 7f7b80169700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2022-09-10T04:18:12.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:11 smithi081 bash[89993]: ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a 2022-09-10T04:18:12.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:12 smithi081 systemd[1]: ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.a.service: Succeeded. 2022-09-10T04:18:12.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:12 smithi081 systemd[1]: Stopped Ceph mon.a for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:18:12.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:12 smithi081 systemd[1]: Starting Ceph mon.a for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:18:13.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:12 smithi081 podman[90093]: 2022-09-10T04:18:13.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:12 smithi081 bash[90093]: 93806b9c523d74305d13b4a6d02922b99bfbdd145271ac6572092f30adff0a93 2022-09-10T04:18:13.050 INFO:teuthology.orchestra.run.smithi081.stdout:Setting mon public_network to 172.21.0.0/20 2022-09-10T04:18:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: set uid:gid to 167:167 (ceph:ceph) 2022-09-10T04:18:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev), process ceph-mon, pid 7 2022-09-10T04:18:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: pidfile_write: ignore empty --pid-file 2022-09-10T04:18:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: load: jerasure load: lrc 2022-09-10T04:18:13.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: RocksDB version: 6.15.5 2022-09-10T04:18:13.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-09-10T04:18:13.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Compile date Sep 9 2022 2022-09-10T04:18:13.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: DB SUMMARY 2022-09-10T04:18:13.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: DB Session ID: 71F0GIKT0XCZF3IUYYZF 2022-09-10T04:18:13.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: CURRENT file: CURRENT 2022-09-10T04:18:13.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: IDENTITY file: IDENTITY 2022-09-10T04:18:13.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2022-09-10T04:18:13.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2022-09-10T04:18:13.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 96567 ; 2022-09-10T04:18:13.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.error_if_exists: 0 2022-09-10T04:18:13.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.create_if_missing: 0 2022-09-10T04:18:13.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.paranoid_checks: 1 2022-09-10T04:18:13.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-09-10T04:18:13.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.env: 0x559515fdb100 2022-09-10T04:18:13.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.fs: Posix File System 2022-09-10T04:18:13.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.info_log: 0x559517be3c20 2022-09-10T04:18:13.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_file_opening_threads: 16 2022-09-10T04:18:13.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.statistics: (nil) 2022-09-10T04:18:13.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.use_fsync: 0 2022-09-10T04:18:13.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_log_file_size: 0 2022-09-10T04:18:13.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-09-10T04:18:13.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.log_file_time_to_roll: 0 2022-09-10T04:18:13.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.keep_log_file_num: 1000 2022-09-10T04:18:13.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.recycle_log_file_num: 0 2022-09-10T04:18:13.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.allow_fallocate: 1 2022-09-10T04:18:13.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.allow_mmap_reads: 0 2022-09-10T04:18:13.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.allow_mmap_writes: 0 2022-09-10T04:18:13.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.use_direct_reads: 0 2022-09-10T04:18:13.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-09-10T04:18:13.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.create_missing_column_families: 0 2022-09-10T04:18:13.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.db_log_dir: 2022-09-10T04:18:13.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2022-09-10T04:18:13.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.table_cache_numshardbits: 6 2022-09-10T04:18:13.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.WAL_ttl_seconds: 0 2022-09-10T04:18:13.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.WAL_size_limit_MB: 0 2022-09-10T04:18:13.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-09-10T04:18:13.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-09-10T04:18:13.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.is_fd_close_on_exec: 1 2022-09-10T04:18:13.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.advise_random_on_open: 1 2022-09-10T04:18:13.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.db_write_buffer_size: 0 2022-09-10T04:18:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.write_buffer_manager: 0x559517cd8b70 2022-09-10T04:18:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-09-10T04:18:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-09-10T04:18:13.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-09-10T04:18:13.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.use_adaptive_mutex: 0 2022-09-10T04:18:13.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.rate_limiter: (nil) 2022-09-10T04:18:13.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-09-10T04:18:13.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.wal_recovery_mode: 2 2022-09-10T04:18:13.317 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.enable_thread_tracking: 0 2022-09-10T04:18:13.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.enable_pipelined_write: 0 2022-09-10T04:18:13.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.unordered_write: 0 2022-09-10T04:18:13.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-09-10T04:18:13.318 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-09-10T04:18:13.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-09-10T04:18:13.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-09-10T04:18:13.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.row_cache: None 2022-09-10T04:18:13.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.wal_filter: None 2022-09-10T04:18:13.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-09-10T04:18:13.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.allow_ingest_behind: 0 2022-09-10T04:18:13.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.preserve_deletes: 0 2022-09-10T04:18:13.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.two_write_queues: 0 2022-09-10T04:18:13.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.manual_wal_flush: 0 2022-09-10T04:18:13.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.atomic_flush: 0 2022-09-10T04:18:13.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-09-10T04:18:13.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.persist_stats_to_disk: 0 2022-09-10T04:18:13.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.write_dbid_to_manifest: 0 2022-09-10T04:18:13.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.log_readahead_size: 0 2022-09-10T04:18:13.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-09-10T04:18:13.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.best_efforts_recovery: 0 2022-09-10T04:18:13.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-09-10T04:18:13.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-09-10T04:18:13.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.allow_data_in_errors: 0 2022-09-10T04:18:13.323 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.db_host_id: __hostname__ 2022-09-10T04:18:13.323 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_background_jobs: 2 2022-09-10T04:18:13.323 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_background_compactions: -1 2022-09-10T04:18:13.324 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_subcompactions: 1 2022-09-10T04:18:13.324 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-09-10T04:18:13.324 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-09-10T04:18:13.324 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.delayed_write_rate : 16777216 2022-09-10T04:18:13.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_total_wal_size: 0 2022-09-10T04:18:13.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-09-10T04:18:13.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.stats_dump_period_sec: 600 2022-09-10T04:18:13.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.stats_persist_period_sec: 600 2022-09-10T04:18:13.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-09-10T04:18:13.326 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_open_files: -1 2022-09-10T04:18:13.326 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bytes_per_sync: 0 2022-09-10T04:18:13.326 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.wal_bytes_per_sync: 0 2022-09-10T04:18:13.326 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.strict_bytes_per_sync: 0 2022-09-10T04:18:13.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_readahead_size: 0 2022-09-10T04:18:13.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_background_flushes: -1 2022-09-10T04:18:13.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Compression algorithms supported: 2022-09-10T04:18:13.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-09-10T04:18:13.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kZSTD supported: 0 2022-09-10T04:18:13.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kXpressCompression supported: 0 2022-09-10T04:18:13.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kLZ4HCCompression supported: 1 2022-09-10T04:18:13.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kZlibCompression supported: 1 2022-09-10T04:18:13.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kSnappyCompression supported: 1 2022-09-10T04:18:13.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kLZ4Compression supported: 1 2022-09-10T04:18:13.329 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: kBZip2Compression supported: 0 2022-09-10T04:18:13.329 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Fast CRC32 supported: Supported on x86 2022-09-10T04:18:13.329 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2022-09-10T04:18:13.329 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2022-09-10T04:18:13.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-09-10T04:18:13.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.merge_operator: 2022-09-10T04:18:13.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_filter: None 2022-09-10T04:18:13.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_filter_factory: None 2022-09-10T04:18:13.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.sst_partitioner_factory: None 2022-09-10T04:18:13.331 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.memtable_factory: SkipListFactory 2022-09-10T04:18:13.331 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.table_factory: BlockBasedTable 2022-09-10T04:18:13.331 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559517be0d48) 2022-09-10T04:18:13.331 INFO:journalctl@ceph.mon.a.smithi081.stdout: cache_index_and_filter_blocks: 1 2022-09-10T04:18:13.332 INFO:journalctl@ceph.mon.a.smithi081.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-09-10T04:18:13.332 INFO:journalctl@ceph.mon.a.smithi081.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-09-10T04:18:13.332 INFO:journalctl@ceph.mon.a.smithi081.stdout: pin_top_level_index_and_filter: 1 2022-09-10T04:18:13.332 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_type: 0 2022-09-10T04:18:13.332 INFO:journalctl@ceph.mon.a.smithi081.stdout: data_block_index_type: 0 2022-09-10T04:18:13.333 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_shortening: 1 2022-09-10T04:18:13.333 INFO:journalctl@ceph.mon.a.smithi081.stdout: data_block_hash_table_util_ratio: 0.750000 2022-09-10T04:18:13.333 INFO:journalctl@ceph.mon.a.smithi081.stdout: hash_index_allow_collision: 1 2022-09-10T04:18:13.334 INFO:journalctl@ceph.mon.a.smithi081.stdout: checksum: 1 2022-09-10T04:18:13.334 INFO:journalctl@ceph.mon.a.smithi081.stdout: no_block_cache: 0 2022-09-10T04:18:13.334 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache: 0x559517c4c9b0 2022-09-10T04:18:13.334 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_name: BinnedLRUCache 2022-09-10T04:18:13.335 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_options: 2022-09-10T04:18:13.335 INFO:journalctl@ceph.mon.a.smithi081.stdout: capacity : 536870912 2022-09-10T04:18:13.335 INFO:journalctl@ceph.mon.a.smithi081.stdout: num_shard_bits : 4 2022-09-10T04:18:13.335 INFO:journalctl@ceph.mon.a.smithi081.stdout: strict_capacity_limit : 0 2022-09-10T04:18:13.335 INFO:journalctl@ceph.mon.a.smithi081.stdout: high_pri_pool_ratio: 0.000 2022-09-10T04:18:13.336 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_compressed: (nil) 2022-09-10T04:18:13.336 INFO:journalctl@ceph.mon.a.smithi081.stdout: persistent_cache: (nil) 2022-09-10T04:18:13.336 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_size: 4096 2022-09-10T04:18:13.336 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_size_deviation: 10 2022-09-10T04:18:13.336 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_restart_interval: 16 2022-09-10T04:18:13.337 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_block_restart_interval: 1 2022-09-10T04:18:13.337 INFO:journalctl@ceph.mon.a.smithi081.stdout: metadata_block_size: 4096 2022-09-10T04:18:13.337 INFO:journalctl@ceph.mon.a.smithi081.stdout: partition_filters: 0 2022-09-10T04:18:13.337 INFO:journalctl@ceph.mon.a.smithi081.stdout: use_delta_encoding: 1 2022-09-10T04:18:13.337 INFO:journalctl@ceph.mon.a.smithi081.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-09-10T04:18:13.338 INFO:journalctl@ceph.mon.a.smithi081.stdout: whole_key_filtering: 1 2022-09-10T04:18:13.338 INFO:journalctl@ceph.mon.a.smithi081.stdout: verify_compression: 0 2022-09-10T04:18:13.338 INFO:journalctl@ceph.mon.a.smithi081.stdout: read_amp_bytes_per_bit: 0 2022-09-10T04:18:13.338 INFO:journalctl@ceph.mon.a.smithi081.stdout: format_version: 4 2022-09-10T04:18:13.339 INFO:journalctl@ceph.mon.a.smithi081.stdout: enable_index_compression: 1 2022-09-10T04:18:13.339 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_align: 0 2022-09-10T04:18:13.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.write_buffer_size: 33554432 2022-09-10T04:18:13.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_write_buffer_number: 2 2022-09-10T04:18:13.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression: NoCompression 2022-09-10T04:18:13.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression: Disabled 2022-09-10T04:18:13.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.prefix_extractor: nullptr 2022-09-10T04:18:13.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-09-10T04:18:13.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.num_levels: 7 2022-09-10T04:18:13.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-09-10T04:18:13.341 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-09-10T04:18:13.341 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-09-10T04:18:13.341 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-09-10T04:18:13.342 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-09-10T04:18:13.342 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-09-10T04:18:13.342 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-09-10T04:18:13.342 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-09-10T04:18:13.342 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-09-10T04:18:13.343 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-09-10T04:18:13.343 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.window_bits: -14 2022-09-10T04:18:13.343 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.level: 32767 2022-09-10T04:18:13.343 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.strategy: 0 2022-09-10T04:18:13.344 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-09-10T04:18:13.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-09-10T04:18:13.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-09-10T04:18:13.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compression_opts.enabled: false 2022-09-10T04:18:13.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-09-10T04:18:13.346 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-09-10T04:18:13.346 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-09-10T04:18:13.346 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.target_file_size_base: 67108864 2022-09-10T04:18:13.346 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.target_file_size_multiplier: 1 2022-09-10T04:18:13.346 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-09-10T04:18:13.347 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-09-10T04:18:13.347 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-09-10T04:18:13.347 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-09-10T04:18:13.347 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-09-10T04:18:13.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-09-10T04:18:13.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-09-10T04:18:13.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-09-10T04:18:13.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-09-10T04:18:13.348 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-09-10T04:18:13.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-09-10T04:18:13.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-09-10T04:18:13.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.arena_block_size: 4194304 2022-09-10T04:18:13.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-09-10T04:18:13.350 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-09-10T04:18:13.350 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-09-10T04:18:13.350 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.disable_auto_compactions: 0 2022-09-10T04:18:13.350 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-09-10T04:18:13.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-09-10T04:18:13.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-09-10T04:18:13.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-09-10T04:18:13.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-09-10T04:18:13.352 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-09-10T04:18:13.352 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-09-10T04:18:13.352 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-09-10T04:18:13.352 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-09-10T04:18:13.352 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-09-10T04:18:13.353 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.table_properties_collectors: 2022-09-10T04:18:13.353 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.inplace_update_support: 0 2022-09-10T04:18:13.353 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.inplace_update_num_locks: 10000 2022-09-10T04:18:13.353 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-09-10T04:18:13.354 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-09-10T04:18:13.354 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.memtable_huge_page_size: 0 2022-09-10T04:18:13.354 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.bloom_locality: 0 2022-09-10T04:18:13.354 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.max_successive_merges: 0 2022-09-10T04:18:13.354 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.optimize_filters_for_hits: 0 2022-09-10T04:18:13.355 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.paranoid_file_checks: 0 2022-09-10T04:18:13.355 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.force_consistency_checks: 1 2022-09-10T04:18:13.355 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.report_bg_io_stats: 0 2022-09-10T04:18:13.355 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.ttl: 2592000 2022-09-10T04:18:13.355 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.periodic_compaction_seconds: 0 2022-09-10T04:18:13.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.enable_blob_files: false 2022-09-10T04:18:13.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.min_blob_size: 0 2022-09-10T04:18:13.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.blob_file_size: 268435456 2022-09-10T04:18:13.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.blob_compression_type: NoCompression 2022-09-10T04:18:13.356 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.enable_blob_garbage_collection: false 2022-09-10T04:18:13.357 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-09-10T04:18:13.357 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-09-10T04:18:13.357 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2022-09-10T04:18:13.357 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2022-09-10T04:18:13.357 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783493016970, "job": 1, "event": "recovery_started", "wal_files": [10]} 2022-09-10T04:18:13.358 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2022-09-10T04:18:13.358 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-09-10T04:18:13.358 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783493018948, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 91863, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 89961, "index_size": 262, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13394, "raw_average_key_size": 49, "raw_value_size": 82612, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1662783493, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "038fb528-c196-467b-85df-e653beda432d", "db_session_id": "71F0GIKT0XCZF3IUYYZF"}} 2022-09-10T04:18:13.359 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2022-09-10T04:18:13.359 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783493019731, "job": 1, "event": "recovery_finished"} 2022-09-10T04:18:13.359 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-09-10T04:18:13.359 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x559517c33880 2022-09-10T04:18:13.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: DB pointer 0x559517ca6000 2022-09-10T04:18:13.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2022-09-10T04:18:13.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-09-10T04:18:13.360 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** DB Stats ** 2022-09-10T04:18:13.361 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:18:13.361 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-09-10T04:18:13.361 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-09-10T04:18:13.361 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-10T04:18:13.361 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-09-10T04:18:13.362 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-09-10T04:18:13.362 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-10T04:18:13.362 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.362 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2022-09-10T04:18:13.362 INFO:journalctl@ceph.mon.a.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:18:13.363 INFO:journalctl@ceph.mon.a.smithi081.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:18:13.363 INFO:journalctl@ceph.mon.a.smithi081.stdout: L0 2/0 91.47 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.363 INFO:journalctl@ceph.mon.a.smithi081.stdout: Sum 2/0 91.47 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.363 INFO:journalctl@ceph.mon.a.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.364 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.364 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2022-09-10T04:18:13.364 INFO:journalctl@ceph.mon.a.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:18:13.364 INFO:journalctl@ceph.mon.a.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:18:13.364 INFO:journalctl@ceph.mon.a.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.365 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:18:13.365 INFO:journalctl@ceph.mon.a.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-10T04:18:13.365 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-10T04:18:13.365 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-10T04:18:13.366 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-10T04:18:13.366 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-10T04:18:13.366 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative compaction: 0.00 GB write, 9.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:18:13.366 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval compaction: 0.00 GB write, 9.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:18:13.366 INFO:journalctl@ceph.mon.a.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-10T04:18:13.367 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.367 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-10T04:18:13.367 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.367 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2022-09-10T04:18:13.368 INFO:journalctl@ceph.mon.a.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:18:13.368 INFO:journalctl@ceph.mon.a.smithi081.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:18:13.368 INFO:journalctl@ceph.mon.a.smithi081.stdout: L0 2/0 91.47 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.369 INFO:journalctl@ceph.mon.a.smithi081.stdout: Sum 2/0 91.47 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.369 INFO:journalctl@ceph.mon.a.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-09-10T04:18:13.369 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.369 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2022-09-10T04:18:13.369 INFO:journalctl@ceph.mon.a.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:18:13.370 INFO:journalctl@ceph.mon.a.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:18:13.370 INFO:journalctl@ceph.mon.a.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 63.5 0.00 0.00 1 0.001 0 0 2022-09-10T04:18:13.370 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:18:13.370 INFO:journalctl@ceph.mon.a.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-10T04:18:13.371 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-10T04:18:13.371 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-10T04:18:13.371 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-10T04:18:13.371 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-10T04:18:13.371 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative compaction: 0.00 GB write, 9.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:18:13.372 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:18:13.372 INFO:journalctl@ceph.mon.a.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-10T04:18:13.372 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.372 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-10T04:18:13.372 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: starting mon.a rank 0 at public addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] at bind addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:13.373 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???) e1 preinit fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:13.373 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).mds e1 new map 2022-09-10T04:18:13.373 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).mds e1 print_map 2022-09-10T04:18:13.373 INFO:journalctl@ceph.mon.a.smithi081.stdout: e1 2022-09-10T04:18:13.374 INFO:journalctl@ceph.mon.a.smithi081.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-09-10T04:18:13.374 INFO:journalctl@ceph.mon.a.smithi081.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-10T04:18:13.374 INFO:journalctl@ceph.mon.a.smithi081.stdout: legacy client fscid: -1 2022-09-10T04:18:13.374 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2022-09-10T04:18:13.374 INFO:journalctl@ceph.mon.a.smithi081.stdout: No filesystems configured 2022-09-10T04:18:13.375 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2022-09-10T04:18:13.375 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:18:13.375 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:18:13.375 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:18:13.375 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2022-09-10T04:18:13.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 systemd[1]: Started Ceph mon.a for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:18:13.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mon.a is new leader, mons a in quorum (ranks 0) 2022-09-10T04:18:13.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: monmap e1: 1 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0]} 2022-09-10T04:18:13.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: fsmap 2022-09-10T04:18:13.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: osdmap e1: 0 total, 0 up, 0 in 2022-09-10T04:18:13.377 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:13 smithi081 ceph-mon[90133]: mgrmap e1: no daemons active 2022-09-10T04:18:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:14 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4258818449' entity='client.admin' 2022-09-10T04:18:16.058 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.632+0000 7fd6be2ee700 1 Processor -- start 2022-09-10T04:18:16.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.632+0000 7fd6be2ee700 1 -- start start 2022-09-10T04:18:16.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.633+0000 7fd6be2ee700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b80782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:16.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.633+0000 7fd6be2ee700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80787e0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.633+0000 7fd6b7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b80782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:16.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.634+0000 7fd6b7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b80782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48914/0 (socket says 172.21.15.81:48914) 2022-09-10T04:18:16.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.634+0000 7fd6b7fff700 1 -- 172.21.15.81:0/1366164295 learned_addr learned my addr 172.21.15.81:0/1366164295 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:16.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.634+0000 7fd6b7fff700 1 -- 172.21.15.81:0/1366164295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b8078920 con 0x7fd6b8079dc0 2022-09-10T04:18:16.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.634+0000 7fd6b7fff700 1 --2- 172.21.15.81:0/1366164295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b80782a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009700 tx=0x7fd6a8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba713f51d84e115 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:16.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.634+0000 7fd6b6ffd700 1 -- 172.21.15.81:0/1366164295 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6a8004160 con 0x7fd6b8079dc0 2022-09-10T04:18:16.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6b6ffd700 1 -- 172.21.15.81:0/1366164295 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fd6a80042c0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6b6ffd700 1 -- 172.21.15.81:0/1366164295 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6a80039d0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 -- 172.21.15.81:0/1366164295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 msgr2=0x7fd6b80782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:16.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 --2- 172.21.15.81:0/1366164295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b80782a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009700 tx=0x7fd6a8009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:16.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 -- 172.21.15.81:0/1366164295 shutdown_connections 2022-09-10T04:18:16.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 --2- 172.21.15.81:0/1366164295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b80782a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:16.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 -- 172.21.15.81:0/1366164295 >> 172.21.15.81:0/1366164295 conn(0x7fd6b80fe9f0 msgr2=0x7fd6b8100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:16.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 -- 172.21.15.81:0/1366164295 shutdown_connections 2022-09-10T04:18:16.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.635+0000 7fd6be2ee700 1 -- 172.21.15.81:0/1366164295 wait complete. 2022-09-10T04:18:16.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6be2ee700 1 Processor -- start 2022-09-10T04:18:16.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6be2ee700 1 -- start start 2022-09-10T04:18:16.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6be2ee700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b8199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:16.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6be2ee700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b81995d0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6b7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b8199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:16.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6b7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b8199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48926/0 (socket says 172.21.15.81:48926) 2022-09-10T04:18:16.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.637+0000 7fd6b7fff700 1 -- 172.21.15.81:0/4258818449 learned_addr learned my addr 172.21.15.81:0/4258818449 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:16.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.638+0000 7fd6b7fff700 1 -- 172.21.15.81:0/4258818449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80093e0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.638+0000 7fd6b7fff700 1 --2- 172.21.15.81:0/4258818449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b8199090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8003fb0 tx=0x7fd6a8004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:16.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.638+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6a80118c0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.638+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fd6a8011a20 con 0x7fd6b8079dc0 2022-09-10T04:18:16.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.638+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b81997d0 con 0x7fd6b8079dc0 2022-09-10T04:18:16.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.638+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd6a8019550 con 0x7fd6b8079dc0 2022-09-10T04:18:16.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.639+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b8199c00 con 0x7fd6b8079dc0 2022-09-10T04:18:16.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.639+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fd6a802a020 con 0x7fd6b8079dc0 2022-09-10T04:18:16.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.639+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd6a802d440 con 0x7fd6b8079dc0 2022-09-10T04:18:16.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.640+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b8060310 con 0x7fd6b8079dc0 2022-09-10T04:18:16.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.643+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7fd6a8011b90 con 0x7fd6b8079dc0 2022-09-10T04:18:16.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.712+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd6b819cb30 con 0x7fd6b8079dc0 2022-09-10T04:18:16.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.715+0000 7fd6b57fa700 1 -- 172.21.15.81:0/4258818449 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd6a801a480 con 0x7fd6b8079dc0 2022-09-10T04:18:16.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 msgr2=0x7fd6b8199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:16.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 --2- 172.21.15.81:0/4258818449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b8199090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8003fb0 tx=0x7fd6a8004090 comp rx=0 tx=0).stop 2022-09-10T04:18:16.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 shutdown_connections 2022-09-10T04:18:16.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 --2- 172.21.15.81:0/4258818449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6b8079dc0 0x7fd6b8199090 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:16.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 >> 172.21.15.81:0/4258818449 conn(0x7fd6b80fe9f0 msgr2=0x7fd6b80ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:16.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 shutdown_connections 2022-09-10T04:18:16.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:13.716+0000 7fd6be2ee700 1 -- 172.21.15.81:0/4258818449 wait complete. 2022-09-10T04:18:16.075 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote config to /etc/ceph/ceph.conf 2022-09-10T04:18:16.075 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2022-09-10T04:18:16.076 INFO:teuthology.orchestra.run.smithi081.stdout:Creating mgr... 2022-09-10T04:18:16.076 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying port 9283 ... 2022-09-10T04:18:16.076 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying port 8765 ... 2022-09-10T04:18:16.451 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.x 2022-09-10T04:18:16.451 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to reset failed state of unit ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.x.service: Unit ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.x.service not loaded. 2022-09-10T04:18:16.716 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-70cf895c-30bf-11ed-8431-001a4aab830c.target.wants/ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-70cf895c-30bf-11ed-8431-001a4aab830c@.service. 2022-09-10T04:18:17.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:16 smithi081 systemd[1]: Starting Ceph mgr.x for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:18:17.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:17 smithi081 podman[90364]: 2022-09-10T04:18:17.633 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-10T04:18:17.633 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2022-09-10T04:18:17.654 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-10T04:18:17.654 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2022-09-10T04:18:17.654 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2022-09-10T04:18:17.655 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to enable service . firewalld.service is not available 2022-09-10T04:18:17.677 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-10T04:18:17.678 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2022-09-10T04:18:17.698 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-10T04:18:17.699 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2022-09-10T04:18:17.699 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2022-09-10T04:18:17.699 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2022-09-10T04:18:17.700 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr to start... 2022-09-10T04:18:17.700 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr... 2022-09-10T04:18:17.889 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:17 smithi081 bash[90364]: 36452bba90b3f826ffd386b278dae17f9a7bf30b98b41e393d3622fbadcf7a8e 2022-09-10T04:18:17.890 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:17 smithi081 systemd[1]: Started Ceph mgr.x for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:18:17.890 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:17.773+0000 7f5fac7f5080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:18:17.890 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:17.887+0000 7f5fac7f5080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:18:18.168 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:18.025+0000 7f5fac7f5080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:18:18.419 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:18.166+0000 7f5fac7f5080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:18:18.419 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:18.417+0000 7f5fac7f5080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:18:18.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:18.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:18:18.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:18:18.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2022-09-10T04:18:18.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-10T04:18:18.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-10T04:18:18.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-10T04:18:18.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-10T04:18:18.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-10T04:18:18.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2022-09-10T04:18:18.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:18.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-10T04:18:18.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2022-09-10T04:18:18.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:18.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2022-09-10T04:18:18.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-10T04:18:18.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:18.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-10T04:18:18.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-10T04:18:18.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-10T04:18:18.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:18.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-10T04:18:18.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-10T04:18:18.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-10T04:18:18.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-10T04:18:18.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-10T04:18:18.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-10T04:18:18.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-10T04:18:18.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-10T04:18:18.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-10T04:18:18.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-10T04:18:18.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-10T04:18:18.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-10T04:18:18.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-10T04:18:18.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-10T04:18:18.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-10T04:18:18.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-10T04:18:18.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:18.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-10T04:18:18.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-10T04:18:18.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-10T04:18:18.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2022-09-10T04:18:18.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-10T04:18:18.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-10T04:18:18.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-10T04:18:18.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-10T04:18:18.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2022-09-10T04:18:18.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:18.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-10T04:18:18.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-10T04:18:18.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:18.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2022-09-10T04:18:07.860491+0000", 2022-09-10T04:18:18.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-10T04:18:18.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:18.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-10T04:18:18.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:18:18.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.294+0000 7efdcc13a700 1 Processor -- start 2022-09-10T04:18:18.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.295+0000 7efdcc13a700 1 -- start start 2022-09-10T04:18:18.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.295+0000 7efdcc13a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc4105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:18.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.295+0000 7efdcc13a700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc41062b0 con 0x7efdc41058d0 2022-09-10T04:18:18.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.296+0000 7efdc9ed6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc4105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:18.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.296+0000 7efdc9ed6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc4105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48956/0 (socket says 172.21.15.81:48956) 2022-09-10T04:18:18.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.296+0000 7efdc9ed6700 1 -- 172.21.15.81:0/667229211 learned_addr learned my addr 172.21.15.81:0/667229211 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:18.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.296+0000 7efdc9ed6700 1 -- 172.21.15.81:0/667229211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc41063f0 con 0x7efdc41058d0 2022-09-10T04:18:18.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.296+0000 7efdc9ed6700 1 --2- 172.21.15.81:0/667229211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc4105ce0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009700 tx=0x7efdc0009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191481c9be583dc5 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:18.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdc8ed4700 1 -- 172.21.15.81:0/667229211 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdc0004160 con 0x7efdc41058d0 2022-09-10T04:18:18.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdc8ed4700 1 -- 172.21.15.81:0/667229211 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7efdc00042c0 con 0x7efdc41058d0 2022-09-10T04:18:18.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdc8ed4700 1 -- 172.21.15.81:0/667229211 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdc00039d0 con 0x7efdc41058d0 2022-09-10T04:18:18.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdcc13a700 1 -- 172.21.15.81:0/667229211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 msgr2=0x7efdc4105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:18.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdcc13a700 1 --2- 172.21.15.81:0/667229211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc4105ce0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efdc0009700 tx=0x7efdc0009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:18.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdcc13a700 1 -- 172.21.15.81:0/667229211 shutdown_connections 2022-09-10T04:18:18.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdcc13a700 1 --2- 172.21.15.81:0/667229211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc4105ce0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:18.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.297+0000 7efdcc13a700 1 -- 172.21.15.81:0/667229211 >> 172.21.15.81:0/667229211 conn(0x7efdc4078a20 msgr2=0x7efdc4078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:18.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.298+0000 7efdcc13a700 1 -- 172.21.15.81:0/667229211 shutdown_connections 2022-09-10T04:18:18.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.298+0000 7efdcc13a700 1 -- 172.21.15.81:0/667229211 wait complete. 2022-09-10T04:18:18.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.299+0000 7efdcc13a700 1 Processor -- start 2022-09-10T04:18:18.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.299+0000 7efdcc13a700 1 -- start start 2022-09-10T04:18:18.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.299+0000 7efdcc13a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc41990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:18.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.299+0000 7efdcc13a700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc4199600 con 0x7efdc41058d0 2022-09-10T04:18:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.300+0000 7efdc9ed6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc41990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.300+0000 7efdc9ed6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc41990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:48964/0 (socket says 172.21.15.81:48964) 2022-09-10T04:18:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.300+0000 7efdc9ed6700 1 -- 172.21.15.81:0/333171168 learned_addr learned my addr 172.21.15.81:0/333171168 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:18.833 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.300+0000 7efdc9ed6700 1 -- 172.21.15.81:0/333171168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00093e0 con 0x7efdc41058d0 2022-09-10T04:18:18.834 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.300+0000 7efdc9ed6700 1 --2- 172.21.15.81:0/333171168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc41990c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efdc0003f10 tx=0x7efdc0003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:18.834 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdc0011950 con 0x7efdc41058d0 2022-09-10T04:18:18.834 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc4199800 con 0x7efdc41058d0 2022-09-10T04:18:18.834 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7efdc0011ab0 con 0x7efdc41058d0 2022-09-10T04:18:18.834 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdc0019550 con 0x7efdc41058d0 2022-09-10T04:18:18.835 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc4199c30 con 0x7efdc41058d0 2022-09-10T04:18:18.835 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7efdc002a020 con 0x7efdc41058d0 2022-09-10T04:18:18.835 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.301+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efdc002d780 con 0x7efdc41058d0 2022-09-10T04:18:18.835 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.302+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc41074e0 con 0x7efdc41058d0 2022-09-10T04:18:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.305+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7efdc0011c20 con 0x7efdc41058d0 2022-09-10T04:18:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.380+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efdc419ca70 con 0x7efdc41058d0 2022-09-10T04:18:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.381+0000 7efdb2ffd700 1 -- 172.21.15.81:0/333171168 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7efdc0019820 con 0x7efdc41058d0 2022-09-10T04:18:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 msgr2=0x7efdc41990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:18.836 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 --2- 172.21.15.81:0/333171168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc41990c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efdc0003f10 tx=0x7efdc0003ff0 comp rx=0 tx=0).stop 2022-09-10T04:18:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 shutdown_connections 2022-09-10T04:18:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 --2- 172.21.15.81:0/333171168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efdc41058d0 0x7efdc41990c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 >> 172.21.15.81:0/333171168 conn(0x7efdc4078a20 msgr2=0x7efdc4103780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:18.837 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 shutdown_connections 2022-09-10T04:18:18.838 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:18.382+0000 7efdcc13a700 1 -- 172.21.15.81:0/333171168 wait complete. 2022-09-10T04:18:18.838 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (1/15)... 2022-09-10T04:18:19.022 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:18.892+0000 7f5fac7f5080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:18:19.022 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:19.020+0000 7f5fac7f5080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:18:19.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:19 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/333171168' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-10T04:18:19.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:19.197+0000 7f5fac7f5080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:18:19.805 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:19.334+0000 7f5fac7f5080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:18:20.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:19.936+0000 7f5fac7f5080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:18:21.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:20.853+0000 7f5fac7f5080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:18:21.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:21.093+0000 7f5fac7f5080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:18:22.556 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:22.171+0000 7f5fac7f5080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:18:22.557 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:22.319+0000 7f5fac7f5080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:18:22.914 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:22.599+0000 7f5fac7f5080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:18:23.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:22.912+0000 7f5fac7f5080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:18:23.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:23.145+0000 7f5fac7f5080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:18:24.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:23.751+0000 7f5fac7f5080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:18:25.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:24.707+0000 7f5fac7f5080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:18:25.472 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:25.142+0000 7f5fac7f5080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:18:25.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:25.470+0000 7f5fac7f5080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:18:25.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:25.671+0000 7f5fac7f5080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:18:26.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:26.053+0000 7f5fac7f5080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:18:26.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:26 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1635267189' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-10T04:18:26.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:26.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:18:26.527 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:18:26.527 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2022-09-10T04:18:26.527 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-10T04:18:26.527 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-10T04:18:26.528 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-10T04:18:26.528 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.528 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-10T04:18:26.528 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-10T04:18:26.529 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2022-09-10T04:18:26.529 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:26.529 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-10T04:18:26.529 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2022-09-10T04:18:26.529 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:26.530 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2022-09-10T04:18:26.530 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-10T04:18:26.530 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:26.531 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-10T04:18:26.531 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-10T04:18:26.531 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.532 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-10T04:18:26.532 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:26.532 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-10T04:18:26.533 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-10T04:18:26.533 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-10T04:18:26.543 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-10T04:18:26.544 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-10T04:18:26.544 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-10T04:18:26.544 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.544 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-10T04:18:26.545 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-10T04:18:26.545 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-10T04:18:26.545 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-10T04:18:26.546 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-10T04:18:26.546 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-10T04:18:26.546 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-10T04:18:26.546 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-10T04:18:26.546 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-10T04:18:26.547 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.547 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-10T04:18:26.547 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:26.547 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-10T04:18:26.548 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-10T04:18:26.548 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.548 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-10T04:18:26.548 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2022-09-10T04:18:26.549 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-10T04:18:26.549 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-10T04:18:26.549 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-10T04:18:26.549 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-10T04:18:26.549 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2022-09-10T04:18:26.550 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:26.550 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-10T04:18:26.550 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.550 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-10T04:18:26.551 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:26.551 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2022-09-10T04:18:07.860491+0000", 2022-09-10T04:18:26.551 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-10T04:18:26.551 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:26.552 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-10T04:18:26.552 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:18:26.552 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.935+0000 7f5d6e145700 1 Processor -- start 2022-09-10T04:18:26.552 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.936+0000 7f5d6e145700 1 -- start start 2022-09-10T04:18:26.552 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.936+0000 7f5d6e145700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 0x7f5d68108360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:26.553 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.936+0000 7f5d6e145700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d68103de0 con 0x7f5d68077ce0 2022-09-10T04:18:26.553 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.937+0000 7f5d677fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 0x7f5d68108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:26.553 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.937+0000 7f5d677fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 0x7f5d68108360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43772/0 (socket says 172.21.15.81:43772) 2022-09-10T04:18:26.553 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.937+0000 7f5d677fe700 1 -- 172.21.15.81:0/562045979 learned_addr learned my addr 172.21.15.81:0/562045979 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:26.554 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.937+0000 7f5d677fe700 1 -- 172.21.15.81:0/562045979 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d681088a0 con 0x7f5d68077ce0 2022-09-10T04:18:26.554 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.937+0000 7f5d677fe700 1 --2- 172.21.15.81:0/562045979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 0x7f5d68108360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009700 tx=0x7f5d58009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb12dc06ba0dfc1b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:26.554 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d667fc700 1 -- 172.21.15.81:0/562045979 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d58004160 con 0x7f5d68077ce0 2022-09-10T04:18:26.554 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d667fc700 1 -- 172.21.15.81:0/562045979 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f5d580042c0 con 0x7f5d68077ce0 2022-09-10T04:18:26.555 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d667fc700 1 -- 172.21.15.81:0/562045979 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d580039d0 con 0x7f5d68077ce0 2022-09-10T04:18:26.555 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d6e145700 1 -- 172.21.15.81:0/562045979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 msgr2=0x7f5d68108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:26.555 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d6e145700 1 --2- 172.21.15.81:0/562045979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 0x7f5d68108360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009700 tx=0x7f5d58009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:26.555 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d6e145700 1 -- 172.21.15.81:0/562045979 shutdown_connections 2022-09-10T04:18:26.556 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d6e145700 1 --2- 172.21.15.81:0/562045979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68077ce0 0x7f5d68108360 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:26.556 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.938+0000 7f5d6e145700 1 -- 172.21.15.81:0/562045979 >> 172.21.15.81:0/562045979 conn(0x7f5d680fd8f0 msgr2=0x7f5d680ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:26.556 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.939+0000 7f5d6e145700 1 -- 172.21.15.81:0/562045979 shutdown_connections 2022-09-10T04:18:26.556 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.939+0000 7f5d6e145700 1 -- 172.21.15.81:0/562045979 wait complete. 2022-09-10T04:18:26.557 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.940+0000 7f5d6e145700 1 Processor -- start 2022-09-10T04:18:26.557 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.941+0000 7f5d6e145700 1 -- start start 2022-09-10T04:18:26.557 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.941+0000 7f5d6e145700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 0x7f5d681994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:26.557 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.941+0000 7f5d6e145700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d681999e0 con 0x7f5d68199090 2022-09-10T04:18:26.558 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.941+0000 7f5d677fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 0x7f5d681994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:26.558 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.941+0000 7f5d677fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 0x7f5d681994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43786/0 (socket says 172.21.15.81:43786) 2022-09-10T04:18:26.558 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.941+0000 7f5d677fe700 1 -- 172.21.15.81:0/1635267189 learned_addr learned my addr 172.21.15.81:0/1635267189 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:26.558 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.942+0000 7f5d677fe700 1 -- 172.21.15.81:0/1635267189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d580093e0 con 0x7f5d68199090 2022-09-10T04:18:26.559 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.942+0000 7f5d677fe700 1 --2- 172.21.15.81:0/1635267189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 0x7f5d681994a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009e90 tx=0x7f5d58004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:26.559 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.942+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d58011840 con 0x7f5d68199090 2022-09-10T04:18:26.559 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.942+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f5d5801a430 con 0x7f5d68199090 2022-09-10T04:18:26.559 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.942+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d58019550 con 0x7f5d68199090 2022-09-10T04:18:26.560 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.942+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d68199be0 con 0x7f5d68199090 2022-09-10T04:18:26.560 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.943+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d6819c840 con 0x7f5d68199090 2022-09-10T04:18:26.560 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.943+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f5d5802a020 con 0x7f5d68199090 2022-09-10T04:18:26.560 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.943+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5d5802d780 con 0x7f5d68199090 2022-09-10T04:18:26.560 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.944+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d68060310 con 0x7f5d68199090 2022-09-10T04:18:26.561 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:25.946+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70815 (secure 0 0 0) 0x7f5d5801a880 con 0x7f5d68199090 2022-09-10T04:18:26.561 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.021+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5d68077ce0 con 0x7f5d68199090 2022-09-10T04:18:26.561 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.022+0000 7f5d64ff9700 1 -- 172.21.15.81:0/1635267189 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (secure 0 0 0) 0x7f5d68077ce0 con 0x7f5d68199090 2022-09-10T04:18:26.561 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 msgr2=0x7f5d681994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:26.562 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 --2- 172.21.15.81:0/1635267189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 0x7f5d681994a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009e90 tx=0x7f5d58004040 comp rx=0 tx=0).stop 2022-09-10T04:18:26.562 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 shutdown_connections 2022-09-10T04:18:26.562 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 --2- 172.21.15.81:0/1635267189 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d68199090 0x7f5d681994a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:26.562 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 >> 172.21.15.81:0/1635267189 conn(0x7f5d680fd8f0 msgr2=0x7f5d680fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:26.563 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 shutdown_connections 2022-09-10T04:18:26.563 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:26.023+0000 7f5d6e145700 1 -- 172.21.15.81:0/1635267189 wait complete. 2022-09-10T04:18:26.563 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (2/15)... 2022-09-10T04:18:27.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:27.089+0000 7f5fac7f5080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:18:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: Activating manager daemon x 2022-09-10T04:18:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: mgrmap e2: x(active, starting, since 0.00441058s) 2022-09-10T04:18:27.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-10T04:18:27.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-10T04:18:27.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-10T04:18:27.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:18:27.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-10T04:18:27.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: Manager daemon x is now available 2022-09-10T04:18:27.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:18:27.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:18:27.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' 2022-09-10T04:18:27.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' 2022-09-10T04:18:27.559 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:27 smithi081 ceph-mon[90133]: from='mgr.14100 172.21.15.81:0/256295000' entity='mgr.x' 2022-09-10T04:18:29.450 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:29 smithi081 ceph-mon[90133]: mgrmap e3: x(active, since 1.01161s) 2022-09-10T04:18:29.834 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:29.852 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:18:29.853 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:18:29.853 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2022-09-10T04:18:29.853 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2022-09-10T04:18:29.853 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2022-09-10T04:18:29.853 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2022-09-10T04:18:29.854 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.854 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2022-09-10T04:18:29.854 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2022-09-10T04:18:29.854 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2022-09-10T04:18:29.855 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:29.855 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2022-09-10T04:18:29.855 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2022-09-10T04:18:29.855 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:29.855 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2022-09-10T04:18:29.856 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2022-09-10T04:18:29.856 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:29.856 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2022-09-10T04:18:29.856 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2022-09-10T04:18:29.857 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.857 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2022-09-10T04:18:29.857 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:29.857 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2022-09-10T04:18:29.857 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2022-09-10T04:18:29.858 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2022-09-10T04:18:29.858 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2022-09-10T04:18:29.858 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2022-09-10T04:18:29.858 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2022-09-10T04:18:29.859 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.859 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2022-09-10T04:18:29.859 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2022-09-10T04:18:29.859 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2022-09-10T04:18:29.859 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2022-09-10T04:18:29.860 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2022-09-10T04:18:29.860 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2022-09-10T04:18:29.860 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2022-09-10T04:18:29.860 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2022-09-10T04:18:29.860 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2022-09-10T04:18:29.861 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.861 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2022-09-10T04:18:29.861 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:29.861 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2022-09-10T04:18:29.861 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2022-09-10T04:18:29.862 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.862 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2022-09-10T04:18:29.862 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2022-09-10T04:18:29.862 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2022-09-10T04:18:29.863 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2022-09-10T04:18:29.863 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2022-09-10T04:18:29.863 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2022-09-10T04:18:29.863 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2022-09-10T04:18:29.863 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2022-09-10T04:18:29.864 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-10T04:18:29.864 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.864 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2022-09-10T04:18:29.864 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2022-09-10T04:18:29.864 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2022-09-10T04:18:07.860491+0000", 2022-09-10T04:18:29.865 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2022-09-10T04:18:29.865 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2022-09-10T04:18:29.865 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2022-09-10T04:18:29.865 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:18:29.865 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.122+0000 7f403810c700 1 Processor -- start 2022-09-10T04:18:29.866 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.122+0000 7f403810c700 1 -- start start 2022-09-10T04:18:29.866 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.123+0000 7f403810c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f40300782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:29.866 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.123+0000 7f403810c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40300787e0 con 0x7f4030079dc0 2022-09-10T04:18:29.866 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.123+0000 7f4035ea8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f40300782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:29.867 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.123+0000 7f4035ea8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f40300782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43854/0 (socket says 172.21.15.81:43854) 2022-09-10T04:18:29.867 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.123+0000 7f4035ea8700 1 -- 172.21.15.81:0/2298441405 learned_addr learned my addr 172.21.15.81:0/2298441405 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:29.867 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.124+0000 7f4035ea8700 1 -- 172.21.15.81:0/2298441405 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030078920 con 0x7f4030079dc0 2022-09-10T04:18:29.867 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.124+0000 7f4035ea8700 1 --2- 172.21.15.81:0/2298441405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f40300782a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f402c009700 tx=0x7f402c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f4d3e4cd3634fa server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:29.867 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.124+0000 7f4034ea6700 1 -- 172.21.15.81:0/2298441405 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c004160 con 0x7f4030079dc0 2022-09-10T04:18:29.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.124+0000 7f4034ea6700 1 -- 172.21.15.81:0/2298441405 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f402c0042c0 con 0x7f4030079dc0 2022-09-10T04:18:29.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.124+0000 7f4034ea6700 1 -- 172.21.15.81:0/2298441405 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c0039d0 con 0x7f4030079dc0 2022-09-10T04:18:29.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 -- 172.21.15.81:0/2298441405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 msgr2=0x7f40300782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:29.868 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 --2- 172.21.15.81:0/2298441405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f40300782a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f402c009700 tx=0x7f402c009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:29.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 -- 172.21.15.81:0/2298441405 shutdown_connections 2022-09-10T04:18:29.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 --2- 172.21.15.81:0/2298441405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f40300782a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:29.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 -- 172.21.15.81:0/2298441405 >> 172.21.15.81:0/2298441405 conn(0x7f40300fe9f0 msgr2=0x7f4030100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:29.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 -- 172.21.15.81:0/2298441405 shutdown_connections 2022-09-10T04:18:29.869 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.125+0000 7f403810c700 1 -- 172.21.15.81:0/2298441405 wait complete. 2022-09-10T04:18:29.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.126+0000 7f403810c700 1 Processor -- start 2022-09-10T04:18:29.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.126+0000 7f403810c700 1 -- start start 2022-09-10T04:18:29.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.127+0000 7f403810c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f4030199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:29.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.127+0000 7f403810c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40301995d0 con 0x7f4030079dc0 2022-09-10T04:18:29.870 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.127+0000 7f4035ea8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f4030199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:29.871 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.127+0000 7f4035ea8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f4030199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43864/0 (socket says 172.21.15.81:43864) 2022-09-10T04:18:29.871 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.127+0000 7f4035ea8700 1 -- 172.21.15.81:0/1799157347 learned_addr learned my addr 172.21.15.81:0/1799157347 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:29.871 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.127+0000 7f4035ea8700 1 -- 172.21.15.81:0/1799157347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402c0093e0 con 0x7f4030079dc0 2022-09-10T04:18:29.871 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.128+0000 7f4035ea8700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f4030199090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f402c003fb0 tx=0x7f402c004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:29.872 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.128+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c0118c0 con 0x7f4030079dc0 2022-09-10T04:18:29.872 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.128+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f402c011a20 con 0x7f4030079dc0 2022-09-10T04:18:29.872 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.128+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f402c019550 con 0x7f4030079dc0 2022-09-10T04:18:29.872 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.128+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40301997d0 con 0x7f4030079dc0 2022-09-10T04:18:29.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.128+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4030199c00 con 0x7f4030079dc0 2022-09-10T04:18:29.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.129+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) v1 ==== 42679+0+0 (secure 0 0 0) 0x7f402c02a020 con 0x7f4030079dc0 2022-09-10T04:18:29.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.129+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4030060310 con 0x7f4030079dc0 2022-09-10T04:18:29.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.130+0000 7f401effd700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4018035770 0x7f4018037c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:29.873 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.130+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f402c0489a0 con 0x7f4030079dc0 2022-09-10T04:18:29.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.130+0000 7f40356a7700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4018035770 0x7f4018037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:29.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.131+0000 7f40356a7700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4018035770 0x7f4018037c20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4020009750 tx=0x7f4020006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:29.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.136+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f402c018b40 con 0x7f4030079dc0 2022-09-10T04:18:29.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.417+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f40301096b0 con 0x7f4030079dc0 2022-09-10T04:18:29.874 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.417+0000 7f401effd700 1 -- 172.21.15.81:0/1799157347 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f402c018b40 con 0x7f4030079dc0 2022-09-10T04:18:29.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4018035770 msgr2=0x7f4018037c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:29.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4018035770 0x7f4018037c20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4020009750 tx=0x7f4020006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:29.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 msgr2=0x7f4030199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:29.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f4030199090 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f402c003fb0 tx=0x7f402c004090 comp rx=0 tx=0).stop 2022-09-10T04:18:29.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 shutdown_connections 2022-09-10T04:18:29.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4018035770 0x7f4018037c20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:29.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 --2- 172.21.15.81:0/1799157347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4030079dc0 0x7f4030199090 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:29.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 >> 172.21.15.81:0/1799157347 conn(0x7f40300fe9f0 msgr2=0x7f40300ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:29.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 shutdown_connections 2022-09-10T04:18:29.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:29.420+0000 7f403810c700 1 -- 172.21.15.81:0/1799157347 wait complete. 2022-09-10T04:18:29.877 INFO:teuthology.orchestra.run.smithi081.stdout:mgr is available 2022-09-10T04:18:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:30 smithi081 ceph-mon[90133]: mgrmap e4: x(active, since 2s) 2022-09-10T04:18:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:30 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1799157347' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2022-09-10T04:18:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:31 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4014354243' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2022-09-10T04:18:31.455 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:31.455 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [global] 2022-09-10T04:18:31.455 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout fsid = 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:31.456 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2022-09-10T04:18:31.456 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2022-09-10T04:18:31.456 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2022-09-10T04:18:31.456 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2022-09-10T04:18:31.457 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:31.457 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [mgr] 2022-09-10T04:18:31.457 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2022-09-10T04:18:31.458 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2022-09-10T04:18:31.458 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [osd] 2022-09-10T04:18:31.458 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2022-09-10T04:18:31.458 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2022-09-10T04:18:31.459 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2022-09-10T04:18:31.459 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2022-09-10T04:18:31.459 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2022-09-10T04:18:31.460 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.783+0000 7f7e19185700 1 Processor -- start 2022-09-10T04:18:31.460 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.784+0000 7f7e19185700 1 -- start start 2022-09-10T04:18:31.460 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.784+0000 7f7e19185700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e140782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:31.461 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.784+0000 7f7e19185700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e14078800 con 0x7f7e14079de0 2022-09-10T04:18:31.461 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.784+0000 7f7e12d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e140782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:31.461 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.784+0000 7f7e12d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e140782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59532/0 (socket says 172.21.15.81:59532) 2022-09-10T04:18:31.462 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.784+0000 7f7e12d9d700 1 -- 172.21.15.81:0/3638197177 learned_addr learned my addr 172.21.15.81:0/3638197177 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:31.462 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.785+0000 7f7e12d9d700 1 -- 172.21.15.81:0/3638197177 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14078940 con 0x7f7e14079de0 2022-09-10T04:18:31.462 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.785+0000 7f7e12d9d700 1 --2- 172.21.15.81:0/3638197177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e140782c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009700 tx=0x7f7e04009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0300e0ec5192360 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:31.462 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.785+0000 7f7e11d9b700 1 -- 172.21.15.81:0/3638197177 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e04004160 con 0x7f7e14079de0 2022-09-10T04:18:31.463 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.785+0000 7f7e11d9b700 1 -- 172.21.15.81:0/3638197177 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f7e040042c0 con 0x7f7e14079de0 2022-09-10T04:18:31.463 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.785+0000 7f7e11d9b700 1 -- 172.21.15.81:0/3638197177 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e040039d0 con 0x7f7e14079de0 2022-09-10T04:18:31.463 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 -- 172.21.15.81:0/3638197177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 msgr2=0x7f7e140782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:31.463 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 --2- 172.21.15.81:0/3638197177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e140782c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009700 tx=0x7f7e04009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:31.463 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 -- 172.21.15.81:0/3638197177 shutdown_connections 2022-09-10T04:18:31.464 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 --2- 172.21.15.81:0/3638197177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e140782c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:31.464 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 -- 172.21.15.81:0/3638197177 >> 172.21.15.81:0/3638197177 conn(0x7f7e140fe9f0 msgr2=0x7f7e14100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:31.464 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 -- 172.21.15.81:0/3638197177 shutdown_connections 2022-09-10T04:18:31.464 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.786+0000 7f7e19185700 1 -- 172.21.15.81:0/3638197177 wait complete. 2022-09-10T04:18:31.465 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.787+0000 7f7e19185700 1 Processor -- start 2022-09-10T04:18:31.465 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.788+0000 7f7e19185700 1 -- start start 2022-09-10T04:18:31.465 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.788+0000 7f7e19185700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e14199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:31.465 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.788+0000 7f7e19185700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e141995c0 con 0x7f7e14079de0 2022-09-10T04:18:31.466 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.788+0000 7f7e12d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e14199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:31.466 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.788+0000 7f7e12d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e14199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59546/0 (socket says 172.21.15.81:59546) 2022-09-10T04:18:31.466 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.788+0000 7f7e12d9d700 1 -- 172.21.15.81:0/4014354243 learned_addr learned my addr 172.21.15.81:0/4014354243 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:31.466 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7e12d9d700 1 -- 172.21.15.81:0/4014354243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e040093e0 con 0x7f7e14079de0 2022-09-10T04:18:31.467 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7e12d9d700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e14199080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7e04003fb0 tx=0x7f7e04004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:31.467 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e040118c0 con 0x7f7e14079de0 2022-09-10T04:18:31.468 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e141997c0 con 0x7f7e14079de0 2022-09-10T04:18:31.468 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f7e04011a20 con 0x7f7e14079de0 2022-09-10T04:18:31.468 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e04019550 con 0x7f7e14079de0 2022-09-10T04:18:31.468 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.789+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e14199bf0 con 0x7f7e14079de0 2022-09-10T04:18:31.469 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.790+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) v1 ==== 42679+0+0 (secure 0 0 0) 0x7f7e0402a020 con 0x7f7e14079de0 2022-09-10T04:18:31.469 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.790+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e14060310 con 0x7f7e14079de0 2022-09-10T04:18:31.469 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.791+0000 7f7dfbfff700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f7e000357d0 0x7f7e00037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:31.469 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.791+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7e04048990 con 0x7f7e14079de0 2022-09-10T04:18:31.470 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.791+0000 7f7e1259c700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f7e000357d0 0x7f7e00037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:31.471 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.792+0000 7f7e1259c700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f7e000357d0 0x7f7e00037c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009750 tx=0x7f7e08006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:31.471 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.796+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f7e04011b90 con 0x7f7e14079de0 2022-09-10T04:18:31.471 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:30.999+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7e1419cc00 con 0x7f7e14079de0 2022-09-10T04:18:31.471 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.000+0000 7f7dfbfff700 1 -- 172.21.15.81:0/4014354243 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f7e04018b40 con 0x7f7e14079de0 2022-09-10T04:18:31.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f7e000357d0 msgr2=0x7f7e00037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:31.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f7e000357d0 0x7f7e00037c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009750 tx=0x7f7e08006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:31.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 msgr2=0x7f7e14199080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:31.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e14199080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7e04003fb0 tx=0x7f7e04004090 comp rx=0 tx=0).stop 2022-09-10T04:18:31.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 shutdown_connections 2022-09-10T04:18:31.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f7e000357d0 0x7f7e00037c80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:31.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 --2- 172.21.15.81:0/4014354243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14079de0 0x7f7e14199080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:31.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 >> 172.21.15.81:0/4014354243 conn(0x7f7e140fe9f0 msgr2=0x7f7e140ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:31.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 shutdown_connections 2022-09-10T04:18:31.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.003+0000 7f7e19185700 1 -- 172.21.15.81:0/4014354243 wait complete. 2022-09-10T04:18:31.474 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling cephadm module... 2022-09-10T04:18:32.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:32 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3030807595' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2022-09-10T04:18:32.695 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: ignoring --setuser ceph since I am not root 2022-09-10T04:18:32.696 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: ignoring --setgroup ceph since I am not root 2022-09-10T04:18:32.696 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:32.693+0000 7fe8c6668080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:18:32.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.999+0000 7fed30774700 1 Processor -- start 2022-09-10T04:18:32.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:31.999+0000 7fed30774700 1 -- start start 2022-09-10T04:18:32.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.000+0000 7fed30774700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed280782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:32.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.000+0000 7fed30774700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed28078800 con 0x7fed28079de0 2022-09-10T04:18:32.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.000+0000 7fed2e510700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed280782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:32.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.000+0000 7fed2e510700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed280782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59548/0 (socket says 172.21.15.81:59548) 2022-09-10T04:18:32.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.000+0000 7fed2e510700 1 -- 172.21.15.81:0/40133329 learned_addr learned my addr 172.21.15.81:0/40133329 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:32.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.000+0000 7fed2e510700 1 -- 172.21.15.81:0/40133329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed28078940 con 0x7fed28079de0 2022-09-10T04:18:32.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.001+0000 7fed2e510700 1 --2- 172.21.15.81:0/40133329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed280782c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fed24009700 tx=0x7fed24009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a6de12ede09884 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:32.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.001+0000 7fed2d50e700 1 -- 172.21.15.81:0/40133329 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed24004160 con 0x7fed28079de0 2022-09-10T04:18:32.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.001+0000 7fed2d50e700 1 -- 172.21.15.81:0/40133329 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fed240042c0 con 0x7fed28079de0 2022-09-10T04:18:32.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.001+0000 7fed2d50e700 1 -- 172.21.15.81:0/40133329 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed240039d0 con 0x7fed28079de0 2022-09-10T04:18:32.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.001+0000 7fed30774700 1 -- 172.21.15.81:0/40133329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 msgr2=0x7fed280782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:32.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.001+0000 7fed30774700 1 --2- 172.21.15.81:0/40133329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed280782c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fed24009700 tx=0x7fed24009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:32.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.002+0000 7fed30774700 1 -- 172.21.15.81:0/40133329 shutdown_connections 2022-09-10T04:18:32.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.002+0000 7fed30774700 1 --2- 172.21.15.81:0/40133329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed280782c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:32.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.002+0000 7fed30774700 1 -- 172.21.15.81:0/40133329 >> 172.21.15.81:0/40133329 conn(0x7fed280fe9f0 msgr2=0x7fed28100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:32.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.002+0000 7fed30774700 1 -- 172.21.15.81:0/40133329 shutdown_connections 2022-09-10T04:18:32.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.002+0000 7fed30774700 1 -- 172.21.15.81:0/40133329 wait complete. 2022-09-10T04:18:32.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.003+0000 7fed30774700 1 Processor -- start 2022-09-10T04:18:32.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.004+0000 7fed30774700 1 -- start start 2022-09-10T04:18:32.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.004+0000 7fed30774700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed281990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:32.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.004+0000 7fed30774700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed28199610 con 0x7fed28079de0 2022-09-10T04:18:32.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.004+0000 7fed2e510700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed281990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:32.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.004+0000 7fed2e510700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed281990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59564/0 (socket says 172.21.15.81:59564) 2022-09-10T04:18:32.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.004+0000 7fed2e510700 1 -- 172.21.15.81:0/3030807595 learned_addr learned my addr 172.21.15.81:0/3030807595 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:32.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.005+0000 7fed2e510700 1 -- 172.21.15.81:0/3030807595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240093e0 con 0x7fed28079de0 2022-09-10T04:18:32.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.005+0000 7fed2e510700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed281990d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fed24003f00 tx=0x7fed24003fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:32.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.005+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed24011950 con 0x7fed28079de0 2022-09-10T04:18:32.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.005+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fed24011ab0 con 0x7fed28079de0 2022-09-10T04:18:32.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.005+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28199810 con 0x7fed28079de0 2022-09-10T04:18:32.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.005+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fed24019550 con 0x7fed28079de0 2022-09-10T04:18:32.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.006+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed28199c40 con 0x7fed28079de0 2022-09-10T04:18:32.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.007+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) v1 ==== 42679+0+0 (secure 0 0 0) 0x7fed2402a020 con 0x7fed28079de0 2022-09-10T04:18:32.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.007+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed281096a0 con 0x7fed28079de0 2022-09-10T04:18:32.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.007+0000 7fed177fe700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7fed100357e0 0x7fed10037c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:32.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.007+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fed24048a50 con 0x7fed28079de0 2022-09-10T04:18:32.761 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.007+0000 7fed2dd0f700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7fed100357e0 0x7fed10037c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:32.761 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.008+0000 7fed2dd0f700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7fed100357e0 0x7fed10037c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed18009750 tx=0x7fed18006cc0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:32.761 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.012+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fed24018d20 con 0x7fed28079de0 2022-09-10T04:18:32.761 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.275+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fed2819ca80 con 0x7fed28079de0 2022-09-10T04:18:32.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.328+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 5) v1 ==== 42690+0+0 (secure 0 0 0) 0x7fed24017020 con 0x7fed28079de0 2022-09-10T04:18:32.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.330+0000 7fed177fe700 1 -- 172.21.15.81:0/3030807595 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fed24023390 con 0x7fed28079de0 2022-09-10T04:18:32.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7fed100357e0 msgr2=0x7fed10037c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:32.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7fed100357e0 0x7fed10037c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fed18009750 tx=0x7fed18006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:32.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 msgr2=0x7fed281990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:32.763 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed281990d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fed24003f00 tx=0x7fed24003fe0 comp rx=0 tx=0).stop 2022-09-10T04:18:32.763 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 shutdown_connections 2022-09-10T04:18:32.763 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7fed100357e0 0x7fed10037c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:32.763 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 --2- 172.21.15.81:0/3030807595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed28079de0 0x7fed281990d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:32.764 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 >> 172.21.15.81:0/3030807595 conn(0x7fed280fe9f0 msgr2=0x7fed281000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:32.764 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 shutdown_connections 2022-09-10T04:18:32.764 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:32.333+0000 7fed30774700 1 -- 172.21.15.81:0/3030807595 wait complete. 2022-09-10T04:18:32.954 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:32.813+0000 7fe8c6668080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:18:33.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:32.949+0000 7fe8c6668080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:18:33.305 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:33.080+0000 7fe8c6668080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:18:33.718 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:33 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3030807595' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2022-09-10T04:18:33.719 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:33 smithi081 ceph-mon[90133]: mgrmap e5: x(active, since 5s) 2022-09-10T04:18:33.719 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:33.341+0000 7fe8c6668080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:18:34.042 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:33.800+0000 7fe8c6668080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:18:34.042 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:33.934+0000 7fe8c6668080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:18:34.062 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:18:34.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 5, 2022-09-10T04:18:34.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2022-09-10T04:18:34.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "active_name": "x", 2022-09-10T04:18:34.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standby": 0 2022-09-10T04:18:34.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:18:34.064 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.405+0000 7f73df59e700 1 Processor -- start 2022-09-10T04:18:34.064 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.405+0000 7f73df59e700 1 -- start start 2022-09-10T04:18:34.064 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.405+0000 7f73df59e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 0x7f73e0105d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:34.064 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.405+0000 7f73df59e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01062a0 con 0x7f73e0077ce0 2022-09-10T04:18:34.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.406+0000 7f73de59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 0x7f73e0105d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:34.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.406+0000 7f73de59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 0x7f73e0105d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59580/0 (socket says 172.21.15.81:59580) 2022-09-10T04:18:34.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.406+0000 7f73de59c700 1 -- 172.21.15.81:0/2918580346 learned_addr learned my addr 172.21.15.81:0/2918580346 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:34.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.406+0000 7f73de59c700 1 -- 172.21.15.81:0/2918580346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e01063e0 con 0x7f73e0077ce0 2022-09-10T04:18:34.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.407+0000 7f73de59c700 1 --2- 172.21.15.81:0/2918580346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 0x7f73e0105d60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009700 tx=0x7f73d0009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63a0d9a41c1bc8 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:34.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.407+0000 7f73dd59a700 1 -- 172.21.15.81:0/2918580346 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73d0004160 con 0x7f73e0077ce0 2022-09-10T04:18:34.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.407+0000 7f73dd59a700 1 -- 172.21.15.81:0/2918580346 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f73d00042c0 con 0x7f73e0077ce0 2022-09-10T04:18:34.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.407+0000 7f73dd59a700 1 -- 172.21.15.81:0/2918580346 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73d00039d0 con 0x7f73e0077ce0 2022-09-10T04:18:34.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.407+0000 7f73df59e700 1 -- 172.21.15.81:0/2918580346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 msgr2=0x7f73e0105d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:34.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.407+0000 7f73df59e700 1 --2- 172.21.15.81:0/2918580346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 0x7f73e0105d60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009700 tx=0x7f73d0009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:34.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.408+0000 7f73df59e700 1 -- 172.21.15.81:0/2918580346 shutdown_connections 2022-09-10T04:18:34.067 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.408+0000 7f73df59e700 1 --2- 172.21.15.81:0/2918580346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0077ce0 0x7f73e0105d60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:34.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.408+0000 7f73df59e700 1 -- 172.21.15.81:0/2918580346 >> 172.21.15.81:0/2918580346 conn(0x7f73e00fd7d0 msgr2=0x7f73e00ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:34.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.408+0000 7f73df59e700 1 -- 172.21.15.81:0/2918580346 shutdown_connections 2022-09-10T04:18:34.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.408+0000 7f73df59e700 1 -- 172.21.15.81:0/2918580346 wait complete. 2022-09-10T04:18:34.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.409+0000 7f73df59e700 1 Processor -- start 2022-09-10T04:18:34.068 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.409+0000 7f73df59e700 1 -- start start 2022-09-10T04:18:34.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.409+0000 7f73df59e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 0x7f73e01970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:34.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.410+0000 7f73df59e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0197610 con 0x7f73e0196ca0 2022-09-10T04:18:34.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.410+0000 7f73de59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 0x7f73e01970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:34.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.410+0000 7f73de59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 0x7f73e01970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59596/0 (socket says 172.21.15.81:59596) 2022-09-10T04:18:34.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.410+0000 7f73de59c700 1 -- 172.21.15.81:0/804237632 learned_addr learned my addr 172.21.15.81:0/804237632 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:34.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.410+0000 7f73de59c700 1 -- 172.21.15.81:0/804237632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00093e0 con 0x7f73e0196ca0 2022-09-10T04:18:34.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.410+0000 7f73de59c700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 0x7f73e01970d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009e90 tx=0x7f73d0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:34.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.411+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73d0011840 con 0x7f73e0196ca0 2022-09-10T04:18:34.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.411+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e0197810 con 0x7f73e0196ca0 2022-09-10T04:18:34.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.411+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f73d001a430 con 0x7f73e0196ca0 2022-09-10T04:18:34.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.411+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73d0019550 con 0x7f73e0196ca0 2022-09-10T04:18:34.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.411+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e019a430 con 0x7f73e0196ca0 2022-09-10T04:18:34.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.412+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f73d002a020 con 0x7f73e0196ca0 2022-09-10T04:18:34.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.412+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e0107210 con 0x7f73e0196ca0 2022-09-10T04:18:34.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.413+0000 7f73cf7fe700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 0x7f73c4037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:34.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.413+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f73d00489b0 con 0x7f73e0196ca0 2022-09-10T04:18:34.072 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.413+0000 7f73ddd9b700 1 -- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 msgr2=0x7f73c4037c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:34.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.413+0000 7f73ddd9b700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 0x7f73c4037c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:18:34.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.418+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f73d00119a0 con 0x7f73e0196ca0 2022-09-10T04:18:34.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.614+0000 7f73ddd9b700 1 -- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 msgr2=0x7f73c4037c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:34.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.614+0000 7f73ddd9b700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 0x7f73c4037c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:18:34.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.684+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f73e019ab20 con 0x7f73e0196ca0 2022-09-10T04:18:34.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.685+0000 7f73cf7fe700 1 -- 172.21.15.81:0/804237632 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f73d0018b40 con 0x7f73e0196ca0 2022-09-10T04:18:34.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 msgr2=0x7f73c4037c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:18:34.074 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 0x7f73c4037c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:34.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 msgr2=0x7f73e01970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:34.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 0x7f73e01970d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73d0009e90 tx=0x7f73d0004040 comp rx=0 tx=0).stop 2022-09-10T04:18:34.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 shutdown_connections 2022-09-10T04:18:34.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f73c40357d0 0x7f73c4037c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:34.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 --2- 172.21.15.81:0/804237632 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73e0196ca0 0x7f73e01970d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:34.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.687+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 >> 172.21.15.81:0/804237632 conn(0x7f73e00fd7d0 msgr2=0x7f73e00fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:34.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.688+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 shutdown_connections 2022-09-10T04:18:34.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:33.688+0000 7f73df59e700 1 -- 172.21.15.81:0/804237632 wait complete. 2022-09-10T04:18:34.076 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for the mgr to restart... 2022-09-10T04:18:34.077 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr epoch 5... 2022-09-10T04:18:34.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:34.120+0000 7fe8c6668080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:18:34.305 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:34.261+0000 7fe8c6668080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:18:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:34 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/804237632' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-09-10T04:18:35.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:34.882+0000 7fe8c6668080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:18:36.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:35.700+0000 7fe8c6668080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:18:36.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:35.932+0000 7fe8c6668080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:18:37.305 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:36.989+0000 7fe8c6668080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:18:37.305 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:37.136+0000 7fe8c6668080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:18:37.724 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:37.415+0000 7fe8c6668080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:18:38.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:37.723+0000 7fe8c6668080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:18:38.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:37.953+0000 7fe8c6668080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:18:38.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:38.546+0000 7fe8c6668080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:18:39.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:39.450+0000 7fe8c6668080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:18:40.196 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:39.872+0000 7fe8c6668080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:18:40.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:40.194+0000 7fe8c6668080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:18:40.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:40.382+0000 7fe8c6668080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:18:41.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:40.739+0000 7fe8c6668080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:18:42.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:18:41.734+0000 7fe8c6668080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:18:42.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:41 smithi081 ceph-mon[90133]: Active manager daemon x restarted 2022-09-10T04:18:42.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:41 smithi081 ceph-mon[90133]: Activating manager daemon x 2022-09-10T04:18:42.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:41 smithi081 ceph-mon[90133]: osdmap e2: 0 total, 0 up, 0 in 2022-09-10T04:18:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: mgrmap e6: x(active, starting, since 0.0528004s) 2022-09-10T04:18:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:18:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-10T04:18:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-10T04:18:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-10T04:18:43.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-10T04:18:43.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: Manager daemon x is now available 2022-09-10T04:18:43.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:43.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:43.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:18:43.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:18:43.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:18:43.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:18:43.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:18:43.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:43.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:42 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:43.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:18:43.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2022-09-10T04:18:43.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "initialized": true 2022-09-10T04:18:43.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:18:43.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.637+0000 7f4aa78b4700 1 Processor -- start 2022-09-10T04:18:43.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.637+0000 7f4aa78b4700 1 -- start start 2022-09-10T04:18:43.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.637+0000 7f4aa78b4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa0105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:43.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.637+0000 7f4aa78b4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa01062b0 con 0x7f4aa01058d0 2022-09-10T04:18:43.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.637+0000 7f4aa5650700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa0105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:43.184 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.638+0000 7f4aa5650700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa0105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59602/0 (socket says 172.21.15.81:59602) 2022-09-10T04:18:43.184 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.638+0000 7f4aa5650700 1 -- 172.21.15.81:0/3951783676 learned_addr learned my addr 172.21.15.81:0/3951783676 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:43.184 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.638+0000 7f4aa5650700 1 -- 172.21.15.81:0/3951783676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa01063f0 con 0x7f4aa01058d0 2022-09-10T04:18:43.184 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.638+0000 7f4aa5650700 1 --2- 172.21.15.81:0/3951783676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa0105ce0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009700 tx=0x7f4a9c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2058c49ac9e82e server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:43.184 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4a8ffff700 1 -- 172.21.15.81:0/3951783676 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a9c004160 con 0x7f4aa01058d0 2022-09-10T04:18:43.185 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4a8ffff700 1 -- 172.21.15.81:0/3951783676 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f4a9c0042c0 con 0x7f4aa01058d0 2022-09-10T04:18:43.185 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4a8ffff700 1 -- 172.21.15.81:0/3951783676 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a9c0039d0 con 0x7f4aa01058d0 2022-09-10T04:18:43.185 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3951783676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 msgr2=0x7f4aa0105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:43.186 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3951783676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa0105ce0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009700 tx=0x7f4a9c009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:43.186 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3951783676 shutdown_connections 2022-09-10T04:18:43.187 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3951783676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa0105ce0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.187 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.639+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3951783676 >> 172.21.15.81:0/3951783676 conn(0x7f4aa0078a20 msgr2=0x7f4aa0078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:43.188 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.640+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3951783676 shutdown_connections 2022-09-10T04:18:43.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.640+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3951783676 wait complete. 2022-09-10T04:18:43.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.641+0000 7f4aa78b4700 1 Processor -- start 2022-09-10T04:18:43.189 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.642+0000 7f4aa78b4700 1 -- start start 2022-09-10T04:18:43.190 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.642+0000 7f4aa78b4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa01990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:43.190 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.642+0000 7f4aa78b4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa0199600 con 0x7f4aa01058d0 2022-09-10T04:18:43.191 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.642+0000 7f4aa5650700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa01990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:43.191 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.642+0000 7f4aa5650700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa01990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59610/0 (socket says 172.21.15.81:59610) 2022-09-10T04:18:43.192 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.642+0000 7f4aa5650700 1 -- 172.21.15.81:0/3611614227 learned_addr learned my addr 172.21.15.81:0/3611614227 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:43.192 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.643+0000 7f4aa5650700 1 -- 172.21.15.81:0/3611614227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c0093e0 con 0x7f4aa01058d0 2022-09-10T04:18:43.193 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.643+0000 7f4aa5650700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa01990c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c003f10 tx=0x7f4a9c003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:43.193 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.643+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a9c011950 con 0x7f4aa01058d0 2022-09-10T04:18:43.194 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.643+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f4a9c011ab0 con 0x7f4aa01058d0 2022-09-10T04:18:43.194 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.643+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4a9c019550 con 0x7f4aa01058d0 2022-09-10T04:18:43.194 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.643+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa0199800 con 0x7f4aa01058d0 2022-09-10T04:18:43.195 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.644+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa0199c30 con 0x7f4aa01058d0 2022-09-10T04:18:43.195 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.645+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f4a9c02a020 con 0x7f4aa01058d0 2022-09-10T04:18:43.195 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.645+0000 7f4a8e7fc700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:43.196 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.645+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 --> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a880383f0 con 0x7f4a88035830 2022-09-10T04:18:43.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.645+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4a9c0489f0 con 0x7f4aa01058d0 2022-09-10T04:18:43.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.645+0000 7f4aa4e4f700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:43.197 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.645+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:18:43.198 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.846+0000 7f4aa4e4f700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:43.198 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:34.846+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:18:43.198 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:35.247+0000 7f4aa4e4f700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:43.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:35.247+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-09-10T04:18:43.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:36.048+0000 7f4aa4e4f700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:43.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:36.048+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-09-10T04:18:43.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:37.650+0000 7f4aa4e4f700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:43.199 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:37.650+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-09-10T04:18:43.200 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:40.853+0000 7f4aa4e4f700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/2567224701 2022-09-10T04:18:43.200 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:40.853+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-09-10T04:18:43.201 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:41.789+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mgrmap(e 6) v1 ==== 42500+0+0 (secure 0 0 0) 0x7f4a9c017020 con 0x7f4aa01058d0 2022-09-10T04:18:43.201 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:41.789+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 msgr2=0x7f4a88037ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:18:43.202 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:41.789+0000 7f4a8e7fc700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.202 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.798+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 7) v1 ==== 42610+0+0 (secure 0 0 0) 0x7f4a9c00f050 con 0x7f4aa01058d0 2022-09-10T04:18:43.203 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.798+0000 7f4a8e7fc700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4a88038fd0 0x7f4a8803b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:43.203 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.798+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a880383f0 con 0x7f4a88038fd0 2022-09-10T04:18:43.203 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.799+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4a88038fd0 0x7f4a8803b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:43.203 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.801+0000 7f4aa4e4f700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4a88038fd0 0x7f4a8803b3b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4a90003890 tx=0x7f4a900092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:43.204 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.802+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6424 (secure 0 0 0) 0x7f4a880383f0 con 0x7f4a88038fd0 2022-09-10T04:18:43.204 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.808+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4aa019a3f0 con 0x7f4a88038fd0 2022-09-10T04:18:43.204 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.808+0000 7f4a8e7fc700 1 -- 172.21.15.81:0/3611614227 <== mgr.14118 v2:172.21.15.81:6800/3224857868 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4aa019a3f0 con 0x7f4a88038fd0 2022-09-10T04:18:43.204 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4a88038fd0 msgr2=0x7f4a8803b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:43.205 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4a88038fd0 0x7f4a8803b3b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4a90003890 tx=0x7f4a900092b0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.205 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 msgr2=0x7f4aa01990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:43.205 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa01990c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c003f10 tx=0x7f4a9c003ff0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.205 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 shutdown_connections 2022-09-10T04:18:43.205 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4a88038fd0 0x7f4a8803b3b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.206 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4aa01058d0 0x7f4aa01990c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.207 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 --2- 172.21.15.81:0/3611614227 >> [v2:172.21.15.81:6800/2567224701,v1:172.21.15.81:6801/2567224701] conn(0x7f4a88035830 0x7f4a88037ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:43.207 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 >> 172.21.15.81:0/3611614227 conn(0x7f4aa0078a20 msgr2=0x7f4aa0103780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:43.207 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 shutdown_connections 2022-09-10T04:18:43.207 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:42.809+0000 7f4aa78b4700 1 -- 172.21.15.81:0/3611614227 wait complete. 2022-09-10T04:18:43.207 INFO:teuthology.orchestra.run.smithi081.stdout:mgr epoch 5 is available 2022-09-10T04:18:43.208 INFO:teuthology.orchestra.run.smithi081.stdout:Setting orchestrator backend to cephadm... 2022-09-10T04:18:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:43 smithi081 ceph-mon[90133]: mgrmap e7: x(active, since 1.0633s) 2022-09-10T04:18:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:43 smithi081 ceph-mon[90133]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-09-10T04:18:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:43 smithi081 ceph-mon[90133]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-09-10T04:18:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:43 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:43 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:44.505 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.836+0000 7fdfaded2700 1 Processor -- start 2022-09-10T04:18:44.505 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.836+0000 7fdfaded2700 1 -- start start 2022-09-10T04:18:44.506 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.836+0000 7fdfaded2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8105be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:44.506 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.836+0000 7fdfaded2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa81061b0 con 0x7fdfa81057d0 2022-09-10T04:18:44.506 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.837+0000 7fdfaced0700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8105be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:44.506 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.837+0000 7fdfaced0700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8105be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43832/0 (socket says 172.21.15.81:43832) 2022-09-10T04:18:44.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.837+0000 7fdfaced0700 1 -- 172.21.15.81:0/1251014602 learned_addr learned my addr 172.21.15.81:0/1251014602 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:44.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.837+0000 7fdfaced0700 1 -- 172.21.15.81:0/1251014602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa81062f0 con 0x7fdfa81057d0 2022-09-10T04:18:44.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.837+0000 7fdfaced0700 1 --2- 172.21.15.81:0/1251014602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8105be0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009700 tx=0x7fdf9c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdb0e514fa93aae server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:44.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfa77fe700 1 -- 172.21.15.81:0/1251014602 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf9c004160 con 0x7fdfa81057d0 2022-09-10T04:18:44.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfa77fe700 1 -- 172.21.15.81:0/1251014602 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fdf9c0042c0 con 0x7fdfa81057d0 2022-09-10T04:18:44.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfa77fe700 1 -- 172.21.15.81:0/1251014602 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf9c0039d0 con 0x7fdfa81057d0 2022-09-10T04:18:44.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfaded2700 1 -- 172.21.15.81:0/1251014602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 msgr2=0x7fdfa8105be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:44.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfaded2700 1 --2- 172.21.15.81:0/1251014602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8105be0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009700 tx=0x7fdf9c009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:44.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfaded2700 1 -- 172.21.15.81:0/1251014602 shutdown_connections 2022-09-10T04:18:44.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfaded2700 1 --2- 172.21.15.81:0/1251014602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8105be0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:44.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.838+0000 7fdfaded2700 1 -- 172.21.15.81:0/1251014602 >> 172.21.15.81:0/1251014602 conn(0x7fdfa80788f0 msgr2=0x7fdfa8078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:44.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.839+0000 7fdfaded2700 1 -- 172.21.15.81:0/1251014602 shutdown_connections 2022-09-10T04:18:44.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.839+0000 7fdfaded2700 1 -- 172.21.15.81:0/1251014602 wait complete. 2022-09-10T04:18:44.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.840+0000 7fdfaded2700 1 Processor -- start 2022-09-10T04:18:44.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.840+0000 7fdfaded2700 1 -- start start 2022-09-10T04:18:44.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.840+0000 7fdfaded2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:44.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.840+0000 7fdfaded2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa8199450 con 0x7fdfa81057d0 2022-09-10T04:18:44.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.841+0000 7fdfaced0700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:44.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.841+0000 7fdfaced0700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43844/0 (socket says 172.21.15.81:43844) 2022-09-10T04:18:44.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.841+0000 7fdfaced0700 1 -- 172.21.15.81:0/1415827258 learned_addr learned my addr 172.21.15.81:0/1415827258 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:44.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.841+0000 7fdfaced0700 1 -- 172.21.15.81:0/1415827258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c0093e0 con 0x7fdfa81057d0 2022-09-10T04:18:44.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.841+0000 7fdfaced0700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8198f10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009e90 tx=0x7fdf9c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:44.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.842+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf9c011840 con 0x7fdfa81057d0 2022-09-10T04:18:44.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.842+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa8199650 con 0x7fdfa81057d0 2022-09-10T04:18:44.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.842+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fdf9c01a430 con 0x7fdfa81057d0 2022-09-10T04:18:44.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.842+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf9c019550 con 0x7fdfa81057d0 2022-09-10T04:18:44.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.842+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa8199a80 con 0x7fdfa81057d0 2022-09-10T04:18:44.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.843+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa8107280 con 0x7fdfa81057d0 2022-09-10T04:18:44.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.843+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 7) v1 ==== 42610+0+0 (secure 0 0 0) 0x7fdf9c02a020 con 0x7fdfa81057d0 2022-09-10T04:18:44.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.843+0000 7fdfa5ffb700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fdf8c035780 0x7fdf8c037c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:44.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.844+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdf9c048920 con 0x7fdfa81057d0 2022-09-10T04:18:44.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.844+0000 7fdfa7fff700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fdf8c035780 0x7fdf8c037c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:44.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.845+0000 7fdfa7fff700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fdf8c035780 0x7fdf8c037c30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009750 tx=0x7fdf94006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:44.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:43.849+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fdf9c018b40 con 0x7fdfa81057d0 2022-09-10T04:18:44.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.085+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfa8103e40 con 0x7fdf8c035780 2022-09-10T04:18:44.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.093+0000 7fdfa5ffb700 1 -- 172.21.15.81:0/1415827258 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdfa8103e40 con 0x7fdf8c035780 2022-09-10T04:18:44.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.096+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fdf8c035780 msgr2=0x7fdf8c037c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:44.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.096+0000 7fdfaded2700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fdf8c035780 0x7fdf8c037c30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009750 tx=0x7fdf94006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:44.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 msgr2=0x7fdfa8198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:44.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8198f10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c009e90 tx=0x7fdf9c004060 comp rx=0 tx=0).stop 2022-09-10T04:18:44.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 shutdown_connections 2022-09-10T04:18:44.517 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fdf8c035780 0x7fdf8c037c30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:44.517 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 --2- 172.21.15.81:0/1415827258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdfa81057d0 0x7fdfa8198f10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:44.517 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 >> 172.21.15.81:0/1415827258 conn(0x7fdfa80788f0 msgr2=0x7fdfa8103730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:44.517 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 shutdown_connections 2022-09-10T04:18:44.518 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:44.097+0000 7fdfaded2700 1 -- 172.21.15.81:0/1415827258 wait complete. 2022-09-10T04:18:45.359 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:45 smithi081 ceph-mon[90133]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:45.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:45 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:45.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:45 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:18:45.361 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:45 smithi081 ceph-mon[90133]: mgrmap e8: x(active, since 2s) 2022-09-10T04:18:45.804 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout value unchanged 2022-09-10T04:18:45.804 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.100+0000 7f25c55e7700 1 Processor -- start 2022-09-10T04:18:45.805 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.100+0000 7f25c55e7700 1 -- start start 2022-09-10T04:18:45.805 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.101+0000 7f25c55e7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c00782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:45.805 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.101+0000 7f25c55e7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c0078800 con 0x7f25c0079de0 2022-09-10T04:18:45.805 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.101+0000 7f25beffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c00782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:45.806 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.101+0000 7f25beffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c00782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43852/0 (socket says 172.21.15.81:43852) 2022-09-10T04:18:45.806 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.101+0000 7f25beffd700 1 -- 172.21.15.81:0/1707061290 learned_addr learned my addr 172.21.15.81:0/1707061290 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:45.806 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.101+0000 7f25beffd700 1 -- 172.21.15.81:0/1707061290 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c0078940 con 0x7f25c0079de0 2022-09-10T04:18:45.806 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.102+0000 7f25beffd700 1 --2- 172.21.15.81:0/1707061290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c00782c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009700 tx=0x7f25b0009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0e2fcdea90e9fc server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:45.807 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.102+0000 7f25bdffb700 1 -- 172.21.15.81:0/1707061290 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25b0004160 con 0x7f25c0079de0 2022-09-10T04:18:45.807 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.102+0000 7f25bdffb700 1 -- 172.21.15.81:0/1707061290 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f25b00042c0 con 0x7f25c0079de0 2022-09-10T04:18:45.807 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.102+0000 7f25bdffb700 1 -- 172.21.15.81:0/1707061290 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25b00039d0 con 0x7f25c0079de0 2022-09-10T04:18:45.807 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 -- 172.21.15.81:0/1707061290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 msgr2=0x7f25c00782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:45.808 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 --2- 172.21.15.81:0/1707061290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c00782c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009700 tx=0x7f25b0009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:45.808 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 -- 172.21.15.81:0/1707061290 shutdown_connections 2022-09-10T04:18:45.808 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 --2- 172.21.15.81:0/1707061290 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c00782c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:45.808 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 -- 172.21.15.81:0/1707061290 >> 172.21.15.81:0/1707061290 conn(0x7f25c00fe9f0 msgr2=0x7f25c0100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:45.809 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 -- 172.21.15.81:0/1707061290 shutdown_connections 2022-09-10T04:18:45.809 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.103+0000 7f25c55e7700 1 -- 172.21.15.81:0/1707061290 wait complete. 2022-09-10T04:18:45.809 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.104+0000 7f25c55e7700 1 Processor -- start 2022-09-10T04:18:45.809 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.105+0000 7f25c55e7700 1 -- start start 2022-09-10T04:18:45.810 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.105+0000 7f25c55e7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c0199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:45.810 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.105+0000 7f25c55e7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c01995c0 con 0x7f25c0079de0 2022-09-10T04:18:45.810 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.105+0000 7f25beffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c0199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:45.810 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.105+0000 7f25beffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c0199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43862/0 (socket says 172.21.15.81:43862) 2022-09-10T04:18:45.810 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.105+0000 7f25beffd700 1 -- 172.21.15.81:0/1654213294 learned_addr learned my addr 172.21.15.81:0/1654213294 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:45.811 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25beffd700 1 -- 172.21.15.81:0/1654213294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00093e0 con 0x7f25c0079de0 2022-09-10T04:18:45.811 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25beffd700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c0199080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25b0003fb0 tx=0x7f25b0004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:45.811 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25b00118c0 con 0x7f25c0079de0 2022-09-10T04:18:45.812 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f25b0011a20 con 0x7f25c0079de0 2022-09-10T04:18:45.812 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c01997c0 con 0x7f25c0079de0 2022-09-10T04:18:45.812 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f25b0019550 con 0x7f25c0079de0 2022-09-10T04:18:45.812 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.106+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c0199bf0 con 0x7f25c0079de0 2022-09-10T04:18:45.812 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.107+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f25b002a020 con 0x7f25c0079de0 2022-09-10T04:18:45.813 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.108+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c0060310 con 0x7f25c0079de0 2022-09-10T04:18:45.813 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.108+0000 7f25a7fff700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f25ac035870 0x7f25ac037d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:45.813 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.108+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f25b0048ad0 con 0x7f25c0079de0 2022-09-10T04:18:45.813 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.108+0000 7f25be7fc700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f25ac035870 0x7f25ac037d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:45.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.109+0000 7f25be7fc700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f25ac035870 0x7f25ac037d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009750 tx=0x7f25b4006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:45.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.113+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f25b0018d20 con 0x7f25c0079de0 2022-09-10T04:18:45.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.326+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f25c00fff00 con 0x7f25ac035870 2022-09-10T04:18:45.814 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.326+0000 7f25a7fff700 1 -- 172.21.15.81:0/1654213294 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f25c00fff00 con 0x7f25ac035870 2022-09-10T04:18:45.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f25ac035870 msgr2=0x7f25ac037d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:45.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f25ac035870 0x7f25ac037d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f25b4009750 tx=0x7f25b4006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:45.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 msgr2=0x7f25c0199080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:45.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c0199080 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25b0003fb0 tx=0x7f25b0004090 comp rx=0 tx=0).stop 2022-09-10T04:18:45.815 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 shutdown_connections 2022-09-10T04:18:45.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f25ac035870 0x7f25ac037d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:45.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 --2- 172.21.15.81:0/1654213294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c0079de0 0x7f25c0199080 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:45.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 >> 172.21.15.81:0/1654213294 conn(0x7f25c00fe9f0 msgr2=0x7f25c00ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:45.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 shutdown_connections 2022-09-10T04:18:45.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:45.329+0000 7f25c55e7700 1 -- 172.21.15.81:0/1654213294 wait complete. 2022-09-10T04:18:45.817 INFO:teuthology.orchestra.run.smithi081.stdout:Generating ssh key... 2022-09-10T04:18:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-mon[90133]: [10/Sep/2022:04:18:44] ENGINE Bus STARTING 2022-09-10T04:18:46.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-mon[90133]: [10/Sep/2022:04:18:45] ENGINE Serving on https://172.21.15.81:7150 2022-09-10T04:18:46.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-mon[90133]: [10/Sep/2022:04:18:45] ENGINE Serving on https://:::8765 2022-09-10T04:18:46.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-mon[90133]: [10/Sep/2022:04:18:45] ENGINE Bus STARTED 2022-09-10T04:18:46.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:18:46.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-mon[90133]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:47.242 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: Generating public/private rsa key pair. 2022-09-10T04:18:47.243 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: Your identification has been saved in /tmp/tmprm0tugbj/key. 2022-09-10T04:18:47.243 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: Your public key has been saved in /tmp/tmprm0tugbj/key.pub. 2022-09-10T04:18:47.243 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: The key fingerprint is: 2022-09-10T04:18:47.243 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: SHA256:PKWQ+veuG7yiTsLOr2JDt0Vvy5AfQs+y0NdzzKuyTR8 ceph-70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:47.244 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: The key's randomart image is: 2022-09-10T04:18:47.244 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: +---[RSA 3072]----+ 2022-09-10T04:18:47.244 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: | | 2022-09-10T04:18:47.245 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: | . | 2022-09-10T04:18:47.245 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: | o . | 2022-09-10T04:18:47.245 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: | + o o | 2022-09-10T04:18:47.245 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: | = = S o | 2022-09-10T04:18:47.246 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: | o o O O + + | 2022-09-10T04:18:47.246 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: |. + = @ *.oE. | 2022-09-10T04:18:47.246 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: |.= + ..*o+... | 2022-09-10T04:18:47.246 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: |..=++. o**+. | 2022-09-10T04:18:47.246 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: +----[SHA256]-----+ 2022-09-10T04:18:47.312 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.404+0000 7ff6422d9700 1 Processor -- start 2022-09-10T04:18:47.312 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.404+0000 7ff6422d9700 1 -- start start 2022-09-10T04:18:47.312 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.404+0000 7ff6422d9700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c0782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:47.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.404+0000 7ff6422d9700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0787e0 con 0x7ff63c079dc0 2022-09-10T04:18:47.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.405+0000 7ff63bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c0782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:47.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.405+0000 7ff63bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c0782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43878/0 (socket says 172.21.15.81:43878) 2022-09-10T04:18:47.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.405+0000 7ff63bfff700 1 -- 172.21.15.81:0/3613279715 learned_addr learned my addr 172.21.15.81:0/3613279715 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:47.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.406+0000 7ff63bfff700 1 -- 172.21.15.81:0/3613279715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c078920 con 0x7ff63c079dc0 2022-09-10T04:18:47.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.406+0000 7ff63bfff700 1 --2- 172.21.15.81:0/3613279715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c0782a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009700 tx=0x7ff62c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d04704168bbbfea server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:47.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.406+0000 7ff63affd700 1 -- 172.21.15.81:0/3613279715 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff62c004160 con 0x7ff63c079dc0 2022-09-10T04:18:47.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.406+0000 7ff63affd700 1 -- 172.21.15.81:0/3613279715 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7ff62c0042c0 con 0x7ff63c079dc0 2022-09-10T04:18:47.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.406+0000 7ff63affd700 1 -- 172.21.15.81:0/3613279715 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff62c0039d0 con 0x7ff63c079dc0 2022-09-10T04:18:47.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 -- 172.21.15.81:0/3613279715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 msgr2=0x7ff63c0782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:47.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 --2- 172.21.15.81:0/3613279715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c0782a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009700 tx=0x7ff62c009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:47.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 -- 172.21.15.81:0/3613279715 shutdown_connections 2022-09-10T04:18:47.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 --2- 172.21.15.81:0/3613279715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c0782a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:47.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 -- 172.21.15.81:0/3613279715 >> 172.21.15.81:0/3613279715 conn(0x7ff63c0fe9f0 msgr2=0x7ff63c100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:47.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 -- 172.21.15.81:0/3613279715 shutdown_connections 2022-09-10T04:18:47.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.407+0000 7ff6422d9700 1 -- 172.21.15.81:0/3613279715 wait complete. 2022-09-10T04:18:47.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.409+0000 7ff6422d9700 1 Processor -- start 2022-09-10T04:18:47.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.409+0000 7ff6422d9700 1 -- start start 2022-09-10T04:18:47.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.409+0000 7ff6422d9700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:47.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.410+0000 7ff6422d9700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c1995d0 con 0x7ff63c079dc0 2022-09-10T04:18:47.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.410+0000 7ff63bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:47.318 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.410+0000 7ff63bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43882/0 (socket says 172.21.15.81:43882) 2022-09-10T04:18:47.318 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.410+0000 7ff63bfff700 1 -- 172.21.15.81:0/3601973503 learned_addr learned my addr 172.21.15.81:0/3601973503 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:47.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.410+0000 7ff63bfff700 1 -- 172.21.15.81:0/3601973503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c0093e0 con 0x7ff63c079dc0 2022-09-10T04:18:47.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.410+0000 7ff63bfff700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c199090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff62c003fb0 tx=0x7ff62c004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:47.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.411+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff62c0118c0 con 0x7ff63c079dc0 2022-09-10T04:18:47.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.411+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c1997d0 con 0x7ff63c079dc0 2022-09-10T04:18:47.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.411+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7ff62c011a20 con 0x7ff63c079dc0 2022-09-10T04:18:47.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.411+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff62c019550 con 0x7ff63c079dc0 2022-09-10T04:18:47.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.411+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c199c00 con 0x7ff63c079dc0 2022-09-10T04:18:47.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.412+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7ff62c02a020 con 0x7ff63c079dc0 2022-09-10T04:18:47.322 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.412+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c060310 con 0x7ff63c079dc0 2022-09-10T04:18:47.322 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.412+0000 7ff6397fa700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7ff628035810 0x7ff628037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:47.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.412+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff62c048a30 con 0x7ff63c079dc0 2022-09-10T04:18:47.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.413+0000 7ff63b7fe700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7ff628035810 0x7ff628037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:47.324 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.413+0000 7ff63b7fe700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7ff628035810 0x7ff628037cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff630009750 tx=0x7ff630006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:47.325 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.417+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff62c018d20 con 0x7ff63c079dc0 2022-09-10T04:18:47.327 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.628+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff63c0fff00 con 0x7ff628035810 2022-09-10T04:18:47.327 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.876+0000 7ff6397fa700 1 -- 172.21.15.81:0/3601973503 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff63c0fff00 con 0x7ff628035810 2022-09-10T04:18:47.327 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7ff628035810 msgr2=0x7ff628037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:47.327 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7ff628035810 0x7ff628037cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff630009750 tx=0x7ff630006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:47.327 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 msgr2=0x7ff63c199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c199090 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff62c003fb0 tx=0x7ff62c004090 comp rx=0 tx=0).stop 2022-09-10T04:18:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 shutdown_connections 2022-09-10T04:18:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7ff628035810 0x7ff628037cc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 --2- 172.21.15.81:0/3601973503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff63c079dc0 0x7ff63c199090 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:47.329 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 >> 172.21.15.81:0/3601973503 conn(0x7ff63c0fe9f0 msgr2=0x7ff63c0ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:47.329 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 shutdown_connections 2022-09-10T04:18:47.329 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:46.879+0000 7ff6422d9700 1 -- 172.21.15.81:0/3601973503 wait complete. 2022-09-10T04:18:48.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:47 smithi081 ceph-mon[90133]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:48.124 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:47 smithi081 ceph-mon[90133]: Generating ssh key... 2022-09-10T04:18:48.124 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:47 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:48.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:47 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:48.506 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:18:48.506 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.863+0000 7feee6fdd700 1 Processor -- start 2022-09-10T04:18:48.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.863+0000 7feee6fdd700 1 -- start start 2022-09-10T04:18:48.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.863+0000 7feee6fdd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:48.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.863+0000 7feee6fdd700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0103da0 con 0x7feee01033c0 2022-09-10T04:18:48.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.864+0000 7feee4d79700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:48.507 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.864+0000 7feee4d79700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43894/0 (socket says 172.21.15.81:43894) 2022-09-10T04:18:48.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.864+0000 7feee4d79700 1 -- 172.21.15.81:0/3925291670 learned_addr learned my addr 172.21.15.81:0/3925291670 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:48.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.864+0000 7feee4d79700 1 -- 172.21.15.81:0/3925291670 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0103ee0 con 0x7feee01033c0 2022-09-10T04:18:48.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.865+0000 7feee4d79700 1 --2- 172.21.15.81:0/3925291670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01037d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feed4009700 tx=0x7feed4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2999a84f278bc1ce server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:48.508 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.865+0000 7feedf7fe700 1 -- 172.21.15.81:0/3925291670 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed4004160 con 0x7feee01033c0 2022-09-10T04:18:48.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.865+0000 7feedf7fe700 1 -- 172.21.15.81:0/3925291670 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7feed40042c0 con 0x7feee01033c0 2022-09-10T04:18:48.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.865+0000 7feedf7fe700 1 -- 172.21.15.81:0/3925291670 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed40039d0 con 0x7feee01033c0 2022-09-10T04:18:48.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 -- 172.21.15.81:0/3925291670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 msgr2=0x7feee01037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:48.509 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 --2- 172.21.15.81:0/3925291670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01037d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feed4009700 tx=0x7feed4009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:48.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 -- 172.21.15.81:0/3925291670 shutdown_connections 2022-09-10T04:18:48.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 --2- 172.21.15.81:0/3925291670 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01037d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:48.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 -- 172.21.15.81:0/3925291670 >> 172.21.15.81:0/3925291670 conn(0x7feee00fe9f0 msgr2=0x7feee0100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:48.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 -- 172.21.15.81:0/3925291670 shutdown_connections 2022-09-10T04:18:48.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.866+0000 7feee6fdd700 1 -- 172.21.15.81:0/3925291670 wait complete. 2022-09-10T04:18:48.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.867+0000 7feee6fdd700 1 Processor -- start 2022-09-10T04:18:48.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.867+0000 7feee6fdd700 1 -- start start 2022-09-10T04:18:48.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.868+0000 7feee6fdd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:48.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.868+0000 7feee6fdd700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee0199600 con 0x7feee01033c0 2022-09-10T04:18:48.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.868+0000 7feee4d79700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:48.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.868+0000 7feee4d79700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43908/0 (socket says 172.21.15.81:43908) 2022-09-10T04:18:48.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.868+0000 7feee4d79700 1 -- 172.21.15.81:0/1472637370 learned_addr learned my addr 172.21.15.81:0/1472637370 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:48.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.868+0000 7feee4d79700 1 -- 172.21.15.81:0/1472637370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40093e0 con 0x7feee01033c0 2022-09-10T04:18:48.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.869+0000 7feee4d79700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01990c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feed4009e90 tx=0x7feed4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:48.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.873+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed4011840 con 0x7feee01033c0 2022-09-10T04:18:48.513 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.873+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee0199800 con 0x7feee01033c0 2022-09-10T04:18:48.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.873+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7feed401a430 con 0x7feee01033c0 2022-09-10T04:18:48.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.873+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feed4019550 con 0x7feee01033c0 2022-09-10T04:18:48.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.873+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee0199c30 con 0x7feee01033c0 2022-09-10T04:18:48.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.874+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7feed402a020 con 0x7feee01033c0 2022-09-10T04:18:48.514 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.874+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee0105290 con 0x7feee01033c0 2022-09-10T04:18:48.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.874+0000 7feeddffb700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7feec40357d0 0x7feec4037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:48.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.874+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7feed4048a50 con 0x7feee01033c0 2022-09-10T04:18:48.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.875+0000 7feedffff700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7feec40357d0 0x7feec4037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:48.515 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.875+0000 7feedffff700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7feec40357d0 0x7feec4037c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feecc009750 tx=0x7feecc006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:48.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:47.879+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7feed40197c0 con 0x7feee01033c0 2022-09-10T04:18:48.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.091+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7feee00ffeb0 con 0x7feec40357d0 2022-09-10T04:18:48.516 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.091+0000 7feeddffb700 1 -- 172.21.15.81:0/1472637370 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7feee00ffeb0 con 0x7feec40357d0 2022-09-10T04:18:48.517 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.093+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7feec40357d0 msgr2=0x7feec4037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:48.518 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.093+0000 7feee6fdd700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7feec40357d0 0x7feec4037c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7feecc009750 tx=0x7feecc006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:48.518 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 msgr2=0x7feee01990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:48.519 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01990c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feed4009e90 tx=0x7feed4004040 comp rx=0 tx=0).stop 2022-09-10T04:18:48.519 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 shutdown_connections 2022-09-10T04:18:48.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7feec40357d0 0x7feec4037c80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:48.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 --2- 172.21.15.81:0/1472637370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee01033c0 0x7feee01990c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:48.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 >> 172.21.15.81:0/1472637370 conn(0x7feee00fe9f0 msgr2=0x7feee00ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:48.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 shutdown_connections 2022-09-10T04:18:48.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:48.094+0000 7feee6fdd700 1 -- 172.21.15.81:0/1472637370 wait complete. 2022-09-10T04:18:48.521 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2022-09-10T04:18:48.521 INFO:teuthology.orchestra.run.smithi081.stdout:Adding key to root@localhost authorized_keys... 2022-09-10T04:18:48.521 INFO:teuthology.orchestra.run.smithi081.stdout:Adding host smithi081... 2022-09-10T04:18:49.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:49 smithi081 ceph-mon[90133]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:50.177 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:50 smithi081 ceph-mon[90133]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi081", "addr": "172.21.15.81", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:51.264 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:51 smithi081 ceph-mon[90133]: Deploying cephadm binary to smithi081 2022-09-10T04:18:53.029 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Added host 'smithi081' with addr '172.21.15.81' 2022-09-10T04:18:53.029 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.091+0000 7f825368b700 1 Processor -- start 2022-09-10T04:18:53.029 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.092+0000 7f825368b700 1 -- start start 2022-09-10T04:18:53.030 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.092+0000 7f825368b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:53.030 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.092+0000 7f825368b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c103da0 con 0x7f824c1033c0 2022-09-10T04:18:53.030 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.092+0000 7f8251427700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:53.030 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.092+0000 7f8251427700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43918/0 (socket says 172.21.15.81:43918) 2022-09-10T04:18:53.031 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.092+0000 7f8251427700 1 -- 172.21.15.81:0/3758750104 learned_addr learned my addr 172.21.15.81:0/3758750104 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:53.031 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.093+0000 7f8251427700 1 -- 172.21.15.81:0/3758750104 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c103ee0 con 0x7f824c1033c0 2022-09-10T04:18:53.031 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.093+0000 7f8251427700 1 --2- 172.21.15.81:0/3758750104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1037d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8248009700 tx=0x7f8248009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdca62d76d1afe2 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:53.032 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.093+0000 7f8243fff700 1 -- 172.21.15.81:0/3758750104 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8248004160 con 0x7f824c1033c0 2022-09-10T04:18:53.032 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.093+0000 7f8243fff700 1 -- 172.21.15.81:0/3758750104 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f82480042c0 con 0x7f824c1033c0 2022-09-10T04:18:53.032 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.093+0000 7f8243fff700 1 -- 172.21.15.81:0/3758750104 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82480039d0 con 0x7f824c1033c0 2022-09-10T04:18:53.032 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 -- 172.21.15.81:0/3758750104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 msgr2=0x7f824c1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:53.033 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 --2- 172.21.15.81:0/3758750104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1037d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8248009700 tx=0x7f8248009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:53.033 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 -- 172.21.15.81:0/3758750104 shutdown_connections 2022-09-10T04:18:53.033 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 --2- 172.21.15.81:0/3758750104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1037d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:53.033 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 -- 172.21.15.81:0/3758750104 >> 172.21.15.81:0/3758750104 conn(0x7f824c0fe9f0 msgr2=0x7f824c100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:53.034 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 -- 172.21.15.81:0/3758750104 shutdown_connections 2022-09-10T04:18:53.034 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.094+0000 7f825368b700 1 -- 172.21.15.81:0/3758750104 wait complete. 2022-09-10T04:18:53.034 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.095+0000 7f825368b700 1 Processor -- start 2022-09-10T04:18:53.034 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f825368b700 1 -- start start 2022-09-10T04:18:53.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f825368b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:53.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f825368b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c199620 con 0x7f824c1033c0 2022-09-10T04:18:53.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f8251427700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:53.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f8251427700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43930/0 (socket says 172.21.15.81:43930) 2022-09-10T04:18:53.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f8251427700 1 -- 172.21.15.81:0/250651480 learned_addr learned my addr 172.21.15.81:0/250651480 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:53.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.096+0000 7f8251427700 1 -- 172.21.15.81:0/250651480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480093e0 con 0x7f824c1033c0 2022-09-10T04:18:53.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.097+0000 7f8251427700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1990e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8248009e90 tx=0x7f8248004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:53.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.097+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8248011840 con 0x7f824c1033c0 2022-09-10T04:18:53.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.097+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f824801a430 con 0x7f824c1033c0 2022-09-10T04:18:53.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.097+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8248019550 con 0x7f824c1033c0 2022-09-10T04:18:53.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.097+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824c199820 con 0x7f824c1033c0 2022-09-10T04:18:53.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.097+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824c199c50 con 0x7f824c1033c0 2022-09-10T04:18:53.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.098+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f824802a020 con 0x7f824c1033c0 2022-09-10T04:18:53.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.098+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824c060310 con 0x7f824c1033c0 2022-09-10T04:18:53.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.099+0000 7f82427fc700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f82300357d0 0x7f8230037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:53.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.099+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f82480489b0 con 0x7f824c1033c0 2022-09-10T04:18:53.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.099+0000 7f8250c26700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f82300357d0 0x7f8230037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:53.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.100+0000 7f8250c26700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f82300357d0 0x7f8230037c80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f823c009750 tx=0x7f823c006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:53.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.104+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f82480119a0 con 0x7f824c1033c0 2022-09-10T04:18:53.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:49.324+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi081", "addr": "172.21.15.81", "target": ["mon-mgr", ""]}) v1 -- 0x7f824c100d10 con 0x7f82300357d0 2022-09-10T04:18:53.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.669+0000 7f82427fc700 1 -- 172.21.15.81:0/250651480 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f824c100d10 con 0x7f82300357d0 2022-09-10T04:18:53.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f82300357d0 msgr2=0x7f8230037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:53.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f82300357d0 0x7f8230037c80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f823c009750 tx=0x7f823c006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:53.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 msgr2=0x7f824c1990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:53.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1990e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8248009e90 tx=0x7f8248004040 comp rx=0 tx=0).stop 2022-09-10T04:18:53.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 shutdown_connections 2022-09-10T04:18:53.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f82300357d0 0x7f8230037c80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:53.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 --2- 172.21.15.81:0/250651480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f824c1033c0 0x7f824c1990e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:53.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 >> 172.21.15.81:0/250651480 conn(0x7f824c0fe9f0 msgr2=0x7f824c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:53.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 shutdown_connections 2022-09-10T04:18:53.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:52.672+0000 7f825368b700 1 -- 172.21.15.81:0/250651480 wait complete. 2022-09-10T04:18:53.043 INFO:teuthology.orchestra.run.smithi081.stdout:Deploying unmanaged mon service... 2022-09-10T04:18:54.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:53 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:54.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:53 smithi081 ceph-mon[90133]: Added host smithi081 2022-09-10T04:18:54.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:53 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:18:54.337 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2022-09-10T04:18:54.337 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.637+0000 7fc34769e700 1 Processor -- start 2022-09-10T04:18:54.337 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.637+0000 7fc34769e700 1 -- start start 2022-09-10T04:18:54.337 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.638+0000 7fc34769e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc3400782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:54.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.638+0000 7fc34769e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3400787e0 con 0x7fc340079dc0 2022-09-10T04:18:54.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.638+0000 7fc34543a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc3400782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:54.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.638+0000 7fc34543a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc3400782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33244/0 (socket says 172.21.15.81:33244) 2022-09-10T04:18:54.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.638+0000 7fc34543a700 1 -- 172.21.15.81:0/343604346 learned_addr learned my addr 172.21.15.81:0/343604346 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:54.339 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.638+0000 7fc34543a700 1 -- 172.21.15.81:0/343604346 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc340078920 con 0x7fc340079dc0 2022-09-10T04:18:54.339 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.639+0000 7fc34543a700 1 --2- 172.21.15.81:0/343604346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc3400782a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009700 tx=0x7fc33c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3383152b77ba1936 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:54.339 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.639+0000 7fc32ffff700 1 -- 172.21.15.81:0/343604346 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc33c004160 con 0x7fc340079dc0 2022-09-10T04:18:54.339 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.639+0000 7fc32ffff700 1 -- 172.21.15.81:0/343604346 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fc33c0042c0 con 0x7fc340079dc0 2022-09-10T04:18:54.340 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.639+0000 7fc32ffff700 1 -- 172.21.15.81:0/343604346 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc33c0039d0 con 0x7fc340079dc0 2022-09-10T04:18:54.340 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 -- 172.21.15.81:0/343604346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 msgr2=0x7fc3400782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:54.340 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 --2- 172.21.15.81:0/343604346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc3400782a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc33c009700 tx=0x7fc33c009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:54.340 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 -- 172.21.15.81:0/343604346 shutdown_connections 2022-09-10T04:18:54.341 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 --2- 172.21.15.81:0/343604346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc3400782a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:54.341 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 -- 172.21.15.81:0/343604346 >> 172.21.15.81:0/343604346 conn(0x7fc3400fe9f0 msgr2=0x7fc340100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:54.341 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 -- 172.21.15.81:0/343604346 shutdown_connections 2022-09-10T04:18:54.342 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.640+0000 7fc34769e700 1 -- 172.21.15.81:0/343604346 wait complete. 2022-09-10T04:18:54.342 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.641+0000 7fc34769e700 1 Processor -- start 2022-09-10T04:18:54.342 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.641+0000 7fc34769e700 1 -- start start 2022-09-10T04:18:54.342 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.642+0000 7fc34769e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc340199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:54.343 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.642+0000 7fc34769e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3401995d0 con 0x7fc340079dc0 2022-09-10T04:18:54.343 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.642+0000 7fc34543a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc340199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:54.343 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.642+0000 7fc34543a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc340199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33252/0 (socket says 172.21.15.81:33252) 2022-09-10T04:18:54.343 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.642+0000 7fc34543a700 1 -- 172.21.15.81:0/2219020085 learned_addr learned my addr 172.21.15.81:0/2219020085 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:54.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.642+0000 7fc34543a700 1 -- 172.21.15.81:0/2219020085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c0093e0 con 0x7fc340079dc0 2022-09-10T04:18:54.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.643+0000 7fc34543a700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc340199090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc33c003fb0 tx=0x7fc33c004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:54.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.643+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc33c0118c0 con 0x7fc340079dc0 2022-09-10T04:18:54.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.643+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fc33c011a20 con 0x7fc340079dc0 2022-09-10T04:18:54.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.643+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3401997d0 con 0x7fc340079dc0 2022-09-10T04:18:54.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.643+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc33c019550 con 0x7fc340079dc0 2022-09-10T04:18:54.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.643+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc340199c00 con 0x7fc340079dc0 2022-09-10T04:18:54.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.644+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7fc33c02a020 con 0x7fc340079dc0 2022-09-10T04:18:54.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.644+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc340060310 con 0x7fc340079dc0 2022-09-10T04:18:54.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.645+0000 7fc32e7fc700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fc328035810 0x7fc328037cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:54.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.645+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc33c048ab0 con 0x7fc340079dc0 2022-09-10T04:18:54.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.645+0000 7fc344c39700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fc328035810 0x7fc328037cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:54.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.646+0000 7fc344c39700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fc328035810 0x7fc328037cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc330009750 tx=0x7fc330006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:54.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.650+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc33c018d20 con 0x7fc340079dc0 2022-09-10T04:18:54.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.873+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc3400fff00 con 0x7fc328035810 2022-09-10T04:18:54.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.879+0000 7fc32e7fc700 1 -- 172.21.15.81:0/2219020085 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc3400fff00 con 0x7fc328035810 2022-09-10T04:18:54.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fc328035810 msgr2=0x7fc328037cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:54.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fc328035810 0x7fc328037cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc330009750 tx=0x7fc330006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:54.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 msgr2=0x7fc340199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:54.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc340199090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc33c003fb0 tx=0x7fc33c004090 comp rx=0 tx=0).stop 2022-09-10T04:18:54.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 shutdown_connections 2022-09-10T04:18:54.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fc328035810 0x7fc328037cc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:54.349 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 --2- 172.21.15.81:0/2219020085 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc340079dc0 0x7fc340199090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:54.349 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 >> 172.21.15.81:0/2219020085 conn(0x7fc3400fe9f0 msgr2=0x7fc3400ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:54.349 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 shutdown_connections 2022-09-10T04:18:54.349 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:53.881+0000 7fc34769e700 1 -- 172.21.15.81:0/2219020085 wait complete. 2022-09-10T04:18:54.350 INFO:teuthology.orchestra.run.smithi081.stdout:Deploying unmanaged mgr service... 2022-09-10T04:18:55.195 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:54 smithi081 ceph-mon[90133]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:55.196 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:54 smithi081 ceph-mon[90133]: Saving service mon spec with placement count:5 2022-09-10T04:18:55.197 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:54 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:55.635 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2022-09-10T04:18:55.636 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.923+0000 7f8bf92d7700 1 Processor -- start 2022-09-10T04:18:55.636 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.924+0000 7f8bf92d7700 1 -- start start 2022-09-10T04:18:55.636 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.924+0000 7f8bf92d7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf40782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:55.637 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.924+0000 7f8bf92d7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf40787e0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.637 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.924+0000 7f8bf2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf40782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:55.637 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.924+0000 7f8bf2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf40782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33262/0 (socket says 172.21.15.81:33262) 2022-09-10T04:18:55.637 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.924+0000 7f8bf2ffd700 1 -- 172.21.15.81:0/123721534 learned_addr learned my addr 172.21.15.81:0/123721534 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:55.638 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.925+0000 7f8bf2ffd700 1 -- 172.21.15.81:0/123721534 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf4078920 con 0x7f8bf4079dc0 2022-09-10T04:18:55.638 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.925+0000 7f8bf2ffd700 1 --2- 172.21.15.81:0/123721534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf40782a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009700 tx=0x7f8be4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc8c5590d095ed2 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:55.638 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.925+0000 7f8bf1ffb700 1 -- 172.21.15.81:0/123721534 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4004160 con 0x7f8bf4079dc0 2022-09-10T04:18:55.638 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.925+0000 7f8bf1ffb700 1 -- 172.21.15.81:0/123721534 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f8be40042c0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.925+0000 7f8bf1ffb700 1 -- 172.21.15.81:0/123721534 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be40039d0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 -- 172.21.15.81:0/123721534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 msgr2=0x7f8bf40782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:55.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 --2- 172.21.15.81:0/123721534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf40782a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8be4009700 tx=0x7f8be4009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:55.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 -- 172.21.15.81:0/123721534 shutdown_connections 2022-09-10T04:18:55.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 --2- 172.21.15.81:0/123721534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf40782a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:55.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 -- 172.21.15.81:0/123721534 >> 172.21.15.81:0/123721534 conn(0x7f8bf40fe9f0 msgr2=0x7f8bf4100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:55.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 -- 172.21.15.81:0/123721534 shutdown_connections 2022-09-10T04:18:55.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.926+0000 7f8bf92d7700 1 -- 172.21.15.81:0/123721534 wait complete. 2022-09-10T04:18:55.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.928+0000 7f8bf92d7700 1 Processor -- start 2022-09-10T04:18:55.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.928+0000 7f8bf92d7700 1 -- start start 2022-09-10T04:18:55.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.929+0000 7f8bf92d7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf4199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:55.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.929+0000 7f8bf92d7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf41995d0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.929+0000 7f8bf2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf4199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:55.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.929+0000 7f8bf2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf4199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33272/0 (socket says 172.21.15.81:33272) 2022-09-10T04:18:55.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.929+0000 7f8bf2ffd700 1 -- 172.21.15.81:0/1584122602 learned_addr learned my addr 172.21.15.81:0/1584122602 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:55.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.929+0000 7f8bf2ffd700 1 -- 172.21.15.81:0/1584122602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be40093e0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.930+0000 7f8bf2ffd700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf4199090 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8be4003fb0 tx=0x7f8be4004090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:55.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.930+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be40118c0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.930+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7f8be4011a20 con 0x7f8bf4079dc0 2022-09-10T04:18:55.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.930+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf41997d0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.930+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8be4019550 con 0x7f8bf4079dc0 2022-09-10T04:18:55.644 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.930+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf4199c00 con 0x7f8bf4079dc0 2022-09-10T04:18:55.644 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.931+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f8be402a020 con 0x7f8bf4079dc0 2022-09-10T04:18:55.644 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.931+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4060310 con 0x7f8bf4079dc0 2022-09-10T04:18:55.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.932+0000 7f8bdbfff700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8be0035870 0x7f8be0037d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:55.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.932+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8be4048ad0 con 0x7f8bf4079dc0 2022-09-10T04:18:55.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.932+0000 7f8bf27fc700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8be0035870 0x7f8be0037d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:55.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.933+0000 7f8bf27fc700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8be0035870 0x7f8be0037d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009750 tx=0x7f8be8006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:55.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:54.937+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8be4018d20 con 0x7f8bf4079dc0 2022-09-10T04:18:55.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.158+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8bf40fff00 con 0x7f8be0035870 2022-09-10T04:18:55.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.163+0000 7f8bdbfff700 1 -- 172.21.15.81:0/1584122602 <== mgr.14118 v2:172.21.15.81:6800/3224857868 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8bf40fff00 con 0x7f8be0035870 2022-09-10T04:18:55.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8be0035870 msgr2=0x7f8be0037d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:55.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8be0035870 0x7f8be0037d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009750 tx=0x7f8be8006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:55.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 msgr2=0x7f8bf4199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:55.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf4199090 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8be4003fb0 tx=0x7f8be4004090 comp rx=0 tx=0).stop 2022-09-10T04:18:55.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 shutdown_connections 2022-09-10T04:18:55.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8be0035870 0x7f8be0037d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:55.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 --2- 172.21.15.81:0/1584122602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bf4079dc0 0x7f8bf4199090 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:55.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 >> 172.21.15.81:0/1584122602 conn(0x7f8bf40fe9f0 msgr2=0x7f8bf40ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:55.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 shutdown_connections 2022-09-10T04:18:55.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:55.165+0000 7f8bf92d7700 1 -- 172.21.15.81:0/1584122602 wait complete. 2022-09-10T04:18:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:56 smithi081 ceph-mon[90133]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:18:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:56 smithi081 ceph-mon[90133]: Saving service mgr spec with placement count:2 2022-09-10T04:18:56.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:56 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:57.034 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.228+0000 7fde7d633700 1 Processor -- start 2022-09-10T04:18:57.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.228+0000 7fde7d633700 1 -- start start 2022-09-10T04:18:57.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.229+0000 7fde7d633700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:57.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.229+0000 7fde7d633700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde781062b0 con 0x7fde781058d0 2022-09-10T04:18:57.035 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.230+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:57.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.230+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33288/0 (socket says 172.21.15.81:33288) 2022-09-10T04:18:57.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.230+0000 7fde76ffd700 1 -- 172.21.15.81:0/2671274615 learned_addr learned my addr 172.21.15.81:0/2671274615 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:57.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.230+0000 7fde76ffd700 1 -- 172.21.15.81:0/2671274615 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde781063f0 con 0x7fde781058d0 2022-09-10T04:18:57.036 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.230+0000 7fde76ffd700 1 --2- 172.21.15.81:0/2671274615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78105ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde68009700 tx=0x7fde68009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e726cedc2d161ed5 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.231+0000 7fde75ffb700 1 -- 172.21.15.81:0/2671274615 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde68004160 con 0x7fde781058d0 2022-09-10T04:18:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.231+0000 7fde75ffb700 1 -- 172.21.15.81:0/2671274615 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fde680042c0 con 0x7fde781058d0 2022-09-10T04:18:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.231+0000 7fde75ffb700 1 -- 172.21.15.81:0/2671274615 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde680039d0 con 0x7fde781058d0 2022-09-10T04:18:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.231+0000 7fde7d633700 1 -- 172.21.15.81:0/2671274615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 msgr2=0x7fde78105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:57.037 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.231+0000 7fde7d633700 1 --2- 172.21.15.81:0/2671274615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78105ce0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde68009700 tx=0x7fde68009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.232+0000 7fde7d633700 1 -- 172.21.15.81:0/2671274615 shutdown_connections 2022-09-10T04:18:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.232+0000 7fde7d633700 1 --2- 172.21.15.81:0/2671274615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78105ce0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.232+0000 7fde7d633700 1 -- 172.21.15.81:0/2671274615 >> 172.21.15.81:0/2671274615 conn(0x7fde78078a20 msgr2=0x7fde78078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:57.038 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.232+0000 7fde7d633700 1 -- 172.21.15.81:0/2671274615 shutdown_connections 2022-09-10T04:18:57.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.232+0000 7fde7d633700 1 -- 172.21.15.81:0/2671274615 wait complete. 2022-09-10T04:18:57.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.234+0000 7fde7d633700 1 Processor -- start 2022-09-10T04:18:57.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.234+0000 7fde7d633700 1 -- start start 2022-09-10T04:18:57.039 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.234+0000 7fde7d633700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:57.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.234+0000 7fde7d633700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78199640 con 0x7fde781058d0 2022-09-10T04:18:57.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.235+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:57.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.235+0000 7fde76ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33304/0 (socket says 172.21.15.81:33304) 2022-09-10T04:18:57.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.235+0000 7fde76ffd700 1 -- 172.21.15.81:0/3350104191 learned_addr learned my addr 172.21.15.81:0/3350104191 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:57.040 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.235+0000 7fde76ffd700 1 -- 172.21.15.81:0/3350104191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680093e0 con 0x7fde781058d0 2022-09-10T04:18:57.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.235+0000 7fde76ffd700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78199100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde68003f10 tx=0x7fde68003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:57.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.236+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde68011950 con 0x7fde781058d0 2022-09-10T04:18:57.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.236+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1049+0+0 (secure 0 0 0) 0x7fde68011ab0 con 0x7fde781058d0 2022-09-10T04:18:57.041 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.236+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde68019550 con 0x7fde781058d0 2022-09-10T04:18:57.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.237+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde78199840 con 0x7fde781058d0 2022-09-10T04:18:57.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.237+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde78199c70 con 0x7fde781058d0 2022-09-10T04:18:57.042 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.238+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde781074e0 con 0x7fde781058d0 2022-09-10T04:18:57.043 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.239+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7fde68017020 con 0x7fde781058d0 2022-09-10T04:18:57.043 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.239+0000 7fde5ffff700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fde64035870 0x7fde64037d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:57.043 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.239+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fde68048c00 con 0x7fde781058d0 2022-09-10T04:18:57.043 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.239+0000 7fde767fc700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fde64035870 0x7fde64037d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:57.043 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.240+0000 7fde767fc700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fde64035870 0x7fde64037d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009750 tx=0x7fde6c006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:57.044 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.248+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fde68011c20 con 0x7fde781058d0 2022-09-10T04:18:57.044 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.481+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fde7819cc40 con 0x7fde781058d0 2022-09-10T04:18:57.044 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.485+0000 7fde5ffff700 1 -- 172.21.15.81:0/3350104191 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) v1 ==== 142+0+0 (secure 0 0 0) 0x7fde68023b80 con 0x7fde781058d0 2022-09-10T04:18:57.044 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.487+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fde64035870 msgr2=0x7fde64037d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:57.045 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.487+0000 7fde7d633700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fde64035870 0x7fde64037d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009750 tx=0x7fde6c006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:57.045 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.487+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 msgr2=0x7fde78199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:57.045 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.487+0000 7fde7d633700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78199100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fde68003f10 tx=0x7fde68003ff0 comp rx=0 tx=0).stop 2022-09-10T04:18:57.045 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.488+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 shutdown_connections 2022-09-10T04:18:57.045 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.488+0000 7fde7d633700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7fde64035870 0x7fde64037d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:57.046 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.488+0000 7fde7d633700 1 --2- 172.21.15.81:0/3350104191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde781058d0 0x7fde78199100 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:57.046 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.488+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 >> 172.21.15.81:0/3350104191 conn(0x7fde78078a20 msgr2=0x7fde78104090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:57.046 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.488+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 shutdown_connections 2022-09-10T04:18:57.046 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:56.488+0000 7fde7d633700 1 -- 172.21.15.81:0/3350104191 wait complete. 2022-09-10T04:18:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:57 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3350104191' entity='client.admin' 2022-09-10T04:18:58.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.972+0000 7f8a60b57700 1 Processor -- start 2022-09-10T04:18:58.639 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.972+0000 7f8a60b57700 1 -- start start 2022-09-10T04:18:58.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.972+0000 7f8a60b57700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:58.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.972+0000 7f8a60b57700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c1062b0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.640 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.973+0000 7f8a5a59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:58.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.973+0000 7f8a5a59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33312/0 (socket says 172.21.15.81:33312) 2022-09-10T04:18:58.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.973+0000 7f8a5a59c700 1 -- 172.21.15.81:0/3345840659 learned_addr learned my addr 172.21.15.81:0/3345840659 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:58.641 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.974+0000 7f8a5a59c700 1 -- 172.21.15.81:0/3345840659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c1063f0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.974+0000 7f8a5a59c700 1 --2- 172.21.15.81:0/3345840659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c105ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009700 tx=0x7f8a4c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591fce2c89b3c061 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:58.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.974+0000 7f8a5959a700 1 -- 172.21.15.81:0/3345840659 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c004160 con 0x7f8a5c1058d0 2022-09-10T04:18:58.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.974+0000 7f8a5959a700 1 -- 172.21.15.81:0/3345840659 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8a4c0042c0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.642 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.974+0000 7f8a5959a700 1 -- 172.21.15.81:0/3345840659 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c003a10 con 0x7f8a5c1058d0 2022-09-10T04:18:58.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 -- 172.21.15.81:0/3345840659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 msgr2=0x7f8a5c105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:58.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 --2- 172.21.15.81:0/3345840659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c105ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009700 tx=0x7f8a4c009a10 comp rx=0 tx=0).stop 2022-09-10T04:18:58.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 -- 172.21.15.81:0/3345840659 shutdown_connections 2022-09-10T04:18:58.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 --2- 172.21.15.81:0/3345840659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c105ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:58.643 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 -- 172.21.15.81:0/3345840659 >> 172.21.15.81:0/3345840659 conn(0x7f8a5c078a20 msgr2=0x7f8a5c078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:58.644 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 -- 172.21.15.81:0/3345840659 shutdown_connections 2022-09-10T04:18:58.644 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.975+0000 7f8a60b57700 1 -- 172.21.15.81:0/3345840659 wait complete. 2022-09-10T04:18:58.644 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.977+0000 7f8a60b57700 1 Processor -- start 2022-09-10T04:18:58.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.977+0000 7f8a60b57700 1 -- start start 2022-09-10T04:18:58.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.978+0000 7f8a60b57700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:58.645 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.978+0000 7f8a60b57700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c1996b0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.978+0000 7f8a5a59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:58.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.978+0000 7f8a5a59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33324/0 (socket says 172.21.15.81:33324) 2022-09-10T04:18:58.646 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.978+0000 7f8a5a59c700 1 -- 172.21.15.81:0/1338538087 learned_addr learned my addr 172.21.15.81:0/1338538087 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:18:58.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.978+0000 7f8a5a59c700 1 -- 172.21.15.81:0/1338538087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0093e0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.979+0000 7f8a5a59c700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c199170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c003560 tx=0x7f8a4c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:58.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.979+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c011840 con 0x7f8a5c1058d0 2022-09-10T04:18:58.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.979+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8a4c01a430 con 0x7f8a5c1058d0 2022-09-10T04:18:58.647 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.979+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c0193c0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.979+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c1998b0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.979+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c199ce0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.980+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f8a4c02a020 con 0x7f8a5c1058d0 2022-09-10T04:18:58.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.981+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c060310 con 0x7f8a5c1058d0 2022-09-10T04:18:58.648 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.981+0000 7f8a4b7fe700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8a40035830 0x7f8a40037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:18:58.649 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.981+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8a4c0487b0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.649 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.981+0000 7f8a59d9b700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8a40035830 0x7f8a40037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:18:58.649 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.982+0000 7f8a59d9b700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8a40035830 0x7f8a40037ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009750 tx=0x7f8a50006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:18:58.649 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:57.989+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8a4c022930 con 0x7f8a5c1058d0 2022-09-10T04:18:58.650 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.220+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f8a5c19ccf0 con 0x7f8a5c1058d0 2022-09-10T04:18:58.650 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.226+0000 7f8a4b7fe700 1 -- 172.21.15.81:0/1338538087 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8a4c00ee60 con 0x7f8a5c1058d0 2022-09-10T04:18:58.650 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8a40035830 msgr2=0x7f8a40037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:58.651 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8a40035830 0x7f8a40037ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009750 tx=0x7f8a50006cc0 comp rx=0 tx=0).stop 2022-09-10T04:18:58.651 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 msgr2=0x7f8a5c199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:18:58.651 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c199170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c003560 tx=0x7f8a4c003f60 comp rx=0 tx=0).stop 2022-09-10T04:18:58.652 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 shutdown_connections 2022-09-10T04:18:58.652 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f8a40035830 0x7f8a40037ce0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:58.652 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 --2- 172.21.15.81:0/1338538087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a5c1058d0 0x7f8a5c199170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:18:58.652 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 >> 172.21.15.81:0/1338538087 conn(0x7f8a5c078a20 msgr2=0x7f8a5c1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:18:58.653 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 shutdown_connections 2022-09-10T04:18:58.653 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:58.229+0000 7f8a60b57700 1 -- 172.21.15.81:0/1338538087 wait complete. 2022-09-10T04:18:58.653 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling the dashboard module... 2022-09-10T04:18:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:58 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:58 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1338538087' entity='client.admin' 2022-09-10T04:18:58.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:58 smithi081 ceph-mon[90133]: from='mgr.14118 172.21.15.81:0/3872574950' entity='mgr.x' 2022-09-10T04:18:59.869 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: ignoring --setuser ceph since I am not root 2022-09-10T04:18:59.869 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:18:59 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2000559563' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2022-09-10T04:19:00.120 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: ignoring --setgroup ceph since I am not root 2022-09-10T04:19:00.121 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:00.042+0000 7f8febac0080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:19:00.145 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.279+0000 7f49b6fcc700 1 Processor -- start 2022-09-10T04:19:00.151 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.279+0000 7f49b6fcc700 1 -- start start 2022-09-10T04:19:00.152 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.280+0000 7f49b6fcc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0102130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:00.152 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.280+0000 7f49b6fcc700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b0077d90 con 0x7f49b0101d20 2022-09-10T04:19:00.152 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.280+0000 7f49b4d68700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0102130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:00.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.280+0000 7f49b4d68700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0102130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33340/0 (socket says 172.21.15.81:33340) 2022-09-10T04:19:00.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.280+0000 7f49b4d68700 1 -- 172.21.15.81:0/3552277875 learned_addr learned my addr 172.21.15.81:0/3552277875 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:00.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.281+0000 7f49b4d68700 1 -- 172.21.15.81:0/3552277875 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0077ed0 con 0x7f49b0101d20 2022-09-10T04:19:00.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.281+0000 7f49b4d68700 1 --2- 172.21.15.81:0/3552277875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0102130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009700 tx=0x7f49a4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eef4893834f2894 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:00.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.281+0000 7f49af7fe700 1 -- 172.21.15.81:0/3552277875 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49a4004160 con 0x7f49b0101d20 2022-09-10T04:19:00.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.281+0000 7f49af7fe700 1 -- 172.21.15.81:0/3552277875 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f49a40042c0 con 0x7f49b0101d20 2022-09-10T04:19:00.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.281+0000 7f49af7fe700 1 -- 172.21.15.81:0/3552277875 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49a4003a10 con 0x7f49b0101d20 2022-09-10T04:19:00.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 -- 172.21.15.81:0/3552277875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 msgr2=0x7f49b0102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:00.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 --2- 172.21.15.81:0/3552277875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0102130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009700 tx=0x7f49a4009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:00.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 -- 172.21.15.81:0/3552277875 shutdown_connections 2022-09-10T04:19:00.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 --2- 172.21.15.81:0/3552277875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0102130 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:00.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 -- 172.21.15.81:0/3552277875 >> 172.21.15.81:0/3552277875 conn(0x7f49b00fd8f0 msgr2=0x7f49b00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:00.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 -- 172.21.15.81:0/3552277875 shutdown_connections 2022-09-10T04:19:00.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.282+0000 7f49b6fcc700 1 -- 172.21.15.81:0/3552277875 wait complete. 2022-09-10T04:19:00.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.283+0000 7f49b6fcc700 1 Processor -- start 2022-09-10T04:19:00.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.283+0000 7f49b6fcc700 1 -- start start 2022-09-10T04:19:00.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.284+0000 7f49b6fcc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:00.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.284+0000 7f49b6fcc700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b0199640 con 0x7f49b0101d20 2022-09-10T04:19:00.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.284+0000 7f49b4d68700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:00.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.284+0000 7f49b4d68700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:33352/0 (socket says 172.21.15.81:33352) 2022-09-10T04:19:00.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.284+0000 7f49b4d68700 1 -- 172.21.15.81:0/2000559563 learned_addr learned my addr 172.21.15.81:0/2000559563 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:00.158 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.284+0000 7f49b4d68700 1 -- 172.21.15.81:0/2000559563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a40093e0 con 0x7f49b0101d20 2022-09-10T04:19:00.158 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.285+0000 7f49b4d68700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0199100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f49a4003560 tx=0x7f49a4003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:00.158 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.285+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49a4011840 con 0x7f49b0101d20 2022-09-10T04:19:00.158 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.285+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b0199840 con 0x7f49b0101d20 2022-09-10T04:19:00.158 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.285+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f49a40119a0 con 0x7f49b0101d20 2022-09-10T04:19:00.159 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.285+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f49a40193c0 con 0x7f49b0101d20 2022-09-10T04:19:00.159 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.285+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b0199c70 con 0x7f49b0101d20 2022-09-10T04:19:00.159 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.286+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) v1 ==== 42690+0+0 (secure 0 0 0) 0x7f49a402a020 con 0x7f49b0101d20 2022-09-10T04:19:00.159 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.286+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b0060310 con 0x7f49b0101d20 2022-09-10T04:19:00.159 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.287+0000 7f49adffb700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4994035830 0x7f4994037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:00.160 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.287+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f49a40488a0 con 0x7f49b0101d20 2022-09-10T04:19:00.160 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.287+0000 7f49affff700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4994035830 0x7f4994037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:00.160 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.288+0000 7f49affff700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4994035830 0x7f4994037ce0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f499c009750 tx=0x7f499c006cc0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:00.160 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.292+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f49a402b940 con 0x7f49b0101d20 2022-09-10T04:19:00.161 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.599+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f49b005f990 con 0x7f49b0101d20 2022-09-10T04:19:00.161 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.626+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 9) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f49a4017020 con 0x7f49b0101d20 2022-09-10T04:19:00.161 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.628+0000 7f49adffb700 1 -- 172.21.15.81:0/2000559563 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f49a402bd30 con 0x7f49b0101d20 2022-09-10T04:19:00.161 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4994035830 msgr2=0x7f4994037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:00.161 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4994035830 0x7f4994037ce0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f499c009750 tx=0x7f499c006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:00.162 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 msgr2=0x7f49b0199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:00.162 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0199100 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f49a4003560 tx=0x7f49a4003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:00.162 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 shutdown_connections 2022-09-10T04:19:00.162 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4994035830 0x7f4994037ce0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:00.163 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 --2- 172.21.15.81:0/2000559563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49b0101d20 0x7f49b0199100 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:00.163 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.631+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 >> 172.21.15.81:0/2000559563 conn(0x7f49b00fd8f0 msgr2=0x7f49b018d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:00.164 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.632+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 shutdown_connections 2022-09-10T04:19:00.164 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:18:59.632+0000 7f49b6fcc700 1 -- 172.21.15.81:0/2000559563 wait complete. 2022-09-10T04:19:00.456 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:00.157+0000 7f8febac0080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:19:00.456 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:00.313+0000 7f8febac0080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:19:00.726 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2000559563' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2022-09-10T04:19:00.727 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-mon[90133]: mgrmap e9: x(active, since 17s) 2022-09-10T04:19:00.727 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:00.454+0000 7f8febac0080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:19:01.049 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:00.724+0000 7f8febac0080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:19:01.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:01.203+0000 7f8febac0080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:19:01.577 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:19:01.586 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 9, 2022-09-10T04:19:01.586 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2022-09-10T04:19:01.587 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "active_name": "x", 2022-09-10T04:19:01.587 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standby": 0 2022-09-10T04:19:01.587 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:19:01.587 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.899+0000 7f489797f700 1 Processor -- start 2022-09-10T04:19:01.588 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.899+0000 7f489797f700 1 -- start start 2022-09-10T04:19:01.588 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.899+0000 7f489797f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890078320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:01.588 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.899+0000 7f489797f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48900788f0 con 0x7f4890079e40 2022-09-10T04:19:01.588 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.900+0000 7f489571b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890078320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:01.588 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.900+0000 7f489571b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890078320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44962/0 (socket says 172.21.15.81:44962) 2022-09-10T04:19:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.900+0000 7f489571b700 1 -- 172.21.15.81:0/2646568987 learned_addr learned my addr 172.21.15.81:0/2646568987 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.901+0000 7f489571b700 1 -- 172.21.15.81:0/2646568987 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4890078a30 con 0x7f4890079e40 2022-09-10T04:19:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.901+0000 7f489571b700 1 --2- 172.21.15.81:0/2646568987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890078320 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f488c009700 tx=0x7f488c009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831ca86d907a86f server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:01.589 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.901+0000 7f487ffff700 1 -- 172.21.15.81:0/2646568987 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f488c004160 con 0x7f4890079e40 2022-09-10T04:19:01.590 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.901+0000 7f487ffff700 1 -- 172.21.15.81:0/2646568987 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f488c0042c0 con 0x7f4890079e40 2022-09-10T04:19:01.590 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.901+0000 7f487ffff700 1 -- 172.21.15.81:0/2646568987 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f488c003a10 con 0x7f4890079e40 2022-09-10T04:19:01.590 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.902+0000 7f489797f700 1 -- 172.21.15.81:0/2646568987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 msgr2=0x7f4890078320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:01.590 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.902+0000 7f489797f700 1 --2- 172.21.15.81:0/2646568987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890078320 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f488c009700 tx=0x7f488c009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:01.591 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.902+0000 7f489797f700 1 -- 172.21.15.81:0/2646568987 shutdown_connections 2022-09-10T04:19:01.591 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.902+0000 7f489797f700 1 --2- 172.21.15.81:0/2646568987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890078320 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:01.591 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.902+0000 7f489797f700 1 -- 172.21.15.81:0/2646568987 >> 172.21.15.81:0/2646568987 conn(0x7f48900fe9f0 msgr2=0x7f4890100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:01.592 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.902+0000 7f489797f700 1 -- 172.21.15.81:0/2646568987 shutdown_connections 2022-09-10T04:19:01.592 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.903+0000 7f489797f700 1 -- 172.21.15.81:0/2646568987 wait complete. 2022-09-10T04:19:01.593 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.904+0000 7f489797f700 1 Processor -- start 2022-09-10T04:19:01.593 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.904+0000 7f489797f700 1 -- start start 2022-09-10T04:19:01.594 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.905+0000 7f489797f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:01.594 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.905+0000 7f489797f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890199640 con 0x7f4890079e40 2022-09-10T04:19:01.595 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.905+0000 7f489571b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:01.595 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.905+0000 7f489571b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44972/0 (socket says 172.21.15.81:44972) 2022-09-10T04:19:01.596 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.905+0000 7f489571b700 1 -- 172.21.15.81:0/1191946188 learned_addr learned my addr 172.21.15.81:0/1191946188 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:01.596 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.906+0000 7f489571b700 1 -- 172.21.15.81:0/1191946188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0093e0 con 0x7f4890079e40 2022-09-10T04:19:01.597 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.906+0000 7f489571b700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890199100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f488c003560 tx=0x7f488c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:01.597 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.906+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f488c011840 con 0x7f4890079e40 2022-09-10T04:19:01.597 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.906+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f488c0119a0 con 0x7f4890079e40 2022-09-10T04:19:01.598 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.906+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f488c0193c0 con 0x7f4890079e40 2022-09-10T04:19:01.598 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.906+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4890199840 con 0x7f4890079e40 2022-09-10T04:19:01.599 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.907+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4890199c70 con 0x7f4890079e40 2022-09-10T04:19:01.599 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.907+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f488c02a020 con 0x7f4890079e40 2022-09-10T04:19:01.599 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.908+0000 7f487e7fc700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 0x7f4878037d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:01.599 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.908+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f488c048940 con 0x7f4890079e40 2022-09-10T04:19:01.599 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.908+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4890060310 con 0x7f4890079e40 2022-09-10T04:19:01.600 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.908+0000 7f4894f1a700 1 -- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 msgr2=0x7f4878037d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:01.600 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.908+0000 7f4894f1a700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 0x7f4878037d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:19:01.600 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:00.913+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f488c02b940 con 0x7f4890079e40 2022-09-10T04:19:01.600 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.109+0000 7f4894f1a700 1 -- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 msgr2=0x7f4878037d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:01.601 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.109+0000 7f4894f1a700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 0x7f4878037d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:19:01.601 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.187+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f489019cef0 con 0x7f4890079e40 2022-09-10T04:19:01.601 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.187+0000 7f487e7fc700 1 -- 172.21.15.81:0/1191946188 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f488c02b760 con 0x7f4890079e40 2022-09-10T04:19:01.602 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 msgr2=0x7f4878037d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:19:01.602 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 0x7f4878037d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:01.602 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 msgr2=0x7f4890199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:01.603 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890199100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f488c003560 tx=0x7f488c003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:01.604 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 shutdown_connections 2022-09-10T04:19:01.604 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f4878035870 0x7f4878037d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:01.605 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 --2- 172.21.15.81:0/1191946188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4890079e40 0x7f4890199100 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:01.605 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 >> 172.21.15.81:0/1191946188 conn(0x7f48900fe9f0 msgr2=0x7f48901000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:01.605 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 shutdown_connections 2022-09-10T04:19:01.606 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:01.190+0000 7f489797f700 1 -- 172.21.15.81:0/1191946188 wait complete. 2022-09-10T04:19:01.606 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for the mgr to restart... 2022-09-10T04:19:01.607 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr epoch 9... 2022-09-10T04:19:01.627 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:01.345+0000 7f8febac0080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:19:01.628 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:01.526+0000 7f8febac0080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:19:01.925 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:01.682+0000 7f8febac0080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:19:01.926 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:01 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1191946188' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2022-09-10T04:19:02.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:02.315+0000 7f8febac0080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:19:03.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:03.131+0000 7f8febac0080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:19:03.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:03.365+0000 7f8febac0080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:19:04.806 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:04.418+0000 7f8febac0080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:19:04.807 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:04.566+0000 7f8febac0080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:19:05.148 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:04.840+0000 7f8febac0080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:19:05.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:05.147+0000 7f8febac0080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:19:05.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:05.377+0000 7f8febac0080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:19:06.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:05.971+0000 7f8febac0080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:19:07.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:06.888+0000 7f8febac0080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:19:07.640 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:07.313+0000 7f8febac0080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:19:08.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:07.639+0000 7f8febac0080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:19:08.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:07.828+0000 7f8febac0080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:19:08.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:08.187+0000 7f8febac0080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:19:09.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:19:09.195+0000 7f8febac0080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:19:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:09 smithi081 ceph-mon[90133]: Active manager daemon x restarted 2022-09-10T04:19:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:09 smithi081 ceph-mon[90133]: Activating manager daemon x 2022-09-10T04:19:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:09 smithi081 ceph-mon[90133]: osdmap e3: 0 total, 0 up, 0 in 2022-09-10T04:19:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: mgrmap e10: x(active, starting, since 0.0534981s) 2022-09-10T04:19:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:19:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-10T04:19:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-10T04:19:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-10T04:19:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-10T04:19:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: Manager daemon x is now available 2022-09-10T04:19:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:10.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:19:10.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:19:10.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:10 smithi081 ceph-mon[90133]: [10/Sep/2022:04:19:09] ENGINE Bus STARTING 2022-09-10T04:19:10.708 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2022-09-10T04:19:10.709 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2022-09-10T04:19:10.709 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "initialized": true 2022-09-10T04:19:10.709 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2022-09-10T04:19:10.709 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f2268f76700 1 Processor -- start 2022-09-10T04:19:10.710 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f2268f76700 1 -- start start 2022-09-10T04:19:10.710 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f2268f76700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264078300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:10.710 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f2268f76700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2264078840 con 0x7f2264079e20 2022-09-10T04:19:10.710 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f226259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264078300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f226259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264078300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44984/0 (socket says 172.21.15.81:44984) 2022-09-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.233+0000 7f226259c700 1 -- 172.21.15.81:0/2040323005 learned_addr learned my addr 172.21.15.81:0/2040323005 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.234+0000 7f226259c700 1 -- 172.21.15.81:0/2040323005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264078980 con 0x7f2264079e20 2022-09-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.234+0000 7f226259c700 1 --2- 172.21.15.81:0/2040323005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264078300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2254009700 tx=0x7f2254009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599cf19bb7020d9d server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:10.711 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.234+0000 7f226159a700 1 -- 172.21.15.81:0/2040323005 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2254004160 con 0x7f2264079e20 2022-09-10T04:19:10.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.234+0000 7f226159a700 1 -- 172.21.15.81:0/2040323005 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f22540042c0 con 0x7f2264079e20 2022-09-10T04:19:10.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.234+0000 7f226159a700 1 -- 172.21.15.81:0/2040323005 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2254003a10 con 0x7f2264079e20 2022-09-10T04:19:10.713 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 -- 172.21.15.81:0/2040323005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 msgr2=0x7f2264078300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:10.713 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 --2- 172.21.15.81:0/2040323005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264078300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2254009700 tx=0x7f2254009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:10.713 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 -- 172.21.15.81:0/2040323005 shutdown_connections 2022-09-10T04:19:10.713 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 --2- 172.21.15.81:0/2040323005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264078300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:10.713 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 -- 172.21.15.81:0/2040323005 >> 172.21.15.81:0/2040323005 conn(0x7f22640fe9f0 msgr2=0x7f2264100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:10.714 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 -- 172.21.15.81:0/2040323005 shutdown_connections 2022-09-10T04:19:10.714 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.236+0000 7f2268f76700 1 -- 172.21.15.81:0/2040323005 wait complete. 2022-09-10T04:19:10.715 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f2268f76700 1 Processor -- start 2022-09-10T04:19:10.715 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f2268f76700 1 -- start start 2022-09-10T04:19:10.715 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f2268f76700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:10.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f2268f76700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2264199650 con 0x7f2264079e20 2022-09-10T04:19:10.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f226259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:10.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f226259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:44992/0 (socket says 172.21.15.81:44992) 2022-09-10T04:19:10.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.237+0000 7f226259c700 1 -- 172.21.15.81:0/88758642 learned_addr learned my addr 172.21.15.81:0/88758642 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:10.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f226259c700 1 -- 172.21.15.81:0/88758642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22540093e0 con 0x7f2264079e20 2022-09-10T04:19:10.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f226259c700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264199110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2254003560 tx=0x7f2254003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:10.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2254011840 con 0x7f2264079e20 2022-09-10T04:19:10.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f22540119a0 con 0x7f2264079e20 2022-09-10T04:19:10.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2264199850 con 0x7f2264079e20 2022-09-10T04:19:10.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22540193c0 con 0x7f2264079e20 2022-09-10T04:19:10.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.239+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2264199c80 con 0x7f2264079e20 2022-09-10T04:19:10.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.240+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f225402a020 con 0x7f2264079e20 2022-09-10T04:19:10.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.240+0000 7f22537fe700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:10.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.240+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 --> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22480383f0 con 0x7f2248035830 2022-09-10T04:19:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.240+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f22540489b0 con 0x7f2264079e20 2022-09-10T04:19:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.240+0000 7f2261d9b700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.240+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:19:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.442+0000 7f2261d9b700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:10.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.442+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:19:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.843+0000 7f2261d9b700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:02.843+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-09-10T04:19:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:03.644+0000 7f2261d9b700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:10.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:03.644+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-09-10T04:19:10.721 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:05.245+0000 7f2261d9b700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:10.721 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:05.245+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-09-10T04:19:10.721 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:08.447+0000 7f2261d9b700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3224857868 2022-09-10T04:19:10.722 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:08.447+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-09-10T04:19:10.722 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:09.250+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mgrmap(e 10) v1 ==== 42513+0+0 (secure 0 0 0) 0x7f2254017020 con 0x7f2264079e20 2022-09-10T04:19:10.723 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:09.250+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 msgr2=0x7f2248037ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:19:10.723 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:09.250+0000 7f22537fe700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:10.724 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.259+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 11) v1 ==== 42623+0+0 (secure 0 0 0) 0x7f225400f050 con 0x7f2264079e20 2022-09-10T04:19:10.724 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.259+0000 7f22537fe700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2248039290 0x7f224803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:10.724 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.259+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22480383f0 con 0x7f2248039290 2022-09-10T04:19:10.725 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.260+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2248039290 0x7f224803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:10.725 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.261+0000 7f2261d9b700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2248039290 0x7f224803b670 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258006dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:10.725 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.261+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6424 (secure 0 0 0) 0x7f22480383f0 con 0x7f2248039290 2022-09-10T04:19:10.725 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.268+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f226419a440 con 0x7f2248039290 2022-09-10T04:19:10.726 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.268+0000 7f22537fe700 1 -- 172.21.15.81:0/88758642 <== mgr.14150 v2:172.21.15.81:6800/532852594 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f226419a440 con 0x7f2248039290 2022-09-10T04:19:10.726 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2248039290 msgr2=0x7f224803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:10.726 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2248039290 0x7f224803b670 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258006dc0 comp rx=0 tx=0).stop 2022-09-10T04:19:10.726 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 msgr2=0x7f2264199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:10.727 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264199110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2254003560 tx=0x7f2254003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:10.727 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 shutdown_connections 2022-09-10T04:19:10.727 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2264079e20 0x7f2264199110 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:10.727 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2248039290 0x7f224803b670 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:10.727 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 --2- 172.21.15.81:0/88758642 >> [v2:172.21.15.81:6800/3224857868,v1:172.21.15.81:6801/3224857868] conn(0x7f2248035830 0x7f2248037ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:10.728 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 >> 172.21.15.81:0/88758642 conn(0x7f22640fe9f0 msgr2=0x7f22641000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:10.728 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 shutdown_connections 2022-09-10T04:19:10.728 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:10.269+0000 7f2268f76700 1 -- 172.21.15.81:0/88758642 wait complete. 2022-09-10T04:19:10.728 INFO:teuthology.orchestra.run.smithi081.stdout:mgr epoch 9 is available 2022-09-10T04:19:10.729 INFO:teuthology.orchestra.run.smithi081.stdout:Generating a dashboard self-signed certificate... 2022-09-10T04:19:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:11 smithi081 ceph-mon[90133]: [10/Sep/2022:04:19:09] ENGINE Serving on https://:::8765 2022-09-10T04:19:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:11 smithi081 ceph-mon[90133]: [10/Sep/2022:04:19:10] ENGINE Serving on https://172.21.15.81:7150 2022-09-10T04:19:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:11 smithi081 ceph-mon[90133]: [10/Sep/2022:04:19:10] ENGINE Bus STARTED 2022-09-10T04:19:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:11 smithi081 ceph-mon[90133]: mgrmap e11: x(active, since 1.0631s) 2022-09-10T04:19:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:11 smithi081 ceph-mon[90133]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2022-09-10T04:19:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:11 smithi081 ceph-mon[90133]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2022-09-10T04:19:12.311 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2022-09-10T04:19:12.311 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.425+0000 7f58c30d3700 1 Processor -- start 2022-09-10T04:19:12.311 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.426+0000 7f58c30d3700 1 -- start start 2022-09-10T04:19:12.312 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.426+0000 7f58c30d3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc1081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:12.312 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.426+0000 7f58c30d3700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc103bd0 con 0x7f58bc077c60 2022-09-10T04:19:12.312 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.427+0000 7f58c20d1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc1081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:12.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.427+0000 7f58c20d1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc1081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38702/0 (socket says 172.21.15.81:38702) 2022-09-10T04:19:12.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.427+0000 7f58c20d1700 1 -- 172.21.15.81:0/3847176771 learned_addr learned my addr 172.21.15.81:0/3847176771 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:12.313 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.427+0000 7f58c20d1700 1 -- 172.21.15.81:0/3847176771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc108720 con 0x7f58bc077c60 2022-09-10T04:19:12.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.427+0000 7f58c20d1700 1 --2- 172.21.15.81:0/3847176771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc1081e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009700 tx=0x7f58b8009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866309c786a43f8f server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:12.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.428+0000 7f58c10cf700 1 -- 172.21.15.81:0/3847176771 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58b8004160 con 0x7f58bc077c60 2022-09-10T04:19:12.314 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.428+0000 7f58c10cf700 1 -- 172.21.15.81:0/3847176771 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f58b80042c0 con 0x7f58bc077c60 2022-09-10T04:19:12.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.428+0000 7f58c10cf700 1 -- 172.21.15.81:0/3847176771 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58b8003a10 con 0x7f58bc077c60 2022-09-10T04:19:12.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.428+0000 7f58c30d3700 1 -- 172.21.15.81:0/3847176771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 msgr2=0x7f58bc1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:12.315 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.428+0000 7f58c30d3700 1 --2- 172.21.15.81:0/3847176771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc1081e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009700 tx=0x7f58b8009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:12.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.429+0000 7f58c30d3700 1 -- 172.21.15.81:0/3847176771 shutdown_connections 2022-09-10T04:19:12.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.429+0000 7f58c30d3700 1 --2- 172.21.15.81:0/3847176771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc1081e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:12.316 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.429+0000 7f58c30d3700 1 -- 172.21.15.81:0/3847176771 >> 172.21.15.81:0/3847176771 conn(0x7f58bc0fd7d0 msgr2=0x7f58bc0ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:12.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.429+0000 7f58c30d3700 1 -- 172.21.15.81:0/3847176771 shutdown_connections 2022-09-10T04:19:12.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.429+0000 7f58c30d3700 1 -- 172.21.15.81:0/3847176771 wait complete. 2022-09-10T04:19:12.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.430+0000 7f58c30d3700 1 Processor -- start 2022-09-10T04:19:12.317 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.431+0000 7f58c30d3700 1 -- start start 2022-09-10T04:19:12.318 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.431+0000 7f58c30d3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:12.318 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.431+0000 7f58c30d3700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1994a0 con 0x7f58bc077c60 2022-09-10T04:19:12.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.431+0000 7f58c20d1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:12.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.431+0000 7f58c20d1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38716/0 (socket says 172.21.15.81:38716) 2022-09-10T04:19:12.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.431+0000 7f58c20d1700 1 -- 172.21.15.81:0/3275854667 learned_addr learned my addr 172.21.15.81:0/3275854667 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:12.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.432+0000 7f58c20d1700 1 -- 172.21.15.81:0/3275854667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b80093e0 con 0x7f58bc077c60 2022-09-10T04:19:12.319 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.432+0000 7f58c20d1700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc198f60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58b8003560 tx=0x7f58b8003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:12.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.432+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58b8011840 con 0x7f58bc077c60 2022-09-10T04:19:12.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.432+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f58b801a430 con 0x7f58bc077c60 2022-09-10T04:19:12.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.433+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc1996a0 con 0x7f58bc077c60 2022-09-10T04:19:12.320 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.433+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58b80193c0 con 0x7f58bc077c60 2022-09-10T04:19:12.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.433+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc199ad0 con 0x7f58bc077c60 2022-09-10T04:19:12.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.434+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc109450 con 0x7f58bc077c60 2022-09-10T04:19:12.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.434+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 12) v1 ==== 42703+0+0 (secure 0 0 0) 0x7f58b802a020 con 0x7f58bc077c60 2022-09-10T04:19:12.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.435+0000 7f58ab7fe700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f58a40358c0 0x7f58a4037d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:12.321 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.435+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f58b80488d0 con 0x7f58bc077c60 2022-09-10T04:19:12.322 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.435+0000 7f58c18d0700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f58a40358c0 0x7f58a4037d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:12.322 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.436+0000 7f58c18d0700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f58a40358c0 0x7f58a4037d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009750 tx=0x7f58ac006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:12.322 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.439+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f58b801a5a0 con 0x7f58bc077c60 2022-09-10T04:19:12.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.679+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f58bc0ffac0 con 0x7f58a40358c0 2022-09-10T04:19:12.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.807+0000 7f58ab7fe700 1 -- 172.21.15.81:0/3275854667 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f58bc0ffac0 con 0x7f58a40358c0 2022-09-10T04:19:12.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f58a40358c0 msgr2=0x7f58a4037d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:12.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f58a40358c0 0x7f58a4037d70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009750 tx=0x7f58ac006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:12.323 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 msgr2=0x7f58bc198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:12.324 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc198f60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58b8003560 tx=0x7f58b8003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:12.324 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 shutdown_connections 2022-09-10T04:19:12.324 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f58a40358c0 0x7f58a4037d70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:12.324 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 --2- 172.21.15.81:0/3275854667 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58bc077c60 0x7f58bc198f60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:12.325 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 >> 172.21.15.81:0/3275854667 conn(0x7f58bc0fd7d0 msgr2=0x7f58bc0fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:12.325 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 shutdown_connections 2022-09-10T04:19:12.325 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:11.811+0000 7f58c30d3700 1 -- 172.21.15.81:0/3275854667 wait complete. 2022-09-10T04:19:12.325 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial admin user... 2022-09-10T04:19:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:12 smithi081 ceph-mon[90133]: mgrmap e12: x(active, since 2s) 2022-09-10T04:19:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:12 smithi081 ceph-mon[90133]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:13.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:13.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$o1HVkwc1vvMrvWc1wJFIouBhbvym.GwwKA8JrOmLK/QiSaENlafL6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1662783553, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2022-09-10T04:19:13.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.927+0000 7fbc35c46700 1 Processor -- start 2022-09-10T04:19:13.816 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.928+0000 7fbc35c46700 1 -- start start 2022-09-10T04:19:13.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.928+0000 7fbc35c46700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc300782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:13.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.928+0000 7fbc35c46700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30078800 con 0x7fbc30079de0 2022-09-10T04:19:13.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.929+0000 7fbc2f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc300782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:13.817 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.929+0000 7fbc2f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc300782c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38720/0 (socket says 172.21.15.81:38720) 2022-09-10T04:19:13.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.929+0000 7fbc2f7fe700 1 -- 172.21.15.81:0/1806043307 learned_addr learned my addr 172.21.15.81:0/1806043307 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:13.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.929+0000 7fbc2f7fe700 1 -- 172.21.15.81:0/1806043307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30078940 con 0x7fbc30079de0 2022-09-10T04:19:13.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.930+0000 7fbc2f7fe700 1 --2- 172.21.15.81:0/1806043307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc300782c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009700 tx=0x7fbc20009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c0281dc6166285 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:13.818 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.930+0000 7fbc2e7fc700 1 -- 172.21.15.81:0/1806043307 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc20004160 con 0x7fbc30079de0 2022-09-10T04:19:13.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.930+0000 7fbc2e7fc700 1 -- 172.21.15.81:0/1806043307 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fbc200042c0 con 0x7fbc30079de0 2022-09-10T04:19:13.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.930+0000 7fbc2e7fc700 1 -- 172.21.15.81:0/1806043307 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc20003a10 con 0x7fbc30079de0 2022-09-10T04:19:13.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.930+0000 7fbc35c46700 1 -- 172.21.15.81:0/1806043307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 msgr2=0x7fbc300782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:13.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.930+0000 7fbc35c46700 1 --2- 172.21.15.81:0/1806043307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc300782c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009700 tx=0x7fbc20009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:13.819 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.931+0000 7fbc35c46700 1 -- 172.21.15.81:0/1806043307 shutdown_connections 2022-09-10T04:19:13.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.931+0000 7fbc35c46700 1 --2- 172.21.15.81:0/1806043307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc300782c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:13.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.931+0000 7fbc35c46700 1 -- 172.21.15.81:0/1806043307 >> 172.21.15.81:0/1806043307 conn(0x7fbc300fe9f0 msgr2=0x7fbc30100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:13.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.932+0000 7fbc35c46700 1 -- 172.21.15.81:0/1806043307 shutdown_connections 2022-09-10T04:19:13.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.932+0000 7fbc35c46700 1 -- 172.21.15.81:0/1806043307 wait complete. 2022-09-10T04:19:13.820 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.933+0000 7fbc35c46700 1 Processor -- start 2022-09-10T04:19:13.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.934+0000 7fbc35c46700 1 -- start start 2022-09-10T04:19:13.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.934+0000 7fbc35c46700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc30199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:13.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.934+0000 7fbc35c46700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30199660 con 0x7fbc30079de0 2022-09-10T04:19:13.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.934+0000 7fbc2f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc30199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:13.821 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.934+0000 7fbc2f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc30199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38724/0 (socket says 172.21.15.81:38724) 2022-09-10T04:19:13.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.934+0000 7fbc2f7fe700 1 -- 172.21.15.81:0/1325217201 learned_addr learned my addr 172.21.15.81:0/1325217201 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:13.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc2f7fe700 1 -- 172.21.15.81:0/1325217201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc200093e0 con 0x7fbc30079de0 2022-09-10T04:19:13.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc2f7fe700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc30199120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc20003560 tx=0x7fbc20003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:13.822 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc20011840 con 0x7fbc30079de0 2022-09-10T04:19:13.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fbc200119a0 con 0x7fbc30079de0 2022-09-10T04:19:13.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc30199860 con 0x7fbc30079de0 2022-09-10T04:19:13.823 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc200193c0 con 0x7fbc30079de0 2022-09-10T04:19:13.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.935+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc30199c90 con 0x7fbc30079de0 2022-09-10T04:19:13.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.936+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 12) v1 ==== 42703+0+0 (secure 0 0 0) 0x7fbc2002a020 con 0x7fbc30079de0 2022-09-10T04:19:13.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.936+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc30060310 con 0x7fbc30079de0 2022-09-10T04:19:13.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.937+0000 7fbc2cff9700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbc1c035870 0x7fbc1c037d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:13.824 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.937+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbc20048a10 con 0x7fbc30079de0 2022-09-10T04:19:13.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.937+0000 7fbc2effd700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbc1c035870 0x7fbc1c037d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:13.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.938+0000 7fbc2effd700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbc1c035870 0x7fbc1c037d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009750 tx=0x7fbc24006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:13.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:12.942+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fbc20022700 con 0x7fbc30079de0 2022-09-10T04:19:13.825 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.177+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fbc3019cca0 con 0x7fbc1c035870 2022-09-10T04:19:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.407+0000 7fbc2cff9700 1 -- 172.21.15.81:0/1325217201 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fbc3019cca0 con 0x7fbc1c035870 2022-09-10T04:19:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbc1c035870 msgr2=0x7fbc1c037d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbc1c035870 0x7fbc1c037d20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009750 tx=0x7fbc24006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 msgr2=0x7fbc30199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:13.826 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc30199120 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc20003560 tx=0x7fbc20003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:13.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 shutdown_connections 2022-09-10T04:19:13.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbc1c035870 0x7fbc1c037d20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:13.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 --2- 172.21.15.81:0/1325217201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc30079de0 0x7fbc30199120 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:13.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 >> 172.21.15.81:0/1325217201 conn(0x7fbc300fe9f0 msgr2=0x7fbc30100110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:13.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 shutdown_connections 2022-09-10T04:19:13.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:13.410+0000 7fbc35c46700 1 -- 172.21.15.81:0/1325217201 wait complete. 2022-09-10T04:19:13.828 INFO:teuthology.orchestra.run.smithi081.stdout:Fetching dashboard port number... 2022-09-10T04:19:14.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:14 smithi081 ceph-mon[90133]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:14.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:14.775 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:15.230 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 8443 2022-09-10T04:19:15.230 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.504+0000 7ff5b4b38700 1 Processor -- start 2022-09-10T04:19:15.231 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.505+0000 7ff5b4b38700 1 -- start start 2022-09-10T04:19:15.231 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.505+0000 7ff5b4b38700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0105ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:15.231 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.505+0000 7ff5b4b38700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b01062b0 con 0x7ff5b01058d0 2022-09-10T04:19:15.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.505+0000 7ff5ae59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0105ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:15.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.506+0000 7ff5ae59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0105ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38728/0 (socket says 172.21.15.81:38728) 2022-09-10T04:19:15.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.506+0000 7ff5ae59c700 1 -- 172.21.15.81:0/2386829908 learned_addr learned my addr 172.21.15.81:0/2386829908 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:15.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.506+0000 7ff5ae59c700 1 -- 172.21.15.81:0/2386829908 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b01063f0 con 0x7ff5b01058d0 2022-09-10T04:19:15.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.506+0000 7ff5ae59c700 1 --2- 172.21.15.81:0/2386829908 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0105ce0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009700 tx=0x7ff5a0009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49198adcb3724ad1 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:15.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.507+0000 7ff5ad59a700 1 -- 172.21.15.81:0/2386829908 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5a0004160 con 0x7ff5b01058d0 2022-09-10T04:19:15.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.507+0000 7ff5ad59a700 1 -- 172.21.15.81:0/2386829908 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff5a00042c0 con 0x7ff5b01058d0 2022-09-10T04:19:15.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.507+0000 7ff5ad59a700 1 -- 172.21.15.81:0/2386829908 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5a0003a10 con 0x7ff5b01058d0 2022-09-10T04:19:15.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.507+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2386829908 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 msgr2=0x7ff5b0105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:15.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.507+0000 7ff5b4b38700 1 --2- 172.21.15.81:0/2386829908 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0105ce0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009700 tx=0x7ff5a0009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:15.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.508+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2386829908 shutdown_connections 2022-09-10T04:19:15.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.508+0000 7ff5b4b38700 1 --2- 172.21.15.81:0/2386829908 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0105ce0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:15.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.508+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2386829908 >> 172.21.15.81:0/2386829908 conn(0x7ff5b0078a20 msgr2=0x7ff5b0078e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:15.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.508+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2386829908 shutdown_connections 2022-09-10T04:19:15.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.508+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2386829908 wait complete. 2022-09-10T04:19:15.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.509+0000 7ff5b4b38700 1 Processor -- start 2022-09-10T04:19:15.236 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.509+0000 7ff5b4b38700 1 -- start start 2022-09-10T04:19:15.236 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.509+0000 7ff5b4b38700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:15.236 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.509+0000 7ff5b4b38700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b01996b0 con 0x7ff5b01058d0 2022-09-10T04:19:15.236 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.510+0000 7ff5ae59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:15.236 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.510+0000 7ff5ae59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38740/0 (socket says 172.21.15.81:38740) 2022-09-10T04:19:15.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.510+0000 7ff5ae59c700 1 -- 172.21.15.81:0/2941718159 learned_addr learned my addr 172.21.15.81:0/2941718159 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:15.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.510+0000 7ff5ae59c700 1 -- 172.21.15.81:0/2941718159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a00093e0 con 0x7ff5b01058d0 2022-09-10T04:19:15.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.511+0000 7ff5ae59c700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0199170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0003560 tx=0x7ff5a0003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:15.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.511+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5a0011840 con 0x7ff5b01058d0 2022-09-10T04:19:15.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.511+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff5a001a430 con 0x7ff5b01058d0 2022-09-10T04:19:15.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.511+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b01998b0 con 0x7ff5b01058d0 2022-09-10T04:19:15.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.511+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff5a00193c0 con 0x7ff5b01058d0 2022-09-10T04:19:15.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.511+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b0199ce0 con 0x7ff5b01058d0 2022-09-10T04:19:15.239 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.512+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b0107500 con 0x7ff5b01058d0 2022-09-10T04:19:15.239 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.513+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 12) v1 ==== 42703+0+0 (secure 0 0 0) 0x7ff5a002a020 con 0x7ff5b01058d0 2022-09-10T04:19:15.239 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.513+0000 7ff59f7fe700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff594035830 0x7ff594037ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:15.239 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.513+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff5a0048900 con 0x7ff5b01058d0 2022-09-10T04:19:15.240 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.513+0000 7ff5add9b700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff594035830 0x7ff594037ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:15.240 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.514+0000 7ff5add9b700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff594035830 0x7ff594037ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4009750 tx=0x7ff5a4006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:15.240 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.518+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff5a001a5a0 con 0x7ff5b01058d0 2022-09-10T04:19:15.240 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.740+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff5b019cb20 con 0x7ff5b01058d0 2022-09-10T04:19:15.241 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.740+0000 7ff59f7fe700 1 -- 172.21.15.81:0/2941718159 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff5a002bb30 con 0x7ff5b01058d0 2022-09-10T04:19:15.241 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff594035830 msgr2=0x7ff594037ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:15.241 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff594035830 0x7ff594037ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4009750 tx=0x7ff5a4006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:15.241 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 msgr2=0x7ff5b0199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:15.242 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0199170 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0003560 tx=0x7ff5a0003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:15.242 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 shutdown_connections 2022-09-10T04:19:15.242 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff594035830 0x7ff594037ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:15.242 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 --2- 172.21.15.81:0/2941718159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5b01058d0 0x7ff5b0199170 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:15.243 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 >> 172.21.15.81:0/2941718159 conn(0x7ff5b0078a20 msgr2=0x7ff5b01037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:15.243 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 shutdown_connections 2022-09-10T04:19:15.243 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:14.743+0000 7ff5b4b38700 1 -- 172.21.15.81:0/2941718159 wait complete. 2022-09-10T04:19:15.253 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2022-09-10T04:19:15.253 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2022-09-10T04:19:15.275 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2022-09-10T04:19:15.275 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2022-09-10T04:19:15.275 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2022-09-10T04:19:15.275 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2022-09-10T04:19:15.277 INFO:teuthology.orchestra.run.smithi081.stdout:Ceph Dashboard is now available at: 2022-09-10T04:19:15.278 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:15.278 INFO:teuthology.orchestra.run.smithi081.stdout: URL: https://smithi081.front.sepia.ceph.com:8443/ 2022-09-10T04:19:15.278 INFO:teuthology.orchestra.run.smithi081.stdout: User: admin 2022-09-10T04:19:15.278 INFO:teuthology.orchestra.run.smithi081.stdout: Password: b13u1v8kgz 2022-09-10T04:19:15.279 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:15.279 INFO:teuthology.orchestra.run.smithi081.stdout:Saving cluster configuration to /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config directory 2022-09-10T04:19:15.280 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling autotune for osd_memory_target 2022-09-10T04:19:15.540 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:15 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2941718159' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2022-09-10T04:19:16.685 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.897+0000 7f74b0258700 1 Processor -- start 2022-09-10T04:19:16.686 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.898+0000 7f74b0258700 1 -- start start 2022-09-10T04:19:16.686 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.898+0000 7f74b0258700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a81037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:16.686 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.898+0000 7f74b0258700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a8103da0 con 0x7f74a81033c0 2022-09-10T04:19:16.687 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.899+0000 7f74adff4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a81037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:16.687 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.899+0000 7f74adff4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a81037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38754/0 (socket says 172.21.15.81:38754) 2022-09-10T04:19:16.687 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.899+0000 7f74adff4700 1 -- 172.21.15.81:0/867878109 learned_addr learned my addr 172.21.15.81:0/867878109 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:16.687 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.899+0000 7f74adff4700 1 -- 172.21.15.81:0/867878109 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8103ee0 con 0x7f74a81033c0 2022-09-10T04:19:16.688 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.899+0000 7f74adff4700 1 --2- 172.21.15.81:0/867878109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a81037d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009700 tx=0x7f74a4009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57d32ca272e53ac server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:16.688 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.900+0000 7f74acff2700 1 -- 172.21.15.81:0/867878109 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74a4004160 con 0x7f74a81033c0 2022-09-10T04:19:16.688 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.900+0000 7f74acff2700 1 -- 172.21.15.81:0/867878109 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f74a40042c0 con 0x7f74a81033c0 2022-09-10T04:19:16.688 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.900+0000 7f74acff2700 1 -- 172.21.15.81:0/867878109 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74a4003a10 con 0x7f74a81033c0 2022-09-10T04:19:16.689 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.900+0000 7f74b0258700 1 -- 172.21.15.81:0/867878109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 msgr2=0x7f74a81037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:16.689 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.900+0000 7f74b0258700 1 --2- 172.21.15.81:0/867878109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a81037d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009700 tx=0x7f74a4009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:16.689 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.901+0000 7f74b0258700 1 -- 172.21.15.81:0/867878109 shutdown_connections 2022-09-10T04:19:16.689 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.901+0000 7f74b0258700 1 --2- 172.21.15.81:0/867878109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a81037d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:16.690 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.901+0000 7f74b0258700 1 -- 172.21.15.81:0/867878109 >> 172.21.15.81:0/867878109 conn(0x7f74a80fe9f0 msgr2=0x7f74a8100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:16.690 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.901+0000 7f74b0258700 1 -- 172.21.15.81:0/867878109 shutdown_connections 2022-09-10T04:19:16.690 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.901+0000 7f74b0258700 1 -- 172.21.15.81:0/867878109 wait complete. 2022-09-10T04:19:16.690 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.902+0000 7f74b0258700 1 Processor -- start 2022-09-10T04:19:16.691 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.902+0000 7f74b0258700 1 -- start start 2022-09-10T04:19:16.691 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.903+0000 7f74b0258700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a8079c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:16.691 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.903+0000 7f74b0258700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a807a180 con 0x7f74a81033c0 2022-09-10T04:19:16.691 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.903+0000 7f74adff4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a8079c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:16.691 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.903+0000 7f74adff4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a8079c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38766/0 (socket says 172.21.15.81:38766) 2022-09-10T04:19:16.692 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.903+0000 7f74adff4700 1 -- 172.21.15.81:0/3432723968 learned_addr learned my addr 172.21.15.81:0/3432723968 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:16.692 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.903+0000 7f74adff4700 1 -- 172.21.15.81:0/3432723968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a40093e0 con 0x7f74a81033c0 2022-09-10T04:19:16.692 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.904+0000 7f74adff4700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a8079c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74a4003560 tx=0x7f74a4003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:16.692 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.904+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74a4011840 con 0x7f74a81033c0 2022-09-10T04:19:16.693 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.904+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f74a40119a0 con 0x7f74a81033c0 2022-09-10T04:19:16.693 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.904+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74a40193c0 con 0x7f74a81033c0 2022-09-10T04:19:16.693 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.904+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a8078280 con 0x7f74a81033c0 2022-09-10T04:19:16.693 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.904+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a80786b0 con 0x7f74a81033c0 2022-09-10T04:19:16.693 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.905+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f74a402a020 con 0x7f74a81033c0 2022-09-10T04:19:16.694 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.905+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a81052b0 con 0x7f74a81033c0 2022-09-10T04:19:16.694 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.906+0000 7f7496ffd700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7490035910 0x7f7490037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:16.694 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.906+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f74a4048a60 con 0x7f74a81033c0 2022-09-10T04:19:16.694 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.906+0000 7f74ad7f3700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7490035910 0x7f7490037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:16.695 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.907+0000 7f74ad7f3700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7490035910 0x7f7490037dc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7498009750 tx=0x7f7498006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:16.695 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:15.911+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f74a4011b10 con 0x7f74a81033c0 2022-09-10T04:19:16.695 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.141+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f74a819cff0 con 0x7f74a81033c0 2022-09-10T04:19:16.695 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.141+0000 7f7496ffd700 1 -- 172.21.15.81:0/3432723968 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f74a402bb30 con 0x7f74a81033c0 2022-09-10T04:19:16.696 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.143+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7490035910 msgr2=0x7f7490037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:16.696 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.143+0000 7f74b0258700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7490035910 0x7f7490037dc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7498009750 tx=0x7f7498006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:16.696 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.143+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 msgr2=0x7f74a8079c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:16.696 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.143+0000 7f74b0258700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a8079c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74a4003560 tx=0x7f74a4003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:16.696 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.144+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 shutdown_connections 2022-09-10T04:19:16.697 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.144+0000 7f74b0258700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7490035910 0x7f7490037dc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:16.697 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.144+0000 7f74b0258700 1 --2- 172.21.15.81:0/3432723968 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74a81033c0 0x7f74a8079c40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:16.697 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.144+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 >> 172.21.15.81:0/3432723968 conn(0x7f74a80fe9f0 msgr2=0x7f74a81000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:16.697 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.144+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 shutdown_connections 2022-09-10T04:19:16.697 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:16.144+0000 7f74b0258700 1 -- 172.21.15.81:0/3432723968 wait complete. 2022-09-10T04:19:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:16 smithi081 ceph-mon[90133]: mgrmap e13: x(active, since 6s) 2022-09-10T04:19:17.992 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.270+0000 7f3f3422f700 1 Processor -- start 2022-09-10T04:19:17.993 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.270+0000 7f3f3422f700 1 -- start start 2022-09-10T04:19:17.993 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.270+0000 7f3f3422f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c078320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:17.994 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.270+0000 7f3f3422f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0788f0 con 0x7f3f2c079e40 2022-09-10T04:19:17.994 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.271+0000 7f3f31fcb700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c078320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:17.995 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.271+0000 7f3f31fcb700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c078320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38772/0 (socket says 172.21.15.81:38772) 2022-09-10T04:19:17.995 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.271+0000 7f3f31fcb700 1 -- 172.21.15.81:0/2016439832 learned_addr learned my addr 172.21.15.81:0/2016439832 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:17.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.271+0000 7f3f31fcb700 1 -- 172.21.15.81:0/2016439832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c078a30 con 0x7f3f2c079e40 2022-09-10T04:19:17.996 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.271+0000 7f3f31fcb700 1 --2- 172.21.15.81:0/2016439832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c078320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009700 tx=0x7f3f28009a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abdd4ae9eb6cdbc server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:17.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f30fc9700 1 -- 172.21.15.81:0/2016439832 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f28004160 con 0x7f3f2c079e40 2022-09-10T04:19:17.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f30fc9700 1 -- 172.21.15.81:0/2016439832 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3f280042c0 con 0x7f3f2c079e40 2022-09-10T04:19:17.997 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f30fc9700 1 -- 172.21.15.81:0/2016439832 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f28003a10 con 0x7f3f2c079e40 2022-09-10T04:19:17.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f3422f700 1 -- 172.21.15.81:0/2016439832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 msgr2=0x7f3f2c078320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:17.998 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f3422f700 1 --2- 172.21.15.81:0/2016439832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c078320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009700 tx=0x7f3f28009a10 comp rx=0 tx=0).stop 2022-09-10T04:19:17.999 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f3422f700 1 -- 172.21.15.81:0/2016439832 shutdown_connections 2022-09-10T04:19:17.999 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f3422f700 1 --2- 172.21.15.81:0/2016439832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c078320 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:18.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f3422f700 1 -- 172.21.15.81:0/2016439832 >> 172.21.15.81:0/2016439832 conn(0x7f3f2c0fe9f0 msgr2=0x7f3f2c100e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:18.000 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.272+0000 7f3f3422f700 1 -- 172.21.15.81:0/2016439832 shutdown_connections 2022-09-10T04:19:18.001 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.273+0000 7f3f3422f700 1 -- 172.21.15.81:0/2016439832 wait complete. 2022-09-10T04:19:18.001 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f3422f700 1 Processor -- start 2022-09-10T04:19:18.002 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f3422f700 1 -- start start 2022-09-10T04:19:18.002 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f3422f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:18.003 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f3422f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c199640 con 0x7f3f2c079e40 2022-09-10T04:19:18.003 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f31fcb700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:18.003 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f31fcb700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38776/0 (socket says 172.21.15.81:38776) 2022-09-10T04:19:18.004 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.274+0000 7f3f31fcb700 1 -- 172.21.15.81:0/3604428277 learned_addr learned my addr 172.21.15.81:0/3604428277 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:18.004 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.275+0000 7f3f31fcb700 1 -- 172.21.15.81:0/3604428277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f280093e0 con 0x7f3f2c079e40 2022-09-10T04:19:18.005 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.275+0000 7f3f31fcb700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c199100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3f28003560 tx=0x7f3f28003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:18.005 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.275+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f28011840 con 0x7f3f2c079e40 2022-09-10T04:19:18.006 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.275+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3f280119a0 con 0x7f3f2c079e40 2022-09-10T04:19:18.006 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.275+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f280193c0 con 0x7f3f2c079e40 2022-09-10T04:19:18.007 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.275+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c199840 con 0x7f3f2c079e40 2022-09-10T04:19:18.007 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.276+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c199c70 con 0x7f3f2c079e40 2022-09-10T04:19:18.008 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.277+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f3f2802a020 con 0x7f3f2c079e40 2022-09-10T04:19:18.008 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.277+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c060310 con 0x7f3f2c079e40 2022-09-10T04:19:18.008 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.277+0000 7f3f1affd700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3f14035910 0x7f3f14037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:18.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.277+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3f280489f0 con 0x7f3f2c079e40 2022-09-10T04:19:18.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.277+0000 7f3f317ca700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3f14035910 0x7f3f14037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:18.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.278+0000 7f3f317ca700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3f14035910 0x7f3f14037dc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009750 tx=0x7f3f1c006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:18.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.282+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3f2802b940 con 0x7f3f2c079e40 2022-09-10T04:19:18.011 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.595+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f3f2c19cab0 con 0x7f3f2c079e40 2022-09-10T04:19:18.011 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.597+0000 7f3f1affd700 1 -- 172.21.15.81:0/3604428277 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f3f2802b760 con 0x7f3f2c079e40 2022-09-10T04:19:18.011 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3f14035910 msgr2=0x7f3f14037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:18.012 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3f14035910 0x7f3f14037dc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009750 tx=0x7f3f1c006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:18.012 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 msgr2=0x7f3f2c199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:18.012 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c199100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3f28003560 tx=0x7f3f28003f60 comp rx=0 tx=0).stop 2022-09-10T04:19:18.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 shutdown_connections 2022-09-10T04:19:18.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3f14035910 0x7f3f14037dc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:18.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 --2- 172.21.15.81:0/3604428277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c079e40 0x7f3f2c199100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:18.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 >> 172.21.15.81:0/3604428277 conn(0x7f3f2c0fe9f0 msgr2=0x7f3f2c1000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:18.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 shutdown_connections 2022-09-10T04:19:18.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2022-09-10T04:19:17.600+0000 7f3f3422f700 1 -- 172.21.15.81:0/3604428277 wait complete. 2022-09-10T04:19:18.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2022-09-10T04:19:18.016 INFO:teuthology.orchestra.run.smithi081.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2022-09-10T04:19:18.016 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.017 INFO:teuthology.orchestra.run.smithi081.stdout:Or, if you are only running a single cluster on this host: 2022-09-10T04:19:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.017 INFO:teuthology.orchestra.run.smithi081.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2022-09-10T04:19:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.018 INFO:teuthology.orchestra.run.smithi081.stdout:Please consider enabling telemetry to help improve Ceph: 2022-09-10T04:19:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.018 INFO:teuthology.orchestra.run.smithi081.stdout: ceph telemetry on 2022-09-10T04:19:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.019 INFO:teuthology.orchestra.run.smithi081.stdout:For more information see: 2022-09-10T04:19:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2022-09-10T04:19:18.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:18.020 INFO:teuthology.orchestra.run.smithi081.stdout:Bootstrap complete. 2022-09-10T04:19:18.059 INFO:tasks.cephadm:Fetching config... 2022-09-10T04:19:18.060 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:19:18.061 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2022-09-10T04:19:18.127 INFO:tasks.cephadm:Fetching client.admin keyring... 2022-09-10T04:19:18.127 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:19:18.128 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2022-09-10T04:19:18.188 INFO:tasks.cephadm:Fetching mon keyring... 2022-09-10T04:19:18.188 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:19:18.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/keyring of=/dev/stdout 2022-09-10T04:19:18.282 INFO:tasks.cephadm:Fetching pub ssh key... 2022-09-10T04:19:18.282 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:19:18.283 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2022-09-10T04:19:18.345 INFO:tasks.cephadm:Installing pub ssh key for root users... 2022-09-10T04:19:18.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-09-10T04:19:18.467 INFO:teuthology.orchestra.run.smithi081.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:19:18.503 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-09-10T04:19:18.549 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:19:18.562 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2022-09-10T04:19:18.608 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8UpKqqejaE/uqkK7vyEs3SwZergcfNd9+KPOyxR7qVJBUtvrnyoAxKYgbJx1zcH56NE3INy6MJY+5AlMelz+Zfh34ToHOK5oVoe/11qrMnDDuikzrBV3r6/i1h1K7RpfgSbl6xC/Rbp68qyIyABg/WzqQY/FuYZVFwv1xVbZ37dPogIpTg+oEMwVWRCR0DYo1hYR4mFa0caVVPaUpseuOqIT4pRFOpMN1rR/paYZomv137mMcHsL6036sDCqDOy+RtB6u3qA1g4LWC7ZJUf3LqTJ69f7bUS6ttpdxMt8zURVjkTq9EXcNY1eP5uNFah7lp3sjxShs/ZAkjCvVJ86AFc8idKoILAzIsFuf52Tc0c17k6+tvosWOzMzu0cRSBiu9VCg4Vm1TFu/yoUG2IOn2wVUChq4mlcQeZ7ks9qToqssrIcn1gwAxIJ/9hC2KPrmEkfaCH1oY4paIPHe8AXACPFk3hpB/Hmm+0wU8c0ftcznz7VVeV1lCaPetxT7PW8= ceph-70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:19:18.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2022-09-10T04:19:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:18 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3604428277' entity='client.admin' 2022-09-10T04:19:19.733 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:20.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:20.964 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:20.964 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:19:20.965 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:20.965 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:20.966 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:21.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 -- 172.21.15.81:0/3861892706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 msgr2=0x7fa70c0ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:21.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 --2- 172.21.15.81:0/3861892706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c0ffe70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009790 tx=0x7fa6fc009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:21.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 -- 172.21.15.81:0/3861892706 shutdown_connections 2022-09-10T04:19:21.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 --2- 172.21.15.81:0/3861892706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c0ffe70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:21.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 -- 172.21.15.81:0/3861892706 >> 172.21.15.81:0/3861892706 conn(0x7fa70c0fb2f0 msgr2=0x7fa70c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:21.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 -- 172.21.15.81:0/3861892706 shutdown_connections 2022-09-10T04:19:21.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.257+0000 7fa71204d700 1 -- 172.21.15.81:0/3861892706 wait complete. 2022-09-10T04:19:21.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.259+0000 7fa71204d700 1 Processor -- start 2022-09-10T04:19:21.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.259+0000 7fa71204d700 1 -- start start 2022-09-10T04:19:21.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.260+0000 7fa71204d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c194a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:21.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.260+0000 7fa71204d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c194f40 con 0x7fa70c0ffa60 2022-09-10T04:19:21.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.260+0000 7fa70b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c194a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:21.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.260+0000 7fa70b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c194a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40726/0 (socket says 172.21.15.81:40726) 2022-09-10T04:19:21.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.260+0000 7fa70b7fe700 1 -- 172.21.15.81:0/2786117418 learned_addr learned my addr 172.21.15.81:0/2786117418 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:21.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.260+0000 7fa70b7fe700 1 -- 172.21.15.81:0/2786117418 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc009470 con 0x7fa70c0ffa60 2022-09-10T04:19:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.261+0000 7fa70b7fe700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c194a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0059c0 tx=0x7fa6fc005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.261+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6fc01d070 con 0x7fa70c0ffa60 2022-09-10T04:19:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.261+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa6fc00fe80 con 0x7fa70c0ffa60 2022-09-10T04:19:21.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.261+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c195140 con 0x7fa70c0ffa60 2022-09-10T04:19:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.261+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6fc00b460 con 0x7fa70c0ffa60 2022-09-10T04:19:21.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.261+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c1955f0 con 0x7fa70c0ffa60 2022-09-10T04:19:21.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.262+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fa6fc028020 con 0x7fa70c0ffa60 2022-09-10T04:19:21.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.262+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c105730 con 0x7fa70c0ffa60 2022-09-10T04:19:21.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.262+0000 7fa708ff9700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa6f4035910 0x7fa6f4037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:21.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.263+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa6fc048c50 con 0x7fa70c0ffa60 2022-09-10T04:19:21.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.263+0000 7fa70affd700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa6f4035910 0x7fa6f4037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:21.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.263+0000 7fa70affd700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa6f4035910 0x7fa6f4037dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa700009750 tx=0x7fa700006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:21.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.268+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa6fc017440 con 0x7fa70c0ffa60 2022-09-10T04:19:21.486 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.485+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa70c05fae0 con 0x7fa70c0ffa60 2022-09-10T04:19:21.490 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.488+0000 7fa708ff9700 1 -- 172.21.15.81:0/2786117418 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa6fc021bb0 con 0x7fa70c0ffa60 2022-09-10T04:19:21.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa6f4035910 msgr2=0x7fa6f4037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:21.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa6f4035910 0x7fa6f4037dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa700009750 tx=0x7fa700006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:21.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 msgr2=0x7fa70c194a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:21.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c194a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0059c0 tx=0x7fa6fc005420 comp rx=0 tx=0).stop 2022-09-10T04:19:21.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 shutdown_connections 2022-09-10T04:19:21.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa6f4035910 0x7fa6f4037dc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:21.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 --2- 172.21.15.81:0/2786117418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa70c0ffa60 0x7fa70c194a00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:21.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 >> 172.21.15.81:0/2786117418 conn(0x7fa70c0fb2f0 msgr2=0x7fa70c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:21.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 shutdown_connections 2022-09-10T04:19:21.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:21.491+0000 7fa71204d700 1 -- 172.21.15.81:0/2786117418 wait complete. 2022-09-10T04:19:22.015 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2022-09-10T04:19:22.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2022-09-10T04:19:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:22 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2786117418' entity='client.admin' 2022-09-10T04:19:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:22.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:22.879 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/3356034311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 msgr2=0x7fcdb0100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 --2- 172.21.15.81:0/3356034311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb0100720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009790 tx=0x7fcdac009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/3356034311 shutdown_connections 2022-09-10T04:19:24.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 --2- 172.21.15.81:0/3356034311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb0100720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:24.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/3356034311 >> 172.21.15.81:0/3356034311 conn(0x7fcdb00f7d60 msgr2=0x7fcdb00fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:24.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/3356034311 shutdown_connections 2022-09-10T04:19:24.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.491+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/3356034311 wait complete. 2022-09-10T04:19:24.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.492+0000 7fcdb7b1f700 1 Processor -- start 2022-09-10T04:19:24.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.493+0000 7fcdb7b1f700 1 -- start start 2022-09-10T04:19:24.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.493+0000 7fcdb7b1f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb01927a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:24.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.493+0000 7fcdb7b1f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb0192ce0 con 0x7fcdb00fe340 2022-09-10T04:19:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.493+0000 7fcdb58bb700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb01927a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.493+0000 7fcdb58bb700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb01927a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40738/0 (socket says 172.21.15.81:40738) 2022-09-10T04:19:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.493+0000 7fcdb58bb700 1 -- 172.21.15.81:0/750611463 learned_addr learned my addr 172.21.15.81:0/750611463 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcdb58bb700 1 -- 172.21.15.81:0/750611463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac009470 con 0x7fcdb00fe340 2022-09-10T04:19:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcdb58bb700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb01927a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0059c0 tx=0x7fcdac00f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdac01d070 con 0x7fcdb00fe340 2022-09-10T04:19:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcdac00fde0 con 0x7fcdb00fe340 2022-09-10T04:19:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb0192ee0 con 0x7fcdb00fe340 2022-09-10T04:19:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdac00b460 con 0x7fcdb00fe340 2022-09-10T04:19:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.494+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb0193390 con 0x7fcdb00fe340 2022-09-10T04:19:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.495+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fcdac028020 con 0x7fcdb00fe340 2022-09-10T04:19:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.495+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb0103500 con 0x7fcdb00fe340 2022-09-10T04:19:24.503 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.496+0000 7fcda6ffd700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcd9c035910 0x7fcd9c037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:24.503 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.496+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcdac048c10 con 0x7fcdb00fe340 2022-09-10T04:19:24.503 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.496+0000 7fcdb50ba700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcd9c035910 0x7fcd9c037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.497+0000 7fcdb50ba700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcd9c035910 0x7fcd9c037dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009750 tx=0x7fcda0006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.501+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcdac017440 con 0x7fcdb00fe340 2022-09-10T04:19:24.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.717+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdb005fae0 con 0x7fcd9c035910 2022-09-10T04:19:24.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.720+0000 7fcda6ffd700 1 -- 172.21.15.81:0/750611463 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcdb005fae0 con 0x7fcd9c035910 2022-09-10T04:19:24.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcd9c035910 msgr2=0x7fcd9c037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:24.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcd9c035910 0x7fcd9c037dc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009750 tx=0x7fcda0006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:24.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 msgr2=0x7fcdb01927a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:24.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb01927a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0059c0 tx=0x7fcdac00f460 comp rx=0 tx=0).stop 2022-09-10T04:19:24.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 shutdown_connections 2022-09-10T04:19:24.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcd9c035910 0x7fcd9c037dc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:24.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 --2- 172.21.15.81:0/750611463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb00fe340 0x7fcdb01927a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:24.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 >> 172.21.15.81:0/750611463 conn(0x7fcdb00f7d60 msgr2=0x7fcdb00f8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:24.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 shutdown_connections 2022-09-10T04:19:24.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:24.723+0000 7fcdb7b1f700 1 -- 172.21.15.81:0/750611463 wait complete. 2022-09-10T04:19:25.151 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2022-09-10T04:19:25.151 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:19:25.152 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2022-09-10T04:19:25.173 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:19:25.174 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:25.233 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2022-09-10T04:19:25.234 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch host add smithi104 2022-09-10T04:19:25.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:25 smithi081 ceph-mon[90133]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:25.968 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:25.968 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:25.968 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:25.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:19:25.969 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:25 smithi081 ceph-mon[90133]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:19:26.124 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:26.842 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:26 smithi081 ceph-mon[90133]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:19:27.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 -- 172.21.15.81:0/48665868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 msgr2=0x7f56cc0660c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:27.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 --2- 172.21.15.81:0/48665868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc0660c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f56bc009790 tx=0x7f56bc009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:27.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 -- 172.21.15.81:0/48665868 shutdown_connections 2022-09-10T04:19:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 --2- 172.21.15.81:0/48665868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc0660c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 -- 172.21.15.81:0/48665868 >> 172.21.15.81:0/48665868 conn(0x7f56cc0727f0 msgr2=0x7f56cc072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:27.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 -- 172.21.15.81:0/48665868 shutdown_connections 2022-09-10T04:19:27.701 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.697+0000 7f56d25c2700 1 -- 172.21.15.81:0/48665868 wait complete. 2022-09-10T04:19:27.703 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.698+0000 7f56d25c2700 1 Processor -- start 2022-09-10T04:19:27.703 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.699+0000 7f56d25c2700 1 -- start start 2022-09-10T04:19:27.703 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.699+0000 7f56d25c2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc194e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.699+0000 7f56d25c2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc195380 con 0x7f56cc065cf0 2022-09-10T04:19:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.699+0000 7f56cbfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc194e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.699+0000 7f56cbfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc194e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40754/0 (socket says 172.21.15.81:40754) 2022-09-10T04:19:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.700+0000 7f56cbfff700 1 -- 172.21.15.81:0/770582231 learned_addr learned my addr 172.21.15.81:0/770582231 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.700+0000 7f56cbfff700 1 -- 172.21.15.81:0/770582231 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc009470 con 0x7f56cc065cf0 2022-09-10T04:19:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.700+0000 7f56cbfff700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc194e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56bc005dd0 tx=0x7f56bc005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.701+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56bc01d070 con 0x7f56cc065cf0 2022-09-10T04:19:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.701+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc195580 con 0x7f56cc065cf0 2022-09-10T04:19:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.701+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f56bc00fcc0 con 0x7f56cc065cf0 2022-09-10T04:19:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.701+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56bc017e20 con 0x7f56cc065cf0 2022-09-10T04:19:27.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.701+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc1959b0 con 0x7f56cc065cf0 2022-09-10T04:19:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.702+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f56bc028020 con 0x7f56cc065cf0 2022-09-10T04:19:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.702+0000 7f56c97fa700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f56b4035910 0x7f56b4037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.702+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f56bc048bf0 con 0x7f56cc065cf0 2022-09-10T04:19:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.703+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc060310 con 0x7f56cc065cf0 2022-09-10T04:19:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.703+0000 7f56cb7fe700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f56b4035910 0x7f56b4037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.703+0000 7f56cb7fe700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f56b4035910 0x7f56b4037dc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009750 tx=0x7f56c0006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:27.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.708+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f56bc021d10 con 0x7f56cc065cf0 2022-09-10T04:19:27.953 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:27.952+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}) v1 -- 0x7f56cc196190 con 0x7f56b4035910 2022-09-10T04:19:28.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:27 smithi081 ceph-mon[90133]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:28 smithi081 ceph-mon[90133]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:19:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:29.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:29.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.820+0000 7f56c97fa700 1 -- 172.21.15.81:0/770582231 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f56cc196190 con 0x7f56b4035910 2022-09-10T04:19:29.822 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi104' with addr '172.21.15.104' 2022-09-10T04:19:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f56b4035910 msgr2=0x7f56b4037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f56b4035910 0x7f56b4037dc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f56c0009750 tx=0x7f56c0006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 msgr2=0x7f56cc194e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:29.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc194e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56bc005dd0 tx=0x7f56bc005390 comp rx=0 tx=0).stop 2022-09-10T04:19:29.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 shutdown_connections 2022-09-10T04:19:29.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f56b4035910 0x7f56b4037dc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:29.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 --2- 172.21.15.81:0/770582231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56cc065cf0 0x7f56cc194e40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 >> 172.21.15.81:0/770582231 conn(0x7f56cc0727f0 msgr2=0x7f56cc0fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 shutdown_connections 2022-09-10T04:19:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:29.823+0000 7f56d25c2700 1 -- 172.21.15.81:0/770582231 wait complete. 2022-09-10T04:19:30.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:29 smithi081 ceph-mon[90133]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:30.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:29 smithi081 ceph-mon[90133]: Deploying cephadm binary to smithi104 2022-09-10T04:19:30.304 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch host ls --format=json 2022-09-10T04:19:31.173 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:31.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:31.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: Added host smithi104 2022-09-10T04:19:31.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:31.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:31.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:19:31.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:32.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:32.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:31 smithi081 ceph-mon[90133]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:32.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.694+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1241873813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 msgr2=0x7f2e20103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:32.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.694+0000 7f2e26e9d700 1 --2- 172.21.15.81:0/1241873813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e20103c30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009790 tx=0x7f2e14009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:32.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.694+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1241873813 shutdown_connections 2022-09-10T04:19:32.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.694+0000 7f2e26e9d700 1 --2- 172.21.15.81:0/1241873813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e20103c30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:32.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.694+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1241873813 >> 172.21.15.81:0/1241873813 conn(0x7f2e200f9420 msgr2=0x7f2e200fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:32.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.694+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1241873813 shutdown_connections 2022-09-10T04:19:32.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.695+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1241873813 wait complete. 2022-09-10T04:19:32.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.696+0000 7f2e26e9d700 1 Processor -- start 2022-09-10T04:19:32.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.696+0000 7f2e26e9d700 1 -- start start 2022-09-10T04:19:32.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.696+0000 7f2e26e9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e2019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:32.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.696+0000 7f2e26e9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2019af10 con 0x7f2e20103860 2022-09-10T04:19:32.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.697+0000 7f2e24c39700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e2019a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:32.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.697+0000 7f2e24c39700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e2019a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51768/0 (socket says 172.21.15.81:51768) 2022-09-10T04:19:32.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.697+0000 7f2e24c39700 1 -- 172.21.15.81:0/1657045001 learned_addr learned my addr 172.21.15.81:0/1657045001 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:32.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.697+0000 7f2e24c39700 1 -- 172.21.15.81:0/1657045001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14009470 con 0x7f2e20103860 2022-09-10T04:19:32.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.697+0000 7f2e24c39700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e2019a9d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e140059c0 tx=0x7f2e14005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:32.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.698+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e1401d070 con 0x7f2e20103860 2022-09-10T04:19:32.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.698+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2e1400fd80 con 0x7f2e20103860 2022-09-10T04:19:32.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.698+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e14017df0 con 0x7f2e20103860 2022-09-10T04:19:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.698+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2019b110 con 0x7f2e20103860 2022-09-10T04:19:32.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.698+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e20194a40 con 0x7f2e20103860 2022-09-10T04:19:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.699+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e20060310 con 0x7f2e20103860 2022-09-10T04:19:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.699+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f2e14028020 con 0x7f2e20103860 2022-09-10T04:19:32.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.700+0000 7f2e1dffb700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2e10035910 0x7f2e10037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:32.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.700+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2e140489c0 con 0x7f2e20103860 2022-09-10T04:19:32.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.700+0000 7f2e1ffff700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2e10035910 0x7f2e10037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:32.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.701+0000 7f2e1ffff700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2e10035910 0x7f2e10037dc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009750 tx=0x7f2e0c006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:32.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.705+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f2e1400f8e0 con 0x7f2e20103860 2022-09-10T04:19:32.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.920+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e20195f50 con 0x7f2e10035910 2022-09-10T04:19:32.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.921+0000 7f2e1dffb700 1 -- 172.21.15.81:0/1657045001 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f2e20195f50 con 0x7f2e10035910 2022-09-10T04:19:32.923 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:32.924 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2022-09-10T04:19:32.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2e10035910 msgr2=0x7f2e10037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:32.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2e10035910 0x7f2e10037dc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009750 tx=0x7f2e0c006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:32.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 msgr2=0x7f2e2019a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:32.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e2019a9d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e140059c0 tx=0x7f2e14005390 comp rx=0 tx=0).stop 2022-09-10T04:19:32.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 shutdown_connections 2022-09-10T04:19:32.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2e10035910 0x7f2e10037dc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:32.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 --2- 172.21.15.81:0/1657045001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e20103860 0x7f2e2019a9d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:32.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 >> 172.21.15.81:0/1657045001 conn(0x7f2e200f9420 msgr2=0x7f2e200729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:32.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 shutdown_connections 2022-09-10T04:19:32.929 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:32.924+0000 7f2e26e9d700 1 -- 172.21.15.81:0/1657045001 wait complete. 2022-09-10T04:19:33.275 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:33 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:33.335 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi110 2022-09-10T04:19:33.335 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:19:33.336 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.conf 2022-09-10T04:19:33.359 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:19:33.359 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:33.419 INFO:tasks.cephadm:Adding host smithi110 to orchestrator... 2022-09-10T04:19:33.420 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch host add smithi110 2022-09-10T04:19:34.293 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:34 smithi081 ceph-mon[90133]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:19:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:34 smithi081 ceph-mon[90133]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:35.818 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 -- 172.21.15.81:0/1610839151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 msgr2=0x7f678c065cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:35.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 --2- 172.21.15.81:0/1610839151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c065cf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f677c009790 tx=0x7f677c009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:35.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 -- 172.21.15.81:0/1610839151 shutdown_connections 2022-09-10T04:19:35.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 --2- 172.21.15.81:0/1610839151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c065cf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:35.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 -- 172.21.15.81:0/1610839151 >> 172.21.15.81:0/1610839151 conn(0x7f678c0727f0 msgr2=0x7f678c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:35.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 -- 172.21.15.81:0/1610839151 shutdown_connections 2022-09-10T04:19:35.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.816+0000 7f67914f3700 1 -- 172.21.15.81:0/1610839151 wait complete. 2022-09-10T04:19:35.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.817+0000 7f67914f3700 1 Processor -- start 2022-09-10T04:19:35.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.818+0000 7f67914f3700 1 -- start start 2022-09-10T04:19:35.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.818+0000 7f67914f3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:35.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.818+0000 7f67914f3700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c199880 con 0x7f678c102a40 2022-09-10T04:19:35.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.818+0000 7f678affd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:35.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.818+0000 7f678affd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51786/0 (socket says 172.21.15.81:51786) 2022-09-10T04:19:35.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.818+0000 7f678affd700 1 -- 172.21.15.81:0/872933904 learned_addr learned my addr 172.21.15.81:0/872933904 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:35.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f678affd700 1 -- 172.21.15.81:0/872933904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c009470 con 0x7f678c102a40 2022-09-10T04:19:35.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f678affd700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c199340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f677c005dd0 tx=0x7f677c005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:35.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f677c01d070 con 0x7f678c102a40 2022-09-10T04:19:35.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f677c00fcc0 con 0x7f678c102a40 2022-09-10T04:19:35.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f677c017e20 con 0x7f678c102a40 2022-09-10T04:19:35.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c199a80 con 0x7f678c102a40 2022-09-10T04:19:35.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.819+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c199eb0 con 0x7f678c102a40 2022-09-10T04:19:35.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.820+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f677c028020 con 0x7f678c102a40 2022-09-10T04:19:35.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.821+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c060310 con 0x7f678c102a40 2022-09-10T04:19:35.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.821+0000 7f6773fff700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6774035910 0x7f6774037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:35.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.821+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f677c048ad0 con 0x7f678c102a40 2022-09-10T04:19:35.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.821+0000 7f678a7fc700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6774035910 0x7f6774037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:35.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.822+0000 7f678a7fc700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6774035910 0x7f6774037dc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6780009750 tx=0x7f6780006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:35.831 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:35.826+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f677c0215a0 con 0x7f678c102a40 2022-09-10T04:19:36.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:36.048+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}) v1 -- 0x7f678c0008d0 con 0x7f6774035910 2022-09-10T04:19:36.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:36.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:19:36.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:36 smithi081 ceph-mon[90133]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:37.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.921+0000 7f6773fff700 1 -- 172.21.15.81:0/872933904 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f678c0008d0 con 0x7f6774035910 2022-09-10T04:19:37.924 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi110' with addr '172.21.15.110' 2022-09-10T04:19:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.923+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6774035910 msgr2=0x7f6774037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.923+0000 7f67914f3700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6774035910 0x7f6774037dc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6780009750 tx=0x7f6780006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:37.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.923+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 msgr2=0x7f678c199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.923+0000 7f67914f3700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c199340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f677c005dd0 tx=0x7f677c005390 comp rx=0 tx=0).stop 2022-09-10T04:19:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.924+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 shutdown_connections 2022-09-10T04:19:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.924+0000 7f67914f3700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6774035910 0x7f6774037dc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:37.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.924+0000 7f67914f3700 1 --2- 172.21.15.81:0/872933904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c102a40 0x7f678c199340 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:37.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.924+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 >> 172.21.15.81:0/872933904 conn(0x7f678c0727f0 msgr2=0x7f678c0fb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:37.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.924+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 shutdown_connections 2022-09-10T04:19:37.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:37.924+0000 7f67914f3700 1 -- 172.21.15.81:0/872933904 wait complete. 2022-09-10T04:19:38.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:37 smithi081 ceph-mon[90133]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:38.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:37 smithi081 ceph-mon[90133]: Deploying cephadm binary to smithi110 2022-09-10T04:19:38.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch host ls --format=json 2022-09-10T04:19:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:38 smithi081 ceph-mon[90133]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:19:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: Added host smithi110 2022-09-10T04:19:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:19:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:19:40.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:40.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:19:40.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:40.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:40.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:40.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 -- 172.21.15.81:0/2997670405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 msgr2=0x7f6f0c102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:40.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 --2- 172.21.15.81:0/2997670405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c102d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009790 tx=0x7f6f08009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:40.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 -- 172.21.15.81:0/2997670405 shutdown_connections 2022-09-10T04:19:40.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 --2- 172.21.15.81:0/2997670405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c102d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:40.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 -- 172.21.15.81:0/2997670405 >> 172.21.15.81:0/2997670405 conn(0x7f6f0c0727f0 msgr2=0x7f6f0c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:40.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 -- 172.21.15.81:0/2997670405 shutdown_connections 2022-09-10T04:19:40.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.605+0000 7f6f140d1700 1 -- 172.21.15.81:0/2997670405 wait complete. 2022-09-10T04:19:40.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.607+0000 7f6f140d1700 1 Processor -- start 2022-09-10T04:19:40.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.607+0000 7f6f140d1700 1 -- start start 2022-09-10T04:19:40.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.608+0000 7f6f140d1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c194eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:40.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.608+0000 7f6f140d1700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c1953f0 con 0x7f6f0c102950 2022-09-10T04:19:40.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.608+0000 7f6f11e6d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c194eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:40.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.608+0000 7f6f11e6d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c194eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50668/0 (socket says 172.21.15.81:50668) 2022-09-10T04:19:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.608+0000 7f6f11e6d700 1 -- 172.21.15.81:0/1038585048 learned_addr learned my addr 172.21.15.81:0/1038585048 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.608+0000 7f6f11e6d700 1 -- 172.21.15.81:0/1038585048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08009470 con 0x7f6f0c102950 2022-09-10T04:19:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.609+0000 7f6f11e6d700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c194eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6f08005dd0 tx=0x7f6f08005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.609+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f0801d070 con 0x7f6f0c102950 2022-09-10T04:19:40.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.609+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6f0800fcc0 con 0x7f6f0c102950 2022-09-10T04:19:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.609+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c1955f0 con 0x7f6f0c102950 2022-09-10T04:19:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.609+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f08017e20 con 0x7f6f0c102950 2022-09-10T04:19:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.609+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c195a20 con 0x7f6f0c102950 2022-09-10T04:19:40.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.610+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f6f08028020 con 0x7f6f0c102950 2022-09-10T04:19:40.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.610+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c060310 con 0x7f6f0c102950 2022-09-10T04:19:40.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.610+0000 7f6f02ffd700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6ef80358d0 0x7f6ef8037d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:40.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.611+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6f08048b30 con 0x7f6f0c102950 2022-09-10T04:19:40.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.611+0000 7f6f1166c700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6ef80358d0 0x7f6ef8037d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:40.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.612+0000 7f6f1166c700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6ef80358d0 0x7f6ef8037d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009750 tx=0x7f6efc006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:40.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.616+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6f08021830 con 0x7f6f0c102950 2022-09-10T04:19:40.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.837+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6f0c196310 con 0x7f6ef80358d0 2022-09-10T04:19:40.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.838+0000 7f6f02ffd700 1 -- 172.21.15.81:0/1038585048 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f6f0c196310 con 0x7f6ef80358d0 2022-09-10T04:19:40.841 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:19:40.842 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}] 2022-09-10T04:19:40.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.840+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6ef80358d0 msgr2=0x7f6ef8037d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:40.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.840+0000 7f6f140d1700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6ef80358d0 0x7f6ef8037d80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6efc009750 tx=0x7f6efc006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:40.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.840+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 msgr2=0x7f6f0c194eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:40.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.840+0000 7f6f140d1700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c194eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6f08005dd0 tx=0x7f6f08005390 comp rx=0 tx=0).stop 2022-09-10T04:19:40.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.841+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 shutdown_connections 2022-09-10T04:19:40.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.841+0000 7f6f140d1700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6ef80358d0 0x7f6ef8037d80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:40.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.841+0000 7f6f140d1700 1 --2- 172.21.15.81:0/1038585048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f0c102950 0x7f6f0c194eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:40.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.841+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 >> 172.21.15.81:0/1038585048 conn(0x7f6f0c0727f0 msgr2=0x7f6f0c0fb710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:40.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.841+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 shutdown_connections 2022-09-10T04:19:40.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:40.841+0000 7f6f140d1700 1 -- 172.21.15.81:0/1038585048 wait complete. 2022-09-10T04:19:41.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:40 smithi081 ceph-mon[90133]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:41.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:40 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:41.305 INFO:tasks.cephadm:Setting crush tunables to default 2022-09-10T04:19:41.306 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd crush tunables default 2022-09-10T04:19:42.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:41 smithi081 ceph-mon[90133]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:19:42.169 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:19:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:19:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:43.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.721+0000 7feff3b00700 1 -- 172.21.15.81:0/1375176139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 msgr2=0x7fefec103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:43.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.721+0000 7feff3b00700 1 --2- 172.21.15.81:0/1375176139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec103c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009790 tx=0x7fefe8009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:43.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.721+0000 7feff3b00700 1 -- 172.21.15.81:0/1375176139 shutdown_connections 2022-09-10T04:19:43.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.721+0000 7feff3b00700 1 --2- 172.21.15.81:0/1375176139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec103c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:43.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.721+0000 7feff3b00700 1 -- 172.21.15.81:0/1375176139 >> 172.21.15.81:0/1375176139 conn(0x7fefec0f9420 msgr2=0x7fefec0fb830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:43.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.722+0000 7feff3b00700 1 -- 172.21.15.81:0/1375176139 shutdown_connections 2022-09-10T04:19:43.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.722+0000 7feff3b00700 1 -- 172.21.15.81:0/1375176139 wait complete. 2022-09-10T04:19:43.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.723+0000 7feff3b00700 1 Processor -- start 2022-09-10T04:19:43.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.723+0000 7feff3b00700 1 -- start start 2022-09-10T04:19:43.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.723+0000 7feff3b00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:43.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.723+0000 7feff3b00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec19af80 con 0x7fefec103860 2022-09-10T04:19:43.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7feff189c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:43.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7feff189c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec19aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50694/0 (socket says 172.21.15.81:50694) 2022-09-10T04:19:43.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7feff189c700 1 -- 172.21.15.81:0/108200684 learned_addr learned my addr 172.21.15.81:0/108200684 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:19:43.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7feff189c700 1 -- 172.21.15.81:0/108200684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe8009470 con 0x7fefec103860 2022-09-10T04:19:43.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7feff189c700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec19aa40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fefe80059c0 tx=0x7fefe8005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:43.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefe801d070 con 0x7fefec103860 2022-09-10T04:19:43.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fefe800fd80 con 0x7fefec103860 2022-09-10T04:19:43.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.724+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec19b180 con 0x7fefec103860 2022-09-10T04:19:43.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.725+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefe8017df0 con 0x7fefec103860 2022-09-10T04:19:43.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.725+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec194a20 con 0x7fefec103860 2022-09-10T04:19:43.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.726+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fefe8028020 con 0x7fefec103860 2022-09-10T04:19:43.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.726+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec060310 con 0x7fefec103860 2022-09-10T04:19:43.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.726+0000 7fefe2ffd700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fefd8035910 0x7fefd8037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:43.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.726+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fefe8048a70 con 0x7fefec103860 2022-09-10T04:19:43.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.726+0000 7feff109b700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fefd8035910 0x7fefd8037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:43.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.727+0000 7feff109b700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fefd8035910 0x7fefd8037dc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009750 tx=0x7fefdc006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:43.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.731+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fefe800f8e0 con 0x7fefec103860 2022-09-10T04:19:43.957 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:43.955+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fefec1960d0 con 0x7fefec103860 2022-09-10T04:19:44.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.051+0000 7fefe2ffd700 1 -- 172.21.15.81:0/108200684 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fefe80213c0 con 0x7fefec103860 2022-09-10T04:19:44.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fefd8035910 msgr2=0x7fefd8037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:44.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fefd8035910 0x7fefd8037dc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009750 tx=0x7fefdc006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:44.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 msgr2=0x7fefec19aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:44.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec19aa40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fefe80059c0 tx=0x7fefe8005390 comp rx=0 tx=0).stop 2022-09-10T04:19:44.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 shutdown_connections 2022-09-10T04:19:44.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fefd8035910 0x7fefd8037dc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 --2- 172.21.15.81:0/108200684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefec103860 0x7fefec19aa40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:44.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.054+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 >> 172.21.15.81:0/108200684 conn(0x7fefec0f9420 msgr2=0x7fefec0fb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:44.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.055+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 shutdown_connections 2022-09-10T04:19:44.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:19:44.055+0000 7feff3b00700 1 -- 172.21.15.81:0/108200684 wait complete. 2022-09-10T04:19:44.061 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2022-09-10T04:19:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:44 smithi081 ceph-mon[90133]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:44 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/108200684' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-09-10T04:19:44.534 INFO:tasks.cephadm:Adding mon.a on smithi081 2022-09-10T04:19:44.535 INFO:tasks.cephadm:Adding mon.b on smithi104 2022-09-10T04:19:44.536 INFO:tasks.cephadm:Adding mon.c on smithi110 2022-09-10T04:19:44.536 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch apply mon '3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c' 2022-09-10T04:19:45.157 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /etc/ceph/ceph.conf 2022-09-10T04:19:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:45 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/108200684' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-09-10T04:19:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:45 smithi081 ceph-mon[90133]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:19:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /etc/ceph/ceph.conf 2022-09-10T04:19:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:46 smithi081 ceph-mon[90133]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.745+0000 7fa40342d700 1 -- 172.21.15.110:0/1873663197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 msgr2=0x7fa3fc0fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.745+0000 7fa40342d700 1 --2- 172.21.15.110:0/1873663197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fe100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009790 tx=0x7fa3ec009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 -- 172.21.15.110:0/1873663197 shutdown_connections 2022-09-10T04:19:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 --2- 172.21.15.110:0/1873663197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fe100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 -- 172.21.15.110:0/1873663197 >> 172.21.15.110:0/1873663197 conn(0x7fa3fc0f74f0 msgr2=0x7fa3fc0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 -- 172.21.15.110:0/1873663197 shutdown_connections 2022-09-10T04:19:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 -- 172.21.15.110:0/1873663197 wait complete. 2022-09-10T04:19:46.751 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 Processor -- start 2022-09-10T04:19:46.751 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 -- start start 2022-09-10T04:19:46.751 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:46.752 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.746+0000 7fa40342d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc195510 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.752 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.747+0000 7fa4011c9700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:46.752 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.747+0000 7fa4011c9700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.110:50036/0 (socket says 172.21.15.110:50036) 2022-09-10T04:19:46.752 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.747+0000 7fa4011c9700 1 -- 172.21.15.110:0/2979300817 learned_addr learned my addr 172.21.15.110:0/2979300817 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:19:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.747+0000 7fa4011c9700 1 -- 172.21.15.110:0/2979300817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ec009470 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.747+0000 7fa4011c9700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fc450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec005850 tx=0x7fa3ec0053e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.748+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3ec01d070 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.748+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc195650 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.748+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa3ec00fd70 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.748+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3ec017dc0 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.748+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc195b00 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.748+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc102400 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.749+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fa3ec028030 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.749+0000 7fa3f27fc700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa3e80358b0 0x7fa3e8037d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.749+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa3ec048870 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.749+0000 7fa4009c8700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa3e80358b0 0x7fa3e8037d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.750+0000 7fa4009c8700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa3e80358b0 0x7fa3e8037d60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009750 tx=0x7fa3f8006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.754+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa3ec0299b0 con 0x7fa3fc0fdcf0 2022-09-10T04:19:46.947 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.946+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3fc05fd10 con 0x7fa3e80358b0 2022-09-10T04:19:46.955 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.953+0000 7fa3f27fc700 1 -- 172.21.15.110:0/2979300817 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa3fc05fd10 con 0x7fa3e80358b0 2022-09-10T04:19:46.956 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mon update... 2022-09-10T04:19:46.958 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa3e80358b0 msgr2=0x7fa3e8037d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:46.958 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa3e80358b0 0x7fa3e8037d60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009750 tx=0x7fa3f8006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:46.959 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 msgr2=0x7fa3fc0fc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:46.960 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fc450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec005850 tx=0x7fa3ec0053e0 comp rx=0 tx=0).stop 2022-09-10T04:19:46.960 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 shutdown_connections 2022-09-10T04:19:46.960 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa3e80358b0 0x7fa3e8037d60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:46.960 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 --2- 172.21.15.110:0/2979300817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3fc0fdcf0 0x7fa3fc0fc450 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 >> 172.21.15.110:0/2979300817 conn(0x7fa3fc0f74f0 msgr2=0x7fa3fc0f8200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 shutdown_connections 2022-09-10T04:19:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:46.955+0000 7fa40342d700 1 -- 172.21.15.110:0/2979300817 wait complete. 2022-09-10T04:19:47.384 DEBUG:teuthology.orchestra.run.smithi104:mon.b> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.b.service 2022-09-10T04:19:47.389 DEBUG:teuthology.orchestra.run.smithi110:mon.c> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mon.c.service 2022-09-10T04:19:47.392 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-09-10T04:19:47.393 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph mon dump -f json 2022-09-10T04:19:47.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:19:47.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:19:48.048 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:19:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: Saving service mon spec with placement smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c;count:3 2022-09-10T04:19:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:19:48.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:19:48.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:48.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:19:48.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:48.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:47 smithi081 ceph-mon[90133]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:19:49.511 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 -- 172.21.15.110:0/3681342161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 msgr2=0x7f79b4102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:49.511 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 --2- 172.21.15.110:0/3681342161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4102b30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009790 tx=0x7f79a8009aa0 comp rx=0 tx=0).stop 2022-09-10T04:19:49.511 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 -- 172.21.15.110:0/3681342161 shutdown_connections 2022-09-10T04:19:49.512 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 --2- 172.21.15.110:0/3681342161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4102b30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:49.512 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 -- 172.21.15.110:0/3681342161 >> 172.21.15.110:0/3681342161 conn(0x7f79b40f8340 msgr2=0x7f79b40fa750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:49.512 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 -- 172.21.15.110:0/3681342161 shutdown_connections 2022-09-10T04:19:49.512 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.508+0000 7f79badd7700 1 -- 172.21.15.110:0/3681342161 wait complete. 2022-09-10T04:19:49.513 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79badd7700 1 Processor -- start 2022-09-10T04:19:49.513 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79badd7700 1 -- start start 2022-09-10T04:19:49.513 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79badd7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4193840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:49.513 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79badd7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4193d80 con 0x7f79b4102760 2022-09-10T04:19:49.514 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79b8b73700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4193840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:49.514 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79b8b73700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4193840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.110:50042/0 (socket says 172.21.15.110:50042) 2022-09-10T04:19:49.514 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.509+0000 7f79b8b73700 1 -- 172.21.15.110:0/200588986 learned_addr learned my addr 172.21.15.110:0/200588986 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:19:49.514 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79b8b73700 1 -- 172.21.15.110:0/200588986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a8009470 con 0x7f79b4102760 2022-09-10T04:19:49.515 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79b8b73700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4193840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f79a80059c0 tx=0x7f79a8005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:49.516 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79a801d070 con 0x7f79b4102760 2022-09-10T04:19:49.517 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b4193f80 con 0x7f79b4102760 2022-09-10T04:19:49.517 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f79a800fd80 con 0x7f79b4102760 2022-09-10T04:19:49.517 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79a8017df0 con 0x7f79b4102760 2022-09-10T04:19:49.517 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.510+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b4194430 con 0x7f79b4102760 2022-09-10T04:19:49.518 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.511+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f79a8028020 con 0x7f79b4102760 2022-09-10T04:19:49.518 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.511+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b4194670 con 0x7f79b4102760 2022-09-10T04:19:49.518 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.512+0000 7f79b1ffb700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f79a4035910 0x7f79a4037dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:49.518 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.512+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f79a8048c40 con 0x7f79b4102760 2022-09-10T04:19:49.519 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.512+0000 7f79b3fff700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f79a4035910 0x7f79a4037dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:49.519 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.513+0000 7f79b3fff700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f79a4035910 0x7f79a4037dc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009750 tx=0x7f79a0006cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:49.519 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.518+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f79a8020370 con 0x7f79b4102760 2022-09-10T04:19:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:19:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:49 smithi081 ceph-mon[90133]: Deploying daemon mon.c on smithi110 2022-09-10T04:19:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.779+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79b4101950 con 0x7f79b4102760 2022-09-10T04:19:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.780+0000 7f79b1ffb700 1 -- 172.21.15.110:0/200588986 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+710 (secure 0 0 0) 0x7f79a8020370 con 0x7f79b4102760 2022-09-10T04:19:49.783 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:19:49.784 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":1,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","modified":"2022-09-10T04:18:01.537043Z","created":"2022-09-10T04:18:01.537043Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2022-09-10T04:19:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f79a4035910 msgr2=0x7f79a4037dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f79a4035910 0x7f79a4037dc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f79a0009750 tx=0x7f79a0006cc0 comp rx=0 tx=0).stop 2022-09-10T04:19:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 msgr2=0x7f79b4193840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4193840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f79a80059c0 tx=0x7f79a8005390 comp rx=0 tx=0).stop 2022-09-10T04:19:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 shutdown_connections 2022-09-10T04:19:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f79a4035910 0x7f79a4037dc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 --2- 172.21.15.110:0/200588986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79b4102760 0x7f79b4193840 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 >> 172.21.15.110:0/200588986 conn(0x7f79b40f8340 msgr2=0x7f79b40f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 shutdown_connections 2022-09-10T04:19:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:49.782+0000 7f79badd7700 1 -- 172.21.15.110:0/200588986 wait complete. 2022-09-10T04:19:49.788 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 1 2022-09-10T04:19:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:50 smithi081 ceph-mon[90133]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:50 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/200588986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:19:51.250 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-09-10T04:19:51.251 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph mon dump -f json 2022-09-10T04:19:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 systemd[1]: Starting Ceph mon.c for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:19:51.921 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 podman[97564]: 2022-09-10T04:19:51.922 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: set uid:gid to 167:167 (ceph:ceph) 2022-09-10T04:19:51.922 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev), process ceph-mon, pid 7 2022-09-10T04:19:51.923 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 bash[97564]: 3c92599d31f1aad87f54084336845c24cd57345cc792dc7e2f4e17cae4ba2254 2022-09-10T04:19:51.923 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pidfile_write: ignore empty --pid-file 2022-09-10T04:19:51.924 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: load: jerasure load: lrc 2022-09-10T04:19:51.924 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: RocksDB version: 6.15.5 2022-09-10T04:19:51.925 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-09-10T04:19:51.925 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Compile date Sep 9 2022 2022-09-10T04:19:51.925 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: DB SUMMARY 2022-09-10T04:19:51.927 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: DB Session ID: 5AIC2KF9M3WGC89RI0Y7 2022-09-10T04:19:51.928 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: CURRENT file: CURRENT 2022-09-10T04:19:51.929 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: IDENTITY file: IDENTITY 2022-09-10T04:19:51.929 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-09-10T04:19:51.929 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2022-09-10T04:19:51.930 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2022-09-10T04:19:51.930 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.error_if_exists: 0 2022-09-10T04:19:51.931 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.create_if_missing: 0 2022-09-10T04:19:51.931 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.paranoid_checks: 1 2022-09-10T04:19:51.932 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-09-10T04:19:51.932 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.env: 0x55a9b2ba4100 2022-09-10T04:19:51.932 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.fs: Posix File System 2022-09-10T04:19:51.933 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.info_log: 0x55a9b4959c20 2022-09-10T04:19:51.933 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_file_opening_threads: 16 2022-09-10T04:19:51.934 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.statistics: (nil) 2022-09-10T04:19:51.934 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.use_fsync: 0 2022-09-10T04:19:51.934 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_log_file_size: 0 2022-09-10T04:19:51.935 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-09-10T04:19:51.935 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.log_file_time_to_roll: 0 2022-09-10T04:19:51.935 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.keep_log_file_num: 1000 2022-09-10T04:19:51.935 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.recycle_log_file_num: 0 2022-09-10T04:19:51.936 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.allow_fallocate: 1 2022-09-10T04:19:51.936 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.allow_mmap_reads: 0 2022-09-10T04:19:51.936 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.allow_mmap_writes: 0 2022-09-10T04:19:51.936 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.use_direct_reads: 0 2022-09-10T04:19:51.936 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-09-10T04:19:51.937 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.create_missing_column_families: 0 2022-09-10T04:19:51.937 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.db_log_dir: 2022-09-10T04:19:51.937 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2022-09-10T04:19:51.938 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.table_cache_numshardbits: 6 2022-09-10T04:19:51.940 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.WAL_ttl_seconds: 0 2022-09-10T04:19:51.941 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.WAL_size_limit_MB: 0 2022-09-10T04:19:51.941 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-09-10T04:19:51.941 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-09-10T04:19:51.941 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.is_fd_close_on_exec: 1 2022-09-10T04:19:51.942 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.advise_random_on_open: 1 2022-09-10T04:19:51.942 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.db_write_buffer_size: 0 2022-09-10T04:19:51.942 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.write_buffer_manager: 0x55a9b4a4eb70 2022-09-10T04:19:51.942 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-09-10T04:19:51.943 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-09-10T04:19:51.943 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-09-10T04:19:51.943 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.use_adaptive_mutex: 0 2022-09-10T04:19:51.943 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.rate_limiter: (nil) 2022-09-10T04:19:51.944 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-09-10T04:19:51.944 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.wal_recovery_mode: 2 2022-09-10T04:19:51.944 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.enable_thread_tracking: 0 2022-09-10T04:19:51.944 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.enable_pipelined_write: 0 2022-09-10T04:19:51.944 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.unordered_write: 0 2022-09-10T04:19:51.945 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-09-10T04:19:51.945 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-09-10T04:19:51.945 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-09-10T04:19:51.945 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-09-10T04:19:51.945 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.row_cache: None 2022-09-10T04:19:51.946 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.wal_filter: None 2022-09-10T04:19:51.946 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-09-10T04:19:51.946 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.allow_ingest_behind: 0 2022-09-10T04:19:51.946 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.preserve_deletes: 0 2022-09-10T04:19:51.947 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.two_write_queues: 0 2022-09-10T04:19:51.947 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.manual_wal_flush: 0 2022-09-10T04:19:51.947 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.atomic_flush: 0 2022-09-10T04:19:51.947 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-09-10T04:19:51.947 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.persist_stats_to_disk: 0 2022-09-10T04:19:51.948 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.write_dbid_to_manifest: 0 2022-09-10T04:19:51.948 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.log_readahead_size: 0 2022-09-10T04:19:51.948 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-09-10T04:19:51.948 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.best_efforts_recovery: 0 2022-09-10T04:19:51.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-09-10T04:19:51.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-09-10T04:19:51.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.allow_data_in_errors: 0 2022-09-10T04:19:51.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.db_host_id: __hostname__ 2022-09-10T04:19:51.950 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_background_jobs: 2 2022-09-10T04:19:51.950 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_background_compactions: -1 2022-09-10T04:19:51.950 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_subcompactions: 1 2022-09-10T04:19:51.950 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-09-10T04:19:51.950 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-09-10T04:19:51.951 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.delayed_write_rate : 16777216 2022-09-10T04:19:51.951 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_total_wal_size: 0 2022-09-10T04:19:51.951 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-09-10T04:19:51.951 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.stats_dump_period_sec: 600 2022-09-10T04:19:51.951 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.stats_persist_period_sec: 600 2022-09-10T04:19:51.952 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-09-10T04:19:51.952 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_open_files: -1 2022-09-10T04:19:51.952 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bytes_per_sync: 0 2022-09-10T04:19:51.952 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.wal_bytes_per_sync: 0 2022-09-10T04:19:51.953 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.strict_bytes_per_sync: 0 2022-09-10T04:19:51.953 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_readahead_size: 0 2022-09-10T04:19:51.953 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_background_flushes: -1 2022-09-10T04:19:51.953 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Compression algorithms supported: 2022-09-10T04:19:51.953 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-09-10T04:19:51.954 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kZSTD supported: 0 2022-09-10T04:19:51.954 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kXpressCompression supported: 0 2022-09-10T04:19:51.954 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kLZ4HCCompression supported: 1 2022-09-10T04:19:51.954 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kZlibCompression supported: 1 2022-09-10T04:19:51.955 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kSnappyCompression supported: 1 2022-09-10T04:19:51.955 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kLZ4Compression supported: 1 2022-09-10T04:19:51.955 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: kBZip2Compression supported: 0 2022-09-10T04:19:51.956 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Fast CRC32 supported: Supported on x86 2022-09-10T04:19:51.956 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2022-09-10T04:19:51.956 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2022-09-10T04:19:51.956 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-09-10T04:19:51.957 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.merge_operator: 2022-09-10T04:19:51.957 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_filter: None 2022-09-10T04:19:51.957 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_filter_factory: None 2022-09-10T04:19:51.957 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.sst_partitioner_factory: None 2022-09-10T04:19:51.957 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.memtable_factory: SkipListFactory 2022-09-10T04:19:51.958 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.table_factory: BlockBasedTable 2022-09-10T04:19:51.958 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a9b4956d48) 2022-09-10T04:19:51.958 INFO:journalctl@ceph.mon.c.smithi110.stdout: cache_index_and_filter_blocks: 1 2022-09-10T04:19:51.958 INFO:journalctl@ceph.mon.c.smithi110.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-09-10T04:19:51.958 INFO:journalctl@ceph.mon.c.smithi110.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-09-10T04:19:51.959 INFO:journalctl@ceph.mon.c.smithi110.stdout: pin_top_level_index_and_filter: 1 2022-09-10T04:19:51.959 INFO:journalctl@ceph.mon.c.smithi110.stdout: index_type: 0 2022-09-10T04:19:51.959 INFO:journalctl@ceph.mon.c.smithi110.stdout: data_block_index_type: 0 2022-09-10T04:19:51.959 INFO:journalctl@ceph.mon.c.smithi110.stdout: index_shortening: 1 2022-09-10T04:19:51.960 INFO:journalctl@ceph.mon.c.smithi110.stdout: data_block_hash_table_util_ratio: 0.750000 2022-09-10T04:19:51.960 INFO:journalctl@ceph.mon.c.smithi110.stdout: hash_index_allow_collision: 1 2022-09-10T04:19:51.960 INFO:journalctl@ceph.mon.c.smithi110.stdout: checksum: 1 2022-09-10T04:19:51.960 INFO:journalctl@ceph.mon.c.smithi110.stdout: no_block_cache: 0 2022-09-10T04:19:51.961 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache: 0x55a9b49c29b0 2022-09-10T04:19:51.961 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache_name: BinnedLRUCache 2022-09-10T04:19:51.961 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache_options: 2022-09-10T04:19:51.961 INFO:journalctl@ceph.mon.c.smithi110.stdout: capacity : 536870912 2022-09-10T04:19:51.961 INFO:journalctl@ceph.mon.c.smithi110.stdout: num_shard_bits : 4 2022-09-10T04:19:51.962 INFO:journalctl@ceph.mon.c.smithi110.stdout: strict_capacity_limit : 0 2022-09-10T04:19:51.962 INFO:journalctl@ceph.mon.c.smithi110.stdout: high_pri_pool_ratio: 0.000 2022-09-10T04:19:51.962 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache_compressed: (nil) 2022-09-10T04:19:51.962 INFO:journalctl@ceph.mon.c.smithi110.stdout: persistent_cache: (nil) 2022-09-10T04:19:51.963 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_size: 4096 2022-09-10T04:19:51.963 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_size_deviation: 10 2022-09-10T04:19:51.963 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_restart_interval: 16 2022-09-10T04:19:51.963 INFO:journalctl@ceph.mon.c.smithi110.stdout: index_block_restart_interval: 1 2022-09-10T04:19:51.963 INFO:journalctl@ceph.mon.c.smithi110.stdout: metadata_block_size: 4096 2022-09-10T04:19:51.964 INFO:journalctl@ceph.mon.c.smithi110.stdout: partition_filters: 0 2022-09-10T04:19:51.964 INFO:journalctl@ceph.mon.c.smithi110.stdout: use_delta_encoding: 1 2022-09-10T04:19:51.964 INFO:journalctl@ceph.mon.c.smithi110.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-09-10T04:19:51.964 INFO:journalctl@ceph.mon.c.smithi110.stdout: whole_key_filtering: 1 2022-09-10T04:19:51.964 INFO:journalctl@ceph.mon.c.smithi110.stdout: verify_compression: 0 2022-09-10T04:19:51.965 INFO:journalctl@ceph.mon.c.smithi110.stdout: read_amp_bytes_per_bit: 0 2022-09-10T04:19:51.965 INFO:journalctl@ceph.mon.c.smithi110.stdout: format_version: 4 2022-09-10T04:19:51.965 INFO:journalctl@ceph.mon.c.smithi110.stdout: enable_index_compression: 1 2022-09-10T04:19:51.965 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_align: 0 2022-09-10T04:19:51.966 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.write_buffer_size: 33554432 2022-09-10T04:19:51.966 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_write_buffer_number: 2 2022-09-10T04:19:51.966 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression: NoCompression 2022-09-10T04:19:51.966 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression: Disabled 2022-09-10T04:19:51.966 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.prefix_extractor: nullptr 2022-09-10T04:19:51.967 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-09-10T04:19:51.967 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.num_levels: 7 2022-09-10T04:19:51.967 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-09-10T04:19:51.967 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-09-10T04:19:51.968 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-09-10T04:19:51.968 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-09-10T04:19:51.968 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-09-10T04:19:51.968 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-09-10T04:19:51.968 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-09-10T04:19:51.969 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-09-10T04:19:51.969 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-09-10T04:19:51.969 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-09-10T04:19:51.969 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.window_bits: -14 2022-09-10T04:19:51.970 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.level: 32767 2022-09-10T04:19:51.970 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.strategy: 0 2022-09-10T04:19:51.970 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-09-10T04:19:51.970 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-09-10T04:19:51.970 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-09-10T04:19:51.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compression_opts.enabled: false 2022-09-10T04:19:51.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-09-10T04:19:51.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-09-10T04:19:51.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-09-10T04:19:51.972 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.target_file_size_base: 67108864 2022-09-10T04:19:51.972 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.target_file_size_multiplier: 1 2022-09-10T04:19:51.972 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-09-10T04:19:51.972 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-09-10T04:19:51.973 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-09-10T04:19:51.973 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-09-10T04:19:51.973 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-09-10T04:19:51.973 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-09-10T04:19:51.974 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-09-10T04:19:51.974 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-09-10T04:19:51.974 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-09-10T04:19:51.974 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-09-10T04:19:51.974 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-09-10T04:19:51.975 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-09-10T04:19:51.975 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.arena_block_size: 4194304 2022-09-10T04:19:51.975 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-09-10T04:19:51.975 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-09-10T04:19:51.976 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-09-10T04:19:51.976 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.disable_auto_compactions: 0 2022-09-10T04:19:51.976 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-09-10T04:19:51.976 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-09-10T04:19:51.977 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-09-10T04:19:51.977 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-09-10T04:19:51.977 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-09-10T04:19:51.978 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-09-10T04:19:51.978 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-09-10T04:19:51.978 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-09-10T04:19:51.978 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-09-10T04:19:51.978 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-09-10T04:19:51.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.table_properties_collectors: 2022-09-10T04:19:51.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.inplace_update_support: 0 2022-09-10T04:19:51.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.inplace_update_num_locks: 10000 2022-09-10T04:19:51.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-09-10T04:19:51.980 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-09-10T04:19:51.980 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.memtable_huge_page_size: 0 2022-09-10T04:19:51.980 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.bloom_locality: 0 2022-09-10T04:19:51.980 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.max_successive_merges: 0 2022-09-10T04:19:51.980 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.optimize_filters_for_hits: 0 2022-09-10T04:19:51.981 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.paranoid_file_checks: 0 2022-09-10T04:19:51.981 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.force_consistency_checks: 1 2022-09-10T04:19:51.981 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.report_bg_io_stats: 0 2022-09-10T04:19:51.981 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.ttl: 2592000 2022-09-10T04:19:51.982 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.periodic_compaction_seconds: 0 2022-09-10T04:19:51.982 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.enable_blob_files: false 2022-09-10T04:19:51.982 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.min_blob_size: 0 2022-09-10T04:19:51.982 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.blob_file_size: 268435456 2022-09-10T04:19:51.982 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.blob_compression_type: NoCompression 2022-09-10T04:19:51.983 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.enable_blob_garbage_collection: false 2022-09-10T04:19:51.983 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-09-10T04:19:51.983 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-09-10T04:19:51.983 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2022-09-10T04:19:51.984 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2022-09-10T04:19:51.984 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783591870366, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-09-10T04:19:51.984 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2022-09-10T04:19:51.984 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-09-10T04:19:51.984 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783591870633, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1662783591, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "acb4bfca-d7e6-4b9b-8ad6-2897305ee00a", "db_session_id": "5AIC2KF9M3WGC89RI0Y7"}} 2022-09-10T04:19:51.985 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2022-09-10T04:19:51.985 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783591870957, "job": 1, "event": "recovery_finished"} 2022-09-10T04:19:51.985 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-09-10T04:19:51.985 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55a9b49a9880 2022-09-10T04:19:51.986 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: DB pointer 0x55a9b49b8000 2022-09-10T04:19:51.986 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2022-09-10T04:19:51.986 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-09-10T04:19:51.986 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** DB Stats ** 2022-09-10T04:19:51.986 INFO:journalctl@ceph.mon.c.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:19:51.987 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-09-10T04:19:51.987 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-09-10T04:19:51.987 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-10T04:19:51.987 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-09-10T04:19:51.988 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-09-10T04:19:51.988 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-10T04:19:51.988 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:51.988 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** Compaction Stats [default] ** 2022-09-10T04:19:51.988 INFO:journalctl@ceph.mon.c.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:19:51.989 INFO:journalctl@ceph.mon.c.smithi110.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:19:51.989 INFO:journalctl@ceph.mon.c.smithi110.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.989 INFO:journalctl@ceph.mon.c.smithi110.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.989 INFO:journalctl@ceph.mon.c.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.989 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:51.990 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** Compaction Stats [default] ** 2022-09-10T04:19:51.990 INFO:journalctl@ceph.mon.c.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:19:51.990 INFO:journalctl@ceph.mon.c.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:19:51.990 INFO:journalctl@ceph.mon.c.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.991 INFO:journalctl@ceph.mon.c.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:19:51.991 INFO:journalctl@ceph.mon.c.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-10T04:19:51.991 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-10T04:19:51.991 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-10T04:19:51.991 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-10T04:19:51.992 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-10T04:19:51.992 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:19:51.992 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:19:51.992 INFO:journalctl@ceph.mon.c.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-10T04:19:51.993 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:51.993 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-10T04:19:51.993 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:51.993 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** Compaction Stats [default] ** 2022-09-10T04:19:51.994 INFO:journalctl@ceph.mon.c.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:19:51.994 INFO:journalctl@ceph.mon.c.smithi110.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:19:51.994 INFO:journalctl@ceph.mon.c.smithi110.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.994 INFO:journalctl@ceph.mon.c.smithi110.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.994 INFO:journalctl@ceph.mon.c.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-09-10T04:19:51.995 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:51.995 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** Compaction Stats [default] ** 2022-09-10T04:19:51.995 INFO:journalctl@ceph.mon.c.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:19:51.995 INFO:journalctl@ceph.mon.c.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:19:51.996 INFO:journalctl@ceph.mon.c.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2022-09-10T04:19:51.996 INFO:journalctl@ceph.mon.c.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:19:51.996 INFO:journalctl@ceph.mon.c.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-10T04:19:51.997 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-10T04:19:51.997 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-10T04:19:51.997 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-10T04:19:51.997 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-10T04:19:51.998 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:19:51.998 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:19:51.998 INFO:journalctl@ceph.mon.c.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-10T04:19:51.998 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:51.999 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-10T04:19:51.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c does not exist in monmap, will attempt to join an existing cluster 2022-09-10T04:19:51.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: using public_addr v2:172.21.15.110:0/0 -> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2022-09-10T04:19:51.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: starting mon.c rank -1 at public addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] at bind addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:19:51.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(???) e0 preinit fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:19:52.000 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).mds e1 new map 2022-09-10T04:19:52.000 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).mds e1 print_map 2022-09-10T04:19:52.000 INFO:journalctl@ceph.mon.c.smithi110.stdout: e1 2022-09-10T04:19:52.001 INFO:journalctl@ceph.mon.c.smithi110.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-09-10T04:19:52.001 INFO:journalctl@ceph.mon.c.smithi110.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-10T04:19:52.001 INFO:journalctl@ceph.mon.c.smithi110.stdout: legacy client fscid: -1 2022-09-10T04:19:52.001 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2022-09-10T04:19:52.002 INFO:journalctl@ceph.mon.c.smithi110.stdout: No filesystems configured 2022-09-10T04:19:52.002 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-09-10T04:19:52.002 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-09-10T04:19:52.002 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-09-10T04:19:52.003 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-09-10T04:19:52.003 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-09-10T04:19:52.003 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-09-10T04:19:52.003 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-09-10T04:19:52.003 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:19:52.004 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:19:52.004 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:19:52.004 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.004 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:19:52.005 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.005 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.005 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.005 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.005 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.006 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:19:52.006 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.006 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.006 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:52.007 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Deploying cephadm binary to smithi110 2022-09-10T04:19:52.007 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.007 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.007 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:52.008 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:52.008 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:19:52.008 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.008 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Added host smithi110 2022-09-10T04:19:52.008 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:19:52.009 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:19:52.009 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:52.009 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:19:52.009 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.010 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.010 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.010 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.010 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.010 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:19:52.011 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.011 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.011 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.011 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.012 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.012 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:19:52.012 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.012 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.012 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/108200684' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-09-10T04:19:52.013 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/108200684' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-09-10T04:19:52.013 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:19:52.013 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.013 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:19:52.014 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Saving service mon spec with placement smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c;count:3 2022-09-10T04:19:52.014 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.014 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:19:52.014 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:52.014 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:19:52.015 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:19:52.015 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.015 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:19:52.015 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:19:52.016 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:19:52.016 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.016 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.016 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:52.016 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:19:52.017 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:52.017 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: Deploying daemon mon.c on smithi110 2022-09-10T04:19:52.017 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:52.017 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/200588986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:19:52.018 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-09-10T04:19:52.018 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 ceph-mon[97633]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2022-09-10T04:19:52.018 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:51 smithi110 systemd[1]: Started Ceph mon.c for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:19:52.089 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:19:57.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:19:57.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: mon.a calling monitor election 2022-09-10T04:19:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: mon.c calling monitor election 2022-09-10T04:19:57.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:57.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-10T04:19:57.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: monmap e2: 2 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:19:57.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: fsmap 2022-09-10T04:19:57.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:19:57.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: mgrmap e13: x(active, since 47s) 2022-09-10T04:19:57.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: overall HEALTH_OK 2022-09-10T04:19:57.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:57.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:57.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:19:57.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:19:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: mon.a calling monitor election 2022-09-10T04:19:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: mon.c calling monitor election 2022-09-10T04:19:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-10T04:19:57.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: monmap e2: 2 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:19:57.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: fsmap 2022-09-10T04:19:57.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:19:57.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: mgrmap e13: x(active, since 47s) 2022-09-10T04:19:57.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: overall HEALTH_OK 2022-09-10T04:19:57.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:57.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:19:57.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:19:57.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:19:57.675 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 -- 172.21.15.110:0/1296778109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 msgr2=0x7f76580058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 --2- 172.21.15.110:0/1296778109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76580058c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f76700fdd90 tx=0x7f7664012d40 comp rx=0 tx=0).stop 2022-09-10T04:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 -- 172.21.15.110:0/1296778109 shutdown_connections 2022-09-10T04:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 --2- 172.21.15.110:0/1296778109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76580058c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:57.676 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 -- 172.21.15.110:0/1296778109 >> 172.21.15.110:0/1296778109 conn(0x7f7670072700 msgr2=0x7f7670072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:57.677 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 -- 172.21.15.110:0/1296778109 shutdown_connections 2022-09-10T04:19:57.677 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.674+0000 7f767638f700 1 -- 172.21.15.110:0/1296778109 wait complete. 2022-09-10T04:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.675+0000 7f767638f700 1 Processor -- start 2022-09-10T04:19:57.682 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.675+0000 7f767638f700 1 -- start start 2022-09-10T04:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f767638f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76700fe510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f767638f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:57.683 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f767638f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7670100490 con 0x7f7658003440 2022-09-10T04:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f767638f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767019ee30 con 0x7f76700fea50 2022-09-10T04:19:57.684 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:57.686 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60558/0 (socket says 172.21.15.110:60558) 2022-09-10T04:19:57.686 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 -- 172.21.15.110:0/670268749 learned_addr learned my addr 172.21.15.110:0/670268749 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:19:57.686 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 -- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 msgr2=0x7f76700feec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-09-10T04:19:57.686 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 -- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 msgr2=0x7f76700feec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-09-10T04:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-09-10T04:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766f7fe700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.676+0000 7f766ffff700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76700fe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:57.687 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.677+0000 7f766ffff700 1 -- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 msgr2=0x7f76700feec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:19:57.688 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.677+0000 7f766ffff700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:57.688 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.677+0000 7f766ffff700 1 -- 172.21.15.110:0/670268749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7664012790 con 0x7f7658003440 2022-09-10T04:19:57.688 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.677+0000 7f766ffff700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76700fe510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7664000c00 tx=0x7f7664005170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.678+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7664003910 con 0x7f7658003440 2022-09-10T04:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.678+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f766401d440 con 0x7f7658003440 2022-09-10T04:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.678+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767019efd0 con 0x7f7658003440 2022-09-10T04:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.678+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7664026400 con 0x7f7658003440 2022-09-10T04:19:57.689 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.678+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767019f450 con 0x7f7658003440 2022-09-10T04:19:57.690 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.679+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f7664036070 con 0x7f7658003440 2022-09-10T04:19:57.690 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.679+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7670060310 con 0x7f7658003440 2022-09-10T04:19:57.690 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.679+0000 7f766d7fa700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f765c035760 0x7f765c037c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:19:57.691 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.679+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f766404a250 con 0x7f7658003440 2022-09-10T04:19:57.691 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.680+0000 7f766f7fe700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f765c035760 0x7f765c037c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:19:57.692 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.680+0000 7f766f7fe700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f765c035760 0x7f765c037c10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7660006fa0 tx=0x7f7660008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:19:57.692 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.684+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f7664025950 con 0x7f7658003440 2022-09-10T04:19:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.943+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76700ff9a0 con 0x7f7658003440 2022-09-10T04:19:57.947 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.944+0000 7f766d7fa700 1 -- 172.21.15.110:0/670268749 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+980 (secure 0 0 0) 0x7f766401b300 con 0x7f7658003440 2022-09-10T04:19:57.948 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:19:57.949 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":2,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","modified":"2022-09-10T04:19:51.894505Z","created":"2022-09-10T04:18:01.537043Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-09-10T04:19:57.953 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f765c035760 msgr2=0x7f765c037c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:57.953 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f765c035760 0x7f765c037c10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7660006fa0 tx=0x7f7660008040 comp rx=0 tx=0).stop 2022-09-10T04:19:57.954 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 msgr2=0x7f76700fe510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:19:57.954 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76700fe510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7664000c00 tx=0x7f7664005170 comp rx=0 tx=0).stop 2022-09-10T04:19:57.955 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 shutdown_connections 2022-09-10T04:19:57.955 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7658003440 0x7f76700fe510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:57.955 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f765c035760 0x7f765c037c10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:57.956 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 --2- 172.21.15.110:0/670268749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76700fea50 0x7f76700feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:19:57.956 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 >> 172.21.15.110:0/670268749 conn(0x7f7670072700 msgr2=0x7f7670066400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:19:57.956 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 shutdown_connections 2022-09-10T04:19:57.957 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:19:57.946+0000 7f767638f700 1 -- 172.21.15.110:0/670268749 wait complete. 2022-09-10T04:19:57.959 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 2 2022-09-10T04:19:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:57 smithi081 ceph-mon[90133]: Deploying daemon mon.b on smithi104 2022-09-10T04:19:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:57 smithi081 ceph-mon[90133]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:58.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:19:57 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/670268749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:19:58.326 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:57 smithi110 ceph-mon[97633]: Deploying daemon mon.b on smithi104 2022-09-10T04:19:58.326 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:57 smithi110 ceph-mon[97633]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:19:58.327 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:19:58.327 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:19:57 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/670268749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:19:59.328 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2022-09-10T04:19:59.329 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph mon dump -f json 2022-09-10T04:19:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:19:59 smithi104 systemd[1]: Starting Ceph mon.b for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:19:59.931 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:20:00.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:19:59 smithi104 podman[97002]: 2022-09-10T04:20:00.187 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: set uid:gid to 167:167 (ceph:ceph) 2022-09-10T04:20:00.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev), process ceph-mon, pid 7 2022-09-10T04:20:00.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 bash[97002]: 0ea210629c23542fe4f181588f1c6e7ddf42349dc02d680ec6c6da67d9d874a8 2022-09-10T04:20:00.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pidfile_write: ignore empty --pid-file 2022-09-10T04:20:00.188 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: load: jerasure load: lrc 2022-09-10T04:20:00.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: RocksDB version: 6.15.5 2022-09-10T04:20:00.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2022-09-10T04:20:00.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Compile date Sep 9 2022 2022-09-10T04:20:00.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: DB SUMMARY 2022-09-10T04:20:00.189 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: DB Session ID: U2ZTSVT407X4KXNGB3NR 2022-09-10T04:20:00.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: CURRENT file: CURRENT 2022-09-10T04:20:00.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: IDENTITY file: IDENTITY 2022-09-10T04:20:00.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2022-09-10T04:20:00.190 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2022-09-10T04:20:00.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2022-09-10T04:20:00.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.error_if_exists: 0 2022-09-10T04:20:00.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.create_if_missing: 0 2022-09-10T04:20:00.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.paranoid_checks: 1 2022-09-10T04:20:00.191 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2022-09-10T04:20:00.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.env: 0x55c6911bd100 2022-09-10T04:20:00.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.fs: Posix File System 2022-09-10T04:20:00.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.info_log: 0x55c691c69c20 2022-09-10T04:20:00.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_file_opening_threads: 16 2022-09-10T04:20:00.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.statistics: (nil) 2022-09-10T04:20:00.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.use_fsync: 0 2022-09-10T04:20:00.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_log_file_size: 0 2022-09-10T04:20:00.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_manifest_file_size: 1073741824 2022-09-10T04:20:00.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.log_file_time_to_roll: 0 2022-09-10T04:20:00.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.keep_log_file_num: 1000 2022-09-10T04:20:00.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.recycle_log_file_num: 0 2022-09-10T04:20:00.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.allow_fallocate: 1 2022-09-10T04:20:00.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.allow_mmap_reads: 0 2022-09-10T04:20:00.195 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.allow_mmap_writes: 0 2022-09-10T04:20:00.195 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.use_direct_reads: 0 2022-09-10T04:20:00.195 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2022-09-10T04:20:00.195 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.create_missing_column_families: 0 2022-09-10T04:20:00.196 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.db_log_dir: 2022-09-10T04:20:00.196 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2022-09-10T04:20:00.196 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.table_cache_numshardbits: 6 2022-09-10T04:20:00.196 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.WAL_ttl_seconds: 0 2022-09-10T04:20:00.196 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.WAL_size_limit_MB: 0 2022-09-10T04:20:00.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2022-09-10T04:20:00.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.manifest_preallocation_size: 4194304 2022-09-10T04:20:00.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.is_fd_close_on_exec: 1 2022-09-10T04:20:00.197 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.advise_random_on_open: 1 2022-09-10T04:20:00.198 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.db_write_buffer_size: 0 2022-09-10T04:20:00.198 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.write_buffer_manager: 0x55c691d5eb10 2022-09-10T04:20:00.198 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.access_hint_on_compaction_start: 1 2022-09-10T04:20:00.198 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2022-09-10T04:20:00.198 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.random_access_max_buffer_size: 1048576 2022-09-10T04:20:00.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.use_adaptive_mutex: 0 2022-09-10T04:20:00.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.rate_limiter: (nil) 2022-09-10T04:20:00.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2022-09-10T04:20:00.199 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.wal_recovery_mode: 2 2022-09-10T04:20:00.200 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.enable_thread_tracking: 0 2022-09-10T04:20:00.200 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.enable_pipelined_write: 0 2022-09-10T04:20:00.200 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.unordered_write: 0 2022-09-10T04:20:00.200 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.allow_concurrent_memtable_write: 1 2022-09-10T04:20:00.200 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2022-09-10T04:20:00.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.write_thread_max_yield_usec: 100 2022-09-10T04:20:00.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.write_thread_slow_yield_usec: 3 2022-09-10T04:20:00.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.row_cache: None 2022-09-10T04:20:00.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.wal_filter: None 2022-09-10T04:20:00.201 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.avoid_flush_during_recovery: 0 2022-09-10T04:20:00.202 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.allow_ingest_behind: 0 2022-09-10T04:20:00.202 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.preserve_deletes: 0 2022-09-10T04:20:00.202 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.two_write_queues: 0 2022-09-10T04:20:00.203 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.manual_wal_flush: 0 2022-09-10T04:20:00.203 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.atomic_flush: 0 2022-09-10T04:20:00.203 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2022-09-10T04:20:00.203 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.persist_stats_to_disk: 0 2022-09-10T04:20:00.204 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.write_dbid_to_manifest: 0 2022-09-10T04:20:00.204 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.log_readahead_size: 0 2022-09-10T04:20:00.204 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.file_checksum_gen_factory: Unknown 2022-09-10T04:20:00.204 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.best_efforts_recovery: 0 2022-09-10T04:20:00.204 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2022-09-10T04:20:00.205 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2022-09-10T04:20:00.205 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.allow_data_in_errors: 0 2022-09-10T04:20:00.205 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.db_host_id: __hostname__ 2022-09-10T04:20:00.205 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_background_jobs: 2 2022-09-10T04:20:00.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_background_compactions: -1 2022-09-10T04:20:00.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_subcompactions: 1 2022-09-10T04:20:00.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.avoid_flush_during_shutdown: 0 2022-09-10T04:20:00.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2022-09-10T04:20:00.207 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.delayed_write_rate : 16777216 2022-09-10T04:20:00.207 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_total_wal_size: 0 2022-09-10T04:20:00.207 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2022-09-10T04:20:00.207 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.stats_dump_period_sec: 600 2022-09-10T04:20:00.207 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.stats_persist_period_sec: 600 2022-09-10T04:20:00.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.stats_history_buffer_size: 1048576 2022-09-10T04:20:00.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_open_files: -1 2022-09-10T04:20:00.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bytes_per_sync: 0 2022-09-10T04:20:00.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.wal_bytes_per_sync: 0 2022-09-10T04:20:00.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.strict_bytes_per_sync: 0 2022-09-10T04:20:00.209 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_readahead_size: 0 2022-09-10T04:20:00.209 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_background_flushes: -1 2022-09-10T04:20:00.209 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Compression algorithms supported: 2022-09-10T04:20:00.209 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kZSTDNotFinalCompression supported: 0 2022-09-10T04:20:00.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kZSTD supported: 0 2022-09-10T04:20:00.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kXpressCompression supported: 0 2022-09-10T04:20:00.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kLZ4HCCompression supported: 1 2022-09-10T04:20:00.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kZlibCompression supported: 1 2022-09-10T04:20:00.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kSnappyCompression supported: 1 2022-09-10T04:20:00.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kLZ4Compression supported: 1 2022-09-10T04:20:00.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: kBZip2Compression supported: 0 2022-09-10T04:20:00.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Fast CRC32 supported: Supported on x86 2022-09-10T04:20:00.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2022-09-10T04:20:00.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2022-09-10T04:20:00.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2022-09-10T04:20:00.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.merge_operator: 2022-09-10T04:20:00.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_filter: None 2022-09-10T04:20:00.212 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_filter_factory: None 2022-09-10T04:20:00.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.sst_partitioner_factory: None 2022-09-10T04:20:00.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.memtable_factory: SkipListFactory 2022-09-10T04:20:00.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.table_factory: BlockBasedTable 2022-09-10T04:20:00.213 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c691c66d48) 2022-09-10T04:20:00.213 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks: 1 2022-09-10T04:20:00.214 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2022-09-10T04:20:00.214 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2022-09-10T04:20:00.214 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_top_level_index_and_filter: 1 2022-09-10T04:20:00.214 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_type: 0 2022-09-10T04:20:00.215 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_index_type: 0 2022-09-10T04:20:00.215 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_shortening: 1 2022-09-10T04:20:00.215 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2022-09-10T04:20:00.215 INFO:journalctl@ceph.mon.b.smithi104.stdout: hash_index_allow_collision: 1 2022-09-10T04:20:00.215 INFO:journalctl@ceph.mon.b.smithi104.stdout: checksum: 1 2022-09-10T04:20:00.216 INFO:journalctl@ceph.mon.b.smithi104.stdout: no_block_cache: 0 2022-09-10T04:20:00.216 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache: 0x55c691cd29b0 2022-09-10T04:20:00.216 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_name: BinnedLRUCache 2022-09-10T04:20:00.216 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_options: 2022-09-10T04:20:00.217 INFO:journalctl@ceph.mon.b.smithi104.stdout: capacity : 536870912 2022-09-10T04:20:00.217 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_shard_bits : 4 2022-09-10T04:20:00.217 INFO:journalctl@ceph.mon.b.smithi104.stdout: strict_capacity_limit : 0 2022-09-10T04:20:00.217 INFO:journalctl@ceph.mon.b.smithi104.stdout: high_pri_pool_ratio: 0.000 2022-09-10T04:20:00.217 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_compressed: (nil) 2022-09-10T04:20:00.218 INFO:journalctl@ceph.mon.b.smithi104.stdout: persistent_cache: (nil) 2022-09-10T04:20:00.218 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size: 4096 2022-09-10T04:20:00.218 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size_deviation: 10 2022-09-10T04:20:00.218 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_restart_interval: 16 2022-09-10T04:20:00.218 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_block_restart_interval: 1 2022-09-10T04:20:00.219 INFO:journalctl@ceph.mon.b.smithi104.stdout: metadata_block_size: 4096 2022-09-10T04:20:00.219 INFO:journalctl@ceph.mon.b.smithi104.stdout: partition_filters: 0 2022-09-10T04:20:00.219 INFO:journalctl@ceph.mon.b.smithi104.stdout: use_delta_encoding: 1 2022-09-10T04:20:00.219 INFO:journalctl@ceph.mon.b.smithi104.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2022-09-10T04:20:00.220 INFO:journalctl@ceph.mon.b.smithi104.stdout: whole_key_filtering: 1 2022-09-10T04:20:00.220 INFO:journalctl@ceph.mon.b.smithi104.stdout: verify_compression: 0 2022-09-10T04:20:00.220 INFO:journalctl@ceph.mon.b.smithi104.stdout: read_amp_bytes_per_bit: 0 2022-09-10T04:20:00.220 INFO:journalctl@ceph.mon.b.smithi104.stdout: format_version: 4 2022-09-10T04:20:00.220 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_index_compression: 1 2022-09-10T04:20:00.221 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_align: 0 2022-09-10T04:20:00.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.write_buffer_size: 33554432 2022-09-10T04:20:00.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_write_buffer_number: 2 2022-09-10T04:20:00.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression: NoCompression 2022-09-10T04:20:00.221 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression: Disabled 2022-09-10T04:20:00.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.prefix_extractor: nullptr 2022-09-10T04:20:00.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2022-09-10T04:20:00.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.num_levels: 7 2022-09-10T04:20:00.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2022-09-10T04:20:00.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2022-09-10T04:20:00.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2022-09-10T04:20:00.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2022-09-10T04:20:00.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.level: 32767 2022-09-10T04:20:00.223 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2022-09-10T04:20:00.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2022-09-10T04:20:00.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2022-09-10T04:20:00.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2022-09-10T04:20:00.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bottommost_compression_opts.enabled: false 2022-09-10T04:20:00.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.window_bits: -14 2022-09-10T04:20:00.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.level: 32767 2022-09-10T04:20:00.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.strategy: 0 2022-09-10T04:20:00.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2022-09-10T04:20:00.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2022-09-10T04:20:00.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.parallel_threads: 1 2022-09-10T04:20:00.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compression_opts.enabled: false 2022-09-10T04:20:00.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2022-09-10T04:20:00.226 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2022-09-10T04:20:00.227 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.level0_stop_writes_trigger: 36 2022-09-10T04:20:00.227 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.target_file_size_base: 67108864 2022-09-10T04:20:00.227 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.target_file_size_multiplier: 1 2022-09-10T04:20:00.227 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_base: 268435456 2022-09-10T04:20:00.228 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2022-09-10T04:20:00.228 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2022-09-10T04:20:00.228 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2022-09-10T04:20:00.228 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2022-09-10T04:20:00.228 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2022-09-10T04:20:00.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2022-09-10T04:20:00.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2022-09-10T04:20:00.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2022-09-10T04:20:00.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2022-09-10T04:20:00.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2022-09-10T04:20:00.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_compaction_bytes: 1677721600 2022-09-10T04:20:00.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.arena_block_size: 4194304 2022-09-10T04:20:00.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2022-09-10T04:20:00.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2022-09-10T04:20:00.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2022-09-10T04:20:00.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.disable_auto_compactions: 0 2022-09-10T04:20:00.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2022-09-10T04:20:00.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2022-09-10T04:20:00.231 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2022-09-10T04:20:00.232 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2022-09-10T04:20:00.232 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2022-09-10T04:20:00.232 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2022-09-10T04:20:00.232 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2022-09-10T04:20:00.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2022-09-10T04:20:00.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2022-09-10T04:20:00.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2022-09-10T04:20:00.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.table_properties_collectors: 2022-09-10T04:20:00.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.inplace_update_support: 0 2022-09-10T04:20:00.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.inplace_update_num_locks: 10000 2022-09-10T04:20:00.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2022-09-10T04:20:00.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.memtable_whole_key_filtering: 0 2022-09-10T04:20:00.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.memtable_huge_page_size: 0 2022-09-10T04:20:00.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.bloom_locality: 0 2022-09-10T04:20:00.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.max_successive_merges: 0 2022-09-10T04:20:00.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.optimize_filters_for_hits: 0 2022-09-10T04:20:00.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.paranoid_file_checks: 0 2022-09-10T04:20:00.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.force_consistency_checks: 1 2022-09-10T04:20:00.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.report_bg_io_stats: 0 2022-09-10T04:20:00.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.ttl: 2592000 2022-09-10T04:20:00.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.periodic_compaction_seconds: 0 2022-09-10T04:20:00.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.enable_blob_files: false 2022-09-10T04:20:00.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.min_blob_size: 0 2022-09-10T04:20:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.blob_file_size: 268435456 2022-09-10T04:20:00.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.blob_compression_type: NoCompression 2022-09-10T04:20:00.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.enable_blob_garbage_collection: false 2022-09-10T04:20:00.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2022-09-10T04:20:00.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2022-09-10T04:20:00.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2022-09-10T04:20:00.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2022-09-10T04:20:00.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783600180696, "job": 1, "event": "recovery_started", "wal_files": [4]} 2022-09-10T04:20:00.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2022-09-10T04:20:00.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2022-09-10T04:20:00.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783600181065, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1662783600, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c2451ed1-ba18-4e3b-814a-a6e31e62f9fe", "db_session_id": "U2ZTSVT407X4KXNGB3NR"}} 2022-09-10T04:20:00.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2022-09-10T04:20:00.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: EVENT_LOG_v1 {"time_micros": 1662783600181496, "job": 1, "event": "recovery_finished"} 2022-09-10T04:20:00.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2022-09-10T04:20:00.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55c691cb9880 2022-09-10T04:20:00.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: DB pointer 0x55c691cc8000 2022-09-10T04:20:00.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2022-09-10T04:20:00.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: rocksdb: [db/db_impl/db_impl.cc:903] 2022-09-10T04:20:00.553 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** DB Stats ** 2022-09-10T04:20:00.553 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:20:00.554 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2022-09-10T04:20:00.554 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2022-09-10T04:20:00.554 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-10T04:20:00.554 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2022-09-10T04:20:00.555 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2022-09-10T04:20:00.555 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2022-09-10T04:20:00.555 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.556 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2022-09-10T04:20:00.556 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:20:00.557 INFO:journalctl@ceph.mon.b.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:20:00.557 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.557 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.557 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.558 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.558 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2022-09-10T04:20:00.558 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:20:00.558 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:20:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:20:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-10T04:20:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-10T04:20:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-10T04:20:00.560 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-10T04:20:00.560 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-10T04:20:00.560 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:20:00.560 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:20:00.561 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-10T04:20:00.561 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.561 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-10T04:20:00.561 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.562 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2022-09-10T04:20:00.562 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:20:00.562 INFO:journalctl@ceph.mon.b.smithi104.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:20:00.563 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.563 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.563 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2022-09-10T04:20:00.563 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2022-09-10T04:20:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2022-09-10T04:20:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2022-09-10T04:20:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2022-09-10T04:20:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2022-09-10T04:20:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2022-09-10T04:20:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2022-09-10T04:20:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2022-09-10T04:20:00.566 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2022-09-10T04:20:00.566 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2022-09-10T04:20:00.566 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:20:00.566 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2022-09-10T04:20:00.567 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2022-09-10T04:20:00.567 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.567 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2022-09-10T04:20:00.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b does not exist in monmap, will attempt to join an existing cluster 2022-09-10T04:20:00.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: using public_addr v2:172.21.15.104:0/0 -> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2022-09-10T04:20:00.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: starting mon.b rank -1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:20:00.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(???) e0 preinit fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:20:00.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).mds e1 new map 2022-09-10T04:20:00.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).mds e1 print_map 2022-09-10T04:20:00.569 INFO:journalctl@ceph.mon.b.smithi104.stdout: e1 2022-09-10T04:20:00.569 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2022-09-10T04:20:00.570 INFO:journalctl@ceph.mon.b.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-10T04:20:00.570 INFO:journalctl@ceph.mon.b.smithi104.stdout: legacy client fscid: -1 2022-09-10T04:20:00.570 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2022-09-10T04:20:00.570 INFO:journalctl@ceph.mon.b.smithi104.stdout: No filesystems configured 2022-09-10T04:20:00.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2022-09-10T04:20:00.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2022-09-10T04:20:00.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2022-09-10T04:20:00.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2022-09-10T04:20:00.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2022-09-10T04:20:00.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2022-09-10T04:20:00.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2022-09-10T04:20:00.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:20:00.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:20:00.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2022-09-10T04:20:00.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:20:00.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:20:00.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:00.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Deploying cephadm binary to smithi110 2022-09-10T04:20:00.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:00.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:00.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Added host smithi110 2022-09-10T04:20:00.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:20:00.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:00.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:20:00.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:20:00.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:20:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:20:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/108200684' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2022-09-10T04:20:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/108200684' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2022-09-10T04:20:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Saving service mon spec with placement smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c;count:3 2022-09-10T04:20:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:00.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:00.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:00.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:20:00.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:00.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:20:00.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:20:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Deploying daemon mon.c on smithi110 2022-09-10T04:20:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/200588986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:20:00.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:20:00.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.a calling monitor election 2022-09-10T04:20:00.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.c calling monitor election 2022-09-10T04:20:00.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-10T04:20:00.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: monmap e2: 2 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:00.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: fsmap 2022-09-10T04:20:00.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:00.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mgrmap e13: x(active, since 47s) 2022-09-10T04:20:00.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: overall HEALTH_OK 2022-09-10T04:20:00.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:00.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:00.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:00.595 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: Deploying daemon mon.b on smithi104 2022-09-10T04:20:00.595 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:00.595 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:00.595 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/670268749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:20:00.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2022-09-10T04:20:00.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 ceph-mon[97044]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2022-09-10T04:20:00.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:00 smithi104 systemd[1]: Started Ceph mon.b for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:20:05.538 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: mon.c calling monitor election 2022-09-10T04:20:05.538 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:20:05.538 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: mon.a calling monitor election 2022-09-10T04:20:05.539 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.539 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:05.539 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.540 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:05.540 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.540 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.540 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:05.541 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.541 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.541 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-10T04:20:05.541 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:05.542 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: fsmap 2022-09-10T04:20:05.542 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:05.542 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: mgrmap e13: x(active, since 56s) 2022-09-10T04:20:05.542 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2022-09-10T04:20:05.543 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-09-10T04:20:05.543 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-09-10T04:20:05.543 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: mon.b (rank 2) addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] is down (out of quorum) 2022-09-10T04:20:05.543 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:05.544 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:05 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:05.547 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 -- 172.21.15.110:0/1004849868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33e0003440 msgr2=0x7f33e00058c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:05.549 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 --2- 172.21.15.110:0/1004849868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33e0003440 0x7f33e00058c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f33f80fed40 tx=0x7f33ec012d10 comp rx=0 tx=0).stop 2022-09-10T04:20:05.549 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 -- 172.21.15.110:0/1004849868 shutdown_connections 2022-09-10T04:20:05.549 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 --2- 172.21.15.110:0/1004849868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33e0003440 0x7f33e00058c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.550 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 -- 172.21.15.110:0/1004849868 >> 172.21.15.110:0/1004849868 conn(0x7f33f80f9550 msgr2=0x7f33f80fb960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:05.550 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 -- 172.21.15.110:0/1004849868 shutdown_connections 2022-09-10T04:20:05.550 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.545+0000 7f33fe14b700 1 -- 172.21.15.110:0/1004849868 wait complete. 2022-09-10T04:20:05.550 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 Processor -- start 2022-09-10T04:20:05.551 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 -- start start 2022-09-10T04:20:05.551 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:05.551 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 0x7f33f8071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:05.552 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33f819f070 0x7f33f81a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:05.552 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f81a1b30 con 0x7f33f819f070 2022-09-10T04:20:05.552 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f81a1ca0 con 0x7f33e0003440 2022-09-10T04:20:05.552 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.546+0000 7f33fe14b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f81a1e60 con 0x7f33f8071520 2022-09-10T04:20:05.553 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f6ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 0x7f33f8071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:05.553 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:05.553 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33f819f070 0x7f33f81a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:05.554 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.110:59870/0 (socket says 172.21.15.110:59870) 2022-09-10T04:20:05.554 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 -- 172.21.15.110:0/3151064421 learned_addr learned my addr 172.21.15.110:0/3151064421 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:20:05.554 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f6ffd700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 msgr2=0x7f33f8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2022-09-10T04:20:05.554 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f6ffd700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 msgr2=0x7f33f8071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-09-10T04:20:05.555 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 msgr2=0x7f33f8070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2022-09-10T04:20:05.555 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f6ffd700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 0x7f33f8071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-09-10T04:20:05.555 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 msgr2=0x7f33f8070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2022-09-10T04:20:05.555 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2022-09-10T04:20:05.556 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f77fe700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:20:05.556 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f6ffd700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 0x7f33f8071990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:20:05.556 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 msgr2=0x7f33f8071990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:20:05.557 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 0x7f33f8071990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.557 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 msgr2=0x7f33f8070fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:20:05.558 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.558 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 -- 172.21.15.110:0/3151064421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec012760 con 0x7f33f819f070 2022-09-10T04:20:05.558 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f7fff700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33f819f070 0x7f33f81a1450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33f000bd50 tx=0x7f33f000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:05.559 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f000cd10 con 0x7f33f819f070 2022-09-10T04:20:05.560 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.547+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f33f000ce70 con 0x7f33f819f070 2022-09-10T04:20:05.560 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.548+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f81a2140 con 0x7f33f819f070 2022-09-10T04:20:05.561 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.548+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0012840 con 0x7f33f819f070 2022-09-10T04:20:05.561 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.548+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f81a2650 con 0x7f33f819f070 2022-09-10T04:20:05.563 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: mon.c calling monitor election 2022-09-10T04:20:05.563 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:20:05.563 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: mon.a calling monitor election 2022-09-10T04:20:05.564 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.564 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:05.564 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:05.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:05.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-10T04:20:05.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:05.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: fsmap 2022-09-10T04:20:05.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:05.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: mgrmap e13: x(active, since 56s) 2022-09-10T04:20:05.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2022-09-10T04:20:05.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-09-10T04:20:05.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-09-10T04:20:05.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: mon.b (rank 2) addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] is down (out of quorum) 2022-09-10T04:20:05.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:05.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:05 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:05.571 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.549+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f33f001b030 con 0x7f33f819f070 2022-09-10T04:20:05.571 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.549+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f8060310 con 0x7f33f819f070 2022-09-10T04:20:05.571 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.549+0000 7f33f4ff9700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33e4035d90 0x7f33e4038240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:05.572 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.549+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f33f00471b0 con 0x7f33f819f070 2022-09-10T04:20:05.572 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.549+0000 7f33f77fe700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33e4035d90 0x7f33e4038240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:05.572 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.550+0000 7f33f77fe700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33e4035d90 0x7f33e4038240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33ec001eb0 tx=0x7f33ec014040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:05.572 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.554+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f33f00116b0 con 0x7f33f819f070 2022-09-10T04:20:05.824 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.823+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33f81055f0 con 0x7f33f819f070 2022-09-10T04:20:05.825 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.823+0000 7f33f4ff9700 1 -- 172.21.15.110:0/3151064421 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1248 (secure 0 0 0) 0x7f33f0019030 con 0x7f33f819f070 2022-09-10T04:20:05.825 INFO:teuthology.orchestra.run.smithi110.stdout: 2022-09-10T04:20:05.826 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","modified":"2022-09-10T04:20:00.207945Z","created":"2022-09-10T04:18:01.537043Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2022-09-10T04:20:05.827 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33e4035d90 msgr2=0x7f33e4038240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:05.827 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33e4035d90 0x7f33e4038240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f33ec001eb0 tx=0x7f33ec014040 comp rx=0 tx=0).stop 2022-09-10T04:20:05.828 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33f819f070 msgr2=0x7f33f81a1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:05.828 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33f819f070 0x7f33f81a1450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f33f000bd50 tx=0x7f33f000be30 comp rx=0 tx=0).stop 2022-09-10T04:20:05.828 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 shutdown_connections 2022-09-10T04:20:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33f819f070 0x7f33f81a1450 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33e0003440 0x7f33f8070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33e4035d90 0x7f33e4038240 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 --2- 172.21.15.110:0/3151064421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33f8071520 0x7f33f8071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:05.829 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 >> 172.21.15.110:0/3151064421 conn(0x7f33f80f9550 msgr2=0x7f33f80ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:05.830 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 shutdown_connections 2022-09-10T04:20:05.830 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:05.825+0000 7f33fe14b700 1 -- 172.21.15.110:0/3151064421 wait complete. 2022-09-10T04:20:05.830 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 3 2022-09-10T04:20:06.274 INFO:tasks.cephadm:Generating final ceph.conf file... 2022-09-10T04:20:06.275 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph config generate-minimal-conf 2022-09-10T04:20:06.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:06.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:06.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:06.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:06.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:20:06.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/3151064421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:20:06.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:20:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/3151064421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:20:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.196 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:20:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: mon.c calling monitor election 2022-09-10T04:20:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:20:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: mon.a calling monitor election 2022-09-10T04:20:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:20:07.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:07.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:07.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:07.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2022-09-10T04:20:07.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:07.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: fsmap 2022-09-10T04:20:07.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:07.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: mgrmap e13: x(active, since 56s) 2022-09-10T04:20:07.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2022-09-10T04:20:07.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2022-09-10T04:20:07.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2022-09-10T04:20:07.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: mon.b (rank 2) addr [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] is down (out of quorum) 2022-09-10T04:20:07.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:07.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:07.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:07.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:07.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:07.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:07.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:20:07.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/3151064421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2022-09-10T04:20:07.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:08.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: mon.b calling monitor election 2022-09-10T04:20:08.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:20:08.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:20:08.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:08.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: mon.b calling monitor election 2022-09-10T04:20:08.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: mon.c calling monitor election 2022-09-10T04:20:08.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: mon.a calling monitor election 2022-09-10T04:20:08.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-09-10T04:20:08.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:08.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: fsmap 2022-09-10T04:20:08.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:08.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: mgrmap e13: x(active, since 58s) 2022-09-10T04:20:08.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-09-10T04:20:08.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Cluster is now healthy 2022-09-10T04:20:08.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:08.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: overall HEALTH_OK 2022-09-10T04:20:08.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:08.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Reconfiguring mon.a (unknown last config time)... 2022-09-10T04:20:08.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:08.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:08.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:08.244 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:08 smithi081 ceph-mon[90133]: Reconfiguring daemon mon.a on smithi081 2022-09-10T04:20:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: mon.b calling monitor election 2022-09-10T04:20:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:20:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:20:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: mon.b calling monitor election 2022-09-10T04:20:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: mon.c calling monitor election 2022-09-10T04:20:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: mon.a calling monitor election 2022-09-10T04:20:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-09-10T04:20:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:08.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: fsmap 2022-09-10T04:20:08.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:08.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: mgrmap e13: x(active, since 58s) 2022-09-10T04:20:08.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-09-10T04:20:08.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Cluster is now healthy 2022-09-10T04:20:08.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:08.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: overall HEALTH_OK 2022-09-10T04:20:08.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:08.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Reconfiguring mon.a (unknown last config time)... 2022-09-10T04:20:08.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:08.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:08.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:08.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:08 smithi104 ceph-mon[97044]: Reconfiguring daemon mon.a on smithi081 2022-09-10T04:20:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: mon.b calling monitor election 2022-09-10T04:20:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:20:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:20:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: mon.b calling monitor election 2022-09-10T04:20:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: mon.c calling monitor election 2022-09-10T04:20:08.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: mon.a calling monitor election 2022-09-10T04:20:08.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2022-09-10T04:20:08.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:20:08.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: fsmap 2022-09-10T04:20:08.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: osdmap e4: 0 total, 0 up, 0 in 2022-09-10T04:20:08.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: mgrmap e13: x(active, since 58s) 2022-09-10T04:20:08.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2022-09-10T04:20:08.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Cluster is now healthy 2022-09-10T04:20:08.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:08.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: overall HEALTH_OK 2022-09-10T04:20:08.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:20:08.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Reconfiguring mon.a (unknown last config time)... 2022-09-10T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:08 smithi110 ceph-mon[97633]: Reconfiguring daemon mon.a on smithi081 2022-09-10T04:20:08.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 -- 172.21.15.81:0/536138131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce8071080 msgr2=0x7f6ce8071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:08.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/536138131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce8071080 0x7f6ce8071450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009790 tx=0x7f6ce4009aa0 comp rx=0 tx=0).stop 2022-09-10T04:20:08.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 -- 172.21.15.81:0/536138131 shutdown_connections 2022-09-10T04:20:08.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/536138131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce8071080 0x7f6ce8071450 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:08.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 -- 172.21.15.81:0/536138131 >> 172.21.15.81:0/536138131 conn(0x7f6ce80f9260 msgr2=0x7f6ce80fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:08.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 -- 172.21.15.81:0/536138131 shutdown_connections 2022-09-10T04:20:08.863 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.859+0000 7f6cee8a4700 1 -- 172.21.15.81:0/536138131 wait complete. 2022-09-10T04:20:08.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 Processor -- start 2022-09-10T04:20:08.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 -- start start 2022-09-10T04:20:08.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ce8071080 0x7f6ce8199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:08.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ce8199660 0x7f6ce819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:08.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 0x7f6ce819e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:08.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce8199d90 con 0x7f6ce819e000 2022-09-10T04:20:08.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81a5d70 con 0x7f6ce8199660 2022-09-10T04:20:08.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.861+0000 7f6cee8a4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81a5ee0 con 0x7f6ce8071080 2022-09-10T04:20:08.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 0x7f6ce819e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:08.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6ced8a2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ce8071080 0x7f6ce8199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:08.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 0x7f6ce819e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52660/0 (socket says 172.21.15.81:52660) 2022-09-10T04:20:08.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6ced8a2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ce8071080 0x7f6ce8199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59528/0 (socket says 172.21.15.81:59528) 2022-09-10T04:20:08.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 -- 172.21.15.81:0/1512015947 learned_addr learned my addr 172.21.15.81:0/1512015947 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:20:08.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6ced0a1700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ce8199660 0x7f6ce819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:08.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 -- 172.21.15.81:0/1512015947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ce8071080 msgr2=0x7f6ce8199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:08.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ce8071080 0x7f6ce8199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:08.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 -- 172.21.15.81:0/1512015947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ce8199660 msgr2=0x7f6ce819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:08.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ce8199660 0x7f6ce819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:08.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 -- 172.21.15.81:0/1512015947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4009470 con 0x7f6ce819e000 2022-09-10T04:20:08.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.862+0000 7f6cee0a3700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 0x7f6ce819e470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000ba50 tx=0x7f6ce000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:08.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.863+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce000c930 con 0x7f6ce819e000 2022-09-10T04:20:08.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.863+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6ce0014440 con 0x7f6ce819e000 2022-09-10T04:20:08.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.863+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce81a61c0 con 0x7f6ce819e000 2022-09-10T04:20:08.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.863+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00075b0 con 0x7f6ce819e000 2022-09-10T04:20:08.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.863+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce81a6680 con 0x7f6ce819e000 2022-09-10T04:20:08.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.864+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f6ce001b030 con 0x7f6ce819e000 2022-09-10T04:20:08.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.864+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce8060310 con 0x7f6ce819e000 2022-09-10T04:20:08.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.864+0000 7f6cdeffd700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6cd4035d90 0x7f6cd4038240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:08.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.864+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6ce0046e30 con 0x7f6ce819e000 2022-09-10T04:20:08.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.865+0000 7f6ced8a2700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6cd4035d90 0x7f6cd4038240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:08.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.865+0000 7f6ced8a2700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6cd4035d90 0x7f6cd4038240 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006010 tx=0x7f6ce4005930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:08.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:08.870+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6ce0014920 con 0x7f6ce819e000 2022-09-10T04:20:09.106 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.104+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6ce81a6d20 con 0x7f6ce819e000 2022-09-10T04:20:09.106 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.104+0000 7f6cdeffd700 1 -- 172.21.15.81:0/1512015947 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f6ce0019030 con 0x7f6ce819e000 2022-09-10T04:20:09.107 INFO:teuthology.orchestra.run.smithi081.stdout:# minimal ceph.conf for 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:20:09.107 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2022-09-10T04:20:09.108 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:20:09.108 INFO:teuthology.orchestra.run.smithi081.stdout: mon_host = [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2022-09-10T04:20:09.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6cd4035d90 msgr2=0x7f6cd4038240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:09.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6cd4035d90 0x7f6cd4038240 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4006010 tx=0x7f6ce4005930 comp rx=0 tx=0).stop 2022-09-10T04:20:09.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 msgr2=0x7f6ce819e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:09.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 0x7f6ce819e470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6ce000ba50 tx=0x7f6ce000bd60 comp rx=0 tx=0).stop 2022-09-10T04:20:09.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 shutdown_connections 2022-09-10T04:20:09.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ce819e000 0x7f6ce819e470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:09.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6cd4035d90 0x7f6cd4038240 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:09.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ce8071080 0x7f6ce8199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:09.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 --2- 172.21.15.81:0/1512015947 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ce8199660 0x7f6ce819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:09.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 >> 172.21.15.81:0/1512015947 conn(0x7f6ce80f9260 msgr2=0x7f6ce80f9630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:09.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 shutdown_connections 2022-09-10T04:20:09.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:09.107+0000 7f6cee8a4700 1 -- 172.21.15.81:0/1512015947 wait complete. 2022-09-10T04:20:09.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:20:09.804 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2022-09-10T04:20:09.805 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:20:09.805 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.conf 2022-09-10T04:20:09.908 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:20:09.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:20:10.035 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:20:10.036 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2022-09-10T04:20:10.067 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:20:10.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:20:10.136 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:20:10.137 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2022-09-10T04:20:10.170 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:20:10.170 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:20:10.239 INFO:tasks.cephadm:Adding mgr.x on smithi081 2022-09-10T04:20:10.240 INFO:tasks.cephadm:Adding mgr.y on smithi104 2022-09-10T04:20:10.240 INFO:tasks.cephadm:Adding mgr.z on smithi110 2022-09-10T04:20:10.240 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch apply mgr '3;smithi081=x;smithi104=y;smithi110=z' 2022-09-10T04:20:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1512015947' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:20:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:20:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1512015947' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:20:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:20:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:10.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:10.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1512015947' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:20:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:20:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:10.871 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:20:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:11 smithi104 ceph-mon[97044]: Reconfiguring mon.b (monmap changed)... 2022-09-10T04:20:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:11 smithi104 ceph-mon[97044]: Reconfiguring daemon mon.b on smithi104 2022-09-10T04:20:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:11 smithi110 ceph-mon[97633]: Reconfiguring mon.b (monmap changed)... 2022-09-10T04:20:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:11 smithi110 ceph-mon[97633]: Reconfiguring daemon mon.b on smithi104 2022-09-10T04:20:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:11 smithi081 ceph-mon[90133]: Reconfiguring mon.b (monmap changed)... 2022-09-10T04:20:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:11 smithi081 ceph-mon[90133]: Reconfiguring daemon mon.b on smithi104 2022-09-10T04:20:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 -- 172.21.15.110:0/1592587738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 msgr2=0x7f09740fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 --2- 172.21.15.110:0/1592587738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f09740fcc50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f096c009790 tx=0x7f096c009aa0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.346 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 -- 172.21.15.110:0/1592587738 shutdown_connections 2022-09-10T04:20:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 --2- 172.21.15.110:0/1592587738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f09740fcc50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 -- 172.21.15.110:0/1592587738 >> 172.21.15.110:0/1592587738 conn(0x7f09740f7f50 msgr2=0x7f09740fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:12.347 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 -- 172.21.15.110:0/1592587738 shutdown_connections 2022-09-10T04:20:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 -- 172.21.15.110:0/1592587738 wait complete. 2022-09-10T04:20:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.341+0000 7f0979776700 1 Processor -- start 2022-09-10T04:20:12.348 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 -- start start 2022-09-10T04:20:12.349 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f0974067760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:12.349 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 0x7f0974197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:12.349 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09741981f0 0x7f097419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:12.349 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097419ae20 con 0x7f09740fc840 2022-09-10T04:20:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097419af90 con 0x7f09741981f0 2022-09-10T04:20:12.350 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0979776700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097419b150 con 0x7f09741957f0 2022-09-10T04:20:12.351 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 0x7f0974197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 0x7f0974197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60596/0 (socket says 172.21.15.110:60596) 2022-09-10T04:20:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 -- 172.21.15.110:0/2924265484 learned_addr learned my addr 172.21.15.110:0/2924265484 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:20:12.352 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0972ffd700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f0974067760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09737fe700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09741981f0 0x7f097419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 -- 172.21.15.110:0/2924265484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09741981f0 msgr2=0x7f097419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:12.353 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09741981f0 0x7f097419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 -- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 msgr2=0x7f0974067760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f0974067760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.354 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 -- 172.21.15.110:0/2924265484 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c009470 con 0x7f09741957f0 2022-09-10T04:20:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f0972ffd700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f0974067760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:20:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f09727fc700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 0x7f0974197bd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f095c009fd0 tx=0x7f095c00b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.342+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c004230 con 0x7f09741957f0 2022-09-10T04:20:12.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.343+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f095c004390 con 0x7f09741957f0 2022-09-10T04:20:12.356 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.343+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097419b430 con 0x7f09741957f0 2022-09-10T04:20:12.356 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.343+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095c003b10 con 0x7f09741957f0 2022-09-10T04:20:12.356 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.343+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097419ba10 con 0x7f09741957f0 2022-09-10T04:20:12.357 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.343+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7f095c017020 con 0x7f09741957f0 2022-09-10T04:20:12.357 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.344+0000 7f096bfff700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0960035d40 0x7f09600381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:12.357 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.344+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f095c046fb0 con 0x7f09741957f0 2022-09-10T04:20:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.344+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0974102440 con 0x7f09741957f0 2022-09-10T04:20:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.344+0000 7f0972ffd700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0960035d40 0x7f09600381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:12.358 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.345+0000 7f0972ffd700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0960035d40 0x7f09600381f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f096c009f20 tx=0x7f096c019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:12.359 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.348+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f095c003c70 con 0x7f09741957f0 2022-09-10T04:20:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: Reconfiguring mon.c (monmap changed)... 2022-09-10T04:20:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: Reconfiguring daemon mon.c on smithi110 2022-09-10T04:20:12.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:12 smithi110 ceph-mon[97633]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: Reconfiguring mon.c (monmap changed)... 2022-09-10T04:20:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: Reconfiguring daemon mon.c on smithi110 2022-09-10T04:20:12.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:12 smithi104 ceph-mon[97044]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:12.550 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.549+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi110=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f097405f1b0 con 0x7f0960035d40 2022-09-10T04:20:12.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: Reconfiguring mon.c (monmap changed)... 2022-09-10T04:20:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2022-09-10T04:20:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2022-09-10T04:20:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: Reconfiguring daemon mon.c on smithi110 2022-09-10T04:20:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:12 smithi081 ceph-mon[90133]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:12.557 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.556+0000 7f096bfff700 1 -- 172.21.15.110:0/2924265484 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f097405f1b0 con 0x7f0960035d40 2022-09-10T04:20:12.557 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mgr update... 2022-09-10T04:20:12.559 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0960035d40 msgr2=0x7f09600381f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:12.559 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0960035d40 0x7f09600381f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f096c009f20 tx=0x7f096c019040 comp rx=0 tx=0).stop 2022-09-10T04:20:12.560 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 msgr2=0x7f0974197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:12.560 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 0x7f0974197bd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f095c009fd0 tx=0x7f095c00b7e0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.560 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 shutdown_connections 2022-09-10T04:20:12.560 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f09741981f0 0x7f097419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.561 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09740fc840 0x7f0974067760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.561 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0960035d40 0x7f09600381f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.561 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 --2- 172.21.15.110:0/2924265484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09741957f0 0x7f0974197bd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:12.562 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 >> 172.21.15.110:0/2924265484 conn(0x7f09740f7f50 msgr2=0x7f09740f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:12.562 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 shutdown_connections 2022-09-10T04:20:12.562 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:20:12.558+0000 7f0979776700 1 -- 172.21.15.110:0/2924265484 wait complete. 2022-09-10T04:20:13.114 DEBUG:teuthology.orchestra.run.smithi104:mgr.y> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.y.service 2022-09-10T04:20:13.117 DEBUG:teuthology.orchestra.run.smithi110:mgr.z> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mgr.z.service 2022-09-10T04:20:13.127 INFO:tasks.cephadm:Deploying OSDs... 2022-09-10T04:20:13.128 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:20:13.128 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2022-09-10T04:20:13.152 INFO:journalctl@ceph.mgr.y.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:20:13.153 INFO:journalctl@ceph.mgr.z.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:20:13.191 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-10T04:20:13.192 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-10T04:20:13.252 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-10T04:20:13.252 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.252 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 448 Links: 1 2022-09-10T04:20:13.253 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.253 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.253 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-09-10 04:19:19.090772209 +0000 2022-09-10T04:20:13.253 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-09-10 04:12:33.386778813 +0000 2022-09-10T04:20:13.254 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-09-10 04:12:33.386778813 +0000 2022-09-10T04:20:13.254 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2022-09-10 04:12:33.349779536 +0000 2022-09-10T04:20:13.283 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-09-10T04:20:13.283 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-09-10T04:20:13.283 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00085829 s, 597 kB/s 2022-09-10T04:20:13.293 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-10T04:20:13.351 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-10T04:20:13.351 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.352 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 454 Links: 1 2022-09-10T04:20:13.352 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.352 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.352 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-09-10 04:19:19.090772209 +0000 2022-09-10T04:20:13.353 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-09-10 04:12:34.420758613 +0000 2022-09-10T04:20:13.353 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-09-10 04:12:34.420758613 +0000 2022-09-10T04:20:13.353 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2022-09-10 04:12:34.387759258 +0000 2022-09-10T04:20:13.380 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-09-10T04:20:13.381 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-09-10T04:20:13.381 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000608919 s, 841 kB/s 2022-09-10T04:20:13.391 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-10T04:20:13.449 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-10T04:20:13.449 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.450 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 460 Links: 1 2022-09-10T04:20:13.450 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.450 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.450 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-09-10 04:19:19.090772209 +0000 2022-09-10T04:20:13.451 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-09-10 04:12:35.453738432 +0000 2022-09-10T04:20:13.451 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-09-10 04:12:35.453738432 +0000 2022-09-10T04:20:13.451 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2022-09-10 04:12:35.418739116 +0000 2022-09-10T04:20:13.479 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-09-10T04:20:13.479 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-09-10T04:20:13.480 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000616223 s, 831 kB/s 2022-09-10T04:20:13.489 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-10T04:20:13.548 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-10T04:20:13.549 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.549 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 466 Links: 1 2022-09-10T04:20:13.549 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.549 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.550 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2022-09-10 04:19:19.090772209 +0000 2022-09-10T04:20:13.550 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-09-10 04:12:36.479718388 +0000 2022-09-10T04:20:13.550 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-09-10 04:12:36.479718388 +0000 2022-09-10T04:20:13.550 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2022-09-10 04:12:36.441719130 +0000 2022-09-10T04:20:13.578 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2022-09-10T04:20:13.579 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2022-09-10T04:20:13.579 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000630457 s, 812 kB/s 2022-09-10T04:20:13.589 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2022-09-10T04:20:13.589 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2022-09-10T04:20:13.611 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-10T04:20:13.612 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-10T04:20:13.671 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-10T04:20:13.672 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 359586 Links: 1 2022-09-10T04:20:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.673 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-09-10 04:19:59.482356691 +0000 2022-09-10T04:20:13.674 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-09-10 04:08:53.204689420 +0000 2022-09-10T04:20:13.675 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-09-10 04:08:53.204689420 +0000 2022-09-10T04:20:13.675 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-09-10T04:20:13.684 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-09-10T04:20:13.684 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-09-10T04:20:13.684 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000244426 s, 2.1 MB/s 2022-09-10T04:20:13.687 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-10T04:20:13.746 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-10T04:20:13.747 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.747 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 355998 Links: 1 2022-09-10T04:20:13.747 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.747 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.748 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-09-10 04:19:59.482356691 +0000 2022-09-10T04:20:13.748 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-09-10 04:08:53.746682970 +0000 2022-09-10T04:20:13.748 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-09-10 04:08:53.746682970 +0000 2022-09-10T04:20:13.748 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-09-10T04:20:13.757 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-09-10T04:20:13.758 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-09-10T04:20:13.758 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000215723 s, 2.4 MB/s 2022-09-10T04:20:13.760 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-10T04:20:13.820 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-10T04:20:13.821 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.821 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 356030 Links: 1 2022-09-10T04:20:13.821 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.821 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.822 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-09-10 04:19:59.483356679 +0000 2022-09-10T04:20:13.822 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-09-10 04:08:54.324676092 +0000 2022-09-10T04:20:13.822 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-09-10 04:08:54.324676092 +0000 2022-09-10T04:20:13.823 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-09-10T04:20:13.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi110=z", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:13.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: Saving service mgr spec with placement smithi081=x;smithi104=y;smithi110=z;count:3 2022-09-10T04:20:13.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-10T04:20:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:13.833 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-09-10T04:20:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-09-10T04:20:13.834 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000183008 s, 2.8 MB/s 2022-09-10T04:20:13.835 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-10T04:20:13.894 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-10T04:20:13.895 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.895 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 6h/6d Inode: 357851 Links: 1 2022-09-10T04:20:13.895 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.896 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.896 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2022-09-10 04:19:59.483356679 +0000 2022-09-10T04:20:13.896 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2022-09-10 04:08:54.895669297 +0000 2022-09-10T04:20:13.897 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2022-09-10 04:08:54.895669297 +0000 2022-09-10T04:20:13.897 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: - 2022-09-10T04:20:13.905 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2022-09-10T04:20:13.905 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2022-09-10T04:20:13.906 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000224891 s, 2.3 MB/s 2022-09-10T04:20:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi110=z", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: Saving service mgr spec with placement smithi081=x;smithi104=y;smithi110=z;count:3 2022-09-10T04:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-10T04:20:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:13.910 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2022-09-10T04:20:13.911 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2022-09-10T04:20:13.929 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-09-10T04:20:13.929 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-09-10T04:20:13.987 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-09-10T04:20:13.987 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:13.987 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 358351 Links: 1 2022-09-10T04:20:13.987 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:13.988 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:13.988 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2022-09-10 04:19:51.281567328 +0000 2022-09-10T04:20:13.988 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2022-09-10 04:08:53.170773029 +0000 2022-09-10T04:20:13.988 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2022-09-10 04:08:53.170773029 +0000 2022-09-10T04:20:13.989 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2022-09-10T04:20:13.999 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2022-09-10T04:20:13.999 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2022-09-10T04:20:13.999 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000113988 s, 4.5 MB/s 2022-09-10T04:20:14.002 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-09-10T04:20:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi110=z", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: Saving service mgr spec with placement smithi081=x;smithi104=y;smithi110=z;count:3 2022-09-10T04:20:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:14.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:14.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:14.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:14.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:14.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-10T04:20:14.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:14.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:14.059 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-09-10T04:20:14.059 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:14.059 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 358755 Links: 1 2022-09-10T04:20:14.060 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:14.060 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:14.060 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2022-09-10 04:19:51.281567328 +0000 2022-09-10T04:20:14.061 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2022-09-10 04:08:53.733763766 +0000 2022-09-10T04:20:14.061 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2022-09-10 04:08:53.733763766 +0000 2022-09-10T04:20:14.061 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2022-09-10T04:20:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2022-09-10T04:20:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2022-09-10T04:20:14.069 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.9152e-05 s, 5.2 MB/s 2022-09-10T04:20:14.072 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-09-10T04:20:14.128 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-09-10T04:20:14.129 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:14.129 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 360093 Links: 1 2022-09-10T04:20:14.129 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:14.129 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:14.130 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2022-09-10 04:19:51.281567328 +0000 2022-09-10T04:20:14.130 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2022-09-10 04:08:54.283754716 +0000 2022-09-10T04:20:14.130 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2022-09-10 04:08:54.283754716 +0000 2022-09-10T04:20:14.130 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2022-09-10T04:20:14.138 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2022-09-10T04:20:14.138 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2022-09-10T04:20:14.138 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000105414 s, 4.9 MB/s 2022-09-10T04:20:14.141 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-09-10T04:20:14.199 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-09-10T04:20:14.199 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-09-10T04:20:14.200 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 354762 Links: 1 2022-09-10T04:20:14.200 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:20:14.200 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2022-09-10T04:20:14.200 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2022-09-10 04:19:51.281567328 +0000 2022-09-10T04:20:14.201 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2022-09-10 04:08:54.851745371 +0000 2022-09-10T04:20:14.201 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2022-09-10 04:08:54.851745371 +0000 2022-09-10T04:20:14.201 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2022-09-10T04:20:14.208 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2022-09-10T04:20:14.209 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2022-09-10T04:20:14.209 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.2854e-05 s, 6.2 MB/s 2022-09-10T04:20:14.211 INFO:tasks.cephadm:Deploying osd.0 on smithi081 with /dev/vg_nvme/lv_4... 2022-09-10T04:20:14.212 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-09-10T04:20:14.796 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:14 smithi110 ceph-mon[97633]: Deploying daemon mgr.z on smithi110 2022-09-10T04:20:14.796 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:14 smithi110 ceph-mon[97633]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:14.797 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:14 smithi110 systemd[1]: Starting Ceph mgr.z for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:20:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:14 smithi104 ceph-mon[97044]: Deploying daemon mgr.z on smithi110 2022-09-10T04:20:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:14 smithi104 ceph-mon[97044]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:14 smithi081 ceph-mon[90133]: Deploying daemon mgr.z on smithi110 2022-09-10T04:20:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:14 smithi081 ceph-mon[90133]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:15.066 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:14 smithi110 podman[99576]: 2022-09-10T04:20:15.067 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-mgr[99615]: -- 172.21.15.110:0/3141771903 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557c0c08a820 con 0x557c0c05e800 2022-09-10T04:20:15.067 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 bash[99576]: 975c4b5e3658d4eeb273222fb32da9b10180a44314f24bb3f02f2f8660feef3e 2022-09-10T04:20:15.074 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:20:15.396 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 systemd[1]: Started Ceph mgr.z for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:20:15.397 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.143+0000 7f64479df080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:20:15.397 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.219+0000 7f64479df080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:20:15.397 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.306+0000 7f64479df080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:20:15.397 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.395+0000 7f64479df080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:20:15.656 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.558+0000 7f64479df080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:20:16.127 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.863+0000 7f64479df080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:20:16.127 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:15 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:15.949+0000 7f64479df080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:20:16.127 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:16.080+0000 7f64479df080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:20:16.359 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:16.359 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:16.360 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:16.360 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-10T04:20:16.360 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:16.360 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: Deploying daemon mgr.y on smithi104 2022-09-10T04:20:16.361 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:16 smithi104 ceph-mon[97044]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:16.407 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:16.173+0000 7f64479df080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:20:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-10T04:20:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: Deploying daemon mgr.y on smithi104 2022-09-10T04:20:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-mon[97633]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:16.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:16.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:16.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:16.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2022-09-10T04:20:16.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:16.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:16.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: Deploying daemon mgr.y on smithi104 2022-09-10T04:20:16.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:16 smithi081 ceph-mon[90133]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:16.796 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:16 smithi104 systemd[1]: Starting Ceph mgr.y for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:20:16.906 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:16 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:16.583+0000 7f64479df080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:20:17.170 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:16 smithi104 podman[98387]: 2022-09-10T04:20:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-09-10T04:20:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-09-10T04:20:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2022-09-10T04:20:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-09-10T04:20:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.046311 s, 226 MB/s 2022-09-10T04:20:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2022-09-10T04:20:17.406 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:17 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:17.146+0000 7f64479df080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:20:17.407 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:17 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:17.313+0000 7f64479df080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:20:17.467 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 bash[98387]: f52df5b9ae1d13f45957dcd9c54bdb8616e565abd278043a2bcb2f50ec8af1d9 2022-09-10T04:20:17.468 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 ceph-mgr[98426]: -- 172.21.15.104:0/575219683 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5623558ce820 con 0x5623558a2c00 2022-09-10T04:20:17.468 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 systemd[1]: Started Ceph mgr.y for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:20:17.469 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:17.292+0000 7f0659f80080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:20:17.469 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:17.380+0000 7f0659f80080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:20:17.470 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:17.466+0000 7f0659f80080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:20:17.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi081:vg_nvme/lv_4 2022-09-10T04:20:17.740 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:17.563+0000 7f0659f80080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:20:18.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:17 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:17.739+0000 7f0659f80080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:20:18.324 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:18.065+0000 7f0659f80080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:20:18.325 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:18.156+0000 7f0659f80080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:20:18.325 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:18.289+0000 7f0659f80080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:20:18.349 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:18.084+0000 7f64479df080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:20:18.350 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:18.178+0000 7f64479df080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:20:18.350 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-mon[97633]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:18.351 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:18.351 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:18.351 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:18.351 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:18.351 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:18.530 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:20:18.656 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:18.348+0000 7f64479df080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:20:18.657 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:18.539+0000 7f64479df080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:20:18.796 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:18.383+0000 7f0659f80080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:20:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-mon[97044]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:18.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:18.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:18.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:18 smithi081 ceph-mon[90133]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:18.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:18.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:19.098 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:18 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:18.692+0000 7f64479df080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:20:19.297 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:18.835+0000 7f0659f80080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:20:19.364 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:19 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:19.097+0000 7f64479df080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:20:19.587 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:19 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:19.419+0000 7f0659f80080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:20:19.906 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:19 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:19.633+0000 7f64479df080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:20:20.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:19 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:19.586+0000 7f0659f80080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:20:20.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2200375222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 msgr2=0x7fcc1c1033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:20.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2200375222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1033a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009810 tx=0x7fcc10009b20 comp rx=0 tx=0).stop 2022-09-10T04:20:20.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2200375222 shutdown_connections 2022-09-10T04:20:20.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2200375222 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc1c103a30 0x7fcc1c105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:20.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2200375222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c0673d0 0x7fcc1c100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:20.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2200375222 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1033a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:20.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2200375222 >> 172.21.15.81:0/2200375222 conn(0x7fcc1c0f8430 msgr2=0x7fcc1c0fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:20.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2200375222 shutdown_connections 2022-09-10T04:20:20.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.094+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2200375222 wait complete. 2022-09-10T04:20:20.101 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.095+0000 7fcc22a1b700 1 Processor -- start 2022-09-10T04:20:20.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.095+0000 7fcc22a1b700 1 -- start start 2022-09-10T04:20:20.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc22a1b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc1c0673d0 0x7fcc1c195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:20.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc22a1b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:20.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc22a1b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c103a30 0x7fcc1c19d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc22a1b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c19d840 con 0x7fcc1c103a30 2022-09-10T04:20:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc22a1b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c19d9b0 con 0x7fcc1c100f40 2022-09-10T04:20:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc22a1b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c19db70 con 0x7fcc1c0673d0 2022-09-10T04:20:20.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:20.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc1b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:46208/0 (socket says 172.21.15.81:46208) 2022-09-10T04:20:20.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc1b7fe700 1 -- 172.21.15.81:0/2272181841 learned_addr learned my addr 172.21.15.81:0/2272181841 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:20:20.105 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc20fb8700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c103a30 0x7fcc1c19d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:20.106 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.096+0000 7fcc1bfff700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc1c0673d0 0x7fcc1c195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:20.106 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc1b7fe700 1 -- 172.21.15.81:0/2272181841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc1c0673d0 msgr2=0x7fcc1c195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:20.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc1b7fe700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc1c0673d0 0x7fcc1c195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:20.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc1b7fe700 1 -- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c103a30 msgr2=0x7fcc1c19d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:20.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc1b7fe700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c103a30 0x7fcc1c19d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:20.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc1b7fe700 1 -- 172.21.15.81:0/2272181841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc10009470 con 0x7fcc1c100f40 2022-09-10T04:20:20.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc20fb8700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c103a30 0x7fcc1c19d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:20:20.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc1b7fe700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1961c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000c00 tx=0x7fcc1000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:20.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1001d070 con 0x7fcc1c100f40 2022-09-10T04:20:20.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcc100054f0 con 0x7fcc1c100f40 2022-09-10T04:20:20.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c19ddf0 con 0x7fcc1c100f40 2022-09-10T04:20:20.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10004880 con 0x7fcc1c100f40 2022-09-10T04:20:20.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.097+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c19e260 con 0x7fcc1c100f40 2022-09-10T04:20:20.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.098+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) v1 ==== 42746+0+0 (secure 0 0 0) 0x7fcc10028030 con 0x7fcc1c100f40 2022-09-10T04:20:20.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.098+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c109640 con 0x7fcc1c100f40 2022-09-10T04:20:20.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.099+0000 7fcc197fa700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcc04035b00 0x7fcc04037fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:20.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.099+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcc100483b0 con 0x7fcc1c100f40 2022-09-10T04:20:20.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.099+0000 7fcc1bfff700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcc04035b00 0x7fcc04037fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:20.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.100+0000 7fcc1bfff700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcc04035b00 0x7fcc04037fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c197150 tx=0x7fcc0c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:20.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.104+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcc10029540 con 0x7fcc1c100f40 2022-09-10T04:20:20.251 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:19 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:19.954+0000 7f64479df080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:20:20.251 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:20.139+0000 7f64479df080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:20:20.252 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.252 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.253 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.253 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.253 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.254 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:20.254 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.255 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: Reconfiguring mgr.x (unknown last config time)... 2022-09-10T04:20:20.255 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:20.255 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:20.256 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:20.256 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-mon[97633]: Reconfiguring daemon mgr.x on smithi081 2022-09-10T04:20:20.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:20.351+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc1c105740 con 0x7fcc04035b00 2022-09-10T04:20:20.417 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:20.328+0000 7f0659f80080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:20:20.418 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.418 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.419 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:20.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: Reconfiguring mgr.x (unknown last config time)... 2022-09-10T04:20:20.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:20.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:20.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:20.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-mon[97044]: Reconfiguring daemon mgr.x on smithi081 2022-09-10T04:20:20.433 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.434 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.434 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.434 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.435 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.435 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:20.435 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:20.435 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: Reconfiguring mgr.x (unknown last config time)... 2022-09-10T04:20:20.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2022-09-10T04:20:20.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2022-09-10T04:20:20.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:20.437 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:20 smithi081 ceph-mon[90133]: Reconfiguring daemon mgr.x on smithi081 2022-09-10T04:20:20.657 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:20.250+0000 7f64479df080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:20:20.657 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:20 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:20.457+0000 7f64479df080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:20:20.762 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:20.416+0000 7f0659f80080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:20:20.762 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:20.580+0000 7f0659f80080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:20:21.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:20.761+0000 7f0659f80080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:20:21.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:20 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:20.904+0000 7f0659f80080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:20:21.165 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:21.163+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 14) v1 ==== 85192+0+0 (secure 0 0 0) 0x7fcc1000c040 con 0x7fcc1c100f40 2022-09-10T04:20:21.375 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:20:21.076+0000 7f64479df080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:20:21.376 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:21.376 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:20:21.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:20:21.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:21.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: Standby manager daemon z started 2022-09-10T04:20:21.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-10T04:20:21.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:20:21.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-10T04:20:21.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:20:21.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:21 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:21.546 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:21.295+0000 7f0659f80080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:20:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:20:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:20:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: Standby manager daemon z started 2022-09-10T04:20:21.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-10T04:20:21.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:20:21.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-10T04:20:21.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:20:21.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:20:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:20:21.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:21.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: Standby manager daemon z started 2022-09-10T04:20:21.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-10T04:20:21.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:20:21.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-10T04:20:21.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/3116306236' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:20:21.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:21 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.116 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:21 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:21.816+0000 7f0659f80080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:20:22.409 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:22.115+0000 7f0659f80080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:20:22.410 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:22.299+0000 7f0659f80080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:20:22.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:22.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:22.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: mgrmap e14: x(active, since 71s), standbys: z 2022-09-10T04:20:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-10T04:20:22.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.414 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-mon[97044]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: mgrmap e14: x(active, since 71s), standbys: z 2022-09-10T04:20:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:22.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-10T04:20:22.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:22 smithi081 ceph-mon[90133]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:22.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: mgrmap e14: x(active, since 71s), standbys: z 2022-09-10T04:20:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-10T04:20:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:22 smithi110 ceph-mon[97633]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:22.796 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:22.408+0000 7f0659f80080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:20:22.796 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:22 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:22.616+0000 7f0659f80080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:20:23.546 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:20:23 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:20:23.232+0000 7f0659f80080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:20:24.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:24.068+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mon.2 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7fcc1002b440 con 0x7fcc1c100f40 2022-09-10T04:20:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: Standby manager daemon y started 2022-09-10T04:20:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-10T04:20:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:20:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-10T04:20:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:20:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/913130110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]: dispatch 2022-09-10T04:20:24.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]: dispatch 2022-09-10T04:20:24.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]': finished 2022-09-10T04:20:24.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: osdmap e5: 1 total, 0 up, 1 in 2022-09-10T04:20:24.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: Standby manager daemon y started 2022-09-10T04:20:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-10T04:20:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:20:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-10T04:20:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:20:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/913130110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]: dispatch 2022-09-10T04:20:24.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]: dispatch 2022-09-10T04:20:24.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]': finished 2022-09-10T04:20:24.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: osdmap e5: 1 total, 0 up, 1 in 2022-09-10T04:20:24.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:24.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: Standby manager daemon y started 2022-09-10T04:20:24.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-10T04:20:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:20:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-10T04:20:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/3905654302' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:20:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:24.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/913130110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]: dispatch 2022-09-10T04:20:24.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]: dispatch 2022-09-10T04:20:24.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06d36b08-d01a-4f7b-87d2-3dd153d2c300"}]': finished 2022-09-10T04:20:24.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: osdmap e5: 1 total, 0 up, 1 in 2022-09-10T04:20:24.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:25 smithi110 ceph-mon[97633]: mgrmap e15: x(active, since 74s), standbys: z, y 2022-09-10T04:20:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-10T04:20:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:25 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2488322803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:20:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:25 smithi104 ceph-mon[97044]: mgrmap e15: x(active, since 74s), standbys: z, y 2022-09-10T04:20:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-10T04:20:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:25 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2488322803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:20:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:25 smithi081 ceph-mon[90133]: mgrmap e15: x(active, since 74s), standbys: z, y 2022-09-10T04:20:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-10T04:20:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:25 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2488322803' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:20:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:26 smithi110 ceph-mon[97633]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:26 smithi104 ceph-mon[97044]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:26 smithi081 ceph-mon[90133]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:28.620 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:28 smithi081 ceph-mon[90133]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:28.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:28 smithi110 ceph-mon[97633]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:28 smithi104 ceph-mon[97044]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:30 smithi081 ceph-mon[90133]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:30 smithi110 ceph-mon[97633]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:30 smithi104 ceph-mon[97044]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:32.485 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:32 smithi081 ceph-mon[90133]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:32.485 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-09-10T04:20:32.486 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:32 smithi110 ceph-mon[97633]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-09-10T04:20:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:32 smithi104 ceph-mon[97044]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2022-09-10T04:20:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:33 smithi110 ceph-mon[97633]: Deploying daemon osd.0 on smithi081 2022-09-10T04:20:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:33 smithi081 ceph-mon[90133]: Deploying daemon osd.0 on smithi081 2022-09-10T04:20:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:33 smithi104 ceph-mon[97044]: Deploying daemon osd.0 on smithi081 2022-09-10T04:20:34.425 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:34 smithi081 ceph-mon[90133]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:34.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:34 smithi110 ceph-mon[97633]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:34 smithi104 ceph-mon[97044]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:36.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:36 smithi081 ceph-mon[90133]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:36 smithi110 ceph-mon[97633]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:36 smithi104 ceph-mon[97044]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:38 smithi110 ceph-mon[97633]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:38 smithi104 ceph-mon[97044]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:38.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:38 smithi081 ceph-mon[90133]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:40 smithi110 ceph-mon[97633]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:40.744 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:40 smithi081 ceph-mon[90133]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:40 smithi104 ceph-mon[97044]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:42 smithi110 ceph-mon[97633]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:42.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:42.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:42.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:42 smithi104 ceph-mon[97044]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:42.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:42 smithi081 ceph-mon[90133]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:42.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:42.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:42.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:44.129 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:44 smithi081 ceph-mon[90133]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:44 smithi110 ceph-mon[97633]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:44 smithi104 ceph-mon[97044]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:46.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:46 smithi110 ceph-mon[97633]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:46 smithi110 ceph-mon[97633]: from='osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-10T04:20:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:46 smithi110 ceph-mon[97633]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-10T04:20:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:46 smithi104 ceph-mon[97044]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:46 smithi104 ceph-mon[97044]: from='osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-10T04:20:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:46 smithi104 ceph-mon[97044]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-10T04:20:47.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:46 smithi081 ceph-mon[90133]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:46 smithi081 ceph-mon[90133]: from='osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-10T04:20:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:46 smithi081 ceph-mon[90133]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2022-09-10T04:20:48.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:47 smithi081 ceph-mon[90133]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-09-10T04:20:48.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:47 smithi081 ceph-mon[90133]: osdmap e6: 1 total, 0 up, 1 in 2022-09-10T04:20:48.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:48.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:47 smithi081 ceph-mon[90133]: from='osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:20:48.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:47 smithi081 ceph-mon[90133]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:20:48.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:47 smithi081 ceph-mon[90133]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:47 smithi104 ceph-mon[97044]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-09-10T04:20:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:47 smithi104 ceph-mon[97044]: osdmap e6: 1 total, 0 up, 1 in 2022-09-10T04:20:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:47 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:47 smithi104 ceph-mon[97044]: from='osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:20:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:47 smithi104 ceph-mon[97044]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:20:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:47 smithi104 ceph-mon[97044]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:47 smithi110 ceph-mon[97633]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2022-09-10T04:20:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:47 smithi110 ceph-mon[97633]: osdmap e6: 1 total, 0 up, 1 in 2022-09-10T04:20:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:47 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:47 smithi110 ceph-mon[97633]: from='osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:20:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:47 smithi110 ceph-mon[97633]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:20:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:47 smithi110 ceph-mon[97633]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:48.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.608+0000 7fcc197fa700 1 -- 172.21.15.81:0/2272181841 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcc1c105740 con 0x7fcc04035b00 2022-09-10T04:20:48.610 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 0 on host 'smithi081' 2022-09-10T04:20:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcc04035b00 msgr2=0x7fcc04037fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:48.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcc04035b00 0x7fcc04037fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c197150 tx=0x7fcc0c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:20:48.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 msgr2=0x7fcc1c1961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:48.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1961c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000c00 tx=0x7fcc1000f690 comp rx=0 tx=0).stop 2022-09-10T04:20:48.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 shutdown_connections 2022-09-10T04:20:48.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc1c103a30 0x7fcc1c19d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcc04035b00 0x7fcc04037fb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc1c0673d0 0x7fcc1c195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 --2- 172.21.15.81:0/2272181841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc1c100f40 0x7fcc1c1961c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 >> 172.21.15.81:0/2272181841 conn(0x7fcc1c0f8430 msgr2=0x7fcc1c0fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:48.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 shutdown_connections 2022-09-10T04:20:48.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:20:48.612+0000 7fcc22a1b700 1 -- 172.21.15.81:0/2272181841 wait complete. 2022-09-10T04:20:48.979 DEBUG:teuthology.orchestra.run.smithi081:osd.0> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.0.service 2022-09-10T04:20:48.984 INFO:tasks.cephadm:Deploying osd.1 on smithi104 with /dev/vg_nvme/lv_4... 2022-09-10T04:20:48.985 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-09-10T04:20:49.072 INFO:journalctl@ceph.osd.0.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:20:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:20:49.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: osdmap e7: 1 total, 0 up, 1 in 2022-09-10T04:20:49.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:49.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:49.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:49.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:49.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:49.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:49.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:49.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:20:49.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: osdmap e7: 1 total, 0 up, 1 in 2022-09-10T04:20:49.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:49.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:49.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:20:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: osdmap e7: 1 total, 0 up, 1 in 2022-09-10T04:20:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:20:49.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:49.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:20:49.578 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:20:50.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:20:50.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:20:50.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:50.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] boot 2022-09-10T04:20:50.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: osdmap e8: 1 total, 1 up, 1 in 2022-09-10T04:20:50.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:50.108 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:49 smithi104 ceph-mon[97044]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:50.183 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:20:50.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:20:50.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:50.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] boot 2022-09-10T04:20:50.184 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: osdmap e8: 1 total, 1 up, 1 in 2022-09-10T04:20:50.185 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:50.185 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:49 smithi081 ceph-mon[90133]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:20:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:20:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: osd.0 [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] boot 2022-09-10T04:20:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: osdmap e8: 1 total, 1 up, 1 in 2022-09-10T04:20:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:20:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:49 smithi110 ceph-mon[97633]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2022-09-10T04:20:51.182 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-09-10T04:20:51.182 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-09-10T04:20:51.183 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2022-09-10T04:20:51.183 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-09-10T04:20:51.183 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111895 s, 937 MB/s 2022-09-10T04:20:51.183 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2022-09-10T04:20:51.217 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:50 smithi104 ceph-mon[97044]: osdmap e9: 1 total, 1 up, 1 in 2022-09-10T04:20:51.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:50 smithi081 ceph-mon[90133]: osdmap e9: 1 total, 1 up, 1 in 2022-09-10T04:20:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:50 smithi110 ceph-mon[97633]: osdmap e9: 1 total, 1 up, 1 in 2022-09-10T04:20:51.587 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2022-09-10T04:20:52.179 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:20:52.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:51 smithi081 ceph-mon[90133]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:51 smithi104 ceph-mon[97044]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:52.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:51 smithi110 ceph-mon[97633]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:53.436 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 -- 172.21.15.104:0/3367652343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fdfe0 msgr2=0x7fcfbc0fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/3367652343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fdfe0 0x7fcfbc0fe430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcfac009810 tx=0x7fcfac009b20 comp rx=0 tx=0).stop 2022-09-10T04:20:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 -- 172.21.15.104:0/3367652343 shutdown_connections 2022-09-10T04:20:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/3367652343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc105230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:53.437 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/3367652343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfbc0fcd60 0x7fcfbc0fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/3367652343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fdfe0 0x7fcfbc0fe430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 -- 172.21.15.104:0/3367652343 >> 172.21.15.104:0/3367652343 conn(0x7fcfbc0f8470 msgr2=0x7fcfbc0fa880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:20:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 -- 172.21.15.104:0/3367652343 shutdown_connections 2022-09-10T04:20:53.438 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.434+0000 7fcfc15e8700 1 -- 172.21.15.104:0/3367652343 wait complete. 2022-09-10T04:20:53.439 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 Processor -- start 2022-09-10T04:20:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 -- start start 2022-09-10T04:20:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfbc0fcd60 0x7fcfbc1929f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:53.442 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfbc0fdfe0 0x7fcfbc192f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc19a570 con 0x7fcfbc0fdfe0 2022-09-10T04:20:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc19a6e0 con 0x7fcfbc0fe970 2022-09-10T04:20:53.443 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfc15e8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfbc19a8a0 con 0x7fcfbc0fcd60 2022-09-10T04:20:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfbb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfbb7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58880/0 (socket says 172.21.15.104:58880) 2022-09-10T04:20:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.435+0000 7fcfbb7fe700 1 -- 172.21.15.104:0/873075776 learned_addr learned my addr 172.21.15.104:0/873075776 (peer_addr_for_me v2:172.21.15.104:0/0) 2022-09-10T04:20:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbaffd700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfbc0fcd60 0x7fcfbc1929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:53.444 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfba7fc700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfbc0fdfe0 0x7fcfbc192f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbb7fe700 1 -- 172.21.15.104:0/873075776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfbc0fcd60 msgr2=0x7fcfbc1929f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbb7fe700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfbc0fcd60 0x7fcfbc1929f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbb7fe700 1 -- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfbc0fdfe0 msgr2=0x7fcfbc192f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:20:53.445 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbb7fe700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfbc0fdfe0 0x7fcfbc192f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:20:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbb7fe700 1 -- 172.21.15.104:0/873075776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfac009470 con 0x7fcfbc0fe970 2022-09-10T04:20:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfbb7fe700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc199fa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400b9e0 tx=0x7fcfb400bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb400c9b0 con 0x7fcfbc0fe970 2022-09-10T04:20:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcfb4014440 con 0x7fcfbc0fe970 2022-09-10T04:20:53.446 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfbc19ab80 con 0x7fcfbc0fe970 2022-09-10T04:20:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb40125a0 con 0x7fcfbc0fe970 2022-09-10T04:20:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.436+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfbc19b040 con 0x7fcfbc0fe970 2022-09-10T04:20:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.437+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfbc108560 con 0x7fcfbc0fe970 2022-09-10T04:20:53.447 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.438+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7fcfb401b030 con 0x7fcfbc0fe970 2022-09-10T04:20:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.438+0000 7fcfa3fff700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfa4098fc0 0x7fcfa409b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:20:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.438+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1763+0+0 (secure 0 0 0) 0x7fcfb40c0100 con 0x7fcfbc0fe970 2022-09-10T04:20:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.439+0000 7fcfbaffd700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfa4098fc0 0x7fcfa409b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:20:53.448 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.439+0000 7fcfbaffd700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfa4098fc0 0x7fcfa409b470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00057d0 tx=0x7fcfb0005760 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:20:53.449 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.442+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcfb404cc80 con 0x7fcfbc0fe970 2022-09-10T04:20:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi081 2022-09-10T04:20:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:20:53.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi081 to 17271M 2022-09-10T04:20:53.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi081 2022-09-10T04:20:53.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:20:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi081 to 17271M 2022-09-10T04:20:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.636 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:20:53.634+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfbc05fbd0 con 0x7fcfa4098fc0 2022-09-10T04:20:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi081 2022-09-10T04:20:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:20:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi081 to 17271M 2022-09-10T04:20:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:20:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:54 smithi104 ceph-mon[97044]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:54 smithi104 ceph-mon[97044]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:20:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:20:54.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:54.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:54 smithi081 ceph-mon[90133]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:54 smithi081 ceph-mon[90133]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:20:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:20:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:54 smithi110 ceph-mon[97633]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:54 smithi110 ceph-mon[97633]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:20:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:20:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:20:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:20:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:56 smithi104 ceph-mon[97044]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:56 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/3461967199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]: dispatch 2022-09-10T04:20:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:56 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]: dispatch 2022-09-10T04:20:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:56 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]': finished 2022-09-10T04:20:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:56 smithi104 ceph-mon[97044]: osdmap e10: 2 total, 1 up, 2 in 2022-09-10T04:20:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:20:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:56 smithi081 ceph-mon[90133]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:56 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/3461967199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]: dispatch 2022-09-10T04:20:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:56 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]: dispatch 2022-09-10T04:20:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:56 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]': finished 2022-09-10T04:20:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:56 smithi081 ceph-mon[90133]: osdmap e10: 2 total, 1 up, 2 in 2022-09-10T04:20:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:20:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:56 smithi110 ceph-mon[97633]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:56 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/3461967199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]: dispatch 2022-09-10T04:20:56.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:56 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]: dispatch 2022-09-10T04:20:56.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:56 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34797803-b77d-4ce2-a459-df89cb7ff968"}]': finished 2022-09-10T04:20:56.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:56 smithi110 ceph-mon[97633]: osdmap e10: 2 total, 1 up, 2 in 2022-09-10T04:20:56.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:20:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:57 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/2955828615' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:20:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:57 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/2955828615' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:20:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:57 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/2955828615' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:20:58.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:20:58 smithi081 ceph-mon[90133]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:58.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:20:58 smithi110 ceph-mon[97633]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:20:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:20:58 smithi104 ceph-mon[97044]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:00 smithi104 ceph-mon[97044]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:00.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:00 smithi081 ceph-mon[90133]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:00.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:00 smithi110 ceph-mon[97633]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:02.503 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:02 smithi104 ceph-mon[97044]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:02 smithi081 ceph-mon[90133]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:02 smithi110 ceph-mon[97633]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:03 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-09-10T04:21:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:03 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:03 smithi104 ceph-mon[97044]: Deploying daemon osd.1 on smithi104 2022-09-10T04:21:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:03 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-09-10T04:21:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:03 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:03 smithi081 ceph-mon[90133]: Deploying daemon osd.1 on smithi104 2022-09-10T04:21:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:03 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2022-09-10T04:21:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:03 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:03 smithi110 ceph-mon[97633]: Deploying daemon osd.1 on smithi104 2022-09-10T04:21:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:04 smithi081 ceph-mon[90133]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:04.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:04 smithi110 ceph-mon[97633]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:04.707 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:04 smithi104 ceph-mon[97044]: pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:06.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:06 smithi081 ceph-mon[90133]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:06 smithi110 ceph-mon[97633]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:06 smithi104 ceph-mon[97044]: pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:08.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:08 smithi081 ceph-mon[90133]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:08.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:08.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:08.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:08.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:08.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:08 smithi110 ceph-mon[97633]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:08 smithi104 ceph-mon[97044]: pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:10 smithi104 ceph-mon[97044]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:21:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:21:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:10 smithi081 ceph-mon[90133]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:21:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:21:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:10 smithi110 ceph-mon[97633]: pgmap v60: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:21:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:21:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.018 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 1 on host 'smithi104' 2022-09-10T04:21:12.018 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.016+0000 7fcfa3fff700 1 -- 172.21.15.104:0/873075776 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcfbc05fbd0 con 0x7fcfa4098fc0 2022-09-10T04:21:12.020 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfa4098fc0 msgr2=0x7fcfa409b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:12.020 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfa4098fc0 0x7fcfa409b470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00057d0 tx=0x7fcfb0005760 comp rx=0 tx=0).stop 2022-09-10T04:21:12.020 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 msgr2=0x7fcfbc199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:12.020 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc199fa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcfb400b9e0 tx=0x7fcfb400bda0 comp rx=0 tx=0).stop 2022-09-10T04:21:12.021 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 shutdown_connections 2022-09-10T04:21:12.021 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfbc0fe970 0x7fcfbc199fa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:12.021 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfa4098fc0 0x7fcfa409b470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:12.021 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfbc0fcd60 0x7fcfbc1929f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 --2- 172.21.15.104:0/873075776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfbc0fdfe0 0x7fcfbc192f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:12.022 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 >> 172.21.15.104:0/873075776 conn(0x7fcfbc0f8470 msgr2=0x7fcfbc0fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:21:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 shutdown_connections 2022-09-10T04:21:12.023 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:21:12.019+0000 7fcfc15e8700 1 -- 172.21.15.104:0/873075776 wait complete. 2022-09-10T04:21:12.447 DEBUG:teuthology.orchestra.run.smithi104:osd.1> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.1.service 2022-09-10T04:21:12.451 INFO:tasks.cephadm:Deploying osd.2 on smithi110 with /dev/vg_nvme/lv_4... 2022-09-10T04:21:12.451 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2022-09-10T04:21:12.479 INFO:journalctl@ceph.osd.1.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:21:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-10T04:21:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-10T04:21:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:12.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-10T04:21:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-10T04:21:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:12.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:12.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-10T04:21:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2022-09-10T04:21:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:13.045 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:21:13.546 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-1[102265]: 2022-09-10T04:21:13.321+0000 7f6a2e2ce700 -1 osd.1 0 waiting for initial osdmap 2022-09-10T04:21:13.547 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-1[102265]: 2022-09-10T04:21:13.324+0000 7f6a2a0c3700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:21:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-mon[97044]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-09-10T04:21:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-mon[97044]: osdmap e11: 2 total, 1 up, 2 in 2022-09-10T04:21:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-mon[97044]: from='osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:21:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:13 smithi104 ceph-mon[97044]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:21:13.574 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:13 smithi110 ceph-mon[97633]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-09-10T04:21:13.574 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:13 smithi110 ceph-mon[97633]: osdmap e11: 2 total, 1 up, 2 in 2022-09-10T04:21:13.575 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:13 smithi110 ceph-mon[97633]: from='osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:21:13.575 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:13.575 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:13 smithi110 ceph-mon[97633]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:21:13.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:13 smithi081 ceph-mon[90133]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2022-09-10T04:21:13.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:13 smithi081 ceph-mon[90133]: osdmap e11: 2 total, 1 up, 2 in 2022-09-10T04:21:13.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:13 smithi081 ceph-mon[90133]: from='osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:21:13.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:13.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:13 smithi081 ceph-mon[90133]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:21:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:14 smithi104 ceph-mon[97044]: pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:14 smithi104 ceph-mon[97044]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:21:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:14 smithi104 ceph-mon[97044]: osdmap e12: 2 total, 1 up, 2 in 2022-09-10T04:21:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:14.572 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_4 2022-09-10T04:21:14.573 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2022-09-10T04:21:14.573 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2022-09-10T04:21:14.573 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2022-09-10T04:21:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00813829 s, 1.3 GB/s 2022-09-10T04:21:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2022-09-10T04:21:14.611 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:14 smithi110 ceph-mon[97633]: pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:14.612 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:14 smithi110 ceph-mon[97633]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:21:14.612 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:14 smithi110 ceph-mon[97633]: osdmap e12: 2 total, 1 up, 2 in 2022-09-10T04:21:14.613 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:14.613 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:14.762 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:14 smithi081 ceph-mon[90133]: pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:14.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:14 smithi081 ceph-mon[90133]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:21:14.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:14 smithi081 ceph-mon[90133]: osdmap e12: 2 total, 1 up, 2 in 2022-09-10T04:21:14.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:14.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:14.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi110:vg_nvme/lv_4 2022-09-10T04:21:15.558 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:21:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:21:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:21:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] boot 2022-09-10T04:21:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: osdmap e13: 2 total, 2 up, 2 in 2022-09-10T04:21:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi104 2022-09-10T04:21:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:21:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi104 to 17168M 2022-09-10T04:21:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:21:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:21:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] boot 2022-09-10T04:21:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: osdmap e13: 2 total, 2 up, 2 in 2022-09-10T04:21:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:15.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi104 2022-09-10T04:21:15.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:21:15.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi104 to 17168M 2022-09-10T04:21:15.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:21:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:21:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: osd.1 [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] boot 2022-09-10T04:21:15.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: osdmap e13: 2 total, 2 up, 2 in 2022-09-10T04:21:15.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:21:15.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi104 2022-09-10T04:21:15.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:21:15.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi104 to 17168M 2022-09-10T04:21:15.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:15.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:16.713 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.709+0000 7f973ca82700 1 -- 172.21.15.110:0/4227587232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fd9d0 msgr2=0x7f9738105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:16.714 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.709+0000 7f973ca82700 1 --2- 172.21.15.110:0/4227587232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fd9d0 0x7f9738105280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9730009bb0 tx=0x7f9730009ec0 comp rx=0 tx=0).stop 2022-09-10T04:21:16.714 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 -- 172.21.15.110:0/4227587232 shutdown_connections 2022-09-10T04:21:16.714 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 --2- 172.21.15.110:0/4227587232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fd9d0 0x7f9738105280 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:16.715 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 --2- 172.21.15.110:0/4227587232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97380fe3d0 0x7f97380fe7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:16.715 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 --2- 172.21.15.110:0/4227587232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 0x7f97380fd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:16.715 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 -- 172.21.15.110:0/4227587232 >> 172.21.15.110:0/4227587232 conn(0x7f97380f7c00 msgr2=0x7f97380fa010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:21:16.715 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 -- 172.21.15.110:0/4227587232 shutdown_connections 2022-09-10T04:21:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 -- 172.21.15.110:0/4227587232 wait complete. 2022-09-10T04:21:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 Processor -- start 2022-09-10T04:21:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.710+0000 7f973ca82700 1 -- start start 2022-09-10T04:21:16.716 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 0x7f97380fcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97380fd9d0 0x7f97381986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 0x7f973819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973819b540 con 0x7f97380fd9d0 2022-09-10T04:21:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973819b6b0 con 0x7f97380fd020 2022-09-10T04:21:16.717 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973819b870 con 0x7f97380fe3d0 2022-09-10T04:21:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 0x7f973819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 0x7f973819ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60628/0 (socket says 172.21.15.110:60628) 2022-09-10T04:21:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 -- 172.21.15.110:0/2740517109 learned_addr learned my addr 172.21.15.110:0/2740517109 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:21:16.718 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973659c700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 0x7f97380fcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:16.719 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 -- 172.21.15.110:0/2740517109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 msgr2=0x7f97380fcaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:16.719 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 0x7f97380fcaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:16.719 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 -- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97380fd9d0 msgr2=0x7f97381986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:16.719 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97380fd9d0 0x7f97381986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:16.720 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 -- 172.21.15.110:0/2740517109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97300097e0 con 0x7f97380fe3d0 2022-09-10T04:21:16.720 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973659c700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 0x7f97380fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:21:16.720 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f9736d9d700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 0x7f973819ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9730011040 tx=0x7f973000bbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:21:16.720 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973001c070 con 0x7f97380fe3d0 2022-09-10T04:21:16.721 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9730005070 con 0x7f97380fe3d0 2022-09-10T04:21:16.721 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973819baf0 con 0x7f97380fe3d0 2022-09-10T04:21:16.721 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730017740 con 0x7f97380fe3d0 2022-09-10T04:21:16.721 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.711+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f973819bf50 con 0x7f97380fe3d0 2022-09-10T04:21:16.721 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.712+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9738108530 con 0x7f97380fe3d0 2022-09-10T04:21:16.723 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.714+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7f9730027030 con 0x7f97380fe3d0 2022-09-10T04:21:16.723 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.714+0000 7f97277fe700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9720099040 0x7f972009b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:16.724 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.715+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2305+0+0 (secure 0 0 0) 0x7f97300c2b00 con 0x7f97380fe3d0 2022-09-10T04:21:16.724 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.715+0000 7f973659c700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9720099040 0x7f972009b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:16.724 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.715+0000 7f973659c700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9720099040 0x7f972009b4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f972c006fd0 tx=0x7f972c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:21:16.724 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.718+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f9730090910 con 0x7f97380fe3d0 2022-09-10T04:21:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:16 smithi110 ceph-mon[97633]: pgmap v66: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:16 smithi110 ceph-mon[97633]: osdmap e14: 2 total, 2 up, 2 in 2022-09-10T04:21:16.908 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:16.907+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f973805fbb0 con 0x7f9720099040 2022-09-10T04:21:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:16 smithi104 ceph-mon[97044]: pgmap v66: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:16 smithi104 ceph-mon[97044]: osdmap e14: 2 total, 2 up, 2 in 2022-09-10T04:21:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:16 smithi081 ceph-mon[90133]: pgmap v66: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2022-09-10T04:21:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:16 smithi081 ceph-mon[90133]: osdmap e14: 2 total, 2 up, 2 in 2022-09-10T04:21:18.021 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:21:18.021 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:21:18.021 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:21:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:21:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:21:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:21:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:18 smithi110 ceph-mon[97633]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:21:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:18 smithi110 ceph-mon[97633]: pgmap v68: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:18 smithi104 ceph-mon[97044]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:21:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:18 smithi104 ceph-mon[97044]: pgmap v68: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:19.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:18 smithi081 ceph-mon[90133]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:21:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:18 smithi081 ceph-mon[90133]: pgmap v68: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:19 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/4215922488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]: dispatch 2022-09-10T04:21:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:19 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]: dispatch 2022-09-10T04:21:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:19 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]': finished 2022-09-10T04:21:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:19 smithi104 ceph-mon[97044]: osdmap e15: 3 total, 2 up, 3 in 2022-09-10T04:21:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:19 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:20.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:19 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/4215922488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]: dispatch 2022-09-10T04:21:20.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:19 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]: dispatch 2022-09-10T04:21:20.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:19 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]': finished 2022-09-10T04:21:20.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:19 smithi081 ceph-mon[90133]: osdmap e15: 3 total, 2 up, 3 in 2022-09-10T04:21:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:19 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:19 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/4215922488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]: dispatch 2022-09-10T04:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:19 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]: dispatch 2022-09-10T04:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:19 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e47fb11-0460-4c91-8284-c8854bcfbfba"}]': finished 2022-09-10T04:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:19 smithi110 ceph-mon[97633]: osdmap e15: 3 total, 2 up, 3 in 2022-09-10T04:21:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:19 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:20 smithi104 ceph-mon[97044]: pgmap v70: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:20 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/2617010284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:21:21.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:20 smithi081 ceph-mon[90133]: pgmap v70: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:21.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:20 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/2617010284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:21:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:20 smithi110 ceph-mon[97633]: pgmap v70: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:20 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/2617010284' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:21:23.012 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:22 smithi110 ceph-mon[97633]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:22 smithi104 ceph-mon[97044]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:23.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:22 smithi081 ceph-mon[90133]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:24.347 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:24 smithi110 ceph-mon[97633]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:24.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:24 smithi104 ceph-mon[97044]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:24.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:24 smithi081 ceph-mon[90133]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:26.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:26 smithi110 ceph-mon[97633]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:26.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:26 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-09-10T04:21:26.607 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:26 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:26.608 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:26 smithi110 ceph-mon[97633]: Deploying daemon osd.2 on smithi110 2022-09-10T04:21:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:26 smithi104 ceph-mon[97044]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:26 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-09-10T04:21:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:26 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:26 smithi104 ceph-mon[97044]: Deploying daemon osd.2 on smithi110 2022-09-10T04:21:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:26 smithi081 ceph-mon[90133]: pgmap v73: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:26 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2022-09-10T04:21:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:26 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:26 smithi081 ceph-mon[90133]: Deploying daemon osd.2 on smithi110 2022-09-10T04:21:28.617 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:28 smithi110 ceph-mon[97633]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:28 smithi104 ceph-mon[97044]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:28.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:28 smithi081 ceph-mon[90133]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:30.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:30 smithi110 ceph-mon[97633]: pgmap v75: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:30 smithi104 ceph-mon[97044]: pgmap v75: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:30.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:30 smithi081 ceph-mon[90133]: pgmap v75: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:32.361 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:32.361 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:32.361 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:32.362 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:32.362 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:32.362 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:32 smithi110 ceph-mon[97633]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:32.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:32 smithi104 ceph-mon[97044]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:32.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:32.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:32 smithi081 ceph-mon[90133]: pgmap v76: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:34.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:34 smithi110 ceph-mon[97633]: pgmap v77: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:34 smithi104 ceph-mon[97044]: pgmap v77: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:34 smithi081 ceph-mon[90133]: pgmap v77: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.183 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.181+0000 7f97277fe700 1 -- 172.21.15.110:0/2740517109 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f973805fbb0 con 0x7f9720099040 2022-09-10T04:21:35.184 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 2 on host 'smithi110' 2022-09-10T04:21:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9720099040 msgr2=0x7f972009b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:35.185 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9720099040 0x7f972009b4f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f972c006fd0 tx=0x7f972c008040 comp rx=0 tx=0).stop 2022-09-10T04:21:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 msgr2=0x7f973819ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 0x7f973819ad70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9730011040 tx=0x7f973000bbd0 comp rx=0 tx=0).stop 2022-09-10T04:21:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 shutdown_connections 2022-09-10T04:21:35.186 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9720099040 0x7f972009b4f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97380fe3d0 0x7f973819ad70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97380fd020 0x7f97380fcaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 --2- 172.21.15.110:0/2740517109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97380fd9d0 0x7f97381986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:35.187 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 >> 172.21.15.110:0/2740517109 conn(0x7f97380f7c00 msgr2=0x7f97380faea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:21:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 shutdown_connections 2022-09-10T04:21:35.188 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:21:35.183+0000 7f973ca82700 1 -- 172.21.15.110:0/2740517109 wait complete. 2022-09-10T04:21:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-10T04:21:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-10T04:21:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:35.606 DEBUG:teuthology.orchestra.run.smithi110:osd.2> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.2.service 2022-09-10T04:21:35.609 INFO:tasks.cephadm:Deploying osd.3 on smithi081 with /dev/vg_nvme/lv_3... 2022-09-10T04:21:35.610 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-09-10T04:21:35.636 INFO:journalctl@ceph.osd.2.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:21:35.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-10T04:21:35.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-10T04:21:35.694 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:35.694 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.694 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:35.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-10T04:21:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2022-09-10T04:21:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:21:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:21:36.479 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:21:36.655 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-2[103510]: 2022-09-10T04:21:36.375+0000 7fe5a95fb700 -1 osd.2 0 waiting for initial osdmap 2022-09-10T04:21:36.656 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-2[103510]: 2022-09-10T04:21:36.380+0000 7fe5a23ea700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:21:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-mon[97633]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-mon[97633]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-09-10T04:21:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-mon[97633]: osdmap e16: 3 total, 2 up, 3 in 2022-09-10T04:21:36.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:36.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-mon[97633]: from='osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:21:36.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:36 smithi110 ceph-mon[97633]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:21:36.713 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:36 smithi081 ceph-mon[90133]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:36.713 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:36 smithi081 ceph-mon[90133]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-09-10T04:21:36.714 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:36 smithi081 ceph-mon[90133]: osdmap e16: 3 total, 2 up, 3 in 2022-09-10T04:21:36.715 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:36.715 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:36 smithi081 ceph-mon[90133]: from='osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:21:36.715 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:36 smithi081 ceph-mon[90133]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:21:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:36 smithi104 ceph-mon[97044]: pgmap v78: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:36 smithi104 ceph-mon[97044]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2022-09-10T04:21:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:36 smithi104 ceph-mon[97044]: osdmap e16: 3 total, 2 up, 3 in 2022-09-10T04:21:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:36 smithi104 ceph-mon[97044]: from='osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:21:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:36 smithi104 ceph-mon[97044]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:21:37.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:37 smithi081 ceph-mon[90133]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:21:37.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:37 smithi081 ceph-mon[90133]: osdmap e17: 3 total, 2 up, 3 in 2022-09-10T04:21:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:37 smithi110 ceph-mon[97633]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:21:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:37 smithi110 ceph-mon[97633]: osdmap e17: 3 total, 2 up, 3 in 2022-09-10T04:21:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:37 smithi104 ceph-mon[97044]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:21:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:37 smithi104 ceph-mon[97044]: osdmap e17: 3 total, 2 up, 3 in 2022-09-10T04:21:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:38.543 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-09-10T04:21:38.543 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-09-10T04:21:38.543 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2022-09-10T04:21:38.543 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-09-10T04:21:38.544 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0460971 s, 227 MB/s 2022-09-10T04:21:38.544 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2022-09-10T04:21:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:21:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:21:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] boot 2022-09-10T04:21:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: osdmap e18: 3 total, 3 up, 3 in 2022-09-10T04:21:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi110 2022-09-10T04:21:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:21:38.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi110 to 17168M 2022-09-10T04:21:38.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:38 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.687 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:21:38.688 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:21:38.688 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:38.688 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] boot 2022-09-10T04:21:38.688 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: osdmap e18: 3 total, 3 up, 3 in 2022-09-10T04:21:38.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:38.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi110 2022-09-10T04:21:38.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.690 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.690 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:21:38.690 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi110 to 17168M 2022-09-10T04:21:38.690 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.691 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:38 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:21:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:21:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2022-09-10T04:21:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: osd.2 [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] boot 2022-09-10T04:21:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: osdmap e18: 3 total, 3 up, 3 in 2022-09-10T04:21:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:21:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi110 2022-09-10T04:21:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:21:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi110 to 17168M 2022-09-10T04:21:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:38 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:38.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi081:vg_nvme/lv_3 2022-09-10T04:21:39.822 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:21:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:40 smithi110 ceph-mon[97633]: osdmap e19: 3 total, 3 up, 3 in 2022-09-10T04:21:40.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:40 smithi081 ceph-mon[90133]: osdmap e19: 3 total, 3 up, 3 in 2022-09-10T04:21:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:40 smithi104 ceph-mon[97044]: osdmap e19: 3 total, 3 up, 3 in 2022-09-10T04:21:41.298 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 -- 172.21.15.81:0/1188045121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c067c20 msgr2=0x7fe59c100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:41.298 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/1188045121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c067c20 0x7fe59c100820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe58c009750 tx=0x7fe58c009a60 comp rx=0 tx=0).stop 2022-09-10T04:21:41.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 -- 172.21.15.81:0/1188045121 shutdown_connections 2022-09-10T04:21:41.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/1188045121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe59c1038c0 0x7fe59c105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:41.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/1188045121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c067c20 0x7fe59c100820 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:41.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/1188045121 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c100df0 0x7fe59c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:41.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 -- 172.21.15.81:0/1188045121 >> 172.21.15.81:0/1188045121 conn(0x7fe59c0f8210 msgr2=0x7fe59c0fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:21:41.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 -- 172.21.15.81:0/1188045121 shutdown_connections 2022-09-10T04:21:41.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.297+0000 7fe5a0e99700 1 -- 172.21.15.81:0/1188045121 wait complete. 2022-09-10T04:21:41.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.298+0000 7fe5a0e99700 1 Processor -- start 2022-09-10T04:21:41.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.298+0000 7fe5a0e99700 1 -- start start 2022-09-10T04:21:41.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe5a0e99700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 0x7fe59c195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:41.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe5a0e99700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe59c100df0 0x7fe59c196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:41.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe5a0e99700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c1038c0 0x7fe59c19d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:41.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe5a0e99700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c19d650 con 0x7fe59c100df0 2022-09-10T04:21:41.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe5a0e99700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c19d7c0 con 0x7fe59c067c20 2022-09-10T04:21:41.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe5a0e99700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59c19d980 con 0x7fe59c1038c0 2022-09-10T04:21:41.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 0x7fe59c195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:41.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 0x7fe59c195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:50712/0 (socket says 172.21.15.81:50712) 2022-09-10T04:21:41.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 -- 172.21.15.81:0/4035957177 learned_addr learned my addr 172.21.15.81:0/4035957177 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:21:41.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59affd700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe59c100df0 0x7fe59c196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:41.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59bfff700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c1038c0 0x7fe59c19d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:41.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 -- 172.21.15.81:0/4035957177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c1038c0 msgr2=0x7fe59c19d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:41.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c1038c0 0x7fe59c19d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:41.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 -- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe59c100df0 msgr2=0x7fe59c196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:21:41.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.299+0000 7fe59b7fe700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe59c100df0 0x7fe59c196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:21:41.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe59b7fe700 1 -- 172.21.15.81:0/4035957177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940097e0 con 0x7fe59c067c20 2022-09-10T04:21:41.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe59bfff700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c1038c0 0x7fe59c19d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:21:41.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe59b7fe700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 0x7fe59c195ad0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe58c004230 tx=0x7fe58c004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:21:41.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c01d070 con 0x7fe59c067c20 2022-09-10T04:21:41.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe58c003bb0 con 0x7fe59c067c20 2022-09-10T04:21:41.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c0093b0 con 0x7fe59c067c20 2022-09-10T04:21:41.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe58c017b90 con 0x7fe59c067c20 2022-09-10T04:21:41.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.300+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59c19dec0 con 0x7fe59c067c20 2022-09-10T04:21:41.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.301+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59c109420 con 0x7fe59c067c20 2022-09-10T04:21:41.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.303+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 15) v1 ==== 127638+0+0 (secure 0 0 0) 0x7fe58c00c040 con 0x7fe59c067c20 2022-09-10T04:21:41.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:41 smithi081 ceph-mon[90133]: pgmap v84: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:41.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:41 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-09-10T04:21:41.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:41 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-09-10T04:21:41.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:41 smithi081 ceph-mon[90133]: osdmap e20: 3 total, 3 up, 3 in 2022-09-10T04:21:41.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:41 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-09-10T04:21:41.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.304+0000 7fe598ff9700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe584098fc0 0x7fe58409b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:21:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.304+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3233+0+0 (secure 0 0 0) 0x7fe58c0c1e00 con 0x7fe59c067c20 2022-09-10T04:21:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.305+0000 7fe59affd700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe584098fc0 0x7fe58409b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:21:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.305+0000 7fe59affd700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe584098fc0 0x7fe58409b470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0679d0 tx=0x7fe590005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:21:41.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.308+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe58c08f870 con 0x7fe59c067c20 2022-09-10T04:21:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:41 smithi110 ceph-mon[97633]: pgmap v84: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:41 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-09-10T04:21:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:41 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-09-10T04:21:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:41 smithi110 ceph-mon[97633]: osdmap e20: 3 total, 3 up, 3 in 2022-09-10T04:21:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:41 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-09-10T04:21:41.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:41.535+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe59c05f120 con 0x7fe584098fc0 2022-09-10T04:21:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:41 smithi104 ceph-mon[97044]: pgmap v84: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:41 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2022-09-10T04:21:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:41 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2022-09-10T04:21:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:41 smithi104 ceph-mon[97044]: osdmap e20: 3 total, 3 up, 3 in 2022-09-10T04:21:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:41 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2022-09-10T04:21:42.143 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:21:42.142+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe58c0c7020 con 0x7fe59c067c20 2022-09-10T04:21:42.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-09-10T04:21:42.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: osdmap e21: 3 total, 3 up, 3 in 2022-09-10T04:21:42.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:42.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:21:42.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:21:42.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:21:42.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:42.306 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:41 smithi081 sudo[105650]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-10T04:21:42.306 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:41 smithi081 sudo[105650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:42.307 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:41 smithi081 sudo[105650]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:42.307 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:42 smithi081 sudo[105650]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-09-10T04:21:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: osdmap e21: 3 total, 3 up, 3 in 2022-09-10T04:21:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:21:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:21:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:21:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2022-09-10T04:21:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: osdmap e21: 3 total, 3 up, 3 in 2022-09-10T04:21:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:21:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:21:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:21:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:43.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:42 smithi081 sudo[105707]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-10T04:21:43.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:42 smithi081 sudo[105707]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:43.054 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:42 smithi081 sudo[105707]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:43.055 INFO:journalctl@ceph.osd.0.smithi081.stdout:Sep 10 04:21:42 smithi081 sudo[105707]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:43.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:43 smithi104 ceph-mon[97044]: osdmap e22: 3 total, 3 up, 3 in 2022-09-10T04:21:43.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:43 smithi104 ceph-mon[97044]: mgrmap e16: x(active, since 2m), standbys: z, y 2022-09-10T04:21:43.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:43.330 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104787]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-10T04:21:43.330 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104787]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:43.330 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104787]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:43.330 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104787]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:43 smithi110 ceph-mon[97633]: osdmap e22: 3 total, 3 up, 3 in 2022-09-10T04:21:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:43 smithi110 ceph-mon[97633]: mgrmap e16: x(active, since 2m), standbys: z, y 2022-09-10T04:21:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:43.437 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:43 smithi081 ceph-mon[90133]: osdmap e22: 3 total, 3 up, 3 in 2022-09-10T04:21:43.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:43 smithi081 ceph-mon[90133]: mgrmap e16: x(active, since 2m), standbys: z, y 2022-09-10T04:21:43.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:43.796 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104790]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-10T04:21:43.796 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104790]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:43.796 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104790]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:43.797 INFO:journalctl@ceph.osd.1.smithi104.stdout:Sep 10 04:21:43 smithi104 sudo[104790]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:43.859 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105848]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-10T04:21:43.860 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:43.860 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105848]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:43.860 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105848]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:44.157 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105851]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-10T04:21:44.157 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105851]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:44.157 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105851]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:44.158 INFO:journalctl@ceph.osd.2.smithi110.stdout:Sep 10 04:21:43 smithi110 sudo[105851]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:45.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:44 smithi081 sudo[105824]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-10T04:21:45.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:44 smithi081 sudo[105824]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:45.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:44 smithi081 sudo[105824]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:45.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:44 smithi081 sudo[105824]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:45 smithi104 ceph-mon[97044]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:45.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 sudo[105848]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-10T04:21:45.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 sudo[105848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:45.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 sudo[105848]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:45.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 sudo[105848]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:45.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:45.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:45.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:45 smithi081 ceph-mon[90133]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:45.888 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105854]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-10T04:21:45.888 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:45.888 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105854]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:45.889 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105854]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105857]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-10T04:21:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105857]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105857]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 sudo[105857]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:21:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:45 smithi110 ceph-mon[97633]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104793]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2022-09-10T04:21:46.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104793]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:46.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104793]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:46.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104793]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4143257532' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]: dispatch 2022-09-10T04:21:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: pgmap v90: 1 pgs: 1 creating+peering; 0 B data, 18 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]: dispatch 2022-09-10T04:21:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]': finished 2022-09-10T04:21:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: osdmap e23: 4 total, 3 up, 4 in 2022-09-10T04:21:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:21:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2018345756' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:21:46.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:46 smithi110 ceph-mon[97633]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104796]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2022-09-10T04:21:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2022-09-10T04:21:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104796]: pam_unix(sudo:session): session opened for user root by (uid=0) 2022-09-10T04:21:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 sudo[104796]: pam_unix(sudo:session): session closed for user root 2022-09-10T04:21:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4143257532' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]: dispatch 2022-09-10T04:21:46.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: pgmap v90: 1 pgs: 1 creating+peering; 0 B data, 18 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:46.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]: dispatch 2022-09-10T04:21:46.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]': finished 2022-09-10T04:21:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: osdmap e23: 4 total, 3 up, 4 in 2022-09-10T04:21:46.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:21:46.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2018345756' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:21:46.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:46 smithi104 ceph-mon[97044]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:46.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4143257532' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]: dispatch 2022-09-10T04:21:46.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: pgmap v90: 1 pgs: 1 creating+peering; 0 B data, 18 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:46.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]: dispatch 2022-09-10T04:21:46.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ada925c5-2eb8-48cf-9a82-bc4c2ff500c2"}]': finished 2022-09-10T04:21:46.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: osdmap e23: 4 total, 3 up, 4 in 2022-09-10T04:21:46.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:21:46.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:46.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2022-09-10T04:21:46.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:21:46.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:21:46.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:21:46.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2018345756' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:21:46.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:46 smithi081 ceph-mon[90133]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2022-09-10T04:21:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:48 smithi104 ceph-mon[97044]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:48 smithi081 ceph-mon[90133]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:48 smithi110 ceph-mon[97633]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:50.685 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:50 smithi081 ceph-mon[90133]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:50 smithi104 ceph-mon[97044]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:50 smithi110 ceph-mon[97633]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:21:52.744 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:52 smithi081 ceph-mon[90133]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 45 KiB/s wr, 4 op/s 2022-09-10T04:21:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:52 smithi104 ceph-mon[97044]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 45 KiB/s wr, 4 op/s 2022-09-10T04:21:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:52 smithi110 ceph-mon[97633]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 45 KiB/s wr, 4 op/s 2022-09-10T04:21:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:54 smithi110 ceph-mon[97633]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 33 KiB/s wr, 3 op/s 2022-09-10T04:21:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-09-10T04:21:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:54 smithi110 ceph-mon[97633]: Deploying daemon osd.3 on smithi081 2022-09-10T04:21:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:54 smithi104 ceph-mon[97044]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 33 KiB/s wr, 3 op/s 2022-09-10T04:21:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-09-10T04:21:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:54 smithi104 ceph-mon[97044]: Deploying daemon osd.3 on smithi081 2022-09-10T04:21:54.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:54 smithi081 ceph-mon[90133]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.3 KiB/s rd, 33 KiB/s wr, 3 op/s 2022-09-10T04:21:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2022-09-10T04:21:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:21:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:54 smithi081 ceph-mon[90133]: Deploying daemon osd.3 on smithi081 2022-09-10T04:21:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:56 smithi110 ceph-mon[97633]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.0 KiB/s rd, 27 KiB/s wr, 2 op/s 2022-09-10T04:21:56.704 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:56 smithi081 ceph-mon[90133]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.0 KiB/s rd, 27 KiB/s wr, 2 op/s 2022-09-10T04:21:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:56 smithi104 ceph-mon[97044]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 1.0 KiB/s rd, 27 KiB/s wr, 2 op/s 2022-09-10T04:21:58.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:21:58 smithi081 ceph-mon[90133]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 862 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:21:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:21:58 smithi110 ceph-mon[97633]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 862 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:21:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:21:58 smithi104 ceph-mon[97044]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 862 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:00 smithi110 ceph-mon[97633]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:00 smithi104 ceph-mon[97044]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:00.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:00 smithi081 ceph-mon[90133]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:02 smithi110 ceph-mon[97633]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:02 smithi104 ceph-mon[97044]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:02 smithi081 ceph-mon[90133]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2022-09-10T04:22:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:04.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:04.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:04 smithi104 ceph-mon[97044]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:04.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:04.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:04 smithi081 ceph-mon[90133]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:04.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:04 smithi110 ceph-mon[97633]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:06 smithi110 ceph-mon[97633]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:06.705 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:06 smithi081 ceph-mon[90133]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:06 smithi104 ceph-mon[97044]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:07.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:07 smithi110 ceph-mon[97633]: from='osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-10T04:22:07.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:07 smithi110 ceph-mon[97633]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-10T04:22:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:07 smithi104 ceph-mon[97044]: from='osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-10T04:22:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:07 smithi104 ceph-mon[97044]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-10T04:22:08.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:08.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:07 smithi081 ceph-mon[90133]: from='osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-10T04:22:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:07 smithi081 ceph-mon[90133]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2022-09-10T04:22:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:08 smithi110 ceph-mon[97633]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:08 smithi110 ceph-mon[97633]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-09-10T04:22:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:08 smithi110 ceph-mon[97633]: osdmap e24: 4 total, 3 up, 4 in 2022-09-10T04:22:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:08 smithi110 ceph-mon[97633]: from='osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:22:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:08 smithi110 ceph-mon[97633]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:08 smithi104 ceph-mon[97044]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:08 smithi104 ceph-mon[97044]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-09-10T04:22:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:08 smithi104 ceph-mon[97044]: osdmap e24: 4 total, 3 up, 4 in 2022-09-10T04:22:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:08 smithi104 ceph-mon[97044]: from='osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:22:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:08 smithi104 ceph-mon[97044]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:22:09.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:08 smithi081 ceph-mon[90133]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 20 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:09.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:08 smithi081 ceph-mon[90133]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2022-09-10T04:22:09.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:08 smithi081 ceph-mon[90133]: osdmap e24: 4 total, 3 up, 4 in 2022-09-10T04:22:09.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:08 smithi081 ceph-mon[90133]: from='osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:22:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:08 smithi081 ceph-mon[90133]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:22:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:09 smithi110 ceph-mon[97633]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:22:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:09 smithi110 ceph-mon[97633]: osdmap e25: 4 total, 3 up, 4 in 2022-09-10T04:22:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:22:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:22:10.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:09 smithi081 ceph-mon[90133]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:22:10.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:09 smithi081 ceph-mon[90133]: osdmap e25: 4 total, 3 up, 4 in 2022-09-10T04:22:10.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:10.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:10.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:22:10.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:22:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:09 smithi104 ceph-mon[97044]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:22:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:09 smithi104 ceph-mon[97044]: osdmap e25: 4 total, 3 up, 4 in 2022-09-10T04:22:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:22:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:22:10.090 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.088+0000 7fe598ff9700 1 -- 172.21.15.81:0/4035957177 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe59c05f120 con 0x7fe584098fc0 2022-09-10T04:22:10.091 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 3 on host 'smithi081' 2022-09-10T04:22:10.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.091+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe584098fc0 msgr2=0x7fe58409b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:10.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe584098fc0 0x7fe58409b470 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe59c0679d0 tx=0x7fe590005ea0 comp rx=0 tx=0).stop 2022-09-10T04:22:10.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 msgr2=0x7fe59c195ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:10.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 0x7fe59c195ad0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe58c004230 tx=0x7fe58c004260 comp rx=0 tx=0).stop 2022-09-10T04:22:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 shutdown_connections 2022-09-10T04:22:10.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe584098fc0 0x7fe58409b470 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe59c1038c0 0x7fe59c19d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:10.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe59c067c20 0x7fe59c195ad0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 --2- 172.21.15.81:0/4035957177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe59c100df0 0x7fe59c196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 >> 172.21.15.81:0/4035957177 conn(0x7fe59c0f8210 msgr2=0x7fe59c104620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:22:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 shutdown_connections 2022-09-10T04:22:10.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:22:10.092+0000 7fe5a0e99700 1 -- 172.21.15.81:0/4035957177 wait complete. 2022-09-10T04:22:10.553 DEBUG:teuthology.orchestra.run.smithi081:osd.3> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.3.service 2022-09-10T04:22:10.561 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_3... 2022-09-10T04:22:10.562 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-09-10T04:22:10.653 INFO:journalctl@ceph.osd.3.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:22:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:22:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:22:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] boot 2022-09-10T04:22:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: osdmap e26: 4 total, 4 up, 4 in 2022-09-10T04:22:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:22:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:22:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] boot 2022-09-10T04:22:11.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: osdmap e26: 4 total, 4 up, 4 in 2022-09-10T04:22:11.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:11.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:11.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:11.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:11.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:11.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:22:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:22:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2022-09-10T04:22:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: osd.3 [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] boot 2022-09-10T04:22:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: osdmap e26: 4 total, 4 up, 4 in 2022-09-10T04:22:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:22:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:11.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:11.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:11.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:11.169 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:22:11.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:11 smithi081 ceph-mon[90133]: osdmap e27: 4 total, 4 up, 4 in 2022-09-10T04:22:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:11 smithi104 ceph-mon[97044]: osdmap e27: 4 total, 4 up, 4 in 2022-09-10T04:22:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:11 smithi110 ceph-mon[97633]: osdmap e27: 4 total, 4 up, 4 in 2022-09-10T04:22:12.658 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-09-10T04:22:12.658 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-09-10T04:22:12.659 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2022-09-10T04:22:12.659 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-09-10T04:22:12.659 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012686 s, 827 MB/s 2022-09-10T04:22:12.659 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2022-09-10T04:22:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:12 smithi081 ceph-mon[90133]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:12 smithi081 ceph-mon[90133]: osdmap e28: 4 total, 4 up, 4 in 2022-09-10T04:22:13.027 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:12 smithi104 ceph-mon[97044]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:13.028 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:12 smithi104 ceph-mon[97044]: osdmap e28: 4 total, 4 up, 4 in 2022-09-10T04:22:13.082 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2022-09-10T04:22:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:12 smithi110 ceph-mon[97633]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:12 smithi110 ceph-mon[97633]: osdmap e28: 4 total, 4 up, 4 in 2022-09-10T04:22:13.674 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:22:13.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:13 smithi104 ceph-mon[97044]: osdmap e29: 4 total, 4 up, 4 in 2022-09-10T04:22:13.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:13 smithi110 ceph-mon[97633]: osdmap e29: 4 total, 4 up, 4 in 2022-09-10T04:22:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:13 smithi081 ceph-mon[90133]: osdmap e29: 4 total, 4 up, 4 in 2022-09-10T04:22:14.859 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 -- 172.21.15.104:0/1157154572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770100fa0 msgr2=0x7f87701033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:14.859 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 --2- 172.21.15.104:0/1157154572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770100fa0 0x7f87701033a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f875c009810 tx=0x7f875c009b20 comp rx=0 tx=0).stop 2022-09-10T04:22:14.860 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 -- 172.21.15.104:0/1157154572 shutdown_connections 2022-09-10T04:22:14.860 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 --2- 172.21.15.104:0/1157154572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8770103a30 0x7f8770105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:14.860 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 --2- 172.21.15.104:0/1157154572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8770067370 0x7f8770100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:14.861 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 --2- 172.21.15.104:0/1157154572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770100fa0 0x7f87701033a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:14.861 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 -- 172.21.15.104:0/1157154572 >> 172.21.15.104:0/1157154572 conn(0x7f87700f8430 msgr2=0x7f87700fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:22:14.861 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 -- 172.21.15.104:0/1157154572 shutdown_connections 2022-09-10T04:22:14.861 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.857+0000 7f877706f700 1 -- 172.21.15.104:0/1157154572 wait complete. 2022-09-10T04:22:14.866 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 Processor -- start 2022-09-10T04:22:14.867 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 -- start start 2022-09-10T04:22:14.867 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8770067370 0x7f8770193b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:14.868 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8770100fa0 0x7f8770194060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:14.868 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 0x7f877019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:14.868 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877019b6a0 con 0x7f8770067370 2022-09-10T04:22:14.869 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877019b810 con 0x7f8770103a30 2022-09-10T04:22:14.869 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877706f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877019b9d0 con 0x7f8770100fa0 2022-09-10T04:22:14.869 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 0x7f877019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:22:14.870 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 0x7f877019b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:58958/0 (socket says 172.21.15.104:58958) 2022-09-10T04:22:14.870 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 -- 172.21.15.104:0/3232724975 learned_addr learned my addr 172.21.15.104:0/3232724975 (peer_addr_for_me v2:172.21.15.104:0/0) 2022-09-10T04:22:14.870 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f876ffff700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8770100fa0 0x7f8770194060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:22:14.871 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 -- 172.21.15.104:0/3232724975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8770100fa0 msgr2=0x7f8770194060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:14.871 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8770100fa0 0x7f8770194060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:14.871 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 -- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8770067370 msgr2=0x7f8770193b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8770067370 0x7f8770193b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 -- 172.21.15.104:0/3232724975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c009470 con 0x7f8770103a30 2022-09-10T04:22:14.872 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.858+0000 7f877560c700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 0x7f877019b0d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f876800e910 tx=0x7f876800ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:22:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.859+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8768004c30 con 0x7f8770103a30 2022-09-10T04:22:14.873 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.859+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f876800ee60 con 0x7f8770103a30 2022-09-10T04:22:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.859+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877019bcb0 con 0x7f8770103a30 2022-09-10T04:22:14.874 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.859+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87680057e0 con 0x7f8770103a30 2022-09-10T04:22:14.875 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.859+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877019c240 con 0x7f8770103a30 2022-09-10T04:22:14.875 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.860+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87701096a0 con 0x7f8770103a30 2022-09-10T04:22:14.876 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.861+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8768021070 con 0x7f8770103a30 2022-09-10T04:22:14.876 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.861+0000 7f876dffb700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8760099090 0x7f876009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:14.877 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.861+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3665+0+0 (secure 0 0 0) 0x7f87680c0f90 con 0x7f8770103a30 2022-09-10T04:22:14.877 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.861+0000 7f8774e0b700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8760099090 0x7f876009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:22:14.877 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.862+0000 7f8774e0b700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8760099090 0x7f876009b540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8770067120 tx=0x7f8764006ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:22:14.878 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:14.865+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f876808e8f0 con 0x7f8770103a30 2022-09-10T04:22:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: osdmap e30: 4 total, 4 up, 4 in 2022-09-10T04:22:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi081 2022-09-10T04:22:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi081 to 8635M 2022-09-10T04:22:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: osdmap e30: 4 total, 4 up, 4 in 2022-09-10T04:22:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi081 2022-09-10T04:22:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi081 to 8635M 2022-09-10T04:22:15.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.057 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:15.054+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f877005faf0 con 0x7f8760099090 2022-09-10T04:22:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: osdmap e30: 4 total, 4 up, 4 in 2022-09-10T04:22:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi081 2022-09-10T04:22:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi081 to 8635M 2022-09-10T04:22:15.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:15.942 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:22:15.943 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:22:15.943 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:22:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:22:16.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:16.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:22:16.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:22:16.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:16 smithi104 ceph-mon[97044]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:22:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:16 smithi104 ceph-mon[97044]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:16 smithi081 ceph-mon[90133]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:22:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:16 smithi081 ceph-mon[90133]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:16 smithi110 ceph-mon[97633]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:22:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:16 smithi110 ceph-mon[97633]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:17 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/3522193638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]: dispatch 2022-09-10T04:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:17 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]: dispatch 2022-09-10T04:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:17 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]': finished 2022-09-10T04:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:17 smithi104 ceph-mon[97044]: osdmap e31: 5 total, 4 up, 5 in 2022-09-10T04:22:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:17 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/3522193638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]: dispatch 2022-09-10T04:22:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:17 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]: dispatch 2022-09-10T04:22:18.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:17 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]': finished 2022-09-10T04:22:18.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:17 smithi081 ceph-mon[90133]: osdmap e31: 5 total, 4 up, 5 in 2022-09-10T04:22:18.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:17 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/3522193638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]: dispatch 2022-09-10T04:22:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:17 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]: dispatch 2022-09-10T04:22:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:17 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73482f9e-093f-408b-96bf-34584c281ea7"}]': finished 2022-09-10T04:22:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:17 smithi110 ceph-mon[97633]: osdmap e31: 5 total, 4 up, 5 in 2022-09-10T04:22:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:18 smithi104 ceph-mon[97044]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:22:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:18 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/1644354141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:22:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:18 smithi081 ceph-mon[90133]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:22:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:18 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/1644354141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:22:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:18 smithi110 ceph-mon[97633]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:22:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:18 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/1644354141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:22:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:20 smithi104 ceph-mon[97044]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-10T04:22:21.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:20 smithi081 ceph-mon[90133]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-10T04:22:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:20 smithi110 ceph-mon[97633]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2022-09-10T04:22:23.041 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:22 smithi104 ceph-mon[97044]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-10T04:22:23.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:22 smithi081 ceph-mon[90133]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-10T04:22:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:22 smithi110 ceph-mon[97633]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-10T04:22:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:24 smithi104 ceph-mon[97044]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-10T04:22:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-09-10T04:22:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:24 smithi110 ceph-mon[97633]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-10T04:22:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-09-10T04:22:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:24.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:24 smithi081 ceph-mon[90133]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2022-09-10T04:22:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2022-09-10T04:22:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:25 smithi104 ceph-mon[97044]: Deploying daemon osd.4 on smithi104 2022-09-10T04:22:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:25 smithi110 ceph-mon[97633]: Deploying daemon osd.4 on smithi104 2022-09-10T04:22:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:25 smithi081 ceph-mon[90133]: Deploying daemon osd.4 on smithi104 2022-09-10T04:22:26.362 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:26 smithi104 ceph-mon[97044]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:22:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:26 smithi110 ceph-mon[97633]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:22:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:26 smithi081 ceph-mon[90133]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:22:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:28 smithi110 ceph-mon[97633]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:28.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:28 smithi104 ceph-mon[97044]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:28 smithi081 ceph-mon[90133]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:30 smithi104 ceph-mon[97044]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:30.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:30 smithi081 ceph-mon[90133]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:30.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:30.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:30 smithi110 ceph-mon[97633]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:32 smithi104 ceph-mon[97044]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:32 smithi081 ceph-mon[90133]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:32 smithi110 ceph-mon[97633]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:33.533 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.531+0000 7f876dffb700 1 -- 172.21.15.104:0/3232724975 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f877005faf0 con 0x7f8760099090 2022-09-10T04:22:33.534 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2022-09-10T04:22:33.536 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8760099090 msgr2=0x7f876009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:33.536 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8760099090 0x7f876009b540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8770067120 tx=0x7f8764006ba0 comp rx=0 tx=0).stop 2022-09-10T04:22:33.537 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 msgr2=0x7f877019b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:33.537 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 0x7f877019b0d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f876800e910 tx=0x7f876800ec20 comp rx=0 tx=0).stop 2022-09-10T04:22:33.537 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 shutdown_connections 2022-09-10T04:22:33.538 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8770103a30 0x7f877019b0d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:33.538 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8770067370 0x7f8770193b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:33.539 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8760099090 0x7f876009b540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:33.539 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 --2- 172.21.15.104:0/3232724975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8770100fa0 0x7f8770194060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:33.540 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.533+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 >> 172.21.15.104:0/3232724975 conn(0x7f87700f8430 msgr2=0x7f87700ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:22:33.540 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.534+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 shutdown_connections 2022-09-10T04:22:33.541 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:22:33.534+0000 7f877706f700 1 -- 172.21.15.104:0/3232724975 wait complete. 2022-09-10T04:22:34.043 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.4.service 2022-09-10T04:22:34.046 INFO:tasks.cephadm:Deploying osd.5 on smithi110 with /dev/vg_nvme/lv_3... 2022-09-10T04:22:34.046 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2022-09-10T04:22:34.073 INFO:journalctl@ceph.osd.4.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:22:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-10T04:22:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-10T04:22:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:34.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:34.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-10T04:22:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-10T04:22:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-10T04:22:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2022-09-10T04:22:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:34.644 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:22:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:35 smithi110 ceph-mon[97633]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-09-10T04:22:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:35 smithi110 ceph-mon[97633]: osdmap e32: 5 total, 4 up, 5 in 2022-09-10T04:22:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:35 smithi110 ceph-mon[97633]: from='osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:22:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:35 smithi110 ceph-mon[97633]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:22:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-mon[97044]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-09-10T04:22:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-mon[97044]: osdmap e32: 5 total, 4 up, 5 in 2022-09-10T04:22:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-mon[97044]: from='osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:22:35.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-mon[97044]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:22:35.548 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-4[108227]: 2022-09-10T04:22:35.115+0000 7fdc179b0700 -1 osd.4 0 waiting for initial osdmap 2022-09-10T04:22:35.548 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 10 04:22:35 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-4[108227]: 2022-09-10T04:22:35.122+0000 7fdc11fa2700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:22:35.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:35 smithi081 ceph-mon[90133]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2022-09-10T04:22:35.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:35 smithi081 ceph-mon[90133]: osdmap e32: 5 total, 4 up, 5 in 2022-09-10T04:22:35.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:35 smithi081 ceph-mon[90133]: from='osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:22:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:35 smithi081 ceph-mon[90133]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:22:36.144 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_3 2022-09-10T04:22:36.144 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2022-09-10T04:22:36.145 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2022-09-10T04:22:36.145 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2022-09-10T04:22:36.145 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00843632 s, 1.2 GB/s 2022-09-10T04:22:36.145 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2022-09-10T04:22:36.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:36 smithi104 ceph-mon[97044]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:22:36.388 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:36 smithi104 ceph-mon[97044]: osdmap e33: 5 total, 4 up, 5 in 2022-09-10T04:22:36.388 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:36.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:36.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:36 smithi104 ceph-mon[97044]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:36 smithi110 ceph-mon[97633]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:22:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:36 smithi110 ceph-mon[97633]: osdmap e33: 5 total, 4 up, 5 in 2022-09-10T04:22:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:36 smithi110 ceph-mon[97633]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:36.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:36 smithi081 ceph-mon[90133]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:22:36.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:36 smithi081 ceph-mon[90133]: osdmap e33: 5 total, 4 up, 5 in 2022-09-10T04:22:36.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:36 smithi081 ceph-mon[90133]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2022-09-10T04:22:36.569 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi110:vg_nvme/lv_3 2022-09-10T04:22:37.155 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:22:37.375 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:22:37.376 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:22:37.376 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] boot 2022-09-10T04:22:37.376 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: osdmap e34: 5 total, 5 up, 5 in 2022-09-10T04:22:37.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:37.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi104 2022-09-10T04:22:37.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:37.378 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:37.379 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi104 to 8584M 2022-09-10T04:22:37.379 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.379 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:22:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:22:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] boot 2022-09-10T04:22:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: osdmap e34: 5 total, 5 up, 5 in 2022-09-10T04:22:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi104 2022-09-10T04:22:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:37.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:37.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi104 to 8584M 2022-09-10T04:22:37.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:22:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:22:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: osd.4 [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] boot 2022-09-10T04:22:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: osdmap e34: 5 total, 5 up, 5 in 2022-09-10T04:22:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:22:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi104 2022-09-10T04:22:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:37.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:22:37.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi104 to 8584M 2022-09-10T04:22:37.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:37.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:38.349 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 -- 172.21.15.110:0/1606934191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fcde0 msgr2=0x7fb8b00fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:38.350 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1606934191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fcde0 0x7fb8b00fd1f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009810 tx=0x7fb8ac009b20 comp rx=0 tx=0).stop 2022-09-10T04:22:38.350 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 -- 172.21.15.110:0/1606934191 shutdown_connections 2022-09-10T04:22:38.351 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1606934191 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b00fe970 0x7fb8b0105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:38.351 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1606934191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fcde0 0x7fb8b00fd1f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:38.351 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1606934191 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b00fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:38.352 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 -- 172.21.15.110:0/1606934191 >> 172.21.15.110:0/1606934191 conn(0x7fb8b00f8430 msgr2=0x7fb8b00fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:22:38.352 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 -- 172.21.15.110:0/1606934191 shutdown_connections 2022-09-10T04:22:38.352 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 -- 172.21.15.110:0/1606934191 wait complete. 2022-09-10T04:22:38.353 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 Processor -- start 2022-09-10T04:22:38.353 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.345+0000 7fb8b8250700 1 -- start start 2022-09-10T04:22:38.354 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8b00fcde0 0x7fb8b0195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:38.354 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b01961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:38.354 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b00fe970 0x7fb8b019d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:38.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b019d830 con 0x7fb8b00fcde0 2022-09-10T04:22:38.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b019d9a0 con 0x7fb8b00fe970 2022-09-10T04:22:38.355 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b019db60 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.356 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b01961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:22:38.356 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b01961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60710/0 (socket says 172.21.15.110:60710) 2022-09-10T04:22:38.356 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 -- 172.21.15.110:0/1820062860 learned_addr learned my addr 172.21.15.110:0/1820062860 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:22:38.357 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b67ed700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b00fe970 0x7fb8b019d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:22:38.357 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 -- 172.21.15.110:0/1820062860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b00fe970 msgr2=0x7fb8b019d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:38.357 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b00fe970 0x7fb8b019d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:38.358 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 -- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8b00fcde0 msgr2=0x7fb8b0195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:38.358 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8b00fcde0 0x7fb8b0195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:38.359 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 -- 172.21.15.110:0/1820062860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac009470 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.359 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b57eb700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b01961c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000b3c0 tx=0x7fb8a000b6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:22:38.359 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0011830 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.360 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb8a000be80 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.360 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b019de40 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.360 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0010de0 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.361 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.346+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b019e340 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.361 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.347+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b0108530 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.363 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.348+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb8a0021070 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.364 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.349+0000 7fb8a6ffd700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb89c099010 0x7fb89c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:22:38.364 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.349+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4097+0+0 (secure 0 0 0) 0x7fb8a00c0c30 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.364 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.349+0000 7fb8b5fec700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb89c099010 0x7fb89c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:22:38.365 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.350+0000 7fb8b5fec700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb89c099010 0x7fb89c09b4c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0196b50 tx=0x7fb8ac000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:22:38.365 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.353+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb8a008e3e0 con 0x7fb8b00fdfc0 2022-09-10T04:22:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:38 smithi110 ceph-mon[97633]: osdmap e35: 5 total, 5 up, 5 in 2022-09-10T04:22:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:38 smithi110 ceph-mon[97633]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:38 smithi104 ceph-mon[97044]: osdmap e35: 5 total, 5 up, 5 in 2022-09-10T04:22:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:38 smithi104 ceph-mon[97044]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:38.547 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:38.545+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8b0066e10 con 0x7fb89c099010 2022-09-10T04:22:38.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:38 smithi081 ceph-mon[90133]: osdmap e35: 5 total, 5 up, 5 in 2022-09-10T04:22:38.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:38 smithi081 ceph-mon[90133]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:39.509 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:39 smithi110 ceph-mon[97633]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:22:39.509 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:22:39.510 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:22:39.510 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:39 smithi104 ceph-mon[97044]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:22:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:22:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:22:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:39 smithi081 ceph-mon[90133]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:22:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:40.256 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:40 smithi110 ceph-mon[97633]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:40 smithi104 ceph-mon[97044]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:40 smithi081 ceph-mon[90133]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:41 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/3351225786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]: dispatch 2022-09-10T04:22:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:41 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]: dispatch 2022-09-10T04:22:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:41 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]': finished 2022-09-10T04:22:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:41 smithi104 ceph-mon[97044]: osdmap e36: 6 total, 5 up, 6 in 2022-09-10T04:22:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:41 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:41.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:41 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/3351225786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]: dispatch 2022-09-10T04:22:41.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:41 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]: dispatch 2022-09-10T04:22:41.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:41 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]': finished 2022-09-10T04:22:41.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:41 smithi081 ceph-mon[90133]: osdmap e36: 6 total, 5 up, 6 in 2022-09-10T04:22:41.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:41 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:41 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/3351225786' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]: dispatch 2022-09-10T04:22:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:41 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]: dispatch 2022-09-10T04:22:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:41 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9dd94ce3-5bdf-4664-97dd-73980655b4a8"}]': finished 2022-09-10T04:22:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:41 smithi110 ceph-mon[97633]: osdmap e36: 6 total, 5 up, 6 in 2022-09-10T04:22:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:41 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:42 smithi104 ceph-mon[97044]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:42 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/1356524693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:22:42.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:42 smithi081 ceph-mon[90133]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:42.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:42 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/1356524693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:22:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:42 smithi110 ceph-mon[97633]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:42 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/1356524693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:22:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:44 smithi110 ceph-mon[97633]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:44 smithi104 ceph-mon[97044]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:44 smithi081 ceph-mon[90133]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:46.534 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:46 smithi110 ceph-mon[97633]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:46 smithi104 ceph-mon[97044]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:46.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:46 smithi081 ceph-mon[90133]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:48.608 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:48 smithi110 ceph-mon[97633]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:48.609 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-09-10T04:22:48.609 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:48 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:48.609 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:48 smithi110 ceph-mon[97633]: Deploying daemon osd.5 on smithi110 2022-09-10T04:22:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:48 smithi104 ceph-mon[97044]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-09-10T04:22:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:48 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:48 smithi104 ceph-mon[97044]: Deploying daemon osd.5 on smithi110 2022-09-10T04:22:48.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:48 smithi081 ceph-mon[90133]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2022-09-10T04:22:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:48 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:48 smithi081 ceph-mon[90133]: Deploying daemon osd.5 on smithi110 2022-09-10T04:22:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:50 smithi110 ceph-mon[97633]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:50 smithi104 ceph-mon[97044]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:50.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:50 smithi081 ceph-mon[90133]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:52 smithi110 ceph-mon[97633]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:52 smithi104 ceph-mon[97044]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:52.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:52 smithi081 ceph-mon[90133]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:54.209 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:54.209 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:54.210 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:54.210 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:54.211 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:54.211 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:54 smithi110 ceph-mon[97633]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:54 smithi104 ceph-mon[97044]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:54.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:54.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:54.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:54 smithi081 ceph-mon[90133]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:56 smithi081 ceph-mon[90133]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.592 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:56 smithi110 ceph-mon[97633]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:56.592 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.593 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.593 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:56 smithi104 ceph-mon[97044]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:56.986 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.985+0000 7fb8a6ffd700 1 -- 172.21.15.110:0/1820062860 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb8b0066e10 con 0x7fb89c099010 2022-09-10T04:22:56.988 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 5 on host 'smithi110' 2022-09-10T04:22:56.989 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb89c099010 msgr2=0x7fb89c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:56.989 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb89c099010 0x7fb89c09b4c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0196b50 tx=0x7fb8ac000bc0 comp rx=0 tx=0).stop 2022-09-10T04:22:56.989 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 msgr2=0x7fb8b01961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:22:56.990 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b01961c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000b3c0 tx=0x7fb8a000b6d0 comp rx=0 tx=0).stop 2022-09-10T04:22:56.990 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 shutdown_connections 2022-09-10T04:22:56.990 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb8b00fe970 0x7fb8b019d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:56.990 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8b00fcde0 0x7fb8b0195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:56.991 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb89c099010 0x7fb89c09b4c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:56.991 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 --2- 172.21.15.110:0/1820062860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b00fdfc0 0x7fb8b01961c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:22:56.991 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.987+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 >> 172.21.15.110:0/1820062860 conn(0x7fb8b00f8430 msgr2=0x7fb8b0066700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:22:56.991 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.988+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 shutdown_connections 2022-09-10T04:22:56.992 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:22:56.988+0000 7fb8b8250700 1 -- 172.21.15.110:0/1820062860 wait complete. 2022-09-10T04:22:57.409 DEBUG:teuthology.orchestra.run.smithi110:osd.5> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.5.service 2022-09-10T04:22:57.414 INFO:tasks.cephadm:Deploying osd.6 on smithi081 with /dev/vg_nvme/lv_2... 2022-09-10T04:22:57.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-09-10T04:22:57.441 INFO:journalctl@ceph.osd.5.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:22:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-10T04:22:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-10T04:22:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:57.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-10T04:22:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-10T04:22:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:57.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-10T04:22:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2022-09-10T04:22:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:22:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:22:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:22:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:22:58.288 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:22:58.606 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:58.607 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-09-10T04:22:58.607 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: osdmap e37: 6 total, 5 up, 6 in 2022-09-10T04:22:58.607 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: from='osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:22:58.608 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:58.608 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:22:58.608 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:22:58.608 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: osdmap e38: 6 total, 5 up, 6 in 2022-09-10T04:22:58.609 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:58.657 INFO:journalctl@ceph.osd.5.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-5[109325]: 2022-09-10T04:22:58.312+0000 7f470c3c3700 -1 osd.5 0 waiting for initial osdmap 2022-09-10T04:22:58.657 INFO:journalctl@ceph.osd.5.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-5[109325]: 2022-09-10T04:22:58.320+0000 7f47071b6700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:22:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-09-10T04:22:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: osdmap e37: 6 total, 5 up, 6 in 2022-09-10T04:22:58.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: from='osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:22:58.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:58.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:22:58.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:22:58.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: osdmap e38: 6 total, 5 up, 6 in 2022-09-10T04:22:58.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:22:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2022-09-10T04:22:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: osdmap e37: 6 total, 5 up, 6 in 2022-09-10T04:22:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: from='osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:22:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:22:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:22:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: osdmap e38: 6 total, 5 up, 6 in 2022-09-10T04:22:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:59.490 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:59 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:59.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:59 smithi081 ceph-mon[90133]: osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] boot 2022-09-10T04:22:59.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:59 smithi081 ceph-mon[90133]: osdmap e39: 6 total, 6 up, 6 in 2022-09-10T04:22:59.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:22:59 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:59.609 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:59 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:59.609 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:59 smithi110 ceph-mon[97633]: osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] boot 2022-09-10T04:22:59.609 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:59 smithi110 ceph-mon[97633]: osdmap e39: 6 total, 6 up, 6 in 2022-09-10T04:22:59.610 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:22:59 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:59 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:22:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:59 smithi104 ceph-mon[97044]: osd.5 [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] boot 2022-09-10T04:22:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:59 smithi104 ceph-mon[97044]: osdmap e39: 6 total, 6 up, 6 in 2022-09-10T04:22:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:22:59 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:23:00.411 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-09-10T04:23:00.411 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-09-10T04:23:00.412 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2022-09-10T04:23:00.412 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-09-10T04:23:00.412 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0444339 s, 236 MB/s 2022-09-10T04:23:00.413 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2022-09-10T04:23:00.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:23:00.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:23:00.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:23:00.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi110 2022-09-10T04:23:00.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:00.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:00.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi110 to 8584M 2022-09-10T04:23:00.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:00 smithi081 ceph-mon[90133]: osdmap e40: 6 total, 6 up, 6 in 2022-09-10T04:23:00.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi081:vg_nvme/lv_2 2022-09-10T04:23:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:23:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:23:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:23:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi110 2022-09-10T04:23:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:00.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi110 to 8584M 2022-09-10T04:23:00.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:00.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:00 smithi110 ceph-mon[97633]: osdmap e40: 6 total, 6 up, 6 in 2022-09-10T04:23:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:23:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:23:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2022-09-10T04:23:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi110 2022-09-10T04:23:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi110 to 8584M 2022-09-10T04:23:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:01.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:00 smithi104 ceph-mon[97044]: osdmap e40: 6 total, 6 up, 6 in 2022-09-10T04:23:01.735 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:23:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:02 smithi110 ceph-mon[97633]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:02.707 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:02 smithi110 ceph-mon[97633]: osdmap e41: 6 total, 6 up, 6 in 2022-09-10T04:23:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:02 smithi104 ceph-mon[97044]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:02 smithi104 ceph-mon[97044]: osdmap e41: 6 total, 6 up, 6 in 2022-09-10T04:23:02.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:02 smithi081 ceph-mon[90133]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:02.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:02 smithi081 ceph-mon[90133]: osdmap e41: 6 total, 6 up, 6 in 2022-09-10T04:23:03.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 -- 172.21.15.81:0/325382550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe364103a30 msgr2=0x7fe364105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:03.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 --2- 172.21.15.81:0/325382550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe364103a30 0x7fe364105eb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009bb0 tx=0x7fe35c009ec0 comp rx=0 tx=0).stop 2022-09-10T04:23:03.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 -- 172.21.15.81:0/325382550 shutdown_connections 2022-09-10T04:23:03.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 --2- 172.21.15.81:0/325382550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe364103a30 0x7fe364105eb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:03.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 --2- 172.21.15.81:0/325382550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3640673d0 0x7fe364100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:03.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 --2- 172.21.15.81:0/325382550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe364100f40 0x7fe3641033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:03.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 -- 172.21.15.81:0/325382550 >> 172.21.15.81:0/325382550 conn(0x7fe3640f8430 msgr2=0x7fe3640fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:23:03.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 -- 172.21.15.81:0/325382550 shutdown_connections 2022-09-10T04:23:03.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.119+0000 7fe368ace700 1 -- 172.21.15.81:0/325382550 wait complete. 2022-09-10T04:23:03.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 Processor -- start 2022-09-10T04:23:03.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 -- start start 2022-09-10T04:23:03.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3640673d0 0x7fe364195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:03.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe364100f40 0x7fe3641961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:03.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe364103a30 0x7fe36419d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:03.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36419d830 con 0x7fe3640673d0 2022-09-10T04:23:03.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36419d9a0 con 0x7fe364100f40 2022-09-10T04:23:03.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe368ace700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36419db60 con 0x7fe364103a30 2022-09-10T04:23:03.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe36259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3640673d0 0x7fe364195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:03.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe362d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe364103a30 0x7fe36419d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:03.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.121+0000 7fe36259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3640673d0 0x7fe364195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56952/0 (socket says 172.21.15.81:56952) 2022-09-10T04:23:03.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe36259c700 1 -- 172.21.15.81:0/3712878134 learned_addr learned my addr 172.21.15.81:0/3712878134 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:23:03.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe361d9b700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe364100f40 0x7fe3641961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:03.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe362d9d700 1 -- 172.21.15.81:0/3712878134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe364100f40 msgr2=0x7fe3641961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:03.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe362d9d700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe364100f40 0x7fe3641961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:03.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe362d9d700 1 -- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3640673d0 msgr2=0x7fe364195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:03.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe362d9d700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3640673d0 0x7fe364195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:03.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe362d9d700 1 -- 172.21.15.81:0/3712878134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0097e0 con 0x7fe364103a30 2022-09-10T04:23:03.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.122+0000 7fe362d9d700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe364103a30 0x7fe36419d1e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009fd0 tx=0x7fe35c00b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:23:03.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.123+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c00f870 con 0x7fe364103a30 2022-09-10T04:23:03.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.123+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36419dde0 con 0x7fe364103a30 2022-09-10T04:23:03.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.123+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36419e1d0 con 0x7fe364103a30 2022-09-10T04:23:03.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.123+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe35c00ba80 con 0x7fe364103a30 2022-09-10T04:23:03.139 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.123+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c005890 con 0x7fe364103a30 2022-09-10T04:23:03.139 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.124+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3440051a0 con 0x7fe364103a30 2022-09-10T04:23:03.140 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.125+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe35c020070 con 0x7fe364103a30 2022-09-10T04:23:03.142 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.127+0000 7fe3537fe700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe34c099010 0x7fe34c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:03.142 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.127+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4529+0+0 (secure 0 0 0) 0x7fe35c0c2b80 con 0x7fe364103a30 2022-09-10T04:23:03.143 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.127+0000 7fe36259c700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe34c099010 0x7fe34c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:03.143 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.128+0000 7fe36259c700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe34c099010 0x7fe34c09b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe364197140 tx=0x7fe354005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:23:03.144 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.131+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe35c090090 con 0x7fe364103a30 2022-09-10T04:23:03.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:03.356+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe344000bf0 con 0x7fe34c099010 2022-09-10T04:23:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:03 smithi110 ceph-mon[97633]: osdmap e42: 6 total, 6 up, 6 in 2022-09-10T04:23:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:03 smithi104 ceph-mon[97044]: osdmap e42: 6 total, 6 up, 6 in 2022-09-10T04:23:03.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:03 smithi081 ceph-mon[90133]: osdmap e42: 6 total, 6 up, 6 in 2022-09-10T04:23:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:04 smithi110 ceph-mon[97633]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:04 smithi110 ceph-mon[97633]: osdmap e43: 6 total, 6 up, 6 in 2022-09-10T04:23:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:04 smithi110 ceph-mon[97633]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:23:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:23:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:23:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:04.678 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:04 smithi081 ceph-mon[90133]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:04.679 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:04 smithi081 ceph-mon[90133]: osdmap e43: 6 total, 6 up, 6 in 2022-09-10T04:23:04.679 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:04 smithi081 ceph-mon[90133]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:23:04.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:23:04.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:23:04.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:04 smithi104 ceph-mon[97044]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:04 smithi104 ceph-mon[97044]: osdmap e43: 6 total, 6 up, 6 in 2022-09-10T04:23:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:04 smithi104 ceph-mon[97044]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:23:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:23:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:23:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:06 smithi110 ceph-mon[97633]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:06 smithi104 ceph-mon[97044]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:06 smithi081 ceph-mon[90133]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:07 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3734835005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36d8ca99-65c5-4528-8ee8-de7cd6faf9fa"}]: dispatch 2022-09-10T04:23:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:07 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3734835005' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36d8ca99-65c5-4528-8ee8-de7cd6faf9fa"}]': finished 2022-09-10T04:23:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:07 smithi110 ceph-mon[97633]: osdmap e44: 7 total, 6 up, 7 in 2022-09-10T04:23:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:07 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3734835005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36d8ca99-65c5-4528-8ee8-de7cd6faf9fa"}]: dispatch 2022-09-10T04:23:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:07 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3734835005' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36d8ca99-65c5-4528-8ee8-de7cd6faf9fa"}]': finished 2022-09-10T04:23:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:07 smithi104 ceph-mon[97044]: osdmap e44: 7 total, 6 up, 7 in 2022-09-10T04:23:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:07 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3734835005' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "36d8ca99-65c5-4528-8ee8-de7cd6faf9fa"}]: dispatch 2022-09-10T04:23:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:07 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3734835005' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "36d8ca99-65c5-4528-8ee8-de7cd6faf9fa"}]': finished 2022-09-10T04:23:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:07 smithi081 ceph-mon[90133]: osdmap e44: 7 total, 6 up, 7 in 2022-09-10T04:23:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:08 smithi110 ceph-mon[97633]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:23:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:08 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1455979096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:23:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:08 smithi104 ceph-mon[97044]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:23:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:08 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1455979096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:23:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:08 smithi081 ceph-mon[90133]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:23:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:08 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1455979096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:23:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:23:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:23:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:23:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:23:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:23:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:23:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:10 smithi110 ceph-mon[97633]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2022-09-10T04:23:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:10 smithi104 ceph-mon[97044]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2022-09-10T04:23:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:10 smithi081 ceph-mon[90133]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2022-09-10T04:23:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:12 smithi110 ceph-mon[97633]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-10T04:23:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:12 smithi104 ceph-mon[97044]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-10T04:23:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:12 smithi081 ceph-mon[90133]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2022-09-10T04:23:14.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:14 smithi081 ceph-mon[90133]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:23:14.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:14 smithi110 ceph-mon[97633]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:23:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:14 smithi104 ceph-mon[97044]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:23:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-09-10T04:23:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-09-10T04:23:15.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2022-09-10T04:23:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:16 smithi110 ceph-mon[97633]: Deploying daemon osd.6 on smithi081 2022-09-10T04:23:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:16 smithi110 ceph-mon[97633]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:23:16.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:16 smithi081 ceph-mon[90133]: Deploying daemon osd.6 on smithi081 2022-09-10T04:23:16.490 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:16 smithi081 ceph-mon[90133]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:23:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:16 smithi104 ceph-mon[97044]: Deploying daemon osd.6 on smithi081 2022-09-10T04:23:16.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:16 smithi104 ceph-mon[97044]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:23:18.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:18 smithi110 ceph-mon[97633]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:18.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:18 smithi081 ceph-mon[90133]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:18 smithi104 ceph-mon[97044]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:20.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:20 smithi110 ceph-mon[97633]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:20 smithi104 ceph-mon[97044]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:20 smithi081 ceph-mon[90133]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:22 smithi110 ceph-mon[97633]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:22 smithi104 ceph-mon[97044]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:22 smithi081 ceph-mon[90133]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:24.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:24 smithi081 ceph-mon[90133]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:24 smithi110 ceph-mon[97633]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:24 smithi104 ceph-mon[97044]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:25.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:25.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:25.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:26 smithi104 ceph-mon[97044]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:26 smithi081 ceph-mon[90133]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:26 smithi110 ceph-mon[97633]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:28 smithi104 ceph-mon[97044]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:28 smithi081 ceph-mon[90133]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:28 smithi110 ceph-mon[97633]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:29 smithi104 ceph-mon[97044]: from='osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-10T04:23:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:29 smithi104 ceph-mon[97044]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-10T04:23:29.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:29 smithi081 ceph-mon[90133]: from='osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-10T04:23:29.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:29 smithi081 ceph-mon[90133]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-10T04:23:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:29 smithi110 ceph-mon[97633]: from='osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-10T04:23:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:29 smithi110 ceph-mon[97633]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2022-09-10T04:23:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:30 smithi104 ceph-mon[97044]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:30 smithi104 ceph-mon[97044]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-09-10T04:23:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:30 smithi104 ceph-mon[97044]: osdmap e45: 7 total, 6 up, 7 in 2022-09-10T04:23:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:30.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:30 smithi104 ceph-mon[97044]: from='osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:23:30.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:30 smithi104 ceph-mon[97044]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:23:30.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:30 smithi081 ceph-mon[90133]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:30.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:30 smithi081 ceph-mon[90133]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-09-10T04:23:30.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:30 smithi081 ceph-mon[90133]: osdmap e45: 7 total, 6 up, 7 in 2022-09-10T04:23:30.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:30.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:30 smithi081 ceph-mon[90133]: from='osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:23:30.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:30 smithi081 ceph-mon[90133]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:23:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:30 smithi110 ceph-mon[97633]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:30 smithi110 ceph-mon[97633]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2022-09-10T04:23:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:30 smithi110 ceph-mon[97633]: osdmap e45: 7 total, 6 up, 7 in 2022-09-10T04:23:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:30.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:30 smithi110 ceph-mon[97633]: from='osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:23:30.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:30 smithi110 ceph-mon[97633]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:23:31.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.225+0000 7fe3537fe700 1 -- 172.21.15.81:0/3712878134 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe344000bf0 con 0x7fe34c099010 2022-09-10T04:23:31.227 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 6 on host 'smithi081' 2022-09-10T04:23:31.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.227+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe34c099010 msgr2=0x7fe34c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:31.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.227+0000 7fe368ace700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe34c099010 0x7fe34c09b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe364197140 tx=0x7fe354005ea0 comp rx=0 tx=0).stop 2022-09-10T04:23:31.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.227+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe364103a30 msgr2=0x7fe36419d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:31.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.227+0000 7fe368ace700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe364103a30 0x7fe36419d1e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009fd0 tx=0x7fe35c00b890 comp rx=0 tx=0).stop 2022-09-10T04:23:31.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 shutdown_connections 2022-09-10T04:23:31.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe34c099010 0x7fe34c09b4c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:31.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe364103a30 0x7fe36419d1e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:31.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3640673d0 0x7fe364195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:31.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 --2- 172.21.15.81:0/3712878134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe364100f40 0x7fe3641961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:31.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 >> 172.21.15.81:0/3712878134 conn(0x7fe3640f8430 msgr2=0x7fe3640ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:23:31.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 shutdown_connections 2022-09-10T04:23:31.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:23:31.228+0000 7fe368ace700 1 -- 172.21.15.81:0/3712878134 wait complete. 2022-09-10T04:23:31.665 DEBUG:teuthology.orchestra.run.smithi081:osd.6> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.6.service 2022-09-10T04:23:31.672 INFO:tasks.cephadm:Deploying osd.7 on smithi104 with /dev/vg_nvme/lv_2... 2022-09-10T04:23:31.673 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-09-10T04:23:31.759 INFO:journalctl@ceph.osd.6.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:23:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:23:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: osdmap e46: 7 total, 6 up, 7 in 2022-09-10T04:23:31.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:31.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:31.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:31.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:31.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:31.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:31.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:23:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: osdmap e46: 7 total, 6 up, 7 in 2022-09-10T04:23:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:31.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: osdmap e46: 7 total, 6 up, 7 in 2022-09-10T04:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:32.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:32.270 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:23:32.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:32 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:23:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:32 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:23:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:32 smithi104 ceph-mon[97044]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:32 smithi104 ceph-mon[97044]: osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] boot 2022-09-10T04:23:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:32 smithi104 ceph-mon[97044]: osdmap e47: 7 total, 7 up, 7 in 2022-09-10T04:23:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:32 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:32 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:23:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:32 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:23:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:32 smithi081 ceph-mon[90133]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:32 smithi081 ceph-mon[90133]: osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] boot 2022-09-10T04:23:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:32 smithi081 ceph-mon[90133]: osdmap e47: 7 total, 7 up, 7 in 2022-09-10T04:23:32.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:32 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:32 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:23:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:32 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:23:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:32 smithi110 ceph-mon[97633]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2022-09-10T04:23:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:32 smithi110 ceph-mon[97633]: osd.6 [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] boot 2022-09-10T04:23:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:32 smithi110 ceph-mon[97633]: osdmap e47: 7 total, 7 up, 7 in 2022-09-10T04:23:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:32 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:23:33.718 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-09-10T04:23:33.718 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-09-10T04:23:33.719 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2022-09-10T04:23:33.719 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-09-10T04:23:33.719 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112798 s, 930 MB/s 2022-09-10T04:23:33.720 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 2022-09-10T04:23:33.720 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2022-09-10T04:23:33.879 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:33 smithi104 ceph-mon[97044]: osdmap e48: 7 total, 7 up, 7 in 2022-09-10T04:23:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:33 smithi110 ceph-mon[97633]: osdmap e48: 7 total, 7 up, 7 in 2022-09-10T04:23:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:33 smithi081 ceph-mon[90133]: osdmap e48: 7 total, 7 up, 7 in 2022-09-10T04:23:34.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2022-09-10T04:23:34.671 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:23:34.902 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:34 smithi104 ceph-mon[97044]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:34 smithi110 ceph-mon[97633]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:34.944 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:34 smithi081 ceph-mon[90133]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:35.909 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 -- 172.21.15.104:0/1979531639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 msgr2=0x7f89400fe430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:35.910 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 --2- 172.21.15.104:0/1979531639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f89400fe430 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8934009810 tx=0x7f8934009b20 comp rx=0 tx=0).stop 2022-09-10T04:23:35.910 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 -- 172.21.15.104:0/1979531639 shutdown_connections 2022-09-10T04:23:35.910 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 --2- 172.21.15.104:0/1979531639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fe970 0x7f8940105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:35.911 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 --2- 172.21.15.104:0/1979531639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fcde0 0x7f89400fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:35.911 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 --2- 172.21.15.104:0/1979531639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f89400fe430 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:35.911 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 -- 172.21.15.104:0/1979531639 >> 172.21.15.104:0/1979531639 conn(0x7f89400f8430 msgr2=0x7f89400fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:23:35.911 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 -- 172.21.15.104:0/1979531639 shutdown_connections 2022-09-10T04:23:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.907+0000 7f894680c700 1 -- 172.21.15.104:0/1979531639 wait complete. 2022-09-10T04:23:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 Processor -- start 2022-09-10T04:23:35.912 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 -- start start 2022-09-10T04:23:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fcde0 0x7f8940067250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:35.915 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f8940198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fe970 0x7f894019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894019b5a0 con 0x7f89400fcde0 2022-09-10T04:23:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894019b710 con 0x7f89400fdfc0 2022-09-10T04:23:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f894680c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894019b8d0 con 0x7f89400fe970 2022-09-10T04:23:35.916 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f893f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f8940198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f893f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f8940198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59068/0 (socket says 172.21.15.104:59068) 2022-09-10T04:23:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f893f7fe700 1 -- 172.21.15.104:0/1251044668 learned_addr learned my addr 172.21.15.104:0/1251044668 (peer_addr_for_me v2:172.21.15.104:0/0) 2022-09-10T04:23:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.908+0000 7f893ffff700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fcde0 0x7f8940067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:35.917 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f8944da9700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fe970 0x7f894019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893f7fe700 1 -- 172.21.15.104:0/1251044668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fe970 msgr2=0x7f894019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893f7fe700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fe970 0x7f894019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893f7fe700 1 -- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fcde0 msgr2=0x7f8940067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:35.918 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893f7fe700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fcde0 0x7f8940067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893f7fe700 1 -- 172.21.15.104:0/1251044668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934009470 con 0x7f89400fdfc0 2022-09-10T04:23:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f8944da9700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fe970 0x7f894019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:23:35.919 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893ffff700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fcde0 0x7f8940067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:23:35.920 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893f7fe700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f8940198720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8934006010 tx=0x7f89340045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:23:35.920 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893401d070 con 0x7f89400fdfc0 2022-09-10T04:23:35.920 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f89340054d0 con 0x7f89400fdfc0 2022-09-10T04:23:35.920 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340049c0 con 0x7f89400fdfc0 2022-09-10T04:23:35.921 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894019bb50 con 0x7f89400fdfc0 2022-09-10T04:23:35.921 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.909+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894019c010 con 0x7f89400fdfc0 2022-09-10T04:23:35.921 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.910+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89401084d0 con 0x7f89400fdfc0 2022-09-10T04:23:35.921 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.911+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8934028030 con 0x7f89400fdfc0 2022-09-10T04:23:35.921 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.911+0000 7f893d7fa700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8928099010 0x7f892809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.911+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4961+0+0 (secure 0 0 0) 0x7f89340c26f0 con 0x7f89400fdfc0 2022-09-10T04:23:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.912+0000 7f893ffff700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8928099010 0x7f892809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:35.922 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.912+0000 7f893ffff700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8928099010 0x7f892809b4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89300057d0 tx=0x7f8930005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:23:35.923 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:35.916+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f893408fb40 con 0x7f89400fdfc0 2022-09-10T04:23:36.110 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:36.108+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f894005fbd0 con 0x7f8928099010 2022-09-10T04:23:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi081 2022-09-10T04:23:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi081 to 5757M 2022-09-10T04:23:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:36 smithi104 ceph-mon[97044]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:36.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi081 2022-09-10T04:23:36.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi081 to 5757M 2022-09-10T04:23:36.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:36 smithi081 ceph-mon[90133]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi081 2022-09-10T04:23:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi081 to 5757M 2022-09-10T04:23:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:36.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:36 smithi110 ceph-mon[97633]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:37 smithi104 ceph-mon[97044]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:23:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:23:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:23:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:37.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:37 smithi081 ceph-mon[90133]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:23:37.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:23:37.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:23:37.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:37 smithi110 ceph-mon[97633]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:23:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:23:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:23:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:38 smithi104 ceph-mon[97044]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:38.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:38 smithi081 ceph-mon[90133]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:38.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:38 smithi110 ceph-mon[97633]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:39 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/2099448579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]: dispatch 2022-09-10T04:23:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:39 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]: dispatch 2022-09-10T04:23:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:39 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]': finished 2022-09-10T04:23:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:39 smithi110 ceph-mon[97633]: osdmap e49: 8 total, 7 up, 8 in 2022-09-10T04:23:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:39 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/1085934244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:23:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:39 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/2099448579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]: dispatch 2022-09-10T04:23:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:39 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]: dispatch 2022-09-10T04:23:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:39 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]': finished 2022-09-10T04:23:39.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:39 smithi104 ceph-mon[97044]: osdmap e49: 8 total, 7 up, 8 in 2022-09-10T04:23:39.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:39.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:39 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/1085934244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:23:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:39 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/2099448579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]: dispatch 2022-09-10T04:23:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:39 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]: dispatch 2022-09-10T04:23:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:39 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0c52ea1-4f7e-4760-b9c5-9b369d476ebe"}]': finished 2022-09-10T04:23:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:39 smithi081 ceph-mon[90133]: osdmap e49: 8 total, 7 up, 8 in 2022-09-10T04:23:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:39 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/1085934244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:23:40.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:40 smithi110 ceph-mon[97633]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:40 smithi104 ceph-mon[97044]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:40 smithi081 ceph-mon[90133]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:42.500 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:42 smithi104 ceph-mon[97044]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:42 smithi110 ceph-mon[97633]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:42.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:42 smithi081 ceph-mon[90133]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:44.261 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:44 smithi104 ceph-mon[97044]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:44.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:44 smithi110 ceph-mon[97633]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:44 smithi081 ceph-mon[90133]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-09-10T04:23:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:45.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-09-10T04:23:45.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2022-09-10T04:23:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:46 smithi104 ceph-mon[97044]: Deploying daemon osd.7 on smithi104 2022-09-10T04:23:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:46 smithi104 ceph-mon[97044]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:46 smithi081 ceph-mon[90133]: Deploying daemon osd.7 on smithi104 2022-09-10T04:23:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:46 smithi081 ceph-mon[90133]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:46 smithi110 ceph-mon[97633]: Deploying daemon osd.7 on smithi104 2022-09-10T04:23:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:46 smithi110 ceph-mon[97633]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:48 smithi110 ceph-mon[97633]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:48 smithi104 ceph-mon[97044]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:48.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:48 smithi081 ceph-mon[90133]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:50.618 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:50 smithi104 ceph-mon[97044]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:50.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:50 smithi110 ceph-mon[97633]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:50.800 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:50 smithi081 ceph-mon[90133]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:51 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:51 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:51 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:51 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:51 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:52.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:51 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:52.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:51 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:52.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:51 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:52.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:51 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:52.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:51 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:52.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:51 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:52 smithi104 ceph-mon[97044]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:52 smithi081 ceph-mon[90133]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:52 smithi110 ceph-mon[97633]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:54.531 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.531 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.531 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:54 smithi104 ceph-mon[97044]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:54 smithi081 ceph-mon[90133]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:54 smithi110 ceph-mon[97633]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:54.923 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 7 on host 'smithi104' 2022-09-10T04:23:54.924 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.920+0000 7f893d7fa700 1 -- 172.21.15.104:0/1251044668 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f894005fbd0 con 0x7f8928099010 2022-09-10T04:23:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8928099010 msgr2=0x7f892809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:54.926 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8928099010 0x7f892809b4c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89300057d0 tx=0x7f8930005dd0 comp rx=0 tx=0).stop 2022-09-10T04:23:54.927 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 msgr2=0x7f8940198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:54.927 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f8940198720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8934006010 tx=0x7f89340045d0 comp rx=0 tx=0).stop 2022-09-10T04:23:54.927 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 shutdown_connections 2022-09-10T04:23:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8928099010 0x7f892809b4c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89400fe970 0x7f894019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:54.928 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89400fcde0 0x7f8940067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 --2- 172.21.15.104:0/1251044668 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f89400fdfc0 0x7f8940198720 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 >> 172.21.15.104:0/1251044668 conn(0x7f89400f8430 msgr2=0x7f89400fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:23:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 shutdown_connections 2022-09-10T04:23:54.929 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:23:54.923+0000 7f894680c700 1 -- 172.21.15.104:0/1251044668 wait complete. 2022-09-10T04:23:55.335 DEBUG:teuthology.orchestra.run.smithi104:osd.7> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.7.service 2022-09-10T04:23:55.338 INFO:tasks.cephadm:Deploying osd.8 on smithi110 with /dev/vg_nvme/lv_2... 2022-09-10T04:23:55.339 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2022-09-10T04:23:55.366 INFO:journalctl@ceph.osd.7.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:23:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-10T04:23:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-10T04:23:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:55 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-10T04:23:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-10T04:23:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:55.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:55.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:55.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:55.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:55 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-10T04:23:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2022-09-10T04:23:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:23:55.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:23:55.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:55.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:55 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:23:55.937 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:23:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:56 smithi110 ceph-mon[97633]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-09-10T04:23:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:56 smithi110 ceph-mon[97633]: osdmap e50: 8 total, 7 up, 8 in 2022-09-10T04:23:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:56 smithi110 ceph-mon[97633]: from='osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:23:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:56 smithi110 ceph-mon[97633]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:23:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:56 smithi110 ceph-mon[97633]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:56.546 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-7[114053]: 2022-09-10T04:23:56.281+0000 7f15a1c87700 -1 osd.7 0 waiting for initial osdmap 2022-09-10T04:23:56.546 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-7[114053]: 2022-09-10T04:23:56.291+0000 7f159ca7a700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:23:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-mon[97044]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-09-10T04:23:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-mon[97044]: osdmap e50: 8 total, 7 up, 8 in 2022-09-10T04:23:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:56.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-mon[97044]: from='osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:23:56.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-mon[97044]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:23:56.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:56 smithi104 ceph-mon[97044]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:56 smithi081 ceph-mon[90133]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2022-09-10T04:23:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:56 smithi081 ceph-mon[90133]: osdmap e50: 8 total, 7 up, 8 in 2022-09-10T04:23:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:56.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:56 smithi081 ceph-mon[90133]: from='osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:23:56.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:56 smithi081 ceph-mon[90133]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:23:56.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:56 smithi081 ceph-mon[90133]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2022-09-10T04:23:57.406 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_2 2022-09-10T04:23:57.406 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2022-09-10T04:23:57.407 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2022-09-10T04:23:57.407 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2022-09-10T04:23:57.407 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832178 s, 1.3 GB/s 2022-09-10T04:23:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2022-09-10T04:23:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:23:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: osdmap e51: 8 total, 7 up, 8 in 2022-09-10T04:23:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] boot 2022-09-10T04:23:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: osdmap e52: 8 total, 8 up, 8 in 2022-09-10T04:23:57.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.549 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:23:57.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: osdmap e51: 8 total, 7 up, 8 in 2022-09-10T04:23:57.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.550 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.551 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] boot 2022-09-10T04:23:57.551 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: osdmap e52: 8 total, 8 up, 8 in 2022-09-10T04:23:57.551 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:23:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: osdmap e51: 8 total, 7 up, 8 in 2022-09-10T04:23:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: osd.7 [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] boot 2022-09-10T04:23:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: osdmap e52: 8 total, 8 up, 8 in 2022-09-10T04:23:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:23:57.849 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi110:vg_nvme/lv_2 2022-09-10T04:23:58.433 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:23:58.834 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:23:58.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:23:58.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:23:58.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi104 2022-09-10T04:23:58.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:58.835 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:58.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:58.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:58.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:58.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi104 to 5722M 2022-09-10T04:23:58.836 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:58.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:58.837 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:23:58 smithi110 ceph-mon[97633]: osdmap e53: 8 total, 8 up, 8 in 2022-09-10T04:23:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:23:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:23:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:23:59.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi104 2022-09-10T04:23:59.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:59.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:59.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:59.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi104 to 5722M 2022-09-10T04:23:59.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:23:58 smithi104 ceph-mon[97044]: osdmap e53: 8 total, 8 up, 8 in 2022-09-10T04:23:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:23:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:23:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:23:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi104 2022-09-10T04:23:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:23:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi104 to 5722M 2022-09-10T04:23:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:23:59.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:23:58 smithi081 ceph-mon[90133]: osdmap e53: 8 total, 8 up, 8 in 2022-09-10T04:23:59.637 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 -- 172.21.15.110:0/295277181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 msgr2=0x7f3a5c0ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:59.638 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/295277181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c0ff500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009840 tx=0x7f3a4c009b50 comp rx=0 tx=0).stop 2022-09-10T04:23:59.638 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 -- 172.21.15.110:0/295277181 shutdown_connections 2022-09-10T04:23:59.639 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/295277181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0ffa40 0x7f3a5c106370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:59.639 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/295277181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 0x7f3a5c0fe230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:59.639 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/295277181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c0ff500 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:59.640 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 -- 172.21.15.110:0/295277181 >> 172.21.15.110:0/295277181 conn(0x7f3a5c0f9540 msgr2=0x7f3a5c0fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:23:59.640 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 -- 172.21.15.110:0/295277181 shutdown_connections 2022-09-10T04:23:59.641 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.630+0000 7f3a61d8b700 1 -- 172.21.15.110:0/295277181 wait complete. 2022-09-10T04:23:59.641 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 Processor -- start 2022-09-10T04:23:59.642 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 -- start start 2022-09-10T04:23:59.642 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 0x7f3a5c193a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:59.643 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c193fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:59.643 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0ffa40 0x7f3a5c19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:59.643 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c19b600 con 0x7f3a5c0ffa40 2022-09-10T04:23:59.644 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c19b770 con 0x7f3a5c0fde20 2022-09-10T04:23:59.644 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a61d8b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c19b930 con 0x7f3a5c0ff090 2022-09-10T04:23:59.645 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a5affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c193fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:59.645 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a5affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c193fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60828/0 (socket says 172.21.15.110:60828) 2022-09-10T04:23:59.646 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.631+0000 7f3a5affd700 1 -- 172.21.15.110:0/4193421850 learned_addr learned my addr 172.21.15.110:0/4193421850 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:23:59.646 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5b7fe700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 0x7f3a5c193a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:59.647 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5affd700 1 -- 172.21.15.110:0/4193421850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 msgr2=0x7f3a5c193a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:59.647 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5affd700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 0x7f3a5c193a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:59.647 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5affd700 1 -- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0ffa40 msgr2=0x7f3a5c19b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:23:59.648 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5affd700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0ffa40 0x7f3a5c19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:23:59.648 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5affd700 1 -- 172.21.15.110:0/4193421850 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c009470 con 0x7f3a5c0ff090 2022-09-10T04:23:59.649 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5b7fe700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 0x7f3a5c193a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:23:59.649 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a5affd700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c193fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009fb0 tx=0x7f3a4c00f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:23:59.650 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c01d070 con 0x7f3a5c0ff090 2022-09-10T04:23:59.650 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3a4c00be50 con 0x7f3a5c0ff090 2022-09-10T04:23:59.651 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c19bbb0 con 0x7f3a5c0ff090 2022-09-10T04:23:59.651 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c004870 con 0x7f3a5c0ff090 2022-09-10T04:23:59.652 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.632+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c19c0e0 con 0x7f3a5c0ff090 2022-09-10T04:23:59.652 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.633+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5c109670 con 0x7f3a5c0ff090 2022-09-10T04:23:59.653 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.634+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3a4c028030 con 0x7f3a5c0ff090 2022-09-10T04:23:59.653 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.634+0000 7f3a58ff9700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3a44099010 0x7f3a4409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:23:59.654 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.634+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5422+0+0 (secure 0 0 0) 0x7f3a4c0c2770 con 0x7f3a5c0ff090 2022-09-10T04:23:59.654 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.635+0000 7f3a5b7fe700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3a44099010 0x7f3a4409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:23:59.654 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.635+0000 7f3a5b7fe700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3a44099010 0x7f3a4409b4c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005fd0 tx=0x7f3a50005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:23:59.656 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.639+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3a4c08fa00 con 0x7f3a5c0ff090 2022-09-10T04:23:59.833 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:23:59.832+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a5c05f120 con 0x7f3a44099010 2022-09-10T04:24:00.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:00 smithi110 ceph-mon[97633]: osdmap e54: 8 total, 8 up, 8 in 2022-09-10T04:24:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:00 smithi110 ceph-mon[97633]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:00 smithi110 ceph-mon[97633]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:00 smithi104 ceph-mon[97044]: osdmap e54: 8 total, 8 up, 8 in 2022-09-10T04:24:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:00 smithi104 ceph-mon[97044]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:00 smithi104 ceph-mon[97044]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:00.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:00.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:00 smithi081 ceph-mon[90133]: osdmap e54: 8 total, 8 up, 8 in 2022-09-10T04:24:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:00 smithi081 ceph-mon[90133]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:00 smithi081 ceph-mon[90133]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:00.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:01 smithi110 ceph-mon[97633]: osdmap e55: 8 total, 8 up, 8 in 2022-09-10T04:24:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:01 smithi104 ceph-mon[97044]: osdmap e55: 8 total, 8 up, 8 in 2022-09-10T04:24:01.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:01 smithi081 ceph-mon[90133]: osdmap e55: 8 total, 8 up, 8 in 2022-09-10T04:24:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:02 smithi110 ceph-mon[97633]: osdmap e56: 8 total, 8 up, 8 in 2022-09-10T04:24:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:02 smithi110 ceph-mon[97633]: pgmap v192: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2022-09-10T04:24:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:02 smithi104 ceph-mon[97044]: osdmap e56: 8 total, 8 up, 8 in 2022-09-10T04:24:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:02 smithi104 ceph-mon[97044]: pgmap v192: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2022-09-10T04:24:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:02 smithi081 ceph-mon[90133]: osdmap e56: 8 total, 8 up, 8 in 2022-09-10T04:24:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:02 smithi081 ceph-mon[90133]: pgmap v192: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2022-09-10T04:24:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-09-10T04:24:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2022-09-10T04:24:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/181459777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]: dispatch 2022-09-10T04:24:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]: dispatch 2022-09-10T04:24:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]': finished 2022-09-10T04:24:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: osdmap e57: 9 total, 8 up, 9 in 2022-09-10T04:24:03.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:03.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:03 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/494288390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:24:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-09-10T04:24:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2022-09-10T04:24:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/181459777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]: dispatch 2022-09-10T04:24:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]: dispatch 2022-09-10T04:24:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]': finished 2022-09-10T04:24:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: osdmap e57: 9 total, 8 up, 9 in 2022-09-10T04:24:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:03 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/494288390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:24:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2022-09-10T04:24:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2022-09-10T04:24:03.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/181459777' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]: dispatch 2022-09-10T04:24:03.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]: dispatch 2022-09-10T04:24:03.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08ff30d-679b-4637-b363-c5fcb59fe112"}]': finished 2022-09-10T04:24:03.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: osdmap e57: 9 total, 8 up, 9 in 2022-09-10T04:24:03.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:03.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:03 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/494288390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:24:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:04 smithi104 ceph-mon[97044]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:04 smithi081 ceph-mon[90133]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:04.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:04 smithi110 ceph-mon[97633]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:05 smithi104 ceph-mon[97044]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2022-09-10T04:24:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:05 smithi104 ceph-mon[97044]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2022-09-10T04:24:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:05 smithi104 ceph-mon[97044]: Cluster is now healthy 2022-09-10T04:24:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:05 smithi081 ceph-mon[90133]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2022-09-10T04:24:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:05 smithi081 ceph-mon[90133]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2022-09-10T04:24:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:05 smithi081 ceph-mon[90133]: Cluster is now healthy 2022-09-10T04:24:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:05 smithi110 ceph-mon[97633]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2022-09-10T04:24:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:05 smithi110 ceph-mon[97633]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2022-09-10T04:24:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:05 smithi110 ceph-mon[97633]: Cluster is now healthy 2022-09-10T04:24:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:06 smithi104 ceph-mon[97044]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:06.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:06 smithi081 ceph-mon[90133]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:06 smithi110 ceph-mon[97633]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:08 smithi110 ceph-mon[97633]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:08 smithi104 ceph-mon[97044]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:08.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:08 smithi081 ceph-mon[90133]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:09.639 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-09-10T04:24:09.640 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:09.640 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:09 smithi110 ceph-mon[97633]: Deploying daemon osd.8 on smithi110 2022-09-10T04:24:09.640 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:24:09.641 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:24:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-09-10T04:24:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:09 smithi104 ceph-mon[97044]: Deploying daemon osd.8 on smithi110 2022-09-10T04:24:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:24:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:24:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2022-09-10T04:24:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:09 smithi081 ceph-mon[90133]: Deploying daemon osd.8 on smithi110 2022-09-10T04:24:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:24:09.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:24:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:10 smithi110 ceph-mon[97633]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:10 smithi104 ceph-mon[97044]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:10 smithi081 ceph-mon[90133]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:12 smithi110 ceph-mon[97633]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:12 smithi104 ceph-mon[97044]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:12.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:12 smithi081 ceph-mon[90133]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:14 smithi110 ceph-mon[97633]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:14 smithi104 ceph-mon[97044]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:14 smithi081 ceph-mon[90133]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:14.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:14.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:14.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:14.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:14.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:16 smithi110 ceph-mon[97633]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:16 smithi104 ceph-mon[97044]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:16.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:16 smithi081 ceph-mon[90133]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:17.955 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.954+0000 7f3a58ff9700 1 -- 172.21.15.110:0/4193421850 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3a5c05f120 con 0x7f3a44099010 2022-09-10T04:24:17.955 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 8 on host 'smithi110' 2022-09-10T04:24:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3a44099010 msgr2=0x7f3a4409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3a44099010 0x7f3a4409b4c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a50005fd0 tx=0x7f3a50005d80 comp rx=0 tx=0).stop 2022-09-10T04:24:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 msgr2=0x7f3a5c193fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c193fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c009fb0 tx=0x7f3a4c00f740 comp rx=0 tx=0).stop 2022-09-10T04:24:17.958 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 shutdown_connections 2022-09-10T04:24:17.958 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c0ffa40 0x7f3a5c19b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:17.958 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c0fde20 0x7f3a5c193a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3a44099010 0x7f3a4409b4c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 --2- 172.21.15.110:0/4193421850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a5c0ff090 0x7f3a5c193fc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 >> 172.21.15.110:0/4193421850 conn(0x7f3a5c0f9540 msgr2=0x7f3a5c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:24:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 shutdown_connections 2022-09-10T04:24:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:24:17.956+0000 7f3a61d8b700 1 -- 172.21.15.110:0/4193421850 wait complete. 2022-09-10T04:24:18.495 DEBUG:teuthology.orchestra.run.smithi110:osd.8> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.8.service 2022-09-10T04:24:18.497 INFO:tasks.cephadm:Deploying osd.9 on smithi081 with /dev/vg_nvme/lv_1... 2022-09-10T04:24:18.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-09-10T04:24:18.526 INFO:journalctl@ceph.osd.8.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:24:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-10T04:24:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:18 smithi110 ceph-mon[97633]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-10T04:24:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-10T04:24:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:18 smithi104 ceph-mon[97044]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-10T04:24:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-10T04:24:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:18 smithi081 ceph-mon[90133]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2022-09-10T04:24:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:24:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-mon[97633]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-09-10T04:24:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-mon[97633]: osdmap e58: 9 total, 8 up, 9 in 2022-09-10T04:24:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-mon[97633]: from='osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:24:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-mon[97633]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:24:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-mon[97633]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:20.159 INFO:journalctl@ceph.osd.8.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-8[115270]: 2022-09-10T04:24:19.967+0000 7f5d01606700 -1 osd.8 0 waiting for initial osdmap 2022-09-10T04:24:20.160 INFO:journalctl@ceph.osd.8.smithi110.stdout:Sep 10 04:24:19 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-8[115270]: 2022-09-10T04:24:19.979+0000 7f5cfbbf8700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:24:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:19 smithi104 ceph-mon[97044]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-09-10T04:24:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:19 smithi104 ceph-mon[97044]: osdmap e58: 9 total, 8 up, 9 in 2022-09-10T04:24:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:19 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:19 smithi104 ceph-mon[97044]: from='osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:24:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:19 smithi104 ceph-mon[97044]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:24:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:19 smithi104 ceph-mon[97044]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:19 smithi081 ceph-mon[90133]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2022-09-10T04:24:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:19 smithi081 ceph-mon[90133]: osdmap e58: 9 total, 8 up, 9 in 2022-09-10T04:24:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:19 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:19 smithi081 ceph-mon[90133]: from='osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:24:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:19 smithi081 ceph-mon[90133]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:24:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:19 smithi081 ceph-mon[90133]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2022-09-10T04:24:20.998 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:24:20.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: osdmap e59: 9 total, 8 up, 9 in 2022-09-10T04:24:20.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:20.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:20.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi110 2022-09-10T04:24:20.999 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.000 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.000 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.000 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.000 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.001 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi110 to 5722M 2022-09-10T04:24:21.001 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.001 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.001 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] boot 2022-09-10T04:24:21.001 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: osdmap e60: 9 total, 9 up, 9 in 2022-09-10T04:24:21.002 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:24:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: osdmap e59: 9 total, 8 up, 9 in 2022-09-10T04:24:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi110 2022-09-10T04:24:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi110 to 5722M 2022-09-10T04:24:21.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] boot 2022-09-10T04:24:21.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: osdmap e60: 9 total, 9 up, 9 in 2022-09-10T04:24:21.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:24:21.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: osdmap e59: 9 total, 8 up, 9 in 2022-09-10T04:24:21.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi110 2022-09-10T04:24:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:21.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi110 to 5722M 2022-09-10T04:24:21.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:21.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: osd.8 [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] boot 2022-09-10T04:24:21.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: osdmap e60: 9 total, 9 up, 9 in 2022-09-10T04:24:21.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:24:21.490 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-09-10T04:24:21.490 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-09-10T04:24:21.491 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2022-09-10T04:24:21.491 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2022-09-10T04:24:21.491 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0436754 s, 240 MB/s 2022-09-10T04:24:21.491 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2022-09-10T04:24:22.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi081:vg_nvme/lv_1 2022-09-10T04:24:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:21 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:24:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:21 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:24:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:21 smithi104 ceph-mon[97044]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:21 smithi104 ceph-mon[97044]: osdmap e61: 9 total, 9 up, 9 in 2022-09-10T04:24:22.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:21 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:24:22.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:21 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:24:22.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:21 smithi081 ceph-mon[90133]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:22.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:21 smithi081 ceph-mon[90133]: osdmap e61: 9 total, 9 up, 9 in 2022-09-10T04:24:22.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:21 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:24:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:21 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:24:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:21 smithi110 ceph-mon[97633]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:21 smithi110 ceph-mon[97633]: osdmap e61: 9 total, 9 up, 9 in 2022-09-10T04:24:22.907 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:24:24.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:23 smithi081 ceph-mon[90133]: osdmap e62: 9 total, 9 up, 9 in 2022-09-10T04:24:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:23 smithi081 ceph-mon[90133]: pgmap v209: 1 pgs: 1 remapped+peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 -- 172.21.15.81:0/3992961256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fcd50 msgr2=0x7fae7c0fd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 --2- 172.21.15.81:0/3992961256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fcd50 0x7fae7c0fd160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fae6c009810 tx=0x7fae6c009b20 comp rx=0 tx=0).stop 2022-09-10T04:24:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 -- 172.21.15.81:0/3992961256 shutdown_connections 2022-09-10T04:24:24.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 --2- 172.21.15.81:0/3992961256 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe960 0x7fae7c105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 --2- 172.21.15.81:0/3992961256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fcd50 0x7fae7c0fd160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 --2- 172.21.15.81:0/3992961256 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fdfd0 0x7fae7c0fe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 -- 172.21.15.81:0/3992961256 >> 172.21.15.81:0/3992961256 conn(0x7fae7c0f8460 msgr2=0x7fae7c0fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:24:24.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 -- 172.21.15.81:0/3992961256 shutdown_connections 2022-09-10T04:24:24.298 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.294+0000 7fae80b17700 1 -- 172.21.15.81:0/3992961256 wait complete. 2022-09-10T04:24:24.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:23 smithi104 ceph-mon[97044]: osdmap e62: 9 total, 9 up, 9 in 2022-09-10T04:24:24.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:23 smithi104 ceph-mon[97044]: pgmap v209: 1 pgs: 1 remapped+peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.295+0000 7fae80b17700 1 Processor -- start 2022-09-10T04:24:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.295+0000 7fae80b17700 1 -- start start 2022-09-10T04:24:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae80b17700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fcd50 0x7fae7c195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae80b17700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fdfd0 0x7fae7c1961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae80b17700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fe960 0x7fae7c19d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae80b17700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c19d830 con 0x7fae7c0fdfd0 2022-09-10T04:24:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae80b17700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c19d9a0 con 0x7fae7c0fcd50 2022-09-10T04:24:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae80b17700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c19db60 con 0x7fae7c0fe960 2022-09-10T04:24:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae79d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fdfd0 0x7fae7c1961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fcd50 0x7fae7c195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae79d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fdfd0 0x7fae7c1961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46622/0 (socket says 172.21.15.81:46622) 2022-09-10T04:24:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae79d9b700 1 -- 172.21.15.81:0/659968915 learned_addr learned my addr 172.21.15.81:0/659968915 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:24:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.296+0000 7fae7ad9d700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fe960 0x7fae7c19d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7a59c700 1 -- 172.21.15.81:0/659968915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fe960 msgr2=0x7fae7c19d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7a59c700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fe960 0x7fae7c19d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7a59c700 1 -- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fdfd0 msgr2=0x7fae7c1961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7a59c700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fdfd0 0x7fae7c1961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7a59c700 1 -- 172.21.15.81:0/659968915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c009470 con 0x7fae7c0fcd50 2022-09-10T04:24:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7ad9d700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fe960 0x7fae7c19d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:24:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae7a59c700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fcd50 0x7fae7c195c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:24:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c01d070 con 0x7fae7c0fcd50 2022-09-10T04:24:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c19dde0 con 0x7fae7c0fcd50 2022-09-10T04:24:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fae6c00f880 con 0x7fae7c0fcd50 2022-09-10T04:24:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.297+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c004760 con 0x7fae7c0fcd50 2022-09-10T04:24:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.298+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c19e220 con 0x7fae7c0fcd50 2022-09-10T04:24:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.299+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c108500 con 0x7fae7c0fcd50 2022-09-10T04:24:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.301+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fae6c028030 con 0x7fae7c0fcd50 2022-09-10T04:24:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.302+0000 7fae6b7fe700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fae64099090 0x7fae6409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.302+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae6c0c26a0 con 0x7fae7c0fcd50 2022-09-10T04:24:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.302+0000 7fae79d9b700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fae64099090 0x7fae6409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.303+0000 7fae79d9b700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fae64099090 0x7fae6409b540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fae7c197180 tx=0x7fae70005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:24:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.304+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fae6c08f6a0 con 0x7fae7c0fcd50 2022-09-10T04:24:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:23 smithi110 ceph-mon[97633]: osdmap e62: 9 total, 9 up, 9 in 2022-09-10T04:24:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:23 smithi110 ceph-mon[97633]: pgmap v209: 1 pgs: 1 remapped+peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:24.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:24.544+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fae7c0676a0 con 0x7fae64099090 2022-09-10T04:24:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:25 smithi110 ceph-mon[97633]: osdmap e63: 9 total, 9 up, 9 in 2022-09-10T04:24:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:25 smithi110 ceph-mon[97633]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:25 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:25 smithi104 ceph-mon[97044]: osdmap e63: 9 total, 9 up, 9 in 2022-09-10T04:24:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:25 smithi104 ceph-mon[97044]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:25 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:25 smithi081 ceph-mon[90133]: osdmap e63: 9 total, 9 up, 9 in 2022-09-10T04:24:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:25 smithi081 ceph-mon[90133]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:25.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:25.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:25 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:26.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:26 smithi110 ceph-mon[97633]: osdmap e64: 9 total, 9 up, 9 in 2022-09-10T04:24:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:26 smithi110 ceph-mon[97633]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:26.428 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:26 smithi081 ceph-mon[90133]: osdmap e64: 9 total, 9 up, 9 in 2022-09-10T04:24:26.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:26 smithi081 ceph-mon[90133]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:26 smithi104 ceph-mon[97044]: osdmap e64: 9 total, 9 up, 9 in 2022-09-10T04:24:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:26 smithi104 ceph-mon[97044]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:28 smithi110 ceph-mon[97633]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 108 KiB/s, 0 objects/s recovering 2022-09-10T04:24:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:28 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4182056290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2"}]: dispatch 2022-09-10T04:24:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:28 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4182056290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2"}]': finished 2022-09-10T04:24:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:28 smithi110 ceph-mon[97633]: osdmap e65: 10 total, 9 up, 10 in 2022-09-10T04:24:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:28 smithi104 ceph-mon[97044]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 108 KiB/s, 0 objects/s recovering 2022-09-10T04:24:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:28 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4182056290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2"}]: dispatch 2022-09-10T04:24:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:28 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4182056290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2"}]': finished 2022-09-10T04:24:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:28 smithi104 ceph-mon[97044]: osdmap e65: 10 total, 9 up, 10 in 2022-09-10T04:24:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:28 smithi081 ceph-mon[90133]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 108 KiB/s, 0 objects/s recovering 2022-09-10T04:24:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:28 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4182056290' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2"}]: dispatch 2022-09-10T04:24:28.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:28 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4182056290' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2"}]': finished 2022-09-10T04:24:28.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:28 smithi081 ceph-mon[90133]: osdmap e65: 10 total, 9 up, 10 in 2022-09-10T04:24:28.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:29 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3995635688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:24:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:29 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3995635688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:24:29.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:29 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3995635688' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:24:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:30 smithi110 ceph-mon[97633]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:24:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:30 smithi104 ceph-mon[97044]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:24:30.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:30 smithi081 ceph-mon[90133]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:24:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:32 smithi110 ceph-mon[97633]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2022-09-10T04:24:32.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:32 smithi081 ceph-mon[90133]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2022-09-10T04:24:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:32 smithi104 ceph-mon[97044]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2022-09-10T04:24:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:34 smithi110 ceph-mon[97633]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2022-09-10T04:24:34.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:34 smithi081 ceph-mon[90133]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2022-09-10T04:24:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:34 smithi104 ceph-mon[97044]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2022-09-10T04:24:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:36 smithi110 ceph-mon[97633]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:24:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:36 smithi104 ceph-mon[97044]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:24:36.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:36 smithi081 ceph-mon[90133]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2022-09-10T04:24:37.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-09-10T04:24:37.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:37.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:37 smithi081 ceph-mon[90133]: Deploying daemon osd.9 on smithi081 2022-09-10T04:24:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-09-10T04:24:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:37 smithi110 ceph-mon[97633]: Deploying daemon osd.9 on smithi081 2022-09-10T04:24:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2022-09-10T04:24:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:37 smithi104 ceph-mon[97044]: Deploying daemon osd.9 on smithi081 2022-09-10T04:24:38.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:38 smithi081 ceph-mon[90133]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:38.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:38 smithi110 ceph-mon[97633]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:38 smithi104 ceph-mon[97044]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:40 smithi081 ceph-mon[90133]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:40.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:40 smithi110 ceph-mon[97633]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:40 smithi104 ceph-mon[97044]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:42 smithi110 ceph-mon[97633]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:42 smithi104 ceph-mon[97044]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:42 smithi081 ceph-mon[90133]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:44 smithi110 ceph-mon[97633]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:44 smithi104 ceph-mon[97044]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:44 smithi081 ceph-mon[90133]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:46 smithi081 ceph-mon[90133]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:46.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:46.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:46 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:46 smithi110 ceph-mon[97633]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:46 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:46 smithi104 ceph-mon[97044]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:46 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:47 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:47 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:47 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:47.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:47.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:47.729 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:47 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:47 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:47 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:47 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:48 smithi110 ceph-mon[97633]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:48 smithi104 ceph-mon[97044]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:48 smithi081 ceph-mon[90133]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:50 smithi104 ceph-mon[97044]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:50 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:50 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:50 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:50 smithi081 ceph-mon[90133]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:50 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:50 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:50 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:50 smithi110 ceph-mon[97633]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:50 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:50 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:50 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:51 smithi110 ceph-mon[97633]: from='osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-10T04:24:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:51 smithi110 ceph-mon[97633]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-10T04:24:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:51 smithi104 ceph-mon[97044]: from='osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-10T04:24:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:51 smithi104 ceph-mon[97044]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-10T04:24:51.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:51 smithi081 ceph-mon[90133]: from='osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-10T04:24:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:51 smithi081 ceph-mon[90133]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2022-09-10T04:24:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:52 smithi110 ceph-mon[97633]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:52.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:52 smithi110 ceph-mon[97633]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-09-10T04:24:52.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:52 smithi110 ceph-mon[97633]: osdmap e66: 10 total, 9 up, 10 in 2022-09-10T04:24:52.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:52.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:52 smithi110 ceph-mon[97633]: from='osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:24:52.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:52 smithi110 ceph-mon[97633]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:24:52.677 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:52 smithi081 ceph-mon[90133]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:52.677 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:52 smithi081 ceph-mon[90133]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-09-10T04:24:52.678 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:52 smithi081 ceph-mon[90133]: osdmap e66: 10 total, 9 up, 10 in 2022-09-10T04:24:52.678 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:52.678 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:52 smithi081 ceph-mon[90133]: from='osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:24:52.679 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:52 smithi081 ceph-mon[90133]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:24:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:52 smithi104 ceph-mon[97044]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:52 smithi104 ceph-mon[97044]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2022-09-10T04:24:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:52 smithi104 ceph-mon[97044]: osdmap e66: 10 total, 9 up, 10 in 2022-09-10T04:24:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:52 smithi104 ceph-mon[97044]: from='osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:24:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:52 smithi104 ceph-mon[97044]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2022-09-10T04:24:53.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.119+0000 7fae6b7fe700 1 -- 172.21.15.81:0/659968915 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fae7c0676a0 con 0x7fae64099090 2022-09-10T04:24:53.125 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 9 on host 'smithi081' 2022-09-10T04:24:53.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fae64099090 msgr2=0x7fae6409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:53.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fae64099090 0x7fae6409b540 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fae7c197180 tx=0x7fae70005ea0 comp rx=0 tx=0).stop 2022-09-10T04:24:53.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fcd50 msgr2=0x7fae7c195c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:53.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fcd50 0x7fae7c195c80 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c00f690 comp rx=0 tx=0).stop 2022-09-10T04:24:53.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 shutdown_connections 2022-09-10T04:24:53.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fae64099090 0x7fae6409b540 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:53.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae7c0fe960 0x7fae7c19d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:53.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fcd50 0x7fae7c195c80 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:53.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 --2- 172.21.15.81:0/659968915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae7c0fdfd0 0x7fae7c1961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:53.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 >> 172.21.15.81:0/659968915 conn(0x7fae7c0f8460 msgr2=0x7fae7c066f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:24:53.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.122+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 shutdown_connections 2022-09-10T04:24:53.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:24:53.123+0000 7fae80b17700 1 -- 172.21.15.81:0/659968915 wait complete. 2022-09-10T04:24:53.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:24:53.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: osdmap e67: 10 total, 9 up, 10 in 2022-09-10T04:24:53.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:53.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:53.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:53.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:53.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:53.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:53.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:53 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:53.642 DEBUG:teuthology.orchestra.run.smithi081:osd.9> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.9.service 2022-09-10T04:24:53.649 INFO:tasks.cephadm:Deploying osd.10 on smithi104 with /dev/vg_nvme/lv_1... 2022-09-10T04:24:53.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-09-10T04:24:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:24:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: osdmap e67: 10 total, 9 up, 10 in 2022-09-10T04:24:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:53.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:53.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:53 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:53.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2022-09-10T04:24:53.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: osdmap e67: 10 total, 9 up, 10 in 2022-09-10T04:24:53.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:53.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:53.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:53.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:53.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:24:53.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:53.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:53 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:24:53.738 INFO:journalctl@ceph.osd.9.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:24:54.246 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:24:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:54 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:24:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:54 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:24:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:54 smithi081 ceph-mon[90133]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:54 smithi081 ceph-mon[90133]: osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] boot 2022-09-10T04:24:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:54 smithi081 ceph-mon[90133]: osdmap e68: 10 total, 10 up, 10 in 2022-09-10T04:24:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:54.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:54 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:24:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:54 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:24:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:54 smithi110 ceph-mon[97633]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:54 smithi110 ceph-mon[97633]: osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] boot 2022-09-10T04:24:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:54 smithi110 ceph-mon[97633]: osdmap e68: 10 total, 10 up, 10 in 2022-09-10T04:24:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:54.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:54 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:24:54.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:54 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:24:54.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:54 smithi104 ceph-mon[97044]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2022-09-10T04:24:54.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:54 smithi104 ceph-mon[97044]: osd.9 [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] boot 2022-09-10T04:24:54.682 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:54 smithi104 ceph-mon[97044]: osdmap e68: 10 total, 10 up, 10 in 2022-09-10T04:24:54.682 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:24:55.857 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-09-10T04:24:55.858 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-09-10T04:24:55.858 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2022-09-10T04:24:55.859 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2022-09-10T04:24:55.859 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112221 s, 934 MB/s 2022-09-10T04:24:55.860 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2022-09-10T04:24:56.275 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2022-09-10T04:24:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:56 smithi110 ceph-mon[97633]: osdmap e69: 10 total, 10 up, 10 in 2022-09-10T04:24:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:56 smithi110 ceph-mon[97633]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:24:56.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:56 smithi081 ceph-mon[90133]: osdmap e69: 10 total, 10 up, 10 in 2022-09-10T04:24:56.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:56 smithi081 ceph-mon[90133]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:24:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:56 smithi104 ceph-mon[97044]: osdmap e69: 10 total, 10 up, 10 in 2022-09-10T04:24:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:56 smithi104 ceph-mon[97044]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:24:56.909 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.b/config 2022-09-10T04:24:58.089 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 -- 172.21.15.104:0/3518405266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 msgr2=0x7f2a2c071110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:58.090 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/3518405266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c071110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009810 tx=0x7f2a20009b20 comp rx=0 tx=0).stop 2022-09-10T04:24:58.091 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 -- 172.21.15.104:0/3518405266 shutdown_connections 2022-09-10T04:24:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/3518405266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a2c105d40 0x7f2a2c108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:58.092 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/3518405266 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c071110 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/3518405266 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a2c071650 0x7f2a2c071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 -- 172.21.15.104:0/3518405266 >> 172.21.15.104:0/3518405266 conn(0x7f2a2c0f9540 msgr2=0x7f2a2c0fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:24:58.093 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 -- 172.21.15.104:0/3518405266 shutdown_connections 2022-09-10T04:24:58.094 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.083+0000 7f2a31d8d700 1 -- 172.21.15.104:0/3518405266 wait complete. 2022-09-10T04:24:58.094 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 Processor -- start 2022-09-10T04:24:58.095 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 -- start start 2022-09-10T04:24:58.095 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a2c071650 0x7f2a2c195c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:58.095 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c1961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a2c105d40 0x7f2a2c19d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:58.096 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c19d840 con 0x7f2a2c105d40 2022-09-10T04:24:58.097 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c19d9b0 con 0x7f2a2c072c30 2022-09-10T04:24:58.097 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a31d8d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c19db70 con 0x7f2a2c071650 2022-09-10T04:24:58.098 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c1961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:58.098 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c1961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:59218/0 (socket says 172.21.15.104:59218) 2022-09-10T04:24:58.099 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 -- 172.21.15.104:0/1002112455 learned_addr learned my addr 172.21.15.104:0/1002112455 (peer_addr_for_me v2:172.21.15.104:0/0) 2022-09-10T04:24:58.099 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2b7fe700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a2c071650 0x7f2a2c195c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:58.099 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 -- 172.21.15.104:0/1002112455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a2c071650 msgr2=0x7f2a2c195c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a2c071650 0x7f2a2c195c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:58.100 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 -- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a2c105d40 msgr2=0x7f2a2c19d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:24:58.101 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a2c105d40 0x7f2a2c19d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:24:58.101 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.084+0000 7f2a2affd700 1 -- 172.21.15.104:0/1002112455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20009470 con 0x7f2a2c072c30 2022-09-10T04:24:58.101 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.085+0000 7f2a2affd700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c1961d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00b3c0 tx=0x7f2a1c00b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:24:58.102 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.085+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c011830 con 0x7f2a2c072c30 2022-09-10T04:24:58.102 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.085+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2a1c011e30 con 0x7f2a2c072c30 2022-09-10T04:24:58.103 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.085+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c19de50 con 0x7f2a2c072c30 2022-09-10T04:24:58.103 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.085+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c00f2e0 con 0x7f2a2c072c30 2022-09-10T04:24:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.085+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c19e3a0 con 0x7f2a2c072c30 2022-09-10T04:24:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.086+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a2c10b870 con 0x7f2a2c072c30 2022-09-10T04:24:58.104 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.088+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f2a1c021070 con 0x7f2a2c072c30 2022-09-10T04:24:58.106 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.088+0000 7f2a28ff9700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a14099090 0x7f2a1409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:24:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.088+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6257+0+0 (secure 0 0 0) 0x7f2a1c0c18b0 con 0x7f2a2c072c30 2022-09-10T04:24:58.107 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.089+0000 7f2a2b7fe700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a14099090 0x7f2a1409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:24:58.108 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.089+0000 7f2a2b7fe700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a14099090 0x7f2a1409b540 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009760 tx=0x7f2a20005910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:24:58.108 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.091+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f2a1c08e7f0 con 0x7f2a2c072c30 2022-09-10T04:24:58.282 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:24:58.281+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a2c05fbb0 con 0x7f2a14099090 2022-09-10T04:24:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:24:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi081 2022-09-10T04:24:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi081 to 4317M 2022-09-10T04:24:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:58.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:58.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:58 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:24:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi081 2022-09-10T04:24:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi081 to 4317M 2022-09-10T04:24:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:59.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:59.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:58 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:24:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi081 2022-09-10T04:24:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:24:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi081 to 4317M 2022-09-10T04:24:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:24:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:24:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:24:59.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:58 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:24:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:24:59 smithi104 ceph-mon[97044]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:24:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:24:59 smithi110 ceph-mon[97633]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:00.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:24:59 smithi081 ceph-mon[90133]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:00 smithi110 ceph-mon[97633]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:00 smithi104 ceph-mon[97044]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:00 smithi081 ceph-mon[90133]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:01 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/3408264319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]: dispatch 2022-09-10T04:25:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:01 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]: dispatch 2022-09-10T04:25:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:01 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]': finished 2022-09-10T04:25:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:01 smithi110 ceph-mon[97633]: osdmap e70: 11 total, 10 up, 11 in 2022-09-10T04:25:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:01 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:01 smithi110 ceph-mon[97633]: from='client.? 172.21.15.104:0/2501543005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:25:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:01 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/3408264319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]: dispatch 2022-09-10T04:25:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:01 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]: dispatch 2022-09-10T04:25:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:01 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]': finished 2022-09-10T04:25:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:01 smithi104 ceph-mon[97044]: osdmap e70: 11 total, 10 up, 11 in 2022-09-10T04:25:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:01 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:01 smithi104 ceph-mon[97044]: from='client.? 172.21.15.104:0/2501543005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:25:02.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:01 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/3408264319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]: dispatch 2022-09-10T04:25:02.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:01 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]: dispatch 2022-09-10T04:25:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:01 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60318b87-7ef8-423c-8f41-1554cd1b4e9a"}]': finished 2022-09-10T04:25:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:01 smithi081 ceph-mon[90133]: osdmap e70: 11 total, 10 up, 11 in 2022-09-10T04:25:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:01 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:02.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:01 smithi081 ceph-mon[90133]: from='client.? 172.21.15.104:0/2501543005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:25:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:02 smithi110 ceph-mon[97633]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:02 smithi104 ceph-mon[97044]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:03.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:02 smithi081 ceph-mon[90133]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:04.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:04 smithi110 ceph-mon[97633]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:04.476 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:04 smithi104 ceph-mon[97044]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:04 smithi081 ceph-mon[90133]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:06 smithi110 ceph-mon[97633]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:06.786 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:06 smithi104 ceph-mon[97044]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:06 smithi081 ceph-mon[90133]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-09-10T04:25:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-09-10T04:25:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2022-09-10T04:25:07.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:08.449 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:08 smithi104 ceph-mon[97044]: Deploying daemon osd.10 on smithi104 2022-09-10T04:25:08.449 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:08 smithi104 ceph-mon[97044]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:08 smithi110 ceph-mon[97633]: Deploying daemon osd.10 on smithi104 2022-09-10T04:25:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:08 smithi110 ceph-mon[97633]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:08 smithi081 ceph-mon[90133]: Deploying daemon osd.10 on smithi104 2022-09-10T04:25:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:08 smithi081 ceph-mon[90133]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:09.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:25:09.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:25:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:25:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:25:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:25:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:25:10.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:10 smithi110 ceph-mon[97633]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:10 smithi104 ceph-mon[97044]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:10.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:10 smithi081 ceph-mon[90133]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:12.625 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:12 smithi104 ceph-mon[97044]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:12 smithi110 ceph-mon[97633]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:12 smithi081 ceph-mon[90133]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:14 smithi104 ceph-mon[97044]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:14 smithi110 ceph-mon[97633]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:14 smithi081 ceph-mon[90133]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:16 smithi104 ceph-mon[97044]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:16.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:16 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.624 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.622+0000 7f2a28ff9700 1 -- 172.21.15.104:0/1002112455 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f2a2c05fbb0 con 0x7f2a14099090 2022-09-10T04:25:16.625 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 10 on host 'smithi104' 2022-09-10T04:25:16.626 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a14099090 msgr2=0x7f2a1409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:16.626 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a14099090 0x7f2a1409b540 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009760 tx=0x7f2a20005910 comp rx=0 tx=0).stop 2022-09-10T04:25:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 msgr2=0x7f2a2c1961d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c1961d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00b3c0 tx=0x7f2a1c00b780 comp rx=0 tx=0).stop 2022-09-10T04:25:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 shutdown_connections 2022-09-10T04:25:16.627 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a2c105d40 0x7f2a2c19d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a14099090 0x7f2a1409b540 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a2c071650 0x7f2a2c195c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 --2- 172.21.15.104:0/1002112455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a2c072c30 0x7f2a2c1961d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:16.628 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 >> 172.21.15.104:0/1002112455 conn(0x7f2a2c0f9540 msgr2=0x7f2a2c0fb920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:16.629 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 shutdown_connections 2022-09-10T04:25:16.629 INFO:teuthology.orchestra.run.smithi104.stderr:2022-09-10T04:25:16.624+0000 7f2a31d8d700 1 -- 172.21.15.104:0/1002112455 wait complete. 2022-09-10T04:25:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:16 smithi110 ceph-mon[97633]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:16.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:16 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:16 smithi081 ceph-mon[90133]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:16.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:16.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:16 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.076 DEBUG:teuthology.orchestra.run.smithi104:osd.10> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.10.service 2022-09-10T04:25:17.080 INFO:tasks.cephadm:Deploying osd.11 on smithi110 with /dev/vg_nvme/lv_1... 2022-09-10T04:25:17.081 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2022-09-10T04:25:17.109 INFO:journalctl@ceph.osd.10.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:25:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-10T04:25:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-10T04:25:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:17.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:17.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:17 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-10T04:25:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-10T04:25:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:17 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:17.681 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:25:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-10T04:25:17.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2022-09-10T04:25:17.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:17.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:17.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:17.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:17 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:18.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-mon[97044]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:18.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-mon[97044]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-09-10T04:25:18.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-mon[97044]: osdmap e71: 11 total, 10 up, 11 in 2022-09-10T04:25:18.621 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:18.621 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-mon[97044]: from='osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:25:18.621 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-mon[97044]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:25:18.621 INFO:journalctl@ceph.osd.10.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-10[119903]: 2022-09-10T04:25:18.342+0000 7fda31b9b700 -1 osd.10 0 waiting for initial osdmap 2022-09-10T04:25:18.622 INFO:journalctl@ceph.osd.10.smithi104.stdout:Sep 10 04:25:18 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-10[119903]: 2022-09-10T04:25:18.357+0000 7fda2c18d700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:25:18.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:18 smithi110 ceph-mon[97633]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:18.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:18 smithi110 ceph-mon[97633]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-09-10T04:25:18.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:18 smithi110 ceph-mon[97633]: osdmap e71: 11 total, 10 up, 11 in 2022-09-10T04:25:18.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:18 smithi110 ceph-mon[97633]: from='osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:25:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:18 smithi110 ceph-mon[97633]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:25:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:18 smithi081 ceph-mon[90133]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:18 smithi081 ceph-mon[90133]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2022-09-10T04:25:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:18 smithi081 ceph-mon[90133]: osdmap e71: 11 total, 10 up, 11 in 2022-09-10T04:25:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:18 smithi081 ceph-mon[90133]: from='osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:25:18.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:18 smithi081 ceph-mon[90133]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2022-09-10T04:25:19.170 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/vg_nvme/lv_1 2022-09-10T04:25:19.170 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2022-09-10T04:25:19.170 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2022-09-10T04:25:19.170 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2022-09-10T04:25:19.171 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00828117 s, 1.3 GB/s 2022-09-10T04:25:19.171 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 2022-09-10T04:25:19.171 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2022-09-10T04:25:19.452 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:25:19.453 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: osdmap e72: 11 total, 10 up, 11 in 2022-09-10T04:25:19.453 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.453 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.454 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] boot 2022-09-10T04:25:19.454 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: osdmap e73: 11 total, 11 up, 11 in 2022-09-10T04:25:19.455 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:19 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.571 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:25:19.571 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: osdmap e72: 11 total, 10 up, 11 in 2022-09-10T04:25:19.572 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.572 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.572 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] boot 2022-09-10T04:25:19.572 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: osdmap e73: 11 total, 11 up, 11 in 2022-09-10T04:25:19.572 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:19 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.605 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch daemon add osd smithi110:vg_nvme/lv_1 2022-09-10T04:25:19.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2022-09-10T04:25:19.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: osdmap e72: 11 total, 10 up, 11 in 2022-09-10T04:25:19.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: osd.10 [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] boot 2022-09-10T04:25:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: osdmap e73: 11 total, 11 up, 11 in 2022-09-10T04:25:19.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:19 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:25:20.202 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:25:20.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:25:20.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:25:20.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:20.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi104 2022-09-10T04:25:20.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi104 to 4292M 2022-09-10T04:25:20.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:20 smithi110 ceph-mon[97633]: osdmap e74: 11 total, 11 up, 11 in 2022-09-10T04:25:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:25:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:25:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi104 2022-09-10T04:25:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi104 to 4292M 2022-09-10T04:25:20.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:20 smithi104 ceph-mon[97044]: osdmap e74: 11 total, 11 up, 11 in 2022-09-10T04:25:20.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:25:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:25:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2022-09-10T04:25:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi104 2022-09-10T04:25:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:20.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi104 to 4292M 2022-09-10T04:25:20.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:20.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:20 smithi081 ceph-mon[90133]: osdmap e74: 11 total, 11 up, 11 in 2022-09-10T04:25:21.379 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/3721295605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140ff060 msgr2=0x7f3d140ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:21.380 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/3721295605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140ff060 0x7f3d140ff4d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009810 tx=0x7f3d04009b20 comp rx=0 tx=0).stop 2022-09-10T04:25:21.380 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/3721295605 shutdown_connections 2022-09-10T04:25:21.380 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/3721295605 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d140ffa10 0x7f3d14106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:21.381 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/3721295605 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d140fde80 0x7f3d140fe290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:21.381 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/3721295605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140ff060 0x7f3d140ff4d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:21.381 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/3721295605 >> 172.21.15.110:0/3721295605 conn(0x7f3d140f9550 msgr2=0x7f3d140fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:21.381 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/3721295605 shutdown_connections 2022-09-10T04:25:21.381 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/3721295605 wait complete. 2022-09-10T04:25:21.382 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 Processor -- start 2022-09-10T04:25:21.382 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.377+0000 7f3d1a8c3700 1 -- start start 2022-09-10T04:25:21.382 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 0x7f3d14072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:21.382 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d140ff060 0x7f3d140710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:21.383 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d140ffa10 0x7f3d14071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:21.383 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d14071e10 con 0x7f3d140ff060 2022-09-10T04:25:21.383 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d14071f80 con 0x7f3d140ffa10 2022-09-10T04:25:21.383 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1419dc10 con 0x7f3d140fde80 2022-09-10T04:25:21.384 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 0x7f3d14072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:21.384 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 0x7f3d14072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60976/0 (socket says 172.21.15.110:60976) 2022-09-10T04:25:21.384 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 -- 172.21.15.110:0/1190014156 learned_addr learned my addr 172.21.15.110:0/1190014156 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:25:21.384 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 -- 172.21.15.110:0/1190014156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d140ffa10 msgr2=0x7f3d14071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:21.385 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d18e60700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d140ffa10 0x7f3d14071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:21.385 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d140ffa10 0x7f3d14071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:21.385 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 -- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d140ff060 msgr2=0x7f3d140710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:21.385 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d140ff060 0x7f3d140710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:21.385 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 -- 172.21.15.110:0/1190014156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d04009470 con 0x7f3d140fde80 2022-09-10T04:25:21.386 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d13fff700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 0x7f3d14072a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800b3c0 tx=0x7f3d0800b780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:21.386 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08011830 con 0x7f3d140fde80 2022-09-10T04:25:21.386 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1419def0 con 0x7f3d140fde80 2022-09-10T04:25:21.386 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3d08011e30 con 0x7f3d140fde80 2022-09-10T04:25:21.387 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0800f2e0 con 0x7f3d140fde80 2022-09-10T04:25:21.387 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.378+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1419e2f0 con 0x7f3d140fde80 2022-09-10T04:25:21.388 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.379+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d14109750 con 0x7f3d140fde80 2022-09-10T04:25:21.388 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.380+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3d08021070 con 0x7f3d140fde80 2022-09-10T04:25:21.388 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.381+0000 7f3d117fa700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3cfc099010 0x7f3cfc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:21.389 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.381+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f3d080c1d80 con 0x7f3d140fde80 2022-09-10T04:25:21.389 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.381+0000 7f3d137fe700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3cfc099010 0x7f3cfc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:21.389 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.382+0000 7f3d137fe700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3cfc099010 0x7f3cfc09b4c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009810 tx=0x7f3d04000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:21.389 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.384+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3d08051c80 con 0x7f3d140fde80 2022-09-10T04:25:21.577 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:21.576+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d1405fbd0 con 0x7f3cfc099010 2022-09-10T04:25:22.601 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:22 smithi110 ceph-mon[97633]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:22.601 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:22 smithi110 ceph-mon[97633]: osdmap e75: 11 total, 11 up, 11 in 2022-09-10T04:25:22.602 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:22 smithi110 ceph-mon[97633]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:22.602 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:25:22.602 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:25:22.602 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:22 smithi104 ceph-mon[97044]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:22 smithi104 ceph-mon[97044]: osdmap e75: 11 total, 11 up, 11 in 2022-09-10T04:25:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:22 smithi104 ceph-mon[97044]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:25:22.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:25:22.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:22 smithi081 ceph-mon[90133]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:22 smithi081 ceph-mon[90133]: osdmap e75: 11 total, 11 up, 11 in 2022-09-10T04:25:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:22 smithi081 ceph-mon[90133]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:22.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2022-09-10T04:25:22.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2022-09-10T04:25:22.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:23 smithi110 ceph-mon[97633]: osdmap e76: 11 total, 11 up, 11 in 2022-09-10T04:25:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:23 smithi104 ceph-mon[97044]: osdmap e76: 11 total, 11 up, 11 in 2022-09-10T04:25:23.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:23 smithi081 ceph-mon[90133]: osdmap e76: 11 total, 11 up, 11 in 2022-09-10T04:25:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: pgmap v253: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-09-10T04:25:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: osdmap e77: 11 total, 11 up, 11 in 2022-09-10T04:25:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/1792675059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]: dispatch 2022-09-10T04:25:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]: dispatch 2022-09-10T04:25:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]': finished 2022-09-10T04:25:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: osdmap e78: 12 total, 11 up, 12 in 2022-09-10T04:25:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: pgmap v253: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-09-10T04:25:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: osdmap e77: 11 total, 11 up, 11 in 2022-09-10T04:25:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/1792675059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]: dispatch 2022-09-10T04:25:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]: dispatch 2022-09-10T04:25:24.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]': finished 2022-09-10T04:25:24.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: osdmap e78: 12 total, 11 up, 12 in 2022-09-10T04:25:24.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:24.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: pgmap v253: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-09-10T04:25:24.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: osdmap e77: 11 total, 11 up, 11 in 2022-09-10T04:25:24.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/1792675059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]: dispatch 2022-09-10T04:25:24.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]: dispatch 2022-09-10T04:25:24.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da2d73f1-bba2-4c62-9463-ca7aa22701d8"}]': finished 2022-09-10T04:25:24.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: osdmap e78: 12 total, 11 up, 12 in 2022-09-10T04:25:24.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:25 smithi110 ceph-mon[97633]: from='client.? 172.21.15.110:0/168483187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:25:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:25 smithi104 ceph-mon[97044]: from='client.? 172.21.15.110:0/168483187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:25:25.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:25 smithi081 ceph-mon[90133]: from='client.? 172.21.15.110:0/168483187' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2022-09-10T04:25:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:26 smithi110 ceph-mon[97633]: pgmap v256: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-09-10T04:25:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:26 smithi104 ceph-mon[97044]: pgmap v256: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-09-10T04:25:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:26 smithi081 ceph-mon[90133]: pgmap v256: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2022-09-10T04:25:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:28 smithi110 ceph-mon[97633]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:25:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:28 smithi104 ceph-mon[97044]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:25:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:28 smithi081 ceph-mon[90133]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2022-09-10T04:25:30.593 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:30 smithi110 ceph-mon[97633]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:30 smithi104 ceph-mon[97044]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:30.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:30 smithi081 ceph-mon[90133]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:31.620 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-09-10T04:25:31.621 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:31 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:31.621 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:31 smithi110 ceph-mon[97633]: Deploying daemon osd.11 on smithi110 2022-09-10T04:25:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-09-10T04:25:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:31 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:31 smithi104 ceph-mon[97044]: Deploying daemon osd.11 on smithi110 2022-09-10T04:25:31.813 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2022-09-10T04:25:31.814 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:31 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:31.814 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:31 smithi081 ceph-mon[90133]: Deploying daemon osd.11 on smithi110 2022-09-10T04:25:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:32 smithi110 ceph-mon[97633]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:32 smithi104 ceph-mon[97044]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:32.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:32 smithi081 ceph-mon[90133]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:34 smithi110 ceph-mon[97633]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:34.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:34 smithi104 ceph-mon[97044]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:34 smithi081 ceph-mon[90133]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:36 smithi110 ceph-mon[97633]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:36.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:36.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:36.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:36 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:36 smithi104 ceph-mon[97044]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:36 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:36 smithi081 ceph-mon[90133]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:36 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:38 smithi110 ceph-mon[97633]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:38 smithi104 ceph-mon[97044]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:38.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:38 smithi081 ceph-mon[90133]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:40.087 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.087 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.088 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:39 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:39 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:39 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:40.340 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.338+0000 7f3d117fa700 1 -- 172.21.15.110:0/1190014156 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f3d1405fbd0 con 0x7f3cfc099010 2022-09-10T04:25:40.341 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 11 on host 'smithi110' 2022-09-10T04:25:40.342 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3cfc099010 msgr2=0x7f3cfc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:40.342 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3cfc099010 0x7f3cfc09b4c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009810 tx=0x7f3d04000bc0 comp rx=0 tx=0).stop 2022-09-10T04:25:40.343 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 msgr2=0x7f3d14072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:40.343 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 0x7f3d14072a90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800b3c0 tx=0x7f3d0800b780 comp rx=0 tx=0).stop 2022-09-10T04:25:40.343 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 shutdown_connections 2022-09-10T04:25:40.343 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d140ffa10 0x7f3d14071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:40.344 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3cfc099010 0x7f3cfc09b4c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:40.344 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d140fde80 0x7f3d14072a90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:40.344 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 --2- 172.21.15.110:0/1190014156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d140ff060 0x7f3d140710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:40.344 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 >> 172.21.15.110:0/1190014156 conn(0x7f3d140f9550 msgr2=0x7f3d141026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:40.345 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 shutdown_connections 2022-09-10T04:25:40.345 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:40.340+0000 7f3d1a8c3700 1 -- 172.21.15.110:0/1190014156 wait complete. 2022-09-10T04:25:40.753 DEBUG:teuthology.orchestra.run.smithi110:osd.11> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@osd.11.service 2022-09-10T04:25:40.756 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2022-09-10T04:25:40.756 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd stat -f json 2022-09-10T04:25:40.786 INFO:journalctl@ceph.osd.11.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:25:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-10T04:25:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-10T04:25:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:40 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-10T04:25:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-10T04:25:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:41.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:41.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:41.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:41.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:40 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:41.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-10T04:25:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2022-09-10T04:25:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:41.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:41.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:41.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:41.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:40 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:41.635 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:25:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-mon[97633]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-09-10T04:25:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-mon[97633]: osdmap e79: 12 total, 11 up, 12 in 2022-09-10T04:25:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:42.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-mon[97633]: from='osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:25:42.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-mon[97633]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:25:42.159 INFO:journalctl@ceph.osd.11.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-11[121123]: 2022-09-10T04:25:41.876+0000 7f5e9bae7700 -1 osd.11 0 waiting for initial osdmap 2022-09-10T04:25:42.160 INFO:journalctl@ceph.osd.11.smithi110.stdout:Sep 10 04:25:41 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-osd-11[121123]: 2022-09-10T04:25:41.897+0000 7f5e960d9700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-09-10T04:25:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:41 smithi104 ceph-mon[97044]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-09-10T04:25:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:41 smithi104 ceph-mon[97044]: osdmap e79: 12 total, 11 up, 12 in 2022-09-10T04:25:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:41 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:41 smithi104 ceph-mon[97044]: from='osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:25:42.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:41 smithi104 ceph-mon[97044]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:25:42.298 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:41 smithi081 ceph-mon[90133]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2022-09-10T04:25:42.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:41 smithi081 ceph-mon[90133]: osdmap e79: 12 total, 11 up, 12 in 2022-09-10T04:25:42.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:41 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:42.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:41 smithi081 ceph-mon[90133]: from='osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:25:42.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:41 smithi081 ceph-mon[90133]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2022-09-10T04:25:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:42 smithi110 ceph-mon[97633]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:42 smithi110 ceph-mon[97633]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:25:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:42 smithi110 ceph-mon[97633]: osdmap e80: 12 total, 11 up, 12 in 2022-09-10T04:25:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:42 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:43.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:42 smithi104 ceph-mon[97044]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:43.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:42 smithi104 ceph-mon[97044]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:25:43.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:42 smithi104 ceph-mon[97044]: osdmap e80: 12 total, 11 up, 12 in 2022-09-10T04:25:43.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:43.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:42 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:43.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 -- 172.21.15.81:0/374077532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 msgr2=0x7f6c70100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:43.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 --2- 172.21.15.81:0/374077532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c70100a00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c60009810 tx=0x7f6c60009b20 comp rx=0 tx=0).stop 2022-09-10T04:25:43.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 -- 172.21.15.81:0/374077532 shutdown_connections 2022-09-10T04:25:43.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 --2- 172.21.15.81:0/374077532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c70103a30 0x7f6c70105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 --2- 172.21.15.81:0/374077532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c70100a00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 --2- 172.21.15.81:0/374077532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c70100f40 0x7f6c701033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 -- 172.21.15.81:0/374077532 >> 172.21.15.81:0/374077532 conn(0x7f6c700f8430 msgr2=0x7f6c700fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:43.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 -- 172.21.15.81:0/374077532 shutdown_connections 2022-09-10T04:25:43.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.202+0000 7f6c7485e700 1 -- 172.21.15.81:0/374077532 wait complete. 2022-09-10T04:25:43.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.204+0000 7f6c7485e700 1 Processor -- start 2022-09-10T04:25:43.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.204+0000 7f6c7485e700 1 -- start start 2022-09-10T04:25:43.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c7485e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c700fe360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:43.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c7485e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c70100f40 0x7f6c700fc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:43.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c7485e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c70103a30 0x7f6c700fcf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:43.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c7485e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c700fd6e0 con 0x7f6c70103a30 2022-09-10T04:25:43.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c7485e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c700fd850 con 0x7f6c70100f40 2022-09-10T04:25:43.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c7485e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7019db60 con 0x7f6c700673d0 2022-09-10T04:25:43.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c6e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c700fe360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:43.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c6e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c700fe360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:50882/0 (socket says 172.21.15.81:50882) 2022-09-10T04:25:43.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c6e59c700 1 -- 172.21.15.81:0/675933895 learned_addr learned my addr 172.21.15.81:0/675933895 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:43.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c6dd9b700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c70100f40 0x7f6c700fc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:43.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.205+0000 7f6c6ed9d700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c70103a30 0x7f6c700fcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:43.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c6e59c700 1 -- 172.21.15.81:0/675933895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c70100f40 msgr2=0x7f6c700fc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:43.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c6e59c700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c70100f40 0x7f6c700fc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c6e59c700 1 -- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c70103a30 msgr2=0x7f6c700fcf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:43.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c6e59c700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c70103a30 0x7f6c700fcf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c6e59c700 1 -- 172.21.15.81:0/675933895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c60009470 con 0x7f6c700673d0 2022-09-10T04:25:43.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c6e59c700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c700fe360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c60000c00 tx=0x7f6c6000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:43.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6001d070 con 0x7f6c700673d0 2022-09-10T04:25:43.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6c6000f930 con 0x7f6c700673d0 2022-09-10T04:25:43.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7019dde0 con 0x7f6c700673d0 2022-09-10T04:25:43.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.206+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c600047d0 con 0x7f6c700673d0 2022-09-10T04:25:43.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.207+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7019e220 con 0x7f6c700673d0 2022-09-10T04:25:43.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.208+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c70109670 con 0x7f6c700673d0 2022-09-10T04:25:43.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.209+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6c60028030 con 0x7f6c700673d0 2022-09-10T04:25:43.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.210+0000 7f6c5f7fe700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c58099090 0x7f6c5809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:43.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.211+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6c600c2b50 con 0x7f6c700673d0 2022-09-10T04:25:43.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.211+0000 7f6c6dd9b700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c58099090 0x7f6c5809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:43.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.211+0000 7f6c6dd9b700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c58099090 0x7f6c5809b540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c64005fd0 tx=0x7f6c64005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:43.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.213+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6c6008f640 con 0x7f6c700673d0 2022-09-10T04:25:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:42 smithi081 ceph-mon[90133]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:42 smithi081 ceph-mon[90133]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2022-09-10T04:25:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:42 smithi081 ceph-mon[90133]: osdmap e80: 12 total, 11 up, 12 in 2022-09-10T04:25:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:42 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:43.427 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.426+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f6c700f9e20 con 0x7f6c700673d0 2022-09-10T04:25:43.428 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.426+0000 7f6c5f7fe700 1 -- 172.21.15.81:0/675933895 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f6c60092c60 con 0x7f6c700673d0 2022-09-10T04:25:43.428 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:25:43.430 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.428+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c58099090 msgr2=0x7f6c5809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:43.430 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.428+0000 7f6c7485e700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c58099090 0x7f6c5809b540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c64005fd0 tx=0x7f6c64005ea0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.428+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 msgr2=0x7f6c700fe360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:43.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.428+0000 7f6c7485e700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c700fe360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c60000c00 tx=0x7f6c6000f690 comp rx=0 tx=0).stop 2022-09-10T04:25:43.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 shutdown_connections 2022-09-10T04:25:43.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c70103a30 0x7f6c700fcf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c58099090 0x7f6c5809b540 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c700673d0 0x7f6c700fe360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 --2- 172.21.15.81:0/675933895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c70100f40 0x7f6c700fc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:43.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 >> 172.21.15.81:0/675933895 conn(0x7f6c700f8430 msgr2=0x7f6c700ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:43.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 shutdown_connections 2022-09-10T04:25:43.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:43.429+0000 7f6c7485e700 1 -- 172.21.15.81:0/675933895 wait complete. 2022-09-10T04:25:43.809 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1662783942,"num_in_osds":12,"osd_in_since":1662783924,"num_remapped_pgs":0} 2022-09-10T04:25:43.811 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd dump --format=json 2022-09-10T04:25:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: purged_snaps scrub starts 2022-09-10T04:25:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: purged_snaps scrub ok 2022-09-10T04:25:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:44.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] boot 2022-09-10T04:25:44.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: osdmap e81: 12 total, 12 up, 12 in 2022-09-10T04:25:44.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:44.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:43 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/675933895' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-09-10T04:25:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: purged_snaps scrub starts 2022-09-10T04:25:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: purged_snaps scrub ok 2022-09-10T04:25:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] boot 2022-09-10T04:25:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: osdmap e81: 12 total, 12 up, 12 in 2022-09-10T04:25:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:43 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/675933895' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-09-10T04:25:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: purged_snaps scrub starts 2022-09-10T04:25:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: purged_snaps scrub ok 2022-09-10T04:25:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: osd.11 [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] boot 2022-09-10T04:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: osdmap e81: 12 total, 12 up, 12 in 2022-09-10T04:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:25:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:43 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/675933895' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2022-09-10T04:25:44.685 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:25:45.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:44 smithi081 ceph-mon[90133]: Detected new or changed devices on smithi110 2022-09-10T04:25:45.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:44 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi110 to 4292M 2022-09-10T04:25:45.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:44 smithi081 ceph-mon[90133]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:45.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:44 smithi081 ceph-mon[90133]: osdmap e82: 12 total, 12 up, 12 in 2022-09-10T04:25:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:44 smithi110 ceph-mon[97633]: Detected new or changed devices on smithi110 2022-09-10T04:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:44 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi110 to 4292M 2022-09-10T04:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:44 smithi110 ceph-mon[97633]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:44 smithi110 ceph-mon[97633]: osdmap e82: 12 total, 12 up, 12 in 2022-09-10T04:25:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:44 smithi104 ceph-mon[97044]: Detected new or changed devices on smithi110 2022-09-10T04:25:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:44 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi110 to 4292M 2022-09-10T04:25:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:44 smithi104 ceph-mon[97044]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2022-09-10T04:25:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:44 smithi104 ceph-mon[97044]: osdmap e82: 12 total, 12 up, 12 in 2022-09-10T04:25:46.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 -- 172.21.15.81:0/3552119904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac100c70 msgr2=0x7f13ac104cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 --2- 172.21.15.81:0/3552119904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac100c70 0x7f13ac104cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f139c009840 tx=0x7f139c009b50 comp rx=0 tx=0).stop 2022-09-10T04:25:46.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 -- 172.21.15.81:0/3552119904 shutdown_connections 2022-09-10T04:25:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 --2- 172.21.15.81:0/3552119904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac1053d0 0x7f13ac10cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 --2- 172.21.15.81:0/3552119904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac1006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 --2- 172.21.15.81:0/3552119904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac100c70 0x7f13ac104cc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 -- 172.21.15.81:0/3552119904 >> 172.21.15.81:0/3552119904 conn(0x7f13ac0fbbf0 msgr2=0x7f13ac0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:46.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 -- 172.21.15.81:0/3552119904 shutdown_connections 2022-09-10T04:25:46.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.212+0000 7f13b4747700 1 -- 172.21.15.81:0/3552119904 wait complete. 2022-09-10T04:25:46.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.214+0000 7f13b4747700 1 Processor -- start 2022-09-10T04:25:46.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.214+0000 7f13b4747700 1 -- start start 2022-09-10T04:25:46.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b4747700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:46.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b4747700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 0x7f13ac196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:46.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b4747700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac1053d0 0x7f13ac19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:46.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b4747700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac19af20 con 0x7f13ac100c70 2022-09-10T04:25:46.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b4747700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac19b090 con 0x7f13ac1053d0 2022-09-10T04:25:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b4747700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac19b250 con 0x7f13ac1002d0 2022-09-10T04:25:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b1ce2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 0x7f13ac196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b24e3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b1ce2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 0x7f13ac196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34646/0 (socket says 172.21.15.81:34646) 2022-09-10T04:25:46.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b24e3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:50888/0 (socket says 172.21.15.81:50888) 2022-09-10T04:25:46.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b1ce2700 1 -- 172.21.15.81:0/2583918181 learned_addr learned my addr 172.21.15.81:0/2583918181 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.215+0000 7f13b2ce4700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac1053d0 0x7f13ac19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b24e3700 1 -- 172.21.15.81:0/2583918181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac1053d0 msgr2=0x7f13ac19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b24e3700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac1053d0 0x7f13ac19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b24e3700 1 -- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 msgr2=0x7f13ac196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b24e3700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 0x7f13ac196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b24e3700 1 -- 172.21.15.81:0/2583918181 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c009470 con 0x7f13ac1002d0 2022-09-10T04:25:46.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b1ce2700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 0x7f13ac196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:25:46.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b24e3700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac195ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13a800e6c0 tx=0x7f13a800e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:46.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8010900 con 0x7f13ac1002d0 2022-09-10T04:25:46.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac19b530 con 0x7f13ac1002d0 2022-09-10T04:25:46.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f13a8018420 con 0x7f13ac1002d0 2022-09-10T04:25:46.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.216+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a8017610 con 0x7f13ac1002d0 2022-09-10T04:25:46.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.217+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac1a2d70 con 0x7f13ac1002d0 2022-09-10T04:25:46.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.218+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac064ea0 con 0x7f13ac1002d0 2022-09-10T04:25:46.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.219+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f13a800c040 con 0x7f13ac1002d0 2022-09-10T04:25:46.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.220+0000 7f13a37fe700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1398099010 0x7f139809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:46.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.221+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f13a8014070 con 0x7f13ac1002d0 2022-09-10T04:25:46.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.221+0000 7f13b1ce2700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1398099010 0x7f139809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:46.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.221+0000 7f13b1ce2700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1398099010 0x7f139809b4c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f139c00b600 tx=0x7f139c005930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:46.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.225+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f13a804fc80 con 0x7f13ac1002d0 2022-09-10T04:25:46.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.442+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13ac1a3380 con 0x7f13ac1002d0 2022-09-10T04:25:46.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.443+0000 7f13a37fe700 1 -- 172.21.15.81:0/2583918181 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18112 (secure 0 0 0) 0x7f13a808eaf0 con 0x7f13ac1002d0 2022-09-10T04:25:46.445 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:25:46.446 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":82,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:25:44.149785+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:25:46.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1398099010 msgr2=0x7f139809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:46.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1398099010 0x7f139809b4c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f139c00b600 tx=0x7f139c005930 comp rx=0 tx=0).stop 2022-09-10T04:25:46.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 msgr2=0x7f13ac195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:46.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac195ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13a800e6c0 tx=0x7f13a800e9d0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 shutdown_connections 2022-09-10T04:25:46.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13ac1053d0 0x7f13ac19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1398099010 0x7f139809b4c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13ac1002d0 0x7f13ac195ef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 --2- 172.21.15.81:0/2583918181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13ac100c70 0x7f13ac196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:46.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.445+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 >> 172.21.15.81:0/2583918181 conn(0x7f13ac0fbbf0 msgr2=0x7f13ac0fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:46.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.446+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 shutdown_connections 2022-09-10T04:25:46.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:46.446+0000 7f13b4747700 1 -- 172.21.15.81:0/2583918181 wait complete. 2022-09-10T04:25:46.863 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-09-10T04:25:46.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd pool get .mgr pg_num 2022-09-10T04:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:46 smithi110 ceph-mon[97633]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:46 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2583918181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:25:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:46 smithi104 ceph-mon[97044]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:46 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2583918181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:25:47.335 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:46 smithi081 ceph-mon[90133]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:47.335 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:46 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2583918181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:25:47.738 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:25:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:48 smithi110 ceph-mon[97633]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 -- 172.21.15.81:0/562576102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6870102fa0 msgr2=0x7f687010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:49.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 --2- 172.21.15.81:0/562576102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6870102fa0 0x7f687010ab40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6868009be0 tx=0x7f6868009ef0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 -- 172.21.15.81:0/562576102 shutdown_connections 2022-09-10T04:25:49.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 --2- 172.21.15.81:0/562576102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6870102fa0 0x7f687010ab40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 --2- 172.21.15.81:0/562576102 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68700fdea0 0x7f68700fe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 --2- 172.21.15.81:0/562576102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fe840 0x7f6870102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 -- 172.21.15.81:0/562576102 >> 172.21.15.81:0/562576102 conn(0x7f68700f9800 msgr2=0x7f68700fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 -- 172.21.15.81:0/562576102 shutdown_connections 2022-09-10T04:25:49.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.267+0000 7f68753d1700 1 -- 172.21.15.81:0/562576102 wait complete. 2022-09-10T04:25:49.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.269+0000 7f68753d1700 1 Processor -- start 2022-09-10T04:25:49.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.269+0000 7f68753d1700 1 -- start start 2022-09-10T04:25:49.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f68753d1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fdea0 0x7f68701998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:49.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f68753d1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68700fe840 0x7f6870199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:49.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f68753d1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6870102fa0 0x7f6870193950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:49.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f68753d1700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870194030 con 0x7f68700fe840 2022-09-10T04:25:49.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f68753d1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701941a0 con 0x7f6870102fa0 2022-09-10T04:25:49.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f68753d1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6870194360 con 0x7f68700fdea0 2022-09-10T04:25:49.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f686f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68700fe840 0x7f6870199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:49.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f686ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fdea0 0x7f68701998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:49.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f686f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68700fe840 0x7f6870199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34662/0 (socket says 172.21.15.81:34662) 2022-09-10T04:25:49.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f686f7fe700 1 -- 172.21.15.81:0/2884277745 learned_addr learned my addr 172.21.15.81:0/2884277745 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:49.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.270+0000 7f6874bd0700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6870102fa0 0x7f6870193950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:49.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686ffff700 1 -- 172.21.15.81:0/2884277745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6870102fa0 msgr2=0x7f6870193950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:49.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686ffff700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6870102fa0 0x7f6870193950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686ffff700 1 -- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68700fe840 msgr2=0x7f6870199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:49.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686ffff700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68700fe840 0x7f6870199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686ffff700 1 -- 172.21.15.81:0/2884277745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68680097e0 con 0x7f68700fdea0 2022-09-10T04:25:49.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f6874bd0700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6870102fa0 0x7f6870193950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:25:49.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686ffff700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fdea0 0x7f68701998e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f686000e6c0 tx=0x7f686000e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:49.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860010900 con 0x7f68700fdea0 2022-09-10T04:25:49.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6870194640 con 0x7f68700fdea0 2022-09-10T04:25:49.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6860018420 con 0x7f68700fdea0 2022-09-10T04:25:49.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860017610 con 0x7f68700fdea0 2022-09-10T04:25:49.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.271+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68701a0930 con 0x7f68700fdea0 2022-09-10T04:25:49.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.272+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687004d480 con 0x7f68700fdea0 2022-09-10T04:25:49.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.274+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f686000c040 con 0x7f68700fdea0 2022-09-10T04:25:49.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.276+0000 7f686d7fa700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6858099010 0x7f685809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:49.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.276+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6860014070 con 0x7f68700fdea0 2022-09-10T04:25:49.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.276+0000 7f686f7fe700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6858099010 0x7f685809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:49.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.277+0000 7f686f7fe700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6858099010 0x7f685809b4c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f68701950b0 tx=0x7f6864005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:49.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.278+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f686008ed00 con 0x7f68700fdea0 2022-09-10T04:25:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:48 smithi104 ceph-mon[97044]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:49.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:48 smithi081 ceph-mon[90133]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:49.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.492+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f687010daf0 con 0x7f68700fdea0 2022-09-10T04:25:49.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.493+0000 7f686d7fa700 1 -- 172.21.15.81:0/2884277745 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6860092320 con 0x7f68700fdea0 2022-09-10T04:25:49.495 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:25:49.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6858099010 msgr2=0x7f685809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:49.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6858099010 0x7f685809b4c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f68701950b0 tx=0x7f6864005ea0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fdea0 msgr2=0x7f68701998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:49.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fdea0 0x7f68701998e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f686000e6c0 tx=0x7f686000e9d0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 shutdown_connections 2022-09-10T04:25:49.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6870102fa0 0x7f6870193950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6858099010 0x7f685809b4c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68700fdea0 0x7f68701998e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 --2- 172.21.15.81:0/2884277745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68700fe840 0x7f6870199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:49.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 >> 172.21.15.81:0/2884277745 conn(0x7f68700f9800 msgr2=0x7f68700fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:49.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 shutdown_connections 2022-09-10T04:25:49.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:49.496+0000 7f68753d1700 1 -- 172.21.15.81:0/2884277745 wait complete. 2022-09-10T04:25:49.916 INFO:tasks.cephadm:Adding mds.a on smithi081 2022-09-10T04:25:49.917 INFO:tasks.cephadm:Adding mds.d on smithi081 2022-09-10T04:25:49.917 INFO:tasks.cephadm:Adding mds.g on smithi081 2022-09-10T04:25:49.918 INFO:tasks.cephadm:Adding mds.j on smithi081 2022-09-10T04:25:49.919 INFO:tasks.cephadm:Adding mds.b on smithi104 2022-09-10T04:25:49.919 INFO:tasks.cephadm:Adding mds.e on smithi104 2022-09-10T04:25:49.920 INFO:tasks.cephadm:Adding mds.h on smithi104 2022-09-10T04:25:49.920 INFO:tasks.cephadm:Adding mds.k on smithi104 2022-09-10T04:25:49.920 INFO:tasks.cephadm:Adding mds.c on smithi110 2022-09-10T04:25:49.921 INFO:tasks.cephadm:Adding mds.f on smithi110 2022-09-10T04:25:49.921 INFO:tasks.cephadm:Adding mds.i on smithi110 2022-09-10T04:25:49.921 INFO:tasks.cephadm:Adding mds.l on smithi110 2022-09-10T04:25:49.922 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph orch apply mds all '12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l' 2022-09-10T04:25:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:49 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2884277745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:25:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:49 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2884277745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:25:50.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:49 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2884277745' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:25:50.522 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.c/config 2022-09-10T04:25:51.027 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:50 smithi110 ceph-mon[97633]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:50 smithi104 ceph-mon[97044]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:51.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:50 smithi081 ceph-mon[90133]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:51.670 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 -- 172.21.15.110:0/1662829824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fe7e0 msgr2=0x7f1a641050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:51.671 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 --2- 172.21.15.110:0/1662829824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fe7e0 0x7f1a641050a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009bb0 tx=0x7f1a5c009ec0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.671 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 -- 172.21.15.110:0/1662829824 shutdown_connections 2022-09-10T04:25:51.671 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 --2- 172.21.15.110:0/1662829824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fe7e0 0x7f1a641050a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.672 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 --2- 172.21.15.110:0/1662829824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a640fcbd0 0x7f1a640fcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 --2- 172.21.15.110:0/1662829824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a640fde50 0x7f1a640fe2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 -- 172.21.15.110:0/1662829824 >> 172.21.15.110:0/1662829824 conn(0x7f1a640f82a0 msgr2=0x7f1a640fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:51.673 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 -- 172.21.15.110:0/1662829824 shutdown_connections 2022-09-10T04:25:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 -- 172.21.15.110:0/1662829824 wait complete. 2022-09-10T04:25:51.674 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.662+0000 7f1a68da0700 1 Processor -- start 2022-09-10T04:25:51.675 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 -- start start 2022-09-10T04:25:51.675 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a640fcbd0 0x7f1a64100bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:51.676 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 0x7f1a640ff230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:51.676 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a640fe7e0 0x7f1a640ff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:51.677 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640fff70 con 0x7f1a640fcbd0 2022-09-10T04:25:51.677 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a641000e0 con 0x7f1a640fe7e0 2022-09-10T04:25:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a68da0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6419dab0 con 0x7f1a640fde50 2022-09-10T04:25:51.678 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 0x7f1a640ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 0x7f1a640ff230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:32868/0 (socket says 172.21.15.110:32868) 2022-09-10T04:25:51.679 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 -- 172.21.15.110:0/3023269906 learned_addr learned my addr 172.21.15.110:0/3023269906 (peer_addr_for_me v2:172.21.15.110:0/0) 2022-09-10T04:25:51.680 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 -- 172.21.15.110:0/3023269906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a640fe7e0 msgr2=0x7f1a640ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:51.680 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a640fe7e0 0x7f1a640ff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.680 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 -- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a640fcbd0 msgr2=0x7f1a64100bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:51.681 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a637fe700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a640fcbd0 0x7f1a64100bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:51.681 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a640fcbd0 0x7f1a64100bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 -- 172.21.15.110:0/3023269906 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0097e0 con 0x7f1a640fde50 2022-09-10T04:25:51.682 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a62ffd700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 0x7f1a640ff230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400e7c0 tx=0x7f1a5400eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.663+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54005220 con 0x7f1a640fde50 2022-09-10T04:25:51.683 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.664+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6419dd90 con 0x7f1a640fde50 2022-09-10T04:25:51.684 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.664+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1a54005380 con 0x7f1a640fde50 2022-09-10T04:25:51.684 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.664+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a54004c90 con 0x7f1a640fde50 2022-09-10T04:25:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.664+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6419e250 con 0x7f1a640fde50 2022-09-10T04:25:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.665+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a641083d0 con 0x7f1a640fde50 2022-09-10T04:25:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.665+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1a5401e020 con 0x7f1a640fde50 2022-09-10T04:25:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.666+0000 7f1a60ff9700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1a4c099010 0x7f1a4c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:51.685 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.666+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1a54014070 con 0x7f1a640fde50 2022-09-10T04:25:51.686 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.666+0000 7f1a637fe700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1a4c099010 0x7f1a4c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:51.686 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.667+0000 7f1a637fe700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1a4c099010 0x7f1a4c09b4c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1a640fdcb0 tx=0x7f1a58008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:51.687 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.670+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1a5408eda0 con 0x7f1a640fde50 2022-09-10T04:25:51.862 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.860+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a640678a0 con 0x7f1a4c099010 2022-09-10T04:25:51.870 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.869+0000 7f1a60ff9700 1 -- 172.21.15.110:0/3023269906 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f1a640678a0 con 0x7f1a4c099010 2022-09-10T04:25:51.871 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mds.all update... 2022-09-10T04:25:51.873 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1a4c099010 msgr2=0x7f1a4c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:51.873 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1a4c099010 0x7f1a4c09b4c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1a640fdcb0 tx=0x7f1a58008040 comp rx=0 tx=0).stop 2022-09-10T04:25:51.874 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 msgr2=0x7f1a640ff230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:51.874 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 0x7f1a640ff230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400e7c0 tx=0x7f1a5400eb80 comp rx=0 tx=0).stop 2022-09-10T04:25:51.875 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 shutdown_connections 2022-09-10T04:25:51.875 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1a640fe7e0 0x7f1a640ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.876 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a640fcbd0 0x7f1a64100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.876 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1a4c099010 0x7f1a4c09b4c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.877 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 --2- 172.21.15.110:0/3023269906 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1a640fde50 0x7f1a640ff230 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:51.877 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 >> 172.21.15.110:0/3023269906 conn(0x7f1a640f82a0 msgr2=0x7f1a64067090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:51.877 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 shutdown_connections 2022-09-10T04:25:51.878 INFO:teuthology.orchestra.run.smithi110.stderr:2022-09-10T04:25:51.871+0000 7f1a68da0700 1 -- 172.21.15.110:0/3023269906 wait complete. 2022-09-10T04:25:52.246 DEBUG:teuthology.orchestra.run.smithi081:mds.a> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.a.service 2022-09-10T04:25:52.255 DEBUG:teuthology.orchestra.run.smithi081:mds.d> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.d.service 2022-09-10T04:25:52.302 DEBUG:teuthology.orchestra.run.smithi081:mds.g> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.g.service 2022-09-10T04:25:52.350 DEBUG:teuthology.orchestra.run.smithi081:mds.j> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.j.service 2022-09-10T04:25:52.351 INFO:journalctl@ceph.mds.a.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:25:52.397 DEBUG:teuthology.orchestra.run.smithi104:mds.b> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.b.service 2022-09-10T04:25:52.399 INFO:journalctl@ceph.mds.d.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:25:52.410 DEBUG:teuthology.orchestra.run.smithi104:mds.e> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.e.service 2022-09-10T04:25:52.432 INFO:journalctl@ceph.mds.b.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:25:52.453 DEBUG:teuthology.orchestra.run.smithi104:mds.h> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.h.service 2022-09-10T04:25:52.456 INFO:journalctl@ceph.mds.g.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:25:52.460 INFO:journalctl@ceph.mds.e.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:25:52.466 DEBUG:teuthology.orchestra.run.smithi104:mds.k> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.k.service 2022-09-10T04:25:52.487 INFO:journalctl@ceph.mds.j.smithi081.stdout:-- Logs begin at Sat 2022-09-10 04:01:02 UTC. -- 2022-09-10T04:25:52.497 INFO:journalctl@ceph.mds.h.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:25:52.499 DEBUG:teuthology.orchestra.run.smithi110:mds.c> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.c.service 2022-09-10T04:25:52.512 DEBUG:teuthology.orchestra.run.smithi110:mds.f> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.f.service 2022-09-10T04:25:52.517 INFO:journalctl@ceph.mds.k.smithi104.stdout:-- Logs begin at Sat 2022-09-10 03:57:08 UTC. -- 2022-09-10T04:25:52.541 INFO:journalctl@ceph.mds.c.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:25:52.549 DEBUG:teuthology.orchestra.run.smithi110:mds.i> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.i.service 2022-09-10T04:25:52.571 INFO:journalctl@ceph.mds.f.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:25:52.583 DEBUG:teuthology.orchestra.run.smithi110:mds.l> sudo journalctl -f -n 0 -u ceph-70cf895c-30bf-11ed-8431-001a4aab830c@mds.l.service 2022-09-10T04:25:52.611 INFO:journalctl@ceph.mds.i.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:25:52.613 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2022-09-10T04:25:52.624 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:25:52.649 INFO:journalctl@ceph.mds.l.smithi110.stdout:-- Logs begin at Sat 2022-09-10 03:56:58 UTC. -- 2022-09-10T04:25:52.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 -- 172.21.15.81:0/2959245427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c1031d0 msgr2=0x7faf1c1055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:52.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 --2- 172.21.15.81:0/2959245427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c1031d0 0x7faf1c1055b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf10009750 tx=0x7faf10009a60 comp rx=0 tx=0).stop 2022-09-10T04:25:52.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 -- 172.21.15.81:0/2959245427 shutdown_connections 2022-09-10T04:25:52.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 --2- 172.21.15.81:0/2959245427 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf1c105af0 0x7faf1c107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:52.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 --2- 172.21.15.81:0/2959245427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf1c066bf0 0x7faf1c102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:52.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 --2- 172.21.15.81:0/2959245427 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c1031d0 0x7faf1c1055b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:52.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 -- 172.21.15.81:0/2959245427 >> 172.21.15.81:0/2959245427 conn(0x7faf1c0f82a0 msgr2=0x7faf1c0fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:52.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 -- 172.21.15.81:0/2959245427 shutdown_connections 2022-09-10T04:25:52.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.896+0000 7faf21fff700 1 -- 172.21.15.81:0/2959245427 wait complete. 2022-09-10T04:25:52.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.897+0000 7faf21fff700 1 Processor -- start 2022-09-10T04:25:52.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.897+0000 7faf21fff700 1 -- start start 2022-09-10T04:25:52.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf21fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 0x7faf1c195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:52.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf21fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf1c1031d0 0x7faf1c1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:52.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf21fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf1c105af0 0x7faf1c19d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:52.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf21fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c19d6e0 con 0x7faf1c1031d0 2022-09-10T04:25:52.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf21fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c19d850 con 0x7faf1c105af0 2022-09-10T04:25:52.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf21fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1c19da10 con 0x7faf1c066bf0 2022-09-10T04:25:52.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 0x7faf1c195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:52.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1affd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf1c1031d0 0x7faf1c1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:52.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 0x7faf1c195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36398/0 (socket says 172.21.15.81:36398) 2022-09-10T04:25:52.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 -- 172.21.15.81:0/2122645524 learned_addr learned my addr 172.21.15.81:0/2122645524 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:52.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1bfff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf1c105af0 0x7faf1c19d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:52.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 -- 172.21.15.81:0/2122645524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf1c105af0 msgr2=0x7faf1c19d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:52.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf1c105af0 0x7faf1c19d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:52.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 -- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf1c1031d0 msgr2=0x7faf1c1960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:52.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.898+0000 7faf1b7fe700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf1c1031d0 0x7faf1c1960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:52.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf1b7fe700 1 -- 172.21.15.81:0/2122645524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf100093b0 con 0x7faf1c066bf0 2022-09-10T04:25:52.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf1b7fe700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 0x7faf1c195b60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00e790 tx=0x7faf0c00eaa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:52.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c009510 con 0x7faf1c066bf0 2022-09-10T04:25:52.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf1c19dcf0 con 0x7faf1c066bf0 2022-09-10T04:25:52.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7faf0c010da0 con 0x7faf1c066bf0 2022-09-10T04:25:52.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c0173d0 con 0x7faf1c066bf0 2022-09-10T04:25:52.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.899+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf1c19e1a0 con 0x7faf1c066bf0 2022-09-10T04:25:52.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.900+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1c10b770 con 0x7faf1c066bf0 2022-09-10T04:25:52.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.902+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7faf0c00c040 con 0x7faf1c066bf0 2022-09-10T04:25:52.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.903+0000 7faf18ff9700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf04099090 0x7faf0409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:52.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.904+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7faf0c014070 con 0x7faf1c066bf0 2022-09-10T04:25:52.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.904+0000 7faf1affd700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf04099090 0x7faf0409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:52.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.904+0000 7faf1affd700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf04099090 0x7faf0409b540 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faf1000f500 tx=0x7faf10005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:52.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:52.910+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7faf0c08eb80 con 0x7faf1c066bf0 2022-09-10T04:25:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:53.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:53.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: Saving service mds.all spec with placement smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l;count:12 2022-09-10T04:25:53.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:53.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:53.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:53.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.066 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:53.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:53.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:52 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:53.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.102+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf1c19e7b0 con 0x7faf1c066bf0 2022-09-10T04:25:53.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.104+0000 7faf18ff9700 1 -- 172.21.15.81:0/2122645524 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18112 (secure 0 0 0) 0x7faf0c0921a0 con 0x7faf1c066bf0 2022-09-10T04:25:53.110 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:25:53.110 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":82,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:25:44.149785+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:25:53.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.106+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf04099090 msgr2=0x7faf0409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.106+0000 7faf21fff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf04099090 0x7faf0409b540 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faf1000f500 tx=0x7faf10005f90 comp rx=0 tx=0).stop 2022-09-10T04:25:53.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.106+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 msgr2=0x7faf1c195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.106+0000 7faf21fff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 0x7faf1c195b60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00e790 tx=0x7faf0c00eaa0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 shutdown_connections 2022-09-10T04:25:53.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf1c105af0 0x7faf1c19d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf04099090 0x7faf0409b540 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf1c066bf0 0x7faf1c195b60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 --2- 172.21.15.81:0/2122645524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf1c1031d0 0x7faf1c1960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 >> 172.21.15.81:0/2122645524 conn(0x7faf1c0f82a0 msgr2=0x7faf1c104630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:53.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 shutdown_connections 2022-09-10T04:25:53.118 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.107+0000 7faf21fff700 1 -- 172.21.15.81:0/2122645524 wait complete. 2022-09-10T04:25:53.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-09-10T04:25:53.142 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:25:53.149 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:53.149 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:53.150 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: Saving service mds.all spec with placement smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l;count:12 2022-09-10T04:25:53.150 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.150 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:53.150 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:53.151 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:53.151 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.151 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.151 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:53.151 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:53.152 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:52 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:25:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: Saving service mds.all spec with placement smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l;count:12 2022-09-10T04:25:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:25:53.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:53.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:25:53.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:53.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:53.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:53.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:52 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:53.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 -- 172.21.15.81:0/1955728311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fdd00 msgr2=0x7f14a80fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 --2- 172.21.15.81:0/1955728311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fdd00 0x7f14a80fe110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14a4009810 tx=0x7f14a4009b20 comp rx=0 tx=0).stop 2022-09-10T04:25:53.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 -- 172.21.15.81:0/1955728311 shutdown_connections 2022-09-10T04:25:53.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 --2- 172.21.15.81:0/1955728311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 0x7f14a81061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 --2- 172.21.15.81:0/1955728311 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fdd00 0x7f14a80fe110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 --2- 172.21.15.81:0/1955728311 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fef80 0x7f14a80ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 -- 172.21.15.81:0/1955728311 >> 172.21.15.81:0/1955728311 conn(0x7f14a80f93f0 msgr2=0x7f14a80fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 -- 172.21.15.81:0/1955728311 shutdown_connections 2022-09-10T04:25:53.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.377+0000 7f14af7ff700 1 -- 172.21.15.81:0/1955728311 wait complete. 2022-09-10T04:25:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.378+0000 7f14af7ff700 1 Processor -- start 2022-09-10T04:25:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 -- start start 2022-09-10T04:25:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 0x7f14a8195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fef80 0x7f14a8196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 0x7f14a819d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a819d6d0 con 0x7f14a80ff910 2022-09-10T04:25:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a819d840 con 0x7f14a80fef80 2022-09-10T04:25:53.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14af7ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a819da00 con 0x7f14a80fdd00 2022-09-10T04:25:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14ad59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 0x7f14a8195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14add9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 0x7f14a819d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14ad59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 0x7f14a8195b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36414/0 (socket says 172.21.15.81:36414) 2022-09-10T04:25:53.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14ad59b700 1 -- 172.21.15.81:0/2727016237 learned_addr learned my addr 172.21.15.81:0/2727016237 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.379+0000 7f14acd9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fef80 0x7f14a8196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14ad59b700 1 -- 172.21.15.81:0/2727016237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fef80 msgr2=0x7f14a8196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14ad59b700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fef80 0x7f14a8196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14ad59b700 1 -- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 msgr2=0x7f14a819d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14ad59b700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 0x7f14a819d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14ad59b700 1 -- 172.21.15.81:0/2727016237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4009470 con 0x7f14a80fdd00 2022-09-10T04:25:53.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14add9c700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 0x7f14a819d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:25:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14ad59b700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 0x7f14a8195b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006010 tx=0x7f14a400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a401d070 con 0x7f14a80fdd00 2022-09-10T04:25:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a819dc80 con 0x7f14a80fdd00 2022-09-10T04:25:53.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f14a400f8b0 con 0x7f14a80fdd00 2022-09-10T04:25:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40047f0 con 0x7f14a80fdd00 2022-09-10T04:25:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.380+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a819e130 con 0x7f14a80fdd00 2022-09-10T04:25:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.381+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a81094b0 con 0x7f14a80fdd00 2022-09-10T04:25:53.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.383+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f14a4028030 con 0x7f14a80fdd00 2022-09-10T04:25:53.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.384+0000 7f149e7fc700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1494099010 0x7f149409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.384+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f14a40c2bf0 con 0x7f14a80fdd00 2022-09-10T04:25:53.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.384+0000 7f14acd9a700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1494099010 0x7f149409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.385+0000 7f14acd9a700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1494099010 0x7f149409b4c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1498005fd0 tx=0x7f1498005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:53.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.388+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f14a408f6e0 con 0x7f14a80fdd00 2022-09-10T04:25:53.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.581+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14a819e990 con 0x7f14a80fdd00 2022-09-10T04:25:53.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.582+0000 7f149e7fc700 1 -- 172.21.15.81:0/2727016237 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14a4092d00 con 0x7f14a80fdd00 2022-09-10T04:25:53.585 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:25:53.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.584+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1494099010 msgr2=0x7f149409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.584+0000 7f14af7ff700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1494099010 0x7f149409b4c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1498005fd0 tx=0x7f1498005ea0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.584+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 msgr2=0x7f14a8195b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.584+0000 7f14af7ff700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 0x7f14a8195b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006010 tx=0x7f14a400f690 comp rx=0 tx=0).stop 2022-09-10T04:25:53.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 shutdown_connections 2022-09-10T04:25:53.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a80ff910 0x7f14a819d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1494099010 0x7f149409b4c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14a80fdd00 0x7f14a8195b50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 --2- 172.21.15.81:0/2727016237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f14a80fef80 0x7f14a8196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 >> 172.21.15.81:0/2727016237 conn(0x7f14a80f93f0 msgr2=0x7f14a80fb680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:53.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 shutdown_connections 2022-09-10T04:25:53.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.585+0000 7f14af7ff700 1 -- 172.21.15.81:0/2727016237 wait complete. 2022-09-10T04:25:53.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:25:53.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 -- 172.21.15.81:0/3770344841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 msgr2=0x7f08cc0ff3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 --2- 172.21.15.81:0/3770344841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 0x7f08cc0ff3c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009810 tx=0x7f08c0009b20 comp rx=0 tx=0).stop 2022-09-10T04:25:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 -- 172.21.15.81:0/3770344841 shutdown_connections 2022-09-10T04:25:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 --2- 172.21.15.81:0/3770344841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08cc0ff900 0x7f08cc1061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 --2- 172.21.15.81:0/3770344841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc0fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 --2- 172.21.15.81:0/3770344841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 0x7f08cc0ff3c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 -- 172.21.15.81:0/3770344841 >> 172.21.15.81:0/3770344841 conn(0x7f08cc0f93c0 msgr2=0x7f08cc0fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 -- 172.21.15.81:0/3770344841 shutdown_connections 2022-09-10T04:25:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.845+0000 7f08d25ff700 1 -- 172.21.15.81:0/3770344841 wait complete. 2022-09-10T04:25:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 Processor -- start 2022-09-10T04:25:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 -- start start 2022-09-10T04:25:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 0x7f08cc196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08cc0ff900 0x7f08cc19d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc19d690 con 0x7f08cc0fef70 2022-09-10T04:25:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc19d800 con 0x7f08cc0ff900 2022-09-10T04:25:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08d25ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc19d9c0 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08cbfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08cbfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36442/0 (socket says 172.21.15.81:36442) 2022-09-10T04:25:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.847+0000 7f08cbfff700 1 -- 172.21.15.81:0/746819733 learned_addr learned my addr 172.21.15.81:0/746819733 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cb7fe700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 0x7f08cc196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08d0b9c700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08cc0ff900 0x7f08cc19d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cbfff700 1 -- 172.21.15.81:0/746819733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08cc0ff900 msgr2=0x7f08cc19d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cbfff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08cc0ff900 0x7f08cc19d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cbfff700 1 -- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 msgr2=0x7f08cc196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:53.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cbfff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 0x7f08cc196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:53.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cbfff700 1 -- 172.21.15.81:0/746819733 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c0009470 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08cbfff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc195b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00e710 tx=0x7f08bc00ea20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc004220 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc19dca0 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f08bc004380 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.848+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc003be0 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.849+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc19e1b0 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.849+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc1094a0 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.854+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f08bc01e020 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.856+0000 7f08c97fa700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f08b4099010 0x7f08b409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:53.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.856+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f08bc014070 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.856+0000 7f08cb7fe700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f08b4099010 0x7f08b409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:53.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.856+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f08bc0c2c60 con 0x7f08cc0fdcf0 2022-09-10T04:25:53.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:53.857+0000 7f08cb7fe700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f08b4099010 0x7f08b409b4c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f08c000b4f0 tx=0x7f08c0009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:54.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.096+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08cc19eab0 con 0x7f08cc0fdcf0 2022-09-10T04:25:54.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.097+0000 7f08c97fa700 1 -- 172.21.15.81:0/746819733 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f08bc08f4d0 con 0x7f08cc0fdcf0 2022-09-10T04:25:54.099 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:25:54.099 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2022-09-10T04:25:54.101 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f08b4099010 msgr2=0x7f08b409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:54.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f08b4099010 0x7f08b409b4c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f08c000b4f0 tx=0x7f08c0009eb0 comp rx=0 tx=0).stop 2022-09-10T04:25:54.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 msgr2=0x7f08cc195b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:54.102 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc195b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00e710 tx=0x7f08bc00ea20 comp rx=0 tx=0).stop 2022-09-10T04:25:54.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 shutdown_connections 2022-09-10T04:25:54.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f08cc0ff900 0x7f08cc19d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:54.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f08b4099010 0x7f08b409b4c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:54.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08cc0fdcf0 0x7f08cc195b10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:54.103 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 --2- 172.21.15.81:0/746819733 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08cc0fef70 0x7f08cc196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:54.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 >> 172.21.15.81:0/746819733 conn(0x7f08cc0f93c0 msgr2=0x7f08cc0fb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:54.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 shutdown_connections 2022-09-10T04:25:54.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:54.100+0000 7f08d25ff700 1 -- 172.21.15.81:0/746819733 wait complete. 2022-09-10T04:25:54.105 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 1 2022-09-10T04:25:54.126 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:53 smithi081 ceph-mon[90133]: Deploying daemon mds.l on smithi110 2022-09-10T04:25:54.126 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:53 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2122645524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:25:54.126 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:53 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2727016237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:25:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:53 smithi110 ceph-mon[97633]: Deploying daemon mds.l on smithi110 2022-09-10T04:25:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:53 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2122645524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:25:54.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:53 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2727016237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:25:54.161 INFO:journalctl@ceph.mds.l.smithi110.stdout:Sep 10 04:25:53 smithi110 systemd[1]: Starting Ceph mds.l for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:25:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:53 smithi104 ceph-mon[97044]: Deploying daemon mds.l on smithi110 2022-09-10T04:25:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:53 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2122645524' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:25:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:53 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2727016237' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:25:54.475 INFO:journalctl@ceph.mds.l.smithi110.stdout:Sep 10 04:25:54 smithi110 podman[124322]: 2022-09-10T04:25:54.873 INFO:journalctl@ceph.mds.l.smithi110.stdout:Sep 10 04:25:54 smithi110 bash[124322]: b77c02ce623df62ebb3106b51a8563e0f82737b92baef9184733fc8abee49795 2022-09-10T04:25:54.873 INFO:journalctl@ceph.mds.l.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mds[124362]: -- 172.21.15.110:0/721221963 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56463dbd4820 con 0x56463dba8c00 2022-09-10T04:25:54.873 INFO:journalctl@ceph.mds.l.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-l[124338]: starting mds.l at 2022-09-10T04:25:54.874 INFO:journalctl@ceph.mds.l.smithi110.stdout:Sep 10 04:25:54 smithi110 systemd[1]: Started Ceph mds.l for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:25:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/746819733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:25:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] up:boot 2022-09-10T04:25:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: fsmap 1 up:standby 2022-09-10T04:25:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-10T04:25:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:55.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:55.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:54 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:55.274 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:55.274 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/746819733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:25:55.275 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] up:boot 2022-09-10T04:25:55.275 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: fsmap 1 up:standby 2022-09-10T04:25:55.275 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-10T04:25:55.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:55.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:55.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:55.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:55.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:54 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/746819733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:25:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] up:boot 2022-09-10T04:25:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: fsmap 1 up:standby 2022-09-10T04:25:55.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-10T04:25:55.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:55.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:55.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:55.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:55.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:54 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:56.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:55 smithi104 ceph-mon[97044]: Deploying daemon mds.k on smithi104 2022-09-10T04:25:56.120 INFO:journalctl@ceph.mds.k.smithi104.stdout:Sep 10 04:25:56 smithi104 systemd[1]: Starting Ceph mds.k for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:25:56.130 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:25:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:55 smithi110 ceph-mon[97633]: Deploying daemon mds.k on smithi104 2022-09-10T04:25:56.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:55 smithi081 ceph-mon[90133]: Deploying daemon mds.k on smithi104 2022-09-10T04:25:56.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 -- 172.21.15.81:0/660062575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 msgr2=0x7f6ab4074d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:56.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/660062575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 0x7f6ab4074d50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009750 tx=0x7f6aa4009a60 comp rx=0 tx=0).stop 2022-09-10T04:25:56.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 -- 172.21.15.81:0/660062575 shutdown_connections 2022-09-10T04:25:56.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/660062575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab4075290 0x7f6ab4075b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/660062575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/660062575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 0x7f6ab4074d50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 -- 172.21.15.81:0/660062575 >> 172.21.15.81:0/660062575 conn(0x7f6ab40fb6b0 msgr2=0x7f6ab40fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:56.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 -- 172.21.15.81:0/660062575 shutdown_connections 2022-09-10T04:25:56.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.363+0000 7f6abb3ff700 1 -- 172.21.15.81:0/660062575 wait complete. 2022-09-10T04:25:56.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.364+0000 7f6abb3ff700 1 Processor -- start 2022-09-10T04:25:56.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 -- start start 2022-09-10T04:25:56.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:56.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 0x7f6ab419a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:56.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab4075290 0x7f6ab41a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:56.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41a1a60 con 0x7f6ab40748e0 2022-09-10T04:25:56.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41a1bd0 con 0x7f6ab4073610 2022-09-10T04:25:56.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6abb3ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41a1d90 con 0x7f6ab4075290 2022-09-10T04:25:56.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6ab919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:56.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6ab919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52166/0 (socket says 172.21.15.81:52166) 2022-09-10T04:25:56.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6ab919b700 1 -- 172.21.15.81:0/2449307407 learned_addr learned my addr 172.21.15.81:0/2449307407 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:56.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6ab899a700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 0x7f6ab419a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:56.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.365+0000 7f6ab999c700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab4075290 0x7f6ab41a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:56.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6ab919b700 1 -- 172.21.15.81:0/2449307407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab4075290 msgr2=0x7f6ab41a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:56.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6ab919b700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab4075290 0x7f6ab41a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6ab919b700 1 -- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 msgr2=0x7f6ab419a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:56.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6ab919b700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 0x7f6ab419a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6ab919b700 1 -- 172.21.15.81:0/2449307407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac0097e0 con 0x7f6ab4073610 2022-09-10T04:25:56.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6ab919b700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4199ee0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e740 tx=0x7f6ab000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:56.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0005220 con 0x7f6ab4073610 2022-09-10T04:25:56.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6ab0005380 con 0x7f6ab4073610 2022-09-10T04:25:56.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0004b70 con 0x7f6ab4073610 2022-09-10T04:25:56.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa40093b0 con 0x7f6ab4073610 2022-09-10T04:25:56.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.366+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab41a23a0 con 0x7f6ab4073610 2022-09-10T04:25:56.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.367+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab410fba0 con 0x7f6ab4073610 2022-09-10T04:25:56.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.369+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6ab001e020 con 0x7f6ab4073610 2022-09-10T04:25:56.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.370+0000 7f6aaa7fc700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6aa0099090 0x7f6aa009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:56.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.370+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6ab0014070 con 0x7f6ab4073610 2022-09-10T04:25:56.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.371+0000 7f6ab899a700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6aa0099090 0x7f6aa009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:56.384 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.371+0000 7f6ab899a700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6aa0099090 0x7f6aa009b540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400f500 tx=0x7f6aa4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:56.384 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.372+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6ab008eef0 con 0x7f6ab4073610 2022-09-10T04:25:56.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.607+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ab405f120 con 0x7f6ab4073610 2022-09-10T04:25:56.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.608+0000 7f6aaa7fc700 1 -- 172.21.15.81:0/2449307407 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f6ab0092510 con 0x7f6ab4073610 2022-09-10T04:25:56.610 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:25:56.611 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-09-10T04:25:56.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.610+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6aa0099090 msgr2=0x7f6aa009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:56.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.610+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6aa0099090 0x7f6aa009b540 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400f500 tx=0x7f6aa4005f90 comp rx=0 tx=0).stop 2022-09-10T04:25:56.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.610+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 msgr2=0x7f6ab4199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:56.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.610+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4199ee0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e740 tx=0x7f6ab000ea50 comp rx=0 tx=0).stop 2022-09-10T04:25:56.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 shutdown_connections 2022-09-10T04:25:56.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6aa0099090 0x7f6aa009b540 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ab4075290 0x7f6ab41a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ab4073610 0x7f6ab4199ee0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 --2- 172.21.15.81:0/2449307407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab40748e0 0x7f6ab419a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:56.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 >> 172.21.15.81:0/2449307407 conn(0x7f6ab40fb6b0 msgr2=0x7f6ab40fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:56.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 shutdown_connections 2022-09-10T04:25:56.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:56.611+0000 7f6abb3ff700 1 -- 172.21.15.81:0/2449307407 wait complete. 2022-09-10T04:25:56.618 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 2 2022-09-10T04:25:56.672 INFO:journalctl@ceph.mds.k.smithi104.stdout:Sep 10 04:25:56 smithi104 podman[122837]: 2022-09-10T04:25:56.955 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:56.955 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2449307407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:25:56.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:56.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:56.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:56.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:56.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:56 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:57.046 INFO:journalctl@ceph.mds.k.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mds[122877]: -- 172.21.15.104:0/3555092063 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eaf239c820 con 0x55eaf2371000 2022-09-10T04:25:57.046 INFO:journalctl@ceph.mds.k.smithi104.stdout:Sep 10 04:25:56 smithi104 bash[122837]: 22ffb927d72c64540c66bc1b1a8bbc11680a24ddaea8f94a83e8a5a231e4341d 2022-09-10T04:25:57.047 INFO:journalctl@ceph.mds.k.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-k[122853]: starting mds.k at 2022-09-10T04:25:57.047 INFO:journalctl@ceph.mds.k.smithi104.stdout:Sep 10 04:25:56 smithi104 systemd[1]: Started Ceph mds.k for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:25:57.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:57.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2449307407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:25:57.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:57.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:57.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:57.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:57.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:56 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2449307407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:25:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:25:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:25:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:25:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:56 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:25:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:57 smithi110 ceph-mon[97633]: Deploying daemon mds.j on smithi081 2022-09-10T04:25:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:57 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.104:6832/777476125,v1:172.21.15.104:6833/777476125] up:boot 2022-09-10T04:25:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:57 smithi110 ceph-mon[97633]: fsmap 2 up:standby 2022-09-10T04:25:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:57 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-10T04:25:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:57 smithi104 ceph-mon[97044]: Deploying daemon mds.j on smithi081 2022-09-10T04:25:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:57 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.104:6832/777476125,v1:172.21.15.104:6833/777476125] up:boot 2022-09-10T04:25:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:57 smithi104 ceph-mon[97044]: fsmap 2 up:standby 2022-09-10T04:25:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:57 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-10T04:25:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:57 smithi081 ceph-mon[90133]: Deploying daemon mds.j on smithi081 2022-09-10T04:25:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:57 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.104:6832/777476125,v1:172.21.15.104:6833/777476125] up:boot 2022-09-10T04:25:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:57 smithi081 ceph-mon[90133]: fsmap 2 up:standby 2022-09-10T04:25:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:57 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-10T04:25:58.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:25:59.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 -- 172.21.15.81:0/2414947203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 msgr2=0x7fb9c81106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:59.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/2414947203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81106f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009bb0 tx=0x7fb9c0009ec0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 -- 172.21.15.81:0/2414947203 shutdown_connections 2022-09-10T04:25:59.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/2414947203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81106f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/2414947203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c8070b10 0x7fb9c806f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/2414947203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9c806f8d0 0x7fb9c806fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.063+0000 7fb9cc99d700 1 -- 172.21.15.81:0/2414947203 >> 172.21.15.81:0/2414947203 conn(0x7fb9c806b0f0 msgr2=0x7fb9c806d520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:59.066 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.064+0000 7fb9cc99d700 1 -- 172.21.15.81:0/2414947203 shutdown_connections 2022-09-10T04:25:59.066 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.064+0000 7fb9cc99d700 1 -- 172.21.15.81:0/2414947203 wait complete. 2022-09-10T04:25:59.068 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.065+0000 7fb9cc99d700 1 Processor -- start 2022-09-10T04:25:59.069 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.065+0000 7fb9cc99d700 1 -- start start 2022-09-10T04:25:59.069 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.065+0000 7fb9cc99d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 0x7fb9c819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:59.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.065+0000 7fb9cc99d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9c8070b10 0x7fb9c819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:59.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9cc99d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:59.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9cc99d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c81a5cc0 con 0x7fb9c8070b10 2022-09-10T04:25:59.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9cc99d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c81a5e30 con 0x7fb9c806f8d0 2022-09-10T04:25:59.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9cc99d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c81a5ff0 con 0x7fb9c810e310 2022-09-10T04:25:59.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 0x7fb9c819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:59.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 0x7fb9c819e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52178/0 (socket says 172.21.15.81:52178) 2022-09-10T04:25:59.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 -- 172.21.15.81:0/1778835455 learned_addr learned my addr 172.21.15.81:0/1778835455 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:25:59.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c6ffd700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9c8070b10 0x7fb9c819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:59.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c7fff700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:59.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 -- 172.21.15.81:0/1778835455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 msgr2=0x7fb9c81a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:59.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 -- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9c8070b10 msgr2=0x7fb9c819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:59.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9c8070b10 0x7fb9c819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.073 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c77fe700 1 -- 172.21.15.81:0/1778835455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c00097e0 con 0x7fb9c806f8d0 2022-09-10T04:25:59.073 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.066+0000 7fb9c7fff700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:25:59.073 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.067+0000 7fb9c77fe700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 0x7fb9c819e140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800e6e0 tx=0x7fb9b800e9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:59.073 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.067+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8004220 con 0x7fb9c806f8d0 2022-09-10T04:25:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.067+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c81a62d0 con 0x7fb9c806f8d0 2022-09-10T04:25:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.067+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb9b8004380 con 0x7fb9c806f8d0 2022-09-10T04:25:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.067+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b8003c20 con 0x7fb9c806f8d0 2022-09-10T04:25:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.067+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c81a6830 con 0x7fb9c806f8d0 2022-09-10T04:25:59.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.069+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c8113ca0 con 0x7fb9c806f8d0 2022-09-10T04:25:59.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.070+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb9b801e020 con 0x7fb9c806f8d0 2022-09-10T04:25:59.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.072+0000 7fb9c4ff9700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9b0099090 0x7fb9b009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:25:59.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.072+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb9b8014070 con 0x7fb9c806f8d0 2022-09-10T04:25:59.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.072+0000 7fb9c6ffd700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9b0099090 0x7fb9b009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:25:59.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.073+0000 7fb9c6ffd700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9b0099090 0x7fb9b009b540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc005fd0 tx=0x7fb9bc005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:25:59.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.075+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb9b808ec60 con 0x7fb9c806f8d0 2022-09-10T04:25:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:58 smithi110 ceph-mon[97633]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:58 smithi104 ceph-mon[97044]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:59.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:58 smithi081 ceph-mon[90133]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:25:59.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.345+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c8113e30 con 0x7fb9c806f8d0 2022-09-10T04:25:59.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.345+0000 7fb9c4ff9700 1 -- 172.21.15.81:0/1778835455 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7fb9c8113e30 con 0x7fb9c806f8d0 2022-09-10T04:25:59.346 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:25:59.347 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2022-09-10T04:25:59.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9b0099090 msgr2=0x7fb9b009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:59.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9b0099090 0x7fb9b009b540 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc005fd0 tx=0x7fb9bc005ea0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 msgr2=0x7fb9c819e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:25:59.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 0x7fb9c819e140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800e6e0 tx=0x7fb9b800e9f0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 shutdown_connections 2022-09-10T04:25:59.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9b0099090 0x7fb9b009b540 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9c810e310 0x7fb9c81a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9c806f8d0 0x7fb9c819e140 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 --2- 172.21.15.81:0/1778835455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9c8070b10 0x7fb9c819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:25:59.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.349+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 >> 172.21.15.81:0/1778835455 conn(0x7fb9c806b0f0 msgr2=0x7fb9c8106c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:25:59.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.350+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 shutdown_connections 2022-09-10T04:25:59.353 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:25:59.350+0000 7fb9cc99d700 1 -- 172.21.15.81:0/1778835455 wait complete. 2022-09-10T04:25:59.354 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 3 2022-09-10T04:25:59.805 INFO:journalctl@ceph.mds.j.smithi081.stdout:Sep 10 04:25:59 smithi081 systemd[1]: Starting Ceph mds.j for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:25:59 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1778835455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:00.200 INFO:journalctl@ceph.mds.j.smithi081.stdout:Sep 10 04:25:59 smithi081 podman[123915]: 2022-09-10T04:26:00.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:25:59 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1778835455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:25:59 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1778835455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:00.546 INFO:journalctl@ceph.mds.j.smithi081.stdout:Sep 10 04:26:00 smithi081 bash[123915]: ff524dd3055ad08bd54d85c91cfaaecd525257441c4af465c105a2258fcb90c9 2022-09-10T04:26:00.547 INFO:journalctl@ceph.mds.j.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mds[123955]: -- 172.21.15.81:0/2375978794 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562c2aac2820 con 0x562c2aa96c00 2022-09-10T04:26:00.547 INFO:journalctl@ceph.mds.j.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-j[123931]: starting mds.j at 2022-09-10T04:26:00.547 INFO:journalctl@ceph.mds.j.smithi081.stdout:Sep 10 04:26:00 smithi081 systemd[1]: Started Ceph mds.j for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:00 smithi110 ceph-mon[97633]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:00 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:00 smithi104 ceph-mon[97044]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:01.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:01.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:01.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:00 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:01.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mon[90133]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:01.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:01.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:01.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:01.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:01.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:00 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:01.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:01.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 -- 172.21.15.81:0/2757211477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 msgr2=0x7f5b840fdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:01.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 --2- 172.21.15.81:0/2757211477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 0x7f5b840fdc00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009bb0 tx=0x7f5b7c009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 -- 172.21.15.81:0/2757211477 shutdown_connections 2022-09-10T04:26:01.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 --2- 172.21.15.81:0/2757211477 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 0x7f5b840fdc00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 --2- 172.21.15.81:0/2757211477 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fc490 0x7f5b840fc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 --2- 172.21.15.81:0/2757211477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b840fcde0 0x7f5b840fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 -- 172.21.15.81:0/2757211477 >> 172.21.15.81:0/2757211477 conn(0x7f5b840f8160 msgr2=0x7f5b840fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:01.636 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 -- 172.21.15.81:0/2757211477 shutdown_connections 2022-09-10T04:26:01.636 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.630+0000 7f5b83fff700 1 -- 172.21.15.81:0/2757211477 wait complete. 2022-09-10T04:26:01.643 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.631+0000 7f5b83fff700 1 Processor -- start 2022-09-10T04:26:01.643 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 -- start start 2022-09-10T04:26:01.644 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b840fc490 0x7f5b84195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:01.644 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 0x7f5b84195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:01.644 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 0x7f5b8419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:01.644 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8419d4a0 con 0x7f5b840fc490 2022-09-10T04:26:01.645 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8419d610 con 0x7f5b840fd790 2022-09-10T04:26:01.645 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b83fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8419d7d0 con 0x7f5b840fcde0 2022-09-10T04:26:01.645 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b827fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 0x7f5b84195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:01.645 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b827fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 0x7f5b84195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:44364/0 (socket says 172.21.15.81:44364) 2022-09-10T04:26:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b827fc700 1 -- 172.21.15.81:0/785578145 learned_addr learned my addr 172.21.15.81:0/785578145 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b82ffd700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b840fc490 0x7f5b84195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.632+0000 7f5b837fe700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 0x7f5b8419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:01.646 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b827fc700 1 -- 172.21.15.81:0/785578145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 msgr2=0x7f5b8419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:01.647 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b827fc700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 0x7f5b8419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.647 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b827fc700 1 -- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b840fc490 msgr2=0x7f5b84195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:01.647 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b827fc700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b840fc490 0x7f5b84195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.647 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b827fc700 1 -- 172.21.15.81:0/785578145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0097e0 con 0x7f5b840fcde0 2022-09-10T04:26:01.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b827fc700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 0x7f5b84195e60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e740 tx=0x7f5b7800ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:01.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78005220 con 0x7f5b840fcde0 2022-09-10T04:26:01.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8419dab0 con 0x7f5b840fcde0 2022-09-10T04:26:01.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5b78005380 con 0x7f5b840fcde0 2022-09-10T04:26:01.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.633+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78004b70 con 0x7f5b840fcde0 2022-09-10T04:26:01.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.634+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8419e010 con 0x7f5b840fcde0 2022-09-10T04:26:01.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.635+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8410b4e0 con 0x7f5b840fcde0 2022-09-10T04:26:01.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.636+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f5b7801e020 con 0x7f5b840fcde0 2022-09-10T04:26:01.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.638+0000 7f5b6bfff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5b6c099090 0x7f5b6c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:01.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.638+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5b78014070 con 0x7f5b840fcde0 2022-09-10T04:26:01.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.638+0000 7f5b82ffd700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5b6c099090 0x7f5b6c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:01.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.639+0000 7f5b82ffd700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5b6c099090 0x7f5b6c09b540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5b74005fd0 tx=0x7f5b74005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:01.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.640+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f5b7808ee70 con 0x7f5b840fcde0 2022-09-10T04:26:01.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.896+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b841070f0 con 0x7f5b840fcde0 2022-09-10T04:26:01.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.897+0000 7f5b6bfff700 1 -- 172.21.15.81:0/785578145 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2774 (secure 0 0 0) 0x7f5b78092490 con 0x7f5b840fcde0 2022-09-10T04:26:01.900 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:01.900 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2022-09-10T04:26:01.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5b6c099090 msgr2=0x7f5b6c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:01.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5b6c099090 0x7f5b6c09b540 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5b74005fd0 tx=0x7f5b74005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 msgr2=0x7f5b84195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:01.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 0x7f5b84195e60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800e740 tx=0x7f5b7800ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:01.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 shutdown_connections 2022-09-10T04:26:01.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b840fd790 0x7f5b8419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b840fc490 0x7f5b84195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5b6c099090 0x7f5b6c09b540 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 --2- 172.21.15.81:0/785578145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b840fcde0 0x7f5b84195e60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:01.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 >> 172.21.15.81:0/785578145 conn(0x7f5b840f8160 msgr2=0x7f5b840faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:01.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 shutdown_connections 2022-09-10T04:26:01.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:01.900+0000 7f5b83fff700 1 -- 172.21.15.81:0/785578145 wait complete. 2022-09-10T04:26:01.906 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 4 2022-09-10T04:26:02.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:01 smithi081 ceph-mon[90133]: Deploying daemon mds.d on smithi081 2022-09-10T04:26:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:01 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.81:6834/1412455487,v1:172.21.15.81:6835/1412455487] up:boot 2022-09-10T04:26:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:01 smithi081 ceph-mon[90133]: fsmap 3 up:standby 2022-09-10T04:26:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:01 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-10T04:26:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:01 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/785578145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:01 smithi104 ceph-mon[97044]: Deploying daemon mds.d on smithi081 2022-09-10T04:26:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:01 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.81:6834/1412455487,v1:172.21.15.81:6835/1412455487] up:boot 2022-09-10T04:26:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:01 smithi104 ceph-mon[97044]: fsmap 3 up:standby 2022-09-10T04:26:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:01 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-10T04:26:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:01 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/785578145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:02.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:01 smithi110 ceph-mon[97633]: Deploying daemon mds.d on smithi081 2022-09-10T04:26:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:01 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.81:6834/1412455487,v1:172.21.15.81:6835/1412455487] up:boot 2022-09-10T04:26:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:01 smithi110 ceph-mon[97633]: fsmap 3 up:standby 2022-09-10T04:26:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:01 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-10T04:26:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:01 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/785578145' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:02 smithi104 ceph-mon[97044]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:02 smithi081 ceph-mon[90133]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 04:26:02 smithi081 systemd[1]: Starting Ceph mds.d for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:02 smithi110 ceph-mon[97633]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 04:26:03 smithi081 podman[124309]: 2022-09-10T04:26:03.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:04.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 -- 172.21.15.81:0/2361751767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8103880 msgr2=0x7fb1a8105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:04.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/2361751767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8103880 0x7fb1a8105c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009bb0 tx=0x7fb1a0009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 -- 172.21.15.81:0/2361751767 shutdown_connections 2022-09-10T04:26:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/2361751767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8103880 0x7fb1a8105c60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/2361751767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1a80fe640 0x7fb1a8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/2361751767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a8100f60 0x7fb1a8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 -- 172.21.15.81:0/2361751767 >> 172.21.15.81:0/2361751767 conn(0x7fb1a80f8160 msgr2=0x7fb1a80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:04.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 -- 172.21.15.81:0/2361751767 shutdown_connections 2022-09-10T04:26:04.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.176+0000 7fb1a7fff700 1 -- 172.21.15.81:0/2361751767 wait complete. 2022-09-10T04:26:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.177+0000 7fb1a7fff700 1 Processor -- start 2022-09-10T04:26:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 -- start start 2022-09-10T04:26:04.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a80fe640 0x7fb1a81958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:04.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 0x7fb1a8195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:04.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1a8103880 0x7fb1a819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:04.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a819d450 con 0x7fb1a80fe640 2022-09-10T04:26:04.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a819d5c0 con 0x7fb1a8100f60 2022-09-10T04:26:04.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a7fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a819d780 con 0x7fb1a8103880 2022-09-10T04:26:04.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 0x7fb1a8195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:04.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a67fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 0x7fb1a8195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38640/0 (socket says 172.21.15.81:38640) 2022-09-10T04:26:04.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a67fc700 1 -- 172.21.15.81:0/676655159 learned_addr learned my addr 172.21.15.81:0/676655159 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:04.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a6ffd700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a80fe640 0x7fb1a81958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.178+0000 7fb1a77fe700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1a8103880 0x7fb1a819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a67fc700 1 -- 172.21.15.81:0/676655159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1a8103880 msgr2=0x7fb1a819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a67fc700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1a8103880 0x7fb1a819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a67fc700 1 -- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a80fe640 msgr2=0x7fb1a81958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:04.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a67fc700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a80fe640 0x7fb1a81958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a67fc700 1 -- 172.21.15.81:0/676655159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a00097e0 con 0x7fb1a8100f60 2022-09-10T04:26:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a67fc700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 0x7fb1a8195e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00e740 tx=0x7fb19c00ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c005220 con 0x7fb1a8100f60 2022-09-10T04:26:04.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a819da60 con 0x7fb1a8100f60 2022-09-10T04:26:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb19c005380 con 0x7fb1a8100f60 2022-09-10T04:26:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c004b70 con 0x7fb1a8100f60 2022-09-10T04:26:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.179+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a819df70 con 0x7fb1a8100f60 2022-09-10T04:26:04.187 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.180+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a81092c0 con 0x7fb1a8100f60 2022-09-10T04:26:04.188 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.182+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb19c01e020 con 0x7fb1a8100f60 2022-09-10T04:26:04.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.183+0000 7fb18ffff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb190099010 0x7fb19009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:04.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.183+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb19c014070 con 0x7fb1a8100f60 2022-09-10T04:26:04.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.184+0000 7fb1a6ffd700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb190099010 0x7fb19009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:04.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.184+0000 7fb1a6ffd700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb190099010 0x7fb19009b4c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb198005fd0 tx=0x7fb198005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:04.190 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.187+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb19c08ec10 con 0x7fb1a8100f60 2022-09-10T04:26:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 04:26:03 smithi081 bash[124309]: 61412591081ac7cac98e1bae1b9983488afd9b2ca7bc4a36d5d76e5c2ca8746e 2022-09-10T04:26:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 04:26:03 smithi081 ceph-mds[124350]: -- 172.21.15.81:0/2835071283 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eaaed64820 con 0x55eaaed38c00 2022-09-10T04:26:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 04:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: starting mds.d at 2022-09-10T04:26:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 04:26:03 smithi081 systemd[1]: Started Ceph mds.d for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:04.425 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.422+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1a819eb10 con 0x7fb1a8100f60 2022-09-10T04:26:04.426 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.423+0000 7fb18ffff700 1 -- 172.21.15.81:0/676655159 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3528 (secure 0 0 0) 0x7fb19c092230 con 0x7fb1a8100f60 2022-09-10T04:26:04.427 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:04.427 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2022-09-10T04:26:04.429 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.425+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb190099010 msgr2=0x7fb19009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:04.430 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.425+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb190099010 0x7fb19009b4c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb198005fd0 tx=0x7fb198005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.425+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 msgr2=0x7fb1a8195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:04.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.425+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 0x7fb1a8195e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00e740 tx=0x7fb19c00ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:04.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 shutdown_connections 2022-09-10T04:26:04.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb190099010 0x7fb19009b4c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1a8103880 0x7fb1a819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1a80fe640 0x7fb1a81958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 --2- 172.21.15.81:0/676655159 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1a8100f60 0x7fb1a8195e10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:04.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 >> 172.21.15.81:0/676655159 conn(0x7fb1a80f8160 msgr2=0x7fb1a80ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:04.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 shutdown_connections 2022-09-10T04:26:04.434 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:04.426+0000 7fb1a7fff700 1 -- 172.21.15.81:0/676655159 wait complete. 2022-09-10T04:26:04.434 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 5 2022-09-10T04:26:05.018 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:05.018 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:05.019 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:05.019 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:05.020 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:05.020 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:05.021 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.81:6836/316573610,v1:172.21.15.81:6837/316573610] up:boot 2022-09-10T04:26:05.021 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: fsmap 4 up:standby 2022-09-10T04:26:05.021 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:26:05.022 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:04 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/676655159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:05.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:05.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:05.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.81:6836/316573610,v1:172.21.15.81:6837/316573610] up:boot 2022-09-10T04:26:05.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: fsmap 4 up:standby 2022-09-10T04:26:05.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:26:05.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:04 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/676655159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:05.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:05.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:05.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:05.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:05.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.81:6836/316573610,v1:172.21.15.81:6837/316573610] up:boot 2022-09-10T04:26:05.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: fsmap 4 up:standby 2022-09-10T04:26:05.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:26:05.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:04 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/676655159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:05.796 INFO:journalctl@ceph.mds.e.smithi104.stdout:Sep 10 04:26:05 smithi104 systemd[1]: Starting Ceph mds.e for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:06.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:05 smithi104 ceph-mon[97044]: Deploying daemon mds.e on smithi104 2022-09-10T04:26:06.060 INFO:journalctl@ceph.mds.e.smithi104.stdout:Sep 10 04:26:05 smithi104 podman[123188]: 2022-09-10T04:26:06.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:05 smithi110 ceph-mon[97633]: Deploying daemon mds.e on smithi104 2022-09-10T04:26:06.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:05 smithi081 ceph-mon[90133]: Deploying daemon mds.e on smithi104 2022-09-10T04:26:06.456 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:06.546 INFO:journalctl@ceph.mds.e.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mds[123226]: -- 172.21.15.104:0/3701314281 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c18d1be820 con 0x55c18d192c00 2022-09-10T04:26:06.546 INFO:journalctl@ceph.mds.e.smithi104.stdout:Sep 10 04:26:06 smithi104 bash[123188]: 5e6a16cb1689fce16cede68a865d696acec5e2b472d1d197e19ab862302cf448 2022-09-10T04:26:06.546 INFO:journalctl@ceph.mds.e.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-e[123204]: starting mds.e at 2022-09-10T04:26:06.547 INFO:journalctl@ceph.mds.e.smithi104.stdout:Sep 10 04:26:06 smithi104 systemd[1]: Started Ceph mds.e for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:06.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/3344801105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8100dd0 msgr2=0x7f6db8103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:06.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/3344801105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8100dd0 0x7f6db8103230 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009810 tx=0x7f6dac009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:06.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/3344801105 shutdown_connections 2022-09-10T04:26:06.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/3344801105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6db81038c0 0x7f6db8105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/3344801105 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db8066bf0 0x7f6db8100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/3344801105 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8100dd0 0x7f6db8103230 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/3344801105 >> 172.21.15.81:0/3344801105 conn(0x7f6db80f82a0 msgr2=0x7f6db80fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:06.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/3344801105 shutdown_connections 2022-09-10T04:26:06.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.687+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/3344801105 wait complete. 2022-09-10T04:26:06.694 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 Processor -- start 2022-09-10T04:26:06.694 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 -- start start 2022-09-10T04:26:06.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8066bf0 0x7f6db8195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:06.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6db8100dd0 0x7f6db81960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:06.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 0x7f6db819d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:06.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db819d6f0 con 0x7f6db8100dd0 2022-09-10T04:26:06.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db819d860 con 0x7f6db8066bf0 2022-09-10T04:26:06.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6dbd5ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db819da20 con 0x7f6db81038c0 2022-09-10T04:26:06.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.689+0000 7f6db77fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 0x7f6db819d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:06.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 0x7f6db819d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:44436/0 (socket says 172.21.15.81:44436) 2022-09-10T04:26:06.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db6ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8066bf0 0x7f6db8195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:06.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 -- 172.21.15.81:0/2909866542 learned_addr learned my addr 172.21.15.81:0/2909866542 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:06.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db67fc700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6db8100dd0 0x7f6db81960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:06.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 -- 172.21.15.81:0/2909866542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8066bf0 msgr2=0x7f6db8195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:06.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8066bf0 0x7f6db8195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 -- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6db8100dd0 msgr2=0x7f6db81960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:06.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6db8100dd0 0x7f6db81960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 -- 172.21.15.81:0/2909866542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac009470 con 0x7f6db81038c0 2022-09-10T04:26:06.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6db77fe700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 0x7f6db819d120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6db000ebf0 tx=0x7f6db000ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:06.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.690+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db00099c0 con 0x7f6db81038c0 2022-09-10T04:26:06.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.691+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6db000cc90 con 0x7f6db81038c0 2022-09-10T04:26:06.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.691+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db819dd00 con 0x7f6db81038c0 2022-09-10T04:26:06.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.691+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0010430 con 0x7f6db81038c0 2022-09-10T04:26:06.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.691+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db819e1c0 con 0x7f6db81038c0 2022-09-10T04:26:06.701 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.692+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db81094f0 con 0x7f6db81038c0 2022-09-10T04:26:06.703 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.693+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6db001e020 con 0x7f6db81038c0 2022-09-10T04:26:06.703 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.695+0000 7f6d9ffff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6da0099010 0x7f6da009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:06.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.695+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6db0014070 con 0x7f6db81038c0 2022-09-10T04:26:06.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.695+0000 7f6db6ffd700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6da0099010 0x7f6da009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:06.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.695+0000 7f6db6ffd700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6da0099010 0x7f6da009b4c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6db8066fc0 tx=0x7f6da8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:06.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.697+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6db008f730 con 0x7f6db81038c0 2022-09-10T04:26:06.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.932+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db80fdc90 con 0x7f6db81038c0 2022-09-10T04:26:06.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.932+0000 7f6d9ffff700 1 -- 172.21.15.81:0/2909866542 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4288 (secure 0 0 0) 0x7f6db0092d50 con 0x7f6db81038c0 2022-09-10T04:26:06.935 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:06.936 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2022-09-10T04:26:06.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6da0099010 msgr2=0x7f6da009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:06.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6da0099010 0x7f6da009b4c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6db8066fc0 tx=0x7f6da8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 msgr2=0x7f6db819d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:06.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 0x7f6db819d120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6db000ebf0 tx=0x7f6db000ef00 comp rx=0 tx=0).stop 2022-09-10T04:26:06.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 shutdown_connections 2022-09-10T04:26:06.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6da0099010 0x7f6da009b4c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6db81038c0 0x7f6db819d120 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6db8066bf0 0x7f6db8195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 --2- 172.21.15.81:0/2909866542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6db8100dd0 0x7f6db81960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 >> 172.21.15.81:0/2909866542 conn(0x7f6db80f82a0 msgr2=0x7f6db80fa680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 shutdown_connections 2022-09-10T04:26:06.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:06.935+0000 7f6dbd5ff700 1 -- 172.21.15.81:0/2909866542 wait complete. 2022-09-10T04:26:06.942 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 6 2022-09-10T04:26:07.124 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:06 smithi110 ceph-mon[97633]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:07.124 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:07.125 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:07.125 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:07.125 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:07.125 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:06 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mon[97044]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:06 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:07.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:06 smithi081 ceph-mon[90133]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:07.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:06 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:07.693 INFO:journalctl@ceph.mds.f.smithi110.stdout:Sep 10 04:26:07 smithi110 systemd[1]: Starting Ceph mds.f for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:07.978 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:07 smithi110 ceph-mon[97633]: Deploying daemon mds.f on smithi110 2022-09-10T04:26:07.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:07 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.104:6834/2709283075,v1:172.21.15.104:6835/2709283075] up:boot 2022-09-10T04:26:07.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:07 smithi110 ceph-mon[97633]: fsmap 5 up:standby 2022-09-10T04:26:07.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:07 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-10T04:26:07.979 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:07 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2909866542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:08.240 INFO:journalctl@ceph.mds.f.smithi110.stdout:Sep 10 04:26:07 smithi110 podman[124671]: 2022-09-10T04:26:08.240 INFO:journalctl@ceph.mds.f.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mds[124710]: -- 172.21.15.110:0/1900114721 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e2913a2820 con 0x55e291377000 2022-09-10T04:26:08.240 INFO:journalctl@ceph.mds.f.smithi110.stdout:Sep 10 04:26:08 smithi110 bash[124671]: b9ebe1cc893467edcfc6ca9c5e9b1bda724fa40864d8d4754a88556e36187146 2022-09-10T04:26:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:07 smithi104 ceph-mon[97044]: Deploying daemon mds.f on smithi110 2022-09-10T04:26:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:07 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.104:6834/2709283075,v1:172.21.15.104:6835/2709283075] up:boot 2022-09-10T04:26:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:07 smithi104 ceph-mon[97044]: fsmap 5 up:standby 2022-09-10T04:26:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:07 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-10T04:26:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:07 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2909866542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:07 smithi081 ceph-mon[90133]: Deploying daemon mds.f on smithi110 2022-09-10T04:26:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:07 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.104:6834/2709283075,v1:172.21.15.104:6835/2709283075] up:boot 2022-09-10T04:26:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:07 smithi081 ceph-mon[90133]: fsmap 5 up:standby 2022-09-10T04:26:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:07 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-10T04:26:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:07 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2909866542' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:08.656 INFO:journalctl@ceph.mds.f.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-f[124687]: starting mds.f at 2022-09-10T04:26:08.657 INFO:journalctl@ceph.mds.f.smithi110.stdout:Sep 10 04:26:08 smithi110 systemd[1]: Started Ceph mds.f for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:08.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mon[97633]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:08 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:09.201 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.199+0000 7f54f95ff700 1 -- 172.21.15.81:0/2731861529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f4100dd0 msgr2=0x7f54f4103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:09.202 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.199+0000 7f54f95ff700 1 --2- 172.21.15.81:0/2731861529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f4100dd0 0x7f54f4103230 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54e8009750 tx=0x7f54e8009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:09.202 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 -- 172.21.15.81:0/2731861529 shutdown_connections 2022-09-10T04:26:09.202 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 --2- 172.21.15.81:0/2731861529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f41038c0 0x7f54f4105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.203 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 --2- 172.21.15.81:0/2731861529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54f4066bf0 0x7f54f4100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.203 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 --2- 172.21.15.81:0/2731861529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f4100dd0 0x7f54f4103230 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.203 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 -- 172.21.15.81:0/2731861529 >> 172.21.15.81:0/2731861529 conn(0x7f54f40f82a0 msgr2=0x7f54f40fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:09.203 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 -- 172.21.15.81:0/2731861529 shutdown_connections 2022-09-10T04:26:09.204 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.200+0000 7f54f95ff700 1 -- 172.21.15.81:0/2731861529 wait complete. 2022-09-10T04:26:09.207 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 Processor -- start 2022-09-10T04:26:09.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 -- start start 2022-09-10T04:26:09.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 0x7f54f4195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:09.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54f4100dd0 0x7f54f4196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:09.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f41038c0 0x7f54f419d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:09.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f419d6d0 con 0x7f54f4066bf0 2022-09-10T04:26:09.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f419d840 con 0x7f54f4100dd0 2022-09-10T04:26:09.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.201+0000 7f54f95ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f419da00 con 0x7f54f41038c0 2022-09-10T04:26:09.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 0x7f54f4195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:09.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 0x7f54f4195b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:36994/0 (socket says 172.21.15.81:36994) 2022-09-10T04:26:09.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 -- 172.21.15.81:0/3318818176 learned_addr learned my addr 172.21.15.81:0/3318818176 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:09.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f37fe700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f41038c0 0x7f54f419d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:09.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f27fc700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54f4100dd0 0x7f54f4196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:09.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 -- 172.21.15.81:0/3318818176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f41038c0 msgr2=0x7f54f419d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f41038c0 0x7f54f419d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 -- 172.21.15.81:0/3318818176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54f4100dd0 msgr2=0x7f54f4196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54f4100dd0 0x7f54f4196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 -- 172.21.15.81:0/3318818176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80093b0 con 0x7f54f4066bf0 2022-09-10T04:26:09.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54f2ffd700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 0x7f54f4195b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54e400e6e0 tx=0x7f54e400e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:09.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.202+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4009500 con 0x7f54f4066bf0 2022-09-10T04:26:09.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.203+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f419dce0 con 0x7f54f4066bf0 2022-09-10T04:26:09.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.203+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f54e4010ce0 con 0x7f54f4066bf0 2022-09-10T04:26:09.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.203+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e400f420 con 0x7f54f4066bf0 2022-09-10T04:26:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.203+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f419e190 con 0x7f54f4066bf0 2022-09-10T04:26:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.204+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f4109540 con 0x7f54f4066bf0 2022-09-10T04:26:09.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.205+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f54e400c040 con 0x7f54f4066bf0 2022-09-10T04:26:09.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.207+0000 7f54dbfff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f54dc099090 0x7f54dc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:09.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.207+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f54e4014070 con 0x7f54f4066bf0 2022-09-10T04:26:09.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.207+0000 7f54f27fc700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f54dc099090 0x7f54dc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:09.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.208+0000 7f54f27fc700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f54dc099090 0x7f54dc09b540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54e800f500 tx=0x7f54e8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:09.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.211+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f54e404fc80 con 0x7f54f4066bf0 2022-09-10T04:26:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:08 smithi104 ceph-mon[97044]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:09.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:08 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:09.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:08 smithi081 ceph-mon[90133]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:09.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:09.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:09.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:09.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:09.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:08 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:09.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.454+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54f405f8f0 con 0x7f54f4066bf0 2022-09-10T04:26:09.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.454+0000 7f54dbfff700 1 -- 172.21.15.81:0/3318818176 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5048 (secure 0 0 0) 0x7f54e408ea70 con 0x7f54f4066bf0 2022-09-10T04:26:09.457 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:09.457 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2022-09-10T04:26:09.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f54dc099090 msgr2=0x7f54dc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:09.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f54dc099090 0x7f54dc09b540 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54e800f500 tx=0x7f54e8005f90 comp rx=0 tx=0).stop 2022-09-10T04:26:09.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 msgr2=0x7f54f4195b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:09.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 0x7f54f4195b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f54e400e6e0 tx=0x7f54e400e9f0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 shutdown_connections 2022-09-10T04:26:09.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f54dc099090 0x7f54dc09b540 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54f41038c0 0x7f54f419d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54f4066bf0 0x7f54f4195b50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 --2- 172.21.15.81:0/3318818176 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f54f4100dd0 0x7f54f4196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:09.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 >> 172.21.15.81:0/3318818176 conn(0x7f54f40f82a0 msgr2=0x7f54f40fa680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:09.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 shutdown_connections 2022-09-10T04:26:09.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:09.457+0000 7f54f95ff700 1 -- 172.21.15.81:0/3318818176 wait complete. 2022-09-10T04:26:09.464 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 7 2022-09-10T04:26:09.874 INFO:journalctl@ceph.mds.c.smithi110.stdout:Sep 10 04:26:09 smithi110 systemd[1]: Starting Ceph mds.c for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:10.144 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: Deploying daemon mds.c on smithi110 2022-09-10T04:26:10.145 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6834/4033961168,v1:172.21.15.110:6835/4033961168] up:boot 2022-09-10T04:26:10.145 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: fsmap 6 up:standby 2022-09-10T04:26:10.145 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-10T04:26:10.146 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:26:10.146 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:26:10.147 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:09 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3318818176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: Deploying daemon mds.c on smithi110 2022-09-10T04:26:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6834/4033961168,v1:172.21.15.110:6835/4033961168] up:boot 2022-09-10T04:26:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: fsmap 6 up:standby 2022-09-10T04:26:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-10T04:26:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:26:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:26:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:09 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3318818176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:10.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: Deploying daemon mds.c on smithi110 2022-09-10T04:26:10.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6834/4033961168,v1:172.21.15.110:6835/4033961168] up:boot 2022-09-10T04:26:10.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: fsmap 6 up:standby 2022-09-10T04:26:10.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-10T04:26:10.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:26:10.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:26:10.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:09 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3318818176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:10.406 INFO:journalctl@ceph.mds.c.smithi110.stdout:Sep 10 04:26:10 smithi110 podman[125018]: 2022-09-10T04:26:10.876 INFO:journalctl@ceph.mds.c.smithi110.stdout:Sep 10 04:26:10 smithi110 bash[125018]: a775a83fe1d76ab8dde389fc10cda26730e63193919035ef2498d237e4ec846c 2022-09-10T04:26:10.876 INFO:journalctl@ceph.mds.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mds[125058]: -- 172.21.15.110:0/1749442699 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561b58f20820 con 0x561b58ef4c00 2022-09-10T04:26:10.876 INFO:journalctl@ceph.mds.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-c[125034]: starting mds.c at 2022-09-10T04:26:10.877 INFO:journalctl@ceph.mds.c.smithi110.stdout:Sep 10 04:26:10 smithi110 systemd[1]: Started Ceph mds.c for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mon[97633]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:11.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:11.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:11.251 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:10 smithi104 ceph-mon[97044]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:11.252 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:11.252 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:11.253 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:11.253 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:11.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:10 smithi081 ceph-mon[90133]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:11.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:11.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:11.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:11.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:11.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2844761195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fdcf0 msgr2=0x7f5f980fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:11.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2844761195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fdcf0 0x7f5f980fe100 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009810 tx=0x7f5f94009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:11.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2844761195 shutdown_connections 2022-09-10T04:26:11.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2844761195 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980ff900 0x7f5f981061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2844761195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fdcf0 0x7f5f980fe100 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2844761195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980fef70 0x7f5f980ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2844761195 >> 172.21.15.81:0/2844761195 conn(0x7f5f980f93c0 msgr2=0x7f5f980fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:11.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2844761195 shutdown_connections 2022-09-10T04:26:11.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.725+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2844761195 wait complete. 2022-09-10T04:26:11.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.726+0000 7f5fa03ff700 1 Processor -- start 2022-09-10T04:26:11.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.726+0000 7f5fa03ff700 1 -- start start 2022-09-10T04:26:11.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5fa03ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 0x7f5f98072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:11.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5fa03ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fef70 0x7f5f980710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:11.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5fa03ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980ff900 0x7f5f98071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:11.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5fa03ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f98071b50 con 0x7f5f980fef70 2022-09-10T04:26:11.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5fa03ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f98071cc0 con 0x7f5f980fdcf0 2022-09-10T04:26:11.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5fa03ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f98071e80 con 0x7f5f980ff900 2022-09-10T04:26:11.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 0x7f5f98072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:11.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 0x7f5f98072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38416/0 (socket says 172.21.15.81:38416) 2022-09-10T04:26:11.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 -- 172.21.15.81:0/2681431583 learned_addr learned my addr 172.21.15.81:0/2681431583 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:11.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9d99a700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fef70 0x7f5f980710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:11.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e99c700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980ff900 0x7f5f98071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:11.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 -- 172.21.15.81:0/2681431583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980ff900 msgr2=0x7f5f98071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:11.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980ff900 0x7f5f98071610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 -- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fef70 msgr2=0x7f5f980710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:11.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fef70 0x7f5f980710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e19b700 1 -- 172.21.15.81:0/2681431583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f94009470 con 0x7f5f980fdcf0 2022-09-10T04:26:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9d99a700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fef70 0x7f5f980710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:26:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.727+0000 7f5f9e99c700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980ff900 0x7f5f98071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:26:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.728+0000 7f5f9e19b700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 0x7f5f98072a90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5f940059c0 tx=0x7f5f9400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.728+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9401d070 con 0x7f5f980fdcf0 2022-09-10T04:26:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.728+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5f940054f0 con 0x7f5f980fdcf0 2022-09-10T04:26:11.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.728+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f981a2150 con 0x7f5f980fdcf0 2022-09-10T04:26:11.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.728+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94004930 con 0x7f5f980fdcf0 2022-09-10T04:26:11.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.728+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f981a2510 con 0x7f5f980fdcf0 2022-09-10T04:26:11.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.729+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f981094a0 con 0x7f5f980fdcf0 2022-09-10T04:26:11.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.731+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f5f94028030 con 0x7f5f980fdcf0 2022-09-10T04:26:11.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.732+0000 7f5f8f7fe700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5f84099010 0x7f5f8409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:11.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.732+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5f940c2e60 con 0x7f5f980fdcf0 2022-09-10T04:26:11.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.732+0000 7f5f9d99a700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5f84099010 0x7f5f8409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:11.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.733+0000 7f5f9d99a700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5f84099010 0x7f5f8409b4c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800bbf0 tx=0x7f5f8800b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:11.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.735+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f5f9408f950 con 0x7f5f980fdcf0 2022-09-10T04:26:11.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.975+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f981a2b50 con 0x7f5f980fdcf0 2022-09-10T04:26:11.977 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.976+0000 7f5f8f7fe700 1 -- 172.21.15.81:0/2681431583 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5808 (secure 0 0 0) 0x7f5f94092f70 con 0x7f5f980fdcf0 2022-09-10T04:26:11.978 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:11.978 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2022-09-10T04:26:11.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5f84099010 msgr2=0x7f5f8409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:11.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5f84099010 0x7f5f8409b4c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800bbf0 tx=0x7f5f8800b5d0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 msgr2=0x7f5f98072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:11.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 0x7f5f98072a90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5f940059c0 tx=0x7f5f9400bfd0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 shutdown_connections 2022-09-10T04:26:11.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5f84099010 0x7f5f8409b4c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f980ff900 0x7f5f98071610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5f980fdcf0 0x7f5f98072a90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 --2- 172.21.15.81:0/2681431583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f980fef70 0x7f5f980710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:11.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 >> 172.21.15.81:0/2681431583 conn(0x7f5f980f93c0 msgr2=0x7f5f980fb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:11.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.978+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 shutdown_connections 2022-09-10T04:26:11.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:11.979+0000 7f5fa03ff700 1 -- 172.21.15.81:0/2681431583 wait complete. 2022-09-10T04:26:11.986 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2022-09-10T04:26:12.118 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:11 smithi104 ceph-mon[97044]: Deploying daemon mds.h on smithi104 2022-09-10T04:26:12.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:11 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] up:boot 2022-09-10T04:26:12.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:11 smithi104 ceph-mon[97044]: fsmap 7 up:standby 2022-09-10T04:26:12.119 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:11 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-10T04:26:12.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:11 smithi110 ceph-mon[97633]: Deploying daemon mds.h on smithi104 2022-09-10T04:26:12.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:11 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] up:boot 2022-09-10T04:26:12.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:11 smithi110 ceph-mon[97633]: fsmap 7 up:standby 2022-09-10T04:26:12.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:11 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-10T04:26:12.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:11 smithi081 ceph-mon[90133]: Deploying daemon mds.h on smithi104 2022-09-10T04:26:12.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:11 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] up:boot 2022-09-10T04:26:12.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:11 smithi081 ceph-mon[90133]: fsmap 7 up:standby 2022-09-10T04:26:12.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:11 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-10T04:26:12.408 INFO:journalctl@ceph.mds.h.smithi104.stdout:Sep 10 04:26:12 smithi104 systemd[1]: Starting Ceph mds.h for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:12.660 INFO:journalctl@ceph.mds.h.smithi104.stdout:Sep 10 04:26:12 smithi104 podman[123531]: 2022-09-10T04:26:13.046 INFO:journalctl@ceph.mds.h.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mds[123570]: -- 172.21.15.104:0/558112204 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558700790820 con 0x558700765000 2022-09-10T04:26:13.046 INFO:journalctl@ceph.mds.h.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-h[123547]: starting mds.h at 2022-09-10T04:26:13.046 INFO:journalctl@ceph.mds.h.smithi104.stdout:Sep 10 04:26:12 smithi104 bash[123531]: 9adbd23de36eb4c1fa9024de013733758ed34c8b863b6158d452bedf5492cc85 2022-09-10T04:26:13.047 INFO:journalctl@ceph.mds.h.smithi104.stdout:Sep 10 04:26:12 smithi104 systemd[1]: Started Ceph mds.h for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2681431583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:13.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:13.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:13.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:13.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:12 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2681431583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:12 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2681431583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:13.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:12 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:14.008 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:14.097 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:13 smithi110 ceph-mon[97633]: Deploying daemon mds.i on smithi110 2022-09-10T04:26:14.097 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:13 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.104:6836/2202378802,v1:172.21.15.104:6837/2202378802] up:boot 2022-09-10T04:26:14.097 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:13 smithi110 ceph-mon[97633]: fsmap 8 up:standby 2022-09-10T04:26:14.098 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:13 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-10T04:26:14.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 -- 172.21.15.81:0/2479856923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 msgr2=0x7f6c8c103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:14.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/2479856923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c103340 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009750 tx=0x7f6c80009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:14.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 -- 172.21.15.81:0/2479856923 shutdown_connections 2022-09-10T04:26:14.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/2479856923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c103880 0x7f6c8c105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/2479856923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/2479856923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c103340 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.241+0000 7f6c8b59e700 1 -- 172.21.15.81:0/2479856923 >> 172.21.15.81:0/2479856923 conn(0x7f6c8c0f8160 msgr2=0x7f6c8c0fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:14.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.242+0000 7f6c8b59e700 1 -- 172.21.15.81:0/2479856923 shutdown_connections 2022-09-10T04:26:14.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.242+0000 7f6c8b59e700 1 -- 172.21.15.81:0/2479856923 wait complete. 2022-09-10T04:26:14.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 Processor -- start 2022-09-10T04:26:14.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 -- start start 2022-09-10T04:26:14.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c0fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:14.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c0fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:14.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c103880 0x7f6c8c0fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:14.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0fd330 con 0x7f6c8c0fe640 2022-09-10T04:26:14.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c0fd4a0 con 0x7f6c8c103880 2022-09-10T04:26:14.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.243+0000 7f6c8b59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c19d7e0 con 0x7f6c8c100f60 2022-09-10T04:26:14.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c8a59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c0fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:14.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c0fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:14.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c8a59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c0fdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55430/0 (socket says 172.21.15.81:55430) 2022-09-10T04:26:14.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c8a59c700 1 -- 172.21.15.81:0/3699092540 learned_addr learned my addr 172.21.15.81:0/3699092540 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:14.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c8ad9d700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c103880 0x7f6c8c0fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:14.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 -- 172.21.15.81:0/3699092540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c103880 msgr2=0x7f6c8c0fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:14.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c103880 0x7f6c8c0fcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 -- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 msgr2=0x7f6c8c0fdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:14.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c0fdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.262 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 -- 172.21.15.81:0/3699092540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c009470 con 0x7f6c8c100f60 2022-09-10T04:26:14.262 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c8ad9d700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c103880 0x7f6c8c0fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:26:14.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c8a59c700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c0fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:14.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.244+0000 7f6c89d9b700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c0fc5f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c800096a0 tx=0x7f6c80004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:14.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.245+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8001d070 con 0x7f6c8c100f60 2022-09-10T04:26:14.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.245+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6c80003bb0 con 0x7f6c8c100f60 2022-09-10T04:26:14.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.245+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c800093b0 con 0x7f6c8c100f60 2022-09-10T04:26:14.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.245+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80017ac0 con 0x7f6c8c100f60 2022-09-10T04:26:14.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.245+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c19dd20 con 0x7f6c8c100f60 2022-09-10T04:26:14.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.246+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c1092c0 con 0x7f6c8c100f60 2022-09-10T04:26:14.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.247+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6c8000c040 con 0x7f6c8c100f60 2022-09-10T04:26:14.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.249+0000 7f6c7b7fe700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c74099010 0x7f6c7409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:14.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.249+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f6c800c2c50 con 0x7f6c8c100f60 2022-09-10T04:26:14.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.249+0000 7f6c8a59c700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c74099010 0x7f6c7409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:14.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.250+0000 7f6c8a59c700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c74099010 0x7f6c7409b4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005b90 tx=0x7f6c7c005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:14.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.253+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6c8008f740 con 0x7f6c8c100f60 2022-09-10T04:26:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:13 smithi104 ceph-mon[97044]: Deploying daemon mds.i on smithi110 2022-09-10T04:26:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:13 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.104:6836/2202378802,v1:172.21.15.104:6837/2202378802] up:boot 2022-09-10T04:26:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:13 smithi104 ceph-mon[97044]: fsmap 8 up:standby 2022-09-10T04:26:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:13 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-10T04:26:14.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:13 smithi081 ceph-mon[90133]: Deploying daemon mds.i on smithi110 2022-09-10T04:26:14.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:13 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.104:6836/2202378802,v1:172.21.15.104:6837/2202378802] up:boot 2022-09-10T04:26:14.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:13 smithi081 ceph-mon[90133]: fsmap 8 up:standby 2022-09-10T04:26:14.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:13 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-10T04:26:14.407 INFO:journalctl@ceph.mds.i.smithi110.stdout:Sep 10 04:26:14 smithi110 systemd[1]: Starting Ceph mds.i for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:14.491 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.489+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8c0fab60 con 0x7f6c8c100f60 2022-09-10T04:26:14.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.490+0000 7f6c7b7fe700 1 -- 172.21.15.81:0/3699092540 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6568 (secure 0 0 0) 0x7f6c80092d60 con 0x7f6c8c100f60 2022-09-10T04:26:14.492 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:14.492 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2022-09-10T04:26:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c74099010 msgr2=0x7f6c7409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c74099010 0x7f6c7409b4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005b90 tx=0x7f6c7c005f60 comp rx=0 tx=0).stop 2022-09-10T04:26:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 msgr2=0x7f6c8c0fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:14.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c0fc5f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6c800096a0 tx=0x7f6c80004230 comp rx=0 tx=0).stop 2022-09-10T04:26:14.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 shutdown_connections 2022-09-10T04:26:14.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c8c103880 0x7f6c8c0fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c8c0fe640 0x7f6c8c0fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c74099010 0x7f6c7409b4c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 --2- 172.21.15.81:0/3699092540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c8c100f60 0x7f6c8c0fc5f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:14.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 >> 172.21.15.81:0/3699092540 conn(0x7f6c8c0f8160 msgr2=0x7f6c8c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:14.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 shutdown_connections 2022-09-10T04:26:14.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:14.493+0000 7f6c8b59e700 1 -- 172.21.15.81:0/3699092540 wait complete. 2022-09-10T04:26:14.499 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 9 2022-09-10T04:26:14.797 INFO:journalctl@ceph.mds.i.smithi110.stdout:Sep 10 04:26:14 smithi110 podman[125366]: 2022-09-10T04:26:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mon[97633]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3699092540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:15.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:15.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:15.161 INFO:journalctl@ceph.mds.i.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-mds[125406]: -- 172.21.15.110:0/2868019062 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560e9a060820 con 0x560e9a035000 2022-09-10T04:26:15.161 INFO:journalctl@ceph.mds.i.smithi110.stdout:Sep 10 04:26:14 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-i[125382]: starting mds.i at 2022-09-10T04:26:15.162 INFO:journalctl@ceph.mds.i.smithi110.stdout:Sep 10 04:26:14 smithi110 bash[125366]: fb42ce2c17f0ccf992064ee3e0343178346f9d66997e1d5c4290def724fcb01b 2022-09-10T04:26:15.162 INFO:journalctl@ceph.mds.i.smithi110.stdout:Sep 10 04:26:14 smithi110 systemd[1]: Started Ceph mds.i for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:14 smithi104 ceph-mon[97044]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:14 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3699092540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:15.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:15.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:15.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:14 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:14 smithi081 ceph-mon[90133]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:15.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:14 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3699092540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:15.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:15.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:15.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:15.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:14 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:15 smithi110 ceph-mon[97633]: Deploying daemon mds.g on smithi081 2022-09-10T04:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:15 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] up:boot 2022-09-10T04:26:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:15 smithi110 ceph-mon[97633]: fsmap 9 up:standby 2022-09-10T04:26:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:15 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-10T04:26:16.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:16.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:15 smithi081 ceph-mon[90133]: Deploying daemon mds.g on smithi081 2022-09-10T04:26:16.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:15 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] up:boot 2022-09-10T04:26:16.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:15 smithi081 ceph-mon[90133]: fsmap 9 up:standby 2022-09-10T04:26:16.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:15 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-10T04:26:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:15 smithi104 ceph-mon[97044]: Deploying daemon mds.g on smithi081 2022-09-10T04:26:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:15 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] up:boot 2022-09-10T04:26:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:15 smithi104 ceph-mon[97044]: fsmap 9 up:standby 2022-09-10T04:26:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:15 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-10T04:26:16.524 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:16.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 -- 172.21.15.81:0/2825344983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 msgr2=0x7f34c4101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:16.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 --2- 172.21.15.81:0/2825344983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c4101020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009810 tx=0x7f34b8009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:16.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 -- 172.21.15.81:0/2825344983 shutdown_connections 2022-09-10T04:26:16.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 --2- 172.21.15.81:0/2825344983 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c41016b0 0x7f34c4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:16.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 --2- 172.21.15.81:0/2825344983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34c4066bf0 0x7f34c40fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:16.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 --2- 172.21.15.81:0/2825344983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c4101020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 -- 172.21.15.81:0/2825344983 >> 172.21.15.81:0/2825344983 conn(0x7f34c40f82a0 msgr2=0x7f34c40fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 -- 172.21.15.81:0/2825344983 shutdown_connections 2022-09-10T04:26:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.785+0000 7f34c8dff700 1 -- 172.21.15.81:0/2825344983 wait complete. 2022-09-10T04:26:16.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.787+0000 7f34c8dff700 1 Processor -- start 2022-09-10T04:26:16.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.787+0000 7f34c8dff700 1 -- start start 2022-09-10T04:26:16.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.787+0000 7f34c8dff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 0x7f34c4199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:16.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.787+0000 7f34c8dff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:16.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c8dff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34c41016b0 0x7f34c41a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:16.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c8dff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c41a1ae0 con 0x7f34c4066bf0 2022-09-10T04:26:16.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c8dff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c41a1c50 con 0x7f34c41016b0 2022-09-10T04:26:16.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c8dff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c41a1e10 con 0x7f34c40febc0 2022-09-10T04:26:16.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 0x7f34c4199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 0x7f34c4199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55446/0 (socket says 172.21.15.81:55446) 2022-09-10T04:26:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c259c700 1 -- 172.21.15.81:0/1365332426 learned_addr learned my addr 172.21.15.81:0/1365332426 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c1d9b700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:16.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c2d9d700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34c41016b0 0x7f34c41a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:16.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c1d9b700 1 -- 172.21.15.81:0/1365332426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34c41016b0 msgr2=0x7f34c41a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:16.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c1d9b700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34c41016b0 0x7f34c41a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:16.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c1d9b700 1 -- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 msgr2=0x7f34c4199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:16.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c1d9b700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 0x7f34c4199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:16.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c1d9b700 1 -- 172.21.15.81:0/1365332426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b8009470 con 0x7f34c40febc0 2022-09-10T04:26:16.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.788+0000 7f34c259c700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 0x7f34c4199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:16.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.789+0000 7f34c1d9b700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c419a4a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34b8006010 tx=0x7f34b800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:16.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.789+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b801d070 con 0x7f34c40febc0 2022-09-10T04:26:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.789+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f34b800f880 con 0x7f34c40febc0 2022-09-10T04:26:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.789+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8004760 con 0x7f34c40febc0 2022-09-10T04:26:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.789+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c41a2090 con 0x7f34c40febc0 2022-09-10T04:26:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.789+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c41a24f0 con 0x7f34c40febc0 2022-09-10T04:26:16.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.790+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c410fad0 con 0x7f34c40febc0 2022-09-10T04:26:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.792+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f34b8028030 con 0x7f34c40febc0 2022-09-10T04:26:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.793+0000 7f34b37fe700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34ac099010 0x7f34ac09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.793+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f34b80c2b80 con 0x7f34c40febc0 2022-09-10T04:26:16.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.794+0000 7f34c259c700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34ac099010 0x7f34ac09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.794+0000 7f34c259c700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34ac099010 0x7f34ac09b4c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34b4005ad0 tx=0x7f34b4005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:16.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:16.797+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f34b808f670 con 0x7f34c40febc0 2022-09-10T04:26:17.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.047+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34c405f120 con 0x7f34c40febc0 2022-09-10T04:26:17.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.048+0000 7f34b37fe700 1 -- 172.21.15.81:0/1365332426 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7f34b8092c90 con 0x7f34c40febc0 2022-09-10T04:26:17.051 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:17.051 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-09-10T04:26:17.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34ac099010 msgr2=0x7f34ac09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34ac099010 0x7f34ac09b4c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34b4005ad0 tx=0x7f34b4005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 msgr2=0x7f34c419a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:17.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c419a4a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34b8006010 tx=0x7f34b800f690 comp rx=0 tx=0).stop 2022-09-10T04:26:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 shutdown_connections 2022-09-10T04:26:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f34c41016b0 0x7f34c41a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:17.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34c4066bf0 0x7f34c4199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34ac099010 0x7f34ac09b4c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 --2- 172.21.15.81:0/1365332426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34c40febc0 0x7f34c419a4a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:17.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 >> 172.21.15.81:0/1365332426 conn(0x7f34c40f82a0 msgr2=0x7f34c40fcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:17.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 shutdown_connections 2022-09-10T04:26:17.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:17.051+0000 7f34c8dff700 1 -- 172.21.15.81:0/1365332426 wait complete. 2022-09-10T04:26:17.057 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 10 2022-09-10T04:26:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:16 smithi110 ceph-mon[97633]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:16 smithi104 ceph-mon[97044]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:16 smithi081 ceph-mon[90133]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 04:26:17 smithi081 systemd[1]: Starting Ceph mds.g for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:17 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1365332426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:18.193 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 04:26:18 smithi081 podman[124929]: 2022-09-10T04:26:18.193 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:17 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1365332426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:17 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1365332426' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:18.666 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 04:26:18 smithi081 bash[124929]: 25648abfd1f2883d9ad850fd40a925e93547093d48c1d1a02bdac214b848b820 2022-09-10T04:26:18.667 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mds[124969]: -- 172.21.15.81:0/22601533 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c910a34820 con 0x55c910a08c00 2022-09-10T04:26:18.668 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: starting mds.g at 2022-09-10T04:26:18.668 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 04:26:18 smithi081 systemd[1]: Started Ceph mds.g for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mon[90133]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:19.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:19.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:19.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:19.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:19.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:18 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:19.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:18 smithi110 ceph-mon[97633]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:19.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:18 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:18 smithi104 ceph-mon[97044]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:18 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:19.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 -- 172.21.15.81:0/755912837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 msgr2=0x7ff7e80ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:19.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/755912837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e80ff410 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009750 tx=0x7ff7d8009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:19.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 -- 172.21.15.81:0/755912837 shutdown_connections 2022-09-10T04:26:19.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/755912837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80ff950 0x7ff7e8106210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/755912837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7e80fdd40 0x7ff7e80fe150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/755912837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e80ff410 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 -- 172.21.15.81:0/755912837 >> 172.21.15.81:0/755912837 conn(0x7ff7e80f9430 msgr2=0x7ff7e80fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:19.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 -- 172.21.15.81:0/755912837 shutdown_connections 2022-09-10T04:26:19.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.357+0000 7ff7f07ff700 1 -- 172.21.15.81:0/755912837 wait complete. 2022-09-10T04:26:19.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.358+0000 7ff7f07ff700 1 Processor -- start 2022-09-10T04:26:19.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 -- start start 2022-09-10T04:26:19.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 0x7ff7e8103eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:19.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e81043f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:19.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7e80ff950 0x7ff7e8100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8101240 con 0x7ff7e80fefc0 2022-09-10T04:26:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e81013b0 con 0x7ff7e80fdd40 2022-09-10T04:26:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7f07ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e8101570 con 0x7ff7e80ff950 2022-09-10T04:26:19.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7ee59b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 0x7ff7e8103eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:19.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7ee59b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 0x7ff7e8103eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38494/0 (socket says 172.21.15.81:38494) 2022-09-10T04:26:19.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7ee59b700 1 -- 172.21.15.81:0/2666178245 learned_addr learned my addr 172.21.15.81:0/2666178245 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:19.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7eed9c700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7e80ff950 0x7ff7e8100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:19.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.359+0000 7ff7edd9a700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e81043f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:19.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7ee59b700 1 -- 172.21.15.81:0/2666178245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7e80ff950 msgr2=0x7ff7e8100aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:19.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7ee59b700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7e80ff950 0x7ff7e8100aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7ee59b700 1 -- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 msgr2=0x7ff7e81043f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:19.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7ee59b700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e81043f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7ee59b700 1 -- 172.21.15.81:0/2666178245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0009730 con 0x7ff7e80fdd40 2022-09-10T04:26:19.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7edd9a700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e81043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:26:19.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7ee59b700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 0x7ff7e8103eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400e710 tx=0x7ff7e400ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:19.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4004220 con 0x7ff7e80fdd40 2022-09-10T04:26:19.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.360+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff7e4004380 con 0x7ff7e80fdd40 2022-09-10T04:26:19.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.361+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d80093b0 con 0x7ff7e80fdd40 2022-09-10T04:26:19.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.361+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e4003c10 con 0x7ff7e80fdd40 2022-09-10T04:26:19.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.361+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e81a2430 con 0x7ff7e80fdd40 2022-09-10T04:26:19.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.362+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e81094f0 con 0x7ff7e80fdd40 2022-09-10T04:26:19.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.364+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff7e401e020 con 0x7ff7e80fdd40 2022-09-10T04:26:19.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.365+0000 7ff7df7fe700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff7d4099010 0x7ff7d409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:19.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.365+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff7e4014070 con 0x7ff7e80fdd40 2022-09-10T04:26:19.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.365+0000 7ff7edd9a700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff7d4099010 0x7ff7d409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:19.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.366+0000 7ff7edd9a700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff7d4099010 0x7ff7d409b4c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8006010 tx=0x7ff7d8009db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:19.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.369+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff7e408ec90 con 0x7ff7e80fdd40 2022-09-10T04:26:19.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.623+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7e81a2d50 con 0x7ff7e80fdd40 2022-09-10T04:26:19.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.624+0000 7ff7df7fe700 1 -- 172.21.15.81:0/2666178245 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7ff7e40922b0 con 0x7ff7e80fdd40 2022-09-10T04:26:19.626 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:19.626 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-09-10T04:26:19.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff7d4099010 msgr2=0x7ff7d409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff7d4099010 0x7ff7d409b4c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8006010 tx=0x7ff7d8009db0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 msgr2=0x7ff7e8103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:19.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 0x7ff7e8103eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7e400e710 tx=0x7ff7e400ea20 comp rx=0 tx=0).stop 2022-09-10T04:26:19.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 shutdown_connections 2022-09-10T04:26:19.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff7d4099010 0x7ff7d409b4c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7e80ff950 0x7ff7e8100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7e80fdd40 0x7ff7e8103eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 --2- 172.21.15.81:0/2666178245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7e80fefc0 0x7ff7e81043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:19.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 >> 172.21.15.81:0/2666178245 conn(0x7ff7e80f9430 msgr2=0x7ff7e80fb720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:19.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 shutdown_connections 2022-09-10T04:26:19.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:19.627+0000 7ff7f07ff700 1 -- 172.21.15.81:0/2666178245 wait complete. 2022-09-10T04:26:19.633 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 11 2022-09-10T04:26:19.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:19 smithi081 ceph-mon[90133]: Deploying daemon mds.a on smithi081 2022-09-10T04:26:19.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:19 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.81:6838/1769781597,v1:172.21.15.81:6839/1769781597] up:boot 2022-09-10T04:26:19.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:19 smithi081 ceph-mon[90133]: fsmap 10 up:standby 2022-09-10T04:26:19.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:19 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:26:19.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:19 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2666178245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:19 smithi110 ceph-mon[97633]: Deploying daemon mds.a on smithi081 2022-09-10T04:26:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:19 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.81:6838/1769781597,v1:172.21.15.81:6839/1769781597] up:boot 2022-09-10T04:26:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:19 smithi110 ceph-mon[97633]: fsmap 10 up:standby 2022-09-10T04:26:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:19 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:26:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:19 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2666178245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:19 smithi104 ceph-mon[97044]: Deploying daemon mds.a on smithi081 2022-09-10T04:26:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:19 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.81:6838/1769781597,v1:172.21.15.81:6839/1769781597] up:boot 2022-09-10T04:26:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:19 smithi104 ceph-mon[97044]: fsmap 10 up:standby 2022-09-10T04:26:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:19 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:26:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:19 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2666178245' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:21.120 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:20 smithi081 ceph-mon[90133]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:20 smithi110 ceph-mon[97633]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:20 smithi104 ceph-mon[97044]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:21.554 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:21 smithi081 systemd[1]: Starting Ceph mds.a for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:21.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:21.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 -- 172.21.15.81:0/3413115964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc101970 msgr2=0x7fc8dc108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:21.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 --2- 172.21.15.81:0/3413115964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc101970 0x7fc8dc108210 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4009bb0 tx=0x7fc8d4009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:21.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 -- 172.21.15.81:0/3413115964 shutdown_connections 2022-09-10T04:26:21.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 --2- 172.21.15.81:0/3413115964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc101970 0x7fc8dc108210 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:21.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 --2- 172.21.15.81:0/3413115964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc0ffde0 0x7fc8dc1001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:21.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 --2- 172.21.15.81:0/3413115964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:21.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 -- 172.21.15.81:0/3413115964 >> 172.21.15.81:0/3413115964 conn(0x7fc8dc0fb4d0 msgr2=0x7fc8dc0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:21.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 -- 172.21.15.81:0/3413115964 shutdown_connections 2022-09-10T04:26:21.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.925+0000 7fc8e139d700 1 -- 172.21.15.81:0/3413115964 wait complete. 2022-09-10T04:26:21.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.926+0000 7fc8e139d700 1 Processor -- start 2022-09-10T04:26:21.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 -- start start 2022-09-10T04:26:21.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8dc0ffde0 0x7fc8dc1958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:21.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:21.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc101970 0x7fc8dc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:21.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc19d440 con 0x7fc8dc0ffde0 2022-09-10T04:26:21.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc19d5b0 con 0x7fc8dc100fc0 2022-09-10T04:26:21.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e139d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc19d770 con 0x7fc8dc101970 2022-09-10T04:26:21.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:21.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8db7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:46888/0 (socket says 172.21.15.81:46888) 2022-09-10T04:26:21.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8db7fe700 1 -- 172.21.15.81:0/123223591 learned_addr learned my addr 172.21.15.81:0/123223591 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:21.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8e0b9c700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc101970 0x7fc8dc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:21.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.927+0000 7fc8dbfff700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8dc0ffde0 0x7fc8dc1958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:21.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8db7fe700 1 -- 172.21.15.81:0/123223591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc101970 msgr2=0x7fc8dc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:21.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8db7fe700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc101970 0x7fc8dc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:21.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8db7fe700 1 -- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8dc0ffde0 msgr2=0x7fc8dc1958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:21.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8db7fe700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8dc0ffde0 0x7fc8dc1958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:21.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8db7fe700 1 -- 172.21.15.81:0/123223591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d40097e0 con 0x7fc8dc100fc0 2022-09-10T04:26:21.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8e0b9c700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc101970 0x7fc8dc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:21.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8db7fe700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc195e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e740 tx=0x7fc8d000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:21.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0005220 con 0x7fc8dc100fc0 2022-09-10T04:26:21.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc8d0005380 con 0x7fc8dc100fc0 2022-09-10T04:26:21.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc19da50 con 0x7fc8dc100fc0 2022-09-10T04:26:21.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0004b70 con 0x7fc8dc100fc0 2022-09-10T04:26:21.949 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.928+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc19df60 con 0x7fc8dc100fc0 2022-09-10T04:26:21.950 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.929+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc10b4a0 con 0x7fc8dc100fc0 2022-09-10T04:26:21.950 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.931+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc8d001e020 con 0x7fc8dc100fc0 2022-09-10T04:26:21.952 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.932+0000 7fc8d97fa700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc8c4099010 0x7fc8c409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:21.952 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.932+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fc8d0014070 con 0x7fc8dc100fc0 2022-09-10T04:26:21.953 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.933+0000 7fc8dbfff700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc8c4099010 0x7fc8c409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:21.953 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.933+0000 7fc8dbfff700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc8c4099010 0x7fc8c409b4c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005fd0 tx=0x7fc8cc005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:21.953 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:21.936+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc8d008ebb0 con 0x7fc8dc100fc0 2022-09-10T04:26:22.004 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:21 smithi081 podman[125331]: 2022-09-10T04:26:22.194 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.193+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8dc10b630 con 0x7fc8dc100fc0 2022-09-10T04:26:22.195 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.194+0000 7fc8d97fa700 1 -- 172.21.15.81:0/123223591 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8088 (secure 0 0 0) 0x7fc8dc10b630 con 0x7fc8dc100fc0 2022-09-10T04:26:22.196 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:22.196 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2022-09-10T04:26:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.196+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc8c4099010 msgr2=0x7fc8c409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:22.198 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.196+0000 7fc8e139d700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc8c4099010 0x7fc8c409b4c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc005fd0 tx=0x7fc8cc005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:22.199 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.196+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 msgr2=0x7fc8dc195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:22.199 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.196+0000 7fc8e139d700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc195e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e740 tx=0x7fc8d000ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:22.199 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 shutdown_connections 2022-09-10T04:26:22.199 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc8c4099010 0x7fc8c409b4c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:22.200 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8dc101970 0x7fc8dc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:22.200 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8dc0ffde0 0x7fc8dc1958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:22.200 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 --2- 172.21.15.81:0/123223591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc8dc100fc0 0x7fc8dc195e00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:22.200 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 >> 172.21.15.81:0/123223591 conn(0x7fc8dc0fb4d0 msgr2=0x7fc8dc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:22.201 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 shutdown_connections 2022-09-10T04:26:22.201 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:22.197+0000 7fc8e139d700 1 -- 172.21.15.81:0/123223591 wait complete. 2022-09-10T04:26:22.202 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 11 2022-09-10T04:26:22.306 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:22 smithi081 bash[125331]: fe187550bf09751713eb1c56b55d2b53b9136860b5a48d1e7bba514206d80554 2022-09-10T04:26:22.306 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mds[125414]: -- 172.21.15.81:0/647921319 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564d31568820 con 0x564d3153c800 2022-09-10T04:26:22.306 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-a[125370]: starting mds.a at 2022-09-10T04:26:22.307 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:22 smithi081 systemd[1]: Started Ceph mds.a for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:23.192 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:23.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:23.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:23.193 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:23.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:23.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:23.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:22 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/123223591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:23.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:23.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:23.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:23.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:23.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:23.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:22 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/123223591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2022-09-10T04:26:23.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2022-09-10T04:26:23.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:23.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:22 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/123223591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:23.911 INFO:journalctl@ceph.mds.b.smithi104.stdout:Sep 10 04:26:23 smithi104 systemd[1]: Starting Ceph mds.b for 70cf895c-30bf-11ed-8431-001a4aab830c... 2022-09-10T04:26:24.209 INFO:journalctl@ceph.mds.b.smithi104.stdout:Sep 10 04:26:23 smithi104 podman[123882]: 2022-09-10T04:26:24.209 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:23 smithi104 ceph-mon[97044]: Deploying daemon mds.b on smithi104 2022-09-10T04:26:24.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:23 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] up:boot 2022-09-10T04:26:24.210 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:23 smithi104 ceph-mon[97044]: fsmap 11 up:standby 2022-09-10T04:26:24.211 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:23 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-10T04:26:24.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:24.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:23 smithi081 ceph-mon[90133]: Deploying daemon mds.b on smithi104 2022-09-10T04:26:24.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:23 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] up:boot 2022-09-10T04:26:24.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:23 smithi081 ceph-mon[90133]: fsmap 11 up:standby 2022-09-10T04:26:24.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:23 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-10T04:26:24.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:23 smithi110 ceph-mon[97633]: Deploying daemon mds.b on smithi104 2022-09-10T04:26:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:23 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] up:boot 2022-09-10T04:26:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:23 smithi110 ceph-mon[97633]: fsmap 11 up:standby 2022-09-10T04:26:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:23 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-10T04:26:24.466 INFO:journalctl@ceph.mds.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mds[123921]: -- 172.21.15.104:0/3472050755 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ae6896820 con 0x564ae686a800 2022-09-10T04:26:24.466 INFO:journalctl@ceph.mds.b.smithi104.stdout:Sep 10 04:26:24 smithi104 bash[123882]: 2d318a5ae4e057b0d31ad4bc7f7265f570e89e1d4ede486c5b24136186a69786 2022-09-10T04:26:24.467 INFO:journalctl@ceph.mds.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-b[123898]: starting mds.b at 2022-09-10T04:26:24.467 INFO:journalctl@ceph.mds.b.smithi104.stdout:Sep 10 04:26:24 smithi104 systemd[1]: Started Ceph mds.b for 70cf895c-30bf-11ed-8431-001a4aab830c. 2022-09-10T04:26:24.491 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 -- 172.21.15.81:0/1569910361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 msgr2=0x7f1f58101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 --2- 172.21.15.81:0/1569910361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 0x7f1f58101020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009810 tx=0x7f1f48009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 -- 172.21.15.81:0/1569910361 shutdown_connections 2022-09-10T04:26:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 --2- 172.21.15.81:0/1569910361 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f581016b0 0x7f1f58101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 --2- 172.21.15.81:0/1569910361 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f580fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 --2- 172.21.15.81:0/1569910361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 0x7f1f58101020 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 -- 172.21.15.81:0/1569910361 >> 172.21.15.81:0/1569910361 conn(0x7f1f580f82a0 msgr2=0x7f1f580fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.484+0000 7f1f607ff700 1 -- 172.21.15.81:0/1569910361 shutdown_connections 2022-09-10T04:26:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.485+0000 7f1f607ff700 1 -- 172.21.15.81:0/1569910361 wait complete. 2022-09-10T04:26:24.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 Processor -- start 2022-09-10T04:26:24.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 -- start start 2022-09-10T04:26:24.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f58199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:24.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 0x7f1f5819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:24.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f581016b0 0x7f1f581a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:24.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581a1b20 con 0x7f1f580febc0 2022-09-10T04:26:24.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581a1c90 con 0x7f1f581016b0 2022-09-10T04:26:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f607ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581a1e50 con 0x7f1f58066bf0 2022-09-10T04:26:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f5dd9a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 0x7f1f5819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f5e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f58199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.486+0000 7f1f5e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f58199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36590/0 (socket says 172.21.15.81:36590) 2022-09-10T04:26:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 -- 172.21.15.81:0/516894725 learned_addr learned my addr 172.21.15.81:0/516894725 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5ed9c700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f581016b0 0x7f1f581a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 -- 172.21.15.81:0/516894725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f581016b0 msgr2=0x7f1f581a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f581016b0 0x7f1f581a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 -- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 msgr2=0x7f1f5819a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 0x7f1f5819a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 -- 172.21.15.81:0/516894725 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f540093b0 con 0x7f1f58066bf0 2022-09-10T04:26:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f5e59b700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f58199fa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400e9f0 tx=0x7f1f5400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.487+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54009750 con 0x7f1f58066bf0 2022-09-10T04:26:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.488+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f50009730 con 0x7f1f58066bf0 2022-09-10T04:26:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.488+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1f54004220 con 0x7f1f58066bf0 2022-09-10T04:26:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.488+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54003cf0 con 0x7f1f58066bf0 2022-09-10T04:26:24.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.488+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f48009470 con 0x7f1f58066bf0 2022-09-10T04:26:24.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.489+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5810fad0 con 0x7f1f58066bf0 2022-09-10T04:26:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.490+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1f5401e020 con 0x7f1f58066bf0 2022-09-10T04:26:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.492+0000 7f1f4f7fe700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f44099010 0x7f1f4409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.492+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f1f54014070 con 0x7f1f58066bf0 2022-09-10T04:26:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.492+0000 7f1f5dd9a700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f44099010 0x7f1f4409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:24.505 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.492+0000 7f1f5dd9a700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f44099010 0x7f1f4409b4c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800b4f0 tx=0x7f1f48009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:24.505 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.496+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1f5408f010 con 0x7f1f58066bf0 2022-09-10T04:26:24.763 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.762+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f580f9c90 con 0x7f1f58066bf0 2022-09-10T04:26:24.764 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.762+0000 7f1f4f7fe700 1 -- 172.21.15.81:0/516894725 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8843 (secure 0 0 0) 0x7f1f54092630 con 0x7f1f58066bf0 2022-09-10T04:26:24.765 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:24.765 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2022-09-10T04:26:24.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.765+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f44099010 msgr2=0x7f1f4409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:24.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.765+0000 7f1f607ff700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f44099010 0x7f1f4409b4c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800b4f0 tx=0x7f1f48009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.765+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 msgr2=0x7f1f58199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:24.772 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.765+0000 7f1f607ff700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f58199fa0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400e9f0 tx=0x7f1f5400ed00 comp rx=0 tx=0).stop 2022-09-10T04:26:24.772 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 shutdown_connections 2022-09-10T04:26:24.772 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f581016b0 0x7f1f581a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.772 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f44099010 0x7f1f4409b4c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f58066bf0 0x7f1f58199fa0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 --2- 172.21.15.81:0/516894725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f580febc0 0x7f1f5819a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:24.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 >> 172.21.15.81:0/516894725 conn(0x7f1f580f82a0 msgr2=0x7f1f580fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:24.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 shutdown_connections 2022-09-10T04:26:24.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:24.766+0000 7f1f607ff700 1 -- 172.21.15.81:0/516894725 wait complete. 2022-09-10T04:26:24.774 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 12 2022-09-10T04:26:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:26:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:25.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:26:25.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:24 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/516894725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:25.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:26:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:26:25.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:24 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/516894725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:26:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:25.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:26:25.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:24 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/516894725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:26 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] up:boot 2022-09-10T04:26:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:26 smithi104 ceph-mon[97044]: fsmap 12 up:standby 2022-09-10T04:26:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:26 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-10T04:26:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:26 smithi104 ceph-mon[97044]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:26 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] up:boot 2022-09-10T04:26:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:26 smithi081 ceph-mon[90133]: fsmap 12 up:standby 2022-09-10T04:26:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:26 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-10T04:26:26.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:26 smithi081 ceph-mon[90133]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:26 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] up:boot 2022-09-10T04:26:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:26 smithi110 ceph-mon[97633]: fsmap 12 up:standby 2022-09-10T04:26:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:26 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-10T04:26:26.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:26 smithi110 ceph-mon[97633]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:26.798 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:27.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 -- 172.21.15.81:0/2899442188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80071650 msgr2=0x7f5a80071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 --2- 172.21.15.81:0/2899442188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80071650 0x7f5a80071ac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009750 tx=0x7f5a70009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:27.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 -- 172.21.15.81:0/2899442188 shutdown_connections 2022-09-10T04:26:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 --2- 172.21.15.81:0/2899442188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80105ae0 0x7f5a80107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 --2- 172.21.15.81:0/2899442188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 0x7f5a80071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 --2- 172.21.15.81:0/2899442188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80071650 0x7f5a80071ac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 -- 172.21.15.81:0/2899442188 >> 172.21.15.81:0/2899442188 conn(0x7f5a800f9260 msgr2=0x7f5a800fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 -- 172.21.15.81:0/2899442188 shutdown_connections 2022-09-10T04:26:27.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.040+0000 7f5a8699d700 1 -- 172.21.15.81:0/2899442188 wait complete. 2022-09-10T04:26:27.048 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.041+0000 7f5a8699d700 1 Processor -- start 2022-09-10T04:26:27.048 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.041+0000 7f5a8699d700 1 -- start start 2022-09-10T04:26:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8699d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 0x7f5a801959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8699d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 0x7f5a80195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8699d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80105ae0 0x7f5a8019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8699d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8019d520 con 0x7f5a80071650 2022-09-10T04:26:27.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8699d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8019d690 con 0x7f5a80072b90 2022-09-10T04:26:27.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8699d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8019d850 con 0x7f5a80105ae0 2022-09-10T04:26:27.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 0x7f5a801959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 0x7f5a801959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56222/0 (socket says 172.21.15.81:56222) 2022-09-10T04:26:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 -- 172.21.15.81:0/677600400 learned_addr learned my addr 172.21.15.81:0/677600400 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8519a700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 0x7f5a80195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8619c700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80105ae0 0x7f5a8019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 -- 172.21.15.81:0/677600400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80105ae0 msgr2=0x7f5a8019cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80105ae0 0x7f5a8019cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 -- 172.21.15.81:0/677600400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 msgr2=0x7f5a80195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 0x7f5a80195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8599b700 1 -- 172.21.15.81:0/677600400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a700093b0 con 0x7f5a80071650 2022-09-10T04:26:27.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.042+0000 7f5a8519a700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 0x7f5a80195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.043+0000 7f5a8599b700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 0x7f5a801959a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00d6a0 tx=0x7f5a7c00d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:27.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.043+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00f890 con 0x7f5a80071650 2022-09-10T04:26:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.043+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8019db30 con 0x7f5a80071650 2022-09-10T04:26:27.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.043+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5a7c00fe90 con 0x7f5a80071650 2022-09-10T04:26:27.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.043+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c019360 con 0x7f5a80071650 2022-09-10T04:26:27.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.043+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8019e090 con 0x7f5a80071650 2022-09-10T04:26:27.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.044+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8010b540 con 0x7f5a80071650 2022-09-10T04:26:27.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.046+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f5a7c022070 con 0x7f5a80071650 2022-09-10T04:26:27.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.048+0000 7f5a76ffd700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5a6c099090 0x7f5a6c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.048+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f5a7c0c1c40 con 0x7f5a80071650 2022-09-10T04:26:27.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.048+0000 7f5a8519a700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5a6c099090 0x7f5a6c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.049+0000 7f5a8519a700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5a6c099090 0x7f5a6c09b540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000f500 tx=0x7f5a70005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:27.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.053+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f5a7c08e730 con 0x7f5a80071650 2022-09-10T04:26:27.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.318+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a8005f120 con 0x7f5a80071650 2022-09-10T04:26:27.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.319+0000 7f5a76ffd700 1 -- 172.21.15.81:0/677600400 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9604 (secure 0 0 0) 0x7f5a7c091d50 con 0x7f5a80071650 2022-09-10T04:26:27.321 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:27.322 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[]} 2022-09-10T04:26:27.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.321+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5a6c099090 msgr2=0x7f5a6c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.321+0000 7f5a8699d700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5a6c099090 0x7f5a6c09b540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000f500 tx=0x7f5a70005f90 comp rx=0 tx=0).stop 2022-09-10T04:26:27.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.321+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 msgr2=0x7f5a801959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.321+0000 7f5a8699d700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 0x7f5a801959a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00d6a0 tx=0x7f5a7c00d9b0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 shutdown_connections 2022-09-10T04:26:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5a6c099090 0x7f5a6c09b540 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a80105ae0 0x7f5a8019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a80071650 0x7f5a801959a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 --2- 172.21.15.81:0/677600400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5a80072b90 0x7f5a80195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 >> 172.21.15.81:0/677600400 conn(0x7f5a800f9260 msgr2=0x7f5a800fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:27.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 shutdown_connections 2022-09-10T04:26:27.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.322+0000 7f5a8699d700 1 -- 172.21.15.81:0/677600400 wait complete. 2022-09-10T04:26:27.328 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 13 2022-09-10T04:26:27.352 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:27.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 -- 172.21.15.81:0/3362545838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b00febc0 msgr2=0x7fb2b0101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/3362545838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b00febc0 0x7fb2b0101020 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009810 tx=0x7fb2a0009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:27.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 -- 172.21.15.81:0/3362545838 shutdown_connections 2022-09-10T04:26:27.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/3362545838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b01016b0 0x7fb2b0101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/3362545838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b0066bf0 0x7fb2b00fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/3362545838 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b00febc0 0x7fb2b0101020 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 -- 172.21.15.81:0/3362545838 >> 172.21.15.81:0/3362545838 conn(0x7fb2b00f82a0 msgr2=0x7fb2b00fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:27.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 -- 172.21.15.81:0/3362545838 shutdown_connections 2022-09-10T04:26:27.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.585+0000 7fb2b83ff700 1 -- 172.21.15.81:0/3362545838 wait complete. 2022-09-10T04:26:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.587+0000 7fb2b83ff700 1 Processor -- start 2022-09-10T04:26:27.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.587+0000 7fb2b83ff700 1 -- start start 2022-09-10T04:26:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b83ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 0x7fb2b0199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b83ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 0x7fb2b019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b83ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b01016b0 0x7fb2b01a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b83ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b01a1a40 con 0x7fb2b00febc0 2022-09-10T04:26:27.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b83ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b01a1bb0 con 0x7fb2b0066bf0 2022-09-10T04:26:27.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b83ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b01a1d70 con 0x7fb2b01016b0 2022-09-10T04:26:27.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 0x7fb2b019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b619b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 0x7fb2b0199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 0x7fb2b019a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56252/0 (socket says 172.21.15.81:56252) 2022-09-10T04:26:27.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 -- 172.21.15.81:0/2733135572 learned_addr learned my addr 172.21.15.81:0/2733135572 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:27.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b619b700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 0x7fb2b0199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:46980/0 (socket says 172.21.15.81:46980) 2022-09-10T04:26:27.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b699c700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b01016b0 0x7fb2b01a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 -- 172.21.15.81:0/2733135572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b01016b0 msgr2=0x7fb2b01a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b01016b0 0x7fb2b01a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 -- 172.21.15.81:0/2733135572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 msgr2=0x7fb2b0199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 0x7fb2b0199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b599a700 1 -- 172.21.15.81:0/2733135572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a0009470 con 0x7fb2b00febc0 2022-09-10T04:26:27.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.588+0000 7fb2b619b700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 0x7fb2b0199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:27.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.589+0000 7fb2b599a700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 0x7fb2b019a400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0000c00 tx=0x7fb2a000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:27.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.589+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a001d070 con 0x7fb2b00febc0 2022-09-10T04:26:27.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.589+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb2a000f8b0 con 0x7fb2b00febc0 2022-09-10T04:26:27.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.589+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b01a1ff0 con 0x7fb2b00febc0 2022-09-10T04:26:27.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.589+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0004870 con 0x7fb2b00febc0 2022-09-10T04:26:27.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.589+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b01a2410 con 0x7fb2b00febc0 2022-09-10T04:26:27.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.591+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b010fa80 con 0x7fb2b00febc0 2022-09-10T04:26:27.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.592+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb2a0028030 con 0x7fb2b00febc0 2022-09-10T04:26:27.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.593+0000 7fb2a77fe700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb29c099010 0x7fb29c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:27.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.593+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7fb2a00c2d00 con 0x7fb2b00febc0 2022-09-10T04:26:27.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.594+0000 7fb2b619b700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb29c099010 0x7fb29c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:27.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.594+0000 7fb2b619b700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb29c099010 0x7fb29c09b4c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0066fc0 tx=0x7fb2ac005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:27.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.597+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb2a008f7f0 con 0x7fb2b00febc0 2022-09-10T04:26:27.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.799+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2b005fb40 con 0x7fb2b00febc0 2022-09-10T04:26:27.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.800+0000 7fb2a77fe700 1 -- 172.21.15.81:0/2733135572 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18112 (secure 0 0 0) 0x7fb2a0092e10 con 0x7fb2b00febc0 2022-09-10T04:26:27.804 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:27.804 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":82,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:25:44.149785+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:27.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb29c099010 msgr2=0x7fb29c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb29c099010 0x7fb29c09b4c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0066fc0 tx=0x7fb2ac005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 msgr2=0x7fb2b019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:27.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 0x7fb2b019a400 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0000c00 tx=0x7fb2a000f690 comp rx=0 tx=0).stop 2022-09-10T04:26:27.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 shutdown_connections 2022-09-10T04:26:27.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb29c099010 0x7fb29c09b4c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b01016b0 0x7fb2b01a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb2b0066bf0 0x7fb2b0199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 --2- 172.21.15.81:0/2733135572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b00febc0 0x7fb2b019a400 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:27.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 >> 172.21.15.81:0/2733135572 conn(0x7fb2b00f82a0 msgr2=0x7fb2b00fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 shutdown_connections 2022-09-10T04:26:27.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:27.804+0000 7fb2b83ff700 1 -- 172.21.15.81:0/2733135572 wait complete. 2022-09-10T04:26:27.839 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2022-09-10T04:26:27.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:26:28.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.090+0000 7f33d659d700 1 -- 172.21.15.81:0/1843918932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 msgr2=0x7f33d00fdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.090+0000 7f33d659d700 1 --2- 172.21.15.81:0/1843918932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d00fdf50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009810 tx=0x7f33cc009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:28.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 -- 172.21.15.81:0/1843918932 shutdown_connections 2022-09-10T04:26:28.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 --2- 172.21.15.81:0/1843918932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00ff510 0x7f33d01060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 --2- 172.21.15.81:0/1843918932 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d00fdf50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 --2- 172.21.15.81:0/1843918932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33d00fe490 0x7f33d00fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 -- 172.21.15.81:0/1843918932 >> 172.21.15.81:0/1843918932 conn(0x7f33d00f9230 msgr2=0x7f33d00fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:28.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 -- 172.21.15.81:0/1843918932 shutdown_connections 2022-09-10T04:26:28.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.091+0000 7f33d659d700 1 -- 172.21.15.81:0/1843918932 wait complete. 2022-09-10T04:26:28.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 Processor -- start 2022-09-10T04:26:28.105 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 -- start start 2022-09-10T04:26:28.105 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.106 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00fe490 0x7f33d00710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33d00ff510 0x7f33d0071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.107 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d0071e10 con 0x7f33d00fe490 2022-09-10T04:26:28.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d0071f80 con 0x7f33d00fdb40 2022-09-10T04:26:28.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.093+0000 7f33d659d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d019d820 con 0x7f33d00ff510 2022-09-10T04:26:28.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d0072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:47000/0 (socket says 172.21.15.81:47000) 2022-09-10T04:26:28.109 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 -- 172.21.15.81:0/390304374 learned_addr learned my addr 172.21.15.81:0/390304374 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:28.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d5d9c700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33d00ff510 0x7f33d0071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.110 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d4d9a700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00fe490 0x7f33d00710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 -- 172.21.15.81:0/390304374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33d00ff510 msgr2=0x7f33d0071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33d00ff510 0x7f33d0071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 -- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00fe490 msgr2=0x7f33d00710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00fe490 0x7f33d00710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 -- 172.21.15.81:0/390304374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00093b0 con 0x7f33d00fdb40 2022-09-10T04:26:28.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d4d9a700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00fe490 0x7f33d00710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:26:28.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.094+0000 7f33d559b700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d0072a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00f690 tx=0x7f33cc00f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.095+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc01d070 con 0x7f33d00fdb40 2022-09-10T04:26:28.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.095+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f33cc0054f0 con 0x7f33d00fdb40 2022-09-10T04:26:28.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.095+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc004860 con 0x7f33d00fdb40 2022-09-10T04:26:28.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.095+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc009470 con 0x7f33d00fdb40 2022-09-10T04:26:28.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.095+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d019dd70 con 0x7f33d00fdb40 2022-09-10T04:26:28.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.096+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d0109330 con 0x7f33d00fdb40 2022-09-10T04:26:28.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.098+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f33cc028030 con 0x7f33d00fdb40 2022-09-10T04:26:28.118 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.099+0000 7f33c67fc700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33bc099010 0x7f33bc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.118 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.100+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f33cc0c2c90 con 0x7f33d00fdb40 2022-09-10T04:26:28.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.100+0000 7f33d4d9a700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33bc099010 0x7f33bc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.100+0000 7f33d4d9a700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33bc099010 0x7f33bc09b4c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f33d00fe860 tx=0x7f33c0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:28.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.101+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f33cc08f780 con 0x7f33d00fdb40 2022-09-10T04:26:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.322+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33d01094c0 con 0x7f33d00fdb40 2022-09-10T04:26:28.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.323+0000 7f33c67fc700 1 -- 172.21.15.81:0/390304374 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33d01094c0 con 0x7f33d00fdb40 2022-09-10T04:26:28.326 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:26:28.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.325+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33bc099010 msgr2=0x7f33bc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.325+0000 7f33d659d700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33bc099010 0x7f33bc09b4c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f33d00fe860 tx=0x7f33c0005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.325+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 msgr2=0x7f33d0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.325+0000 7f33d659d700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d0072a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00f690 tx=0x7f33cc00f6c0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.325+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 shutdown_connections 2022-09-10T04:26:28.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.325+0000 7f33d659d700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f33bc099010 0x7f33bc09b4c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.326+0000 7f33d659d700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33d00ff510 0x7f33d0071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.326+0000 7f33d659d700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33d00fdb40 0x7f33d0072a90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.326+0000 7f33d659d700 1 --2- 172.21.15.81:0/390304374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33d00fe490 0x7f33d00710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.326+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 >> 172.21.15.81:0/390304374 conn(0x7f33d00f9230 msgr2=0x7f33d00fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:28.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.326+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 shutdown_connections 2022-09-10T04:26:28.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.326+0000 7f33d659d700 1 -- 172.21.15.81:0/390304374 wait complete. 2022-09-10T04:26:28.358 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-09-10T04:26:28.439 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.439 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.439 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.440 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.440 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/677600400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:28.440 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:28.440 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:28 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2733135572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/677600400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:28.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:28.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:28 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2733135572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:28.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.588+0000 7f428c9fe700 1 -- 172.21.15.81:0/2297831652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 msgr2=0x7f4288103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.588+0000 7f428c9fe700 1 --2- 172.21.15.81:0/2297831652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288103340 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f427c009810 tx=0x7f427c009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:28.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 -- 172.21.15.81:0/2297831652 shutdown_connections 2022-09-10T04:26:28.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 --2- 172.21.15.81:0/2297831652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4288103880 0x7f4288105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 --2- 172.21.15.81:0/2297831652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42880fe640 0x7f4288100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 --2- 172.21.15.81:0/2297831652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288103340 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 -- 172.21.15.81:0/2297831652 >> 172.21.15.81:0/2297831652 conn(0x7f42880f8160 msgr2=0x7f42880fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:28.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 -- 172.21.15.81:0/2297831652 shutdown_connections 2022-09-10T04:26:28.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.589+0000 7f428c9fe700 1 -- 172.21.15.81:0/2297831652 wait complete. 2022-09-10T04:26:28.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.590+0000 7f428c9fe700 1 Processor -- start 2022-09-10T04:26:28.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.590+0000 7f428c9fe700 1 -- start start 2022-09-10T04:26:28.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.590+0000 7f428c9fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42880fe640 0x7f4288193700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.590+0000 7f428c9fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288193c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428c9fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4288103880 0x7f428819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428c9fe700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428819b280 con 0x7f4288100f60 2022-09-10T04:26:28.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428c9fe700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428819b3f0 con 0x7f42880fe640 2022-09-10T04:26:28.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428c9fe700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f428819b5b0 con 0x7f4288103880 2022-09-10T04:26:28.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288193c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f4286d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42880fe640 0x7f4288193700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288193c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56290/0 (socket says 172.21.15.81:56290) 2022-09-10T04:26:28.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 -- 172.21.15.81:0/388546283 learned_addr learned my addr 172.21.15.81:0/388546283 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:28.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428759e700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4288103880 0x7f428819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 -- 172.21.15.81:0/388546283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4288103880 msgr2=0x7f428819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4288103880 0x7f428819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 -- 172.21.15.81:0/388546283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42880fe640 msgr2=0x7f4288193700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42880fe640 0x7f4288193700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.591+0000 7f428659c700 1 -- 172.21.15.81:0/388546283 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c009470 con 0x7f4288100f60 2022-09-10T04:26:28.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.592+0000 7f428659c700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288193c40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f427c00f690 tx=0x7f427c00f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:28.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.592+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c01d070 con 0x7f4288100f60 2022-09-10T04:26:28.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.592+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f428819b830 con 0x7f4288100f60 2022-09-10T04:26:28.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.592+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f427c00f8e0 con 0x7f4288100f60 2022-09-10T04:26:28.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.592+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c0047f0 con 0x7f4288100f60 2022-09-10T04:26:28.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.592+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f428819bce0 con 0x7f4288100f60 2022-09-10T04:26:28.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.593+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42881092c0 con 0x7f4288100f60 2022-09-10T04:26:28.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.595+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f427c028030 con 0x7f4288100f60 2022-09-10T04:26:28.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.596+0000 7f426ffff700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4270099010 0x7f427009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:28.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.596+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7f427c0c2c10 con 0x7f4288100f60 2022-09-10T04:26:28.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.596+0000 7f4286d9d700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4270099010 0x7f427009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:28.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.597+0000 7f4286d9d700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4270099010 0x7f427009b4c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4278005fd0 tx=0x7f4278005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:28.619 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.600+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f427c08f700 con 0x7f4288100f60 2022-09-10T04:26:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/677600400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:28 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2733135572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:28.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.804+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f428805f990 con 0x7f4288100f60 2022-09-10T04:26:28.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.805+0000 7f426ffff700 1 -- 172.21.15.81:0/388546283 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f427c092d20 con 0x7f4288100f60 2022-09-10T04:26:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:28.807 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2022-09-10T04:26:28.807 INFO:teuthology.orchestra.run.smithi081.stdout: { 2022-09-10T04:26:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2022-09-10T04:26:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2022-09-10T04:26:28.808 INFO:teuthology.orchestra.run.smithi081.stdout: } 2022-09-10T04:26:28.808 INFO:teuthology.orchestra.run.smithi081.stdout:] 2022-09-10T04:26:28.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4270099010 msgr2=0x7f427009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4270099010 0x7f427009b4c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4278005fd0 tx=0x7f4278005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 msgr2=0x7f4288193c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:28.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288193c40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f427c00f690 tx=0x7f427c00f6c0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 shutdown_connections 2022-09-10T04:26:28.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4270099010 0x7f427009b4c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4288103880 0x7f428819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42880fe640 0x7f4288193700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 --2- 172.21.15.81:0/388546283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4288100f60 0x7f4288193c40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:28.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 >> 172.21.15.81:0/388546283 conn(0x7f42880f8160 msgr2=0x7f42880ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:28.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 shutdown_connections 2022-09-10T04:26:28.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:28.808+0000 7f428c9fe700 1 -- 172.21.15.81:0/388546283 wait complete. 2022-09-10T04:26:28.838 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-09-10T04:26:28.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2022-09-10T04:26:29.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 -- 172.21.15.81:0/526405400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2980fe640 msgr2=0x7ff298100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:29.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 --2- 172.21.15.81:0/526405400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2980fe640 0x7ff298100a20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff288009810 tx=0x7ff288009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:29.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 -- 172.21.15.81:0/526405400 shutdown_connections 2022-09-10T04:26:29.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 --2- 172.21.15.81:0/526405400 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff298103880 0x7ff298105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:29.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 --2- 172.21.15.81:0/526405400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2980fe640 0x7ff298100a20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:29.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 --2- 172.21.15.81:0/526405400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff298100f60 0x7ff298103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:29.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 -- 172.21.15.81:0/526405400 >> 172.21.15.81:0/526405400 conn(0x7ff2980f8160 msgr2=0x7ff2980fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:29.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 -- 172.21.15.81:0/526405400 shutdown_connections 2022-09-10T04:26:29.086 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.081+0000 7ff297fff700 1 -- 172.21.15.81:0/526405400 wait complete. 2022-09-10T04:26:29.089 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.082+0000 7ff297fff700 1 Processor -- start 2022-09-10T04:26:29.089 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.082+0000 7ff297fff700 1 -- start start 2022-09-10T04:26:29.089 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff297fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980fe640 0x7ff298193710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:29.090 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff297fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 0x7ff298193c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:29.091 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff297fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff298103880 0x7ff29819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:29.091 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff297fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29819b240 con 0x7ff2980fe640 2022-09-10T04:26:29.091 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff297fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29819b3b0 con 0x7ff298100f60 2022-09-10T04:26:29.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff297fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29819b570 con 0x7ff298103880 2022-09-10T04:26:29.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff296ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980fe640 0x7ff298193710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:29.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff2967fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 0x7ff298193c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:29.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff2967fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 0x7ff298193c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:47036/0 (socket says 172.21.15.81:47036) 2022-09-10T04:26:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff2967fc700 1 -- 172.21.15.81:0/1979857017 learned_addr learned my addr 172.21.15.81:0/1979857017 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.083+0000 7ff2977fe700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff298103880 0x7ff29819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff2967fc700 1 -- 172.21.15.81:0/1979857017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff298103880 msgr2=0x7ff29819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff2967fc700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff298103880 0x7ff29819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:29.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff2967fc700 1 -- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980fe640 msgr2=0x7ff298193710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff2967fc700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980fe640 0x7ff298193710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:29.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff2967fc700 1 -- 172.21.15.81:0/1979857017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff288009470 con 0x7ff298100f60 2022-09-10T04:26:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff296ffd700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980fe640 0x7ff298193710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff2967fc700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 0x7ff298193c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e740 tx=0x7ff28c00ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c005220 con 0x7ff298100f60 2022-09-10T04:26:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff28c005380 con 0x7ff298100f60 2022-09-10T04:26:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2980fc490 con 0x7ff298100f60 2022-09-10T04:26:29.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c004b70 con 0x7ff298100f60 2022-09-10T04:26:29.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.084+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2980fc9f0 con 0x7ff298100f60 2022-09-10T04:26:29.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.085+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff298109270 con 0x7ff298100f60 2022-09-10T04:26:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.087+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff28c01e020 con 0x7ff298100f60 2022-09-10T04:26:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.089+0000 7ff29c9fd700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff280099090 0x7ff28009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:29.100 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.089+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7123+0+0 (secure 0 0 0) 0x7ff28c014070 con 0x7ff298100f60 2022-09-10T04:26:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.089+0000 7ff296ffd700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff280099090 0x7ff28009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.090+0000 7ff296ffd700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff280099090 0x7ff28009b540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff28800b4f0 tx=0x7ff288009e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.091+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff28c08eef0 con 0x7ff298100f60 2022-09-10T04:26:29.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:29.289+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff2980fab60 con 0x7ff298100f60 2022-09-10T04:26:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:29 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/390304374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:29.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:29 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/388546283' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-09-10T04:26:29.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:29 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:29.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:29 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:29 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/390304374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:29 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/388546283' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-09-10T04:26:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:29 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:29 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:29 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/390304374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:29 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/388546283' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2022-09-10T04:26:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:29 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:29 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:30.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.301+0000 7ff29c9fd700 1 -- 172.21.15.81:0/1979857017 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff28c092510 con 0x7ff298100f60 2022-09-10T04:26:30.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff280099090 msgr2=0x7ff28009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:30.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff280099090 0x7ff28009b540 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff28800b4f0 tx=0x7ff288009e70 comp rx=0 tx=0).stop 2022-09-10T04:26:30.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 msgr2=0x7ff298193c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:30.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 0x7ff298193c50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e740 tx=0x7ff28c00ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:30.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 shutdown_connections 2022-09-10T04:26:30.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff280099090 0x7ff28009b540 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff298103880 0x7ff29819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980fe640 0x7ff298193710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 --2- 172.21.15.81:0/1979857017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff298100f60 0x7ff298193c50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 >> 172.21.15.81:0/1979857017 conn(0x7ff2980f8160 msgr2=0x7ff2980fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:30.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 shutdown_connections 2022-09-10T04:26:30.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.306+0000 7ff297fff700 1 -- 172.21.15.81:0/1979857017 wait complete. 2022-09-10T04:26:30.318 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2022-09-10T04:26:30.334 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1979857017' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-10T04:26:30.334 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-10T04:26:30.334 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:30.335 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:30.335 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:26:30.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:30.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:30 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:26:30.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2022-09-10T04:26:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1979857017' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-10T04:26:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-10T04:26:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:30.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:26:30.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:30.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:30 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:26:30.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.593+0000 7f7980f9d700 1 -- 172.21.15.81:0/1309957348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c103880 msgr2=0x7f797c105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.593+0000 7f7980f9d700 1 --2- 172.21.15.81:0/1309957348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c103880 0x7f797c105c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7974009b00 tx=0x7f7974009e10 comp rx=0 tx=0).stop 2022-09-10T04:26:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 -- 172.21.15.81:0/1309957348 shutdown_connections 2022-09-10T04:26:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 --2- 172.21.15.81:0/1309957348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c103880 0x7f797c105c60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 --2- 172.21.15.81:0/1309957348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f797c0fe640 0x7f797c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 --2- 172.21.15.81:0/1309957348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f797c100f60 0x7f797c103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 -- 172.21.15.81:0/1309957348 >> 172.21.15.81:0/1309957348 conn(0x7f797c0f8160 msgr2=0x7f797c0fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:30.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 -- 172.21.15.81:0/1309957348 shutdown_connections 2022-09-10T04:26:30.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.594+0000 7f7980f9d700 1 -- 172.21.15.81:0/1309957348 wait complete. 2022-09-10T04:26:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.595+0000 7f7980f9d700 1 Processor -- start 2022-09-10T04:26:30.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.595+0000 7f7980f9d700 1 -- start start 2022-09-10T04:26:30.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.595+0000 7f7980f9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 0x7f797c0fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:30.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f7980f9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f797c100f60 0x7f797c0fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:30.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f7980f9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f797c103880 0x7f797c0fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:30.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f7980f9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0fd330 con 0x7f797c103880 2022-09-10T04:26:30.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f7980f9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c0fd4a0 con 0x7f797c100f60 2022-09-10T04:26:30.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f7980f9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f797c19d7d0 con 0x7f797c0fe640 2022-09-10T04:26:30.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 0x7f797c0fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:30.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 0x7f797c0fdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43238/0 (socket says 172.21.15.81:43238) 2022-09-10T04:26:30.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797affd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f797c100f60 0x7f797c0fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:30.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 -- 172.21.15.81:0/4215489472 learned_addr learned my addr 172.21.15.81:0/4215489472 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:30.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797bfff700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f797c103880 0x7f797c0fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:30.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 -- 172.21.15.81:0/4215489472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f797c100f60 msgr2=0x7f797c0fc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:30.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f797c100f60 0x7f797c0fc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 -- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f797c103880 msgr2=0x7f797c0fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:30.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f797c103880 0x7f797c0fcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:30.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.596+0000 7f797b7fe700 1 -- 172.21.15.81:0/4215489472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7974009730 con 0x7f797c0fe640 2022-09-10T04:26:30.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.597+0000 7f797b7fe700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 0x7f797c0fdfb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f796c00e740 tx=0x7f796c00ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.597+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c005220 con 0x7f797c0fe640 2022-09-10T04:26:30.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.597+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f796c005380 con 0x7f797c0fe640 2022-09-10T04:26:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.597+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f797c19dab0 con 0x7f797c0fe640 2022-09-10T04:26:30.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.597+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796c004c10 con 0x7f797c0fe640 2022-09-10T04:26:30.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.597+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797c19df90 con 0x7f797c0fe640 2022-09-10T04:26:30.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.598+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797c1092c0 con 0x7f797c0fe640 2022-09-10T04:26:30.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.600+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f796c01e020 con 0x7f797c0fe640 2022-09-10T04:26:30.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.601+0000 7f7978ff9700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7964099010 0x7f796409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:30.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.601+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f796c014070 con 0x7f797c0fe640 2022-09-10T04:26:30.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.602+0000 7f797affd700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7964099010 0x7f796409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:30.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.602+0000 7f797affd700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7964099010 0x7f796409b4c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7970005fd0 tx=0x7f7970005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:30.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.605+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f796c08ed50 con 0x7f797c0fe640 2022-09-10T04:26:30.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1979857017' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-10T04:26:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2022-09-10T04:26:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:26:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:26:30.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:30 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:26:30.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:30.805+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f797c05f120 con 0x7f797c0fe640 2022-09-10T04:26:31.542 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.538+0000 7f7978ff9700 1 -- 172.21.15.81:0/4215489472 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) v1 ==== 171+0+0 (secure 0 0 0) 0x7f796c092370 con 0x7f797c0fe640 2022-09-10T04:26:31.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.548+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7964099010 msgr2=0x7f796409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.548+0000 7f7980f9d700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7964099010 0x7f796409b4c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7970005fd0 tx=0x7f7970005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.548+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 msgr2=0x7f797c0fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:31.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.548+0000 7f7980f9d700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 0x7f797c0fdfb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f796c00e740 tx=0x7f796c00ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:31.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.550+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 shutdown_connections 2022-09-10T04:26:31.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.550+0000 7f7980f9d700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f797c103880 0x7f797c0fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.550+0000 7f7980f9d700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7964099010 0x7f796409b4c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.550+0000 7f7980f9d700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f797c0fe640 0x7f797c0fdfb0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f796c00e740 tx=0x7f796c00ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:31.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.550+0000 7f7980f9d700 1 --2- 172.21.15.81:0/4215489472 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f797c100f60 0x7f797c0fc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.550+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 >> 172.21.15.81:0/4215489472 conn(0x7f797c0f8160 msgr2=0x7f797c0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:31.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.552+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 shutdown_connections 2022-09-10T04:26:31.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.555+0000 7f7980f9d700 1 -- 172.21.15.81:0/4215489472 wait complete. 2022-09-10T04:26:31.563 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2022-09-10T04:26:31.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-09-10T04:26:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:31 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-09-10T04:26:31.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:31 smithi081 ceph-mon[90133]: osdmap e83: 12 total, 12 up, 12 in 2022-09-10T04:26:31.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.829+0000 7f4a919ff700 1 -- 172.21.15.81:0/790933676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0ff910 msgr2=0x7f4a8c1061d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:31.831 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.829+0000 7f4a919ff700 1 --2- 172.21.15.81:0/790933676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0ff910 0x7f4a8c1061d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4a84009bb0 tx=0x7f4a84009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.832 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 -- 172.21.15.81:0/790933676 shutdown_connections 2022-09-10T04:26:31.832 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 --2- 172.21.15.81:0/790933676 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0ff910 0x7f4a8c1061d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.832 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 --2- 172.21.15.81:0/790933676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a8c0fdd00 0x7f4a8c0fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 --2- 172.21.15.81:0/790933676 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 -- 172.21.15.81:0/790933676 >> 172.21.15.81:0/790933676 conn(0x7f4a8c0f93f0 msgr2=0x7f4a8c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:31.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 -- 172.21.15.81:0/790933676 shutdown_connections 2022-09-10T04:26:31.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.830+0000 7f4a919ff700 1 -- 172.21.15.81:0/790933676 wait complete. 2022-09-10T04:26:31.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 Processor -- start 2022-09-10T04:26:31.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 -- start start 2022-09-10T04:26:31.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a8c0fdd00 0x7f4a8c072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:31.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:31.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8c0ff910 0x7f4a8c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:31.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c071e10 con 0x7f4a8c0fdd00 2022-09-10T04:26:31.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c071f80 con 0x7f4a8c0fef80 2022-09-10T04:26:31.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.832+0000 7f4a919ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8c19da90 con 0x7f4a8c0ff910 2022-09-10T04:26:31.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:31.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56594/0 (socket says 172.21.15.81:56594) 2022-09-10T04:26:31.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 -- 172.21.15.81:0/1192390483 learned_addr learned my addr 172.21.15.81:0/1192390483 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8affd700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a8c0fdd00 0x7f4a8c072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8b7fe700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8c0ff910 0x7f4a8c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 -- 172.21.15.81:0/1192390483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8c0ff910 msgr2=0x7f4a8c071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:31.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8c0ff910 0x7f4a8c071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 -- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a8c0fdd00 msgr2=0x7f4a8c072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a8c0fdd00 0x7f4a8c072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 -- 172.21.15.81:0/1192390483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a840097e0 con 0x7f4a8c0fef80 2022-09-10T04:26:31.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.833+0000 7f4a8a7fc700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0710d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000e740 tx=0x7f4a8000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:31.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.834+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80005220 con 0x7f4a8c0fef80 2022-09-10T04:26:31.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.834+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c19dd70 con 0x7f4a8c0fef80 2022-09-10T04:26:31.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.834+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4a80005380 con 0x7f4a8c0fef80 2022-09-10T04:26:31.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.834+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c19e240 con 0x7f4a8c0fef80 2022-09-10T04:26:31.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.834+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a80004c10 con 0x7f4a8c0fef80 2022-09-10T04:26:31.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.835+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6c0051a0 con 0x7f4a8c0fef80 2022-09-10T04:26:31.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.837+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f4a8001e020 con 0x7f4a8c0fef80 2022-09-10T04:26:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.838+0000 7f4a909fd700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4a74099090 0x7f4a7409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.838+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7875+0+0 (secure 0 0 0) 0x7f4a80014070 con 0x7f4a8c0fef80 2022-09-10T04:26:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.839+0000 7f4a8affd700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4a74099090 0x7f4a7409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:31.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.839+0000 7f4a8affd700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4a74099090 0x7f4a7409b540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0fede0 tx=0x7f4a7c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:31.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:31.843+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f4a8008ec30 con 0x7f4a8c0fef80 2022-09-10T04:26:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:31 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-09-10T04:26:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:31 smithi110 ceph-mon[97633]: osdmap e83: 12 total, 12 up, 12 in 2022-09-10T04:26:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:31 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2022-09-10T04:26:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:31 smithi104 ceph-mon[97044]: osdmap e83: 12 total, 12 up, 12 in 2022-09-10T04:26:32.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.094+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4a6c005df0 con 0x7f4a8c0fef80 2022-09-10T04:26:32.104 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.103+0000 7f4a909fd700 1 -- 172.21.15.81:0/1192390483 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4a80092250 con 0x7f4a8c0fef80 2022-09-10T04:26:32.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.114+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4a74099090 msgr2=0x7f4a7409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:32.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.114+0000 7f4a919ff700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4a74099090 0x7f4a7409b540 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0fede0 tx=0x7f4a7c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.114+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 msgr2=0x7f4a8c0710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:32.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.114+0000 7f4a919ff700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0710d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000e740 tx=0x7f4a8000ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:32.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.121+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 shutdown_connections 2022-09-10T04:26:32.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.121+0000 7f4a919ff700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4a74099090 0x7f4a7409b540 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.121+0000 7f4a919ff700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8c0ff910 0x7f4a8c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.121+0000 7f4a919ff700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a8c0fdd00 0x7f4a8c072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.121+0000 7f4a919ff700 1 --2- 172.21.15.81:0/1192390483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4a8c0fef80 0x7f4a8c0710d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.121+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 >> 172.21.15.81:0/1192390483 conn(0x7f4a8c0f93f0 msgr2=0x7f4a8c102590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:32.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.127+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 shutdown_connections 2022-09-10T04:26:32.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.127+0000 7f4a919ff700 1 -- 172.21.15.81:0/1192390483 wait complete. 2022-09-10T04:26:32.135 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 2 and data pool 3 2022-09-10T04:26:32.183 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2022-09-10T04:26:32.184 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2022-09-10T04:26:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 -- 172.21.15.81:0/2748675489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e500fffa0 msgr2=0x7f1e501003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:32.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 --2- 172.21.15.81:0/2748675489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e500fffa0 0x7f1e501003b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009750 tx=0x7f1e4c009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:32.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 -- 172.21.15.81:0/2748675489 shutdown_connections 2022-09-10T04:26:32.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 --2- 172.21.15.81:0/2748675489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e50101bb0 0x7f1e50108470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 --2- 172.21.15.81:0/2748675489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e500fffa0 0x7f1e501003b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 --2- 172.21.15.81:0/2748675489 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 -- 172.21.15.81:0/2748675489 >> 172.21.15.81:0/2748675489 conn(0x7f1e500fb6b0 msgr2=0x7f1e500fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:32.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 -- 172.21.15.81:0/2748675489 shutdown_connections 2022-09-10T04:26:32.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.433+0000 7f1e583ff700 1 -- 172.21.15.81:0/2748675489 wait complete. 2022-09-10T04:26:32.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.434+0000 7f1e583ff700 1 Processor -- start 2022-09-10T04:26:32.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 -- start start 2022-09-10T04:26:32.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e500fffa0 0x7f1e50195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:32.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:32.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50101bb0 0x7f1e5019d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:32.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5019d6b0 con 0x7f1e50101bb0 2022-09-10T04:26:32.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5019d820 con 0x7f1e50101220 2022-09-10T04:26:32.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e583ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5019d9e0 con 0x7f1e500fffa0 2022-09-10T04:26:32.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e5599a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:32.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e5699c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50101bb0 0x7f1e5019d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:32.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e5599a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50196070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56618/0 (socket says 172.21.15.81:56618) 2022-09-10T04:26:32.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e5619b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e500fffa0 0x7f1e50195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:32.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.435+0000 7f1e5599a700 1 -- 172.21.15.81:0/188340048 learned_addr learned my addr 172.21.15.81:0/188340048 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:32.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5599a700 1 -- 172.21.15.81:0/188340048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e500fffa0 msgr2=0x7f1e50195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:32.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5599a700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e500fffa0 0x7f1e50195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5599a700 1 -- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50101bb0 msgr2=0x7f1e5019d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:32.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5599a700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50101bb0 0x7f1e5019d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:32.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5599a700 1 -- 172.21.15.81:0/188340048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4c0093b0 con 0x7f1e50101220 2022-09-10T04:26:32.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5619b700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e500fffa0 0x7f1e50195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:32.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5699c700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50101bb0 0x7f1e5019d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:32.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e5599a700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50196070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000e740 tx=0x7f1e4000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:32.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40005220 con 0x7f1e50101220 2022-09-10T04:26:32.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1e40005380 con 0x7f1e50101220 2022-09-10T04:26:32.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5019dcc0 con 0x7f1e50101220 2022-09-10T04:26:32.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40004c10 con 0x7f1e50101220 2022-09-10T04:26:32.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.436+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5019e1d0 con 0x7f1e50101220 2022-09-10T04:26:32.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.437+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5010b750 con 0x7f1e50101220 2022-09-10T04:26:32.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.439+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1e4001e020 con 0x7f1e50101220 2022-09-10T04:26:32.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.440+0000 7f1e477fe700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e3c099010 0x7f1e3c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:32.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.440+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7975+0+0 (secure 0 0 0) 0x7f1e40014070 con 0x7f1e50101220 2022-09-10T04:26:32.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.441+0000 7f1e5619b700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e3c099010 0x7f1e3c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:32.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.441+0000 7f1e5619b700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e3c099010 0x7f1e3c09b4c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00f460 tx=0x7f1e4c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:32.461 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.445+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1e4008ed80 con 0x7f1e50101220 2022-09-10T04:26:32.673 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:32.671+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f1e50106e90 con 0x7f1e50101220 2022-09-10T04:26:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4215489472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-10T04:26:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-10T04:26:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: pgmap v294: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2022-09-10T04:26:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: osdmap e84: 12 total, 12 up, 12 in 2022-09-10T04:26:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1192390483' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-10T04:26:32.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-10T04:26:32.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: daemon mds.a assigned to filesystem cephfs as rank 0 2022-09-10T04:26:32.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: osdmap e85: 12 total, 12 up, 12 in 2022-09-10T04:26:32.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-09-10T04:26:32.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2022-09-10T04:26:32.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:32 smithi081 ceph-mon[90133]: daemon mds.a is now active in filesystem cephfs as rank 0 2022-09-10T04:26:32.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4215489472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-10T04:26:32.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-10T04:26:32.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: pgmap v294: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:32.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2022-09-10T04:26:32.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: osdmap e84: 12 total, 12 up, 12 in 2022-09-10T04:26:32.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1192390483' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-10T04:26:32.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-10T04:26:32.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: daemon mds.a assigned to filesystem cephfs as rank 0 2022-09-10T04:26:32.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: osdmap e85: 12 total, 12 up, 12 in 2022-09-10T04:26:32.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-09-10T04:26:32.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2022-09-10T04:26:32.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:32 smithi110 ceph-mon[97633]: daemon mds.a is now active in filesystem cephfs as rank 0 2022-09-10T04:26:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4215489472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-10T04:26:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2022-09-10T04:26:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: pgmap v294: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:26:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2022-09-10T04:26:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: osdmap e84: 12 total, 12 up, 12 in 2022-09-10T04:26:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1192390483' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-10T04:26:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2022-09-10T04:26:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: daemon mds.a assigned to filesystem cephfs as rank 0 2022-09-10T04:26:33.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: osdmap e85: 12 total, 12 up, 12 in 2022-09-10T04:26:33.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2022-09-10T04:26:33.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2022-09-10T04:26:33.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:32 smithi104 ceph-mon[97044]: daemon mds.a is now active in filesystem cephfs as rank 0 2022-09-10T04:26:33.108 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.107+0000 7f1e477fe700 1 -- 172.21.15.81:0/188340048 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) v1 ==== 157+0+0 (secure 0 0 0) 0x7f1e400923a0 con 0x7f1e50101220 2022-09-10T04:26:33.111 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e3c099010 msgr2=0x7f1e3c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:33.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e3c099010 0x7f1e3c09b4c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00f460 tx=0x7f1e4c005f90 comp rx=0 tx=0).stop 2022-09-10T04:26:33.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 msgr2=0x7f1e50196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:33.112 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50196070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000e740 tx=0x7f1e4000ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:33.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 shutdown_connections 2022-09-10T04:26:33.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e50101bb0 0x7f1e5019d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e3c099010 0x7f1e3c09b4c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e500fffa0 0x7f1e50195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 --2- 172.21.15.81:0/188340048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e50101220 0x7f1e50196070 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 >> 172.21.15.81:0/188340048 conn(0x7f1e500fb6b0 msgr2=0x7f1e500fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:33.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 shutdown_connections 2022-09-10T04:26:33.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.110+0000 7f1e583ff700 1 -- 172.21.15.81:0/188340048 wait complete. 2022-09-10T04:26:33.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2022-09-10T04:26:33.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 -- 172.21.15.81:0/3255272412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9580ffdf0 msgr2=0x7fb958100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 --2- 172.21.15.81:0/3255272412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9580ffdf0 0x7fb958100200 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb948009750 tx=0x7fb948009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:33.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 -- 172.21.15.81:0/3255272412 shutdown_connections 2022-09-10T04:26:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 --2- 172.21.15.81:0/3255272412 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb958101980 0x7fb958108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 --2- 172.21.15.81:0/3255272412 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9580ffdf0 0x7fb958100200 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 --2- 172.21.15.81:0/3255272412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb958100fd0 0x7fb958101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 -- 172.21.15.81:0/3255272412 >> 172.21.15.81:0/3255272412 conn(0x7fb9580fb4d0 msgr2=0x7fb9580fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:33.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 -- 172.21.15.81:0/3255272412 shutdown_connections 2022-09-10T04:26:33.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.387+0000 7fb95d59d700 1 -- 172.21.15.81:0/3255272412 wait complete. 2022-09-10T04:26:33.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.388+0000 7fb95d59d700 1 Processor -- start 2022-09-10T04:26:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.388+0000 7fb95d59d700 1 -- start start 2022-09-10T04:26:33.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95d59d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580ffdf0 0x7fb958195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:33.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95d59d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb958100fd0 0x7fb958195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:33.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95d59d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 0x7fb95819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:33.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95d59d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95819d410 con 0x7fb958100fd0 2022-09-10T04:26:33.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95d59d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95819d580 con 0x7fb958101980 2022-09-10T04:26:33.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95d59d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95819d740 con 0x7fb9580ffdf0 2022-09-10T04:26:33.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95cd9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 0x7fb95819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:33.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95cd9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 0x7fb95819ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56632/0 (socket says 172.21.15.81:56632) 2022-09-10T04:26:33.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95cd9c700 1 -- 172.21.15.81:0/837957578 learned_addr learned my addr 172.21.15.81:0/837957578 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb94f7fe700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb958100fd0 0x7fb958195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb957fff700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580ffdf0 0x7fb958195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:33.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.389+0000 7fb95cd9c700 1 -- 172.21.15.81:0/837957578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580ffdf0 msgr2=0x7fb958195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95cd9c700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580ffdf0 0x7fb958195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95cd9c700 1 -- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb958100fd0 msgr2=0x7fb958195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95cd9c700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb958100fd0 0x7fb958195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95cd9c700 1 -- 172.21.15.81:0/837957578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9480093b0 con 0x7fb958101980 2022-09-10T04:26:33.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95cd9c700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 0x7fb95819ce40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb95000ebf0 tx=0x7fb95000ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9500099c0 con 0x7fb958101980 2022-09-10T04:26:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb95000cc80 con 0x7fb958101980 2022-09-10T04:26:33.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95819da20 con 0x7fb958101980 2022-09-10T04:26:33.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950005920 con 0x7fb958101980 2022-09-10T04:26:33.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.390+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95819dee0 con 0x7fb958101980 2022-09-10T04:26:33.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.391+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95810b4b0 con 0x7fb958101980 2022-09-10T04:26:33.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.393+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb95001e020 con 0x7fb958101980 2022-09-10T04:26:33.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.394+0000 7fb955ffb700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb944099010 0x7fb94409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:33.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.394+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8201+0+0 (secure 0 0 0) 0x7fb950014070 con 0x7fb958101980 2022-09-10T04:26:33.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.394+0000 7fb957fff700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb944099010 0x7fb94409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:33.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.395+0000 7fb957fff700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb944099010 0x7fb94409b4c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb94800f500 tx=0x7fb948005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:33.411 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.398+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb95008f000 con 0x7fb958101980 2022-09-10T04:26:33.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:33.614+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb958107240 con 0x7fb958101980 2022-09-10T04:26:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/188340048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-10T04:26:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-10T04:26:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2022-09-10T04:26:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: osdmap e86: 12 total, 12 up, 12 in 2022-09-10T04:26:33.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] up:active 2022-09-10T04:26:33.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:33.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:33 smithi110 ceph-mon[97633]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/188340048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-10T04:26:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-10T04:26:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2022-09-10T04:26:34.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: osdmap e86: 12 total, 12 up, 12 in 2022-09-10T04:26:34.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] up:active 2022-09-10T04:26:34.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:34.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:33 smithi104 ceph-mon[97044]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/188340048' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-10T04:26:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2022-09-10T04:26:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2022-09-10T04:26:34.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: osdmap e86: 12 total, 12 up, 12 in 2022-09-10T04:26:34.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] up:active 2022-09-10T04:26:34.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:34.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:33 smithi081 ceph-mon[90133]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:34 smithi081 ceph-mon[90133]: pgmap v298: 97 pgs: 15 creating+peering, 20 active+clean, 62 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:34 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/837957578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-10T04:26:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:34 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-10T04:26:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:34 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:34 smithi110 ceph-mon[97633]: pgmap v298: 97 pgs: 15 creating+peering, 20 active+clean, 62 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:34 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/837957578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-10T04:26:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:34 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-10T04:26:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:34 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:34 smithi104 ceph-mon[97044]: pgmap v298: 97 pgs: 15 creating+peering, 20 active+clean, 62 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:34 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/837957578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-10T04:26:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:34 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2022-09-10T04:26:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:35.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:34 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:26:35.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.579+0000 7fb955ffb700 1 -- 172.21.15.81:0/837957578 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) v1 ==== 228+0+0 (secure 0 0 0) 0x7fb950092620 con 0x7fb958101980 2022-09-10T04:26:35.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.584+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb944099010 msgr2=0x7fb94409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:35.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.584+0000 7fb95d59d700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb944099010 0x7fb94409b4c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb94800f500 tx=0x7fb948005f90 comp rx=0 tx=0).stop 2022-09-10T04:26:35.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.584+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 msgr2=0x7fb95819ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:35.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.584+0000 7fb95d59d700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 0x7fb95819ce40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb95000ebf0 tx=0x7fb95000ef00 comp rx=0 tx=0).stop 2022-09-10T04:26:35.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 shutdown_connections 2022-09-10T04:26:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb958101980 0x7fb95819ce40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb944099010 0x7fb94409b4c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580ffdf0 0x7fb958195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 --2- 172.21.15.81:0/837957578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb958100fd0 0x7fb958195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 >> 172.21.15.81:0/837957578 conn(0x7fb9580fb4d0 msgr2=0x7fb958104500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:35.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 shutdown_connections 2022-09-10T04:26:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.585+0000 7fb95d59d700 1 -- 172.21.15.81:0/837957578 wait complete. 2022-09-10T04:26:35.591 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data_ec' created 2022-09-10T04:26:35.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2022-09-10T04:26:35.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 -- 172.21.15.81:0/3347849383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0fdd00 msgr2=0x7f900c0fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:35.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 --2- 172.21.15.81:0/3347849383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0fdd00 0x7f900c0fe110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009810 tx=0x7f8ffc009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:35.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 -- 172.21.15.81:0/3347849383 shutdown_connections 2022-09-10T04:26:35.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 --2- 172.21.15.81:0/3347849383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f900c0ff910 0x7f900c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 --2- 172.21.15.81:0/3347849383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0fdd00 0x7f900c0fe110 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 --2- 172.21.15.81:0/3347849383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fef80 0x7f900c0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 -- 172.21.15.81:0/3347849383 >> 172.21.15.81:0/3347849383 conn(0x7f900c0f93f0 msgr2=0x7f900c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 -- 172.21.15.81:0/3347849383 shutdown_connections 2022-09-10T04:26:35.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.847+0000 7f90111ff700 1 -- 172.21.15.81:0/3347849383 wait complete. 2022-09-10T04:26:35.863 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.849+0000 7f90111ff700 1 Processor -- start 2022-09-10T04:26:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.849+0000 7f90111ff700 1 -- start start 2022-09-10T04:26:35.864 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f90111ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 0x7f900c193950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:35.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f90111ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f900c0fef80 0x7f900c193e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:35.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f90111ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0ff910 0x7f900c19af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:35.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f90111ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c19b4d0 con 0x7f900c0ff910 2022-09-10T04:26:35.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f90111ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c19b640 con 0x7f900c0fdd00 2022-09-10T04:26:35.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f90111ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c19b800 con 0x7f900c0fef80 2022-09-10T04:26:35.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 0x7f900c193950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:35.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900b59e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0ff910 0x7f900c19af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:35.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900ad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 0x7f900c193950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56648/0 (socket says 172.21.15.81:56648) 2022-09-10T04:26:35.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900ad9d700 1 -- 172.21.15.81:0/2557540396 learned_addr learned my addr 172.21.15.81:0/2557540396 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:35.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900a59c700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f900c0fef80 0x7f900c193e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900ad9d700 1 -- 172.21.15.81:0/2557540396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f900c0fef80 msgr2=0x7f900c193e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900ad9d700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f900c0fef80 0x7f900c193e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.850+0000 7f900ad9d700 1 -- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0ff910 msgr2=0x7f900c19af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:35.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f900ad9d700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0ff910 0x7f900c19af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f900ad9d700 1 -- 172.21.15.81:0/2557540396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc009470 con 0x7f900c0fdd00 2022-09-10T04:26:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f900b59e700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0ff910 0x7f900c19af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f900ad9d700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 0x7f900c193950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006010 tx=0x7f8ffc00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:35.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc01d070 con 0x7f900c0fdd00 2022-09-10T04:26:35.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c070f10 con 0x7f900c0fdd00 2022-09-10T04:26:35.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8ffc00f8b0 con 0x7f900c0fdd00 2022-09-10T04:26:35.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc0047f0 con 0x7f900c0fdd00 2022-09-10T04:26:35.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.851+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c071410 con 0x7f900c0fdd00 2022-09-10T04:26:35.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.852+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c1094b0 con 0x7f900c0fdd00 2022-09-10T04:26:35.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.854+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8ffc028030 con 0x7f900c0fdd00 2022-09-10T04:26:35.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.855+0000 7f8feffff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ff4099010 0x7f8ff409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:35.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.856+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8694+0+0 (secure 0 0 0) 0x7f8ffc0c3660 con 0x7f900c0fdd00 2022-09-10T04:26:35.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.856+0000 7f900a59c700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ff4099010 0x7f8ff409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:35.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.856+0000 7f900a59c700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ff4099010 0x7f8ff409b4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9000005fd0 tx=0x7f9000012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:35.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:35.858+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f8ffc08fb20 con 0x7f900c0fdd00 2022-09-10T04:26:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:35 smithi110 ceph-mon[97633]: osdmap e87: 12 total, 12 up, 12 in 2022-09-10T04:26:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:35 smithi104 ceph-mon[97044]: osdmap e87: 12 total, 12 up, 12 in 2022-09-10T04:26:36.048 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.047+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f900c071d00 con 0x7f900c0fdd00 2022-09-10T04:26:36.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:35 smithi081 ceph-mon[90133]: osdmap e87: 12 total, 12 up, 12 in 2022-09-10T04:26:36.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.581+0000 7f8feffff700 1 -- 172.21.15.81:0/2557540396 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8ffc0260c0 con 0x7f900c0fdd00 2022-09-10T04:26:36.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.591+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ff4099010 msgr2=0x7f8ff409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:36.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.591+0000 7f90111ff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ff4099010 0x7f8ff409b4c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9000005fd0 tx=0x7f9000012040 comp rx=0 tx=0).stop 2022-09-10T04:26:36.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.591+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 msgr2=0x7f900c193950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:36.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.591+0000 7f90111ff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 0x7f900c193950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006010 tx=0x7f8ffc00f690 comp rx=0 tx=0).stop 2022-09-10T04:26:36.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.592+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 shutdown_connections 2022-09-10T04:26:36.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.592+0000 7f90111ff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f900c0ff910 0x7f900c19af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.592+0000 7f90111ff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f900c0fdd00 0x7f900c193950 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc006010 tx=0x7f8ffc00f690 comp rx=0 tx=0).stop 2022-09-10T04:26:36.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.592+0000 7f90111ff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ff4099010 0x7f8ff409b4c0 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9000005fd0 tx=0x7f9000012040 comp rx=0 tx=0).stop 2022-09-10T04:26:36.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.592+0000 7f90111ff700 1 --2- 172.21.15.81:0/2557540396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f900c0fef80 0x7f900c193e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.592+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 >> 172.21.15.81:0/2557540396 conn(0x7f900c0f93f0 msgr2=0x7f900c0fb710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:36.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.595+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 shutdown_connections 2022-09-10T04:26:36.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.595+0000 7f90111ff700 1 -- 172.21.15.81:0/2557540396 wait complete. 2022-09-10T04:26:36.607 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 4 allow_ec_overwrites to true 2022-09-10T04:26:36.634 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2022-09-10T04:26:36.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 -- 172.21.15.81:0/3331793964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 msgr2=0x7f73f8100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:36.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 --2- 172.21.15.81:0/3331793964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f8100a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009810 tx=0x7f73f4009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:36.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 -- 172.21.15.81:0/3331793964 shutdown_connections 2022-09-10T04:26:36.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 --2- 172.21.15.81:0/3331793964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 0x7f73f8105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 --2- 172.21.15.81:0/3331793964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f8100a20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 --2- 172.21.15.81:0/3331793964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73f8100f60 0x7f73f8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.863+0000 7f73fe59d700 1 -- 172.21.15.81:0/3331793964 >> 172.21.15.81:0/3331793964 conn(0x7f73f80f8160 msgr2=0x7f73f80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:36.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.864+0000 7f73fe59d700 1 -- 172.21.15.81:0/3331793964 shutdown_connections 2022-09-10T04:26:36.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.864+0000 7f73fe59d700 1 -- 172.21.15.81:0/3331793964 wait complete. 2022-09-10T04:26:36.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 Processor -- start 2022-09-10T04:26:36.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 -- start start 2022-09-10T04:26:36.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f80fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:36.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73f8100f60 0x7f73f80fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:36.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 0x7f73f80fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:36.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80fd330 con 0x7f73f8103880 2022-09-10T04:26:36.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80fd4a0 con 0x7f73f8100f60 2022-09-10T04:26:36.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fe59d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f819d800 con 0x7f73f80fe640 2022-09-10T04:26:36.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.865+0000 7f73fdd9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 0x7f73f80fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:36.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f80fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:36.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fdd9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 0x7f73f80fcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45762/0 (socket says 172.21.15.81:45762) 2022-09-10T04:26:36.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fdd9c700 1 -- 172.21.15.81:0/2982577704 learned_addr learned my addr 172.21.15.81:0/2982577704 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:36.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f80fdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43318/0 (socket says 172.21.15.81:43318) 2022-09-10T04:26:36.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fcd9a700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73f8100f60 0x7f73f80fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:36.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 -- 172.21.15.81:0/2982577704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73f8100f60 msgr2=0x7f73f80fc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:36.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73f8100f60 0x7f73f80fc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 -- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 msgr2=0x7f73f80fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:36.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 0x7f73f80fcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:36.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 -- 172.21.15.81:0/2982577704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f4009470 con 0x7f73f80fe640 2022-09-10T04:26:36.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.866+0000 7f73fd59b700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f80fdfb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73f4000c00 tx=0x7f73f400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:36.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.867+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f401d070 con 0x7f73f80fe640 2022-09-10T04:26:36.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.867+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f73f400f880 con 0x7f73f80fe640 2022-09-10T04:26:36.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.867+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4004760 con 0x7f73f80fe640 2022-09-10T04:26:36.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.867+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f819da80 con 0x7f73f80fe640 2022-09-10T04:26:36.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.867+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f819dee0 con 0x7f73f80fe640 2022-09-10T04:26:36.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.868+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f81092c0 con 0x7f73f80fe640 2022-09-10T04:26:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.870+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f73f4028030 con 0x7f73f80fe640 2022-09-10T04:26:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.871+0000 7f73ee7fc700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f73e4099110 0x7f73e409b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:36.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.871+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8694+0+0 (secure 0 0 0) 0x7f73f40c3530 con 0x7f73f80fe640 2022-09-10T04:26:36.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.871+0000 7f73fcd9a700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f73e4099110 0x7f73e409b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:36.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.872+0000 7f73fcd9a700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f73e4099110 0x7f73e409b5c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005fd0 tx=0x7f73e8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:36.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:36.873+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f73f408f9f0 con 0x7f73f80fe640 2022-09-10T04:26:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:36 smithi110 ceph-mon[97633]: pgmap v300: 97 pgs: 15 creating+peering, 64 active+clean, 18 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2022-09-10T04:26:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:36 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-09-10T04:26:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:36 smithi110 ceph-mon[97633]: osdmap e88: 12 total, 12 up, 12 in 2022-09-10T04:26:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:36 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2557540396' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-10T04:26:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:36 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-10T04:26:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:36 smithi104 ceph-mon[97044]: pgmap v300: 97 pgs: 15 creating+peering, 64 active+clean, 18 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2022-09-10T04:26:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:36 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-09-10T04:26:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:36 smithi104 ceph-mon[97044]: osdmap e88: 12 total, 12 up, 12 in 2022-09-10T04:26:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:36 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2557540396' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-10T04:26:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:36 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-10T04:26:37.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:36 smithi081 ceph-mon[90133]: pgmap v300: 97 pgs: 15 creating+peering, 64 active+clean, 18 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 5 op/s 2022-09-10T04:26:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:36 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2022-09-10T04:26:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:36 smithi081 ceph-mon[90133]: osdmap e88: 12 total, 12 up, 12 in 2022-09-10T04:26:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:36 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2557540396' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-10T04:26:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:36 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2022-09-10T04:26:37.113 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.112+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f73f80fab60 con 0x7f73f80fe640 2022-09-10T04:26:37.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.118+0000 7f73ee7fc700 1 -- 172.21.15.81:0/2982577704 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f73f4026020 con 0x7f73f80fe640 2022-09-10T04:26:37.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f73e4099110 msgr2=0x7f73e409b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f73e4099110 0x7f73e409b5c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f73e8005fd0 tx=0x7f73e8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 msgr2=0x7f73f80fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f80fdfb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f73f4000c00 tx=0x7f73f400f690 comp rx=0 tx=0).stop 2022-09-10T04:26:37.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 shutdown_connections 2022-09-10T04:26:37.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73f8103880 0x7f73f80fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f73e4099110 0x7f73e409b5c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73f80fe640 0x7f73f80fdfb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 --2- 172.21.15.81:0/2982577704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73f8100f60 0x7f73f80fc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 >> 172.21.15.81:0/2982577704 conn(0x7f73f80f8160 msgr2=0x7f73f80fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:37.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 shutdown_connections 2022-09-10T04:26:37.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.121+0000 7f73fe59d700 1 -- 172.21.15.81:0/2982577704 wait complete. 2022-09-10T04:26:37.125 INFO:teuthology.orchestra.run.smithi081.stderr:added data pool 4 to fsmap 2022-09-10T04:26:37.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:37.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 -- 172.21.15.81:0/2515801386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8100f60 msgr2=0x7f71f8103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 --2- 172.21.15.81:0/2515801386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8100f60 0x7f71f8103340 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009810 tx=0x7f71e8009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:37.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 -- 172.21.15.81:0/2515801386 shutdown_connections 2022-09-10T04:26:37.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 --2- 172.21.15.81:0/2515801386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8103880 0x7f71f8105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 --2- 172.21.15.81:0/2515801386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 0x7f71f8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 --2- 172.21.15.81:0/2515801386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8100f60 0x7f71f8103340 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 -- 172.21.15.81:0/2515801386 >> 172.21.15.81:0/2515801386 conn(0x7f71f80f8160 msgr2=0x7f71f80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.374+0000 7f71fef9d700 1 -- 172.21.15.81:0/2515801386 shutdown_connections 2022-09-10T04:26:37.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.375+0000 7f71fef9d700 1 -- 172.21.15.81:0/2515801386 wait complete. 2022-09-10T04:26:37.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 Processor -- start 2022-09-10T04:26:37.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 -- start start 2022-09-10T04:26:37.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 0x7f71f81958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 0x7f71f8195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8103880 0x7f71f819cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f819d470 con 0x7f71f80fe640 2022-09-10T04:26:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f819d5e0 con 0x7f71f8100f60 2022-09-10T04:26:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.376+0000 7f71fef9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f819d7a0 con 0x7f71f8103880 2022-09-10T04:26:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 0x7f71f8195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fdf9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 0x7f71f81958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 0x7f71f8195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56676/0 (socket says 172.21.15.81:56676) 2022-09-10T04:26:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 -- 172.21.15.81:0/1198304253 learned_addr learned my addr 172.21.15.81:0/1198304253 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fe79c700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8103880 0x7f71f819cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.384 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 -- 172.21.15.81:0/1198304253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8103880 msgr2=0x7f71f819cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8103880 0x7f71f819cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 -- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 msgr2=0x7f71f81958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 0x7f71f81958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 -- 172.21.15.81:0/1198304253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8009470 con 0x7f71f8100f60 2022-09-10T04:26:37.385 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fdf9b700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 0x7f71f81958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.377+0000 7f71fd79a700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 0x7f71f8195e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f71e80059c0 tx=0x7f71e800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.378+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e801d070 con 0x7f71f8100f60 2022-09-10T04:26:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.378+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f71e800f880 con 0x7f71f8100f60 2022-09-10T04:26:37.386 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.378+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f819da20 con 0x7f71f8100f60 2022-09-10T04:26:37.387 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.378+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e80047d0 con 0x7f71f8100f60 2022-09-10T04:26:37.387 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.378+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f819ded0 con 0x7f71f8100f60 2022-09-10T04:26:37.387 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.379+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f81092c0 con 0x7f71f8100f60 2022-09-10T04:26:37.388 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.381+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f71e8028030 con 0x7f71f8100f60 2022-09-10T04:26:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.382+0000 7f71eeffd700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f71e4099010 0x7f71e409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.382+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f71e80c3680 con 0x7f71f8100f60 2022-09-10T04:26:37.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.382+0000 7f71fdf9b700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f71e4099010 0x7f71e409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.383+0000 7f71fdf9b700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f71e4099010 0x7f71e409b4c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f71f4005fd0 tx=0x7f71f4005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:37.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.384+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f71e808fb20 con 0x7f71f8100f60 2022-09-10T04:26:37.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.623+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f819e480 con 0x7f71f8100f60 2022-09-10T04:26:37.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.624+0000 7f71eeffd700 1 -- 172.21.15.81:0/1198304253 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10676 (secure 0 0 0) 0x7f71e8026090 con 0x7f71f8100f60 2022-09-10T04:26:37.626 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:37.627 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-09-10T04:26:32.097174+0000","modified":"2022-09-10T04:26:37.114599+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24577":{"gid":24577,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2022-09-10T04:26:37.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f71e4099010 msgr2=0x7f71e409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f71e4099010 0x7f71e409b4c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f71f4005fd0 tx=0x7f71f4005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 msgr2=0x7f71f8195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 0x7f71f8195e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f71e80059c0 tx=0x7f71e800f690 comp rx=0 tx=0).stop 2022-09-10T04:26:37.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 shutdown_connections 2022-09-10T04:26:37.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f71e4099010 0x7f71e409b4c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71f8103880 0x7f71f819cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71f80fe640 0x7f71f81958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 --2- 172.21.15.81:0/1198304253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71f8100f60 0x7f71f8195e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 >> 172.21.15.81:0/1198304253 conn(0x7f71f80f8160 msgr2=0x7f71f80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:37.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 shutdown_connections 2022-09-10T04:26:37.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.627+0000 7f71fef9d700 1 -- 172.21.15.81:0/1198304253 wait complete. 2022-09-10T04:26:37.632 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2022-09-10T04:26:37.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:37.888 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 -- 172.21.15.81:0/799986817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f00fff20 msgr2=0x7f01f0100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.888 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 --2- 172.21.15.81:0/799986817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f00fff20 0x7f01f0100330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009810 tx=0x7f01ec009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:37.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 -- 172.21.15.81:0/799986817 shutdown_connections 2022-09-10T04:26:37.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 --2- 172.21.15.81:0/799986817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f0101b30 0x7f01f01083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 --2- 172.21.15.81:0/799986817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f00fff20 0x7f01f0100330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 --2- 172.21.15.81:0/799986817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f01011a0 0x7f01f01015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 -- 172.21.15.81:0/799986817 >> 172.21.15.81:0/799986817 conn(0x7f01f00fb6b0 msgr2=0x7f01f00fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 -- 172.21.15.81:0/799986817 shutdown_connections 2022-09-10T04:26:37.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.881+0000 7f01f77ff700 1 -- 172.21.15.81:0/799986817 wait complete. 2022-09-10T04:26:37.892 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.882+0000 7f01f77ff700 1 Processor -- start 2022-09-10T04:26:37.892 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 -- start start 2022-09-10T04:26:37.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 0x7f01f0195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f01011a0 0x7f01f0196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f0101b30 0x7f01f019d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f019d690 con 0x7f01f01011a0 2022-09-10T04:26:37.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f019d800 con 0x7f01f00fff20 2022-09-10T04:26:37.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f77ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f019d9c0 con 0x7f01f0101b30 2022-09-10T04:26:37.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f559b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 0x7f01f0195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f559b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 0x7f01f0195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56698/0 (socket says 172.21.15.81:56698) 2022-09-10T04:26:37.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f559b700 1 -- 172.21.15.81:0/4137078579 learned_addr learned my addr 172.21.15.81:0/4137078579 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:37.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f4d9a700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f01011a0 0x7f01f0196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.883+0000 7f01f5d9c700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f0101b30 0x7f01f019d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f559b700 1 -- 172.21.15.81:0/4137078579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f0101b30 msgr2=0x7f01f019d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f559b700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f0101b30 0x7f01f019d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f559b700 1 -- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f01011a0 msgr2=0x7f01f0196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:37.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f559b700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f01011a0 0x7f01f0196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:37.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f559b700 1 -- 172.21.15.81:0/4137078579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec009470 con 0x7f01f00fff20 2022-09-10T04:26:37.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f4d9a700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f01011a0 0x7f01f0196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:37.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f559b700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 0x7f01f0195b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006010 tx=0x7f01ec00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:37.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec01d070 con 0x7f01f00fff20 2022-09-10T04:26:37.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f019dc40 con 0x7f01f00fff20 2022-09-10T04:26:37.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f01ec00f8b0 con 0x7f01f00fff20 2022-09-10T04:26:37.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ec0047f0 con 0x7f01f00fff20 2022-09-10T04:26:37.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.884+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f019e0f0 con 0x7f01f00fff20 2022-09-10T04:26:37.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.885+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f010b6d0 con 0x7f01f00fff20 2022-09-10T04:26:37.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.887+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f01ec028030 con 0x7f01f00fff20 2022-09-10T04:26:37.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.889+0000 7f01e67fc700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f01dc099010 0x7f01dc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:37.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.889+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f01ec0c3280 con 0x7f01f00fff20 2022-09-10T04:26:37.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.889+0000 7f01f4d9a700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f01dc099010 0x7f01dc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:37.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.890+0000 7f01f4d9a700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f01dc099010 0x7f01dc09b4c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f01e0005fd0 tx=0x7f01e0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:37.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:37.891+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f01ec08f720 con 0x7f01f00fff20 2022-09-10T04:26:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2022-09-10T04:26:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: osdmap e89: 12 total, 12 up, 12 in 2022-09-10T04:26:37.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2982577704' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-10T04:26:37.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-10T04:26:37.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: osdmap e90: 12 total, 12 up, 12 in 2022-09-10T04:26:37.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2022-09-10T04:26:37.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:37 smithi110 ceph-mon[97633]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:37.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2022-09-10T04:26:37.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: osdmap e89: 12 total, 12 up, 12 in 2022-09-10T04:26:37.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2982577704' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-10T04:26:37.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-10T04:26:37.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: osdmap e90: 12 total, 12 up, 12 in 2022-09-10T04:26:37.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2022-09-10T04:26:37.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-mon[90133]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2022-09-10T04:26:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: osdmap e89: 12 total, 12 up, 12 in 2022-09-10T04:26:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2982577704' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-10T04:26:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2022-09-10T04:26:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: osdmap e90: 12 total, 12 up, 12 in 2022-09-10T04:26:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2022-09-10T04:26:38.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:37 smithi104 ceph-mon[97044]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:38.088 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.086+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01f019e6a0 con 0x7f01f00fff20 2022-09-10T04:26:38.089 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.088+0000 7f01e67fc700 1 -- 172.21.15.81:0/4137078579 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7f01ec092d40 con 0x7f01f00fff20 2022-09-10T04:26:38.090 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:38.091 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f01dc099010 msgr2=0x7f01dc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f01dc099010 0x7f01dc09b4c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f01e0005fd0 tx=0x7f01e0005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 msgr2=0x7f01f0195b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 0x7f01f0195b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01ec006010 tx=0x7f01ec00f690 comp rx=0 tx=0).stop 2022-09-10T04:26:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 shutdown_connections 2022-09-10T04:26:38.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f01dc099010 0x7f01dc09b4c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f0101b30 0x7f01f019d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f00fff20 0x7f01f0195b10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 --2- 172.21.15.81:0/4137078579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f01011a0 0x7f01f0196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.090+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 >> 172.21.15.81:0/4137078579 conn(0x7f01f00fb6b0 msgr2=0x7f01f00fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.091+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 shutdown_connections 2022-09-10T04:26:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.091+0000 7f01f77ff700 1 -- 172.21.15.81:0/4137078579 wait complete. 2022-09-10T04:26:38.120 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:38.196 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-a[125370]: 2022-09-10T04:26:37.912+0000 7f1cb22fb700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2022-09-10T04:26:38.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 -- 172.21.15.81:0/936350345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 msgr2=0x7f80440ff3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 --2- 172.21.15.81:0/936350345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440ff3c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8038009750 tx=0x7f8038009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:38.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 -- 172.21.15.81:0/936350345 shutdown_connections 2022-09-10T04:26:38.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 --2- 172.21.15.81:0/936350345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80440ff900 0x7f80441061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 --2- 172.21.15.81:0/936350345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80440fdcf0 0x7f80440fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 --2- 172.21.15.81:0/936350345 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440ff3c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 -- 172.21.15.81:0/936350345 >> 172.21.15.81:0/936350345 conn(0x7f80440f93c0 msgr2=0x7f80440fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 -- 172.21.15.81:0/936350345 shutdown_connections 2022-09-10T04:26:38.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.346+0000 7f80495ff700 1 -- 172.21.15.81:0/936350345 wait complete. 2022-09-10T04:26:38.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.347+0000 7f80495ff700 1 Processor -- start 2022-09-10T04:26:38.353 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.347+0000 7f80495ff700 1 -- start start 2022-09-10T04:26:38.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80495ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80440fdcf0 0x7f8044072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80495ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80495ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80440ff900 0x7f8044071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80495ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044071e10 con 0x7f80440ff900 2022-09-10T04:26:38.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80495ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044071f80 con 0x7f80440fdcf0 2022-09-10T04:26:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80495ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804419d9d0 con 0x7f80440fef70 2022-09-10T04:26:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80427fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80427fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43368/0 (socket says 172.21.15.81:43368) 2022-09-10T04:26:38.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80427fc700 1 -- 172.21.15.81:0/2932496721 learned_addr learned my addr 172.21.15.81:0/2932496721 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:38.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f80437fe700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80440ff900 0x7f8044071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.348+0000 7f8042ffd700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80440fdcf0 0x7f8044072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80427fc700 1 -- 172.21.15.81:0/2932496721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80440fdcf0 msgr2=0x7f8044072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80427fc700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80440fdcf0 0x7f8044072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80427fc700 1 -- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80440ff900 msgr2=0x7f8044071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80427fc700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80440ff900 0x7f8044071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80427fc700 1 -- 172.21.15.81:0/2932496721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80380093b0 con 0x7f80440fef70 2022-09-10T04:26:38.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80427fc700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440710d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8038004230 tx=0x7f8038004260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:38.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803801d070 con 0x7f80440fef70 2022-09-10T04:26:38.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8038003bb0 con 0x7f80440fef70 2022-09-10T04:26:38.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8038017b80 con 0x7f80440fef70 2022-09-10T04:26:38.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804419dc50 con 0x7f80440fef70 2022-09-10T04:26:38.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.349+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804419e050 con 0x7f80440fef70 2022-09-10T04:26:38.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.350+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8044109510 con 0x7f80440fef70 2022-09-10T04:26:38.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.352+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f803800c040 con 0x7f80440fef70 2022-09-10T04:26:38.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.353+0000 7f8027fff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f802c099090 0x7f802c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.353+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f80380c3200 con 0x7f80440fef70 2022-09-10T04:26:38.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.354+0000 7f8042ffd700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f802c099090 0x7f802c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.354+0000 7f8042ffd700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f802c099090 0x7f802c09b540 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f80440fedd0 tx=0x7f8034005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:38.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.357+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f803808f6a0 con 0x7f80440fef70 2022-09-10T04:26:38.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.549+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8044104b80 con 0x7f80440fef70 2022-09-10T04:26:38.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.551+0000 7f8027fff700 1 -- 172.21.15.81:0/2932496721 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7f8038092cc0 con 0x7f80440fef70 2022-09-10T04:26:38.553 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:38.553 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:38.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f802c099090 msgr2=0x7f802c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f802c099090 0x7f802c09b540 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f80440fedd0 tx=0x7f8034005f60 comp rx=0 tx=0).stop 2022-09-10T04:26:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 msgr2=0x7f80440710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440710d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8038004230 tx=0x7f8038004260 comp rx=0 tx=0).stop 2022-09-10T04:26:38.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 shutdown_connections 2022-09-10T04:26:38.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80440ff900 0x7f8044071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80440fdcf0 0x7f8044072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f802c099090 0x7f802c09b540 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 --2- 172.21.15.81:0/2932496721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80440fef70 0x7f80440710d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 >> 172.21.15.81:0/2932496721 conn(0x7f80440f93c0 msgr2=0x7f80440fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:38.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 shutdown_connections 2022-09-10T04:26:38.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.554+0000 7f80495ff700 1 -- 172.21.15.81:0/2932496721 wait complete. 2022-09-10T04:26:38.602 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:38.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 -- 172.21.15.81:0/2694273560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 msgr2=0x7f70500fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 --2- 172.21.15.81:0/2694273560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f70500fe100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7040009750 tx=0x7f7040009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:38.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 -- 172.21.15.81:0/2694273560 shutdown_connections 2022-09-10T04:26:38.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 --2- 172.21.15.81:0/2694273560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500ff900 0x7f70501061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 --2- 172.21.15.81:0/2694273560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f70500fe100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 --2- 172.21.15.81:0/2694273560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500fef70 0x7f70500ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.835 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 -- 172.21.15.81:0/2694273560 >> 172.21.15.81:0/2694273560 conn(0x7f70500f93c0 msgr2=0x7f70500fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:38.835 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 -- 172.21.15.81:0/2694273560 shutdown_connections 2022-09-10T04:26:38.835 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.828+0000 7f70567ff700 1 -- 172.21.15.81:0/2694273560 wait complete. 2022-09-10T04:26:38.835 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 Processor -- start 2022-09-10T04:26:38.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 -- start start 2022-09-10T04:26:38.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f7050195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 0x7f7050196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500ff900 0x7f705019d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019d760 con 0x7f70500fef70 2022-09-10T04:26:38.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019d8d0 con 0x7f70500fdcf0 2022-09-10T04:26:38.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f70567ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705019da90 con 0x7f70500ff900 2022-09-10T04:26:38.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f704f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 0x7f7050196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f704ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f7050195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f704f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 0x7f7050196120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45844/0 (socket says 172.21.15.81:45844) 2022-09-10T04:26:38.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f704ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f7050195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56734/0 (socket says 172.21.15.81:56734) 2022-09-10T04:26:38.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f704f7fe700 1 -- 172.21.15.81:0/2008131352 learned_addr learned my addr 172.21.15.81:0/2008131352 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:38.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.830+0000 7f7054d9c700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500ff900 0x7f705019d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704ffff700 1 -- 172.21.15.81:0/2008131352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500ff900 msgr2=0x7f705019d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704ffff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500ff900 0x7f705019d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704ffff700 1 -- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 msgr2=0x7f7050196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:38.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704ffff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 0x7f7050196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:38.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704ffff700 1 -- 172.21.15.81:0/2008131352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70400093b0 con 0x7f70500fdcf0 2022-09-10T04:26:38.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f7054d9c700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500ff900 0x7f705019d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:26:38.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704f7fe700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 0x7f7050196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:38.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704ffff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f7050195be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f70400096a0 tx=0x7f7040004230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:38.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.831+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704001d070 con 0x7f70500fdcf0 2022-09-10T04:26:38.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.832+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7040003bb0 con 0x7f70500fdcf0 2022-09-10T04:26:38.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.832+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040017ac0 con 0x7f70500fdcf0 2022-09-10T04:26:38.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.832+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705019dd10 con 0x7f70500fdcf0 2022-09-10T04:26:38.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.832+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705019e110 con 0x7f70500fdcf0 2022-09-10T04:26:38.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.833+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7050109510 con 0x7f70500fdcf0 2022-09-10T04:26:38.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.834+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f704000c040 con 0x7f70500fdcf0 2022-09-10T04:26:38.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.836+0000 7f704d7fa700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7038099090 0x7f703809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:38.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.836+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f70400c31f0 con 0x7f70500fdcf0 2022-09-10T04:26:38.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.836+0000 7f704f7fe700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7038099090 0x7f703809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:38.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.837+0000 7f704f7fe700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7038099090 0x7f703809b540 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f70440057d0 tx=0x7f7044005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:38.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:38.840+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f704008f690 con 0x7f70500fdcf0 2022-09-10T04:26:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:38 smithi110 ceph-mon[97633]: pgmap v304: 129 pgs: 6 creating+peering, 105 active+clean, 18 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2022-09-10T04:26:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:38 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1198304253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:38 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4137078579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:38 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2932496721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:38 smithi104 ceph-mon[97044]: pgmap v304: 129 pgs: 6 creating+peering, 105 active+clean, 18 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2022-09-10T04:26:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:38 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1198304253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:38 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4137078579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:38 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2932496721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:38 smithi081 ceph-mon[90133]: pgmap v304: 129 pgs: 6 creating+peering, 105 active+clean, 18 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2022-09-10T04:26:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:38 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1198304253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:38 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4137078579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:38 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2932496721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.074+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f705005f8f0 con 0x7f70500fdcf0 2022-09-10T04:26:39.079 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.075+0000 7f704d7fa700 1 -- 172.21.15.81:0/2008131352 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10676 (secure 0 0 0) 0x7f7040092cb0 con 0x7f70500fdcf0 2022-09-10T04:26:39.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:39.080 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-09-10T04:26:32.097174+0000","modified":"2022-09-10T04:26:37.114599+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24577},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24577":{"gid":24577,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2022-09-10T04:26:39.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7038099090 msgr2=0x7f703809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7038099090 0x7f703809b540 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f70440057d0 tx=0x7f7044005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 msgr2=0x7f7050195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f7050195be0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f70400096a0 tx=0x7f7040004230 comp rx=0 tx=0).stop 2022-09-10T04:26:39.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 shutdown_connections 2022-09-10T04:26:39.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7038099090 0x7f703809b540 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70500ff900 0x7f705019d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f70500fdcf0 0x7f7050195be0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 --2- 172.21.15.81:0/2008131352 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70500fef70 0x7f7050196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 >> 172.21.15.81:0/2008131352 conn(0x7f70500f93c0 msgr2=0x7f7050104990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 shutdown_connections 2022-09-10T04:26:39.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.078+0000 7f70567ff700 1 -- 172.21.15.81:0/2008131352 wait complete. 2022-09-10T04:26:39.084 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2022-09-10T04:26:39.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-09-10T04:26:32.097174+0000', 'modified': '2022-09-10T04:26:37.114599+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24577}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24577': {'gid': 24577, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.81:6841/953424075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6840', 'nonce': 953424075}, {'type': 'v1', 'addr': '172.21.15.81:6841', 'nonce': 953424075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2022-09-10T04:26:39.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-09-10T04:26:39.108 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2022-09-10T04:26:39.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.333+0000 7f44b0d9d700 1 -- 172.21.15.81:0/3421373942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 msgr2=0x7f44ac103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.333+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/3421373942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac103340 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4494009810 tx=0x7f4494009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:39.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.333+0000 7f44b0d9d700 1 -- 172.21.15.81:0/3421373942 shutdown_connections 2022-09-10T04:26:39.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.333+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/3421373942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44ac103880 0x7f44ac105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.333+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/3421373942 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac0fe640 0x7f44ac100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.333+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/3421373942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac103340 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.334+0000 7f44b0d9d700 1 -- 172.21.15.81:0/3421373942 >> 172.21.15.81:0/3421373942 conn(0x7f44ac0f8160 msgr2=0x7f44ac0fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.334+0000 7f44b0d9d700 1 -- 172.21.15.81:0/3421373942 shutdown_connections 2022-09-10T04:26:39.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.334+0000 7f44b0d9d700 1 -- 172.21.15.81:0/3421373942 wait complete. 2022-09-10T04:26:39.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 Processor -- start 2022-09-10T04:26:39.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 -- start start 2022-09-10T04:26:39.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44ac0fe640 0x7f44ac1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 0x7f44ac19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac19d470 con 0x7f44ac100f60 2022-09-10T04:26:39.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac19d5e0 con 0x7f44ac0fe640 2022-09-10T04:26:39.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.335+0000 7f44b0d9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac19d7a0 con 0x7f44ac103880 2022-09-10T04:26:39.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 0x7f44ac19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44a3fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 0x7f44ac19cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43400/0 (socket says 172.21.15.81:43400) 2022-09-10T04:26:39.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44ac0fe640 0x7f44ac1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 -- 172.21.15.81:0/2611611599 learned_addr learned my addr 172.21.15.81:0/2611611599 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:39.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 -- 172.21.15.81:0/2611611599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44ac0fe640 msgr2=0x7f44ac1958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44ac0fe640 0x7f44ac1958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 -- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 msgr2=0x7f44ac195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 -- 172.21.15.81:0/2611611599 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494009470 con 0x7f44ac103880 2022-09-10T04:26:39.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44a3fff700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:39.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.336+0000 7f44abfff700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 0x7f44ac19cea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44a400ebf0 tx=0x7f44a400ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:39.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.337+0000 7f44a97fa700 1 -- 172.21.15.81:0/2611611599 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a40099c0 con 0x7f44ac103880 2022-09-10T04:26:39.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.337+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac19da80 con 0x7f44ac103880 2022-09-10T04:26:39.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.337+0000 7f44a97fa700 1 -- 172.21.15.81:0/2611611599 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f44a400cc90 con 0x7f44ac103880 2022-09-10T04:26:39.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.337+0000 7f44a97fa700 1 -- 172.21.15.81:0/2611611599 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a4010430 con 0x7f44ac103880 2022-09-10T04:26:39.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.337+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac19df40 con 0x7f44ac103880 2022-09-10T04:26:39.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.340+0000 7f44a97fa700 1 -- 172.21.15.81:0/2611611599 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f44a401e020 con 0x7f44ac103880 2022-09-10T04:26:39.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.342+0000 7f44c5556b80 1 Processor -- start 2022-09-10T04:26:39.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.342+0000 7f44c5556b80 1 -- start start 2022-09-10T04:26:39.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.342+0000 7f44a97fa700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4498099010 0x7f449809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.342+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.342+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbbebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbbf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44c5556b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdaa9080 con 0x562ccdbc0000 2022-09-10T04:26:39.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44c5556b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdb139e0 con 0x562ccdbc0950 2022-09-10T04:26:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44c5556b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdb0d180 con 0x562ccdbbf1c0 2022-09-10T04:26:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44a97fa700 1 -- 172.21.15.81:0/2611611599 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f44a4014070 con 0x7f44ac103880 2022-09-10T04:26:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44a3fff700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4498099010 0x7f449809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbbf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbbf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43412/0 (socket says 172.21.15.81:43412) 2022-09-10T04:26:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 -- 172.21.15.81:0/460491584 learned_addr learned my addr 172.21.15.81:0/460491584 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44a3fff700 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbbebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 -- 172.21.15.81:0/460491584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 msgr2=0x562ccdbbebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbbebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 -- 172.21.15.81:0/460491584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 msgr2=0x562ccdbc0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.343+0000 7f44abfff700 1 -- 172.21.15.81:0/460491584 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ccd9955e0 con 0x562ccdbbf1c0 2022-09-10T04:26:39.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44abfff700 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbbf630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44a400c5b0 tx=0x7f44a40928d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a718c671fab9723 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:39.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44a3fff700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4498099010 0x7f449809b4c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f449400b4f0 tx=0x7f449401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:39.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44a1ffb700 1 -- 172.21.15.81:0/460491584 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a40980d0 con 0x562ccdbbf1c0 2022-09-10T04:26:39.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44a1ffb700 1 -- 172.21.15.81:0/460491584 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f44a40c0580 con 0x562ccdbbf1c0 2022-09-10T04:26:39.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44a1ffb700 1 -- 172.21.15.81:0/460491584 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a408f3a0 con 0x562ccdbbf1c0 2022-09-10T04:26:39.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44c5556b80 1 -- 172.21.15.81:0/460491584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 msgr2=0x562ccdbbf630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.344+0000 7f44c5556b80 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbbf630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f44a400c5b0 tx=0x7f44a40928d0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 -- 172.21.15.81:0/460491584 shutdown_connections 2022-09-10T04:26:39.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbbf630 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 --2- 172.21.15.81:0/460491584 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbbebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 -- 172.21.15.81:0/460491584 >> 172.21.15.81:0/460491584 conn(0x562ccdbbb170 msgr2=0x562ccdbb95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 -- 172.21.15.81:0/460491584 shutdown_connections 2022-09-10T04:26:39.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 -- 172.21.15.81:0/460491584 wait complete. 2022-09-10T04:26:39.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.345+0000 7f44c5556b80 1 -- start start 2022-09-10T04:26:39.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 Processor -- start 2022-09-10T04:26:39.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbc8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 0x562ccdbcfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccd9fb0c0 con 0x562ccdbc0950 2022-09-10T04:26:39.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbd0120 con 0x562ccdbc0000 2022-09-10T04:26:39.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbd02b0 con 0x562ccdbbf1c0 2022-09-10T04:26:39.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44c5556b80 10 client.0 _ll_register_callbacks cb 0x562ccdab8370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:26:39.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44abfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 0x562ccdbcfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44abfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 0x562ccdbcfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45890/0 (socket says 172.21.15.81:45890) 2022-09-10T04:26:39.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44abfff700 1 -- 172.21.15.81:0/2448921611 learned_addr learned my addr 172.21.15.81:0/2448921611 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:39.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44a3fff700 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbc8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.346+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43414/0 (socket says 172.21.15.81:43414) 2022-09-10T04:26:39.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44abfff700 1 -- 172.21.15.81:0/2448921611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 msgr2=0x562ccdbc8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44abfff700 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44abfff700 1 -- 172.21.15.81:0/2448921611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 msgr2=0x562ccdbc8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44abfff700 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbc8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44abfff700 1 -- 172.21.15.81:0/2448921611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac19df40 con 0x562ccdbc0950 2022-09-10T04:26:39.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44abfff700 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 0x562ccdbcfbe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f44a4096040 tx=0x7f44a4005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fc39b98a75670f server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:39.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f448affd700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:26:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f448affd700 1 -- 172.21.15.81:0/2448921611 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a40be070 con 0x562ccdbc0950 2022-09-10T04:26:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f448affd700 1 -- 172.21.15.81:0/2448921611 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f44a4092c90 con 0x562ccdbc0950 2022-09-10T04:26:39.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.347+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ccdbb1cf0 con 0x562ccdbc0950 2022-09-10T04:26:39.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f448affd700 1 -- 172.21.15.81:0/2448921611 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44a4093390 con 0x562ccdbc0950 2022-09-10T04:26:39.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 10 client.14586 fetch_fsmap learned FSMap version 17 2022-09-10T04:26:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562ccd941d90 con 0x562ccdbc0950 2022-09-10T04:26:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f448affd700 1 -- 172.21.15.81:0/2448921611 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x562ccdbb1cf0 con 0x562ccdbc0950 2022-09-10T04:26:39.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f448affd700 1 -- 172.21.15.81:0/2448921611 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) v1 ==== 7468+0+0 (secure 0 0 0) 0x7f44a404b2c0 con 0x562ccdbc0950 2022-09-10T04:26:39.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 10 client.14586 fetch_fsmap finished waiting for FSMap version 17 2022-09-10T04:26:39.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 10 client.14586 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-10T04:26:39.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 20 client.14586 populate_metadata read hostname 'smithi081' 2022-09-10T04:26:39.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdbd34d0 0x562ccdbd1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 4 client.14586 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:26:39.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ccdb8e4c0 con 0x562ccdbd34d0 2022-09-10T04:26:39.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.348+0000 7f44ab7fe700 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdbd34d0 0x562ccdbd1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.349+0000 7f448affd700 10 client.14586 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:26:39.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.349+0000 7f44ab7fe700 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdbd34d0 0x562ccdbd1ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4577872c3eb9e492 server_cookie=1a5e89d2ffdb2cb1 in_seq=0 out_seq=0 2022-09-10T04:26:39.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f448affd700 1 -- 172.21.15.81:0/2448921611 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x562ccdb8e4c0 con 0x562ccdbd34d0 2022-09-10T04:26:39.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f448affd700 10 client.14586 handle_command_reply: tid=0 2022-09-10T04:26:39.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 client.14586 shutdown 2022-09-10T04:26:39.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 msgr2=0x562ccdbcfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 0x562ccdbcfbe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f44a4096040 tx=0x7f44a4005db0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 shutdown_connections 2022-09-10T04:26:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0950 0x562ccdbcfbe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdbd34d0 0x562ccdbd1ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.382 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2448921611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbc8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 >> 172.21.15.81:0/2448921611 conn(0x562ccdbbb170 msgr2=0x562ccdbb8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.350+0000 7f448affd700 0 client.14586 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:26:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.351+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 shutdown_connections 2022-09-10T04:26:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.351+0000 7f44c5556b80 1 -- 172.21.15.81:0/2448921611 wait complete. 2022-09-10T04:26:39.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.351+0000 7f44c5556b80 20 client.14586 trim_cache size 0 max 16384 2022-09-10T04:26:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.368+0000 7f44c5556b80 1 Processor -- start 2022-09-10T04:26:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.368+0000 7f44c5556b80 1 -- start start 2022-09-10T04:26:39.389 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbc8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44c5556b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbc8230 con 0x562ccdbc0000 2022-09-10T04:26:39.390 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44c5556b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbba9f0 con 0x562ccdbc0950 2022-09-10T04:26:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44c5556b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbc9850 con 0x562ccdbbf1c0 2022-09-10T04:26:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44a3fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44a3fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45896/0 (socket says 172.21.15.81:45896) 2022-09-10T04:26:39.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44a3fff700 1 -- 172.21.15.81:0/760169721 learned_addr learned my addr 172.21.15.81:0/760169721 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44abfff700 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbc8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.369+0000 7f44ab7fe700 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a3fff700 1 -- 172.21.15.81:0/760169721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 msgr2=0x562ccdbc6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a3fff700 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a3fff700 1 -- 172.21.15.81:0/760169721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 msgr2=0x562ccdbc8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a3fff700 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbc8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a3fff700 1 -- 172.21.15.81:0/760169721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ccd940260 con 0x562ccdbc0000 2022-09-10T04:26:39.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a3fff700 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc8410 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4494015000 tx=0x7f4494009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e028f567c38aaf server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:39.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a27fc700 1 -- 172.21.15.81:0/760169721 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449402d050 con 0x562ccdbc0000 2022-09-10T04:26:39.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a27fc700 1 -- 172.21.15.81:0/760169721 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f449402e040 con 0x562ccdbc0000 2022-09-10T04:26:39.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.370+0000 7f44a27fc700 1 -- 172.21.15.81:0/760169721 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494033e40 con 0x562ccdbc0000 2022-09-10T04:26:39.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 -- 172.21.15.81:0/760169721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 msgr2=0x562ccdbc8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc8410 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4494015000 tx=0x7f4494009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 -- 172.21.15.81:0/760169721 shutdown_connections 2022-09-10T04:26:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0950 0x562ccdbc8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbbf1c0 0x562ccdbc6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 --2- 172.21.15.81:0/760169721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbc0000 0x562ccdbc8410 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 -- 172.21.15.81:0/760169721 >> 172.21.15.81:0/760169721 conn(0x562ccdbbb170 msgr2=0x562ccdbc63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 -- 172.21.15.81:0/760169721 shutdown_connections 2022-09-10T04:26:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 -- 172.21.15.81:0/760169721 wait complete. 2022-09-10T04:26:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.371+0000 7f44c5556b80 1 -- start start 2022-09-10T04:26:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 Processor -- start 2022-09-10T04:26:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 0x562ccdbd9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbda0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 0x562ccdbdcf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbdd460 con 0x562ccdbbf1c0 2022-09-10T04:26:39.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbdd5d0 con 0x562ccdbc0000 2022-09-10T04:26:39.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ccdbdd790 con 0x562ccdbc0950 2022-09-10T04:26:39.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44c5556b80 10 client.0 _ll_register_callbacks cb 0x562ccdab8370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:26:39.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 0x562ccdbd9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 0x562ccdbdcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbda0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbda0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56784/0 (socket says 172.21.15.81:56784) 2022-09-10T04:26:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44ab7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 0x562ccdbd9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45908/0 (socket says 172.21.15.81:45908) 2022-09-10T04:26:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 0x562ccdbdcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:43430/0 (socket says 172.21.15.81:43430) 2022-09-10T04:26:39.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.372+0000 7f44ab7fe700 1 -- 172.21.15.81:0/2440066127 learned_addr learned my addr 172.21.15.81:0/2440066127 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44abfff700 1 -- 172.21.15.81:0/2440066127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 msgr2=0x562ccdbda0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44abfff700 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbda0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44abfff700 1 -- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 msgr2=0x562ccdbd9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:26:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44abfff700 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 0x562ccdbd9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44abfff700 1 -- 172.21.15.81:0/2440066127 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4494020040 con 0x562ccdbc0950 2022-09-10T04:26:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44ab7fe700 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 0x562ccdbd9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:39.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44abfff700 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 0x562ccdbdcf20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f44a408f700 tx=0x7f44a4099e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ac5027d8d3a547b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:39.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f448affd700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:26:39.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f448affd700 1 -- 172.21.15.81:0/2440066127 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a4016530 con 0x562ccdbc0950 2022-09-10T04:26:39.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44a40c0580 con 0x562ccdbc0950 2022-09-10T04:26:39.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f448affd700 1 -- 172.21.15.81:0/2440066127 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f44a4091030 con 0x562ccdbc0950 2022-09-10T04:26:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f448affd700 1 -- 172.21.15.81:0/2440066127 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44a4092390 con 0x562ccdbc0950 2022-09-10T04:26:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44c5556b80 10 client.24694 fetch_fsmap learned FSMap version 17 2022-09-10T04:26:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.373+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x562ccda0ea30 con 0x562ccdbc0950 2022-09-10T04:26:39.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f448affd700 1 -- 172.21.15.81:0/2440066127 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f44a40c0580 con 0x562ccdbc0950 2022-09-10T04:26:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f448affd700 1 -- 172.21.15.81:0/2440066127 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 17) v1 ==== 7468+0+0 (secure 0 0 0) 0x7f44a408ea80 con 0x562ccdbc0950 2022-09-10T04:26:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44c5556b80 10 client.24694 fetch_fsmap finished waiting for FSMap version 17 2022-09-10T04:26:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44c5556b80 10 client.24694 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-10T04:26:39.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44c5556b80 20 client.24694 populate_metadata read hostname 'smithi081' 2022-09-10T04:26:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdc2a3f0 0x562ccdc27070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44c5556b80 4 client.24694 mds_command: new command op to 24577 tid=0[{"prefix": "status"}] 2022-09-10T04:26:39.406 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562ccdb066f0 con 0x562ccdc2a3f0 2022-09-10T04:26:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.374+0000 7f44ab7fe700 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdc2a3f0 0x562ccdc27070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f448affd700 10 client.24694 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:26:39.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44ab7fe700 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdc2a3f0 0x562ccdc27070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=464ea80112b66201 server_cookie=c8da2c64b3cba110 in_seq=0 out_seq=0 2022-09-10T04:26:39.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f448affd700 1 -- 172.21.15.81:0/2440066127 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x562ccdb066f0 con 0x562ccdc2a3f0 2022-09-10T04:26:39.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f448affd700 10 client.24694 handle_command_reply: tid=0 2022-09-10T04:26:39.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 client.24694 shutdown 2022-09-10T04:26:39.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 msgr2=0x562ccdbdcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 0x562ccdbdcf20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f44a408f700 tx=0x7f44a4099e20 comp rx=0 tx=0).stop 2022-09-10T04:26:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 shutdown_connections 2022-09-10T04:26:39.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562ccdbc0950 0x562ccdbdcf20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562ccdbbf1c0 0x562ccdbd9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.411 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x562ccdc2a3f0 0x562ccdc27070 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.412 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 --2- 172.21.15.81:0/2440066127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562ccdbc0000 0x562ccdbda0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.412 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 >> 172.21.15.81:0/2440066127 conn(0x562ccdbbb170 msgr2=0x562ccdbc8410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.412 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.375+0000 7f448affd700 0 client.24694 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:26:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 shutdown_connections 2022-09-10T04:26:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44c5556b80 1 -- 172.21.15.81:0/2440066127 wait complete. 2022-09-10T04:26:39.413 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44c5556b80 20 client.24694 trim_cache size 0 max 16384 2022-09-10T04:26:39.414 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:26:39.414 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:26:39.414 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2022-09-10T04:26:39.415 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24577, 2022-09-10T04:26:39.415 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:26:39.415 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:26:39.415 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:26:39.415 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 7.2758232710000001, 2022-09-10T04:26:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2022-09-10T04:26:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 90, 2022-09-10T04:26:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 89, 2022-09-10T04:26:39.416 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 17.363438163000001 2022-09-10T04:26:39.417 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:26:39.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4498099010 msgr2=0x7f449809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4498099010 0x7f449809b4c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f449400b4f0 tx=0x7f449401a040 comp rx=0 tx=0).stop 2022-09-10T04:26:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 msgr2=0x7f44ac19cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 0x7f44ac19cea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f44a400ebf0 tx=0x7f44a400ef00 comp rx=0 tx=0).stop 2022-09-10T04:26:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 shutdown_connections 2022-09-10T04:26:39.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4498099010 0x7f449809b4c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44ac103880 0x7f44ac19cea0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44ac0fe640 0x7f44ac1958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 --2- 172.21.15.81:0/2611611599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44ac100f60 0x7f44ac195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.376+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 >> 172.21.15.81:0/2611611599 conn(0x7f44ac0f8160 msgr2=0x7f44ac0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.377+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 shutdown_connections 2022-09-10T04:26:39.421 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:39.377+0000 7f44b0d9d700 1 -- 172.21.15.81:0/2611611599 wait complete. 2022-09-10T04:26:39.421 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2022-09-10T04:26:39.422 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:39.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph fs ls 2022-09-10T04:26:39.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:39 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2008131352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:39 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2008131352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:39 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2008131352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:40.295 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:26:40.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:40 smithi081 ceph-mon[90133]: pgmap v305: 129 pgs: 6 creating+peering, 116 active+clean, 7 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 214 B/s wr, 5 op/s 2022-09-10T04:26:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:40 smithi110 ceph-mon[97633]: pgmap v305: 129 pgs: 6 creating+peering, 116 active+clean, 7 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 214 B/s wr, 5 op/s 2022-09-10T04:26:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:40 smithi104 ceph-mon[97044]: pgmap v305: 129 pgs: 6 creating+peering, 116 active+clean, 7 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 214 B/s wr, 5 op/s 2022-09-10T04:26:41.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 -- 172.21.15.81:0/2187525818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47400666c0 msgr2=0x7f4740066a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 --2- 172.21.15.81:0/2187525818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47400666c0 0x7f4740066a90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4734009750 tx=0x7f4734009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 -- 172.21.15.81:0/2187525818 shutdown_connections 2022-09-10T04:26:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 --2- 172.21.15.81:0/2187525818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f474010aa10 0x7f474010cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:41.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 --2- 172.21.15.81:0/2187525818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47400666c0 0x7f4740066a90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 --2- 172.21.15.81:0/2187525818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4740066fd0 0x7f474010a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 -- 172.21.15.81:0/2187525818 >> 172.21.15.81:0/2187525818 conn(0x7f4740073cf0 msgr2=0x7f47400740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 -- 172.21.15.81:0/2187525818 shutdown_connections 2022-09-10T04:26:41.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.812+0000 7f4746bb3700 1 -- 172.21.15.81:0/2187525818 wait complete. 2022-09-10T04:26:41.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.813+0000 7f4746bb3700 1 Processor -- start 2022-09-10T04:26:41.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 -- start start 2022-09-10T04:26:41.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 0x7f4740195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:41.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4740066fd0 0x7f47401963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:41.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f474010aa10 0x7f474019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:41.821 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474019aed0 con 0x7f474010aa10 2022-09-10T04:26:41.821 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474019b040 con 0x7f47400666c0 2022-09-10T04:26:41.822 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4746bb3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474019b200 con 0x7f4740066fd0 2022-09-10T04:26:41.822 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f473ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4740066fd0 0x7f47401963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:41.822 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f474494f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 0x7f4740195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:41.823 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f474494f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 0x7f4740195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:58580/0 (socket says 172.21.15.81:58580) 2022-09-10T04:26:41.823 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f474494f700 1 -- 172.21.15.81:0/398924691 learned_addr learned my addr 172.21.15.81:0/398924691 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:41.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f473ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4740066fd0 0x7f47401963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:33592/0 (socket says 172.21.15.81:33592) 2022-09-10T04:26:41.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.814+0000 7f4745150700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f474010aa10 0x7f474019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:41.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f474494f700 1 -- 172.21.15.81:0/398924691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4740066fd0 msgr2=0x7f47401963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:41.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f474494f700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4740066fd0 0x7f47401963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:41.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f474494f700 1 -- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f474010aa10 msgr2=0x7f474019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:41.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f474494f700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f474010aa10 0x7f474019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:41.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f474494f700 1 -- 172.21.15.81:0/398924691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c009470 con 0x7f47400666c0 2022-09-10T04:26:41.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f474494f700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 0x7f4740195ea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4734006010 tx=0x7f4734004230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:41.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473401d070 con 0x7f47400666c0 2022-09-10T04:26:41.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4734003500 con 0x7f47400666c0 2022-09-10T04:26:41.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47340093b0 con 0x7f47400666c0 2022-09-10T04:26:41.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.815+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734004760 con 0x7f47400666c0 2022-09-10T04:26:41.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.816+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47401a2af0 con 0x7f47400666c0 2022-09-10T04:26:41.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.816+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4740064ea0 con 0x7f47400666c0 2022-09-10T04:26:41.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.818+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f473400c040 con 0x7f47400666c0 2022-09-10T04:26:41.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.820+0000 7f473dffb700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4730099010 0x7f473009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:41.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.820+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f47340c36b0 con 0x7f47400666c0 2022-09-10T04:26:41.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.820+0000 7f473ffff700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4730099010 0x7f473009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:41.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.820+0000 7f473ffff700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4730099010 0x7f473009b4c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f47401974b0 tx=0x7f472c005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:41.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:41.822+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f473408fb50 con 0x7f47400666c0 2022-09-10T04:26:42.078 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.076+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f474010ff20 con 0x7f47400666c0 2022-09-10T04:26:42.079 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.077+0000 7f473dffb700 1 -- 172.21.15.81:0/398924691 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4734093170 con 0x7f47400666c0 2022-09-10T04:26:42.080 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2022-09-10T04:26:42.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.079+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4730099010 msgr2=0x7f473009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:42.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.079+0000 7f4746bb3700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4730099010 0x7f473009b4c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f47401974b0 tx=0x7f472c005f60 comp rx=0 tx=0).stop 2022-09-10T04:26:42.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.079+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 msgr2=0x7f4740195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:42.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.079+0000 7f4746bb3700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 0x7f4740195ea0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4734006010 tx=0x7f4734004230 comp rx=0 tx=0).stop 2022-09-10T04:26:42.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 shutdown_connections 2022-09-10T04:26:42.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f474010aa10 0x7f474019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:42.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47400666c0 0x7f4740195ea0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:42.086 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4730099010 0x7f473009b4c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:42.086 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 --2- 172.21.15.81:0/398924691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4740066fd0 0x7f47401963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:42.087 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 >> 172.21.15.81:0/398924691 conn(0x7f4740073cf0 msgr2=0x7f47400fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:42.087 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 shutdown_connections 2022-09-10T04:26:42.088 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:42.080+0000 7f4746bb3700 1 -- 172.21.15.81:0/398924691 wait complete. 2022-09-10T04:26:42.562 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2022-09-10T04:26:42.563 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-09-10T04:26:42.563 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2022-09-10T04:26:42.564 INFO:tasks.cephfs.mount:self.client.name = client.admin 2022-09-10T04:26:42.564 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:42.564 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2022-09-10T04:26:42.565 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-09-10T04:26:42.566 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-09-10T04:26:42.566 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2022-09-10T04:26:42.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:42.569 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2022-09-10T04:26:42.630 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-09-10T04:26:42.631 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-09-10T04:26:42.632 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2022-09-10T04:26:42.632 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-09-10T04:26:42.632 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2022-09-10T04:26:42.633 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-09-10T04:26:42.633 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-10T04:26:42.633 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:26:42.633 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2022-09-10T04:26:42.634 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-10T04:26:42.634 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:26:42.634 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2022-09-10T04:26:42.634 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-10T04:26:42.635 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:26:42.635 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-09-10T04:26:42.635 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:26:42.635 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-09-10T04:26:42.636 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41676sec preferred_lft 41676sec 2022-09-10T04:26:42.636 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2022-09-10T04:26:42.636 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-09-10T04:26:42.639 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-09-10T04:26:42.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:42.640 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:26:42.641 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:26:42.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2022-09-10T04:26:42.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2022-09-10T04:26:42.642 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2022-09-10T04:26:42.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-09-10T04:26:42.644 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:26:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:42 smithi081 ceph-mon[90133]: pgmap v306: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2022-09-10T04:26:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:42 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/398924691' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-10T04:26:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:42 smithi110 ceph-mon[97633]: pgmap v306: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2022-09-10T04:26:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:42 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/398924691' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-10T04:26:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:42 smithi104 ceph-mon[97044]: pgmap v306: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2022-09-10T04:26:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:42 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/398924691' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-10T04:26:43.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:43.145 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-09-10T04:26:43.236 INFO:teuthology.orchestra.run.smithi081.stdout:1 2022-09-10T04:26:43.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:43.247 DEBUG:teuthology.orchestra.run.smithi081:> route 2022-09-10T04:26:43.328 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2022-09-10T04:26:43.329 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-09-10T04:26:43.329 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-09-10T04:26:43.329 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-09-10T04:26:43.330 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-09-10T04:26:43.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:43.331 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:26:43.331 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:26:43.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-09-10T04:26:43.332 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-09-10T04:26:43.333 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-09-10T04:26:43.333 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:26:44.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:44.107 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2022-09-10T04:26:44.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:44.168 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2022-09-10T04:26:44.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:44.231 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:26:44.232 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:26:44.232 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2022-09-10T04:26:44.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2022-09-10T04:26:44.233 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:26:44.365 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:44 smithi081 ceph-mon[90133]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s wr, 3 op/s 2022-09-10T04:26:44.417 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2022-09-10T04:26:44.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:44.419 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:26:44.419 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:26:44.420 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2022-09-10T04:26:44.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-09-10T04:26:44.421 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2022-09-10T04:26:44.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2022-09-10T04:26:44.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2022-09-10T04:26:44.423 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:26:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:44 smithi104 ceph-mon[97044]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s wr, 3 op/s 2022-09-10T04:26:44.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:44 smithi110 ceph-mon[97633]: pgmap v307: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 131 B/s wr, 3 op/s 2022-09-10T04:26:44.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:44.764 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:26:44.765 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:26:44.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2022-09-10T04:26:44.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2022-09-10T04:26:44.767 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:26:44.982 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2022-09-10T04:26:44.983 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-10T04:26:44.984 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:45.044 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2022-09-10T04:26:45.045 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-10T04:26:45.045 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:45.103 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2022-09-10T04:26:45.194 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-09-10T04:26:45.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-09-10T04:26:45.282 INFO:teuthology.orchestra.run.smithi081.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-09-10T04:26:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-09-10T04:26:45.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:45.288 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2022-09-10T04:26:45.346 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-09-10T04:26:45.348 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_mountpoint=/ --client_fs=cephfs) 2022-09-10T04:26:45.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2022-09-10T04:26:45.445 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-09-10T04:26:45.446 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-09-10T04:26:45.535 INFO:teuthology.orchestra.run.smithi081.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-09-10T04:26:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-09-10T04:26:45.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:45.617 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2022-09-10T04:26:45.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:2022-09-10T04:26:45.580+0000 7f6023152400 -1 init, newargv = 0x563cbb6d6730 newargc=15 2022-09-10T04:26:45.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:ceph-fuse[127937]: starting ceph client 2022-09-10T04:26:45.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:ceph-fuse[127937]: starting fuse 2022-09-10T04:26:45.677 INFO:teuthology.orchestra.run.smithi081.stdout:120 2022-09-10T04:26:45.677 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2022-09-10T04:26:45.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-09-10T04:26:45.679 DEBUG:teuthology.orchestra.run.smithi081:> import glob 2022-09-10T04:26:45.679 DEBUG:teuthology.orchestra.run.smithi081:> import re 2022-09-10T04:26:45.679 DEBUG:teuthology.orchestra.run.smithi081:> import os 2022-09-10T04:26:45.680 DEBUG:teuthology.orchestra.run.smithi081:> import subprocess 2022-09-10T04:26:45.680 DEBUG:teuthology.orchestra.run.smithi081:> 2022-09-10T04:26:45.681 DEBUG:teuthology.orchestra.run.smithi081:> def _find_admin_socket(client_name): 2022-09-10T04:26:45.681 DEBUG:teuthology.orchestra.run.smithi081:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2022-09-10T04:26:45.682 DEBUG:teuthology.orchestra.run.smithi081:> files = glob.glob(asok_path) 2022-09-10T04:26:45.682 DEBUG:teuthology.orchestra.run.smithi081:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2022-09-10T04:26:45.682 DEBUG:teuthology.orchestra.run.smithi081:> 2022-09-10T04:26:45.683 DEBUG:teuthology.orchestra.run.smithi081:> # Given a non-glob path, it better be there 2022-09-10T04:26:45.684 DEBUG:teuthology.orchestra.run.smithi081:> if "*" not in asok_path: 2022-09-10T04:26:45.684 DEBUG:teuthology.orchestra.run.smithi081:> assert(len(files) == 1) 2022-09-10T04:26:45.685 DEBUG:teuthology.orchestra.run.smithi081:> return files[0] 2022-09-10T04:26:45.686 DEBUG:teuthology.orchestra.run.smithi081:> 2022-09-10T04:26:45.687 DEBUG:teuthology.orchestra.run.smithi081:> for f in files: 2022-09-10T04:26:45.688 DEBUG:teuthology.orchestra.run.smithi081:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-09-10T04:26:45.688 DEBUG:teuthology.orchestra.run.smithi081:> if os.path.exists("/proc/{0}".format(pid)): 2022-09-10T04:26:45.689 DEBUG:teuthology.orchestra.run.smithi081:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2022-09-10T04:26:45.689 DEBUG:teuthology.orchestra.run.smithi081:> contents = proc_f.read() 2022-09-10T04:26:45.690 DEBUG:teuthology.orchestra.run.smithi081:> if mountpoint in contents: 2022-09-10T04:26:45.690 DEBUG:teuthology.orchestra.run.smithi081:> return f 2022-09-10T04:26:45.691 DEBUG:teuthology.orchestra.run.smithi081:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-09-10T04:26:45.691 DEBUG:teuthology.orchestra.run.smithi081:> 2022-09-10T04:26:45.691 DEBUG:teuthology.orchestra.run.smithi081:> print(_find_admin_socket("client.admin")) 2022-09-10T04:26:45.692 DEBUG:teuthology.orchestra.run.smithi081:> ' 2022-09-10T04:26:45.862 INFO:teuthology.orchestra.run.smithi081.stdout:/var/run/ceph/ceph-client.admin.127937.asok 2022-09-10T04:26:46.040 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.127937.asok 2022-09-10T04:26:46.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:46.042 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.127937.asok status 2022-09-10T04:26:46.291 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:26:46.291 INFO:teuthology.orchestra.run.smithi081.stdout: "metadata": { 2022-09-10T04:26:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_sha1": "244418605927bb7842e73fc5d2a53889b760937f", 2022-09-10T04:26:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 17.0.0-14898-g24441860 (244418605927bb7842e73fc5d2a53889b760937f) quincy (dev)", 2022-09-10T04:26:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "entity_id": "admin", 2022-09-10T04:26:46.292 INFO:teuthology.orchestra.run.smithi081.stdout: "hostname": "smithi081", 2022-09-10T04:26:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2022-09-10T04:26:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "pid": "127937", 2022-09-10T04:26:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "root": "/" 2022-09-10T04:26:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2022-09-10T04:26:46.293 INFO:teuthology.orchestra.run.smithi081.stdout: "dentry_count": 0, 2022-09-10T04:26:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "dentry_pinned_count": 0, 2022-09-10T04:26:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 14604, 2022-09-10T04:26:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "inst": { 2022-09-10T04:26:46.294 INFO:teuthology.orchestra.run.smithi081.stdout: "name": { 2022-09-10T04:26:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client", 2022-09-10T04:26:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "num": 14604 2022-09-10T04:26:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2022-09-10T04:26:46.295 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": { 2022-09-10T04:26:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2022-09-10T04:26:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "192.168.0.1:0", 2022-09-10T04:26:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2927908702 2022-09-10T04:26:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: } 2022-09-10T04:26:46.296 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2022-09-10T04:26:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": { 2022-09-10T04:26:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2022-09-10T04:26:46.297 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "192.168.0.1:0", 2022-09-10T04:26:46.298 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2927908702 2022-09-10T04:26:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2022-09-10T04:26:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "inst_str": "client.14604 192.168.0.1:0/2927908702", 2022-09-10T04:26:46.300 INFO:teuthology.orchestra.run.smithi081.stdout: "addr_str": "192.168.0.1:0/2927908702", 2022-09-10T04:26:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "inode_count": 1, 2022-09-10T04:26:46.302 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_epoch": 17, 2022-09-10T04:26:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_epoch": 90, 2022-09-10T04:26:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_epoch_barrier": 0, 2022-09-10T04:26:46.303 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklisted": false, 2022-09-10T04:26:46.304 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs" 2022-09-10T04:26:46.304 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:26:46.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:46.314 DEBUG:teuthology.orchestra.run.smithi081:> stat --file-system '--printf=%T 2022-09-10T04:26:46.316 DEBUG:teuthology.orchestra.run.smithi081:> ' -- /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:46.376 INFO:teuthology.orchestra.run.smithi081.stdout:fuseblk 2022-09-10T04:26:46.376 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:46.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:46.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:46.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:46.475 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2022-09-10T04:26:46.625 INFO:teuthology.orchestra.run.smithi081.stdout:# file: . 2022-09-10T04:26:46.626 INFO:teuthology.orchestra.run.smithi081.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2022-09-10T04:26:46.626 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:46.629 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:26:46.629 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2022-09-10T04:26:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:46 smithi110 ceph-mon[97633]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:46.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:46.767 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:46 smithi081 ceph-mon[90133]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:46 smithi104 ceph-mon[97044]: pgmap v308: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 -- 172.21.15.81:0/855094144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500ff900 msgr2=0x7f5d501061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 --2- 172.21.15.81:0/855094144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500ff900 0x7f5d501061c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009bb0 tx=0x7f5d48009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:46.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 -- 172.21.15.81:0/855094144 shutdown_connections 2022-09-10T04:26:46.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 --2- 172.21.15.81:0/855094144 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500ff900 0x7f5d501061c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:46.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 --2- 172.21.15.81:0/855094144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d500fdcf0 0x7f5d500fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:46.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 --2- 172.21.15.81:0/855094144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d500ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:46.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 -- 172.21.15.81:0/855094144 >> 172.21.15.81:0/855094144 conn(0x7f5d500f93c0 msgr2=0x7f5d500fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:46.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 -- 172.21.15.81:0/855094144 shutdown_connections 2022-09-10T04:26:46.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.918+0000 7f5d55bff700 1 -- 172.21.15.81:0/855094144 wait complete. 2022-09-10T04:26:46.929 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.920+0000 7f5d55bff700 1 Processor -- start 2022-09-10T04:26:46.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.920+0000 7f5d55bff700 1 -- start start 2022-09-10T04:26:46.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d55bff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500fdcf0 0x7f5d50193940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:46.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d55bff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d50193e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:46.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d55bff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d500ff900 0x7f5d5019aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:46.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d55bff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5019b4c0 con 0x7f5d500fdcf0 2022-09-10T04:26:46.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d55bff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5019b630 con 0x7f5d500fef70 2022-09-10T04:26:46.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d55bff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5019b7f0 con 0x7f5d500ff900 2022-09-10T04:26:46.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500fdcf0 0x7f5d50193940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:46.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d50193e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:46.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d50193e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:58600/0 (socket says 172.21.15.81:58600) 2022-09-10T04:26:46.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4effd700 1 -- 172.21.15.81:0/1792493985 learned_addr learned my addr 172.21.15.81:0/1792493985 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:46.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4ffff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d500ff900 0x7f5d5019aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:46.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4f7fe700 1 -- 172.21.15.81:0/1792493985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d500ff900 msgr2=0x7f5d5019aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:46.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4f7fe700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d500ff900 0x7f5d5019aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:46.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4f7fe700 1 -- 172.21.15.81:0/1792493985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 msgr2=0x7f5d50193e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:46.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4f7fe700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d50193e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:46.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4f7fe700 1 -- 172.21.15.81:0/1792493985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480097e0 con 0x7f5d500fdcf0 2022-09-10T04:26:46.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.921+0000 7f5d4effd700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d50193e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:46.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.922+0000 7f5d4f7fe700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500fdcf0 0x7f5d50193940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005fd0 tx=0x7f5d4000e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:46.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.922+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40004220 con 0x7f5d500fdcf0 2022-09-10T04:26:46.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.922+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5d40004380 con 0x7f5d500fdcf0 2022-09-10T04:26:46.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.922+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5019bad0 con 0x7f5d500fdcf0 2022-09-10T04:26:46.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.922+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40003b60 con 0x7f5d500fdcf0 2022-09-10T04:26:46.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.922+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5019c030 con 0x7f5d500fdcf0 2022-09-10T04:26:46.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.923+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d501094b0 con 0x7f5d500fdcf0 2022-09-10T04:26:46.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.925+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f5d4001e020 con 0x7f5d500fdcf0 2022-09-10T04:26:46.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.927+0000 7f5d4cff9700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5d38099090 0x7f5d3809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:46.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.927+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f5d40014070 con 0x7f5d500fdcf0 2022-09-10T04:26:46.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.927+0000 7f5d4effd700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5d38099090 0x7f5d3809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:46.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.928+0000 7f5d4effd700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5d38099090 0x7f5d3809b540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d440057d0 tx=0x7f5d44005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:46.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:46.932+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f5d4008eb20 con 0x7f5d500fdcf0 2022-09-10T04:26:47.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.125+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d50104bd0 con 0x7f5d500fdcf0 2022-09-10T04:26:47.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.126+0000 7f5d4cff9700 1 -- 172.21.15.81:0/1792493985 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7f5d4000eea0 con 0x7f5d500fdcf0 2022-09-10T04:26:47.127 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:47.128 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:47.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5d38099090 msgr2=0x7f5d3809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5d38099090 0x7f5d3809b540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d440057d0 tx=0x7f5d44005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500fdcf0 msgr2=0x7f5d50193940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500fdcf0 0x7f5d50193940 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005fd0 tx=0x7f5d4000e9a0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 shutdown_connections 2022-09-10T04:26:47.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5d38099090 0x7f5d3809b540 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d500ff900 0x7f5d5019aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d500fdcf0 0x7f5d50193940 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 --2- 172.21.15.81:0/1792493985 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5d500fef70 0x7f5d50193e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 >> 172.21.15.81:0/1792493985 conn(0x7f5d500f93c0 msgr2=0x7f5d500fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:47.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 shutdown_connections 2022-09-10T04:26:47.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.129+0000 7f5d55bff700 1 -- 172.21.15.81:0/1792493985 wait complete. 2022-09-10T04:26:47.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-10T04:26:30.005808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-10T04:26:31.529234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-10T04:26:35.269129+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-10T04:26:47.158 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:26:47.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 -- 172.21.15.81:0/2578318999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200ff910 msgr2=0x7fa4201061d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 --2- 172.21.15.81:0/2578318999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200ff910 0x7fa4201061d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa418009b00 tx=0x7fa418009e10 comp rx=0 tx=0).stop 2022-09-10T04:26:47.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 -- 172.21.15.81:0/2578318999 shutdown_connections 2022-09-10T04:26:47.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 --2- 172.21.15.81:0/2578318999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200ff910 0x7fa4201061d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 --2- 172.21.15.81:0/2578318999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4200fdd00 0x7fa4200fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 --2- 172.21.15.81:0/2578318999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4200fef80 0x7fa4200ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.389+0000 7fa4263ff700 1 -- 172.21.15.81:0/2578318999 >> 172.21.15.81:0/2578318999 conn(0x7fa4200f93f0 msgr2=0x7fa4200fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:47.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.390+0000 7fa4263ff700 1 -- 172.21.15.81:0/2578318999 shutdown_connections 2022-09-10T04:26:47.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.390+0000 7fa4263ff700 1 -- 172.21.15.81:0/2578318999 wait complete. 2022-09-10T04:26:47.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 Processor -- start 2022-09-10T04:26:47.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 -- start start 2022-09-10T04:26:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4200fdd00 0x7fa420195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 0x7fa420196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4200ff910 0x7fa42019d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42019d6c0 con 0x7fa4200ff910 2022-09-10T04:26:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42019d830 con 0x7fa4200fdd00 2022-09-10T04:26:47.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.391+0000 7fa4263ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42019d9f0 con 0x7fa4200fef80 2022-09-10T04:26:47.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 0x7fa420196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 0x7fa420196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:33618/0 (socket says 172.21.15.81:33618) 2022-09-10T04:26:47.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 -- 172.21.15.81:0/1346237572 learned_addr learned my addr 172.21.15.81:0/1346237572 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:47.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa42499c700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4200ff910 0x7fa42019d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41ffff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4200fdd00 0x7fa420195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 -- 172.21.15.81:0/1346237572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4200fdd00 msgr2=0x7fa420195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4200fdd00 0x7fa420195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.416 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 -- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4200ff910 msgr2=0x7fa42019d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4200ff910 0x7fa42019d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 -- 172.21.15.81:0/1346237572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa418009730 con 0x7fa4200fef80 2022-09-10T04:26:47.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.392+0000 7fa41f7fe700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 0x7fa420196080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa41400e740 tx=0x7fa41400ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.393+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414005220 con 0x7fa4200fef80 2022-09-10T04:26:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.393+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42019dcd0 con 0x7fa4200fef80 2022-09-10T04:26:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.393+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa414005380 con 0x7fa4200fef80 2022-09-10T04:26:47.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.393+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa414010ba0 con 0x7fa4200fef80 2022-09-10T04:26:47.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.393+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42019e1d0 con 0x7fa4200fef80 2022-09-10T04:26:47.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.394+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4201094c0 con 0x7fa4200fef80 2022-09-10T04:26:47.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.396+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fa41401e020 con 0x7fa4200fef80 2022-09-10T04:26:47.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.397+0000 7fa41d7fa700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa408099090 0x7fa40809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.397+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fa414014070 con 0x7fa4200fef80 2022-09-10T04:26:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.398+0000 7fa41ffff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa408099090 0x7fa40809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.421 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.398+0000 7fa41ffff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa408099090 0x7fa40809b540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa4200fede0 tx=0x7fa410005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:47.422 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.399+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa41408f2d0 con 0x7fa4200fef80 2022-09-10T04:26:47.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.596+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa42019e840 con 0x7fa4200fef80 2022-09-10T04:26:47.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.597+0000 7fa41d7fa700 1 -- 172.21.15.81:0/1346237572 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4140928f0 con 0x7fa4200fef80 2022-09-10T04:26:47.599 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:26:47.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa408099090 msgr2=0x7fa40809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa408099090 0x7fa40809b540 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa4200fede0 tx=0x7fa410005f60 comp rx=0 tx=0).stop 2022-09-10T04:26:47.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 msgr2=0x7fa420196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 0x7fa420196080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa41400e740 tx=0x7fa41400ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:47.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 shutdown_connections 2022-09-10T04:26:47.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4200ff910 0x7fa42019d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa4200fdd00 0x7fa420195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa408099090 0x7fa40809b540 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 --2- 172.21.15.81:0/1346237572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4200fef80 0x7fa420196080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 >> 172.21.15.81:0/1346237572 conn(0x7fa4200f93f0 msgr2=0x7fa4200fb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:47.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 shutdown_connections 2022-09-10T04:26:47.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.599+0000 7fa4263ff700 1 -- 172.21.15.81:0/1346237572 wait complete. 2022-09-10T04:26:47.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-10T04:26:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:47 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1792493985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:47.705 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:47 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1792493985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:47 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1792493985' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:47.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 -- 172.21.15.81:0/1735456846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280fe490 msgr2=0x7f7e280fee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/1735456846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280fe490 0x7f7e280fee80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009810 tx=0x7f7e14009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:47.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 -- 172.21.15.81:0/1735456846 shutdown_connections 2022-09-10T04:26:47.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/1735456846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280ff510 0x7f7e281060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/1735456846 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fdb40 0x7f7e280fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/1735456846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280fe490 0x7f7e280fee80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 -- 172.21.15.81:0/1735456846 >> 172.21.15.81:0/1735456846 conn(0x7f7e280f9230 msgr2=0x7f7e280fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:47.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 -- 172.21.15.81:0/1735456846 shutdown_connections 2022-09-10T04:26:47.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.856+0000 7f7e2d99d700 1 -- 172.21.15.81:0/1735456846 wait complete. 2022-09-10T04:26:47.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.858+0000 7f7e2d99d700 1 Processor -- start 2022-09-10T04:26:47.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.858+0000 7f7e2d99d700 1 -- start start 2022-09-10T04:26:47.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d99d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280fdb40 0x7f7e28072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d99d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fe490 0x7f7e280710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d99d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280ff510 0x7f7e28071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d99d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28071e10 con 0x7f7e280ff510 2022-09-10T04:26:47.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d99d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e28071f80 con 0x7f7e280fe490 2022-09-10T04:26:47.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d99d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2819d820 con 0x7f7e280fdb40 2022-09-10T04:26:47.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d19c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280ff510 0x7f7e28071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2c99b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280fdb40 0x7f7e28072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d19c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280ff510 0x7f7e28071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40474/0 (socket says 172.21.15.81:40474) 2022-09-10T04:26:47.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e27fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fe490 0x7f7e280710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2d19c700 1 -- 172.21.15.81:0/2217738744 learned_addr learned my addr 172.21.15.81:0/2217738744 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:47.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2c99b700 1 -- 172.21.15.81:0/2217738744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fe490 msgr2=0x7f7e280710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2c99b700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fe490 0x7f7e280710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2c99b700 1 -- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280ff510 msgr2=0x7f7e28071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:47.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2c99b700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280ff510 0x7f7e28071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:47.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e2c99b700 1 -- 172.21.15.81:0/2217738744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e14009470 con 0x7f7e280fdb40 2022-09-10T04:26:47.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.859+0000 7f7e27fff700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fe490 0x7f7e280710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:47.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.860+0000 7f7e2c99b700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280fdb40 0x7f7e28072a90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00e6c0 tx=0x7f7e1c00e9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:47.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.860+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c010900 con 0x7f7e280fdb40 2022-09-10T04:26:47.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.860+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2819db00 con 0x7f7e280fdb40 2022-09-10T04:26:47.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.860+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7e1c018420 con 0x7f7e280fdb40 2022-09-10T04:26:47.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.860+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c017610 con 0x7f7e280fdb40 2022-09-10T04:26:47.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.860+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2819e010 con 0x7f7e280fdb40 2022-09-10T04:26:47.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.861+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e28109300 con 0x7f7e280fdb40 2022-09-10T04:26:47.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.863+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f7e1c00c040 con 0x7f7e280fdb40 2022-09-10T04:26:47.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.864+0000 7f7e25ffb700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7e18099010 0x7f7e1809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:47.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.865+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f7e1c014070 con 0x7f7e280fdb40 2022-09-10T04:26:47.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.865+0000 7f7e27fff700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7e18099010 0x7f7e1809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:47.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.865+0000 7f7e27fff700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7e18099010 0x7f7e1809b4c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400b4f0 tx=0x7f7e14009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:47.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:47.867+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f7e1c08ead0 con 0x7f7e280fdb40 2022-09-10T04:26:48.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.062+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e28104aa0 con 0x7f7e280fdb40 2022-09-10T04:26:48.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.062+0000 7f7e25ffb700 1 -- 172.21.15.81:0/2217738744 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e1c0970a0 con 0x7f7e280fdb40 2022-09-10T04:26:48.065 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:26:48.066 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7e18099010 msgr2=0x7f7e1809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.067 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7e18099010 0x7f7e1809b4c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400b4f0 tx=0x7f7e14009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.067 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280fdb40 msgr2=0x7f7e28072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.067 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280fdb40 0x7f7e28072a90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00e6c0 tx=0x7f7e1c00e9d0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.067 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 shutdown_connections 2022-09-10T04:26:48.068 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e280ff510 0x7f7e28071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7e18099010 0x7f7e1809b4c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e280fdb40 0x7f7e28072a90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 --2- 172.21.15.81:0/2217738744 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7e280fe490 0x7f7e280710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 >> 172.21.15.81:0/2217738744 conn(0x7f7e280f9230 msgr2=0x7f7e280fb540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:48.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 shutdown_connections 2022-09-10T04:26:48.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.065+0000 7f7e2d99d700 1 -- 172.21.15.81:0/2217738744 wait complete. 2022-09-10T04:26:48.093 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-10T04:26:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1146314742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40fdb40 msgr2=0x7fd2a40fdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.337 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1146314742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40fdb40 0x7fd2a40fdf50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd294009810 tx=0x7fd294009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1146314742 shutdown_connections 2022-09-10T04:26:48.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1146314742 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40ff510 0x7fd2a41060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1146314742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40fdb40 0x7fd2a40fdf50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1146314742 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2a40fe490 0x7fd2a40fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1146314742 >> 172.21.15.81:0/1146314742 conn(0x7fd2a40f9230 msgr2=0x7fd2a40fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1146314742 shutdown_connections 2022-09-10T04:26:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.333+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1146314742 wait complete. 2022-09-10T04:26:48.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 Processor -- start 2022-09-10T04:26:48.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 -- start start 2022-09-10T04:26:48.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40fdb40 0x7fd2a4193750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2a40fe490 0x7fd2a4193c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 0x7fd2a419ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a419b2d0 con 0x7fd2a40ff510 2022-09-10T04:26:48.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a419b440 con 0x7fd2a40fdb40 2022-09-10T04:26:48.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.334+0000 7fd2a3fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a419b600 con 0x7fd2a40fe490 2022-09-10T04:26:48.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 0x7fd2a419ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a27fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2a40fe490 0x7fd2a4193c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 0x7fd2a419ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40504/0 (socket says 172.21.15.81:40504) 2022-09-10T04:26:48.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 -- 172.21.15.81:0/1501113450 learned_addr learned my addr 172.21.15.81:0/1501113450 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:48.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a2ffd700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40fdb40 0x7fd2a4193750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 -- 172.21.15.81:0/1501113450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2a40fe490 msgr2=0x7fd2a4193c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2a40fe490 0x7fd2a4193c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 -- 172.21.15.81:0/1501113450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40fdb40 msgr2=0x7fd2a4193750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40fdb40 0x7fd2a4193750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a37fe700 1 -- 172.21.15.81:0/1501113450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009470 con 0x7fd2a40ff510 2022-09-10T04:26:48.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.335+0000 7fd2a2ffd700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40fdb40 0x7fd2a4193750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:48.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.336+0000 7fd2a37fe700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 0x7fd2a419ad00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00ebf0 tx=0x7fd29c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:48.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.336+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0099c0 con 0x7fd2a40ff510 2022-09-10T04:26:48.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.336+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fd29c00cc90 con 0x7fd2a40ff510 2022-09-10T04:26:48.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.336+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a419b8e0 con 0x7fd2a40ff510 2022-09-10T04:26:48.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.336+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c010430 con 0x7fd2a40ff510 2022-09-10T04:26:48.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.336+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a419bdf0 con 0x7fd2a40ff510 2022-09-10T04:26:48.353 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.337+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a4109350 con 0x7fd2a40ff510 2022-09-10T04:26:48.353 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.339+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fd29c01e020 con 0x7fd2a40ff510 2022-09-10T04:26:48.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.340+0000 7fd287fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fd28c099010 0x7fd28c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.341+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fd29c014070 con 0x7fd2a40ff510 2022-09-10T04:26:48.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.341+0000 7fd2a2ffd700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fd28c099010 0x7fd28c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.341+0000 7fd2a2ffd700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fd28c099010 0x7fd28c09b4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd29400b4f0 tx=0x7fd294009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:48.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.344+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fd29c08ef80 con 0x7fd2a40ff510 2022-09-10T04:26:48.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.538+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd2a405fb00 con 0x7fd2a40ff510 2022-09-10T04:26:48.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.539+0000 7fd287fff700 1 -- 172.21.15.81:0/1501113450 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd29c0925a0 con 0x7fd2a40ff510 2022-09-10T04:26:48.541 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2022-09-10T04:26:48.543 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fd28c099010 msgr2=0x7fd28c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.543 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fd28c099010 0x7fd28c09b4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd29400b4f0 tx=0x7fd294009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.543 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 msgr2=0x7fd2a419ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 0x7fd2a419ad00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00ebf0 tx=0x7fd29c00ef00 comp rx=0 tx=0).stop 2022-09-10T04:26:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 shutdown_connections 2022-09-10T04:26:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2a40ff510 0x7fd2a419ad00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2a40fdb40 0x7fd2a4193750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fd28c099010 0x7fd28c09b4c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 --2- 172.21.15.81:0/1501113450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2a40fe490 0x7fd2a4193c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 >> 172.21.15.81:0/1501113450 conn(0x7fd2a40f9230 msgr2=0x7fd2a40fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:48.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 shutdown_connections 2022-09-10T04:26:48.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.541+0000 7fd2a3fff700 1 -- 172.21.15.81:0/1501113450 wait complete. 2022-09-10T04:26:48.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-10T04:26:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:48 smithi110 ceph-mon[97633]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:48 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1346237572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:48 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2217738744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:26:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:48 smithi104 ceph-mon[97044]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:48.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:48 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1346237572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:48.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:48 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2217738744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:26:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:48 smithi081 ceph-mon[90133]: pgmap v309: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:48 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1346237572' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:48.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:48 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2217738744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:26:48.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.804+0000 7f0d627ff700 1 -- 172.21.15.81:0/129168903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0fef80 msgr2=0x7f0d5c0ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.804+0000 7f0d627ff700 1 --2- 172.21.15.81:0/129168903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c0ff3d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009810 tx=0x7f0d50009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:48.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 -- 172.21.15.81:0/129168903 shutdown_connections 2022-09-10T04:26:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 --2- 172.21.15.81:0/129168903 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d5c0ff910 0x7f0d5c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 --2- 172.21.15.81:0/129168903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fdd00 0x7f0d5c0fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 --2- 172.21.15.81:0/129168903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c0ff3d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 -- 172.21.15.81:0/129168903 >> 172.21.15.81:0/129168903 conn(0x7f0d5c0f93f0 msgr2=0x7f0d5c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 -- 172.21.15.81:0/129168903 shutdown_connections 2022-09-10T04:26:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.805+0000 7f0d627ff700 1 -- 172.21.15.81:0/129168903 wait complete. 2022-09-10T04:26:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 Processor -- start 2022-09-10T04:26:48.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 -- start start 2022-09-10T04:26:48.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d5c0fdd00 0x7f0d5c195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0ff910 0x7f0d5c19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c19d6c0 con 0x7f0d5c0fef80 2022-09-10T04:26:48.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c19d830 con 0x7f0d5c0fdd00 2022-09-10T04:26:48.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.806+0000 7f0d627ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c19d9f0 con 0x7f0d5c0ff910 2022-09-10T04:26:48.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40526/0 (socket says 172.21.15.81:40526) 2022-09-10T04:26:48.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 -- 172.21.15.81:0/1602076076 learned_addr learned my addr 172.21.15.81:0/1602076076 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:48.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d60d9c700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0ff910 0x7f0d5c19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5bfff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d5c0fdd00 0x7f0d5c195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 -- 172.21.15.81:0/1602076076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0ff910 msgr2=0x7f0d5c19d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0ff910 0x7f0d5c19d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 -- 172.21.15.81:0/1602076076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d5c0fdd00 msgr2=0x7f0d5c195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d5c0fdd00 0x7f0d5c195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:48.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.807+0000 7f0d5b7fe700 1 -- 172.21.15.81:0/1602076076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50009470 con 0x7f0d5c0fef80 2022-09-10T04:26:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.808+0000 7f0d5b7fe700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c196080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0d50000c00 tx=0x7f0d50004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.808+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5001d070 con 0x7f0d5c0fef80 2022-09-10T04:26:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.808+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0d50005880 con 0x7f0d5c0fef80 2022-09-10T04:26:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.808+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c19dc70 con 0x7f0d5c0fef80 2022-09-10T04:26:48.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.808+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50004550 con 0x7f0d5c0fef80 2022-09-10T04:26:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.808+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c19e120 con 0x7f0d5c0fef80 2022-09-10T04:26:48.818 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.809+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c1094b0 con 0x7f0d5c0fef80 2022-09-10T04:26:48.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.811+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f0d50028030 con 0x7f0d5c0fef80 2022-09-10T04:26:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.812+0000 7f0d597fa700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0d44099010 0x7f0d4409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.813+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f0d500c3770 con 0x7f0d5c0fef80 2022-09-10T04:26:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.813+0000 7f0d5bfff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0d44099010 0x7f0d4409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:48.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.813+0000 7f0d5bfff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0d44099010 0x7f0d4409b4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0fede0 tx=0x7f0d4c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:48.821 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:48.815+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f0d5008fc10 con 0x7f0d5c0fef80 2022-09-10T04:26:49.006 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.005+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0d5c19e790 con 0x7f0d5c0fef80 2022-09-10T04:26:49.006 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.005+0000 7f0d597fa700 1 -- 172.21.15.81:0/1602076076 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0d50026020 con 0x7f0d5c0fef80 2022-09-10T04:26:49.007 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:26:49.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0d44099010 msgr2=0x7f0d4409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.011 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0d44099010 0x7f0d4409b4c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0fede0 tx=0x7f0d4c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.012 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 msgr2=0x7f0d5c196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.012 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c196080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0d50000c00 tx=0x7f0d50004230 comp rx=0 tx=0).stop 2022-09-10T04:26:49.013 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 shutdown_connections 2022-09-10T04:26:49.013 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0d44099010 0x7f0d4409b4c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.013 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d5c0ff910 0x7f0d5c19d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.014 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0d5c0fdd00 0x7f0d5c195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.014 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 --2- 172.21.15.81:0/1602076076 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d5c0fef80 0x7f0d5c196080 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.015 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 >> 172.21.15.81:0/1602076076 conn(0x7f0d5c0f93f0 msgr2=0x7f0d5c0fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:49.015 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 shutdown_connections 2022-09-10T04:26:49.016 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.008+0000 7f0d627ff700 1 -- 172.21.15.81:0/1602076076 wait complete. 2022-09-10T04:26:49.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2022-09-10T04:26:49.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.300+0000 7faf55f9d700 1 -- 172.21.15.81:0/697533401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 msgr2=0x7faf50105c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.300+0000 7faf55f9d700 1 --2- 172.21.15.81:0/697533401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 0x7faf50105c60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faf48009b00 tx=0x7faf48009e10 comp rx=0 tx=0).stop 2022-09-10T04:26:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.300+0000 7faf55f9d700 1 -- 172.21.15.81:0/697533401 shutdown_connections 2022-09-10T04:26:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.300+0000 7faf55f9d700 1 --2- 172.21.15.81:0/697533401 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 0x7faf50105c60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.300+0000 7faf55f9d700 1 --2- 172.21.15.81:0/697533401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf500fe640 0x7faf50100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.300+0000 7faf55f9d700 1 --2- 172.21.15.81:0/697533401 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf50100f60 0x7faf50103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.301+0000 7faf55f9d700 1 -- 172.21.15.81:0/697533401 >> 172.21.15.81:0/697533401 conn(0x7faf500f8160 msgr2=0x7faf500fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:49.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.301+0000 7faf55f9d700 1 -- 172.21.15.81:0/697533401 shutdown_connections 2022-09-10T04:26:49.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.301+0000 7faf55f9d700 1 -- 172.21.15.81:0/697533401 wait complete. 2022-09-10T04:26:49.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 Processor -- start 2022-09-10T04:26:49.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 -- start start 2022-09-10T04:26:49.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 0x7faf501958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf50100f60 0x7faf50195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 0x7faf5019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5019d450 con 0x7faf50100f60 2022-09-10T04:26:49.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5019d5c0 con 0x7faf50103880 2022-09-10T04:26:49.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.302+0000 7faf55f9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5019d780 con 0x7faf500fe640 2022-09-10T04:26:49.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 0x7faf501958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 0x7faf501958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:33702/0 (socket says 172.21.15.81:33702) 2022-09-10T04:26:49.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf4ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf50100f60 0x7faf50195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 -- 172.21.15.81:0/915634852 learned_addr learned my addr 172.21.15.81:0/915634852 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:49.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf5579c700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 0x7faf5019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 -- 172.21.15.81:0/915634852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 msgr2=0x7faf5019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 0x7faf5019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 -- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf50100f60 msgr2=0x7faf50195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf50100f60 0x7faf50195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 -- 172.21.15.81:0/915634852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48009730 con 0x7faf500fe640 2022-09-10T04:26:49.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.303+0000 7faf54f9b700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 0x7faf501958d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faf4400e740 tx=0x7faf4400ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:49.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.304+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44005220 con 0x7faf500fe640 2022-09-10T04:26:49.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.304+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7faf44005380 con 0x7faf500fe640 2022-09-10T04:26:49.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.304+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5019da60 con 0x7faf500fe640 2022-09-10T04:26:49.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.304+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44004c10 con 0x7faf500fe640 2022-09-10T04:26:49.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.304+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5019df70 con 0x7faf500fe640 2022-09-10T04:26:49.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.305+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf501092c0 con 0x7faf500fe640 2022-09-10T04:26:49.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.307+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7faf4401e020 con 0x7faf500fe640 2022-09-10T04:26:49.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.308+0000 7faf4dffb700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf40099010 0x7faf4009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.308+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7faf44014070 con 0x7faf500fe640 2022-09-10T04:26:49.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.308+0000 7faf4ffff700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf40099010 0x7faf4009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.309+0000 7faf4ffff700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf40099010 0x7faf4009b4c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faf3c005fd0 tx=0x7faf3c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:49.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.312+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7faf4408ed80 con 0x7faf500fe640 2022-09-10T04:26:49.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.526+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faf5019e860 con 0x7faf500fe640 2022-09-10T04:26:49.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.527+0000 7faf4dffb700 1 -- 172.21.15.81:0/915634852 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+591 (secure 0 0 0) 0x7faf440923a0 con 0x7faf500fe640 2022-09-10T04:26:49.528 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2927238869 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:49.529 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/3943821768 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:49.529 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2359268015 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:49.529 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/2567224701 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:49.529 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/2567224701 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:49.530 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2774479929 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:49.530 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/1348172231 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:49.530 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/596170975 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:49.531 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/3224857868 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:49.531 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/3224857868 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:49.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf40099010 msgr2=0x7faf4009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf40099010 0x7faf4009b4c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7faf3c005fd0 tx=0x7faf3c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 msgr2=0x7faf501958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 0x7faf501958d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faf4400e740 tx=0x7faf4400ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:49.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 shutdown_connections 2022-09-10T04:26:49.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf50103880 0x7faf5019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7faf40099010 0x7faf4009b4c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf500fe640 0x7faf501958d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 --2- 172.21.15.81:0/915634852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf50100f60 0x7faf50195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 >> 172.21.15.81:0/915634852 conn(0x7faf500f8160 msgr2=0x7faf500ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:49.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.530+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 shutdown_connections 2022-09-10T04:26:49.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.531+0000 7faf55f9d700 1 -- 172.21.15.81:0/915634852 wait complete. 2022-09-10T04:26:49.537 INFO:teuthology.orchestra.run.smithi081.stderr:listed 10 entries 2022-09-10T04:26:49.561 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:26:49.561 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2022-09-10T04:26:49.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:49.697 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:49 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1501113450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:26:49.698 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:49 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1602076076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:26:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:49 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1501113450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:26:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:49 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1602076076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:26:49.851 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 -- 172.21.15.81:0/2651779285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2544066bf0 msgr2=0x7f2544100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.851 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 --2- 172.21.15.81:0/2651779285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2544066bf0 0x7f2544100890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2534009750 tx=0x7f2534009a60 comp rx=0 tx=0).stop 2022-09-10T04:26:49.851 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 -- 172.21.15.81:0/2651779285 shutdown_connections 2022-09-10T04:26:49.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 --2- 172.21.15.81:0/2651779285 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25441038c0 0x7f2544105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 --2- 172.21.15.81:0/2651779285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2544066bf0 0x7f2544100890 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 --2- 172.21.15.81:0/2651779285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f2544103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.846+0000 7f25489ff700 1 -- 172.21.15.81:0/2651779285 >> 172.21.15.81:0/2651779285 conn(0x7f25440f82a0 msgr2=0x7f25440fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:49.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.847+0000 7f25489ff700 1 -- 172.21.15.81:0/2651779285 shutdown_connections 2022-09-10T04:26:49.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.847+0000 7f25489ff700 1 -- 172.21.15.81:0/2651779285 wait complete. 2022-09-10T04:26:49.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 Processor -- start 2022-09-10T04:26:49.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 -- start start 2022-09-10T04:26:49.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2544066bf0 0x7f25440fe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f25440fc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25441038c0 0x7f25440fcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25440fd570 con 0x7f2544100dd0 2022-09-10T04:26:49.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25440fd6e0 con 0x7f2544066bf0 2022-09-10T04:26:49.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.848+0000 7f25489ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254419da10 con 0x7f25441038c0 2022-09-10T04:26:49.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f25440fc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f25440fc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42422/0 (socket says 172.21.15.81:42422) 2022-09-10T04:26:49.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 -- 172.21.15.81:0/3743713803 learned_addr learned my addr 172.21.15.81:0/3743713803 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:49.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f254259c700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2544066bf0 0x7f25440fe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2542d9d700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25441038c0 0x7f25440fcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 -- 172.21.15.81:0/3743713803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25441038c0 msgr2=0x7f25440fcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25441038c0 0x7f25440fcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 -- 172.21.15.81:0/3743713803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2544066bf0 msgr2=0x7f25440fe1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:49.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2544066bf0 0x7f25440fe1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:49.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f2541d9b700 1 -- 172.21.15.81:0/3743713803 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25340093b0 con 0x7f2544100dd0 2022-09-10T04:26:49.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.849+0000 7f254259c700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2544066bf0 0x7f25440fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:49.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.850+0000 7f2541d9b700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f25440fc830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f253800e740 tx=0x7f253800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:49.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.850+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538005220 con 0x7f2544100dd0 2022-09-10T04:26:49.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.850+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254419dcf0 con 0x7f2544100dd0 2022-09-10T04:26:49.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.850+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2538005380 con 0x7f2544100dd0 2022-09-10T04:26:49.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.850+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538004c10 con 0x7f2544100dd0 2022-09-10T04:26:49.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.850+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254419e1f0 con 0x7f2544100dd0 2022-09-10T04:26:49.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.851+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2544109500 con 0x7f2544100dd0 2022-09-10T04:26:49.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.853+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f253801e020 con 0x7f2544100dd0 2022-09-10T04:26:49.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.854+0000 7f25337fe700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f252c099090 0x7f252c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:49.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.854+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f2538014070 con 0x7f2544100dd0 2022-09-10T04:26:49.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.854+0000 7f254259c700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f252c099090 0x7f252c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:49.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.855+0000 7f254259c700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f252c099090 0x7f252c09b540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2534009e30 tx=0x7f2534005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:49.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:49.858+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f253808ece0 con 0x7f2544100dd0 2022-09-10T04:26:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:49 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1501113450' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:26:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:49 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1602076076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:26:50.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.051+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25440f9c90 con 0x7f2544100dd0 2022-09-10T04:26:50.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.052+0000 7f25337fe700 1 -- 172.21.15.81:0/3743713803 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7f2538092300 con 0x7f2544100dd0 2022-09-10T04:26:50.054 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:50.055 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:50.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f252c099090 msgr2=0x7f252c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f252c099090 0x7f252c09b540 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2534009e30 tx=0x7f2534005f90 comp rx=0 tx=0).stop 2022-09-10T04:26:50.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 msgr2=0x7f25440fc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f25440fc830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f253800e740 tx=0x7f253800ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:50.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 shutdown_connections 2022-09-10T04:26:50.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f252c099090 0x7f252c09b540 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25441038c0 0x7f25440fcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2544066bf0 0x7f25440fe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 --2- 172.21.15.81:0/3743713803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2544100dd0 0x7f25440fc830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 >> 172.21.15.81:0/3743713803 conn(0x7f25440f82a0 msgr2=0x7f25440ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:50.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 shutdown_connections 2022-09-10T04:26:50.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.055+0000 7f25489ff700 1 -- 172.21.15.81:0/3743713803 wait complete. 2022-09-10T04:26:50.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-10T04:26:30.005808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-10T04:26:31.529234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-10T04:26:35.269129+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-10T04:26:50.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:26:50.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 -- 172.21.15.81:0/2505004326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 msgr2=0x7f65dc0fdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/2505004326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc0fdf50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009810 tx=0x7f65d0009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 -- 172.21.15.81:0/2505004326 shutdown_connections 2022-09-10T04:26:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/2505004326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65dc0ff510 0x7f65dc1060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/2505004326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc0fdf50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/2505004326 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65dc0fe490 0x7f65dc0fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.312+0000 7f65e1d9d700 1 -- 172.21.15.81:0/2505004326 >> 172.21.15.81:0/2505004326 conn(0x7f65dc0f9230 msgr2=0x7f65dc0fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:50.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.313+0000 7f65e1d9d700 1 -- 172.21.15.81:0/2505004326 shutdown_connections 2022-09-10T04:26:50.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.313+0000 7f65e1d9d700 1 -- 172.21.15.81:0/2505004326 wait complete. 2022-09-10T04:26:50.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 Processor -- start 2022-09-10T04:26:50.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 -- start start 2022-09-10T04:26:50.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65dc0fe490 0x7f65dc195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65dc0ff510 0x7f65dc19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc19d4c0 con 0x7f65dc0fe490 2022-09-10T04:26:50.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc19d630 con 0x7f65dc0ff510 2022-09-10T04:26:50.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.314+0000 7f65e1d9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65dc19d7f0 con 0x7f65dc0fdb40 2022-09-10T04:26:50.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65dbfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65dc0fe490 0x7f65dc195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51754/0 (socket says 172.21.15.81:51754) 2022-09-10T04:26:50.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 -- 172.21.15.81:0/1198177337 learned_addr learned my addr 172.21.15.81:0/1198177337 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:50.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e159c700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65dc0ff510 0x7f65dc19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 -- 172.21.15.81:0/1198177337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65dc0ff510 msgr2=0x7f65dc19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65dc0ff510 0x7f65dc19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 -- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65dc0fe490 msgr2=0x7f65dc195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65dc0fe490 0x7f65dc195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 -- 172.21.15.81:0/1198177337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0009470 con 0x7f65dc0fdb40 2022-09-10T04:26:50.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.315+0000 7f65e0d9b700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc195940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65d000f690 tx=0x7f65d000f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:50.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.316+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d001d070 con 0x7f65dc0fdb40 2022-09-10T04:26:50.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.316+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f65d00054f0 con 0x7f65dc0fdb40 2022-09-10T04:26:50.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.316+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d0004860 con 0x7f65dc0fdb40 2022-09-10T04:26:50.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.316+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65dc19da70 con 0x7f65dc0fdb40 2022-09-10T04:26:50.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.316+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65dc19df20 con 0x7f65dc0fdb40 2022-09-10T04:26:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.317+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65dc109330 con 0x7f65dc0fdb40 2022-09-10T04:26:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.319+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f65d0028030 con 0x7f65dc0fdb40 2022-09-10T04:26:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.320+0000 7f65d9ffb700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f65cc099090 0x7f65cc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.320+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f65d00c3270 con 0x7f65dc0fdb40 2022-09-10T04:26:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.320+0000 7f65dbfff700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f65cc099090 0x7f65cc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.321+0000 7f65dbfff700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f65cc099090 0x7f65cc09b540 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0fe860 tx=0x7f65c8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:50.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.322+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f65d008f710 con 0x7f65dc0fdb40 2022-09-10T04:26:50.517 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.517+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65dc19e7d0 con 0x7f65dc0fdb40 2022-09-10T04:26:50.518 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.517+0000 7f65d9ffb700 1 -- 172.21.15.81:0/1198177337 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65d0026020 con 0x7f65dc0fdb40 2022-09-10T04:26:50.518 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:26:50.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f65cc099090 msgr2=0x7f65cc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f65cc099090 0x7f65cc09b540 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f65dc0fe860 tx=0x7f65c8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 msgr2=0x7f65dc195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc195940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f65d000f690 tx=0x7f65d000f6c0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 shutdown_connections 2022-09-10T04:26:50.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f65dc0ff510 0x7f65dc19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f65cc099090 0x7f65cc09b540 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65dc0fdb40 0x7f65dc195940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 --2- 172.21.15.81:0/1198177337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65dc0fe490 0x7f65dc195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 >> 172.21.15.81:0/1198177337 conn(0x7f65dc0f9230 msgr2=0x7f65dc0fb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:50.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 shutdown_connections 2022-09-10T04:26:50.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.520+0000 7f65e1d9d700 1 -- 172.21.15.81:0/1198177337 wait complete. 2022-09-10T04:26:50.548 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-10T04:26:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 -- 172.21.15.81:0/607996625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200ff510 msgr2=0x7f98201060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 --2- 172.21.15.81:0/607996625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200ff510 0x7f98201060b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9818009bb0 tx=0x7f9818009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 -- 172.21.15.81:0/607996625 shutdown_connections 2022-09-10T04:26:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 --2- 172.21.15.81:0/607996625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200ff510 0x7f98201060b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 --2- 172.21.15.81:0/607996625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98200fdb40 0x7f98200fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 --2- 172.21.15.81:0/607996625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fe490 0x7f98200fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 -- 172.21.15.81:0/607996625 >> 172.21.15.81:0/607996625 conn(0x7f98200f9250 msgr2=0x7f98200fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:50.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 -- 172.21.15.81:0/607996625 shutdown_connections 2022-09-10T04:26:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.778+0000 7f982699d700 1 -- 172.21.15.81:0/607996625 wait complete. 2022-09-10T04:26:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.779+0000 7f982699d700 1 Processor -- start 2022-09-10T04:26:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 -- start start 2022-09-10T04:26:50.784 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fdb40 0x7f9820195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.784 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 0x7f9820195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98200ff510 0x7f982019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d4a0 con 0x7f98200fe490 2022-09-10T04:26:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d610 con 0x7f98200ff510 2022-09-10T04:26:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982699d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d7d0 con 0x7f98200fdb40 2022-09-10T04:26:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982519a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 0x7f9820195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982599b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fdb40 0x7f9820195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982599b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fdb40 0x7f9820195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51776/0 (socket says 172.21.15.81:51776) 2022-09-10T04:26:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982519a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 0x7f9820195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42460/0 (socket says 172.21.15.81:42460) 2022-09-10T04:26:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982599b700 1 -- 172.21.15.81:0/1274180296 learned_addr learned my addr 172.21.15.81:0/1274180296 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.780+0000 7f982619c700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98200ff510 0x7f982019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982519a700 1 -- 172.21.15.81:0/1274180296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fdb40 msgr2=0x7f9820195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982519a700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fdb40 0x7f9820195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982519a700 1 -- 172.21.15.81:0/1274180296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98200ff510 msgr2=0x7f982019ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982519a700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98200ff510 0x7f982019ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982519a700 1 -- 172.21.15.81:0/1274180296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0093b0 con 0x7f98200fe490 2022-09-10T04:26:50.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982519a700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 0x7f9820195e60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9810005fd0 tx=0x7f981000e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:50.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810010900 con 0x7f98200fe490 2022-09-10T04:26:50.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9810018420 con 0x7f98200fe490 2022-09-10T04:26:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98180097e0 con 0x7f98200fe490 2022-09-10T04:26:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.781+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810017610 con 0x7f98200fe490 2022-09-10T04:26:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.782+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982019de80 con 0x7f98200fe490 2022-09-10T04:26:50.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.783+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820109330 con 0x7f98200fe490 2022-09-10T04:26:50.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.785+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f981000c040 con 0x7f98200fe490 2022-09-10T04:26:50.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.786+0000 7f9816ffd700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f980c099090 0x7f980c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:50.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.786+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f9810014070 con 0x7f98200fe490 2022-09-10T04:26:50.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.786+0000 7f982599b700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f980c099090 0x7f980c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:50.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.787+0000 7f982599b700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f980c099090 0x7f980c09b540 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f981c009d40 tx=0x7f981c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:50.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.789+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f981004fc80 con 0x7f98200fe490 2022-09-10T04:26:50.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:50 smithi104 ceph-mon[97044]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:50 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/915634852' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-10T04:26:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:50 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3743713803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:50.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:50 smithi081 ceph-mon[90133]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:50.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:50 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/915634852' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-10T04:26:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:50 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3743713803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:50 smithi110 ceph-mon[97633]: pgmap v310: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2022-09-10T04:26:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:50 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/915634852' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-10T04:26:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:50 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3743713803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:50.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.982+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f982018ff40 con 0x7f98200fe490 2022-09-10T04:26:50.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.982+0000 7f9816ffd700 1 -- 172.21.15.81:0/1274180296 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f981008ea50 con 0x7f98200fe490 2022-09-10T04:26:50.984 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:26:50.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f980c099090 msgr2=0x7f980c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f980c099090 0x7f980c09b540 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f981c009d40 tx=0x7f981c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 msgr2=0x7f9820195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 0x7f9820195e60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9810005fd0 tx=0x7f981000e9a0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 shutdown_connections 2022-09-10T04:26:50.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98200ff510 0x7f982019ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f980c099090 0x7f980c09b540 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98200fdb40 0x7f9820195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 --2- 172.21.15.81:0/1274180296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98200fe490 0x7f9820195e60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:50.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 >> 172.21.15.81:0/1274180296 conn(0x7f98200f9250 msgr2=0x7f9820104770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:50.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 shutdown_connections 2022-09-10T04:26:50.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:50.985+0000 7f982699d700 1 -- 172.21.15.81:0/1274180296 wait complete. 2022-09-10T04:26:51.012 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-10T04:26:51.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.240+0000 7f256699d700 1 -- 172.21.15.81:0/455762118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560072bf0 msgr2=0x7f2560071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.240+0000 7f256699d700 1 --2- 172.21.15.81:0/455762118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560072bf0 0x7f2560071170 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f255c009810 tx=0x7f255c009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:51.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 -- 172.21.15.81:0/455762118 shutdown_connections 2022-09-10T04:26:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 --2- 172.21.15.81:0/455762118 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2560105a50 0x7f2560107e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 --2- 172.21.15.81:0/455762118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560072bf0 0x7f2560071170 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 --2- 172.21.15.81:0/455762118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 -- 172.21.15.81:0/455762118 >> 172.21.15.81:0/455762118 conn(0x7f25600f9230 msgr2=0x7f25600fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 -- 172.21.15.81:0/455762118 shutdown_connections 2022-09-10T04:26:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.241+0000 7f256699d700 1 -- 172.21.15.81:0/455762118 wait complete. 2022-09-10T04:26:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.242+0000 7f256699d700 1 Processor -- start 2022-09-10T04:26:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 -- start start 2022-09-10T04:26:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2560072bf0 0x7f2560195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560105a50 0x7f256019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256019d4c0 con 0x7f2560105a50 2022-09-10T04:26:51.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256019d630 con 0x7f2560072bf0 2022-09-10T04:26:51.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.243+0000 7f256699d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256019d7f0 con 0x7f25600716b0 2022-09-10T04:26:51.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256619c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560105a50 0x7f256019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51796/0 (socket says 172.21.15.81:51796) 2022-09-10T04:26:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 -- 172.21.15.81:0/979177763 learned_addr learned my addr 172.21.15.81:0/979177763 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256519a700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2560072bf0 0x7f2560195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 -- 172.21.15.81:0/979177763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2560072bf0 msgr2=0x7f2560195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2560072bf0 0x7f2560195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 -- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560105a50 msgr2=0x7f256019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560105a50 0x7f256019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 -- 172.21.15.81:0/979177763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c009470 con 0x7f25600716b0 2022-09-10T04:26:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256619c700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560105a50 0x7f256019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:51.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.244+0000 7f256599b700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560195940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f255c006010 tx=0x7f255c00f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:51.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.245+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c01d070 con 0x7f25600716b0 2022-09-10T04:26:51.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.245+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256019da70 con 0x7f25600716b0 2022-09-10T04:26:51.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.245+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f255c00f8b0 con 0x7f25600716b0 2022-09-10T04:26:51.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.245+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c0047f0 con 0x7f25600716b0 2022-09-10T04:26:51.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.245+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256019df70 con 0x7f25600716b0 2022-09-10T04:26:51.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.246+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256010b500 con 0x7f25600716b0 2022-09-10T04:26:51.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.248+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f255c028030 con 0x7f25600716b0 2022-09-10T04:26:51.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.249+0000 7f2556ffd700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f254c099010 0x7f254c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.249+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f255c0c3670 con 0x7f25600716b0 2022-09-10T04:26:51.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.250+0000 7f256519a700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f254c099010 0x7f254c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.250+0000 7f256519a700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f254c099010 0x7f254c09b4c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2550005fd0 tx=0x7f2550005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.251+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f255c08fb10 con 0x7f25600716b0 2022-09-10T04:26:51.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.446+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f256010b690 con 0x7f25600716b0 2022-09-10T04:26:51.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.447+0000 7f2556ffd700 1 -- 172.21.15.81:0/979177763 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f256010b690 con 0x7f25600716b0 2022-09-10T04:26:51.447 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2022-09-10T04:26:51.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f254c099010 msgr2=0x7f254c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f254c099010 0x7f254c09b4c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2550005fd0 tx=0x7f2550005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 msgr2=0x7f2560195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560195940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f255c006010 tx=0x7f255c00f690 comp rx=0 tx=0).stop 2022-09-10T04:26:51.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 shutdown_connections 2022-09-10T04:26:51.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2560105a50 0x7f256019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f254c099010 0x7f254c09b4c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25600716b0 0x7f2560195940 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 --2- 172.21.15.81:0/979177763 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2560072bf0 0x7f2560195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.449+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 >> 172.21.15.81:0/979177763 conn(0x7f25600f9230 msgr2=0x7f25600fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.450+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 shutdown_connections 2022-09-10T04:26:51.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.450+0000 7f256699d700 1 -- 172.21.15.81:0/979177763 wait complete. 2022-09-10T04:26:51.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-10T04:26:51.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 -- 172.21.15.81:0/2931091490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 msgr2=0x7f9998071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 --2- 172.21.15.81:0/2931091490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f9998071170 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9988009810 tx=0x7f9988009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:51.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 -- 172.21.15.81:0/2931091490 shutdown_connections 2022-09-10T04:26:51.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 --2- 172.21.15.81:0/2931091490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998105bc0 0x7f9998108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 --2- 172.21.15.81:0/2931091490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f9998071170 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 --2- 172.21.15.81:0/2931091490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9998071740 0x7f9998071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 -- 172.21.15.81:0/2931091490 >> 172.21.15.81:0/2931091490 conn(0x7f99980f93f0 msgr2=0x7f99980fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:51.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 -- 172.21.15.81:0/2931091490 shutdown_connections 2022-09-10T04:26:51.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.709+0000 7f999cbff700 1 -- 172.21.15.81:0/2931091490 wait complete. 2022-09-10T04:26:51.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.711+0000 7f999cbff700 1 Processor -- start 2022-09-10T04:26:51.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.711+0000 7f999cbff700 1 -- start start 2022-09-10T04:26:51.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999cbff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998071740 0x7f9998195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999cbff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f99981960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999cbff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9998105bc0 0x7f999819d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999cbff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999819d710 con 0x7f9998072c90 2022-09-10T04:26:51.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999cbff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999819d880 con 0x7f9998071740 2022-09-10T04:26:51.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999cbff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999819da40 con 0x7f9998105bc0 2022-09-10T04:26:51.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998071740 0x7f9998195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f9995d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f99981960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998071740 0x7f9998195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:36514/0 (socket says 172.21.15.81:36514) 2022-09-10T04:26:51.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f999659c700 1 -- 172.21.15.81:0/3432859727 learned_addr learned my addr 172.21.15.81:0/3432859727 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:51.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.712+0000 7f9996d9d700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9998105bc0 0x7f999819d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f9995d9b700 1 -- 172.21.15.81:0/3432859727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9998105bc0 msgr2=0x7f999819d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f9995d9b700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9998105bc0 0x7f999819d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f9995d9b700 1 -- 172.21.15.81:0/3432859727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998071740 msgr2=0x7f9998195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f9995d9b700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998071740 0x7f9998195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f9995d9b700 1 -- 172.21.15.81:0/3432859727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990009730 con 0x7f9998072c90 2022-09-10T04:26:51.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f9995d9b700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f99981960d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f998c00e6c0 tx=0x7f998c00e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:51.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c010840 con 0x7f9998072c90 2022-09-10T04:26:51.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f998c010e40 con 0x7f9998072c90 2022-09-10T04:26:51.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9988009470 con 0x7f9998072c90 2022-09-10T04:26:51.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c00f420 con 0x7f9998072c90 2022-09-10T04:26:51.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.713+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999819e0f0 con 0x7f9998072c90 2022-09-10T04:26:51.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.715+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999810b700 con 0x7f9998072c90 2022-09-10T04:26:51.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.717+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f998c00c040 con 0x7f9998072c90 2022-09-10T04:26:51.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.718+0000 7f99877fe700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9980099090 0x7f998009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:51.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.718+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f998c014070 con 0x7f9998072c90 2022-09-10T04:26:51.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.718+0000 7f999659c700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9980099090 0x7f998009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:51.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.719+0000 7f999659c700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9980099090 0x7f998009b540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f998800b4f0 tx=0x7f9988009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:51.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.722+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f998c04fc80 con 0x7f9998072c90 2022-09-10T04:26:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:51 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1198177337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:51 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1274180296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:26:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:51 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1198177337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:51 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1274180296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:26:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:51 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1198177337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:26:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:51 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1274180296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:26:51.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.912+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9998104180 con 0x7f9998072c90 2022-09-10T04:26:51.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.912+0000 7f99877fe700 1 -- 172.21.15.81:0/3432859727 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f998c08ea40 con 0x7f9998072c90 2022-09-10T04:26:51.915 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:26:51.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9980099090 msgr2=0x7f998009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9980099090 0x7f998009b540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f998800b4f0 tx=0x7f9988009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 msgr2=0x7f99981960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:51.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f99981960d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f998c00e6c0 tx=0x7f998c00e9d0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 shutdown_connections 2022-09-10T04:26:51.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9980099090 0x7f998009b540 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9998105bc0 0x7f999819d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9998071740 0x7f9998195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 --2- 172.21.15.81:0/3432859727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9998072c90 0x7f99981960d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:51.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 >> 172.21.15.81:0/3432859727 conn(0x7f99980f93f0 msgr2=0x7f99980fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:51.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 shutdown_connections 2022-09-10T04:26:51.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:51.915+0000 7f999cbff700 1 -- 172.21.15.81:0/3432859727 wait complete. 2022-09-10T04:26:51.945 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2022-09-10T04:26:52.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 -- 172.21.15.81:0/2346091351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0fef80 msgr2=0x7f672c0ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 --2- 172.21.15.81:0/2346091351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0fef80 0x7f672c0ff3d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6720009810 tx=0x7f6720009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:52.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 -- 172.21.15.81:0/2346091351 shutdown_connections 2022-09-10T04:26:52.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 --2- 172.21.15.81:0/2346091351 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c0ff910 0x7f672c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 --2- 172.21.15.81:0/2346091351 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c0fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 --2- 172.21.15.81:0/2346091351 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0fef80 0x7f672c0ff3d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 -- 172.21.15.81:0/2346091351 >> 172.21.15.81:0/2346091351 conn(0x7f672c0f93f0 msgr2=0x7f672c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:52.176 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 -- 172.21.15.81:0/2346091351 shutdown_connections 2022-09-10T04:26:52.176 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.172+0000 7f67327ff700 1 -- 172.21.15.81:0/2346091351 wait complete. 2022-09-10T04:26:52.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.174+0000 7f67327ff700 1 Processor -- start 2022-09-10T04:26:52.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.174+0000 7f67327ff700 1 -- start start 2022-09-10T04:26:52.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f67327ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f67327ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c0fef80 0x7f672c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f67327ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0ff910 0x7f672c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f67327ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c071e10 con 0x7f672c0ff910 2022-09-10T04:26:52.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f67327ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c071f80 con 0x7f672c0fdd00 2022-09-10T04:26:52.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f67327ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c19da20 con 0x7f672c0fef80 2022-09-10T04:26:52.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f6730d9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0ff910 0x7f672c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f672bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f672b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c0fef80 0x7f672c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f672bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:36520/0 (socket says 172.21.15.81:36520) 2022-09-10T04:26:52.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f6730d9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0ff910 0x7f672c071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42540/0 (socket says 172.21.15.81:42540) 2022-09-10T04:26:52.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.175+0000 7f672bfff700 1 -- 172.21.15.81:0/1894292179 learned_addr learned my addr 172.21.15.81:0/1894292179 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:52.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f672bfff700 1 -- 172.21.15.81:0/1894292179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c0fef80 msgr2=0x7f672c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f672bfff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c0fef80 0x7f672c0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f672bfff700 1 -- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0ff910 msgr2=0x7f672c071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f672bfff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0ff910 0x7f672c071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f672bfff700 1 -- 172.21.15.81:0/1894292179 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6720009470 con 0x7f672c0fdd00 2022-09-10T04:26:52.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f672bfff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c072a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e710 tx=0x7f671c00ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:52.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c004220 con 0x7f672c0fdd00 2022-09-10T04:26:52.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f671c004380 con 0x7f672c0fdd00 2022-09-10T04:26:52.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c19dd00 con 0x7f672c0fdd00 2022-09-10T04:26:52.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.176+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671c003cf0 con 0x7f672c0fdd00 2022-09-10T04:26:52.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.177+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c19e1c0 con 0x7f672c0fdd00 2022-09-10T04:26:52.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.177+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c1094b0 con 0x7f672c0fdd00 2022-09-10T04:26:52.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.180+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f671c01e020 con 0x7f672c0fdd00 2022-09-10T04:26:52.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.181+0000 7f67297fa700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6714099010 0x7f671409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.181+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f671c014070 con 0x7f672c0fdd00 2022-09-10T04:26:52.186 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.181+0000 7f672b7fe700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6714099010 0x7f671409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.187 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.182+0000 7f672b7fe700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6714099010 0x7f671409b4c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f672000b4f0 tx=0x7f6720009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:52.187 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.183+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f671c08eff0 con 0x7f672c0fdd00 2022-09-10T04:26:52.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.389+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f672c104bf0 con 0x7f672c0fdd00 2022-09-10T04:26:52.391 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.390+0000 7f67297fa700 1 -- 172.21.15.81:0/1894292179 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+591 (secure 0 0 0) 0x7f671c092610 con 0x7f672c0fdd00 2022-09-10T04:26:52.392 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2927238869 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:52.392 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/3943821768 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:52.392 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2359268015 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:52.393 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/2567224701 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:52.393 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/2567224701 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:52.394 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2774479929 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:52.394 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/1348172231 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:52.395 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/596170975 2022-09-11T04:18:41.737767+0000 2022-09-10T04:26:52.396 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/3224857868 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:52.396 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/3224857868 2022-09-11T04:19:09.198520+0000 2022-09-10T04:26:52.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.392+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6714099010 msgr2=0x7f671409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.392+0000 7f67327ff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6714099010 0x7f671409b4c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f672000b4f0 tx=0x7f6720009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.392+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 msgr2=0x7f672c072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.392+0000 7f67327ff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c072a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f671c00e710 tx=0x7f671c00ea20 comp rx=0 tx=0).stop 2022-09-10T04:26:52.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 shutdown_connections 2022-09-10T04:26:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f672c0ff910 0x7f672c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f672c0fdd00 0x7f672c072a90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6714099010 0x7f671409b4c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 --2- 172.21.15.81:0/1894292179 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f672c0fef80 0x7f672c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 >> 172.21.15.81:0/1894292179 conn(0x7f672c0f93f0 msgr2=0x7f672c0fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:52.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 shutdown_connections 2022-09-10T04:26:52.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.393+0000 7f67327ff700 1 -- 172.21.15.81:0/1894292179 wait complete. 2022-09-10T04:26:52.403 INFO:teuthology.orchestra.run.smithi081.stderr:listed 10 entries 2022-09-10T04:26:52.421 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi081.front.sepia.ceph.com... 2022-09-10T04:26:52.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:52.426 DEBUG:teuthology.orchestra.run.smithi081:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2022-09-10T04:26:52.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:ceph-fuse[127937]: fuse finished with error 0 and tester_r 0 2022-09-10T04:26:52.516 INFO:teuthology.orchestra.run:waiting for 300 2022-09-10T04:26:52.562 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:26:52.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:26:52.563 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2022-09-10T04:26:52.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:52.711 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:52 smithi081 ceph-mon[90133]: pgmap v311: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2022-09-10T04:26:52.711 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:52 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/979177763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:26:52.712 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:52 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3432859727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:26:52.713 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:52 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1894292179' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-10T04:26:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:52 smithi104 ceph-mon[97044]: pgmap v311: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2022-09-10T04:26:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:52 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/979177763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:26:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:52 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3432859727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:26:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:52 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1894292179' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-10T04:26:52.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 -- 172.21.15.81:0/4286844141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05501038c0 msgr2=0x7f0550105d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 --2- 172.21.15.81:0/4286844141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05501038c0 0x7f0550105d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0548009bb0 tx=0x7f0548009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 -- 172.21.15.81:0/4286844141 shutdown_connections 2022-09-10T04:26:52.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 --2- 172.21.15.81:0/4286844141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05501038c0 0x7f0550105d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 --2- 172.21.15.81:0/4286844141 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550066bf0 0x7f0550100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 --2- 172.21.15.81:0/4286844141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0550100dd0 0x7f0550103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 -- 172.21.15.81:0/4286844141 >> 172.21.15.81:0/4286844141 conn(0x7f05500f82a0 msgr2=0x7f05500fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:52.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 -- 172.21.15.81:0/4286844141 shutdown_connections 2022-09-10T04:26:52.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.861+0000 7f05587ff700 1 -- 172.21.15.81:0/4286844141 wait complete. 2022-09-10T04:26:52.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.862+0000 7f05587ff700 1 Processor -- start 2022-09-10T04:26:52.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.862+0000 7f05587ff700 1 -- start start 2022-09-10T04:26:52.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f05587ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 0x7f0550195b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f05587ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550100dd0 0x7f0550196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f05587ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05501038c0 0x7f055019d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f05587ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055019d690 con 0x7f0550066bf0 2022-09-10T04:26:52.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f05587ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055019d800 con 0x7f0550100dd0 2022-09-10T04:26:52.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f05587ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055019d9c0 con 0x7f05501038c0 2022-09-10T04:26:52.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 0x7f0550195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 0x7f0550195b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42562/0 (socket says 172.21.15.81:42562) 2022-09-10T04:26:52.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 -- 172.21.15.81:0/56082865 learned_addr learned my addr 172.21.15.81:0/56082865 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:52.882 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f0555d9a700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550100dd0 0x7f0550196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.882 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f0556d9c700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05501038c0 0x7f055019d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 -- 172.21.15.81:0/56082865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05501038c0 msgr2=0x7f055019d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05501038c0 0x7f055019d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 -- 172.21.15.81:0/56082865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550100dd0 msgr2=0x7f0550196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:52.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550100dd0 0x7f0550196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:52.885 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f055659b700 1 -- 172.21.15.81:0/56082865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480097e0 con 0x7f0550066bf0 2022-09-10T04:26:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.863+0000 7f0555d9a700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550100dd0 0x7f0550196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:52.886 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.864+0000 7f055659b700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 0x7f0550195b10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f054c00e6e0 tx=0x7f054c00e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:52.887 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.864+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c009500 con 0x7f0550066bf0 2022-09-10T04:26:52.887 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.864+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f054c010da0 con 0x7f0550066bf0 2022-09-10T04:26:52.888 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.864+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055019dca0 con 0x7f0550066bf0 2022-09-10T04:26:52.888 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.864+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c0173d0 con 0x7f0550066bf0 2022-09-10T04:26:52.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.864+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055019e1b0 con 0x7f0550066bf0 2022-09-10T04:26:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.865+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05501094d0 con 0x7f0550066bf0 2022-09-10T04:26:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.867+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f054c00c040 con 0x7f0550066bf0 2022-09-10T04:26:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.868+0000 7f05477fe700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f053c099090 0x7f053c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:52.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.868+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f054c014070 con 0x7f0550066bf0 2022-09-10T04:26:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.869+0000 7f0555d9a700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f053c099090 0x7f053c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:52.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.869+0000 7f0555d9a700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f053c099090 0x7f053c09b540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0540005fd0 tx=0x7f0540005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:52.892 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:52.872+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f054c04fc80 con 0x7f0550066bf0 2022-09-10T04:26:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:52 smithi110 ceph-mon[97633]: pgmap v311: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2022-09-10T04:26:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:52 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/979177763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:26:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:52 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3432859727' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:26:52.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:52 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1894292179' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2022-09-10T04:26:53.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.063+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f055019ed20 con 0x7f0550066bf0 2022-09-10T04:26:53.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.064+0000 7f05477fe700 1 -- 172.21.15.81:0/56082865 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7f054c08eb40 con 0x7f0550066bf0 2022-09-10T04:26:53.066 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:53.067 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:53.069 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f053c099090 msgr2=0x7f053c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f053c099090 0x7f053c09b540 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0540005fd0 tx=0x7f0540005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.070 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 msgr2=0x7f0550195b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 0x7f0550195b10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f054c00e6e0 tx=0x7f054c00e9f0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 shutdown_connections 2022-09-10T04:26:53.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f053c099090 0x7f053c09b540 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05501038c0 0x7f055019d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0550066bf0 0x7f0550195b10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.073 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 --2- 172.21.15.81:0/56082865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0550100dd0 0x7f0550196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.073 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.066+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 >> 172.21.15.81:0/56082865 conn(0x7f05500f82a0 msgr2=0x7f05500ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:53.074 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.067+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 shutdown_connections 2022-09-10T04:26:53.074 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.067+0000 7f05587ff700 1 -- 172.21.15.81:0/56082865 wait complete. 2022-09-10T04:26:53.095 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:53.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 -- 172.21.15.81:0/3480737900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fc430 msgr2=0x7fb1e40fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 --2- 172.21.15.81:0/3480737900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fc430 0x7fb1e40fc840 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009810 tx=0x7fb1d4009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:53.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 -- 172.21.15.81:0/3480737900 shutdown_connections 2022-09-10T04:26:53.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 --2- 172.21.15.81:0/3480737900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fd730 0x7fb1e40fdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 --2- 172.21.15.81:0/3480737900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fc430 0x7fb1e40fc840 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 --2- 172.21.15.81:0/3480737900 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e40fcd80 0x7fb1e40fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 -- 172.21.15.81:0/3480737900 >> 172.21.15.81:0/3480737900 conn(0x7fb1e40f8160 msgr2=0x7fb1e40fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:53.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 -- 172.21.15.81:0/3480737900 shutdown_connections 2022-09-10T04:26:53.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.331+0000 7fb1e899d700 1 -- 172.21.15.81:0/3480737900 wait complete. 2022-09-10T04:26:53.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.332+0000 7fb1e899d700 1 Processor -- start 2022-09-10T04:26:53.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.332+0000 7fb1e899d700 1 -- start start 2022-09-10T04:26:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e899d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 0x7fb1e41959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e899d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fcd80 0x7fb1e4195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e899d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e40fd730 0x7fb1e419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e899d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e419d560 con 0x7fb1e40fcd80 2022-09-10T04:26:53.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e899d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e419d6d0 con 0x7fb1e40fd730 2022-09-10T04:26:53.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e899d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e419d890 con 0x7fb1e40fc430 2022-09-10T04:26:53.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 0x7fb1e41959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 0x7fb1e41959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51868/0 (socket says 172.21.15.81:51868) 2022-09-10T04:26:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 -- 172.21.15.81:0/879318686 learned_addr learned my addr 172.21.15.81:0/879318686 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:53.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e2ffd700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fcd80 0x7fb1e4195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e3fff700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e40fd730 0x7fb1e419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 -- 172.21.15.81:0/879318686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e40fd730 msgr2=0x7fb1e419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e40fd730 0x7fb1e419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 -- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fcd80 msgr2=0x7fb1e4195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fcd80 0x7fb1e4195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.333+0000 7fb1e37fe700 1 -- 172.21.15.81:0/879318686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d80093b0 con 0x7fb1e40fc430 2022-09-10T04:26:53.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.334+0000 7fb1e37fe700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 0x7fb1e41959e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb1d40059c0 tx=0x7fb1d400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:53.348 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.334+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d401d070 con 0x7fb1e40fc430 2022-09-10T04:26:53.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.334+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d4009470 con 0x7fb1e40fc430 2022-09-10T04:26:53.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.334+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e419de80 con 0x7fb1e40fc430 2022-09-10T04:26:53.349 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.334+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb1d400f880 con 0x7fb1e40fc430 2022-09-10T04:26:53.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.334+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d40047f0 con 0x7fb1e40fc430 2022-09-10T04:26:53.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.335+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e410b4d0 con 0x7fb1e40fc430 2022-09-10T04:26:53.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.337+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb1d4028030 con 0x7fb1e40fc430 2022-09-10T04:26:53.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.338+0000 7fb1e0ff9700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1cc099010 0x7fb1cc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.338+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fb1d40c31d0 con 0x7fb1e40fc430 2022-09-10T04:26:53.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.339+0000 7fb1e2ffd700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1cc099010 0x7fb1cc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.353 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.339+0000 7fb1e2ffd700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1cc099010 0x7fb1cc09b4c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8005fd0 tx=0x7fb1d8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:53.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.342+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb1d4059c80 con 0x7fb1e40fc430 2022-09-10T04:26:53.542 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.540+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e405f990 con 0x7fb1e40fc430 2022-09-10T04:26:53.543 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.542+0000 7fb1e0ff9700 1 -- 172.21.15.81:0/879318686 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7fb1d408f670 con 0x7fb1e40fc430 2022-09-10T04:26:53.543 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:53.544 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:53.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1cc099010 msgr2=0x7fb1cc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1cc099010 0x7fb1cc09b4c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8005fd0 tx=0x7fb1d8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 msgr2=0x7fb1e41959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 0x7fb1e41959e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb1d40059c0 tx=0x7fb1d400f690 comp rx=0 tx=0).stop 2022-09-10T04:26:53.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 shutdown_connections 2022-09-10T04:26:53.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e40fd730 0x7fb1e419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1cc099010 0x7fb1cc09b4c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1e40fc430 0x7fb1e41959e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 --2- 172.21.15.81:0/879318686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e40fcd80 0x7fb1e4195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 >> 172.21.15.81:0/879318686 conn(0x7fb1e40f8160 msgr2=0x7fb1e40faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:53.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 shutdown_connections 2022-09-10T04:26:53.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.544+0000 7fb1e899d700 1 -- 172.21.15.81:0/879318686 wait complete. 2022-09-10T04:26:53.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-09-10T04:26:53.692 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:53 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/56082865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:53 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/56082865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:53.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 -- 172.21.15.81:0/1109607083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee84066bf0 msgr2=0x7fee840fe6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 --2- 172.21.15.81:0/1109607083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee84066bf0 0x7fee840fe6a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fee74009810 tx=0x7fee74009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 -- 172.21.15.81:0/1109607083 shutdown_connections 2022-09-10T04:26:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 --2- 172.21.15.81:0/1109607083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee841016d0 0x7fee84101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 --2- 172.21.15.81:0/1109607083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee84066bf0 0x7fee840fe6a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 --2- 172.21.15.81:0/1109607083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee840febe0 0x7fee84101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 -- 172.21.15.81:0/1109607083 >> 172.21.15.81:0/1109607083 conn(0x7fee840f82e0 msgr2=0x7fee840fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.845+0000 7fee891ff700 1 -- 172.21.15.81:0/1109607083 shutdown_connections 2022-09-10T04:26:53.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.846+0000 7fee891ff700 1 -- 172.21.15.81:0/1109607083 wait complete. 2022-09-10T04:26:53.851 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 Processor -- start 2022-09-10T04:26:53.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 -- start start 2022-09-10T04:26:53.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee84066bf0 0x7fee84199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee840febe0 0x7fee8419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 0x7fee841a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee841a1b20 con 0x7fee840febe0 2022-09-10T04:26:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee841a1c90 con 0x7fee84066bf0 2022-09-10T04:26:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.847+0000 7fee891ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee841a1e50 con 0x7fee841016d0 2022-09-10T04:26:53.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8259c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee840febe0 0x7fee8419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 0x7fee841a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 0x7fee841a1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51882/0 (socket says 172.21.15.81:51882) 2022-09-10T04:26:53.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 -- 172.21.15.81:0/4040772809 learned_addr learned my addr 172.21.15.81:0/4040772809 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:53.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee82d9d700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee84066bf0 0x7fee84199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 -- 172.21.15.81:0/4040772809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee84066bf0 msgr2=0x7fee84199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee84066bf0 0x7fee84199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 -- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee840febe0 msgr2=0x7fee8419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:53.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee840febe0 0x7fee8419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:53.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 -- 172.21.15.81:0/4040772809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74009470 con 0x7fee841016d0 2022-09-10T04:26:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8259c700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee840febe0 0x7fee8419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:53.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.848+0000 7fee8359e700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 0x7fee841a1550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00ebf0 tx=0x7fee7c00ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:53.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.849+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0099c0 con 0x7fee841016d0 2022-09-10T04:26:53.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.849+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee841a2130 con 0x7fee841016d0 2022-09-10T04:26:53.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.849+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fee7c00cc90 con 0x7fee841016d0 2022-09-10T04:26:53.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.849+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c005850 con 0x7fee841016d0 2022-09-10T04:26:53.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.849+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee841a2640 con 0x7fee841016d0 2022-09-10T04:26:53.863 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.850+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8410fab0 con 0x7fee841016d0 2022-09-10T04:26:53.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.852+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fee7c01e020 con 0x7fee841016d0 2022-09-10T04:26:53.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.853+0000 7fee67fff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fee6c099090 0x7fee6c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:53.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.853+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fee7c014070 con 0x7fee841016d0 2022-09-10T04:26:53.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.854+0000 7fee82d9d700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fee6c099090 0x7fee6c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:53.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.854+0000 7fee82d9d700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fee6c099090 0x7fee6c09b540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fee74009760 tx=0x7fee74009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:53.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:53.855+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fee7c08f720 con 0x7fee841016d0 2022-09-10T04:26:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:53 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/56082865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:54.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.091+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fee840f9cd0 con 0x7fee841016d0 2022-09-10T04:26:54.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.431+0000 7fee67fff700 1 -- 172.21.15.81:0/4040772809 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fee7c092d40 con 0x7fee841016d0 2022-09-10T04:26:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fee6c099090 msgr2=0x7fee6c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fee6c099090 0x7fee6c09b540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fee74009760 tx=0x7fee74009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 msgr2=0x7fee841a1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 0x7fee841a1550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00ebf0 tx=0x7fee7c00ef00 comp rx=0 tx=0).stop 2022-09-10T04:26:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 shutdown_connections 2022-09-10T04:26:54.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fee6c099090 0x7fee6c09b540 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee841016d0 0x7fee841a1550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee84066bf0 0x7fee84199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 --2- 172.21.15.81:0/4040772809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee840febe0 0x7fee8419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.434+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 >> 172.21.15.81:0/4040772809 conn(0x7fee840f82e0 msgr2=0x7fee840fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.435+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 shutdown_connections 2022-09-10T04:26:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.435+0000 7fee891ff700 1 -- 172.21.15.81:0/4040772809 wait complete. 2022-09-10T04:26:54.463 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True} 2022-09-10T04:26:54.464 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2022-09-10T04:26:54.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 -- 172.21.15.81:0/195969004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 msgr2=0x7f52300fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:54.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 --2- 172.21.15.81:0/195969004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f52300fe110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5220009810 tx=0x7f5220009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:54.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 -- 172.21.15.81:0/195969004 shutdown_connections 2022-09-10T04:26:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 --2- 172.21.15.81:0/195969004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52300ff910 0x7f52301061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 --2- 172.21.15.81:0/195969004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f52300fe110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 --2- 172.21.15.81:0/195969004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52300ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 -- 172.21.15.81:0/195969004 >> 172.21.15.81:0/195969004 conn(0x7f52300f93f0 msgr2=0x7f52300fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 -- 172.21.15.81:0/195969004 shutdown_connections 2022-09-10T04:26:54.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.693+0000 7f52355ff700 1 -- 172.21.15.81:0/195969004 wait complete. 2022-09-10T04:26:54.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.694+0000 7f52355ff700 1 Processor -- start 2022-09-10T04:26:54.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 -- start start 2022-09-10T04:26:54.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f5230195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:54.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52301960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:54.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52300ff910 0x7f523019d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:54.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523019d6e0 con 0x7f52300fef80 2022-09-10T04:26:54.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523019d850 con 0x7f52300fdd00 2022-09-10T04:26:54.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f52355ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523019da10 con 0x7f52300ff910 2022-09-10T04:26:54.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f522e7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52301960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:54.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f522e7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52301960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42612/0 (socket says 172.21.15.81:42612) 2022-09-10T04:26:54.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f522effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f5230195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:54.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f522e7fc700 1 -- 172.21.15.81:0/1185028661 learned_addr learned my addr 172.21.15.81:0/1185028661 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:54.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.695+0000 7f522f7fe700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52300ff910 0x7f523019d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:54.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522e7fc700 1 -- 172.21.15.81:0/1185028661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52300ff910 msgr2=0x7f523019d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:54.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522e7fc700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52300ff910 0x7f523019d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522e7fc700 1 -- 172.21.15.81:0/1185028661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 msgr2=0x7f5230195b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:54.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522e7fc700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f5230195b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:54.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522e7fc700 1 -- 172.21.15.81:0/1185028661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5220009470 con 0x7f52300fef80 2022-09-10T04:26:54.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522effd700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f5230195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:54.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f522e7fc700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52301960a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f522400e740 tx=0x7f522400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:54.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224005220 con 0x7f52300fef80 2022-09-10T04:26:54.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5224005380 con 0x7f52300fef80 2022-09-10T04:26:54.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523019dcf0 con 0x7f52300fef80 2022-09-10T04:26:54.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.696+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224004b70 con 0x7f52300fef80 2022-09-10T04:26:54.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.697+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523019e1f0 con 0x7f52300fef80 2022-09-10T04:26:54.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.698+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52301094b0 con 0x7f52300fef80 2022-09-10T04:26:54.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.699+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f522401e020 con 0x7f52300fef80 2022-09-10T04:26:54.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.701+0000 7f5217fff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5218099090 0x7f521809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:54.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.701+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f5224014070 con 0x7f52300fef80 2022-09-10T04:26:54.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.701+0000 7f522effd700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5218099090 0x7f521809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:54.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.702+0000 7f522effd700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5218099090 0x7f521809b540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f522000b4f0 tx=0x7f5220009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:54.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.705+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f522408eb80 con 0x7f52300fef80 2022-09-10T04:26:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:54 smithi104 ceph-mon[97044]: pgmap v312: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2022-09-10T04:26:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:54 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/879318686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:54 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4040772809' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-10T04:26:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:54 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-10T04:26:54.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:54 smithi081 ceph-mon[90133]: pgmap v312: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2022-09-10T04:26:54.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:54 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/879318686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:54.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:54 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4040772809' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-10T04:26:54.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:54 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-10T04:26:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:54 smithi110 ceph-mon[97633]: pgmap v312: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2022-09-10T04:26:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:54 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/879318686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:54 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4040772809' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-10T04:26:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:54 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2022-09-10T04:26:54.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:54.944+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f52301900c0 con 0x7f52300fef80 2022-09-10T04:26:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.441+0000 7f5217fff700 1 -- 172.21.15.81:0/1185028661 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f52240921a0 con 0x7f52300fef80 2022-09-10T04:26:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.444+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5218099090 msgr2=0x7f521809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.444+0000 7f52355ff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5218099090 0x7f521809b540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f522000b4f0 tx=0x7f5220009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.444+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 msgr2=0x7f52301960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.444+0000 7f52355ff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52301960a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f522400e740 tx=0x7f522400ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 shutdown_connections 2022-09-10T04:26:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5218099090 0x7f521809b540 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f52300ff910 0x7f523019d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52300fdd00 0x7f5230195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 --2- 172.21.15.81:0/1185028661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52300fef80 0x7f52301960a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 >> 172.21.15.81:0/1185028661 conn(0x7f52300f93f0 msgr2=0x7f5230104960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 shutdown_connections 2022-09-10T04:26:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.445+0000 7f52355ff700 1 -- 172.21.15.81:0/1185028661 wait complete. 2022-09-10T04:26:55.478 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2022-09-10T04:26:55.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 -- 172.21.15.81:0/2932804624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0fdd00 msgr2=0x7f181c0fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:55.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 --2- 172.21.15.81:0/2932804624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0fdd00 0x7f181c0fe110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1818009810 tx=0x7f1818009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:55.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 -- 172.21.15.81:0/2932804624 shutdown_connections 2022-09-10T04:26:55.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 --2- 172.21.15.81:0/2932804624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0ff910 0x7f181c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 --2- 172.21.15.81:0/2932804624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0fdd00 0x7f181c0fe110 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 --2- 172.21.15.81:0/2932804624 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fef80 0x7f181c0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.709+0000 7f18241ff700 1 -- 172.21.15.81:0/2932804624 >> 172.21.15.81:0/2932804624 conn(0x7f181c0f93f0 msgr2=0x7f181c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:55.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.710+0000 7f18241ff700 1 -- 172.21.15.81:0/2932804624 shutdown_connections 2022-09-10T04:26:55.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.710+0000 7f18241ff700 1 -- 172.21.15.81:0/2932804624 wait complete. 2022-09-10T04:26:55.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 Processor -- start 2022-09-10T04:26:55.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 -- start start 2022-09-10T04:26:55.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fdd00 0x7f181c072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:55.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 0x7f181c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:55.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0ff910 0x7f181c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:55.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c071e10 con 0x7f181c0fef80 2022-09-10T04:26:55.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c071f80 con 0x7f181c0ff910 2022-09-10T04:26:55.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.711+0000 7f18241ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c19da40 con 0x7f181c0fdd00 2022-09-10T04:26:55.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f182179a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 0x7f181c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:55.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fdd00 0x7f181c072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:55.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f182179a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 0x7f181c0710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42634/0 (socket says 172.21.15.81:42634) 2022-09-10T04:26:55.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f182179a700 1 -- 172.21.15.81:0/3833987204 learned_addr learned my addr 172.21.15.81:0/3833987204 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:55.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f182279c700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0ff910 0x7f181c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:55.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 -- 172.21.15.81:0/3833987204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0ff910 msgr2=0x7f181c071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:55.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0ff910 0x7f181c071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 -- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 msgr2=0x7f181c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:55.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 0x7f181c0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:55.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 -- 172.21.15.81:0/3833987204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818009470 con 0x7f181c0fdd00 2022-09-10T04:26:55.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f182179a700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 0x7f181c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:55.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.712+0000 7f1821f9b700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fdd00 0x7f181c072a90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1818006010 tx=0x7f181800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:55.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.713+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181801d070 con 0x7f181c0fdd00 2022-09-10T04:26:55.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.713+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f181800f880 con 0x7f181c0fdd00 2022-09-10T04:26:55.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.713+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818004780 con 0x7f181c0fdd00 2022-09-10T04:26:55.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.713+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c19dcc0 con 0x7f181c0fdd00 2022-09-10T04:26:55.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.713+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c19e120 con 0x7f181c0fdd00 2022-09-10T04:26:55.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.714+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c1094b0 con 0x7f181c0fdd00 2022-09-10T04:26:55.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.716+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1818028030 con 0x7f181c0fdd00 2022-09-10T04:26:55.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.717+0000 7f1812ffd700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1808099010 0x7f180809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:55.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.717+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f18180c3060 con 0x7f181c0fdd00 2022-09-10T04:26:55.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.717+0000 7f182179a700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1808099010 0x7f180809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:55.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.718+0000 7f182179a700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1808099010 0x7f180809b4c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f180c0057d0 tx=0x7f180c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:55.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.719+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f181808f500 con 0x7f181c0fdd00 2022-09-10T04:26:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:55 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-09-10T04:26:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:55 smithi104 ceph-mon[97044]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:55 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1185028661' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-09-10T04:26:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:55 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-09-10T04:26:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:55 smithi081 ceph-mon[90133]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:55 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1185028661' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-09-10T04:26:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:55 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2022-09-10T04:26:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:55 smithi110 ceph-mon[97633]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:55 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1185028661' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2022-09-10T04:26:55.960 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:55.959+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f181c05f950 con 0x7f181c0fdd00 2022-09-10T04:26:56.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.447+0000 7f1812ffd700 1 -- 172.21.15.81:0/3833987204 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1818092b20 con 0x7f181c0fdd00 2022-09-10T04:26:56.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1808099010 msgr2=0x7f180809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:56.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1808099010 0x7f180809b4c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f180c0057d0 tx=0x7f180c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fdd00 msgr2=0x7f181c072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fdd00 0x7f181c072a90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1818006010 tx=0x7f181800f690 comp rx=0 tx=0).stop 2022-09-10T04:26:56.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 shutdown_connections 2022-09-10T04:26:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f181c0ff910 0x7f181c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1808099010 0x7f180809b4c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f181c0fdd00 0x7f181c072a90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.450+0000 7f18241ff700 1 --2- 172.21.15.81:0/3833987204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0fef80 0x7f181c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.451+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 >> 172.21.15.81:0/3833987204 conn(0x7f181c0f93f0 msgr2=0x7f181c0fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:56.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.451+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 shutdown_connections 2022-09-10T04:26:56.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.451+0000 7f18241ff700 1 -- 172.21.15.81:0/3833987204 wait complete. 2022-09-10T04:26:56.483 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2022-09-10T04:26:56.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.711+0000 7f6e19d9d700 1 -- 172.21.15.81:0/2156926944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 msgr2=0x7f6e14107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.711+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/2156926944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 0x7f6e14107ec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009bb0 tx=0x7f6e0c009ec0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.711+0000 7f6e19d9d700 1 -- 172.21.15.81:0/2156926944 shutdown_connections 2022-09-10T04:26:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.712+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/2156926944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 0x7f6e14107ec0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.712+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/2156926944 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 0x7f6e14071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.712+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/2156926944 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.712+0000 7f6e19d9d700 1 -- 172.21.15.81:0/2156926944 >> 172.21.15.81:0/2156926944 conn(0x7f6e140f9260 msgr2=0x7f6e140fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:56.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.712+0000 7f6e19d9d700 1 -- 172.21.15.81:0/2156926944 shutdown_connections 2022-09-10T04:26:56.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.712+0000 7f6e19d9d700 1 -- 172.21.15.81:0/2156926944 wait complete. 2022-09-10T04:26:56.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.713+0000 7f6e19d9d700 1 Processor -- start 2022-09-10T04:26:56.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 -- start start 2022-09-10T04:26:56.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:56.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 0x7f6e14195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:56.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 0x7f6e1419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1419d4a0 con 0x7f6e14105ae0 2022-09-10T04:26:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1419d610 con 0x7f6e14072b90 2022-09-10T04:26:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.714+0000 7f6e19d9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1419d7d0 con 0x7f6e14071650 2022-09-10T04:26:56.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e18d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 0x7f6e14195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e18d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51946/0 (socket says 172.21.15.81:51946) 2022-09-10T04:26:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e18d9b700 1 -- 172.21.15.81:0/4163190094 learned_addr learned my addr 172.21.15.81:0/4163190094 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:56.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e1959c700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 0x7f6e1419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 -- 172.21.15.81:0/4163190094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 msgr2=0x7f6e14195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 -- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 msgr2=0x7f6e1419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 0x7f6e1419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:56.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 -- 172.21.15.81:0/4163190094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c0097e0 con 0x7f6e14072b90 2022-09-10T04:26:56.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e18d9b700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:56.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.715+0000 7f6e13fff700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 0x7f6e14195e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000e6c0 tx=0x7f6e0000e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:56.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.716+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00010900 con 0x7f6e14072b90 2022-09-10T04:26:56.724 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.716+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1419dab0 con 0x7f6e14072b90 2022-09-10T04:26:56.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.716+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6e00018420 con 0x7f6e14072b90 2022-09-10T04:26:56.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.716+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e00017610 con 0x7f6e14072b90 2022-09-10T04:26:56.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.716+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1419dfc0 con 0x7f6e14072b90 2022-09-10T04:26:56.725 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.717+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1410b520 con 0x7f6e14072b90 2022-09-10T04:26:56.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.719+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6e0000c040 con 0x7f6e14072b90 2022-09-10T04:26:56.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.720+0000 7f6e11ffb700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6e04099010 0x7f6e0409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:56.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.720+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f6e00014070 con 0x7f6e14072b90 2022-09-10T04:26:56.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.721+0000 7f6e18d9b700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6e04099010 0x7f6e0409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:56.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.721+0000 7f6e18d9b700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6e04099010 0x7f6e0409b4c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009f60 tx=0x7f6e08005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:56.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.722+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6e0008ed00 con 0x7f6e14072b90 2022-09-10T04:26:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: pgmap v313: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2022-09-10T04:26:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-09-10T04:26:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1185028661' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2022-09-10T04:26:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-09-10T04:26:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2022-09-10T04:26:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: daemon mds.b is now active in filesystem cephfs as rank 1 2022-09-10T04:26:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3833987204' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-10T04:26:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:56 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-10T04:26:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: pgmap v313: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2022-09-10T04:26:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-09-10T04:26:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1185028661' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2022-09-10T04:26:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-09-10T04:26:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2022-09-10T04:26:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: daemon mds.b is now active in filesystem cephfs as rank 1 2022-09-10T04:26:56.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3833987204' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-10T04:26:56.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:56 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-10T04:26:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: pgmap v313: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2022-09-10T04:26:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2022-09-10T04:26:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1185028661' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2022-09-10T04:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2022-09-10T04:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2022-09-10T04:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2022-09-10T04:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: daemon mds.b is now active in filesystem cephfs as rank 1 2022-09-10T04:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3833987204' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-10T04:26:56.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:56 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2022-09-10T04:26:56.960 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:56.959+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f6e141040f0 con 0x7f6e14072b90 2022-09-10T04:26:57.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.457+0000 7f6e11ffb700 1 -- 172.21.15.81:0/4163190094 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6e00092320 con 0x7f6e14072b90 2022-09-10T04:26:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6e04099010 msgr2=0x7f6e0409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6e04099010 0x7f6e0409b4c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6e08009f60 tx=0x7f6e08005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 msgr2=0x7f6e14195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 0x7f6e14195e60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000e6c0 tx=0x7f6e0000e9d0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 shutdown_connections 2022-09-10T04:26:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e14105ae0 0x7f6e1419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6e04099010 0x7f6e0409b4c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e14071650 0x7f6e14195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 --2- 172.21.15.81:0/4163190094 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e14072b90 0x7f6e14195e60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 >> 172.21.15.81:0/4163190094 conn(0x7f6e140f9260 msgr2=0x7f6e140fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:57.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.460+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 shutdown_connections 2022-09-10T04:26:57.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.461+0000 7f6e19d9d700 1 -- 172.21.15.81:0/4163190094 wait complete. 2022-09-10T04:26:57.492 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2022-09-10T04:26:57.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:26:57.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 -- 172.21.15.81:0/490412834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850072c90 msgr2=0x7f0850071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 --2- 172.21.15.81:0/490412834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850072c90 0x7f0850071170 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0840009810 tx=0x7f0840009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:57.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 -- 172.21.15.81:0/490412834 shutdown_connections 2022-09-10T04:26:57.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 --2- 172.21.15.81:0/490412834 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850105bb0 0x7f0850107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 --2- 172.21.15.81:0/490412834 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850072c90 0x7f0850071170 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 --2- 172.21.15.81:0/490412834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850071740 0x7f0850071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 -- 172.21.15.81:0/490412834 >> 172.21.15.81:0/490412834 conn(0x7f08500f93c0 msgr2=0x7f08500fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:57.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 -- 172.21.15.81:0/490412834 shutdown_connections 2022-09-10T04:26:57.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.718+0000 7f0854bff700 1 -- 172.21.15.81:0/490412834 wait complete. 2022-09-10T04:26:57.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.719+0000 7f0854bff700 1 Processor -- start 2022-09-10T04:26:57.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 -- start start 2022-09-10T04:26:57.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850071740 0x7f0850195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:57.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 0x7f08501960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:57.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850105bb0 0x7f085019d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:57.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085019d700 con 0x7f0850071740 2022-09-10T04:26:57.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085019d870 con 0x7f0850105bb0 2022-09-10T04:26:57.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f0854bff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085019da30 con 0x7f0850072c90 2022-09-10T04:26:57.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f084e59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850071740 0x7f0850195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:57.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f084dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 0x7f08501960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:57.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f084dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 0x7f08501960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51974/0 (socket says 172.21.15.81:51974) 2022-09-10T04:26:57.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f084dd9b700 1 -- 172.21.15.81:0/2797980960 learned_addr learned my addr 172.21.15.81:0/2797980960 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:57.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.720+0000 7f084e59c700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850071740 0x7f0850195b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42678/0 (socket says 172.21.15.81:42678) 2022-09-10T04:26:57.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084ed9d700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850105bb0 0x7f085019d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:57.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084dd9b700 1 -- 172.21.15.81:0/2797980960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850105bb0 msgr2=0x7f085019d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084dd9b700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850105bb0 0x7f085019d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084dd9b700 1 -- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850071740 msgr2=0x7f0850195b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084dd9b700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850071740 0x7f0850195b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084dd9b700 1 -- 172.21.15.81:0/2797980960 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0840009470 con 0x7f0850072c90 2022-09-10T04:26:57.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f084dd9b700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 0x7f08501960c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f084400e740 tx=0x7f084400ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:57.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844005220 con 0x7f0850072c90 2022-09-10T04:26:57.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085019dd10 con 0x7f0850072c90 2022-09-10T04:26:57.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.721+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f0844005380 con 0x7f0850072c90 2022-09-10T04:26:57.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.722+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844004b50 con 0x7f0850072c90 2022-09-10T04:26:57.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.722+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085019e1c0 con 0x7f0850072c90 2022-09-10T04:26:57.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.722+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085010b760 con 0x7f0850072c90 2022-09-10T04:26:57.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.724+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f084401e020 con 0x7f0850072c90 2022-09-10T04:26:57.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.726+0000 7f083f7fe700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0838099010 0x7f083809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:57.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.726+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f0844014070 con 0x7f0850072c90 2022-09-10T04:26:57.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.726+0000 7f084e59c700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0838099010 0x7f083809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:57.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.727+0000 7f084e59c700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0838099010 0x7f083809b4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f084000b4f0 tx=0x7f0840009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:57.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.728+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f084408eda0 con 0x7f0850072c90 2022-09-10T04:26:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] up:active 2022-09-10T04:26:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-09-10T04:26:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-09-10T04:26:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2022-09-10T04:26:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2022-09-10T04:26:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: daemon mds.i is now active in filesystem cephfs as rank 2 2022-09-10T04:26:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4163190094' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-10T04:26:57.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:57 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-10T04:26:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] up:active 2022-09-10T04:26:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-09-10T04:26:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-09-10T04:26:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2022-09-10T04:26:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2022-09-10T04:26:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: daemon mds.i is now active in filesystem cephfs as rank 2 2022-09-10T04:26:57.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4163190094' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-10T04:26:57.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-10T04:26:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] up:active 2022-09-10T04:26:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2022-09-10T04:26:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2022-09-10T04:26:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2022-09-10T04:26:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2022-09-10T04:26:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: daemon mds.i is now active in filesystem cephfs as rank 2 2022-09-10T04:26:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4163190094' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-10T04:26:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:57 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2022-09-10T04:26:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.961+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0850104170 con 0x7f0850072c90 2022-09-10T04:26:57.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.962+0000 7f083f7fe700 1 -- 172.21.15.81:0/2797980960 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10730 (secure 0 0 0) 0x7f08440923c0 con 0x7f0850072c90 2022-09-10T04:26:57.965 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:57.966 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24461,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24473,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24497,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-10T04:26:32.097174+0000","modified":"2022-09-10T04:26:57.458164+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3],"up":{"mds_0":24577,"mds_1":24572,"mds_2":24547,"mds_3":24532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24532":{"gid":24532,"name":"c","rank":3,"incarnation":24,"state":"up:creating","state_seq":1,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24577":{"gid":24577,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-10T04:26:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0838099010 msgr2=0x7f083809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0838099010 0x7f083809b4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f084000b4f0 tx=0x7f0840009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 msgr2=0x7f08501960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:57.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 0x7f08501960c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f084400e740 tx=0x7f084400ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 shutdown_connections 2022-09-10T04:26:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0850105bb0 0x7f085019d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0850071740 0x7f0850195b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f0838099010 0x7f083809b4c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 --2- 172.21.15.81:0/2797980960 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0850072c90 0x7f08501960c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 >> 172.21.15.81:0/2797980960 conn(0x7f08500f93c0 msgr2=0x7f08500fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 shutdown_connections 2022-09-10T04:26:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:57.965+0000 7f0854bff700 1 -- 172.21.15.81:0/2797980960 wait complete. 2022-09-10T04:26:57.972 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 24 2022-09-10T04:26:57.994 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:26:58.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.229+0000 7fa7dd39d700 1 -- 172.21.15.81:0/2985581405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 msgr2=0x7fa7d8100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.229+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/2985581405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 0x7fa7d8100a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8009810 tx=0x7fa7c8009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:58.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.229+0000 7fa7dd39d700 1 -- 172.21.15.81:0/2985581405 shutdown_connections 2022-09-10T04:26:58.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.229+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/2985581405 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8103880 0x7fa7d8105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.229+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/2985581405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 0x7fa7d8100a20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.230+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/2985581405 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d8100f60 0x7fa7d8103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.230+0000 7fa7dd39d700 1 -- 172.21.15.81:0/2985581405 >> 172.21.15.81:0/2985581405 conn(0x7fa7d80f8160 msgr2=0x7fa7d80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:58.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.230+0000 7fa7dd39d700 1 -- 172.21.15.81:0/2985581405 shutdown_connections 2022-09-10T04:26:58.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.230+0000 7fa7dd39d700 1 -- 172.21.15.81:0/2985581405 wait complete. 2022-09-10T04:26:58.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.231+0000 7fa7dd39d700 1 Processor -- start 2022-09-10T04:26:58.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 -- start start 2022-09-10T04:26:58.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 0x7fa7d80fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 0x7fa7d80fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d8103880 0x7fa7d80fcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d80fd330 con 0x7fa7d80fe640 2022-09-10T04:26:58.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d80fd4a0 con 0x7fa7d8100f60 2022-09-10T04:26:58.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7dd39d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7d819d830 con 0x7fa7d8103880 2022-09-10T04:26:58.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7cffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 0x7fa7d80fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7d7fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 0x7fa7d80fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7cffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 0x7fa7d80fc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:36660/0 (socket says 172.21.15.81:36660) 2022-09-10T04:26:58.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.232+0000 7fa7cffff700 1 -- 172.21.15.81:0/328278190 learned_addr learned my addr 172.21.15.81:0/328278190 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:58.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7dcb9c700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d8103880 0x7fa7d80fcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7cffff700 1 -- 172.21.15.81:0/328278190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d8103880 msgr2=0x7fa7d80fcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7cffff700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d8103880 0x7fa7d80fcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7cffff700 1 -- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 msgr2=0x7fa7d80fdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7cffff700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 0x7fa7d80fdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7cffff700 1 -- 172.21.15.81:0/328278190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c8009470 con 0x7fa7d8100f60 2022-09-10T04:26:58.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7cffff700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 0x7fa7d80fc5f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000e740 tx=0x7fa7c000ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:58.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.233+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0005220 con 0x7fa7d8100f60 2022-09-10T04:26:58.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.234+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d819db10 con 0x7fa7d8100f60 2022-09-10T04:26:58.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.234+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa7c0005380 con 0x7fa7d8100f60 2022-09-10T04:26:58.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.234+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c0004b70 con 0x7fa7d8100f60 2022-09-10T04:26:58.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.234+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d819e070 con 0x7fa7d8100f60 2022-09-10T04:26:58.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.235+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d81092c0 con 0x7fa7d8100f60 2022-09-10T04:26:58.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.237+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fa7c001e020 con 0x7fa7d8100f60 2022-09-10T04:26:58.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.238+0000 7fa7d5ffb700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7c4099010 0x7fa7c409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.238+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fa7c0014070 con 0x7fa7d8100f60 2022-09-10T04:26:58.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.238+0000 7fa7d7fff700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7c4099010 0x7fa7c409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.239+0000 7fa7d7fff700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7c4099010 0x7fa7c409b4c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800b4f0 tx=0x7fa7c8009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:58.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.240+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa7c008ef40 con 0x7fa7d8100f60 2022-09-10T04:26:58.304 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-a[125370]: 2022-09-10T04:26:57.913+0000 7f1cb22fb700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2022-09-10T04:26:58.304 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-a[125370]: 2022-09-10T04:26:57.913+0000 7f1cb22fb700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2022-09-10T04:26:58.436 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.434+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7d80fab60 con 0x7fa7d8100f60 2022-09-10T04:26:58.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.436+0000 7fa7d5ffb700 1 -- 172.21.15.81:0/328278190 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7fa7c0092560 con 0x7fa7d8100f60 2022-09-10T04:26:58.438 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:58.438 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:26:58.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.438+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7c4099010 msgr2=0x7fa7c409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.438+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7c4099010 0x7fa7c409b4c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800b4f0 tx=0x7fa7c8009eb0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.438+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 msgr2=0x7fa7d80fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.438+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 0x7fa7d80fc5f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa7c000e740 tx=0x7fa7c000ea50 comp rx=0 tx=0).stop 2022-09-10T04:26:58.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 shutdown_connections 2022-09-10T04:26:58.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7c4099010 0x7fa7c409b4c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7d8103880 0x7fa7d80fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7d80fe640 0x7fa7d80fdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 --2- 172.21.15.81:0/328278190 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa7d8100f60 0x7fa7d80fc5f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 >> 172.21.15.81:0/328278190 conn(0x7fa7d80f8160 msgr2=0x7fa7d8100010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 shutdown_connections 2022-09-10T04:26:58.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.439+0000 7fa7dd39d700 1 -- 172.21.15.81:0/328278190 wait complete. 2022-09-10T04:26:58.470 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2022-09-10T04:26:58.471 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-09-10T04:26:58.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 -- 172.21.15.81:0/2929128799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 msgr2=0x7fc5a8101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.704 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/2929128799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a8101210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc598009810 tx=0x7fc598009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:58.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 -- 172.21.15.81:0/2929128799 shutdown_connections 2022-09-10T04:26:58.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/2929128799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a8101750 0x7fc5a8101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/2929128799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a80fc430 0x7fc5a80fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/2929128799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a8101210 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 -- 172.21.15.81:0/2929128799 >> 172.21.15.81:0/2929128799 conn(0x7fc5a80f8160 msgr2=0x7fc5a80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:58.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 -- 172.21.15.81:0/2929128799 shutdown_connections 2022-09-10T04:26:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.697+0000 7fc5ae79d700 1 -- 172.21.15.81:0/2929128799 wait complete. 2022-09-10T04:26:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.698+0000 7fc5ae79d700 1 Processor -- start 2022-09-10T04:26:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.698+0000 7fc5ae79d700 1 -- start start 2022-09-10T04:26:58.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.698+0000 7fc5ae79d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 0x7fc5a8199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.698+0000 7fc5ae79d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.698+0000 7fc5ae79d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a8101750 0x7fc5a81a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ae79d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a81a1910 con 0x7fc5a80fed90 2022-09-10T04:26:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ae79d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a81a1a80 con 0x7fc5a8101750 2022-09-10T04:26:58.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ae79d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a81a1c40 con 0x7fc5a80fc430 2022-09-10T04:26:58.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ad79b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 0x7fc5a8199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5adf9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a8101750 0x7fc5a81a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5acf9a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.712 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ad79b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 0x7fc5a8199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:52020/0 (socket says 172.21.15.81:52020) 2022-09-10T04:26:58.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ad79b700 1 -- 172.21.15.81:0/731585857 learned_addr learned my addr 172.21.15.81:0/731585857 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:58.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5adf9c700 1 -- 172.21.15.81:0/731585857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 msgr2=0x7fc5a8199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5adf9c700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 0x7fc5a8199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.713 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5adf9c700 1 -- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 msgr2=0x7fc5a819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5adf9c700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5adf9c700 1 -- 172.21.15.81:0/731585857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598009470 con 0x7fc5a8101750 2022-09-10T04:26:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5ad79b700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 0x7fc5a8199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.699+0000 7fc5acf9a700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:58.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.700+0000 7fc5adf9c700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a8101750 0x7fc5a81a12c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000ea80 tx=0x7fc5a000ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.700+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a00098c0 con 0x7fc5a8101750 2022-09-10T04:26:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.700+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a81a1f20 con 0x7fc5a8101750 2022-09-10T04:26:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.700+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc5a000ce10 con 0x7fc5a8101750 2022-09-10T04:26:58.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.700+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a81a2340 con 0x7fc5a8101750 2022-09-10T04:26:58.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.700+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0004280 con 0x7fc5a8101750 2022-09-10T04:26:58.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.701+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5840051a0 con 0x7fc5a8101750 2022-09-10T04:26:58.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.703+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc5a000c040 con 0x7fc5a8101750 2022-09-10T04:26:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.704+0000 7fc59e7fc700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc594099090 0x7fc59409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.704+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fc5a0014070 con 0x7fc5a8101750 2022-09-10T04:26:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.705+0000 7fc5ad79b700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc594099090 0x7fc59409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:58.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.705+0000 7fc5ad79b700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc594099090 0x7fc59409b540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc5a819b220 tx=0x7fc5a4005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:58.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.707+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fc5a008f220 con 0x7fc5a8101750 2022-09-10T04:26:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: pgmap v314: 129 pgs: 129 active+clean; 582 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2022-09-10T04:26:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] up:active 2022-09-10T04:26:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-09-10T04:26:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2022-09-10T04:26:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2022-09-10T04:26:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2022-09-10T04:26:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: daemon mds.c is now active in filesystem cephfs as rank 3 2022-09-10T04:26:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2797980960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:58.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] up:active 2022-09-10T04:26:58.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2022-09-10T04:26:58.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-09-10T04:26:58.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: Cluster is now healthy 2022-09-10T04:26:58.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2022-09-10T04:26:58.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2022-09-10T04:26:58.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: daemon mds.l is now active in filesystem cephfs as rank 4 2022-09-10T04:26:58.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:58 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/328278190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: pgmap v314: 129 pgs: 129 active+clean; 582 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2022-09-10T04:26:58.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] up:active 2022-09-10T04:26:58.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-09-10T04:26:58.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2022-09-10T04:26:58.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2022-09-10T04:26:58.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2022-09-10T04:26:58.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: daemon mds.c is now active in filesystem cephfs as rank 3 2022-09-10T04:26:58.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2797980960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:58.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] up:active 2022-09-10T04:26:58.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2022-09-10T04:26:58.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-09-10T04:26:58.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: Cluster is now healthy 2022-09-10T04:26:58.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2022-09-10T04:26:58.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2022-09-10T04:26:58.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: daemon mds.l is now active in filesystem cephfs as rank 4 2022-09-10T04:26:58.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:58 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/328278190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:58.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.896+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc584000bf0 con 0x7fc594099090 2022-09-10T04:26:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: pgmap v314: 129 pgs: 129 active+clean; 582 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2022-09-10T04:26:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] up:active 2022-09-10T04:26:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2022-09-10T04:26:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2022-09-10T04:26:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2022-09-10T04:26:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2022-09-10T04:26:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: daemon mds.c is now active in filesystem cephfs as rank 3 2022-09-10T04:26:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2797980960' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:26:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] up:active 2022-09-10T04:26:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2022-09-10T04:26:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2022-09-10T04:26:58.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: Cluster is now healthy 2022-09-10T04:26:58.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2022-09-10T04:26:58.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2022-09-10T04:26:58.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: daemon mds.l is now active in filesystem cephfs as rank 4 2022-09-10T04:26:58.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:58 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/328278190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:26:58.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.912+0000 7fc59e7fc700 1 -- 172.21.15.81:0/731585857 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422683 (secure 0 0 0) 0x7fc584000bf0 con 0x7fc594099090 2022-09-10T04:26:58.925 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc594099090 msgr2=0x7fc59409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc594099090 0x7fc59409b540 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc5a819b220 tx=0x7fc5a4005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a8101750 msgr2=0x7fc5a81a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:58.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a8101750 0x7fc5a81a12c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5a000ea80 tx=0x7fc5a000ed90 comp rx=0 tx=0).stop 2022-09-10T04:26:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 shutdown_connections 2022-09-10T04:26:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5a8101750 0x7fc5a81a12c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc594099090 0x7fc59409b540 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5a80fc430 0x7fc5a8199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 --2- 172.21.15.81:0/731585857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5a80fed90 0x7fc5a819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:58.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 >> 172.21.15.81:0/731585857 conn(0x7fc5a80f8160 msgr2=0x7fc5a80fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:58.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 shutdown_connections 2022-09-10T04:26:58.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:58.919+0000 7fc5ae79d700 1 -- 172.21.15.81:0/731585857 wait complete. 2022-09-10T04:26:58.932 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2022-09-10T04:26:58.949 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2022-09-10T04:26:57.342929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595675,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":263,"num_write_kb":2968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80960,"kb_used_data":13408,"kb_used_omap":0,"kb_used_meta":67456,"kb_avail":1124860864,"statfs":{"total":1151940427776,"available":1151857524736,"internally_reserved":0,"allocated":13729792,"data_stored":7527079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69074944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2977,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003329"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583240+0000","last_change":"2022-09-10T04:26:32.116469+0000","last_active":"2022-09-10T04:26:36.583240+0000","last_peered":"2022-09-10T04:26:36.583240+0000","last_clean":"2022-09-10T04:26:36.583240+0000","last_became_active":"2022-09-10T04:26:32.116060+0000","last_became_peered":"2022-09-10T04:26:32.116060+0000","last_unstale":"2022-09-10T04:26:36.583240+0000","last_undegraded":"2022-09-10T04:26:36.583240+0000","last_fullsized":"2022-09-10T04:26:36.583240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:22:05.868811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585910+0000","last_change":"2022-09-10T04:26:32.114659+0000","last_active":"2022-09-10T04:26:36.585910+0000","last_peered":"2022-09-10T04:26:36.585910+0000","last_clean":"2022-09-10T04:26:36.585910+0000","last_became_active":"2022-09-10T04:26:32.114381+0000","last_became_peered":"2022-09-10T04:26:32.114381+0000","last_unstale":"2022-09-10T04:26:36.585910+0000","last_undegraded":"2022-09-10T04:26:36.585910+0000","last_fullsized":"2022-09-10T04:26:36.585910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:53:50.267695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585535+0000","last_change":"2022-09-10T04:26:32.117123+0000","last_active":"2022-09-10T04:26:36.585535+0000","last_peered":"2022-09-10T04:26:36.585535+0000","last_clean":"2022-09-10T04:26:36.585535+0000","last_became_active":"2022-09-10T04:26:32.116753+0000","last_became_peered":"2022-09-10T04:26:32.116753+0000","last_unstale":"2022-09-10T04:26:36.585535+0000","last_undegraded":"2022-09-10T04:26:36.585535+0000","last_fullsized":"2022-09-10T04:26:36.585535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:52.713788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589699+0000","last_change":"2022-09-10T04:26:32.120151+0000","last_active":"2022-09-10T04:26:36.589699+0000","last_peered":"2022-09-10T04:26:36.589699+0000","last_clean":"2022-09-10T04:26:36.589699+0000","last_became_active":"2022-09-10T04:26:32.119510+0000","last_became_peered":"2022-09-10T04:26:32.119510+0000","last_unstale":"2022-09-10T04:26:36.589699+0000","last_undegraded":"2022-09-10T04:26:36.589699+0000","last_fullsized":"2022-09-10T04:26:36.589699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:56:23.232054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581285+0000","last_change":"2022-09-10T04:26:32.138820+0000","last_active":"2022-09-10T04:26:36.581285+0000","last_peered":"2022-09-10T04:26:36.581285+0000","last_clean":"2022-09-10T04:26:36.581285+0000","last_became_active":"2022-09-10T04:26:32.138466+0000","last_became_peered":"2022-09-10T04:26:32.138466+0000","last_unstale":"2022-09-10T04:26:36.581285+0000","last_undegraded":"2022-09-10T04:26:36.581285+0000","last_fullsized":"2022-09-10T04:26:36.581285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:16:22.128598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789266+0000","last_change":"2022-09-10T04:26:32.124561+0000","last_active":"2022-09-10T04:26:36.789266+0000","last_peered":"2022-09-10T04:26:36.789266+0000","last_clean":"2022-09-10T04:26:36.789266+0000","last_became_active":"2022-09-10T04:26:32.124263+0000","last_became_peered":"2022-09-10T04:26:32.124263+0000","last_unstale":"2022-09-10T04:26:36.789266+0000","last_undegraded":"2022-09-10T04:26:36.789266+0000","last_fullsized":"2022-09-10T04:26:36.789266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:35:16.940139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585041+0000","last_change":"2022-09-10T04:26:32.122869+0000","last_active":"2022-09-10T04:26:36.585041+0000","last_peered":"2022-09-10T04:26:36.585041+0000","last_clean":"2022-09-10T04:26:36.585041+0000","last_became_active":"2022-09-10T04:26:32.122457+0000","last_became_peered":"2022-09-10T04:26:32.122457+0000","last_unstale":"2022-09-10T04:26:36.585041+0000","last_undegraded":"2022-09-10T04:26:36.585041+0000","last_fullsized":"2022-09-10T04:26:36.585041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:08.945173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581301+0000","last_change":"2022-09-10T04:26:32.146033+0000","last_active":"2022-09-10T04:26:36.581301+0000","last_peered":"2022-09-10T04:26:36.581301+0000","last_clean":"2022-09-10T04:26:36.581301+0000","last_became_active":"2022-09-10T04:26:32.145039+0000","last_became_peered":"2022-09-10T04:26:32.145039+0000","last_unstale":"2022-09-10T04:26:36.581301+0000","last_undegraded":"2022-09-10T04:26:36.581301+0000","last_fullsized":"2022-09-10T04:26:36.581301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:44:45.332855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583717+0000","last_change":"2022-09-10T04:26:32.113787+0000","last_active":"2022-09-10T04:26:36.583717+0000","last_peered":"2022-09-10T04:26:36.583717+0000","last_clean":"2022-09-10T04:26:36.583717+0000","last_became_active":"2022-09-10T04:26:32.113454+0000","last_became_peered":"2022-09-10T04:26:32.113454+0000","last_unstale":"2022-09-10T04:26:36.583717+0000","last_undegraded":"2022-09-10T04:26:36.583717+0000","last_fullsized":"2022-09-10T04:26:36.583717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:11:23.375628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789594+0000","last_change":"2022-09-10T04:26:32.123249+0000","last_active":"2022-09-10T04:26:36.789594+0000","last_peered":"2022-09-10T04:26:36.789594+0000","last_clean":"2022-09-10T04:26:36.789594+0000","last_became_active":"2022-09-10T04:26:32.121836+0000","last_became_peered":"2022-09-10T04:26:32.121836+0000","last_unstale":"2022-09-10T04:26:36.789594+0000","last_undegraded":"2022-09-10T04:26:36.789594+0000","last_fullsized":"2022-09-10T04:26:36.789594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:22.711515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584738+0000","last_change":"2022-09-10T04:26:32.118213+0000","last_active":"2022-09-10T04:26:36.584738+0000","last_peered":"2022-09-10T04:26:36.584738+0000","last_clean":"2022-09-10T04:26:36.584738+0000","last_became_active":"2022-09-10T04:26:32.117863+0000","last_became_peered":"2022-09-10T04:26:32.117863+0000","last_unstale":"2022-09-10T04:26:36.584738+0000","last_undegraded":"2022-09-10T04:26:36.584738+0000","last_fullsized":"2022-09-10T04:26:36.584738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:26:35.632083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589648+0000","last_change":"2022-09-10T04:26:32.127748+0000","last_active":"2022-09-10T04:26:36.589648+0000","last_peered":"2022-09-10T04:26:36.589648+0000","last_clean":"2022-09-10T04:26:36.589648+0000","last_became_active":"2022-09-10T04:26:32.127470+0000","last_became_peered":"2022-09-10T04:26:32.127470+0000","last_unstale":"2022-09-10T04:26:36.589648+0000","last_undegraded":"2022-09-10T04:26:36.589648+0000","last_fullsized":"2022-09-10T04:26:36.589648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:01:58.438614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586289+0000","last_change":"2022-09-10T04:26:32.126378+0000","last_active":"2022-09-10T04:26:36.586289+0000","last_peered":"2022-09-10T04:26:36.586289+0000","last_clean":"2022-09-10T04:26:36.586289+0000","last_became_active":"2022-09-10T04:26:32.125510+0000","last_became_peered":"2022-09-10T04:26:32.125510+0000","last_unstale":"2022-09-10T04:26:36.586289+0000","last_undegraded":"2022-09-10T04:26:36.586289+0000","last_fullsized":"2022-09-10T04:26:36.586289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:41.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.587049+0000","last_change":"2022-09-10T04:26:32.118515+0000","last_active":"2022-09-10T04:26:36.587049+0000","last_peered":"2022-09-10T04:26:36.587049+0000","last_clean":"2022-09-10T04:26:36.587049+0000","last_became_active":"2022-09-10T04:26:32.118163+0000","last_became_peered":"2022-09-10T04:26:32.118163+0000","last_unstale":"2022-09-10T04:26:36.587049+0000","last_undegraded":"2022-09-10T04:26:36.587049+0000","last_fullsized":"2022-09-10T04:26:36.587049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:39:42.245166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.592195+0000","last_change":"2022-09-10T04:26:32.122232+0000","last_active":"2022-09-10T04:26:36.592195+0000","last_peered":"2022-09-10T04:26:36.592195+0000","last_clean":"2022-09-10T04:26:36.592195+0000","last_became_active":"2022-09-10T04:26:32.121906+0000","last_became_peered":"2022-09-10T04:26:32.121906+0000","last_unstale":"2022-09-10T04:26:36.592195+0000","last_undegraded":"2022-09-10T04:26:36.592195+0000","last_fullsized":"2022-09-10T04:26:36.592195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:49:50.711773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583704+0000","last_change":"2022-09-10T04:26:32.117413+0000","last_active":"2022-09-10T04:26:36.583704+0000","last_peered":"2022-09-10T04:26:36.583704+0000","last_clean":"2022-09-10T04:26:36.583704+0000","last_became_active":"2022-09-10T04:26:32.117120+0000","last_became_peered":"2022-09-10T04:26:32.117120+0000","last_unstale":"2022-09-10T04:26:36.583704+0000","last_undegraded":"2022-09-10T04:26:36.583704+0000","last_fullsized":"2022-09-10T04:26:36.583704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:50:28.426517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583151+0000","last_change":"2022-09-10T04:26:32.117852+0000","last_active":"2022-09-10T04:26:36.583151+0000","last_peered":"2022-09-10T04:26:36.583151+0000","last_clean":"2022-09-10T04:26:36.583151+0000","last_became_active":"2022-09-10T04:26:32.117526+0000","last_became_peered":"2022-09-10T04:26:32.117526+0000","last_unstale":"2022-09-10T04:26:36.583151+0000","last_undegraded":"2022-09-10T04:26:36.583151+0000","last_fullsized":"2022-09-10T04:26:36.583151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:02:45.194870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582629+0000","last_change":"2022-09-10T04:26:32.122937+0000","last_active":"2022-09-10T04:26:36.582629+0000","last_peered":"2022-09-10T04:26:36.582629+0000","last_clean":"2022-09-10T04:26:36.582629+0000","last_became_active":"2022-09-10T04:26:32.122549+0000","last_became_peered":"2022-09-10T04:26:32.122549+0000","last_unstale":"2022-09-10T04:26:36.582629+0000","last_undegraded":"2022-09-10T04:26:36.582629+0000","last_fullsized":"2022-09-10T04:26:36.582629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:19:48.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577863+0000","last_change":"2022-09-10T04:26:32.139841+0000","last_active":"2022-09-10T04:26:36.577863+0000","last_peered":"2022-09-10T04:26:36.577863+0000","last_clean":"2022-09-10T04:26:36.577863+0000","last_became_active":"2022-09-10T04:26:32.138962+0000","last_became_peered":"2022-09-10T04:26:32.138962+0000","last_unstale":"2022-09-10T04:26:36.577863+0000","last_undegraded":"2022-09-10T04:26:36.577863+0000","last_fullsized":"2022-09-10T04:26:36.577863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:51.196716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.243899+0000","last_change":"2022-09-10T04:26:36.594956+0000","last_active":"2022-09-10T04:26:37.243899+0000","last_peered":"2022-09-10T04:26:37.243899+0000","last_clean":"2022-09-10T04:26:37.243899+0000","last_became_active":"2022-09-10T04:26:36.594507+0000","last_became_peered":"2022-09-10T04:26:36.594507+0000","last_unstale":"2022-09-10T04:26:37.243899+0000","last_undegraded":"2022-09-10T04:26:37.243899+0000","last_fullsized":"2022-09-10T04:26:37.243899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:05:36.037017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580745+0000","last_change":"2022-09-10T04:26:31.553482+0000","last_active":"2022-09-10T04:26:36.580745+0000","last_peered":"2022-09-10T04:26:36.580745+0000","last_clean":"2022-09-10T04:26:36.580745+0000","last_became_active":"2022-09-10T04:26:31.553170+0000","last_became_peered":"2022-09-10T04:26:31.553170+0000","last_unstale":"2022-09-10T04:26:36.580745+0000","last_undegraded":"2022-09-10T04:26:36.580745+0000","last_fullsized":"2022-09-10T04:26:36.580745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:52:35.054369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580763+0000","last_change":"2022-09-10T04:26:32.139920+0000","last_active":"2022-09-10T04:26:36.580763+0000","last_peered":"2022-09-10T04:26:36.580763+0000","last_clean":"2022-09-10T04:26:36.580763+0000","last_became_active":"2022-09-10T04:26:32.139414+0000","last_became_peered":"2022-09-10T04:26:32.139414+0000","last_unstale":"2022-09-10T04:26:36.580763+0000","last_undegraded":"2022-09-10T04:26:36.580763+0000","last_fullsized":"2022-09-10T04:26:36.580763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:12.288511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119112+0000","last_change":"2022-09-10T04:26:36.603332+0000","last_active":"2022-09-10T04:26:37.119112+0000","last_peered":"2022-09-10T04:26:37.119112+0000","last_clean":"2022-09-10T04:26:37.119112+0000","last_became_active":"2022-09-10T04:26:36.602947+0000","last_became_peered":"2022-09-10T04:26:36.602947+0000","last_unstale":"2022-09-10T04:26:37.119112+0000","last_undegraded":"2022-09-10T04:26:37.119112+0000","last_fullsized":"2022-09-10T04:26:37.119112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:20:15.790086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581013+0000","last_change":"2022-09-10T04:26:31.559001+0000","last_active":"2022-09-10T04:26:36.581013+0000","last_peered":"2022-09-10T04:26:36.581013+0000","last_clean":"2022-09-10T04:26:36.581013+0000","last_became_active":"2022-09-10T04:26:31.558675+0000","last_became_peered":"2022-09-10T04:26:31.558675+0000","last_unstale":"2022-09-10T04:26:36.581013+0000","last_undegraded":"2022-09-10T04:26:36.581013+0000","last_fullsized":"2022-09-10T04:26:36.581013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:31:20.091765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.596004+0000","last_change":"2022-09-10T04:26:32.122545+0000","last_active":"2022-09-10T04:26:36.596004+0000","last_peered":"2022-09-10T04:26:36.596004+0000","last_clean":"2022-09-10T04:26:36.596004+0000","last_became_active":"2022-09-10T04:26:32.122237+0000","last_became_peered":"2022-09-10T04:26:32.122237+0000","last_unstale":"2022-09-10T04:26:36.596004+0000","last_undegraded":"2022-09-10T04:26:36.596004+0000","last_fullsized":"2022-09-10T04:26:36.596004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:16:01.235524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119323+0000","last_change":"2022-09-10T04:26:36.603544+0000","last_active":"2022-09-10T04:26:37.119323+0000","last_peered":"2022-09-10T04:26:37.119323+0000","last_clean":"2022-09-10T04:26:37.119323+0000","last_became_active":"2022-09-10T04:26:36.602905+0000","last_became_peered":"2022-09-10T04:26:36.602905+0000","last_unstale":"2022-09-10T04:26:37.119323+0000","last_undegraded":"2022-09-10T04:26:37.119323+0000","last_fullsized":"2022-09-10T04:26:37.119323+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:17:44.678651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580890+0000","last_change":"2022-09-10T04:26:31.551550+0000","last_active":"2022-09-10T04:26:36.580890+0000","last_peered":"2022-09-10T04:26:36.580890+0000","last_clean":"2022-09-10T04:26:36.580890+0000","last_became_active":"2022-09-10T04:26:31.551161+0000","last_became_peered":"2022-09-10T04:26:31.551161+0000","last_unstale":"2022-09-10T04:26:36.580890+0000","last_undegraded":"2022-09-10T04:26:36.580890+0000","last_fullsized":"2022-09-10T04:26:36.580890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:59:32.430914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582594+0000","last_change":"2022-09-10T04:26:32.122935+0000","last_active":"2022-09-10T04:26:36.582594+0000","last_peered":"2022-09-10T04:26:36.582594+0000","last_clean":"2022-09-10T04:26:36.582594+0000","last_became_active":"2022-09-10T04:26:32.122471+0000","last_became_peered":"2022-09-10T04:26:32.122471+0000","last_unstale":"2022-09-10T04:26:36.582594+0000","last_undegraded":"2022-09-10T04:26:36.582594+0000","last_fullsized":"2022-09-10T04:26:36.582594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:29:29.965140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119344+0000","last_change":"2022-09-10T04:26:36.604924+0000","last_active":"2022-09-10T04:26:37.119344+0000","last_peered":"2022-09-10T04:26:37.119344+0000","last_clean":"2022-09-10T04:26:37.119344+0000","last_became_active":"2022-09-10T04:26:36.604620+0000","last_became_peered":"2022-09-10T04:26:36.604620+0000","last_unstale":"2022-09-10T04:26:37.119344+0000","last_undegraded":"2022-09-10T04:26:37.119344+0000","last_fullsized":"2022-09-10T04:26:37.119344+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:41:34.668107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581512+0000","last_change":"2022-09-10T04:26:31.547564+0000","last_active":"2022-09-10T04:26:36.581512+0000","last_peered":"2022-09-10T04:26:36.581512+0000","last_clean":"2022-09-10T04:26:36.581512+0000","last_became_active":"2022-09-10T04:26:31.547180+0000","last_became_peered":"2022-09-10T04:26:31.547180+0000","last_unstale":"2022-09-10T04:26:36.581512+0000","last_undegraded":"2022-09-10T04:26:36.581512+0000","last_fullsized":"2022-09-10T04:26:36.581512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:23:12.454750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582914+0000","last_change":"2022-09-10T04:26:32.119307+0000","last_active":"2022-09-10T04:26:36.582914+0000","last_peered":"2022-09-10T04:26:36.582914+0000","last_clean":"2022-09-10T04:26:36.582914+0000","last_became_active":"2022-09-10T04:26:32.118995+0000","last_became_peered":"2022-09-10T04:26:32.118995+0000","last_unstale":"2022-09-10T04:26:36.582914+0000","last_undegraded":"2022-09-10T04:26:36.582914+0000","last_fullsized":"2022-09-10T04:26:36.582914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:25.888725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.244142+0000","last_change":"2022-09-10T04:26:36.591310+0000","last_active":"2022-09-10T04:26:37.244142+0000","last_peered":"2022-09-10T04:26:37.244142+0000","last_clean":"2022-09-10T04:26:37.244142+0000","last_became_active":"2022-09-10T04:26:36.590907+0000","last_became_peered":"2022-09-10T04:26:36.590907+0000","last_unstale":"2022-09-10T04:26:37.244142+0000","last_undegraded":"2022-09-10T04:26:37.244142+0000","last_fullsized":"2022-09-10T04:26:37.244142+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:05:45.016925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582839+0000","last_change":"2022-09-10T04:26:31.550524+0000","last_active":"2022-09-10T04:26:36.582839+0000","last_peered":"2022-09-10T04:26:36.582839+0000","last_clean":"2022-09-10T04:26:36.582839+0000","last_became_active":"2022-09-10T04:26:31.550259+0000","last_became_peered":"2022-09-10T04:26:31.550259+0000","last_unstale":"2022-09-10T04:26:36.582839+0000","last_undegraded":"2022-09-10T04:26:36.582839+0000","last_fullsized":"2022-09-10T04:26:36.582839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:53.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585280+0000","last_change":"2022-09-10T04:26:32.123377+0000","last_active":"2022-09-10T04:26:36.585280+0000","last_peered":"2022-09-10T04:26:36.585280+0000","last_clean":"2022-09-10T04:26:36.585280+0000","last_became_active":"2022-09-10T04:26:32.123076+0000","last_became_peered":"2022-09-10T04:26:32.123076+0000","last_unstale":"2022-09-10T04:26:36.585280+0000","last_undegraded":"2022-09-10T04:26:36.585280+0000","last_fullsized":"2022-09-10T04:26:36.585280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:40.425861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.244104+0000","last_change":"2022-09-10T04:26:36.593491+0000","last_active":"2022-09-10T04:26:37.244104+0000","last_peered":"2022-09-10T04:26:37.244104+0000","last_clean":"2022-09-10T04:26:37.244104+0000","last_became_active":"2022-09-10T04:26:36.593064+0000","last_became_peered":"2022-09-10T04:26:36.593064+0000","last_unstale":"2022-09-10T04:26:37.244104+0000","last_undegraded":"2022-09-10T04:26:37.244104+0000","last_fullsized":"2022-09-10T04:26:37.244104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:03:44.752739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789087+0000","last_change":"2022-09-10T04:26:31.555492+0000","last_active":"2022-09-10T04:26:36.789087+0000","last_peered":"2022-09-10T04:26:36.789087+0000","last_clean":"2022-09-10T04:26:36.789087+0000","last_became_active":"2022-09-10T04:26:31.555216+0000","last_became_peered":"2022-09-10T04:26:31.555216+0000","last_unstale":"2022-09-10T04:26:36.789087+0000","last_undegraded":"2022-09-10T04:26:36.789087+0000","last_fullsized":"2022-09-10T04:26:36.789087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:50:32.775631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582937+0000","last_change":"2022-09-10T04:26:32.139851+0000","last_active":"2022-09-10T04:26:36.582937+0000","last_peered":"2022-09-10T04:26:36.582937+0000","last_clean":"2022-09-10T04:26:36.582937+0000","last_became_active":"2022-09-10T04:26:32.139515+0000","last_became_peered":"2022-09-10T04:26:32.139515+0000","last_unstale":"2022-09-10T04:26:36.582937+0000","last_undegraded":"2022-09-10T04:26:36.582937+0000","last_fullsized":"2022-09-10T04:26:36.582937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:46:04.342743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.100507+0000","last_change":"2022-09-10T04:26:36.598406+0000","last_active":"2022-09-10T04:26:38.100507+0000","last_peered":"2022-09-10T04:26:38.100507+0000","last_clean":"2022-09-10T04:26:38.100507+0000","last_became_active":"2022-09-10T04:26:36.598118+0000","last_became_peered":"2022-09-10T04:26:36.598118+0000","last_unstale":"2022-09-10T04:26:38.100507+0000","last_undegraded":"2022-09-10T04:26:38.100507+0000","last_fullsized":"2022-09-10T04:26:38.100507+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:35.903423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.461177+0000","last_change":"2022-09-10T04:26:31.541593+0000","last_active":"2022-09-10T04:26:56.461177+0000","last_peered":"2022-09-10T04:26:56.461177+0000","last_clean":"2022-09-10T04:26:56.461177+0000","last_became_active":"2022-09-10T04:26:31.541236+0000","last_became_peered":"2022-09-10T04:26:31.541236+0000","last_unstale":"2022-09-10T04:26:56.461177+0000","last_undegraded":"2022-09-10T04:26:56.461177+0000","last_fullsized":"2022-09-10T04:26:56.461177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:30:03.723483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581002+0000","last_change":"2022-09-10T04:26:32.147698+0000","last_active":"2022-09-10T04:26:36.581002+0000","last_peered":"2022-09-10T04:26:36.581002+0000","last_clean":"2022-09-10T04:26:36.581002+0000","last_became_active":"2022-09-10T04:26:32.147334+0000","last_became_peered":"2022-09-10T04:26:32.147334+0000","last_unstale":"2022-09-10T04:26:36.581002+0000","last_undegraded":"2022-09-10T04:26:36.581002+0000","last_fullsized":"2022-09-10T04:26:36.581002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:33:21.907371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.988827+0000","last_change":"2022-09-10T04:26:36.597450+0000","last_active":"2022-09-10T04:26:38.988827+0000","last_peered":"2022-09-10T04:26:38.988827+0000","last_clean":"2022-09-10T04:26:38.988827+0000","last_became_active":"2022-09-10T04:26:36.597087+0000","last_became_peered":"2022-09-10T04:26:36.597087+0000","last_unstale":"2022-09-10T04:26:38.988827+0000","last_undegraded":"2022-09-10T04:26:38.988827+0000","last_fullsized":"2022-09-10T04:26:38.988827+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:27.788857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583682+0000","last_change":"2022-09-10T04:26:31.550451+0000","last_active":"2022-09-10T04:26:36.583682+0000","last_peered":"2022-09-10T04:26:36.583682+0000","last_clean":"2022-09-10T04:26:36.583682+0000","last_became_active":"2022-09-10T04:26:31.550158+0000","last_became_peered":"2022-09-10T04:26:31.550158+0000","last_unstale":"2022-09-10T04:26:36.583682+0000","last_undegraded":"2022-09-10T04:26:36.583682+0000","last_fullsized":"2022-09-10T04:26:36.583682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:21:27.132856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580285+0000","last_change":"2022-09-10T04:26:32.136694+0000","last_active":"2022-09-10T04:26:36.580285+0000","last_peered":"2022-09-10T04:26:36.580285+0000","last_clean":"2022-09-10T04:26:36.580285+0000","last_became_active":"2022-09-10T04:26:32.136398+0000","last_became_peered":"2022-09-10T04:26:32.136398+0000","last_unstale":"2022-09-10T04:26:36.580285+0000","last_undegraded":"2022-09-10T04:26:36.580285+0000","last_fullsized":"2022-09-10T04:26:36.580285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:32:22.413618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.602343+0000","last_change":"2022-09-10T04:26:36.599433+0000","last_active":"2022-09-10T04:26:36.602343+0000","last_peered":"2022-09-10T04:26:36.602343+0000","last_clean":"2022-09-10T04:26:36.602343+0000","last_became_active":"2022-09-10T04:26:36.599013+0000","last_became_peered":"2022-09-10T04:26:36.599013+0000","last_unstale":"2022-09-10T04:26:36.602343+0000","last_undegraded":"2022-09-10T04:26:36.602343+0000","last_fullsized":"2022-09-10T04:26:36.602343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:23:18.397219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789436+0000","last_change":"2022-09-10T04:26:31.548631+0000","last_active":"2022-09-10T04:26:36.789436+0000","last_peered":"2022-09-10T04:26:36.789436+0000","last_clean":"2022-09-10T04:26:36.789436+0000","last_became_active":"2022-09-10T04:26:31.548292+0000","last_became_peered":"2022-09-10T04:26:31.548292+0000","last_unstale":"2022-09-10T04:26:36.789436+0000","last_undegraded":"2022-09-10T04:26:36.789436+0000","last_fullsized":"2022-09-10T04:26:36.789436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:40.825422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583168+0000","last_change":"2022-09-10T04:26:32.124469+0000","last_active":"2022-09-10T04:26:36.583168+0000","last_peered":"2022-09-10T04:26:36.583168+0000","last_clean":"2022-09-10T04:26:36.583168+0000","last_became_active":"2022-09-10T04:26:32.124084+0000","last_became_peered":"2022-09-10T04:26:32.124084+0000","last_unstale":"2022-09-10T04:26:36.583168+0000","last_undegraded":"2022-09-10T04:26:36.583168+0000","last_fullsized":"2022-09-10T04:26:36.583168+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:39:04.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420161+0000","last_change":"2022-09-10T04:26:36.601607+0000","last_active":"2022-09-10T04:26:38.420161+0000","last_peered":"2022-09-10T04:26:38.420161+0000","last_clean":"2022-09-10T04:26:38.420161+0000","last_became_active":"2022-09-10T04:26:36.601191+0000","last_became_peered":"2022-09-10T04:26:36.601191+0000","last_unstale":"2022-09-10T04:26:38.420161+0000","last_undegraded":"2022-09-10T04:26:38.420161+0000","last_fullsized":"2022-09-10T04:26:38.420161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:52:58.815905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.455549+0000","last_change":"2022-09-10T04:26:31.548524+0000","last_active":"2022-09-10T04:26:55.455549+0000","last_peered":"2022-09-10T04:26:55.455549+0000","last_clean":"2022-09-10T04:26:55.455549+0000","last_became_active":"2022-09-10T04:26:31.548073+0000","last_became_peered":"2022-09-10T04:26:31.548073+0000","last_unstale":"2022-09-10T04:26:55.455549+0000","last_undegraded":"2022-09-10T04:26:55.455549+0000","last_fullsized":"2022-09-10T04:26:55.455549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:01:02.489915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.596090+0000","last_change":"2022-09-10T04:26:32.120523+0000","last_active":"2022-09-10T04:26:36.596090+0000","last_peered":"2022-09-10T04:26:36.596090+0000","last_clean":"2022-09-10T04:26:36.596090+0000","last_became_active":"2022-09-10T04:26:32.120076+0000","last_became_peered":"2022-09-10T04:26:32.120076+0000","last_unstale":"2022-09-10T04:26:36.596090+0000","last_undegraded":"2022-09-10T04:26:36.596090+0000","last_fullsized":"2022-09-10T04:26:36.596090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:26:07.907475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119606+0000","last_change":"2022-09-10T04:26:36.601632+0000","last_active":"2022-09-10T04:26:37.119606+0000","last_peered":"2022-09-10T04:26:37.119606+0000","last_clean":"2022-09-10T04:26:37.119606+0000","last_became_active":"2022-09-10T04:26:36.601237+0000","last_became_peered":"2022-09-10T04:26:36.601237+0000","last_unstale":"2022-09-10T04:26:37.119606+0000","last_undegraded":"2022-09-10T04:26:37.119606+0000","last_fullsized":"2022-09-10T04:26:37.119606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:08:34.270892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582804+0000","last_change":"2022-09-10T04:26:31.557753+0000","last_active":"2022-09-10T04:26:36.582804+0000","last_peered":"2022-09-10T04:26:36.582804+0000","last_clean":"2022-09-10T04:26:36.582804+0000","last_became_active":"2022-09-10T04:26:31.557432+0000","last_became_peered":"2022-09-10T04:26:31.557432+0000","last_unstale":"2022-09-10T04:26:36.582804+0000","last_undegraded":"2022-09-10T04:26:36.582804+0000","last_fullsized":"2022-09-10T04:26:36.582804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:14:51.954481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582777+0000","last_change":"2022-09-10T04:26:32.122753+0000","last_active":"2022-09-10T04:26:36.582777+0000","last_peered":"2022-09-10T04:26:36.582777+0000","last_clean":"2022-09-10T04:26:36.582777+0000","last_became_active":"2022-09-10T04:26:32.122052+0000","last_became_peered":"2022-09-10T04:26:32.122052+0000","last_unstale":"2022-09-10T04:26:36.582777+0000","last_undegraded":"2022-09-10T04:26:36.582777+0000","last_fullsized":"2022-09-10T04:26:36.582777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:00:07.053739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420184+0000","last_change":"2022-09-10T04:26:36.591159+0000","last_active":"2022-09-10T04:26:38.420184+0000","last_peered":"2022-09-10T04:26:38.420184+0000","last_clean":"2022-09-10T04:26:38.420184+0000","last_became_active":"2022-09-10T04:26:36.590788+0000","last_became_peered":"2022-09-10T04:26:36.590788+0000","last_unstale":"2022-09-10T04:26:38.420184+0000","last_undegraded":"2022-09-10T04:26:38.420184+0000","last_fullsized":"2022-09-10T04:26:38.420184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:50:34.597783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:52.513644+0000","last_change":"2022-09-10T04:26:31.547702+0000","last_active":"2022-09-10T04:26:52.513644+0000","last_peered":"2022-09-10T04:26:52.513644+0000","last_clean":"2022-09-10T04:26:52.513644+0000","last_became_active":"2022-09-10T04:26:31.547388+0000","last_became_peered":"2022-09-10T04:26:31.547388+0000","last_unstale":"2022-09-10T04:26:52.513644+0000","last_undegraded":"2022-09-10T04:26:52.513644+0000","last_fullsized":"2022-09-10T04:26:52.513644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:42:28.378515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3749,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580578+0000","last_change":"2022-09-10T04:26:32.117260+0000","last_active":"2022-09-10T04:26:36.580578+0000","last_peered":"2022-09-10T04:26:36.580578+0000","last_clean":"2022-09-10T04:26:36.580578+0000","last_became_active":"2022-09-10T04:26:32.116928+0000","last_became_peered":"2022-09-10T04:26:32.116928+0000","last_unstale":"2022-09-10T04:26:36.580578+0000","last_undegraded":"2022-09-10T04:26:36.580578+0000","last_fullsized":"2022-09-10T04:26:36.580578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:00:10.528509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.288747+0000","last_change":"2022-09-10T04:26:36.600239+0000","last_active":"2022-09-10T04:26:37.288747+0000","last_peered":"2022-09-10T04:26:37.288747+0000","last_clean":"2022-09-10T04:26:37.288747+0000","last_became_active":"2022-09-10T04:26:36.599614+0000","last_became_peered":"2022-09-10T04:26:36.599614+0000","last_unstale":"2022-09-10T04:26:37.288747+0000","last_undegraded":"2022-09-10T04:26:37.288747+0000","last_fullsized":"2022-09-10T04:26:37.288747+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:36.535130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577343+0000","last_change":"2022-09-10T04:26:31.552412+0000","last_active":"2022-09-10T04:26:36.577343+0000","last_peered":"2022-09-10T04:26:36.577343+0000","last_clean":"2022-09-10T04:26:36.577343+0000","last_became_active":"2022-09-10T04:26:31.552145+0000","last_became_peered":"2022-09-10T04:26:31.552145+0000","last_unstale":"2022-09-10T04:26:36.577343+0000","last_undegraded":"2022-09-10T04:26:36.577343+0000","last_fullsized":"2022-09-10T04:26:36.577343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:16:36.041950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585882+0000","last_change":"2022-09-10T04:26:32.136276+0000","last_active":"2022-09-10T04:26:36.585882+0000","last_peered":"2022-09-10T04:26:36.585882+0000","last_clean":"2022-09-10T04:26:36.585882+0000","last_became_active":"2022-09-10T04:26:32.135894+0000","last_became_peered":"2022-09-10T04:26:32.135894+0000","last_unstale":"2022-09-10T04:26:36.585882+0000","last_undegraded":"2022-09-10T04:26:36.585882+0000","last_fullsized":"2022-09-10T04:26:36.585882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:12:32.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.100262+0000","last_change":"2022-09-10T04:26:36.600069+0000","last_active":"2022-09-10T04:26:38.100262+0000","last_peered":"2022-09-10T04:26:38.100262+0000","last_clean":"2022-09-10T04:26:38.100262+0000","last_became_active":"2022-09-10T04:26:36.599592+0000","last_became_peered":"2022-09-10T04:26:36.599592+0000","last_unstale":"2022-09-10T04:26:38.100262+0000","last_undegraded":"2022-09-10T04:26:38.100262+0000","last_fullsized":"2022-09-10T04:26:38.100262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:25:58.722442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582707+0000","last_change":"2022-09-10T04:26:31.556823+0000","last_active":"2022-09-10T04:26:36.582707+0000","last_peered":"2022-09-10T04:26:36.582707+0000","last_clean":"2022-09-10T04:26:36.582707+0000","last_became_active":"2022-09-10T04:26:31.556509+0000","last_became_peered":"2022-09-10T04:26:31.556509+0000","last_unstale":"2022-09-10T04:26:36.582707+0000","last_undegraded":"2022-09-10T04:26:36.582707+0000","last_fullsized":"2022-09-10T04:26:36.582707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:35:16.227863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585116+0000","last_change":"2022-09-10T04:26:32.130565+0000","last_active":"2022-09-10T04:26:36.585116+0000","last_peered":"2022-09-10T04:26:36.585116+0000","last_clean":"2022-09-10T04:26:36.585116+0000","last_became_active":"2022-09-10T04:26:32.130121+0000","last_became_peered":"2022-09-10T04:26:32.130121+0000","last_unstale":"2022-09-10T04:26:36.585116+0000","last_undegraded":"2022-09-10T04:26:36.585116+0000","last_fullsized":"2022-09-10T04:26:36.585116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:55:03.504615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420227+0000","last_change":"2022-09-10T04:26:36.603262+0000","last_active":"2022-09-10T04:26:38.420227+0000","last_peered":"2022-09-10T04:26:38.420227+0000","last_clean":"2022-09-10T04:26:38.420227+0000","last_became_active":"2022-09-10T04:26:36.602590+0000","last_became_peered":"2022-09-10T04:26:36.602590+0000","last_unstale":"2022-09-10T04:26:38.420227+0000","last_undegraded":"2022-09-10T04:26:38.420227+0000","last_fullsized":"2022-09-10T04:26:38.420227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:19:03.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.459902+0000","last_change":"2022-09-10T04:26:31.545898+0000","last_active":"2022-09-10T04:26:56.459902+0000","last_peered":"2022-09-10T04:26:56.459902+0000","last_clean":"2022-09-10T04:26:56.459902+0000","last_became_active":"2022-09-10T04:26:31.545355+0000","last_became_peered":"2022-09-10T04:26:31.545355+0000","last_unstale":"2022-09-10T04:26:56.459902+0000","last_undegraded":"2022-09-10T04:26:56.459902+0000","last_fullsized":"2022-09-10T04:26:56.459902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:50:36.187534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585167+0000","last_change":"2022-09-10T04:26:32.116268+0000","last_active":"2022-09-10T04:26:36.585167+0000","last_peered":"2022-09-10T04:26:36.585167+0000","last_clean":"2022-09-10T04:26:36.585167+0000","last_became_active":"2022-09-10T04:26:32.115133+0000","last_became_peered":"2022-09-10T04:26:32.115133+0000","last_unstale":"2022-09-10T04:26:36.585167+0000","last_undegraded":"2022-09-10T04:26:36.585167+0000","last_fullsized":"2022-09-10T04:26:36.585167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:38:05.319082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.606364+0000","last_change":"2022-09-10T04:26:36.605212+0000","last_active":"2022-09-10T04:26:36.606364+0000","last_peered":"2022-09-10T04:26:36.606364+0000","last_clean":"2022-09-10T04:26:36.606364+0000","last_became_active":"2022-09-10T04:26:36.604767+0000","last_became_peered":"2022-09-10T04:26:36.604767+0000","last_unstale":"2022-09-10T04:26:36.606364+0000","last_undegraded":"2022-09-10T04:26:36.606364+0000","last_fullsized":"2022-09-10T04:26:36.606364+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:33:55.048936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.454477+0000","last_change":"2022-09-10T04:26:31.561097+0000","last_active":"2022-09-10T04:26:55.454477+0000","last_peered":"2022-09-10T04:26:55.454477+0000","last_clean":"2022-09-10T04:26:55.454477+0000","last_became_active":"2022-09-10T04:26:31.560748+0000","last_became_peered":"2022-09-10T04:26:31.560748+0000","last_unstale":"2022-09-10T04:26:55.454477+0000","last_undegraded":"2022-09-10T04:26:55.454477+0000","last_fullsized":"2022-09-10T04:26:55.454477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:44:32.921350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582638+0000","last_change":"2022-09-10T04:26:32.122012+0000","last_active":"2022-09-10T04:26:36.582638+0000","last_peered":"2022-09-10T04:26:36.582638+0000","last_clean":"2022-09-10T04:26:36.582638+0000","last_became_active":"2022-09-10T04:26:32.121668+0000","last_became_peered":"2022-09-10T04:26:32.121668+0000","last_unstale":"2022-09-10T04:26:36.582638+0000","last_undegraded":"2022-09-10T04:26:36.582638+0000","last_fullsized":"2022-09-10T04:26:36.582638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:49:57.312550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.288088+0000","last_change":"2022-09-10T04:26:36.594964+0000","last_active":"2022-09-10T04:26:37.288088+0000","last_peered":"2022-09-10T04:26:37.288088+0000","last_clean":"2022-09-10T04:26:37.288088+0000","last_became_active":"2022-09-10T04:26:36.594548+0000","last_became_peered":"2022-09-10T04:26:36.594548+0000","last_unstale":"2022-09-10T04:26:37.288088+0000","last_undegraded":"2022-09-10T04:26:37.288088+0000","last_fullsized":"2022-09-10T04:26:37.288088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:01.446152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580676+0000","last_change":"2022-09-10T04:26:31.549608+0000","last_active":"2022-09-10T04:26:36.580676+0000","last_peered":"2022-09-10T04:26:36.580676+0000","last_clean":"2022-09-10T04:26:36.580676+0000","last_became_active":"2022-09-10T04:26:31.549300+0000","last_became_peered":"2022-09-10T04:26:31.549300+0000","last_unstale":"2022-09-10T04:26:36.580676+0000","last_undegraded":"2022-09-10T04:26:36.580676+0000","last_fullsized":"2022-09-10T04:26:36.580676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:53:51.183777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582208+0000","last_change":"2022-09-10T04:26:32.119540+0000","last_active":"2022-09-10T04:26:36.582208+0000","last_peered":"2022-09-10T04:26:36.582208+0000","last_clean":"2022-09-10T04:26:36.582208+0000","last_became_active":"2022-09-10T04:26:32.119341+0000","last_became_peered":"2022-09-10T04:26:32.119341+0000","last_unstale":"2022-09-10T04:26:36.582208+0000","last_undegraded":"2022-09-10T04:26:36.582208+0000","last_fullsized":"2022-09-10T04:26:36.582208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:21:13.815142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.791499+0000","last_change":"2022-09-10T04:26:36.790770+0000","last_active":"2022-09-10T04:26:36.791499+0000","last_peered":"2022-09-10T04:26:36.791499+0000","last_clean":"2022-09-10T04:26:36.791499+0000","last_became_active":"2022-09-10T04:26:36.790366+0000","last_became_peered":"2022-09-10T04:26:36.790366+0000","last_unstale":"2022-09-10T04:26:36.791499+0000","last_undegraded":"2022-09-10T04:26:36.791499+0000","last_fullsized":"2022-09-10T04:26:36.791499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:40:18.004852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580385+0000","last_change":"2022-09-10T04:26:31.557145+0000","last_active":"2022-09-10T04:26:36.580385+0000","last_peered":"2022-09-10T04:26:36.580385+0000","last_clean":"2022-09-10T04:26:36.580385+0000","last_became_active":"2022-09-10T04:26:31.556805+0000","last_became_peered":"2022-09-10T04:26:31.556805+0000","last_unstale":"2022-09-10T04:26:36.580385+0000","last_undegraded":"2022-09-10T04:26:36.580385+0000","last_fullsized":"2022-09-10T04:26:36.580385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:53:41.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789300+0000","last_change":"2022-09-10T04:26:32.139758+0000","last_active":"2022-09-10T04:26:36.789300+0000","last_peered":"2022-09-10T04:26:36.789300+0000","last_clean":"2022-09-10T04:26:36.789300+0000","last_became_active":"2022-09-10T04:26:32.139469+0000","last_became_peered":"2022-09-10T04:26:32.139469+0000","last_unstale":"2022-09-10T04:26:36.789300+0000","last_undegraded":"2022-09-10T04:26:36.789300+0000","last_fullsized":"2022-09-10T04:26:36.789300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:09:52.224744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.606481+0000","last_change":"2022-09-10T04:26:36.600083+0000","last_active":"2022-09-10T04:26:36.606481+0000","last_peered":"2022-09-10T04:26:36.606481+0000","last_clean":"2022-09-10T04:26:36.606481+0000","last_became_active":"2022-09-10T04:26:36.599648+0000","last_became_peered":"2022-09-10T04:26:36.599648+0000","last_unstale":"2022-09-10T04:26:36.606481+0000","last_undegraded":"2022-09-10T04:26:36.606481+0000","last_fullsized":"2022-09-10T04:26:36.606481+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:03:24.897645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.231607+0000","last_change":"2022-09-10T04:26:31.560147+0000","last_active":"2022-09-10T04:26:38.231607+0000","last_peered":"2022-09-10T04:26:38.231607+0000","last_clean":"2022-09-10T04:26:38.231607+0000","last_became_active":"2022-09-10T04:26:31.559868+0000","last_became_peered":"2022-09-10T04:26:31.559868+0000","last_unstale":"2022-09-10T04:26:38.231607+0000","last_undegraded":"2022-09-10T04:26:38.231607+0000","last_fullsized":"2022-09-10T04:26:38.231607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:02:46.652743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789118+0000","last_change":"2022-09-10T04:26:32.125369+0000","last_active":"2022-09-10T04:26:36.789118+0000","last_peered":"2022-09-10T04:26:36.789118+0000","last_clean":"2022-09-10T04:26:36.789118+0000","last_became_active":"2022-09-10T04:26:32.125077+0000","last_became_peered":"2022-09-10T04:26:32.125077+0000","last_unstale":"2022-09-10T04:26:36.789118+0000","last_undegraded":"2022-09-10T04:26:36.789118+0000","last_fullsized":"2022-09-10T04:26:36.789118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:03:13.265143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.791633+0000","last_change":"2022-09-10T04:26:36.790278+0000","last_active":"2022-09-10T04:26:36.791633+0000","last_peered":"2022-09-10T04:26:36.791633+0000","last_clean":"2022-09-10T04:26:36.791633+0000","last_became_active":"2022-09-10T04:26:36.789852+0000","last_became_peered":"2022-09-10T04:26:36.789852+0000","last_unstale":"2022-09-10T04:26:36.791633+0000","last_undegraded":"2022-09-10T04:26:36.791633+0000","last_fullsized":"2022-09-10T04:26:36.791633+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:26:38.960022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583882+0000","last_change":"2022-09-10T04:26:32.119857+0000","last_active":"2022-09-10T04:26:36.583882+0000","last_peered":"2022-09-10T04:26:36.583882+0000","last_clean":"2022-09-10T04:26:36.583882+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:26:36.583882+0000","last_undegraded":"2022-09-10T04:26:36.583882+0000","last_fullsized":"2022-09-10T04:26:36.583882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:21:40.941165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.462233+0000","last_change":"2022-09-10T04:26:31.555699+0000","last_active":"2022-09-10T04:26:56.462233+0000","last_peered":"2022-09-10T04:26:56.462233+0000","last_clean":"2022-09-10T04:26:56.462233+0000","last_became_active":"2022-09-10T04:26:31.555337+0000","last_became_peered":"2022-09-10T04:26:31.555337+0000","last_unstale":"2022-09-10T04:26:56.462233+0000","last_undegraded":"2022-09-10T04:26:56.462233+0000","last_fullsized":"2022-09-10T04:26:56.462233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:38:23.689077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.298100+0000","last_change":"2022-09-10T04:26:36.592313+0000","last_active":"2022-09-10T04:26:37.298100+0000","last_peered":"2022-09-10T04:26:37.298100+0000","last_clean":"2022-09-10T04:26:37.298100+0000","last_became_active":"2022-09-10T04:26:36.591703+0000","last_became_peered":"2022-09-10T04:26:36.591703+0000","last_unstale":"2022-09-10T04:26:37.298100+0000","last_undegraded":"2022-09-10T04:26:37.298100+0000","last_fullsized":"2022-09-10T04:26:37.298100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:19:26.704605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581155+0000","last_change":"2022-09-10T04:26:32.138958+0000","last_active":"2022-09-10T04:26:36.581155+0000","last_peered":"2022-09-10T04:26:36.581155+0000","last_clean":"2022-09-10T04:26:36.581155+0000","last_became_active":"2022-09-10T04:26:32.138655+0000","last_became_peered":"2022-09-10T04:26:32.138655+0000","last_unstale":"2022-09-10T04:26:36.581155+0000","last_undegraded":"2022-09-10T04:26:36.581155+0000","last_fullsized":"2022-09-10T04:26:36.581155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:25:07.759875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.455863+0000","last_change":"2022-09-10T04:26:31.544708+0000","last_active":"2022-09-10T04:26:55.455863+0000","last_peered":"2022-09-10T04:26:55.455863+0000","last_clean":"2022-09-10T04:26:55.455863+0000","last_became_active":"2022-09-10T04:26:31.544386+0000","last_became_peered":"2022-09-10T04:26:31.544386+0000","last_unstale":"2022-09-10T04:26:55.455863+0000","last_undegraded":"2022-09-10T04:26:55.455863+0000","last_fullsized":"2022-09-10T04:26:55.455863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:07:18.444015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":377,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584162+0000","last_change":"2022-09-10T04:25:23.357461+0000","last_active":"2022-09-10T04:26:36.584162+0000","last_peered":"2022-09-10T04:26:36.584162+0000","last_clean":"2022-09-10T04:26:36.584162+0000","last_became_active":"2022-09-10T04:25:23.356951+0000","last_became_peered":"2022-09-10T04:25:23.356951+0000","last_unstale":"2022-09-10T04:26:36.584162+0000","last_undegraded":"2022-09-10T04:26:36.584162+0000","last_fullsized":"2022-09-10T04:26:36.584162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_clean_scrub_stamp":"2022-09-10T04:21:40.122270+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:01:20.972512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.288579+0000","last_change":"2022-09-10T04:26:36.790925+0000","last_active":"2022-09-10T04:26:37.288579+0000","last_peered":"2022-09-10T04:26:37.288579+0000","last_clean":"2022-09-10T04:26:37.288579+0000","last_became_active":"2022-09-10T04:26:36.790332+0000","last_became_peered":"2022-09-10T04:26:36.790332+0000","last_unstale":"2022-09-10T04:26:37.288579+0000","last_undegraded":"2022-09-10T04:26:37.288579+0000","last_fullsized":"2022-09-10T04:26:37.288579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:10:55.153959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586285+0000","last_change":"2022-09-10T04:26:32.122356+0000","last_active":"2022-09-10T04:26:36.586285+0000","last_peered":"2022-09-10T04:26:36.586285+0000","last_clean":"2022-09-10T04:26:36.586285+0000","last_became_active":"2022-09-10T04:26:32.122028+0000","last_became_peered":"2022-09-10T04:26:32.122028+0000","last_unstale":"2022-09-10T04:26:36.586285+0000","last_undegraded":"2022-09-10T04:26:36.586285+0000","last_fullsized":"2022-09-10T04:26:36.586285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:33.837184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.460004+0000","last_change":"2022-09-10T04:26:31.556537+0000","last_active":"2022-09-10T04:26:56.460004+0000","last_peered":"2022-09-10T04:26:56.460004+0000","last_clean":"2022-09-10T04:26:56.460004+0000","last_became_active":"2022-09-10T04:26:31.556209+0000","last_became_peered":"2022-09-10T04:26:31.556209+0000","last_unstale":"2022-09-10T04:26:56.460004+0000","last_undegraded":"2022-09-10T04:26:56.460004+0000","last_fullsized":"2022-09-10T04:26:56.460004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:42:05.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.100221+0000","last_change":"2022-09-10T04:26:36.600051+0000","last_active":"2022-09-10T04:26:38.100221+0000","last_peered":"2022-09-10T04:26:38.100221+0000","last_clean":"2022-09-10T04:26:38.100221+0000","last_became_active":"2022-09-10T04:26:36.599603+0000","last_became_peered":"2022-09-10T04:26:36.599603+0000","last_unstale":"2022-09-10T04:26:38.100221+0000","last_undegraded":"2022-09-10T04:26:38.100221+0000","last_fullsized":"2022-09-10T04:26:38.100221+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:48:12.857851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.989022+0000","last_change":"2022-09-10T04:26:31.560780+0000","last_active":"2022-09-10T04:26:38.989022+0000","last_peered":"2022-09-10T04:26:38.989022+0000","last_clean":"2022-09-10T04:26:38.989022+0000","last_became_active":"2022-09-10T04:26:31.560462+0000","last_became_peered":"2022-09-10T04:26:31.560462+0000","last_unstale":"2022-09-10T04:26:38.989022+0000","last_undegraded":"2022-09-10T04:26:38.989022+0000","last_fullsized":"2022-09-10T04:26:38.989022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:05:15.437095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589341+0000","last_change":"2022-09-10T04:26:32.127579+0000","last_active":"2022-09-10T04:26:36.589341+0000","last_peered":"2022-09-10T04:26:36.589341+0000","last_clean":"2022-09-10T04:26:36.589341+0000","last_became_active":"2022-09-10T04:26:32.127237+0000","last_became_peered":"2022-09-10T04:26:32.127237+0000","last_unstale":"2022-09-10T04:26:36.589341+0000","last_undegraded":"2022-09-10T04:26:36.589341+0000","last_fullsized":"2022-09-10T04:26:36.589341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:09:28.061156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.791816+0000","last_change":"2022-09-10T04:26:36.790777+0000","last_active":"2022-09-10T04:26:36.791816+0000","last_peered":"2022-09-10T04:26:36.791816+0000","last_clean":"2022-09-10T04:26:36.791816+0000","last_became_active":"2022-09-10T04:26:36.789974+0000","last_became_peered":"2022-09-10T04:26:36.789974+0000","last_unstale":"2022-09-10T04:26:36.791816+0000","last_undegraded":"2022-09-10T04:26:36.791816+0000","last_fullsized":"2022-09-10T04:26:36.791816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:41:20.214040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.018591+0000","last_change":"2022-09-10T04:26:31.552253+0000","last_active":"2022-09-10T04:26:37.018591+0000","last_peered":"2022-09-10T04:26:37.018591+0000","last_clean":"2022-09-10T04:26:37.018591+0000","last_became_active":"2022-09-10T04:26:31.551846+0000","last_became_peered":"2022-09-10T04:26:31.551846+0000","last_unstale":"2022-09-10T04:26:37.018591+0000","last_undegraded":"2022-09-10T04:26:37.018591+0000","last_fullsized":"2022-09-10T04:26:37.018591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:28:49.941688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581283+0000","last_change":"2022-09-10T04:26:32.145463+0000","last_active":"2022-09-10T04:26:36.581283+0000","last_peered":"2022-09-10T04:26:36.581283+0000","last_clean":"2022-09-10T04:26:36.581283+0000","last_became_active":"2022-09-10T04:26:32.145115+0000","last_became_peered":"2022-09-10T04:26:32.145115+0000","last_unstale":"2022-09-10T04:26:36.581283+0000","last_undegraded":"2022-09-10T04:26:36.581283+0000","last_fullsized":"2022-09-10T04:26:36.581283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:11.597005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.243771+0000","last_change":"2022-09-10T04:26:36.594514+0000","last_active":"2022-09-10T04:26:37.243771+0000","last_peered":"2022-09-10T04:26:37.243771+0000","last_clean":"2022-09-10T04:26:37.243771+0000","last_became_active":"2022-09-10T04:26:36.594092+0000","last_became_peered":"2022-09-10T04:26:36.594092+0000","last_unstale":"2022-09-10T04:26:37.243771+0000","last_undegraded":"2022-09-10T04:26:37.243771+0000","last_fullsized":"2022-09-10T04:26:37.243771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:34:31.510362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.453895+0000","last_change":"2022-09-10T04:26:31.548844+0000","last_active":"2022-09-10T04:26:55.453895+0000","last_peered":"2022-09-10T04:26:55.453895+0000","last_clean":"2022-09-10T04:26:55.453895+0000","last_became_active":"2022-09-10T04:26:31.548535+0000","last_became_peered":"2022-09-10T04:26:31.548535+0000","last_unstale":"2022-09-10T04:26:55.453895+0000","last_undegraded":"2022-09-10T04:26:55.453895+0000","last_fullsized":"2022-09-10T04:26:55.453895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:40:49.443190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586008+0000","last_change":"2022-09-10T04:26:32.136407+0000","last_active":"2022-09-10T04:26:36.586008+0000","last_peered":"2022-09-10T04:26:36.586008+0000","last_clean":"2022-09-10T04:26:36.586008+0000","last_became_active":"2022-09-10T04:26:32.136116+0000","last_became_peered":"2022-09-10T04:26:32.136116+0000","last_unstale":"2022-09-10T04:26:36.586008+0000","last_undegraded":"2022-09-10T04:26:36.586008+0000","last_fullsized":"2022-09-10T04:26:36.586008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:32:26.673896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119584+0000","last_change":"2022-09-10T04:26:36.790658+0000","last_active":"2022-09-10T04:26:37.119584+0000","last_peered":"2022-09-10T04:26:37.119584+0000","last_clean":"2022-09-10T04:26:37.119584+0000","last_became_active":"2022-09-10T04:26:36.790203+0000","last_became_peered":"2022-09-10T04:26:36.790203+0000","last_unstale":"2022-09-10T04:26:37.119584+0000","last_undegraded":"2022-09-10T04:26:37.119584+0000","last_fullsized":"2022-09-10T04:26:37.119584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:53:08.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583789+0000","last_change":"2022-09-10T04:26:31.549875+0000","last_active":"2022-09-10T04:26:36.583789+0000","last_peered":"2022-09-10T04:26:36.583789+0000","last_clean":"2022-09-10T04:26:36.583789+0000","last_became_active":"2022-09-10T04:26:31.549578+0000","last_became_peered":"2022-09-10T04:26:31.549578+0000","last_unstale":"2022-09-10T04:26:36.583789+0000","last_undegraded":"2022-09-10T04:26:36.583789+0000","last_fullsized":"2022-09-10T04:26:36.583789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:52:51.521360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580312+0000","last_change":"2022-09-10T04:26:32.111710+0000","last_active":"2022-09-10T04:26:36.580312+0000","last_peered":"2022-09-10T04:26:36.580312+0000","last_clean":"2022-09-10T04:26:36.580312+0000","last_became_active":"2022-09-10T04:26:32.111314+0000","last_became_peered":"2022-09-10T04:26:32.111314+0000","last_unstale":"2022-09-10T04:26:36.580312+0000","last_undegraded":"2022-09-10T04:26:36.580312+0000","last_fullsized":"2022-09-10T04:26:36.580312+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:42.589093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420237+0000","last_change":"2022-09-10T04:26:36.596926+0000","last_active":"2022-09-10T04:26:38.420237+0000","last_peered":"2022-09-10T04:26:38.420237+0000","last_clean":"2022-09-10T04:26:38.420237+0000","last_became_active":"2022-09-10T04:26:36.596287+0000","last_became_peered":"2022-09-10T04:26:36.596287+0000","last_unstale":"2022-09-10T04:26:38.420237+0000","last_undegraded":"2022-09-10T04:26:38.420237+0000","last_fullsized":"2022-09-10T04:26:38.420237+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:52:54.252320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.455040+0000","last_change":"2022-09-10T04:26:31.552277+0000","last_active":"2022-09-10T04:26:55.455040+0000","last_peered":"2022-09-10T04:26:55.455040+0000","last_clean":"2022-09-10T04:26:55.455040+0000","last_became_active":"2022-09-10T04:26:31.551904+0000","last_became_peered":"2022-09-10T04:26:31.551904+0000","last_unstale":"2022-09-10T04:26:55.455040+0000","last_undegraded":"2022-09-10T04:26:55.455040+0000","last_fullsized":"2022-09-10T04:26:55.455040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:28:52.507833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583873+0000","last_change":"2022-09-10T04:26:32.116288+0000","last_active":"2022-09-10T04:26:36.583873+0000","last_peered":"2022-09-10T04:26:36.583873+0000","last_clean":"2022-09-10T04:26:36.583873+0000","last_became_active":"2022-09-10T04:26:32.115955+0000","last_became_peered":"2022-09-10T04:26:32.115955+0000","last_unstale":"2022-09-10T04:26:36.583873+0000","last_undegraded":"2022-09-10T04:26:36.583873+0000","last_fullsized":"2022-09-10T04:26:36.583873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:02:30.470550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119460+0000","last_change":"2022-09-10T04:26:36.594357+0000","last_active":"2022-09-10T04:26:37.119460+0000","last_peered":"2022-09-10T04:26:37.119460+0000","last_clean":"2022-09-10T04:26:37.119460+0000","last_became_active":"2022-09-10T04:26:36.593865+0000","last_became_peered":"2022-09-10T04:26:36.593865+0000","last_unstale":"2022-09-10T04:26:37.119460+0000","last_undegraded":"2022-09-10T04:26:37.119460+0000","last_fullsized":"2022-09-10T04:26:37.119460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:11:14.675252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577076+0000","last_change":"2022-09-10T04:26:31.551838+0000","last_active":"2022-09-10T04:26:36.577076+0000","last_peered":"2022-09-10T04:26:36.577076+0000","last_clean":"2022-09-10T04:26:36.577076+0000","last_became_active":"2022-09-10T04:26:31.551516+0000","last_became_peered":"2022-09-10T04:26:31.551516+0000","last_unstale":"2022-09-10T04:26:36.577076+0000","last_undegraded":"2022-09-10T04:26:36.577076+0000","last_fullsized":"2022-09-10T04:26:36.577076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:18:58.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581086+0000","last_change":"2022-09-10T04:26:32.145692+0000","last_active":"2022-09-10T04:26:36.581086+0000","last_peered":"2022-09-10T04:26:36.581086+0000","last_clean":"2022-09-10T04:26:36.581086+0000","last_became_active":"2022-09-10T04:26:32.144922+0000","last_became_peered":"2022-09-10T04:26:32.144922+0000","last_unstale":"2022-09-10T04:26:36.581086+0000","last_undegraded":"2022-09-10T04:26:36.581086+0000","last_fullsized":"2022-09-10T04:26:36.581086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:12:29.711076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119772+0000","last_change":"2022-09-10T04:26:36.791321+0000","last_active":"2022-09-10T04:26:37.119772+0000","last_peered":"2022-09-10T04:26:37.119772+0000","last_clean":"2022-09-10T04:26:37.119772+0000","last_became_active":"2022-09-10T04:26:36.789918+0000","last_became_peered":"2022-09-10T04:26:36.789918+0000","last_unstale":"2022-09-10T04:26:37.119772+0000","last_undegraded":"2022-09-10T04:26:37.119772+0000","last_fullsized":"2022-09-10T04:26:37.119772+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:49:56.538141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.454909+0000","last_change":"2022-09-10T04:26:31.552140+0000","last_active":"2022-09-10T04:26:55.454909+0000","last_peered":"2022-09-10T04:26:55.454909+0000","last_clean":"2022-09-10T04:26:55.454909+0000","last_became_active":"2022-09-10T04:26:31.551191+0000","last_became_peered":"2022-09-10T04:26:31.551191+0000","last_unstale":"2022-09-10T04:26:55.454909+0000","last_undegraded":"2022-09-10T04:26:55.454909+0000","last_fullsized":"2022-09-10T04:26:55.454909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:52.681536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584586+0000","last_change":"2022-09-10T04:26:32.125775+0000","last_active":"2022-09-10T04:26:36.584586+0000","last_peered":"2022-09-10T04:26:36.584586+0000","last_clean":"2022-09-10T04:26:36.584586+0000","last_became_active":"2022-09-10T04:26:32.125463+0000","last_became_peered":"2022-09-10T04:26:32.125463+0000","last_unstale":"2022-09-10T04:26:36.584586+0000","last_undegraded":"2022-09-10T04:26:36.584586+0000","last_fullsized":"2022-09-10T04:26:36.584586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:33:26.816695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.215638+0000","last_change":"2022-09-10T04:26:36.593873+0000","last_active":"2022-09-10T04:26:38.215638+0000","last_peered":"2022-09-10T04:26:38.215638+0000","last_clean":"2022-09-10T04:26:38.215638+0000","last_became_active":"2022-09-10T04:26:36.593418+0000","last_became_peered":"2022-09-10T04:26:36.593418+0000","last_unstale":"2022-09-10T04:26:38.215638+0000","last_undegraded":"2022-09-10T04:26:38.215638+0000","last_fullsized":"2022-09-10T04:26:38.215638+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:21.834135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577448+0000","last_change":"2022-09-10T04:26:31.555352+0000","last_active":"2022-09-10T04:26:36.577448+0000","last_peered":"2022-09-10T04:26:36.577448+0000","last_clean":"2022-09-10T04:26:36.577448+0000","last_became_active":"2022-09-10T04:26:31.555092+0000","last_became_peered":"2022-09-10T04:26:31.555092+0000","last_unstale":"2022-09-10T04:26:36.577448+0000","last_undegraded":"2022-09-10T04:26:36.577448+0000","last_fullsized":"2022-09-10T04:26:36.577448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:30:17.812089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584742+0000","last_change":"2022-09-10T04:26:32.115687+0000","last_active":"2022-09-10T04:26:36.584742+0000","last_peered":"2022-09-10T04:26:36.584742+0000","last_clean":"2022-09-10T04:26:36.584742+0000","last_became_active":"2022-09-10T04:26:32.115369+0000","last_became_peered":"2022-09-10T04:26:32.115369+0000","last_unstale":"2022-09-10T04:26:36.584742+0000","last_undegraded":"2022-09-10T04:26:36.584742+0000","last_fullsized":"2022-09-10T04:26:36.584742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:59:31.105825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.988722+0000","last_change":"2022-09-10T04:26:36.602214+0000","last_active":"2022-09-10T04:26:38.988722+0000","last_peered":"2022-09-10T04:26:38.988722+0000","last_clean":"2022-09-10T04:26:38.988722+0000","last_became_active":"2022-09-10T04:26:36.601785+0000","last_became_peered":"2022-09-10T04:26:36.601785+0000","last_unstale":"2022-09-10T04:26:38.988722+0000","last_undegraded":"2022-09-10T04:26:38.988722+0000","last_fullsized":"2022-09-10T04:26:38.988722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:52:42.795971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583014+0000","last_change":"2022-09-10T04:26:31.556182+0000","last_active":"2022-09-10T04:26:36.583014+0000","last_peered":"2022-09-10T04:26:36.583014+0000","last_clean":"2022-09-10T04:26:36.583014+0000","last_became_active":"2022-09-10T04:26:31.555873+0000","last_became_peered":"2022-09-10T04:26:31.555873+0000","last_unstale":"2022-09-10T04:26:36.583014+0000","last_undegraded":"2022-09-10T04:26:36.583014+0000","last_fullsized":"2022-09-10T04:26:36.583014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:56:49.406476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585008+0000","last_change":"2022-09-10T04:26:32.134925+0000","last_active":"2022-09-10T04:26:36.585008+0000","last_peered":"2022-09-10T04:26:36.585008+0000","last_clean":"2022-09-10T04:26:36.585008+0000","last_became_active":"2022-09-10T04:26:32.134581+0000","last_became_peered":"2022-09-10T04:26:32.134581+0000","last_unstale":"2022-09-10T04:26:36.585008+0000","last_undegraded":"2022-09-10T04:26:36.585008+0000","last_fullsized":"2022-09-10T04:26:36.585008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:06:09.815252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.287892+0000","last_change":"2022-09-10T04:26:36.590902+0000","last_active":"2022-09-10T04:26:37.287892+0000","last_peered":"2022-09-10T04:26:37.287892+0000","last_clean":"2022-09-10T04:26:37.287892+0000","last_became_active":"2022-09-10T04:26:36.590468+0000","last_became_peered":"2022-09-10T04:26:36.590468+0000","last_unstale":"2022-09-10T04:26:37.287892+0000","last_undegraded":"2022-09-10T04:26:37.287892+0000","last_fullsized":"2022-09-10T04:26:37.287892+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:49:40.841273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586989+0000","last_change":"2022-09-10T04:26:31.554090+0000","last_active":"2022-09-10T04:26:36.586989+0000","last_peered":"2022-09-10T04:26:36.586989+0000","last_clean":"2022-09-10T04:26:36.586989+0000","last_became_active":"2022-09-10T04:26:31.553675+0000","last_became_peered":"2022-09-10T04:26:31.553675+0000","last_unstale":"2022-09-10T04:26:36.586989+0000","last_undegraded":"2022-09-10T04:26:36.586989+0000","last_fullsized":"2022-09-10T04:26:36.586989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:35:33.619937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584919+0000","last_change":"2022-09-10T04:26:32.118923+0000","last_active":"2022-09-10T04:26:36.584919+0000","last_peered":"2022-09-10T04:26:36.584919+0000","last_clean":"2022-09-10T04:26:36.584919+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:26:36.584919+0000","last_undegraded":"2022-09-10T04:26:36.584919+0000","last_fullsized":"2022-09-10T04:26:36.584919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:03.377179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583988+0000","last_change":"2022-09-10T04:26:32.115339+0000","last_active":"2022-09-10T04:26:36.583988+0000","last_peered":"2022-09-10T04:26:36.583988+0000","last_clean":"2022-09-10T04:26:36.583988+0000","last_became_active":"2022-09-10T04:26:32.115044+0000","last_became_peered":"2022-09-10T04:26:32.115044+0000","last_unstale":"2022-09-10T04:26:36.583988+0000","last_undegraded":"2022-09-10T04:26:36.583988+0000","last_fullsized":"2022-09-10T04:26:36.583988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:06:39.719939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577922+0000","last_change":"2022-09-10T04:26:32.121701+0000","last_active":"2022-09-10T04:26:36.577922+0000","last_peered":"2022-09-10T04:26:36.577922+0000","last_clean":"2022-09-10T04:26:36.577922+0000","last_became_active":"2022-09-10T04:26:32.121417+0000","last_became_peered":"2022-09-10T04:26:32.121417+0000","last_unstale":"2022-09-10T04:26:36.577922+0000","last_undegraded":"2022-09-10T04:26:36.577922+0000","last_fullsized":"2022-09-10T04:26:36.577922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:16:44.194677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.576913+0000","last_change":"2022-09-10T04:26:32.139762+0000","last_active":"2022-09-10T04:26:36.576913+0000","last_peered":"2022-09-10T04:26:36.576913+0000","last_clean":"2022-09-10T04:26:36.576913+0000","last_became_active":"2022-09-10T04:26:32.139468+0000","last_became_peered":"2022-09-10T04:26:32.139468+0000","last_unstale":"2022-09-10T04:26:36.576913+0000","last_undegraded":"2022-09-10T04:26:36.576913+0000","last_fullsized":"2022-09-10T04:26:36.576913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:37:25.448977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.588356+0000","last_change":"2022-09-10T04:26:32.131899+0000","last_active":"2022-09-10T04:26:36.588356+0000","last_peered":"2022-09-10T04:26:36.588356+0000","last_clean":"2022-09-10T04:26:36.588356+0000","last_became_active":"2022-09-10T04:26:32.130047+0000","last_became_peered":"2022-09-10T04:26:32.130047+0000","last_unstale":"2022-09-10T04:26:36.588356+0000","last_undegraded":"2022-09-10T04:26:36.588356+0000","last_fullsized":"2022-09-10T04:26:36.588356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:49:48.001008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589423+0000","last_change":"2022-09-10T04:26:32.126445+0000","last_active":"2022-09-10T04:26:36.589423+0000","last_peered":"2022-09-10T04:26:36.589423+0000","last_clean":"2022-09-10T04:26:36.589423+0000","last_became_active":"2022-09-10T04:26:32.125935+0000","last_became_peered":"2022-09-10T04:26:32.125935+0000","last_unstale":"2022-09-10T04:26:36.589423+0000","last_undegraded":"2022-09-10T04:26:36.589423+0000","last_fullsized":"2022-09-10T04:26:36.589423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:27:44.987973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582203+0000","last_change":"2022-09-10T04:26:32.118684+0000","last_active":"2022-09-10T04:26:36.582203+0000","last_peered":"2022-09-10T04:26:36.582203+0000","last_clean":"2022-09-10T04:26:36.582203+0000","last_became_active":"2022-09-10T04:26:32.118399+0000","last_became_peered":"2022-09-10T04:26:32.118399+0000","last_unstale":"2022-09-10T04:26:36.582203+0000","last_undegraded":"2022-09-10T04:26:36.582203+0000","last_fullsized":"2022-09-10T04:26:36.582203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:55:18.907421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577149+0000","last_change":"2022-09-10T04:26:32.122790+0000","last_active":"2022-09-10T04:26:36.577149+0000","last_peered":"2022-09-10T04:26:36.577149+0000","last_clean":"2022-09-10T04:26:36.577149+0000","last_became_active":"2022-09-10T04:26:32.122379+0000","last_became_peered":"2022-09-10T04:26:32.122379+0000","last_unstale":"2022-09-10T04:26:36.577149+0000","last_undegraded":"2022-09-10T04:26:36.577149+0000","last_fullsized":"2022-09-10T04:26:36.577149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:36:19.747563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581383+0000","last_change":"2022-09-10T04:26:32.145947+0000","last_active":"2022-09-10T04:26:36.581383+0000","last_peered":"2022-09-10T04:26:36.581383+0000","last_clean":"2022-09-10T04:26:36.581383+0000","last_became_active":"2022-09-10T04:26:32.145570+0000","last_became_peered":"2022-09-10T04:26:32.145570+0000","last_unstale":"2022-09-10T04:26:36.581383+0000","last_undegraded":"2022-09-10T04:26:36.581383+0000","last_fullsized":"2022-09-10T04:26:36.581383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:38:56.028983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583785+0000","last_change":"2022-09-10T04:26:32.140019+0000","last_active":"2022-09-10T04:26:36.583785+0000","last_peered":"2022-09-10T04:26:36.583785+0000","last_clean":"2022-09-10T04:26:36.583785+0000","last_became_active":"2022-09-10T04:26:32.139709+0000","last_became_peered":"2022-09-10T04:26:32.139709+0000","last_unstale":"2022-09-10T04:26:36.583785+0000","last_undegraded":"2022-09-10T04:26:36.583785+0000","last_fullsized":"2022-09-10T04:26:36.583785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:56:11.850506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581578+0000","last_change":"2022-09-10T04:26:32.145867+0000","last_active":"2022-09-10T04:26:36.581578+0000","last_peered":"2022-09-10T04:26:36.581578+0000","last_clean":"2022-09-10T04:26:36.581578+0000","last_became_active":"2022-09-10T04:26:32.145582+0000","last_became_peered":"2022-09-10T04:26:32.145582+0000","last_unstale":"2022-09-10T04:26:36.581578+0000","last_undegraded":"2022-09-10T04:26:36.581578+0000","last_fullsized":"2022-09-10T04:26:36.581578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:30:11.784617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584565+0000","last_change":"2022-09-10T04:26:32.134641+0000","last_active":"2022-09-10T04:26:36.584565+0000","last_peered":"2022-09-10T04:26:36.584565+0000","last_clean":"2022-09-10T04:26:36.584565+0000","last_became_active":"2022-09-10T04:26:32.134320+0000","last_became_peered":"2022-09-10T04:26:32.134320+0000","last_unstale":"2022-09-10T04:26:36.584565+0000","last_undegraded":"2022-09-10T04:26:36.584565+0000","last_fullsized":"2022-09-10T04:26:36.584565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:58:29.800916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586176+0000","last_change":"2022-09-10T04:26:32.126597+0000","last_active":"2022-09-10T04:26:36.586176+0000","last_peered":"2022-09-10T04:26:36.586176+0000","last_clean":"2022-09-10T04:26:36.586176+0000","last_became_active":"2022-09-10T04:26:32.125895+0000","last_became_peered":"2022-09-10T04:26:32.125895+0000","last_unstale":"2022-09-10T04:26:36.586176+0000","last_undegraded":"2022-09-10T04:26:36.586176+0000","last_fullsized":"2022-09-10T04:26:36.586176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:17:41.456304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581777+0000","last_change":"2022-09-10T04:26:32.122525+0000","last_active":"2022-09-10T04:26:36.581777+0000","last_peered":"2022-09-10T04:26:36.581777+0000","last_clean":"2022-09-10T04:26:36.581777+0000","last_became_active":"2022-09-10T04:26:32.122113+0000","last_became_peered":"2022-09-10T04:26:32.122113+0000","last_unstale":"2022-09-10T04:26:36.581777+0000","last_undegraded":"2022-09-10T04:26:36.581777+0000","last_fullsized":"2022-09-10T04:26:36.581777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:29:19.351147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5307,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":15483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350992,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6028,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739124,"statfs":{"total":95995035648,"available":95988862976,"internally_reserved":0,"allocated":987136,"data_stored":478526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.70799999999999996}]},{"osd":6,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.65000000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.68700000000000006}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":9,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.89300000000000002}]},{"osd":10,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.69599999999999995}]}]},{"osd":10,"up_from":73,"seq":313532612629,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6820,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738332,"statfs":{"total":95995035648,"available":95988051968,"internally_reserved":0,"allocated":1601536,"data_stored":1072514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.438}]},{"osd":5,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":9.7629999999999999,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.7349999999999994,"5min":9.7349999999999994,"15min":9.7349999999999994},"last":0.64400000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.873}]},{"osd":8,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.65900000000000003}]},{"osd":9,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.70099999999999996}]},{"osd":11,"last update":"Sat Sep 10 04:26:44 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.67400000000000004}]}]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6224,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738928,"statfs":{"total":95995035648,"available":95988662272,"internally_reserved":0,"allocated":991232,"data_stored":479092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.65800000000000003}]},{"osd":1,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.85099999999999998}]},{"osd":2,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.72899999999999998}]},{"osd":4,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.272,"5min":1.272,"15min":1.272},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.68400000000000005}]},{"osd":5,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.54500000000000004}]},{"osd":6,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.79700000000000004}]},{"osd":7,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.81699999999999995}]},{"osd":8,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.083,"15min":1.083},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.624},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.91800000000000004}]},{"osd":10,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.82899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]},{"osd":8,"up_from":60,"seq":257698037792,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6360,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":999424,"data_stored":479136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71399999999999997,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.58299999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76700000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.436},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":4,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.622,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83999999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.79300000000000004}]},{"osd":5,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.5},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80500000000000005,"5min":1.046,"15min":1.046},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69799999999999995,"15min":0.70799999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69099999999999995,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.72699999999999998}]},{"osd":7,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.747},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.92200000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.73699999999999999}]},{"osd":9,"last update":"Sat Sep 10 04:25:55 2022","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.80300000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7068,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738084,"statfs":{"total":95995035648,"available":95987798016,"internally_reserved":0,"allocated":1593344,"data_stored":1070251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.77100000000000002,"15min":0.79900000000000004},"min":{"1min":0.41299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":5.3719999999999999,"15min":5.3719999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.75,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39900000000000002,"15min":0.39800000000000002},"min":{"1min":0.30099999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61099999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.441,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.628,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77400000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.77600000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.65000000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.45800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.48899999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.628}]},{"osd":4,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67800000000000005,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.373}]},{"osd":5,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76200000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84199999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.50800000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83299999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69399999999999995,"15min":0.69699999999999995},"min":{"1min":0.51600000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.84699999999999998,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.47499999999999998}]},{"osd":8,"last update":"Sat Sep 10 04:26:24 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.68500000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67000000000000004,"15min":0.67800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81200000000000006,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63900000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:25:56 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68700000000000006}]},{"osd":10,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6348,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738804,"statfs":{"total":95995035648,"available":95988535296,"internally_reserved":0,"allocated":987136,"data_stored":478526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.76400000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.93600000000000005,"5min":1.046,"15min":1.046},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64900000000000002,"15min":0.63400000000000001},"min":{"1min":0.441,"5min":0.436,"15min":0.436},"max":{"1min":0.85999999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68600000000000005,"15min":0.66900000000000004},"min":{"1min":0.504,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.61299999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84599999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0209999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.998,"5min":1.103,"15min":1.103},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.79100000000000004}]},{"osd":4,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68799999999999994,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.92400000000000004,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.76600000000000001,"15min":0.76100000000000001},"min":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.96299999999999997,"5min":1.167,"15min":1.167},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.69099999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92900000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70699999999999996,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94799999999999995,"5min":1.036,"15min":1.036},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:26:03 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.78300000000000003,"15min":0.76100000000000001},"min":{"1min":0.66200000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.82599999999999996,"15min":0.80500000000000005},"min":{"1min":0.71999999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.80000000000000004}]},{"osd":8,"last update":"Sat Sep 10 04:26:30 2022","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.85499999999999998,"15min":0.85099999999999998},"min":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1459999999999999,"5min":1.181,"15min":1.181},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72399999999999998,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0840000000000001,"5min":1.097,"15min":1.097},"last":0.88100000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.90100000000000002}]},{"osd":10,"last update":"Sat Sep 10 04:26:22 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.81399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7252,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737900,"statfs":{"total":95995035648,"available":95987609600,"internally_reserved":0,"allocated":995328,"data_stored":482010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:29 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.51700000000000002},"min":{"1min":0.41399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.67500000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.503},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53400000000000003,"15min":0.52400000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.32300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:26:17 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84299999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.60699999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.68700000000000006}]},{"osd":4,"last update":"Sat Sep 10 04:26:41 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.42599999999999999,"15min":0.40500000000000003},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70199999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.32400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Sep 10 04:26:09 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.52800000000000002,"15min":0.502},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.53000000000000003,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.71599999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:26:34 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.46700000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.749},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.81000000000000005}]},{"osd":7,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77000000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.73699999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.437,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Sat Sep 10 04:25:54 2022","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.876}]},{"osd":10,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.80300000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7248,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737904,"statfs":{"total":95995035648,"available":95987613696,"internally_reserved":0,"allocated":991232,"data_stored":478616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.86699999999999999},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":8.3729999999999993,"15min":8.3729999999999993},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.45500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0209999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.88300000000000001},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96699999999999997,"5min":8.3260000000000005,"15min":8.3260000000000005},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:25:11 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.52500000000000002,"15min":0.503},"min":{"1min":0.37,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.435},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.86299999999999999}]},{"osd":4,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63100000000000001,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.628,"15min":0.58599999999999997},"min":{"1min":0.435,"5min":0.372,"15min":0.372},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.53000000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:26:42 2022","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.65700000000000003},"min":{"1min":0.55600000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72199999999999998,"15min":0.68700000000000006},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71899999999999997,"15min":0.71099999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.72299999999999998,"15min":0.70399999999999996},"min":{"1min":0.68700000000000006,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1140000000000001,"5min":1.145,"15min":1.145},"last":0.53600000000000003}]},{"osd":8,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80400000000000005,"15min":0.80000000000000004},"min":{"1min":0.59499999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.099,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.79000000000000004,"15min":0.78400000000000003},"min":{"1min":0.59599999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.70499999999999996}]},{"osd":9,"last update":"Sat Sep 10 04:25:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.879}]},{"osd":10,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.65100000000000002}]},{"osd":11,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.84499999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7320,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737832,"statfs":{"total":95995035648,"available":95987539968,"internally_reserved":0,"allocated":999424,"data_stored":481903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68200000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33500000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72899999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.64300000000000002}]},{"osd":1,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40400000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77200000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52600000000000002,"15min":0.504},"min":{"1min":0.378,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71299999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Sep 10 04:26:16 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.80500000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.59999999999999998},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.80500000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Sep 10 04:25:40 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.54900000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73599999999999999,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84399999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.79000000000000004}]},{"osd":5,"last update":"Sat Sep 10 04:26:07 2022","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57699999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.754,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66200000000000003}]},{"osd":6,"last update":"Sat Sep 10 04:26:37 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82199999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84199999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.68999999999999995}]},{"osd":7,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.86199999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.76500000000000001}]},{"osd":8,"last update":"Sat Sep 10 04:26:28 2022","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.83399999999999996}]},{"osd":9,"last update":"Sat Sep 10 04:25:56 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.80800000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:26:22 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.91900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6612,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738540,"statfs":{"total":95995035648,"available":95988264960,"internally_reserved":0,"allocated":995328,"data_stored":478630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:15 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.41699999999999998},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68899999999999995,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.432},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46700000000000003,"15min":0.44400000000000001},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.81899999999999995,"5min":1.242,"15min":1.242},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Sep 10 04:26:15 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65300000000000002,"15min":0.61599999999999999},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0409999999999999,"5min":1.494,"15min":1.494},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63,"15min":0.60799999999999998},"min":{"1min":0.47299999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.88200000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Sep 10 04:26:15 2022","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.59599999999999997},"min":{"1min":0.437,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88400000000000001,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.67700000000000005,"15min":0.64900000000000002},"min":{"1min":0.59099999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99299999999999999,"5min":1.454,"15min":1.454},"last":0.79300000000000004}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.626},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"min":{"1min":0.45800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.95499999999999996,"5min":1.528,"15min":1.528},"last":0.71099999999999997}]},{"osd":5,"last update":"Sat Sep 10 04:26:02 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66900000000000004,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0069999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.47999999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94299999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64500000000000002},"min":{"1min":0.501,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.72699999999999998,"15min":0.70799999999999996},"min":{"1min":0.54400000000000004,"5min":0.433,"15min":0.433},"max":{"1min":0.98799999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:26:02 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.76200000000000001,"15min":0.75900000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1719999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81599999999999995}]},{"osd":8,"last update":"Sat Sep 10 04:26:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1060000000000001,"5min":1.208,"15min":1.208},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79200000000000004,"15min":0.78600000000000003},"min":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.099,"5min":1.139,"15min":1.139},"last":0.73299999999999998}]},{"osd":9,"last update":"Sat Sep 10 04:25:57 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.83599999999999997}]},{"osd":10,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.90300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":999424,"data_stored":479122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:25:39 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73199999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54500000000000004,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.378,"15min":0.34399999999999997},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71499999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.36499999999999999},"min":{"1min":0.29799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80800000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.55900000000000005},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.86299999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79700000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:26:09 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57599999999999996,"15min":0.54300000000000004},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.92500000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.39800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66700000000000004,"15min":0.67400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.504},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.77000000000000002,"15min":0.73699999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.61699999999999999,"15min":0.61199999999999999},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55400000000000005}]},{"osd":8,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.48999999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0089999999999999,"5min":1.133,"15min":1.133},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.875,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.59499999999999997}]},{"osd":9,"last update":"Sat Sep 10 04:25:54 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67500000000000004}]},{"osd":10,"last update":"Sat Sep 10 04:26:19 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.57199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7192,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737960,"statfs":{"total":95995035648,"available":95987671040,"internally_reserved":0,"allocated":1589248,"data_stored":1068753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77100000000000002,"5min":1.879,"15min":1.879},"last":0.42899999999999999}]},{"osd":1,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.503,"15min":0.48099999999999998},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.501,"15min":0.47999999999999998},"min":{"1min":0.38500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63200000000000001}]},{"osd":2,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.375,"15min":0.37},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.55500000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.53700000000000003}]},{"osd":3,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.74099999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.46999999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.503},"min":{"1min":0.41199999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.75600000000000001}]},{"osd":6,"last update":"Sat Sep 10 04:26:36 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.45200000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.80500000000000005}]},{"osd":8,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56799999999999995}]},{"osd":9,"last update":"Sat Sep 10 04:25:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.77100000000000002}]},{"osd":10,"last update":"Sat Sep 10 04:26:19 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.745},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-10T04:26:58.971 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-09-10T04:26:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 -- 172.21.15.81:0/767274180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 msgr2=0x7fb9f81001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/767274180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f81001f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009810 tx=0x7fb9f4009b20 comp rx=0 tx=0).stop 2022-09-10T04:26:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 -- 172.21.15.81:0/767274180 shutdown_connections 2022-09-10T04:26:59.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/767274180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8101970 0x7fb9f8108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/767274180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f81001f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/767274180 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 0x7fb9f8101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.218+0000 7fb9ff19d700 1 -- 172.21.15.81:0/767274180 >> 172.21.15.81:0/767274180 conn(0x7fb9f80fb4d0 msgr2=0x7fb9f80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.219+0000 7fb9ff19d700 1 -- 172.21.15.81:0/767274180 shutdown_connections 2022-09-10T04:26:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.219+0000 7fb9ff19d700 1 -- 172.21.15.81:0/767274180 wait complete. 2022-09-10T04:26:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.220+0000 7fb9ff19d700 1 Processor -- start 2022-09-10T04:26:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 -- start start 2022-09-10T04:26:59.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f8195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:59.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 0x7fb9f8195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:59.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8101970 0x7fb9f819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:59.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f819d480 con 0x7fb9f80ffde0 2022-09-10T04:26:59.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f819d5f0 con 0x7fb9f8100fc0 2022-09-10T04:26:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9ff19d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f819d7b0 con 0x7fb9f8101970 2022-09-10T04:26:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9fe19b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f8195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9fe19b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f8195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42730/0 (socket says 172.21.15.81:42730) 2022-09-10T04:26:59.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9fe19b700 1 -- 172.21.15.81:0/3033010148 learned_addr learned my addr 172.21.15.81:0/3033010148 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:26:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.221+0000 7fb9fd99a700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 0x7fb9f8195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe99c700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8101970 0x7fb9f819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe19b700 1 -- 172.21.15.81:0/3033010148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8101970 msgr2=0x7fb9f819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe19b700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8101970 0x7fb9f819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe19b700 1 -- 172.21.15.81:0/3033010148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 msgr2=0x7fb9f8195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe19b700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 0x7fb9f8195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe19b700 1 -- 172.21.15.81:0/3033010148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e80093b0 con 0x7fb9f80ffde0 2022-09-10T04:26:59.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fd99a700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 0x7fb9f8195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:26:59.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9fe19b700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f8195900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40059c0 tx=0x7fb9f4018630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:59.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f401d070 con 0x7fb9f80ffde0 2022-09-10T04:26:59.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb9f4018820 con 0x7fb9f80ffde0 2022-09-10T04:26:59.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40047f0 con 0x7fb9f80ffde0 2022-09-10T04:26:59.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.222+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f4009470 con 0x7fb9f80ffde0 2022-09-10T04:26:59.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.223+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f819dd00 con 0x7fb9f80ffde0 2022-09-10T04:26:59.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.224+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f810b500 con 0x7fb9f80ffde0 2022-09-10T04:26:59.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.225+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb9f4029030 con 0x7fb9f80ffde0 2022-09-10T04:26:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.227+0000 7fb9ef7fe700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9e4099090 0x7fb9e409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:26:59.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.227+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fb9f40c3640 con 0x7fb9f80ffde0 2022-09-10T04:26:59.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.227+0000 7fb9fd99a700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9e4099090 0x7fb9e409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:26:59.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.228+0000 7fb9fd99a700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9e4099090 0x7fb9e409b540 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8005fd0 tx=0x7fb9e8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:26:59.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.231+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb9f408fae0 con 0x7fb9f80ffde0 2022-09-10T04:26:59.424 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.423+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9f805f120 con 0x7fb9e4099090 2022-09-10T04:26:59.436 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.435+0000 7fb9ef7fe700 1 -- 172.21.15.81:0/3033010148 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422871 (secure 0 0 0) 0x7fb9f805f120 con 0x7fb9e4099090 2022-09-10T04:26:59.444 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:26:59.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9e4099090 msgr2=0x7fb9e409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9e4099090 0x7fb9e409b540 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8005fd0 tx=0x7fb9e8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 msgr2=0x7fb9f8195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:26:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f8195900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40059c0 tx=0x7fb9f4018630 comp rx=0 tx=0).stop 2022-09-10T04:26:59.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 shutdown_connections 2022-09-10T04:26:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9e4099090 0x7fb9e409b540 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9f8101970 0x7fb9f819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9f80ffde0 0x7fb9f8195900 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 --2- 172.21.15.81:0/3033010148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9f8100fc0 0x7fb9f8195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:26:59.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 >> 172.21.15.81:0/3033010148 conn(0x7fb9f80fb4d0 msgr2=0x7fb9f80fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:26:59.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 shutdown_connections 2022-09-10T04:26:59.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:26:59.441+0000 7fb9ff19d700 1 -- 172.21.15.81:0/3033010148 wait complete. 2022-09-10T04:26:59.450 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2022-09-10T04:26:59.473 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2022-09-10T04:26:59.343640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596905,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":142,"num_read_kb":124,"num_write":297,"num_write_kb":2972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81212,"kb_used_data":13468,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":1124860612,"statfs":{"total":1151940427776,"available":1151857266688,"internally_reserved":0,"allocated":13791232,"data_stored":7530500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4207,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":71,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003494"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583240+0000","last_change":"2022-09-10T04:26:32.116469+0000","last_active":"2022-09-10T04:26:36.583240+0000","last_peered":"2022-09-10T04:26:36.583240+0000","last_clean":"2022-09-10T04:26:36.583240+0000","last_became_active":"2022-09-10T04:26:32.116060+0000","last_became_peered":"2022-09-10T04:26:32.116060+0000","last_unstale":"2022-09-10T04:26:36.583240+0000","last_undegraded":"2022-09-10T04:26:36.583240+0000","last_fullsized":"2022-09-10T04:26:36.583240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:22:05.868811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585910+0000","last_change":"2022-09-10T04:26:32.114659+0000","last_active":"2022-09-10T04:26:36.585910+0000","last_peered":"2022-09-10T04:26:36.585910+0000","last_clean":"2022-09-10T04:26:36.585910+0000","last_became_active":"2022-09-10T04:26:32.114381+0000","last_became_peered":"2022-09-10T04:26:32.114381+0000","last_unstale":"2022-09-10T04:26:36.585910+0000","last_undegraded":"2022-09-10T04:26:36.585910+0000","last_fullsized":"2022-09-10T04:26:36.585910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:53:50.267695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585535+0000","last_change":"2022-09-10T04:26:32.117123+0000","last_active":"2022-09-10T04:26:36.585535+0000","last_peered":"2022-09-10T04:26:36.585535+0000","last_clean":"2022-09-10T04:26:36.585535+0000","last_became_active":"2022-09-10T04:26:32.116753+0000","last_became_peered":"2022-09-10T04:26:32.116753+0000","last_unstale":"2022-09-10T04:26:36.585535+0000","last_undegraded":"2022-09-10T04:26:36.585535+0000","last_fullsized":"2022-09-10T04:26:36.585535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:52.713788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589699+0000","last_change":"2022-09-10T04:26:32.120151+0000","last_active":"2022-09-10T04:26:36.589699+0000","last_peered":"2022-09-10T04:26:36.589699+0000","last_clean":"2022-09-10T04:26:36.589699+0000","last_became_active":"2022-09-10T04:26:32.119510+0000","last_became_peered":"2022-09-10T04:26:32.119510+0000","last_unstale":"2022-09-10T04:26:36.589699+0000","last_undegraded":"2022-09-10T04:26:36.589699+0000","last_fullsized":"2022-09-10T04:26:36.589699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:56:23.232054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581285+0000","last_change":"2022-09-10T04:26:32.138820+0000","last_active":"2022-09-10T04:26:36.581285+0000","last_peered":"2022-09-10T04:26:36.581285+0000","last_clean":"2022-09-10T04:26:36.581285+0000","last_became_active":"2022-09-10T04:26:32.138466+0000","last_became_peered":"2022-09-10T04:26:32.138466+0000","last_unstale":"2022-09-10T04:26:36.581285+0000","last_undegraded":"2022-09-10T04:26:36.581285+0000","last_fullsized":"2022-09-10T04:26:36.581285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:16:22.128598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789266+0000","last_change":"2022-09-10T04:26:32.124561+0000","last_active":"2022-09-10T04:26:36.789266+0000","last_peered":"2022-09-10T04:26:36.789266+0000","last_clean":"2022-09-10T04:26:36.789266+0000","last_became_active":"2022-09-10T04:26:32.124263+0000","last_became_peered":"2022-09-10T04:26:32.124263+0000","last_unstale":"2022-09-10T04:26:36.789266+0000","last_undegraded":"2022-09-10T04:26:36.789266+0000","last_fullsized":"2022-09-10T04:26:36.789266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:35:16.940139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585041+0000","last_change":"2022-09-10T04:26:32.122869+0000","last_active":"2022-09-10T04:26:36.585041+0000","last_peered":"2022-09-10T04:26:36.585041+0000","last_clean":"2022-09-10T04:26:36.585041+0000","last_became_active":"2022-09-10T04:26:32.122457+0000","last_became_peered":"2022-09-10T04:26:32.122457+0000","last_unstale":"2022-09-10T04:26:36.585041+0000","last_undegraded":"2022-09-10T04:26:36.585041+0000","last_fullsized":"2022-09-10T04:26:36.585041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:08.945173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581301+0000","last_change":"2022-09-10T04:26:32.146033+0000","last_active":"2022-09-10T04:26:36.581301+0000","last_peered":"2022-09-10T04:26:36.581301+0000","last_clean":"2022-09-10T04:26:36.581301+0000","last_became_active":"2022-09-10T04:26:32.145039+0000","last_became_peered":"2022-09-10T04:26:32.145039+0000","last_unstale":"2022-09-10T04:26:36.581301+0000","last_undegraded":"2022-09-10T04:26:36.581301+0000","last_fullsized":"2022-09-10T04:26:36.581301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:44:45.332855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583717+0000","last_change":"2022-09-10T04:26:32.113787+0000","last_active":"2022-09-10T04:26:36.583717+0000","last_peered":"2022-09-10T04:26:36.583717+0000","last_clean":"2022-09-10T04:26:36.583717+0000","last_became_active":"2022-09-10T04:26:32.113454+0000","last_became_peered":"2022-09-10T04:26:32.113454+0000","last_unstale":"2022-09-10T04:26:36.583717+0000","last_undegraded":"2022-09-10T04:26:36.583717+0000","last_fullsized":"2022-09-10T04:26:36.583717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:11:23.375628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789594+0000","last_change":"2022-09-10T04:26:32.123249+0000","last_active":"2022-09-10T04:26:36.789594+0000","last_peered":"2022-09-10T04:26:36.789594+0000","last_clean":"2022-09-10T04:26:36.789594+0000","last_became_active":"2022-09-10T04:26:32.121836+0000","last_became_peered":"2022-09-10T04:26:32.121836+0000","last_unstale":"2022-09-10T04:26:36.789594+0000","last_undegraded":"2022-09-10T04:26:36.789594+0000","last_fullsized":"2022-09-10T04:26:36.789594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:22.711515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584738+0000","last_change":"2022-09-10T04:26:32.118213+0000","last_active":"2022-09-10T04:26:36.584738+0000","last_peered":"2022-09-10T04:26:36.584738+0000","last_clean":"2022-09-10T04:26:36.584738+0000","last_became_active":"2022-09-10T04:26:32.117863+0000","last_became_peered":"2022-09-10T04:26:32.117863+0000","last_unstale":"2022-09-10T04:26:36.584738+0000","last_undegraded":"2022-09-10T04:26:36.584738+0000","last_fullsized":"2022-09-10T04:26:36.584738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:26:35.632083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589648+0000","last_change":"2022-09-10T04:26:32.127748+0000","last_active":"2022-09-10T04:26:36.589648+0000","last_peered":"2022-09-10T04:26:36.589648+0000","last_clean":"2022-09-10T04:26:36.589648+0000","last_became_active":"2022-09-10T04:26:32.127470+0000","last_became_peered":"2022-09-10T04:26:32.127470+0000","last_unstale":"2022-09-10T04:26:36.589648+0000","last_undegraded":"2022-09-10T04:26:36.589648+0000","last_fullsized":"2022-09-10T04:26:36.589648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:01:58.438614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586289+0000","last_change":"2022-09-10T04:26:32.126378+0000","last_active":"2022-09-10T04:26:36.586289+0000","last_peered":"2022-09-10T04:26:36.586289+0000","last_clean":"2022-09-10T04:26:36.586289+0000","last_became_active":"2022-09-10T04:26:32.125510+0000","last_became_peered":"2022-09-10T04:26:32.125510+0000","last_unstale":"2022-09-10T04:26:36.586289+0000","last_undegraded":"2022-09-10T04:26:36.586289+0000","last_fullsized":"2022-09-10T04:26:36.586289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:41.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.587049+0000","last_change":"2022-09-10T04:26:32.118515+0000","last_active":"2022-09-10T04:26:36.587049+0000","last_peered":"2022-09-10T04:26:36.587049+0000","last_clean":"2022-09-10T04:26:36.587049+0000","last_became_active":"2022-09-10T04:26:32.118163+0000","last_became_peered":"2022-09-10T04:26:32.118163+0000","last_unstale":"2022-09-10T04:26:36.587049+0000","last_undegraded":"2022-09-10T04:26:36.587049+0000","last_fullsized":"2022-09-10T04:26:36.587049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:39:42.245166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.592195+0000","last_change":"2022-09-10T04:26:32.122232+0000","last_active":"2022-09-10T04:26:36.592195+0000","last_peered":"2022-09-10T04:26:36.592195+0000","last_clean":"2022-09-10T04:26:36.592195+0000","last_became_active":"2022-09-10T04:26:32.121906+0000","last_became_peered":"2022-09-10T04:26:32.121906+0000","last_unstale":"2022-09-10T04:26:36.592195+0000","last_undegraded":"2022-09-10T04:26:36.592195+0000","last_fullsized":"2022-09-10T04:26:36.592195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:49:50.711773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583704+0000","last_change":"2022-09-10T04:26:32.117413+0000","last_active":"2022-09-10T04:26:36.583704+0000","last_peered":"2022-09-10T04:26:36.583704+0000","last_clean":"2022-09-10T04:26:36.583704+0000","last_became_active":"2022-09-10T04:26:32.117120+0000","last_became_peered":"2022-09-10T04:26:32.117120+0000","last_unstale":"2022-09-10T04:26:36.583704+0000","last_undegraded":"2022-09-10T04:26:36.583704+0000","last_fullsized":"2022-09-10T04:26:36.583704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:50:28.426517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583151+0000","last_change":"2022-09-10T04:26:32.117852+0000","last_active":"2022-09-10T04:26:36.583151+0000","last_peered":"2022-09-10T04:26:36.583151+0000","last_clean":"2022-09-10T04:26:36.583151+0000","last_became_active":"2022-09-10T04:26:32.117526+0000","last_became_peered":"2022-09-10T04:26:32.117526+0000","last_unstale":"2022-09-10T04:26:36.583151+0000","last_undegraded":"2022-09-10T04:26:36.583151+0000","last_fullsized":"2022-09-10T04:26:36.583151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:02:45.194870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582629+0000","last_change":"2022-09-10T04:26:32.122937+0000","last_active":"2022-09-10T04:26:36.582629+0000","last_peered":"2022-09-10T04:26:36.582629+0000","last_clean":"2022-09-10T04:26:36.582629+0000","last_became_active":"2022-09-10T04:26:32.122549+0000","last_became_peered":"2022-09-10T04:26:32.122549+0000","last_unstale":"2022-09-10T04:26:36.582629+0000","last_undegraded":"2022-09-10T04:26:36.582629+0000","last_fullsized":"2022-09-10T04:26:36.582629+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:19:48.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577863+0000","last_change":"2022-09-10T04:26:32.139841+0000","last_active":"2022-09-10T04:26:36.577863+0000","last_peered":"2022-09-10T04:26:36.577863+0000","last_clean":"2022-09-10T04:26:36.577863+0000","last_became_active":"2022-09-10T04:26:32.138962+0000","last_became_peered":"2022-09-10T04:26:32.138962+0000","last_unstale":"2022-09-10T04:26:36.577863+0000","last_undegraded":"2022-09-10T04:26:36.577863+0000","last_fullsized":"2022-09-10T04:26:36.577863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:51.196716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.243899+0000","last_change":"2022-09-10T04:26:36.594956+0000","last_active":"2022-09-10T04:26:37.243899+0000","last_peered":"2022-09-10T04:26:37.243899+0000","last_clean":"2022-09-10T04:26:37.243899+0000","last_became_active":"2022-09-10T04:26:36.594507+0000","last_became_peered":"2022-09-10T04:26:36.594507+0000","last_unstale":"2022-09-10T04:26:37.243899+0000","last_undegraded":"2022-09-10T04:26:37.243899+0000","last_fullsized":"2022-09-10T04:26:37.243899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:05:36.037017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:58.191392+0000","last_change":"2022-09-10T04:26:31.553482+0000","last_active":"2022-09-10T04:26:58.191392+0000","last_peered":"2022-09-10T04:26:58.191392+0000","last_clean":"2022-09-10T04:26:58.191392+0000","last_became_active":"2022-09-10T04:26:31.553170+0000","last_became_peered":"2022-09-10T04:26:31.553170+0000","last_unstale":"2022-09-10T04:26:58.191392+0000","last_undegraded":"2022-09-10T04:26:58.191392+0000","last_fullsized":"2022-09-10T04:26:58.191392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:52:35.054369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580763+0000","last_change":"2022-09-10T04:26:32.139920+0000","last_active":"2022-09-10T04:26:36.580763+0000","last_peered":"2022-09-10T04:26:36.580763+0000","last_clean":"2022-09-10T04:26:36.580763+0000","last_became_active":"2022-09-10T04:26:32.139414+0000","last_became_peered":"2022-09-10T04:26:32.139414+0000","last_unstale":"2022-09-10T04:26:36.580763+0000","last_undegraded":"2022-09-10T04:26:36.580763+0000","last_fullsized":"2022-09-10T04:26:36.580763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:12.288511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119112+0000","last_change":"2022-09-10T04:26:36.603332+0000","last_active":"2022-09-10T04:26:37.119112+0000","last_peered":"2022-09-10T04:26:37.119112+0000","last_clean":"2022-09-10T04:26:37.119112+0000","last_became_active":"2022-09-10T04:26:36.602947+0000","last_became_peered":"2022-09-10T04:26:36.602947+0000","last_unstale":"2022-09-10T04:26:37.119112+0000","last_undegraded":"2022-09-10T04:26:37.119112+0000","last_fullsized":"2022-09-10T04:26:37.119112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:20:15.790086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581013+0000","last_change":"2022-09-10T04:26:31.559001+0000","last_active":"2022-09-10T04:26:36.581013+0000","last_peered":"2022-09-10T04:26:36.581013+0000","last_clean":"2022-09-10T04:26:36.581013+0000","last_became_active":"2022-09-10T04:26:31.558675+0000","last_became_peered":"2022-09-10T04:26:31.558675+0000","last_unstale":"2022-09-10T04:26:36.581013+0000","last_undegraded":"2022-09-10T04:26:36.581013+0000","last_fullsized":"2022-09-10T04:26:36.581013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:31:20.091765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.596004+0000","last_change":"2022-09-10T04:26:32.122545+0000","last_active":"2022-09-10T04:26:36.596004+0000","last_peered":"2022-09-10T04:26:36.596004+0000","last_clean":"2022-09-10T04:26:36.596004+0000","last_became_active":"2022-09-10T04:26:32.122237+0000","last_became_peered":"2022-09-10T04:26:32.122237+0000","last_unstale":"2022-09-10T04:26:36.596004+0000","last_undegraded":"2022-09-10T04:26:36.596004+0000","last_fullsized":"2022-09-10T04:26:36.596004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:16:01.235524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119323+0000","last_change":"2022-09-10T04:26:36.603544+0000","last_active":"2022-09-10T04:26:37.119323+0000","last_peered":"2022-09-10T04:26:37.119323+0000","last_clean":"2022-09-10T04:26:37.119323+0000","last_became_active":"2022-09-10T04:26:36.602905+0000","last_became_peered":"2022-09-10T04:26:36.602905+0000","last_unstale":"2022-09-10T04:26:37.119323+0000","last_undegraded":"2022-09-10T04:26:37.119323+0000","last_fullsized":"2022-09-10T04:26:37.119323+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:17:44.678651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580890+0000","last_change":"2022-09-10T04:26:31.551550+0000","last_active":"2022-09-10T04:26:36.580890+0000","last_peered":"2022-09-10T04:26:36.580890+0000","last_clean":"2022-09-10T04:26:36.580890+0000","last_became_active":"2022-09-10T04:26:31.551161+0000","last_became_peered":"2022-09-10T04:26:31.551161+0000","last_unstale":"2022-09-10T04:26:36.580890+0000","last_undegraded":"2022-09-10T04:26:36.580890+0000","last_fullsized":"2022-09-10T04:26:36.580890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:59:32.430914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582594+0000","last_change":"2022-09-10T04:26:32.122935+0000","last_active":"2022-09-10T04:26:36.582594+0000","last_peered":"2022-09-10T04:26:36.582594+0000","last_clean":"2022-09-10T04:26:36.582594+0000","last_became_active":"2022-09-10T04:26:32.122471+0000","last_became_peered":"2022-09-10T04:26:32.122471+0000","last_unstale":"2022-09-10T04:26:36.582594+0000","last_undegraded":"2022-09-10T04:26:36.582594+0000","last_fullsized":"2022-09-10T04:26:36.582594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:29:29.965140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119344+0000","last_change":"2022-09-10T04:26:36.604924+0000","last_active":"2022-09-10T04:26:37.119344+0000","last_peered":"2022-09-10T04:26:37.119344+0000","last_clean":"2022-09-10T04:26:37.119344+0000","last_became_active":"2022-09-10T04:26:36.604620+0000","last_became_peered":"2022-09-10T04:26:36.604620+0000","last_unstale":"2022-09-10T04:26:37.119344+0000","last_undegraded":"2022-09-10T04:26:37.119344+0000","last_fullsized":"2022-09-10T04:26:37.119344+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:41:34.668107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581512+0000","last_change":"2022-09-10T04:26:31.547564+0000","last_active":"2022-09-10T04:26:36.581512+0000","last_peered":"2022-09-10T04:26:36.581512+0000","last_clean":"2022-09-10T04:26:36.581512+0000","last_became_active":"2022-09-10T04:26:31.547180+0000","last_became_peered":"2022-09-10T04:26:31.547180+0000","last_unstale":"2022-09-10T04:26:36.581512+0000","last_undegraded":"2022-09-10T04:26:36.581512+0000","last_fullsized":"2022-09-10T04:26:36.581512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:23:12.454750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582914+0000","last_change":"2022-09-10T04:26:32.119307+0000","last_active":"2022-09-10T04:26:36.582914+0000","last_peered":"2022-09-10T04:26:36.582914+0000","last_clean":"2022-09-10T04:26:36.582914+0000","last_became_active":"2022-09-10T04:26:32.118995+0000","last_became_peered":"2022-09-10T04:26:32.118995+0000","last_unstale":"2022-09-10T04:26:36.582914+0000","last_undegraded":"2022-09-10T04:26:36.582914+0000","last_fullsized":"2022-09-10T04:26:36.582914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:25.888725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.244142+0000","last_change":"2022-09-10T04:26:36.591310+0000","last_active":"2022-09-10T04:26:37.244142+0000","last_peered":"2022-09-10T04:26:37.244142+0000","last_clean":"2022-09-10T04:26:37.244142+0000","last_became_active":"2022-09-10T04:26:36.590907+0000","last_became_peered":"2022-09-10T04:26:36.590907+0000","last_unstale":"2022-09-10T04:26:37.244142+0000","last_undegraded":"2022-09-10T04:26:37.244142+0000","last_fullsized":"2022-09-10T04:26:37.244142+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:05:45.016925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582839+0000","last_change":"2022-09-10T04:26:31.550524+0000","last_active":"2022-09-10T04:26:36.582839+0000","last_peered":"2022-09-10T04:26:36.582839+0000","last_clean":"2022-09-10T04:26:36.582839+0000","last_became_active":"2022-09-10T04:26:31.550259+0000","last_became_peered":"2022-09-10T04:26:31.550259+0000","last_unstale":"2022-09-10T04:26:36.582839+0000","last_undegraded":"2022-09-10T04:26:36.582839+0000","last_fullsized":"2022-09-10T04:26:36.582839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:53.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585280+0000","last_change":"2022-09-10T04:26:32.123377+0000","last_active":"2022-09-10T04:26:36.585280+0000","last_peered":"2022-09-10T04:26:36.585280+0000","last_clean":"2022-09-10T04:26:36.585280+0000","last_became_active":"2022-09-10T04:26:32.123076+0000","last_became_peered":"2022-09-10T04:26:32.123076+0000","last_unstale":"2022-09-10T04:26:36.585280+0000","last_undegraded":"2022-09-10T04:26:36.585280+0000","last_fullsized":"2022-09-10T04:26:36.585280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:40.425861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.244104+0000","last_change":"2022-09-10T04:26:36.593491+0000","last_active":"2022-09-10T04:26:37.244104+0000","last_peered":"2022-09-10T04:26:37.244104+0000","last_clean":"2022-09-10T04:26:37.244104+0000","last_became_active":"2022-09-10T04:26:36.593064+0000","last_became_peered":"2022-09-10T04:26:36.593064+0000","last_unstale":"2022-09-10T04:26:37.244104+0000","last_undegraded":"2022-09-10T04:26:37.244104+0000","last_fullsized":"2022-09-10T04:26:37.244104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:03:44.752739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789087+0000","last_change":"2022-09-10T04:26:31.555492+0000","last_active":"2022-09-10T04:26:36.789087+0000","last_peered":"2022-09-10T04:26:36.789087+0000","last_clean":"2022-09-10T04:26:36.789087+0000","last_became_active":"2022-09-10T04:26:31.555216+0000","last_became_peered":"2022-09-10T04:26:31.555216+0000","last_unstale":"2022-09-10T04:26:36.789087+0000","last_undegraded":"2022-09-10T04:26:36.789087+0000","last_fullsized":"2022-09-10T04:26:36.789087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:50:32.775631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582937+0000","last_change":"2022-09-10T04:26:32.139851+0000","last_active":"2022-09-10T04:26:36.582937+0000","last_peered":"2022-09-10T04:26:36.582937+0000","last_clean":"2022-09-10T04:26:36.582937+0000","last_became_active":"2022-09-10T04:26:32.139515+0000","last_became_peered":"2022-09-10T04:26:32.139515+0000","last_unstale":"2022-09-10T04:26:36.582937+0000","last_undegraded":"2022-09-10T04:26:36.582937+0000","last_fullsized":"2022-09-10T04:26:36.582937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:46:04.342743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.100507+0000","last_change":"2022-09-10T04:26:36.598406+0000","last_active":"2022-09-10T04:26:38.100507+0000","last_peered":"2022-09-10T04:26:38.100507+0000","last_clean":"2022-09-10T04:26:38.100507+0000","last_became_active":"2022-09-10T04:26:36.598118+0000","last_became_peered":"2022-09-10T04:26:36.598118+0000","last_unstale":"2022-09-10T04:26:38.100507+0000","last_undegraded":"2022-09-10T04:26:38.100507+0000","last_fullsized":"2022-09-10T04:26:38.100507+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:35.903423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.461177+0000","last_change":"2022-09-10T04:26:31.541593+0000","last_active":"2022-09-10T04:26:56.461177+0000","last_peered":"2022-09-10T04:26:56.461177+0000","last_clean":"2022-09-10T04:26:56.461177+0000","last_became_active":"2022-09-10T04:26:31.541236+0000","last_became_peered":"2022-09-10T04:26:31.541236+0000","last_unstale":"2022-09-10T04:26:56.461177+0000","last_undegraded":"2022-09-10T04:26:56.461177+0000","last_fullsized":"2022-09-10T04:26:56.461177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:30:03.723483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581002+0000","last_change":"2022-09-10T04:26:32.147698+0000","last_active":"2022-09-10T04:26:36.581002+0000","last_peered":"2022-09-10T04:26:36.581002+0000","last_clean":"2022-09-10T04:26:36.581002+0000","last_became_active":"2022-09-10T04:26:32.147334+0000","last_became_peered":"2022-09-10T04:26:32.147334+0000","last_unstale":"2022-09-10T04:26:36.581002+0000","last_undegraded":"2022-09-10T04:26:36.581002+0000","last_fullsized":"2022-09-10T04:26:36.581002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:33:21.907371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.988827+0000","last_change":"2022-09-10T04:26:36.597450+0000","last_active":"2022-09-10T04:26:38.988827+0000","last_peered":"2022-09-10T04:26:38.988827+0000","last_clean":"2022-09-10T04:26:38.988827+0000","last_became_active":"2022-09-10T04:26:36.597087+0000","last_became_peered":"2022-09-10T04:26:36.597087+0000","last_unstale":"2022-09-10T04:26:38.988827+0000","last_undegraded":"2022-09-10T04:26:38.988827+0000","last_fullsized":"2022-09-10T04:26:38.988827+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:27.788857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583682+0000","last_change":"2022-09-10T04:26:31.550451+0000","last_active":"2022-09-10T04:26:36.583682+0000","last_peered":"2022-09-10T04:26:36.583682+0000","last_clean":"2022-09-10T04:26:36.583682+0000","last_became_active":"2022-09-10T04:26:31.550158+0000","last_became_peered":"2022-09-10T04:26:31.550158+0000","last_unstale":"2022-09-10T04:26:36.583682+0000","last_undegraded":"2022-09-10T04:26:36.583682+0000","last_fullsized":"2022-09-10T04:26:36.583682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:21:27.132856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580285+0000","last_change":"2022-09-10T04:26:32.136694+0000","last_active":"2022-09-10T04:26:36.580285+0000","last_peered":"2022-09-10T04:26:36.580285+0000","last_clean":"2022-09-10T04:26:36.580285+0000","last_became_active":"2022-09-10T04:26:32.136398+0000","last_became_peered":"2022-09-10T04:26:32.136398+0000","last_unstale":"2022-09-10T04:26:36.580285+0000","last_undegraded":"2022-09-10T04:26:36.580285+0000","last_fullsized":"2022-09-10T04:26:36.580285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:32:22.413618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.602343+0000","last_change":"2022-09-10T04:26:36.599433+0000","last_active":"2022-09-10T04:26:36.602343+0000","last_peered":"2022-09-10T04:26:36.602343+0000","last_clean":"2022-09-10T04:26:36.602343+0000","last_became_active":"2022-09-10T04:26:36.599013+0000","last_became_peered":"2022-09-10T04:26:36.599013+0000","last_unstale":"2022-09-10T04:26:36.602343+0000","last_undegraded":"2022-09-10T04:26:36.602343+0000","last_fullsized":"2022-09-10T04:26:36.602343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:23:18.397219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789436+0000","last_change":"2022-09-10T04:26:31.548631+0000","last_active":"2022-09-10T04:26:36.789436+0000","last_peered":"2022-09-10T04:26:36.789436+0000","last_clean":"2022-09-10T04:26:36.789436+0000","last_became_active":"2022-09-10T04:26:31.548292+0000","last_became_peered":"2022-09-10T04:26:31.548292+0000","last_unstale":"2022-09-10T04:26:36.789436+0000","last_undegraded":"2022-09-10T04:26:36.789436+0000","last_fullsized":"2022-09-10T04:26:36.789436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:40.825422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583168+0000","last_change":"2022-09-10T04:26:32.124469+0000","last_active":"2022-09-10T04:26:36.583168+0000","last_peered":"2022-09-10T04:26:36.583168+0000","last_clean":"2022-09-10T04:26:36.583168+0000","last_became_active":"2022-09-10T04:26:32.124084+0000","last_became_peered":"2022-09-10T04:26:32.124084+0000","last_unstale":"2022-09-10T04:26:36.583168+0000","last_undegraded":"2022-09-10T04:26:36.583168+0000","last_fullsized":"2022-09-10T04:26:36.583168+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:39:04.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420161+0000","last_change":"2022-09-10T04:26:36.601607+0000","last_active":"2022-09-10T04:26:38.420161+0000","last_peered":"2022-09-10T04:26:38.420161+0000","last_clean":"2022-09-10T04:26:38.420161+0000","last_became_active":"2022-09-10T04:26:36.601191+0000","last_became_peered":"2022-09-10T04:26:36.601191+0000","last_unstale":"2022-09-10T04:26:38.420161+0000","last_undegraded":"2022-09-10T04:26:38.420161+0000","last_fullsized":"2022-09-10T04:26:38.420161+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:52:58.815905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.455549+0000","last_change":"2022-09-10T04:26:31.548524+0000","last_active":"2022-09-10T04:26:55.455549+0000","last_peered":"2022-09-10T04:26:55.455549+0000","last_clean":"2022-09-10T04:26:55.455549+0000","last_became_active":"2022-09-10T04:26:31.548073+0000","last_became_peered":"2022-09-10T04:26:31.548073+0000","last_unstale":"2022-09-10T04:26:55.455549+0000","last_undegraded":"2022-09-10T04:26:55.455549+0000","last_fullsized":"2022-09-10T04:26:55.455549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:01:02.489915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.596090+0000","last_change":"2022-09-10T04:26:32.120523+0000","last_active":"2022-09-10T04:26:36.596090+0000","last_peered":"2022-09-10T04:26:36.596090+0000","last_clean":"2022-09-10T04:26:36.596090+0000","last_became_active":"2022-09-10T04:26:32.120076+0000","last_became_peered":"2022-09-10T04:26:32.120076+0000","last_unstale":"2022-09-10T04:26:36.596090+0000","last_undegraded":"2022-09-10T04:26:36.596090+0000","last_fullsized":"2022-09-10T04:26:36.596090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:26:07.907475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119606+0000","last_change":"2022-09-10T04:26:36.601632+0000","last_active":"2022-09-10T04:26:37.119606+0000","last_peered":"2022-09-10T04:26:37.119606+0000","last_clean":"2022-09-10T04:26:37.119606+0000","last_became_active":"2022-09-10T04:26:36.601237+0000","last_became_peered":"2022-09-10T04:26:36.601237+0000","last_unstale":"2022-09-10T04:26:37.119606+0000","last_undegraded":"2022-09-10T04:26:37.119606+0000","last_fullsized":"2022-09-10T04:26:37.119606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:08:34.270892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582804+0000","last_change":"2022-09-10T04:26:31.557753+0000","last_active":"2022-09-10T04:26:36.582804+0000","last_peered":"2022-09-10T04:26:36.582804+0000","last_clean":"2022-09-10T04:26:36.582804+0000","last_became_active":"2022-09-10T04:26:31.557432+0000","last_became_peered":"2022-09-10T04:26:31.557432+0000","last_unstale":"2022-09-10T04:26:36.582804+0000","last_undegraded":"2022-09-10T04:26:36.582804+0000","last_fullsized":"2022-09-10T04:26:36.582804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:14:51.954481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582777+0000","last_change":"2022-09-10T04:26:32.122753+0000","last_active":"2022-09-10T04:26:36.582777+0000","last_peered":"2022-09-10T04:26:36.582777+0000","last_clean":"2022-09-10T04:26:36.582777+0000","last_became_active":"2022-09-10T04:26:32.122052+0000","last_became_peered":"2022-09-10T04:26:32.122052+0000","last_unstale":"2022-09-10T04:26:36.582777+0000","last_undegraded":"2022-09-10T04:26:36.582777+0000","last_fullsized":"2022-09-10T04:26:36.582777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:00:07.053739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420184+0000","last_change":"2022-09-10T04:26:36.591159+0000","last_active":"2022-09-10T04:26:38.420184+0000","last_peered":"2022-09-10T04:26:38.420184+0000","last_clean":"2022-09-10T04:26:38.420184+0000","last_became_active":"2022-09-10T04:26:36.590788+0000","last_became_peered":"2022-09-10T04:26:36.590788+0000","last_unstale":"2022-09-10T04:26:38.420184+0000","last_undegraded":"2022-09-10T04:26:38.420184+0000","last_fullsized":"2022-09-10T04:26:38.420184+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:50:34.597783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'10","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:52.513644+0000","last_change":"2022-09-10T04:26:31.547702+0000","last_active":"2022-09-10T04:26:52.513644+0000","last_peered":"2022-09-10T04:26:52.513644+0000","last_clean":"2022-09-10T04:26:52.513644+0000","last_became_active":"2022-09-10T04:26:31.547388+0000","last_became_peered":"2022-09-10T04:26:31.547388+0000","last_unstale":"2022-09-10T04:26:52.513644+0000","last_undegraded":"2022-09-10T04:26:52.513644+0000","last_fullsized":"2022-09-10T04:26:52.513644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:42:28.378515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3749,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580578+0000","last_change":"2022-09-10T04:26:32.117260+0000","last_active":"2022-09-10T04:26:36.580578+0000","last_peered":"2022-09-10T04:26:36.580578+0000","last_clean":"2022-09-10T04:26:36.580578+0000","last_became_active":"2022-09-10T04:26:32.116928+0000","last_became_peered":"2022-09-10T04:26:32.116928+0000","last_unstale":"2022-09-10T04:26:36.580578+0000","last_undegraded":"2022-09-10T04:26:36.580578+0000","last_fullsized":"2022-09-10T04:26:36.580578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:00:10.528509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.288747+0000","last_change":"2022-09-10T04:26:36.600239+0000","last_active":"2022-09-10T04:26:37.288747+0000","last_peered":"2022-09-10T04:26:37.288747+0000","last_clean":"2022-09-10T04:26:37.288747+0000","last_became_active":"2022-09-10T04:26:36.599614+0000","last_became_peered":"2022-09-10T04:26:36.599614+0000","last_unstale":"2022-09-10T04:26:37.288747+0000","last_undegraded":"2022-09-10T04:26:37.288747+0000","last_fullsized":"2022-09-10T04:26:37.288747+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:36.535130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:58.191682+0000","last_change":"2022-09-10T04:26:31.552412+0000","last_active":"2022-09-10T04:26:58.191682+0000","last_peered":"2022-09-10T04:26:58.191682+0000","last_clean":"2022-09-10T04:26:58.191682+0000","last_became_active":"2022-09-10T04:26:31.552145+0000","last_became_peered":"2022-09-10T04:26:31.552145+0000","last_unstale":"2022-09-10T04:26:58.191682+0000","last_undegraded":"2022-09-10T04:26:58.191682+0000","last_fullsized":"2022-09-10T04:26:58.191682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:16:36.041950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585882+0000","last_change":"2022-09-10T04:26:32.136276+0000","last_active":"2022-09-10T04:26:36.585882+0000","last_peered":"2022-09-10T04:26:36.585882+0000","last_clean":"2022-09-10T04:26:36.585882+0000","last_became_active":"2022-09-10T04:26:32.135894+0000","last_became_peered":"2022-09-10T04:26:32.135894+0000","last_unstale":"2022-09-10T04:26:36.585882+0000","last_undegraded":"2022-09-10T04:26:36.585882+0000","last_fullsized":"2022-09-10T04:26:36.585882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:12:32.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.100262+0000","last_change":"2022-09-10T04:26:36.600069+0000","last_active":"2022-09-10T04:26:38.100262+0000","last_peered":"2022-09-10T04:26:38.100262+0000","last_clean":"2022-09-10T04:26:38.100262+0000","last_became_active":"2022-09-10T04:26:36.599592+0000","last_became_peered":"2022-09-10T04:26:36.599592+0000","last_unstale":"2022-09-10T04:26:38.100262+0000","last_undegraded":"2022-09-10T04:26:38.100262+0000","last_fullsized":"2022-09-10T04:26:38.100262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:25:58.722442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582707+0000","last_change":"2022-09-10T04:26:31.556823+0000","last_active":"2022-09-10T04:26:36.582707+0000","last_peered":"2022-09-10T04:26:36.582707+0000","last_clean":"2022-09-10T04:26:36.582707+0000","last_became_active":"2022-09-10T04:26:31.556509+0000","last_became_peered":"2022-09-10T04:26:31.556509+0000","last_unstale":"2022-09-10T04:26:36.582707+0000","last_undegraded":"2022-09-10T04:26:36.582707+0000","last_fullsized":"2022-09-10T04:26:36.582707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:35:16.227863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585116+0000","last_change":"2022-09-10T04:26:32.130565+0000","last_active":"2022-09-10T04:26:36.585116+0000","last_peered":"2022-09-10T04:26:36.585116+0000","last_clean":"2022-09-10T04:26:36.585116+0000","last_became_active":"2022-09-10T04:26:32.130121+0000","last_became_peered":"2022-09-10T04:26:32.130121+0000","last_unstale":"2022-09-10T04:26:36.585116+0000","last_undegraded":"2022-09-10T04:26:36.585116+0000","last_fullsized":"2022-09-10T04:26:36.585116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:55:03.504615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420227+0000","last_change":"2022-09-10T04:26:36.603262+0000","last_active":"2022-09-10T04:26:38.420227+0000","last_peered":"2022-09-10T04:26:38.420227+0000","last_clean":"2022-09-10T04:26:38.420227+0000","last_became_active":"2022-09-10T04:26:36.602590+0000","last_became_peered":"2022-09-10T04:26:36.602590+0000","last_unstale":"2022-09-10T04:26:38.420227+0000","last_undegraded":"2022-09-10T04:26:38.420227+0000","last_fullsized":"2022-09-10T04:26:38.420227+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:19:03.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.459902+0000","last_change":"2022-09-10T04:26:31.545898+0000","last_active":"2022-09-10T04:26:56.459902+0000","last_peered":"2022-09-10T04:26:56.459902+0000","last_clean":"2022-09-10T04:26:56.459902+0000","last_became_active":"2022-09-10T04:26:31.545355+0000","last_became_peered":"2022-09-10T04:26:31.545355+0000","last_unstale":"2022-09-10T04:26:56.459902+0000","last_undegraded":"2022-09-10T04:26:56.459902+0000","last_fullsized":"2022-09-10T04:26:56.459902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:50:36.187534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585167+0000","last_change":"2022-09-10T04:26:32.116268+0000","last_active":"2022-09-10T04:26:36.585167+0000","last_peered":"2022-09-10T04:26:36.585167+0000","last_clean":"2022-09-10T04:26:36.585167+0000","last_became_active":"2022-09-10T04:26:32.115133+0000","last_became_peered":"2022-09-10T04:26:32.115133+0000","last_unstale":"2022-09-10T04:26:36.585167+0000","last_undegraded":"2022-09-10T04:26:36.585167+0000","last_fullsized":"2022-09-10T04:26:36.585167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:38:05.319082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.606364+0000","last_change":"2022-09-10T04:26:36.605212+0000","last_active":"2022-09-10T04:26:36.606364+0000","last_peered":"2022-09-10T04:26:36.606364+0000","last_clean":"2022-09-10T04:26:36.606364+0000","last_became_active":"2022-09-10T04:26:36.604767+0000","last_became_peered":"2022-09-10T04:26:36.604767+0000","last_unstale":"2022-09-10T04:26:36.606364+0000","last_undegraded":"2022-09-10T04:26:36.606364+0000","last_fullsized":"2022-09-10T04:26:36.606364+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:33:55.048936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.454477+0000","last_change":"2022-09-10T04:26:31.561097+0000","last_active":"2022-09-10T04:26:55.454477+0000","last_peered":"2022-09-10T04:26:55.454477+0000","last_clean":"2022-09-10T04:26:55.454477+0000","last_became_active":"2022-09-10T04:26:31.560748+0000","last_became_peered":"2022-09-10T04:26:31.560748+0000","last_unstale":"2022-09-10T04:26:55.454477+0000","last_undegraded":"2022-09-10T04:26:55.454477+0000","last_fullsized":"2022-09-10T04:26:55.454477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:44:32.921350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582638+0000","last_change":"2022-09-10T04:26:32.122012+0000","last_active":"2022-09-10T04:26:36.582638+0000","last_peered":"2022-09-10T04:26:36.582638+0000","last_clean":"2022-09-10T04:26:36.582638+0000","last_became_active":"2022-09-10T04:26:32.121668+0000","last_became_peered":"2022-09-10T04:26:32.121668+0000","last_unstale":"2022-09-10T04:26:36.582638+0000","last_undegraded":"2022-09-10T04:26:36.582638+0000","last_fullsized":"2022-09-10T04:26:36.582638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:49:57.312550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.288088+0000","last_change":"2022-09-10T04:26:36.594964+0000","last_active":"2022-09-10T04:26:37.288088+0000","last_peered":"2022-09-10T04:26:37.288088+0000","last_clean":"2022-09-10T04:26:37.288088+0000","last_became_active":"2022-09-10T04:26:36.594548+0000","last_became_peered":"2022-09-10T04:26:36.594548+0000","last_unstale":"2022-09-10T04:26:37.288088+0000","last_undegraded":"2022-09-10T04:26:37.288088+0000","last_fullsized":"2022-09-10T04:26:37.288088+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:01.446152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580676+0000","last_change":"2022-09-10T04:26:31.549608+0000","last_active":"2022-09-10T04:26:36.580676+0000","last_peered":"2022-09-10T04:26:36.580676+0000","last_clean":"2022-09-10T04:26:36.580676+0000","last_became_active":"2022-09-10T04:26:31.549300+0000","last_became_peered":"2022-09-10T04:26:31.549300+0000","last_unstale":"2022-09-10T04:26:36.580676+0000","last_undegraded":"2022-09-10T04:26:36.580676+0000","last_fullsized":"2022-09-10T04:26:36.580676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:53:51.183777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582208+0000","last_change":"2022-09-10T04:26:32.119540+0000","last_active":"2022-09-10T04:26:36.582208+0000","last_peered":"2022-09-10T04:26:36.582208+0000","last_clean":"2022-09-10T04:26:36.582208+0000","last_became_active":"2022-09-10T04:26:32.119341+0000","last_became_peered":"2022-09-10T04:26:32.119341+0000","last_unstale":"2022-09-10T04:26:36.582208+0000","last_undegraded":"2022-09-10T04:26:36.582208+0000","last_fullsized":"2022-09-10T04:26:36.582208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:21:13.815142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.791499+0000","last_change":"2022-09-10T04:26:36.790770+0000","last_active":"2022-09-10T04:26:36.791499+0000","last_peered":"2022-09-10T04:26:36.791499+0000","last_clean":"2022-09-10T04:26:36.791499+0000","last_became_active":"2022-09-10T04:26:36.790366+0000","last_became_peered":"2022-09-10T04:26:36.790366+0000","last_unstale":"2022-09-10T04:26:36.791499+0000","last_undegraded":"2022-09-10T04:26:36.791499+0000","last_fullsized":"2022-09-10T04:26:36.791499+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:40:18.004852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.642044+0000","last_change":"2022-09-10T04:26:31.557145+0000","last_active":"2022-09-10T04:26:38.642044+0000","last_peered":"2022-09-10T04:26:38.642044+0000","last_clean":"2022-09-10T04:26:38.642044+0000","last_became_active":"2022-09-10T04:26:31.556805+0000","last_became_peered":"2022-09-10T04:26:31.556805+0000","last_unstale":"2022-09-10T04:26:38.642044+0000","last_undegraded":"2022-09-10T04:26:38.642044+0000","last_fullsized":"2022-09-10T04:26:38.642044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:53:41.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789300+0000","last_change":"2022-09-10T04:26:32.139758+0000","last_active":"2022-09-10T04:26:36.789300+0000","last_peered":"2022-09-10T04:26:36.789300+0000","last_clean":"2022-09-10T04:26:36.789300+0000","last_became_active":"2022-09-10T04:26:32.139469+0000","last_became_peered":"2022-09-10T04:26:32.139469+0000","last_unstale":"2022-09-10T04:26:36.789300+0000","last_undegraded":"2022-09-10T04:26:36.789300+0000","last_fullsized":"2022-09-10T04:26:36.789300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:09:52.224744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.606481+0000","last_change":"2022-09-10T04:26:36.600083+0000","last_active":"2022-09-10T04:26:36.606481+0000","last_peered":"2022-09-10T04:26:36.606481+0000","last_clean":"2022-09-10T04:26:36.606481+0000","last_became_active":"2022-09-10T04:26:36.599648+0000","last_became_peered":"2022-09-10T04:26:36.599648+0000","last_unstale":"2022-09-10T04:26:36.606481+0000","last_undegraded":"2022-09-10T04:26:36.606481+0000","last_fullsized":"2022-09-10T04:26:36.606481+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:03:24.897645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.231607+0000","last_change":"2022-09-10T04:26:31.560147+0000","last_active":"2022-09-10T04:26:38.231607+0000","last_peered":"2022-09-10T04:26:38.231607+0000","last_clean":"2022-09-10T04:26:38.231607+0000","last_became_active":"2022-09-10T04:26:31.559868+0000","last_became_peered":"2022-09-10T04:26:31.559868+0000","last_unstale":"2022-09-10T04:26:38.231607+0000","last_undegraded":"2022-09-10T04:26:38.231607+0000","last_fullsized":"2022-09-10T04:26:38.231607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:02:46.652743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.789118+0000","last_change":"2022-09-10T04:26:32.125369+0000","last_active":"2022-09-10T04:26:36.789118+0000","last_peered":"2022-09-10T04:26:36.789118+0000","last_clean":"2022-09-10T04:26:36.789118+0000","last_became_active":"2022-09-10T04:26:32.125077+0000","last_became_peered":"2022-09-10T04:26:32.125077+0000","last_unstale":"2022-09-10T04:26:36.789118+0000","last_undegraded":"2022-09-10T04:26:36.789118+0000","last_fullsized":"2022-09-10T04:26:36.789118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:03:13.265143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.791633+0000","last_change":"2022-09-10T04:26:36.790278+0000","last_active":"2022-09-10T04:26:36.791633+0000","last_peered":"2022-09-10T04:26:36.791633+0000","last_clean":"2022-09-10T04:26:36.791633+0000","last_became_active":"2022-09-10T04:26:36.789852+0000","last_became_peered":"2022-09-10T04:26:36.789852+0000","last_unstale":"2022-09-10T04:26:36.791633+0000","last_undegraded":"2022-09-10T04:26:36.791633+0000","last_fullsized":"2022-09-10T04:26:36.791633+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:26:38.960022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583882+0000","last_change":"2022-09-10T04:26:32.119857+0000","last_active":"2022-09-10T04:26:36.583882+0000","last_peered":"2022-09-10T04:26:36.583882+0000","last_clean":"2022-09-10T04:26:36.583882+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:26:36.583882+0000","last_undegraded":"2022-09-10T04:26:36.583882+0000","last_fullsized":"2022-09-10T04:26:36.583882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:21:40.941165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.462233+0000","last_change":"2022-09-10T04:26:31.555699+0000","last_active":"2022-09-10T04:26:56.462233+0000","last_peered":"2022-09-10T04:26:56.462233+0000","last_clean":"2022-09-10T04:26:56.462233+0000","last_became_active":"2022-09-10T04:26:31.555337+0000","last_became_peered":"2022-09-10T04:26:31.555337+0000","last_unstale":"2022-09-10T04:26:56.462233+0000","last_undegraded":"2022-09-10T04:26:56.462233+0000","last_fullsized":"2022-09-10T04:26:56.462233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:38:23.689077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.298100+0000","last_change":"2022-09-10T04:26:36.592313+0000","last_active":"2022-09-10T04:26:37.298100+0000","last_peered":"2022-09-10T04:26:37.298100+0000","last_clean":"2022-09-10T04:26:37.298100+0000","last_became_active":"2022-09-10T04:26:36.591703+0000","last_became_peered":"2022-09-10T04:26:36.591703+0000","last_unstale":"2022-09-10T04:26:37.298100+0000","last_undegraded":"2022-09-10T04:26:37.298100+0000","last_fullsized":"2022-09-10T04:26:37.298100+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:19:26.704605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581155+0000","last_change":"2022-09-10T04:26:32.138958+0000","last_active":"2022-09-10T04:26:36.581155+0000","last_peered":"2022-09-10T04:26:36.581155+0000","last_clean":"2022-09-10T04:26:36.581155+0000","last_became_active":"2022-09-10T04:26:32.138655+0000","last_became_peered":"2022-09-10T04:26:32.138655+0000","last_unstale":"2022-09-10T04:26:36.581155+0000","last_undegraded":"2022-09-10T04:26:36.581155+0000","last_fullsized":"2022-09-10T04:26:36.581155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:25:07.759875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.455863+0000","last_change":"2022-09-10T04:26:31.544708+0000","last_active":"2022-09-10T04:26:55.455863+0000","last_peered":"2022-09-10T04:26:55.455863+0000","last_clean":"2022-09-10T04:26:55.455863+0000","last_became_active":"2022-09-10T04:26:31.544386+0000","last_became_peered":"2022-09-10T04:26:31.544386+0000","last_unstale":"2022-09-10T04:26:55.455863+0000","last_undegraded":"2022-09-10T04:26:55.455863+0000","last_fullsized":"2022-09-10T04:26:55.455863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:07:18.444015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":377,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584162+0000","last_change":"2022-09-10T04:25:23.357461+0000","last_active":"2022-09-10T04:26:36.584162+0000","last_peered":"2022-09-10T04:26:36.584162+0000","last_clean":"2022-09-10T04:26:36.584162+0000","last_became_active":"2022-09-10T04:25:23.356951+0000","last_became_peered":"2022-09-10T04:25:23.356951+0000","last_unstale":"2022-09-10T04:26:36.584162+0000","last_undegraded":"2022-09-10T04:26:36.584162+0000","last_fullsized":"2022-09-10T04:26:36.584162+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_clean_scrub_stamp":"2022-09-10T04:21:40.122270+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:01:20.972512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.288579+0000","last_change":"2022-09-10T04:26:36.790925+0000","last_active":"2022-09-10T04:26:37.288579+0000","last_peered":"2022-09-10T04:26:37.288579+0000","last_clean":"2022-09-10T04:26:37.288579+0000","last_became_active":"2022-09-10T04:26:36.790332+0000","last_became_peered":"2022-09-10T04:26:36.790332+0000","last_unstale":"2022-09-10T04:26:37.288579+0000","last_undegraded":"2022-09-10T04:26:37.288579+0000","last_fullsized":"2022-09-10T04:26:37.288579+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:10:55.153959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586285+0000","last_change":"2022-09-10T04:26:32.122356+0000","last_active":"2022-09-10T04:26:36.586285+0000","last_peered":"2022-09-10T04:26:36.586285+0000","last_clean":"2022-09-10T04:26:36.586285+0000","last_became_active":"2022-09-10T04:26:32.122028+0000","last_became_peered":"2022-09-10T04:26:32.122028+0000","last_unstale":"2022-09-10T04:26:36.586285+0000","last_undegraded":"2022-09-10T04:26:36.586285+0000","last_fullsized":"2022-09-10T04:26:36.586285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:33.837184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:56.460004+0000","last_change":"2022-09-10T04:26:31.556537+0000","last_active":"2022-09-10T04:26:56.460004+0000","last_peered":"2022-09-10T04:26:56.460004+0000","last_clean":"2022-09-10T04:26:56.460004+0000","last_became_active":"2022-09-10T04:26:31.556209+0000","last_became_peered":"2022-09-10T04:26:31.556209+0000","last_unstale":"2022-09-10T04:26:56.460004+0000","last_undegraded":"2022-09-10T04:26:56.460004+0000","last_fullsized":"2022-09-10T04:26:56.460004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:42:05.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.100221+0000","last_change":"2022-09-10T04:26:36.600051+0000","last_active":"2022-09-10T04:26:38.100221+0000","last_peered":"2022-09-10T04:26:38.100221+0000","last_clean":"2022-09-10T04:26:38.100221+0000","last_became_active":"2022-09-10T04:26:36.599603+0000","last_became_peered":"2022-09-10T04:26:36.599603+0000","last_unstale":"2022-09-10T04:26:38.100221+0000","last_undegraded":"2022-09-10T04:26:38.100221+0000","last_fullsized":"2022-09-10T04:26:38.100221+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:48:12.857851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.989022+0000","last_change":"2022-09-10T04:26:31.560780+0000","last_active":"2022-09-10T04:26:38.989022+0000","last_peered":"2022-09-10T04:26:38.989022+0000","last_clean":"2022-09-10T04:26:38.989022+0000","last_became_active":"2022-09-10T04:26:31.560462+0000","last_became_peered":"2022-09-10T04:26:31.560462+0000","last_unstale":"2022-09-10T04:26:38.989022+0000","last_undegraded":"2022-09-10T04:26:38.989022+0000","last_fullsized":"2022-09-10T04:26:38.989022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:05:15.437095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589341+0000","last_change":"2022-09-10T04:26:32.127579+0000","last_active":"2022-09-10T04:26:36.589341+0000","last_peered":"2022-09-10T04:26:36.589341+0000","last_clean":"2022-09-10T04:26:36.589341+0000","last_became_active":"2022-09-10T04:26:32.127237+0000","last_became_peered":"2022-09-10T04:26:32.127237+0000","last_unstale":"2022-09-10T04:26:36.589341+0000","last_undegraded":"2022-09-10T04:26:36.589341+0000","last_fullsized":"2022-09-10T04:26:36.589341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:09:28.061156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.791816+0000","last_change":"2022-09-10T04:26:36.790777+0000","last_active":"2022-09-10T04:26:36.791816+0000","last_peered":"2022-09-10T04:26:36.791816+0000","last_clean":"2022-09-10T04:26:36.791816+0000","last_became_active":"2022-09-10T04:26:36.789974+0000","last_became_peered":"2022-09-10T04:26:36.789974+0000","last_unstale":"2022-09-10T04:26:36.791816+0000","last_undegraded":"2022-09-10T04:26:36.791816+0000","last_fullsized":"2022-09-10T04:26:36.791816+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:41:20.214040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:57.470209+0000","last_change":"2022-09-10T04:26:31.552253+0000","last_active":"2022-09-10T04:26:57.470209+0000","last_peered":"2022-09-10T04:26:57.470209+0000","last_clean":"2022-09-10T04:26:57.470209+0000","last_became_active":"2022-09-10T04:26:31.551846+0000","last_became_peered":"2022-09-10T04:26:31.551846+0000","last_unstale":"2022-09-10T04:26:57.470209+0000","last_undegraded":"2022-09-10T04:26:57.470209+0000","last_fullsized":"2022-09-10T04:26:57.470209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:28:49.941688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581283+0000","last_change":"2022-09-10T04:26:32.145463+0000","last_active":"2022-09-10T04:26:36.581283+0000","last_peered":"2022-09-10T04:26:36.581283+0000","last_clean":"2022-09-10T04:26:36.581283+0000","last_became_active":"2022-09-10T04:26:32.145115+0000","last_became_peered":"2022-09-10T04:26:32.145115+0000","last_unstale":"2022-09-10T04:26:36.581283+0000","last_undegraded":"2022-09-10T04:26:36.581283+0000","last_fullsized":"2022-09-10T04:26:36.581283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:11.597005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.243771+0000","last_change":"2022-09-10T04:26:36.594514+0000","last_active":"2022-09-10T04:26:37.243771+0000","last_peered":"2022-09-10T04:26:37.243771+0000","last_clean":"2022-09-10T04:26:37.243771+0000","last_became_active":"2022-09-10T04:26:36.594092+0000","last_became_peered":"2022-09-10T04:26:36.594092+0000","last_unstale":"2022-09-10T04:26:37.243771+0000","last_undegraded":"2022-09-10T04:26:37.243771+0000","last_fullsized":"2022-09-10T04:26:37.243771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:34:31.510362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.453895+0000","last_change":"2022-09-10T04:26:31.548844+0000","last_active":"2022-09-10T04:26:55.453895+0000","last_peered":"2022-09-10T04:26:55.453895+0000","last_clean":"2022-09-10T04:26:55.453895+0000","last_became_active":"2022-09-10T04:26:31.548535+0000","last_became_peered":"2022-09-10T04:26:31.548535+0000","last_unstale":"2022-09-10T04:26:55.453895+0000","last_undegraded":"2022-09-10T04:26:55.453895+0000","last_fullsized":"2022-09-10T04:26:55.453895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:40:49.443190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586008+0000","last_change":"2022-09-10T04:26:32.136407+0000","last_active":"2022-09-10T04:26:36.586008+0000","last_peered":"2022-09-10T04:26:36.586008+0000","last_clean":"2022-09-10T04:26:36.586008+0000","last_became_active":"2022-09-10T04:26:32.136116+0000","last_became_peered":"2022-09-10T04:26:32.136116+0000","last_unstale":"2022-09-10T04:26:36.586008+0000","last_undegraded":"2022-09-10T04:26:36.586008+0000","last_fullsized":"2022-09-10T04:26:36.586008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:32:26.673896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119584+0000","last_change":"2022-09-10T04:26:36.790658+0000","last_active":"2022-09-10T04:26:37.119584+0000","last_peered":"2022-09-10T04:26:37.119584+0000","last_clean":"2022-09-10T04:26:37.119584+0000","last_became_active":"2022-09-10T04:26:36.790203+0000","last_became_peered":"2022-09-10T04:26:36.790203+0000","last_unstale":"2022-09-10T04:26:37.119584+0000","last_undegraded":"2022-09-10T04:26:37.119584+0000","last_fullsized":"2022-09-10T04:26:37.119584+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:53:08.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583789+0000","last_change":"2022-09-10T04:26:31.549875+0000","last_active":"2022-09-10T04:26:36.583789+0000","last_peered":"2022-09-10T04:26:36.583789+0000","last_clean":"2022-09-10T04:26:36.583789+0000","last_became_active":"2022-09-10T04:26:31.549578+0000","last_became_peered":"2022-09-10T04:26:31.549578+0000","last_unstale":"2022-09-10T04:26:36.583789+0000","last_undegraded":"2022-09-10T04:26:36.583789+0000","last_fullsized":"2022-09-10T04:26:36.583789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:52:51.521360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.580312+0000","last_change":"2022-09-10T04:26:32.111710+0000","last_active":"2022-09-10T04:26:36.580312+0000","last_peered":"2022-09-10T04:26:36.580312+0000","last_clean":"2022-09-10T04:26:36.580312+0000","last_became_active":"2022-09-10T04:26:32.111314+0000","last_became_peered":"2022-09-10T04:26:32.111314+0000","last_unstale":"2022-09-10T04:26:36.580312+0000","last_undegraded":"2022-09-10T04:26:36.580312+0000","last_fullsized":"2022-09-10T04:26:36.580312+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:42.589093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.420237+0000","last_change":"2022-09-10T04:26:36.596926+0000","last_active":"2022-09-10T04:26:38.420237+0000","last_peered":"2022-09-10T04:26:38.420237+0000","last_clean":"2022-09-10T04:26:38.420237+0000","last_became_active":"2022-09-10T04:26:36.596287+0000","last_became_peered":"2022-09-10T04:26:36.596287+0000","last_unstale":"2022-09-10T04:26:38.420237+0000","last_undegraded":"2022-09-10T04:26:38.420237+0000","last_fullsized":"2022-09-10T04:26:38.420237+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:52:54.252320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.455040+0000","last_change":"2022-09-10T04:26:31.552277+0000","last_active":"2022-09-10T04:26:55.455040+0000","last_peered":"2022-09-10T04:26:55.455040+0000","last_clean":"2022-09-10T04:26:55.455040+0000","last_became_active":"2022-09-10T04:26:31.551904+0000","last_became_peered":"2022-09-10T04:26:31.551904+0000","last_unstale":"2022-09-10T04:26:55.455040+0000","last_undegraded":"2022-09-10T04:26:55.455040+0000","last_fullsized":"2022-09-10T04:26:55.455040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:28:52.507833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583873+0000","last_change":"2022-09-10T04:26:32.116288+0000","last_active":"2022-09-10T04:26:36.583873+0000","last_peered":"2022-09-10T04:26:36.583873+0000","last_clean":"2022-09-10T04:26:36.583873+0000","last_became_active":"2022-09-10T04:26:32.115955+0000","last_became_peered":"2022-09-10T04:26:32.115955+0000","last_unstale":"2022-09-10T04:26:36.583873+0000","last_undegraded":"2022-09-10T04:26:36.583873+0000","last_fullsized":"2022-09-10T04:26:36.583873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:02:30.470550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119460+0000","last_change":"2022-09-10T04:26:36.594357+0000","last_active":"2022-09-10T04:26:37.119460+0000","last_peered":"2022-09-10T04:26:37.119460+0000","last_clean":"2022-09-10T04:26:37.119460+0000","last_became_active":"2022-09-10T04:26:36.593865+0000","last_became_peered":"2022-09-10T04:26:36.593865+0000","last_unstale":"2022-09-10T04:26:37.119460+0000","last_undegraded":"2022-09-10T04:26:37.119460+0000","last_fullsized":"2022-09-10T04:26:37.119460+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:11:14.675252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":16,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.243846+0000","last_change":"2022-09-10T04:26:31.551838+0000","last_active":"2022-09-10T04:26:37.243846+0000","last_peered":"2022-09-10T04:26:37.243846+0000","last_clean":"2022-09-10T04:26:37.243846+0000","last_became_active":"2022-09-10T04:26:31.551516+0000","last_became_peered":"2022-09-10T04:26:31.551516+0000","last_unstale":"2022-09-10T04:26:37.243846+0000","last_undegraded":"2022-09-10T04:26:37.243846+0000","last_fullsized":"2022-09-10T04:26:37.243846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:18:58.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581086+0000","last_change":"2022-09-10T04:26:32.145692+0000","last_active":"2022-09-10T04:26:36.581086+0000","last_peered":"2022-09-10T04:26:36.581086+0000","last_clean":"2022-09-10T04:26:36.581086+0000","last_became_active":"2022-09-10T04:26:32.144922+0000","last_became_peered":"2022-09-10T04:26:32.144922+0000","last_unstale":"2022-09-10T04:26:36.581086+0000","last_undegraded":"2022-09-10T04:26:36.581086+0000","last_fullsized":"2022-09-10T04:26:36.581086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:12:29.711076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.119772+0000","last_change":"2022-09-10T04:26:36.791321+0000","last_active":"2022-09-10T04:26:37.119772+0000","last_peered":"2022-09-10T04:26:37.119772+0000","last_clean":"2022-09-10T04:26:37.119772+0000","last_became_active":"2022-09-10T04:26:36.789918+0000","last_became_peered":"2022-09-10T04:26:36.789918+0000","last_unstale":"2022-09-10T04:26:37.119772+0000","last_undegraded":"2022-09-10T04:26:37.119772+0000","last_fullsized":"2022-09-10T04:26:37.119772+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:49:56.538141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:55.454909+0000","last_change":"2022-09-10T04:26:31.552140+0000","last_active":"2022-09-10T04:26:55.454909+0000","last_peered":"2022-09-10T04:26:55.454909+0000","last_clean":"2022-09-10T04:26:55.454909+0000","last_became_active":"2022-09-10T04:26:31.551191+0000","last_became_peered":"2022-09-10T04:26:31.551191+0000","last_unstale":"2022-09-10T04:26:55.454909+0000","last_undegraded":"2022-09-10T04:26:55.454909+0000","last_fullsized":"2022-09-10T04:26:55.454909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:52.681536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584586+0000","last_change":"2022-09-10T04:26:32.125775+0000","last_active":"2022-09-10T04:26:36.584586+0000","last_peered":"2022-09-10T04:26:36.584586+0000","last_clean":"2022-09-10T04:26:36.584586+0000","last_became_active":"2022-09-10T04:26:32.125463+0000","last_became_peered":"2022-09-10T04:26:32.125463+0000","last_unstale":"2022-09-10T04:26:36.584586+0000","last_undegraded":"2022-09-10T04:26:36.584586+0000","last_fullsized":"2022-09-10T04:26:36.584586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:33:26.816695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.215638+0000","last_change":"2022-09-10T04:26:36.593873+0000","last_active":"2022-09-10T04:26:38.215638+0000","last_peered":"2022-09-10T04:26:38.215638+0000","last_clean":"2022-09-10T04:26:38.215638+0000","last_became_active":"2022-09-10T04:26:36.593418+0000","last_became_peered":"2022-09-10T04:26:36.593418+0000","last_unstale":"2022-09-10T04:26:38.215638+0000","last_undegraded":"2022-09-10T04:26:38.215638+0000","last_fullsized":"2022-09-10T04:26:38.215638+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:21.834135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:58.184424+0000","last_change":"2022-09-10T04:26:31.555352+0000","last_active":"2022-09-10T04:26:58.184424+0000","last_peered":"2022-09-10T04:26:58.184424+0000","last_clean":"2022-09-10T04:26:58.184424+0000","last_became_active":"2022-09-10T04:26:31.555092+0000","last_became_peered":"2022-09-10T04:26:31.555092+0000","last_unstale":"2022-09-10T04:26:58.184424+0000","last_undegraded":"2022-09-10T04:26:58.184424+0000","last_fullsized":"2022-09-10T04:26:58.184424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:30:17.812089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1106,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584742+0000","last_change":"2022-09-10T04:26:32.115687+0000","last_active":"2022-09-10T04:26:36.584742+0000","last_peered":"2022-09-10T04:26:36.584742+0000","last_clean":"2022-09-10T04:26:36.584742+0000","last_became_active":"2022-09-10T04:26:32.115369+0000","last_became_peered":"2022-09-10T04:26:32.115369+0000","last_unstale":"2022-09-10T04:26:36.584742+0000","last_undegraded":"2022-09-10T04:26:36.584742+0000","last_fullsized":"2022-09-10T04:26:36.584742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:59:31.105825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:38.988722+0000","last_change":"2022-09-10T04:26:36.602214+0000","last_active":"2022-09-10T04:26:38.988722+0000","last_peered":"2022-09-10T04:26:38.988722+0000","last_clean":"2022-09-10T04:26:38.988722+0000","last_became_active":"2022-09-10T04:26:36.601785+0000","last_became_peered":"2022-09-10T04:26:36.601785+0000","last_unstale":"2022-09-10T04:26:38.988722+0000","last_undegraded":"2022-09-10T04:26:38.988722+0000","last_fullsized":"2022-09-10T04:26:38.988722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:52:42.795971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583014+0000","last_change":"2022-09-10T04:26:31.556182+0000","last_active":"2022-09-10T04:26:36.583014+0000","last_peered":"2022-09-10T04:26:36.583014+0000","last_clean":"2022-09-10T04:26:36.583014+0000","last_became_active":"2022-09-10T04:26:31.555873+0000","last_became_peered":"2022-09-10T04:26:31.555873+0000","last_unstale":"2022-09-10T04:26:36.583014+0000","last_undegraded":"2022-09-10T04:26:36.583014+0000","last_fullsized":"2022-09-10T04:26:36.583014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:56:49.406476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.585008+0000","last_change":"2022-09-10T04:26:32.134925+0000","last_active":"2022-09-10T04:26:36.585008+0000","last_peered":"2022-09-10T04:26:36.585008+0000","last_clean":"2022-09-10T04:26:36.585008+0000","last_became_active":"2022-09-10T04:26:32.134581+0000","last_became_peered":"2022-09-10T04:26:32.134581+0000","last_unstale":"2022-09-10T04:26:36.585008+0000","last_undegraded":"2022-09-10T04:26:36.585008+0000","last_fullsized":"2022-09-10T04:26:36.585008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:06:09.815252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":90,"state":"active+clean","last_fresh":"2022-09-10T04:26:37.287892+0000","last_change":"2022-09-10T04:26:36.590902+0000","last_active":"2022-09-10T04:26:37.287892+0000","last_peered":"2022-09-10T04:26:37.287892+0000","last_clean":"2022-09-10T04:26:37.287892+0000","last_became_active":"2022-09-10T04:26:36.590468+0000","last_became_peered":"2022-09-10T04:26:36.590468+0000","last_unstale":"2022-09-10T04:26:37.287892+0000","last_undegraded":"2022-09-10T04:26:37.287892+0000","last_fullsized":"2022-09-10T04:26:37.287892+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:49:40.841273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586989+0000","last_change":"2022-09-10T04:26:31.554090+0000","last_active":"2022-09-10T04:26:36.586989+0000","last_peered":"2022-09-10T04:26:36.586989+0000","last_clean":"2022-09-10T04:26:36.586989+0000","last_became_active":"2022-09-10T04:26:31.553675+0000","last_became_peered":"2022-09-10T04:26:31.553675+0000","last_unstale":"2022-09-10T04:26:36.586989+0000","last_undegraded":"2022-09-10T04:26:36.586989+0000","last_fullsized":"2022-09-10T04:26:36.586989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:35:33.619937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584919+0000","last_change":"2022-09-10T04:26:32.118923+0000","last_active":"2022-09-10T04:26:36.584919+0000","last_peered":"2022-09-10T04:26:36.584919+0000","last_clean":"2022-09-10T04:26:36.584919+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:26:36.584919+0000","last_undegraded":"2022-09-10T04:26:36.584919+0000","last_fullsized":"2022-09-10T04:26:36.584919+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:03.377179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583988+0000","last_change":"2022-09-10T04:26:32.115339+0000","last_active":"2022-09-10T04:26:36.583988+0000","last_peered":"2022-09-10T04:26:36.583988+0000","last_clean":"2022-09-10T04:26:36.583988+0000","last_became_active":"2022-09-10T04:26:32.115044+0000","last_became_peered":"2022-09-10T04:26:32.115044+0000","last_unstale":"2022-09-10T04:26:36.583988+0000","last_undegraded":"2022-09-10T04:26:36.583988+0000","last_fullsized":"2022-09-10T04:26:36.583988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:06:39.719939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577922+0000","last_change":"2022-09-10T04:26:32.121701+0000","last_active":"2022-09-10T04:26:36.577922+0000","last_peered":"2022-09-10T04:26:36.577922+0000","last_clean":"2022-09-10T04:26:36.577922+0000","last_became_active":"2022-09-10T04:26:32.121417+0000","last_became_peered":"2022-09-10T04:26:32.121417+0000","last_unstale":"2022-09-10T04:26:36.577922+0000","last_undegraded":"2022-09-10T04:26:36.577922+0000","last_fullsized":"2022-09-10T04:26:36.577922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:16:44.194677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.576913+0000","last_change":"2022-09-10T04:26:32.139762+0000","last_active":"2022-09-10T04:26:36.576913+0000","last_peered":"2022-09-10T04:26:36.576913+0000","last_clean":"2022-09-10T04:26:36.576913+0000","last_became_active":"2022-09-10T04:26:32.139468+0000","last_became_peered":"2022-09-10T04:26:32.139468+0000","last_unstale":"2022-09-10T04:26:36.576913+0000","last_undegraded":"2022-09-10T04:26:36.576913+0000","last_fullsized":"2022-09-10T04:26:36.576913+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:37:25.448977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.588356+0000","last_change":"2022-09-10T04:26:32.131899+0000","last_active":"2022-09-10T04:26:36.588356+0000","last_peered":"2022-09-10T04:26:36.588356+0000","last_clean":"2022-09-10T04:26:36.588356+0000","last_became_active":"2022-09-10T04:26:32.130047+0000","last_became_peered":"2022-09-10T04:26:32.130047+0000","last_unstale":"2022-09-10T04:26:36.588356+0000","last_undegraded":"2022-09-10T04:26:36.588356+0000","last_fullsized":"2022-09-10T04:26:36.588356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:49:48.001008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.589423+0000","last_change":"2022-09-10T04:26:32.126445+0000","last_active":"2022-09-10T04:26:36.589423+0000","last_peered":"2022-09-10T04:26:36.589423+0000","last_clean":"2022-09-10T04:26:36.589423+0000","last_became_active":"2022-09-10T04:26:32.125935+0000","last_became_peered":"2022-09-10T04:26:32.125935+0000","last_unstale":"2022-09-10T04:26:36.589423+0000","last_undegraded":"2022-09-10T04:26:36.589423+0000","last_fullsized":"2022-09-10T04:26:36.589423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:27:44.987973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.582203+0000","last_change":"2022-09-10T04:26:32.118684+0000","last_active":"2022-09-10T04:26:36.582203+0000","last_peered":"2022-09-10T04:26:36.582203+0000","last_clean":"2022-09-10T04:26:36.582203+0000","last_became_active":"2022-09-10T04:26:32.118399+0000","last_became_peered":"2022-09-10T04:26:32.118399+0000","last_unstale":"2022-09-10T04:26:36.582203+0000","last_undegraded":"2022-09-10T04:26:36.582203+0000","last_fullsized":"2022-09-10T04:26:36.582203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:55:18.907421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.577149+0000","last_change":"2022-09-10T04:26:32.122790+0000","last_active":"2022-09-10T04:26:36.577149+0000","last_peered":"2022-09-10T04:26:36.577149+0000","last_clean":"2022-09-10T04:26:36.577149+0000","last_became_active":"2022-09-10T04:26:32.122379+0000","last_became_peered":"2022-09-10T04:26:32.122379+0000","last_unstale":"2022-09-10T04:26:36.577149+0000","last_undegraded":"2022-09-10T04:26:36.577149+0000","last_fullsized":"2022-09-10T04:26:36.577149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:36:19.747563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581383+0000","last_change":"2022-09-10T04:26:32.145947+0000","last_active":"2022-09-10T04:26:36.581383+0000","last_peered":"2022-09-10T04:26:36.581383+0000","last_clean":"2022-09-10T04:26:36.581383+0000","last_became_active":"2022-09-10T04:26:32.145570+0000","last_became_peered":"2022-09-10T04:26:32.145570+0000","last_unstale":"2022-09-10T04:26:36.581383+0000","last_undegraded":"2022-09-10T04:26:36.581383+0000","last_fullsized":"2022-09-10T04:26:36.581383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:38:56.028983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.583785+0000","last_change":"2022-09-10T04:26:32.140019+0000","last_active":"2022-09-10T04:26:36.583785+0000","last_peered":"2022-09-10T04:26:36.583785+0000","last_clean":"2022-09-10T04:26:36.583785+0000","last_became_active":"2022-09-10T04:26:32.139709+0000","last_became_peered":"2022-09-10T04:26:32.139709+0000","last_unstale":"2022-09-10T04:26:36.583785+0000","last_undegraded":"2022-09-10T04:26:36.583785+0000","last_fullsized":"2022-09-10T04:26:36.583785+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:56:11.850506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581578+0000","last_change":"2022-09-10T04:26:32.145867+0000","last_active":"2022-09-10T04:26:36.581578+0000","last_peered":"2022-09-10T04:26:36.581578+0000","last_clean":"2022-09-10T04:26:36.581578+0000","last_became_active":"2022-09-10T04:26:32.145582+0000","last_became_peered":"2022-09-10T04:26:32.145582+0000","last_unstale":"2022-09-10T04:26:36.581578+0000","last_undegraded":"2022-09-10T04:26:36.581578+0000","last_fullsized":"2022-09-10T04:26:36.581578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:30:11.784617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.584565+0000","last_change":"2022-09-10T04:26:32.134641+0000","last_active":"2022-09-10T04:26:36.584565+0000","last_peered":"2022-09-10T04:26:36.584565+0000","last_clean":"2022-09-10T04:26:36.584565+0000","last_became_active":"2022-09-10T04:26:32.134320+0000","last_became_peered":"2022-09-10T04:26:32.134320+0000","last_unstale":"2022-09-10T04:26:36.584565+0000","last_undegraded":"2022-09-10T04:26:36.584565+0000","last_fullsized":"2022-09-10T04:26:36.584565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:58:29.800916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.586176+0000","last_change":"2022-09-10T04:26:32.126597+0000","last_active":"2022-09-10T04:26:36.586176+0000","last_peered":"2022-09-10T04:26:36.586176+0000","last_clean":"2022-09-10T04:26:36.586176+0000","last_became_active":"2022-09-10T04:26:32.125895+0000","last_became_peered":"2022-09-10T04:26:32.125895+0000","last_unstale":"2022-09-10T04:26:36.586176+0000","last_undegraded":"2022-09-10T04:26:36.586176+0000","last_fullsized":"2022-09-10T04:26:36.586176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:17:41.456304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2022-09-10T04:26:36.581777+0000","last_change":"2022-09-10T04:26:32.122525+0000","last_active":"2022-09-10T04:26:36.581777+0000","last_peered":"2022-09-10T04:26:36.581777+0000","last_clean":"2022-09-10T04:26:36.581777+0000","last_became_active":"2022-09-10T04:26:32.122113+0000","last_became_peered":"2022-09-10T04:26:32.122113+0000","last_unstale":"2022-09-10T04:26:36.581777+0000","last_undegraded":"2022-09-10T04:26:36.581777+0000","last_fullsized":"2022-09-10T04:26:36.581777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:29:19.351147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6537,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":116,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":18977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350993,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6104,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739048,"statfs":{"total":95995035648,"available":95988785152,"internally_reserved":0,"allocated":999424,"data_stored":479654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.623},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.49399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.71799999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.78500000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":9,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.80700000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.79800000000000004}]}]},{"osd":10,"up_from":73,"seq":313532612629,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6820,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738332,"statfs":{"total":95995035648,"available":95988051968,"internally_reserved":0,"allocated":1601536,"data_stored":1072514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.59499999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.438}]},{"osd":5,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":9.7629999999999999,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":9.7349999999999994,"5min":9.7349999999999994,"15min":9.7349999999999994},"last":0.64400000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.873}]},{"osd":8,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.65900000000000003}]},{"osd":9,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.883,"5min":1.883,"15min":1.883},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":0.70099999999999996}]},{"osd":11,"last update":"Sat Sep 10 04:26:44 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.67400000000000004}]}]},{"osd":9,"up_from":68,"seq":292057776154,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6308,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738844,"statfs":{"total":95995035648,"available":95988576256,"internally_reserved":0,"allocated":1011712,"data_stored":479418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83799999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.39700000000000002}]},{"osd":1,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.69399999999999995,"15min":0.70599999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84499999999999997,"5min":1.024,"15min":1.024},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.79600000000000004,"15min":0.81799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.84099999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.625},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.86299999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.79200000000000004}]},{"osd":3,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89800000000000002,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.80900000000000005}]},{"osd":4,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.80300000000000005,"15min":0.83199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.82599999999999996,"5min":1.272,"15min":1.272},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.81100000000000005,"15min":0.83299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.85199999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.86399999999999999,"15min":0.89500000000000002},"min":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.879,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.88400000000000001,"15min":0.91400000000000003},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.85599999999999998,"5min":1.262,"15min":1.262},"last":0.746}]},{"osd":6,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.79700000000000004}]},{"osd":7,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.93700000000000006,"15min":0.97299999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.89100000000000001,"5min":1.429,"15min":1.429},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.97999999999999998,"15min":1.018},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95199999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.66700000000000004}]},{"osd":8,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.0189999999999999,"15min":1.0609999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.96899999999999997,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.96699999999999997,"15min":1.0089999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91800000000000004,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.70699999999999996}]},{"osd":10,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68700000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]}]},{"osd":8,"up_from":60,"seq":257698037792,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6360,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738792,"statfs":{"total":95995035648,"available":95988523008,"internally_reserved":0,"allocated":999424,"data_stored":479136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71399999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71399999999999997,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57599999999999996,"15min":0.58299999999999996},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.76700000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54000000000000004,"15min":0.54600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.433,"15min":0.436},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.441,"15min":0.443},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.61699999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.59399999999999997,"15min":0.59899999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":4,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56899999999999995,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.622,"15min":0.628},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.83999999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.79300000000000004}]},{"osd":5,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.5},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.56599999999999995,"15min":0.57899999999999996},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80500000000000005,"5min":1.046,"15min":1.046},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.69799999999999995,"15min":0.70799999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.69099999999999995,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.72699999999999998}]},{"osd":7,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65200000000000002,"15min":0.65800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.747},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.67100000000000004,"15min":0.68300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.92200000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.73699999999999999}]},{"osd":9,"last update":"Sat Sep 10 04:25:55 2022","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.80300000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.82299999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]}]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7068,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738084,"statfs":{"total":95995035648,"available":95987798016,"internally_reserved":0,"allocated":1593344,"data_stored":1070251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.77100000000000002,"15min":0.79900000000000004},"min":{"1min":0.41299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":5.3719999999999999,"15min":5.3719999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.75,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39900000000000002,"15min":0.39800000000000002},"min":{"1min":0.30099999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.61099999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.441,"15min":0.442},"min":{"1min":0.32800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.628,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77400000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.77600000000000002,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.65000000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.45800000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.61399999999999999,"15min":0.61199999999999999},"min":{"1min":0.48899999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.628}]},{"osd":4,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.67800000000000005,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65800000000000003,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.373}]},{"osd":5,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76200000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84199999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.50800000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.83299999999999996,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69399999999999995,"15min":0.69699999999999995},"min":{"1min":0.51600000000000001,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.84699999999999998,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.47499999999999998}]},{"osd":8,"last update":"Sat Sep 10 04:26:24 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.68500000000000005,"15min":0.69499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89900000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.67000000000000004,"15min":0.67800000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81200000000000006,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63900000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:25:56 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68700000000000006}]},{"osd":10,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6348,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738804,"statfs":{"total":95995035648,"available":95988535296,"internally_reserved":0,"allocated":987136,"data_stored":478526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.5,"15min":0.48499999999999999},"min":{"1min":0.379,"5min":0.318,"15min":0.318},"max":{"1min":0.76400000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54000000000000004,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.93600000000000005,"5min":1.046,"15min":1.046},"last":0.57299999999999995}]},{"osd":1,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64900000000000002,"15min":0.63400000000000001},"min":{"1min":0.441,"5min":0.436,"15min":0.436},"max":{"1min":0.85999999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.68600000000000005,"15min":0.66900000000000004},"min":{"1min":0.504,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.995,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66100000000000003}]},{"osd":2,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.622,"15min":0.61299999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84599999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0209999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.83899999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69099999999999995,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.998,"5min":1.103,"15min":1.103},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.58599999999999997},"min":{"1min":0.49099999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.79100000000000004}]},{"osd":4,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68799999999999994,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.496,"15min":0.496},"max":{"1min":0.92400000000000004,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.76600000000000001,"15min":0.76100000000000001},"min":{"1min":0.57699999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.96299999999999997,"5min":1.167,"15min":1.167},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.70099999999999996,"15min":0.69099999999999995},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92900000000000005,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70699999999999996,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94799999999999995,"5min":1.036,"15min":1.036},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:26:03 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.78300000000000003,"15min":0.76100000000000001},"min":{"1min":0.66200000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.82599999999999996,"15min":0.80500000000000005},"min":{"1min":0.71999999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.80000000000000004}]},{"osd":8,"last update":"Sat Sep 10 04:26:30 2022","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.85499999999999998,"15min":0.85099999999999998},"min":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1459999999999999,"5min":1.181,"15min":1.181},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.72399999999999998,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0840000000000001,"5min":1.097,"15min":1.097},"last":0.88100000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.90100000000000002}]},{"osd":10,"last update":"Sat Sep 10 04:26:22 2022","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.81399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7252,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737900,"statfs":{"total":95995035648,"available":95987609600,"internally_reserved":0,"allocated":995328,"data_stored":482010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:29 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.53500000000000003,"15min":0.51700000000000002},"min":{"1min":0.41399999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.67500000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.503},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53400000000000003,"15min":0.52400000000000002},"min":{"1min":0.34899999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.32300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:26:17 2022","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.40999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84299999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.60699999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.68700000000000006}]},{"osd":4,"last update":"Sat Sep 10 04:26:41 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.42599999999999999,"15min":0.40500000000000003},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70199999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.32400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Sep 10 04:26:09 2022","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.52800000000000002,"15min":0.502},"min":{"1min":0.35899999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.53000000000000003,"15min":0.502},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.583,"5min":1.583,"15min":1.583},"last":0.71599999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:26:34 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.60599999999999998,"15min":0.59999999999999998},"min":{"1min":0.46700000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.749},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.90200000000000002,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.81000000000000005}]},{"osd":7,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.35599999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77000000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.73699999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.437,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.79000000000000004}]},{"osd":9,"last update":"Sat Sep 10 04:25:54 2022","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.876}]},{"osd":10,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.80300000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7248,"kb_used_data":968,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93737904,"statfs":{"total":95995035648,"available":95987613696,"internally_reserved":0,"allocated":991232,"data_stored":478616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.86699999999999999},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.94799999999999995,"5min":8.3729999999999993,"15min":8.3729999999999993},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.58699999999999997,"15min":0.60099999999999998},"min":{"1min":0.45500000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0209999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.88300000000000001},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96699999999999997,"5min":8.3260000000000005,"15min":8.3260000000000005},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:25:11 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.52500000000000002,"15min":0.503},"min":{"1min":0.37,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.435},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.36499999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.86299999999999999}]},{"osd":4,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63100000000000001,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.628,"15min":0.58599999999999997},"min":{"1min":0.435,"5min":0.372,"15min":0.372},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61899999999999999}]},{"osd":5,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.63800000000000001,"15min":0.60599999999999998},"min":{"1min":0.53000000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.64400000000000002,"15min":0.61499999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.48299999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:26:42 2022","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.65700000000000003},"min":{"1min":0.55600000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72199999999999998,"15min":0.68700000000000006},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78000000000000003}]},{"osd":7,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.71899999999999997,"15min":0.71099999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.72299999999999998,"15min":0.70399999999999996},"min":{"1min":0.68700000000000006,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.1140000000000001,"5min":1.145,"15min":1.145},"last":0.53600000000000003}]},{"osd":8,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80400000000000005,"15min":0.80000000000000004},"min":{"1min":0.59499999999999997,"5min":0.495,"15min":0.495},"max":{"1min":1.099,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.79000000000000004,"15min":0.78400000000000003},"min":{"1min":0.59599999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.70499999999999996}]},{"osd":9,"last update":"Sat Sep 10 04:25:55 2022","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.879}]},{"osd":10,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.65100000000000002}]},{"osd":11,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.84499999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7412,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737740,"statfs":{"total":95995035648,"available":95987445760,"internally_reserved":0,"allocated":1028096,"data_stored":483870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68200000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33500000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72899999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40400000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77200000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52600000000000002,"15min":0.504},"min":{"1min":0.378,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71299999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65300000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:26:16 2022","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.59499999999999997,"15min":0.60099999999999998},"min":{"1min":0.41699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.80500000000000005,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58599999999999997,"15min":0.59999999999999998},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.80500000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.58799999999999997}]},{"osd":4,"last update":"Sat Sep 10 04:25:40 2022","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.54900000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.73599999999999999,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84399999999999997,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66200000000000003}]},{"osd":5,"last update":"Sat Sep 10 04:26:07 2022","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57699999999999996,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.754,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41599999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:26:37 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82199999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84199999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63}]},{"osd":7,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60699999999999998,"15min":0.61199999999999999},"min":{"1min":0.40400000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.86199999999999999,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69399999999999995}]},{"osd":8,"last update":"Sat Sep 10 04:26:28 2022","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59699999999999998,"15min":0.60599999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78300000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61699999999999999}]},{"osd":9,"last update":"Sat Sep 10 04:25:56 2022","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.73299999999999998}]},{"osd":10,"last update":"Sat Sep 10 04:26:22 2022","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.76200000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6612,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738540,"statfs":{"total":95995035648,"available":95988264960,"internally_reserved":0,"allocated":995328,"data_stored":478630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:15 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45000000000000001,"15min":0.41699999999999998},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68899999999999995,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.432},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.46700000000000003,"15min":0.44400000000000001},"min":{"1min":0.38700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.81899999999999995,"5min":1.242,"15min":1.242},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Sep 10 04:26:15 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65300000000000002,"15min":0.61599999999999999},"min":{"1min":0.52100000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.0409999999999999,"5min":1.494,"15min":1.494},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.63,"15min":0.60799999999999998},"min":{"1min":0.47299999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.88200000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Sat Sep 10 04:26:15 2022","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.623,"15min":0.59599999999999997},"min":{"1min":0.437,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88400000000000001,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.745,"5min":0.67700000000000005,"15min":0.64900000000000002},"min":{"1min":0.59099999999999997,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.99299999999999999,"5min":1.454,"15min":1.454},"last":0.79300000000000004}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.626},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"min":{"1min":0.45800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.95499999999999996,"5min":1.528,"15min":1.528},"last":0.71099999999999997}]},{"osd":5,"last update":"Sat Sep 10 04:26:02 2022","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66900000000000004,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0069999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.67900000000000005,"15min":0.67300000000000004},"min":{"1min":0.47999999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94299999999999995,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Sep 10 04:26:35 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.66700000000000004,"15min":0.64500000000000002},"min":{"1min":0.501,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.72699999999999998,"15min":0.70799999999999996},"min":{"1min":0.54400000000000004,"5min":0.433,"15min":0.433},"max":{"1min":0.98799999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:26:02 2022","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71499999999999997,"15min":0.71099999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.76200000000000001,"15min":0.75900000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1719999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.81599999999999995}]},{"osd":8,"last update":"Sat Sep 10 04:26:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1060000000000001,"5min":1.208,"15min":1.208},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79200000000000004,"15min":0.78600000000000003},"min":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.099,"5min":1.139,"15min":1.139},"last":0.73299999999999998}]},{"osd":9,"last update":"Sat Sep 10 04:25:57 2022","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.83599999999999997}]},{"osd":10,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.90300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":999424,"data_stored":479122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:25:39 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73199999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54500000000000004,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.378,"15min":0.34399999999999997},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71499999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.36499999999999999},"min":{"1min":0.29799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80800000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42199999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.55900000000000005},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.86299999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79700000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:26:09 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57599999999999996,"15min":0.54300000000000004},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.92500000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56599999999999995,"15min":0.55200000000000005},"min":{"1min":0.39800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.66000000000000003}]},{"osd":6,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66700000000000004,"15min":0.67400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.504},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.77000000000000002,"15min":0.73699999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.61699999999999999,"15min":0.61199999999999999},"min":{"1min":0.35099999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.55200000000000005,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55400000000000005}]},{"osd":8,"last update":"Sat Sep 10 04:26:25 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.48999999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0089999999999999,"5min":1.133,"15min":1.133},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66800000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.875,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.59499999999999997}]},{"osd":9,"last update":"Sat Sep 10 04:25:54 2022","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67500000000000004}]},{"osd":10,"last update":"Sat Sep 10 04:26:19 2022","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.57199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7192,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737960,"statfs":{"total":95995035648,"available":95987671040,"internally_reserved":0,"allocated":1589248,"data_stored":1068753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.36799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77100000000000002,"5min":1.879,"15min":1.879},"last":0.42899999999999999}]},{"osd":1,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.503,"15min":0.48099999999999998},"min":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.501,"15min":0.47999999999999998},"min":{"1min":0.38500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.63200000000000001}]},{"osd":2,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.375,"15min":0.37},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.55500000000000005,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38700000000000001,"15min":0.38},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.53700000000000003}]},{"osd":3,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.74099999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.57799999999999996,"15min":0.56799999999999995},"min":{"1min":0.46999999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Sep 10 04:26:01 2022","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51700000000000002,"15min":0.503},"min":{"1min":0.41199999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.75600000000000001}]},{"osd":6,"last update":"Sat Sep 10 04:26:36 2022","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.45200000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.218},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Sat Sep 10 04:25:59 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.42899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.80500000000000005}]},{"osd":8,"last update":"Sat Sep 10 04:26:23 2022","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.56799999999999995}]},{"osd":9,"last update":"Sat Sep 10 04:25:56 2022","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.77100000000000002}]},{"osd":10,"last update":"Sat Sep 10 04:26:19 2022","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.745},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-10T04:26:59.483 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2022-09-10T04:26:59.484 INFO:tasks.cephadm:Setting up client nodes... 2022-09-10T04:26:59.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2022-09-10T04:26:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:59 smithi104 ceph-mon[97044]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:26:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:59 smithi104 ceph-mon[97044]: mds.? [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] up:active 2022-09-10T04:26:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:59 smithi104 ceph-mon[97044]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2022-09-10T04:26:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:26:59 smithi104 ceph-mon[97044]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-09-10T04:26:59.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:59 smithi081 ceph-mon[90133]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:26:59.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:59 smithi081 ceph-mon[90133]: mds.? [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] up:active 2022-09-10T04:26:59.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:59 smithi081 ceph-mon[90133]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2022-09-10T04:26:59.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:26:59 smithi081 ceph-mon[90133]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-09-10T04:26:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:59 smithi110 ceph-mon[97633]: from='client.24824 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:26:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:59 smithi110 ceph-mon[97633]: mds.? [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] up:active 2022-09-10T04:26:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:59 smithi110 ceph-mon[97633]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2022-09-10T04:26:59.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:26:59 smithi110 ceph-mon[97633]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-09-10T04:27:00.388 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:00.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:00 smithi081 ceph-mon[90133]: pgmap v315: 129 pgs: 129 active+clean; 583 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.7 KiB/s wr, 6 op/s 2022-09-10T04:27:00.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:00 smithi081 ceph-mon[90133]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:00 smithi104 ceph-mon[97044]: pgmap v315: 129 pgs: 129 active+clean; 583 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.7 KiB/s wr, 6 op/s 2022-09-10T04:27:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:00 smithi104 ceph-mon[97044]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:00 smithi110 ceph-mon[97633]: pgmap v315: 129 pgs: 129 active+clean; 583 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.7 KiB/s wr, 6 op/s 2022-09-10T04:27:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:00 smithi110 ceph-mon[97633]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:01.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 -- 172.21.15.81:0/1281138465 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 msgr2=0x7f5580106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 --2- 172.21.15.81:0/1281138465 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 0x7f5580106360 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5578009b30 tx=0x7f5578009e40 comp rx=0 tx=0).stop 2022-09-10T04:27:01.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 -- 172.21.15.81:0/1281138465 shutdown_connections 2022-09-10T04:27:01.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 --2- 172.21.15.81:0/1281138465 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 0x7f5580106360 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:01.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 --2- 172.21.15.81:0/1281138465 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55800fde10 0x7f55800fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:01.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 --2- 172.21.15.81:0/1281138465 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f55800ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:01.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 -- 172.21.15.81:0/1281138465 >> 172.21.15.81:0/1281138465 conn(0x7f55800f9570 msgr2=0x7f55800fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:01.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 -- 172.21.15.81:0/1281138465 shutdown_connections 2022-09-10T04:27:01.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.905+0000 7f55865df700 1 -- 172.21.15.81:0/1281138465 wait complete. 2022-09-10T04:27:01.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.906+0000 7f55865df700 1 Processor -- start 2022-09-10T04:27:01.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.906+0000 7f55865df700 1 -- start start 2022-09-10T04:27:01.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.906+0000 7f55865df700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55800fde10 0x7f5580195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:01.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.906+0000 7f55865df700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f5580196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:01.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.906+0000 7f55865df700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 0x7f558019d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:01.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f55865df700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558019d7f0 con 0x7f55800ffa30 2022-09-10T04:27:01.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f55865df700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558019d960 con 0x7f55800ff080 2022-09-10T04:27:01.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f55865df700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558019db20 con 0x7f55800fde10 2022-09-10T04:27:01.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f5580196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:01.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f5580196170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:35992/0 (socket says 172.21.15.81:35992) 2022-09-10T04:27:01.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 -- 172.21.15.81:0/3747179103 learned_addr learned my addr 172.21.15.81:0/3747179103 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:01.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557ffff700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55800fde10 0x7f5580195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:01.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f5584b7c700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 0x7f558019d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:01.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 -- 172.21.15.81:0/3747179103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55800fde10 msgr2=0x7f5580195c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:01.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55800fde10 0x7f5580195c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:01.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 -- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 msgr2=0x7f558019d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:01.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 0x7f558019d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:01.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.907+0000 7f557f7fe700 1 -- 172.21.15.81:0/3747179103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5578009730 con 0x7f55800ff080 2022-09-10T04:27:01.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.908+0000 7f557f7fe700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f5580196170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f557400e710 tx=0x7f557400ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:01.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.908+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5574004220 con 0x7f55800ff080 2022-09-10T04:27:01.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.908+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558019de00 con 0x7f55800ff080 2022-09-10T04:27:01.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.908+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f5574004380 con 0x7f55800ff080 2022-09-10T04:27:01.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.908+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5574003b70 con 0x7f55800ff080 2022-09-10T04:27:01.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.908+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558019e280 con 0x7f55800ff080 2022-09-10T04:27:01.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.909+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5580109630 con 0x7f55800ff080 2022-09-10T04:27:01.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.911+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f557401e020 con 0x7f55800ff080 2022-09-10T04:27:01.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.912+0000 7f557d7fa700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5568099010 0x7f556809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:01.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.913+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f5574014070 con 0x7f55800ff080 2022-09-10T04:27:01.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.913+0000 7f557ffff700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5568099010 0x7f556809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:01.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.913+0000 7f557ffff700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5568099010 0x7f556809b4c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5580197100 tx=0x7f5570005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:01.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:01.914+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f557408eeb0 con 0x7f55800ff080 2022-09-10T04:27:02.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.208+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f5580190240 con 0x7f55800ff080 2022-09-10T04:27:02.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.215+0000 7f557d7fa700 1 -- 172.21.15.81:0/3747179103 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f55740924d0 con 0x7f55800ff080 2022-09-10T04:27:02.217 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2022-09-10T04:27:02.217 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAWEhxjCQ6SDBAA1iwB4x4ExMWhBbzTZV6gog== 2022-09-10T04:27:02.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5568099010 msgr2=0x7f556809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:02.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5568099010 0x7f556809b4c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5580197100 tx=0x7f5570005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:02.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 msgr2=0x7f5580196170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:02.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f5580196170 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f557400e710 tx=0x7f557400ea20 comp rx=0 tx=0).stop 2022-09-10T04:27:02.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 shutdown_connections 2022-09-10T04:27:02.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55800ffa30 0x7f558019d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:02.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f5568099010 0x7f556809b4c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:02.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f55800fde10 0x7f5580195c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:02.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 --2- 172.21.15.81:0/3747179103 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55800ff080 0x7f5580196170 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:02.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.217+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 >> 172.21.15.81:0/3747179103 conn(0x7f55800f9570 msgr2=0x7f55801049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:02.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.218+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 shutdown_connections 2022-09-10T04:27:02.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:02.218+0000 7f55865df700 1 -- 172.21.15.81:0/3747179103 wait complete. 2022-09-10T04:27:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:02 smithi081 ceph-mon[90133]: pgmap v316: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.6 KiB/s wr, 14 op/s 2022-09-10T04:27:02.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:02 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3747179103' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-10T04:27:02.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:02 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-10T04:27:02.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:02 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-09-10T04:27:02.656 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:27:02.657 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2022-09-10T04:27:02.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-09-10T04:27:02.786 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-09-10T04:27:02.787 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2022-09-10T04:27:02.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph mgr dump --format=json 2022-09-10T04:27:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:02 smithi104 ceph-mon[97044]: pgmap v316: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.6 KiB/s wr, 14 op/s 2022-09-10T04:27:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:02 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3747179103' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-10T04:27:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:02 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-10T04:27:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:02 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-09-10T04:27:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:02 smithi110 ceph-mon[97633]: pgmap v316: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.6 KiB/s wr, 14 op/s 2022-09-10T04:27:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:02 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3747179103' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-10T04:27:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:02 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2022-09-10T04:27:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:02 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2022-09-10T04:27:03.305 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-a[125370]: 2022-09-10T04:27:02.913+0000 7f1cb22fb700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2022-09-10T04:27:03.306 INFO:journalctl@ceph.mds.a.smithi081.stdout:Sep 10 04:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-a[125370]: 2022-09-10T04:27:02.913+0000 7f1cb22fb700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2022-09-10T04:27:03.698 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:04.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:04 smithi081 ceph-mon[90133]: pgmap v317: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.2 KiB/s wr, 17 op/s 2022-09-10T04:27:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:04 smithi104 ceph-mon[97044]: pgmap v317: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.2 KiB/s wr, 17 op/s 2022-09-10T04:27:04.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:04 smithi110 ceph-mon[97633]: pgmap v317: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.2 KiB/s wr, 17 op/s 2022-09-10T04:27:05.206 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 -- 172.21.15.81:0/3958158574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe340100c70 msgr2=0x7fe340104cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:05.207 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 --2- 172.21.15.81:0/3958158574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe340100c70 0x7fe340104cc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe330009810 tx=0x7fe330009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:05.207 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 -- 172.21.15.81:0/3958158574 shutdown_connections 2022-09-10T04:27:05.207 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 --2- 172.21.15.81:0/3958158574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401053d0 0x7fe34010cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.207 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 --2- 172.21.15.81:0/3958158574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe3401002d0 0x7fe3401006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 --2- 172.21.15.81:0/3958158574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe340100c70 0x7fe340104cc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 -- 172.21.15.81:0/3958158574 >> 172.21.15.81:0/3958158574 conn(0x7fe3400fbbf0 msgr2=0x7fe3400fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:05.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.204+0000 7fe347308700 1 -- 172.21.15.81:0/3958158574 shutdown_connections 2022-09-10T04:27:05.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.205+0000 7fe347308700 1 -- 172.21.15.81:0/3958158574 wait complete. 2022-09-10T04:27:05.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 Processor -- start 2022-09-10T04:27:05.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 -- start start 2022-09-10T04:27:05.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 0x7fe340195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:05.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe340100c70 0x7fe340196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3401053d0 0x7fe34019a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34019af10 con 0x7fe3401002d0 2022-09-10T04:27:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34019b080 con 0x7fe340100c70 2022-09-10T04:27:05.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.206+0000 7fe347308700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34019b240 con 0x7fe3401053d0 2022-09-10T04:27:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3450a4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 0x7fe340195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3450a4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 0x7fe340195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55500/0 (socket says 172.21.15.81:55500) 2022-09-10T04:27:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3450a4700 1 -- 172.21.15.81:0/1157578303 learned_addr learned my addr 172.21.15.81:0/1157578303 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:05.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe340100c70 0x7fe340196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:05.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3458a5700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3401053d0 0x7fe34019a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:05.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 -- 172.21.15.81:0/1157578303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3401053d0 msgr2=0x7fe34019a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:05.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3401053d0 0x7fe34019a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 -- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 msgr2=0x7fe340195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:05.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 0x7fe340195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 -- 172.21.15.81:0/1157578303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe330009470 con 0x7fe340100c70 2022-09-10T04:27:05.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3450a4700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 0x7fe340195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:05.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.207+0000 7fe3448a3700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe340100c70 0x7fe340196420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe3300059c0 tx=0x7fe33000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:05.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.208+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33001d070 con 0x7fe340100c70 2022-09-10T04:27:05.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.208+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34019b4c0 con 0x7fe340100c70 2022-09-10T04:27:05.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.208+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe33000f880 con 0x7fe340100c70 2022-09-10T04:27:05.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.208+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330004760 con 0x7fe340100c70 2022-09-10T04:27:05.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.208+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3401a2d10 con 0x7fe340100c70 2022-09-10T04:27:05.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.209+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe340064ea0 con 0x7fe340100c70 2022-09-10T04:27:05.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.211+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe330028030 con 0x7fe340100c70 2022-09-10T04:27:05.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.212+0000 7fe3367fc700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe32c099010 0x7fe32c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:05.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.212+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7fe3300c34c0 con 0x7fe340100c70 2022-09-10T04:27:05.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.212+0000 7fe3450a4700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe32c099010 0x7fe32c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:05.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.213+0000 7fe3450a4700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe32c099010 0x7fe32c09b4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0057d0 tx=0x7fe33c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:05.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.214+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fe33008f960 con 0x7fe340100c70 2022-09-10T04:27:05.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.484+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe3401972d0 con 0x7fe340100c70 2022-09-10T04:27:05.490 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.489+0000 7fe3367fc700 1 -- 172.21.15.81:0/1157578303 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+245987 (secure 0 0 0) 0x7fe330092f80 con 0x7fe340100c70 2022-09-10T04:27:05.497 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:05.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe32c099010 msgr2=0x7fe32c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:05.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe32c099010 0x7fe32c09b4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0057d0 tx=0x7fe33c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe340100c70 msgr2=0x7fe340196420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:05.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe340100c70 0x7fe340196420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe3300059c0 tx=0x7fe33000f690 comp rx=0 tx=0).stop 2022-09-10T04:27:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 shutdown_connections 2022-09-10T04:27:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe32c099010 0x7fe32c09b4c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3401053d0 0x7fe34019a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3401002d0 0x7fe340195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 --2- 172.21.15.81:0/1157578303 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe340100c70 0x7fe340196420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:05.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 >> 172.21.15.81:0/1157578303 conn(0x7fe3400fbbf0 msgr2=0x7fe3400fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:05.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 shutdown_connections 2022-09-10T04:27:05.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:05.495+0000 7fe347308700 1 -- 172.21.15.81:0/1157578303 wait complete. 2022-09-10T04:27:06.066 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":16,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":532852594},{"type":"v1","addr":"172.21.15.81:6801","nonce":532852594}]},"active_addr":"172.21.15.81:6801/532852594","active_change":"2022-09-10T04:19:09.199013+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.81:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":936949817}]},{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":700200539}]},{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":3312490}]},{"addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1948013399}]}]}} 2022-09-10T04:27:06.073 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2022-09-10T04:27:06.073 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2022-09-10T04:27:06.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd dump --format=json 2022-09-10T04:27:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:06 smithi110 ceph-mon[97633]: pgmap v318: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 4.2 KiB/s wr, 20 op/s 2022-09-10T04:27:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:06 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1157578303' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-09-10T04:27:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:06 smithi104 ceph-mon[97044]: pgmap v318: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 4.2 KiB/s wr, 20 op/s 2022-09-10T04:27:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:06 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1157578303' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-09-10T04:27:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:06 smithi081 ceph-mon[90133]: pgmap v318: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 4.2 KiB/s wr, 20 op/s 2022-09-10T04:27:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:06 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1157578303' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2022-09-10T04:27:06.946 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:08.514 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3251581991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38102d40 msgr2=0x7f3d380666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:08.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3251581991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38102d40 0x7f3d380666c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009810 tx=0x7f3d28009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:08.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3251581991 shutdown_connections 2022-09-10T04:27:08.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3251581991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d381008d0 0x7f3d38100d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3251581991 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38102d40 0x7f3d380666c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3251581991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d38066c00 0x7f3d38067050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3251581991 >> 172.21.15.81:0/3251581991 conn(0x7f3d38073cf0 msgr2=0x7f3d380740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:08.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3251581991 shutdown_connections 2022-09-10T04:27:08.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.512+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3251581991 wait complete. 2022-09-10T04:27:08.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.513+0000 7f3d3d49b700 1 Processor -- start 2022-09-10T04:27:08.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.513+0000 7f3d3d49b700 1 -- start start 2022-09-10T04:27:08.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d3d49b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38066c00 0x7f3d3819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d3d49b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 0x7f3d3819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:08.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d3d49b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d38102d40 0x7f3d3819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:08.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d3d49b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3819f250 con 0x7f3d381008d0 2022-09-10T04:27:08.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d3d49b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3819f3c0 con 0x7f3d38066c00 2022-09-10T04:27:08.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d3d49b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3819f580 con 0x7f3d38102d40 2022-09-10T04:27:08.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d2ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 0x7f3d3819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d36ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38066c00 0x7f3d3819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:08.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d36ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38066c00 0x7f3d3819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:36042/0 (socket says 172.21.15.81:36042) 2022-09-10T04:27:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d36ffd700 1 -- 172.21.15.81:0/3377395480 learned_addr learned my addr 172.21.15.81:0/3377395480 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:08.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d2ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 0x7f3d3819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55518/0 (socket says 172.21.15.81:55518) 2022-09-10T04:27:08.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.514+0000 7f3d377fe700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d38102d40 0x7f3d3819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:08.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d2ffff700 1 -- 172.21.15.81:0/3377395480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d38102d40 msgr2=0x7f3d3819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:08.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d2ffff700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d38102d40 0x7f3d3819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d2ffff700 1 -- 172.21.15.81:0/3377395480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38066c00 msgr2=0x7f3d3819a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:08.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d2ffff700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38066c00 0x7f3d3819a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d2ffff700 1 -- 172.21.15.81:0/3377395480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28009470 con 0x7f3d381008d0 2022-09-10T04:27:08.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d2ffff700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 0x7f3d3819a760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000e6c0 tx=0x7f3d2000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:08.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20010900 con 0x7f3d381008d0 2022-09-10T04:27:08.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3819f860 con 0x7f3d381008d0 2022-09-10T04:27:08.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3d20018420 con 0x7f3d381008d0 2022-09-10T04:27:08.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20017610 con 0x7f3d381008d0 2022-09-10T04:27:08.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.515+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d381a70a0 con 0x7f3d381008d0 2022-09-10T04:27:08.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.517+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d38064ea0 con 0x7f3d381008d0 2022-09-10T04:27:08.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.518+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3d2000c040 con 0x7f3d381008d0 2022-09-10T04:27:08.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.519+0000 7f3d34ff9700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3d24099010 0x7f3d2409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:08.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.520+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8726+0+0 (secure 0 0 0) 0x7f3d20014070 con 0x7f3d381008d0 2022-09-10T04:27:08.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.520+0000 7f3d36ffd700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3d24099010 0x7f3d2409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:08.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.521+0000 7f3d36ffd700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3d24099010 0x7f3d2409b4c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009760 tx=0x7f3d28009e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:08.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.524+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3d2004fc80 con 0x7f3d381008d0 2022-09-10T04:27:08.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:08 smithi081 ceph-mon[90133]: pgmap v319: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.0 KiB/s wr, 22 op/s 2022-09-10T04:27:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:08 smithi110 ceph-mon[97633]: pgmap v319: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.0 KiB/s wr, 22 op/s 2022-09-10T04:27:08.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.737+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d38114230 con 0x7f3d381008d0 2022-09-10T04:27:08.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.738+0000 7f3d34ff9700 1 -- 172.21.15.81:0/3377395480 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+23178 (secure 0 0 0) 0x7f3d2008eaf0 con 0x7f3d381008d0 2022-09-10T04:27:08.740 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:08.740 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":90,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:26:37.114323+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:27:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.740+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3d24099010 msgr2=0x7f3d2409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.740+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3d24099010 0x7f3d2409b4c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d28009760 tx=0x7f3d28009e70 comp rx=0 tx=0).stop 2022-09-10T04:27:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.740+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 msgr2=0x7f3d3819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:08.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.740+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 0x7f3d3819a760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000e6c0 tx=0x7f3d2000e9d0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.740+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 shutdown_connections 2022-09-10T04:27:08.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3d24099010 0x7f3d2409b4c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d38102d40 0x7f3d3819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d38066c00 0x7f3d3819a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 --2- 172.21.15.81:0/3377395480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d381008d0 0x7f3d3819a760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 >> 172.21.15.81:0/3377395480 conn(0x7f3d38073cf0 msgr2=0x7f3d380ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:08.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 shutdown_connections 2022-09-10T04:27:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:08.741+0000 7f3d3d49b700 1 -- 172.21.15.81:0/3377395480 wait complete. 2022-09-10T04:27:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:08 smithi104 ceph-mon[97044]: pgmap v319: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.0 KiB/s wr, 22 op/s 2022-09-10T04:27:09.221 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2022-09-10T04:27:09.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd dump --format=json 2022-09-10T04:27:09.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3377395480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:09.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2022-09-10T04:27:09.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2022-09-10T04:27:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2022-09-10T04:27:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2022-09-10T04:27:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2022-09-10T04:27:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2022-09-10T04:27:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2022-09-10T04:27:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2022-09-10T04:27:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2022-09-10T04:27:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2022-09-10T04:27:09.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:27:09.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:27:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3377395480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2022-09-10T04:27:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2022-09-10T04:27:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2022-09-10T04:27:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2022-09-10T04:27:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2022-09-10T04:27:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2022-09-10T04:27:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2022-09-10T04:27:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2022-09-10T04:27:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2022-09-10T04:27:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2022-09-10T04:27:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:27:09.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:27:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3377395480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2022-09-10T04:27:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2022-09-10T04:27:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2022-09-10T04:27:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2022-09-10T04:27:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2022-09-10T04:27:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]: dispatch 2022-09-10T04:27:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2022-09-10T04:27:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2022-09-10T04:27:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2022-09-10T04:27:09.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2022-09-10T04:27:09.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:27:09.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:27:10.105 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:10.751 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: pgmap v320: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 20 op/s 2022-09-10T04:27:10.751 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2022-09-10T04:27:10.752 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2022-09-10T04:27:10.752 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2022-09-10T04:27:10.752 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2022-09-10T04:27:10.753 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2022-09-10T04:27:10.753 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2022-09-10T04:27:10.753 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2022-09-10T04:27:10.754 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2022-09-10T04:27:10.754 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2022-09-10T04:27:10.754 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2022-09-10T04:27:10.754 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:10 smithi081 ceph-mon[90133]: osdmap e91: 12 total, 12 up, 12 in 2022-09-10T04:27:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: pgmap v320: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 20 op/s 2022-09-10T04:27:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2022-09-10T04:27:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2022-09-10T04:27:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2022-09-10T04:27:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2022-09-10T04:27:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2022-09-10T04:27:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2022-09-10T04:27:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2022-09-10T04:27:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2022-09-10T04:27:10.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2022-09-10T04:27:10.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2022-09-10T04:27:10.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:10 smithi104 ceph-mon[97044]: osdmap e91: 12 total, 12 up, 12 in 2022-09-10T04:27:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: pgmap v320: 129 pgs: 129 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.1 KiB/s wr, 20 op/s 2022-09-10T04:27:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2022-09-10T04:27:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2022-09-10T04:27:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2022-09-10T04:27:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2022-09-10T04:27:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2022-09-10T04:27:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.25", "id": [5, 9]}]': finished 2022-09-10T04:27:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2022-09-10T04:27:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2022-09-10T04:27:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2022-09-10T04:27:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2022-09-10T04:27:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:10 smithi110 ceph-mon[97633]: osdmap e91: 12 total, 12 up, 12 in 2022-09-10T04:27:11.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 -- 172.21.15.81:0/2463536360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b181031d0 msgr2=0x7f3b1810ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:11.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/2463536360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b181031d0 0x7f3b1810ad70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009b30 tx=0x7f3b10009e40 comp rx=0 tx=0).stop 2022-09-10T04:27:11.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 -- 172.21.15.81:0/2463536360 shutdown_connections 2022-09-10T04:27:11.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/2463536360 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b181031d0 0x7f3b1810ad70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/2463536360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b180fe0d0 0x7f3b180fe4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/2463536360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fea70 0x7f3b18102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 -- 172.21.15.81:0/2463536360 >> 172.21.15.81:0/2463536360 conn(0x7f3b180f9a10 msgr2=0x7f3b180fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:11.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 -- 172.21.15.81:0/2463536360 shutdown_connections 2022-09-10T04:27:11.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.622+0000 7f3b1cb27700 1 -- 172.21.15.81:0/2463536360 wait complete. 2022-09-10T04:27:11.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.623+0000 7f3b1cb27700 1 Processor -- start 2022-09-10T04:27:11.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.623+0000 7f3b1cb27700 1 -- start start 2022-09-10T04:27:11.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1cb27700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fe0d0 0x7f3b18195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:11.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1cb27700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 0x7f3b181963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:11.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1cb27700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b181031d0 0x7f3b1819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:11.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1cb27700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1819aed0 con 0x7f3b180fe0d0 2022-09-10T04:27:11.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1cb27700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1819b040 con 0x7f3b180fea70 2022-09-10T04:27:11.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1cb27700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1819b200 con 0x7f3b181031d0 2022-09-10T04:27:11.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b1659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fe0d0 0x7f3b18195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:11.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b15d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 0x7f3b181963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b15d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 0x7f3b181963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:37866/0 (socket says 172.21.15.81:37866) 2022-09-10T04:27:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b15d9b700 1 -- 172.21.15.81:0/1032781036 learned_addr learned my addr 172.21.15.81:0/1032781036 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:11.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.624+0000 7f3b16d9d700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b181031d0 0x7f3b1819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1659c700 1 -- 172.21.15.81:0/1032781036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b181031d0 msgr2=0x7f3b1819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1659c700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b181031d0 0x7f3b1819a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1659c700 1 -- 172.21.15.81:0/1032781036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 msgr2=0x7f3b181963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1659c700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 0x7f3b181963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1659c700 1 -- 172.21.15.81:0/1032781036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c009470 con 0x7f3b180fe0d0 2022-09-10T04:27:11.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b15d9b700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 0x7f3b181963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:11.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b16d9d700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b181031d0 0x7f3b1819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:11.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1659c700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fe0d0 0x7f3b18195ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800e6c0 tx=0x7f3b0800e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:11.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08010900 con 0x7f3b180fe0d0 2022-09-10T04:27:11.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b10009730 con 0x7f3b180fe0d0 2022-09-10T04:27:11.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3b08018420 con 0x7f3b180fe0d0 2022-09-10T04:27:11.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.625+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08017610 con 0x7f3b180fe0d0 2022-09-10T04:27:11.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.626+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b181a2cd0 con 0x7f3b180fe0d0 2022-09-10T04:27:11.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.627+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b18064ea0 con 0x7f3b180fe0d0 2022-09-10T04:27:11.637 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.628+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3b0800c040 con 0x7f3b180fe0d0 2022-09-10T04:27:11.637 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.630+0000 7f3b077fe700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3b00099090 0x7f3b0009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:11.638 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.631+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f3b08014070 con 0x7f3b180fe0d0 2022-09-10T04:27:11.638 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.631+0000 7f3b15d9b700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3b00099090 0x7f3b0009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:11.638 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.631+0000 7f3b15d9b700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3b00099090 0x7f3b0009b540 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3b181974b0 tx=0x7f3b0c005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:11.638 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.635+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3b0804fc80 con 0x7f3b180fe0d0 2022-09-10T04:27:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:11 smithi104 ceph-mon[97044]: osdmap e92: 12 total, 12 up, 12 in 2022-09-10T04:27:11.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:11 smithi081 ceph-mon[90133]: osdmap e92: 12 total, 12 up, 12 in 2022-09-10T04:27:11.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.846+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b1810dce0 con 0x7f3b180fe0d0 2022-09-10T04:27:11.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.847+0000 7f3b077fe700 1 -- 172.21.15.81:0/1032781036 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+23649 (secure 0 0 0) 0x7f3b0808ea50 con 0x7f3b180fe0d0 2022-09-10T04:27:11.849 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:11.850 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":92,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:27:10.414885+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:27:11.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3b00099090 msgr2=0x7f3b0009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:11.852 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3b00099090 0x7f3b0009b540 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3b181974b0 tx=0x7f3b0c005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:11.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fe0d0 msgr2=0x7f3b18195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:11.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fe0d0 0x7f3b18195ea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800e6c0 tx=0x7f3b0800e9d0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.853 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 shutdown_connections 2022-09-10T04:27:11.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3b00099090 0x7f3b0009b540 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b181031d0 0x7f3b1819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b180fe0d0 0x7f3b18195ea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 --2- 172.21.15.81:0/1032781036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3b180fea70 0x7f3b181963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:11.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 >> 172.21.15.81:0/1032781036 conn(0x7f3b180f9a10 msgr2=0x7f3b180fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:11.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 shutdown_connections 2022-09-10T04:27:11.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:11.850+0000 7f3b1cb27700 1 -- 172.21.15.81:0/1032781036 wait complete. 2022-09-10T04:27:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:11 smithi110 ceph-mon[97633]: osdmap e92: 12 total, 12 up, 12 in 2022-09-10T04:27:12.384 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2022-09-10T04:27:12.387 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2022-09-10T04:27:12.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2022-09-10T04:27:12.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2022-09-10T04:27:12.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2022-09-10T04:27:12.392 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2022-09-10T04:27:12.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2022-09-10T04:27:12.394 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2022-09-10T04:27:12.394 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2022-09-10T04:27:12.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2022-09-10T04:27:12.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2022-09-10T04:27:12.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2022-09-10T04:27:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:12 smithi081 ceph-mon[90133]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2022-09-10T04:27:12.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:12 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1032781036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:12 smithi104 ceph-mon[97044]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2022-09-10T04:27:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:12 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1032781036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:12 smithi110 ceph-mon[97633]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 0 B/s wr, 10 op/s 2022-09-10T04:27:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:12 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1032781036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:14.464 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:14 smithi104 ceph-mon[97044]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:27:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:14 smithi081 ceph-mon[90133]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:27:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:14 smithi110 ceph-mon[97633]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:27:14.709 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:14.958 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.034 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.064 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.264 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.470 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:15.472 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:16 smithi110 ceph-mon[97633]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:16.694 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:16 smithi081 ceph-mon[90133]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:16 smithi104 ceph-mon[97044]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:16.797 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:18.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:18 smithi081 ceph-mon[90133]: pgmap v326: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:18.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:18 smithi110 ceph-mon[97633]: pgmap v326: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:18 smithi104 ceph-mon[97044]: pgmap v326: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:19.392 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 -- 172.21.15.81:0/3419778923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e80fea70 msgr2=0x7f40e8102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 --2- 172.21.15.81:0/3419778923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e80fea70 0x7f40e8102ac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f40d8009750 tx=0x7f40d8009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:19.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 -- 172.21.15.81:0/3419778923 shutdown_connections 2022-09-10T04:27:19.393 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 --2- 172.21.15.81:0/3419778923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40e81031d0 0x7f40e810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 --2- 172.21.15.81:0/3419778923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 0x7f40e80fe4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.394 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 --2- 172.21.15.81:0/3419778923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e80fea70 0x7f40e8102ac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.395 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 -- 172.21.15.81:0/3419778923 >> 172.21.15.81:0/3419778923 conn(0x7f40e80f9a10 msgr2=0x7f40e80fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:19.396 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 -- 172.21.15.81:0/3419778923 shutdown_connections 2022-09-10T04:27:19.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.299+0000 7f40ef90f700 1 -- 172.21.15.81:0/3419778923 wait complete. 2022-09-10T04:27:19.397 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.300+0000 7f40ef90f700 1 Processor -- start 2022-09-10T04:27:19.398 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.300+0000 7f40ef90f700 1 -- start start 2022-09-10T04:27:19.399 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ef90f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 0x7f40e8195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ef90f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40e80fea70 0x7f40e8196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.400 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ef90f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e81031d0 0x7f40e819a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ef90f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e819ae90 con 0x7f40e80fe0d0 2022-09-10T04:27:19.401 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ef90f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e819b000 con 0x7f40e80fea70 2022-09-10T04:27:19.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ef90f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40e819b1c0 con 0x7f40e81031d0 2022-09-10T04:27:19.402 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ed6ab700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 0x7f40e8195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ed6ab700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 0x7f40e8195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45614/0 (socket says 172.21.15.81:45614) 2022-09-10T04:27:19.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40ed6ab700 1 -- 172.21.15.81:0/1893940699 learned_addr learned my addr 172.21.15.81:0/1893940699 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:19.403 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40eceaa700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40e80fea70 0x7f40e8196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.301+0000 7f40edeac700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e81031d0 0x7f40e819a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.404 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ed6ab700 1 -- 172.21.15.81:0/1893940699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e81031d0 msgr2=0x7f40e819a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ed6ab700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40e81031d0 0x7f40e819a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.405 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ed6ab700 1 -- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40e80fea70 msgr2=0x7f40e8196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.406 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ed6ab700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f40e80fea70 0x7f40e8196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ed6ab700 1 -- 172.21.15.81:0/1893940699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80093b0 con 0x7f40e80fe0d0 2022-09-10T04:27:19.407 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ed6ab700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 0x7f40e8195f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40e400e740 tx=0x7f40e400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:19.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4005220 con 0x7f40e80fe0d0 2022-09-10T04:27:19.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40e819b4a0 con 0x7f40e80fe0d0 2022-09-10T04:27:19.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f40e4005380 con 0x7f40e80fe0d0 2022-09-10T04:27:19.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e4010ba0 con 0x7f40e80fe0d0 2022-09-10T04:27:19.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.302+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e81a2ce0 con 0x7f40e80fe0d0 2022-09-10T04:27:19.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.303+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40e8072260 con 0x7f40e80fe0d0 2022-09-10T04:27:19.411 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.305+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f40e401e020 con 0x7f40e80fe0d0 2022-09-10T04:27:19.411 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.306+0000 7f40de7fc700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f40d4099010 0x7f40d409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.411 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f40e4014070 con 0x7f40e80fe0d0 2022-09-10T04:27:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40eceaa700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f40d4099010 0x7f40d409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.412 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40de7fc700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] conn(0x7f40d40a4020 0x7f40d40a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.413 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 --> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40d40a6b80 con 0x7f40d40a4020 2022-09-10T04:27:19.413 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40e40c3470 con 0x7f40e80fe0d0 2022-09-10T04:27:19.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40edeac700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] conn(0x7f40d40a4020 0x7f40d40a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.307+0000 7f40eceaa700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f40d4099010 0x7f40d409b4c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40d800f5e0 tx=0x7f40d8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:19.415 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.308+0000 7f40edeac700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] conn(0x7f40d40a4020 0x7f40d40a64b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:19.415 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.309+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== osd.7 v2:172.21.15.104:6816/4207080292 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f40d40a6b80 con 0x7f40d40a4020 2022-09-10T04:27:19.415 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.327+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 --> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40e8064ea0 con 0x7f40d40a4020 2022-09-10T04:27:19.416 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.327+0000 7f40de7fc700 1 -- 172.21.15.81:0/1893940699 <== osd.7 v2:172.21.15.104:6816/4207080292 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f40e8064ea0 con 0x7f40d40a4020 2022-09-10T04:27:19.416 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] conn(0x7f40d40a4020 msgr2=0x7f40d40a64b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] conn(0x7f40d40a4020 0x7f40d40a64b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f40d4099010 msgr2=0x7f40d409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f40d4099010 0x7f40d409b4c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f40d800f5e0 tx=0x7f40d8005fd0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 msgr2=0x7f40e8195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 --2- 172.21.15.81:0/1893940699 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40e80fe0d0 0x7f40e8195f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f40e400e740 tx=0x7f40e400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40edeac700 1 -- 172.21.15.81:0/1893940699 reap_dead start 2022-09-10T04:27:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 shutdown_connections 2022-09-10T04:27:19.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 >> 172.21.15.81:0/1893940699 conn(0x7f40e80f9a10 msgr2=0x7f40e80fa790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:19.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 shutdown_connections 2022-09-10T04:27:19.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.328+0000 7f40ef90f700 1 -- 172.21.15.81:0/1893940699 wait complete. 2022-09-10T04:27:19.548 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:19.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 -- 172.21.15.81:0/2791206978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb880fea70 msgr2=0x7fcb88102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/2791206978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb880fea70 0x7fcb88102ac0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb78009810 tx=0x7fcb78009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:19.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 -- 172.21.15.81:0/2791206978 shutdown_connections 2022-09-10T04:27:19.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/2791206978 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb881031d0 0x7fcb8810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/2791206978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 0x7fcb880fe4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/2791206978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb880fea70 0x7fcb88102ac0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 -- 172.21.15.81:0/2791206978 >> 172.21.15.81:0/2791206978 conn(0x7fcb880f9a10 msgr2=0x7fcb880fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:19.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 -- 172.21.15.81:0/2791206978 shutdown_connections 2022-09-10T04:27:19.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.889+0000 7fcb8f29e700 1 -- 172.21.15.81:0/2791206978 wait complete. 2022-09-10T04:27:19.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.890+0000 7fcb8f29e700 1 Processor -- start 2022-09-10T04:27:19.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.890+0000 7fcb8f29e700 1 -- start start 2022-09-10T04:27:19.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8f29e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 0x7fcb88195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8f29e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb880fea70 0x7fcb88196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8f29e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb881031d0 0x7fcb8819a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8f29e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8819aee0 con 0x7fcb880fe0d0 2022-09-10T04:27:19.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8f29e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8819b050 con 0x7fcb880fea70 2022-09-10T04:27:19.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8f29e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb8819b210 con 0x7fcb881031d0 2022-09-10T04:27:19.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8d03a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 0x7fcb88195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8c839700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb880fea70 0x7fcb88196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8d03a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 0x7fcb88195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60184/0 (socket says 172.21.15.81:60184) 2022-09-10T04:27:19.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8d03a700 1 -- 172.21.15.81:0/1316528768 learned_addr learned my addr 172.21.15.81:0/1316528768 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:19.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.891+0000 7fcb8d83b700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb881031d0 0x7fcb8819a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d03a700 1 -- 172.21.15.81:0/1316528768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb881031d0 msgr2=0x7fcb8819a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d03a700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb881031d0 0x7fcb8819a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d03a700 1 -- 172.21.15.81:0/1316528768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb880fea70 msgr2=0x7fcb88196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d03a700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb880fea70 0x7fcb88196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d03a700 1 -- 172.21.15.81:0/1316528768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb840093b0 con 0x7fcb880fe0d0 2022-09-10T04:27:19.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d83b700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb881031d0 0x7fcb8819a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:27:19.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8c839700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcb880fea70 0x7fcb88196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:19.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8d03a700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 0x7fcb88195f50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e9c0 tx=0x7fcb8400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:19.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb840097b0 con 0x7fcb880fe0d0 2022-09-10T04:27:19.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb78009470 con 0x7fcb880fe0d0 2022-09-10T04:27:19.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcb8400ce70 con 0x7fcb880fe0d0 2022-09-10T04:27:19.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb84003e30 con 0x7fcb880fe0d0 2022-09-10T04:27:19.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.892+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb881a2ce0 con 0x7fcb880fe0d0 2022-09-10T04:27:19.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.893+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb88072260 con 0x7fcb880fe0d0 2022-09-10T04:27:19.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.895+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fcb8401e020 con 0x7fcb880fe0d0 2022-09-10T04:27:19.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.896+0000 7fcb7e7fc700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcb74099090 0x7fcb7409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.897+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fcb84014070 con 0x7fcb880fe0d0 2022-09-10T04:27:19.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.897+0000 7fcb8c839700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcb74099090 0x7fcb7409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.897+0000 7fcb7e7fc700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] conn(0x7fcb740a40a0 0x7fcb740a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:19.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.897+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 --> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb740a6c00 con 0x7fcb740a40a0 2022-09-10T04:27:19.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.897+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb840c3040 con 0x7fcb880fe0d0 2022-09-10T04:27:19.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.897+0000 7fcb8d83b700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] conn(0x7fcb740a40a0 0x7fcb740a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:19.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.898+0000 7fcb8c839700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcb74099090 0x7fcb7409b540 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb88197560 tx=0x7fcb78009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:19.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.898+0000 7fcb8d83b700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] conn(0x7fcb740a40a0 0x7fcb740a6530 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:19.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.899+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== osd.3 v2:172.21.15.81:6810/1343686033 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fcb740a6c00 con 0x7fcb740a40a0 2022-09-10T04:27:19.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.916+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 --> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb88064ea0 con 0x7fcb740a40a0 2022-09-10T04:27:19.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.916+0000 7fcb7e7fc700 1 -- 172.21.15.81:0/1316528768 <== osd.3 v2:172.21.15.81:6810/1343686033 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcb88064ea0 con 0x7fcb740a40a0 2022-09-10T04:27:19.929 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] conn(0x7fcb740a40a0 msgr2=0x7fcb740a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.929 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] conn(0x7fcb740a40a0 0x7fcb740a6530 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcb74099090 msgr2=0x7fcb7409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcb74099090 0x7fcb7409b540 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb88197560 tx=0x7fcb78009eb0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 msgr2=0x7fcb88195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:19.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 --2- 172.21.15.81:0/1316528768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcb880fe0d0 0x7fcb88195f50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400e9c0 tx=0x7fcb8400ecd0 comp rx=0 tx=0).stop 2022-09-10T04:27:19.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8d83b700 1 -- 172.21.15.81:0/1316528768 reap_dead start 2022-09-10T04:27:19.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 shutdown_connections 2022-09-10T04:27:19.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 >> 172.21.15.81:0/1316528768 conn(0x7fcb880f9a10 msgr2=0x7fcb880fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:19.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 shutdown_connections 2022-09-10T04:27:19.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:19.917+0000 7fcb8f29e700 1 -- 172.21.15.81:0/1316528768 wait complete. 2022-09-10T04:27:20.593 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:20 smithi081 ceph-mon[90133]: pgmap v327: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:20.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:20 smithi104 ceph-mon[97044]: pgmap v327: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:20 smithi110 ceph-mon[97633]: pgmap v327: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:21.681 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 -- 172.21.15.81:0/1485082615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face00ff5e0 msgr2=0x7face0107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 --2- 172.21.15.81:0/1485082615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face00ff5e0 0x7face0107d40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7facd4009810 tx=0x7facd4009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 -- 172.21.15.81:0/1485082615 shutdown_connections 2022-09-10T04:27:21.682 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 --2- 172.21.15.81:0/1485082615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7face0108280 0x7face010a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.683 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 --2- 172.21.15.81:0/1485082615 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7face00fecd0 0x7face00ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.683 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 --2- 172.21.15.81:0/1485082615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face00ff5e0 0x7face0107d40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.683 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 -- 172.21.15.81:0/1485082615 >> 172.21.15.81:0/1485082615 conn(0x7face00f8710 msgr2=0x7face00fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:21.684 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 -- 172.21.15.81:0/1485082615 shutdown_connections 2022-09-10T04:27:21.684 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.856+0000 7facdf59e700 1 -- 172.21.15.81:0/1485082615 wait complete. 2022-09-10T04:27:21.684 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.858+0000 7facdf59e700 1 Processor -- start 2022-09-10T04:27:21.684 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.858+0000 7facdf59e700 1 -- start start 2022-09-10T04:27:21.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.859+0000 7facdf59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7face00fecd0 0x7face00ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.859+0000 7facdf59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7face00ff5e0 0x7face0100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.859+0000 7facdf59e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face0108280 0x7face01007d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.859+0000 7facdf59e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face01a6440 con 0x7face0108280 2022-09-10T04:27:21.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.859+0000 7facdf59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face0195a90 con 0x7face00fecd0 2022-09-10T04:27:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.859+0000 7facdf59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face0195c50 con 0x7face00ff5e0 2022-09-10T04:27:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.860+0000 7facded9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face0108280 0x7face01007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.860+0000 7facded9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face0108280 0x7face01007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60194/0 (socket says 172.21.15.81:60194) 2022-09-10T04:27:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.860+0000 7facded9d700 1 -- 172.21.15.81:0/3133689010 learned_addr learned my addr 172.21.15.81:0/3133689010 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:21.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.860+0000 7facde59c700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7face00fecd0 0x7face00ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.860+0000 7facddd9b700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7face00ff5e0 0x7face0100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facded9d700 1 -- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7face00ff5e0 msgr2=0x7face0100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facded9d700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7face00ff5e0 0x7face0100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facded9d700 1 -- 172.21.15.81:0/3133689010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7face00fecd0 msgr2=0x7face00ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facded9d700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7face00fecd0 0x7face00ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facded9d700 1 -- 172.21.15.81:0/3133689010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4009470 con 0x7face0108280 2022-09-10T04:27:21.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facde59c700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7face00fecd0 0x7face00ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:21.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facded9d700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face0108280 0x7face01007d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7facd800eaf0 tx=0x7facd800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:21.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd8009960 con 0x7face0108280 2022-09-10T04:27:21.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.861+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face0195f30 con 0x7face0108280 2022-09-10T04:27:21.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.862+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7facd800ce70 con 0x7face0108280 2022-09-10T04:27:21.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.862+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd80058b0 con 0x7face0108280 2022-09-10T04:27:21.692 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.862+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face0196440 con 0x7face0108280 2022-09-10T04:27:21.692 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.863+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7face00fe640 con 0x7face0108280 2022-09-10T04:27:21.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.864+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7facd801e020 con 0x7face0108280 2022-09-10T04:27:21.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.866+0000 7faccf7fe700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facc8099090 0x7facc809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.694 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.866+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7facd8014070 con 0x7face0108280 2022-09-10T04:27:21.694 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.866+0000 7facde59c700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facc8099090 0x7facc809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.866+0000 7faccf7fe700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] conn(0x7facc80a40a0 0x7facc80a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.866+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 --> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7facc80a6c00 con 0x7facc80a40a0 2022-09-10T04:27:21.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.866+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7facd80c72a0 con 0x7face0108280 2022-09-10T04:27:21.695 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.867+0000 7facddd9b700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] conn(0x7facc80a40a0 0x7facc80a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.867+0000 7facde59c700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facc8099090 0x7facc809b540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7facd0005fd0 tx=0x7facd0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:21.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.868+0000 7facddd9b700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] conn(0x7facc80a40a0 0x7facc80a6530 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:21.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.868+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== osd.8 v2:172.21.15.110:6816/462969903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7facc80a6c00 con 0x7facc80a40a0 2022-09-10T04:27:21.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.889+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 --> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7face004d480 con 0x7facc80a40a0 2022-09-10T04:27:21.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7faccf7fe700 1 -- 172.21.15.81:0/3133689010 <== osd.8 v2:172.21.15.110:6816/462969903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7face004d480 con 0x7facc80a40a0 2022-09-10T04:27:21.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] conn(0x7facc80a40a0 msgr2=0x7facc80a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] conn(0x7facc80a40a0 0x7facc80a6530 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facc8099090 msgr2=0x7facc809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facc8099090 0x7facc809b540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7facd0005fd0 tx=0x7facd0005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face0108280 msgr2=0x7face01007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 --2- 172.21.15.81:0/3133689010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7face0108280 0x7face01007d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7facd800eaf0 tx=0x7facd800ee00 comp rx=0 tx=0).stop 2022-09-10T04:27:21.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facded9d700 1 -- 172.21.15.81:0/3133689010 reap_dead start 2022-09-10T04:27:21.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 shutdown_connections 2022-09-10T04:27:21.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.890+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 >> 172.21.15.81:0/3133689010 conn(0x7face00f8710 msgr2=0x7face010a400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:21.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.891+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 shutdown_connections 2022-09-10T04:27:21.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:20.891+0000 7facdf59e700 1 -- 172.21.15.81:0/3133689010 wait complete. 2022-09-10T04:27:21.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.824+0000 7fafe1172700 1 -- 172.21.15.81:0/696442949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100ab0 msgr2=0x7fafdc104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.824+0000 7fafe1172700 1 --2- 172.21.15.81:0/696442949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100ab0 0x7fafdc104b00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009840 tx=0x7fafd0009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:21.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.824+0000 7fafe1172700 1 -- 172.21.15.81:0/696442949 shutdown_connections 2022-09-10T04:27:21.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.825+0000 7fafe1172700 1 --2- 172.21.15.81:0/696442949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafdc105210 0x7fafdc10cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.825+0000 7fafe1172700 1 --2- 172.21.15.81:0/696442949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafdc100110 0x7fafdc1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.825+0000 7fafe1172700 1 --2- 172.21.15.81:0/696442949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100ab0 0x7fafdc104b00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.825+0000 7fafe1172700 1 -- 172.21.15.81:0/696442949 >> 172.21.15.81:0/696442949 conn(0x7fafdc0fba90 msgr2=0x7fafdc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:21.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.825+0000 7fafe1172700 1 -- 172.21.15.81:0/696442949 shutdown_connections 2022-09-10T04:27:21.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.825+0000 7fafe1172700 1 -- 172.21.15.81:0/696442949 wait complete. 2022-09-10T04:27:21.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.826+0000 7fafe1172700 1 Processor -- start 2022-09-10T04:27:21.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 -- start start 2022-09-10T04:27:21.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100110 0x7fafdc10c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafdc100ab0 0x7fafdc10cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafdc105210 0x7fafdc109ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc10a280 con 0x7fafdc105210 2022-09-10T04:27:21.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc10a3f0 con 0x7fafdc100110 2022-09-10T04:27:21.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.827+0000 7fafe1172700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc10a5b0 con 0x7fafdc100ab0 2022-09-10T04:27:21.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100110 0x7fafdc10c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100110 0x7fafdc10c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45640/0 (socket says 172.21.15.81:45640) 2022-09-10T04:27:21.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 -- 172.21.15.81:0/2392294156 learned_addr learned my addr 172.21.15.81:0/2392294156 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdb7fe700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafdc100ab0 0x7fafdc10cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafe0971700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafdc105210 0x7fafdc109ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 -- 172.21.15.81:0/2392294156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafdc100ab0 msgr2=0x7fafdc10cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fafdc100ab0 0x7fafdc10cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 -- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafdc105210 msgr2=0x7fafdc109ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafdc105210 0x7fafdc109ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.828+0000 7fafdbfff700 1 -- 172.21.15.81:0/2392294156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd0009470 con 0x7fafdc100110 2022-09-10T04:27:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.829+0000 7fafdbfff700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100110 0x7fafdc10c840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00e740 tx=0x7fafcc00ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:21.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.829+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc005220 con 0x7fafdc100110 2022-09-10T04:27:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.829+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fafcc005380 con 0x7fafdc100110 2022-09-10T04:27:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.829+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc004c10 con 0x7fafdc100110 2022-09-10T04:27:21.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.829+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc19c230 con 0x7fafdc100110 2022-09-10T04:27:21.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.829+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc19c6c0 con 0x7fafdc100110 2022-09-10T04:27:21.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.831+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafdc0fd000 con 0x7fafdc100110 2022-09-10T04:27:21.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.833+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fafcc01e020 con 0x7fafdc100110 2022-09-10T04:27:21.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.834+0000 7fafd97fa700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fafc4099090 0x7fafc409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.834+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fafcc014070 con 0x7fafdc100110 2022-09-10T04:27:21.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.835+0000 7fafd97fa700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] conn(0x7fafc40a4020 0x7fafc40a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:21.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.835+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 --> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafc40a6b80 con 0x7fafc40a4020 2022-09-10T04:27:21.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.835+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafcc092820 con 0x7fafdc100110 2022-09-10T04:27:21.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.835+0000 7fafe0971700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] conn(0x7fafc40a4020 0x7fafc40a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.836+0000 7fafe0971700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] conn(0x7fafc40a4020 0x7fafc40a64b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:21.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.837+0000 7fafdb7fe700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fafc4099090 0x7fafc409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.838+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== osd.6 v2:172.21.15.81:6818/2282910145 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fafc40a6b80 con 0x7fafc40a4020 2022-09-10T04:27:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.838+0000 7fafdb7fe700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fafc4099090 0x7fafc409b540 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009810 tx=0x7fafd0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:21.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.859+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 --> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fafdc04d480 con 0x7fafc40a4020 2022-09-10T04:27:21.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafd97fa700 1 -- 172.21.15.81:0/2392294156 <== osd.6 v2:172.21.15.81:6818/2282910145 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fafdc04d480 con 0x7fafc40a4020 2022-09-10T04:27:21.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] conn(0x7fafc40a4020 msgr2=0x7fafc40a64b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] conn(0x7fafc40a4020 0x7fafc40a64b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fafc4099090 msgr2=0x7fafc409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fafc4099090 0x7fafc409b540 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009810 tx=0x7fafd0000bc0 comp rx=0 tx=0).stop 2022-09-10T04:27:21.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100110 msgr2=0x7fafdc10c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:21.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 --2- 172.21.15.81:0/2392294156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafdc100110 0x7fafdc10c840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00e740 tx=0x7fafcc00ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:21.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe0971700 1 -- 172.21.15.81:0/2392294156 reap_dead start 2022-09-10T04:27:21.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 shutdown_connections 2022-09-10T04:27:21.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.860+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 >> 172.21.15.81:0/2392294156 conn(0x7fafdc0fba90 msgr2=0x7fafdc0fc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:21.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.861+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 shutdown_connections 2022-09-10T04:27:21.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:21.861+0000 7fafe1172700 1 -- 172.21.15.81:0/2392294156 wait complete. 2022-09-10T04:27:22.535 INFO:teuthology.orchestra.run.smithi081.stdout:223338299434 2022-09-10T04:27:22.536 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.7 2022-09-10T04:27:22.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:22 smithi081 ceph-mon[90133]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:22.748 INFO:teuthology.orchestra.run.smithi081.stdout:111669149760 2022-09-10T04:27:22.749 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.3 2022-09-10T04:27:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:22 smithi104 ceph-mon[97044]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:22 smithi110 ceph-mon[97633]: pgmap v328: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:23.447 INFO:teuthology.orchestra.run.smithi081.stdout:257698037797 2022-09-10T04:27:23.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.8 2022-09-10T04:27:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.424+0000 7facc9380700 1 -- 172.21.15.81:0/44874070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4105400 msgr2=0x7facc410cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.424+0000 7facc9380700 1 --2- 172.21.15.81:0/44874070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4105400 0x7facc410cfa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7facbc009be0 tx=0x7facbc009ef0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 -- 172.21.15.81:0/44874070 shutdown_connections 2022-09-10T04:27:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 --2- 172.21.15.81:0/44874070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4105400 0x7facc410cfa0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 --2- 172.21.15.81:0/44874070 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc4100300 0x7facc41006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 --2- 172.21.15.81:0/44874070 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc4100ca0 0x7facc4104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 -- 172.21.15.81:0/44874070 >> 172.21.15.81:0/44874070 conn(0x7facc40fbc20 msgr2=0x7facc40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 -- 172.21.15.81:0/44874070 shutdown_connections 2022-09-10T04:27:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.425+0000 7facc9380700 1 -- 172.21.15.81:0/44874070 wait complete. 2022-09-10T04:27:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.426+0000 7facc9380700 1 Processor -- start 2022-09-10T04:27:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.426+0000 7facc9380700 1 -- start start 2022-09-10T04:27:23.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc9380700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc4100300 0x7facc4195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc9380700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4100ca0 0x7facc4196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc9380700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc4105400 0x7facc419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc9380700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc419af40 con 0x7facc4100ca0 2022-09-10T04:27:23.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc9380700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc419b0b0 con 0x7facc4105400 2022-09-10T04:27:23.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc9380700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc419b270 con 0x7facc4100300 2022-09-10T04:27:23.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc27fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4100ca0 0x7facc4196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc27fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4100ca0 0x7facc4196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60246/0 (socket says 172.21.15.81:60246) 2022-09-10T04:27:23.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc27fc700 1 -- 172.21.15.81:0/3622640212 learned_addr learned my addr 172.21.15.81:0/3622640212 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:23.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.427+0000 7facc2ffd700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc4100300 0x7facc4195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc27fc700 1 -- 172.21.15.81:0/3622640212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc4100300 msgr2=0x7facc4195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc27fc700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc4100300 0x7facc4195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc27fc700 1 -- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc4105400 msgr2=0x7facc419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc37fe700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc4105400 0x7facc419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc27fc700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7facc4105400 0x7facc419a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc27fc700 1 -- 172.21.15.81:0/3622640212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0097e0 con 0x7facc4100ca0 2022-09-10T04:27:23.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.428+0000 7facc27fc700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4100ca0 0x7facc4196450 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7facb800e740 tx=0x7facb800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:23.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.429+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8005220 con 0x7facc4100ca0 2022-09-10T04:27:23.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.429+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7facb8005380 con 0x7facc4100ca0 2022-09-10T04:27:23.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.429+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8004c10 con 0x7facc4100ca0 2022-09-10T04:27:23.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.429+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc419b550 con 0x7facc4100ca0 2022-09-10T04:27:23.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.429+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc41a2d40 con 0x7facc4100ca0 2022-09-10T04:27:23.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.430+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7facc40fd100 con 0x7facc4100ca0 2022-09-10T04:27:23.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.432+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7facb801e020 con 0x7facc4100ca0 2022-09-10T04:27:23.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.434+0000 7facabfff700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facac099110 0x7facac09b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.434+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7facb8014070 con 0x7facc4100ca0 2022-09-10T04:27:23.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.435+0000 7facabfff700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] conn(0x7facac0a4120 0x7facac0a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.435+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 --> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7facac0a6c80 con 0x7facac0a4120 2022-09-10T04:27:23.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.435+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7facb80c3720 con 0x7facc4100ca0 2022-09-10T04:27:23.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.436+0000 7facc2ffd700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facac099110 0x7facac09b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.436+0000 7facc37fe700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] conn(0x7facac0a4120 0x7facac0a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.437+0000 7facc2ffd700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facac099110 0x7facac09b5c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7facb4005fd0 tx=0x7facb4005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:23.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.437+0000 7facc37fe700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] conn(0x7facac0a4120 0x7facac0a65b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:23.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.438+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== osd.10 v2:172.21.15.104:6824/675537330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7facac0a6c80 con 0x7facac0a4120 2022-09-10T04:27:23.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.457+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 --> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7facc4064ea0 con 0x7facac0a4120 2022-09-10T04:27:23.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facabfff700 1 -- 172.21.15.81:0/3622640212 <== osd.10 v2:172.21.15.104:6824/675537330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7facc4064ea0 con 0x7facac0a4120 2022-09-10T04:27:23.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] conn(0x7facac0a4120 msgr2=0x7facac0a65b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] conn(0x7facac0a4120 0x7facac0a65b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facac099110 msgr2=0x7facac09b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7facac099110 0x7facac09b5c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7facb4005fd0 tx=0x7facb4005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4100ca0 msgr2=0x7facc4196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.882 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 --2- 172.21.15.81:0/3622640212 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc4100ca0 0x7facc4196450 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7facb800e740 tx=0x7facb800ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:23.882 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc37fe700 1 -- 172.21.15.81:0/3622640212 reap_dead start 2022-09-10T04:27:23.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 shutdown_connections 2022-09-10T04:27:23.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.458+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 >> 172.21.15.81:0/3622640212 conn(0x7facc40fbc20 msgr2=0x7facc40fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.459+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 shutdown_connections 2022-09-10T04:27:23.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.459+0000 7facc9380700 1 -- 172.21.15.81:0/3622640212 wait complete. 2022-09-10T04:27:23.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.685+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/2153943808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 msgr2=0x7fc5f41006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.685+0000 7fc5f9ae6700 1 --2- 172.21.15.81:0/2153943808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f41006d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4009840 tx=0x7fc5e4009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:23.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/2153943808 shutdown_connections 2022-09-10T04:27:23.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 --2- 172.21.15.81:0/2153943808 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5f4105400 0x7fc5f410cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 --2- 172.21.15.81:0/2153943808 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f41006d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 --2- 172.21.15.81:0/2153943808 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5f4100ca0 0x7fc5f4104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/2153943808 >> 172.21.15.81:0/2153943808 conn(0x7fc5f40fbc20 msgr2=0x7fc5f40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:23.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/2153943808 shutdown_connections 2022-09-10T04:27:23.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.686+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/2153943808 wait complete. 2022-09-10T04:27:23.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.687+0000 7fc5f9ae6700 1 Processor -- start 2022-09-10T04:27:23.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 -- start start 2022-09-10T04:27:23.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f4195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5f4100ca0 0x7fc5f4196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5f4105400 0x7fc5f419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f419aef0 con 0x7fc5f4100ca0 2022-09-10T04:27:23.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f419b060 con 0x7fc5f4100300 2022-09-10T04:27:23.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.688+0000 7fc5f9ae6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f419b220 con 0x7fc5f4105400 2022-09-10T04:27:23.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f4195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f4195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45690/0 (socket says 172.21.15.81:45690) 2022-09-10T04:27:23.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 -- 172.21.15.81:0/844714871 learned_addr learned my addr 172.21.15.81:0/844714871 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:23.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f3fff700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5f4105400 0x7fc5f419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f2ffd700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5f4100ca0 0x7fc5f4196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 -- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5f4105400 msgr2=0x7fc5f419a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5f4105400 0x7fc5f419a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 -- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5f4100ca0 msgr2=0x7fc5f4196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5f4100ca0 0x7fc5f4196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f37fe700 1 -- 172.21.15.81:0/844714871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4009470 con 0x7fc5f4100300 2022-09-10T04:27:23.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.689+0000 7fc5f2ffd700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5f4100ca0 0x7fc5f4196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:27:23.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.690+0000 7fc5f37fe700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f4195ec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400f690 tx=0x7fc5e400f770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:23.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.690+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e401d070 con 0x7fc5f4100300 2022-09-10T04:27:23.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.690+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fc5e400b480 con 0x7fc5f4100300 2022-09-10T04:27:23.917 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.690+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4004980 con 0x7fc5f4100300 2022-09-10T04:27:23.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.690+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/844714871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f419b4a0 con 0x7fc5f4100300 2022-09-10T04:27:23.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.690+0000 7fc5f9ae6700 1 -- 172.21.15.81:0/844714871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f41a2cf0 con 0x7fc5f4100300 2022-09-10T04:27:23.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.691+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc5d0000fc0 con 0x7fc5f4100300 2022-09-10T04:27:23.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.693+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fc5e4028030 con 0x7fc5f4100300 2022-09-10T04:27:23.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.695+0000 7fc5f0ff9700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc5dc099090 0x7fc5dc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.695+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fc5e40c3610 con 0x7fc5f4100300 2022-09-10T04:27:23.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.696+0000 7fc5f0ff9700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] conn(0x7fc5dc0a40a0 0x7fc5dc0a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:23.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.696+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 --> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5dc0a6c00 con 0x7fc5dc0a40a0 2022-09-10T04:27:23.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.696+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5e40c3980 con 0x7fc5f4100300 2022-09-10T04:27:23.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.696+0000 7fc5f2ffd700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc5dc099090 0x7fc5dc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.697+0000 7fc5f3fff700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] conn(0x7fc5dc0a40a0 0x7fc5dc0a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:23.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.697+0000 7fc5f3fff700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] conn(0x7fc5dc0a40a0 0x7fc5dc0a6530 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:23.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.697+0000 7fc5f2ffd700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc5dc099090 0x7fc5dc09b540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc5f41974d0 tx=0x7fc5e8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:23.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.698+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== osd.11 v2:172.21.15.110:6824/595897724 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fc5dc0a6c00 con 0x7fc5dc0a40a0 2022-09-10T04:27:23.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.735+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 --> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc5d0002bc0 con 0x7fc5dc0a40a0 2022-09-10T04:27:23.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.737+0000 7fc5f0ff9700 1 -- 172.21.15.81:0/844714871 <== osd.11 v2:172.21.15.110:6824/595897724 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc5d0002bc0 con 0x7fc5dc0a40a0 2022-09-10T04:27:23.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] conn(0x7fc5dc0a40a0 msgr2=0x7fc5dc0a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5da7fc700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] conn(0x7fc5dc0a40a0 0x7fc5dc0a6530 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc5dc099090 msgr2=0x7fc5dc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5da7fc700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fc5dc099090 0x7fc5dc09b540 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc5f41974d0 tx=0x7fc5e8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:23.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 msgr2=0x7fc5f4195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:23.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5da7fc700 1 --2- 172.21.15.81:0/844714871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc5f4100300 0x7fc5f4195ec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400f690 tx=0x7fc5e400f770 comp rx=0 tx=0).stop 2022-09-10T04:27:23.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.738+0000 7fc5f3fff700 1 -- 172.21.15.81:0/844714871 reap_dead start 2022-09-10T04:27:23.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.739+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 shutdown_connections 2022-09-10T04:27:23.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.739+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 >> 172.21.15.81:0/844714871 conn(0x7fc5f40fbc20 msgr2=0x7fc5f40fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:23.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.739+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 shutdown_connections 2022-09-10T04:27:23.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.739+0000 7fc5da7fc700 1 -- 172.21.15.81:0/844714871 wait complete. 2022-09-10T04:27:24.022 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 -- 172.21.15.81:0/2027084545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 msgr2=0x7f888c1006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.022 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 --2- 172.21.15.81:0/2027084545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c1006b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f887c009840 tx=0x7f887c009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:24.022 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 -- 172.21.15.81:0/2027084545 shutdown_connections 2022-09-10T04:27:24.022 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 --2- 172.21.15.81:0/2027084545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f888c1053e0 0x7f888c10cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.023 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 --2- 172.21.15.81:0/2027084545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c1006b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.023 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 --2- 172.21.15.81:0/2027084545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888c100c80 0x7f888c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.023 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.141+0000 7f8891785700 1 -- 172.21.15.81:0/2027084545 >> 172.21.15.81:0/2027084545 conn(0x7f888c0fbc20 msgr2=0x7f888c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:24.023 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.142+0000 7f8891785700 1 -- 172.21.15.81:0/2027084545 shutdown_connections 2022-09-10T04:27:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.142+0000 7f8891785700 1 -- 172.21.15.81:0/2027084545 wait complete. 2022-09-10T04:27:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 Processor -- start 2022-09-10T04:27:24.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 -- start start 2022-09-10T04:27:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f888c100c80 0x7f888c196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888c1053e0 0x7f888c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.026 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c19aef0 con 0x7f888c1002e0 2022-09-10T04:27:24.026 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c19b060 con 0x7f888c100c80 2022-09-10T04:27:24.026 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.143+0000 7f8891785700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c19b220 con 0x7f888c1053e0 2022-09-10T04:27:24.027 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.027 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60230/0 (socket says 172.21.15.81:60230) 2022-09-10T04:27:24.028 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 -- 172.21.15.81:0/3021976567 learned_addr learned my addr 172.21.15.81:0/3021976567 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:24.028 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888a7fc700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f888c100c80 0x7f888c196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.028 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888b7fe700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888c1053e0 0x7f888c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 -- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888c1053e0 msgr2=0x7f888c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f888c1053e0 0x7f888c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 -- 172.21.15.81:0/3021976567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f888c100c80 msgr2=0x7f888c196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.030 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f888c100c80 0x7f888c196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.030 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.144+0000 7f888affd700 1 -- 172.21.15.81:0/3021976567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c009470 con 0x7f888c1002e0 2022-09-10T04:27:24.030 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.145+0000 7f888affd700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c195ec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c010 tx=0x7f887c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:24.031 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.145+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c01d070 con 0x7f888c1002e0 2022-09-10T04:27:24.031 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.145+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c19b4a0 con 0x7f888c1002e0 2022-09-10T04:27:24.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.145+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f887c00b480 con 0x7f888c1002e0 2022-09-10T04:27:24.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.145+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c004760 con 0x7f888c1002e0 2022-09-10T04:27:24.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.145+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c1a2cf0 con 0x7f888c1002e0 2022-09-10T04:27:24.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.146+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f888c0fd140 con 0x7f888c1002e0 2022-09-10T04:27:24.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.148+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f887c028030 con 0x7f888c1002e0 2022-09-10T04:27:24.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.149+0000 7f886ffff700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8874099090 0x7f887409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.150+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f887c0c3ca0 con 0x7f888c1002e0 2022-09-10T04:27:24.034 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.150+0000 7f888a7fc700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8874099090 0x7f887409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.150+0000 7f886ffff700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] conn(0x7f88740a40a0 0x7f88740a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.150+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 --> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f88740a6c00 con 0x7f88740a40a0 2022-09-10T04:27:24.035 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.150+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f887c0c4010 con 0x7f888c1002e0 2022-09-10T04:27:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.150+0000 7f888b7fe700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] conn(0x7f88740a40a0 0x7f88740a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.151+0000 7f888a7fc700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8874099090 0x7f887409b540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f888c1974d0 tx=0x7f8880005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:24.036 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.151+0000 7f888b7fe700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] conn(0x7f88740a40a0 0x7f88740a6530 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.152+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== osd.5 v2:172.21.15.110:6808/3775240590 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f88740a6c00 con 0x7f88740a40a0 2022-09-10T04:27:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.169+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 --> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f888c064ea0 con 0x7f88740a40a0 2022-09-10T04:27:24.037 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.170+0000 7f886ffff700 1 -- 172.21.15.81:0/3021976567 <== osd.5 v2:172.21.15.110:6808/3775240590 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f888c064ea0 con 0x7f88740a40a0 2022-09-10T04:27:24.039 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.170+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] conn(0x7f88740a40a0 msgr2=0x7f88740a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.039 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.170+0000 7f8891785700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] conn(0x7f88740a40a0 0x7f88740a6530 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8874099090 msgr2=0x7f887409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8874099090 0x7f887409b540 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f888c1974d0 tx=0x7f8880005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 msgr2=0x7f888c195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 --2- 172.21.15.81:0/3021976567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888c1002e0 0x7f888c195ec0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c010 tx=0x7f887c00f690 comp rx=0 tx=0).stop 2022-09-10T04:27:24.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f888b7fe700 1 -- 172.21.15.81:0/3021976567 reap_dead start 2022-09-10T04:27:24.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 shutdown_connections 2022-09-10T04:27:24.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 >> 172.21.15.81:0/3021976567 conn(0x7f888c0fbc20 msgr2=0x7f888c0fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:24.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 shutdown_connections 2022-09-10T04:27:24.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:23.171+0000 7f8891785700 1 -- 172.21.15.81:0/3021976567 wait complete. 2022-09-10T04:27:24.062 INFO:teuthology.orchestra.run.smithi081.stdout:201863462961 2022-09-10T04:27:24.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.6 2022-09-10T04:27:24.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:24 smithi081 ceph-mon[90133]: pgmap v329: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:24.488 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 -- 172.21.15.81:0/3767969232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 msgr2=0x7f495410cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.489 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 --2- 172.21.15.81:0/3767969232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 0x7f495410cb90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f494c009b00 tx=0x7f494c009e10 comp rx=0 tx=0).stop 2022-09-10T04:27:24.489 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 -- 172.21.15.81:0/3767969232 shutdown_connections 2022-09-10T04:27:24.489 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 --2- 172.21.15.81:0/3767969232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 0x7f495410cb90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.490 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 --2- 172.21.15.81:0/3767969232 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071000 0x7f49540713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.490 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 --2- 172.21.15.81:0/3767969232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4954071910 0x7f495410a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.491 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 -- 172.21.15.81:0/3767969232 >> 172.21.15.81:0/3767969232 conn(0x7f49540f9a10 msgr2=0x7f49540fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.486+0000 7f495bfa2700 1 -- 172.21.15.81:0/3767969232 shutdown_connections 2022-09-10T04:27:24.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.487+0000 7f495bfa2700 1 -- 172.21.15.81:0/3767969232 wait complete. 2022-09-10T04:27:24.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.488+0000 7f495bfa2700 1 Processor -- start 2022-09-10T04:27:24.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 -- start start 2022-09-10T04:27:24.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4954071000 0x7f4954195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071910 0x7f4954196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.497 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 0x7f495419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495419af50 con 0x7f4954071000 2022-09-10T04:27:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495419b0c0 con 0x7f4954071910 2022-09-10T04:27:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.489+0000 7f495bfa2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495419b280 con 0x7f495410a7b0 2022-09-10T04:27:24.498 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071910 0x7f4954196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071910 0x7f4954196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45714/0 (socket says 172.21.15.81:45714) 2022-09-10T04:27:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495a53f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 0x7f495419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 -- 172.21.15.81:0/498757533 learned_addr learned my addr 172.21.15.81:0/498757533 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:24.499 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f4959d3e700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4954071000 0x7f4954195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 -- 172.21.15.81:0/498757533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 msgr2=0x7f495419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 0x7f495419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.500 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 -- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4954071000 msgr2=0x7f4954195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4954071000 0x7f4954195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495953d700 1 -- 172.21.15.81:0/498757533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944009470 con 0x7f4954071910 2022-09-10T04:27:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.490+0000 7f495a53f700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495410a7b0 0x7f495419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:24.501 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.491+0000 7f495953d700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071910 0x7f4954196460 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f494400e9c0 tx=0x7f494400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:24.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.491+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944004220 con 0x7f4954071910 2022-09-10T04:27:24.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.491+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c009730 con 0x7f4954071910 2022-09-10T04:27:24.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.491+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f494400eeb0 con 0x7f4954071910 2022-09-10T04:27:24.502 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.491+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944003d10 con 0x7f4954071910 2022-09-10T04:27:24.503 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.491+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495419b700 con 0x7f4954071910 2022-09-10T04:27:24.503 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.492+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f49540fb9a0 con 0x7f4954071910 2022-09-10T04:27:24.504 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.495+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f494400c040 con 0x7f4954071910 2022-09-10T04:27:24.505 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.496+0000 7f494affd700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4940099040 0x7f494009b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.505 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.497+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4944014070 con 0x7f4954071910 2022-09-10T04:27:24.505 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.497+0000 7f494affd700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] conn(0x7f49400a4050 0x7f49400a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:24.505 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.497+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 --> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f49400a6bb0 con 0x7f49400a4050 2022-09-10T04:27:24.506 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.497+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49440c3430 con 0x7f4954071910 2022-09-10T04:27:24.507 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.498+0000 7f495a53f700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] conn(0x7f49400a4050 0x7f49400a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.507 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.499+0000 7f4959d3e700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4940099040 0x7f494009b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:24.507 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.499+0000 7f495a53f700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] conn(0x7f49400a4050 0x7f49400a64e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:24.507 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.500+0000 7f4959d3e700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4940099040 0x7f494009b4f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4950005fd0 tx=0x7f4950005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:24.507 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.500+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== osd.9 v2:172.21.15.81:6826/3176150026 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f49400a6bb0 con 0x7f49400a4050 2022-09-10T04:27:24.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.522+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 --> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4954064ea0 con 0x7f49400a4050 2022-09-10T04:27:24.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.522+0000 7f494affd700 1 -- 172.21.15.81:0/498757533 <== osd.9 v2:172.21.15.81:6826/3176150026 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4954064ea0 con 0x7f49400a4050 2022-09-10T04:27:24.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] conn(0x7f49400a4050 msgr2=0x7f49400a64e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] conn(0x7f49400a4050 0x7f49400a64e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4940099040 msgr2=0x7f494009b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4940099040 0x7f494009b4f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4950005fd0 tx=0x7f4950005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071910 msgr2=0x7f4954196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:24.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 --2- 172.21.15.81:0/498757533 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4954071910 0x7f4954196460 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f494400e9c0 tx=0x7f494400ecd0 comp rx=0 tx=0).stop 2022-09-10T04:27:24.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495a53f700 1 -- 172.21.15.81:0/498757533 reap_dead start 2022-09-10T04:27:24.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 shutdown_connections 2022-09-10T04:27:24.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 >> 172.21.15.81:0/498757533 conn(0x7f49540f9a10 msgr2=0x7f495410c980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:24.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 shutdown_connections 2022-09-10T04:27:24.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.523+0000 7f495bfa2700 1 -- 172.21.15.81:0/498757533 wait complete. 2022-09-10T04:27:24.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:24 smithi104 ceph-mon[97044]: pgmap v329: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:24.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:24 smithi110 ceph-mon[97633]: pgmap v329: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:25.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 -- 172.21.15.81:0/3579683765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 msgr2=0x7f76b40666c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:25.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 --2- 172.21.15.81:0/3579683765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 0x7f76b40666c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009810 tx=0x7f76a8009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:25.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 -- 172.21.15.81:0/3579683765 shutdown_connections 2022-09-10T04:27:25.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 --2- 172.21.15.81:0/3579683765 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 0x7f76b410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:25.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 --2- 172.21.15.81:0/3579683765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4103ea0 0x7f76b4104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:25.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 --2- 172.21.15.81:0/3579683765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 0x7f76b40666c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:25.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 -- 172.21.15.81:0/3579683765 >> 172.21.15.81:0/3579683765 conn(0x7f76b4073cf0 msgr2=0x7f76b40740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:25.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 -- 172.21.15.81:0/3579683765 shutdown_connections 2022-09-10T04:27:25.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.995+0000 7f76b984f700 1 -- 172.21.15.81:0/3579683765 wait complete. 2022-09-10T04:27:25.118 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.996+0000 7f76b984f700 1 Processor -- start 2022-09-10T04:27:25.118 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.996+0000 7f76b984f700 1 -- start start 2022-09-10T04:27:25.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b984f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 0x7f76b4195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:25.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b984f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4103ea0 0x7f76b4196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:25.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b984f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 0x7f76b419a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:25.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b984f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b419af00 con 0x7f76b4104840 2022-09-10T04:27:25.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b984f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b419b070 con 0x7f76b4066da0 2022-09-10T04:27:25.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b984f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b419b230 con 0x7f76b4103ea0 2022-09-10T04:27:25.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 0x7f76b4195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:25.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b37fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 0x7f76b419a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:25.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 0x7f76b4195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45736/0 (socket says 172.21.15.81:45736) 2022-09-10T04:27:25.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 -- 172.21.15.81:0/3359334513 learned_addr learned my addr 172.21.15.81:0/3359334513 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:25.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b27fc700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4103ea0 0x7f76b4196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:25.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 -- 172.21.15.81:0/3359334513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4103ea0 msgr2=0x7f76b4196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b4103ea0 0x7f76b4196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:25.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 -- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 msgr2=0x7f76b419a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:25.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76b4104840 0x7f76b419a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:25.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.997+0000 7f76b2ffd700 1 -- 172.21.15.81:0/3359334513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8009470 con 0x7f76b4066da0 2022-09-10T04:27:25.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.998+0000 7f76b2ffd700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 0x7f76b4195ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f76a400e740 tx=0x7f76a400ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:25.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.998+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4005220 con 0x7f76b4066da0 2022-09-10T04:27:25.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.998+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b419b510 con 0x7f76b4066da0 2022-09-10T04:27:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.998+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f76a4005380 con 0x7f76b4066da0 2022-09-10T04:27:25.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.998+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a4004a30 con 0x7f76b4066da0 2022-09-10T04:27:25.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.998+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b41a2d00 con 0x7f76b4066da0 2022-09-10T04:27:25.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:24.999+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76b4109b00 con 0x7f76b4066da0 2022-09-10T04:27:25.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.001+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f76a401e020 con 0x7f76b4066da0 2022-09-10T04:27:25.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b884d700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f769c099090 0x7f769c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:25.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f76a4014070 con 0x7f76b4066da0 2022-09-10T04:27:25.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b27fc700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f769c099090 0x7f769c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:25.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b884d700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] conn(0x7f769c0a40a0 0x7f769c0a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 --> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f769c0a6c00 con 0x7f769c0a40a0 2022-09-10T04:27:25.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76a40c2f00 con 0x7f76b4066da0 2022-09-10T04:27:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.003+0000 7f76b37fe700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] conn(0x7f769c0a40a0 0x7f769c0a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.004+0000 7f76b27fc700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f769c099090 0x7f769c09b540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009ef0 tx=0x7f76a8005910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:25.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.004+0000 7f76b37fe700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] conn(0x7f769c0a40a0 0x7f769c0a6530 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:25.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.005+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== osd.4 v2:172.21.15.104:6808/2302505309 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f769c0a6c00 con 0x7f769c0a40a0 2022-09-10T04:27:25.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.021+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 --> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76b4064ea0 con 0x7f769c0a40a0 2022-09-10T04:27:25.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.022+0000 7f76b884d700 1 -- 172.21.15.81:0/3359334513 <== osd.4 v2:172.21.15.104:6808/2302505309 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f76b4064ea0 con 0x7f769c0a40a0 2022-09-10T04:27:25.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] conn(0x7f769c0a40a0 msgr2=0x7f769c0a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:25.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] conn(0x7f769c0a40a0 0x7f769c0a6530 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:25.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f769c099090 msgr2=0x7f769c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:25.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f769c099090 0x7f769c09b540 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f76a8009ef0 tx=0x7f76a8005910 comp rx=0 tx=0).stop 2022-09-10T04:27:25.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 msgr2=0x7f76b4195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:25.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 --2- 172.21.15.81:0/3359334513 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f76b4066da0 0x7f76b4195ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f76a400e740 tx=0x7f76a400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:25.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b37fe700 1 -- 172.21.15.81:0/3359334513 reap_dead start 2022-09-10T04:27:25.139 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 shutdown_connections 2022-09-10T04:27:25.139 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 >> 172.21.15.81:0/3359334513 conn(0x7f76b4073cf0 msgr2=0x7f76b41090b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:25.140 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 shutdown_connections 2022-09-10T04:27:25.140 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.023+0000 7f76b984f700 1 -- 172.21.15.81:0/3359334513 wait complete. 2022-09-10T04:27:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:347892350998 2022-09-10T04:27:25.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.11 2022-09-10T04:27:25.816 INFO:teuthology.orchestra.run.smithi081.stdout:313532612635 2022-09-10T04:27:25.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.10 2022-09-10T04:27:26.114 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 -- 172.21.15.81:0/851643422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4102fa0 msgr2=0x7f63d410ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 --2- 172.21.15.81:0/851643422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4102fa0 0x7f63d410ab40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009bb0 tx=0x7f63cc009ec0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.115 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 -- 172.21.15.81:0/851643422 shutdown_connections 2022-09-10T04:27:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 --2- 172.21.15.81:0/851643422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4102fa0 0x7f63d410ab40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 --2- 172.21.15.81:0/851643422 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d40fdea0 0x7f63d40fe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.116 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 --2- 172.21.15.81:0/851643422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d40fe840 0x7f63d4102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 -- 172.21.15.81:0/851643422 >> 172.21.15.81:0/851643422 conn(0x7f63d40f9800 msgr2=0x7f63d40fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 -- 172.21.15.81:0/851643422 shutdown_connections 2022-09-10T04:27:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.523+0000 7f63d89d7700 1 -- 172.21.15.81:0/851643422 wait complete. 2022-09-10T04:27:26.117 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.524+0000 7f63d89d7700 1 Processor -- start 2022-09-10T04:27:26.118 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 -- start start 2022-09-10T04:27:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d40fdea0 0x7f63d4195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d40fe840 0x7f63d4196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d4102fa0 0x7f63d419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.119 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d419ad50 con 0x7f63d40fe840 2022-09-10T04:27:26.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d419aec0 con 0x7f63d4102fa0 2022-09-10T04:27:26.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d89d7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d419b080 con 0x7f63d40fdea0 2022-09-10T04:27:26.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d2d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d40fdea0 0x7f63d4195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d2d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d40fdea0 0x7f63d4195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59834/0 (socket says 172.21.15.81:59834) 2022-09-10T04:27:26.120 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.525+0000 7f63d2d9d700 1 -- 172.21.15.81:0/2166327145 learned_addr learned my addr 172.21.15.81:0/2166327145 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:26.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d259c700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d40fe840 0x7f63d4196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d359e700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d4102fa0 0x7f63d419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d2d9d700 1 -- 172.21.15.81:0/2166327145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d4102fa0 msgr2=0x7f63d419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d2d9d700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63d4102fa0 0x7f63d419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d2d9d700 1 -- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d40fe840 msgr2=0x7f63d4196260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.122 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d2d9d700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d40fe840 0x7f63d4196260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d2d9d700 1 -- 172.21.15.81:0/2166327145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0097e0 con 0x7f63d40fdea0 2022-09-10T04:27:26.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.526+0000 7f63d2d9d700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d40fdea0 0x7f63d4195d20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f63c400e740 tx=0x7f63c400ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:26.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.527+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4005220 con 0x7f63d40fdea0 2022-09-10T04:27:26.123 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.527+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f63c4005380 con 0x7f63d40fdea0 2022-09-10T04:27:26.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.527+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4004c10 con 0x7f63d40fdea0 2022-09-10T04:27:26.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.527+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d419b360 con 0x7f63d40fdea0 2022-09-10T04:27:26.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.527+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d41a2b50 con 0x7f63d40fdea0 2022-09-10T04:27:26.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.528+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63d40721e0 con 0x7f63d40fdea0 2022-09-10T04:27:26.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.530+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f63c401e020 con 0x7f63d40fdea0 2022-09-10T04:27:26.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.531+0000 7f63bbfff700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f63bc099090 0x7f63bc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.531+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f63c4014070 con 0x7f63d40fdea0 2022-09-10T04:27:26.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.531+0000 7f63d259c700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f63bc099090 0x7f63bc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.532+0000 7f63bbfff700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] conn(0x7f63bc0a40a0 0x7f63bc0a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.532+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 --> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63bc0a6c00 con 0x7f63bc0a40a0 2022-09-10T04:27:26.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.532+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63c40c3450 con 0x7f63d40fdea0 2022-09-10T04:27:26.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.532+0000 7f63d259c700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f63bc099090 0x7f63bc09b540 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63d4197330 tx=0x7f63c8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:26.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.532+0000 7f63d359e700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] conn(0x7f63bc0a40a0 0x7f63bc0a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.533+0000 7f63d359e700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] conn(0x7f63bc0a40a0 0x7f63bc0a6530 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:26.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.533+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== osd.2 v2:172.21.15.110:6800/829575986 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f63bc0a6c00 con 0x7f63bc0a40a0 2022-09-10T04:27:26.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.552+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 --> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f63d404d480 con 0x7f63bc0a40a0 2022-09-10T04:27:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63bbfff700 1 -- 172.21.15.81:0/2166327145 <== osd.2 v2:172.21.15.110:6800/829575986 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f63d404d480 con 0x7f63bc0a40a0 2022-09-10T04:27:26.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] conn(0x7f63bc0a40a0 msgr2=0x7f63bc0a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] conn(0x7f63bc0a40a0 0x7f63bc0a6530 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f63bc099090 msgr2=0x7f63bc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f63bc099090 0x7f63bc09b540 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f63d4197330 tx=0x7f63c8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d40fdea0 msgr2=0x7f63d4195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 --2- 172.21.15.81:0/2166327145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d40fdea0 0x7f63d4195d20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f63c400e740 tx=0x7f63c400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:26.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d359e700 1 -- 172.21.15.81:0/2166327145 reap_dead start 2022-09-10T04:27:26.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 shutdown_connections 2022-09-10T04:27:26.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.553+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 >> 172.21.15.81:0/2166327145 conn(0x7f63d40f9800 msgr2=0x7f63d40fa490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:26.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.554+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 shutdown_connections 2022-09-10T04:27:26.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:25.554+0000 7f63d89d7700 1 -- 172.21.15.81:0/2166327145 wait complete. 2022-09-10T04:27:26.413 INFO:teuthology.orchestra.run.smithi081.stdout:167503724598 2022-09-10T04:27:26.414 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.5 2022-09-10T04:27:26.423 INFO:teuthology.orchestra.run.smithi081.stdout:292057776160 2022-09-10T04:27:26.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.9 2022-09-10T04:27:26.519 INFO:teuthology.orchestra.run.smithi081.stdout:146028888125 2022-09-10T04:27:26.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.4 2022-09-10T04:27:26.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 -- 172.21.15.81:0/4126812235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea41053e0 msgr2=0x7f1ea410cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 --2- 172.21.15.81:0/4126812235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea41053e0 0x7f1ea410cf80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009b30 tx=0x7f1e9c009e40 comp rx=0 tx=0).stop 2022-09-10T04:27:26.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 -- 172.21.15.81:0/4126812235 shutdown_connections 2022-09-10T04:27:26.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 --2- 172.21.15.81:0/4126812235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea41053e0 0x7f1ea410cf80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 --2- 172.21.15.81:0/4126812235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea41002e0 0x7f1ea41006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 --2- 172.21.15.81:0/4126812235 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea4100c80 0x7f1ea4104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 -- 172.21.15.81:0/4126812235 >> 172.21.15.81:0/4126812235 conn(0x7f1ea40fbc20 msgr2=0x7f1ea40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:26.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 -- 172.21.15.81:0/4126812235 shutdown_connections 2022-09-10T04:27:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.335+0000 7f1eab29b700 1 -- 172.21.15.81:0/4126812235 wait complete. 2022-09-10T04:27:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.336+0000 7f1eab29b700 1 Processor -- start 2022-09-10T04:27:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 -- start start 2022-09-10T04:27:26.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea41002e0 0x7f1ea4195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea4100c80 0x7f1ea4196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea41053e0 0x7f1ea419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea419ae70 con 0x7f1ea4100c80 2022-09-10T04:27:26.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea419afe0 con 0x7f1ea41053e0 2022-09-10T04:27:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.337+0000 7f1eab29b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea419b1a0 con 0x7f1ea41002e0 2022-09-10T04:27:26.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea4100c80 0x7f1ea4196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea9037700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea41002e0 0x7f1ea4195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea4100c80 0x7f1ea4196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60338/0 (socket says 172.21.15.81:60338) 2022-09-10T04:27:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 -- 172.21.15.81:0/1649656399 learned_addr learned my addr 172.21.15.81:0/1649656399 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:26.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea9838700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea41053e0 0x7f1ea419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 -- 172.21.15.81:0/1649656399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea41002e0 msgr2=0x7f1ea4195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea41002e0 0x7f1ea4195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 -- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea41053e0 msgr2=0x7f1ea419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ea41053e0 0x7f1ea419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 -- 172.21.15.81:0/1649656399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea0009470 con 0x7f1ea4100c80 2022-09-10T04:27:26.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.338+0000 7f1ea8836700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea4100c80 0x7f1ea4196380 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400e740 tx=0x7f1e9400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:26.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.339+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94005220 con 0x7f1ea4100c80 2022-09-10T04:27:26.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.339+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9c009730 con 0x7f1ea4100c80 2022-09-10T04:27:26.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.339+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1e94005380 con 0x7f1ea4100c80 2022-09-10T04:27:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.339+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e94004b70 con 0x7f1ea4100c80 2022-09-10T04:27:26.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.339+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea41a2c70 con 0x7f1ea4100c80 2022-09-10T04:27:26.619 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.340+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ea40fd140 con 0x7f1ea4100c80 2022-09-10T04:27:26.619 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.343+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1e9401e020 con 0x7f1ea4100c80 2022-09-10T04:27:26.620 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.345+0000 7f1e9a7fc700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e90099090 0x7f1e9009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.620 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.345+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f1e94014070 con 0x7f1ea4100c80 2022-09-10T04:27:26.621 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.346+0000 7f1e9a7fc700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] conn(0x7f1e900a40a0 0x7f1e900a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:26.621 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.346+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 --> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e900a6c00 con 0x7f1e900a40a0 2022-09-10T04:27:26.621 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.346+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e940c4ba0 con 0x7f1ea4100c80 2022-09-10T04:27:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.347+0000 7f1ea9838700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] conn(0x7f1e900a40a0 0x7f1e900a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.622 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.347+0000 7f1ea9838700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] conn(0x7f1e900a40a0 0x7f1e900a6530 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:26.623 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.348+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== osd.1 v2:172.21.15.104:6800/757653704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7f1e900a6c00 con 0x7f1e900a40a0 2022-09-10T04:27:26.623 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.352+0000 7f1ea9037700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e90099090 0x7f1e9009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.353+0000 7f1ea9037700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e90099090 0x7f1e9009b540 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009440 tx=0x7f1ea0005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.377+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 --> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ea4064ea0 con 0x7f1e900a40a0 2022-09-10T04:27:26.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.378+0000 7f1e9a7fc700 1 -- 172.21.15.81:0/1649656399 <== osd.1 v2:172.21.15.104:6800/757653704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1ea4064ea0 con 0x7f1e900a40a0 2022-09-10T04:27:26.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.379+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] conn(0x7f1e900a40a0 msgr2=0x7f1e900a6530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.379+0000 7f1eab29b700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] conn(0x7f1e900a40a0 0x7f1e900a6530 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:26.627 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.379+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e90099090 msgr2=0x7f1e9009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.627 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.379+0000 7f1eab29b700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e90099090 0x7f1e9009b540 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009440 tx=0x7f1ea0005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:26.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.379+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea4100c80 msgr2=0x7f1ea4196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:26.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.379+0000 7f1eab29b700 1 --2- 172.21.15.81:0/1649656399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ea4100c80 0x7f1ea4196380 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400e740 tx=0x7f1e9400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:26.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.380+0000 7f1ea9838700 1 -- 172.21.15.81:0/1649656399 reap_dead start 2022-09-10T04:27:26.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.380+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 shutdown_connections 2022-09-10T04:27:26.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.381+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 >> 172.21.15.81:0/1649656399 conn(0x7f1ea40fbc20 msgr2=0x7f1ea40fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:26.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.381+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 shutdown_connections 2022-09-10T04:27:26.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:26.381+0000 7f1eab29b700 1 -- 172.21.15.81:0/1649656399 wait complete. 2022-09-10T04:27:26.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:26 smithi081 ceph-mon[90133]: pgmap v330: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:26 smithi110 ceph-mon[97633]: pgmap v330: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:26 smithi104 ceph-mon[97044]: pgmap v330: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2022-09-10T04:27:27.342 INFO:teuthology.orchestra.run.smithi081.stdout:55834574925 2022-09-10T04:27:27.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.1 2022-09-10T04:27:27.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 -- 172.21.15.81:0/2495559484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1480fca80 msgr2=0x7fe1480fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:27.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 --2- 172.21.15.81:0/2495559484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1480fca80 0x7fe1480fce50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009750 tx=0x7fe13c009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:27.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 -- 172.21.15.81:0/2495559484 shutdown_connections 2022-09-10T04:27:27.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 --2- 172.21.15.81:0/2495559484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe14810a630 0x7fe14810ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 --2- 172.21.15.81:0/2495559484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1480fca80 0x7fe1480fce50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 --2- 172.21.15.81:0/2495559484 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 0x7fe14810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.277+0000 7fe14dd72700 1 -- 172.21.15.81:0/2495559484 >> 172.21.15.81:0/2495559484 conn(0x7fe1480f8710 msgr2=0x7fe1480fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:27.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.278+0000 7fe14dd72700 1 -- 172.21.15.81:0/2495559484 shutdown_connections 2022-09-10T04:27:27.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.278+0000 7fe14dd72700 1 -- 172.21.15.81:0/2495559484 wait complete. 2022-09-10T04:27:27.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.279+0000 7fe14dd72700 1 Processor -- start 2022-09-10T04:27:27.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.279+0000 7fe14dd72700 1 -- start start 2022-09-10T04:27:27.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14dd72700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1480fca80 0x7fe1480fed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:27.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14dd72700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 0x7fe1480ff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:27.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14dd72700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe14810a630 0x7fe1480ff7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:27.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14dd72700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1481a6390 con 0x7fe14810a630 2022-09-10T04:27:27.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14dd72700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1481959e0 con 0x7fe1480fd390 2022-09-10T04:27:27.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14dd72700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe148195ba0 con 0x7fe1480fca80 2022-09-10T04:27:27.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe147fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 0x7fe1480ff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:27.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe147fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 0x7fe1480ff2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45800/0 (socket says 172.21.15.81:45800) 2022-09-10T04:27:27.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe147fff700 1 -- 172.21.15.81:0/925692026 learned_addr learned my addr 172.21.15.81:0/925692026 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:27.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.280+0000 7fe14cd70700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1480fca80 0x7fe1480fed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:27.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe14d571700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe14810a630 0x7fe1480ff7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:27.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe147fff700 1 -- 172.21.15.81:0/925692026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1480fca80 msgr2=0x7fe1480fed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:27.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe147fff700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1480fca80 0x7fe1480fed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe147fff700 1 -- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe14810a630 msgr2=0x7fe1480ff7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:27.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe147fff700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe14810a630 0x7fe1480ff7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe147fff700 1 -- 172.21.15.81:0/925692026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0093b0 con 0x7fe1480fd390 2022-09-10T04:27:27.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe14d571700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe14810a630 0x7fe1480ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:27:27.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe14cd70700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1480fca80 0x7fe1480fed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:27.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.281+0000 7fe147fff700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 0x7fe1480ff2a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe13400e740 tx=0x7fe13400ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:27.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.282+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134005220 con 0x7fe1480fd390 2022-09-10T04:27:27.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.282+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe148195e80 con 0x7fe1480fd390 2022-09-10T04:27:27.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.282+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fe134005380 con 0x7fe1480fd390 2022-09-10T04:27:27.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.282+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134004d10 con 0x7fe1480fd390 2022-09-10T04:27:27.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.282+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe148196390 con 0x7fe1480fd390 2022-09-10T04:27:27.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.283+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe14810ca70 con 0x7fe1480fd390 2022-09-10T04:27:27.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.285+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fe13401e020 con 0x7fe1480fd390 2022-09-10T04:27:27.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.286+0000 7fe145ffb700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe138099010 0x7fe13809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:27.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.287+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fe134014070 con 0x7fe1480fd390 2022-09-10T04:27:27.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.287+0000 7fe14cd70700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe138099010 0x7fe13809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:27.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.287+0000 7fe145ffb700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] conn(0x7fe1380a4020 0x7fe1380a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:27.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.287+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 --> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1380a6b80 con 0x7fe1380a4020 2022-09-10T04:27:27.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.287+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1340c2e20 con 0x7fe1480fd390 2022-09-10T04:27:27.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.287+0000 7fe14d571700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] conn(0x7fe1380a4020 0x7fe1380a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:27.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.288+0000 7fe14cd70700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe138099010 0x7fe13809b4c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009e30 tx=0x7fe13c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:27.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.288+0000 7fe14d571700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] conn(0x7fe1380a4020 0x7fe1380a64b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:27.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.289+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== osd.0 v2:172.21.15.81:6802/860456526 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24493 (crc 0 0 0) 0x7fe1380a6b80 con 0x7fe1380a4020 2022-09-10T04:27:27.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.313+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 --> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe14804d480 con 0x7fe1380a4020 2022-09-10T04:27:27.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe145ffb700 1 -- 172.21.15.81:0/925692026 <== osd.0 v2:172.21.15.81:6802/860456526 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe14804d480 con 0x7fe1380a4020 2022-09-10T04:27:27.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] conn(0x7fe1380a4020 msgr2=0x7fe1380a64b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:27.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] conn(0x7fe1380a4020 0x7fe1380a64b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe138099010 msgr2=0x7fe13809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:27.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fe138099010 0x7fe13809b4c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009e30 tx=0x7fe13c005fd0 comp rx=0 tx=0).stop 2022-09-10T04:27:27.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 msgr2=0x7fe1480ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:27.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 --2- 172.21.15.81:0/925692026 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1480fd390 0x7fe1480ff2a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe13400e740 tx=0x7fe13400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:27.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14d571700 1 -- 172.21.15.81:0/925692026 reap_dead start 2022-09-10T04:27:27.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 shutdown_connections 2022-09-10T04:27:27.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.314+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 >> 172.21.15.81:0/925692026 conn(0x7fe1480f8710 msgr2=0x7fe14810b960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:27.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.315+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 shutdown_connections 2022-09-10T04:27:27.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:27.315+0000 7fe14dd72700 1 -- 172.21.15.81:0/925692026 wait complete. 2022-09-10T04:27:27.387 INFO:teuthology.orchestra.run.smithi081.stdout:77309411400 2022-09-10T04:27:27.388 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.2 2022-09-10T04:27:27.920 INFO:teuthology.orchestra.run.smithi081.stdout:34359738451 2022-09-10T04:27:27.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph osd last-stat-seq osd.0 2022-09-10T04:27:27.949 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:27.973 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:28.068 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:28.141 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:28.343 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:28.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:28 smithi081 ceph-mon[90133]: pgmap v331: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T04:27:28.657 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:28.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:28 smithi104 ceph-mon[97044]: pgmap v331: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T04:27:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:28 smithi110 ceph-mon[97633]: pgmap v331: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T04:27:29.647 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:29.800 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:30.547 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:30 smithi081 ceph-mon[90133]: pgmap v332: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:30 smithi104 ceph-mon[97044]: pgmap v332: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:30 smithi110 ceph-mon[97633]: pgmap v332: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:31.671 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:32.486 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:32 smithi081 ceph-mon[90133]: pgmap v333: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:32.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:32 smithi104 ceph-mon[97044]: pgmap v333: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:32.886 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 -- 172.21.15.81:0/3444954035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 msgr2=0x7f33a810a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:32.886 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 --2- 172.21.15.81:0/3444954035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 0x7f33a810a660 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009bb0 tx=0x7f33a0009ec0 comp rx=0 tx=0).stop 2022-09-10T04:27:32.887 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 -- 172.21.15.81:0/3444954035 shutdown_connections 2022-09-10T04:27:32.887 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 --2- 172.21.15.81:0/3444954035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 0x7f33a810a660 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:32.888 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 --2- 172.21.15.81:0/3444954035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a80ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:32.888 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 --2- 172.21.15.81:0/3444954035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33a80ff5e0 0x7f33a8107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:32.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.882+0000 7f33aec22700 1 -- 172.21.15.81:0/3444954035 >> 172.21.15.81:0/3444954035 conn(0x7f33a80f8710 msgr2=0x7f33a80fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:32.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.883+0000 7f33aec22700 1 -- 172.21.15.81:0/3444954035 shutdown_connections 2022-09-10T04:27:32.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.883+0000 7f33aec22700 1 -- 172.21.15.81:0/3444954035 wait complete. 2022-09-10T04:27:32.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.884+0000 7f33aec22700 1 Processor -- start 2022-09-10T04:27:32.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.884+0000 7f33aec22700 1 -- start start 2022-09-10T04:27:32.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33aec22700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:32.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33aec22700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33a80ff5e0 0x7f33a819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:32.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33aec22700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 0x7f33a8195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:32.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33aec22700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a81962f0 con 0x7f33a80fecd0 2022-09-10T04:27:32.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33aec22700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8196460 con 0x7f33a8108280 2022-09-10T04:27:32.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33aec22700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a8196620 con 0x7f33a80ff5e0 2022-09-10T04:27:32.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33adc20700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:32.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33adc20700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a819bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42070/0 (socket says 172.21.15.81:42070) 2022-09-10T04:27:32.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33adc20700 1 -- 172.21.15.81:0/636597502 learned_addr learned my addr 172.21.15.81:0/636597502 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:32.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33ae421700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 0x7f33a8195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:32.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.885+0000 7f33ad41f700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33a80ff5e0 0x7f33a819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:32.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33adc20700 1 -- 172.21.15.81:0/636597502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33a80ff5e0 msgr2=0x7f33a819c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:32.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33adc20700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33a80ff5e0 0x7f33a819c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:32.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33adc20700 1 -- 172.21.15.81:0/636597502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 msgr2=0x7f33a8195b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:32.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33adc20700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 0x7f33a8195b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:32.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33adc20700 1 -- 172.21.15.81:0/636597502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a00097e0 con 0x7f33a80fecd0 2022-09-10T04:27:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33adc20700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a819bae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e740 tx=0x7f33a400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:32.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4005220 con 0x7f33a80fecd0 2022-09-10T04:27:32.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f33a4005380 con 0x7f33a80fecd0 2022-09-10T04:27:32.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a8196900 con 0x7f33a80fecd0 2022-09-10T04:27:32.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.886+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a4004b70 con 0x7f33a80fecd0 2022-09-10T04:27:32.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.887+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a81a2b80 con 0x7f33a80fecd0 2022-09-10T04:27:32.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.888+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a804d480 con 0x7f33a80fecd0 2022-09-10T04:27:32.918 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.890+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f33a401e020 con 0x7f33a80fecd0 2022-09-10T04:27:32.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.891+0000 7f339effd700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3394099090 0x7f339409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:32.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.892+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f33a4014070 con 0x7f33a80fecd0 2022-09-10T04:27:32.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.892+0000 7f33ad41f700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3394099090 0x7f339409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:32.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.892+0000 7f33ad41f700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3394099090 0x7f339409b540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f33a81097e0 tx=0x7f3398005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:32.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:32.895+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f33a408eb80 con 0x7f33a80fecd0 2022-09-10T04:27:32.922 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:32 smithi110 ceph-mon[97633]: pgmap v333: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:33.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.118+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f33a81a3250 con 0x7f33a80fecd0 2022-09-10T04:27:33.121 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.119+0000 7f339effd700 1 -- 172.21.15.81:0/636597502 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f33a40921a0 con 0x7f33a80fecd0 2022-09-10T04:27:33.122 INFO:teuthology.orchestra.run.smithi081.stdout:313532612637 2022-09-10T04:27:33.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.121+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3394099090 msgr2=0x7f339409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:33.124 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.121+0000 7f33aec22700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3394099090 0x7f339409b540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f33a81097e0 tx=0x7f3398005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:33.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.121+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 msgr2=0x7f33a819bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:33.125 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.121+0000 7f33aec22700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a819bae0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e740 tx=0x7f33a400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:33.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 shutdown_connections 2022-09-10T04:27:33.126 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f33a8108280 0x7f33a8195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:33.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33a80fecd0 0x7f33a819bae0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:33.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3394099090 0x7f339409b540 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:33.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 --2- 172.21.15.81:0/636597502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33a80ff5e0 0x7f33a819c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:33.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 >> 172.21.15.81:0/636597502 conn(0x7f33a80f8710 msgr2=0x7f33a810a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:33.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 shutdown_connections 2022-09-10T04:27:33.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.122+0000 7f33aec22700 1 -- 172.21.15.81:0/636597502 wait complete. 2022-09-10T04:27:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:33 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/636597502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-09-10T04:27:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:33 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/636597502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-09-10T04:27:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:33 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/636597502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2022-09-10T04:27:33.919 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612635 got 313532612637 for osd.10 2022-09-10T04:27:33.919 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:34.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.760+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/5386036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28100110 msgr2=0x7f4d281004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.760+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/5386036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28100110 0x7f4d281004e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009840 tx=0x7f4d24009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:34.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/5386036 shutdown_connections 2022-09-10T04:27:34.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/5386036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28105210 0x7f4d2810cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/5386036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28100110 0x7f4d281004e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/5386036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d28100ab0 0x7f4d28104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/5386036 >> 172.21.15.81:0/5386036 conn(0x7f4d280fba90 msgr2=0x7f4d280fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:34.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/5386036 shutdown_connections 2022-09-10T04:27:34.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.761+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/5386036 wait complete. 2022-09-10T04:27:34.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.762+0000 7f4d2e5a7700 1 Processor -- start 2022-09-10T04:27:34.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 -- start start 2022-09-10T04:27:34.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 0x7f4d28195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d28100ab0 0x7f4d28196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28105210 0x7f4d281995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28199c20 con 0x7f4d28105210 2022-09-10T04:27:34.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28199d90 con 0x7f4d28100110 2022-09-10T04:27:34.282 INFO:teuthology.orchestra.run.smithi081.stdout:201863462962 2022-09-10T04:27:34.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.763+0000 7f4d2e5a7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d28199f50 con 0x7f4d28100ab0 2022-09-10T04:27:34.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 0x7f4d28195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2dda6700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28105210 0x7f4d281995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 0x7f4d28195cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49352/0 (socket says 172.21.15.81:49352) 2022-09-10T04:27:34.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 -- 172.21.15.81:0/1479846955 learned_addr learned my addr 172.21.15.81:0/1479846955 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:34.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2cda4700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d28100ab0 0x7f4d28196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 -- 172.21.15.81:0/1479846955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d28100ab0 msgr2=0x7f4d28196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d28100ab0 0x7f4d28196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 -- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28105210 msgr2=0x7f4d281995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28105210 0x7f4d281995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.764+0000 7f4d2d5a5700 1 -- 172.21.15.81:0/1479846955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d180093b0 con 0x7f4d28100110 2022-09-10T04:27:34.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.765+0000 7f4d2d5a5700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 0x7f4d28195cd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400c010 tx=0x7f4d2400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:34.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.765+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2401d070 con 0x7f4d28100110 2022-09-10T04:27:34.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.765+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d24009470 con 0x7f4d28100110 2022-09-10T04:27:34.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.765+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d281a2950 con 0x7f4d28100110 2022-09-10T04:27:34.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.765+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4d2400b480 con 0x7f4d28100110 2022-09-10T04:27:34.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.765+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d24004760 con 0x7f4d28100110 2022-09-10T04:27:34.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.766+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0c0051a0 con 0x7f4d28100110 2022-09-10T04:27:34.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.768+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f4d24028030 con 0x7f4d28100110 2022-09-10T04:27:34.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.769+0000 7f4d1e7fc700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4d14099090 0x7f4d1409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.769+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f4d240c3100 con 0x7f4d28100110 2022-09-10T04:27:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.770+0000 7f4d2cda4700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4d14099090 0x7f4d1409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.770+0000 7f4d2cda4700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4d14099090 0x7f4d1409b540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d281972a0 tx=0x7f4d18005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:33.772+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f4d2408f480 con 0x7f4d28100110 2022-09-10T04:27:34.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.071+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f4d0c005010 con 0x7f4d28100110 2022-09-10T04:27:34.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.071+0000 7f4d1e7fc700 1 -- 172.21.15.81:0/1479846955 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4d24026020 con 0x7f4d28100110 2022-09-10T04:27:34.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4d14099090 msgr2=0x7f4d1409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4d14099090 0x7f4d1409b540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4d281972a0 tx=0x7f4d18005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 msgr2=0x7f4d28195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 0x7f4d28195cd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d2400c010 tx=0x7f4d2400f690 comp rx=0 tx=0).stop 2022-09-10T04:27:34.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 shutdown_connections 2022-09-10T04:27:34.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d28105210 0x7f4d281995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4d28100110 0x7f4d28195cd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4d14099090 0x7f4d1409b540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 --2- 172.21.15.81:0/1479846955 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d28100ab0 0x7f4d28196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 >> 172.21.15.81:0/1479846955 conn(0x7f4d280fba90 msgr2=0x7f4d280fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 shutdown_connections 2022-09-10T04:27:34.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7f4d2e5a7700 1 -- 172.21.15.81:0/1479846955 wait complete. 2022-09-10T04:27:34.696 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:34 smithi104 ceph-mon[97044]: pgmap v334: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:34.696 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:34 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1479846955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-09-10T04:27:34.697 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:34 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1497119996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-09-10T04:27:34.698 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:34 smithi110 ceph-mon[97633]: pgmap v334: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:34.698 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:34 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1479846955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-09-10T04:27:34.699 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:34 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1497119996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-09-10T04:27:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 -- 172.21.15.81:0/3326110053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c073b20 msgr2=0x7fcf1c10e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 --2- 172.21.15.81:0/3326110053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c073b20 0x7fcf1c10e690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009780 tx=0x7fcf10009a90 comp rx=0 tx=0).stop 2022-09-10T04:27:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 -- 172.21.15.81:0/3326110053 shutdown_connections 2022-09-10T04:27:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 --2- 172.21.15.81:0/3326110053 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf1c10ebd0 0x7fcf1c110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 --2- 172.21.15.81:0/3326110053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073210 0x7fcf1c0735e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 --2- 172.21.15.81:0/3326110053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c073b20 0x7fcf1c10e690 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.074+0000 7fcf21505700 1 -- 172.21.15.81:0/3326110053 >> 172.21.15.81:0/3326110053 conn(0x7fcf1c0fba90 msgr2=0x7fcf1c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:34.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7fcf21505700 1 -- 172.21.15.81:0/3326110053 shutdown_connections 2022-09-10T04:27:34.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.075+0000 7fcf21505700 1 -- 172.21.15.81:0/3326110053 wait complete. 2022-09-10T04:27:34.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.076+0000 7fcf21505700 1 Processor -- start 2022-09-10T04:27:34.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 -- start start 2022-09-10T04:27:34.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf1c073210 0x7fcf1c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 0x7fcf1c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c10ebd0 0x7fcf1c19eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c19f1c0 con 0x7fcf1c073b20 2022-09-10T04:27:34.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c19f330 con 0x7fcf1c073210 2022-09-10T04:27:34.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.077+0000 7fcf21505700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c19f4f0 con 0x7fcf1c10ebd0 2022-09-10T04:27:34.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.078+0000 7fcf1b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 0x7fcf1c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.078+0000 7fcf1b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 0x7fcf1c19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42116/0 (socket says 172.21.15.81:42116) 2022-09-10T04:27:34.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.078+0000 7fcf1b7fe700 1 -- 172.21.15.81:0/1497119996 learned_addr learned my addr 172.21.15.81:0/1497119996 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:34.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.078+0000 7fcf1bfff700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf1c073210 0x7fcf1c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.078+0000 7fcf20d04700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c10ebd0 0x7fcf1c19eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf1b7fe700 1 -- 172.21.15.81:0/1497119996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c10ebd0 msgr2=0x7fcf1c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf1b7fe700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c10ebd0 0x7fcf1c19eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf1b7fe700 1 -- 172.21.15.81:0/1497119996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf1c073210 msgr2=0x7fcf1c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf1b7fe700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf1c073210 0x7fcf1c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf1b7fe700 1 -- 172.21.15.81:0/1497119996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf100093b0 con 0x7fcf1c073b20 2022-09-10T04:27:34.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf1b7fe700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 0x7fcf1c19a6d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcf10018630 tx=0x7fcf10018660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:34.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1001d070 con 0x7fcf1c073b20 2022-09-10T04:27:34.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcf10018930 con 0x7fcf1c073b20 2022-09-10T04:27:34.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10004760 con 0x7fcf1c073b20 2022-09-10T04:27:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.079+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c19f770 con 0x7fcf1c073b20 2022-09-10T04:27:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.080+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c19f990 con 0x7fcf1c073b20 2022-09-10T04:27:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.081+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c04d480 con 0x7fcf1c073b20 2022-09-10T04:27:34.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.083+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fcf1000c040 con 0x7fcf1c073b20 2022-09-10T04:27:34.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.085+0000 7fcf197fa700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcf04099090 0x7fcf0409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:34.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.085+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fcf100c3b20 con 0x7fcf1c073b20 2022-09-10T04:27:34.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.085+0000 7fcf1bfff700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcf04099090 0x7fcf0409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:34.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.086+0000 7fcf1bfff700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcf04099090 0x7fcf0409b540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c005fd0 tx=0x7fcf0c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:34.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.091+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcf1008fea0 con 0x7fcf1c073b20 2022-09-10T04:27:34.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.341+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fcf1c1a74d0 con 0x7fcf1c073b20 2022-09-10T04:27:34.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.342+0000 7fcf197fa700 1 -- 172.21.15.81:0/1497119996 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcf10030020 con 0x7fcf1c073b20 2022-09-10T04:27:34.746 INFO:teuthology.orchestra.run.smithi081.stdout:223338299437 2022-09-10T04:27:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcf04099090 msgr2=0x7fcf0409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcf04099090 0x7fcf0409b540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c005fd0 tx=0x7fcf0c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 msgr2=0x7fcf1c19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:34.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 0x7fcf1c19a6d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcf10018630 tx=0x7fcf10018660 comp rx=0 tx=0).stop 2022-09-10T04:27:34.749 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 shutdown_connections 2022-09-10T04:27:34.749 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcf04099090 0x7fcf0409b540 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.750 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf1c10ebd0 0x7fcf1c19eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.750 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcf1c073210 0x7fcf1c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.750 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 --2- 172.21.15.81:0/1497119996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf1c073b20 0x7fcf1c19a6d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 >> 172.21.15.81:0/1497119996 conn(0x7fcf1c0fba90 msgr2=0x7fcf1c0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:34.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 shutdown_connections 2022-09-10T04:27:34.752 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.345+0000 7fcf21505700 1 -- 172.21.15.81:0/1497119996 wait complete. 2022-09-10T04:27:34.753 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:34 smithi081 ceph-mon[90133]: pgmap v334: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:34.753 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:34 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1479846955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2022-09-10T04:27:34.754 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:34 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1497119996' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2022-09-10T04:27:35.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 -- 172.21.15.81:0/2972263380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071080 msgr2=0x7ff734071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 --2- 172.21.15.81:0/2972263380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071080 0x7ff734071450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff724009810 tx=0x7ff724009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:35.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 -- 172.21.15.81:0/2972263380 shutdown_connections 2022-09-10T04:27:35.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 --2- 172.21.15.81:0/2972263380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff73410ef10 0x7ff7341112f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 --2- 172.21.15.81:0/2972263380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071080 0x7ff734071450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 --2- 172.21.15.81:0/2972263380 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734071990 0x7ff73410e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 -- 172.21.15.81:0/2972263380 >> 172.21.15.81:0/2972263380 conn(0x7ff7340f9a20 msgr2=0x7ff7340fbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:35.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 -- 172.21.15.81:0/2972263380 shutdown_connections 2022-09-10T04:27:35.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.820+0000 7ff738d87700 1 -- 172.21.15.81:0/2972263380 wait complete. 2022-09-10T04:27:35.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 Processor -- start 2022-09-10T04:27:35.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 -- start start 2022-09-10T04:27:35.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff734071080 0x7ff73419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 0x7ff73419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff73410ef10 0x7ff73419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73419f340 con 0x7ff73410ef10 2022-09-10T04:27:35.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73419f4b0 con 0x7ff734071990 2022-09-10T04:27:35.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff738d87700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73419f670 con 0x7ff734071080 2022-09-10T04:27:35.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.822+0000 7ff731d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 0x7ff73419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff73259c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff734071080 0x7ff73419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 0x7ff73419a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49374/0 (socket says 172.21.15.81:49374) 2022-09-10T04:27:35.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 -- 172.21.15.81:0/3493059362 learned_addr learned my addr 172.21.15.81:0/3493059362 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:35.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff732d9d700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff73410ef10 0x7ff73419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.252 INFO:teuthology.orchestra.run.smithi081.stdout:257698037800 2022-09-10T04:27:35.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 -- 172.21.15.81:0/3493059362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff734071080 msgr2=0x7ff73419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff734071080 0x7ff73419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 -- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff73410ef10 msgr2=0x7ff73419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff73410ef10 0x7ff73419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 -- 172.21.15.81:0/3493059362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72c009730 con 0x7ff734071990 2022-09-10T04:27:35.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff73259c700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff734071080 0x7ff73419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:35.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.823+0000 7ff731d9b700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 0x7ff73419a850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff72800e740 tx=0x7ff72800ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:35.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.824+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728005220 con 0x7ff734071990 2022-09-10T04:27:35.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.824+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff728005380 con 0x7ff734071990 2022-09-10T04:27:35.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.824+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728010ba0 con 0x7ff734071990 2022-09-10T04:27:35.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.824+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff724009470 con 0x7ff734071990 2022-09-10T04:27:35.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.824+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7341a6f60 con 0x7ff734071990 2022-09-10T04:27:35.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.825+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff734064ea0 con 0x7ff734071990 2022-09-10T04:27:35.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.827+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff72801e020 con 0x7ff734071990 2022-09-10T04:27:35.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.828+0000 7ff7237fe700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff71c099090 0x7ff71c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.828+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7ff728014070 con 0x7ff734071990 2022-09-10T04:27:35.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.829+0000 7ff73259c700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff71c099090 0x7ff71c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.829+0000 7ff73259c700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff71c099090 0x7ff71c09b540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff724009760 tx=0x7ff724009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:35.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:34.836+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff72808ecc0 con 0x7ff734071990 2022-09-10T04:27:35.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.069+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7ff734114330 con 0x7ff734071990 2022-09-10T04:27:35.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.070+0000 7ff7237fe700 1 -- 172.21.15.81:0/3493059362 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff7280922e0 con 0x7ff734071990 2022-09-10T04:27:35.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff71c099090 msgr2=0x7ff71c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff71c099090 0x7ff71c09b540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff724009760 tx=0x7ff724009eb0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 msgr2=0x7ff73419a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 0x7ff73419a850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff72800e740 tx=0x7ff72800ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:35.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 shutdown_connections 2022-09-10T04:27:35.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff73410ef10 0x7ff73419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff71c099090 0x7ff71c09b540 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff734071080 0x7ff73419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 --2- 172.21.15.81:0/3493059362 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734071990 0x7ff73419a850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 >> 172.21.15.81:0/3493059362 conn(0x7ff7340f9a20 msgr2=0x7ff7340fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:35.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 shutdown_connections 2022-09-10T04:27:35.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.073+0000 7ff738d87700 1 -- 172.21.15.81:0/3493059362 wait complete. 2022-09-10T04:27:35.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 -- 172.21.15.81:0/1917565602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c1002e0 msgr2=0x7f9c6c1006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 --2- 172.21.15.81:0/1917565602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c1006b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009840 tx=0x7f9c5c009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:35.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 -- 172.21.15.81:0/1917565602 shutdown_connections 2022-09-10T04:27:35.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 --2- 172.21.15.81:0/1917565602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 0x7f9c6c10cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 --2- 172.21.15.81:0/1917565602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c1006b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 --2- 172.21.15.81:0/1917565602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c100c80 0x7f9c6c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.736+0000 7f9c71835700 1 -- 172.21.15.81:0/1917565602 >> 172.21.15.81:0/1917565602 conn(0x7f9c6c0fbc20 msgr2=0x7f9c6c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:35.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.737+0000 7f9c71835700 1 -- 172.21.15.81:0/1917565602 shutdown_connections 2022-09-10T04:27:35.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.737+0000 7f9c71835700 1 -- 172.21.15.81:0/1917565602 wait complete. 2022-09-10T04:27:35.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.738+0000 7f9c71835700 1 Processor -- start 2022-09-10T04:27:35.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 -- start start 2022-09-10T04:27:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c100c80 0x7f9c6c1963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 0x7f9c6c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c19aec0 con 0x7f9c6c100c80 2022-09-10T04:27:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c19b030 con 0x7f9c6c1053e0 2022-09-10T04:27:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.739+0000 7f9c71835700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c19b1f0 con 0x7f9c6c1002e0 2022-09-10T04:27:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:49722/0 (socket says 172.21.15.81:49722) 2022-09-10T04:27:35.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 -- 172.21.15.81:0/4189414918 learned_addr learned my addr 172.21.15.81:0/4189414918 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6b7fe700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 0x7f9c6c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6a7fc700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c100c80 0x7f9c6c1963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 -- 172.21.15.81:0/4189414918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 msgr2=0x7f9c6c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 0x7f9c6c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 -- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c100c80 msgr2=0x7f9c6c1963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c100c80 0x7f9c6c1963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 -- 172.21.15.81:0/4189414918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c009470 con 0x7f9c6c1002e0 2022-09-10T04:27:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6b7fe700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 0x7f9c6c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:27:35.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.740+0000 7f9c6affd700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c195e90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00c010 tx=0x7f9c5c00f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:35.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.741+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c01d070 con 0x7f9c6c1002e0 2022-09-10T04:27:35.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.741+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9c5c00b480 con 0x7f9c6c1002e0 2022-09-10T04:27:35.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.741+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c19b470 con 0x7f9c6c1002e0 2022-09-10T04:27:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.741+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c004890 con 0x7f9c6c1002e0 2022-09-10T04:27:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.741+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c1a2cc0 con 0x7f9c6c1002e0 2022-09-10T04:27:35.749 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.742+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c064ea0 con 0x7f9c6c1002e0 2022-09-10T04:27:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.744+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f9c5c028030 con 0x7f9c6c1002e0 2022-09-10T04:27:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.745+0000 7f9c70833700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9c54099090 0x7f9c5409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.745+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f9c5c0c3380 con 0x7f9c6c1002e0 2022-09-10T04:27:35.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.745+0000 7f9c6a7fc700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9c54099090 0x7f9c5409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.752 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.746+0000 7f9c6a7fc700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9c54099090 0x7f9c5409b540 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c1974a0 tx=0x7f9c60005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:35.752 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.749+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f9c5c08f700 con 0x7f9c6c1002e0 2022-09-10T04:27:35.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 -- 172.21.15.81:0/1510122438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fe840 msgr2=0x7f4134102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 --2- 172.21.15.81:0/1510122438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fe840 0x7f4134102890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4128009810 tx=0x7f4128009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:35.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 -- 172.21.15.81:0/1510122438 shutdown_connections 2022-09-10T04:27:35.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 --2- 172.21.15.81:0/1510122438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4134102fa0 0x7f413410ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 --2- 172.21.15.81:0/1510122438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41340fdea0 0x7f41340fe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 --2- 172.21.15.81:0/1510122438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fe840 0x7f4134102890 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 -- 172.21.15.81:0/1510122438 >> 172.21.15.81:0/1510122438 conn(0x7f41340f9800 msgr2=0x7f41340fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:35.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 -- 172.21.15.81:0/1510122438 shutdown_connections 2022-09-10T04:27:35.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.779+0000 7f4138b96700 1 -- 172.21.15.81:0/1510122438 wait complete. 2022-09-10T04:27:35.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.780+0000 7f4138b96700 1 Processor -- start 2022-09-10T04:27:35.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 -- start start 2022-09-10T04:27:35.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fdea0 0x7f4134199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 0x7f4134199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4134102fa0 0x7f4134193990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4134194070 con 0x7f4134102fa0 2022-09-10T04:27:35.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41341941e0 con 0x7f41340fe840 2022-09-10T04:27:35.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4138b96700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41341943a0 con 0x7f41340fdea0 2022-09-10T04:27:35.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f41337fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fdea0 0x7f4134199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4132ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 0x7f4134199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4132ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 0x7f4134199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49420/0 (socket says 172.21.15.81:49420) 2022-09-10T04:27:35.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f41337fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fdea0 0x7f4134199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:49734/0 (socket says 172.21.15.81:49734) 2022-09-10T04:27:35.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4132ffd700 1 -- 172.21.15.81:0/1734583639 learned_addr learned my addr 172.21.15.81:0/1734583639 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:35.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.781+0000 7f4133fff700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4134102fa0 0x7f4134193990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.782+0000 7f4132ffd700 1 -- 172.21.15.81:0/1734583639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fdea0 msgr2=0x7f4134199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.782+0000 7f4132ffd700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fdea0 0x7f4134199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.782+0000 7f4132ffd700 1 -- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4134102fa0 msgr2=0x7f4134193990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:35.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.782+0000 7f4132ffd700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4134102fa0 0x7f4134193990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:35.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.782+0000 7f4132ffd700 1 -- 172.21.15.81:0/1734583639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4128009470 con 0x7f41340fe840 2022-09-10T04:27:35.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.782+0000 7f4132ffd700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 0x7f4134199e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f41280059c0 tx=0x7f412800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:35.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.783+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128025070 con 0x7f41340fe840 2022-09-10T04:27:35.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.783+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f412800f800 con 0x7f41340fe840 2022-09-10T04:27:35.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.783+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4134194620 con 0x7f41340fe840 2022-09-10T04:27:35.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.783+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128004790 con 0x7f41340fe840 2022-09-10T04:27:35.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.783+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41340710f0 con 0x7f41340fe840 2022-09-10T04:27:35.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.784+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413404d480 con 0x7f41340fe840 2022-09-10T04:27:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:27:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:27:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:35 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:27:35.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:35 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3493059362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-09-10T04:27:35.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.786+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f4128031030 con 0x7f41340fe840 2022-09-10T04:27:35.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.788+0000 7f4130ff9700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f411c099010 0x7f411c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:35.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.788+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f41280cc2d0 con 0x7f41340fe840 2022-09-10T04:27:35.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.789+0000 7f41337fe700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f411c099010 0x7f411c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:35.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.789+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f41280986b0 con 0x7f41340fe840 2022-09-10T04:27:35.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:35.791+0000 7f41337fe700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f411c099010 0x7f411c09b4c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f412400baf0 tx=0x7f412400b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:35.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:27:35.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:27:35.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:35 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:27:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:35 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3493059362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-09-10T04:27:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:27:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:27:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:35 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:27:35.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:35 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3493059362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2022-09-10T04:27:36.002 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.000+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f9c6c1a3270 con 0x7f9c6c1002e0 2022-09-10T04:27:36.003 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.001+0000 7f9c70833700 1 -- 172.21.15.81:0/4189414918 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9c5c026020 con 0x7f9c6c1002e0 2022-09-10T04:27:36.003 INFO:teuthology.orchestra.run.smithi081.stdout:292057776162 2022-09-10T04:27:36.006 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9c54099090 msgr2=0x7f9c5409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.006 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9c54099090 0x7f9c5409b540 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c1974a0 tx=0x7f9c60005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.007 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 msgr2=0x7f9c6c195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.007 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c195e90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00c010 tx=0x7f9c5c00f740 comp rx=0 tx=0).stop 2022-09-10T04:27:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 shutdown_connections 2022-09-10T04:27:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9c6c1053e0 0x7f9c6c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9c54099090 0x7f9c5409b540 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c1974a0 tx=0x7f9c60005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c6c1002e0 0x7f9c6c195e90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.004+0000 7f9c71835700 1 --2- 172.21.15.81:0/4189414918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c6c100c80 0x7f9c6c1963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.005+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 >> 172.21.15.81:0/4189414918 conn(0x7f9c6c0fbc20 msgr2=0x7f9c6c0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:36.010 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.005+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 shutdown_connections 2022-09-10T04:27:36.010 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.005+0000 7f9c71835700 1 -- 172.21.15.81:0/4189414918 wait complete. 2022-09-10T04:27:36.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.069+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f413410dac0 con 0x7f41340fe840 2022-09-10T04:27:36.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.070+0000 7f4130ff9700 1 -- 172.21.15.81:0/1734583639 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f412802f020 con 0x7f41340fe840 2022-09-10T04:27:36.073 INFO:teuthology.orchestra.run.smithi081.stdout:111669149763 2022-09-10T04:27:36.075 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f411c099010 msgr2=0x7f411c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.075 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f411c099010 0x7f411c09b4c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f412400baf0 tx=0x7f412400b2e0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 msgr2=0x7f4134199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 0x7f4134199e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f41280059c0 tx=0x7f412800f690 comp rx=0 tx=0).stop 2022-09-10T04:27:36.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 shutdown_connections 2022-09-10T04:27:36.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4134102fa0 0x7f4134193990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f411c099010 0x7f411c09b4c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41340fdea0 0x7f4134199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 --2- 172.21.15.81:0/1734583639 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41340fe840 0x7f4134199e60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 >> 172.21.15.81:0/1734583639 conn(0x7f41340f9800 msgr2=0x7f41340fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:36.078 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 shutdown_connections 2022-09-10T04:27:36.078 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.073+0000 7f4138b96700 1 -- 172.21.15.81:0/1734583639 wait complete. 2022-09-10T04:27:36.472 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037797 got 257698037800 for osd.8 2022-09-10T04:27:36.472 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:36.682 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:36 smithi081 ceph-mon[90133]: pgmap v335: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T04:27:36.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:36 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4189414918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-09-10T04:27:36.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:36 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1734583639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-09-10T04:27:36.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 -- 172.21.15.81:0/4056079948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100ab0 msgr2=0x7fedfc104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 --2- 172.21.15.81:0/4056079948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100ab0 0x7fedfc104b00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fedec009840 tx=0x7fedec009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 -- 172.21.15.81:0/4056079948 shutdown_connections 2022-09-10T04:27:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 --2- 172.21.15.81:0/4056079948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedfc105210 0x7fedfc10cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 --2- 172.21.15.81:0/4056079948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100110 0x7fedfc1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 --2- 172.21.15.81:0/4056079948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100ab0 0x7fedfc104b00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 -- 172.21.15.81:0/4056079948 >> 172.21.15.81:0/4056079948 conn(0x7fedfc0fbab0 msgr2=0x7fedfc0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 -- 172.21.15.81:0/4056079948 shutdown_connections 2022-09-10T04:27:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.592+0000 7fee02ac4700 1 -- 172.21.15.81:0/4056079948 wait complete. 2022-09-10T04:27:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.593+0000 7fee02ac4700 1 Processor -- start 2022-09-10T04:27:36.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.593+0000 7fee02ac4700 1 -- start start 2022-09-10T04:27:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee02ac4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100110 0x7fedfc19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee02ac4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100ab0 0x7fedfc19c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:36.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee02ac4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedfc105210 0x7fedfc195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee02ac4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc1962d0 con 0x7fedfc100ab0 2022-09-10T04:27:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee02ac4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc196440 con 0x7fedfc100110 2022-09-10T04:27:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee02ac4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc196600 con 0x7fedfc105210 2022-09-10T04:27:36.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee01ac2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100110 0x7fedfc19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee01ac2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100110 0x7fedfc19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49442/0 (socket says 172.21.15.81:49442) 2022-09-10T04:27:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee01ac2700 1 -- 172.21.15.81:0/44137827 learned_addr learned my addr 172.21.15.81:0/44137827 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee012c1700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100ab0 0x7fedfc19c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:36.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.594+0000 7fee022c3700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedfc105210 0x7fedfc195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:36.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee012c1700 1 -- 172.21.15.81:0/44137827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedfc105210 msgr2=0x7fedfc195b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee012c1700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedfc105210 0x7fedfc195b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee012c1700 1 -- 172.21.15.81:0/44137827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100110 msgr2=0x7fedfc19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.744 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee012c1700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100110 0x7fedfc19bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee012c1700 1 -- 172.21.15.81:0/44137827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedec009470 con 0x7fedfc100ab0 2022-09-10T04:27:36.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee012c1700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100ab0 0x7fedfc19c000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fedec006010 tx=0x7fedec00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:36.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec01d070 con 0x7fedfc100ab0 2022-09-10T04:27:36.745 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc196880 con 0x7fedfc100ab0 2022-09-10T04:27:36.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fedec00b480 con 0x7fedfc100ab0 2022-09-10T04:27:36.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec004810 con 0x7fedfc100ab0 2022-09-10T04:27:36.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.595+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc1a2ab0 con 0x7fedfc100ab0 2022-09-10T04:27:36.746 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.597+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc04d480 con 0x7fedfc100ab0 2022-09-10T04:27:36.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.599+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fedec028030 con 0x7fedfc100ab0 2022-09-10T04:27:36.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.601+0000 7fedf2ffd700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fede8099010 0x7fede809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:36.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.601+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fedec0c3310 con 0x7fedfc100ab0 2022-09-10T04:27:36.747 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.601+0000 7fee01ac2700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fede8099010 0x7fede809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:36.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.602+0000 7fee01ac2700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fede8099010 0x7fede809b4c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fedf80057d0 tx=0x7fedf8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:36.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.603+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fedec08f6f0 con 0x7fedfc100ab0 2022-09-10T04:27:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:36 smithi104 ceph-mon[97044]: pgmap v335: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T04:27:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:36 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4189414918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-09-10T04:27:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:36 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1734583639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-09-10T04:27:36.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.839+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fedfc10fd20 con 0x7fedfc100ab0 2022-09-10T04:27:36.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.839+0000 7fedf2ffd700 1 -- 172.21.15.81:0/44137827 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fedec030020 con 0x7fedfc100ab0 2022-09-10T04:27:36.841 INFO:teuthology.orchestra.run.smithi081.stdout:347892351001 2022-09-10T04:27:36.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fede8099010 msgr2=0x7fede809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fede8099010 0x7fede809b4c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fedf80057d0 tx=0x7fedf8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100ab0 msgr2=0x7fedfc19c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:36.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100ab0 0x7fedfc19c000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fedec006010 tx=0x7fedec00f690 comp rx=0 tx=0).stop 2022-09-10T04:27:36.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 shutdown_connections 2022-09-10T04:27:36.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fede8099010 0x7fede809b4c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fedfc105210 0x7fedfc195b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fedfc100110 0x7fedfc19bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 --2- 172.21.15.81:0/44137827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedfc100ab0 0x7fedfc19c000 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:36.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 >> 172.21.15.81:0/44137827 conn(0x7fedfc0fbab0 msgr2=0x7fedfc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:36.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.842+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 shutdown_connections 2022-09-10T04:27:36.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:36.843+0000 7fee02ac4700 1 -- 172.21.15.81:0/44137827 wait complete. 2022-09-10T04:27:36.856 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299434 got 223338299437 for osd.7 2022-09-10T04:27:36.856 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:36 smithi110 ceph-mon[97633]: pgmap v335: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T04:27:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:36 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4189414918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2022-09-10T04:27:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:36 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1734583639' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2022-09-10T04:27:37.589 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:37.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462961 got 201863462962 for osd.6 2022-09-10T04:27:37.621 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:37.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:37 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/44137827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-09-10T04:27:37.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.687 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.687 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:37 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.696 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149760 got 111669149763 for osd.3 2022-09-10T04:27:37.696 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:37.769 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 -- 172.21.15.81:0/3427500665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980066fd0 msgr2=0x7f398010a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:37.769 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 --2- 172.21.15.81:0/3427500665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980066fd0 0x7f398010a4d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3974009750 tx=0x7f3974009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:37.770 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 -- 172.21.15.81:0/3427500665 shutdown_connections 2022-09-10T04:27:37.770 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 --2- 172.21.15.81:0/3427500665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f398010aa10 0x7f398010cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.770 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 --2- 172.21.15.81:0/3427500665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39800666c0 0x7f3980066a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.770 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 --2- 172.21.15.81:0/3427500665 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3980066fd0 0x7f398010a4d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 -- 172.21.15.81:0/3427500665 >> 172.21.15.81:0/3427500665 conn(0x7f3980073cf0 msgr2=0x7f39800740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:37.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 -- 172.21.15.81:0/3427500665 shutdown_connections 2022-09-10T04:27:37.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.768+0000 7f3984860700 1 -- 172.21.15.81:0/3427500665 wait complete. 2022-09-10T04:27:37.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 Processor -- start 2022-09-10T04:27:37.774 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 -- start start 2022-09-10T04:27:37.774 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39800666c0 0x7f3980195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3980066fd0 0x7f39801964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.775 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f398010aa10 0x7f398019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398019afa0 con 0x7f398010aa10 2022-09-10T04:27:37.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398019b110 con 0x7f39800666c0 2022-09-10T04:27:37.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.770+0000 7f3984860700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398019b2d0 con 0x7f3980066fd0 2022-09-10T04:27:37.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39800666c0 0x7f3980195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3980066fd0 0x7f39801964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397e59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39800666c0 0x7f3980195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49470/0 (socket says 172.21.15.81:49470) 2022-09-10T04:27:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397e59c700 1 -- 172.21.15.81:0/2983614866 learned_addr learned my addr 172.21.15.81:0/2983614866 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:37.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397ed9d700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f398010aa10 0x7f398019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 -- 172.21.15.81:0/2983614866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39800666c0 msgr2=0x7f3980195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39800666c0 0x7f3980195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 -- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f398010aa10 msgr2=0x7f398019a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:37.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f398010aa10 0x7f398019a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 -- 172.21.15.81:0/2983614866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970009470 con 0x7f3980066fd0 2022-09-10T04:27:37.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.771+0000 7f397dd9b700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3980066fd0 0x7f39801964b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3974009750 tx=0x7f3974004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:37.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.772+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397401d070 con 0x7f3980066fd0 2022-09-10T04:27:37.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.772+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39740093b0 con 0x7f3980066fd0 2022-09-10T04:27:37.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.772+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3974003500 con 0x7f3980066fd0 2022-09-10T04:27:37.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.772+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974004760 con 0x7f3980066fd0 2022-09-10T04:27:37.783 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.772+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39801a2bc0 con 0x7f3980066fd0 2022-09-10T04:27:37.783 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.773+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3980064ea0 con 0x7f3980066fd0 2022-09-10T04:27:37.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.775+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f397400c040 con 0x7f3980066fd0 2022-09-10T04:27:37.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.776+0000 7f396f7fe700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3968099010 0x7f396809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.776+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f39740c35f0 con 0x7f3980066fd0 2022-09-10T04:27:37.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.776+0000 7f397e59c700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3968099010 0x7f396809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.777+0000 7f397e59c700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3968099010 0x7f396809b4c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3970005b90 tx=0x7f3970005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:37.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.780+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f397408f970 con 0x7f3980066fd0 2022-09-10T04:27:37.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:37 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/44137827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-09-10T04:27:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:37 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 -- 172.21.15.81:0/1348703959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 msgr2=0x7fb1fc066f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:37.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1348703959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc066f30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009750 tx=0x7fb1f0009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:37.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 -- 172.21.15.81:0/1348703959 shutdown_connections 2022-09-10T04:27:37.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1348703959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc067610 0x7fb1fc10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1348703959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1fc103ea0 0x7fb1fc104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1348703959 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc066f30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 -- 172.21.15.81:0/1348703959 >> 172.21.15.81:0/1348703959 conn(0x7fb1fc073cf0 msgr2=0x7fb1fc0740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 -- 172.21.15.81:0/1348703959 shutdown_connections 2022-09-10T04:27:37.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.862+0000 7fb200f5c700 1 -- 172.21.15.81:0/1348703959 wait complete. 2022-09-10T04:27:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.864+0000 7fb200f5c700 1 Processor -- start 2022-09-10T04:27:37.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.864+0000 7fb200f5c700 1 -- start start 2022-09-10T04:27:37.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.864+0000 7fb200f5c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1fc067610 0x7fb1fc195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.864+0000 7fb200f5c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc103ea0 0x7fb1fc196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.864+0000 7fb200f5c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc19a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb200f5c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc19af10 con 0x7fb1fc103ea0 2022-09-10T04:27:37.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb200f5c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc19b080 con 0x7fb1fc104840 2022-09-10T04:27:37.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb200f5c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc19b240 con 0x7fb1fc067610 2022-09-10T04:27:37.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb1fad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc19a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb1fad9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc19a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49494/0 (socket says 172.21.15.81:49494) 2022-09-10T04:27:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb1fad9d700 1 -- 172.21.15.81:0/1996856280 learned_addr learned my addr 172.21.15.81:0/1996856280 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:37.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.865+0000 7fb1fa59c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1fc067610 0x7fb1fc195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.866+0000 7fb1fad9d700 1 -- 172.21.15.81:0/1996856280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1fc067610 msgr2=0x7fb1fc195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:37.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.866+0000 7fb1fad9d700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1fc067610 0x7fb1fc195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.866+0000 7fb1fad9d700 1 -- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc103ea0 msgr2=0x7fb1fc196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:37.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.866+0000 7fb1fad9d700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc103ea0 0x7fb1fc196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:37.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.866+0000 7fb1fad9d700 1 -- 172.21.15.81:0/1996856280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00093b0 con 0x7fb1fc104840 2022-09-10T04:27:37.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.866+0000 7fb1fad9d700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc19a880 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400ebf0 tx=0x7fb1f4009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:37.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.867+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40099c0 con 0x7fb1fc104840 2022-09-10T04:27:37.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.867+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb1f400ccb0 con 0x7fb1fc104840 2022-09-10T04:27:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.867+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4010430 con 0x7fb1fc104840 2022-09-10T04:27:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.868+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc19b520 con 0x7fb1fc104840 2022-09-10T04:27:37.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.868+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc0688e0 con 0x7fb1fc104840 2022-09-10T04:27:37.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.870+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc064ea0 con 0x7fb1fc104840 2022-09-10T04:27:37.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.873+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb1f400c040 con 0x7fb1fc104840 2022-09-10T04:27:37.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.875+0000 7fb1eb7fe700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1e4099090 0x7fb1e409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:37.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.875+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fb1f4014070 con 0x7fb1fc104840 2022-09-10T04:27:37.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.879+0000 7fb1fa59c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1e4099090 0x7fb1e409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:37.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.880+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb1f408f000 con 0x7fb1fc104840 2022-09-10T04:27:37.885 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:37.880+0000 7fb1fa59c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1e4099090 0x7fb1e409b540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec005fd0 tx=0x7fb1ec005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:37 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/44137827' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2022-09-10T04:27:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:37 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:37.945 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776160 got 292057776162 for osd.9 2022-09-10T04:27:37.945 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:38.091 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.090+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f398010ff10 con 0x7f3980066fd0 2022-09-10T04:27:38.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.090+0000 7f396f7fe700 1 -- 172.21.15.81:0/2983614866 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3974092f90 con 0x7f3980066fd0 2022-09-10T04:27:38.092 INFO:teuthology.orchestra.run.smithi081.stdout:167503724601 2022-09-10T04:27:38.094 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3968099010 msgr2=0x7f396809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3968099010 0x7f396809b4c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3970005b90 tx=0x7f3970005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3980066fd0 msgr2=0x7f39801964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3980066fd0 0x7f39801964b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3974009750 tx=0x7f3974004230 comp rx=0 tx=0).stop 2022-09-10T04:27:38.095 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 shutdown_connections 2022-09-10T04:27:38.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f398010aa10 0x7f398019a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39800666c0 0x7f3980195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3968099010 0x7f396809b4c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 --2- 172.21.15.81:0/2983614866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3980066fd0 0x7f39801964b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.093+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 >> 172.21.15.81:0/2983614866 conn(0x7f3980073cf0 msgr2=0x7f39800fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.094+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 shutdown_connections 2022-09-10T04:27:38.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.094+0000 7f3984860700 1 -- 172.21.15.81:0/2983614866 wait complete. 2022-09-10T04:27:38.151 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.150+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb1fc068e90 con 0x7fb1fc104840 2022-09-10T04:27:38.152 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.151+0000 7fb1eb7fe700 1 -- 172.21.15.81:0/1996856280 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb1f4092620 con 0x7fb1fc104840 2022-09-10T04:27:38.152 INFO:teuthology.orchestra.run.smithi081.stdout:55834574926 2022-09-10T04:27:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1e4099090 msgr2=0x7fb1e409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1e4099090 0x7fb1e409b540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec005fd0 tx=0x7fb1ec005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 msgr2=0x7fb1fc19a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.155 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc19a880 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1f400ebf0 tx=0x7fb1f4009f60 comp rx=0 tx=0).stop 2022-09-10T04:27:38.156 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 shutdown_connections 2022-09-10T04:27:38.156 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1fc104840 0x7fb1fc19a880 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.156 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1e4099090 0x7fb1e409b540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.156 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1fc067610 0x7fb1fc195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.157 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 --2- 172.21.15.81:0/1996856280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1fc103ea0 0x7fb1fc196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.157 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 >> 172.21.15.81:0/1996856280 conn(0x7fb1fc073cf0 msgr2=0x7fb1fc0fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:38.157 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 shutdown_connections 2022-09-10T04:27:38.157 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.154+0000 7fb200f5c700 1 -- 172.21.15.81:0/1996856280 wait complete. 2022-09-10T04:27:38.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.273+0000 7f91561c1700 1 -- 172.21.15.81:0/3334086981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fca80 msgr2=0x7f91500fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.273+0000 7f91561c1700 1 --2- 172.21.15.81:0/3334086981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fca80 0x7f91500fce50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f914c009750 tx=0x7f914c009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:38.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 -- 172.21.15.81:0/3334086981 shutdown_connections 2022-09-10T04:27:38.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 --2- 172.21.15.81:0/3334086981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915010ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 --2- 172.21.15.81:0/3334086981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fca80 0x7f91500fce50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 --2- 172.21.15.81:0/3334086981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91500fd390 0x7f915010a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 -- 172.21.15.81:0/3334086981 >> 172.21.15.81:0/3334086981 conn(0x7f91500f8710 msgr2=0x7f91500fab20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:38.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 -- 172.21.15.81:0/3334086981 shutdown_connections 2022-09-10T04:27:38.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.274+0000 7f91561c1700 1 -- 172.21.15.81:0/3334086981 wait complete. 2022-09-10T04:27:38.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.275+0000 7f91561c1700 1 Processor -- start 2022-09-10T04:27:38.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.275+0000 7f91561c1700 1 -- start start 2022-09-10T04:27:38.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91561c1700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91500fca80 0x7f9150195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:38.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91561c1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fd390 0x7f91501962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:38.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91561c1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:38.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91561c1700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915019ada0 con 0x7f91500fca80 2022-09-10T04:27:38.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91561c1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915019af10 con 0x7f915010a630 2022-09-10T04:27:38.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91561c1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915019b0d0 con 0x7f91500fd390 2022-09-10T04:27:38.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91559c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:38.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91559c0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915019a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49522/0 (socket says 172.21.15.81:49522) 2022-09-10T04:27:38.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91559c0700 1 -- 172.21.15.81:0/2133653818 learned_addr learned my addr 172.21.15.81:0/2133653818 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:38.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91549be700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fd390 0x7f91501962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:38.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91551bf700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91500fca80 0x7f9150195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:38.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91559c0700 1 -- 172.21.15.81:0/2133653818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fd390 msgr2=0x7f91501962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.276+0000 7f91559c0700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fd390 0x7f91501962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91559c0700 1 -- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91500fca80 msgr2=0x7f9150195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91559c0700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91500fca80 0x7f9150195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91559c0700 1 -- 172.21.15.81:0/2133653818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140009470 con 0x7f915010a630 2022-09-10T04:27:38.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91559c0700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915019a710 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f914800eaf0 tx=0x7f914800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:38.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148009960 con 0x7f915010a630 2022-09-10T04:27:38.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f914800cc90 con 0x7f915010a630 2022-09-10T04:27:38.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0093b0 con 0x7f915010a630 2022-09-10T04:27:38.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9148010430 con 0x7f915010a630 2022-09-10T04:27:38.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.277+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91501a29c0 con 0x7f915010a630 2022-09-10T04:27:38.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.279+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915004d480 con 0x7f915010a630 2022-09-10T04:27:38.294 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.281+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f914801e020 con 0x7f915010a630 2022-09-10T04:27:38.294 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.282+0000 7f91467fc700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f913c099010 0x7f913c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:38.295 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.282+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f9148014070 con 0x7f915010a630 2022-09-10T04:27:38.295 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.282+0000 7f91551bf700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f913c099010 0x7f913c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:38.295 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.283+0000 7f91551bf700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f913c099010 0x7f913c09b4c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f914c009e30 tx=0x7f914c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:38.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.286+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f914808f060 con 0x7f915010a630 2022-09-10T04:27:38.325 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:38.364 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350998 got 347892351001 for osd.11 2022-09-10T04:27:38.364 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:38.453 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:38 smithi081 ceph-mon[90133]: pgmap v336: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:27:38.454 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:38 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2983614866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-09-10T04:27:38.455 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:38 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1996856280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-09-10T04:27:38.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.537+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f91501a3030 con 0x7f915010a630 2022-09-10T04:27:38.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.538+0000 7f91467fc700 1 -- 172.21.15.81:0/2133653818 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9148092680 con 0x7f915010a630 2022-09-10T04:27:38.541 INFO:teuthology.orchestra.run.smithi081.stdout:146028888128 2022-09-10T04:27:38.542 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.540+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f913c099010 msgr2=0x7f913c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.543 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f913c099010 0x7f913c09b4c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f914c009e30 tx=0x7f914c005f90 comp rx=0 tx=0).stop 2022-09-10T04:27:38.543 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 msgr2=0x7f915019a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:38.544 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915019a710 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f914800eaf0 tx=0x7f914800ee00 comp rx=0 tx=0).stop 2022-09-10T04:27:38.544 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 shutdown_connections 2022-09-10T04:27:38.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f915010a630 0x7f915019a710 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91500fca80 0x7f9150195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f913c099010 0x7f913c09b4c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 --2- 172.21.15.81:0/2133653818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91500fd390 0x7f91501962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:38.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 >> 172.21.15.81:0/2133653818 conn(0x7f91500f8710 msgr2=0x7f915010b960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:38.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 shutdown_connections 2022-09-10T04:27:38.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:38.541+0000 7f91561c1700 1 -- 172.21.15.81:0/2133653818 wait complete. 2022-09-10T04:27:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:38 smithi104 ceph-mon[97044]: pgmap v336: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:27:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:38 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2983614866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-09-10T04:27:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:38 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1996856280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-09-10T04:27:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:38 smithi110 ceph-mon[97633]: pgmap v336: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:27:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:38 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2983614866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2022-09-10T04:27:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:38 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1996856280' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2022-09-10T04:27:38.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724598 got 167503724601 for osd.5 2022-09-10T04:27:38.914 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:39.473 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888125 got 146028888128 for osd.4 2022-09-10T04:27:39.474 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:39.477 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574926 for osd.1 2022-09-10T04:27:39.478 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:39.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:39 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2133653818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-09-10T04:27:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:39 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2133653818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-09-10T04:27:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:39 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2133653818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2022-09-10T04:27:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:40 smithi081 ceph-mon[90133]: pgmap v337: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:27:40.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:40 smithi104 ceph-mon[97044]: pgmap v337: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:27:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:40 smithi110 ceph-mon[97633]: pgmap v337: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:27:41.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 -- 172.21.15.81:0/2334070619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8100110 msgr2=0x7f7ca81004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2334070619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8100110 0x7f7ca81004e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009810 tx=0x7f7c98009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:41.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 -- 172.21.15.81:0/2334070619 shutdown_connections 2022-09-10T04:27:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2334070619 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8105210 0x7f7ca810cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2334070619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8100110 0x7f7ca81004e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2334070619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.054 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.890+0000 7f7ca759e700 1 -- 172.21.15.81:0/2334070619 >> 172.21.15.81:0/2334070619 conn(0x7f7ca80fba90 msgr2=0x7f7ca80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:41.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.891+0000 7f7ca759e700 1 -- 172.21.15.81:0/2334070619 shutdown_connections 2022-09-10T04:27:41.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.891+0000 7f7ca759e700 1 -- 172.21.15.81:0/2334070619 wait complete. 2022-09-10T04:27:41.055 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.892+0000 7f7ca759e700 1 Processor -- start 2022-09-10T04:27:41.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 -- start start 2022-09-10T04:27:41.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100110 0x7f7ca8195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8105210 0x7f7ca81995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8199c20 con 0x7f7ca8100110 2022-09-10T04:27:41.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8199d90 con 0x7f7ca8100ab0 2022-09-10T04:27:41.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.893+0000 7f7ca759e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8199f50 con 0x7f7ca8105210 2022-09-10T04:27:41.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca5d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100110 0x7f7ca8195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca5d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8196210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:35872/0 (socket says 172.21.15.81:35872) 2022-09-10T04:27:41.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca5d9b700 1 -- 172.21.15.81:0/2471325148 learned_addr learned my addr 172.21.15.81:0/2471325148 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca6d9d700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8105210 0x7f7ca81995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca659c700 1 -- 172.21.15.81:0/2471325148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8105210 msgr2=0x7f7ca81995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca659c700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8105210 0x7f7ca81995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca659c700 1 -- 172.21.15.81:0/2471325148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 msgr2=0x7f7ca8196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca659c700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca659c700 1 -- 172.21.15.81:0/2471325148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c98009470 con 0x7f7ca8100110 2022-09-10T04:27:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.894+0000 7f7ca5d9b700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.895+0000 7f7ca659c700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100110 0x7f7ca8195cd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c98000c00 tx=0x7f7c9800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.895+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9801d070 con 0x7f7ca8100110 2022-09-10T04:27:41.063 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.895+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca819a1d0 con 0x7f7ca8100110 2022-09-10T04:27:41.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.895+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f7c9800f880 con 0x7f7ca8100110 2022-09-10T04:27:41.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.895+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98004760 con 0x7f7ca8100110 2022-09-10T04:27:41.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.895+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca81a2b30 con 0x7f7ca8100110 2022-09-10T04:27:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.897+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca804d480 con 0x7f7ca8100110 2022-09-10T04:27:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.900+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f7c98028030 con 0x7f7ca8100110 2022-09-10T04:27:41.066 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.901+0000 7f7c977fe700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7c90099090 0x7f7c9009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.067 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.901+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f7c980c3150 con 0x7f7ca8100110 2022-09-10T04:27:41.067 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.902+0000 7f7ca5d9b700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7c90099090 0x7f7c9009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.068 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.903+0000 7f7ca5d9b700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7c90099090 0x7f7c9009b540 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7ca81972a0 tx=0x7f7c9c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:41.068 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:40.906+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f7c9808f530 con 0x7f7ca8100110 2022-09-10T04:27:41.071 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 -- 172.21.15.81:0/2380552154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100140 msgr2=0x7f712c100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 --2- 172.21.15.81:0/2380552154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100140 0x7f712c100510 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7120009840 tx=0x7f7120009b50 comp rx=0 tx=0).stop 2022-09-10T04:27:41.072 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 -- 172.21.15.81:0/2380552154 shutdown_connections 2022-09-10T04:27:41.075 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 --2- 172.21.15.81:0/2380552154 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 0x7f712c10cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 --2- 172.21.15.81:0/2380552154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100140 0x7f712c100510 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 --2- 172.21.15.81:0/2380552154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100ae0 0x7f712c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.068+0000 7f7131b49700 1 -- 172.21.15.81:0/2380552154 >> 172.21.15.81:0/2380552154 conn(0x7f712c0fba20 msgr2=0x7f712c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:41.076 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.069+0000 7f7131b49700 1 -- 172.21.15.81:0/2380552154 shutdown_connections 2022-09-10T04:27:41.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.069+0000 7f7131b49700 1 -- 172.21.15.81:0/2380552154 wait complete. 2022-09-10T04:27:41.077 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.070+0000 7f7131b49700 1 Processor -- start 2022-09-10T04:27:41.079 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.070+0000 7f7131b49700 1 -- start start 2022-09-10T04:27:41.080 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7131b49700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 0x7f712c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.080 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7131b49700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100ae0 0x7f712c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.080 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7131b49700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 0x7f712c195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.080 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7131b49700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c196270 con 0x7f712c100ae0 2022-09-10T04:27:41.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7131b49700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c1963e0 con 0x7f712c100140 2022-09-10T04:27:41.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7131b49700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c1965a0 con 0x7f712c1051f0 2022-09-10T04:27:41.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7130b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 0x7f712c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.081 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7130b47700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 0x7f712c19ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:35888/0 (socket says 172.21.15.81:35888) 2022-09-10T04:27:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.071+0000 7f7130b47700 1 -- 172.21.15.81:0/29188201 learned_addr learned my addr 172.21.15.81:0/29188201 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7131348700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 0x7f712c195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7130b47700 1 -- 172.21.15.81:0/29188201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 msgr2=0x7f712c195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7130b47700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 0x7f712c195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.082 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7130b47700 1 -- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100ae0 msgr2=0x7f712c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7130b47700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100ae0 0x7f712c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7130b47700 1 -- 172.21.15.81:0/29188201 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180093b0 con 0x7f712c100140 2022-09-10T04:27:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7131348700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 0x7f712c195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:27:41.083 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7130b47700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 0x7f712c19ba60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7120009760 tx=0x7f712000f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712001d070 con 0x7f712c100140 2022-09-10T04:27:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7120009470 con 0x7f712c100140 2022-09-10T04:27:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f712000f8e0 con 0x7f712c100140 2022-09-10T04:27:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120004880 con 0x7f712c100140 2022-09-10T04:27:41.084 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.072+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c1a2a50 con 0x7f712c100140 2022-09-10T04:27:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.074+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c04d480 con 0x7f712c100140 2022-09-10T04:27:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.076+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f7120028030 con 0x7f712c100140 2022-09-10T04:27:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.077+0000 7f7129ffb700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f711c099010 0x7f711c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:41.085 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.077+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f71200c3510 con 0x7f712c100140 2022-09-10T04:27:41.086 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.078+0000 7f712bfff700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f711c099010 0x7f711c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:41.086 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.078+0000 7f712bfff700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f711c099010 0x7f711c09b4c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7118009380 tx=0x7f71180092d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:41.086 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.079+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f712008f890 con 0x7f712c100140 2022-09-10T04:27:41.144 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.143+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7ca81a30e0 con 0x7f7ca8100110 2022-09-10T04:27:41.145 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.144+0000 7f7c977fe700 1 -- 172.21.15.81:0/2471325148 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c98026020 con 0x7f7ca8100110 2022-09-10T04:27:41.145 INFO:teuthology.orchestra.run.smithi081.stdout:34359738453 2022-09-10T04:27:41.148 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.146+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7c90099090 msgr2=0x7f7c9009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.148 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.146+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7c90099090 0x7f7c9009b540 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7ca81972a0 tx=0x7f7c9c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.148 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.146+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100110 msgr2=0x7f7ca8195cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.148 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.146+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100110 0x7f7ca8195cd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c98000c00 tx=0x7f7c9800f690 comp rx=0 tx=0).stop 2022-09-10T04:27:41.149 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 shutdown_connections 2022-09-10T04:27:41.149 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f7c90099090 0x7f7c9009b540 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.149 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ca8105210 0x7f7ca81995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.149 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ca8100110 0x7f7ca8195cd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.149 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 --2- 172.21.15.81:0/2471325148 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ca8100ab0 0x7f7ca8196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 >> 172.21.15.81:0/2471325148 conn(0x7f7ca80fba90 msgr2=0x7f7ca8103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:41.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 shutdown_connections 2022-09-10T04:27:41.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.147+0000 7f7ca759e700 1 -- 172.21.15.81:0/2471325148 wait complete. 2022-09-10T04:27:41.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.337+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f712c197040 con 0x7f712c100140 2022-09-10T04:27:41.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.338+0000 7f7129ffb700 1 -- 172.21.15.81:0/29188201 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7120026020 con 0x7f712c100140 2022-09-10T04:27:41.340 INFO:teuthology.orchestra.run.smithi081.stdout:77309411403 2022-09-10T04:27:41.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f711c099010 msgr2=0x7f711c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f711c099010 0x7f711c09b4c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7118009380 tx=0x7f71180092d0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 msgr2=0x7f712c19ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:41.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 0x7f712c19ba60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7120009760 tx=0x7f712000f740 comp rx=0 tx=0).stop 2022-09-10T04:27:41.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 shutdown_connections 2022-09-10T04:27:41.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f711c099010 0x7f711c09b4c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f712c1051f0 0x7f712c195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f712c100140 0x7f712c19ba60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 --2- 172.21.15.81:0/29188201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f712c100ae0 0x7f712c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 >> 172.21.15.81:0/29188201 conn(0x7f712c0fba20 msgr2=0x7f712c0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:41.346 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.341+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 shutdown_connections 2022-09-10T04:27:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:41.342+0000 7f7131b49700 1 -- 172.21.15.81:0/29188201 wait complete. 2022-09-10T04:27:41.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738453 for osd.0 2022-09-10T04:27:41.612 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:41 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2471325148' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-09-10T04:27:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:41 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/29188201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-09-10T04:27:41.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:41 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2471325148' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-09-10T04:27:41.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:41 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/29188201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-09-10T04:27:41.906 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411403 for osd.2 2022-09-10T04:27:41.906 DEBUG:teuthology.parallel:result is None 2022-09-10T04:27:41.907 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2022-09-10T04:27:41.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph pg dump --format=json 2022-09-10T04:27:41.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:41 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2471325148' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2022-09-10T04:27:41.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:41 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/29188201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2022-09-10T04:27:42.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:42 smithi104 ceph-mon[97044]: pgmap v338: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:42 smithi081 ceph-mon[90133]: pgmap v338: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:42.807 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:42 smithi110 ceph-mon[97633]: pgmap v338: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:44 smithi081 ceph-mon[90133]: pgmap v339: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:44 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:44 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:44.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 -- 172.21.15.81:0/1520213675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 msgr2=0x7f041410cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:44.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 --2- 172.21.15.81:0/1520213675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 0x7f041410cb10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f040c009bb0 tx=0x7f040c009ec0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 -- 172.21.15.81:0/1520213675 shutdown_connections 2022-09-10T04:27:44.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 --2- 172.21.15.81:0/1520213675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 0x7f041410cb10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 --2- 172.21.15.81:0/1520213675 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 --2- 172.21.15.81:0/1520213675 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 0x7f041410a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 -- 172.21.15.81:0/1520213675 >> 172.21.15.81:0/1520213675 conn(0x7f04140f9a10 msgr2=0x7f04140fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:44.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 -- 172.21.15.81:0/1520213675 shutdown_connections 2022-09-10T04:27:44.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.360+0000 7f0418d73700 1 -- 172.21.15.81:0/1520213675 wait complete. 2022-09-10T04:27:44.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.362+0000 7f0418d73700 1 Processor -- start 2022-09-10T04:27:44.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.362+0000 7f0418d73700 1 -- start start 2022-09-10T04:27:44.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0418d73700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:44.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0418d73700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 0x7f0414196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:44.373 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0418d73700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 0x7f041419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:44.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0418d73700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041419ae80 con 0x7f041410a730 2022-09-10T04:27:44.374 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0418d73700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041419aff0 con 0x7f0414071080 2022-09-10T04:27:44.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0418d73700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041419b1b0 con 0x7f0414071990 2022-09-10T04:27:44.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f041259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:44.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f041259c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414195e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:35906/0 (socket says 172.21.15.81:35906) 2022-09-10T04:27:44.375 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0412d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 0x7f041419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:44.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f0411d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 0x7f0414196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:44.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.363+0000 7f041259c700 1 -- 172.21.15.81:0/1939655840 learned_addr learned my addr 172.21.15.81:0/1939655840 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:44.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f041259c700 1 -- 172.21.15.81:0/1939655840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 msgr2=0x7f0414196390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:44.376 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f041259c700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 0x7f0414196390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f041259c700 1 -- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 msgr2=0x7f041419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:44.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f041259c700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 0x7f041419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f041259c700 1 -- 172.21.15.81:0/1939655840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040093b0 con 0x7f0414071080 2022-09-10T04:27:44.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f0411d9b700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 0x7f0414196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:44.377 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f041259c700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414195e50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f040400e9c0 tx=0x7f040400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:44.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040097b0 con 0x7f0414071080 2022-09-10T04:27:44.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c0097e0 con 0x7f0414071080 2022-09-10T04:27:44.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f040400ce70 con 0x7f0414071080 2022-09-10T04:27:44.378 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404003cf0 con 0x7f0414071080 2022-09-10T04:27:44.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.364+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04141a2c80 con 0x7f0414071080 2022-09-10T04:27:44.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.365+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0414064ea0 con 0x7f0414071080 2022-09-10T04:27:44.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.368+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f040401e020 con 0x7f0414071080 2022-09-10T04:27:44.379 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.369+0000 7f04037fe700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f03fc099090 0x7f03fc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:44.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.369+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f0404014070 con 0x7f0414071080 2022-09-10T04:27:44.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.370+0000 7f0411d9b700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f03fc099090 0x7f03fc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:44.380 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.370+0000 7f0411d9b700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f03fc099090 0x7f03fc09b540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0414197460 tx=0x7f0408005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:44.381 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.373+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f040408ee30 con 0x7f0414071080 2022-09-10T04:27:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:44 smithi104 ceph-mon[97044]: pgmap v339: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:44 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:44 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:44.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.595+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04140fb660 con 0x7f03fc099090 2022-09-10T04:27:44.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.609+0000 7f04037fe700 1 -- 172.21.15.81:0/1939655840 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426916 (secure 0 0 0) 0x7f04140fb660 con 0x7f03fc099090 2022-09-10T04:27:44.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:44.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f03fc099090 msgr2=0x7f03fc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:44.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f03fc099090 0x7f03fc09b540 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0414197460 tx=0x7f0408005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 msgr2=0x7f0414195e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:44.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414195e50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f040400e9c0 tx=0x7f040400ecd0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 shutdown_connections 2022-09-10T04:27:44.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f041410a730 0x7f041419a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0414071080 0x7f0414195e50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f03fc099090 0x7f03fc09b540 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 --2- 172.21.15.81:0/1939655840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0414071990 0x7f0414196390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:44.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.616+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 >> 172.21.15.81:0/1939655840 conn(0x7f04140f9a10 msgr2=0x7f041410c920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:44.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.617+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 shutdown_connections 2022-09-10T04:27:44.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:44.617+0000 7f0418d73700 1 -- 172.21.15.81:0/1939655840 wait complete. 2022-09-10T04:27:44.633 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2022-09-10T04:27:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:44 smithi110 ceph-mon[97633]: pgmap v339: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:44 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:44 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:45.093 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2022-09-10T04:27:43.356017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600267,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":591,"num_read_kb":364,"num_write":390,"num_write_kb":3005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":320,"ondisk_log_size":320,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82504,"kb_used_data":14056,"kb_used_omap":0,"kb_used_meta":68352,"kb_avail":1124859320,"statfs":{"total":1151940427776,"available":1151855943680,"internally_reserved":0,"allocated":14393344,"data_stored":7766662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69992448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003433"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422009+0000","last_change":"2022-09-10T04:26:32.116469+0000","last_active":"2022-09-10T04:27:09.422009+0000","last_peered":"2022-09-10T04:27:09.422009+0000","last_clean":"2022-09-10T04:27:09.422009+0000","last_became_active":"2022-09-10T04:26:32.116060+0000","last_became_peered":"2022-09-10T04:26:32.116060+0000","last_unstale":"2022-09-10T04:27:09.422009+0000","last_undegraded":"2022-09-10T04:27:09.422009+0000","last_fullsized":"2022-09-10T04:27:09.422009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:22:05.868811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429692+0000","last_change":"2022-09-10T04:26:32.114659+0000","last_active":"2022-09-10T04:27:09.429692+0000","last_peered":"2022-09-10T04:27:09.429692+0000","last_clean":"2022-09-10T04:27:09.429692+0000","last_became_active":"2022-09-10T04:26:32.114381+0000","last_became_peered":"2022-09-10T04:26:32.114381+0000","last_unstale":"2022-09-10T04:27:09.429692+0000","last_undegraded":"2022-09-10T04:27:09.429692+0000","last_fullsized":"2022-09-10T04:27:09.429692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:53:50.267695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429488+0000","last_change":"2022-09-10T04:26:32.117123+0000","last_active":"2022-09-10T04:27:09.429488+0000","last_peered":"2022-09-10T04:27:09.429488+0000","last_clean":"2022-09-10T04:27:09.429488+0000","last_became_active":"2022-09-10T04:26:32.116753+0000","last_became_peered":"2022-09-10T04:26:32.116753+0000","last_unstale":"2022-09-10T04:27:09.429488+0000","last_undegraded":"2022-09-10T04:27:09.429488+0000","last_fullsized":"2022-09-10T04:27:09.429488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:52.713788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421339+0000","last_change":"2022-09-10T04:26:32.120151+0000","last_active":"2022-09-10T04:27:09.421339+0000","last_peered":"2022-09-10T04:27:09.421339+0000","last_clean":"2022-09-10T04:27:09.421339+0000","last_became_active":"2022-09-10T04:26:32.119510+0000","last_became_peered":"2022-09-10T04:26:32.119510+0000","last_unstale":"2022-09-10T04:27:09.421339+0000","last_undegraded":"2022-09-10T04:27:09.421339+0000","last_fullsized":"2022-09-10T04:27:09.421339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:56:23.232054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423492+0000","last_change":"2022-09-10T04:26:32.138820+0000","last_active":"2022-09-10T04:27:09.423492+0000","last_peered":"2022-09-10T04:27:09.423492+0000","last_clean":"2022-09-10T04:27:09.423492+0000","last_became_active":"2022-09-10T04:26:32.138466+0000","last_became_peered":"2022-09-10T04:26:32.138466+0000","last_unstale":"2022-09-10T04:27:09.423492+0000","last_undegraded":"2022-09-10T04:27:09.423492+0000","last_fullsized":"2022-09-10T04:27:09.423492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:16:22.128598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.424838+0000","last_change":"2022-09-10T04:27:10.424838+0000","last_active":"2022-09-10T04:27:10.424838+0000","last_peered":"2022-09-10T04:27:10.424838+0000","last_clean":"2022-09-10T04:27:10.424838+0000","last_became_active":"2022-09-10T04:27:10.424425+0000","last_became_peered":"2022-09-10T04:27:10.424425+0000","last_unstale":"2022-09-10T04:27:10.424838+0000","last_undegraded":"2022-09-10T04:27:10.424838+0000","last_fullsized":"2022-09-10T04:27:10.424838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:51:59.529546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426584+0000","last_change":"2022-09-10T04:26:32.122869+0000","last_active":"2022-09-10T04:27:09.426584+0000","last_peered":"2022-09-10T04:27:09.426584+0000","last_clean":"2022-09-10T04:27:09.426584+0000","last_became_active":"2022-09-10T04:26:32.122457+0000","last_became_peered":"2022-09-10T04:26:32.122457+0000","last_unstale":"2022-09-10T04:27:09.426584+0000","last_undegraded":"2022-09-10T04:27:09.426584+0000","last_fullsized":"2022-09-10T04:27:09.426584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:08.945173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740810+0000","last_change":"2022-09-10T04:27:10.739898+0000","last_active":"2022-09-10T04:27:10.740810+0000","last_peered":"2022-09-10T04:27:10.740810+0000","last_clean":"2022-09-10T04:27:10.740810+0000","last_became_active":"2022-09-10T04:27:10.739397+0000","last_became_peered":"2022-09-10T04:27:10.739397+0000","last_unstale":"2022-09-10T04:27:10.740810+0000","last_undegraded":"2022-09-10T04:27:10.740810+0000","last_fullsized":"2022-09-10T04:27:10.740810+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:40:17.580037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425123+0000","last_change":"2022-09-10T04:26:32.113787+0000","last_active":"2022-09-10T04:27:09.425123+0000","last_peered":"2022-09-10T04:27:09.425123+0000","last_clean":"2022-09-10T04:27:09.425123+0000","last_became_active":"2022-09-10T04:26:32.113454+0000","last_became_peered":"2022-09-10T04:26:32.113454+0000","last_unstale":"2022-09-10T04:27:09.425123+0000","last_undegraded":"2022-09-10T04:27:09.425123+0000","last_fullsized":"2022-09-10T04:27:09.425123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:11:23.375628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423927+0000","last_change":"2022-09-10T04:26:32.123249+0000","last_active":"2022-09-10T04:27:09.423927+0000","last_peered":"2022-09-10T04:27:09.423927+0000","last_clean":"2022-09-10T04:27:09.423927+0000","last_became_active":"2022-09-10T04:26:32.121836+0000","last_became_peered":"2022-09-10T04:26:32.121836+0000","last_unstale":"2022-09-10T04:27:09.423927+0000","last_undegraded":"2022-09-10T04:27:09.423927+0000","last_fullsized":"2022-09-10T04:27:09.423927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:22.711515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427289+0000","last_change":"2022-09-10T04:26:32.118213+0000","last_active":"2022-09-10T04:27:09.427289+0000","last_peered":"2022-09-10T04:27:09.427289+0000","last_clean":"2022-09-10T04:27:09.427289+0000","last_became_active":"2022-09-10T04:26:32.117863+0000","last_became_peered":"2022-09-10T04:26:32.117863+0000","last_unstale":"2022-09-10T04:27:09.427289+0000","last_undegraded":"2022-09-10T04:27:09.427289+0000","last_fullsized":"2022-09-10T04:27:09.427289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:26:35.632083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421238+0000","last_change":"2022-09-10T04:26:32.127748+0000","last_active":"2022-09-10T04:27:09.421238+0000","last_peered":"2022-09-10T04:27:09.421238+0000","last_clean":"2022-09-10T04:27:09.421238+0000","last_became_active":"2022-09-10T04:26:32.127470+0000","last_became_peered":"2022-09-10T04:26:32.127470+0000","last_unstale":"2022-09-10T04:27:09.421238+0000","last_undegraded":"2022-09-10T04:27:09.421238+0000","last_fullsized":"2022-09-10T04:27:09.421238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:01:58.438614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427201+0000","last_change":"2022-09-10T04:26:32.126378+0000","last_active":"2022-09-10T04:27:09.427201+0000","last_peered":"2022-09-10T04:27:09.427201+0000","last_clean":"2022-09-10T04:27:09.427201+0000","last_became_active":"2022-09-10T04:26:32.125510+0000","last_became_peered":"2022-09-10T04:26:32.125510+0000","last_unstale":"2022-09-10T04:27:09.427201+0000","last_undegraded":"2022-09-10T04:27:09.427201+0000","last_fullsized":"2022-09-10T04:27:09.427201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:41.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429073+0000","last_change":"2022-09-10T04:26:32.118515+0000","last_active":"2022-09-10T04:27:09.429073+0000","last_peered":"2022-09-10T04:27:09.429073+0000","last_clean":"2022-09-10T04:27:09.429073+0000","last_became_active":"2022-09-10T04:26:32.118163+0000","last_became_peered":"2022-09-10T04:26:32.118163+0000","last_unstale":"2022-09-10T04:27:09.429073+0000","last_undegraded":"2022-09-10T04:27:09.429073+0000","last_fullsized":"2022-09-10T04:27:09.429073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:39:42.245166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429375+0000","last_change":"2022-09-10T04:26:32.122232+0000","last_active":"2022-09-10T04:27:09.429375+0000","last_peered":"2022-09-10T04:27:09.429375+0000","last_clean":"2022-09-10T04:27:09.429375+0000","last_became_active":"2022-09-10T04:26:32.121906+0000","last_became_peered":"2022-09-10T04:26:32.121906+0000","last_unstale":"2022-09-10T04:27:09.429375+0000","last_undegraded":"2022-09-10T04:27:09.429375+0000","last_fullsized":"2022-09-10T04:27:09.429375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:49:50.711773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425182+0000","last_change":"2022-09-10T04:26:32.117413+0000","last_active":"2022-09-10T04:27:09.425182+0000","last_peered":"2022-09-10T04:27:09.425182+0000","last_clean":"2022-09-10T04:27:09.425182+0000","last_became_active":"2022-09-10T04:26:32.117120+0000","last_became_peered":"2022-09-10T04:26:32.117120+0000","last_unstale":"2022-09-10T04:27:09.425182+0000","last_undegraded":"2022-09-10T04:27:09.425182+0000","last_fullsized":"2022-09-10T04:27:09.425182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:50:28.426517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421816+0000","last_change":"2022-09-10T04:26:32.117852+0000","last_active":"2022-09-10T04:27:09.421816+0000","last_peered":"2022-09-10T04:27:09.421816+0000","last_clean":"2022-09-10T04:27:09.421816+0000","last_became_active":"2022-09-10T04:26:32.117526+0000","last_became_peered":"2022-09-10T04:26:32.117526+0000","last_unstale":"2022-09-10T04:27:09.421816+0000","last_undegraded":"2022-09-10T04:27:09.421816+0000","last_fullsized":"2022-09-10T04:27:09.421816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:02:45.194870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419780+0000","last_change":"2022-09-10T04:26:32.122937+0000","last_active":"2022-09-10T04:27:09.419780+0000","last_peered":"2022-09-10T04:27:09.419780+0000","last_clean":"2022-09-10T04:27:09.419780+0000","last_became_active":"2022-09-10T04:26:32.122549+0000","last_became_peered":"2022-09-10T04:26:32.122549+0000","last_unstale":"2022-09-10T04:27:09.419780+0000","last_undegraded":"2022-09-10T04:27:09.419780+0000","last_fullsized":"2022-09-10T04:27:09.419780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:19:48.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418617+0000","last_change":"2022-09-10T04:26:32.139841+0000","last_active":"2022-09-10T04:27:09.418617+0000","last_peered":"2022-09-10T04:27:09.418617+0000","last_clean":"2022-09-10T04:27:09.418617+0000","last_became_active":"2022-09-10T04:26:32.138962+0000","last_became_peered":"2022-09-10T04:26:32.138962+0000","last_unstale":"2022-09-10T04:27:09.418617+0000","last_undegraded":"2022-09-10T04:27:09.418617+0000","last_fullsized":"2022-09-10T04:27:09.418617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:51.196716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.417950+0000","last_change":"2022-09-10T04:26:36.594956+0000","last_active":"2022-09-10T04:27:09.417950+0000","last_peered":"2022-09-10T04:27:09.417950+0000","last_clean":"2022-09-10T04:27:09.417950+0000","last_became_active":"2022-09-10T04:26:36.594507+0000","last_became_peered":"2022-09-10T04:26:36.594507+0000","last_unstale":"2022-09-10T04:27:09.417950+0000","last_undegraded":"2022-09-10T04:27:09.417950+0000","last_fullsized":"2022-09-10T04:27:09.417950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:05:36.037017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":72,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:37.251607+0000","last_change":"2022-09-10T04:26:31.553482+0000","last_active":"2022-09-10T04:27:37.251607+0000","last_peered":"2022-09-10T04:27:37.251607+0000","last_clean":"2022-09-10T04:27:37.251607+0000","last_became_active":"2022-09-10T04:26:31.553170+0000","last_became_peered":"2022-09-10T04:26:31.553170+0000","last_unstale":"2022-09-10T04:27:37.251607+0000","last_undegraded":"2022-09-10T04:27:37.251607+0000","last_fullsized":"2022-09-10T04:27:37.251607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:52:35.054369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425342+0000","last_change":"2022-09-10T04:26:32.139920+0000","last_active":"2022-09-10T04:27:09.425342+0000","last_peered":"2022-09-10T04:27:09.425342+0000","last_clean":"2022-09-10T04:27:09.425342+0000","last_became_active":"2022-09-10T04:26:32.139414+0000","last_became_peered":"2022-09-10T04:26:32.139414+0000","last_unstale":"2022-09-10T04:27:09.425342+0000","last_undegraded":"2022-09-10T04:27:09.425342+0000","last_fullsized":"2022-09-10T04:27:09.425342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:12.288511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420458+0000","last_change":"2022-09-10T04:26:36.603332+0000","last_active":"2022-09-10T04:27:09.420458+0000","last_peered":"2022-09-10T04:27:09.420458+0000","last_clean":"2022-09-10T04:27:09.420458+0000","last_became_active":"2022-09-10T04:26:36.602947+0000","last_became_peered":"2022-09-10T04:26:36.602947+0000","last_unstale":"2022-09-10T04:27:09.420458+0000","last_undegraded":"2022-09-10T04:27:09.420458+0000","last_fullsized":"2022-09-10T04:27:09.420458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:20:15.790086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425618+0000","last_change":"2022-09-10T04:26:31.559001+0000","last_active":"2022-09-10T04:27:09.425618+0000","last_peered":"2022-09-10T04:27:09.425618+0000","last_clean":"2022-09-10T04:27:09.425618+0000","last_became_active":"2022-09-10T04:26:31.558675+0000","last_became_peered":"2022-09-10T04:26:31.558675+0000","last_unstale":"2022-09-10T04:27:09.425618+0000","last_undegraded":"2022-09-10T04:27:09.425618+0000","last_fullsized":"2022-09-10T04:27:09.425618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:31:20.091765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429942+0000","last_change":"2022-09-10T04:26:32.122545+0000","last_active":"2022-09-10T04:27:09.429942+0000","last_peered":"2022-09-10T04:27:09.429942+0000","last_clean":"2022-09-10T04:27:09.429942+0000","last_became_active":"2022-09-10T04:26:32.122237+0000","last_became_peered":"2022-09-10T04:26:32.122237+0000","last_unstale":"2022-09-10T04:27:09.429942+0000","last_undegraded":"2022-09-10T04:27:09.429942+0000","last_fullsized":"2022-09-10T04:27:09.429942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:16:01.235524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422082+0000","last_change":"2022-09-10T04:26:36.603544+0000","last_active":"2022-09-10T04:27:09.422082+0000","last_peered":"2022-09-10T04:27:09.422082+0000","last_clean":"2022-09-10T04:27:09.422082+0000","last_became_active":"2022-09-10T04:26:36.602905+0000","last_became_peered":"2022-09-10T04:26:36.602905+0000","last_unstale":"2022-09-10T04:27:09.422082+0000","last_undegraded":"2022-09-10T04:27:09.422082+0000","last_fullsized":"2022-09-10T04:27:09.422082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:17:44.678651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424219+0000","last_change":"2022-09-10T04:26:31.551550+0000","last_active":"2022-09-10T04:27:09.424219+0000","last_peered":"2022-09-10T04:27:09.424219+0000","last_clean":"2022-09-10T04:27:09.424219+0000","last_became_active":"2022-09-10T04:26:31.551161+0000","last_became_peered":"2022-09-10T04:26:31.551161+0000","last_unstale":"2022-09-10T04:27:09.424219+0000","last_undegraded":"2022-09-10T04:27:09.424219+0000","last_fullsized":"2022-09-10T04:27:09.424219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:59:32.430914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425021+0000","last_change":"2022-09-10T04:26:32.122935+0000","last_active":"2022-09-10T04:27:09.425021+0000","last_peered":"2022-09-10T04:27:09.425021+0000","last_clean":"2022-09-10T04:27:09.425021+0000","last_became_active":"2022-09-10T04:26:32.122471+0000","last_became_peered":"2022-09-10T04:26:32.122471+0000","last_unstale":"2022-09-10T04:27:09.425021+0000","last_undegraded":"2022-09-10T04:27:09.425021+0000","last_fullsized":"2022-09-10T04:27:09.425021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:29:29.965140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420594+0000","last_change":"2022-09-10T04:26:36.604924+0000","last_active":"2022-09-10T04:27:09.420594+0000","last_peered":"2022-09-10T04:27:09.420594+0000","last_clean":"2022-09-10T04:27:09.420594+0000","last_became_active":"2022-09-10T04:26:36.604620+0000","last_became_peered":"2022-09-10T04:26:36.604620+0000","last_unstale":"2022-09-10T04:27:09.420594+0000","last_undegraded":"2022-09-10T04:27:09.420594+0000","last_fullsized":"2022-09-10T04:27:09.420594+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:41:34.668107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423261+0000","last_change":"2022-09-10T04:26:31.547564+0000","last_active":"2022-09-10T04:27:09.423261+0000","last_peered":"2022-09-10T04:27:09.423261+0000","last_clean":"2022-09-10T04:27:09.423261+0000","last_became_active":"2022-09-10T04:26:31.547180+0000","last_became_peered":"2022-09-10T04:26:31.547180+0000","last_unstale":"2022-09-10T04:27:09.423261+0000","last_undegraded":"2022-09-10T04:27:09.423261+0000","last_fullsized":"2022-09-10T04:27:09.423261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:23:12.454750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426158+0000","last_change":"2022-09-10T04:26:32.119307+0000","last_active":"2022-09-10T04:27:09.426158+0000","last_peered":"2022-09-10T04:27:09.426158+0000","last_clean":"2022-09-10T04:27:09.426158+0000","last_became_active":"2022-09-10T04:26:32.118995+0000","last_became_peered":"2022-09-10T04:26:32.118995+0000","last_unstale":"2022-09-10T04:27:09.426158+0000","last_undegraded":"2022-09-10T04:27:09.426158+0000","last_fullsized":"2022-09-10T04:27:09.426158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:25.888725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418383+0000","last_change":"2022-09-10T04:26:36.591310+0000","last_active":"2022-09-10T04:27:09.418383+0000","last_peered":"2022-09-10T04:27:09.418383+0000","last_clean":"2022-09-10T04:27:09.418383+0000","last_became_active":"2022-09-10T04:26:36.590907+0000","last_became_peered":"2022-09-10T04:26:36.590907+0000","last_unstale":"2022-09-10T04:27:09.418383+0000","last_undegraded":"2022-09-10T04:27:09.418383+0000","last_fullsized":"2022-09-10T04:27:09.418383+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:05:45.016925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":66,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:38.243206+0000","last_change":"2022-09-10T04:26:31.550524+0000","last_active":"2022-09-10T04:27:38.243206+0000","last_peered":"2022-09-10T04:27:38.243206+0000","last_clean":"2022-09-10T04:27:38.243206+0000","last_became_active":"2022-09-10T04:26:31.550259+0000","last_became_peered":"2022-09-10T04:26:31.550259+0000","last_unstale":"2022-09-10T04:27:38.243206+0000","last_undegraded":"2022-09-10T04:27:38.243206+0000","last_fullsized":"2022-09-10T04:27:38.243206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:53.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423230+0000","last_change":"2022-09-10T04:26:32.123377+0000","last_active":"2022-09-10T04:27:09.423230+0000","last_peered":"2022-09-10T04:27:09.423230+0000","last_clean":"2022-09-10T04:27:09.423230+0000","last_became_active":"2022-09-10T04:26:32.123076+0000","last_became_peered":"2022-09-10T04:26:32.123076+0000","last_unstale":"2022-09-10T04:27:09.423230+0000","last_undegraded":"2022-09-10T04:27:09.423230+0000","last_fullsized":"2022-09-10T04:27:09.423230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:40.425861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418231+0000","last_change":"2022-09-10T04:26:36.593491+0000","last_active":"2022-09-10T04:27:09.418231+0000","last_peered":"2022-09-10T04:27:09.418231+0000","last_clean":"2022-09-10T04:27:09.418231+0000","last_became_active":"2022-09-10T04:26:36.593064+0000","last_became_peered":"2022-09-10T04:26:36.593064+0000","last_unstale":"2022-09-10T04:27:09.418231+0000","last_undegraded":"2022-09-10T04:27:09.418231+0000","last_fullsized":"2022-09-10T04:27:09.418231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:03:44.752739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424560+0000","last_change":"2022-09-10T04:26:31.555492+0000","last_active":"2022-09-10T04:27:09.424560+0000","last_peered":"2022-09-10T04:27:09.424560+0000","last_clean":"2022-09-10T04:27:09.424560+0000","last_became_active":"2022-09-10T04:26:31.555216+0000","last_became_peered":"2022-09-10T04:26:31.555216+0000","last_unstale":"2022-09-10T04:27:09.424560+0000","last_undegraded":"2022-09-10T04:27:09.424560+0000","last_fullsized":"2022-09-10T04:27:09.424560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:50:32.775631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427293+0000","last_change":"2022-09-10T04:26:32.139851+0000","last_active":"2022-09-10T04:27:09.427293+0000","last_peered":"2022-09-10T04:27:09.427293+0000","last_clean":"2022-09-10T04:27:09.427293+0000","last_became_active":"2022-09-10T04:26:32.139515+0000","last_became_peered":"2022-09-10T04:26:32.139515+0000","last_unstale":"2022-09-10T04:27:09.427293+0000","last_undegraded":"2022-09-10T04:27:09.427293+0000","last_fullsized":"2022-09-10T04:27:09.427293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:46:04.342743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421704+0000","last_change":"2022-09-10T04:26:36.598406+0000","last_active":"2022-09-10T04:27:09.421704+0000","last_peered":"2022-09-10T04:27:09.421704+0000","last_clean":"2022-09-10T04:27:09.421704+0000","last_became_active":"2022-09-10T04:26:36.598118+0000","last_became_peered":"2022-09-10T04:26:36.598118+0000","last_unstale":"2022-09-10T04:27:09.421704+0000","last_undegraded":"2022-09-10T04:27:09.421704+0000","last_fullsized":"2022-09-10T04:27:09.421704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:35.903423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.428122+0000","last_change":"2022-09-10T04:26:31.541593+0000","last_active":"2022-09-10T04:27:09.428122+0000","last_peered":"2022-09-10T04:27:09.428122+0000","last_clean":"2022-09-10T04:27:09.428122+0000","last_became_active":"2022-09-10T04:26:31.541236+0000","last_became_peered":"2022-09-10T04:26:31.541236+0000","last_unstale":"2022-09-10T04:27:09.428122+0000","last_undegraded":"2022-09-10T04:27:09.428122+0000","last_fullsized":"2022-09-10T04:27:09.428122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:30:03.723483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423245+0000","last_change":"2022-09-10T04:26:32.147698+0000","last_active":"2022-09-10T04:27:09.423245+0000","last_peered":"2022-09-10T04:27:09.423245+0000","last_clean":"2022-09-10T04:27:09.423245+0000","last_became_active":"2022-09-10T04:26:32.147334+0000","last_became_peered":"2022-09-10T04:26:32.147334+0000","last_unstale":"2022-09-10T04:27:09.423245+0000","last_undegraded":"2022-09-10T04:27:09.423245+0000","last_fullsized":"2022-09-10T04:27:09.423245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:33:21.907371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429661+0000","last_change":"2022-09-10T04:26:36.597450+0000","last_active":"2022-09-10T04:27:09.429661+0000","last_peered":"2022-09-10T04:27:09.429661+0000","last_clean":"2022-09-10T04:27:09.429661+0000","last_became_active":"2022-09-10T04:26:36.597087+0000","last_became_peered":"2022-09-10T04:26:36.597087+0000","last_unstale":"2022-09-10T04:27:09.429661+0000","last_undegraded":"2022-09-10T04:27:09.429661+0000","last_fullsized":"2022-09-10T04:27:09.429661+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:27.788857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425510+0000","last_change":"2022-09-10T04:26:31.550451+0000","last_active":"2022-09-10T04:27:09.425510+0000","last_peered":"2022-09-10T04:27:09.425510+0000","last_clean":"2022-09-10T04:27:09.425510+0000","last_became_active":"2022-09-10T04:26:31.550158+0000","last_became_peered":"2022-09-10T04:26:31.550158+0000","last_unstale":"2022-09-10T04:27:09.425510+0000","last_undegraded":"2022-09-10T04:27:09.425510+0000","last_fullsized":"2022-09-10T04:27:09.425510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:21:27.132856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419212+0000","last_change":"2022-09-10T04:26:32.136694+0000","last_active":"2022-09-10T04:27:09.419212+0000","last_peered":"2022-09-10T04:27:09.419212+0000","last_clean":"2022-09-10T04:27:09.419212+0000","last_became_active":"2022-09-10T04:26:32.136398+0000","last_became_peered":"2022-09-10T04:26:32.136398+0000","last_unstale":"2022-09-10T04:27:09.419212+0000","last_undegraded":"2022-09-10T04:27:09.419212+0000","last_fullsized":"2022-09-10T04:27:09.419212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:32:22.413618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423754+0000","last_change":"2022-09-10T04:26:36.599433+0000","last_active":"2022-09-10T04:27:09.423754+0000","last_peered":"2022-09-10T04:27:09.423754+0000","last_clean":"2022-09-10T04:27:09.423754+0000","last_became_active":"2022-09-10T04:26:36.599013+0000","last_became_peered":"2022-09-10T04:26:36.599013+0000","last_unstale":"2022-09-10T04:27:09.423754+0000","last_undegraded":"2022-09-10T04:27:09.423754+0000","last_fullsized":"2022-09-10T04:27:09.423754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:23:18.397219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":106,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:39.243969+0000","last_change":"2022-09-10T04:26:31.548631+0000","last_active":"2022-09-10T04:27:39.243969+0000","last_peered":"2022-09-10T04:27:39.243969+0000","last_clean":"2022-09-10T04:27:39.243969+0000","last_became_active":"2022-09-10T04:26:31.548292+0000","last_became_peered":"2022-09-10T04:26:31.548292+0000","last_unstale":"2022-09-10T04:27:39.243969+0000","last_undegraded":"2022-09-10T04:27:39.243969+0000","last_fullsized":"2022-09-10T04:27:39.243969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:40.825422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425167+0000","last_change":"2022-09-10T04:26:32.124469+0000","last_active":"2022-09-10T04:27:09.425167+0000","last_peered":"2022-09-10T04:27:09.425167+0000","last_clean":"2022-09-10T04:27:09.425167+0000","last_became_active":"2022-09-10T04:26:32.124084+0000","last_became_peered":"2022-09-10T04:26:32.124084+0000","last_unstale":"2022-09-10T04:27:09.425167+0000","last_undegraded":"2022-09-10T04:27:09.425167+0000","last_fullsized":"2022-09-10T04:27:09.425167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:39:04.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419136+0000","last_change":"2022-09-10T04:26:36.601607+0000","last_active":"2022-09-10T04:27:09.419136+0000","last_peered":"2022-09-10T04:27:09.419136+0000","last_clean":"2022-09-10T04:27:09.419136+0000","last_became_active":"2022-09-10T04:26:36.601191+0000","last_became_peered":"2022-09-10T04:26:36.601191+0000","last_unstale":"2022-09-10T04:27:09.419136+0000","last_undegraded":"2022-09-10T04:27:09.419136+0000","last_fullsized":"2022-09-10T04:27:09.419136+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:52:58.815905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427014+0000","last_change":"2022-09-10T04:26:31.548524+0000","last_active":"2022-09-10T04:27:09.427014+0000","last_peered":"2022-09-10T04:27:09.427014+0000","last_clean":"2022-09-10T04:27:09.427014+0000","last_became_active":"2022-09-10T04:26:31.548073+0000","last_became_peered":"2022-09-10T04:26:31.548073+0000","last_unstale":"2022-09-10T04:27:09.427014+0000","last_undegraded":"2022-09-10T04:27:09.427014+0000","last_fullsized":"2022-09-10T04:27:09.427014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:01:02.489915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740368+0000","last_change":"2022-09-10T04:27:10.740368+0000","last_active":"2022-09-10T04:27:10.740368+0000","last_peered":"2022-09-10T04:27:10.740368+0000","last_clean":"2022-09-10T04:27:10.740368+0000","last_became_active":"2022-09-10T04:27:10.739903+0000","last_became_peered":"2022-09-10T04:27:10.739903+0000","last_unstale":"2022-09-10T04:27:10.740368+0000","last_undegraded":"2022-09-10T04:27:10.740368+0000","last_fullsized":"2022-09-10T04:27:10.740368+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:59:30.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.427026+0000","last_change":"2022-09-10T04:27:10.427026+0000","last_active":"2022-09-10T04:27:10.427026+0000","last_peered":"2022-09-10T04:27:10.427026+0000","last_clean":"2022-09-10T04:27:10.427026+0000","last_became_active":"2022-09-10T04:27:10.426556+0000","last_became_peered":"2022-09-10T04:27:10.426556+0000","last_unstale":"2022-09-10T04:27:10.427026+0000","last_undegraded":"2022-09-10T04:27:10.427026+0000","last_fullsized":"2022-09-10T04:27:10.427026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:34:43.204361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425683+0000","last_change":"2022-09-10T04:26:31.557753+0000","last_active":"2022-09-10T04:27:09.425683+0000","last_peered":"2022-09-10T04:27:09.425683+0000","last_clean":"2022-09-10T04:27:09.425683+0000","last_became_active":"2022-09-10T04:26:31.557432+0000","last_became_peered":"2022-09-10T04:26:31.557432+0000","last_unstale":"2022-09-10T04:27:09.425683+0000","last_undegraded":"2022-09-10T04:27:09.425683+0000","last_fullsized":"2022-09-10T04:27:09.425683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:14:51.954481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740377+0000","last_change":"2022-09-10T04:27:10.740377+0000","last_active":"2022-09-10T04:27:10.740377+0000","last_peered":"2022-09-10T04:27:10.740377+0000","last_clean":"2022-09-10T04:27:10.740377+0000","last_became_active":"2022-09-10T04:27:10.739989+0000","last_became_peered":"2022-09-10T04:27:10.739989+0000","last_unstale":"2022-09-10T04:27:10.740377+0000","last_undegraded":"2022-09-10T04:27:10.740377+0000","last_fullsized":"2022-09-10T04:27:10.740377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:46:22.492574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419267+0000","last_change":"2022-09-10T04:26:36.591159+0000","last_active":"2022-09-10T04:27:09.419267+0000","last_peered":"2022-09-10T04:27:09.419267+0000","last_clean":"2022-09-10T04:27:09.419267+0000","last_became_active":"2022-09-10T04:26:36.590788+0000","last_became_peered":"2022-09-10T04:26:36.590788+0000","last_unstale":"2022-09-10T04:27:09.419267+0000","last_undegraded":"2022-09-10T04:27:09.419267+0000","last_fullsized":"2022-09-10T04:27:09.419267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:50:34.597783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'14","reported_seq":118,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:39.243663+0000","last_change":"2022-09-10T04:26:31.547702+0000","last_active":"2022-09-10T04:27:39.243663+0000","last_peered":"2022-09-10T04:27:39.243663+0000","last_clean":"2022-09-10T04:27:39.243663+0000","last_became_active":"2022-09-10T04:26:31.547388+0000","last_became_peered":"2022-09-10T04:26:31.547388+0000","last_unstale":"2022-09-10T04:27:39.243663+0000","last_undegraded":"2022-09-10T04:27:39.243663+0000","last_fullsized":"2022-09-10T04:27:39.243663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:42:28.378515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3749,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":86,"num_read_kb":47,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424056+0000","last_change":"2022-09-10T04:26:32.117260+0000","last_active":"2022-09-10T04:27:09.424056+0000","last_peered":"2022-09-10T04:27:09.424056+0000","last_clean":"2022-09-10T04:27:09.424056+0000","last_became_active":"2022-09-10T04:26:32.116928+0000","last_became_peered":"2022-09-10T04:26:32.116928+0000","last_unstale":"2022-09-10T04:27:09.424056+0000","last_undegraded":"2022-09-10T04:27:09.424056+0000","last_fullsized":"2022-09-10T04:27:09.424056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:00:10.528509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426431+0000","last_change":"2022-09-10T04:26:36.600239+0000","last_active":"2022-09-10T04:27:09.426431+0000","last_peered":"2022-09-10T04:27:09.426431+0000","last_clean":"2022-09-10T04:27:09.426431+0000","last_became_active":"2022-09-10T04:26:36.599614+0000","last_became_peered":"2022-09-10T04:26:36.599614+0000","last_unstale":"2022-09-10T04:27:09.426431+0000","last_undegraded":"2022-09-10T04:27:09.426431+0000","last_fullsized":"2022-09-10T04:27:09.426431+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:36.535130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418087+0000","last_change":"2022-09-10T04:26:31.552412+0000","last_active":"2022-09-10T04:27:09.418087+0000","last_peered":"2022-09-10T04:27:09.418087+0000","last_clean":"2022-09-10T04:27:09.418087+0000","last_became_active":"2022-09-10T04:26:31.552145+0000","last_became_peered":"2022-09-10T04:26:31.552145+0000","last_unstale":"2022-09-10T04:27:09.418087+0000","last_undegraded":"2022-09-10T04:27:09.418087+0000","last_fullsized":"2022-09-10T04:27:09.418087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:16:36.041950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427385+0000","last_change":"2022-09-10T04:26:32.136276+0000","last_active":"2022-09-10T04:27:09.427385+0000","last_peered":"2022-09-10T04:27:09.427385+0000","last_clean":"2022-09-10T04:27:09.427385+0000","last_became_active":"2022-09-10T04:26:32.135894+0000","last_became_peered":"2022-09-10T04:26:32.135894+0000","last_unstale":"2022-09-10T04:27:09.427385+0000","last_undegraded":"2022-09-10T04:27:09.427385+0000","last_fullsized":"2022-09-10T04:27:09.427385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:12:32.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740446+0000","last_change":"2022-09-10T04:27:10.740446+0000","last_active":"2022-09-10T04:27:10.740446+0000","last_peered":"2022-09-10T04:27:10.740446+0000","last_clean":"2022-09-10T04:27:10.740446+0000","last_became_active":"2022-09-10T04:27:10.739986+0000","last_became_peered":"2022-09-10T04:27:10.739986+0000","last_unstale":"2022-09-10T04:27:10.740446+0000","last_undegraded":"2022-09-10T04:27:10.740446+0000","last_fullsized":"2022-09-10T04:27:10.740446+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:12:17.918925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426192+0000","last_change":"2022-09-10T04:26:31.556823+0000","last_active":"2022-09-10T04:27:09.426192+0000","last_peered":"2022-09-10T04:27:09.426192+0000","last_clean":"2022-09-10T04:27:09.426192+0000","last_became_active":"2022-09-10T04:26:31.556509+0000","last_became_peered":"2022-09-10T04:26:31.556509+0000","last_unstale":"2022-09-10T04:27:09.426192+0000","last_undegraded":"2022-09-10T04:27:09.426192+0000","last_fullsized":"2022-09-10T04:27:09.426192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:35:16.227863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427136+0000","last_change":"2022-09-10T04:26:32.130565+0000","last_active":"2022-09-10T04:27:09.427136+0000","last_peered":"2022-09-10T04:27:09.427136+0000","last_clean":"2022-09-10T04:27:09.427136+0000","last_became_active":"2022-09-10T04:26:32.130121+0000","last_became_peered":"2022-09-10T04:26:32.130121+0000","last_unstale":"2022-09-10T04:27:09.427136+0000","last_undegraded":"2022-09-10T04:27:09.427136+0000","last_fullsized":"2022-09-10T04:27:09.427136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:55:03.504615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419883+0000","last_change":"2022-09-10T04:26:36.603262+0000","last_active":"2022-09-10T04:27:09.419883+0000","last_peered":"2022-09-10T04:27:09.419883+0000","last_clean":"2022-09-10T04:27:09.419883+0000","last_became_active":"2022-09-10T04:26:36.602590+0000","last_became_peered":"2022-09-10T04:26:36.602590+0000","last_unstale":"2022-09-10T04:27:09.419883+0000","last_undegraded":"2022-09-10T04:27:09.419883+0000","last_fullsized":"2022-09-10T04:27:09.419883+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:19:03.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:40.245556+0000","last_change":"2022-09-10T04:26:31.545898+0000","last_active":"2022-09-10T04:27:40.245556+0000","last_peered":"2022-09-10T04:27:40.245556+0000","last_clean":"2022-09-10T04:27:40.245556+0000","last_became_active":"2022-09-10T04:26:31.545355+0000","last_became_peered":"2022-09-10T04:26:31.545355+0000","last_unstale":"2022-09-10T04:27:40.245556+0000","last_undegraded":"2022-09-10T04:27:40.245556+0000","last_fullsized":"2022-09-10T04:27:40.245556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:50:36.187534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426475+0000","last_change":"2022-09-10T04:26:32.116268+0000","last_active":"2022-09-10T04:27:09.426475+0000","last_peered":"2022-09-10T04:27:09.426475+0000","last_clean":"2022-09-10T04:27:09.426475+0000","last_became_active":"2022-09-10T04:26:32.115133+0000","last_became_peered":"2022-09-10T04:26:32.115133+0000","last_unstale":"2022-09-10T04:27:09.426475+0000","last_undegraded":"2022-09-10T04:27:09.426475+0000","last_fullsized":"2022-09-10T04:27:09.426475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:38:05.319082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427708+0000","last_change":"2022-09-10T04:26:36.605212+0000","last_active":"2022-09-10T04:27:09.427708+0000","last_peered":"2022-09-10T04:27:09.427708+0000","last_clean":"2022-09-10T04:27:09.427708+0000","last_became_active":"2022-09-10T04:26:36.604767+0000","last_became_peered":"2022-09-10T04:26:36.604767+0000","last_unstale":"2022-09-10T04:27:09.427708+0000","last_undegraded":"2022-09-10T04:27:09.427708+0000","last_fullsized":"2022-09-10T04:27:09.427708+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:33:55.048936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422700+0000","last_change":"2022-09-10T04:26:31.561097+0000","last_active":"2022-09-10T04:27:09.422700+0000","last_peered":"2022-09-10T04:27:09.422700+0000","last_clean":"2022-09-10T04:27:09.422700+0000","last_became_active":"2022-09-10T04:26:31.560748+0000","last_became_peered":"2022-09-10T04:26:31.560748+0000","last_unstale":"2022-09-10T04:27:09.422700+0000","last_undegraded":"2022-09-10T04:27:09.422700+0000","last_fullsized":"2022-09-10T04:27:09.422700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:44:32.921350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425328+0000","last_change":"2022-09-10T04:26:32.122012+0000","last_active":"2022-09-10T04:27:09.425328+0000","last_peered":"2022-09-10T04:27:09.425328+0000","last_clean":"2022-09-10T04:27:09.425328+0000","last_became_active":"2022-09-10T04:26:32.121668+0000","last_became_peered":"2022-09-10T04:26:32.121668+0000","last_unstale":"2022-09-10T04:27:09.425328+0000","last_undegraded":"2022-09-10T04:27:09.425328+0000","last_fullsized":"2022-09-10T04:27:09.425328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:49:57.312550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425296+0000","last_change":"2022-09-10T04:26:36.594964+0000","last_active":"2022-09-10T04:27:09.425296+0000","last_peered":"2022-09-10T04:27:09.425296+0000","last_clean":"2022-09-10T04:27:09.425296+0000","last_became_active":"2022-09-10T04:26:36.594548+0000","last_became_peered":"2022-09-10T04:26:36.594548+0000","last_unstale":"2022-09-10T04:27:09.425296+0000","last_undegraded":"2022-09-10T04:27:09.425296+0000","last_fullsized":"2022-09-10T04:27:09.425296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:01.446152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423752+0000","last_change":"2022-09-10T04:26:31.549608+0000","last_active":"2022-09-10T04:27:09.423752+0000","last_peered":"2022-09-10T04:27:09.423752+0000","last_clean":"2022-09-10T04:27:09.423752+0000","last_became_active":"2022-09-10T04:26:31.549300+0000","last_became_peered":"2022-09-10T04:26:31.549300+0000","last_unstale":"2022-09-10T04:27:09.423752+0000","last_undegraded":"2022-09-10T04:27:09.423752+0000","last_fullsized":"2022-09-10T04:27:09.423752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:53:51.183777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421594+0000","last_change":"2022-09-10T04:26:32.119540+0000","last_active":"2022-09-10T04:27:09.421594+0000","last_peered":"2022-09-10T04:27:09.421594+0000","last_clean":"2022-09-10T04:27:09.421594+0000","last_became_active":"2022-09-10T04:26:32.119341+0000","last_became_peered":"2022-09-10T04:26:32.119341+0000","last_unstale":"2022-09-10T04:27:09.421594+0000","last_undegraded":"2022-09-10T04:27:09.421594+0000","last_fullsized":"2022-09-10T04:27:09.421594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:21:13.815142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423202+0000","last_change":"2022-09-10T04:26:36.790770+0000","last_active":"2022-09-10T04:27:09.423202+0000","last_peered":"2022-09-10T04:27:09.423202+0000","last_clean":"2022-09-10T04:27:09.423202+0000","last_became_active":"2022-09-10T04:26:36.790366+0000","last_became_peered":"2022-09-10T04:26:36.790366+0000","last_unstale":"2022-09-10T04:27:09.423202+0000","last_undegraded":"2022-09-10T04:27:09.423202+0000","last_fullsized":"2022-09-10T04:27:09.423202+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:40:18.004852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419529+0000","last_change":"2022-09-10T04:26:31.557145+0000","last_active":"2022-09-10T04:27:09.419529+0000","last_peered":"2022-09-10T04:27:09.419529+0000","last_clean":"2022-09-10T04:27:09.419529+0000","last_became_active":"2022-09-10T04:26:31.556805+0000","last_became_peered":"2022-09-10T04:26:31.556805+0000","last_unstale":"2022-09-10T04:27:09.419529+0000","last_undegraded":"2022-09-10T04:27:09.419529+0000","last_fullsized":"2022-09-10T04:27:09.419529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:53:41.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423637+0000","last_change":"2022-09-10T04:26:32.139758+0000","last_active":"2022-09-10T04:27:09.423637+0000","last_peered":"2022-09-10T04:27:09.423637+0000","last_clean":"2022-09-10T04:27:09.423637+0000","last_became_active":"2022-09-10T04:26:32.139469+0000","last_became_peered":"2022-09-10T04:26:32.139469+0000","last_unstale":"2022-09-10T04:27:09.423637+0000","last_undegraded":"2022-09-10T04:27:09.423637+0000","last_fullsized":"2022-09-10T04:27:09.423637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:09:52.224744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427720+0000","last_change":"2022-09-10T04:26:36.600083+0000","last_active":"2022-09-10T04:27:09.427720+0000","last_peered":"2022-09-10T04:27:09.427720+0000","last_clean":"2022-09-10T04:27:09.427720+0000","last_became_active":"2022-09-10T04:26:36.599648+0000","last_became_peered":"2022-09-10T04:26:36.599648+0000","last_unstale":"2022-09-10T04:27:09.427720+0000","last_undegraded":"2022-09-10T04:27:09.427720+0000","last_fullsized":"2022-09-10T04:27:09.427720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:03:24.897645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:41.257579+0000","last_change":"2022-09-10T04:26:31.560147+0000","last_active":"2022-09-10T04:27:41.257579+0000","last_peered":"2022-09-10T04:27:41.257579+0000","last_clean":"2022-09-10T04:27:41.257579+0000","last_became_active":"2022-09-10T04:26:31.559868+0000","last_became_peered":"2022-09-10T04:26:31.559868+0000","last_unstale":"2022-09-10T04:27:41.257579+0000","last_undegraded":"2022-09-10T04:27:41.257579+0000","last_fullsized":"2022-09-10T04:27:41.257579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:02:46.652743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422976+0000","last_change":"2022-09-10T04:26:32.125369+0000","last_active":"2022-09-10T04:27:09.422976+0000","last_peered":"2022-09-10T04:27:09.422976+0000","last_clean":"2022-09-10T04:27:09.422976+0000","last_became_active":"2022-09-10T04:26:32.125077+0000","last_became_peered":"2022-09-10T04:26:32.125077+0000","last_unstale":"2022-09-10T04:27:09.422976+0000","last_undegraded":"2022-09-10T04:27:09.422976+0000","last_fullsized":"2022-09-10T04:27:09.422976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:03:13.265143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427814+0000","last_change":"2022-09-10T04:26:36.790278+0000","last_active":"2022-09-10T04:27:09.427814+0000","last_peered":"2022-09-10T04:27:09.427814+0000","last_clean":"2022-09-10T04:27:09.427814+0000","last_became_active":"2022-09-10T04:26:36.789852+0000","last_became_peered":"2022-09-10T04:26:36.789852+0000","last_unstale":"2022-09-10T04:27:09.427814+0000","last_undegraded":"2022-09-10T04:27:09.427814+0000","last_fullsized":"2022-09-10T04:27:09.427814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:26:38.960022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425728+0000","last_change":"2022-09-10T04:26:32.119857+0000","last_active":"2022-09-10T04:27:09.425728+0000","last_peered":"2022-09-10T04:27:09.425728+0000","last_clean":"2022-09-10T04:27:09.425728+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:27:09.425728+0000","last_undegraded":"2022-09-10T04:27:09.425728+0000","last_fullsized":"2022-09-10T04:27:09.425728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:21:40.941165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:40.244940+0000","last_change":"2022-09-10T04:26:31.555699+0000","last_active":"2022-09-10T04:27:40.244940+0000","last_peered":"2022-09-10T04:27:40.244940+0000","last_clean":"2022-09-10T04:27:40.244940+0000","last_became_active":"2022-09-10T04:26:31.555337+0000","last_became_peered":"2022-09-10T04:26:31.555337+0000","last_unstale":"2022-09-10T04:27:40.244940+0000","last_undegraded":"2022-09-10T04:27:40.244940+0000","last_fullsized":"2022-09-10T04:27:40.244940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:38:23.689077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":45,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427200+0000","last_change":"2022-09-10T04:26:36.592313+0000","last_active":"2022-09-10T04:27:09.427200+0000","last_peered":"2022-09-10T04:27:09.427200+0000","last_clean":"2022-09-10T04:27:09.427200+0000","last_became_active":"2022-09-10T04:26:36.591703+0000","last_became_peered":"2022-09-10T04:26:36.591703+0000","last_unstale":"2022-09-10T04:27:09.427200+0000","last_undegraded":"2022-09-10T04:27:09.427200+0000","last_fullsized":"2022-09-10T04:27:09.427200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:19:26.704605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740776+0000","last_change":"2022-09-10T04:27:10.739699+0000","last_active":"2022-09-10T04:27:10.740776+0000","last_peered":"2022-09-10T04:27:10.740776+0000","last_clean":"2022-09-10T04:27:10.740776+0000","last_became_active":"2022-09-10T04:27:10.739321+0000","last_became_peered":"2022-09-10T04:27:10.739321+0000","last_unstale":"2022-09-10T04:27:10.740776+0000","last_undegraded":"2022-09-10T04:27:10.740776+0000","last_fullsized":"2022-09-10T04:27:10.740776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:36:46.680589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422262+0000","last_change":"2022-09-10T04:26:31.544708+0000","last_active":"2022-09-10T04:27:09.422262+0000","last_peered":"2022-09-10T04:27:09.422262+0000","last_clean":"2022-09-10T04:27:09.422262+0000","last_became_active":"2022-09-10T04:26:31.544386+0000","last_became_peered":"2022-09-10T04:26:31.544386+0000","last_unstale":"2022-09-10T04:27:09.422262+0000","last_undegraded":"2022-09-10T04:27:09.422262+0000","last_fullsized":"2022-09-10T04:27:09.422262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:07:18.444015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":379,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426280+0000","last_change":"2022-09-10T04:25:23.357461+0000","last_active":"2022-09-10T04:27:09.426280+0000","last_peered":"2022-09-10T04:27:09.426280+0000","last_clean":"2022-09-10T04:27:09.426280+0000","last_became_active":"2022-09-10T04:25:23.356951+0000","last_became_peered":"2022-09-10T04:25:23.356951+0000","last_unstale":"2022-09-10T04:27:09.426280+0000","last_undegraded":"2022-09-10T04:27:09.426280+0000","last_fullsized":"2022-09-10T04:27:09.426280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_clean_scrub_stamp":"2022-09-10T04:21:40.122270+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:01:20.972512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425953+0000","last_change":"2022-09-10T04:26:36.790925+0000","last_active":"2022-09-10T04:27:09.425953+0000","last_peered":"2022-09-10T04:27:09.425953+0000","last_clean":"2022-09-10T04:27:09.425953+0000","last_became_active":"2022-09-10T04:26:36.790332+0000","last_became_peered":"2022-09-10T04:26:36.790332+0000","last_unstale":"2022-09-10T04:27:09.425953+0000","last_undegraded":"2022-09-10T04:27:09.425953+0000","last_fullsized":"2022-09-10T04:27:09.425953+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:10:55.153959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427972+0000","last_change":"2022-09-10T04:26:32.122356+0000","last_active":"2022-09-10T04:27:09.427972+0000","last_peered":"2022-09-10T04:27:09.427972+0000","last_clean":"2022-09-10T04:27:09.427972+0000","last_became_active":"2022-09-10T04:26:32.122028+0000","last_became_peered":"2022-09-10T04:26:32.122028+0000","last_unstale":"2022-09-10T04:27:09.427972+0000","last_undegraded":"2022-09-10T04:27:09.427972+0000","last_fullsized":"2022-09-10T04:27:09.427972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:33.837184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426406+0000","last_change":"2022-09-10T04:26:31.556537+0000","last_active":"2022-09-10T04:27:09.426406+0000","last_peered":"2022-09-10T04:27:09.426406+0000","last_clean":"2022-09-10T04:27:09.426406+0000","last_became_active":"2022-09-10T04:26:31.556209+0000","last_became_peered":"2022-09-10T04:26:31.556209+0000","last_unstale":"2022-09-10T04:27:09.426406+0000","last_undegraded":"2022-09-10T04:27:09.426406+0000","last_fullsized":"2022-09-10T04:27:09.426406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:42:05.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421174+0000","last_change":"2022-09-10T04:26:36.600051+0000","last_active":"2022-09-10T04:27:09.421174+0000","last_peered":"2022-09-10T04:27:09.421174+0000","last_clean":"2022-09-10T04:27:09.421174+0000","last_became_active":"2022-09-10T04:26:36.599603+0000","last_became_peered":"2022-09-10T04:26:36.599603+0000","last_unstale":"2022-09-10T04:27:09.421174+0000","last_undegraded":"2022-09-10T04:27:09.421174+0000","last_fullsized":"2022-09-10T04:27:09.421174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:48:12.857851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424105+0000","last_change":"2022-09-10T04:26:31.560780+0000","last_active":"2022-09-10T04:27:09.424105+0000","last_peered":"2022-09-10T04:27:09.424105+0000","last_clean":"2022-09-10T04:27:09.424105+0000","last_became_active":"2022-09-10T04:26:31.560462+0000","last_became_peered":"2022-09-10T04:26:31.560462+0000","last_unstale":"2022-09-10T04:27:09.424105+0000","last_undegraded":"2022-09-10T04:27:09.424105+0000","last_fullsized":"2022-09-10T04:27:09.424105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:05:15.437095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427857+0000","last_change":"2022-09-10T04:26:32.127579+0000","last_active":"2022-09-10T04:27:09.427857+0000","last_peered":"2022-09-10T04:27:09.427857+0000","last_clean":"2022-09-10T04:27:09.427857+0000","last_became_active":"2022-09-10T04:26:32.127237+0000","last_became_peered":"2022-09-10T04:26:32.127237+0000","last_unstale":"2022-09-10T04:27:09.427857+0000","last_undegraded":"2022-09-10T04:27:09.427857+0000","last_fullsized":"2022-09-10T04:27:09.427857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:09:28.061156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423213+0000","last_change":"2022-09-10T04:26:36.790777+0000","last_active":"2022-09-10T04:27:09.423213+0000","last_peered":"2022-09-10T04:27:09.423213+0000","last_clean":"2022-09-10T04:27:09.423213+0000","last_became_active":"2022-09-10T04:26:36.789974+0000","last_became_peered":"2022-09-10T04:26:36.789974+0000","last_unstale":"2022-09-10T04:27:09.423213+0000","last_undegraded":"2022-09-10T04:27:09.423213+0000","last_fullsized":"2022-09-10T04:27:09.423213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:41:20.214040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.437497+0000","last_change":"2022-09-10T04:27:10.437497+0000","last_active":"2022-09-10T04:27:10.437497+0000","last_peered":"2022-09-10T04:27:10.437497+0000","last_clean":"2022-09-10T04:27:10.437497+0000","last_became_active":"2022-09-10T04:27:10.427029+0000","last_became_peered":"2022-09-10T04:27:10.427029+0000","last_unstale":"2022-09-10T04:27:10.437497+0000","last_undegraded":"2022-09-10T04:27:10.437497+0000","last_fullsized":"2022-09-10T04:27:10.437497+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:01:01.308164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424813+0000","last_change":"2022-09-10T04:26:32.145463+0000","last_active":"2022-09-10T04:27:09.424813+0000","last_peered":"2022-09-10T04:27:09.424813+0000","last_clean":"2022-09-10T04:27:09.424813+0000","last_became_active":"2022-09-10T04:26:32.145115+0000","last_became_peered":"2022-09-10T04:26:32.145115+0000","last_unstale":"2022-09-10T04:27:09.424813+0000","last_undegraded":"2022-09-10T04:27:09.424813+0000","last_fullsized":"2022-09-10T04:27:09.424813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:11.597005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.417822+0000","last_change":"2022-09-10T04:26:36.594514+0000","last_active":"2022-09-10T04:27:09.417822+0000","last_peered":"2022-09-10T04:27:09.417822+0000","last_clean":"2022-09-10T04:27:09.417822+0000","last_became_active":"2022-09-10T04:26:36.594092+0000","last_became_peered":"2022-09-10T04:26:36.594092+0000","last_unstale":"2022-09-10T04:27:09.417822+0000","last_undegraded":"2022-09-10T04:27:09.417822+0000","last_fullsized":"2022-09-10T04:27:09.417822+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:34:31.510362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421375+0000","last_change":"2022-09-10T04:26:31.548844+0000","last_active":"2022-09-10T04:27:09.421375+0000","last_peered":"2022-09-10T04:27:09.421375+0000","last_clean":"2022-09-10T04:27:09.421375+0000","last_became_active":"2022-09-10T04:26:31.548535+0000","last_became_peered":"2022-09-10T04:26:31.548535+0000","last_unstale":"2022-09-10T04:27:09.421375+0000","last_undegraded":"2022-09-10T04:27:09.421375+0000","last_fullsized":"2022-09-10T04:27:09.421375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:40:49.443190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429750+0000","last_change":"2022-09-10T04:26:32.136407+0000","last_active":"2022-09-10T04:27:09.429750+0000","last_peered":"2022-09-10T04:27:09.429750+0000","last_clean":"2022-09-10T04:27:09.429750+0000","last_became_active":"2022-09-10T04:26:32.136116+0000","last_became_peered":"2022-09-10T04:26:32.136116+0000","last_unstale":"2022-09-10T04:27:09.429750+0000","last_undegraded":"2022-09-10T04:27:09.429750+0000","last_fullsized":"2022-09-10T04:27:09.429750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:32:26.673896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421066+0000","last_change":"2022-09-10T04:26:36.790658+0000","last_active":"2022-09-10T04:27:09.421066+0000","last_peered":"2022-09-10T04:27:09.421066+0000","last_clean":"2022-09-10T04:27:09.421066+0000","last_became_active":"2022-09-10T04:26:36.790203+0000","last_became_peered":"2022-09-10T04:26:36.790203+0000","last_unstale":"2022-09-10T04:27:09.421066+0000","last_undegraded":"2022-09-10T04:27:09.421066+0000","last_fullsized":"2022-09-10T04:27:09.421066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:53:08.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425301+0000","last_change":"2022-09-10T04:26:31.549875+0000","last_active":"2022-09-10T04:27:09.425301+0000","last_peered":"2022-09-10T04:27:09.425301+0000","last_clean":"2022-09-10T04:27:09.425301+0000","last_became_active":"2022-09-10T04:26:31.549578+0000","last_became_peered":"2022-09-10T04:26:31.549578+0000","last_unstale":"2022-09-10T04:27:09.425301+0000","last_undegraded":"2022-09-10T04:27:09.425301+0000","last_fullsized":"2022-09-10T04:27:09.425301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:52:51.521360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422810+0000","last_change":"2022-09-10T04:26:32.111710+0000","last_active":"2022-09-10T04:27:09.422810+0000","last_peered":"2022-09-10T04:27:09.422810+0000","last_clean":"2022-09-10T04:27:09.422810+0000","last_became_active":"2022-09-10T04:26:32.111314+0000","last_became_peered":"2022-09-10T04:26:32.111314+0000","last_unstale":"2022-09-10T04:27:09.422810+0000","last_undegraded":"2022-09-10T04:27:09.422810+0000","last_fullsized":"2022-09-10T04:27:09.422810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:42.589093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420029+0000","last_change":"2022-09-10T04:26:36.596926+0000","last_active":"2022-09-10T04:27:09.420029+0000","last_peered":"2022-09-10T04:27:09.420029+0000","last_clean":"2022-09-10T04:27:09.420029+0000","last_became_active":"2022-09-10T04:26:36.596287+0000","last_became_peered":"2022-09-10T04:26:36.596287+0000","last_unstale":"2022-09-10T04:27:09.420029+0000","last_undegraded":"2022-09-10T04:27:09.420029+0000","last_fullsized":"2022-09-10T04:27:09.420029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:52:54.252320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423848+0000","last_change":"2022-09-10T04:26:31.552277+0000","last_active":"2022-09-10T04:27:09.423848+0000","last_peered":"2022-09-10T04:27:09.423848+0000","last_clean":"2022-09-10T04:27:09.423848+0000","last_became_active":"2022-09-10T04:26:31.551904+0000","last_became_peered":"2022-09-10T04:26:31.551904+0000","last_unstale":"2022-09-10T04:27:09.423848+0000","last_undegraded":"2022-09-10T04:27:09.423848+0000","last_fullsized":"2022-09-10T04:27:09.423848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:28:52.507833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420912+0000","last_change":"2022-09-10T04:26:32.116288+0000","last_active":"2022-09-10T04:27:09.420912+0000","last_peered":"2022-09-10T04:27:09.420912+0000","last_clean":"2022-09-10T04:27:09.420912+0000","last_became_active":"2022-09-10T04:26:32.115955+0000","last_became_peered":"2022-09-10T04:26:32.115955+0000","last_unstale":"2022-09-10T04:27:09.420912+0000","last_undegraded":"2022-09-10T04:27:09.420912+0000","last_fullsized":"2022-09-10T04:27:09.420912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:02:30.470550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420960+0000","last_change":"2022-09-10T04:26:36.594357+0000","last_active":"2022-09-10T04:27:09.420960+0000","last_peered":"2022-09-10T04:27:09.420960+0000","last_clean":"2022-09-10T04:27:09.420960+0000","last_became_active":"2022-09-10T04:26:36.593865+0000","last_became_peered":"2022-09-10T04:26:36.593865+0000","last_unstale":"2022-09-10T04:27:09.420960+0000","last_undegraded":"2022-09-10T04:27:09.420960+0000","last_fullsized":"2022-09-10T04:27:09.420960+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:11:14.675252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.432478+0000","last_change":"2022-09-10T04:27:10.432089+0000","last_active":"2022-09-10T04:27:10.432478+0000","last_peered":"2022-09-10T04:27:10.432478+0000","last_clean":"2022-09-10T04:27:10.432478+0000","last_became_active":"2022-09-10T04:27:10.427334+0000","last_became_peered":"2022-09-10T04:27:10.427334+0000","last_unstale":"2022-09-10T04:27:10.432478+0000","last_undegraded":"2022-09-10T04:27:10.432478+0000","last_fullsized":"2022-09-10T04:27:10.432478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:04:04.064842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424238+0000","last_change":"2022-09-10T04:26:32.145692+0000","last_active":"2022-09-10T04:27:09.424238+0000","last_peered":"2022-09-10T04:27:09.424238+0000","last_clean":"2022-09-10T04:27:09.424238+0000","last_became_active":"2022-09-10T04:26:32.144922+0000","last_became_peered":"2022-09-10T04:26:32.144922+0000","last_unstale":"2022-09-10T04:27:09.424238+0000","last_undegraded":"2022-09-10T04:27:09.424238+0000","last_fullsized":"2022-09-10T04:27:09.424238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:12:29.711076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420911+0000","last_change":"2022-09-10T04:26:36.791321+0000","last_active":"2022-09-10T04:27:09.420911+0000","last_peered":"2022-09-10T04:27:09.420911+0000","last_clean":"2022-09-10T04:27:09.420911+0000","last_became_active":"2022-09-10T04:26:36.789918+0000","last_became_peered":"2022-09-10T04:26:36.789918+0000","last_unstale":"2022-09-10T04:27:09.420911+0000","last_undegraded":"2022-09-10T04:27:09.420911+0000","last_fullsized":"2022-09-10T04:27:09.420911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:49:56.538141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423474+0000","last_change":"2022-09-10T04:26:31.552140+0000","last_active":"2022-09-10T04:27:09.423474+0000","last_peered":"2022-09-10T04:27:09.423474+0000","last_clean":"2022-09-10T04:27:09.423474+0000","last_became_active":"2022-09-10T04:26:31.551191+0000","last_became_peered":"2022-09-10T04:26:31.551191+0000","last_unstale":"2022-09-10T04:27:09.423474+0000","last_undegraded":"2022-09-10T04:27:09.423474+0000","last_fullsized":"2022-09-10T04:27:09.423474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:52.681536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426681+0000","last_change":"2022-09-10T04:26:32.125775+0000","last_active":"2022-09-10T04:27:09.426681+0000","last_peered":"2022-09-10T04:27:09.426681+0000","last_clean":"2022-09-10T04:27:09.426681+0000","last_became_active":"2022-09-10T04:26:32.125463+0000","last_became_peered":"2022-09-10T04:26:32.125463+0000","last_unstale":"2022-09-10T04:27:09.426681+0000","last_undegraded":"2022-09-10T04:27:09.426681+0000","last_fullsized":"2022-09-10T04:27:09.426681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:33:26.816695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424706+0000","last_change":"2022-09-10T04:26:36.593873+0000","last_active":"2022-09-10T04:27:09.424706+0000","last_peered":"2022-09-10T04:27:09.424706+0000","last_clean":"2022-09-10T04:27:09.424706+0000","last_became_active":"2022-09-10T04:26:36.593418+0000","last_became_peered":"2022-09-10T04:26:36.593418+0000","last_unstale":"2022-09-10T04:27:09.424706+0000","last_undegraded":"2022-09-10T04:27:09.424706+0000","last_fullsized":"2022-09-10T04:27:09.424706+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:21.834135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":65,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:37.292191+0000","last_change":"2022-09-10T04:26:31.555352+0000","last_active":"2022-09-10T04:27:37.292191+0000","last_peered":"2022-09-10T04:27:37.292191+0000","last_clean":"2022-09-10T04:27:37.292191+0000","last_became_active":"2022-09-10T04:26:31.555092+0000","last_became_peered":"2022-09-10T04:26:31.555092+0000","last_unstale":"2022-09-10T04:27:37.292191+0000","last_undegraded":"2022-09-10T04:27:37.292191+0000","last_fullsized":"2022-09-10T04:27:37.292191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:30:17.812089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1106,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427305+0000","last_change":"2022-09-10T04:26:32.115687+0000","last_active":"2022-09-10T04:27:09.427305+0000","last_peered":"2022-09-10T04:27:09.427305+0000","last_clean":"2022-09-10T04:27:09.427305+0000","last_became_active":"2022-09-10T04:26:32.115369+0000","last_became_peered":"2022-09-10T04:26:32.115369+0000","last_unstale":"2022-09-10T04:27:09.427305+0000","last_undegraded":"2022-09-10T04:27:09.427305+0000","last_fullsized":"2022-09-10T04:27:09.427305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:59:31.105825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429329+0000","last_change":"2022-09-10T04:26:36.602214+0000","last_active":"2022-09-10T04:27:09.429329+0000","last_peered":"2022-09-10T04:27:09.429329+0000","last_clean":"2022-09-10T04:27:09.429329+0000","last_became_active":"2022-09-10T04:26:36.601785+0000","last_became_peered":"2022-09-10T04:26:36.601785+0000","last_unstale":"2022-09-10T04:27:09.429329+0000","last_undegraded":"2022-09-10T04:27:09.429329+0000","last_fullsized":"2022-09-10T04:27:09.429329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:52:42.795971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426294+0000","last_change":"2022-09-10T04:26:31.556182+0000","last_active":"2022-09-10T04:27:09.426294+0000","last_peered":"2022-09-10T04:27:09.426294+0000","last_clean":"2022-09-10T04:27:09.426294+0000","last_became_active":"2022-09-10T04:26:31.555873+0000","last_became_peered":"2022-09-10T04:26:31.555873+0000","last_unstale":"2022-09-10T04:27:09.426294+0000","last_undegraded":"2022-09-10T04:27:09.426294+0000","last_fullsized":"2022-09-10T04:27:09.426294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:56:49.406476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427147+0000","last_change":"2022-09-10T04:26:32.134925+0000","last_active":"2022-09-10T04:27:09.427147+0000","last_peered":"2022-09-10T04:27:09.427147+0000","last_clean":"2022-09-10T04:27:09.427147+0000","last_became_active":"2022-09-10T04:26:32.134581+0000","last_became_peered":"2022-09-10T04:26:32.134581+0000","last_unstale":"2022-09-10T04:27:09.427147+0000","last_undegraded":"2022-09-10T04:27:09.427147+0000","last_fullsized":"2022-09-10T04:27:09.427147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:06:09.815252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425087+0000","last_change":"2022-09-10T04:26:36.590902+0000","last_active":"2022-09-10T04:27:09.425087+0000","last_peered":"2022-09-10T04:27:09.425087+0000","last_clean":"2022-09-10T04:27:09.425087+0000","last_became_active":"2022-09-10T04:26:36.590468+0000","last_became_peered":"2022-09-10T04:26:36.590468+0000","last_unstale":"2022-09-10T04:27:09.425087+0000","last_undegraded":"2022-09-10T04:27:09.425087+0000","last_fullsized":"2022-09-10T04:27:09.425087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:49:40.841273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427621+0000","last_change":"2022-09-10T04:26:31.554090+0000","last_active":"2022-09-10T04:27:09.427621+0000","last_peered":"2022-09-10T04:27:09.427621+0000","last_clean":"2022-09-10T04:27:09.427621+0000","last_became_active":"2022-09-10T04:26:31.553675+0000","last_became_peered":"2022-09-10T04:26:31.553675+0000","last_unstale":"2022-09-10T04:27:09.427621+0000","last_undegraded":"2022-09-10T04:27:09.427621+0000","last_fullsized":"2022-09-10T04:27:09.427621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:35:33.619937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427241+0000","last_change":"2022-09-10T04:26:32.118923+0000","last_active":"2022-09-10T04:27:09.427241+0000","last_peered":"2022-09-10T04:27:09.427241+0000","last_clean":"2022-09-10T04:27:09.427241+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:27:09.427241+0000","last_undegraded":"2022-09-10T04:27:09.427241+0000","last_fullsized":"2022-09-10T04:27:09.427241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:03.377179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420987+0000","last_change":"2022-09-10T04:26:32.115339+0000","last_active":"2022-09-10T04:27:09.420987+0000","last_peered":"2022-09-10T04:27:09.420987+0000","last_clean":"2022-09-10T04:27:09.420987+0000","last_became_active":"2022-09-10T04:26:32.115044+0000","last_became_peered":"2022-09-10T04:26:32.115044+0000","last_unstale":"2022-09-10T04:27:09.420987+0000","last_undegraded":"2022-09-10T04:27:09.420987+0000","last_fullsized":"2022-09-10T04:27:09.420987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:06:39.719939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418101+0000","last_change":"2022-09-10T04:26:32.121701+0000","last_active":"2022-09-10T04:27:09.418101+0000","last_peered":"2022-09-10T04:27:09.418101+0000","last_clean":"2022-09-10T04:27:09.418101+0000","last_became_active":"2022-09-10T04:26:32.121417+0000","last_became_peered":"2022-09-10T04:26:32.121417+0000","last_unstale":"2022-09-10T04:27:09.418101+0000","last_undegraded":"2022-09-10T04:27:09.418101+0000","last_fullsized":"2022-09-10T04:27:09.418101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:16:44.194677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.417864+0000","last_change":"2022-09-10T04:26:32.139762+0000","last_active":"2022-09-10T04:27:09.417864+0000","last_peered":"2022-09-10T04:27:09.417864+0000","last_clean":"2022-09-10T04:27:09.417864+0000","last_became_active":"2022-09-10T04:26:32.139468+0000","last_became_peered":"2022-09-10T04:26:32.139468+0000","last_unstale":"2022-09-10T04:27:09.417864+0000","last_undegraded":"2022-09-10T04:27:09.417864+0000","last_fullsized":"2022-09-10T04:27:09.417864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:37:25.448977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429667+0000","last_change":"2022-09-10T04:26:32.131899+0000","last_active":"2022-09-10T04:27:09.429667+0000","last_peered":"2022-09-10T04:27:09.429667+0000","last_clean":"2022-09-10T04:27:09.429667+0000","last_became_active":"2022-09-10T04:26:32.130047+0000","last_became_peered":"2022-09-10T04:26:32.130047+0000","last_unstale":"2022-09-10T04:27:09.429667+0000","last_undegraded":"2022-09-10T04:27:09.429667+0000","last_fullsized":"2022-09-10T04:27:09.429667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:49:48.001008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427928+0000","last_change":"2022-09-10T04:26:32.126445+0000","last_active":"2022-09-10T04:27:09.427928+0000","last_peered":"2022-09-10T04:27:09.427928+0000","last_clean":"2022-09-10T04:27:09.427928+0000","last_became_active":"2022-09-10T04:26:32.125935+0000","last_became_peered":"2022-09-10T04:26:32.125935+0000","last_unstale":"2022-09-10T04:27:09.427928+0000","last_undegraded":"2022-09-10T04:27:09.427928+0000","last_fullsized":"2022-09-10T04:27:09.427928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:27:44.987973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.427571+0000","last_change":"2022-09-10T04:27:10.427571+0000","last_active":"2022-09-10T04:27:10.427571+0000","last_peered":"2022-09-10T04:27:10.427571+0000","last_clean":"2022-09-10T04:27:10.427571+0000","last_became_active":"2022-09-10T04:27:10.427153+0000","last_became_peered":"2022-09-10T04:27:10.427153+0000","last_unstale":"2022-09-10T04:27:10.427571+0000","last_undegraded":"2022-09-10T04:27:10.427571+0000","last_fullsized":"2022-09-10T04:27:10.427571+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:23:54.710033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418703+0000","last_change":"2022-09-10T04:26:32.122790+0000","last_active":"2022-09-10T04:27:09.418703+0000","last_peered":"2022-09-10T04:27:09.418703+0000","last_clean":"2022-09-10T04:27:09.418703+0000","last_became_active":"2022-09-10T04:26:32.122379+0000","last_became_peered":"2022-09-10T04:26:32.122379+0000","last_unstale":"2022-09-10T04:27:09.418703+0000","last_undegraded":"2022-09-10T04:27:09.418703+0000","last_fullsized":"2022-09-10T04:27:09.418703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:36:19.747563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424872+0000","last_change":"2022-09-10T04:26:32.145947+0000","last_active":"2022-09-10T04:27:09.424872+0000","last_peered":"2022-09-10T04:27:09.424872+0000","last_clean":"2022-09-10T04:27:09.424872+0000","last_became_active":"2022-09-10T04:26:32.145570+0000","last_became_peered":"2022-09-10T04:26:32.145570+0000","last_unstale":"2022-09-10T04:27:09.424872+0000","last_undegraded":"2022-09-10T04:27:09.424872+0000","last_fullsized":"2022-09-10T04:27:09.424872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:38:56.028983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425606+0000","last_change":"2022-09-10T04:26:32.140019+0000","last_active":"2022-09-10T04:27:09.425606+0000","last_peered":"2022-09-10T04:27:09.425606+0000","last_clean":"2022-09-10T04:27:09.425606+0000","last_became_active":"2022-09-10T04:26:32.139709+0000","last_became_peered":"2022-09-10T04:26:32.139709+0000","last_unstale":"2022-09-10T04:27:09.425606+0000","last_undegraded":"2022-09-10T04:27:09.425606+0000","last_fullsized":"2022-09-10T04:27:09.425606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:56:11.850506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424959+0000","last_change":"2022-09-10T04:26:32.145867+0000","last_active":"2022-09-10T04:27:09.424959+0000","last_peered":"2022-09-10T04:27:09.424959+0000","last_clean":"2022-09-10T04:27:09.424959+0000","last_became_active":"2022-09-10T04:26:32.145582+0000","last_became_peered":"2022-09-10T04:26:32.145582+0000","last_unstale":"2022-09-10T04:27:09.424959+0000","last_undegraded":"2022-09-10T04:27:09.424959+0000","last_fullsized":"2022-09-10T04:27:09.424959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:30:11.784617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425612+0000","last_change":"2022-09-10T04:26:32.134641+0000","last_active":"2022-09-10T04:27:09.425612+0000","last_peered":"2022-09-10T04:27:09.425612+0000","last_clean":"2022-09-10T04:27:09.425612+0000","last_became_active":"2022-09-10T04:26:32.134320+0000","last_became_peered":"2022-09-10T04:26:32.134320+0000","last_unstale":"2022-09-10T04:27:09.425612+0000","last_undegraded":"2022-09-10T04:27:09.425612+0000","last_fullsized":"2022-09-10T04:27:09.425612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:58:29.800916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427023+0000","last_change":"2022-09-10T04:26:32.126597+0000","last_active":"2022-09-10T04:27:09.427023+0000","last_peered":"2022-09-10T04:27:09.427023+0000","last_clean":"2022-09-10T04:27:09.427023+0000","last_became_active":"2022-09-10T04:26:32.125895+0000","last_became_peered":"2022-09-10T04:26:32.125895+0000","last_unstale":"2022-09-10T04:27:09.427023+0000","last_undegraded":"2022-09-10T04:27:09.427023+0000","last_fullsized":"2022-09-10T04:27:09.427023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:17:41.456304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422818+0000","last_change":"2022-09-10T04:26:32.122525+0000","last_active":"2022-09-10T04:27:09.422818+0000","last_peered":"2022-09-10T04:27:09.422818+0000","last_clean":"2022-09-10T04:27:09.422818+0000","last_became_active":"2022-09-10T04:26:32.122113+0000","last_became_peered":"2022-09-10T04:26:32.122113+0000","last_unstale":"2022-09-10T04:27:09.422818+0000","last_undegraded":"2022-09-10T04:27:09.422818+0000","last_fullsized":"2022-09-10T04:27:09.422818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:29:19.351147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9899,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":455,"num_read_kb":245,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":29697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351002,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6136,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739016,"statfs":{"total":95995035648,"available":95988752384,"internally_reserved":0,"allocated":1032192,"data_stored":498408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.60099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.69899999999999995}]},{"osd":5,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.754},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.70699999999999996}]},{"osd":6,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.85299999999999998}]},{"osd":7,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.76700000000000002}]},{"osd":8,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.80100000000000005}]},{"osd":9,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.871},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.82199999999999995}]},{"osd":10,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.83199999999999996}]}]},{"osd":10,"up_from":73,"seq":313532612639,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7012,"kb_used_data":1628,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738140,"statfs":{"total":95995035648,"available":95987855360,"internally_reserved":0,"allocated":1667072,"data_stored":1092746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.442},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.46999999999999997}]},{"osd":1,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.40300000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64200000000000002,"15min":0.65300000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80600000000000005,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.627},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59399999999999997,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79500000000000004,"5min":1.147,"15min":1.147},"last":0.55700000000000005}]},{"osd":3,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.67100000000000004,"15min":0.69099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"last":0.45600000000000002}]},{"osd":4,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.35899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84199999999999997,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.38900000000000001}]},{"osd":5,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70199999999999996,"15min":0.71199999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.96099999999999997,"5min":1.903,"15min":1.903},"last":0.498},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94199999999999995,"5min":1.034,"15min":1.034},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":1.056,"15min":1.121},"min":{"1min":0.49199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.1000000000000001,"15min":1.1619999999999999},"min":{"1min":0.503,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.91200000000000003,"5min":9.7349999999999994,"15min":9.7349999999999994},"last":0.51200000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.873}]},{"osd":8,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.753,"15min":0.75900000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0049999999999999,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.77100000000000002,"15min":0.78300000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.98399999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.60899999999999999}]},{"osd":9,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.89800000000000002,"15min":0.88},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":5.3289999999999997,"5min":5.3289999999999997,"15min":5.3289999999999997},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81100000000000005,"15min":0.82199999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.087,"5min":1.972,"15min":1.972},"last":0.56299999999999994}]},{"osd":11,"last update":"Sat Sep 10 04:26:44 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.63300000000000001}]}]},{"osd":9,"up_from":68,"seq":292057776163,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":1056768,"data_stored":498386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83799999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.80800000000000005}]},{"osd":1,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.69399999999999995,"15min":0.70599999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84499999999999997,"5min":1.024,"15min":1.024},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.79600000000000004,"15min":0.81799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.84099999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.86299999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.59699999999999998}]},{"osd":3,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89800000000000002,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.95399999999999996}]},{"osd":4,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.80300000000000005,"15min":0.83199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.82599999999999996,"5min":1.272,"15min":1.272},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.81100000000000005,"15min":0.83299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.85199999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.86399999999999999,"15min":0.89500000000000002},"min":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.879,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.88400000000000001,"15min":0.91400000000000003},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.85599999999999998,"5min":1.262,"15min":1.262},"last":0.60599999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.79700000000000004}]},{"osd":7,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.93700000000000006,"15min":0.97299999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.89100000000000001,"5min":1.429,"15min":1.429},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.97999999999999998,"15min":1.018},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95199999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.88100000000000001}]},{"osd":8,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.0189999999999999,"15min":1.0609999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.96899999999999997,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.96699999999999997,"15min":1.0089999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91800000000000004,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.501}]},{"osd":10,"last update":"Sat Sep 10 04:27:28 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.89000000000000001,"15min":0.91200000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.96699999999999997,"5min":1.278,"15min":1.278},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.98499999999999999,"15min":1.0189999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":0.95199999999999996,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.65600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]}]},{"osd":8,"up_from":60,"seq":257698037802,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":1052672,"data_stored":498748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.499},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.442}]},{"osd":1,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36199999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.41599999999999998}]},{"osd":2,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.441,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.624,"15min":0.61599999999999999},"min":{"1min":0.41299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.379},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.63900000000000001,"15min":0.63300000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53100000000000003}]},{"osd":5,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.88800000000000001,"5min":1.046,"15min":1.046},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.70299999999999996,"15min":0.70999999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.69299999999999995,"15min":0.69499999999999995},"min":{"1min":0.46400000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.377,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.51100000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:26:57 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.74299999999999999,"15min":0.747},"min":{"1min":0.503,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.66600000000000004}]},{"osd":10,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68999999999999995,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59699999999999998}]},{"osd":11,"last update":"Sat Sep 10 04:27:33 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.58199999999999996}]}]},{"osd":7,"up_from":52,"seq":223338299439,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93737972,"statfs":{"total":95995035648,"available":95987683328,"internally_reserved":0,"allocated":1642496,"data_stored":1090291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.77100000000000002,"15min":0.79900000000000004},"min":{"1min":0.41299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":5.3719999999999999,"15min":5.3719999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.75,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53500000000000003,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.433,"15min":0.439},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.499}]},{"osd":2,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.57199999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.78600000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.73399999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.622,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80400000000000005,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76000000000000001,"5min":0.875,"15min":0.875},"last":0.65800000000000003}]},{"osd":4,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48699999999999999,"15min":0.49399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.629},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.49399999999999999,"15min":0.51200000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67100000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81499999999999995,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83499999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.66500000000000004,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.753},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.93000000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.72499999999999998}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.68799999999999994,"15min":0.69599999999999995},"min":{"1min":0.48699999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.875,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66900000000000004,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88300000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.79200000000000004}]},{"osd":9,"last update":"Sat Sep 10 04:26:59 2022","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.83299999999999996}]},{"osd":10,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68500000000000005,"15min":0.68100000000000005},"min":{"1min":0.48399999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.91200000000000003,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.80300000000000005}]},{"osd":11,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.81599999999999995}]}]},{"osd":6,"up_from":47,"seq":201863462965,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":1036288,"data_stored":497954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51300000000000001,"15min":0.48899999999999999},"min":{"1min":0.41399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71799999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54400000000000004,"15min":0.51800000000000002},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77100000000000002,"5min":1.046,"15min":1.046},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66300000000000003,"15min":0.63900000000000001},"min":{"1min":0.51200000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.96499999999999997,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69499999999999995,"15min":0.67200000000000004},"min":{"1min":0.502,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.012,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.73499999999999999}]},{"osd":2,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.71099999999999997,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.135,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.69799999999999995}]},{"osd":3,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70799999999999996,"15min":0.68000000000000005},"min":{"1min":0.51600000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.056,"5min":1.103,"15min":1.103},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.621,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.627}]},{"osd":4,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.69799999999999995,"15min":0.68200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.98699999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":0.76100000000000001},"min":{"1min":0.63,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.97599999999999998,"5min":1.167,"15min":1.167},"last":0.89200000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.71599999999999997,"15min":0.69599999999999995},"min":{"1min":0.57899999999999996,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93999999999999995,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.72099999999999997,"15min":0.69899999999999995},"min":{"1min":0.54600000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.99299999999999999,"5min":1.036,"15min":1.036},"last":0.754}]},{"osd":7,"last update":"Sat Sep 10 04:27:05 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.76300000000000001},"min":{"1min":0.61799999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.115,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83499999999999996,"15min":0.80800000000000005},"min":{"1min":0.64900000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.171,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.80600000000000005}]},{"osd":8,"last update":"Sat Sep 10 04:27:30 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.85999999999999999,"15min":0.85299999999999998},"min":{"1min":0.67000000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.147,"5min":1.181,"15min":1.181},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73699999999999999,"15min":0.71499999999999997},"min":{"1min":0.56399999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99399999999999999,"5min":1.097,"15min":1.097},"last":0.74199999999999999}]},{"osd":9,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.0069999999999999,"15min":1.0289999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.89100000000000001,"15min":0.90200000000000002},"min":{"1min":0.66400000000000003,"5min":0.624,"15min":0.624},"max":{"1min":1.1259999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.65900000000000003}]},{"osd":10,"last update":"Sat Sep 10 04:27:24 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.71799999999999997,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.0289999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85799999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0669999999999999,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72699999999999998}]},{"osd":11,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.96399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7360,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737792,"statfs":{"total":95995035648,"available":95987499008,"internally_reserved":0,"allocated":1040384,"data_stored":502436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:29 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55400000000000005,"15min":0.52300000000000002},"min":{"1min":0.44900000000000001,"5min":0.314,"15min":0.312},"max":{"1min":0.85299999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.52900000000000003},"min":{"1min":0.435,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45900000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:26:57 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.48099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81399999999999995,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.621},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.55100000000000005,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83499999999999996,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.50800000000000001}]},{"osd":3,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.39000000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.63600000000000001,"15min":0.60199999999999998},"min":{"1min":0.48799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Sep 10 04:26:41 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.42599999999999999,"15min":0.40500000000000003},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70199999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.32400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.434}]},{"osd":5,"last update":"Sat Sep 10 04:27:09 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.51200000000000001},"min":{"1min":0.46100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91100000000000003,"5min":1.577,"15min":1.577},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.39500000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91700000000000004,"5min":1.583,"15min":1.583},"last":0.60399999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.65800000000000003,"15min":0.625},"min":{"1min":0.45700000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Sep 10 04:27:00 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.57999999999999996,"15min":0.56699999999999995},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.42399999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.66100000000000003,"15min":0.621},"min":{"1min":0.497,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98799999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.68200000000000005,"15min":0.65400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.97499999999999998,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.70099999999999996}]},{"osd":9,"last update":"Sat Sep 10 04:26:57 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.88,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.68999999999999995}]},{"osd":10,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96499999999999997,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.51400000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.56100000000000005}]},{"osd":11,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.71599999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":1060864,"data_stored":499378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 10 04:27:20 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.58199999999999996,"15min":0.84399999999999997},"min":{"1min":0.39500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":8.3729999999999993},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.49299999999999999}]},{"osd":2,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0209999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.88300000000000001},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96699999999999997,"5min":8.3260000000000005,"15min":8.3260000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:27:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.65000000000000002}]},{"osd":4,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63100000000000001,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.628,"15min":0.58599999999999997},"min":{"1min":0.435,"5min":0.372,"15min":0.372},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:27:02 2022","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.66000000000000003,"15min":0.61399999999999999},"min":{"1min":0.56200000000000006,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.67400000000000004,"15min":0.625},"min":{"1min":0.48299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.70999999999999996}]},{"osd":6,"last update":"Sat Sep 10 04:26:42 2022","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.65700000000000003},"min":{"1min":0.55600000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.72199999999999998,"15min":0.68700000000000006},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0329999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.81599999999999995}]},{"osd":7,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.745}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81899999999999995,"15min":0.80500000000000005},"min":{"1min":0.59899999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.81399999999999995,"15min":0.79200000000000004},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.77500000000000002}]},{"osd":9,"last update":"Sat Sep 10 04:26:56 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.88700000000000001,"15min":0.90500000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.046,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.91100000000000003,"15min":0.92700000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.099,"5min":1.21,"15min":1.21},"last":1.0049999999999999}]},{"osd":10,"last update":"Sat Sep 10 04:27:20 2022","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.91100000000000003,"15min":0.90400000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91500000000000004,"15min":0.91400000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.29,"5min":1.306,"15min":1.306},"last":0.79100000000000004}]},{"osd":11,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.85899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7444,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737708,"statfs":{"total":95995035648,"available":95987412992,"internally_reserved":0,"allocated":1060864,"data_stored":502624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68200000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33500000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72899999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.54000000000000004}]},{"osd":1,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40400000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77200000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52600000000000002,"15min":0.504},"min":{"1min":0.378,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71299999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.36599999999999999,"5min":0.311,"15min":0.311},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.372},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.497}]},{"osd":4,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.53200000000000003}]},{"osd":5,"last update":"Sat Sep 10 04:27:13 2022","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42699999999999999,"15min":0.41499999999999998},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.62,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.438},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47199999999999998,"15min":0.44400000000000001},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80500000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.35499999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:26:37 2022","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82199999999999995,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.64500000000000002,"15min":0.65500000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.84199999999999997,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.51700000000000002}]},{"osd":8,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.54700000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.622,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.47399999999999998}]},{"osd":9,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72399999999999998,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0189999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.57499999999999996}]},{"osd":10,"last update":"Sat Sep 10 04:27:23 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.70599999999999996,"15min":0.68799999999999994},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.56599999999999995}]},{"osd":11,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.61099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6728,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738424,"statfs":{"total":95995035648,"available":95988146176,"internally_reserved":0,"allocated":1048576,"data_stored":497710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.56299999999999994,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":11.026999999999999,"5min":11.026999999999999,"15min":11.026999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.497,"15min":0.45400000000000001},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.89800000000000002}]},{"osd":1,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67500000000000004,"15min":0.623},"min":{"1min":0.439,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.3919999999999999,"5min":1.494,"15min":1.494},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.65400000000000003,"15min":0.61599999999999999},"min":{"1min":0.54800000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.238,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.60299999999999998},"min":{"1min":0.46600000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69499999999999995,"15min":0.65500000000000003},"min":{"1min":0.50900000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0880000000000001,"5min":1.454,"15min":1.454},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.626},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"min":{"1min":0.45800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.95499999999999996,"5min":1.528,"15min":1.528},"last":1.1619999999999999}]},{"osd":5,"last update":"Sat Sep 10 04:27:04 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.66900000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94599999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.68500000000000005,"15min":0.67500000000000004},"min":{"1min":0.5,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93799999999999994,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.73899999999999999}]},{"osd":6,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.69499999999999995,"15min":0.65400000000000003},"min":{"1min":0.46100000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.755,"15min":0.71699999999999997},"min":{"1min":0.57799999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":1.2330000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:27:04 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.72599999999999998,"15min":0.71399999999999997},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95299999999999996,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.76000000000000001},"min":{"1min":0.57899999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0649999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.03}]},{"osd":8,"last update":"Sat Sep 10 04:26:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1060000000000001,"5min":1.208,"15min":1.208},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79200000000000004,"15min":0.78600000000000003},"min":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.099,"5min":1.139,"15min":1.139},"last":0.73299999999999998}]},{"osd":9,"last update":"Sat Sep 10 04:27:00 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.91200000000000003,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0289999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85099999999999998,"15min":0.85599999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":1.101}]},{"osd":10,"last update":"Sat Sep 10 04:27:24 2022","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.879,"15min":0.88500000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.871,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.67600000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":1.0640000000000001}]},{"osd":11,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":1},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.2010000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888129,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6592,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738560,"statfs":{"total":95995035648,"available":95988285440,"internally_reserved":0,"allocated":1040384,"data_stored":497932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:25:39 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73199999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54500000000000004,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.378,"15min":0.34399999999999997},"min":{"1min":0.245,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.71499999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39500000000000002,"15min":0.36499999999999999},"min":{"1min":0.29799999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69199999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.77100000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54600000000000004,"15min":0.52400000000000002},"min":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80800000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.59499999999999997,"15min":0.55900000000000005},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.86299999999999999,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79700000000000004,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Sep 10 04:27:09 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59099999999999997,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76300000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57099999999999995,"15min":0.55400000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Sep 10 04:26:40 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66700000000000004,"15min":0.67400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.77000000000000002,"15min":0.73699999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":5.5880000000000001,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Sep 10 04:27:03 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.38600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84299999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.44},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81599999999999995,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.39700000000000002}]},{"osd":8,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88700000000000001,"5min":1.133,"15min":1.133},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93700000000000006,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.58799999999999997}]},{"osd":9,"last update":"Sat Sep 10 04:26:58 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78500000000000003,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.80500000000000005,"15min":0.81699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91000000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.64500000000000002}]},{"osd":10,"last update":"Sat Sep 10 04:27:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94699999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84999999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.54700000000000004}]},{"osd":11,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.624},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.61399999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7320,"kb_used_data":1616,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93737832,"statfs":{"total":95995035648,"available":95987539968,"internally_reserved":0,"allocated":1654784,"data_stored":1090049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53700000000000003,"15min":0.52500000000000002},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":1.879,"15min":1.879},"last":0.52000000000000002}]},{"osd":1,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52300000000000002,"15min":0.48799999999999999},"min":{"1min":0.38,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79300000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.52800000000000002,"15min":0.48899999999999999},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70799999999999996,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.373},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.317},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.307,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42099999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72499999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73999999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54800000000000004,"15min":0.51400000000000001},"min":{"1min":0.41199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.435},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53700000000000003,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.44}]},{"osd":6,"last update":"Sat Sep 10 04:27:37 2022","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.76800000000000002,"15min":0.64900000000000002},"min":{"1min":0.504,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":7.9470000000000001,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.381,"5min":0.79600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.9029999999999996,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.52500000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.49399999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.41499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.41499999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76900000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.38900000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.57299999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.52700000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.627}]},{"osd":10,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63,"15min":0.61699999999999999},"min":{"1min":0.56299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.59899999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.51200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-10T04:27:45.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph pg dump --format=json 2022-09-10T04:27:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:45 smithi104 ceph-mon[97044]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:45 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:45.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:45 smithi081 ceph-mon[90133]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:45.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:45 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:45 smithi110 ceph-mon[97633]: from='client.25001 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:45 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' 2022-09-10T04:27:45.988 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:46.462 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:46 smithi081 ceph-mon[90133]: pgmap v340: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:27:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:46 smithi104 ceph-mon[97044]: pgmap v340: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:27:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:46 smithi110 ceph-mon[97633]: pgmap v340: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:27:47.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 -- 172.21.15.81:0/4084661083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d410ef00 msgr2=0x7ff9d41112e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:47.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/4084661083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d410ef00 0x7ff9d41112e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009b00 tx=0x7ff9cc009e10 comp rx=0 tx=0).stop 2022-09-10T04:27:47.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 -- 172.21.15.81:0/4084661083 shutdown_connections 2022-09-10T04:27:47.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/4084661083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d410ef00 0x7ff9d41112e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/4084661083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071080 0x7ff9d4071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/4084661083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d4071990 0x7ff9d410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 -- 172.21.15.81:0/4084661083 >> 172.21.15.81:0/4084661083 conn(0x7ff9d40f9a10 msgr2=0x7ff9d40fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:47.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 -- 172.21.15.81:0/4084661083 shutdown_connections 2022-09-10T04:27:47.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.533+0000 7ff9dafd9700 1 -- 172.21.15.81:0/4084661083 wait complete. 2022-09-10T04:27:47.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 Processor -- start 2022-09-10T04:27:47.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 -- start start 2022-09-10T04:27:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d4071080 0x7ff9d419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071990 0x7ff9d419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 0x7ff9d419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d419f320 con 0x7ff9d4071990 2022-09-10T04:27:47.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d419f490 con 0x7ff9d4071080 2022-09-10T04:27:47.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.535+0000 7ff9dafd9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d419f650 con 0x7ff9d410ef00 2022-09-10T04:27:47.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 0x7ff9d419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:47.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d8d75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d4071080 0x7ff9d419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d3fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071990 0x7ff9d419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 0x7ff9d419ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:51544/0 (socket says 172.21.15.81:51544) 2022-09-10T04:27:47.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 -- 172.21.15.81:0/882427656 learned_addr learned my addr 172.21.15.81:0/882427656 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:47.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 -- 172.21.15.81:0/882427656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d4071080 msgr2=0x7ff9d419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:47.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d4071080 0x7ff9d419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 -- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071990 msgr2=0x7ff9d419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:47.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071990 0x7ff9d419a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 -- 172.21.15.81:0/882427656 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc009730 con 0x7ff9d410ef00 2022-09-10T04:27:47.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d3fff700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071990 0x7ff9d419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:47.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.536+0000 7ff9d9576700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 0x7ff9d419ec90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00bb40 tx=0x7ff9cc00bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.537+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc00f7e0 con 0x7ff9d410ef00 2022-09-10T04:27:47.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.537+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d419f8d0 con 0x7ff9d410ef00 2022-09-10T04:27:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.537+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff9cc005040 con 0x7ff9d410ef00 2022-09-10T04:27:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.537+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc017660 con 0x7ff9d410ef00 2022-09-10T04:27:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.537+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d41a7120 con 0x7ff9d410ef00 2022-09-10T04:27:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.538+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d4064ea0 con 0x7ff9d410ef00 2022-09-10T04:27:47.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.540+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff9cc027020 con 0x7ff9d410ef00 2022-09-10T04:27:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.541+0000 7ff9d1ffb700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff9c4099010 0x7ff9c409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.541+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7ff9cc0c3f10 con 0x7ff9d410ef00 2022-09-10T04:27:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.541+0000 7ff9d8d75700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff9c4099010 0x7ff9c409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:47.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.542+0000 7ff9d8d75700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff9c4099010 0x7ff9c409b4c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8005fd0 tx=0x7ff9c8005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:47.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.543+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff9cc090290 con 0x7ff9d410ef00 2022-09-10T04:27:47.748 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.746+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9d405f120 con 0x7ff9c4099010 2022-09-10T04:27:47.760 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.759+0000 7ff9d1ffb700 1 -- 172.21.15.81:0/882427656 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427186 (secure 0 0 0) 0x7ff9d405f120 con 0x7ff9c4099010 2022-09-10T04:27:47.773 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:47.777 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff9c4099010 msgr2=0x7ff9c409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:47.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff9c4099010 0x7ff9c409b4c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8005fd0 tx=0x7ff9c8005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:47.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 msgr2=0x7ff9d419ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:47.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 0x7ff9d419ec90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00bb40 tx=0x7ff9cc00bb70 comp rx=0 tx=0).stop 2022-09-10T04:27:47.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 shutdown_connections 2022-09-10T04:27:47.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff9c4099010 0x7ff9c409b4c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d410ef00 0x7ff9d419ec90 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff9d4071080 0x7ff9d419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.766+0000 7ff9dafd9700 1 --2- 172.21.15.81:0/882427656 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9d4071990 0x7ff9d419a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:47.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.767+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 >> 172.21.15.81:0/882427656 conn(0x7ff9d40f9a10 msgr2=0x7ff9d40fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:47.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.767+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 shutdown_connections 2022-09-10T04:27:47.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:47.767+0000 7ff9dafd9700 1 -- 172.21.15.81:0/882427656 wait complete. 2022-09-10T04:27:47.781 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2022-09-10T04:27:48.269 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2022-09-10T04:27:47.357086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600267,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":641,"num_read_kb":389,"num_write":390,"num_write_kb":3005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":320,"ondisk_log_size":320,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82504,"kb_used_data":14056,"kb_used_omap":0,"kb_used_meta":68352,"kb_avail":1124859320,"statfs":{"total":1151940427776,"available":1151855943680,"internally_reserved":0,"allocated":14393344,"data_stored":7766662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69992448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003373"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422009+0000","last_change":"2022-09-10T04:26:32.116469+0000","last_active":"2022-09-10T04:27:09.422009+0000","last_peered":"2022-09-10T04:27:09.422009+0000","last_clean":"2022-09-10T04:27:09.422009+0000","last_became_active":"2022-09-10T04:26:32.116060+0000","last_became_peered":"2022-09-10T04:26:32.116060+0000","last_unstale":"2022-09-10T04:27:09.422009+0000","last_undegraded":"2022-09-10T04:27:09.422009+0000","last_fullsized":"2022-09-10T04:27:09.422009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:22:05.868811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429692+0000","last_change":"2022-09-10T04:26:32.114659+0000","last_active":"2022-09-10T04:27:09.429692+0000","last_peered":"2022-09-10T04:27:09.429692+0000","last_clean":"2022-09-10T04:27:09.429692+0000","last_became_active":"2022-09-10T04:26:32.114381+0000","last_became_peered":"2022-09-10T04:26:32.114381+0000","last_unstale":"2022-09-10T04:27:09.429692+0000","last_undegraded":"2022-09-10T04:27:09.429692+0000","last_fullsized":"2022-09-10T04:27:09.429692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:53:50.267695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429488+0000","last_change":"2022-09-10T04:26:32.117123+0000","last_active":"2022-09-10T04:27:09.429488+0000","last_peered":"2022-09-10T04:27:09.429488+0000","last_clean":"2022-09-10T04:27:09.429488+0000","last_became_active":"2022-09-10T04:26:32.116753+0000","last_became_peered":"2022-09-10T04:26:32.116753+0000","last_unstale":"2022-09-10T04:27:09.429488+0000","last_undegraded":"2022-09-10T04:27:09.429488+0000","last_fullsized":"2022-09-10T04:27:09.429488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:52.713788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421339+0000","last_change":"2022-09-10T04:26:32.120151+0000","last_active":"2022-09-10T04:27:09.421339+0000","last_peered":"2022-09-10T04:27:09.421339+0000","last_clean":"2022-09-10T04:27:09.421339+0000","last_became_active":"2022-09-10T04:26:32.119510+0000","last_became_peered":"2022-09-10T04:26:32.119510+0000","last_unstale":"2022-09-10T04:27:09.421339+0000","last_undegraded":"2022-09-10T04:27:09.421339+0000","last_fullsized":"2022-09-10T04:27:09.421339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:56:23.232054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423492+0000","last_change":"2022-09-10T04:26:32.138820+0000","last_active":"2022-09-10T04:27:09.423492+0000","last_peered":"2022-09-10T04:27:09.423492+0000","last_clean":"2022-09-10T04:27:09.423492+0000","last_became_active":"2022-09-10T04:26:32.138466+0000","last_became_peered":"2022-09-10T04:26:32.138466+0000","last_unstale":"2022-09-10T04:27:09.423492+0000","last_undegraded":"2022-09-10T04:27:09.423492+0000","last_fullsized":"2022-09-10T04:27:09.423492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:16:22.128598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.424838+0000","last_change":"2022-09-10T04:27:10.424838+0000","last_active":"2022-09-10T04:27:10.424838+0000","last_peered":"2022-09-10T04:27:10.424838+0000","last_clean":"2022-09-10T04:27:10.424838+0000","last_became_active":"2022-09-10T04:27:10.424425+0000","last_became_peered":"2022-09-10T04:27:10.424425+0000","last_unstale":"2022-09-10T04:27:10.424838+0000","last_undegraded":"2022-09-10T04:27:10.424838+0000","last_fullsized":"2022-09-10T04:27:10.424838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:51:59.529546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426584+0000","last_change":"2022-09-10T04:26:32.122869+0000","last_active":"2022-09-10T04:27:09.426584+0000","last_peered":"2022-09-10T04:27:09.426584+0000","last_clean":"2022-09-10T04:27:09.426584+0000","last_became_active":"2022-09-10T04:26:32.122457+0000","last_became_peered":"2022-09-10T04:26:32.122457+0000","last_unstale":"2022-09-10T04:27:09.426584+0000","last_undegraded":"2022-09-10T04:27:09.426584+0000","last_fullsized":"2022-09-10T04:27:09.426584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:08.945173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740810+0000","last_change":"2022-09-10T04:27:10.739898+0000","last_active":"2022-09-10T04:27:10.740810+0000","last_peered":"2022-09-10T04:27:10.740810+0000","last_clean":"2022-09-10T04:27:10.740810+0000","last_became_active":"2022-09-10T04:27:10.739397+0000","last_became_peered":"2022-09-10T04:27:10.739397+0000","last_unstale":"2022-09-10T04:27:10.740810+0000","last_undegraded":"2022-09-10T04:27:10.740810+0000","last_fullsized":"2022-09-10T04:27:10.740810+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:40:17.580037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425123+0000","last_change":"2022-09-10T04:26:32.113787+0000","last_active":"2022-09-10T04:27:09.425123+0000","last_peered":"2022-09-10T04:27:09.425123+0000","last_clean":"2022-09-10T04:27:09.425123+0000","last_became_active":"2022-09-10T04:26:32.113454+0000","last_became_peered":"2022-09-10T04:26:32.113454+0000","last_unstale":"2022-09-10T04:27:09.425123+0000","last_undegraded":"2022-09-10T04:27:09.425123+0000","last_fullsized":"2022-09-10T04:27:09.425123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:11:23.375628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423927+0000","last_change":"2022-09-10T04:26:32.123249+0000","last_active":"2022-09-10T04:27:09.423927+0000","last_peered":"2022-09-10T04:27:09.423927+0000","last_clean":"2022-09-10T04:27:09.423927+0000","last_became_active":"2022-09-10T04:26:32.121836+0000","last_became_peered":"2022-09-10T04:26:32.121836+0000","last_unstale":"2022-09-10T04:27:09.423927+0000","last_undegraded":"2022-09-10T04:27:09.423927+0000","last_fullsized":"2022-09-10T04:27:09.423927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:22.711515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427289+0000","last_change":"2022-09-10T04:26:32.118213+0000","last_active":"2022-09-10T04:27:09.427289+0000","last_peered":"2022-09-10T04:27:09.427289+0000","last_clean":"2022-09-10T04:27:09.427289+0000","last_became_active":"2022-09-10T04:26:32.117863+0000","last_became_peered":"2022-09-10T04:26:32.117863+0000","last_unstale":"2022-09-10T04:27:09.427289+0000","last_undegraded":"2022-09-10T04:27:09.427289+0000","last_fullsized":"2022-09-10T04:27:09.427289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:26:35.632083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421238+0000","last_change":"2022-09-10T04:26:32.127748+0000","last_active":"2022-09-10T04:27:09.421238+0000","last_peered":"2022-09-10T04:27:09.421238+0000","last_clean":"2022-09-10T04:27:09.421238+0000","last_became_active":"2022-09-10T04:26:32.127470+0000","last_became_peered":"2022-09-10T04:26:32.127470+0000","last_unstale":"2022-09-10T04:27:09.421238+0000","last_undegraded":"2022-09-10T04:27:09.421238+0000","last_fullsized":"2022-09-10T04:27:09.421238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:01:58.438614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427201+0000","last_change":"2022-09-10T04:26:32.126378+0000","last_active":"2022-09-10T04:27:09.427201+0000","last_peered":"2022-09-10T04:27:09.427201+0000","last_clean":"2022-09-10T04:27:09.427201+0000","last_became_active":"2022-09-10T04:26:32.125510+0000","last_became_peered":"2022-09-10T04:26:32.125510+0000","last_unstale":"2022-09-10T04:27:09.427201+0000","last_undegraded":"2022-09-10T04:27:09.427201+0000","last_fullsized":"2022-09-10T04:27:09.427201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:41.235347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429073+0000","last_change":"2022-09-10T04:26:32.118515+0000","last_active":"2022-09-10T04:27:09.429073+0000","last_peered":"2022-09-10T04:27:09.429073+0000","last_clean":"2022-09-10T04:27:09.429073+0000","last_became_active":"2022-09-10T04:26:32.118163+0000","last_became_peered":"2022-09-10T04:26:32.118163+0000","last_unstale":"2022-09-10T04:27:09.429073+0000","last_undegraded":"2022-09-10T04:27:09.429073+0000","last_fullsized":"2022-09-10T04:27:09.429073+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:39:42.245166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429375+0000","last_change":"2022-09-10T04:26:32.122232+0000","last_active":"2022-09-10T04:27:09.429375+0000","last_peered":"2022-09-10T04:27:09.429375+0000","last_clean":"2022-09-10T04:27:09.429375+0000","last_became_active":"2022-09-10T04:26:32.121906+0000","last_became_peered":"2022-09-10T04:26:32.121906+0000","last_unstale":"2022-09-10T04:27:09.429375+0000","last_undegraded":"2022-09-10T04:27:09.429375+0000","last_fullsized":"2022-09-10T04:27:09.429375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:49:50.711773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425182+0000","last_change":"2022-09-10T04:26:32.117413+0000","last_active":"2022-09-10T04:27:09.425182+0000","last_peered":"2022-09-10T04:27:09.425182+0000","last_clean":"2022-09-10T04:27:09.425182+0000","last_became_active":"2022-09-10T04:26:32.117120+0000","last_became_peered":"2022-09-10T04:26:32.117120+0000","last_unstale":"2022-09-10T04:27:09.425182+0000","last_undegraded":"2022-09-10T04:27:09.425182+0000","last_fullsized":"2022-09-10T04:27:09.425182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:50:28.426517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421816+0000","last_change":"2022-09-10T04:26:32.117852+0000","last_active":"2022-09-10T04:27:09.421816+0000","last_peered":"2022-09-10T04:27:09.421816+0000","last_clean":"2022-09-10T04:27:09.421816+0000","last_became_active":"2022-09-10T04:26:32.117526+0000","last_became_peered":"2022-09-10T04:26:32.117526+0000","last_unstale":"2022-09-10T04:27:09.421816+0000","last_undegraded":"2022-09-10T04:27:09.421816+0000","last_fullsized":"2022-09-10T04:27:09.421816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:02:45.194870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419780+0000","last_change":"2022-09-10T04:26:32.122937+0000","last_active":"2022-09-10T04:27:09.419780+0000","last_peered":"2022-09-10T04:27:09.419780+0000","last_clean":"2022-09-10T04:27:09.419780+0000","last_became_active":"2022-09-10T04:26:32.122549+0000","last_became_peered":"2022-09-10T04:26:32.122549+0000","last_unstale":"2022-09-10T04:27:09.419780+0000","last_undegraded":"2022-09-10T04:27:09.419780+0000","last_fullsized":"2022-09-10T04:27:09.419780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:19:48.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418617+0000","last_change":"2022-09-10T04:26:32.139841+0000","last_active":"2022-09-10T04:27:09.418617+0000","last_peered":"2022-09-10T04:27:09.418617+0000","last_clean":"2022-09-10T04:27:09.418617+0000","last_became_active":"2022-09-10T04:26:32.138962+0000","last_became_peered":"2022-09-10T04:26:32.138962+0000","last_unstale":"2022-09-10T04:27:09.418617+0000","last_undegraded":"2022-09-10T04:27:09.418617+0000","last_fullsized":"2022-09-10T04:27:09.418617+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:51.196716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.417950+0000","last_change":"2022-09-10T04:26:36.594956+0000","last_active":"2022-09-10T04:27:09.417950+0000","last_peered":"2022-09-10T04:27:09.417950+0000","last_clean":"2022-09-10T04:27:09.417950+0000","last_became_active":"2022-09-10T04:26:36.594507+0000","last_became_peered":"2022-09-10T04:26:36.594507+0000","last_unstale":"2022-09-10T04:27:09.417950+0000","last_undegraded":"2022-09-10T04:27:09.417950+0000","last_fullsized":"2022-09-10T04:27:09.417950+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:05:36.037017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:42.258223+0000","last_change":"2022-09-10T04:26:31.553482+0000","last_active":"2022-09-10T04:27:42.258223+0000","last_peered":"2022-09-10T04:27:42.258223+0000","last_clean":"2022-09-10T04:27:42.258223+0000","last_became_active":"2022-09-10T04:26:31.553170+0000","last_became_peered":"2022-09-10T04:26:31.553170+0000","last_unstale":"2022-09-10T04:27:42.258223+0000","last_undegraded":"2022-09-10T04:27:42.258223+0000","last_fullsized":"2022-09-10T04:27:42.258223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:52:35.054369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425342+0000","last_change":"2022-09-10T04:26:32.139920+0000","last_active":"2022-09-10T04:27:09.425342+0000","last_peered":"2022-09-10T04:27:09.425342+0000","last_clean":"2022-09-10T04:27:09.425342+0000","last_became_active":"2022-09-10T04:26:32.139414+0000","last_became_peered":"2022-09-10T04:26:32.139414+0000","last_unstale":"2022-09-10T04:27:09.425342+0000","last_undegraded":"2022-09-10T04:27:09.425342+0000","last_fullsized":"2022-09-10T04:27:09.425342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:12.288511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420458+0000","last_change":"2022-09-10T04:26:36.603332+0000","last_active":"2022-09-10T04:27:09.420458+0000","last_peered":"2022-09-10T04:27:09.420458+0000","last_clean":"2022-09-10T04:27:09.420458+0000","last_became_active":"2022-09-10T04:26:36.602947+0000","last_became_peered":"2022-09-10T04:26:36.602947+0000","last_unstale":"2022-09-10T04:27:09.420458+0000","last_undegraded":"2022-09-10T04:27:09.420458+0000","last_fullsized":"2022-09-10T04:27:09.420458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:20:15.790086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425618+0000","last_change":"2022-09-10T04:26:31.559001+0000","last_active":"2022-09-10T04:27:09.425618+0000","last_peered":"2022-09-10T04:27:09.425618+0000","last_clean":"2022-09-10T04:27:09.425618+0000","last_became_active":"2022-09-10T04:26:31.558675+0000","last_became_peered":"2022-09-10T04:26:31.558675+0000","last_unstale":"2022-09-10T04:27:09.425618+0000","last_undegraded":"2022-09-10T04:27:09.425618+0000","last_fullsized":"2022-09-10T04:27:09.425618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:31:20.091765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429942+0000","last_change":"2022-09-10T04:26:32.122545+0000","last_active":"2022-09-10T04:27:09.429942+0000","last_peered":"2022-09-10T04:27:09.429942+0000","last_clean":"2022-09-10T04:27:09.429942+0000","last_became_active":"2022-09-10T04:26:32.122237+0000","last_became_peered":"2022-09-10T04:26:32.122237+0000","last_unstale":"2022-09-10T04:27:09.429942+0000","last_undegraded":"2022-09-10T04:27:09.429942+0000","last_fullsized":"2022-09-10T04:27:09.429942+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:16:01.235524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422082+0000","last_change":"2022-09-10T04:26:36.603544+0000","last_active":"2022-09-10T04:27:09.422082+0000","last_peered":"2022-09-10T04:27:09.422082+0000","last_clean":"2022-09-10T04:27:09.422082+0000","last_became_active":"2022-09-10T04:26:36.602905+0000","last_became_peered":"2022-09-10T04:26:36.602905+0000","last_unstale":"2022-09-10T04:27:09.422082+0000","last_undegraded":"2022-09-10T04:27:09.422082+0000","last_fullsized":"2022-09-10T04:27:09.422082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:17:44.678651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424219+0000","last_change":"2022-09-10T04:26:31.551550+0000","last_active":"2022-09-10T04:27:09.424219+0000","last_peered":"2022-09-10T04:27:09.424219+0000","last_clean":"2022-09-10T04:27:09.424219+0000","last_became_active":"2022-09-10T04:26:31.551161+0000","last_became_peered":"2022-09-10T04:26:31.551161+0000","last_unstale":"2022-09-10T04:27:09.424219+0000","last_undegraded":"2022-09-10T04:27:09.424219+0000","last_fullsized":"2022-09-10T04:27:09.424219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:59:32.430914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425021+0000","last_change":"2022-09-10T04:26:32.122935+0000","last_active":"2022-09-10T04:27:09.425021+0000","last_peered":"2022-09-10T04:27:09.425021+0000","last_clean":"2022-09-10T04:27:09.425021+0000","last_became_active":"2022-09-10T04:26:32.122471+0000","last_became_peered":"2022-09-10T04:26:32.122471+0000","last_unstale":"2022-09-10T04:27:09.425021+0000","last_undegraded":"2022-09-10T04:27:09.425021+0000","last_fullsized":"2022-09-10T04:27:09.425021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:29:29.965140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420594+0000","last_change":"2022-09-10T04:26:36.604924+0000","last_active":"2022-09-10T04:27:09.420594+0000","last_peered":"2022-09-10T04:27:09.420594+0000","last_clean":"2022-09-10T04:27:09.420594+0000","last_became_active":"2022-09-10T04:26:36.604620+0000","last_became_peered":"2022-09-10T04:26:36.604620+0000","last_unstale":"2022-09-10T04:27:09.420594+0000","last_undegraded":"2022-09-10T04:27:09.420594+0000","last_fullsized":"2022-09-10T04:27:09.420594+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:41:34.668107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423261+0000","last_change":"2022-09-10T04:26:31.547564+0000","last_active":"2022-09-10T04:27:09.423261+0000","last_peered":"2022-09-10T04:27:09.423261+0000","last_clean":"2022-09-10T04:27:09.423261+0000","last_became_active":"2022-09-10T04:26:31.547180+0000","last_became_peered":"2022-09-10T04:26:31.547180+0000","last_unstale":"2022-09-10T04:27:09.423261+0000","last_undegraded":"2022-09-10T04:27:09.423261+0000","last_fullsized":"2022-09-10T04:27:09.423261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:23:12.454750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426158+0000","last_change":"2022-09-10T04:26:32.119307+0000","last_active":"2022-09-10T04:27:09.426158+0000","last_peered":"2022-09-10T04:27:09.426158+0000","last_clean":"2022-09-10T04:27:09.426158+0000","last_became_active":"2022-09-10T04:26:32.118995+0000","last_became_peered":"2022-09-10T04:26:32.118995+0000","last_unstale":"2022-09-10T04:27:09.426158+0000","last_undegraded":"2022-09-10T04:27:09.426158+0000","last_fullsized":"2022-09-10T04:27:09.426158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:28:25.888725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418383+0000","last_change":"2022-09-10T04:26:36.591310+0000","last_active":"2022-09-10T04:27:09.418383+0000","last_peered":"2022-09-10T04:27:09.418383+0000","last_clean":"2022-09-10T04:27:09.418383+0000","last_became_active":"2022-09-10T04:26:36.590907+0000","last_became_peered":"2022-09-10T04:26:36.590907+0000","last_unstale":"2022-09-10T04:27:09.418383+0000","last_undegraded":"2022-09-10T04:27:09.418383+0000","last_fullsized":"2022-09-10T04:27:09.418383+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:05:45.016925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:43.249975+0000","last_change":"2022-09-10T04:26:31.550524+0000","last_active":"2022-09-10T04:27:43.249975+0000","last_peered":"2022-09-10T04:27:43.249975+0000","last_clean":"2022-09-10T04:27:43.249975+0000","last_became_active":"2022-09-10T04:26:31.550259+0000","last_became_peered":"2022-09-10T04:26:31.550259+0000","last_unstale":"2022-09-10T04:27:43.249975+0000","last_undegraded":"2022-09-10T04:27:43.249975+0000","last_fullsized":"2022-09-10T04:27:43.249975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:53.982221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423230+0000","last_change":"2022-09-10T04:26:32.123377+0000","last_active":"2022-09-10T04:27:09.423230+0000","last_peered":"2022-09-10T04:27:09.423230+0000","last_clean":"2022-09-10T04:27:09.423230+0000","last_became_active":"2022-09-10T04:26:32.123076+0000","last_became_peered":"2022-09-10T04:26:32.123076+0000","last_unstale":"2022-09-10T04:27:09.423230+0000","last_undegraded":"2022-09-10T04:27:09.423230+0000","last_fullsized":"2022-09-10T04:27:09.423230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:40.425861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418231+0000","last_change":"2022-09-10T04:26:36.593491+0000","last_active":"2022-09-10T04:27:09.418231+0000","last_peered":"2022-09-10T04:27:09.418231+0000","last_clean":"2022-09-10T04:27:09.418231+0000","last_became_active":"2022-09-10T04:26:36.593064+0000","last_became_peered":"2022-09-10T04:26:36.593064+0000","last_unstale":"2022-09-10T04:27:09.418231+0000","last_undegraded":"2022-09-10T04:27:09.418231+0000","last_fullsized":"2022-09-10T04:27:09.418231+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:03:44.752739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424560+0000","last_change":"2022-09-10T04:26:31.555492+0000","last_active":"2022-09-10T04:27:09.424560+0000","last_peered":"2022-09-10T04:27:09.424560+0000","last_clean":"2022-09-10T04:27:09.424560+0000","last_became_active":"2022-09-10T04:26:31.555216+0000","last_became_peered":"2022-09-10T04:26:31.555216+0000","last_unstale":"2022-09-10T04:27:09.424560+0000","last_undegraded":"2022-09-10T04:27:09.424560+0000","last_fullsized":"2022-09-10T04:27:09.424560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:50:32.775631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427293+0000","last_change":"2022-09-10T04:26:32.139851+0000","last_active":"2022-09-10T04:27:09.427293+0000","last_peered":"2022-09-10T04:27:09.427293+0000","last_clean":"2022-09-10T04:27:09.427293+0000","last_became_active":"2022-09-10T04:26:32.139515+0000","last_became_peered":"2022-09-10T04:26:32.139515+0000","last_unstale":"2022-09-10T04:27:09.427293+0000","last_undegraded":"2022-09-10T04:27:09.427293+0000","last_fullsized":"2022-09-10T04:27:09.427293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:46:04.342743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421704+0000","last_change":"2022-09-10T04:26:36.598406+0000","last_active":"2022-09-10T04:27:09.421704+0000","last_peered":"2022-09-10T04:27:09.421704+0000","last_clean":"2022-09-10T04:27:09.421704+0000","last_became_active":"2022-09-10T04:26:36.598118+0000","last_became_peered":"2022-09-10T04:26:36.598118+0000","last_unstale":"2022-09-10T04:27:09.421704+0000","last_undegraded":"2022-09-10T04:27:09.421704+0000","last_fullsized":"2022-09-10T04:27:09.421704+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:35.903423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.428122+0000","last_change":"2022-09-10T04:26:31.541593+0000","last_active":"2022-09-10T04:27:09.428122+0000","last_peered":"2022-09-10T04:27:09.428122+0000","last_clean":"2022-09-10T04:27:09.428122+0000","last_became_active":"2022-09-10T04:26:31.541236+0000","last_became_peered":"2022-09-10T04:26:31.541236+0000","last_unstale":"2022-09-10T04:27:09.428122+0000","last_undegraded":"2022-09-10T04:27:09.428122+0000","last_fullsized":"2022-09-10T04:27:09.428122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:30:03.723483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423245+0000","last_change":"2022-09-10T04:26:32.147698+0000","last_active":"2022-09-10T04:27:09.423245+0000","last_peered":"2022-09-10T04:27:09.423245+0000","last_clean":"2022-09-10T04:27:09.423245+0000","last_became_active":"2022-09-10T04:26:32.147334+0000","last_became_peered":"2022-09-10T04:26:32.147334+0000","last_unstale":"2022-09-10T04:27:09.423245+0000","last_undegraded":"2022-09-10T04:27:09.423245+0000","last_fullsized":"2022-09-10T04:27:09.423245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:33:21.907371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429661+0000","last_change":"2022-09-10T04:26:36.597450+0000","last_active":"2022-09-10T04:27:09.429661+0000","last_peered":"2022-09-10T04:27:09.429661+0000","last_clean":"2022-09-10T04:27:09.429661+0000","last_became_active":"2022-09-10T04:26:36.597087+0000","last_became_peered":"2022-09-10T04:26:36.597087+0000","last_unstale":"2022-09-10T04:27:09.429661+0000","last_undegraded":"2022-09-10T04:27:09.429661+0000","last_fullsized":"2022-09-10T04:27:09.429661+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:06:27.788857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425510+0000","last_change":"2022-09-10T04:26:31.550451+0000","last_active":"2022-09-10T04:27:09.425510+0000","last_peered":"2022-09-10T04:27:09.425510+0000","last_clean":"2022-09-10T04:27:09.425510+0000","last_became_active":"2022-09-10T04:26:31.550158+0000","last_became_peered":"2022-09-10T04:26:31.550158+0000","last_unstale":"2022-09-10T04:27:09.425510+0000","last_undegraded":"2022-09-10T04:27:09.425510+0000","last_fullsized":"2022-09-10T04:27:09.425510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:21:27.132856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419212+0000","last_change":"2022-09-10T04:26:32.136694+0000","last_active":"2022-09-10T04:27:09.419212+0000","last_peered":"2022-09-10T04:27:09.419212+0000","last_clean":"2022-09-10T04:27:09.419212+0000","last_became_active":"2022-09-10T04:26:32.136398+0000","last_became_peered":"2022-09-10T04:26:32.136398+0000","last_unstale":"2022-09-10T04:27:09.419212+0000","last_undegraded":"2022-09-10T04:27:09.419212+0000","last_fullsized":"2022-09-10T04:27:09.419212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:32:22.413618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423754+0000","last_change":"2022-09-10T04:26:36.599433+0000","last_active":"2022-09-10T04:27:09.423754+0000","last_peered":"2022-09-10T04:27:09.423754+0000","last_clean":"2022-09-10T04:27:09.423754+0000","last_became_active":"2022-09-10T04:26:36.599013+0000","last_became_peered":"2022-09-10T04:26:36.599013+0000","last_unstale":"2022-09-10T04:27:09.423754+0000","last_undegraded":"2022-09-10T04:27:09.423754+0000","last_fullsized":"2022-09-10T04:27:09.423754+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:23:18.397219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":116,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:44.250737+0000","last_change":"2022-09-10T04:26:31.548631+0000","last_active":"2022-09-10T04:27:44.250737+0000","last_peered":"2022-09-10T04:27:44.250737+0000","last_clean":"2022-09-10T04:27:44.250737+0000","last_became_active":"2022-09-10T04:26:31.548292+0000","last_became_peered":"2022-09-10T04:26:31.548292+0000","last_unstale":"2022-09-10T04:27:44.250737+0000","last_undegraded":"2022-09-10T04:27:44.250737+0000","last_fullsized":"2022-09-10T04:27:44.250737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:12:40.825422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425167+0000","last_change":"2022-09-10T04:26:32.124469+0000","last_active":"2022-09-10T04:27:09.425167+0000","last_peered":"2022-09-10T04:27:09.425167+0000","last_clean":"2022-09-10T04:27:09.425167+0000","last_became_active":"2022-09-10T04:26:32.124084+0000","last_became_peered":"2022-09-10T04:26:32.124084+0000","last_unstale":"2022-09-10T04:27:09.425167+0000","last_undegraded":"2022-09-10T04:27:09.425167+0000","last_fullsized":"2022-09-10T04:27:09.425167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:39:04.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419136+0000","last_change":"2022-09-10T04:26:36.601607+0000","last_active":"2022-09-10T04:27:09.419136+0000","last_peered":"2022-09-10T04:27:09.419136+0000","last_clean":"2022-09-10T04:27:09.419136+0000","last_became_active":"2022-09-10T04:26:36.601191+0000","last_became_peered":"2022-09-10T04:26:36.601191+0000","last_unstale":"2022-09-10T04:27:09.419136+0000","last_undegraded":"2022-09-10T04:27:09.419136+0000","last_fullsized":"2022-09-10T04:27:09.419136+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:52:58.815905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427014+0000","last_change":"2022-09-10T04:26:31.548524+0000","last_active":"2022-09-10T04:27:09.427014+0000","last_peered":"2022-09-10T04:27:09.427014+0000","last_clean":"2022-09-10T04:27:09.427014+0000","last_became_active":"2022-09-10T04:26:31.548073+0000","last_became_peered":"2022-09-10T04:26:31.548073+0000","last_unstale":"2022-09-10T04:27:09.427014+0000","last_undegraded":"2022-09-10T04:27:09.427014+0000","last_fullsized":"2022-09-10T04:27:09.427014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:01:02.489915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740368+0000","last_change":"2022-09-10T04:27:10.740368+0000","last_active":"2022-09-10T04:27:10.740368+0000","last_peered":"2022-09-10T04:27:10.740368+0000","last_clean":"2022-09-10T04:27:10.740368+0000","last_became_active":"2022-09-10T04:27:10.739903+0000","last_became_peered":"2022-09-10T04:27:10.739903+0000","last_unstale":"2022-09-10T04:27:10.740368+0000","last_undegraded":"2022-09-10T04:27:10.740368+0000","last_fullsized":"2022-09-10T04:27:10.740368+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:59:30.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.427026+0000","last_change":"2022-09-10T04:27:10.427026+0000","last_active":"2022-09-10T04:27:10.427026+0000","last_peered":"2022-09-10T04:27:10.427026+0000","last_clean":"2022-09-10T04:27:10.427026+0000","last_became_active":"2022-09-10T04:27:10.426556+0000","last_became_peered":"2022-09-10T04:27:10.426556+0000","last_unstale":"2022-09-10T04:27:10.427026+0000","last_undegraded":"2022-09-10T04:27:10.427026+0000","last_fullsized":"2022-09-10T04:27:10.427026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:34:43.204361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425683+0000","last_change":"2022-09-10T04:26:31.557753+0000","last_active":"2022-09-10T04:27:09.425683+0000","last_peered":"2022-09-10T04:27:09.425683+0000","last_clean":"2022-09-10T04:27:09.425683+0000","last_became_active":"2022-09-10T04:26:31.557432+0000","last_became_peered":"2022-09-10T04:26:31.557432+0000","last_unstale":"2022-09-10T04:27:09.425683+0000","last_undegraded":"2022-09-10T04:27:09.425683+0000","last_fullsized":"2022-09-10T04:27:09.425683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:14:51.954481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740377+0000","last_change":"2022-09-10T04:27:10.740377+0000","last_active":"2022-09-10T04:27:10.740377+0000","last_peered":"2022-09-10T04:27:10.740377+0000","last_clean":"2022-09-10T04:27:10.740377+0000","last_became_active":"2022-09-10T04:27:10.739989+0000","last_became_peered":"2022-09-10T04:27:10.739989+0000","last_unstale":"2022-09-10T04:27:10.740377+0000","last_undegraded":"2022-09-10T04:27:10.740377+0000","last_fullsized":"2022-09-10T04:27:10.740377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:46:22.492574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419267+0000","last_change":"2022-09-10T04:26:36.591159+0000","last_active":"2022-09-10T04:27:09.419267+0000","last_peered":"2022-09-10T04:27:09.419267+0000","last_clean":"2022-09-10T04:27:09.419267+0000","last_became_active":"2022-09-10T04:26:36.590788+0000","last_became_peered":"2022-09-10T04:26:36.590788+0000","last_unstale":"2022-09-10T04:27:09.419267+0000","last_undegraded":"2022-09-10T04:27:09.419267+0000","last_fullsized":"2022-09-10T04:27:09.419267+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:50:34.597783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'14","reported_seq":128,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:44.250698+0000","last_change":"2022-09-10T04:26:31.547702+0000","last_active":"2022-09-10T04:27:44.250698+0000","last_peered":"2022-09-10T04:27:44.250698+0000","last_clean":"2022-09-10T04:27:44.250698+0000","last_became_active":"2022-09-10T04:26:31.547388+0000","last_became_peered":"2022-09-10T04:26:31.547388+0000","last_unstale":"2022-09-10T04:27:44.250698+0000","last_undegraded":"2022-09-10T04:27:44.250698+0000","last_fullsized":"2022-09-10T04:27:44.250698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:42:28.378515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3749,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":96,"num_read_kb":52,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424056+0000","last_change":"2022-09-10T04:26:32.117260+0000","last_active":"2022-09-10T04:27:09.424056+0000","last_peered":"2022-09-10T04:27:09.424056+0000","last_clean":"2022-09-10T04:27:09.424056+0000","last_became_active":"2022-09-10T04:26:32.116928+0000","last_became_peered":"2022-09-10T04:26:32.116928+0000","last_unstale":"2022-09-10T04:27:09.424056+0000","last_undegraded":"2022-09-10T04:27:09.424056+0000","last_fullsized":"2022-09-10T04:27:09.424056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:00:10.528509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426431+0000","last_change":"2022-09-10T04:26:36.600239+0000","last_active":"2022-09-10T04:27:09.426431+0000","last_peered":"2022-09-10T04:27:09.426431+0000","last_clean":"2022-09-10T04:27:09.426431+0000","last_became_active":"2022-09-10T04:26:36.599614+0000","last_became_peered":"2022-09-10T04:26:36.599614+0000","last_unstale":"2022-09-10T04:27:09.426431+0000","last_undegraded":"2022-09-10T04:27:09.426431+0000","last_fullsized":"2022-09-10T04:27:09.426431+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:46:36.535130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418087+0000","last_change":"2022-09-10T04:26:31.552412+0000","last_active":"2022-09-10T04:27:09.418087+0000","last_peered":"2022-09-10T04:27:09.418087+0000","last_clean":"2022-09-10T04:27:09.418087+0000","last_became_active":"2022-09-10T04:26:31.552145+0000","last_became_peered":"2022-09-10T04:26:31.552145+0000","last_unstale":"2022-09-10T04:27:09.418087+0000","last_undegraded":"2022-09-10T04:27:09.418087+0000","last_fullsized":"2022-09-10T04:27:09.418087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:16:36.041950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427385+0000","last_change":"2022-09-10T04:26:32.136276+0000","last_active":"2022-09-10T04:27:09.427385+0000","last_peered":"2022-09-10T04:27:09.427385+0000","last_clean":"2022-09-10T04:27:09.427385+0000","last_became_active":"2022-09-10T04:26:32.135894+0000","last_became_peered":"2022-09-10T04:26:32.135894+0000","last_unstale":"2022-09-10T04:27:09.427385+0000","last_undegraded":"2022-09-10T04:27:09.427385+0000","last_fullsized":"2022-09-10T04:27:09.427385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:12:32.912358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740446+0000","last_change":"2022-09-10T04:27:10.740446+0000","last_active":"2022-09-10T04:27:10.740446+0000","last_peered":"2022-09-10T04:27:10.740446+0000","last_clean":"2022-09-10T04:27:10.740446+0000","last_became_active":"2022-09-10T04:27:10.739986+0000","last_became_peered":"2022-09-10T04:27:10.739986+0000","last_unstale":"2022-09-10T04:27:10.740446+0000","last_undegraded":"2022-09-10T04:27:10.740446+0000","last_fullsized":"2022-09-10T04:27:10.740446+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:12:17.918925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426192+0000","last_change":"2022-09-10T04:26:31.556823+0000","last_active":"2022-09-10T04:27:09.426192+0000","last_peered":"2022-09-10T04:27:09.426192+0000","last_clean":"2022-09-10T04:27:09.426192+0000","last_became_active":"2022-09-10T04:26:31.556509+0000","last_became_peered":"2022-09-10T04:26:31.556509+0000","last_unstale":"2022-09-10T04:27:09.426192+0000","last_undegraded":"2022-09-10T04:27:09.426192+0000","last_fullsized":"2022-09-10T04:27:09.426192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:35:16.227863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427136+0000","last_change":"2022-09-10T04:26:32.130565+0000","last_active":"2022-09-10T04:27:09.427136+0000","last_peered":"2022-09-10T04:27:09.427136+0000","last_clean":"2022-09-10T04:27:09.427136+0000","last_became_active":"2022-09-10T04:26:32.130121+0000","last_became_peered":"2022-09-10T04:26:32.130121+0000","last_unstale":"2022-09-10T04:27:09.427136+0000","last_undegraded":"2022-09-10T04:27:09.427136+0000","last_fullsized":"2022-09-10T04:27:09.427136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:55:03.504615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419883+0000","last_change":"2022-09-10T04:26:36.603262+0000","last_active":"2022-09-10T04:27:09.419883+0000","last_peered":"2022-09-10T04:27:09.419883+0000","last_clean":"2022-09-10T04:27:09.419883+0000","last_became_active":"2022-09-10T04:26:36.602590+0000","last_became_peered":"2022-09-10T04:26:36.602590+0000","last_unstale":"2022-09-10T04:27:09.419883+0000","last_undegraded":"2022-09-10T04:27:09.419883+0000","last_fullsized":"2022-09-10T04:27:09.419883+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:19:03.583123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:45.252498+0000","last_change":"2022-09-10T04:26:31.545898+0000","last_active":"2022-09-10T04:27:45.252498+0000","last_peered":"2022-09-10T04:27:45.252498+0000","last_clean":"2022-09-10T04:27:45.252498+0000","last_became_active":"2022-09-10T04:26:31.545355+0000","last_became_peered":"2022-09-10T04:26:31.545355+0000","last_unstale":"2022-09-10T04:27:45.252498+0000","last_undegraded":"2022-09-10T04:27:45.252498+0000","last_fullsized":"2022-09-10T04:27:45.252498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:50:36.187534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426475+0000","last_change":"2022-09-10T04:26:32.116268+0000","last_active":"2022-09-10T04:27:09.426475+0000","last_peered":"2022-09-10T04:27:09.426475+0000","last_clean":"2022-09-10T04:27:09.426475+0000","last_became_active":"2022-09-10T04:26:32.115133+0000","last_became_peered":"2022-09-10T04:26:32.115133+0000","last_unstale":"2022-09-10T04:27:09.426475+0000","last_undegraded":"2022-09-10T04:27:09.426475+0000","last_fullsized":"2022-09-10T04:27:09.426475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:38:05.319082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427708+0000","last_change":"2022-09-10T04:26:36.605212+0000","last_active":"2022-09-10T04:27:09.427708+0000","last_peered":"2022-09-10T04:27:09.427708+0000","last_clean":"2022-09-10T04:27:09.427708+0000","last_became_active":"2022-09-10T04:26:36.604767+0000","last_became_peered":"2022-09-10T04:26:36.604767+0000","last_unstale":"2022-09-10T04:27:09.427708+0000","last_undegraded":"2022-09-10T04:27:09.427708+0000","last_fullsized":"2022-09-10T04:27:09.427708+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:33:55.048936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422700+0000","last_change":"2022-09-10T04:26:31.561097+0000","last_active":"2022-09-10T04:27:09.422700+0000","last_peered":"2022-09-10T04:27:09.422700+0000","last_clean":"2022-09-10T04:27:09.422700+0000","last_became_active":"2022-09-10T04:26:31.560748+0000","last_became_peered":"2022-09-10T04:26:31.560748+0000","last_unstale":"2022-09-10T04:27:09.422700+0000","last_undegraded":"2022-09-10T04:27:09.422700+0000","last_fullsized":"2022-09-10T04:27:09.422700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:44:32.921350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425328+0000","last_change":"2022-09-10T04:26:32.122012+0000","last_active":"2022-09-10T04:27:09.425328+0000","last_peered":"2022-09-10T04:27:09.425328+0000","last_clean":"2022-09-10T04:27:09.425328+0000","last_became_active":"2022-09-10T04:26:32.121668+0000","last_became_peered":"2022-09-10T04:26:32.121668+0000","last_unstale":"2022-09-10T04:27:09.425328+0000","last_undegraded":"2022-09-10T04:27:09.425328+0000","last_fullsized":"2022-09-10T04:27:09.425328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:49:57.312550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425296+0000","last_change":"2022-09-10T04:26:36.594964+0000","last_active":"2022-09-10T04:27:09.425296+0000","last_peered":"2022-09-10T04:27:09.425296+0000","last_clean":"2022-09-10T04:27:09.425296+0000","last_became_active":"2022-09-10T04:26:36.594548+0000","last_became_peered":"2022-09-10T04:26:36.594548+0000","last_unstale":"2022-09-10T04:27:09.425296+0000","last_undegraded":"2022-09-10T04:27:09.425296+0000","last_fullsized":"2022-09-10T04:27:09.425296+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:01.446152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423752+0000","last_change":"2022-09-10T04:26:31.549608+0000","last_active":"2022-09-10T04:27:09.423752+0000","last_peered":"2022-09-10T04:27:09.423752+0000","last_clean":"2022-09-10T04:27:09.423752+0000","last_became_active":"2022-09-10T04:26:31.549300+0000","last_became_peered":"2022-09-10T04:26:31.549300+0000","last_unstale":"2022-09-10T04:27:09.423752+0000","last_undegraded":"2022-09-10T04:27:09.423752+0000","last_fullsized":"2022-09-10T04:27:09.423752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:53:51.183777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421594+0000","last_change":"2022-09-10T04:26:32.119540+0000","last_active":"2022-09-10T04:27:09.421594+0000","last_peered":"2022-09-10T04:27:09.421594+0000","last_clean":"2022-09-10T04:27:09.421594+0000","last_became_active":"2022-09-10T04:26:32.119341+0000","last_became_peered":"2022-09-10T04:26:32.119341+0000","last_unstale":"2022-09-10T04:27:09.421594+0000","last_undegraded":"2022-09-10T04:27:09.421594+0000","last_fullsized":"2022-09-10T04:27:09.421594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:21:13.815142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423202+0000","last_change":"2022-09-10T04:26:36.790770+0000","last_active":"2022-09-10T04:27:09.423202+0000","last_peered":"2022-09-10T04:27:09.423202+0000","last_clean":"2022-09-10T04:27:09.423202+0000","last_became_active":"2022-09-10T04:26:36.790366+0000","last_became_peered":"2022-09-10T04:26:36.790366+0000","last_unstale":"2022-09-10T04:27:09.423202+0000","last_undegraded":"2022-09-10T04:27:09.423202+0000","last_fullsized":"2022-09-10T04:27:09.423202+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:40:18.004852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.419529+0000","last_change":"2022-09-10T04:26:31.557145+0000","last_active":"2022-09-10T04:27:09.419529+0000","last_peered":"2022-09-10T04:27:09.419529+0000","last_clean":"2022-09-10T04:27:09.419529+0000","last_became_active":"2022-09-10T04:26:31.556805+0000","last_became_peered":"2022-09-10T04:26:31.556805+0000","last_unstale":"2022-09-10T04:27:09.419529+0000","last_undegraded":"2022-09-10T04:27:09.419529+0000","last_fullsized":"2022-09-10T04:27:09.419529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:53:41.074803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423637+0000","last_change":"2022-09-10T04:26:32.139758+0000","last_active":"2022-09-10T04:27:09.423637+0000","last_peered":"2022-09-10T04:27:09.423637+0000","last_clean":"2022-09-10T04:27:09.423637+0000","last_became_active":"2022-09-10T04:26:32.139469+0000","last_became_peered":"2022-09-10T04:26:32.139469+0000","last_unstale":"2022-09-10T04:27:09.423637+0000","last_undegraded":"2022-09-10T04:27:09.423637+0000","last_fullsized":"2022-09-10T04:27:09.423637+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:09:52.224744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427720+0000","last_change":"2022-09-10T04:26:36.600083+0000","last_active":"2022-09-10T04:27:09.427720+0000","last_peered":"2022-09-10T04:27:09.427720+0000","last_clean":"2022-09-10T04:27:09.427720+0000","last_became_active":"2022-09-10T04:26:36.599648+0000","last_became_peered":"2022-09-10T04:26:36.599648+0000","last_unstale":"2022-09-10T04:27:09.427720+0000","last_undegraded":"2022-09-10T04:27:09.427720+0000","last_fullsized":"2022-09-10T04:27:09.427720+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:03:24.897645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:46.265482+0000","last_change":"2022-09-10T04:26:31.560147+0000","last_active":"2022-09-10T04:27:46.265482+0000","last_peered":"2022-09-10T04:27:46.265482+0000","last_clean":"2022-09-10T04:27:46.265482+0000","last_became_active":"2022-09-10T04:26:31.559868+0000","last_became_peered":"2022-09-10T04:26:31.559868+0000","last_unstale":"2022-09-10T04:27:46.265482+0000","last_undegraded":"2022-09-10T04:27:46.265482+0000","last_fullsized":"2022-09-10T04:27:46.265482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:02:46.652743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422976+0000","last_change":"2022-09-10T04:26:32.125369+0000","last_active":"2022-09-10T04:27:09.422976+0000","last_peered":"2022-09-10T04:27:09.422976+0000","last_clean":"2022-09-10T04:27:09.422976+0000","last_became_active":"2022-09-10T04:26:32.125077+0000","last_became_peered":"2022-09-10T04:26:32.125077+0000","last_unstale":"2022-09-10T04:27:09.422976+0000","last_undegraded":"2022-09-10T04:27:09.422976+0000","last_fullsized":"2022-09-10T04:27:09.422976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:03:13.265143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427814+0000","last_change":"2022-09-10T04:26:36.790278+0000","last_active":"2022-09-10T04:27:09.427814+0000","last_peered":"2022-09-10T04:27:09.427814+0000","last_clean":"2022-09-10T04:27:09.427814+0000","last_became_active":"2022-09-10T04:26:36.789852+0000","last_became_peered":"2022-09-10T04:26:36.789852+0000","last_unstale":"2022-09-10T04:27:09.427814+0000","last_undegraded":"2022-09-10T04:27:09.427814+0000","last_fullsized":"2022-09-10T04:27:09.427814+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:26:38.960022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425728+0000","last_change":"2022-09-10T04:26:32.119857+0000","last_active":"2022-09-10T04:27:09.425728+0000","last_peered":"2022-09-10T04:27:09.425728+0000","last_clean":"2022-09-10T04:27:09.425728+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:27:09.425728+0000","last_undegraded":"2022-09-10T04:27:09.425728+0000","last_fullsized":"2022-09-10T04:27:09.425728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:21:40.941165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:45.251930+0000","last_change":"2022-09-10T04:26:31.555699+0000","last_active":"2022-09-10T04:27:45.251930+0000","last_peered":"2022-09-10T04:27:45.251930+0000","last_clean":"2022-09-10T04:27:45.251930+0000","last_became_active":"2022-09-10T04:26:31.555337+0000","last_became_peered":"2022-09-10T04:26:31.555337+0000","last_unstale":"2022-09-10T04:27:45.251930+0000","last_undegraded":"2022-09-10T04:27:45.251930+0000","last_fullsized":"2022-09-10T04:27:45.251930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:38:23.689077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427200+0000","last_change":"2022-09-10T04:26:36.592313+0000","last_active":"2022-09-10T04:27:09.427200+0000","last_peered":"2022-09-10T04:27:09.427200+0000","last_clean":"2022-09-10T04:27:09.427200+0000","last_became_active":"2022-09-10T04:26:36.591703+0000","last_became_peered":"2022-09-10T04:26:36.591703+0000","last_unstale":"2022-09-10T04:27:09.427200+0000","last_undegraded":"2022-09-10T04:27:09.427200+0000","last_fullsized":"2022-09-10T04:27:09.427200+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:19:26.704605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.740776+0000","last_change":"2022-09-10T04:27:10.739699+0000","last_active":"2022-09-10T04:27:10.740776+0000","last_peered":"2022-09-10T04:27:10.740776+0000","last_clean":"2022-09-10T04:27:10.740776+0000","last_became_active":"2022-09-10T04:27:10.739321+0000","last_became_peered":"2022-09-10T04:27:10.739321+0000","last_unstale":"2022-09-10T04:27:10.740776+0000","last_undegraded":"2022-09-10T04:27:10.740776+0000","last_fullsized":"2022-09-10T04:27:10.740776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:36:46.680589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422262+0000","last_change":"2022-09-10T04:26:31.544708+0000","last_active":"2022-09-10T04:27:09.422262+0000","last_peered":"2022-09-10T04:27:09.422262+0000","last_clean":"2022-09-10T04:27:09.422262+0000","last_became_active":"2022-09-10T04:26:31.544386+0000","last_became_peered":"2022-09-10T04:26:31.544386+0000","last_unstale":"2022-09-10T04:27:09.422262+0000","last_undegraded":"2022-09-10T04:27:09.422262+0000","last_fullsized":"2022-09-10T04:27:09.422262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:07:18.444015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":379,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426280+0000","last_change":"2022-09-10T04:25:23.357461+0000","last_active":"2022-09-10T04:27:09.426280+0000","last_peered":"2022-09-10T04:27:09.426280+0000","last_clean":"2022-09-10T04:27:09.426280+0000","last_became_active":"2022-09-10T04:25:23.356951+0000","last_became_peered":"2022-09-10T04:25:23.356951+0000","last_unstale":"2022-09-10T04:27:09.426280+0000","last_undegraded":"2022-09-10T04:27:09.426280+0000","last_fullsized":"2022-09-10T04:27:09.426280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:21:40.122270+0000","last_clean_scrub_stamp":"2022-09-10T04:21:40.122270+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:01:20.972512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425953+0000","last_change":"2022-09-10T04:26:36.790925+0000","last_active":"2022-09-10T04:27:09.425953+0000","last_peered":"2022-09-10T04:27:09.425953+0000","last_clean":"2022-09-10T04:27:09.425953+0000","last_became_active":"2022-09-10T04:26:36.790332+0000","last_became_peered":"2022-09-10T04:26:36.790332+0000","last_unstale":"2022-09-10T04:27:09.425953+0000","last_undegraded":"2022-09-10T04:27:09.425953+0000","last_fullsized":"2022-09-10T04:27:09.425953+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:10:55.153959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427972+0000","last_change":"2022-09-10T04:26:32.122356+0000","last_active":"2022-09-10T04:27:09.427972+0000","last_peered":"2022-09-10T04:27:09.427972+0000","last_clean":"2022-09-10T04:27:09.427972+0000","last_became_active":"2022-09-10T04:26:32.122028+0000","last_became_peered":"2022-09-10T04:26:32.122028+0000","last_unstale":"2022-09-10T04:27:09.427972+0000","last_undegraded":"2022-09-10T04:27:09.427972+0000","last_fullsized":"2022-09-10T04:27:09.427972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:51:33.837184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426406+0000","last_change":"2022-09-10T04:26:31.556537+0000","last_active":"2022-09-10T04:27:09.426406+0000","last_peered":"2022-09-10T04:27:09.426406+0000","last_clean":"2022-09-10T04:27:09.426406+0000","last_became_active":"2022-09-10T04:26:31.556209+0000","last_became_peered":"2022-09-10T04:26:31.556209+0000","last_unstale":"2022-09-10T04:27:09.426406+0000","last_undegraded":"2022-09-10T04:27:09.426406+0000","last_fullsized":"2022-09-10T04:27:09.426406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:42:05.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421174+0000","last_change":"2022-09-10T04:26:36.600051+0000","last_active":"2022-09-10T04:27:09.421174+0000","last_peered":"2022-09-10T04:27:09.421174+0000","last_clean":"2022-09-10T04:27:09.421174+0000","last_became_active":"2022-09-10T04:26:36.599603+0000","last_became_peered":"2022-09-10T04:26:36.599603+0000","last_unstale":"2022-09-10T04:27:09.421174+0000","last_undegraded":"2022-09-10T04:27:09.421174+0000","last_fullsized":"2022-09-10T04:27:09.421174+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:48:12.857851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424105+0000","last_change":"2022-09-10T04:26:31.560780+0000","last_active":"2022-09-10T04:27:09.424105+0000","last_peered":"2022-09-10T04:27:09.424105+0000","last_clean":"2022-09-10T04:27:09.424105+0000","last_became_active":"2022-09-10T04:26:31.560462+0000","last_became_peered":"2022-09-10T04:26:31.560462+0000","last_unstale":"2022-09-10T04:27:09.424105+0000","last_undegraded":"2022-09-10T04:27:09.424105+0000","last_fullsized":"2022-09-10T04:27:09.424105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:05:15.437095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427857+0000","last_change":"2022-09-10T04:26:32.127579+0000","last_active":"2022-09-10T04:27:09.427857+0000","last_peered":"2022-09-10T04:27:09.427857+0000","last_clean":"2022-09-10T04:27:09.427857+0000","last_became_active":"2022-09-10T04:26:32.127237+0000","last_became_peered":"2022-09-10T04:26:32.127237+0000","last_unstale":"2022-09-10T04:27:09.427857+0000","last_undegraded":"2022-09-10T04:27:09.427857+0000","last_fullsized":"2022-09-10T04:27:09.427857+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:09:28.061156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423213+0000","last_change":"2022-09-10T04:26:36.790777+0000","last_active":"2022-09-10T04:27:09.423213+0000","last_peered":"2022-09-10T04:27:09.423213+0000","last_clean":"2022-09-10T04:27:09.423213+0000","last_became_active":"2022-09-10T04:26:36.789974+0000","last_became_peered":"2022-09-10T04:26:36.789974+0000","last_unstale":"2022-09-10T04:27:09.423213+0000","last_undegraded":"2022-09-10T04:27:09.423213+0000","last_fullsized":"2022-09-10T04:27:09.423213+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:41:20.214040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.437497+0000","last_change":"2022-09-10T04:27:10.437497+0000","last_active":"2022-09-10T04:27:10.437497+0000","last_peered":"2022-09-10T04:27:10.437497+0000","last_clean":"2022-09-10T04:27:10.437497+0000","last_became_active":"2022-09-10T04:27:10.427029+0000","last_became_peered":"2022-09-10T04:27:10.427029+0000","last_unstale":"2022-09-10T04:27:10.437497+0000","last_undegraded":"2022-09-10T04:27:10.437497+0000","last_fullsized":"2022-09-10T04:27:10.437497+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:01:01.308164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424813+0000","last_change":"2022-09-10T04:26:32.145463+0000","last_active":"2022-09-10T04:27:09.424813+0000","last_peered":"2022-09-10T04:27:09.424813+0000","last_clean":"2022-09-10T04:27:09.424813+0000","last_became_active":"2022-09-10T04:26:32.145115+0000","last_became_peered":"2022-09-10T04:26:32.145115+0000","last_unstale":"2022-09-10T04:27:09.424813+0000","last_undegraded":"2022-09-10T04:27:09.424813+0000","last_fullsized":"2022-09-10T04:27:09.424813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:14:11.597005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.417822+0000","last_change":"2022-09-10T04:26:36.594514+0000","last_active":"2022-09-10T04:27:09.417822+0000","last_peered":"2022-09-10T04:27:09.417822+0000","last_clean":"2022-09-10T04:27:09.417822+0000","last_became_active":"2022-09-10T04:26:36.594092+0000","last_became_peered":"2022-09-10T04:26:36.594092+0000","last_unstale":"2022-09-10T04:27:09.417822+0000","last_undegraded":"2022-09-10T04:27:09.417822+0000","last_fullsized":"2022-09-10T04:27:09.417822+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:34:31.510362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421375+0000","last_change":"2022-09-10T04:26:31.548844+0000","last_active":"2022-09-10T04:27:09.421375+0000","last_peered":"2022-09-10T04:27:09.421375+0000","last_clean":"2022-09-10T04:27:09.421375+0000","last_became_active":"2022-09-10T04:26:31.548535+0000","last_became_peered":"2022-09-10T04:26:31.548535+0000","last_unstale":"2022-09-10T04:27:09.421375+0000","last_undegraded":"2022-09-10T04:27:09.421375+0000","last_fullsized":"2022-09-10T04:27:09.421375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:40:49.443190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429750+0000","last_change":"2022-09-10T04:26:32.136407+0000","last_active":"2022-09-10T04:27:09.429750+0000","last_peered":"2022-09-10T04:27:09.429750+0000","last_clean":"2022-09-10T04:27:09.429750+0000","last_became_active":"2022-09-10T04:26:32.136116+0000","last_became_peered":"2022-09-10T04:26:32.136116+0000","last_unstale":"2022-09-10T04:27:09.429750+0000","last_undegraded":"2022-09-10T04:27:09.429750+0000","last_fullsized":"2022-09-10T04:27:09.429750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T10:32:26.673896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.421066+0000","last_change":"2022-09-10T04:26:36.790658+0000","last_active":"2022-09-10T04:27:09.421066+0000","last_peered":"2022-09-10T04:27:09.421066+0000","last_clean":"2022-09-10T04:27:09.421066+0000","last_became_active":"2022-09-10T04:26:36.790203+0000","last_became_peered":"2022-09-10T04:26:36.790203+0000","last_unstale":"2022-09-10T04:27:09.421066+0000","last_undegraded":"2022-09-10T04:27:09.421066+0000","last_fullsized":"2022-09-10T04:27:09.421066+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:53:08.131113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425301+0000","last_change":"2022-09-10T04:26:31.549875+0000","last_active":"2022-09-10T04:27:09.425301+0000","last_peered":"2022-09-10T04:27:09.425301+0000","last_clean":"2022-09-10T04:27:09.425301+0000","last_became_active":"2022-09-10T04:26:31.549578+0000","last_became_peered":"2022-09-10T04:26:31.549578+0000","last_unstale":"2022-09-10T04:27:09.425301+0000","last_undegraded":"2022-09-10T04:27:09.425301+0000","last_fullsized":"2022-09-10T04:27:09.425301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:52:51.521360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422810+0000","last_change":"2022-09-10T04:26:32.111710+0000","last_active":"2022-09-10T04:27:09.422810+0000","last_peered":"2022-09-10T04:27:09.422810+0000","last_clean":"2022-09-10T04:27:09.422810+0000","last_became_active":"2022-09-10T04:26:32.111314+0000","last_became_peered":"2022-09-10T04:26:32.111314+0000","last_unstale":"2022-09-10T04:27:09.422810+0000","last_undegraded":"2022-09-10T04:27:09.422810+0000","last_fullsized":"2022-09-10T04:27:09.422810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:36:42.589093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420029+0000","last_change":"2022-09-10T04:26:36.596926+0000","last_active":"2022-09-10T04:27:09.420029+0000","last_peered":"2022-09-10T04:27:09.420029+0000","last_clean":"2022-09-10T04:27:09.420029+0000","last_became_active":"2022-09-10T04:26:36.596287+0000","last_became_peered":"2022-09-10T04:26:36.596287+0000","last_unstale":"2022-09-10T04:27:09.420029+0000","last_undegraded":"2022-09-10T04:27:09.420029+0000","last_fullsized":"2022-09-10T04:27:09.420029+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:52:54.252320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423848+0000","last_change":"2022-09-10T04:26:31.552277+0000","last_active":"2022-09-10T04:27:09.423848+0000","last_peered":"2022-09-10T04:27:09.423848+0000","last_clean":"2022-09-10T04:27:09.423848+0000","last_became_active":"2022-09-10T04:26:31.551904+0000","last_became_peered":"2022-09-10T04:26:31.551904+0000","last_unstale":"2022-09-10T04:27:09.423848+0000","last_undegraded":"2022-09-10T04:27:09.423848+0000","last_fullsized":"2022-09-10T04:27:09.423848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:28:52.507833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420912+0000","last_change":"2022-09-10T04:26:32.116288+0000","last_active":"2022-09-10T04:27:09.420912+0000","last_peered":"2022-09-10T04:27:09.420912+0000","last_clean":"2022-09-10T04:27:09.420912+0000","last_became_active":"2022-09-10T04:26:32.115955+0000","last_became_peered":"2022-09-10T04:26:32.115955+0000","last_unstale":"2022-09-10T04:27:09.420912+0000","last_undegraded":"2022-09-10T04:27:09.420912+0000","last_fullsized":"2022-09-10T04:27:09.420912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T11:02:30.470550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420960+0000","last_change":"2022-09-10T04:26:36.594357+0000","last_active":"2022-09-10T04:27:09.420960+0000","last_peered":"2022-09-10T04:27:09.420960+0000","last_clean":"2022-09-10T04:27:09.420960+0000","last_became_active":"2022-09-10T04:26:36.593865+0000","last_became_peered":"2022-09-10T04:26:36.593865+0000","last_unstale":"2022-09-10T04:27:09.420960+0000","last_undegraded":"2022-09-10T04:27:09.420960+0000","last_fullsized":"2022-09-10T04:27:09.420960+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:11:14.675252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.432478+0000","last_change":"2022-09-10T04:27:10.432089+0000","last_active":"2022-09-10T04:27:10.432478+0000","last_peered":"2022-09-10T04:27:10.432478+0000","last_clean":"2022-09-10T04:27:10.432478+0000","last_became_active":"2022-09-10T04:27:10.427334+0000","last_became_peered":"2022-09-10T04:27:10.427334+0000","last_unstale":"2022-09-10T04:27:10.432478+0000","last_undegraded":"2022-09-10T04:27:10.432478+0000","last_fullsized":"2022-09-10T04:27:10.432478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:04:04.064842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424238+0000","last_change":"2022-09-10T04:26:32.145692+0000","last_active":"2022-09-10T04:27:09.424238+0000","last_peered":"2022-09-10T04:27:09.424238+0000","last_clean":"2022-09-10T04:27:09.424238+0000","last_became_active":"2022-09-10T04:26:32.144922+0000","last_became_peered":"2022-09-10T04:26:32.144922+0000","last_unstale":"2022-09-10T04:27:09.424238+0000","last_undegraded":"2022-09-10T04:27:09.424238+0000","last_fullsized":"2022-09-10T04:27:09.424238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:12:29.711076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420911+0000","last_change":"2022-09-10T04:26:36.791321+0000","last_active":"2022-09-10T04:27:09.420911+0000","last_peered":"2022-09-10T04:27:09.420911+0000","last_clean":"2022-09-10T04:27:09.420911+0000","last_became_active":"2022-09-10T04:26:36.789918+0000","last_became_peered":"2022-09-10T04:26:36.789918+0000","last_unstale":"2022-09-10T04:27:09.420911+0000","last_undegraded":"2022-09-10T04:27:09.420911+0000","last_fullsized":"2022-09-10T04:27:09.420911+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T15:49:56.538141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":22,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.423474+0000","last_change":"2022-09-10T04:26:31.552140+0000","last_active":"2022-09-10T04:27:09.423474+0000","last_peered":"2022-09-10T04:27:09.423474+0000","last_clean":"2022-09-10T04:27:09.423474+0000","last_became_active":"2022-09-10T04:26:31.551191+0000","last_became_peered":"2022-09-10T04:26:31.551191+0000","last_unstale":"2022-09-10T04:27:09.423474+0000","last_undegraded":"2022-09-10T04:27:09.423474+0000","last_fullsized":"2022-09-10T04:27:09.423474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T13:08:52.681536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426681+0000","last_change":"2022-09-10T04:26:32.125775+0000","last_active":"2022-09-10T04:27:09.426681+0000","last_peered":"2022-09-10T04:27:09.426681+0000","last_clean":"2022-09-10T04:27:09.426681+0000","last_became_active":"2022-09-10T04:26:32.125463+0000","last_became_peered":"2022-09-10T04:26:32.125463+0000","last_unstale":"2022-09-10T04:27:09.426681+0000","last_undegraded":"2022-09-10T04:27:09.426681+0000","last_fullsized":"2022-09-10T04:27:09.426681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:33:26.816695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424706+0000","last_change":"2022-09-10T04:26:36.593873+0000","last_active":"2022-09-10T04:27:09.424706+0000","last_peered":"2022-09-10T04:27:09.424706+0000","last_clean":"2022-09-10T04:27:09.424706+0000","last_became_active":"2022-09-10T04:26:36.593418+0000","last_became_peered":"2022-09-10T04:26:36.593418+0000","last_unstale":"2022-09-10T04:27:09.424706+0000","last_undegraded":"2022-09-10T04:27:09.424706+0000","last_fullsized":"2022-09-10T04:27:09.424706+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:19:21.834135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:42.300019+0000","last_change":"2022-09-10T04:26:31.555352+0000","last_active":"2022-09-10T04:27:42.300019+0000","last_peered":"2022-09-10T04:27:42.300019+0000","last_clean":"2022-09-10T04:27:42.300019+0000","last_became_active":"2022-09-10T04:26:31.555092+0000","last_became_peered":"2022-09-10T04:26:31.555092+0000","last_unstale":"2022-09-10T04:27:42.300019+0000","last_undegraded":"2022-09-10T04:27:42.300019+0000","last_fullsized":"2022-09-10T04:27:42.300019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:30:17.812089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1106,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427305+0000","last_change":"2022-09-10T04:26:32.115687+0000","last_active":"2022-09-10T04:27:09.427305+0000","last_peered":"2022-09-10T04:27:09.427305+0000","last_clean":"2022-09-10T04:27:09.427305+0000","last_became_active":"2022-09-10T04:26:32.115369+0000","last_became_peered":"2022-09-10T04:26:32.115369+0000","last_unstale":"2022-09-10T04:27:09.427305+0000","last_undegraded":"2022-09-10T04:27:09.427305+0000","last_fullsized":"2022-09-10T04:27:09.427305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T08:59:31.105825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429329+0000","last_change":"2022-09-10T04:26:36.602214+0000","last_active":"2022-09-10T04:27:09.429329+0000","last_peered":"2022-09-10T04:27:09.429329+0000","last_clean":"2022-09-10T04:27:09.429329+0000","last_became_active":"2022-09-10T04:26:36.601785+0000","last_became_peered":"2022-09-10T04:26:36.601785+0000","last_unstale":"2022-09-10T04:27:09.429329+0000","last_undegraded":"2022-09-10T04:27:09.429329+0000","last_fullsized":"2022-09-10T04:27:09.429329+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:52:42.795971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.426294+0000","last_change":"2022-09-10T04:26:31.556182+0000","last_active":"2022-09-10T04:27:09.426294+0000","last_peered":"2022-09-10T04:27:09.426294+0000","last_clean":"2022-09-10T04:27:09.426294+0000","last_became_active":"2022-09-10T04:26:31.555873+0000","last_became_peered":"2022-09-10T04:26:31.555873+0000","last_unstale":"2022-09-10T04:27:09.426294+0000","last_undegraded":"2022-09-10T04:27:09.426294+0000","last_fullsized":"2022-09-10T04:27:09.426294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:56:49.406476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427147+0000","last_change":"2022-09-10T04:26:32.134925+0000","last_active":"2022-09-10T04:27:09.427147+0000","last_peered":"2022-09-10T04:27:09.427147+0000","last_clean":"2022-09-10T04:27:09.427147+0000","last_became_active":"2022-09-10T04:26:32.134581+0000","last_became_peered":"2022-09-10T04:26:32.134581+0000","last_unstale":"2022-09-10T04:27:09.427147+0000","last_undegraded":"2022-09-10T04:27:09.427147+0000","last_fullsized":"2022-09-10T04:27:09.427147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:06:09.815252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425087+0000","last_change":"2022-09-10T04:26:36.590902+0000","last_active":"2022-09-10T04:27:09.425087+0000","last_peered":"2022-09-10T04:27:09.425087+0000","last_clean":"2022-09-10T04:27:09.425087+0000","last_became_active":"2022-09-10T04:26:36.590468+0000","last_became_peered":"2022-09-10T04:26:36.590468+0000","last_unstale":"2022-09-10T04:27:09.425087+0000","last_undegraded":"2022-09-10T04:27:09.425087+0000","last_fullsized":"2022-09-10T04:27:09.425087+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:35.566588+0000","last_clean_scrub_stamp":"2022-09-10T04:26:35.566588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:49:40.841273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427621+0000","last_change":"2022-09-10T04:26:31.554090+0000","last_active":"2022-09-10T04:27:09.427621+0000","last_peered":"2022-09-10T04:27:09.427621+0000","last_clean":"2022-09-10T04:27:09.427621+0000","last_became_active":"2022-09-10T04:26:31.553675+0000","last_became_peered":"2022-09-10T04:26:31.553675+0000","last_unstale":"2022-09-10T04:27:09.427621+0000","last_undegraded":"2022-09-10T04:27:09.427621+0000","last_fullsized":"2022-09-10T04:27:09.427621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:30.287151+0000","last_clean_scrub_stamp":"2022-09-10T04:26:30.287151+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:35:33.619937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427241+0000","last_change":"2022-09-10T04:26:32.118923+0000","last_active":"2022-09-10T04:27:09.427241+0000","last_peered":"2022-09-10T04:27:09.427241+0000","last_clean":"2022-09-10T04:27:09.427241+0000","last_became_active":"2022-09-10T04:26:32.118560+0000","last_became_peered":"2022-09-10T04:26:32.118560+0000","last_unstale":"2022-09-10T04:27:09.427241+0000","last_undegraded":"2022-09-10T04:27:09.427241+0000","last_fullsized":"2022-09-10T04:27:09.427241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T16:24:03.377179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.420987+0000","last_change":"2022-09-10T04:26:32.115339+0000","last_active":"2022-09-10T04:27:09.420987+0000","last_peered":"2022-09-10T04:27:09.420987+0000","last_clean":"2022-09-10T04:27:09.420987+0000","last_became_active":"2022-09-10T04:26:32.115044+0000","last_became_peered":"2022-09-10T04:26:32.115044+0000","last_unstale":"2022-09-10T04:27:09.420987+0000","last_undegraded":"2022-09-10T04:27:09.420987+0000","last_fullsized":"2022-09-10T04:27:09.420987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T07:06:39.719939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418101+0000","last_change":"2022-09-10T04:26:32.121701+0000","last_active":"2022-09-10T04:27:09.418101+0000","last_peered":"2022-09-10T04:27:09.418101+0000","last_clean":"2022-09-10T04:27:09.418101+0000","last_became_active":"2022-09-10T04:26:32.121417+0000","last_became_peered":"2022-09-10T04:26:32.121417+0000","last_unstale":"2022-09-10T04:27:09.418101+0000","last_undegraded":"2022-09-10T04:27:09.418101+0000","last_fullsized":"2022-09-10T04:27:09.418101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T05:16:44.194677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.417864+0000","last_change":"2022-09-10T04:26:32.139762+0000","last_active":"2022-09-10T04:27:09.417864+0000","last_peered":"2022-09-10T04:27:09.417864+0000","last_clean":"2022-09-10T04:27:09.417864+0000","last_became_active":"2022-09-10T04:26:32.139468+0000","last_became_peered":"2022-09-10T04:26:32.139468+0000","last_unstale":"2022-09-10T04:27:09.417864+0000","last_undegraded":"2022-09-10T04:27:09.417864+0000","last_fullsized":"2022-09-10T04:27:09.417864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:37:25.448977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.429667+0000","last_change":"2022-09-10T04:26:32.131899+0000","last_active":"2022-09-10T04:27:09.429667+0000","last_peered":"2022-09-10T04:27:09.429667+0000","last_clean":"2022-09-10T04:27:09.429667+0000","last_became_active":"2022-09-10T04:26:32.130047+0000","last_became_peered":"2022-09-10T04:26:32.130047+0000","last_unstale":"2022-09-10T04:27:09.429667+0000","last_undegraded":"2022-09-10T04:27:09.429667+0000","last_fullsized":"2022-09-10T04:27:09.429667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:49:48.001008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427928+0000","last_change":"2022-09-10T04:26:32.126445+0000","last_active":"2022-09-10T04:27:09.427928+0000","last_peered":"2022-09-10T04:27:09.427928+0000","last_clean":"2022-09-10T04:27:09.427928+0000","last_became_active":"2022-09-10T04:26:32.125935+0000","last_became_peered":"2022-09-10T04:26:32.125935+0000","last_unstale":"2022-09-10T04:27:09.427928+0000","last_undegraded":"2022-09-10T04:27:09.427928+0000","last_fullsized":"2022-09-10T04:27:09.427928+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:27:44.987973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2022-09-10T04:27:10.427571+0000","last_change":"2022-09-10T04:27:10.427571+0000","last_active":"2022-09-10T04:27:10.427571+0000","last_peered":"2022-09-10T04:27:10.427571+0000","last_clean":"2022-09-10T04:27:10.427571+0000","last_became_active":"2022-09-10T04:27:10.427153+0000","last_became_peered":"2022-09-10T04:27:10.427153+0000","last_unstale":"2022-09-10T04:27:10.427571+0000","last_undegraded":"2022-09-10T04:27:10.427571+0000","last_fullsized":"2022-09-10T04:27:10.427571+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:23:54.710033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,9],"acting":[11,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.418703+0000","last_change":"2022-09-10T04:26:32.122790+0000","last_active":"2022-09-10T04:27:09.418703+0000","last_peered":"2022-09-10T04:27:09.418703+0000","last_clean":"2022-09-10T04:27:09.418703+0000","last_became_active":"2022-09-10T04:26:32.122379+0000","last_became_peered":"2022-09-10T04:26:32.122379+0000","last_unstale":"2022-09-10T04:27:09.418703+0000","last_undegraded":"2022-09-10T04:27:09.418703+0000","last_fullsized":"2022-09-10T04:27:09.418703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:36:19.747563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424872+0000","last_change":"2022-09-10T04:26:32.145947+0000","last_active":"2022-09-10T04:27:09.424872+0000","last_peered":"2022-09-10T04:27:09.424872+0000","last_clean":"2022-09-10T04:27:09.424872+0000","last_became_active":"2022-09-10T04:26:32.145570+0000","last_became_peered":"2022-09-10T04:26:32.145570+0000","last_unstale":"2022-09-10T04:27:09.424872+0000","last_undegraded":"2022-09-10T04:27:09.424872+0000","last_fullsized":"2022-09-10T04:27:09.424872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T06:38:56.028983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425606+0000","last_change":"2022-09-10T04:26:32.140019+0000","last_active":"2022-09-10T04:27:09.425606+0000","last_peered":"2022-09-10T04:27:09.425606+0000","last_clean":"2022-09-10T04:27:09.425606+0000","last_became_active":"2022-09-10T04:26:32.139709+0000","last_became_peered":"2022-09-10T04:26:32.139709+0000","last_unstale":"2022-09-10T04:27:09.425606+0000","last_undegraded":"2022-09-10T04:27:09.425606+0000","last_fullsized":"2022-09-10T04:27:09.425606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T04:56:11.850506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.424959+0000","last_change":"2022-09-10T04:26:32.145867+0000","last_active":"2022-09-10T04:27:09.424959+0000","last_peered":"2022-09-10T04:27:09.424959+0000","last_clean":"2022-09-10T04:27:09.424959+0000","last_became_active":"2022-09-10T04:26:32.145582+0000","last_became_peered":"2022-09-10T04:26:32.145582+0000","last_unstale":"2022-09-10T04:27:09.424959+0000","last_undegraded":"2022-09-10T04:27:09.424959+0000","last_fullsized":"2022-09-10T04:27:09.424959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:30:11.784617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.425612+0000","last_change":"2022-09-10T04:26:32.134641+0000","last_active":"2022-09-10T04:27:09.425612+0000","last_peered":"2022-09-10T04:27:09.425612+0000","last_clean":"2022-09-10T04:27:09.425612+0000","last_became_active":"2022-09-10T04:26:32.134320+0000","last_became_peered":"2022-09-10T04:26:32.134320+0000","last_unstale":"2022-09-10T04:27:09.425612+0000","last_undegraded":"2022-09-10T04:27:09.425612+0000","last_fullsized":"2022-09-10T04:27:09.425612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T09:58:29.800916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.427023+0000","last_change":"2022-09-10T04:26:32.126597+0000","last_active":"2022-09-10T04:27:09.427023+0000","last_peered":"2022-09-10T04:27:09.427023+0000","last_clean":"2022-09-10T04:27:09.427023+0000","last_became_active":"2022-09-10T04:26:32.125895+0000","last_became_peered":"2022-09-10T04:26:32.125895+0000","last_unstale":"2022-09-10T04:27:09.427023+0000","last_undegraded":"2022-09-10T04:27:09.427023+0000","last_fullsized":"2022-09-10T04:27:09.427023+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T12:17:41.456304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2022-09-10T04:27:09.422818+0000","last_change":"2022-09-10T04:26:32.122525+0000","last_active":"2022-09-10T04:27:09.422818+0000","last_peered":"2022-09-10T04:27:09.422818+0000","last_clean":"2022-09-10T04:27:09.422818+0000","last_became_active":"2022-09-10T04:26:32.122113+0000","last_became_peered":"2022-09-10T04:26:32.122113+0000","last_unstale":"2022-09-10T04:27:09.422818+0000","last_undegraded":"2022-09-10T04:27:09.422818+0000","last_fullsized":"2022-09-10T04:27:09.422818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-10T04:26:31.529403+0000","last_clean_scrub_stamp":"2022-09-10T04:26:31.529403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-09-11T14:29:19.351147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9899,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":505,"num_read_kb":270,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":29697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351003,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6136,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739016,"statfs":{"total":95995035648,"available":95988752384,"internally_reserved":0,"allocated":1032192,"data_stored":498408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.63},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.64500000000000002}]},{"osd":1,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.67400000000000004}]},{"osd":5,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.72199999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.79000000000000004}]},{"osd":7,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.73899999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.746}]},{"osd":9,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.82299999999999995}]}]},{"osd":10,"up_from":73,"seq":313532612640,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7012,"kb_used_data":1628,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738140,"statfs":{"total":95995035648,"available":95987855360,"internally_reserved":0,"allocated":1667072,"data_stored":1092746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.5709999999999997,"5min":5.5709999999999997,"15min":5.5709999999999997},"last":0.60199999999999998}]},{"osd":1,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.33600000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64200000000000002,"15min":0.65300000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.80600000000000005,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.59399999999999997,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.79500000000000004,"5min":1.147,"15min":1.147},"last":0.65400000000000003}]},{"osd":3,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.67100000000000004,"15min":0.69099999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":1.9490000000000001,"15min":1.9490000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.74399999999999999,"15min":0.72399999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"last":0.69599999999999995}]},{"osd":4,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.35899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84199999999999997,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.57099999999999995}]},{"osd":5,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.70199999999999996,"15min":0.71199999999999997},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.96099999999999997,"5min":1.903,"15min":1.903},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.40799999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94199999999999995,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":6,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":1.056,"15min":1.121},"min":{"1min":0.49199999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":9.7629999999999999,"15min":9.7629999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.1000000000000001,"15min":1.1619999999999999},"min":{"1min":0.503,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.91200000000000003,"5min":9.7349999999999994,"15min":9.7349999999999994},"last":0.81499999999999995}]},{"osd":7,"last update":"Sat Sep 10 04:26:20 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.873}]},{"osd":8,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.753,"15min":0.75900000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0049999999999999,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.748},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.77100000000000002,"15min":0.78300000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.98399999999999999,"5min":1.9259999999999999,"15min":1.9259999999999999},"last":0.621}]},{"osd":9,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.89800000000000002,"15min":0.88},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":5.3289999999999997,"5min":5.3289999999999997,"15min":5.3289999999999997},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.81100000000000005,"15min":0.82199999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.087,"5min":1.972,"15min":1.972},"last":0.79500000000000004}]},{"osd":11,"last update":"Sat Sep 10 04:26:44 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.75800000000000001}]}]},{"osd":9,"up_from":68,"seq":292057776164,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6352,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738800,"statfs":{"total":95995035648,"available":95988531200,"internally_reserved":0,"allocated":1056768,"data_stored":498386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83799999999999997,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.69399999999999995,"15min":0.70599999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84499999999999997,"5min":1.024,"15min":1.024},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.79600000000000004,"15min":0.81699999999999995},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.71099999999999997}]},{"osd":2,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.79600000000000004,"15min":0.81799999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.84099999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.754},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.86299999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.747,"15min":0.76000000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.78400000000000003,"15min":0.80100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.89800000000000002,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.51400000000000001}]},{"osd":4,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.80300000000000005,"15min":0.83199999999999996},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.82599999999999996,"5min":1.272,"15min":1.272},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.81100000000000005,"15min":0.83299999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.85199999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.75600000000000001}]},{"osd":5,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.86399999999999999,"15min":0.89500000000000002},"min":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.879,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.88400000000000001,"15min":0.91400000000000003},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.85599999999999998,"5min":1.262,"15min":1.262},"last":0.65200000000000002}]},{"osd":6,"last update":"Sat Sep 10 04:25:53 2022","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.79700000000000004}]},{"osd":7,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.93700000000000006,"15min":0.97299999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.89100000000000001,"5min":1.429,"15min":1.429},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.97999999999999998,"15min":1.018},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.95199999999999996,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.73899999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:26:54 2022","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.0189999999999999,"15min":1.0609999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.96899999999999997,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.96699999999999997,"15min":1.0089999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.91800000000000004,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.59999999999999998}]},{"osd":10,"last update":"Sat Sep 10 04:27:28 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.89000000000000001,"15min":0.91200000000000003},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.96699999999999997,"5min":1.278,"15min":1.278},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.98499999999999999,"15min":1.0189999999999999},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":0.95199999999999996,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78900000000000003}]},{"osd":11,"last update":"Sat Sep 10 04:27:39 2022","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.76900000000000002}]}]},{"osd":8,"up_from":60,"seq":257698037803,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6540,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738612,"statfs":{"total":95995035648,"available":95988338688,"internally_reserved":0,"allocated":1052672,"data_stored":498748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55300000000000005,"15min":0.54900000000000004},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.67800000000000005}]},{"osd":1,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.36199999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.55000000000000004,"15min":0.54900000000000004},"min":{"1min":0.34499999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.441,"15min":0.439},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.753,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.45100000000000001,"15min":0.44700000000000001},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56299999999999994}]},{"osd":3,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59199999999999997,"15min":0.59799999999999998},"min":{"1min":0.32900000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.63},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.624,"15min":0.61599999999999999},"min":{"1min":0.41299999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.78700000000000003}]},{"osd":4,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.503},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.63900000000000001,"15min":0.63300000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.748}]},{"osd":5,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53300000000000003,"15min":0.53100000000000003},"min":{"1min":0.32400000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.35199999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.88800000000000001,"5min":1.046,"15min":1.046},"last":0.72399999999999998}]},{"osd":6,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.70299999999999996,"15min":0.70999999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.69299999999999995,"15min":0.69499999999999995},"min":{"1min":0.46400000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.77000000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.377,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.38600000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.93700000000000006,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.75800000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:26:57 2022","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.74299999999999999,"15min":0.747},"min":{"1min":0.503,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.74299999999999999,"15min":0.749},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.80700000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.68799999999999994,"15min":0.68000000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68999999999999995,"15min":0.67600000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.88800000000000001}]},{"osd":11,"last update":"Sat Sep 10 04:27:33 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.86599999999999999}]}]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7180,"kb_used_data":1604,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93737972,"statfs":{"total":95995035648,"available":95987683328,"internally_reserved":0,"allocated":1642496,"data_stored":1090291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:00 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.77100000000000002,"15min":0.79900000000000004},"min":{"1min":0.41299999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":5.3719999999999999,"15min":5.3719999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.44500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.75,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.51000000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53500000000000003,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.433,"15min":0.439},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.55900000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.44}]},{"osd":2,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.57199999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72499999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.38600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.78600000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.622,"15min":0.626},"min":{"1min":0.42099999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80400000000000005,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60699999999999998,"15min":0.60899999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76000000000000001,"5min":0.875,"15min":0.875},"last":0.57299999999999995}]},{"osd":4,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48699999999999999,"15min":0.49399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.623},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.49399999999999999,"15min":0.51200000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67100000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47199999999999998}]},{"osd":5,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.61699999999999999,"15min":0.63200000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.81499999999999995,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.63800000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83499999999999996,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.66500000000000004,"15min":0.66400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68100000000000005,"15min":0.69299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.93000000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.75}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.68799999999999994,"15min":0.69599999999999995},"min":{"1min":0.48699999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.875,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.66900000000000004,"15min":0.67800000000000005},"min":{"1min":0.52100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.88300000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63500000000000001}]},{"osd":9,"last update":"Sat Sep 10 04:26:59 2022","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68799999999999994}]},{"osd":10,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.45500000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.68500000000000005,"15min":0.68100000000000005},"min":{"1min":0.48399999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.91200000000000003,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.70599999999999996}]},{"osd":11,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.77100000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6460,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738692,"statfs":{"total":95995035648,"available":95988420608,"internally_reserved":0,"allocated":1036288,"data_stored":497954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51300000000000001,"15min":0.48899999999999999},"min":{"1min":0.41399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71799999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54400000000000004,"15min":0.51800000000000002},"min":{"1min":0.376,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77100000000000002,"5min":1.046,"15min":1.046},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66300000000000003,"15min":0.63900000000000001},"min":{"1min":0.51200000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.96499999999999997,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":1.004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.69499999999999995,"15min":0.67200000000000004},"min":{"1min":0.502,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.012,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.747}]},{"osd":2,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.63800000000000001,"15min":0.61799999999999999},"min":{"1min":0.49199999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.71099999999999997,"15min":0.68700000000000006},"min":{"1min":0.51900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.135,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.875}]},{"osd":3,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70799999999999996,"15min":0.68000000000000005},"min":{"1min":0.51600000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.056,"5min":1.103,"15min":1.103},"last":1.153},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.621,"15min":0.59299999999999997},"min":{"1min":0.42799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.115}]},{"osd":4,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.69799999999999995,"15min":0.68200000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.98699999999999999,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.76600000000000001,"15min":0.76100000000000001},"min":{"1min":0.63,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.97599999999999998,"5min":1.167,"15min":1.167},"last":0.81299999999999994}]},{"osd":5,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.71599999999999997,"15min":0.69599999999999995},"min":{"1min":0.57899999999999996,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93999999999999995,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.72099999999999997,"15min":0.69899999999999995},"min":{"1min":0.54600000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.99299999999999999,"5min":1.036,"15min":1.036},"last":1.046}]},{"osd":7,"last update":"Sat Sep 10 04:27:05 2022","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79100000000000004,"15min":0.76300000000000001},"min":{"1min":0.61799999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.115,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83499999999999996,"15min":0.80800000000000005},"min":{"1min":0.64900000000000002,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.171,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.84099999999999997}]},{"osd":8,"last update":"Sat Sep 10 04:27:30 2022","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.85999999999999999,"15min":0.85299999999999998},"min":{"1min":0.67000000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.147,"5min":1.181,"15min":1.181},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73699999999999999,"15min":0.71499999999999997},"min":{"1min":0.56399999999999995,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.99399999999999999,"5min":1.097,"15min":1.097},"last":1.081}]},{"osd":9,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":1.0069999999999999,"15min":1.0289999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.2,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.214},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.89100000000000001,"15min":0.90200000000000002},"min":{"1min":0.66400000000000003,"5min":0.624,"15min":0.624},"max":{"1min":1.1259999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":1.2909999999999999}]},{"osd":10,"last update":"Sat Sep 10 04:27:24 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.90600000000000003,"15min":0.91800000000000004},"min":{"1min":0.71799999999999997,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.0289999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":1.052},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.84999999999999998,"15min":0.85799999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.0669999999999999,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":1.4399999999999999}]},{"osd":11,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":1.3879999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":1.173}]}]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7360,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737792,"statfs":{"total":95995035648,"available":95987499008,"internally_reserved":0,"allocated":1040384,"data_stored":502436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:29 2022","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55400000000000005,"15min":0.52300000000000002},"min":{"1min":0.44900000000000001,"5min":0.314,"15min":0.312},"max":{"1min":0.85299999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54800000000000004,"15min":0.52900000000000003},"min":{"1min":0.435,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81200000000000006,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.66700000000000004}]},{"osd":2,"last update":"Sat Sep 10 04:26:57 2022","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.48099999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81399999999999995,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.55100000000000005,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83499999999999996,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":0.66200000000000003}]},{"osd":3,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.39000000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.63600000000000001,"15min":0.60199999999999998},"min":{"1min":0.48799999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.7969999999999999,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.629}]},{"osd":4,"last update":"Sat Sep 10 04:26:41 2022","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.42599999999999999,"15min":0.40500000000000003},"min":{"1min":0.27200000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.70199999999999996,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.32400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.434}]},{"osd":5,"last update":"Sat Sep 10 04:27:09 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.51200000000000001},"min":{"1min":0.46100000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.91100000000000003,"5min":1.577,"15min":1.577},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.39500000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91700000000000004,"5min":1.583,"15min":1.583},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.629,"15min":0.60699999999999998},"min":{"1min":0.42899999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.65800000000000003,"15min":0.625},"min":{"1min":0.45700000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.435}]},{"osd":7,"last update":"Sat Sep 10 04:27:00 2022","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.57999999999999996,"15min":0.56699999999999995},"min":{"1min":0.33000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79900000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.505},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.36499999999999999}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.66100000000000003,"15min":0.621},"min":{"1min":0.497,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98799999999999999,"5min":1.5960000000000001,"15min":1.5960000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.68200000000000005,"15min":0.65400000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.97499999999999998,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.59299999999999997}]},{"osd":9,"last update":"Sat Sep 10 04:26:57 2022","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.74399999999999999,"15min":0.73999999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.88,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.56000000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96499999999999997,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.51400000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.53300000000000003}]},{"osd":11,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.69499999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":1060864,"data_stored":499378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 10 04:27:20 2022","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.58199999999999996,"15min":0.84399999999999997},"min":{"1min":0.39500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":8.3729999999999993},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.80300000000000005}]},{"osd":2,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.61399999999999999,"15min":0.60299999999999998},"min":{"1min":0.46400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0209999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.749},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69899999999999995,"15min":0.88300000000000001},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96699999999999997,"5min":8.3260000000000005,"15min":8.3260000000000005},"last":0.86199999999999999}]},{"osd":3,"last update":"Sat Sep 10 04:27:34 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.622}]},{"osd":4,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63100000000000001,"15min":0.57499999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.628,"15min":0.58599999999999997},"min":{"1min":0.435,"5min":0.372,"15min":0.372},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69399999999999995}]},{"osd":5,"last update":"Sat Sep 10 04:27:02 2022","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.66000000000000003,"15min":0.61399999999999999},"min":{"1min":0.56200000000000006,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.67400000000000004,"15min":0.625},"min":{"1min":0.48299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.83599999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:27:42 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.74199999999999999,"15min":0.67300000000000004},"min":{"1min":0.67600000000000005,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.77000000000000002,"15min":0.70299999999999996},"min":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.798,"5min":1.798,"15min":1.798},"last":1.0109999999999999}]},{"osd":7,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.93999999999999995}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.81899999999999995,"15min":0.80500000000000005},"min":{"1min":0.59899999999999998,"5min":0.495,"15min":0.495},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.81399999999999995,"15min":0.79200000000000004},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.90300000000000002}]},{"osd":9,"last update":"Sat Sep 10 04:26:56 2022","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.88700000000000001,"15min":0.90500000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.046,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.91100000000000003,"15min":0.92700000000000005},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.099,"5min":1.21,"15min":1.21},"last":0.94699999999999995}]},{"osd":10,"last update":"Sat Sep 10 04:27:20 2022","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.91100000000000003,"15min":0.90400000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.91500000000000004,"15min":0.91400000000000003},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.29,"5min":1.306,"15min":1.306},"last":0.84499999999999997}]},{"osd":11,"last update":"Sat Sep 10 04:26:47 2022","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":1.02},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":1.052}]}]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7444,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737708,"statfs":{"total":95995035648,"available":95987412992,"internally_reserved":0,"allocated":1060864,"data_stored":502624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68200000000000005,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.33500000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72899999999999998,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.83899999999999997}]},{"osd":1,"last update":"Sat Sep 10 04:26:49 2022","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40400000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77200000000000002,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52600000000000002,"15min":0.504},"min":{"1min":0.378,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71299999999999997,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.622}]},{"osd":3,"last update":"Sat Sep 10 04:27:21 2022","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.36599999999999999,"5min":0.311,"15min":0.311},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.372},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.876}]},{"osd":4,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.751},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.90300000000000002}]},{"osd":5,"last update":"Sat Sep 10 04:27:13 2022","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.42699999999999999,"15min":0.41499999999999998},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.62,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.47199999999999998,"15min":0.44400000000000001},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80500000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.92800000000000005}]},{"osd":6,"last update":"Sat Sep 10 04:27:39 2022","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":0.74099999999999999,"15min":0.70599999999999996},"min":{"1min":0.48699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.6459999999999999,"5min":4.6459999999999999,"15min":4.6459999999999999},"last":1.712},{"interface":"front","average":{"1min":1.1100000000000001,"5min":0.73499999999999999,"15min":0.68500000000000005},"min":{"1min":0.51100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":4.6310000000000002,"5min":4.6310000000000002,"15min":4.6310000000000002},"last":0.97199999999999998}]},{"osd":7,"last update":"Sat Sep 10 04:27:36 2022","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.89700000000000002}]},{"osd":8,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.54700000000000004,"15min":0.53700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.622,"15min":0.61399999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.96599999999999997}]},{"osd":9,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72399999999999998,"15min":0.72099999999999997},"min":{"1min":0.496,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.0189999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":1.022},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93500000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":1.0449999999999999}]},{"osd":10,"last update":"Sat Sep 10 04:27:23 2022","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.70599999999999996,"15min":0.68799999999999994},"min":{"1min":0.53000000000000003,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.98199999999999998}]},{"osd":11,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.0009999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6728,"kb_used_data":1024,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738424,"statfs":{"total":95995035648,"available":95988146176,"internally_reserved":0,"allocated":1048576,"data_stored":497710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.56299999999999994,"15min":0.45500000000000002},"min":{"1min":0.372,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":11.026999999999999,"5min":11.026999999999999,"15min":11.026999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.497,"15min":0.45400000000000001},"min":{"1min":0.35699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.47699999999999998}]},{"osd":1,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67500000000000004,"15min":0.623},"min":{"1min":0.439,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.3919999999999999,"5min":1.494,"15min":1.494},"last":0.747},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.65400000000000003,"15min":0.61599999999999999},"min":{"1min":0.54800000000000004,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.238,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Sep 10 04:27:18 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.64300000000000002,"15min":0.60299999999999998},"min":{"1min":0.46600000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93400000000000005,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69499999999999995,"15min":0.65500000000000003},"min":{"1min":0.50900000000000001,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0880000000000001,"5min":1.454,"15min":1.454},"last":0.70499999999999996}]},{"osd":4,"last update":"Sat Sep 10 04:26:46 2022","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64400000000000002,"15min":0.626},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84599999999999997,"5min":1.4710000000000001,"15min":1.4710000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"min":{"1min":0.45800000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.95499999999999996,"5min":1.528,"15min":1.528},"last":0.66900000000000004}]},{"osd":5,"last update":"Sat Sep 10 04:27:04 2022","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.67900000000000005,"15min":0.66900000000000004},"min":{"1min":0.54300000000000004,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94599999999999995,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.68500000000000005,"15min":0.67500000000000004},"min":{"1min":0.5,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.93799999999999994,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.72099999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.69499999999999995,"15min":0.65400000000000003},"min":{"1min":0.46100000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.755,"15min":0.71699999999999997},"min":{"1min":0.57799999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.8340000000000001,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.63700000000000001}]},{"osd":7,"last update":"Sat Sep 10 04:27:04 2022","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.72599999999999998,"15min":0.71399999999999997},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95299999999999996,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76400000000000001,"15min":0.76000000000000001},"min":{"1min":0.57899999999999996,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0649999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.79000000000000004}]},{"osd":8,"last update":"Sat Sep 10 04:26:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.56699999999999995,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.1060000000000001,"5min":1.208,"15min":1.208},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.79200000000000004,"15min":0.78600000000000003},"min":{"1min":0.64500000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.099,"5min":1.139,"15min":1.139},"last":0.73299999999999998}]},{"osd":9,"last update":"Sat Sep 10 04:27:00 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.91200000000000003,"15min":0.92500000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0289999999999999,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85099999999999998,"15min":0.85599999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.80500000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:27:24 2022","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.879,"15min":0.88500000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.86699999999999999,"15min":0.86599999999999999},"min":{"1min":0.67600000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.81599999999999995}]},{"osd":11,"last update":"Sat Sep 10 04:27:35 2022","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.81000000000000005}]}]},{"osd":4,"up_from":34,"seq":146028888130,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6592,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738560,"statfs":{"total":95995035648,"available":95988285440,"internally_reserved":0,"allocated":1040384,"data_stored":497932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:25:39 2022","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.51600000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73199999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54500000000000004,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Sep 10 04:27:43 2022","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.39700000000000002,"15min":0.34999999999999998},"min":{"1min":0.32000000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55300000000000005,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.41199999999999998,"15min":0.37},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66200000000000003,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.438}]},{"osd":2,"last update":"Sat Sep 10 04:27:43 2022","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54900000000000004,"15min":0.52400000000000002},"min":{"1min":0.46400000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55600000000000005,"15min":0.52700000000000002},"min":{"1min":0.42199999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69899999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Sep 10 04:27:43 2022","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61099999999999999,"15min":0.56399999999999995},"min":{"1min":0.48399999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.81399999999999995,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.624},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.39000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77900000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48999999999999999}]},{"osd":5,"last update":"Sat Sep 10 04:27:09 2022","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59099999999999997,"15min":0.54800000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76300000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57099999999999995,"15min":0.55400000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Sat Sep 10 04:27:43 2022","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65300000000000002,"15min":0.66900000000000004},"min":{"1min":0.44700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83499999999999996,"5min":1.123,"15min":1.123},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.76200000000000001,"15min":0.73399999999999999},"min":{"1min":0.51200000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91400000000000003,"5min":5.5880000000000001,"15min":5.5880000000000001},"last":0.77100000000000002}]},{"osd":7,"last update":"Sat Sep 10 04:27:03 2022","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.38600000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84299999999999997,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.747},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54700000000000004,"15min":0.54400000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81599999999999995,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.56399999999999995}]},{"osd":8,"last update":"Sat Sep 10 04:27:26 2022","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72199999999999998,"15min":0.71999999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.88700000000000001,"5min":1.133,"15min":1.133},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.67200000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93700000000000006,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.65400000000000003}]},{"osd":9,"last update":"Sat Sep 10 04:26:58 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.78500000000000003,"15min":0.76500000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.80500000000000005,"15min":0.81699999999999995},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91000000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.81100000000000005}]},{"osd":10,"last update":"Sat Sep 10 04:27:20 2022","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94699999999999995,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.67200000000000004,"15min":0.67800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.84999999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.72099999999999997}]},{"osd":11,"last update":"Sat Sep 10 04:27:32 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.754}]}]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7320,"kb_used_data":1616,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93737832,"statfs":{"total":95995035648,"available":95987539968,"internally_reserved":0,"allocated":1654784,"data_stored":1090049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53700000000000003,"15min":0.52500000000000002},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74199999999999999,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71099999999999997,"5min":1.879,"15min":1.879},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.52300000000000002,"15min":0.48799999999999999},"min":{"1min":0.38,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79300000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.52800000000000002,"15min":0.48899999999999999},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70799999999999996,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.373},"min":{"1min":0.29299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.307,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.434}]},{"osd":3,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.42099999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72499999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.56999999999999995},"min":{"1min":0.47399999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73999999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.58599999999999997}]},{"osd":4,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.54800000000000004,"15min":0.51400000000000001},"min":{"1min":0.41199999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.53700000000000003,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76700000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Sep 10 04:27:37 2022","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.76800000000000002,"15min":0.64900000000000002},"min":{"1min":0.504,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":7.9470000000000001,"5min":7.9470000000000001,"15min":7.9470000000000001},"last":0.504},{"interface":"front","average":{"1min":1.381,"5min":0.79600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":7.9029999999999996,"5min":7.9029999999999996,"15min":7.9029999999999996},"last":0.621}]},{"osd":7,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57999999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.68000000000000005}]},{"osd":8,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56699999999999995,"15min":0.57699999999999996},"min":{"1min":0.41499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.56999999999999995},"min":{"1min":0.41499999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76900000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55300000000000005}]},{"osd":9,"last update":"Sat Sep 10 04:27:01 2022","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.57299999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.52700000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.73099999999999998}]},{"osd":10,"last update":"Sat Sep 10 04:27:25 2022","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.63,"15min":0.61699999999999999},"min":{"1min":0.56299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.59899999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":5421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2022-09-10T04:27:48.284 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2022-09-10T04:27:48.284 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-09-10T04:27:48.284 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2022-09-10T04:27:48.285 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph health --format=json 2022-09-10T04:27:48.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:48 smithi081 ceph-mon[90133]: pgmap v341: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:48.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:48 smithi081 ceph-mon[90133]: from='client.25015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:48.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:48 smithi104 ceph-mon[97044]: pgmap v341: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:48 smithi104 ceph-mon[97044]: from='client.25015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:48 smithi110 ceph-mon[97633]: pgmap v341: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:48 smithi110 ceph-mon[97633]: from='client.25015 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2022-09-10T04:27:49.165 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:50.680 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 -- 172.21.15.81:0/1572585153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec1053e0 msgr2=0x7fb9ec10cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 --2- 172.21.15.81:0/1572585153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec10cf80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4009be0 tx=0x7fb9e4009ef0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 -- 172.21.15.81:0/1572585153 shutdown_connections 2022-09-10T04:27:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 --2- 172.21.15.81:0/1572585153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec10cf80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.681 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 --2- 172.21.15.81:0/1572585153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 --2- 172.21.15.81:0/1572585153 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec100c80 0x7fb9ec104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 -- 172.21.15.81:0/1572585153 >> 172.21.15.81:0/1572585153 conn(0x7fb9ec0fbc20 msgr2=0x7fb9ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:50.682 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 -- 172.21.15.81:0/1572585153 shutdown_connections 2022-09-10T04:27:50.683 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.678+0000 7fb9f1521700 1 -- 172.21.15.81:0/1572585153 wait complete. 2022-09-10T04:27:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 Processor -- start 2022-09-10T04:27:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 -- start start 2022-09-10T04:27:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:50.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 0x7fb9ec196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec19ae70 con 0x7fb9ec100c80 2022-09-10T04:27:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec19afe0 con 0x7fb9ec1053e0 2022-09-10T04:27:50.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.680+0000 7fb9f1521700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec19b1a0 con 0x7fb9ec1002e0 2022-09-10T04:27:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9ea7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 0x7fb9ec196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36740/0 (socket says 172.21.15.81:36740) 2022-09-10T04:27:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9ea7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 0x7fb9ec196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59016/0 (socket says 172.21.15.81:59016) 2022-09-10T04:27:50.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 -- 172.21.15.81:0/3929536013 learned_addr learned my addr 172.21.15.81:0/3929536013 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eb7fe700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 -- 172.21.15.81:0/3929536013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec1053e0 msgr2=0x7fb9ec19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec19a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 -- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 msgr2=0x7fb9ec196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 0x7fb9ec196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eaffd700 1 -- 172.21.15.81:0/3929536013 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e40097e0 con 0x7fb9ec1002e0 2022-09-10T04:27:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9eb7fe700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.681+0000 7fb9ea7fc700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 0x7fb9ec196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:50.689 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.682+0000 7fb9eaffd700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec195e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00e740 tx=0x7fb9dc00ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.682+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc005220 con 0x7fb9ec1002e0 2022-09-10T04:27:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.682+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb9dc005380 con 0x7fb9ec1002e0 2022-09-10T04:27:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.682+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec19b480 con 0x7fb9ec1002e0 2022-09-10T04:27:50.690 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.682+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc004b70 con 0x7fb9ec1002e0 2022-09-10T04:27:50.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.682+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec1a2cc0 con 0x7fb9ec1002e0 2022-09-10T04:27:50.691 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.683+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec064ea0 con 0x7fb9ec1002e0 2022-09-10T04:27:50.692 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.685+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb9dc01e020 con 0x7fb9ec1002e0 2022-09-10T04:27:50.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.686+0000 7fb9cffff700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9d4099090 0x7fb9d409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:50.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.686+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fb9dc014070 con 0x7fb9ec1002e0 2022-09-10T04:27:50.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.687+0000 7fb9ea7fc700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9d4099090 0x7fb9d409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:50.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.687+0000 7fb9ea7fc700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9d4099090 0x7fb9d409b540 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec197450 tx=0x7fb9e0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:50.694 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.688+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb9dc08ef70 con 0x7fb9ec1002e0 2022-09-10T04:27:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:50 smithi104 ceph-mon[97044]: pgmap v342: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:50.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:50 smithi081 ceph-mon[90133]: pgmap v342: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:50 smithi110 ceph-mon[97633]: pgmap v342: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:50.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.939+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb9ec10fef0 con 0x7fb9ec1002e0 2022-09-10T04:27:50.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.940+0000 7fb9cffff700 1 -- 172.21.15.81:0/3929536013 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb9dc092590 con 0x7fb9ec1002e0 2022-09-10T04:27:50.942 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:50.942 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-09-10T04:27:50.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9d4099090 msgr2=0x7fb9d409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:50.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9d4099090 0x7fb9d409b540 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec197450 tx=0x7fb9e0005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 msgr2=0x7fb9ec195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:50.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec195e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00e740 tx=0x7fb9dc00ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:50.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 shutdown_connections 2022-09-10T04:27:50.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9ec1053e0 0x7fb9ec19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb9d4099090 0x7fb9d409b540 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9ec1002e0 0x7fb9ec195e40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 --2- 172.21.15.81:0/3929536013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9ec100c80 0x7fb9ec196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:50.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 >> 172.21.15.81:0/3929536013 conn(0x7fb9ec0fbc20 msgr2=0x7fb9ec0fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:50.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.942+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 shutdown_connections 2022-09-10T04:27:50.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:50.943+0000 7fb9f1521700 1 -- 172.21.15.81:0/3929536013 wait complete. 2022-09-10T04:27:51.333 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2022-09-10T04:27:51.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:27:51.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 -- 172.21.15.81:0/4026350177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140ff510 msgr2=0x7f9b141060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:51.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 --2- 172.21.15.81:0/4026350177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140ff510 0x7f9b141060b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c009b00 tx=0x7f9b0c009e10 comp rx=0 tx=0).stop 2022-09-10T04:27:51.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 -- 172.21.15.81:0/4026350177 shutdown_connections 2022-09-10T04:27:51.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 --2- 172.21.15.81:0/4026350177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140ff510 0x7f9b141060b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 --2- 172.21.15.81:0/4026350177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fdb40 0x7f9b140fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 --2- 172.21.15.81:0/4026350177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140fe490 0x7f9b140fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 -- 172.21.15.81:0/4026350177 >> 172.21.15.81:0/4026350177 conn(0x7f9b140f9230 msgr2=0x7f9b140fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:51.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 -- 172.21.15.81:0/4026350177 shutdown_connections 2022-09-10T04:27:51.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.568+0000 7f9b13fff700 1 -- 172.21.15.81:0/4026350177 wait complete. 2022-09-10T04:27:51.574 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.569+0000 7f9b13fff700 1 Processor -- start 2022-09-10T04:27:51.574 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.569+0000 7f9b13fff700 1 -- start start 2022-09-10T04:27:51.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b13fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140fdb40 0x7f9b141958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:51.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b13fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 0x7f9b14195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:51.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b13fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140ff510 0x7f9b1419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:51.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b13fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1419d470 con 0x7f9b140fe490 2022-09-10T04:27:51.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b13fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1419d5e0 con 0x7f9b140ff510 2022-09-10T04:27:51.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b13fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1419d7a0 con 0x7f9b140fdb40 2022-09-10T04:27:51.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b127fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 0x7f9b14195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:51.577 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b12ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140fdb40 0x7f9b141958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:51.577 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b127fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 0x7f9b14195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59024/0 (socket says 172.21.15.81:59024) 2022-09-10T04:27:51.577 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b127fc700 1 -- 172.21.15.81:0/394362948 learned_addr learned my addr 172.21.15.81:0/394362948 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:51.577 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b137fe700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140ff510 0x7f9b1419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:51.578 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b127fc700 1 -- 172.21.15.81:0/394362948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140fdb40 msgr2=0x7f9b141958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b127fc700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140fdb40 0x7f9b141958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.570+0000 7f9b127fc700 1 -- 172.21.15.81:0/394362948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140ff510 msgr2=0x7f9b1419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b127fc700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140ff510 0x7f9b1419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.579 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b127fc700 1 -- 172.21.15.81:0/394362948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b04009470 con 0x7f9b140fe490 2022-09-10T04:27:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b137fe700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140ff510 0x7f9b1419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b127fc700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 0x7f9b14195e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800e710 tx=0x7f9b0800ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:51.580 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08009500 con 0x7f9b140fe490 2022-09-10T04:27:51.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c009730 con 0x7f9b140fe490 2022-09-10T04:27:51.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9b08010da0 con 0x7f9b140fe490 2022-09-10T04:27:51.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b080173d0 con 0x7f9b140fe490 2022-09-10T04:27:51.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.571+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1419de00 con 0x7f9b140fe490 2022-09-10T04:27:51.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.573+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b14109330 con 0x7f9b140fe490 2022-09-10T04:27:51.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.574+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f9b0800c040 con 0x7f9b140fe490 2022-09-10T04:27:51.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.575+0000 7f9b189fd700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9afc099010 0x7f9afc09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:51.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.576+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f9b08014070 con 0x7f9b140fe490 2022-09-10T04:27:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.576+0000 7f9b12ffd700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9afc099010 0x7f9afc09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.577+0000 7f9b12ffd700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9afc099010 0x7f9afc09b4c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009440 tx=0x7f9b04005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:51.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.580+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f9b0804fc80 con 0x7f9b140fe490 2022-09-10T04:27:51.774 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.772+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b1419e440 con 0x7f9b140fe490 2022-09-10T04:27:51.775 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.774+0000 7f9b189fd700 1 -- 172.21.15.81:0/394362948 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+23649 (secure 0 0 0) 0x7f9b0808ec20 con 0x7f9b140fe490 2022-09-10T04:27:51.776 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:51.776 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":92,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:27:10.414885+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:27:51.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.776+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9afc099010 msgr2=0x7f9afc09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:51.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.776+0000 7f9b13fff700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9afc099010 0x7f9afc09b4c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009440 tx=0x7f9b04005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:51.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.776+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 msgr2=0x7f9b14195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:51.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.776+0000 7f9b13fff700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 0x7f9b14195e30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800e710 tx=0x7f9b0800ea20 comp rx=0 tx=0).stop 2022-09-10T04:27:51.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 shutdown_connections 2022-09-10T04:27:51.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9b140ff510 0x7f9b1419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9afc099010 0x7f9afc09b4c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b140fdb40 0x7f9b141958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 --2- 172.21.15.81:0/394362948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b140fe490 0x7f9b14195e30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:51.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 >> 172.21.15.81:0/394362948 conn(0x7f9b140f9230 msgr2=0x7f9b140fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:51.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 shutdown_connections 2022-09-10T04:27:51.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:51.777+0000 7f9b13fff700 1 -- 172.21.15.81:0/394362948 wait complete. 2022-09-10T04:27:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:51 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3929536013' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-09-10T04:27:51.802 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:51 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3929536013' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-09-10T04:27:51.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-10T04:26:30.005808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-10T04:26:31.529234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-10T04:26:35.269129+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-10T04:27:51.807 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:27:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:51 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3929536013' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2022-09-10T04:27:52.038 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 -- 172.21.15.81:0/579103972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca80fff30 msgr2=0x7f2ca8100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.039 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/579103972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca80fff30 0x7f2ca8100340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009810 tx=0x7f2ca4009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:52.039 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 -- 172.21.15.81:0/579103972 shutdown_connections 2022-09-10T04:27:52.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/579103972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca8108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/579103972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca80fff30 0x7f2ca8100340 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/579103972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ca81011b0 0x7f2ca8101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 -- 172.21.15.81:0/579103972 >> 172.21.15.81:0/579103972 conn(0x7f2ca80fb6b0 msgr2=0x7f2ca80fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:52.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 -- 172.21.15.81:0/579103972 shutdown_connections 2022-09-10T04:27:52.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.035+0000 7f2cb03ff700 1 -- 172.21.15.81:0/579103972 wait complete. 2022-09-10T04:27:52.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 Processor -- start 2022-09-10T04:27:52.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 -- start start 2022-09-10T04:27:52.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ca80fff30 0x7f2ca8195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.042 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca81011b0 0x7f2ca8196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca819d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca819d6a0 con 0x7f2ca8101b40 2022-09-10T04:27:52.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca819d810 con 0x7f2ca80fff30 2022-09-10T04:27:52.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cb03ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca819d9d0 con 0x7f2ca81011b0 2022-09-10T04:27:52.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cae99c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca819d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cae99c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca819d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59042/0 (socket says 172.21.15.81:59042) 2022-09-10T04:27:52.045 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cae99c700 1 -- 172.21.15.81:0/3943365946 learned_addr learned my addr 172.21.15.81:0/3943365946 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:52.045 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cad99a700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca81011b0 0x7f2ca8196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.045 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.037+0000 7f2cae19b700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ca80fff30 0x7f2ca8195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.048 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cae99c700 1 -- 172.21.15.81:0/3943365946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca81011b0 msgr2=0x7f2ca8196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.048 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cae99c700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca81011b0 0x7f2ca8196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.048 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cae99c700 1 -- 172.21.15.81:0/3943365946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ca80fff30 msgr2=0x7f2ca8195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cae99c700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ca80fff30 0x7f2ca8195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cae99c700 1 -- 172.21.15.81:0/3943365946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca4009470 con 0x7f2ca8101b40 2022-09-10T04:27:52.049 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cad99a700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca81011b0 0x7f2ca8196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2cae99c700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca819d0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000eaf0 tx=0x7f2ca000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0009960 con 0x7f2ca8101b40 2022-09-10T04:27:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2ca000ce70 con 0x7f2ca8101b40 2022-09-10T04:27:52.050 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.038+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00058b0 con 0x7f2ca8101b40 2022-09-10T04:27:52.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.039+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca819dcb0 con 0x7f2ca8101b40 2022-09-10T04:27:52.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.039+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca819e170 con 0x7f2ca8101b40 2022-09-10T04:27:52.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.040+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca810b6e0 con 0x7f2ca8101b40 2022-09-10T04:27:52.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.041+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f2ca001e020 con 0x7f2ca8101b40 2022-09-10T04:27:52.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.043+0000 7f2c9f7fe700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2c94099090 0x7f2c9409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.043+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f2ca0014070 con 0x7f2ca8101b40 2022-09-10T04:27:52.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.043+0000 7f2cae19b700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2c94099090 0x7f2c9409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.044+0000 7f2cae19b700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2c94099090 0x7f2c9409b540 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400b4f0 tx=0x7f2ca4009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:52.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.046+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f2ca008ef40 con 0x7f2ca8101b40 2022-09-10T04:27:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.244+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ca805faf0 con 0x7f2ca8101b40 2022-09-10T04:27:52.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.245+0000 7f2c9f7fe700 1 -- 172.21.15.81:0/3943365946 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ca0092560 con 0x7f2ca8101b40 2022-09-10T04:27:52.248 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:27:52.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2c94099090 msgr2=0x7f2c9409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2c94099090 0x7f2c9409b540 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400b4f0 tx=0x7f2ca4009eb0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 msgr2=0x7f2ca819d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca819d0d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000eaf0 tx=0x7f2ca000ee00 comp rx=0 tx=0).stop 2022-09-10T04:27:52.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 shutdown_connections 2022-09-10T04:27:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca8101b40 0x7f2ca819d0d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2ca80fff30 0x7f2ca8195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2c94099090 0x7f2c9409b540 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 --2- 172.21.15.81:0/3943365946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2ca81011b0 0x7f2ca8196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 >> 172.21.15.81:0/3943365946 conn(0x7f2ca80fb6b0 msgr2=0x7f2ca80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 shutdown_connections 2022-09-10T04:27:52.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.247+0000 7f2cb03ff700 1 -- 172.21.15.81:0/3943365946 wait complete. 2022-09-10T04:27:52.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-10T04:27:52.507 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 -- 172.21.15.81:0/1726341167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fdd00 msgr2=0x7f6c480fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.508 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/1726341167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fdd00 0x7f6c480fe110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6c38009810 tx=0x7f6c38009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:52.508 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 -- 172.21.15.81:0/1726341167 shutdown_connections 2022-09-10T04:27:52.508 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/1726341167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c480ff910 0x7f6c481061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.508 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/1726341167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fdd00 0x7f6c480fe110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.509 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/1726341167 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fef80 0x7f6c480ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.509 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 -- 172.21.15.81:0/1726341167 >> 172.21.15.81:0/1726341167 conn(0x7f6c480f93f0 msgr2=0x7f6c480fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:52.509 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 -- 172.21.15.81:0/1726341167 shutdown_connections 2022-09-10T04:27:52.509 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.506+0000 7f6c4dbff700 1 -- 172.21.15.81:0/1726341167 wait complete. 2022-09-10T04:27:52.512 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.507+0000 7f6c4dbff700 1 Processor -- start 2022-09-10T04:27:52.512 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.507+0000 7f6c4dbff700 1 -- start start 2022-09-10T04:27:52.512 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.507+0000 7f6c4dbff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fdd00 0x7f6c48195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c4dbff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 0x7f6c48196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c4dbff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c480ff910 0x7f6c4819d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c4dbff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4819d6c0 con 0x7f6c480fef80 2022-09-10T04:27:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c4dbff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4819d830 con 0x7f6c480ff910 2022-09-10T04:27:52.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c4dbff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4819d9f0 con 0x7f6c480fdd00 2022-09-10T04:27:52.514 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 0x7f6c48196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.514 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 0x7f6c48196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59050/0 (socket says 172.21.15.81:59050) 2022-09-10T04:27:52.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 -- 172.21.15.81:0/961033693 learned_addr learned my addr 172.21.15.81:0/961033693 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:52.517 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c477fe700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fdd00 0x7f6c48195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.517 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c47fff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c480ff910 0x7f6c4819d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.518 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 -- 172.21.15.81:0/961033693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fdd00 msgr2=0x7f6c48195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fdd00 0x7f6c48195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 -- 172.21.15.81:0/961033693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c480ff910 msgr2=0x7f6c4819d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c480ff910 0x7f6c4819d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c46ffd700 1 -- 172.21.15.81:0/961033693 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38009470 con 0x7f6c480fef80 2022-09-10T04:27:52.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.508+0000 7f6c477fe700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fdd00 0x7f6c48195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:52.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.509+0000 7f6c46ffd700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 0x7f6c48196080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00e740 tx=0x7f6c3c00ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:52.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.509+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c005220 con 0x7f6c480fef80 2022-09-10T04:27:52.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.509+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6c3c005380 con 0x7f6c480fef80 2022-09-10T04:27:52.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.509+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4819dcd0 con 0x7f6c480fef80 2022-09-10T04:27:52.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.509+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c004b70 con 0x7f6c480fef80 2022-09-10T04:27:52.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.509+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4819e1d0 con 0x7f6c480fef80 2022-09-10T04:27:52.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.510+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c481094b0 con 0x7f6c480fef80 2022-09-10T04:27:52.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.512+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f6c3c01e020 con 0x7f6c480fef80 2022-09-10T04:27:52.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.513+0000 7f6c44ff9700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c30099090 0x7f6c3009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.513+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f6c3c014070 con 0x7f6c480fef80 2022-09-10T04:27:52.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.514+0000 7f6c477fe700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c30099090 0x7f6c3009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.515+0000 7f6c477fe700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c30099090 0x7f6c3009b540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800b4f0 tx=0x7f6c38009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:52.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.517+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f6c3c08eb80 con 0x7f6c480fef80 2022-09-10T04:27:52.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:52 smithi081 ceph-mon[90133]: pgmap v343: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:52.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:52 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/394362948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:52 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3943365946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:27:52.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.709+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6c4819eb20 con 0x7f6c480fef80 2022-09-10T04:27:52.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.710+0000 7f6c44ff9700 1 -- 172.21.15.81:0/961033693 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6c3c0921a0 con 0x7f6c480fef80 2022-09-10T04:27:52.712 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:27:52.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c30099090 msgr2=0x7f6c3009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c30099090 0x7f6c3009b540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800b4f0 tx=0x7f6c38009eb0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 msgr2=0x7f6c48196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 0x7f6c48196080 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00e740 tx=0x7f6c3c00ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 shutdown_connections 2022-09-10T04:27:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c480ff910 0x7f6c4819d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6c30099090 0x7f6c3009b540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c480fdd00 0x7f6c48195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 --2- 172.21.15.81:0/961033693 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c480fef80 0x7f6c48196080 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 >> 172.21.15.81:0/961033693 conn(0x7f6c480f93f0 msgr2=0x7f6c480fb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:52.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 shutdown_connections 2022-09-10T04:27:52.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.712+0000 7f6c4dbff700 1 -- 172.21.15.81:0/961033693 wait complete. 2022-09-10T04:27:52.741 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-10T04:27:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:52 smithi104 ceph-mon[97044]: pgmap v343: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:52 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/394362948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:52 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3943365946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:27:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:52 smithi110 ceph-mon[97633]: pgmap v343: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:27:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:52 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/394362948' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:27:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:52 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3943365946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:27:52.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 -- 172.21.15.81:0/637386446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0072bf0 msgr2=0x7f27a0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 --2- 172.21.15.81:0/637386446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0072bf0 0x7f27a0071170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2790009810 tx=0x7f2790009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:52.977 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 -- 172.21.15.81:0/637386446 shutdown_connections 2022-09-10T04:27:52.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 --2- 172.21.15.81:0/637386446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0101640 0x7f27a010c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 --2- 172.21.15.81:0/637386446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0072bf0 0x7f27a0071170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 --2- 172.21.15.81:0/637386446 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 0x7f27a0071b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 -- 172.21.15.81:0/637386446 >> 172.21.15.81:0/637386446 conn(0x7f27a00f9230 msgr2=0x7f27a00fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:52.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 -- 172.21.15.81:0/637386446 shutdown_connections 2022-09-10T04:27:52.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.969+0000 7f27a559d700 1 -- 172.21.15.81:0/637386446 wait complete. 2022-09-10T04:27:52.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 Processor -- start 2022-09-10T04:27:52.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 -- start start 2022-09-10T04:27:52.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 0x7f27a0199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0072bf0 0x7f27a019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 0x7f27a01a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01a18e0 con 0x7f27a0101640 2022-09-10T04:27:52.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01a1a50 con 0x7f27a00716b0 2022-09-10T04:27:52.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.971+0000 7f27a559d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01a1c10 con 0x7f27a0072bf0 2022-09-10T04:27:52.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 0x7f27a01a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f279f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0072bf0 0x7f27a019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f279ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 0x7f27a0199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 0x7f27a01a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59064/0 (socket says 172.21.15.81:59064) 2022-09-10T04:27:52.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 -- 172.21.15.81:0/818978343 learned_addr learned my addr 172.21.15.81:0/818978343 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:52.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 -- 172.21.15.81:0/818978343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0072bf0 msgr2=0x7f27a019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0072bf0 0x7f27a019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 -- 172.21.15.81:0/818978343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 msgr2=0x7f27a0199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:52.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 0x7f27a0199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:52.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f27a4d9c700 1 -- 172.21.15.81:0/818978343 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27940093b0 con 0x7f27a0101640 2022-09-10T04:27:52.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.972+0000 7f279ffff700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 0x7f27a0199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:52.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.973+0000 7f27a4d9c700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 0x7f27a01a1310 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f279800eb40 tx=0x7f279800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:52.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.973+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27980099b0 con 0x7f27a0101640 2022-09-10T04:27:52.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.973+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f279800ce70 con 0x7f27a0101640 2022-09-10T04:27:52.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.973+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2798010430 con 0x7f27a0101640 2022-09-10T04:27:52.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.973+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2790009470 con 0x7f27a0101640 2022-09-10T04:27:52.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.973+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a01a21c0 con 0x7f27a0101640 2022-09-10T04:27:52.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.974+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a010f900 con 0x7f27a0101640 2022-09-10T04:27:52.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.976+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f279801e020 con 0x7f27a0101640 2022-09-10T04:27:52.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.977+0000 7f279d7fa700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2788099090 0x7f278809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:52.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.977+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f2798014070 con 0x7f27a0101640 2022-09-10T04:27:52.989 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.977+0000 7f279ffff700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2788099090 0x7f278809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:52.989 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.978+0000 7f279ffff700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2788099090 0x7f278809b540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f279000b4f0 tx=0x7f2790009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:52.989 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:52.981+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f279808efd0 con 0x7f27a0101640 2022-09-10T04:27:53.173 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.172+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f27a005faf0 con 0x7f27a0101640 2022-09-10T04:27:53.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.172+0000 7f279d7fa700 1 -- 172.21.15.81:0/818978343 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f27980925f0 con 0x7f27a0101640 2022-09-10T04:27:53.175 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2022-09-10T04:27:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2788099090 msgr2=0x7f278809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2788099090 0x7f278809b540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f279000b4f0 tx=0x7f2790009eb0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 msgr2=0x7f27a01a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 0x7f27a01a1310 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f279800eb40 tx=0x7f279800ee50 comp rx=0 tx=0).stop 2022-09-10T04:27:53.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 shutdown_connections 2022-09-10T04:27:53.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27a0101640 0x7f27a01a1310 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f27a00716b0 0x7f27a0199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2788099090 0x7f278809b540 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 --2- 172.21.15.81:0/818978343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27a0072bf0 0x7f27a019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 >> 172.21.15.81:0/818978343 conn(0x7f27a00f9230 msgr2=0x7f27a00fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:53.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 shutdown_connections 2022-09-10T04:27:53.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.175+0000 7f27a559d700 1 -- 172.21.15.81:0/818978343 wait complete. 2022-09-10T04:27:53.204 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-10T04:27:53.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 -- 172.21.15.81:0/3221916258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0fef80 msgr2=0x7f909c0ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 --2- 172.21.15.81:0/3221916258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0fef80 0x7f909c0ff3d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9090009750 tx=0x7f9090009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:53.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 -- 172.21.15.81:0/3221916258 shutdown_connections 2022-09-10T04:27:53.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 --2- 172.21.15.81:0/3221916258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f909c0ff910 0x7f909c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 --2- 172.21.15.81:0/3221916258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c0fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 --2- 172.21.15.81:0/3221916258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0fef80 0x7f909c0ff3d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 -- 172.21.15.81:0/3221916258 >> 172.21.15.81:0/3221916258 conn(0x7f909c0f93f0 msgr2=0x7f909c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:53.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 -- 172.21.15.81:0/3221916258 shutdown_connections 2022-09-10T04:27:53.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.437+0000 7f90a29ff700 1 -- 172.21.15.81:0/3221916258 wait complete. 2022-09-10T04:27:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 Processor -- start 2022-09-10T04:27:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 -- start start 2022-09-10T04:27:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f909c0fef80 0x7f909c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0ff910 0x7f909c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c071e10 con 0x7f909c0fdd00 2022-09-10T04:27:53.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c071f80 con 0x7f909c0fef80 2022-09-10T04:27:53.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f90a29ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c19da20 con 0x7f909c0ff910 2022-09-10T04:27:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.439+0000 7f909bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59080/0 (socket says 172.21.15.81:59080) 2022-09-10T04:27:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 -- 172.21.15.81:0/1359424347 learned_addr learned my addr 172.21.15.81:0/1359424347 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909b7fe700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f909c0fef80 0x7f909c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f90a0f9c700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0ff910 0x7f909c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 -- 172.21.15.81:0/1359424347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0ff910 msgr2=0x7f909c071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0ff910 0x7f909c071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 -- 172.21.15.81:0/1359424347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f909c0fef80 msgr2=0x7f909c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f909c0fef80 0x7f909c0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 -- 172.21.15.81:0/1359424347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90900093b0 con 0x7f909c0fdd00 2022-09-10T04:27:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f909bfff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c072a90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f908c00e710 tx=0x7f908c00ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.440+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c004220 con 0x7f909c0fdd00 2022-09-10T04:27:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.441+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f908c004380 con 0x7f909c0fdd00 2022-09-10T04:27:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.441+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c19dd00 con 0x7f909c0fdd00 2022-09-10T04:27:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.441+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f908c003c80 con 0x7f909c0fdd00 2022-09-10T04:27:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.441+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c19e1c0 con 0x7f909c0fdd00 2022-09-10T04:27:53.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.442+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c1094b0 con 0x7f909c0fdd00 2022-09-10T04:27:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.444+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f908c01e020 con 0x7f909c0fdd00 2022-09-10T04:27:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.445+0000 7f90997fa700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9084099010 0x7f908409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.445+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f908c014070 con 0x7f909c0fdd00 2022-09-10T04:27:53.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.445+0000 7f909b7fe700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9084099010 0x7f908409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.446+0000 7f909b7fe700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9084099010 0x7f908409b4c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f909000f500 tx=0x7f9090005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:53.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.449+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f908c08ebb0 con 0x7f909c0fdd00 2022-09-10T04:27:53.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:53 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/961033693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:27:53.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:53 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/818978343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:27:53.643 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.641+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f909c05fb30 con 0x7f909c0fdd00 2022-09-10T04:27:53.644 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.642+0000 7f90997fa700 1 -- 172.21.15.81:0/1359424347 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f908c0921d0 con 0x7f909c0fdd00 2022-09-10T04:27:53.644 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:27:53.647 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9084099010 msgr2=0x7f908409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.647 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9084099010 0x7f908409b4c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f909000f500 tx=0x7f9090005f90 comp rx=0 tx=0).stop 2022-09-10T04:27:53.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 msgr2=0x7f909c072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.648 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c072a90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f908c00e710 tx=0x7f908c00ea20 comp rx=0 tx=0).stop 2022-09-10T04:27:53.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 shutdown_connections 2022-09-10T04:27:53.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9084099010 0x7f908409b4c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f909c0ff910 0x7f909c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0fdd00 0x7f909c072a90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 --2- 172.21.15.81:0/1359424347 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f909c0fef80 0x7f909c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.644+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 >> 172.21.15.81:0/1359424347 conn(0x7f909c0f93f0 msgr2=0x7f909c0fb740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.645+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 shutdown_connections 2022-09-10T04:27:53.652 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.645+0000 7f90a29ff700 1 -- 172.21.15.81:0/1359424347 wait complete. 2022-09-10T04:27:53.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:27:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:53 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/961033693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:27:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:53 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/818978343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:27:53.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 -- 172.21.15.81:0/360705814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80066bf0 msgr2=0x7ffb80100890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 --2- 172.21.15.81:0/360705814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80066bf0 0x7ffb80100890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009810 tx=0x7ffb70009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:53.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 -- 172.21.15.81:0/360705814 shutdown_connections 2022-09-10T04:27:53.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 --2- 172.21.15.81:0/360705814 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 0x7ffb80105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 --2- 172.21.15.81:0/360705814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80066bf0 0x7ffb80100890 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 --2- 172.21.15.81:0/360705814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80100dd0 0x7ffb80103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 -- 172.21.15.81:0/360705814 >> 172.21.15.81:0/360705814 conn(0x7ffb800f82a0 msgr2=0x7ffb800fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:53.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.899+0000 7ffb857ff700 1 -- 172.21.15.81:0/360705814 shutdown_connections 2022-09-10T04:27:53.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.900+0000 7ffb857ff700 1 -- 172.21.15.81:0/360705814 wait complete. 2022-09-10T04:27:53.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 Processor -- start 2022-09-10T04:27:53.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 -- start start 2022-09-10T04:27:53.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80066bf0 0x7ffb80195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80100dd0 0x7ffb80196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 0x7ffb8019d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019d6c0 con 0x7ffb80100dd0 2022-09-10T04:27:53.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019d830 con 0x7ffb80066bf0 2022-09-10T04:27:53.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.901+0000 7ffb857ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8019d9f0 con 0x7ffb801038c0 2022-09-10T04:27:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 0x7ffb8019d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80100dd0 0x7ffb80196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 0x7ffb8019d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36852/0 (socket says 172.21.15.81:36852) 2022-09-10T04:27:53.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7f7fe700 1 -- 172.21.15.81:0/106271251 learned_addr learned my addr 172.21.15.81:0/106271251 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7effd700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80066bf0 0x7ffb80195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 -- 172.21.15.81:0/106271251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 msgr2=0x7ffb8019d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 0x7ffb8019d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 -- 172.21.15.81:0/106271251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80066bf0 msgr2=0x7ffb80195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80066bf0 0x7ffb80195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 -- 172.21.15.81:0/106271251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70009470 con 0x7ffb80100dd0 2022-09-10T04:27:53.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7effd700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80066bf0 0x7ffb80195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:53.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.902+0000 7ffb7e7fc700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80100dd0 0x7ffb80196080 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400e6c0 tx=0x7ffb7400e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:53.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.903+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74010900 con 0x7ffb80100dd0 2022-09-10T04:27:53.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.903+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ffb74018420 con 0x7ffb80100dd0 2022-09-10T04:27:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.903+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8019dcd0 con 0x7ffb80100dd0 2022-09-10T04:27:53.915 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.903+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74017610 con 0x7ffb80100dd0 2022-09-10T04:27:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.903+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8019e1d0 con 0x7ffb80100dd0 2022-09-10T04:27:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.904+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb80109490 con 0x7ffb80100dd0 2022-09-10T04:27:53.916 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.906+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ffb7400c040 con 0x7ffb80100dd0 2022-09-10T04:27:53.918 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:53 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/961033693' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:27:53.918 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:53 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/818978343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:27:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.907+0000 7ffb63fff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ffb68099090 0x7ffb6809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.907+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7ffb74014070 con 0x7ffb80100dd0 2022-09-10T04:27:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.907+0000 7ffb7effd700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ffb68099090 0x7ffb6809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.908+0000 7ffb7effd700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ffb68099090 0x7ffb6809b540 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000b4f0 tx=0x7ffb70009e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:53.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:53.911+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ffb7404fc80 con 0x7ffb80100dd0 2022-09-10T04:27:54.146 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.145+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb80105350 con 0x7ffb80100dd0 2022-09-10T04:27:54.147 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.145+0000 7ffb63fff700 1 -- 172.21.15.81:0/106271251 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10787 (secure 0 0 0) 0x7ffb7408ea90 con 0x7ffb80100dd0 2022-09-10T04:27:54.147 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:27:54.148 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-10T04:26:32.097174+0000","modified":"2022-09-10T04:26:59.184557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24577,"mds_1":24572,"mds_2":24547,"mds_3":24532,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24466":{"gid":24466,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"j","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24497":{"gid":24497,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24577":{"gid":24577,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-10T04:27:54.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ffb68099090 msgr2=0x7ffb6809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ffb68099090 0x7ffb6809b540 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000b4f0 tx=0x7ffb70009e70 comp rx=0 tx=0).stop 2022-09-10T04:27:54.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80100dd0 msgr2=0x7ffb80196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.150 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80100dd0 0x7ffb80196080 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400e6c0 tx=0x7ffb7400e9d0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.151 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 shutdown_connections 2022-09-10T04:27:54.151 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ffb68099090 0x7ffb6809b540 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.151 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffb801038c0 0x7ffb8019d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.151 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ffb80066bf0 0x7ffb80195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.152 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 --2- 172.21.15.81:0/106271251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb80100dd0 0x7ffb80196080 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.152 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 >> 172.21.15.81:0/106271251 conn(0x7ffb800f82a0 msgr2=0x7ffb800ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.152 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 shutdown_connections 2022-09-10T04:27:54.152 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.148+0000 7ffb857ff700 1 -- 172.21.15.81:0/106271251 wait complete. 2022-09-10T04:27:54.153 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 28 2022-09-10T04:27:54.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-09-10T04:26:32.097174+0000', 'modified': '2022-09-10T04:26:59.184557+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24577, 'mds_1': 24572, 'mds_2': 24547, 'mds_3': 24532, 'mds_4': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24461': {'gid': 24461, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6833/777476125', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 777476125}, {'type': 'v1', 'addr': '172.21.15.104:6833', 'nonce': 777476125}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24466': {'gid': 24466, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.110:6833/2514050760', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6832', 'nonce': 2514050760}, {'type': 'v1', 'addr': '172.21.15.110:6833', 'nonce': 2514050760}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24473': {'gid': 24473, 'name': 'j', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.81:6835/1412455487', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1412455487}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1412455487}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24497': {'gid': 24497, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6835/2709283075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 2709283075}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 2709283075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/4033961168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 4033961168}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 4033961168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6837/2202378802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6836', 'nonce': 2202378802}, {'type': 'v1', 'addr': '172.21.15.104:6837', 'nonce': 2202378802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.110:6837/1467999026', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 1467999026}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 1467999026}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.110:6839/1670334720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 1670334720}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 1670334720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.104:6839/2357131382', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 2357131382}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 2357131382}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24577': {'gid': 24577, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.81:6841/953424075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6840', 'nonce': 953424075}, {'type': 'v1', 'addr': '172.21.15.81:6841', 'nonce': 953424075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-09-10T04:27:54.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2022-09-10T04:27:54.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-09-10T04:27:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 -- 172.21.15.81:0/3790731890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0fef80 msgr2=0x7f293c0ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 --2- 172.21.15.81:0/3790731890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0fef80 0x7f293c0ff3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2930009810 tx=0x7f2930009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:54.408 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 -- 172.21.15.81:0/3790731890 shutdown_connections 2022-09-10T04:27:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 --2- 172.21.15.81:0/3790731890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f293c0ff910 0x7f293c1061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 --2- 172.21.15.81:0/3790731890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c0fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 --2- 172.21.15.81:0/3790731890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0fef80 0x7f293c0ff3d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.409 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 -- 172.21.15.81:0/3790731890 >> 172.21.15.81:0/3790731890 conn(0x7f293c0f93f0 msgr2=0x7f293c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 -- 172.21.15.81:0/3790731890 shutdown_connections 2022-09-10T04:27:54.410 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.406+0000 7f29427ff700 1 -- 172.21.15.81:0/3790731890 wait complete. 2022-09-10T04:27:54.413 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.407+0000 7f29427ff700 1 Processor -- start 2022-09-10T04:27:54.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 -- start start 2022-09-10T04:27:54.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f293c0fef80 0x7f293c196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0ff910 0x7f293c19d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.414 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c19d750 con 0x7f293c0ff910 2022-09-10T04:27:54.415 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c19d8c0 con 0x7f293c0fdd00 2022-09-10T04:27:54.415 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f29427ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c19da80 con 0x7f293c0fef80 2022-09-10T04:27:54.415 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f293bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.416 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f293b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f293c0fef80 0x7f293c196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.416 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f293bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52004/0 (socket says 172.21.15.81:52004) 2022-09-10T04:27:54.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f293bfff700 1 -- 172.21.15.81:0/3885227419 learned_addr learned my addr 172.21.15.81:0/3885227419 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.408+0000 7f2940d9c700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0ff910 0x7f293c19d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f293bfff700 1 -- 172.21.15.81:0/3885227419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f293c0fef80 msgr2=0x7f293c196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.417 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f293bfff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f293c0fef80 0x7f293c196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f293bfff700 1 -- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0ff910 msgr2=0x7f293c19d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f293bfff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0ff910 0x7f293c19d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f293bfff700 1 -- 172.21.15.81:0/3885227419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c0093b0 con 0x7f293c0fdd00 2022-09-10T04:27:54.418 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f293bfff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c195bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f292c00e9a0 tx=0x7f292c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f29397fa700 1 -- 172.21.15.81:0/3885227419 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c00ca10 con 0x7f293c0fdd00 2022-09-10T04:27:54.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2930009470 con 0x7f293c0fdd00 2022-09-10T04:27:54.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f29397fa700 1 -- 172.21.15.81:0/3885227419 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f292c00cb70 con 0x7f293c0fdd00 2022-09-10T04:27:54.419 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f29397fa700 1 -- 172.21.15.81:0/3885227419 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c018680 con 0x7f293c0fdd00 2022-09-10T04:27:54.420 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.409+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c19e0e0 con 0x7f293c0fdd00 2022-09-10T04:27:54.430 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.413+0000 7f29397fa700 1 -- 172.21.15.81:0/3885227419 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f292c00c040 con 0x7f293c0fdd00 2022-09-10T04:27:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 Processor -- start 2022-09-10T04:27:54.431 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 -- start start 2022-09-10T04:27:54.432 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd8ae0 0x555c29dd8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 0x555c29dd9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.433 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dda1f0 0x555c29dda640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.434 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f29397fa700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2924099010 0x7f292409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29cc3080 con 0x555c29dda1f0 2022-09-10T04:27:54.435 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29d2d9e0 con 0x555c29dd8ae0 2022-09-10T04:27:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f2955d52b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29d27180 con 0x555c29dd9430 2022-09-10T04:27:54.436 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.414+0000 7f29397fa700 1 -- 172.21.15.81:0/3885227419 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f292c014070 con 0x7f293c0fdd00 2022-09-10T04:27:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 0x555c29dd9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f2940d9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd8ae0 0x555c29dd8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.438 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 0x555c29dd9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36884/0 (socket says 172.21.15.81:36884) 2022-09-10T04:27:54.438 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 -- 172.21.15.81:0/3987628447 learned_addr learned my addr 172.21.15.81:0/3987628447 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f2940d9c700 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dda1f0 0x555c29dda640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293b7fe700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2924099010 0x7f292409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 -- 172.21.15.81:0/3987628447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd8ae0 msgr2=0x555c29dd8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd8ae0 0x555c29dd8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 -- 172.21.15.81:0/3987628447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dda1f0 msgr2=0x555c29dda640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dda1f0 0x555c29dda640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 -- 172.21.15.81:0/3987628447 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c29baf5e0 con 0x555c29dd9430 2022-09-10T04:27:54.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293bfff700 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 0x555c29dd9c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f292c000c00 tx=0x7f292c092b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9166591d677152 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f29227fc700 1 -- 172.21.15.81:0/3987628447 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c0980c0 con 0x555c29dd9430 2022-09-10T04:27:54.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f293b7fe700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2924099010 0x7f292409b4c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f293000b4f0 tx=0x7f2930009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.415+0000 7f29227fc700 1 -- 172.21.15.81:0/3987628447 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f292c0c0540 con 0x555c29dd9430 2022-09-10T04:27:54.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f29227fc700 1 -- 172.21.15.81:0/3987628447 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c07d170 con 0x555c29dd9430 2022-09-10T04:27:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 -- 172.21.15.81:0/3987628447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 msgr2=0x555c29dd9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 0x555c29dd9c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f292c000c00 tx=0x7f292c092b50 comp rx=0 tx=0).stop 2022-09-10T04:27:54.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 -- 172.21.15.81:0/3987628447 shutdown_connections 2022-09-10T04:27:54.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dda1f0 0x555c29dda640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd8ae0 0x555c29dd8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3987628447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dd9430 0x555c29dd9c60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 -- 172.21.15.81:0/3987628447 >> 172.21.15.81:0/3987628447 conn(0x555c29dd4fa0 msgr2=0x555c29dd3370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 -- 172.21.15.81:0/3987628447 shutdown_connections 2022-09-10T04:27:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.416+0000 7f2955d52b80 1 -- 172.21.15.81:0/3987628447 wait complete. 2022-09-10T04:27:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.417+0000 7f2955d52b80 1 -- start start 2022-09-10T04:27:54.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.417+0000 7f2955d52b80 1 Processor -- start 2022-09-10T04:27:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29c150c0 con 0x555c29dd8ae0 2022-09-10T04:27:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29de9fe0 con 0x555c29dd9430 2022-09-10T04:27:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29dea170 con 0x555c29dda1f0 2022-09-10T04:27:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2955d52b80 10 client.0 _ll_register_callbacks cb 0x555c29cd2370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f293b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36892/0 (socket says 172.21.15.81:36892) 2022-09-10T04:27:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 -- 172.21.15.81:0/2577576790 learned_addr learned my addr 172.21.15.81:0/2577576790 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f293bfff700 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.450 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 -- 172.21.15.81:0/2577576790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 msgr2=0x555c29de2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 -- 172.21.15.81:0/2577576790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 msgr2=0x555c29de2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.451 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.418+0000 7f2940d9c700 1 -- 172.21.15.81:0/2577576790 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c19e0e0 con 0x555c29dda1f0 2022-09-10T04:27:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f2940d9c700 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de9aa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2934010b40 tx=0x7f2934005ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e964e2c21736d55 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f29177fe700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:27:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f29177fe700 1 -- 172.21.15.81:0/2577576790 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934004a30 con 0x555c29dda1f0 2022-09-10T04:27:54.452 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c29dcbaa0 con 0x555c29dda1f0 2022-09-10T04:27:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f29177fe700 1 -- 172.21.15.81:0/2577576790 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f2934004b90 con 0x555c29dda1f0 2022-09-10T04:27:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f29177fe700 1 -- 172.21.15.81:0/2577576790 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2934004d50 con 0x555c29dda1f0 2022-09-10T04:27:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f2955d52b80 10 client.25069 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:54.453 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555c29b5bd90 con 0x555c29dda1f0 2022-09-10T04:27:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.419+0000 7f29177fe700 1 -- 172.21.15.81:0/2577576790 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x555c29dcbaa0 con 0x555c29dda1f0 2022-09-10T04:27:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f29177fe700 1 -- 172.21.15.81:0/2577576790 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f2934005470 con 0x555c29dda1f0 2022-09-10T04:27:54.454 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f2955d52b80 10 client.25069 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f2955d52b80 10 client.25069 resolve_mds: resolved name 'l' to daemon mds.l 2022-09-10T04:27:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f2955d52b80 20 client.25069 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29ded8d0 0x555c29df2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f2955d52b80 4 client.25069 mds_command: new command op to 24466 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:27:54.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 --> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c29da84c0 con 0x555c29ded8d0 2022-09-10T04:27:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.420+0000 7f293bfff700 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29ded8d0 0x555c29df2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.421+0000 7f293bfff700 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29ded8d0 0x555c29df2bb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=39cff453a0245ae2 server_cookie=de9f579a63e6d701 in_seq=0 out_seq=0 2022-09-10T04:27:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.421+0000 7f29177fe700 10 client.25069 ms_handle_connect on v2:172.21.15.110:6832/2514050760 2022-09-10T04:27:54.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f29177fe700 1 -- 172.21.15.81:0/2577576790 <== mds.4 v2:172.21.15.110:6832/2514050760 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x555c29da84c0 con 0x555c29ded8d0 2022-09-10T04:27:54.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f29177fe700 10 client.25069 handle_command_reply: tid=0 2022-09-10T04:27:54.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 client.25069 shutdown 2022-09-10T04:27:54.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 msgr2=0x555c29de9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de9aa0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2934010b40 tx=0x7f2934005ae0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 shutdown_connections 2022-09-10T04:27:54.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29ded8d0 0x555c29df2bb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de9aa0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f29177fe700 0 client.25069 ms_handle_reset on v2:172.21.15.110:6832/2514050760 2022-09-10T04:27:54.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2577576790 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 >> 172.21.15.81:0/2577576790 conn(0x555c29dd4fa0 msgr2=0x555c29dd2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 shutdown_connections 2022-09-10T04:27:54.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 1 -- 172.21.15.81:0/2577576790 wait complete. 2022-09-10T04:27:54.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.422+0000 7f2955d52b80 20 client.25069 trim_cache size 0 max 16384 2022-09-10T04:27:54.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:54 smithi081 ceph-mon[90133]: pgmap v344: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:54.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:54 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1359424347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:27:54.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:54 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/106271251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:27:54.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 Processor -- start 2022-09-10T04:27:54.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 -- start start 2022-09-10T04:27:54.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29dd4e40 con 0x555c29dd8ae0 2022-09-10T04:27:54.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29da84c0 con 0x555c29dd9430 2022-09-10T04:27:54.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2955d52b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29dcbaa0 con 0x555c29dda1f0 2022-09-10T04:27:54.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f293bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f293bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59140/0 (socket says 172.21.15.81:59140) 2022-09-10T04:27:54.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f2940d9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f293bfff700 1 -- 172.21.15.81:0/3438448263 learned_addr learned my addr 172.21.15.81:0/3438448263 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.442+0000 7f293b7fe700 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2940d9c700 1 -- 172.21.15.81:0/3438448263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 msgr2=0x555c29de07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2940d9c700 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2940d9c700 1 -- 172.21.15.81:0/3438448263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 msgr2=0x555c29de02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2940d9c700 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2940d9c700 1 -- 172.21.15.81:0/3438448263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c29b5a260 con 0x555c29dda1f0 2022-09-10T04:27:54.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2940d9c700 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de22d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2934005850 tx=0x7f293400f6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da3ba0d1188b2167 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2922ffd700 1 -- 172.21.15.81:0/3438448263 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934009eb0 con 0x555c29dda1f0 2022-09-10T04:27:54.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2922ffd700 1 -- 172.21.15.81:0/3438448263 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2934017040 con 0x555c29dda1f0 2022-09-10T04:27:54.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.443+0000 7f2922ffd700 1 -- 172.21.15.81:0/3438448263 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293401dcf0 con 0x555c29dda1f0 2022-09-10T04:27:54.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 -- 172.21.15.81:0/3438448263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 msgr2=0x555c29de22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de22d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2934005850 tx=0x7f293400f6c0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 -- 172.21.15.81:0/3438448263 shutdown_connections 2022-09-10T04:27:54.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29de22d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 --2- 172.21.15.81:0/3438448263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.473 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 -- 172.21.15.81:0/3438448263 >> 172.21.15.81:0/3438448263 conn(0x555c29dd4fa0 msgr2=0x555c29ddce90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.473 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 -- 172.21.15.81:0/3438448263 shutdown_connections 2022-09-10T04:27:54.473 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.444+0000 7f2955d52b80 1 -- 172.21.15.81:0/3438448263 wait complete. 2022-09-10T04:27:54.473 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.445+0000 7f2955d52b80 1 -- start start 2022-09-10T04:27:54.474 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.445+0000 7f2955d52b80 1 Processor -- start 2022-09-10T04:27:54.474 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.474 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29e3fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29e403c0 con 0x555c29dd8ae0 2022-09-10T04:27:54.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29e40530 con 0x555c29dd9430 2022-09-10T04:27:54.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c29e406f0 con 0x555c29dda1f0 2022-09-10T04:27:54.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2955d52b80 10 client.0 _ll_register_callbacks cb 0x555c29cd2370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f293bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f293b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f293bfff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59154/0 (socket says 172.21.15.81:59154) 2022-09-10T04:27:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f293bfff700 1 -- 172.21.15.81:0/2370514230 learned_addr learned my addr 172.21.15.81:0/2370514230 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.446+0000 7f2940d9c700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29e3fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.477 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293b7fe700 1 -- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 msgr2=0x555c29e3fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.477 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293b7fe700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29e3fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.477 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293b7fe700 1 -- 172.21.15.81:0/2370514230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 msgr2=0x555c29de6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.478 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293b7fe700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.478 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293b7fe700 1 -- 172.21.15.81:0/2370514230 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c29b5bd90 con 0x555c29dd9430 2022-09-10T04:27:54.478 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293bfff700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:54.478 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f293b7fe700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de7180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2930000c00 tx=0x7f2930009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8664e3b7f9b35096 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f29177fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:27:54.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f29177fe700 1 -- 172.21.15.81:0/2370514230 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930004dc0 con 0x555c29dd9430 2022-09-10T04:27:54.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f292c0c0540 con 0x555c29dd9430 2022-09-10T04:27:54.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.447+0000 7f29177fe700 1 -- 172.21.15.81:0/2370514230 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f293000f460 con 0x555c29dd9430 2022-09-10T04:27:54.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f29177fe700 1 -- 172.21.15.81:0/2370514230 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f293000f620 con 0x555c29dd9430 2022-09-10T04:27:54.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f2955d52b80 10 client.25067 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:54.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555c29c28a30 con 0x555c29dd9430 2022-09-10T04:27:54.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f29177fe700 1 -- 172.21.15.81:0/2370514230 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f292c0c0540 con 0x555c29dd9430 2022-09-10T04:27:54.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f29177fe700 1 -- 172.21.15.81:0/2370514230 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f2930032800 con 0x555c29dd9430 2022-09-10T04:27:54.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f2955d52b80 10 client.25067 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:54.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f2955d52b80 10 client.25067 resolve_mds: resolved name 'l' to daemon mds.l 2022-09-10T04:27:54.482 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f2955d52b80 20 client.25067 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:54.482 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.448+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29e45560 0x555c29e41eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.482 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.449+0000 7f2955d52b80 4 client.25067 mds_command: new command op to 24466 tid=0[{"prefix": "status"}] 2022-09-10T04:27:54.482 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.449+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 --> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555c29d206f0 con 0x555c29e45560 2022-09-10T04:27:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.449+0000 7f293bfff700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29e45560 0x555c29e41eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.449+0000 7f293bfff700 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29e45560 0x555c29e41eb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a7055aa504f15b35 server_cookie=1f2be65a79914749 in_seq=0 out_seq=0 2022-09-10T04:27:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.449+0000 7f29177fe700 10 client.25067 ms_handle_connect on v2:172.21.15.110:6832/2514050760 2022-09-10T04:27:54.483 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f29177fe700 1 -- 172.21.15.81:0/2370514230 <== mds.4 v2:172.21.15.110:6832/2514050760 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555c29d206f0 con 0x555c29e45560 2022-09-10T04:27:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f29177fe700 10 client.25067 handle_command_reply: tid=0 2022-09-10T04:27:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 client.25067 shutdown 2022-09-10T04:27:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 msgr2=0x555c29de7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.484 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de7180 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2930000c00 tx=0x7f2930009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 shutdown_connections 2022-09-10T04:27:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x555c29e45560 0x555c29e41eb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x555c29dda1f0 0x555c29e3fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555c29dd8ae0 0x555c29de6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 --2- 172.21.15.81:0/2370514230 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555c29dd9430 0x555c29de7180 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f29177fe700 0 client.25067 ms_handle_reset on v2:172.21.15.110:6832/2514050760 2022-09-10T04:27:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 >> 172.21.15.81:0/2370514230 conn(0x555c29dd4fa0 msgr2=0x555c29de02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.486 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 shutdown_connections 2022-09-10T04:27:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 1 -- 172.21.15.81:0/2370514230 wait complete. 2022-09-10T04:27:54.487 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.450+0000 7f2955d52b80 20 client.25067 trim_cache size 0 max 16384 2022-09-10T04:27:54.487 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:27:54.488 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:27:54.488 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 4, 2022-09-10T04:27:54.488 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24466, 2022-09-10T04:27:54.488 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:27:54.489 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:27:54.489 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:27:54.489 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 56.270805946999999, 2022-09-10T04:27:54.489 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:27:54.489 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 92, 2022-09-10T04:27:54.490 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:27:54.490 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 119.97331680400001 2022-09-10T04:27:54.490 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:27:54.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2924099010 msgr2=0x7f292409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2924099010 0x7f292409b4c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f293000b4f0 tx=0x7f2930009eb0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 msgr2=0x7f293c195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c195bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f292c00e9a0 tx=0x7f292c00ecb0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 shutdown_connections 2022-09-10T04:27:54.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f293c0ff910 0x7f293c19d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f293c0fdd00 0x7f293c195bd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2924099010 0x7f292409b4c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 --2- 172.21.15.81:0/3885227419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f293c0fef80 0x7f293c196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 >> 172.21.15.81:0/3885227419 conn(0x7f293c0f93f0 msgr2=0x7f293c104960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 shutdown_connections 2022-09-10T04:27:54.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.451+0000 7f29427ff700 1 -- 172.21.15.81:0/3885227419 wait complete. 2022-09-10T04:27:54.495 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2022-09-10T04:27:54.726 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.720+0000 7f39f559d700 1 -- 172.21.15.81:0/134068058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00fe490 msgr2=0x7f39f00fee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.720+0000 7f39f559d700 1 --2- 172.21.15.81:0/134068058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00fe490 0x7f39f00fee80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009750 tx=0x7f39e4009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:54.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 -- 172.21.15.81:0/134068058 shutdown_connections 2022-09-10T04:27:54.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 --2- 172.21.15.81:0/134068058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39f00ff510 0x7f39f01060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 --2- 172.21.15.81:0/134068058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f00fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 --2- 172.21.15.81:0/134068058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00fe490 0x7f39f00fee80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 -- 172.21.15.81:0/134068058 >> 172.21.15.81:0/134068058 conn(0x7f39f00f9230 msgr2=0x7f39f00fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 -- 172.21.15.81:0/134068058 shutdown_connections 2022-09-10T04:27:54.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.721+0000 7f39f559d700 1 -- 172.21.15.81:0/134068058 wait complete. 2022-09-10T04:27:54.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.722+0000 7f39f559d700 1 Processor -- start 2022-09-10T04:27:54.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.722+0000 7f39f559d700 1 -- start start 2022-09-10T04:27:54.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.722+0000 7f39f559d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f0072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f559d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39f00fe490 0x7f39f00710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f559d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00ff510 0x7f39f0071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.736 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f559d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f0071e10 con 0x7f39f00fdb40 2022-09-10T04:27:54.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f559d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f0071f80 con 0x7f39f00fe490 2022-09-10T04:27:54.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f559d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f019d840 con 0x7f39f00ff510 2022-09-10T04:27:54.737 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f0072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f4d9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00ff510 0x7f39f0071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f0072a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59168/0 (socket says 172.21.15.81:59168) 2022-09-10T04:27:54.738 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 -- 172.21.15.81:0/758292020 learned_addr learned my addr 172.21.15.81:0/758292020 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39f00fe490 0x7f39f00710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 -- 172.21.15.81:0/758292020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00ff510 msgr2=0x7f39f0071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.739 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00ff510 0x7f39f0071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 -- 172.21.15.81:0/758292020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39f00fe490 msgr2=0x7f39f00710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39f00fe490 0x7f39f00710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.740 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39effff700 1 -- 172.21.15.81:0/758292020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e40093b0 con 0x7f39f00fdb40 2022-09-10T04:27:54.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.723+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00ff510 0x7f39f0071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:54.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.724+0000 7f39effff700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f0072a90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f39e000e740 tx=0x7f39e000ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.724+0000 7f39ed7fa700 1 -- 172.21.15.81:0/758292020 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0005220 con 0x7f39f00fdb40 2022-09-10T04:27:54.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.724+0000 7f39ed7fa700 1 -- 172.21.15.81:0/758292020 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39e0005380 con 0x7f39f00fdb40 2022-09-10T04:27:54.741 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.724+0000 7f39ed7fa700 1 -- 172.21.15.81:0/758292020 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0004c10 con 0x7f39f00fdb40 2022-09-10T04:27:54.742 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.724+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f019db20 con 0x7f39f00fdb40 2022-09-10T04:27:54.743 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.724+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f019e030 con 0x7f39f00fdb40 2022-09-10T04:27:54.751 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.729+0000 7f39ed7fa700 1 -- 172.21.15.81:0/758292020 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f39e001e020 con 0x7f39f00fdb40 2022-09-10T04:27:54.752 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 Processor -- start 2022-09-10T04:27:54.752 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 -- start start 2022-09-10T04:27:54.753 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936feb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.753 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fe9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.753 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fea630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.753 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936ed4080 con 0x556936feb950 2022-09-10T04:27:54.754 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936f3e9e0 con 0x556936feb000 2022-09-10T04:27:54.754 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936f38180 con 0x556936fea1c0 2022-09-10T04:27:54.754 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.730+0000 7f39ed7fa700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f39d8099090 0x7f39d809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.754 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39effff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fe9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.755 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.755 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fea630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36938/0 (socket says 172.21.15.81:36938) 2022-09-10T04:27:54.755 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 -- 172.21.15.81:0/2870602080 learned_addr learned my addr 172.21.15.81:0/2870602080 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.755 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ed7fa700 1 -- 172.21.15.81:0/758292020 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f39e0014070 con 0x7f39f00fdb40 2022-09-10T04:27:54.756 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936feb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.756 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f39d8099090 0x7f39d809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.756 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 -- 172.21.15.81:0/2870602080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 msgr2=0x556936feb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.757 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936feb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.757 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 -- 172.21.15.81:0/2870602080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 msgr2=0x556936fe9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.758 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fe9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.758 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 -- 172.21.15.81:0/2870602080 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556936dc05e0 con 0x556936fea1c0 2022-09-10T04:27:54.758 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.731+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fea630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39f00fe860 tx=0x7f39e40059c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95da8b6ea6883eb2 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.758 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.732+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f39d8099090 0x7f39d809b540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39e80099b0 tx=0x7f39e8009410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.759 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.732+0000 7f39d67fc700 1 -- 172.21.15.81:0/2870602080 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e401d070 con 0x556936fea1c0 2022-09-10T04:27:54.759 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.732+0000 7f39d67fc700 1 -- 172.21.15.81:0/2870602080 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39e401e040 con 0x556936fea1c0 2022-09-10T04:27:54.759 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.732+0000 7f39d67fc700 1 -- 172.21.15.81:0/2870602080 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4004760 con 0x556936fea1c0 2022-09-10T04:27:54.760 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.732+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2870602080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 msgr2=0x556936fea630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.760 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.732+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fea630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39f00fe860 tx=0x7f39e40059c0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.760 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2870602080 shutdown_connections 2022-09-10T04:27:54.760 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fea630 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.760 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936feb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.761 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2870602080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fe9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.761 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2870602080 >> 172.21.15.81:0/2870602080 conn(0x556936fe6170 msgr2=0x556936fe45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2870602080 shutdown_connections 2022-09-10T04:27:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2870602080 wait complete. 2022-09-10T04:27:54.762 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.733+0000 7f3a09dc9b80 1 -- start start 2022-09-10T04:27:54.763 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.734+0000 7f3a09dc9b80 1 Processor -- start 2022-09-10T04:27:54.764 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.734+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 0x556936ff3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.764 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936feb000 0x556936ff3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.764 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb950 0x556936ffaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936e260c0 con 0x556936fea1c0 2022-09-10T04:27:54.765 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936ffafd0 con 0x556936feb950 2022-09-10T04:27:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936ffb160 con 0x556936feb000 2022-09-10T04:27:54.766 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f3a09dc9b80 10 client.0 _ll_register_callbacks cb 0x556936ee3370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39effff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 0x556936ff3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.768 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39f4d9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb950 0x556936ffaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.769 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39effff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 0x556936ff3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59186/0 (socket says 172.21.15.81:59186) 2022-09-10T04:27:54.769 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39effff700 1 -- 172.21.15.81:0/2139064077 learned_addr learned my addr 172.21.15.81:0/2139064077 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.770 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936feb000 0x556936ff3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.770 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39f4d9c700 1 -- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936feb000 msgr2=0x556936ff3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936feb000 0x556936ff3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39f4d9c700 1 -- 172.21.15.81:0/2139064077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 msgr2=0x556936ff3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.771 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 0x556936ff3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.772 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39f4d9c700 1 -- 172.21.15.81:0/2139064077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f0106870 con 0x556936feb950 2022-09-10T04:27:54.772 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.735+0000 7f39effff700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 0x556936ff3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:54.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb950 0x556936ffaa90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f39e800fdd0 tx=0x7f39e8004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35581d308268fb89 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.773 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f39cb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:27:54.774 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f39cb7fe700 1 -- 172.21.15.81:0/2139064077 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e8023070 con 0x556936feb950 2022-09-10T04:27:54.774 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556936fdccf0 con 0x556936feb950 2022-09-10T04:27:54.774 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f39cb7fe700 1 -- 172.21.15.81:0/2139064077 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f39e800fe00 con 0x556936feb950 2022-09-10T04:27:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f39cb7fe700 1 -- 172.21.15.81:0/2139064077 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39e8011050 con 0x556936feb950 2022-09-10T04:27:54.775 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f3a09dc9b80 10 client.25079 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:54.776 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.736+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556936d6cd90 con 0x556936feb950 2022-09-10T04:27:54.776 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f39cb7fe700 1 -- 172.21.15.81:0/2139064077 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x556936fdccf0 con 0x556936feb950 2022-09-10T04:27:54.777 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f39cb7fe700 1 -- 172.21.15.81:0/2139064077 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f39e8010830 con 0x556936feb950 2022-09-10T04:27:54.777 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f3a09dc9b80 10 client.25079 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:54.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f3a09dc9b80 10 client.25079 resolve_mds: resolved name 'c' to daemon mds.c 2022-09-10T04:27:54.778 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f3a09dc9b80 20 client.25079 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556936ffe8a0 0x556937003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f3a09dc9b80 4 client.25079 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:27:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 --> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556936fb94c0 con 0x556936ffe8a0 2022-09-10T04:27:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.737+0000 7f39effff700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556936ffe8a0 0x556937003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.738+0000 7f39effff700 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556936ffe8a0 0x556937003bc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5aaba30d8a0e6e01 server_cookie=b6239bdfdf1176c5 in_seq=0 out_seq=0 2022-09-10T04:27:54.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.738+0000 7f39cb7fe700 10 client.25079 ms_handle_connect on v2:172.21.15.110:6836/1467999026 2022-09-10T04:27:54.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f39cb7fe700 1 -- 172.21.15.81:0/2139064077 <== mds.3 v2:172.21.15.110:6836/1467999026 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x556936fb94c0 con 0x556936ffe8a0 2022-09-10T04:27:54.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f39cb7fe700 10 client.25079 handle_command_reply: tid=0 2022-09-10T04:27:54.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 client.25079 shutdown 2022-09-10T04:27:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb950 msgr2=0x556936ffaa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb950 0x556936ffaa90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f39e800fdd0 tx=0x7f39e8004950 comp rx=0 tx=0).stop 2022-09-10T04:27:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 shutdown_connections 2022-09-10T04:27:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb950 0x556936ffaa90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556936ffe8a0 0x556937003bc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936fea1c0 0x556936ff3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/2139064077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936feb000 0x556936ff3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 >> 172.21.15.81:0/2139064077 conn(0x556936fe6170 msgr2=0x556936fe3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f39cb7fe700 0 client.25079 ms_handle_reset on v2:172.21.15.110:6836/1467999026 2022-09-10T04:27:54.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 shutdown_connections 2022-09-10T04:27:54.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.739+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/2139064077 wait complete. 2022-09-10T04:27:54.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.740+0000 7f3a09dc9b80 20 client.25079 trim_cache size 0 max 16384 2022-09-10T04:27:54.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.757+0000 7f3a09dc9b80 1 Processor -- start 2022-09-10T04:27:54.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.757+0000 7f3a09dc9b80 1 -- start start 2022-09-10T04:27:54.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.757+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fe9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ffb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fee040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936efc050 con 0x556936feb950 2022-09-10T04:27:54.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936eda4f0 con 0x556936feb000 2022-09-10T04:27:54.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936dc4a30 con 0x556936fea1c0 2022-09-10T04:27:54.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39f4d9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ffb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39effff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fe9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ffb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52098/0 (socket says 172.21.15.81:52098) 2022-09-10T04:27:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 -- 172.21.15.81:0/1114397671 learned_addr learned my addr 172.21.15.81:0/1114397671 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 -- 172.21.15.81:0/1114397671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 msgr2=0x556936fe9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fe9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 -- 172.21.15.81:0/1114397671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 msgr2=0x556936fee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.758+0000 7f39ef7fe700 1 -- 172.21.15.81:0/1114397671 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556936d6b260 con 0x556936feb000 2022-09-10T04:27:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.759+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ffb540 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39e4023000 tx=0x7f39e4009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26cdf6bf9ccdfcf5 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.759+0000 7f39d6ffd700 1 -- 172.21.15.81:0/1114397671 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e401d070 con 0x556936feb000 2022-09-10T04:27:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.759+0000 7f39d6ffd700 1 -- 172.21.15.81:0/1114397671 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39e400f6d0 con 0x556936feb000 2022-09-10T04:27:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.759+0000 7f39d6ffd700 1 -- 172.21.15.81:0/1114397671 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e4004c10 con 0x556936feb000 2022-09-10T04:27:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.759+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1114397671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 msgr2=0x556936ffb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ffb540 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39e4023000 tx=0x7f39e4009fc0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1114397671 shutdown_connections 2022-09-10T04:27:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936fee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936fe9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1114397671 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ffb540 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1114397671 >> 172.21.15.81:0/1114397671 conn(0x556936fe6170 msgr2=0x556937007be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1114397671 shutdown_connections 2022-09-10T04:27:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1114397671 wait complete. 2022-09-10T04:27:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.760+0000 7f3a09dc9b80 1 -- start start 2022-09-10T04:27:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 Processor -- start 2022-09-10T04:27:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936ff1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ff7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936ff8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936ff87d0 con 0x556936feb950 2022-09-10T04:27:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x556936ff8940 con 0x556936feb000 2022-09-10T04:27:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x556937050be0 con 0x556936fea1c0 2022-09-10T04:27:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.761+0000 7f3a09dc9b80 10 client.0 _ll_register_callbacks cb 0x556936ee3370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936ff1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936ff1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:36974/0 (socket says 172.21.15.81:36974) 2022-09-10T04:27:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 -- 172.21.15.81:0/1585141420 learned_addr learned my addr 172.21.15.81:0/1585141420 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39f4d9c700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936ff8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ff7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 -- 172.21.15.81:0/1585141420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 msgr2=0x556936ff7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ff7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 -- 172.21.15.81:0/1585141420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 msgr2=0x556936ff8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936ff8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 -- 172.21.15.81:0/1585141420 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556936dc05e0 con 0x556936fea1c0 2022-09-10T04:27:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39effff700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936ff1e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f39e008eeb0 tx=0x7f39e001cd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78299a6c060daab8 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39cb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:27:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39cb7fe700 1 -- 172.21.15.81:0/1585141420 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e00930b0 con 0x556936fea1c0 2022-09-10T04:27:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f39cb7fe700 1 -- 172.21.15.81:0/1585141420 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f39e00929b0 con 0x556936fea1c0 2022-09-10T04:27:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.762+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39e401e040 con 0x556936fea1c0 2022-09-10T04:27:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f39cb7fe700 1 -- 172.21.15.81:0/1585141420 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39e0099c50 con 0x556936fea1c0 2022-09-10T04:27:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 10 client.25093 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556936e39a30 con 0x556936fea1c0 2022-09-10T04:27:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f39cb7fe700 1 -- 172.21.15.81:0/1585141420 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f39e401e040 con 0x556936fea1c0 2022-09-10T04:27:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f39cb7fe700 1 -- 172.21.15.81:0/1585141420 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f39e00af2a0 con 0x556936fea1c0 2022-09-10T04:27:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 10 client.25093 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 10 client.25093 resolve_mds: resolved name 'c' to daemon mds.c 2022-09-10T04:27:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 20 client.25093 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556937055990 0x556937052420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 4 client.25093 mds_command: new command op to 24532 tid=0[{"prefix": "status"}] 2022-09-10T04:27:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.763+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 --> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556936f316f0 con 0x556937055990 2022-09-10T04:27:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556937055990 0x556937052420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f39ef7fe700 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556937055990 0x556937052420 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=940ad554564e4b53 server_cookie=ca886606d7a67343 in_seq=0 out_seq=0 2022-09-10T04:27:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f39cb7fe700 10 client.25093 ms_handle_connect on v2:172.21.15.110:6836/1467999026 2022-09-10T04:27:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f39cb7fe700 1 -- 172.21.15.81:0/1585141420 <== mds.3 v2:172.21.15.110:6836/1467999026 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x556936f316f0 con 0x556937055990 2022-09-10T04:27:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f39cb7fe700 10 client.25093 handle_command_reply: tid=0 2022-09-10T04:27:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f3a09dc9b80 1 client.25093 shutdown 2022-09-10T04:27:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 msgr2=0x556936ff1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.764+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936ff1e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f39e008eeb0 tx=0x7f39e001cd40 comp rx=0 tx=0).stop 2022-09-10T04:27:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 shutdown_connections 2022-09-10T04:27:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x556937055990 0x556937052420 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556936feb950 0x556936ff8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f39cb7fe700 0 client.25093 ms_handle_reset on v2:172.21.15.110:6836/1467999026 2022-09-10T04:27:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x556936fea1c0 0x556936ff1e10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 --2- 172.21.15.81:0/1585141420 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x556936feb000 0x556936ff7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 >> 172.21.15.81:0/1585141420 conn(0x556936fe6170 msgr2=0x556936fee040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 shutdown_connections 2022-09-10T04:27:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 1 -- 172.21.15.81:0/1585141420 wait complete. 2022-09-10T04:27:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f3a09dc9b80 20 client.25093 trim_cache size 0 max 16384 2022-09-10T04:27:54.816 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:27:54.817 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:27:54.817 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 3, 2022-09-10T04:27:54.817 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24532, 2022-09-10T04:27:54.818 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:27:54.818 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:27:54.818 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:27:54.819 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 57.305359897999999, 2022-09-10T04:27:54.819 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:27:54.819 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 92, 2022-09-10T04:27:54.819 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:27:54.819 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 104.34032806499999 2022-09-10T04:27:54.820 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:27:54.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:54 smithi104 ceph-mon[97044]: pgmap v344: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:54.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:54 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1359424347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:27:54.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:54 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/106271251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:27:54.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.765+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f39d8099090 msgr2=0x7f39d809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f39d8099090 0x7f39d809b540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f39e80099b0 tx=0x7f39e8009410 comp rx=0 tx=0).stop 2022-09-10T04:27:54.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 msgr2=0x7f39f0072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:54.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f0072a90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f39e000e740 tx=0x7f39e000ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:54.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 shutdown_connections 2022-09-10T04:27:54.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f39d8099090 0x7f39d809b540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39f00ff510 0x7f39f0071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39f00fdb40 0x7f39f0072a90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 --2- 172.21.15.81:0/758292020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f39f00fe490 0x7f39f00710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:54.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 >> 172.21.15.81:0/758292020 conn(0x7f39f00f9230 msgr2=0x7f39f00fb530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:54.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 shutdown_connections 2022-09-10T04:27:54.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:54.766+0000 7f39f559d700 1 -- 172.21.15.81:0/758292020 wait complete. 2022-09-10T04:27:54.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2022-09-10T04:27:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:54 smithi110 ceph-mon[97633]: pgmap v344: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:54 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1359424347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:27:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:54 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/106271251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:27:55.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 -- 172.21.15.81:0/800572702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fef80 msgr2=0x7fb9800ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.166 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 --2- 172.21.15.81:0/800572702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fef80 0x7fb9800ff3d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009810 tx=0x7fb96c009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:55.167 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 -- 172.21.15.81:0/800572702 shutdown_connections 2022-09-10T04:27:55.167 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 --2- 172.21.15.81:0/800572702 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9800ff910 0x7fb9801061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.168 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 --2- 172.21.15.81:0/800572702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800fdd00 0x7fb9800fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.168 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 --2- 172.21.15.81:0/800572702 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fef80 0x7fb9800ff3d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 -- 172.21.15.81:0/800572702 >> 172.21.15.81:0/800572702 conn(0x7fb9800f93f0 msgr2=0x7fb9800fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 -- 172.21.15.81:0/800572702 shutdown_connections 2022-09-10T04:27:55.170 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.053+0000 7fb986fff700 1 -- 172.21.15.81:0/800572702 wait complete. 2022-09-10T04:27:55.170 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.054+0000 7fb986fff700 1 Processor -- start 2022-09-10T04:27:55.171 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.054+0000 7fb986fff700 1 -- start start 2022-09-10T04:27:55.171 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.054+0000 7fb986fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fdd00 0x7fb980195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.172 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb986fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9800fef80 0x7fb9801960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.172 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb986fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 0x7fb98019d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.173 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb986fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98019d6f0 con 0x7fb9800ff910 2022-09-10T04:27:55.173 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb986fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98019d860 con 0x7fb9800fef80 2022-09-10T04:27:55.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb986fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98019da20 con 0x7fb9800fdd00 2022-09-10T04:27:55.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 0x7fb98019d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 0x7fb98019d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59220/0 (socket says 172.21.15.81:59220) 2022-09-10T04:27:55.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 -- 172.21.15.81:0/1363245002 learned_addr learned my addr 172.21.15.81:0/1363245002 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.176 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fdd00 0x7fb980195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.176 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb97ffff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9800fef80 0x7fb9801960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 -- 172.21.15.81:0/1363245002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fdd00 msgr2=0x7fb980195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fdd00 0x7fb980195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 -- 172.21.15.81:0/1363245002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9800fef80 msgr2=0x7fb9801960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9800fef80 0x7fb9801960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb98559c700 1 -- 172.21.15.81:0/1363245002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9740093b0 con 0x7fb9800ff910 2022-09-10T04:27:55.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.055+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fdd00 0x7fb980195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.056+0000 7fb98559c700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 0x7fb98019d120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb97800eaf0 tx=0x7fb97800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.056+0000 7fb97dffb700 1 -- 172.21.15.81:0/1363245002 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978009960 con 0x7fb9800ff910 2022-09-10T04:27:55.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.056+0000 7fb97dffb700 1 -- 172.21.15.81:0/1363245002 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb97800ce70 con 0x7fb9800ff910 2022-09-10T04:27:55.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.056+0000 7fb97dffb700 1 -- 172.21.15.81:0/1363245002 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9780058b0 con 0x7fb9800ff910 2022-09-10T04:27:55.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.056+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c009470 con 0x7fb9800ff910 2022-09-10T04:27:55.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.056+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98019e030 con 0x7fb9800ff910 2022-09-10T04:27:55.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.059+0000 7fb97dffb700 1 -- 172.21.15.81:0/1363245002 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb97801e020 con 0x7fb9800ff910 2022-09-10T04:27:55.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.060+0000 7fb97dffb700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb970099090 0x7fb97009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 Processor -- start 2022-09-10T04:27:55.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb97dffb700 1 -- 172.21.15.81:0/1363245002 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fb978014070 con 0x7fb9800ff910 2022-09-10T04:27:55.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 -- start start 2022-09-10T04:27:55.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb970099090 0x7fb97009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.211 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61dae0 0x56162f61def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 0x56162f61ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.212 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f61f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f508080 con 0x56162f61dae0 2022-09-10T04:27:55.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f5729e0 con 0x56162f61e430 2022-09-10T04:27:55.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.061+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f56c180 con 0x56162f61f1f0 2022-09-10T04:27:55.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 0x56162f61ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 0x56162f61ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52138/0 (socket says 172.21.15.81:52138) 2022-09-10T04:27:55.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 -- 172.21.15.81:0/568618750 learned_addr learned my addr 172.21.15.81:0/568618750 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb98559c700 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f61f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb984d9b700 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61dae0 0x56162f61def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 -- 172.21.15.81:0/568618750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 msgr2=0x56162f61f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f61f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 -- 172.21.15.81:0/568618750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61dae0 msgr2=0x56162f61def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61dae0 0x56162f61def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 -- 172.21.15.81:0/568618750 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56162f3f45e0 con 0x56162f61e430 2022-09-10T04:27:55.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb984d9b700 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61dae0 0x56162f61def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:27:55.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb970099090 0x7fb97009b540 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb9800fede0 tx=0x7fb9740118e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb97ffff700 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 0x56162f61ec60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb96c015000 tx=0x7fb96c005d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b007d4c4e4986b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.062+0000 7fb96affd700 1 -- 172.21.15.81:0/568618750 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c01d070 con 0x56162f61e430 2022-09-10T04:27:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb96affd700 1 -- 172.21.15.81:0/568618750 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb96c01e040 con 0x56162f61e430 2022-09-10T04:27:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb96affd700 1 -- 172.21.15.81:0/568618750 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c004760 con 0x56162f61e430 2022-09-10T04:27:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 -- 172.21.15.81:0/568618750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 msgr2=0x56162f61ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 0x56162f61ec60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb96c015000 tx=0x7fb96c005d00 comp rx=0 tx=0).stop 2022-09-10T04:27:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 -- 172.21.15.81:0/568618750 shutdown_connections 2022-09-10T04:27:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f61f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61dae0 0x56162f61def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/568618750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61e430 0x56162f61ec60 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 -- 172.21.15.81:0/568618750 >> 172.21.15.81:0/568618750 conn(0x56162f619fa0 msgr2=0x56162f618370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 -- 172.21.15.81:0/568618750 shutdown_connections 2022-09-10T04:27:55.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.063+0000 7fb99a5cab80 1 -- 172.21.15.81:0/568618750 wait complete. 2022-09-10T04:27:55.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.064+0000 7fb99a5cab80 1 -- start start 2022-09-10T04:27:55.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.064+0000 7fb99a5cab80 1 Processor -- start 2022-09-10T04:27:55.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6274a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 0x56162f6279e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f62ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f45a0c0 con 0x56162f61e430 2022-09-10T04:27:55.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f62ef60 con 0x56162f61dae0 2022-09-10T04:27:55.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f62f0f0 con 0x56162f61f1f0 2022-09-10T04:27:55.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb99a5cab80 10 client.0 _ll_register_callbacks cb 0x56162f517370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:55.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 0x56162f6279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb97ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 0x56162f6279e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59244/0 (socket says 172.21.15.81:59244) 2022-09-10T04:27:55.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb97ffff700 1 -- 172.21.15.81:0/95094300 learned_addr learned my addr 172.21.15.81:0/95094300 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb984d9b700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6274a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb98559c700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f62ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.065+0000 7fb97ffff700 1 -- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 msgr2=0x56162f62ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb97ffff700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f62ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb97ffff700 1 -- 172.21.15.81:0/95094300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 msgr2=0x56162f6274a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb97ffff700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6274a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb97ffff700 1 -- 172.21.15.81:0/95094300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb980106a40 con 0x56162f61e430 2022-09-10T04:27:55.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb984d9b700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb97ffff700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 0x56162f6279e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00b4f0 tx=0x7fb96c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55827d66fde55315 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb95bfff700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:27:55.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb95bfff700 1 -- 172.21.15.81:0/95094300 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c01d070 con 0x56162f61e430 2022-09-10T04:27:55.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56162f610aa0 con 0x56162f61e430 2022-09-10T04:27:55.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb95bfff700 1 -- 172.21.15.81:0/95094300 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fb96c00baf0 con 0x56162f61e430 2022-09-10T04:27:55.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb95bfff700 1 -- 172.21.15.81:0/95094300 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb96c009260 con 0x56162f61e430 2022-09-10T04:27:55.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb99a5cab80 10 client.14994 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:55.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.066+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56162f3a0d90 con 0x56162f61e430 2022-09-10T04:27:55.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb95bfff700 1 -- 172.21.15.81:0/95094300 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x56162f610aa0 con 0x56162f61e430 2022-09-10T04:27:55.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb95bfff700 1 -- 172.21.15.81:0/95094300 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fb96c0173f0 con 0x56162f61e430 2022-09-10T04:27:55.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb99a5cab80 10 client.14994 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:55.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb99a5cab80 10 client.14994 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-10T04:27:55.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb99a5cab80 20 client.14994 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:55.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f6321d0 0x56162f630ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb99a5cab80 4 client.14994 mds_command: new command op to 24547 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:27:55.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.067+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 --> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56162f5ed4c0 con 0x56162f6321d0 2022-09-10T04:27:55.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.068+0000 7fb984d9b700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f6321d0 0x56162f630ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.068+0000 7fb984d9b700 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f6321d0 0x56162f630ae0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=882176c9f9780b92 server_cookie=907da8ee91c74032 in_seq=0 out_seq=0 2022-09-10T04:27:55.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.068+0000 7fb95bfff700 10 client.14994 ms_handle_connect on v2:172.21.15.110:6838/1670334720 2022-09-10T04:27:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.069+0000 7fb95bfff700 1 -- 172.21.15.81:0/95094300 <== mds.2 v2:172.21.15.110:6838/1670334720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x56162f5ed4c0 con 0x56162f6321d0 2022-09-10T04:27:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.069+0000 7fb95bfff700 10 client.14994 handle_command_reply: tid=0 2022-09-10T04:27:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.069+0000 7fb99a5cab80 1 client.14994 shutdown 2022-09-10T04:27:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.069+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 msgr2=0x56162f6279e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.069+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 0x56162f6279e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00b4f0 tx=0x7fb96c00f690 comp rx=0 tx=0).stop 2022-09-10T04:27:55.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 shutdown_connections 2022-09-10T04:27:55.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f6321d0 0x56162f630ae0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f62ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6274a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/95094300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f61e430 0x56162f6279e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb95bfff700 0 client.14994 ms_handle_reset on v2:172.21.15.110:6838/1670334720 2022-09-10T04:27:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 >> 172.21.15.81:0/95094300 conn(0x56162f619fa0 msgr2=0x56162f617a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 shutdown_connections 2022-09-10T04:27:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 1 -- 172.21.15.81:0/95094300 wait complete. 2022-09-10T04:27:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.070+0000 7fb99a5cab80 20 client.14994 trim_cache size 0 max 16384 2022-09-10T04:27:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.088+0000 7fb99a5cab80 1 Processor -- start 2022-09-10T04:27:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.088+0000 7fb99a5cab80 1 -- start start 2022-09-10T04:27:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.088+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61dae0 0x56162f621e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.088+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61f1f0 0x56162f627250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f627790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f5ed4c0 con 0x56162f63c670 2022-09-10T04:27:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f610aa0 con 0x56162f61f1f0 2022-09-10T04:27:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f47eed0 con 0x56162f61dae0 2022-09-10T04:27:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f627790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f627790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59256/0 (socket says 172.21.15.81:59256) 2022-09-10T04:27:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 -- 172.21.15.81:0/1314123924 learned_addr learned my addr 172.21.15.81:0/1314123924 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61dae0 0x56162f621e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb97ffff700 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61f1f0 0x56162f627250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 -- 172.21.15.81:0/1314123924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61dae0 msgr2=0x56162f621e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61dae0 0x56162f621e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 -- 172.21.15.81:0/1314123924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61f1f0 msgr2=0x56162f627250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61f1f0 0x56162f627250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb98559c700 1 -- 172.21.15.81:0/1314123924 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56162f39f260 con 0x56162f63c670 2022-09-10T04:27:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.089+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61dae0 0x56162f621e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.090+0000 7fb98559c700 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f627790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb978092b90 tx=0x7fb978095040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c01e71991e3fdd server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.090+0000 7fb96b7fe700 1 -- 172.21.15.81:0/1314123924 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978098070 con 0x56162f63c670 2022-09-10T04:27:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.090+0000 7fb96b7fe700 1 -- 172.21.15.81:0/1314123924 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb978093300 con 0x56162f63c670 2022-09-10T04:27:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.090+0000 7fb96b7fe700 1 -- 172.21.15.81:0/1314123924 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978004c30 con 0x56162f63c670 2022-09-10T04:27:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.090+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1314123924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 msgr2=0x56162f627790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.090+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f627790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb978092b90 tx=0x7fb978095040 comp rx=0 tx=0).stop 2022-09-10T04:27:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1314123924 shutdown_connections 2022-09-10T04:27:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f627790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61dae0 0x56162f621e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1314123924 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61f1f0 0x56162f627250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1314123924 >> 172.21.15.81:0/1314123924 conn(0x56162f619fa0 msgr2=0x56162f617a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1314123924 shutdown_connections 2022-09-10T04:27:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1314123924 wait complete. 2022-09-10T04:27:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 -- start start 2022-09-10T04:27:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.091+0000 7fb99a5cab80 1 Processor -- start 2022-09-10T04:27:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f638de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f63b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f63bce0 con 0x56162f63c670 2022-09-10T04:27:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f63be50 con 0x56162f61dae0 2022-09-10T04:27:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x56162f63c010 con 0x56162f61f1f0 2022-09-10T04:27:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb99a5cab80 10 client.0 _ll_register_callbacks cb 0x56162f517370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb98559c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f63b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb984d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb98559c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f63b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59270/0 (socket says 172.21.15.81:59270) 2022-09-10T04:27:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb98559c700 1 -- 172.21.15.81:0/1162185829 learned_addr learned my addr 172.21.15.81:0/1162185829 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.092+0000 7fb97ffff700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f638de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb98559c700 1 -- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 msgr2=0x56162f638de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb98559c700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f638de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb98559c700 1 -- 172.21.15.81:0/1162185829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 msgr2=0x56162f6388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb98559c700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb98559c700 1 -- 172.21.15.81:0/1162185829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97400fc20 con 0x56162f63c670 2022-09-10T04:27:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb98559c700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f63b7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb978092b40 tx=0x7fb978004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de77411754edea9 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb95bfff700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:27:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb95bfff700 1 -- 172.21.15.81:0/1162185829 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978092c00 con 0x56162f63c670 2022-09-10T04:27:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb96c01e040 con 0x56162f63c670 2022-09-10T04:27:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb95bfff700 1 -- 172.21.15.81:0/1162185829 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fb9780a9e40 con 0x56162f63c670 2022-09-10T04:27:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb95bfff700 1 -- 172.21.15.81:0/1162185829 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb978099ea0 con 0x56162f63c670 2022-09-10T04:27:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb99a5cab80 10 client.15000 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.093+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fb9780a6040 con 0x56162f63c670 2022-09-10T04:27:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb95bfff700 1 -- 172.21.15.81:0/1162185829 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb96c01e040 con 0x56162f63c670 2022-09-10T04:27:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb95bfff700 1 -- 172.21.15.81:0/1162185829 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fb9780920f0 con 0x56162f63c670 2022-09-10T04:27:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb99a5cab80 10 client.15000 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb99a5cab80 10 client.15000 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-10T04:27:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb99a5cab80 20 client.15000 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f689520 0x56162f685ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb99a5cab80 4 client.15000 mds_command: new command op to 24547 tid=0[{"prefix": "status"}] 2022-09-10T04:27:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 --> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56162f5656f0 con 0x56162f689520 2022-09-10T04:27:55.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.094+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f689520 0x56162f685ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb984d9b700 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f689520 0x56162f685ed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4a5dba7375b4d79 server_cookie=da326296db0734f4 in_seq=0 out_seq=0 2022-09-10T04:27:55.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb95bfff700 10 client.15000 ms_handle_connect on v2:172.21.15.110:6838/1670334720 2022-09-10T04:27:55.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb95bfff700 1 -- 172.21.15.81:0/1162185829 <== mds.2 v2:172.21.15.110:6838/1670334720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56162f5656f0 con 0x56162f689520 2022-09-10T04:27:55.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb95bfff700 10 client.15000 handle_command_reply: tid=0 2022-09-10T04:27:55.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 client.15000 shutdown 2022-09-10T04:27:55.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 msgr2=0x56162f63b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f63b7a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb978092b40 tx=0x7fb978004950 comp rx=0 tx=0).stop 2022-09-10T04:27:55.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 shutdown_connections 2022-09-10T04:27:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x56162f689520 0x56162f685ed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56162f63c670 0x56162f63b7a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56162f61dae0 0x56162f6388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 --2- 172.21.15.81:0/1162185829 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x56162f61f1f0 0x56162f638de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb95bfff700 0 client.15000 ms_handle_reset on v2:172.21.15.110:6838/1670334720 2022-09-10T04:27:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.095+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 >> 172.21.15.81:0/1162185829 conn(0x56162f619fa0 msgr2=0x56162f617a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 shutdown_connections 2022-09-10T04:27:55.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb99a5cab80 1 -- 172.21.15.81:0/1162185829 wait complete. 2022-09-10T04:27:55.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb99a5cab80 20 client.15000 trim_cache size 0 max 16384 2022-09-10T04:27:55.252 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:27:55.253 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:27:55.253 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 2, 2022-09-10T04:27:55.253 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24547, 2022-09-10T04:27:55.253 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:27:55.253 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:27:55.254 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:27:55.254 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 58.645156172999997, 2022-09-10T04:27:55.254 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:27:55.254 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 92, 2022-09-10T04:27:55.255 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:27:55.255 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 100.29661622899999 2022-09-10T04:27:55.255 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:27:55.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb970099090 msgr2=0x7fb97009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb970099090 0x7fb97009b540 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb9800fede0 tx=0x7fb9740118e0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 msgr2=0x7fb98019d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 0x7fb98019d120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb97800eaf0 tx=0x7fb97800ee00 comp rx=0 tx=0).stop 2022-09-10T04:27:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 shutdown_connections 2022-09-10T04:27:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9800ff910 0x7fb98019d120 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb970099090 0x7fb97009b540 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9800fdd00 0x7fb980195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 --2- 172.21.15.81:0/1363245002 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9800fef80 0x7fb9801960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.096+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 >> 172.21.15.81:0/1363245002 conn(0x7fb9800f93f0 msgr2=0x7fb9800fb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.097+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 shutdown_connections 2022-09-10T04:27:55.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.097+0000 7fb986fff700 1 -- 172.21.15.81:0/1363245002 wait complete. 2022-09-10T04:27:55.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2022-09-10T04:27:55.490 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 -- 172.21.15.81:0/1801207237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fd730 msgr2=0x7f8bb80fdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.491 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/1801207237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fd730 0x7f8bb80fdba0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009bb0 tx=0x7f8bb0009ec0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.491 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 -- 172.21.15.81:0/1801207237 shutdown_connections 2022-09-10T04:27:55.491 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/1801207237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fd730 0x7f8bb80fdba0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.492 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/1801207237 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fc430 0x7f8bb80fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/1801207237 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bb80fcd80 0x7f8bb80fd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.488+0000 7f8bbe99d700 1 -- 172.21.15.81:0/1801207237 >> 172.21.15.81:0/1801207237 conn(0x7f8bb80f8160 msgr2=0x7f8bb80fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.489+0000 7f8bbe99d700 1 -- 172.21.15.81:0/1801207237 shutdown_connections 2022-09-10T04:27:55.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.489+0000 7f8bbe99d700 1 -- 172.21.15.81:0/1801207237 wait complete. 2022-09-10T04:27:55.512 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 Processor -- start 2022-09-10T04:27:55.512 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 -- start start 2022-09-10T04:27:55.512 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 0x7f8bb8195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bb80fcd80 0x7f8bb8195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fd730 0x7f8bb819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb819d490 con 0x7f8bb80fc430 2022-09-10T04:27:55.513 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb819d600 con 0x7f8bb80fcd80 2022-09-10T04:27:55.514 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.490+0000 7f8bbe99d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb819d7c0 con 0x7f8bb80fd730 2022-09-10T04:27:55.514 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 0x7f8bb8195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.514 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 0x7f8bb8195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59300/0 (socket says 172.21.15.81:59300) 2022-09-10T04:27:55.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 -- 172.21.15.81:0/685757551 learned_addr learned my addr 172.21.15.81:0/685757551 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fd730 0x7f8bb819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bb80fcd80 0x7f8bb8195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.515 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 -- 172.21.15.81:0/685757551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fd730 msgr2=0x7f8bb819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fd730 0x7f8bb819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 -- 172.21.15.81:0/685757551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bb80fcd80 msgr2=0x7f8bb8195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bb80fcd80 0x7f8bb8195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbd99b700 1 -- 172.21.15.81:0/685757551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb00097e0 con 0x7f8bb80fc430 2022-09-10T04:27:55.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.491+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fd730 0x7f8bb819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.517 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.492+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 0x7f8bb8195910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400e740 tx=0x7f8bb400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.517 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.492+0000 7f8baeffd700 1 -- 172.21.15.81:0/685757551 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4005220 con 0x7f8bb80fc430 2022-09-10T04:27:55.517 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.492+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb819daa0 con 0x7f8bb80fc430 2022-09-10T04:27:55.518 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.492+0000 7f8baeffd700 1 -- 172.21.15.81:0/685757551 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8bb4005380 con 0x7f8bb80fc430 2022-09-10T04:27:55.518 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.492+0000 7f8baeffd700 1 -- 172.21.15.81:0/685757551 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4004b70 con 0x7f8bb80fc430 2022-09-10T04:27:55.518 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.492+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb819e000 con 0x7f8bb80fc430 2022-09-10T04:27:55.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.495+0000 7f8baeffd700 1 -- 172.21.15.81:0/685757551 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8bb401e020 con 0x7f8bb80fc430 2022-09-10T04:27:55.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.496+0000 7f8baeffd700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ba4099090 0x7f8ba409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8baeffd700 1 -- 172.21.15.81:0/685757551 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f8bb4014070 con 0x7f8bb80fc430 2022-09-10T04:27:55.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8bd30b4b80 1 Processor -- start 2022-09-10T04:27:55.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ba4099090 0x7f8ba409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8bd30b4b80 1 -- start start 2022-09-10T04:27:55.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ba4099090 0x7f8ba409b540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8005fd0 tx=0x7f8ba8005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417af410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.497+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561141698080 con 0x5611417ae1c0 2022-09-10T04:27:55.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611417029e0 con 0x5611417af000 2022-09-10T04:27:55.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611416fc180 con 0x5611417af950 2022-09-10T04:27:55.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbe19c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd19a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbe19c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417ae630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59314/0 (socket says 172.21.15.81:59314) 2022-09-10T04:27:55.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbe19c700 1 -- 172.21.15.81:0/885479488 learned_addr learned my addr 172.21.15.81:0/885479488 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd19a700 1 -- 172.21.15.81:0/885479488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 msgr2=0x5611417af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd19a700 1 -- 172.21.15.81:0/885479488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 msgr2=0x5611417ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd19a700 1 -- 172.21.15.81:0/885479488 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611415845e0 con 0x5611417af950 2022-09-10T04:27:55.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.498+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.499+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417adbe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8011b20 tx=0x7f8ba8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3251dfe6713ea4c server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.499+0000 7f8ba3fff700 1 -- 172.21.15.81:0/885479488 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba800ea40 con 0x5611417af950 2022-09-10T04:27:55.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.499+0000 7f8ba3fff700 1 -- 172.21.15.81:0/885479488 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8ba8021070 con 0x5611417af950 2022-09-10T04:27:55.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.499+0000 7f8ba3fff700 1 -- 172.21.15.81:0/885479488 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba801d410 con 0x5611417af950 2022-09-10T04:27:55.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/885479488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 msgr2=0x5611417adbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.526 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417adbe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8011b20 tx=0x7f8ba8014000 comp rx=0 tx=0).stop 2022-09-10T04:27:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/885479488 shutdown_connections 2022-09-10T04:27:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/885479488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417adbe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/885479488 >> 172.21.15.81:0/885479488 conn(0x5611417aa170 msgr2=0x5611417a85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/885479488 shutdown_connections 2022-09-10T04:27:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.500+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/885479488 wait complete. 2022-09-10T04:27:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 -- start start 2022-09-10T04:27:55.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 Processor -- start 2022-09-10T04:27:55.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417ae1c0 0x5611417b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x5611417bec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611415ea0c0 con 0x5611417af950 2022-09-10T04:27:55.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611417bf1b0 con 0x5611417af000 2022-09-10T04:27:55.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.501+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611417bf340 con 0x5611417ae1c0 2022-09-10T04:27:55.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bd30b4b80 10 client.0 _ll_register_callbacks cb 0x5611416a7370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:55.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbe19c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x5611417bec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417ae1c0 0x5611417b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbe19c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x5611417bec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59324/0 (socket says 172.21.15.81:59324) 2022-09-10T04:27:55.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbe19c700 1 -- 172.21.15.81:0/2681903314 learned_addr learned my addr 172.21.15.81:0/2681903314 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 msgr2=0x5611417b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2681903314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 msgr2=0x5611417bec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x5611417bec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2681903314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb8108a80 con 0x5611417ae1c0 2022-09-10T04:27:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x5611417bec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417ae1c0 0x5611417b76f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4092780 tx=0x7f8bb40921d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb6f6a34119112aa server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.502+0000 7f8ba17fa700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:27:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8ba17fa700 1 -- 172.21.15.81:0/2681903314 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4099a90 con 0x5611417ae1c0 2022-09-10T04:27:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8ba17fa700 1 -- 172.21.15.81:0/2681903314 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f8bb408ef00 con 0x5611417ae1c0 2022-09-10T04:27:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611417a0cf0 con 0x5611417ae1c0 2022-09-10T04:27:55.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8ba17fa700 1 -- 172.21.15.81:0/2681903314 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bb40af2b0 con 0x5611417ae1c0 2022-09-10T04:27:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8bd30b4b80 10 client.25123 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8bb819e000 con 0x5611417ae1c0 2022-09-10T04:27:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8ba17fa700 1 -- 172.21.15.81:0/2681903314 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8bb4078080 con 0x5611417ae1c0 2022-09-10T04:27:55.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8ba17fa700 1 -- 172.21.15.81:0/2681903314 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f8bb4091ce0 con 0x5611417ae1c0 2022-09-10T04:27:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.503+0000 7f8bd30b4b80 10 client.25123 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.504+0000 7f8bd30b4b80 10 client.25123 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-10T04:27:55.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.504+0000 7f8bd30b4b80 20 client.25123 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:55.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.504+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x5611417c29d0 0x5611417c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.504+0000 7f8bd30b4b80 4 client.25123 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:27:55.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.504+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 --> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56114177d4c0 con 0x5611417c29d0 2022-09-10T04:27:55.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.504+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x5611417c29d0 0x5611417c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.505+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x5611417c29d0 0x5611417c7c90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2acc5d02f2ae285 server_cookie=ca5326f808fd8bb6 in_seq=0 out_seq=0 2022-09-10T04:27:55.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.505+0000 7f8ba17fa700 10 client.25123 ms_handle_connect on v2:172.21.15.104:6838/2357131382 2022-09-10T04:27:55.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.505+0000 7f8ba17fa700 1 -- 172.21.15.81:0/2681903314 <== mds.1 v2:172.21.15.104:6838/2357131382 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x56114177d4c0 con 0x5611417c29d0 2022-09-10T04:27:55.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.505+0000 7f8ba17fa700 10 client.25123 handle_command_reply: tid=0 2022-09-10T04:27:55.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.505+0000 7f8bd30b4b80 1 client.25123 shutdown 2022-09-10T04:27:55.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417ae1c0 msgr2=0x5611417b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417ae1c0 0x5611417b76f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4092780 tx=0x7f8bb40921d0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 shutdown_connections 2022-09-10T04:27:55.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x5611417c29d0 0x5611417c7c90 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x5611417bec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417ae1c0 0x5611417b76f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2681903314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.541 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8ba17fa700 0 client.25123 ms_handle_reset on v2:172.21.15.104:6838/2357131382 2022-09-10T04:27:55.541 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 >> 172.21.15.81:0/2681903314 conn(0x5611417aa170 msgr2=0x5611417a7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.541 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 shutdown_connections 2022-09-10T04:27:55.541 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2681903314 wait complete. 2022-09-10T04:27:55.542 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.506+0000 7f8bd30b4b80 20 client.25123 trim_cache size 0 max 16384 2022-09-10T04:27:55.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 Processor -- start 2022-09-10T04:27:55.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 -- start start 2022-09-10T04:27:55.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611416c0080 con 0x5611417ae1c0 2022-09-10T04:27:55.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56114169e520 con 0x5611417af000 2022-09-10T04:27:55.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611416c8690 con 0x5611417af950 2022-09-10T04:27:55.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417cbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59340/0 (socket says 172.21.15.81:59340) 2022-09-10T04:27:55.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2442258927 learned_addr learned my addr 172.21.15.81:0/2442258927 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.524+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2442258927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 msgr2=0x5611417b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2442258927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 msgr2=0x5611417b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd99b700 1 -- 172.21.15.81:0/2442258927 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba8017070 con 0x5611417ae1c0 2022-09-10T04:27:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417cbcb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8bb408f260 tx=0x7f8bb4090da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65eecb2aa933ff server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bacbff700 1 -- 172.21.15.81:0/2442258927 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb40af2b0 con 0x5611417ae1c0 2022-09-10T04:27:55.552 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bacbff700 1 -- 172.21.15.81:0/2442258927 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5611417a0cf0 con 0x5611417ae1c0 2022-09-10T04:27:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.525+0000 7f8bacbff700 1 -- 172.21.15.81:0/2442258927 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4092a50 con 0x5611417ae1c0 2022-09-10T04:27:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2442258927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 msgr2=0x5611417cbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417cbcb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8bb408f260 tx=0x7f8bb4090da0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.553 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2442258927 shutdown_connections 2022-09-10T04:27:55.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af950 0x5611417b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417ae1c0 0x5611417cbcb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/2442258927 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417af000 0x5611417b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2442258927 >> 172.21.15.81:0/2442258927 conn(0x5611417aa170 msgr2=0x5611417a7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2442258927 shutdown_connections 2022-09-10T04:27:55.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.526+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/2442258927 wait complete. 2022-09-10T04:27:55.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.527+0000 7f8bd30b4b80 1 -- start start 2022-09-10T04:27:55.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.527+0000 7f8bd30b4b80 1 Processor -- start 2022-09-10T04:27:55.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 0x5611417bb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af000 0x5611417bbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x561141814c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611417bc570 con 0x5611417af950 2022-09-10T04:27:55.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x561141815170 con 0x5611417ae1c0 2022-09-10T04:27:55.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611418152b0 con 0x5611417af000 2022-09-10T04:27:55.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bd30b4b80 10 client.0 _ll_register_callbacks cb 0x5611416a7370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:55.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 0x5611417bb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bbd99b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 0x5611417bb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52242/0 (socket says 172.21.15.81:52242) 2022-09-10T04:27:55.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bbd99b700 1 -- 172.21.15.81:0/3980635820 learned_addr learned my addr 172.21.15.81:0/3980635820 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x561141814c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.528+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af000 0x5611417bbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbd99b700 1 -- 172.21.15.81:0/3980635820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af000 msgr2=0x5611417bbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af000 0x5611417bbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbd99b700 1 -- 172.21.15.81:0/3980635820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 msgr2=0x561141814c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x561141814c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbd99b700 1 -- 172.21.15.81:0/3980635820 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba802a040 con 0x5611417ae1c0 2022-09-10T04:27:55.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbe19c700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x561141814c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:27:55.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bbd99b700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 0x5611417bb950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400eff0 tx=0x7f8bb4092970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fec8a250d11235e5 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8ba17fa700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:27:55.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8ba17fa700 1 -- 172.21.15.81:0/3980635820 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4092ef0 con 0x5611417ae1c0 2022-09-10T04:27:55.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8ba17fa700 1 -- 172.21.15.81:0/3980635820 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f8bb4092c60 con 0x5611417ae1c0 2022-09-10T04:27:55.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ba8021070 con 0x5611417ae1c0 2022-09-10T04:27:55.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8ba17fa700 1 -- 172.21.15.81:0/3980635820 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bb408e5e0 con 0x5611417ae1c0 2022-09-10T04:27:55.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.529+0000 7f8bd30b4b80 10 client.25121 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:55.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8bb819e000 con 0x5611417ae1c0 2022-09-10T04:27:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8ba17fa700 1 -- 172.21.15.81:0/3980635820 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8ba8021070 con 0x5611417ae1c0 2022-09-10T04:27:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8ba17fa700 1 -- 172.21.15.81:0/3980635820 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f8bb40a9a30 con 0x5611417ae1c0 2022-09-10T04:27:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 10 client.25121 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 10 client.25121 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-10T04:27:55.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 20 client.25121 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x561141819ad0 0x561141816590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 4 client.25121 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2022-09-10T04:27:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 --> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5611416f56f0 con 0x561141819ad0 2022-09-10T04:27:55.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.530+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x561141819ad0 0x561141816590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8ba17fa700 10 client.25121 ms_handle_connect on v2:172.21.15.104:6838/2357131382 2022-09-10T04:27:55.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bbd19a700 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x561141819ad0 0x561141816590 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d57adc61944d0426 server_cookie=7669b9c8d4a350cf in_seq=0 out_seq=0 2022-09-10T04:27:55.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8ba17fa700 1 -- 172.21.15.81:0/3980635820 <== mds.1 v2:172.21.15.104:6838/2357131382 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5611416f56f0 con 0x561141819ad0 2022-09-10T04:27:55.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8ba17fa700 10 client.25121 handle_command_reply: tid=0 2022-09-10T04:27:55.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 client.25121 shutdown 2022-09-10T04:27:55.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 msgr2=0x5611417bb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 0x5611417bb950 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400eff0 tx=0x7f8bb4092970 comp rx=0 tx=0).stop 2022-09-10T04:27:55.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 shutdown_connections 2022-09-10T04:27:55.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x561141819ad0 0x561141816590 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5611417af950 0x561141814c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8ba17fa700 0 client.25121 ms_handle_reset on v2:172.21.15.104:6838/2357131382 2022-09-10T04:27:55.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5611417ae1c0 0x5611417bb950 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 --2- 172.21.15.81:0/3980635820 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5611417af000 0x5611417bbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 >> 172.21.15.81:0/3980635820 conn(0x5611417aa170 msgr2=0x5611417b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.568 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 shutdown_connections 2022-09-10T04:27:55.568 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.531+0000 7f8bd30b4b80 1 -- 172.21.15.81:0/3980635820 wait complete. 2022-09-10T04:27:55.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bd30b4b80 20 client.25121 trim_cache size 0 max 16384 2022-09-10T04:27:55.570 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:27:55.570 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:27:55.571 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 1, 2022-09-10T04:27:55.571 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24572, 2022-09-10T04:27:55.572 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:27:55.572 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:27:55.573 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:27:55.573 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 60.086537595000003, 2022-09-10T04:27:55.573 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:27:55.574 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 92, 2022-09-10T04:27:55.575 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:27:55.576 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 91.322811240999997 2022-09-10T04:27:55.576 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:27:55.580 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ba4099090 msgr2=0x7f8ba409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.580 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ba4099090 0x7f8ba409b540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8005fd0 tx=0x7f8ba8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 msgr2=0x7f8bb8195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 0x7f8bb8195910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8bb400e740 tx=0x7f8bb400ea50 comp rx=0 tx=0).stop 2022-09-10T04:27:55.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 shutdown_connections 2022-09-10T04:27:55.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8ba4099090 0x7f8ba409b540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bb80fd730 0x7f8bb819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb80fc430 0x7f8bb8195910 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 --2- 172.21.15.81:0/685757551 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8bb80fcd80 0x7f8bb8195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.532+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 >> 172.21.15.81:0/685757551 conn(0x7f8bb80f8160 msgr2=0x7f8bb80faf40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.533+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 shutdown_connections 2022-09-10T04:27:55.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.533+0000 7f8bbe99d700 1 -- 172.21.15.81:0/685757551 wait complete. 2022-09-10T04:27:55.587 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2022-09-10T04:27:55.816 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 -- 172.21.15.81:0/218307654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0072c90 msgr2=0x7f8fd0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/218307654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0072c90 0x7f8fd0071170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009750 tx=0x7f8fc4009a60 comp rx=0 tx=0).stop 2022-09-10T04:27:55.817 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 -- 172.21.15.81:0/218307654 shutdown_connections 2022-09-10T04:27:55.818 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/218307654 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd0105bc0 0x7f8fd0108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.818 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/218307654 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0072c90 0x7f8fd0071170 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.819 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/218307654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 -- 172.21.15.81:0/218307654 >> 172.21.15.81:0/218307654 conn(0x7f8fd00f93f0 msgr2=0x7f8fd00fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.820 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 -- 172.21.15.81:0/218307654 shutdown_connections 2022-09-10T04:27:55.821 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.812+0000 7f8fd6dff700 1 -- 172.21.15.81:0/218307654 wait complete. 2022-09-10T04:27:55.822 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 Processor -- start 2022-09-10T04:27:55.822 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 -- start start 2022-09-10T04:27:55.823 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd0072c90 0x7f8fd01961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0105bc0 0x7f8fd019d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.824 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd019d820 con 0x7f8fd0071740 2022-09-10T04:27:55.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd019d990 con 0x7f8fd0105bc0 2022-09-10T04:27:55.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd6dff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd019db50 con 0x7f8fd0072c90 2022-09-10T04:27:55.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd4b9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd4b9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59358/0 (socket says 172.21.15.81:59358) 2022-09-10T04:27:55.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd0072c90 0x7f8fd01961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3564473520 learned_addr learned my addr 172.21.15.81:0/3564473520 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.814+0000 7f8fd539c700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0105bc0 0x7f8fd019d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3564473520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd0072c90 msgr2=0x7f8fd01961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd0072c90 0x7f8fd01961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3564473520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0105bc0 msgr2=0x7f8fd019d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0105bc0 0x7f8fd019d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3564473520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc40093b0 con 0x7f8fd0071740 2022-09-10T04:27:55.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0195c60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4004230 tx=0x7f8fc4004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fcdffb700 1 -- 172.21.15.81:0/3564473520 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc401d070 con 0x7f8fd0071740 2022-09-10T04:27:55.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.815+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd019ddd0 con 0x7f8fd0071740 2022-09-10T04:27:55.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.816+0000 7f8fcdffb700 1 -- 172.21.15.81:0/3564473520 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8fc4003bb0 con 0x7f8fd0071740 2022-09-10T04:27:55.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.816+0000 7f8fcdffb700 1 -- 172.21.15.81:0/3564473520 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4017b80 con 0x7f8fd0071740 2022-09-10T04:27:55.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.816+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd019e1c0 con 0x7f8fd0071740 2022-09-10T04:27:55.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.820+0000 7f8fcdffb700 1 -- 172.21.15.81:0/3564473520 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f8fc400c040 con 0x7f8fd0071740 2022-09-10T04:27:55.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 Processor -- start 2022-09-10T04:27:55.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 -- start start 2022-09-10T04:27:55.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fcdffb700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8fc0099090 0x7f8fc009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fcdffb700 1 -- 172.21.15.81:0/3564473520 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f8fc40c3320 con 0x7f8fd0071740 2022-09-10T04:27:55.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521d2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d3430 0x55f3521d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d41f0 0x55f3521d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3520bd080 con 0x55f3521d41f0 2022-09-10T04:27:55.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3521279e0 con 0x55f3521d2ae0 2022-09-10T04:27:55.847 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.821+0000 7f8fea26db80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f352121180 con 0x55f3521d3430 2022-09-10T04:27:55.848 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521d2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd539c700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8fc0099090 0x7f8fc009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521d2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52278/0 (socket says 172.21.15.81:52278) 2022-09-10T04:27:55.849 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3420114599 learned_addr learned my addr 172.21.15.81:0/3420114599 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.850 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fcffff700 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d3430 0x55f3521d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.850 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd539c700 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d41f0 0x55f3521d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.850 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3420114599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d3430 msgr2=0x55f3521d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.851 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d3430 0x55f3521d3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3420114599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d41f0 msgr2=0x55f3521d4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.854 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d41f0 0x55f3521d4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.822+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3420114599 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f351fa95e0 con 0x55f3521d2ae0 2022-09-10T04:27:55.855 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521d2ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fc408f0e0 tx=0x7f8fc4017640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c98286310a693c5a server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fd539c700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8fc0099090 0x7f8fc009b540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009fd0 tx=0x7f8fc8009b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.856 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fbaffd700 1 -- 172.21.15.81:0/3420114599 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4093a60 con 0x55f3521d2ae0 2022-09-10T04:27:55.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fbaffd700 1 -- 172.21.15.81:0/3420114599 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8fc4093360 con 0x55f3521d2ae0 2022-09-10T04:27:55.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fbaffd700 1 -- 172.21.15.81:0/3420114599 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4028350 con 0x55f3521d2ae0 2022-09-10T04:27:55.857 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fea26db80 1 -- 172.21.15.81:0/3420114599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 msgr2=0x55f3521d2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.823+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521d2ef0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8fc408f0e0 tx=0x7f8fc4017640 comp rx=0 tx=0).stop 2022-09-10T04:27:55.858 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 -- 172.21.15.81:0/3420114599 shutdown_connections 2022-09-10T04:27:55.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d41f0 0x55f3521d4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.859 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521d2ef0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.860 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3420114599 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d3430 0x55f3521d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 -- 172.21.15.81:0/3420114599 >> 172.21.15.81:0/3420114599 conn(0x55f3521cefa0 msgr2=0x55f3521cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.861 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 -- 172.21.15.81:0/3420114599 shutdown_connections 2022-09-10T04:27:55.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 -- 172.21.15.81:0/3420114599 wait complete. 2022-09-10T04:27:55.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 -- start start 2022-09-10T04:27:55.862 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.824+0000 7f8fea26db80 1 Processor -- start 2022-09-10T04:27:55.863 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.863 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521dc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.864 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.864 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f35200f0c0 con 0x55f3521d3430 2022-09-10T04:27:55.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3521e3b10 con 0x55f3521d2ae0 2022-09-10T04:27:55.865 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3521e3ca0 con 0x55f3521d41f0 2022-09-10T04:27:55.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fea26db80 10 client.0 _ll_register_callbacks cb 0x55f3520cc370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:55.866 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fd4b9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fd4b9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521dc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52282/0 (socket says 172.21.15.81:52282) 2022-09-10T04:27:55.867 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3611679494 learned_addr learned my addr 172.21.15.81:0/3611679494 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.868 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521dc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.825+0000 7f8fd539c700 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.869 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3611679494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 msgr2=0x55f3521e35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521e35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.870 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 msgr2=0x55f3521dc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521dc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.871 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fd4b9b700 1 -- 172.21.15.81:0/3611679494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd019e1c0 con 0x55f3521d2ae0 2022-09-10T04:27:55.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521dc050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400f500 tx=0x7f8fc400ff90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4da62e14fb9f6751 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fabfff700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:27:55.872 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fabfff700 1 -- 172.21.15.81:0/3611679494 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc409a600 con 0x55f3521d2ae0 2022-09-10T04:27:55.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fabfff700 1 -- 172.21.15.81:0/3611679494 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f8fc408fa70 con 0x55f3521d2ae0 2022-09-10T04:27:55.873 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f3521c5aa0 con 0x55f3521d2ae0 2022-09-10T04:27:55.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.826+0000 7f8fabfff700 1 -- 172.21.15.81:0/3611679494 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fc408fc30 con 0x55f3521d2ae0 2022-09-10T04:27:55.874 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 10 client.25133 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:55.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f351f55d90 con 0x55f3521d2ae0 2022-09-10T04:27:55.875 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fabfff700 1 -- 172.21.15.81:0/3611679494 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55f3521c5aa0 con 0x55f3521d2ae0 2022-09-10T04:27:55.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fabfff700 1 -- 172.21.15.81:0/3611679494 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f8fc40928e0 con 0x55f3521d2ae0 2022-09-10T04:27:55.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 10 client.25133 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:55.876 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 10 client.25133 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-10T04:27:55.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 20 client.25133 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:55.877 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f3521e6e80 0x55f3521e5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 4 client.25133 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:27:55.878 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.827+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3521a24c0 con 0x55f3521e6e80 2022-09-10T04:27:55.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.828+0000 7f8fcffff700 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f3521e6e80 0x55f3521e5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.879 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.828+0000 7f8fabfff700 10 client.25133 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:27:55.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.828+0000 7f8fcffff700 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f3521e6e80 0x55f3521e5690 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95092444c2362f5d server_cookie=f529901b22c5466a in_seq=0 out_seq=0 2022-09-10T04:27:55.880 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fabfff700 1 -- 172.21.15.81:0/3611679494 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55f3521a24c0 con 0x55f3521e6e80 2022-09-10T04:27:55.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fabfff700 10 client.25133 handle_command_reply: tid=0 2022-09-10T04:27:55.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 client.25133 shutdown 2022-09-10T04:27:55.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 msgr2=0x55f3521dc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.881 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521dc050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400f500 tx=0x7f8fc400ff90 comp rx=0 tx=0).stop 2022-09-10T04:27:55.882 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 shutdown_connections 2022-09-10T04:27:55.882 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f3521e6e80 0x55f3521e5690 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521dc050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 --2- 172.21.15.81:0/3611679494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521dc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.883 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fabfff700 0 client.25133 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:27:55.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 >> 172.21.15.81:0/3611679494 conn(0x55f3521cefa0 msgr2=0x55f3521cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 shutdown_connections 2022-09-10T04:27:55.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.829+0000 7f8fea26db80 1 -- 172.21.15.81:0/3611679494 wait complete. 2022-09-10T04:27:55.884 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.830+0000 7f8fea26db80 20 client.25133 trim_cache size 0 max 16384 2022-09-10T04:27:55.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 Processor -- start 2022-09-10T04:27:55.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 -- start start 2022-09-10T04:27:55.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521d6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.892 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521dbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.892 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f352033ed0 con 0x55f3521d3430 2022-09-10T04:27:55.892 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3520e5050 con 0x55f3521d2ae0 2022-09-10T04:27:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fea26db80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f351fada30 con 0x55f3521d41f0 2022-09-10T04:27:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.848+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521d6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521d6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59386/0 (socket says 172.21.15.81:59386) 2022-09-10T04:27:55.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 -- 172.21.15.81:0/944521986 learned_addr learned my addr 172.21.15.81:0/944521986 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fd539c700 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521dbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 -- 172.21.15.81:0/944521986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 msgr2=0x55f3521dbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521dbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 -- 172.21.15.81:0/944521986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 msgr2=0x55f3521cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 -- 172.21.15.81:0/944521986 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f351f54260 con 0x55f3521d3430 2022-09-10T04:27:55.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:27:55.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fcffff700 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521d6e90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55f3521dd530 tx=0x7f8fbc005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265fdecca6b9b746 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.849+0000 7f8fbb7fe700 1 -- 172.21.15.81:0/944521986 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc017040 con 0x55f3521d3430 2022-09-10T04:27:55.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fbb7fe700 1 -- 172.21.15.81:0/944521986 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8fbc018040 con 0x55f3521d3430 2022-09-10T04:27:55.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fbb7fe700 1 -- 172.21.15.81:0/944521986 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc010690 con 0x55f3521d3430 2022-09-10T04:27:55.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 -- 172.21.15.81:0/944521986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 msgr2=0x55f3521d6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521d6e90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x55f3521dd530 tx=0x7f8fbc005f60 comp rx=0 tx=0).stop 2022-09-10T04:27:55.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 -- 172.21.15.81:0/944521986 shutdown_connections 2022-09-10T04:27:55.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521dbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 --2- 172.21.15.81:0/944521986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521d6e90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 -- 172.21.15.81:0/944521986 >> 172.21.15.81:0/944521986 conn(0x55f3521cefa0 msgr2=0x55f3521cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 -- 172.21.15.81:0/944521986 shutdown_connections 2022-09-10T04:27:55.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.850+0000 7f8fea26db80 1 -- 172.21.15.81:0/944521986 wait complete. 2022-09-10T04:27:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 -- start start 2022-09-10T04:27:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 Processor -- start 2022-09-10T04:27:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521ed230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521ed770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521f0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3521f0cd0 con 0x55f3521d3430 2022-09-10T04:27:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3521f0e40 con 0x55f3521d2ae0 2022-09-10T04:27:55.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.851+0000 7f8fea26db80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3521f1000 con 0x55f3521d41f0 2022-09-10T04:27:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fea26db80 10 client.0 _ll_register_callbacks cb 0x55f3520cc370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:27:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fd539c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521f0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fcffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521ed770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fd539c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521f0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:37134/0 (socket says 172.21.15.81:37134) 2022-09-10T04:27:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fd539c700 1 -- 172.21.15.81:0/1613565031 learned_addr learned my addr 172.21.15.81:0/1613565031 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521ed230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.904 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fcffff700 1 -- 172.21.15.81:0/1613565031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 msgr2=0x55f3521f0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fcffff700 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521f0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fcffff700 1 -- 172.21.15.81:0/1613565031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 msgr2=0x55f3521ed230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:27:55.905 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fcffff700 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521ed230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.852+0000 7f8fcffff700 1 -- 172.21.15.81:0/1613565031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f351f55d90 con 0x55f3521d3430 2022-09-10T04:27:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fcffff700 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521ed770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc004d90 tx=0x7f8fbc009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305587113cd772ca server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fabfff700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:27:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fabfff700 1 -- 172.21.15.81:0/1613565031 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc02c070 con 0x55f3521d3430 2022-09-10T04:27:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fc4093360 con 0x55f3521d3430 2022-09-10T04:27:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fabfff700 1 -- 172.21.15.81:0/1613565031 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f8fbc004820 con 0x55f3521d3430 2022-09-10T04:27:55.907 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fabfff700 1 -- 172.21.15.81:0/1613565031 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fbc0049e0 con 0x55f3521d3430 2022-09-10T04:27:55.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fea26db80 10 client.15036 fetch_fsmap learned FSMap version 28 2022-09-10T04:27:55.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8fbc009e10 con 0x55f3521d3430 2022-09-10T04:27:55.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.853+0000 7f8fabfff700 1 -- 172.21.15.81:0/1613565031 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8fc4093360 con 0x55f3521d3430 2022-09-10T04:27:55.908 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fabfff700 1 -- 172.21.15.81:0/1613565031 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f8fbc0109f0 con 0x55f3521d3430 2022-09-10T04:27:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fea26db80 10 client.15036 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:27:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fea26db80 10 client.15036 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-10T04:27:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fea26db80 20 client.15036 populate_metadata read hostname 'smithi081' 2022-09-10T04:27:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fea26db80 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f352246bf0 0x55f352243560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:55.909 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fea26db80 4 client.15036 mds_command: new command op to 24577 tid=0[{"prefix": "status"}] 2022-09-10T04:27:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f35211a6f0 con 0x55f352246bf0 2022-09-10T04:27:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.854+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f352246bf0 0x55f352243560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fabfff700 10 client.15036 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:27:55.910 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fd4b9b700 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f352246bf0 0x55f352243560 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3887623d9e69609 server_cookie=ad8b7a1c89d0a0f1 in_seq=0 out_seq=0 2022-09-10T04:27:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fabfff700 1 -- 172.21.15.81:0/1613565031 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f35211a6f0 con 0x55f352246bf0 2022-09-10T04:27:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fabfff700 10 client.15036 handle_command_reply: tid=0 2022-09-10T04:27:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 client.15036 shutdown 2022-09-10T04:27:55.911 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 msgr2=0x55f3521ed770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521ed770 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc004d90 tx=0x7f8fbc009850 comp rx=0 tx=0).stop 2022-09-10T04:27:55.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 shutdown_connections 2022-09-10T04:27:55.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f3521d41f0 0x55f3521f0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55f352246bf0 0x55f352243560 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.912 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f3521d2ae0 0x55f3521ed230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 --2- 172.21.15.81:0/1613565031 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3521d3430 0x55f3521ed770 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fabfff700 0 client.15036 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:27:55.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 >> 172.21.15.81:0/1613565031 conn(0x55f3521cefa0 msgr2=0x55f3521dbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.913 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 shutdown_connections 2022-09-10T04:27:55.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 1 -- 172.21.15.81:0/1613565031 wait complete. 2022-09-10T04:27:55.914 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.855+0000 7f8fea26db80 20 client.15036 trim_cache size 0 max 16384 2022-09-10T04:27:55.915 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:27:55.915 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:27:55.915 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2022-09-10T04:27:55.915 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24577, 2022-09-10T04:27:55.916 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:27:55.916 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:27:55.916 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:27:55.917 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 83.755612338000006, 2022-09-10T04:27:55.917 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:27:55.917 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 92, 2022-09-10T04:27:55.917 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:27:55.918 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 93.843226580000007 2022-09-10T04:27:55.918 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:27:55.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8fc0099090 msgr2=0x7f8fc009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8fc0099090 0x7f8fc009b540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8009fd0 tx=0x7f8fc8009b60 comp rx=0 tx=0).stop 2022-09-10T04:27:55.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 msgr2=0x7f8fd0195c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:55.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0195c60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4004230 tx=0x7f8fc4004260 comp rx=0 tx=0).stop 2022-09-10T04:27:55.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 shutdown_connections 2022-09-10T04:27:55.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd0105bc0 0x7f8fd019d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0071740 0x7f8fd0195c60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f8fc0099090 0x7f8fc009b540 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 --2- 172.21.15.81:0/3564473520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8fd0072c90 0x7f8fd01961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:55.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 >> 172.21.15.81:0/3564473520 conn(0x7f8fd00f93f0 msgr2=0x7f8fd00fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:55.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 shutdown_connections 2022-09-10T04:27:55.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:55.856+0000 7f8fd6dff700 1 -- 172.21.15.81:0/3564473520 wait complete. 2022-09-10T04:27:55.922 INFO:tasks.cephadm:Setup complete, yielding 2022-09-10T04:27:55.923 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-09-10T04:27:55.946 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:27:55.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph orch status' 2022-09-10T04:27:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:56 smithi104 ceph-mon[97044]: pgmap v345: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:27:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:56 smithi081 ceph-mon[90133]: pgmap v345: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:27:56.823 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:56 smithi110 ceph-mon[97633]: pgmap v345: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:27:58.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 -- 172.21.15.81:0/4193206078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40feed0 msgr2=0x7fb1d40ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:58.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/4193206078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40feed0 0x7fb1d40ff320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009810 tx=0x7fb1c4009b20 comp rx=0 tx=0).stop 2022-09-10T04:27:58.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 -- 172.21.15.81:0/4193206078 shutdown_connections 2022-09-10T04:27:58.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/4193206078 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40ff860 0x7fb1d4106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/4193206078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40fdc50 0x7fb1d40fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/4193206078 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40feed0 0x7fb1d40ff320 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 -- 172.21.15.81:0/4193206078 >> 172.21.15.81:0/4193206078 conn(0x7fb1d40f9300 msgr2=0x7fb1d40fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:58.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 -- 172.21.15.81:0/4193206078 shutdown_connections 2022-09-10T04:27:58.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.217+0000 7fb1db0c5700 1 -- 172.21.15.81:0/4193206078 wait complete. 2022-09-10T04:27:58.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.218+0000 7fb1db0c5700 1 Processor -- start 2022-09-10T04:27:58.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.218+0000 7fb1db0c5700 1 -- start start 2022-09-10T04:27:58.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.218+0000 7fb1db0c5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40fdc50 0x7fb1d4195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:58.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.218+0000 7fb1db0c5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40feed0 0x7fb1d4195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:58.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1db0c5700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40ff860 0x7fb1d419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:58.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1db0c5700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d419d5c0 con 0x7fb1d40ff860 2022-09-10T04:27:58.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1db0c5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d419d730 con 0x7fb1d40fdc50 2022-09-10T04:27:58.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1db0c5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d419d8f0 con 0x7fb1d40feed0 2022-09-10T04:27:58.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1d98c2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40feed0 0x7fb1d4195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:58.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1da0c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40fdc50 0x7fb1d4195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:58.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1da0c3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40fdc50 0x7fb1d4195a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52312/0 (socket says 172.21.15.81:52312) 2022-09-10T04:27:58.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1da0c3700 1 -- 172.21.15.81:0/613542706 learned_addr learned my addr 172.21.15.81:0/613542706 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:27:58.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1da8c4700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40ff860 0x7fb1d419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:58.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1d98c2700 1 -- 172.21.15.81:0/613542706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40fdc50 msgr2=0x7fb1d4195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:58.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1d98c2700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40fdc50 0x7fb1d4195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1d98c2700 1 -- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40ff860 msgr2=0x7fb1d419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1d98c2700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40ff860 0x7fb1d419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1d98c2700 1 -- 172.21.15.81:0/613542706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4009470 con 0x7fb1d40feed0 2022-09-10T04:27:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.219+0000 7fb1da8c4700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40ff860 0x7fb1d419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:27:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.220+0000 7fb1d98c2700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40feed0 0x7fb1d4195f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4000c00 tx=0x7fb1c400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:58.229 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.220+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c401d070 con 0x7fb1d40feed0 2022-09-10T04:27:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.220+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d419db70 con 0x7fb1d40feed0 2022-09-10T04:27:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.220+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d419e070 con 0x7fb1d40feed0 2022-09-10T04:27:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.220+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fb1c400f930 con 0x7fb1d40feed0 2022-09-10T04:27:58.230 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.220+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4004870 con 0x7fb1d40feed0 2022-09-10T04:27:58.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.221+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b80051a0 con 0x7fb1d40feed0 2022-09-10T04:27:58.231 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.223+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fb1c4028030 con 0x7fb1d40feed0 2022-09-10T04:27:58.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.224+0000 7fb1cb7fe700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1c0099010 0x7fb1c009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:27:58.232 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.224+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fb1c40c3200 con 0x7fb1d40feed0 2022-09-10T04:27:58.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.225+0000 7fb1da0c3700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1c0099010 0x7fb1c009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:27:58.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.225+0000 7fb1da0c3700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1c0099010 0x7fb1c009b4c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009b70 tx=0x7fb1d0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:27:58.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.226+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fb1c408f580 con 0x7fb1d40feed0 2022-09-10T04:27:58.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.459+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1b8000bf0 con 0x7fb1c0099010 2022-09-10T04:27:58.461 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.460+0000 7fb1cb7fe700 1 -- 172.21.15.81:0/613542706 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb1b8000bf0 con 0x7fb1c0099010 2022-09-10T04:27:58.462 INFO:teuthology.orchestra.run.smithi081.stdout:Backend: cephadm 2022-09-10T04:27:58.462 INFO:teuthology.orchestra.run.smithi081.stdout:Available: Yes 2022-09-10T04:27:58.462 INFO:teuthology.orchestra.run.smithi081.stdout:Paused: No 2022-09-10T04:27:58.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.462+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1c0099010 msgr2=0x7fb1c009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:58.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.462+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1c0099010 0x7fb1c009b4c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009b70 tx=0x7fb1d0005ea0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.462+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40feed0 msgr2=0x7fb1d4195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:27:58.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.462+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40feed0 0x7fb1d4195f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4000c00 tx=0x7fb1c400f690 comp rx=0 tx=0).stop 2022-09-10T04:27:58.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.462+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 shutdown_connections 2022-09-10T04:27:58.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.462+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d40ff860 0x7fb1d419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.463+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1d40fdc50 0x7fb1d4195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.463+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fb1c0099010 0x7fb1c009b4c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.463+0000 7fb1db0c5700 1 --2- 172.21.15.81:0/613542706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1d40feed0 0x7fb1d4195f80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:27:58.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.463+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 >> 172.21.15.81:0/613542706 conn(0x7fb1d40f9300 msgr2=0x7fb1d40fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:27:58.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.463+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 shutdown_connections 2022-09-10T04:27:58.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:27:58.463+0000 7fb1db0c5700 1 -- 172.21.15.81:0/613542706 wait complete. 2022-09-10T04:27:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:58 smithi104 ceph-mon[97044]: pgmap v346: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:58 smithi081 ceph-mon[90133]: pgmap v346: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:58.891 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph orch ps' 2022-09-10T04:27:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:58 smithi110 ceph-mon[97633]: pgmap v346: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:27:59.770 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:27:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:27:59 smithi104 ceph-mon[97044]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:27:59.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:27:59 smithi081 ceph-mon[90133]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:27:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:27:59 smithi110 ceph-mon[97633]: from='client.25153 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:00.717 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:00 smithi081 ceph-mon[90133]: pgmap v347: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:00 smithi104 ceph-mon[97044]: pgmap v347: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:00.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:00 smithi110 ceph-mon[97633]: pgmap v347: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:01.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 -- 172.21.15.81:0/1599085363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164075290 msgr2=0x7f4164075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:01.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 --2- 172.21.15.81:0/1599085363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164075290 0x7f4164075b20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f415c009bb0 tx=0x7f415c009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 -- 172.21.15.81:0/1599085363 shutdown_connections 2022-09-10T04:28:01.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 --2- 172.21.15.81:0/1599085363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164075290 0x7f4164075b20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 --2- 172.21.15.81:0/1599085363 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164073610 0x7f4164073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 --2- 172.21.15.81:0/1599085363 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41640748e0 0x7f4164074d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 -- 172.21.15.81:0/1599085363 >> 172.21.15.81:0/1599085363 conn(0x7f41640fb6b0 msgr2=0x7f41640fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:01.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 -- 172.21.15.81:0/1599085363 shutdown_connections 2022-09-10T04:28:01.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.292+0000 7f4169eb4700 1 -- 172.21.15.81:0/1599085363 wait complete. 2022-09-10T04:28:01.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.293+0000 7f4169eb4700 1 Processor -- start 2022-09-10T04:28:01.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.293+0000 7f4169eb4700 1 -- start start 2022-09-10T04:28:01.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4169eb4700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164073610 0x7f4164199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:01.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4169eb4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41640748e0 0x7f416419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:01.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4169eb4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164075290 0x7f41641a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:01.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4169eb4700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a19a0 con 0x7f4164073610 2022-09-10T04:28:01.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4169eb4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a1b10 con 0x7f4164075290 2022-09-10T04:28:01.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4169eb4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a1cd0 con 0x7f41640748e0 2022-09-10T04:28:01.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4168eb2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164073610 0x7f4164199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:01.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f41696b3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164075290 0x7f41641a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:01.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4168eb2700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164073610 0x7f4164199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55168/0 (socket says 172.21.15.81:55168) 2022-09-10T04:28:01.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4163fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41640748e0 0x7f416419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:01.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.294+0000 7f4168eb2700 1 -- 172.21.15.81:0/2992185041 learned_addr learned my addr 172.21.15.81:0/2992185041 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:01.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f41696b3700 1 -- 172.21.15.81:0/2992185041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41640748e0 msgr2=0x7f416419a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:01.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f41696b3700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41640748e0 0x7f416419a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f41696b3700 1 -- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164073610 msgr2=0x7f4164199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:01.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f41696b3700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164073610 0x7f4164199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f41696b3700 1 -- 172.21.15.81:0/2992185041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0097e0 con 0x7f4164075290 2022-09-10T04:28:01.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f41696b3700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164075290 0x7f41641a13d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f415c009fd0 tx=0x7f415c00b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:01.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c00f870 con 0x7f4164075290 2022-09-10T04:28:01.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41641a1f50 con 0x7f4164075290 2022-09-10T04:28:01.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f415c00ba80 con 0x7f4164075290 2022-09-10T04:28:01.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.295+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c017470 con 0x7f4164075290 2022-09-10T04:28:01.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.296+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41641a2370 con 0x7f4164075290 2022-09-10T04:28:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.297+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416410fae0 con 0x7f4164075290 2022-09-10T04:28:01.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.299+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f415c029070 con 0x7f4164075290 2022-09-10T04:28:01.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.300+0000 7f4161ffb700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4154099090 0x7f415409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:01.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.300+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f415c0c3600 con 0x7f4164075290 2022-09-10T04:28:01.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.300+0000 7f4168eb2700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4154099090 0x7f415409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:01.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.301+0000 7f4168eb2700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4154099090 0x7f415409b540 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41580057d0 tx=0x7f4158005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:01.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.302+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f415c08f980 con 0x7f4164075290 2022-09-10T04:28:01.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.525+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f416405fbb0 con 0x7f4154099090 2022-09-10T04:28:01.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.538+0000 7f4161ffb700 1 -- 172.21.15.81:0/2992185041 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4216 (secure 0 0 0) 0x7f416405fbb0 con 0x7f4154099090 2022-09-10T04:28:01.540 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2022-09-10T04:28:01.541 INFO:teuthology.orchestra.run.smithi081.stdout:mds.a smithi081 running (99s) 17s ago 99s 22.0M - 17.0.0-14898-g24441860 99abe8ecf349 fe187550bf09 2022-09-10T04:28:01.541 INFO:teuthology.orchestra.run.smithi081.stdout:mds.b smithi104 running (97s) 24s ago 97s 19.7M - 17.0.0-14898-g24441860 99abe8ecf349 2d318a5ae4e0 2022-09-10T04:28:01.541 INFO:teuthology.orchestra.run.smithi081.stdout:mds.c smithi110 running (111s) 24s ago 111s 17.2M - 17.0.0-14898-g24441860 99abe8ecf349 a775a83fe1d7 2022-09-10T04:28:01.541 INFO:teuthology.orchestra.run.smithi081.stdout:mds.d smithi081 running (118s) 17s ago 117s 13.5M - 17.0.0-14898-g24441860 99abe8ecf349 61412591081a 2022-09-10T04:28:01.542 INFO:teuthology.orchestra.run.smithi081.stdout:mds.e smithi104 running (115s) 24s ago 115s 19.4M - 17.0.0-14898-g24441860 99abe8ecf349 5e6a16cb1689 2022-09-10T04:28:01.542 INFO:teuthology.orchestra.run.smithi081.stdout:mds.f smithi110 running (113s) 24s ago 113s 17.2M - 17.0.0-14898-g24441860 99abe8ecf349 b9ebe1cc8934 2022-09-10T04:28:01.542 INFO:teuthology.orchestra.run.smithi081.stdout:mds.g smithi081 running (103s) 17s ago 103s 13.4M - 17.0.0-14898-g24441860 99abe8ecf349 25648abfd1f2 2022-09-10T04:28:01.543 INFO:teuthology.orchestra.run.smithi081.stdout:mds.h smithi104 running (109s) 24s ago 108s 17.5M - 17.0.0-14898-g24441860 99abe8ecf349 9adbd23de36e 2022-09-10T04:28:01.543 INFO:teuthology.orchestra.run.smithi081.stdout:mds.i smithi110 running (107s) 24s ago 106s 18.2M - 17.0.0-14898-g24441860 99abe8ecf349 fb42ce2c17f0 2022-09-10T04:28:01.543 INFO:teuthology.orchestra.run.smithi081.stdout:mds.j smithi081 running (2m) 17s ago 2m 16.1M - 17.0.0-14898-g24441860 99abe8ecf349 ff524dd3055a 2022-09-10T04:28:01.543 INFO:teuthology.orchestra.run.smithi081.stdout:mds.k smithi104 running (2m) 24s ago 2m 17.3M - 17.0.0-14898-g24441860 99abe8ecf349 22ffb927d72c 2022-09-10T04:28:01.543 INFO:teuthology.orchestra.run.smithi081.stdout:mds.l smithi110 running (2m) 24s ago 2m 20.7M - 17.0.0-14898-g24441860 99abe8ecf349 b77c02ce623d 2022-09-10T04:28:01.544 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.x smithi081 *:9283,8765 running (9m) 17s ago 9m 499M - 17.0.0-14898-g24441860 99abe8ecf349 36452bba90b3 2022-09-10T04:28:01.544 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.y smithi104 *:8443 running (7m) 24s ago 7m 431M - 17.0.0-14898-g24441860 99abe8ecf349 f52df5b9ae1d 2022-09-10T04:28:01.544 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.z smithi110 *:8443 running (7m) 24s ago 7m 431M - 17.0.0-14898-g24441860 99abe8ecf349 975c4b5e3658 2022-09-10T04:28:01.544 INFO:teuthology.orchestra.run.smithi081.stdout:mon.a smithi081 running (9m) 17s ago 9m 94.9M 2048M 17.0.0-14898-g24441860 99abe8ecf349 93806b9c523d 2022-09-10T04:28:01.545 INFO:teuthology.orchestra.run.smithi081.stdout:mon.b smithi104 running (8m) 24s ago 8m 65.7M 2048M 17.0.0-14898-g24441860 99abe8ecf349 0ea210629c23 2022-09-10T04:28:01.545 INFO:teuthology.orchestra.run.smithi081.stdout:mon.c smithi110 running (8m) 24s ago 8m 66.0M 2048M 17.0.0-14898-g24441860 99abe8ecf349 3c92599d31f1 2022-09-10T04:28:01.545 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi081 running (7m) 17s ago 7m 70.6M 4317M 17.0.0-14898-g24441860 99abe8ecf349 1f9ce7781477 2022-09-10T04:28:01.545 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi104 running (6m) 24s ago 6m 78.5M 4292M 17.0.0-14898-g24441860 99abe8ecf349 e696d78e1aae 2022-09-10T04:28:01.545 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi110 running (6m) 24s ago 6m 74.0M 4292M 17.0.0-14898-g24441860 99abe8ecf349 f3f90def5a88 2022-09-10T04:28:01.546 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi081 running (5m) 17s ago 5m 68.4M 4317M 17.0.0-14898-g24441860 99abe8ecf349 1d8e9a8938ed 2022-09-10T04:28:01.546 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi104 running (5m) 24s ago 5m 66.4M 4292M 17.0.0-14898-g24441860 99abe8ecf349 a01e15294b87 2022-09-10T04:28:01.546 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi110 running (5m) 24s ago 5m 69.7M 4292M 17.0.0-14898-g24441860 99abe8ecf349 71d0e0f56b45 2022-09-10T04:28:01.546 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi081 running (4m) 17s ago 4m 63.0M 4317M 17.0.0-14898-g24441860 99abe8ecf349 3ae375b1e8a4 2022-09-10T04:28:01.547 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi104 running (4m) 24s ago 4m 65.8M 4292M 17.0.0-14898-g24441860 99abe8ecf349 bc11a4fff98b 2022-09-10T04:28:01.547 INFO:teuthology.orchestra.run.smithi081.stdout:osd.8 smithi110 running (3m) 24s ago 3m 66.0M 4292M 17.0.0-14898-g24441860 99abe8ecf349 1a0fc8b47ffb 2022-09-10T04:28:01.547 INFO:teuthology.orchestra.run.smithi081.stdout:osd.9 smithi081 running (3m) 17s ago 3m 60.0M 4317M 17.0.0-14898-g24441860 99abe8ecf349 e4d601f20edb 2022-09-10T04:28:01.547 INFO:teuthology.orchestra.run.smithi081.stdout:osd.10 smithi104 running (2m) 24s ago 2m 61.3M 4292M 17.0.0-14898-g24441860 99abe8ecf349 a42a43e779d5 2022-09-10T04:28:01.548 INFO:teuthology.orchestra.run.smithi081.stdout:osd.11 smithi110 running (2m) 24s ago 2m 55.8M 4292M 17.0.0-14898-g24441860 99abe8ecf349 8921640f84fe 2022-09-10T04:28:01.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4154099090 msgr2=0x7f415409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:01.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4154099090 0x7f415409b540 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f41580057d0 tx=0x7f4158005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164075290 msgr2=0x7f41641a13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:01.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164075290 0x7f41641a13d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f415c009fd0 tx=0x7f415c00b890 comp rx=0 tx=0).stop 2022-09-10T04:28:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 shutdown_connections 2022-09-10T04:28:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4164075290 0x7f41641a13d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4164073610 0x7f4164199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.550 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4154099090 0x7f415409b540 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 --2- 172.21.15.81:0/2992185041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41640748e0 0x7f416419a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 >> 172.21.15.81:0/2992185041 conn(0x7f41640fb6b0 msgr2=0x7f41640fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 shutdown_connections 2022-09-10T04:28:01.551 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:01.541+0000 7f4169eb4700 1 -- 172.21.15.81:0/2992185041 wait complete. 2022-09-10T04:28:02.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph orch ls' 2022-09-10T04:28:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:02 smithi104 ceph-mon[97044]: pgmap v348: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:28:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:02 smithi104 ceph-mon[97044]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:02 smithi081 ceph-mon[90133]: pgmap v348: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:28:02.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:02 smithi081 ceph-mon[90133]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:02 smithi110 ceph-mon[97633]: pgmap v348: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T04:28:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:02 smithi110 ceph-mon[97633]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:02.917 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:28:04.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.466+0000 7fcfcd657700 1 -- 172.21.15.81:0/2432965581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fc630 msgr2=0x7fcfc80fca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:04.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.466+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2432965581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fc630 0x7fcfc80fca40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8009750 tx=0x7fcfb8009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:04.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 -- 172.21.15.81:0/2432965581 shutdown_connections 2022-09-10T04:28:04.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2432965581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fd930 0x7fcfc80fdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2432965581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fc630 0x7fcfc80fca40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2432965581 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fcf80 0x7fcfc80fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 -- 172.21.15.81:0/2432965581 >> 172.21.15.81:0/2432965581 conn(0x7fcfc80f82a0 msgr2=0x7fcfc80fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:04.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 -- 172.21.15.81:0/2432965581 shutdown_connections 2022-09-10T04:28:04.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.467+0000 7fcfcd657700 1 -- 172.21.15.81:0/2432965581 wait complete. 2022-09-10T04:28:04.474 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.468+0000 7fcfcd657700 1 Processor -- start 2022-09-10T04:28:04.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.468+0000 7fcfcd657700 1 -- start start 2022-09-10T04:28:04.475 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcd657700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fc630 0x7fcfc8195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:04.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcd657700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 0x7fcfc8195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:04.476 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcd657700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fd930 0x7fcfc819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:04.477 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcd657700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc819d5e0 con 0x7fcfc80fcf80 2022-09-10T04:28:04.477 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcd657700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc819d750 con 0x7fcfc80fc630 2022-09-10T04:28:04.477 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcd657700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfc819d910 con 0x7fcfc80fd930 2022-09-10T04:28:04.478 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 0x7fcfc8195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:04.478 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 0x7fcfc8195fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55176/0 (socket says 172.21.15.81:55176) 2022-09-10T04:28:04.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 -- 172.21.15.81:0/2597855140 learned_addr learned my addr 172.21.15.81:0/2597855140 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:04.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc7fff700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fc630 0x7fcfc8195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:04.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfcce56700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fd930 0x7fcfc819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:04.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 -- 172.21.15.81:0/2597855140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fd930 msgr2=0x7fcfc819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:04.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fd930 0x7fcfc819d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 -- 172.21.15.81:0/2597855140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fc630 msgr2=0x7fcfc8195a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:04.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.469+0000 7fcfc77fe700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fc630 0x7fcfc8195a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.482 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfc77fe700 1 -- 172.21.15.81:0/2597855140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb80093b0 con 0x7fcfc80fcf80 2022-09-10T04:28:04.482 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfc7fff700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fc630 0x7fcfc8195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:04.483 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfc77fe700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 0x7fcfc8195fa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00e740 tx=0x7fcfbc00ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:04.483 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc005220 con 0x7fcfc80fcf80 2022-09-10T04:28:04.484 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcfbc005380 con 0x7fcfc80fcf80 2022-09-10T04:28:04.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfc819dbf0 con 0x7fcfc80fcf80 2022-09-10T04:28:04.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfbc010ba0 con 0x7fcfc80fcf80 2022-09-10T04:28:04.485 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.470+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfc819e150 con 0x7fcfc80fcf80 2022-09-10T04:28:04.486 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.471+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfc810b600 con 0x7fcfc80fcf80 2022-09-10T04:28:04.486 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.473+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fcfbc01e020 con 0x7fcfc80fcf80 2022-09-10T04:28:04.488 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.474+0000 7fcfc57fa700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfb0099090 0x7fcfb009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:04.489 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.474+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7fcfbc014070 con 0x7fcfc80fcf80 2022-09-10T04:28:04.489 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.475+0000 7fcfc7fff700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfb0099090 0x7fcfb009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:04.489 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.475+0000 7fcfc7fff700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfb0099090 0x7fcfb009b540 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800f500 tx=0x7fcfb8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:04.490 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.478+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcfbc08ecc0 con 0x7fcfc80fcf80 2022-09-10T04:28:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:04 smithi104 ceph-mon[97044]: pgmap v349: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:04 smithi081 ceph-mon[90133]: pgmap v349: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:04 smithi110 ceph-mon[97633]: pgmap v349: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:04.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.704+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfc805fbb0 con 0x7fcfb0099090 2022-09-10T04:28:04.711 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.710+0000 7fcfc57fa700 1 -- 172.21.15.81:0/2597855140 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fcfc805fbb0 con 0x7fcfb0099090 2022-09-10T04:28:04.712 INFO:teuthology.orchestra.run.smithi081.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2022-09-10T04:28:04.712 INFO:teuthology.orchestra.run.smithi081.stdout:mds.all 12/12 27s ago 2m smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi110=c;smithi110=f;smithi110=i;smithi110=l;count:12 2022-09-10T04:28:04.712 INFO:teuthology.orchestra.run.smithi081.stdout:mgr 3/3 27s ago 7m smithi081=x;smithi104=y;smithi110=z;count:3 2022-09-10T04:28:04.712 INFO:teuthology.orchestra.run.smithi081.stdout:mon 3/3 27s ago 8m smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi110:172.21.15.110=c;count:3 2022-09-10T04:28:04.713 INFO:teuthology.orchestra.run.smithi081.stdout:osd 12 27s ago - 2022-09-10T04:28:04.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfb0099090 msgr2=0x7fcfb009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:04.714 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfb0099090 0x7fcfb009b540 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800f500 tx=0x7fcfb8005f90 comp rx=0 tx=0).stop 2022-09-10T04:28:04.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 msgr2=0x7fcfc8195fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:04.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 0x7fcfc8195fa0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc00e740 tx=0x7fcfbc00ea50 comp rx=0 tx=0).stop 2022-09-10T04:28:04.715 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 shutdown_connections 2022-09-10T04:28:04.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfb0099090 0x7fcfb009b540 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcfc80fd930 0x7fcfc819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcfc80fc630 0x7fcfc8195a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 --2- 172.21.15.81:0/2597855140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcfc80fcf80 0x7fcfc8195fa0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:04.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 >> 172.21.15.81:0/2597855140 conn(0x7fcfc80f82a0 msgr2=0x7fcfc80f9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:04.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 shutdown_connections 2022-09-10T04:28:04.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:04.712+0000 7fcfcd657700 1 -- 172.21.15.81:0/2597855140 wait complete. 2022-09-10T04:28:05.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:05 smithi081 ceph-mon[90133]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:05.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph orch host ls' 2022-09-10T04:28:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:05 smithi104 ceph-mon[97044]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:05 smithi110 ceph-mon[97633]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:06.360 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:28:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:06 smithi104 ceph-mon[97044]: pgmap v350: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:06.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:06 smithi081 ceph-mon[90133]: pgmap v350: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:06 smithi110 ceph-mon[97633]: pgmap v350: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:07.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.917+0000 7f57c881b700 1 -- 172.21.15.81:0/888816585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4067310 msgr2=0x7f57c4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:07.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.917+0000 7f57c881b700 1 --2- 172.21.15.81:0/888816585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4067310 0x7f57c4100ac0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009810 tx=0x7f57b4009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:07.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.917+0000 7f57c881b700 1 -- 172.21.15.81:0/888816585 shutdown_connections 2022-09-10T04:28:07.922 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.918+0000 7f57c881b700 1 --2- 172.21.15.81:0/888816585 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57c4103a50 0x7f57c4105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:07.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.918+0000 7f57c881b700 1 --2- 172.21.15.81:0/888816585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4067310 0x7f57c4100ac0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:07.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.918+0000 7f57c881b700 1 --2- 172.21.15.81:0/888816585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4101000 0x7f57c41033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:07.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.918+0000 7f57c881b700 1 -- 172.21.15.81:0/888816585 >> 172.21.15.81:0/888816585 conn(0x7f57c40f8430 msgr2=0x7f57c40fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:07.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.918+0000 7f57c881b700 1 -- 172.21.15.81:0/888816585 shutdown_connections 2022-09-10T04:28:07.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.918+0000 7f57c881b700 1 -- 172.21.15.81:0/888816585 wait complete. 2022-09-10T04:28:07.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.919+0000 7f57c881b700 1 Processor -- start 2022-09-10T04:28:07.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.919+0000 7f57c881b700 1 -- start start 2022-09-10T04:28:07.930 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c881b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4067310 0x7f57c4195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:07.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c881b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 0x7f57c41961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:07.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c881b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57c4103a50 0x7f57c419d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:07.931 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c881b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419d820 con 0x7f57c4101000 2022-09-10T04:28:07.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c881b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419d990 con 0x7f57c4103a50 2022-09-10T04:28:07.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c881b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c419db50 con 0x7f57c4067310 2022-09-10T04:28:07.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c1d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 0x7f57c41961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:07.932 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c259c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4067310 0x7f57c4195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:07.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c1d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 0x7f57c41961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:55194/0 (socket says 172.21.15.81:55194) 2022-09-10T04:28:07.933 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c2d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57c4103a50 0x7f57c419d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:07.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c1d9b700 1 -- 172.21.15.81:0/3371556837 learned_addr learned my addr 172.21.15.81:0/3371556837 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:07.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c259c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4067310 0x7f57c4195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:41372/0 (socket says 172.21.15.81:41372) 2022-09-10T04:28:07.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.920+0000 7f57c1d9b700 1 -- 172.21.15.81:0/3371556837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4067310 msgr2=0x7f57c4195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:07.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c1d9b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4067310 0x7f57c4195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:07.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c1d9b700 1 -- 172.21.15.81:0/3371556837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57c4103a50 msgr2=0x7f57c419d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:07.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c1d9b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57c4103a50 0x7f57c419d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:07.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c1d9b700 1 -- 172.21.15.81:0/3371556837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b4009470 con 0x7f57c4101000 2022-09-10T04:28:07.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c1d9b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 0x7f57c41961b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f57b800e740 tx=0x7f57b800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:07.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8005220 con 0x7f57c4101000 2022-09-10T04:28:07.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c419de30 con 0x7f57c4101000 2022-09-10T04:28:07.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f57b8005380 con 0x7f57c4101000 2022-09-10T04:28:07.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8004b70 con 0x7f57c4101000 2022-09-10T04:28:07.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.921+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c419e300 con 0x7f57c4101000 2022-09-10T04:28:07.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.922+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c4109620 con 0x7f57c4101000 2022-09-10T04:28:07.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.924+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f57b801e020 con 0x7f57c4101000 2022-09-10T04:28:07.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.926+0000 7f57b37fe700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f57ac099090 0x7f57ac09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:07.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.926+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9016+0+0 (secure 0 0 0) 0x7f57b8014070 con 0x7f57c4101000 2022-09-10T04:28:07.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.926+0000 7f57c259c700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f57ac099090 0x7f57ac09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:07.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.927+0000 7f57c259c700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f57ac099090 0x7f57ac09b540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f57c4196b40 tx=0x7f57b4009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:07.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:07.930+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f57b808eb20 con 0x7f57c4101000 2022-09-10T04:28:08.164 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.161+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f57c405fb70 con 0x7f57ac099090 2022-09-10T04:28:08.165 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.163+0000 7f57b37fe700 1 -- 172.21.15.81:0/3371556837 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f57c405fb70 con 0x7f57ac099090 2022-09-10T04:28:08.165 INFO:teuthology.orchestra.run.smithi081.stdout:HOST ADDR LABELS STATUS 2022-09-10T04:28:08.165 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 172.21.15.81 2022-09-10T04:28:08.166 INFO:teuthology.orchestra.run.smithi081.stdout:smithi104 172.21.15.104 2022-09-10T04:28:08.166 INFO:teuthology.orchestra.run.smithi081.stdout:smithi110 172.21.15.110 2022-09-10T04:28:08.166 INFO:teuthology.orchestra.run.smithi081.stdout:3 hosts in cluster 2022-09-10T04:28:08.167 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f57ac099090 msgr2=0x7f57ac09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:08.167 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f57ac099090 0x7f57ac09b540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f57c4196b40 tx=0x7f57b4009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:08.168 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 msgr2=0x7f57c41961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:08.168 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 0x7f57c41961b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f57b800e740 tx=0x7f57b800ea50 comp rx=0 tx=0).stop 2022-09-10T04:28:08.168 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 shutdown_connections 2022-09-10T04:28:08.168 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57c4103a50 0x7f57c419d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f57ac099090 0x7f57ac09b540 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57c4067310 0x7f57c4195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 --2- 172.21.15.81:0/3371556837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57c4101000 0x7f57c41961b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 >> 172.21.15.81:0/3371556837 conn(0x7f57c40f8430 msgr2=0x7f57c40fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:08.169 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 shutdown_connections 2022-09-10T04:28:08.170 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:08.165+0000 7f57c881b700 1 -- 172.21.15.81:0/3371556837 wait complete. 2022-09-10T04:28:08.529 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:08 smithi081 ceph-mon[90133]: pgmap v351: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:08.627 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph orch device ls' 2022-09-10T04:28:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:08 smithi104 ceph-mon[97044]: pgmap v351: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:08 smithi110 ceph-mon[97633]: pgmap v351: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:09.503 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:28:09.791 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:09 smithi081 ceph-mon[90133]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:09.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:09.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:09 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:09.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:09 smithi104 ceph-mon[97044]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:09 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:09 smithi110 ceph-mon[97633]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:09 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:10.719 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:10 smithi081 ceph-mon[90133]: pgmap v352: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:10.719 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2022-09-10T04:28:10.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:10 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]: dispatch 2022-09-10T04:28:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:10 smithi104 ceph-mon[97044]: pgmap v352: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2022-09-10T04:28:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:10 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]: dispatch 2022-09-10T04:28:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:10 smithi110 ceph-mon[97633]: pgmap v352: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T04:28:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2022-09-10T04:28:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:10 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]: dispatch 2022-09-10T04:28:11.018 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 -- 172.21.15.81:0/3238579402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c105b90 msgr2=0x7fa12c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 --2- 172.21.15.81:0/3238579402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c105b90 0x7fa12c107fd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa124009bb0 tx=0x7fa124009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 -- 172.21.15.81:0/3238579402 shutdown_connections 2022-09-10T04:28:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 --2- 172.21.15.81:0/3238579402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c105b90 0x7fa12c107fd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.019 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 --2- 172.21.15.81:0/3238579402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c072c90 0x7fa12c071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 --2- 172.21.15.81:0/3238579402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.016+0000 7fa13129a700 1 -- 172.21.15.81:0/3238579402 >> 172.21.15.81:0/3238579402 conn(0x7fa12c0f93a0 msgr2=0x7fa12c0fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.017+0000 7fa13129a700 1 -- 172.21.15.81:0/3238579402 shutdown_connections 2022-09-10T04:28:11.020 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.017+0000 7fa13129a700 1 -- 172.21.15.81:0/3238579402 wait complete. 2022-09-10T04:28:11.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.018+0000 7fa13129a700 1 Processor -- start 2022-09-10T04:28:11.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 -- start start 2022-09-10T04:28:11.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:11.024 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c072c90 0x7fa12c195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:11.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c105b90 0x7fa12c19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:11.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c19d5f0 con 0x7fa12c072c90 2022-09-10T04:28:11.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c19d760 con 0x7fa12c105b90 2022-09-10T04:28:11.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa13129a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c19d920 con 0x7fa12c071740 2022-09-10T04:28:11.027 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa12bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:11.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa12bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c195a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:47160/0 (socket says 172.21.15.81:47160) 2022-09-10T04:28:11.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.019+0000 7fa12bfff700 1 -- 172.21.15.81:0/886315652 learned_addr learned my addr 172.21.15.81:0/886315652 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:11.030 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12b7fe700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c072c90 0x7fa12c195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:11.030 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa130a99700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c105b90 0x7fa12c19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:11.031 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12bfff700 1 -- 172.21.15.81:0/886315652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c105b90 msgr2=0x7fa12c19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:11.031 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12bfff700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c105b90 0x7fa12c19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12bfff700 1 -- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c072c90 msgr2=0x7fa12c195fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:11.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12bfff700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c072c90 0x7fa12c195fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12bfff700 1 -- 172.21.15.81:0/886315652 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1200093b0 con 0x7fa12c071740 2022-09-10T04:28:11.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12b7fe700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c072c90 0x7fa12c195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:28:11.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa12bfff700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c195a70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00e740 tx=0x7fa11c00ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:11.034 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.020+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c005220 con 0x7fa12c071740 2022-09-10T04:28:11.034 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.021+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa11c005380 con 0x7fa12c071740 2022-09-10T04:28:11.035 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.021+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c004c10 con 0x7fa12c071740 2022-09-10T04:28:11.035 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.021+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1240097e0 con 0x7fa12c071740 2022-09-10T04:28:11.036 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.021+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12c19df30 con 0x7fa12c071740 2022-09-10T04:28:11.036 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.022+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12c10b610 con 0x7fa12c071740 2022-09-10T04:28:11.039 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.024+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fa11c01e020 con 0x7fa12c071740 2022-09-10T04:28:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.025+0000 7fa1297fa700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa114099010 0x7fa11409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.025+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fa11c014070 con 0x7fa12c071740 2022-09-10T04:28:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.025+0000 7fa12b7fe700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa114099010 0x7fa11409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:11.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.026+0000 7fa12b7fe700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa114099010 0x7fa11409b4c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa120005fd0 tx=0x7fa120005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:11.041 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.027+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa11c08f3e0 con 0x7fa12c071740 2022-09-10T04:28:11.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.255+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa12c05fb90 con 0x7fa114099010 2022-09-10T04:28:11.258 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.256+0000 7fa1297fa700 1 -- 172.21.15.81:0/886315652 <== mgr.14150 v2:172.21.15.81:6800/532852594 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7fa12c05fb90 con 0x7fa114099010 2022-09-10T04:28:11.259 INFO:teuthology.orchestra.run.smithi081.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2022-09-10T04:28:11.259 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400N7400BGN 400G 3m ago LVM detected, locked 2022-09-10T04:28:11.259 INFO:teuthology.orchestra.run.smithi081.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 400G 2m ago LVM detected, locked 2022-09-10T04:28:11.260 INFO:teuthology.orchestra.run.smithi081.stdout:smithi110 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002T480DGN 480G 2m ago LVM detected, locked 2022-09-10T04:28:11.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa114099010 msgr2=0x7fa11409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:11.262 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa114099010 0x7fa11409b4c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa120005fd0 tx=0x7fa120005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.262 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 msgr2=0x7fa12c195a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:11.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c195a70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00e740 tx=0x7fa11c00ea50 comp rx=0 tx=0).stop 2022-09-10T04:28:11.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 shutdown_connections 2022-09-10T04:28:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa12c105b90 0x7fa12c19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa114099010 0x7fa11409b4c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa12c071740 0x7fa12c195a70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 --2- 172.21.15.81:0/886315652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa12c072c90 0x7fa12c195fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:11.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 >> 172.21.15.81:0/886315652 conn(0x7fa12c0f93a0 msgr2=0x7fa12c0fb780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:11.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 shutdown_connections 2022-09-10T04:28:11.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:11.259+0000 7fa13129a700 1 -- 172.21.15.81:0/886315652 wait complete. 2022-09-10T04:28:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:11 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2022-09-10T04:28:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:11 smithi081 ceph-mon[90133]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]': finished 2022-09-10T04:28:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:11 smithi081 ceph-mon[90133]: osdmap e93: 12 total, 12 up, 12 in 2022-09-10T04:28:11.775 INFO:teuthology.run_tasks:Running task cephadm.shell... 2022-09-10T04:28:11.787 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:28:11.787 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph fs dump' 2022-09-10T04:28:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:11 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2022-09-10T04:28:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:11 smithi104 ceph-mon[97044]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]': finished 2022-09-10T04:28:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:11 smithi104 ceph-mon[97044]: osdmap e93: 12 total, 12 up, 12 in 2022-09-10T04:28:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:11 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2022-09-10T04:28:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:11 smithi110 ceph-mon[97633]: from='mgr.14150 172.21.15.81:0/4048390995' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]': finished 2022-09-10T04:28:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:11 smithi110 ceph-mon[97633]: osdmap e93: 12 total, 12 up, 12 in 2022-09-10T04:28:12.663 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:28:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:12 smithi104 ceph-mon[97044]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:12 smithi104 ceph-mon[97044]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:28:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:12 smithi104 ceph-mon[97044]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-09-10T04:28:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:12 smithi104 ceph-mon[97044]: osdmap e94: 12 total, 12 up, 12 in 2022-09-10T04:28:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:12 smithi081 ceph-mon[90133]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:12 smithi081 ceph-mon[90133]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:28:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:12 smithi081 ceph-mon[90133]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-09-10T04:28:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:12 smithi081 ceph-mon[90133]: osdmap e94: 12 total, 12 up, 12 in 2022-09-10T04:28:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:12 smithi110 ceph-mon[97633]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:12 smithi110 ceph-mon[97633]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:28:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:12 smithi110 ceph-mon[97633]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2022-09-10T04:28:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:12 smithi110 ceph-mon[97633]: osdmap e94: 12 total, 12 up, 12 in 2022-09-10T04:28:14.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 -- 172.21.15.81:0/4127580083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 msgr2=0x7ff7840ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:14.208 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 --2- 172.21.15.81:0/4127580083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff7840ff320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff778009810 tx=0x7ff778009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:14.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 -- 172.21.15.81:0/4127580083 shutdown_connections 2022-09-10T04:28:14.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 --2- 172.21.15.81:0/4127580083 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7840ff860 0x7ff784106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 --2- 172.21.15.81:0/4127580083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7840fdc50 0x7ff7840fe060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.209 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 --2- 172.21.15.81:0/4127580083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff7840ff320 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 -- 172.21.15.81:0/4127580083 >> 172.21.15.81:0/4127580083 conn(0x7ff7840f9320 msgr2=0x7ff7840fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:14.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 -- 172.21.15.81:0/4127580083 shutdown_connections 2022-09-10T04:28:14.210 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.206+0000 7ff783fff700 1 -- 172.21.15.81:0/4127580083 wait complete. 2022-09-10T04:28:14.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.207+0000 7ff783fff700 1 Processor -- start 2022-09-10T04:28:14.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.207+0000 7ff783fff700 1 -- start start 2022-09-10T04:28:14.213 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff783fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7840fdc50 0x7ff784195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:14.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff783fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff784195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:14.214 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff783fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7840ff860 0x7ff78419d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff783fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78419d600 con 0x7ff7840feed0 2022-09-10T04:28:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff783fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78419d770 con 0x7ff7840fdc50 2022-09-10T04:28:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff783fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78419d930 con 0x7ff7840ff860 2022-09-10T04:28:14.215 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff7827fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff784195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff7827fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff784195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34986/0 (socket says 172.21.15.81:34986) 2022-09-10T04:28:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff7827fc700 1 -- 172.21.15.81:0/502942870 learned_addr learned my addr 172.21.15.81:0/502942870 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff782ffd700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7840fdc50 0x7ff784195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff7837fe700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7840ff860 0x7ff78419d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:14.216 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff7827fc700 1 -- 172.21.15.81:0/502942870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7840ff860 msgr2=0x7ff78419d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:14.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.208+0000 7ff7827fc700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7840ff860 0x7ff78419d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff7827fc700 1 -- 172.21.15.81:0/502942870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7840fdc50 msgr2=0x7ff784195a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:14.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff7827fc700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7840fdc50 0x7ff784195a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.217 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff7827fc700 1 -- 172.21.15.81:0/502942870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff778009470 con 0x7ff7840feed0 2022-09-10T04:28:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff7827fc700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff784195fc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff77800f690 tx=0x7ff77800f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77801d070 con 0x7ff7840feed0 2022-09-10T04:28:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff77800f8e0 con 0x7ff7840feed0 2022-09-10T04:28:14.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7780047f0 con 0x7ff7840feed0 2022-09-10T04:28:14.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78419dbb0 con 0x7ff7840feed0 2022-09-10T04:28:14.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.209+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78419e010 con 0x7ff7840feed0 2022-09-10T04:28:14.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.210+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff784109400 con 0x7ff7840feed0 2022-09-10T04:28:14.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.212+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff778028030 con 0x7ff7840feed0 2022-09-10T04:28:14.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.214+0000 7ff767fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff76c099090 0x7ff76c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:14.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.214+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7ff7780c3370 con 0x7ff7840feed0 2022-09-10T04:28:14.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.214+0000 7ff782ffd700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff76c099090 0x7ff76c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:14.221 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.215+0000 7ff782ffd700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff76c099090 0x7ff76c09b540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff7840fed30 tx=0x7ff774005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:14.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.218+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff77808f6b0 con 0x7ff7840feed0 2022-09-10T04:28:14.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.477+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff78419ebd0 con 0x7ff7840feed0 2022-09-10T04:28:14.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.478+0000 7ff767fff700 1 -- 172.21.15.81:0/502942870 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+2894 (secure 0 0 0) 0x7ff778092cd0 con 0x7ff7840feed0 2022-09-10T04:28:14.480 INFO:teuthology.orchestra.run.smithi081.stdout:e28 2022-09-10T04:28:14.480 INFO:teuthology.orchestra.run.smithi081.stdout:enable_multiple, ever_enabled_multiple: 1,1 2022-09-10T04:28:14.480 INFO:teuthology.orchestra.run.smithi081.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-10T04:28:14.481 INFO:teuthology.orchestra.run.smithi081.stdout:legacy client fscid: 1 2022-09-10T04:28:14.481 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:14.481 INFO:teuthology.orchestra.run.smithi081.stdout:Filesystem 'cephfs' (1) 2022-09-10T04:28:14.481 INFO:teuthology.orchestra.run.smithi081.stdout:fs_name cephfs 2022-09-10T04:28:14.482 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 28 2022-09-10T04:28:14.482 INFO:teuthology.orchestra.run.smithi081.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2022-09-10T04:28:14.482 INFO:teuthology.orchestra.run.smithi081.stdout:created 2022-09-10T04:26:32.097174+0000 2022-09-10T04:28:14.482 INFO:teuthology.orchestra.run.smithi081.stdout:modified 2022-09-10T04:26:59.184557+0000 2022-09-10T04:28:14.483 INFO:teuthology.orchestra.run.smithi081.stdout:tableserver 0 2022-09-10T04:28:14.483 INFO:teuthology.orchestra.run.smithi081.stdout:root 0 2022-09-10T04:28:14.484 INFO:teuthology.orchestra.run.smithi081.stdout:session_timeout 300 2022-09-10T04:28:14.484 INFO:teuthology.orchestra.run.smithi081.stdout:session_autoclose 300 2022-09-10T04:28:14.484 INFO:teuthology.orchestra.run.smithi081.stdout:max_file_size 1099511627776 2022-09-10T04:28:14.484 INFO:teuthology.orchestra.run.smithi081.stdout:required_client_features {} 2022-09-10T04:28:14.485 INFO:teuthology.orchestra.run.smithi081.stdout:last_failure 0 2022-09-10T04:28:14.485 INFO:teuthology.orchestra.run.smithi081.stdout:last_failure_osd_epoch 0 2022-09-10T04:28:14.485 INFO:teuthology.orchestra.run.smithi081.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2022-09-10T04:28:14.485 INFO:teuthology.orchestra.run.smithi081.stdout:max_mds 5 2022-09-10T04:28:14.485 INFO:teuthology.orchestra.run.smithi081.stdout:in 0,1,2,3,4 2022-09-10T04:28:14.486 INFO:teuthology.orchestra.run.smithi081.stdout:up {0=24577,1=24572,2=24547,3=24532,4=24466} 2022-09-10T04:28:14.486 INFO:teuthology.orchestra.run.smithi081.stdout:failed 2022-09-10T04:28:14.486 INFO:teuthology.orchestra.run.smithi081.stdout:damaged 2022-09-10T04:28:14.487 INFO:teuthology.orchestra.run.smithi081.stdout:stopped 2022-09-10T04:28:14.487 INFO:teuthology.orchestra.run.smithi081.stdout:data_pools [3,4] 2022-09-10T04:28:14.487 INFO:teuthology.orchestra.run.smithi081.stdout:metadata_pool 2 2022-09-10T04:28:14.488 INFO:teuthology.orchestra.run.smithi081.stdout:inline_data disabled 2022-09-10T04:28:14.488 INFO:teuthology.orchestra.run.smithi081.stdout:balancer 2022-09-10T04:28:14.488 INFO:teuthology.orchestra.run.smithi081.stdout:standby_count_wanted 0 2022-09-10T04:28:14.488 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a{0:24577} state up:active seq 4 addr [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.h{0:24530} state up:standby-replay seq 1 addr [v2:172.21.15.104:6836/2202378802,v1:172.21.15.104:6837/2202378802] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b{1:24572} state up:active seq 9 addr [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.f{1:24520} state up:standby-replay seq 1 addr [v2:172.21.15.110:6834/4033961168,v1:172.21.15.110:6835/4033961168] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.i{2:24547} state up:active seq 12 addr [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.e{2:24497} state up:standby-replay seq 1 addr [v2:172.21.15.104:6834/2709283075,v1:172.21.15.104:6835/2709283075] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.490 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c{3:24532} state up:active seq 13 addr [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.490 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.j{3:24473} state up:standby-replay seq 1 addr [v2:172.21.15.81:6834/1412455487,v1:172.21.15.81:6835/1412455487] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.490 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.l{4:24466} state up:active seq 17 addr [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.490 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.k{4:24461} state up:standby-replay seq 1 addr [v2:172.21.15.104:6832/777476125,v1:172.21.15.104:6833/777476125] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.491 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:14.491 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:14.491 INFO:teuthology.orchestra.run.smithi081.stdout:Standby daemons: 2022-09-10T04:28:14.491 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:14.492 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d{-1:14415} state up:standby seq 1 addr [v2:172.21.15.81:6836/316573610,v1:172.21.15.81:6837/316573610] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.492 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.g{-1:14460} state up:standby seq 1 addr [v2:172.21.15.81:6838/1769781597,v1:172.21.15.81:6839/1769781597] compat {c=[1],r=[1],i=[7ff]}] 2022-09-10T04:28:14.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff76c099090 msgr2=0x7ff76c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:14.493 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff76c099090 0x7ff76c09b540 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff7840fed30 tx=0x7ff774005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 msgr2=0x7ff784195fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:14.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff784195fc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff77800f690 tx=0x7ff77800f6c0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.494 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 shutdown_connections 2022-09-10T04:28:14.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff76c099090 0x7ff76c09b540 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff7840ff860 0x7ff78419d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7840fdc50 0x7ff784195a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.495 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 --2- 172.21.15.81:0/502942870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7840feed0 0x7ff784195fc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 >> 172.21.15.81:0/502942870 conn(0x7ff7840f9320 msgr2=0x7ff7840fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.480+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 shutdown_connections 2022-09-10T04:28:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:14.481+0000 7ff783fff700 1 -- 172.21.15.81:0/502942870 wait complete. 2022-09-10T04:28:14.496 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 28 2022-09-10T04:28:14.511 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:14 smithi081 ceph-mon[90133]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:28:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:14 smithi104 ceph-mon[97044]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:28:14.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:14 smithi110 ceph-mon[97633]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:28:14.937 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- bash -c 'ceph osd dump' 2022-09-10T04:28:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:15 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/502942870' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-09-10T04:28:15.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:15 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/502942870' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-09-10T04:28:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:15 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/502942870' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2022-09-10T04:28:15.809 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:28:16.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:16 smithi081 ceph-mon[90133]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T04:28:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:16 smithi104 ceph-mon[97044]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T04:28:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:16 smithi110 ceph-mon[97633]: pgmap v357: 129 pgs: 1 peering, 128 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T04:28:17.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 -- 172.21.15.81:0/3426648550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810103a30 msgr2=0x7fa810105eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:17.233 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 --2- 172.21.15.81:0/3426648550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810103a30 0x7fa810105eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa808009bb0 tx=0x7fa808009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 -- 172.21.15.81:0/3426648550 shutdown_connections 2022-09-10T04:28:17.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 --2- 172.21.15.81:0/3426648550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810103a30 0x7fa810105eb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 --2- 172.21.15.81:0/3426648550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8100673d0 0x7fa810100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.234 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 --2- 172.21.15.81:0/3426648550 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810100f40 0x7fa8101033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 -- 172.21.15.81:0/3426648550 >> 172.21.15.81:0/3426648550 conn(0x7fa8100f8430 msgr2=0x7fa8100fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:17.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 -- 172.21.15.81:0/3426648550 shutdown_connections 2022-09-10T04:28:17.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.231+0000 7fa818090700 1 -- 172.21.15.81:0/3426648550 wait complete. 2022-09-10T04:28:17.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.232+0000 7fa818090700 1 Processor -- start 2022-09-10T04:28:17.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.232+0000 7fa818090700 1 -- start start 2022-09-10T04:28:17.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa818090700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8100673d0 0x7fa810195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:17.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa818090700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810100f40 0x7fa8101961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:17.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa818090700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 0x7fa81019d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:17.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa818090700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81019d840 con 0x7fa8100673d0 2022-09-10T04:28:17.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa818090700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81019d9b0 con 0x7fa810103a30 2022-09-10T04:28:17.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa818090700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81019db70 con 0x7fa810100f40 2022-09-10T04:28:17.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa815e2c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8100673d0 0x7fa810195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:17.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa81662d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 0x7fa81019d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:17.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa81562b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810100f40 0x7fa8101961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:17.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa81662d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 0x7fa81019d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:33444/0 (socket says 172.21.15.81:33444) 2022-09-10T04:28:17.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.233+0000 7fa81662d700 1 -- 172.21.15.81:0/2492547302 learned_addr learned my addr 172.21.15.81:0/2492547302 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:17.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81662d700 1 -- 172.21.15.81:0/2492547302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810100f40 msgr2=0x7fa8101961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:17.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81662d700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810100f40 0x7fa8101961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81662d700 1 -- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8100673d0 msgr2=0x7fa810195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:17.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81662d700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8100673d0 0x7fa810195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81662d700 1 -- 172.21.15.81:0/2492547302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c0093b0 con 0x7fa810103a30 2022-09-10T04:28:17.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81562b700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810100f40 0x7fa8101961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:17.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa81662d700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 0x7fa81019d1f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa808005b50 tx=0x7fa80800b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:17.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80800f870 con 0x7fa810103a30 2022-09-10T04:28:17.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa80800ba80 con 0x7fa810103a30 2022-09-10T04:28:17.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8080097e0 con 0x7fa810103a30 2022-09-10T04:28:17.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.234+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8080176d0 con 0x7fa810103a30 2022-09-10T04:28:17.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.235+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81019e0c0 con 0x7fa810103a30 2022-09-10T04:28:17.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.236+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa810109640 con 0x7fa810103a30 2022-09-10T04:28:17.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.239+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fa808020070 con 0x7fa810103a30 2022-09-10T04:28:17.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.240+0000 7fa806ffd700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7fc099090 0x7fa7fc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:17.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.240+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fa8080c4080 con 0x7fa810103a30 2022-09-10T04:28:17.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.240+0000 7fa815e2c700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7fc099090 0x7fa7fc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:17.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.241+0000 7fa815e2c700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7fc099090 0x7fa7fc09b540 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa810197150 tx=0x7fa80c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:17.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.241+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fa80805ac80 con 0x7fa810103a30 2022-09-10T04:28:17.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.457+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fa810105420 con 0x7fa810103a30 2022-09-10T04:28:17.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.458+0000 7fa806ffd700 1 -- 172.21.15.81:0/2492547302 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) v1 ==== 56+0+5571 (secure 0 0 0) 0x7fa8080903c0 con 0x7fa810103a30 2022-09-10T04:28:17.460 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 94 2022-09-10T04:28:17.460 INFO:teuthology.orchestra.run.smithi081.stdout:fsid 70cf895c-30bf-11ed-8431-001a4aab830c 2022-09-10T04:28:17.461 INFO:teuthology.orchestra.run.smithi081.stdout:created 2022-09-10T04:18:07.859996+0000 2022-09-10T04:28:17.461 INFO:teuthology.orchestra.run.smithi081.stdout:modified 2022-09-10T04:28:11.428965+0000 2022-09-10T04:28:17.461 INFO:teuthology.orchestra.run.smithi081.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2022-09-10T04:28:17.461 INFO:teuthology.orchestra.run.smithi081.stdout:crush_version 27 2022-09-10T04:28:17.462 INFO:teuthology.orchestra.run.smithi081.stdout:full_ratio 0.95 2022-09-10T04:28:17.462 INFO:teuthology.orchestra.run.smithi081.stdout:backfillfull_ratio 0.9 2022-09-10T04:28:17.462 INFO:teuthology.orchestra.run.smithi081.stdout:nearfull_ratio 0.85 2022-09-10T04:28:17.463 INFO:teuthology.orchestra.run.smithi081.stdout:require_min_compat_client luminous 2022-09-10T04:28:17.463 INFO:teuthology.orchestra.run.smithi081.stdout:min_compat_client luminous 2022-09-10T04:28:17.464 INFO:teuthology.orchestra.run.smithi081.stdout:require_osd_release quincy 2022-09-10T04:28:17.464 INFO:teuthology.orchestra.run.smithi081.stdout:stretch_mode_enabled false 2022-09-10T04:28:17.465 INFO:teuthology.orchestra.run.smithi081.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2022-09-10T04:28:17.465 INFO:teuthology.orchestra.run.smithi081.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs 2022-09-10T04:28:17.466 INFO:teuthology.orchestra.run.smithi081.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs 2022-09-10T04:28:17.466 INFO:teuthology.orchestra.run.smithi081.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2022-09-10T04:28:17.467 INFO:teuthology.orchestra.run.smithi081.stdout:max_osd 12 2022-09-10T04:28:17.467 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6802/860456526,v1:172.21.15.81:6803/860456526] [v2:172.21.15.81:6804/860456526,v1:172.21.15.81:6805/860456526] exists,up 06d36b08-d01a-4f7b-87d2-3dd153d2c300 2022-09-10T04:28:17.468 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 up in weight 1 up_from 13 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6800/757653704,v1:172.21.15.104:6801/757653704] [v2:172.21.15.104:6802/757653704,v1:172.21.15.104:6803/757653704] exists,up 34797803-b77d-4ce2-a459-df89cb7ff968 2022-09-10T04:28:17.468 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6800/829575986,v1:172.21.15.110:6801/829575986] [v2:172.21.15.110:6802/829575986,v1:172.21.15.110:6803/829575986] exists,up 9e47fb11-0460-4c91-8284-c8854bcfbfba 2022-09-10T04:28:17.468 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 up in weight 1 up_from 26 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6810/1343686033,v1:172.21.15.81:6811/1343686033] [v2:172.21.15.81:6812/1343686033,v1:172.21.15.81:6813/1343686033] exists,up ada925c5-2eb8-48cf-9a82-bc4c2ff500c2 2022-09-10T04:28:17.469 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 up in weight 1 up_from 34 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6808/2302505309,v1:172.21.15.104:6809/2302505309] [v2:172.21.15.104:6810/2302505309,v1:172.21.15.104:6811/2302505309] exists,up 73482f9e-093f-408b-96bf-34584c281ea7 2022-09-10T04:28:17.469 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 up in weight 1 up_from 39 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6808/3775240590,v1:172.21.15.110:6809/3775240590] [v2:172.21.15.110:6810/3775240590,v1:172.21.15.110:6811/3775240590] exists,up 9dd94ce3-5bdf-4664-97dd-73980655b4a8 2022-09-10T04:28:17.470 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6818/2282910145,v1:172.21.15.81:6819/2282910145] [v2:172.21.15.81:6820/2282910145,v1:172.21.15.81:6821/2282910145] exists,up 36d8ca99-65c5-4528-8ee8-de7cd6faf9fa 2022-09-10T04:28:17.470 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 up in weight 1 up_from 52 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6816/4207080292,v1:172.21.15.104:6817/4207080292] [v2:172.21.15.104:6818/4207080292,v1:172.21.15.104:6819/4207080292] exists,up c0c52ea1-4f7e-4760-b9c5-9b369d476ebe 2022-09-10T04:28:17.470 INFO:teuthology.orchestra.run.smithi081.stdout:osd.8 up in weight 1 up_from 60 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6816/462969903,v1:172.21.15.110:6817/462969903] [v2:172.21.15.110:6818/462969903,v1:172.21.15.110:6819/462969903] exists,up c08ff30d-679b-4637-b363-c5fcb59fe112 2022-09-10T04:28:17.471 INFO:teuthology.orchestra.run.smithi081.stdout:osd.9 up in weight 1 up_from 68 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6826/3176150026,v1:172.21.15.81:6827/3176150026] [v2:172.21.15.81:6828/3176150026,v1:172.21.15.81:6829/3176150026] exists,up 9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2 2022-09-10T04:28:17.471 INFO:teuthology.orchestra.run.smithi081.stdout:osd.10 up in weight 1 up_from 73 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6824/675537330,v1:172.21.15.104:6825/675537330] [v2:172.21.15.104:6826/675537330,v1:172.21.15.104:6827/675537330] exists,up 60318b87-7ef8-423c-8f41-1554cd1b4e9a 2022-09-10T04:28:17.472 INFO:teuthology.orchestra.run.smithi081.stdout:osd.11 up in weight 1 up_from 81 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.110:6824/595897724,v1:172.21.15.110:6825/595897724] [v2:172.21.15.110:6826/595897724,v1:172.21.15.110:6827/595897724] exists,up da2d73f1-bba2-4c62-9463-ca7aa22701d8 2022-09-10T04:28:17.472 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 2.6 [8,11] 2022-09-10T04:28:17.472 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 2.a [2,11] 2022-09-10T04:28:17.473 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.2 [3,9] 2022-09-10T04:28:17.473 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.14 [4,6] 2022-09-10T04:28:17.473 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.17 [4,7] 2022-09-10T04:28:17.473 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.25 [5,9] 2022-09-10T04:28:17.474 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.38 [3,9] 2022-09-10T04:28:17.474 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.3a [4,8] 2022-09-10T04:28:17.474 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.2 [9,1] 2022-09-10T04:28:17.474 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.13 [9,8] 2022-09-10T04:28:17.475 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.14 [9,8] 2022-09-10T04:28:17.475 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.1d [5,10] 2022-09-10T04:28:17.475 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/2927238869 expires 2022-09-11T04:19:09.198520+0000 2022-09-10T04:28:17.475 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/3943821768 expires 2022-09-11T04:19:09.198520+0000 2022-09-10T04:28:17.475 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/2359268015 expires 2022-09-11T04:19:09.198520+0000 2022-09-10T04:28:17.476 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/2567224701 expires 2022-09-11T04:18:41.737767+0000 2022-09-10T04:28:17.476 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6801/2567224701 expires 2022-09-11T04:18:41.737767+0000 2022-09-10T04:28:17.476 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/2774479929 expires 2022-09-11T04:18:41.737767+0000 2022-09-10T04:28:17.477 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/1348172231 expires 2022-09-11T04:18:41.737767+0000 2022-09-10T04:28:17.477 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/596170975 expires 2022-09-11T04:18:41.737767+0000 2022-09-10T04:28:17.477 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6801/3224857868 expires 2022-09-11T04:19:09.198520+0000 2022-09-10T04:28:17.477 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/3224857868 expires 2022-09-11T04:19:09.198520+0000 2022-09-10T04:28:17.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7fc099090 msgr2=0x7fa7fc09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:17.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7fc099090 0x7fa7fc09b540 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa810197150 tx=0x7fa80c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.479 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 msgr2=0x7fa81019d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:17.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 0x7fa81019d1f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa808005b50 tx=0x7fa80800b890 comp rx=0 tx=0).stop 2022-09-10T04:28:17.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 shutdown_connections 2022-09-10T04:28:17.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa810103a30 0x7fa81019d1f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8100673d0 0x7fa810195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.480 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa7fc099090 0x7fa7fc09b540 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 --2- 172.21.15.81:0/2492547302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa810100f40 0x7fa8101961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:17.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 >> 172.21.15.81:0/2492547302 conn(0x7fa8100f8430 msgr2=0x7fa8100ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:17.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 shutdown_connections 2022-09-10T04:28:17.481 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:17.461+0000 7fa818090700 1 -- 172.21.15.81:0/2492547302 wait complete. 2022-09-10T04:28:17.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:17 smithi104 ceph-mon[97044]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-09-10T04:28:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:17 smithi104 ceph-mon[97044]: Cluster is now healthy 2022-09-10T04:28:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:17 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2492547302' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-09-10T04:28:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:17 smithi081 ceph-mon[90133]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-09-10T04:28:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:17 smithi081 ceph-mon[90133]: Cluster is now healthy 2022-09-10T04:28:17.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:17 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2492547302' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-09-10T04:28:17.876 INFO:teuthology.run_tasks:Running task fs.ready... 2022-09-10T04:28:17.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:28:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:17 smithi110 ceph-mon[97633]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2022-09-10T04:28:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:17 smithi110 ceph-mon[97633]: Cluster is now healthy 2022-09-10T04:28:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:17 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2492547302' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2022-09-10T04:28:18.127 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 -- 172.21.15.81:0/3224519587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 msgr2=0x7f50fc1060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.128 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 --2- 172.21.15.81:0/3224519587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc1060b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009b00 tx=0x7f50f4009e10 comp rx=0 tx=0).stop 2022-09-10T04:28:18.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 -- 172.21.15.81:0/3224519587 shutdown_connections 2022-09-10T04:28:18.129 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 --2- 172.21.15.81:0/3224519587 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc1060b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 --2- 172.21.15.81:0/3224519587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 0x7f50fc0fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.130 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 --2- 172.21.15.81:0/3224519587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc0fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 -- 172.21.15.81:0/3224519587 >> 172.21.15.81:0/3224519587 conn(0x7f50fc0f9230 msgr2=0x7f50fc0fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:18.131 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 -- 172.21.15.81:0/3224519587 shutdown_connections 2022-09-10T04:28:18.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.119+0000 7f5100b9d700 1 -- 172.21.15.81:0/3224519587 wait complete. 2022-09-10T04:28:18.132 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 Processor -- start 2022-09-10T04:28:18.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 -- start start 2022-09-10T04:28:18.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 0x7f50fc195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.133 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc19d4b0 con 0x7f50fc0fe490 2022-09-10T04:28:18.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc19d620 con 0x7f50fc0ff510 2022-09-10T04:28:18.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f5100b9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc19d7e0 con 0x7f50fc0fdb40 2022-09-10T04:28:18.134 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f50faffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f50faffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35042/0 (socket says 172.21.15.81:35042) 2022-09-10T04:28:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f50faffd700 1 -- 172.21.15.81:0/2464606718 learned_addr learned my addr 172.21.15.81:0/2464606718 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.121+0000 7f50fb7fe700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 0x7f50fc195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.135 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50fbfff700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50faffd700 1 -- 172.21.15.81:0/2464606718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 msgr2=0x7f50fc195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50faffd700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 0x7f50fc195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.136 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50faffd700 1 -- 172.21.15.81:0/2464606718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 msgr2=0x7f50fc19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50faffd700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50faffd700 1 -- 172.21.15.81:0/2464606718 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0093b0 con 0x7f50fc0fe490 2022-09-10T04:28:18.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50fbfff700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:28:18.137 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50fb7fe700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 0x7f50fc195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:18.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50faffd700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc195e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50f000e7c0 tx=0x7f50f000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:18.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.122+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0009510 con 0x7f50fc0fe490 2022-09-10T04:28:18.138 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.123+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f4009730 con 0x7f50fc0fe490 2022-09-10T04:28:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.123+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f50f0010da0 con 0x7f50fc0fe490 2022-09-10T04:28:18.139 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.123+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f000f420 con 0x7f50fc0fe490 2022-09-10T04:28:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.123+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc19de90 con 0x7f50fc0fe490 2022-09-10T04:28:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.124+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50fc109330 con 0x7f50fc0fe490 2022-09-10T04:28:18.140 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.125+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f50f000c040 con 0x7f50fc0fe490 2022-09-10T04:28:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.127+0000 7f50f8ff9700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f50e4099010 0x7f50e409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.141 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.127+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f50f0014070 con 0x7f50fc0fe490 2022-09-10T04:28:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.127+0000 7f50fb7fe700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f50e4099010 0x7f50e409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.128+0000 7f50fb7fe700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f50e4099010 0x7f50e409b4c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50ec005fd0 tx=0x7f50ec005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:18.142 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.130+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f50f004fc80 con 0x7f50fc0fe490 2022-09-10T04:28:18.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.325+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50fc18ff40 con 0x7f50fc0fe490 2022-09-10T04:28:18.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.326+0000 7f50f8ff9700 1 -- 172.21.15.81:0/2464606718 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+23743 (secure 0 0 0) 0x7f50f0092710 con 0x7f50fc0fe490 2022-09-10T04:28:18.328 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:18.328 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":94,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:28:11.428965+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:28:18.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f50e4099010 msgr2=0x7f50e409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f50e4099010 0x7f50e409b4c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50ec005fd0 tx=0x7f50ec005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 msgr2=0x7f50fc195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc195e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50f000e7c0 tx=0x7f50f000ead0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 shutdown_connections 2022-09-10T04:28:18.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f50fc0ff510 0x7f50fc19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f50e4099010 0x7f50e409b4c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50fc0fdb40 0x7f50fc195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 --2- 172.21.15.81:0/2464606718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50fc0fe490 0x7f50fc195e70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 >> 172.21.15.81:0/2464606718 conn(0x7f50fc0f9230 msgr2=0x7f50fc104790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:18.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 shutdown_connections 2022-09-10T04:28:18.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.329+0000 7f5100b9d700 1 -- 172.21.15.81:0/2464606718 wait complete. 2022-09-10T04:28:18.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-10T04:26:30.005808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-10T04:26:31.529234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-10T04:26:35.269129+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-10T04:28:18.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:28:18.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 -- 172.21.15.81:0/3535866700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff740066bf0 msgr2=0x7ff7401008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 --2- 172.21.15.81:0/3535866700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff740066bf0 0x7ff7401008b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009810 tx=0x7ff73c009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:18.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 -- 172.21.15.81:0/3535866700 shutdown_connections 2022-09-10T04:28:18.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 --2- 172.21.15.81:0/3535866700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7401038e0 0x7ff740105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 --2- 172.21.15.81:0/3535866700 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff740066bf0 0x7ff7401008b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 --2- 172.21.15.81:0/3535866700 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 0x7ff740103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 -- 172.21.15.81:0/3535866700 >> 172.21.15.81:0/3535866700 conn(0x7ff7400f82e0 msgr2=0x7ff7400fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:18.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 -- 172.21.15.81:0/3535866700 shutdown_connections 2022-09-10T04:28:18.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.588+0000 7ff747fff700 1 -- 172.21.15.81:0/3535866700 wait complete. 2022-09-10T04:28:18.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.589+0000 7ff747fff700 1 Processor -- start 2022-09-10T04:28:18.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.589+0000 7ff747fff700 1 -- start start 2022-09-10T04:28:18.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff747fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 0x7ff740195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff747fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 0x7ff7401960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff747fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7401038e0 0x7ff74019d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff747fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74019d6e0 con 0x7ff740066bf0 2022-09-10T04:28:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff747fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74019d850 con 0x7ff7401038e0 2022-09-10T04:28:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff747fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74019da10 con 0x7ff740100df0 2022-09-10T04:28:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff745d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 0x7ff740195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7401038e0 0x7ff74019d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff745d9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 0x7ff740195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35060/0 (socket says 172.21.15.81:35060) 2022-09-10T04:28:18.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff745d9b700 1 -- 172.21.15.81:0/3331066799 learned_addr learned my addr 172.21.15.81:0/3331066799 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:18.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74559a700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 0x7ff7401960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74659c700 1 -- 172.21.15.81:0/3331066799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 msgr2=0x7ff7401960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74659c700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 0x7ff7401960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74659c700 1 -- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 msgr2=0x7ff740195b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74659c700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 0x7ff740195b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74659c700 1 -- 172.21.15.81:0/3331066799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c009470 con 0x7ff7401038e0 2022-09-10T04:28:18.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff74559a700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 0x7ff7401960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:28:18.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.590+0000 7ff745d9b700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 0x7ff740195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:18.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.591+0000 7ff74659c700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7401038e0 0x7ff74019d110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff73800eab0 tx=0x7ff73800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:18.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.591+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738004c30 con 0x7ff7401038e0 2022-09-10T04:28:18.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.591+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74019dcf0 con 0x7ff7401038e0 2022-09-10T04:28:18.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.591+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff73800ce10 con 0x7ff7401038e0 2022-09-10T04:28:18.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.591+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738004280 con 0x7ff7401038e0 2022-09-10T04:28:18.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.591+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74019e1b0 con 0x7ff7401038e0 2022-09-10T04:28:18.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.592+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7401094b0 con 0x7ff7401038e0 2022-09-10T04:28:18.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.594+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7ff73800c040 con 0x7ff7401038e0 2022-09-10T04:28:18.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.595+0000 7ff736ffd700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff72c099010 0x7ff72c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:18.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.595+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7ff738014070 con 0x7ff7401038e0 2022-09-10T04:28:18.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.595+0000 7ff745d9b700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff72c099010 0x7ff72c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:18.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.596+0000 7ff745d9b700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff72c099010 0x7ff72c09b4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00b4f0 tx=0x7ff73c009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:18.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.599+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff73808ed20 con 0x7ff7401038e0 2022-09-10T04:28:18.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.789+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7400fdc90 con 0x7ff7401038e0 2022-09-10T04:28:18.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.789+0000 7ff736ffd700 1 -- 172.21.15.81:0/3331066799 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff73801c0a0 con 0x7ff7401038e0 2022-09-10T04:28:18.791 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:28:18.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff72c099010 msgr2=0x7ff72c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff72c099010 0x7ff72c09b4c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00b4f0 tx=0x7ff73c009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7401038e0 msgr2=0x7ff74019d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:18.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7401038e0 0x7ff74019d110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff73800eab0 tx=0x7ff73800edc0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 shutdown_connections 2022-09-10T04:28:18.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff7401038e0 0x7ff74019d110 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff740066bf0 0x7ff740195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7ff72c099010 0x7ff72c09b4c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 --2- 172.21.15.81:0/3331066799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff740100df0 0x7ff7401960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:18.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 >> 172.21.15.81:0/3331066799 conn(0x7ff7400f82e0 msgr2=0x7ff7400fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:18.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 shutdown_connections 2022-09-10T04:28:18.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:18.792+0000 7ff747fff700 1 -- 172.21.15.81:0/3331066799 wait complete. 2022-09-10T04:28:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:18 smithi104 ceph-mon[97044]: pgmap v358: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:28:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:18 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2464606718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:18.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:18 smithi081 ceph-mon[90133]: pgmap v358: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:28:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:18 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2464606718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:18.821 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-10T04:28:18.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:18 smithi110 ceph-mon[97633]: pgmap v358: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2022-09-10T04:28:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:18 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2464606718' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:19.051 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 -- 172.21.15.81:0/1484354557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 msgr2=0x7f45e01008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1484354557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 0x7f45e01008b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009810 tx=0x7f45d0009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:19.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 -- 172.21.15.81:0/1484354557 shutdown_connections 2022-09-10T04:28:19.052 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1484354557 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e01038e0 0x7f45e0105d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1484354557 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 0x7f45e01008b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1484354557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.049+0000 7f45e69ff700 1 -- 172.21.15.81:0/1484354557 >> 172.21.15.81:0/1484354557 conn(0x7f45e00f82e0 msgr2=0x7f45e00fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:19.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.050+0000 7f45e69ff700 1 -- 172.21.15.81:0/1484354557 shutdown_connections 2022-09-10T04:28:19.053 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.050+0000 7f45e69ff700 1 -- 172.21.15.81:0/1484354557 wait complete. 2022-09-10T04:28:19.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 Processor -- start 2022-09-10T04:28:19.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 -- start start 2022-09-10T04:28:19.056 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 0x7f45e0195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e01960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e01038e0 0x7f45e019d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.057 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e019d6e0 con 0x7f45e0100df0 2022-09-10T04:28:19.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e019d850 con 0x7f45e0066bf0 2022-09-10T04:28:19.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.051+0000 7f45e69ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e019da10 con 0x7f45e01038e0 2022-09-10T04:28:19.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e01960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.058 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e01960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35088/0 (socket says 172.21.15.81:35088) 2022-09-10T04:28:19.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 -- 172.21.15.81:0/1107294545 learned_addr learned my addr 172.21.15.81:0/1107294545 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:19.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45dffff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 0x7f45e0195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45e4f9c700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e01038e0 0x7f45e019d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.059 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 -- 172.21.15.81:0/1107294545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e01038e0 msgr2=0x7f45e019d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e01038e0 0x7f45e019d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 -- 172.21.15.81:0/1107294545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 msgr2=0x7f45e0195b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.060 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 0x7f45e0195b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 -- 172.21.15.81:0/1107294545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0009470 con 0x7f45e0100df0 2022-09-10T04:28:19.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.052+0000 7f45df7fe700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e01960a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f45d400e740 tx=0x7f45d400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:19.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.053+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4005220 con 0x7f45e0100df0 2022-09-10T04:28:19.061 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.053+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f45d4005380 con 0x7f45e0100df0 2022-09-10T04:28:19.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.053+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e019dcf0 con 0x7f45e0100df0 2022-09-10T04:28:19.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.053+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d4004b70 con 0x7f45e0100df0 2022-09-10T04:28:19.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.053+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e019e1f0 con 0x7f45e0100df0 2022-09-10T04:28:19.062 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.054+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e01094b0 con 0x7f45e0100df0 2022-09-10T04:28:19.063 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.056+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f45d401e020 con 0x7f45e0100df0 2022-09-10T04:28:19.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.057+0000 7f45dd7fa700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f45c8099090 0x7f45c809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.057+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f45d4014070 con 0x7f45e0100df0 2022-09-10T04:28:19.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.057+0000 7f45dffff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f45c8099090 0x7f45c809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.064 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.058+0000 7f45dffff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f45c8099090 0x7f45c809b540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d0009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:19.065 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.061+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f45d408eb80 con 0x7f45e0100df0 2022-09-10T04:28:19.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.254+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f45e0105370 con 0x7f45e0100df0 2022-09-10T04:28:19.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.255+0000 7f45dd7fa700 1 -- 172.21.15.81:0/1107294545 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f45d40921a0 con 0x7f45e0100df0 2022-09-10T04:28:19.257 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:28:19.259 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f45c8099090 msgr2=0x7f45c809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f45c8099090 0x7f45c809b540 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d0009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 msgr2=0x7f45e01960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e01960a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f45d400e740 tx=0x7f45d400ea50 comp rx=0 tx=0).stop 2022-09-10T04:28:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 shutdown_connections 2022-09-10T04:28:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f45c8099090 0x7f45c809b540 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45e01038e0 0x7f45e019d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.262 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f45e0066bf0 0x7f45e0195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.262 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 --2- 172.21.15.81:0/1107294545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45e0100df0 0x7f45e01960a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 >> 172.21.15.81:0/1107294545 conn(0x7f45e00f82e0 msgr2=0x7f45e00ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:19.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 shutdown_connections 2022-09-10T04:28:19.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.258+0000 7f45e69ff700 1 -- 172.21.15.81:0/1107294545 wait complete. 2022-09-10T04:28:19.287 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-10T04:28:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 -- 172.21.15.81:0/4077150456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 msgr2=0x7fbf9c108400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/4077150456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c108400 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009bb0 tx=0x7fbf94009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 -- 172.21.15.81:0/4077150456 shutdown_connections 2022-09-10T04:28:19.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/4077150456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c108400 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.523 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/4077150456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 0x7fbf9c100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/4077150456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 0x7fbf9c101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.518+0000 7fbfa1fff700 1 -- 172.21.15.81:0/4077150456 >> 172.21.15.81:0/4077150456 conn(0x7fbf9c0fb6b0 msgr2=0x7fbf9c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:19.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.519+0000 7fbfa1fff700 1 -- 172.21.15.81:0/4077150456 shutdown_connections 2022-09-10T04:28:19.524 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.519+0000 7fbfa1fff700 1 -- 172.21.15.81:0/4077150456 wait complete. 2022-09-10T04:28:19.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 Processor -- start 2022-09-10T04:28:19.525 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 -- start start 2022-09-10T04:28:19.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 0x7fbf9c195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 0x7fbf9c196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c19d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c19d660 con 0x7fbf9c1011b0 2022-09-10T04:28:19.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c19d7d0 con 0x7fbf9c101b40 2022-09-10T04:28:19.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbfa1fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c19d990 con 0x7fbf9c0fff30 2022-09-10T04:28:19.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.520+0000 7fbf9b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 0x7fbf9c195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9affd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 0x7fbf9c196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 0x7fbf9c195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:47270/0 (socket says 172.21.15.81:47270) 2022-09-10T04:28:19.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c19d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9b7fe700 1 -- 172.21.15.81:0/1308734122 learned_addr learned my addr 172.21.15.81:0/1308734122 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:19.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 -- 172.21.15.81:0/1308734122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 msgr2=0x7fbf9c195ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 0x7fbf9c195ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 -- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 msgr2=0x7fbf9c196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 0x7fbf9c196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 -- 172.21.15.81:0/1308734122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf940097e0 con 0x7fbf9c101b40 2022-09-10T04:28:19.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9affd700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 0x7fbf9c196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:19.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf9bfff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c19d090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009fd0 tx=0x7fbf9400b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:19.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.521+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9400f870 con 0x7fbf9c101b40 2022-09-10T04:28:19.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.522+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fbf9400ba80 con 0x7fbf9c101b40 2022-09-10T04:28:19.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.522+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94017470 con 0x7fbf9c101b40 2022-09-10T04:28:19.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.522+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c19dc10 con 0x7fbf9c101b40 2022-09-10T04:28:19.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.522+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c19e070 con 0x7fbf9c101b40 2022-09-10T04:28:19.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.523+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c10b6f0 con 0x7fbf9c101b40 2022-09-10T04:28:19.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.525+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fbf94029070 con 0x7fbf9c101b40 2022-09-10T04:28:19.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.526+0000 7fbf98ff9700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbf84099010 0x7fbf8409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.526+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fbf940c3670 con 0x7fbf9c101b40 2022-09-10T04:28:19.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.526+0000 7fbf9b7fe700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbf84099010 0x7fbf8409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.527+0000 7fbf9b7fe700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbf84099010 0x7fbf8409b4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009b70 tx=0x7fbf8c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:19.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.530+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fbf9408f9b0 con 0x7fbf9c101b40 2022-09-10T04:28:19.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:19 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/3331066799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:28:19.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:19 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1107294545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:28:19.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.721+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf9c19e950 con 0x7fbf9c101b40 2022-09-10T04:28:19.723 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.721+0000 7fbf98ff9700 1 -- 172.21.15.81:0/1308734122 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf94027020 con 0x7fbf9c101b40 2022-09-10T04:28:19.723 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2022-09-10T04:28:19.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbf84099010 msgr2=0x7fbf8409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.727 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbf84099010 0x7fbf8409b4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009b70 tx=0x7fbf8c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 msgr2=0x7fbf9c19d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.728 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c19d090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009fd0 tx=0x7fbf9400b890 comp rx=0 tx=0).stop 2022-09-10T04:28:19.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 shutdown_connections 2022-09-10T04:28:19.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf9c101b40 0x7fbf9c19d090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.729 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fbf84099010 0x7fbf8409b4c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf9c0fff30 0x7fbf9c195ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.730 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 --2- 172.21.15.81:0/1308734122 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c1011b0 0x7fbf9c196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 >> 172.21.15.81:0/1308734122 conn(0x7fbf9c0fb6b0 msgr2=0x7fbf9c0fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:19.731 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 shutdown_connections 2022-09-10T04:28:19.732 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.724+0000 7fbfa1fff700 1 -- 172.21.15.81:0/1308734122 wait complete. 2022-09-10T04:28:19.753 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-10T04:28:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:19 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/3331066799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:28:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:19 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1107294545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:28:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:19 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/3331066799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:28:19.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:19 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1107294545' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:28:19.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.981+0000 7f35069ff700 1 -- 172.21.15.81:0/598244520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35000febe0 msgr2=0x7f3500101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.981+0000 7f35069ff700 1 --2- 172.21.15.81:0/598244520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35000febe0 0x7f3500101040 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f34f4009750 tx=0x7f34f4009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:19.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 -- 172.21.15.81:0/598244520 shutdown_connections 2022-09-10T04:28:19.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 --2- 172.21.15.81:0/598244520 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f35001016d0 0x7f3500101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 --2- 172.21.15.81:0/598244520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3500066bf0 0x7f35000fe6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 --2- 172.21.15.81:0/598244520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35000febe0 0x7f3500101040 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 -- 172.21.15.81:0/598244520 >> 172.21.15.81:0/598244520 conn(0x7f35000f82e0 msgr2=0x7f35000fa710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:19.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 -- 172.21.15.81:0/598244520 shutdown_connections 2022-09-10T04:28:19.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.982+0000 7f35069ff700 1 -- 172.21.15.81:0/598244520 wait complete. 2022-09-10T04:28:19.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 Processor -- start 2022-09-10T04:28:19.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 -- start start 2022-09-10T04:28:19.993 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 0x7f3500199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.994 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 0x7f350019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.994 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35001016d0 0x7f35001a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:19.995 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35001a1a60 con 0x7f35000febe0 2022-09-10T04:28:19.995 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35001a1bd0 con 0x7f3500066bf0 2022-09-10T04:28:19.995 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.983+0000 7f35069ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35001a1d90 con 0x7f35001016d0 2022-09-10T04:28:19.996 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 0x7f350019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.996 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 0x7f3500199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.997 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f3504f9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35001016d0 0x7f35001a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:19.997 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 0x7f350019a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56014/0 (socket says 172.21.15.81:56014) 2022-09-10T04:28:19.997 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 0x7f3500199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51706/0 (socket says 172.21.15.81:51706) 2022-09-10T04:28:19.997 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34ff7fe700 1 -- 172.21.15.81:0/1567576688 learned_addr learned my addr 172.21.15.81:0/1567576688 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:19.998 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 -- 172.21.15.81:0/1567576688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35001016d0 msgr2=0x7f35001a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.998 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35001016d0 0x7f35001a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.998 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 -- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 msgr2=0x7f350019a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:19.998 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 0x7f350019a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:19.999 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 -- 172.21.15.81:0/1567576688 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f0009470 con 0x7f3500066bf0 2022-09-10T04:28:19.999 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34ff7fe700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 0x7f350019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:19.999 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.984+0000 7f34fffff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 0x7f3500199ee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3500066fc0 tx=0x7f34f000eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:19.999 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.985+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0004220 con 0x7f3500066bf0 2022-09-10T04:28:19.999 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.985+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f34f0004380 con 0x7f3500066bf0 2022-09-10T04:28:20.000 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.985+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f0003be0 con 0x7f3500066bf0 2022-09-10T04:28:20.000 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.985+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f40093b0 con 0x7f3500066bf0 2022-09-10T04:28:20.000 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.985+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35001a23a0 con 0x7f3500066bf0 2022-09-10T04:28:20.000 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.986+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350010faa0 con 0x7f3500066bf0 2022-09-10T04:28:20.001 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.988+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f34f001e020 con 0x7f3500066bf0 2022-09-10T04:28:20.001 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.989+0000 7f34fd7fa700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34e8099010 0x7f34e809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.001 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.989+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f34f0014070 con 0x7f3500066bf0 2022-09-10T04:28:20.001 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.989+0000 7f34ff7fe700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34e8099010 0x7f34e809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.001 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.990+0000 7f34ff7fe700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34e8099010 0x7f34e809b4c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f34f4006010 tx=0x7f34f40092d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:20.002 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:19.993+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f34f008eca0 con 0x7f3500066bf0 2022-09-10T04:28:20.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.183+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f350005fa30 con 0x7f3500066bf0 2022-09-10T04:28:20.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.184+0000 7f34fd7fa700 1 -- 172.21.15.81:0/1567576688 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7f34f00922c0 con 0x7f3500066bf0 2022-09-10T04:28:20.186 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:28:20.188 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.186+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34e8099010 msgr2=0x7f34e809b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.188 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.186+0000 7f35069ff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34e8099010 0x7f34e809b4c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f34f4006010 tx=0x7f34f40092d0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 msgr2=0x7f3500199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 0x7f3500199ee0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3500066fc0 tx=0x7f34f000eca0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 shutdown_connections 2022-09-10T04:28:20.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f34e8099010 0x7f34e809b4c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.189 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35001016d0 0x7f35001a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.190 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3500066bf0 0x7f3500199ee0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.190 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 --2- 172.21.15.81:0/1567576688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35000febe0 0x7f350019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.190 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 >> 172.21.15.81:0/1567576688 conn(0x7f35000f82e0 msgr2=0x7f35000fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:20.190 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 shutdown_connections 2022-09-10T04:28:20.191 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.187+0000 7f35069ff700 1 -- 172.21.15.81:0/1567576688 wait complete. 2022-09-10T04:28:20.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:28:20.455 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 -- 172.21.15.81:0/609569499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fdb40 msgr2=0x7f259c0fdf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 --2- 172.21.15.81:0/609569499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fdb40 0x7f259c0fdf50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f258c009750 tx=0x7f258c009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:20.456 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 -- 172.21.15.81:0/609569499 shutdown_connections 2022-09-10T04:28:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 --2- 172.21.15.81:0/609569499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f259c0ff510 0x7f259c1060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.457 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 --2- 172.21.15.81:0/609569499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fdb40 0x7f259c0fdf50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 --2- 172.21.15.81:0/609569499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fe490 0x7f259c0fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.458 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 -- 172.21.15.81:0/609569499 >> 172.21.15.81:0/609569499 conn(0x7f259c0f9230 msgr2=0x7f259c0fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:20.459 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 -- 172.21.15.81:0/609569499 shutdown_connections 2022-09-10T04:28:20.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.441+0000 7f25a099d700 1 -- 172.21.15.81:0/609569499 wait complete. 2022-09-10T04:28:20.460 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.442+0000 7f25a099d700 1 Processor -- start 2022-09-10T04:28:20.461 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.442+0000 7f25a099d700 1 -- start start 2022-09-10T04:28:20.461 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f25a099d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 0x7f259c195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f25a099d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fe490 0x7f259c195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.462 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f25a099d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f259c0ff510 0x7f259c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f25a099d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c19d4c0 con 0x7f259c0fdb40 2022-09-10T04:28:20.463 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f25a099d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c19d630 con 0x7f259c0ff510 2022-09-10T04:28:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f25a099d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259c19d7f0 con 0x7f259c0fe490 2022-09-10T04:28:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 0x7f259c195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.464 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 0x7f259c195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56032/0 (socket says 172.21.15.81:56032) 2022-09-10T04:28:20.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 -- 172.21.15.81:0/2536069411 learned_addr learned my addr 172.21.15.81:0/2536069411 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:20.465 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259affd700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fe490 0x7f259c195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259bfff700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f259c0ff510 0x7f259c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 -- 172.21.15.81:0/2536069411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fe490 msgr2=0x7f259c195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.466 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fe490 0x7f259c195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 -- 172.21.15.81:0/2536069411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f259c0ff510 msgr2=0x7f259c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.467 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f259c0ff510 0x7f259c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.443+0000 7f259b7fe700 1 -- 172.21.15.81:0/2536069411 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2590009470 con 0x7f259c0fdb40 2022-09-10T04:28:20.468 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.444+0000 7f259b7fe700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 0x7f259c195940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f258c004230 tx=0x7f258c004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:20.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.444+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c01d070 con 0x7f259c0fdb40 2022-09-10T04:28:20.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.444+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f258c005590 con 0x7f259c0fdb40 2022-09-10T04:28:20.469 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.444+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c017a90 con 0x7f259c0fdb40 2022-09-10T04:28:20.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.444+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c0093b0 con 0x7f259c0fdb40 2022-09-10T04:28:20.470 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.444+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259c19dce0 con 0x7f259c0fdb40 2022-09-10T04:28:20.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.445+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c109330 con 0x7f259c0fdb40 2022-09-10T04:28:20.471 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.447+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f258c00c040 con 0x7f259c0fdb40 2022-09-10T04:28:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.449+0000 7f2598ff9700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2584099090 0x7f258409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.449+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f258c0c32e0 con 0x7f259c0fdb40 2022-09-10T04:28:20.472 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.449+0000 7f259affd700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2584099090 0x7f258409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.450+0000 7f259affd700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2584099090 0x7f258409b540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2590009440 tx=0x7f2590005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:20.473 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.453+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f258c08f620 con 0x7f259c0fdb40 2022-09-10T04:28:20.692 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.690+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f259c19e630 con 0x7f259c0fdb40 2022-09-10T04:28:20.693 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.691+0000 7f2598ff9700 1 -- 172.21.15.81:0/2536069411 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10787 (secure 0 0 0) 0x7f258c092c40 con 0x7f259c0fdb40 2022-09-10T04:28:20.693 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:20.694 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-10T04:26:32.097174+0000","modified":"2022-09-10T04:26:59.184557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24577,"mds_1":24572,"mds_2":24547,"mds_3":24532,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24466":{"gid":24466,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"j","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24497":{"gid":24497,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24577":{"gid":24577,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-10T04:28:20.696 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2584099090 msgr2=0x7f258409b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2584099090 0x7f258409b540 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2590009440 tx=0x7f2590005f60 comp rx=0 tx=0).stop 2022-09-10T04:28:20.697 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 msgr2=0x7f259c195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 0x7f259c195940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f258c004230 tx=0x7f258c004260 comp rx=0 tx=0).stop 2022-09-10T04:28:20.698 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 shutdown_connections 2022-09-10T04:28:20.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f259c0ff510 0x7f259c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.699 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f259c0fdb40 0x7f259c195940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2584099090 0x7f258409b540 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 --2- 172.21.15.81:0/2536069411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f259c0fe490 0x7f259c195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 >> 172.21.15.81:0/2536069411 conn(0x7f259c0f9230 msgr2=0x7f259c0fb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 shutdown_connections 2022-09-10T04:28:20.702 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.694+0000 7f25a099d700 1 -- 172.21.15.81:0/2536069411 wait complete. 2022-09-10T04:28:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 28 2022-09-10T04:28:20.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:28:20.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:20 smithi104 ceph-mon[97044]: pgmap v359: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2022-09-10T04:28:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:20 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1308734122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:28:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:20 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1567576688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:28:20.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:20 smithi081 ceph-mon[90133]: pgmap v359: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2022-09-10T04:28:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:20 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1308734122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:28:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:20 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1567576688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:28:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:20 smithi110 ceph-mon[97633]: pgmap v359: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2022-09-10T04:28:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:20 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1308734122' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:28:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:20 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1567576688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:28:20.958 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.951+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/2586156061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f78100dd0 msgr2=0x7f2f78103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.951+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/2586156061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f78100dd0 0x7f2f78103230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c009810 tx=0x7f2f6c009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:20.959 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/2586156061 shutdown_connections 2022-09-10T04:28:20.960 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/2586156061 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f781038c0 0x7f2f78105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.961 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/2586156061 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.961 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/2586156061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f78100dd0 0x7f2f78103230 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.962 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/2586156061 >> 172.21.15.81:0/2586156061 conn(0x7f2f780f82a0 msgr2=0x7f2f780fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:20.962 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/2586156061 shutdown_connections 2022-09-10T04:28:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.952+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/2586156061 wait complete. 2022-09-10T04:28:20.963 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 Processor -- start 2022-09-10T04:28:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 -- start start 2022-09-10T04:28:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f78100dd0 0x7f2f78196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.965 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f781038c0 0x7f2f7819d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.965 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7819d6a0 con 0x7f2f78066bf0 2022-09-10T04:28:20.966 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7819d810 con 0x7f2f78100dd0 2022-09-10T04:28:20.967 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.953+0000 7f2f7e1ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7819d9d0 con 0x7f2f781038c0 2022-09-10T04:28:20.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56046/0 (socket says 172.21.15.81:56046) 2022-09-10T04:28:20.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 -- 172.21.15.81:0/921927077 learned_addr learned my addr 172.21.15.81:0/921927077 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:20.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f76ffd700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f78100dd0 0x7f2f78196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f77fff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f781038c0 0x7f2f7819d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 -- 172.21.15.81:0/921927077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f781038c0 msgr2=0x7f2f7819d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f781038c0 0x7f2f7819d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 -- 172.21.15.81:0/921927077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f78100dd0 msgr2=0x7f2f78196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:20.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f78100dd0 0x7f2f78196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:20.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f777fe700 1 -- 172.21.15.81:0/921927077 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c009470 con 0x7f2f78066bf0 2022-09-10T04:28:20.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.954+0000 7f2f77fff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f781038c0 0x7f2f7819d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:28:20.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.955+0000 7f2f777fe700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78195b20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800e6e0 tx=0x7f2f6800e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:20.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.955+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68009500 con 0x7f2f78066bf0 2022-09-10T04:28:20.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.955+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2f68010da0 con 0x7f2f78066bf0 2022-09-10T04:28:20.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.955+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7819dcb0 con 0x7f2f78066bf0 2022-09-10T04:28:20.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.955+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6800f420 con 0x7f2f78066bf0 2022-09-10T04:28:20.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.955+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7819e1c0 con 0x7f2f78066bf0 2022-09-10T04:28:20.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.956+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f781094d0 con 0x7f2f78066bf0 2022-09-10T04:28:20.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.958+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f2f6800c040 con 0x7f2f78066bf0 2022-09-10T04:28:20.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.959+0000 7f2f74ff9700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2f60099010 0x7f2f6009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:20.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.959+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f2f68014070 con 0x7f2f78066bf0 2022-09-10T04:28:20.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.960+0000 7f2f76ffd700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2f60099010 0x7f2f6009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:20.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.960+0000 7f2f76ffd700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2f60099010 0x7f2f6009b4c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00b4f0 tx=0x7f2f6c009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:20.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:20.963+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f2f6804fc80 con 0x7f2f78066bf0 2022-09-10T04:28:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.155+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f78105350 con 0x7f2f78066bf0 2022-09-10T04:28:21.158 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.156+0000 7f2f74ff9700 1 -- 172.21.15.81:0/921927077 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+23743 (secure 0 0 0) 0x7f2f68092710 con 0x7f2f78066bf0 2022-09-10T04:28:21.159 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:21.159 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":94,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:28:11.428965+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:28:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2f60099010 msgr2=0x7f2f6009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2f60099010 0x7f2f6009b4c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00b4f0 tx=0x7f2f6c009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 msgr2=0x7f2f78195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78195b20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800e6e0 tx=0x7f2f6800e9f0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 shutdown_connections 2022-09-10T04:28:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2f60099010 0x7f2f6009b4c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f781038c0 0x7f2f7819d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.163 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f78066bf0 0x7f2f78195b20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.163 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 --2- 172.21.15.81:0/921927077 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f78100dd0 0x7f2f78196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.163 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 >> 172.21.15.81:0/921927077 conn(0x7f2f780f82a0 msgr2=0x7f2f780ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:21.163 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 shutdown_connections 2022-09-10T04:28:21.164 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.159+0000 7f2f7e1ff700 1 -- 172.21.15.81:0/921927077 wait complete. 2022-09-10T04:28:21.188 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2022-09-10T04:21:40.072108+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2022-09-10T04:26:30.005808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2022-09-10T04:26:31.529234+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2022-09-10T04:26:35.269129+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-09-10T04:28:21.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2022-09-10T04:28:21.424 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 -- 172.21.15.81:0/3735976826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fe4c0 msgr2=0x7f3fc00feeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.425 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 --2- 172.21.15.81:0/3735976826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc00feeb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009750 tx=0x7f3fac009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:21.426 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 -- 172.21.15.81:0/3735976826 shutdown_connections 2022-09-10T04:28:21.427 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 --2- 172.21.15.81:0/3735976826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc00ff540 0x7f3fc01060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.427 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 --2- 172.21.15.81:0/3735976826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fdb70 0x7f3fc00fdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.428 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 --2- 172.21.15.81:0/3735976826 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc00feeb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.428 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 -- 172.21.15.81:0/3735976826 >> 172.21.15.81:0/3735976826 conn(0x7f3fc00f9260 msgr2=0x7f3fc00fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:21.428 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 -- 172.21.15.81:0/3735976826 shutdown_connections 2022-09-10T04:28:21.429 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.421+0000 7f3fc599d700 1 -- 172.21.15.81:0/3735976826 wait complete. 2022-09-10T04:28:21.429 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.423+0000 7f3fc599d700 1 Processor -- start 2022-09-10T04:28:21.430 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.423+0000 7f3fc599d700 1 -- start start 2022-09-10T04:28:21.436 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc599d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 0x7f3fc0193740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc599d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc0193c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc599d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc00ff540 0x7f3fc019acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc599d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc019b2c0 con 0x7f3fc00fe4c0 2022-09-10T04:28:21.438 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc599d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc019b430 con 0x7f3fc00fdb70 2022-09-10T04:28:21.438 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc599d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc019b5f0 con 0x7f3fc00ff540 2022-09-10T04:28:21.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc499b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 0x7f3fc0193740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.439 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fbffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc0193c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc499b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 0x7f3fc0193740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51766/0 (socket says 172.21.15.81:51766) 2022-09-10T04:28:21.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc499b700 1 -- 172.21.15.81:0/4270975086 learned_addr learned my addr 172.21.15.81:0/4270975086 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:21.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.424+0000 7f3fc519c700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc00ff540 0x7f3fc019acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc499b700 1 -- 172.21.15.81:0/4270975086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc00ff540 msgr2=0x7f3fc019acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc499b700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc00ff540 0x7f3fc019acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc499b700 1 -- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fe4c0 msgr2=0x7f3fc0193c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc499b700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc0193c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc499b700 1 -- 172.21.15.81:0/4270975086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac0093b0 con 0x7f3fc00fdb70 2022-09-10T04:28:21.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fbffff700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc0193c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:21.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc499b700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 0x7f3fc0193740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400e740 tx=0x7f3fb400ea50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:21.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4005220 con 0x7f3fc00fdb70 2022-09-10T04:28:21.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f3fb4005380 con 0x7f3fc00fdb70 2022-09-10T04:28:21.445 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4004c10 con 0x7f3fc00fdb70 2022-09-10T04:28:21.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc019b8d0 con 0x7f3fc00fdb70 2022-09-10T04:28:21.446 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.425+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc019bd90 con 0x7f3fc00fdb70 2022-09-10T04:28:21.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.426+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc0109330 con 0x7f3fc00fdb70 2022-09-10T04:28:21.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.429+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f3fb401e020 con 0x7f3fc00fdb70 2022-09-10T04:28:21.447 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.430+0000 7f3fbdffb700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3fb0099090 0x7f3fb009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.430+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f3fb4014070 con 0x7f3fc00fdb70 2022-09-10T04:28:21.448 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.430+0000 7f3fbffff700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3fb0099090 0x7f3fb009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.431+0000 7f3fbffff700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3fb0099090 0x7f3fb009b540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00f460 tx=0x7f3fac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:21.449 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.432+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f3fb408f2b0 con 0x7f3fc00fdb70 2022-09-10T04:28:21.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.626+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3fc0104ad0 con 0x7f3fc00fdb70 2022-09-10T04:28:21.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.627+0000 7f3fbdffb700 1 -- 172.21.15.81:0/4270975086 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3fb40928d0 con 0x7f3fc00fdb70 2022-09-10T04:28:21.629 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2022-09-10T04:28:21.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3fb0099090 msgr2=0x7f3fb009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3fb0099090 0x7f3fb009b540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00f460 tx=0x7f3fac005f90 comp rx=0 tx=0).stop 2022-09-10T04:28:21.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 msgr2=0x7f3fc0193740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 0x7f3fc0193740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400e740 tx=0x7f3fb400ea50 comp rx=0 tx=0).stop 2022-09-10T04:28:21.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 shutdown_connections 2022-09-10T04:28:21.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f3fb0099090 0x7f3fb009b540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fc00ff540 0x7f3fc019acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fc00fdb70 0x7f3fc0193740 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 --2- 172.21.15.81:0/4270975086 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc00fe4c0 0x7f3fc0193c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 >> 172.21.15.81:0/4270975086 conn(0x7f3fc00f9260 msgr2=0x7f3fc00fb520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:21.636 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 shutdown_connections 2022-09-10T04:28:21.636 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.630+0000 7f3fc599d700 1 -- 172.21.15.81:0/4270975086 wait complete. 2022-09-10T04:28:21.659 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-09-10T04:28:21.736 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:21 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/2536069411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:28:21.737 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:21 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/921927077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:21 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/2536069411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:28:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:21 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/921927077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:21.889 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 -- 172.21.15.81:0/3408513009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fef80 msgr2=0x7f97500ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 --2- 172.21.15.81:0/3408513009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fef80 0x7f97500ff3d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f973c009810 tx=0x7f973c009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:21.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 -- 172.21.15.81:0/3408513009 shutdown_connections 2022-09-10T04:28:21.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 --2- 172.21.15.81:0/3408513009 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97500ff910 0x7f97501061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.890 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 --2- 172.21.15.81:0/3408513009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fdd00 0x7f97500fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 --2- 172.21.15.81:0/3408513009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fef80 0x7f97500ff3d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 -- 172.21.15.81:0/3408513009 >> 172.21.15.81:0/3408513009 conn(0x7f97500f93f0 msgr2=0x7f97500fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:21.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 -- 172.21.15.81:0/3408513009 shutdown_connections 2022-09-10T04:28:21.891 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.887+0000 7f9756fff700 1 -- 172.21.15.81:0/3408513009 wait complete. 2022-09-10T04:28:21.893 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 Processor -- start 2022-09-10T04:28:21.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 -- start start 2022-09-10T04:28:21.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 0x7f9750195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fef80 0x7f9750196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.894 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97500ff910 0x7f975019d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975019d6c0 con 0x7f97500fef80 2022-09-10T04:28:21.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975019d830 con 0x7f97500fdd00 2022-09-10T04:28:21.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9756fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975019d9f0 con 0x7f97500ff910 2022-09-10T04:28:21.895 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9754d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 0x7f9750195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9754d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 0x7f9750195b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51780/0 (socket says 172.21.15.81:51780) 2022-09-10T04:28:21.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f974ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fef80 0x7f9750196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.889+0000 7f9754d9b700 1 -- 172.21.15.81:0/1799108666 learned_addr learned my addr 172.21.15.81:0/1799108666 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:21.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f975559c700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97500ff910 0x7f975019d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9754d9b700 1 -- 172.21.15.81:0/1799108666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97500ff910 msgr2=0x7f975019d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.897 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9754d9b700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97500ff910 0x7f975019d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9754d9b700 1 -- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fef80 msgr2=0x7f9750196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9754d9b700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fef80 0x7f9750196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9754d9b700 1 -- 172.21.15.81:0/1799108666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c009470 con 0x7f97500fdd00 2022-09-10T04:28:21.898 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f974ffff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fef80 0x7f9750196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9754d9b700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 0x7f9750195b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f974400e710 tx=0x7f974400ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9744004220 con 0x7f97500fdd00 2022-09-10T04:28:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975019dcd0 con 0x7f97500fdd00 2022-09-10T04:28:21.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9744004380 con 0x7f97500fdd00 2022-09-10T04:28:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.890+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9744003cf0 con 0x7f97500fdd00 2022-09-10T04:28:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.891+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975019e190 con 0x7f97500fdd00 2022-09-10T04:28:21.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.892+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97501094b0 con 0x7f97500fdd00 2022-09-10T04:28:21.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.894+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f974401e020 con 0x7f97500fdd00 2022-09-10T04:28:21.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.895+0000 7f974dffb700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9740099010 0x7f974009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:21.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.895+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f9744014070 con 0x7f97500fdd00 2022-09-10T04:28:21.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.895+0000 7f974ffff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9740099010 0x7f974009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:21.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.896+0000 7f974ffff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9740099010 0x7f974009b4c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f973c00b4f0 tx=0x7f973c009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:21.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:21.899+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f974408ecd0 con 0x7f97500fdd00 2022-09-10T04:28:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:21 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/2536069411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:28:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:21 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/921927077' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:22.092 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.090+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9750109640 con 0x7f97500fdd00 2022-09-10T04:28:22.093 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.091+0000 7f974dffb700 1 -- 172.21.15.81:0/1799108666 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9750109640 con 0x7f97500fdd00 2022-09-10T04:28:22.094 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:28:22.096 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.093+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9740099010 msgr2=0x7f974009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.093+0000 7f9756fff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9740099010 0x7f974009b4c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f973c00b4f0 tx=0x7f973c009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.093+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 msgr2=0x7f9750195b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.097 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.093+0000 7f9756fff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 0x7f9750195b40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f974400e710 tx=0x7f974400ea20 comp rx=0 tx=0).stop 2022-09-10T04:28:22.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 shutdown_connections 2022-09-10T04:28:22.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f9740099010 0x7f974009b4c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97500ff910 0x7f975019d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.098 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97500fdd00 0x7f9750195b40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 --2- 172.21.15.81:0/1799108666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97500fef80 0x7f9750196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.099 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 >> 172.21.15.81:0/1799108666 conn(0x7f97500f93f0 msgr2=0x7f97500fb710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:22.100 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 shutdown_connections 2022-09-10T04:28:22.100 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.094+0000 7f9756fff700 1 -- 172.21.15.81:0/1799108666 wait complete. 2022-09-10T04:28:22.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-09-10T04:28:22.354 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 -- 172.21.15.81:0/3829690815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0ff950 msgr2=0x7f421c106210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 --2- 172.21.15.81:0/3829690815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0ff950 0x7f421c106210 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4214009bb0 tx=0x7f4214009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 -- 172.21.15.81:0/3829690815 shutdown_connections 2022-09-10T04:28:22.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 --2- 172.21.15.81:0/3829690815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0ff950 0x7f421c106210 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.355 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 --2- 172.21.15.81:0/3829690815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421c0fdd40 0x7f421c0fe150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 --2- 172.21.15.81:0/3829690815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.352+0000 7f42219ff700 1 -- 172.21.15.81:0/3829690815 >> 172.21.15.81:0/3829690815 conn(0x7f421c0f9430 msgr2=0x7f421c0fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:22.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.353+0000 7f42219ff700 1 -- 172.21.15.81:0/3829690815 shutdown_connections 2022-09-10T04:28:22.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.353+0000 7f42219ff700 1 -- 172.21.15.81:0/3829690815 wait complete. 2022-09-10T04:28:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 Processor -- start 2022-09-10T04:28:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 -- start start 2022-09-10T04:28:22.360 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0fdd40 0x7f421c195af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421c0ff950 0x7f421c19d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c19d670 con 0x7f421c0fefc0 2022-09-10T04:28:22.361 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c19d7e0 con 0x7f421c0ff950 2022-09-10T04:28:22.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.354+0000 7f42219ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421c19d9a0 con 0x7f421c0fdd40 2022-09-10T04:28:22.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421a7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0fdd40 0x7f421c195af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421a7fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c196030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56106/0 (socket says 172.21.15.81:56106) 2022-09-10T04:28:22.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421a7fc700 1 -- 172.21.15.81:0/1806795164 learned_addr learned my addr 172.21.15.81:0/1806795164 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:22.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421b7fe700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421c0ff950 0x7f421c19d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 -- 172.21.15.81:0/1806795164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421c0ff950 msgr2=0x7f421c19d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421c0ff950 0x7f421c19d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 -- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 msgr2=0x7f421c196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 -- 172.21.15.81:0/1806795164 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140097e0 con 0x7f421c0fdd40 2022-09-10T04:28:22.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421a7fc700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:22.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.355+0000 7f421affd700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0fdd40 0x7f421c195af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f420c005fd0 tx=0x7f420c00e9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.356+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c004220 con 0x7f421c0fdd40 2022-09-10T04:28:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.356+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f420c004380 con 0x7f421c0fdd40 2022-09-10T04:28:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.356+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c003b60 con 0x7f421c0fdd40 2022-09-10T04:28:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.356+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421c19dc80 con 0x7f421c0fdd40 2022-09-10T04:28:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.356+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421c19e140 con 0x7f421c0fdd40 2022-09-10T04:28:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.357+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421c109500 con 0x7f421c0fdd40 2022-09-10T04:28:22.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.359+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f420c01e020 con 0x7f421c0fdd40 2022-09-10T04:28:22.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.360+0000 7f42209fd700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4204099010 0x7f420409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.360+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f420c014070 con 0x7f421c0fdd40 2022-09-10T04:28:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.361+0000 7f421a7fc700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4204099010 0x7f420409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.361+0000 7f421a7fc700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4204099010 0x7f420409b4c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f42100057d0 tx=0x7f4210005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:22.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.362+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f420c08ed80 con 0x7f421c0fdd40 2022-09-10T04:28:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.556+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f421c190190 con 0x7f421c0fdd40 2022-09-10T04:28:22.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.556+0000 7f42209fd700 1 -- 172.21.15.81:0/1806795164 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f420c00eea0 con 0x7f421c0fdd40 2022-09-10T04:28:22.560 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2022-09-10T04:28:22.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4204099010 msgr2=0x7f420409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4204099010 0x7f420409b4c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f42100057d0 tx=0x7f4210005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0fdd40 msgr2=0x7f421c195af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0fdd40 0x7f421c195af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f420c005fd0 tx=0x7f420c00e9a0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 shutdown_connections 2022-09-10T04:28:22.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421c0ff950 0x7f421c19d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f4204099010 0x7f420409b4c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f421c0fdd40 0x7f421c195af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 --2- 172.21.15.81:0/1806795164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f421c0fefc0 0x7f421c196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 >> 172.21.15.81:0/1806795164 conn(0x7f421c0f9430 msgr2=0x7f421c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:22.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 shutdown_connections 2022-09-10T04:28:22.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.559+0000 7f42219ff700 1 -- 172.21.15.81:0/1806795164 wait complete. 2022-09-10T04:28:22.595 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2022-09-10T04:28:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:22 smithi104 ceph-mon[97044]: pgmap v360: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:28:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:22 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4270975086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:28:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:22 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1799108666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:28:22.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:22 smithi081 ceph-mon[90133]: pgmap v360: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:28:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:22 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4270975086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:28:22.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:22 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1799108666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:28:22.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.823+0000 7f1f297ff700 1 -- 172.21.15.81:0/1000195950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fef80 msgr2=0x7f1f240ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.831 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.823+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1000195950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fef80 0x7f1f240ff3d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009750 tx=0x7f1f18009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:22.831 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 -- 172.21.15.81:0/1000195950 shutdown_connections 2022-09-10T04:28:22.832 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1000195950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f240ff910 0x7f1f241061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.832 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1000195950 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fdd00 0x7f1f240fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1000195950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fef80 0x7f1f240ff3d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 -- 172.21.15.81:0/1000195950 >> 172.21.15.81:0/1000195950 conn(0x7f1f240f93f0 msgr2=0x7f1f240fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:22.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 -- 172.21.15.81:0/1000195950 shutdown_connections 2022-09-10T04:28:22.835 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.824+0000 7f1f297ff700 1 -- 172.21.15.81:0/1000195950 wait complete. 2022-09-10T04:28:22.835 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 Processor -- start 2022-09-10T04:28:22.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 -- start start 2022-09-10T04:28:22.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fdd00 0x7f1f24072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 0x7f1f240710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f240ff910 0x7f1f24071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.837 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f24071e10 con 0x7f1f240fdd00 2022-09-10T04:28:22.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f24071f80 con 0x7f1f240fef80 2022-09-10T04:28:22.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.825+0000 7f1f297ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2419da10 con 0x7f1f240ff910 2022-09-10T04:28:22.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 0x7f1f240710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f22ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fdd00 0x7f1f24072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 0x7f1f240710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51820/0 (socket says 172.21.15.81:51820) 2022-09-10T04:28:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 -- 172.21.15.81:0/1125894348 learned_addr learned my addr 172.21.15.81:0/1125894348 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f237fe700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f240ff910 0x7f1f24071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 -- 172.21.15.81:0/1125894348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f240ff910 msgr2=0x7f1f24071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f240ff910 0x7f1f24071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 -- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fdd00 msgr2=0x7f1f24072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:22.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fdd00 0x7f1f24072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:22.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 -- 172.21.15.81:0/1125894348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f14009470 con 0x7f1f240fef80 2022-09-10T04:28:22.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.826+0000 7f1f227fc700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 0x7f1f240710d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1f18004230 tx=0x7f1f18004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:22.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.827+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1801d070 con 0x7f1f240fef80 2022-09-10T04:28:22.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.827+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c009730 con 0x7f1f240fef80 2022-09-10T04:28:22.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.827+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1f18005590 con 0x7f1f240fef80 2022-09-10T04:28:22.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.827+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18017a90 con 0x7f1f240fef80 2022-09-10T04:28:22.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.827+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f180093b0 con 0x7f1f240fef80 2022-09-10T04:28:22.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.828+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f241094b0 con 0x7f1f240fef80 2022-09-10T04:28:22.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.830+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f1f1800c040 con 0x7f1f240fef80 2022-09-10T04:28:22.845 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.831+0000 7f1f07fff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f0c099090 0x7f1f0c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:22.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.831+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f1f180c35e0 con 0x7f1f240fef80 2022-09-10T04:28:22.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.831+0000 7f1f22ffd700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f0c099090 0x7f1f0c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:22.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.832+0000 7f1f22ffd700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f0c099090 0x7f1f0c09b540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009440 tx=0x7f1f14005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:22.846 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:22.833+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1f1808f920 con 0x7f1f240fef80 2022-09-10T04:28:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:22 smithi110 ceph-mon[97633]: pgmap v360: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:28:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:22 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4270975086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2022-09-10T04:28:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:22 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1799108666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2022-09-10T04:28:23.025 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.024+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1f24109640 con 0x7f1f240fef80 2022-09-10T04:28:23.027 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.024+0000 7f1f07fff700 1 -- 172.21.15.81:0/1125894348 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1f24109640 con 0x7f1f240fef80 2022-09-10T04:28:23.027 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2022-09-10T04:28:23.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f0c099090 msgr2=0x7f1f0c09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f0c099090 0x7f1f0c09b540 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009440 tx=0x7f1f14005f60 comp rx=0 tx=0).stop 2022-09-10T04:28:23.029 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 msgr2=0x7f1f240710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.030 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 0x7f1f240710d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1f18004230 tx=0x7f1f18004260 comp rx=0 tx=0).stop 2022-09-10T04:28:23.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 shutdown_connections 2022-09-10T04:28:23.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1f0c099090 0x7f1f0c09b540 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f240ff910 0x7f1f24071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f240fdd00 0x7f1f24072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 --2- 172.21.15.81:0/1125894348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1f240fef80 0x7f1f240710d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.027+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 >> 172.21.15.81:0/1125894348 conn(0x7f1f240f93f0 msgr2=0x7f1f240fb750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:23.034 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.028+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 shutdown_connections 2022-09-10T04:28:23.034 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.028+0000 7f1f297ff700 1 -- 172.21.15.81:0/1125894348 wait complete. 2022-09-10T04:28:23.056 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-09-10T04:28:23.287 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 -- 172.21.15.81:0/2389209109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 msgr2=0x7efc4c1061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 --2- 172.21.15.81:0/2389209109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c1061c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efc44009bb0 tx=0x7efc44009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 -- 172.21.15.81:0/2389209109 shutdown_connections 2022-09-10T04:28:23.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 --2- 172.21.15.81:0/2389209109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c1061c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.288 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 --2- 172.21.15.81:0/2389209109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc4c0fdcf0 0x7efc4c0fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 --2- 172.21.15.81:0/2389209109 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 0x7efc4c0ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 -- 172.21.15.81:0/2389209109 >> 172.21.15.81:0/2389209109 conn(0x7efc4c0f93c0 msgr2=0x7efc4c0fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:23.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.285+0000 7efc50dff700 1 -- 172.21.15.81:0/2389209109 shutdown_connections 2022-09-10T04:28:23.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.286+0000 7efc50dff700 1 -- 172.21.15.81:0/2389209109 wait complete. 2022-09-10T04:28:23.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 Processor -- start 2022-09-10T04:28:23.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 -- start start 2022-09-10T04:28:23.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc4c0fdcf0 0x7efc4c072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 0x7efc4c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.294 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c071e10 con 0x7efc4c0ff900 2022-09-10T04:28:23.294 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c071f80 con 0x7efc4c0fef70 2022-09-10T04:28:23.294 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc50dff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c19da20 con 0x7efc4c0fdcf0 2022-09-10T04:28:23.295 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc4ad9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.295 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.287+0000 7efc4ad9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56144/0 (socket says 172.21.15.81:56144) 2022-09-10T04:28:23.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc4ad9d700 1 -- 172.21.15.81:0/88476424 learned_addr learned my addr 172.21.15.81:0/88476424 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:23.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 0x7efc4c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.296 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc4a59c700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc4c0fdcf0 0x7efc4c072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 -- 172.21.15.81:0/88476424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc4c0fdcf0 msgr2=0x7efc4c072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.297 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc4c0fdcf0 0x7efc4c072a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.298 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 -- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 msgr2=0x7efc4c071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.298 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.298 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 -- 172.21.15.81:0/88476424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc440097e0 con 0x7efc4c0fef70 2022-09-10T04:28:23.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc4ad9d700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:23.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.288+0000 7efc49d9b700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 0x7efc4c0710d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e6c0 tx=0x7efc4000e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:23.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.289+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40010900 con 0x7efc4c0fef70 2022-09-10T04:28:23.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.289+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4c19dd00 con 0x7efc4c0fef70 2022-09-10T04:28:23.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.289+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7efc40018420 con 0x7efc4c0fef70 2022-09-10T04:28:23.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.289+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40017610 con 0x7efc4c0fef70 2022-09-10T04:28:23.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.289+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4c19e200 con 0x7efc4c0fef70 2022-09-10T04:28:23.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.290+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c109520 con 0x7efc4c0fef70 2022-09-10T04:28:23.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.292+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7efc4000c040 con 0x7efc4c0fef70 2022-09-10T04:28:23.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.293+0000 7efc3b7fe700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7efc34099010 0x7efc3409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.293+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7efc40014070 con 0x7efc4c0fef70 2022-09-10T04:28:23.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.293+0000 7efc4a59c700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7efc34099010 0x7efc3409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.294+0000 7efc4a59c700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7efc34099010 0x7efc3409b4c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0fedd0 tx=0x7efc3c005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:23.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.295+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7efc4008ead0 con 0x7efc4c0fef70 2022-09-10T04:28:23.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.537+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc4c19e900 con 0x7efc4c0fef70 2022-09-10T04:28:23.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.538+0000 7efc3b7fe700 1 -- 172.21.15.81:0/88476424 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10787 (secure 0 0 0) 0x7efc40097020 con 0x7efc4c0fef70 2022-09-10T04:28:23.541 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:23.541 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14415,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/316573610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":316573610},{"type":"v1","addr":"172.21.15.81:6837","nonce":316573610}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14460,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1769781597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1769781597},{"type":"v1","addr":"172.21.15.81:6839","nonce":1769781597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2022-09-10T04:26:32.097174+0000","modified":"2022-09-10T04:26:59.184557+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24577,"mds_1":24572,"mds_2":24547,"mds_3":24532,"mds_4":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24461":{"gid":24461,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6833/777476125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":777476125},{"type":"v1","addr":"172.21.15.104:6833","nonce":777476125}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24466":{"gid":24466,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6833/2514050760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2514050760},{"type":"v1","addr":"172.21.15.110:6833","nonce":2514050760}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"j","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.81:6835/1412455487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1412455487},{"type":"v1","addr":"172.21.15.81:6835","nonce":1412455487}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24497":{"gid":24497,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6835/2709283075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":2709283075},{"type":"v1","addr":"172.21.15.104:6835","nonce":2709283075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.110:6835/4033961168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6834","nonce":4033961168},{"type":"v1","addr":"172.21.15.110:6835","nonce":4033961168}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.104:6837/2202378802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":2202378802},{"type":"v1","addr":"172.21.15.104:6837","nonce":2202378802}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.110:6837/1467999026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6836","nonce":1467999026},{"type":"v1","addr":"172.21.15.110:6837","nonce":1467999026}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24547":{"gid":24547,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.110:6839/1670334720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6838","nonce":1670334720},{"type":"v1","addr":"172.21.15.110:6839","nonce":1670334720}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.104:6839/2357131382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":2357131382},{"type":"v1","addr":"172.21.15.104:6839","nonce":2357131382}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24577":{"gid":24577,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6841/953424075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":953424075},{"type":"v1","addr":"172.21.15.81:6841","nonce":953424075}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-09-10T04:28:23.544 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7efc34099010 msgr2=0x7efc3409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.544 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7efc34099010 0x7efc3409b4c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0fedd0 tx=0x7efc3c005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 msgr2=0x7efc4c0710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.545 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 0x7efc4c0710d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e6c0 tx=0x7efc4000e9d0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 shutdown_connections 2022-09-10T04:28:23.546 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc4c0ff900 0x7efc4c071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7efc34099010 0x7efc3409b4c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.547 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc4c0fdcf0 0x7efc4c072a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 --2- 172.21.15.81:0/88476424 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc4c0fef70 0x7efc4c0710d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 >> 172.21.15.81:0/88476424 conn(0x7efc4c0f93c0 msgr2=0x7efc4c0fb700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:23.548 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 shutdown_connections 2022-09-10T04:28:23.549 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.541+0000 7efc50dff700 1 -- 172.21.15.81:0/88476424 wait complete. 2022-09-10T04:28:23.549 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 28 2022-09-10T04:28:23.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-09-10T04:28:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:23 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1806795164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:28:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:23 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1125894348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:28:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 -- 172.21.15.81:0/925941050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40ff900 msgr2=0x7f15a41061c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/925941050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40ff900 0x7f15a41061c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f159c009b00 tx=0x7f159c009e10 comp rx=0 tx=0).stop 2022-09-10T04:28:23.798 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 -- 172.21.15.81:0/925941050 shutdown_connections 2022-09-10T04:28:23.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/925941050 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40ff900 0x7f15a41061c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/925941050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15a40fdcf0 0x7f15a40fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/925941050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40fef70 0x7f15a40ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 -- 172.21.15.81:0/925941050 >> 172.21.15.81:0/925941050 conn(0x7f15a40f93c0 msgr2=0x7f15a40fb810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:23.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 -- 172.21.15.81:0/925941050 shutdown_connections 2022-09-10T04:28:23.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.796+0000 7f15ab3ff700 1 -- 172.21.15.81:0/925941050 wait complete. 2022-09-10T04:28:23.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.797+0000 7f15ab3ff700 1 Processor -- start 2022-09-10T04:28:23.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 -- start start 2022-09-10T04:28:23.803 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40fdcf0 0x7f15a4195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15a40fef70 0x7f15a4196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40ff900 0x7f15a419d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a419d650 con 0x7f15a40fef70 2022-09-10T04:28:23.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a419d7c0 con 0x7f15a40ff900 2022-09-10T04:28:23.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15ab3ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a419d980 con 0x7f15a40fdcf0 2022-09-10T04:28:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15a919b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40fdcf0 0x7f15a4195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15a919b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40fdcf0 0x7f15a4195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39548/0 (socket says 172.21.15.81:39548) 2022-09-10T04:28:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15a919b700 1 -- 172.21.15.81:0/4000341459 learned_addr learned my addr 172.21.15.81:0/4000341459 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:23.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15a999c700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40ff900 0x7f15a419d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.798+0000 7f15a899a700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15a40fef70 0x7f15a4196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15a999c700 1 -- 172.21.15.81:0/4000341459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40fdcf0 msgr2=0x7f15a4195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15a999c700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40fdcf0 0x7f15a4195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15a999c700 1 -- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15a40fef70 msgr2=0x7f15a4196010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:23.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15a999c700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15a40fef70 0x7f15a4196010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:23.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15a999c700 1 -- 172.21.15.81:0/4000341459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1594009470 con 0x7f15a40ff900 2022-09-10T04:28:23.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15a999c700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40ff900 0x7f15a419d080 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f159c009a20 tx=0x7f159c00f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:23.807 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c00bbe0 con 0x7f15a40ff900 2022-09-10T04:28:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f159c00fb20 con 0x7f15a40ff900 2022-09-10T04:28:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c009730 con 0x7f15a40ff900 2022-09-10T04:28:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.799+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f159c017660 con 0x7f15a40ff900 2022-09-10T04:28:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.800+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a419def0 con 0x7f15a40ff900 2022-09-10T04:28:23.808 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.801+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a41094b0 con 0x7f15a40ff900 2022-09-10T04:28:23.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:23 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1806795164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:28:23.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:23 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1125894348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:28:23.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.803+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f159c027020 con 0x7f15a40ff900 2022-09-10T04:28:23.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.804+0000 7f159a7fc700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1590099010 0x7f159009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:23.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.804+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f159c0c3fe0 con 0x7f15a40ff900 2022-09-10T04:28:23.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.804+0000 7f15a919b700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1590099010 0x7f159009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:23.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.805+0000 7f15a919b700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1590099010 0x7f159009b4c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009b70 tx=0x7f15a0005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:23.811 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:23.806+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f159c090340 con 0x7f15a40ff900 2022-09-10T04:28:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:23 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1806795164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2022-09-10T04:28:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:23 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1125894348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2022-09-10T04:28:24.003 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.002+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15a41900c0 con 0x7f15a40ff900 2022-09-10T04:28:24.004 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.004+0000 7f159a7fc700 1 -- 172.21.15.81:0/4000341459 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+23743 (secure 0 0 0) 0x7f159c058e60 con 0x7f15a40ff900 2022-09-10T04:28:24.005 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-10T04:28:24.005 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":94,"fsid":"70cf895c-30bf-11ed-8431-001a4aab830c","created":"2022-09-10T04:18:07.859996+0000","modified":"2022-09-10T04:28:11.428965+0000","last_up_change":"2022-09-10T04:25:42.874308+0000","last_in_change":"2022-09-10T04:25:24.003587+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2022-09-10T04:21:40.072108+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2022-09-10T04:26:30.005808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2022-09-10T04:26:31.529234+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2022-09-10T04:26:35.269129+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"06d36b08-d01a-4f7b-87d2-3dd153d2c300","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6803","nonce":860456526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6805","nonce":860456526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6809","nonce":860456526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":860456526},{"type":"v1","addr":"172.21.15.81:6807","nonce":860456526}]},"public_addr":"172.21.15.81:6803/860456526","cluster_addr":"172.21.15.81:6805/860456526","heartbeat_back_addr":"172.21.15.81:6809/860456526","heartbeat_front_addr":"172.21.15.81:6807/860456526","state":["exists","up"]},{"osd":1,"uuid":"34797803-b77d-4ce2-a459-df89cb7ff968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6801","nonce":757653704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6803","nonce":757653704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6807","nonce":757653704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":757653704},{"type":"v1","addr":"172.21.15.104:6805","nonce":757653704}]},"public_addr":"172.21.15.104:6801/757653704","cluster_addr":"172.21.15.104:6803/757653704","heartbeat_back_addr":"172.21.15.104:6807/757653704","heartbeat_front_addr":"172.21.15.104:6805/757653704","state":["exists","up"]},{"osd":2,"uuid":"9e47fb11-0460-4c91-8284-c8854bcfbfba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6801","nonce":829575986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6803","nonce":829575986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6807","nonce":829575986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":829575986},{"type":"v1","addr":"172.21.15.110:6805","nonce":829575986}]},"public_addr":"172.21.15.110:6801/829575986","cluster_addr":"172.21.15.110:6803/829575986","heartbeat_back_addr":"172.21.15.110:6807/829575986","heartbeat_front_addr":"172.21.15.110:6805/829575986","state":["exists","up"]},{"osd":3,"uuid":"ada925c5-2eb8-48cf-9a82-bc4c2ff500c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6811","nonce":1343686033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6813","nonce":1343686033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6817","nonce":1343686033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":1343686033},{"type":"v1","addr":"172.21.15.81:6815","nonce":1343686033}]},"public_addr":"172.21.15.81:6811/1343686033","cluster_addr":"172.21.15.81:6813/1343686033","heartbeat_back_addr":"172.21.15.81:6817/1343686033","heartbeat_front_addr":"172.21.15.81:6815/1343686033","state":["exists","up"]},{"osd":4,"uuid":"73482f9e-093f-408b-96bf-34584c281ea7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6809","nonce":2302505309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6811","nonce":2302505309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6815","nonce":2302505309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":2302505309},{"type":"v1","addr":"172.21.15.104:6813","nonce":2302505309}]},"public_addr":"172.21.15.104:6809/2302505309","cluster_addr":"172.21.15.104:6811/2302505309","heartbeat_back_addr":"172.21.15.104:6815/2302505309","heartbeat_front_addr":"172.21.15.104:6813/2302505309","state":["exists","up"]},{"osd":5,"uuid":"9dd94ce3-5bdf-4664-97dd-73980655b4a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6809","nonce":3775240590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6811","nonce":3775240590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6815","nonce":3775240590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":3775240590},{"type":"v1","addr":"172.21.15.110:6813","nonce":3775240590}]},"public_addr":"172.21.15.110:6809/3775240590","cluster_addr":"172.21.15.110:6811/3775240590","heartbeat_back_addr":"172.21.15.110:6815/3775240590","heartbeat_front_addr":"172.21.15.110:6813/3775240590","state":["exists","up"]},{"osd":6,"uuid":"36d8ca99-65c5-4528-8ee8-de7cd6faf9fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6819","nonce":2282910145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6821","nonce":2282910145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6825","nonce":2282910145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2282910145},{"type":"v1","addr":"172.21.15.81:6823","nonce":2282910145}]},"public_addr":"172.21.15.81:6819/2282910145","cluster_addr":"172.21.15.81:6821/2282910145","heartbeat_back_addr":"172.21.15.81:6825/2282910145","heartbeat_front_addr":"172.21.15.81:6823/2282910145","state":["exists","up"]},{"osd":7,"uuid":"c0c52ea1-4f7e-4760-b9c5-9b369d476ebe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6817","nonce":4207080292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6819","nonce":4207080292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6823","nonce":4207080292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":4207080292},{"type":"v1","addr":"172.21.15.104:6821","nonce":4207080292}]},"public_addr":"172.21.15.104:6817/4207080292","cluster_addr":"172.21.15.104:6819/4207080292","heartbeat_back_addr":"172.21.15.104:6823/4207080292","heartbeat_front_addr":"172.21.15.104:6821/4207080292","state":["exists","up"]},{"osd":8,"uuid":"c08ff30d-679b-4637-b363-c5fcb59fe112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6817","nonce":462969903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6819","nonce":462969903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6823","nonce":462969903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":462969903},{"type":"v1","addr":"172.21.15.110:6821","nonce":462969903}]},"public_addr":"172.21.15.110:6817/462969903","cluster_addr":"172.21.15.110:6819/462969903","heartbeat_back_addr":"172.21.15.110:6823/462969903","heartbeat_front_addr":"172.21.15.110:6821/462969903","state":["exists","up"]},{"osd":9,"uuid":"9ba7689f-d148-4c4a-8f0f-54eb9c4e8cd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6827","nonce":3176150026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6829","nonce":3176150026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6833","nonce":3176150026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":3176150026},{"type":"v1","addr":"172.21.15.81:6831","nonce":3176150026}]},"public_addr":"172.21.15.81:6827/3176150026","cluster_addr":"172.21.15.81:6829/3176150026","heartbeat_back_addr":"172.21.15.81:6833/3176150026","heartbeat_front_addr":"172.21.15.81:6831/3176150026","state":["exists","up"]},{"osd":10,"uuid":"60318b87-7ef8-423c-8f41-1554cd1b4e9a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6825","nonce":675537330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6827","nonce":675537330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6831","nonce":675537330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":675537330},{"type":"v1","addr":"172.21.15.104:6829","nonce":675537330}]},"public_addr":"172.21.15.104:6825/675537330","cluster_addr":"172.21.15.104:6827/675537330","heartbeat_back_addr":"172.21.15.104:6831/675537330","heartbeat_front_addr":"172.21.15.104:6829/675537330","state":["exists","up"]},{"osd":11,"uuid":"da2d73f1-bba2-4c62-9463-ca7aa22701d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6825","nonce":595897724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6827","nonce":595897724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6831","nonce":595897724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":595897724},{"type":"v1","addr":"172.21.15.110:6829","nonce":595897724}]},"public_addr":"172.21.15.110:6825/595897724","cluster_addr":"172.21.15.110:6827/595897724","heartbeat_back_addr":"172.21.15.110:6831/595897724","heartbeat_front_addr":"172.21.15.110:6829/595897724","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:20:47.013802+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:12.617823+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:21:35.879988+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:08.603861+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:34.542686+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:22:57.755363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:29.840084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:23:55.686129+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:19.097994+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:24:51.420658+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:17.596001+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2022-09-10T04:25:41.048223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.25","mappings":[{"from":5,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/2927238869":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/3943821768":"2022-09-11T04:19:09.198520+0000","172.21.15.81:0/2359268015":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/2567224701":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/2774479929":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/1348172231":"2022-09-11T04:18:41.737767+0000","172.21.15.81:0/596170975":"2022-09-11T04:18:41.737767+0000","172.21.15.81:6801/3224857868":"2022-09-11T04:19:09.198520+0000","172.21.15.81:6800/3224857868":"2022-09-11T04:19:09.198520+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2022-09-10T04:28:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1590099010 msgr2=0x7f159009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.007 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1590099010 0x7f159009b4c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009b70 tx=0x7f15a0005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40ff900 msgr2=0x7f15a419d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40ff900 0x7f15a419d080 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f159c009a20 tx=0x7f159c00f870 comp rx=0 tx=0).stop 2022-09-10T04:28:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 shutdown_connections 2022-09-10T04:28:24.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f15a40ff900 0x7f15a419d080 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1590099010 0x7f159009b4c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15a40fdcf0 0x7f15a4195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 --2- 172.21.15.81:0/4000341459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15a40fef70 0x7f15a4196010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 >> 172.21.15.81:0/4000341459 conn(0x7f15a40f93c0 msgr2=0x7f15a4104890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.009 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 shutdown_connections 2022-09-10T04:28:24.010 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.006+0000 7f15ab3ff700 1 -- 172.21.15.81:0/4000341459 wait complete. 2022-09-10T04:28:24.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2022-09-10T04:26:32.097174+0000', 'modified': '2022-09-10T04:26:59.184557+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24577, 'mds_1': 24572, 'mds_2': 24547, 'mds_3': 24532, 'mds_4': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24461': {'gid': 24461, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6833/777476125', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6832', 'nonce': 777476125}, {'type': 'v1', 'addr': '172.21.15.104:6833', 'nonce': 777476125}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24466': {'gid': 24466, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.110:6833/2514050760', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6832', 'nonce': 2514050760}, {'type': 'v1', 'addr': '172.21.15.110:6833', 'nonce': 2514050760}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24473': {'gid': 24473, 'name': 'j', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.81:6835/1412455487', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1412455487}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1412455487}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24497': {'gid': 24497, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6835/2709283075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6834', 'nonce': 2709283075}, {'type': 'v1', 'addr': '172.21.15.104:6835', 'nonce': 2709283075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.110:6835/4033961168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6834', 'nonce': 4033961168}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 4033961168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.104:6837/2202378802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6836', 'nonce': 2202378802}, {'type': 'v1', 'addr': '172.21.15.104:6837', 'nonce': 2202378802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.110:6837/1467999026', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6836', 'nonce': 1467999026}, {'type': 'v1', 'addr': '172.21.15.110:6837', 'nonce': 1467999026}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.110:6839/1670334720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6838', 'nonce': 1670334720}, {'type': 'v1', 'addr': '172.21.15.110:6839', 'nonce': 1670334720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.104:6839/2357131382', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 2357131382}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 2357131382}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24577': {'gid': 24577, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.81:6841/953424075', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6840', 'nonce': 953424075}, {'type': 'v1', 'addr': '172.21.15.81:6841', 'nonce': 953424075}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-09-10T04:28:24.036 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2022-09-10T04:28:24.036 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2022-09-10T04:28:24.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 -- 172.21.15.81:0/3022305750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 msgr2=0x7f6274071b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 --2- 172.21.15.81:0/3022305750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274071b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6264009810 tx=0x7f6264009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 -- 172.21.15.81:0/3022305750 shutdown_connections 2022-09-10T04:28:24.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 --2- 172.21.15.81:0/3022305750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6274105a50 0x7f6274107e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 --2- 172.21.15.81:0/3022305750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6274072bf0 0x7f6274071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 --2- 172.21.15.81:0/3022305750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274071b20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 -- 172.21.15.81:0/3022305750 >> 172.21.15.81:0/3022305750 conn(0x7f62740f9230 msgr2=0x7f62740fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 -- 172.21.15.81:0/3022305750 shutdown_connections 2022-09-10T04:28:24.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.268+0000 7f627a19d700 1 -- 172.21.15.81:0/3022305750 wait complete. 2022-09-10T04:28:24.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.269+0000 7f627a19d700 1 Processor -- start 2022-09-10T04:28:24.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.269+0000 7f627a19d700 1 -- start start 2022-09-10T04:28:24.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627a19d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627a19d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6274072bf0 0x7f6274195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627a19d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6274105a50 0x7f627419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627a19d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627419d4a0 con 0x7f62740716b0 2022-09-10T04:28:24.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627a19d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627419d610 con 0x7f6274072bf0 2022-09-10T04:28:24.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627a19d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627419d7d0 con 0x7f6274105a50 2022-09-10T04:28:24.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56172/0 (socket says 172.21.15.81:56172) 2022-09-10T04:28:24.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 -- 172.21.15.81:0/1051066338 learned_addr learned my addr 172.21.15.81:0/1051066338 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627899a700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6274072bf0 0x7f6274195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627999c700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6274105a50 0x7f627419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.283 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 -- 172.21.15.81:0/1051066338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6274105a50 msgr2=0x7f627419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6274105a50 0x7f627419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 -- 172.21.15.81:0/1051066338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6274072bf0 msgr2=0x7f6274195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6274072bf0 0x7f6274195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.284 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.270+0000 7f627919b700 1 -- 172.21.15.81:0/1051066338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6264009470 con 0x7f62740716b0 2022-09-10T04:28:24.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.271+0000 7f627919b700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274195920 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f627000e710 tx=0x7f627000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.271+0000 7f626a7fc700 1 -- 172.21.15.81:0/1051066338 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270004220 con 0x7f62740716b0 2022-09-10T04:28:24.285 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.271+0000 7f626a7fc700 1 -- 172.21.15.81:0/1051066338 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6270004380 con 0x7f62740716b0 2022-09-10T04:28:24.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.271+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627419dab0 con 0x7f62740716b0 2022-09-10T04:28:24.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.271+0000 7f626a7fc700 1 -- 172.21.15.81:0/1051066338 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270003cf0 con 0x7f62740716b0 2022-09-10T04:28:24.286 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.271+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627419e010 con 0x7f62740716b0 2022-09-10T04:28:24.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.274+0000 7f626a7fc700 1 -- 172.21.15.81:0/1051066338 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f627001e020 con 0x7f62740716b0 2022-09-10T04:28:24.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.276+0000 7f626a7fc700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6260099090 0x7f626009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.276+0000 7f626a7fc700 1 -- 172.21.15.81:0/1051066338 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f6270014070 con 0x7f62740716b0 2022-09-10T04:28:24.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.276+0000 7f628e86ab80 1 Processor -- start 2022-09-10T04:28:24.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.276+0000 7f627899a700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6260099090 0x7f626009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.276+0000 7f628e86ab80 1 -- start start 2022-09-10T04:28:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbcb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbc9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbca1c0 0x557fabbca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f628e86ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabab4080 con 0x557fabbca1c0 2022-09-10T04:28:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f628e86ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabb1e9e0 con 0x557fabbcb000 2022-09-10T04:28:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f628e86ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabb18180 con 0x557fabbcb950 2022-09-10T04:28:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f627899a700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6260099090 0x7f626009b540 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f626400b4f0 tx=0x7f6264009eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f627919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbcb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f627899a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbc9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f627999c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbca1c0 0x557fabbca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f627919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbcb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51892/0 (socket says 172.21.15.81:51892) 2022-09-10T04:28:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.277+0000 7f627919b700 1 -- 172.21.15.81:0/1057353425 learned_addr learned my addr 172.21.15.81:0/1057353425 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.303 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f627919b700 1 -- 172.21.15.81:0/1057353425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 msgr2=0x557fabbc9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f627919b700 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbc9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f627919b700 1 -- 172.21.15.81:0/1057353425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbca1c0 msgr2=0x557fabbca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f627919b700 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbca1c0 0x557fabbca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.304 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f627919b700 1 -- 172.21.15.81:0/1057353425 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fab9a05e0 con 0x557fabbcb000 2022-09-10T04:28:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f627919b700 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbcb410 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62700923d0 tx=0x7f6270092400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1925b18b8efa882 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f62577fe700 1 -- 172.21.15.81:0/1057353425 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270092f90 con 0x557fabbcb000 2022-09-10T04:28:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f62577fe700 1 -- 172.21.15.81:0/1057353425 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6270092890 con 0x557fabbcb000 2022-09-10T04:28:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.278+0000 7f62577fe700 1 -- 172.21.15.81:0/1057353425 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270092af0 con 0x557fabbcb000 2022-09-10T04:28:24.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 -- 172.21.15.81:0/1057353425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 msgr2=0x557fabbcb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbcb410 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f62700923d0 tx=0x7f6270092400 comp rx=0 tx=0).stop 2022-09-10T04:28:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 -- 172.21.15.81:0/1057353425 shutdown_connections 2022-09-10T04:28:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbca1c0 0x557fabbca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbcb410 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 --2- 172.21.15.81:0/1057353425 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbc9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 -- 172.21.15.81:0/1057353425 >> 172.21.15.81:0/1057353425 conn(0x557fabbc6170 msgr2=0x557fabbc45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 -- 172.21.15.81:0/1057353425 shutdown_connections 2022-09-10T04:28:24.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.279+0000 7f628e86ab80 1 -- 172.21.15.81:0/1057353425 wait complete. 2022-09-10T04:28:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 -- start start 2022-09-10T04:28:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 Processor -- start 2022-09-10T04:28:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb000 0x557fabbd3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbdabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557faba060c0 con 0x557fabbcb950 2022-09-10T04:28:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabbdb120 con 0x557fabbca1c0 2022-09-10T04:28:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabbdb2b0 con 0x557fabbcb000 2022-09-10T04:28:24.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.280+0000 7f628e86ab80 10 client.0 _ll_register_callbacks cb 0x557fabac3370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627999c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbdabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627899a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb000 0x557fabbd3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51894/0 (socket says 172.21.15.81:51894) 2022-09-10T04:28:24.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 -- 172.21.15.81:0/224048981 learned_addr learned my addr 172.21.15.81:0/224048981 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 -- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb000 msgr2=0x557fabbd3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb000 0x557fabbd3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 -- 172.21.15.81:0/224048981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 msgr2=0x557fabbdabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbdabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 -- 172.21.15.81:0/224048981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627419e010 con 0x557fabbca1c0 2022-09-10T04:28:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f627919b700 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f627000e6e0 tx=0x7f62700920d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f0daf623b137f server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.281+0000 7f6254ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:28:24.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f6254ff9700 1 -- 172.21.15.81:0/224048981 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270092f90 con 0x557fabbca1c0 2022-09-10T04:28:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f6254ff9700 1 -- 172.21.15.81:0/224048981 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f6270099b30 con 0x557fabbca1c0 2022-09-10T04:28:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557fabbbccf0 con 0x557fabbca1c0 2022-09-10T04:28:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f6254ff9700 1 -- 172.21.15.81:0/224048981 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62700af320 con 0x557fabbca1c0 2022-09-10T04:28:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 10 client.25271 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:24.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557fab94cd90 con 0x557fabbca1c0 2022-09-10T04:28:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f6254ff9700 1 -- 172.21.15.81:0/224048981 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x557fabbbccf0 con 0x557fabbca1c0 2022-09-10T04:28:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f6254ff9700 1 -- 172.21.15.81:0/224048981 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f62700a9ad0 con 0x557fabbca1c0 2022-09-10T04:28:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 10 client.25271 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:24.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 10 client.25271 resolve_mds: resolved name 'l' to daemon mds.l 2022-09-10T04:28:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 20 client.25271 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabbde6a0 0x557fabbe3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 4 client.25271 mds_command: new command op to 24466 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:28:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.282+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 --> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557fabb994c0 con 0x557fabbde6a0 2022-09-10T04:28:24.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.283+0000 7f627899a700 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabbde6a0 0x557fabbe3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.283+0000 7f627899a700 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabbde6a0 0x557fabbe3940 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a1d21fddcd0e41b4 server_cookie=4d960ec036edd9ae in_seq=0 out_seq=0 2022-09-10T04:28:24.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.283+0000 7f6254ff9700 10 client.25271 ms_handle_connect on v2:172.21.15.110:6832/2514050760 2022-09-10T04:28:24.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.284+0000 7f6254ff9700 1 -- 172.21.15.81:0/224048981 <== mds.4 v2:172.21.15.110:6832/2514050760 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x557fabb994c0 con 0x557fabbde6a0 2022-09-10T04:28:24.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.284+0000 7f6254ff9700 10 client.25271 handle_command_reply: tid=0 2022-09-10T04:28:24.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.284+0000 7f628e86ab80 1 client.25271 shutdown 2022-09-10T04:28:24.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 msgr2=0x557fabbd3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f627000e6e0 tx=0x7f62700920d0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 shutdown_connections 2022-09-10T04:28:24.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabbde6a0 0x557fabbe3940 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbdabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd3660 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 --2- 172.21.15.81:0/224048981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb000 0x557fabbd3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 >> 172.21.15.81:0/224048981 conn(0x557fabbc6170 msgr2=0x557fabbc3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f6254ff9700 0 client.25271 ms_handle_reset on v2:172.21.15.110:6832/2514050760 2022-09-10T04:28:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 shutdown_connections 2022-09-10T04:28:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 1 -- 172.21.15.81:0/224048981 wait complete. 2022-09-10T04:28:24.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.285+0000 7f628e86ab80 20 client.25271 trim_cache size 0 max 16384 2022-09-10T04:28:24.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.302+0000 7f628e86ab80 1 Processor -- start 2022-09-10T04:28:24.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.302+0000 7f628e86ab80 1 -- start start 2022-09-10T04:28:24.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbca1c0 0x557fabbd1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.326 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbd18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbd3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f628e86ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabbd3230 con 0x557fabbcb950 2022-09-10T04:28:24.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f628e86ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabbbf580 con 0x557fabbcb000 2022-09-10T04:28:24.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f628e86ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabbc5440 con 0x557fabbca1c0 2022-09-10T04:28:24.327 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f627899a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbd18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f627919b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbca1c0 0x557fabbd1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f627899a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbd18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51896/0 (socket says 172.21.15.81:51896) 2022-09-10T04:28:24.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f627899a700 1 -- 172.21.15.81:0/4171147575 learned_addr learned my addr 172.21.15.81:0/4171147575 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.328 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.303+0000 7f627999c700 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbd3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627899a700 1 -- 172.21.15.81:0/4171147575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbca1c0 msgr2=0x557fabbd1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627899a700 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbca1c0 0x557fabbd1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627899a700 1 -- 172.21.15.81:0/4171147575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 msgr2=0x557fabbd3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627899a700 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbd3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.329 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627899a700 1 -- 172.21.15.81:0/4171147575 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fab94b260 con 0x557fabbcb000 2022-09-10T04:28:24.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627919b700 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbca1c0 0x557fabbd1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:24.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f627899a700 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbd18d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6264006010 tx=0x7f6264020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=252ccb372af08c54 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f6257fff700 1 -- 172.21.15.81:0/4171147575 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626400bbc0 con 0x557fabbcb000 2022-09-10T04:28:24.330 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f6257fff700 1 -- 172.21.15.81:0/4171147575 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f626401e040 con 0x557fabbcb000 2022-09-10T04:28:24.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.304+0000 7f6257fff700 1 -- 172.21.15.81:0/4171147575 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264038480 con 0x557fabbcb000 2022-09-10T04:28:24.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 -- 172.21.15.81:0/4171147575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 msgr2=0x557fabbd18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbd18d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6264006010 tx=0x7f6264020000 comp rx=0 tx=0).stop 2022-09-10T04:28:24.331 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 -- 172.21.15.81:0/4171147575 shutdown_connections 2022-09-10T04:28:24.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb950 0x557fabbd3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbca1c0 0x557fabbd1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 --2- 172.21.15.81:0/4171147575 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbcb000 0x557fabbd18d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.332 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 -- 172.21.15.81:0/4171147575 >> 172.21.15.81:0/4171147575 conn(0x557fabbc6170 msgr2=0x557fabbce040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 -- 172.21.15.81:0/4171147575 shutdown_connections 2022-09-10T04:28:24.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 -- 172.21.15.81:0/4171147575 wait complete. 2022-09-10T04:28:24.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.305+0000 7f628e86ab80 1 -- start start 2022-09-10T04:28:24.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.306+0000 7f628e86ab80 1 Processor -- start 2022-09-10T04:28:24.333 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.306+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.306+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb000 0x557fabbd79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f628e86ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbd7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f628e86ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabc30930 con 0x557fabbcb000 2022-09-10T04:28:24.334 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f628e86ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabc30aa0 con 0x557fabbca1c0 2022-09-10T04:28:24.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f628e86ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fabc30c60 con 0x557fabbcb950 2022-09-10T04:28:24.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f628e86ab80 10 client.0 _ll_register_callbacks cb 0x557fabac3370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:24.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.335 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51908/0 (socket says 172.21.15.81:51908) 2022-09-10T04:28:24.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 -- 172.21.15.81:0/2713651943 learned_addr learned my addr 172.21.15.81:0/2713651943 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627899a700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb000 0x557fabbd79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627999c700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbd7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 -- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 msgr2=0x557fabbd7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.336 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbd7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.337 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 -- 172.21.15.81:0/2713651943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb000 msgr2=0x557fabbd79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.337 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb000 0x557fabbd79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.337 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 -- 172.21.15.81:0/2713651943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557fab94cd90 con 0x557fabbca1c0 2022-09-10T04:28:24.337 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f627919b700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd7460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6270092590 tx=0x7f62700a9d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8274788f4f92666 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.307+0000 7f6254ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:28:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f6254ff9700 1 -- 172.21.15.81:0/2713651943 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270092f90 con 0x557fabbca1c0 2022-09-10T04:28:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f6254ff9700 1 -- 172.21.15.81:0/2713651943 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f6270099b30 con 0x557fabbca1c0 2022-09-10T04:28:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6270092890 con 0x557fabbca1c0 2022-09-10T04:28:24.338 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f6254ff9700 1 -- 172.21.15.81:0/2713651943 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62700af320 con 0x557fabbca1c0 2022-09-10T04:28:24.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 10 client.25277 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:24.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f6274108a60 con 0x557fabbca1c0 2022-09-10T04:28:24.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f6254ff9700 1 -- 172.21.15.81:0/2713651943 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f62700b3080 con 0x557fabbca1c0 2022-09-10T04:28:24.339 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f6254ff9700 1 -- 172.21.15.81:0/2713651943 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f62700917f0 con 0x557fabbca1c0 2022-09-10T04:28:24.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 10 client.25277 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:24.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 10 client.25277 resolve_mds: resolved name 'l' to daemon mds.l 2022-09-10T04:28:24.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 20 client.25277 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:24.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabc354e0 0x557fabc31f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.340 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 4 client.25277 mds_command: new command op to 24466 tid=0[{"prefix": "status"}] 2022-09-10T04:28:24.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.308+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 --> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557fabb116f0 con 0x557fabc354e0 2022-09-10T04:28:24.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.309+0000 7f627899a700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabc354e0 0x557fabc31f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.309+0000 7f627899a700 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabc354e0 0x557fabc31f40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=27eb3722e735855d server_cookie=1d6abad11e7394e3 in_seq=0 out_seq=0 2022-09-10T04:28:24.341 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.309+0000 7f6254ff9700 10 client.25277 ms_handle_connect on v2:172.21.15.110:6832/2514050760 2022-09-10T04:28:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f6254ff9700 1 -- 172.21.15.81:0/2713651943 <== mds.4 v2:172.21.15.110:6832/2514050760 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x557fabb116f0 con 0x557fabc354e0 2022-09-10T04:28:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f6254ff9700 10 client.25277 handle_command_reply: tid=0 2022-09-10T04:28:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 client.25277 shutdown 2022-09-10T04:28:24.342 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 msgr2=0x557fabbd7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd7460 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6270092590 tx=0x7f62700a9d60 comp rx=0 tx=0).stop 2022-09-10T04:28:24.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 shutdown_connections 2022-09-10T04:28:24.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:6832/2514050760,v1:172.21.15.110:6833/2514050760] conn(0x557fabc354e0 0x557fabc31f40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x557fabbcb950 0x557fabbd7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.343 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f6254ff9700 0 client.25277 ms_handle_reset on v2:172.21.15.110:6832/2514050760 2022-09-10T04:28:24.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x557fabbca1c0 0x557fabbd7460 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 --2- 172.21.15.81:0/2713651943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557fabbcb000 0x557fabbd79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 >> 172.21.15.81:0/2713651943 conn(0x557fabbc6170 msgr2=0x557fabbd1390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.344 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 shutdown_connections 2022-09-10T04:28:24.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 1 -- 172.21.15.81:0/2713651943 wait complete. 2022-09-10T04:28:24.345 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.310+0000 7f628e86ab80 20 client.25277 trim_cache size 0 max 16384 2022-09-10T04:28:24.345 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:28:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:28:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 4, 2022-09-10T04:28:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24466, 2022-09-10T04:28:24.346 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:28:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:28:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:28:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 86.130786603999994, 2022-09-10T04:28:24.347 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:28:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 94, 2022-09-10T04:28:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:28:24.348 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 149.83329746300001 2022-09-10T04:28:24.348 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:28:24.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6260099090 msgr2=0x7f626009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6260099090 0x7f626009b540 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f626400b4f0 tx=0x7f6264009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.350 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 msgr2=0x7f6274195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274195920 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f627000e710 tx=0x7f627000ea20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 shutdown_connections 2022-09-10T04:28:24.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6260099090 0x7f626009b540 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6274105a50 0x7f627419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.351 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62740716b0 0x7f6274195920 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 --2- 172.21.15.81:0/1051066338 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6274072bf0 0x7f6274195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 >> 172.21.15.81:0/1051066338 conn(0x7f62740f9230 msgr2=0x7f62740fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 shutdown_connections 2022-09-10T04:28:24.352 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.311+0000 7f627a19d700 1 -- 172.21.15.81:0/1051066338 wait complete. 2022-09-10T04:28:24.353 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2022-09-10T04:28:24.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 -- 172.21.15.81:0/2269569948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 msgr2=0x7f2334100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.581 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 --2- 172.21.15.81:0/2269569948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334100a20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2324009810 tx=0x7f2324009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 -- 172.21.15.81:0/2269569948 shutdown_connections 2022-09-10T04:28:24.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 --2- 172.21.15.81:0/2269569948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2334103880 0x7f2334105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.582 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 --2- 172.21.15.81:0/2269569948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334100a20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 --2- 172.21.15.81:0/2269569948 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2334100f60 0x7f2334103340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.579+0000 7f2333fff700 1 -- 172.21.15.81:0/2269569948 >> 172.21.15.81:0/2269569948 conn(0x7f23340f8160 msgr2=0x7f23340fa570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.580+0000 7f2333fff700 1 -- 172.21.15.81:0/2269569948 shutdown_connections 2022-09-10T04:28:24.583 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.580+0000 7f2333fff700 1 -- 172.21.15.81:0/2269569948 wait complete. 2022-09-10T04:28:24.585 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 Processor -- start 2022-09-10T04:28:24.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 -- start start 2022-09-10T04:28:24.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2334100f60 0x7f2334195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2334103880 0x7f233419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233419d4b0 con 0x7f23340fe640 2022-09-10T04:28:24.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233419d620 con 0x7f2334100f60 2022-09-10T04:28:24.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2333fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233419d7e0 con 0x7f2334103880 2022-09-10T04:28:24.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.581+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56220/0 (socket says 172.21.15.81:56220) 2022-09-10T04:28:24.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 -- 172.21.15.81:0/3298296175 learned_addr learned my addr 172.21.15.81:0/3298296175 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f23337fe700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2334103880 0x7f233419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f23327fc700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2334100f60 0x7f2334195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 -- 172.21.15.81:0/3298296175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2334103880 msgr2=0x7f233419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2334103880 0x7f233419cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 -- 172.21.15.81:0/3298296175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2334100f60 msgr2=0x7f2334195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2334100f60 0x7f2334195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 -- 172.21.15.81:0/3298296175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2324009470 con 0x7f23340fe640 2022-09-10T04:28:24.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.582+0000 7f2332ffd700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334195930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f232400f690 tx=0x7f232400f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.583+0000 7f231bfff700 1 -- 172.21.15.81:0/3298296175 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232401d070 con 0x7f23340fe640 2022-09-10T04:28:24.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.583+0000 7f231bfff700 1 -- 172.21.15.81:0/3298296175 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f232400f8e0 con 0x7f23340fe640 2022-09-10T04:28:24.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.583+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233419da60 con 0x7f23340fe640 2022-09-10T04:28:24.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.583+0000 7f231bfff700 1 -- 172.21.15.81:0/3298296175 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23240047f0 con 0x7f23340fe640 2022-09-10T04:28:24.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.583+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233419df10 con 0x7f23340fe640 2022-09-10T04:28:24.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.586+0000 7f231bfff700 1 -- 172.21.15.81:0/3298296175 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f2324028030 con 0x7f23340fe640 2022-09-10T04:28:24.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.587+0000 7f231bfff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f231c099010 0x7f231c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.587+0000 7f231bfff700 1 -- 172.21.15.81:0/3298296175 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f23240c33c0 con 0x7f23340fe640 2022-09-10T04:28:24.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.587+0000 7f234cc9ab80 1 Processor -- start 2022-09-10T04:28:24.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.587+0000 7f23327fc700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f231c099010 0x7f231c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 -- start start 2022-09-10T04:28:24.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 0x5636bd0f0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0950 0x5636bd0eebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0ef1c0 0x5636bd0ef630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcfd9080 con 0x5636bd0ef1c0 2022-09-10T04:28:24.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd0439e0 con 0x5636bd0f0950 2022-09-10T04:28:24.610 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd03d180 con 0x5636bd0f0000 2022-09-10T04:28:24.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 0x5636bd0f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f23327fc700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f231c099010 0x7f231c09b4c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2328005fd0 tx=0x7f2328005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 0x5636bd0f0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39620/0 (socket says 172.21.15.81:39620) 2022-09-10T04:28:24.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f2332ffd700 1 -- 172.21.15.81:0/1414795951 learned_addr learned my addr 172.21.15.81:0/1414795951 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.611 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f23337fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0ef1c0 0x5636bd0ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.588+0000 7f23327fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0950 0x5636bd0eebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f2332ffd700 1 -- 172.21.15.81:0/1414795951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0950 msgr2=0x5636bd0eebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f2332ffd700 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0950 0x5636bd0eebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.612 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f2332ffd700 1 -- 172.21.15.81:0/1414795951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0ef1c0 msgr2=0x5636bd0ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f2332ffd700 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0ef1c0 0x5636bd0ef630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f2332ffd700 1 -- 172.21.15.81:0/1414795951 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bcec55e0 con 0x5636bd0f0000 2022-09-10T04:28:24.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f2332ffd700 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 0x5636bd0f0410 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23240c0fd0 tx=0x7f2324009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fe15f904b0e5cb0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f23197fa700 1 -- 172.21.15.81:0/1414795951 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324093ac0 con 0x5636bd0f0000 2022-09-10T04:28:24.613 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f23197fa700 1 -- 172.21.15.81:0/1414795951 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f23240933c0 con 0x5636bd0f0000 2022-09-10T04:28:24.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.589+0000 7f23197fa700 1 -- 172.21.15.81:0/1414795951 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324093620 con 0x5636bd0f0000 2022-09-10T04:28:24.614 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1414795951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 msgr2=0x5636bd0f0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 0x5636bd0f0410 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23240c0fd0 tx=0x7f2324009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1414795951 shutdown_connections 2022-09-10T04:28:24.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0ef1c0 0x5636bd0ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0000 0x5636bd0f0410 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1414795951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0950 0x5636bd0eebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1414795951 >> 172.21.15.81:0/1414795951 conn(0x5636bd0eb170 msgr2=0x5636bd0e95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1414795951 shutdown_connections 2022-09-10T04:28:24.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.590+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1414795951 wait complete. 2022-09-10T04:28:24.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.591+0000 7f234cc9ab80 1 -- start start 2022-09-10T04:28:24.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.591+0000 7f234cc9ab80 1 Processor -- start 2022-09-10T04:28:24.619 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.619 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0000 0x5636bd0f87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.619 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0950 0x5636bd0ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.620 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcf2b0c0 con 0x5636bd0f0950 2022-09-10T04:28:24.620 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd0ffd50 con 0x5636bd0f0000 2022-09-10T04:28:24.621 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd0ffee0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.621 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f234cc9ab80 10 client.0 _ll_register_callbacks cb 0x5636bcfe8370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:24.622 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f23337fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0950 0x5636bd0ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.622 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.623 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.592+0000 7f23327fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0000 0x5636bd0f87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.623 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39634/0 (socket says 172.21.15.81:39634) 2022-09-10T04:28:24.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 -- 172.21.15.81:0/1754728075 learned_addr learned my addr 172.21.15.81:0/1754728075 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 -- 172.21.15.81:0/1754728075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0000 msgr2=0x5636bd0f87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.624 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0000 0x5636bd0f87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 -- 172.21.15.81:0/1754728075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0950 msgr2=0x5636bd0ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.625 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0950 0x5636bd0ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 -- 172.21.15.81:0/1754728075 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233419df10 con 0x5636bd0ef1c0 2022-09-10T04:28:24.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f2332ffd700 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f8290 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2324093350 tx=0x7f2324009f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e7fede7a7e9c88 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.626 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f23127fc700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:28:24.627 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f23127fc700 1 -- 172.21.15.81:0/1754728075 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324093ac0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.627 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f23127fc700 1 -- 172.21.15.81:0/1754728075 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f232409a660 con 0x5636bd0ef1c0 2022-09-10T04:28:24.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.593+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636bd0e1cf0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.628 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f23127fc700 1 -- 172.21.15.81:0/1754728075 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2324093830 con 0x5636bd0ef1c0 2022-09-10T04:28:24.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 10 client.25297 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:24.629 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5636bce71d90 con 0x5636bd0ef1c0 2022-09-10T04:28:24.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f23127fc700 1 -- 172.21.15.81:0/1754728075 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f232405f020 con 0x5636bd0ef1c0 2022-09-10T04:28:24.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f23127fc700 1 -- 172.21.15.81:0/1754728075 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f232408f1b0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.630 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 10 client.25297 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:24.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 10 client.25297 resolve_mds: resolved name 'c' to daemon mds.c 2022-09-10T04:28:24.631 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 20 client.25297 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:24.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd103090 0x5636bd1018d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.632 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 4 client.25297 mds_command: new command op to 24532 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:28:24.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.594+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 --> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636bd0be4c0 con 0x5636bd103090 2022-09-10T04:28:24.633 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.595+0000 7f23327fc700 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd103090 0x5636bd1018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.634 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.595+0000 7f23327fc700 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd103090 0x5636bd1018d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ddc08d0321d50177 server_cookie=be6986dd0a872509 in_seq=0 out_seq=0 2022-09-10T04:28:24.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.595+0000 7f23127fc700 10 client.25297 ms_handle_connect on v2:172.21.15.110:6836/1467999026 2022-09-10T04:28:24.635 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.596+0000 7f23127fc700 1 -- 172.21.15.81:0/1754728075 <== mds.3 v2:172.21.15.110:6836/1467999026 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5636bd0be4c0 con 0x5636bd103090 2022-09-10T04:28:24.636 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.596+0000 7f23127fc700 10 client.25297 handle_command_reply: tid=0 2022-09-10T04:28:24.636 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.596+0000 7f234cc9ab80 1 client.25297 shutdown 2022-09-10T04:28:24.637 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.596+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 msgr2=0x5636bd0f8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.637 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.596+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f8290 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2324093350 tx=0x7f2324009f20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.637 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 shutdown_connections 2022-09-10T04:28:24.638 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd103090 0x5636bd1018d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.638 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0950 0x5636bd0ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.639 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f23127fc700 0 client.25297 ms_handle_reset on v2:172.21.15.110:6836/1467999026 2022-09-10T04:28:24.639 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f8290 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.640 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/1754728075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0f0000 0x5636bd0f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.640 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 >> 172.21.15.81:0/1754728075 conn(0x5636bd0eb170 msgr2=0x5636bd0e8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.640 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 shutdown_connections 2022-09-10T04:28:24.641 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 1 -- 172.21.15.81:0/1754728075 wait complete. 2022-09-10T04:28:24.641 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.597+0000 7f234cc9ab80 20 client.25297 trim_cache size 0 max 16384 2022-09-10T04:28:24.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.614+0000 7f234cc9ab80 1 Processor -- start 2022-09-10T04:28:24.649 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.614+0000 7f234cc9ab80 1 -- start start 2022-09-10T04:28:24.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd0f8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd0f8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.650 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd0ea440 con 0x5636bd0f0000 2022-09-10T04:28:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd0f9480 con 0x5636bd0ef1c0 2022-09-10T04:28:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bcf4fed0 con 0x5636bd0f0950 2022-09-10T04:28:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd0f8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd0f8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56232/0 (socket says 172.21.15.81:56232) 2022-09-10T04:28:24.651 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 -- 172.21.15.81:0/399020016 learned_addr learned my addr 172.21.15.81:0/399020016 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.652 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23337fe700 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd0f8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.652 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f2332ffd700 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.652 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 -- 172.21.15.81:0/399020016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 msgr2=0x5636bd0f8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd0f8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 -- 172.21.15.81:0/399020016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 msgr2=0x5636bd0f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.653 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f23327fc700 1 -- 172.21.15.81:0/399020016 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636bce70260 con 0x5636bd0f0000 2022-09-10T04:28:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.615+0000 7f2332ffd700 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:28:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.616+0000 7f23327fc700 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd0f8040 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f23280098f0 tx=0x7f2328014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e425e576d763b3e9 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.654 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.616+0000 7f2319ffb700 1 -- 172.21.15.81:0/399020016 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328022070 con 0x5636bd0f0000 2022-09-10T04:28:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.616+0000 7f2319ffb700 1 -- 172.21.15.81:0/399020016 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2328023040 con 0x5636bd0f0000 2022-09-10T04:28:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.616+0000 7f2319ffb700 1 -- 172.21.15.81:0/399020016 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232802b740 con 0x5636bd0f0000 2022-09-10T04:28:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 -- 172.21.15.81:0/399020016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 msgr2=0x5636bd0f8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd0f8040 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f23280098f0 tx=0x7f2328014000 comp rx=0 tx=0).stop 2022-09-10T04:28:24.655 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 -- 172.21.15.81:0/399020016 shutdown_connections 2022-09-10T04:28:24.656 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd0f8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.656 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd0f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.656 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/399020016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd0f8040 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.656 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 -- 172.21.15.81:0/399020016 >> 172.21.15.81:0/399020016 conn(0x5636bd0eb170 msgr2=0x5636bd0ef590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 -- 172.21.15.81:0/399020016 shutdown_connections 2022-09-10T04:28:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 -- 172.21.15.81:0/399020016 wait complete. 2022-09-10T04:28:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 -- start start 2022-09-10T04:28:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.617+0000 7f234cc9ab80 1 Processor -- start 2022-09-10T04:28:24.657 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.658 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd1098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.658 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd10c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.658 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd10ce00 con 0x5636bd0f0000 2022-09-10T04:28:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd10cf70 con 0x5636bd0ef1c0 2022-09-10T04:28:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636bd10d130 con 0x5636bd0f0950 2022-09-10T04:28:24.659 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f234cc9ab80 10 client.0 _ll_register_callbacks cb 0x5636bcfe8370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:24.660 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.660 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51964/0 (socket says 172.21.15.81:51964) 2022-09-10T04:28:24.660 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 -- 172.21.15.81:0/4287111783 learned_addr learned my addr 172.21.15.81:0/4287111783 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.660 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f23327fc700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd1098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.661 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f23337fe700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd10c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.661 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 -- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 msgr2=0x5636bd10c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.661 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd10c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.661 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 -- 172.21.15.81:0/4287111783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 msgr2=0x5636bd1098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.661 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.618+0000 7f2332ffd700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd1098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.662 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f2332ffd700 1 -- 172.21.15.81:0/4287111783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2328017070 con 0x5636bd0ef1c0 2022-09-10T04:28:24.662 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f2332ffd700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd109360 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2324005b50 tx=0x7f2324009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bdb83b0a1d864dd server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.662 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f23127fc700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:28:24.662 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f23127fc700 1 -- 172.21.15.81:0/4287111783 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2324093ac0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f23127fc700 1 -- 172.21.15.81:0/4287111783 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f2324060080 con 0x5636bd0ef1c0 2022-09-10T04:28:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23240933c0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f23127fc700 1 -- 172.21.15.81:0/4287111783 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23240c0cb0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.663 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f234cc9ab80 10 client.25295 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5636bcf3ea30 con 0x5636bd0ef1c0 2022-09-10T04:28:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.619+0000 7f23127fc700 1 -- 172.21.15.81:0/4287111783 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f23240933c0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f23127fc700 1 -- 172.21.15.81:0/4287111783 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f232409a1d0 con 0x5636bd0ef1c0 2022-09-10T04:28:24.664 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f234cc9ab80 10 client.25295 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f234cc9ab80 10 client.25295 resolve_mds: resolved name 'c' to daemon mds.c 2022-09-10T04:28:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f234cc9ab80 20 client.25295 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd159ec0 0x5636bd156a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f234cc9ab80 4 client.25295 mds_command: new command op to 24532 tid=0[{"prefix": "status"}] 2022-09-10T04:28:24.665 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 --> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5636bd0366f0 con 0x5636bd159ec0 2022-09-10T04:28:24.666 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.620+0000 7f23327fc700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd159ec0 0x5636bd156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.666 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.621+0000 7f23327fc700 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd159ec0 0x5636bd156a00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=25e362ce4f4aa3b5 server_cookie=9925b84a9237cdab in_seq=0 out_seq=0 2022-09-10T04:28:24.667 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.621+0000 7f23127fc700 10 client.25295 ms_handle_connect on v2:172.21.15.110:6836/1467999026 2022-09-10T04:28:24.667 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.621+0000 7f23127fc700 1 -- 172.21.15.81:0/4287111783 <== mds.3 v2:172.21.15.110:6836/1467999026 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5636bd0366f0 con 0x5636bd159ec0 2022-09-10T04:28:24.668 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.621+0000 7f23127fc700 10 client.25295 handle_command_reply: tid=0 2022-09-10T04:28:24.668 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.621+0000 7f234cc9ab80 1 client.25295 shutdown 2022-09-10T04:28:24.669 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 msgr2=0x5636bd109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.669 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd109360 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2324005b50 tx=0x7f2324009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.669 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 shutdown_connections 2022-09-10T04:28:24.670 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5636bd0f0950 0x5636bd10c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.670 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5636bd0ef1c0 0x5636bd109360 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.671 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026] conn(0x5636bd159ec0 0x5636bd156a00 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.671 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 --2- 172.21.15.81:0/4287111783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5636bd0f0000 0x5636bd1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.672 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f23127fc700 0 client.25295 ms_handle_reset on v2:172.21.15.110:6836/1467999026 2022-09-10T04:28:24.673 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 >> 172.21.15.81:0/4287111783 conn(0x5636bd0eb170 msgr2=0x5636bd0f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.673 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 shutdown_connections 2022-09-10T04:28:24.674 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 1 -- 172.21.15.81:0/4287111783 wait complete. 2022-09-10T04:28:24.674 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.622+0000 7f234cc9ab80 20 client.25295 trim_cache size 0 max 16384 2022-09-10T04:28:24.675 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:28:24.676 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:28:24.676 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 3, 2022-09-10T04:28:24.677 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24532, 2022-09-10T04:28:24.677 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:28:24.678 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:28:24.678 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:28:24.679 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 87.162231706, 2022-09-10T04:28:24.679 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:28:24.680 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 94, 2022-09-10T04:28:24.680 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:28:24.680 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 134.19719973700001 2022-09-10T04:28:24.681 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:28:24.683 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f231c099010 msgr2=0x7f231c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.684 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f231c099010 0x7f231c09b4c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2328005fd0 tx=0x7f2328005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.684 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 msgr2=0x7f2334195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334195930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f232400f690 tx=0x7f232400f6c0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 shutdown_connections 2022-09-10T04:28:24.685 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f231c099010 0x7f231c09b4c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2334103880 0x7f233419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.686 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23340fe640 0x7f2334195930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 --2- 172.21.15.81:0/3298296175 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2334100f60 0x7f2334195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 >> 172.21.15.81:0/3298296175 conn(0x7f23340f8160 msgr2=0x7f23340fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.687 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 shutdown_connections 2022-09-10T04:28:24.688 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.623+0000 7f2333fff700 1 -- 172.21.15.81:0/3298296175 wait complete. 2022-09-10T04:28:24.688 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:24 smithi081 ceph-mon[90133]: pgmap v361: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:24.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:24 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/88476424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:28:24.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:24 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/4000341459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:24.691 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2022-09-10T04:28:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:24 smithi104 ceph-mon[97044]: pgmap v361: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:24 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/88476424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:28:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:24 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/4000341459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:24 smithi110 ceph-mon[97633]: pgmap v361: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T04:28:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:24 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/88476424' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2022-09-10T04:28:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:24 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/4000341459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2022-09-10T04:28:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/1050304139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 msgr2=0x7f2a980fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/1050304139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a980fe110 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009810 tx=0x7f2a88009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.919 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/1050304139 shutdown_connections 2022-09-10T04:28:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/1050304139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a980ff910 0x7f2a981061d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/1050304139 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a980fe110 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/1050304139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a980fef80 0x7f2a980ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.920 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/1050304139 >> 172.21.15.81:0/1050304139 conn(0x7f2a980f93f0 msgr2=0x7f2a980fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/1050304139 shutdown_connections 2022-09-10T04:28:24.921 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.917+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/1050304139 wait complete. 2022-09-10T04:28:24.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.918+0000 7f2a9e3ff700 1 Processor -- start 2022-09-10T04:28:24.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 -- start start 2022-09-10T04:28:24.923 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a98195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a980fef80 0x7f2a981960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a980ff910 0x7f2a9819d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9819d710 con 0x7f2a980fef80 2022-09-10T04:28:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9819d880 con 0x7f2a980ff910 2022-09-10T04:28:24.924 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9e3ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9819da40 con 0x7f2a980fdd00 2022-09-10T04:28:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a98195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a977fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a980fef80 0x7f2a981960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a98195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39660/0 (socket says 172.21.15.81:39660) 2022-09-10T04:28:24.925 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a97fff700 1 -- 172.21.15.81:0/96271156 learned_addr learned my addr 172.21.15.81:0/96271156 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.919+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a980ff910 0x7f2a9819d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a97fff700 1 -- 172.21.15.81:0/96271156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a980ff910 msgr2=0x7f2a9819d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a97fff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a980ff910 0x7f2a9819d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a97fff700 1 -- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a980fef80 msgr2=0x7f2a981960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.926 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a97fff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a980fef80 0x7f2a981960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a97fff700 1 -- 172.21.15.81:0/96271156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a88009470 con 0x7f2a980fdd00 2022-09-10T04:28:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a97fff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a98195b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a88006010 tx=0x7f2a8800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a957fa700 1 -- 172.21.15.81:0/96271156 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8801d070 con 0x7f2a980fdd00 2022-09-10T04:28:24.927 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a957fa700 1 -- 172.21.15.81:0/96271156 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2a8800f8b0 con 0x7f2a980fdd00 2022-09-10T04:28:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a9819dcc0 con 0x7f2a980fdd00 2022-09-10T04:28:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.920+0000 7f2a957fa700 1 -- 172.21.15.81:0/96271156 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880047f0 con 0x7f2a980fdd00 2022-09-10T04:28:24.928 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.921+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9819e170 con 0x7f2a980fdd00 2022-09-10T04:28:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.924+0000 7f2a957fa700 1 -- 172.21.15.81:0/96271156 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f2a88028030 con 0x7f2a980fdd00 2022-09-10T04:28:24.934 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.925+0000 7f2ab19cab80 1 Processor -- start 2022-09-10T04:28:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.925+0000 7f2a957fa700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a80099010 0x7f2a8009b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.925+0000 7f2ab19cab80 1 -- start start 2022-09-10T04:28:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.925+0000 7f2a957fa700 1 -- 172.21.15.81:0/96271156 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f2a880c33a0 con 0x7f2a980fdd00 2022-09-10T04:28:24.935 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.925+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a80099010 0x7f2a8009b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c33628ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33629430 0x561c33629c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c3362a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.936 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33513080 con 0x561c3362a1f0 2022-09-10T04:28:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3357d9e0 con 0x561c33629430 2022-09-10T04:28:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33577180 con 0x561c33628ae0 2022-09-10T04:28:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c33628ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c33628ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39676/0 (socket says 172.21.15.81:39676) 2022-09-10T04:28:24.937 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2a97fff700 1 -- 172.21.15.81:0/1738514307 learned_addr learned my addr 172.21.15.81:0/1738514307 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2a977fe700 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33629430 0x561c33629c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.926+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c3362a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a97fff700 1 -- 172.21.15.81:0/1738514307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33629430 msgr2=0x561c33629c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.938 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a97fff700 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33629430 0x561c33629c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a97fff700 1 -- 172.21.15.81:0/1738514307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 msgr2=0x561c3362a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a97fff700 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c3362a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a97fff700 1 -- 172.21.15.81:0/1738514307 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c333ff5e0 con 0x561c33628ae0 2022-09-10T04:28:24.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c3362a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:24.939 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a80099010 0x7f2a8009b4c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2a980fdab0 tx=0x7f2a90011720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a97fff700 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c33628ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a88000c00 tx=0x7f2a88005f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=928c3ba89f12da12 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a7a7fc700 1 -- 172.21.15.81:0/1738514307 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880c3160 con 0x561c33628ae0 2022-09-10T04:28:24.940 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a7a7fc700 1 -- 172.21.15.81:0/1738514307 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2a880933a0 con 0x561c33628ae0 2022-09-10T04:28:24.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.927+0000 7f2a7a7fc700 1 -- 172.21.15.81:0/1738514307 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a88093800 con 0x561c33628ae0 2022-09-10T04:28:24.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 -- 172.21.15.81:0/1738514307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 msgr2=0x561c33628ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c33628ef0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a88000c00 tx=0x7f2a88005f60 comp rx=0 tx=0).stop 2022-09-10T04:28:24.941 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 -- 172.21.15.81:0/1738514307 shutdown_connections 2022-09-10T04:28:24.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c3362a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c33628ef0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/1738514307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33629430 0x561c33629c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.942 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 -- 172.21.15.81:0/1738514307 >> 172.21.15.81:0/1738514307 conn(0x561c33624fa0 msgr2=0x561c33623370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 -- 172.21.15.81:0/1738514307 shutdown_connections 2022-09-10T04:28:24.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.928+0000 7f2ab19cab80 1 -- 172.21.15.81:0/1738514307 wait complete. 2022-09-10T04:28:24.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 -- start start 2022-09-10T04:28:24.943 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 Processor -- start 2022-09-10T04:28:24.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33632190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33629430 0x561c336326d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c33639710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c334650c0 con 0x561c3362a1f0 2022-09-10T04:28:24.944 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33639c50 con 0x561c33628ae0 2022-09-10T04:28:24.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.929+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33639de0 con 0x561c33629430 2022-09-10T04:28:24.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2ab19cab80 10 client.0 _ll_register_callbacks cb 0x561c33522370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:24.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a9c99c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c33639710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33632190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.945 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a9c99c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c33639710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56282/0 (socket says 172.21.15.81:56282) 2022-09-10T04:28:24.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a9c99c700 1 -- 172.21.15.81:0/3170517739 learned_addr learned my addr 172.21.15.81:0/3170517739 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a977fe700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33629430 0x561c336326d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a97fff700 1 -- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33629430 msgr2=0x561c336326d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.946 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33629430 0x561c336326d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a97fff700 1 -- 172.21.15.81:0/3170517739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 msgr2=0x561c33639710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c33639710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a97fff700 1 -- 172.21.15.81:0/3170517739 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a90011c50 con 0x561c33628ae0 2022-09-10T04:28:24.947 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.930+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c33639710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:24.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33632190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800fb70 tx=0x7f2a88092c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e7d9c78d762eff8 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2a737fe700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:28:24.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2a737fe700 1 -- 172.21.15.81:0/3170517739 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880c3160 con 0x561c33628ae0 2022-09-10T04:28:24.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2a737fe700 1 -- 172.21.15.81:0/3170517739 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f2a880aa370 con 0x561c33628ae0 2022-09-10T04:28:24.948 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c3361baa0 con 0x561c33628ae0 2022-09-10T04:28:24.949 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2a737fe700 1 -- 172.21.15.81:0/3170517739 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a8809a640 con 0x561c33628ae0 2022-09-10T04:28:24.949 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2ab19cab80 10 client.25307 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:24.949 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f2a9819e170 con 0x561c33628ae0 2022-09-10T04:28:24.949 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.931+0000 7f2a737fe700 1 -- 172.21.15.81:0/3170517739 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2a8805f020 con 0x561c33628ae0 2022-09-10T04:28:24.949 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2a737fe700 1 -- 172.21.15.81:0/3170517739 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f2a8808f190 con 0x561c33628ae0 2022-09-10T04:28:24.950 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2ab19cab80 10 client.25307 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:24.950 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2ab19cab80 10 client.25307 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-10T04:28:24.950 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2ab19cab80 20 client.25307 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:24.950 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c3363ce40 0x561c3363b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.951 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2ab19cab80 4 client.25307 mds_command: new command op to 24547 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:28:24.951 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 --> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c335f84c0 con 0x561c3363ce40 2022-09-10T04:28:24.951 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.932+0000 7f2a977fe700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c3363ce40 0x561c3363b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.951 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.933+0000 7f2a977fe700 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c3363ce40 0x561c3363b7d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9a14a19a98ab7166 server_cookie=b1594adb7520b90e in_seq=0 out_seq=0 2022-09-10T04:28:24.951 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.933+0000 7f2a737fe700 10 client.25307 ms_handle_connect on v2:172.21.15.110:6838/1670334720 2022-09-10T04:28:24.953 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2a737fe700 1 -- 172.21.15.81:0/3170517739 <== mds.2 v2:172.21.15.110:6838/1670334720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x561c335f84c0 con 0x561c3363ce40 2022-09-10T04:28:24.953 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2a737fe700 10 client.25307 handle_command_reply: tid=0 2022-09-10T04:28:24.954 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 client.25307 shutdown 2022-09-10T04:28:24.954 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 msgr2=0x561c33632190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.954 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33632190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800fb70 tx=0x7f2a88092c20 comp rx=0 tx=0).stop 2022-09-10T04:28:24.954 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 shutdown_connections 2022-09-10T04:28:24.955 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c3363ce40 0x561c3363b7d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.955 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c3362a1f0 0x561c33639710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.955 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2a737fe700 0 client.25307 ms_handle_reset on v2:172.21.15.110:6838/1670334720 2022-09-10T04:28:24.955 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33632190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.956 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3170517739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33629430 0x561c336326d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.956 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 >> 172.21.15.81:0/3170517739 conn(0x561c33624fa0 msgr2=0x561c33622a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.956 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.934+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 shutdown_connections 2022-09-10T04:28:24.956 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.935+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3170517739 wait complete. 2022-09-10T04:28:24.957 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.935+0000 7f2ab19cab80 20 client.25307 trim_cache size 0 max 16384 2022-09-10T04:28:24.957 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.952+0000 7f2ab19cab80 1 Processor -- start 2022-09-10T04:28:24.957 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.952+0000 7f2ab19cab80 1 -- start start 2022-09-10T04:28:24.962 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c336302c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.963 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c33630800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.963 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c3362a1f0 0x561c33631f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.963 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c335f84c0 con 0x561c33629430 2022-09-10T04:28:24.963 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33489ed0 con 0x561c3362a1f0 2022-09-10T04:28:24.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3353b050 con 0x561c33628ae0 2022-09-10T04:28:24.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a977fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c33630800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c336302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a977fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c33630800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56286/0 (socket says 172.21.15.81:56286) 2022-09-10T04:28:24.964 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a9c99c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c3362a1f0 0x561c33631f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.965 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a977fe700 1 -- 172.21.15.81:0/3082384861 learned_addr learned my addr 172.21.15.81:0/3082384861 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.965 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a97fff700 1 -- 172.21.15.81:0/3082384861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c3362a1f0 msgr2=0x561c33631f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.965 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c3362a1f0 0x561c33631f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.965 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a97fff700 1 -- 172.21.15.81:0/3082384861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 msgr2=0x561c33630800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.966 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c33630800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.966 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.953+0000 7f2a97fff700 1 -- 172.21.15.81:0/3082384861 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c333abd90 con 0x561c33628ae0 2022-09-10T04:28:24.966 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c336302c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a880916f0 tx=0x7f2a88005f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11faabcd786025ff server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.966 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2a7affd700 1 -- 172.21.15.81:0/3082384861 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880c3160 con 0x561c33628ae0 2022-09-10T04:28:24.967 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2a7affd700 1 -- 172.21.15.81:0/3082384861 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2a8805f070 con 0x561c33628ae0 2022-09-10T04:28:24.967 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2a7affd700 1 -- 172.21.15.81:0/3082384861 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880aa3d0 con 0x561c33628ae0 2022-09-10T04:28:24.967 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3082384861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 msgr2=0x561c336302c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.967 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c336302c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2a880916f0 tx=0x7f2a88005f60 comp rx=0 tx=0).stop 2022-09-10T04:28:24.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3082384861 shutdown_connections 2022-09-10T04:28:24.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c3362a1f0 0x561c33631f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c33628ae0 0x561c336302c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3082384861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c33630800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.968 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.954+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3082384861 >> 172.21.15.81:0/3082384861 conn(0x561c33624fa0 msgr2=0x561c3362ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.955+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3082384861 shutdown_connections 2022-09-10T04:28:24.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.955+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3082384861 wait complete. 2022-09-10T04:28:24.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.955+0000 7f2ab19cab80 1 -- start start 2022-09-10T04:28:24.969 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.955+0000 7f2ab19cab80 1 Processor -- start 2022-09-10T04:28:24.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33642db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c336432f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c3362a1f0 0x561c33646260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c336467a0 con 0x561c33629430 2022-09-10T04:28:24.970 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33646910 con 0x561c33628ae0 2022-09-10T04:28:24.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c33646ad0 con 0x561c3362a1f0 2022-09-10T04:28:24.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2ab19cab80 10 client.0 _ll_register_callbacks cb 0x561c33522370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:24.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2a977fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c336432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.971 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2a97fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33642db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2a977fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c336432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56298/0 (socket says 172.21.15.81:56298) 2022-09-10T04:28:24.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2a977fe700 1 -- 172.21.15.81:0/3679367841 learned_addr learned my addr 172.21.15.81:0/3679367841 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:24.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.956+0000 7f2a9c99c700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c3362a1f0 0x561c33646260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.972 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a977fe700 1 -- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c3362a1f0 msgr2=0x561c33646260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a977fe700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c3362a1f0 0x561c33646260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a977fe700 1 -- 172.21.15.81:0/3679367841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 msgr2=0x561c33642db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:24.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a977fe700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33642db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a977fe700 1 -- 172.21.15.81:0/3679367841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a9819e170 con 0x561c33629430 2022-09-10T04:28:24.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33642db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:24.974 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a977fe700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c336432f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009fc0 tx=0x7f2a8c01f9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e307e9eaad16e96 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:24.974 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a737fe700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:28:24.974 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a737fe700 1 -- 172.21.15.81:0/3679367841 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c017040 con 0x561c33629430 2022-09-10T04:28:24.974 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a880933a0 con 0x561c33629430 2022-09-10T04:28:24.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.957+0000 7f2a737fe700 1 -- 172.21.15.81:0/3679367841 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f2a8c005220 con 0x561c33629430 2022-09-10T04:28:24.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2a737fe700 1 -- 172.21.15.81:0/3679367841 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a8c0053e0 con 0x561c33629430 2022-09-10T04:28:24.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 10 client.15210 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:24.975 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561c333aa260 con 0x561c33629430 2022-09-10T04:28:24.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2a737fe700 1 -- 172.21.15.81:0/3679367841 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2a880933a0 con 0x561c33629430 2022-09-10T04:28:24.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2a737fe700 1 -- 172.21.15.81:0/3679367841 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f2a8c010a10 con 0x561c33629430 2022-09-10T04:28:24.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 10 client.15210 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:24.976 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 10 client.15210 resolve_mds: resolved name 'i' to daemon mds.i 2022-09-10T04:28:24.977 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 20 client.15210 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:24.977 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c33693ff0 0x561c33690960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:24.977 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 4 client.15210 mds_command: new command op to 24547 tid=0[{"prefix": "status"}] 2022-09-10T04:28:24.977 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 --> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561c335706f0 con 0x561c33693ff0 2022-09-10T04:28:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.958+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c33693ff0 0x561c33690960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2a97fff700 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c33693ff0 0x561c33690960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d63b4baa44df8b06 server_cookie=6ec496be9df8c86e in_seq=0 out_seq=0 2022-09-10T04:28:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2a737fe700 10 client.15210 ms_handle_connect on v2:172.21.15.110:6838/1670334720 2022-09-10T04:28:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2a737fe700 1 -- 172.21.15.81:0/3679367841 <== mds.2 v2:172.21.15.110:6838/1670334720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x561c335706f0 con 0x561c33693ff0 2022-09-10T04:28:24.978 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2a737fe700 10 client.15210 handle_command_reply: tid=0 2022-09-10T04:28:24.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 client.15210 shutdown 2022-09-10T04:28:24.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 msgr2=0x561c336432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c336432f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c009fc0 tx=0x7f2a8c01f9b0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.979 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 shutdown_connections 2022-09-10T04:28:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:6838/1670334720,v1:172.21.15.110:6839/1670334720] conn(0x561c33693ff0 0x561c33690960 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x561c3362a1f0 0x561c33646260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x561c33628ae0 0x561c33642db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2a737fe700 0 client.15210 ms_handle_reset on v2:172.21.15.110:6838/1670334720 2022-09-10T04:28:24.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 --2- 172.21.15.81:0/3679367841 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561c33629430 0x561c336432f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.959+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 >> 172.21.15.81:0/3679367841 conn(0x561c33624fa0 msgr2=0x561c336302c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 shutdown_connections 2022-09-10T04:28:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2ab19cab80 1 -- 172.21.15.81:0/3679367841 wait complete. 2022-09-10T04:28:24.981 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2ab19cab80 20 client.15210 trim_cache size 0 max 16384 2022-09-10T04:28:24.982 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:28:24.982 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:28:24.982 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 2, 2022-09-10T04:28:24.983 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24547, 2022-09-10T04:28:24.983 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:28:24.983 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:28:24.983 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:28:24.984 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 88.509199538000004, 2022-09-10T04:28:24.984 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:28:24.984 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 94, 2022-09-10T04:28:24.985 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:28:24.985 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 130.16065947499999 2022-09-10T04:28:24.985 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:28:24.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a80099010 msgr2=0x7f2a8009b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a80099010 0x7f2a8009b4c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2a980fdab0 tx=0x7f2a90011720 comp rx=0 tx=0).stop 2022-09-10T04:28:24.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 msgr2=0x7f2a98195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:24.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a98195b90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2a88006010 tx=0x7f2a8800f690 comp rx=0 tx=0).stop 2022-09-10T04:28:24.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 shutdown_connections 2022-09-10T04:28:24.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2a980ff910 0x7f2a9819d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f2a80099010 0x7f2a8009b4c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.988 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a980fdd00 0x7f2a98195b90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.989 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 --2- 172.21.15.81:0/96271156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a980fef80 0x7f2a981960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:24.989 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.960+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 >> 172.21.15.81:0/96271156 conn(0x7f2a980f93f0 msgr2=0x7f2a980fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:24.989 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.961+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 shutdown_connections 2022-09-10T04:28:24.990 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:24.961+0000 7f2a9e3ff700 1 -- 172.21.15.81:0/96271156 wait complete. 2022-09-10T04:28:24.990 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2022-09-10T04:28:25.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 -- 172.21.15.81:0/1887275537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 msgr2=0x7fa3301060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.218 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 --2- 172.21.15.81:0/1887275537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa3301060b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa328009b00 tx=0x7fa328009e10 comp rx=0 tx=0).stop 2022-09-10T04:28:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 -- 172.21.15.81:0/1887275537 shutdown_connections 2022-09-10T04:28:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 --2- 172.21.15.81:0/1887275537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa3301060b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 --2- 172.21.15.81:0/1887275537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa3300fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 --2- 172.21.15.81:0/1887275537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3300fe490 0x7fa3300fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.219 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 -- 172.21.15.81:0/1887275537 >> 172.21.15.81:0/1887275537 conn(0x7fa3300f9230 msgr2=0x7fa3300fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 -- 172.21.15.81:0/1887275537 shutdown_connections 2022-09-10T04:28:25.220 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.216+0000 7fa33659d700 1 -- 172.21.15.81:0/1887275537 wait complete. 2022-09-10T04:28:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.217+0000 7fa33659d700 1 Processor -- start 2022-09-10T04:28:25.222 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.217+0000 7fa33659d700 1 -- start start 2022-09-10T04:28:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33659d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa330195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33659d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3300fe490 0x7fa330195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33659d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa33019cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.223 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33659d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33019d490 con 0x7fa3300fdb40 2022-09-10T04:28:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33659d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33019d600 con 0x7fa3300ff510 2022-09-10T04:28:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33659d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33019d7c0 con 0x7fa3300fe490 2022-09-10T04:28:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa330195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.224 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa330195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56322/0 (socket says 172.21.15.81:56322) 2022-09-10T04:28:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 -- 172.21.15.81:0/3494417270 learned_addr learned my addr 172.21.15.81:0/3494417270 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa33019cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa334d9a700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3300fe490 0x7fa330195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.225 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 -- 172.21.15.81:0/3494417270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3300fe490 msgr2=0x7fa330195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3300fe490 0x7fa330195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 -- 172.21.15.81:0/3494417270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 msgr2=0x7fa33019cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa33019cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.226 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa33559b700 1 -- 172.21.15.81:0/3494417270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c0093b0 con 0x7fa3300fdb40 2022-09-10T04:28:25.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.218+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa33019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:25.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.219+0000 7fa33559b700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa330195910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e9c0 tx=0x7fa32c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.219+0000 7fa3267fc700 1 -- 172.21.15.81:0/3494417270 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0097b0 con 0x7fa3300fdb40 2022-09-10T04:28:25.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.219+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa328009730 con 0x7fa3300fdb40 2022-09-10T04:28:25.227 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.219+0000 7fa3267fc700 1 -- 172.21.15.81:0/3494417270 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa32c00ce70 con 0x7fa3300fdb40 2022-09-10T04:28:25.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.219+0000 7fa3267fc700 1 -- 172.21.15.81:0/3494417270 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c003d20 con 0x7fa3300fdb40 2022-09-10T04:28:25.228 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.219+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33019de20 con 0x7fa3300fdb40 2022-09-10T04:28:25.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.222+0000 7fa3267fc700 1 -- 172.21.15.81:0/3494417270 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7fa32c01e020 con 0x7fa3300fdb40 2022-09-10T04:28:25.235 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.224+0000 7fa34ad86b80 1 Processor -- start 2022-09-10T04:28:25.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 -- start start 2022-09-10T04:28:25.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa3267fc700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa31c099010 0x7fa31c09b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa3267fc700 1 -- 172.21.15.81:0/3494417270 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fa32c014070 con 0x7fa3300fdb40 2022-09-10T04:28:25.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980f000 0x55f2e980f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.236 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e980dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 0x55f2e980e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e96f8080 con 0x55f2e980f000 2022-09-10T04:28:25.237 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e97629e0 con 0x55f2e980e1c0 2022-09-10T04:28:25.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e975c180 con 0x55f2e980f950 2022-09-10T04:28:25.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa31c099010 0x7fa31c09b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.225+0000 7fa335d9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 0x55f2e980e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.238 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa335d9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 0x55f2e980e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52042/0 (socket says 172.21.15.81:52042) 2022-09-10T04:28:25.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa335d9c700 1 -- 172.21.15.81:0/1220205378 learned_addr learned my addr 172.21.15.81:0/1220205378 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980f000 0x55f2e980f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa334d9a700 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e980dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.239 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 -- 172.21.15.81:0/1220205378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 msgr2=0x55f2e980dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e980dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 -- 172.21.15.81:0/1220205378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 msgr2=0x55f2e980e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.240 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 0x55f2e980e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 -- 172.21.15.81:0/1220205378 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e95e45e0 con 0x55f2e980f000 2022-09-10T04:28:25.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa335d9c700 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 0x55f2e980e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:25.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa31c099010 0x7fa31c09b4c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa32800fe20 tx=0x7fa328004d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa33559b700 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980f000 0x55f2e980f410 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa32c08f560 tx=0x7fa32c08eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2094728361acd5f server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.226+0000 7fa31b7fe700 1 -- 172.21.15.81:0/1220205378 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c003e80 con 0x55f2e980f000 2022-09-10T04:28:25.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa31b7fe700 1 -- 172.21.15.81:0/1220205378 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa32c0931f0 con 0x55f2e980f000 2022-09-10T04:28:25.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa31b7fe700 1 -- 172.21.15.81:0/1220205378 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c096170 con 0x55f2e980f000 2022-09-10T04:28:25.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 -- 172.21.15.81:0/1220205378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980f000 msgr2=0x55f2e980f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980f000 0x55f2e980f410 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa32c08f560 tx=0x7fa32c08eeb0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 -- 172.21.15.81:0/1220205378 shutdown_connections 2022-09-10T04:28:25.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980e1c0 0x55f2e980e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980f000 0x55f2e980f410 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/1220205378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e980dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.243 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 -- 172.21.15.81:0/1220205378 >> 172.21.15.81:0/1220205378 conn(0x55f2e980a170 msgr2=0x55f2e98085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 -- 172.21.15.81:0/1220205378 shutdown_connections 2022-09-10T04:28:25.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.227+0000 7fa34ad86b80 1 -- 172.21.15.81:0/1220205378 wait complete. 2022-09-10T04:28:25.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.228+0000 7fa34ad86b80 1 -- start start 2022-09-10T04:28:25.244 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.228+0000 7fa34ad86b80 1 Processor -- start 2022-09-10T04:28:25.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e98176d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e9817c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e981ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e964a0c0 con 0x55f2e980e1c0 2022-09-10T04:28:25.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e981f190 con 0x55f2e980f000 2022-09-10T04:28:25.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e981f320 con 0x55f2e980f950 2022-09-10T04:28:25.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa34ad86b80 10 client.0 _ll_register_callbacks cb 0x55f2e9707370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:25.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e98176d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e98176d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56352/0 (socket says 172.21.15.81:56352) 2022-09-10T04:28:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 -- 172.21.15.81:0/2645161028 learned_addr learned my addr 172.21.15.81:0/2645161028 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa334d9a700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e9817c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa335d9c700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e981ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 -- 172.21.15.81:0/2645161028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 msgr2=0x55f2e981ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e981ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 -- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 msgr2=0x55f2e9817c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.229+0000 7fa33559b700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e9817c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa33559b700 1 -- 172.21.15.81:0/2645161028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa330106870 con 0x55f2e980e1c0 2022-09-10T04:28:25.248 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa334d9a700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e9817c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:25.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa33559b700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e98176d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa32c092a80 tx=0x7fa32c0036e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63edf7f6a8ae7f80 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa318ff9700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:28:25.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa318ff9700 1 -- 172.21.15.81:0/2645161028 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c092af0 con 0x55f2e980e1c0 2022-09-10T04:28:25.249 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa318ff9700 1 -- 172.21.15.81:0/2645161028 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fa32c099d90 con 0x55f2e980e1c0 2022-09-10T04:28:25.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2e9800cf0 con 0x55f2e980e1c0 2022-09-10T04:28:25.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa318ff9700 1 -- 172.21.15.81:0/2645161028 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa32c092d50 con 0x55f2e980e1c0 2022-09-10T04:28:25.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa34ad86b80 10 client.15222 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:25.250 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.230+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fa33019de20 con 0x55f2e980e1c0 2022-09-10T04:28:25.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa318ff9700 1 -- 172.21.15.81:0/2645161028 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa32c076020 con 0x55f2e980e1c0 2022-09-10T04:28:25.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa318ff9700 1 -- 172.21.15.81:0/2645161028 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa32c099a00 con 0x55f2e980e1c0 2022-09-10T04:28:25.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa34ad86b80 10 client.15222 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:25.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa34ad86b80 10 client.15222 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-10T04:28:25.251 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa34ad86b80 20 client.15222 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:25.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e9822680 0x55f2e9827920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa34ad86b80 4 client.15222 mds_command: new command op to 24572 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:28:25.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 --> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2e97dd4c0 con 0x55f2e9822680 2022-09-10T04:28:25.252 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.231+0000 7fa334d9a700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e9822680 0x55f2e9827920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.232+0000 7fa318ff9700 10 client.15222 ms_handle_connect on v2:172.21.15.104:6838/2357131382 2022-09-10T04:28:25.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.232+0000 7fa334d9a700 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e9822680 0x55f2e9827920 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21f9aaa7147f6f6c server_cookie=2264ede616ea1d10 in_seq=0 out_seq=0 2022-09-10T04:28:25.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa318ff9700 1 -- 172.21.15.81:0/2645161028 <== mds.1 v2:172.21.15.104:6838/2357131382 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55f2e97dd4c0 con 0x55f2e9822680 2022-09-10T04:28:25.253 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa318ff9700 10 client.15222 handle_command_reply: tid=0 2022-09-10T04:28:25.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 client.15222 shutdown 2022-09-10T04:28:25.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 msgr2=0x55f2e98176d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e98176d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa32c092a80 tx=0x7fa32c0036e0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.254 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 shutdown_connections 2022-09-10T04:28:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e9822680 0x55f2e9827920 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e981ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa318ff9700 0 client.15222 ms_handle_reset on v2:172.21.15.104:6838/2357131382 2022-09-10T04:28:25.255 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e98176d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/2645161028 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e9817c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.233+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 >> 172.21.15.81:0/2645161028 conn(0x55f2e980a170 msgr2=0x55f2e9807c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.234+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 shutdown_connections 2022-09-10T04:28:25.256 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.234+0000 7fa34ad86b80 1 -- 172.21.15.81:0/2645161028 wait complete. 2022-09-10T04:28:25.257 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.234+0000 7fa34ad86b80 20 client.15222 trim_cache size 0 max 16384 2022-09-10T04:28:25.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.251+0000 7fa34ad86b80 1 Processor -- start 2022-09-10T04:28:25.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.251+0000 7fa34ad86b80 1 -- start start 2022-09-10T04:28:25.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e9807c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.263 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e982b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e9812040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e980eca0 con 0x55f2e980e1c0 2022-09-10T04:28:25.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e966eed0 con 0x55f2e980f000 2022-09-10T04:28:25.264 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e96fe4f0 con 0x55f2e980f950 2022-09-10T04:28:25.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e9807c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa334d9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e982b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e9807c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56356/0 (socket says 172.21.15.81:56356) 2022-09-10T04:28:25.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa33559b700 1 -- 172.21.15.81:0/3583125730 learned_addr learned my addr 172.21.15.81:0/3583125730 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.265 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.252+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e9812040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa334d9a700 1 -- 172.21.15.81:0/3583125730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 msgr2=0x55f2e9812040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.266 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa334d9a700 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e9812040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa334d9a700 1 -- 172.21.15.81:0/3583125730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 msgr2=0x55f2e9807c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa334d9a700 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e9807c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa334d9a700 1 -- 172.21.15.81:0/3583125730 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e9590d90 con 0x55f2e980f000 2022-09-10T04:28:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa334d9a700 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e982b940 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55f2e9823ea0 tx=0x7fa320005f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec0fc4a44e3287ae server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.267 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa31bfff700 1 -- 172.21.15.81:0/3583125730 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320017040 con 0x55f2e980f000 2022-09-10T04:28:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa31bfff700 1 -- 172.21.15.81:0/3583125730 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa320018040 con 0x55f2e980f000 2022-09-10T04:28:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.253+0000 7fa31bfff700 1 -- 172.21.15.81:0/3583125730 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32001f650 con 0x55f2e980f000 2022-09-10T04:28:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3583125730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 msgr2=0x55f2e982b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.268 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e982b940 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55f2e9823ea0 tx=0x7fa320005f60 comp rx=0 tx=0).stop 2022-09-10T04:28:25.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3583125730 shutdown_connections 2022-09-10T04:28:25.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f950 0x55f2e9812040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e9807c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3583125730 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f000 0x55f2e982b940 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.269 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3583125730 >> 172.21.15.81:0/3583125730 conn(0x55f2e980a170 msgr2=0x55f2e980e590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3583125730 shutdown_connections 2022-09-10T04:28:25.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.254+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3583125730 wait complete. 2022-09-10T04:28:25.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 -- start start 2022-09-10T04:28:25.270 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 Processor -- start 2022-09-10T04:28:25.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e981b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 0x55f2e981b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 0x55f2e981bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e981ce10 con 0x55f2e980e1c0 2022-09-10T04:28:25.271 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e98748c0 con 0x55f2e980f950 2022-09-10T04:28:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e9874a00 con 0x55f2e980f000 2022-09-10T04:28:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.255+0000 7fa34ad86b80 10 client.0 _ll_register_callbacks cb 0x55f2e9707370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 0x55f2e981bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.272 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa334d9a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 0x55f2e981b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa33559b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e981b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 0x55f2e981bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52076/0 (socket says 172.21.15.81:52076) 2022-09-10T04:28:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa334d9a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 0x55f2e981b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39750/0 (socket says 172.21.15.81:39750) 2022-09-10T04:28:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 -- 172.21.15.81:0/3419392003 learned_addr learned my addr 172.21.15.81:0/3419392003 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.273 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 -- 172.21.15.81:0/3419392003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 msgr2=0x55f2e981b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 0x55f2e981b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 -- 172.21.15.81:0/3419392003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 msgr2=0x55f2e981b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e981b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.274 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 -- 172.21.15.81:0/3419392003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3200216e0 con 0x55f2e980f950 2022-09-10T04:28:25.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa334d9a700 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 0x55f2e981b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:25.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa335d9c700 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 0x55f2e981bd20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ad0 tx=0x7fa328009e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=264633d67cd26a10 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.256+0000 7fa318ff9700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:28:25.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa318ff9700 1 -- 172.21.15.81:0/3419392003 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328035640 con 0x55f2e980f950 2022-09-10T04:28:25.275 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa32c0931f0 con 0x55f2e980f950 2022-09-10T04:28:25.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa318ff9700 1 -- 172.21.15.81:0/3419392003 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fa3280357a0 con 0x55f2e980f950 2022-09-10T04:28:25.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa318ff9700 1 -- 172.21.15.81:0/3419392003 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa328035960 con 0x55f2e980f950 2022-09-10T04:28:25.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa34ad86b80 10 client.25331 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:25.276 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f2e958f260 con 0x55f2e980f950 2022-09-10T04:28:25.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa318ff9700 1 -- 172.21.15.81:0/3419392003 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa32c0931f0 con 0x55f2e980f950 2022-09-10T04:28:25.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa318ff9700 1 -- 172.21.15.81:0/3419392003 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa328017c40 con 0x55f2e980f950 2022-09-10T04:28:25.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa34ad86b80 10 client.25331 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:25.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa34ad86b80 10 client.25331 resolve_mds: resolved name 'b' to daemon mds.b 2022-09-10T04:28:25.277 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.257+0000 7fa34ad86b80 20 client.25331 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:25.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.258+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e98798a0 0x55f2e98761d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.258+0000 7fa34ad86b80 4 client.25331 mds_command: new command op to 24572 tid=0[{"prefix": "status"}] 2022-09-10T04:28:25.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.258+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 --> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f2e97556f0 con 0x55f2e98798a0 2022-09-10T04:28:25.278 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.258+0000 7fa33559b700 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e98798a0 0x55f2e98761d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.258+0000 7fa33559b700 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e98798a0 0x55f2e98761d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e353e7c5a5f1c4f server_cookie=4182e9f86d92d37b in_seq=0 out_seq=0 2022-09-10T04:28:25.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.258+0000 7fa318ff9700 10 client.25331 ms_handle_connect on v2:172.21.15.104:6838/2357131382 2022-09-10T04:28:25.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa318ff9700 1 -- 172.21.15.81:0/3419392003 <== mds.1 v2:172.21.15.104:6838/2357131382 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55f2e97556f0 con 0x55f2e98798a0 2022-09-10T04:28:25.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa318ff9700 10 client.25331 handle_command_reply: tid=0 2022-09-10T04:28:25.279 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 client.25331 shutdown 2022-09-10T04:28:25.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 msgr2=0x55f2e981bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 0x55f2e981bd20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ad0 tx=0x7fa328009e10 comp rx=0 tx=0).stop 2022-09-10T04:28:25.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 shutdown_connections 2022-09-10T04:28:25.280 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:6838/2357131382,v1:172.21.15.104:6839/2357131382] conn(0x55f2e98798a0 0x55f2e98761d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55f2e980f950 0x55f2e981bd20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f2e980e1c0 0x55f2e981b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 --2- 172.21.15.81:0/3419392003 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55f2e980f000 0x55f2e981b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 >> 172.21.15.81:0/3419392003 conn(0x55f2e980a170 msgr2=0x55f2e9807c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.281 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa318ff9700 0 client.25331 ms_handle_reset on v2:172.21.15.104:6838/2357131382 2022-09-10T04:28:25.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 shutdown_connections 2022-09-10T04:28:25.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 1 -- 172.21.15.81:0/3419392003 wait complete. 2022-09-10T04:28:25.282 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.259+0000 7fa34ad86b80 20 client.25331 trim_cache size 0 max 16384 2022-09-10T04:28:25.283 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:28:25.283 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:28:25.283 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 1, 2022-09-10T04:28:25.284 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24572, 2022-09-10T04:28:25.284 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:28:25.284 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:28:25.284 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:28:25.284 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 89.814004956999995, 2022-09-10T04:28:25.285 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:28:25.285 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 94, 2022-09-10T04:28:25.285 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:28:25.286 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 121.05027800000001 2022-09-10T04:28:25.286 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:28:25.289 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa31c099010 msgr2=0x7fa31c09b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa31c099010 0x7fa31c09b4c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa32800fe20 tx=0x7fa328004d70 comp rx=0 tx=0).stop 2022-09-10T04:28:25.290 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 msgr2=0x7fa330195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa330195910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e9c0 tx=0x7fa32c00ecd0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.291 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 shutdown_connections 2022-09-10T04:28:25.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa3300ff510 0x7fa33019cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3300fdb40 0x7fa330195910 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.292 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fa31c099010 0x7fa31c09b4c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 --2- 172.21.15.81:0/3494417270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3300fe490 0x7fa330195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 >> 172.21.15.81:0/3494417270 conn(0x7fa3300f9230 msgr2=0x7fa3300fb460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 shutdown_connections 2022-09-10T04:28:25.293 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.260+0000 7fa33659d700 1 -- 172.21.15.81:0/3494417270 wait complete. 2022-09-10T04:28:25.294 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2022-09-10T04:28:25.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.522+0000 7f2305dff700 1 -- 172.21.15.81:0/114334831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 msgr2=0x7f23001061d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.522+0000 7f2305dff700 1 --2- 172.21.15.81:0/114334831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 0x7f23001061d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009b00 tx=0x7f22f8009e10 comp rx=0 tx=0).stop 2022-09-10T04:28:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.522+0000 7f2305dff700 1 -- 172.21.15.81:0/114334831 shutdown_connections 2022-09-10T04:28:25.529 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.522+0000 7f2305dff700 1 --2- 172.21.15.81:0/114334831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 0x7f23001061d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.522+0000 7f2305dff700 1 --2- 172.21.15.81:0/114334831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fdd00 0x7f23000fe110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.523+0000 7f2305dff700 1 --2- 172.21.15.81:0/114334831 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fef80 0x7f23000ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.530 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.523+0000 7f2305dff700 1 -- 172.21.15.81:0/114334831 >> 172.21.15.81:0/114334831 conn(0x7f23000f93f0 msgr2=0x7f23000fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.523+0000 7f2305dff700 1 -- 172.21.15.81:0/114334831 shutdown_connections 2022-09-10T04:28:25.531 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.523+0000 7f2305dff700 1 -- 172.21.15.81:0/114334831 wait complete. 2022-09-10T04:28:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 Processor -- start 2022-09-10T04:28:25.532 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 -- start start 2022-09-10T04:28:25.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 0x7f2300195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fef80 0x7f2300196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 0x7f230019d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230019d6a0 con 0x7f23000ff910 2022-09-10T04:28:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230019d810 con 0x7f23000fef80 2022-09-10T04:28:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.524+0000 7f2305dff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230019d9d0 con 0x7f23000fdd00 2022-09-10T04:28:25.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22fffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 0x7f230019d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fef80 0x7f2300196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 0x7f2300195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 0x7f2300195b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39778/0 (socket says 172.21.15.81:39778) 2022-09-10T04:28:25.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 -- 172.21.15.81:0/606100942 learned_addr learned my addr 172.21.15.81:0/606100942 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 -- 172.21.15.81:0/606100942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fef80 msgr2=0x7f2300196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fef80 0x7f2300196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 -- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 msgr2=0x7f230019d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 0x7f230019d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.538 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 -- 172.21.15.81:0/606100942 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0009470 con 0x7f23000fdd00 2022-09-10T04:28:25.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22feffd700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fef80 0x7f2300196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:25.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.525+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 0x7f2300195b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f22f000e9c0 tx=0x7f22f000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.539 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.526+0000 7f22fcff9700 1 -- 172.21.15.81:0/606100942 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f000c9c0 con 0x7f23000fdd00 2022-09-10T04:28:25.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.526+0000 7f22fcff9700 1 -- 172.21.15.81:0/606100942 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f22f0010430 con 0x7f23000fdd00 2022-09-10T04:28:25.540 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.526+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f8009730 con 0x7f23000fdd00 2022-09-10T04:28:25.541 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.526+0000 7f22fcff9700 1 -- 172.21.15.81:0/606100942 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0018660 con 0x7f23000fdd00 2022-09-10T04:28:25.541 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.526+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230019e080 con 0x7f23000fdd00 2022-09-10T04:28:25.554 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.529+0000 7f22fcff9700 1 -- 172.21.15.81:0/606100942 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f22f000c040 con 0x7f23000fdd00 2022-09-10T04:28:25.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.530+0000 7f2319338b80 1 Processor -- start 2022-09-10T04:28:25.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.530+0000 7f22fcff9700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f22e8099090 0x7f22e809b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.555 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.530+0000 7f2319338b80 1 -- start start 2022-09-10T04:28:25.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.530+0000 7f22fcff9700 1 -- 172.21.15.81:0/606100942 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f22f0014070 con 0x7f23000fdd00 2022-09-10T04:28:25.556 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f22fffff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f22e8099090 0x7f22e809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f2319338b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea88ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f2319338b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea89c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.557 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f2319338b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea8a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f2319338b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fe973080 con 0x5628fea88ae0 2022-09-10T04:28:25.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f2319338b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fe9dd9e0 con 0x5628fea8a1f0 2022-09-10T04:28:25.558 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f2319338b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fe9d7180 con 0x5628fea89430 2022-09-10T04:28:25.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea88ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f22feffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea89c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f22feffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea89c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39792/0 (socket says 172.21.15.81:39792) 2022-09-10T04:28:25.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f22feffd700 1 -- 172.21.15.81:0/3698851344 learned_addr learned my addr 172.21.15.81:0/3698851344 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.559 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.531+0000 7f22fffff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f22e8099090 0x7f22e809b540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009ad0 tx=0x7f22f801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22fffff700 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea8a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22feffd700 1 -- 172.21.15.81:0/3698851344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 msgr2=0x5628fea8a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22feffd700 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea8a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.560 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22feffd700 1 -- 172.21.15.81:0/3698851344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 msgr2=0x5628fea88ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22feffd700 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea88ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22feffd700 1 -- 172.21.15.81:0/3698851344 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628fe85f5e0 con 0x5628fea89430 2022-09-10T04:28:25.561 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22fffff700 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea8a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:28:25.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22feffd700 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea89c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009fd0 tx=0x7f22f4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ed6b5f9a931a4ae server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22e5ffb700 1 -- 172.21.15.81:0/3698851344 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004230 con 0x5628fea89430 2022-09-10T04:28:25.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22e5ffb700 1 -- 172.21.15.81:0/3698851344 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f22f4004390 con 0x5628fea89430 2022-09-10T04:28:25.562 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.532+0000 7f22e5ffb700 1 -- 172.21.15.81:0/3698851344 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4019920 con 0x5628fea89430 2022-09-10T04:28:25.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 -- 172.21.15.81:0/3698851344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 msgr2=0x5628fea89c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea89c60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f22f4009fd0 tx=0x7f22f4011720 comp rx=0 tx=0).stop 2022-09-10T04:28:25.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 -- 172.21.15.81:0/3698851344 shutdown_connections 2022-09-10T04:28:25.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea8a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.563 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea88ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 --2- 172.21.15.81:0/3698851344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea89c60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 -- 172.21.15.81:0/3698851344 >> 172.21.15.81:0/3698851344 conn(0x5628fea84fa0 msgr2=0x5628fea83370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 -- 172.21.15.81:0/3698851344 shutdown_connections 2022-09-10T04:28:25.564 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.533+0000 7f2319338b80 1 -- 172.21.15.81:0/3698851344 wait complete. 2022-09-10T04:28:25.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 -- start start 2022-09-10T04:28:25.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 Processor -- start 2022-09-10T04:28:25.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea929e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.565 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea99a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fe8c50c0 con 0x5628fea88ae0 2022-09-10T04:28:25.566 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fea99f60 con 0x5628fea8a1f0 2022-09-10T04:28:25.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.534+0000 7f2319338b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fea9a0f0 con 0x5628fea89430 2022-09-10T04:28:25.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f2319338b80 10 client.0 _ll_register_callbacks cb 0x5628fe982370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:25.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.567 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea924a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56404/0 (socket says 172.21.15.81:56404) 2022-09-10T04:28:25.568 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 -- 172.21.15.81:0/575172821 learned_addr learned my addr 172.21.15.81:0/575172821 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.568 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22fffff700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea99a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.568 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22feffd700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea929e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.568 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 -- 172.21.15.81:0/575172821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 msgr2=0x5628fea929e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea929e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 -- 172.21.15.81:0/575172821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 msgr2=0x5628fea99a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea99a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 -- 172.21.15.81:0/575172821 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f4009c00 con 0x5628fea88ae0 2022-09-10T04:28:25.569 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea924a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22f0021000 tx=0x7f22f00926b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db3f29f0b2cd543 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.535+0000 7f22daffd700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:28:25.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.536+0000 7f22daffd700 1 -- 172.21.15.81:0/575172821 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0092a80 con 0x5628fea88ae0 2022-09-10T04:28:25.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.536+0000 7f22daffd700 1 -- 172.21.15.81:0/575172821 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f22f0099d20 con 0x5628fea88ae0 2022-09-10T04:28:25.570 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.536+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5628fea7baa0 con 0x5628fea88ae0 2022-09-10T04:28:25.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.536+0000 7f22daffd700 1 -- 172.21.15.81:0/575172821 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f0092ce0 con 0x5628fea88ae0 2022-09-10T04:28:25.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.536+0000 7f2319338b80 10 client.15240 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:25.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.536+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5628fe80bd90 con 0x5628fea88ae0 2022-09-10T04:28:25.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f22daffd700 1 -- 172.21.15.81:0/575172821 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f22f00ad030 con 0x5628fea88ae0 2022-09-10T04:28:25.571 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f22daffd700 1 -- 172.21.15.81:0/575172821 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f22f0091f70 con 0x5628fea88ae0 2022-09-10T04:28:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f2319338b80 10 client.15240 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f2319338b80 10 client.15240 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-10T04:28:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f2319338b80 20 client.15240 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f2319338b80 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628fea9d770 0x5628feaa2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.572 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f2319338b80 4 client.15240 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:28:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628fea584c0 con 0x5628fea9d770 2022-09-10T04:28:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.537+0000 7f22feffd700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628fea9d770 0x5628feaa2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.573 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.538+0000 7f22feffd700 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628fea9d770 0x5628feaa2a90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdec3da749d958ec server_cookie=d1a57ee149c77938 in_seq=0 out_seq=0 2022-09-10T04:28:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.538+0000 7f22daffd700 10 client.15240 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.538+0000 7f22daffd700 1 -- 172.21.15.81:0/575172821 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5628fea584c0 con 0x5628fea9d770 2022-09-10T04:28:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.538+0000 7f22daffd700 10 client.15240 handle_command_reply: tid=0 2022-09-10T04:28:25.574 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 client.15240 shutdown 2022-09-10T04:28:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 msgr2=0x5628fea924a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea924a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22f0021000 tx=0x7f22f00926b0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 shutdown_connections 2022-09-10T04:28:25.575 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628fea9d770 0x5628feaa2a90 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea99a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea924a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 --2- 172.21.15.81:0/575172821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea929e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 >> 172.21.15.81:0/575172821 conn(0x5628fea84fa0 msgr2=0x5628fea82a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.576 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f22daffd700 0 client.15240 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 shutdown_connections 2022-09-10T04:28:25.577 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 1 -- 172.21.15.81:0/575172821 wait complete. 2022-09-10T04:28:25.578 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.539+0000 7f2319338b80 20 client.15240 trim_cache size 0 max 16384 2022-09-10T04:28:25.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 Processor -- start 2022-09-10T04:28:25.584 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 -- start start 2022-09-10T04:28:25.585 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628feaa6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.585 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea8ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.585 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea92250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.585 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fe863a30 con 0x5628fea88ae0 2022-09-10T04:28:25.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fe9a3690 con 0x5628fea8a1f0 2022-09-10T04:28:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f2319338b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fea9f010 con 0x5628fea89430 2022-09-10T04:28:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.557+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628feaa6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628feaa6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56414/0 (socket says 172.21.15.81:56414) 2022-09-10T04:28:25.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 -- 172.21.15.81:0/2091489850 learned_addr learned my addr 172.21.15.81:0/2091489850 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22fffff700 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea92250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.588 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22feffd700 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea8ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 -- 172.21.15.81:0/2091489850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 msgr2=0x5628fea8ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea8ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 -- 172.21.15.81:0/2091489850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 msgr2=0x5628fea92250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea92250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 -- 172.21.15.81:0/2091489850 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628fe80a260 con 0x5628fea88ae0 2022-09-10T04:28:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628feaa6ab0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f22f000c5b0 tx=0x7f22f0096380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d755bd6f21d5ed server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.558+0000 7f22e67fc700 1 -- 172.21.15.81:0/2091489850 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0099d20 con 0x5628fea88ae0 2022-09-10T04:28:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f22e67fc700 1 -- 172.21.15.81:0/2091489850 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5628fea7baa0 con 0x5628fea88ae0 2022-09-10T04:28:25.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f22e67fc700 1 -- 172.21.15.81:0/2091489850 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f0092ce0 con 0x5628fea88ae0 2022-09-10T04:28:25.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 -- 172.21.15.81:0/2091489850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 msgr2=0x5628feaa6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628feaa6ab0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f22f000c5b0 tx=0x7f22f0096380 comp rx=0 tx=0).stop 2022-09-10T04:28:25.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 -- 172.21.15.81:0/2091489850 shutdown_connections 2022-09-10T04:28:25.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea8a1f0 0x5628fea92250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628feaa6ab0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 --2- 172.21.15.81:0/2091489850 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea89430 0x5628fea8ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 -- 172.21.15.81:0/2091489850 >> 172.21.15.81:0/2091489850 conn(0x5628fea84fa0 msgr2=0x5628fea9a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 -- 172.21.15.81:0/2091489850 shutdown_connections 2022-09-10T04:28:25.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.559+0000 7f2319338b80 1 -- 172.21.15.81:0/2091489850 wait complete. 2022-09-10T04:28:25.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.560+0000 7f2319338b80 1 -- start start 2022-09-10T04:28:25.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.560+0000 7f2319338b80 1 Processor -- start 2022-09-10T04:28:25.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea96700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 0x5628fea96c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea8a1f0 0x5628feaefa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fea97320 con 0x5628fea88ae0 2022-09-10T04:28:25.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628feaeff70 con 0x5628fea89430 2022-09-10T04:28:25.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628feaf00b0 con 0x5628fea8a1f0 2022-09-10T04:28:25.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f2319338b80 10 client.0 _ll_register_callbacks cb 0x5628fe982370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:25.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 0x5628fea96c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22ff7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea96700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 0x5628fea96c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52114/0 (socket says 172.21.15.81:52114) 2022-09-10T04:28:25.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 -- 172.21.15.81:0/3435227760 learned_addr learned my addr 172.21.15.81:0/3435227760 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:25.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22fffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea8a1f0 0x5628feaefa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 -- 172.21.15.81:0/3435227760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea8a1f0 msgr2=0x5628feaefa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea8a1f0 0x5628feaefa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 -- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 msgr2=0x5628fea96700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:25.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea96700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.561+0000 7f22feffd700 1 -- 172.21.15.81:0/3435227760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628fe80bd90 con 0x5628fea89430 2022-09-10T04:28:25.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f22feffd700 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 0x5628fea96c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x5628fea89840 tx=0x7f22f4004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bf4ae6e89fdb83d server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:25.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f22daffd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:28:25.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f22daffd700 1 -- 172.21.15.81:0/3435227760 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4004770 con 0x5628fea89430 2022-09-10T04:28:25.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22f4004390 con 0x5628fea89430 2022-09-10T04:28:25.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f22daffd700 1 -- 172.21.15.81:0/3435227760 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f22f4005b60 con 0x5628fea89430 2022-09-10T04:28:25.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f22daffd700 1 -- 172.21.15.81:0/3435227760 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22f4004100 con 0x5628fea89430 2022-09-10T04:28:25.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f2319338b80 10 client.25346 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:25.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5628fe8d8a30 con 0x5628fea89430 2022-09-10T04:28:25.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.562+0000 7f22daffd700 1 -- 172.21.15.81:0/3435227760 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f22f4004390 con 0x5628fea89430 2022-09-10T04:28:25.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f22daffd700 1 -- 172.21.15.81:0/3435227760 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f22f4011440 con 0x5628fea89430 2022-09-10T04:28:25.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f2319338b80 10 client.25346 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:25.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f2319338b80 10 client.25346 resolve_mds: resolved name 'a' to daemon mds.a 2022-09-10T04:28:25.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f2319338b80 20 client.25346 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:25.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f2319338b80 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628feaf4f70 0x5628feaf18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:25.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f2319338b80 4 client.25346 mds_command: new command op to 24577 tid=0[{"prefix": "status"}] 2022-09-10T04:28:25.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5628fe9d06f0 con 0x5628feaf4f70 2022-09-10T04:28:25.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628feaf4f70 0x5628feaf18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:25.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f22ff7fe700 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628feaf4f70 0x5628feaf18e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd1ea661fa03a138 server_cookie=5b6449d83cc8866d in_seq=0 out_seq=0 2022-09-10T04:28:25.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.563+0000 7f22daffd700 10 client.25346 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:25.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f22daffd700 1 -- 172.21.15.81:0/3435227760 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5628fe9d06f0 con 0x5628feaf4f70 2022-09-10T04:28:25.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f22daffd700 10 client.25346 handle_command_reply: tid=0 2022-09-10T04:28:25.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 client.25346 shutdown 2022-09-10T04:28:25.605 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 msgr2=0x5628fea96c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 0x5628fea96c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x5628fea89840 tx=0x7f22f4004500 comp rx=0 tx=0).stop 2022-09-10T04:28:25.606 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 shutdown_connections 2022-09-10T04:28:25.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628fea8a1f0 0x5628feaefa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628fea88ae0 0x5628fea96700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5628feaf4f70 0x5628feaf18e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 --2- 172.21.15.81:0/3435227760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5628fea89430 0x5628fea96c40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 >> 172.21.15.81:0/3435227760 conn(0x5628fea84fa0 msgr2=0x5628fea8ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.608 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f22daffd700 0 client.25346 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:25.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 shutdown_connections 2022-09-10T04:28:25.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 1 -- 172.21.15.81:0/3435227760 wait complete. 2022-09-10T04:28:25.609 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.564+0000 7f2319338b80 20 client.25346 trim_cache size 0 max 16384 2022-09-10T04:28:25.610 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2022-09-10T04:28:25.610 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "70cf895c-30bf-11ed-8431-001a4aab830c", 2022-09-10T04:28:25.611 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2022-09-10T04:28:25.611 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24577, 2022-09-10T04:28:25.611 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2022-09-10T04:28:25.611 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2022-09-10T04:28:25.612 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2022-09-10T04:28:25.612 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 113.46452340099999, 2022-09-10T04:28:25.612 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 28, 2022-09-10T04:28:25.612 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 94, 2022-09-10T04:28:25.613 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 90, 2022-09-10T04:28:25.613 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 123.552143947 2022-09-10T04:28:25.613 INFO:teuthology.orchestra.run.smithi081.stdout:} 2022-09-10T04:28:25.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f22e8099090 msgr2=0x7f22e809b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f22e8099090 0x7f22e809b540 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009ad0 tx=0x7f22f801a040 comp rx=0 tx=0).stop 2022-09-10T04:28:25.615 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 msgr2=0x7f2300195b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:25.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 0x7f2300195b20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f22f000e9c0 tx=0x7f22f000ecd0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 shutdown_connections 2022-09-10T04:28:25.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23000ff910 0x7f230019d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f22e8099090 0x7f22e809b540 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23000fdd00 0x7f2300195b20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 --2- 172.21.15.81:0/606100942 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f23000fef80 0x7f2300196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:25.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 >> 172.21.15.81:0/606100942 conn(0x7f23000f93f0 msgr2=0x7f23000fb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:25.617 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 shutdown_connections 2022-09-10T04:28:25.618 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:25.565+0000 7f2305dff700 1 -- 172.21.15.81:0/606100942 wait complete. 2022-09-10T04:28:25.618 INFO:teuthology.run_tasks:Running task kclient... 2022-09-10T04:28:25.635 INFO:tasks.kclient:Mounting kernel clients... 2022-09-10T04:28:25.635 INFO:tasks.kclient:config is {'client.0': {}} 2022-09-10T04:28:25.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:25.636 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2022-09-10T04:28:25.695 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2022-09-10T04:28:25.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:25.697 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2022-09-10T04:28:25.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:25.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2022-09-10T04:28:25.943 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2'} 2022-09-10T04:28:25.944 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2022-09-10T04:28:25.945 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:25.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph fs ls 2022-09-10T04:28:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:26 smithi104 ceph-mon[97044]: pgmap v362: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T04:28:26.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:26 smithi081 ceph-mon[90133]: pgmap v362: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T04:28:26.824 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T04:28:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:26 smithi110 ceph-mon[97633]: pgmap v362: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T04:28:28.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 -- 172.21.15.81:0/2044550717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf80666c0 msgr2=0x7febf8066a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:28.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 --2- 172.21.15.81:0/2044550717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf80666c0 0x7febf8066a90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7febe8009750 tx=0x7febe8009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:28.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 -- 172.21.15.81:0/2044550717 shutdown_connections 2022-09-10T04:28:28.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 --2- 172.21.15.81:0/2044550717 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf810aa10 0x7febf810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.314 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 --2- 172.21.15.81:0/2044550717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf80666c0 0x7febf8066a90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 --2- 172.21.15.81:0/2044550717 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf8066fd0 0x7febf810a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 -- 172.21.15.81:0/2044550717 >> 172.21.15.81:0/2044550717 conn(0x7febf8073cf0 msgr2=0x7febf80740f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:28.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 -- 172.21.15.81:0/2044550717 shutdown_connections 2022-09-10T04:28:28.315 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.311+0000 7febfc94f700 1 -- 172.21.15.81:0/2044550717 wait complete. 2022-09-10T04:28:28.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.313+0000 7febfc94f700 1 Processor -- start 2022-09-10T04:28:28.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.313+0000 7febfc94f700 1 -- start start 2022-09-10T04:28:28.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febfc94f700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf80666c0 0x7febf8195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:28.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febfc94f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf8066fd0 0x7febf8196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:28.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febfc94f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf810aa10 0x7febf819a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:28.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febfc94f700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf819aeb0 con 0x7febf80666c0 2022-09-10T04:28:28.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febfc94f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf819b020 con 0x7febf810aa10 2022-09-10T04:28:28.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febfc94f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf819b1e0 con 0x7febf8066fd0 2022-09-10T04:28:28.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.314+0000 7febf659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf80666c0 0x7febf8195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:28.319 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf80666c0 0x7febf8195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56446/0 (socket says 172.21.15.81:56446) 2022-09-10T04:28:28.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf810aa10 0x7febf819a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:28.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf659c700 1 -- 172.21.15.81:0/1517880060 learned_addr learned my addr 172.21.15.81:0/1517880060 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:28.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf5d9b700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf8066fd0 0x7febf8196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:28.320 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 -- 172.21.15.81:0/1517880060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf8066fd0 msgr2=0x7febf8196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:28.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf8066fd0 0x7febf8196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 -- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf80666c0 msgr2=0x7febf8195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:28.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf80666c0 0x7febf8195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 -- 172.21.15.81:0/1517880060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe80093b0 con 0x7febf810aa10 2022-09-10T04:28:28.321 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.315+0000 7febf6d9d700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf810aa10 0x7febf819a820 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7febf000e940 tx=0x7febf000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:28.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.316+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0004c30 con 0x7febf810aa10 2022-09-10T04:28:28.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.316+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7febf0004d90 con 0x7febf810aa10 2022-09-10T04:28:28.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.316+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf819b4c0 con 0x7febf810aa10 2022-09-10T04:28:28.322 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.316+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf00057d0 con 0x7febf810aa10 2022-09-10T04:28:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.316+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf81a2d00 con 0x7febf810aa10 2022-09-10T04:28:28.323 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.317+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8064ea0 con 0x7febf810aa10 2022-09-10T04:28:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.319+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7febf001e020 con 0x7febf810aa10 2022-09-10T04:28:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.320+0000 7febe77fe700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7febe0099090 0x7febe009b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:28.324 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.320+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7febf0014070 con 0x7febf810aa10 2022-09-10T04:28:28.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.320+0000 7febf659c700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7febe0099090 0x7febe009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:28.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.321+0000 7febf659c700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7febe0099090 0x7febe009b540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7febe8009e30 tx=0x7febe80092d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:28.325 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.322+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7febf008ee30 con 0x7febf810aa10 2022-09-10T04:28:28.586 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.586+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7febf810ff00 con 0x7febf810aa10 2022-09-10T04:28:28.587 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.587+0000 7febe77fe700 1 -- 172.21.15.81:0/1517880060 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7febf0092450 con 0x7febf810aa10 2022-09-10T04:28:28.588 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2022-09-10T04:28:28.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7febe0099090 msgr2=0x7febe009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:28.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7febe0099090 0x7febe009b540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7febe8009e30 tx=0x7febe80092d0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf810aa10 msgr2=0x7febf819a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:28.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf810aa10 0x7febf819a820 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7febf000e940 tx=0x7febf000ec50 comp rx=0 tx=0).stop 2022-09-10T04:28:28.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 shutdown_connections 2022-09-10T04:28:28.591 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febf810aa10 0x7febf819a820 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febf80666c0 0x7febf8195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7febe0099090 0x7febe009b540 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 --2- 172.21.15.81:0/1517880060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf8066fd0 0x7febf8196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:28.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 >> 172.21.15.81:0/1517880060 conn(0x7febf8073cf0 msgr2=0x7febf80fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:28.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 shutdown_connections 2022-09-10T04:28:28.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:28.589+0000 7febfc94f700 1 -- 172.21.15.81:0/1517880060 wait complete. 2022-09-10T04:28:28.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:28 smithi104 ceph-mon[97044]: pgmap v363: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:28:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:28 smithi081 ceph-mon[90133]: pgmap v363: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:28:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:28 smithi110 ceph-mon[97633]: pgmap v363: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T04:28:29.090 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2022-09-10T04:28:29.091 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2022-09-10T04:28:29.091 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2022-09-10T04:28:29.092 INFO:tasks.cephfs.mount:self.client.name = client.0 2022-09-10T04:28:29.092 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:29.092 INFO:tasks.cephfs.mount:self.cephfs_name = None 2022-09-10T04:28:29.093 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2022-09-10T04:28:29.093 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2022-09-10T04:28:29.095 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2022-09-10T04:28:29.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.096 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2022-09-10T04:28:29.156 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2022-09-10T04:28:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2022-09-10T04:28:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2022-09-10T04:28:29.157 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-09-10T04:28:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2022-09-10T04:28:29.158 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-09-10T04:28:29.159 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-10T04:28:29.159 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:28:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2022-09-10T04:28:29.160 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-10T04:28:29.160 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:28:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2022-09-10T04:28:29.161 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2022-09-10T04:28:29.161 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:28:29.161 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2022-09-10T04:28:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2022-09-10T04:28:29.162 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2022-09-10T04:28:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41570sec preferred_lft 41570sec 2022-09-10T04:28:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2022-09-10T04:28:29.163 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2022-09-10T04:28:29.164 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2022-09-10T04:28:29.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.165 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:28:29.166 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:28:29.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2022-09-10T04:28:29.166 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2022-09-10T04:28:29.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2022-09-10T04:28:29.167 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2022-09-10T04:28:29.167 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:28:29.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.429 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2022-09-10T04:28:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:1 2022-09-10T04:28:29.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.526 DEBUG:teuthology.orchestra.run.smithi081:> route 2022-09-10T04:28:29.588 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2022-09-10T04:28:29.588 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2022-09-10T04:28:29.589 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2022-09-10T04:28:29.589 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2022-09-10T04:28:29.589 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2022-09-10T04:28:29.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.591 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:28:29.591 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:28:29.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2022-09-10T04:28:29.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2022-09-10T04:28:29.592 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2022-09-10T04:28:29.593 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:28:29.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:29 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/1517880060' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-10T04:28:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:29 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/1517880060' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-10T04:28:29.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.810 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2022-09-10T04:28:29.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.870 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2022-09-10T04:28:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:29 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/1517880060' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2022-09-10T04:28:29.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:29.930 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:28:29.930 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:28:29.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2022-09-10T04:28:29.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2022-09-10T04:28:29.932 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:28:30.118 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2022-09-10T04:28:30.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:30.119 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:28:30.120 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:28:30.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2022-09-10T04:28:30.121 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2022-09-10T04:28:30.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2022-09-10T04:28:30.122 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2022-09-10T04:28:30.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2022-09-10T04:28:30.123 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:28:30.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:30.436 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2022-09-10T04:28:30.437 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2022-09-10T04:28:30.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2022-09-10T04:28:30.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2022-09-10T04:28:30.438 DEBUG:teuthology.orchestra.run.smithi081:> ') 2022-09-10T04:28:30.610 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:30 smithi081 ceph-mon[90133]: pgmap v364: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T04:28:30.654 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-10T04:28:30.654 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:30.715 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2022-09-10T04:28:30.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-09-10T04:28:30.716 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:30.774 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2022-09-10T04:28:30.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:30.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2022-09-10T04:28:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:30 smithi104 ceph-mon[97044]: pgmap v364: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T04:28:30.904 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2022-09-10T04:28:30.904 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2022-09-10T04:28:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:30 smithi110 ceph-mon[97633]: pgmap v364: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T04:28:31.396 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2022-09-10T04:28:31.396 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2022-09-10T04:28:31.397 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:3300,172.21.15.104:3300,172.21.15.110:3300" 2022-09-10T04:28:31.397 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@70cf895c-30bf-11ed-8431-001a4aab830c.cephfs=/ 2022-09-10T04:28:31.398 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.81:3300/172.21.15.104:3300/172.21.15.110:3300" will pass to kernel 2022-09-10T04:28:31.406 INFO:tasks.cephfs.kernel_mount:mount command passed 2022-09-10T04:28:31.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:31.407 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:31.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:31.497 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2022-09-10T04:28:31.615 INFO:teuthology.orchestra.run.smithi081.stdout:client25355 2022-09-10T04:28:31.616 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:28:31.617 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2022-09-10T04:28:31.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:31.684 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2022-09-10T04:28:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:client25355 2022-09-10T04:28:31.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:31.817 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2022-09-10T04:28:31.934 INFO:teuthology.orchestra.run.smithi081.stdout:client25355 2022-09-10T04:28:31.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-09-10T04:28:31.935 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/70cf895c-30bf-11ed-8431-001a4aab830c.client25355/status') 2022-09-10T04:28:32.082 INFO:teuthology.orchestra.run.smithi081.stdout:instance: client.25355 (3)172.21.15.81:0/197519246 2022-09-10T04:28:32.083 INFO:teuthology.orchestra.run.smithi081.stdout:blocklisted: false 2022-09-10T04:28:32.084 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records in 2022-09-10T04:28:32.084 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records out 2022-09-10T04:28:32.084 INFO:teuthology.orchestra.run.smithi081.stderr:70 bytes copied, 6.9787e-05 s, 1.0 MB/s 2022-09-10T04:28:32.091 INFO:teuthology.run_tasks:Running task check-counter... 2022-09-10T04:28:32.111 INFO:teuthology.run_tasks:Running task exec... 2022-09-10T04:28:32.132 INFO:teuthology.task.exec:Executing custom commands... 2022-09-10T04:28:32.133 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:28:32.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2022-09-10T04:28:32.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 -- 172.21.15.81:0/3093470125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6278103b10 msgr2=0x7f6278105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:32.356 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 --2- 172.21.15.81:0/3093470125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6278103b10 0x7f6278105ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6270009bb0 tx=0x7f6270009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:32.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 -- 172.21.15.81:0/3093470125 shutdown_connections 2022-09-10T04:28:32.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 --2- 172.21.15.81:0/3093470125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6278103b10 0x7f6278105ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:32.357 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 --2- 172.21.15.81:0/3093470125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:32.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 --2- 172.21.15.81:0/3093470125 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62781011f0 0x7f62781035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:32.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 -- 172.21.15.81:0/3093470125 >> 172.21.15.81:0/3093470125 conn(0x7f62780f83b0 msgr2=0x7f62780fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:32.358 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 -- 172.21.15.81:0/3093470125 shutdown_connections 2022-09-10T04:28:32.359 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.354+0000 7f627f9ff700 1 -- 172.21.15.81:0/3093470125 wait complete. 2022-09-10T04:28:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 Processor -- start 2022-09-10T04:28:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 -- start start 2022-09-10T04:28:32.362 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:32.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62781011f0 0x7f62781960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:32.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6278103b10 0x7f627819d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:32.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627819d6f0 con 0x7f62780fe8d0 2022-09-10T04:28:32.363 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627819d860 con 0x7f62781011f0 2022-09-10T04:28:32.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.356+0000 7f627f9ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627819da20 con 0x7f6278103b10 2022-09-10T04:28:32.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:32.364 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627df9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6278103b10 0x7f627819d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627cf9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62781011f0 0x7f62781960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34242/0 (socket says 172.21.15.81:34242) 2022-09-10T04:28:32.365 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 -- 172.21.15.81:0/932160056 learned_addr learned my addr 172.21.15.81:0/932160056 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:32.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 -- 172.21.15.81:0/932160056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6278103b10 msgr2=0x7f627819d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:32.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6278103b10 0x7f627819d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:32.366 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 -- 172.21.15.81:0/932160056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62781011f0 msgr2=0x7f62781960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:32.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62781011f0 0x7f62781960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:32.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 -- 172.21.15.81:0/932160056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62740093b0 con 0x7f62780fe8d0 2022-09-10T04:28:32.367 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627cf9a700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62781011f0 0x7f62781960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:32.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.357+0000 7f627d79b700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278195b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f627400e9c0 tx=0x7f627400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:32.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.358+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62740097b0 con 0x7f62780fe8d0 2022-09-10T04:28:32.368 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.358+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62700097e0 con 0x7f62780fe8d0 2022-09-10T04:28:32.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.358+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f627400ce70 con 0x7f62780fe8d0 2022-09-10T04:28:32.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.358+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6274003cf0 con 0x7f62780fe8d0 2022-09-10T04:28:32.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.358+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627819e0d0 con 0x7f62780fe8d0 2022-09-10T04:28:32.369 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.359+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6278109550 con 0x7f62780fe8d0 2022-09-10T04:28:32.370 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.361+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) v1 ==== 127678+0+0 (secure 0 0 0) 0x7f627401e020 con 0x7f62780fe8d0 2022-09-10T04:28:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.362+0000 7f626e7fc700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6264099010 0x7f626409b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:32.371 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.362+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f6274014070 con 0x7f62780fe8d0 2022-09-10T04:28:32.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.362+0000 7f627cf9a700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6264099010 0x7f626409b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:32.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.363+0000 7f627cf9a700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6264099010 0x7f626409b4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6268005fd0 tx=0x7f6268005ea0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:32.372 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.366+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f627408eed0 con 0x7f62780fe8d0 2022-09-10T04:28:32.616 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:32.614+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f627819e680 con 0x7f62780fe8d0 2022-09-10T04:28:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:32 smithi104 ceph-mon[97044]: pgmap v365: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:28:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:32 smithi081 ceph-mon[90133]: pgmap v365: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:28:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:32 smithi110 ceph-mon[97633]: pgmap v365: 129 pgs: 129 active+clean; 586 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:28:33.527 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.526+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 17) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f627400c040 con 0x7f62780fe8d0 2022-09-10T04:28:33.528 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.527+0000 7f626e7fc700 1 -- 172.21.15.81:0/932160056 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7f6274092960 con 0x7f62780fe8d0 2022-09-10T04:28:33.533 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.531+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6264099010 msgr2=0x7f626409b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:33.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.531+0000 7f627f9ff700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6264099010 0x7f626409b4c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6268005fd0 tx=0x7f6268005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.531+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 msgr2=0x7f6278195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:33.534 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.531+0000 7f627f9ff700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278195b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f627400e9c0 tx=0x7f627400ecd0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 shutdown_connections 2022-09-10T04:28:33.535 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f6264099010 0x7f626409b4c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6278103b10 0x7f627819d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62780fe8d0 0x7f6278195b70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.536 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 --2- 172.21.15.81:0/932160056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62781011f0 0x7f62781960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 >> 172.21.15.81:0/932160056 conn(0x7f62780f83b0 msgr2=0x7f62780ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 shutdown_connections 2022-09-10T04:28:33.537 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.532+0000 7f627f9ff700 1 -- 172.21.15.81:0/932160056 wait complete. 2022-09-10T04:28:33.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2022-09-10T04:28:33.766 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: ignoring --setuser ceph since I am not root 2022-09-10T04:28:33.767 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: ignoring --setgroup ceph since I am not root 2022-09-10T04:28:33.767 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:33 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/932160056' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-10T04:28:33.773 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/932160056' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-10T04:28:33.773 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: ignoring --setuser ceph since I am not root 2022-09-10T04:28:33.774 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: ignoring --setgroup ceph since I am not root 2022-09-10T04:28:33.775 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-mgr[98426]: -- 172.21.15.104:0/2084147237 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e804e5c820 con 0x55e804e31000 2022-09-10T04:28:33.775 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:33.682+0000 7f143826d080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:28:33.775 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:33.770+0000 7f143826d080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:28:33.777 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/932160056' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2022-09-10T04:28:33.777 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: ignoring --setuser ceph since I am not root 2022-09-10T04:28:33.779 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: ignoring --setgroup ceph since I am not root 2022-09-10T04:28:33.779 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-mgr[99615]: -- 172.21.15.110:0/1914203158 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d9d1854820 con 0x55d9d1828800 2022-09-10T04:28:33.780 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:33.681+0000 7f1e0820e080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:28:33.780 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:33.770+0000 7f1e0820e080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:28:33.825 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1074642860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200fdc50 msgr2=0x7f1e200fe060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:33.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1074642860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200fdc50 0x7f1e200fe060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009750 tx=0x7f1e10009a60 comp rx=0 tx=0).stop 2022-09-10T04:28:33.826 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1074642860 shutdown_connections 2022-09-10T04:28:33.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1074642860 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 0x7f1e20106120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1074642860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200fdc50 0x7f1e200fe060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.827 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1074642860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200feed0 0x7f1e200ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1074642860 >> 172.21.15.81:0/1074642860 conn(0x7f1e200f9360 msgr2=0x7f1e200fb770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:33.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1074642860 shutdown_connections 2022-09-10T04:28:33.828 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.822+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1074642860 wait complete. 2022-09-10T04:28:33.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.823+0000 7f1e1f59e700 1 Processor -- start 2022-09-10T04:28:33.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 -- start start 2022-09-10T04:28:33.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200fdc50 0x7f1e20195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:33.829 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200feed0 0x7f1e20195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:33.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 0x7f1e2019cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:33.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2019d4c0 con 0x7f1e200feed0 2022-09-10T04:28:33.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2019d630 con 0x7f1e200ff860 2022-09-10T04:28:33.830 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1f59e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2019d7f0 con 0x7f1e200fdc50 2022-09-10T04:28:33.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1dd9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200feed0 0x7f1e20195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:33.833 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 0x7f1e2019cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:33.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 0x7f1e2019cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45756/0 (socket says 172.21.15.81:45756) 2022-09-10T04:28:33.834 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200fdc50 0x7f1e20195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:33.838 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.824+0000 7f1e1ed9d700 1 -- 172.21.15.81:0/1973191224 learned_addr learned my addr 172.21.15.81:0/1973191224 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:33.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1e59c700 1 -- 172.21.15.81:0/1973191224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 msgr2=0x7f1e2019cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:33.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1e59c700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 0x7f1e2019cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1e59c700 1 -- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200feed0 msgr2=0x7f1e20195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:33.839 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1e59c700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200feed0 0x7f1e20195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:33.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1e59c700 1 -- 172.21.15.81:0/1973191224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180097e0 con 0x7f1e200fdc50 2022-09-10T04:28:33.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1e59c700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200fdc50 0x7f1e20195940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1e100096a0 tx=0x7f1e10004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:33.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1001d070 con 0x7f1e200fdc50 2022-09-10T04:28:33.840 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e100093b0 con 0x7f1e200fdc50 2022-09-10T04:28:33.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f1e10005590 con 0x7f1e200fdc50 2022-09-10T04:28:33.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.825+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e100179f0 con 0x7f1e200fdc50 2022-09-10T04:28:33.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.826+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2019dd30 con 0x7f1e200fdc50 2022-09-10T04:28:33.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.827+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e20109400 con 0x7f1e200fdc50 2022-09-10T04:28:33.841 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.828+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) v1 ==== 127695+0+0 (secure 0 0 0) 0x7f1e1000c040 con 0x7f1e200fdc50 2022-09-10T04:28:33.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.829+0000 7f1e0f7fe700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 0x7f1e0809b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:33.842 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.830+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7f1e100c3310 con 0x7f1e200fdc50 2022-09-10T04:28:33.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.830+0000 7f1e1dd9b700 1 -- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 msgr2=0x7f1e0809b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:33.843 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.830+0000 7f1e1dd9b700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 0x7f1e0809b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:28:33.844 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:33.833+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f1e1008f650 con 0x7f1e200fdc50 2022-09-10T04:28:34.032 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.031+0000 7f1e1dd9b700 1 -- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 msgr2=0x7f1e0809b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:34.033 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.031+0000 7f1e1dd9b700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 0x7f1e0809b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:28:34.036 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.033+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f1e2019e3d0 con 0x7f1e200fdc50 2022-09-10T04:28:34.040 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.039+0000 7f1e0f7fe700 1 -- 172.21.15.81:0/1973191224 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1e10092c70 con 0x7f1e200fdc50 2022-09-10T04:28:34.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 msgr2=0x7f1e0809b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:28:34.043 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 0x7f1e0809b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200fdc50 msgr2=0x7f1e20195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200fdc50 0x7f1e20195940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1e100096a0 tx=0x7f1e10004230 comp rx=0 tx=0).stop 2022-09-10T04:28:34.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 shutdown_connections 2022-09-10T04:28:34.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e200ff860 0x7f1e2019cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.044 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7f1e08099110 0x7f1e0809b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.045 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e200fdc50 0x7f1e20195940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.045 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 --2- 172.21.15.81:0/1973191224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1e200feed0 0x7f1e20195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.045 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.041+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 >> 172.21.15.81:0/1973191224 conn(0x7f1e200f9360 msgr2=0x7f1e20104700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:34.046 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.042+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 shutdown_connections 2022-09-10T04:28:34.046 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.042+0000 7f1e1f59e700 1 -- 172.21.15.81:0/1973191224 wait complete. 2022-09-10T04:28:34.047 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:33.858+0000 7f143826d080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:28:34.047 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:33 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:33.950+0000 7f143826d080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:28:34.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:33 smithi081 ceph-mgr[90405]: -- 172.21.15.81:0/1497204851 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644ea166820 con 0x5644ea13a800 2022-09-10T04:28:34.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:33.946+0000 7f9b87366080 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2022-09-10T04:28:34.070 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2022-09-10T04:28:34.117 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:33.859+0000 7f1e0820e080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:28:34.117 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:33 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:33.950+0000 7f1e0820e080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:28:34.299 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 -- 172.21.15.81:0/1217194928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd004071740 msgr2=0x7fd004071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 --2- 172.21.15.81:0/1217194928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd004071740 0x7fd004071bb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009810 tx=0x7fcff8009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 -- 172.21.15.81:0/1217194928 shutdown_connections 2022-09-10T04:28:34.300 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 --2- 172.21.15.81:0/1217194928 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd00410a430 0x7fd00410c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 --2- 172.21.15.81:0/1217194928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004072c90 0x7fd004071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.301 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 --2- 172.21.15.81:0/1217194928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd004071740 0x7fd004071bb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.297+0000 7fd0099ff700 1 -- 172.21.15.81:0/1217194928 >> 172.21.15.81:0/1217194928 conn(0x7fd0040f9480 msgr2=0x7fd0040fb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.298+0000 7fd0099ff700 1 -- 172.21.15.81:0/1217194928 shutdown_connections 2022-09-10T04:28:34.302 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.298+0000 7fd0099ff700 1 -- 172.21.15.81:0/1217194928 wait complete. 2022-09-10T04:28:34.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.299+0000 7fd0099ff700 1 Processor -- start 2022-09-10T04:28:34.305 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 -- start start 2022-09-10T04:28:34.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 0x7fd004199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd004072c90 0x7fd00419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00410a430 0x7fd0041a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.306 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041a1b70 con 0x7fd00410a430 2022-09-10T04:28:34.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041a1ce0 con 0x7fd004072c90 2022-09-10T04:28:34.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd0099ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041a1ea0 con 0x7fd004071740 2022-09-10T04:28:34.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd002ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 0x7fd004199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:34.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd002ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 0x7fd004199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59356/0 (socket says 172.21.15.81:59356) 2022-09-10T04:28:34.307 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.300+0000 7fd002ffd700 1 -- 172.21.15.81:0/2506383495 learned_addr learned my addr 172.21.15.81:0/2506383495 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:34.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd0027fc700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd004072c90 0x7fd00419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:34.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd0037fe700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00410a430 0x7fd0041a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:34.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd002ffd700 1 -- 172.21.15.81:0/2506383495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd004072c90 msgr2=0x7fd00419a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.308 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd002ffd700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd004072c90 0x7fd00419a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd002ffd700 1 -- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00410a430 msgr2=0x7fd0041a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd002ffd700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00410a430 0x7fd0041a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd002ffd700 1 -- 172.21.15.81:0/2506383495 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8009470 con 0x7fd004071740 2022-09-10T04:28:34.309 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd002ffd700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 0x7fd004199ff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e740 tx=0x7fcff400ea50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:34.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4005220 con 0x7fd004071740 2022-09-10T04:28:34.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fcff4005380 con 0x7fd004071740 2022-09-10T04:28:34.310 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0041a2180 con 0x7fd004071740 2022-09-10T04:28:34.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.301+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4004c10 con 0x7fd004071740 2022-09-10T04:28:34.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.302+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0041a2690 con 0x7fd004071740 2022-09-10T04:28:34.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.303+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00410fbe0 con 0x7fd004071740 2022-09-10T04:28:34.311 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.304+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 17) v1 ==== 127695+0+0 (secure 0 0 0) 0x7fcff401e020 con 0x7fd004071740 2022-09-10T04:28:34.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.305+0000 7fd0089fd700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 0x7fcfec09b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.312 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.306+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fcff4014070 con 0x7fd004071740 2022-09-10T04:28:34.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.306+0000 7fd0027fc700 1 -- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 msgr2=0x7fcfec09b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:34.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.306+0000 7fd0027fc700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 0x7fcfec09b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:28:34.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.309+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fcff408ee70 con 0x7fd004071740 2022-09-10T04:28:34.347 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:34.062+0000 7f9b87366080 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2022-09-10T04:28:34.347 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:34.202+0000 7f9b87366080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2022-09-10T04:28:34.406 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:34.114+0000 7f1e0820e080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:28:34.425 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:34.116+0000 7f143826d080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:28:34.509 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.507+0000 7fd0027fc700 1 -- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 msgr2=0x7fcfec09b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:34.509 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.507+0000 7fd0027fc700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 0x7fcfec09b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:28:34.510 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.507+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fd00405f960 con 0x7fd004071740 2022-09-10T04:28:34.516 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.515+0000 7fd0089fd700 1 -- 172.21.15.81:0/2506383495 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcff4092490 con 0x7fd004071740 2022-09-10T04:28:34.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.517+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 msgr2=0x7fcfec09b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:28:34.519 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.517+0000 7fd0099ff700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 0x7fcfec09b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.517+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 msgr2=0x7fd004199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.517+0000 7fd0099ff700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 0x7fd004199ff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e740 tx=0x7fcff400ea50 comp rx=0 tx=0).stop 2022-09-10T04:28:34.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 shutdown_connections 2022-09-10T04:28:34.520 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd00410a430 0x7fd0041a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fcfec099060 0x7fcfec09b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd004071740 0x7fd004199ff0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 --2- 172.21.15.81:0/2506383495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd004072c90 0x7fd00419a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.521 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 >> 172.21.15.81:0/2506383495 conn(0x7fd0040f9480 msgr2=0x7fd0040fb860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:34.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 shutdown_connections 2022-09-10T04:28:34.522 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.518+0000 7fd0099ff700 1 -- 172.21.15.81:0/2506383495 wait complete. 2022-09-10T04:28:34.547 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2022-09-10T04:28:34.605 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:34.345+0000 7f9b87366080 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2022-09-10T04:28:34.606 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-mon[90133]: from='client.? 172.21.15.81:0/932160056' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2022-09-10T04:28:34.606 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-mon[90133]: mgrmap e17: x(active, since 9m), standbys: z, y 2022-09-10T04:28:34.607 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' 2022-09-10T04:28:34.607 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-mon[90133]: from='client.? ' entity='client.admin' 2022-09-10T04:28:34.731 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:34.424+0000 7f143826d080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:28:34.732 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:34.511+0000 7f143826d080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:28:34.732 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:34.638+0000 7f143826d080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:28:34.732 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:34.730+0000 7f143826d080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:28:34.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-mon[97044]: from='client.? 172.21.15.81:0/932160056' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2022-09-10T04:28:34.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-mon[97044]: mgrmap e17: x(active, since 9m), standbys: z, y 2022-09-10T04:28:34.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' 2022-09-10T04:28:34.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:34 smithi104 ceph-mon[97044]: from='client.? ' entity='client.admin' 2022-09-10T04:28:34.734 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:34.426+0000 7f1e0820e080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:28:34.735 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:34.513+0000 7f1e0820e080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:28:34.735 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:34.641+0000 7f1e0820e080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:28:34.735 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-mon[97633]: from='client.? 172.21.15.81:0/932160056' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2022-09-10T04:28:34.736 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-mon[97633]: mgrmap e17: x(active, since 9m), standbys: z, y 2022-09-10T04:28:34.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' 2022-09-10T04:28:34.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-mon[97633]: from='client.? ' entity='client.admin' 2022-09-10T04:28:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.772+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/2337993626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100000 msgr2=0x7fdb08100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.779 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.772+0000 7fdb0f9ff700 1 --2- 172.21.15.81:0/2337993626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100000 0x7fdb08100410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009810 tx=0x7fdb04009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/2337993626 shutdown_connections 2022-09-10T04:28:34.780 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 --2- 172.21.15.81:0/2337993626 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb081019d0 0x7fdb08108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 --2- 172.21.15.81:0/2337993626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100000 0x7fdb08100410 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.781 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 --2- 172.21.15.81:0/2337993626 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb08100950 0x7fdb08101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/2337993626 >> 172.21.15.81:0/2337993626 conn(0x7fdb080fb710 msgr2=0x7fdb080fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/2337993626 shutdown_connections 2022-09-10T04:28:34.782 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.773+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/2337993626 wait complete. 2022-09-10T04:28:34.783 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 Processor -- start 2022-09-10T04:28:34.784 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 -- start start 2022-09-10T04:28:34.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb08100000 0x7fdb08195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.785 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100950 0x7fdb08196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb081019d0 0x7fdb0819d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0819d750 con 0x7fdb081019d0 2022-09-10T04:28:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0819d8c0 con 0x7fdb08100000 2022-09-10T04:28:34.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.774+0000 7fdb0f9ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0819da80 con 0x7fdb08100950 2022-09-10T04:28:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb08100000 0x7fdb08195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:34.788 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb08100000 0x7fdb08195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:45784/0 (socket says 172.21.15.81:45784) 2022-09-10T04:28:34.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 -- 172.21.15.81:0/3237569529 learned_addr learned my addr 172.21.15.81:0/3237569529 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:34.789 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100950 0x7fdb08196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:34.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0df9c700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb081019d0 0x7fdb0819d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:34.790 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100950 msgr2=0x7fdb08196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb08100950 0x7fdb08196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb081019d0 msgr2=0x7fdb0819d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:34.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb081019d0 0x7fdb0819d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:34.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 -- 172.21.15.81:0/3237569529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb00009730 con 0x7fdb08100000 2022-09-10T04:28:34.791 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.775+0000 7fdb0d79b700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb08100000 0x7fdb08195bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400f690 tx=0x7fdb0400f6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:34.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.776+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0401d070 con 0x7fdb08100000 2022-09-10T04:28:34.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.776+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb04009470 con 0x7fdb08100000 2022-09-10T04:28:34.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.776+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fdb040054f0 con 0x7fdb08100000 2022-09-10T04:28:34.792 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.776+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04004860 con 0x7fdb08100000 2022-09-10T04:28:34.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.776+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0819e020 con 0x7fdb08100000 2022-09-10T04:28:34.793 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.777+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0810b7f0 con 0x7fdb08100000 2022-09-10T04:28:34.794 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.778+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) v1 ==== 127695+0+0 (secure 0 0 0) 0x7fdb04028030 con 0x7fdb08100000 2022-09-10T04:28:34.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.780+0000 7fdafe7fc700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:34.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.780+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9074+0+0 (secure 0 0 0) 0x7fdb040c34a0 con 0x7fdb08100000 2022-09-10T04:28:34.795 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.780+0000 7fdb0cf9a700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:34.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.780+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2022-09-10T04:28:34.796 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.783+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7fdb0408f7e0 con 0x7fdb08100000 2022-09-10T04:28:34.982 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.981+0000 7fdb0cf9a700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:34.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:34.981+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2022-09-10T04:28:35.008 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:35.007+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 --> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb0805f120 con 0x7fdaf4099060 2022-09-10T04:28:35.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:34.603+0000 7f9b87366080 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2022-09-10T04:28:35.149 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:34 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:34.733+0000 7f1e0820e080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:28:35.376 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:35.061+0000 7f9b87366080 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2022-09-10T04:28:35.377 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:35.194+0000 7f9b87366080 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2022-09-10T04:28:35.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:35.382+0000 7fdb0cf9a700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:35.383 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:35.382+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2022-09-10T04:28:35.406 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:35 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:35.146+0000 7f1e0820e080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:28:35.546 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:35 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:35.142+0000 7f143826d080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:28:35.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:35.374+0000 7f9b87366080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2022-09-10T04:28:35.804 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:35.515+0000 7f9b87366080 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2022-09-10T04:28:36.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:35 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:35.699+0000 7f143826d080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:28:36.047 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:35 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:35.867+0000 7f143826d080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:28:36.157 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:35 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:35.707+0000 7f1e0820e080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:28:36.157 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:35 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:35.874+0000 7f1e0820e080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:28:36.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:36.183+0000 7fdb0cf9a700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:36.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:36.183+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2022-09-10T04:28:36.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:36.119+0000 7f9b87366080 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2022-09-10T04:28:36.872 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:36 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:36.615+0000 7f143826d080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:28:36.873 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:36 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:36.704+0000 7f143826d080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:28:36.875 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:36 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:36.617+0000 7f1e0820e080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:28:36.876 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:36 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:36.705+0000 7f1e0820e080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:28:37.156 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:36 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:36.873+0000 7f1e0820e080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:28:37.157 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:37 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:37.056+0000 7f1e0820e080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:28:37.196 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:36 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:36.871+0000 7f143826d080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:28:37.197 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:37 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:37.052+0000 7f143826d080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:28:37.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:36.957+0000 7f9b87366080 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2022-09-10T04:28:37.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:37.193+0000 7f9b87366080 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2022-09-10T04:28:37.546 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:37 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:37.196+0000 7f143826d080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:28:37.595 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:37 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:37.203+0000 7f1e0820e080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:28:37.786 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:37.785+0000 7fdb0cf9a700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:37.787 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:37.785+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2022-09-10T04:28:38.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:37 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:37.586+0000 7f143826d080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:28:38.405 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:38 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:38.103+0000 7f143826d080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:28:38.408 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:37 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:37.593+0000 7f1e0820e080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:28:38.408 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:38 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:38.112+0000 7f1e0820e080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:28:38.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:38.271+0000 7f9b87366080 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:28:38.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:38.419+0000 7f9b87366080 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2022-09-10T04:28:38.700 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:38 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:38.405+0000 7f143826d080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:28:38.700 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:38 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:38.589+0000 7f143826d080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:28:38.712 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:38 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:38.416+0000 7f1e0820e080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:28:38.712 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:38 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:38.600+0000 7f1e0820e080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:28:39.014 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:38.702+0000 7f9b87366080 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2022-09-10T04:28:39.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:38 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:38.699+0000 7f143826d080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:28:39.046 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:38 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:38.906+0000 7f143826d080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:28:39.157 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:38 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:38.710+0000 7f1e0820e080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:28:39.157 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:38 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:38.918+0000 7f1e0820e080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:28:39.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:39.012+0000 7f9b87366080 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2022-09-10T04:28:39.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:39.246+0000 7f9b87366080 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2022-09-10T04:28:39.607 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:39.606+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 18) v1 ==== 127695+0+0 (secure 0 0 0) 0x7fdb0400c040 con 0x7fdb08100000 2022-09-10T04:28:39.795 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-y[98403]: 2022-09-10T04:28:39.529+0000 7f143826d080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:28:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: Standby manager daemon y restarted 2022-09-10T04:28:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: Standby manager daemon y started 2022-09-10T04:28:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-10T04:28:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:28:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-10T04:28:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:28:39.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: Standby manager daemon z restarted 2022-09-10T04:28:39.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: Standby manager daemon z started 2022-09-10T04:28:39.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-10T04:28:39.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:28:39.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-10T04:28:39.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:39 smithi104 ceph-mon[97044]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:28:39.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: Standby manager daemon y restarted 2022-09-10T04:28:39.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: Standby manager daemon y started 2022-09-10T04:28:39.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-10T04:28:39.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:28:39.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-10T04:28:39.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:28:39.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: Standby manager daemon z restarted 2022-09-10T04:28:39.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: Standby manager daemon z started 2022-09-10T04:28:39.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-10T04:28:39.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:28:39.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-10T04:28:39.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-mon[90133]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:28:39.907 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-z[99591]: 2022-09-10T04:28:39.539+0000 7f1e0820e080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:28:39.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: Standby manager daemon y restarted 2022-09-10T04:28:39.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: Standby manager daemon y started 2022-09-10T04:28:39.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2022-09-10T04:28:39.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:28:39.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2022-09-10T04:28:39.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.104:0/2011760039' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:28:39.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: Standby manager daemon z restarted 2022-09-10T04:28:39.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: Standby manager daemon z started 2022-09-10T04:28:39.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2022-09-10T04:28:39.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2022-09-10T04:28:39.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2022-09-10T04:28:39.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:39 smithi110 ceph-mon[97633]: from='mgr.? 172.21.15.110:0/2718806046' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2022-09-10T04:28:40.304 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:39.864+0000 7f9b87366080 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2022-09-10T04:28:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:40 smithi110 ceph-mon[97633]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-09-10T04:28:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:40.988+0000 7fdb0cf9a700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/532852594 2022-09-10T04:28:40.990 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:40.988+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2022-09-10T04:28:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:40 smithi104 ceph-mon[97044]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-09-10T04:28:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:40 smithi081 ceph-mon[90133]: mgrmap e18: x(active, since 9m), standbys: y, z 2022-09-10T04:28:41.055 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:40.804+0000 7f9b87366080 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2022-09-10T04:28:41.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:41.227+0000 7f9b87366080 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2022-09-10T04:28:42.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:41.560+0000 7f9b87366080 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2022-09-10T04:28:42.054 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:41.750+0000 7f9b87366080 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2022-09-10T04:28:42.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:42.114+0000 7f9b87366080 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2022-09-10T04:28:43.204 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:43.202+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 19) v1 ==== 127422+0+0 (secure 0 0 0) 0x7fdb0400fba0 con 0x7fdb08100000 2022-09-10T04:28:43.204 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:43.203+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 msgr2=0x7fdaf409b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2022-09-10T04:28:43.205 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:43.203+0000 7fdafe7fc700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/532852594,v1:172.21.15.81:6801/532852594] conn(0x7fdaf4099060 0x7fdaf409b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:43 smithi104 ceph-mon[97044]: Active manager daemon x restarted 2022-09-10T04:28:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:43 smithi104 ceph-mon[97044]: Activating manager daemon x 2022-09-10T04:28:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:43 smithi104 ceph-mon[97044]: osdmap e95: 12 total, 12 up, 12 in 2022-09-10T04:28:43.554 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.134+0000 7f9b87366080 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-09-10T04:28:43.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-mon[90133]: Active manager daemon x restarted 2022-09-10T04:28:43.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-mon[90133]: Activating manager daemon x 2022-09-10T04:28:43.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-mon[90133]: osdmap e95: 12 total, 12 up, 12 in 2022-09-10T04:28:43.614 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:43 smithi110 ceph-mon[97633]: Active manager daemon x restarted 2022-09-10T04:28:43.615 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:43 smithi110 ceph-mon[97633]: Activating manager daemon x 2022-09-10T04:28:43.615 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:43 smithi110 ceph-mon[97633]: osdmap e95: 12 total, 12 up, 12 in 2022-09-10T04:28:43.925 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.682+0000 7f9ae9e49700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.926 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.682+0000 7f9ae9e49700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.926 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.682+0000 7f9ae9e49700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.927 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.682+0000 7f9ae9e49700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.927 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.682+0000 7f9ae9e49700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.928 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.685+0000 7f9aede51700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.928 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.685+0000 7f9aede51700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.929 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.685+0000 7f9aede51700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.929 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.685+0000 7f9aede51700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:43.930 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:28:43.685+0000 7f9aede51700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:28:44.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.239+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mon.2 v2:172.21.15.104:3300/0 9 ==== mgrmap(e 20) v1 ==== 127532+0+0 (secure 0 0 0) 0x7fdb0404e3f0 con 0x7fdb08100000 2022-09-10T04:28:44.241 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.240+0000 7fdafe7fc700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fdaf402efc0 0x7fdaf402f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:44.242 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.240+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 --> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb0805f120 con 0x7fdaf402efc0 2022-09-10T04:28:44.245 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.242+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fdaf402efc0 0x7fdaf402f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:44.247 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.245+0000 7fdb0cf9a700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fdaf402efc0 0x7fdaf402f390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdb08100d20 tx=0x7fdaf8005ea0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:44.313 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.312+0000 7fdafe7fc700 1 -- 172.21.15.81:0/3237569529 <== mgr.25379 v2:172.21.15.81:6800/3853788614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdb0805f120 con 0x7fdaf402efc0 2022-09-10T04:28:44.314 INFO:teuthology.orchestra.run.smithi081.stdout:Schedule set for path / 2022-09-10T04:28:44.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fdaf402efc0 msgr2=0x7fdaf402f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.316 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0f9ff700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fdaf402efc0 0x7fdaf402f390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdb08100d20 tx=0x7fdaf8005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb08100000 msgr2=0x7fdb08195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0f9ff700 1 --2- 172.21.15.81:0/3237569529 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdb08100000 0x7fdb08195bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400f690 tx=0x7fdb0400f6c0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.317 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0df9c700 1 -- 172.21.15.81:0/3237569529 reap_dead start 2022-09-10T04:28:44.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 shutdown_connections 2022-09-10T04:28:44.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.314+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 >> 172.21.15.81:0/3237569529 conn(0x7fdb080fb710 msgr2=0x7fdb08104860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:44.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.315+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 shutdown_connections 2022-09-10T04:28:44.318 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.315+0000 7fdb0f9ff700 1 -- 172.21.15.81:0/3237569529 wait complete. 2022-09-10T04:28:44.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2022-09-10T04:28:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: mgrmap e19: x(active, starting, since 0.0579582s), standbys: y, z 2022-09-10T04:28:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:28:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-10T04:28:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-10T04:28:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-10T04:28:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-10T04:28:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-10T04:28:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-10T04:28:44.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-10T04:28:44.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-10T04:28:44.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-10T04:28:44.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-10T04:28:44.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-10T04:28:44.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-10T04:28:44.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-10T04:28:44.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:28:44.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:28:44.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:28:44.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:28:44.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:28:44.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:28:44.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:28:44.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:28:44.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:28:44.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:28:44.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:28:44.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:28:44.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-10T04:28:44.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-10T04:28:44.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-10T04:28:44.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: Manager daemon x is now available 2022-09-10T04:28:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:28:44.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:28:44.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:44.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:44.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:44.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:44.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: mgrmap e19: x(active, starting, since 0.0579582s), standbys: y, z 2022-09-10T04:28:44.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:28:44.567 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:28:44.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:28:44.568 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:28:44.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:28:44.569 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-10T04:28:44.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-10T04:28:44.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-10T04:28:44.571 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-10T04:28:44.572 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-10T04:28:44.574 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-10T04:28:44.575 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-10T04:28:44.575 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-10T04:28:44.576 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-10T04:28:44.576 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-10T04:28:44.576 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-10T04:28:44.577 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-10T04:28:44.577 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-10T04:28:44.578 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:28:44.578 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:28:44.579 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:28:44.579 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:28:44.580 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:28:44.580 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:28:44.581 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:28:44.581 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:28:44.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:28:44.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:28:44.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:28:44.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:28:44.584 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-10T04:28:44.584 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-10T04:28:44.584 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-10T04:28:44.584 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: Manager daemon x is now available 2022-09-10T04:28:44.585 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:44.585 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:44.585 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:28:44.585 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:28:44.585 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:44.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:44.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:44.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:44 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:44.590 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 -- 172.21.15.81:0/555236390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 msgr2=0x7f9720103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 --2- 172.21.15.81:0/555236390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 0x7f9720103130 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f970c009810 tx=0x7f970c009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:44.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 -- 172.21.15.81:0/555236390 shutdown_connections 2022-09-10T04:28:44.592 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 --2- 172.21.15.81:0/555236390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f9720105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 --2- 172.21.15.81:0/555236390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9720066e80 0x7f9720100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 --2- 172.21.15.81:0/555236390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 0x7f9720103130 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 -- 172.21.15.81:0/555236390 >> 172.21.15.81:0/555236390 conn(0x7f97200f8180 msgr2=0x7f97200fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:44.593 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 -- 172.21.15.81:0/555236390 shutdown_connections 2022-09-10T04:28:44.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.574+0000 7f9725d9d700 1 -- 172.21.15.81:0/555236390 wait complete. 2022-09-10T04:28:44.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.575+0000 7f9725d9d700 1 Processor -- start 2022-09-10T04:28:44.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.575+0000 7f9725d9d700 1 -- start start 2022-09-10T04:28:44.594 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9725d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9720066e80 0x7f97201959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:44.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9725d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 0x7f9720195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:44.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9725d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f972019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:44.595 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9725d9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972019d520 con 0x7f9720100cd0 2022-09-10T04:28:44.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9725d9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972019d690 con 0x7f97201037c0 2022-09-10T04:28:44.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9725d9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972019d850 con 0x7f9720066e80 2022-09-10T04:28:44.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f972559c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f972019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:44.596 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f972559c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f972019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:55508/0 (socket says 172.21.15.81:55508) 2022-09-10T04:28:44.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f972559c700 1 -- 172.21.15.81:0/1298694614 learned_addr learned my addr 172.21.15.81:0/1298694614 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:44.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f971ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 0x7f9720195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:44.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.576+0000 7f9724d9b700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9720066e80 0x7f97201959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:44.597 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f972559c700 1 -- 172.21.15.81:0/1298694614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9720066e80 msgr2=0x7f97201959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f972559c700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9720066e80 0x7f97201959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f972559c700 1 -- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 msgr2=0x7f9720195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f972559c700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 0x7f9720195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.598 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f972559c700 1 -- 172.21.15.81:0/1298694614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9718009730 con 0x7f97201037c0 2022-09-10T04:28:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f972559c700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f972019cf50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f971800ec50 tx=0x7f971800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971800ccc0 con 0x7f97201037c0 2022-09-10T04:28:44.599 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f971800ce20 con 0x7f97201037c0 2022-09-10T04:28:44.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c009470 con 0x7f97201037c0 2022-09-10T04:28:44.600 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718010430 con 0x7f97201037c0 2022-09-10T04:28:44.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.577+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972019de60 con 0x7f97201037c0 2022-09-10T04:28:44.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.579+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97201093d0 con 0x7f97201037c0 2022-09-10T04:28:44.601 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.580+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 20) v1 ==== 127532+0+0 (secure 0 0 0) 0x7f971801e020 con 0x7f97201037c0 2022-09-10T04:28:44.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.581+0000 7f971dffb700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9710098f80 0x7f971009b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:44.602 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.582+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f9718014070 con 0x7f97201037c0 2022-09-10T04:28:44.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.582+0000 7f9724d9b700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9710098f80 0x7f971009b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:44.603 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.582+0000 7f9724d9b700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9710098f80 0x7f971009b430 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9720067250 tx=0x7f9714005ea0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:44.604 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.585+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f971808f060 con 0x7f97201037c0 2022-09-10T04:28:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: mgrmap e19: x(active, starting, since 0.0579582s), standbys: y, z 2022-09-10T04:28:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2022-09-10T04:28:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2022-09-10T04:28:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2022-09-10T04:28:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:28:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:28:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2022-09-10T04:28:44.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2022-09-10T04:28:44.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2022-09-10T04:28:44.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2022-09-10T04:28:44.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2022-09-10T04:28:44.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2022-09-10T04:28:44.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2022-09-10T04:28:44.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2022-09-10T04:28:44.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2022-09-10T04:28:44.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2022-09-10T04:28:44.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2022-09-10T04:28:44.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2022-09-10T04:28:44.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2022-09-10T04:28:44.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2022-09-10T04:28:44.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2022-09-10T04:28:44.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2022-09-10T04:28:44.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2022-09-10T04:28:44.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2022-09-10T04:28:44.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2022-09-10T04:28:44.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2022-09-10T04:28:44.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2022-09-10T04:28:44.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2022-09-10T04:28:44.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2022-09-10T04:28:44.668 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2022-09-10T04:28:44.668 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2022-09-10T04:28:44.668 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2022-09-10T04:28:44.669 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2022-09-10T04:28:44.669 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2022-09-10T04:28:44.669 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: Manager daemon x is now available 2022-09-10T04:28:44.670 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:44.670 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:44.670 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:28:44.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:28:44.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:44.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:28:44.672 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:44.672 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:28:44.836 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.834+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 --> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f97201054b0 con 0x7f9710098f80 2022-09-10T04:28:44.896 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.893+0000 7f971dffb700 1 -- 172.21.15.81:0/1298694614 <== mgr.25379 v2:172.21.15.81:6800/3853788614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f97201054b0 con 0x7f9710098f80 2022-09-10T04:28:44.896 INFO:teuthology.orchestra.run.smithi081.stdout:Retention added to path / 2022-09-10T04:28:44.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9710098f80 msgr2=0x7f971009b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.899 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9710098f80 0x7f971009b430 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9720067250 tx=0x7f9714005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 msgr2=0x7f972019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:44.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f972019cf50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f971800ec50 tx=0x7f971800ef60 comp rx=0 tx=0).stop 2022-09-10T04:28:44.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 shutdown_connections 2022-09-10T04:28:44.900 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9710098f80 0x7f971009b430 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f97201037c0 0x7f972019cf50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.901 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9720066e80 0x7f97201959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 --2- 172.21.15.81:0/1298694614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9720100cd0 0x7f9720195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:44.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 >> 172.21.15.81:0/1298694614 conn(0x7f97200f8180 msgr2=0x7f97200fa560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:44.902 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 shutdown_connections 2022-09-10T04:28:44.903 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:44.896+0000 7f9725d9d700 1 -- 172.21.15.81:0/1298694614 wait complete. 2022-09-10T04:28:44.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2022-09-10T04:28:45.163 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 -- 172.21.15.81:0/2643942646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078071740 msgr2=0x7ff078071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.164 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 --2- 172.21.15.81:0/2643942646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078071740 0x7ff078071bb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009810 tx=0x7ff06c009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:45.165 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 -- 172.21.15.81:0/2643942646 shutdown_connections 2022-09-10T04:28:45.165 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 --2- 172.21.15.81:0/2643942646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078105ad0 0x7ff078107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.165 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 --2- 172.21.15.81:0/2643942646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078072c90 0x7ff078071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.166 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 --2- 172.21.15.81:0/2643942646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078071740 0x7ff078071bb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.166 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 -- 172.21.15.81:0/2643942646 >> 172.21.15.81:0/2643942646 conn(0x7ff0780f92c0 msgr2=0x7ff0780fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 -- 172.21.15.81:0/2643942646 shutdown_connections 2022-09-10T04:28:45.167 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.161+0000 7ff07759e700 1 -- 172.21.15.81:0/2643942646 wait complete. 2022-09-10T04:28:45.172 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.163+0000 7ff07759e700 1 Processor -- start 2022-09-10T04:28:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.163+0000 7ff07759e700 1 -- start start 2022-09-10T04:28:45.173 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07759e700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 0x7ff078195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07759e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078072c90 0x7ff078195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.174 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078105ad0 0x7ff07819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07759e700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819d510 con 0x7ff078071740 2022-09-10T04:28:45.175 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07759e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819d680 con 0x7ff078072c90 2022-09-10T04:28:45.176 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07759e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07819d840 con 0x7ff078105ad0 2022-09-10T04:28:45.176 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 0x7ff078195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff075d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078072c90 0x7ff078195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff075d9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078072c90 0x7ff078195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:55534/0 (socket says 172.21.15.81:55534) 2022-09-10T04:28:45.177 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff07659c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 0x7ff078195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39024/0 (socket says 172.21.15.81:39024) 2022-09-10T04:28:45.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff075d9b700 1 -- 172.21.15.81:0/3498686193 learned_addr learned my addr 172.21.15.81:0/3498686193 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:45.178 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.164+0000 7ff076d9d700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078105ad0 0x7ff07819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07659c700 1 -- 172.21.15.81:0/3498686193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078105ad0 msgr2=0x7ff07819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07659c700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078105ad0 0x7ff07819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.179 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07659c700 1 -- 172.21.15.81:0/3498686193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078072c90 msgr2=0x7ff078195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07659c700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078072c90 0x7ff078195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.180 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07659c700 1 -- 172.21.15.81:0/3498686193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c009470 con 0x7ff078071740 2022-09-10T04:28:45.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff076d9d700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078105ad0 0x7ff07819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:45.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07659c700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 0x7ff078195990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff06800e6c0 tx=0x7ff06800e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:45.181 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068010900 con 0x7ff078071740 2022-09-10T04:28:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff068018420 con 0x7ff078071740 2022-09-10T04:28:45.182 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07819db20 con 0x7ff078071740 2022-09-10T04:28:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068017610 con 0x7ff078071740 2022-09-10T04:28:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.165+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07819e030 con 0x7ff078071740 2022-09-10T04:28:45.183 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.166+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07810b610 con 0x7ff078071740 2022-09-10T04:28:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.168+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 20) v1 ==== 127532+0+0 (secure 0 0 0) 0x7ff06800c040 con 0x7ff078071740 2022-09-10T04:28:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.169+0000 7ff0677fe700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff060098f80 0x7ff06009b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.170+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7ff068014070 con 0x7ff078071740 2022-09-10T04:28:45.184 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.170+0000 7ff075d9b700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff060098f80 0x7ff06009b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.170+0000 7ff075d9b700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff060098f80 0x7ff06009b430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00b4f0 tx=0x7ff06c009eb0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:45.185 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.172+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7ff06808ea00 con 0x7ff078071740 2022-09-10T04:28:45.246 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.244+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 21) v1 ==== 127815+0+0 (secure 0 0 0) 0x7ff0680188e0 con 0x7ff078071740 2022-09-10T04:28:45.426 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.425+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 --> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7ff07805f120 con 0x7ff060098f80 2022-09-10T04:28:45.437 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.433+0000 7ff0677fe700 1 -- 172.21.15.81:0/3498686193 <== mgr.25379 v2:172.21.15.81:6800/3853788614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7ff07805f120 con 0x7ff060098f80 2022-09-10T04:28:45.438 INFO:teuthology.orchestra.run.smithi081.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2022-09-10T00:00:00", "created": "2022-09-10T04:28:44", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2022-09-10T04:28:45.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff060098f80 msgr2=0x7ff06009b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.440 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff060098f80 0x7ff06009b430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff06c00b4f0 tx=0x7ff06c009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 msgr2=0x7ff078195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.441 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 0x7ff078195990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff06800e6c0 tx=0x7ff06800e9d0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 shutdown_connections 2022-09-10T04:28:45.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff078105ad0 0x7ff07819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.442 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff078071740 0x7ff078195990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff060098f80 0x7ff06009b430 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 --2- 172.21.15.81:0/3498686193 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff078072c90 0x7ff078195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.443 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 >> 172.21.15.81:0/3498686193 conn(0x7ff0780f92c0 msgr2=0x7ff0780fb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:45.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 shutdown_connections 2022-09-10T04:28:45.444 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.436+0000 7ff07759e700 1 -- 172.21.15.81:0/3498686193 wait complete. 2022-09-10T04:28:45.468 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2022-09-10T04:28:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: [10/Sep/2022:04:28:44] ENGINE Bus STARTING 2022-09-10T04:28:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: [10/Sep/2022:04:28:44] ENGINE Serving on https://:::8765 2022-09-10T04:28:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: [10/Sep/2022:04:28:44] ENGINE Serving on https://172.21.15.81:7150 2022-09-10T04:28:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: [10/Sep/2022:04:28:44] ENGINE Bus STARTED 2022-09-10T04:28:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: mgrmap e20: x(active, since 1.096s), standbys: y, z 2022-09-10T04:28:45.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: from='client.25397 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:45.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:45 smithi104 ceph-mon[97044]: pgmap v3: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:45.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: [10/Sep/2022:04:28:44] ENGINE Bus STARTING 2022-09-10T04:28:45.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: [10/Sep/2022:04:28:44] ENGINE Serving on https://:::8765 2022-09-10T04:28:45.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: [10/Sep/2022:04:28:44] ENGINE Serving on https://172.21.15.81:7150 2022-09-10T04:28:45.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: [10/Sep/2022:04:28:44] ENGINE Bus STARTED 2022-09-10T04:28:45.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: mgrmap e20: x(active, since 1.096s), standbys: y, z 2022-09-10T04:28:45.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: from='client.25397 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:45.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:45 smithi081 ceph-mon[90133]: pgmap v3: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: [10/Sep/2022:04:28:44] ENGINE Bus STARTING 2022-09-10T04:28:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: [10/Sep/2022:04:28:44] ENGINE Serving on https://:::8765 2022-09-10T04:28:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: [10/Sep/2022:04:28:44] ENGINE Serving on https://172.21.15.81:7150 2022-09-10T04:28:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: [10/Sep/2022:04:28:44] ENGINE Bus STARTED 2022-09-10T04:28:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: mgrmap e20: x(active, since 1.096s), standbys: y, z 2022-09-10T04:28:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: from='client.25397 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:45 smithi110 ceph-mon[97633]: pgmap v3: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:45.705 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 -- 172.21.15.81:0/4219186851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8270103b10 msgr2=0x7f8270105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.706 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 --2- 172.21.15.81:0/4219186851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8270103b10 0x7f8270105ef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8268009bb0 tx=0x7f8268009ec0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 -- 172.21.15.81:0/4219186851 shutdown_connections 2022-09-10T04:28:45.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 --2- 172.21.15.81:0/4219186851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8270103b10 0x7f8270105ef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.707 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 --2- 172.21.15.81:0/4219186851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 0x7f8270100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 --2- 172.21.15.81:0/4219186851 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f82701011f0 0x7f82701035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.708 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 -- 172.21.15.81:0/4219186851 >> 172.21.15.81:0/4219186851 conn(0x7f82700f83b0 msgr2=0x7f82700fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:45.709 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 -- 172.21.15.81:0/4219186851 shutdown_connections 2022-09-10T04:28:45.710 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.703+0000 7f8275fff700 1 -- 172.21.15.81:0/4219186851 wait complete. 2022-09-10T04:28:45.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 Processor -- start 2022-09-10T04:28:45.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 -- start start 2022-09-10T04:28:45.716 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 0x7f8270195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82701011f0 0x7f82701960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8270103b10 0x7f827019d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827019d6f0 con 0x7f82700fe8d0 2022-09-10T04:28:45.717 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827019d860 con 0x7f8270103b10 2022-09-10T04:28:45.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.705+0000 7f8275fff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827019da20 con 0x7f82701011f0 2022-09-10T04:28:45.718 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 0x7f8270195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82701011f0 0x7f82701960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82701011f0 0x7f82701960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:46184/0 (socket says 172.21.15.81:46184) 2022-09-10T04:28:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826effd700 1 -- 172.21.15.81:0/1849450496 learned_addr learned my addr 172.21.15.81:0/1849450496 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826ffff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8270103b10 0x7f827019d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.719 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 -- 172.21.15.81:0/1849450496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82701011f0 msgr2=0x7f82701960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82701011f0 0x7f82701960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 -- 172.21.15.81:0/1849450496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8270103b10 msgr2=0x7f827019d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8270103b10 0x7f827019d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.720 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 -- 172.21.15.81:0/1849450496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82680097e0 con 0x7f82700fe8d0 2022-09-10T04:28:45.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.706+0000 7f826f7fe700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 0x7f8270195b70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f826000e6c0 tx=0x7f826000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:45.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.707+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260010900 con 0x7f82700fe8d0 2022-09-10T04:28:45.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.707+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f8260018420 con 0x7f82700fe8d0 2022-09-10T04:28:45.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.707+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827019dd00 con 0x7f82700fe8d0 2022-09-10T04:28:45.721 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.707+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260017610 con 0x7f82700fe8d0 2022-09-10T04:28:45.722 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.707+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827019e1c0 con 0x7f82700fe8d0 2022-09-10T04:28:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.708+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270109550 con 0x7f82700fe8d0 2022-09-10T04:28:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.710+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 21) v1 ==== 127815+0+0 (secure 0 0 0) 0x7f826000c040 con 0x7f82700fe8d0 2022-09-10T04:28:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.711+0000 7f826cff9700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f8258099130 0x7f825809b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.711+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f8260014070 con 0x7f82700fe8d0 2022-09-10T04:28:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.711+0000 7f826effd700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f8258099130 0x7f825809b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.712+0000 7f826effd700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f8258099130 0x7f825809b5e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f82640057d0 tx=0x7f8264005ea0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.715+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+171119 (secure 0 0 0) 0x7f826004fc80 con 0x7f82700fe8d0 2022-09-10T04:28:45.973 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.971+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 --> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8270100440 con 0x7f8258099130 2022-09-10T04:28:45.980 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.978+0000 7f826cff9700 1 -- 172.21.15.81:0/1849450496 <== mgr.25379 v2:172.21.15.81:6800/3853788614 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f8270100440 con 0x7f8258099130 2022-09-10T04:28:45.980 INFO:teuthology.orchestra.run.smithi081.stdout:/ 1M 6M3h 2022-09-10T04:28:45.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f8258099130 msgr2=0x7f825809b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f8258099130 0x7f825809b5e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f82640057d0 tx=0x7f8264005ea0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.983 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 msgr2=0x7f8270195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:45.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 0x7f8270195b70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f826000e6c0 tx=0x7f826000e9d0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.984 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 shutdown_connections 2022-09-10T04:28:45.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f8258099130 0x7f825809b5e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8270103b10 0x7f827019d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.985 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82700fe8d0 0x7f8270195b70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 --2- 172.21.15.81:0/1849450496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82701011f0 0x7f82701960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 >> 172.21.15.81:0/1849450496 conn(0x7f82700f83b0 msgr2=0x7f82700ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 shutdown_connections 2022-09-10T04:28:45.987 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:28:45.981+0000 7f8275fff700 1 -- 172.21.15.81:0/1849450496 wait complete. 2022-09-10T04:28:46.008 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2022-09-10T04:28:46.156 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2022-09-10T04:28:46.173 INFO:teuthology.run_tasks:Running task background_exec... 2022-09-10T04:28:46.185 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:28:46.186 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2022-09-10T04:28:46.192 INFO:teuthology.run_tasks:Running task workunit... 2022-09-10T04:28:46.207 INFO:tasks.workunit:Pulling workunits from ref 244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:28:46.209 INFO:tasks.workunit:Making a separate scratch dir for every client... 2022-09-10T04:28:46.209 INFO:tasks.workunit:timeout=3h 2022-09-10T04:28:46.210 INFO:tasks.workunit:cleanup=True 2022-09-10T04:28:46.210 DEBUG:teuthology.orchestra.run.smithi081:> stat -- /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:46.268 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:46 smithi081 ceph-mon[90133]: from='client.25442 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:46.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:46 smithi081 ceph-mon[90133]: pgmap v4: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:46.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:46 smithi081 ceph-mon[90133]: mgrmap e21: x(active, since 2s), standbys: y, z 2022-09-10T04:28:46.270 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:46 smithi081 ceph-mon[90133]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: File: /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:46.280 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2022-09-10T04:28:46.281 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 78h/120d Inode: 1 Links: 2 2022-09-10T04:28:46.281 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2022-09-10T04:28:46.282 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:cephfs_t:s0 2022-09-10T04:28:46.282 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2022-09-10T04:28:46.283 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2022-09-10 04:26:32.102354648 +0000 2022-09-10T04:28:46.283 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2022-09-10 04:26:46.616573507 +0000 2022-09-10T04:28:46.283 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2022-09-10 04:26:32.102354648 +0000 2022-09-10T04:28:46.283 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2022-09-10T04:28:46.284 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2022-09-10T04:28:46.394 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 244418605927bb7842e73fc5d2a53889b760937f 2022-09-10T04:28:46.457 INFO:tasks.workunit.client.0.smithi081.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2022-09-10T04:28:46.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:46 smithi104 ceph-mon[97044]: from='client.25442 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:46.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:46 smithi104 ceph-mon[97044]: pgmap v4: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:46.517 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:46 smithi104 ceph-mon[97044]: mgrmap e21: x(active, since 2s), standbys: y, z 2022-09-10T04:28:46.518 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:46 smithi104 ceph-mon[97044]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:46.619 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:46 smithi110 ceph-mon[97633]: from='client.25442 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:46.619 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:46 smithi110 ceph-mon[97633]: pgmap v4: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:46.620 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:46 smithi110 ceph-mon[97633]: mgrmap e21: x(active, since 2s), standbys: y, z 2022-09-10T04:28:46.620 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:46 smithi110 ceph-mon[97633]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:47.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:47.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.681 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.682 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.682 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.682 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.682 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.683 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.684 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.686 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.686 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.733 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:47.734 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.735 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.735 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.735 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.735 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.736 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.736 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.736 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.736 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.736 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.737 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.738 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.738 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.738 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.738 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2022-09-10T04:28:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:47.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:47.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:47 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: pgmap v5: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi104 to 196.2M 2022-09-10T04:28:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Unable to set osd_memory_target on smithi104 to 205768908: error parsing value: Value '205768908' is below minimum 939524096 2022-09-10T04:28:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:28:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi110 to 196.2M 2022-09-10T04:28:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Unable to set osd_memory_target on smithi110 to 205747404: error parsing value: Value '205747404' is below minimum 939524096 2022-09-10T04:28:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:28:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:48.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:48.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:48.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:48.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: pgmap v5: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi104 to 196.2M 2022-09-10T04:28:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Unable to set osd_memory_target on smithi104 to 205768908: error parsing value: Value '205768908' is below minimum 939524096 2022-09-10T04:28:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:28:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi110 to 196.2M 2022-09-10T04:28:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Unable to set osd_memory_target on smithi110 to 205747404: error parsing value: Value '205747404' is below minimum 939524096 2022-09-10T04:28:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:28:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: pgmap v5: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi104 to 196.2M 2022-09-10T04:28:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Unable to set osd_memory_target on smithi104 to 205768908: error parsing value: Value '205768908' is below minimum 939524096 2022-09-10T04:28:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Updating smithi104:/etc/ceph/ceph.conf 2022-09-10T04:28:49.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi110 to 196.2M 2022-09-10T04:28:49.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Unable to set osd_memory_target on smithi110 to 205747404: error parsing value: Value '205747404' is below minimum 939524096 2022-09-10T04:28:49.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Updating smithi110:/etc/ceph/ceph.conf 2022-09-10T04:28:49.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:49.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:49.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:48 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:49 smithi110 ceph-mon[97633]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:49.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:49 smithi110 ceph-mon[97633]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:49.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:49 smithi110 ceph-mon[97633]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:49.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:49 smithi110 ceph-mon[97633]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:49 smithi104 ceph-mon[97044]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:49 smithi104 ceph-mon[97044]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:49 smithi104 ceph-mon[97044]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:49 smithi104 ceph-mon[97044]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:50.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:49 smithi081 ceph-mon[90133]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:50.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:49 smithi081 ceph-mon[90133]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:50.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:49 smithi081 ceph-mon[90133]: Updating smithi104:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:49 smithi081 ceph-mon[90133]: Updating smithi110:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:50.758 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: pgmap v6: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:50.759 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:50.759 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:50.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.761 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.761 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.762 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.762 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.763 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:50 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: pgmap v6: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:50.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:50.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:50 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: pgmap v6: 129 pgs: 129 active+clean; 972 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail 2022-09-10T04:28:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:50 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2022-09-10T04:28:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:51 smithi110 ceph-mon[97633]: Adjusting osd_memory_target on smithi081 to 221.9M 2022-09-10T04:28:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:51 smithi110 ceph-mon[97633]: Unable to set osd_memory_target on smithi081 to 232724172: error parsing value: Value '232724172' is below minimum 939524096 2022-09-10T04:28:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:51 smithi110 ceph-mon[97633]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:28:51.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:51 smithi081 ceph-mon[90133]: Adjusting osd_memory_target on smithi081 to 221.9M 2022-09-10T04:28:51.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:51 smithi081 ceph-mon[90133]: Unable to set osd_memory_target on smithi081 to 232724172: error parsing value: Value '232724172' is below minimum 939524096 2022-09-10T04:28:51.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:51 smithi081 ceph-mon[90133]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:28:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:51 smithi104 ceph-mon[97044]: Adjusting osd_memory_target on smithi081 to 221.9M 2022-09-10T04:28:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:51 smithi104 ceph-mon[97044]: Unable to set osd_memory_target on smithi081 to 232724172: error parsing value: Value '232724172' is below minimum 939524096 2022-09-10T04:28:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:51 smithi104 ceph-mon[97044]: Updating smithi081:/etc/ceph/ceph.conf 2022-09-10T04:28:52.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:52 smithi081 ceph-mon[90133]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:52.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:52 smithi081 ceph-mon[90133]: pgmap v7: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 132 KiB/s wr, 53 op/s 2022-09-10T04:28:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:52 smithi110 ceph-mon[97633]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:52 smithi110 ceph-mon[97633]: pgmap v7: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 132 KiB/s wr, 53 op/s 2022-09-10T04:28:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:52 smithi104 ceph-mon[97044]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.conf 2022-09-10T04:28:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:52 smithi104 ceph-mon[97044]: pgmap v7: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 132 KiB/s wr, 53 op/s 2022-09-10T04:28:53.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:53 smithi081 ceph-mon[90133]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:53.740 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:53 smithi081 ceph-mon[90133]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:53.882 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:53 smithi110 ceph-mon[97633]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:53.882 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:53 smithi110 ceph-mon[97633]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:53 smithi104 ceph-mon[97044]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2022-09-10T04:28:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:53 smithi104 ceph-mon[97044]: Updating smithi081:/var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/config/ceph.client.admin.keyring 2022-09-10T04:28:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: pgmap v8: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 103 KiB/s wr, 41 op/s 2022-09-10T04:28:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:28:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:28:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:54 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: pgmap v8: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 103 KiB/s wr, 41 op/s 2022-09-10T04:28:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:55.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:28:55.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:28:55.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:54 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: pgmap v8: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 103 KiB/s wr, 41 op/s 2022-09-10T04:28:55.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:55.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:28:55.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:28:55.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:28:55.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:54 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:28:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:56 smithi110 ceph-mon[97633]: pgmap v9: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 84 KiB/s wr, 38 op/s 2022-09-10T04:28:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:56 smithi104 ceph-mon[97044]: pgmap v9: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 84 KiB/s wr, 38 op/s 2022-09-10T04:28:57.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:56 smithi081 ceph-mon[90133]: pgmap v9: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 84 KiB/s wr, 38 op/s 2022-09-10T04:28:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:28:58 smithi104 ceph-mon[97044]: pgmap v10: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-10T04:28:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:28:58 smithi081 ceph-mon[90133]: pgmap v10: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-10T04:28:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:28:58 smithi110 ceph-mon[97633]: pgmap v10: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-10T04:28:59.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.485+0000 7fa17639d700 1 -- 172.21.15.81:0/2242053413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 msgr2=0x7fa170066170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.485+0000 7fa17639d700 1 --2- 172.21.15.81:0/2242053413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa170066170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009810 tx=0x7fa16c009b20 comp rx=0 tx=0).stop 2022-09-10T04:28:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 -- 172.21.15.81:0/2242053413 shutdown_connections 2022-09-10T04:28:59.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 --2- 172.21.15.81:0/2242053413 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa170100680 0x7fa170100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 --2- 172.21.15.81:0/2242053413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa170066170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 --2- 172.21.15.81:0/2242053413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa1700666b0 0x7fa170066b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 -- 172.21.15.81:0/2242053413 >> 172.21.15.81:0/2242053413 conn(0x7fa170074870 msgr2=0x7fa170074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 -- 172.21.15.81:0/2242053413 shutdown_connections 2022-09-10T04:28:59.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.486+0000 7fa17639d700 1 -- 172.21.15.81:0/2242053413 wait complete. 2022-09-10T04:28:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.487+0000 7fa17639d700 1 Processor -- start 2022-09-10T04:28:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.487+0000 7fa17639d700 1 -- start start 2022-09-10T04:28:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17639d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1700666b0 0x7fa170199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17639d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa170100680 0x7fa17019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17639d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa17019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17639d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17019ef00 con 0x7fa170102aa0 2022-09-10T04:28:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17639d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17019f070 con 0x7fa1700666b0 2022-09-10T04:28:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17639d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17019f230 con 0x7fa170100680 2022-09-10T04:28:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa175b9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa17019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa17539b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1700666b0 0x7fa170199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa174b9a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa170100680 0x7fa17019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa175b9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa17019e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54786/0 (socket says 172.21.15.81:54786) 2022-09-10T04:28:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.488+0000 7fa175b9c700 1 -- 172.21.15.81:0/2392502262 learned_addr learned my addr 172.21.15.81:0/2392502262 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa175b9c700 1 -- 172.21.15.81:0/2392502262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa170100680 msgr2=0x7fa17019a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa175b9c700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa170100680 0x7fa17019a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa175b9c700 1 -- 172.21.15.81:0/2392502262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1700666b0 msgr2=0x7fa170199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.497 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa175b9c700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1700666b0 0x7fa170199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa175b9c700 1 -- 172.21.15.81:0/2392502262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1600093b0 con 0x7fa170102aa0 2022-09-10T04:28:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa174b9a700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa170100680 0x7fa17019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa17539b700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1700666b0 0x7fa170199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:59.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa175b9c700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa17019e870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa16800ebf0 tx=0x7fa16800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa1667fc700 1 -- 172.21.15.81:0/2392502262 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680099c0 con 0x7fa170102aa0 2022-09-10T04:28:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16c009470 con 0x7fa170102aa0 2022-09-10T04:28:59.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa1667fc700 1 -- 172.21.15.81:0/2392502262 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa16800cc90 con 0x7fa170102aa0 2022-09-10T04:28:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa1667fc700 1 -- 172.21.15.81:0/2392502262 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168010430 con 0x7fa170102aa0 2022-09-10T04:28:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.489+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17019f6b0 con 0x7fa170102aa0 2022-09-10T04:28:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.494+0000 7fa1667fc700 1 -- 172.21.15.81:0/2392502262 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 21) v1 ==== 127815+0+0 (secure 0 0 0) 0x7fa16801e020 con 0x7fa170102aa0 2022-09-10T04:28:59.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.496+0000 7fa1667fc700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa15c099260 0x7fa15c09b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.496+0000 7fa1667fc700 1 -- 172.21.15.81:0/2392502262 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa168014070 con 0x7fa170102aa0 2022-09-10T04:28:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.496+0000 7fa17539b700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa15c099260 0x7fa15c09b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 Processor -- start 2022-09-10T04:28:59.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 -- start start 2022-09-10T04:28:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa17539b700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa15c099260 0x7fa15c09b710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009760 tx=0x7fa16c009eb0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x55803777d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037777b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037783f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x558037662070 con 0x5580377776f0 2022-09-10T04:28:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580376cc9d0 con 0x5580377780a0 2022-09-10T04:28:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.497+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580376c6170 con 0x55803777d200 2022-09-10T04:28:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x55803777d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa174b9a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037777b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x55803777d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:38478/0 (socket says 172.21.15.81:38478) 2022-09-10T04:28:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 -- 172.21.15.81:0/654740766 learned_addr learned my addr 172.21.15.81:0/654740766 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa175b9c700 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037783f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 -- 172.21.15.81:0/654740766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 msgr2=0x558037783f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037783f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 -- 172.21.15.81:0/654740766 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 msgr2=0x558037777b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037777b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.498+0000 7fa17539b700 1 -- 172.21.15.81:0/654740766 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55803754e5d0 con 0x55803777d200 2022-09-10T04:28:59.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.499+0000 7fa17539b700 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x55803777d5d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00bd90 tx=0x7fa16c020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f802c2d59a8362bf server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.499+0000 7fa15b7fe700 1 -- 172.21.15.81:0/654740766 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02a070 con 0x55803777d200 2022-09-10T04:28:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.499+0000 7fa15b7fe700 1 -- 172.21.15.81:0/654740766 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa16c01e040 con 0x55803777d200 2022-09-10T04:28:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.499+0000 7fa15b7fe700 1 -- 172.21.15.81:0/654740766 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c00f750 con 0x55803777d200 2022-09-10T04:28:59.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.499+0000 7fa18ab75b80 1 -- 172.21.15.81:0/654740766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 msgr2=0x55803777d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x55803777d5d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00bd90 tx=0x7fa16c020000 comp rx=0 tx=0).stop 2022-09-10T04:28:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 -- 172.21.15.81:0/654740766 shutdown_connections 2022-09-10T04:28:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037783f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x55803777d5d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/654740766 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037777b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 -- 172.21.15.81:0/654740766 >> 172.21.15.81:0/654740766 conn(0x5580375d76b0 msgr2=0x558037772010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 -- 172.21.15.81:0/654740766 shutdown_connections 2022-09-10T04:28:59.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.500+0000 7fa18ab75b80 1 -- 172.21.15.81:0/654740766 wait complete. 2022-09-10T04:28:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 -- start start 2022-09-10T04:28:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 Processor -- start 2022-09-10T04:28:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x558037783bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037780dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x558037781310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580375b40b0 con 0x55803777d200 2022-09-10T04:28:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580377818c0 con 0x5580377780a0 2022-09-10T04:28:59.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x558037781a80 con 0x5580377776f0 2022-09-10T04:28:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.501+0000 7fa18ab75b80 10 client.0 _ll_register_callbacks cb 0x558037671360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x558037781310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.517 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x558037781310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54798/0 (socket says 172.21.15.81:54798) 2022-09-10T04:28:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 -- 172.21.15.81:0/3537345723 learned_addr learned my addr 172.21.15.81:0/3537345723 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa174b9a700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037780dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa17539b700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x558037783bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 -- 172.21.15.81:0/3537345723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 msgr2=0x558037783bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x558037783bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 -- 172.21.15.81:0/3537345723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 msgr2=0x558037780dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037780dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.502+0000 7fa175b9c700 1 -- 172.21.15.81:0/3537345723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa170111a70 con 0x55803777d200 2022-09-10T04:28:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa175b9c700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x558037781310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa168005ab0 tx=0x7fa168092670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b53499604bb650a server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa158ff9700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:28:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa158ff9700 1 -- 172.21.15.81:0/3537345723 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680980d0 con 0x55803777d200 2022-09-10T04:28:59.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa158ff9700 1 -- 172.21.15.81:0/3537345723 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa168099f30 con 0x55803777d200 2022-09-10T04:28:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55803776a930 con 0x55803777d200 2022-09-10T04:28:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa158ff9700 1 -- 172.21.15.81:0/3537345723 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa168092ef0 con 0x55803777d200 2022-09-10T04:28:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa18ab75b80 10 client.15354 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5580374fad80 con 0x55803777d200 2022-09-10T04:28:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.503+0000 7fa158ff9700 1 -- 172.21.15.81:0/3537345723 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa168076070 con 0x55803777d200 2022-09-10T04:28:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa158ff9700 1 -- 172.21.15.81:0/3537345723 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa168099aa0 con 0x55803777d200 2022-09-10T04:28:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa18ab75b80 10 client.15354 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa18ab75b80 10 client.15354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:28:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa18ab75b80 20 client.15354 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:59.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377914a0 0x5580377938b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa18ab75b80 4 client.15354 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:28:59.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580377474b0 con 0x5580377914a0 2022-09-10T04:28:59.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.504+0000 7fa17539b700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377914a0 0x5580377938b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.505+0000 7fa17539b700 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377914a0 0x5580377938b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7869bbaae9d229f server_cookie=ae923479c7205986 in_seq=0 out_seq=0 2022-09-10T04:28:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.505+0000 7fa158ff9700 10 client.15354 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa158ff9700 1 -- 172.21.15.81:0/3537345723 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5580377474b0 con 0x5580377914a0 2022-09-10T04:28:59.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa158ff9700 10 client.15354 handle_command_reply: tid=0 2022-09-10T04:28:59.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 client.15354 shutdown 2022-09-10T04:28:59.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 msgr2=0x558037781310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x558037781310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa168005ab0 tx=0x7fa168092670 comp rx=0 tx=0).stop 2022-09-10T04:28:59.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 shutdown_connections 2022-09-10T04:28:59.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x558037781310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x558037783bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377914a0 0x5580377938b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/3537345723 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x558037780dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 >> 172.21.15.81:0/3537345723 conn(0x5580375d76b0 msgr2=0x5580377766e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa158ff9700 0 client.15354 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 shutdown_connections 2022-09-10T04:28:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.506+0000 7fa18ab75b80 1 -- 172.21.15.81:0/3537345723 wait complete. 2022-09-10T04:28:59.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.507+0000 7fa18ab75b80 20 client.15354 trim_cache size 0 max 16384 2022-09-10T04:28:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.524+0000 7fa18ab75b80 1 Processor -- start 2022-09-10T04:28:59.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.524+0000 7fa18ab75b80 1 -- start start 2022-09-10T04:28:59.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x55803777c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377836f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x5580377766e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55803777cbf0 con 0x55803777d200 2022-09-10T04:28:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55803776ea00 con 0x5580377780a0 2022-09-10T04:28:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x558037798460 con 0x5580377776f0 2022-09-10T04:28:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa174b9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377836f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa174b9a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377836f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:53114/0 (socket says 172.21.15.81:53114) 2022-09-10T04:28:59.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa174b9a700 1 -- 172.21.15.81:0/1337783143 learned_addr learned my addr 172.21.15.81:0/1337783143 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:59.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa175b9c700 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x5580377766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.525+0000 7fa17539b700 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x55803777c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa17539b700 1 -- 172.21.15.81:0/1337783143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 msgr2=0x5580377836f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa17539b700 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377836f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa17539b700 1 -- 172.21.15.81:0/1337783143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 msgr2=0x5580377766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa17539b700 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x5580377766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa17539b700 1 -- 172.21.15.81:0/1337783143 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580374f9250 con 0x5580377776f0 2022-09-10T04:28:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa174b9a700 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377836f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:28:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa17539b700 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x55803777c6b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009810 tx=0x7fa16c004e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=781860c8309456e4 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa15bfff700 1 -- 172.21.15.81:0/1337783143 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c02a040 con 0x5580377776f0 2022-09-10T04:28:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa15bfff700 1 -- 172.21.15.81:0/1337783143 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa16c00bec0 con 0x5580377776f0 2022-09-10T04:28:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.526+0000 7fa15bfff700 1 -- 172.21.15.81:0/1337783143 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c004b10 con 0x5580377776f0 2022-09-10T04:28:59.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1337783143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 msgr2=0x55803777c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x55803777c6b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009810 tx=0x7fa16c004e50 comp rx=0 tx=0).stop 2022-09-10T04:28:59.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1337783143 shutdown_connections 2022-09-10T04:28:59.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55803777d200 0x5580377766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5580377776f0 0x55803777c6b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1337783143 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377836f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.541 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1337783143 >> 172.21.15.81:0/1337783143 conn(0x5580375d76b0 msgr2=0x55803777bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:59.541 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1337783143 shutdown_connections 2022-09-10T04:28:59.541 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.527+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1337783143 wait complete. 2022-09-10T04:28:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.528+0000 7fa18ab75b80 1 -- start start 2022-09-10T04:28:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.528+0000 7fa18ab75b80 1 Processor -- start 2022-09-10T04:28:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.528+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037785c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.528+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377861a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa18ab75b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x5580377e1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x558037786ab0 con 0x5580377776f0 2022-09-10T04:28:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580377e21b0 con 0x5580377780a0 2022-09-10T04:28:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa18ab75b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580377e22f0 con 0x55803777d200 2022-09-10T04:28:59.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa18ab75b80 10 client.0 _ll_register_callbacks cb 0x558037671360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:28:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa17539b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037785c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x5580377e1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x5580377e1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:38490/0 (socket says 172.21.15.81:38490) 2022-09-10T04:28:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 -- 172.21.15.81:0/1131258407 learned_addr learned my addr 172.21.15.81:0/1131258407 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:28:59.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa174b9a700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377861a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 -- 172.21.15.81:0/1131258407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 msgr2=0x5580377861a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377861a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 -- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 msgr2=0x558037785c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:28:59.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037785c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa175b9c700 1 -- 172.21.15.81:0/1131258407 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa160009e30 con 0x55803777d200 2022-09-10T04:28:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.529+0000 7fa174b9a700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:28:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa175b9c700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x5580377e1c70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa16808f190 tx=0x7fa16800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=701a2ab44b6c70e9 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:28:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa158ff9700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:28:59.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa158ff9700 1 -- 172.21.15.81:0/1131258407 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1680980d0 con 0x55803777d200 2022-09-10T04:28:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa158ff9700 1 -- 172.21.15.81:0/1131258407 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa168099f30 con 0x55803777d200 2022-09-10T04:28:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580376bf6e0 con 0x55803777d200 2022-09-10T04:28:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa158ff9700 1 -- 172.21.15.81:0/1131258407 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa16808ea80 con 0x55803777d200 2022-09-10T04:28:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa18ab75b80 10 client.25480 fetch_fsmap learned FSMap version 28 2022-09-10T04:28:59.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55803754e5d0 con 0x55803777d200 2022-09-10T04:28:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.530+0000 7fa158ff9700 1 -- 172.21.15.81:0/1131258407 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa1680a0020 con 0x55803777d200 2022-09-10T04:28:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa158ff9700 1 -- 172.21.15.81:0/1131258407 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa168092f80 con 0x55803777d200 2022-09-10T04:28:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa18ab75b80 10 client.25480 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:28:59.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa18ab75b80 10 client.25480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:28:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa18ab75b80 20 client.25480 populate_metadata read hostname 'smithi081' 2022-09-10T04:28:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377e35d0 0x5580377e59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:28:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa18ab75b80 4 client.25480 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:28:59.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5580376bf400 con 0x5580377e35d0 2022-09-10T04:28:59.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.531+0000 7fa17539b700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377e35d0 0x5580377e59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:28:59.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.532+0000 7fa158ff9700 10 client.25480 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:59.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.532+0000 7fa17539b700 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377e35d0 0x5580377e59e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1024d6510451b8a0 server_cookie=7f2d4eed727e23d9 in_seq=0 out_seq=0 2022-09-10T04:28:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.540+0000 7fa158ff9700 1 -- 172.21.15.81:0/1131258407 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5580376bf400 con 0x5580377e35d0 2022-09-10T04:28:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.540+0000 7fa158ff9700 10 client.25480 handle_command_reply: tid=0 2022-09-10T04:28:59.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.540+0000 7fa18ab75b80 1 client.25480 shutdown 2022-09-10T04:28:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 msgr2=0x5580377e1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x5580377e1c70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa16808f190 tx=0x7fa16800ef70 comp rx=0 tx=0).stop 2022-09-10T04:28:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 shutdown_connections 2022-09-10T04:28:59.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55803777d200 0x5580377e1c70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5580377776f0 0x558037785c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5580377e35d0 0x5580377e59e0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 --2- 172.21.15.81:0/1131258407 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5580377780a0 0x5580377861a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 >> 172.21.15.81:0/1131258407 conn(0x5580375d76b0 msgr2=0x55803776bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa158ff9700 0 client.25480 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:28:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 shutdown_connections 2022-09-10T04:28:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 1 -- 172.21.15.81:0/1131258407 wait complete. 2022-09-10T04:28:59.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.541+0000 7fa18ab75b80 20 client.25480 trim_cache size 0 max 16384 2022-09-10T04:28:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:28:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:28:59.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:28:59.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:28:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa15c099260 msgr2=0x7fa15c09b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa15c099260 0x7fa15c09b710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009760 tx=0x7fa16c009eb0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 msgr2=0x7fa17019e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:28:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa17019e870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa16800ebf0 tx=0x7fa16800ef00 comp rx=0 tx=0).stop 2022-09-10T04:28:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 shutdown_connections 2022-09-10T04:28:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa170102aa0 0x7fa17019e870 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa15c099260 0x7fa15c09b710 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa1700666b0 0x7fa170199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 --2- 172.21.15.81:0/2392502262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa170100680 0x7fa17019a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:28:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 >> 172.21.15.81:0/2392502262 conn(0x7fa170074870 msgr2=0x7fa1700ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:28:59.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.542+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 shutdown_connections 2022-09-10T04:28:59.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:28:59.543+0000 7fa17639d700 1 -- 172.21.15.81:0/2392502262 wait complete. 2022-09-10T04:29:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:00 smithi104 ceph-mon[97044]: pgmap v11: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-10T04:29:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:00 smithi081 ceph-mon[90133]: pgmap v11: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-10T04:29:01.055 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:29:00.905+0000 7f9acc30e700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:29:01.055 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:29:00.905+0000 7f9acc30e700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:29:01.055 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:29:00.905+0000 7f9acc30e700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:29:01.055 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:29:00.905+0000 7f9acc30e700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:29:01.056 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Sep 10 04:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mgr-x[90380]: 2022-09-10T04:29:00.905+0000 7f9acc30e700 -1 client.0 error registering admin socket command: (17) File exists 2022-09-10T04:29:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:00 smithi110 ceph-mon[97633]: pgmap v11: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 35 op/s 2022-09-10T04:29:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:02 smithi104 ceph-mon[97044]: pgmap v12: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 39 op/s 2022-09-10T04:29:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:02 smithi104 ceph-mon[97044]: mgrmap e22: x(active, since 18s), standbys: y, z 2022-09-10T04:29:03.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:02 smithi081 ceph-mon[90133]: pgmap v12: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 39 op/s 2022-09-10T04:29:03.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:02 smithi081 ceph-mon[90133]: mgrmap e22: x(active, since 18s), standbys: y, z 2022-09-10T04:29:03.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:02 smithi110 ceph-mon[97633]: pgmap v12: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 39 op/s 2022-09-10T04:29:03.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:02 smithi110 ceph-mon[97633]: mgrmap e22: x(active, since 18s), standbys: y, z 2022-09-10T04:29:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:04 smithi104 ceph-mon[97044]: pgmap v13: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:29:05.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:04 smithi081 ceph-mon[90133]: pgmap v13: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:29:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:04 smithi110 ceph-mon[97633]: pgmap v13: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:29:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:06 smithi104 ceph-mon[97044]: pgmap v14: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2022-09-10T04:29:07.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:06 smithi081 ceph-mon[90133]: pgmap v14: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2022-09-10T04:29:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:06 smithi110 ceph-mon[97633]: pgmap v14: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2022-09-10T04:29:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:08 smithi104 ceph-mon[97044]: pgmap v15: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:09.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:08 smithi081 ceph-mon[90133]: pgmap v15: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:08 smithi110 ceph-mon[97633]: pgmap v15: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:10 smithi104 ceph-mon[97044]: pgmap v16: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:10 smithi081 ceph-mon[90133]: pgmap v16: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:10 smithi110 ceph-mon[97633]: pgmap v16: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 -- 172.21.15.81:0/3959765040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8104580 msgr2=0x7fa5d8066170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.801 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/3959765040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8104580 0x7fa5d8066170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009810 tx=0x7fa5cc009b20 comp rx=0 tx=0).stop 2022-09-10T04:29:12.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 -- 172.21.15.81:0/3959765040 shutdown_connections 2022-09-10T04:29:12.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/3959765040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d8066850 0x7fa5d810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/3959765040 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 0x7fa5d8103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/3959765040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8104580 0x7fa5d8066170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 -- 172.21.15.81:0/3959765040 >> 172.21.15.81:0/3959765040 conn(0x7fa5d8074870 msgr2=0x7fa5d8074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:12.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.799+0000 7fa5dd39d700 1 -- 172.21.15.81:0/3959765040 shutdown_connections 2022-09-10T04:29:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.800+0000 7fa5dd39d700 1 -- 172.21.15.81:0/3959765040 wait complete. 2022-09-10T04:29:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 Processor -- start 2022-09-10T04:29:12.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 -- start start 2022-09-10T04:29:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 0x7fa5d80691b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 0x7fa5d80696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d8104580 0x7fa5d81a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81a84e0 con 0x7fa5d8104580 2022-09-10T04:29:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81a8650 con 0x7fa5d8103be0 2022-09-10T04:29:12.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.802+0000 7fa5dd39d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d81a8810 con 0x7fa5d8066850 2022-09-10T04:29:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 0x7fa5d80691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 0x7fa5d80691b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59280/0 (socket says 172.21.15.81:59280) 2022-09-10T04:29:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 -- 172.21.15.81:0/2909019993 learned_addr learned my addr 172.21.15.81:0/2909019993 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:12.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 0x7fa5d80696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d8104580 0x7fa5d81a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 -- 172.21.15.81:0/2909019993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 msgr2=0x7fa5d80696f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 0x7fa5d80696f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 -- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d8104580 msgr2=0x7fa5d81a7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d8104580 0x7fa5d81a7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.803+0000 7fa5d7fff700 1 -- 172.21.15.81:0/2909019993 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc009470 con 0x7fa5d8066850 2022-09-10T04:29:12.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.804+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 0x7fa5d80696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:29:12.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.804+0000 7fa5d7fff700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 0x7fa5d80691b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800d700 tx=0x7fa5c800da10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:12.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.804+0000 7fa5d57fa700 1 -- 172.21.15.81:0/2909019993 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c800f890 con 0x7fa5d8066850 2022-09-10T04:29:12.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.804+0000 7fa5d57fa700 1 -- 172.21.15.81:0/2909019993 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa5c800fe90 con 0x7fa5d8066850 2022-09-10T04:29:12.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.804+0000 7fa5d57fa700 1 -- 172.21.15.81:0/2909019993 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c800e3b0 con 0x7fa5d8066850 2022-09-10T04:29:12.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.805+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d80fcc70 con 0x7fa5d8066850 2022-09-10T04:29:12.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.806+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d80fd100 con 0x7fa5d8066850 2022-09-10T04:29:12.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.811+0000 7fa5d57fa700 1 -- 172.21.15.81:0/2909019993 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) v1 ==== 127855+0+0 (secure 0 0 0) 0x7fa5c8019030 con 0x7fa5d8066850 2022-09-10T04:29:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.813+0000 7fa5f1b69b80 1 Processor -- start 2022-09-10T04:29:12.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.813+0000 7fa5f1b69b80 1 -- start start 2022-09-10T04:29:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85e2a10 0x55e4b85e2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5d57fa700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa5c0099360 0x7fa5c009b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dd8b0 0x55e4b85e9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.839 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b84bf070 con 0x55e4b85e2a10 2022-09-10T04:29:12.839 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85299d0 con 0x55e4b85dcf00 2022-09-10T04:29:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b8523170 con 0x55e4b85dd8b0 2022-09-10T04:29:12.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.814+0000 7fa5d57fa700 1 -- 172.21.15.81:0/2909019993 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa5c80c2510 con 0x7fa5d8066850 2022-09-10T04:29:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85e2a10 0x55e4b85e2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.841 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5d77fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85e2a10 0x55e4b85e2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58346/0 (socket says 172.21.15.81:58346) 2022-09-10T04:29:12.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/1737451691 learned_addr learned my addr 172.21.15.81:0/1737451691 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dd8b0 0x55e4b85e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5d77fe700 1 -- 172.21.15.81:0/1737451691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dd8b0 msgr2=0x55e4b85e9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dd8b0 0x55e4b85e9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5d77fe700 1 -- 172.21.15.81:0/1737451691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85e2a10 msgr2=0x55e4b85e2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85e2a10 0x55e4b85e2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.815+0000 7fa5d77fe700 1 -- 172.21.15.81:0/1737451691 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4b83ab5d0 con 0x55e4b85dcf00 2022-09-10T04:29:12.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.816+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd370 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc005a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b1164467958d65c server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:12.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.816+0000 7fa5d7fff700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa5c0099360 0x7fa5c009b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.816+0000 7fa5be7fc700 1 -- 172.21.15.81:0/1737451691 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc01cdc0 con 0x55e4b85dcf00 2022-09-10T04:29:12.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.816+0000 7fa5be7fc700 1 -- 172.21.15.81:0/1737451691 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa5cc0054e0 con 0x55e4b85dcf00 2022-09-10T04:29:12.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.816+0000 7fa5be7fc700 1 -- 172.21.15.81:0/1737451691 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc00b560 con 0x55e4b85dcf00 2022-09-10T04:29:12.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/1737451691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85dcf00 msgr2=0x55e4b85dd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5d7fff700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa5c0099360 0x7fa5c009b810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80105a0 tx=0x7fa5c8010000 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:12.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd370 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc005a80 comp rx=0 tx=0).stop 2022-09-10T04:29:12.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/1737451691 shutdown_connections 2022-09-10T04:29:12.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dd8b0 0x55e4b85e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85e2a10 0x55e4b85e2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/1737451691 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd370 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/1737451691 >> 172.21.15.81:0/1737451691 conn(0x55e4b84346b0 msgr2=0x55e4b85cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:12.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/1737451691 shutdown_connections 2022-09-10T04:29:12.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.817+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/1737451691 wait complete. 2022-09-10T04:29:12.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.818+0000 7fa5f1b69b80 1 -- start start 2022-09-10T04:29:12.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.819+0000 7fa5f1b69b80 1 Processor -- start 2022-09-10T04:29:12.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85d5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b84110b0 con 0x55e4b85dd8b0 2022-09-10T04:29:12.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d1fa0 con 0x55e4b85e2a10 2022-09-10T04:29:12.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d2160 con 0x55e4b85dcf00 2022-09-10T04:29:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5f1b69b80 10 client.0 _ll_register_callbacks cb 0x55e4b84ce360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:12.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85d5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85d5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59302/0 (socket says 172.21.15.81:59302) 2022-09-10T04:29:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/3097273748 learned_addr learned my addr 172.21.15.81:0/3097273748 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:12.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.820+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/3097273748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 msgr2=0x55e4b85d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 msgr2=0x55e4b85d6380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-09-10T04:29:12.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d6380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/3097273748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d810b290 con 0x55e4b85dcf00 2022-09-10T04:29:12.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85d5e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00103d0 tx=0x7fa5d0009d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da05e51689f22c21 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.821+0000 7fa5b37fe700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:29:12.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.822+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3097273748 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0010d70 con 0x55e4b85dcf00 2022-09-10T04:29:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.822+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e4b85c7930 con 0x55e4b85dcf00 2022-09-10T04:29:12.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.822+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3097273748 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa5d0003880 con 0x55e4b85dcf00 2022-09-10T04:29:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.822+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3097273748 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5d0010ed0 con 0x55e4b85dcf00 2022-09-10T04:29:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.822+0000 7fa5f1b69b80 10 client.25498 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:12.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.822+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e4b8357d80 con 0x55e4b85dcf00 2022-09-10T04:29:12.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3097273748 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55e4b85c7930 con 0x55e4b85dcf00 2022-09-10T04:29:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3097273748 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa5d0020470 con 0x55e4b85dcf00 2022-09-10T04:29:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5f1b69b80 10 client.25498 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:12.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5f1b69b80 10 client.25498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5f1b69b80 20 client.25498 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:12.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85ff3d0 0x55e4b86017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5f1b69b80 4 client.25498 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:29:12.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.823+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4b85a44b0 con 0x55e4b85ff3d0 2022-09-10T04:29:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.824+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85ff3d0 0x55e4b86017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.825+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85ff3d0 0x55e4b86017e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e4bb2249e0aded6 server_cookie=d4a4425e8574f56c in_seq=0 out_seq=0 2022-09-10T04:29:12.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.825+0000 7fa5b37fe700 10 client.25498 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.826+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3097273748 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55e4b85a44b0 con 0x55e4b85ff3d0 2022-09-10T04:29:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.826+0000 7fa5b37fe700 10 client.25498 handle_command_reply: tid=0 2022-09-10T04:29:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.826+0000 7fa5f1b69b80 1 client.25498 shutdown 2022-09-10T04:29:12.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.826+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 msgr2=0x55e4b85d5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.826+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85d5e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00103d0 tx=0x7fa5d0009d40 comp rx=0 tx=0).stop 2022-09-10T04:29:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 shutdown_connections 2022-09-10T04:29:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85ff3d0 0x55e4b86017e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85d5e40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3097273748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 >> 172.21.15.81:0/3097273748 conn(0x55e4b84346b0 msgr2=0x55e4b85e0390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5b37fe700 0 client.25498 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 shutdown_connections 2022-09-10T04:29:12.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.827+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3097273748 wait complete. 2022-09-10T04:29:12.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.828+0000 7fa5f1b69b80 20 client.25498 trim_cache size 0 max 16384 2022-09-10T04:29:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.855+0000 7fa5f1b69b80 1 Processor -- start 2022-09-10T04:29:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.855+0000 7fa5f1b69b80 1 -- start start 2022-09-10T04:29:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85e13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85c9950 con 0x55e4b85dd8b0 2022-09-10T04:29:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d2790 con 0x55e4b85e2a10 2022-09-10T04:29:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d0dc0 con 0x55e4b85dcf00 2022-09-10T04:29:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59308/0 (socket says 172.21.15.81:59308) 2022-09-10T04:29:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.856+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/2708700300 learned_addr learned my addr 172.21.15.81:0/2708700300 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85e13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/2708700300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 msgr2=0x55e4b85dd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/2708700300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 msgr2=0x55e4b85c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/2708700300 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4b8356250 con 0x55e4b85e2a10 2022-09-10T04:29:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2022-09-10T04:29:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.857+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85e13a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00096e0 tx=0x7fa5d0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ecfb051cf43a105 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.858+0000 7fa5beffd700 1 -- 172.21.15.81:0/2708700300 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0009a20 con 0x55e4b85e2a10 2022-09-10T04:29:12.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.858+0000 7fa5beffd700 1 -- 172.21.15.81:0/2708700300 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa5d0010d70 con 0x55e4b85e2a10 2022-09-10T04:29:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.858+0000 7fa5beffd700 1 -- 172.21.15.81:0/2708700300 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0020e20 con 0x55e4b85e2a10 2022-09-10T04:29:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/2708700300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 msgr2=0x55e4b85e13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85e13a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00096e0 tx=0x7fa5d0009f90 comp rx=0 tx=0).stop 2022-09-10T04:29:12.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/2708700300 shutdown_connections 2022-09-10T04:29:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85e13a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85dd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/2708700300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/2708700300 >> 172.21.15.81:0/2708700300 conn(0x55e4b84346b0 msgr2=0x55e4b85d6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/2708700300 shutdown_connections 2022-09-10T04:29:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.859+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/2708700300 wait complete. 2022-09-10T04:29:12.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.860+0000 7fa5f1b69b80 1 -- start start 2022-09-10T04:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.860+0000 7fa5f1b69b80 1 Processor -- start 2022-09-10T04:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85db850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d8780 con 0x55e4b85dd8b0 2022-09-10T04:29:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d88f0 con 0x55e4b85e2a10 2022-09-10T04:29:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4b85d8ab0 con 0x55e4b85dcf00 2022-09-10T04:29:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5f1b69b80 10 client.0 _ll_register_callbacks cb 0x55e4b84ce360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:12.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85db850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5dcb9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85db850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:59324/0 (socket says 172.21.15.81:59324) 2022-09-10T04:29:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5dcb9c700 1 -- 172.21.15.81:0/3911851937 learned_addr learned my addr 172.21.15.81:0/3911851937 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.862+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.861+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.862+0000 7fa5d77fe700 1 -- 172.21.15.81:0/3911851937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 msgr2=0x55e4b85db850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.862+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85db850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.862+0000 7fa5d77fe700 1 -- 172.21.15.81:0/3911851937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 msgr2=0x55e4b85d8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:12.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.862+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.862+0000 7fa5d77fe700 1 -- 172.21.15.81:0/3911851937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4b8357d80 con 0x55e4b85dd8b0 2022-09-10T04:29:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.863+0000 7fa5d77fe700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d7d00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31db25f08dc3bf75 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.863+0000 7fa5b37fe700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:29:12.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.863+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3911851937 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc02a070 con 0x55e4b85dd8b0 2022-09-10T04:29:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.863+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3911851937 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7fa5cc01cdc0 con 0x55e4b85dd8b0 2022-09-10T04:29:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.863+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5cc0054e0 con 0x55e4b85dd8b0 2022-09-10T04:29:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.864+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3911851937 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5cc02e410 con 0x55e4b85dd8b0 2022-09-10T04:29:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.864+0000 7fa5f1b69b80 10 client.15378 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.864+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e4b8424a20 con 0x55e4b85dd8b0 2022-09-10T04:29:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.864+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3911851937 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7fa5cc0054e0 con 0x55e4b85dd8b0 2022-09-10T04:29:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3911851937 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7fa5cc036070 con 0x55e4b85dd8b0 2022-09-10T04:29:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5f1b69b80 10 client.15378 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5f1b69b80 10 client.15378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5f1b69b80 20 client.15378 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85e9c70 0x55e4b85ec050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:12.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5f1b69b80 4 client.15378 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:29:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.865+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e4b851c6e0 con 0x55e4b85e9c70 2022-09-10T04:29:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.866+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85e9c70 0x55e4b85ec050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.867+0000 7fa5dcb9c700 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85e9c70 0x55e4b85ec050 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87074b134388c28a server_cookie=fb34320152b10cbe in_seq=0 out_seq=0 2022-09-10T04:29:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.867+0000 7fa5b37fe700 10 client.15378 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:12.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5b37fe700 1 -- 172.21.15.81:0/3911851937 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e4b851c6e0 con 0x55e4b85e9c70 2022-09-10T04:29:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5b37fe700 10 client.15378 handle_command_reply: tid=0 2022-09-10T04:29:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 client.15378 shutdown 2022-09-10T04:29:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 msgr2=0x55e4b85d7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d7d00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc006010 tx=0x7fa5cc009f60 comp rx=0 tx=0).stop 2022-09-10T04:29:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 shutdown_connections 2022-09-10T04:29:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e4b85e9c70 0x55e4b85ec050 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e4b85e2a10 0x55e4b85d8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e4b85dcf00 0x55e4b85db850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 --2- 172.21.15.81:0/3911851937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e4b85dd8b0 0x55e4b85d7d00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5b37fe700 0 client.15378 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.875+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 >> 172.21.15.81:0/3911851937 conn(0x55e4b84346b0 msgr2=0x55e4b85d7270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.876+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 shutdown_connections 2022-09-10T04:29:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.876+0000 7fa5f1b69b80 1 -- 172.21.15.81:0/3911851937 wait complete. 2022-09-10T04:29:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.876+0000 7fa5f1b69b80 20 client.15378 trim_cache size 0 max 16384 2022-09-10T04:29:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:29:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:29:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:29:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:29:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa5c0099360 msgr2=0x7fa5c009b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa5c0099360 0x7fa5c009b810 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80105a0 tx=0x7fa5c8010000 comp rx=0 tx=0).stop 2022-09-10T04:29:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 msgr2=0x7fa5d80691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 0x7fa5d80691b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800d700 tx=0x7fa5c800da10 comp rx=0 tx=0).stop 2022-09-10T04:29:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 shutdown_connections 2022-09-10T04:29:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d8104580 0x7fa5d81a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d8066850 0x7fa5d80691b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7fa5c0099360 0x7fa5c009b810 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 --2- 172.21.15.81:0/2909019993 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5d8103be0 0x7fa5d80696f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 >> 172.21.15.81:0/2909019993 conn(0x7fa5d8074870 msgr2=0x7fa5d80fb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 shutdown_connections 2022-09-10T04:29:12.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:12.877+0000 7fa5dd39d700 1 -- 172.21.15.81:0/2909019993 wait complete. 2022-09-10T04:29:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:12 smithi104 ceph-mon[97044]: pgmap v17: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2022-09-10T04:29:13.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:12 smithi081 ceph-mon[90133]: pgmap v17: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2022-09-10T04:29:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:12 smithi110 ceph-mon[97633]: pgmap v17: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 26 op/s 2022-09-10T04:29:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:14 smithi104 ceph-mon[97044]: pgmap v18: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:14 smithi081 ceph-mon[90133]: pgmap v18: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:14 smithi110 ceph-mon[97633]: pgmap v18: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 22 op/s 2022-09-10T04:29:17.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:16 smithi104 ceph-mon[97044]: pgmap v19: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2022-09-10T04:29:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:16 smithi081 ceph-mon[90133]: pgmap v19: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2022-09-10T04:29:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:16 smithi110 ceph-mon[97633]: pgmap v19: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 27 op/s 2022-09-10T04:29:19.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:18 smithi104 ceph-mon[97044]: pgmap v20: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 9 op/s 2022-09-10T04:29:19.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:18 smithi081 ceph-mon[90133]: pgmap v20: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 9 op/s 2022-09-10T04:29:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:18 smithi110 ceph-mon[97633]: pgmap v20: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 9 op/s 2022-09-10T04:29:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:20 smithi104 ceph-mon[97044]: pgmap v21: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 9 op/s 2022-09-10T04:29:21.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:20 smithi081 ceph-mon[90133]: pgmap v21: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 9 op/s 2022-09-10T04:29:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:20 smithi110 ceph-mon[97633]: pgmap v21: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 9 op/s 2022-09-10T04:29:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:22 smithi104 ceph-mon[97044]: pgmap v22: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 13 op/s 2022-09-10T04:29:23.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:22 smithi081 ceph-mon[90133]: pgmap v22: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 13 op/s 2022-09-10T04:29:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:22 smithi110 ceph-mon[97633]: pgmap v22: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 13 op/s 2022-09-10T04:29:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:24 smithi104 ceph-mon[97044]: pgmap v23: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2022-09-10T04:29:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:24 smithi081 ceph-mon[90133]: pgmap v23: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2022-09-10T04:29:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:24 smithi110 ceph-mon[97633]: pgmap v23: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2022-09-10T04:29:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 -- 172.21.15.81:0/3088508474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1051b0 msgr2=0x7f266c10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 --2- 172.21.15.81:0/3088508474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1051b0 0x7f266c10cd50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2664009be0 tx=0x7f2664009ef0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 -- 172.21.15.81:0/3088508474 shutdown_connections 2022-09-10T04:29:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 --2- 172.21.15.81:0/3088508474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1051b0 0x7f266c10cd50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 --2- 172.21.15.81:0/3088508474 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f266c1000b0 0x7f266c100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 --2- 172.21.15.81:0/3088508474 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f266c100a50 0x7f266c104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.058+0000 7f267299d700 1 -- 172.21.15.81:0/3088508474 >> 172.21.15.81:0/3088508474 conn(0x7f266c0fba30 msgr2=0x7f266c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.059+0000 7f267299d700 1 -- 172.21.15.81:0/3088508474 shutdown_connections 2022-09-10T04:29:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.059+0000 7f267299d700 1 -- 172.21.15.81:0/3088508474 wait complete. 2022-09-10T04:29:26.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 Processor -- start 2022-09-10T04:29:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 -- start start 2022-09-10T04:29:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 0x7f266c19b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f266c100a50 0x7f266c19bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f266c1051b0 0x7f266c195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c196130 con 0x7f266c1000b0 2022-09-10T04:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1962a0 con 0x7f266c100a50 2022-09-10T04:29:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.060+0000 7f267299d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c196460 con 0x7f266c1051b0 2022-09-10T04:29:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 0x7f266c19b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 0x7f266c19b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45046/0 (socket says 172.21.15.81:45046) 2022-09-10T04:29:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 -- 172.21.15.81:0/2191262020 learned_addr learned my addr 172.21.15.81:0/2191262020 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267219c700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f266c1051b0 0x7f266c195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267119a700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f266c100a50 0x7f266c19bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 -- 172.21.15.81:0/2191262020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f266c1051b0 msgr2=0x7f266c195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f266c1051b0 0x7f266c195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 -- 172.21.15.81:0/2191262020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f266c100a50 msgr2=0x7f266c19bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f266c100a50 0x7f266c19bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.061+0000 7f267199b700 1 -- 172.21.15.81:0/2191262020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26640097e0 con 0x7f266c1000b0 2022-09-10T04:29:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.062+0000 7f267199b700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 0x7f266c19b980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f266800e740 tx=0x7f266800ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.062+0000 7f2662ffd700 1 -- 172.21.15.81:0/2191262020 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668005220 con 0x7f266c1000b0 2022-09-10T04:29:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.062+0000 7f2662ffd700 1 -- 172.21.15.81:0/2191262020 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2668005380 con 0x7f266c1000b0 2022-09-10T04:29:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.062+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c196740 con 0x7f266c1000b0 2022-09-10T04:29:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.062+0000 7f2662ffd700 1 -- 172.21.15.81:0/2191262020 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668004b70 con 0x7f266c1000b0 2022-09-10T04:29:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.062+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1a29b0 con 0x7f266c1000b0 2022-09-10T04:29:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f2662ffd700 1 -- 172.21.15.81:0/2191262020 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 22) v1 ==== 127855+0+0 (secure 0 0 0) 0x7f266801e020 con 0x7f266c1000b0 2022-09-10T04:29:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 Processor -- start 2022-09-10T04:29:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 -- start start 2022-09-10T04:29:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb21499b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb21384070 con 0x55fb2149f200 2022-09-10T04:29:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.067+0000 7f268702bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb213ee9d0 con 0x55fb214996f0 2022-09-10T04:29:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f268702bb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb213e8170 con 0x55fb2149a0a0 2022-09-10T04:29:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267199b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb21499b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45058/0 (socket says 172.21.15.81:45058) 2022-09-10T04:29:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 -- 172.21.15.81:0/1057098899 learned_addr learned my addr 172.21.15.81:0/1057098899 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267119a700 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 -- 172.21.15.81:0/1057098899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 msgr2=0x55fb214a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 -- 172.21.15.81:0/1057098899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 msgr2=0x55fb21499b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb21499b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267219c700 1 -- 172.21.15.81:0/1057098899 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb212705d0 con 0x55fb2149f200 2022-09-10T04:29:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.068+0000 7f267119a700 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:29:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f267219c700 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149f5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2664016000 tx=0x7f26640092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4e01880d56603b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f2662ffd700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f26580992a0 0x7f265809b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f2662ffd700 1 -- 172.21.15.81:0/2191262020 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f2668014070 con 0x7f266c1000b0 2022-09-10T04:29:26.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f264ffff700 1 -- 172.21.15.81:0/1057098899 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266401d040 con 0x55fb2149f200 2022-09-10T04:29:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f267199b700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f26580992a0 0x7f265809b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f264ffff700 1 -- 172.21.15.81:0/1057098899 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2664003d20 con 0x55fb2149f200 2022-09-10T04:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.069+0000 7f264ffff700 1 -- 172.21.15.81:0/1057098899 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664005670 con 0x55fb2149f200 2022-09-10T04:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 -- 172.21.15.81:0/1057098899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 msgr2=0x55fb2149f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149f5d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2664016000 tx=0x7f26640092d0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f267199b700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f26580992a0 0x7f265809b750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26680928d0 tx=0x7f26680c8530 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 -- 172.21.15.81:0/1057098899 shutdown_connections 2022-09-10T04:29:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149f5d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 --2- 172.21.15.81:0/1057098899 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb21499b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 -- 172.21.15.81:0/1057098899 >> 172.21.15.81:0/1057098899 conn(0x55fb212f96b0 msgr2=0x55fb21494010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.070+0000 7f268702bb80 1 -- 172.21.15.81:0/1057098899 shutdown_connections 2022-09-10T04:29:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.071+0000 7f268702bb80 1 -- 172.21.15.81:0/1057098899 wait complete. 2022-09-10T04:29:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.071+0000 7f268702bb80 1 -- start start 2022-09-10T04:29:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.071+0000 7f268702bb80 1 Processor -- start 2022-09-10T04:29:26.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb214a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb214a3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb212d60b0 con 0x55fb2149f200 2022-09-10T04:29:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214a3770 con 0x55fb214996f0 2022-09-10T04:29:26.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214a3930 con 0x55fb2149a0a0 2022-09-10T04:29:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f268702bb80 10 client.0 _ll_register_callbacks cb 0x55fb21393360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f267219c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb214a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f267119a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f267219c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb214a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f267119a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:47476/0 (socket says 172.21.15.81:47476) 2022-09-10T04:29:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f267219c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb214a3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45070/0 (socket says 172.21.15.81:45070) 2022-09-10T04:29:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.072+0000 7f267119a700 1 -- 172.21.15.81:0/1941526922 learned_addr learned my addr 172.21.15.81:0/1941526922 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267119a700 1 -- 172.21.15.81:0/1941526922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 msgr2=0x55fb214a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267119a700 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb214a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267119a700 1 -- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 msgr2=0x55fb214a3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267119a700 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb214a3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267119a700 1 -- 172.21.15.81:0/1941526922 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c009790 con 0x55fb2149a0a0 2022-09-10T04:29:26.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267219c700 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb214a3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:29:26.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f267119a700 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a2cf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f265c0127d0 tx=0x7f265c012ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b73b3fb1c8bb4898 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:26.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.073+0000 7f264d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:29:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f264d7fa700 1 -- 172.21.15.81:0/1941526922 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c017070 con 0x55fb2149a0a0 2022-09-10T04:29:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb2148c930 con 0x55fb2149a0a0 2022-09-10T04:29:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f264d7fa700 1 -- 172.21.15.81:0/1941526922 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f265c004020 con 0x55fb2149a0a0 2022-09-10T04:29:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f264d7fa700 1 -- 172.21.15.81:0/1941526922 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f265c0041e0 con 0x55fb2149a0a0 2022-09-10T04:29:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f268702bb80 10 client.25516 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f266c10d510 con 0x55fb2149a0a0 2022-09-10T04:29:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f264d7fa700 1 -- 172.21.15.81:0/1941526922 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55fb2148c930 con 0x55fb2149a0a0 2022-09-10T04:29:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f264d7fa700 1 -- 172.21.15.81:0/1941526922 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f265c00f480 con 0x55fb2149a0a0 2022-09-10T04:29:26.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.074+0000 7f268702bb80 10 client.25516 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:26.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.075+0000 7f268702bb80 10 client.25516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:26.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.075+0000 7f268702bb80 20 client.25516 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:26.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.075+0000 7f268702bb80 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb214b3400 0x55fb214b5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.075+0000 7f268702bb80 4 client.25516 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:29:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.075+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb214694b0 con 0x55fb214b3400 2022-09-10T04:29:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.075+0000 7f267219c700 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb214b3400 0x55fb214b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.076+0000 7f267219c700 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb214b3400 0x55fb214b5810 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97e1eb2302a34874 server_cookie=e1abb9461b7619da in_seq=0 out_seq=0 2022-09-10T04:29:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.076+0000 7f264d7fa700 10 client.25516 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.076+0000 7f264d7fa700 1 -- 172.21.15.81:0/1941526922 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55fb214694b0 con 0x55fb214b3400 2022-09-10T04:29:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.076+0000 7f264d7fa700 10 client.25516 handle_command_reply: tid=0 2022-09-10T04:29:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.076+0000 7f268702bb80 1 client.25516 shutdown 2022-09-10T04:29:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 msgr2=0x55fb214a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a2cf0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f265c0127d0 tx=0x7f265c012ae0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 shutdown_connections 2022-09-10T04:29:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb214a3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb214b3400 0x55fb214b5810 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb214996f0 0x55fb214a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 --2- 172.21.15.81:0/1941526922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb2149a0a0 0x55fb214a2cf0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f264d7fa700 0 client.25516 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 >> 172.21.15.81:0/1941526922 conn(0x55fb212f96b0 msgr2=0x55fb214986e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 shutdown_connections 2022-09-10T04:29:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 1 -- 172.21.15.81:0/1941526922 wait complete. 2022-09-10T04:29:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.077+0000 7f268702bb80 20 client.25516 trim_cache size 0 max 16384 2022-09-10T04:29:26.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.092+0000 7f268702bb80 1 Processor -- start 2022-09-10T04:29:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.092+0000 7f268702bb80 1 -- start start 2022-09-10T04:29:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.092+0000 7f268702bb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb2149cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.092+0000 7f268702bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 0x55fb2149d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.092+0000 7f268702bb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f268702bb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb21496250 con 0x55fb2149f200 2022-09-10T04:29:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f268702bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214a0e20 con 0x55fb2149a0a0 2022-09-10T04:29:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f268702bb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214ba3c0 con 0x55fb214996f0 2022-09-10T04:29:26.104 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 0x55fb2149d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267219c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb2149cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 0x55fb2149d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:34800/0 (socket says 172.21.15.81:34800) 2022-09-10T04:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 -- 172.21.15.81:0/293094156 learned_addr learned my addr 172.21.15.81:0/293094156 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267219c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb2149cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:47478/0 (socket says 172.21.15.81:47478) 2022-09-10T04:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267219c700 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 -- 172.21.15.81:0/293094156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 msgr2=0x55fb2149cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb2149cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 -- 172.21.15.81:0/293094156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 msgr2=0x55fb2149de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267119a700 1 -- 172.21.15.81:0/293094156 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb2121cd80 con 0x55fb2149a0a0 2022-09-10T04:29:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.093+0000 7f267219c700 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb2149cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:29:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.094+0000 7f267119a700 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 0x55fb2149d0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f265c012400 tx=0x7f265c005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68ed8812e14cc25e server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.094+0000 7f2660bff700 1 -- 172.21.15.81:0/293094156 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c017070 con 0x55fb2149a0a0 2022-09-10T04:29:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.094+0000 7f2660bff700 1 -- 172.21.15.81:0/293094156 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f265c018040 con 0x55fb2149a0a0 2022-09-10T04:29:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.094+0000 7f2660bff700 1 -- 172.21.15.81:0/293094156 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0037f0 con 0x55fb2149a0a0 2022-09-10T04:29:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 -- 172.21.15.81:0/293094156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 msgr2=0x55fb2149d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 0x55fb2149d0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f265c012400 tx=0x7f265c005f10 comp rx=0 tx=0).stop 2022-09-10T04:29:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 -- 172.21.15.81:0/293094156 shutdown_connections 2022-09-10T04:29:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149f200 0x55fb2149de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb2149cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 --2- 172.21.15.81:0/293094156 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149a0a0 0x55fb2149d0c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 -- 172.21.15.81:0/293094156 >> 172.21.15.81:0/293094156 conn(0x55fb212f96b0 msgr2=0x55fb21498f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 -- 172.21.15.81:0/293094156 shutdown_connections 2022-09-10T04:29:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 -- 172.21.15.81:0/293094156 wait complete. 2022-09-10T04:29:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.095+0000 7f268702bb80 1 -- start start 2022-09-10T04:29:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 Processor -- start 2022-09-10T04:29:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb214fe460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 0x55fb214fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149f200 0x55fb214f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214f9240 con 0x55fb2149a0a0 2022-09-10T04:29:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214f93b0 con 0x55fb2149f200 2022-09-10T04:29:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb214f9570 con 0x55fb214996f0 2022-09-10T04:29:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f268702bb80 10 client.0 _ll_register_callbacks cb 0x55fb21393360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f267219c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb214fe460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f267119a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 0x55fb214fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f267119a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 0x55fb214fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45098/0 (socket says 172.21.15.81:45098) 2022-09-10T04:29:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 -- 172.21.15.81:0/1431621462 learned_addr learned my addr 172.21.15.81:0/1431621462 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:26.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.096+0000 7f267219c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb214fe460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:47482/0 (socket says 172.21.15.81:47482) 2022-09-10T04:29:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267219c700 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149f200 0x55fb214f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 -- 172.21.15.81:0/1431621462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 msgr2=0x55fb214fe460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb214fe460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 -- 172.21.15.81:0/1431621462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149f200 msgr2=0x55fb214f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:26.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149f200 0x55fb214f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 -- 172.21.15.81:0/1431621462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2664026040 con 0x55fb2149a0a0 2022-09-10T04:29:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f267119a700 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 0x55fb214fe9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f265c00f930 tx=0x7f265c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ef3408ecc70fa1 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f264d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2022-09-10T04:29:26.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.097+0000 7f264d7fa700 1 -- 172.21.15.81:0/1431621462 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c017040 con 0x55fb2149a0a0 2022-09-10T04:29:26.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2664003d20 con 0x55fb2149a0a0 2022-09-10T04:29:26.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f264d7fa700 1 -- 172.21.15.81:0/1431621462 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f265c00fa00 con 0x55fb2149a0a0 2022-09-10T04:29:26.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f264d7fa700 1 -- 172.21.15.81:0/1431621462 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f265c003700 con 0x55fb2149a0a0 2022-09-10T04:29:26.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 10 client.15396 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:26.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f266c10d510 con 0x55fb2149a0a0 2022-09-10T04:29:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f264d7fa700 1 -- 172.21.15.81:0/1431621462 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f2664003d20 con 0x55fb2149a0a0 2022-09-10T04:29:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f264d7fa700 1 -- 172.21.15.81:0/1431621462 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f265c010430 con 0x55fb2149a0a0 2022-09-10T04:29:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 10 client.15396 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 10 client.15396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:26.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 20 client.15396 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb21505eb0 0x55fb215082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.098+0000 7f268702bb80 4 client.15396 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:29:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.099+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fb213e16e0 con 0x55fb21505eb0 2022-09-10T04:29:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.099+0000 7f267219c700 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb21505eb0 0x55fb215082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:26.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.099+0000 7f267219c700 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb21505eb0 0x55fb215082c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a356ef76b3ad907d server_cookie=5a8bea36c54d3c60 in_seq=0 out_seq=0 2022-09-10T04:29:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.099+0000 7f264d7fa700 10 client.15396 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f264d7fa700 1 -- 172.21.15.81:0/1431621462 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fb213e16e0 con 0x55fb21505eb0 2022-09-10T04:29:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f264d7fa700 10 client.15396 handle_command_reply: tid=0 2022-09-10T04:29:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 client.15396 shutdown 2022-09-10T04:29:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 msgr2=0x55fb214fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 0x55fb214fe9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f265c00f930 tx=0x7f265c005710 comp rx=0 tx=0).stop 2022-09-10T04:29:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 shutdown_connections 2022-09-10T04:29:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55fb21505eb0 0x55fb215082c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55fb2149f200 0x55fb214f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55fb214996f0 0x55fb214fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 --2- 172.21.15.81:0/1431621462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55fb2149a0a0 0x55fb214fe9a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 >> 172.21.15.81:0/1431621462 conn(0x55fb212f96b0 msgr2=0x55fb2149d280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f264d7fa700 0 client.15396 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 shutdown_connections 2022-09-10T04:29:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.106+0000 7f268702bb80 1 -- 172.21.15.81:0/1431621462 wait complete. 2022-09-10T04:29:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f268702bb80 20 client.15396 trim_cache size 0 max 16384 2022-09-10T04:29:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:29:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:29:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:29:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:29:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f26580992a0 msgr2=0x7f265809b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f26580992a0 0x7f265809b750 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f26680928d0 tx=0x7f26680c8530 comp rx=0 tx=0).stop 2022-09-10T04:29:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 msgr2=0x7f266c19b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 0x7f266c19b980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f266800e740 tx=0x7f266800ea50 comp rx=0 tx=0).stop 2022-09-10T04:29:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 shutdown_connections 2022-09-10T04:29:26.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f266c1051b0 0x7f266c195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f266c1000b0 0x7f266c19b980 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f26580992a0 0x7f265809b750 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 --2- 172.21.15.81:0/2191262020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f266c100a50 0x7f266c19bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.107+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 >> 172.21.15.81:0/2191262020 conn(0x7f266c0fba30 msgr2=0x7f266c0fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:26.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.108+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 shutdown_connections 2022-09-10T04:29:26.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:26.108+0000 7f267299d700 1 -- 172.21.15.81:0/2191262020 wait complete. 2022-09-10T04:29:26.498 INFO:tasks.workunit.client.0.smithi081.stderr:Note: switching to '244418605927bb7842e73fc5d2a53889b760937f'. 2022-09-10T04:29:26.499 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.499 INFO:tasks.workunit.client.0.smithi081.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2022-09-10T04:29:26.499 INFO:tasks.workunit.client.0.smithi081.stderr:changes and commit them, and you can discard any commits you make in this 2022-09-10T04:29:26.499 INFO:tasks.workunit.client.0.smithi081.stderr:state without impacting any branches by switching back to a branch. 2022-09-10T04:29:26.500 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.500 INFO:tasks.workunit.client.0.smithi081.stderr:If you want to create a new branch to retain commits you create, you may 2022-09-10T04:29:26.500 INFO:tasks.workunit.client.0.smithi081.stderr:do so (now or later) by using -c with the switch command. Example: 2022-09-10T04:29:26.500 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.501 INFO:tasks.workunit.client.0.smithi081.stderr: git switch -c 2022-09-10T04:29:26.501 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.501 INFO:tasks.workunit.client.0.smithi081.stderr:Or undo this operation with: 2022-09-10T04:29:26.501 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.501 INFO:tasks.workunit.client.0.smithi081.stderr: git switch - 2022-09-10T04:29:26.502 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.502 INFO:tasks.workunit.client.0.smithi081.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2022-09-10T04:29:26.502 INFO:tasks.workunit.client.0.smithi081.stderr: 2022-09-10T04:29:26.503 INFO:tasks.workunit.client.0.smithi081.stderr:HEAD is now at 24441860592 Merge PR #46821 into wip-pdonnell-testing-20220909.143836 2022-09-10T04:29:26.542 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2022-09-10T04:29:26.613 INFO:tasks.workunit.client.0.smithi081.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2022-09-10T04:29:26.619 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-09-10T04:29:26.619 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2022-09-10T04:29:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:26 smithi104 ceph-mon[97044]: pgmap v24: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2022-09-10T04:29:27.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:26 smithi081 ceph-mon[90133]: pgmap v24: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2022-09-10T04:29:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:26 smithi110 ceph-mon[97633]: pgmap v24: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2022-09-10T04:29:27.204 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2022-09-10T04:29:27.326 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2022-09-10T04:29:27.407 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2022-09-10T04:29:27.412 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-09-10T04:29:27.413 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2022-09-10T04:29:27.493 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2022-09-10T04:29:27.504 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2022-09-10T04:29:27.505 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2022-09-10T04:29:27.565 INFO:tasks.workunit:Running workunits matching direct_io on client.0... 2022-09-10T04:29:27.568 INFO:tasks.workunit:Running workunit direct_io/big.sh... 2022-09-10T04:29:27.569 DEBUG:teuthology.orchestra.run.smithi081:workunit test direct_io/big.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=244418605927bb7842e73fc5d2a53889b760937f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/big.sh 2022-09-10T04:29:27.650 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 'test large (16MB) dio write' 2022-09-10T04:29:27.651 INFO:tasks.workunit.client.0.smithi081.stdout:test large (16MB) dio write 2022-09-10T04:29:27.651 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=/dev/zero of=foo.big bs=16M count=1 oflag=direct 2022-09-10T04:29:27.849 INFO:tasks.workunit.client.0.smithi081.stderr:1+0 records in 2022-09-10T04:29:27.849 INFO:tasks.workunit.client.0.smithi081.stderr:1+0 records out 2022-09-10T04:29:27.850 INFO:tasks.workunit.client.0.smithi081.stderr:16777216 bytes (17 MB, 16 MiB) copied, 0.193985 s, 86.5 MB/s 2022-09-10T04:29:27.856 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2022-09-10T04:29:27.857 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2022-09-10T04:29:27.857 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-09-10T04:29:27.858 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-09-10T04:29:27.960 INFO:tasks.workunit:Running workunit direct_io/direct_io_test... 2022-09-10T04:29:27.961 DEBUG:teuthology.orchestra.run.smithi081:workunit test direct_io/direct_io_test> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=244418605927bb7842e73fc5d2a53889b760937f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/direct_io_test 2022-09-10T04:29:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:28 smithi104 ceph-mon[97044]: pgmap v25: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2022-09-10T04:29:29.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:28 smithi081 ceph-mon[90133]: pgmap v25: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2022-09-10T04:29:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:28 smithi110 ceph-mon[97633]: pgmap v25: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2022-09-10T04:29:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:30 smithi104 ceph-mon[97044]: pgmap v26: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2022-09-10T04:29:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:30 smithi081 ceph-mon[90133]: pgmap v26: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2022-09-10T04:29:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:30 smithi110 ceph-mon[97633]: pgmap v26: 129 pgs: 129 active+clean; 1.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 8 op/s 2022-09-10T04:29:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:32 smithi104 ceph-mon[97044]: pgmap v27: 129 pgs: 129 active+clean; 17 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 MiB/s wr, 14 op/s 2022-09-10T04:29:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:32 smithi081 ceph-mon[90133]: pgmap v27: 129 pgs: 129 active+clean; 17 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 MiB/s wr, 14 op/s 2022-09-10T04:29:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:32 smithi110 ceph-mon[97633]: pgmap v27: 129 pgs: 129 active+clean; 17 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.3 MiB/s wr, 14 op/s 2022-09-10T04:29:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:34 smithi104 ceph-mon[97044]: pgmap v28: 129 pgs: 129 active+clean; 17 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2022-09-10T04:29:35.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:34 smithi081 ceph-mon[90133]: pgmap v28: 129 pgs: 129 active+clean; 17 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2022-09-10T04:29:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:34 smithi110 ceph-mon[97633]: pgmap v28: 129 pgs: 129 active+clean; 17 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.3 MiB/s wr, 10 op/s 2022-09-10T04:29:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:36 smithi104 ceph-mon[97044]: pgmap v29: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.4 MiB/s wr, 241 op/s 2022-09-10T04:29:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:36 smithi081 ceph-mon[90133]: pgmap v29: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.4 MiB/s wr, 241 op/s 2022-09-10T04:29:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:36 smithi110 ceph-mon[97633]: pgmap v29: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.4 MiB/s wr, 241 op/s 2022-09-10T04:29:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:38 smithi104 ceph-mon[97044]: pgmap v30: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 237 op/s 2022-09-10T04:29:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:38 smithi081 ceph-mon[90133]: pgmap v30: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 237 op/s 2022-09-10T04:29:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:38 smithi110 ceph-mon[97633]: pgmap v30: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 237 op/s 2022-09-10T04:29:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 -- 172.21.15.81:0/847683685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960104580 msgr2=0x7f9960066170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 --2- 172.21.15.81:0/847683685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960104580 0x7f9960066170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9954009810 tx=0x7f9954009b20 comp rx=0 tx=0).stop 2022-09-10T04:29:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 -- 172.21.15.81:0/847683685 shutdown_connections 2022-09-10T04:29:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 --2- 172.21.15.81:0/847683685 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9960066850 0x7f996010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 --2- 172.21.15.81:0/847683685 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960103be0 0x7f9960103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 --2- 172.21.15.81:0/847683685 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960104580 0x7f9960066170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 -- 172.21.15.81:0/847683685 >> 172.21.15.81:0/847683685 conn(0x7f9960074870 msgr2=0x7f9960074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 -- 172.21.15.81:0/847683685 shutdown_connections 2022-09-10T04:29:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.282+0000 7f9964d9d700 1 -- 172.21.15.81:0/847683685 wait complete. 2022-09-10T04:29:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.283+0000 7f9964d9d700 1 Processor -- start 2022-09-10T04:29:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.283+0000 7f9964d9d700 1 -- start start 2022-09-10T04:29:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f9964d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9960066850 0x7f996019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.294 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f9964d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 0x7f996019bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f9964d9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960104580 0x7f99601959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f9964d9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601961a0 con 0x7f9960104580 2022-09-10T04:29:39.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f9964d9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9960196310 con 0x7f9960066850 2022-09-10T04:29:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f9964d9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99601964d0 con 0x7f9960103be0 2022-09-10T04:29:39.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f995effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 0x7f996019bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f995effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 0x7f996019bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:48284/0 (socket says 172.21.15.81:48284) 2022-09-10T04:29:39.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f995effd700 1 -- 172.21.15.81:0/2508889549 learned_addr learned my addr 172.21.15.81:0/2508889549 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:39.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9960066850 0x7f996019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.284+0000 7f995ffff700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960104580 0x7f99601959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995effd700 1 -- 172.21.15.81:0/2508889549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9960066850 msgr2=0x7f996019b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995effd700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9960066850 0x7f996019b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995effd700 1 -- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960104580 msgr2=0x7f99601959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995effd700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960104580 0x7f99601959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995effd700 1 -- 172.21.15.81:0/2508889549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954009470 con 0x7f9960103be0 2022-09-10T04:29:39.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995ffff700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960104580 0x7f99601959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:29:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995effd700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 0x7f996019bf30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99540059c0 tx=0x7f995400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:39.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995cff9700 1 -- 172.21.15.81:0/2508889549 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995401d070 con 0x7f9960103be0 2022-09-10T04:29:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995cff9700 1 -- 172.21.15.81:0/2508889549 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f995400f930 con 0x7f9960103be0 2022-09-10T04:29:39.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9960196750 con 0x7f9960103be0 2022-09-10T04:29:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f995cff9700 1 -- 172.21.15.81:0/2508889549 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954004760 con 0x7f9960103be0 2022-09-10T04:29:39.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.285+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601a2970 con 0x7f9960103be0 2022-09-10T04:29:39.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.289+0000 7f995cff9700 1 -- 172.21.15.81:0/2508889549 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) v1 ==== 127855+0+0 (secure 0 0 0) 0x7f9954028030 con 0x7f9960103be0 2022-09-10T04:29:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.290+0000 7f99794e9b80 1 Processor -- start 2022-09-10T04:29:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.290+0000 7f99794e9b80 1 -- start start 2022-09-10T04:29:39.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70df5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70d9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70e5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f99794e9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f6fc4070 con 0x5614f70d96f0 2022-09-10T04:29:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f99794e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f702e9d0 con 0x5614f70da0a0 2022-09-10T04:29:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f99794e9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f7028170 con 0x5614f70df200 2022-09-10T04:29:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f995cff9700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9948099220 0x7f994809b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f995ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70df5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f995ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70df5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:48288/0 (socket says 172.21.15.81:48288) 2022-09-10T04:29:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f995ffff700 1 -- 172.21.15.81:0/2349049843 learned_addr learned my addr 172.21.15.81:0/2349049843 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.291+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70d9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995cff9700 1 -- 172.21.15.81:0/2508889549 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f99540c36a0 con 0x7f9960103be0 2022-09-10T04:29:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995effd700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70e5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995ffff700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9948099220 0x7f994809b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995f7fe700 1 -- 172.21.15.81:0/2349049843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 msgr2=0x5614f70df5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70df5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995f7fe700 1 -- 172.21.15.81:0/2349049843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 msgr2=0x5614f70e5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70e5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995f7fe700 1 -- 172.21.15.81:0/2349049843 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614f6eb05d0 con 0x5614f70d96f0 2022-09-10T04:29:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995effd700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70e5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:29:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995ffff700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70df5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:29:39.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.292+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70d9b60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9950009b30 tx=0x7f9950009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0420d98c017acb server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.293+0000 7f995ffff700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9948099220 0x7f994809b6d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9958013ab0 tx=0x7f99580049c0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.293+0000 7f9945ffb700 1 -- 172.21.15.81:0/2349049843 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950004310 con 0x5614f70d96f0 2022-09-10T04:29:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.293+0000 7f9945ffb700 1 -- 172.21.15.81:0/2349049843 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9950011dd0 con 0x5614f70d96f0 2022-09-10T04:29:39.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.293+0000 7f9945ffb700 1 -- 172.21.15.81:0/2349049843 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950003ea0 con 0x5614f70d96f0 2022-09-10T04:29:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.293+0000 7f99794e9b80 1 -- 172.21.15.81:0/2349049843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 msgr2=0x5614f70d9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.293+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70d9b60 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9950009b30 tx=0x7f9950009e40 comp rx=0 tx=0).stop 2022-09-10T04:29:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 -- 172.21.15.81:0/2349049843 shutdown_connections 2022-09-10T04:29:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70e5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70df5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2349049843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70d9b60 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 -- 172.21.15.81:0/2349049843 >> 172.21.15.81:0/2349049843 conn(0x5614f6f396b0 msgr2=0x5614f70d4010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 -- 172.21.15.81:0/2349049843 shutdown_connections 2022-09-10T04:29:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 -- 172.21.15.81:0/2349049843 wait complete. 2022-09-10T04:29:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.294+0000 7f99794e9b80 1 -- start start 2022-09-10T04:29:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 Processor -- start 2022-09-10T04:29:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70d96f0 0x5614f70e5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 0x5614f70e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f6f160b0 con 0x5614f70df200 2022-09-10T04:29:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70e3720 con 0x5614f70d96f0 2022-09-10T04:29:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70e38e0 con 0x5614f70da0a0 2022-09-10T04:29:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.295+0000 7f99794e9b80 10 client.0 _ll_register_callbacks cb 0x5614f6fd3360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 0x5614f70e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 0x5614f70e2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:48298/0 (socket says 172.21.15.81:48298) 2022-09-10T04:29:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 -- 172.21.15.81:0/2063724436 learned_addr learned my addr 172.21.15.81:0/2063724436 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70d96f0 0x5614f70e5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995ffff700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 -- 172.21.15.81:0/2063724436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70d96f0 msgr2=0x5614f70e5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70d96f0 0x5614f70e5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 -- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 msgr2=0x5614f70e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 -- 172.21.15.81:0/2063724436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9958013560 con 0x5614f70da0a0 2022-09-10T04:29:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f995effd700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 0x5614f70e2ca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f995408fef0 tx=0x7f9954092e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d78c47b217403d0a server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.296+0000 7f993affd700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:29:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f993affd700 1 -- 172.21.15.81:0/2063724436 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540c34e0 con 0x5614f70da0a0 2022-09-10T04:29:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f993affd700 1 -- 172.21.15.81:0/2063724436 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f9954093480 con 0x5614f70da0a0 2022-09-10T04:29:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614f70cc930 con 0x5614f70da0a0 2022-09-10T04:29:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f993affd700 1 -- 172.21.15.81:0/2063724436 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f995409a720 con 0x5614f70da0a0 2022-09-10T04:29:39.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f99794e9b80 10 client.25534 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f9950009790 con 0x5614f70da0a0 2022-09-10T04:29:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f993affd700 1 -- 172.21.15.81:0/2063724436 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5614f70cc930 con 0x5614f70da0a0 2022-09-10T04:29:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f993affd700 1 -- 172.21.15.81:0/2063724436 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f9954092a70 con 0x5614f70da0a0 2022-09-10T04:29:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f99794e9b80 10 client.25534 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:39.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f99794e9b80 10 client.25534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.297+0000 7f99794e9b80 20 client.25534 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.298+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f70f3000 0x5614f70f5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.298+0000 7f99794e9b80 4 client.25534 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:29:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.298+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5614f70a94b0 con 0x5614f70f3000 2022-09-10T04:29:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.298+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f70f3000 0x5614f70f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.298+0000 7f995f7fe700 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f70f3000 0x5614f70f5410 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae8861703d7548f server_cookie=5521b447db0052bc in_seq=0 out_seq=0 2022-09-10T04:29:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.298+0000 7f993affd700 10 client.25534 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.299+0000 7f993affd700 1 -- 172.21.15.81:0/2063724436 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5614f70a94b0 con 0x5614f70f3000 2022-09-10T04:29:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.299+0000 7f993affd700 10 client.25534 handle_command_reply: tid=0 2022-09-10T04:29:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.299+0000 7f99794e9b80 1 client.25534 shutdown 2022-09-10T04:29:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 msgr2=0x5614f70e2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 0x5614f70e2ca0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f995408fef0 tx=0x7f9954092e60 comp rx=0 tx=0).stop 2022-09-10T04:29:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 shutdown_connections 2022-09-10T04:29:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f70f3000 0x5614f70f5410 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70d96f0 0x5614f70e5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 --2- 172.21.15.81:0/2063724436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70da0a0 0x5614f70e2ca0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 >> 172.21.15.81:0/2063724436 conn(0x5614f6f396b0 msgr2=0x5614f70d86e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f993affd700 0 client.25534 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 shutdown_connections 2022-09-10T04:29:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 1 -- 172.21.15.81:0/2063724436 wait complete. 2022-09-10T04:29:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.300+0000 7f99794e9b80 20 client.25534 trim_cache size 0 max 16384 2022-09-10T04:29:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 Processor -- start 2022-09-10T04:29:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 -- start start 2022-09-10T04:29:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70dcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70dd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70cc910 con 0x5614f70d96f0 2022-09-10T04:29:39.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70f9fc0 con 0x5614f70da0a0 2022-09-10T04:29:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f99794e9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70da580 con 0x5614f70df200 2022-09-10T04:29:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f995effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70dcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f995ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70dd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f995effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70dcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38852/0 (socket says 172.21.15.81:38852) 2022-09-10T04:29:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.321+0000 7f995effd700 1 -- 172.21.15.81:0/3928799876 learned_addr learned my addr 172.21.15.81:0/3928799876 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995f7fe700 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995effd700 1 -- 172.21.15.81:0/3928799876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 msgr2=0x5614f70dd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995effd700 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70dd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995effd700 1 -- 172.21.15.81:0/3928799876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 msgr2=0x5614f70e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995effd700 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70e55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995effd700 1 -- 172.21.15.81:0/3928799876 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614f6e5cd80 con 0x5614f70da0a0 2022-09-10T04:29:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f995effd700 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70dcb80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99540917d0 tx=0x7f995409a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c2617a5e11d88fb server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f99467fc700 1 -- 172.21.15.81:0/3928799876 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540c34e0 con 0x5614f70da0a0 2022-09-10T04:29:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f99467fc700 1 -- 172.21.15.81:0/3928799876 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9954093250 con 0x5614f70da0a0 2022-09-10T04:29:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.322+0000 7f99467fc700 1 -- 172.21.15.81:0/3928799876 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99540936e0 con 0x5614f70da0a0 2022-09-10T04:29:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 -- 172.21.15.81:0/3928799876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 msgr2=0x5614f70dcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70dcb80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99540917d0 tx=0x7f995409a330 comp rx=0 tx=0).stop 2022-09-10T04:29:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 -- 172.21.15.81:0/3928799876 shutdown_connections 2022-09-10T04:29:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70df200 0x5614f70dd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70d96f0 0x5614f70e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 --2- 172.21.15.81:0/3928799876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70dcb80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 -- 172.21.15.81:0/3928799876 >> 172.21.15.81:0/3928799876 conn(0x5614f6f396b0 msgr2=0x5614f70d3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 -- 172.21.15.81:0/3928799876 shutdown_connections 2022-09-10T04:29:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.323+0000 7f99794e9b80 1 -- 172.21.15.81:0/3928799876 wait complete. 2022-09-10T04:29:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.324+0000 7f99794e9b80 1 -- start start 2022-09-10T04:29:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.324+0000 7f99794e9b80 1 Processor -- start 2022-09-10T04:29:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.324+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70d96f0 0x5614f70e9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.324+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70ea2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f99794e9b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f99794e9b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70e8d30 con 0x5614f70df200 2022-09-10T04:29:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f99794e9b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70e8ea0 con 0x5614f70da0a0 2022-09-10T04:29:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f99794e9b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614f70e9060 con 0x5614f70d96f0 2022-09-10T04:29:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f99794e9b80 10 client.0 _ll_register_callbacks cb 0x5614f6fd3360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70ea2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70ea2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38866/0 (socket says 172.21.15.81:38866) 2022-09-10T04:29:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 -- 172.21.15.81:0/262777499 learned_addr learned my addr 172.21.15.81:0/262777499 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995f7fe700 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70d96f0 0x5614f70e9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 -- 172.21.15.81:0/262777499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70d96f0 msgr2=0x5614f70e9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70d96f0 0x5614f70e9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 -- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 msgr2=0x5614f70e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.325+0000 7f995effd700 1 -- 172.21.15.81:0/262777499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9950009790 con 0x5614f70da0a0 2022-09-10T04:29:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f995effd700 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70ea2a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f995408f980 tx=0x7f9954093990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fd88f6da5ff3287 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f993affd700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:29:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f993affd700 1 -- 172.21.15.81:0/262777499 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995400fdf0 con 0x5614f70da0a0 2022-09-10T04:29:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f993affd700 1 -- 172.21.15.81:0/262777499 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f9954093480 con 0x5614f70da0a0 2022-09-10T04:29:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9950011dd0 con 0x5614f70da0a0 2022-09-10T04:29:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f993affd700 1 -- 172.21.15.81:0/262777499 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f995409a720 con 0x5614f70da0a0 2022-09-10T04:29:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f99794e9b80 10 client.25532 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.326+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5614f6e5b250 con 0x5614f70da0a0 2022-09-10T04:29:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f993affd700 1 -- 172.21.15.81:0/262777499 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f9950011dd0 con 0x5614f70da0a0 2022-09-10T04:29:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f993affd700 1 -- 172.21.15.81:0/262777499 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f99540924e0 con 0x5614f70da0a0 2022-09-10T04:29:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f99794e9b80 10 client.25532 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f99794e9b80 10 client.25532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f99794e9b80 20 client.25532 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f99794e9b80 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f7145500 0x5614f71478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f99794e9b80 4 client.25532 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:29:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5614f70216e0 con 0x5614f7145500 2022-09-10T04:29:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.327+0000 7f995f7fe700 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f7145500 0x5614f71478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.328+0000 7f993affd700 10 client.25532 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.328+0000 7f995f7fe700 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f7145500 0x5614f71478e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f81262969dc75399 server_cookie=efecfb101efd8add in_seq=0 out_seq=0 2022-09-10T04:29:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.338+0000 7f993affd700 1 -- 172.21.15.81:0/262777499 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5614f70216e0 con 0x5614f7145500 2022-09-10T04:29:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.338+0000 7f993affd700 10 client.25532 handle_command_reply: tid=0 2022-09-10T04:29:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.338+0000 7f99794e9b80 1 client.25532 shutdown 2022-09-10T04:29:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 msgr2=0x5614f70ea2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70ea2a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f995408f980 tx=0x7f9954093990 comp rx=0 tx=0).stop 2022-09-10T04:29:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 shutdown_connections 2022-09-10T04:29:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5614f70df200 0x5614f70e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5614f70d96f0 0x5614f70e9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x5614f7145500 0x5614f71478e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 --2- 172.21.15.81:0/262777499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5614f70da0a0 0x5614f70ea2a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 >> 172.21.15.81:0/262777499 conn(0x5614f6f396b0 msgr2=0x5614f70dcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f993affd700 0 client.25532 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 shutdown_connections 2022-09-10T04:29:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 1 -- 172.21.15.81:0/262777499 wait complete. 2022-09-10T04:29:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.339+0000 7f99794e9b80 20 client.25532 trim_cache size 0 max 16384 2022-09-10T04:29:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:29:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:29:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:29:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:29:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9948099220 msgr2=0x7f994809b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9948099220 0x7f994809b6d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9958013ab0 tx=0x7f99580049c0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 msgr2=0x7f996019bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 0x7f996019bf30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99540059c0 tx=0x7f995400f690 comp rx=0 tx=0).stop 2022-09-10T04:29:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 shutdown_connections 2022-09-10T04:29:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9960104580 0x7f99601959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f9948099220 0x7f994809b6d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9960066850 0x7f996019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 --2- 172.21.15.81:0/2508889549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9960103be0 0x7f996019bf30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 >> 172.21.15.81:0/2508889549 conn(0x7f9960074870 msgr2=0x7f9960108df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 shutdown_connections 2022-09-10T04:29:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:39.340+0000 7f9964d9d700 1 -- 172.21.15.81:0/2508889549 wait complete. 2022-09-10T04:29:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:40 smithi104 ceph-mon[97044]: pgmap v31: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 237 op/s 2022-09-10T04:29:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:40 smithi081 ceph-mon[90133]: pgmap v31: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 237 op/s 2022-09-10T04:29:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:40 smithi110 ceph-mon[97633]: pgmap v31: 129 pgs: 129 active+clean; 18 MiB data, 118 MiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 1.4 MiB/s wr, 237 op/s 2022-09-10T04:29:42.001 INFO:tasks.workunit.client.0.smithi081.stdout:.......... 2022-09-10T04:29:42.001 INFO:tasks.workunit.client.0.smithi081.stdout:do_o_direct_reads: SUCCESS 2022-09-10T04:29:42.007 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-09-10T04:29:42.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-09-10T04:29:42.106 INFO:tasks.workunit:Running workunit direct_io/misc.sh... 2022-09-10T04:29:42.107 DEBUG:teuthology.orchestra.run.smithi081:workunit test direct_io/misc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=244418605927bb7842e73fc5d2a53889b760937f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/misc.sh 2022-09-10T04:29:43.046 INFO:tasks.workunit.client.0.smithi081.stdout:test read from hole 2022-09-10T04:29:43.046 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 'test read from hole' 2022-09-10T04:29:43.047 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=/dev/zero of=dd3 bs=1 seek=1048576 count=0 2022-09-10T04:29:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:42 smithi104 ceph-mon[97044]: pgmap v32: 129 pgs: 129 active+clean; 1.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 771 op/s 2022-09-10T04:29:43.049 INFO:tasks.workunit.client.0.smithi081.stderr:0+0 records in 2022-09-10T04:29:43.049 INFO:tasks.workunit.client.0.smithi081.stderr:0+0 records out 2022-09-10T04:29:43.049 INFO:tasks.workunit.client.0.smithi081.stderr:0 bytes copied, 0.000106281 s, 0.0 kB/s 2022-09-10T04:29:43.050 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=dd3 of=/tmp/ddout1 skip=8 bs=512 count=2 iflag=direct 2022-09-10T04:29:43.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:42 smithi081 ceph-mon[90133]: pgmap v32: 129 pgs: 129 active+clean; 1.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 771 op/s 2022-09-10T04:29:43.054 INFO:tasks.workunit.client.0.smithi081.stderr:2+0 records in 2022-09-10T04:29:43.054 INFO:tasks.workunit.client.0.smithi081.stderr:2+0 records out 2022-09-10T04:29:43.055 INFO:tasks.workunit.client.0.smithi081.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00248263 s, 412 kB/s 2022-09-10T04:29:43.056 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=/dev/zero of=/tmp/dd3 bs=512 count=2 2022-09-10T04:29:43.057 INFO:tasks.workunit.client.0.smithi081.stderr:2+0 records in 2022-09-10T04:29:43.057 INFO:tasks.workunit.client.0.smithi081.stderr:2+0 records out 2022-09-10T04:29:43.057 INFO:tasks.workunit.client.0.smithi081.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000162695 s, 6.3 MB/s 2022-09-10T04:29:43.058 INFO:tasks.workunit.client.0.smithi081.stderr:+ cmp /tmp/dd3 /tmp/ddout1 2022-09-10T04:29:43.060 INFO:tasks.workunit.client.0.smithi081.stdout:other thing 2022-09-10T04:29:43.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 'other thing' 2022-09-10T04:29:43.061 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=/dev/urandom of=/tmp/dd10 bs=500 count=1 2022-09-10T04:29:43.063 INFO:tasks.workunit.client.0.smithi081.stderr:1+0 records in 2022-09-10T04:29:43.063 INFO:tasks.workunit.client.0.smithi081.stderr:1+0 records out 2022-09-10T04:29:43.064 INFO:tasks.workunit.client.0.smithi081.stderr:500 bytes copied, 0.000120441 s, 4.2 MB/s 2022-09-10T04:29:43.064 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=/tmp/dd10 of=dd10 bs=512 seek=8388 count=1 2022-09-10T04:29:43.077 INFO:tasks.workunit.client.0.smithi081.stderr:0+1 records in 2022-09-10T04:29:43.077 INFO:tasks.workunit.client.0.smithi081.stderr:0+1 records out 2022-09-10T04:29:43.077 INFO:tasks.workunit.client.0.smithi081.stderr:500 bytes copied, 0.010342 s, 48.3 kB/s 2022-09-10T04:29:43.078 INFO:tasks.workunit.client.0.smithi081.stderr:+ dd if=dd10 of=/tmp/dd10out bs=512 skip=8388 count=1 iflag=direct 2022-09-10T04:29:43.085 INFO:tasks.workunit.client.0.smithi081.stderr:0+1 records in 2022-09-10T04:29:43.085 INFO:tasks.workunit.client.0.smithi081.stderr:0+1 records out 2022-09-10T04:29:43.085 INFO:tasks.workunit.client.0.smithi081.stderr:500 bytes copied, 0.00484031 s, 103 kB/s 2022-09-10T04:29:43.086 INFO:tasks.workunit.client.0.smithi081.stderr:+ cmp /tmp/dd10 /tmp/dd10out 2022-09-10T04:29:43.088 INFO:tasks.workunit.client.0.smithi081.stdout:OK 2022-09-10T04:29:43.090 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo OK 2022-09-10T04:29:43.091 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-09-10T04:29:43.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-09-10T04:29:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:42 smithi110 ceph-mon[97633]: pgmap v32: 129 pgs: 129 active+clean; 1.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 771 op/s 2022-09-10T04:29:43.193 INFO:tasks.workunit:Running workunit direct_io/test_short_dio_read... 2022-09-10T04:29:43.194 DEBUG:teuthology.orchestra.run.smithi081:workunit test direct_io/test_short_dio_read> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=244418605927bb7842e73fc5d2a53889b760937f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/test_short_dio_read 2022-09-10T04:29:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:44 smithi104 ceph-mon[97044]: pgmap v33: 129 pgs: 129 active+clean; 1.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 36 KiB/s wr, 765 op/s 2022-09-10T04:29:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:29:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:44 smithi081 ceph-mon[90133]: pgmap v33: 129 pgs: 129 active+clean; 1.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 36 KiB/s wr, 765 op/s 2022-09-10T04:29:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:29:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:44 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:29:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:29:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:44 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:29:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:44 smithi110 ceph-mon[97633]: pgmap v33: 129 pgs: 129 active+clean; 1.6 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 36 KiB/s wr, 765 op/s 2022-09-10T04:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:29:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:46 smithi104 ceph-mon[97044]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 41 KiB/s wr, 1.06k op/s 2022-09-10T04:29:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:46 smithi081 ceph-mon[90133]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 41 KiB/s wr, 1.06k op/s 2022-09-10T04:29:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:46 smithi110 ceph-mon[97633]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 41 KiB/s wr, 1.06k op/s 2022-09-10T04:29:47.059 INFO:tasks.workunit.client.0.smithi081.stdout:writing first 3 bytes of 10k file 2022-09-10T04:29:47.059 INFO:tasks.workunit.client.0.smithi081.stdout:reading O_DIRECT 2022-09-10T04:29:47.059 INFO:tasks.workunit.client.0.smithi081.stdout:got 10000 2022-09-10T04:29:47.061 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-09-10T04:29:47.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-09-10T04:29:47.169 INFO:tasks.workunit:Running workunit direct_io/test_sync_io... 2022-09-10T04:29:47.170 DEBUG:teuthology.orchestra.run.smithi081:workunit test direct_io/test_sync_io> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=244418605927bb7842e73fc5d2a53889b760937f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io/test_sync_io 2022-09-10T04:29:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:48 smithi104 ceph-mon[97044]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 KiB/s wr, 831 op/s 2022-09-10T04:29:48.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:48 smithi081 ceph-mon[90133]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 KiB/s wr, 831 op/s 2022-09-10T04:29:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:48 smithi110 ceph-mon[97633]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 KiB/s wr, 831 op/s 2022-09-10T04:29:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:50 smithi104 ceph-mon[97044]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 KiB/s wr, 831 op/s 2022-09-10T04:29:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:50 smithi081 ceph-mon[90133]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 KiB/s wr, 831 op/s 2022-09-10T04:29:50.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:50 smithi110 ceph-mon[97633]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 6.7 KiB/s wr, 831 op/s 2022-09-10T04:29:51.957 INFO:tasks.workunit.client.0.smithi081.stdout:writing pattern 2022-09-10T04:29:51.958 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190208 len 1024 2022-09-10T04:29:51.958 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 1024 2022-09-10T04:29:51.959 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190208 len 1024 2022-09-10T04:29:51.959 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 1024 2022-09-10T04:29:51.960 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190208 len 2048 2022-09-10T04:29:51.960 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 2048 2022-09-10T04:29:51.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190208 len 2048 2022-09-10T04:29:51.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 2048 2022-09-10T04:29:51.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190208 len 4096 2022-09-10T04:29:51.962 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 4096 2022-09-10T04:29:51.962 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190208 len 4096 2022-09-10T04:29:51.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 4096 2022-09-10T04:29:51.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190208 len 8192 2022-09-10T04:29:51.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 8192 2022-09-10T04:29:51.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190208 len 8192 2022-09-10T04:29:51.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 8192 2022-09-10T04:29:51.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190208 len 16384 2022-09-10T04:29:51.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 16384 2022-09-10T04:29:51.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190208 len 16384 2022-09-10T04:29:51.965 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190208 len 16384 2022-09-10T04:29:51.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:51.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:51.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:51.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:51.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:51.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:51.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:51.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:51.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:51.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:51.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:51.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:51.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:51.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:51.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:51.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:51.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:51.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:51.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:51.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:51.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:51.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:51.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:51.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:51.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:51.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:51.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:51.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:51.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:51.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:51.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:51.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:51.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:51.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:51.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:51.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:51.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:51.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:51.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:51.975 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:51.975 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:51.975 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:51.975 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:51.976 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:51.976 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:51.976 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:51.976 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:51.977 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:51.977 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:51.977 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:51.977 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:51.977 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:51.978 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:51.978 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:51.978 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:51.978 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:51.979 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:51.979 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:51.979 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:51.979 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:51.979 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:51.980 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:51.980 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:51.980 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:51.980 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:51.981 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:51.981 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:52.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:52.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:52.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:52.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:52.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:52.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:52.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:52.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:52.032 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:52.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:52.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:52.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:52.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:52.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:52.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:52.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:52.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:52.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:52.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:52.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:52.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:52.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:52.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:52.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:52.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:52.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:52.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:52.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:52.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:52.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:52.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:52.039 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:52.039 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:52.039 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:52.040 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:52.040 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:52.040 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:52.041 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:52.041 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:52.041 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:52.041 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:52.042 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:52.042 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:52.042 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:52.042 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:52.043 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:52.043 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:52.043 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:52.043 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:52.044 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:52.044 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:52.044 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:52.045 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:52.045 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:52.045 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:52.045 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:52.046 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:52.046 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:52.047 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:52.047 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:52.047 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:52.047 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:52.048 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:52.048 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:52.048 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:52.048 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:52.048 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:52.049 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:52.049 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:52.050 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:52.050 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:52.050 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:52.051 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:52.051 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:52.051 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:52.051 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:52.052 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:52.052 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:52.052 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:52.052 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:52.052 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:52.053 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:52.053 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:52.053 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:52.053 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:52.054 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:52.054 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:52.054 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:52.064 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:52.065 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:52.065 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:52.065 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:52.065 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:52.066 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:52.066 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:52.066 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:52.066 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:52.067 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:52.067 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:52.067 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:52.067 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:52.068 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:52.068 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:52.068 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:52.068 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:52.068 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:52.069 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:52.069 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:52.069 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:52.069 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:52.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:52.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:52.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:52.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:52.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:52.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:52.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:52.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:52.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:52.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:52.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:52.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:52.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:52.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:52.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:52.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:52.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:52.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:52.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:52.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:52.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:52.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:52.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:52.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:52.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:52.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:52.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:52.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:52.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:52.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:52.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:52.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:52.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:52.078 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:52.078 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:52.078 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:52.078 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:52.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:52.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:52.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:52.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:52.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:52.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:52.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:52.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:52.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:52.081 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:52.081 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:52.081 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:52.081 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:52.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:52.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:52.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:52.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:52.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:52.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:52.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:52.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:52.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:52.084 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:52.084 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:52.084 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:52.084 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:52.085 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:52.085 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:52.104 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:52.105 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:52.105 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:52.106 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:52.107 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:52.108 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:52.108 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:52.108 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:52.108 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:52.109 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:52.109 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:52.109 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:52.109 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:52.110 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:52.110 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:52.110 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:52.110 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:52.111 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:52.111 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:52.111 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:52.111 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:52.111 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:52.112 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:52.112 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:52.112 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:52.113 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:52.113 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:52.114 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:52.114 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:52.114 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:52.114 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:52.115 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:52.115 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:52.115 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:52.115 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:52.116 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:52.116 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:52.116 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:52.116 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:52.117 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:52.117 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:52.117 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:52.117 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:52.117 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:52.118 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:52.118 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:52.118 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:52.118 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:52.119 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:52.119 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:52.119 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:52.119 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:52.120 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:52.120 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:52.120 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:52.120 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:52.121 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:52.121 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:52.121 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190208 len 1024 2022-09-10T04:29:52.121 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 1024 2022-09-10T04:29:52.121 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190208 len 1024 2022-09-10T04:29:52.122 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 1024 2022-09-10T04:29:52.122 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190208 len 2048 2022-09-10T04:29:52.122 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 2048 2022-09-10T04:29:52.123 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190208 len 2048 2022-09-10T04:29:52.123 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 2048 2022-09-10T04:29:52.123 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190208 len 4096 2022-09-10T04:29:52.123 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 4096 2022-09-10T04:29:52.123 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190208 len 4096 2022-09-10T04:29:52.124 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 4096 2022-09-10T04:29:52.124 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190208 len 8192 2022-09-10T04:29:52.124 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 8192 2022-09-10T04:29:52.124 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190208 len 8192 2022-09-10T04:29:52.125 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 8192 2022-09-10T04:29:52.125 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190208 len 16384 2022-09-10T04:29:52.125 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 16384 2022-09-10T04:29:52.125 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190208 len 16384 2022-09-10T04:29:52.125 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190208 len 16384 2022-09-10T04:29:52.126 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:52.126 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:52.126 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:52.127 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:52.127 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:52.127 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:52.128 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:52.128 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:52.128 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:52.165 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:52.166 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:52.166 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:52.166 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:52.166 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:52.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:52.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:52.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:52.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:52.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:52.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:52.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:52.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:52.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:52.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:52.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:52.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:52.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:52.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:52.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:52.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:52.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:52.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:52.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:52.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:52.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:52.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:52.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:52.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:52.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:52.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:52.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:52.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:52.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:52.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:52.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:52.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:52.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:52.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:52.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:52.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:52.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:52.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:52.176 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:52.176 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:52.176 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:52.176 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:52.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:52.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:52.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:52.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:52.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192256 len 1024 2022-09-10T04:29:52.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 1024 2022-09-10T04:29:52.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192256 len 1024 2022-09-10T04:29:52.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 1024 2022-09-10T04:29:52.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192256 len 2048 2022-09-10T04:29:52.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 2048 2022-09-10T04:29:52.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192256 len 2048 2022-09-10T04:29:52.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 2048 2022-09-10T04:29:52.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192256 len 4096 2022-09-10T04:29:52.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 4096 2022-09-10T04:29:52.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192256 len 4096 2022-09-10T04:29:52.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 4096 2022-09-10T04:29:52.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192256 len 8192 2022-09-10T04:29:52.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 8192 2022-09-10T04:29:52.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192256 len 8192 2022-09-10T04:29:52.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 8192 2022-09-10T04:29:52.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192256 len 16384 2022-09-10T04:29:52.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 16384 2022-09-10T04:29:52.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192256 len 16384 2022-09-10T04:29:52.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192256 len 16384 2022-09-10T04:29:52.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192768 len 1024 2022-09-10T04:29:52.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 1024 2022-09-10T04:29:52.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192768 len 1024 2022-09-10T04:29:52.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 1024 2022-09-10T04:29:52.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192768 len 2048 2022-09-10T04:29:52.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 2048 2022-09-10T04:29:52.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192768 len 2048 2022-09-10T04:29:52.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 2048 2022-09-10T04:29:52.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192768 len 4096 2022-09-10T04:29:52.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 4096 2022-09-10T04:29:52.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192768 len 4096 2022-09-10T04:29:52.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 4096 2022-09-10T04:29:52.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192768 len 8192 2022-09-10T04:29:52.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 8192 2022-09-10T04:29:52.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192768 len 8192 2022-09-10T04:29:52.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 8192 2022-09-10T04:29:52.219 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4192768 len 16384 2022-09-10T04:29:52.219 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 16384 2022-09-10T04:29:52.219 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4192768 len 16384 2022-09-10T04:29:52.219 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4192768 len 16384 2022-09-10T04:29:52.220 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193280 len 1024 2022-09-10T04:29:52.220 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 1024 2022-09-10T04:29:52.220 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193280 len 1024 2022-09-10T04:29:52.221 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 1024 2022-09-10T04:29:52.221 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193280 len 2048 2022-09-10T04:29:52.222 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 2048 2022-09-10T04:29:52.222 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193280 len 2048 2022-09-10T04:29:52.223 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 2048 2022-09-10T04:29:52.223 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193280 len 4096 2022-09-10T04:29:52.223 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 4096 2022-09-10T04:29:52.224 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193280 len 4096 2022-09-10T04:29:52.224 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 4096 2022-09-10T04:29:52.224 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193280 len 8192 2022-09-10T04:29:52.225 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 8192 2022-09-10T04:29:52.225 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193280 len 8192 2022-09-10T04:29:52.226 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 8192 2022-09-10T04:29:52.226 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193280 len 16384 2022-09-10T04:29:52.227 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 16384 2022-09-10T04:29:52.227 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193280 len 16384 2022-09-10T04:29:52.227 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193280 len 16384 2022-09-10T04:29:52.228 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193792 len 1024 2022-09-10T04:29:52.228 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 1024 2022-09-10T04:29:52.228 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193792 len 1024 2022-09-10T04:29:52.229 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 1024 2022-09-10T04:29:52.229 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193792 len 2048 2022-09-10T04:29:52.229 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 2048 2022-09-10T04:29:52.230 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193792 len 2048 2022-09-10T04:29:52.230 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 2048 2022-09-10T04:29:52.231 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193792 len 4096 2022-09-10T04:29:52.231 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 4096 2022-09-10T04:29:52.231 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193792 len 4096 2022-09-10T04:29:52.231 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 4096 2022-09-10T04:29:52.232 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193792 len 8192 2022-09-10T04:29:52.232 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 8192 2022-09-10T04:29:52.232 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193792 len 8192 2022-09-10T04:29:52.232 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 8192 2022-09-10T04:29:52.232 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4193792 len 16384 2022-09-10T04:29:52.233 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 16384 2022-09-10T04:29:52.233 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4193792 len 16384 2022-09-10T04:29:52.233 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4193792 len 16384 2022-09-10T04:29:52.233 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194304 len 1024 2022-09-10T04:29:52.234 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 1024 2022-09-10T04:29:52.234 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194304 len 1024 2022-09-10T04:29:52.234 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 1024 2022-09-10T04:29:52.234 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194304 len 2048 2022-09-10T04:29:52.234 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 2048 2022-09-10T04:29:52.235 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194304 len 2048 2022-09-10T04:29:52.235 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 2048 2022-09-10T04:29:52.235 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194304 len 4096 2022-09-10T04:29:52.235 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 4096 2022-09-10T04:29:52.236 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194304 len 4096 2022-09-10T04:29:52.236 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 4096 2022-09-10T04:29:52.236 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194304 len 8192 2022-09-10T04:29:52.236 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 8192 2022-09-10T04:29:52.236 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194304 len 8192 2022-09-10T04:29:52.237 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 8192 2022-09-10T04:29:52.237 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194304 len 16384 2022-09-10T04:29:52.237 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 16384 2022-09-10T04:29:52.237 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194304 len 16384 2022-09-10T04:29:52.238 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194304 len 16384 2022-09-10T04:29:52.238 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194816 len 1024 2022-09-10T04:29:52.238 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 1024 2022-09-10T04:29:52.238 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194816 len 1024 2022-09-10T04:29:52.239 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 1024 2022-09-10T04:29:52.239 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194816 len 2048 2022-09-10T04:29:52.239 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 2048 2022-09-10T04:29:52.239 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194816 len 2048 2022-09-10T04:29:52.239 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 2048 2022-09-10T04:29:52.240 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194816 len 4096 2022-09-10T04:29:52.240 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 4096 2022-09-10T04:29:52.240 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194816 len 4096 2022-09-10T04:29:52.240 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 4096 2022-09-10T04:29:52.241 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194816 len 8192 2022-09-10T04:29:52.241 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 8192 2022-09-10T04:29:52.241 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194816 len 8192 2022-09-10T04:29:52.242 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 8192 2022-09-10T04:29:52.242 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4194816 len 16384 2022-09-10T04:29:52.253 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 16384 2022-09-10T04:29:52.253 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4194816 len 16384 2022-09-10T04:29:52.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4194816 len 16384 2022-09-10T04:29:52.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195328 len 1024 2022-09-10T04:29:52.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 1024 2022-09-10T04:29:52.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195328 len 1024 2022-09-10T04:29:52.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 1024 2022-09-10T04:29:52.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195328 len 2048 2022-09-10T04:29:52.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 2048 2022-09-10T04:29:52.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195328 len 2048 2022-09-10T04:29:52.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 2048 2022-09-10T04:29:52.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195328 len 4096 2022-09-10T04:29:52.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 4096 2022-09-10T04:29:52.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195328 len 4096 2022-09-10T04:29:52.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 4096 2022-09-10T04:29:52.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195328 len 8192 2022-09-10T04:29:52.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 8192 2022-09-10T04:29:52.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195328 len 8192 2022-09-10T04:29:52.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 8192 2022-09-10T04:29:52.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195328 len 16384 2022-09-10T04:29:52.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 16384 2022-09-10T04:29:52.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195328 len 16384 2022-09-10T04:29:52.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195328 len 16384 2022-09-10T04:29:52.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195840 len 1024 2022-09-10T04:29:52.259 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 1024 2022-09-10T04:29:52.259 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195840 len 1024 2022-09-10T04:29:52.259 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 1024 2022-09-10T04:29:52.259 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195840 len 2048 2022-09-10T04:29:52.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 2048 2022-09-10T04:29:52.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195840 len 2048 2022-09-10T04:29:52.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 2048 2022-09-10T04:29:52.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195840 len 4096 2022-09-10T04:29:52.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 4096 2022-09-10T04:29:52.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195840 len 4096 2022-09-10T04:29:52.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 4096 2022-09-10T04:29:52.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195840 len 8192 2022-09-10T04:29:52.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 8192 2022-09-10T04:29:52.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195840 len 8192 2022-09-10T04:29:52.262 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 8192 2022-09-10T04:29:52.262 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4195840 len 16384 2022-09-10T04:29:52.262 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 16384 2022-09-10T04:29:52.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4195840 len 16384 2022-09-10T04:29:52.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4195840 len 16384 2022-09-10T04:29:52.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196352 len 1024 2022-09-10T04:29:52.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 1024 2022-09-10T04:29:52.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196352 len 1024 2022-09-10T04:29:52.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 1024 2022-09-10T04:29:52.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196352 len 2048 2022-09-10T04:29:52.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 2048 2022-09-10T04:29:52.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196352 len 2048 2022-09-10T04:29:52.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 2048 2022-09-10T04:29:52.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196352 len 4096 2022-09-10T04:29:52.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 4096 2022-09-10T04:29:52.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196352 len 4096 2022-09-10T04:29:52.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 4096 2022-09-10T04:29:52.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196352 len 8192 2022-09-10T04:29:52.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 8192 2022-09-10T04:29:52.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196352 len 8192 2022-09-10T04:29:52.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 8192 2022-09-10T04:29:52.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196352 len 16384 2022-09-10T04:29:52.268 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 16384 2022-09-10T04:29:52.268 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196352 len 16384 2022-09-10T04:29:52.268 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196352 len 16384 2022-09-10T04:29:52.268 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196864 len 1024 2022-09-10T04:29:52.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 1024 2022-09-10T04:29:52.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196864 len 1024 2022-09-10T04:29:52.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 1024 2022-09-10T04:29:52.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196864 len 2048 2022-09-10T04:29:52.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 2048 2022-09-10T04:29:52.270 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196864 len 2048 2022-09-10T04:29:52.270 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 2048 2022-09-10T04:29:52.270 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196864 len 4096 2022-09-10T04:29:52.270 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 4096 2022-09-10T04:29:52.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196864 len 4096 2022-09-10T04:29:52.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 4096 2022-09-10T04:29:52.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196864 len 8192 2022-09-10T04:29:52.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 8192 2022-09-10T04:29:52.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196864 len 8192 2022-09-10T04:29:52.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 8192 2022-09-10T04:29:52.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4196864 len 16384 2022-09-10T04:29:52.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 16384 2022-09-10T04:29:52.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4196864 len 16384 2022-09-10T04:29:52.273 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4196864 len 16384 2022-09-10T04:29:52.273 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197376 len 1024 2022-09-10T04:29:52.300 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 1024 2022-09-10T04:29:52.301 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197376 len 1024 2022-09-10T04:29:52.301 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 1024 2022-09-10T04:29:52.301 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197376 len 2048 2022-09-10T04:29:52.301 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 2048 2022-09-10T04:29:52.302 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197376 len 2048 2022-09-10T04:29:52.302 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 2048 2022-09-10T04:29:52.302 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197376 len 4096 2022-09-10T04:29:52.302 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 4096 2022-09-10T04:29:52.303 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197376 len 4096 2022-09-10T04:29:52.303 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 4096 2022-09-10T04:29:52.303 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197376 len 8192 2022-09-10T04:29:52.303 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 8192 2022-09-10T04:29:52.303 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197376 len 8192 2022-09-10T04:29:52.304 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 8192 2022-09-10T04:29:52.304 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197376 len 16384 2022-09-10T04:29:52.304 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 16384 2022-09-10T04:29:52.304 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197376 len 16384 2022-09-10T04:29:52.305 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197376 len 16384 2022-09-10T04:29:52.305 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197888 len 1024 2022-09-10T04:29:52.305 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 1024 2022-09-10T04:29:52.305 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197888 len 1024 2022-09-10T04:29:52.305 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 1024 2022-09-10T04:29:52.306 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197888 len 2048 2022-09-10T04:29:52.306 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 2048 2022-09-10T04:29:52.306 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197888 len 2048 2022-09-10T04:29:52.306 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 2048 2022-09-10T04:29:52.307 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197888 len 4096 2022-09-10T04:29:52.307 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 4096 2022-09-10T04:29:52.307 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197888 len 4096 2022-09-10T04:29:52.308 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 4096 2022-09-10T04:29:52.308 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197888 len 8192 2022-09-10T04:29:52.308 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 8192 2022-09-10T04:29:52.309 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197888 len 8192 2022-09-10T04:29:52.309 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 8192 2022-09-10T04:29:52.309 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 512 offset 4197888 len 16384 2022-09-10T04:29:52.309 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 16384 2022-09-10T04:29:52.310 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 512 offset 4197888 len 16384 2022-09-10T04:29:52.310 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 512 offset 4197888 len 16384 2022-09-10T04:29:52.310 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190208 len 1024 2022-09-10T04:29:52.310 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 1024 2022-09-10T04:29:52.310 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190208 len 1024 2022-09-10T04:29:52.311 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 1024 2022-09-10T04:29:52.311 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190208 len 2048 2022-09-10T04:29:52.311 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 2048 2022-09-10T04:29:52.311 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190208 len 2048 2022-09-10T04:29:52.311 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 2048 2022-09-10T04:29:52.312 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190208 len 4096 2022-09-10T04:29:52.312 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 4096 2022-09-10T04:29:52.312 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190208 len 4096 2022-09-10T04:29:52.312 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 4096 2022-09-10T04:29:52.313 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190208 len 8192 2022-09-10T04:29:52.313 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 8192 2022-09-10T04:29:52.313 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190208 len 8192 2022-09-10T04:29:52.313 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 8192 2022-09-10T04:29:52.313 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190208 len 16384 2022-09-10T04:29:52.314 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 16384 2022-09-10T04:29:52.314 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190208 len 16384 2022-09-10T04:29:52.314 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190208 len 16384 2022-09-10T04:29:52.314 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190720 len 1024 2022-09-10T04:29:52.314 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 1024 2022-09-10T04:29:52.315 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190720 len 1024 2022-09-10T04:29:52.315 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 1024 2022-09-10T04:29:52.315 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190720 len 2048 2022-09-10T04:29:52.315 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 2048 2022-09-10T04:29:52.315 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190720 len 2048 2022-09-10T04:29:52.316 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 2048 2022-09-10T04:29:52.316 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190720 len 4096 2022-09-10T04:29:52.316 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 4096 2022-09-10T04:29:52.316 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190720 len 4096 2022-09-10T04:29:52.316 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 4096 2022-09-10T04:29:52.317 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190720 len 8192 2022-09-10T04:29:52.317 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 8192 2022-09-10T04:29:52.317 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190720 len 8192 2022-09-10T04:29:52.317 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 8192 2022-09-10T04:29:52.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4190720 len 16384 2022-09-10T04:29:52.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 16384 2022-09-10T04:29:52.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4190720 len 16384 2022-09-10T04:29:52.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4190720 len 16384 2022-09-10T04:29:52.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191232 len 1024 2022-09-10T04:29:52.319 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 1024 2022-09-10T04:29:52.319 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191232 len 1024 2022-09-10T04:29:52.319 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 1024 2022-09-10T04:29:52.355 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191232 len 2048 2022-09-10T04:29:52.355 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 2048 2022-09-10T04:29:52.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191232 len 2048 2022-09-10T04:29:52.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 2048 2022-09-10T04:29:52.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191232 len 4096 2022-09-10T04:29:52.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 4096 2022-09-10T04:29:52.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191232 len 4096 2022-09-10T04:29:52.357 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 4096 2022-09-10T04:29:52.357 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191232 len 8192 2022-09-10T04:29:52.357 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 8192 2022-09-10T04:29:52.357 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191232 len 8192 2022-09-10T04:29:52.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 8192 2022-09-10T04:29:52.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191232 len 16384 2022-09-10T04:29:52.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 16384 2022-09-10T04:29:52.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191232 len 16384 2022-09-10T04:29:52.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191232 len 16384 2022-09-10T04:29:52.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191744 len 1024 2022-09-10T04:29:52.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 1024 2022-09-10T04:29:52.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191744 len 1024 2022-09-10T04:29:52.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 1024 2022-09-10T04:29:52.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191744 len 2048 2022-09-10T04:29:52.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 2048 2022-09-10T04:29:52.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191744 len 2048 2022-09-10T04:29:52.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 2048 2022-09-10T04:29:52.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191744 len 4096 2022-09-10T04:29:52.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 4096 2022-09-10T04:29:52.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191744 len 4096 2022-09-10T04:29:52.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 4096 2022-09-10T04:29:52.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191744 len 8192 2022-09-10T04:29:52.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 8192 2022-09-10T04:29:52.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191744 len 8192 2022-09-10T04:29:52.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 8192 2022-09-10T04:29:52.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4191744 len 16384 2022-09-10T04:29:52.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 16384 2022-09-10T04:29:52.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4191744 len 16384 2022-09-10T04:29:52.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4191744 len 16384 2022-09-10T04:29:52.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192256 len 1024 2022-09-10T04:29:52.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 1024 2022-09-10T04:29:52.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192256 len 1024 2022-09-10T04:29:52.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 1024 2022-09-10T04:29:52.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192256 len 2048 2022-09-10T04:29:52.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 2048 2022-09-10T04:29:52.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192256 len 2048 2022-09-10T04:29:52.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 2048 2022-09-10T04:29:52.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192256 len 4096 2022-09-10T04:29:52.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 4096 2022-09-10T04:29:52.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192256 len 4096 2022-09-10T04:29:52.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 4096 2022-09-10T04:29:52.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192256 len 8192 2022-09-10T04:29:52.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 8192 2022-09-10T04:29:52.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192256 len 8192 2022-09-10T04:29:52.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 8192 2022-09-10T04:29:52.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192256 len 16384 2022-09-10T04:29:52.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 16384 2022-09-10T04:29:52.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192256 len 16384 2022-09-10T04:29:52.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192256 len 16384 2022-09-10T04:29:52.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192768 len 1024 2022-09-10T04:29:52.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 1024 2022-09-10T04:29:52.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192768 len 1024 2022-09-10T04:29:52.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 1024 2022-09-10T04:29:52.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192768 len 2048 2022-09-10T04:29:52.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 2048 2022-09-10T04:29:52.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192768 len 2048 2022-09-10T04:29:52.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 2048 2022-09-10T04:29:52.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192768 len 4096 2022-09-10T04:29:52.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 4096 2022-09-10T04:29:52.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192768 len 4096 2022-09-10T04:29:52.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 4096 2022-09-10T04:29:52.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192768 len 8192 2022-09-10T04:29:52.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 8192 2022-09-10T04:29:52.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192768 len 8192 2022-09-10T04:29:52.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 8192 2022-09-10T04:29:52.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4192768 len 16384 2022-09-10T04:29:52.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 16384 2022-09-10T04:29:52.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4192768 len 16384 2022-09-10T04:29:52.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4192768 len 16384 2022-09-10T04:29:52.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193280 len 1024 2022-09-10T04:29:52.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 1024 2022-09-10T04:29:52.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193280 len 1024 2022-09-10T04:29:52.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 1024 2022-09-10T04:29:52.373 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193280 len 2048 2022-09-10T04:29:52.373 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 2048 2022-09-10T04:29:52.373 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193280 len 2048 2022-09-10T04:29:52.403 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 2048 2022-09-10T04:29:52.403 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193280 len 4096 2022-09-10T04:29:52.404 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 4096 2022-09-10T04:29:52.404 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193280 len 4096 2022-09-10T04:29:52.404 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 4096 2022-09-10T04:29:52.404 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193280 len 8192 2022-09-10T04:29:52.404 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 8192 2022-09-10T04:29:52.405 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193280 len 8192 2022-09-10T04:29:52.405 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 8192 2022-09-10T04:29:52.405 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193280 len 16384 2022-09-10T04:29:52.405 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 16384 2022-09-10T04:29:52.406 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193280 len 16384 2022-09-10T04:29:52.406 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193280 len 16384 2022-09-10T04:29:52.406 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193792 len 1024 2022-09-10T04:29:52.406 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 1024 2022-09-10T04:29:52.406 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193792 len 1024 2022-09-10T04:29:52.407 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 1024 2022-09-10T04:29:52.407 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193792 len 2048 2022-09-10T04:29:52.407 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 2048 2022-09-10T04:29:52.407 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193792 len 2048 2022-09-10T04:29:52.408 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 2048 2022-09-10T04:29:52.408 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193792 len 4096 2022-09-10T04:29:52.408 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 4096 2022-09-10T04:29:52.408 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193792 len 4096 2022-09-10T04:29:52.408 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 4096 2022-09-10T04:29:52.409 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193792 len 8192 2022-09-10T04:29:52.409 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 8192 2022-09-10T04:29:52.409 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193792 len 8192 2022-09-10T04:29:52.410 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 8192 2022-09-10T04:29:52.410 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4193792 len 16384 2022-09-10T04:29:52.410 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 16384 2022-09-10T04:29:52.410 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4193792 len 16384 2022-09-10T04:29:52.411 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4193792 len 16384 2022-09-10T04:29:52.411 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194304 len 1024 2022-09-10T04:29:52.411 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 1024 2022-09-10T04:29:52.411 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194304 len 1024 2022-09-10T04:29:52.412 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 1024 2022-09-10T04:29:52.412 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194304 len 2048 2022-09-10T04:29:52.412 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 2048 2022-09-10T04:29:52.412 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194304 len 2048 2022-09-10T04:29:52.412 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 2048 2022-09-10T04:29:52.413 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194304 len 4096 2022-09-10T04:29:52.413 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 4096 2022-09-10T04:29:52.413 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194304 len 4096 2022-09-10T04:29:52.413 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 4096 2022-09-10T04:29:52.413 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194304 len 8192 2022-09-10T04:29:52.414 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 8192 2022-09-10T04:29:52.414 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194304 len 8192 2022-09-10T04:29:52.414 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 8192 2022-09-10T04:29:52.414 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194304 len 16384 2022-09-10T04:29:52.415 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 16384 2022-09-10T04:29:52.415 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194304 len 16384 2022-09-10T04:29:52.415 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194304 len 16384 2022-09-10T04:29:52.415 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194816 len 1024 2022-09-10T04:29:52.415 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 1024 2022-09-10T04:29:52.416 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194816 len 1024 2022-09-10T04:29:52.416 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 1024 2022-09-10T04:29:52.416 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194816 len 2048 2022-09-10T04:29:52.416 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 2048 2022-09-10T04:29:52.417 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194816 len 2048 2022-09-10T04:29:52.417 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 2048 2022-09-10T04:29:52.417 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194816 len 4096 2022-09-10T04:29:52.417 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 4096 2022-09-10T04:29:52.418 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194816 len 4096 2022-09-10T04:29:52.418 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 4096 2022-09-10T04:29:52.418 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194816 len 8192 2022-09-10T04:29:52.418 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 8192 2022-09-10T04:29:52.418 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194816 len 8192 2022-09-10T04:29:52.419 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 8192 2022-09-10T04:29:52.419 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4194816 len 16384 2022-09-10T04:29:52.419 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 16384 2022-09-10T04:29:52.420 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4194816 len 16384 2022-09-10T04:29:52.420 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4194816 len 16384 2022-09-10T04:29:52.420 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195328 len 1024 2022-09-10T04:29:52.420 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 1024 2022-09-10T04:29:52.421 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195328 len 1024 2022-09-10T04:29:52.421 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 1024 2022-09-10T04:29:52.421 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195328 len 2048 2022-09-10T04:29:52.421 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 2048 2022-09-10T04:29:52.421 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195328 len 2048 2022-09-10T04:29:52.422 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 2048 2022-09-10T04:29:52.422 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195328 len 4096 2022-09-10T04:29:52.438 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 4096 2022-09-10T04:29:52.438 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195328 len 4096 2022-09-10T04:29:52.439 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 4096 2022-09-10T04:29:52.439 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195328 len 8192 2022-09-10T04:29:52.439 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 8192 2022-09-10T04:29:52.439 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195328 len 8192 2022-09-10T04:29:52.440 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 8192 2022-09-10T04:29:52.440 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195328 len 16384 2022-09-10T04:29:52.440 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 16384 2022-09-10T04:29:52.440 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195328 len 16384 2022-09-10T04:29:52.440 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195328 len 16384 2022-09-10T04:29:52.441 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195840 len 1024 2022-09-10T04:29:52.441 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 1024 2022-09-10T04:29:52.441 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195840 len 1024 2022-09-10T04:29:52.441 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 1024 2022-09-10T04:29:52.441 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195840 len 2048 2022-09-10T04:29:52.442 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 2048 2022-09-10T04:29:52.442 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195840 len 2048 2022-09-10T04:29:52.442 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 2048 2022-09-10T04:29:52.442 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195840 len 4096 2022-09-10T04:29:52.443 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 4096 2022-09-10T04:29:52.443 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195840 len 4096 2022-09-10T04:29:52.443 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 4096 2022-09-10T04:29:52.443 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195840 len 8192 2022-09-10T04:29:52.444 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 8192 2022-09-10T04:29:52.444 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195840 len 8192 2022-09-10T04:29:52.444 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 8192 2022-09-10T04:29:52.444 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4195840 len 16384 2022-09-10T04:29:52.445 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 16384 2022-09-10T04:29:52.445 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4195840 len 16384 2022-09-10T04:29:52.445 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4195840 len 16384 2022-09-10T04:29:52.445 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196352 len 1024 2022-09-10T04:29:52.446 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 1024 2022-09-10T04:29:52.446 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196352 len 1024 2022-09-10T04:29:52.446 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 1024 2022-09-10T04:29:52.446 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196352 len 2048 2022-09-10T04:29:52.447 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 2048 2022-09-10T04:29:52.447 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196352 len 2048 2022-09-10T04:29:52.447 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 2048 2022-09-10T04:29:52.447 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196352 len 4096 2022-09-10T04:29:52.448 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 4096 2022-09-10T04:29:52.448 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196352 len 4096 2022-09-10T04:29:52.448 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 4096 2022-09-10T04:29:52.449 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196352 len 8192 2022-09-10T04:29:52.449 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 8192 2022-09-10T04:29:52.449 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196352 len 8192 2022-09-10T04:29:52.449 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 8192 2022-09-10T04:29:52.450 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196352 len 16384 2022-09-10T04:29:52.450 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 16384 2022-09-10T04:29:52.450 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196352 len 16384 2022-09-10T04:29:52.450 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196352 len 16384 2022-09-10T04:29:52.450 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196864 len 1024 2022-09-10T04:29:52.451 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 1024 2022-09-10T04:29:52.451 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196864 len 1024 2022-09-10T04:29:52.451 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 1024 2022-09-10T04:29:52.451 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196864 len 2048 2022-09-10T04:29:52.452 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 2048 2022-09-10T04:29:52.452 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196864 len 2048 2022-09-10T04:29:52.452 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 2048 2022-09-10T04:29:52.452 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196864 len 4096 2022-09-10T04:29:52.453 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 4096 2022-09-10T04:29:52.453 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196864 len 4096 2022-09-10T04:29:52.453 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 4096 2022-09-10T04:29:52.453 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196864 len 8192 2022-09-10T04:29:52.453 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 8192 2022-09-10T04:29:52.454 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196864 len 8192 2022-09-10T04:29:52.454 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 8192 2022-09-10T04:29:52.454 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4196864 len 16384 2022-09-10T04:29:52.454 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 16384 2022-09-10T04:29:52.455 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4196864 len 16384 2022-09-10T04:29:52.455 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4196864 len 16384 2022-09-10T04:29:52.455 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197376 len 1024 2022-09-10T04:29:52.455 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 1024 2022-09-10T04:29:52.456 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197376 len 1024 2022-09-10T04:29:52.456 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 1024 2022-09-10T04:29:52.456 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197376 len 2048 2022-09-10T04:29:52.456 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 2048 2022-09-10T04:29:52.457 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197376 len 2048 2022-09-10T04:29:52.457 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 2048 2022-09-10T04:29:52.457 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197376 len 4096 2022-09-10T04:29:52.457 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 4096 2022-09-10T04:29:52.458 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197376 len 4096 2022-09-10T04:29:52.458 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 4096 2022-09-10T04:29:52.493 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197376 len 8192 2022-09-10T04:29:52.494 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 8192 2022-09-10T04:29:52.494 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197376 len 8192 2022-09-10T04:29:52.494 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 8192 2022-09-10T04:29:52.494 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197376 len 16384 2022-09-10T04:29:52.494 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 16384 2022-09-10T04:29:52.495 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197376 len 16384 2022-09-10T04:29:52.495 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197376 len 16384 2022-09-10T04:29:52.496 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197888 len 1024 2022-09-10T04:29:52.496 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 1024 2022-09-10T04:29:52.496 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197888 len 1024 2022-09-10T04:29:52.497 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 1024 2022-09-10T04:29:52.497 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197888 len 2048 2022-09-10T04:29:52.498 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 2048 2022-09-10T04:29:52.498 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197888 len 2048 2022-09-10T04:29:52.498 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 2048 2022-09-10T04:29:52.499 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197888 len 4096 2022-09-10T04:29:52.499 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 4096 2022-09-10T04:29:52.499 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197888 len 4096 2022-09-10T04:29:52.499 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 4096 2022-09-10T04:29:52.499 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197888 len 8192 2022-09-10T04:29:52.500 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 8192 2022-09-10T04:29:52.500 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197888 len 8192 2022-09-10T04:29:52.500 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 8192 2022-09-10T04:29:52.500 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1024 offset 4197888 len 16384 2022-09-10T04:29:52.501 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 16384 2022-09-10T04:29:52.501 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1024 offset 4197888 len 16384 2022-09-10T04:29:52.501 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1024 offset 4197888 len 16384 2022-09-10T04:29:52.502 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190208 len 1024 2022-09-10T04:29:52.502 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 1024 2022-09-10T04:29:52.502 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190208 len 1024 2022-09-10T04:29:52.502 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 1024 2022-09-10T04:29:52.502 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190208 len 2048 2022-09-10T04:29:52.503 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 2048 2022-09-10T04:29:52.503 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190208 len 2048 2022-09-10T04:29:52.503 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 2048 2022-09-10T04:29:52.503 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190208 len 4096 2022-09-10T04:29:52.503 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 4096 2022-09-10T04:29:52.504 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190208 len 4096 2022-09-10T04:29:52.504 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 4096 2022-09-10T04:29:52.504 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190208 len 8192 2022-09-10T04:29:52.504 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 8192 2022-09-10T04:29:52.505 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190208 len 8192 2022-09-10T04:29:52.505 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 8192 2022-09-10T04:29:52.505 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190208 len 16384 2022-09-10T04:29:52.505 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 16384 2022-09-10T04:29:52.505 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190208 len 16384 2022-09-10T04:29:52.506 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190208 len 16384 2022-09-10T04:29:52.506 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190720 len 1024 2022-09-10T04:29:52.506 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 1024 2022-09-10T04:29:52.506 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190720 len 1024 2022-09-10T04:29:52.506 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 1024 2022-09-10T04:29:52.507 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190720 len 2048 2022-09-10T04:29:52.507 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 2048 2022-09-10T04:29:52.507 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190720 len 2048 2022-09-10T04:29:52.507 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 2048 2022-09-10T04:29:52.507 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190720 len 4096 2022-09-10T04:29:52.508 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 4096 2022-09-10T04:29:52.508 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190720 len 4096 2022-09-10T04:29:52.508 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 4096 2022-09-10T04:29:52.508 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190720 len 8192 2022-09-10T04:29:52.509 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 8192 2022-09-10T04:29:52.509 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190720 len 8192 2022-09-10T04:29:52.509 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 8192 2022-09-10T04:29:52.510 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4190720 len 16384 2022-09-10T04:29:52.510 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 16384 2022-09-10T04:29:52.510 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4190720 len 16384 2022-09-10T04:29:52.510 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4190720 len 16384 2022-09-10T04:29:52.510 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191232 len 1024 2022-09-10T04:29:52.511 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 1024 2022-09-10T04:29:52.511 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191232 len 1024 2022-09-10T04:29:52.511 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 1024 2022-09-10T04:29:52.511 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191232 len 2048 2022-09-10T04:29:52.511 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 2048 2022-09-10T04:29:52.512 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191232 len 2048 2022-09-10T04:29:52.512 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 2048 2022-09-10T04:29:52.512 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191232 len 4096 2022-09-10T04:29:52.512 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 4096 2022-09-10T04:29:52.513 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191232 len 4096 2022-09-10T04:29:52.513 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 4096 2022-09-10T04:29:52.513 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191232 len 8192 2022-09-10T04:29:52.513 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 8192 2022-09-10T04:29:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 -- 172.21.15.81:0/4117981357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff918071910 msgr2=0x7ff91810a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 --2- 172.21.15.81:0/4117981357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff918071910 0x7ff91810a060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff908009810 tx=0x7ff908009b20 comp rx=0 tx=0).stop 2022-09-10T04:29:52.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 -- 172.21.15.81:0/4117981357 shutdown_connections 2022-09-10T04:29:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 --2- 172.21.15.81:0/4117981357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff91810a5a0 0x7ff91810c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 --2- 172.21.15.81:0/4117981357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 0x7ff9180713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 --2- 172.21.15.81:0/4117981357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff918071910 0x7ff91810a060 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 -- 172.21.15.81:0/4117981357 >> 172.21.15.81:0/4117981357 conn(0x7ff9180f9880 msgr2=0x7ff9180fbc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 -- 172.21.15.81:0/4117981357 shutdown_connections 2022-09-10T04:29:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.535+0000 7ff91f59d700 1 -- 172.21.15.81:0/4117981357 wait complete. 2022-09-10T04:29:52.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:52 smithi104 ceph-mon[97044]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 11 KiB/s wr, 837 op/s 2022-09-10T04:29:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 Processor -- start 2022-09-10T04:29:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 -- start start 2022-09-10T04:29:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 0x7ff91819b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff918071910 0x7ff91819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff91810a5a0 0x7ff918195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181961d0 con 0x7ff918071910 2022-09-10T04:29:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918196340 con 0x7ff918071000 2022-09-10T04:29:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.537+0000 7ff91f59d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff918196500 con 0x7ff91810a5a0 2022-09-10T04:29:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 0x7ff91819b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff91810a5a0 0x7ff918195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 0x7ff91819b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:50224/0 (socket says 172.21.15.81:50224) 2022-09-10T04:29:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91e59b700 1 -- 172.21.15.81:0/3779330473 learned_addr learned my addr 172.21.15.81:0/3779330473 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff918071910 0x7ff91819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 -- 172.21.15.81:0/3779330473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 msgr2=0x7ff91819b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 0x7ff91819b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 -- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff918071910 msgr2=0x7ff91819bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff918071910 0x7ff91819bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 -- 172.21.15.81:0/3779330473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff908009470 con 0x7ff91810a5a0 2022-09-10T04:29:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.538+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff91810a5a0 0x7ff918195a30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff91000ea80 tx=0x7ff91000ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.539+0000 7ff90f7fe700 1 -- 172.21.15.81:0/3779330473 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9100098c0 con 0x7ff91810a5a0 2022-09-10T04:29:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.539+0000 7ff90f7fe700 1 -- 172.21.15.81:0/3779330473 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff91000ce10 con 0x7ff91810a5a0 2022-09-10T04:29:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.539+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9181967e0 con 0x7ff91810a5a0 2022-09-10T04:29:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.539+0000 7ff90f7fe700 1 -- 172.21.15.81:0/3779330473 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910004280 con 0x7ff91810a5a0 2022-09-10T04:29:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.539+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9181a2a00 con 0x7ff91810a5a0 2022-09-10T04:29:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.542+0000 7ff90f7fe700 1 -- 172.21.15.81:0/3779330473 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) v1 ==== 127855+0+0 (secure 0 0 0) 0x7ff91000c040 con 0x7ff91810a5a0 2022-09-10T04:29:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.544+0000 7ff90f7fe700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff9040992a0 0x7ff90409b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.544+0000 7ff90f7fe700 1 -- 172.21.15.81:0/3779330473 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7ff910014070 con 0x7ff91810a5a0 2022-09-10T04:29:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.544+0000 7ff933ddcb80 1 Processor -- start 2022-09-10T04:29:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.544+0000 7ff91e59b700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff9040992a0 0x7ff90409b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 -- start start 2022-09-10T04:29:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 0x55a32da515d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4b6f0 0x55a32da4bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32d936070 con 0x55a32da4b6f0 2022-09-10T04:29:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32d9a09d0 con 0x55a32da4c0a0 2022-09-10T04:29:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32d99a170 con 0x55a32da51200 2022-09-10T04:29:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 0x55a32da515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 0x55a32da515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39436/0 (socket says 172.21.15.81:39436) 2022-09-10T04:29:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff91e59b700 1 -- 172.21.15.81:0/1289298852 learned_addr learned my addr 172.21.15.81:0/1289298852 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.545+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4b6f0 0x55a32da4bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff9040992a0 0x7ff90409b750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9140057d0 tx=0x7ff914005ea0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 -- 172.21.15.81:0/1289298852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 msgr2=0x55a32da57f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 -- 172.21.15.81:0/1289298852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4b6f0 msgr2=0x55a32da4bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4b6f0 0x55a32da4bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 -- 172.21.15.81:0/1289298852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a32d8225d0 con 0x55a32da51200 2022-09-10T04:29:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4b6f0 0x55a32da4bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:29:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff91e59b700 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 0x55a32da515d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff914004330 tx=0x7ff914014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1660a51beca58fc server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.546+0000 7ff90cff9700 1 -- 172.21.15.81:0/1289298852 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff914005220 con 0x55a32da51200 2022-09-10T04:29:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff90cff9700 1 -- 172.21.15.81:0/1289298852 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff914005380 con 0x55a32da51200 2022-09-10T04:29:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff90cff9700 1 -- 172.21.15.81:0/1289298852 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9140039e0 con 0x55a32da51200 2022-09-10T04:29:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1289298852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 msgr2=0x55a32da515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 0x55a32da515d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff914004330 tx=0x7ff914014000 comp rx=0 tx=0).stop 2022-09-10T04:29:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1289298852 shutdown_connections 2022-09-10T04:29:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da51200 0x55a32da515d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1289298852 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4b6f0 0x55a32da4bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1289298852 >> 172.21.15.81:0/1289298852 conn(0x55a32d8ab6b0 msgr2=0x55a32da46010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.547+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1289298852 shutdown_connections 2022-09-10T04:29:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.548+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1289298852 wait complete. 2022-09-10T04:29:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.548+0000 7ff933ddcb80 1 -- start start 2022-09-10T04:29:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 Processor -- start 2022-09-10T04:29:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da57a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4c0a0 0x55a32da54c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da51200 0x55a32da551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32d8880b0 con 0x55a32da4c0a0 2022-09-10T04:29:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32da556f0 con 0x55a32da51200 2022-09-10T04:29:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32da558b0 con 0x55a32da4b6f0 2022-09-10T04:29:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.549+0000 7ff933ddcb80 10 client.0 _ll_register_callbacks cb 0x55a32d945360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da57a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da57a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39442/0 (socket says 172.21.15.81:39442) 2022-09-10T04:29:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 -- 172.21.15.81:0/390647056 learned_addr learned my addr 172.21.15.81:0/390647056 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4c0a0 0x55a32da54c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da51200 0x55a32da551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 -- 172.21.15.81:0/390647056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da51200 msgr2=0x55a32da551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da51200 0x55a32da551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 -- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4c0a0 msgr2=0x55a32da54c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4c0a0 0x55a32da54c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 -- 172.21.15.81:0/390647056 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff914017070 con 0x55a32da4b6f0 2022-09-10T04:29:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4c0a0 0x55a32da54c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:29:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff91e59b700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da57a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff914004800 tx=0x7ff9140048e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c69b2e96d1ceb72 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.550+0000 7ff901ffb700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:29:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff901ffb700 1 -- 172.21.15.81:0/390647056 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91402c070 con 0x55a32da4b6f0 2022-09-10T04:29:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff901ffb700 1 -- 172.21.15.81:0/390647056 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7ff9140054f0 con 0x55a32da4b6f0 2022-09-10T04:29:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a32da3e930 con 0x55a32da4b6f0 2022-09-10T04:29:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff901ffb700 1 -- 172.21.15.81:0/390647056 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff914004de0 con 0x55a32da4b6f0 2022-09-10T04:29:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff933ddcb80 10 client.25552 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a32d7ced80 con 0x55a32da4b6f0 2022-09-10T04:29:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.551+0000 7ff901ffb700 1 -- 172.21.15.81:0/390647056 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55a32da3e930 con 0x55a32da4b6f0 2022-09-10T04:29:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff901ffb700 1 -- 172.21.15.81:0/390647056 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7ff914024030 con 0x55a32da4b6f0 2022-09-10T04:29:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff933ddcb80 10 client.25552 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff933ddcb80 10 client.25552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff933ddcb80 20 client.25552 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32da65360 0x55a32da67770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff933ddcb80 4 client.25552 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:29:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a32da1b4b0 con 0x55a32da65360 2022-09-10T04:29:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.552+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32da65360 0x55a32da67770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.581 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:52 smithi081 ceph-mon[90133]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 11 KiB/s wr, 837 op/s 2022-09-10T04:29:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.553+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32da65360 0x55a32da67770 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c644b814fa1350ef server_cookie=da3bcc502ed9b11d in_seq=0 out_seq=0 2022-09-10T04:29:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.553+0000 7ff901ffb700 10 client.25552 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:52.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff901ffb700 1 -- 172.21.15.81:0/390647056 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55a32da1b4b0 con 0x55a32da65360 2022-09-10T04:29:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff901ffb700 10 client.25552 handle_command_reply: tid=0 2022-09-10T04:29:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 client.25552 shutdown 2022-09-10T04:29:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 msgr2=0x55a32da57a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da57a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff914004800 tx=0x7ff9140048e0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 shutdown_connections 2022-09-10T04:29:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32da65360 0x55a32da67770 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da51200 0x55a32da551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.554+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da57a60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.555+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/390647056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da4c0a0 0x55a32da54c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.555+0000 7ff901ffb700 0 client.25552 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.555+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 >> 172.21.15.81:0/390647056 conn(0x55a32d8ab6b0 msgr2=0x55a32da4a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.555+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 shutdown_connections 2022-09-10T04:29:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.555+0000 7ff933ddcb80 1 -- 172.21.15.81:0/390647056 wait complete. 2022-09-10T04:29:52.587 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.555+0000 7ff933ddcb80 20 client.25552 trim_cache size 0 max 16384 2022-09-10T04:29:52.587 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191232 len 8192 2022-09-10T04:29:52.588 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 8192 2022-09-10T04:29:52.588 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191232 len 16384 2022-09-10T04:29:52.588 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 16384 2022-09-10T04:29:52.588 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191232 len 16384 2022-09-10T04:29:52.588 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191232 len 16384 2022-09-10T04:29:52.589 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191744 len 1024 2022-09-10T04:29:52.589 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 1024 2022-09-10T04:29:52.589 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191744 len 1024 2022-09-10T04:29:52.589 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 1024 2022-09-10T04:29:52.590 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191744 len 2048 2022-09-10T04:29:52.590 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 2048 2022-09-10T04:29:52.590 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191744 len 2048 2022-09-10T04:29:52.590 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 2048 2022-09-10T04:29:52.591 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191744 len 4096 2022-09-10T04:29:52.591 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 4096 2022-09-10T04:29:52.591 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191744 len 4096 2022-09-10T04:29:52.591 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 4096 2022-09-10T04:29:52.592 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191744 len 8192 2022-09-10T04:29:52.592 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 8192 2022-09-10T04:29:52.592 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191744 len 8192 2022-09-10T04:29:52.592 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 8192 2022-09-10T04:29:52.592 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4191744 len 16384 2022-09-10T04:29:52.593 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 16384 2022-09-10T04:29:52.593 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4191744 len 16384 2022-09-10T04:29:52.593 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4191744 len 16384 2022-09-10T04:29:52.593 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192256 len 1024 2022-09-10T04:29:52.593 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 1024 2022-09-10T04:29:52.594 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192256 len 1024 2022-09-10T04:29:52.594 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 1024 2022-09-10T04:29:52.594 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192256 len 2048 2022-09-10T04:29:52.594 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 2048 2022-09-10T04:29:52.595 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192256 len 2048 2022-09-10T04:29:52.595 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 2048 2022-09-10T04:29:52.595 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192256 len 4096 2022-09-10T04:29:52.595 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 4096 2022-09-10T04:29:52.595 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192256 len 4096 2022-09-10T04:29:52.596 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 4096 2022-09-10T04:29:52.596 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192256 len 8192 2022-09-10T04:29:52.596 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 8192 2022-09-10T04:29:52.596 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192256 len 8192 2022-09-10T04:29:52.597 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 8192 2022-09-10T04:29:52.597 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192256 len 16384 2022-09-10T04:29:52.597 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 16384 2022-09-10T04:29:52.597 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192256 len 16384 2022-09-10T04:29:52.597 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192256 len 16384 2022-09-10T04:29:52.598 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192768 len 1024 2022-09-10T04:29:52.598 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 1024 2022-09-10T04:29:52.598 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192768 len 1024 2022-09-10T04:29:52.598 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 1024 2022-09-10T04:29:52.598 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192768 len 2048 2022-09-10T04:29:52.599 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 2048 2022-09-10T04:29:52.599 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192768 len 2048 2022-09-10T04:29:52.599 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 2048 2022-09-10T04:29:52.600 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192768 len 4096 2022-09-10T04:29:52.600 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 4096 2022-09-10T04:29:52.600 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192768 len 4096 2022-09-10T04:29:52.600 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 4096 2022-09-10T04:29:52.601 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192768 len 8192 2022-09-10T04:29:52.601 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 8192 2022-09-10T04:29:52.601 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192768 len 8192 2022-09-10T04:29:52.601 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 8192 2022-09-10T04:29:52.601 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4192768 len 16384 2022-09-10T04:29:52.602 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 16384 2022-09-10T04:29:52.602 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4192768 len 16384 2022-09-10T04:29:52.602 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4192768 len 16384 2022-09-10T04:29:52.602 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193280 len 1024 2022-09-10T04:29:52.603 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 1024 2022-09-10T04:29:52.603 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193280 len 1024 2022-09-10T04:29:52.603 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 1024 2022-09-10T04:29:52.603 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193280 len 2048 2022-09-10T04:29:52.603 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 2048 2022-09-10T04:29:52.604 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193280 len 2048 2022-09-10T04:29:52.604 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 2048 2022-09-10T04:29:52.604 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193280 len 4096 2022-09-10T04:29:52.604 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 4096 2022-09-10T04:29:52.604 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193280 len 4096 2022-09-10T04:29:52.605 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 4096 2022-09-10T04:29:52.605 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193280 len 8192 2022-09-10T04:29:52.605 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 8192 2022-09-10T04:29:52.605 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193280 len 8192 2022-09-10T04:29:52.606 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 8192 2022-09-10T04:29:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 Processor -- start 2022-09-10T04:29:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 -- start start 2022-09-10T04:29:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da40d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32da4eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32da481f0 con 0x55a32da51200 2022-09-10T04:29:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32da554e0 con 0x55a32da4c0a0 2022-09-10T04:29:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.571+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32da4fa80 con 0x55a32da4b6f0 2022-09-10T04:29:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da40d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da40d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:39454/0 (socket says 172.21.15.81:39454) 2022-09-10T04:29:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 -- 172.21.15.81:0/2941141302 learned_addr learned my addr 172.21.15.81:0/2941141302 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32da4eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 -- 172.21.15.81:0/2941141302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 msgr2=0x55a32da57590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 -- 172.21.15.81:0/2941141302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 msgr2=0x55a32da4eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32da4eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.572+0000 7ff91e59b700 1 -- 172.21.15.81:0/2941141302 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a32d7cd250 con 0x55a32da4b6f0 2022-09-10T04:29:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.573+0000 7ff91e59b700 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da40d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff9140092f0 tx=0x7ff9140059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e35adc513705a2c server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.573+0000 7ff90d7fa700 1 -- 172.21.15.81:0/2941141302 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91402c070 con 0x55a32da4b6f0 2022-09-10T04:29:52.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.573+0000 7ff90d7fa700 1 -- 172.21.15.81:0/2941141302 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff9140054f0 con 0x55a32da4b6f0 2022-09-10T04:29:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.573+0000 7ff90d7fa700 1 -- 172.21.15.81:0/2941141302 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91401d530 con 0x55a32da4b6f0 2022-09-10T04:29:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 -- 172.21.15.81:0/2941141302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 msgr2=0x55a32da40d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da40d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff9140092f0 tx=0x7ff9140059e0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 -- 172.21.15.81:0/2941141302 shutdown_connections 2022-09-10T04:29:52.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32da4eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4b6f0 0x55a32da40d10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/2941141302 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4c0a0 0x55a32da57590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 -- 172.21.15.81:0/2941141302 >> 172.21.15.81:0/2941141302 conn(0x55a32d8ab6b0 msgr2=0x55a32da4a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 -- 172.21.15.81:0/2941141302 shutdown_connections 2022-09-10T04:29:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 -- 172.21.15.81:0/2941141302 wait complete. 2022-09-10T04:29:52.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.574+0000 7ff933ddcb80 1 -- start start 2022-09-10T04:29:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 Processor -- start 2022-09-10T04:29:52.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4b6f0 0x55a32da5a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4c0a0 0x55a32da5ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32dab5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32da5b3a0 con 0x55a32da51200 2022-09-10T04:29:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32dab6070 con 0x55a32da4b6f0 2022-09-10T04:29:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a32dab6230 con 0x55a32da4c0a0 2022-09-10T04:29:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.575+0000 7ff933ddcb80 10 client.0 _ll_register_callbacks cb 0x55a32d945360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:29:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4b6f0 0x55a32da5a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91e59b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4b6f0 0x55a32da5a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:50286/0 (socket says 172.21.15.81:50286) 2022-09-10T04:29:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91dd9a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4c0a0 0x55a32da5ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91e59b700 1 -- 172.21.15.81:0/1208552907 learned_addr learned my addr 172.21.15.81:0/1208552907 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:29:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91ed9c700 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32dab5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91dd9a700 1 -- 172.21.15.81:0/1208552907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4b6f0 msgr2=0x55a32da5a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4b6f0 0x55a32da5a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91dd9a700 1 -- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 msgr2=0x55a32dab5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:29:52.639 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32dab5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.576+0000 7ff91dd9a700 1 -- 172.21.15.81:0/1208552907 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a32d7ced80 con 0x55a32da4c0a0 2022-09-10T04:29:52.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff91dd9a700 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4c0a0 0x55a32da5ac30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff908009760 tx=0x7ff908005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7b9eabd1f0b24c server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:29:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff901ffb700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:29:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff901ffb700 1 -- 172.21.15.81:0/1208552907 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90801d040 con 0x55a32da4c0a0 2022-09-10T04:29:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff901ffb700 1 -- 172.21.15.81:0/1208552907 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7ff908028820 con 0x55a32da4c0a0 2022-09-10T04:29:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff914005380 con 0x55a32da4c0a0 2022-09-10T04:29:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff901ffb700 1 -- 172.21.15.81:0/1208552907 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9080282b0 con 0x55a32da4c0a0 2022-09-10T04:29:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff933ddcb80 10 client.25558 fetch_fsmap learned FSMap version 28 2022-09-10T04:29:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a32d89ba20 con 0x55a32da4c0a0 2022-09-10T04:29:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.577+0000 7ff901ffb700 1 -- 172.21.15.81:0/1208552907 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7ff914005380 con 0x55a32da4c0a0 2022-09-10T04:29:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff901ffb700 1 -- 172.21.15.81:0/1208552907 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7ff90800bcc0 con 0x55a32da4c0a0 2022-09-10T04:29:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff933ddcb80 10 client.25558 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:29:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff933ddcb80 10 client.25558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:29:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff933ddcb80 20 client.25558 populate_metadata read hostname 'smithi081' 2022-09-10T04:29:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32dab7a20 0x55a32dab9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:29:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff933ddcb80 4 client.25558 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:29:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a32d9936e0 con 0x55a32dab7a20 2022-09-10T04:29:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.578+0000 7ff91e59b700 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32dab7a20 0x55a32dab9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:29:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.579+0000 7ff91e59b700 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32dab7a20 0x55a32dab9e30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fa7721915cbd30f server_cookie=2ffe27e9ede4fc2b in_seq=0 out_seq=0 2022-09-10T04:29:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.579+0000 7ff901ffb700 10 client.25558 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff901ffb700 1 -- 172.21.15.81:0/1208552907 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a32d9936e0 con 0x55a32dab7a20 2022-09-10T04:29:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff901ffb700 10 client.25558 handle_command_reply: tid=0 2022-09-10T04:29:52.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 client.25558 shutdown 2022-09-10T04:29:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4c0a0 msgr2=0x55a32da5ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4c0a0 0x55a32da5ac30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff908009760 tx=0x7ff908005b10 comp rx=0 tx=0).stop 2022-09-10T04:29:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 shutdown_connections 2022-09-10T04:29:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a32da51200 0x55a32dab5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55a32dab7a20 0x55a32dab9e30 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55a32da4b6f0 0x55a32da5a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 --2- 172.21.15.81:0/1208552907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a32da4c0a0 0x55a32da5ac30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 >> 172.21.15.81:0/1208552907 conn(0x55a32d8ab6b0 msgr2=0x55a32da4eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.588+0000 7ff901ffb700 0 client.25558 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:29:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.589+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 shutdown_connections 2022-09-10T04:29:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.589+0000 7ff933ddcb80 1 -- 172.21.15.81:0/1208552907 wait complete. 2022-09-10T04:29:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.589+0000 7ff933ddcb80 20 client.25558 trim_cache size 0 max 16384 2022-09-10T04:29:52.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:29:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:29:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:29:52.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:29:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff9040992a0 msgr2=0x7ff90409b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.652 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff9040992a0 0x7ff90409b750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff9140057d0 tx=0x7ff914005ea0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff91810a5a0 msgr2=0x7ff918195a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:29:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff91810a5a0 0x7ff918195a30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff91000ea80 tx=0x7ff91000ed90 comp rx=0 tx=0).stop 2022-09-10T04:29:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 shutdown_connections 2022-09-10T04:29:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff91810a5a0 0x7ff918195a30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7ff9040992a0 0x7ff90409b750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff918071000 0x7ff91819b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 --2- 172.21.15.81:0/3779330473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff918071910 0x7ff91819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:29:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 >> 172.21.15.81:0/3779330473 conn(0x7ff9180f9880 msgr2=0x7ff918102000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:29:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 shutdown_connections 2022-09-10T04:29:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:29:52.590+0000 7ff91f59d700 1 -- 172.21.15.81:0/3779330473 wait complete. 2022-09-10T04:29:52.656 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193280 len 16384 2022-09-10T04:29:52.656 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 16384 2022-09-10T04:29:52.656 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193280 len 16384 2022-09-10T04:29:52.656 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193280 len 16384 2022-09-10T04:29:52.657 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193792 len 1024 2022-09-10T04:29:52.657 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 1024 2022-09-10T04:29:52.657 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193792 len 1024 2022-09-10T04:29:52.658 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 1024 2022-09-10T04:29:52.658 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193792 len 2048 2022-09-10T04:29:52.658 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 2048 2022-09-10T04:29:52.658 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193792 len 2048 2022-09-10T04:29:52.659 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 2048 2022-09-10T04:29:52.659 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193792 len 4096 2022-09-10T04:29:52.659 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 4096 2022-09-10T04:29:52.659 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193792 len 4096 2022-09-10T04:29:52.659 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 4096 2022-09-10T04:29:52.660 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193792 len 8192 2022-09-10T04:29:52.660 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 8192 2022-09-10T04:29:52.660 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193792 len 8192 2022-09-10T04:29:52.660 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 8192 2022-09-10T04:29:52.661 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4193792 len 16384 2022-09-10T04:29:52.661 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 16384 2022-09-10T04:29:52.661 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4193792 len 16384 2022-09-10T04:29:52.661 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4193792 len 16384 2022-09-10T04:29:52.661 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194304 len 1024 2022-09-10T04:29:52.662 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 1024 2022-09-10T04:29:52.662 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194304 len 1024 2022-09-10T04:29:52.662 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 1024 2022-09-10T04:29:52.662 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194304 len 2048 2022-09-10T04:29:52.663 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 2048 2022-09-10T04:29:52.663 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194304 len 2048 2022-09-10T04:29:52.663 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 2048 2022-09-10T04:29:52.663 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194304 len 4096 2022-09-10T04:29:52.664 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 4096 2022-09-10T04:29:52.664 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194304 len 4096 2022-09-10T04:29:52.664 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 4096 2022-09-10T04:29:52.664 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194304 len 8192 2022-09-10T04:29:52.665 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 8192 2022-09-10T04:29:52.665 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194304 len 8192 2022-09-10T04:29:52.665 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 8192 2022-09-10T04:29:52.665 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194304 len 16384 2022-09-10T04:29:52.666 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 16384 2022-09-10T04:29:52.666 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194304 len 16384 2022-09-10T04:29:52.666 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194304 len 16384 2022-09-10T04:29:52.666 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194816 len 1024 2022-09-10T04:29:52.666 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 1024 2022-09-10T04:29:52.667 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194816 len 1024 2022-09-10T04:29:52.667 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 1024 2022-09-10T04:29:52.667 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194816 len 2048 2022-09-10T04:29:52.668 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 2048 2022-09-10T04:29:52.668 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194816 len 2048 2022-09-10T04:29:52.668 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 2048 2022-09-10T04:29:52.669 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194816 len 4096 2022-09-10T04:29:52.669 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 4096 2022-09-10T04:29:52.669 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194816 len 4096 2022-09-10T04:29:52.670 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 4096 2022-09-10T04:29:52.670 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194816 len 8192 2022-09-10T04:29:52.670 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 8192 2022-09-10T04:29:52.671 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194816 len 8192 2022-09-10T04:29:52.671 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 8192 2022-09-10T04:29:52.671 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4194816 len 16384 2022-09-10T04:29:52.671 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 16384 2022-09-10T04:29:52.672 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4194816 len 16384 2022-09-10T04:29:52.672 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4194816 len 16384 2022-09-10T04:29:52.672 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195328 len 1024 2022-09-10T04:29:52.672 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 1024 2022-09-10T04:29:52.673 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195328 len 1024 2022-09-10T04:29:52.673 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 1024 2022-09-10T04:29:52.673 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195328 len 2048 2022-09-10T04:29:52.673 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 2048 2022-09-10T04:29:52.673 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195328 len 2048 2022-09-10T04:29:52.674 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 2048 2022-09-10T04:29:52.674 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195328 len 4096 2022-09-10T04:29:52.674 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 4096 2022-09-10T04:29:52.674 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195328 len 4096 2022-09-10T04:29:52.675 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 4096 2022-09-10T04:29:52.675 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195328 len 8192 2022-09-10T04:29:52.675 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 8192 2022-09-10T04:29:52.675 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195328 len 8192 2022-09-10T04:29:52.675 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 8192 2022-09-10T04:29:52.676 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195328 len 16384 2022-09-10T04:29:52.676 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 16384 2022-09-10T04:29:52.676 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195328 len 16384 2022-09-10T04:29:52.676 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195328 len 16384 2022-09-10T04:29:52.677 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195840 len 1024 2022-09-10T04:29:52.677 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 1024 2022-09-10T04:29:52.677 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195840 len 1024 2022-09-10T04:29:52.677 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 1024 2022-09-10T04:29:52.677 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195840 len 2048 2022-09-10T04:29:52.678 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 2048 2022-09-10T04:29:52.678 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195840 len 2048 2022-09-10T04:29:52.678 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 2048 2022-09-10T04:29:52.678 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195840 len 4096 2022-09-10T04:29:52.679 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 4096 2022-09-10T04:29:52.679 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195840 len 4096 2022-09-10T04:29:52.679 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 4096 2022-09-10T04:29:52.680 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195840 len 8192 2022-09-10T04:29:52.680 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 8192 2022-09-10T04:29:52.680 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195840 len 8192 2022-09-10T04:29:52.680 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 8192 2022-09-10T04:29:52.681 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4195840 len 16384 2022-09-10T04:29:52.681 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 16384 2022-09-10T04:29:52.681 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4195840 len 16384 2022-09-10T04:29:52.681 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4195840 len 16384 2022-09-10T04:29:52.681 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196352 len 1024 2022-09-10T04:29:52.682 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 1024 2022-09-10T04:29:52.682 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196352 len 1024 2022-09-10T04:29:52.682 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 1024 2022-09-10T04:29:52.682 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196352 len 2048 2022-09-10T04:29:52.683 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 2048 2022-09-10T04:29:52.683 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196352 len 2048 2022-09-10T04:29:52.683 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 2048 2022-09-10T04:29:52.683 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196352 len 4096 2022-09-10T04:29:52.683 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 4096 2022-09-10T04:29:52.684 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196352 len 4096 2022-09-10T04:29:52.684 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 4096 2022-09-10T04:29:52.684 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196352 len 8192 2022-09-10T04:29:52.684 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 8192 2022-09-10T04:29:52.684 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196352 len 8192 2022-09-10T04:29:52.685 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 8192 2022-09-10T04:29:52.685 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196352 len 16384 2022-09-10T04:29:52.685 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 16384 2022-09-10T04:29:52.685 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196352 len 16384 2022-09-10T04:29:52.685 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196352 len 16384 2022-09-10T04:29:52.686 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196864 len 1024 2022-09-10T04:29:52.686 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 1024 2022-09-10T04:29:52.686 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196864 len 1024 2022-09-10T04:29:52.686 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 1024 2022-09-10T04:29:52.687 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196864 len 2048 2022-09-10T04:29:52.687 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 2048 2022-09-10T04:29:52.687 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196864 len 2048 2022-09-10T04:29:52.687 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 2048 2022-09-10T04:29:52.688 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196864 len 4096 2022-09-10T04:29:52.688 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 4096 2022-09-10T04:29:52.688 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196864 len 4096 2022-09-10T04:29:52.688 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 4096 2022-09-10T04:29:52.688 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196864 len 8192 2022-09-10T04:29:52.689 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 8192 2022-09-10T04:29:52.689 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196864 len 8192 2022-09-10T04:29:52.690 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 8192 2022-09-10T04:29:52.690 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4196864 len 16384 2022-09-10T04:29:52.691 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 16384 2022-09-10T04:29:52.691 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4196864 len 16384 2022-09-10T04:29:52.692 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4196864 len 16384 2022-09-10T04:29:52.692 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197376 len 1024 2022-09-10T04:29:52.692 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 1024 2022-09-10T04:29:52.693 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197376 len 1024 2022-09-10T04:29:52.693 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 1024 2022-09-10T04:29:52.694 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197376 len 2048 2022-09-10T04:29:52.694 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 2048 2022-09-10T04:29:52.694 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197376 len 2048 2022-09-10T04:29:52.695 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 2048 2022-09-10T04:29:52.695 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197376 len 4096 2022-09-10T04:29:52.695 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 4096 2022-09-10T04:29:52.696 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197376 len 4096 2022-09-10T04:29:52.696 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 4096 2022-09-10T04:29:52.696 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197376 len 8192 2022-09-10T04:29:52.696 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 8192 2022-09-10T04:29:52.696 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197376 len 8192 2022-09-10T04:29:52.697 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 8192 2022-09-10T04:29:52.697 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197376 len 16384 2022-09-10T04:29:52.697 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 16384 2022-09-10T04:29:52.697 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197376 len 16384 2022-09-10T04:29:52.698 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197376 len 16384 2022-09-10T04:29:52.698 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197888 len 1024 2022-09-10T04:29:52.698 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:52 smithi110 ceph-mon[97633]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 11 KiB/s wr, 837 op/s 2022-09-10T04:29:52.700 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 1024 2022-09-10T04:29:52.700 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197888 len 1024 2022-09-10T04:29:52.701 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 1024 2022-09-10T04:29:52.701 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197888 len 2048 2022-09-10T04:29:52.701 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 2048 2022-09-10T04:29:52.701 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197888 len 2048 2022-09-10T04:29:52.701 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 2048 2022-09-10T04:29:52.702 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197888 len 4096 2022-09-10T04:29:52.702 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 4096 2022-09-10T04:29:52.702 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197888 len 4096 2022-09-10T04:29:52.702 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 4096 2022-09-10T04:29:52.703 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197888 len 8192 2022-09-10T04:29:52.703 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 8192 2022-09-10T04:29:52.703 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197888 len 8192 2022-09-10T04:29:52.703 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 8192 2022-09-10T04:29:52.703 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 1536 offset 4197888 len 16384 2022-09-10T04:29:52.704 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 16384 2022-09-10T04:29:52.704 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 1536 offset 4197888 len 16384 2022-09-10T04:29:52.704 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 1536 offset 4197888 len 16384 2022-09-10T04:29:52.704 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190208 len 1024 2022-09-10T04:29:52.705 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 1024 2022-09-10T04:29:52.705 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190208 len 1024 2022-09-10T04:29:52.705 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 1024 2022-09-10T04:29:52.705 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190208 len 2048 2022-09-10T04:29:52.705 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 2048 2022-09-10T04:29:52.706 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190208 len 2048 2022-09-10T04:29:52.706 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 2048 2022-09-10T04:29:52.706 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190208 len 4096 2022-09-10T04:29:52.706 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 4096 2022-09-10T04:29:52.707 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190208 len 4096 2022-09-10T04:29:52.707 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 4096 2022-09-10T04:29:52.707 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190208 len 8192 2022-09-10T04:29:52.707 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 8192 2022-09-10T04:29:52.707 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190208 len 8192 2022-09-10T04:29:52.708 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 8192 2022-09-10T04:29:52.708 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190208 len 16384 2022-09-10T04:29:52.708 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 16384 2022-09-10T04:29:52.708 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190208 len 16384 2022-09-10T04:29:52.708 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190208 len 16384 2022-09-10T04:29:52.709 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190720 len 1024 2022-09-10T04:29:52.709 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 1024 2022-09-10T04:29:52.709 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190720 len 1024 2022-09-10T04:29:52.709 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 1024 2022-09-10T04:29:52.710 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190720 len 2048 2022-09-10T04:29:52.710 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 2048 2022-09-10T04:29:52.710 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190720 len 2048 2022-09-10T04:29:52.710 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 2048 2022-09-10T04:29:52.710 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190720 len 4096 2022-09-10T04:29:52.711 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 4096 2022-09-10T04:29:52.711 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190720 len 4096 2022-09-10T04:29:52.711 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 4096 2022-09-10T04:29:52.711 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190720 len 8192 2022-09-10T04:29:52.712 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 8192 2022-09-10T04:29:52.712 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190720 len 8192 2022-09-10T04:29:52.712 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 8192 2022-09-10T04:29:52.712 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4190720 len 16384 2022-09-10T04:29:52.712 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 16384 2022-09-10T04:29:52.713 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4190720 len 16384 2022-09-10T04:29:52.713 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4190720 len 16384 2022-09-10T04:29:52.713 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191232 len 1024 2022-09-10T04:29:52.713 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 1024 2022-09-10T04:29:52.713 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191232 len 1024 2022-09-10T04:29:52.714 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 1024 2022-09-10T04:29:52.714 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191232 len 2048 2022-09-10T04:29:52.714 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 2048 2022-09-10T04:29:52.714 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191232 len 2048 2022-09-10T04:29:52.715 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 2048 2022-09-10T04:29:52.715 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191232 len 4096 2022-09-10T04:29:52.715 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 4096 2022-09-10T04:29:52.715 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191232 len 4096 2022-09-10T04:29:52.715 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 4096 2022-09-10T04:29:52.716 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191232 len 8192 2022-09-10T04:29:52.716 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 8192 2022-09-10T04:29:52.716 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191232 len 8192 2022-09-10T04:29:52.716 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 8192 2022-09-10T04:29:52.716 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191232 len 16384 2022-09-10T04:29:52.717 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 16384 2022-09-10T04:29:52.717 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191232 len 16384 2022-09-10T04:29:52.717 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191232 len 16384 2022-09-10T04:29:52.717 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191744 len 1024 2022-09-10T04:29:52.717 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 1024 2022-09-10T04:29:52.718 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191744 len 1024 2022-09-10T04:29:52.718 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 1024 2022-09-10T04:29:52.760 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191744 len 2048 2022-09-10T04:29:52.761 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 2048 2022-09-10T04:29:52.761 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191744 len 2048 2022-09-10T04:29:52.761 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 2048 2022-09-10T04:29:52.761 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191744 len 4096 2022-09-10T04:29:52.761 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 4096 2022-09-10T04:29:52.762 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191744 len 4096 2022-09-10T04:29:52.762 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 4096 2022-09-10T04:29:52.763 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191744 len 8192 2022-09-10T04:29:52.763 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 8192 2022-09-10T04:29:52.763 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191744 len 8192 2022-09-10T04:29:52.764 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 8192 2022-09-10T04:29:52.764 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4191744 len 16384 2022-09-10T04:29:52.765 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 16384 2022-09-10T04:29:52.765 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4191744 len 16384 2022-09-10T04:29:52.765 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4191744 len 16384 2022-09-10T04:29:52.766 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192256 len 1024 2022-09-10T04:29:52.766 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 1024 2022-09-10T04:29:52.766 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192256 len 1024 2022-09-10T04:29:52.767 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 1024 2022-09-10T04:29:52.768 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192256 len 2048 2022-09-10T04:29:52.768 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 2048 2022-09-10T04:29:52.768 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192256 len 2048 2022-09-10T04:29:52.769 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 2048 2022-09-10T04:29:52.769 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192256 len 4096 2022-09-10T04:29:52.770 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 4096 2022-09-10T04:29:52.770 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192256 len 4096 2022-09-10T04:29:52.770 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 4096 2022-09-10T04:29:52.771 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192256 len 8192 2022-09-10T04:29:52.771 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 8192 2022-09-10T04:29:52.772 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192256 len 8192 2022-09-10T04:29:52.772 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 8192 2022-09-10T04:29:52.773 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192256 len 16384 2022-09-10T04:29:52.773 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 16384 2022-09-10T04:29:52.773 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192256 len 16384 2022-09-10T04:29:52.774 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192256 len 16384 2022-09-10T04:29:52.774 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192768 len 1024 2022-09-10T04:29:52.774 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 1024 2022-09-10T04:29:52.775 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192768 len 1024 2022-09-10T04:29:52.775 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 1024 2022-09-10T04:29:52.776 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192768 len 2048 2022-09-10T04:29:52.776 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 2048 2022-09-10T04:29:52.776 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192768 len 2048 2022-09-10T04:29:52.777 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 2048 2022-09-10T04:29:52.777 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192768 len 4096 2022-09-10T04:29:52.777 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 4096 2022-09-10T04:29:52.777 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192768 len 4096 2022-09-10T04:29:52.778 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 4096 2022-09-10T04:29:52.778 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192768 len 8192 2022-09-10T04:29:52.778 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 8192 2022-09-10T04:29:52.778 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192768 len 8192 2022-09-10T04:29:52.779 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 8192 2022-09-10T04:29:52.779 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4192768 len 16384 2022-09-10T04:29:52.779 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 16384 2022-09-10T04:29:52.779 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4192768 len 16384 2022-09-10T04:29:52.779 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4192768 len 16384 2022-09-10T04:29:52.780 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193280 len 1024 2022-09-10T04:29:52.780 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 1024 2022-09-10T04:29:52.780 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193280 len 1024 2022-09-10T04:29:52.780 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 1024 2022-09-10T04:29:52.780 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193280 len 2048 2022-09-10T04:29:52.781 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 2048 2022-09-10T04:29:52.781 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193280 len 2048 2022-09-10T04:29:52.781 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 2048 2022-09-10T04:29:52.781 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193280 len 4096 2022-09-10T04:29:52.781 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 4096 2022-09-10T04:29:52.782 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193280 len 4096 2022-09-10T04:29:52.782 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 4096 2022-09-10T04:29:52.782 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193280 len 8192 2022-09-10T04:29:52.782 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 8192 2022-09-10T04:29:52.783 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193280 len 8192 2022-09-10T04:29:52.783 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 8192 2022-09-10T04:29:52.783 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193280 len 16384 2022-09-10T04:29:52.783 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 16384 2022-09-10T04:29:52.783 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193280 len 16384 2022-09-10T04:29:52.784 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193280 len 16384 2022-09-10T04:29:52.784 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193792 len 1024 2022-09-10T04:29:52.784 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 1024 2022-09-10T04:29:52.784 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193792 len 1024 2022-09-10T04:29:52.784 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 1024 2022-09-10T04:29:52.785 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193792 len 2048 2022-09-10T04:29:52.785 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 2048 2022-09-10T04:29:52.799 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193792 len 2048 2022-09-10T04:29:52.799 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 2048 2022-09-10T04:29:52.799 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193792 len 4096 2022-09-10T04:29:52.799 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 4096 2022-09-10T04:29:52.800 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193792 len 4096 2022-09-10T04:29:52.800 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 4096 2022-09-10T04:29:52.800 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193792 len 8192 2022-09-10T04:29:52.801 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 8192 2022-09-10T04:29:52.801 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193792 len 8192 2022-09-10T04:29:52.801 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 8192 2022-09-10T04:29:52.801 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4193792 len 16384 2022-09-10T04:29:52.802 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 16384 2022-09-10T04:29:52.802 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4193792 len 16384 2022-09-10T04:29:52.802 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4193792 len 16384 2022-09-10T04:29:52.802 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194304 len 1024 2022-09-10T04:29:52.803 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 1024 2022-09-10T04:29:52.803 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194304 len 1024 2022-09-10T04:29:52.803 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 1024 2022-09-10T04:29:52.803 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194304 len 2048 2022-09-10T04:29:52.803 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 2048 2022-09-10T04:29:52.804 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194304 len 2048 2022-09-10T04:29:52.804 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 2048 2022-09-10T04:29:52.804 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194304 len 4096 2022-09-10T04:29:52.804 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 4096 2022-09-10T04:29:52.805 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194304 len 4096 2022-09-10T04:29:52.805 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 4096 2022-09-10T04:29:52.805 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194304 len 8192 2022-09-10T04:29:52.805 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 8192 2022-09-10T04:29:52.805 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194304 len 8192 2022-09-10T04:29:52.806 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 8192 2022-09-10T04:29:52.806 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194304 len 16384 2022-09-10T04:29:52.806 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 16384 2022-09-10T04:29:52.806 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194304 len 16384 2022-09-10T04:29:52.806 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194304 len 16384 2022-09-10T04:29:52.807 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194816 len 1024 2022-09-10T04:29:52.807 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 1024 2022-09-10T04:29:52.807 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194816 len 1024 2022-09-10T04:29:52.807 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 1024 2022-09-10T04:29:52.807 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194816 len 2048 2022-09-10T04:29:52.808 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 2048 2022-09-10T04:29:52.808 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194816 len 2048 2022-09-10T04:29:52.808 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 2048 2022-09-10T04:29:52.808 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194816 len 4096 2022-09-10T04:29:52.809 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 4096 2022-09-10T04:29:52.809 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194816 len 4096 2022-09-10T04:29:52.809 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 4096 2022-09-10T04:29:52.809 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194816 len 8192 2022-09-10T04:29:52.809 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 8192 2022-09-10T04:29:52.810 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194816 len 8192 2022-09-10T04:29:52.810 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 8192 2022-09-10T04:29:52.810 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4194816 len 16384 2022-09-10T04:29:52.810 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 16384 2022-09-10T04:29:52.811 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4194816 len 16384 2022-09-10T04:29:52.811 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4194816 len 16384 2022-09-10T04:29:52.811 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195328 len 1024 2022-09-10T04:29:52.812 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 1024 2022-09-10T04:29:52.812 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195328 len 1024 2022-09-10T04:29:52.812 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 1024 2022-09-10T04:29:52.812 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195328 len 2048 2022-09-10T04:29:52.812 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 2048 2022-09-10T04:29:52.813 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195328 len 2048 2022-09-10T04:29:52.813 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 2048 2022-09-10T04:29:52.813 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195328 len 4096 2022-09-10T04:29:52.813 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 4096 2022-09-10T04:29:52.813 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195328 len 4096 2022-09-10T04:29:52.814 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 4096 2022-09-10T04:29:52.814 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195328 len 8192 2022-09-10T04:29:52.814 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 8192 2022-09-10T04:29:52.814 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195328 len 8192 2022-09-10T04:29:52.815 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 8192 2022-09-10T04:29:52.815 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195328 len 16384 2022-09-10T04:29:52.815 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 16384 2022-09-10T04:29:52.815 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195328 len 16384 2022-09-10T04:29:52.815 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195328 len 16384 2022-09-10T04:29:52.816 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195840 len 1024 2022-09-10T04:29:52.816 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 1024 2022-09-10T04:29:52.816 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195840 len 1024 2022-09-10T04:29:52.816 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 1024 2022-09-10T04:29:52.816 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195840 len 2048 2022-09-10T04:29:52.817 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 2048 2022-09-10T04:29:52.817 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195840 len 2048 2022-09-10T04:29:52.817 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 2048 2022-09-10T04:29:52.832 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195840 len 4096 2022-09-10T04:29:52.833 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 4096 2022-09-10T04:29:52.833 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195840 len 4096 2022-09-10T04:29:52.833 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 4096 2022-09-10T04:29:52.833 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195840 len 8192 2022-09-10T04:29:52.833 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 8192 2022-09-10T04:29:52.834 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195840 len 8192 2022-09-10T04:29:52.834 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 8192 2022-09-10T04:29:52.834 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4195840 len 16384 2022-09-10T04:29:52.834 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 16384 2022-09-10T04:29:52.835 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4195840 len 16384 2022-09-10T04:29:52.835 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4195840 len 16384 2022-09-10T04:29:52.835 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196352 len 1024 2022-09-10T04:29:52.835 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 1024 2022-09-10T04:29:52.835 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196352 len 1024 2022-09-10T04:29:52.836 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 1024 2022-09-10T04:29:52.836 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196352 len 2048 2022-09-10T04:29:52.836 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 2048 2022-09-10T04:29:52.836 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196352 len 2048 2022-09-10T04:29:52.836 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 2048 2022-09-10T04:29:52.837 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196352 len 4096 2022-09-10T04:29:52.837 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 4096 2022-09-10T04:29:52.837 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196352 len 4096 2022-09-10T04:29:52.837 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 4096 2022-09-10T04:29:52.838 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196352 len 8192 2022-09-10T04:29:52.838 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 8192 2022-09-10T04:29:52.838 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196352 len 8192 2022-09-10T04:29:52.838 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 8192 2022-09-10T04:29:52.839 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196352 len 16384 2022-09-10T04:29:52.839 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 16384 2022-09-10T04:29:52.839 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196352 len 16384 2022-09-10T04:29:52.839 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196352 len 16384 2022-09-10T04:29:52.839 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196864 len 1024 2022-09-10T04:29:52.840 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 1024 2022-09-10T04:29:52.840 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196864 len 1024 2022-09-10T04:29:52.840 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 1024 2022-09-10T04:29:52.840 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196864 len 2048 2022-09-10T04:29:52.840 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 2048 2022-09-10T04:29:52.841 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196864 len 2048 2022-09-10T04:29:52.841 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 2048 2022-09-10T04:29:52.841 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196864 len 4096 2022-09-10T04:29:52.841 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 4096 2022-09-10T04:29:52.842 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196864 len 4096 2022-09-10T04:29:52.842 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 4096 2022-09-10T04:29:52.842 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196864 len 8192 2022-09-10T04:29:52.842 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 8192 2022-09-10T04:29:52.842 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196864 len 8192 2022-09-10T04:29:52.843 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 8192 2022-09-10T04:29:52.843 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4196864 len 16384 2022-09-10T04:29:52.843 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 16384 2022-09-10T04:29:52.843 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4196864 len 16384 2022-09-10T04:29:52.843 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4196864 len 16384 2022-09-10T04:29:52.844 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197376 len 1024 2022-09-10T04:29:52.844 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 1024 2022-09-10T04:29:52.844 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197376 len 1024 2022-09-10T04:29:52.844 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 1024 2022-09-10T04:29:52.844 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197376 len 2048 2022-09-10T04:29:52.845 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 2048 2022-09-10T04:29:52.845 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197376 len 2048 2022-09-10T04:29:52.845 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 2048 2022-09-10T04:29:52.845 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197376 len 4096 2022-09-10T04:29:52.845 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 4096 2022-09-10T04:29:52.846 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197376 len 4096 2022-09-10T04:29:52.846 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 4096 2022-09-10T04:29:52.846 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197376 len 8192 2022-09-10T04:29:52.846 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 8192 2022-09-10T04:29:52.846 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197376 len 8192 2022-09-10T04:29:52.847 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 8192 2022-09-10T04:29:52.847 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197376 len 16384 2022-09-10T04:29:52.847 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 16384 2022-09-10T04:29:52.847 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197376 len 16384 2022-09-10T04:29:52.848 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197376 len 16384 2022-09-10T04:29:52.848 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197888 len 1024 2022-09-10T04:29:52.848 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 1024 2022-09-10T04:29:52.848 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197888 len 1024 2022-09-10T04:29:52.848 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 1024 2022-09-10T04:29:52.849 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197888 len 2048 2022-09-10T04:29:52.849 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 2048 2022-09-10T04:29:52.849 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197888 len 2048 2022-09-10T04:29:52.849 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 2048 2022-09-10T04:29:52.849 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197888 len 4096 2022-09-10T04:29:52.850 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 4096 2022-09-10T04:29:52.850 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197888 len 4096 2022-09-10T04:29:52.886 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 4096 2022-09-10T04:29:52.886 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197888 len 8192 2022-09-10T04:29:52.887 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 8192 2022-09-10T04:29:52.887 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197888 len 8192 2022-09-10T04:29:52.887 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 8192 2022-09-10T04:29:52.887 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2048 offset 4197888 len 16384 2022-09-10T04:29:52.888 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 16384 2022-09-10T04:29:52.888 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2048 offset 4197888 len 16384 2022-09-10T04:29:52.888 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2048 offset 4197888 len 16384 2022-09-10T04:29:52.896 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190208 len 1024 2022-09-10T04:29:52.896 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 1024 2022-09-10T04:29:52.897 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190208 len 1024 2022-09-10T04:29:52.897 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 1024 2022-09-10T04:29:52.898 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190208 len 2048 2022-09-10T04:29:52.898 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 2048 2022-09-10T04:29:52.899 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190208 len 2048 2022-09-10T04:29:52.899 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 2048 2022-09-10T04:29:52.900 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190208 len 4096 2022-09-10T04:29:52.900 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 4096 2022-09-10T04:29:52.900 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190208 len 4096 2022-09-10T04:29:52.901 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 4096 2022-09-10T04:29:52.901 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190208 len 8192 2022-09-10T04:29:52.902 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 8192 2022-09-10T04:29:52.902 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190208 len 8192 2022-09-10T04:29:52.903 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 8192 2022-09-10T04:29:52.903 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190208 len 16384 2022-09-10T04:29:52.903 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 16384 2022-09-10T04:29:52.904 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190208 len 16384 2022-09-10T04:29:52.904 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190208 len 16384 2022-09-10T04:29:52.905 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190720 len 1024 2022-09-10T04:29:52.905 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 1024 2022-09-10T04:29:52.906 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190720 len 1024 2022-09-10T04:29:52.906 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 1024 2022-09-10T04:29:52.907 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190720 len 2048 2022-09-10T04:29:52.907 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 2048 2022-09-10T04:29:52.907 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190720 len 2048 2022-09-10T04:29:52.908 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 2048 2022-09-10T04:29:52.908 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190720 len 4096 2022-09-10T04:29:52.909 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 4096 2022-09-10T04:29:52.909 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190720 len 4096 2022-09-10T04:29:52.910 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 4096 2022-09-10T04:29:52.910 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190720 len 8192 2022-09-10T04:29:52.910 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 8192 2022-09-10T04:29:52.911 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190720 len 8192 2022-09-10T04:29:52.911 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 8192 2022-09-10T04:29:52.911 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4190720 len 16384 2022-09-10T04:29:52.912 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 16384 2022-09-10T04:29:52.912 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4190720 len 16384 2022-09-10T04:29:52.912 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4190720 len 16384 2022-09-10T04:29:52.912 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191232 len 1024 2022-09-10T04:29:52.913 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 1024 2022-09-10T04:29:52.913 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191232 len 1024 2022-09-10T04:29:52.913 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 1024 2022-09-10T04:29:52.913 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191232 len 2048 2022-09-10T04:29:52.914 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 2048 2022-09-10T04:29:52.914 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191232 len 2048 2022-09-10T04:29:52.914 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 2048 2022-09-10T04:29:52.914 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191232 len 4096 2022-09-10T04:29:52.915 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 4096 2022-09-10T04:29:52.915 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191232 len 4096 2022-09-10T04:29:52.915 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 4096 2022-09-10T04:29:52.915 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191232 len 8192 2022-09-10T04:29:52.916 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 8192 2022-09-10T04:29:52.916 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191232 len 8192 2022-09-10T04:29:52.916 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 8192 2022-09-10T04:29:52.916 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191232 len 16384 2022-09-10T04:29:52.917 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 16384 2022-09-10T04:29:52.917 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191232 len 16384 2022-09-10T04:29:52.917 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191232 len 16384 2022-09-10T04:29:52.917 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191744 len 1024 2022-09-10T04:29:52.918 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 1024 2022-09-10T04:29:52.918 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191744 len 1024 2022-09-10T04:29:52.918 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 1024 2022-09-10T04:29:52.919 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191744 len 2048 2022-09-10T04:29:52.919 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 2048 2022-09-10T04:29:52.919 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191744 len 2048 2022-09-10T04:29:52.919 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 2048 2022-09-10T04:29:52.920 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191744 len 4096 2022-09-10T04:29:52.920 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 4096 2022-09-10T04:29:52.920 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191744 len 4096 2022-09-10T04:29:52.921 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 4096 2022-09-10T04:29:52.921 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191744 len 8192 2022-09-10T04:29:52.950 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 8192 2022-09-10T04:29:52.951 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191744 len 8192 2022-09-10T04:29:52.951 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 8192 2022-09-10T04:29:52.951 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4191744 len 16384 2022-09-10T04:29:52.952 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 16384 2022-09-10T04:29:52.952 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4191744 len 16384 2022-09-10T04:29:52.953 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4191744 len 16384 2022-09-10T04:29:52.953 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192256 len 1024 2022-09-10T04:29:52.953 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 1024 2022-09-10T04:29:52.954 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192256 len 1024 2022-09-10T04:29:52.954 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 1024 2022-09-10T04:29:52.955 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192256 len 2048 2022-09-10T04:29:52.955 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 2048 2022-09-10T04:29:52.955 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192256 len 2048 2022-09-10T04:29:52.955 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 2048 2022-09-10T04:29:52.956 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192256 len 4096 2022-09-10T04:29:52.956 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 4096 2022-09-10T04:29:52.956 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192256 len 4096 2022-09-10T04:29:52.956 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 4096 2022-09-10T04:29:52.957 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192256 len 8192 2022-09-10T04:29:52.957 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 8192 2022-09-10T04:29:52.957 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192256 len 8192 2022-09-10T04:29:52.957 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 8192 2022-09-10T04:29:52.958 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192256 len 16384 2022-09-10T04:29:52.958 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 16384 2022-09-10T04:29:52.958 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192256 len 16384 2022-09-10T04:29:52.958 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192256 len 16384 2022-09-10T04:29:52.959 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192768 len 1024 2022-09-10T04:29:52.959 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 1024 2022-09-10T04:29:52.959 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192768 len 1024 2022-09-10T04:29:52.960 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 1024 2022-09-10T04:29:52.960 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192768 len 2048 2022-09-10T04:29:52.960 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 2048 2022-09-10T04:29:52.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192768 len 2048 2022-09-10T04:29:52.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 2048 2022-09-10T04:29:52.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192768 len 4096 2022-09-10T04:29:52.961 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 4096 2022-09-10T04:29:52.962 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192768 len 4096 2022-09-10T04:29:52.962 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 4096 2022-09-10T04:29:52.962 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192768 len 8192 2022-09-10T04:29:52.962 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 8192 2022-09-10T04:29:52.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192768 len 8192 2022-09-10T04:29:52.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 8192 2022-09-10T04:29:52.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4192768 len 16384 2022-09-10T04:29:52.963 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 16384 2022-09-10T04:29:52.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4192768 len 16384 2022-09-10T04:29:52.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4192768 len 16384 2022-09-10T04:29:52.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193280 len 1024 2022-09-10T04:29:52.964 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 1024 2022-09-10T04:29:52.965 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193280 len 1024 2022-09-10T04:29:52.965 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 1024 2022-09-10T04:29:52.965 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193280 len 2048 2022-09-10T04:29:52.965 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 2048 2022-09-10T04:29:52.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193280 len 2048 2022-09-10T04:29:52.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 2048 2022-09-10T04:29:52.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193280 len 4096 2022-09-10T04:29:52.966 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 4096 2022-09-10T04:29:52.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193280 len 4096 2022-09-10T04:29:52.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 4096 2022-09-10T04:29:52.967 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193280 len 8192 2022-09-10T04:29:52.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 8192 2022-09-10T04:29:52.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193280 len 8192 2022-09-10T04:29:52.968 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 8192 2022-09-10T04:29:52.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193280 len 16384 2022-09-10T04:29:52.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 16384 2022-09-10T04:29:52.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193280 len 16384 2022-09-10T04:29:52.969 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193280 len 16384 2022-09-10T04:29:52.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193792 len 1024 2022-09-10T04:29:52.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 1024 2022-09-10T04:29:52.970 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193792 len 1024 2022-09-10T04:29:52.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 1024 2022-09-10T04:29:52.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193792 len 2048 2022-09-10T04:29:52.971 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 2048 2022-09-10T04:29:52.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193792 len 2048 2022-09-10T04:29:52.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 2048 2022-09-10T04:29:52.972 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193792 len 4096 2022-09-10T04:29:52.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 4096 2022-09-10T04:29:52.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193792 len 4096 2022-09-10T04:29:52.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 4096 2022-09-10T04:29:52.973 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193792 len 8192 2022-09-10T04:29:52.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 8192 2022-09-10T04:29:52.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193792 len 8192 2022-09-10T04:29:52.974 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 8192 2022-09-10T04:29:52.983 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4193792 len 16384 2022-09-10T04:29:52.984 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 16384 2022-09-10T04:29:52.984 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4193792 len 16384 2022-09-10T04:29:52.984 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4193792 len 16384 2022-09-10T04:29:52.984 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194304 len 1024 2022-09-10T04:29:52.985 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 1024 2022-09-10T04:29:52.985 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194304 len 1024 2022-09-10T04:29:52.985 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 1024 2022-09-10T04:29:52.985 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194304 len 2048 2022-09-10T04:29:52.985 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 2048 2022-09-10T04:29:52.986 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194304 len 2048 2022-09-10T04:29:52.986 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 2048 2022-09-10T04:29:52.986 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194304 len 4096 2022-09-10T04:29:52.987 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 4096 2022-09-10T04:29:52.987 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194304 len 4096 2022-09-10T04:29:52.988 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 4096 2022-09-10T04:29:52.988 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194304 len 8192 2022-09-10T04:29:52.988 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 8192 2022-09-10T04:29:52.989 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194304 len 8192 2022-09-10T04:29:52.989 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 8192 2022-09-10T04:29:52.989 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194304 len 16384 2022-09-10T04:29:52.990 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 16384 2022-09-10T04:29:52.990 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194304 len 16384 2022-09-10T04:29:52.990 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194304 len 16384 2022-09-10T04:29:52.990 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194816 len 1024 2022-09-10T04:29:52.991 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 1024 2022-09-10T04:29:52.991 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194816 len 1024 2022-09-10T04:29:52.991 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 1024 2022-09-10T04:29:52.991 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194816 len 2048 2022-09-10T04:29:52.992 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 2048 2022-09-10T04:29:52.992 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194816 len 2048 2022-09-10T04:29:52.992 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 2048 2022-09-10T04:29:52.993 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194816 len 4096 2022-09-10T04:29:52.993 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 4096 2022-09-10T04:29:52.993 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194816 len 4096 2022-09-10T04:29:52.993 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 4096 2022-09-10T04:29:52.994 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194816 len 8192 2022-09-10T04:29:52.994 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 8192 2022-09-10T04:29:52.994 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194816 len 8192 2022-09-10T04:29:52.994 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 8192 2022-09-10T04:29:52.995 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4194816 len 16384 2022-09-10T04:29:52.995 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 16384 2022-09-10T04:29:52.995 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4194816 len 16384 2022-09-10T04:29:52.996 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4194816 len 16384 2022-09-10T04:29:52.996 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195328 len 1024 2022-09-10T04:29:52.996 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 1024 2022-09-10T04:29:52.996 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195328 len 1024 2022-09-10T04:29:52.997 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 1024 2022-09-10T04:29:52.997 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195328 len 2048 2022-09-10T04:29:52.997 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 2048 2022-09-10T04:29:52.997 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195328 len 2048 2022-09-10T04:29:52.998 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 2048 2022-09-10T04:29:52.998 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195328 len 4096 2022-09-10T04:29:52.998 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 4096 2022-09-10T04:29:52.998 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195328 len 4096 2022-09-10T04:29:52.999 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 4096 2022-09-10T04:29:52.999 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195328 len 8192 2022-09-10T04:29:52.999 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 8192 2022-09-10T04:29:52.999 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195328 len 8192 2022-09-10T04:29:52.999 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 8192 2022-09-10T04:29:53.000 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195328 len 16384 2022-09-10T04:29:53.000 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 16384 2022-09-10T04:29:53.000 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195328 len 16384 2022-09-10T04:29:53.000 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195328 len 16384 2022-09-10T04:29:53.001 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195840 len 1024 2022-09-10T04:29:53.001 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 1024 2022-09-10T04:29:53.001 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195840 len 1024 2022-09-10T04:29:53.001 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 1024 2022-09-10T04:29:53.002 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195840 len 2048 2022-09-10T04:29:53.002 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 2048 2022-09-10T04:29:53.002 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195840 len 2048 2022-09-10T04:29:53.002 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 2048 2022-09-10T04:29:53.002 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195840 len 4096 2022-09-10T04:29:53.003 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 4096 2022-09-10T04:29:53.003 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195840 len 4096 2022-09-10T04:29:53.003 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 4096 2022-09-10T04:29:53.003 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195840 len 8192 2022-09-10T04:29:53.003 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 8192 2022-09-10T04:29:53.004 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195840 len 8192 2022-09-10T04:29:53.004 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 8192 2022-09-10T04:29:53.004 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4195840 len 16384 2022-09-10T04:29:53.004 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 16384 2022-09-10T04:29:53.017 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4195840 len 16384 2022-09-10T04:29:53.018 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4195840 len 16384 2022-09-10T04:29:53.018 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196352 len 1024 2022-09-10T04:29:53.018 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 1024 2022-09-10T04:29:53.019 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196352 len 1024 2022-09-10T04:29:53.020 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 1024 2022-09-10T04:29:53.020 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196352 len 2048 2022-09-10T04:29:53.020 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 2048 2022-09-10T04:29:53.021 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196352 len 2048 2022-09-10T04:29:53.021 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 2048 2022-09-10T04:29:53.021 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196352 len 4096 2022-09-10T04:29:53.021 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 4096 2022-09-10T04:29:53.021 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196352 len 4096 2022-09-10T04:29:53.022 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 4096 2022-09-10T04:29:53.022 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196352 len 8192 2022-09-10T04:29:53.022 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 8192 2022-09-10T04:29:53.023 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196352 len 8192 2022-09-10T04:29:53.023 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 8192 2022-09-10T04:29:53.023 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196352 len 16384 2022-09-10T04:29:53.024 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 16384 2022-09-10T04:29:53.024 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196352 len 16384 2022-09-10T04:29:53.024 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196352 len 16384 2022-09-10T04:29:53.025 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196864 len 1024 2022-09-10T04:29:53.025 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 1024 2022-09-10T04:29:53.025 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196864 len 1024 2022-09-10T04:29:53.026 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 1024 2022-09-10T04:29:53.026 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196864 len 2048 2022-09-10T04:29:53.026 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 2048 2022-09-10T04:29:53.026 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196864 len 2048 2022-09-10T04:29:53.027 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 2048 2022-09-10T04:29:53.027 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196864 len 4096 2022-09-10T04:29:53.027 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 4096 2022-09-10T04:29:53.027 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196864 len 4096 2022-09-10T04:29:53.028 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 4096 2022-09-10T04:29:53.028 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196864 len 8192 2022-09-10T04:29:53.028 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 8192 2022-09-10T04:29:53.028 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196864 len 8192 2022-09-10T04:29:53.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 8192 2022-09-10T04:29:53.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4196864 len 16384 2022-09-10T04:29:53.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 16384 2022-09-10T04:29:53.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4196864 len 16384 2022-09-10T04:29:53.029 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4196864 len 16384 2022-09-10T04:29:53.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197376 len 1024 2022-09-10T04:29:53.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 1024 2022-09-10T04:29:53.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197376 len 1024 2022-09-10T04:29:53.030 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 1024 2022-09-10T04:29:53.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197376 len 2048 2022-09-10T04:29:53.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 2048 2022-09-10T04:29:53.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197376 len 2048 2022-09-10T04:29:53.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 2048 2022-09-10T04:29:53.031 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197376 len 4096 2022-09-10T04:29:53.032 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 4096 2022-09-10T04:29:53.032 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197376 len 4096 2022-09-10T04:29:53.032 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 4096 2022-09-10T04:29:53.032 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197376 len 8192 2022-09-10T04:29:53.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 8192 2022-09-10T04:29:53.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197376 len 8192 2022-09-10T04:29:53.033 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 8192 2022-09-10T04:29:53.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197376 len 16384 2022-09-10T04:29:53.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 16384 2022-09-10T04:29:53.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197376 len 16384 2022-09-10T04:29:53.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197376 len 16384 2022-09-10T04:29:53.034 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197888 len 1024 2022-09-10T04:29:53.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 1024 2022-09-10T04:29:53.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197888 len 1024 2022-09-10T04:29:53.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 1024 2022-09-10T04:29:53.035 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197888 len 2048 2022-09-10T04:29:53.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 2048 2022-09-10T04:29:53.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197888 len 2048 2022-09-10T04:29:53.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 2048 2022-09-10T04:29:53.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197888 len 4096 2022-09-10T04:29:53.036 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 4096 2022-09-10T04:29:53.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197888 len 4096 2022-09-10T04:29:53.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 4096 2022-09-10T04:29:53.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197888 len 8192 2022-09-10T04:29:53.037 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 8192 2022-09-10T04:29:53.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197888 len 8192 2022-09-10T04:29:53.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 8192 2022-09-10T04:29:53.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 2560 offset 4197888 len 16384 2022-09-10T04:29:53.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 16384 2022-09-10T04:29:53.038 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 2560 offset 4197888 len 16384 2022-09-10T04:29:53.039 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 2560 offset 4197888 len 16384 2022-09-10T04:29:53.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190208 len 1024 2022-09-10T04:29:53.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 1024 2022-09-10T04:29:53.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190208 len 1024 2022-09-10T04:29:53.070 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 1024 2022-09-10T04:29:53.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190208 len 2048 2022-09-10T04:29:53.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 2048 2022-09-10T04:29:53.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190208 len 2048 2022-09-10T04:29:53.071 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 2048 2022-09-10T04:29:53.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190208 len 4096 2022-09-10T04:29:53.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 4096 2022-09-10T04:29:53.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190208 len 4096 2022-09-10T04:29:53.072 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 4096 2022-09-10T04:29:53.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190208 len 8192 2022-09-10T04:29:53.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 8192 2022-09-10T04:29:53.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190208 len 8192 2022-09-10T04:29:53.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 8192 2022-09-10T04:29:53.073 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190208 len 16384 2022-09-10T04:29:53.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 16384 2022-09-10T04:29:53.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190208 len 16384 2022-09-10T04:29:53.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190208 len 16384 2022-09-10T04:29:53.074 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190720 len 1024 2022-09-10T04:29:53.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 1024 2022-09-10T04:29:53.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190720 len 1024 2022-09-10T04:29:53.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 1024 2022-09-10T04:29:53.075 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190720 len 2048 2022-09-10T04:29:53.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 2048 2022-09-10T04:29:53.076 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190720 len 2048 2022-09-10T04:29:53.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 2048 2022-09-10T04:29:53.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190720 len 4096 2022-09-10T04:29:53.077 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 4096 2022-09-10T04:29:53.078 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190720 len 4096 2022-09-10T04:29:53.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 4096 2022-09-10T04:29:53.079 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190720 len 8192 2022-09-10T04:29:53.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 8192 2022-09-10T04:29:53.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190720 len 8192 2022-09-10T04:29:53.080 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 8192 2022-09-10T04:29:53.081 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4190720 len 16384 2022-09-10T04:29:53.081 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 16384 2022-09-10T04:29:53.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4190720 len 16384 2022-09-10T04:29:53.082 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4190720 len 16384 2022-09-10T04:29:53.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191232 len 1024 2022-09-10T04:29:53.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 1024 2022-09-10T04:29:53.083 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191232 len 1024 2022-09-10T04:29:53.084 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 1024 2022-09-10T04:29:53.084 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191232 len 2048 2022-09-10T04:29:53.085 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 2048 2022-09-10T04:29:53.085 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191232 len 2048 2022-09-10T04:29:53.086 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 2048 2022-09-10T04:29:53.086 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191232 len 4096 2022-09-10T04:29:53.087 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 4096 2022-09-10T04:29:53.087 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191232 len 4096 2022-09-10T04:29:53.088 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 4096 2022-09-10T04:29:53.088 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191232 len 8192 2022-09-10T04:29:53.089 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 8192 2022-09-10T04:29:53.089 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191232 len 8192 2022-09-10T04:29:53.090 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 8192 2022-09-10T04:29:53.090 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191232 len 16384 2022-09-10T04:29:53.090 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 16384 2022-09-10T04:29:53.091 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191232 len 16384 2022-09-10T04:29:53.091 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191232 len 16384 2022-09-10T04:29:53.092 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191744 len 1024 2022-09-10T04:29:53.092 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 1024 2022-09-10T04:29:53.092 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191744 len 1024 2022-09-10T04:29:53.092 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 1024 2022-09-10T04:29:53.093 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191744 len 2048 2022-09-10T04:29:53.093 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 2048 2022-09-10T04:29:53.093 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191744 len 2048 2022-09-10T04:29:53.093 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 2048 2022-09-10T04:29:53.094 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191744 len 4096 2022-09-10T04:29:53.094 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 4096 2022-09-10T04:29:53.094 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191744 len 4096 2022-09-10T04:29:53.095 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 4096 2022-09-10T04:29:53.095 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191744 len 8192 2022-09-10T04:29:53.095 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 8192 2022-09-10T04:29:53.095 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191744 len 8192 2022-09-10T04:29:53.095 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 8192 2022-09-10T04:29:53.096 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4191744 len 16384 2022-09-10T04:29:53.096 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 16384 2022-09-10T04:29:53.096 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4191744 len 16384 2022-09-10T04:29:53.096 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4191744 len 16384 2022-09-10T04:29:53.097 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192256 len 1024 2022-09-10T04:29:53.097 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 1024 2022-09-10T04:29:53.098 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192256 len 1024 2022-09-10T04:29:53.133 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 1024 2022-09-10T04:29:53.133 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192256 len 2048 2022-09-10T04:29:53.134 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 2048 2022-09-10T04:29:53.134 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192256 len 2048 2022-09-10T04:29:53.134 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 2048 2022-09-10T04:29:53.135 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192256 len 4096 2022-09-10T04:29:53.135 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 4096 2022-09-10T04:29:53.135 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192256 len 4096 2022-09-10T04:29:53.135 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 4096 2022-09-10T04:29:53.135 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192256 len 8192 2022-09-10T04:29:53.136 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 8192 2022-09-10T04:29:53.136 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192256 len 8192 2022-09-10T04:29:53.136 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 8192 2022-09-10T04:29:53.137 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192256 len 16384 2022-09-10T04:29:53.137 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 16384 2022-09-10T04:29:53.137 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192256 len 16384 2022-09-10T04:29:53.138 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192256 len 16384 2022-09-10T04:29:53.138 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192768 len 1024 2022-09-10T04:29:53.138 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 1024 2022-09-10T04:29:53.139 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192768 len 1024 2022-09-10T04:29:53.139 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 1024 2022-09-10T04:29:53.139 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192768 len 2048 2022-09-10T04:29:53.139 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 2048 2022-09-10T04:29:53.140 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192768 len 2048 2022-09-10T04:29:53.140 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 2048 2022-09-10T04:29:53.140 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192768 len 4096 2022-09-10T04:29:53.140 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 4096 2022-09-10T04:29:53.140 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192768 len 4096 2022-09-10T04:29:53.141 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 4096 2022-09-10T04:29:53.141 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192768 len 8192 2022-09-10T04:29:53.141 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 8192 2022-09-10T04:29:53.141 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192768 len 8192 2022-09-10T04:29:53.142 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 8192 2022-09-10T04:29:53.142 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4192768 len 16384 2022-09-10T04:29:53.142 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 16384 2022-09-10T04:29:53.142 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4192768 len 16384 2022-09-10T04:29:53.142 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4192768 len 16384 2022-09-10T04:29:53.143 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193280 len 1024 2022-09-10T04:29:53.143 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 1024 2022-09-10T04:29:53.143 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193280 len 1024 2022-09-10T04:29:53.143 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 1024 2022-09-10T04:29:53.144 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193280 len 2048 2022-09-10T04:29:53.144 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 2048 2022-09-10T04:29:53.144 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193280 len 2048 2022-09-10T04:29:53.144 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 2048 2022-09-10T04:29:53.144 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193280 len 4096 2022-09-10T04:29:53.145 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 4096 2022-09-10T04:29:53.145 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193280 len 4096 2022-09-10T04:29:53.145 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 4096 2022-09-10T04:29:53.145 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193280 len 8192 2022-09-10T04:29:53.146 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 8192 2022-09-10T04:29:53.146 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193280 len 8192 2022-09-10T04:29:53.146 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 8192 2022-09-10T04:29:53.147 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193280 len 16384 2022-09-10T04:29:53.147 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 16384 2022-09-10T04:29:53.147 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193280 len 16384 2022-09-10T04:29:53.147 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193280 len 16384 2022-09-10T04:29:53.148 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193792 len 1024 2022-09-10T04:29:53.149 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 1024 2022-09-10T04:29:53.149 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193792 len 1024 2022-09-10T04:29:53.149 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 1024 2022-09-10T04:29:53.150 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193792 len 2048 2022-09-10T04:29:53.150 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 2048 2022-09-10T04:29:53.150 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193792 len 2048 2022-09-10T04:29:53.150 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 2048 2022-09-10T04:29:53.151 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193792 len 4096 2022-09-10T04:29:53.151 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 4096 2022-09-10T04:29:53.151 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193792 len 4096 2022-09-10T04:29:53.152 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 4096 2022-09-10T04:29:53.152 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193792 len 8192 2022-09-10T04:29:53.152 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 8192 2022-09-10T04:29:53.152 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193792 len 8192 2022-09-10T04:29:53.153 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 8192 2022-09-10T04:29:53.153 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4193792 len 16384 2022-09-10T04:29:53.153 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 16384 2022-09-10T04:29:53.154 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4193792 len 16384 2022-09-10T04:29:53.154 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4193792 len 16384 2022-09-10T04:29:53.155 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194304 len 1024 2022-09-10T04:29:53.155 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 1024 2022-09-10T04:29:53.155 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194304 len 1024 2022-09-10T04:29:53.156 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 1024 2022-09-10T04:29:53.156 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194304 len 2048 2022-09-10T04:29:53.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 2048 2022-09-10T04:29:53.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194304 len 2048 2022-09-10T04:29:53.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 2048 2022-09-10T04:29:53.167 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194304 len 4096 2022-09-10T04:29:53.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 4096 2022-09-10T04:29:53.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194304 len 4096 2022-09-10T04:29:53.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 4096 2022-09-10T04:29:53.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194304 len 8192 2022-09-10T04:29:53.168 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 8192 2022-09-10T04:29:53.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194304 len 8192 2022-09-10T04:29:53.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 8192 2022-09-10T04:29:53.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194304 len 16384 2022-09-10T04:29:53.169 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 16384 2022-09-10T04:29:53.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194304 len 16384 2022-09-10T04:29:53.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194304 len 16384 2022-09-10T04:29:53.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194816 len 1024 2022-09-10T04:29:53.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 1024 2022-09-10T04:29:53.170 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194816 len 1024 2022-09-10T04:29:53.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 1024 2022-09-10T04:29:53.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194816 len 2048 2022-09-10T04:29:53.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 2048 2022-09-10T04:29:53.171 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194816 len 2048 2022-09-10T04:29:53.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 2048 2022-09-10T04:29:53.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194816 len 4096 2022-09-10T04:29:53.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 4096 2022-09-10T04:29:53.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194816 len 4096 2022-09-10T04:29:53.172 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 4096 2022-09-10T04:29:53.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194816 len 8192 2022-09-10T04:29:53.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 8192 2022-09-10T04:29:53.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194816 len 8192 2022-09-10T04:29:53.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 8192 2022-09-10T04:29:53.173 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4194816 len 16384 2022-09-10T04:29:53.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 16384 2022-09-10T04:29:53.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4194816 len 16384 2022-09-10T04:29:53.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4194816 len 16384 2022-09-10T04:29:53.174 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195328 len 1024 2022-09-10T04:29:53.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 1024 2022-09-10T04:29:53.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195328 len 1024 2022-09-10T04:29:53.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 1024 2022-09-10T04:29:53.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195328 len 2048 2022-09-10T04:29:53.175 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 2048 2022-09-10T04:29:53.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195328 len 2048 2022-09-10T04:29:53.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 2048 2022-09-10T04:29:53.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195328 len 4096 2022-09-10T04:29:53.177 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 4096 2022-09-10T04:29:53.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195328 len 4096 2022-09-10T04:29:53.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 4096 2022-09-10T04:29:53.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195328 len 8192 2022-09-10T04:29:53.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 8192 2022-09-10T04:29:53.178 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195328 len 8192 2022-09-10T04:29:53.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 8192 2022-09-10T04:29:53.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195328 len 16384 2022-09-10T04:29:53.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 16384 2022-09-10T04:29:53.179 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195328 len 16384 2022-09-10T04:29:53.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195328 len 16384 2022-09-10T04:29:53.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195840 len 1024 2022-09-10T04:29:53.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 1024 2022-09-10T04:29:53.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195840 len 1024 2022-09-10T04:29:53.180 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 1024 2022-09-10T04:29:53.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195840 len 2048 2022-09-10T04:29:53.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 2048 2022-09-10T04:29:53.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195840 len 2048 2022-09-10T04:29:53.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 2048 2022-09-10T04:29:53.181 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195840 len 4096 2022-09-10T04:29:53.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 4096 2022-09-10T04:29:53.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195840 len 4096 2022-09-10T04:29:53.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 4096 2022-09-10T04:29:53.182 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195840 len 8192 2022-09-10T04:29:53.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 8192 2022-09-10T04:29:53.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195840 len 8192 2022-09-10T04:29:53.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 8192 2022-09-10T04:29:53.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4195840 len 16384 2022-09-10T04:29:53.183 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 16384 2022-09-10T04:29:53.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4195840 len 16384 2022-09-10T04:29:53.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4195840 len 16384 2022-09-10T04:29:53.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196352 len 1024 2022-09-10T04:29:53.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 1024 2022-09-10T04:29:53.184 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196352 len 1024 2022-09-10T04:29:53.185 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 1024 2022-09-10T04:29:53.185 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196352 len 2048 2022-09-10T04:29:53.185 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 2048 2022-09-10T04:29:53.185 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196352 len 2048 2022-09-10T04:29:53.186 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 2048 2022-09-10T04:29:53.200 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196352 len 4096 2022-09-10T04:29:53.200 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 4096 2022-09-10T04:29:53.200 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196352 len 4096 2022-09-10T04:29:53.201 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 4096 2022-09-10T04:29:53.201 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196352 len 8192 2022-09-10T04:29:53.201 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 8192 2022-09-10T04:29:53.201 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196352 len 8192 2022-09-10T04:29:53.202 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 8192 2022-09-10T04:29:53.202 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196352 len 16384 2022-09-10T04:29:53.202 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 16384 2022-09-10T04:29:53.202 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196352 len 16384 2022-09-10T04:29:53.202 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196352 len 16384 2022-09-10T04:29:53.203 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196864 len 1024 2022-09-10T04:29:53.203 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 1024 2022-09-10T04:29:53.203 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196864 len 1024 2022-09-10T04:29:53.203 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 1024 2022-09-10T04:29:53.204 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196864 len 2048 2022-09-10T04:29:53.204 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 2048 2022-09-10T04:29:53.204 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196864 len 2048 2022-09-10T04:29:53.204 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 2048 2022-09-10T04:29:53.204 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196864 len 4096 2022-09-10T04:29:53.205 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 4096 2022-09-10T04:29:53.205 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196864 len 4096 2022-09-10T04:29:53.205 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 4096 2022-09-10T04:29:53.205 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196864 len 8192 2022-09-10T04:29:53.206 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 8192 2022-09-10T04:29:53.206 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196864 len 8192 2022-09-10T04:29:53.206 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 8192 2022-09-10T04:29:53.206 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4196864 len 16384 2022-09-10T04:29:53.206 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 16384 2022-09-10T04:29:53.207 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4196864 len 16384 2022-09-10T04:29:53.207 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4196864 len 16384 2022-09-10T04:29:53.209 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197376 len 1024 2022-09-10T04:29:53.209 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 1024 2022-09-10T04:29:53.210 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197376 len 1024 2022-09-10T04:29:53.210 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 1024 2022-09-10T04:29:53.210 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197376 len 2048 2022-09-10T04:29:53.210 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 2048 2022-09-10T04:29:53.211 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197376 len 2048 2022-09-10T04:29:53.211 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 2048 2022-09-10T04:29:53.211 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197376 len 4096 2022-09-10T04:29:53.211 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 4096 2022-09-10T04:29:53.211 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197376 len 4096 2022-09-10T04:29:53.212 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 4096 2022-09-10T04:29:53.212 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197376 len 8192 2022-09-10T04:29:53.212 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 8192 2022-09-10T04:29:53.212 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197376 len 8192 2022-09-10T04:29:53.213 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 8192 2022-09-10T04:29:53.213 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197376 len 16384 2022-09-10T04:29:53.213 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 16384 2022-09-10T04:29:53.213 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197376 len 16384 2022-09-10T04:29:53.213 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197376 len 16384 2022-09-10T04:29:53.214 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197888 len 1024 2022-09-10T04:29:53.214 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 1024 2022-09-10T04:29:53.214 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197888 len 1024 2022-09-10T04:29:53.215 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 1024 2022-09-10T04:29:53.215 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197888 len 2048 2022-09-10T04:29:53.215 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 2048 2022-09-10T04:29:53.215 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197888 len 2048 2022-09-10T04:29:53.215 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 2048 2022-09-10T04:29:53.216 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197888 len 4096 2022-09-10T04:29:53.216 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 4096 2022-09-10T04:29:53.216 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197888 len 4096 2022-09-10T04:29:53.216 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 4096 2022-09-10T04:29:53.217 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197888 len 8192 2022-09-10T04:29:53.217 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 8192 2022-09-10T04:29:53.217 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197888 len 8192 2022-09-10T04:29:53.217 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 8192 2022-09-10T04:29:53.217 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3072 offset 4197888 len 16384 2022-09-10T04:29:53.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 16384 2022-09-10T04:29:53.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3072 offset 4197888 len 16384 2022-09-10T04:29:53.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3072 offset 4197888 len 16384 2022-09-10T04:29:53.218 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190208 len 1024 2022-09-10T04:29:53.219 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 1024 2022-09-10T04:29:53.219 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190208 len 1024 2022-09-10T04:29:53.220 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 1024 2022-09-10T04:29:53.220 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190208 len 2048 2022-09-10T04:29:53.220 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 2048 2022-09-10T04:29:53.221 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190208 len 2048 2022-09-10T04:29:53.221 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 2048 2022-09-10T04:29:53.222 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190208 len 4096 2022-09-10T04:29:53.222 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 4096 2022-09-10T04:29:53.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190208 len 4096 2022-09-10T04:29:53.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 4096 2022-09-10T04:29:53.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190208 len 8192 2022-09-10T04:29:53.254 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 8192 2022-09-10T04:29:53.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190208 len 8192 2022-09-10T04:29:53.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 8192 2022-09-10T04:29:53.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190208 len 16384 2022-09-10T04:29:53.255 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 16384 2022-09-10T04:29:53.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190208 len 16384 2022-09-10T04:29:53.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190208 len 16384 2022-09-10T04:29:53.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190720 len 1024 2022-09-10T04:29:53.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 1024 2022-09-10T04:29:53.256 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190720 len 1024 2022-09-10T04:29:53.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 1024 2022-09-10T04:29:53.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190720 len 2048 2022-09-10T04:29:53.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 2048 2022-09-10T04:29:53.257 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190720 len 2048 2022-09-10T04:29:53.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 2048 2022-09-10T04:29:53.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190720 len 4096 2022-09-10T04:29:53.258 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 4096 2022-09-10T04:29:53.259 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190720 len 4096 2022-09-10T04:29:53.259 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 4096 2022-09-10T04:29:53.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190720 len 8192 2022-09-10T04:29:53.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 8192 2022-09-10T04:29:53.260 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190720 len 8192 2022-09-10T04:29:53.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 8192 2022-09-10T04:29:53.261 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4190720 len 16384 2022-09-10T04:29:53.262 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 16384 2022-09-10T04:29:53.262 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4190720 len 16384 2022-09-10T04:29:53.263 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4190720 len 16384 2022-09-10T04:29:53.263 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191232 len 1024 2022-09-10T04:29:53.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 1024 2022-09-10T04:29:53.264 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191232 len 1024 2022-09-10T04:29:53.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 1024 2022-09-10T04:29:53.265 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191232 len 2048 2022-09-10T04:29:53.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 2048 2022-09-10T04:29:53.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191232 len 2048 2022-09-10T04:29:53.266 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 2048 2022-09-10T04:29:53.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191232 len 4096 2022-09-10T04:29:53.267 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 4096 2022-09-10T04:29:53.268 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191232 len 4096 2022-09-10T04:29:53.268 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 4096 2022-09-10T04:29:53.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191232 len 8192 2022-09-10T04:29:53.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 8192 2022-09-10T04:29:53.269 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191232 len 8192 2022-09-10T04:29:53.270 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 8192 2022-09-10T04:29:53.270 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191232 len 16384 2022-09-10T04:29:53.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 16384 2022-09-10T04:29:53.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191232 len 16384 2022-09-10T04:29:53.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191232 len 16384 2022-09-10T04:29:53.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191744 len 1024 2022-09-10T04:29:53.271 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 1024 2022-09-10T04:29:53.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191744 len 1024 2022-09-10T04:29:53.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 1024 2022-09-10T04:29:53.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191744 len 2048 2022-09-10T04:29:53.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 2048 2022-09-10T04:29:53.272 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191744 len 2048 2022-09-10T04:29:53.273 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 2048 2022-09-10T04:29:53.273 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191744 len 4096 2022-09-10T04:29:53.273 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 4096 2022-09-10T04:29:53.273 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191744 len 4096 2022-09-10T04:29:53.274 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 4096 2022-09-10T04:29:53.274 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191744 len 8192 2022-09-10T04:29:53.274 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 8192 2022-09-10T04:29:53.274 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191744 len 8192 2022-09-10T04:29:53.274 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 8192 2022-09-10T04:29:53.275 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4191744 len 16384 2022-09-10T04:29:53.275 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 16384 2022-09-10T04:29:53.275 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4191744 len 16384 2022-09-10T04:29:53.275 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4191744 len 16384 2022-09-10T04:29:53.275 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192256 len 1024 2022-09-10T04:29:53.276 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 1024 2022-09-10T04:29:53.276 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192256 len 1024 2022-09-10T04:29:53.276 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 1024 2022-09-10T04:29:53.276 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192256 len 2048 2022-09-10T04:29:53.276 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 2048 2022-09-10T04:29:53.277 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192256 len 2048 2022-09-10T04:29:53.277 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 2048 2022-09-10T04:29:53.277 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192256 len 4096 2022-09-10T04:29:53.277 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 4096 2022-09-10T04:29:53.277 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192256 len 4096 2022-09-10T04:29:53.278 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 4096 2022-09-10T04:29:53.278 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192256 len 8192 2022-09-10T04:29:53.317 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 8192 2022-09-10T04:29:53.317 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192256 len 8192 2022-09-10T04:29:53.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 8192 2022-09-10T04:29:53.318 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192256 len 16384 2022-09-10T04:29:53.319 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 16384 2022-09-10T04:29:53.319 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192256 len 16384 2022-09-10T04:29:53.320 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192256 len 16384 2022-09-10T04:29:53.320 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192768 len 1024 2022-09-10T04:29:53.321 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 1024 2022-09-10T04:29:53.321 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192768 len 1024 2022-09-10T04:29:53.321 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 1024 2022-09-10T04:29:53.322 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192768 len 2048 2022-09-10T04:29:53.322 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 2048 2022-09-10T04:29:53.323 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192768 len 2048 2022-09-10T04:29:53.323 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 2048 2022-09-10T04:29:53.324 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192768 len 4096 2022-09-10T04:29:53.324 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 4096 2022-09-10T04:29:53.325 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192768 len 4096 2022-09-10T04:29:53.325 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 4096 2022-09-10T04:29:53.325 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192768 len 8192 2022-09-10T04:29:53.326 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 8192 2022-09-10T04:29:53.326 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192768 len 8192 2022-09-10T04:29:53.327 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 8192 2022-09-10T04:29:53.327 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4192768 len 16384 2022-09-10T04:29:53.327 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 16384 2022-09-10T04:29:53.327 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4192768 len 16384 2022-09-10T04:29:53.328 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4192768 len 16384 2022-09-10T04:29:53.328 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193280 len 1024 2022-09-10T04:29:53.328 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 1024 2022-09-10T04:29:53.328 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193280 len 1024 2022-09-10T04:29:53.328 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 1024 2022-09-10T04:29:53.329 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193280 len 2048 2022-09-10T04:29:53.329 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 2048 2022-09-10T04:29:53.329 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193280 len 2048 2022-09-10T04:29:53.329 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 2048 2022-09-10T04:29:53.330 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193280 len 4096 2022-09-10T04:29:53.330 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 4096 2022-09-10T04:29:53.330 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193280 len 4096 2022-09-10T04:29:53.330 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 4096 2022-09-10T04:29:53.331 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193280 len 8192 2022-09-10T04:29:53.331 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 8192 2022-09-10T04:29:53.331 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193280 len 8192 2022-09-10T04:29:53.332 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 8192 2022-09-10T04:29:53.332 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193280 len 16384 2022-09-10T04:29:53.332 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 16384 2022-09-10T04:29:53.332 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193280 len 16384 2022-09-10T04:29:53.332 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193280 len 16384 2022-09-10T04:29:53.333 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193792 len 1024 2022-09-10T04:29:53.333 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 1024 2022-09-10T04:29:53.333 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193792 len 1024 2022-09-10T04:29:53.333 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 1024 2022-09-10T04:29:53.334 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193792 len 2048 2022-09-10T04:29:53.334 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 2048 2022-09-10T04:29:53.334 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193792 len 2048 2022-09-10T04:29:53.335 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 2048 2022-09-10T04:29:53.335 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193792 len 4096 2022-09-10T04:29:53.335 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 4096 2022-09-10T04:29:53.336 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193792 len 4096 2022-09-10T04:29:53.336 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 4096 2022-09-10T04:29:53.336 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193792 len 8192 2022-09-10T04:29:53.336 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 8192 2022-09-10T04:29:53.336 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193792 len 8192 2022-09-10T04:29:53.337 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 8192 2022-09-10T04:29:53.337 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4193792 len 16384 2022-09-10T04:29:53.337 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 16384 2022-09-10T04:29:53.337 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4193792 len 16384 2022-09-10T04:29:53.338 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4193792 len 16384 2022-09-10T04:29:53.338 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194304 len 1024 2022-09-10T04:29:53.338 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 1024 2022-09-10T04:29:53.339 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194304 len 1024 2022-09-10T04:29:53.339 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 1024 2022-09-10T04:29:53.339 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194304 len 2048 2022-09-10T04:29:53.339 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 2048 2022-09-10T04:29:53.339 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194304 len 2048 2022-09-10T04:29:53.340 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 2048 2022-09-10T04:29:53.340 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194304 len 4096 2022-09-10T04:29:53.340 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 4096 2022-09-10T04:29:53.340 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194304 len 4096 2022-09-10T04:29:53.341 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 4096 2022-09-10T04:29:53.341 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194304 len 8192 2022-09-10T04:29:53.341 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 8192 2022-09-10T04:29:53.341 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194304 len 8192 2022-09-10T04:29:53.350 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 8192 2022-09-10T04:29:53.351 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194304 len 16384 2022-09-10T04:29:53.351 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 16384 2022-09-10T04:29:53.351 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194304 len 16384 2022-09-10T04:29:53.352 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194304 len 16384 2022-09-10T04:29:53.352 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194816 len 1024 2022-09-10T04:29:53.352 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 1024 2022-09-10T04:29:53.353 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194816 len 1024 2022-09-10T04:29:53.353 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 1024 2022-09-10T04:29:53.354 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194816 len 2048 2022-09-10T04:29:53.354 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 2048 2022-09-10T04:29:53.354 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194816 len 2048 2022-09-10T04:29:53.355 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 2048 2022-09-10T04:29:53.355 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194816 len 4096 2022-09-10T04:29:53.355 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 4096 2022-09-10T04:29:53.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194816 len 4096 2022-09-10T04:29:53.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 4096 2022-09-10T04:29:53.356 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194816 len 8192 2022-09-10T04:29:53.357 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 8192 2022-09-10T04:29:53.357 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194816 len 8192 2022-09-10T04:29:53.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 8192 2022-09-10T04:29:53.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4194816 len 16384 2022-09-10T04:29:53.358 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 16384 2022-09-10T04:29:53.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4194816 len 16384 2022-09-10T04:29:53.359 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4194816 len 16384 2022-09-10T04:29:53.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195328 len 1024 2022-09-10T04:29:53.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 1024 2022-09-10T04:29:53.360 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195328 len 1024 2022-09-10T04:29:53.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 1024 2022-09-10T04:29:53.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195328 len 2048 2022-09-10T04:29:53.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 2048 2022-09-10T04:29:53.361 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195328 len 2048 2022-09-10T04:29:53.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 2048 2022-09-10T04:29:53.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195328 len 4096 2022-09-10T04:29:53.362 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 4096 2022-09-10T04:29:53.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195328 len 4096 2022-09-10T04:29:53.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 4096 2022-09-10T04:29:53.363 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195328 len 8192 2022-09-10T04:29:53.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 8192 2022-09-10T04:29:53.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195328 len 8192 2022-09-10T04:29:53.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 8192 2022-09-10T04:29:53.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195328 len 16384 2022-09-10T04:29:53.364 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 16384 2022-09-10T04:29:53.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195328 len 16384 2022-09-10T04:29:53.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195328 len 16384 2022-09-10T04:29:53.365 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195840 len 1024 2022-09-10T04:29:53.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 1024 2022-09-10T04:29:53.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195840 len 1024 2022-09-10T04:29:53.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 1024 2022-09-10T04:29:53.366 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195840 len 2048 2022-09-10T04:29:53.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 2048 2022-09-10T04:29:53.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195840 len 2048 2022-09-10T04:29:53.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 2048 2022-09-10T04:29:53.367 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195840 len 4096 2022-09-10T04:29:53.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 4096 2022-09-10T04:29:53.368 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195840 len 4096 2022-09-10T04:29:53.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 4096 2022-09-10T04:29:53.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195840 len 8192 2022-09-10T04:29:53.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 8192 2022-09-10T04:29:53.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195840 len 8192 2022-09-10T04:29:53.369 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 8192 2022-09-10T04:29:53.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4195840 len 16384 2022-09-10T04:29:53.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 16384 2022-09-10T04:29:53.370 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4195840 len 16384 2022-09-10T04:29:53.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4195840 len 16384 2022-09-10T04:29:53.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196352 len 1024 2022-09-10T04:29:53.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 1024 2022-09-10T04:29:53.371 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196352 len 1024 2022-09-10T04:29:53.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 1024 2022-09-10T04:29:53.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196352 len 2048 2022-09-10T04:29:53.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 2048 2022-09-10T04:29:53.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196352 len 2048 2022-09-10T04:29:53.372 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 2048 2022-09-10T04:29:53.373 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196352 len 4096 2022-09-10T04:29:53.373 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 4096 2022-09-10T04:29:53.373 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196352 len 4096 2022-09-10T04:29:53.374 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 4096 2022-09-10T04:29:53.374 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196352 len 8192 2022-09-10T04:29:53.374 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 8192 2022-09-10T04:29:53.374 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196352 len 8192 2022-09-10T04:29:53.374 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 8192 2022-09-10T04:29:53.375 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196352 len 16384 2022-09-10T04:29:53.862 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 16384 2022-09-10T04:29:53.862 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196352 len 16384 2022-09-10T04:29:53.863 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196352 len 16384 2022-09-10T04:29:53.864 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196864 len 1024 2022-09-10T04:29:53.864 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 1024 2022-09-10T04:29:53.864 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196864 len 1024 2022-09-10T04:29:53.865 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 1024 2022-09-10T04:29:53.865 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196864 len 2048 2022-09-10T04:29:53.866 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 2048 2022-09-10T04:29:53.866 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196864 len 2048 2022-09-10T04:29:53.866 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 2048 2022-09-10T04:29:53.867 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196864 len 4096 2022-09-10T04:29:53.868 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 4096 2022-09-10T04:29:53.868 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196864 len 4096 2022-09-10T04:29:53.868 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 4096 2022-09-10T04:29:53.869 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196864 len 8192 2022-09-10T04:29:53.869 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 8192 2022-09-10T04:29:53.869 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196864 len 8192 2022-09-10T04:29:53.870 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 8192 2022-09-10T04:29:53.870 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4196864 len 16384 2022-09-10T04:29:53.870 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 16384 2022-09-10T04:29:53.871 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4196864 len 16384 2022-09-10T04:29:53.871 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4196864 len 16384 2022-09-10T04:29:53.871 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197376 len 1024 2022-09-10T04:29:53.871 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 1024 2022-09-10T04:29:53.871 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197376 len 1024 2022-09-10T04:29:53.872 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 1024 2022-09-10T04:29:53.872 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197376 len 2048 2022-09-10T04:29:53.872 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 2048 2022-09-10T04:29:53.872 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197376 len 2048 2022-09-10T04:29:53.872 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 2048 2022-09-10T04:29:53.873 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197376 len 4096 2022-09-10T04:29:53.873 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 4096 2022-09-10T04:29:53.873 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197376 len 4096 2022-09-10T04:29:53.873 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 4096 2022-09-10T04:29:53.874 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197376 len 8192 2022-09-10T04:29:53.874 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 8192 2022-09-10T04:29:53.874 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197376 len 8192 2022-09-10T04:29:53.874 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 8192 2022-09-10T04:29:53.874 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197376 len 16384 2022-09-10T04:29:53.875 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 16384 2022-09-10T04:29:53.875 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197376 len 16384 2022-09-10T04:29:53.876 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197376 len 16384 2022-09-10T04:29:53.876 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197888 len 1024 2022-09-10T04:29:53.876 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 1024 2022-09-10T04:29:53.876 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197888 len 1024 2022-09-10T04:29:53.876 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 1024 2022-09-10T04:29:53.877 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197888 len 2048 2022-09-10T04:29:53.877 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 2048 2022-09-10T04:29:53.877 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197888 len 2048 2022-09-10T04:29:53.877 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 2048 2022-09-10T04:29:53.877 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197888 len 4096 2022-09-10T04:29:53.878 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 4096 2022-09-10T04:29:53.878 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197888 len 4096 2022-09-10T04:29:53.878 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 4096 2022-09-10T04:29:53.878 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197888 len 8192 2022-09-10T04:29:53.879 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 8192 2022-09-10T04:29:53.879 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197888 len 8192 2022-09-10T04:29:53.879 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 8192 2022-09-10T04:29:53.879 INFO:tasks.workunit.client.0.smithi081.stdout:read_direct buf_align 3584 offset 4197888 len 16384 2022-09-10T04:29:53.879 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 16384 2022-09-10T04:29:53.880 INFO:tasks.workunit.client.0.smithi081.stdout:read_sync buf_align 3584 offset 4197888 len 16384 2022-09-10T04:29:53.880 INFO:tasks.workunit.client.0.smithi081.stdout:read_file buf_align 3584 offset 4197888 len 16384 2022-09-10T04:29:53.880 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:53.880 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:53.880 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:53.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 1024 2022-09-10T04:29:53.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:53.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:53.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:53.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 2048 2022-09-10T04:29:53.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:53.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:53.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:53.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 4096 2022-09-10T04:29:53.883 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:53.883 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:53.883 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:53.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 8192 2022-09-10T04:29:53.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:53.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:53.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:53.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4190720 len 16384 2022-09-10T04:29:53.885 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:54 smithi104 ceph-mon[97044]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.4 KiB/s wr, 302 op/s 2022-09-10T04:29:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:29:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:29:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:29:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:54 smithi081 ceph-mon[90133]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.4 KiB/s wr, 302 op/s 2022-09-10T04:29:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:54 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:29:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:54 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:29:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:54 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:29:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:54 smithi110 ceph-mon[97633]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 87 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 9.4 KiB/s wr, 302 op/s 2022-09-10T04:29:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:29:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:29:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:29:55.782 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:55.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:55.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 1024 2022-09-10T04:29:55.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:55.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:55.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:55.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 2048 2022-09-10T04:29:55.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:55.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:55.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:55.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 4096 2022-09-10T04:29:55.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:55.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:55.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:55.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 8192 2022-09-10T04:29:55.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:55.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:55.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:55.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191232 len 16384 2022-09-10T04:29:55.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:55.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:55.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:55.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 1024 2022-09-10T04:29:55.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:55.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:55.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:55.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 2048 2022-09-10T04:29:55.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:55.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:55.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:55.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 4096 2022-09-10T04:29:55.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:55.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:55.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:55.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 8192 2022-09-10T04:29:55.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:55.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:55.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:55.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4191744 len 16384 2022-09-10T04:29:55.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:55.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:55.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:55.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 1024 2022-09-10T04:29:55.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:55.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:55.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:55.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 2048 2022-09-10T04:29:55.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:55.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:55.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:55.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 4096 2022-09-10T04:29:55.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:55.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:55.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:55.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 8192 2022-09-10T04:29:55.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:55.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:55.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:55.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192256 len 16384 2022-09-10T04:29:55.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:55.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:55.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:55.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 1024 2022-09-10T04:29:55.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:55.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:55.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:55.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 2048 2022-09-10T04:29:55.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:55.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:55.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:55.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 4096 2022-09-10T04:29:55.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:55.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:55.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:55.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 8192 2022-09-10T04:29:55.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:55.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:55.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:55.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4192768 len 16384 2022-09-10T04:29:55.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:55.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:55.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:55.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 1024 2022-09-10T04:29:55.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:55.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:55.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:55 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:55 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:55.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:55 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:55.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:55 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:55.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:55 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:55.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:55 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:55 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:55 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:55 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:55 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:55 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:55 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:55 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:55 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:55 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:55 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:55 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:55 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:56 smithi104 ceph-mon[97044]: pgmap v39: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 717 KiB/s wr, 460 op/s 2022-09-10T04:29:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:56 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:56 smithi110 ceph-mon[97633]: pgmap v39: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 717 KiB/s wr, 460 op/s 2022-09-10T04:29:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:56 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:57.023 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 2048 2022-09-10T04:29:57.024 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:57.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:57.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:57.026 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 4096 2022-09-10T04:29:57.026 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:57.027 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:57.027 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:57.028 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 8192 2022-09-10T04:29:57.028 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:57.029 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:57.029 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:57.030 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193280 len 16384 2022-09-10T04:29:57.030 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:57.031 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:57.031 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:57.032 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 1024 2022-09-10T04:29:57.032 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:57.033 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:57.033 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:57.034 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 2048 2022-09-10T04:29:57.034 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:57.034 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:57.036 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:57.037 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 4096 2022-09-10T04:29:57.037 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:57.037 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:57.038 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:57.038 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 8192 2022-09-10T04:29:57.039 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:57.039 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:57.040 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:57.040 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4193792 len 16384 2022-09-10T04:29:57.041 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:57.041 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:57.041 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:57.042 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 1024 2022-09-10T04:29:57.042 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:57.042 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:57.043 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:57.043 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 2048 2022-09-10T04:29:57.044 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:57.044 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:57.045 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:57.045 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 4096 2022-09-10T04:29:57.045 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:57.046 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:57.046 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:57.046 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 8192 2022-09-10T04:29:57.047 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:57.047 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:57.047 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:57.047 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194304 len 16384 2022-09-10T04:29:57.048 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:57.048 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:57.048 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:57.049 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 1024 2022-09-10T04:29:57.049 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:57.049 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:57.050 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:57.050 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 2048 2022-09-10T04:29:57.051 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:57.051 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:57.051 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:57.052 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 4096 2022-09-10T04:29:57.052 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:57.053 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:57.053 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:57.054 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 8192 2022-09-10T04:29:57.054 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:57.055 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:57.055 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:57.056 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4194816 len 16384 2022-09-10T04:29:57.056 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:57.057 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:57.057 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:57.057 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 1024 2022-09-10T04:29:57.058 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:57.058 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:57.059 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:57.059 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 2048 2022-09-10T04:29:57.060 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:57.060 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:57.061 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:57.061 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 4096 2022-09-10T04:29:57.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:56 smithi081 ceph-mon[90133]: pgmap v39: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 717 KiB/s wr, 460 op/s 2022-09-10T04:29:57.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:56 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:29:57.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:57.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:57.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:57.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 8192 2022-09-10T04:29:57.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:57.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:57.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:57.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195328 len 16384 2022-09-10T04:29:57.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:57.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:57.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:57.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 1024 2022-09-10T04:29:57.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:57.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:57.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:57.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 2048 2022-09-10T04:29:57.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:57.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:57.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:57.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 4096 2022-09-10T04:29:57.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:57.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:57.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:57.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 8192 2022-09-10T04:29:57.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:57.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:57.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:57.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4195840 len 16384 2022-09-10T04:29:57.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:57.639 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:57.639 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:57.639 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 1024 2022-09-10T04:29:57.639 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:57.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:57.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:57.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 2048 2022-09-10T04:29:57.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:57.641 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:57.641 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:57.641 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 4096 2022-09-10T04:29:57.641 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:57.641 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:57.642 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:57.642 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 8192 2022-09-10T04:29:57.642 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:57.642 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:57.643 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:57.643 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196352 len 16384 2022-09-10T04:29:57.643 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:57.643 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:57.643 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:57.644 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 1024 2022-09-10T04:29:57.644 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:57.644 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:57.644 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:57.645 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 2048 2022-09-10T04:29:57.645 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:57.645 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:57.645 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:57.646 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 4096 2022-09-10T04:29:57.646 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:57.646 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:57.646 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:57.647 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 8192 2022-09-10T04:29:57.647 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:57.647 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:57.648 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:57.648 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4196864 len 16384 2022-09-10T04:29:57.648 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:57.648 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:57.648 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:57.649 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 1024 2022-09-10T04:29:57.649 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:57.649 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:57.649 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:57.650 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 2048 2022-09-10T04:29:57.650 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:57.650 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:57.650 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:57.650 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 4096 2022-09-10T04:29:57.651 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:57.651 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:57.651 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:57.651 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 8192 2022-09-10T04:29:57.652 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:57.652 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:29:58 smithi110 ceph-mon[97633]: pgmap v40: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 712 KiB/s wr, 163 op/s 2022-09-10T04:29:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:29:58 smithi104 ceph-mon[97044]: pgmap v40: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 712 KiB/s wr, 163 op/s 2022-09-10T04:29:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:29:58 smithi081 ceph-mon[90133]: pgmap v40: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 712 KiB/s wr, 163 op/s 2022-09-10T04:29:59.110 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:59.111 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197376 len 16384 2022-09-10T04:29:59.111 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:59.112 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:59.112 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:59.113 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 1024 2022-09-10T04:29:59.113 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:59.113 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:59.114 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:59.114 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 2048 2022-09-10T04:29:59.115 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:59.115 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:59.116 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:59.116 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 4096 2022-09-10T04:29:59.116 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:59.117 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:59.117 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:59.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 8192 2022-09-10T04:29:59.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:59.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:59.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:59.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 0 offset 4197888 len 16384 2022-09-10T04:29:59.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:59.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:59.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:59.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 1024 2022-09-10T04:29:59.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:59.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:59.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:59.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 2048 2022-09-10T04:29:59.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:59.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:59.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:59.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 4096 2022-09-10T04:29:59.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:59.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:59.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:59.123 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 8192 2022-09-10T04:29:59.123 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:59.124 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:59.124 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:59.124 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4190720 len 16384 2022-09-10T04:29:59.125 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:59.125 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:59.126 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:59.126 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 1024 2022-09-10T04:29:59.127 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:59.127 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:59.128 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:59.128 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 2048 2022-09-10T04:29:59.129 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:59.129 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:59.129 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:59.130 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 4096 2022-09-10T04:29:59.130 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:59.130 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:59.131 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:59.131 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 8192 2022-09-10T04:29:59.131 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:59.131 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:59.132 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:59.132 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191232 len 16384 2022-09-10T04:29:59.132 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:59.132 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:59.133 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:59.133 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 1024 2022-09-10T04:29:59.133 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:59.133 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:59.133 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:59.134 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 2048 2022-09-10T04:29:59.134 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:59.134 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:59.134 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:59.135 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 4096 2022-09-10T04:29:59.135 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:59.135 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:59.135 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:59.136 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 8192 2022-09-10T04:29:59.136 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:59.136 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:59.136 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:59.137 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4191744 len 16384 2022-09-10T04:29:59.137 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192256 len 1024 2022-09-10T04:29:59.137 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 1024 2022-09-10T04:30:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:00 smithi110 ceph-mon[97633]: pgmap v41: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 712 KiB/s wr, 163 op/s 2022-09-10T04:30:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:00 smithi110 ceph-mon[97633]: overall HEALTH_OK 2022-09-10T04:30:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:00 smithi104 ceph-mon[97044]: pgmap v41: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 712 KiB/s wr, 163 op/s 2022-09-10T04:30:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:00 smithi104 ceph-mon[97044]: overall HEALTH_OK 2022-09-10T04:30:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:00 smithi081 ceph-mon[90133]: pgmap v41: 129 pgs: 129 active+clean; 86 MiB data, 161 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 712 KiB/s wr, 163 op/s 2022-09-10T04:30:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:00 smithi081 ceph-mon[90133]: overall HEALTH_OK 2022-09-10T04:30:01.294 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192256 len 1024 2022-09-10T04:30:01.295 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 1024 2022-09-10T04:30:01.295 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192256 len 2048 2022-09-10T04:30:01.295 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 2048 2022-09-10T04:30:01.295 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192256 len 2048 2022-09-10T04:30:01.296 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 2048 2022-09-10T04:30:01.296 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192256 len 4096 2022-09-10T04:30:01.296 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 4096 2022-09-10T04:30:01.297 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192256 len 4096 2022-09-10T04:30:01.297 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 4096 2022-09-10T04:30:01.297 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192256 len 8192 2022-09-10T04:30:01.297 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 8192 2022-09-10T04:30:01.297 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192256 len 8192 2022-09-10T04:30:01.298 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 8192 2022-09-10T04:30:01.298 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192256 len 16384 2022-09-10T04:30:01.298 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 16384 2022-09-10T04:30:01.299 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192256 len 16384 2022-09-10T04:30:01.299 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192256 len 16384 2022-09-10T04:30:01.299 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192768 len 1024 2022-09-10T04:30:01.300 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 1024 2022-09-10T04:30:01.300 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192768 len 1024 2022-09-10T04:30:01.300 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 1024 2022-09-10T04:30:01.300 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192768 len 2048 2022-09-10T04:30:01.301 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 2048 2022-09-10T04:30:01.301 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192768 len 2048 2022-09-10T04:30:01.301 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 2048 2022-09-10T04:30:01.302 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192768 len 4096 2022-09-10T04:30:01.302 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 4096 2022-09-10T04:30:01.302 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192768 len 4096 2022-09-10T04:30:01.303 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 4096 2022-09-10T04:30:01.304 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192768 len 8192 2022-09-10T04:30:01.304 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 8192 2022-09-10T04:30:01.304 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192768 len 8192 2022-09-10T04:30:01.305 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 8192 2022-09-10T04:30:01.305 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4192768 len 16384 2022-09-10T04:30:01.306 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 16384 2022-09-10T04:30:01.306 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4192768 len 16384 2022-09-10T04:30:01.307 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4192768 len 16384 2022-09-10T04:30:01.307 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193280 len 1024 2022-09-10T04:30:01.307 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 1024 2022-09-10T04:30:01.308 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193280 len 1024 2022-09-10T04:30:01.308 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 1024 2022-09-10T04:30:01.308 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193280 len 2048 2022-09-10T04:30:01.308 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 2048 2022-09-10T04:30:01.309 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193280 len 2048 2022-09-10T04:30:01.309 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 2048 2022-09-10T04:30:01.309 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193280 len 4096 2022-09-10T04:30:01.309 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 4096 2022-09-10T04:30:01.309 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193280 len 4096 2022-09-10T04:30:01.310 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 4096 2022-09-10T04:30:01.310 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193280 len 8192 2022-09-10T04:30:01.310 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 8192 2022-09-10T04:30:01.310 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193280 len 8192 2022-09-10T04:30:01.311 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 8192 2022-09-10T04:30:01.312 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193280 len 16384 2022-09-10T04:30:01.312 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 16384 2022-09-10T04:30:01.312 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193280 len 16384 2022-09-10T04:30:01.312 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193280 len 16384 2022-09-10T04:30:01.313 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193792 len 1024 2022-09-10T04:30:01.313 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 1024 2022-09-10T04:30:01.313 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193792 len 1024 2022-09-10T04:30:01.313 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 1024 2022-09-10T04:30:01.314 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193792 len 2048 2022-09-10T04:30:01.314 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 2048 2022-09-10T04:30:01.314 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193792 len 2048 2022-09-10T04:30:01.314 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 2048 2022-09-10T04:30:01.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193792 len 4096 2022-09-10T04:30:01.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 4096 2022-09-10T04:30:01.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193792 len 4096 2022-09-10T04:30:01.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 4096 2022-09-10T04:30:01.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193792 len 8192 2022-09-10T04:30:01.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 8192 2022-09-10T04:30:01.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193792 len 8192 2022-09-10T04:30:01.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 8192 2022-09-10T04:30:01.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4193792 len 16384 2022-09-10T04:30:01.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 16384 2022-09-10T04:30:01.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4193792 len 16384 2022-09-10T04:30:01.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4193792 len 16384 2022-09-10T04:30:01.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194304 len 1024 2022-09-10T04:30:01.318 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 1024 2022-09-10T04:30:01.318 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194304 len 1024 2022-09-10T04:30:01.318 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 1024 2022-09-10T04:30:02.098 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194304 len 2048 2022-09-10T04:30:02.099 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 2048 2022-09-10T04:30:02.099 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194304 len 2048 2022-09-10T04:30:02.100 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 2048 2022-09-10T04:30:02.100 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194304 len 4096 2022-09-10T04:30:02.100 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 4096 2022-09-10T04:30:02.100 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194304 len 4096 2022-09-10T04:30:02.100 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 4096 2022-09-10T04:30:02.101 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194304 len 8192 2022-09-10T04:30:02.101 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 8192 2022-09-10T04:30:02.101 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194304 len 8192 2022-09-10T04:30:02.101 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 8192 2022-09-10T04:30:02.102 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194304 len 16384 2022-09-10T04:30:02.102 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 16384 2022-09-10T04:30:02.102 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194304 len 16384 2022-09-10T04:30:02.102 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194304 len 16384 2022-09-10T04:30:02.103 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194816 len 1024 2022-09-10T04:30:02.103 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 1024 2022-09-10T04:30:02.103 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194816 len 1024 2022-09-10T04:30:02.103 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 1024 2022-09-10T04:30:02.104 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194816 len 2048 2022-09-10T04:30:02.104 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 2048 2022-09-10T04:30:02.104 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194816 len 2048 2022-09-10T04:30:02.105 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 2048 2022-09-10T04:30:02.105 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194816 len 4096 2022-09-10T04:30:02.106 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 4096 2022-09-10T04:30:02.106 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194816 len 4096 2022-09-10T04:30:02.106 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 4096 2022-09-10T04:30:02.107 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194816 len 8192 2022-09-10T04:30:02.107 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 8192 2022-09-10T04:30:02.107 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194816 len 8192 2022-09-10T04:30:02.108 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 8192 2022-09-10T04:30:02.108 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4194816 len 16384 2022-09-10T04:30:02.109 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 16384 2022-09-10T04:30:02.110 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4194816 len 16384 2022-09-10T04:30:02.110 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4194816 len 16384 2022-09-10T04:30:02.111 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195328 len 1024 2022-09-10T04:30:02.111 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 1024 2022-09-10T04:30:02.111 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195328 len 1024 2022-09-10T04:30:02.112 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 1024 2022-09-10T04:30:02.112 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195328 len 2048 2022-09-10T04:30:02.113 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 2048 2022-09-10T04:30:02.113 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195328 len 2048 2022-09-10T04:30:02.114 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 2048 2022-09-10T04:30:02.114 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195328 len 4096 2022-09-10T04:30:02.114 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 4096 2022-09-10T04:30:02.115 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195328 len 4096 2022-09-10T04:30:02.115 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 4096 2022-09-10T04:30:02.115 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195328 len 8192 2022-09-10T04:30:02.116 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 8192 2022-09-10T04:30:02.116 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195328 len 8192 2022-09-10T04:30:02.117 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 8192 2022-09-10T04:30:02.117 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195328 len 16384 2022-09-10T04:30:02.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 16384 2022-09-10T04:30:02.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195328 len 16384 2022-09-10T04:30:02.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195328 len 16384 2022-09-10T04:30:02.118 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195840 len 1024 2022-09-10T04:30:02.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 1024 2022-09-10T04:30:02.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195840 len 1024 2022-09-10T04:30:02.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 1024 2022-09-10T04:30:02.119 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195840 len 2048 2022-09-10T04:30:02.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 2048 2022-09-10T04:30:02.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195840 len 2048 2022-09-10T04:30:02.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 2048 2022-09-10T04:30:02.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195840 len 4096 2022-09-10T04:30:02.120 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 4096 2022-09-10T04:30:02.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195840 len 4096 2022-09-10T04:30:02.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 4096 2022-09-10T04:30:02.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195840 len 8192 2022-09-10T04:30:02.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 8192 2022-09-10T04:30:02.121 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195840 len 8192 2022-09-10T04:30:02.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 8192 2022-09-10T04:30:02.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4195840 len 16384 2022-09-10T04:30:02.122 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 16384 2022-09-10T04:30:02.123 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4195840 len 16384 2022-09-10T04:30:02.123 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4195840 len 16384 2022-09-10T04:30:02.123 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196352 len 1024 2022-09-10T04:30:02.123 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 1024 2022-09-10T04:30:02.126 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196352 len 1024 2022-09-10T04:30:02.126 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 1024 2022-09-10T04:30:02.126 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196352 len 2048 2022-09-10T04:30:02.127 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 2048 2022-09-10T04:30:02.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196352 len 2048 2022-09-10T04:30:02.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 2048 2022-09-10T04:30:02.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196352 len 4096 2022-09-10T04:30:02.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 4096 2022-09-10T04:30:02.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196352 len 4096 2022-09-10T04:30:02.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 4096 2022-09-10T04:30:02.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196352 len 8192 2022-09-10T04:30:02.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 8192 2022-09-10T04:30:02.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196352 len 8192 2022-09-10T04:30:02.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 8192 2022-09-10T04:30:02.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196352 len 16384 2022-09-10T04:30:02.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 16384 2022-09-10T04:30:02.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196352 len 16384 2022-09-10T04:30:02.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196352 len 16384 2022-09-10T04:30:02.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196864 len 1024 2022-09-10T04:30:02.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 1024 2022-09-10T04:30:02.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196864 len 1024 2022-09-10T04:30:02.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 1024 2022-09-10T04:30:02.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196864 len 2048 2022-09-10T04:30:02.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 2048 2022-09-10T04:30:02.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196864 len 2048 2022-09-10T04:30:02.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 2048 2022-09-10T04:30:02.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196864 len 4096 2022-09-10T04:30:02.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 4096 2022-09-10T04:30:02.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196864 len 4096 2022-09-10T04:30:02.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 4096 2022-09-10T04:30:02.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196864 len 8192 2022-09-10T04:30:02.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 8192 2022-09-10T04:30:02.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196864 len 8192 2022-09-10T04:30:02.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 8192 2022-09-10T04:30:02.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4196864 len 16384 2022-09-10T04:30:02.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 16384 2022-09-10T04:30:02.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4196864 len 16384 2022-09-10T04:30:02.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4196864 len 16384 2022-09-10T04:30:02.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197376 len 1024 2022-09-10T04:30:02.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 1024 2022-09-10T04:30:02.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197376 len 1024 2022-09-10T04:30:02.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 1024 2022-09-10T04:30:02.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197376 len 2048 2022-09-10T04:30:02.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 2048 2022-09-10T04:30:02.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197376 len 2048 2022-09-10T04:30:02.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 2048 2022-09-10T04:30:02.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197376 len 4096 2022-09-10T04:30:02.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 4096 2022-09-10T04:30:02.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197376 len 4096 2022-09-10T04:30:02.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 4096 2022-09-10T04:30:02.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197376 len 8192 2022-09-10T04:30:02.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 8192 2022-09-10T04:30:02.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197376 len 8192 2022-09-10T04:30:02.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 8192 2022-09-10T04:30:02.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197376 len 16384 2022-09-10T04:30:02.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 16384 2022-09-10T04:30:02.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197376 len 16384 2022-09-10T04:30:02.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197376 len 16384 2022-09-10T04:30:02.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197888 len 1024 2022-09-10T04:30:02.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 1024 2022-09-10T04:30:02.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197888 len 1024 2022-09-10T04:30:02.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 1024 2022-09-10T04:30:02.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197888 len 2048 2022-09-10T04:30:02.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 2048 2022-09-10T04:30:02.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197888 len 2048 2022-09-10T04:30:02.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 2048 2022-09-10T04:30:02.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197888 len 4096 2022-09-10T04:30:02.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 4096 2022-09-10T04:30:02.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197888 len 4096 2022-09-10T04:30:02.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 4096 2022-09-10T04:30:02.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197888 len 8192 2022-09-10T04:30:02.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 8192 2022-09-10T04:30:02.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197888 len 8192 2022-09-10T04:30:02.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 8192 2022-09-10T04:30:02.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 512 offset 4197888 len 16384 2022-09-10T04:30:02.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 16384 2022-09-10T04:30:02.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 512 offset 4197888 len 16384 2022-09-10T04:30:02.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 512 offset 4197888 len 16384 2022-09-10T04:30:02.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4190720 len 1024 2022-09-10T04:30:02.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 1024 2022-09-10T04:30:02.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4190720 len 1024 2022-09-10T04:30:02.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 1024 2022-09-10T04:30:02.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4190720 len 2048 2022-09-10T04:30:02.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 2048 2022-09-10T04:30:02.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4190720 len 2048 2022-09-10T04:30:02.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 2048 2022-09-10T04:30:02.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4190720 len 4096 2022-09-10T04:30:02.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:02 smithi110 ceph-mon[97633]: pgmap v42: 129 pgs: 129 active+clean; 432 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 913 KiB/s wr, 218 op/s 2022-09-10T04:30:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:02 smithi104 ceph-mon[97044]: pgmap v42: 129 pgs: 129 active+clean; 432 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 913 KiB/s wr, 218 op/s 2022-09-10T04:30:03.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:02 smithi081 ceph-mon[90133]: pgmap v42: 129 pgs: 129 active+clean; 432 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 913 KiB/s wr, 218 op/s 2022-09-10T04:30:04.520 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 4096 2022-09-10T04:30:04.521 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4190720 len 4096 2022-09-10T04:30:04.521 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 4096 2022-09-10T04:30:04.521 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4190720 len 8192 2022-09-10T04:30:04.521 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 8192 2022-09-10T04:30:04.522 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4190720 len 8192 2022-09-10T04:30:04.522 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 8192 2022-09-10T04:30:04.522 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4190720 len 16384 2022-09-10T04:30:04.522 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 16384 2022-09-10T04:30:04.523 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4190720 len 16384 2022-09-10T04:30:04.523 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4190720 len 16384 2022-09-10T04:30:04.523 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191232 len 1024 2022-09-10T04:30:04.523 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 1024 2022-09-10T04:30:04.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191232 len 1024 2022-09-10T04:30:04.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 1024 2022-09-10T04:30:04.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191232 len 2048 2022-09-10T04:30:04.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 2048 2022-09-10T04:30:04.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191232 len 2048 2022-09-10T04:30:04.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 2048 2022-09-10T04:30:04.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191232 len 4096 2022-09-10T04:30:04.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 4096 2022-09-10T04:30:04.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191232 len 4096 2022-09-10T04:30:04.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 4096 2022-09-10T04:30:04.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191232 len 8192 2022-09-10T04:30:04.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 8192 2022-09-10T04:30:04.527 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191232 len 8192 2022-09-10T04:30:04.527 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 8192 2022-09-10T04:30:04.527 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191232 len 16384 2022-09-10T04:30:04.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 16384 2022-09-10T04:30:04.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191232 len 16384 2022-09-10T04:30:04.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191232 len 16384 2022-09-10T04:30:04.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191744 len 1024 2022-09-10T04:30:04.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 1024 2022-09-10T04:30:04.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191744 len 1024 2022-09-10T04:30:04.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 1024 2022-09-10T04:30:04.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191744 len 2048 2022-09-10T04:30:04.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 2048 2022-09-10T04:30:04.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191744 len 2048 2022-09-10T04:30:04.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 2048 2022-09-10T04:30:04.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191744 len 4096 2022-09-10T04:30:04.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 4096 2022-09-10T04:30:04.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191744 len 4096 2022-09-10T04:30:04.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 4096 2022-09-10T04:30:04.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191744 len 8192 2022-09-10T04:30:04.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 8192 2022-09-10T04:30:04.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191744 len 8192 2022-09-10T04:30:04.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 8192 2022-09-10T04:30:04.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4191744 len 16384 2022-09-10T04:30:04.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 16384 2022-09-10T04:30:04.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4191744 len 16384 2022-09-10T04:30:04.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4191744 len 16384 2022-09-10T04:30:04.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192256 len 1024 2022-09-10T04:30:04.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 1024 2022-09-10T04:30:04.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192256 len 1024 2022-09-10T04:30:04.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 1024 2022-09-10T04:30:04.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192256 len 2048 2022-09-10T04:30:04.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 2048 2022-09-10T04:30:04.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192256 len 2048 2022-09-10T04:30:04.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 2048 2022-09-10T04:30:04.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192256 len 4096 2022-09-10T04:30:04.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 4096 2022-09-10T04:30:04.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192256 len 4096 2022-09-10T04:30:04.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 4096 2022-09-10T04:30:04.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192256 len 8192 2022-09-10T04:30:04.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 8192 2022-09-10T04:30:04.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192256 len 8192 2022-09-10T04:30:04.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 8192 2022-09-10T04:30:04.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192256 len 16384 2022-09-10T04:30:04.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 16384 2022-09-10T04:30:04.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192256 len 16384 2022-09-10T04:30:04.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192256 len 16384 2022-09-10T04:30:04.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192768 len 1024 2022-09-10T04:30:04.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 1024 2022-09-10T04:30:04.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192768 len 1024 2022-09-10T04:30:04.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 1024 2022-09-10T04:30:04.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192768 len 2048 2022-09-10T04:30:04.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 2048 2022-09-10T04:30:04.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192768 len 2048 2022-09-10T04:30:04.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 2048 2022-09-10T04:30:04.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192768 len 4096 2022-09-10T04:30:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:04 smithi110 ceph-mon[97633]: pgmap v43: 129 pgs: 129 active+clean; 432 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 909 KiB/s wr, 213 op/s 2022-09-10T04:30:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:04 smithi104 ceph-mon[97044]: pgmap v43: 129 pgs: 129 active+clean; 432 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 909 KiB/s wr, 213 op/s 2022-09-10T04:30:05.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:04 smithi081 ceph-mon[90133]: pgmap v43: 129 pgs: 129 active+clean; 432 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s rd, 909 KiB/s wr, 213 op/s 2022-09-10T04:30:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.783+0000 7f415cb9d700 1 -- 172.21.15.81:0/2428264931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 msgr2=0x7f4158100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.783+0000 7f415cb9d700 1 --2- 172.21.15.81:0/2428264931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f4158100480 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4148009840 tx=0x7f4148009b50 comp rx=0 tx=0).stop 2022-09-10T04:30:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 -- 172.21.15.81:0/2428264931 shutdown_connections 2022-09-10T04:30:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 --2- 172.21.15.81:0/2428264931 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41581051b0 0x7f415810cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 --2- 172.21.15.81:0/2428264931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f4158100480 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 --2- 172.21.15.81:0/2428264931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4158100a50 0x7f4158104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 -- 172.21.15.81:0/2428264931 >> 172.21.15.81:0/2428264931 conn(0x7f41580fba30 msgr2=0x7f41580fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 -- 172.21.15.81:0/2428264931 shutdown_connections 2022-09-10T04:30:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.784+0000 7f415cb9d700 1 -- 172.21.15.81:0/2428264931 wait complete. 2022-09-10T04:30:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 Processor -- start 2022-09-10T04:30:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 -- start start 2022-09-10T04:30:05.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f415819b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 0x7f415819bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41581051b0 0x7f4158195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158196150 con 0x7f41581000b0 2022-09-10T04:30:05.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41581962c0 con 0x7f41581051b0 2022-09-10T04:30:05.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.786+0000 7f415cb9d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158196480 con 0x7f4158100a50 2022-09-10T04:30:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 0x7f415819bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 0x7f415819bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:37632/0 (socket says 172.21.15.81:37632) 2022-09-10T04:30:05.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 -- 172.21.15.81:0/491935459 learned_addr learned my addr 172.21.15.81:0/491935459 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:05.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f41577fe700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f415819b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f4157fff700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41581051b0 0x7f4158195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 -- 172.21.15.81:0/491935459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41581051b0 msgr2=0x7f4158195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41581051b0 0x7f4158195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 -- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 msgr2=0x7f415819b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f415819b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.787+0000 7f414ffff700 1 -- 172.21.15.81:0/491935459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4148009470 con 0x7f4158100a50 2022-09-10T04:30:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f4157fff700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41581051b0 0x7f4158195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:30:05.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f41577fe700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f415819b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:30:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f414ffff700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 0x7f415819bee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f414000e740 tx=0x7f414000eb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:05.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f41557fa700 1 -- 172.21.15.81:0/491935459 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140005220 con 0x7f4158100a50 2022-09-10T04:30:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f41557fa700 1 -- 172.21.15.81:0/491935459 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4140005380 con 0x7f4158100a50 2022-09-10T04:30:05.813 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f41557fa700 1 -- 172.21.15.81:0/491935459 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140004c70 con 0x7f4158100a50 2022-09-10T04:30:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4158196760 con 0x7f4158100a50 2022-09-10T04:30:05.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.788+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41581a2920 con 0x7f4158100a50 2022-09-10T04:30:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.792+0000 7f41557fa700 1 -- 172.21.15.81:0/491935459 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 22) v1 ==== 127855+0+0 (secure 0 0 0) 0x7f414001e020 con 0x7f4158100a50 2022-09-10T04:30:05.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.793+0000 7f41557fa700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f41440992a0 0x7f414409b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.793+0000 7f41557fa700 1 -- 172.21.15.81:0/491935459 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f4140014070 con 0x7f4158100a50 2022-09-10T04:30:05.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.794+0000 7f41577fe700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f41440992a0 0x7f414409b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.794+0000 7f41712d1b80 1 Processor -- start 2022-09-10T04:30:05.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.794+0000 7f41712d1b80 1 -- start start 2022-09-10T04:30:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860d6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860e2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f41712d1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e985fb8070 con 0x55e9860d5f00 2022-09-10T04:30:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f41712d1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9860229d0 con 0x55e9860d68b0 2022-09-10T04:30:05.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f41712d1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e98601c170 con 0x55e9860dba10 2022-09-10T04:30:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f4157fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860e2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f4157fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860e2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:53250/0 (socket says 172.21.15.81:53250) 2022-09-10T04:30:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f4157fff700 1 -- 172.21.15.81:0/805183611 learned_addr learned my addr 172.21.15.81:0/805183611 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.795+0000 7f414ffff700 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860d6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f41577fe700 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f4157fff700 1 -- 172.21.15.81:0/805183611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 msgr2=0x55e9860dbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f4157fff700 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f4157fff700 1 -- 172.21.15.81:0/805183611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 msgr2=0x55e9860d6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f4157fff700 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860d6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f41577fe700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f41440992a0 0x7f414409b750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4148000c00 tx=0x7f4148005ac0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:05.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f4157fff700 1 -- 172.21.15.81:0/805183611 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e985ea45d0 con 0x55e9860d68b0 2022-09-10T04:30:05.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.796+0000 7f4157fff700 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860e2730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4150009e10 tx=0x7f415000f640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30fe0ba5a965c3e4 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.797+0000 7f414dffb700 1 -- 172.21.15.81:0/805183611 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150009ea0 con 0x55e9860d68b0 2022-09-10T04:30:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.797+0000 7f414dffb700 1 -- 172.21.15.81:0/805183611 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f415000fda0 con 0x55e9860d68b0 2022-09-10T04:30:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.797+0000 7f414dffb700 1 -- 172.21.15.81:0/805183611 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150011440 con 0x55e9860d68b0 2022-09-10T04:30:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 -- 172.21.15.81:0/805183611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 msgr2=0x55e9860e2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860e2730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4150009e10 tx=0x7f415000f640 comp rx=0 tx=0).stop 2022-09-10T04:30:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 -- 172.21.15.81:0/805183611 shutdown_connections 2022-09-10T04:30:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860e2730 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 --2- 172.21.15.81:0/805183611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860d6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 -- 172.21.15.81:0/805183611 >> 172.21.15.81:0/805183611 conn(0x55e985f2d6b0 msgr2=0x55e9860c8010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 -- 172.21.15.81:0/805183611 shutdown_connections 2022-09-10T04:30:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.798+0000 7f41712d1b80 1 -- 172.21.15.81:0/805183611 wait complete. 2022-09-10T04:30:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.799+0000 7f41712d1b80 1 -- start start 2022-09-10T04:30:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.799+0000 7f41712d1b80 1 Processor -- start 2022-09-10T04:30:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860e22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860df530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e985f0a0b0 con 0x55e9860d5f00 2022-09-10T04:30:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9860dffb0 con 0x55e9860d68b0 2022-09-10T04:30:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9860e0120 con 0x55e9860dba10 2022-09-10T04:30:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.800+0000 7f41712d1b80 10 client.0 _ll_register_callbacks cb 0x55e985fc7360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:30:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f414ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860df530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f41577fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860e22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:37660/0 (socket says 172.21.15.81:37660) 2022-09-10T04:30:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 -- 172.21.15.81:0/2180484890 learned_addr learned my addr 172.21.15.81:0/2180484890 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f414ffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860df530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:53258/0 (socket says 172.21.15.81:53258) 2022-09-10T04:30:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 -- 172.21.15.81:0/2180484890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 msgr2=0x55e9860df530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860df530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 -- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 msgr2=0x55e9860e22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860e22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 -- 172.21.15.81:0/2180484890 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4150009a40 con 0x55e9860dba10 2022-09-10T04:30:05.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f414ffff700 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860df530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:30:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.801+0000 7f4157fff700 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dfa70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4150005230 tx=0x7f4150005260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=230615453ca51c2b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f4136ffd700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:30:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f4136ffd700 1 -- 172.21.15.81:0/2180484890 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150021070 con 0x55e9860dba10 2022-09-10T04:30:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f4136ffd700 1 -- 172.21.15.81:0/2180484890 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f415000fba0 con 0x55e9860dba10 2022-09-10T04:30:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e9860c0930 con 0x55e9860dba10 2022-09-10T04:30:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f4136ffd700 1 -- 172.21.15.81:0/2180484890 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4150005380 con 0x55e9860dba10 2022-09-10T04:30:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f41712d1b80 10 client.25570 fetch_fsmap learned FSMap version 28 2022-09-10T04:30:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.802+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e985e50d80 con 0x55e9860dba10 2022-09-10T04:30:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f4136ffd700 1 -- 172.21.15.81:0/2180484890 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x55e9860c0930 con 0x55e9860dba10 2022-09-10T04:30:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f4136ffd700 1 -- 172.21.15.81:0/2180484890 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f4150011740 con 0x55e9860dba10 2022-09-10T04:30:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41712d1b80 10 client.25570 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:30:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41712d1b80 10 client.25570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:30:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41712d1b80 20 client.25570 populate_metadata read hostname 'smithi081' 2022-09-10T04:30:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41712d1b80 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e9860efc00 0x55e9860f2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41712d1b80 4 client.25570 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:30:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e98609d4b0 con 0x55e9860efc00 2022-09-10T04:30:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.803+0000 7f41577fe700 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e9860efc00 0x55e9860f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.804+0000 7f41577fe700 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e9860efc00 0x55e9860f2010 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d342ea584f58221 server_cookie=1db91c3150efdaf3 in_seq=0 out_seq=0 2022-09-10T04:30:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.804+0000 7f4136ffd700 10 client.25570 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f4136ffd700 1 -- 172.21.15.81:0/2180484890 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x55e98609d4b0 con 0x55e9860efc00 2022-09-10T04:30:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f4136ffd700 10 client.25570 handle_command_reply: tid=0 2022-09-10T04:30:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 client.25570 shutdown 2022-09-10T04:30:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 msgr2=0x55e9860dfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dfa70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4150005230 tx=0x7f4150005260 comp rx=0 tx=0).stop 2022-09-10T04:30:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 shutdown_connections 2022-09-10T04:30:05.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860dfa70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e9860e22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e9860efc00 0x55e9860f2010 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 --2- 172.21.15.81:0/2180484890 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d68b0 0x55e9860df530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 >> 172.21.15.81:0/2180484890 conn(0x55e985f2d6b0 msgr2=0x55e9860d4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f4136ffd700 0 client.25570 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 shutdown_connections 2022-09-10T04:30:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.805+0000 7f41712d1b80 1 -- 172.21.15.81:0/2180484890 wait complete. 2022-09-10T04:30:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.806+0000 7f41712d1b80 20 client.25570 trim_cache size 0 max 16384 2022-09-10T04:30:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 Processor -- start 2022-09-10T04:30:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 -- start start 2022-09-10T04:30:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 0x55e9860eb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d68b0 0x55e9860c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860da790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9860db630 con 0x55e9860d68b0 2022-09-10T04:30:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9860dda90 con 0x55e9860d5f00 2022-09-10T04:30:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.823+0000 7f41712d1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9860da380 con 0x55e9860dba10 2022-09-10T04:30:05.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f41577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 0x55e9860eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f4157fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860da790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f41577fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 0x55e9860eb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:53274/0 (socket says 172.21.15.81:53274) 2022-09-10T04:30:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f41577fe700 1 -- 172.21.15.81:0/727225896 learned_addr learned my addr 172.21.15.81:0/727225896 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:05.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f414ffff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d68b0 0x55e9860c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f4157fff700 1 -- 172.21.15.81:0/727225896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 msgr2=0x55e9860eb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f4157fff700 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 0x55e9860eb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f4157fff700 1 -- 172.21.15.81:0/727225896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d68b0 msgr2=0x55e9860c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f4157fff700 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d68b0 0x55e9860c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f4157fff700 1 -- 172.21.15.81:0/727225896 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e985e4f250 con 0x55e9860dba10 2022-09-10T04:30:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.824+0000 7f41577fe700 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 0x55e9860eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:30:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.825+0000 7f4157fff700 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860da790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4150002230 tx=0x7f415000f640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba2d6f78112f6cd9 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.825+0000 7f414e7fc700 1 -- 172.21.15.81:0/727225896 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150021020 con 0x55e9860dba10 2022-09-10T04:30:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.825+0000 7f414e7fc700 1 -- 172.21.15.81:0/727225896 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f4150011d20 con 0x55e9860dba10 2022-09-10T04:30:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.825+0000 7f414e7fc700 1 -- 172.21.15.81:0/727225896 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4150025be0 con 0x55e9860dba10 2022-09-10T04:30:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 -- 172.21.15.81:0/727225896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 msgr2=0x55e9860da790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860da790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4150002230 tx=0x7f415000f640 comp rx=0 tx=0).stop 2022-09-10T04:30:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 -- 172.21.15.81:0/727225896 shutdown_connections 2022-09-10T04:30:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860dba10 0x55e9860da790 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860d5f00 0x55e9860eb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 --2- 172.21.15.81:0/727225896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d68b0 0x55e9860c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 -- 172.21.15.81:0/727225896 >> 172.21.15.81:0/727225896 conn(0x55e985f2d6b0 msgr2=0x55e9860eb280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 -- 172.21.15.81:0/727225896 shutdown_connections 2022-09-10T04:30:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 -- 172.21.15.81:0/727225896 wait complete. 2022-09-10T04:30:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.826+0000 7f41712d1b80 1 -- start start 2022-09-10T04:30:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 Processor -- start 2022-09-10T04:30:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e986135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 0x55e986135a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860dba10 0x55e9861403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e986140920 con 0x55e9860d5f00 2022-09-10T04:30:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e986140a60 con 0x55e9860dba10 2022-09-10T04:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.827+0000 7f41712d1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e986140ba0 con 0x55e9860d68b0 2022-09-10T04:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f41712d1b80 10 client.0 _ll_register_callbacks cb 0x55e985fc7360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 0x55e986135a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f41577fe700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e986135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 0x55e986135a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:37678/0 (socket says 172.21.15.81:37678) 2022-09-10T04:30:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 -- 172.21.15.81:0/4031207383 learned_addr learned my addr 172.21.15.81:0/4031207383 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f4157fff700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860dba10 0x55e9861403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 -- 172.21.15.81:0/4031207383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860dba10 msgr2=0x55e9861403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860dba10 0x55e9861403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 -- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 msgr2=0x55e986135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e986135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 -- 172.21.15.81:0/4031207383 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415810d510 con 0x55e9860d68b0 2022-09-10T04:30:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f41577fe700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e986135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:30:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f4157fff700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860dba10 0x55e9861403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:30:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f414ffff700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 0x55e986135a90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4140000c00 tx=0x7f41400925e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d94ff0f9b96f05aa server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.828+0000 7f4136ffd700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:30:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f4136ffd700 1 -- 172.21.15.81:0/4031207383 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140092c00 con 0x55e9860d68b0 2022-09-10T04:30:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f4136ffd700 1 -- 172.21.15.81:0/4031207383 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f41400a9e40 con 0x55e9860d68b0 2022-09-10T04:30:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f415000fda0 con 0x55e9860d68b0 2022-09-10T04:30:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f4136ffd700 1 -- 172.21.15.81:0/4031207383 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4140099ea0 con 0x55e9860d68b0 2022-09-10T04:30:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f41712d1b80 10 client.25576 fetch_fsmap learned FSMap version 28 2022-09-10T04:30:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e985e50d80 con 0x55e9860d68b0 2022-09-10T04:30:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f4136ffd700 1 -- 172.21.15.81:0/4031207383 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f415000fda0 con 0x55e9860d68b0 2022-09-10T04:30:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f4136ffd700 1 -- 172.21.15.81:0/4031207383 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f4140099a10 con 0x55e9860d68b0 2022-09-10T04:30:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f41712d1b80 10 client.25576 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:30:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f41712d1b80 10 client.25576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:30:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.829+0000 7f41712d1b80 20 client.25576 populate_metadata read hostname 'smithi081' 2022-09-10T04:30:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.830+0000 7f41712d1b80 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e986142450 0x55e986144860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.830+0000 7f41712d1b80 4 client.25576 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:30:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.830+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e9860156e0 con 0x55e986142450 2022-09-10T04:30:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.830+0000 7f41577fe700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e986142450 0x55e986144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.831+0000 7f41577fe700 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e986142450 0x55e986144860 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba9333cf423616f server_cookie=8f0dfa4c8121dde5 in_seq=0 out_seq=0 2022-09-10T04:30:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.831+0000 7f4136ffd700 10 client.25576 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:05.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f4136ffd700 1 -- 172.21.15.81:0/4031207383 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e9860156e0 con 0x55e986142450 2022-09-10T04:30:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f4136ffd700 10 client.25576 handle_command_reply: tid=0 2022-09-10T04:30:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 client.25576 shutdown 2022-09-10T04:30:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 msgr2=0x55e986135a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 0x55e986135a90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4140000c00 tx=0x7f41400925e0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 shutdown_connections 2022-09-10T04:30:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x55e986142450 0x55e986144860 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x55e9860dba10 0x55e9861403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55e9860d5f00 0x55e986135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 --2- 172.21.15.81:0/4031207383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e9860d68b0 0x55e986135a90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 >> 172.21.15.81:0/4031207383 conn(0x55e985f2d6b0 msgr2=0x55e9860da790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:05.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.916+0000 7f4136ffd700 0 client.25576 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.917+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 shutdown_connections 2022-09-10T04:30:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.917+0000 7f41712d1b80 1 -- 172.21.15.81:0/4031207383 wait complete. 2022-09-10T04:30:05.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.917+0000 7f41712d1b80 20 client.25576 trim_cache size 0 max 16384 2022-09-10T04:30:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:30:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:30:05.926 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:30:05.927 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:30:05.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f41440992a0 msgr2=0x7f414409b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f41440992a0 0x7f414409b750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4148000c00 tx=0x7f4148005ac0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 msgr2=0x7f415819bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 0x7f415819bee0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f414000e740 tx=0x7f414000eb00 comp rx=0 tx=0).stop 2022-09-10T04:30:05.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 shutdown_connections 2022-09-10T04:30:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f41440992a0 0x7f414409b750 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41581051b0 0x7f4158195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41581000b0 0x7f415819b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 --2- 172.21.15.81:0/491935459 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4158100a50 0x7f415819bee0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 >> 172.21.15.81:0/491935459 conn(0x7f41580fba30 msgr2=0x7f41580fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 shutdown_connections 2022-09-10T04:30:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:05.918+0000 7f415cb9d700 1 -- 172.21.15.81:0/491935459 wait complete. 2022-09-10T04:30:06.314 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 4096 2022-09-10T04:30:06.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192768 len 4096 2022-09-10T04:30:06.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 4096 2022-09-10T04:30:06.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192768 len 8192 2022-09-10T04:30:06.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 8192 2022-09-10T04:30:06.315 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192768 len 8192 2022-09-10T04:30:06.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 8192 2022-09-10T04:30:06.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4192768 len 16384 2022-09-10T04:30:06.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 16384 2022-09-10T04:30:06.316 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4192768 len 16384 2022-09-10T04:30:06.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4192768 len 16384 2022-09-10T04:30:06.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193280 len 1024 2022-09-10T04:30:06.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 1024 2022-09-10T04:30:06.317 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193280 len 1024 2022-09-10T04:30:06.318 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 1024 2022-09-10T04:30:06.318 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193280 len 2048 2022-09-10T04:30:06.318 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 2048 2022-09-10T04:30:06.322 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193280 len 2048 2022-09-10T04:30:06.322 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 2048 2022-09-10T04:30:06.322 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193280 len 4096 2022-09-10T04:30:06.323 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 4096 2022-09-10T04:30:06.323 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193280 len 4096 2022-09-10T04:30:06.323 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 4096 2022-09-10T04:30:06.323 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193280 len 8192 2022-09-10T04:30:06.324 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 8192 2022-09-10T04:30:06.324 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193280 len 8192 2022-09-10T04:30:06.324 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 8192 2022-09-10T04:30:06.324 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193280 len 16384 2022-09-10T04:30:06.325 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 16384 2022-09-10T04:30:06.325 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193280 len 16384 2022-09-10T04:30:06.325 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193280 len 16384 2022-09-10T04:30:06.325 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193792 len 1024 2022-09-10T04:30:06.326 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 1024 2022-09-10T04:30:06.326 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193792 len 1024 2022-09-10T04:30:06.326 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 1024 2022-09-10T04:30:06.326 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193792 len 2048 2022-09-10T04:30:06.327 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 2048 2022-09-10T04:30:06.327 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193792 len 2048 2022-09-10T04:30:06.327 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 2048 2022-09-10T04:30:06.327 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193792 len 4096 2022-09-10T04:30:06.327 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 4096 2022-09-10T04:30:06.328 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193792 len 4096 2022-09-10T04:30:06.328 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 4096 2022-09-10T04:30:06.328 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193792 len 8192 2022-09-10T04:30:06.328 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 8192 2022-09-10T04:30:06.328 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193792 len 8192 2022-09-10T04:30:06.329 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 8192 2022-09-10T04:30:06.329 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4193792 len 16384 2022-09-10T04:30:06.329 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 16384 2022-09-10T04:30:06.329 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4193792 len 16384 2022-09-10T04:30:06.330 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4193792 len 16384 2022-09-10T04:30:06.330 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194304 len 1024 2022-09-10T04:30:06.330 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 1024 2022-09-10T04:30:06.330 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194304 len 1024 2022-09-10T04:30:06.330 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 1024 2022-09-10T04:30:06.331 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194304 len 2048 2022-09-10T04:30:06.331 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 2048 2022-09-10T04:30:06.331 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194304 len 2048 2022-09-10T04:30:06.331 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 2048 2022-09-10T04:30:06.332 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194304 len 4096 2022-09-10T04:30:06.332 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 4096 2022-09-10T04:30:06.332 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194304 len 4096 2022-09-10T04:30:06.332 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 4096 2022-09-10T04:30:06.332 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194304 len 8192 2022-09-10T04:30:06.333 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 8192 2022-09-10T04:30:06.333 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194304 len 8192 2022-09-10T04:30:06.333 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 8192 2022-09-10T04:30:06.334 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194304 len 16384 2022-09-10T04:30:06.334 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 16384 2022-09-10T04:30:06.334 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194304 len 16384 2022-09-10T04:30:06.334 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194304 len 16384 2022-09-10T04:30:06.334 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194816 len 1024 2022-09-10T04:30:06.335 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 1024 2022-09-10T04:30:06.335 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194816 len 1024 2022-09-10T04:30:06.335 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 1024 2022-09-10T04:30:06.335 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194816 len 2048 2022-09-10T04:30:06.336 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 2048 2022-09-10T04:30:06.336 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194816 len 2048 2022-09-10T04:30:06.336 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 2048 2022-09-10T04:30:06.336 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194816 len 4096 2022-09-10T04:30:06.336 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 4096 2022-09-10T04:30:06.863 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194816 len 4096 2022-09-10T04:30:06.864 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 4096 2022-09-10T04:30:06.864 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194816 len 8192 2022-09-10T04:30:06.864 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 8192 2022-09-10T04:30:06.864 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194816 len 8192 2022-09-10T04:30:06.865 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 8192 2022-09-10T04:30:06.865 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4194816 len 16384 2022-09-10T04:30:06.865 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 16384 2022-09-10T04:30:06.865 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4194816 len 16384 2022-09-10T04:30:06.866 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4194816 len 16384 2022-09-10T04:30:06.866 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195328 len 1024 2022-09-10T04:30:06.866 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 1024 2022-09-10T04:30:06.866 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195328 len 1024 2022-09-10T04:30:06.867 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 1024 2022-09-10T04:30:06.867 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195328 len 2048 2022-09-10T04:30:06.867 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 2048 2022-09-10T04:30:06.868 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195328 len 2048 2022-09-10T04:30:06.868 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 2048 2022-09-10T04:30:06.868 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195328 len 4096 2022-09-10T04:30:06.869 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 4096 2022-09-10T04:30:06.869 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195328 len 4096 2022-09-10T04:30:06.869 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 4096 2022-09-10T04:30:06.869 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195328 len 8192 2022-09-10T04:30:06.870 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 8192 2022-09-10T04:30:06.870 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195328 len 8192 2022-09-10T04:30:06.870 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 8192 2022-09-10T04:30:06.870 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195328 len 16384 2022-09-10T04:30:06.871 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 16384 2022-09-10T04:30:06.871 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195328 len 16384 2022-09-10T04:30:06.871 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195328 len 16384 2022-09-10T04:30:06.872 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195840 len 1024 2022-09-10T04:30:06.872 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 1024 2022-09-10T04:30:06.872 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195840 len 1024 2022-09-10T04:30:06.872 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 1024 2022-09-10T04:30:06.873 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195840 len 2048 2022-09-10T04:30:06.873 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 2048 2022-09-10T04:30:06.873 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195840 len 2048 2022-09-10T04:30:06.874 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 2048 2022-09-10T04:30:06.874 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195840 len 4096 2022-09-10T04:30:06.875 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 4096 2022-09-10T04:30:06.875 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195840 len 4096 2022-09-10T04:30:06.876 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 4096 2022-09-10T04:30:06.876 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195840 len 8192 2022-09-10T04:30:06.877 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 8192 2022-09-10T04:30:06.877 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195840 len 8192 2022-09-10T04:30:06.877 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 8192 2022-09-10T04:30:06.878 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4195840 len 16384 2022-09-10T04:30:06.878 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 16384 2022-09-10T04:30:06.879 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4195840 len 16384 2022-09-10T04:30:06.879 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4195840 len 16384 2022-09-10T04:30:06.879 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196352 len 1024 2022-09-10T04:30:06.879 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 1024 2022-09-10T04:30:06.880 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196352 len 1024 2022-09-10T04:30:06.880 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 1024 2022-09-10T04:30:06.880 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196352 len 2048 2022-09-10T04:30:06.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 2048 2022-09-10T04:30:06.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196352 len 2048 2022-09-10T04:30:06.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 2048 2022-09-10T04:30:06.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196352 len 4096 2022-09-10T04:30:06.881 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 4096 2022-09-10T04:30:06.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196352 len 4096 2022-09-10T04:30:06.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 4096 2022-09-10T04:30:06.882 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196352 len 8192 2022-09-10T04:30:06.883 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 8192 2022-09-10T04:30:06.883 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196352 len 8192 2022-09-10T04:30:06.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 8192 2022-09-10T04:30:06.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196352 len 16384 2022-09-10T04:30:06.884 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 16384 2022-09-10T04:30:06.885 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196352 len 16384 2022-09-10T04:30:06.885 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196352 len 16384 2022-09-10T04:30:06.885 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196864 len 1024 2022-09-10T04:30:06.886 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 1024 2022-09-10T04:30:06.886 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196864 len 1024 2022-09-10T04:30:06.887 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 1024 2022-09-10T04:30:06.887 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196864 len 2048 2022-09-10T04:30:06.887 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 2048 2022-09-10T04:30:06.888 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196864 len 2048 2022-09-10T04:30:06.888 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 2048 2022-09-10T04:30:06.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196864 len 4096 2022-09-10T04:30:06.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 4096 2022-09-10T04:30:06.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196864 len 4096 2022-09-10T04:30:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:06 smithi110 ceph-mon[97633]: pgmap v44: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s rd, 1.2 MiB/s wr, 302 op/s 2022-09-10T04:30:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:06 smithi104 ceph-mon[97044]: pgmap v44: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s rd, 1.2 MiB/s wr, 302 op/s 2022-09-10T04:30:07.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:06 smithi081 ceph-mon[90133]: pgmap v44: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s rd, 1.2 MiB/s wr, 302 op/s 2022-09-10T04:30:07.887 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 4096 2022-09-10T04:30:07.887 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196864 len 8192 2022-09-10T04:30:07.888 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 8192 2022-09-10T04:30:07.888 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196864 len 8192 2022-09-10T04:30:07.888 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 8192 2022-09-10T04:30:07.888 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4196864 len 16384 2022-09-10T04:30:07.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 16384 2022-09-10T04:30:07.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4196864 len 16384 2022-09-10T04:30:07.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4196864 len 16384 2022-09-10T04:30:07.889 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197376 len 1024 2022-09-10T04:30:07.890 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 1024 2022-09-10T04:30:07.890 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197376 len 1024 2022-09-10T04:30:07.890 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 1024 2022-09-10T04:30:07.890 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197376 len 2048 2022-09-10T04:30:07.891 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 2048 2022-09-10T04:30:07.891 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197376 len 2048 2022-09-10T04:30:07.891 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 2048 2022-09-10T04:30:07.891 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197376 len 4096 2022-09-10T04:30:07.892 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 4096 2022-09-10T04:30:07.892 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197376 len 4096 2022-09-10T04:30:07.892 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 4096 2022-09-10T04:30:07.892 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197376 len 8192 2022-09-10T04:30:07.893 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 8192 2022-09-10T04:30:07.893 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197376 len 8192 2022-09-10T04:30:07.893 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 8192 2022-09-10T04:30:07.893 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197376 len 16384 2022-09-10T04:30:07.894 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 16384 2022-09-10T04:30:07.894 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197376 len 16384 2022-09-10T04:30:07.894 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197376 len 16384 2022-09-10T04:30:07.895 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197888 len 1024 2022-09-10T04:30:07.895 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 1024 2022-09-10T04:30:07.895 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197888 len 1024 2022-09-10T04:30:07.895 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 1024 2022-09-10T04:30:07.896 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197888 len 2048 2022-09-10T04:30:07.896 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 2048 2022-09-10T04:30:07.896 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197888 len 2048 2022-09-10T04:30:07.896 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 2048 2022-09-10T04:30:07.897 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197888 len 4096 2022-09-10T04:30:07.897 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 4096 2022-09-10T04:30:07.897 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197888 len 4096 2022-09-10T04:30:07.897 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 4096 2022-09-10T04:30:07.898 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197888 len 8192 2022-09-10T04:30:07.898 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 8192 2022-09-10T04:30:07.898 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197888 len 8192 2022-09-10T04:30:07.898 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 8192 2022-09-10T04:30:07.898 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1024 offset 4197888 len 16384 2022-09-10T04:30:07.899 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 16384 2022-09-10T04:30:07.899 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1024 offset 4197888 len 16384 2022-09-10T04:30:07.899 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1024 offset 4197888 len 16384 2022-09-10T04:30:07.899 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4190720 len 1024 2022-09-10T04:30:07.900 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 1024 2022-09-10T04:30:07.900 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4190720 len 1024 2022-09-10T04:30:07.900 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 1024 2022-09-10T04:30:07.900 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4190720 len 2048 2022-09-10T04:30:07.900 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 2048 2022-09-10T04:30:07.901 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4190720 len 2048 2022-09-10T04:30:07.901 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 2048 2022-09-10T04:30:07.901 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4190720 len 4096 2022-09-10T04:30:07.901 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 4096 2022-09-10T04:30:07.902 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4190720 len 4096 2022-09-10T04:30:07.902 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 4096 2022-09-10T04:30:07.902 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4190720 len 8192 2022-09-10T04:30:07.902 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 8192 2022-09-10T04:30:07.903 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4190720 len 8192 2022-09-10T04:30:07.903 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 8192 2022-09-10T04:30:07.903 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4190720 len 16384 2022-09-10T04:30:07.903 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 16384 2022-09-10T04:30:07.903 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4190720 len 16384 2022-09-10T04:30:07.904 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4190720 len 16384 2022-09-10T04:30:07.904 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191232 len 1024 2022-09-10T04:30:07.904 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 1024 2022-09-10T04:30:07.904 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191232 len 1024 2022-09-10T04:30:07.905 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 1024 2022-09-10T04:30:07.905 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191232 len 2048 2022-09-10T04:30:07.905 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 2048 2022-09-10T04:30:07.905 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191232 len 2048 2022-09-10T04:30:07.905 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 2048 2022-09-10T04:30:07.906 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191232 len 4096 2022-09-10T04:30:07.906 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 4096 2022-09-10T04:30:07.906 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191232 len 4096 2022-09-10T04:30:07.906 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 4096 2022-09-10T04:30:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:08 smithi110 ceph-mon[97633]: pgmap v45: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 484 KiB/s wr, 145 op/s 2022-09-10T04:30:09.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:08 smithi104 ceph-mon[97044]: pgmap v45: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 484 KiB/s wr, 145 op/s 2022-09-10T04:30:09.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:08 smithi081 ceph-mon[90133]: pgmap v45: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 484 KiB/s wr, 145 op/s 2022-09-10T04:30:09.618 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191232 len 8192 2022-09-10T04:30:09.619 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 8192 2022-09-10T04:30:09.619 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191232 len 8192 2022-09-10T04:30:09.620 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 8192 2022-09-10T04:30:09.620 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191232 len 16384 2022-09-10T04:30:09.621 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 16384 2022-09-10T04:30:09.621 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191232 len 16384 2022-09-10T04:30:09.621 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191232 len 16384 2022-09-10T04:30:09.621 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191744 len 1024 2022-09-10T04:30:09.622 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 1024 2022-09-10T04:30:09.622 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191744 len 1024 2022-09-10T04:30:09.622 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 1024 2022-09-10T04:30:09.622 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191744 len 2048 2022-09-10T04:30:09.622 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 2048 2022-09-10T04:30:09.623 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191744 len 2048 2022-09-10T04:30:09.623 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 2048 2022-09-10T04:30:09.623 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191744 len 4096 2022-09-10T04:30:09.623 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 4096 2022-09-10T04:30:09.624 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191744 len 4096 2022-09-10T04:30:09.624 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 4096 2022-09-10T04:30:09.624 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191744 len 8192 2022-09-10T04:30:09.624 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 8192 2022-09-10T04:30:09.624 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191744 len 8192 2022-09-10T04:30:09.625 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 8192 2022-09-10T04:30:09.625 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4191744 len 16384 2022-09-10T04:30:09.625 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 16384 2022-09-10T04:30:09.626 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4191744 len 16384 2022-09-10T04:30:09.626 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4191744 len 16384 2022-09-10T04:30:09.626 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192256 len 1024 2022-09-10T04:30:09.627 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 1024 2022-09-10T04:30:09.627 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192256 len 1024 2022-09-10T04:30:09.627 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 1024 2022-09-10T04:30:09.627 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192256 len 2048 2022-09-10T04:30:09.628 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 2048 2022-09-10T04:30:09.628 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192256 len 2048 2022-09-10T04:30:09.628 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 2048 2022-09-10T04:30:09.629 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192256 len 4096 2022-09-10T04:30:09.629 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 4096 2022-09-10T04:30:09.629 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192256 len 4096 2022-09-10T04:30:09.629 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 4096 2022-09-10T04:30:09.630 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192256 len 8192 2022-09-10T04:30:09.630 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 8192 2022-09-10T04:30:09.630 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192256 len 8192 2022-09-10T04:30:09.630 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 8192 2022-09-10T04:30:09.630 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192256 len 16384 2022-09-10T04:30:09.631 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 16384 2022-09-10T04:30:09.631 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192256 len 16384 2022-09-10T04:30:09.631 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192256 len 16384 2022-09-10T04:30:09.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192768 len 1024 2022-09-10T04:30:09.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 1024 2022-09-10T04:30:09.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192768 len 1024 2022-09-10T04:30:09.632 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 1024 2022-09-10T04:30:09.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192768 len 2048 2022-09-10T04:30:09.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 2048 2022-09-10T04:30:09.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192768 len 2048 2022-09-10T04:30:09.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 2048 2022-09-10T04:30:09.633 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192768 len 4096 2022-09-10T04:30:09.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 4096 2022-09-10T04:30:09.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192768 len 4096 2022-09-10T04:30:09.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 4096 2022-09-10T04:30:09.634 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192768 len 8192 2022-09-10T04:30:09.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 8192 2022-09-10T04:30:09.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192768 len 8192 2022-09-10T04:30:09.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 8192 2022-09-10T04:30:09.635 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4192768 len 16384 2022-09-10T04:30:09.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 16384 2022-09-10T04:30:09.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4192768 len 16384 2022-09-10T04:30:09.636 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4192768 len 16384 2022-09-10T04:30:09.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193280 len 1024 2022-09-10T04:30:09.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 1024 2022-09-10T04:30:09.637 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193280 len 1024 2022-09-10T04:30:09.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 1024 2022-09-10T04:30:09.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193280 len 2048 2022-09-10T04:30:09.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 2048 2022-09-10T04:30:09.638 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193280 len 2048 2022-09-10T04:30:09.639 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 2048 2022-09-10T04:30:09.639 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193280 len 4096 2022-09-10T04:30:09.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 4096 2022-09-10T04:30:09.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193280 len 4096 2022-09-10T04:30:09.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 4096 2022-09-10T04:30:09.640 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193280 len 8192 2022-09-10T04:30:10.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 8192 2022-09-10T04:30:10.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193280 len 8192 2022-09-10T04:30:10.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 8192 2022-09-10T04:30:10.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193280 len 16384 2022-09-10T04:30:10.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 16384 2022-09-10T04:30:10.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193280 len 16384 2022-09-10T04:30:10.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193280 len 16384 2022-09-10T04:30:10.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193792 len 1024 2022-09-10T04:30:10.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 1024 2022-09-10T04:30:10.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193792 len 1024 2022-09-10T04:30:10.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 1024 2022-09-10T04:30:10.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193792 len 2048 2022-09-10T04:30:10.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 2048 2022-09-10T04:30:10.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193792 len 2048 2022-09-10T04:30:10.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 2048 2022-09-10T04:30:10.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193792 len 4096 2022-09-10T04:30:10.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 4096 2022-09-10T04:30:10.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193792 len 4096 2022-09-10T04:30:10.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 4096 2022-09-10T04:30:10.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193792 len 8192 2022-09-10T04:30:10.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 8192 2022-09-10T04:30:10.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193792 len 8192 2022-09-10T04:30:10.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 8192 2022-09-10T04:30:10.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4193792 len 16384 2022-09-10T04:30:10.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 16384 2022-09-10T04:30:10.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4193792 len 16384 2022-09-10T04:30:10.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4193792 len 16384 2022-09-10T04:30:10.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194304 len 1024 2022-09-10T04:30:10.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 1024 2022-09-10T04:30:10.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194304 len 1024 2022-09-10T04:30:10.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 1024 2022-09-10T04:30:10.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194304 len 2048 2022-09-10T04:30:10.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 2048 2022-09-10T04:30:10.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194304 len 2048 2022-09-10T04:30:10.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 2048 2022-09-10T04:30:10.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194304 len 4096 2022-09-10T04:30:10.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 4096 2022-09-10T04:30:10.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194304 len 4096 2022-09-10T04:30:10.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 4096 2022-09-10T04:30:10.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194304 len 8192 2022-09-10T04:30:10.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 8192 2022-09-10T04:30:10.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194304 len 8192 2022-09-10T04:30:10.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 8192 2022-09-10T04:30:10.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194304 len 16384 2022-09-10T04:30:10.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 16384 2022-09-10T04:30:10.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194304 len 16384 2022-09-10T04:30:10.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194304 len 16384 2022-09-10T04:30:10.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194816 len 1024 2022-09-10T04:30:10.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 1024 2022-09-10T04:30:10.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194816 len 1024 2022-09-10T04:30:10.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 1024 2022-09-10T04:30:10.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194816 len 2048 2022-09-10T04:30:10.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 2048 2022-09-10T04:30:10.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194816 len 2048 2022-09-10T04:30:10.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 2048 2022-09-10T04:30:10.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194816 len 4096 2022-09-10T04:30:10.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 4096 2022-09-10T04:30:10.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194816 len 4096 2022-09-10T04:30:10.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 4096 2022-09-10T04:30:10.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194816 len 8192 2022-09-10T04:30:10.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 8192 2022-09-10T04:30:10.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194816 len 8192 2022-09-10T04:30:10.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 8192 2022-09-10T04:30:10.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4194816 len 16384 2022-09-10T04:30:10.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 16384 2022-09-10T04:30:10.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4194816 len 16384 2022-09-10T04:30:10.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4194816 len 16384 2022-09-10T04:30:10.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195328 len 1024 2022-09-10T04:30:10.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 1024 2022-09-10T04:30:10.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195328 len 1024 2022-09-10T04:30:10.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 1024 2022-09-10T04:30:10.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195328 len 2048 2022-09-10T04:30:10.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 2048 2022-09-10T04:30:10.811 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195328 len 2048 2022-09-10T04:30:10.811 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 2048 2022-09-10T04:30:10.811 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195328 len 4096 2022-09-10T04:30:10.811 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 4096 2022-09-10T04:30:10.812 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195328 len 4096 2022-09-10T04:30:10.812 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 4096 2022-09-10T04:30:10.812 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195328 len 8192 2022-09-10T04:30:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:10 smithi110 ceph-mon[97633]: pgmap v46: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 484 KiB/s wr, 145 op/s 2022-09-10T04:30:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:10 smithi104 ceph-mon[97044]: pgmap v46: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 484 KiB/s wr, 145 op/s 2022-09-10T04:30:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:10 smithi081 ceph-mon[90133]: pgmap v46: 129 pgs: 129 active+clean; 238 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 484 KiB/s wr, 145 op/s 2022-09-10T04:30:11.382 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 8192 2022-09-10T04:30:11.383 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195328 len 8192 2022-09-10T04:30:11.383 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 8192 2022-09-10T04:30:11.384 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195328 len 16384 2022-09-10T04:30:11.384 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 16384 2022-09-10T04:30:11.384 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195328 len 16384 2022-09-10T04:30:11.385 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195328 len 16384 2022-09-10T04:30:11.385 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195840 len 1024 2022-09-10T04:30:11.385 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 1024 2022-09-10T04:30:11.385 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195840 len 1024 2022-09-10T04:30:11.386 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 1024 2022-09-10T04:30:11.386 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195840 len 2048 2022-09-10T04:30:11.386 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 2048 2022-09-10T04:30:11.386 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195840 len 2048 2022-09-10T04:30:11.387 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 2048 2022-09-10T04:30:11.387 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195840 len 4096 2022-09-10T04:30:11.387 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 4096 2022-09-10T04:30:11.387 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195840 len 4096 2022-09-10T04:30:11.388 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 4096 2022-09-10T04:30:11.388 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195840 len 8192 2022-09-10T04:30:11.388 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 8192 2022-09-10T04:30:11.388 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195840 len 8192 2022-09-10T04:30:11.388 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 8192 2022-09-10T04:30:11.389 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4195840 len 16384 2022-09-10T04:30:11.389 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 16384 2022-09-10T04:30:11.389 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4195840 len 16384 2022-09-10T04:30:11.389 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4195840 len 16384 2022-09-10T04:30:11.390 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196352 len 1024 2022-09-10T04:30:11.390 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 1024 2022-09-10T04:30:11.390 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196352 len 1024 2022-09-10T04:30:11.390 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 1024 2022-09-10T04:30:11.390 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196352 len 2048 2022-09-10T04:30:11.391 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 2048 2022-09-10T04:30:11.391 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196352 len 2048 2022-09-10T04:30:11.391 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 2048 2022-09-10T04:30:11.391 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196352 len 4096 2022-09-10T04:30:11.392 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 4096 2022-09-10T04:30:11.392 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196352 len 4096 2022-09-10T04:30:11.392 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 4096 2022-09-10T04:30:11.392 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196352 len 8192 2022-09-10T04:30:11.392 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 8192 2022-09-10T04:30:11.393 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196352 len 8192 2022-09-10T04:30:11.393 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 8192 2022-09-10T04:30:11.393 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196352 len 16384 2022-09-10T04:30:11.393 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 16384 2022-09-10T04:30:11.393 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196352 len 16384 2022-09-10T04:30:11.394 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196352 len 16384 2022-09-10T04:30:11.394 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196864 len 1024 2022-09-10T04:30:11.395 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 1024 2022-09-10T04:30:11.395 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196864 len 1024 2022-09-10T04:30:11.395 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 1024 2022-09-10T04:30:11.396 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196864 len 2048 2022-09-10T04:30:11.397 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 2048 2022-09-10T04:30:11.397 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196864 len 2048 2022-09-10T04:30:11.397 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 2048 2022-09-10T04:30:11.398 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196864 len 4096 2022-09-10T04:30:11.398 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 4096 2022-09-10T04:30:11.398 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196864 len 4096 2022-09-10T04:30:11.399 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 4096 2022-09-10T04:30:11.399 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196864 len 8192 2022-09-10T04:30:11.400 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 8192 2022-09-10T04:30:11.400 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196864 len 8192 2022-09-10T04:30:11.401 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 8192 2022-09-10T04:30:11.401 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4196864 len 16384 2022-09-10T04:30:11.402 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 16384 2022-09-10T04:30:11.402 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4196864 len 16384 2022-09-10T04:30:11.403 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4196864 len 16384 2022-09-10T04:30:11.403 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197376 len 1024 2022-09-10T04:30:11.403 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 1024 2022-09-10T04:30:11.404 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197376 len 1024 2022-09-10T04:30:11.404 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 1024 2022-09-10T04:30:11.404 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197376 len 2048 2022-09-10T04:30:11.405 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 2048 2022-09-10T04:30:11.405 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197376 len 2048 2022-09-10T04:30:11.406 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 2048 2022-09-10T04:30:11.406 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197376 len 4096 2022-09-10T04:30:11.406 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 4096 2022-09-10T04:30:11.407 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197376 len 4096 2022-09-10T04:30:11.407 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 4096 2022-09-10T04:30:11.407 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197376 len 8192 2022-09-10T04:30:11.407 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 8192 2022-09-10T04:30:12.715 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197376 len 8192 2022-09-10T04:30:12.715 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 8192 2022-09-10T04:30:12.716 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197376 len 16384 2022-09-10T04:30:12.716 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 16384 2022-09-10T04:30:12.716 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197376 len 16384 2022-09-10T04:30:12.716 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197376 len 16384 2022-09-10T04:30:12.717 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197888 len 1024 2022-09-10T04:30:12.717 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 1024 2022-09-10T04:30:12.717 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197888 len 1024 2022-09-10T04:30:12.717 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 1024 2022-09-10T04:30:12.717 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197888 len 2048 2022-09-10T04:30:12.718 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 2048 2022-09-10T04:30:12.718 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197888 len 2048 2022-09-10T04:30:12.718 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 2048 2022-09-10T04:30:12.718 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197888 len 4096 2022-09-10T04:30:12.719 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 4096 2022-09-10T04:30:12.719 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197888 len 4096 2022-09-10T04:30:12.719 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 4096 2022-09-10T04:30:12.719 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197888 len 8192 2022-09-10T04:30:12.720 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 8192 2022-09-10T04:30:12.720 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197888 len 8192 2022-09-10T04:30:12.720 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 8192 2022-09-10T04:30:12.720 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 1536 offset 4197888 len 16384 2022-09-10T04:30:12.721 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 16384 2022-09-10T04:30:12.721 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 1536 offset 4197888 len 16384 2022-09-10T04:30:12.722 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 1536 offset 4197888 len 16384 2022-09-10T04:30:12.722 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4190720 len 1024 2022-09-10T04:30:12.722 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 1024 2022-09-10T04:30:12.723 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4190720 len 1024 2022-09-10T04:30:12.723 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 1024 2022-09-10T04:30:12.723 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4190720 len 2048 2022-09-10T04:30:12.723 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 2048 2022-09-10T04:30:12.724 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4190720 len 2048 2022-09-10T04:30:12.724 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 2048 2022-09-10T04:30:12.724 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4190720 len 4096 2022-09-10T04:30:12.724 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 4096 2022-09-10T04:30:12.724 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4190720 len 4096 2022-09-10T04:30:12.725 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 4096 2022-09-10T04:30:12.725 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4190720 len 8192 2022-09-10T04:30:12.725 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 8192 2022-09-10T04:30:12.726 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4190720 len 8192 2022-09-10T04:30:12.730 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 8192 2022-09-10T04:30:12.730 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4190720 len 16384 2022-09-10T04:30:12.731 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 16384 2022-09-10T04:30:12.731 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4190720 len 16384 2022-09-10T04:30:12.731 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4190720 len 16384 2022-09-10T04:30:12.732 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191232 len 1024 2022-09-10T04:30:12.732 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 1024 2022-09-10T04:30:12.732 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191232 len 1024 2022-09-10T04:30:12.733 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 1024 2022-09-10T04:30:12.733 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191232 len 2048 2022-09-10T04:30:12.733 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 2048 2022-09-10T04:30:12.733 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191232 len 2048 2022-09-10T04:30:12.734 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 2048 2022-09-10T04:30:12.734 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191232 len 4096 2022-09-10T04:30:12.734 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 4096 2022-09-10T04:30:12.734 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191232 len 4096 2022-09-10T04:30:12.734 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 4096 2022-09-10T04:30:12.735 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191232 len 8192 2022-09-10T04:30:12.735 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 8192 2022-09-10T04:30:12.735 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191232 len 8192 2022-09-10T04:30:12.735 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 8192 2022-09-10T04:30:12.735 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191232 len 16384 2022-09-10T04:30:12.736 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 16384 2022-09-10T04:30:12.736 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191232 len 16384 2022-09-10T04:30:12.736 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191232 len 16384 2022-09-10T04:30:12.736 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191744 len 1024 2022-09-10T04:30:12.737 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 1024 2022-09-10T04:30:12.737 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191744 len 1024 2022-09-10T04:30:12.737 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 1024 2022-09-10T04:30:12.737 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191744 len 2048 2022-09-10T04:30:12.737 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 2048 2022-09-10T04:30:12.738 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191744 len 2048 2022-09-10T04:30:12.738 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 2048 2022-09-10T04:30:12.738 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191744 len 4096 2022-09-10T04:30:12.738 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 4096 2022-09-10T04:30:12.738 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191744 len 4096 2022-09-10T04:30:12.739 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 4096 2022-09-10T04:30:12.739 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191744 len 8192 2022-09-10T04:30:12.739 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 8192 2022-09-10T04:30:12.740 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191744 len 8192 2022-09-10T04:30:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:12 smithi110 ceph-mon[97633]: pgmap v47: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s rd, 748 KiB/s wr, 244 op/s 2022-09-10T04:30:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:12 smithi104 ceph-mon[97044]: pgmap v47: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s rd, 748 KiB/s wr, 244 op/s 2022-09-10T04:30:13.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:12 smithi081 ceph-mon[90133]: pgmap v47: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s rd, 748 KiB/s wr, 244 op/s 2022-09-10T04:30:14.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 8192 2022-09-10T04:30:14.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4191744 len 16384 2022-09-10T04:30:14.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 16384 2022-09-10T04:30:14.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4191744 len 16384 2022-09-10T04:30:14.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4191744 len 16384 2022-09-10T04:30:14.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192256 len 1024 2022-09-10T04:30:14.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 1024 2022-09-10T04:30:14.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192256 len 1024 2022-09-10T04:30:14.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 1024 2022-09-10T04:30:14.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192256 len 2048 2022-09-10T04:30:14.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 2048 2022-09-10T04:30:14.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192256 len 2048 2022-09-10T04:30:14.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 2048 2022-09-10T04:30:14.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192256 len 4096 2022-09-10T04:30:14.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 4096 2022-09-10T04:30:14.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192256 len 4096 2022-09-10T04:30:14.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 4096 2022-09-10T04:30:14.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192256 len 8192 2022-09-10T04:30:14.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 8192 2022-09-10T04:30:14.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192256 len 8192 2022-09-10T04:30:14.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 8192 2022-09-10T04:30:14.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192256 len 16384 2022-09-10T04:30:14.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 16384 2022-09-10T04:30:14.541 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192256 len 16384 2022-09-10T04:30:14.541 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192256 len 16384 2022-09-10T04:30:14.542 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192768 len 1024 2022-09-10T04:30:14.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 1024 2022-09-10T04:30:14.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192768 len 1024 2022-09-10T04:30:14.544 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 1024 2022-09-10T04:30:14.544 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192768 len 2048 2022-09-10T04:30:14.545 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 2048 2022-09-10T04:30:14.545 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192768 len 2048 2022-09-10T04:30:14.545 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 2048 2022-09-10T04:30:14.546 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192768 len 4096 2022-09-10T04:30:14.546 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 4096 2022-09-10T04:30:14.546 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192768 len 4096 2022-09-10T04:30:14.546 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 4096 2022-09-10T04:30:14.547 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192768 len 8192 2022-09-10T04:30:14.547 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 8192 2022-09-10T04:30:14.547 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192768 len 8192 2022-09-10T04:30:14.547 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 8192 2022-09-10T04:30:14.548 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4192768 len 16384 2022-09-10T04:30:14.548 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 16384 2022-09-10T04:30:14.548 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4192768 len 16384 2022-09-10T04:30:14.548 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4192768 len 16384 2022-09-10T04:30:14.549 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193280 len 1024 2022-09-10T04:30:14.549 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 1024 2022-09-10T04:30:14.549 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193280 len 1024 2022-09-10T04:30:14.549 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 1024 2022-09-10T04:30:14.549 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193280 len 2048 2022-09-10T04:30:14.550 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 2048 2022-09-10T04:30:14.550 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193280 len 2048 2022-09-10T04:30:14.550 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 2048 2022-09-10T04:30:14.550 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193280 len 4096 2022-09-10T04:30:14.551 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 4096 2022-09-10T04:30:14.551 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193280 len 4096 2022-09-10T04:30:14.551 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 4096 2022-09-10T04:30:14.551 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193280 len 8192 2022-09-10T04:30:14.552 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 8192 2022-09-10T04:30:14.552 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193280 len 8192 2022-09-10T04:30:14.552 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 8192 2022-09-10T04:30:14.552 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193280 len 16384 2022-09-10T04:30:14.552 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 16384 2022-09-10T04:30:14.553 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193280 len 16384 2022-09-10T04:30:14.553 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193280 len 16384 2022-09-10T04:30:14.553 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193792 len 1024 2022-09-10T04:30:14.553 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 1024 2022-09-10T04:30:14.554 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193792 len 1024 2022-09-10T04:30:14.554 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 1024 2022-09-10T04:30:14.554 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193792 len 2048 2022-09-10T04:30:14.554 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 2048 2022-09-10T04:30:14.555 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193792 len 2048 2022-09-10T04:30:14.555 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 2048 2022-09-10T04:30:14.555 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193792 len 4096 2022-09-10T04:30:14.555 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 4096 2022-09-10T04:30:14.555 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193792 len 4096 2022-09-10T04:30:14.556 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 4096 2022-09-10T04:30:14.556 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193792 len 8192 2022-09-10T04:30:14.556 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 8192 2022-09-10T04:30:14.556 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193792 len 8192 2022-09-10T04:30:14.557 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 8192 2022-09-10T04:30:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:14 smithi110 ceph-mon[97633]: pgmap v48: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s rd, 547 KiB/s wr, 188 op/s 2022-09-10T04:30:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:14 smithi104 ceph-mon[97044]: pgmap v48: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s rd, 547 KiB/s wr, 188 op/s 2022-09-10T04:30:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:14 smithi081 ceph-mon[90133]: pgmap v48: 129 pgs: 129 active+clean; 532 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s rd, 547 KiB/s wr, 188 op/s 2022-09-10T04:30:15.192 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4193792 len 16384 2022-09-10T04:30:15.194 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 16384 2022-09-10T04:30:15.194 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4193792 len 16384 2022-09-10T04:30:15.194 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4193792 len 16384 2022-09-10T04:30:15.194 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194304 len 1024 2022-09-10T04:30:15.195 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 1024 2022-09-10T04:30:15.195 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194304 len 1024 2022-09-10T04:30:15.195 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 1024 2022-09-10T04:30:15.196 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194304 len 2048 2022-09-10T04:30:15.196 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 2048 2022-09-10T04:30:15.196 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194304 len 2048 2022-09-10T04:30:15.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 2048 2022-09-10T04:30:15.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194304 len 4096 2022-09-10T04:30:15.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 4096 2022-09-10T04:30:15.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194304 len 4096 2022-09-10T04:30:15.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 4096 2022-09-10T04:30:15.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194304 len 8192 2022-09-10T04:30:15.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 8192 2022-09-10T04:30:15.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194304 len 8192 2022-09-10T04:30:15.199 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 8192 2022-09-10T04:30:15.199 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194304 len 16384 2022-09-10T04:30:15.199 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 16384 2022-09-10T04:30:15.200 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194304 len 16384 2022-09-10T04:30:15.200 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194304 len 16384 2022-09-10T04:30:15.200 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194816 len 1024 2022-09-10T04:30:15.200 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 1024 2022-09-10T04:30:15.201 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194816 len 1024 2022-09-10T04:30:15.201 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 1024 2022-09-10T04:30:15.201 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194816 len 2048 2022-09-10T04:30:15.202 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 2048 2022-09-10T04:30:15.202 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194816 len 2048 2022-09-10T04:30:15.202 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 2048 2022-09-10T04:30:15.202 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194816 len 4096 2022-09-10T04:30:15.203 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 4096 2022-09-10T04:30:15.203 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194816 len 4096 2022-09-10T04:30:15.203 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 4096 2022-09-10T04:30:15.203 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194816 len 8192 2022-09-10T04:30:15.203 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 8192 2022-09-10T04:30:15.204 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194816 len 8192 2022-09-10T04:30:15.204 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 8192 2022-09-10T04:30:15.205 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4194816 len 16384 2022-09-10T04:30:15.205 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 16384 2022-09-10T04:30:15.205 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4194816 len 16384 2022-09-10T04:30:15.205 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4194816 len 16384 2022-09-10T04:30:15.206 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195328 len 1024 2022-09-10T04:30:15.206 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 1024 2022-09-10T04:30:15.206 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195328 len 1024 2022-09-10T04:30:15.207 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 1024 2022-09-10T04:30:15.207 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195328 len 2048 2022-09-10T04:30:15.208 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 2048 2022-09-10T04:30:15.208 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195328 len 2048 2022-09-10T04:30:15.208 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 2048 2022-09-10T04:30:15.208 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195328 len 4096 2022-09-10T04:30:15.208 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 4096 2022-09-10T04:30:15.209 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195328 len 4096 2022-09-10T04:30:15.209 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 4096 2022-09-10T04:30:15.209 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195328 len 8192 2022-09-10T04:30:15.209 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 8192 2022-09-10T04:30:15.210 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195328 len 8192 2022-09-10T04:30:15.210 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 8192 2022-09-10T04:30:15.210 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195328 len 16384 2022-09-10T04:30:15.210 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 16384 2022-09-10T04:30:15.211 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195328 len 16384 2022-09-10T04:30:15.211 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195328 len 16384 2022-09-10T04:30:15.211 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195840 len 1024 2022-09-10T04:30:15.211 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 1024 2022-09-10T04:30:15.212 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195840 len 1024 2022-09-10T04:30:15.212 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 1024 2022-09-10T04:30:15.212 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195840 len 2048 2022-09-10T04:30:15.212 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 2048 2022-09-10T04:30:15.212 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195840 len 2048 2022-09-10T04:30:15.213 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 2048 2022-09-10T04:30:15.213 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195840 len 4096 2022-09-10T04:30:15.213 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 4096 2022-09-10T04:30:15.213 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195840 len 4096 2022-09-10T04:30:15.213 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 4096 2022-09-10T04:30:15.214 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195840 len 8192 2022-09-10T04:30:15.214 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 8192 2022-09-10T04:30:15.215 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195840 len 8192 2022-09-10T04:30:15.215 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 8192 2022-09-10T04:30:15.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4195840 len 16384 2022-09-10T04:30:15.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 16384 2022-09-10T04:30:15.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4195840 len 16384 2022-09-10T04:30:15.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4195840 len 16384 2022-09-10T04:30:15.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196352 len 1024 2022-09-10T04:30:15.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 1024 2022-09-10T04:30:15.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196352 len 1024 2022-09-10T04:30:15.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 1024 2022-09-10T04:30:15.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196352 len 2048 2022-09-10T04:30:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 2048 2022-09-10T04:30:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196352 len 2048 2022-09-10T04:30:15.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 2048 2022-09-10T04:30:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196352 len 4096 2022-09-10T04:30:15.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 4096 2022-09-10T04:30:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196352 len 4096 2022-09-10T04:30:15.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 4096 2022-09-10T04:30:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196352 len 8192 2022-09-10T04:30:15.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 8192 2022-09-10T04:30:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196352 len 8192 2022-09-10T04:30:15.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 8192 2022-09-10T04:30:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196352 len 16384 2022-09-10T04:30:15.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 16384 2022-09-10T04:30:15.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196352 len 16384 2022-09-10T04:30:15.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196352 len 16384 2022-09-10T04:30:15.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196864 len 1024 2022-09-10T04:30:15.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 1024 2022-09-10T04:30:15.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196864 len 1024 2022-09-10T04:30:15.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 1024 2022-09-10T04:30:15.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196864 len 2048 2022-09-10T04:30:15.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 2048 2022-09-10T04:30:15.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196864 len 2048 2022-09-10T04:30:15.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 2048 2022-09-10T04:30:15.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196864 len 4096 2022-09-10T04:30:15.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 4096 2022-09-10T04:30:15.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196864 len 4096 2022-09-10T04:30:15.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 4096 2022-09-10T04:30:15.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196864 len 8192 2022-09-10T04:30:15.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 8192 2022-09-10T04:30:15.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196864 len 8192 2022-09-10T04:30:15.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 8192 2022-09-10T04:30:15.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4196864 len 16384 2022-09-10T04:30:15.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 16384 2022-09-10T04:30:15.803 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4196864 len 16384 2022-09-10T04:30:15.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4196864 len 16384 2022-09-10T04:30:15.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197376 len 1024 2022-09-10T04:30:15.804 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 1024 2022-09-10T04:30:15.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197376 len 1024 2022-09-10T04:30:15.805 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 1024 2022-09-10T04:30:15.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197376 len 2048 2022-09-10T04:30:15.806 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 2048 2022-09-10T04:30:15.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197376 len 2048 2022-09-10T04:30:15.807 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 2048 2022-09-10T04:30:15.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197376 len 4096 2022-09-10T04:30:15.808 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 4096 2022-09-10T04:30:15.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197376 len 4096 2022-09-10T04:30:15.809 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 4096 2022-09-10T04:30:15.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197376 len 8192 2022-09-10T04:30:15.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 8192 2022-09-10T04:30:15.810 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197376 len 8192 2022-09-10T04:30:15.811 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 8192 2022-09-10T04:30:15.811 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197376 len 16384 2022-09-10T04:30:15.812 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 16384 2022-09-10T04:30:15.812 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197376 len 16384 2022-09-10T04:30:15.813 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197376 len 16384 2022-09-10T04:30:15.813 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197888 len 1024 2022-09-10T04:30:15.814 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 1024 2022-09-10T04:30:15.814 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197888 len 1024 2022-09-10T04:30:15.815 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 1024 2022-09-10T04:30:15.815 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197888 len 2048 2022-09-10T04:30:15.816 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 2048 2022-09-10T04:30:15.816 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197888 len 2048 2022-09-10T04:30:15.816 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 2048 2022-09-10T04:30:15.817 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197888 len 4096 2022-09-10T04:30:15.817 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 4096 2022-09-10T04:30:15.818 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197888 len 4096 2022-09-10T04:30:15.818 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 4096 2022-09-10T04:30:15.819 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197888 len 8192 2022-09-10T04:30:15.819 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 8192 2022-09-10T04:30:15.820 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197888 len 8192 2022-09-10T04:30:15.820 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 8192 2022-09-10T04:30:15.820 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2048 offset 4197888 len 16384 2022-09-10T04:30:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:16 smithi104 ceph-mon[97044]: pgmap v49: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s rd, 787 KiB/s wr, 282 op/s 2022-09-10T04:30:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:16 smithi104 ceph-mon[97044]: mgrmap e23: x(active, since 92s), standbys: y, z 2022-09-10T04:30:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:16 smithi081 ceph-mon[90133]: pgmap v49: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s rd, 787 KiB/s wr, 282 op/s 2022-09-10T04:30:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:16 smithi081 ceph-mon[90133]: mgrmap e23: x(active, since 92s), standbys: y, z 2022-09-10T04:30:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:16 smithi110 ceph-mon[97633]: pgmap v49: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s rd, 787 KiB/s wr, 282 op/s 2022-09-10T04:30:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:16 smithi110 ceph-mon[97633]: mgrmap e23: x(active, since 92s), standbys: y, z 2022-09-10T04:30:17.490 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 16384 2022-09-10T04:30:17.491 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2048 offset 4197888 len 16384 2022-09-10T04:30:17.491 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2048 offset 4197888 len 16384 2022-09-10T04:30:17.491 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4190720 len 1024 2022-09-10T04:30:17.492 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 1024 2022-09-10T04:30:17.492 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4190720 len 1024 2022-09-10T04:30:17.492 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 1024 2022-09-10T04:30:17.492 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4190720 len 2048 2022-09-10T04:30:17.493 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 2048 2022-09-10T04:30:17.493 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4190720 len 2048 2022-09-10T04:30:17.493 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 2048 2022-09-10T04:30:17.493 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4190720 len 4096 2022-09-10T04:30:17.494 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 4096 2022-09-10T04:30:17.494 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4190720 len 4096 2022-09-10T04:30:17.494 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 4096 2022-09-10T04:30:17.494 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4190720 len 8192 2022-09-10T04:30:17.495 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 8192 2022-09-10T04:30:17.495 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4190720 len 8192 2022-09-10T04:30:17.495 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 8192 2022-09-10T04:30:17.496 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4190720 len 16384 2022-09-10T04:30:17.496 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 16384 2022-09-10T04:30:17.496 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4190720 len 16384 2022-09-10T04:30:17.497 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4190720 len 16384 2022-09-10T04:30:17.497 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191232 len 1024 2022-09-10T04:30:17.497 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 1024 2022-09-10T04:30:17.497 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191232 len 1024 2022-09-10T04:30:17.498 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 1024 2022-09-10T04:30:17.498 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191232 len 2048 2022-09-10T04:30:17.498 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 2048 2022-09-10T04:30:17.499 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191232 len 2048 2022-09-10T04:30:17.499 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 2048 2022-09-10T04:30:17.499 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191232 len 4096 2022-09-10T04:30:17.501 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 4096 2022-09-10T04:30:17.501 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191232 len 4096 2022-09-10T04:30:17.501 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 4096 2022-09-10T04:30:17.501 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191232 len 8192 2022-09-10T04:30:17.502 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 8192 2022-09-10T04:30:17.502 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191232 len 8192 2022-09-10T04:30:17.502 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 8192 2022-09-10T04:30:17.502 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191232 len 16384 2022-09-10T04:30:17.502 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 16384 2022-09-10T04:30:17.503 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191232 len 16384 2022-09-10T04:30:17.503 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191232 len 16384 2022-09-10T04:30:17.503 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191744 len 1024 2022-09-10T04:30:17.503 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 1024 2022-09-10T04:30:17.504 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191744 len 1024 2022-09-10T04:30:17.504 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 1024 2022-09-10T04:30:17.504 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191744 len 2048 2022-09-10T04:30:17.504 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 2048 2022-09-10T04:30:17.504 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191744 len 2048 2022-09-10T04:30:17.505 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 2048 2022-09-10T04:30:17.505 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191744 len 4096 2022-09-10T04:30:17.505 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 4096 2022-09-10T04:30:17.505 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191744 len 4096 2022-09-10T04:30:17.505 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 4096 2022-09-10T04:30:17.506 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191744 len 8192 2022-09-10T04:30:17.506 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 8192 2022-09-10T04:30:17.506 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191744 len 8192 2022-09-10T04:30:17.506 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 8192 2022-09-10T04:30:17.507 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4191744 len 16384 2022-09-10T04:30:17.507 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 16384 2022-09-10T04:30:17.507 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4191744 len 16384 2022-09-10T04:30:17.507 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4191744 len 16384 2022-09-10T04:30:17.507 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192256 len 1024 2022-09-10T04:30:17.508 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 1024 2022-09-10T04:30:17.508 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192256 len 1024 2022-09-10T04:30:17.508 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 1024 2022-09-10T04:30:17.508 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192256 len 2048 2022-09-10T04:30:17.508 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 2048 2022-09-10T04:30:17.509 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192256 len 2048 2022-09-10T04:30:17.509 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 2048 2022-09-10T04:30:17.509 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192256 len 4096 2022-09-10T04:30:17.509 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 4096 2022-09-10T04:30:17.509 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192256 len 4096 2022-09-10T04:30:17.510 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 4096 2022-09-10T04:30:17.510 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192256 len 8192 2022-09-10T04:30:17.510 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 8192 2022-09-10T04:30:17.510 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192256 len 8192 2022-09-10T04:30:17.511 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 8192 2022-09-10T04:30:17.511 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192256 len 16384 2022-09-10T04:30:17.511 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 16384 2022-09-10T04:30:19.004 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192256 len 16384 2022-09-10T04:30:19.005 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192256 len 16384 2022-09-10T04:30:19.005 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192768 len 1024 2022-09-10T04:30:19.005 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 1024 2022-09-10T04:30:19.006 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192768 len 1024 2022-09-10T04:30:19.006 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 1024 2022-09-10T04:30:19.006 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192768 len 2048 2022-09-10T04:30:19.007 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 2048 2022-09-10T04:30:19.007 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192768 len 2048 2022-09-10T04:30:19.007 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 2048 2022-09-10T04:30:19.007 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192768 len 4096 2022-09-10T04:30:19.008 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 4096 2022-09-10T04:30:19.008 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192768 len 4096 2022-09-10T04:30:19.008 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 4096 2022-09-10T04:30:19.008 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192768 len 8192 2022-09-10T04:30:19.009 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 8192 2022-09-10T04:30:19.009 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192768 len 8192 2022-09-10T04:30:19.009 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 8192 2022-09-10T04:30:19.010 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4192768 len 16384 2022-09-10T04:30:19.010 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 16384 2022-09-10T04:30:19.011 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4192768 len 16384 2022-09-10T04:30:19.011 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4192768 len 16384 2022-09-10T04:30:19.011 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193280 len 1024 2022-09-10T04:30:19.011 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 1024 2022-09-10T04:30:19.012 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193280 len 1024 2022-09-10T04:30:19.012 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 1024 2022-09-10T04:30:19.012 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193280 len 2048 2022-09-10T04:30:19.012 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 2048 2022-09-10T04:30:19.012 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193280 len 2048 2022-09-10T04:30:19.013 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 2048 2022-09-10T04:30:19.013 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193280 len 4096 2022-09-10T04:30:19.013 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 4096 2022-09-10T04:30:19.013 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193280 len 4096 2022-09-10T04:30:19.014 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 4096 2022-09-10T04:30:19.014 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193280 len 8192 2022-09-10T04:30:19.014 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 8192 2022-09-10T04:30:19.014 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193280 len 8192 2022-09-10T04:30:19.015 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 8192 2022-09-10T04:30:19.015 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193280 len 16384 2022-09-10T04:30:19.016 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 16384 2022-09-10T04:30:19.016 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193280 len 16384 2022-09-10T04:30:19.016 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193280 len 16384 2022-09-10T04:30:19.016 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193792 len 1024 2022-09-10T04:30:19.017 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 1024 2022-09-10T04:30:19.017 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193792 len 1024 2022-09-10T04:30:19.017 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 1024 2022-09-10T04:30:19.018 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193792 len 2048 2022-09-10T04:30:19.018 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 2048 2022-09-10T04:30:19.018 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193792 len 2048 2022-09-10T04:30:19.018 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 2048 2022-09-10T04:30:19.018 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193792 len 4096 2022-09-10T04:30:19.019 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 4096 2022-09-10T04:30:19.019 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193792 len 4096 2022-09-10T04:30:19.019 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 4096 2022-09-10T04:30:19.020 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193792 len 8192 2022-09-10T04:30:19.020 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 8192 2022-09-10T04:30:19.020 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193792 len 8192 2022-09-10T04:30:19.021 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 8192 2022-09-10T04:30:19.021 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4193792 len 16384 2022-09-10T04:30:19.021 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 16384 2022-09-10T04:30:19.022 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4193792 len 16384 2022-09-10T04:30:19.022 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4193792 len 16384 2022-09-10T04:30:19.022 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194304 len 1024 2022-09-10T04:30:19.022 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 1024 2022-09-10T04:30:19.023 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194304 len 1024 2022-09-10T04:30:19.023 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 1024 2022-09-10T04:30:19.023 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194304 len 2048 2022-09-10T04:30:19.023 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 2048 2022-09-10T04:30:19.023 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194304 len 2048 2022-09-10T04:30:19.024 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 2048 2022-09-10T04:30:19.024 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194304 len 4096 2022-09-10T04:30:19.024 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 4096 2022-09-10T04:30:19.024 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194304 len 4096 2022-09-10T04:30:19.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 4096 2022-09-10T04:30:19.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194304 len 8192 2022-09-10T04:30:19.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 8192 2022-09-10T04:30:19.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194304 len 8192 2022-09-10T04:30:19.025 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 8192 2022-09-10T04:30:19.026 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194304 len 16384 2022-09-10T04:30:19.026 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 16384 2022-09-10T04:30:19.026 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194304 len 16384 2022-09-10T04:30:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:18 smithi104 ceph-mon[97044]: pgmap v50: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 504 KiB/s wr, 193 op/s 2022-09-10T04:30:19.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:18 smithi081 ceph-mon[90133]: pgmap v50: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 504 KiB/s wr, 193 op/s 2022-09-10T04:30:19.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.106+0000 7f6554bff700 1 -- 172.21.15.81:0/1899285059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100280 msgr2=0x7f6550100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.106+0000 7f6554bff700 1 --2- 172.21.15.81:0/1899285059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100280 0x7f6550100650 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540009b50 comp rx=0 tx=0).stop 2022-09-10T04:30:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 -- 172.21.15.81:0/1899285059 shutdown_connections 2022-09-10T04:30:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 --2- 172.21.15.81:0/1899285059 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550105380 0x7f655010cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 --2- 172.21.15.81:0/1899285059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100280 0x7f6550100650 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 --2- 172.21.15.81:0/1899285059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550100c20 0x7f6550104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 -- 172.21.15.81:0/1899285059 >> 172.21.15.81:0/1899285059 conn(0x7f65500fbba0 msgr2=0x7f65500fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 -- 172.21.15.81:0/1899285059 shutdown_connections 2022-09-10T04:30:19.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.107+0000 7f6554bff700 1 -- 172.21.15.81:0/1899285059 wait complete. 2022-09-10T04:30:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.108+0000 7f6554bff700 1 Processor -- start 2022-09-10T04:30:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.108+0000 7f6554bff700 1 -- start start 2022-09-10T04:30:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f6554bff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550100280 0x7f6550195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f6554bff700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100c20 0x7f6550196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f6554bff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 0x7f655019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f6554bff700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655019af00 con 0x7f6550100c20 2022-09-10T04:30:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f6554bff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655019b070 con 0x7f6550105380 2022-09-10T04:30:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f6554bff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655019b230 con 0x7f6550100280 2022-09-10T04:30:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f654ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 0x7f655019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f654e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550100280 0x7f6550195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f654ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 0x7f655019a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56304/0 (socket says 172.21.15.81:56304) 2022-09-10T04:30:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f654ed9d700 1 -- 172.21.15.81:0/455842423 learned_addr learned my addr 172.21.15.81:0/455842423 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.109+0000 7f654dd9b700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100c20 0x7f6550196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f654ed9d700 1 -- 172.21.15.81:0/455842423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550100280 msgr2=0x7f6550195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f654ed9d700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550100280 0x7f6550195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f654ed9d700 1 -- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100c20 msgr2=0x7f6550196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f654ed9d700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100c20 0x7f6550196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f654ed9d700 1 -- 172.21.15.81:0/455842423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540009470 con 0x7f6550105380 2022-09-10T04:30:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f654ed9d700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 0x7f655019a870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f654800ebf0 tx=0x7f654800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f653f7fe700 1 -- 172.21.15.81:0/455842423 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480099c0 con 0x7f6550105380 2022-09-10T04:30:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655019b510 con 0x7f6550105380 2022-09-10T04:30:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65501a2d00 con 0x7f6550105380 2022-09-10T04:30:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f653f7fe700 1 -- 172.21.15.81:0/455842423 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f654800cd40 con 0x7f6550105380 2022-09-10T04:30:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.110+0000 7f653f7fe700 1 -- 172.21.15.81:0/455842423 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6548005900 con 0x7f6550105380 2022-09-10T04:30:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.115+0000 7f653f7fe700 1 -- 172.21.15.81:0/455842423 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 23) v1 ==== 127775+0+0 (secure 0 0 0) 0x7f654801e020 con 0x7f6550105380 2022-09-10T04:30:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.117+0000 7f65681b1b80 1 Processor -- start 2022-09-10T04:30:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.117+0000 7f65681b1b80 1 -- start start 2022-09-10T04:30:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f653f7fe700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f65380990f0 0x7f653809b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x562851938320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x5628519324a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x562851932e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f65681b1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285181d070 con 0x562851937f50 2022-09-10T04:30:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f65681b1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628518879d0 con 0x562851938860 2022-09-10T04:30:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f65681b1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x562851881170 con 0x5628519329e0 2022-09-10T04:30:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f653f7fe700 1 -- 172.21.15.81:0/455842423 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f6548014070 con 0x7f6550105380 2022-09-10T04:30:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f654dd9b700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f65380990f0 0x7f653809b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f654e59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x562851938320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f654e59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x562851938320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43206/0 (socket says 172.21.15.81:43206) 2022-09-10T04:30:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f654e59c700 1 -- 172.21.15.81:0/3647586020 learned_addr learned my addr 172.21.15.81:0/3647586020 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.118+0000 7f654dd9b700 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x5628519324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654ed9d700 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x562851932e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654e59c700 1 -- 172.21.15.81:0/3647586020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 msgr2=0x562851932e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654e59c700 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x562851932e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654e59c700 1 -- 172.21.15.81:0/3647586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 msgr2=0x5628519324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654e59c700 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x5628519324a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654e59c700 1 -- 172.21.15.81:0/3647586020 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628517095d0 con 0x562851937f50 2022-09-10T04:30:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654dd9b700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f65380990f0 0x7f653809b5a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6544009360 tx=0x7f65440099a0 comp rx=0 tx=0).ready entity=mgr.25379 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.119+0000 7f654e59c700 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x562851938320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c6d11bca7c389b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.120+0000 7f653cff9700 1 -- 172.21.15.81:0/3647586020 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654001d070 con 0x562851937f50 2022-09-10T04:30:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.120+0000 7f653cff9700 1 -- 172.21.15.81:0/3647586020 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f654001e040 con 0x562851937f50 2022-09-10T04:30:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.120+0000 7f653cff9700 1 -- 172.21.15.81:0/3647586020 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540004500 con 0x562851937f50 2022-09-10T04:30:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.120+0000 7f65681b1b80 1 -- 172.21.15.81:0/3647586020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 msgr2=0x562851938320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.120+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x562851938320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6540009840 tx=0x7f6540005a80 comp rx=0 tx=0).stop 2022-09-10T04:30:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 -- 172.21.15.81:0/3647586020 shutdown_connections 2022-09-10T04:30:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x562851932e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x562851938320 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3647586020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x5628519324a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 -- 172.21.15.81:0/3647586020 >> 172.21.15.81:0/3647586020 conn(0x5628517926b0 msgr2=0x56285192ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 -- 172.21.15.81:0/3647586020 shutdown_connections 2022-09-10T04:30:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.121+0000 7f65681b1b80 1 -- 172.21.15.81:0/3647586020 wait complete. 2022-09-10T04:30:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 -- start start 2022-09-10T04:30:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 Processor -- start 2022-09-10T04:30:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x56285193e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x56285193b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x56285193bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285176f0b0 con 0x562851937f50 2022-09-10T04:30:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285193c190 con 0x562851938860 2022-09-10T04:30:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.122+0000 7f65681b1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285193c350 con 0x5628519329e0 2022-09-10T04:30:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f65681b1b80 10 client.0 _ll_register_callbacks cb 0x56285182c360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:30:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x56285193e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654dd9b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x56285193b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x56285193e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.81:53244/0 (socket says 172.21.15.81:53244) 2022-09-10T04:30:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 -- 172.21.15.81:0/1164191865 learned_addr learned my addr 172.21.15.81:0/1164191865 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654ed9d700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x56285193bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 -- 172.21.15.81:0/1164191865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 msgr2=0x56285193bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x56285193bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 -- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 msgr2=0x56285193b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x56285193b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654e59c700 1 -- 172.21.15.81:0/1164191865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655010d6e0 con 0x5628519329e0 2022-09-10T04:30:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654ed9d700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x56285193bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:30:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.123+0000 7f654dd9b700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x56285193b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2022-09-10T04:30:19.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f654e59c700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x56285193e490 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6540009b90 tx=0x7f654000bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35c968e49ce5fa05 server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f652dffb700 10 client.0 ms_handle_connect on v2:172.21.15.110:3300/0 2022-09-10T04:30:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f652dffb700 1 -- 172.21.15.81:0/1164191865 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654001d070 con 0x5628519329e0 2022-09-10T04:30:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f652dffb700 1 -- 172.21.15.81:0/1164191865 <== mon.1 v2:172.21.15.110:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f6540005250 con 0x5628519329e0 2022-09-10T04:30:19.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5628519256e0 con 0x5628519329e0 2022-09-10T04:30:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f652dffb700 1 -- 172.21.15.81:0/1164191865 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f654000b480 con 0x5628519329e0 2022-09-10T04:30:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f65681b1b80 10 client.25588 fetch_fsmap learned FSMap version 28 2022-09-10T04:30:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5628516b5d80 con 0x5628519329e0 2022-09-10T04:30:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.124+0000 7f652dffb700 1 -- 172.21.15.81:0/1164191865 <== mon.1 v2:172.21.15.110:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x5628519256e0 con 0x5628519329e0 2022-09-10T04:30:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f652dffb700 1 -- 172.21.15.81:0/1164191865 <== mon.1 v2:172.21.15.110:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f654000f4f0 con 0x5628519329e0 2022-09-10T04:30:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f65681b1b80 10 client.25588 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:30:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f65681b1b80 10 client.25588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:30:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f65681b1b80 20 client.25588 populate_metadata read hostname 'smithi081' 2022-09-10T04:30:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285194bcc0 0x56285194e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f65681b1b80 4 client.25588 mds_command: new command op to 24577 tid=0[{"prefix": "get_command_descriptions"}] 2022-09-10T04:30:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628519024b0 con 0x56285194bcc0 2022-09-10T04:30:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.125+0000 7f654dd9b700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285194bcc0 0x56285194e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.126+0000 7f654dd9b700 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285194bcc0 0x56285194e0a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e62306f672b3e4f4 server_cookie=c24eb0afef2c3907 in_seq=0 out_seq=0 2022-09-10T04:30:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.126+0000 7f652dffb700 10 client.25588 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.126+0000 7f652dffb700 1 -- 172.21.15.81:0/1164191865 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17402 (crc 0 0 0) 0x5628519024b0 con 0x56285194bcc0 2022-09-10T04:30:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.126+0000 7f652dffb700 10 client.25588 handle_command_reply: tid=0 2022-09-10T04:30:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.126+0000 7f65681b1b80 1 client.25588 shutdown 2022-09-10T04:30:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 msgr2=0x56285193e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x56285193e490 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6540009b90 tx=0x7f654000bdb0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 shutdown_connections 2022-09-10T04:30:19.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285194bcc0 0x56285194e0a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851938860 0x56285193bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x5628519329e0 0x56285193e490 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f652dffb700 0 client.25588 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:19.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1164191865 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562851937f50 0x56285193b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 >> 172.21.15.81:0/1164191865 conn(0x5628517926b0 msgr2=0x562851931490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 shutdown_connections 2022-09-10T04:30:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 1 -- 172.21.15.81:0/1164191865 wait complete. 2022-09-10T04:30:19.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.127+0000 7f65681b1b80 20 client.25588 trim_cache size 0 max 16384 2022-09-10T04:30:19.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:18 smithi110 ceph-mon[97633]: pgmap v50: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 504 KiB/s wr, 193 op/s 2022-09-10T04:30:19.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.143+0000 7f65681b1b80 1 Processor -- start 2022-09-10T04:30:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 -- start start 2022-09-10T04:30:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x5628519324a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x56285193aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x56285193dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562851927800 con 0x5628519329e0 2022-09-10T04:30:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x562851937b90 con 0x562851937f50 2022-09-10T04:30:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f65681b1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x562851939c90 con 0x562851938860 2022-09-10T04:30:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f654dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x56285193aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f654dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x56285193aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56324/0 (socket says 172.21.15.81:56324) 2022-09-10T04:30:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.144+0000 7f654dd9b700 1 -- 172.21.15.81:0/3301042369 learned_addr learned my addr 172.21.15.81:0/3301042369 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654e59c700 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x5628519324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654ed9d700 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x56285193dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654dd9b700 1 -- 172.21.15.81:0/3301042369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 msgr2=0x56285193dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654dd9b700 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x56285193dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654dd9b700 1 -- 172.21.15.81:0/3301042369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 msgr2=0x5628519324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654dd9b700 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x5628519324a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654dd9b700 1 -- 172.21.15.81:0/3301042369 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628516b4250 con 0x562851937f50 2022-09-10T04:30:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654ed9d700 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x56285193dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2022-09-10T04:30:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f654dd9b700 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x56285193aef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6544015010 tx=0x7f6544005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9c0960b3ea78bc server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f653d7fa700 1 -- 172.21.15.81:0/3301042369 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544026040 con 0x562851937f50 2022-09-10T04:30:19.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f653d7fa700 1 -- 172.21.15.81:0/3301042369 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f6544009620 con 0x562851937f50 2022-09-10T04:30:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.145+0000 7f653d7fa700 1 -- 172.21.15.81:0/3301042369 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544003d00 con 0x562851937f50 2022-09-10T04:30:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 -- 172.21.15.81:0/3301042369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 msgr2=0x56285193aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x56285193aef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6544015010 tx=0x7f6544005ea0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 -- 172.21.15.81:0/3301042369 shutdown_connections 2022-09-10T04:30:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x56285193dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x5628519324a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 --2- 172.21.15.81:0/3301042369 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x56285193aef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 -- 172.21.15.81:0/3301042369 >> 172.21.15.81:0/3301042369 conn(0x5628517926b0 msgr2=0x56285193b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 -- 172.21.15.81:0/3301042369 shutdown_connections 2022-09-10T04:30:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.146+0000 7f65681b1b80 1 -- 172.21.15.81:0/3301042369 wait complete. 2022-09-10T04:30:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 -- start start 2022-09-10T04:30:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 Processor -- start 2022-09-10T04:30:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x562851942f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x562851941250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x562851941790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562851941e60 con 0x5628519329e0 2022-09-10T04:30:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x562851941fd0 con 0x562851937f50 2022-09-10T04:30:19.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.147+0000 7f65681b1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x56285199d470 con 0x562851938860 2022-09-10T04:30:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f65681b1b80 10 client.0 _ll_register_callbacks cb 0x56285182c360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-09-10T04:30:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654e59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x562851942f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654dd9b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x562851941250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654e59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x562851942f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:43230/0 (socket says 172.21.15.81:43230) 2022-09-10T04:30:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654e59c700 1 -- 172.21.15.81:0/1590076140 learned_addr learned my addr 172.21.15.81:0/1590076140 (peer_addr_for_me v2:172.21.15.81:0/0) 2022-09-10T04:30:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654ed9d700 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x562851941790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654dd9b700 1 -- 172.21.15.81:0/1590076140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 msgr2=0x562851941790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654dd9b700 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x562851941790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654dd9b700 1 -- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 msgr2=0x562851942f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-09-10T04:30:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654dd9b700 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x562851942f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.148+0000 7f654dd9b700 1 -- 172.21.15.81:0/1590076140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654401b040 con 0x562851937f50 2022-09-10T04:30:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f654dd9b700 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x562851941250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6544015010 tx=0x7f6544009d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ddcfa9a164861b server_cookie=0 in_seq=0 out_seq=0 2022-09-10T04:30:19.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f652dffb700 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2022-09-10T04:30:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f652dffb700 1 -- 172.21.15.81:0/1590076140 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544026070 con 0x562851937f50 2022-09-10T04:30:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f654001e040 con 0x562851937f50 2022-09-10T04:30:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f652dffb700 1 -- 172.21.15.81:0/1590076140 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9332+0+0 (secure 0 0 0) 0x7f654402ca30 con 0x562851937f50 2022-09-10T04:30:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f652dffb700 1 -- 172.21.15.81:0/1590076140 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f654402cbf0 con 0x562851937f50 2022-09-10T04:30:19.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f65681b1b80 10 client.25586 fetch_fsmap learned FSMap version 28 2022-09-10T04:30:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.149+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562851782a20 con 0x562851937f50 2022-09-10T04:30:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f652dffb700 1 -- 172.21.15.81:0/1590076140 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(26 keys) v1 ==== 1080+0+0 (secure 0 0 0) 0x7f654001e040 con 0x562851937f50 2022-09-10T04:30:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f652dffb700 1 -- 172.21.15.81:0/1590076140 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 28) v1 ==== 7456+0+0 (secure 0 0 0) 0x7f6544033020 con 0x562851937f50 2022-09-10T04:30:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f65681b1b80 10 client.25586 fetch_fsmap finished waiting for FSMap version 28 2022-09-10T04:30:19.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f65681b1b80 10 client.25586 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2022-09-10T04:30:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f65681b1b80 20 client.25586 populate_metadata read hostname 'smithi081' 2022-09-10T04:30:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285199eb20 0x5628519a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2022-09-10T04:30:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f65681b1b80 4 client.25586 mds_command: new command op to 24577 tid=0[{"prefix": "flush journal"}] 2022-09-10T04:30:19.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 --> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56285187a6e0 con 0x56285199eb20 2022-09-10T04:30:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.150+0000 7f654e59c700 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285199eb20 0x5628519a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2022-09-10T04:30:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.151+0000 7f654e59c700 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285199eb20 0x5628519a0f30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3d5529704f0e07c server_cookie=67466669e0c3492c in_seq=0 out_seq=0 2022-09-10T04:30:19.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.151+0000 7f652dffb700 10 client.25586 ms_handle_connect on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f652dffb700 1 -- 172.21.15.81:0/1590076140 <== mds.0 v2:172.21.15.81:6840/953424075 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56285187a6e0 con 0x56285199eb20 2022-09-10T04:30:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f652dffb700 10 client.25586 handle_command_reply: tid=0 2022-09-10T04:30:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 client.25586 shutdown 2022-09-10T04:30:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 msgr2=0x562851941250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x562851941250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6544015010 tx=0x7f6544009d80 comp rx=0 tx=0).stop 2022-09-10T04:30:19.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 shutdown_connections 2022-09-10T04:30:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x562851938860 0x562851941790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5628519329e0 0x562851942f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.81:6840/953424075,v1:172.21.15.81:6841/953424075] conn(0x56285199eb20 0x5628519a0f30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 --2- 172.21.15.81:0/1590076140 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x562851937f50 0x562851941250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f652dffb700 0 client.25586 ms_handle_reset on v2:172.21.15.81:6840/953424075 2022-09-10T04:30:19.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.362+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 >> 172.21.15.81:0/1590076140 conn(0x5628517926b0 msgr2=0x562851936800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:19.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 shutdown_connections 2022-09-10T04:30:19.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f65681b1b80 1 -- 172.21.15.81:0/1590076140 wait complete. 2022-09-10T04:30:19.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f65681b1b80 20 client.25586 trim_cache size 0 max 16384 2022-09-10T04:30:19.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:{ 2022-09-10T04:30:19.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "message": "", 2022-09-10T04:30:19.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout: "return_code": 0 2022-09-10T04:30:19.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stdout:} 2022-09-10T04:30:19.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f65380990f0 msgr2=0x7f653809b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f6554bff700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f65380990f0 0x7f653809b5a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6544009360 tx=0x7f65440099a0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 msgr2=0x7f655019a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-09-10T04:30:19.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.363+0000 7f6554bff700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 0x7f655019a870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f654800ebf0 tx=0x7f654800ef00 comp rx=0 tx=0).stop 2022-09-10T04:30:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 shutdown_connections 2022-09-10T04:30:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:6800/3853788614,v1:172.21.15.81:6801/3853788614] conn(0x7f65380990f0 0x7f653809b5a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6550105380 0x7f655019a870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6550100280 0x7f6550195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 --2- 172.21.15.81:0/455842423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6550100c20 0x7f6550196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2022-09-10T04:30:19.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 >> 172.21.15.81:0/455842423 conn(0x7f65500fbba0 msgr2=0x7f65500fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-09-10T04:30:19.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 shutdown_connections 2022-09-10T04:30:19.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2022-09-10T04:30:19.364+0000 7f6554bff700 1 -- 172.21.15.81:0/455842423 wait complete. 2022-09-10T04:30:19.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194304 len 16384 2022-09-10T04:30:19.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194816 len 1024 2022-09-10T04:30:19.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 1024 2022-09-10T04:30:19.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194816 len 1024 2022-09-10T04:30:19.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 1024 2022-09-10T04:30:19.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194816 len 2048 2022-09-10T04:30:19.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 2048 2022-09-10T04:30:19.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194816 len 2048 2022-09-10T04:30:19.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 2048 2022-09-10T04:30:19.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194816 len 4096 2022-09-10T04:30:19.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 4096 2022-09-10T04:30:19.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194816 len 4096 2022-09-10T04:30:19.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 4096 2022-09-10T04:30:19.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194816 len 8192 2022-09-10T04:30:19.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 8192 2022-09-10T04:30:19.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194816 len 8192 2022-09-10T04:30:19.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 8192 2022-09-10T04:30:19.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4194816 len 16384 2022-09-10T04:30:19.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 16384 2022-09-10T04:30:19.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4194816 len 16384 2022-09-10T04:30:19.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4194816 len 16384 2022-09-10T04:30:19.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195328 len 1024 2022-09-10T04:30:19.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 1024 2022-09-10T04:30:19.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195328 len 1024 2022-09-10T04:30:19.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 1024 2022-09-10T04:30:19.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195328 len 2048 2022-09-10T04:30:19.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 2048 2022-09-10T04:30:19.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195328 len 2048 2022-09-10T04:30:19.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 2048 2022-09-10T04:30:19.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195328 len 4096 2022-09-10T04:30:19.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 4096 2022-09-10T04:30:19.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195328 len 4096 2022-09-10T04:30:19.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 4096 2022-09-10T04:30:19.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195328 len 8192 2022-09-10T04:30:19.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 8192 2022-09-10T04:30:19.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195328 len 8192 2022-09-10T04:30:19.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 8192 2022-09-10T04:30:19.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195328 len 16384 2022-09-10T04:30:19.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 16384 2022-09-10T04:30:19.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195328 len 16384 2022-09-10T04:30:19.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195328 len 16384 2022-09-10T04:30:19.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195840 len 1024 2022-09-10T04:30:19.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 1024 2022-09-10T04:30:19.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195840 len 1024 2022-09-10T04:30:19.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 1024 2022-09-10T04:30:19.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195840 len 2048 2022-09-10T04:30:19.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 2048 2022-09-10T04:30:19.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195840 len 2048 2022-09-10T04:30:19.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 2048 2022-09-10T04:30:19.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195840 len 4096 2022-09-10T04:30:19.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 4096 2022-09-10T04:30:19.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195840 len 4096 2022-09-10T04:30:19.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 4096 2022-09-10T04:30:19.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195840 len 8192 2022-09-10T04:30:19.795 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 8192 2022-09-10T04:30:19.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195840 len 8192 2022-09-10T04:30:19.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 8192 2022-09-10T04:30:19.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4195840 len 16384 2022-09-10T04:30:19.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 16384 2022-09-10T04:30:19.796 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4195840 len 16384 2022-09-10T04:30:19.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4195840 len 16384 2022-09-10T04:30:19.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196352 len 1024 2022-09-10T04:30:19.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 1024 2022-09-10T04:30:19.797 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196352 len 1024 2022-09-10T04:30:19.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 1024 2022-09-10T04:30:19.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196352 len 2048 2022-09-10T04:30:19.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 2048 2022-09-10T04:30:19.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196352 len 2048 2022-09-10T04:30:19.798 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 2048 2022-09-10T04:30:19.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196352 len 4096 2022-09-10T04:30:19.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 4096 2022-09-10T04:30:19.799 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196352 len 4096 2022-09-10T04:30:19.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 4096 2022-09-10T04:30:19.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196352 len 8192 2022-09-10T04:30:19.800 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 8192 2022-09-10T04:30:19.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196352 len 8192 2022-09-10T04:30:19.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 8192 2022-09-10T04:30:19.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196352 len 16384 2022-09-10T04:30:19.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 16384 2022-09-10T04:30:19.801 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196352 len 16384 2022-09-10T04:30:19.802 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196352 len 16384 2022-09-10T04:30:20.659 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196864 len 1024 2022-09-10T04:30:20.660 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 1024 2022-09-10T04:30:20.660 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196864 len 1024 2022-09-10T04:30:20.660 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 1024 2022-09-10T04:30:20.661 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196864 len 2048 2022-09-10T04:30:20.661 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 2048 2022-09-10T04:30:20.661 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196864 len 2048 2022-09-10T04:30:20.661 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 2048 2022-09-10T04:30:20.662 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196864 len 4096 2022-09-10T04:30:20.662 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 4096 2022-09-10T04:30:20.662 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196864 len 4096 2022-09-10T04:30:20.663 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 4096 2022-09-10T04:30:20.663 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196864 len 8192 2022-09-10T04:30:20.663 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 8192 2022-09-10T04:30:20.664 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196864 len 8192 2022-09-10T04:30:20.664 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 8192 2022-09-10T04:30:20.664 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4196864 len 16384 2022-09-10T04:30:20.664 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 16384 2022-09-10T04:30:20.665 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4196864 len 16384 2022-09-10T04:30:20.665 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4196864 len 16384 2022-09-10T04:30:20.665 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197376 len 1024 2022-09-10T04:30:20.665 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 1024 2022-09-10T04:30:20.666 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197376 len 1024 2022-09-10T04:30:20.667 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 1024 2022-09-10T04:30:20.667 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197376 len 2048 2022-09-10T04:30:20.667 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 2048 2022-09-10T04:30:20.668 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197376 len 2048 2022-09-10T04:30:20.668 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 2048 2022-09-10T04:30:20.668 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197376 len 4096 2022-09-10T04:30:20.668 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 4096 2022-09-10T04:30:20.669 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197376 len 4096 2022-09-10T04:30:20.669 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 4096 2022-09-10T04:30:20.669 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197376 len 8192 2022-09-10T04:30:20.670 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 8192 2022-09-10T04:30:20.670 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197376 len 8192 2022-09-10T04:30:20.670 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 8192 2022-09-10T04:30:20.670 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197376 len 16384 2022-09-10T04:30:20.671 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 16384 2022-09-10T04:30:20.671 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197376 len 16384 2022-09-10T04:30:20.671 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197376 len 16384 2022-09-10T04:30:20.672 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197888 len 1024 2022-09-10T04:30:20.672 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 1024 2022-09-10T04:30:20.672 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197888 len 1024 2022-09-10T04:30:20.672 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 1024 2022-09-10T04:30:20.673 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197888 len 2048 2022-09-10T04:30:20.673 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 2048 2022-09-10T04:30:20.673 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197888 len 2048 2022-09-10T04:30:20.673 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 2048 2022-09-10T04:30:20.674 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197888 len 4096 2022-09-10T04:30:20.674 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 4096 2022-09-10T04:30:20.674 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197888 len 4096 2022-09-10T04:30:20.674 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 4096 2022-09-10T04:30:20.675 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197888 len 8192 2022-09-10T04:30:20.675 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 8192 2022-09-10T04:30:20.675 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197888 len 8192 2022-09-10T04:30:20.675 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 8192 2022-09-10T04:30:20.675 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 2560 offset 4197888 len 16384 2022-09-10T04:30:20.676 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 16384 2022-09-10T04:30:20.676 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 2560 offset 4197888 len 16384 2022-09-10T04:30:20.676 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 2560 offset 4197888 len 16384 2022-09-10T04:30:20.676 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4190720 len 1024 2022-09-10T04:30:20.677 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 1024 2022-09-10T04:30:20.677 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4190720 len 1024 2022-09-10T04:30:20.677 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 1024 2022-09-10T04:30:20.677 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4190720 len 2048 2022-09-10T04:30:20.678 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 2048 2022-09-10T04:30:20.678 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4190720 len 2048 2022-09-10T04:30:20.678 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 2048 2022-09-10T04:30:20.678 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4190720 len 4096 2022-09-10T04:30:20.679 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 4096 2022-09-10T04:30:20.679 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4190720 len 4096 2022-09-10T04:30:20.679 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 4096 2022-09-10T04:30:20.679 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4190720 len 8192 2022-09-10T04:30:20.680 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 8192 2022-09-10T04:30:20.680 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4190720 len 8192 2022-09-10T04:30:20.680 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 8192 2022-09-10T04:30:20.680 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4190720 len 16384 2022-09-10T04:30:20.681 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 16384 2022-09-10T04:30:20.681 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4190720 len 16384 2022-09-10T04:30:20.681 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4190720 len 16384 2022-09-10T04:30:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:20 smithi104 ceph-mon[97044]: pgmap v51: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 504 KiB/s wr, 193 op/s 2022-09-10T04:30:21.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:20 smithi081 ceph-mon[90133]: pgmap v51: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 504 KiB/s wr, 193 op/s 2022-09-10T04:30:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:20 smithi110 ceph-mon[97633]: pgmap v51: 129 pgs: 129 active+clean; 582 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 504 KiB/s wr, 193 op/s 2022-09-10T04:30:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:22 smithi104 ceph-mon[97044]: pgmap v52: 129 pgs: 129 active+clean; 881 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s rd, 768 KiB/s wr, 338 op/s 2022-09-10T04:30:23.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:22 smithi081 ceph-mon[90133]: pgmap v52: 129 pgs: 129 active+clean; 881 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s rd, 768 KiB/s wr, 338 op/s 2022-09-10T04:30:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:22 smithi110 ceph-mon[97633]: pgmap v52: 129 pgs: 129 active+clean; 881 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s rd, 768 KiB/s wr, 338 op/s 2022-09-10T04:30:23.523 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191232 len 1024 2022-09-10T04:30:23.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 1024 2022-09-10T04:30:23.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191232 len 1024 2022-09-10T04:30:23.524 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 1024 2022-09-10T04:30:23.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191232 len 2048 2022-09-10T04:30:23.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 2048 2022-09-10T04:30:23.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191232 len 2048 2022-09-10T04:30:23.525 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 2048 2022-09-10T04:30:23.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191232 len 4096 2022-09-10T04:30:23.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 4096 2022-09-10T04:30:23.526 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191232 len 4096 2022-09-10T04:30:23.527 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 4096 2022-09-10T04:30:23.527 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191232 len 8192 2022-09-10T04:30:23.527 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 8192 2022-09-10T04:30:23.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191232 len 8192 2022-09-10T04:30:23.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 8192 2022-09-10T04:30:23.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191232 len 16384 2022-09-10T04:30:23.528 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 16384 2022-09-10T04:30:23.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191232 len 16384 2022-09-10T04:30:23.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191232 len 16384 2022-09-10T04:30:23.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191744 len 1024 2022-09-10T04:30:23.529 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 1024 2022-09-10T04:30:23.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191744 len 1024 2022-09-10T04:30:23.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 1024 2022-09-10T04:30:23.530 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191744 len 2048 2022-09-10T04:30:23.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 2048 2022-09-10T04:30:23.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191744 len 2048 2022-09-10T04:30:23.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 2048 2022-09-10T04:30:23.531 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191744 len 4096 2022-09-10T04:30:23.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 4096 2022-09-10T04:30:23.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191744 len 4096 2022-09-10T04:30:23.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 4096 2022-09-10T04:30:23.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191744 len 8192 2022-09-10T04:30:23.532 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 8192 2022-09-10T04:30:23.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191744 len 8192 2022-09-10T04:30:23.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 8192 2022-09-10T04:30:23.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4191744 len 16384 2022-09-10T04:30:23.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 16384 2022-09-10T04:30:23.533 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4191744 len 16384 2022-09-10T04:30:23.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4191744 len 16384 2022-09-10T04:30:23.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192256 len 1024 2022-09-10T04:30:23.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 1024 2022-09-10T04:30:23.534 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192256 len 1024 2022-09-10T04:30:23.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 1024 2022-09-10T04:30:23.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192256 len 2048 2022-09-10T04:30:23.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 2048 2022-09-10T04:30:23.535 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192256 len 2048 2022-09-10T04:30:23.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 2048 2022-09-10T04:30:23.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192256 len 4096 2022-09-10T04:30:23.536 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 4096 2022-09-10T04:30:23.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192256 len 4096 2022-09-10T04:30:23.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 4096 2022-09-10T04:30:23.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192256 len 8192 2022-09-10T04:30:23.537 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 8192 2022-09-10T04:30:23.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192256 len 8192 2022-09-10T04:30:23.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 8192 2022-09-10T04:30:23.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192256 len 16384 2022-09-10T04:30:23.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 16384 2022-09-10T04:30:23.538 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192256 len 16384 2022-09-10T04:30:23.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192256 len 16384 2022-09-10T04:30:23.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192768 len 1024 2022-09-10T04:30:23.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 1024 2022-09-10T04:30:23.539 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192768 len 1024 2022-09-10T04:30:23.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 1024 2022-09-10T04:30:23.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192768 len 2048 2022-09-10T04:30:23.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 2048 2022-09-10T04:30:23.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192768 len 2048 2022-09-10T04:30:23.540 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 2048 2022-09-10T04:30:23.541 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192768 len 4096 2022-09-10T04:30:23.541 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 4096 2022-09-10T04:30:23.541 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192768 len 4096 2022-09-10T04:30:23.541 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 4096 2022-09-10T04:30:23.542 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192768 len 8192 2022-09-10T04:30:23.542 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 8192 2022-09-10T04:30:23.542 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192768 len 8192 2022-09-10T04:30:23.542 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 8192 2022-09-10T04:30:23.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4192768 len 16384 2022-09-10T04:30:23.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 16384 2022-09-10T04:30:23.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4192768 len 16384 2022-09-10T04:30:23.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4192768 len 16384 2022-09-10T04:30:23.543 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193280 len 1024 2022-09-10T04:30:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:24 smithi104 ceph-mon[97044]: pgmap v53: 129 pgs: 129 active+clean; 881 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 503 KiB/s wr, 239 op/s 2022-09-10T04:30:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:24 smithi081 ceph-mon[90133]: pgmap v53: 129 pgs: 129 active+clean; 881 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 503 KiB/s wr, 239 op/s 2022-09-10T04:30:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:24 smithi110 ceph-mon[97633]: pgmap v53: 129 pgs: 129 active+clean; 881 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s rd, 503 KiB/s wr, 239 op/s 2022-09-10T04:30:25.162 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 1024 2022-09-10T04:30:25.162 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193280 len 1024 2022-09-10T04:30:25.163 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 1024 2022-09-10T04:30:25.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193280 len 2048 2022-09-10T04:30:25.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 2048 2022-09-10T04:30:25.165 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193280 len 2048 2022-09-10T04:30:25.165 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 2048 2022-09-10T04:30:25.166 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193280 len 4096 2022-09-10T04:30:25.166 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 4096 2022-09-10T04:30:25.167 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193280 len 4096 2022-09-10T04:30:25.167 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 4096 2022-09-10T04:30:25.168 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193280 len 8192 2022-09-10T04:30:25.168 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 8192 2022-09-10T04:30:25.169 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193280 len 8192 2022-09-10T04:30:25.169 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 8192 2022-09-10T04:30:25.170 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193280 len 16384 2022-09-10T04:30:25.171 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 16384 2022-09-10T04:30:25.171 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193280 len 16384 2022-09-10T04:30:25.171 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193280 len 16384 2022-09-10T04:30:25.172 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193792 len 1024 2022-09-10T04:30:25.172 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 1024 2022-09-10T04:30:25.173 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193792 len 1024 2022-09-10T04:30:25.173 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 1024 2022-09-10T04:30:25.174 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193792 len 2048 2022-09-10T04:30:25.175 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 2048 2022-09-10T04:30:25.175 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193792 len 2048 2022-09-10T04:30:25.175 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 2048 2022-09-10T04:30:25.176 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193792 len 4096 2022-09-10T04:30:25.176 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 4096 2022-09-10T04:30:25.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193792 len 4096 2022-09-10T04:30:25.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 4096 2022-09-10T04:30:25.178 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193792 len 8192 2022-09-10T04:30:25.178 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 8192 2022-09-10T04:30:25.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193792 len 8192 2022-09-10T04:30:25.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 8192 2022-09-10T04:30:25.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4193792 len 16384 2022-09-10T04:30:25.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 16384 2022-09-10T04:30:25.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4193792 len 16384 2022-09-10T04:30:25.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4193792 len 16384 2022-09-10T04:30:25.182 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194304 len 1024 2022-09-10T04:30:25.182 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 1024 2022-09-10T04:30:25.183 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194304 len 1024 2022-09-10T04:30:25.183 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 1024 2022-09-10T04:30:25.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194304 len 2048 2022-09-10T04:30:25.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 2048 2022-09-10T04:30:25.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194304 len 2048 2022-09-10T04:30:25.185 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 2048 2022-09-10T04:30:25.186 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194304 len 4096 2022-09-10T04:30:25.186 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 4096 2022-09-10T04:30:25.187 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194304 len 4096 2022-09-10T04:30:25.187 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 4096 2022-09-10T04:30:25.188 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194304 len 8192 2022-09-10T04:30:25.188 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 8192 2022-09-10T04:30:25.188 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194304 len 8192 2022-09-10T04:30:25.189 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 8192 2022-09-10T04:30:25.189 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194304 len 16384 2022-09-10T04:30:25.190 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 16384 2022-09-10T04:30:25.190 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194304 len 16384 2022-09-10T04:30:25.191 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194304 len 16384 2022-09-10T04:30:25.191 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194816 len 1024 2022-09-10T04:30:25.192 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 1024 2022-09-10T04:30:25.192 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194816 len 1024 2022-09-10T04:30:25.192 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 1024 2022-09-10T04:30:25.193 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194816 len 2048 2022-09-10T04:30:25.193 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 2048 2022-09-10T04:30:25.194 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194816 len 2048 2022-09-10T04:30:25.194 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 2048 2022-09-10T04:30:25.195 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194816 len 4096 2022-09-10T04:30:25.195 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 4096 2022-09-10T04:30:25.195 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194816 len 4096 2022-09-10T04:30:25.196 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 4096 2022-09-10T04:30:25.196 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194816 len 8192 2022-09-10T04:30:25.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 8192 2022-09-10T04:30:25.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194816 len 8192 2022-09-10T04:30:25.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 8192 2022-09-10T04:30:25.197 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4194816 len 16384 2022-09-10T04:30:25.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 16384 2022-09-10T04:30:25.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4194816 len 16384 2022-09-10T04:30:25.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4194816 len 16384 2022-09-10T04:30:25.198 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195328 len 1024 2022-09-10T04:30:25.199 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 1024 2022-09-10T04:30:25.761 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195328 len 1024 2022-09-10T04:30:25.762 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 1024 2022-09-10T04:30:25.762 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195328 len 2048 2022-09-10T04:30:25.763 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 2048 2022-09-10T04:30:25.763 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195328 len 2048 2022-09-10T04:30:25.764 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 2048 2022-09-10T04:30:25.764 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195328 len 4096 2022-09-10T04:30:25.765 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 4096 2022-09-10T04:30:25.765 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195328 len 4096 2022-09-10T04:30:25.765 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 4096 2022-09-10T04:30:25.766 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195328 len 8192 2022-09-10T04:30:25.766 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 8192 2022-09-10T04:30:25.767 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195328 len 8192 2022-09-10T04:30:25.767 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 8192 2022-09-10T04:30:25.767 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195328 len 16384 2022-09-10T04:30:25.768 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 16384 2022-09-10T04:30:25.768 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195328 len 16384 2022-09-10T04:30:25.768 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195328 len 16384 2022-09-10T04:30:25.768 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195840 len 1024 2022-09-10T04:30:25.769 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 1024 2022-09-10T04:30:25.769 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195840 len 1024 2022-09-10T04:30:25.769 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 1024 2022-09-10T04:30:25.769 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195840 len 2048 2022-09-10T04:30:25.770 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 2048 2022-09-10T04:30:25.770 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195840 len 2048 2022-09-10T04:30:25.770 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 2048 2022-09-10T04:30:25.770 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195840 len 4096 2022-09-10T04:30:25.771 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 4096 2022-09-10T04:30:25.771 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195840 len 4096 2022-09-10T04:30:25.771 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 4096 2022-09-10T04:30:25.771 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195840 len 8192 2022-09-10T04:30:25.772 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 8192 2022-09-10T04:30:25.772 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195840 len 8192 2022-09-10T04:30:25.772 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 8192 2022-09-10T04:30:25.772 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4195840 len 16384 2022-09-10T04:30:25.773 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 16384 2022-09-10T04:30:25.773 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4195840 len 16384 2022-09-10T04:30:25.773 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4195840 len 16384 2022-09-10T04:30:25.773 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196352 len 1024 2022-09-10T04:30:25.774 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 1024 2022-09-10T04:30:25.774 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196352 len 1024 2022-09-10T04:30:25.774 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 1024 2022-09-10T04:30:25.775 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196352 len 2048 2022-09-10T04:30:25.775 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 2048 2022-09-10T04:30:25.775 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196352 len 2048 2022-09-10T04:30:25.776 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 2048 2022-09-10T04:30:25.777 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196352 len 4096 2022-09-10T04:30:25.777 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 4096 2022-09-10T04:30:25.778 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196352 len 4096 2022-09-10T04:30:25.778 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 4096 2022-09-10T04:30:25.779 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196352 len 8192 2022-09-10T04:30:25.781 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 8192 2022-09-10T04:30:25.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196352 len 8192 2022-09-10T04:30:25.783 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 8192 2022-09-10T04:30:25.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196352 len 16384 2022-09-10T04:30:25.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 16384 2022-09-10T04:30:25.784 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196352 len 16384 2022-09-10T04:30:25.785 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196352 len 16384 2022-09-10T04:30:25.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196864 len 1024 2022-09-10T04:30:25.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 1024 2022-09-10T04:30:25.786 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196864 len 1024 2022-09-10T04:30:25.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 1024 2022-09-10T04:30:25.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196864 len 2048 2022-09-10T04:30:25.787 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 2048 2022-09-10T04:30:25.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196864 len 2048 2022-09-10T04:30:25.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 2048 2022-09-10T04:30:25.788 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196864 len 4096 2022-09-10T04:30:25.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 4096 2022-09-10T04:30:25.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196864 len 4096 2022-09-10T04:30:25.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 4096 2022-09-10T04:30:25.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196864 len 8192 2022-09-10T04:30:25.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 8192 2022-09-10T04:30:25.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196864 len 8192 2022-09-10T04:30:25.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 8192 2022-09-10T04:30:25.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4196864 len 16384 2022-09-10T04:30:25.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 16384 2022-09-10T04:30:25.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4196864 len 16384 2022-09-10T04:30:25.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4196864 len 16384 2022-09-10T04:30:25.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197376 len 1024 2022-09-10T04:30:25.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 1024 2022-09-10T04:30:25.794 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197376 len 1024 2022-09-10T04:30:26.980 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 1024 2022-09-10T04:30:26.980 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197376 len 2048 2022-09-10T04:30:26.980 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 2048 2022-09-10T04:30:26.981 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197376 len 2048 2022-09-10T04:30:26.981 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 2048 2022-09-10T04:30:26.981 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197376 len 4096 2022-09-10T04:30:26.981 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 4096 2022-09-10T04:30:26.982 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197376 len 4096 2022-09-10T04:30:26.982 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 4096 2022-09-10T04:30:26.982 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197376 len 8192 2022-09-10T04:30:26.982 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 8192 2022-09-10T04:30:26.983 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197376 len 8192 2022-09-10T04:30:26.983 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 8192 2022-09-10T04:30:26.983 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197376 len 16384 2022-09-10T04:30:26.983 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 16384 2022-09-10T04:30:26.984 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197376 len 16384 2022-09-10T04:30:26.984 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197376 len 16384 2022-09-10T04:30:26.984 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197888 len 1024 2022-09-10T04:30:26.984 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 1024 2022-09-10T04:30:26.985 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197888 len 1024 2022-09-10T04:30:26.985 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 1024 2022-09-10T04:30:26.985 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197888 len 2048 2022-09-10T04:30:26.985 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 2048 2022-09-10T04:30:26.985 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197888 len 2048 2022-09-10T04:30:26.986 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 2048 2022-09-10T04:30:26.986 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197888 len 4096 2022-09-10T04:30:26.986 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 4096 2022-09-10T04:30:26.986 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197888 len 4096 2022-09-10T04:30:26.987 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 4096 2022-09-10T04:30:26.987 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197888 len 8192 2022-09-10T04:30:26.988 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 8192 2022-09-10T04:30:26.988 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197888 len 8192 2022-09-10T04:30:26.989 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 8192 2022-09-10T04:30:26.989 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3072 offset 4197888 len 16384 2022-09-10T04:30:26.990 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 16384 2022-09-10T04:30:26.990 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3072 offset 4197888 len 16384 2022-09-10T04:30:26.991 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3072 offset 4197888 len 16384 2022-09-10T04:30:26.991 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4190720 len 1024 2022-09-10T04:30:26.991 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 1024 2022-09-10T04:30:26.992 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4190720 len 1024 2022-09-10T04:30:26.992 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 1024 2022-09-10T04:30:26.992 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4190720 len 2048 2022-09-10T04:30:26.993 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 2048 2022-09-10T04:30:26.993 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4190720 len 2048 2022-09-10T04:30:26.993 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 2048 2022-09-10T04:30:26.993 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4190720 len 4096 2022-09-10T04:30:26.994 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 4096 2022-09-10T04:30:26.994 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4190720 len 4096 2022-09-10T04:30:26.994 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 4096 2022-09-10T04:30:26.994 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4190720 len 8192 2022-09-10T04:30:26.995 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 8192 2022-09-10T04:30:26.995 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4190720 len 8192 2022-09-10T04:30:26.995 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 8192 2022-09-10T04:30:26.995 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4190720 len 16384 2022-09-10T04:30:26.995 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 16384 2022-09-10T04:30:26.996 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4190720 len 16384 2022-09-10T04:30:26.996 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4190720 len 16384 2022-09-10T04:30:26.996 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191232 len 1024 2022-09-10T04:30:26.996 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 1024 2022-09-10T04:30:26.997 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191232 len 1024 2022-09-10T04:30:26.997 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 1024 2022-09-10T04:30:26.997 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191232 len 2048 2022-09-10T04:30:26.997 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 2048 2022-09-10T04:30:26.998 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191232 len 2048 2022-09-10T04:30:26.998 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 2048 2022-09-10T04:30:26.998 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191232 len 4096 2022-09-10T04:30:26.998 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 4096 2022-09-10T04:30:26.999 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191232 len 4096 2022-09-10T04:30:26.999 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 4096 2022-09-10T04:30:26.999 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191232 len 8192 2022-09-10T04:30:26.999 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 8192 2022-09-10T04:30:27.000 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191232 len 8192 2022-09-10T04:30:27.000 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 8192 2022-09-10T04:30:27.001 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191232 len 16384 2022-09-10T04:30:27.001 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 16384 2022-09-10T04:30:27.002 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191232 len 16384 2022-09-10T04:30:27.002 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191232 len 16384 2022-09-10T04:30:27.002 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191744 len 1024 2022-09-10T04:30:27.002 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 1024 2022-09-10T04:30:27.003 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191744 len 1024 2022-09-10T04:30:27.003 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 1024 2022-09-10T04:30:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:26 smithi104 ceph-mon[97044]: pgmap v54: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 812 KiB/s wr, 414 op/s 2022-09-10T04:30:27.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:26 smithi081 ceph-mon[90133]: pgmap v54: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 812 KiB/s wr, 414 op/s 2022-09-10T04:30:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:26 smithi110 ceph-mon[97633]: pgmap v54: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s rd, 812 KiB/s wr, 414 op/s 2022-09-10T04:30:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:28 smithi104 ceph-mon[97044]: pgmap v55: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 572 KiB/s wr, 320 op/s 2022-09-10T04:30:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:28 smithi081 ceph-mon[90133]: pgmap v55: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 572 KiB/s wr, 320 op/s 2022-09-10T04:30:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:28 smithi110 ceph-mon[97633]: pgmap v55: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 572 KiB/s wr, 320 op/s 2022-09-10T04:30:29.163 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191744 len 2048 2022-09-10T04:30:29.163 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 2048 2022-09-10T04:30:29.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191744 len 2048 2022-09-10T04:30:29.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 2048 2022-09-10T04:30:29.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191744 len 4096 2022-09-10T04:30:29.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 4096 2022-09-10T04:30:29.164 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191744 len 4096 2022-09-10T04:30:29.165 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 4096 2022-09-10T04:30:29.165 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191744 len 8192 2022-09-10T04:30:29.166 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 8192 2022-09-10T04:30:29.167 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191744 len 8192 2022-09-10T04:30:29.167 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 8192 2022-09-10T04:30:29.167 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4191744 len 16384 2022-09-10T04:30:29.168 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 16384 2022-09-10T04:30:29.168 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4191744 len 16384 2022-09-10T04:30:29.169 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4191744 len 16384 2022-09-10T04:30:29.169 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192256 len 1024 2022-09-10T04:30:29.169 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 1024 2022-09-10T04:30:29.170 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192256 len 1024 2022-09-10T04:30:29.171 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 1024 2022-09-10T04:30:29.171 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192256 len 2048 2022-09-10T04:30:29.172 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 2048 2022-09-10T04:30:29.172 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192256 len 2048 2022-09-10T04:30:29.172 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 2048 2022-09-10T04:30:29.173 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192256 len 4096 2022-09-10T04:30:29.173 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 4096 2022-09-10T04:30:29.173 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192256 len 4096 2022-09-10T04:30:29.174 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 4096 2022-09-10T04:30:29.175 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192256 len 8192 2022-09-10T04:30:29.175 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 8192 2022-09-10T04:30:29.175 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192256 len 8192 2022-09-10T04:30:29.176 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 8192 2022-09-10T04:30:29.176 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192256 len 16384 2022-09-10T04:30:29.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 16384 2022-09-10T04:30:29.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192256 len 16384 2022-09-10T04:30:29.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192256 len 16384 2022-09-10T04:30:29.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192768 len 1024 2022-09-10T04:30:29.177 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 1024 2022-09-10T04:30:29.178 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192768 len 1024 2022-09-10T04:30:29.178 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 1024 2022-09-10T04:30:29.178 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192768 len 2048 2022-09-10T04:30:29.178 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 2048 2022-09-10T04:30:29.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192768 len 2048 2022-09-10T04:30:29.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 2048 2022-09-10T04:30:29.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192768 len 4096 2022-09-10T04:30:29.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 4096 2022-09-10T04:30:29.179 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192768 len 4096 2022-09-10T04:30:29.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 4096 2022-09-10T04:30:29.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192768 len 8192 2022-09-10T04:30:29.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 8192 2022-09-10T04:30:29.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192768 len 8192 2022-09-10T04:30:29.180 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 8192 2022-09-10T04:30:29.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4192768 len 16384 2022-09-10T04:30:29.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 16384 2022-09-10T04:30:29.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4192768 len 16384 2022-09-10T04:30:29.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4192768 len 16384 2022-09-10T04:30:29.181 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193280 len 1024 2022-09-10T04:30:29.182 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 1024 2022-09-10T04:30:29.182 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193280 len 1024 2022-09-10T04:30:29.182 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 1024 2022-09-10T04:30:29.182 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193280 len 2048 2022-09-10T04:30:29.183 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 2048 2022-09-10T04:30:29.183 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193280 len 2048 2022-09-10T04:30:29.183 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 2048 2022-09-10T04:30:29.183 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193280 len 4096 2022-09-10T04:30:29.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 4096 2022-09-10T04:30:29.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193280 len 4096 2022-09-10T04:30:29.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 4096 2022-09-10T04:30:29.184 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193280 len 8192 2022-09-10T04:30:29.185 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 8192 2022-09-10T04:30:29.185 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193280 len 8192 2022-09-10T04:30:29.185 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 8192 2022-09-10T04:30:29.185 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193280 len 16384 2022-09-10T04:30:29.185 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 16384 2022-09-10T04:30:29.186 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193280 len 16384 2022-09-10T04:30:29.186 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193280 len 16384 2022-09-10T04:30:29.186 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193792 len 1024 2022-09-10T04:30:29.186 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 1024 2022-09-10T04:30:29.187 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193792 len 1024 2022-09-10T04:30:29.187 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 1024 2022-09-10T04:30:30.078 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193792 len 2048 2022-09-10T04:30:30.078 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 2048 2022-09-10T04:30:30.078 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193792 len 2048 2022-09-10T04:30:30.079 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 2048 2022-09-10T04:30:30.079 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193792 len 4096 2022-09-10T04:30:30.079 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 4096 2022-09-10T04:30:30.079 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193792 len 4096 2022-09-10T04:30:30.080 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 4096 2022-09-10T04:30:30.080 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193792 len 8192 2022-09-10T04:30:30.080 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 8192 2022-09-10T04:30:30.080 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193792 len 8192 2022-09-10T04:30:30.081 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 8192 2022-09-10T04:30:30.081 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4193792 len 16384 2022-09-10T04:30:30.081 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 16384 2022-09-10T04:30:30.081 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4193792 len 16384 2022-09-10T04:30:30.081 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4193792 len 16384 2022-09-10T04:30:30.082 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194304 len 1024 2022-09-10T04:30:30.082 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 1024 2022-09-10T04:30:30.082 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194304 len 1024 2022-09-10T04:30:30.082 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 1024 2022-09-10T04:30:30.083 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194304 len 2048 2022-09-10T04:30:30.083 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 2048 2022-09-10T04:30:30.083 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194304 len 2048 2022-09-10T04:30:30.083 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 2048 2022-09-10T04:30:30.083 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194304 len 4096 2022-09-10T04:30:30.084 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 4096 2022-09-10T04:30:30.084 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194304 len 4096 2022-09-10T04:30:30.084 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 4096 2022-09-10T04:30:30.084 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194304 len 8192 2022-09-10T04:30:30.084 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 8192 2022-09-10T04:30:30.085 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194304 len 8192 2022-09-10T04:30:30.085 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 8192 2022-09-10T04:30:30.085 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194304 len 16384 2022-09-10T04:30:30.085 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 16384 2022-09-10T04:30:30.086 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194304 len 16384 2022-09-10T04:30:30.086 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194304 len 16384 2022-09-10T04:30:30.087 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194816 len 1024 2022-09-10T04:30:30.087 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 1024 2022-09-10T04:30:30.088 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194816 len 1024 2022-09-10T04:30:30.088 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 1024 2022-09-10T04:30:30.088 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194816 len 2048 2022-09-10T04:30:30.088 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 2048 2022-09-10T04:30:30.089 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194816 len 2048 2022-09-10T04:30:30.089 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 2048 2022-09-10T04:30:30.089 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194816 len 4096 2022-09-10T04:30:30.089 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 4096 2022-09-10T04:30:30.089 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194816 len 4096 2022-09-10T04:30:30.090 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 4096 2022-09-10T04:30:30.090 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194816 len 8192 2022-09-10T04:30:30.090 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 8192 2022-09-10T04:30:30.090 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194816 len 8192 2022-09-10T04:30:30.091 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 8192 2022-09-10T04:30:30.091 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4194816 len 16384 2022-09-10T04:30:30.091 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 16384 2022-09-10T04:30:30.091 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4194816 len 16384 2022-09-10T04:30:30.092 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4194816 len 16384 2022-09-10T04:30:30.092 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195328 len 1024 2022-09-10T04:30:30.092 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 1024 2022-09-10T04:30:30.092 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195328 len 1024 2022-09-10T04:30:30.092 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 1024 2022-09-10T04:30:30.093 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195328 len 2048 2022-09-10T04:30:30.093 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 2048 2022-09-10T04:30:30.093 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195328 len 2048 2022-09-10T04:30:30.093 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 2048 2022-09-10T04:30:30.094 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195328 len 4096 2022-09-10T04:30:30.094 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 4096 2022-09-10T04:30:30.094 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195328 len 4096 2022-09-10T04:30:30.094 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 4096 2022-09-10T04:30:30.095 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195328 len 8192 2022-09-10T04:30:30.095 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 8192 2022-09-10T04:30:30.095 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195328 len 8192 2022-09-10T04:30:30.095 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 8192 2022-09-10T04:30:30.095 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195328 len 16384 2022-09-10T04:30:30.096 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 16384 2022-09-10T04:30:30.096 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195328 len 16384 2022-09-10T04:30:30.097 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195328 len 16384 2022-09-10T04:30:30.097 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195840 len 1024 2022-09-10T04:30:30.097 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 1024 2022-09-10T04:30:30.098 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195840 len 1024 2022-09-10T04:30:30.098 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 1024 2022-09-10T04:30:30.098 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195840 len 2048 2022-09-10T04:30:30.687 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 2048 2022-09-10T04:30:30.688 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195840 len 2048 2022-09-10T04:30:30.688 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 2048 2022-09-10T04:30:30.688 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195840 len 4096 2022-09-10T04:30:30.688 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 4096 2022-09-10T04:30:30.688 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195840 len 4096 2022-09-10T04:30:30.689 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 4096 2022-09-10T04:30:30.689 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195840 len 8192 2022-09-10T04:30:30.689 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 8192 2022-09-10T04:30:30.689 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195840 len 8192 2022-09-10T04:30:30.690 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 8192 2022-09-10T04:30:30.690 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4195840 len 16384 2022-09-10T04:30:30.690 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 16384 2022-09-10T04:30:30.690 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4195840 len 16384 2022-09-10T04:30:30.690 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4195840 len 16384 2022-09-10T04:30:30.691 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196352 len 1024 2022-09-10T04:30:30.691 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 1024 2022-09-10T04:30:30.691 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196352 len 1024 2022-09-10T04:30:30.691 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 1024 2022-09-10T04:30:30.692 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196352 len 2048 2022-09-10T04:30:30.692 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 2048 2022-09-10T04:30:30.692 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196352 len 2048 2022-09-10T04:30:30.692 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 2048 2022-09-10T04:30:30.692 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196352 len 4096 2022-09-10T04:30:30.693 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 4096 2022-09-10T04:30:30.693 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196352 len 4096 2022-09-10T04:30:30.693 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 4096 2022-09-10T04:30:30.694 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196352 len 8192 2022-09-10T04:30:30.694 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 8192 2022-09-10T04:30:30.694 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196352 len 8192 2022-09-10T04:30:30.694 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 8192 2022-09-10T04:30:30.695 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196352 len 16384 2022-09-10T04:30:30.695 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 16384 2022-09-10T04:30:30.695 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196352 len 16384 2022-09-10T04:30:30.695 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196352 len 16384 2022-09-10T04:30:30.696 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196864 len 1024 2022-09-10T04:30:30.696 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 1024 2022-09-10T04:30:30.696 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196864 len 1024 2022-09-10T04:30:30.696 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 1024 2022-09-10T04:30:30.696 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196864 len 2048 2022-09-10T04:30:30.697 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 2048 2022-09-10T04:30:30.697 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196864 len 2048 2022-09-10T04:30:30.697 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 2048 2022-09-10T04:30:30.697 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196864 len 4096 2022-09-10T04:30:30.698 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 4096 2022-09-10T04:30:30.698 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196864 len 4096 2022-09-10T04:30:30.698 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 4096 2022-09-10T04:30:30.698 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196864 len 8192 2022-09-10T04:30:30.698 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 8192 2022-09-10T04:30:30.699 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196864 len 8192 2022-09-10T04:30:30.699 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 8192 2022-09-10T04:30:30.699 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4196864 len 16384 2022-09-10T04:30:30.699 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 16384 2022-09-10T04:30:30.699 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4196864 len 16384 2022-09-10T04:30:30.700 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4196864 len 16384 2022-09-10T04:30:30.700 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197376 len 1024 2022-09-10T04:30:30.700 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 1024 2022-09-10T04:30:30.700 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197376 len 1024 2022-09-10T04:30:30.700 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 1024 2022-09-10T04:30:30.701 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197376 len 2048 2022-09-10T04:30:30.701 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 2048 2022-09-10T04:30:30.701 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197376 len 2048 2022-09-10T04:30:30.701 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 2048 2022-09-10T04:30:30.702 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197376 len 4096 2022-09-10T04:30:30.702 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 4096 2022-09-10T04:30:30.702 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197376 len 4096 2022-09-10T04:30:30.702 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 4096 2022-09-10T04:30:30.703 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197376 len 8192 2022-09-10T04:30:30.703 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 8192 2022-09-10T04:30:30.703 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197376 len 8192 2022-09-10T04:30:30.703 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 8192 2022-09-10T04:30:30.704 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197376 len 16384 2022-09-10T04:30:30.704 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 16384 2022-09-10T04:30:30.704 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197376 len 16384 2022-09-10T04:30:30.705 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197376 len 16384 2022-09-10T04:30:30.705 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197888 len 1024 2022-09-10T04:30:30.705 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 1024 2022-09-10T04:30:30.705 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197888 len 1024 2022-09-10T04:30:30.705 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 1024 2022-09-10T04:30:30.706 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197888 len 2048 2022-09-10T04:30:30.706 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 2048 2022-09-10T04:30:30.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197888 len 2048 2022-09-10T04:30:30.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 2048 2022-09-10T04:30:30.789 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197888 len 4096 2022-09-10T04:30:30.790 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 4096 2022-09-10T04:30:30.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197888 len 4096 2022-09-10T04:30:30.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 4096 2022-09-10T04:30:30.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197888 len 8192 2022-09-10T04:30:30.791 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 8192 2022-09-10T04:30:30.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197888 len 8192 2022-09-10T04:30:30.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 8192 2022-09-10T04:30:30.792 INFO:tasks.workunit.client.0.smithi081.stdout:write_direct buf_align 3584 offset 4197888 len 16384 2022-09-10T04:30:30.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 16384 2022-09-10T04:30:30.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_sync buf_align 3584 offset 4197888 len 16384 2022-09-10T04:30:30.793 INFO:tasks.workunit.client.0.smithi081.stdout:write_file buf_align 3584 offset 4197888 len 16384 2022-09-10T04:30:30.795 INFO:teuthology.orchestra.run:Running command with timeout 3600 2022-09-10T04:30:30.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2022-09-10T04:30:30.902 INFO:tasks.workunit:Stopping ['direct_io'] on client.0... 2022-09-10T04:30:30.903 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2022-09-10T04:30:30.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:30 smithi081 ceph-mon[90133]: pgmap v56: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 572 KiB/s wr, 320 op/s 2022-09-10T04:30:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:30 smithi104 ceph-mon[97044]: pgmap v56: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 572 KiB/s wr, 320 op/s 2022-09-10T04:30:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:30 smithi110 ceph-mon[97633]: pgmap v56: 129 pgs: 129 active+clean; 241 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 572 KiB/s wr, 320 op/s 2022-09-10T04:30:32.270 DEBUG:teuthology.parallel:result is None 2022-09-10T04:30:32.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2022-09-10T04:30:32.365 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2022-09-10T04:30:32.366 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2022-09-10T04:30:32.385 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:30:32.401 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2022-09-10T04:30:32.420 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2022-09-10T04:30:32.421 INFO:teuthology.task.exec:Executing custom commands... 2022-09-10T04:30:32.422 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2022-09-10T04:30:32.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2022-09-10T04:30:32.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2022-09-10T04:30:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:32 smithi081 ceph-mon[90133]: pgmap v57: 129 pgs: 129 active+clean; 175 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 822 KiB/s wr, 416 op/s 2022-09-10T04:30:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:32 smithi104 ceph-mon[97044]: pgmap v57: 129 pgs: 129 active+clean; 175 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 822 KiB/s wr, 416 op/s 2022-09-10T04:30:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:32 smithi110 ceph-mon[97633]: pgmap v57: 129 pgs: 129 active+clean; 175 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s rd, 822 KiB/s wr, 416 op/s 2022-09-10T04:30:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:34 smithi104 ceph-mon[97044]: pgmap v58: 129 pgs: 129 active+clean; 175 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 558 KiB/s wr, 270 op/s 2022-09-10T04:30:35.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:34 smithi081 ceph-mon[90133]: pgmap v58: 129 pgs: 129 active+clean; 175 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 558 KiB/s wr, 270 op/s 2022-09-10T04:30:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:34 smithi110 ceph-mon[97633]: pgmap v58: 129 pgs: 129 active+clean; 175 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s rd, 558 KiB/s wr, 270 op/s 2022-09-10T04:30:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:36 smithi104 ceph-mon[97044]: pgmap v59: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 703 KiB/s wr, 298 op/s 2022-09-10T04:30:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:36 smithi081 ceph-mon[90133]: pgmap v59: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 703 KiB/s wr, 298 op/s 2022-09-10T04:30:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:36 smithi110 ceph-mon[97633]: pgmap v59: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s rd, 703 KiB/s wr, 298 op/s 2022-09-10T04:30:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:38 smithi104 ceph-mon[97044]: pgmap v60: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 394 KiB/s wr, 124 op/s 2022-09-10T04:30:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:38 smithi081 ceph-mon[90133]: pgmap v60: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 394 KiB/s wr, 124 op/s 2022-09-10T04:30:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:38 smithi110 ceph-mon[97633]: pgmap v60: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 394 KiB/s wr, 124 op/s 2022-09-10T04:30:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:40 smithi104 ceph-mon[97044]: pgmap v61: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 394 KiB/s wr, 124 op/s 2022-09-10T04:30:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:40 smithi081 ceph-mon[90133]: pgmap v61: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 394 KiB/s wr, 124 op/s 2022-09-10T04:30:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:40 smithi110 ceph-mon[97633]: pgmap v61: 129 pgs: 129 active+clean; 184 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 394 KiB/s wr, 124 op/s 2022-09-10T04:30:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:42 smithi104 ceph-mon[97044]: pgmap v62: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 403 KiB/s wr, 143 op/s 2022-09-10T04:30:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:42 smithi081 ceph-mon[90133]: pgmap v62: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 403 KiB/s wr, 143 op/s 2022-09-10T04:30:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:42 smithi110 ceph-mon[97633]: pgmap v62: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 403 KiB/s wr, 143 op/s 2022-09-10T04:30:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:44 smithi104 ceph-mon[97044]: pgmap v63: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 153 KiB/s wr, 47 op/s 2022-09-10T04:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:30:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:44 smithi081 ceph-mon[90133]: pgmap v63: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 153 KiB/s wr, 47 op/s 2022-09-10T04:30:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:30:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:44 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:30:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:44 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:30:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:44 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:30:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:44 smithi110 ceph-mon[97633]: pgmap v63: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 153 KiB/s wr, 47 op/s 2022-09-10T04:30:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:30:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:30:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:30:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:30:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:46 smithi104 ceph-mon[97044]: pgmap v64: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 153 KiB/s wr, 52 op/s 2022-09-10T04:30:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:46 smithi081 ceph-mon[90133]: pgmap v64: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 153 KiB/s wr, 52 op/s 2022-09-10T04:30:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:46 smithi110 ceph-mon[97633]: pgmap v64: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 153 KiB/s wr, 52 op/s 2022-09-10T04:30:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:48 smithi104 ceph-mon[97044]: pgmap v65: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2022-09-10T04:30:48.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:48 smithi081 ceph-mon[90133]: pgmap v65: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2022-09-10T04:30:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:48 smithi110 ceph-mon[97633]: pgmap v65: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2022-09-10T04:30:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:50 smithi104 ceph-mon[97044]: pgmap v66: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2022-09-10T04:30:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:50 smithi081 ceph-mon[90133]: pgmap v66: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2022-09-10T04:30:50.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:50 smithi110 ceph-mon[97633]: pgmap v66: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2022-09-10T04:30:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:52 smithi104 ceph-mon[97044]: pgmap v67: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 KiB/s wr, 27 op/s 2022-09-10T04:30:52.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:52 smithi081 ceph-mon[90133]: pgmap v67: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 KiB/s wr, 27 op/s 2022-09-10T04:30:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:52 smithi110 ceph-mon[97633]: pgmap v67: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 KiB/s wr, 27 op/s 2022-09-10T04:30:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:54 smithi104 ceph-mon[97044]: pgmap v68: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2022-09-10T04:30:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:54 smithi081 ceph-mon[90133]: pgmap v68: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2022-09-10T04:30:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:54 smithi110 ceph-mon[97633]: pgmap v68: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2022-09-10T04:30:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:56 smithi104 ceph-mon[97044]: pgmap v69: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2022-09-10T04:30:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:56 smithi081 ceph-mon[90133]: pgmap v69: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2022-09-10T04:30:56.565 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:56 smithi110 ceph-mon[97633]: pgmap v69: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2022-09-10T04:30:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:30:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:30:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:30:57.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:57 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:30:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:57 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:30:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:57 smithi081 ceph-mon[90133]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:30:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:30:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:30:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:30:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:58 smithi104 ceph-mon[97044]: pgmap v70: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:30:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:30:58 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:30:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:58 smithi081 ceph-mon[90133]: pgmap v70: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:30:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:30:58 smithi081 ceph-mon[90133]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:30:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:58 smithi110 ceph-mon[97633]: pgmap v70: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:30:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:30:58 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' 2022-09-10T04:31:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:31:02 smithi081 ceph-mon[90133]: pgmap v71: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:02.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:31:02 smithi081 ceph-mon[90133]: pgmap v72: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2022-09-10T04:31:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:02 smithi110 ceph-mon[97633]: pgmap v71: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:02 smithi110 ceph-mon[97633]: pgmap v72: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2022-09-10T04:31:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:02 smithi104 ceph-mon[97044]: pgmap v71: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:02 smithi104 ceph-mon[97044]: pgmap v72: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2022-09-10T04:31:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v73: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T04:31:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v74: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 23 op/s 2022-09-10T04:31:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v75: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2022-09-10T04:31:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v76: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 23 op/s 2022-09-10T04:31:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v77: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 23 op/s 2022-09-10T04:31:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: mon.c calling monitor election 2022-09-10T04:31:17.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v78: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:31:17.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: pgmap v79: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T04:31:17.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: mon.c is new leader, mons c,b in quorum (ranks 1,2) 2022-09-10T04:31:17.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:31:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-09-10T04:31:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: osdmap e95: 12 total, 12 up, 12 in 2022-09-10T04:31:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: mgrmap e23: x(active, since 2m), standbys: y, z 2022-09-10T04:31:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: Health check failed: 1/3 mons down, quorum c,b (MON_DOWN) 2022-09-10T04:31:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T04:31:17.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-10T04:31:17.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:17 smithi104 ceph-mon[97044]: mon.a (rank 0) addr [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] is down (out of quorum) 2022-09-10T04:31:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v73: 129 pgs: 129 active+clean; 12 MiB data, 289 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T04:31:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v74: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 23 op/s 2022-09-10T04:31:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v75: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2022-09-10T04:31:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v76: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 23 op/s 2022-09-10T04:31:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v77: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 21 KiB/s wr, 23 op/s 2022-09-10T04:31:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: mon.c calling monitor election 2022-09-10T04:31:17.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v78: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:31:17.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: pgmap v79: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T04:31:17.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: mon.c is new leader, mons c,b in quorum (ranks 1,2) 2022-09-10T04:31:17.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: monmap e3: 3 mons at {a=[v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} 2022-09-10T04:31:17.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2022-09-10T04:31:17.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: osdmap e95: 12 total, 12 up, 12 in 2022-09-10T04:31:17.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: mgrmap e23: x(active, since 2m), standbys: y, z 2022-09-10T04:31:17.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: Health check failed: 1/3 mons down, quorum c,b (MON_DOWN) 2022-09-10T04:31:17.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: Health detail: HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T04:31:17.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: [WRN] MON_DOWN: 1/3 mons down, quorum c,b 2022-09-10T04:31:17.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:17 smithi110 ceph-mon[97633]: mon.a (rank 0) addr [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] is down (out of quorum) 2022-09-10T04:31:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:18 smithi104 ceph-mon[97044]: pgmap v80: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:31:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:18 smithi110 ceph-mon[97633]: pgmap v80: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:31:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:19 smithi110 ceph-mon[97633]: pgmap v81: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:31:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:19 smithi104 ceph-mon[97044]: pgmap v81: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:31:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:21 smithi110 ceph-mon[97633]: pgmap v82: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:21 smithi104 ceph-mon[97044]: pgmap v82: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:23 smithi110 ceph-mon[97633]: pgmap v83: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:23 smithi104 ceph-mon[97044]: pgmap v83: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:25 smithi110 ceph-mon[97633]: pgmap v84: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:25 smithi104 ceph-mon[97044]: pgmap v84: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:27 smithi104 ceph-mon[97044]: pgmap v85: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:27 smithi110 ceph-mon[97633]: pgmap v85: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:29 smithi110 ceph-mon[97633]: pgmap v86: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:29 smithi104 ceph-mon[97044]: pgmap v86: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:31 smithi110 ceph-mon[97633]: pgmap v87: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:31 smithi104 ceph-mon[97044]: pgmap v87: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:33 smithi110 ceph-mon[97633]: pgmap v88: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:33 smithi104 ceph-mon[97044]: pgmap v88: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:36.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:35 smithi110 ceph-mon[97633]: pgmap v89: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:35 smithi104 ceph-mon[97044]: pgmap v89: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:37 smithi104 ceph-mon[97044]: pgmap v90: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:37 smithi110 ceph-mon[97633]: pgmap v90: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:39 smithi110 ceph-mon[97633]: pgmap v91: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:39 smithi104 ceph-mon[97044]: pgmap v91: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:41 smithi110 ceph-mon[97633]: pgmap v92: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:41 smithi104 ceph-mon[97044]: pgmap v92: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:31:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:31:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:31:43.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:31:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:31:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:31:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:31:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:31:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:44 smithi110 ceph-mon[97633]: pgmap v93: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:44 smithi104 ceph-mon[97044]: pgmap v93: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:46.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:45 smithi110 ceph-mon[97633]: pgmap v94: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:45 smithi104 ceph-mon[97044]: pgmap v94: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:47 smithi104 ceph-mon[97044]: pgmap v95: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:31:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:47 smithi110 ceph-mon[97633]: pgmap v95: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:31:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:49 smithi110 ceph-mon[97633]: pgmap v96: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:49 smithi104 ceph-mon[97044]: pgmap v96: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:51 smithi110 ceph-mon[97633]: pgmap v97: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:31:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:51 smithi104 ceph-mon[97044]: pgmap v97: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:31:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:53 smithi110 ceph-mon[97633]: pgmap v98: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:31:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:53 smithi104 ceph-mon[97044]: pgmap v98: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:31:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:55 smithi110 ceph-mon[97633]: pgmap v99: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:56.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:55 smithi104 ceph-mon[97044]: pgmap v99: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:31:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:31:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:31:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:31:58.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:57 smithi104 ceph-mon[97044]: pgmap v100: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:31:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2022-09-10T04:31:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2022-09-10T04:31:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2022-09-10T04:31:58.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:57 smithi110 ceph-mon[97633]: pgmap v100: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:31:59 smithi110 ceph-mon[97633]: pgmap v101: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:31:59 smithi104 ceph-mon[97044]: pgmap v101: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:01 smithi110 ceph-mon[97633]: pgmap v102: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:01 smithi104 ceph-mon[97044]: pgmap v102: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:03 smithi110 ceph-mon[97633]: pgmap v103: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:03 smithi104 ceph-mon[97044]: pgmap v103: 129 pgs: 129 active+clean; 12 MiB data, 290 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:05 smithi110 ceph-mon[97633]: pgmap v104: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:32:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:05 smithi104 ceph-mon[97044]: pgmap v104: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:32:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:07 smithi104 ceph-mon[97044]: pgmap v105: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:32:08.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:07 smithi110 ceph-mon[97633]: pgmap v105: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:32:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:09 smithi110 ceph-mon[97633]: pgmap v106: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:32:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:09 smithi104 ceph-mon[97044]: pgmap v106: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:32:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:11 smithi110 ceph-mon[97633]: pgmap v107: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:32:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:11 smithi104 ceph-mon[97044]: pgmap v107: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:32:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:13 smithi110 ceph-mon[97633]: pgmap v108: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:32:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:13 smithi104 ceph-mon[97044]: pgmap v108: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:32:16.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:15 smithi110 ceph-mon[97633]: pgmap v109: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:32:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:15 smithi104 ceph-mon[97044]: pgmap v109: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:32:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:17 smithi104 ceph-mon[97044]: pgmap v110: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:32:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:17 smithi110 ceph-mon[97633]: pgmap v110: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:32:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:19 smithi110 ceph-mon[97633]: pgmap v111: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:32:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:19 smithi104 ceph-mon[97044]: pgmap v111: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:32:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:21 smithi110 ceph-mon[97633]: pgmap v112: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:21 smithi104 ceph-mon[97044]: pgmap v112: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:22 smithi110 ceph-mon[97633]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-09-10T04:32:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:22 smithi110 ceph-mon[97633]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-09-10T04:32:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:22 smithi110 ceph-mon[97633]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2022-09-10T04:32:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:22 smithi104 ceph-mon[97044]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2022-09-10T04:32:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:22 smithi104 ceph-mon[97044]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2022-09-10T04:32:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:22 smithi104 ceph-mon[97044]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2022-09-10T04:32:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:23 smithi110 ceph-mon[97633]: pgmap v113: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:23 smithi104 ceph-mon[97044]: pgmap v113: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:25 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:26.182 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:25 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:26.182 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:25 smithi110 ceph-mon[97633]: pgmap v114: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:25 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:25 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:25 smithi104 ceph-mon[97044]: pgmap v114: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:26 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:26 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:26 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:26 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:27 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:27 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:27 smithi104 ceph-mon[97044]: pgmap v115: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:27 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:27 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:27 smithi110 ceph-mon[97633]: pgmap v115: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:28 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:28 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:28 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:28 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:29 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:29 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:29 smithi104 ceph-mon[97044]: pgmap v116: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:29 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:29 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:29 smithi110 ceph-mon[97633]: pgmap v116: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:30 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:31.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:30 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:31.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:30 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:31.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:30 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:31 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:31 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:31 smithi104 ceph-mon[97044]: pgmap v117: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:31 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:31 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:31 smithi110 ceph-mon[97633]: pgmap v117: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:32 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:32 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:32 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:32 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:33 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:33 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:33 smithi104 ceph-mon[97044]: pgmap v118: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:33 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:33 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:33 smithi110 ceph-mon[97633]: pgmap v118: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:34 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:34 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:34 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:34 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:35 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:35 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:35 smithi104 ceph-mon[97044]: pgmap v119: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:35 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:35 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:35 smithi110 ceph-mon[97633]: pgmap v119: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:36 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:36 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:36 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:36 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:37 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:37 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:37 smithi104 ceph-mon[97044]: pgmap v120: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:37 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:37 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:37 smithi110 ceph-mon[97633]: pgmap v120: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:38 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:38 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:38 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:38 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:39 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:39 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:39 smithi104 ceph-mon[97044]: pgmap v121: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:39 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:39 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:39 smithi110 ceph-mon[97633]: pgmap v121: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:41 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:41 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:40 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:40 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:42 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:42 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:42 smithi104 ceph-mon[97044]: pgmap v122: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:42 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:42 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:42 smithi110 ceph-mon[97633]: pgmap v122: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:32:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:32:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:32:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:32:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:44 smithi104 ceph-mon[97044]: pgmap v123: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:32:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:44 smithi110 ceph-mon[97633]: pgmap v123: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:46 smithi104 ceph-mon[97044]: pgmap v124: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:46 smithi110 ceph-mon[97633]: pgmap v124: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:48 smithi104 ceph-mon[97044]: pgmap v125: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:48 smithi110 ceph-mon[97633]: pgmap v125: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:49 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:49 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:49 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:49 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:50 smithi104 ceph-mon[97044]: pgmap v126: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:50.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:50.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:50.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:50 smithi110 ceph-mon[97633]: pgmap v126: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:51 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:51 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:51 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:51.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:51 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:52 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:52.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:52 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:52.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:52 smithi104 ceph-mon[97044]: pgmap v127: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:52 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:52 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:52 smithi110 ceph-mon[97633]: pgmap v127: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:53 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:53 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:53 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:53 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:54.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:54 smithi110 ceph-mon[97633]: pgmap v128: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:54 smithi104 ceph-mon[97044]: pgmap v128: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:55 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:55 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:55 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:55 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:56 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:56 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:56 smithi110 ceph-mon[97633]: pgmap v129: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:56 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:56 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:56 smithi104 ceph-mon[97044]: pgmap v129: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:32:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:58 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:58 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:58 smithi110 ceph-mon[97633]: pgmap v130: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:58 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:58 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:58 smithi104 ceph-mon[97044]: pgmap v130: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:32:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:59 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:32:59 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:32:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:59 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:32:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:32:59 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:00.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:00 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:00.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:00 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:00.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:00 smithi110 ceph-mon[97633]: pgmap v131: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:00 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:00 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:00 smithi104 ceph-mon[97044]: pgmap v131: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:01 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:01 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:01 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:01 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:02 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:02 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:02 smithi104 ceph-mon[97044]: pgmap v132: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:02.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:02 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:02 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:02 smithi110 ceph-mon[97633]: pgmap v132: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:03 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:03 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:03 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:03 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:04 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:04 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:04 smithi110 ceph-mon[97633]: pgmap v133: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:04.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:04 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:04.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:04 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:04.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:04 smithi104 ceph-mon[97044]: pgmap v133: 129 pgs: 129 active+clean; 12 MiB data, 294 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:05 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:05 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:05 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:05 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:06 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:06 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:06 smithi110 ceph-mon[97633]: pgmap v134: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:33:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:06 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:06 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:06.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:06 smithi104 ceph-mon[97044]: pgmap v134: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:33:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:07 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:07 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:07 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:07 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:08 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:08 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:08 smithi110 ceph-mon[97633]: pgmap v135: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:33:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:08 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:08 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:08 smithi104 ceph-mon[97044]: pgmap v135: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:33:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:09 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:09 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:09 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:09 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:10.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:10 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:10 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:10 smithi110 ceph-mon[97633]: pgmap v136: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:33:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:10 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:10 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:10 smithi104 ceph-mon[97044]: pgmap v136: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:33:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:11 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:11 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:11 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:11 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:12 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:12 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:12 smithi110 ceph-mon[97633]: pgmap v137: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:33:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:12 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:12 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:12 smithi104 ceph-mon[97044]: pgmap v137: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:33:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:13 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:13 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:13 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:13 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:14 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:14 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:14 smithi110 ceph-mon[97633]: pgmap v138: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:33:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:14 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:14 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:14 smithi104 ceph-mon[97044]: pgmap v138: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:33:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:15 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:15 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:15 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:15 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:16 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:16 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:16 smithi110 ceph-mon[97633]: pgmap v139: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:33:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:16 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:16 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:16 smithi104 ceph-mon[97044]: pgmap v139: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:33:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:17 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:17 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:17 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:17 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:18 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:18 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:18 smithi110 ceph-mon[97633]: pgmap v140: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:33:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:18 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:18 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:18 smithi104 ceph-mon[97044]: pgmap v140: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:33:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:19 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:19 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:19 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:19 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:20 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:20 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:20 smithi110 ceph-mon[97633]: pgmap v141: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:33:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:20 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:20 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:20 smithi104 ceph-mon[97044]: pgmap v141: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:33:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:21 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:21 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:21 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:21 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:22 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:22 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:22 smithi110 ceph-mon[97633]: pgmap v142: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:22 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:22 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:22 smithi104 ceph-mon[97044]: pgmap v142: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:23 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:23 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:23 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:23 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:24.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:24 smithi110 ceph-mon[97633]: pgmap v143: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:24 smithi104 ceph-mon[97044]: pgmap v143: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:25 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:25 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:25 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:25 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:26 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:26 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:26 smithi110 ceph-mon[97633]: pgmap v144: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:26 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:26 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:26 smithi104 ceph-mon[97044]: pgmap v144: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:27 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:27 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:27 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:27 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:28 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:28 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:28 smithi104 ceph-mon[97044]: pgmap v145: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:28.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:28 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:28 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:28 smithi110 ceph-mon[97633]: pgmap v145: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:29 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:29 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:29 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:29 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:30 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:30 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:30 smithi104 ceph-mon[97044]: pgmap v146: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:30 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:30 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:30 smithi110 ceph-mon[97633]: pgmap v146: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:31 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:31 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:31 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:31 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:32 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:32 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:32 smithi104 ceph-mon[97044]: pgmap v147: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:32 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:32 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:32 smithi110 ceph-mon[97633]: pgmap v147: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:33 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:33 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:33 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:33 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:34 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:34 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:34 smithi104 ceph-mon[97044]: pgmap v148: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:34 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:34 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:34 smithi110 ceph-mon[97633]: pgmap v148: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:35 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:35 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:35 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:35 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:36 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:36 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:36 smithi104 ceph-mon[97044]: pgmap v149: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:36 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:36 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:36.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:36 smithi110 ceph-mon[97633]: pgmap v149: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:37 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:37 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:37 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:37 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:38 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:38 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:38 smithi104 ceph-mon[97044]: pgmap v150: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:38 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:38 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:38 smithi110 ceph-mon[97633]: pgmap v150: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:39 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:39 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:39 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:39 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:40 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:40 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:40 smithi104 ceph-mon[97044]: pgmap v151: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:40 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:40.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:40 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:40.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:40 smithi110 ceph-mon[97633]: pgmap v151: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:41.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:41 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:41.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:41 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:41.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:41 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:41.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:41 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:42 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:42 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:42 smithi104 ceph-mon[97044]: pgmap v152: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:42 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:42 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:42.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:42 smithi110 ceph-mon[97633]: pgmap v152: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:33:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:33:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:33:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:33:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:44 smithi104 ceph-mon[97044]: pgmap v153: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:33:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:33:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:33:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:33:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:44 smithi110 ceph-mon[97633]: pgmap v153: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:46 smithi104 ceph-mon[97044]: pgmap v154: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:46 smithi110 ceph-mon[97633]: pgmap v154: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:48 smithi104 ceph-mon[97044]: pgmap v155: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:48 smithi110 ceph-mon[97633]: pgmap v155: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:49 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:49 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:49 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:49 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:50 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:50 smithi104 ceph-mon[97044]: pgmap v156: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:50.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:50 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:50 smithi110 ceph-mon[97633]: pgmap v156: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:51 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:51 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:51 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:51 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:52 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:52 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:52 smithi104 ceph-mon[97044]: pgmap v157: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:52 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:52 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:52 smithi110 ceph-mon[97633]: pgmap v157: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:53 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:53 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:53 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:53 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:54 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:54 smithi104 ceph-mon[97044]: pgmap v158: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:54.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:54 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:54 smithi110 ceph-mon[97633]: pgmap v158: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:55 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:55 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:55 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:55 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:56 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:56 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:56 smithi104 ceph-mon[97044]: pgmap v159: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:56 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:56 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:56 smithi110 ceph-mon[97633]: pgmap v159: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:33:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:57 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:57 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:58 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:58 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:58 smithi104 ceph-mon[97044]: pgmap v160: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:58 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:58 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:58 smithi110 ceph-mon[97633]: pgmap v160: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:33:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:59 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:59 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:59 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:33:59 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:59 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:59 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:33:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:59 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:33:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:33:59 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:00 smithi104 ceph-mon[97044]: pgmap v161: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:00 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:00 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:00 smithi110 ceph-mon[97633]: pgmap v161: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:00.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:00 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:00.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:00 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:01 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:01 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:01 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:01 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:02.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:02 smithi104 ceph-mon[97044]: pgmap v162: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:02 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:02 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:02 smithi110 ceph-mon[97633]: pgmap v162: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:02 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:02 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:03 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:03 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:03 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:03 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:04.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:04 smithi104 ceph-mon[97044]: pgmap v163: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:04 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:04 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:04 smithi110 ceph-mon[97633]: pgmap v163: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:04 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:04 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:05 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:05 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:05 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:05 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:06 smithi104 ceph-mon[97044]: pgmap v164: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:34:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:06 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:06.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:06 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:06 smithi110 ceph-mon[97633]: pgmap v164: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:34:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:06 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:06 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:07 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:07 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:07 smithi104 ceph-mon[97044]: pgmap v165: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:34:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:07 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:07 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:07 smithi110 ceph-mon[97633]: pgmap v165: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T04:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:08 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:08 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:08 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:08 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:09 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:09 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:09 smithi104 ceph-mon[97044]: pgmap v166: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:34:10.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:09 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:09 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:09 smithi110 ceph-mon[97633]: pgmap v166: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:34:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:10 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:10 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:10 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:10 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:12 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:12 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:12 smithi104 ceph-mon[97044]: pgmap v167: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:34:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:12 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:12 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:12 smithi110 ceph-mon[97633]: pgmap v167: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:34:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:13 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:13 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:13 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:13 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:14 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:14 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:14 smithi104 ceph-mon[97044]: pgmap v168: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:34:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:14 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:14 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:14 smithi110 ceph-mon[97633]: pgmap v168: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T04:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:15 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:15 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:15 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:15 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:16 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:16 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:16 smithi104 ceph-mon[97044]: pgmap v169: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:34:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:16 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:16 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:16 smithi110 ceph-mon[97633]: pgmap v169: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T04:34:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:17 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:17 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:17 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:17 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:18 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:18 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:18 smithi104 ceph-mon[97044]: pgmap v170: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:34:18.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:18 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:18.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:18 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:18.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:18 smithi110 ceph-mon[97633]: pgmap v170: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:34:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:19 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:19 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:19.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:19 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:19.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:19 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:20 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:20 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:20 smithi104 ceph-mon[97044]: pgmap v171: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:34:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:20 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:20 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:20 smithi110 ceph-mon[97633]: pgmap v171: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:34:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:21 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:21 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:21 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:21 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:22 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:22 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:22 smithi104 ceph-mon[97044]: pgmap v172: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:22 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:22 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:22 smithi110 ceph-mon[97633]: pgmap v172: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:23.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:23 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:23.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:23 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:23.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:23 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:23.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:23 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:24 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:24 smithi104 ceph-mon[97044]: pgmap v173: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:24 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:24 smithi110 ceph-mon[97633]: pgmap v173: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:25 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:25 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:25 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:25 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:26 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:26 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:26 smithi110 ceph-mon[97633]: pgmap v174: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:26 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:26 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:26 smithi104 ceph-mon[97044]: pgmap v174: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:27 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:27 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:27 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:27 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:28 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:28 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:28 smithi110 ceph-mon[97633]: pgmap v175: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:28 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:28 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:28 smithi104 ceph-mon[97044]: pgmap v175: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:29 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:29 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:29 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:29 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:30 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:30 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:30 smithi110 ceph-mon[97633]: pgmap v176: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:30 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:30 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:30 smithi104 ceph-mon[97044]: pgmap v176: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:31 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:31 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:31 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:31 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:32.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:32 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:32 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:32 smithi110 ceph-mon[97633]: pgmap v177: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:32 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:32 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:32 smithi104 ceph-mon[97044]: pgmap v177: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:33 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:33 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:33 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:33 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:34 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:34 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:34 smithi110 ceph-mon[97633]: pgmap v178: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:34 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:34 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:34 smithi104 ceph-mon[97044]: pgmap v178: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:35 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:35 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:35 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:35 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:36 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:36 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:36 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:36 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:37 smithi110 ceph-mon[97633]: pgmap v179: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:37 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:37 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:37 smithi104 ceph-mon[97044]: pgmap v179: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:37 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:37 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:38 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:38 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:38 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:38 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:39 smithi110 ceph-mon[97633]: pgmap v180: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:39 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:39 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:39 smithi104 ceph-mon[97044]: pgmap v180: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:39 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2022-09-10T04:34:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:39 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2022-09-10T04:34:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:41 smithi110 ceph-mon[97633]: pgmap v181: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:41 smithi104 ceph-mon[97044]: pgmap v181: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:43 smithi110 ceph-mon[97633]: pgmap v182: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:43 smithi104 ceph-mon[97044]: pgmap v182: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:34:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:34:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:34:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:34:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:34:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:34:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:34:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:34:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:45 smithi110 ceph-mon[97633]: pgmap v183: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:45 smithi104 ceph-mon[97044]: pgmap v183: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:47 smithi110 ceph-mon[97633]: pgmap v184: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:47 smithi104 ceph-mon[97044]: pgmap v184: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:49 smithi110 ceph-mon[97633]: pgmap v185: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:49 smithi104 ceph-mon[97044]: pgmap v185: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:51 smithi110 ceph-mon[97633]: pgmap v186: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:51 smithi104 ceph-mon[97044]: pgmap v186: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:53 smithi110 ceph-mon[97633]: pgmap v187: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:53 smithi104 ceph-mon[97044]: pgmap v187: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:55 smithi110 ceph-mon[97633]: pgmap v188: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:55 smithi104 ceph-mon[97044]: pgmap v188: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:57 smithi110 ceph-mon[97633]: pgmap v189: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:57 smithi104 ceph-mon[97044]: pgmap v189: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:34:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:34:59 smithi110 ceph-mon[97633]: pgmap v190: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:34:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:34:59 smithi104 ceph-mon[97044]: pgmap v190: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:01 smithi110 ceph-mon[97633]: pgmap v191: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:01 smithi104 ceph-mon[97044]: pgmap v191: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:03 smithi110 ceph-mon[97633]: pgmap v192: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:03 smithi104 ceph-mon[97044]: pgmap v192: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:05 smithi110 ceph-mon[97633]: pgmap v193: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:05 smithi104 ceph-mon[97044]: pgmap v193: 129 pgs: 129 active+clean; 12 MiB data, 295 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:07 smithi110 ceph-mon[97633]: pgmap v194: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:35:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:07 smithi104 ceph-mon[97044]: pgmap v194: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:08 smithi104 ceph-mon[97044]: pgmap v195: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:35:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:08 smithi104 ceph-mon[97044]: osdmap e96: 12 total, 12 up, 12 in 2022-09-10T04:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:08 smithi110 ceph-mon[97633]: pgmap v195: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:08 smithi110 ceph-mon[97633]: osdmap e96: 12 total, 12 up, 12 in 2022-09-10T04:35:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:10 smithi104 ceph-mon[97044]: pgmap v197: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:35:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:10 smithi110 ceph-mon[97633]: pgmap v197: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:35:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:12 smithi104 ceph-mon[97044]: pgmap v198: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:35:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:12 smithi104 ceph-mon[97044]: osdmap e97: 12 total, 12 up, 12 in 2022-09-10T04:35:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:12 smithi110 ceph-mon[97633]: pgmap v198: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:35:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:12 smithi110 ceph-mon[97633]: osdmap e97: 12 total, 12 up, 12 in 2022-09-10T04:35:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:14 smithi104 ceph-mon[97044]: pgmap v200: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:35:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:14 smithi110 ceph-mon[97633]: pgmap v200: 129 pgs: 129 active+clean; 12 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:35:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:17 smithi104 ceph-mon[97044]: pgmap v201: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:35:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:17 smithi110 ceph-mon[97633]: pgmap v201: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:35:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:19 smithi104 ceph-mon[97044]: pgmap v202: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T04:35:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:19 smithi110 ceph-mon[97633]: pgmap v202: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T04:35:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:21 smithi104 ceph-mon[97044]: pgmap v203: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:35:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:21 smithi110 ceph-mon[97633]: pgmap v203: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:35:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:23 smithi104 ceph-mon[97044]: pgmap v204: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:35:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:23 smithi110 ceph-mon[97633]: pgmap v204: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:35:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:25 smithi104 ceph-mon[97044]: pgmap v205: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:35:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:25 smithi110 ceph-mon[97633]: pgmap v205: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:35:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:27 smithi104 ceph-mon[97044]: pgmap v206: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:35:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:27 smithi110 ceph-mon[97633]: pgmap v206: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:35:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:29 smithi104 ceph-mon[97044]: pgmap v207: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:29 smithi110 ceph-mon[97633]: pgmap v207: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:31 smithi104 ceph-mon[97044]: pgmap v208: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:31 smithi110 ceph-mon[97633]: pgmap v208: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:33 smithi104 ceph-mon[97044]: pgmap v209: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:33 smithi110 ceph-mon[97633]: pgmap v209: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:35 smithi104 ceph-mon[97044]: pgmap v210: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:35 smithi110 ceph-mon[97633]: pgmap v210: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:37 smithi104 ceph-mon[97044]: pgmap v211: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:37 smithi110 ceph-mon[97633]: pgmap v211: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:39 smithi104 ceph-mon[97044]: pgmap v212: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:39 smithi110 ceph-mon[97633]: pgmap v212: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:35:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:41 smithi104 ceph-mon[97044]: pgmap v213: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:41 smithi104 ceph-mon[97044]: evicting unresponsive client smithi081:0 (25355), after 352.299 seconds 2022-09-10T04:35:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:41 smithi104 ceph-mon[97044]: Evicting (and blocklisting) client session 25355 (172.21.15.81:0/197519246) 2022-09-10T04:35:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:41 smithi104 ceph-mon[97044]: from='mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026]' entity='mds.c' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.81:0/197519246"}]: dispatch 2022-09-10T04:35:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:41 smithi110 ceph-mon[97633]: pgmap v213: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:35:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:41 smithi110 ceph-mon[97633]: evicting unresponsive client smithi081:0 (25355), after 352.299 seconds 2022-09-10T04:35:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:41 smithi110 ceph-mon[97633]: Evicting (and blocklisting) client session 25355 (172.21.15.81:0/197519246) 2022-09-10T04:35:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:41 smithi110 ceph-mon[97633]: from='mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026]' entity='mds.c' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.81:0/197519246"}]: dispatch 2022-09-10T04:35:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:42 smithi110 ceph-mon[97633]: from='mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026]' entity='mds.c' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.81:0/197519246"}]': finished 2022-09-10T04:35:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:42 smithi110 ceph-mon[97633]: osdmap e98: 12 total, 12 up, 12 in 2022-09-10T04:35:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:42 smithi104 ceph-mon[97044]: from='mds.? [v2:172.21.15.110:6836/1467999026,v1:172.21.15.110:6837/1467999026]' entity='mds.c' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.81:0/197519246"}]': finished 2022-09-10T04:35:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:42 smithi104 ceph-mon[97044]: osdmap e98: 12 total, 12 up, 12 in 2022-09-10T04:35:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:43 smithi110 ceph-mon[97633]: pgmap v215: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:35:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:43 smithi104 ceph-mon[97044]: pgmap v215: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:35:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:35:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:35:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:35:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:35:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:35:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:35:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:35:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:35:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:45 smithi110 ceph-mon[97633]: pgmap v216: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:35:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:45 smithi104 ceph-mon[97044]: pgmap v216: 129 pgs: 129 active+clean; 12 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T04:35:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:47 smithi110 ceph-mon[97633]: pgmap v217: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2022-09-10T04:35:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:47 smithi104 ceph-mon[97044]: pgmap v217: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2022-09-10T04:35:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:49 smithi110 ceph-mon[97633]: pgmap v218: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2022-09-10T04:35:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:49 smithi104 ceph-mon[97044]: pgmap v218: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2022-09-10T04:35:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:51 smithi110 ceph-mon[97633]: pgmap v219: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2022-09-10T04:35:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:51 smithi104 ceph-mon[97044]: pgmap v219: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2022-09-10T04:35:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:53 smithi110 ceph-mon[97633]: pgmap v220: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 473 B/s wr, 10 op/s 2022-09-10T04:35:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:53 smithi104 ceph-mon[97044]: pgmap v220: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 473 B/s wr, 10 op/s 2022-09-10T04:35:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:55 smithi110 ceph-mon[97633]: pgmap v221: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2022-09-10T04:35:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:55 smithi104 ceph-mon[97044]: pgmap v221: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2022-09-10T04:35:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:57 smithi110 ceph-mon[97633]: pgmap v222: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 13 op/s 2022-09-10T04:35:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:57 smithi104 ceph-mon[97044]: pgmap v222: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 13 op/s 2022-09-10T04:35:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:35:59 smithi110 ceph-mon[97633]: pgmap v223: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2022-09-10T04:35:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:35:59 smithi104 ceph-mon[97044]: pgmap v223: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2022-09-10T04:36:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:01 smithi110 ceph-mon[97633]: pgmap v224: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2022-09-10T04:36:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:01 smithi104 ceph-mon[97044]: pgmap v224: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 0 B/s wr, 12 op/s 2022-09-10T04:36:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:03 smithi110 ceph-mon[97633]: pgmap v225: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:03 smithi104 ceph-mon[97044]: pgmap v225: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:05 smithi110 ceph-mon[97633]: pgmap v226: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:05 smithi104 ceph-mon[97044]: pgmap v226: 129 pgs: 129 active+clean; 12 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:07 smithi110 ceph-mon[97633]: pgmap v227: 129 pgs: 129 active+clean; 12 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:36:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:07 smithi104 ceph-mon[97044]: pgmap v227: 129 pgs: 129 active+clean; 12 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:36:09.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:09 smithi104 ceph-mon[97044]: pgmap v228: 129 pgs: 129 active+clean; 12 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:36:09.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:09 smithi104 ceph-mon[97044]: osdmap e99: 12 total, 12 up, 12 in 2022-09-10T04:36:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:09 smithi110 ceph-mon[97633]: pgmap v228: 129 pgs: 129 active+clean; 12 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:09 smithi110 ceph-mon[97633]: osdmap e99: 12 total, 12 up, 12 in 2022-09-10T04:36:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:11 smithi104 ceph-mon[97044]: pgmap v230: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:36:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:11 smithi110 ceph-mon[97633]: pgmap v230: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:36:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:13 smithi104 ceph-mon[97044]: pgmap v231: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:36:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:13 smithi104 ceph-mon[97044]: osdmap e100: 12 total, 12 up, 12 in 2022-09-10T04:36:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:13 smithi110 ceph-mon[97633]: pgmap v231: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:36:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:13 smithi110 ceph-mon[97633]: osdmap e100: 12 total, 12 up, 12 in 2022-09-10T04:36:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:15 smithi104 ceph-mon[97044]: pgmap v233: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:36:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:15 smithi110 ceph-mon[97633]: pgmap v233: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:36:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:17 smithi104 ceph-mon[97044]: pgmap v234: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:36:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:17 smithi110 ceph-mon[97633]: pgmap v234: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:36:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:19 smithi104 ceph-mon[97044]: pgmap v235: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T04:36:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:19 smithi110 ceph-mon[97633]: pgmap v235: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T04:36:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:21 smithi104 ceph-mon[97044]: pgmap v236: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:36:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:21 smithi110 ceph-mon[97633]: pgmap v236: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:36:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:23 smithi104 ceph-mon[97044]: pgmap v237: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:36:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:23 smithi110 ceph-mon[97633]: pgmap v237: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:36:25.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:25 smithi104 ceph-mon[97044]: pgmap v238: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:36:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:25 smithi110 ceph-mon[97633]: pgmap v238: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:36:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:27 smithi104 ceph-mon[97044]: pgmap v239: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:36:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:27 smithi110 ceph-mon[97633]: pgmap v239: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:36:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:29 smithi104 ceph-mon[97044]: pgmap v240: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:29 smithi110 ceph-mon[97633]: pgmap v240: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:31 smithi104 ceph-mon[97044]: pgmap v241: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:31.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:31 smithi110 ceph-mon[97633]: pgmap v241: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:33 smithi104 ceph-mon[97044]: pgmap v242: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:33 smithi110 ceph-mon[97633]: pgmap v242: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:35 smithi110 ceph-mon[97633]: pgmap v243: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:35 smithi104 ceph-mon[97044]: pgmap v243: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:37 smithi110 ceph-mon[97633]: pgmap v244: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:37 smithi104 ceph-mon[97044]: pgmap v244: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:39 smithi110 ceph-mon[97633]: pgmap v245: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:36:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:39 smithi104 ceph-mon[97044]: pgmap v245: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:36:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:41 smithi110 ceph-mon[97633]: pgmap v246: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:41 smithi104 ceph-mon[97044]: pgmap v246: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:43 smithi110 ceph-mon[97633]: pgmap v247: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:36:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:43 smithi104 ceph-mon[97044]: pgmap v247: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:36:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:36:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:36:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:36:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:36:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:36:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:36:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:36:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:36:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:45 smithi110 ceph-mon[97633]: pgmap v248: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:36:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:45 smithi104 ceph-mon[97044]: pgmap v248: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:36:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:47 smithi110 ceph-mon[97633]: pgmap v249: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:47 smithi104 ceph-mon[97044]: pgmap v249: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:49 smithi110 ceph-mon[97633]: pgmap v250: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:49 smithi104 ceph-mon[97044]: pgmap v250: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:51 smithi110 ceph-mon[97633]: pgmap v251: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:51 smithi104 ceph-mon[97044]: pgmap v251: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:53 smithi110 ceph-mon[97633]: pgmap v252: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:53 smithi104 ceph-mon[97044]: pgmap v252: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:55 smithi110 ceph-mon[97633]: pgmap v253: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:55 smithi104 ceph-mon[97044]: pgmap v253: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:57 smithi110 ceph-mon[97633]: pgmap v254: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:57 smithi104 ceph-mon[97044]: pgmap v254: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:36:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:36:59 smithi110 ceph-mon[97633]: pgmap v255: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:36:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:36:59 smithi104 ceph-mon[97044]: pgmap v255: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:01 smithi110 ceph-mon[97633]: pgmap v256: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:01 smithi104 ceph-mon[97044]: pgmap v256: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:03 smithi110 ceph-mon[97633]: pgmap v257: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:03 smithi104 ceph-mon[97044]: pgmap v257: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:05 smithi110 ceph-mon[97633]: pgmap v258: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:05 smithi104 ceph-mon[97044]: pgmap v258: 129 pgs: 129 active+clean; 7.7 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:07 smithi110 ceph-mon[97633]: pgmap v259: 129 pgs: 129 active+clean; 7.7 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:37:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:07 smithi104 ceph-mon[97044]: pgmap v259: 129 pgs: 129 active+clean; 7.7 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:37:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:09 smithi104 ceph-mon[97044]: pgmap v260: 129 pgs: 129 active+clean; 7.7 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:37:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:09 smithi104 ceph-mon[97044]: osdmap e101: 12 total, 12 up, 12 in 2022-09-10T04:37:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:09 smithi110 ceph-mon[97633]: pgmap v260: 129 pgs: 129 active+clean; 7.7 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:37:09.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:09 smithi110 ceph-mon[97633]: osdmap e101: 12 total, 12 up, 12 in 2022-09-10T04:37:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:11 smithi104 ceph-mon[97044]: pgmap v262: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:37:11.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:11 smithi110 ceph-mon[97633]: pgmap v262: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:37:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:13 smithi104 ceph-mon[97044]: pgmap v263: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:37:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:13 smithi104 ceph-mon[97044]: osdmap e102: 12 total, 12 up, 12 in 2022-09-10T04:37:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:13 smithi110 ceph-mon[97633]: pgmap v263: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:37:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:13 smithi110 ceph-mon[97633]: osdmap e102: 12 total, 12 up, 12 in 2022-09-10T04:37:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:15 smithi104 ceph-mon[97044]: pgmap v265: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:37:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:15 smithi110 ceph-mon[97633]: pgmap v265: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:37:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:17 smithi104 ceph-mon[97044]: pgmap v266: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:37:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:17 smithi110 ceph-mon[97633]: pgmap v266: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:37:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:19 smithi104 ceph-mon[97044]: pgmap v267: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T04:37:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:19 smithi110 ceph-mon[97633]: pgmap v267: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T04:37:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:21 smithi104 ceph-mon[97044]: pgmap v268: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2022-09-10T04:37:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:21 smithi110 ceph-mon[97633]: pgmap v268: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2022-09-10T04:37:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:23 smithi110 ceph-mon[97633]: pgmap v269: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2022-09-10T04:37:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:23 smithi104 ceph-mon[97044]: pgmap v269: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 307 B/s wr, 10 op/s 2022-09-10T04:37:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:25 smithi110 ceph-mon[97633]: pgmap v270: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 259 B/s wr, 8 op/s 2022-09-10T04:37:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:25 smithi104 ceph-mon[97044]: pgmap v270: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 259 B/s wr, 8 op/s 2022-09-10T04:37:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:27 smithi110 ceph-mon[97633]: pgmap v271: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:37:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:27 smithi104 ceph-mon[97044]: pgmap v271: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:37:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:29 smithi110 ceph-mon[97633]: pgmap v272: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2022-09-10T04:37:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:29 smithi104 ceph-mon[97044]: pgmap v272: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2022-09-10T04:37:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:31 smithi110 ceph-mon[97633]: pgmap v273: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 170 B/s wr, 12 op/s 2022-09-10T04:37:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:31 smithi104 ceph-mon[97044]: pgmap v273: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 170 B/s wr, 12 op/s 2022-09-10T04:37:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:33 smithi110 ceph-mon[97633]: pgmap v274: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:33 smithi104 ceph-mon[97044]: pgmap v274: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:35 smithi110 ceph-mon[97633]: pgmap v275: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:35 smithi104 ceph-mon[97044]: pgmap v275: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:37.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:37 smithi110 ceph-mon[97633]: pgmap v276: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:37 smithi104 ceph-mon[97044]: pgmap v276: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:39 smithi110 ceph-mon[97633]: pgmap v277: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:39 smithi104 ceph-mon[97044]: pgmap v277: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:41 smithi110 ceph-mon[97633]: pgmap v278: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:41 smithi104 ceph-mon[97044]: pgmap v278: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:43 smithi110 ceph-mon[97633]: pgmap v279: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:43 smithi104 ceph-mon[97044]: pgmap v279: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:37:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:37:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:37:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:37:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:37:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:37:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:37:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:37:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:45 smithi110 ceph-mon[97633]: pgmap v280: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:45 smithi104 ceph-mon[97044]: pgmap v280: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:47 smithi110 ceph-mon[97633]: pgmap v281: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:47 smithi104 ceph-mon[97044]: pgmap v281: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:49 smithi110 ceph-mon[97633]: pgmap v282: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:49 smithi104 ceph-mon[97044]: pgmap v282: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:51 smithi110 ceph-mon[97633]: pgmap v283: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:51 smithi104 ceph-mon[97044]: pgmap v283: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:53 smithi110 ceph-mon[97633]: pgmap v284: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:53 smithi104 ceph-mon[97044]: pgmap v284: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:55 smithi110 ceph-mon[97633]: pgmap v285: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:55 smithi104 ceph-mon[97044]: pgmap v285: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:57 smithi110 ceph-mon[97633]: pgmap v286: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:57 smithi104 ceph-mon[97044]: pgmap v286: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:37:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:37:59 smithi110 ceph-mon[97633]: pgmap v287: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:37:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:37:59 smithi104 ceph-mon[97044]: pgmap v287: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:01 smithi110 ceph-mon[97633]: pgmap v288: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:01 smithi104 ceph-mon[97044]: pgmap v288: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:03 smithi110 ceph-mon[97633]: pgmap v289: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:03 smithi104 ceph-mon[97044]: pgmap v289: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:05 smithi110 ceph-mon[97633]: pgmap v290: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:05 smithi104 ceph-mon[97044]: pgmap v290: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:07 smithi110 ceph-mon[97633]: pgmap v291: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:38:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:07 smithi104 ceph-mon[97044]: pgmap v291: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:38:08.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:08 smithi110 ceph-mon[97633]: osdmap e103: 12 total, 12 up, 12 in 2022-09-10T04:38:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:08 smithi104 ceph-mon[97044]: osdmap e103: 12 total, 12 up, 12 in 2022-09-10T04:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:09 smithi110 ceph-mon[97633]: pgmap v293: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:38:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:09 smithi104 ceph-mon[97044]: pgmap v293: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:38:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:11 smithi110 ceph-mon[97633]: pgmap v294: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:38:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:11 smithi104 ceph-mon[97044]: pgmap v294: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:38:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:13 smithi110 ceph-mon[97633]: pgmap v295: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:38:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:13 smithi110 ceph-mon[97633]: osdmap e104: 12 total, 12 up, 12 in 2022-09-10T04:38:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:13 smithi104 ceph-mon[97044]: pgmap v295: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:38:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:13 smithi104 ceph-mon[97044]: osdmap e104: 12 total, 12 up, 12 in 2022-09-10T04:38:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:15 smithi110 ceph-mon[97633]: pgmap v297: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:38:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:15 smithi104 ceph-mon[97044]: pgmap v297: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:38:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:17 smithi110 ceph-mon[97633]: pgmap v298: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:38:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:17 smithi104 ceph-mon[97044]: pgmap v298: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:38:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:19 smithi110 ceph-mon[97633]: pgmap v299: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:38:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:19 smithi104 ceph-mon[97044]: pgmap v299: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:38:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:21 smithi110 ceph-mon[97633]: pgmap v300: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:38:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:21 smithi104 ceph-mon[97044]: pgmap v300: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:38:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:23 smithi110 ceph-mon[97633]: pgmap v301: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:38:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:23 smithi104 ceph-mon[97044]: pgmap v301: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:38:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:25 smithi110 ceph-mon[97633]: pgmap v302: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:38:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:25 smithi104 ceph-mon[97044]: pgmap v302: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:38:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:27 smithi110 ceph-mon[97633]: pgmap v303: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:38:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:27 smithi104 ceph-mon[97044]: pgmap v303: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:38:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:29 smithi110 ceph-mon[97633]: pgmap v304: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:29 smithi104 ceph-mon[97044]: pgmap v304: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:31 smithi110 ceph-mon[97633]: pgmap v305: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:31 smithi104 ceph-mon[97044]: pgmap v305: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:33 smithi110 ceph-mon[97633]: pgmap v306: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:33 smithi104 ceph-mon[97044]: pgmap v306: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:35 smithi110 ceph-mon[97633]: pgmap v307: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:35 smithi104 ceph-mon[97044]: pgmap v307: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:37.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:37 smithi110 ceph-mon[97633]: pgmap v308: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:37 smithi104 ceph-mon[97044]: pgmap v308: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:39 smithi110 ceph-mon[97633]: pgmap v309: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:39 smithi104 ceph-mon[97044]: pgmap v309: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:41 smithi110 ceph-mon[97633]: pgmap v310: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:41 smithi104 ceph-mon[97044]: pgmap v310: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:43 smithi110 ceph-mon[97633]: pgmap v311: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:43 smithi104 ceph-mon[97044]: pgmap v311: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:44.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:38:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:38:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:38:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:38:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:38:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:38:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:38:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:38:45.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:45 smithi110 ceph-mon[97633]: pgmap v312: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:45 smithi104 ceph-mon[97044]: pgmap v312: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:46.635 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2022-09-10T04:38:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:47 smithi110 ceph-mon[97633]: pgmap v313: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:47 smithi104 ceph-mon[97044]: pgmap v313: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:49 smithi110 ceph-mon[97633]: pgmap v314: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:49 smithi104 ceph-mon[97044]: pgmap v314: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:51 smithi110 ceph-mon[97633]: pgmap v315: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:51 smithi104 ceph-mon[97044]: pgmap v315: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:53 smithi110 ceph-mon[97633]: pgmap v316: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:53 smithi104 ceph-mon[97044]: pgmap v316: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:55 smithi110 ceph-mon[97633]: pgmap v317: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:55 smithi104 ceph-mon[97044]: pgmap v317: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:57 smithi110 ceph-mon[97633]: pgmap v318: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:57 smithi104 ceph-mon[97044]: pgmap v318: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:38:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:38:59 smithi110 ceph-mon[97633]: pgmap v319: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:38:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:38:59 smithi104 ceph-mon[97044]: pgmap v319: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:01 smithi110 ceph-mon[97633]: pgmap v320: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:01 smithi104 ceph-mon[97044]: pgmap v320: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:03 smithi110 ceph-mon[97633]: pgmap v321: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:03 smithi104 ceph-mon[97044]: pgmap v321: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:05 smithi110 ceph-mon[97633]: pgmap v322: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:05 smithi104 ceph-mon[97044]: pgmap v322: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:07 smithi110 ceph-mon[97633]: pgmap v323: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:39:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:07 smithi104 ceph-mon[97044]: pgmap v323: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:39:08.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:08 smithi110 ceph-mon[97633]: osdmap e105: 12 total, 12 up, 12 in 2022-09-10T04:39:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:08 smithi104 ceph-mon[97044]: osdmap e105: 12 total, 12 up, 12 in 2022-09-10T04:39:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:09 smithi110 ceph-mon[97633]: pgmap v325: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:39:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:09 smithi104 ceph-mon[97044]: pgmap v325: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:39:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:11 smithi110 ceph-mon[97633]: pgmap v326: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:39:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:11 smithi104 ceph-mon[97044]: pgmap v326: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:39:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:13 smithi110 ceph-mon[97633]: pgmap v327: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:39:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:13 smithi110 ceph-mon[97633]: osdmap e106: 12 total, 12 up, 12 in 2022-09-10T04:39:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:13 smithi104 ceph-mon[97044]: pgmap v327: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:39:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:13 smithi104 ceph-mon[97044]: osdmap e106: 12 total, 12 up, 12 in 2022-09-10T04:39:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:15 smithi110 ceph-mon[97633]: pgmap v329: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:39:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:15 smithi104 ceph-mon[97044]: pgmap v329: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:39:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:17 smithi110 ceph-mon[97633]: pgmap v330: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:39:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:17 smithi104 ceph-mon[97044]: pgmap v330: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:39:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:19 smithi110 ceph-mon[97633]: pgmap v331: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:39:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:19 smithi104 ceph-mon[97044]: pgmap v331: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:39:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:21 smithi110 ceph-mon[97633]: pgmap v332: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:39:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:21 smithi104 ceph-mon[97044]: pgmap v332: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:39:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:23 smithi110 ceph-mon[97633]: pgmap v333: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:39:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:23 smithi104 ceph-mon[97044]: pgmap v333: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:39:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:25 smithi110 ceph-mon[97633]: pgmap v334: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:39:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:25 smithi104 ceph-mon[97044]: pgmap v334: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:39:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:27 smithi110 ceph-mon[97633]: pgmap v335: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:39:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:27 smithi104 ceph-mon[97044]: pgmap v335: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:39:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:29 smithi110 ceph-mon[97633]: pgmap v336: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:29 smithi104 ceph-mon[97044]: pgmap v336: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:31 smithi110 ceph-mon[97633]: pgmap v337: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:31 smithi104 ceph-mon[97044]: pgmap v337: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:33 smithi110 ceph-mon[97633]: pgmap v338: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:33 smithi104 ceph-mon[97044]: pgmap v338: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:35 smithi110 ceph-mon[97633]: pgmap v339: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:35 smithi104 ceph-mon[97044]: pgmap v339: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:37 smithi110 ceph-mon[97633]: pgmap v340: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:37 smithi104 ceph-mon[97044]: pgmap v340: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:39 smithi110 ceph-mon[97633]: pgmap v341: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:39 smithi104 ceph-mon[97044]: pgmap v341: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:41 smithi110 ceph-mon[97633]: pgmap v342: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:41 smithi104 ceph-mon[97044]: pgmap v342: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:43 smithi110 ceph-mon[97633]: pgmap v343: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:43 smithi104 ceph-mon[97044]: pgmap v343: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:39:44.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:39:44.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:39:44.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:39:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:39:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:39:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:39:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:39:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:45 smithi110 ceph-mon[97633]: pgmap v344: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:45 smithi104 ceph-mon[97044]: pgmap v344: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:47 smithi110 ceph-mon[97633]: pgmap v345: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:47 smithi104 ceph-mon[97044]: pgmap v345: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:49 smithi110 ceph-mon[97633]: pgmap v346: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:49 smithi104 ceph-mon[97044]: pgmap v346: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:51 smithi110 ceph-mon[97633]: pgmap v347: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:51 smithi104 ceph-mon[97044]: pgmap v347: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:53 smithi110 ceph-mon[97633]: pgmap v348: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:53 smithi104 ceph-mon[97044]: pgmap v348: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:55 smithi110 ceph-mon[97633]: pgmap v349: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:55 smithi104 ceph-mon[97044]: pgmap v349: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:57 smithi110 ceph-mon[97633]: pgmap v350: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:57 smithi104 ceph-mon[97044]: pgmap v350: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:39:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:39:59 smithi110 ceph-mon[97633]: pgmap v351: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:39:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:39:59 smithi104 ceph-mon[97044]: pgmap v351: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:00.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T04:40:00.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T04:40:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:01 smithi110 ceph-mon[97633]: pgmap v352: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:01.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:01 smithi104 ceph-mon[97044]: pgmap v352: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:03 smithi110 ceph-mon[97633]: pgmap v353: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:03 smithi104 ceph-mon[97044]: pgmap v353: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:05.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:05 smithi110 ceph-mon[97633]: pgmap v354: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:05 smithi104 ceph-mon[97044]: pgmap v354: 129 pgs: 129 active+clean; 7.7 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:07 smithi110 ceph-mon[97633]: pgmap v355: 129 pgs: 129 active+clean; 7.7 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:40:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:07 smithi104 ceph-mon[97044]: pgmap v355: 129 pgs: 129 active+clean; 7.7 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:40:08.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:08 smithi110 ceph-mon[97633]: osdmap e107: 12 total, 12 up, 12 in 2022-09-10T04:40:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:08 smithi104 ceph-mon[97044]: osdmap e107: 12 total, 12 up, 12 in 2022-09-10T04:40:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:09 smithi104 ceph-mon[97044]: pgmap v357: 129 pgs: 129 active+clean; 7.7 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:40:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:09 smithi110 ceph-mon[97633]: pgmap v357: 129 pgs: 129 active+clean; 7.7 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:40:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:11 smithi104 ceph-mon[97044]: pgmap v358: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:40:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:11 smithi110 ceph-mon[97633]: pgmap v358: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:40:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:13 smithi110 ceph-mon[97633]: pgmap v359: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:40:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:13 smithi110 ceph-mon[97633]: osdmap e108: 12 total, 12 up, 12 in 2022-09-10T04:40:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:13 smithi104 ceph-mon[97044]: pgmap v359: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:40:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:13 smithi104 ceph-mon[97044]: osdmap e108: 12 total, 12 up, 12 in 2022-09-10T04:40:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:15 smithi110 ceph-mon[97633]: pgmap v361: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:40:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:15 smithi104 ceph-mon[97044]: pgmap v361: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:40:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:17 smithi110 ceph-mon[97633]: pgmap v362: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:40:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:17 smithi104 ceph-mon[97044]: pgmap v362: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:40:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:19 smithi110 ceph-mon[97633]: pgmap v363: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:40:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:19 smithi104 ceph-mon[97044]: pgmap v363: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:40:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:21 smithi110 ceph-mon[97633]: pgmap v364: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:40:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:21 smithi104 ceph-mon[97044]: pgmap v364: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:40:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:23 smithi110 ceph-mon[97633]: pgmap v365: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:40:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:23 smithi104 ceph-mon[97044]: pgmap v365: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:40:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:25 smithi110 ceph-mon[97633]: pgmap v366: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:40:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:25 smithi104 ceph-mon[97044]: pgmap v366: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:40:27.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:27 smithi110 ceph-mon[97633]: pgmap v367: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:40:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:27 smithi104 ceph-mon[97044]: pgmap v367: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:40:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:29 smithi110 ceph-mon[97633]: pgmap v368: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:29 smithi104 ceph-mon[97044]: pgmap v368: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:31 smithi110 ceph-mon[97633]: pgmap v369: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:31 smithi104 ceph-mon[97044]: pgmap v369: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:33 smithi110 ceph-mon[97633]: pgmap v370: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:33 smithi104 ceph-mon[97044]: pgmap v370: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:35 smithi110 ceph-mon[97633]: pgmap v371: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:35 smithi104 ceph-mon[97044]: pgmap v371: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:37 smithi110 ceph-mon[97633]: pgmap v372: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:37 smithi104 ceph-mon[97044]: pgmap v372: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:39 smithi110 ceph-mon[97633]: pgmap v373: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:39 smithi104 ceph-mon[97044]: pgmap v373: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:41 smithi110 ceph-mon[97633]: pgmap v374: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:41 smithi104 ceph-mon[97044]: pgmap v374: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:43 smithi110 ceph-mon[97633]: pgmap v375: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:43 smithi104 ceph-mon[97044]: pgmap v375: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:40:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:40:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:40:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:40:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:40:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:40:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:40:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:40:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:45 smithi110 ceph-mon[97633]: pgmap v376: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:45 smithi104 ceph-mon[97044]: pgmap v376: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:47 smithi110 ceph-mon[97633]: pgmap v377: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:47 smithi104 ceph-mon[97044]: pgmap v377: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:49 smithi110 ceph-mon[97633]: pgmap v378: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:49 smithi104 ceph-mon[97044]: pgmap v378: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:51 smithi110 ceph-mon[97633]: pgmap v379: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:51 smithi104 ceph-mon[97044]: pgmap v379: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:53 smithi110 ceph-mon[97633]: pgmap v380: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:53 smithi104 ceph-mon[97044]: pgmap v380: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:55 smithi110 ceph-mon[97633]: pgmap v381: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:55 smithi104 ceph-mon[97044]: pgmap v381: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:40:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:57 smithi110 ceph-mon[97633]: pgmap v382: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:57 smithi104 ceph-mon[97044]: pgmap v382: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:40:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:40:59 smithi110 ceph-mon[97633]: pgmap v383: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:40:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:40:59 smithi104 ceph-mon[97044]: pgmap v383: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:01.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:01 smithi110 ceph-mon[97633]: pgmap v384: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:01 smithi104 ceph-mon[97044]: pgmap v384: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:03 smithi104 ceph-mon[97044]: pgmap v385: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:03 smithi110 ceph-mon[97633]: pgmap v385: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:05 smithi104 ceph-mon[97044]: pgmap v386: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:05 smithi110 ceph-mon[97633]: pgmap v386: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:07 smithi104 ceph-mon[97044]: pgmap v387: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:41:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:07 smithi110 ceph-mon[97633]: pgmap v387: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:41:08.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:08 smithi104 ceph-mon[97044]: osdmap e109: 12 total, 12 up, 12 in 2022-09-10T04:41:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:08 smithi110 ceph-mon[97633]: osdmap e109: 12 total, 12 up, 12 in 2022-09-10T04:41:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:09 smithi104 ceph-mon[97044]: pgmap v389: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:41:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:09 smithi110 ceph-mon[97633]: pgmap v389: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:41:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:11 smithi104 ceph-mon[97044]: pgmap v390: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:41:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:11 smithi110 ceph-mon[97633]: pgmap v390: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:41:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:13 smithi110 ceph-mon[97633]: pgmap v391: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:41:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:13 smithi110 ceph-mon[97633]: osdmap e110: 12 total, 12 up, 12 in 2022-09-10T04:41:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:13 smithi104 ceph-mon[97044]: pgmap v391: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:41:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:13 smithi104 ceph-mon[97044]: osdmap e110: 12 total, 12 up, 12 in 2022-09-10T04:41:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:15 smithi110 ceph-mon[97633]: pgmap v393: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:41:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:15 smithi104 ceph-mon[97044]: pgmap v393: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:41:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:17 smithi110 ceph-mon[97633]: pgmap v394: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:41:17.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:17 smithi104 ceph-mon[97044]: pgmap v394: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:41:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:19 smithi110 ceph-mon[97633]: pgmap v395: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:41:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:19 smithi104 ceph-mon[97044]: pgmap v395: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:41:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:21 smithi110 ceph-mon[97633]: pgmap v396: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:41:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:21 smithi104 ceph-mon[97044]: pgmap v396: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:41:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:23 smithi110 ceph-mon[97633]: pgmap v397: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:41:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:23 smithi104 ceph-mon[97044]: pgmap v397: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:41:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:25 smithi110 ceph-mon[97633]: pgmap v398: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:41:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:25 smithi104 ceph-mon[97044]: pgmap v398: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:41:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:27 smithi110 ceph-mon[97633]: pgmap v399: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:41:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:27 smithi104 ceph-mon[97044]: pgmap v399: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:41:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:29 smithi110 ceph-mon[97633]: pgmap v400: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:29 smithi104 ceph-mon[97044]: pgmap v400: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:31 smithi110 ceph-mon[97633]: pgmap v401: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:31 smithi104 ceph-mon[97044]: pgmap v401: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:33 smithi110 ceph-mon[97633]: pgmap v402: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:33 smithi104 ceph-mon[97044]: pgmap v402: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:35 smithi110 ceph-mon[97633]: pgmap v403: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:35 smithi104 ceph-mon[97044]: pgmap v403: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:37 smithi110 ceph-mon[97633]: pgmap v404: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:37 smithi104 ceph-mon[97044]: pgmap v404: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:39 smithi110 ceph-mon[97633]: pgmap v405: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:39 smithi104 ceph-mon[97044]: pgmap v405: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:41 smithi110 ceph-mon[97633]: pgmap v406: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:41 smithi104 ceph-mon[97044]: pgmap v406: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:43 smithi110 ceph-mon[97633]: pgmap v407: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:43 smithi104 ceph-mon[97044]: pgmap v407: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:41:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:41:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:41:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:41:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:41:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:41:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:41:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:41:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:45 smithi110 ceph-mon[97633]: pgmap v408: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:45 smithi104 ceph-mon[97044]: pgmap v408: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:47 smithi110 ceph-mon[97633]: pgmap v409: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:47 smithi104 ceph-mon[97044]: pgmap v409: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:49 smithi110 ceph-mon[97633]: pgmap v410: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:49 smithi104 ceph-mon[97044]: pgmap v410: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:51 smithi110 ceph-mon[97633]: pgmap v411: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:51 smithi104 ceph-mon[97044]: pgmap v411: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:53 smithi110 ceph-mon[97633]: pgmap v412: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:53 smithi104 ceph-mon[97044]: pgmap v412: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:55 smithi104 ceph-mon[97044]: pgmap v413: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:55 smithi110 ceph-mon[97633]: pgmap v413: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:41:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:57 smithi104 ceph-mon[97044]: pgmap v414: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:57 smithi110 ceph-mon[97633]: pgmap v414: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:41:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:41:59 smithi104 ceph-mon[97044]: pgmap v415: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:41:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:41:59 smithi110 ceph-mon[97633]: pgmap v415: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:01 smithi104 ceph-mon[97044]: pgmap v416: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:01 smithi110 ceph-mon[97633]: pgmap v416: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:03 smithi104 ceph-mon[97044]: pgmap v417: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:03 smithi110 ceph-mon[97633]: pgmap v417: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:05 smithi104 ceph-mon[97044]: pgmap v418: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:05 smithi110 ceph-mon[97633]: pgmap v418: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:07 smithi104 ceph-mon[97044]: pgmap v419: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:42:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:07 smithi110 ceph-mon[97633]: pgmap v419: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:42:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:08 smithi104 ceph-mon[97044]: osdmap e111: 12 total, 12 up, 12 in 2022-09-10T04:42:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:08 smithi110 ceph-mon[97633]: osdmap e111: 12 total, 12 up, 12 in 2022-09-10T04:42:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:09 smithi104 ceph-mon[97044]: pgmap v421: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:42:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:09 smithi110 ceph-mon[97633]: pgmap v421: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:42:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:11 smithi104 ceph-mon[97044]: pgmap v422: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:42:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:11 smithi110 ceph-mon[97633]: pgmap v422: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:42:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:13 smithi110 ceph-mon[97633]: pgmap v423: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:42:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:13 smithi110 ceph-mon[97633]: osdmap e112: 12 total, 12 up, 12 in 2022-09-10T04:42:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:13 smithi104 ceph-mon[97044]: pgmap v423: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:42:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:13 smithi104 ceph-mon[97044]: osdmap e112: 12 total, 12 up, 12 in 2022-09-10T04:42:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:15 smithi110 ceph-mon[97633]: pgmap v425: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:42:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:15 smithi104 ceph-mon[97044]: pgmap v425: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:42:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:17 smithi110 ceph-mon[97633]: pgmap v426: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:42:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:17 smithi104 ceph-mon[97044]: pgmap v426: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:42:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:19 smithi110 ceph-mon[97633]: pgmap v427: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:42:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:19 smithi104 ceph-mon[97044]: pgmap v427: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:42:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:21 smithi110 ceph-mon[97633]: pgmap v428: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:42:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:21 smithi104 ceph-mon[97044]: pgmap v428: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:42:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:23 smithi110 ceph-mon[97633]: pgmap v429: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:42:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:23 smithi104 ceph-mon[97044]: pgmap v429: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:42:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:25 smithi110 ceph-mon[97633]: pgmap v430: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:42:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:25 smithi104 ceph-mon[97044]: pgmap v430: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:42:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:27 smithi110 ceph-mon[97633]: pgmap v431: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:42:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:27 smithi104 ceph-mon[97044]: pgmap v431: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:42:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:29 smithi110 ceph-mon[97633]: pgmap v432: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:29.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:29 smithi104 ceph-mon[97044]: pgmap v432: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:31 smithi110 ceph-mon[97633]: pgmap v433: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:31 smithi104 ceph-mon[97044]: pgmap v433: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:33 smithi110 ceph-mon[97633]: pgmap v434: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:33 smithi104 ceph-mon[97044]: pgmap v434: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:35 smithi110 ceph-mon[97633]: pgmap v435: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:35 smithi104 ceph-mon[97044]: pgmap v435: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:37 smithi110 ceph-mon[97633]: pgmap v436: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:37 smithi104 ceph-mon[97044]: pgmap v436: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:39 smithi110 ceph-mon[97633]: pgmap v437: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:39 smithi104 ceph-mon[97044]: pgmap v437: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:41 smithi110 ceph-mon[97633]: pgmap v438: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:41 smithi104 ceph-mon[97044]: pgmap v438: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:43 smithi110 ceph-mon[97633]: pgmap v439: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:43 smithi104 ceph-mon[97044]: pgmap v439: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:42:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:42:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:42:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:42:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:42:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:42:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:42:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:42:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:45 smithi104 ceph-mon[97044]: pgmap v440: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:45 smithi110 ceph-mon[97633]: pgmap v440: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:47 smithi104 ceph-mon[97044]: pgmap v441: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:47 smithi110 ceph-mon[97633]: pgmap v441: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:49 smithi104 ceph-mon[97044]: pgmap v442: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:49 smithi110 ceph-mon[97633]: pgmap v442: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:51 smithi104 ceph-mon[97044]: pgmap v443: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:51 smithi110 ceph-mon[97633]: pgmap v443: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:53 smithi104 ceph-mon[97044]: pgmap v444: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:53 smithi110 ceph-mon[97633]: pgmap v444: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:55 smithi104 ceph-mon[97044]: pgmap v445: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:55 smithi110 ceph-mon[97633]: pgmap v445: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:57 smithi104 ceph-mon[97044]: pgmap v446: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:57 smithi110 ceph-mon[97633]: pgmap v446: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:42:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:42:59 smithi104 ceph-mon[97044]: pgmap v447: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:42:59.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:42:59 smithi110 ceph-mon[97633]: pgmap v447: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:01 smithi104 ceph-mon[97044]: pgmap v448: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:01 smithi110 ceph-mon[97633]: pgmap v448: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:03 smithi104 ceph-mon[97044]: pgmap v449: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:03 smithi110 ceph-mon[97633]: pgmap v449: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:05 smithi104 ceph-mon[97044]: pgmap v450: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:05 smithi110 ceph-mon[97633]: pgmap v450: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:07 smithi104 ceph-mon[97044]: pgmap v451: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:43:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:07 smithi110 ceph-mon[97633]: pgmap v451: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:43:08.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:08 smithi104 ceph-mon[97044]: osdmap e113: 12 total, 12 up, 12 in 2022-09-10T04:43:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:08 smithi110 ceph-mon[97633]: osdmap e113: 12 total, 12 up, 12 in 2022-09-10T04:43:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:09 smithi104 ceph-mon[97044]: pgmap v453: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:43:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:09 smithi110 ceph-mon[97633]: pgmap v453: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:43:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:11 smithi104 ceph-mon[97044]: pgmap v454: 129 pgs: 129 active+clean; 7.7 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:43:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:11 smithi110 ceph-mon[97633]: pgmap v454: 129 pgs: 129 active+clean; 7.7 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:43:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:13 smithi110 ceph-mon[97633]: pgmap v455: 129 pgs: 129 active+clean; 7.7 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:43:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:13 smithi110 ceph-mon[97633]: osdmap e114: 12 total, 12 up, 12 in 2022-09-10T04:43:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:13 smithi104 ceph-mon[97044]: pgmap v455: 129 pgs: 129 active+clean; 7.7 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:43:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:13 smithi104 ceph-mon[97044]: osdmap e114: 12 total, 12 up, 12 in 2022-09-10T04:43:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:15 smithi110 ceph-mon[97633]: pgmap v457: 129 pgs: 129 active+clean; 7.7 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:43:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:15 smithi104 ceph-mon[97044]: pgmap v457: 129 pgs: 129 active+clean; 7.7 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:43:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:17 smithi110 ceph-mon[97633]: pgmap v458: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:43:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:17 smithi104 ceph-mon[97044]: pgmap v458: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:43:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:19 smithi110 ceph-mon[97633]: pgmap v459: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:43:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:19 smithi104 ceph-mon[97044]: pgmap v459: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:43:21.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:21 smithi110 ceph-mon[97633]: pgmap v460: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:43:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:21 smithi104 ceph-mon[97044]: pgmap v460: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:43:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:23 smithi110 ceph-mon[97633]: pgmap v461: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:43:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:23 smithi104 ceph-mon[97044]: pgmap v461: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:43:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:25 smithi110 ceph-mon[97633]: pgmap v462: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:43:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:25 smithi104 ceph-mon[97044]: pgmap v462: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:43:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:27 smithi110 ceph-mon[97633]: pgmap v463: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:43:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:27 smithi104 ceph-mon[97044]: pgmap v463: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:43:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:29 smithi110 ceph-mon[97633]: pgmap v464: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:29 smithi104 ceph-mon[97044]: pgmap v464: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:31 smithi110 ceph-mon[97633]: pgmap v465: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:31 smithi104 ceph-mon[97044]: pgmap v465: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:33 smithi110 ceph-mon[97633]: pgmap v466: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:33 smithi104 ceph-mon[97044]: pgmap v466: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:35 smithi110 ceph-mon[97633]: pgmap v467: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:35 smithi104 ceph-mon[97044]: pgmap v467: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:37 smithi104 ceph-mon[97044]: pgmap v468: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:37 smithi110 ceph-mon[97633]: pgmap v468: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:39 smithi104 ceph-mon[97044]: pgmap v469: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:39 smithi110 ceph-mon[97633]: pgmap v469: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:41 smithi104 ceph-mon[97044]: pgmap v470: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:41 smithi110 ceph-mon[97633]: pgmap v470: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:43 smithi104 ceph-mon[97044]: pgmap v471: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:43 smithi110 ceph-mon[97633]: pgmap v471: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:43:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:43:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:43:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:43:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:44.241+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:44.442+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:44.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:43:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:43:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:43:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:43:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:44.843+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:45 smithi104 ceph-mon[97044]: pgmap v472: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:45 smithi110 ceph-mon[97633]: pgmap v472: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:46.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:45.644+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:46.799 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:43:46.799+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T04:43:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:47 smithi104 ceph-mon[97044]: pgmap v473: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:47.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:47.245+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:47 smithi110 ceph-mon[97633]: pgmap v473: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:49.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:49 smithi104 ceph-mon[97044]: pgmap v474: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:49 smithi110 ceph-mon[97633]: pgmap v474: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:50.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:50.448+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:51 smithi104 ceph-mon[97044]: pgmap v475: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:51 smithi110 ceph-mon[97633]: pgmap v475: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:53 smithi104 ceph-mon[97044]: pgmap v476: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:53 smithi110 ceph-mon[97633]: pgmap v476: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:55 smithi104 ceph-mon[97044]: pgmap v477: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:55 smithi110 ceph-mon[97633]: pgmap v477: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:43:56.855+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:43:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:57 smithi104 ceph-mon[97044]: pgmap v478: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:57 smithi110 ceph-mon[97633]: pgmap v478: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:43:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:43:59 smithi104 ceph-mon[97044]: pgmap v479: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:43:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:43:59 smithi110 ceph-mon[97633]: pgmap v479: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:01 smithi104 ceph-mon[97044]: pgmap v480: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:01 smithi110 ceph-mon[97633]: pgmap v480: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:03 smithi104 ceph-mon[97044]: pgmap v481: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:03 smithi110 ceph-mon[97633]: pgmap v481: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:05 smithi104 ceph-mon[97044]: pgmap v482: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:05 smithi110 ceph-mon[97633]: pgmap v482: 129 pgs: 129 active+clean; 7.7 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:07 smithi104 ceph-mon[97044]: pgmap v483: 129 pgs: 129 active+clean; 7.7 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:44:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:07 smithi110 ceph-mon[97633]: pgmap v483: 129 pgs: 129 active+clean; 7.7 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:44:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:08 smithi104 ceph-mon[97044]: osdmap e115: 12 total, 12 up, 12 in 2022-09-10T04:44:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:08 smithi110 ceph-mon[97633]: osdmap e115: 12 total, 12 up, 12 in 2022-09-10T04:44:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:09 smithi104 ceph-mon[97044]: pgmap v485: 129 pgs: 129 active+clean; 7.7 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:44:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:09 smithi110 ceph-mon[97633]: pgmap v485: 129 pgs: 129 active+clean; 7.7 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:44:10.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:44:09.668+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:44:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:11 smithi104 ceph-mon[97044]: pgmap v486: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:44:11.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:11 smithi110 ceph-mon[97633]: pgmap v486: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:44:13.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:13 smithi110 ceph-mon[97633]: pgmap v487: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:44:13.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:13 smithi110 ceph-mon[97633]: osdmap e116: 12 total, 12 up, 12 in 2022-09-10T04:44:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:13 smithi104 ceph-mon[97044]: pgmap v487: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:44:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:13 smithi104 ceph-mon[97044]: osdmap e116: 12 total, 12 up, 12 in 2022-09-10T04:44:15.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:15 smithi110 ceph-mon[97633]: pgmap v489: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:44:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:15 smithi104 ceph-mon[97044]: pgmap v489: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:44:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:17 smithi110 ceph-mon[97633]: pgmap v490: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:44:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:17 smithi104 ceph-mon[97044]: pgmap v490: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2022-09-10T04:44:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:19 smithi110 ceph-mon[97633]: pgmap v491: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:44:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:19 smithi104 ceph-mon[97044]: pgmap v491: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:44:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:21 smithi104 ceph-mon[97044]: pgmap v492: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:44:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:21 smithi110 ceph-mon[97633]: pgmap v492: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:44:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:23 smithi104 ceph-mon[97044]: pgmap v493: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:44:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:23 smithi110 ceph-mon[97633]: pgmap v493: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:44:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:44:24.680+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:44:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:25 smithi104 ceph-mon[97044]: pgmap v494: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:44:25.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:25 smithi110 ceph-mon[97633]: pgmap v494: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:44:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:27 smithi104 ceph-mon[97044]: pgmap v495: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:44:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:27 smithi110 ceph-mon[97633]: pgmap v495: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:44:29.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:29 smithi104 ceph-mon[97044]: pgmap v496: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:29 smithi110 ceph-mon[97633]: pgmap v496: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:31 smithi104 ceph-mon[97044]: pgmap v497: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:31 smithi110 ceph-mon[97633]: pgmap v497: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:33 smithi104 ceph-mon[97044]: pgmap v498: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:33 smithi110 ceph-mon[97633]: pgmap v498: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:35 smithi104 ceph-mon[97044]: pgmap v499: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:35 smithi110 ceph-mon[97633]: pgmap v499: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:37 smithi104 ceph-mon[97044]: pgmap v500: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:37 smithi110 ceph-mon[97633]: pgmap v500: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:39 smithi104 ceph-mon[97044]: pgmap v501: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:39 smithi110 ceph-mon[97633]: pgmap v501: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:44:39.692+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:44:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:41 smithi104 ceph-mon[97044]: pgmap v502: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:41 smithi110 ceph-mon[97633]: pgmap v502: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:43 smithi104 ceph-mon[97044]: pgmap v503: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:43 smithi110 ceph-mon[97633]: pgmap v503: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:44:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:44:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:44:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:44:44.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:44:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:44:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:44:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:44:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:45 smithi104 ceph-mon[97044]: pgmap v504: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:45 smithi110 ceph-mon[97633]: pgmap v504: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:47 smithi104 ceph-mon[97044]: pgmap v505: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:47 smithi110 ceph-mon[97633]: pgmap v505: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:49 smithi104 ceph-mon[97044]: pgmap v506: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:49 smithi110 ceph-mon[97633]: pgmap v506: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:51 smithi104 ceph-mon[97044]: pgmap v507: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:51 smithi110 ceph-mon[97633]: pgmap v507: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:53 smithi104 ceph-mon[97044]: pgmap v508: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:53 smithi110 ceph-mon[97633]: pgmap v508: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:55.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:44:54.695+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:44:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:55 smithi104 ceph-mon[97044]: pgmap v509: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:55 smithi110 ceph-mon[97633]: pgmap v509: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:57 smithi104 ceph-mon[97044]: pgmap v510: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:57 smithi110 ceph-mon[97633]: pgmap v510: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:44:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:44:59 smithi104 ceph-mon[97044]: pgmap v511: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:44:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:44:59 smithi110 ceph-mon[97633]: pgmap v511: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:01 smithi104 ceph-mon[97044]: pgmap v512: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:01 smithi110 ceph-mon[97633]: pgmap v512: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:03 smithi104 ceph-mon[97044]: pgmap v513: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:03 smithi110 ceph-mon[97633]: pgmap v513: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:05 smithi104 ceph-mon[97044]: pgmap v514: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:05 smithi110 ceph-mon[97633]: pgmap v514: 129 pgs: 129 active+clean; 7.7 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:07 smithi104 ceph-mon[97044]: pgmap v515: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:45:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:07 smithi110 ceph-mon[97633]: pgmap v515: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:45:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:08 smithi104 ceph-mon[97044]: osdmap e117: 12 total, 12 up, 12 in 2022-09-10T04:45:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:08 smithi110 ceph-mon[97633]: osdmap e117: 12 total, 12 up, 12 in 2022-09-10T04:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:09 smithi104 ceph-mon[97044]: pgmap v517: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:45:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:09 smithi110 ceph-mon[97633]: pgmap v517: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:45:10.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:45:09.707+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:45:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:11 smithi104 ceph-mon[97044]: pgmap v518: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:45:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:11 smithi110 ceph-mon[97633]: pgmap v518: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:45:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:13 smithi104 ceph-mon[97044]: pgmap v519: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:45:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:13 smithi104 ceph-mon[97044]: osdmap e118: 12 total, 12 up, 12 in 2022-09-10T04:45:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:13 smithi110 ceph-mon[97633]: pgmap v519: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:45:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:13 smithi110 ceph-mon[97633]: osdmap e118: 12 total, 12 up, 12 in 2022-09-10T04:45:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:15 smithi104 ceph-mon[97044]: pgmap v521: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:45:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:15 smithi110 ceph-mon[97633]: pgmap v521: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:45:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:17 smithi104 ceph-mon[97044]: pgmap v522: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:45:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:17 smithi110 ceph-mon[97633]: pgmap v522: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:45:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:19 smithi104 ceph-mon[97044]: pgmap v523: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:45:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:19 smithi110 ceph-mon[97633]: pgmap v523: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:45:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:21 smithi104 ceph-mon[97044]: pgmap v524: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:45:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:21 smithi110 ceph-mon[97633]: pgmap v524: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:45:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:23 smithi104 ceph-mon[97044]: pgmap v525: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:45:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:23 smithi110 ceph-mon[97633]: pgmap v525: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:45:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:45:24.718+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:45:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:25 smithi104 ceph-mon[97044]: pgmap v526: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:45:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:25 smithi110 ceph-mon[97633]: pgmap v526: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:45:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:27 smithi104 ceph-mon[97044]: pgmap v527: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:45:27.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:27 smithi110 ceph-mon[97633]: pgmap v527: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:45:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:29 smithi104 ceph-mon[97044]: pgmap v528: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:29 smithi110 ceph-mon[97633]: pgmap v528: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:31 smithi104 ceph-mon[97044]: pgmap v529: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:31 smithi110 ceph-mon[97633]: pgmap v529: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:33 smithi104 ceph-mon[97044]: pgmap v530: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:33.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:33 smithi110 ceph-mon[97633]: pgmap v530: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:35 smithi104 ceph-mon[97044]: pgmap v531: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:35 smithi110 ceph-mon[97633]: pgmap v531: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:37 smithi104 ceph-mon[97044]: pgmap v532: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:37 smithi110 ceph-mon[97633]: pgmap v532: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:39 smithi104 ceph-mon[97044]: pgmap v533: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:39 smithi110 ceph-mon[97633]: pgmap v533: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:45:39.730+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:45:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:41 smithi104 ceph-mon[97044]: pgmap v534: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:41 smithi110 ceph-mon[97633]: pgmap v534: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:43.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:43 smithi104 ceph-mon[97044]: pgmap v535: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:43 smithi110 ceph-mon[97633]: pgmap v535: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:45:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:45:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:45:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:45:44.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:45:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:45:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:45:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:45:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:45 smithi104 ceph-mon[97044]: pgmap v536: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:45 smithi110 ceph-mon[97633]: pgmap v536: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:47 smithi104 ceph-mon[97044]: pgmap v537: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:47 smithi110 ceph-mon[97633]: pgmap v537: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:49 smithi104 ceph-mon[97044]: pgmap v538: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:49 smithi110 ceph-mon[97633]: pgmap v538: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:51 smithi104 ceph-mon[97044]: pgmap v539: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:51 smithi110 ceph-mon[97633]: pgmap v539: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:53 smithi104 ceph-mon[97044]: pgmap v540: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:53 smithi110 ceph-mon[97633]: pgmap v540: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:55.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:45:54.743+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:45:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:55 smithi104 ceph-mon[97044]: pgmap v541: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:55 smithi110 ceph-mon[97633]: pgmap v541: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:57 smithi104 ceph-mon[97044]: pgmap v542: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:57 smithi110 ceph-mon[97633]: pgmap v542: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:45:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:45:59 smithi104 ceph-mon[97044]: pgmap v543: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:45:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:45:59 smithi110 ceph-mon[97633]: pgmap v543: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:01 smithi104 ceph-mon[97044]: pgmap v544: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:01 smithi110 ceph-mon[97633]: pgmap v544: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:03 smithi104 ceph-mon[97044]: pgmap v545: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:03 smithi110 ceph-mon[97633]: pgmap v545: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:05 smithi104 ceph-mon[97044]: pgmap v546: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:05 smithi110 ceph-mon[97633]: pgmap v546: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:07 smithi104 ceph-mon[97044]: pgmap v547: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:46:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:07 smithi110 ceph-mon[97633]: pgmap v547: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:46:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:08 smithi104 ceph-mon[97044]: osdmap e119: 12 total, 12 up, 12 in 2022-09-10T04:46:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:08 smithi110 ceph-mon[97633]: osdmap e119: 12 total, 12 up, 12 in 2022-09-10T04:46:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:09 smithi104 ceph-mon[97044]: pgmap v549: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:46:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:09 smithi110 ceph-mon[97633]: pgmap v549: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:46:10.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:46:09.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:46:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:11 smithi104 ceph-mon[97044]: pgmap v550: 129 pgs: 129 active+clean; 7.7 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:46:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:11 smithi110 ceph-mon[97633]: pgmap v550: 129 pgs: 129 active+clean; 7.7 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:46:13.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:13 smithi104 ceph-mon[97044]: pgmap v551: 129 pgs: 129 active+clean; 7.7 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:46:13.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:13 smithi104 ceph-mon[97044]: osdmap e120: 12 total, 12 up, 12 in 2022-09-10T04:46:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:13 smithi110 ceph-mon[97633]: pgmap v551: 129 pgs: 129 active+clean; 7.7 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:46:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:13 smithi110 ceph-mon[97633]: osdmap e120: 12 total, 12 up, 12 in 2022-09-10T04:46:15.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:15 smithi104 ceph-mon[97044]: pgmap v553: 129 pgs: 129 active+clean; 7.7 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:46:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:15 smithi110 ceph-mon[97633]: pgmap v553: 129 pgs: 129 active+clean; 7.7 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:46:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:17 smithi104 ceph-mon[97044]: pgmap v554: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:46:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:17 smithi110 ceph-mon[97633]: pgmap v554: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:46:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:19 smithi104 ceph-mon[97044]: pgmap v555: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:46:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:19 smithi110 ceph-mon[97633]: pgmap v555: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:46:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:21 smithi104 ceph-mon[97044]: pgmap v556: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:46:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:21 smithi110 ceph-mon[97633]: pgmap v556: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:46:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:23 smithi104 ceph-mon[97044]: pgmap v557: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:46:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:23 smithi110 ceph-mon[97633]: pgmap v557: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:46:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:46:24.768+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:46:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:25 smithi104 ceph-mon[97044]: pgmap v558: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:46:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:25 smithi110 ceph-mon[97633]: pgmap v558: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:46:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:27 smithi104 ceph-mon[97044]: pgmap v559: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:46:27.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:27 smithi110 ceph-mon[97633]: pgmap v559: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:46:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:29 smithi104 ceph-mon[97044]: pgmap v560: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:29 smithi110 ceph-mon[97633]: pgmap v560: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:31 smithi104 ceph-mon[97044]: pgmap v561: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:31 smithi110 ceph-mon[97633]: pgmap v561: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:33 smithi104 ceph-mon[97044]: pgmap v562: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:33 smithi110 ceph-mon[97633]: pgmap v562: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:35 smithi104 ceph-mon[97044]: pgmap v563: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:35 smithi110 ceph-mon[97633]: pgmap v563: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:37 smithi104 ceph-mon[97044]: pgmap v564: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:37 smithi110 ceph-mon[97633]: pgmap v564: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:39 smithi104 ceph-mon[97044]: pgmap v565: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:39 smithi110 ceph-mon[97633]: pgmap v565: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:46:39.783+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:46:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:41 smithi104 ceph-mon[97044]: pgmap v566: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:41 smithi110 ceph-mon[97633]: pgmap v566: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:43 smithi104 ceph-mon[97044]: pgmap v567: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:43 smithi110 ceph-mon[97633]: pgmap v567: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:44.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:46:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:46:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:46:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:46:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:46:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:46:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:46:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:46:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:45 smithi104 ceph-mon[97044]: pgmap v568: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:45 smithi110 ceph-mon[97633]: pgmap v568: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:47 smithi104 ceph-mon[97044]: pgmap v569: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:47 smithi110 ceph-mon[97633]: pgmap v569: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:49 smithi104 ceph-mon[97044]: pgmap v570: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:49 smithi110 ceph-mon[97633]: pgmap v570: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:51 smithi104 ceph-mon[97044]: pgmap v571: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:51 smithi110 ceph-mon[97633]: pgmap v571: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:53 smithi104 ceph-mon[97044]: pgmap v572: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:53 smithi110 ceph-mon[97633]: pgmap v572: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:55.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:46:54.798+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:46:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:55 smithi104 ceph-mon[97044]: pgmap v573: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:55 smithi110 ceph-mon[97633]: pgmap v573: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:57 smithi104 ceph-mon[97044]: pgmap v574: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:57 smithi110 ceph-mon[97633]: pgmap v574: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:46:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:46:59 smithi104 ceph-mon[97044]: pgmap v575: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:46:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:46:59 smithi110 ceph-mon[97633]: pgmap v575: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:01 smithi104 ceph-mon[97044]: pgmap v576: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:01 smithi110 ceph-mon[97633]: pgmap v576: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:03 smithi104 ceph-mon[97044]: pgmap v577: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:03 smithi110 ceph-mon[97633]: pgmap v577: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:05 smithi104 ceph-mon[97044]: pgmap v578: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:05 smithi110 ceph-mon[97633]: pgmap v578: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:07 smithi104 ceph-mon[97044]: pgmap v579: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:47:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:07 smithi110 ceph-mon[97633]: pgmap v579: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:47:08.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:08 smithi104 ceph-mon[97044]: osdmap e121: 12 total, 12 up, 12 in 2022-09-10T04:47:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:08 smithi110 ceph-mon[97633]: osdmap e121: 12 total, 12 up, 12 in 2022-09-10T04:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:09 smithi104 ceph-mon[97044]: pgmap v581: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:47:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:09 smithi110 ceph-mon[97633]: pgmap v581: 129 pgs: 129 active+clean; 7.7 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:47:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:47:09.810+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:47:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:11 smithi104 ceph-mon[97044]: pgmap v582: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:47:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:11 smithi110 ceph-mon[97633]: pgmap v582: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:47:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:13 smithi104 ceph-mon[97044]: pgmap v583: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:47:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:13 smithi104 ceph-mon[97044]: osdmap e122: 12 total, 12 up, 12 in 2022-09-10T04:47:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:13 smithi110 ceph-mon[97633]: pgmap v583: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:47:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:13 smithi110 ceph-mon[97633]: osdmap e122: 12 total, 12 up, 12 in 2022-09-10T04:47:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:15 smithi104 ceph-mon[97044]: pgmap v585: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:47:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:15 smithi110 ceph-mon[97633]: pgmap v585: 129 pgs: 129 active+clean; 7.7 MiB data, 296 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:47:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:17 smithi104 ceph-mon[97044]: pgmap v586: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:47:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:17 smithi110 ceph-mon[97633]: pgmap v586: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:47:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:19 smithi104 ceph-mon[97044]: pgmap v587: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:47:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:19 smithi110 ceph-mon[97633]: pgmap v587: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:47:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:21 smithi104 ceph-mon[97044]: pgmap v588: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:47:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:21 smithi110 ceph-mon[97633]: pgmap v588: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:47:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:23 smithi104 ceph-mon[97044]: pgmap v589: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:47:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:23 smithi110 ceph-mon[97633]: pgmap v589: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:47:25.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:47:24.825+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:47:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:25 smithi104 ceph-mon[97044]: pgmap v590: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:47:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:25 smithi110 ceph-mon[97633]: pgmap v590: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:47:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:27 smithi104 ceph-mon[97044]: pgmap v591: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:47:27.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:27 smithi110 ceph-mon[97633]: pgmap v591: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:47:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:29 smithi104 ceph-mon[97044]: pgmap v592: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:29 smithi110 ceph-mon[97633]: pgmap v592: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:31 smithi104 ceph-mon[97044]: pgmap v593: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:31 smithi110 ceph-mon[97633]: pgmap v593: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:33 smithi104 ceph-mon[97044]: pgmap v594: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:47:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:33 smithi110 ceph-mon[97633]: pgmap v594: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:47:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:35 smithi104 ceph-mon[97044]: pgmap v595: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:47:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:35 smithi110 ceph-mon[97633]: pgmap v595: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:47:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:37 smithi104 ceph-mon[97044]: pgmap v596: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:37 smithi110 ceph-mon[97633]: pgmap v596: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:39 smithi104 ceph-mon[97044]: pgmap v597: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:47:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:39 smithi110 ceph-mon[97633]: pgmap v597: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:47:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:47:39.840+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:47:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:41 smithi104 ceph-mon[97044]: pgmap v598: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:41 smithi110 ceph-mon[97633]: pgmap v598: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:43 smithi104 ceph-mon[97044]: pgmap v599: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:43 smithi110 ceph-mon[97633]: pgmap v599: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:47:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:47:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:47:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:47:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:47:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:47:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:47:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:47:45.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:45 smithi104 ceph-mon[97044]: pgmap v600: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:45 smithi110 ceph-mon[97633]: pgmap v600: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:47 smithi104 ceph-mon[97044]: pgmap v601: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:47 smithi110 ceph-mon[97633]: pgmap v601: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:49 smithi104 ceph-mon[97044]: pgmap v602: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:49 smithi110 ceph-mon[97633]: pgmap v602: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:51 smithi104 ceph-mon[97044]: pgmap v603: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:51 smithi110 ceph-mon[97633]: pgmap v603: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:53.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:53 smithi104 ceph-mon[97044]: pgmap v604: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:53 smithi110 ceph-mon[97633]: pgmap v604: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:47:54.856+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:47:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:55 smithi104 ceph-mon[97044]: pgmap v605: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:55.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:55 smithi110 ceph-mon[97633]: pgmap v605: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:57 smithi104 ceph-mon[97044]: pgmap v606: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:57 smithi110 ceph-mon[97633]: pgmap v606: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:47:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:47:59 smithi104 ceph-mon[97044]: pgmap v607: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:47:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:47:59 smithi110 ceph-mon[97633]: pgmap v607: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:01 smithi104 ceph-mon[97044]: pgmap v608: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:01 smithi110 ceph-mon[97633]: pgmap v608: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:03 smithi104 ceph-mon[97044]: pgmap v609: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:03 smithi110 ceph-mon[97633]: pgmap v609: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:05 smithi104 ceph-mon[97044]: pgmap v610: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:05 smithi110 ceph-mon[97633]: pgmap v610: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:07 smithi104 ceph-mon[97044]: pgmap v611: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:48:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:07 smithi110 ceph-mon[97633]: pgmap v611: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:48:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:08 smithi104 ceph-mon[97044]: osdmap e123: 12 total, 12 up, 12 in 2022-09-10T04:48:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:08 smithi110 ceph-mon[97633]: osdmap e123: 12 total, 12 up, 12 in 2022-09-10T04:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:09 smithi104 ceph-mon[97044]: pgmap v613: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:48:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:09 smithi110 ceph-mon[97633]: pgmap v613: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:48:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:48:09.871+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:48:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:11 smithi104 ceph-mon[97044]: pgmap v614: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:48:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:11 smithi110 ceph-mon[97633]: pgmap v614: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:13 smithi104 ceph-mon[97044]: pgmap v615: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:13 smithi104 ceph-mon[97044]: osdmap e124: 12 total, 12 up, 12 in 2022-09-10T04:48:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:13 smithi110 ceph-mon[97633]: pgmap v615: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:48:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:13 smithi110 ceph-mon[97633]: osdmap e124: 12 total, 12 up, 12 in 2022-09-10T04:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:15 smithi104 ceph-mon[97044]: pgmap v617: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:48:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:15 smithi110 ceph-mon[97633]: pgmap v617: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:17 smithi104 ceph-mon[97044]: pgmap v618: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:48:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:17 smithi110 ceph-mon[97633]: pgmap v618: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:48:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:19 smithi104 ceph-mon[97044]: pgmap v619: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:48:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:19 smithi110 ceph-mon[97633]: pgmap v619: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:48:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:21 smithi104 ceph-mon[97044]: pgmap v620: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:48:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:21 smithi110 ceph-mon[97633]: pgmap v620: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:48:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:23 smithi104 ceph-mon[97044]: pgmap v621: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:48:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:23 smithi110 ceph-mon[97633]: pgmap v621: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:48:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:48:24.886+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:48:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:25 smithi104 ceph-mon[97044]: pgmap v622: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:48:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:25 smithi110 ceph-mon[97633]: pgmap v622: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:48:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:27 smithi104 ceph-mon[97044]: pgmap v623: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:48:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:27 smithi110 ceph-mon[97633]: pgmap v623: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:48:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:29 smithi104 ceph-mon[97044]: pgmap v624: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:48:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:29 smithi110 ceph-mon[97633]: pgmap v624: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:48:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:31 smithi104 ceph-mon[97044]: pgmap v625: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:31 smithi110 ceph-mon[97633]: pgmap v625: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:33 smithi104 ceph-mon[97044]: pgmap v626: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:48:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:33 smithi110 ceph-mon[97633]: pgmap v626: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:48:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:35 smithi104 ceph-mon[97044]: pgmap v627: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:48:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:35 smithi110 ceph-mon[97633]: pgmap v627: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:48:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:37 smithi104 ceph-mon[97044]: pgmap v628: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:37 smithi110 ceph-mon[97633]: pgmap v628: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:39 smithi104 ceph-mon[97044]: pgmap v629: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:39 smithi110 ceph-mon[97633]: pgmap v629: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:48:39.902+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:48:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:41 smithi104 ceph-mon[97044]: pgmap v630: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:41 smithi110 ceph-mon[97633]: pgmap v630: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:43 smithi104 ceph-mon[97044]: pgmap v631: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:43 smithi110 ceph-mon[97633]: pgmap v631: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:48:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:48:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:48:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:48:44.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:48:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:48:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:48:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:48:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:45 smithi104 ceph-mon[97044]: pgmap v632: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:45 smithi110 ceph-mon[97633]: pgmap v632: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:46.800 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:48:46.799+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T04:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:47 smithi104 ceph-mon[97044]: pgmap v633: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:47 smithi110 ceph-mon[97633]: pgmap v633: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:49 smithi104 ceph-mon[97044]: pgmap v634: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:49 smithi110 ceph-mon[97633]: pgmap v634: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:51 smithi104 ceph-mon[97044]: pgmap v635: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:51 smithi110 ceph-mon[97633]: pgmap v635: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:53 smithi104 ceph-mon[97044]: pgmap v636: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:53 smithi110 ceph-mon[97633]: pgmap v636: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:55.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:48:54.913+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:48:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:55 smithi104 ceph-mon[97044]: pgmap v637: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:55 smithi110 ceph-mon[97633]: pgmap v637: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:57 smithi104 ceph-mon[97044]: pgmap v638: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:57 smithi110 ceph-mon[97633]: pgmap v638: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:48:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:48:59 smithi104 ceph-mon[97044]: pgmap v639: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:48:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:48:59 smithi110 ceph-mon[97633]: pgmap v639: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:01 smithi104 ceph-mon[97044]: pgmap v640: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:01 smithi110 ceph-mon[97633]: pgmap v640: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:03 smithi104 ceph-mon[97044]: pgmap v641: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:03 smithi110 ceph-mon[97633]: pgmap v641: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:05 smithi104 ceph-mon[97044]: pgmap v642: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:05 smithi110 ceph-mon[97633]: pgmap v642: 129 pgs: 129 active+clean; 7.7 MiB data, 299 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:07 smithi104 ceph-mon[97044]: pgmap v643: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:49:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:07 smithi110 ceph-mon[97633]: pgmap v643: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:49:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:08 smithi110 ceph-mon[97633]: osdmap e125: 12 total, 12 up, 12 in 2022-09-10T04:49:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:08 smithi104 ceph-mon[97044]: osdmap e125: 12 total, 12 up, 12 in 2022-09-10T04:49:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:09 smithi110 ceph-mon[97633]: pgmap v645: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:49:09.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:09 smithi104 ceph-mon[97044]: pgmap v645: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:49:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:49:09.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:49:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:11 smithi110 ceph-mon[97633]: pgmap v646: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:49:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:11 smithi104 ceph-mon[97044]: pgmap v646: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:49:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:13 smithi104 ceph-mon[97044]: pgmap v647: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:49:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:13 smithi104 ceph-mon[97044]: osdmap e126: 12 total, 12 up, 12 in 2022-09-10T04:49:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:13 smithi110 ceph-mon[97633]: pgmap v647: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:49:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:13 smithi110 ceph-mon[97633]: osdmap e126: 12 total, 12 up, 12 in 2022-09-10T04:49:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:15 smithi104 ceph-mon[97044]: pgmap v649: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:49:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:15 smithi110 ceph-mon[97633]: pgmap v649: 129 pgs: 129 active+clean; 7.7 MiB data, 301 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:49:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:17 smithi104 ceph-mon[97044]: pgmap v650: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:49:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:17 smithi110 ceph-mon[97633]: pgmap v650: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:49:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:19 smithi104 ceph-mon[97044]: pgmap v651: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:49:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:19 smithi110 ceph-mon[97633]: pgmap v651: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:49:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:21 smithi104 ceph-mon[97044]: pgmap v652: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:49:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:21 smithi110 ceph-mon[97633]: pgmap v652: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:49:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:23 smithi104 ceph-mon[97044]: pgmap v653: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:49:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:23 smithi110 ceph-mon[97633]: pgmap v653: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:49:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:49:24.940+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:49:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:25 smithi104 ceph-mon[97044]: pgmap v654: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:49:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:25 smithi110 ceph-mon[97633]: pgmap v654: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:49:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:27 smithi104 ceph-mon[97044]: pgmap v655: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:49:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:27 smithi110 ceph-mon[97633]: pgmap v655: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:49:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:29 smithi104 ceph-mon[97044]: pgmap v656: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:29 smithi110 ceph-mon[97633]: pgmap v656: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:31 smithi104 ceph-mon[97044]: pgmap v657: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:31 smithi110 ceph-mon[97633]: pgmap v657: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:33 smithi104 ceph-mon[97044]: pgmap v658: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:33 smithi110 ceph-mon[97633]: pgmap v658: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:35 smithi104 ceph-mon[97044]: pgmap v659: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:35 smithi110 ceph-mon[97633]: pgmap v659: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:37 smithi104 ceph-mon[97044]: pgmap v660: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:37 smithi110 ceph-mon[97633]: pgmap v660: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:39 smithi104 ceph-mon[97044]: pgmap v661: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:39 smithi110 ceph-mon[97633]: pgmap v661: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:49:39.955+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:49:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:41 smithi104 ceph-mon[97044]: pgmap v662: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:41 smithi110 ceph-mon[97633]: pgmap v662: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:43 smithi104 ceph-mon[97044]: pgmap v663: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:43 smithi110 ceph-mon[97633]: pgmap v663: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:49:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:49:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:49:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:49:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:49:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:49:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:49:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:49:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:45 smithi104 ceph-mon[97044]: pgmap v664: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:45 smithi110 ceph-mon[97633]: pgmap v664: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:47 smithi104 ceph-mon[97044]: pgmap v665: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:47 smithi110 ceph-mon[97633]: pgmap v665: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:49 smithi104 ceph-mon[97044]: pgmap v666: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:49 smithi110 ceph-mon[97633]: pgmap v666: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:51 smithi104 ceph-mon[97044]: pgmap v667: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:51 smithi110 ceph-mon[97633]: pgmap v667: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:53 smithi104 ceph-mon[97044]: pgmap v668: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:53 smithi110 ceph-mon[97633]: pgmap v668: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:49:54.957+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:49:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:55 smithi104 ceph-mon[97044]: pgmap v669: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:55 smithi110 ceph-mon[97633]: pgmap v669: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:57 smithi104 ceph-mon[97044]: pgmap v670: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:57 smithi110 ceph-mon[97633]: pgmap v670: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:49:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:49:59 smithi104 ceph-mon[97044]: pgmap v671: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:49:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:49:59 smithi110 ceph-mon[97633]: pgmap v671: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:00.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T04:50:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T04:50:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:01 smithi110 ceph-mon[97633]: pgmap v672: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:01 smithi104 ceph-mon[97044]: pgmap v672: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:03 smithi110 ceph-mon[97633]: pgmap v673: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:03 smithi104 ceph-mon[97044]: pgmap v673: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:05 smithi110 ceph-mon[97633]: pgmap v674: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:05 smithi104 ceph-mon[97044]: pgmap v674: 129 pgs: 129 active+clean; 7.7 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:07 smithi110 ceph-mon[97633]: pgmap v675: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:50:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:07 smithi104 ceph-mon[97044]: pgmap v675: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:50:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:08 smithi110 ceph-mon[97633]: osdmap e127: 12 total, 12 up, 12 in 2022-09-10T04:50:08.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:08 smithi104 ceph-mon[97044]: osdmap e127: 12 total, 12 up, 12 in 2022-09-10T04:50:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:09 smithi110 ceph-mon[97633]: pgmap v677: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:50:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:09 smithi104 ceph-mon[97044]: pgmap v677: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:50:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:50:09.973+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:50:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:11 smithi110 ceph-mon[97633]: pgmap v678: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:50:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:11 smithi104 ceph-mon[97044]: pgmap v678: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:50:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:13 smithi104 ceph-mon[97044]: pgmap v679: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:50:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:13 smithi104 ceph-mon[97044]: osdmap e128: 12 total, 12 up, 12 in 2022-09-10T04:50:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:13 smithi110 ceph-mon[97633]: pgmap v679: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:50:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:13 smithi110 ceph-mon[97633]: osdmap e128: 12 total, 12 up, 12 in 2022-09-10T04:50:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:15 smithi104 ceph-mon[97044]: pgmap v681: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:50:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:15 smithi110 ceph-mon[97633]: pgmap v681: 129 pgs: 129 active+clean; 7.7 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:50:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:17 smithi104 ceph-mon[97044]: pgmap v682: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:50:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:17 smithi110 ceph-mon[97633]: pgmap v682: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:50:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:19 smithi104 ceph-mon[97044]: pgmap v683: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:50:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:19 smithi110 ceph-mon[97633]: pgmap v683: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:50:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:21 smithi104 ceph-mon[97044]: pgmap v684: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:50:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:21 smithi110 ceph-mon[97633]: pgmap v684: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:50:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:23 smithi104 ceph-mon[97044]: pgmap v685: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:50:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:23 smithi110 ceph-mon[97633]: pgmap v685: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:50:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:50:24.978+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:50:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:25 smithi104 ceph-mon[97044]: pgmap v686: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:50:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:25 smithi110 ceph-mon[97633]: pgmap v686: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:50:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:27 smithi104 ceph-mon[97044]: pgmap v687: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:50:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:27 smithi110 ceph-mon[97633]: pgmap v687: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:50:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:29 smithi104 ceph-mon[97044]: pgmap v688: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:29 smithi110 ceph-mon[97633]: pgmap v688: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:31 smithi104 ceph-mon[97044]: pgmap v689: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:31 smithi110 ceph-mon[97633]: pgmap v689: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:33 smithi104 ceph-mon[97044]: pgmap v690: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:33 smithi110 ceph-mon[97633]: pgmap v690: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:35 smithi104 ceph-mon[97044]: pgmap v691: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:35 smithi110 ceph-mon[97633]: pgmap v691: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:37 smithi104 ceph-mon[97044]: pgmap v692: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:50:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:37 smithi110 ceph-mon[97633]: pgmap v692: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:50:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:39 smithi104 ceph-mon[97044]: pgmap v693: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T04:50:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:39 smithi110 ceph-mon[97633]: pgmap v693: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T04:50:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:50:39.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:50:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:41 smithi104 ceph-mon[97044]: pgmap v694: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:50:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:41 smithi110 ceph-mon[97633]: pgmap v694: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:50:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:43 smithi104 ceph-mon[97044]: pgmap v695: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T04:50:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:43 smithi110 ceph-mon[97633]: pgmap v695: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T04:50:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:50:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:50:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:50:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:50:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:50:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:50:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:50:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:50:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:45 smithi104 ceph-mon[97044]: pgmap v696: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T04:50:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:45 smithi110 ceph-mon[97633]: pgmap v696: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T04:50:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:47 smithi104 ceph-mon[97044]: pgmap v697: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:50:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:47 smithi110 ceph-mon[97633]: pgmap v697: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:50:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:49 smithi110 ceph-mon[97633]: pgmap v698: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:49 smithi104 ceph-mon[97044]: pgmap v698: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:51 smithi110 ceph-mon[97633]: pgmap v699: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:51 smithi104 ceph-mon[97044]: pgmap v699: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:53 smithi110 ceph-mon[97633]: pgmap v700: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:53 smithi104 ceph-mon[97044]: pgmap v700: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:50:54.985+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:50:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:55 smithi110 ceph-mon[97633]: pgmap v701: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:55 smithi104 ceph-mon[97044]: pgmap v701: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:57 smithi110 ceph-mon[97633]: pgmap v702: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:57 smithi104 ceph-mon[97044]: pgmap v702: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:50:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:50:59 smithi110 ceph-mon[97633]: pgmap v703: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:50:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:50:59 smithi104 ceph-mon[97044]: pgmap v703: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:01 smithi110 ceph-mon[97633]: pgmap v704: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:01 smithi104 ceph-mon[97044]: pgmap v704: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:03 smithi110 ceph-mon[97633]: pgmap v705: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:03 smithi104 ceph-mon[97044]: pgmap v705: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:05 smithi110 ceph-mon[97633]: pgmap v706: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:05 smithi104 ceph-mon[97044]: pgmap v706: 129 pgs: 129 active+clean; 7.7 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:07 smithi110 ceph-mon[97633]: pgmap v707: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:51:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:07 smithi104 ceph-mon[97044]: pgmap v707: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:51:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:08 smithi110 ceph-mon[97633]: osdmap e129: 12 total, 12 up, 12 in 2022-09-10T04:51:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:08 smithi104 ceph-mon[97044]: osdmap e129: 12 total, 12 up, 12 in 2022-09-10T04:51:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:09 smithi110 ceph-mon[97633]: pgmap v709: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:51:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:09 smithi104 ceph-mon[97044]: pgmap v709: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:51:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:51:09.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:51:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:11 smithi110 ceph-mon[97633]: pgmap v710: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:51:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:11 smithi104 ceph-mon[97044]: pgmap v710: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:51:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:13 smithi104 ceph-mon[97044]: pgmap v711: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:51:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:13 smithi104 ceph-mon[97044]: osdmap e130: 12 total, 12 up, 12 in 2022-09-10T04:51:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:13 smithi110 ceph-mon[97633]: pgmap v711: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:51:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:13 smithi110 ceph-mon[97633]: osdmap e130: 12 total, 12 up, 12 in 2022-09-10T04:51:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:15 smithi104 ceph-mon[97044]: pgmap v713: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:51:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:15 smithi110 ceph-mon[97633]: pgmap v713: 129 pgs: 129 active+clean; 7.7 MiB data, 309 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:51:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:17 smithi104 ceph-mon[97044]: pgmap v714: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:51:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:17 smithi110 ceph-mon[97633]: pgmap v714: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:51:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:19 smithi104 ceph-mon[97044]: pgmap v715: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:51:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:19 smithi110 ceph-mon[97633]: pgmap v715: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:51:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:21 smithi104 ceph-mon[97044]: pgmap v716: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:51:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:21 smithi110 ceph-mon[97633]: pgmap v716: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:51:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:23 smithi104 ceph-mon[97044]: pgmap v717: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:51:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:23 smithi110 ceph-mon[97633]: pgmap v717: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:51:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:51:24.994+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:51:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:25 smithi104 ceph-mon[97044]: pgmap v718: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:51:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:25 smithi110 ceph-mon[97633]: pgmap v718: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:51:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:27 smithi104 ceph-mon[97044]: pgmap v719: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:51:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:27 smithi110 ceph-mon[97633]: pgmap v719: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:51:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:29 smithi104 ceph-mon[97044]: pgmap v720: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:29 smithi110 ceph-mon[97633]: pgmap v720: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:31 smithi104 ceph-mon[97044]: pgmap v721: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:31 smithi110 ceph-mon[97633]: pgmap v721: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:33 smithi104 ceph-mon[97044]: pgmap v722: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:33 smithi110 ceph-mon[97633]: pgmap v722: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:35 smithi104 ceph-mon[97044]: pgmap v723: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:35 smithi110 ceph-mon[97633]: pgmap v723: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:37 smithi104 ceph-mon[97044]: pgmap v724: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:37 smithi110 ceph-mon[97633]: pgmap v724: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:39 smithi104 ceph-mon[97044]: pgmap v725: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:39 smithi110 ceph-mon[97633]: pgmap v725: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:51:39.998+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:51:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:41 smithi104 ceph-mon[97044]: pgmap v726: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:41 smithi110 ceph-mon[97633]: pgmap v726: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:43 smithi110 ceph-mon[97633]: pgmap v727: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:43 smithi104 ceph-mon[97044]: pgmap v727: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:51:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:51:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:51:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:51:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:51:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:51:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:51:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:51:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:45 smithi110 ceph-mon[97633]: pgmap v728: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:45 smithi104 ceph-mon[97044]: pgmap v728: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:47 smithi110 ceph-mon[97633]: pgmap v729: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:47 smithi104 ceph-mon[97044]: pgmap v729: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:49 smithi110 ceph-mon[97633]: pgmap v730: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:49 smithi104 ceph-mon[97044]: pgmap v730: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:51 smithi110 ceph-mon[97633]: pgmap v731: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:51 smithi104 ceph-mon[97044]: pgmap v731: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:53 smithi110 ceph-mon[97633]: pgmap v732: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:53 smithi104 ceph-mon[97044]: pgmap v732: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:51:55.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:51:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:55 smithi110 ceph-mon[97633]: pgmap v733: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:55 smithi104 ceph-mon[97044]: pgmap v733: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:57 smithi110 ceph-mon[97633]: pgmap v734: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:57 smithi104 ceph-mon[97044]: pgmap v734: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:51:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:51:59 smithi110 ceph-mon[97633]: pgmap v735: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:51:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:51:59 smithi104 ceph-mon[97044]: pgmap v735: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:01 smithi110 ceph-mon[97633]: pgmap v736: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:01 smithi104 ceph-mon[97044]: pgmap v736: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:03 smithi110 ceph-mon[97633]: pgmap v737: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:03 smithi104 ceph-mon[97044]: pgmap v737: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:05 smithi110 ceph-mon[97633]: pgmap v738: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:05 smithi104 ceph-mon[97044]: pgmap v738: 129 pgs: 129 active+clean; 7.7 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:07 smithi110 ceph-mon[97633]: pgmap v739: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:52:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:07 smithi104 ceph-mon[97044]: pgmap v739: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:52:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:08 smithi110 ceph-mon[97633]: osdmap e131: 12 total, 12 up, 12 in 2022-09-10T04:52:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:08 smithi104 ceph-mon[97044]: osdmap e131: 12 total, 12 up, 12 in 2022-09-10T04:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:09 smithi110 ceph-mon[97633]: pgmap v741: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:52:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:09 smithi104 ceph-mon[97044]: pgmap v741: 129 pgs: 129 active+clean; 7.7 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:52:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:52:10.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:52:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:11 smithi110 ceph-mon[97633]: pgmap v742: 129 pgs: 129 active+clean; 7.7 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:52:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:11 smithi104 ceph-mon[97044]: pgmap v742: 129 pgs: 129 active+clean; 7.7 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:52:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:13 smithi104 ceph-mon[97044]: pgmap v743: 129 pgs: 129 active+clean; 7.7 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:52:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:13 smithi104 ceph-mon[97044]: osdmap e132: 12 total, 12 up, 12 in 2022-09-10T04:52:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:13 smithi110 ceph-mon[97633]: pgmap v743: 129 pgs: 129 active+clean; 7.7 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:52:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:13 smithi110 ceph-mon[97633]: osdmap e132: 12 total, 12 up, 12 in 2022-09-10T04:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:15 smithi104 ceph-mon[97044]: pgmap v745: 129 pgs: 129 active+clean; 7.7 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:52:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:15 smithi110 ceph-mon[97633]: pgmap v745: 129 pgs: 129 active+clean; 7.7 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:52:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:17 smithi104 ceph-mon[97044]: pgmap v746: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:52:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:17 smithi110 ceph-mon[97633]: pgmap v746: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:52:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:19 smithi104 ceph-mon[97044]: pgmap v747: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:52:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:19 smithi110 ceph-mon[97633]: pgmap v747: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:52:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:21 smithi104 ceph-mon[97044]: pgmap v748: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:52:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:21 smithi110 ceph-mon[97633]: pgmap v748: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:52:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:23 smithi104 ceph-mon[97044]: pgmap v749: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:52:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:23 smithi110 ceph-mon[97633]: pgmap v749: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:52:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:52:25.009+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:52:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:25 smithi104 ceph-mon[97044]: pgmap v750: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:52:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:25 smithi110 ceph-mon[97633]: pgmap v750: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:52:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:27 smithi104 ceph-mon[97044]: pgmap v751: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:52:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:27 smithi110 ceph-mon[97633]: pgmap v751: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:52:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:29 smithi104 ceph-mon[97044]: pgmap v752: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:29 smithi110 ceph-mon[97633]: pgmap v752: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:31 smithi110 ceph-mon[97633]: pgmap v753: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:31 smithi104 ceph-mon[97044]: pgmap v753: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:33 smithi110 ceph-mon[97633]: pgmap v754: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:33 smithi104 ceph-mon[97044]: pgmap v754: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:35 smithi110 ceph-mon[97633]: pgmap v755: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:35 smithi104 ceph-mon[97044]: pgmap v755: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:37 smithi110 ceph-mon[97633]: pgmap v756: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:37 smithi104 ceph-mon[97044]: pgmap v756: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:39 smithi110 ceph-mon[97633]: pgmap v757: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:39 smithi104 ceph-mon[97044]: pgmap v757: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:52:40.014+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:52:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:41 smithi110 ceph-mon[97633]: pgmap v758: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:41 smithi104 ceph-mon[97044]: pgmap v758: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:43 smithi110 ceph-mon[97633]: pgmap v759: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:43 smithi104 ceph-mon[97044]: pgmap v759: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:52:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:52:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:52:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:52:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:52:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:52:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:52:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:52:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:45 smithi110 ceph-mon[97633]: pgmap v760: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:45 smithi104 ceph-mon[97044]: pgmap v760: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:47 smithi110 ceph-mon[97633]: pgmap v761: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:47 smithi104 ceph-mon[97044]: pgmap v761: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:49 smithi110 ceph-mon[97633]: pgmap v762: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:52:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:49 smithi104 ceph-mon[97044]: pgmap v762: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:52:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:51 smithi110 ceph-mon[97633]: pgmap v763: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:51 smithi104 ceph-mon[97044]: pgmap v763: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:53 smithi110 ceph-mon[97633]: pgmap v764: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:52:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:53 smithi104 ceph-mon[97044]: pgmap v764: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:52:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:52:55.019+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:52:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:55 smithi110 ceph-mon[97633]: pgmap v765: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:52:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:55 smithi104 ceph-mon[97044]: pgmap v765: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:52:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:57 smithi110 ceph-mon[97633]: pgmap v766: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:57 smithi104 ceph-mon[97044]: pgmap v766: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:52:59.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:52:59 smithi110 ceph-mon[97633]: pgmap v767: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:52:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:52:59 smithi104 ceph-mon[97044]: pgmap v767: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:01 smithi110 ceph-mon[97633]: pgmap v768: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:01 smithi104 ceph-mon[97044]: pgmap v768: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:03 smithi110 ceph-mon[97633]: pgmap v769: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:03 smithi104 ceph-mon[97044]: pgmap v769: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:05 smithi110 ceph-mon[97633]: pgmap v770: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:05 smithi104 ceph-mon[97044]: pgmap v770: 129 pgs: 129 active+clean; 7.7 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:07 smithi110 ceph-mon[97633]: pgmap v771: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:53:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:07 smithi104 ceph-mon[97044]: pgmap v771: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:53:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:08 smithi110 ceph-mon[97633]: osdmap e133: 12 total, 12 up, 12 in 2022-09-10T04:53:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:08 smithi104 ceph-mon[97044]: osdmap e133: 12 total, 12 up, 12 in 2022-09-10T04:53:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:09 smithi110 ceph-mon[97633]: pgmap v773: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:53:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:09 smithi104 ceph-mon[97044]: pgmap v773: 129 pgs: 129 active+clean; 7.7 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:53:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:53:10.025+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:53:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:11 smithi110 ceph-mon[97633]: pgmap v774: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:53:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:11 smithi104 ceph-mon[97044]: pgmap v774: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:53:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:13 smithi104 ceph-mon[97044]: pgmap v775: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:53:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:13 smithi104 ceph-mon[97044]: osdmap e134: 12 total, 12 up, 12 in 2022-09-10T04:53:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:13 smithi110 ceph-mon[97633]: pgmap v775: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:53:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:13 smithi110 ceph-mon[97633]: osdmap e134: 12 total, 12 up, 12 in 2022-09-10T04:53:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:15 smithi104 ceph-mon[97044]: pgmap v777: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:53:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:15 smithi110 ceph-mon[97633]: pgmap v777: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:53:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:17 smithi104 ceph-mon[97044]: pgmap v778: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:53:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:17 smithi110 ceph-mon[97633]: pgmap v778: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T04:53:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:19 smithi104 ceph-mon[97044]: pgmap v779: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:53:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:19 smithi110 ceph-mon[97633]: pgmap v779: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:53:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:21 smithi110 ceph-mon[97633]: pgmap v780: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:53:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:21 smithi104 ceph-mon[97044]: pgmap v780: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:53:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:23 smithi110 ceph-mon[97633]: pgmap v781: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:53:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:23 smithi104 ceph-mon[97044]: pgmap v781: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:53:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:53:25.027+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:53:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:25 smithi110 ceph-mon[97633]: pgmap v782: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:53:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:25 smithi104 ceph-mon[97044]: pgmap v782: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:53:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:27 smithi110 ceph-mon[97633]: pgmap v783: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:53:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:27 smithi104 ceph-mon[97044]: pgmap v783: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:53:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:29 smithi110 ceph-mon[97633]: pgmap v784: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:29 smithi104 ceph-mon[97044]: pgmap v784: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:31 smithi110 ceph-mon[97633]: pgmap v785: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:31 smithi104 ceph-mon[97044]: pgmap v785: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:33 smithi110 ceph-mon[97633]: pgmap v786: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:33 smithi104 ceph-mon[97044]: pgmap v786: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:35 smithi110 ceph-mon[97633]: pgmap v787: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:35 smithi104 ceph-mon[97044]: pgmap v787: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:37 smithi110 ceph-mon[97633]: pgmap v788: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:37 smithi104 ceph-mon[97044]: pgmap v788: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:39 smithi110 ceph-mon[97633]: pgmap v789: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:39 smithi104 ceph-mon[97044]: pgmap v789: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:53:40.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:53:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:41 smithi110 ceph-mon[97633]: pgmap v790: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:41 smithi104 ceph-mon[97044]: pgmap v790: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:43 smithi110 ceph-mon[97633]: pgmap v791: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:43 smithi104 ceph-mon[97044]: pgmap v791: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:53:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:53:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:53:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:53:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:53:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:53:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:53:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:53:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:45 smithi110 ceph-mon[97633]: pgmap v792: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:45 smithi104 ceph-mon[97044]: pgmap v792: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:46.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:53:46.800+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T04:53:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:47 smithi110 ceph-mon[97633]: pgmap v793: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:47 smithi104 ceph-mon[97044]: pgmap v793: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:49 smithi110 ceph-mon[97633]: pgmap v794: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:49 smithi104 ceph-mon[97044]: pgmap v794: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:51 smithi110 ceph-mon[97633]: pgmap v795: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:51 smithi104 ceph-mon[97044]: pgmap v795: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:53 smithi110 ceph-mon[97633]: pgmap v796: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:53 smithi104 ceph-mon[97044]: pgmap v796: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:53:55.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:53:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:55 smithi110 ceph-mon[97633]: pgmap v797: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:55 smithi104 ceph-mon[97044]: pgmap v797: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:57 smithi110 ceph-mon[97633]: pgmap v798: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:57 smithi104 ceph-mon[97044]: pgmap v798: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:53:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:53:59 smithi110 ceph-mon[97633]: pgmap v799: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:53:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:53:59 smithi104 ceph-mon[97044]: pgmap v799: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:01 smithi110 ceph-mon[97633]: pgmap v800: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:01 smithi104 ceph-mon[97044]: pgmap v800: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:03 smithi110 ceph-mon[97633]: pgmap v801: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:03 smithi104 ceph-mon[97044]: pgmap v801: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:05 smithi110 ceph-mon[97633]: pgmap v802: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:05 smithi104 ceph-mon[97044]: pgmap v802: 129 pgs: 129 active+clean; 7.7 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:07 smithi110 ceph-mon[97633]: pgmap v803: 129 pgs: 129 active+clean; 7.7 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:54:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:07 smithi104 ceph-mon[97044]: pgmap v803: 129 pgs: 129 active+clean; 7.7 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:54:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:08 smithi110 ceph-mon[97633]: osdmap e135: 12 total, 12 up, 12 in 2022-09-10T04:54:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:08 smithi104 ceph-mon[97044]: osdmap e135: 12 total, 12 up, 12 in 2022-09-10T04:54:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:09 smithi110 ceph-mon[97633]: pgmap v805: 129 pgs: 129 active+clean; 7.7 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:54:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:09 smithi104 ceph-mon[97044]: pgmap v805: 129 pgs: 129 active+clean; 7.7 MiB data, 332 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:54:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:54:10.038+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:54:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:11 smithi110 ceph-mon[97633]: pgmap v806: 129 pgs: 129 active+clean; 7.7 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:54:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:11 smithi104 ceph-mon[97044]: pgmap v806: 129 pgs: 129 active+clean; 7.7 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:54:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:13 smithi110 ceph-mon[97633]: pgmap v807: 129 pgs: 129 active+clean; 7.7 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:54:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:13 smithi110 ceph-mon[97633]: osdmap e136: 12 total, 12 up, 12 in 2022-09-10T04:54:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:13 smithi104 ceph-mon[97044]: pgmap v807: 129 pgs: 129 active+clean; 7.7 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:54:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:13 smithi104 ceph-mon[97044]: osdmap e136: 12 total, 12 up, 12 in 2022-09-10T04:54:15.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:15 smithi110 ceph-mon[97633]: pgmap v809: 129 pgs: 129 active+clean; 7.7 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:54:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:15 smithi104 ceph-mon[97044]: pgmap v809: 129 pgs: 129 active+clean; 7.7 MiB data, 336 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:54:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:17 smithi110 ceph-mon[97633]: pgmap v810: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:54:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:17 smithi104 ceph-mon[97044]: pgmap v810: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:54:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:19 smithi110 ceph-mon[97633]: pgmap v811: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:54:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:19 smithi104 ceph-mon[97044]: pgmap v811: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:54:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:21 smithi110 ceph-mon[97633]: pgmap v812: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:54:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:21 smithi104 ceph-mon[97044]: pgmap v812: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:54:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:23 smithi110 ceph-mon[97633]: pgmap v813: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:54:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:23 smithi104 ceph-mon[97044]: pgmap v813: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:54:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:54:25.049+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:54:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:25 smithi110 ceph-mon[97633]: pgmap v814: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:54:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:25 smithi104 ceph-mon[97044]: pgmap v814: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:54:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:27 smithi110 ceph-mon[97633]: pgmap v815: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:54:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:27 smithi104 ceph-mon[97044]: pgmap v815: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:54:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:29 smithi110 ceph-mon[97633]: pgmap v816: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:29 smithi104 ceph-mon[97044]: pgmap v816: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:31 smithi110 ceph-mon[97633]: pgmap v817: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:31 smithi104 ceph-mon[97044]: pgmap v817: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:33 smithi110 ceph-mon[97633]: pgmap v818: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:33 smithi104 ceph-mon[97044]: pgmap v818: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:35 smithi110 ceph-mon[97633]: pgmap v819: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:35 smithi104 ceph-mon[97044]: pgmap v819: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:37 smithi110 ceph-mon[97633]: pgmap v820: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:37 smithi104 ceph-mon[97044]: pgmap v820: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:39 smithi110 ceph-mon[97633]: pgmap v821: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:39 smithi104 ceph-mon[97044]: pgmap v821: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:54:40.049+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:54:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:41 smithi110 ceph-mon[97633]: pgmap v822: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:41 smithi104 ceph-mon[97044]: pgmap v822: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:43 smithi110 ceph-mon[97633]: pgmap v823: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:43 smithi104 ceph-mon[97044]: pgmap v823: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:54:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:54:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:54:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:54:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:54:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:45 smithi110 ceph-mon[97633]: pgmap v824: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:45 smithi104 ceph-mon[97044]: pgmap v824: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:47 smithi110 ceph-mon[97633]: pgmap v825: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:47 smithi104 ceph-mon[97044]: pgmap v825: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:49 smithi110 ceph-mon[97633]: pgmap v826: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:49 smithi104 ceph-mon[97044]: pgmap v826: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:51 smithi110 ceph-mon[97633]: pgmap v827: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:51 smithi104 ceph-mon[97044]: pgmap v827: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:53 smithi110 ceph-mon[97633]: pgmap v828: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:53 smithi104 ceph-mon[97044]: pgmap v828: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:55.561 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:54:55.057+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:54:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:55 smithi110 ceph-mon[97633]: pgmap v829: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:55 smithi104 ceph-mon[97044]: pgmap v829: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:57 smithi110 ceph-mon[97633]: pgmap v830: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:57 smithi104 ceph-mon[97044]: pgmap v830: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:54:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:54:59 smithi110 ceph-mon[97633]: pgmap v831: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:54:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:54:59 smithi104 ceph-mon[97044]: pgmap v831: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:01 smithi110 ceph-mon[97633]: pgmap v832: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:01 smithi104 ceph-mon[97044]: pgmap v832: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:03 smithi110 ceph-mon[97633]: pgmap v833: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:03 smithi104 ceph-mon[97044]: pgmap v833: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:05 smithi110 ceph-mon[97633]: pgmap v834: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:05 smithi104 ceph-mon[97044]: pgmap v834: 129 pgs: 129 active+clean; 7.7 MiB data, 337 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:07 smithi110 ceph-mon[97633]: pgmap v835: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:55:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:07 smithi104 ceph-mon[97044]: pgmap v835: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:55:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:08 smithi110 ceph-mon[97633]: osdmap e137: 12 total, 12 up, 12 in 2022-09-10T04:55:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:08 smithi104 ceph-mon[97044]: osdmap e137: 12 total, 12 up, 12 in 2022-09-10T04:55:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:09 smithi110 ceph-mon[97633]: pgmap v837: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:55:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:09 smithi104 ceph-mon[97044]: pgmap v837: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:55:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:55:10.060+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:55:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:11 smithi110 ceph-mon[97633]: pgmap v838: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:55:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:11 smithi104 ceph-mon[97044]: pgmap v838: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:55:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:13 smithi110 ceph-mon[97633]: pgmap v839: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:55:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:13 smithi110 ceph-mon[97633]: osdmap e138: 12 total, 12 up, 12 in 2022-09-10T04:55:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:13 smithi104 ceph-mon[97044]: pgmap v839: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:55:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:13 smithi104 ceph-mon[97044]: osdmap e138: 12 total, 12 up, 12 in 2022-09-10T04:55:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:15 smithi110 ceph-mon[97633]: pgmap v841: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:55:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:15 smithi104 ceph-mon[97044]: pgmap v841: 129 pgs: 129 active+clean; 7.7 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:55:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:17 smithi110 ceph-mon[97633]: pgmap v842: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:55:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:17 smithi104 ceph-mon[97044]: pgmap v842: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:55:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:19 smithi110 ceph-mon[97633]: pgmap v843: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:55:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:19 smithi104 ceph-mon[97044]: pgmap v843: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:55:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:21 smithi110 ceph-mon[97633]: pgmap v844: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:55:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:21 smithi104 ceph-mon[97044]: pgmap v844: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:55:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:23 smithi110 ceph-mon[97633]: pgmap v845: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:55:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:23 smithi104 ceph-mon[97044]: pgmap v845: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:55:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:55:25.067+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:55:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:25 smithi110 ceph-mon[97633]: pgmap v846: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:55:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:25 smithi104 ceph-mon[97044]: pgmap v846: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:55:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:27 smithi110 ceph-mon[97633]: pgmap v847: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:55:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:27 smithi104 ceph-mon[97044]: pgmap v847: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:55:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:29 smithi110 ceph-mon[97633]: pgmap v848: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:29 smithi104 ceph-mon[97044]: pgmap v848: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:31 smithi110 ceph-mon[97633]: pgmap v849: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:31 smithi104 ceph-mon[97044]: pgmap v849: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:33 smithi110 ceph-mon[97633]: pgmap v850: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:33 smithi104 ceph-mon[97044]: pgmap v850: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:35 smithi110 ceph-mon[97633]: pgmap v851: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:35 smithi104 ceph-mon[97044]: pgmap v851: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:37 smithi110 ceph-mon[97633]: pgmap v852: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:37 smithi104 ceph-mon[97044]: pgmap v852: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:39.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:39 smithi110 ceph-mon[97633]: pgmap v853: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:39 smithi104 ceph-mon[97044]: pgmap v853: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:55:40.068+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:55:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:41 smithi110 ceph-mon[97633]: pgmap v854: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:41 smithi104 ceph-mon[97044]: pgmap v854: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:43 smithi110 ceph-mon[97633]: pgmap v855: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:43 smithi104 ceph-mon[97044]: pgmap v855: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:55:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:55:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:55:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:55:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:55:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:45 smithi110 ceph-mon[97633]: pgmap v856: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:45 smithi104 ceph-mon[97044]: pgmap v856: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:47 smithi110 ceph-mon[97633]: pgmap v857: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:47 smithi104 ceph-mon[97044]: pgmap v857: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:49 smithi110 ceph-mon[97633]: pgmap v858: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:49 smithi104 ceph-mon[97044]: pgmap v858: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:51 smithi110 ceph-mon[97633]: pgmap v859: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:51 smithi104 ceph-mon[97044]: pgmap v859: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:53 smithi110 ceph-mon[97633]: pgmap v860: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:53 smithi104 ceph-mon[97044]: pgmap v860: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:55:55.074+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:55:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:55 smithi110 ceph-mon[97633]: pgmap v861: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:55 smithi104 ceph-mon[97044]: pgmap v861: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:57 smithi110 ceph-mon[97633]: pgmap v862: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:57 smithi104 ceph-mon[97044]: pgmap v862: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:55:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:55:59 smithi110 ceph-mon[97633]: pgmap v863: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:55:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:55:59 smithi104 ceph-mon[97044]: pgmap v863: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:01 smithi110 ceph-mon[97633]: pgmap v864: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:01 smithi104 ceph-mon[97044]: pgmap v864: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:03 smithi110 ceph-mon[97633]: pgmap v865: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:03 smithi104 ceph-mon[97044]: pgmap v865: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:05 smithi110 ceph-mon[97633]: pgmap v866: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:05 smithi104 ceph-mon[97044]: pgmap v866: 129 pgs: 129 active+clean; 7.7 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:07 smithi110 ceph-mon[97633]: pgmap v867: 129 pgs: 129 active+clean; 7.7 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:56:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:07 smithi104 ceph-mon[97044]: pgmap v867: 129 pgs: 129 active+clean; 7.7 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:56:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:08 smithi104 ceph-mon[97044]: osdmap e139: 12 total, 12 up, 12 in 2022-09-10T04:56:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:08 smithi110 ceph-mon[97633]: osdmap e139: 12 total, 12 up, 12 in 2022-09-10T04:56:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:09 smithi104 ceph-mon[97044]: pgmap v869: 129 pgs: 129 active+clean; 7.7 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:56:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:09 smithi110 ceph-mon[97633]: pgmap v869: 129 pgs: 129 active+clean; 7.7 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:56:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:56:10.085+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:56:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:11 smithi104 ceph-mon[97044]: pgmap v870: 129 pgs: 129 active+clean; 7.7 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:56:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:11 smithi110 ceph-mon[97633]: pgmap v870: 129 pgs: 129 active+clean; 7.7 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:56:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:13 smithi110 ceph-mon[97633]: pgmap v871: 129 pgs: 129 active+clean; 7.7 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:56:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:13 smithi110 ceph-mon[97633]: osdmap e140: 12 total, 12 up, 12 in 2022-09-10T04:56:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:13 smithi104 ceph-mon[97044]: pgmap v871: 129 pgs: 129 active+clean; 7.7 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T04:56:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:13 smithi104 ceph-mon[97044]: osdmap e140: 12 total, 12 up, 12 in 2022-09-10T04:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:15 smithi110 ceph-mon[97633]: pgmap v873: 129 pgs: 129 active+clean; 7.7 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:56:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:15 smithi104 ceph-mon[97044]: pgmap v873: 129 pgs: 129 active+clean; 7.7 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:56:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:17 smithi110 ceph-mon[97633]: pgmap v874: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:56:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:17 smithi104 ceph-mon[97044]: pgmap v874: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:56:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:19 smithi110 ceph-mon[97633]: pgmap v875: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:56:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:19 smithi104 ceph-mon[97044]: pgmap v875: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:56:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:21 smithi110 ceph-mon[97633]: pgmap v876: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:56:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:21 smithi104 ceph-mon[97044]: pgmap v876: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:56:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:23 smithi110 ceph-mon[97633]: pgmap v877: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:56:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:23 smithi104 ceph-mon[97044]: pgmap v877: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:56:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:56:25.087+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:56:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:25 smithi110 ceph-mon[97633]: pgmap v878: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:56:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:25 smithi104 ceph-mon[97044]: pgmap v878: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T04:56:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:27 smithi110 ceph-mon[97633]: pgmap v879: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:56:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:27 smithi104 ceph-mon[97044]: pgmap v879: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:56:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:29 smithi110 ceph-mon[97633]: pgmap v880: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:29 smithi104 ceph-mon[97044]: pgmap v880: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:31 smithi110 ceph-mon[97633]: pgmap v881: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:31 smithi104 ceph-mon[97044]: pgmap v881: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:33 smithi110 ceph-mon[97633]: pgmap v882: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:33 smithi104 ceph-mon[97044]: pgmap v882: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:35 smithi110 ceph-mon[97633]: pgmap v883: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:35 smithi104 ceph-mon[97044]: pgmap v883: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:37 smithi110 ceph-mon[97633]: pgmap v884: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:37 smithi104 ceph-mon[97044]: pgmap v884: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:39 smithi110 ceph-mon[97633]: pgmap v885: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:39 smithi104 ceph-mon[97044]: pgmap v885: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:56:40.096+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:56:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:41 smithi110 ceph-mon[97633]: pgmap v886: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:41 smithi104 ceph-mon[97044]: pgmap v886: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:43 smithi110 ceph-mon[97633]: pgmap v887: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:43 smithi104 ceph-mon[97044]: pgmap v887: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:56:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:56:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:56:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:56:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:56:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:56:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:56:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:56:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:45 smithi110 ceph-mon[97633]: pgmap v888: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:45 smithi104 ceph-mon[97044]: pgmap v888: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:47 smithi110 ceph-mon[97633]: pgmap v889: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:47 smithi104 ceph-mon[97044]: pgmap v889: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:49 smithi110 ceph-mon[97633]: pgmap v890: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:49 smithi104 ceph-mon[97044]: pgmap v890: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:51 smithi110 ceph-mon[97633]: pgmap v891: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:51 smithi104 ceph-mon[97044]: pgmap v891: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:53 smithi110 ceph-mon[97633]: pgmap v892: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:53 smithi104 ceph-mon[97044]: pgmap v892: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:56:55.099+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:56:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:55 smithi110 ceph-mon[97633]: pgmap v893: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:55 smithi104 ceph-mon[97044]: pgmap v893: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:57 smithi110 ceph-mon[97633]: pgmap v894: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:57 smithi104 ceph-mon[97044]: pgmap v894: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:56:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:56:59 smithi104 ceph-mon[97044]: pgmap v895: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:56:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:56:59 smithi110 ceph-mon[97633]: pgmap v895: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:01 smithi104 ceph-mon[97044]: pgmap v896: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:01 smithi110 ceph-mon[97633]: pgmap v896: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:03 smithi104 ceph-mon[97044]: pgmap v897: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:03 smithi110 ceph-mon[97633]: pgmap v897: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:05 smithi104 ceph-mon[97044]: pgmap v898: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:05 smithi110 ceph-mon[97633]: pgmap v898: 129 pgs: 129 active+clean; 7.7 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:07 smithi104 ceph-mon[97044]: pgmap v899: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:57:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:07 smithi110 ceph-mon[97633]: pgmap v899: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:57:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:08 smithi104 ceph-mon[97044]: osdmap e141: 12 total, 12 up, 12 in 2022-09-10T04:57:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:08 smithi110 ceph-mon[97633]: osdmap e141: 12 total, 12 up, 12 in 2022-09-10T04:57:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:09 smithi104 ceph-mon[97044]: pgmap v901: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:57:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:09 smithi110 ceph-mon[97633]: pgmap v901: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:57:10.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:57:10.107+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:57:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:11 smithi104 ceph-mon[97044]: pgmap v902: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:57:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:11 smithi110 ceph-mon[97633]: pgmap v902: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:57:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:13 smithi110 ceph-mon[97633]: pgmap v903: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:57:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:13 smithi110 ceph-mon[97633]: osdmap e142: 12 total, 12 up, 12 in 2022-09-10T04:57:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:13 smithi104 ceph-mon[97044]: pgmap v903: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:57:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:13 smithi104 ceph-mon[97044]: osdmap e142: 12 total, 12 up, 12 in 2022-09-10T04:57:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:15 smithi110 ceph-mon[97633]: pgmap v905: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:57:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:15 smithi104 ceph-mon[97044]: pgmap v905: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:57:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:17 smithi110 ceph-mon[97633]: pgmap v906: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:57:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:17 smithi104 ceph-mon[97044]: pgmap v906: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:57:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:19 smithi110 ceph-mon[97633]: pgmap v907: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:57:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:19 smithi104 ceph-mon[97044]: pgmap v907: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:57:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:21 smithi110 ceph-mon[97633]: pgmap v908: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:57:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:21 smithi104 ceph-mon[97044]: pgmap v908: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:57:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:23 smithi110 ceph-mon[97633]: pgmap v909: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:57:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:23 smithi104 ceph-mon[97044]: pgmap v909: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:57:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:57:25.108+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:57:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:25 smithi110 ceph-mon[97633]: pgmap v910: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:57:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:25 smithi104 ceph-mon[97044]: pgmap v910: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:57:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:27 smithi110 ceph-mon[97633]: pgmap v911: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:57:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:27 smithi104 ceph-mon[97044]: pgmap v911: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:57:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:29 smithi110 ceph-mon[97633]: pgmap v912: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:29 smithi104 ceph-mon[97044]: pgmap v912: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:31 smithi110 ceph-mon[97633]: pgmap v913: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:31 smithi104 ceph-mon[97044]: pgmap v913: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:33 smithi110 ceph-mon[97633]: pgmap v914: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:33 smithi104 ceph-mon[97044]: pgmap v914: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:35 smithi110 ceph-mon[97633]: pgmap v915: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:35 smithi104 ceph-mon[97044]: pgmap v915: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:37 smithi110 ceph-mon[97633]: pgmap v916: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:37 smithi104 ceph-mon[97044]: pgmap v916: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:39 smithi110 ceph-mon[97633]: pgmap v917: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:39 smithi104 ceph-mon[97044]: pgmap v917: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:57:40.111+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:57:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:41 smithi110 ceph-mon[97633]: pgmap v918: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:41 smithi104 ceph-mon[97044]: pgmap v918: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:43 smithi110 ceph-mon[97633]: pgmap v919: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:43 smithi104 ceph-mon[97044]: pgmap v919: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:57:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:57:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:57:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:45 smithi110 ceph-mon[97633]: pgmap v920: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:45 smithi104 ceph-mon[97044]: pgmap v920: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:47 smithi110 ceph-mon[97633]: pgmap v921: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:47 smithi104 ceph-mon[97044]: pgmap v921: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:49 smithi104 ceph-mon[97044]: pgmap v922: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:57:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:49 smithi110 ceph-mon[97633]: pgmap v922: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:57:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:51 smithi104 ceph-mon[97044]: pgmap v923: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:51 smithi110 ceph-mon[97633]: pgmap v923: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:53 smithi104 ceph-mon[97044]: pgmap v924: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:57:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:53 smithi110 ceph-mon[97633]: pgmap v924: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:57:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:57:55.119+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:57:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:55 smithi104 ceph-mon[97044]: pgmap v925: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:57:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:55 smithi110 ceph-mon[97633]: pgmap v925: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T04:57:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:57 smithi104 ceph-mon[97044]: pgmap v926: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:57 smithi110 ceph-mon[97633]: pgmap v926: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:57:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:57:59 smithi104 ceph-mon[97044]: pgmap v927: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:57:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:57:59 smithi110 ceph-mon[97633]: pgmap v927: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:01 smithi104 ceph-mon[97044]: pgmap v928: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:01 smithi110 ceph-mon[97633]: pgmap v928: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:03 smithi104 ceph-mon[97044]: pgmap v929: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:03 smithi110 ceph-mon[97633]: pgmap v929: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:05 smithi104 ceph-mon[97044]: pgmap v930: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:05 smithi110 ceph-mon[97633]: pgmap v930: 129 pgs: 129 active+clean; 7.7 MiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:07 smithi104 ceph-mon[97044]: pgmap v931: 129 pgs: 129 active+clean; 7.7 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:58:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:07 smithi110 ceph-mon[97633]: pgmap v931: 129 pgs: 129 active+clean; 7.7 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:58:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:08 smithi104 ceph-mon[97044]: osdmap e143: 12 total, 12 up, 12 in 2022-09-10T04:58:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:08 smithi110 ceph-mon[97633]: osdmap e143: 12 total, 12 up, 12 in 2022-09-10T04:58:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:09 smithi104 ceph-mon[97044]: pgmap v933: 129 pgs: 129 active+clean; 7.7 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:58:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:09 smithi110 ceph-mon[97633]: pgmap v933: 129 pgs: 129 active+clean; 7.7 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:58:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:58:10.120+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:58:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:11 smithi104 ceph-mon[97044]: pgmap v934: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:58:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:11 smithi110 ceph-mon[97633]: pgmap v934: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:58:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:13 smithi110 ceph-mon[97633]: pgmap v935: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:58:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:13 smithi110 ceph-mon[97633]: osdmap e144: 12 total, 12 up, 12 in 2022-09-10T04:58:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:13 smithi104 ceph-mon[97044]: pgmap v935: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:58:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:13 smithi104 ceph-mon[97044]: osdmap e144: 12 total, 12 up, 12 in 2022-09-10T04:58:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:15 smithi110 ceph-mon[97633]: pgmap v937: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:58:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:15 smithi104 ceph-mon[97044]: pgmap v937: 129 pgs: 129 active+clean; 7.7 MiB data, 297 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T04:58:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:17 smithi110 ceph-mon[97633]: pgmap v938: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:58:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:17 smithi104 ceph-mon[97044]: pgmap v938: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:58:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:19 smithi110 ceph-mon[97633]: pgmap v939: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:58:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:19 smithi104 ceph-mon[97044]: pgmap v939: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:58:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:21 smithi110 ceph-mon[97633]: pgmap v940: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:58:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:21 smithi104 ceph-mon[97044]: pgmap v940: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:58:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:23 smithi110 ceph-mon[97633]: pgmap v941: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:58:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:23 smithi104 ceph-mon[97044]: pgmap v941: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T04:58:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:58:25.121+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:58:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:25 smithi110 ceph-mon[97633]: pgmap v942: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:58:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:25 smithi104 ceph-mon[97044]: pgmap v942: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T04:58:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:27 smithi110 ceph-mon[97633]: pgmap v943: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:58:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:27 smithi104 ceph-mon[97044]: pgmap v943: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:58:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:29 smithi110 ceph-mon[97633]: pgmap v944: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:29.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:29 smithi104 ceph-mon[97044]: pgmap v944: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:31 smithi110 ceph-mon[97633]: pgmap v945: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:31 smithi104 ceph-mon[97044]: pgmap v945: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:33 smithi110 ceph-mon[97633]: pgmap v946: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:33 smithi104 ceph-mon[97044]: pgmap v946: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:35 smithi110 ceph-mon[97633]: pgmap v947: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:35 smithi104 ceph-mon[97044]: pgmap v947: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:37 smithi110 ceph-mon[97633]: pgmap v948: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:37 smithi104 ceph-mon[97044]: pgmap v948: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:39 smithi110 ceph-mon[97633]: pgmap v949: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:39 smithi104 ceph-mon[97044]: pgmap v949: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:58:40.125+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:58:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:41 smithi104 ceph-mon[97044]: pgmap v950: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:41.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:41 smithi110 ceph-mon[97633]: pgmap v950: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:58:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:43 smithi104 ceph-mon[97044]: pgmap v951: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:43 smithi110 ceph-mon[97633]: pgmap v951: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:58:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:58:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:58:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:58:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:58:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:58:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:58:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:58:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:45 smithi104 ceph-mon[97044]: pgmap v952: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:45 smithi110 ceph-mon[97633]: pgmap v952: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:46.801 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T04:58:46.801+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T04:58:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:47 smithi104 ceph-mon[97044]: pgmap v953: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:47 smithi110 ceph-mon[97633]: pgmap v953: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:49 smithi104 ceph-mon[97044]: pgmap v954: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T04:58:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:49 smithi110 ceph-mon[97633]: pgmap v954: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T04:58:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:51 smithi104 ceph-mon[97044]: pgmap v955: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:51 smithi110 ceph-mon[97633]: pgmap v955: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:53 smithi104 ceph-mon[97044]: pgmap v956: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T04:58:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:53 smithi110 ceph-mon[97633]: pgmap v956: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T04:58:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:58:55.131+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:58:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:55 smithi104 ceph-mon[97044]: pgmap v957: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:55 smithi110 ceph-mon[97633]: pgmap v957: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T04:58:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:57 smithi104 ceph-mon[97044]: pgmap v958: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:57 smithi110 ceph-mon[97633]: pgmap v958: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:58:59 smithi104 ceph-mon[97044]: pgmap v959: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:58:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:58:59 smithi110 ceph-mon[97633]: pgmap v959: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:01 smithi104 ceph-mon[97044]: pgmap v960: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:01 smithi110 ceph-mon[97633]: pgmap v960: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:03 smithi104 ceph-mon[97044]: pgmap v961: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:03 smithi110 ceph-mon[97633]: pgmap v961: 129 pgs: 129 active+clean; 7.7 MiB data, 298 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:05 smithi104 ceph-mon[97044]: pgmap v962: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:59:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:05 smithi110 ceph-mon[97633]: pgmap v962: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T04:59:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:07 smithi104 ceph-mon[97044]: pgmap v963: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:59:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:07 smithi110 ceph-mon[97633]: pgmap v963: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T04:59:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:08 smithi104 ceph-mon[97044]: osdmap e145: 12 total, 12 up, 12 in 2022-09-10T04:59:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:08 smithi110 ceph-mon[97633]: osdmap e145: 12 total, 12 up, 12 in 2022-09-10T04:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:09 smithi104 ceph-mon[97044]: pgmap v965: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:59:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:09 smithi110 ceph-mon[97633]: pgmap v965: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:59:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:59:10.133+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:59:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:11 smithi104 ceph-mon[97044]: pgmap v966: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:59:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:11 smithi110 ceph-mon[97633]: pgmap v966: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:59:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:13 smithi110 ceph-mon[97633]: pgmap v967: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:59:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:13 smithi110 ceph-mon[97633]: osdmap e146: 12 total, 12 up, 12 in 2022-09-10T04:59:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:13 smithi104 ceph-mon[97044]: pgmap v967: 129 pgs: 129 active+clean; 7.8 MiB data, 303 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T04:59:13.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:13 smithi104 ceph-mon[97044]: osdmap e146: 12 total, 12 up, 12 in 2022-09-10T04:59:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:15 smithi110 ceph-mon[97633]: pgmap v969: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:59:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:15 smithi104 ceph-mon[97044]: pgmap v969: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T04:59:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:17 smithi110 ceph-mon[97633]: pgmap v970: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:59:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:17 smithi104 ceph-mon[97044]: pgmap v970: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T04:59:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:19 smithi110 ceph-mon[97633]: pgmap v971: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:59:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:19 smithi104 ceph-mon[97044]: pgmap v971: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T04:59:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:21 smithi110 ceph-mon[97633]: pgmap v972: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T04:59:21.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:21 smithi104 ceph-mon[97044]: pgmap v972: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T04:59:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:23 smithi110 ceph-mon[97633]: pgmap v973: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T04:59:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:23 smithi104 ceph-mon[97044]: pgmap v973: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T04:59:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:59:25.138+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:59:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:25 smithi110 ceph-mon[97633]: pgmap v974: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:59:25.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:25 smithi104 ceph-mon[97044]: pgmap v974: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T04:59:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:27 smithi110 ceph-mon[97633]: pgmap v975: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:27 smithi104 ceph-mon[97044]: pgmap v975: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:29 smithi110 ceph-mon[97633]: pgmap v976: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:29 smithi104 ceph-mon[97044]: pgmap v976: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:31 smithi104 ceph-mon[97044]: pgmap v977: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:31 smithi110 ceph-mon[97633]: pgmap v977: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:33 smithi104 ceph-mon[97044]: pgmap v978: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:33 smithi110 ceph-mon[97633]: pgmap v978: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:35 smithi104 ceph-mon[97044]: pgmap v979: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:35 smithi110 ceph-mon[97633]: pgmap v979: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:37 smithi104 ceph-mon[97044]: pgmap v980: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:37 smithi110 ceph-mon[97633]: pgmap v980: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:39 smithi104 ceph-mon[97044]: pgmap v981: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:39.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:39 smithi110 ceph-mon[97633]: pgmap v981: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:59:40.140+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:59:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:41 smithi104 ceph-mon[97044]: pgmap v982: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:41 smithi110 ceph-mon[97633]: pgmap v982: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:43 smithi104 ceph-mon[97044]: pgmap v983: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:43 smithi110 ceph-mon[97633]: pgmap v983: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:59:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T04:59:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T04:59:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:45 smithi104 ceph-mon[97044]: pgmap v984: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:45 smithi110 ceph-mon[97633]: pgmap v984: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:47 smithi104 ceph-mon[97044]: pgmap v985: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:47 smithi110 ceph-mon[97633]: pgmap v985: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:49 smithi104 ceph-mon[97044]: pgmap v986: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:49 smithi110 ceph-mon[97633]: pgmap v986: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:51 smithi104 ceph-mon[97044]: pgmap v987: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:51 smithi110 ceph-mon[97633]: pgmap v987: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:53 smithi104 ceph-mon[97044]: pgmap v988: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:53 smithi110 ceph-mon[97633]: pgmap v988: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 04:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T04:59:55.141+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T04:59:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:55 smithi104 ceph-mon[97044]: pgmap v989: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:55 smithi110 ceph-mon[97633]: pgmap v989: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T04:59:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:57 smithi104 ceph-mon[97044]: pgmap v990: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:57 smithi110 ceph-mon[97633]: pgmap v990: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 04:59:59 smithi104 ceph-mon[97044]: pgmap v991: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T04:59:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 04:59:59 smithi110 ceph-mon[97633]: pgmap v991: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:00:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:00:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:01 smithi104 ceph-mon[97044]: pgmap v992: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:01 smithi110 ceph-mon[97633]: pgmap v992: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:03 smithi104 ceph-mon[97044]: pgmap v993: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:03.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:03 smithi110 ceph-mon[97633]: pgmap v993: 129 pgs: 129 active+clean; 7.8 MiB data, 304 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:05 smithi104 ceph-mon[97044]: pgmap v994: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:00:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:05 smithi110 ceph-mon[97633]: pgmap v994: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:00:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:07 smithi104 ceph-mon[97044]: pgmap v995: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:00:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:07 smithi110 ceph-mon[97633]: pgmap v995: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:00:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:08 smithi104 ceph-mon[97044]: osdmap e147: 12 total, 12 up, 12 in 2022-09-10T05:00:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:08 smithi110 ceph-mon[97633]: osdmap e147: 12 total, 12 up, 12 in 2022-09-10T05:00:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:09 smithi104 ceph-mon[97044]: pgmap v997: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:00:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:09 smithi110 ceph-mon[97633]: pgmap v997: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:00:10.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:00:10.150+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:00:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:11 smithi104 ceph-mon[97044]: pgmap v998: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:00:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:11 smithi110 ceph-mon[97633]: pgmap v998: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:00:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:13 smithi110 ceph-mon[97633]: pgmap v999: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:00:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:13 smithi110 ceph-mon[97633]: osdmap e148: 12 total, 12 up, 12 in 2022-09-10T05:00:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:13 smithi104 ceph-mon[97044]: pgmap v999: 129 pgs: 129 active+clean; 7.8 MiB data, 305 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:00:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:13 smithi104 ceph-mon[97044]: osdmap e148: 12 total, 12 up, 12 in 2022-09-10T05:00:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:15 smithi110 ceph-mon[97633]: pgmap v1001: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:00:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:15 smithi104 ceph-mon[97044]: pgmap v1001: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:00:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:17 smithi110 ceph-mon[97633]: pgmap v1002: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:00:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:17 smithi104 ceph-mon[97044]: pgmap v1002: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:00:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:19 smithi104 ceph-mon[97044]: pgmap v1003: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:00:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:19 smithi110 ceph-mon[97633]: pgmap v1003: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:00:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:21 smithi104 ceph-mon[97044]: pgmap v1004: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:00:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:21 smithi110 ceph-mon[97633]: pgmap v1004: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:00:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:23 smithi104 ceph-mon[97044]: pgmap v1005: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:00:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:23 smithi110 ceph-mon[97633]: pgmap v1005: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:00:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:00:25.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:00:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:25 smithi104 ceph-mon[97044]: pgmap v1006: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:00:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:25 smithi110 ceph-mon[97633]: pgmap v1006: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:00:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:27 smithi104 ceph-mon[97044]: pgmap v1007: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:27 smithi110 ceph-mon[97633]: pgmap v1007: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:29 smithi104 ceph-mon[97044]: pgmap v1008: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:29 smithi110 ceph-mon[97633]: pgmap v1008: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:31 smithi104 ceph-mon[97044]: pgmap v1009: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:31 smithi110 ceph-mon[97633]: pgmap v1009: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:33 smithi104 ceph-mon[97044]: pgmap v1010: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:33 smithi110 ceph-mon[97633]: pgmap v1010: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:35 smithi104 ceph-mon[97044]: pgmap v1011: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:35 smithi110 ceph-mon[97633]: pgmap v1011: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:37 smithi104 ceph-mon[97044]: pgmap v1012: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:37 smithi110 ceph-mon[97633]: pgmap v1012: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:39 smithi104 ceph-mon[97044]: pgmap v1013: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:39 smithi110 ceph-mon[97633]: pgmap v1013: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:00:40.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:00:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:41 smithi104 ceph-mon[97044]: pgmap v1014: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:41 smithi110 ceph-mon[97633]: pgmap v1014: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:43 smithi104 ceph-mon[97044]: pgmap v1015: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:43.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:43 smithi110 ceph-mon[97633]: pgmap v1015: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:00:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:00:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:45 smithi104 ceph-mon[97044]: pgmap v1016: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:45 smithi110 ceph-mon[97633]: pgmap v1016: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:47 smithi104 ceph-mon[97044]: pgmap v1017: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:00:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:47 smithi110 ceph-mon[97633]: pgmap v1017: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:00:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:49 smithi104 ceph-mon[97044]: pgmap v1018: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:00:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:49 smithi110 ceph-mon[97633]: pgmap v1018: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:00:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:51 smithi104 ceph-mon[97044]: pgmap v1019: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:51.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:51 smithi110 ceph-mon[97633]: pgmap v1019: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:53 smithi104 ceph-mon[97044]: pgmap v1020: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:00:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:53 smithi110 ceph-mon[97633]: pgmap v1020: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:00:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:00:55.155+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:00:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:55 smithi104 ceph-mon[97044]: pgmap v1021: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:55 smithi110 ceph-mon[97633]: pgmap v1021: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:00:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:57 smithi104 ceph-mon[97044]: pgmap v1022: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:57 smithi110 ceph-mon[97633]: pgmap v1022: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:00:59 smithi104 ceph-mon[97044]: pgmap v1023: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:00:59.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:00:59 smithi110 ceph-mon[97633]: pgmap v1023: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:01 smithi104 ceph-mon[97044]: pgmap v1024: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:01 smithi110 ceph-mon[97633]: pgmap v1024: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:03 smithi104 ceph-mon[97044]: pgmap v1025: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:03 smithi110 ceph-mon[97633]: pgmap v1025: 129 pgs: 129 active+clean; 7.8 MiB data, 306 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:05 smithi104 ceph-mon[97044]: pgmap v1026: 129 pgs: 129 active+clean; 7.8 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:01:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:05 smithi110 ceph-mon[97633]: pgmap v1026: 129 pgs: 129 active+clean; 7.8 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:01:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:07 smithi104 ceph-mon[97044]: pgmap v1027: 129 pgs: 129 active+clean; 7.8 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:01:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:07 smithi110 ceph-mon[97633]: pgmap v1027: 129 pgs: 129 active+clean; 7.8 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:01:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:08 smithi104 ceph-mon[97044]: osdmap e149: 12 total, 12 up, 12 in 2022-09-10T05:01:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:08 smithi110 ceph-mon[97633]: osdmap e149: 12 total, 12 up, 12 in 2022-09-10T05:01:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:09 smithi104 ceph-mon[97044]: pgmap v1029: 129 pgs: 129 active+clean; 7.8 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:01:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:09 smithi110 ceph-mon[97633]: pgmap v1029: 129 pgs: 129 active+clean; 7.8 MiB data, 307 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:01:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:01:10.158+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:01:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:11 smithi104 ceph-mon[97044]: pgmap v1030: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:01:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:11 smithi110 ceph-mon[97633]: pgmap v1030: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:01:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:13 smithi104 ceph-mon[97044]: pgmap v1031: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:01:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:13 smithi104 ceph-mon[97044]: osdmap e150: 12 total, 12 up, 12 in 2022-09-10T05:01:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:13 smithi110 ceph-mon[97633]: pgmap v1031: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:01:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:13 smithi110 ceph-mon[97633]: osdmap e150: 12 total, 12 up, 12 in 2022-09-10T05:01:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:15 smithi104 ceph-mon[97044]: pgmap v1033: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:01:15.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:15 smithi110 ceph-mon[97633]: pgmap v1033: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:01:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:17 smithi104 ceph-mon[97044]: pgmap v1034: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:01:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:17 smithi110 ceph-mon[97633]: pgmap v1034: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:01:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:19 smithi104 ceph-mon[97044]: pgmap v1035: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:01:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:19 smithi110 ceph-mon[97633]: pgmap v1035: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:01:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:21 smithi104 ceph-mon[97044]: pgmap v1036: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:01:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:21 smithi110 ceph-mon[97633]: pgmap v1036: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:01:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:23 smithi104 ceph-mon[97044]: pgmap v1037: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:01:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:23 smithi110 ceph-mon[97633]: pgmap v1037: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:01:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:01:25.158+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:01:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:25 smithi104 ceph-mon[97044]: pgmap v1038: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:01:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:25 smithi110 ceph-mon[97633]: pgmap v1038: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:01:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:27 smithi104 ceph-mon[97044]: pgmap v1039: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:27 smithi110 ceph-mon[97633]: pgmap v1039: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:29 smithi104 ceph-mon[97044]: pgmap v1040: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:29 smithi110 ceph-mon[97633]: pgmap v1040: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:31 smithi104 ceph-mon[97044]: pgmap v1041: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:31 smithi110 ceph-mon[97633]: pgmap v1041: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:33 smithi104 ceph-mon[97044]: pgmap v1042: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:33 smithi110 ceph-mon[97633]: pgmap v1042: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:35 smithi104 ceph-mon[97044]: pgmap v1043: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:35 smithi110 ceph-mon[97633]: pgmap v1043: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:37 smithi104 ceph-mon[97044]: pgmap v1044: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:37 smithi110 ceph-mon[97633]: pgmap v1044: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:39 smithi104 ceph-mon[97044]: pgmap v1045: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:39 smithi110 ceph-mon[97633]: pgmap v1045: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:01:40.160+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:01:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:41 smithi104 ceph-mon[97044]: pgmap v1046: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:41 smithi110 ceph-mon[97633]: pgmap v1046: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:43 smithi104 ceph-mon[97044]: pgmap v1047: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:43 smithi110 ceph-mon[97633]: pgmap v1047: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:01:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:01:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:01:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:01:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:01:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:01:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:01:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:01:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:45 smithi104 ceph-mon[97044]: pgmap v1048: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:45 smithi110 ceph-mon[97633]: pgmap v1048: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:47 smithi104 ceph-mon[97044]: pgmap v1049: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:47 smithi110 ceph-mon[97633]: pgmap v1049: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:49 smithi104 ceph-mon[97044]: pgmap v1050: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:49 smithi110 ceph-mon[97633]: pgmap v1050: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:51 smithi104 ceph-mon[97044]: pgmap v1051: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:51 smithi110 ceph-mon[97633]: pgmap v1051: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:53 smithi104 ceph-mon[97044]: pgmap v1052: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:53 smithi110 ceph-mon[97633]: pgmap v1052: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:01:55.169+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:01:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:55 smithi104 ceph-mon[97044]: pgmap v1053: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:55 smithi110 ceph-mon[97633]: pgmap v1053: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:01:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:57 smithi104 ceph-mon[97044]: pgmap v1054: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:57 smithi110 ceph-mon[97633]: pgmap v1054: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:01:59 smithi104 ceph-mon[97044]: pgmap v1055: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:01:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:01:59 smithi110 ceph-mon[97633]: pgmap v1055: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:01 smithi104 ceph-mon[97044]: pgmap v1056: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:01 smithi110 ceph-mon[97633]: pgmap v1056: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:03 smithi104 ceph-mon[97044]: pgmap v1057: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:03 smithi110 ceph-mon[97633]: pgmap v1057: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:05 smithi104 ceph-mon[97044]: pgmap v1058: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:02:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:05 smithi110 ceph-mon[97633]: pgmap v1058: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:02:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:07 smithi104 ceph-mon[97044]: pgmap v1059: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:02:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:07 smithi110 ceph-mon[97633]: pgmap v1059: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:02:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:08 smithi104 ceph-mon[97044]: osdmap e151: 12 total, 12 up, 12 in 2022-09-10T05:02:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:08 smithi110 ceph-mon[97633]: osdmap e151: 12 total, 12 up, 12 in 2022-09-10T05:02:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:09 smithi104 ceph-mon[97044]: pgmap v1061: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:02:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:09 smithi110 ceph-mon[97633]: pgmap v1061: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:02:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:02:10.171+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:02:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:11 smithi104 ceph-mon[97044]: pgmap v1062: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:02:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:11 smithi110 ceph-mon[97633]: pgmap v1062: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:02:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:13 smithi104 ceph-mon[97044]: pgmap v1063: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:02:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:13 smithi104 ceph-mon[97044]: osdmap e152: 12 total, 12 up, 12 in 2022-09-10T05:02:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:13 smithi110 ceph-mon[97633]: pgmap v1063: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:02:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:13 smithi110 ceph-mon[97633]: osdmap e152: 12 total, 12 up, 12 in 2022-09-10T05:02:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:15 smithi104 ceph-mon[97044]: pgmap v1065: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:02:15.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:15 smithi110 ceph-mon[97633]: pgmap v1065: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:02:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:17 smithi104 ceph-mon[97044]: pgmap v1066: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:02:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:17 smithi110 ceph-mon[97633]: pgmap v1066: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:02:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:19 smithi104 ceph-mon[97044]: pgmap v1067: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:02:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:19 smithi110 ceph-mon[97633]: pgmap v1067: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:02:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:21 smithi104 ceph-mon[97044]: pgmap v1068: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:02:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:21 smithi110 ceph-mon[97633]: pgmap v1068: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:02:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:23 smithi104 ceph-mon[97044]: pgmap v1069: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:02:23.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:23 smithi110 ceph-mon[97633]: pgmap v1069: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:02:25.563 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:02:25.172+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:02:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:25 smithi104 ceph-mon[97044]: pgmap v1070: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:02:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:25 smithi110 ceph-mon[97633]: pgmap v1070: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:02:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:27 smithi104 ceph-mon[97044]: pgmap v1071: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:27 smithi110 ceph-mon[97633]: pgmap v1071: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:29 smithi104 ceph-mon[97044]: pgmap v1072: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:29.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:29 smithi110 ceph-mon[97633]: pgmap v1072: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:31 smithi104 ceph-mon[97044]: pgmap v1073: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:31 smithi110 ceph-mon[97633]: pgmap v1073: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:33 smithi104 ceph-mon[97044]: pgmap v1074: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:33 smithi110 ceph-mon[97633]: pgmap v1074: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:35 smithi104 ceph-mon[97044]: pgmap v1075: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:35 smithi110 ceph-mon[97633]: pgmap v1075: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:37 smithi104 ceph-mon[97044]: pgmap v1076: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:37 smithi110 ceph-mon[97633]: pgmap v1076: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:39 smithi104 ceph-mon[97044]: pgmap v1077: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:39 smithi110 ceph-mon[97633]: pgmap v1077: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:02:40.173+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:02:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:41 smithi104 ceph-mon[97044]: pgmap v1078: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:41 smithi110 ceph-mon[97633]: pgmap v1078: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:43 smithi104 ceph-mon[97044]: pgmap v1079: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:43 smithi110 ceph-mon[97633]: pgmap v1079: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:02:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:02:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:02:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:02:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:02:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:45 smithi104 ceph-mon[97044]: pgmap v1080: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:45 smithi110 ceph-mon[97633]: pgmap v1080: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:47 smithi104 ceph-mon[97044]: pgmap v1081: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:47 smithi110 ceph-mon[97633]: pgmap v1081: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:49 smithi104 ceph-mon[97044]: pgmap v1082: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:49 smithi110 ceph-mon[97633]: pgmap v1082: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:51 smithi104 ceph-mon[97044]: pgmap v1083: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:51 smithi110 ceph-mon[97633]: pgmap v1083: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:53 smithi104 ceph-mon[97044]: pgmap v1084: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:53 smithi110 ceph-mon[97633]: pgmap v1084: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:02:55.180+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:02:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:55 smithi104 ceph-mon[97044]: pgmap v1085: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:55 smithi110 ceph-mon[97633]: pgmap v1085: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:02:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:57 smithi104 ceph-mon[97044]: pgmap v1086: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:57 smithi110 ceph-mon[97633]: pgmap v1086: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:02:59 smithi104 ceph-mon[97044]: pgmap v1087: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:02:59.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:02:59 smithi110 ceph-mon[97633]: pgmap v1087: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:01 smithi104 ceph-mon[97044]: pgmap v1088: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:01 smithi110 ceph-mon[97633]: pgmap v1088: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:03 smithi104 ceph-mon[97044]: pgmap v1089: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:03 smithi110 ceph-mon[97633]: pgmap v1089: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:05 smithi104 ceph-mon[97044]: pgmap v1090: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:03:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:05 smithi110 ceph-mon[97633]: pgmap v1090: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:03:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:07 smithi104 ceph-mon[97044]: pgmap v1091: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:03:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:07 smithi110 ceph-mon[97633]: pgmap v1091: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:03:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:08 smithi104 ceph-mon[97044]: osdmap e153: 12 total, 12 up, 12 in 2022-09-10T05:03:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:08 smithi110 ceph-mon[97633]: osdmap e153: 12 total, 12 up, 12 in 2022-09-10T05:03:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:09 smithi104 ceph-mon[97044]: pgmap v1093: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:03:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:09 smithi110 ceph-mon[97633]: pgmap v1093: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:03:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:03:10.180+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:03:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:11 smithi104 ceph-mon[97044]: pgmap v1094: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:03:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:11 smithi110 ceph-mon[97633]: pgmap v1094: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:03:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:13 smithi104 ceph-mon[97044]: pgmap v1095: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:03:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:13 smithi104 ceph-mon[97044]: osdmap e154: 12 total, 12 up, 12 in 2022-09-10T05:03:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:13 smithi110 ceph-mon[97633]: pgmap v1095: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:03:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:13 smithi110 ceph-mon[97633]: osdmap e154: 12 total, 12 up, 12 in 2022-09-10T05:03:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:15 smithi104 ceph-mon[97044]: pgmap v1097: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:03:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:15 smithi110 ceph-mon[97633]: pgmap v1097: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:03:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:17 smithi104 ceph-mon[97044]: pgmap v1098: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:03:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:17 smithi110 ceph-mon[97633]: pgmap v1098: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:03:19.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:19 smithi104 ceph-mon[97044]: pgmap v1099: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:03:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:19 smithi110 ceph-mon[97633]: pgmap v1099: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:03:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:21 smithi104 ceph-mon[97044]: pgmap v1100: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:03:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:21 smithi110 ceph-mon[97633]: pgmap v1100: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:03:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:23 smithi104 ceph-mon[97044]: pgmap v1101: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:03:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:23 smithi110 ceph-mon[97633]: pgmap v1101: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:03:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:03:25.185+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:03:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:25 smithi104 ceph-mon[97044]: pgmap v1102: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:03:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:25 smithi110 ceph-mon[97633]: pgmap v1102: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:03:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:27 smithi104 ceph-mon[97044]: pgmap v1103: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:27 smithi110 ceph-mon[97633]: pgmap v1103: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:29 smithi104 ceph-mon[97044]: pgmap v1104: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:29 smithi110 ceph-mon[97633]: pgmap v1104: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:31 smithi104 ceph-mon[97044]: pgmap v1105: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:31 smithi110 ceph-mon[97633]: pgmap v1105: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:33 smithi104 ceph-mon[97044]: pgmap v1106: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:33 smithi110 ceph-mon[97633]: pgmap v1106: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:35 smithi104 ceph-mon[97044]: pgmap v1107: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:35 smithi110 ceph-mon[97633]: pgmap v1107: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:37 smithi104 ceph-mon[97044]: pgmap v1108: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:37 smithi110 ceph-mon[97633]: pgmap v1108: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:39 smithi104 ceph-mon[97044]: pgmap v1109: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:39 smithi110 ceph-mon[97633]: pgmap v1109: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:03:40.188+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:03:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:41 smithi104 ceph-mon[97044]: pgmap v1110: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:41 smithi110 ceph-mon[97633]: pgmap v1110: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:43 smithi104 ceph-mon[97044]: pgmap v1111: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:43 smithi110 ceph-mon[97633]: pgmap v1111: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:03:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:03:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:45 smithi104 ceph-mon[97044]: pgmap v1112: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:45 smithi110 ceph-mon[97633]: pgmap v1112: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:46.802 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:03:46.800+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T05:03:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:47 smithi104 ceph-mon[97044]: pgmap v1113: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:47 smithi110 ceph-mon[97633]: pgmap v1113: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:49 smithi104 ceph-mon[97044]: pgmap v1114: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:49 smithi110 ceph-mon[97633]: pgmap v1114: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:51 smithi104 ceph-mon[97044]: pgmap v1115: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:51 smithi110 ceph-mon[97633]: pgmap v1115: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:53 smithi104 ceph-mon[97044]: pgmap v1116: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:53 smithi110 ceph-mon[97633]: pgmap v1116: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:03:55.190+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:03:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:55 smithi104 ceph-mon[97044]: pgmap v1117: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:55 smithi110 ceph-mon[97633]: pgmap v1117: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:03:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:57 smithi104 ceph-mon[97044]: pgmap v1118: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:57 smithi110 ceph-mon[97633]: pgmap v1118: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:03:59 smithi104 ceph-mon[97044]: pgmap v1119: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:03:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:03:59 smithi110 ceph-mon[97633]: pgmap v1119: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:01 smithi104 ceph-mon[97044]: pgmap v1120: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:01 smithi110 ceph-mon[97633]: pgmap v1120: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:03 smithi104 ceph-mon[97044]: pgmap v1121: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:03 smithi110 ceph-mon[97633]: pgmap v1121: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:05 smithi104 ceph-mon[97044]: pgmap v1122: 129 pgs: 129 active+clean; 7.8 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:04:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:05 smithi110 ceph-mon[97633]: pgmap v1122: 129 pgs: 129 active+clean; 7.8 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:04:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:07 smithi104 ceph-mon[97044]: pgmap v1123: 129 pgs: 129 active+clean; 7.8 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:04:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:07 smithi110 ceph-mon[97633]: pgmap v1123: 129 pgs: 129 active+clean; 7.8 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:04:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:08 smithi110 ceph-mon[97633]: osdmap e155: 12 total, 12 up, 12 in 2022-09-10T05:04:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:08 smithi104 ceph-mon[97044]: osdmap e155: 12 total, 12 up, 12 in 2022-09-10T05:04:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:09 smithi110 ceph-mon[97633]: pgmap v1125: 129 pgs: 129 active+clean; 7.8 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:04:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:09 smithi104 ceph-mon[97044]: pgmap v1125: 129 pgs: 129 active+clean; 7.8 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:04:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:04:10.192+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:04:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:11 smithi110 ceph-mon[97633]: pgmap v1126: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:04:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:11 smithi104 ceph-mon[97044]: pgmap v1126: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:04:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:13 smithi104 ceph-mon[97044]: pgmap v1127: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:04:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:13 smithi104 ceph-mon[97044]: osdmap e156: 12 total, 12 up, 12 in 2022-09-10T05:04:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:13 smithi110 ceph-mon[97633]: pgmap v1127: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:04:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:13 smithi110 ceph-mon[97633]: osdmap e156: 12 total, 12 up, 12 in 2022-09-10T05:04:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:15 smithi104 ceph-mon[97044]: pgmap v1129: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:04:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:15 smithi110 ceph-mon[97633]: pgmap v1129: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:04:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:17 smithi104 ceph-mon[97044]: pgmap v1130: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:04:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:17 smithi110 ceph-mon[97633]: pgmap v1130: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:04:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:19 smithi104 ceph-mon[97044]: pgmap v1131: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:04:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:19 smithi110 ceph-mon[97633]: pgmap v1131: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:04:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:21 smithi104 ceph-mon[97044]: pgmap v1132: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:04:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:21 smithi110 ceph-mon[97633]: pgmap v1132: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:04:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:23 smithi104 ceph-mon[97044]: pgmap v1133: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:04:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:23 smithi110 ceph-mon[97633]: pgmap v1133: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:04:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:04:25.194+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:04:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:25 smithi104 ceph-mon[97044]: pgmap v1134: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:04:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:25 smithi110 ceph-mon[97633]: pgmap v1134: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:04:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:27 smithi104 ceph-mon[97044]: pgmap v1135: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:27 smithi110 ceph-mon[97633]: pgmap v1135: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:29 smithi104 ceph-mon[97044]: pgmap v1136: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:29 smithi110 ceph-mon[97633]: pgmap v1136: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:31 smithi104 ceph-mon[97044]: pgmap v1137: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:31 smithi110 ceph-mon[97633]: pgmap v1137: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:33 smithi104 ceph-mon[97044]: pgmap v1138: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:33 smithi110 ceph-mon[97633]: pgmap v1138: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:35 smithi104 ceph-mon[97044]: pgmap v1139: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:35 smithi110 ceph-mon[97633]: pgmap v1139: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:37 smithi104 ceph-mon[97044]: pgmap v1140: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:37 smithi110 ceph-mon[97633]: pgmap v1140: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:39 smithi104 ceph-mon[97044]: pgmap v1141: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:39 smithi110 ceph-mon[97633]: pgmap v1141: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:04:40.201+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:04:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:41 smithi104 ceph-mon[97044]: pgmap v1142: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:41 smithi110 ceph-mon[97633]: pgmap v1142: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:43 smithi104 ceph-mon[97044]: pgmap v1143: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:43 smithi110 ceph-mon[97633]: pgmap v1143: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:04:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:04:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:04:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:04:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:04:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:45 smithi104 ceph-mon[97044]: pgmap v1144: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:45 smithi110 ceph-mon[97633]: pgmap v1144: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:47 smithi104 ceph-mon[97044]: pgmap v1145: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:47 smithi110 ceph-mon[97633]: pgmap v1145: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:49 smithi104 ceph-mon[97044]: pgmap v1146: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:49 smithi110 ceph-mon[97633]: pgmap v1146: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:51 smithi104 ceph-mon[97044]: pgmap v1147: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:51 smithi110 ceph-mon[97633]: pgmap v1147: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:53 smithi104 ceph-mon[97044]: pgmap v1148: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:53 smithi110 ceph-mon[97633]: pgmap v1148: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:04:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:04:55.201+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:04:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:55 smithi104 ceph-mon[97044]: pgmap v1149: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:55 smithi110 ceph-mon[97633]: pgmap v1149: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:04:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:57 smithi104 ceph-mon[97044]: pgmap v1150: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:04:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:57 smithi110 ceph-mon[97633]: pgmap v1150: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:04:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:04:59 smithi104 ceph-mon[97044]: pgmap v1151: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:04:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:04:59 smithi110 ceph-mon[97633]: pgmap v1151: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:05:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:01 smithi104 ceph-mon[97044]: pgmap v1152: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:01 smithi110 ceph-mon[97633]: pgmap v1152: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:03 smithi104 ceph-mon[97044]: pgmap v1153: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:05:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:03 smithi110 ceph-mon[97633]: pgmap v1153: 129 pgs: 129 active+clean; 7.8 MiB data, 323 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:05:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:05 smithi104 ceph-mon[97044]: pgmap v1154: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T05:05:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:05 smithi110 ceph-mon[97633]: pgmap v1154: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T05:05:07.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:07 smithi110 ceph-mon[97633]: pgmap v1155: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T05:05:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:07 smithi104 ceph-mon[97044]: pgmap v1155: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T05:05:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:09 smithi110 ceph-mon[97633]: pgmap v1156: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T05:05:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:09 smithi104 ceph-mon[97044]: pgmap v1156: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T05:05:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:05:10.206+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:05:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:11 smithi110 ceph-mon[97633]: pgmap v1157: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T05:05:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:11 smithi104 ceph-mon[97044]: pgmap v1157: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T05:05:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:13 smithi110 ceph-mon[97633]: pgmap v1158: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T05:05:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:13 smithi104 ceph-mon[97044]: pgmap v1158: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T05:05:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:15 smithi110 ceph-mon[97633]: pgmap v1159: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T05:05:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:15 smithi104 ceph-mon[97044]: pgmap v1159: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T05:05:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:17 smithi110 ceph-mon[97633]: pgmap v1160: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T05:05:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:17 smithi104 ceph-mon[97044]: pgmap v1160: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T05:05:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:19 smithi110 ceph-mon[97633]: pgmap v1161: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T05:05:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:19 smithi104 ceph-mon[97044]: pgmap v1161: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T05:05:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:21 smithi110 ceph-mon[97633]: pgmap v1162: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:05:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:21 smithi104 ceph-mon[97044]: pgmap v1162: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:05:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:23 smithi110 ceph-mon[97633]: pgmap v1163: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:23 smithi104 ceph-mon[97044]: pgmap v1163: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:05:25.214+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:05:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:25 smithi110 ceph-mon[97633]: pgmap v1164: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:25 smithi104 ceph-mon[97044]: pgmap v1164: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:27 smithi110 ceph-mon[97633]: pgmap v1165: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:27 smithi104 ceph-mon[97044]: pgmap v1165: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:29 smithi110 ceph-mon[97633]: pgmap v1166: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:29 smithi104 ceph-mon[97044]: pgmap v1166: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:31 smithi110 ceph-mon[97633]: pgmap v1167: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:31 smithi104 ceph-mon[97044]: pgmap v1167: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:33 smithi110 ceph-mon[97633]: pgmap v1168: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:33 smithi104 ceph-mon[97044]: pgmap v1168: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:35 smithi110 ceph-mon[97633]: pgmap v1169: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:35 smithi104 ceph-mon[97044]: pgmap v1169: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:37 smithi110 ceph-mon[97633]: pgmap v1170: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:37 smithi104 ceph-mon[97044]: pgmap v1170: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:39 smithi110 ceph-mon[97633]: pgmap v1171: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:39 smithi104 ceph-mon[97044]: pgmap v1171: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:05:40.217+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:05:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:41 smithi110 ceph-mon[97633]: pgmap v1172: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:41 smithi104 ceph-mon[97044]: pgmap v1172: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:43 smithi110 ceph-mon[97633]: pgmap v1173: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:43 smithi104 ceph-mon[97044]: pgmap v1173: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:05:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:05:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:05:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:05:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:05:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:05:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:45 smithi110 ceph-mon[97633]: pgmap v1174: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:45 smithi104 ceph-mon[97044]: pgmap v1174: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:47 smithi110 ceph-mon[97633]: pgmap v1175: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:47 smithi104 ceph-mon[97044]: pgmap v1175: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:49 smithi110 ceph-mon[97633]: pgmap v1176: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:49 smithi104 ceph-mon[97044]: pgmap v1176: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:51 smithi110 ceph-mon[97633]: pgmap v1177: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:51 smithi104 ceph-mon[97044]: pgmap v1177: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:53 smithi110 ceph-mon[97633]: pgmap v1178: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:53 smithi104 ceph-mon[97044]: pgmap v1178: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:05:55.224+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:05:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:55 smithi110 ceph-mon[97633]: pgmap v1179: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:55 smithi104 ceph-mon[97044]: pgmap v1179: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:05:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:57 smithi110 ceph-mon[97633]: pgmap v1180: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:57 smithi104 ceph-mon[97044]: pgmap v1180: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:05:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:05:59 smithi110 ceph-mon[97633]: pgmap v1181: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:05:59 smithi104 ceph-mon[97044]: pgmap v1181: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:01 smithi110 ceph-mon[97633]: pgmap v1182: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:01 smithi104 ceph-mon[97044]: pgmap v1182: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:03 smithi110 ceph-mon[97633]: pgmap v1183: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:03 smithi104 ceph-mon[97044]: pgmap v1183: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:05 smithi110 ceph-mon[97633]: pgmap v1184: 129 pgs: 129 active+clean; 7.8 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:06:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:05 smithi104 ceph-mon[97044]: pgmap v1184: 129 pgs: 129 active+clean; 7.8 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:06:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:07 smithi110 ceph-mon[97633]: pgmap v1185: 129 pgs: 129 active+clean; 7.8 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:06:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:07 smithi104 ceph-mon[97044]: pgmap v1185: 129 pgs: 129 active+clean; 7.8 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:06:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:08 smithi110 ceph-mon[97633]: osdmap e157: 12 total, 12 up, 12 in 2022-09-10T05:06:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:08 smithi104 ceph-mon[97044]: osdmap e157: 12 total, 12 up, 12 in 2022-09-10T05:06:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:09 smithi110 ceph-mon[97633]: pgmap v1187: 129 pgs: 129 active+clean; 7.8 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:06:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:09 smithi104 ceph-mon[97044]: pgmap v1187: 129 pgs: 129 active+clean; 7.8 MiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:06:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:06:10.227+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:06:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:11 smithi110 ceph-mon[97633]: pgmap v1188: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:06:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:11 smithi104 ceph-mon[97044]: pgmap v1188: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:06:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:13 smithi104 ceph-mon[97044]: pgmap v1189: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:06:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:13 smithi104 ceph-mon[97044]: osdmap e158: 12 total, 12 up, 12 in 2022-09-10T05:06:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:13 smithi110 ceph-mon[97633]: pgmap v1189: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:06:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:13 smithi110 ceph-mon[97633]: osdmap e158: 12 total, 12 up, 12 in 2022-09-10T05:06:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:15 smithi104 ceph-mon[97044]: pgmap v1191: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:06:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:15 smithi110 ceph-mon[97633]: pgmap v1191: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:06:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:17 smithi104 ceph-mon[97044]: pgmap v1192: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T05:06:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:17 smithi110 ceph-mon[97633]: pgmap v1192: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 232 B/s wr, 11 op/s 2022-09-10T05:06:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:19 smithi104 ceph-mon[97044]: pgmap v1193: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:06:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:19 smithi110 ceph-mon[97633]: pgmap v1193: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:06:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:21 smithi104 ceph-mon[97044]: pgmap v1194: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:06:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:21 smithi110 ceph-mon[97633]: pgmap v1194: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:06:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:23 smithi104 ceph-mon[97044]: pgmap v1195: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:06:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:23 smithi110 ceph-mon[97633]: pgmap v1195: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:06:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:06:25.233+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:06:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:25 smithi104 ceph-mon[97044]: pgmap v1196: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:06:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:25 smithi110 ceph-mon[97633]: pgmap v1196: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:06:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:27 smithi104 ceph-mon[97044]: pgmap v1197: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:27 smithi110 ceph-mon[97633]: pgmap v1197: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:29 smithi104 ceph-mon[97044]: pgmap v1198: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:29 smithi110 ceph-mon[97633]: pgmap v1198: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:31 smithi104 ceph-mon[97044]: pgmap v1199: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:31 smithi110 ceph-mon[97633]: pgmap v1199: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:33 smithi104 ceph-mon[97044]: pgmap v1200: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:33 smithi110 ceph-mon[97633]: pgmap v1200: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:35 smithi104 ceph-mon[97044]: pgmap v1201: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:35 smithi110 ceph-mon[97633]: pgmap v1201: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:37 smithi104 ceph-mon[97044]: pgmap v1202: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:37 smithi110 ceph-mon[97633]: pgmap v1202: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:39 smithi104 ceph-mon[97044]: pgmap v1203: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:39 smithi110 ceph-mon[97633]: pgmap v1203: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:06:40.234+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:06:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:41 smithi104 ceph-mon[97044]: pgmap v1204: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:41 smithi110 ceph-mon[97633]: pgmap v1204: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:43 smithi104 ceph-mon[97044]: pgmap v1205: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:43.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:43 smithi110 ceph-mon[97633]: pgmap v1205: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:06:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:06:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:06:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:06:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:06:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:06:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:45 smithi110 ceph-mon[97633]: pgmap v1206: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:45 smithi104 ceph-mon[97044]: pgmap v1206: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:47 smithi110 ceph-mon[97633]: pgmap v1207: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:47 smithi104 ceph-mon[97044]: pgmap v1207: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:49 smithi110 ceph-mon[97633]: pgmap v1208: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:49 smithi104 ceph-mon[97044]: pgmap v1208: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:51 smithi110 ceph-mon[97633]: pgmap v1209: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:51 smithi104 ceph-mon[97044]: pgmap v1209: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:53 smithi110 ceph-mon[97633]: pgmap v1210: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:53 smithi104 ceph-mon[97044]: pgmap v1210: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:06:55.238+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:06:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:55 smithi110 ceph-mon[97633]: pgmap v1211: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:55 smithi104 ceph-mon[97044]: pgmap v1211: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:06:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:57 smithi110 ceph-mon[97633]: pgmap v1212: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:58.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:57 smithi104 ceph-mon[97044]: pgmap v1212: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:06:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:06:59 smithi110 ceph-mon[97633]: pgmap v1213: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:06:59 smithi104 ceph-mon[97044]: pgmap v1213: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:01 smithi110 ceph-mon[97633]: pgmap v1214: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:01 smithi104 ceph-mon[97044]: pgmap v1214: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:03 smithi110 ceph-mon[97633]: pgmap v1215: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:03 smithi104 ceph-mon[97044]: pgmap v1215: 129 pgs: 129 active+clean; 7.8 MiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:05 smithi110 ceph-mon[97633]: pgmap v1216: 129 pgs: 129 active+clean; 7.8 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:07:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:05 smithi104 ceph-mon[97044]: pgmap v1216: 129 pgs: 129 active+clean; 7.8 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:07:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:07 smithi110 ceph-mon[97633]: pgmap v1217: 129 pgs: 129 active+clean; 7.8 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:07:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:07 smithi104 ceph-mon[97044]: pgmap v1217: 129 pgs: 129 active+clean; 7.8 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:07:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:08 smithi110 ceph-mon[97633]: osdmap e159: 12 total, 12 up, 12 in 2022-09-10T05:07:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:08 smithi104 ceph-mon[97044]: osdmap e159: 12 total, 12 up, 12 in 2022-09-10T05:07:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:09 smithi110 ceph-mon[97633]: pgmap v1219: 129 pgs: 129 active+clean; 7.8 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:07:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:09 smithi104 ceph-mon[97044]: pgmap v1219: 129 pgs: 129 active+clean; 7.8 MiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:07:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:07:10.244+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:07:11.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:11 smithi110 ceph-mon[97633]: pgmap v1220: 129 pgs: 129 active+clean; 7.8 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:07:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:11 smithi104 ceph-mon[97044]: pgmap v1220: 129 pgs: 129 active+clean; 7.8 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:07:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:13 smithi104 ceph-mon[97044]: pgmap v1221: 129 pgs: 129 active+clean; 7.8 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:07:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:13 smithi104 ceph-mon[97044]: osdmap e160: 12 total, 12 up, 12 in 2022-09-10T05:07:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:13 smithi110 ceph-mon[97633]: pgmap v1221: 129 pgs: 129 active+clean; 7.8 MiB data, 315 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:07:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:13 smithi110 ceph-mon[97633]: osdmap e160: 12 total, 12 up, 12 in 2022-09-10T05:07:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:15 smithi104 ceph-mon[97044]: pgmap v1223: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:07:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:15 smithi110 ceph-mon[97633]: pgmap v1223: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:07:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:17 smithi104 ceph-mon[97044]: pgmap v1224: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:07:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:17 smithi110 ceph-mon[97633]: pgmap v1224: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:07:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:19 smithi104 ceph-mon[97044]: pgmap v1225: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:07:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:19 smithi110 ceph-mon[97633]: pgmap v1225: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:07:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:21 smithi104 ceph-mon[97044]: pgmap v1226: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:07:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:21 smithi110 ceph-mon[97633]: pgmap v1226: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:07:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:23 smithi104 ceph-mon[97044]: pgmap v1227: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:07:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:23 smithi110 ceph-mon[97633]: pgmap v1227: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:07:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:07:25.247+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:07:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:25 smithi104 ceph-mon[97044]: pgmap v1228: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:07:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:25 smithi110 ceph-mon[97633]: pgmap v1228: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:07:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:27 smithi104 ceph-mon[97044]: pgmap v1229: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:27 smithi110 ceph-mon[97633]: pgmap v1229: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:29 smithi104 ceph-mon[97044]: pgmap v1230: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:29 smithi110 ceph-mon[97633]: pgmap v1230: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:31 smithi104 ceph-mon[97044]: pgmap v1231: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:31 smithi110 ceph-mon[97633]: pgmap v1231: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:33 smithi104 ceph-mon[97044]: pgmap v1232: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:33 smithi110 ceph-mon[97633]: pgmap v1232: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:35 smithi110 ceph-mon[97633]: pgmap v1233: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:35 smithi104 ceph-mon[97044]: pgmap v1233: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:37 smithi110 ceph-mon[97633]: pgmap v1234: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:07:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:37 smithi104 ceph-mon[97044]: pgmap v1234: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:07:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:39 smithi110 ceph-mon[97633]: pgmap v1235: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:07:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:39 smithi104 ceph-mon[97044]: pgmap v1235: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:07:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:07:40.254+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:07:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:41 smithi110 ceph-mon[97633]: pgmap v1236: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:41 smithi104 ceph-mon[97044]: pgmap v1236: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:43 smithi110 ceph-mon[97633]: pgmap v1237: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:07:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:43 smithi104 ceph-mon[97044]: pgmap v1237: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:07:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:07:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:07:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:07:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:07:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:07:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:07:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:07:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:07:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:45 smithi110 ceph-mon[97633]: pgmap v1238: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:45 smithi104 ceph-mon[97044]: pgmap v1238: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:47 smithi110 ceph-mon[97633]: pgmap v1239: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:47 smithi104 ceph-mon[97044]: pgmap v1239: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:49 smithi110 ceph-mon[97633]: pgmap v1240: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:49 smithi104 ceph-mon[97044]: pgmap v1240: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:51 smithi110 ceph-mon[97633]: pgmap v1241: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:51 smithi104 ceph-mon[97044]: pgmap v1241: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:53 smithi110 ceph-mon[97633]: pgmap v1242: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:53 smithi104 ceph-mon[97044]: pgmap v1242: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:07:55.256+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:07:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:55 smithi110 ceph-mon[97633]: pgmap v1243: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:55 smithi104 ceph-mon[97044]: pgmap v1243: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:07:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:57 smithi110 ceph-mon[97633]: pgmap v1244: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:57 smithi104 ceph-mon[97044]: pgmap v1244: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:07:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:07:59 smithi110 ceph-mon[97633]: pgmap v1245: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:07:59 smithi104 ceph-mon[97044]: pgmap v1245: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:01 smithi110 ceph-mon[97633]: pgmap v1246: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:01 smithi104 ceph-mon[97044]: pgmap v1246: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:03 smithi110 ceph-mon[97633]: pgmap v1247: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:03 smithi104 ceph-mon[97044]: pgmap v1247: 129 pgs: 129 active+clean; 7.8 MiB data, 316 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:05 smithi110 ceph-mon[97633]: pgmap v1248: 129 pgs: 129 active+clean; 7.8 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:08:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:05 smithi104 ceph-mon[97044]: pgmap v1248: 129 pgs: 129 active+clean; 7.8 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:08:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:07 smithi110 ceph-mon[97633]: pgmap v1249: 129 pgs: 129 active+clean; 7.8 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:08:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:07 smithi104 ceph-mon[97044]: pgmap v1249: 129 pgs: 129 active+clean; 7.8 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:08:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:08 smithi110 ceph-mon[97633]: osdmap e161: 12 total, 12 up, 12 in 2022-09-10T05:08:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:08 smithi104 ceph-mon[97044]: osdmap e161: 12 total, 12 up, 12 in 2022-09-10T05:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:09 smithi110 ceph-mon[97633]: pgmap v1251: 129 pgs: 129 active+clean; 7.8 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:08:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:09 smithi104 ceph-mon[97044]: pgmap v1251: 129 pgs: 129 active+clean; 7.8 MiB data, 317 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:08:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:08:10.262+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:08:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:11 smithi110 ceph-mon[97633]: pgmap v1252: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:08:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:11 smithi104 ceph-mon[97044]: pgmap v1252: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:08:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:13 smithi104 ceph-mon[97044]: pgmap v1253: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:08:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:13 smithi104 ceph-mon[97044]: osdmap e162: 12 total, 12 up, 12 in 2022-09-10T05:08:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:13 smithi110 ceph-mon[97633]: pgmap v1253: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:08:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:13 smithi110 ceph-mon[97633]: osdmap e162: 12 total, 12 up, 12 in 2022-09-10T05:08:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:15 smithi104 ceph-mon[97044]: pgmap v1255: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:08:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:15 smithi110 ceph-mon[97633]: pgmap v1255: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:08:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:17 smithi104 ceph-mon[97044]: pgmap v1256: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:08:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:17 smithi110 ceph-mon[97633]: pgmap v1256: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:08:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:19 smithi104 ceph-mon[97044]: pgmap v1257: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:08:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:19 smithi110 ceph-mon[97633]: pgmap v1257: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:08:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:21 smithi104 ceph-mon[97044]: pgmap v1258: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:08:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:21 smithi110 ceph-mon[97633]: pgmap v1258: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:08:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:23 smithi104 ceph-mon[97044]: pgmap v1259: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:08:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:23 smithi110 ceph-mon[97633]: pgmap v1259: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:08:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:08:25.263+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:08:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:25 smithi110 ceph-mon[97633]: pgmap v1260: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:08:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:25 smithi104 ceph-mon[97044]: pgmap v1260: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:08:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:27 smithi110 ceph-mon[97633]: pgmap v1261: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:27 smithi104 ceph-mon[97044]: pgmap v1261: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:29 smithi110 ceph-mon[97633]: pgmap v1262: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:29 smithi104 ceph-mon[97044]: pgmap v1262: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:31 smithi110 ceph-mon[97633]: pgmap v1263: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:31 smithi104 ceph-mon[97044]: pgmap v1263: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:33 smithi110 ceph-mon[97633]: pgmap v1264: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:33 smithi104 ceph-mon[97044]: pgmap v1264: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:35 smithi110 ceph-mon[97633]: pgmap v1265: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:35 smithi104 ceph-mon[97044]: pgmap v1265: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:37 smithi110 ceph-mon[97633]: pgmap v1266: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:37 smithi104 ceph-mon[97044]: pgmap v1266: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:39 smithi110 ceph-mon[97633]: pgmap v1267: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:39 smithi104 ceph-mon[97044]: pgmap v1267: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:08:40.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:08:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:41 smithi110 ceph-mon[97633]: pgmap v1268: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:42.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:41 smithi104 ceph-mon[97044]: pgmap v1268: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:43 smithi110 ceph-mon[97633]: pgmap v1269: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:43 smithi104 ceph-mon[97044]: pgmap v1269: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:08:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:08:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:08:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:08:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:08:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:08:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:45 smithi110 ceph-mon[97633]: pgmap v1270: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:45 smithi104 ceph-mon[97044]: pgmap v1270: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:08:46.802+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T05:08:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:47 smithi110 ceph-mon[97633]: pgmap v1271: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:47 smithi104 ceph-mon[97044]: pgmap v1271: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:49 smithi110 ceph-mon[97633]: pgmap v1272: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:49 smithi104 ceph-mon[97044]: pgmap v1272: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:51 smithi110 ceph-mon[97633]: pgmap v1273: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:51 smithi104 ceph-mon[97044]: pgmap v1273: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:53 smithi110 ceph-mon[97633]: pgmap v1274: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:53 smithi104 ceph-mon[97044]: pgmap v1274: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:08:55.276+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:08:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:55 smithi110 ceph-mon[97633]: pgmap v1275: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:55 smithi104 ceph-mon[97044]: pgmap v1275: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:08:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:57 smithi110 ceph-mon[97633]: pgmap v1276: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:57 smithi104 ceph-mon[97044]: pgmap v1276: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:08:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:08:59 smithi110 ceph-mon[97633]: pgmap v1277: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:08:59 smithi104 ceph-mon[97044]: pgmap v1277: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:01 smithi110 ceph-mon[97633]: pgmap v1278: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:01 smithi104 ceph-mon[97044]: pgmap v1278: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:03 smithi110 ceph-mon[97633]: pgmap v1279: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:03 smithi104 ceph-mon[97044]: pgmap v1279: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:05 smithi110 ceph-mon[97633]: pgmap v1280: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:09:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:05 smithi104 ceph-mon[97044]: pgmap v1280: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:09:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:07 smithi110 ceph-mon[97633]: pgmap v1281: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:09:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:07 smithi104 ceph-mon[97044]: pgmap v1281: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:09:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:08 smithi110 ceph-mon[97633]: osdmap e163: 12 total, 12 up, 12 in 2022-09-10T05:09:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:08 smithi104 ceph-mon[97044]: osdmap e163: 12 total, 12 up, 12 in 2022-09-10T05:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:09 smithi110 ceph-mon[97633]: pgmap v1283: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:09:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:09 smithi104 ceph-mon[97044]: pgmap v1283: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:09:10.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:09:10.279+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:09:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:11 smithi110 ceph-mon[97633]: pgmap v1284: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:09:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:11 smithi104 ceph-mon[97044]: pgmap v1284: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:09:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:13 smithi110 ceph-mon[97633]: pgmap v1285: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:09:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:13 smithi110 ceph-mon[97633]: osdmap e164: 12 total, 12 up, 12 in 2022-09-10T05:09:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:13 smithi104 ceph-mon[97044]: pgmap v1285: 129 pgs: 129 active+clean; 7.8 MiB data, 320 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:09:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:13 smithi104 ceph-mon[97044]: osdmap e164: 12 total, 12 up, 12 in 2022-09-10T05:09:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:15 smithi110 ceph-mon[97633]: pgmap v1287: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:09:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:15 smithi104 ceph-mon[97044]: pgmap v1287: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:09:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:17 smithi110 ceph-mon[97633]: pgmap v1288: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:09:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:17 smithi104 ceph-mon[97044]: pgmap v1288: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:09:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:19 smithi110 ceph-mon[97633]: pgmap v1289: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:09:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:19 smithi104 ceph-mon[97044]: pgmap v1289: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:09:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:21 smithi110 ceph-mon[97633]: pgmap v1290: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:09:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:21 smithi104 ceph-mon[97044]: pgmap v1290: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:09:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:23 smithi110 ceph-mon[97633]: pgmap v1291: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:09:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:23 smithi104 ceph-mon[97044]: pgmap v1291: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:09:25.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:09:25.288+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:09:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:25 smithi110 ceph-mon[97633]: pgmap v1292: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:09:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:25 smithi104 ceph-mon[97044]: pgmap v1292: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:09:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:27 smithi110 ceph-mon[97633]: pgmap v1293: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:27 smithi104 ceph-mon[97044]: pgmap v1293: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:29 smithi110 ceph-mon[97633]: pgmap v1294: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:29 smithi104 ceph-mon[97044]: pgmap v1294: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:31 smithi110 ceph-mon[97633]: pgmap v1295: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:31 smithi104 ceph-mon[97044]: pgmap v1295: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:33 smithi110 ceph-mon[97633]: pgmap v1296: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:33 smithi104 ceph-mon[97044]: pgmap v1296: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:35 smithi110 ceph-mon[97633]: pgmap v1297: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:35 smithi104 ceph-mon[97044]: pgmap v1297: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:37 smithi110 ceph-mon[97633]: pgmap v1298: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:37 smithi104 ceph-mon[97044]: pgmap v1298: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:39 smithi110 ceph-mon[97633]: pgmap v1299: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:39 smithi104 ceph-mon[97044]: pgmap v1299: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:40.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:09:40.291+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:09:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:41 smithi110 ceph-mon[97633]: pgmap v1300: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:41 smithi104 ceph-mon[97044]: pgmap v1300: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:43 smithi110 ceph-mon[97633]: pgmap v1301: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:43 smithi104 ceph-mon[97044]: pgmap v1301: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:09:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:09:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:09:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:09:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:09:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:45 smithi110 ceph-mon[97633]: pgmap v1302: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:45 smithi104 ceph-mon[97044]: pgmap v1302: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:47 smithi110 ceph-mon[97633]: pgmap v1303: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:47 smithi104 ceph-mon[97044]: pgmap v1303: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:49 smithi110 ceph-mon[97633]: pgmap v1304: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:49 smithi104 ceph-mon[97044]: pgmap v1304: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:09:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:51 smithi110 ceph-mon[97633]: pgmap v1305: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:51 smithi104 ceph-mon[97044]: pgmap v1305: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:53 smithi110 ceph-mon[97633]: pgmap v1306: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:53 smithi104 ceph-mon[97044]: pgmap v1306: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:55.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:09:55.292+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:09:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:55 smithi110 ceph-mon[97633]: pgmap v1307: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:55 smithi104 ceph-mon[97044]: pgmap v1307: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:09:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:57 smithi110 ceph-mon[97633]: pgmap v1308: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:57 smithi104 ceph-mon[97044]: pgmap v1308: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:09:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:09:59 smithi110 ceph-mon[97633]: pgmap v1309: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:10:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:09:59 smithi104 ceph-mon[97044]: pgmap v1309: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:10:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:10:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:10:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:01 smithi104 ceph-mon[97044]: pgmap v1310: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:01 smithi110 ceph-mon[97633]: pgmap v1310: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:03 smithi110 ceph-mon[97633]: pgmap v1311: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:03 smithi104 ceph-mon[97044]: pgmap v1311: 129 pgs: 129 active+clean; 7.8 MiB data, 321 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:05 smithi110 ceph-mon[97633]: pgmap v1312: 129 pgs: 129 active+clean; 7.8 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:10:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:05 smithi104 ceph-mon[97044]: pgmap v1312: 129 pgs: 129 active+clean; 7.8 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:10:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:07 smithi110 ceph-mon[97633]: pgmap v1313: 129 pgs: 129 active+clean; 7.8 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:10:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:07 smithi104 ceph-mon[97044]: pgmap v1313: 129 pgs: 129 active+clean; 7.8 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:10:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:08 smithi110 ceph-mon[97633]: osdmap e165: 12 total, 12 up, 12 in 2022-09-10T05:10:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:08 smithi104 ceph-mon[97044]: osdmap e165: 12 total, 12 up, 12 in 2022-09-10T05:10:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:09 smithi110 ceph-mon[97633]: pgmap v1315: 129 pgs: 129 active+clean; 7.8 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:10:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:09 smithi104 ceph-mon[97044]: pgmap v1315: 129 pgs: 129 active+clean; 7.8 MiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:10:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:10:10.303+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:10:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:11 smithi110 ceph-mon[97633]: pgmap v1316: 129 pgs: 129 active+clean; 7.8 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:10:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:11 smithi104 ceph-mon[97044]: pgmap v1316: 129 pgs: 129 active+clean; 7.8 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:10:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:13 smithi110 ceph-mon[97633]: pgmap v1317: 129 pgs: 129 active+clean; 7.8 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:10:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:13 smithi110 ceph-mon[97633]: osdmap e166: 12 total, 12 up, 12 in 2022-09-10T05:10:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:13 smithi104 ceph-mon[97044]: pgmap v1317: 129 pgs: 129 active+clean; 7.8 MiB data, 310 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:10:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:13 smithi104 ceph-mon[97044]: osdmap e166: 12 total, 12 up, 12 in 2022-09-10T05:10:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:15 smithi110 ceph-mon[97633]: pgmap v1319: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:10:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:15 smithi104 ceph-mon[97044]: pgmap v1319: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:10:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:17 smithi110 ceph-mon[97633]: pgmap v1320: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:10:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:17 smithi104 ceph-mon[97044]: pgmap v1320: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T05:10:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:19 smithi110 ceph-mon[97633]: pgmap v1321: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:10:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:19 smithi104 ceph-mon[97044]: pgmap v1321: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:10:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:21 smithi110 ceph-mon[97633]: pgmap v1322: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:10:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:21 smithi104 ceph-mon[97044]: pgmap v1322: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:10:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:23 smithi110 ceph-mon[97633]: pgmap v1323: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:10:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:23 smithi104 ceph-mon[97044]: pgmap v1323: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:10:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:10:25.308+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:10:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:25 smithi110 ceph-mon[97633]: pgmap v1324: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:10:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:25 smithi104 ceph-mon[97044]: pgmap v1324: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:10:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:27 smithi110 ceph-mon[97633]: pgmap v1325: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:27 smithi104 ceph-mon[97044]: pgmap v1325: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:29 smithi110 ceph-mon[97633]: pgmap v1326: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:29 smithi104 ceph-mon[97044]: pgmap v1326: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:31 smithi110 ceph-mon[97633]: pgmap v1327: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:31 smithi104 ceph-mon[97044]: pgmap v1327: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:33 smithi110 ceph-mon[97633]: pgmap v1328: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:33 smithi104 ceph-mon[97044]: pgmap v1328: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:35 smithi110 ceph-mon[97633]: pgmap v1329: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:35 smithi104 ceph-mon[97044]: pgmap v1329: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:37 smithi110 ceph-mon[97633]: pgmap v1330: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:37 smithi104 ceph-mon[97044]: pgmap v1330: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:39 smithi110 ceph-mon[97633]: pgmap v1331: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:39 smithi104 ceph-mon[97044]: pgmap v1331: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:10:40.309+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:10:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:41 smithi110 ceph-mon[97633]: pgmap v1332: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:41 smithi104 ceph-mon[97044]: pgmap v1332: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:43 smithi110 ceph-mon[97633]: pgmap v1333: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:43 smithi104 ceph-mon[97044]: pgmap v1333: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:10:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:10:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:10:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:10:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:10:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:10:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:10:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:45 smithi110 ceph-mon[97633]: pgmap v1334: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:45 smithi104 ceph-mon[97044]: pgmap v1334: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:47 smithi110 ceph-mon[97633]: pgmap v1335: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:47 smithi104 ceph-mon[97044]: pgmap v1335: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:49 smithi110 ceph-mon[97633]: pgmap v1336: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:49 smithi104 ceph-mon[97044]: pgmap v1336: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:51 smithi110 ceph-mon[97633]: pgmap v1337: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:51 smithi104 ceph-mon[97044]: pgmap v1337: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:53 smithi110 ceph-mon[97633]: pgmap v1338: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:53 smithi104 ceph-mon[97044]: pgmap v1338: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:10:55.315+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:10:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:55 smithi110 ceph-mon[97633]: pgmap v1339: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:55 smithi104 ceph-mon[97044]: pgmap v1339: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:10:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:57 smithi110 ceph-mon[97633]: pgmap v1340: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:57 smithi104 ceph-mon[97044]: pgmap v1340: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:10:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:10:59 smithi110 ceph-mon[97633]: pgmap v1341: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:10:59 smithi104 ceph-mon[97044]: pgmap v1341: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:01 smithi110 ceph-mon[97633]: pgmap v1342: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:01 smithi104 ceph-mon[97044]: pgmap v1342: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:03 smithi110 ceph-mon[97633]: pgmap v1343: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:03 smithi104 ceph-mon[97044]: pgmap v1343: 129 pgs: 129 active+clean; 7.8 MiB data, 311 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:05 smithi110 ceph-mon[97633]: pgmap v1344: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:11:06.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:05 smithi104 ceph-mon[97044]: pgmap v1344: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:11:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:07 smithi110 ceph-mon[97633]: pgmap v1345: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:11:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:07 smithi104 ceph-mon[97044]: pgmap v1345: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:11:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:08 smithi104 ceph-mon[97044]: osdmap e167: 12 total, 12 up, 12 in 2022-09-10T05:11:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:08 smithi110 ceph-mon[97633]: osdmap e167: 12 total, 12 up, 12 in 2022-09-10T05:11:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:09 smithi104 ceph-mon[97044]: pgmap v1347: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:11:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:09 smithi110 ceph-mon[97633]: pgmap v1347: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:11:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:11:10.319+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:11:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:11 smithi104 ceph-mon[97044]: pgmap v1348: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:11:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:11 smithi110 ceph-mon[97633]: pgmap v1348: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:13 smithi110 ceph-mon[97633]: pgmap v1349: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:13 smithi110 ceph-mon[97633]: osdmap e168: 12 total, 12 up, 12 in 2022-09-10T05:11:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:13 smithi104 ceph-mon[97044]: pgmap v1349: 129 pgs: 129 active+clean; 7.8 MiB data, 312 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:11:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:13 smithi104 ceph-mon[97044]: osdmap e168: 12 total, 12 up, 12 in 2022-09-10T05:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:15 smithi110 ceph-mon[97633]: pgmap v1351: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:11:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:15 smithi104 ceph-mon[97044]: pgmap v1351: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:11:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:17 smithi110 ceph-mon[97633]: pgmap v1352: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:11:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:17 smithi104 ceph-mon[97044]: pgmap v1352: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:11:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:19 smithi110 ceph-mon[97633]: pgmap v1353: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:11:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:19 smithi104 ceph-mon[97044]: pgmap v1353: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:11:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:21 smithi110 ceph-mon[97633]: pgmap v1354: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:11:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:21 smithi104 ceph-mon[97044]: pgmap v1354: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:11:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:23 smithi110 ceph-mon[97633]: pgmap v1355: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:11:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:23 smithi104 ceph-mon[97044]: pgmap v1355: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:11:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:11:25.328+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:11:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:25 smithi110 ceph-mon[97633]: pgmap v1356: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:11:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:25 smithi104 ceph-mon[97044]: pgmap v1356: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:11:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:27 smithi110 ceph-mon[97633]: pgmap v1357: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:27 smithi104 ceph-mon[97044]: pgmap v1357: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:29 smithi110 ceph-mon[97633]: pgmap v1358: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:29 smithi104 ceph-mon[97044]: pgmap v1358: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:31 smithi110 ceph-mon[97633]: pgmap v1359: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:31 smithi104 ceph-mon[97044]: pgmap v1359: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:33 smithi110 ceph-mon[97633]: pgmap v1360: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:33 smithi104 ceph-mon[97044]: pgmap v1360: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:35 smithi110 ceph-mon[97633]: pgmap v1361: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:35 smithi104 ceph-mon[97044]: pgmap v1361: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:37 smithi110 ceph-mon[97633]: pgmap v1362: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:37 smithi104 ceph-mon[97044]: pgmap v1362: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:39 smithi110 ceph-mon[97633]: pgmap v1363: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:39 smithi104 ceph-mon[97044]: pgmap v1363: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:11:40.331+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:11:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:41 smithi110 ceph-mon[97633]: pgmap v1364: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:41 smithi104 ceph-mon[97044]: pgmap v1364: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:43 smithi110 ceph-mon[97633]: pgmap v1365: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:43 smithi104 ceph-mon[97044]: pgmap v1365: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:11:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:11:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:11:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:11:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:11:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:11:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:11:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:11:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:45 smithi110 ceph-mon[97633]: pgmap v1366: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:45 smithi104 ceph-mon[97044]: pgmap v1366: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:47 smithi110 ceph-mon[97633]: pgmap v1367: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:47 smithi104 ceph-mon[97044]: pgmap v1367: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:49 smithi110 ceph-mon[97633]: pgmap v1368: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:49 smithi104 ceph-mon[97044]: pgmap v1368: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:11:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:51 smithi110 ceph-mon[97633]: pgmap v1369: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:51 smithi104 ceph-mon[97044]: pgmap v1369: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:53 smithi110 ceph-mon[97633]: pgmap v1370: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:11:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:53 smithi104 ceph-mon[97044]: pgmap v1370: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:11:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:11:55.337+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:11:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:55 smithi110 ceph-mon[97633]: pgmap v1371: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:55 smithi104 ceph-mon[97044]: pgmap v1371: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:11:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:57 smithi110 ceph-mon[97633]: pgmap v1372: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:11:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:57 smithi104 ceph-mon[97044]: pgmap v1372: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:11:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:11:59 smithi110 ceph-mon[97633]: pgmap v1373: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:12:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:11:59 smithi104 ceph-mon[97044]: pgmap v1373: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:12:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:01 smithi110 ceph-mon[97633]: pgmap v1374: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:01 smithi104 ceph-mon[97044]: pgmap v1374: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:03 smithi104 ceph-mon[97044]: pgmap v1375: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:03 smithi110 ceph-mon[97633]: pgmap v1375: 129 pgs: 129 active+clean; 7.8 MiB data, 313 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:05 smithi104 ceph-mon[97044]: pgmap v1376: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:12:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:05 smithi110 ceph-mon[97633]: pgmap v1376: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:12:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:07 smithi104 ceph-mon[97044]: pgmap v1377: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:12:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:07 smithi110 ceph-mon[97633]: pgmap v1377: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:12:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:08 smithi104 ceph-mon[97044]: osdmap e169: 12 total, 12 up, 12 in 2022-09-10T05:12:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:08 smithi110 ceph-mon[97633]: osdmap e169: 12 total, 12 up, 12 in 2022-09-10T05:12:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:09 smithi104 ceph-mon[97044]: pgmap v1379: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:12:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:09 smithi110 ceph-mon[97633]: pgmap v1379: 129 pgs: 129 active+clean; 7.8 MiB data, 318 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:12:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:12:10.347+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:12:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:11 smithi104 ceph-mon[97044]: pgmap v1380: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:12:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:11 smithi110 ceph-mon[97633]: pgmap v1380: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:13 smithi110 ceph-mon[97633]: pgmap v1381: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:13 smithi110 ceph-mon[97633]: osdmap e170: 12 total, 12 up, 12 in 2022-09-10T05:12:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:13 smithi104 ceph-mon[97044]: pgmap v1381: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:12:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:13 smithi104 ceph-mon[97044]: osdmap e170: 12 total, 12 up, 12 in 2022-09-10T05:12:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:15 smithi110 ceph-mon[97633]: pgmap v1383: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:12:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:15 smithi104 ceph-mon[97044]: pgmap v1383: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:12:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:17 smithi110 ceph-mon[97633]: pgmap v1384: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:12:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:17 smithi104 ceph-mon[97044]: pgmap v1384: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:12:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:19 smithi110 ceph-mon[97633]: pgmap v1385: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:12:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:19 smithi104 ceph-mon[97044]: pgmap v1385: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:12:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:21 smithi110 ceph-mon[97633]: pgmap v1386: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:12:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:21 smithi104 ceph-mon[97044]: pgmap v1386: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:12:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:23 smithi110 ceph-mon[97633]: pgmap v1387: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:12:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:23 smithi104 ceph-mon[97044]: pgmap v1387: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:12:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:12:25.351+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:12:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:25 smithi110 ceph-mon[97633]: pgmap v1388: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:12:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:25 smithi104 ceph-mon[97044]: pgmap v1388: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:12:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:27 smithi110 ceph-mon[97633]: pgmap v1389: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:27 smithi104 ceph-mon[97044]: pgmap v1389: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:29 smithi110 ceph-mon[97633]: pgmap v1390: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:29 smithi104 ceph-mon[97044]: pgmap v1390: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:31 smithi110 ceph-mon[97633]: pgmap v1391: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:31 smithi104 ceph-mon[97044]: pgmap v1391: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:33 smithi110 ceph-mon[97633]: pgmap v1392: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:33 smithi104 ceph-mon[97044]: pgmap v1392: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:35 smithi110 ceph-mon[97633]: pgmap v1393: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:35 smithi104 ceph-mon[97044]: pgmap v1393: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:37 smithi110 ceph-mon[97633]: pgmap v1394: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:37 smithi104 ceph-mon[97044]: pgmap v1394: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:39 smithi110 ceph-mon[97633]: pgmap v1395: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:39 smithi104 ceph-mon[97044]: pgmap v1395: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:12:40.353+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:12:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:41 smithi110 ceph-mon[97633]: pgmap v1396: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:41 smithi104 ceph-mon[97044]: pgmap v1396: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:43 smithi110 ceph-mon[97633]: pgmap v1397: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:43 smithi104 ceph-mon[97044]: pgmap v1397: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:12:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:12:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:12:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:12:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:12:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:12:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:12:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:45 smithi110 ceph-mon[97633]: pgmap v1398: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:45 smithi104 ceph-mon[97044]: pgmap v1398: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:47 smithi110 ceph-mon[97633]: pgmap v1399: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:47 smithi104 ceph-mon[97044]: pgmap v1399: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:49 smithi110 ceph-mon[97633]: pgmap v1400: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:49 smithi104 ceph-mon[97044]: pgmap v1400: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:51 smithi110 ceph-mon[97633]: pgmap v1401: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:51 smithi104 ceph-mon[97044]: pgmap v1401: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:53 smithi110 ceph-mon[97633]: pgmap v1402: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:53 smithi104 ceph-mon[97044]: pgmap v1402: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:12:55.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:12:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:55 smithi104 ceph-mon[97044]: pgmap v1403: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:55 smithi110 ceph-mon[97633]: pgmap v1403: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:12:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:57 smithi104 ceph-mon[97044]: pgmap v1404: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:12:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:57 smithi110 ceph-mon[97633]: pgmap v1404: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:12:59 smithi104 ceph-mon[97044]: pgmap v1405: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:12:59 smithi110 ceph-mon[97633]: pgmap v1405: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:01 smithi104 ceph-mon[97044]: pgmap v1406: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:01 smithi110 ceph-mon[97633]: pgmap v1406: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:03 smithi104 ceph-mon[97044]: pgmap v1407: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:03 smithi110 ceph-mon[97633]: pgmap v1407: 129 pgs: 129 active+clean; 7.8 MiB data, 319 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:05 smithi104 ceph-mon[97044]: pgmap v1408: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:13:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:05 smithi110 ceph-mon[97633]: pgmap v1408: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:13:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:07 smithi104 ceph-mon[97044]: pgmap v1409: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:13:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:07 smithi110 ceph-mon[97633]: pgmap v1409: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:13:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:08 smithi104 ceph-mon[97044]: osdmap e171: 12 total, 12 up, 12 in 2022-09-10T05:13:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:08 smithi110 ceph-mon[97633]: osdmap e171: 12 total, 12 up, 12 in 2022-09-10T05:13:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:09 smithi104 ceph-mon[97044]: pgmap v1411: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:13:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:09 smithi110 ceph-mon[97633]: pgmap v1411: 129 pgs: 129 active+clean; 7.8 MiB data, 324 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:13:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:13:10.369+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:13:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:11 smithi104 ceph-mon[97044]: pgmap v1412: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:13:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:11 smithi110 ceph-mon[97633]: pgmap v1412: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:13:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:13 smithi110 ceph-mon[97633]: pgmap v1413: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:13:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:13 smithi110 ceph-mon[97633]: osdmap e172: 12 total, 12 up, 12 in 2022-09-10T05:13:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:13 smithi104 ceph-mon[97044]: pgmap v1413: 129 pgs: 129 active+clean; 7.8 MiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:13:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:13 smithi104 ceph-mon[97044]: osdmap e172: 12 total, 12 up, 12 in 2022-09-10T05:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:15 smithi110 ceph-mon[97633]: pgmap v1415: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:13:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:15 smithi104 ceph-mon[97044]: pgmap v1415: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:13:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:17 smithi110 ceph-mon[97633]: pgmap v1416: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:13:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:17 smithi104 ceph-mon[97044]: pgmap v1416: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:13:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:19 smithi110 ceph-mon[97633]: pgmap v1417: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:13:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:19 smithi104 ceph-mon[97044]: pgmap v1417: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:13:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:21 smithi110 ceph-mon[97633]: pgmap v1418: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:13:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:21 smithi104 ceph-mon[97044]: pgmap v1418: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:13:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:23 smithi110 ceph-mon[97633]: pgmap v1419: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:13:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:23 smithi104 ceph-mon[97044]: pgmap v1419: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:13:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:13:25.385+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:13:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:25 smithi110 ceph-mon[97633]: pgmap v1420: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:13:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:25 smithi104 ceph-mon[97044]: pgmap v1420: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:13:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:27 smithi110 ceph-mon[97633]: pgmap v1421: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:27 smithi104 ceph-mon[97044]: pgmap v1421: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:29 smithi110 ceph-mon[97633]: pgmap v1422: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:29 smithi104 ceph-mon[97044]: pgmap v1422: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:31 smithi110 ceph-mon[97633]: pgmap v1423: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:31 smithi104 ceph-mon[97044]: pgmap v1423: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:33 smithi110 ceph-mon[97633]: pgmap v1424: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:33 smithi104 ceph-mon[97044]: pgmap v1424: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:35 smithi110 ceph-mon[97633]: pgmap v1425: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:35 smithi104 ceph-mon[97044]: pgmap v1425: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:37 smithi110 ceph-mon[97633]: pgmap v1426: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:37 smithi104 ceph-mon[97044]: pgmap v1426: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:39 smithi110 ceph-mon[97633]: pgmap v1427: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:39 smithi104 ceph-mon[97044]: pgmap v1427: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:13:40.390+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:13:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:41 smithi110 ceph-mon[97633]: pgmap v1428: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:41 smithi104 ceph-mon[97044]: pgmap v1428: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:43 smithi110 ceph-mon[97633]: pgmap v1429: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:43 smithi104 ceph-mon[97044]: pgmap v1429: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:13:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:13:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:13:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:13:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:13:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:45 smithi104 ceph-mon[97044]: pgmap v1430: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:45 smithi110 ceph-mon[97633]: pgmap v1430: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:46.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:13:46.803+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T05:13:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:47 smithi104 ceph-mon[97044]: pgmap v1431: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:47 smithi110 ceph-mon[97633]: pgmap v1431: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:49 smithi104 ceph-mon[97044]: pgmap v1432: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:49 smithi110 ceph-mon[97633]: pgmap v1432: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:51 smithi104 ceph-mon[97044]: pgmap v1433: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:51 smithi110 ceph-mon[97633]: pgmap v1433: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:53 smithi104 ceph-mon[97044]: pgmap v1434: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:53 smithi110 ceph-mon[97633]: pgmap v1434: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:13:55.398+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:13:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:55 smithi104 ceph-mon[97044]: pgmap v1435: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:55 smithi110 ceph-mon[97633]: pgmap v1435: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:13:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:57 smithi104 ceph-mon[97044]: pgmap v1436: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:13:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:57 smithi110 ceph-mon[97633]: pgmap v1436: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:13:59 smithi104 ceph-mon[97044]: pgmap v1437: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:13:59 smithi110 ceph-mon[97633]: pgmap v1437: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:01 smithi104 ceph-mon[97044]: pgmap v1438: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:01 smithi110 ceph-mon[97633]: pgmap v1438: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:03 smithi104 ceph-mon[97044]: pgmap v1439: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:03 smithi110 ceph-mon[97633]: pgmap v1439: 129 pgs: 129 active+clean; 7.8 MiB data, 338 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:05 smithi104 ceph-mon[97044]: pgmap v1440: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:14:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:05 smithi110 ceph-mon[97633]: pgmap v1440: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:14:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:07 smithi104 ceph-mon[97044]: pgmap v1441: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:14:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:07 smithi110 ceph-mon[97633]: pgmap v1441: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:14:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:08 smithi104 ceph-mon[97044]: osdmap e173: 12 total, 12 up, 12 in 2022-09-10T05:14:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:08 smithi110 ceph-mon[97633]: osdmap e173: 12 total, 12 up, 12 in 2022-09-10T05:14:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:09 smithi104 ceph-mon[97044]: pgmap v1443: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:14:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:09 smithi110 ceph-mon[97633]: pgmap v1443: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:14:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:14:10.402+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:14:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:11 smithi104 ceph-mon[97044]: pgmap v1444: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:14:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:11 smithi110 ceph-mon[97633]: pgmap v1444: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:14:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:13 smithi110 ceph-mon[97633]: pgmap v1445: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:14:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:13 smithi110 ceph-mon[97633]: osdmap e174: 12 total, 12 up, 12 in 2022-09-10T05:14:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:13 smithi104 ceph-mon[97044]: pgmap v1445: 129 pgs: 129 active+clean; 7.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:14:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:13 smithi104 ceph-mon[97044]: osdmap e174: 12 total, 12 up, 12 in 2022-09-10T05:14:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:15 smithi110 ceph-mon[97633]: pgmap v1447: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:14:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:15 smithi104 ceph-mon[97044]: pgmap v1447: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:14:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:17 smithi110 ceph-mon[97633]: pgmap v1448: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:14:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:17 smithi104 ceph-mon[97044]: pgmap v1448: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:14:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:19 smithi110 ceph-mon[97633]: pgmap v1449: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:14:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:19 smithi104 ceph-mon[97044]: pgmap v1449: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:14:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:21 smithi110 ceph-mon[97633]: pgmap v1450: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:14:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:21 smithi104 ceph-mon[97044]: pgmap v1450: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:14:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:23 smithi110 ceph-mon[97633]: pgmap v1451: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:14:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:23 smithi104 ceph-mon[97044]: pgmap v1451: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:14:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:14:25.413+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:14:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:25 smithi110 ceph-mon[97633]: pgmap v1452: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:14:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:25 smithi104 ceph-mon[97044]: pgmap v1452: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:14:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:27 smithi110 ceph-mon[97633]: pgmap v1453: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:27 smithi104 ceph-mon[97044]: pgmap v1453: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:29 smithi110 ceph-mon[97633]: pgmap v1454: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:29 smithi104 ceph-mon[97044]: pgmap v1454: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:31 smithi110 ceph-mon[97633]: pgmap v1455: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:31 smithi104 ceph-mon[97044]: pgmap v1455: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:33 smithi110 ceph-mon[97633]: pgmap v1456: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:33 smithi104 ceph-mon[97044]: pgmap v1456: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:35 smithi104 ceph-mon[97044]: pgmap v1457: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:35 smithi110 ceph-mon[97633]: pgmap v1457: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:37 smithi104 ceph-mon[97044]: pgmap v1458: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:14:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:37 smithi110 ceph-mon[97633]: pgmap v1458: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:14:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:39 smithi104 ceph-mon[97044]: pgmap v1459: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:14:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:39 smithi110 ceph-mon[97633]: pgmap v1459: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:14:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:14:40.417+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:14:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:41 smithi104 ceph-mon[97044]: pgmap v1460: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:41 smithi110 ceph-mon[97633]: pgmap v1460: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:43 smithi104 ceph-mon[97044]: pgmap v1461: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:14:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:43 smithi110 ceph-mon[97633]: pgmap v1461: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:14:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:14:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:14:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:14:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:14:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:14:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:14:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:14:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:14:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:45 smithi104 ceph-mon[97044]: pgmap v1462: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:45 smithi110 ceph-mon[97633]: pgmap v1462: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:47 smithi104 ceph-mon[97044]: pgmap v1463: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:47 smithi110 ceph-mon[97633]: pgmap v1463: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:49 smithi104 ceph-mon[97044]: pgmap v1464: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:49 smithi110 ceph-mon[97633]: pgmap v1464: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:51 smithi104 ceph-mon[97044]: pgmap v1465: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:51 smithi110 ceph-mon[97633]: pgmap v1465: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:53 smithi104 ceph-mon[97044]: pgmap v1466: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:53 smithi110 ceph-mon[97633]: pgmap v1466: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:14:55.429+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:14:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:55 smithi104 ceph-mon[97044]: pgmap v1467: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:55 smithi110 ceph-mon[97633]: pgmap v1467: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:14:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:57 smithi104 ceph-mon[97044]: pgmap v1468: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:14:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:57 smithi110 ceph-mon[97633]: pgmap v1468: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:14:59 smithi104 ceph-mon[97044]: pgmap v1469: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:14:59 smithi110 ceph-mon[97633]: pgmap v1469: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:01 smithi104 ceph-mon[97044]: pgmap v1470: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:01 smithi110 ceph-mon[97633]: pgmap v1470: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:03 smithi104 ceph-mon[97044]: pgmap v1471: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:03 smithi110 ceph-mon[97633]: pgmap v1471: 129 pgs: 129 active+clean; 7.8 MiB data, 340 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:05 smithi104 ceph-mon[97044]: pgmap v1472: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:15:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:05 smithi110 ceph-mon[97633]: pgmap v1472: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:15:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:07 smithi104 ceph-mon[97044]: pgmap v1473: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:15:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:07 smithi110 ceph-mon[97633]: pgmap v1473: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:15:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:08 smithi104 ceph-mon[97044]: osdmap e175: 12 total, 12 up, 12 in 2022-09-10T05:15:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:08 smithi110 ceph-mon[97633]: osdmap e175: 12 total, 12 up, 12 in 2022-09-10T05:15:10.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:09 smithi104 ceph-mon[97044]: pgmap v1475: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:15:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:09 smithi110 ceph-mon[97633]: pgmap v1475: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:15:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:15:10.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:15:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:11 smithi104 ceph-mon[97044]: pgmap v1476: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:15:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:11 smithi110 ceph-mon[97633]: pgmap v1476: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:15:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:13 smithi110 ceph-mon[97633]: pgmap v1477: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:15:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:13 smithi110 ceph-mon[97633]: osdmap e176: 12 total, 12 up, 12 in 2022-09-10T05:15:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:13 smithi104 ceph-mon[97044]: pgmap v1477: 129 pgs: 129 active+clean; 7.8 MiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:15:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:13 smithi104 ceph-mon[97044]: osdmap e176: 12 total, 12 up, 12 in 2022-09-10T05:15:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:15 smithi110 ceph-mon[97633]: pgmap v1479: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:15:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:15 smithi104 ceph-mon[97044]: pgmap v1479: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:15:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:17 smithi110 ceph-mon[97633]: pgmap v1480: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:15:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:17 smithi104 ceph-mon[97044]: pgmap v1480: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:15:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:19 smithi110 ceph-mon[97633]: pgmap v1481: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:15:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:19 smithi104 ceph-mon[97044]: pgmap v1481: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:15:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:21 smithi110 ceph-mon[97633]: pgmap v1482: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:15:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:21 smithi104 ceph-mon[97044]: pgmap v1482: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:15:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:23 smithi110 ceph-mon[97633]: pgmap v1483: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:15:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:23 smithi104 ceph-mon[97044]: pgmap v1483: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:15:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:15:25.442+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:15:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:25 smithi104 ceph-mon[97044]: pgmap v1484: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:15:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:25 smithi110 ceph-mon[97633]: pgmap v1484: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:15:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:27 smithi104 ceph-mon[97044]: pgmap v1485: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:27 smithi110 ceph-mon[97633]: pgmap v1485: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:29 smithi104 ceph-mon[97044]: pgmap v1486: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:29 smithi110 ceph-mon[97633]: pgmap v1486: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:31 smithi104 ceph-mon[97044]: pgmap v1487: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:31 smithi110 ceph-mon[97633]: pgmap v1487: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:33 smithi104 ceph-mon[97044]: pgmap v1488: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:33 smithi110 ceph-mon[97633]: pgmap v1488: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:35 smithi104 ceph-mon[97044]: pgmap v1489: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:35 smithi110 ceph-mon[97633]: pgmap v1489: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:37 smithi104 ceph-mon[97044]: pgmap v1490: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:37 smithi110 ceph-mon[97633]: pgmap v1490: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:39 smithi104 ceph-mon[97044]: pgmap v1491: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:39 smithi110 ceph-mon[97633]: pgmap v1491: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:15:40.442+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:15:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:41 smithi104 ceph-mon[97044]: pgmap v1492: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:41 smithi110 ceph-mon[97633]: pgmap v1492: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:43 smithi104 ceph-mon[97044]: pgmap v1493: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:43 smithi110 ceph-mon[97633]: pgmap v1493: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:15:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:15:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:15:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:15:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:15:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:15:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:15:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:15:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:45 smithi104 ceph-mon[97044]: pgmap v1494: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:45 smithi110 ceph-mon[97633]: pgmap v1494: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:47 smithi104 ceph-mon[97044]: pgmap v1495: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:47 smithi110 ceph-mon[97633]: pgmap v1495: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:50.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:49 smithi104 ceph-mon[97044]: pgmap v1496: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:49 smithi110 ceph-mon[97633]: pgmap v1496: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:15:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:51 smithi104 ceph-mon[97044]: pgmap v1497: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:51 smithi110 ceph-mon[97633]: pgmap v1497: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:53 smithi104 ceph-mon[97044]: pgmap v1498: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:15:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:53 smithi110 ceph-mon[97633]: pgmap v1498: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:15:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:15:55.445+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:15:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:55 smithi104 ceph-mon[97044]: pgmap v1499: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:55 smithi110 ceph-mon[97633]: pgmap v1499: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:15:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:57 smithi104 ceph-mon[97044]: pgmap v1500: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:15:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:57 smithi110 ceph-mon[97633]: pgmap v1500: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:16:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:15:59 smithi104 ceph-mon[97044]: pgmap v1501: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:16:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:15:59 smithi110 ceph-mon[97633]: pgmap v1501: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:16:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:01 smithi104 ceph-mon[97044]: pgmap v1502: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:01 smithi110 ceph-mon[97633]: pgmap v1502: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:03 smithi104 ceph-mon[97044]: pgmap v1503: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:03 smithi110 ceph-mon[97633]: pgmap v1503: 129 pgs: 129 active+clean; 7.8 MiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:05 smithi104 ceph-mon[97044]: pgmap v1504: 129 pgs: 129 active+clean; 7.8 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:16:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:05 smithi110 ceph-mon[97633]: pgmap v1504: 129 pgs: 129 active+clean; 7.8 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:16:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:07 smithi104 ceph-mon[97044]: pgmap v1505: 129 pgs: 129 active+clean; 7.8 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:16:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:07 smithi110 ceph-mon[97633]: pgmap v1505: 129 pgs: 129 active+clean; 7.8 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:16:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:08 smithi104 ceph-mon[97044]: osdmap e177: 12 total, 12 up, 12 in 2022-09-10T05:16:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:08 smithi110 ceph-mon[97633]: osdmap e177: 12 total, 12 up, 12 in 2022-09-10T05:16:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:09 smithi104 ceph-mon[97044]: pgmap v1507: 129 pgs: 129 active+clean; 7.8 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:16:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:09 smithi110 ceph-mon[97633]: pgmap v1507: 129 pgs: 129 active+clean; 7.8 MiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:16:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:16:10.454+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:16:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:11 smithi104 ceph-mon[97044]: pgmap v1508: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:16:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:11 smithi110 ceph-mon[97633]: pgmap v1508: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:16:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:13 smithi104 ceph-mon[97044]: pgmap v1509: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:16:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:13 smithi104 ceph-mon[97044]: osdmap e178: 12 total, 12 up, 12 in 2022-09-10T05:16:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:13 smithi110 ceph-mon[97633]: pgmap v1509: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:16:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:13 smithi110 ceph-mon[97633]: osdmap e178: 12 total, 12 up, 12 in 2022-09-10T05:16:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:15 smithi104 ceph-mon[97044]: pgmap v1511: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:16:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:15 smithi110 ceph-mon[97633]: pgmap v1511: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:16:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:17 smithi104 ceph-mon[97044]: pgmap v1512: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:16:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:17 smithi110 ceph-mon[97633]: pgmap v1512: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:16:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:19 smithi104 ceph-mon[97044]: pgmap v1513: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:16:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:19 smithi110 ceph-mon[97633]: pgmap v1513: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:16:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:21 smithi104 ceph-mon[97044]: pgmap v1514: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:16:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:21 smithi110 ceph-mon[97633]: pgmap v1514: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:16:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:23 smithi104 ceph-mon[97044]: pgmap v1515: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:16:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:23 smithi110 ceph-mon[97633]: pgmap v1515: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:16:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:16:25.464+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:16:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:25 smithi104 ceph-mon[97044]: pgmap v1516: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:16:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:25 smithi110 ceph-mon[97633]: pgmap v1516: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:16:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:27 smithi104 ceph-mon[97044]: pgmap v1517: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:27 smithi110 ceph-mon[97633]: pgmap v1517: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:29 smithi104 ceph-mon[97044]: pgmap v1518: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:29 smithi110 ceph-mon[97633]: pgmap v1518: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:31 smithi104 ceph-mon[97044]: pgmap v1519: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:31 smithi110 ceph-mon[97633]: pgmap v1519: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:33 smithi104 ceph-mon[97044]: pgmap v1520: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:33 smithi110 ceph-mon[97633]: pgmap v1520: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:35 smithi104 ceph-mon[97044]: pgmap v1521: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:35 smithi110 ceph-mon[97633]: pgmap v1521: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:37 smithi104 ceph-mon[97044]: pgmap v1522: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:37 smithi110 ceph-mon[97633]: pgmap v1522: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:39 smithi104 ceph-mon[97044]: pgmap v1523: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:39 smithi110 ceph-mon[97633]: pgmap v1523: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:40.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:16:40.473+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:16:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:41 smithi104 ceph-mon[97044]: pgmap v1524: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:41 smithi110 ceph-mon[97633]: pgmap v1524: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:43 smithi104 ceph-mon[97044]: pgmap v1525: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:16:44.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:16:44.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:16:44.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:16:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:43 smithi110 ceph-mon[97633]: pgmap v1525: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:16:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:16:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:16:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:16:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:45 smithi104 ceph-mon[97044]: pgmap v1526: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:45 smithi110 ceph-mon[97633]: pgmap v1526: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:47 smithi104 ceph-mon[97044]: pgmap v1527: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:47 smithi110 ceph-mon[97633]: pgmap v1527: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:49 smithi104 ceph-mon[97044]: pgmap v1528: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:49 smithi110 ceph-mon[97633]: pgmap v1528: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:51 smithi104 ceph-mon[97044]: pgmap v1529: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:51 smithi110 ceph-mon[97633]: pgmap v1529: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:53 smithi104 ceph-mon[97044]: pgmap v1530: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:53 smithi110 ceph-mon[97633]: pgmap v1530: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:55.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:16:55.482+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:16:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:55 smithi104 ceph-mon[97044]: pgmap v1531: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:55 smithi110 ceph-mon[97633]: pgmap v1531: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:16:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:57 smithi104 ceph-mon[97044]: pgmap v1532: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:16:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:57 smithi110 ceph-mon[97633]: pgmap v1532: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:16:59 smithi104 ceph-mon[97044]: pgmap v1533: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:16:59 smithi110 ceph-mon[97633]: pgmap v1533: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:01 smithi104 ceph-mon[97044]: pgmap v1534: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:01 smithi110 ceph-mon[97633]: pgmap v1534: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:03 smithi104 ceph-mon[97044]: pgmap v1535: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:03 smithi110 ceph-mon[97633]: pgmap v1535: 129 pgs: 129 active+clean; 7.8 MiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:05 smithi104 ceph-mon[97044]: pgmap v1536: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:17:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:05 smithi110 ceph-mon[97633]: pgmap v1536: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:17:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:07 smithi104 ceph-mon[97044]: pgmap v1537: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:17:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:07 smithi110 ceph-mon[97633]: pgmap v1537: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:17:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:08 smithi104 ceph-mon[97044]: osdmap e179: 12 total, 12 up, 12 in 2022-09-10T05:17:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:08 smithi110 ceph-mon[97633]: osdmap e179: 12 total, 12 up, 12 in 2022-09-10T05:17:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:09 smithi104 ceph-mon[97044]: pgmap v1539: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:17:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:09 smithi110 ceph-mon[97633]: pgmap v1539: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:17:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:17:10.484+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:17:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:11 smithi104 ceph-mon[97044]: pgmap v1540: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:17:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:11 smithi110 ceph-mon[97633]: pgmap v1540: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:17:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:13 smithi104 ceph-mon[97044]: pgmap v1541: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:17:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:13 smithi104 ceph-mon[97044]: osdmap e180: 12 total, 12 up, 12 in 2022-09-10T05:17:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:13 smithi110 ceph-mon[97633]: pgmap v1541: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:17:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:13 smithi110 ceph-mon[97633]: osdmap e180: 12 total, 12 up, 12 in 2022-09-10T05:17:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:15 smithi104 ceph-mon[97044]: pgmap v1543: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:17:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:15 smithi110 ceph-mon[97633]: pgmap v1543: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:17:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:17 smithi104 ceph-mon[97044]: pgmap v1544: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:17:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:17 smithi110 ceph-mon[97633]: pgmap v1544: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:17:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:19 smithi104 ceph-mon[97044]: pgmap v1545: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:17:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:19 smithi110 ceph-mon[97633]: pgmap v1545: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:17:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:21 smithi104 ceph-mon[97044]: pgmap v1546: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:17:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:21 smithi110 ceph-mon[97633]: pgmap v1546: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:17:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:23 smithi104 ceph-mon[97044]: pgmap v1547: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:17:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:23 smithi110 ceph-mon[97633]: pgmap v1547: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:17:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:17:25.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:17:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:25 smithi104 ceph-mon[97044]: pgmap v1548: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:17:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:25 smithi110 ceph-mon[97633]: pgmap v1548: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:17:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:27 smithi104 ceph-mon[97044]: pgmap v1549: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:27 smithi110 ceph-mon[97633]: pgmap v1549: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:29 smithi104 ceph-mon[97044]: pgmap v1550: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:29 smithi110 ceph-mon[97633]: pgmap v1550: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:31 smithi104 ceph-mon[97044]: pgmap v1551: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:31 smithi110 ceph-mon[97633]: pgmap v1551: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:33 smithi104 ceph-mon[97044]: pgmap v1552: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:33 smithi110 ceph-mon[97633]: pgmap v1552: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:35 smithi104 ceph-mon[97044]: pgmap v1553: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:35 smithi110 ceph-mon[97633]: pgmap v1553: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:37 smithi104 ceph-mon[97044]: pgmap v1554: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:37 smithi110 ceph-mon[97633]: pgmap v1554: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:39 smithi104 ceph-mon[97044]: pgmap v1555: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:39 smithi110 ceph-mon[97633]: pgmap v1555: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:17:40.487+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:17:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:41 smithi104 ceph-mon[97044]: pgmap v1556: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:41 smithi110 ceph-mon[97633]: pgmap v1556: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:43 smithi104 ceph-mon[97044]: pgmap v1557: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:17:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:17:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:17:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:17:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:43 smithi110 ceph-mon[97633]: pgmap v1557: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:17:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:17:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:17:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:17:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:45 smithi104 ceph-mon[97044]: pgmap v1558: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:45 smithi110 ceph-mon[97633]: pgmap v1558: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:47 smithi104 ceph-mon[97044]: pgmap v1559: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:47 smithi110 ceph-mon[97633]: pgmap v1559: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:49 smithi104 ceph-mon[97044]: pgmap v1560: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:49 smithi110 ceph-mon[97633]: pgmap v1560: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:51 smithi104 ceph-mon[97044]: pgmap v1561: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:51 smithi110 ceph-mon[97633]: pgmap v1561: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:53 smithi104 ceph-mon[97044]: pgmap v1562: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:53 smithi110 ceph-mon[97633]: pgmap v1562: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:17:55.502+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:17:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:55 smithi104 ceph-mon[97044]: pgmap v1563: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:55 smithi110 ceph-mon[97633]: pgmap v1563: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:17:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:57 smithi104 ceph-mon[97044]: pgmap v1564: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:17:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:57 smithi110 ceph-mon[97633]: pgmap v1564: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:17:59 smithi104 ceph-mon[97044]: pgmap v1565: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:17:59 smithi110 ceph-mon[97633]: pgmap v1565: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:01 smithi104 ceph-mon[97044]: pgmap v1566: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:01 smithi110 ceph-mon[97633]: pgmap v1566: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:03 smithi104 ceph-mon[97044]: pgmap v1567: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:03 smithi110 ceph-mon[97633]: pgmap v1567: 129 pgs: 129 active+clean; 7.8 MiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:05 smithi104 ceph-mon[97044]: pgmap v1568: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:18:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:05 smithi110 ceph-mon[97633]: pgmap v1568: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:18:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:07 smithi104 ceph-mon[97044]: pgmap v1569: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:18:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:07 smithi110 ceph-mon[97633]: pgmap v1569: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:18:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:08 smithi104 ceph-mon[97044]: osdmap e181: 12 total, 12 up, 12 in 2022-09-10T05:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:08 smithi110 ceph-mon[97633]: osdmap e181: 12 total, 12 up, 12 in 2022-09-10T05:18:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:09 smithi104 ceph-mon[97044]: pgmap v1571: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:18:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:09 smithi110 ceph-mon[97633]: pgmap v1571: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:18:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:18:10.516+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:18:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:11 smithi104 ceph-mon[97044]: pgmap v1572: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:18:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:11 smithi110 ceph-mon[97633]: pgmap v1572: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:18:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:13 smithi104 ceph-mon[97044]: pgmap v1573: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:18:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:13 smithi104 ceph-mon[97044]: osdmap e182: 12 total, 12 up, 12 in 2022-09-10T05:18:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:13 smithi110 ceph-mon[97633]: pgmap v1573: 129 pgs: 129 active+clean; 7.8 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:18:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:13 smithi110 ceph-mon[97633]: osdmap e182: 12 total, 12 up, 12 in 2022-09-10T05:18:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:15 smithi104 ceph-mon[97044]: pgmap v1575: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:18:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:15 smithi110 ceph-mon[97633]: pgmap v1575: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:18:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:17 smithi104 ceph-mon[97044]: pgmap v1576: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:18:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:17 smithi110 ceph-mon[97633]: pgmap v1576: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:18:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:19 smithi104 ceph-mon[97044]: pgmap v1577: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:18:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:19 smithi110 ceph-mon[97633]: pgmap v1577: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:18:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:21 smithi104 ceph-mon[97044]: pgmap v1578: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:18:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:21 smithi110 ceph-mon[97633]: pgmap v1578: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:18:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:23 smithi104 ceph-mon[97044]: pgmap v1579: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:18:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:23 smithi110 ceph-mon[97633]: pgmap v1579: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:18:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:18:25.532+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:18:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:25 smithi104 ceph-mon[97044]: pgmap v1580: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:18:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:25 smithi110 ceph-mon[97633]: pgmap v1580: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:18:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:27 smithi104 ceph-mon[97044]: pgmap v1581: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:27 smithi110 ceph-mon[97633]: pgmap v1581: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:29 smithi104 ceph-mon[97044]: pgmap v1582: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:29 smithi110 ceph-mon[97633]: pgmap v1582: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:31 smithi104 ceph-mon[97044]: pgmap v1583: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:31 smithi110 ceph-mon[97633]: pgmap v1583: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:33 smithi104 ceph-mon[97044]: pgmap v1584: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:33 smithi110 ceph-mon[97633]: pgmap v1584: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:35 smithi104 ceph-mon[97044]: pgmap v1585: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:35 smithi110 ceph-mon[97633]: pgmap v1585: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:37.970 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:37 smithi104 ceph-mon[97044]: pgmap v1586: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:37 smithi110 ceph-mon[97633]: pgmap v1586: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:39 smithi104 ceph-mon[97044]: pgmap v1587: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:39 smithi110 ceph-mon[97633]: pgmap v1587: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:18:40.547+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:18:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:41 smithi104 ceph-mon[97044]: pgmap v1588: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:41 smithi110 ceph-mon[97633]: pgmap v1588: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:43 smithi104 ceph-mon[97044]: pgmap v1589: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:18:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:18:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:18:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:18:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:43 smithi110 ceph-mon[97633]: pgmap v1589: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:18:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:18:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:18:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:18:46.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:45 smithi104 ceph-mon[97044]: pgmap v1590: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:45 smithi110 ceph-mon[97633]: pgmap v1590: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:46.804 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:18:46.803+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T05:18:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:47 smithi104 ceph-mon[97044]: pgmap v1591: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:47 smithi110 ceph-mon[97633]: pgmap v1591: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:49 smithi104 ceph-mon[97044]: pgmap v1592: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:49 smithi110 ceph-mon[97633]: pgmap v1592: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:51 smithi104 ceph-mon[97044]: pgmap v1593: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:51 smithi110 ceph-mon[97633]: pgmap v1593: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:53 smithi104 ceph-mon[97044]: pgmap v1594: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:53 smithi110 ceph-mon[97633]: pgmap v1594: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:18:55.550+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:18:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:55 smithi104 ceph-mon[97044]: pgmap v1595: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:55 smithi110 ceph-mon[97633]: pgmap v1595: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:18:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:57 smithi104 ceph-mon[97044]: pgmap v1596: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:18:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:57 smithi110 ceph-mon[97633]: pgmap v1596: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:18:59 smithi104 ceph-mon[97044]: pgmap v1597: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:00.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:18:59 smithi110 ceph-mon[97633]: pgmap v1597: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:01 smithi104 ceph-mon[97044]: pgmap v1598: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:01 smithi110 ceph-mon[97633]: pgmap v1598: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:03 smithi104 ceph-mon[97044]: pgmap v1599: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:03 smithi110 ceph-mon[97633]: pgmap v1599: 129 pgs: 129 active+clean; 7.8 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:05 smithi104 ceph-mon[97044]: pgmap v1600: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:19:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:05 smithi110 ceph-mon[97633]: pgmap v1600: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:19:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:07 smithi104 ceph-mon[97044]: pgmap v1601: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:19:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:07 smithi110 ceph-mon[97633]: pgmap v1601: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:19:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:08 smithi104 ceph-mon[97044]: osdmap e183: 12 total, 12 up, 12 in 2022-09-10T05:19:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:08 smithi110 ceph-mon[97633]: osdmap e183: 12 total, 12 up, 12 in 2022-09-10T05:19:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:09 smithi104 ceph-mon[97044]: pgmap v1603: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:19:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:09 smithi110 ceph-mon[97633]: pgmap v1603: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:19:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:19:10.564+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:19:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:11 smithi104 ceph-mon[97044]: pgmap v1604: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:19:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:11 smithi110 ceph-mon[97633]: pgmap v1604: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:19:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:13 smithi104 ceph-mon[97044]: pgmap v1605: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:19:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:13 smithi104 ceph-mon[97044]: osdmap e184: 12 total, 12 up, 12 in 2022-09-10T05:19:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:13 smithi110 ceph-mon[97633]: pgmap v1605: 129 pgs: 129 active+clean; 7.8 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:19:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:13 smithi110 ceph-mon[97633]: osdmap e184: 12 total, 12 up, 12 in 2022-09-10T05:19:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:15 smithi104 ceph-mon[97044]: pgmap v1607: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:19:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:15 smithi110 ceph-mon[97633]: pgmap v1607: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:19:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:17 smithi104 ceph-mon[97044]: pgmap v1608: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:19:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:17 smithi110 ceph-mon[97633]: pgmap v1608: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:19:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:19 smithi104 ceph-mon[97044]: pgmap v1609: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:19:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:19 smithi110 ceph-mon[97633]: pgmap v1609: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:19:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:21 smithi104 ceph-mon[97044]: pgmap v1610: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:19:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:21 smithi110 ceph-mon[97633]: pgmap v1610: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:19:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:23 smithi104 ceph-mon[97044]: pgmap v1611: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:19:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:23 smithi110 ceph-mon[97633]: pgmap v1611: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:19:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:25 smithi104 ceph-mon[97044]: pgmap v1612: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:19:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:19:25.570+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:19:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:25 smithi110 ceph-mon[97633]: pgmap v1612: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:19:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:27 smithi104 ceph-mon[97044]: pgmap v1613: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:27 smithi110 ceph-mon[97633]: pgmap v1613: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:29 smithi104 ceph-mon[97044]: pgmap v1614: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:29 smithi110 ceph-mon[97633]: pgmap v1614: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:31 smithi104 ceph-mon[97044]: pgmap v1615: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:31 smithi110 ceph-mon[97633]: pgmap v1615: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:33 smithi104 ceph-mon[97044]: pgmap v1616: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:33 smithi110 ceph-mon[97633]: pgmap v1616: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:35 smithi104 ceph-mon[97044]: pgmap v1617: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:35 smithi110 ceph-mon[97633]: pgmap v1617: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:37 smithi104 ceph-mon[97044]: pgmap v1618: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:37 smithi110 ceph-mon[97633]: pgmap v1618: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:39 smithi104 ceph-mon[97044]: pgmap v1619: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:39 smithi110 ceph-mon[97633]: pgmap v1619: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:19:40.580+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:19:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:41 smithi104 ceph-mon[97044]: pgmap v1620: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:41 smithi110 ceph-mon[97633]: pgmap v1620: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:43 smithi104 ceph-mon[97044]: pgmap v1621: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:19:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:19:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:19:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:19:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:43 smithi110 ceph-mon[97633]: pgmap v1621: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:19:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:19:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:19:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:19:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:45 smithi104 ceph-mon[97044]: pgmap v1622: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:45 smithi110 ceph-mon[97633]: pgmap v1622: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:47 smithi104 ceph-mon[97044]: pgmap v1623: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:47 smithi110 ceph-mon[97633]: pgmap v1623: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:49 smithi104 ceph-mon[97044]: pgmap v1624: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:49 smithi110 ceph-mon[97633]: pgmap v1624: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:51 smithi104 ceph-mon[97044]: pgmap v1625: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:51 smithi110 ceph-mon[97633]: pgmap v1625: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:53 smithi104 ceph-mon[97044]: pgmap v1626: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:53 smithi110 ceph-mon[97633]: pgmap v1626: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:55 smithi104 ceph-mon[97044]: pgmap v1627: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:19:55.581+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:19:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:55 smithi110 ceph-mon[97633]: pgmap v1627: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:19:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:57 smithi104 ceph-mon[97044]: pgmap v1628: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:19:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:57 smithi110 ceph-mon[97633]: pgmap v1628: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:19:59 smithi104 ceph-mon[97044]: pgmap v1629: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:19:59 smithi110 ceph-mon[97633]: pgmap v1629: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:20:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:20:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:01 smithi104 ceph-mon[97044]: pgmap v1630: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:01 smithi110 ceph-mon[97633]: pgmap v1630: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:03 smithi104 ceph-mon[97044]: pgmap v1631: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:03 smithi110 ceph-mon[97633]: pgmap v1631: 129 pgs: 129 active+clean; 7.8 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:05 smithi104 ceph-mon[97044]: pgmap v1632: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:20:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:05 smithi110 ceph-mon[97633]: pgmap v1632: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:20:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:07 smithi110 ceph-mon[97633]: pgmap v1633: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:20:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:07 smithi104 ceph-mon[97044]: pgmap v1633: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:20:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:08 smithi110 ceph-mon[97633]: osdmap e185: 12 total, 12 up, 12 in 2022-09-10T05:20:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:08 smithi104 ceph-mon[97044]: osdmap e185: 12 total, 12 up, 12 in 2022-09-10T05:20:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:09 smithi110 ceph-mon[97633]: pgmap v1635: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:20:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:09 smithi104 ceph-mon[97044]: pgmap v1635: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:20:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:20:10.595+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:20:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:11 smithi110 ceph-mon[97633]: pgmap v1636: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:20:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:11 smithi104 ceph-mon[97044]: pgmap v1636: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:20:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:13 smithi104 ceph-mon[97044]: pgmap v1637: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:20:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:13 smithi104 ceph-mon[97044]: osdmap e186: 12 total, 12 up, 12 in 2022-09-10T05:20:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:13 smithi110 ceph-mon[97633]: pgmap v1637: 129 pgs: 129 active+clean; 7.8 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:20:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:13 smithi110 ceph-mon[97633]: osdmap e186: 12 total, 12 up, 12 in 2022-09-10T05:20:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:15 smithi104 ceph-mon[97044]: pgmap v1639: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:20:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:15 smithi110 ceph-mon[97633]: pgmap v1639: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:20:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:17 smithi104 ceph-mon[97044]: pgmap v1640: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:20:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:17 smithi110 ceph-mon[97633]: pgmap v1640: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T05:20:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:19 smithi104 ceph-mon[97044]: pgmap v1641: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:20:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:19 smithi110 ceph-mon[97633]: pgmap v1641: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:20:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:21 smithi104 ceph-mon[97044]: pgmap v1642: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:20:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:21 smithi110 ceph-mon[97633]: pgmap v1642: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:20:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:23 smithi104 ceph-mon[97044]: pgmap v1643: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:20:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:23 smithi110 ceph-mon[97633]: pgmap v1643: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:20:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:25 smithi104 ceph-mon[97044]: pgmap v1644: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:20:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:20:25.599+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:20:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:25 smithi110 ceph-mon[97633]: pgmap v1644: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:20:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:27 smithi104 ceph-mon[97044]: pgmap v1645: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:20:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:27 smithi110 ceph-mon[97633]: pgmap v1645: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:20:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:29 smithi104 ceph-mon[97044]: pgmap v1646: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:20:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:29 smithi110 ceph-mon[97633]: pgmap v1646: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:20:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:31 smithi104 ceph-mon[97044]: pgmap v1647: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:20:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:31 smithi110 ceph-mon[97633]: pgmap v1647: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:20:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:33 smithi104 ceph-mon[97044]: pgmap v1648: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:20:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:33 smithi110 ceph-mon[97633]: pgmap v1648: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:20:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:35 smithi104 ceph-mon[97044]: pgmap v1649: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:20:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:35 smithi110 ceph-mon[97633]: pgmap v1649: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:20:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:37 smithi104 ceph-mon[97044]: pgmap v1650: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:38.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:37 smithi110 ceph-mon[97633]: pgmap v1650: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:39 smithi104 ceph-mon[97044]: pgmap v1651: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:39 smithi110 ceph-mon[97633]: pgmap v1651: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:20:40.613+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:20:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:41 smithi104 ceph-mon[97044]: pgmap v1652: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:41 smithi110 ceph-mon[97633]: pgmap v1652: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:43 smithi104 ceph-mon[97044]: pgmap v1653: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:20:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:20:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:20:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:20:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:43 smithi110 ceph-mon[97633]: pgmap v1653: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:20:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:20:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:20:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:20:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:45 smithi104 ceph-mon[97044]: pgmap v1654: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:45 smithi110 ceph-mon[97633]: pgmap v1654: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:47 smithi104 ceph-mon[97044]: pgmap v1655: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:47 smithi110 ceph-mon[97633]: pgmap v1655: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:49 smithi104 ceph-mon[97044]: pgmap v1656: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:49 smithi110 ceph-mon[97633]: pgmap v1656: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:51 smithi104 ceph-mon[97044]: pgmap v1657: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:51 smithi110 ceph-mon[97633]: pgmap v1657: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:53 smithi104 ceph-mon[97044]: pgmap v1658: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:53 smithi110 ceph-mon[97633]: pgmap v1658: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:55 smithi104 ceph-mon[97044]: pgmap v1659: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:20:55.628+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:20:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:55 smithi110 ceph-mon[97633]: pgmap v1659: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:20:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:57 smithi104 ceph-mon[97044]: pgmap v1660: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:20:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:57 smithi110 ceph-mon[97633]: pgmap v1660: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:20:59 smithi104 ceph-mon[97044]: pgmap v1661: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:20:59 smithi110 ceph-mon[97633]: pgmap v1661: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:01 smithi104 ceph-mon[97044]: pgmap v1662: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:01 smithi110 ceph-mon[97633]: pgmap v1662: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:03 smithi110 ceph-mon[97633]: pgmap v1663: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:03 smithi104 ceph-mon[97044]: pgmap v1663: 129 pgs: 129 active+clean; 7.8 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:05 smithi110 ceph-mon[97633]: pgmap v1664: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:21:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:05 smithi104 ceph-mon[97044]: pgmap v1664: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:21:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:07 smithi110 ceph-mon[97633]: pgmap v1665: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:21:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:07 smithi104 ceph-mon[97044]: pgmap v1665: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:08 smithi110 ceph-mon[97633]: osdmap e187: 12 total, 12 up, 12 in 2022-09-10T05:21:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:08 smithi104 ceph-mon[97044]: osdmap e187: 12 total, 12 up, 12 in 2022-09-10T05:21:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:09 smithi110 ceph-mon[97633]: pgmap v1667: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:21:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:09 smithi104 ceph-mon[97044]: pgmap v1667: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:21:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:21:10.642+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:21:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:11 smithi110 ceph-mon[97633]: pgmap v1668: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:21:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:11 smithi104 ceph-mon[97044]: pgmap v1668: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:21:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:13 smithi104 ceph-mon[97044]: pgmap v1669: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:21:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:13 smithi104 ceph-mon[97044]: osdmap e188: 12 total, 12 up, 12 in 2022-09-10T05:21:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:13 smithi110 ceph-mon[97633]: pgmap v1669: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:21:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:13 smithi110 ceph-mon[97633]: osdmap e188: 12 total, 12 up, 12 in 2022-09-10T05:21:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:15 smithi104 ceph-mon[97044]: pgmap v1671: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:21:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:15 smithi110 ceph-mon[97633]: pgmap v1671: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:21:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:17 smithi104 ceph-mon[97044]: pgmap v1672: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:21:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:17 smithi110 ceph-mon[97633]: pgmap v1672: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:21:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:19 smithi104 ceph-mon[97044]: pgmap v1673: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:19 smithi110 ceph-mon[97633]: pgmap v1673: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:21:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:21 smithi104 ceph-mon[97044]: pgmap v1674: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:21:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:21 smithi110 ceph-mon[97633]: pgmap v1674: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:21:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:23 smithi104 ceph-mon[97044]: pgmap v1675: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:21:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:23 smithi110 ceph-mon[97633]: pgmap v1675: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:21:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:25 smithi104 ceph-mon[97044]: pgmap v1676: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:21:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:21:25.654+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:21:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:25 smithi110 ceph-mon[97633]: pgmap v1676: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:21:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:27 smithi104 ceph-mon[97044]: pgmap v1677: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:27 smithi110 ceph-mon[97633]: pgmap v1677: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:29 smithi104 ceph-mon[97044]: pgmap v1678: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:29 smithi110 ceph-mon[97633]: pgmap v1678: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:31 smithi104 ceph-mon[97044]: pgmap v1679: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:31 smithi110 ceph-mon[97633]: pgmap v1679: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:33 smithi104 ceph-mon[97044]: pgmap v1680: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:33 smithi110 ceph-mon[97633]: pgmap v1680: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:35 smithi104 ceph-mon[97044]: pgmap v1681: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:35 smithi110 ceph-mon[97633]: pgmap v1681: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:37 smithi104 ceph-mon[97044]: pgmap v1682: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:37 smithi110 ceph-mon[97633]: pgmap v1682: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:39 smithi104 ceph-mon[97044]: pgmap v1683: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:39 smithi110 ceph-mon[97633]: pgmap v1683: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:21:40.660+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:21:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:41 smithi104 ceph-mon[97044]: pgmap v1684: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:41 smithi110 ceph-mon[97633]: pgmap v1684: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:43 smithi104 ceph-mon[97044]: pgmap v1685: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:21:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:21:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:21:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:21:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:43 smithi110 ceph-mon[97633]: pgmap v1685: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:21:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:21:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:21:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:21:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:45 smithi104 ceph-mon[97044]: pgmap v1686: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:45 smithi110 ceph-mon[97633]: pgmap v1686: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:47 smithi104 ceph-mon[97044]: pgmap v1687: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:47 smithi110 ceph-mon[97633]: pgmap v1687: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:49 smithi104 ceph-mon[97044]: pgmap v1688: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:49 smithi110 ceph-mon[97633]: pgmap v1688: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:51 smithi110 ceph-mon[97633]: pgmap v1689: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:51 smithi104 ceph-mon[97044]: pgmap v1689: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:53 smithi110 ceph-mon[97633]: pgmap v1690: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:53 smithi104 ceph-mon[97044]: pgmap v1690: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:21:55.667+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:21:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:55 smithi110 ceph-mon[97633]: pgmap v1691: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:56.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:55 smithi104 ceph-mon[97044]: pgmap v1691: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:21:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:57 smithi110 ceph-mon[97633]: pgmap v1692: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:21:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:57 smithi104 ceph-mon[97044]: pgmap v1692: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:21:59 smithi110 ceph-mon[97633]: pgmap v1693: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:21:59 smithi104 ceph-mon[97044]: pgmap v1693: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:01 smithi110 ceph-mon[97633]: pgmap v1694: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:01 smithi104 ceph-mon[97044]: pgmap v1694: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:03 smithi110 ceph-mon[97633]: pgmap v1695: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:03 smithi104 ceph-mon[97044]: pgmap v1695: 129 pgs: 129 active+clean; 7.8 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:05 smithi110 ceph-mon[97633]: pgmap v1696: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:22:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:05 smithi104 ceph-mon[97044]: pgmap v1696: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:22:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:07 smithi110 ceph-mon[97633]: pgmap v1697: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:22:08.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:07 smithi104 ceph-mon[97044]: pgmap v1697: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:22:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:08 smithi110 ceph-mon[97633]: osdmap e189: 12 total, 12 up, 12 in 2022-09-10T05:22:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:08 smithi104 ceph-mon[97044]: osdmap e189: 12 total, 12 up, 12 in 2022-09-10T05:22:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:09 smithi110 ceph-mon[97633]: pgmap v1699: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:22:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:09 smithi104 ceph-mon[97044]: pgmap v1699: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:22:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:22:10.683+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:22:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:11 smithi110 ceph-mon[97633]: pgmap v1700: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:22:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:11 smithi104 ceph-mon[97044]: pgmap v1700: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:22:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:13 smithi104 ceph-mon[97044]: pgmap v1701: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:22:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:13 smithi104 ceph-mon[97044]: osdmap e190: 12 total, 12 up, 12 in 2022-09-10T05:22:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:13 smithi110 ceph-mon[97633]: pgmap v1701: 129 pgs: 129 active+clean; 7.8 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:22:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:13 smithi110 ceph-mon[97633]: osdmap e190: 12 total, 12 up, 12 in 2022-09-10T05:22:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:15 smithi104 ceph-mon[97044]: pgmap v1703: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:22:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:15 smithi110 ceph-mon[97633]: pgmap v1703: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:17 smithi104 ceph-mon[97044]: pgmap v1704: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:22:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:17 smithi110 ceph-mon[97633]: pgmap v1704: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:22:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:19 smithi104 ceph-mon[97044]: pgmap v1705: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:22:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:19 smithi110 ceph-mon[97633]: pgmap v1705: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:22:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:21 smithi104 ceph-mon[97044]: pgmap v1706: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:22:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:21 smithi110 ceph-mon[97633]: pgmap v1706: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:22:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:23 smithi104 ceph-mon[97044]: pgmap v1707: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:22:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:23 smithi110 ceph-mon[97633]: pgmap v1707: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:22:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:25 smithi104 ceph-mon[97044]: pgmap v1708: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:22:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:22:25.693+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:22:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:25 smithi110 ceph-mon[97633]: pgmap v1708: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:22:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:27 smithi104 ceph-mon[97044]: pgmap v1709: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:27 smithi110 ceph-mon[97633]: pgmap v1709: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:29 smithi104 ceph-mon[97044]: pgmap v1710: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:29 smithi110 ceph-mon[97633]: pgmap v1710: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:31 smithi104 ceph-mon[97044]: pgmap v1711: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:31 smithi110 ceph-mon[97633]: pgmap v1711: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:33 smithi104 ceph-mon[97044]: pgmap v1712: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:33 smithi110 ceph-mon[97633]: pgmap v1712: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:35 smithi104 ceph-mon[97044]: pgmap v1713: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:35 smithi110 ceph-mon[97633]: pgmap v1713: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:37 smithi104 ceph-mon[97044]: pgmap v1714: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:37 smithi110 ceph-mon[97633]: pgmap v1714: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:39 smithi104 ceph-mon[97044]: pgmap v1715: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:39 smithi110 ceph-mon[97633]: pgmap v1715: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:22:40.696+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:22:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:41 smithi110 ceph-mon[97633]: pgmap v1716: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:41 smithi104 ceph-mon[97044]: pgmap v1716: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:43 smithi110 ceph-mon[97633]: pgmap v1717: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:22:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:22:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:22:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:22:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:43 smithi104 ceph-mon[97044]: pgmap v1717: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:22:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:22:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:22:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:22:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:45 smithi110 ceph-mon[97633]: pgmap v1718: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:45 smithi104 ceph-mon[97044]: pgmap v1718: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:47 smithi110 ceph-mon[97633]: pgmap v1719: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:47 smithi104 ceph-mon[97044]: pgmap v1719: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:49 smithi110 ceph-mon[97633]: pgmap v1720: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:49 smithi104 ceph-mon[97044]: pgmap v1720: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:22:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:51 smithi110 ceph-mon[97633]: pgmap v1721: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:51 smithi104 ceph-mon[97044]: pgmap v1721: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:53 smithi110 ceph-mon[97633]: pgmap v1722: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:22:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:53 smithi104 ceph-mon[97044]: pgmap v1722: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:22:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:22:55.706+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:22:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:55 smithi110 ceph-mon[97633]: pgmap v1723: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:55 smithi104 ceph-mon[97044]: pgmap v1723: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:22:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:57 smithi110 ceph-mon[97633]: pgmap v1724: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:22:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:57 smithi104 ceph-mon[97044]: pgmap v1724: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:23:00.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:22:59 smithi110 ceph-mon[97633]: pgmap v1725: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:23:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:22:59 smithi104 ceph-mon[97044]: pgmap v1725: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:23:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:01 smithi110 ceph-mon[97633]: pgmap v1726: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:01 smithi104 ceph-mon[97044]: pgmap v1726: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:03 smithi110 ceph-mon[97633]: pgmap v1727: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:03 smithi104 ceph-mon[97044]: pgmap v1727: 129 pgs: 129 active+clean; 7.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:05 smithi110 ceph-mon[97633]: pgmap v1728: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:23:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:05 smithi104 ceph-mon[97044]: pgmap v1728: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:23:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:07 smithi110 ceph-mon[97633]: pgmap v1729: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:23:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:07 smithi104 ceph-mon[97044]: pgmap v1729: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:23:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:08 smithi110 ceph-mon[97633]: osdmap e191: 12 total, 12 up, 12 in 2022-09-10T05:23:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:08 smithi104 ceph-mon[97044]: osdmap e191: 12 total, 12 up, 12 in 2022-09-10T05:23:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:09 smithi110 ceph-mon[97633]: pgmap v1731: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:23:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:09 smithi104 ceph-mon[97044]: pgmap v1731: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:23:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:23:10.720+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:23:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:11 smithi110 ceph-mon[97633]: pgmap v1732: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:23:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:11 smithi104 ceph-mon[97044]: pgmap v1732: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:23:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:13 smithi104 ceph-mon[97044]: pgmap v1733: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:23:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:13 smithi104 ceph-mon[97044]: osdmap e192: 12 total, 12 up, 12 in 2022-09-10T05:23:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:13 smithi110 ceph-mon[97633]: pgmap v1733: 129 pgs: 129 active+clean; 7.8 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:23:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:13 smithi110 ceph-mon[97633]: osdmap e192: 12 total, 12 up, 12 in 2022-09-10T05:23:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:15 smithi104 ceph-mon[97044]: pgmap v1735: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:23:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:15 smithi110 ceph-mon[97633]: pgmap v1735: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:23:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:17 smithi104 ceph-mon[97044]: pgmap v1736: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:23:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:17 smithi110 ceph-mon[97633]: pgmap v1736: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:23:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:19 smithi104 ceph-mon[97044]: pgmap v1737: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:23:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:19 smithi110 ceph-mon[97633]: pgmap v1737: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:23:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:21 smithi104 ceph-mon[97044]: pgmap v1738: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:23:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:21 smithi110 ceph-mon[97633]: pgmap v1738: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:23:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:23 smithi104 ceph-mon[97044]: pgmap v1739: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:23:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:23 smithi110 ceph-mon[97633]: pgmap v1739: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:23:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:25 smithi104 ceph-mon[97044]: pgmap v1740: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:23:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:23:25.722+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:23:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:25 smithi110 ceph-mon[97633]: pgmap v1740: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:23:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:27 smithi104 ceph-mon[97044]: pgmap v1741: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:28.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:27 smithi110 ceph-mon[97633]: pgmap v1741: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:29 smithi110 ceph-mon[97633]: pgmap v1742: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:29 smithi104 ceph-mon[97044]: pgmap v1742: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:31 smithi110 ceph-mon[97633]: pgmap v1743: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:31 smithi104 ceph-mon[97044]: pgmap v1743: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:33 smithi110 ceph-mon[97633]: pgmap v1744: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:34.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:33 smithi104 ceph-mon[97044]: pgmap v1744: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:35 smithi110 ceph-mon[97633]: pgmap v1745: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:35 smithi104 ceph-mon[97044]: pgmap v1745: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:37 smithi110 ceph-mon[97633]: pgmap v1746: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:37 smithi104 ceph-mon[97044]: pgmap v1746: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:39 smithi110 ceph-mon[97633]: pgmap v1747: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:39 smithi104 ceph-mon[97044]: pgmap v1747: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:23:40.730+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:23:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:41 smithi110 ceph-mon[97633]: pgmap v1748: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:42.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:41 smithi104 ceph-mon[97044]: pgmap v1748: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:43 smithi110 ceph-mon[97633]: pgmap v1749: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:23:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:23:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:23:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:23:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:43 smithi104 ceph-mon[97044]: pgmap v1749: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:23:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:23:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:23:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:23:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:45 smithi110 ceph-mon[97633]: pgmap v1750: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:45 smithi104 ceph-mon[97044]: pgmap v1750: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:46.805 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:23:46.804+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T05:23:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:47 smithi110 ceph-mon[97633]: pgmap v1751: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:47 smithi104 ceph-mon[97044]: pgmap v1751: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:49 smithi110 ceph-mon[97633]: pgmap v1752: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:49 smithi104 ceph-mon[97044]: pgmap v1752: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:51 smithi110 ceph-mon[97633]: pgmap v1753: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:51 smithi104 ceph-mon[97044]: pgmap v1753: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:54.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:53 smithi110 ceph-mon[97633]: pgmap v1754: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:53 smithi104 ceph-mon[97044]: pgmap v1754: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:23:55.735+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:23:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:55 smithi110 ceph-mon[97633]: pgmap v1755: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:55 smithi104 ceph-mon[97044]: pgmap v1755: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:23:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:57 smithi110 ceph-mon[97633]: pgmap v1756: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:23:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:57 smithi104 ceph-mon[97044]: pgmap v1756: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:23:59 smithi110 ceph-mon[97633]: pgmap v1757: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:23:59 smithi104 ceph-mon[97044]: pgmap v1757: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:01 smithi110 ceph-mon[97633]: pgmap v1758: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:01 smithi104 ceph-mon[97044]: pgmap v1758: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:03 smithi110 ceph-mon[97633]: pgmap v1759: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:03 smithi104 ceph-mon[97044]: pgmap v1759: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:05 smithi110 ceph-mon[97633]: pgmap v1760: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:24:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:05 smithi104 ceph-mon[97044]: pgmap v1760: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:24:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:07 smithi110 ceph-mon[97633]: pgmap v1761: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:24:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:07 smithi104 ceph-mon[97044]: pgmap v1761: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:08 smithi110 ceph-mon[97633]: osdmap e193: 12 total, 12 up, 12 in 2022-09-10T05:24:09.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:08 smithi104 ceph-mon[97044]: osdmap e193: 12 total, 12 up, 12 in 2022-09-10T05:24:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:09 smithi110 ceph-mon[97633]: pgmap v1763: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:24:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:09 smithi104 ceph-mon[97044]: pgmap v1763: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:24:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:24:10.747+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:24:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:11 smithi110 ceph-mon[97633]: pgmap v1764: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:24:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:11 smithi104 ceph-mon[97044]: pgmap v1764: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:24:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:13 smithi104 ceph-mon[97044]: pgmap v1765: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:24:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:13 smithi104 ceph-mon[97044]: osdmap e194: 12 total, 12 up, 12 in 2022-09-10T05:24:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:13 smithi110 ceph-mon[97633]: pgmap v1765: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:24:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:13 smithi110 ceph-mon[97633]: osdmap e194: 12 total, 12 up, 12 in 2022-09-10T05:24:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:15 smithi104 ceph-mon[97044]: pgmap v1767: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:24:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:15 smithi110 ceph-mon[97633]: pgmap v1767: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:24:18.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:17 smithi104 ceph-mon[97044]: pgmap v1768: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:24:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:17 smithi110 ceph-mon[97633]: pgmap v1768: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:24:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:19 smithi110 ceph-mon[97633]: pgmap v1769: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:24:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:19 smithi104 ceph-mon[97044]: pgmap v1769: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:24:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:21 smithi110 ceph-mon[97633]: pgmap v1770: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:24:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:21 smithi104 ceph-mon[97044]: pgmap v1770: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:24:24.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:23 smithi110 ceph-mon[97633]: pgmap v1771: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:24:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:23 smithi104 ceph-mon[97044]: pgmap v1771: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:24:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:24:25.758+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:24:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:25 smithi110 ceph-mon[97633]: pgmap v1772: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:24:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:25 smithi104 ceph-mon[97044]: pgmap v1772: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:24:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:27 smithi110 ceph-mon[97633]: pgmap v1773: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:27 smithi104 ceph-mon[97044]: pgmap v1773: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:29 smithi110 ceph-mon[97633]: pgmap v1774: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:29 smithi104 ceph-mon[97044]: pgmap v1774: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:31 smithi110 ceph-mon[97633]: pgmap v1775: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:31 smithi104 ceph-mon[97044]: pgmap v1775: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:33 smithi110 ceph-mon[97633]: pgmap v1776: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:33 smithi104 ceph-mon[97044]: pgmap v1776: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:35 smithi110 ceph-mon[97633]: pgmap v1777: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:35 smithi104 ceph-mon[97044]: pgmap v1777: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:37 smithi110 ceph-mon[97633]: pgmap v1778: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:37 smithi104 ceph-mon[97044]: pgmap v1778: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:39 smithi110 ceph-mon[97633]: pgmap v1779: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:39 smithi104 ceph-mon[97044]: pgmap v1779: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:24:40.764+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:24:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:41 smithi110 ceph-mon[97633]: pgmap v1780: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:41 smithi104 ceph-mon[97044]: pgmap v1780: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:43 smithi110 ceph-mon[97633]: pgmap v1781: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:24:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:24:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:24:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:24:44.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:43 smithi104 ceph-mon[97044]: pgmap v1781: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:44.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:24:44.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:24:44.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:24:44.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:24:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:45 smithi110 ceph-mon[97633]: pgmap v1782: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:45 smithi104 ceph-mon[97044]: pgmap v1782: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:47 smithi110 ceph-mon[97633]: pgmap v1783: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:47 smithi104 ceph-mon[97044]: pgmap v1783: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:49 smithi110 ceph-mon[97633]: pgmap v1784: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:49 smithi104 ceph-mon[97044]: pgmap v1784: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:24:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:51 smithi110 ceph-mon[97633]: pgmap v1785: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:51 smithi104 ceph-mon[97044]: pgmap v1785: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:53 smithi110 ceph-mon[97633]: pgmap v1786: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:53 smithi104 ceph-mon[97044]: pgmap v1786: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:24:55.771+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:24:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:55 smithi110 ceph-mon[97633]: pgmap v1787: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:55 smithi104 ceph-mon[97044]: pgmap v1787: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:57 smithi110 ceph-mon[97633]: pgmap v1788: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:24:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:57 smithi104 ceph-mon[97044]: pgmap v1788: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:25:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:24:59 smithi110 ceph-mon[97633]: pgmap v1789: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:25:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:24:59 smithi104 ceph-mon[97044]: pgmap v1789: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:25:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:01 smithi110 ceph-mon[97633]: pgmap v1790: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:01 smithi104 ceph-mon[97044]: pgmap v1790: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:03 smithi110 ceph-mon[97633]: pgmap v1791: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:03 smithi104 ceph-mon[97044]: pgmap v1791: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:05 smithi110 ceph-mon[97633]: pgmap v1792: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:25:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:05 smithi104 ceph-mon[97044]: pgmap v1792: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:25:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:07 smithi110 ceph-mon[97633]: pgmap v1793: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:25:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:07 smithi104 ceph-mon[97044]: pgmap v1793: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:08 smithi110 ceph-mon[97633]: osdmap e195: 12 total, 12 up, 12 in 2022-09-10T05:25:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:08 smithi104 ceph-mon[97044]: osdmap e195: 12 total, 12 up, 12 in 2022-09-10T05:25:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:09 smithi110 ceph-mon[97633]: pgmap v1795: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:25:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:09 smithi104 ceph-mon[97044]: pgmap v1795: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:25:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:25:10.787+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:25:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:11 smithi110 ceph-mon[97633]: pgmap v1796: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:25:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:11 smithi104 ceph-mon[97044]: pgmap v1796: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:25:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:13 smithi110 ceph-mon[97633]: pgmap v1797: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:25:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:13 smithi110 ceph-mon[97633]: osdmap e196: 12 total, 12 up, 12 in 2022-09-10T05:25:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:13 smithi104 ceph-mon[97044]: pgmap v1797: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:25:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:13 smithi104 ceph-mon[97044]: osdmap e196: 12 total, 12 up, 12 in 2022-09-10T05:25:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:15 smithi110 ceph-mon[97633]: pgmap v1799: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:25:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:15 smithi104 ceph-mon[97044]: pgmap v1799: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:25:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:17 smithi110 ceph-mon[97633]: pgmap v1800: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:25:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:17 smithi104 ceph-mon[97044]: pgmap v1800: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:25:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:19 smithi110 ceph-mon[97633]: pgmap v1801: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:25:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:19 smithi104 ceph-mon[97044]: pgmap v1801: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:25:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:21 smithi110 ceph-mon[97633]: pgmap v1802: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:25:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:21 smithi104 ceph-mon[97044]: pgmap v1802: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:25:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:23 smithi110 ceph-mon[97633]: pgmap v1803: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:25:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:23 smithi104 ceph-mon[97044]: pgmap v1803: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:25:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:25:25.795+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:25:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:25 smithi110 ceph-mon[97633]: pgmap v1804: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:25:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:25 smithi104 ceph-mon[97044]: pgmap v1804: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:25:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:27 smithi110 ceph-mon[97633]: pgmap v1805: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:27 smithi104 ceph-mon[97044]: pgmap v1805: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:29 smithi110 ceph-mon[97633]: pgmap v1806: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:29 smithi104 ceph-mon[97044]: pgmap v1806: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:31 smithi110 ceph-mon[97633]: pgmap v1807: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:32.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:31 smithi104 ceph-mon[97044]: pgmap v1807: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:33 smithi110 ceph-mon[97633]: pgmap v1808: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:33 smithi104 ceph-mon[97044]: pgmap v1808: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:35 smithi110 ceph-mon[97633]: pgmap v1809: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:35 smithi104 ceph-mon[97044]: pgmap v1809: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:37 smithi110 ceph-mon[97633]: pgmap v1810: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:37 smithi104 ceph-mon[97044]: pgmap v1810: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:40.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:39 smithi110 ceph-mon[97633]: pgmap v1811: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:40.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:39 smithi104 ceph-mon[97044]: pgmap v1811: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:25:40.810+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:25:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:41 smithi110 ceph-mon[97633]: pgmap v1812: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:41 smithi104 ceph-mon[97044]: pgmap v1812: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:43 smithi110 ceph-mon[97633]: pgmap v1813: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:25:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:25:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:25:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:25:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:43 smithi104 ceph-mon[97044]: pgmap v1813: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:25:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:25:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:45 smithi110 ceph-mon[97633]: pgmap v1814: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:45 smithi104 ceph-mon[97044]: pgmap v1814: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:47 smithi110 ceph-mon[97633]: pgmap v1815: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:47 smithi104 ceph-mon[97044]: pgmap v1815: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:49 smithi110 ceph-mon[97633]: pgmap v1816: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:49 smithi104 ceph-mon[97044]: pgmap v1816: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:51 smithi110 ceph-mon[97633]: pgmap v1817: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:51 smithi104 ceph-mon[97044]: pgmap v1817: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:53 smithi110 ceph-mon[97633]: pgmap v1818: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:53 smithi104 ceph-mon[97044]: pgmap v1818: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:55 smithi110 ceph-mon[97633]: pgmap v1819: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:55 smithi104 ceph-mon[97044]: pgmap v1819: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:25:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:25:55.819+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:25:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:57 smithi110 ceph-mon[97633]: pgmap v1820: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:25:58.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:57 smithi104 ceph-mon[97044]: pgmap v1820: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:00.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:25:59 smithi110 ceph-mon[97633]: pgmap v1821: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:25:59 smithi104 ceph-mon[97044]: pgmap v1821: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:01 smithi110 ceph-mon[97633]: pgmap v1822: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:01 smithi104 ceph-mon[97044]: pgmap v1822: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:03 smithi110 ceph-mon[97633]: pgmap v1823: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:03 smithi104 ceph-mon[97044]: pgmap v1823: 129 pgs: 129 active+clean; 7.8 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:05 smithi110 ceph-mon[97633]: pgmap v1824: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:26:06.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:05 smithi104 ceph-mon[97044]: pgmap v1824: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:26:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:07 smithi110 ceph-mon[97633]: pgmap v1825: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:26:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:07 smithi104 ceph-mon[97044]: pgmap v1825: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:08 smithi110 ceph-mon[97633]: osdmap e197: 12 total, 12 up, 12 in 2022-09-10T05:26:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:08 smithi104 ceph-mon[97044]: osdmap e197: 12 total, 12 up, 12 in 2022-09-10T05:26:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:09 smithi110 ceph-mon[97633]: pgmap v1827: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:26:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:09 smithi104 ceph-mon[97044]: pgmap v1827: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:26:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:26:10.823+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:26:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:11 smithi110 ceph-mon[97633]: pgmap v1828: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:26:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:11 smithi104 ceph-mon[97044]: pgmap v1828: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:26:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:13 smithi110 ceph-mon[97633]: pgmap v1829: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:26:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:13 smithi110 ceph-mon[97633]: osdmap e198: 12 total, 12 up, 12 in 2022-09-10T05:26:14.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:13 smithi104 ceph-mon[97044]: pgmap v1829: 129 pgs: 129 active+clean; 7.8 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:26:14.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:13 smithi104 ceph-mon[97044]: osdmap e198: 12 total, 12 up, 12 in 2022-09-10T05:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:15 smithi110 ceph-mon[97633]: pgmap v1831: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:26:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:15 smithi104 ceph-mon[97044]: pgmap v1831: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:26:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:17 smithi110 ceph-mon[97633]: pgmap v1832: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:26:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:17 smithi104 ceph-mon[97044]: pgmap v1832: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:26:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:19 smithi110 ceph-mon[97633]: pgmap v1833: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:26:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:19 smithi104 ceph-mon[97044]: pgmap v1833: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:26:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:21 smithi110 ceph-mon[97633]: pgmap v1834: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:26:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:21 smithi104 ceph-mon[97044]: pgmap v1834: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:26:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:23 smithi110 ceph-mon[97633]: pgmap v1835: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:26:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:23 smithi104 ceph-mon[97044]: pgmap v1835: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:26:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:25 smithi110 ceph-mon[97633]: pgmap v1836: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:26:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:25 smithi104 ceph-mon[97044]: pgmap v1836: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:26:26.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:26:25.836+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:26:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:27 smithi110 ceph-mon[97633]: pgmap v1837: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:27 smithi104 ceph-mon[97044]: pgmap v1837: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:29 smithi110 ceph-mon[97633]: pgmap v1838: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:29 smithi104 ceph-mon[97044]: pgmap v1838: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:31 smithi110 ceph-mon[97633]: pgmap v1839: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:31 smithi104 ceph-mon[97044]: pgmap v1839: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:33 smithi110 ceph-mon[97633]: pgmap v1840: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:33 smithi104 ceph-mon[97044]: pgmap v1840: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:36.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:35 smithi110 ceph-mon[97633]: pgmap v1841: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:35 smithi104 ceph-mon[97044]: pgmap v1841: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:37 smithi110 ceph-mon[97633]: pgmap v1842: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:37 smithi104 ceph-mon[97044]: pgmap v1842: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:40.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:39 smithi110 ceph-mon[97633]: pgmap v1843: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:40.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:39 smithi104 ceph-mon[97044]: pgmap v1843: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:26:40.844+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:26:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:41 smithi110 ceph-mon[97633]: pgmap v1844: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:41 smithi104 ceph-mon[97044]: pgmap v1844: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:43 smithi110 ceph-mon[97633]: pgmap v1845: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:26:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:26:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:26:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:26:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:43 smithi104 ceph-mon[97044]: pgmap v1845: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:26:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:26:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:26:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:26:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:45 smithi110 ceph-mon[97633]: pgmap v1846: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:45 smithi104 ceph-mon[97044]: pgmap v1846: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:47 smithi110 ceph-mon[97633]: pgmap v1847: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:47 smithi104 ceph-mon[97044]: pgmap v1847: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:49 smithi110 ceph-mon[97633]: pgmap v1848: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:49 smithi104 ceph-mon[97044]: pgmap v1848: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:51 smithi110 ceph-mon[97633]: pgmap v1849: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:51 smithi104 ceph-mon[97044]: pgmap v1849: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:53 smithi110 ceph-mon[97633]: pgmap v1850: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:53 smithi104 ceph-mon[97044]: pgmap v1850: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:55 smithi110 ceph-mon[97633]: pgmap v1851: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:55 smithi104 ceph-mon[97044]: pgmap v1851: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:26:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:26:55.852+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:26:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:57 smithi110 ceph-mon[97633]: pgmap v1852: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:26:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:57 smithi104 ceph-mon[97044]: pgmap v1852: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:26:59 smithi110 ceph-mon[97633]: pgmap v1853: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:26:59 smithi104 ceph-mon[97044]: pgmap v1853: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:01 smithi110 ceph-mon[97633]: pgmap v1854: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:01 smithi104 ceph-mon[97044]: pgmap v1854: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:03 smithi110 ceph-mon[97633]: pgmap v1855: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:03 smithi104 ceph-mon[97044]: pgmap v1855: 129 pgs: 129 active+clean; 7.8 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:05 smithi110 ceph-mon[97633]: pgmap v1856: 129 pgs: 129 active+clean; 7.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:27:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:05 smithi104 ceph-mon[97044]: pgmap v1856: 129 pgs: 129 active+clean; 7.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:27:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:07 smithi110 ceph-mon[97633]: pgmap v1857: 129 pgs: 129 active+clean; 7.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:27:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:07 smithi104 ceph-mon[97044]: pgmap v1857: 129 pgs: 129 active+clean; 7.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:27:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:08 smithi104 ceph-mon[97044]: osdmap e199: 12 total, 12 up, 12 in 2022-09-10T05:27:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:08 smithi110 ceph-mon[97633]: osdmap e199: 12 total, 12 up, 12 in 2022-09-10T05:27:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:09 smithi104 ceph-mon[97044]: pgmap v1859: 129 pgs: 129 active+clean; 7.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:27:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:09 smithi110 ceph-mon[97633]: pgmap v1859: 129 pgs: 129 active+clean; 7.8 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:27:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:27:10.868+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:27:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:11 smithi104 ceph-mon[97044]: pgmap v1860: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:27:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:11 smithi110 ceph-mon[97633]: pgmap v1860: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:27:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:13 smithi110 ceph-mon[97633]: pgmap v1861: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:27:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:13 smithi110 ceph-mon[97633]: osdmap e200: 12 total, 12 up, 12 in 2022-09-10T05:27:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:13 smithi104 ceph-mon[97044]: pgmap v1861: 129 pgs: 129 active+clean; 7.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:27:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:13 smithi104 ceph-mon[97044]: osdmap e200: 12 total, 12 up, 12 in 2022-09-10T05:27:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:15 smithi110 ceph-mon[97633]: pgmap v1863: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:27:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:15 smithi104 ceph-mon[97044]: pgmap v1863: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:27:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:17 smithi110 ceph-mon[97633]: pgmap v1864: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:27:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:17 smithi104 ceph-mon[97044]: pgmap v1864: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:27:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:19 smithi110 ceph-mon[97633]: pgmap v1865: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:27:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:19 smithi104 ceph-mon[97044]: pgmap v1865: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:27:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:21 smithi110 ceph-mon[97633]: pgmap v1866: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:27:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:21 smithi104 ceph-mon[97044]: pgmap v1866: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:27:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:23 smithi110 ceph-mon[97633]: pgmap v1867: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:27:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:23 smithi104 ceph-mon[97044]: pgmap v1867: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:27:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:25 smithi110 ceph-mon[97633]: pgmap v1868: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:27:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:25 smithi104 ceph-mon[97044]: pgmap v1868: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:27:26.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:27:25.874+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:27:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:27 smithi110 ceph-mon[97633]: pgmap v1869: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:27 smithi104 ceph-mon[97044]: pgmap v1869: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:29 smithi110 ceph-mon[97633]: pgmap v1870: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:29 smithi104 ceph-mon[97044]: pgmap v1870: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:31 smithi110 ceph-mon[97633]: pgmap v1871: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:31 smithi104 ceph-mon[97044]: pgmap v1871: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:33 smithi110 ceph-mon[97633]: pgmap v1872: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:33 smithi104 ceph-mon[97044]: pgmap v1872: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:35 smithi110 ceph-mon[97633]: pgmap v1873: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:35 smithi104 ceph-mon[97044]: pgmap v1873: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:37 smithi110 ceph-mon[97633]: pgmap v1874: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:37 smithi104 ceph-mon[97044]: pgmap v1874: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:39 smithi110 ceph-mon[97633]: pgmap v1875: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:39 smithi104 ceph-mon[97044]: pgmap v1875: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:27:40.890+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:27:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:41 smithi110 ceph-mon[97633]: pgmap v1876: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:41 smithi104 ceph-mon[97044]: pgmap v1876: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:43 smithi110 ceph-mon[97633]: pgmap v1877: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:27:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:27:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:27:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:27:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:43 smithi104 ceph-mon[97044]: pgmap v1877: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:27:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:27:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:27:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:27:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:45 smithi110 ceph-mon[97633]: pgmap v1878: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:45 smithi104 ceph-mon[97044]: pgmap v1878: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:47 smithi110 ceph-mon[97633]: pgmap v1879: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:47 smithi104 ceph-mon[97044]: pgmap v1879: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:49 smithi110 ceph-mon[97633]: pgmap v1880: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:49 smithi104 ceph-mon[97044]: pgmap v1880: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:51 smithi110 ceph-mon[97633]: pgmap v1881: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:51 smithi104 ceph-mon[97044]: pgmap v1881: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:53 smithi110 ceph-mon[97633]: pgmap v1882: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:53 smithi104 ceph-mon[97044]: pgmap v1882: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:55 smithi110 ceph-mon[97633]: pgmap v1883: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:55 smithi104 ceph-mon[97044]: pgmap v1883: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:27:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:27:55.894+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:27:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:57 smithi110 ceph-mon[97633]: pgmap v1884: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:27:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:57 smithi104 ceph-mon[97044]: pgmap v1884: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:27:59 smithi110 ceph-mon[97633]: pgmap v1885: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:27:59 smithi104 ceph-mon[97044]: pgmap v1885: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:01 smithi110 ceph-mon[97633]: pgmap v1886: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:01 smithi104 ceph-mon[97044]: pgmap v1886: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:03 smithi104 ceph-mon[97044]: pgmap v1887: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:03 smithi110 ceph-mon[97633]: pgmap v1887: 129 pgs: 129 active+clean; 7.8 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:05 smithi104 ceph-mon[97044]: pgmap v1888: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:28:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:05 smithi110 ceph-mon[97633]: pgmap v1888: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:28:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:07 smithi104 ceph-mon[97044]: pgmap v1889: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:28:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:07 smithi110 ceph-mon[97633]: pgmap v1889: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:28:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:08 smithi104 ceph-mon[97044]: osdmap e201: 12 total, 12 up, 12 in 2022-09-10T05:28:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:08 smithi110 ceph-mon[97633]: osdmap e201: 12 total, 12 up, 12 in 2022-09-10T05:28:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:09 smithi104 ceph-mon[97044]: pgmap v1891: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:28:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:09 smithi110 ceph-mon[97633]: pgmap v1891: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:28:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:28:10.910+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:28:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:11 smithi104 ceph-mon[97044]: pgmap v1892: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:28:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:11 smithi110 ceph-mon[97633]: pgmap v1892: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:28:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:13 smithi110 ceph-mon[97633]: pgmap v1893: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:28:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:13 smithi110 ceph-mon[97633]: osdmap e202: 12 total, 12 up, 12 in 2022-09-10T05:28:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:13 smithi104 ceph-mon[97044]: pgmap v1893: 129 pgs: 129 active+clean; 7.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:28:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:13 smithi104 ceph-mon[97044]: osdmap e202: 12 total, 12 up, 12 in 2022-09-10T05:28:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:15 smithi110 ceph-mon[97633]: pgmap v1895: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:28:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:15 smithi104 ceph-mon[97044]: pgmap v1895: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:28:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:17 smithi110 ceph-mon[97633]: pgmap v1896: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:28:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:17 smithi104 ceph-mon[97044]: pgmap v1896: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:28:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:19 smithi110 ceph-mon[97633]: pgmap v1897: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:28:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:19 smithi104 ceph-mon[97044]: pgmap v1897: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:28:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:21 smithi110 ceph-mon[97633]: pgmap v1898: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:28:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:21 smithi104 ceph-mon[97044]: pgmap v1898: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:28:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:23 smithi110 ceph-mon[97633]: pgmap v1899: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:28:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:23 smithi104 ceph-mon[97044]: pgmap v1899: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:28:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:25 smithi110 ceph-mon[97633]: pgmap v1900: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:28:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:25 smithi104 ceph-mon[97044]: pgmap v1900: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:28:26.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:28:25.916+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:28:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:27 smithi110 ceph-mon[97633]: pgmap v1901: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:27 smithi104 ceph-mon[97044]: pgmap v1901: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:29 smithi110 ceph-mon[97633]: pgmap v1902: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:29 smithi104 ceph-mon[97044]: pgmap v1902: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:31 smithi110 ceph-mon[97633]: pgmap v1903: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:31 smithi104 ceph-mon[97044]: pgmap v1903: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:33 smithi110 ceph-mon[97633]: pgmap v1904: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:33 smithi104 ceph-mon[97044]: pgmap v1904: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:35 smithi110 ceph-mon[97633]: pgmap v1905: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:35 smithi104 ceph-mon[97044]: pgmap v1905: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:37 smithi110 ceph-mon[97633]: pgmap v1906: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:38.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:37 smithi104 ceph-mon[97044]: pgmap v1906: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:39 smithi110 ceph-mon[97633]: pgmap v1907: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:39 smithi104 ceph-mon[97044]: pgmap v1907: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:28:40.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:28:42.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:41 smithi110 ceph-mon[97633]: pgmap v1908: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:41 smithi104 ceph-mon[97044]: pgmap v1908: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:43 smithi110 ceph-mon[97633]: pgmap v1909: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:28:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:28:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:28:44.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:28:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:43 smithi104 ceph-mon[97044]: pgmap v1909: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:28:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:28:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:28:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:45 smithi110 ceph-mon[97633]: pgmap v1910: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:45 smithi104 ceph-mon[97044]: pgmap v1910: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:46.806 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:28:46.805+0000 7f0f077ff700 -1 monclient: get_monmap_and_config failed to get config 2022-09-10T05:28:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:47 smithi110 ceph-mon[97633]: pgmap v1911: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:47 smithi104 ceph-mon[97044]: pgmap v1911: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:50.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:49 smithi110 ceph-mon[97633]: pgmap v1912: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:49 smithi104 ceph-mon[97044]: pgmap v1912: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:51 smithi104 ceph-mon[97044]: pgmap v1913: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:51 smithi110 ceph-mon[97633]: pgmap v1913: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:53 smithi104 ceph-mon[97044]: pgmap v1914: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:53 smithi110 ceph-mon[97633]: pgmap v1914: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:55 smithi104 ceph-mon[97044]: pgmap v1915: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:56.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:28:55.934+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:28:56.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:55 smithi110 ceph-mon[97633]: pgmap v1915: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:28:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:57 smithi104 ceph-mon[97044]: pgmap v1916: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:28:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:57 smithi110 ceph-mon[97633]: pgmap v1916: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:28:59 smithi104 ceph-mon[97044]: pgmap v1917: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:28:59 smithi110 ceph-mon[97633]: pgmap v1917: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:01 smithi104 ceph-mon[97044]: pgmap v1918: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:01 smithi110 ceph-mon[97633]: pgmap v1918: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:04.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:03 smithi110 ceph-mon[97633]: pgmap v1919: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:03 smithi104 ceph-mon[97044]: pgmap v1919: 129 pgs: 129 active+clean; 7.8 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:05 smithi110 ceph-mon[97633]: pgmap v1920: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:29:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:05 smithi104 ceph-mon[97044]: pgmap v1920: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:29:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:07 smithi110 ceph-mon[97633]: pgmap v1921: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:29:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:07 smithi104 ceph-mon[97044]: pgmap v1921: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:29:09.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:08 smithi110 ceph-mon[97633]: osdmap e203: 12 total, 12 up, 12 in 2022-09-10T05:29:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:08 smithi104 ceph-mon[97044]: osdmap e203: 12 total, 12 up, 12 in 2022-09-10T05:29:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:09 smithi110 ceph-mon[97633]: pgmap v1923: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:29:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:09 smithi104 ceph-mon[97044]: pgmap v1923: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:29:11.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:29:10.948+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:29:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:11 smithi110 ceph-mon[97633]: pgmap v1924: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:29:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:11 smithi104 ceph-mon[97044]: pgmap v1924: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:29:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:13 smithi110 ceph-mon[97633]: pgmap v1925: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:29:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:13 smithi110 ceph-mon[97633]: osdmap e204: 12 total, 12 up, 12 in 2022-09-10T05:29:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:13 smithi104 ceph-mon[97044]: pgmap v1925: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:29:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:13 smithi104 ceph-mon[97044]: osdmap e204: 12 total, 12 up, 12 in 2022-09-10T05:29:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:15 smithi110 ceph-mon[97633]: pgmap v1927: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:29:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:15 smithi104 ceph-mon[97044]: pgmap v1927: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:29:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:17 smithi110 ceph-mon[97633]: pgmap v1928: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 227 B/s wr, 11 op/s 2022-09-10T05:29:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:17 smithi104 ceph-mon[97044]: pgmap v1928: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 227 B/s wr, 11 op/s 2022-09-10T05:29:20.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:19 smithi110 ceph-mon[97633]: pgmap v1929: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:29:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:19 smithi104 ceph-mon[97044]: pgmap v1929: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:29:22.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:21 smithi110 ceph-mon[97633]: pgmap v1930: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:29:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:21 smithi104 ceph-mon[97044]: pgmap v1930: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:29:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:23 smithi110 ceph-mon[97633]: pgmap v1931: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:29:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:23 smithi104 ceph-mon[97044]: pgmap v1931: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:29:26.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:25 smithi110 ceph-mon[97633]: pgmap v1932: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:29:26.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:25 smithi104 ceph-mon[97044]: pgmap v1932: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:29:26.315 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:29:25.953+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:29:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:27 smithi110 ceph-mon[97633]: pgmap v1933: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:27 smithi104 ceph-mon[97044]: pgmap v1933: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:29 smithi110 ceph-mon[97633]: pgmap v1934: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:29 smithi104 ceph-mon[97044]: pgmap v1934: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:31 smithi110 ceph-mon[97633]: pgmap v1935: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:31 smithi104 ceph-mon[97044]: pgmap v1935: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:33 smithi110 ceph-mon[97633]: pgmap v1936: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:33 smithi104 ceph-mon[97044]: pgmap v1936: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:35 smithi110 ceph-mon[97633]: pgmap v1937: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:35 smithi104 ceph-mon[97044]: pgmap v1937: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:37 smithi110 ceph-mon[97633]: pgmap v1938: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:37 smithi104 ceph-mon[97044]: pgmap v1938: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:39 smithi110 ceph-mon[97633]: pgmap v1939: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:39 smithi104 ceph-mon[97044]: pgmap v1939: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:41.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:29:40.958+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:29:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:41 smithi110 ceph-mon[97633]: pgmap v1940: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:41 smithi104 ceph-mon[97044]: pgmap v1940: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:43 smithi110 ceph-mon[97633]: pgmap v1941: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:29:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:29:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:29:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:29:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:43 smithi104 ceph-mon[97044]: pgmap v1941: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:29:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:29:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:29:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:29:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:45 smithi110 ceph-mon[97633]: pgmap v1942: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:45 smithi104 ceph-mon[97044]: pgmap v1942: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:47 smithi110 ceph-mon[97633]: pgmap v1943: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:47 smithi104 ceph-mon[97044]: pgmap v1943: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:49 smithi110 ceph-mon[97633]: pgmap v1944: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:49 smithi104 ceph-mon[97044]: pgmap v1944: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:51 smithi110 ceph-mon[97633]: pgmap v1945: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:51 smithi104 ceph-mon[97044]: pgmap v1945: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:53 smithi110 ceph-mon[97633]: pgmap v1946: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:53 smithi104 ceph-mon[97044]: pgmap v1946: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:56.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:55 smithi110 ceph-mon[97633]: pgmap v1947: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:55 smithi104 ceph-mon[97044]: pgmap v1947: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:29:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:29:55.961+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:29:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:57 smithi110 ceph-mon[97633]: pgmap v1948: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:29:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:57 smithi104 ceph-mon[97044]: pgmap v1948: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:00.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:29:59 smithi110 ceph-mon[97633]: pgmap v1949: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:29:59 smithi104 ceph-mon[97044]: pgmap v1949: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:30:01.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:30:02.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:01 smithi110 ceph-mon[97633]: pgmap v1950: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:01 smithi104 ceph-mon[97044]: pgmap v1950: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:04.172 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:03 smithi110 ceph-mon[97633]: pgmap v1951: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:04.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:03 smithi104 ceph-mon[97044]: pgmap v1951: 129 pgs: 129 active+clean; 7.9 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:05 smithi104 ceph-mon[97044]: pgmap v1952: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:30:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:05 smithi110 ceph-mon[97633]: pgmap v1952: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:30:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:07 smithi104 ceph-mon[97044]: pgmap v1953: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:30:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:07 smithi110 ceph-mon[97633]: pgmap v1953: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:08 smithi104 ceph-mon[97044]: osdmap e205: 12 total, 12 up, 12 in 2022-09-10T05:30:09.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:08 smithi110 ceph-mon[97633]: osdmap e205: 12 total, 12 up, 12 in 2022-09-10T05:30:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:09 smithi104 ceph-mon[97044]: pgmap v1955: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:30:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:09 smithi110 ceph-mon[97633]: pgmap v1955: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:30:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:30:10.976+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:30:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:11 smithi104 ceph-mon[97044]: pgmap v1956: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:30:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:11 smithi110 ceph-mon[97633]: pgmap v1956: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:30:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:13 smithi110 ceph-mon[97633]: osdmap e206: 12 total, 12 up, 12 in 2022-09-10T05:30:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:13 smithi110 ceph-mon[97633]: pgmap v1958: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:30:14.306 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:13 smithi104 ceph-mon[97044]: osdmap e206: 12 total, 12 up, 12 in 2022-09-10T05:30:14.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:13 smithi104 ceph-mon[97044]: pgmap v1958: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:30:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:15 smithi104 ceph-mon[97044]: pgmap v1959: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:30:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:15 smithi110 ceph-mon[97633]: pgmap v1959: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:30:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:17 smithi104 ceph-mon[97044]: pgmap v1960: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:30:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:17 smithi110 ceph-mon[97633]: pgmap v1960: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:30:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:19 smithi104 ceph-mon[97044]: pgmap v1961: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:30:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:19 smithi110 ceph-mon[97633]: pgmap v1961: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:30:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:21 smithi104 ceph-mon[97044]: pgmap v1962: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:30:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:21 smithi110 ceph-mon[97633]: pgmap v1962: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:30:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:23 smithi110 ceph-mon[97633]: pgmap v1963: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:30:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:23 smithi104 ceph-mon[97044]: pgmap v1963: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:30:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:25 smithi104 ceph-mon[97044]: pgmap v1964: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:30:26.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:30:25.992+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:30:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:25 smithi110 ceph-mon[97633]: pgmap v1964: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:30:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:27 smithi104 ceph-mon[97044]: pgmap v1965: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:27 smithi110 ceph-mon[97633]: pgmap v1965: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:29 smithi104 ceph-mon[97044]: pgmap v1966: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:29 smithi110 ceph-mon[97633]: pgmap v1966: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:31 smithi104 ceph-mon[97044]: pgmap v1967: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:32.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:31 smithi110 ceph-mon[97633]: pgmap v1967: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:34.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:33 smithi110 ceph-mon[97633]: pgmap v1968: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:34.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:33 smithi104 ceph-mon[97044]: pgmap v1968: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:35 smithi104 ceph-mon[97044]: pgmap v1969: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:35 smithi110 ceph-mon[97633]: pgmap v1969: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:38.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:37 smithi104 ceph-mon[97044]: pgmap v1970: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:37 smithi110 ceph-mon[97633]: pgmap v1970: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:39 smithi104 ceph-mon[97044]: pgmap v1971: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:40.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:39 smithi110 ceph-mon[97633]: pgmap v1971: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:30:41.007+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:30:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:41 smithi104 ceph-mon[97044]: pgmap v1972: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:41 smithi110 ceph-mon[97633]: pgmap v1972: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:43 smithi110 ceph-mon[97633]: pgmap v1973: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:30:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:30:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:30:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:30:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:43 smithi104 ceph-mon[97044]: pgmap v1973: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:30:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:30:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:30:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:30:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:45 smithi104 ceph-mon[97044]: pgmap v1974: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:46.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:45 smithi110 ceph-mon[97633]: pgmap v1974: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:47 smithi104 ceph-mon[97044]: pgmap v1975: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:47 smithi110 ceph-mon[97633]: pgmap v1975: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:49 smithi104 ceph-mon[97044]: pgmap v1976: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:50.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:49 smithi110 ceph-mon[97633]: pgmap v1976: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:51 smithi104 ceph-mon[97044]: pgmap v1977: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:51 smithi110 ceph-mon[97633]: pgmap v1977: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:53 smithi110 ceph-mon[97633]: pgmap v1978: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:53 smithi104 ceph-mon[97044]: pgmap v1978: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:55 smithi104 ceph-mon[97044]: pgmap v1979: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:30:56.023+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:30:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:55 smithi110 ceph-mon[97633]: pgmap v1979: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:30:58.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:57 smithi104 ceph-mon[97044]: pgmap v1980: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:30:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:57 smithi110 ceph-mon[97633]: pgmap v1980: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:30:59 smithi104 ceph-mon[97044]: pgmap v1981: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:30:59 smithi110 ceph-mon[97633]: pgmap v1981: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:01 smithi104 ceph-mon[97044]: pgmap v1982: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:01 smithi110 ceph-mon[97633]: pgmap v1982: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:03 smithi110 ceph-mon[97633]: pgmap v1983: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:03 smithi104 ceph-mon[97044]: pgmap v1983: 129 pgs: 129 active+clean; 7.9 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:05 smithi104 ceph-mon[97044]: pgmap v1984: 129 pgs: 129 active+clean; 7.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:31:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:05 smithi110 ceph-mon[97633]: pgmap v1984: 129 pgs: 129 active+clean; 7.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:31:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:07 smithi104 ceph-mon[97044]: pgmap v1985: 129 pgs: 129 active+clean; 7.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:31:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:07 smithi110 ceph-mon[97633]: pgmap v1985: 129 pgs: 129 active+clean; 7.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:08 smithi104 ceph-mon[97044]: osdmap e207: 12 total, 12 up, 12 in 2022-09-10T05:31:09.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:08 smithi110 ceph-mon[97633]: osdmap e207: 12 total, 12 up, 12 in 2022-09-10T05:31:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:09 smithi104 ceph-mon[97044]: pgmap v1987: 129 pgs: 129 active+clean; 7.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:31:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:09 smithi110 ceph-mon[97633]: pgmap v1987: 129 pgs: 129 active+clean; 7.9 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:31:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:31:11.024+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:31:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:11 smithi104 ceph-mon[97044]: pgmap v1988: 129 pgs: 129 active+clean; 7.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:31:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:11 smithi110 ceph-mon[97633]: pgmap v1988: 129 pgs: 129 active+clean; 7.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:31:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:13 smithi110 ceph-mon[97633]: osdmap e208: 12 total, 12 up, 12 in 2022-09-10T05:31:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:13 smithi110 ceph-mon[97633]: pgmap v1990: 129 pgs: 129 active+clean; 7.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:31:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:13 smithi104 ceph-mon[97044]: osdmap e208: 12 total, 12 up, 12 in 2022-09-10T05:31:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:13 smithi104 ceph-mon[97044]: pgmap v1990: 129 pgs: 129 active+clean; 7.9 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:31:16.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:15 smithi104 ceph-mon[97044]: pgmap v1991: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:31:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:15 smithi110 ceph-mon[97633]: pgmap v1991: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:31:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:17 smithi104 ceph-mon[97044]: pgmap v1992: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:31:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:17 smithi110 ceph-mon[97633]: pgmap v1992: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:31:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:19 smithi104 ceph-mon[97044]: pgmap v1993: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:31:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:19 smithi110 ceph-mon[97633]: pgmap v1993: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:31:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:21 smithi104 ceph-mon[97044]: pgmap v1994: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:31:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:21 smithi110 ceph-mon[97633]: pgmap v1994: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:31:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:23 smithi104 ceph-mon[97044]: pgmap v1995: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:31:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:23 smithi110 ceph-mon[97633]: pgmap v1995: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:31:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:25 smithi104 ceph-mon[97044]: pgmap v1996: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:31:26.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:31:26.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:31:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:25 smithi110 ceph-mon[97633]: pgmap v1996: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:31:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:27 smithi104 ceph-mon[97044]: pgmap v1997: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:27 smithi110 ceph-mon[97633]: pgmap v1997: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:29 smithi104 ceph-mon[97044]: pgmap v1998: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:29 smithi110 ceph-mon[97633]: pgmap v1998: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:31 smithi104 ceph-mon[97044]: pgmap v1999: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:31 smithi110 ceph-mon[97633]: pgmap v1999: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:33 smithi104 ceph-mon[97044]: pgmap v2000: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:33 smithi110 ceph-mon[97633]: pgmap v2000: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:35 smithi104 ceph-mon[97044]: pgmap v2001: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:35 smithi110 ceph-mon[97633]: pgmap v2001: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:37 smithi104 ceph-mon[97044]: pgmap v2002: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:37 smithi110 ceph-mon[97633]: pgmap v2002: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:39 smithi104 ceph-mon[97044]: pgmap v2003: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:39 smithi110 ceph-mon[97633]: pgmap v2003: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:31:41.039+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:31:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:41 smithi104 ceph-mon[97044]: pgmap v2004: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:41 smithi110 ceph-mon[97633]: pgmap v2004: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:43 smithi104 ceph-mon[97044]: pgmap v2005: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:31:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:31:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:31:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:31:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:43 smithi110 ceph-mon[97633]: pgmap v2005: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:31:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:31:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:31:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:45 smithi104 ceph-mon[97044]: pgmap v2006: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:45 smithi110 ceph-mon[97633]: pgmap v2006: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:47 smithi104 ceph-mon[97044]: pgmap v2007: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:47 smithi110 ceph-mon[97633]: pgmap v2007: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:49 smithi104 ceph-mon[97044]: pgmap v2008: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:50.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:49 smithi110 ceph-mon[97633]: pgmap v2008: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:51 smithi104 ceph-mon[97044]: pgmap v2009: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:51 smithi110 ceph-mon[97633]: pgmap v2009: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:53 smithi104 ceph-mon[97044]: pgmap v2010: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:53 smithi110 ceph-mon[97633]: pgmap v2010: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:55 smithi104 ceph-mon[97044]: pgmap v2011: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:56.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:31:56.043+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:31:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:55 smithi110 ceph-mon[97633]: pgmap v2011: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:31:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:57 smithi104 ceph-mon[97044]: pgmap v2012: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:31:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:57 smithi110 ceph-mon[97633]: pgmap v2012: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:31:59 smithi104 ceph-mon[97044]: pgmap v2013: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:31:59 smithi110 ceph-mon[97633]: pgmap v2013: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:02.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:01 smithi104 ceph-mon[97044]: pgmap v2014: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:01 smithi110 ceph-mon[97633]: pgmap v2014: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:03 smithi104 ceph-mon[97044]: pgmap v2015: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:32:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:03 smithi110 ceph-mon[97633]: pgmap v2015: 129 pgs: 129 active+clean; 7.9 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:32:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:05 smithi104 ceph-mon[97044]: pgmap v2016: 129 pgs: 129 active+clean; 7.9 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:32:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:05 smithi110 ceph-mon[97633]: pgmap v2016: 129 pgs: 129 active+clean; 7.9 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:32:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:07 smithi104 ceph-mon[97044]: pgmap v2017: 129 pgs: 129 active+clean; 7.9 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T05:32:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:07 smithi110 ceph-mon[97633]: pgmap v2017: 129 pgs: 129 active+clean; 7.9 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T05:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:08 smithi104 ceph-mon[97044]: osdmap e209: 12 total, 12 up, 12 in 2022-09-10T05:32:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:08 smithi110 ceph-mon[97633]: osdmap e209: 12 total, 12 up, 12 in 2022-09-10T05:32:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:09 smithi104 ceph-mon[97044]: pgmap v2019: 129 pgs: 129 active+clean; 7.9 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:32:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:09 smithi110 ceph-mon[97633]: pgmap v2019: 129 pgs: 129 active+clean; 7.9 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:32:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:32:11.047+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:32:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:11 smithi104 ceph-mon[97044]: pgmap v2020: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:32:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:11 smithi110 ceph-mon[97633]: pgmap v2020: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:32:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:13 smithi104 ceph-mon[97044]: osdmap e210: 12 total, 12 up, 12 in 2022-09-10T05:32:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:13 smithi104 ceph-mon[97044]: pgmap v2022: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:32:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:13 smithi110 ceph-mon[97633]: osdmap e210: 12 total, 12 up, 12 in 2022-09-10T05:32:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:13 smithi110 ceph-mon[97633]: pgmap v2022: 129 pgs: 129 active+clean; 7.9 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:32:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:15 smithi104 ceph-mon[97044]: pgmap v2023: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:32:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:15 smithi110 ceph-mon[97633]: pgmap v2023: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:32:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:17 smithi104 ceph-mon[97044]: pgmap v2024: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:32:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:17 smithi110 ceph-mon[97633]: pgmap v2024: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:32:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:19 smithi104 ceph-mon[97044]: pgmap v2025: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:32:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:19 smithi110 ceph-mon[97633]: pgmap v2025: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:32:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:21 smithi104 ceph-mon[97044]: pgmap v2026: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:32:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:21 smithi110 ceph-mon[97633]: pgmap v2026: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:32:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:23 smithi104 ceph-mon[97044]: pgmap v2027: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:32:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:23 smithi110 ceph-mon[97633]: pgmap v2027: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:32:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:25 smithi104 ceph-mon[97044]: pgmap v2028: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:32:26.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:32:26.049+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:32:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:25 smithi110 ceph-mon[97633]: pgmap v2028: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:32:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:27 smithi104 ceph-mon[97044]: pgmap v2029: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:27 smithi110 ceph-mon[97633]: pgmap v2029: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:29 smithi104 ceph-mon[97044]: pgmap v2030: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:29 smithi110 ceph-mon[97633]: pgmap v2030: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:31 smithi104 ceph-mon[97044]: pgmap v2031: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:31 smithi110 ceph-mon[97633]: pgmap v2031: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:33 smithi104 ceph-mon[97044]: pgmap v2032: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:33 smithi110 ceph-mon[97633]: pgmap v2032: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:35 smithi104 ceph-mon[97044]: pgmap v2033: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:35 smithi110 ceph-mon[97633]: pgmap v2033: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:37 smithi104 ceph-mon[97044]: pgmap v2034: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:37 smithi110 ceph-mon[97633]: pgmap v2034: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:39 smithi104 ceph-mon[97044]: pgmap v2035: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:39 smithi110 ceph-mon[97633]: pgmap v2035: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:32:41.051+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:32:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:41 smithi104 ceph-mon[97044]: pgmap v2036: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:42.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:41 smithi110 ceph-mon[97633]: pgmap v2036: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:43 smithi104 ceph-mon[97044]: pgmap v2037: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:32:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:32:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:32:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:32:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:43 smithi110 ceph-mon[97633]: pgmap v2037: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:32:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:45 smithi104 ceph-mon[97044]: pgmap v2038: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:45 smithi110 ceph-mon[97633]: pgmap v2038: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:47 smithi104 ceph-mon[97044]: pgmap v2039: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:47 smithi110 ceph-mon[97633]: pgmap v2039: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:50.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:49 smithi104 ceph-mon[97044]: pgmap v2040: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:49 smithi110 ceph-mon[97633]: pgmap v2040: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:52 smithi104 ceph-mon[97044]: pgmap v2041: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:52 smithi110 ceph-mon[97633]: pgmap v2041: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:32:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:53 smithi104 ceph-mon[97044]: pgmap v2042: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:53 smithi110 ceph-mon[97633]: pgmap v2042: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:32:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:55 smithi104 ceph-mon[97044]: pgmap v2043: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:32:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:55 smithi110 ceph-mon[97633]: pgmap v2043: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:32:56.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:32:56.054+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:32:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:32:57 smithi104 ceph-mon[97044]: pgmap v2044: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:32:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:32:57 smithi110 ceph-mon[97633]: pgmap v2044: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:33:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:00 smithi104 ceph-mon[97044]: pgmap v2045: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:33:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:00 smithi110 ceph-mon[97633]: pgmap v2045: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:33:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:02 smithi104 ceph-mon[97044]: pgmap v2046: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:33:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:02 smithi110 ceph-mon[97633]: pgmap v2046: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:33:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:03 smithi104 ceph-mon[97044]: pgmap v2047: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:33:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:03 smithi110 ceph-mon[97633]: pgmap v2047: 129 pgs: 129 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T05:33:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:05 smithi104 ceph-mon[97044]: pgmap v2048: 129 pgs: 129 active+clean; 7.9 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:33:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:05 smithi110 ceph-mon[97633]: pgmap v2048: 129 pgs: 129 active+clean; 7.9 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:33:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:08 smithi104 ceph-mon[97044]: pgmap v2049: 129 pgs: 129 active+clean; 7.9 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:33:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:08 smithi110 ceph-mon[97633]: pgmap v2049: 129 pgs: 129 active+clean; 7.9 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:09 smithi104 ceph-mon[97044]: osdmap e211: 12 total, 12 up, 12 in 2022-09-10T05:33:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:09 smithi110 ceph-mon[97633]: osdmap e211: 12 total, 12 up, 12 in 2022-09-10T05:33:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:10 smithi104 ceph-mon[97044]: pgmap v2051: 129 pgs: 129 active+clean; 7.9 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:33:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:10 smithi110 ceph-mon[97633]: pgmap v2051: 129 pgs: 129 active+clean; 7.9 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:33:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:33:11.070+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:33:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:12 smithi104 ceph-mon[97044]: pgmap v2052: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:33:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:12 smithi110 ceph-mon[97633]: pgmap v2052: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:33:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:13 smithi104 ceph-mon[97044]: osdmap e212: 12 total, 12 up, 12 in 2022-09-10T05:33:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:13 smithi104 ceph-mon[97044]: pgmap v2054: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:33:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:13 smithi110 ceph-mon[97633]: osdmap e212: 12 total, 12 up, 12 in 2022-09-10T05:33:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:13 smithi110 ceph-mon[97633]: pgmap v2054: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:33:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:16 smithi104 ceph-mon[97044]: pgmap v2055: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:33:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:16 smithi110 ceph-mon[97633]: pgmap v2055: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:33:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:18 smithi104 ceph-mon[97044]: pgmap v2056: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:33:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:18 smithi110 ceph-mon[97633]: pgmap v2056: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:33:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:20 smithi104 ceph-mon[97044]: pgmap v2057: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:33:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:20 smithi110 ceph-mon[97633]: pgmap v2057: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:33:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:22 smithi104 ceph-mon[97044]: pgmap v2058: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:33:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:22 smithi110 ceph-mon[97633]: pgmap v2058: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:33:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:23 smithi104 ceph-mon[97044]: pgmap v2059: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:33:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:23 smithi110 ceph-mon[97633]: pgmap v2059: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:33:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:26 smithi104 ceph-mon[97044]: pgmap v2060: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:33:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:26 smithi110 ceph-mon[97633]: pgmap v2060: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:33:26.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:33:26.072+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:33:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:28 smithi104 ceph-mon[97044]: pgmap v2061: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:28 smithi110 ceph-mon[97633]: pgmap v2061: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:30 smithi104 ceph-mon[97044]: pgmap v2062: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:30 smithi110 ceph-mon[97633]: pgmap v2062: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:32 smithi104 ceph-mon[97044]: pgmap v2063: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:32 smithi110 ceph-mon[97633]: pgmap v2063: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:33 smithi104 ceph-mon[97044]: pgmap v2064: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:33 smithi110 ceph-mon[97633]: pgmap v2064: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:36 smithi104 ceph-mon[97044]: pgmap v2065: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:36 smithi110 ceph-mon[97633]: pgmap v2065: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:38 smithi104 ceph-mon[97044]: pgmap v2066: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:33:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:38 smithi110 ceph-mon[97633]: pgmap v2066: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:33:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:40 smithi104 ceph-mon[97044]: pgmap v2067: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:33:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:40 smithi110 ceph-mon[97633]: pgmap v2067: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:33:41.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:33:41.079+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:33:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:42 smithi104 ceph-mon[97044]: pgmap v2068: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:42 smithi110 ceph-mon[97633]: pgmap v2068: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:43 smithi104 ceph-mon[97044]: pgmap v2069: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:33:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:33:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:33:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:33:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:33:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:43 smithi110 ceph-mon[97633]: pgmap v2069: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:33:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:33:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:33:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:33:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:33:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:46 smithi104 ceph-mon[97044]: pgmap v2070: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:46 smithi110 ceph-mon[97633]: pgmap v2070: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:2022-09-10T05:33:46.808+0000 7f0f077ff700 0 librados: client.admin authentication error (1) Operation not permitted 2022-09-10T05:33:46.810 INFO:teuthology.orchestra.run.smithi081.stderr:[errno 1] RADOS permission error (error connecting to the cluster) 2022-09-10T05:33:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-09-10T05:33:46.819 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/exec.py", line 72, in task c], File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/'" 2022-09-10T05:33:46.820 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2022-09-10T05:33:46.831 DEBUG:tasks.check_counter:Getting stats from a 2022-09-10T05:33:46.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:244418605927bb7842e73fc5d2a53889b760937f shell --fsid 70cf895c-30bf-11ed-8431-001a4aab830c -- ceph daemon mds.a perf dump 2022-09-10T05:33:47.512 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/70cf895c-30bf-11ed-8431-001a4aab830c/mon.a/config 2022-09-10T05:33:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:48 smithi104 ceph-mon[97044]: pgmap v2071: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:48 smithi110 ceph-mon[97633]: pgmap v2071: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:50 smithi104 ceph-mon[97044]: pgmap v2072: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:50 smithi110 ceph-mon[97633]: pgmap v2072: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:52 smithi104 ceph-mon[97044]: pgmap v2073: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:52 smithi110 ceph-mon[97633]: pgmap v2073: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:53 smithi104 ceph-mon[97044]: pgmap v2074: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:53 smithi110 ceph-mon[97633]: pgmap v2074: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:56 smithi104 ceph-mon[97044]: pgmap v2075: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:56 smithi110 ceph-mon[97633]: pgmap v2075: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:33:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:33:56.095+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:33:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:33:58 smithi104 ceph-mon[97044]: pgmap v2076: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:33:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:33:58 smithi110 ceph-mon[97633]: pgmap v2076: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:00 smithi104 ceph-mon[97044]: pgmap v2077: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:00 smithi110 ceph-mon[97633]: pgmap v2077: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:02 smithi104 ceph-mon[97044]: pgmap v2078: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:02 smithi110 ceph-mon[97633]: pgmap v2078: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:03 smithi104 ceph-mon[97044]: pgmap v2079: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:03 smithi110 ceph-mon[97633]: pgmap v2079: 129 pgs: 129 active+clean; 7.9 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:06 smithi104 ceph-mon[97044]: pgmap v2080: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:34:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:06 smithi110 ceph-mon[97633]: pgmap v2080: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:34:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:08 smithi104 ceph-mon[97044]: pgmap v2081: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:34:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:08 smithi110 ceph-mon[97633]: pgmap v2081: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:34:08.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:08.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:08.587786+0000) 2022-09-10T05:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:09 smithi104 ceph-mon[97044]: osdmap e213: 12 total, 12 up, 12 in 2022-09-10T05:34:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:08.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:08.995076+0000) 2022-09-10T05:34:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:09 smithi110 ceph-mon[97633]: osdmap e213: 12 total, 12 up, 12 in 2022-09-10T05:34:09.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:09.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:09.588049+0000) 2022-09-10T05:34:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:10 smithi104 ceph-mon[97044]: pgmap v2083: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:34:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:09.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:09.995282+0000) 2022-09-10T05:34:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:10 smithi110 ceph-mon[97633]: pgmap v2083: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:34:10.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:10.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:10.588264+0000) 2022-09-10T05:34:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:10.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:10.995496+0000) 2022-09-10T05:34:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:34:11.110+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:34:11.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:11.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:11.588452+0000) 2022-09-10T05:34:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:12 smithi104 ceph-mon[97044]: pgmap v2084: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:34:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:11.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:11.995747+0000) 2022-09-10T05:34:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:12 smithi110 ceph-mon[97633]: pgmap v2084: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:34:12.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:12.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:12.588605+0000) 2022-09-10T05:34:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:12.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:12.995973+0000) 2022-09-10T05:34:13.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:13.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:13.588788+0000) 2022-09-10T05:34:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:13 smithi104 ceph-mon[97044]: osdmap e214: 12 total, 12 up, 12 in 2022-09-10T05:34:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:13 smithi104 ceph-mon[97044]: pgmap v2086: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:34:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:13.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:13.996242+0000) 2022-09-10T05:34:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:13 smithi110 ceph-mon[97633]: osdmap e214: 12 total, 12 up, 12 in 2022-09-10T05:34:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:13 smithi110 ceph-mon[97633]: pgmap v2086: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:34:14.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:14.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:14.588973+0000) 2022-09-10T05:34:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:14.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:14.996378+0000) 2022-09-10T05:34:15.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:15.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:15.589133+0000) 2022-09-10T05:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:16 smithi104 ceph-mon[97044]: pgmap v2087: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:34:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:15.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:15.996588+0000) 2022-09-10T05:34:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:16 smithi110 ceph-mon[97633]: pgmap v2087: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:34:16.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:16.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:16.589355+0000) 2022-09-10T05:34:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:16.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:16.996855+0000) 2022-09-10T05:34:17.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:17.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:17.589624+0000) 2022-09-10T05:34:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:18 smithi104 ceph-mon[97044]: pgmap v2088: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:34:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:17.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:17.997122+0000) 2022-09-10T05:34:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:18 smithi110 ceph-mon[97633]: pgmap v2088: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:34:18.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:18.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:18.589872+0000) 2022-09-10T05:34:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:18.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:18.997355+0000) 2022-09-10T05:34:19.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:19.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:19.590089+0000) 2022-09-10T05:34:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:20 smithi104 ceph-mon[97044]: pgmap v2089: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:34:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:19.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:19.997581+0000) 2022-09-10T05:34:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:20 smithi110 ceph-mon[97633]: pgmap v2089: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:34:20.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:20.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:20.590250+0000) 2022-09-10T05:34:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:20.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:20.997855+0000) 2022-09-10T05:34:21.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:21.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:21.590473+0000) 2022-09-10T05:34:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:22 smithi104 ceph-mon[97044]: pgmap v2090: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:34:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:21.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:21.998134+0000) 2022-09-10T05:34:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:22 smithi110 ceph-mon[97633]: pgmap v2090: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:34:22.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:22.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:22.590688+0000) 2022-09-10T05:34:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:22.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:22.998363+0000) 2022-09-10T05:34:23.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:23.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:23.590863+0000) 2022-09-10T05:34:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:23 smithi104 ceph-mon[97044]: pgmap v2091: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:34:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:23.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:23.998682+0000) 2022-09-10T05:34:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:23 smithi110 ceph-mon[97633]: pgmap v2091: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:34:25.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:24.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:24.591116+0000) 2022-09-10T05:34:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:24.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:24.998872+0000) 2022-09-10T05:34:25.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:25.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:25.591397+0000) 2022-09-10T05:34:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:26 smithi104 ceph-mon[97044]: pgmap v2092: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:34:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:25.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:25.999037+0000) 2022-09-10T05:34:26.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:34:26.112+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:34:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:26 smithi110 ceph-mon[97633]: pgmap v2092: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:34:27.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:26.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:26.591658+0000) 2022-09-10T05:34:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:26.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:26.999257+0000) 2022-09-10T05:34:28.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:27.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:27.591854+0000) 2022-09-10T05:34:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:28 smithi104 ceph-mon[97044]: pgmap v2093: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:27.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:27.999452+0000) 2022-09-10T05:34:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:28 smithi110 ceph-mon[97633]: pgmap v2093: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:29.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:28.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:28.592128+0000) 2022-09-10T05:34:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:28.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:28.999611+0000) 2022-09-10T05:34:30.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:29.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:29.592368+0000) 2022-09-10T05:34:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:30 smithi104 ceph-mon[97044]: pgmap v2094: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:29.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:29.999780+0000) 2022-09-10T05:34:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:30 smithi110 ceph-mon[97633]: pgmap v2094: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:31.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:30.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:30.592592+0000) 2022-09-10T05:34:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:30.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:30.999905+0000) 2022-09-10T05:34:32.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:31.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:31.592836+0000) 2022-09-10T05:34:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:32 smithi104 ceph-mon[97044]: pgmap v2095: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:31.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:32.000134+0000) 2022-09-10T05:34:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:32 smithi110 ceph-mon[97633]: pgmap v2095: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:33.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:32.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:32.593072+0000) 2022-09-10T05:34:33.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:32.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:33.000384+0000) 2022-09-10T05:34:34.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:33.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:33.593306+0000) 2022-09-10T05:34:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:33 smithi104 ceph-mon[97044]: pgmap v2096: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:34.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:34.000624+0000) 2022-09-10T05:34:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:33 smithi110 ceph-mon[97633]: pgmap v2096: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:35.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:34.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:34.593538+0000) 2022-09-10T05:34:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:35.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:35.000842+0000) 2022-09-10T05:34:36.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:35.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:35.593746+0000) 2022-09-10T05:34:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:36 smithi104 ceph-mon[97044]: pgmap v2097: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:36.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:36.001120+0000) 2022-09-10T05:34:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:36 smithi110 ceph-mon[97633]: pgmap v2097: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:37.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:36.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:36.593874+0000) 2022-09-10T05:34:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:37.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:37.001359+0000) 2022-09-10T05:34:38.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:37.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:37.594030+0000) 2022-09-10T05:34:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:38 smithi104 ceph-mon[97044]: pgmap v2098: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:38.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:38.001523+0000) 2022-09-10T05:34:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:38 smithi110 ceph-mon[97633]: pgmap v2098: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:39.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:38.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:38.594271+0000) 2022-09-10T05:34:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:39.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:39.001732+0000) 2022-09-10T05:34:40.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:39.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:39.594530+0000) 2022-09-10T05:34:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:40 smithi104 ceph-mon[97044]: pgmap v2099: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:40.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:40.002000+0000) 2022-09-10T05:34:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:40 smithi110 ceph-mon[97633]: pgmap v2099: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:41.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:40.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:40.594770+0000) 2022-09-10T05:34:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:34:41.124+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:34:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:41.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:41.002254+0000) 2022-09-10T05:34:42.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:41.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:41.595017+0000) 2022-09-10T05:34:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:42 smithi104 ceph-mon[97044]: pgmap v2100: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:42.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:42.002504+0000) 2022-09-10T05:34:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:42 smithi110 ceph-mon[97633]: pgmap v2100: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:43.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:42.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:42.595165+0000) 2022-09-10T05:34:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:43.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:43.002741+0000) 2022-09-10T05:34:44.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:43.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:43.595364+0000) 2022-09-10T05:34:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:43 smithi104 ceph-mon[97044]: pgmap v2101: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:34:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:34:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:34:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:34:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:44.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:44.002999+0000) 2022-09-10T05:34:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:43 smithi110 ceph-mon[97633]: pgmap v2101: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:34:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:34:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:34:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:34:45.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:44.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:44.595557+0000) 2022-09-10T05:34:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:45.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:45.003220+0000) 2022-09-10T05:34:46.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:45.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:45.595763+0000) 2022-09-10T05:34:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:46 smithi104 ceph-mon[97044]: pgmap v2102: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:46.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:46.003480+0000) 2022-09-10T05:34:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:46 smithi110 ceph-mon[97633]: pgmap v2102: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:47.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:46.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:46.595989+0000) 2022-09-10T05:34:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:47.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:47.003675+0000) 2022-09-10T05:34:48.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:47.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:47.596249+0000) 2022-09-10T05:34:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:48 smithi104 ceph-mon[97044]: pgmap v2103: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:48.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:48.003815+0000) 2022-09-10T05:34:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:48 smithi110 ceph-mon[97633]: pgmap v2103: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:49.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:48.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:48.596419+0000) 2022-09-10T05:34:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:49.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:49.003960+0000) 2022-09-10T05:34:50.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:49.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:49.596669+0000) 2022-09-10T05:34:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:50 smithi104 ceph-mon[97044]: pgmap v2104: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:50.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:50.004220+0000) 2022-09-10T05:34:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:50 smithi110 ceph-mon[97633]: pgmap v2104: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:51.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:50.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:50.596856+0000) 2022-09-10T05:34:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:51.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:51.004445+0000) 2022-09-10T05:34:52.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:51.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:51.597056+0000) 2022-09-10T05:34:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:52 smithi104 ceph-mon[97044]: pgmap v2105: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:52.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:52.004630+0000) 2022-09-10T05:34:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:52 smithi110 ceph-mon[97633]: pgmap v2105: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:53.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:52.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:52.597298+0000) 2022-09-10T05:34:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:53.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:53.004868+0000) 2022-09-10T05:34:54.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:53.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:53.597520+0000) 2022-09-10T05:34:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:53 smithi104 ceph-mon[97044]: pgmap v2106: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:54.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:54.005165+0000) 2022-09-10T05:34:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:53 smithi110 ceph-mon[97633]: pgmap v2106: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:55.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:54.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:54.597721+0000) 2022-09-10T05:34:55.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:55.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:55.005395+0000) 2022-09-10T05:34:56.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:55.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:55.597937+0000) 2022-09-10T05:34:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:56 smithi104 ceph-mon[97044]: pgmap v2107: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:56.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:56.005645+0000) 2022-09-10T05:34:56.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:34:56.133+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:34:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:56 smithi110 ceph-mon[97633]: pgmap v2107: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:34:57.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:56.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:56.598139+0000) 2022-09-10T05:34:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:57.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:57.005894+0000) 2022-09-10T05:34:58.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:57.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:57.598296+0000) 2022-09-10T05:34:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:34:58 smithi104 ceph-mon[97044]: pgmap v2108: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:58.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:58.006189+0000) 2022-09-10T05:34:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:34:58 smithi110 ceph-mon[97633]: pgmap v2108: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:34:59.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:58.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:58.598496+0000) 2022-09-10T05:34:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:34:59.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:59.006394+0000) 2022-09-10T05:35:00.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:34:59.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:34:59.598670+0000) 2022-09-10T05:35:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:00 smithi104 ceph-mon[97044]: pgmap v2109: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:00.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:00.006649+0000) 2022-09-10T05:35:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:00 smithi110 ceph-mon[97633]: pgmap v2109: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:01.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:00.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:00.598888+0000) 2022-09-10T05:35:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:01.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:01.006804+0000) 2022-09-10T05:35:02.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:01.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:01.599058+0000) 2022-09-10T05:35:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:02 smithi104 ceph-mon[97044]: pgmap v2110: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:02.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:02.006989+0000) 2022-09-10T05:35:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:02 smithi110 ceph-mon[97633]: pgmap v2110: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:03.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:02.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:02.599238+0000) 2022-09-10T05:35:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:03.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:03.007245+0000) 2022-09-10T05:35:04.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:03.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:03.599398+0000) 2022-09-10T05:35:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:03 smithi104 ceph-mon[97044]: pgmap v2111: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:04.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:04.007490+0000) 2022-09-10T05:35:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:03 smithi110 ceph-mon[97633]: pgmap v2111: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:05.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:04.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:04.599536+0000) 2022-09-10T05:35:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:05.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:05.007668+0000) 2022-09-10T05:35:06.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:05.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:05.599719+0000) 2022-09-10T05:35:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:06 smithi104 ceph-mon[97044]: pgmap v2112: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:35:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:06.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:06.007886+0000) 2022-09-10T05:35:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:06 smithi110 ceph-mon[97633]: pgmap v2112: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:35:07.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:06.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:06.599956+0000) 2022-09-10T05:35:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:07.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:07.008040+0000) 2022-09-10T05:35:08.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:07.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:07.600203+0000) 2022-09-10T05:35:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:08 smithi104 ceph-mon[97044]: pgmap v2113: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:35:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:08.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:08.008272+0000) 2022-09-10T05:35:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:08 smithi110 ceph-mon[97633]: pgmap v2113: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:35:09.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:08.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:08.600414+0000) 2022-09-10T05:35:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:09.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:09.008521+0000) 2022-09-10T05:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:09 smithi110 ceph-mon[97633]: osdmap e215: 12 total, 12 up, 12 in 2022-09-10T05:35:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:09 smithi104 ceph-mon[97044]: osdmap e215: 12 total, 12 up, 12 in 2022-09-10T05:35:10.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:09.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:09.600605+0000) 2022-09-10T05:35:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:10.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:10.008749+0000) 2022-09-10T05:35:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:10 smithi110 ceph-mon[97633]: pgmap v2115: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:35:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:10 smithi104 ceph-mon[97044]: pgmap v2115: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:35:11.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:10.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:10.600812+0000) 2022-09-10T05:35:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:11.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:11.008994+0000) 2022-09-10T05:35:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:35:11.144+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:35:12.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:11.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:11.601028+0000) 2022-09-10T05:35:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:12.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:12.009248+0000) 2022-09-10T05:35:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:12 smithi110 ceph-mon[97633]: pgmap v2116: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:35:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:12 smithi104 ceph-mon[97044]: pgmap v2116: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:35:13.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:12.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:12.601255+0000) 2022-09-10T05:35:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:13.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:13.009453+0000) 2022-09-10T05:35:14.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:13.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:13.601537+0000) 2022-09-10T05:35:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:13 smithi104 ceph-mon[97044]: osdmap e216: 12 total, 12 up, 12 in 2022-09-10T05:35:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:13 smithi104 ceph-mon[97044]: pgmap v2118: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:35:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:14.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:14.009704+0000) 2022-09-10T05:35:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:13 smithi110 ceph-mon[97633]: osdmap e216: 12 total, 12 up, 12 in 2022-09-10T05:35:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:13 smithi110 ceph-mon[97633]: pgmap v2118: 129 pgs: 129 active+clean; 7.9 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:35:15.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:14.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:14.601750+0000) 2022-09-10T05:35:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:15.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:15.009918+0000) 2022-09-10T05:35:16.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:15.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:15.601958+0000) 2022-09-10T05:35:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:16 smithi104 ceph-mon[97044]: pgmap v2119: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:35:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:16.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:16.010188+0000) 2022-09-10T05:35:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:16 smithi110 ceph-mon[97633]: pgmap v2119: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:35:17.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:16.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:16.602203+0000) 2022-09-10T05:35:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:17.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:17.010458+0000) 2022-09-10T05:35:18.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:17.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:17.602423+0000) 2022-09-10T05:35:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:18 smithi104 ceph-mon[97044]: pgmap v2120: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:35:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:18.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:18.010668+0000) 2022-09-10T05:35:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:18 smithi110 ceph-mon[97633]: pgmap v2120: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:35:19.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:18.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:18.602693+0000) 2022-09-10T05:35:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:19.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:19.010886+0000) 2022-09-10T05:35:20.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:19.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:19.602909+0000) 2022-09-10T05:35:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:20 smithi104 ceph-mon[97044]: pgmap v2121: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:35:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:20.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:20.011175+0000) 2022-09-10T05:35:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:20 smithi110 ceph-mon[97633]: pgmap v2121: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:35:21.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:20.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:20.603113+0000) 2022-09-10T05:35:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:21.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:21.011369+0000) 2022-09-10T05:35:22.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:21.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:21.603279+0000) 2022-09-10T05:35:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:22.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:22.011568+0000) 2022-09-10T05:35:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:22 smithi110 ceph-mon[97633]: pgmap v2122: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:35:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:22 smithi104 ceph-mon[97044]: pgmap v2122: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:35:23.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:22.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:22.603528+0000) 2022-09-10T05:35:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:23.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:23.011804+0000) 2022-09-10T05:35:24.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:23.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:23.603745+0000) 2022-09-10T05:35:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:23 smithi104 ceph-mon[97044]: pgmap v2123: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:35:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:24.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:24.012126+0000) 2022-09-10T05:35:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:23 smithi110 ceph-mon[97633]: pgmap v2123: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:35:25.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:24.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:24.604031+0000) 2022-09-10T05:35:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:25.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:25.012278+0000) 2022-09-10T05:35:25.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:25.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:25.604294+0000) 2022-09-10T05:35:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:26 smithi104 ceph-mon[97044]: pgmap v2124: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:35:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:26.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:26.012445+0000) 2022-09-10T05:35:26.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:35:26.147+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:35:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:26 smithi110 ceph-mon[97633]: pgmap v2124: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:35:27.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:26.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:26.604565+0000) 2022-09-10T05:35:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:27.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:27.012620+0000) 2022-09-10T05:35:28.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:27.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:27.604814+0000) 2022-09-10T05:35:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:28.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:28.012853+0000) 2022-09-10T05:35:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:28 smithi110 ceph-mon[97633]: pgmap v2125: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:28 smithi104 ceph-mon[97044]: pgmap v2125: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:29.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:28.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:28.605069+0000) 2022-09-10T05:35:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:29.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:29.013141+0000) 2022-09-10T05:35:30.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:29.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:29.605357+0000) 2022-09-10T05:35:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:30.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:30.013327+0000) 2022-09-10T05:35:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:30 smithi110 ceph-mon[97633]: pgmap v2126: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:30 smithi104 ceph-mon[97044]: pgmap v2126: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:31.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:30.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:30.605627+0000) 2022-09-10T05:35:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:31.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:31.013560+0000) 2022-09-10T05:35:32.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:31.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:31.605834+0000) 2022-09-10T05:35:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:32.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:32.013806+0000) 2022-09-10T05:35:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:32 smithi110 ceph-mon[97633]: pgmap v2127: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:32 smithi104 ceph-mon[97044]: pgmap v2127: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:33.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:32.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:32.605992+0000) 2022-09-10T05:35:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:33.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:33.014043+0000) 2022-09-10T05:35:34.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:33.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:33.606266+0000) 2022-09-10T05:35:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:33 smithi104 ceph-mon[97044]: pgmap v2128: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:34.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:34.014288+0000) 2022-09-10T05:35:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:33 smithi110 ceph-mon[97633]: pgmap v2128: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:35.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:34.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:34.606494+0000) 2022-09-10T05:35:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:35.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:35.014480+0000) 2022-09-10T05:35:36.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:35.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:35.606633+0000) 2022-09-10T05:35:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:36.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:36.014684+0000) 2022-09-10T05:35:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:36 smithi110 ceph-mon[97633]: pgmap v2129: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:36 smithi104 ceph-mon[97044]: pgmap v2129: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:37.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:36.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:36.606768+0000) 2022-09-10T05:35:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:37.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:37.014873+0000) 2022-09-10T05:35:38.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:37.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:37.606953+0000) 2022-09-10T05:35:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:38.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:38.015066+0000) 2022-09-10T05:35:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:38 smithi110 ceph-mon[97633]: pgmap v2130: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:38 smithi104 ceph-mon[97044]: pgmap v2130: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:39.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:38.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:38.607168+0000) 2022-09-10T05:35:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:39.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:39.015300+0000) 2022-09-10T05:35:40.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:39.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:39.607431+0000) 2022-09-10T05:35:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:40.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:40.015496+0000) 2022-09-10T05:35:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:40 smithi110 ceph-mon[97633]: pgmap v2131: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:40 smithi104 ceph-mon[97044]: pgmap v2131: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:41.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:40.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:40.607705+0000) 2022-09-10T05:35:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:41.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:41.015694+0000) 2022-09-10T05:35:41.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:35:41.162+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:35:42.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:41.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:41.607906+0000) 2022-09-10T05:35:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:42.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:42.015939+0000) 2022-09-10T05:35:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:42 smithi110 ceph-mon[97633]: pgmap v2132: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:42 smithi104 ceph-mon[97044]: pgmap v2132: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:43.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:42.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:42.608131+0000) 2022-09-10T05:35:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:43.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:43.016189+0000) 2022-09-10T05:35:44.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:43.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:43.608352+0000) 2022-09-10T05:35:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:43 smithi104 ceph-mon[97044]: osdmap e217: 12 total, 12 up, 12 in 2022-09-10T05:35:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:43 smithi104 ceph-mon[97044]: pgmap v2134: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:35:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:35:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:35:44.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:35:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:44.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:44.016403+0000) 2022-09-10T05:35:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:43 smithi110 ceph-mon[97633]: osdmap e217: 12 total, 12 up, 12 in 2022-09-10T05:35:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:43 smithi110 ceph-mon[97633]: pgmap v2134: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:35:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:35:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:35:44.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:35:45.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:44.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:44.608593+0000) 2022-09-10T05:35:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:45.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:45.016609+0000) 2022-09-10T05:35:46.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:45.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:45.608791+0000) 2022-09-10T05:35:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:46.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:46.016747+0000) 2022-09-10T05:35:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:46 smithi110 ceph-mon[97633]: pgmap v2135: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:46 smithi104 ceph-mon[97044]: pgmap v2135: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:47.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:46.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:46.609036+0000) 2022-09-10T05:35:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:47.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:47.016939+0000) 2022-09-10T05:35:48.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:47.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:47.609275+0000) 2022-09-10T05:35:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:48.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:48.017130+0000) 2022-09-10T05:35:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:48 smithi110 ceph-mon[97633]: pgmap v2136: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:48 smithi104 ceph-mon[97044]: pgmap v2136: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:49.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:48.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:48.609532+0000) 2022-09-10T05:35:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:49.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:49.017361+0000) 2022-09-10T05:35:50.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:49.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:49.609729+0000) 2022-09-10T05:35:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:50.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:50.017556+0000) 2022-09-10T05:35:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:50 smithi110 ceph-mon[97633]: pgmap v2137: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:50 smithi104 ceph-mon[97044]: pgmap v2137: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:51.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:50.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:50.609923+0000) 2022-09-10T05:35:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:51.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:51.017764+0000) 2022-09-10T05:35:52.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:51.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:51.610135+0000) 2022-09-10T05:35:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:52.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:52.018022+0000) 2022-09-10T05:35:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:52 smithi110 ceph-mon[97633]: pgmap v2138: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:52 smithi104 ceph-mon[97044]: pgmap v2138: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:53.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:52.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:52.610343+0000) 2022-09-10T05:35:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:53.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:53.018242+0000) 2022-09-10T05:35:54.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:53.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:53.610643+0000) 2022-09-10T05:35:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:53 smithi104 ceph-mon[97044]: pgmap v2139: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:54.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:54.018484+0000) 2022-09-10T05:35:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:53 smithi110 ceph-mon[97633]: pgmap v2139: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:35:55.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:54.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:54.610836+0000) 2022-09-10T05:35:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:55.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:55.018721+0000) 2022-09-10T05:35:56.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:55.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:55.611082+0000) 2022-09-10T05:35:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:56.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:56.018916+0000) 2022-09-10T05:35:56.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:35:56.177+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:35:56.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:56 smithi110 ceph-mon[97633]: pgmap v2140: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:56 smithi104 ceph-mon[97044]: pgmap v2140: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:35:57.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:56.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:56.611332+0000) 2022-09-10T05:35:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:57.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:57.019078+0000) 2022-09-10T05:35:58.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:57.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:57.611564+0000) 2022-09-10T05:35:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:58.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:58.019238+0000) 2022-09-10T05:35:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:35:58 smithi110 ceph-mon[97633]: pgmap v2141: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:35:58 smithi104 ceph-mon[97044]: pgmap v2141: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:35:59.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:58.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:58.611801+0000) 2022-09-10T05:35:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:35:59.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:59.019422+0000) 2022-09-10T05:36:00.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:35:59.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:35:59.612033+0000) 2022-09-10T05:36:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:00.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:00.019663+0000) 2022-09-10T05:36:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:00 smithi110 ceph-mon[97633]: pgmap v2142: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:00 smithi104 ceph-mon[97044]: pgmap v2142: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:01.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:00.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:00.612274+0000) 2022-09-10T05:36:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:01.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:01.019833+0000) 2022-09-10T05:36:02.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:01.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:01.612501+0000) 2022-09-10T05:36:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:02.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:02.020047+0000) 2022-09-10T05:36:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:02 smithi110 ceph-mon[97633]: pgmap v2143: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:02 smithi104 ceph-mon[97044]: pgmap v2143: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:03.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:02.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:02.612718+0000) 2022-09-10T05:36:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:03.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:03.020284+0000) 2022-09-10T05:36:04.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:03.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:03.612927+0000) 2022-09-10T05:36:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:03 smithi104 ceph-mon[97044]: pgmap v2144: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:04.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:04.020411+0000) 2022-09-10T05:36:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:03 smithi110 ceph-mon[97633]: pgmap v2144: 129 pgs: 129 active+clean; 7.9 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:05.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:04.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:04.613185+0000) 2022-09-10T05:36:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:05.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:05.020635+0000) 2022-09-10T05:36:06.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:05.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:05.613399+0000) 2022-09-10T05:36:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:06.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:06.020887+0000) 2022-09-10T05:36:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:06 smithi110 ceph-mon[97633]: pgmap v2145: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:36:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:06 smithi104 ceph-mon[97044]: pgmap v2145: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:36:07.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:06.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:06.613679+0000) 2022-09-10T05:36:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:07.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:07.021164+0000) 2022-09-10T05:36:08.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:07.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:07.613906+0000) 2022-09-10T05:36:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:08.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:08.021392+0000) 2022-09-10T05:36:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:08 smithi110 ceph-mon[97633]: pgmap v2146: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:36:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:08 smithi104 ceph-mon[97044]: pgmap v2146: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:36:09.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:08.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:08.614078+0000) 2022-09-10T05:36:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:09.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:09.021587+0000) 2022-09-10T05:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:09 smithi110 ceph-mon[97633]: osdmap e218: 12 total, 12 up, 12 in 2022-09-10T05:36:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:09 smithi104 ceph-mon[97044]: osdmap e218: 12 total, 12 up, 12 in 2022-09-10T05:36:10.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:09.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:09.614286+0000) 2022-09-10T05:36:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:10.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:10.021835+0000) 2022-09-10T05:36:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:10 smithi110 ceph-mon[97633]: pgmap v2148: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:36:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:10 smithi104 ceph-mon[97044]: pgmap v2148: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:36:11.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:10.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:10.614428+0000) 2022-09-10T05:36:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:11.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:11.022050+0000) 2022-09-10T05:36:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:36:11.186+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:36:12.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:11.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:11.614663+0000) 2022-09-10T05:36:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:12.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:12.022315+0000) 2022-09-10T05:36:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:12 smithi110 ceph-mon[97633]: pgmap v2149: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:36:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:12 smithi104 ceph-mon[97044]: pgmap v2149: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:36:13.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:12.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:12.614883+0000) 2022-09-10T05:36:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:13.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:13.022516+0000) 2022-09-10T05:36:14.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:13.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:13.615118+0000) 2022-09-10T05:36:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:13 smithi104 ceph-mon[97044]: osdmap e219: 12 total, 12 up, 12 in 2022-09-10T05:36:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:13 smithi104 ceph-mon[97044]: pgmap v2151: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:36:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:14.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:14.022779+0000) 2022-09-10T05:36:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:13 smithi110 ceph-mon[97633]: osdmap e219: 12 total, 12 up, 12 in 2022-09-10T05:36:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:13 smithi110 ceph-mon[97633]: pgmap v2151: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:36:15.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:14.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:14.615328+0000) 2022-09-10T05:36:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:15.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:15.022968+0000) 2022-09-10T05:36:16.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:15.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:15.615492+0000) 2022-09-10T05:36:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:16.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:16.023194+0000) 2022-09-10T05:36:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:16 smithi110 ceph-mon[97633]: pgmap v2152: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:36:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:16 smithi104 ceph-mon[97044]: pgmap v2152: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:36:17.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:16.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:16.615678+0000) 2022-09-10T05:36:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:17.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:17.023422+0000) 2022-09-10T05:36:18.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:17.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:17.615916+0000) 2022-09-10T05:36:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:18.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:18.023636+0000) 2022-09-10T05:36:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:18 smithi110 ceph-mon[97633]: pgmap v2153: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:36:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:18 smithi104 ceph-mon[97044]: pgmap v2153: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T05:36:19.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:18.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:18.616164+0000) 2022-09-10T05:36:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:19.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:19.023833+0000) 2022-09-10T05:36:20.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:19.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:19.616432+0000) 2022-09-10T05:36:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:20.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:20.024034+0000) 2022-09-10T05:36:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:20 smithi110 ceph-mon[97633]: pgmap v2154: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:36:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:20 smithi104 ceph-mon[97044]: pgmap v2154: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:36:21.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:20.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:20.616627+0000) 2022-09-10T05:36:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:21.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:21.024264+0000) 2022-09-10T05:36:22.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:21.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:21.616843+0000) 2022-09-10T05:36:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:22.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:22.024477+0000) 2022-09-10T05:36:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:22 smithi110 ceph-mon[97633]: pgmap v2155: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:36:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:22 smithi104 ceph-mon[97044]: pgmap v2155: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:36:23.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:22.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:22.617127+0000) 2022-09-10T05:36:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:23.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:23.024716+0000) 2022-09-10T05:36:24.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:23.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:23.617362+0000) 2022-09-10T05:36:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:23 smithi104 ceph-mon[97044]: pgmap v2156: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:36:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:24.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:24.024999+0000) 2022-09-10T05:36:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:23 smithi110 ceph-mon[97633]: pgmap v2156: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:36:25.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:24.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:24.617572+0000) 2022-09-10T05:36:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:25.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:25.025249+0000) 2022-09-10T05:36:26.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:25.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:25.617776+0000) 2022-09-10T05:36:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:26.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:26.025438+0000) 2022-09-10T05:36:26.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:36:26.198+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:36:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:26 smithi110 ceph-mon[97633]: pgmap v2157: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:36:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:26 smithi104 ceph-mon[97044]: pgmap v2157: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:36:26.869 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:26.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:26.617972+0000) 2022-09-10T05:36:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:27.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:27.025670+0000) 2022-09-10T05:36:28.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:27.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:27.618203+0000) 2022-09-10T05:36:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:28.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:28.025916+0000) 2022-09-10T05:36:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:28 smithi110 ceph-mon[97633]: pgmap v2158: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:28 smithi104 ceph-mon[97044]: pgmap v2158: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:29.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:28.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:28.618462+0000) 2022-09-10T05:36:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:29.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:29.026207+0000) 2022-09-10T05:36:30.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:29.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:29.618753+0000) 2022-09-10T05:36:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:30.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:30.026458+0000) 2022-09-10T05:36:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:30 smithi110 ceph-mon[97633]: pgmap v2159: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:30 smithi104 ceph-mon[97044]: pgmap v2159: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:31.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:30.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:30.618998+0000) 2022-09-10T05:36:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:31.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:31.026682+0000) 2022-09-10T05:36:32.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:31.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:31.619208+0000) 2022-09-10T05:36:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:32.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:32.026926+0000) 2022-09-10T05:36:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:32 smithi110 ceph-mon[97633]: pgmap v2160: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:32 smithi104 ceph-mon[97044]: pgmap v2160: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:33.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:32.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:32.619450+0000) 2022-09-10T05:36:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:33.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:33.027196+0000) 2022-09-10T05:36:34.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:33.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:33.619742+0000) 2022-09-10T05:36:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:33 smithi104 ceph-mon[97044]: pgmap v2161: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:34.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:34.027474+0000) 2022-09-10T05:36:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:33 smithi110 ceph-mon[97633]: pgmap v2161: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:35.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:34.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:34.619985+0000) 2022-09-10T05:36:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:35.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:35.027658+0000) 2022-09-10T05:36:36.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:35.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:35.620267+0000) 2022-09-10T05:36:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:36.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:36.027920+0000) 2022-09-10T05:36:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:36 smithi110 ceph-mon[97633]: pgmap v2162: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:36 smithi104 ceph-mon[97044]: pgmap v2162: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:37.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:36.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:36.620567+0000) 2022-09-10T05:36:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:37.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:37.028142+0000) 2022-09-10T05:36:38.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:37.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:37.620776+0000) 2022-09-10T05:36:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:38.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:38.028382+0000) 2022-09-10T05:36:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:38 smithi110 ceph-mon[97633]: pgmap v2163: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:38 smithi104 ceph-mon[97044]: pgmap v2163: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:39.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:38.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:38.620994+0000) 2022-09-10T05:36:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:39.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:39.028600+0000) 2022-09-10T05:36:40.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:39.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:39.621275+0000) 2022-09-10T05:36:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:40.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:40.028850+0000) 2022-09-10T05:36:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:40 smithi110 ceph-mon[97633]: pgmap v2164: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:40 smithi104 ceph-mon[97044]: pgmap v2164: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:41.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:40.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:40.621476+0000) 2022-09-10T05:36:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:36:41.204+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:36:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:41.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:41.029066+0000) 2022-09-10T05:36:42.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:41.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:41.621702+0000) 2022-09-10T05:36:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:42.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:42.029312+0000) 2022-09-10T05:36:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:42 smithi110 ceph-mon[97633]: pgmap v2165: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:42 smithi104 ceph-mon[97044]: pgmap v2165: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:43.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:42.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:42.621906+0000) 2022-09-10T05:36:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:43.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:43.029522+0000) 2022-09-10T05:36:44.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:43.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:43.622171+0000) 2022-09-10T05:36:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:43 smithi104 ceph-mon[97044]: pgmap v2166: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:36:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:36:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:43 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:36:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:43 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:36:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:44.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:44.029717+0000) 2022-09-10T05:36:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:43 smithi110 ceph-mon[97633]: pgmap v2166: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:36:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:36:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:43 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:36:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:43 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:36:45.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:44.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:44.622388+0000) 2022-09-10T05:36:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:45.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:45.029936+0000) 2022-09-10T05:36:46.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:45.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:45.622657+0000) 2022-09-10T05:36:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:46.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:46.030153+0000) 2022-09-10T05:36:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:46 smithi110 ceph-mon[97633]: pgmap v2167: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:46 smithi104 ceph-mon[97044]: pgmap v2167: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:47.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:46.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:46.622898+0000) 2022-09-10T05:36:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:47.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:47.030388+0000) 2022-09-10T05:36:48.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:47.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:47.623194+0000) 2022-09-10T05:36:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:48.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:48.030606+0000) 2022-09-10T05:36:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:48 smithi110 ceph-mon[97633]: pgmap v2168: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:48 smithi104 ceph-mon[97044]: pgmap v2168: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:49.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:48.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:48.623366+0000) 2022-09-10T05:36:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:49.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:49.030849+0000) 2022-09-10T05:36:50.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:49.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:49.623585+0000) 2022-09-10T05:36:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:50.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:50.030994+0000) 2022-09-10T05:36:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:50 smithi110 ceph-mon[97633]: pgmap v2169: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:50 smithi104 ceph-mon[97044]: pgmap v2169: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:51.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:50.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:50.623785+0000) 2022-09-10T05:36:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:51.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:51.031207+0000) 2022-09-10T05:36:52.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:51.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:51.623972+0000) 2022-09-10T05:36:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:52.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:52.031406+0000) 2022-09-10T05:36:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:52 smithi110 ceph-mon[97633]: pgmap v2170: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:52 smithi104 ceph-mon[97044]: pgmap v2170: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:53.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:52.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:52.624198+0000) 2022-09-10T05:36:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:53.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:53.031577+0000) 2022-09-10T05:36:54.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:53.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:53.624365+0000) 2022-09-10T05:36:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:53 smithi104 ceph-mon[97044]: pgmap v2171: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:54.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:54.031815+0000) 2022-09-10T05:36:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:53 smithi110 ceph-mon[97633]: pgmap v2171: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:55.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:54.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:54.624556+0000) 2022-09-10T05:36:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:55.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:55.032041+0000) 2022-09-10T05:36:56.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:55.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:55.624806+0000) 2022-09-10T05:36:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:56.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:56.032228+0000) 2022-09-10T05:36:56.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:36:56.219+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:36:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:56 smithi110 ceph-mon[97633]: pgmap v2172: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:56 smithi104 ceph-mon[97044]: pgmap v2172: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:36:57.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:56.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:56.625043+0000) 2022-09-10T05:36:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:57.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:57.032510+0000) 2022-09-10T05:36:58.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:57.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:57.625300+0000) 2022-09-10T05:36:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:58.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:58.032692+0000) 2022-09-10T05:36:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:36:58 smithi110 ceph-mon[97633]: pgmap v2173: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:36:58 smithi104 ceph-mon[97044]: pgmap v2173: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:36:59.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:58.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:58.625482+0000) 2022-09-10T05:36:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:36:59.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:59.032896+0000) 2022-09-10T05:37:00.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:36:59.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:36:59.625695+0000) 2022-09-10T05:37:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:00.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:00.033076+0000) 2022-09-10T05:37:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:00 smithi110 ceph-mon[97633]: pgmap v2174: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:00 smithi104 ceph-mon[97044]: pgmap v2174: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:01.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:00.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:00.625919+0000) 2022-09-10T05:37:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:01.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:01.033322+0000) 2022-09-10T05:37:02.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:01.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:01.626136+0000) 2022-09-10T05:37:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:02.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:02.033532+0000) 2022-09-10T05:37:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:02 smithi110 ceph-mon[97633]: pgmap v2175: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:02 smithi104 ceph-mon[97044]: pgmap v2175: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:03.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:02.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:02.626375+0000) 2022-09-10T05:37:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:03.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:03.033697+0000) 2022-09-10T05:37:04.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:03.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:03.626567+0000) 2022-09-10T05:37:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:04 smithi104 ceph-mon[97044]: pgmap v2176: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:04.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:04.033870+0000) 2022-09-10T05:37:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:04 smithi110 ceph-mon[97633]: pgmap v2176: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:05.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:04.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:04.626804+0000) 2022-09-10T05:37:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:05.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:05.034074+0000) 2022-09-10T05:37:06.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:05.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:05.627037+0000) 2022-09-10T05:37:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:06.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:06.034214+0000) 2022-09-10T05:37:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:06 smithi110 ceph-mon[97633]: pgmap v2177: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:37:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:06 smithi104 ceph-mon[97044]: pgmap v2177: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:37:07.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:06.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:06.627283+0000) 2022-09-10T05:37:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:07.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:07.034401+0000) 2022-09-10T05:37:08.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:07.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:07.627536+0000) 2022-09-10T05:37:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:08.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:08.034551+0000) 2022-09-10T05:37:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:08 smithi110 ceph-mon[97633]: pgmap v2178: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:37:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:08 smithi104 ceph-mon[97044]: pgmap v2178: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:37:09.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:08.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:08.627761+0000) 2022-09-10T05:37:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:09.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:09.034830+0000) 2022-09-10T05:37:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:09 smithi110 ceph-mon[97633]: osdmap e220: 12 total, 12 up, 12 in 2022-09-10T05:37:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:09 smithi104 ceph-mon[97044]: osdmap e220: 12 total, 12 up, 12 in 2022-09-10T05:37:10.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:09.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:09.628015+0000) 2022-09-10T05:37:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:10.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:10.035032+0000) 2022-09-10T05:37:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:10 smithi110 ceph-mon[97633]: pgmap v2180: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:37:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:10 smithi104 ceph-mon[97044]: pgmap v2180: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:37:11.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:10.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:10.628221+0000) 2022-09-10T05:37:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:11.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:11.035257+0000) 2022-09-10T05:37:11.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:37:11.233+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:37:12.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:11.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:11.628437+0000) 2022-09-10T05:37:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:12.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:12.035501+0000) 2022-09-10T05:37:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:12 smithi110 ceph-mon[97633]: pgmap v2181: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:37:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:12 smithi104 ceph-mon[97044]: pgmap v2181: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:37:13.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:12.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:12.628622+0000) 2022-09-10T05:37:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:13.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:13.035739+0000) 2022-09-10T05:37:14.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:13.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:13.628886+0000) 2022-09-10T05:37:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:13 smithi104 ceph-mon[97044]: osdmap e221: 12 total, 12 up, 12 in 2022-09-10T05:37:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:13 smithi104 ceph-mon[97044]: pgmap v2183: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:37:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:14.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:14.035995+0000) 2022-09-10T05:37:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:13 smithi110 ceph-mon[97633]: osdmap e221: 12 total, 12 up, 12 in 2022-09-10T05:37:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:13 smithi110 ceph-mon[97633]: pgmap v2183: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:37:14.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:14.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:14.629132+0000) 2022-09-10T05:37:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:15.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:15.036254+0000) 2022-09-10T05:37:16.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:15.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:15.629373+0000) 2022-09-10T05:37:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:16.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:16.036434+0000) 2022-09-10T05:37:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:16 smithi110 ceph-mon[97633]: pgmap v2184: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:37:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:16 smithi104 ceph-mon[97044]: pgmap v2184: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:37:17.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:16.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:16.629608+0000) 2022-09-10T05:37:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:17.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:17.036633+0000) 2022-09-10T05:37:18.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:17.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:17.629829+0000) 2022-09-10T05:37:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:18.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:18.036772+0000) 2022-09-10T05:37:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:18 smithi110 ceph-mon[97633]: pgmap v2185: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:37:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:18 smithi104 ceph-mon[97044]: pgmap v2185: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:37:19.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:18.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:18.630057+0000) 2022-09-10T05:37:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:19.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:19.037036+0000) 2022-09-10T05:37:20.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:19.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:19.630285+0000) 2022-09-10T05:37:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:20.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:20.037213+0000) 2022-09-10T05:37:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:20 smithi110 ceph-mon[97633]: pgmap v2186: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:37:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:20 smithi104 ceph-mon[97044]: pgmap v2186: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:37:21.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:20.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:20.630467+0000) 2022-09-10T05:37:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:21.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:21.037370+0000) 2022-09-10T05:37:22.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:21.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:21.630681+0000) 2022-09-10T05:37:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:22.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:22.037608+0000) 2022-09-10T05:37:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:22 smithi110 ceph-mon[97633]: pgmap v2187: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:37:22.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:22 smithi104 ceph-mon[97044]: pgmap v2187: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:37:23.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:22.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:22.630941+0000) 2022-09-10T05:37:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:23.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:23.037822+0000) 2022-09-10T05:37:24.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:23.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:23.631190+0000) 2022-09-10T05:37:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:24 smithi104 ceph-mon[97044]: pgmap v2188: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:37:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:24.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:24.038068+0000) 2022-09-10T05:37:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:24 smithi110 ceph-mon[97633]: pgmap v2188: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:37:25.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:24.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:24.631411+0000) 2022-09-10T05:37:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:25.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:25.038319+0000) 2022-09-10T05:37:26.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:25.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:25.631684+0000) 2022-09-10T05:37:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:26.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:26.038568+0000) 2022-09-10T05:37:26.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:37:26.244+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:37:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:26 smithi110 ceph-mon[97633]: pgmap v2189: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:37:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:26 smithi104 ceph-mon[97044]: pgmap v2189: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:37:27.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:26.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:26.631888+0000) 2022-09-10T05:37:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:27.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:27.038793+0000) 2022-09-10T05:37:28.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:27.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:27.632130+0000) 2022-09-10T05:37:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:28.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:28.039025+0000) 2022-09-10T05:37:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:28 smithi110 ceph-mon[97633]: pgmap v2190: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:28 smithi104 ceph-mon[97044]: pgmap v2190: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:29.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:28.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:28.632371+0000) 2022-09-10T05:37:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:29.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:29.039236+0000) 2022-09-10T05:37:30.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:29.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:29.632653+0000) 2022-09-10T05:37:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:30.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:30.039417+0000) 2022-09-10T05:37:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:30 smithi110 ceph-mon[97633]: pgmap v2191: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:30 smithi104 ceph-mon[97044]: pgmap v2191: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:31.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:30.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:30.632902+0000) 2022-09-10T05:37:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:31.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:31.039555+0000) 2022-09-10T05:37:32.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:31.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:31.633131+0000) 2022-09-10T05:37:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:32.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:32.039761+0000) 2022-09-10T05:37:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:32 smithi110 ceph-mon[97633]: pgmap v2192: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:32 smithi104 ceph-mon[97044]: pgmap v2192: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:33.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:32.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:32.633343+0000) 2022-09-10T05:37:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:33.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:33.039948+0000) 2022-09-10T05:37:34.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:33.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:33.633568+0000) 2022-09-10T05:37:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:34 smithi104 ceph-mon[97044]: pgmap v2193: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:34.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:34.040191+0000) 2022-09-10T05:37:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:34 smithi110 ceph-mon[97633]: pgmap v2193: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:35.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:34.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:34.633823+0000) 2022-09-10T05:37:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:35.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:35.040434+0000) 2022-09-10T05:37:36.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:35.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:35.634066+0000) 2022-09-10T05:37:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:36.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:36.040640+0000) 2022-09-10T05:37:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:36 smithi110 ceph-mon[97633]: pgmap v2194: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:36 smithi104 ceph-mon[97044]: pgmap v2194: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:37.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:36.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:36.634302+0000) 2022-09-10T05:37:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:37.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:37.040870+0000) 2022-09-10T05:37:38.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:37.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:37.634446+0000) 2022-09-10T05:37:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:38.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:38.041062+0000) 2022-09-10T05:37:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:38 smithi110 ceph-mon[97633]: pgmap v2195: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:38 smithi104 ceph-mon[97044]: pgmap v2195: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:39.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:38.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:38.634693+0000) 2022-09-10T05:37:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:39.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:39.041268+0000) 2022-09-10T05:37:40.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:39.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:39.634976+0000) 2022-09-10T05:37:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:40.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:40.041485+0000) 2022-09-10T05:37:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:40 smithi110 ceph-mon[97633]: pgmap v2196: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:40 smithi104 ceph-mon[97044]: pgmap v2196: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:41.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:40.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:40.635186+0000) 2022-09-10T05:37:41.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:37:41.250+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:37:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:41.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:41.041707+0000) 2022-09-10T05:37:42.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:41.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:41.635402+0000) 2022-09-10T05:37:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:42.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:42.041952+0000) 2022-09-10T05:37:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:42 smithi110 ceph-mon[97633]: pgmap v2197: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:42 smithi104 ceph-mon[97044]: pgmap v2197: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:43.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:42.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:42.635635+0000) 2022-09-10T05:37:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:43.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:43.042156+0000) 2022-09-10T05:37:44.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:43.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:43.635875+0000) 2022-09-10T05:37:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:44 smithi104 ceph-mon[97044]: pgmap v2198: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:37:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:37:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:37:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:37:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:44.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:44.042342+0000) 2022-09-10T05:37:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:44 smithi110 ceph-mon[97633]: pgmap v2198: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:37:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:37:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:37:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:37:45.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:44.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:44.636131+0000) 2022-09-10T05:37:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:45.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:45.042520+0000) 2022-09-10T05:37:46.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:45.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:45.636338+0000) 2022-09-10T05:37:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:46.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:46.042782+0000) 2022-09-10T05:37:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:46 smithi110 ceph-mon[97633]: pgmap v2199: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:46 smithi104 ceph-mon[97044]: pgmap v2199: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:47.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:46.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:46.636603+0000) 2022-09-10T05:37:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:47.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:47.043049+0000) 2022-09-10T05:37:48.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:47.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:47.636874+0000) 2022-09-10T05:37:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:48.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:48.043295+0000) 2022-09-10T05:37:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:48 smithi110 ceph-mon[97633]: pgmap v2200: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:48 smithi104 ceph-mon[97044]: pgmap v2200: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:49.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:48.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:48.637086+0000) 2022-09-10T05:37:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:49.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:49.043527+0000) 2022-09-10T05:37:50.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:49.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:49.637342+0000) 2022-09-10T05:37:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:50.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:50.043663+0000) 2022-09-10T05:37:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:50 smithi110 ceph-mon[97633]: pgmap v2201: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:50 smithi104 ceph-mon[97044]: pgmap v2201: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:51.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:50.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:50.637523+0000) 2022-09-10T05:37:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:51.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:51.043871+0000) 2022-09-10T05:37:52.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:51.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:51.637700+0000) 2022-09-10T05:37:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:52.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:52.044135+0000) 2022-09-10T05:37:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:52 smithi110 ceph-mon[97633]: pgmap v2202: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:52 smithi104 ceph-mon[97044]: pgmap v2202: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:53.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:52.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:52.637932+0000) 2022-09-10T05:37:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:53.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:53.044286+0000) 2022-09-10T05:37:54.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:53.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:53.638235+0000) 2022-09-10T05:37:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:54 smithi104 ceph-mon[97044]: pgmap v2203: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:54.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:54.044488+0000) 2022-09-10T05:37:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:54 smithi110 ceph-mon[97633]: pgmap v2203: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:55.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:54.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:54.638397+0000) 2022-09-10T05:37:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:55.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:55.044696+0000) 2022-09-10T05:37:56.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:55.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:55.638526+0000) 2022-09-10T05:37:56.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:37:56.259+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:37:56.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:56.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:56.044887+0000) 2022-09-10T05:37:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:56 smithi110 ceph-mon[97633]: pgmap v2204: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:56 smithi104 ceph-mon[97044]: pgmap v2204: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:37:57.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:56.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:56.638730+0000) 2022-09-10T05:37:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:57.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:57.045179+0000) 2022-09-10T05:37:58.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:57.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:57.638987+0000) 2022-09-10T05:37:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:58.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:58.045399+0000) 2022-09-10T05:37:58.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:37:58 smithi110 ceph-mon[97633]: pgmap v2205: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:37:58 smithi104 ceph-mon[97044]: pgmap v2205: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:37:59.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:58.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:58.639268+0000) 2022-09-10T05:37:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:37:59.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:59.045633+0000) 2022-09-10T05:38:00.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:37:59.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:37:59.639527+0000) 2022-09-10T05:38:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:00.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:00.045907+0000) 2022-09-10T05:38:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:00 smithi110 ceph-mon[97633]: pgmap v2206: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:00 smithi104 ceph-mon[97044]: pgmap v2206: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:01.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:00.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:00.639704+0000) 2022-09-10T05:38:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:01.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:01.046182+0000) 2022-09-10T05:38:02.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:01.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:01.639884+0000) 2022-09-10T05:38:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:02.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:02.046394+0000) 2022-09-10T05:38:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:02 smithi110 ceph-mon[97633]: pgmap v2207: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:02 smithi104 ceph-mon[97044]: pgmap v2207: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:03.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:02.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:02.640064+0000) 2022-09-10T05:38:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:03.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:03.046615+0000) 2022-09-10T05:38:04.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:03.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:03.640294+0000) 2022-09-10T05:38:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:04 smithi104 ceph-mon[97044]: pgmap v2208: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:04.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:04.046896+0000) 2022-09-10T05:38:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:04 smithi110 ceph-mon[97633]: pgmap v2208: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:05.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:04.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:04.640541+0000) 2022-09-10T05:38:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:05.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:05.047190+0000) 2022-09-10T05:38:06.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:05.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:05.640811+0000) 2022-09-10T05:38:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:06.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:06.047422+0000) 2022-09-10T05:38:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:06 smithi110 ceph-mon[97633]: pgmap v2209: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:38:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:06 smithi104 ceph-mon[97044]: pgmap v2209: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:38:07.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:06.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:06.640983+0000) 2022-09-10T05:38:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:07.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:07.047616+0000) 2022-09-10T05:38:08.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:07.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:07.641189+0000) 2022-09-10T05:38:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:08.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:08.047832+0000) 2022-09-10T05:38:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:08 smithi110 ceph-mon[97633]: pgmap v2210: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:38:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:08 smithi104 ceph-mon[97044]: pgmap v2210: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:38:09.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:08.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:08.641468+0000) 2022-09-10T05:38:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:09.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:09.048064+0000) 2022-09-10T05:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:09 smithi110 ceph-mon[97633]: osdmap e222: 12 total, 12 up, 12 in 2022-09-10T05:38:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:09 smithi104 ceph-mon[97044]: osdmap e222: 12 total, 12 up, 12 in 2022-09-10T05:38:10.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:09.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:09.641660+0000) 2022-09-10T05:38:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:10.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:10.048244+0000) 2022-09-10T05:38:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:10 smithi110 ceph-mon[97633]: pgmap v2212: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:38:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:10 smithi104 ceph-mon[97044]: pgmap v2212: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:38:11.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:10.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:10.641897+0000) 2022-09-10T05:38:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:38:11.268+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:38:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:11.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:11.048431+0000) 2022-09-10T05:38:12.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:11.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:11.642163+0000) 2022-09-10T05:38:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:12.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:12.048692+0000) 2022-09-10T05:38:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:12 smithi110 ceph-mon[97633]: pgmap v2213: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:38:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:12 smithi104 ceph-mon[97044]: pgmap v2213: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:38:13.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:12.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:12.642377+0000) 2022-09-10T05:38:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:13.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:13.048903+0000) 2022-09-10T05:38:14.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:13.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:13.642562+0000) 2022-09-10T05:38:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:14 smithi104 ceph-mon[97044]: osdmap e223: 12 total, 12 up, 12 in 2022-09-10T05:38:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:14 smithi104 ceph-mon[97044]: pgmap v2215: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:38:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:14.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:14.049166+0000) 2022-09-10T05:38:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:14 smithi110 ceph-mon[97633]: osdmap e223: 12 total, 12 up, 12 in 2022-09-10T05:38:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:14 smithi110 ceph-mon[97633]: pgmap v2215: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:38:15.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:14.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:14.642774+0000) 2022-09-10T05:38:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:15.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:15.049355+0000) 2022-09-10T05:38:16.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:15.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:15.643028+0000) 2022-09-10T05:38:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:16.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:16.049522+0000) 2022-09-10T05:38:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:16 smithi110 ceph-mon[97633]: pgmap v2216: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:38:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:16 smithi104 ceph-mon[97044]: pgmap v2216: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:38:17.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:16.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:16.643291+0000) 2022-09-10T05:38:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:17.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:17.049773+0000) 2022-09-10T05:38:18.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:17.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:17.643529+0000) 2022-09-10T05:38:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:18.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:18.050015+0000) 2022-09-10T05:38:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:18 smithi110 ceph-mon[97633]: pgmap v2217: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:38:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:18 smithi104 ceph-mon[97044]: pgmap v2217: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:38:19.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:18.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:18.643789+0000) 2022-09-10T05:38:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:19.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:19.050204+0000) 2022-09-10T05:38:20.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:19.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:19.644025+0000) 2022-09-10T05:38:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:20.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:20.050469+0000) 2022-09-10T05:38:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:20 smithi110 ceph-mon[97633]: pgmap v2218: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:38:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:20 smithi104 ceph-mon[97044]: pgmap v2218: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:38:21.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:20.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:20.644288+0000) 2022-09-10T05:38:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:21.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:21.050664+0000) 2022-09-10T05:38:22.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:21.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:21.644584+0000) 2022-09-10T05:38:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:22.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:22.050917+0000) 2022-09-10T05:38:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:22 smithi110 ceph-mon[97633]: pgmap v2219: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:38:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:22 smithi104 ceph-mon[97044]: pgmap v2219: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:38:23.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:22.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:22.644826+0000) 2022-09-10T05:38:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:23.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:23.051183+0000) 2022-09-10T05:38:24.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:23.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:23.645070+0000) 2022-09-10T05:38:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:24 smithi104 ceph-mon[97044]: pgmap v2220: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:38:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:24.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:24.051436+0000) 2022-09-10T05:38:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:24 smithi110 ceph-mon[97633]: pgmap v2220: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:38:25.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:24.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:24.645375+0000) 2022-09-10T05:38:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:25.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:25.051676+0000) 2022-09-10T05:38:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:25.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:25.645635+0000) 2022-09-10T05:38:26.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:38:26.277+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:38:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:26.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:26.051856+0000) 2022-09-10T05:38:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:26 smithi110 ceph-mon[97633]: pgmap v2221: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:38:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:26 smithi104 ceph-mon[97044]: pgmap v2221: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:38:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:26.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:26.645840+0000) 2022-09-10T05:38:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:27.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:27.052074+0000) 2022-09-10T05:38:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:27.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:27.646070+0000) 2022-09-10T05:38:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:28.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:28.052347+0000) 2022-09-10T05:38:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:28 smithi110 ceph-mon[97633]: pgmap v2222: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:28 smithi104 ceph-mon[97044]: pgmap v2222: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:28.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:28.646340+0000) 2022-09-10T05:38:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:29.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:29.052546+0000) 2022-09-10T05:38:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:29.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:29.646584+0000) 2022-09-10T05:38:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:30.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:30.052795+0000) 2022-09-10T05:38:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:30 smithi110 ceph-mon[97633]: pgmap v2223: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:30 smithi104 ceph-mon[97044]: pgmap v2223: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:30.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:30.646805+0000) 2022-09-10T05:38:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:31.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:31.053030+0000) 2022-09-10T05:38:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:31.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:31.646965+0000) 2022-09-10T05:38:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:32.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:32.053247+0000) 2022-09-10T05:38:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:32 smithi110 ceph-mon[97633]: pgmap v2224: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:32 smithi104 ceph-mon[97044]: pgmap v2224: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:32.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:32.647187+0000) 2022-09-10T05:38:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:33.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:33.053403+0000) 2022-09-10T05:38:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:33.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:33.647447+0000) 2022-09-10T05:38:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:34 smithi104 ceph-mon[97044]: pgmap v2225: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:34 smithi110 ceph-mon[97633]: pgmap v2225: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:34.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:34.053675+0000) 2022-09-10T05:38:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:34.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:34.647715+0000) 2022-09-10T05:38:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:35.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:35.053996+0000) 2022-09-10T05:38:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:35.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:35.647993+0000) 2022-09-10T05:38:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:36 smithi110 ceph-mon[97633]: pgmap v2226: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:36 smithi104 ceph-mon[97044]: pgmap v2226: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:36.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:36.054155+0000) 2022-09-10T05:38:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:36.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:36.648271+0000) 2022-09-10T05:38:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:37.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:37.054289+0000) 2022-09-10T05:38:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:37.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:37.648523+0000) 2022-09-10T05:38:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:38 smithi110 ceph-mon[97633]: pgmap v2227: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:38 smithi104 ceph-mon[97044]: pgmap v2227: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:38.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:38.054456+0000) 2022-09-10T05:38:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:38.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:38.648805+0000) 2022-09-10T05:38:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:39.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:39.054625+0000) 2022-09-10T05:38:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:39.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:39.649085+0000) 2022-09-10T05:38:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:40 smithi110 ceph-mon[97633]: pgmap v2228: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:40 smithi104 ceph-mon[97044]: pgmap v2228: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:40.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:40.054841+0000) 2022-09-10T05:38:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:40.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:40.649323+0000) 2022-09-10T05:38:41.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:38:41.291+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:38:41.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:41.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:41.054984+0000) 2022-09-10T05:38:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:41.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:41.649563+0000) 2022-09-10T05:38:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:42 smithi110 ceph-mon[97633]: pgmap v2229: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:42 smithi104 ceph-mon[97044]: pgmap v2229: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:42.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:42.055203+0000) 2022-09-10T05:38:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:42.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:42.649701+0000) 2022-09-10T05:38:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:43.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:43.055380+0000) 2022-09-10T05:38:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:43.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:43.649964+0000) 2022-09-10T05:38:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:44 smithi104 ceph-mon[97044]: pgmap v2230: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:38:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:38:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:38:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:38:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:44 smithi110 ceph-mon[97633]: pgmap v2230: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:38:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:38:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:38:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:38:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:44.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:44.055578+0000) 2022-09-10T05:38:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:44.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:44.650221+0000) 2022-09-10T05:38:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:45.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:45.055760+0000) 2022-09-10T05:38:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:45.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:45.650419+0000) 2022-09-10T05:38:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:46 smithi110 ceph-mon[97633]: pgmap v2231: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:46 smithi104 ceph-mon[97044]: pgmap v2231: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:46.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:46.055943+0000) 2022-09-10T05:38:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:46.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:46.650600+0000) 2022-09-10T05:38:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:47.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:47.056148+0000) 2022-09-10T05:38:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:47.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:47.650838+0000) 2022-09-10T05:38:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:48 smithi110 ceph-mon[97633]: pgmap v2232: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:48 smithi104 ceph-mon[97044]: pgmap v2232: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:48.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:48.056348+0000) 2022-09-10T05:38:49.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:48.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:48.651029+0000) 2022-09-10T05:38:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:49.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:49.056561+0000) 2022-09-10T05:38:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:49.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:49.651251+0000) 2022-09-10T05:38:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:50 smithi110 ceph-mon[97633]: pgmap v2233: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:50 smithi104 ceph-mon[97044]: pgmap v2233: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:50.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:50.056728+0000) 2022-09-10T05:38:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:50.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:50.651493+0000) 2022-09-10T05:38:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:51.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:51.056944+0000) 2022-09-10T05:38:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:51.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:51.651680+0000) 2022-09-10T05:38:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:52 smithi110 ceph-mon[97633]: pgmap v2234: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:52 smithi104 ceph-mon[97044]: pgmap v2234: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:52.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:52.057169+0000) 2022-09-10T05:38:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:52.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:52.651868+0000) 2022-09-10T05:38:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:53.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:53.057379+0000) 2022-09-10T05:38:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:53.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:53.652191+0000) 2022-09-10T05:38:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:54 smithi104 ceph-mon[97044]: pgmap v2235: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:54 smithi110 ceph-mon[97633]: pgmap v2235: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:54.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:54.057597+0000) 2022-09-10T05:38:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:54.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:54.652358+0000) 2022-09-10T05:38:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:55.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:55.057726+0000) 2022-09-10T05:38:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:55.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:55.652584+0000) 2022-09-10T05:38:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:56 smithi110 ceph-mon[97633]: pgmap v2236: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:56 smithi104 ceph-mon[97044]: pgmap v2236: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:38:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:56.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:56.057940+0000) 2022-09-10T05:38:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:38:56.304+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:38:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:56.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:56.652810+0000) 2022-09-10T05:38:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:57.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:57.058126+0000) 2022-09-10T05:38:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:57.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:57.653036+0000) 2022-09-10T05:38:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:38:58 smithi110 ceph-mon[97633]: pgmap v2237: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:38:58 smithi104 ceph-mon[97044]: pgmap v2237: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:38:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:58.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:58.058306+0000) 2022-09-10T05:38:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:58.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:58.653246+0000) 2022-09-10T05:38:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:38:59.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:59.058498+0000) 2022-09-10T05:39:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:38:59.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:38:59.653428+0000) 2022-09-10T05:39:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:00 smithi110 ceph-mon[97633]: pgmap v2238: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:00 smithi104 ceph-mon[97044]: pgmap v2238: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:00.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:00.058655+0000) 2022-09-10T05:39:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:00.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:00.653637+0000) 2022-09-10T05:39:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:01.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:01.058816+0000) 2022-09-10T05:39:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:01.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:01.653814+0000) 2022-09-10T05:39:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:02 smithi110 ceph-mon[97633]: pgmap v2239: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:02 smithi104 ceph-mon[97044]: pgmap v2239: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:02.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:02.058994+0000) 2022-09-10T05:39:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:02.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:02.654013+0000) 2022-09-10T05:39:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:03.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:03.059197+0000) 2022-09-10T05:39:04.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:03.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:03.654237+0000) 2022-09-10T05:39:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:04 smithi104 ceph-mon[97044]: pgmap v2240: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:04 smithi110 ceph-mon[97633]: pgmap v2240: 129 pgs: 129 active+clean; 7.9 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:04.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:04.059387+0000) 2022-09-10T05:39:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:04.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:04.654447+0000) 2022-09-10T05:39:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:05.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:05.059530+0000) 2022-09-10T05:39:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:05.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:05.654641+0000) 2022-09-10T05:39:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:06 smithi110 ceph-mon[97633]: pgmap v2241: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:39:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:06 smithi104 ceph-mon[97044]: pgmap v2241: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:39:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:06.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:06.059652+0000) 2022-09-10T05:39:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:06.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:06.654807+0000) 2022-09-10T05:39:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:07.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:07.059870+0000) 2022-09-10T05:39:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:07.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:07.655020+0000) 2022-09-10T05:39:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:08 smithi110 ceph-mon[97633]: pgmap v2242: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:39:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:08 smithi104 ceph-mon[97044]: pgmap v2242: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:39:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:08.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:08.060034+0000) 2022-09-10T05:39:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:08.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:08.655168+0000) 2022-09-10T05:39:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:09 smithi110 ceph-mon[97633]: osdmap e224: 12 total, 12 up, 12 in 2022-09-10T05:39:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:09 smithi104 ceph-mon[97044]: osdmap e224: 12 total, 12 up, 12 in 2022-09-10T05:39:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:09.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:09.060207+0000) 2022-09-10T05:39:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:09.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:09.655389+0000) 2022-09-10T05:39:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:10 smithi110 ceph-mon[97633]: pgmap v2244: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:39:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:10 smithi104 ceph-mon[97044]: pgmap v2244: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:39:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:10.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:10.060357+0000) 2022-09-10T05:39:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:10.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:10.655584+0000) 2022-09-10T05:39:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:39:11.307+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:39:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:11.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:11.060473+0000) 2022-09-10T05:39:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:11.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:11.655761+0000) 2022-09-10T05:39:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:12 smithi110 ceph-mon[97633]: pgmap v2245: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:39:12.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:12.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:12.060603+0000) 2022-09-10T05:39:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:12 smithi104 ceph-mon[97044]: pgmap v2245: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:39:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:12.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:12.655906+0000) 2022-09-10T05:39:13.327 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:13.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:13.060771+0000) 2022-09-10T05:39:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:13.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:13.656044+0000) 2022-09-10T05:39:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:14 smithi104 ceph-mon[97044]: osdmap e225: 12 total, 12 up, 12 in 2022-09-10T05:39:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:14 smithi104 ceph-mon[97044]: pgmap v2247: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:39:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:14 smithi110 ceph-mon[97633]: osdmap e225: 12 total, 12 up, 12 in 2022-09-10T05:39:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:14 smithi110 ceph-mon[97633]: pgmap v2247: 129 pgs: 129 active+clean; 7.9 MiB data, 391 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:39:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:14.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:14.060921+0000) 2022-09-10T05:39:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:14.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:14.656168+0000) 2022-09-10T05:39:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:15.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:15.061076+0000) 2022-09-10T05:39:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:15.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:15.656315+0000) 2022-09-10T05:39:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:16 smithi110 ceph-mon[97633]: pgmap v2248: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:39:16.446 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:16.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:16.061188+0000) 2022-09-10T05:39:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:16 smithi104 ceph-mon[97044]: pgmap v2248: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:39:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:16.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:16.656482+0000) 2022-09-10T05:39:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:17.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:17.061348+0000) 2022-09-10T05:39:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:17.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:17.656639+0000) 2022-09-10T05:39:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:18 smithi110 ceph-mon[97633]: pgmap v2249: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:39:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:18 smithi104 ceph-mon[97044]: pgmap v2249: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T05:39:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:18.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:18.061515+0000) 2022-09-10T05:39:19.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:18.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:18.656795+0000) 2022-09-10T05:39:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:19.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:19.061707+0000) 2022-09-10T05:39:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:19.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:19.656960+0000) 2022-09-10T05:39:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:20 smithi110 ceph-mon[97633]: pgmap v2250: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:39:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:20 smithi104 ceph-mon[97044]: pgmap v2250: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:39:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:20.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:20.061868+0000) 2022-09-10T05:39:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:20.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:20.657120+0000) 2022-09-10T05:39:21.353 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:21.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:21.062005+0000) 2022-09-10T05:39:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:21.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:21.657263+0000) 2022-09-10T05:39:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:22 smithi110 ceph-mon[97633]: pgmap v2251: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:39:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:22 smithi104 ceph-mon[97044]: pgmap v2251: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:39:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:22.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:22.062166+0000) 2022-09-10T05:39:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:22.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:22.657412+0000) 2022-09-10T05:39:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:23.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:23.062314+0000) 2022-09-10T05:39:24.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:23.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:23.657567+0000) 2022-09-10T05:39:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:24 smithi104 ceph-mon[97044]: pgmap v2252: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:39:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:24 smithi110 ceph-mon[97633]: pgmap v2252: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:39:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:24.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:24.062496+0000) 2022-09-10T05:39:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:24.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:24.657718+0000) 2022-09-10T05:39:25.456 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:25.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:25.062622+0000) 2022-09-10T05:39:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:25.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:25.657883+0000) 2022-09-10T05:39:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:26.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:26.062795+0000) 2022-09-10T05:39:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:26 smithi110 ceph-mon[97633]: pgmap v2253: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:39:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:26 smithi104 ceph-mon[97044]: pgmap v2253: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:39:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:39:26.318+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:39:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:26.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:26.658041+0000) 2022-09-10T05:39:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:27.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:27.062956+0000) 2022-09-10T05:39:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:27.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:27.658207+0000) 2022-09-10T05:39:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:28 smithi110 ceph-mon[97633]: pgmap v2254: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:28 smithi104 ceph-mon[97044]: pgmap v2254: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:28.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:28.063126+0000) 2022-09-10T05:39:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:28.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:28.658367+0000) 2022-09-10T05:39:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:29.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:29.063252+0000) 2022-09-10T05:39:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:29.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:29.658540+0000) 2022-09-10T05:39:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:30 smithi110 ceph-mon[97633]: pgmap v2255: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:30 smithi104 ceph-mon[97044]: pgmap v2255: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:30.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:30.063373+0000) 2022-09-10T05:39:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:30.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:30.658706+0000) 2022-09-10T05:39:31.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:31.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:31.063509+0000) 2022-09-10T05:39:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:31.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:31.658880+0000) 2022-09-10T05:39:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:32 smithi110 ceph-mon[97633]: pgmap v2256: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:32 smithi104 ceph-mon[97044]: pgmap v2256: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:32.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:32.063642+0000) 2022-09-10T05:39:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:32.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:32.659043+0000) 2022-09-10T05:39:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:33.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:33.063802+0000) 2022-09-10T05:39:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:33.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:33.659197+0000) 2022-09-10T05:39:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:34 smithi104 ceph-mon[97044]: pgmap v2257: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:34 smithi110 ceph-mon[97633]: pgmap v2257: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:34.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:34.063975+0000) 2022-09-10T05:39:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:34.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:34.659365+0000) 2022-09-10T05:39:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:35.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:35.064129+0000) 2022-09-10T05:39:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:35.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:35.659514+0000) 2022-09-10T05:39:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:36 smithi110 ceph-mon[97633]: pgmap v2258: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:36 smithi104 ceph-mon[97044]: pgmap v2258: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:36.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:36.064288+0000) 2022-09-10T05:39:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:36.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:36.659658+0000) 2022-09-10T05:39:37.456 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:37.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:37.064424+0000) 2022-09-10T05:39:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:37.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:37.659901+0000) 2022-09-10T05:39:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:38 smithi110 ceph-mon[97633]: pgmap v2259: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:38 smithi104 ceph-mon[97044]: pgmap v2259: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:38.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:38.064589+0000) 2022-09-10T05:39:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:38.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:38.660081+0000) 2022-09-10T05:39:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:39.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:39.064875+0000) 2022-09-10T05:39:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:39.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:39.660316+0000) 2022-09-10T05:39:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:40 smithi110 ceph-mon[97633]: pgmap v2260: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:40 smithi104 ceph-mon[97044]: pgmap v2260: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:40.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:40.065069+0000) 2022-09-10T05:39:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:40.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:40.660511+0000) 2022-09-10T05:39:41.335 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:41.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:41.065278+0000) 2022-09-10T05:39:41.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:39:41.333+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:39:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:41.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:41.660788+0000) 2022-09-10T05:39:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:42 smithi110 ceph-mon[97633]: pgmap v2261: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:42 smithi104 ceph-mon[97044]: pgmap v2261: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:42.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:42.065464+0000) 2022-09-10T05:39:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:42.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:42.661037+0000) 2022-09-10T05:39:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:43.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:43.065679+0000) 2022-09-10T05:39:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:43.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:43.661315+0000) 2022-09-10T05:39:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:44 smithi104 ceph-mon[97044]: pgmap v2262: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:39:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:39:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:39:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:39:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:44 smithi110 ceph-mon[97633]: pgmap v2262: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:39:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:39:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:39:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:39:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:44.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:44.065930+0000) 2022-09-10T05:39:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:44.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:44.661594+0000) 2022-09-10T05:39:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:45.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:45.066188+0000) 2022-09-10T05:39:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:45.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:45.661754+0000) 2022-09-10T05:39:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:46 smithi110 ceph-mon[97633]: pgmap v2263: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:46 smithi104 ceph-mon[97044]: pgmap v2263: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:46.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:46.066377+0000) 2022-09-10T05:39:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:46.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:46.661961+0000) 2022-09-10T05:39:47.471 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:47.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:47.066576+0000) 2022-09-10T05:39:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:47.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:47.662484+0000) 2022-09-10T05:39:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:48 smithi110 ceph-mon[97633]: pgmap v2264: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:48 smithi104 ceph-mon[97044]: pgmap v2264: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:48.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:48.066777+0000) 2022-09-10T05:39:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:48.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:48.662688+0000) 2022-09-10T05:39:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:49.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:49.066971+0000) 2022-09-10T05:39:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:49.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:49.662908+0000) 2022-09-10T05:39:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:50 smithi110 ceph-mon[97633]: pgmap v2265: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:50 smithi104 ceph-mon[97044]: pgmap v2265: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:50.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:50.067226+0000) 2022-09-10T05:39:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:50.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:50.663187+0000) 2022-09-10T05:39:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:51.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:51.067480+0000) 2022-09-10T05:39:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:51.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:51.663389+0000) 2022-09-10T05:39:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:52 smithi110 ceph-mon[97633]: pgmap v2266: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:52 smithi104 ceph-mon[97044]: pgmap v2266: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:52.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:52.067682+0000) 2022-09-10T05:39:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:52.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:52.663635+0000) 2022-09-10T05:39:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:53.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:53.067898+0000) 2022-09-10T05:39:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:53.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:53.663915+0000) 2022-09-10T05:39:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:54 smithi110 ceph-mon[97633]: pgmap v2267: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:54 smithi104 ceph-mon[97044]: pgmap v2267: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:54.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:54.068135+0000) 2022-09-10T05:39:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:54.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:54.664162+0000) 2022-09-10T05:39:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:55.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:55.068306+0000) 2022-09-10T05:39:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:55.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:55.664367+0000) 2022-09-10T05:39:56.351 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:56.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:56.068537+0000) 2022-09-10T05:39:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:56 smithi110 ceph-mon[97633]: pgmap v2268: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:56 smithi104 ceph-mon[97044]: pgmap v2268: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:39:56.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:39:56.349+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:39:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:56.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:56.664557+0000) 2022-09-10T05:39:57.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:57.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:57.068751+0000) 2022-09-10T05:39:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:57.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:57.664721+0000) 2022-09-10T05:39:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:39:58 smithi110 ceph-mon[97633]: pgmap v2269: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:39:58 smithi104 ceph-mon[97044]: pgmap v2269: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:39:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:58.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:58.068968+0000) 2022-09-10T05:39:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:58.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:58.664888+0000) 2022-09-10T05:39:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:39:59.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:59.069205+0000) 2022-09-10T05:40:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:39:59.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:39:59.665134+0000) 2022-09-10T05:40:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:00 smithi110 ceph-mon[97633]: pgmap v2270: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:40:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:00 smithi104 ceph-mon[97044]: pgmap v2270: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:00.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:40:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:00.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:00.069422+0000) 2022-09-10T05:40:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:00.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:00.665396+0000) 2022-09-10T05:40:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:01.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:01.069630+0000) 2022-09-10T05:40:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:01.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:01.665640+0000) 2022-09-10T05:40:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:02 smithi110 ceph-mon[97633]: pgmap v2271: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:02 smithi104 ceph-mon[97044]: pgmap v2271: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:02.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:02.069861+0000) 2022-09-10T05:40:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:02.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:02.665908+0000) 2022-09-10T05:40:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:03.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:03.070144+0000) 2022-09-10T05:40:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:03.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:03.666159+0000) 2022-09-10T05:40:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:04 smithi110 ceph-mon[97633]: pgmap v2272: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:04 smithi104 ceph-mon[97044]: pgmap v2272: 129 pgs: 129 active+clean; 7.9 MiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:04.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:04.070353+0000) 2022-09-10T05:40:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:04.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:04.666326+0000) 2022-09-10T05:40:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:05.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:05.070604+0000) 2022-09-10T05:40:06.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:05.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:05.666480+0000) 2022-09-10T05:40:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:06 smithi110 ceph-mon[97633]: pgmap v2273: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:40:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:06 smithi104 ceph-mon[97044]: pgmap v2273: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:40:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:06.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:06.070848+0000) 2022-09-10T05:40:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:06.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:06.666653+0000) 2022-09-10T05:40:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:07.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:07.071139+0000) 2022-09-10T05:40:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:07.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:07.666894+0000) 2022-09-10T05:40:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:08 smithi110 ceph-mon[97633]: pgmap v2274: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:40:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:08 smithi110 ceph-mon[97633]: osdmap e226: 12 total, 12 up, 12 in 2022-09-10T05:40:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:08 smithi104 ceph-mon[97044]: pgmap v2274: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:40:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:08 smithi104 ceph-mon[97044]: osdmap e226: 12 total, 12 up, 12 in 2022-09-10T05:40:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:08.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:08.071387+0000) 2022-09-10T05:40:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:08.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:08.667158+0000) 2022-09-10T05:40:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:09.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:09.071567+0000) 2022-09-10T05:40:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:09.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:09.667445+0000) 2022-09-10T05:40:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:10 smithi110 ceph-mon[97633]: pgmap v2276: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:40:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:10 smithi104 ceph-mon[97044]: pgmap v2276: 129 pgs: 129 active+clean; 7.9 MiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:40:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:10.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:10.071803+0000) 2022-09-10T05:40:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:10.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:10.667653+0000) 2022-09-10T05:40:11.365 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:11.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:11.072063+0000) 2022-09-10T05:40:11.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:40:11.363+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:40:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:11.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:11.667820+0000) 2022-09-10T05:40:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:12 smithi110 ceph-mon[97633]: pgmap v2277: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:40:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:12 smithi104 ceph-mon[97044]: pgmap v2277: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:40:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:12.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:12.072302+0000) 2022-09-10T05:40:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:12.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:12.668080+0000) 2022-09-10T05:40:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:13.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:13.072626+0000) 2022-09-10T05:40:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:13.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:13.668362+0000) 2022-09-10T05:40:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:14 smithi110 ceph-mon[97633]: osdmap e227: 12 total, 12 up, 12 in 2022-09-10T05:40:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:14 smithi110 ceph-mon[97633]: pgmap v2279: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:40:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:14 smithi104 ceph-mon[97044]: osdmap e227: 12 total, 12 up, 12 in 2022-09-10T05:40:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:14 smithi104 ceph-mon[97044]: pgmap v2279: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:40:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:14.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:14.072879+0000) 2022-09-10T05:40:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:14.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:14.668630+0000) 2022-09-10T05:40:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:15.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:15.073149+0000) 2022-09-10T05:40:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:15.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:15.668810+0000) 2022-09-10T05:40:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:16 smithi110 ceph-mon[97633]: pgmap v2280: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:40:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:16 smithi104 ceph-mon[97044]: pgmap v2280: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:40:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:16.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:16.073319+0000) 2022-09-10T05:40:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:16.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:16.669072+0000) 2022-09-10T05:40:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:17.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:17.073514+0000) 2022-09-10T05:40:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:17.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:17.669387+0000) 2022-09-10T05:40:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:18 smithi110 ceph-mon[97633]: pgmap v2281: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:40:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:18 smithi104 ceph-mon[97044]: pgmap v2281: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:40:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:18.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:18.073776+0000) 2022-09-10T05:40:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:18.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:18.669636+0000) 2022-09-10T05:40:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:19.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:19.074021+0000) 2022-09-10T05:40:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:19.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:19.669851+0000) 2022-09-10T05:40:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:20 smithi110 ceph-mon[97633]: pgmap v2282: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:40:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:20 smithi104 ceph-mon[97044]: pgmap v2282: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:40:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:20.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:20.074274+0000) 2022-09-10T05:40:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:20.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:20.670078+0000) 2022-09-10T05:40:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:21.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:21.074492+0000) 2022-09-10T05:40:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:21.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:21.670336+0000) 2022-09-10T05:40:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:22 smithi110 ceph-mon[97633]: pgmap v2283: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:40:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:22 smithi104 ceph-mon[97044]: pgmap v2283: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:40:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:22.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:22.074678+0000) 2022-09-10T05:40:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:22.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:22.670554+0000) 2022-09-10T05:40:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:23.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:23.074930+0000) 2022-09-10T05:40:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:23.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:23.670805+0000) 2022-09-10T05:40:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:24 smithi110 ceph-mon[97633]: pgmap v2284: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:40:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:24 smithi104 ceph-mon[97044]: pgmap v2284: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:40:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:24.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:24.075213+0000) 2022-09-10T05:40:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:24.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:24.671025+0000) 2022-09-10T05:40:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:25.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:25.075449+0000) 2022-09-10T05:40:26.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:25.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:25.671308+0000) 2022-09-10T05:40:26.366 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:26.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:26.075698+0000) 2022-09-10T05:40:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:26 smithi110 ceph-mon[97633]: pgmap v2285: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:40:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:26 smithi104 ceph-mon[97044]: pgmap v2285: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:40:26.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:40:26.364+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:40:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:26.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:26.671543+0000) 2022-09-10T05:40:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:27.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:27.075885+0000) 2022-09-10T05:40:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:27.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:27.671745+0000) 2022-09-10T05:40:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:28 smithi110 ceph-mon[97633]: pgmap v2286: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:28 smithi104 ceph-mon[97044]: pgmap v2286: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:28.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:28.076117+0000) 2022-09-10T05:40:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:28.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:28.671962+0000) 2022-09-10T05:40:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:29.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:29.076376+0000) 2022-09-10T05:40:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:29.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:29.672211+0000) 2022-09-10T05:40:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:30 smithi110 ceph-mon[97633]: pgmap v2287: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:30 smithi104 ceph-mon[97044]: pgmap v2287: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:30.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:30.076639+0000) 2022-09-10T05:40:31.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:30.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:30.672413+0000) 2022-09-10T05:40:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:31.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:31.076808+0000) 2022-09-10T05:40:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:31.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:31.672634+0000) 2022-09-10T05:40:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:32 smithi110 ceph-mon[97633]: pgmap v2288: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:32 smithi104 ceph-mon[97044]: pgmap v2288: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:32.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:32.077035+0000) 2022-09-10T05:40:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:32.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:32.672872+0000) 2022-09-10T05:40:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:33.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:33.077270+0000) 2022-09-10T05:40:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:33.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:33.673129+0000) 2022-09-10T05:40:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:34 smithi110 ceph-mon[97633]: pgmap v2289: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:34 smithi104 ceph-mon[97044]: pgmap v2289: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:34.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:34.077511+0000) 2022-09-10T05:40:35.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:34.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:34.673317+0000) 2022-09-10T05:40:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:35.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:35.077739+0000) 2022-09-10T05:40:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:35.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:35.673510+0000) 2022-09-10T05:40:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:36 smithi110 ceph-mon[97633]: pgmap v2290: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:36 smithi104 ceph-mon[97044]: pgmap v2290: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:36.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:36.077929+0000) 2022-09-10T05:40:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:36.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:36.673724+0000) 2022-09-10T05:40:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:37.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:37.078192+0000) 2022-09-10T05:40:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:37.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:37.673913+0000) 2022-09-10T05:40:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:38 smithi110 ceph-mon[97633]: pgmap v2291: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:38 smithi104 ceph-mon[97044]: pgmap v2291: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:38.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:38.078325+0000) 2022-09-10T05:40:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:38.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:38.674146+0000) 2022-09-10T05:40:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:39.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:39.078540+0000) 2022-09-10T05:40:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:39.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:39.674286+0000) 2022-09-10T05:40:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:40 smithi110 ceph-mon[97633]: pgmap v2292: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:40 smithi104 ceph-mon[97044]: pgmap v2292: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:40.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:40.078789+0000) 2022-09-10T05:40:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:40.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:40.674480+0000) 2022-09-10T05:40:41.382 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:41.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:41.079021+0000) 2022-09-10T05:40:41.675 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:40:41.380+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:40:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:41.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:41.674718+0000) 2022-09-10T05:40:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:42 smithi110 ceph-mon[97633]: pgmap v2293: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:42 smithi104 ceph-mon[97044]: pgmap v2293: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:42.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:42.079274+0000) 2022-09-10T05:40:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:42.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:42.674938+0000) 2022-09-10T05:40:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:43.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:43.079473+0000) 2022-09-10T05:40:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:43.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:43.675215+0000) 2022-09-10T05:40:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:44 smithi110 ceph-mon[97633]: pgmap v2294: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:40:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:40:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:40:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:40:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:44 smithi104 ceph-mon[97044]: pgmap v2294: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:40:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:40:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:40:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:40:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:44.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:44.079738+0000) 2022-09-10T05:40:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:44.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:44.675456+0000) 2022-09-10T05:40:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:45.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:45.079969+0000) 2022-09-10T05:40:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:45.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:45.675712+0000) 2022-09-10T05:40:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:46 smithi110 ceph-mon[97633]: pgmap v2295: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:46 smithi104 ceph-mon[97044]: pgmap v2295: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:46.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:46.080217+0000) 2022-09-10T05:40:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:46.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:46.675939+0000) 2022-09-10T05:40:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:47.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:47.080440+0000) 2022-09-10T05:40:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:47.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:47.676195+0000) 2022-09-10T05:40:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:48 smithi110 ceph-mon[97633]: pgmap v2296: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:48 smithi104 ceph-mon[97044]: pgmap v2296: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:48.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:48.080709+0000) 2022-09-10T05:40:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:48.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:48.676400+0000) 2022-09-10T05:40:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:49.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:49.080855+0000) 2022-09-10T05:40:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:49.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:49.676577+0000) 2022-09-10T05:40:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:50 smithi110 ceph-mon[97633]: pgmap v2297: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:50 smithi104 ceph-mon[97044]: pgmap v2297: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:50.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:50.081073+0000) 2022-09-10T05:40:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:50.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:50.676802+0000) 2022-09-10T05:40:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:51.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:51.081257+0000) 2022-09-10T05:40:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:51.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:51.677033+0000) 2022-09-10T05:40:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:52 smithi110 ceph-mon[97633]: pgmap v2298: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:52 smithi104 ceph-mon[97044]: pgmap v2298: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:52.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:52.081407+0000) 2022-09-10T05:40:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:52.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:52.677254+0000) 2022-09-10T05:40:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:53.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:53.081632+0000) 2022-09-10T05:40:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:53.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:53.677471+0000) 2022-09-10T05:40:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:54 smithi110 ceph-mon[97633]: pgmap v2299: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:54 smithi104 ceph-mon[97044]: pgmap v2299: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:54.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:54.081819+0000) 2022-09-10T05:40:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:54.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:54.677745+0000) 2022-09-10T05:40:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:55.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:55.081950+0000) 2022-09-10T05:40:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:55.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:55.677997+0000) 2022-09-10T05:40:56.396 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:56.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:56.082194+0000) 2022-09-10T05:40:56.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:56 smithi110 ceph-mon[97633]: pgmap v2300: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:56 smithi104 ceph-mon[97044]: pgmap v2300: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:40:56.679 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:40:56.394+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:40:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:56.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:56.678218+0000) 2022-09-10T05:40:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:57.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:57.082441+0000) 2022-09-10T05:40:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:57.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:57.678397+0000) 2022-09-10T05:40:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:40:58 smithi110 ceph-mon[97633]: pgmap v2301: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:40:58 smithi104 ceph-mon[97044]: pgmap v2301: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:40:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:58.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:58.082621+0000) 2022-09-10T05:40:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:58.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:58.678637+0000) 2022-09-10T05:40:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:40:59.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:59.082850+0000) 2022-09-10T05:41:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:40:59.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:40:59.678942+0000) 2022-09-10T05:41:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:00 smithi110 ceph-mon[97633]: pgmap v2302: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:00.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:00 smithi104 ceph-mon[97044]: pgmap v2302: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:00.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:00.083088+0000) 2022-09-10T05:41:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:00.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:00.679179+0000) 2022-09-10T05:41:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:01.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:01.083266+0000) 2022-09-10T05:41:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:01.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:01.679368+0000) 2022-09-10T05:41:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:02 smithi110 ceph-mon[97633]: pgmap v2303: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:02 smithi104 ceph-mon[97044]: pgmap v2303: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:02.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:02.083499+0000) 2022-09-10T05:41:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:02.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:02.679615+0000) 2022-09-10T05:41:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:03.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:03.083741+0000) 2022-09-10T05:41:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:03.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:03.679781+0000) 2022-09-10T05:41:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:04 smithi110 ceph-mon[97633]: pgmap v2304: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:04 smithi104 ceph-mon[97044]: pgmap v2304: 129 pgs: 129 active+clean; 7.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:04.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:04.084033+0000) 2022-09-10T05:41:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:04.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:04.679983+0000) 2022-09-10T05:41:05.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:05.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:05.084234+0000) 2022-09-10T05:41:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:05.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:05.680248+0000) 2022-09-10T05:41:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:06 smithi110 ceph-mon[97633]: pgmap v2305: 129 pgs: 129 active+clean; 7.9 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:41:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:06 smithi104 ceph-mon[97044]: pgmap v2305: 129 pgs: 129 active+clean; 7.9 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:41:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:06.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:06.084457+0000) 2022-09-10T05:41:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:06.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:06.680473+0000) 2022-09-10T05:41:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:07.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:07.084638+0000) 2022-09-10T05:41:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:07.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:07.680703+0000) 2022-09-10T05:41:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:08 smithi104 ceph-mon[97044]: pgmap v2306: 129 pgs: 129 active+clean; 7.9 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:41:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:08 smithi104 ceph-mon[97044]: osdmap e228: 12 total, 12 up, 12 in 2022-09-10T05:41:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:08.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:08.084849+0000) 2022-09-10T05:41:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:08 smithi110 ceph-mon[97633]: pgmap v2306: 129 pgs: 129 active+clean; 7.9 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:41:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:08 smithi110 ceph-mon[97633]: osdmap e228: 12 total, 12 up, 12 in 2022-09-10T05:41:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:08.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:08.680856+0000) 2022-09-10T05:41:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:09.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:09.085076+0000) 2022-09-10T05:41:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:09.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:09.681047+0000) 2022-09-10T05:41:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:10 smithi104 ceph-mon[97044]: pgmap v2308: 129 pgs: 129 active+clean; 7.9 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:41:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:10.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:10.085302+0000) 2022-09-10T05:41:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:10 smithi110 ceph-mon[97633]: pgmap v2308: 129 pgs: 129 active+clean; 7.9 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:41:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:10.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:10.681300+0000) 2022-09-10T05:41:11.411 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:11.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:11.085506+0000) 2022-09-10T05:41:11.683 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:41:11.409+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:41:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:11.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:11.681519+0000) 2022-09-10T05:41:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:12 smithi104 ceph-mon[97044]: pgmap v2309: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:41:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:12.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:12.085785+0000) 2022-09-10T05:41:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:12 smithi110 ceph-mon[97633]: pgmap v2309: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:41:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:12.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:12.681701+0000) 2022-09-10T05:41:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:13.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:13.086006+0000) 2022-09-10T05:41:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:13.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:13.681938+0000) 2022-09-10T05:41:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:14 smithi110 ceph-mon[97633]: osdmap e229: 12 total, 12 up, 12 in 2022-09-10T05:41:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:14 smithi110 ceph-mon[97633]: pgmap v2311: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:41:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:14 smithi104 ceph-mon[97044]: osdmap e229: 12 total, 12 up, 12 in 2022-09-10T05:41:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:14 smithi104 ceph-mon[97044]: pgmap v2311: 129 pgs: 129 active+clean; 7.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:41:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:14.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:14.086270+0000) 2022-09-10T05:41:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:14.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:14.682157+0000) 2022-09-10T05:41:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:15.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:15.086523+0000) 2022-09-10T05:41:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:15.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:15.682297+0000) 2022-09-10T05:41:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:16 smithi110 ceph-mon[97633]: pgmap v2312: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:41:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:16 smithi104 ceph-mon[97044]: pgmap v2312: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:41:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:16.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:16.086760+0000) 2022-09-10T05:41:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:16.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:16.682521+0000) 2022-09-10T05:41:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:17.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:17.086983+0000) 2022-09-10T05:41:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:17.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:17.682724+0000) 2022-09-10T05:41:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:18 smithi110 ceph-mon[97633]: pgmap v2313: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:41:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:18 smithi104 ceph-mon[97044]: pgmap v2313: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:41:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:18.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:18.087235+0000) 2022-09-10T05:41:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:18.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:18.682942+0000) 2022-09-10T05:41:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:19.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:19.087395+0000) 2022-09-10T05:41:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:19.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:19.683180+0000) 2022-09-10T05:41:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:20 smithi104 ceph-mon[97044]: pgmap v2314: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:41:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:20.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:20.087604+0000) 2022-09-10T05:41:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:20 smithi110 ceph-mon[97633]: pgmap v2314: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:41:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:20.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:20.683416+0000) 2022-09-10T05:41:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:21.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:21.087839+0000) 2022-09-10T05:41:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:21.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:21.683688+0000) 2022-09-10T05:41:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:22 smithi104 ceph-mon[97044]: pgmap v2315: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:41:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:22.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:22.088069+0000) 2022-09-10T05:41:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:22 smithi110 ceph-mon[97633]: pgmap v2315: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:41:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:22.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:22.683933+0000) 2022-09-10T05:41:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:23.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:23.088320+0000) 2022-09-10T05:41:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:23.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:23.684196+0000) 2022-09-10T05:41:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:24 smithi110 ceph-mon[97633]: pgmap v2316: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:41:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:24 smithi104 ceph-mon[97044]: pgmap v2316: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:41:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:24.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:24.088547+0000) 2022-09-10T05:41:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:24.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:24.684443+0000) 2022-09-10T05:41:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:25.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:25.088765+0000) 2022-09-10T05:41:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:25.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:25.684660+0000) 2022-09-10T05:41:26.424 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:26.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:26.089017+0000) 2022-09-10T05:41:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:26 smithi104 ceph-mon[97044]: pgmap v2317: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:41:26.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:26 smithi110 ceph-mon[97633]: pgmap v2317: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:41:26.685 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:41:26.423+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:41:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:26.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:26.684901+0000) 2022-09-10T05:41:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:27.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:27.089217+0000) 2022-09-10T05:41:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:27.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:27.685088+0000) 2022-09-10T05:41:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:28 smithi104 ceph-mon[97044]: pgmap v2318: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:28.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:28.089457+0000) 2022-09-10T05:41:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:28 smithi110 ceph-mon[97633]: pgmap v2318: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:28.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:28.685383+0000) 2022-09-10T05:41:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:29.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:29.089684+0000) 2022-09-10T05:41:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:29.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:29.685624+0000) 2022-09-10T05:41:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:30 smithi104 ceph-mon[97044]: pgmap v2319: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:30.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:30.089934+0000) 2022-09-10T05:41:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:30 smithi110 ceph-mon[97633]: pgmap v2319: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:30.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:30.685855+0000) 2022-09-10T05:41:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:31.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:31.090147+0000) 2022-09-10T05:41:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:31.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:31.686074+0000) 2022-09-10T05:41:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:32 smithi104 ceph-mon[97044]: pgmap v2320: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:32.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:32.090317+0000) 2022-09-10T05:41:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:32 smithi110 ceph-mon[97633]: pgmap v2320: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:32.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:32.686344+0000) 2022-09-10T05:41:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:33.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:33.090553+0000) 2022-09-10T05:41:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:33.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:33.686537+0000) 2022-09-10T05:41:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:34 smithi110 ceph-mon[97633]: pgmap v2321: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:34 smithi104 ceph-mon[97044]: pgmap v2321: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:34.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:34.090824+0000) 2022-09-10T05:41:35.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:34.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:34.686772+0000) 2022-09-10T05:41:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:35.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:35.091049+0000) 2022-09-10T05:41:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:35.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:35.687006+0000) 2022-09-10T05:41:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:36 smithi104 ceph-mon[97044]: pgmap v2322: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:36.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:36.091291+0000) 2022-09-10T05:41:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:36 smithi110 ceph-mon[97633]: pgmap v2322: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:36.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:36.687235+0000) 2022-09-10T05:41:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:37.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:37.091529+0000) 2022-09-10T05:41:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:37.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:37.687527+0000) 2022-09-10T05:41:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:38 smithi104 ceph-mon[97044]: pgmap v2323: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:38.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:38.091736+0000) 2022-09-10T05:41:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:38 smithi110 ceph-mon[97633]: pgmap v2323: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:38.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:38.687725+0000) 2022-09-10T05:41:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:39.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:39.091931+0000) 2022-09-10T05:41:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:39.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:39.688034+0000) 2022-09-10T05:41:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:40 smithi104 ceph-mon[97044]: pgmap v2324: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:40.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:40.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:40.092182+0000) 2022-09-10T05:41:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:40 smithi110 ceph-mon[97633]: pgmap v2324: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:40.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:40.688277+0000) 2022-09-10T05:41:41.436 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:41.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:41.092400+0000) 2022-09-10T05:41:41.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:41:41.433+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:41:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:41.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:41.688571+0000) 2022-09-10T05:41:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:42 smithi104 ceph-mon[97044]: pgmap v2325: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:42.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:42.092639+0000) 2022-09-10T05:41:42.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:42 smithi110 ceph-mon[97633]: pgmap v2325: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:42.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:42.688836+0000) 2022-09-10T05:41:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:43.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:43.092866+0000) 2022-09-10T05:41:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:43.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:43.689127+0000) 2022-09-10T05:41:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:44 smithi110 ceph-mon[97633]: pgmap v2326: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:41:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:41:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:41:44.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:41:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:44 smithi104 ceph-mon[97044]: pgmap v2326: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:41:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:41:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:41:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:41:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:44.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:44.093180+0000) 2022-09-10T05:41:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:44.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:44.689402+0000) 2022-09-10T05:41:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:45.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:45.093380+0000) 2022-09-10T05:41:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:45.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:45.689622+0000) 2022-09-10T05:41:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:46 smithi104 ceph-mon[97044]: pgmap v2327: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:46.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:46.093627+0000) 2022-09-10T05:41:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:46 smithi110 ceph-mon[97633]: pgmap v2327: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:46.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:46.689867+0000) 2022-09-10T05:41:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:47.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:47.093805+0000) 2022-09-10T05:41:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:47.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:47.690177+0000) 2022-09-10T05:41:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:48 smithi104 ceph-mon[97044]: pgmap v2328: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:48.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:48.093977+0000) 2022-09-10T05:41:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:48 smithi110 ceph-mon[97633]: pgmap v2328: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:48.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:48.690368+0000) 2022-09-10T05:41:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:49.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:49.094225+0000) 2022-09-10T05:41:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:49.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:49.690598+0000) 2022-09-10T05:41:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:50 smithi104 ceph-mon[97044]: pgmap v2329: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:50.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:50.094472+0000) 2022-09-10T05:41:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:50 smithi110 ceph-mon[97633]: pgmap v2329: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:50.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:50.690810+0000) 2022-09-10T05:41:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:51.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:51.094701+0000) 2022-09-10T05:41:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:51.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:51.691053+0000) 2022-09-10T05:41:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:52 smithi104 ceph-mon[97044]: pgmap v2330: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:52.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:52.094927+0000) 2022-09-10T05:41:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:52 smithi110 ceph-mon[97633]: pgmap v2330: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:52.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:52.691326+0000) 2022-09-10T05:41:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:53.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:53.095197+0000) 2022-09-10T05:41:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:53.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:53.691475+0000) 2022-09-10T05:41:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:54 smithi110 ceph-mon[97633]: pgmap v2331: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:54 smithi104 ceph-mon[97044]: pgmap v2331: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:54.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:54.095451+0000) 2022-09-10T05:41:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:54.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:54.691685+0000) 2022-09-10T05:41:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:55.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:55.095665+0000) 2022-09-10T05:41:56.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:55.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:55.691860+0000) 2022-09-10T05:41:56.447 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:56.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:56.095821+0000) 2022-09-10T05:41:56.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:56 smithi104 ceph-mon[97044]: pgmap v2332: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:56.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:56 smithi110 ceph-mon[97633]: pgmap v2332: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:41:56.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:41:56.444+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:41:56.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:56.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:56.692123+0000) 2022-09-10T05:41:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:57.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:57.096033+0000) 2022-09-10T05:41:58.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:57.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:57.692349+0000) 2022-09-10T05:41:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:41:58 smithi104 ceph-mon[97044]: pgmap v2333: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:58.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:58.096271+0000) 2022-09-10T05:41:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:41:58 smithi110 ceph-mon[97633]: pgmap v2333: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:41:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:58.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:58.692579+0000) 2022-09-10T05:41:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:41:59.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:59.096507+0000) 2022-09-10T05:42:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:41:59.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:41:59.692735+0000) 2022-09-10T05:42:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:00 smithi104 ceph-mon[97044]: pgmap v2334: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:00.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:00.096766+0000) 2022-09-10T05:42:00.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:00 smithi110 ceph-mon[97633]: pgmap v2334: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:00.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:00.692923+0000) 2022-09-10T05:42:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:01.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:01.096992+0000) 2022-09-10T05:42:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:01.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:01.693169+0000) 2022-09-10T05:42:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:02 smithi104 ceph-mon[97044]: pgmap v2335: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:02.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:02.097240+0000) 2022-09-10T05:42:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:02 smithi110 ceph-mon[97633]: pgmap v2335: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:02.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:02.693330+0000) 2022-09-10T05:42:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:03.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:03.097440+0000) 2022-09-10T05:42:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:03.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:03.693474+0000) 2022-09-10T05:42:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:04 smithi110 ceph-mon[97633]: pgmap v2336: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:04 smithi104 ceph-mon[97044]: pgmap v2336: 129 pgs: 129 active+clean; 7.9 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:04.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:04.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:04.097651+0000) 2022-09-10T05:42:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:04.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:04.693649+0000) 2022-09-10T05:42:05.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:05.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:05.097860+0000) 2022-09-10T05:42:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:05.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:05.693839+0000) 2022-09-10T05:42:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:06 smithi104 ceph-mon[97044]: pgmap v2337: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:42:06.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:06.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:06.098125+0000) 2022-09-10T05:42:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:06 smithi110 ceph-mon[97633]: pgmap v2337: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:42:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:06.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:06.694005+0000) 2022-09-10T05:42:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:07.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:07.098275+0000) 2022-09-10T05:42:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:07.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:07.694224+0000) 2022-09-10T05:42:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:08 smithi104 ceph-mon[97044]: pgmap v2338: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:42:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:08 smithi104 ceph-mon[97044]: osdmap e230: 12 total, 12 up, 12 in 2022-09-10T05:42:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:08.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:08.098474+0000) 2022-09-10T05:42:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:08 smithi110 ceph-mon[97633]: pgmap v2338: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:42:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:08 smithi110 ceph-mon[97633]: osdmap e230: 12 total, 12 up, 12 in 2022-09-10T05:42:09.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:08.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:08.694404+0000) 2022-09-10T05:42:09.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:09.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:09.098724+0000) 2022-09-10T05:42:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:09.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:09.694608+0000) 2022-09-10T05:42:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:10 smithi104 ceph-mon[97044]: pgmap v2340: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:42:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:10.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:10.098924+0000) 2022-09-10T05:42:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:10 smithi110 ceph-mon[97633]: pgmap v2340: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:42:11.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:10.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:10.694767+0000) 2022-09-10T05:42:11.462 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:11.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:11.099163+0000) 2022-09-10T05:42:11.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:42:11.460+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:42:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:11.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:11.694991+0000) 2022-09-10T05:42:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:12 smithi104 ceph-mon[97044]: pgmap v2341: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:42:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:12.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:12.099377+0000) 2022-09-10T05:42:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:12 smithi110 ceph-mon[97633]: pgmap v2341: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:42:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:12.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:12.695239+0000) 2022-09-10T05:42:13.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:13.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:13.099590+0000) 2022-09-10T05:42:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:13.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:13.695426+0000) 2022-09-10T05:42:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:14 smithi110 ceph-mon[97633]: osdmap e231: 12 total, 12 up, 12 in 2022-09-10T05:42:14.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:14 smithi110 ceph-mon[97633]: pgmap v2343: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:42:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:14 smithi104 ceph-mon[97044]: osdmap e231: 12 total, 12 up, 12 in 2022-09-10T05:42:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:14 smithi104 ceph-mon[97044]: pgmap v2343: 129 pgs: 129 active+clean; 7.9 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:42:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:14.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:14.099795+0000) 2022-09-10T05:42:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:14.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:14.695692+0000) 2022-09-10T05:42:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:15.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:15.100001+0000) 2022-09-10T05:42:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:15.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:15.695889+0000) 2022-09-10T05:42:16.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:16 smithi104 ceph-mon[97044]: pgmap v2344: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:42:16.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:16.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:16.100234+0000) 2022-09-10T05:42:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:16 smithi110 ceph-mon[97633]: pgmap v2344: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:42:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:16.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:16.696050+0000) 2022-09-10T05:42:17.563 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:17.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:17.100490+0000) 2022-09-10T05:42:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:17.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:17.696307+0000) 2022-09-10T05:42:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:18 smithi104 ceph-mon[97044]: pgmap v2345: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:42:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:18.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:18.100773+0000) 2022-09-10T05:42:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:18 smithi110 ceph-mon[97633]: pgmap v2345: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:42:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:18.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:18.696525+0000) 2022-09-10T05:42:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:19.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:19.101005+0000) 2022-09-10T05:42:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:19.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:19.696801+0000) 2022-09-10T05:42:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:20 smithi104 ceph-mon[97044]: pgmap v2346: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:42:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:20.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:20.101240+0000) 2022-09-10T05:42:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:20 smithi110 ceph-mon[97633]: pgmap v2346: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:42:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:20.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:20.696935+0000) 2022-09-10T05:42:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:21.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:21.101451+0000) 2022-09-10T05:42:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:21.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:21.697128+0000) 2022-09-10T05:42:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:22 smithi104 ceph-mon[97044]: pgmap v2347: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:42:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:22.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:22.101621+0000) 2022-09-10T05:42:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:22 smithi110 ceph-mon[97633]: pgmap v2347: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:42:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:22.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:22.697329+0000) 2022-09-10T05:42:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:23.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:23.101759+0000) 2022-09-10T05:42:24.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:23.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:23.697527+0000) 2022-09-10T05:42:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:24 smithi110 ceph-mon[97633]: pgmap v2348: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:42:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:24 smithi104 ceph-mon[97044]: pgmap v2348: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:42:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:24.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:24.101988+0000) 2022-09-10T05:42:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:24.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:24.697742+0000) 2022-09-10T05:42:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:25.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:25.102193+0000) 2022-09-10T05:42:26.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:25.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:25.697976+0000) 2022-09-10T05:42:26.469 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:26.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:26.102397+0000) 2022-09-10T05:42:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:26 smithi104 ceph-mon[97044]: pgmap v2349: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:42:26.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:26 smithi110 ceph-mon[97633]: pgmap v2349: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:42:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:42:26.467+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:42:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:26.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:26.698248+0000) 2022-09-10T05:42:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:27.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:27.102657+0000) 2022-09-10T05:42:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:27.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:27.698462+0000) 2022-09-10T05:42:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:28 smithi104 ceph-mon[97044]: pgmap v2350: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:28.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:28.102929+0000) 2022-09-10T05:42:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:28 smithi110 ceph-mon[97633]: pgmap v2350: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:28.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:28.698638+0000) 2022-09-10T05:42:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:29.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:29.103192+0000) 2022-09-10T05:42:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:29.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:29.698898+0000) 2022-09-10T05:42:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:30 smithi104 ceph-mon[97044]: pgmap v2351: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:30.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:30.103403+0000) 2022-09-10T05:42:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:30 smithi110 ceph-mon[97633]: pgmap v2351: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:30.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:30.699079+0000) 2022-09-10T05:42:31.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:31.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:31.103635+0000) 2022-09-10T05:42:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:31.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:31.699397+0000) 2022-09-10T05:42:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:32 smithi104 ceph-mon[97044]: pgmap v2352: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:32.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:32.103874+0000) 2022-09-10T05:42:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:32 smithi110 ceph-mon[97633]: pgmap v2352: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:32.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:32.699582+0000) 2022-09-10T05:42:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:33.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:33.104072+0000) 2022-09-10T05:42:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:33.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:33.699799+0000) 2022-09-10T05:42:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:34 smithi110 ceph-mon[97633]: pgmap v2353: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:34 smithi104 ceph-mon[97044]: pgmap v2353: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:34.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:34.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:34.104333+0000) 2022-09-10T05:42:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:34.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:34.700029+0000) 2022-09-10T05:42:35.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:35.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:35.104516+0000) 2022-09-10T05:42:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:35.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:35.700247+0000) 2022-09-10T05:42:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:36 smithi104 ceph-mon[97044]: pgmap v2354: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:36.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:36.104756+0000) 2022-09-10T05:42:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:36 smithi110 ceph-mon[97633]: pgmap v2354: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:36.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:36.700519+0000) 2022-09-10T05:42:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:37.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:37.104985+0000) 2022-09-10T05:42:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:37.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:37.700687+0000) 2022-09-10T05:42:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:38 smithi104 ceph-mon[97044]: pgmap v2355: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:42:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:38.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:38.105262+0000) 2022-09-10T05:42:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:38 smithi110 ceph-mon[97633]: pgmap v2355: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:42:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:38.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:38.700881+0000) 2022-09-10T05:42:39.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:39.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:39.105423+0000) 2022-09-10T05:42:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:39.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:39.701154+0000) 2022-09-10T05:42:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:40 smithi104 ceph-mon[97044]: pgmap v2356: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:42:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:40.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:40.105675+0000) 2022-09-10T05:42:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:40 smithi110 ceph-mon[97633]: pgmap v2356: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:42:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:40.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:40.701360+0000) 2022-09-10T05:42:41.485 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:41.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:41.105944+0000) 2022-09-10T05:42:41.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:42:41.482+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:42:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:41.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:41.701521+0000) 2022-09-10T05:42:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:42 smithi104 ceph-mon[97044]: pgmap v2357: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:42.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:42.106230+0000) 2022-09-10T05:42:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:42 smithi110 ceph-mon[97633]: pgmap v2357: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:42.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:42.701766+0000) 2022-09-10T05:42:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:43.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:43.106494+0000) 2022-09-10T05:42:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:43.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:43.702045+0000) 2022-09-10T05:42:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:44 smithi110 ceph-mon[97633]: pgmap v2358: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:42:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:42:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:42:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:42:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:42:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:44 smithi104 ceph-mon[97044]: pgmap v2358: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:42:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:42:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:42:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:42:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:42:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:44.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:44.106762+0000) 2022-09-10T05:42:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:44.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:44.702285+0000) 2022-09-10T05:42:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:45.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:45.106985+0000) 2022-09-10T05:42:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:45.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:45.702483+0000) 2022-09-10T05:42:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:46 smithi104 ceph-mon[97044]: pgmap v2359: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:46.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:46.107270+0000) 2022-09-10T05:42:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:46 smithi110 ceph-mon[97633]: pgmap v2359: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:46.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:46.702641+0000) 2022-09-10T05:42:47.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:47.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:47.107494+0000) 2022-09-10T05:42:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:47.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:47.702843+0000) 2022-09-10T05:42:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:48 smithi104 ceph-mon[97044]: pgmap v2360: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:48.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:48.107769+0000) 2022-09-10T05:42:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:48 smithi110 ceph-mon[97633]: pgmap v2360: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:48.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:48.703050+0000) 2022-09-10T05:42:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:49.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:49.108008+0000) 2022-09-10T05:42:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:49.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:49.703316+0000) 2022-09-10T05:42:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:50 smithi104 ceph-mon[97044]: pgmap v2361: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:50.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:50.108258+0000) 2022-09-10T05:42:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:50 smithi110 ceph-mon[97633]: pgmap v2361: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:50.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:50.703496+0000) 2022-09-10T05:42:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:51.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:51.108413+0000) 2022-09-10T05:42:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:51.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:51.703755+0000) 2022-09-10T05:42:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:52 smithi104 ceph-mon[97044]: pgmap v2362: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:52.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:52.108629+0000) 2022-09-10T05:42:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:52 smithi110 ceph-mon[97633]: pgmap v2362: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:52.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:52.703905+0000) 2022-09-10T05:42:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:53.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:53.108879+0000) 2022-09-10T05:42:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:53.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:53.704156+0000) 2022-09-10T05:42:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:54 smithi110 ceph-mon[97633]: pgmap v2363: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:54 smithi104 ceph-mon[97044]: pgmap v2363: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:54.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:54.109167+0000) 2022-09-10T05:42:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:54.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:54.704324+0000) 2022-09-10T05:42:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:55.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:55.109409+0000) 2022-09-10T05:42:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:55.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:55.704572+0000) 2022-09-10T05:42:56.500 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:56.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:56.109580+0000) 2022-09-10T05:42:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:56 smithi104 ceph-mon[97044]: pgmap v2364: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:56 smithi110 ceph-mon[97633]: pgmap v2364: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:42:56.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:42:56.498+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:42:56.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:56.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:56.704734+0000) 2022-09-10T05:42:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:57.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:57.109821+0000) 2022-09-10T05:42:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:57.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:57.704982+0000) 2022-09-10T05:42:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:42:58 smithi104 ceph-mon[97044]: pgmap v2365: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:58.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:58.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:58.110022+0000) 2022-09-10T05:42:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:42:58 smithi110 ceph-mon[97633]: pgmap v2365: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:42:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:58.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:58.705202+0000) 2022-09-10T05:42:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:42:59.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:59.110286+0000) 2022-09-10T05:43:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:42:59.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:42:59.705379+0000) 2022-09-10T05:43:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:00 smithi104 ceph-mon[97044]: pgmap v2366: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:00.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:00.110509+0000) 2022-09-10T05:43:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:00 smithi110 ceph-mon[97633]: pgmap v2366: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:00.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:00.705632+0000) 2022-09-10T05:43:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:01.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:01.110744+0000) 2022-09-10T05:43:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:01.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:01.705880+0000) 2022-09-10T05:43:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:02 smithi104 ceph-mon[97044]: pgmap v2367: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:02.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:02.110990+0000) 2022-09-10T05:43:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:02 smithi110 ceph-mon[97633]: pgmap v2367: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:02.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:02.706156+0000) 2022-09-10T05:43:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:03.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:03.111260+0000) 2022-09-10T05:43:04.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:03.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:03.706364+0000) 2022-09-10T05:43:04.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:04 smithi110 ceph-mon[97633]: pgmap v2368: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:04 smithi104 ceph-mon[97044]: pgmap v2368: 129 pgs: 129 active+clean; 7.9 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:04.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:04.111437+0000) 2022-09-10T05:43:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:04.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:04.706599+0000) 2022-09-10T05:43:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:05.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:05.111676+0000) 2022-09-10T05:43:06.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:05.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:05.706871+0000) 2022-09-10T05:43:06.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:06 smithi104 ceph-mon[97044]: pgmap v2369: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:43:06.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:06.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:06.111873+0000) 2022-09-10T05:43:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:06 smithi110 ceph-mon[97633]: pgmap v2369: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:43:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:06.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:06.707155+0000) 2022-09-10T05:43:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:07.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:07.112168+0000) 2022-09-10T05:43:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:07.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:07.707438+0000) 2022-09-10T05:43:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:08 smithi104 ceph-mon[97044]: pgmap v2370: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:43:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:08 smithi104 ceph-mon[97044]: osdmap e232: 12 total, 12 up, 12 in 2022-09-10T05:43:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:08.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:08.112341+0000) 2022-09-10T05:43:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:08 smithi110 ceph-mon[97633]: pgmap v2370: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:43:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:08 smithi110 ceph-mon[97633]: osdmap e232: 12 total, 12 up, 12 in 2022-09-10T05:43:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:08.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:08.707591+0000) 2022-09-10T05:43:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:09.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:09.112496+0000) 2022-09-10T05:43:10.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:09.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:09.707839+0000) 2022-09-10T05:43:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:10 smithi104 ceph-mon[97044]: pgmap v2372: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:43:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:10.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:10.112678+0000) 2022-09-10T05:43:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:10 smithi110 ceph-mon[97633]: pgmap v2372: 129 pgs: 129 active+clean; 7.9 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:43:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:10.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:10.708039+0000) 2022-09-10T05:43:11.517 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:11.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:11.112904+0000) 2022-09-10T05:43:11.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:43:11.514+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:43:11.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:11.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:11.708218+0000) 2022-09-10T05:43:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:12 smithi104 ceph-mon[97044]: pgmap v2373: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:43:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:12.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:12.113121+0000) 2022-09-10T05:43:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:12 smithi110 ceph-mon[97633]: pgmap v2373: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:43:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:12.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:12.708467+0000) 2022-09-10T05:43:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:13.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:13.113341+0000) 2022-09-10T05:43:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:13.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:13.708687+0000) 2022-09-10T05:43:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:14 smithi110 ceph-mon[97633]: osdmap e233: 12 total, 12 up, 12 in 2022-09-10T05:43:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:14 smithi110 ceph-mon[97633]: pgmap v2375: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:43:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:14 smithi104 ceph-mon[97044]: osdmap e233: 12 total, 12 up, 12 in 2022-09-10T05:43:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:14 smithi104 ceph-mon[97044]: pgmap v2375: 129 pgs: 129 active+clean; 7.9 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:43:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:14.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:14.113519+0000) 2022-09-10T05:43:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:14.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:14.708914+0000) 2022-09-10T05:43:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:15.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:15.113744+0000) 2022-09-10T05:43:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:15.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:15.709172+0000) 2022-09-10T05:43:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:16 smithi104 ceph-mon[97044]: pgmap v2376: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:43:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:16.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:16.113948+0000) 2022-09-10T05:43:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:16 smithi110 ceph-mon[97633]: pgmap v2376: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:43:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:16.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:16.709398+0000) 2022-09-10T05:43:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:17.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:17.114205+0000) 2022-09-10T05:43:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:17.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:17.709603+0000) 2022-09-10T05:43:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:18 smithi104 ceph-mon[97044]: pgmap v2377: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:43:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:18.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:18.114400+0000) 2022-09-10T05:43:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:18 smithi110 ceph-mon[97633]: pgmap v2377: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:43:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:18.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:18.709809+0000) 2022-09-10T05:43:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:19.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:19.114524+0000) 2022-09-10T05:43:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:19.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:19.710049+0000) 2022-09-10T05:43:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:20 smithi104 ceph-mon[97044]: pgmap v2378: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:43:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:20.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:20.114698+0000) 2022-09-10T05:43:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:20 smithi110 ceph-mon[97633]: pgmap v2378: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:43:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:20.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:20.710299+0000) 2022-09-10T05:43:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:21.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:21.114880+0000) 2022-09-10T05:43:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:21.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:21.710529+0000) 2022-09-10T05:43:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:22 smithi104 ceph-mon[97044]: pgmap v2379: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:43:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:22.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:22.115146+0000) 2022-09-10T05:43:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:22 smithi110 ceph-mon[97633]: pgmap v2379: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:43:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:22.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:22.710744+0000) 2022-09-10T05:43:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:23.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:23.115417+0000) 2022-09-10T05:43:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:23.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:23.710992+0000) 2022-09-10T05:43:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:24 smithi110 ceph-mon[97633]: pgmap v2380: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:43:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:24 smithi104 ceph-mon[97044]: pgmap v2380: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:43:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:24.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:24.115624+0000) 2022-09-10T05:43:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:24.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:24.711218+0000) 2022-09-10T05:43:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:25.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:25.115886+0000) 2022-09-10T05:43:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:25.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:25.711409+0000) 2022-09-10T05:43:26.520 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:26.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:26.116090+0000) 2022-09-10T05:43:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:26 smithi104 ceph-mon[97044]: pgmap v2381: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:43:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:26 smithi110 ceph-mon[97633]: pgmap v2381: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:43:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:26.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:26.711658+0000) 2022-09-10T05:43:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:43:26.517+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:43:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:27.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:27.116356+0000) 2022-09-10T05:43:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:27.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:27.711964+0000) 2022-09-10T05:43:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:28 smithi104 ceph-mon[97044]: pgmap v2382: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:28.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:28.116532+0000) 2022-09-10T05:43:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:28 smithi110 ceph-mon[97633]: pgmap v2382: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:28.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:28.712194+0000) 2022-09-10T05:43:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:29.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:29.116746+0000) 2022-09-10T05:43:30.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:29.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:29.712469+0000) 2022-09-10T05:43:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:30 smithi104 ceph-mon[97044]: pgmap v2383: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:30.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:30.116986+0000) 2022-09-10T05:43:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:30 smithi110 ceph-mon[97633]: pgmap v2383: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:30.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:30.712733+0000) 2022-09-10T05:43:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:31.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:31.117237+0000) 2022-09-10T05:43:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:31.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:31.712930+0000) 2022-09-10T05:43:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:32 smithi104 ceph-mon[97044]: pgmap v2384: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:32.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:32.117518+0000) 2022-09-10T05:43:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:32 smithi110 ceph-mon[97633]: pgmap v2384: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:32.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:32.713204+0000) 2022-09-10T05:43:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:33.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:33.117744+0000) 2022-09-10T05:43:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:33.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:33.713370+0000) 2022-09-10T05:43:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:34 smithi110 ceph-mon[97633]: pgmap v2385: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:34 smithi104 ceph-mon[97044]: pgmap v2385: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:43:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:34.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:34.117946+0000) 2022-09-10T05:43:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:34.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:34.713507+0000) 2022-09-10T05:43:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:35.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:35.118201+0000) 2022-09-10T05:43:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:35.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:35.713694+0000) 2022-09-10T05:43:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:36 smithi104 ceph-mon[97044]: pgmap v2386: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:36.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:36.118425+0000) 2022-09-10T05:43:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:36 smithi110 ceph-mon[97633]: pgmap v2386: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:36.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:36.713933+0000) 2022-09-10T05:43:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:37.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:37.118656+0000) 2022-09-10T05:43:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:37.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:37.714199+0000) 2022-09-10T05:43:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:38 smithi104 ceph-mon[97044]: pgmap v2387: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:38.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:38.118825+0000) 2022-09-10T05:43:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:38 smithi110 ceph-mon[97633]: pgmap v2387: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:38.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:38.714439+0000) 2022-09-10T05:43:39.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:39.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:39.119006+0000) 2022-09-10T05:43:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:39.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:39.714644+0000) 2022-09-10T05:43:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:40 smithi104 ceph-mon[97044]: pgmap v2388: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:40.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:40.119171+0000) 2022-09-10T05:43:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:40 smithi110 ceph-mon[97633]: pgmap v2388: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:40.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:40.714823+0000) 2022-09-10T05:43:41.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:41.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:41.119407+0000) 2022-09-10T05:43:41.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:43:41.525+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:43:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:41.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:41.715060+0000) 2022-09-10T05:43:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:42 smithi104 ceph-mon[97044]: pgmap v2389: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:42.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:42.119660+0000) 2022-09-10T05:43:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:42 smithi110 ceph-mon[97633]: pgmap v2389: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:43:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:42.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:42.715348+0000) 2022-09-10T05:43:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:43.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:43.119894+0000) 2022-09-10T05:43:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:43.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:43.715589+0000) 2022-09-10T05:43:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:44 smithi110 ceph-mon[97633]: pgmap v2390: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:43:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:43:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:43:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:43:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:44 smithi104 ceph-mon[97044]: pgmap v2390: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:43:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:43:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:43:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:43:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:44.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:44.120200+0000) 2022-09-10T05:43:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:44.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:44.715854+0000) 2022-09-10T05:43:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:45.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:45.120436+0000) 2022-09-10T05:43:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:45.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:45.716110+0000) 2022-09-10T05:43:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:46 smithi104 ceph-mon[97044]: pgmap v2391: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T05:43:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:46.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:46.120628+0000) 2022-09-10T05:43:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:46 smithi110 ceph-mon[97633]: pgmap v2391: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T05:43:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:46.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:46.716377+0000) 2022-09-10T05:43:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:47.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:47.120816+0000) 2022-09-10T05:43:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:47.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:47.716575+0000) 2022-09-10T05:43:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:48 smithi104 ceph-mon[97044]: pgmap v2392: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T05:43:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:48.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:48.121005+0000) 2022-09-10T05:43:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:48 smithi110 ceph-mon[97633]: pgmap v2392: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T05:43:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:48.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:48.716765+0000) 2022-09-10T05:43:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:49.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:49.121234+0000) 2022-09-10T05:43:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:49.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:49.717024+0000) 2022-09-10T05:43:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:50 smithi104 ceph-mon[97044]: pgmap v2393: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T05:43:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:50.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:50.121473+0000) 2022-09-10T05:43:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:50 smithi110 ceph-mon[97633]: pgmap v2393: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T05:43:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:50.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:50.717299+0000) 2022-09-10T05:43:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:51.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:51.121679+0000) 2022-09-10T05:43:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:51.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:51.717559+0000) 2022-09-10T05:43:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:52 smithi104 ceph-mon[97044]: pgmap v2394: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:43:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:52.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:52.121940+0000) 2022-09-10T05:43:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:52 smithi110 ceph-mon[97633]: pgmap v2394: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T05:43:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:52.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:52.717785+0000) 2022-09-10T05:43:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:53.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:53.122167+0000) 2022-09-10T05:43:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:53.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:53.718022+0000) 2022-09-10T05:43:54.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:54 smithi110 ceph-mon[97633]: pgmap v2395: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T05:43:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:54 smithi104 ceph-mon[97044]: pgmap v2395: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T05:43:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:54.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:54.122350+0000) 2022-09-10T05:43:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:54.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:54.718293+0000) 2022-09-10T05:43:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:55.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:55.122526+0000) 2022-09-10T05:43:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:55.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:55.718558+0000) 2022-09-10T05:43:56.537 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:56.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:56.122697+0000) 2022-09-10T05:43:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:56 smithi104 ceph-mon[97044]: pgmap v2396: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2022-09-10T05:43:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:56 smithi110 ceph-mon[97633]: pgmap v2396: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2022-09-10T05:43:56.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:56.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:56.718776+0000) 2022-09-10T05:43:56.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:43:56.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:43:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:57.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:57.122885+0000) 2022-09-10T05:43:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:57.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:57.719050+0000) 2022-09-10T05:43:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:43:58 smithi104 ceph-mon[97044]: pgmap v2397: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:58.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:58.123091+0000) 2022-09-10T05:43:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:43:58 smithi110 ceph-mon[97633]: pgmap v2397: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:43:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:58.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:58.719227+0000) 2022-09-10T05:43:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:43:59.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:59.123311+0000) 2022-09-10T05:44:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:43:59.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:43:59.719447+0000) 2022-09-10T05:44:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:00 smithi104 ceph-mon[97044]: pgmap v2398: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:44:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:00.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:00.123520+0000) 2022-09-10T05:44:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:00 smithi110 ceph-mon[97633]: pgmap v2398: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:44:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:00.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:00.719673+0000) 2022-09-10T05:44:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:01.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:01.123738+0000) 2022-09-10T05:44:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:01.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:01.719909+0000) 2022-09-10T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:02 smithi104 ceph-mon[97044]: pgmap v2399: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:02.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:02.123927+0000) 2022-09-10T05:44:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:02 smithi110 ceph-mon[97633]: pgmap v2399: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:02.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:02.720046+0000) 2022-09-10T05:44:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:03.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:03.124201+0000) 2022-09-10T05:44:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:03.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:03.720261+0000) 2022-09-10T05:44:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:04 smithi110 ceph-mon[97633]: pgmap v2400: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:44:04.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:04 smithi104 ceph-mon[97044]: pgmap v2400: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:44:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:04.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:04.124461+0000) 2022-09-10T05:44:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:04.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:04.720540+0000) 2022-09-10T05:44:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:05.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:05.124617+0000) 2022-09-10T05:44:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:05.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:05.720710+0000) 2022-09-10T05:44:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:06 smithi104 ceph-mon[97044]: pgmap v2401: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:44:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:06.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:06.124861+0000) 2022-09-10T05:44:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:06 smithi110 ceph-mon[97633]: pgmap v2401: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:44:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:06.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:06.720925+0000) 2022-09-10T05:44:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:07.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:07.125049+0000) 2022-09-10T05:44:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:07.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:07.721158+0000) 2022-09-10T05:44:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:08 smithi104 ceph-mon[97044]: pgmap v2402: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:44:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:08 smithi104 ceph-mon[97044]: osdmap e234: 12 total, 12 up, 12 in 2022-09-10T05:44:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:08.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:08.125264+0000) 2022-09-10T05:44:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:08 smithi110 ceph-mon[97633]: pgmap v2402: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:44:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:08 smithi110 ceph-mon[97633]: osdmap e234: 12 total, 12 up, 12 in 2022-09-10T05:44:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:08.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:08.721411+0000) 2022-09-10T05:44:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:09.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:09.125488+0000) 2022-09-10T05:44:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:09.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:09.721695+0000) 2022-09-10T05:44:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:10 smithi104 ceph-mon[97044]: pgmap v2404: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:44:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:10.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:10.125715+0000) 2022-09-10T05:44:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:10 smithi110 ceph-mon[97633]: pgmap v2404: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:44:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:10.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:10.721962+0000) 2022-09-10T05:44:11.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:11.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:11.125916+0000) 2022-09-10T05:44:11.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:44:11.549+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:44:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:11.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:11.722231+0000) 2022-09-10T05:44:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:12 smithi104 ceph-mon[97044]: pgmap v2405: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:44:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:12.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:12.126061+0000) 2022-09-10T05:44:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:12 smithi110 ceph-mon[97633]: pgmap v2405: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:44:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:12.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:12.722485+0000) 2022-09-10T05:44:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:13.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:13.126274+0000) 2022-09-10T05:44:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:13.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:13.722667+0000) 2022-09-10T05:44:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:14 smithi110 ceph-mon[97633]: osdmap e235: 12 total, 12 up, 12 in 2022-09-10T05:44:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:14 smithi110 ceph-mon[97633]: pgmap v2407: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:44:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:14 smithi104 ceph-mon[97044]: osdmap e235: 12 total, 12 up, 12 in 2022-09-10T05:44:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:14 smithi104 ceph-mon[97044]: pgmap v2407: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:44:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:14.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:14.126518+0000) 2022-09-10T05:44:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:14.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:14.722938+0000) 2022-09-10T05:44:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:15.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:15.126695+0000) 2022-09-10T05:44:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:15.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:15.723129+0000) 2022-09-10T05:44:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:16 smithi104 ceph-mon[97044]: pgmap v2408: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:44:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:16.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:16.126919+0000) 2022-09-10T05:44:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:16 smithi110 ceph-mon[97633]: pgmap v2408: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:44:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:16.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:16.723380+0000) 2022-09-10T05:44:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:17.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:17.127173+0000) 2022-09-10T05:44:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:17.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:17.723571+0000) 2022-09-10T05:44:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:18 smithi104 ceph-mon[97044]: pgmap v2409: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:44:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:18.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:18.127353+0000) 2022-09-10T05:44:18.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:18 smithi110 ceph-mon[97633]: pgmap v2409: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:44:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:18.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:18.723756+0000) 2022-09-10T05:44:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:19.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:19.127486+0000) 2022-09-10T05:44:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:19.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:19.723998+0000) 2022-09-10T05:44:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:20 smithi104 ceph-mon[97044]: pgmap v2410: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:44:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:20.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:20.127693+0000) 2022-09-10T05:44:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:20 smithi110 ceph-mon[97633]: pgmap v2410: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:44:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:20.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:20.724215+0000) 2022-09-10T05:44:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:21.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:21.127914+0000) 2022-09-10T05:44:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:21.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:21.724346+0000) 2022-09-10T05:44:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:22 smithi104 ceph-mon[97044]: pgmap v2411: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:44:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:22.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:22.128142+0000) 2022-09-10T05:44:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:22 smithi110 ceph-mon[97633]: pgmap v2411: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:44:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:22.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:22.724535+0000) 2022-09-10T05:44:23.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:23.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:23.128418+0000) 2022-09-10T05:44:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:23.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:23.724719+0000) 2022-09-10T05:44:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:24 smithi110 ceph-mon[97633]: pgmap v2412: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:44:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:24 smithi104 ceph-mon[97044]: pgmap v2412: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:44:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:24.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:24.128676+0000) 2022-09-10T05:44:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:24.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:24.724937+0000) 2022-09-10T05:44:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:25.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:25.128819+0000) 2022-09-10T05:44:26.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:25.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:25.725196+0000) 2022-09-10T05:44:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:26 smithi104 ceph-mon[97044]: pgmap v2413: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:44:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:26.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:26.129070+0000) 2022-09-10T05:44:26.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:26 smithi110 ceph-mon[97633]: pgmap v2413: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:44:27.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:44:26.565+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:44:27.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:26.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:26.725381+0000) 2022-09-10T05:44:27.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:27.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:27.129335+0000) 2022-09-10T05:44:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:27.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:27.725614+0000) 2022-09-10T05:44:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:28 smithi104 ceph-mon[97044]: pgmap v2414: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:28.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:28.129506+0000) 2022-09-10T05:44:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:28 smithi110 ceph-mon[97633]: pgmap v2414: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:28.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:28.725873+0000) 2022-09-10T05:44:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:29.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:29.129735+0000) 2022-09-10T05:44:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:29.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:29.726187+0000) 2022-09-10T05:44:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:30 smithi104 ceph-mon[97044]: pgmap v2415: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:30.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:30.129983+0000) 2022-09-10T05:44:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:30 smithi110 ceph-mon[97633]: pgmap v2415: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:30.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:30.726387+0000) 2022-09-10T05:44:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:31.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:31.130150+0000) 2022-09-10T05:44:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:31.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:31.726541+0000) 2022-09-10T05:44:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:32 smithi104 ceph-mon[97044]: pgmap v2416: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:32.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:32.130385+0000) 2022-09-10T05:44:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:32 smithi110 ceph-mon[97633]: pgmap v2416: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:32.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:32.726710+0000) 2022-09-10T05:44:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:33.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:33.130580+0000) 2022-09-10T05:44:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:33.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:33.726898+0000) 2022-09-10T05:44:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:34 smithi110 ceph-mon[97633]: pgmap v2417: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:34 smithi104 ceph-mon[97044]: pgmap v2417: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:34.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:34.130828+0000) 2022-09-10T05:44:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:34.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:34.727134+0000) 2022-09-10T05:44:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:35.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:35.131047+0000) 2022-09-10T05:44:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:35.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:35.727380+0000) 2022-09-10T05:44:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:36 smithi104 ceph-mon[97044]: pgmap v2418: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:36.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:36.131293+0000) 2022-09-10T05:44:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:36 smithi110 ceph-mon[97633]: pgmap v2418: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:36.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:36.727572+0000) 2022-09-10T05:44:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:37.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:37.131510+0000) 2022-09-10T05:44:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:37.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:37.727817+0000) 2022-09-10T05:44:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:38 smithi104 ceph-mon[97044]: pgmap v2419: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:38.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:38.131682+0000) 2022-09-10T05:44:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:38 smithi110 ceph-mon[97633]: pgmap v2419: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:38.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:38.728054+0000) 2022-09-10T05:44:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:39.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:39.131919+0000) 2022-09-10T05:44:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:39.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:39.728340+0000) 2022-09-10T05:44:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:40 smithi104 ceph-mon[97044]: pgmap v2420: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:40.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:40.132138+0000) 2022-09-10T05:44:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:40 smithi110 ceph-mon[97633]: pgmap v2420: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:40.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:40.728587+0000) 2022-09-10T05:44:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:41.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:41.132312+0000) 2022-09-10T05:44:42.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:44:41.580+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:44:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:41.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:41.728817+0000) 2022-09-10T05:44:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:42 smithi104 ceph-mon[97044]: pgmap v2421: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:42.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:42.132498+0000) 2022-09-10T05:44:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:42 smithi110 ceph-mon[97633]: pgmap v2421: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:42.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:42.729014+0000) 2022-09-10T05:44:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:43.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:43.132697+0000) 2022-09-10T05:44:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:43.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:43.729258+0000) 2022-09-10T05:44:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:44 smithi110 ceph-mon[97633]: pgmap v2422: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:44:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:44:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:44:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:44:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:44 smithi104 ceph-mon[97044]: pgmap v2422: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:44:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:44:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:44:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:44:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:44.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:44.132879+0000) 2022-09-10T05:44:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:44.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:44.729473+0000) 2022-09-10T05:44:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:45.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:45.133068+0000) 2022-09-10T05:44:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:45.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:45.729696+0000) 2022-09-10T05:44:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:46 smithi104 ceph-mon[97044]: pgmap v2423: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:46.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:46.133286+0000) 2022-09-10T05:44:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:46 smithi110 ceph-mon[97633]: pgmap v2423: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:46.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:46.729923+0000) 2022-09-10T05:44:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:47.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:47.133442+0000) 2022-09-10T05:44:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:47.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:47.730159+0000) 2022-09-10T05:44:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:48 smithi104 ceph-mon[97044]: pgmap v2424: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:48.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:48.133605+0000) 2022-09-10T05:44:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:48 smithi110 ceph-mon[97633]: pgmap v2424: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:48.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:48.730392+0000) 2022-09-10T05:44:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:49.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:49.133785+0000) 2022-09-10T05:44:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:49.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:49.730612+0000) 2022-09-10T05:44:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:50 smithi104 ceph-mon[97044]: pgmap v2425: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:50.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:50.133965+0000) 2022-09-10T05:44:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:50 smithi110 ceph-mon[97633]: pgmap v2425: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:50.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:50.730851+0000) 2022-09-10T05:44:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:51.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:51.134184+0000) 2022-09-10T05:44:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:51.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:51.731089+0000) 2022-09-10T05:44:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:52 smithi104 ceph-mon[97044]: pgmap v2426: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:52.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:52.134375+0000) 2022-09-10T05:44:52.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:52 smithi110 ceph-mon[97633]: pgmap v2426: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:53.065 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:52.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:52.731329+0000) 2022-09-10T05:44:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:53.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:53.134550+0000) 2022-09-10T05:44:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:53.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:53.731494+0000) 2022-09-10T05:44:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:54 smithi110 ceph-mon[97633]: pgmap v2427: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:54 smithi104 ceph-mon[97044]: pgmap v2427: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:54.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:54.134741+0000) 2022-09-10T05:44:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:54.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:54.731706+0000) 2022-09-10T05:44:55.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:55.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:55.134877+0000) 2022-09-10T05:44:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:55.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:55.731881+0000) 2022-09-10T05:44:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:56 smithi104 ceph-mon[97044]: pgmap v2428: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:56.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:56.135047+0000) 2022-09-10T05:44:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:56 smithi110 ceph-mon[97633]: pgmap v2428: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:44:57.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:56.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:56.732090+0000) 2022-09-10T05:44:57.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:44:56.595+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:44:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:57.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:57.135253+0000) 2022-09-10T05:44:58.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:57.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:57.732331+0000) 2022-09-10T05:44:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:44:58 smithi104 ceph-mon[97044]: pgmap v2429: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:58.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:58.135512+0000) 2022-09-10T05:44:58.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:44:58 smithi110 ceph-mon[97633]: pgmap v2429: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:44:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:58.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:58.732555+0000) 2022-09-10T05:44:59.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:44:59.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:59.135772+0000) 2022-09-10T05:45:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:44:59.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:44:59.732805+0000) 2022-09-10T05:45:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:00 smithi104 ceph-mon[97044]: pgmap v2430: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:00.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:00.136012+0000) 2022-09-10T05:45:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:00 smithi110 ceph-mon[97633]: pgmap v2430: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:00.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:00.732956+0000) 2022-09-10T05:45:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:01.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:01.136246+0000) 2022-09-10T05:45:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:01.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:01.733195+0000) 2022-09-10T05:45:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:02 smithi104 ceph-mon[97044]: pgmap v2431: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:02.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:02.136483+0000) 2022-09-10T05:45:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:02 smithi110 ceph-mon[97633]: pgmap v2431: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:02.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:02.733397+0000) 2022-09-10T05:45:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:03.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:03.136727+0000) 2022-09-10T05:45:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:03.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:03.733782+0000) 2022-09-10T05:45:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:04 smithi110 ceph-mon[97633]: pgmap v2432: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:04 smithi104 ceph-mon[97044]: pgmap v2432: 129 pgs: 129 active+clean; 7.9 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:04.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:04.136960+0000) 2022-09-10T05:45:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:04.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:04.733968+0000) 2022-09-10T05:45:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:05.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:05.137214+0000) 2022-09-10T05:45:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:05.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:05.734214+0000) 2022-09-10T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:06 smithi104 ceph-mon[97044]: pgmap v2433: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:45:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:06.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:06.137444+0000) 2022-09-10T05:45:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:06 smithi110 ceph-mon[97633]: pgmap v2433: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:45:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:06.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:06.734419+0000) 2022-09-10T05:45:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:07.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:07.137650+0000) 2022-09-10T05:45:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:07.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:07.734651+0000) 2022-09-10T05:45:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:08 smithi104 ceph-mon[97044]: pgmap v2434: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:45:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:08 smithi104 ceph-mon[97044]: osdmap e236: 12 total, 12 up, 12 in 2022-09-10T05:45:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:08.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:08.137891+0000) 2022-09-10T05:45:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:08 smithi110 ceph-mon[97633]: pgmap v2434: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:45:08.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:08 smithi110 ceph-mon[97633]: osdmap e236: 12 total, 12 up, 12 in 2022-09-10T05:45:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:08.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:08.734872+0000) 2022-09-10T05:45:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:09.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:09.138186+0000) 2022-09-10T05:45:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:09.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:09.735178+0000) 2022-09-10T05:45:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:10 smithi104 ceph-mon[97044]: pgmap v2436: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:45:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:10.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:10.138473+0000) 2022-09-10T05:45:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:10 smithi110 ceph-mon[97633]: pgmap v2436: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:45:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:10.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:10.735336+0000) 2022-09-10T05:45:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:11.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:11.138716+0000) 2022-09-10T05:45:12.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:45:11.608+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:45:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:11.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:11.735588+0000) 2022-09-10T05:45:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:12 smithi104 ceph-mon[97044]: pgmap v2437: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:45:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:12.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:12.138954+0000) 2022-09-10T05:45:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:12 smithi110 ceph-mon[97633]: pgmap v2437: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:45:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:12.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:12.735814+0000) 2022-09-10T05:45:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:13.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:13.139221+0000) 2022-09-10T05:45:14.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:13.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:13.736053+0000) 2022-09-10T05:45:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:14 smithi110 ceph-mon[97633]: osdmap e237: 12 total, 12 up, 12 in 2022-09-10T05:45:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:14 smithi110 ceph-mon[97633]: pgmap v2439: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:45:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:14 smithi104 ceph-mon[97044]: osdmap e237: 12 total, 12 up, 12 in 2022-09-10T05:45:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:14 smithi104 ceph-mon[97044]: pgmap v2439: 129 pgs: 129 active+clean; 7.9 MiB data, 417 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:45:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:14.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:14.139423+0000) 2022-09-10T05:45:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:14.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:14.736306+0000) 2022-09-10T05:45:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:15.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:15.139663+0000) 2022-09-10T05:45:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:15.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:15.736565+0000) 2022-09-10T05:45:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:16 smithi104 ceph-mon[97044]: pgmap v2440: 129 pgs: 129 active+clean; 7.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:45:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:16.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:16.139930+0000) 2022-09-10T05:45:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:16 smithi110 ceph-mon[97633]: pgmap v2440: 129 pgs: 129 active+clean; 7.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:45:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:16.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:16.736817+0000) 2022-09-10T05:45:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:17.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:17.140230+0000) 2022-09-10T05:45:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:17.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:17.737072+0000) 2022-09-10T05:45:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:18 smithi104 ceph-mon[97044]: pgmap v2441: 129 pgs: 129 active+clean; 7.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:45:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:18.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:18.140439+0000) 2022-09-10T05:45:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:18 smithi110 ceph-mon[97633]: pgmap v2441: 129 pgs: 129 active+clean; 7.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:45:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:18.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:18.737316+0000) 2022-09-10T05:45:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:19.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:19.140624+0000) 2022-09-10T05:45:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:19.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:19.737515+0000) 2022-09-10T05:45:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:20 smithi104 ceph-mon[97044]: pgmap v2442: 129 pgs: 129 active+clean; 7.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:45:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:20.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:20.140817+0000) 2022-09-10T05:45:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:20 smithi110 ceph-mon[97633]: pgmap v2442: 129 pgs: 129 active+clean; 7.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:45:21.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:20.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:20.737770+0000) 2022-09-10T05:45:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:21.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:21.141008+0000) 2022-09-10T05:45:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:21.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:21.737958+0000) 2022-09-10T05:45:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:22 smithi104 ceph-mon[97044]: pgmap v2443: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:45:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:22.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:22.141252+0000) 2022-09-10T05:45:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:22 smithi110 ceph-mon[97633]: pgmap v2443: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:45:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:22.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:22.738203+0000) 2022-09-10T05:45:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:23.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:23.141495+0000) 2022-09-10T05:45:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:23.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:23.738453+0000) 2022-09-10T05:45:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:24 smithi110 ceph-mon[97633]: pgmap v2444: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:45:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:24 smithi104 ceph-mon[97044]: pgmap v2444: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:45:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:24.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:24.141654+0000) 2022-09-10T05:45:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:24.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:24.738607+0000) 2022-09-10T05:45:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:25.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:25.141832+0000) 2022-09-10T05:45:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:25.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:25.738849+0000) 2022-09-10T05:45:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:26 smithi104 ceph-mon[97044]: pgmap v2445: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:45:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:26.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:26.142055+0000) 2022-09-10T05:45:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:26 smithi110 ceph-mon[97633]: pgmap v2445: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:45:27.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:45:26.624+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:45:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:26.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:26.739138+0000) 2022-09-10T05:45:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:27.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:27.142239+0000) 2022-09-10T05:45:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:27.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:27.739346+0000) 2022-09-10T05:45:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:28 smithi104 ceph-mon[97044]: pgmap v2446: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:28.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:28.142463+0000) 2022-09-10T05:45:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:28 smithi110 ceph-mon[97633]: pgmap v2446: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:28.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:28.739507+0000) 2022-09-10T05:45:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:29.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:29.142660+0000) 2022-09-10T05:45:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:29.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:29.739709+0000) 2022-09-10T05:45:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:30 smithi104 ceph-mon[97044]: pgmap v2447: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:30.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:30.142922+0000) 2022-09-10T05:45:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:30 smithi110 ceph-mon[97633]: pgmap v2447: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:30.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:30.739879+0000) 2022-09-10T05:45:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:31.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:31.143193+0000) 2022-09-10T05:45:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:31.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:31.740066+0000) 2022-09-10T05:45:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:32 smithi104 ceph-mon[97044]: pgmap v2448: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:32.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:32.143466+0000) 2022-09-10T05:45:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:32 smithi110 ceph-mon[97633]: pgmap v2448: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:32.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:32.740309+0000) 2022-09-10T05:45:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:33.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:33.143709+0000) 2022-09-10T05:45:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:33.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:33.740478+0000) 2022-09-10T05:45:34.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:34 smithi110 ceph-mon[97633]: pgmap v2449: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:34 smithi104 ceph-mon[97044]: pgmap v2449: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:34.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:34.143906+0000) 2022-09-10T05:45:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:34.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:34.740691+0000) 2022-09-10T05:45:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:35.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:35.144151+0000) 2022-09-10T05:45:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:35.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:35.740859+0000) 2022-09-10T05:45:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:36 smithi104 ceph-mon[97044]: pgmap v2450: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:36.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:36.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:36.144323+0000) 2022-09-10T05:45:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:36 smithi110 ceph-mon[97633]: pgmap v2450: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:36.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:36.741017+0000) 2022-09-10T05:45:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:37.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:37.144574+0000) 2022-09-10T05:45:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:37.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:37.741251+0000) 2022-09-10T05:45:38.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:38 smithi104 ceph-mon[97044]: pgmap v2451: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:38.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:38.144810+0000) 2022-09-10T05:45:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:38 smithi110 ceph-mon[97633]: pgmap v2451: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:38.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:38.741455+0000) 2022-09-10T05:45:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:39.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:39.145026+0000) 2022-09-10T05:45:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:39.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:39.741691+0000) 2022-09-10T05:45:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:40 smithi104 ceph-mon[97044]: pgmap v2452: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:40.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:40.145219+0000) 2022-09-10T05:45:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:40 smithi110 ceph-mon[97633]: pgmap v2452: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:40.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:40.741896+0000) 2022-09-10T05:45:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:41.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:41.145406+0000) 2022-09-10T05:45:42.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:45:41.635+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:45:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:41.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:41.742051+0000) 2022-09-10T05:45:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:42 smithi104 ceph-mon[97044]: pgmap v2453: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:42.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:42.145623+0000) 2022-09-10T05:45:42.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:42 smithi110 ceph-mon[97633]: pgmap v2453: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:42.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:42.742255+0000) 2022-09-10T05:45:43.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:43.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:43.145873+0000) 2022-09-10T05:45:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:43.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:43.742402+0000) 2022-09-10T05:45:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:44 smithi110 ceph-mon[97633]: pgmap v2454: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:45:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:45:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:45:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:45:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:44 smithi104 ceph-mon[97044]: pgmap v2454: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:45:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:45:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:45:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:45:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:44.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:44.146060+0000) 2022-09-10T05:45:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:44.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:44.742675+0000) 2022-09-10T05:45:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:45.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:45.146276+0000) 2022-09-10T05:45:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:45.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:45.742894+0000) 2022-09-10T05:45:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:46 smithi104 ceph-mon[97044]: pgmap v2455: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:46.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:46.146535+0000) 2022-09-10T05:45:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:46 smithi110 ceph-mon[97633]: pgmap v2455: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:46.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:46.743090+0000) 2022-09-10T05:45:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:47.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:47.146807+0000) 2022-09-10T05:45:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:47.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:47.743293+0000) 2022-09-10T05:45:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:48 smithi104 ceph-mon[97044]: pgmap v2456: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:48.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:48.147028+0000) 2022-09-10T05:45:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:48 smithi110 ceph-mon[97633]: pgmap v2456: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:48.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:48.743497+0000) 2022-09-10T05:45:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:49.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:49.147196+0000) 2022-09-10T05:45:50.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:49.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:49.743745+0000) 2022-09-10T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:50 smithi104 ceph-mon[97044]: pgmap v2457: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:50.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:50.147390+0000) 2022-09-10T05:45:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:50 smithi110 ceph-mon[97633]: pgmap v2457: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:50.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:50.743961+0000) 2022-09-10T05:45:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:51.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:51.147563+0000) 2022-09-10T05:45:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:51.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:51.744131+0000) 2022-09-10T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:52 smithi104 ceph-mon[97044]: pgmap v2458: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:52.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:52.147805+0000) 2022-09-10T05:45:52.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:52 smithi110 ceph-mon[97633]: pgmap v2458: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:52.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:52.744332+0000) 2022-09-10T05:45:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:53.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:53.148038+0000) 2022-09-10T05:45:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:53.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:53.744514+0000) 2022-09-10T05:45:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:54 smithi110 ceph-mon[97633]: pgmap v2459: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:54 smithi104 ceph-mon[97044]: pgmap v2459: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:54.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:54.148338+0000) 2022-09-10T05:45:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:54.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:54.744712+0000) 2022-09-10T05:45:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:55.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:55.148536+0000) 2022-09-10T05:45:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:55.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:55.744873+0000) 2022-09-10T05:45:56.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:56 smithi104 ceph-mon[97044]: pgmap v2460: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:56.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:56.148714+0000) 2022-09-10T05:45:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:56 smithi110 ceph-mon[97633]: pgmap v2460: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:45:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:56.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:56.745088+0000) 2022-09-10T05:45:57.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:45:56.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:45:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:57.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:57.148975+0000) 2022-09-10T05:45:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:57.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:57.745355+0000) 2022-09-10T05:45:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:45:58 smithi104 ceph-mon[97044]: pgmap v2461: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:58.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:58.149202+0000) 2022-09-10T05:45:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:45:58 smithi110 ceph-mon[97633]: pgmap v2461: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:45:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:58.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:58.745536+0000) 2022-09-10T05:45:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:45:59.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:59.149444+0000) 2022-09-10T05:46:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:45:59.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:45:59.745714+0000) 2022-09-10T05:46:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:00 smithi104 ceph-mon[97044]: pgmap v2462: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:00.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:00.149691+0000) 2022-09-10T05:46:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:00 smithi110 ceph-mon[97633]: pgmap v2462: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:00.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:00.745883+0000) 2022-09-10T05:46:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:01.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:01.149941+0000) 2022-09-10T05:46:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:01.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:01.746134+0000) 2022-09-10T05:46:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:02 smithi104 ceph-mon[97044]: pgmap v2463: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:02.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:02.150185+0000) 2022-09-10T05:46:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:02 smithi110 ceph-mon[97633]: pgmap v2463: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:02.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:02.746328+0000) 2022-09-10T05:46:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:03.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:03.150392+0000) 2022-09-10T05:46:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:03.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:03.746528+0000) 2022-09-10T05:46:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:04 smithi110 ceph-mon[97633]: pgmap v2464: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:04 smithi104 ceph-mon[97044]: pgmap v2464: 129 pgs: 129 active+clean; 7.9 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:04.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:04.150570+0000) 2022-09-10T05:46:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:04.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:04.746700+0000) 2022-09-10T05:46:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:05.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:05.150805+0000) 2022-09-10T05:46:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:05.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:05.746868+0000) 2022-09-10T05:46:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:06 smithi104 ceph-mon[97044]: pgmap v2465: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:46:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:06.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:06.150984+0000) 2022-09-10T05:46:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:06 smithi110 ceph-mon[97633]: pgmap v2465: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:46:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:06.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:06.747134+0000) 2022-09-10T05:46:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:07.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:07.151223+0000) 2022-09-10T05:46:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:07.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:07.747316+0000) 2022-09-10T05:46:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:08 smithi104 ceph-mon[97044]: pgmap v2466: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:46:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:08 smithi104 ceph-mon[97044]: osdmap e238: 12 total, 12 up, 12 in 2022-09-10T05:46:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:08.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:08.151468+0000) 2022-09-10T05:46:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:08 smithi110 ceph-mon[97633]: pgmap v2466: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:46:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:08 smithi110 ceph-mon[97633]: osdmap e238: 12 total, 12 up, 12 in 2022-09-10T05:46:09.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:08.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:08.747547+0000) 2022-09-10T05:46:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:09.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:09.151749+0000) 2022-09-10T05:46:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:09.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:09.747817+0000) 2022-09-10T05:46:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:10 smithi104 ceph-mon[97044]: pgmap v2468: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:46:10.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:10.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:10.152020+0000) 2022-09-10T05:46:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:10 smithi110 ceph-mon[97633]: pgmap v2468: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:46:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:10.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:10.747981+0000) 2022-09-10T05:46:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:11.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:11.152282+0000) 2022-09-10T05:46:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:11.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:11.748157+0000) 2022-09-10T05:46:12.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:46:11.648+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:46:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:12 smithi104 ceph-mon[97044]: pgmap v2469: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:46:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:12.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:12.152520+0000) 2022-09-10T05:46:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:12 smithi110 ceph-mon[97633]: pgmap v2469: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:46:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:12.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:12.748285+0000) 2022-09-10T05:46:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:13.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:13.152720+0000) 2022-09-10T05:46:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:13.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:13.748510+0000) 2022-09-10T05:46:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:14 smithi110 ceph-mon[97633]: osdmap e239: 12 total, 12 up, 12 in 2022-09-10T05:46:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:14 smithi110 ceph-mon[97633]: pgmap v2471: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:46:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:14 smithi104 ceph-mon[97044]: osdmap e239: 12 total, 12 up, 12 in 2022-09-10T05:46:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:14 smithi104 ceph-mon[97044]: pgmap v2471: 129 pgs: 129 active+clean; 7.9 MiB data, 423 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:46:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:14.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:14.153006+0000) 2022-09-10T05:46:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:14.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:14.748686+0000) 2022-09-10T05:46:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:15.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:15.153247+0000) 2022-09-10T05:46:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:15.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:15.748936+0000) 2022-09-10T05:46:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:16 smithi104 ceph-mon[97044]: pgmap v2472: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:46:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:16.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:16.153496+0000) 2022-09-10T05:46:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:16 smithi110 ceph-mon[97633]: pgmap v2472: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:46:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:16.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:16.749189+0000) 2022-09-10T05:46:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:17.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:17.153756+0000) 2022-09-10T05:46:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:17.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:17.749348+0000) 2022-09-10T05:46:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:18 smithi104 ceph-mon[97044]: pgmap v2473: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:46:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:18.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:18.154019+0000) 2022-09-10T05:46:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:18 smithi110 ceph-mon[97633]: pgmap v2473: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:46:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:18.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:18.749506+0000) 2022-09-10T05:46:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:19.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:19.154284+0000) 2022-09-10T05:46:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:19.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:19.749758+0000) 2022-09-10T05:46:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:20 smithi104 ceph-mon[97044]: pgmap v2474: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:46:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:20.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:20.154551+0000) 2022-09-10T05:46:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:20 smithi110 ceph-mon[97633]: pgmap v2474: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:46:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:20.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:20.750021+0000) 2022-09-10T05:46:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:21.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:21.154755+0000) 2022-09-10T05:46:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:21.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:21.750255+0000) 2022-09-10T05:46:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:22 smithi104 ceph-mon[97044]: pgmap v2475: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:46:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:22.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:22.154974+0000) 2022-09-10T05:46:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:22 smithi110 ceph-mon[97633]: pgmap v2475: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:46:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:22.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:22.750491+0000) 2022-09-10T05:46:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:23.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:23.155223+0000) 2022-09-10T05:46:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:23.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:23.750674+0000) 2022-09-10T05:46:24.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:24 smithi110 ceph-mon[97633]: pgmap v2476: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:46:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:24 smithi104 ceph-mon[97044]: pgmap v2476: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:46:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:24.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:24.155466+0000) 2022-09-10T05:46:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:24.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:24.750929+0000) 2022-09-10T05:46:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:25.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:25.155669+0000) 2022-09-10T05:46:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:25.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:25.751190+0000) 2022-09-10T05:46:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:26 smithi104 ceph-mon[97044]: pgmap v2477: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:46:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:26.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:26.155925+0000) 2022-09-10T05:46:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:26 smithi110 ceph-mon[97633]: pgmap v2477: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:46:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:26.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:26.751389+0000) 2022-09-10T05:46:27.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:46:26.661+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:46:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:27.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:27.156192+0000) 2022-09-10T05:46:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:27.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:27.751568+0000) 2022-09-10T05:46:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:28 smithi104 ceph-mon[97044]: pgmap v2478: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:28.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:28.156327+0000) 2022-09-10T05:46:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:28 smithi110 ceph-mon[97633]: pgmap v2478: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:28.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:28.751791+0000) 2022-09-10T05:46:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:29.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:29.156497+0000) 2022-09-10T05:46:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:29.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:29.752053+0000) 2022-09-10T05:46:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:30 smithi104 ceph-mon[97044]: pgmap v2479: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:30.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:30.156683+0000) 2022-09-10T05:46:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:30 smithi110 ceph-mon[97633]: pgmap v2479: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:30.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:30.752215+0000) 2022-09-10T05:46:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:31.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:31.156914+0000) 2022-09-10T05:46:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:31.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:31.752435+0000) 2022-09-10T05:46:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:32 smithi104 ceph-mon[97044]: pgmap v2480: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:32.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:32.157180+0000) 2022-09-10T05:46:32.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:32 smithi110 ceph-mon[97633]: pgmap v2480: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:32.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:32.752631+0000) 2022-09-10T05:46:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:33.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:33.157370+0000) 2022-09-10T05:46:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:33.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:33.752859+0000) 2022-09-10T05:46:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:34 smithi110 ceph-mon[97633]: pgmap v2481: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:34 smithi104 ceph-mon[97044]: pgmap v2481: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:34.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:34.157608+0000) 2022-09-10T05:46:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:34.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:34.753086+0000) 2022-09-10T05:46:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:35.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:35.157884+0000) 2022-09-10T05:46:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:35.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:35.753281+0000) 2022-09-10T05:46:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:36 smithi104 ceph-mon[97044]: pgmap v2482: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:36.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:36.158083+0000) 2022-09-10T05:46:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:36 smithi110 ceph-mon[97633]: pgmap v2482: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:36.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:36.753469+0000) 2022-09-10T05:46:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:37.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:37.158277+0000) 2022-09-10T05:46:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:37.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:37.753697+0000) 2022-09-10T05:46:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:38 smithi104 ceph-mon[97044]: pgmap v2483: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:38.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:38.158470+0000) 2022-09-10T05:46:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:38 smithi110 ceph-mon[97633]: pgmap v2483: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:38.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:38.753923+0000) 2022-09-10T05:46:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:39.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:39.158681+0000) 2022-09-10T05:46:40.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:39.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:39.754159+0000) 2022-09-10T05:46:40.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:40 smithi104 ceph-mon[97044]: pgmap v2484: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:40.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:40.158863+0000) 2022-09-10T05:46:40.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:40 smithi110 ceph-mon[97633]: pgmap v2484: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:40.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:40.754382+0000) 2022-09-10T05:46:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:41.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:41.159076+0000) 2022-09-10T05:46:42.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:46:41.670+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:46:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:41.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:41.754624+0000) 2022-09-10T05:46:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:42 smithi104 ceph-mon[97044]: pgmap v2485: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:42.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:42.159281+0000) 2022-09-10T05:46:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:42 smithi110 ceph-mon[97633]: pgmap v2485: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:42.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:42.754881+0000) 2022-09-10T05:46:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:43.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:43.159502+0000) 2022-09-10T05:46:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:43.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:43.755120+0000) 2022-09-10T05:46:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:44 smithi104 ceph-mon[97044]: pgmap v2486: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:46:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:46:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:46:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:46:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:44.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:44.159706+0000) 2022-09-10T05:46:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:44 smithi110 ceph-mon[97633]: pgmap v2486: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:46:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:46:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:46:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:46:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:44.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:44.755335+0000) 2022-09-10T05:46:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:45.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:45.159865+0000) 2022-09-10T05:46:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:45.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:45.755490+0000) 2022-09-10T05:46:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:46 smithi104 ceph-mon[97044]: pgmap v2487: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:46.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:46.160183+0000) 2022-09-10T05:46:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:46 smithi110 ceph-mon[97633]: pgmap v2487: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:46.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:46.755649+0000) 2022-09-10T05:46:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:47.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:47.160419+0000) 2022-09-10T05:46:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:47.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:47.755904+0000) 2022-09-10T05:46:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:48 smithi104 ceph-mon[97044]: pgmap v2488: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:48.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:48.160566+0000) 2022-09-10T05:46:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:48 smithi110 ceph-mon[97633]: pgmap v2488: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:48.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:48.756187+0000) 2022-09-10T05:46:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:49.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:49.160795+0000) 2022-09-10T05:46:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:49.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:49.756369+0000) 2022-09-10T05:46:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:50 smithi104 ceph-mon[97044]: pgmap v2489: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:50.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:50.161004+0000) 2022-09-10T05:46:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:50 smithi110 ceph-mon[97633]: pgmap v2489: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:50.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:50.756528+0000) 2022-09-10T05:46:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:51.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:51.161258+0000) 2022-09-10T05:46:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:51.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:51.756662+0000) 2022-09-10T05:46:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:52 smithi104 ceph-mon[97044]: pgmap v2490: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:52.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:52.161460+0000) 2022-09-10T05:46:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:52 smithi110 ceph-mon[97633]: pgmap v2490: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:52.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:52.756888+0000) 2022-09-10T05:46:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:53.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:53.161657+0000) 2022-09-10T05:46:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:53.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:53.757171+0000) 2022-09-10T05:46:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:54 smithi104 ceph-mon[97044]: pgmap v2491: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:54.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:54.161845+0000) 2022-09-10T05:46:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:54 smithi110 ceph-mon[97633]: pgmap v2491: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:54.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:54.757343+0000) 2022-09-10T05:46:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:55.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:55.162065+0000) 2022-09-10T05:46:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:55.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:55.757629+0000) 2022-09-10T05:46:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:56 smithi104 ceph-mon[97044]: pgmap v2492: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:56.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:56.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:56.162260+0000) 2022-09-10T05:46:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:56 smithi110 ceph-mon[97633]: pgmap v2492: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:46:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:56.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:56.757916+0000) 2022-09-10T05:46:57.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:46:56.681+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:46:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:57.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:57.162519+0000) 2022-09-10T05:46:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:57.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:57.758242+0000) 2022-09-10T05:46:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:46:58 smithi104 ceph-mon[97044]: pgmap v2493: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:58.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:58.162768+0000) 2022-09-10T05:46:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:46:58 smithi110 ceph-mon[97633]: pgmap v2493: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:46:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:58.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:58.758488+0000) 2022-09-10T05:46:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:46:59.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:59.163077+0000) 2022-09-10T05:47:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:46:59.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:46:59.758732+0000) 2022-09-10T05:47:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:00 smithi104 ceph-mon[97044]: pgmap v2494: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:00.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:00.163315+0000) 2022-09-10T05:47:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:00 smithi110 ceph-mon[97633]: pgmap v2494: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:00.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:00.758990+0000) 2022-09-10T05:47:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:01.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:01.163582+0000) 2022-09-10T05:47:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:01.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:01.759252+0000) 2022-09-10T05:47:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:02 smithi104 ceph-mon[97044]: pgmap v2495: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:02.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:02.163799+0000) 2022-09-10T05:47:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:02 smithi110 ceph-mon[97633]: pgmap v2495: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:02.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:02.759495+0000) 2022-09-10T05:47:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:03.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:03.164015+0000) 2022-09-10T05:47:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:03.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:03.759714+0000) 2022-09-10T05:47:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:04 smithi104 ceph-mon[97044]: pgmap v2496: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:04.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:04.164183+0000) 2022-09-10T05:47:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:04 smithi110 ceph-mon[97633]: pgmap v2496: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:04.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:04.759959+0000) 2022-09-10T05:47:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:05.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:05.164332+0000) 2022-09-10T05:47:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:05.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:05.760229+0000) 2022-09-10T05:47:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:06 smithi104 ceph-mon[97044]: pgmap v2497: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:47:06.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:06.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:06.164577+0000) 2022-09-10T05:47:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:06 smithi110 ceph-mon[97633]: pgmap v2497: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:47:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:06.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:06.760458+0000) 2022-09-10T05:47:07.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:07.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:07.164747+0000) 2022-09-10T05:47:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:07.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:07.760721+0000) 2022-09-10T05:47:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:08 smithi104 ceph-mon[97044]: pgmap v2498: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:47:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:08 smithi104 ceph-mon[97044]: osdmap e240: 12 total, 12 up, 12 in 2022-09-10T05:47:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:08.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:08.164991+0000) 2022-09-10T05:47:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:08 smithi110 ceph-mon[97633]: pgmap v2498: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:47:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:08 smithi110 ceph-mon[97633]: osdmap e240: 12 total, 12 up, 12 in 2022-09-10T05:47:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:08.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:08.760890+0000) 2022-09-10T05:47:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:09.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:09.165272+0000) 2022-09-10T05:47:10.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:09.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:09.761120+0000) 2022-09-10T05:47:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:10 smithi104 ceph-mon[97044]: pgmap v2500: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:47:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:10.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:10.165496+0000) 2022-09-10T05:47:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:10 smithi110 ceph-mon[97633]: pgmap v2500: 129 pgs: 129 active+clean; 7.9 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:47:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:10.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:10.761370+0000) 2022-09-10T05:47:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:11.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:11.165655+0000) 2022-09-10T05:47:12.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:47:11.697+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:47:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:11.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:11.761600+0000) 2022-09-10T05:47:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:12 smithi104 ceph-mon[97044]: pgmap v2501: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:47:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:12.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:12.165842+0000) 2022-09-10T05:47:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:12 smithi110 ceph-mon[97633]: pgmap v2501: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:47:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:12.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:12.761791+0000) 2022-09-10T05:47:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:13.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:13.165998+0000) 2022-09-10T05:47:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:13.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:13.761979+0000) 2022-09-10T05:47:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:14 smithi104 ceph-mon[97044]: osdmap e241: 12 total, 12 up, 12 in 2022-09-10T05:47:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:14 smithi104 ceph-mon[97044]: pgmap v2503: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:47:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:14.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:14.166190+0000) 2022-09-10T05:47:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:14 smithi110 ceph-mon[97633]: osdmap e241: 12 total, 12 up, 12 in 2022-09-10T05:47:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:14 smithi110 ceph-mon[97633]: pgmap v2503: 129 pgs: 129 active+clean; 7.9 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:47:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:14.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:14.762143+0000) 2022-09-10T05:47:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:15.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:15.166354+0000) 2022-09-10T05:47:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:15.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:15.762287+0000) 2022-09-10T05:47:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:16 smithi104 ceph-mon[97044]: pgmap v2504: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:47:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:16.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:16.166519+0000) 2022-09-10T05:47:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:16 smithi110 ceph-mon[97633]: pgmap v2504: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:47:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:16.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:16.762435+0000) 2022-09-10T05:47:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:17.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:17.166722+0000) 2022-09-10T05:47:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:17.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:17.762645+0000) 2022-09-10T05:47:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:18 smithi104 ceph-mon[97044]: pgmap v2505: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:47:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:18.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:18.166924+0000) 2022-09-10T05:47:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:18 smithi110 ceph-mon[97633]: pgmap v2505: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:47:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:18.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:18.762916+0000) 2022-09-10T05:47:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:19.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:19.167139+0000) 2022-09-10T05:47:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:19.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:19.763196+0000) 2022-09-10T05:47:20.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:20 smithi104 ceph-mon[97044]: pgmap v2506: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:47:20.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:20.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:20.167377+0000) 2022-09-10T05:47:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:20 smithi110 ceph-mon[97633]: pgmap v2506: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:47:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:20.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:20.763442+0000) 2022-09-10T05:47:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:21.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:21.167633+0000) 2022-09-10T05:47:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:21.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:21.763638+0000) 2022-09-10T05:47:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:22 smithi104 ceph-mon[97044]: pgmap v2507: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:47:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:22.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:22.167850+0000) 2022-09-10T05:47:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:22 smithi110 ceph-mon[97633]: pgmap v2507: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:47:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:22.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:22.763786+0000) 2022-09-10T05:47:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:23.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:23.168061+0000) 2022-09-10T05:47:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:23.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:23.764014+0000) 2022-09-10T05:47:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:24 smithi104 ceph-mon[97044]: pgmap v2508: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:47:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:24.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:24.168209+0000) 2022-09-10T05:47:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:24 smithi110 ceph-mon[97633]: pgmap v2508: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:47:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:24.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:24.764257+0000) 2022-09-10T05:47:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:25.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:25.168406+0000) 2022-09-10T05:47:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:25.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:25.764485+0000) 2022-09-10T05:47:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:26 smithi104 ceph-mon[97044]: pgmap v2509: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:47:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:26.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:26.168588+0000) 2022-09-10T05:47:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:26 smithi110 ceph-mon[97633]: pgmap v2509: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:47:27.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:47:26.713+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:47:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:26.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:26.764701+0000) 2022-09-10T05:47:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:27.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:27.168838+0000) 2022-09-10T05:47:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:27.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:27.764955+0000) 2022-09-10T05:47:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:28 smithi104 ceph-mon[97044]: pgmap v2510: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:28.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:28.169146+0000) 2022-09-10T05:47:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:28 smithi110 ceph-mon[97633]: pgmap v2510: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:28.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:28.765204+0000) 2022-09-10T05:47:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:29.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:29.169326+0000) 2022-09-10T05:47:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:29.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:29.765392+0000) 2022-09-10T05:47:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:30 smithi104 ceph-mon[97044]: pgmap v2511: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:30.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:30.169531+0000) 2022-09-10T05:47:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:30 smithi110 ceph-mon[97633]: pgmap v2511: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:30.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:30.765614+0000) 2022-09-10T05:47:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:31.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:31.169752+0000) 2022-09-10T05:47:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:31.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:31.765879+0000) 2022-09-10T05:47:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:32 smithi104 ceph-mon[97044]: pgmap v2512: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:32.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:32.169993+0000) 2022-09-10T05:47:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:32 smithi110 ceph-mon[97633]: pgmap v2512: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:32.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:32.766173+0000) 2022-09-10T05:47:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:33.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:33.170190+0000) 2022-09-10T05:47:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:33.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:33.766348+0000) 2022-09-10T05:47:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:34 smithi104 ceph-mon[97044]: pgmap v2513: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:34.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:34.170453+0000) 2022-09-10T05:47:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:34 smithi110 ceph-mon[97633]: pgmap v2513: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:34.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:34.766561+0000) 2022-09-10T05:47:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:35.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:35.170664+0000) 2022-09-10T05:47:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:35.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:35.766847+0000) 2022-09-10T05:47:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:36 smithi104 ceph-mon[97044]: pgmap v2514: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:36.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:36.170847+0000) 2022-09-10T05:47:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:36 smithi110 ceph-mon[97633]: pgmap v2514: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:36.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:36.767156+0000) 2022-09-10T05:47:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:37.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:37.171066+0000) 2022-09-10T05:47:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:37.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:37.767422+0000) 2022-09-10T05:47:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:38 smithi104 ceph-mon[97044]: pgmap v2515: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:38.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:38.171308+0000) 2022-09-10T05:47:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:38 smithi110 ceph-mon[97633]: pgmap v2515: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:38.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:38.767615+0000) 2022-09-10T05:47:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:39.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:39.171621+0000) 2022-09-10T05:47:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:39.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:39.767872+0000) 2022-09-10T05:47:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:40 smithi104 ceph-mon[97044]: pgmap v2516: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:40.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:40.171788+0000) 2022-09-10T05:47:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:40 smithi110 ceph-mon[97633]: pgmap v2516: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:40.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:40.768066+0000) 2022-09-10T05:47:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:41.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:41.172019+0000) 2022-09-10T05:47:42.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:47:41.729+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:47:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:41.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:41.768266+0000) 2022-09-10T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:42 smithi104 ceph-mon[97044]: pgmap v2517: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:42.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:42.172261+0000) 2022-09-10T05:47:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:42 smithi110 ceph-mon[97633]: pgmap v2517: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:42.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:42.768511+0000) 2022-09-10T05:47:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:43.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:43.172507+0000) 2022-09-10T05:47:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:43.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:43.768768+0000) 2022-09-10T05:47:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:44 smithi104 ceph-mon[97044]: pgmap v2518: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:47:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:47:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:47:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:47:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:44.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:44.172799+0000) 2022-09-10T05:47:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:44 smithi110 ceph-mon[97633]: pgmap v2518: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:47:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:47:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:47:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:47:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:44.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:44.769013+0000) 2022-09-10T05:47:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:45.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:45.173028+0000) 2022-09-10T05:47:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:45.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:45.769231+0000) 2022-09-10T05:47:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:46 smithi104 ceph-mon[97044]: pgmap v2519: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:46.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:46.173277+0000) 2022-09-10T05:47:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:46 smithi110 ceph-mon[97633]: pgmap v2519: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:46.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:46.769460+0000) 2022-09-10T05:47:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:47.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:47.173492+0000) 2022-09-10T05:47:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:47.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:47.769682+0000) 2022-09-10T05:47:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:48 smithi104 ceph-mon[97044]: pgmap v2520: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:48.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:48.173622+0000) 2022-09-10T05:47:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:48 smithi110 ceph-mon[97633]: pgmap v2520: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:48.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:48.769825+0000) 2022-09-10T05:47:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:49.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:49.173865+0000) 2022-09-10T05:47:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:49.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:49.770025+0000) 2022-09-10T05:47:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:50 smithi104 ceph-mon[97044]: pgmap v2521: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:50.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:50.174108+0000) 2022-09-10T05:47:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:50 smithi110 ceph-mon[97633]: pgmap v2521: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:50.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:50.770269+0000) 2022-09-10T05:47:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:51.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:51.174299+0000) 2022-09-10T05:47:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:51.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:51.770441+0000) 2022-09-10T05:47:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:52 smithi104 ceph-mon[97044]: pgmap v2522: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:52.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:52.174528+0000) 2022-09-10T05:47:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:52 smithi110 ceph-mon[97633]: pgmap v2522: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:52.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:52.770681+0000) 2022-09-10T05:47:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:53.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:53.174752+0000) 2022-09-10T05:47:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:53.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:53.770961+0000) 2022-09-10T05:47:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:54 smithi104 ceph-mon[97044]: pgmap v2523: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:54.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:54.175002+0000) 2022-09-10T05:47:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:54 smithi110 ceph-mon[97633]: pgmap v2523: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:54.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:54.771261+0000) 2022-09-10T05:47:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:55.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:55.175191+0000) 2022-09-10T05:47:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:55.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:55.771421+0000) 2022-09-10T05:47:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:56 smithi104 ceph-mon[97044]: pgmap v2524: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:56.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:56.175388+0000) 2022-09-10T05:47:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:56 smithi110 ceph-mon[97633]: pgmap v2524: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:47:57.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:47:56.740+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:47:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:56.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:56.771632+0000) 2022-09-10T05:47:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:57.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:57.175519+0000) 2022-09-10T05:47:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:57.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:57.771888+0000) 2022-09-10T05:47:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:47:58 smithi104 ceph-mon[97044]: pgmap v2525: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:58.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:58.175739+0000) 2022-09-10T05:47:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:47:58 smithi110 ceph-mon[97633]: pgmap v2525: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:47:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:58.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:58.772168+0000) 2022-09-10T05:47:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:47:59.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:59.175981+0000) 2022-09-10T05:48:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:47:59.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:47:59.772351+0000) 2022-09-10T05:48:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:00 smithi104 ceph-mon[97044]: pgmap v2526: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:00.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:00.176231+0000) 2022-09-10T05:48:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:00 smithi110 ceph-mon[97633]: pgmap v2526: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:00.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:00.772542+0000) 2022-09-10T05:48:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:01.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:01.176447+0000) 2022-09-10T05:48:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:01.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:01.772770+0000) 2022-09-10T05:48:02.508 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:02.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:02.176712+0000) 2022-09-10T05:48:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:02 smithi104 ceph-mon[97044]: pgmap v2527: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:02 smithi110 ceph-mon[97633]: pgmap v2527: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:02.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:02.773024+0000) 2022-09-10T05:48:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:03.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:03.176922+0000) 2022-09-10T05:48:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:03.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:03.773258+0000) 2022-09-10T05:48:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:04 smithi104 ceph-mon[97044]: pgmap v2528: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:04.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:04.177189+0000) 2022-09-10T05:48:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:04 smithi110 ceph-mon[97633]: pgmap v2528: 129 pgs: 129 active+clean; 7.9 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:04.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:04.773449+0000) 2022-09-10T05:48:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:05.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:05.177431+0000) 2022-09-10T05:48:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:05.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:05.773682+0000) 2022-09-10T05:48:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:06 smithi104 ceph-mon[97044]: pgmap v2529: 129 pgs: 129 active+clean; 7.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:48:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:06.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:06.177690+0000) 2022-09-10T05:48:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:06 smithi110 ceph-mon[97633]: pgmap v2529: 129 pgs: 129 active+clean; 7.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:48:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:06.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:06.773926+0000) 2022-09-10T05:48:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:07.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:07.177920+0000) 2022-09-10T05:48:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:07.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:07.774213+0000) 2022-09-10T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:08 smithi104 ceph-mon[97044]: pgmap v2530: 129 pgs: 129 active+clean; 7.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:08 smithi104 ceph-mon[97044]: osdmap e242: 12 total, 12 up, 12 in 2022-09-10T05:48:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:08.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:08.178200+0000) 2022-09-10T05:48:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:08 smithi110 ceph-mon[97633]: pgmap v2530: 129 pgs: 129 active+clean; 7.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:48:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:08 smithi110 ceph-mon[97633]: osdmap e242: 12 total, 12 up, 12 in 2022-09-10T05:48:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:08.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:08.774408+0000) 2022-09-10T05:48:09.488 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:09.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:09.178399+0000) 2022-09-10T05:48:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:09.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:09.774575+0000) 2022-09-10T05:48:10.520 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:10.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:10.178572+0000) 2022-09-10T05:48:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:10 smithi110 ceph-mon[97633]: pgmap v2532: 129 pgs: 129 active+clean; 7.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:48:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:10 smithi104 ceph-mon[97044]: pgmap v2532: 129 pgs: 129 active+clean; 7.9 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:48:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:10.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:10.774742+0000) 2022-09-10T05:48:11.541 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:11.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:11.178721+0000) 2022-09-10T05:48:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:11.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:11.774904+0000) 2022-09-10T05:48:11.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:48:11.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:48:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:12.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:12.178868+0000) 2022-09-10T05:48:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:12 smithi110 ceph-mon[97633]: pgmap v2533: 129 pgs: 129 active+clean; 7.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:48:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:12 smithi104 ceph-mon[97044]: pgmap v2533: 129 pgs: 129 active+clean; 7.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:48:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:12.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:12.775031+0000) 2022-09-10T05:48:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:13.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:13.179073+0000) 2022-09-10T05:48:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:13.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:13.775250+0000) 2022-09-10T05:48:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:14 smithi104 ceph-mon[97044]: osdmap e243: 12 total, 12 up, 12 in 2022-09-10T05:48:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:14 smithi104 ceph-mon[97044]: pgmap v2535: 129 pgs: 129 active+clean; 7.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:48:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:14.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:14.179312+0000) 2022-09-10T05:48:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:14 smithi110 ceph-mon[97633]: osdmap e243: 12 total, 12 up, 12 in 2022-09-10T05:48:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:14 smithi110 ceph-mon[97633]: pgmap v2535: 129 pgs: 129 active+clean; 7.9 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:48:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:14.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:14.775503+0000) 2022-09-10T05:48:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:15.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:15.179486+0000) 2022-09-10T05:48:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:15.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:15.775697+0000) 2022-09-10T05:48:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:16 smithi104 ceph-mon[97044]: pgmap v2536: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:48:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:16.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:16.179708+0000) 2022-09-10T05:48:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:16 smithi110 ceph-mon[97633]: pgmap v2536: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:48:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:16.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:16.775943+0000) 2022-09-10T05:48:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:17.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:17.179867+0000) 2022-09-10T05:48:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:17.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:17.776190+0000) 2022-09-10T05:48:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:18 smithi104 ceph-mon[97044]: pgmap v2537: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:48:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:18.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:18.180156+0000) 2022-09-10T05:48:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:18 smithi110 ceph-mon[97633]: pgmap v2537: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:48:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:18.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:18.776353+0000) 2022-09-10T05:48:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:19.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:19.180392+0000) 2022-09-10T05:48:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:19.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:19.776550+0000) 2022-09-10T05:48:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:20.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:20.180632+0000) 2022-09-10T05:48:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:20 smithi110 ceph-mon[97633]: pgmap v2538: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:48:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:20 smithi104 ceph-mon[97044]: pgmap v2538: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:48:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:20.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:20.776835+0000) 2022-09-10T05:48:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:21.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:21.180905+0000) 2022-09-10T05:48:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:21.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:21.777083+0000) 2022-09-10T05:48:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:22.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:22.181184+0000) 2022-09-10T05:48:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:22 smithi110 ceph-mon[97633]: pgmap v2539: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:48:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:22 smithi104 ceph-mon[97044]: pgmap v2539: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:48:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:22.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:22.777319+0000) 2022-09-10T05:48:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:23.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:23.181411+0000) 2022-09-10T05:48:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:23.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:23.777570+0000) 2022-09-10T05:48:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:24 smithi104 ceph-mon[97044]: pgmap v2540: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:48:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:24.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:24.181717+0000) 2022-09-10T05:48:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:24 smithi110 ceph-mon[97633]: pgmap v2540: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:48:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:24.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:24.777779+0000) 2022-09-10T05:48:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:25.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:25.181928+0000) 2022-09-10T05:48:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:25.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:25.777973+0000) 2022-09-10T05:48:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:26.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:26.182169+0000) 2022-09-10T05:48:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:26 smithi110 ceph-mon[97633]: pgmap v2541: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:48:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:26 smithi104 ceph-mon[97044]: pgmap v2541: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:48:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:26.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:26.778132+0000) 2022-09-10T05:48:27.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:48:26.766+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:48:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:27.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:27.182398+0000) 2022-09-10T05:48:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:27.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:27.778262+0000) 2022-09-10T05:48:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:28.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:28.182686+0000) 2022-09-10T05:48:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:28 smithi110 ceph-mon[97633]: pgmap v2542: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:28 smithi104 ceph-mon[97044]: pgmap v2542: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:28.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:28.778444+0000) 2022-09-10T05:48:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:29.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:29.182902+0000) 2022-09-10T05:48:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:29.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:29.778623+0000) 2022-09-10T05:48:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:30.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:30.183032+0000) 2022-09-10T05:48:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:30 smithi110 ceph-mon[97633]: pgmap v2543: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:30 smithi104 ceph-mon[97044]: pgmap v2543: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:30.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:30.778763+0000) 2022-09-10T05:48:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:31.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:31.183277+0000) 2022-09-10T05:48:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:31.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:31.778906+0000) 2022-09-10T05:48:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:32.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:32.183538+0000) 2022-09-10T05:48:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:32 smithi110 ceph-mon[97633]: pgmap v2544: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:32 smithi104 ceph-mon[97044]: pgmap v2544: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:32.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:32.779228+0000) 2022-09-10T05:48:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:33.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:33.183736+0000) 2022-09-10T05:48:34.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:33.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:33.779476+0000) 2022-09-10T05:48:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:34 smithi104 ceph-mon[97044]: pgmap v2545: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:34.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:34.183971+0000) 2022-09-10T05:48:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:34 smithi110 ceph-mon[97633]: pgmap v2545: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:34.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:34.779691+0000) 2022-09-10T05:48:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:35.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:35.184168+0000) 2022-09-10T05:48:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:35.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:35.779938+0000) 2022-09-10T05:48:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:36.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:36.184356+0000) 2022-09-10T05:48:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:36 smithi110 ceph-mon[97633]: pgmap v2546: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:36 smithi104 ceph-mon[97044]: pgmap v2546: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:36.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:36.780219+0000) 2022-09-10T05:48:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:37.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:37.184603+0000) 2022-09-10T05:48:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:37.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:37.780422+0000) 2022-09-10T05:48:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:38.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:38.184866+0000) 2022-09-10T05:48:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:38 smithi110 ceph-mon[97633]: pgmap v2547: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:38 smithi104 ceph-mon[97044]: pgmap v2547: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:38.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:38.780611+0000) 2022-09-10T05:48:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:39.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:39.185127+0000) 2022-09-10T05:48:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:39.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:39.780823+0000) 2022-09-10T05:48:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:40.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:40.185366+0000) 2022-09-10T05:48:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:40 smithi110 ceph-mon[97633]: pgmap v2548: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:40 smithi104 ceph-mon[97044]: pgmap v2548: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:40.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:40.781146+0000) 2022-09-10T05:48:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:41.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:41.185637+0000) 2022-09-10T05:48:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:41.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:41.781295+0000) 2022-09-10T05:48:42.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:48:41.779+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:48:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:42.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:42.185903+0000) 2022-09-10T05:48:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:42 smithi110 ceph-mon[97633]: pgmap v2549: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:42 smithi104 ceph-mon[97044]: pgmap v2549: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:42.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:42.781426+0000) 2022-09-10T05:48:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:43.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:43.186025+0000) 2022-09-10T05:48:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:43.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:43.781687+0000) 2022-09-10T05:48:44.376 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:44 smithi110 ceph-mon[97633]: pgmap v2550: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:44.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:48:44.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:48:44.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:48:44.377 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:48:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:44 smithi104 ceph-mon[97044]: pgmap v2550: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:48:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:48:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:48:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:48:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:44.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:44.186279+0000) 2022-09-10T05:48:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:44.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:44.781926+0000) 2022-09-10T05:48:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:45.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:45.186504+0000) 2022-09-10T05:48:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:45.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:45.782148+0000) 2022-09-10T05:48:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:46.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:46.186760+0000) 2022-09-10T05:48:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:46 smithi110 ceph-mon[97633]: pgmap v2551: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:46 smithi104 ceph-mon[97044]: pgmap v2551: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:46.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:46.782354+0000) 2022-09-10T05:48:47.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:47.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:47.186963+0000) 2022-09-10T05:48:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:47.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:47.782579+0000) 2022-09-10T05:48:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:48.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:48.187205+0000) 2022-09-10T05:48:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:48 smithi110 ceph-mon[97633]: pgmap v2552: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:48 smithi104 ceph-mon[97044]: pgmap v2552: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:48.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:48.782826+0000) 2022-09-10T05:48:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:49.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:49.187358+0000) 2022-09-10T05:48:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:49.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:49.783064+0000) 2022-09-10T05:48:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:50.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:50.187586+0000) 2022-09-10T05:48:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:50 smithi110 ceph-mon[97633]: pgmap v2553: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:50 smithi104 ceph-mon[97044]: pgmap v2553: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:50.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:50.783310+0000) 2022-09-10T05:48:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:51.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:51.187795+0000) 2022-09-10T05:48:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:51.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:51.783563+0000) 2022-09-10T05:48:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:52.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:52.188025+0000) 2022-09-10T05:48:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:52 smithi110 ceph-mon[97633]: pgmap v2554: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:52 smithi104 ceph-mon[97044]: pgmap v2554: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:52.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:52.783805+0000) 2022-09-10T05:48:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:53.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:53.188213+0000) 2022-09-10T05:48:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:53.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:53.784053+0000) 2022-09-10T05:48:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:54 smithi104 ceph-mon[97044]: pgmap v2555: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:54.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:54.188412+0000) 2022-09-10T05:48:54.586 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:54 smithi110 ceph-mon[97633]: pgmap v2555: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:54.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:54.784280+0000) 2022-09-10T05:48:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:55.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:55.188585+0000) 2022-09-10T05:48:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:55.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:55.784471+0000) 2022-09-10T05:48:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:56.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:56.188796+0000) 2022-09-10T05:48:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:56 smithi110 ceph-mon[97633]: pgmap v2556: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:56 smithi104 ceph-mon[97044]: pgmap v2556: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:48:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:56.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:56.784746+0000) 2022-09-10T05:48:57.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:48:56.790+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:48:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:57.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:57.189045+0000) 2022-09-10T05:48:58.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:57.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:57.784984+0000) 2022-09-10T05:48:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:58.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:58.189308+0000) 2022-09-10T05:48:58.668 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:48:58 smithi110 ceph-mon[97633]: pgmap v2557: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:48:58 smithi104 ceph-mon[97044]: pgmap v2557: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:48:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:58.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:58.785153+0000) 2022-09-10T05:48:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:48:59.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:59.189566+0000) 2022-09-10T05:49:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:48:59.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:48:59.785362+0000) 2022-09-10T05:49:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:00.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:00.189790+0000) 2022-09-10T05:49:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:00 smithi110 ceph-mon[97633]: pgmap v2558: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:00 smithi104 ceph-mon[97044]: pgmap v2558: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:00.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:00.785582+0000) 2022-09-10T05:49:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:01.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:01.190032+0000) 2022-09-10T05:49:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:01.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:01.785848+0000) 2022-09-10T05:49:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:02.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:02.190266+0000) 2022-09-10T05:49:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:02 smithi110 ceph-mon[97633]: pgmap v2559: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:02 smithi104 ceph-mon[97044]: pgmap v2559: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:02.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:02.786067+0000) 2022-09-10T05:49:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:03.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:03.190496+0000) 2022-09-10T05:49:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:03.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:03.786309+0000) 2022-09-10T05:49:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:04 smithi104 ceph-mon[97044]: pgmap v2560: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:04.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:04.190674+0000) 2022-09-10T05:49:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:04 smithi110 ceph-mon[97633]: pgmap v2560: 129 pgs: 129 active+clean; 7.9 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:04.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:04.786535+0000) 2022-09-10T05:49:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:05.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:05.190942+0000) 2022-09-10T05:49:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:05.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:05.786770+0000) 2022-09-10T05:49:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:06.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:06.191210+0000) 2022-09-10T05:49:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:06 smithi110 ceph-mon[97633]: pgmap v2561: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:49:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:06 smithi104 ceph-mon[97044]: pgmap v2561: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:49:07.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:06.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:06.786972+0000) 2022-09-10T05:49:07.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:07.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:07.191340+0000) 2022-09-10T05:49:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:07.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:07.787251+0000) 2022-09-10T05:49:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:08.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:08.191496+0000) 2022-09-10T05:49:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:08 smithi110 ceph-mon[97633]: pgmap v2562: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:49:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:08 smithi110 ceph-mon[97633]: osdmap e244: 12 total, 12 up, 12 in 2022-09-10T05:49:08.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:08 smithi104 ceph-mon[97044]: pgmap v2562: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:49:08.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:08 smithi104 ceph-mon[97044]: osdmap e244: 12 total, 12 up, 12 in 2022-09-10T05:49:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:08.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:08.787508+0000) 2022-09-10T05:49:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:09.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:09.191743+0000) 2022-09-10T05:49:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:09.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:09.787768+0000) 2022-09-10T05:49:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:10.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:10.191923+0000) 2022-09-10T05:49:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:10 smithi110 ceph-mon[97633]: pgmap v2564: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:49:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:10 smithi104 ceph-mon[97044]: pgmap v2564: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:49:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:10.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:10.788026+0000) 2022-09-10T05:49:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:11.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:11.192151+0000) 2022-09-10T05:49:12.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:49:11.805+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:49:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:11.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:11.788263+0000) 2022-09-10T05:49:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:12.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:12.192387+0000) 2022-09-10T05:49:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:12 smithi110 ceph-mon[97633]: pgmap v2565: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:49:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:12 smithi104 ceph-mon[97044]: pgmap v2565: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:49:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:12.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:12.788477+0000) 2022-09-10T05:49:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:13.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:13.192625+0000) 2022-09-10T05:49:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:13.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:13.788725+0000) 2022-09-10T05:49:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:14 smithi104 ceph-mon[97044]: osdmap e245: 12 total, 12 up, 12 in 2022-09-10T05:49:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:14 smithi104 ceph-mon[97044]: pgmap v2567: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:49:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:14.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:14.192892+0000) 2022-09-10T05:49:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:14 smithi110 ceph-mon[97633]: osdmap e245: 12 total, 12 up, 12 in 2022-09-10T05:49:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:14 smithi110 ceph-mon[97633]: pgmap v2567: 129 pgs: 129 active+clean; 7.9 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:49:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:14.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:14.788904+0000) 2022-09-10T05:49:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:15.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:15.193137+0000) 2022-09-10T05:49:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:15.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:15.789207+0000) 2022-09-10T05:49:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:16.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:16.193376+0000) 2022-09-10T05:49:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:16 smithi110 ceph-mon[97633]: pgmap v2568: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:49:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:16 smithi104 ceph-mon[97044]: pgmap v2568: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:49:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:16.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:16.789437+0000) 2022-09-10T05:49:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:17.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:17.193605+0000) 2022-09-10T05:49:18.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:17.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:17.789654+0000) 2022-09-10T05:49:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:18.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:18.193818+0000) 2022-09-10T05:49:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:18 smithi110 ceph-mon[97633]: pgmap v2569: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:49:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:18 smithi104 ceph-mon[97044]: pgmap v2569: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:49:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:18.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:18.789898+0000) 2022-09-10T05:49:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:19.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:19.194044+0000) 2022-09-10T05:49:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:19.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:19.790207+0000) 2022-09-10T05:49:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:20.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:20.194244+0000) 2022-09-10T05:49:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:20 smithi110 ceph-mon[97633]: pgmap v2570: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:49:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:20 smithi104 ceph-mon[97044]: pgmap v2570: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:49:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:20.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:20.790385+0000) 2022-09-10T05:49:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:21.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:21.194429+0000) 2022-09-10T05:49:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:21.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:21.790593+0000) 2022-09-10T05:49:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:22.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:22.194590+0000) 2022-09-10T05:49:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:22 smithi110 ceph-mon[97633]: pgmap v2571: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:49:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:22 smithi104 ceph-mon[97044]: pgmap v2571: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:49:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:22.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:22.790847+0000) 2022-09-10T05:49:23.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:23.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:23.194789+0000) 2022-09-10T05:49:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:23.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:23.791058+0000) 2022-09-10T05:49:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:24 smithi104 ceph-mon[97044]: pgmap v2572: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:49:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:24.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:24.194980+0000) 2022-09-10T05:49:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:24 smithi110 ceph-mon[97633]: pgmap v2572: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:49:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:24.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:24.791216+0000) 2022-09-10T05:49:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:25.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:25.195207+0000) 2022-09-10T05:49:26.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:25.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:25.791444+0000) 2022-09-10T05:49:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:26.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:26.195431+0000) 2022-09-10T05:49:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:26 smithi110 ceph-mon[97633]: pgmap v2573: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:49:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:26 smithi104 ceph-mon[97044]: pgmap v2573: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:49:27.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:49:26.812+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:49:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:26.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:26.791633+0000) 2022-09-10T05:49:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:27.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:27.195659+0000) 2022-09-10T05:49:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:27.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:27.791795+0000) 2022-09-10T05:49:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:28.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:28.195907+0000) 2022-09-10T05:49:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:28 smithi110 ceph-mon[97633]: pgmap v2574: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:28 smithi104 ceph-mon[97044]: pgmap v2574: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:28.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:28.792034+0000) 2022-09-10T05:49:29.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:29.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:29.196114+0000) 2022-09-10T05:49:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:29.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:29.792303+0000) 2022-09-10T05:49:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:30.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:30.196287+0000) 2022-09-10T05:49:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:30 smithi110 ceph-mon[97633]: pgmap v2575: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:30 smithi104 ceph-mon[97044]: pgmap v2575: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:30.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:30.792537+0000) 2022-09-10T05:49:31.563 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:31.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:31.196455+0000) 2022-09-10T05:49:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:31.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:31.792778+0000) 2022-09-10T05:49:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:32.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:32.196671+0000) 2022-09-10T05:49:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:32 smithi110 ceph-mon[97633]: pgmap v2576: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:32 smithi104 ceph-mon[97044]: pgmap v2576: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:32.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:32.792982+0000) 2022-09-10T05:49:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:33.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:33.196938+0000) 2022-09-10T05:49:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:33.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:33.793220+0000) 2022-09-10T05:49:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:34 smithi104 ceph-mon[97044]: pgmap v2577: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:34.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:34.197166+0000) 2022-09-10T05:49:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:34 smithi110 ceph-mon[97633]: pgmap v2577: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:34.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:34.793402+0000) 2022-09-10T05:49:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:35.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:35.197337+0000) 2022-09-10T05:49:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:35.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:35.793542+0000) 2022-09-10T05:49:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:36.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:36.197492+0000) 2022-09-10T05:49:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:36 smithi110 ceph-mon[97633]: pgmap v2578: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:36 smithi104 ceph-mon[97044]: pgmap v2578: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:36.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:36.793774+0000) 2022-09-10T05:49:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:37.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:37.197666+0000) 2022-09-10T05:49:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:37.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:37.794009+0000) 2022-09-10T05:49:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:38.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:38.197852+0000) 2022-09-10T05:49:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:38 smithi110 ceph-mon[97633]: pgmap v2579: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:38 smithi104 ceph-mon[97044]: pgmap v2579: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:38.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:38.794212+0000) 2022-09-10T05:49:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:39.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:39.198051+0000) 2022-09-10T05:49:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:39.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:39.794437+0000) 2022-09-10T05:49:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:40.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:40.198313+0000) 2022-09-10T05:49:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:40 smithi110 ceph-mon[97633]: pgmap v2580: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:40 smithi104 ceph-mon[97044]: pgmap v2580: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:40.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:40.794690+0000) 2022-09-10T05:49:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:41.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:41.198520+0000) 2022-09-10T05:49:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:41.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:41.794965+0000) 2022-09-10T05:49:42.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:49:41.820+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:49:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:42.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:42.198680+0000) 2022-09-10T05:49:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:42 smithi110 ceph-mon[97633]: pgmap v2581: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:42 smithi104 ceph-mon[97044]: pgmap v2581: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:42.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:42.795197+0000) 2022-09-10T05:49:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:43.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:43.198931+0000) 2022-09-10T05:49:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:43.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:43.795384+0000) 2022-09-10T05:49:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:44 smithi104 ceph-mon[97044]: pgmap v2582: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:49:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:49:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:49:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:49:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:44.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:44.199161+0000) 2022-09-10T05:49:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:44 smithi110 ceph-mon[97633]: pgmap v2582: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:49:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:49:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:49:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:49:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:44.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:44.795555+0000) 2022-09-10T05:49:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:45.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:45.199339+0000) 2022-09-10T05:49:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:45.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:45.795795+0000) 2022-09-10T05:49:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:46.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:46.199583+0000) 2022-09-10T05:49:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:46 smithi110 ceph-mon[97633]: pgmap v2583: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:46 smithi104 ceph-mon[97044]: pgmap v2583: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:47.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:46.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:46.796019+0000) 2022-09-10T05:49:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:47.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:47.199840+0000) 2022-09-10T05:49:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:47.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:47.796201+0000) 2022-09-10T05:49:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:48.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:48.200053+0000) 2022-09-10T05:49:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:48 smithi110 ceph-mon[97633]: pgmap v2584: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:48 smithi104 ceph-mon[97044]: pgmap v2584: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:48.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:48.796414+0000) 2022-09-10T05:49:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:49.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:49.200246+0000) 2022-09-10T05:49:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:49.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:49.796663+0000) 2022-09-10T05:49:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:50.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:50.200441+0000) 2022-09-10T05:49:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:50 smithi110 ceph-mon[97633]: pgmap v2585: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:50 smithi104 ceph-mon[97044]: pgmap v2585: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:50.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:50.796915+0000) 2022-09-10T05:49:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:51.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:51.200671+0000) 2022-09-10T05:49:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:51.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:51.797200+0000) 2022-09-10T05:49:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:52.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:52.200905+0000) 2022-09-10T05:49:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:52 smithi110 ceph-mon[97633]: pgmap v2586: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:52 smithi104 ceph-mon[97044]: pgmap v2586: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:52.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:52.797451+0000) 2022-09-10T05:49:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:53.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:53.201164+0000) 2022-09-10T05:49:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:53.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:53.797673+0000) 2022-09-10T05:49:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:54 smithi104 ceph-mon[97044]: pgmap v2587: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:54.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:54.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:54.201330+0000) 2022-09-10T05:49:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:54 smithi110 ceph-mon[97633]: pgmap v2587: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:55.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:54.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:54.797948+0000) 2022-09-10T05:49:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:55.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:55.201553+0000) 2022-09-10T05:49:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:55.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:55.798195+0000) 2022-09-10T05:49:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:56.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:56.201815+0000) 2022-09-10T05:49:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:56 smithi110 ceph-mon[97633]: pgmap v2588: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:56 smithi104 ceph-mon[97044]: pgmap v2588: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:49:57.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:49:56.830+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:49:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:56.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:56.798371+0000) 2022-09-10T05:49:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:57.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:57.202067+0000) 2022-09-10T05:49:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:57.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:57.798590+0000) 2022-09-10T05:49:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:58.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:58.202318+0000) 2022-09-10T05:49:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:49:58 smithi110 ceph-mon[97633]: pgmap v2589: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:49:58 smithi104 ceph-mon[97044]: pgmap v2589: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:49:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:58.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:58.798787+0000) 2022-09-10T05:49:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:49:59.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:59.202578+0000) 2022-09-10T05:50:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:49:59.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:49:59.799027+0000) 2022-09-10T05:50:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:00.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:00.202763+0000) 2022-09-10T05:50:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:00 smithi110 ceph-mon[97633]: pgmap v2590: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:00.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:50:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:00 smithi104 ceph-mon[97044]: pgmap v2590: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T05:50:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:00.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:00.799258+0000) 2022-09-10T05:50:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:01.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:01.202958+0000) 2022-09-10T05:50:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:01.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:01.799448+0000) 2022-09-10T05:50:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:02.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:02.203171+0000) 2022-09-10T05:50:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:02 smithi110 ceph-mon[97633]: pgmap v2591: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:02 smithi104 ceph-mon[97044]: pgmap v2591: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:02.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:02.799684+0000) 2022-09-10T05:50:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:03.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:03.203380+0000) 2022-09-10T05:50:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:03.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:03.799883+0000) 2022-09-10T05:50:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:04 smithi104 ceph-mon[97044]: pgmap v2592: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:04.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:04.203576+0000) 2022-09-10T05:50:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:04 smithi110 ceph-mon[97633]: pgmap v2592: 129 pgs: 129 active+clean; 7.9 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:04.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:04.800129+0000) 2022-09-10T05:50:05.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:05.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:05.203813+0000) 2022-09-10T05:50:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:05.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:05.800345+0000) 2022-09-10T05:50:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:06.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:06.204051+0000) 2022-09-10T05:50:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:06 smithi110 ceph-mon[97633]: pgmap v2593: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:50:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:06 smithi104 ceph-mon[97044]: pgmap v2593: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:50:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:06.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:06.800563+0000) 2022-09-10T05:50:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:07.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:07.204159+0000) 2022-09-10T05:50:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:07.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:07.800826+0000) 2022-09-10T05:50:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:08.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:08.204378+0000) 2022-09-10T05:50:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:08 smithi110 ceph-mon[97633]: pgmap v2594: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:50:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:08 smithi110 ceph-mon[97633]: osdmap e246: 12 total, 12 up, 12 in 2022-09-10T05:50:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:08 smithi104 ceph-mon[97044]: pgmap v2594: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:50:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:08 smithi104 ceph-mon[97044]: osdmap e246: 12 total, 12 up, 12 in 2022-09-10T05:50:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:08.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:08.801007+0000) 2022-09-10T05:50:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:09.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:09.204650+0000) 2022-09-10T05:50:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:09.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:09.801158+0000) 2022-09-10T05:50:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:10.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:10.204836+0000) 2022-09-10T05:50:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:10 smithi110 ceph-mon[97633]: pgmap v2596: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:50:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:10 smithi104 ceph-mon[97044]: pgmap v2596: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:50:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:10.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:10.801387+0000) 2022-09-10T05:50:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:11.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:11.205053+0000) 2022-09-10T05:50:12.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:50:11.846+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:50:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:11.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:11.801636+0000) 2022-09-10T05:50:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:12.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:12.205300+0000) 2022-09-10T05:50:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:12 smithi110 ceph-mon[97633]: pgmap v2597: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:50:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:12 smithi104 ceph-mon[97044]: pgmap v2597: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:50:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:12.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:12.801889+0000) 2022-09-10T05:50:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:13.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:13.205457+0000) 2022-09-10T05:50:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:13.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:13.802190+0000) 2022-09-10T05:50:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:14 smithi104 ceph-mon[97044]: osdmap e247: 12 total, 12 up, 12 in 2022-09-10T05:50:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:14 smithi104 ceph-mon[97044]: pgmap v2599: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:50:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:14.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:14.205672+0000) 2022-09-10T05:50:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:14 smithi110 ceph-mon[97633]: osdmap e247: 12 total, 12 up, 12 in 2022-09-10T05:50:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:14 smithi110 ceph-mon[97633]: pgmap v2599: 129 pgs: 129 active+clean; 7.9 MiB data, 428 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:50:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:14.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:14.802422+0000) 2022-09-10T05:50:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:15.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:15.205838+0000) 2022-09-10T05:50:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:15.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:15.802636+0000) 2022-09-10T05:50:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:16.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:16.206073+0000) 2022-09-10T05:50:16.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:16 smithi110 ceph-mon[97633]: pgmap v2600: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:50:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:16 smithi104 ceph-mon[97044]: pgmap v2600: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:50:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:16.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:16.802868+0000) 2022-09-10T05:50:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:17.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:17.206268+0000) 2022-09-10T05:50:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:17.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:17.803121+0000) 2022-09-10T05:50:18.565 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:18.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:18.206459+0000) 2022-09-10T05:50:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:18 smithi110 ceph-mon[97633]: pgmap v2601: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:50:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:18 smithi104 ceph-mon[97044]: pgmap v2601: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:50:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:18.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:18.803315+0000) 2022-09-10T05:50:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:19.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:19.206650+0000) 2022-09-10T05:50:20.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:19.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:19.803534+0000) 2022-09-10T05:50:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:20.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:20.206845+0000) 2022-09-10T05:50:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:20 smithi110 ceph-mon[97633]: pgmap v2602: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:50:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:20 smithi104 ceph-mon[97044]: pgmap v2602: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:50:21.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:20.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:20.803748+0000) 2022-09-10T05:50:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:21.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:21.207087+0000) 2022-09-10T05:50:22.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:21.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:21.803981+0000) 2022-09-10T05:50:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:22.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:22.207331+0000) 2022-09-10T05:50:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:22 smithi110 ceph-mon[97633]: pgmap v2603: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:50:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:22 smithi104 ceph-mon[97044]: pgmap v2603: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:50:23.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:22.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:22.804191+0000) 2022-09-10T05:50:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:23.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:23.207562+0000) 2022-09-10T05:50:24.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:23.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:23.804357+0000) 2022-09-10T05:50:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:24 smithi104 ceph-mon[97044]: pgmap v2604: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:50:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:24.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:24.207754+0000) 2022-09-10T05:50:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:24 smithi110 ceph-mon[97633]: pgmap v2604: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:50:25.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:24.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:24.804590+0000) 2022-09-10T05:50:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:25.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:25.207925+0000) 2022-09-10T05:50:26.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:25.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:25.804752+0000) 2022-09-10T05:50:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:26.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:26.208110+0000) 2022-09-10T05:50:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:26 smithi110 ceph-mon[97633]: pgmap v2605: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:50:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:26 smithi104 ceph-mon[97044]: pgmap v2605: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:50:27.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:26.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:26.805002+0000) 2022-09-10T05:50:27.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:50:26.848+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:50:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:27.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:27.208365+0000) 2022-09-10T05:50:28.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:27.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:27.805218+0000) 2022-09-10T05:50:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:28.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:28.208564+0000) 2022-09-10T05:50:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:28 smithi110 ceph-mon[97633]: pgmap v2606: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:28 smithi104 ceph-mon[97044]: pgmap v2606: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:29.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:28.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:28.805413+0000) 2022-09-10T05:50:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:29.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:29.208793+0000) 2022-09-10T05:50:30.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:29.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:29.805607+0000) 2022-09-10T05:50:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:30.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:30.209063+0000) 2022-09-10T05:50:30.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:30 smithi110 ceph-mon[97633]: pgmap v2607: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:30 smithi104 ceph-mon[97044]: pgmap v2607: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:31.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:30.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:30.805864+0000) 2022-09-10T05:50:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:31.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:31.209363+0000) 2022-09-10T05:50:32.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:31.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:31.806126+0000) 2022-09-10T05:50:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:32.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:32.209556+0000) 2022-09-10T05:50:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:32 smithi110 ceph-mon[97633]: pgmap v2608: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:32 smithi104 ceph-mon[97044]: pgmap v2608: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:33.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:32.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:32.806364+0000) 2022-09-10T05:50:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:33.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:33.209806+0000) 2022-09-10T05:50:34.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:33.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:33.806635+0000) 2022-09-10T05:50:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:34 smithi104 ceph-mon[97044]: pgmap v2609: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:34.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:34.210056+0000) 2022-09-10T05:50:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:34 smithi110 ceph-mon[97633]: pgmap v2609: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:35.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:34.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:34.806859+0000) 2022-09-10T05:50:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:35.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:35.210245+0000) 2022-09-10T05:50:36.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:35.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:35.807047+0000) 2022-09-10T05:50:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:36.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:36.210438+0000) 2022-09-10T05:50:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:36 smithi110 ceph-mon[97633]: pgmap v2610: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:36 smithi104 ceph-mon[97044]: pgmap v2610: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:37.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:36.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:36.807282+0000) 2022-09-10T05:50:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:37.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:37.210631+0000) 2022-09-10T05:50:38.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:37.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:37.807479+0000) 2022-09-10T05:50:38.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:38.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:38.210788+0000) 2022-09-10T05:50:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:38 smithi110 ceph-mon[97633]: pgmap v2611: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:38 smithi104 ceph-mon[97044]: pgmap v2611: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:39.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:38.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:38.807678+0000) 2022-09-10T05:50:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:39.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:39.210947+0000) 2022-09-10T05:50:40.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:39.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:39.807888+0000) 2022-09-10T05:50:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:40.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:40.211211+0000) 2022-09-10T05:50:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:40 smithi110 ceph-mon[97633]: pgmap v2612: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:40 smithi104 ceph-mon[97044]: pgmap v2612: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:40.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:40.808156+0000) 2022-09-10T05:50:41.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:41.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:41.211450+0000) 2022-09-10T05:50:42.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:41.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:41.808283+0000) 2022-09-10T05:50:42.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:50:41.858+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:50:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:42.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:42.211730+0000) 2022-09-10T05:50:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:42 smithi110 ceph-mon[97633]: pgmap v2613: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:42 smithi104 ceph-mon[97044]: pgmap v2613: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:43.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:42.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:42.808512+0000) 2022-09-10T05:50:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:43.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:43.211920+0000) 2022-09-10T05:50:44.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:43.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:43.808676+0000) 2022-09-10T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:44 smithi104 ceph-mon[97044]: pgmap v2614: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:50:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:50:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:50:44.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:50:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:44.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:44.212216+0000) 2022-09-10T05:50:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:44 smithi110 ceph-mon[97633]: pgmap v2614: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:50:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:50:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:50:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:50:45.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:44.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:44.808925+0000) 2022-09-10T05:50:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:45.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:45.212459+0000) 2022-09-10T05:50:46.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:45.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:45.809204+0000) 2022-09-10T05:50:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:46.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:46.212665+0000) 2022-09-10T05:50:46.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:46 smithi110 ceph-mon[97633]: pgmap v2615: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:46 smithi104 ceph-mon[97044]: pgmap v2615: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:47.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:46.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:46.809367+0000) 2022-09-10T05:50:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:47.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:47.212883+0000) 2022-09-10T05:50:48.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:47.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:47.809619+0000) 2022-09-10T05:50:48.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:48.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:48.213055+0000) 2022-09-10T05:50:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:48 smithi110 ceph-mon[97633]: pgmap v2616: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:48 smithi104 ceph-mon[97044]: pgmap v2616: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:49.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:48.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:48.809819+0000) 2022-09-10T05:50:49.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:49.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:49.213226+0000) 2022-09-10T05:50:50.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:49.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:49.810059+0000) 2022-09-10T05:50:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:50.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:50.213447+0000) 2022-09-10T05:50:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:50 smithi110 ceph-mon[97633]: pgmap v2617: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:50 smithi104 ceph-mon[97044]: pgmap v2617: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:51.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:50.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:50.810279+0000) 2022-09-10T05:50:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:51.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:51.213627+0000) 2022-09-10T05:50:52.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:51.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:51.810533+0000) 2022-09-10T05:50:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:52.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:52.213843+0000) 2022-09-10T05:50:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:52 smithi110 ceph-mon[97633]: pgmap v2618: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:52 smithi104 ceph-mon[97044]: pgmap v2618: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:53.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:52.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:52.810710+0000) 2022-09-10T05:50:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:53.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:53.214138+0000) 2022-09-10T05:50:54.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:53.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:53.810925+0000) 2022-09-10T05:50:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:54 smithi104 ceph-mon[97044]: pgmap v2619: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:54.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:54.214334+0000) 2022-09-10T05:50:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:54 smithi110 ceph-mon[97633]: pgmap v2619: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:55.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:54.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:54.811190+0000) 2022-09-10T05:50:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:55.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:55.214499+0000) 2022-09-10T05:50:56.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:55.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:55.811373+0000) 2022-09-10T05:50:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:56.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:56.214658+0000) 2022-09-10T05:50:56.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:56 smithi110 ceph-mon[97633]: pgmap v2620: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:56 smithi104 ceph-mon[97044]: pgmap v2620: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:50:57.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:56.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:56.811528+0000) 2022-09-10T05:50:57.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:50:56.869+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:50:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:57.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:57.214908+0000) 2022-09-10T05:50:58.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:57.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:57.811750+0000) 2022-09-10T05:50:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:58.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:58.215157+0000) 2022-09-10T05:50:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:50:58 smithi110 ceph-mon[97633]: pgmap v2621: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:50:58 smithi104 ceph-mon[97044]: pgmap v2621: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:50:59.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:58.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:58.811942+0000) 2022-09-10T05:50:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:50:59.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:59.215422+0000) 2022-09-10T05:51:00.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:50:59.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:50:59.812191+0000) 2022-09-10T05:51:00.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:00.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:00.215569+0000) 2022-09-10T05:51:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:00 smithi110 ceph-mon[97633]: pgmap v2622: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:00 smithi104 ceph-mon[97044]: pgmap v2622: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:01.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:00.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:00.812373+0000) 2022-09-10T05:51:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:01.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:01.215827+0000) 2022-09-10T05:51:02.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:01.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:01.812572+0000) 2022-09-10T05:51:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:02.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:02.216030+0000) 2022-09-10T05:51:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:02 smithi110 ceph-mon[97633]: pgmap v2623: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:02 smithi104 ceph-mon[97044]: pgmap v2623: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:03.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:02.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:02.812807+0000) 2022-09-10T05:51:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:03.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:03.216253+0000) 2022-09-10T05:51:04.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:03.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:03.813026+0000) 2022-09-10T05:51:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:04 smithi104 ceph-mon[97044]: pgmap v2624: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:04.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:04.216401+0000) 2022-09-10T05:51:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:04 smithi110 ceph-mon[97633]: pgmap v2624: 129 pgs: 129 active+clean; 7.9 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:05.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:04.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:04.813204+0000) 2022-09-10T05:51:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:05.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:05.216630+0000) 2022-09-10T05:51:06.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:05.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:05.813423+0000) 2022-09-10T05:51:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:06.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:06.216892+0000) 2022-09-10T05:51:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:06 smithi110 ceph-mon[97633]: pgmap v2625: 129 pgs: 129 active+clean; 7.9 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:51:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:06 smithi104 ceph-mon[97044]: pgmap v2625: 129 pgs: 129 active+clean; 7.9 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:51:07.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:06.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:06.813667+0000) 2022-09-10T05:51:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:07.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:07.217127+0000) 2022-09-10T05:51:08.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:07.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:07.813937+0000) 2022-09-10T05:51:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:08.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:08.217342+0000) 2022-09-10T05:51:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:08 smithi110 ceph-mon[97633]: pgmap v2626: 129 pgs: 129 active+clean; 7.9 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:51:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:08 smithi110 ceph-mon[97633]: osdmap e248: 12 total, 12 up, 12 in 2022-09-10T05:51:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:08 smithi104 ceph-mon[97044]: pgmap v2626: 129 pgs: 129 active+clean; 7.9 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:51:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:08 smithi104 ceph-mon[97044]: osdmap e248: 12 total, 12 up, 12 in 2022-09-10T05:51:09.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:08.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:08.814173+0000) 2022-09-10T05:51:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:09.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:09.217549+0000) 2022-09-10T05:51:10.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:09.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:09.814445+0000) 2022-09-10T05:51:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:10.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:10.217790+0000) 2022-09-10T05:51:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:10 smithi110 ceph-mon[97633]: pgmap v2628: 129 pgs: 129 active+clean; 7.9 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:51:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:10 smithi104 ceph-mon[97044]: pgmap v2628: 129 pgs: 129 active+clean; 7.9 MiB data, 438 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:51:11.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:10.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:10.814698+0000) 2022-09-10T05:51:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:11.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:11.218049+0000) 2022-09-10T05:51:12.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:11.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:11.814895+0000) 2022-09-10T05:51:12.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:51:11.883+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:51:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:12.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:12.218355+0000) 2022-09-10T05:51:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:12 smithi110 ceph-mon[97633]: pgmap v2629: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:51:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:12 smithi104 ceph-mon[97044]: pgmap v2629: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:51:13.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:12.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:12.815167+0000) 2022-09-10T05:51:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:13.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:13.218628+0000) 2022-09-10T05:51:14.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:13.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:13.815303+0000) 2022-09-10T05:51:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:14 smithi104 ceph-mon[97044]: osdmap e249: 12 total, 12 up, 12 in 2022-09-10T05:51:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:14 smithi104 ceph-mon[97044]: pgmap v2631: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:51:14.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:14.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:14.218905+0000) 2022-09-10T05:51:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:14 smithi110 ceph-mon[97633]: osdmap e249: 12 total, 12 up, 12 in 2022-09-10T05:51:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:14 smithi110 ceph-mon[97633]: pgmap v2631: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:51:15.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:14.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:14.815528+0000) 2022-09-10T05:51:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:15.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:15.219151+0000) 2022-09-10T05:51:16.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:15.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:15.815777+0000) 2022-09-10T05:51:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:16.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:16.219397+0000) 2022-09-10T05:51:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:16 smithi110 ceph-mon[97633]: pgmap v2632: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:51:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:16 smithi104 ceph-mon[97044]: pgmap v2632: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:51:17.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:16.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:16.815993+0000) 2022-09-10T05:51:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:17.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:17.219638+0000) 2022-09-10T05:51:18.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:17.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:17.816191+0000) 2022-09-10T05:51:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:18.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:18.219872+0000) 2022-09-10T05:51:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:18 smithi110 ceph-mon[97633]: pgmap v2633: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:51:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:18 smithi104 ceph-mon[97044]: pgmap v2633: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:51:19.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:18.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:18.816363+0000) 2022-09-10T05:51:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:19.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:19.220060+0000) 2022-09-10T05:51:20.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:19.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:19.816576+0000) 2022-09-10T05:51:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:20.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:20.220269+0000) 2022-09-10T05:51:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:20 smithi110 ceph-mon[97633]: pgmap v2634: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:51:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:20 smithi104 ceph-mon[97044]: pgmap v2634: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:51:21.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:20.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:20.816825+0000) 2022-09-10T05:51:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:21.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:21.220446+0000) 2022-09-10T05:51:22.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:21.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:21.817063+0000) 2022-09-10T05:51:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:22.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:22.220620+0000) 2022-09-10T05:51:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:22 smithi110 ceph-mon[97633]: pgmap v2635: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:51:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:22 smithi104 ceph-mon[97044]: pgmap v2635: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:51:23.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:22.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:22.817277+0000) 2022-09-10T05:51:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:23.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:23.220866+0000) 2022-09-10T05:51:24.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:23.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:23.817517+0000) 2022-09-10T05:51:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:24 smithi104 ceph-mon[97044]: pgmap v2636: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:51:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:24.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:24.221067+0000) 2022-09-10T05:51:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:24 smithi110 ceph-mon[97633]: pgmap v2636: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:51:25.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:24.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:24.817742+0000) 2022-09-10T05:51:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:25.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:25.221252+0000) 2022-09-10T05:51:26.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:25.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:25.817917+0000) 2022-09-10T05:51:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:26.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:26.221509+0000) 2022-09-10T05:51:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:26 smithi110 ceph-mon[97633]: pgmap v2637: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:51:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:26 smithi104 ceph-mon[97044]: pgmap v2637: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:51:27.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:26.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:26.818149+0000) 2022-09-10T05:51:27.223 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:51:26.889+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:51:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:27.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:27.221758+0000) 2022-09-10T05:51:28.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:27.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:27.818341+0000) 2022-09-10T05:51:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:28.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:28.222005+0000) 2022-09-10T05:51:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:28 smithi110 ceph-mon[97633]: pgmap v2638: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:28 smithi104 ceph-mon[97044]: pgmap v2638: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:29.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:28.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:28.818498+0000) 2022-09-10T05:51:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:29.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:29.222251+0000) 2022-09-10T05:51:30.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:29.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:29.818713+0000) 2022-09-10T05:51:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:30.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:30.222519+0000) 2022-09-10T05:51:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:30 smithi110 ceph-mon[97633]: pgmap v2639: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:30 smithi104 ceph-mon[97044]: pgmap v2639: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:31.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:30.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:30.818952+0000) 2022-09-10T05:51:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:31.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:31.222846+0000) 2022-09-10T05:51:32.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:31.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:31.819180+0000) 2022-09-10T05:51:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:32.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:32.223056+0000) 2022-09-10T05:51:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:32 smithi110 ceph-mon[97633]: pgmap v2640: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:32 smithi104 ceph-mon[97044]: pgmap v2640: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:33.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:32.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:32.819386+0000) 2022-09-10T05:51:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:33.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:33.223300+0000) 2022-09-10T05:51:34.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:33.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:33.819662+0000) 2022-09-10T05:51:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:34 smithi104 ceph-mon[97044]: pgmap v2641: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:34.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:34.223586+0000) 2022-09-10T05:51:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:34 smithi110 ceph-mon[97633]: pgmap v2641: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:35.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:34.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:34.819844+0000) 2022-09-10T05:51:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:35.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:35.223838+0000) 2022-09-10T05:51:36.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:35.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:35.820132+0000) 2022-09-10T05:51:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:36.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:36.224085+0000) 2022-09-10T05:51:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:36 smithi110 ceph-mon[97633]: pgmap v2642: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:36 smithi104 ceph-mon[97044]: pgmap v2642: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:37.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:36.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:36.820339+0000) 2022-09-10T05:51:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:37.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:37.224278+0000) 2022-09-10T05:51:38.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:37.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:37.820526+0000) 2022-09-10T05:51:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:38.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:38.224479+0000) 2022-09-10T05:51:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:38 smithi110 ceph-mon[97633]: pgmap v2643: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:38 smithi104 ceph-mon[97044]: pgmap v2643: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:39.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:38.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:38.820782+0000) 2022-09-10T05:51:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:39.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:39.224680+0000) 2022-09-10T05:51:40.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:39.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:39.821074+0000) 2022-09-10T05:51:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:40.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:40.224940+0000) 2022-09-10T05:51:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:40 smithi110 ceph-mon[97633]: pgmap v2644: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T05:51:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:40 smithi104 ceph-mon[97044]: pgmap v2644: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T05:51:41.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:40.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:40.821205+0000) 2022-09-10T05:51:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:41.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:41.225235+0000) 2022-09-10T05:51:42.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:41.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:41.821425+0000) 2022-09-10T05:51:42.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:51:41.898+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:51:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:42.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:42.225502+0000) 2022-09-10T05:51:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:42 smithi110 ceph-mon[97633]: pgmap v2645: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:42 smithi104 ceph-mon[97044]: pgmap v2645: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:43.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:42.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:42.821604+0000) 2022-09-10T05:51:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:43.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:43.225697+0000) 2022-09-10T05:51:44.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:43.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:43.821782+0000) 2022-09-10T05:51:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:44 smithi104 ceph-mon[97044]: pgmap v2646: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:51:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:51:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:51:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:51:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:44.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:44.225945+0000) 2022-09-10T05:51:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:44 smithi110 ceph-mon[97633]: pgmap v2646: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:51:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:51:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:51:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:51:45.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:44.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:44.822030+0000) 2022-09-10T05:51:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:45.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:45.226225+0000) 2022-09-10T05:51:46.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:45.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:45.822239+0000) 2022-09-10T05:51:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:46.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:46.226489+0000) 2022-09-10T05:51:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:46 smithi110 ceph-mon[97633]: pgmap v2647: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:46.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:46 smithi104 ceph-mon[97044]: pgmap v2647: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:47.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:46.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:46.822473+0000) 2022-09-10T05:51:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:47.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:47.226743+0000) 2022-09-10T05:51:48.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:47.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:47.822714+0000) 2022-09-10T05:51:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:48.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:48.226940+0000) 2022-09-10T05:51:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:48 smithi110 ceph-mon[97633]: pgmap v2648: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:48 smithi104 ceph-mon[97044]: pgmap v2648: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:49.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:48.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:48.822961+0000) 2022-09-10T05:51:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:49.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:49.227214+0000) 2022-09-10T05:51:50.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:49.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:49.823196+0000) 2022-09-10T05:51:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:50.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:50.227416+0000) 2022-09-10T05:51:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:50 smithi110 ceph-mon[97633]: pgmap v2649: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T05:51:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:50 smithi104 ceph-mon[97044]: pgmap v2649: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T05:51:51.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:50.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:50.823443+0000) 2022-09-10T05:51:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:51.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:51.227641+0000) 2022-09-10T05:51:52.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:51.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:51.823663+0000) 2022-09-10T05:51:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:52.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:52.227899+0000) 2022-09-10T05:51:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:52 smithi110 ceph-mon[97633]: pgmap v2650: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2022-09-10T05:51:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:52 smithi104 ceph-mon[97044]: pgmap v2650: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2022-09-10T05:51:53.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:52.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:52.823876+0000) 2022-09-10T05:51:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:53.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:53.228183+0000) 2022-09-10T05:51:54.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:53.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:53.824146+0000) 2022-09-10T05:51:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:54 smithi104 ceph-mon[97044]: pgmap v2651: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:54.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:54.228488+0000) 2022-09-10T05:51:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:54 smithi110 ceph-mon[97633]: pgmap v2651: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:55.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:54.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:54.824374+0000) 2022-09-10T05:51:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:55.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:55.228736+0000) 2022-09-10T05:51:56.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:55.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:55.824603+0000) 2022-09-10T05:51:56.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:56.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:56.228949+0000) 2022-09-10T05:51:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:56 smithi110 ceph-mon[97633]: pgmap v2652: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:56 smithi104 ceph-mon[97044]: pgmap v2652: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:51:57.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:56.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:56.824834+0000) 2022-09-10T05:51:57.231 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:51:56.909+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:51:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:57.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:57.229217+0000) 2022-09-10T05:51:58.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:57.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:57.825047+0000) 2022-09-10T05:51:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:58.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:58.229460+0000) 2022-09-10T05:51:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:51:58 smithi110 ceph-mon[97633]: pgmap v2653: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:51:58 smithi104 ceph-mon[97044]: pgmap v2653: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:51:59.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:58.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:58.825278+0000) 2022-09-10T05:51:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:51:59.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:59.229615+0000) 2022-09-10T05:52:00.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:51:59.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:51:59.825477+0000) 2022-09-10T05:52:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:00.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:00.229750+0000) 2022-09-10T05:52:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:00 smithi110 ceph-mon[97633]: pgmap v2654: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T05:52:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:00 smithi104 ceph-mon[97044]: pgmap v2654: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2022-09-10T05:52:01.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:00.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:00.825708+0000) 2022-09-10T05:52:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:01.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:01.230016+0000) 2022-09-10T05:52:02.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:01.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:01.825935+0000) 2022-09-10T05:52:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:02.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:02.230170+0000) 2022-09-10T05:52:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:02 smithi110 ceph-mon[97633]: pgmap v2655: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2022-09-10T05:52:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:02 smithi104 ceph-mon[97044]: pgmap v2655: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2022-09-10T05:52:03.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:02.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:02.826197+0000) 2022-09-10T05:52:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:03.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:03.230323+0000) 2022-09-10T05:52:04.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:03.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:03.826398+0000) 2022-09-10T05:52:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:04 smithi104 ceph-mon[97044]: pgmap v2656: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:04.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:04.230564+0000) 2022-09-10T05:52:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:04 smithi110 ceph-mon[97633]: pgmap v2656: 129 pgs: 129 active+clean; 7.9 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:05.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:04.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:04.826644+0000) 2022-09-10T05:52:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:05.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:05.230778+0000) 2022-09-10T05:52:06.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:05.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:05.826830+0000) 2022-09-10T05:52:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:06.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:06.231001+0000) 2022-09-10T05:52:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:06 smithi110 ceph-mon[97633]: pgmap v2657: 129 pgs: 129 active+clean; 7.9 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:52:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:06 smithi104 ceph-mon[97044]: pgmap v2657: 129 pgs: 129 active+clean; 7.9 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:52:07.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:06.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:06.827178+0000) 2022-09-10T05:52:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:07.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:07.231184+0000) 2022-09-10T05:52:08.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:07.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:07.827437+0000) 2022-09-10T05:52:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:08.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:08.231391+0000) 2022-09-10T05:52:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:08 smithi110 ceph-mon[97633]: pgmap v2658: 129 pgs: 129 active+clean; 7.9 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:52:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:08 smithi110 ceph-mon[97633]: osdmap e250: 12 total, 12 up, 12 in 2022-09-10T05:52:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:08 smithi104 ceph-mon[97044]: pgmap v2658: 129 pgs: 129 active+clean; 7.9 MiB data, 440 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:52:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:08 smithi104 ceph-mon[97044]: osdmap e250: 12 total, 12 up, 12 in 2022-09-10T05:52:09.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:08.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:08.827684+0000) 2022-09-10T05:52:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:09.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:09.231649+0000) 2022-09-10T05:52:10.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:09.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:09.828067+0000) 2022-09-10T05:52:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:10.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:10.231937+0000) 2022-09-10T05:52:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:10 smithi110 ceph-mon[97633]: pgmap v2660: 129 pgs: 129 active+clean; 7.9 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:52:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:10 smithi104 ceph-mon[97044]: pgmap v2660: 129 pgs: 129 active+clean; 7.9 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:52:11.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:10.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:10.828370+0000) 2022-09-10T05:52:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:11.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:11.232228+0000) 2022-09-10T05:52:12.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:11.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:11.828554+0000) 2022-09-10T05:52:12.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:52:11.919+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:52:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:12.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:12.232471+0000) 2022-09-10T05:52:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:12 smithi110 ceph-mon[97633]: pgmap v2661: 129 pgs: 129 active+clean; 7.9 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:52:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:12 smithi104 ceph-mon[97044]: pgmap v2661: 129 pgs: 129 active+clean; 7.9 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:52:13.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:12.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:12.828789+0000) 2022-09-10T05:52:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:13.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:13.232643+0000) 2022-09-10T05:52:14.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:13.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:13.829059+0000) 2022-09-10T05:52:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:14 smithi104 ceph-mon[97044]: osdmap e251: 12 total, 12 up, 12 in 2022-09-10T05:52:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:14 smithi104 ceph-mon[97044]: pgmap v2663: 129 pgs: 129 active+clean; 7.9 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:52:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:14.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:14.232896+0000) 2022-09-10T05:52:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:14 smithi110 ceph-mon[97633]: osdmap e251: 12 total, 12 up, 12 in 2022-09-10T05:52:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:14 smithi110 ceph-mon[97633]: pgmap v2663: 129 pgs: 129 active+clean; 7.9 MiB data, 445 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:52:15.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:14.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:14.829305+0000) 2022-09-10T05:52:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:15.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:15.233178+0000) 2022-09-10T05:52:16.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:15.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:15.829468+0000) 2022-09-10T05:52:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:16.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:16.233449+0000) 2022-09-10T05:52:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:16 smithi110 ceph-mon[97633]: pgmap v2664: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:52:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:16 smithi104 ceph-mon[97044]: pgmap v2664: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:52:17.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:16.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:16.829631+0000) 2022-09-10T05:52:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:17.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:17.233729+0000) 2022-09-10T05:52:18.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:17.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:17.829828+0000) 2022-09-10T05:52:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:18.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:18.233905+0000) 2022-09-10T05:52:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:18 smithi110 ceph-mon[97633]: pgmap v2665: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:52:18.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:18 smithi104 ceph-mon[97044]: pgmap v2665: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:52:19.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:18.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:18.830050+0000) 2022-09-10T05:52:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:19.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:19.234166+0000) 2022-09-10T05:52:20.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:19.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:19.830317+0000) 2022-09-10T05:52:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:20.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:20.234432+0000) 2022-09-10T05:52:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:20 smithi110 ceph-mon[97633]: pgmap v2666: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:52:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:20 smithi104 ceph-mon[97044]: pgmap v2666: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:52:21.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:20.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:20.830572+0000) 2022-09-10T05:52:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:21.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:21.234697+0000) 2022-09-10T05:52:22.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:21.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:21.830861+0000) 2022-09-10T05:52:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:22.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:22.234962+0000) 2022-09-10T05:52:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:22 smithi110 ceph-mon[97633]: pgmap v2667: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:52:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:22 smithi104 ceph-mon[97044]: pgmap v2667: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:52:23.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:22.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:22.831042+0000) 2022-09-10T05:52:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:23.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:23.235178+0000) 2022-09-10T05:52:24.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:23.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:23.831259+0000) 2022-09-10T05:52:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:24 smithi104 ceph-mon[97044]: pgmap v2668: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:52:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:24.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:24.235451+0000) 2022-09-10T05:52:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:24 smithi110 ceph-mon[97633]: pgmap v2668: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:52:25.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:24.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:24.831536+0000) 2022-09-10T05:52:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:25.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:25.235735+0000) 2022-09-10T05:52:26.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:25.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:25.831771+0000) 2022-09-10T05:52:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:26.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:26.235995+0000) 2022-09-10T05:52:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:26 smithi110 ceph-mon[97633]: pgmap v2669: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:52:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:26 smithi104 ceph-mon[97044]: pgmap v2669: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:52:27.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:26.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:26.831987+0000) 2022-09-10T05:52:27.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:52:26.935+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:52:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:27.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:27.236226+0000) 2022-09-10T05:52:28.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:27.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:27.832185+0000) 2022-09-10T05:52:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:28.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:28.236488+0000) 2022-09-10T05:52:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:28 smithi110 ceph-mon[97633]: pgmap v2670: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:28 smithi104 ceph-mon[97044]: pgmap v2670: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:29.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:28.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:28.832420+0000) 2022-09-10T05:52:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:29.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:29.236696+0000) 2022-09-10T05:52:30.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:29.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:29.832693+0000) 2022-09-10T05:52:30.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:30.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:30.236958+0000) 2022-09-10T05:52:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:30 smithi110 ceph-mon[97633]: pgmap v2671: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:52:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:30 smithi104 ceph-mon[97044]: pgmap v2671: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:52:31.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:30.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:30.832909+0000) 2022-09-10T05:52:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:31.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:31.237213+0000) 2022-09-10T05:52:32.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:31.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:31.833140+0000) 2022-09-10T05:52:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:32.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:32.237449+0000) 2022-09-10T05:52:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:32 smithi110 ceph-mon[97633]: pgmap v2672: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2022-09-10T05:52:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:32 smithi104 ceph-mon[97044]: pgmap v2672: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2022-09-10T05:52:33.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:32.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:32.833228+0000) 2022-09-10T05:52:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:33.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:33.237685+0000) 2022-09-10T05:52:34.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:33.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:33.833435+0000) 2022-09-10T05:52:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:34 smithi104 ceph-mon[97044]: pgmap v2673: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:34.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:34.237917+0000) 2022-09-10T05:52:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:34 smithi110 ceph-mon[97633]: pgmap v2673: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:35.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:34.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:34.833619+0000) 2022-09-10T05:52:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:35.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:35.238186+0000) 2022-09-10T05:52:36.085 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:35.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:35.833830+0000) 2022-09-10T05:52:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:36.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:36.238462+0000) 2022-09-10T05:52:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:36 smithi110 ceph-mon[97633]: pgmap v2674: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:52:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:36 smithi104 ceph-mon[97044]: pgmap v2674: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:52:37.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:36.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:36.834092+0000) 2022-09-10T05:52:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:37.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:37.238685+0000) 2022-09-10T05:52:38.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:37.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:37.834298+0000) 2022-09-10T05:52:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:38.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:38.238830+0000) 2022-09-10T05:52:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:38 smithi110 ceph-mon[97633]: pgmap v2675: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:38 smithi104 ceph-mon[97044]: pgmap v2675: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:39.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:38.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:38.834492+0000) 2022-09-10T05:52:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:39.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:39.239060+0000) 2022-09-10T05:52:40.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:39.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:39.834677+0000) 2022-09-10T05:52:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:40.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:40.239308+0000) 2022-09-10T05:52:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:40 smithi110 ceph-mon[97633]: pgmap v2676: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:52:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:40 smithi104 ceph-mon[97044]: pgmap v2676: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:52:41.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:40.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:40.834900+0000) 2022-09-10T05:52:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:41.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:41.239572+0000) 2022-09-10T05:52:42.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:41.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:41.835072+0000) 2022-09-10T05:52:42.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:52:41.943+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:52:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:42.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:42.239823+0000) 2022-09-10T05:52:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:42 smithi110 ceph-mon[97633]: pgmap v2677: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:52:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:42 smithi104 ceph-mon[97044]: pgmap v2677: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:52:43.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:42.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:42.835201+0000) 2022-09-10T05:52:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:43.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:43.240086+0000) 2022-09-10T05:52:44.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:43.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:43.835410+0000) 2022-09-10T05:52:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:44 smithi104 ceph-mon[97044]: pgmap v2678: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:52:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:52:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:52:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:52:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:44.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:44.240358+0000) 2022-09-10T05:52:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:44 smithi110 ceph-mon[97633]: pgmap v2678: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:52:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:52:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:52:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:52:45.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:44.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:44.835686+0000) 2022-09-10T05:52:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:45.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:45.240593+0000) 2022-09-10T05:52:46.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:45.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:45.835907+0000) 2022-09-10T05:52:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:46.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:46.240878+0000) 2022-09-10T05:52:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:46 smithi110 ceph-mon[97633]: pgmap v2679: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:52:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:46 smithi104 ceph-mon[97044]: pgmap v2679: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:52:47.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:46.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:46.836070+0000) 2022-09-10T05:52:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:47.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:47.241156+0000) 2022-09-10T05:52:48.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:47.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:47.836343+0000) 2022-09-10T05:52:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:48.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:48.241415+0000) 2022-09-10T05:52:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:48 smithi110 ceph-mon[97633]: pgmap v2680: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:48 smithi104 ceph-mon[97044]: pgmap v2680: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:49.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:48.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:48.836545+0000) 2022-09-10T05:52:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:49.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:49.241656+0000) 2022-09-10T05:52:50.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:49.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:49.836683+0000) 2022-09-10T05:52:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:50.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:50.241828+0000) 2022-09-10T05:52:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:50 smithi110 ceph-mon[97633]: pgmap v2681: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:52:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:50 smithi104 ceph-mon[97044]: pgmap v2681: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:52:51.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:50.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:50.836877+0000) 2022-09-10T05:52:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:51.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:51.242064+0000) 2022-09-10T05:52:52.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:51.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:51.837157+0000) 2022-09-10T05:52:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:52.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:52.242316+0000) 2022-09-10T05:52:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:52 smithi110 ceph-mon[97633]: pgmap v2682: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:52:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:52 smithi104 ceph-mon[97044]: pgmap v2682: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:52:53.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:52.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:52.837320+0000) 2022-09-10T05:52:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:53.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:53.242540+0000) 2022-09-10T05:52:54.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:53.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:53.837577+0000) 2022-09-10T05:52:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:54 smithi104 ceph-mon[97044]: pgmap v2683: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:54.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:54.242788+0000) 2022-09-10T05:52:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:54 smithi110 ceph-mon[97633]: pgmap v2683: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:52:55.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:54.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:54.837824+0000) 2022-09-10T05:52:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:55.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:55.242979+0000) 2022-09-10T05:52:56.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:55.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:55.838075+0000) 2022-09-10T05:52:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:56.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:56.243218+0000) 2022-09-10T05:52:56.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:56 smithi110 ceph-mon[97633]: pgmap v2684: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:52:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:56 smithi104 ceph-mon[97044]: pgmap v2684: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:52:57.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:56.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:56.838350+0000) 2022-09-10T05:52:57.245 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:52:56.953+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:52:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:57.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:57.243445+0000) 2022-09-10T05:52:58.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:57.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:57.838584+0000) 2022-09-10T05:52:58.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:58.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:58.243643+0000) 2022-09-10T05:52:58.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:52:58 smithi110 ceph-mon[97633]: pgmap v2685: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:52:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:52:58 smithi104 ceph-mon[97044]: pgmap v2685: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:52:59.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:58.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:58.838813+0000) 2022-09-10T05:52:59.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:52:59.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:59.243895+0000) 2022-09-10T05:53:00.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:52:59.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:52:59.839044+0000) 2022-09-10T05:53:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:00.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:00.244059+0000) 2022-09-10T05:53:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:00 smithi110 ceph-mon[97633]: pgmap v2686: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2022-09-10T05:53:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:00 smithi104 ceph-mon[97044]: pgmap v2686: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2022-09-10T05:53:01.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:00.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:00.839256+0000) 2022-09-10T05:53:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:01.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:01.244227+0000) 2022-09-10T05:53:02.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:01.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:01.839467+0000) 2022-09-10T05:53:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:02.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:02.244469+0000) 2022-09-10T05:53:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:02 smithi110 ceph-mon[97633]: pgmap v2687: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T05:53:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:02 smithi104 ceph-mon[97044]: pgmap v2687: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T05:53:03.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:02.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:02.839729+0000) 2022-09-10T05:53:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:03.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:03.244709+0000) 2022-09-10T05:53:04.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:03.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:03.839982+0000) 2022-09-10T05:53:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:04 smithi104 ceph-mon[97044]: pgmap v2688: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:53:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:04.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:04.244908+0000) 2022-09-10T05:53:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:04 smithi110 ceph-mon[97633]: pgmap v2688: 129 pgs: 129 active+clean; 7.9 MiB data, 446 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:53:05.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:04.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:04.840234+0000) 2022-09-10T05:53:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:05.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:05.245111+0000) 2022-09-10T05:53:06.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:05.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:05.840411+0000) 2022-09-10T05:53:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:06.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:06.245395+0000) 2022-09-10T05:53:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:06 smithi110 ceph-mon[97633]: pgmap v2689: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:53:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:06 smithi104 ceph-mon[97044]: pgmap v2689: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:53:07.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:06.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:06.840636+0000) 2022-09-10T05:53:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:07.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:07.245669+0000) 2022-09-10T05:53:08.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:07.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:07.840818+0000) 2022-09-10T05:53:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:08.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:08.245914+0000) 2022-09-10T05:53:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:08 smithi110 ceph-mon[97633]: pgmap v2690: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:53:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:08 smithi110 ceph-mon[97633]: osdmap e252: 12 total, 12 up, 12 in 2022-09-10T05:53:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:08 smithi104 ceph-mon[97044]: pgmap v2690: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:53:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:08 smithi104 ceph-mon[97044]: osdmap e252: 12 total, 12 up, 12 in 2022-09-10T05:53:09.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:08.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:08.841032+0000) 2022-09-10T05:53:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:09.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:09.246156+0000) 2022-09-10T05:53:10.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:09.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:09.841261+0000) 2022-09-10T05:53:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:10.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:10.246432+0000) 2022-09-10T05:53:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:10 smithi110 ceph-mon[97633]: pgmap v2692: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:53:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:10 smithi104 ceph-mon[97044]: pgmap v2692: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:53:11.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:10.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:10.841419+0000) 2022-09-10T05:53:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:11.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:11.246695+0000) 2022-09-10T05:53:12.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:11.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:11.841609+0000) 2022-09-10T05:53:12.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:53:11.969+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:53:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:12.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:12.246889+0000) 2022-09-10T05:53:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:12 smithi110 ceph-mon[97633]: pgmap v2693: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:53:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:12 smithi104 ceph-mon[97044]: pgmap v2693: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:53:13.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:12.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:12.841845+0000) 2022-09-10T05:53:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:13.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:13.247086+0000) 2022-09-10T05:53:14.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:13.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:13.842162+0000) 2022-09-10T05:53:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:14 smithi104 ceph-mon[97044]: osdmap e253: 12 total, 12 up, 12 in 2022-09-10T05:53:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:14 smithi104 ceph-mon[97044]: pgmap v2695: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:53:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:14.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:14.247335+0000) 2022-09-10T05:53:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:14 smithi110 ceph-mon[97633]: osdmap e253: 12 total, 12 up, 12 in 2022-09-10T05:53:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:14 smithi110 ceph-mon[97633]: pgmap v2695: 129 pgs: 129 active+clean; 7.9 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:53:15.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:14.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:14.842347+0000) 2022-09-10T05:53:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:15.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:15.247584+0000) 2022-09-10T05:53:16.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:15.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:15.842541+0000) 2022-09-10T05:53:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:16.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:16.247861+0000) 2022-09-10T05:53:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:16 smithi110 ceph-mon[97633]: pgmap v2696: 129 pgs: 129 active+clean; 7.9 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:53:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:16 smithi104 ceph-mon[97044]: pgmap v2696: 129 pgs: 129 active+clean; 7.9 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:53:17.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:16.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:16.842740+0000) 2022-09-10T05:53:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:17.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:17.248149+0000) 2022-09-10T05:53:18.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:17.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:17.842939+0000) 2022-09-10T05:53:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:18.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:18.248326+0000) 2022-09-10T05:53:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:18 smithi110 ceph-mon[97633]: pgmap v2697: 129 pgs: 129 active+clean; 7.9 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:53:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:18 smithi104 ceph-mon[97044]: pgmap v2697: 129 pgs: 129 active+clean; 7.9 MiB data, 451 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:53:19.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:18.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:18.843183+0000) 2022-09-10T05:53:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:19.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:19.248490+0000) 2022-09-10T05:53:20.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:19.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:19.843449+0000) 2022-09-10T05:53:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:20.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:20.248714+0000) 2022-09-10T05:53:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:20 smithi110 ceph-mon[97633]: pgmap v2698: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:53:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:20 smithi104 ceph-mon[97044]: pgmap v2698: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:53:21.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:20.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:20.843672+0000) 2022-09-10T05:53:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:21.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:21.248941+0000) 2022-09-10T05:53:22.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:21.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:21.843883+0000) 2022-09-10T05:53:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:22.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:22.249203+0000) 2022-09-10T05:53:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:22 smithi110 ceph-mon[97633]: pgmap v2699: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:53:22.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:22 smithi104 ceph-mon[97044]: pgmap v2699: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:53:23.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:22.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:22.844136+0000) 2022-09-10T05:53:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:23.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:23.249470+0000) 2022-09-10T05:53:24.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:23.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:23.844341+0000) 2022-09-10T05:53:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:24 smithi104 ceph-mon[97044]: pgmap v2700: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:53:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:24.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:24.249647+0000) 2022-09-10T05:53:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:24 smithi110 ceph-mon[97633]: pgmap v2700: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:53:25.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:24.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:24.844602+0000) 2022-09-10T05:53:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:25.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:25.249888+0000) 2022-09-10T05:53:26.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:25.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:25.844789+0000) 2022-09-10T05:53:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:26.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:26.250157+0000) 2022-09-10T05:53:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:26 smithi110 ceph-mon[97633]: pgmap v2701: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:53:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:26 smithi104 ceph-mon[97044]: pgmap v2701: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:53:27.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:26.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:26.844961+0000) 2022-09-10T05:53:27.252 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:53:26.978+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:53:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:27.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:27.250384+0000) 2022-09-10T05:53:28.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:27.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:27.845197+0000) 2022-09-10T05:53:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:28.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:28.250528+0000) 2022-09-10T05:53:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:28 smithi110 ceph-mon[97633]: pgmap v2702: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:28 smithi104 ceph-mon[97044]: pgmap v2702: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:29.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:28.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:28.845428+0000) 2022-09-10T05:53:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:29.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:29.250710+0000) 2022-09-10T05:53:30.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:29.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:29.845616+0000) 2022-09-10T05:53:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:30.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:30.250974+0000) 2022-09-10T05:53:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:30 smithi110 ceph-mon[97633]: pgmap v2703: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:53:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:30 smithi104 ceph-mon[97044]: pgmap v2703: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:53:31.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:30.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:30.845841+0000) 2022-09-10T05:53:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:31.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:31.251225+0000) 2022-09-10T05:53:32.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:31.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:31.846009+0000) 2022-09-10T05:53:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:32.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:32.251402+0000) 2022-09-10T05:53:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:32 smithi110 ceph-mon[97633]: pgmap v2704: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:53:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:32 smithi104 ceph-mon[97044]: pgmap v2704: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:53:33.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:32.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:32.846243+0000) 2022-09-10T05:53:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:33.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:33.251658+0000) 2022-09-10T05:53:34.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:33.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:33.846488+0000) 2022-09-10T05:53:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:34 smithi104 ceph-mon[97044]: pgmap v2705: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:34.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:34.251910+0000) 2022-09-10T05:53:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:34 smithi110 ceph-mon[97633]: pgmap v2705: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:35.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:34.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:34.846701+0000) 2022-09-10T05:53:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:35.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:35.252184+0000) 2022-09-10T05:53:36.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:35.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:35.846930+0000) 2022-09-10T05:53:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:36.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:36.252416+0000) 2022-09-10T05:53:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:36 smithi110 ceph-mon[97633]: pgmap v2706: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:53:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:36 smithi104 ceph-mon[97044]: pgmap v2706: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:53:37.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:36.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:36.847170+0000) 2022-09-10T05:53:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:37.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:37.252607+0000) 2022-09-10T05:53:38.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:37.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:37.847375+0000) 2022-09-10T05:53:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:38.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:38.252777+0000) 2022-09-10T05:53:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:38 smithi110 ceph-mon[97633]: pgmap v2707: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:38 smithi104 ceph-mon[97044]: pgmap v2707: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:39.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:38.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:38.847552+0000) 2022-09-10T05:53:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:39.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:39.253022+0000) 2022-09-10T05:53:40.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:39.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:39.847822+0000) 2022-09-10T05:53:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:40.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:40.253253+0000) 2022-09-10T05:53:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:40 smithi110 ceph-mon[97633]: pgmap v2708: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:53:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:40 smithi104 ceph-mon[97044]: pgmap v2708: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:53:41.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:40.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:40.848044+0000) 2022-09-10T05:53:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:41.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:41.253433+0000) 2022-09-10T05:53:42.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:41.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:41.848274+0000) 2022-09-10T05:53:42.255 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:53:41.989+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:53:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:42.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:42.253666+0000) 2022-09-10T05:53:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:42 smithi110 ceph-mon[97633]: pgmap v2709: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:53:42.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:42 smithi104 ceph-mon[97044]: pgmap v2709: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:53:43.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:42.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:42.848524+0000) 2022-09-10T05:53:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:43.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:43.253882+0000) 2022-09-10T05:53:44.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:43.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:43.848755+0000) 2022-09-10T05:53:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:44 smithi104 ceph-mon[97044]: pgmap v2710: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:53:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:53:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:53:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:53:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:44.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:44.254187+0000) 2022-09-10T05:53:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:44 smithi110 ceph-mon[97633]: pgmap v2710: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:53:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:53:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:53:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:53:45.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:44.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:44.848967+0000) 2022-09-10T05:53:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:45.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:45.254368+0000) 2022-09-10T05:53:46.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:45.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:45.849174+0000) 2022-09-10T05:53:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:46.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:46.254581+0000) 2022-09-10T05:53:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:46 smithi110 ceph-mon[97633]: pgmap v2711: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:53:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:46 smithi104 ceph-mon[97044]: pgmap v2711: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:53:47.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:46.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:46.849312+0000) 2022-09-10T05:53:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:47.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:47.254786+0000) 2022-09-10T05:53:48.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:47.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:47.849515+0000) 2022-09-10T05:53:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:48.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:48.255005+0000) 2022-09-10T05:53:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:48 smithi110 ceph-mon[97633]: pgmap v2712: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:53:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:48 smithi104 ceph-mon[97044]: pgmap v2712: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:53:49.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:48.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:48.849665+0000) 2022-09-10T05:53:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:49.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:49.255168+0000) 2022-09-10T05:53:50.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:49.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:49.849940+0000) 2022-09-10T05:53:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:50.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:50.255379+0000) 2022-09-10T05:53:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:50 smithi104 ceph-mon[97044]: pgmap v2713: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2022-09-10T05:53:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:50 smithi110 ceph-mon[97633]: pgmap v2713: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2022-09-10T05:53:51.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:50.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:50.850207+0000) 2022-09-10T05:53:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:51.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:51.255611+0000) 2022-09-10T05:53:52.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:51.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:51.850464+0000) 2022-09-10T05:53:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:52.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:52.255803+0000) 2022-09-10T05:53:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:52 smithi104 ceph-mon[97044]: pgmap v2714: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T05:53:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:52 smithi110 ceph-mon[97633]: pgmap v2714: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T05:53:53.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:52.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:52.850690+0000) 2022-09-10T05:53:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:53.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:53.256041+0000) 2022-09-10T05:53:54.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:53.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:53.850965+0000) 2022-09-10T05:53:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:54 smithi104 ceph-mon[97044]: pgmap v2715: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:53:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:54.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:54.256301+0000) 2022-09-10T05:53:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:54 smithi110 ceph-mon[97633]: pgmap v2715: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:53:55.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:54.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:54.851194+0000) 2022-09-10T05:53:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:55.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:55.256515+0000) 2022-09-10T05:53:56.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:55.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:55.851417+0000) 2022-09-10T05:53:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:56.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:56.256767+0000) 2022-09-10T05:53:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:56 smithi110 ceph-mon[97633]: pgmap v2716: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:53:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:56 smithi104 ceph-mon[97044]: pgmap v2716: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:53:57.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:56.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:56.851619+0000) 2022-09-10T05:53:57.258 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:53:56.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:53:57.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:57.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:57.256953+0000) 2022-09-10T05:53:58.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:57.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:57.851826+0000) 2022-09-10T05:53:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:58.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:58.257208+0000) 2022-09-10T05:53:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:53:58 smithi104 ceph-mon[97044]: pgmap v2717: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:53:58 smithi110 ceph-mon[97633]: pgmap v2717: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:53:59.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:58.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:58.852135+0000) 2022-09-10T05:53:59.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:53:59.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:59.257448+0000) 2022-09-10T05:54:00.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:53:59.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:53:59.852330+0000) 2022-09-10T05:54:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:00.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:00.257641+0000) 2022-09-10T05:54:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:00 smithi104 ceph-mon[97044]: pgmap v2718: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:00 smithi110 ceph-mon[97633]: pgmap v2718: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:01.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:00.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:00.852561+0000) 2022-09-10T05:54:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:01.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:01.257884+0000) 2022-09-10T05:54:02.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:01.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:01.852819+0000) 2022-09-10T05:54:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:02.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:02.258086+0000) 2022-09-10T05:54:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:02 smithi104 ceph-mon[97044]: pgmap v2719: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:02.916 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:02 smithi110 ceph-mon[97633]: pgmap v2719: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:03.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:02.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:02.853082+0000) 2022-09-10T05:54:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:03.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:03.258295+0000) 2022-09-10T05:54:04.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:03.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:03.853345+0000) 2022-09-10T05:54:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:04 smithi104 ceph-mon[97044]: pgmap v2720: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:04.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:04.258547+0000) 2022-09-10T05:54:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:04 smithi110 ceph-mon[97633]: pgmap v2720: 129 pgs: 129 active+clean; 7.9 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:05.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:04.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:04.853517+0000) 2022-09-10T05:54:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:05.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:05.258778+0000) 2022-09-10T05:54:06.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:05.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:05.853742+0000) 2022-09-10T05:54:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:06.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:06.258985+0000) 2022-09-10T05:54:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:06 smithi104 ceph-mon[97044]: pgmap v2721: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:54:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:06 smithi110 ceph-mon[97633]: pgmap v2721: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:54:07.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:06.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:06.853961+0000) 2022-09-10T05:54:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:07.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:07.259214+0000) 2022-09-10T05:54:08.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:07.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:07.854194+0000) 2022-09-10T05:54:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:08.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:08.259419+0000) 2022-09-10T05:54:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:08 smithi104 ceph-mon[97044]: pgmap v2722: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:54:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:08 smithi104 ceph-mon[97044]: osdmap e254: 12 total, 12 up, 12 in 2022-09-10T05:54:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:08 smithi110 ceph-mon[97633]: pgmap v2722: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:54:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:08 smithi110 ceph-mon[97633]: osdmap e254: 12 total, 12 up, 12 in 2022-09-10T05:54:09.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:08.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:08.854445+0000) 2022-09-10T05:54:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:09.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:09.259616+0000) 2022-09-10T05:54:10.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:09.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:09.854708+0000) 2022-09-10T05:54:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:10.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:10.259857+0000) 2022-09-10T05:54:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:10 smithi104 ceph-mon[97044]: pgmap v2724: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:54:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:10 smithi110 ceph-mon[97633]: pgmap v2724: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:54:11.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:10.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:10.854935+0000) 2022-09-10T05:54:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:11.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:11.260173+0000) 2022-09-10T05:54:12.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:11.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:11.855140+0000) 2022-09-10T05:54:12.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:54:12.003+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:54:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:12.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:12.260403+0000) 2022-09-10T05:54:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:12 smithi104 ceph-mon[97044]: pgmap v2725: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:54:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:12 smithi110 ceph-mon[97633]: pgmap v2725: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:54:13.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:12.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:12.855343+0000) 2022-09-10T05:54:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:13.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:13.260596+0000) 2022-09-10T05:54:14.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:13.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:13.855568+0000) 2022-09-10T05:54:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:14 smithi104 ceph-mon[97044]: osdmap e255: 12 total, 12 up, 12 in 2022-09-10T05:54:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:14 smithi104 ceph-mon[97044]: pgmap v2727: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:54:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:14.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:14.260817+0000) 2022-09-10T05:54:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:14 smithi110 ceph-mon[97633]: osdmap e255: 12 total, 12 up, 12 in 2022-09-10T05:54:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:14 smithi110 ceph-mon[97633]: pgmap v2727: 129 pgs: 129 active+clean; 7.9 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:54:15.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:14.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:14.855776+0000) 2022-09-10T05:54:15.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:15.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:15.260996+0000) 2022-09-10T05:54:16.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:15.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:15.855992+0000) 2022-09-10T05:54:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:16.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:16.261166+0000) 2022-09-10T05:54:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:16 smithi104 ceph-mon[97044]: pgmap v2728: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:54:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:16 smithi110 ceph-mon[97633]: pgmap v2728: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:54:17.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:16.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:16.856190+0000) 2022-09-10T05:54:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:17.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:17.261294+0000) 2022-09-10T05:54:18.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:17.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:17.856387+0000) 2022-09-10T05:54:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:18.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:18.261457+0000) 2022-09-10T05:54:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:18 smithi104 ceph-mon[97044]: pgmap v2729: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:54:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:18 smithi110 ceph-mon[97633]: pgmap v2729: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:54:19.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:18.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:18.856656+0000) 2022-09-10T05:54:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:19.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:19.261701+0000) 2022-09-10T05:54:20.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:19.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:19.856890+0000) 2022-09-10T05:54:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:20.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:20.261884+0000) 2022-09-10T05:54:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:20 smithi104 ceph-mon[97044]: pgmap v2730: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:54:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:20 smithi110 ceph-mon[97633]: pgmap v2730: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:54:21.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:20.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:20.857066+0000) 2022-09-10T05:54:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:21.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:21.262084+0000) 2022-09-10T05:54:22.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:21.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:21.857302+0000) 2022-09-10T05:54:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:22.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:22.262336+0000) 2022-09-10T05:54:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:22 smithi104 ceph-mon[97044]: pgmap v2731: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:54:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:22 smithi110 ceph-mon[97633]: pgmap v2731: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:54:23.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:22.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:22.857504+0000) 2022-09-10T05:54:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:23.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:23.262552+0000) 2022-09-10T05:54:24.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:23.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:23.857700+0000) 2022-09-10T05:54:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:24 smithi104 ceph-mon[97044]: pgmap v2732: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:54:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:24.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:24.262792+0000) 2022-09-10T05:54:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:24 smithi110 ceph-mon[97633]: pgmap v2732: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:54:25.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:24.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:24.857892+0000) 2022-09-10T05:54:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:25.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:25.262968+0000) 2022-09-10T05:54:26.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:25.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:25.858070+0000) 2022-09-10T05:54:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:26.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:26.263171+0000) 2022-09-10T05:54:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:26 smithi104 ceph-mon[97044]: pgmap v2733: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:54:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:26 smithi110 ceph-mon[97633]: pgmap v2733: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:54:27.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:54:27.016+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:54:27.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:26.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:26.858256+0000) 2022-09-10T05:54:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:27.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:27.263378+0000) 2022-09-10T05:54:28.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:27.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:27.858464+0000) 2022-09-10T05:54:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:28.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:28.263621+0000) 2022-09-10T05:54:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:28 smithi104 ceph-mon[97044]: pgmap v2734: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:28 smithi110 ceph-mon[97633]: pgmap v2734: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:29.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:28.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:28.858712+0000) 2022-09-10T05:54:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:29.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:29.263826+0000) 2022-09-10T05:54:30.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:29.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:29.858911+0000) 2022-09-10T05:54:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:30.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:30.264037+0000) 2022-09-10T05:54:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:30 smithi104 ceph-mon[97044]: pgmap v2735: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:30 smithi110 ceph-mon[97633]: pgmap v2735: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:31.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:30.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:30.859156+0000) 2022-09-10T05:54:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:31.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:31.264312+0000) 2022-09-10T05:54:32.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:31.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:31.859460+0000) 2022-09-10T05:54:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:32.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:32.264592+0000) 2022-09-10T05:54:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:32 smithi104 ceph-mon[97044]: pgmap v2736: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:32.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:32 smithi110 ceph-mon[97633]: pgmap v2736: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:33.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:32.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:32.859691+0000) 2022-09-10T05:54:33.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:33.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:33.264806+0000) 2022-09-10T05:54:34.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:33.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:33.859921+0000) 2022-09-10T05:54:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:34 smithi104 ceph-mon[97044]: pgmap v2737: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:34.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:34.265075+0000) 2022-09-10T05:54:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:34 smithi110 ceph-mon[97633]: pgmap v2737: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:35.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:34.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:34.860216+0000) 2022-09-10T05:54:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:35.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:35.265392+0000) 2022-09-10T05:54:36.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:35.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:35.860435+0000) 2022-09-10T05:54:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:36.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:36.265647+0000) 2022-09-10T05:54:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:36 smithi104 ceph-mon[97044]: pgmap v2738: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:54:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:36 smithi110 ceph-mon[97633]: pgmap v2738: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:54:37.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:36.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:36.860603+0000) 2022-09-10T05:54:37.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:37.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:37.265911+0000) 2022-09-10T05:54:38.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:37.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:37.860744+0000) 2022-09-10T05:54:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:38.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:38.266211+0000) 2022-09-10T05:54:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:38 smithi104 ceph-mon[97044]: pgmap v2739: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:38.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:38 smithi110 ceph-mon[97633]: pgmap v2739: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:39.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:38.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:38.860873+0000) 2022-09-10T05:54:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:39.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:39.266452+0000) 2022-09-10T05:54:40.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:39.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:39.861160+0000) 2022-09-10T05:54:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:40.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:40.266625+0000) 2022-09-10T05:54:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:40 smithi104 ceph-mon[97044]: pgmap v2740: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:40 smithi110 ceph-mon[97633]: pgmap v2740: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:41.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:40.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:40.861390+0000) 2022-09-10T05:54:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:41.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:41.266876+0000) 2022-09-10T05:54:42.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:41.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:41.861656+0000) 2022-09-10T05:54:42.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:54:42.028+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:54:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:42.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:42.267069+0000) 2022-09-10T05:54:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:42 smithi104 ceph-mon[97044]: pgmap v2741: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:42 smithi110 ceph-mon[97633]: pgmap v2741: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:43.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:42.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:42.861850+0000) 2022-09-10T05:54:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:43.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:43.267200+0000) 2022-09-10T05:54:44.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:43.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:43.862059+0000) 2022-09-10T05:54:44.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:44 smithi104 ceph-mon[97044]: pgmap v2742: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:44.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:54:44.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:54:44.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:54:44.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:54:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:44.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:44.267330+0000) 2022-09-10T05:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:44 smithi110 ceph-mon[97633]: pgmap v2742: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:54:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:54:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:54:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:54:45.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:44.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:44.862349+0000) 2022-09-10T05:54:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:45.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:45.267568+0000) 2022-09-10T05:54:46.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:45.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:45.862524+0000) 2022-09-10T05:54:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:46.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:46.267740+0000) 2022-09-10T05:54:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:46 smithi104 ceph-mon[97044]: pgmap v2743: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:54:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:46 smithi110 ceph-mon[97633]: pgmap v2743: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:54:47.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:46.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:46.862698+0000) 2022-09-10T05:54:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:47.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:47.267986+0000) 2022-09-10T05:54:48.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:47.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:47.862976+0000) 2022-09-10T05:54:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:48.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:48.268189+0000) 2022-09-10T05:54:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:48 smithi104 ceph-mon[97044]: pgmap v2744: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:48 smithi110 ceph-mon[97633]: pgmap v2744: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:49.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:48.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:48.863222+0000) 2022-09-10T05:54:49.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:49.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:49.268432+0000) 2022-09-10T05:54:50.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:49.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:49.863406+0000) 2022-09-10T05:54:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:50.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:50.268677+0000) 2022-09-10T05:54:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:50 smithi104 ceph-mon[97044]: pgmap v2745: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:50 smithi110 ceph-mon[97633]: pgmap v2745: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:54:51.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:50.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:50.863688+0000) 2022-09-10T05:54:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:51.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:51.268934+0000) 2022-09-10T05:54:52.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:51.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:51.863874+0000) 2022-09-10T05:54:52.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:52.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:52.269207+0000) 2022-09-10T05:54:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:52 smithi104 ceph-mon[97044]: pgmap v2746: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:52 smithi110 ceph-mon[97633]: pgmap v2746: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:54:53.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:52.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:52.864124+0000) 2022-09-10T05:54:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:53.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:53.269544+0000) 2022-09-10T05:54:54.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:53.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:53.864393+0000) 2022-09-10T05:54:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:54 smithi104 ceph-mon[97044]: pgmap v2747: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:54.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:54.269781+0000) 2022-09-10T05:54:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:54 smithi110 ceph-mon[97633]: pgmap v2747: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:55.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:54.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:54.864591+0000) 2022-09-10T05:54:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:55.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:55.269984+0000) 2022-09-10T05:54:56.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:55.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:55.864867+0000) 2022-09-10T05:54:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:56.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:56.270270+0000) 2022-09-10T05:54:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:56 smithi104 ceph-mon[97044]: pgmap v2748: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:54:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:56 smithi110 ceph-mon[97633]: pgmap v2748: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:54:57.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:56.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:56.865026+0000) 2022-09-10T05:54:57.273 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:54:57.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:54:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:57.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:57.270524+0000) 2022-09-10T05:54:58.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:57.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:57.865267+0000) 2022-09-10T05:54:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:58.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:58.270798+0000) 2022-09-10T05:54:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:54:58 smithi104 ceph-mon[97044]: pgmap v2749: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:54:58 smithi110 ceph-mon[97633]: pgmap v2749: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:54:59.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:58.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:58.865505+0000) 2022-09-10T05:54:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:54:59.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:59.271074+0000) 2022-09-10T05:55:00.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:54:59.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:54:59.865704+0000) 2022-09-10T05:55:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:00.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:00.271374+0000) 2022-09-10T05:55:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:00 smithi104 ceph-mon[97044]: pgmap v2750: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:00 smithi110 ceph-mon[97633]: pgmap v2750: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:01.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:00.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:00.865874+0000) 2022-09-10T05:55:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:01.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:01.271596+0000) 2022-09-10T05:55:02.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:01.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:01.866116+0000) 2022-09-10T05:55:02.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:02.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:02.271861+0000) 2022-09-10T05:55:02.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:02 smithi104 ceph-mon[97044]: pgmap v2751: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:02 smithi110 ceph-mon[97633]: pgmap v2751: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:03.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:02.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:02.866304+0000) 2022-09-10T05:55:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:03.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:03.272047+0000) 2022-09-10T05:55:04.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:03.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:03.866580+0000) 2022-09-10T05:55:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:04.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:04.272293+0000) 2022-09-10T05:55:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:04 smithi110 ceph-mon[97633]: pgmap v2752: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:04.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:04 smithi104 ceph-mon[97044]: pgmap v2752: 129 pgs: 129 active+clean; 7.9 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:05.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:04.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:04.866809+0000) 2022-09-10T05:55:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:05.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:05.272559+0000) 2022-09-10T05:55:06.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:05.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:05.867083+0000) 2022-09-10T05:55:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:06.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:06.272824+0000) 2022-09-10T05:55:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:06 smithi104 ceph-mon[97044]: pgmap v2753: 129 pgs: 129 active+clean; 7.9 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:55:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:06 smithi110 ceph-mon[97633]: pgmap v2753: 129 pgs: 129 active+clean; 7.9 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:55:07.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:06.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:06.867341+0000) 2022-09-10T05:55:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:07.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:07.273117+0000) 2022-09-10T05:55:08.149 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:07.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:07.867597+0000) 2022-09-10T05:55:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:08.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:08.273328+0000) 2022-09-10T05:55:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:08 smithi104 ceph-mon[97044]: pgmap v2754: 129 pgs: 129 active+clean; 7.9 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:55:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:08 smithi104 ceph-mon[97044]: osdmap e256: 12 total, 12 up, 12 in 2022-09-10T05:55:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:08 smithi110 ceph-mon[97633]: pgmap v2754: 129 pgs: 129 active+clean; 7.9 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:55:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:08 smithi110 ceph-mon[97633]: osdmap e256: 12 total, 12 up, 12 in 2022-09-10T05:55:09.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:08.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:08.867845+0000) 2022-09-10T05:55:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:09.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:09.273523+0000) 2022-09-10T05:55:10.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:09.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:09.868047+0000) 2022-09-10T05:55:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:10.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:10.273802+0000) 2022-09-10T05:55:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:10 smithi104 ceph-mon[97044]: pgmap v2756: 129 pgs: 129 active+clean; 7.9 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:55:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:10 smithi110 ceph-mon[97633]: pgmap v2756: 129 pgs: 129 active+clean; 7.9 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:55:11.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:10.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:10.868283+0000) 2022-09-10T05:55:11.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:11.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:11.274043+0000) 2022-09-10T05:55:12.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:11.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:11.868460+0000) 2022-09-10T05:55:12.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:55:12.051+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:55:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:12.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:12.274262+0000) 2022-09-10T05:55:12.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:12 smithi104 ceph-mon[97044]: pgmap v2757: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:55:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:12 smithi110 ceph-mon[97633]: pgmap v2757: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:55:13.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:12.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:12.868636+0000) 2022-09-10T05:55:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:13.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:13.274484+0000) 2022-09-10T05:55:14.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:13.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:13.868885+0000) 2022-09-10T05:55:14.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:14 smithi104 ceph-mon[97044]: osdmap e257: 12 total, 12 up, 12 in 2022-09-10T05:55:14.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:14 smithi104 ceph-mon[97044]: pgmap v2759: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:55:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:14.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:14.274720+0000) 2022-09-10T05:55:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:14 smithi110 ceph-mon[97633]: osdmap e257: 12 total, 12 up, 12 in 2022-09-10T05:55:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:14 smithi110 ceph-mon[97633]: pgmap v2759: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:55:15.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:14.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:14.869135+0000) 2022-09-10T05:55:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:15.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:15.275003+0000) 2022-09-10T05:55:16.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:15.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:15.869319+0000) 2022-09-10T05:55:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:16.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:16.275274+0000) 2022-09-10T05:55:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:16 smithi104 ceph-mon[97044]: pgmap v2760: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:55:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:16 smithi110 ceph-mon[97633]: pgmap v2760: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:55:17.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:16.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:16.869508+0000) 2022-09-10T05:55:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:17.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:17.275468+0000) 2022-09-10T05:55:18.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:17.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:17.869696+0000) 2022-09-10T05:55:18.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:18.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:18.275709+0000) 2022-09-10T05:55:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:18 smithi104 ceph-mon[97044]: pgmap v2761: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:55:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:18 smithi110 ceph-mon[97633]: pgmap v2761: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:55:19.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:18.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:18.869873+0000) 2022-09-10T05:55:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:19.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:19.275963+0000) 2022-09-10T05:55:20.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:19.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:19.870027+0000) 2022-09-10T05:55:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:20.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:20.276196+0000) 2022-09-10T05:55:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:20 smithi104 ceph-mon[97044]: pgmap v2762: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:55:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:20 smithi110 ceph-mon[97633]: pgmap v2762: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:55:21.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:20.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:20.870236+0000) 2022-09-10T05:55:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:21.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:21.276451+0000) 2022-09-10T05:55:22.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:21.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:21.870418+0000) 2022-09-10T05:55:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:22.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:22.276701+0000) 2022-09-10T05:55:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:22 smithi104 ceph-mon[97044]: pgmap v2763: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:55:22.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:22 smithi110 ceph-mon[97633]: pgmap v2763: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:55:23.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:22.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:22.870645+0000) 2022-09-10T05:55:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:23.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:23.276942+0000) 2022-09-10T05:55:24.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:23.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:23.870897+0000) 2022-09-10T05:55:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:24.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:24.277213+0000) 2022-09-10T05:55:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:24 smithi110 ceph-mon[97633]: pgmap v2764: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:55:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:24 smithi104 ceph-mon[97044]: pgmap v2764: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:55:25.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:24.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:24.871144+0000) 2022-09-10T05:55:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:25.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:25.277466+0000) 2022-09-10T05:55:26.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:25.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:25.871327+0000) 2022-09-10T05:55:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:26.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:26.277711+0000) 2022-09-10T05:55:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:26 smithi104 ceph-mon[97044]: pgmap v2765: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:55:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:26 smithi110 ceph-mon[97633]: pgmap v2765: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:55:27.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:26.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:26.871530+0000) 2022-09-10T05:55:27.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:55:27.067+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:55:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:27.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:27.277955+0000) 2022-09-10T05:55:28.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:27.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:27.871745+0000) 2022-09-10T05:55:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:28.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:28.278228+0000) 2022-09-10T05:55:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:28 smithi104 ceph-mon[97044]: pgmap v2766: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:28 smithi110 ceph-mon[97633]: pgmap v2766: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:29.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:28.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:28.871921+0000) 2022-09-10T05:55:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:29.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:29.278480+0000) 2022-09-10T05:55:30.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:29.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:29.872176+0000) 2022-09-10T05:55:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:30.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:30.278715+0000) 2022-09-10T05:55:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:30 smithi104 ceph-mon[97044]: pgmap v2767: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:30 smithi110 ceph-mon[97633]: pgmap v2767: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:31.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:30.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:30.872462+0000) 2022-09-10T05:55:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:31.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:31.278922+0000) 2022-09-10T05:55:32.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:31.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:31.872725+0000) 2022-09-10T05:55:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:32.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:32.279164+0000) 2022-09-10T05:55:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:32 smithi104 ceph-mon[97044]: pgmap v2768: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:32.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:32 smithi110 ceph-mon[97633]: pgmap v2768: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:33.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:32.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:32.872963+0000) 2022-09-10T05:55:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:33.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:33.279393+0000) 2022-09-10T05:55:34.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:33.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:33.873241+0000) 2022-09-10T05:55:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:34.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:34.279679+0000) 2022-09-10T05:55:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:34 smithi110 ceph-mon[97633]: pgmap v2769: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:34 smithi104 ceph-mon[97044]: pgmap v2769: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:35.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:34.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:34.873417+0000) 2022-09-10T05:55:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:35.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:35.279890+0000) 2022-09-10T05:55:36.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:35.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:35.873652+0000) 2022-09-10T05:55:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:36.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:36.280068+0000) 2022-09-10T05:55:36.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:36 smithi104 ceph-mon[97044]: pgmap v2770: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:55:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:36 smithi110 ceph-mon[97633]: pgmap v2770: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:55:37.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:36.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:36.873825+0000) 2022-09-10T05:55:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:37.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:37.280273+0000) 2022-09-10T05:55:38.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:37.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:37.874036+0000) 2022-09-10T05:55:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:38.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:38.280522+0000) 2022-09-10T05:55:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:38 smithi104 ceph-mon[97044]: pgmap v2771: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:38 smithi110 ceph-mon[97633]: pgmap v2771: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:39.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:38.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:38.874252+0000) 2022-09-10T05:55:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:39.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:39.280760+0000) 2022-09-10T05:55:40.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:39.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:39.874513+0000) 2022-09-10T05:55:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:40.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:40.280943+0000) 2022-09-10T05:55:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:40 smithi104 ceph-mon[97044]: pgmap v2772: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:40 smithi110 ceph-mon[97633]: pgmap v2772: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:41.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:40.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:40.874821+0000) 2022-09-10T05:55:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:41.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:41.281201+0000) 2022-09-10T05:55:42.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:41.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:41.875178+0000) 2022-09-10T05:55:42.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:55:42.082+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:55:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:42.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:42.281446+0000) 2022-09-10T05:55:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:42 smithi104 ceph-mon[97044]: pgmap v2773: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:42 smithi110 ceph-mon[97633]: pgmap v2773: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:43.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:42.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:42.875412+0000) 2022-09-10T05:55:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:43.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:43.281638+0000) 2022-09-10T05:55:44.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:43.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:43.875567+0000) 2022-09-10T05:55:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:44.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:44.281894+0000) 2022-09-10T05:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:44 smithi110 ceph-mon[97633]: pgmap v2774: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:55:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:55:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:55:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:55:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:44 smithi104 ceph-mon[97044]: pgmap v2774: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:55:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:55:44.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:55:44.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:55:45.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:44.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:44.875731+0000) 2022-09-10T05:55:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:45.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:45.282184+0000) 2022-09-10T05:55:46.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:45.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:45.875917+0000) 2022-09-10T05:55:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:46.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:46.282413+0000) 2022-09-10T05:55:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:46 smithi104 ceph-mon[97044]: pgmap v2775: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:55:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:46 smithi110 ceph-mon[97633]: pgmap v2775: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:55:47.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:46.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:46.876182+0000) 2022-09-10T05:55:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:47.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:47.282560+0000) 2022-09-10T05:55:48.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:47.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:47.876404+0000) 2022-09-10T05:55:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:48.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:48.282716+0000) 2022-09-10T05:55:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:48 smithi104 ceph-mon[97044]: pgmap v2776: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:48 smithi110 ceph-mon[97633]: pgmap v2776: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:49.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:48.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:48.876551+0000) 2022-09-10T05:55:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:49.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:49.282949+0000) 2022-09-10T05:55:50.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:49.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:49.876830+0000) 2022-09-10T05:55:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:50.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:50.283205+0000) 2022-09-10T05:55:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:50 smithi104 ceph-mon[97044]: pgmap v2777: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:50.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:50 smithi110 ceph-mon[97633]: pgmap v2777: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:55:51.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:50.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:50.877072+0000) 2022-09-10T05:55:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:51.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:51.283416+0000) 2022-09-10T05:55:52.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:51.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:51.877331+0000) 2022-09-10T05:55:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:52.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:52.283650+0000) 2022-09-10T05:55:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:52 smithi104 ceph-mon[97044]: pgmap v2778: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:52 smithi110 ceph-mon[97633]: pgmap v2778: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:55:53.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:52.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:52.877514+0000) 2022-09-10T05:55:53.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:53.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:53.283862+0000) 2022-09-10T05:55:54.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:53.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:53.877696+0000) 2022-09-10T05:55:54.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:54.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:54.284064+0000) 2022-09-10T05:55:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:54 smithi110 ceph-mon[97633]: pgmap v2779: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:54 smithi104 ceph-mon[97044]: pgmap v2779: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:55.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:54.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:54.877866+0000) 2022-09-10T05:55:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:55.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:55.284282+0000) 2022-09-10T05:55:56.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:55.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:55.878003+0000) 2022-09-10T05:55:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:56.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:56.284501+0000) 2022-09-10T05:55:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:56 smithi104 ceph-mon[97044]: pgmap v2780: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:55:56.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:56 smithi110 ceph-mon[97633]: pgmap v2780: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:55:57.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:56.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:56.878231+0000) 2022-09-10T05:55:57.324 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:55:57.094+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:55:57.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:57.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:57.284722+0000) 2022-09-10T05:55:58.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:57.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:57.878444+0000) 2022-09-10T05:55:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:58.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:58.284930+0000) 2022-09-10T05:55:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:55:58 smithi104 ceph-mon[97044]: pgmap v2781: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:55:58 smithi110 ceph-mon[97633]: pgmap v2781: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:55:59.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:58.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:58.878641+0000) 2022-09-10T05:55:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:55:59.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:59.285200+0000) 2022-09-10T05:56:00.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:55:59.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:55:59.878876+0000) 2022-09-10T05:56:00.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:00.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:00.285377+0000) 2022-09-10T05:56:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:00 smithi104 ceph-mon[97044]: pgmap v2782: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:56:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:00 smithi110 ceph-mon[97633]: pgmap v2782: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:56:01.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:00.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:00.879022+0000) 2022-09-10T05:56:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:01.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:01.285635+0000) 2022-09-10T05:56:02.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:01.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:01.879263+0000) 2022-09-10T05:56:02.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:02.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:02.285850+0000) 2022-09-10T05:56:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:02 smithi104 ceph-mon[97044]: pgmap v2783: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:56:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:02 smithi110 ceph-mon[97633]: pgmap v2783: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:56:03.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:02.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:02.879530+0000) 2022-09-10T05:56:03.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:03.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:03.286067+0000) 2022-09-10T05:56:04.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:03.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:03.879751+0000) 2022-09-10T05:56:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:04.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:04.286353+0000) 2022-09-10T05:56:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:04 smithi110 ceph-mon[97633]: pgmap v2784: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:04 smithi104 ceph-mon[97044]: pgmap v2784: 129 pgs: 129 active+clean; 7.9 MiB data, 456 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:05.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:04.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:04.879951+0000) 2022-09-10T05:56:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:05.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:05.286600+0000) 2022-09-10T05:56:06.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:05.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:05.880223+0000) 2022-09-10T05:56:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:06.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:06.286813+0000) 2022-09-10T05:56:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:06 smithi104 ceph-mon[97044]: pgmap v2785: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:56:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:06 smithi110 ceph-mon[97633]: pgmap v2785: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:56:07.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:06.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:06.880489+0000) 2022-09-10T05:56:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:07.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:07.287041+0000) 2022-09-10T05:56:08.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:07.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:07.880716+0000) 2022-09-10T05:56:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:08.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:08.287332+0000) 2022-09-10T05:56:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:08 smithi104 ceph-mon[97044]: pgmap v2786: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:56:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:08 smithi104 ceph-mon[97044]: osdmap e258: 12 total, 12 up, 12 in 2022-09-10T05:56:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:08 smithi110 ceph-mon[97633]: pgmap v2786: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:56:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:08 smithi110 ceph-mon[97633]: osdmap e258: 12 total, 12 up, 12 in 2022-09-10T05:56:09.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:08.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:08.880980+0000) 2022-09-10T05:56:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:09.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:09.287544+0000) 2022-09-10T05:56:10.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:09.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:09.881269+0000) 2022-09-10T05:56:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:10.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:10.287785+0000) 2022-09-10T05:56:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:10 smithi104 ceph-mon[97044]: pgmap v2788: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:56:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:10 smithi110 ceph-mon[97633]: pgmap v2788: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:56:11.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:10.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:10.881467+0000) 2022-09-10T05:56:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:11.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:11.287998+0000) 2022-09-10T05:56:12.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:11.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:11.881649+0000) 2022-09-10T05:56:12.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:56:12.110+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:56:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:12.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:12.288160+0000) 2022-09-10T05:56:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:12 smithi104 ceph-mon[97044]: pgmap v2789: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:56:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:12 smithi110 ceph-mon[97633]: pgmap v2789: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:56:13.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:12.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:12.881882+0000) 2022-09-10T05:56:13.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:13.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:13.288353+0000) 2022-09-10T05:56:14.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:13.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:13.882141+0000) 2022-09-10T05:56:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:14.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:14.288601+0000) 2022-09-10T05:56:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:14 smithi110 ceph-mon[97633]: osdmap e259: 12 total, 12 up, 12 in 2022-09-10T05:56:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:14 smithi110 ceph-mon[97633]: pgmap v2791: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:56:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:14 smithi104 ceph-mon[97044]: osdmap e259: 12 total, 12 up, 12 in 2022-09-10T05:56:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:14 smithi104 ceph-mon[97044]: pgmap v2791: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:56:15.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:14.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:14.882301+0000) 2022-09-10T05:56:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:15.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:15.288837+0000) 2022-09-10T05:56:16.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:15.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:15.882551+0000) 2022-09-10T05:56:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:16.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:16.289129+0000) 2022-09-10T05:56:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:16 smithi104 ceph-mon[97044]: pgmap v2792: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:56:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:16 smithi110 ceph-mon[97633]: pgmap v2792: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:56:17.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:16.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:16.882815+0000) 2022-09-10T05:56:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:17.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:17.289275+0000) 2022-09-10T05:56:18.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:17.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:17.883081+0000) 2022-09-10T05:56:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:18.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:18.289508+0000) 2022-09-10T05:56:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:18 smithi104 ceph-mon[97044]: pgmap v2793: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:56:18.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:18 smithi110 ceph-mon[97633]: pgmap v2793: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:56:19.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:18.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:18.883342+0000) 2022-09-10T05:56:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:19.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:19.289715+0000) 2022-09-10T05:56:20.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:19.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:19.883560+0000) 2022-09-10T05:56:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:20.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:20.289938+0000) 2022-09-10T05:56:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:20 smithi104 ceph-mon[97044]: pgmap v2794: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:56:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:20 smithi110 ceph-mon[97633]: pgmap v2794: 129 pgs: 129 active+clean; 7.9 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:56:21.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:20.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:20.883712+0000) 2022-09-10T05:56:21.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:21.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:21.290210+0000) 2022-09-10T05:56:22.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:21.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:21.883962+0000) 2022-09-10T05:56:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:22.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:22.290502+0000) 2022-09-10T05:56:22.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:22 smithi104 ceph-mon[97044]: pgmap v2795: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:56:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:22 smithi110 ceph-mon[97633]: pgmap v2795: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:56:23.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:22.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:22.884144+0000) 2022-09-10T05:56:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:23.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:23.290722+0000) 2022-09-10T05:56:24.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:23.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:23.884362+0000) 2022-09-10T05:56:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:24.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:24.290993+0000) 2022-09-10T05:56:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:24 smithi110 ceph-mon[97633]: pgmap v2796: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:56:24.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:24 smithi104 ceph-mon[97044]: pgmap v2796: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:56:25.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:24.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:24.884541+0000) 2022-09-10T05:56:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:25.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:25.291267+0000) 2022-09-10T05:56:26.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:25.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:25.884808+0000) 2022-09-10T05:56:26.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:26.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:26.291527+0000) 2022-09-10T05:56:26.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:26 smithi104 ceph-mon[97044]: pgmap v2797: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:56:26.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:26 smithi110 ceph-mon[97633]: pgmap v2797: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:56:27.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:26.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:26.885025+0000) 2022-09-10T05:56:27.300 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:56:27.125+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:56:27.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:27.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:27.291712+0000) 2022-09-10T05:56:28.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:27.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:27.885255+0000) 2022-09-10T05:56:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:28.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:28.291888+0000) 2022-09-10T05:56:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:28 smithi104 ceph-mon[97044]: pgmap v2798: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:56:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:28 smithi110 ceph-mon[97633]: pgmap v2798: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:56:29.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:28.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:28.885475+0000) 2022-09-10T05:56:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:29.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:29.292125+0000) 2022-09-10T05:56:30.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:29.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:29.885683+0000) 2022-09-10T05:56:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:30.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:30.292311+0000) 2022-09-10T05:56:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:30 smithi104 ceph-mon[97044]: pgmap v2799: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2022-09-10T05:56:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:30 smithi110 ceph-mon[97633]: pgmap v2799: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2022-09-10T05:56:31.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:30.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:30.885864+0000) 2022-09-10T05:56:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:31.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:31.292570+0000) 2022-09-10T05:56:32.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:31.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:31.886145+0000) 2022-09-10T05:56:32.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:32.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:32.292784+0000) 2022-09-10T05:56:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:32 smithi104 ceph-mon[97044]: pgmap v2800: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T05:56:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:32 smithi110 ceph-mon[97633]: pgmap v2800: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T05:56:33.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:32.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:32.886380+0000) 2022-09-10T05:56:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:33.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:33.292970+0000) 2022-09-10T05:56:34.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:33.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:33.886630+0000) 2022-09-10T05:56:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:34.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:34.293178+0000) 2022-09-10T05:56:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:34 smithi110 ceph-mon[97633]: pgmap v2801: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:56:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:34 smithi104 ceph-mon[97044]: pgmap v2801: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T05:56:35.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:34.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:34.886855+0000) 2022-09-10T05:56:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:35.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:35.293387+0000) 2022-09-10T05:56:36.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:35.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:35.887049+0000) 2022-09-10T05:56:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:36.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:36.293584+0000) 2022-09-10T05:56:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:36 smithi104 ceph-mon[97044]: pgmap v2802: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:56:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:36 smithi110 ceph-mon[97633]: pgmap v2802: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:56:37.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:36.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:36.887249+0000) 2022-09-10T05:56:37.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:37.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:37.293796+0000) 2022-09-10T05:56:38.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:37.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:37.887512+0000) 2022-09-10T05:56:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:38.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:38.294035+0000) 2022-09-10T05:56:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:38 smithi104 ceph-mon[97044]: pgmap v2803: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:38 smithi110 ceph-mon[97633]: pgmap v2803: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:39.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:38.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:38.887689+0000) 2022-09-10T05:56:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:39.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:39.294218+0000) 2022-09-10T05:56:40.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:39.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:39.887941+0000) 2022-09-10T05:56:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:40.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:40.294358+0000) 2022-09-10T05:56:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:40 smithi104 ceph-mon[97044]: pgmap v2804: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:56:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:40 smithi110 ceph-mon[97633]: pgmap v2804: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:56:41.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:40.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:40.888156+0000) 2022-09-10T05:56:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:41.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:41.294582+0000) 2022-09-10T05:56:42.141 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:41.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:41.888386+0000) 2022-09-10T05:56:42.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:56:42.140+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:56:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:42.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:42.294854+0000) 2022-09-10T05:56:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:42 smithi104 ceph-mon[97044]: pgmap v2805: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:56:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:42 smithi110 ceph-mon[97633]: pgmap v2805: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:56:43.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:42.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:42.888599+0000) 2022-09-10T05:56:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:43.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:43.295079+0000) 2022-09-10T05:56:44.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:43.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:43.888818+0000) 2022-09-10T05:56:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:44.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:44.295347+0000) 2022-09-10T05:56:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:44 smithi110 ceph-mon[97633]: pgmap v2806: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:56:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:56:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:56:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:56:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:44 smithi104 ceph-mon[97044]: pgmap v2806: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:56:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:56:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:56:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:56:45.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:44.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:44.889030+0000) 2022-09-10T05:56:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:45.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:45.295626+0000) 2022-09-10T05:56:46.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:45.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:45.889197+0000) 2022-09-10T05:56:46.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:46.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:46.295879+0000) 2022-09-10T05:56:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:46 smithi104 ceph-mon[97044]: pgmap v2807: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:56:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:46 smithi110 ceph-mon[97633]: pgmap v2807: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:56:47.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:46.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:46.889447+0000) 2022-09-10T05:56:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:47.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:47.296048+0000) 2022-09-10T05:56:48.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:47.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:47.889628+0000) 2022-09-10T05:56:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:48.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:48.296208+0000) 2022-09-10T05:56:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:48 smithi104 ceph-mon[97044]: pgmap v2808: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:48 smithi110 ceph-mon[97633]: pgmap v2808: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:49.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:48.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:48.889861+0000) 2022-09-10T05:56:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:49.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:49.296389+0000) 2022-09-10T05:56:50.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:49.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:49.890078+0000) 2022-09-10T05:56:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:50.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:50.296635+0000) 2022-09-10T05:56:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:50 smithi104 ceph-mon[97044]: pgmap v2809: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:56:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:50 smithi110 ceph-mon[97633]: pgmap v2809: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:56:51.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:50.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:50.890322+0000) 2022-09-10T05:56:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:51.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:51.296846+0000) 2022-09-10T05:56:52.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:51.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:51.890558+0000) 2022-09-10T05:56:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:52.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:52.297087+0000) 2022-09-10T05:56:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:52 smithi104 ceph-mon[97044]: pgmap v2810: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:56:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:52 smithi110 ceph-mon[97633]: pgmap v2810: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:56:53.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:52.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:52.890761+0000) 2022-09-10T05:56:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:53.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:53.297344+0000) 2022-09-10T05:56:54.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:53.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:53.890957+0000) 2022-09-10T05:56:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:54.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:54.297601+0000) 2022-09-10T05:56:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:54 smithi110 ceph-mon[97633]: pgmap v2811: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:54 smithi104 ceph-mon[97044]: pgmap v2811: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:55.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:54.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:54.891192+0000) 2022-09-10T05:56:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:55.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:55.297849+0000) 2022-09-10T05:56:56.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:55.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:55.891425+0000) 2022-09-10T05:56:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:56.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:56.298047+0000) 2022-09-10T05:56:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:56 smithi104 ceph-mon[97044]: pgmap v2812: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:56:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:56 smithi110 ceph-mon[97633]: pgmap v2812: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:56:57.167 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:56.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:56.891576+0000) 2022-09-10T05:56:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:56:57.152+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:56:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:57.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:57.298309+0000) 2022-09-10T05:56:58.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:57.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:57.891796+0000) 2022-09-10T05:56:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:58.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:58.298595+0000) 2022-09-10T05:56:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:56:58 smithi104 ceph-mon[97044]: pgmap v2813: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:56:58 smithi110 ceph-mon[97633]: pgmap v2813: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:56:59.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:58.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:58.892048+0000) 2022-09-10T05:56:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:56:59.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:59.298856+0000) 2022-09-10T05:57:00.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:56:59.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:56:59.892281+0000) 2022-09-10T05:57:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:00.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:00.299160+0000) 2022-09-10T05:57:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:00 smithi104 ceph-mon[97044]: pgmap v2814: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:00.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:00 smithi110 ceph-mon[97633]: pgmap v2814: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:01.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:00.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:00.892420+0000) 2022-09-10T05:57:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:01.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:01.299368+0000) 2022-09-10T05:57:02.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:01.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:01.892628+0000) 2022-09-10T05:57:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:02.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:02.299572+0000) 2022-09-10T05:57:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:02 smithi104 ceph-mon[97044]: pgmap v2815: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:02 smithi110 ceph-mon[97633]: pgmap v2815: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:03.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:02.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:02.892896+0000) 2022-09-10T05:57:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:03.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:03.299813+0000) 2022-09-10T05:57:04.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:03.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:03.893081+0000) 2022-09-10T05:57:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:04.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:04.300056+0000) 2022-09-10T05:57:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:04 smithi110 ceph-mon[97633]: pgmap v2816: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:04.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:04 smithi104 ceph-mon[97044]: pgmap v2816: 129 pgs: 129 active+clean; 7.9 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:05.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:04.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:04.893295+0000) 2022-09-10T05:57:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:05.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:05.300299+0000) 2022-09-10T05:57:06.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:05.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:05.893498+0000) 2022-09-10T05:57:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:06.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:06.300511+0000) 2022-09-10T05:57:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:06 smithi104 ceph-mon[97044]: pgmap v2817: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:57:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:06 smithi110 ceph-mon[97633]: pgmap v2817: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:57:07.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:06.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:06.893705+0000) 2022-09-10T05:57:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:07.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:07.300769+0000) 2022-09-10T05:57:08.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:07.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:07.893968+0000) 2022-09-10T05:57:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:08.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:08.300995+0000) 2022-09-10T05:57:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:08 smithi104 ceph-mon[97044]: pgmap v2818: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:57:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:08 smithi104 ceph-mon[97044]: osdmap e260: 12 total, 12 up, 12 in 2022-09-10T05:57:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:08 smithi110 ceph-mon[97633]: pgmap v2818: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:57:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:08 smithi110 ceph-mon[97633]: osdmap e260: 12 total, 12 up, 12 in 2022-09-10T05:57:09.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:08.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:08.894205+0000) 2022-09-10T05:57:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:09.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:09.301234+0000) 2022-09-10T05:57:10.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:09.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:09.894445+0000) 2022-09-10T05:57:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:10.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:10.301500+0000) 2022-09-10T05:57:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:10 smithi104 ceph-mon[97044]: pgmap v2820: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:57:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:10 smithi110 ceph-mon[97633]: pgmap v2820: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:57:11.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:10.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:10.894605+0000) 2022-09-10T05:57:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:11.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:11.301711+0000) 2022-09-10T05:57:12.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:11.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:11.894820+0000) 2022-09-10T05:57:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:57:12.160+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:57:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:12.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:12.301848+0000) 2022-09-10T05:57:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:12 smithi104 ceph-mon[97044]: pgmap v2821: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:57:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:12 smithi110 ceph-mon[97633]: pgmap v2821: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:57:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:12.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:12.895048+0000) 2022-09-10T05:57:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:13.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:13.302078+0000) 2022-09-10T05:57:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:13.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:13.895179+0000) 2022-09-10T05:57:14.564 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:14.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:14.302325+0000) 2022-09-10T05:57:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:14 smithi110 ceph-mon[97633]: osdmap e261: 12 total, 12 up, 12 in 2022-09-10T05:57:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:14 smithi110 ceph-mon[97633]: pgmap v2823: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:57:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:14 smithi104 ceph-mon[97044]: osdmap e261: 12 total, 12 up, 12 in 2022-09-10T05:57:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:14 smithi104 ceph-mon[97044]: pgmap v2823: 129 pgs: 129 active+clean; 7.9 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:57:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:14.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:14.895402+0000) 2022-09-10T05:57:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:15.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:15.302595+0000) 2022-09-10T05:57:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:15.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:15.895636+0000) 2022-09-10T05:57:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:16.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:16.302854+0000) 2022-09-10T05:57:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:16 smithi104 ceph-mon[97044]: pgmap v2824: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:57:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:16 smithi110 ceph-mon[97633]: pgmap v2824: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:57:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:16.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:16.895816+0000) 2022-09-10T05:57:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:17.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:17.303163+0000) 2022-09-10T05:57:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:17.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:17.896004+0000) 2022-09-10T05:57:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:18 smithi104 ceph-mon[97044]: pgmap v2825: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:57:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:18.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:18.303377+0000) 2022-09-10T05:57:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:18 smithi110 ceph-mon[97633]: pgmap v2825: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T05:57:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:18.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:18.896203+0000) 2022-09-10T05:57:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:19.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:19.303552+0000) 2022-09-10T05:57:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:19.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:19.896416+0000) 2022-09-10T05:57:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:20 smithi104 ceph-mon[97044]: pgmap v2826: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:57:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:20.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:20.303758+0000) 2022-09-10T05:57:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:20 smithi110 ceph-mon[97633]: pgmap v2826: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:57:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:20.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:20.896638+0000) 2022-09-10T05:57:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:21.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:21.303944+0000) 2022-09-10T05:57:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:21.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:21.896864+0000) 2022-09-10T05:57:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:22 smithi104 ceph-mon[97044]: pgmap v2827: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:57:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:22.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:22.304080+0000) 2022-09-10T05:57:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:22 smithi110 ceph-mon[97633]: pgmap v2827: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:57:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:22.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:22.897040+0000) 2022-09-10T05:57:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:23.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:23.304224+0000) 2022-09-10T05:57:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:23.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:23.897237+0000) 2022-09-10T05:57:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:24 smithi110 ceph-mon[97633]: pgmap v2828: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:57:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:24 smithi104 ceph-mon[97044]: pgmap v2828: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:57:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:24.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:24.304367+0000) 2022-09-10T05:57:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:24.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:24.897446+0000) 2022-09-10T05:57:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:25.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:25.304533+0000) 2022-09-10T05:57:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:25.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:25.897647+0000) 2022-09-10T05:57:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:26 smithi104 ceph-mon[97044]: pgmap v2829: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:57:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:26.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:26.304728+0000) 2022-09-10T05:57:26.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:26 smithi110 ceph-mon[97633]: pgmap v2829: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:57:27.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:26.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:26.897870+0000) 2022-09-10T05:57:27.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:57:27.176+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:57:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:27.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:27.304911+0000) 2022-09-10T05:57:28.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:27.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:27.898144+0000) 2022-09-10T05:57:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:28 smithi104 ceph-mon[97044]: pgmap v2830: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:28.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:28.305081+0000) 2022-09-10T05:57:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:28 smithi110 ceph-mon[97633]: pgmap v2830: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:28.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:28.898322+0000) 2022-09-10T05:57:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:29.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:29.305312+0000) 2022-09-10T05:57:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:29.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:29.898536+0000) 2022-09-10T05:57:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:30 smithi104 ceph-mon[97044]: pgmap v2831: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:30.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:30.305488+0000) 2022-09-10T05:57:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:30 smithi110 ceph-mon[97633]: pgmap v2831: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:30.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:30.898795+0000) 2022-09-10T05:57:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:31.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:31.305607+0000) 2022-09-10T05:57:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:31.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:31.899057+0000) 2022-09-10T05:57:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:32 smithi104 ceph-mon[97044]: pgmap v2832: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:32.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:32.305733+0000) 2022-09-10T05:57:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:32 smithi110 ceph-mon[97633]: pgmap v2832: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:32.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:32.899301+0000) 2022-09-10T05:57:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:33.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:33.305857+0000) 2022-09-10T05:57:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:33.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:33.899531+0000) 2022-09-10T05:57:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:34 smithi110 ceph-mon[97633]: pgmap v2833: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:34 smithi104 ceph-mon[97044]: pgmap v2833: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:34.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:34.306001+0000) 2022-09-10T05:57:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:34.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:34.899764+0000) 2022-09-10T05:57:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:35.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:35.306161+0000) 2022-09-10T05:57:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:35.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:35.899963+0000) 2022-09-10T05:57:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:36 smithi104 ceph-mon[97044]: pgmap v2834: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:57:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:36.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:36.306358+0000) 2022-09-10T05:57:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:36 smithi110 ceph-mon[97633]: pgmap v2834: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:57:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:36.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:36.900210+0000) 2022-09-10T05:57:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:37.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:37.306496+0000) 2022-09-10T05:57:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:37.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:37.900469+0000) 2022-09-10T05:57:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:38 smithi104 ceph-mon[97044]: pgmap v2835: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:38.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:38.306695+0000) 2022-09-10T05:57:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:38 smithi110 ceph-mon[97633]: pgmap v2835: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:38.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:38.900635+0000) 2022-09-10T05:57:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:39.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:39.306948+0000) 2022-09-10T05:57:40.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:39.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:39.900935+0000) 2022-09-10T05:57:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:40 smithi104 ceph-mon[97044]: pgmap v2836: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:40.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:40.307156+0000) 2022-09-10T05:57:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:40 smithi110 ceph-mon[97633]: pgmap v2836: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:40.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:40.901178+0000) 2022-09-10T05:57:41.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:41.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:41.307314+0000) 2022-09-10T05:57:42.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:41.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:41.901340+0000) 2022-09-10T05:57:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:42.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:42.307473+0000) 2022-09-10T05:57:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:57:42.187+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:57:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:42 smithi104 ceph-mon[97044]: pgmap v2837: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:42 smithi110 ceph-mon[97633]: pgmap v2837: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:43.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:42.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:42.901590+0000) 2022-09-10T05:57:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:43.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:43.307667+0000) 2022-09-10T05:57:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:43.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:43.901777+0000) 2022-09-10T05:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:44 smithi110 ceph-mon[97633]: pgmap v2838: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:57:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:57:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:57:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:44 smithi104 ceph-mon[97044]: pgmap v2838: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:57:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:57:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:57:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:57:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:44.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:44.307923+0000) 2022-09-10T05:57:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:44.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:44.902017+0000) 2022-09-10T05:57:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:45.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:45.308179+0000) 2022-09-10T05:57:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:45.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:45.902198+0000) 2022-09-10T05:57:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:46 smithi104 ceph-mon[97044]: pgmap v2839: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:57:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:46.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:46.308352+0000) 2022-09-10T05:57:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:46 smithi110 ceph-mon[97633]: pgmap v2839: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:57:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:46.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:46.902423+0000) 2022-09-10T05:57:47.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:47.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:47.308499+0000) 2022-09-10T05:57:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:47.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:47.902675+0000) 2022-09-10T05:57:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:48 smithi104 ceph-mon[97044]: pgmap v2840: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:48.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:48.308704+0000) 2022-09-10T05:57:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:48 smithi110 ceph-mon[97633]: pgmap v2840: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:49.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:48.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:48.902913+0000) 2022-09-10T05:57:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:49.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:49.308838+0000) 2022-09-10T05:57:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:49.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:49.903206+0000) 2022-09-10T05:57:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:50 smithi104 ceph-mon[97044]: pgmap v2841: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:50.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:50.309026+0000) 2022-09-10T05:57:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:50 smithi110 ceph-mon[97633]: pgmap v2841: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:57:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:50.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:50.903392+0000) 2022-09-10T05:57:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:51.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:51.309216+0000) 2022-09-10T05:57:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:51.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:51.903618+0000) 2022-09-10T05:57:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:52 smithi104 ceph-mon[97044]: pgmap v2842: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:52.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:52.309347+0000) 2022-09-10T05:57:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:52 smithi110 ceph-mon[97633]: pgmap v2842: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:57:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:52.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:52.903860+0000) 2022-09-10T05:57:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:53.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:53.309513+0000) 2022-09-10T05:57:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:53.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:53.903992+0000) 2022-09-10T05:57:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:54 smithi110 ceph-mon[97633]: pgmap v2843: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:54 smithi104 ceph-mon[97044]: pgmap v2843: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:54.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:54.309688+0000) 2022-09-10T05:57:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:54.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:54.904241+0000) 2022-09-10T05:57:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:55.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:55.309877+0000) 2022-09-10T05:57:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:55.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:55.904408+0000) 2022-09-10T05:57:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:56 smithi104 ceph-mon[97044]: pgmap v2844: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:57:56.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:56.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:56.310037+0000) 2022-09-10T05:57:56.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:56 smithi110 ceph-mon[97633]: pgmap v2844: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:57:57.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:56.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:56.904581+0000) 2022-09-10T05:57:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:57.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:57.310203+0000) 2022-09-10T05:57:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:57:57.189+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:57:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:57.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:57.904794+0000) 2022-09-10T05:57:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:57:58 smithi104 ceph-mon[97044]: pgmap v2845: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:58.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:58.310385+0000) 2022-09-10T05:57:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:57:58 smithi110 ceph-mon[97633]: pgmap v2845: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:57:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:58.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:58.905021+0000) 2022-09-10T05:57:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:57:59.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:59.310584+0000) 2022-09-10T05:58:00.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:57:59.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:57:59.905282+0000) 2022-09-10T05:58:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:00 smithi104 ceph-mon[97044]: pgmap v2846: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:58:00.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:00.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:00.310741+0000) 2022-09-10T05:58:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:00 smithi110 ceph-mon[97633]: pgmap v2846: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:58:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:00.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:00.905541+0000) 2022-09-10T05:58:01.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:01.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:01.310927+0000) 2022-09-10T05:58:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:01.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:01.905776+0000) 2022-09-10T05:58:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:02 smithi104 ceph-mon[97044]: pgmap v2847: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:02.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:02.311134+0000) 2022-09-10T05:58:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:02 smithi110 ceph-mon[97633]: pgmap v2847: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:02.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:02.905938+0000) 2022-09-10T05:58:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:03.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:03.311290+0000) 2022-09-10T05:58:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:03.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:03.906182+0000) 2022-09-10T05:58:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:04 smithi110 ceph-mon[97633]: pgmap v2848: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:04 smithi104 ceph-mon[97044]: pgmap v2848: 129 pgs: 129 active+clean; 7.9 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:04.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:04.311468+0000) 2022-09-10T05:58:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:04.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:04.906362+0000) 2022-09-10T05:58:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:05.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:05.311638+0000) 2022-09-10T05:58:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:05.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:05.906595+0000) 2022-09-10T05:58:06.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:06 smithi104 ceph-mon[97044]: pgmap v2849: 129 pgs: 129 active+clean; 8.0 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:58:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:06.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:06.311899+0000) 2022-09-10T05:58:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:06 smithi110 ceph-mon[97633]: pgmap v2849: 129 pgs: 129 active+clean; 8.0 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:58:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:06.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:06.906827+0000) 2022-09-10T05:58:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:07.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:07.312190+0000) 2022-09-10T05:58:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:07.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:07.907031+0000) 2022-09-10T05:58:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:08 smithi104 ceph-mon[97044]: pgmap v2850: 129 pgs: 129 active+clean; 8.0 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:58:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:08 smithi104 ceph-mon[97044]: osdmap e262: 12 total, 12 up, 12 in 2022-09-10T05:58:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:08.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:08.312436+0000) 2022-09-10T05:58:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:08 smithi110 ceph-mon[97633]: pgmap v2850: 129 pgs: 129 active+clean; 8.0 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:58:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:08 smithi110 ceph-mon[97633]: osdmap e262: 12 total, 12 up, 12 in 2022-09-10T05:58:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:08.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:08.907218+0000) 2022-09-10T05:58:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:09.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:09.312607+0000) 2022-09-10T05:58:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:09.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:09.907489+0000) 2022-09-10T05:58:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:10 smithi104 ceph-mon[97044]: pgmap v2852: 129 pgs: 129 active+clean; 8.0 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:58:10.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:10.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:10.312804+0000) 2022-09-10T05:58:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:10 smithi110 ceph-mon[97633]: pgmap v2852: 129 pgs: 129 active+clean; 8.0 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:58:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:10.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:10.907718+0000) 2022-09-10T05:58:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:11.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:11.313040+0000) 2022-09-10T05:58:12.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:11.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:11.907973+0000) 2022-09-10T05:58:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:58:12.203+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:58:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:12.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:12.313200+0000) 2022-09-10T05:58:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:12 smithi104 ceph-mon[97044]: pgmap v2853: 129 pgs: 129 active+clean; 8.0 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:58:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:12 smithi110 ceph-mon[97633]: pgmap v2853: 129 pgs: 129 active+clean; 8.0 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:58:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:12.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:12.908224+0000) 2022-09-10T05:58:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:13.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:13.313405+0000) 2022-09-10T05:58:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:13.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:13.908452+0000) 2022-09-10T05:58:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:14 smithi110 ceph-mon[97633]: osdmap e263: 12 total, 12 up, 12 in 2022-09-10T05:58:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:14 smithi110 ceph-mon[97633]: pgmap v2855: 129 pgs: 129 active+clean; 8.0 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:58:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:14 smithi104 ceph-mon[97044]: osdmap e263: 12 total, 12 up, 12 in 2022-09-10T05:58:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:14 smithi104 ceph-mon[97044]: pgmap v2855: 129 pgs: 129 active+clean; 8.0 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:58:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:14.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:14.313661+0000) 2022-09-10T05:58:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:14.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:14.908666+0000) 2022-09-10T05:58:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:15.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:15.313902+0000) 2022-09-10T05:58:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:15.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:15.908905+0000) 2022-09-10T05:58:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:16 smithi104 ceph-mon[97044]: pgmap v2856: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:58:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:16.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:16.314197+0000) 2022-09-10T05:58:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:16 smithi110 ceph-mon[97633]: pgmap v2856: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:58:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:16.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:16.909186+0000) 2022-09-10T05:58:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:17.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:17.314427+0000) 2022-09-10T05:58:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:17.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:17.909415+0000) 2022-09-10T05:58:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:18 smithi104 ceph-mon[97044]: pgmap v2857: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:58:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:18.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:18.314615+0000) 2022-09-10T05:58:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:18 smithi110 ceph-mon[97633]: pgmap v2857: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:58:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:18.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:18.909638+0000) 2022-09-10T05:58:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:19.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:19.314858+0000) 2022-09-10T05:58:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:19.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:19.909915+0000) 2022-09-10T05:58:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:20 smithi104 ceph-mon[97044]: pgmap v2858: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:58:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:20.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:20.315056+0000) 2022-09-10T05:58:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:20 smithi110 ceph-mon[97633]: pgmap v2858: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T05:58:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:20.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:20.910112+0000) 2022-09-10T05:58:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:21.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:21.315241+0000) 2022-09-10T05:58:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:21.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:21.910215+0000) 2022-09-10T05:58:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:22 smithi104 ceph-mon[97044]: pgmap v2859: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:58:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:22.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:22.315510+0000) 2022-09-10T05:58:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:22 smithi110 ceph-mon[97633]: pgmap v2859: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:58:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:22.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:22.910419+0000) 2022-09-10T05:58:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:23.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:23.315707+0000) 2022-09-10T05:58:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:23.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:23.910628+0000) 2022-09-10T05:58:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:24 smithi110 ceph-mon[97633]: pgmap v2860: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:58:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:24 smithi104 ceph-mon[97044]: pgmap v2860: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:58:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:24.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:24.315975+0000) 2022-09-10T05:58:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:24.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:24.910868+0000) 2022-09-10T05:58:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:25.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:25.316245+0000) 2022-09-10T05:58:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:25.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:25.911084+0000) 2022-09-10T05:58:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:26 smithi104 ceph-mon[97044]: pgmap v2861: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:58:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:26.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:26.316477+0000) 2022-09-10T05:58:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:26 smithi110 ceph-mon[97633]: pgmap v2861: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:58:27.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:26.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:26.911375+0000) 2022-09-10T05:58:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:27.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:27.316701+0000) 2022-09-10T05:58:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:58:27.217+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:58:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:27.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:27.911601+0000) 2022-09-10T05:58:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:28 smithi104 ceph-mon[97044]: pgmap v2862: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:28.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:28.316966+0000) 2022-09-10T05:58:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:28 smithi110 ceph-mon[97633]: pgmap v2862: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:28.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:28.911823+0000) 2022-09-10T05:58:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:29.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:29.317238+0000) 2022-09-10T05:58:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:29.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:29.912062+0000) 2022-09-10T05:58:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:30 smithi104 ceph-mon[97044]: pgmap v2863: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:58:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:30.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:30.317510+0000) 2022-09-10T05:58:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:30 smithi110 ceph-mon[97633]: pgmap v2863: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:58:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:30.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:30.912261+0000) 2022-09-10T05:58:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:31.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:31.317692+0000) 2022-09-10T05:58:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:31.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:31.912506+0000) 2022-09-10T05:58:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:32 smithi104 ceph-mon[97044]: pgmap v2864: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:32.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:32.317964+0000) 2022-09-10T05:58:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:32 smithi110 ceph-mon[97633]: pgmap v2864: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:32.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:32.912709+0000) 2022-09-10T05:58:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:33.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:33.318178+0000) 2022-09-10T05:58:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:33.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:33.912959+0000) 2022-09-10T05:58:34.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:34 smithi110 ceph-mon[97633]: pgmap v2865: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:34 smithi104 ceph-mon[97044]: pgmap v2865: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:34.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:34.318394+0000) 2022-09-10T05:58:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:34.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:34.913161+0000) 2022-09-10T05:58:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:35.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:35.318554+0000) 2022-09-10T05:58:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:35.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:35.913332+0000) 2022-09-10T05:58:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:36 smithi104 ceph-mon[97044]: pgmap v2866: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:36.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:36.318720+0000) 2022-09-10T05:58:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:36 smithi110 ceph-mon[97633]: pgmap v2866: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:36.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:36.913585+0000) 2022-09-10T05:58:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:37.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:37.318948+0000) 2022-09-10T05:58:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:37.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:37.913832+0000) 2022-09-10T05:58:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:38 smithi104 ceph-mon[97044]: pgmap v2867: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:38.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:38.319231+0000) 2022-09-10T05:58:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:38 smithi110 ceph-mon[97633]: pgmap v2867: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:38.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:38.914039+0000) 2022-09-10T05:58:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:39.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:39.319468+0000) 2022-09-10T05:58:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:39.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:39.914279+0000) 2022-09-10T05:58:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:40 smithi104 ceph-mon[97044]: pgmap v2868: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:58:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:40.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:40.319661+0000) 2022-09-10T05:58:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:40 smithi110 ceph-mon[97633]: pgmap v2868: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2022-09-10T05:58:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:40.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:40.914461+0000) 2022-09-10T05:58:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:41.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:41.319901+0000) 2022-09-10T05:58:42.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:41.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:41.914676+0000) 2022-09-10T05:58:42.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:58:42.230+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:58:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:42.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:42.320190+0000) 2022-09-10T05:58:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:42 smithi104 ceph-mon[97044]: pgmap v2869: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:42 smithi110 ceph-mon[97633]: pgmap v2869: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:42.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:42.914906+0000) 2022-09-10T05:58:43.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:43.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:43.320402+0000) 2022-09-10T05:58:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:43.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:43.915174+0000) 2022-09-10T05:58:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:44 smithi110 ceph-mon[97633]: pgmap v2870: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:58:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:58:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:58:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:58:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:44 smithi104 ceph-mon[97044]: pgmap v2870: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:58:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:58:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:58:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:58:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:44.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:44.320669+0000) 2022-09-10T05:58:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:44.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:44.915350+0000) 2022-09-10T05:58:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:45.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:45.320930+0000) 2022-09-10T05:58:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:45.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:45.915578+0000) 2022-09-10T05:58:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:46 smithi104 ceph-mon[97044]: pgmap v2871: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:46.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:46.321244+0000) 2022-09-10T05:58:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:46 smithi110 ceph-mon[97633]: pgmap v2871: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:46.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:46.915816+0000) 2022-09-10T05:58:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:47.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:47.321494+0000) 2022-09-10T05:58:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:47.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:47.916014+0000) 2022-09-10T05:58:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:48 smithi104 ceph-mon[97044]: pgmap v2872: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:48.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:48.321763+0000) 2022-09-10T05:58:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:48 smithi110 ceph-mon[97633]: pgmap v2872: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:48.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:48.916257+0000) 2022-09-10T05:58:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:49.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:49.321972+0000) 2022-09-10T05:58:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:49.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:49.916411+0000) 2022-09-10T05:58:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:50 smithi104 ceph-mon[97044]: pgmap v2873: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:50.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:50.322233+0000) 2022-09-10T05:58:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:50 smithi110 ceph-mon[97633]: pgmap v2873: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:50.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:50.916616+0000) 2022-09-10T05:58:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:51.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:51.322460+0000) 2022-09-10T05:58:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:51.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:51.916788+0000) 2022-09-10T05:58:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:52 smithi104 ceph-mon[97044]: pgmap v2874: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:52.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:52.322717+0000) 2022-09-10T05:58:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:52 smithi110 ceph-mon[97633]: pgmap v2874: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T05:58:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:52.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:52.916969+0000) 2022-09-10T05:58:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:53.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:53.322929+0000) 2022-09-10T05:58:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:53.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:53.917136+0000) 2022-09-10T05:58:54.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:54 smithi110 ceph-mon[97633]: pgmap v2875: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:54 smithi104 ceph-mon[97044]: pgmap v2875: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:54.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:54.323215+0000) 2022-09-10T05:58:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:54.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:54.917306+0000) 2022-09-10T05:58:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:55.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:55.323404+0000) 2022-09-10T05:58:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:55.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:55.917540+0000) 2022-09-10T05:58:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:56 smithi104 ceph-mon[97044]: pgmap v2876: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:56.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:56.323642+0000) 2022-09-10T05:58:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:56 smithi110 ceph-mon[97633]: pgmap v2876: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:58:57.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:56.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:56.917762+0000) 2022-09-10T05:58:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:58:57.242+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:58:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:57.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:57.323872+0000) 2022-09-10T05:58:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:57.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:57.917983+0000) 2022-09-10T05:58:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:58:58 smithi104 ceph-mon[97044]: pgmap v2877: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:58.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:58.324091+0000) 2022-09-10T05:58:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:58:58 smithi110 ceph-mon[97633]: pgmap v2877: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:58:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:58.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:58.918213+0000) 2022-09-10T05:58:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:58:59.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:59.324348+0000) 2022-09-10T05:59:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:58:59.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:58:59.918466+0000) 2022-09-10T05:59:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:00 smithi104 ceph-mon[97044]: pgmap v2878: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:00.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:00.324569+0000) 2022-09-10T05:59:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:00 smithi110 ceph-mon[97633]: pgmap v2878: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:00.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:00.918644+0000) 2022-09-10T05:59:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:01.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:01.324738+0000) 2022-09-10T05:59:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:01.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:01.918847+0000) 2022-09-10T05:59:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:02 smithi104 ceph-mon[97044]: pgmap v2879: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:02.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:02.324986+0000) 2022-09-10T05:59:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:02 smithi110 ceph-mon[97633]: pgmap v2879: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:02.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:02.919069+0000) 2022-09-10T05:59:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:03.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:03.325228+0000) 2022-09-10T05:59:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:03.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:03.919297+0000) 2022-09-10T05:59:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:04 smithi110 ceph-mon[97633]: pgmap v2880: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:04.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:04 smithi104 ceph-mon[97044]: pgmap v2880: 129 pgs: 129 active+clean; 8.0 MiB data, 467 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:04.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:04.325418+0000) 2022-09-10T05:59:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:04.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:04.919458+0000) 2022-09-10T05:59:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:05.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:05.325634+0000) 2022-09-10T05:59:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:05.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:05.919712+0000) 2022-09-10T05:59:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:06 smithi104 ceph-mon[97044]: pgmap v2881: 129 pgs: 129 active+clean; 8.0 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:59:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:06.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:06.325859+0000) 2022-09-10T05:59:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:06 smithi110 ceph-mon[97633]: pgmap v2881: 129 pgs: 129 active+clean; 8.0 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T05:59:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:06.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:06.919935+0000) 2022-09-10T05:59:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:07.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:07.326117+0000) 2022-09-10T05:59:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:07.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:07.920219+0000) 2022-09-10T05:59:08.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:08.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:08.326365+0000) 2022-09-10T05:59:08.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:08 smithi104 ceph-mon[97044]: pgmap v2882: 129 pgs: 129 active+clean; 8.0 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:59:08.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:08 smithi104 ceph-mon[97044]: osdmap e264: 12 total, 12 up, 12 in 2022-09-10T05:59:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:08 smithi110 ceph-mon[97633]: pgmap v2882: 129 pgs: 129 active+clean; 8.0 MiB data, 468 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T05:59:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:08 smithi110 ceph-mon[97633]: osdmap e264: 12 total, 12 up, 12 in 2022-09-10T05:59:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:08.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:08.920446+0000) 2022-09-10T05:59:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:09.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:09.326537+0000) 2022-09-10T05:59:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:09.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:09.920634+0000) 2022-09-10T05:59:10.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:10 smithi104 ceph-mon[97044]: pgmap v2884: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:59:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:10.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:10.326749+0000) 2022-09-10T05:59:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:10 smithi110 ceph-mon[97633]: pgmap v2884: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:59:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:10.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:10.920816+0000) 2022-09-10T05:59:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:11.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:11.326951+0000) 2022-09-10T05:59:12.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:11.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:11.921079+0000) 2022-09-10T05:59:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:12.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:12.327198+0000) 2022-09-10T05:59:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:59:12.245+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:59:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:12 smithi104 ceph-mon[97044]: pgmap v2885: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:59:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:12 smithi110 ceph-mon[97633]: pgmap v2885: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T05:59:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:12.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:12.921352+0000) 2022-09-10T05:59:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:13.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:13.327417+0000) 2022-09-10T05:59:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:13.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:13.921605+0000) 2022-09-10T05:59:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:14 smithi110 ceph-mon[97633]: osdmap e265: 12 total, 12 up, 12 in 2022-09-10T05:59:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:14 smithi110 ceph-mon[97633]: pgmap v2887: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:59:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:14 smithi104 ceph-mon[97044]: osdmap e265: 12 total, 12 up, 12 in 2022-09-10T05:59:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:14 smithi104 ceph-mon[97044]: pgmap v2887: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T05:59:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:14.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:14.327611+0000) 2022-09-10T05:59:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:14.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:14.921818+0000) 2022-09-10T05:59:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:15.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:15.327776+0000) 2022-09-10T05:59:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:15.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:15.922050+0000) 2022-09-10T05:59:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:16 smithi104 ceph-mon[97044]: pgmap v2888: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:59:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:16.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:16.327977+0000) 2022-09-10T05:59:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:16 smithi110 ceph-mon[97633]: pgmap v2888: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T05:59:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:16.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:16.922251+0000) 2022-09-10T05:59:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:17.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:17.328272+0000) 2022-09-10T05:59:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:17.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:17.922482+0000) 2022-09-10T05:59:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:18 smithi104 ceph-mon[97044]: pgmap v2889: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:59:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:18.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:18.328443+0000) 2022-09-10T05:59:18.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:18 smithi110 ceph-mon[97633]: pgmap v2889: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T05:59:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:18.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:18.922739+0000) 2022-09-10T05:59:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:19.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:19.328610+0000) 2022-09-10T05:59:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:19.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:19.922962+0000) 2022-09-10T05:59:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:20 smithi104 ceph-mon[97044]: pgmap v2890: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:59:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:20.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:20.328860+0000) 2022-09-10T05:59:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:20 smithi110 ceph-mon[97633]: pgmap v2890: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:59:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:20.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:20.923204+0000) 2022-09-10T05:59:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:21.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:21.329078+0000) 2022-09-10T05:59:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:21.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:21.923424+0000) 2022-09-10T05:59:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:22 smithi104 ceph-mon[97044]: pgmap v2891: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:59:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:22.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:22.329346+0000) 2022-09-10T05:59:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:22 smithi110 ceph-mon[97633]: pgmap v2891: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T05:59:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:22.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:22.923642+0000) 2022-09-10T05:59:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:23.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:23.329565+0000) 2022-09-10T05:59:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:23.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:23.923855+0000) 2022-09-10T05:59:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:24 smithi110 ceph-mon[97633]: pgmap v2892: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:59:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:24 smithi104 ceph-mon[97044]: pgmap v2892: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T05:59:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:24.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:24.329803+0000) 2022-09-10T05:59:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:24.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:24.924081+0000) 2022-09-10T05:59:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:25.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:25.330041+0000) 2022-09-10T05:59:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:25.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:25.924314+0000) 2022-09-10T05:59:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:26 smithi104 ceph-mon[97044]: pgmap v2893: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:59:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:26.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:26.330337+0000) 2022-09-10T05:59:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:26 smithi110 ceph-mon[97633]: pgmap v2893: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T05:59:27.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:26.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:26.924513+0000) 2022-09-10T05:59:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:27.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:27.330559+0000) 2022-09-10T05:59:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:59:27.259+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:59:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:27.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:27.924754+0000) 2022-09-10T05:59:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:28 smithi104 ceph-mon[97044]: pgmap v2894: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:28.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:28.330701+0000) 2022-09-10T05:59:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:28 smithi110 ceph-mon[97633]: pgmap v2894: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:28.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:28.925025+0000) 2022-09-10T05:59:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:29.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:29.330948+0000) 2022-09-10T05:59:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:29.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:29.925271+0000) 2022-09-10T05:59:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:30 smithi104 ceph-mon[97044]: pgmap v2895: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:30.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:30.331186+0000) 2022-09-10T05:59:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:30 smithi110 ceph-mon[97633]: pgmap v2895: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:30.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:30.925461+0000) 2022-09-10T05:59:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:31.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:31.331431+0000) 2022-09-10T05:59:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:31.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:31.925681+0000) 2022-09-10T05:59:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:32 smithi104 ceph-mon[97044]: pgmap v2896: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:32.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:32.331679+0000) 2022-09-10T05:59:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:32 smithi110 ceph-mon[97633]: pgmap v2896: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:32.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:32.925897+0000) 2022-09-10T05:59:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:33.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:33.331919+0000) 2022-09-10T05:59:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:33.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:33.926180+0000) 2022-09-10T05:59:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:34 smithi110 ceph-mon[97633]: pgmap v2897: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:34 smithi104 ceph-mon[97044]: pgmap v2897: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:34.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:34.332146+0000) 2022-09-10T05:59:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:34.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:34.926402+0000) 2022-09-10T05:59:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:35.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:35.332308+0000) 2022-09-10T05:59:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:35.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:35.926629+0000) 2022-09-10T05:59:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:36 smithi104 ceph-mon[97044]: pgmap v2898: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:36.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:36.332605+0000) 2022-09-10T05:59:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:36 smithi110 ceph-mon[97633]: pgmap v2898: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:36.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:36.926874+0000) 2022-09-10T05:59:37.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:37.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:37.332806+0000) 2022-09-10T05:59:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:37.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:37.927176+0000) 2022-09-10T05:59:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:38 smithi104 ceph-mon[97044]: pgmap v2899: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:38.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:38.333049+0000) 2022-09-10T05:59:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:38 smithi110 ceph-mon[97633]: pgmap v2899: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:38.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:38.927457+0000) 2022-09-10T05:59:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:39.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:39.333350+0000) 2022-09-10T05:59:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:39.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:39.927686+0000) 2022-09-10T05:59:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:40 smithi104 ceph-mon[97044]: pgmap v2900: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:40.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:40.333566+0000) 2022-09-10T05:59:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:40 smithi110 ceph-mon[97633]: pgmap v2900: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:40.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:40.927953+0000) 2022-09-10T05:59:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:41.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:41.333750+0000) 2022-09-10T05:59:42.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:41.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:41.928212+0000) 2022-09-10T05:59:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:59:42.273+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:59:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:42.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:42.334022+0000) 2022-09-10T05:59:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:42 smithi104 ceph-mon[97044]: pgmap v2901: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:42 smithi110 ceph-mon[97633]: pgmap v2901: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:42.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:42.928454+0000) 2022-09-10T05:59:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:43.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:43.334237+0000) 2022-09-10T05:59:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:43.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:43.928685+0000) 2022-09-10T05:59:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:44 smithi110 ceph-mon[97633]: pgmap v2902: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:59:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:59:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:59:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:59:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:44 smithi104 ceph-mon[97044]: pgmap v2902: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T05:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T05:59:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:44.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:44.334400+0000) 2022-09-10T05:59:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:44.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:44.928928+0000) 2022-09-10T05:59:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:45.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:45.334637+0000) 2022-09-10T05:59:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:45.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:45.929076+0000) 2022-09-10T05:59:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:46 smithi104 ceph-mon[97044]: pgmap v2903: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:46.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:46.334826+0000) 2022-09-10T05:59:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:46 smithi110 ceph-mon[97633]: pgmap v2903: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:46.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:46.929297+0000) 2022-09-10T05:59:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:47.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:47.334996+0000) 2022-09-10T05:59:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:47.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:47.929522+0000) 2022-09-10T05:59:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:48 smithi104 ceph-mon[97044]: pgmap v2904: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:48.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:48.335294+0000) 2022-09-10T05:59:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:48 smithi110 ceph-mon[97633]: pgmap v2904: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:48.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:48.929778+0000) 2022-09-10T05:59:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:49.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:49.335553+0000) 2022-09-10T05:59:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:49.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:49.930020+0000) 2022-09-10T05:59:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:50 smithi104 ceph-mon[97044]: pgmap v2905: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:50.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:50.335852+0000) 2022-09-10T05:59:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:50 smithi110 ceph-mon[97633]: pgmap v2905: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:50.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:50.930272+0000) 2022-09-10T05:59:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:51.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:51.336038+0000) 2022-09-10T05:59:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:51.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:51.930500+0000) 2022-09-10T05:59:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:52 smithi104 ceph-mon[97044]: pgmap v2906: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:52.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:52.336167+0000) 2022-09-10T05:59:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:52 smithi110 ceph-mon[97633]: pgmap v2906: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:52.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:52.930687+0000) 2022-09-10T05:59:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:53.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:53.336362+0000) 2022-09-10T05:59:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:53.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:53.930958+0000) 2022-09-10T05:59:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:54 smithi110 ceph-mon[97633]: pgmap v2907: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:54 smithi104 ceph-mon[97044]: pgmap v2907: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:54.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:54.336541+0000) 2022-09-10T05:59:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:54.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:54.931181+0000) 2022-09-10T05:59:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:55.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:55.336719+0000) 2022-09-10T05:59:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:55.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:55.931414+0000) 2022-09-10T05:59:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:56 smithi104 ceph-mon[97044]: pgmap v2908: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:56.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:56.336882+0000) 2022-09-10T05:59:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:56 smithi110 ceph-mon[97633]: pgmap v2908: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T05:59:57.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:56.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:56.931663+0000) 2022-09-10T05:59:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:57.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:57.337150+0000) 2022-09-10T05:59:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 05:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T05:59:57.287+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T05:59:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:57.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:57.931938+0000) 2022-09-10T05:59:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 05:59:58 smithi104 ceph-mon[97044]: pgmap v2909: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:58.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:58.337414+0000) 2022-09-10T05:59:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 05:59:58 smithi110 ceph-mon[97633]: pgmap v2909: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T05:59:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:58.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:58.932198+0000) 2022-09-10T05:59:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 05:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T05:59:59.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:59.337677+0000) 2022-09-10T06:00:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 05:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T05:59:59.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T04:59:59.932363+0000) 2022-09-10T06:00:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:00 smithi104 ceph-mon[97044]: pgmap v2910: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:00:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:00.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:00.337890+0000) 2022-09-10T06:00:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:00 smithi110 ceph-mon[97633]: pgmap v2910: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:00:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:00.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:00.932499+0000) 2022-09-10T06:00:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:01.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:01.338153+0000) 2022-09-10T06:00:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:01.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:01.932705+0000) 2022-09-10T06:00:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:02 smithi104 ceph-mon[97044]: pgmap v2911: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:02.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:02.338371+0000) 2022-09-10T06:00:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:02 smithi110 ceph-mon[97633]: pgmap v2911: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:02.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:02.932928+0000) 2022-09-10T06:00:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:03.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:03.338553+0000) 2022-09-10T06:00:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:03.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:03.933129+0000) 2022-09-10T06:00:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:04 smithi110 ceph-mon[97633]: pgmap v2912: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:04 smithi104 ceph-mon[97044]: pgmap v2912: 129 pgs: 129 active+clean; 8.0 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:04.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:04.338751+0000) 2022-09-10T06:00:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:04.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:04.933357+0000) 2022-09-10T06:00:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:05.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:05.339014+0000) 2022-09-10T06:00:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:05.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:05.933605+0000) 2022-09-10T06:00:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:06 smithi104 ceph-mon[97044]: pgmap v2913: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:00:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:06.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:06.339214+0000) 2022-09-10T06:00:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:06 smithi110 ceph-mon[97633]: pgmap v2913: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:00:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:06.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:06.933827+0000) 2022-09-10T06:00:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:07.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:07.339352+0000) 2022-09-10T06:00:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:07.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:07.934062+0000) 2022-09-10T06:00:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:08 smithi104 ceph-mon[97044]: pgmap v2914: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:00:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:08 smithi104 ceph-mon[97044]: osdmap e266: 12 total, 12 up, 12 in 2022-09-10T06:00:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:08.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:08.339590+0000) 2022-09-10T06:00:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:08 smithi110 ceph-mon[97633]: pgmap v2914: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:00:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:08 smithi110 ceph-mon[97633]: osdmap e266: 12 total, 12 up, 12 in 2022-09-10T06:00:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:08.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:08.934292+0000) 2022-09-10T06:00:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:09.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:09.339792+0000) 2022-09-10T06:00:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:09.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:09.934478+0000) 2022-09-10T06:00:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:10 smithi104 ceph-mon[97044]: pgmap v2916: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:00:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:10.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:10.339999+0000) 2022-09-10T06:00:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:10 smithi110 ceph-mon[97633]: pgmap v2916: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:00:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:10.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:10.934656+0000) 2022-09-10T06:00:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:11.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:11.340218+0000) 2022-09-10T06:00:12.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:11.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:11.934865+0000) 2022-09-10T06:00:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:12.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:12.340417+0000) 2022-09-10T06:00:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:00:12.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:00:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:12 smithi104 ceph-mon[97044]: pgmap v2917: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:00:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:12 smithi110 ceph-mon[97633]: pgmap v2917: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:00:13.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:12.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:12.935147+0000) 2022-09-10T06:00:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:13.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:13.340638+0000) 2022-09-10T06:00:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:13.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:13.935409+0000) 2022-09-10T06:00:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:14 smithi110 ceph-mon[97633]: osdmap e267: 12 total, 12 up, 12 in 2022-09-10T06:00:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:14 smithi110 ceph-mon[97633]: pgmap v2919: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:00:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:14 smithi104 ceph-mon[97044]: osdmap e267: 12 total, 12 up, 12 in 2022-09-10T06:00:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:14 smithi104 ceph-mon[97044]: pgmap v2919: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:00:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:14.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:14.340918+0000) 2022-09-10T06:00:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:14.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:14.935657+0000) 2022-09-10T06:00:15.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:15.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:15.341219+0000) 2022-09-10T06:00:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:15.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:15.935853+0000) 2022-09-10T06:00:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:16 smithi104 ceph-mon[97044]: pgmap v2920: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:00:16.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:16.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:16.341460+0000) 2022-09-10T06:00:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:16 smithi110 ceph-mon[97633]: pgmap v2920: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:00:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:16.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:16.936086+0000) 2022-09-10T06:00:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:17.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:17.341619+0000) 2022-09-10T06:00:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:17.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:17.936365+0000) 2022-09-10T06:00:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:18 smithi104 ceph-mon[97044]: pgmap v2921: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:00:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:18.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:18.341850+0000) 2022-09-10T06:00:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:18 smithi110 ceph-mon[97633]: pgmap v2921: 129 pgs: 129 active+clean; 8.0 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:00:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:18.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:18.936599+0000) 2022-09-10T06:00:19.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:19.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:19.342175+0000) 2022-09-10T06:00:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:19.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:19.936786+0000) 2022-09-10T06:00:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:20 smithi104 ceph-mon[97044]: pgmap v2922: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:00:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:20.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:20.342454+0000) 2022-09-10T06:00:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:20 smithi110 ceph-mon[97633]: pgmap v2922: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:00:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:20.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:20.937057+0000) 2022-09-10T06:00:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:21.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:21.342614+0000) 2022-09-10T06:00:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:21.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:21.937227+0000) 2022-09-10T06:00:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:22 smithi104 ceph-mon[97044]: pgmap v2923: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:00:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:22.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:22.342852+0000) 2022-09-10T06:00:22.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:22 smithi110 ceph-mon[97633]: pgmap v2923: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:00:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:22.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:22.937411+0000) 2022-09-10T06:00:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:23.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:23.343069+0000) 2022-09-10T06:00:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:23.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:23.937628+0000) 2022-09-10T06:00:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:24 smithi110 ceph-mon[97633]: pgmap v2924: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:00:24.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:24 smithi104 ceph-mon[97044]: pgmap v2924: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:00:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:24.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:24.343343+0000) 2022-09-10T06:00:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:24.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:24.937853+0000) 2022-09-10T06:00:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:25.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:25.343561+0000) 2022-09-10T06:00:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:25.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:25.938135+0000) 2022-09-10T06:00:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:26 smithi104 ceph-mon[97044]: pgmap v2925: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:00:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:26.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:26.343763+0000) 2022-09-10T06:00:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:26 smithi110 ceph-mon[97633]: pgmap v2925: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:00:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:26.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:26.938375+0000) 2022-09-10T06:00:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:27.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:27.343974+0000) 2022-09-10T06:00:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:00:27.316+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:00:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:27.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:27.938612+0000) 2022-09-10T06:00:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:28 smithi104 ceph-mon[97044]: pgmap v2926: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:28.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:28.344221+0000) 2022-09-10T06:00:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:28 smithi110 ceph-mon[97633]: pgmap v2926: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:28.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:28.938879+0000) 2022-09-10T06:00:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:29.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:29.344476+0000) 2022-09-10T06:00:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:29.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:29.939178+0000) 2022-09-10T06:00:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:30 smithi104 ceph-mon[97044]: pgmap v2927: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:30.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:30.344706+0000) 2022-09-10T06:00:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:30 smithi110 ceph-mon[97633]: pgmap v2927: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:30.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:30.939414+0000) 2022-09-10T06:00:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:31.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:31.344944+0000) 2022-09-10T06:00:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:31.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:31.939643+0000) 2022-09-10T06:00:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:32 smithi104 ceph-mon[97044]: pgmap v2928: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:32.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:32.345191+0000) 2022-09-10T06:00:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:32 smithi110 ceph-mon[97633]: pgmap v2928: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:32.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:32.939874+0000) 2022-09-10T06:00:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:33.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:33.345417+0000) 2022-09-10T06:00:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:33.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:33.940175+0000) 2022-09-10T06:00:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:34 smithi110 ceph-mon[97633]: pgmap v2929: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:34 smithi104 ceph-mon[97044]: pgmap v2929: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:34.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:34.345655+0000) 2022-09-10T06:00:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:34.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:34.940320+0000) 2022-09-10T06:00:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:35.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:35.345828+0000) 2022-09-10T06:00:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:35.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:35.940503+0000) 2022-09-10T06:00:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:36 smithi104 ceph-mon[97044]: pgmap v2930: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:36.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:36.346073+0000) 2022-09-10T06:00:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:36 smithi110 ceph-mon[97633]: pgmap v2930: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:36.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:36.940720+0000) 2022-09-10T06:00:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:37.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:37.346289+0000) 2022-09-10T06:00:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:37.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:37.940968+0000) 2022-09-10T06:00:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:38 smithi104 ceph-mon[97044]: pgmap v2931: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:38.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:38.346521+0000) 2022-09-10T06:00:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:38 smithi110 ceph-mon[97633]: pgmap v2931: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:38.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:38.941181+0000) 2022-09-10T06:00:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:39.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:39.346720+0000) 2022-09-10T06:00:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:39.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:39.941371+0000) 2022-09-10T06:00:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:40.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:40.347003+0000) 2022-09-10T06:00:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:40 smithi110 ceph-mon[97633]: pgmap v2932: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:40 smithi104 ceph-mon[97044]: pgmap v2932: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:40.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:40.941600+0000) 2022-09-10T06:00:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:41.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:41.347285+0000) 2022-09-10T06:00:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:41.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:41.941742+0000) 2022-09-10T06:00:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:42.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:42.347470+0000) 2022-09-10T06:00:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:00:42.332+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:00:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:42 smithi110 ceph-mon[97633]: pgmap v2933: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:42 smithi104 ceph-mon[97044]: pgmap v2933: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:42.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:42.941970+0000) 2022-09-10T06:00:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:43.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:43.347704+0000) 2022-09-10T06:00:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:43.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:43.942143+0000) 2022-09-10T06:00:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:44 smithi110 ceph-mon[97633]: pgmap v2934: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:00:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:00:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:00:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:00:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:44 smithi104 ceph-mon[97044]: pgmap v2934: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:00:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:44.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:44.348008+0000) 2022-09-10T06:00:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:44.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:44.942380+0000) 2022-09-10T06:00:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:45.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:45.348206+0000) 2022-09-10T06:00:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:45.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:45.942637+0000) 2022-09-10T06:00:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:46 smithi104 ceph-mon[97044]: pgmap v2935: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:46.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:46.348346+0000) 2022-09-10T06:00:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:46 smithi110 ceph-mon[97633]: pgmap v2935: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:46.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:46.942836+0000) 2022-09-10T06:00:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:47.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:47.348535+0000) 2022-09-10T06:00:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:47.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:47.943059+0000) 2022-09-10T06:00:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:48.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:48.348780+0000) 2022-09-10T06:00:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:48 smithi110 ceph-mon[97633]: pgmap v2936: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:48 smithi104 ceph-mon[97044]: pgmap v2936: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:00:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:48.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:48.943285+0000) 2022-09-10T06:00:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:49.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:49.349017+0000) 2022-09-10T06:00:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:49.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:49.943521+0000) 2022-09-10T06:00:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:50.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:50.349276+0000) 2022-09-10T06:00:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:50 smithi110 ceph-mon[97633]: pgmap v2937: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:50 smithi104 ceph-mon[97044]: pgmap v2937: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:50.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:50.943786+0000) 2022-09-10T06:00:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:51.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:51.349546+0000) 2022-09-10T06:00:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:51.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:51.944030+0000) 2022-09-10T06:00:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:52.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:52.349819+0000) 2022-09-10T06:00:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:52 smithi110 ceph-mon[97633]: pgmap v2938: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:00:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:52 smithi104 ceph-mon[97044]: pgmap v2938: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:00:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:52.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:52.944202+0000) 2022-09-10T06:00:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:53.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:53.350056+0000) 2022-09-10T06:00:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:53.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:53.944409+0000) 2022-09-10T06:00:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:54 smithi110 ceph-mon[97633]: pgmap v2939: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:00:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:54 smithi104 ceph-mon[97044]: pgmap v2939: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:00:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:54.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:54.350261+0000) 2022-09-10T06:00:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:54.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:54.944673+0000) 2022-09-10T06:00:55.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:55.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:55.350480+0000) 2022-09-10T06:00:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:55.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:55.944898+0000) 2022-09-10T06:00:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:56.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:56.350640+0000) 2022-09-10T06:00:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:56 smithi110 ceph-mon[97633]: pgmap v2940: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:56 smithi104 ceph-mon[97044]: pgmap v2940: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:00:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:56.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:56.945125+0000) 2022-09-10T06:00:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:00:57.339+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:00:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:57.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:57.350877+0000) 2022-09-10T06:00:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:57.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:57.945307+0000) 2022-09-10T06:00:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:58.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:58.351167+0000) 2022-09-10T06:00:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:00:58 smithi110 ceph-mon[97633]: pgmap v2941: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:00:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:00:58 smithi104 ceph-mon[97044]: pgmap v2941: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:00:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:58.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:58.945542+0000) 2022-09-10T06:00:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:00:59.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:59.351425+0000) 2022-09-10T06:01:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:00:59.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:00:59.945775+0000) 2022-09-10T06:01:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:00.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:00.351600+0000) 2022-09-10T06:01:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:00 smithi110 ceph-mon[97633]: pgmap v2942: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:00 smithi104 ceph-mon[97044]: pgmap v2942: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:00.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:00.945936+0000) 2022-09-10T06:01:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:01.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:01.351821+0000) 2022-09-10T06:01:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:01.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:01.946169+0000) 2022-09-10T06:01:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:02.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:02.352049+0000) 2022-09-10T06:01:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:02 smithi110 ceph-mon[97633]: pgmap v2943: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:02 smithi104 ceph-mon[97044]: pgmap v2943: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:03.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:02.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:02.946358+0000) 2022-09-10T06:01:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:03.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:03.352335+0000) 2022-09-10T06:01:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:03.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:03.946595+0000) 2022-09-10T06:01:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:04 smithi110 ceph-mon[97633]: pgmap v2944: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:04 smithi104 ceph-mon[97044]: pgmap v2944: 129 pgs: 129 active+clean; 8.0 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:04.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:04.352542+0000) 2022-09-10T06:01:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:04.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:04.946775+0000) 2022-09-10T06:01:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:05.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:05.352762+0000) 2022-09-10T06:01:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:05.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:05.946941+0000) 2022-09-10T06:01:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:06.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:06.353000+0000) 2022-09-10T06:01:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:06 smithi110 ceph-mon[97633]: pgmap v2945: 129 pgs: 129 active+clean; 8.0 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:01:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:06 smithi104 ceph-mon[97044]: pgmap v2945: 129 pgs: 129 active+clean; 8.0 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:01:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:06.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:06.947191+0000) 2022-09-10T06:01:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:07.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:07.353217+0000) 2022-09-10T06:01:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:07.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:07.947454+0000) 2022-09-10T06:01:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:08.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:08.353434+0000) 2022-09-10T06:01:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:08 smithi110 ceph-mon[97633]: pgmap v2946: 129 pgs: 129 active+clean; 8.0 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:01:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:08 smithi110 ceph-mon[97633]: osdmap e268: 12 total, 12 up, 12 in 2022-09-10T06:01:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:08 smithi104 ceph-mon[97044]: pgmap v2946: 129 pgs: 129 active+clean; 8.0 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:01:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:08 smithi104 ceph-mon[97044]: osdmap e268: 12 total, 12 up, 12 in 2022-09-10T06:01:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:08.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:08.947676+0000) 2022-09-10T06:01:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:09.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:09.353682+0000) 2022-09-10T06:01:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:09.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:09.947925+0000) 2022-09-10T06:01:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:10.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:10.353902+0000) 2022-09-10T06:01:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:10 smithi110 ceph-mon[97633]: pgmap v2948: 129 pgs: 129 active+clean; 8.0 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:01:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:10 smithi104 ceph-mon[97044]: pgmap v2948: 129 pgs: 129 active+clean; 8.0 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:01:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:10.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:10.948177+0000) 2022-09-10T06:01:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:11.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:11.354147+0000) 2022-09-10T06:01:12.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:11.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:11.948373+0000) 2022-09-10T06:01:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:12.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:12.354397+0000) 2022-09-10T06:01:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:01:12.353+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:01:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:12 smithi110 ceph-mon[97633]: pgmap v2949: 129 pgs: 129 active+clean; 8.0 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:01:13.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:12 smithi104 ceph-mon[97044]: pgmap v2949: 129 pgs: 129 active+clean; 8.0 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:01:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:12.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:12.948553+0000) 2022-09-10T06:01:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:13.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:13.354584+0000) 2022-09-10T06:01:14.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:13.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:13.948734+0000) 2022-09-10T06:01:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:14 smithi110 ceph-mon[97633]: osdmap e269: 12 total, 12 up, 12 in 2022-09-10T06:01:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:14 smithi110 ceph-mon[97633]: pgmap v2951: 129 pgs: 129 active+clean; 8.0 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:01:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:14 smithi104 ceph-mon[97044]: osdmap e269: 12 total, 12 up, 12 in 2022-09-10T06:01:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:14 smithi104 ceph-mon[97044]: pgmap v2951: 129 pgs: 129 active+clean; 8.0 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:01:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:14.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:14.354823+0000) 2022-09-10T06:01:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:14.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:14.948994+0000) 2022-09-10T06:01:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:15.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:15.355023+0000) 2022-09-10T06:01:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:15.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:15.949198+0000) 2022-09-10T06:01:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:16.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:16.355237+0000) 2022-09-10T06:01:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:16 smithi110 ceph-mon[97633]: pgmap v2952: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:01:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:16 smithi104 ceph-mon[97044]: pgmap v2952: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:01:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:16.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:16.949445+0000) 2022-09-10T06:01:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:17.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:17.355405+0000) 2022-09-10T06:01:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:17.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:17.949677+0000) 2022-09-10T06:01:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:18.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:18.355637+0000) 2022-09-10T06:01:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:18 smithi110 ceph-mon[97633]: pgmap v2953: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:01:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:18 smithi104 ceph-mon[97044]: pgmap v2953: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:01:19.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:18.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:18.949875+0000) 2022-09-10T06:01:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:19.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:19.355860+0000) 2022-09-10T06:01:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:19.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:19.950148+0000) 2022-09-10T06:01:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:20.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:20.356051+0000) 2022-09-10T06:01:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:20 smithi110 ceph-mon[97633]: pgmap v2954: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:01:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:20 smithi104 ceph-mon[97044]: pgmap v2954: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:01:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:20.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:20.950316+0000) 2022-09-10T06:01:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:21.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:21.356223+0000) 2022-09-10T06:01:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:21.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:21.950557+0000) 2022-09-10T06:01:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:22.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:22.356413+0000) 2022-09-10T06:01:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:22 smithi110 ceph-mon[97633]: pgmap v2955: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:01:23.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:22 smithi104 ceph-mon[97044]: pgmap v2955: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:01:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:22.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:22.950819+0000) 2022-09-10T06:01:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:23.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:23.356591+0000) 2022-09-10T06:01:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:23.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:23.951068+0000) 2022-09-10T06:01:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:24 smithi110 ceph-mon[97633]: pgmap v2956: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:01:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:24 smithi104 ceph-mon[97044]: pgmap v2956: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:01:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:24.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:24.356806+0000) 2022-09-10T06:01:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:24.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:24.951244+0000) 2022-09-10T06:01:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:25.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:25.356987+0000) 2022-09-10T06:01:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:25.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:25.951387+0000) 2022-09-10T06:01:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:26.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:26.357205+0000) 2022-09-10T06:01:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:26 smithi110 ceph-mon[97633]: pgmap v2957: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:01:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:26 smithi104 ceph-mon[97044]: pgmap v2957: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:01:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:26.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:26.951547+0000) 2022-09-10T06:01:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:27.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:27.357395+0000) 2022-09-10T06:01:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:01:27.369+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:01:28.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:27.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:27.951748+0000) 2022-09-10T06:01:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:28.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:28.357650+0000) 2022-09-10T06:01:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:28 smithi110 ceph-mon[97633]: pgmap v2958: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:28 smithi104 ceph-mon[97044]: pgmap v2958: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:28.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:28.952001+0000) 2022-09-10T06:01:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:29.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:29.357878+0000) 2022-09-10T06:01:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:29.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:29.952287+0000) 2022-09-10T06:01:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:30.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:30.358113+0000) 2022-09-10T06:01:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:30 smithi110 ceph-mon[97633]: pgmap v2959: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:30 smithi104 ceph-mon[97044]: pgmap v2959: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:30.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:30.952457+0000) 2022-09-10T06:01:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:31.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:31.358358+0000) 2022-09-10T06:01:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:31.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:31.952705+0000) 2022-09-10T06:01:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:32.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:32.358513+0000) 2022-09-10T06:01:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:32 smithi110 ceph-mon[97633]: pgmap v2960: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:33.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:32 smithi104 ceph-mon[97044]: pgmap v2960: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:32.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:32.952971+0000) 2022-09-10T06:01:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:33.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:33.358692+0000) 2022-09-10T06:01:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:33.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:33.953222+0000) 2022-09-10T06:01:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:34 smithi110 ceph-mon[97633]: pgmap v2961: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:34 smithi104 ceph-mon[97044]: pgmap v2961: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:34.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:34.358922+0000) 2022-09-10T06:01:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:34.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:34.953445+0000) 2022-09-10T06:01:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:35.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:35.359212+0000) 2022-09-10T06:01:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:35.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:35.953712+0000) 2022-09-10T06:01:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:36.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:36.359341+0000) 2022-09-10T06:01:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:36 smithi110 ceph-mon[97633]: pgmap v2962: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:36 smithi104 ceph-mon[97044]: pgmap v2962: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:36.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:36.953871+0000) 2022-09-10T06:01:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:37.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:37.359580+0000) 2022-09-10T06:01:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:37.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:37.954148+0000) 2022-09-10T06:01:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:38.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:38.359822+0000) 2022-09-10T06:01:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:38 smithi110 ceph-mon[97633]: pgmap v2963: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:38 smithi104 ceph-mon[97044]: pgmap v2963: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:38.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:38.954318+0000) 2022-09-10T06:01:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:39.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:39.360081+0000) 2022-09-10T06:01:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:39.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:39.954535+0000) 2022-09-10T06:01:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:40.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:40.360360+0000) 2022-09-10T06:01:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:40 smithi110 ceph-mon[97633]: pgmap v2964: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:40 smithi104 ceph-mon[97044]: pgmap v2964: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:40.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:40.954717+0000) 2022-09-10T06:01:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:41.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:41.360539+0000) 2022-09-10T06:01:42.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:41.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:41.954960+0000) 2022-09-10T06:01:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:01:42.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:01:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:42.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:42.360748+0000) 2022-09-10T06:01:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:42 smithi110 ceph-mon[97633]: pgmap v2965: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:42 smithi104 ceph-mon[97044]: pgmap v2965: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:43.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:42.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:42.955208+0000) 2022-09-10T06:01:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:43.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:43.360916+0000) 2022-09-10T06:01:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:43.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:43.955467+0000) 2022-09-10T06:01:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:44 smithi110 ceph-mon[97633]: pgmap v2966: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:01:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:01:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:01:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:01:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:44 smithi104 ceph-mon[97044]: pgmap v2966: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:01:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:01:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:01:44.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:01:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:44.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:44.361172+0000) 2022-09-10T06:01:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:44.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:44.955704+0000) 2022-09-10T06:01:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:45.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:45.361395+0000) 2022-09-10T06:01:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:45.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:45.955913+0000) 2022-09-10T06:01:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:46.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:46.361619+0000) 2022-09-10T06:01:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:46 smithi110 ceph-mon[97633]: pgmap v2967: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:46 smithi104 ceph-mon[97044]: pgmap v2967: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:46.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:46.956067+0000) 2022-09-10T06:01:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:47.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:47.361813+0000) 2022-09-10T06:01:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:47.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:47.956266+0000) 2022-09-10T06:01:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:48.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:48.362056+0000) 2022-09-10T06:01:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:48 smithi110 ceph-mon[97633]: pgmap v2968: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:48 smithi104 ceph-mon[97044]: pgmap v2968: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:48.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:48.956500+0000) 2022-09-10T06:01:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:49.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:49.362320+0000) 2022-09-10T06:01:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:49.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:49.956773+0000) 2022-09-10T06:01:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:50.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:50.362471+0000) 2022-09-10T06:01:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:50 smithi110 ceph-mon[97633]: pgmap v2969: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:50 smithi104 ceph-mon[97044]: pgmap v2969: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:50.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:50.957000+0000) 2022-09-10T06:01:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:51.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:51.362684+0000) 2022-09-10T06:01:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:51.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:51.957211+0000) 2022-09-10T06:01:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:52.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:52.362872+0000) 2022-09-10T06:01:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:52 smithi110 ceph-mon[97633]: pgmap v2970: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:52 smithi104 ceph-mon[97044]: pgmap v2970: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:52.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:52.957459+0000) 2022-09-10T06:01:53.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:53.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:53.363012+0000) 2022-09-10T06:01:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:53.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:53.957633+0000) 2022-09-10T06:01:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:54 smithi110 ceph-mon[97633]: pgmap v2971: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:54 smithi104 ceph-mon[97044]: pgmap v2971: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:54.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:54.363173+0000) 2022-09-10T06:01:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:54.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:54.957864+0000) 2022-09-10T06:01:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:55.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:55.363396+0000) 2022-09-10T06:01:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:55.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:55.958116+0000) 2022-09-10T06:01:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:56.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:56.363644+0000) 2022-09-10T06:01:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:56 smithi110 ceph-mon[97633]: pgmap v2972: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:56 smithi104 ceph-mon[97044]: pgmap v2972: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:01:57.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:56.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:56.958351+0000) 2022-09-10T06:01:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:57.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:57.363839+0000) 2022-09-10T06:01:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:01:57.398+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:01:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:57.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:57.958603+0000) 2022-09-10T06:01:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:58.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:58.364084+0000) 2022-09-10T06:01:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:01:58 smithi110 ceph-mon[97633]: pgmap v2973: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:01:58 smithi104 ceph-mon[97044]: pgmap v2973: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:01:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:58.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:58.958854+0000) 2022-09-10T06:01:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:01:59.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:59.364337+0000) 2022-09-10T06:02:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:01:59.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:01:59.959105+0000) 2022-09-10T06:02:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:00.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:00.364581+0000) 2022-09-10T06:02:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:00 smithi110 ceph-mon[97633]: pgmap v2974: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:00 smithi104 ceph-mon[97044]: pgmap v2974: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:00.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:00.959271+0000) 2022-09-10T06:02:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:01.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:01.364779+0000) 2022-09-10T06:02:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:01.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:01.959436+0000) 2022-09-10T06:02:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:02.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:02.364963+0000) 2022-09-10T06:02:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:02 smithi110 ceph-mon[97633]: pgmap v2975: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:02 smithi104 ceph-mon[97044]: pgmap v2975: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:03.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:02.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:02.959686+0000) 2022-09-10T06:02:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:03.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:03.365148+0000) 2022-09-10T06:02:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:03.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:03.959894+0000) 2022-09-10T06:02:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:04 smithi110 ceph-mon[97633]: pgmap v2976: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:04 smithi104 ceph-mon[97044]: pgmap v2976: 129 pgs: 129 active+clean; 8.0 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:04.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:04.365331+0000) 2022-09-10T06:02:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:04.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:04.960135+0000) 2022-09-10T06:02:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:05.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:05.365534+0000) 2022-09-10T06:02:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:05.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:05.960397+0000) 2022-09-10T06:02:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:06.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:06.365791+0000) 2022-09-10T06:02:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:06 smithi110 ceph-mon[97633]: pgmap v2977: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:02:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:06 smithi104 ceph-mon[97044]: pgmap v2977: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:02:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:06.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:06.960672+0000) 2022-09-10T06:02:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:07.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:07.365983+0000) 2022-09-10T06:02:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:07.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:07.960929+0000) 2022-09-10T06:02:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:08.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:08.366146+0000) 2022-09-10T06:02:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:08 smithi110 ceph-mon[97633]: pgmap v2978: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:02:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:08 smithi110 ceph-mon[97633]: osdmap e270: 12 total, 12 up, 12 in 2022-09-10T06:02:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:08 smithi104 ceph-mon[97044]: pgmap v2978: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:02:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:08 smithi104 ceph-mon[97044]: osdmap e270: 12 total, 12 up, 12 in 2022-09-10T06:02:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:08.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:08.961149+0000) 2022-09-10T06:02:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:09.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:09.366357+0000) 2022-09-10T06:02:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:09.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:09.961378+0000) 2022-09-10T06:02:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:10.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:10.366534+0000) 2022-09-10T06:02:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:10 smithi110 ceph-mon[97633]: pgmap v2980: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:02:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:10 smithi104 ceph-mon[97044]: pgmap v2980: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:02:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:10.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:10.961582+0000) 2022-09-10T06:02:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:11.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:11.366715+0000) 2022-09-10T06:02:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:11.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:11.961817+0000) 2022-09-10T06:02:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:12.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:12.366906+0000) 2022-09-10T06:02:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:02:12.401+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:02:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:12 smithi110 ceph-mon[97633]: pgmap v2981: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:02:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:12 smithi104 ceph-mon[97044]: pgmap v2981: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:02:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:12.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:12.961988+0000) 2022-09-10T06:02:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:13.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:13.367183+0000) 2022-09-10T06:02:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:13.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:13.962220+0000) 2022-09-10T06:02:14.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:14 smithi110 ceph-mon[97633]: osdmap e271: 12 total, 12 up, 12 in 2022-09-10T06:02:14.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:14 smithi110 ceph-mon[97633]: pgmap v2983: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:02:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:14 smithi104 ceph-mon[97044]: osdmap e271: 12 total, 12 up, 12 in 2022-09-10T06:02:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:14 smithi104 ceph-mon[97044]: pgmap v2983: 129 pgs: 129 active+clean; 8.0 MiB data, 479 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:02:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:14.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:14.367388+0000) 2022-09-10T06:02:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:14.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:14.962471+0000) 2022-09-10T06:02:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:15.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:15.367590+0000) 2022-09-10T06:02:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:15.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:15.962707+0000) 2022-09-10T06:02:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:16.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:16.367828+0000) 2022-09-10T06:02:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:16 smithi110 ceph-mon[97633]: pgmap v2984: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:02:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:16 smithi104 ceph-mon[97044]: pgmap v2984: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:02:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:16.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:16.962951+0000) 2022-09-10T06:02:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:17.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:17.368054+0000) 2022-09-10T06:02:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:17.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:17.963196+0000) 2022-09-10T06:02:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:18.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:18.368255+0000) 2022-09-10T06:02:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:18 smithi110 ceph-mon[97633]: pgmap v2985: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:02:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:18 smithi104 ceph-mon[97044]: pgmap v2985: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:02:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:18.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:18.963439+0000) 2022-09-10T06:02:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:19.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:19.368459+0000) 2022-09-10T06:02:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:19.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:19.963666+0000) 2022-09-10T06:02:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:20.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:20.368708+0000) 2022-09-10T06:02:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:20 smithi110 ceph-mon[97633]: pgmap v2986: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:02:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:20 smithi104 ceph-mon[97044]: pgmap v2986: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:02:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:20.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:20.963882+0000) 2022-09-10T06:02:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:21.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:21.368964+0000) 2022-09-10T06:02:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:21.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:21.964138+0000) 2022-09-10T06:02:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:22.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:22.369176+0000) 2022-09-10T06:02:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:22 smithi110 ceph-mon[97633]: pgmap v2987: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:02:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:22 smithi104 ceph-mon[97044]: pgmap v2987: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:02:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:22.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:22.964323+0000) 2022-09-10T06:02:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:23.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:23.369338+0000) 2022-09-10T06:02:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:23.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:23.964527+0000) 2022-09-10T06:02:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:24 smithi104 ceph-mon[97044]: pgmap v2988: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:02:24.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:24.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:24.369479+0000) 2022-09-10T06:02:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:24 smithi110 ceph-mon[97633]: pgmap v2988: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:02:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:24.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:24.964784+0000) 2022-09-10T06:02:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:25.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:25.369696+0000) 2022-09-10T06:02:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:25.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:25.964965+0000) 2022-09-10T06:02:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:26.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:26.369942+0000) 2022-09-10T06:02:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:26 smithi110 ceph-mon[97633]: pgmap v2989: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:02:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:26 smithi104 ceph-mon[97044]: pgmap v2989: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:02:27.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:26.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:26.965135+0000) 2022-09-10T06:02:27.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:27.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:27.370178+0000) 2022-09-10T06:02:27.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:02:27.404+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:02:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:27.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:27.965265+0000) 2022-09-10T06:02:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:28.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:28.370394+0000) 2022-09-10T06:02:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:28 smithi110 ceph-mon[97633]: pgmap v2990: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:28 smithi104 ceph-mon[97044]: pgmap v2990: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:29.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:28.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:28.965444+0000) 2022-09-10T06:02:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:29.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:29.370569+0000) 2022-09-10T06:02:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:29.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:29.965722+0000) 2022-09-10T06:02:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:30.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:30.370787+0000) 2022-09-10T06:02:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:30 smithi110 ceph-mon[97633]: pgmap v2991: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:30 smithi104 ceph-mon[97044]: pgmap v2991: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:30.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:30.965966+0000) 2022-09-10T06:02:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:31.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:31.370999+0000) 2022-09-10T06:02:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:31.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:31.966190+0000) 2022-09-10T06:02:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:32.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:32.371192+0000) 2022-09-10T06:02:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:32 smithi110 ceph-mon[97633]: pgmap v2992: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:32 smithi104 ceph-mon[97044]: pgmap v2992: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:32.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:32.966373+0000) 2022-09-10T06:02:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:33.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:33.371380+0000) 2022-09-10T06:02:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:33.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:33.966590+0000) 2022-09-10T06:02:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:34 smithi104 ceph-mon[97044]: pgmap v2993: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:34.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:34.371575+0000) 2022-09-10T06:02:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:34 smithi110 ceph-mon[97633]: pgmap v2993: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:34.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:34.966790+0000) 2022-09-10T06:02:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:35.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:35.371791+0000) 2022-09-10T06:02:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:35.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:35.966952+0000) 2022-09-10T06:02:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:36.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:36.372026+0000) 2022-09-10T06:02:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:36 smithi110 ceph-mon[97633]: pgmap v2994: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:37.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:36 smithi104 ceph-mon[97044]: pgmap v2994: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:36.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:36.967217+0000) 2022-09-10T06:02:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:37.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:37.372244+0000) 2022-09-10T06:02:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:37.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:37.967412+0000) 2022-09-10T06:02:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:38.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:38.372468+0000) 2022-09-10T06:02:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:38 smithi110 ceph-mon[97633]: pgmap v2995: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:38 smithi104 ceph-mon[97044]: pgmap v2995: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:38.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:38.967651+0000) 2022-09-10T06:02:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:39.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:39.372670+0000) 2022-09-10T06:02:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:39.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:39.967875+0000) 2022-09-10T06:02:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:40.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:40.372894+0000) 2022-09-10T06:02:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:40 smithi110 ceph-mon[97633]: pgmap v2996: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:40 smithi104 ceph-mon[97044]: pgmap v2996: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:40.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:40.968119+0000) 2022-09-10T06:02:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:41.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:41.373158+0000) 2022-09-10T06:02:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:41.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:41.968296+0000) 2022-09-10T06:02:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:42.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:42.373380+0000) 2022-09-10T06:02:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:02:42.419+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:02:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:42 smithi110 ceph-mon[97633]: pgmap v2997: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:42 smithi104 ceph-mon[97044]: pgmap v2997: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:42.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:42.968472+0000) 2022-09-10T06:02:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:43.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:43.373582+0000) 2022-09-10T06:02:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:43.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:43.968723+0000) 2022-09-10T06:02:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:44 smithi104 ceph-mon[97044]: pgmap v2998: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:02:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:02:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:02:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:02:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:44.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:44.373810+0000) 2022-09-10T06:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:44 smithi110 ceph-mon[97633]: pgmap v2998: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:02:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:02:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:02:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:02:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:44.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:44.969003+0000) 2022-09-10T06:02:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:45.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:45.373989+0000) 2022-09-10T06:02:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:45.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:45.969261+0000) 2022-09-10T06:02:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:46.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:46.374208+0000) 2022-09-10T06:02:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:46 smithi110 ceph-mon[97633]: pgmap v2999: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:46 smithi104 ceph-mon[97044]: pgmap v2999: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:46.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:46.969461+0000) 2022-09-10T06:02:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:47.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:47.374421+0000) 2022-09-10T06:02:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:47.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:47.969771+0000) 2022-09-10T06:02:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:48.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:48.374659+0000) 2022-09-10T06:02:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:48 smithi110 ceph-mon[97633]: pgmap v3000: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:48 smithi104 ceph-mon[97044]: pgmap v3000: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:48.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:48.969982+0000) 2022-09-10T06:02:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:49.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:49.374933+0000) 2022-09-10T06:02:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:49.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:49.970208+0000) 2022-09-10T06:02:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:50.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:50.375180+0000) 2022-09-10T06:02:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:50 smithi110 ceph-mon[97633]: pgmap v3001: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:50 smithi104 ceph-mon[97044]: pgmap v3001: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:50.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:50.970385+0000) 2022-09-10T06:02:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:51.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:51.375394+0000) 2022-09-10T06:02:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:51.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:51.970578+0000) 2022-09-10T06:02:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:52.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:52.375651+0000) 2022-09-10T06:02:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:52 smithi110 ceph-mon[97633]: pgmap v3002: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:52 smithi104 ceph-mon[97044]: pgmap v3002: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:52.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:52.970732+0000) 2022-09-10T06:02:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:53.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:53.375865+0000) 2022-09-10T06:02:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:53.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:53.970921+0000) 2022-09-10T06:02:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:54 smithi104 ceph-mon[97044]: pgmap v3003: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:54.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:54.376159+0000) 2022-09-10T06:02:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:54 smithi110 ceph-mon[97633]: pgmap v3003: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:54.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:54.971076+0000) 2022-09-10T06:02:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:55.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:55.376392+0000) 2022-09-10T06:02:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:55.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:55.971326+0000) 2022-09-10T06:02:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:56.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:56.376654+0000) 2022-09-10T06:02:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:56 smithi110 ceph-mon[97633]: pgmap v3004: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:56 smithi104 ceph-mon[97044]: pgmap v3004: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:02:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:56.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:56.971542+0000) 2022-09-10T06:02:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:57.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:57.376854+0000) 2022-09-10T06:02:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:02:57.424+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:02:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:57.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:57.971767+0000) 2022-09-10T06:02:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:58.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:58.377086+0000) 2022-09-10T06:02:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:02:58 smithi110 ceph-mon[97633]: pgmap v3005: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:02:58 smithi104 ceph-mon[97044]: pgmap v3005: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:02:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:58.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:58.971981+0000) 2022-09-10T06:02:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:02:59.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:59.377273+0000) 2022-09-10T06:03:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:02:59.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:02:59.972239+0000) 2022-09-10T06:03:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:00.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:00.377499+0000) 2022-09-10T06:03:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:00 smithi110 ceph-mon[97633]: pgmap v3006: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:00 smithi104 ceph-mon[97044]: pgmap v3006: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:01.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:00.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:00.972407+0000) 2022-09-10T06:03:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:01.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:01.377730+0000) 2022-09-10T06:03:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:01.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:01.972635+0000) 2022-09-10T06:03:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:02.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:02.377993+0000) 2022-09-10T06:03:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:02 smithi110 ceph-mon[97633]: pgmap v3007: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:02 smithi104 ceph-mon[97044]: pgmap v3007: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:02.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:02.972852+0000) 2022-09-10T06:03:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:03.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:03.378221+0000) 2022-09-10T06:03:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:03.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:03.973092+0000) 2022-09-10T06:03:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:04 smithi104 ceph-mon[97044]: pgmap v3008: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:04.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:04.378460+0000) 2022-09-10T06:03:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:04 smithi110 ceph-mon[97633]: pgmap v3008: 129 pgs: 129 active+clean; 8.0 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:04.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:04.973387+0000) 2022-09-10T06:03:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:05.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:05.378704+0000) 2022-09-10T06:03:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:05.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:05.973566+0000) 2022-09-10T06:03:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:06.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:06.378923+0000) 2022-09-10T06:03:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:06 smithi110 ceph-mon[97633]: pgmap v3009: 129 pgs: 129 active+clean; 8.0 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:03:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:06 smithi104 ceph-mon[97044]: pgmap v3009: 129 pgs: 129 active+clean; 8.0 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:03:07.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:06.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:06.973704+0000) 2022-09-10T06:03:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:07.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:07.379172+0000) 2022-09-10T06:03:08.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:07.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:07.973947+0000) 2022-09-10T06:03:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:08.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:08.379361+0000) 2022-09-10T06:03:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:08 smithi110 ceph-mon[97633]: pgmap v3010: 129 pgs: 129 active+clean; 8.0 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:03:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:08 smithi110 ceph-mon[97633]: osdmap e272: 12 total, 12 up, 12 in 2022-09-10T06:03:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:08 smithi104 ceph-mon[97044]: pgmap v3010: 129 pgs: 129 active+clean; 8.0 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:03:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:08 smithi104 ceph-mon[97044]: osdmap e272: 12 total, 12 up, 12 in 2022-09-10T06:03:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:08.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:08.974153+0000) 2022-09-10T06:03:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:09.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:09.379607+0000) 2022-09-10T06:03:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:09.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:09.974346+0000) 2022-09-10T06:03:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:10.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:10.379842+0000) 2022-09-10T06:03:10.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:10 smithi110 ceph-mon[97633]: pgmap v3012: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:03:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:10 smithi104 ceph-mon[97044]: pgmap v3012: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:03:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:10.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:10.974515+0000) 2022-09-10T06:03:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:11.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:11.380060+0000) 2022-09-10T06:03:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:11.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:11.974736+0000) 2022-09-10T06:03:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:12.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:12.380291+0000) 2022-09-10T06:03:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:03:12.439+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:03:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:12 smithi110 ceph-mon[97633]: pgmap v3013: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:03:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:12 smithi104 ceph-mon[97044]: pgmap v3013: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:03:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:12.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:12.974980+0000) 2022-09-10T06:03:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:13.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:13.380448+0000) 2022-09-10T06:03:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:13.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:13.975231+0000) 2022-09-10T06:03:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:14 smithi104 ceph-mon[97044]: osdmap e273: 12 total, 12 up, 12 in 2022-09-10T06:03:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:14 smithi104 ceph-mon[97044]: pgmap v3015: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:03:14.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:14.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:14.380705+0000) 2022-09-10T06:03:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:14 smithi110 ceph-mon[97633]: osdmap e273: 12 total, 12 up, 12 in 2022-09-10T06:03:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:14 smithi110 ceph-mon[97633]: pgmap v3015: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:03:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:14.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:14.975503+0000) 2022-09-10T06:03:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:15.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:15.380930+0000) 2022-09-10T06:03:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:15.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:15.975776+0000) 2022-09-10T06:03:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:16.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:16.381170+0000) 2022-09-10T06:03:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:16 smithi110 ceph-mon[97633]: pgmap v3016: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:03:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:16 smithi104 ceph-mon[97044]: pgmap v3016: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:03:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:16.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:16.975992+0000) 2022-09-10T06:03:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:17.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:17.381299+0000) 2022-09-10T06:03:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:17.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:17.976228+0000) 2022-09-10T06:03:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:18.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:18.381520+0000) 2022-09-10T06:03:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:18 smithi110 ceph-mon[97633]: pgmap v3017: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:03:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:18 smithi104 ceph-mon[97044]: pgmap v3017: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:03:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:18.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:18.976410+0000) 2022-09-10T06:03:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:19.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:19.381729+0000) 2022-09-10T06:03:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:19.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:19.976621+0000) 2022-09-10T06:03:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:20.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:20.381955+0000) 2022-09-10T06:03:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:20 smithi110 ceph-mon[97633]: pgmap v3018: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:03:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:20 smithi104 ceph-mon[97044]: pgmap v3018: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:03:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:20.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:20.976857+0000) 2022-09-10T06:03:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:21.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:21.382205+0000) 2022-09-10T06:03:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:21.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:21.977032+0000) 2022-09-10T06:03:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:22.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:22.382372+0000) 2022-09-10T06:03:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:22 smithi110 ceph-mon[97633]: pgmap v3019: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:03:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:22 smithi104 ceph-mon[97044]: pgmap v3019: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:03:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:22.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:22.977200+0000) 2022-09-10T06:03:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:23.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:23.382542+0000) 2022-09-10T06:03:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:23.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:23.977463+0000) 2022-09-10T06:03:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:24 smithi104 ceph-mon[97044]: pgmap v3020: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:03:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:24.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:24.382781+0000) 2022-09-10T06:03:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:24 smithi110 ceph-mon[97633]: pgmap v3020: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:03:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:24.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:24.977675+0000) 2022-09-10T06:03:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:25.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:25.382974+0000) 2022-09-10T06:03:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:25.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:25.977857+0000) 2022-09-10T06:03:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:26.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:26.383201+0000) 2022-09-10T06:03:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:26 smithi110 ceph-mon[97633]: pgmap v3021: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:03:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:26 smithi104 ceph-mon[97044]: pgmap v3021: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:03:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:26.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:26.978038+0000) 2022-09-10T06:03:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:27.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:27.383390+0000) 2022-09-10T06:03:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:03:27.456+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:03:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:27.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:27.978223+0000) 2022-09-10T06:03:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:28.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:28.383539+0000) 2022-09-10T06:03:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:28 smithi110 ceph-mon[97633]: pgmap v3022: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:28 smithi104 ceph-mon[97044]: pgmap v3022: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:28.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:28.978432+0000) 2022-09-10T06:03:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:29.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:29.383761+0000) 2022-09-10T06:03:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:29.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:29.978713+0000) 2022-09-10T06:03:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:30.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:30.383952+0000) 2022-09-10T06:03:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:30 smithi110 ceph-mon[97633]: pgmap v3023: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:30 smithi104 ceph-mon[97044]: pgmap v3023: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:30.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:30.978976+0000) 2022-09-10T06:03:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:31.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:31.384203+0000) 2022-09-10T06:03:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:31 smithi110 ceph-mon[97633]: pgmap v3024: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:31 smithi104 ceph-mon[97044]: pgmap v3024: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:31.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:31.979220+0000) 2022-09-10T06:03:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:32.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:32.384378+0000) 2022-09-10T06:03:33.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:32.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:32.979452+0000) 2022-09-10T06:03:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:33.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:33.384547+0000) 2022-09-10T06:03:34.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:33.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:33.979611+0000) 2022-09-10T06:03:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:34 smithi104 ceph-mon[97044]: pgmap v3025: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:34.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:34.384802+0000) 2022-09-10T06:03:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:34 smithi110 ceph-mon[97633]: pgmap v3025: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:34.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:34.979796+0000) 2022-09-10T06:03:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:35.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:35.385045+0000) 2022-09-10T06:03:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:35.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:35.980062+0000) 2022-09-10T06:03:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:36.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:36.385159+0000) 2022-09-10T06:03:36.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:36 smithi110 ceph-mon[97633]: pgmap v3026: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:36 smithi104 ceph-mon[97044]: pgmap v3026: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:36.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:36.980305+0000) 2022-09-10T06:03:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:37.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:37.385361+0000) 2022-09-10T06:03:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:37.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:37.980537+0000) 2022-09-10T06:03:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:38.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:38.385600+0000) 2022-09-10T06:03:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:38 smithi110 ceph-mon[97633]: pgmap v3027: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:38 smithi104 ceph-mon[97044]: pgmap v3027: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:38.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:38.980789+0000) 2022-09-10T06:03:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:39.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:39.385782+0000) 2022-09-10T06:03:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:39.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:39.981047+0000) 2022-09-10T06:03:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:40.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:40.386031+0000) 2022-09-10T06:03:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:40 smithi110 ceph-mon[97633]: pgmap v3028: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:40 smithi104 ceph-mon[97044]: pgmap v3028: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:40.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:40.981310+0000) 2022-09-10T06:03:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:41.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:41.386265+0000) 2022-09-10T06:03:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:41.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:41.981526+0000) 2022-09-10T06:03:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:42.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:42.386480+0000) 2022-09-10T06:03:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:03:42.466+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:03:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:42 smithi110 ceph-mon[97633]: pgmap v3029: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:42 smithi104 ceph-mon[97044]: pgmap v3029: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:42.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:42.981754+0000) 2022-09-10T06:03:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:43.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:43.386678+0000) 2022-09-10T06:03:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:43.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:43.982006+0000) 2022-09-10T06:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:44 smithi104 ceph-mon[97044]: pgmap v3030: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:03:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:03:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:03:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:03:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:44.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:44.386905+0000) 2022-09-10T06:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:44 smithi110 ceph-mon[97633]: pgmap v3030: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:03:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:44.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:44.982253+0000) 2022-09-10T06:03:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:45.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:45.387179+0000) 2022-09-10T06:03:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:45.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:45.982425+0000) 2022-09-10T06:03:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:46.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:46.387436+0000) 2022-09-10T06:03:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:46 smithi110 ceph-mon[97633]: pgmap v3031: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:46 smithi104 ceph-mon[97044]: pgmap v3031: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:46.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:46.982600+0000) 2022-09-10T06:03:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:47.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:47.387632+0000) 2022-09-10T06:03:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:47.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:47.982831+0000) 2022-09-10T06:03:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:48.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:48.387862+0000) 2022-09-10T06:03:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:48 smithi110 ceph-mon[97633]: pgmap v3032: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:48 smithi104 ceph-mon[97044]: pgmap v3032: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:48.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:48.983054+0000) 2022-09-10T06:03:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:49.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:49.388165+0000) 2022-09-10T06:03:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:49.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:49.983239+0000) 2022-09-10T06:03:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:50.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:50.388349+0000) 2022-09-10T06:03:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:50 smithi110 ceph-mon[97633]: pgmap v3033: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:50 smithi104 ceph-mon[97044]: pgmap v3033: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:50.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:50.983468+0000) 2022-09-10T06:03:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:51.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:51.388512+0000) 2022-09-10T06:03:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:51 smithi104 ceph-mon[97044]: pgmap v3034: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:51 smithi110 ceph-mon[97633]: pgmap v3034: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:51.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:51.983634+0000) 2022-09-10T06:03:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:52.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:52.388756+0000) 2022-09-10T06:03:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:52.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:52.983813+0000) 2022-09-10T06:03:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:53.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:53.389026+0000) 2022-09-10T06:03:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:53.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:53.983992+0000) 2022-09-10T06:03:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:54 smithi104 ceph-mon[97044]: pgmap v3035: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:54.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:54.389269+0000) 2022-09-10T06:03:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:54 smithi110 ceph-mon[97633]: pgmap v3035: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:54.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:54.984216+0000) 2022-09-10T06:03:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:55.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:55.389480+0000) 2022-09-10T06:03:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:55.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:55.984360+0000) 2022-09-10T06:03:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:56.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:56.389639+0000) 2022-09-10T06:03:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:56 smithi110 ceph-mon[97633]: pgmap v3036: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:56 smithi104 ceph-mon[97044]: pgmap v3036: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:03:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:56.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:56.984579+0000) 2022-09-10T06:03:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:57.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:57.389840+0000) 2022-09-10T06:03:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:03:57.478+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:03:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:57.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:57.984773+0000) 2022-09-10T06:03:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:58.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:58.390046+0000) 2022-09-10T06:03:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:03:58 smithi110 ceph-mon[97633]: pgmap v3037: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:03:58 smithi104 ceph-mon[97044]: pgmap v3037: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:03:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:58.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:58.984982+0000) 2022-09-10T06:03:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:03:59.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:59.390280+0000) 2022-09-10T06:04:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:03:59.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:03:59.985221+0000) 2022-09-10T06:04:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:00.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:00.390504+0000) 2022-09-10T06:04:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:00 smithi110 ceph-mon[97633]: pgmap v3038: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:00 smithi104 ceph-mon[97044]: pgmap v3038: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:00.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:00.985392+0000) 2022-09-10T06:04:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:01.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:01.390644+0000) 2022-09-10T06:04:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:01 smithi110 ceph-mon[97633]: pgmap v3039: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:01 smithi104 ceph-mon[97044]: pgmap v3039: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:01.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:01.985576+0000) 2022-09-10T06:04:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:02.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:02.390824+0000) 2022-09-10T06:04:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:02.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:02.985822+0000) 2022-09-10T06:04:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:03.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:03.391073+0000) 2022-09-10T06:04:04.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:03.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:03.986128+0000) 2022-09-10T06:04:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:04 smithi104 ceph-mon[97044]: pgmap v3040: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:04.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:04.391329+0000) 2022-09-10T06:04:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:04 smithi110 ceph-mon[97633]: pgmap v3040: 129 pgs: 129 active+clean; 8.0 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:04.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:04.986347+0000) 2022-09-10T06:04:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:05.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:05.391528+0000) 2022-09-10T06:04:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:05.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:05.986577+0000) 2022-09-10T06:04:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:06.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:06.391748+0000) 2022-09-10T06:04:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:06 smithi110 ceph-mon[97633]: pgmap v3041: 129 pgs: 129 active+clean; 8.0 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:04:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:06 smithi104 ceph-mon[97044]: pgmap v3041: 129 pgs: 129 active+clean; 8.0 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:04:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:06.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:06.986793+0000) 2022-09-10T06:04:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:07.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:07.391941+0000) 2022-09-10T06:04:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:07 smithi104 ceph-mon[97044]: pgmap v3042: 129 pgs: 129 active+clean; 8.0 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:04:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:07 smithi104 ceph-mon[97044]: osdmap e274: 12 total, 12 up, 12 in 2022-09-10T06:04:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:07 smithi110 ceph-mon[97633]: pgmap v3042: 129 pgs: 129 active+clean; 8.0 MiB data, 484 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:04:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:07 smithi110 ceph-mon[97633]: osdmap e274: 12 total, 12 up, 12 in 2022-09-10T06:04:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:07.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:07.986997+0000) 2022-09-10T06:04:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:08.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:08.392165+0000) 2022-09-10T06:04:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:08.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:08.987270+0000) 2022-09-10T06:04:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:09.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:09.392336+0000) 2022-09-10T06:04:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:09.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:09.987517+0000) 2022-09-10T06:04:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:10.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:10.392557+0000) 2022-09-10T06:04:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:10 smithi110 ceph-mon[97633]: pgmap v3044: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:04:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:10 smithi104 ceph-mon[97044]: pgmap v3044: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:04:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:10.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:10.987708+0000) 2022-09-10T06:04:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:11.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:11.392751+0000) 2022-09-10T06:04:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:11 smithi104 ceph-mon[97044]: pgmap v3045: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:04:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:11 smithi110 ceph-mon[97633]: pgmap v3045: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:04:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:11.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:11.987932+0000) 2022-09-10T06:04:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:12.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:12.392965+0000) 2022-09-10T06:04:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:04:12.480+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:04:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:12.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:12.988139+0000) 2022-09-10T06:04:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:13.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:13.393177+0000) 2022-09-10T06:04:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:13.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:13.988273+0000) 2022-09-10T06:04:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:14 smithi104 ceph-mon[97044]: osdmap e275: 12 total, 12 up, 12 in 2022-09-10T06:04:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:14 smithi104 ceph-mon[97044]: pgmap v3047: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:04:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:14.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:14.393422+0000) 2022-09-10T06:04:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:14 smithi110 ceph-mon[97633]: osdmap e275: 12 total, 12 up, 12 in 2022-09-10T06:04:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:14 smithi110 ceph-mon[97633]: pgmap v3047: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:04:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:14.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:14.988485+0000) 2022-09-10T06:04:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:15.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:15.393620+0000) 2022-09-10T06:04:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:15.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:15.988710+0000) 2022-09-10T06:04:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:16.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:16.393868+0000) 2022-09-10T06:04:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:16 smithi110 ceph-mon[97633]: pgmap v3048: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:04:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:16 smithi104 ceph-mon[97044]: pgmap v3048: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:04:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:16.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:16.988925+0000) 2022-09-10T06:04:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:17.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:17.394138+0000) 2022-09-10T06:04:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:17 smithi110 ceph-mon[97633]: pgmap v3049: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:04:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:17 smithi104 ceph-mon[97044]: pgmap v3049: 129 pgs: 129 active+clean; 8.0 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:04:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:17.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:17.989158+0000) 2022-09-10T06:04:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:18.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:18.394388+0000) 2022-09-10T06:04:19.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:18.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:18.989400+0000) 2022-09-10T06:04:19.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:19.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:19.394581+0000) 2022-09-10T06:04:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:19.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:19.989621+0000) 2022-09-10T06:04:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:20.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:20.394769+0000) 2022-09-10T06:04:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:20 smithi110 ceph-mon[97633]: pgmap v3050: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:04:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:20 smithi104 ceph-mon[97044]: pgmap v3050: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:04:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:20.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:20.989823+0000) 2022-09-10T06:04:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:21.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:21.394953+0000) 2022-09-10T06:04:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:21 smithi110 ceph-mon[97633]: pgmap v3051: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:04:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:21 smithi104 ceph-mon[97044]: pgmap v3051: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:04:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:21.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:21.990004+0000) 2022-09-10T06:04:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:22.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:22.395218+0000) 2022-09-10T06:04:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:22.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:22.990226+0000) 2022-09-10T06:04:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:23.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:23.395438+0000) 2022-09-10T06:04:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:23.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:23.990427+0000) 2022-09-10T06:04:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:24 smithi104 ceph-mon[97044]: pgmap v3052: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:04:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:24.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:24.395752+0000) 2022-09-10T06:04:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:24 smithi110 ceph-mon[97633]: pgmap v3052: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:04:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:24.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:24.990670+0000) 2022-09-10T06:04:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:25.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:25.396008+0000) 2022-09-10T06:04:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:25 smithi104 ceph-mon[97044]: pgmap v3053: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:04:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:25 smithi110 ceph-mon[97633]: pgmap v3053: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:04:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:25.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:25.990918+0000) 2022-09-10T06:04:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:26.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:26.396267+0000) 2022-09-10T06:04:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:26.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:26.991178+0000) 2022-09-10T06:04:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:27.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:27.396440+0000) 2022-09-10T06:04:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:04:27.492+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:04:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:27 smithi104 ceph-mon[97044]: pgmap v3054: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:27 smithi110 ceph-mon[97633]: pgmap v3054: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:27.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:27.991351+0000) 2022-09-10T06:04:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:28.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:28.396701+0000) 2022-09-10T06:04:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:28.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:28.991579+0000) 2022-09-10T06:04:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:29.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:29.396896+0000) 2022-09-10T06:04:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:29 smithi104 ceph-mon[97044]: pgmap v3055: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:29 smithi110 ceph-mon[97633]: pgmap v3055: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:29.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:29.991774+0000) 2022-09-10T06:04:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:30.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:30.397080+0000) 2022-09-10T06:04:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:30.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:30.991987+0000) 2022-09-10T06:04:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:31.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:31.397329+0000) 2022-09-10T06:04:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:31 smithi104 ceph-mon[97044]: pgmap v3056: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:31 smithi110 ceph-mon[97633]: pgmap v3056: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:31.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:31.992221+0000) 2022-09-10T06:04:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:32.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:32.397543+0000) 2022-09-10T06:04:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:32.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:32.992404+0000) 2022-09-10T06:04:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:33.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:33.397752+0000) 2022-09-10T06:04:34.315 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:33.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:33.992579+0000) 2022-09-10T06:04:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:34 smithi104 ceph-mon[97044]: pgmap v3057: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:34.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:34.398007+0000) 2022-09-10T06:04:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:34 smithi110 ceph-mon[97633]: pgmap v3057: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:34.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:34.992775+0000) 2022-09-10T06:04:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:35.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:35.398250+0000) 2022-09-10T06:04:36.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:35 smithi104 ceph-mon[97044]: pgmap v3058: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:35 smithi110 ceph-mon[97633]: pgmap v3058: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:35.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:35.993010+0000) 2022-09-10T06:04:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:36.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:36.398544+0000) 2022-09-10T06:04:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:36.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:36.993202+0000) 2022-09-10T06:04:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:37.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:37.398740+0000) 2022-09-10T06:04:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:37 smithi104 ceph-mon[97044]: pgmap v3059: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:37 smithi110 ceph-mon[97633]: pgmap v3059: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:37.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:37.993446+0000) 2022-09-10T06:04:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:38.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:38.398907+0000) 2022-09-10T06:04:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:38.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:38.993706+0000) 2022-09-10T06:04:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:39.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:39.399186+0000) 2022-09-10T06:04:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:39 smithi104 ceph-mon[97044]: pgmap v3060: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:39 smithi110 ceph-mon[97633]: pgmap v3060: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:39.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:39.993945+0000) 2022-09-10T06:04:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:40.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:40.399411+0000) 2022-09-10T06:04:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:40.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:40.994172+0000) 2022-09-10T06:04:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:41.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:41.399577+0000) 2022-09-10T06:04:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:41 smithi104 ceph-mon[97044]: pgmap v3061: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:41 smithi110 ceph-mon[97633]: pgmap v3061: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:41.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:41.994351+0000) 2022-09-10T06:04:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:04:42.508+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:04:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:42.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:42.399784+0000) 2022-09-10T06:04:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:42.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:42.994517+0000) 2022-09-10T06:04:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:43.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:43.399919+0000) 2022-09-10T06:04:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:43.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:43.994690+0000) 2022-09-10T06:04:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:44 smithi104 ceph-mon[97044]: pgmap v3062: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:04:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:04:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:04:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:04:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:44.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:44.400158+0000) 2022-09-10T06:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:44 smithi110 ceph-mon[97633]: pgmap v3062: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:04:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:04:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:04:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:44.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:44.994946+0000) 2022-09-10T06:04:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:45.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:45.400379+0000) 2022-09-10T06:04:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:45 smithi104 ceph-mon[97044]: pgmap v3063: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:45 smithi110 ceph-mon[97633]: pgmap v3063: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:45.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:45.995213+0000) 2022-09-10T06:04:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:46.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:46.400561+0000) 2022-09-10T06:04:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:46.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:46.995439+0000) 2022-09-10T06:04:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:47.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:47.400745+0000) 2022-09-10T06:04:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:47 smithi104 ceph-mon[97044]: pgmap v3064: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:47 smithi110 ceph-mon[97633]: pgmap v3064: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:04:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:47.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:47.995659+0000) 2022-09-10T06:04:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:48.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:48.400983+0000) 2022-09-10T06:04:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:48.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:48.995941+0000) 2022-09-10T06:04:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:49.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:49.401195+0000) 2022-09-10T06:04:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:49 smithi104 ceph-mon[97044]: pgmap v3065: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:49 smithi110 ceph-mon[97633]: pgmap v3065: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:50.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:49.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:49.996216+0000) 2022-09-10T06:04:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:50.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:50.401391+0000) 2022-09-10T06:04:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:50.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:50.996469+0000) 2022-09-10T06:04:51.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:51.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:51.401607+0000) 2022-09-10T06:04:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:51 smithi104 ceph-mon[97044]: pgmap v3066: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:04:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:51 smithi110 ceph-mon[97633]: pgmap v3066: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:04:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:51.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:51.996719+0000) 2022-09-10T06:04:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:52.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:52.401845+0000) 2022-09-10T06:04:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:52.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:52.996981+0000) 2022-09-10T06:04:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:53.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:53.402079+0000) 2022-09-10T06:04:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:53.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:53.997195+0000) 2022-09-10T06:04:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:54 smithi104 ceph-mon[97044]: pgmap v3067: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:04:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:54.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:54.402264+0000) 2022-09-10T06:04:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:54 smithi110 ceph-mon[97633]: pgmap v3067: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:04:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:54.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:54.997423+0000) 2022-09-10T06:04:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:55.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:55.402501+0000) 2022-09-10T06:04:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:55 smithi104 ceph-mon[97044]: pgmap v3068: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:55 smithi110 ceph-mon[97633]: pgmap v3068: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:04:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:55.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:55.997645+0000) 2022-09-10T06:04:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:56.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:56.402685+0000) 2022-09-10T06:04:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:56.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:56.997863+0000) 2022-09-10T06:04:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:57.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:57.402933+0000) 2022-09-10T06:04:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:04:57.509+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:04:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:57 smithi104 ceph-mon[97044]: pgmap v3069: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:04:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:57 smithi110 ceph-mon[97633]: pgmap v3069: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:04:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:57.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:57.998085+0000) 2022-09-10T06:04:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:58.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:58.403143+0000) 2022-09-10T06:04:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:58.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:58.998359+0000) 2022-09-10T06:04:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:04:59.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:59.403359+0000) 2022-09-10T06:05:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:04:59 smithi104 ceph-mon[97044]: pgmap v3070: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:04:59 smithi110 ceph-mon[97633]: pgmap v3070: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:04:59.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:04:59.998579+0000) 2022-09-10T06:05:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:00.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:00.403583+0000) 2022-09-10T06:05:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:00.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:00.998765+0000) 2022-09-10T06:05:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:01.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:01.403805+0000) 2022-09-10T06:05:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:01 smithi104 ceph-mon[97044]: pgmap v3071: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:01 smithi110 ceph-mon[97633]: pgmap v3071: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:01.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:01.998979+0000) 2022-09-10T06:05:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:02.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:02.404072+0000) 2022-09-10T06:05:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:02.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:02.999180+0000) 2022-09-10T06:05:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:03.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:03.404331+0000) 2022-09-10T06:05:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:03.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:03.999355+0000) 2022-09-10T06:05:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:04 smithi104 ceph-mon[97044]: pgmap v3072: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:04.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:04.404535+0000) 2022-09-10T06:05:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:04 smithi110 ceph-mon[97633]: pgmap v3072: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:04.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:04.999585+0000) 2022-09-10T06:05:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:05.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:05.404750+0000) 2022-09-10T06:05:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:05 smithi104 ceph-mon[97044]: pgmap v3073: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T06:05:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:05 smithi110 ceph-mon[97633]: pgmap v3073: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T06:05:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:05.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:05.999763+0000) 2022-09-10T06:05:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:06.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:06.404913+0000) 2022-09-10T06:05:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:06.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:06.999987+0000) 2022-09-10T06:05:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:07.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:07.405177+0000) 2022-09-10T06:05:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:07 smithi104 ceph-mon[97044]: pgmap v3074: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T06:05:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:07 smithi110 ceph-mon[97633]: pgmap v3074: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T06:05:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:08.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:08.000184+0000) 2022-09-10T06:05:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:08.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:08.405381+0000) 2022-09-10T06:05:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:09.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:09.000380+0000) 2022-09-10T06:05:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:09.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:09.405542+0000) 2022-09-10T06:05:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:09 smithi104 ceph-mon[97044]: pgmap v3075: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T06:05:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:09 smithi110 ceph-mon[97633]: pgmap v3075: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T06:05:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:10.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:10.000584+0000) 2022-09-10T06:05:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:10.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:10.405719+0000) 2022-09-10T06:05:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:11.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:11.000846+0000) 2022-09-10T06:05:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:11.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:11.405966+0000) 2022-09-10T06:05:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:11 smithi104 ceph-mon[97044]: pgmap v3076: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T06:05:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:11 smithi110 ceph-mon[97633]: pgmap v3076: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T06:05:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:11.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:12.000976+0000) 2022-09-10T06:05:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:12.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:12.406152+0000) 2022-09-10T06:05:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:05:12.512+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:05:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:13.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:13.001226+0000) 2022-09-10T06:05:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:13.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:13.406400+0000) 2022-09-10T06:05:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:14.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:14.001445+0000) 2022-09-10T06:05:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:14 smithi104 ceph-mon[97044]: pgmap v3077: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T06:05:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:14.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:14.406578+0000) 2022-09-10T06:05:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:14 smithi110 ceph-mon[97633]: pgmap v3077: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T06:05:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:15.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:15.001650+0000) 2022-09-10T06:05:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:15.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:15.406799+0000) 2022-09-10T06:05:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:15 smithi104 ceph-mon[97044]: pgmap v3078: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T06:05:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:15 smithi110 ceph-mon[97633]: pgmap v3078: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T06:05:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:16.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:16.001902+0000) 2022-09-10T06:05:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:16.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:16.407068+0000) 2022-09-10T06:05:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:17.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:17.002142+0000) 2022-09-10T06:05:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:17.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:17.407290+0000) 2022-09-10T06:05:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:17 smithi104 ceph-mon[97044]: pgmap v3079: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T06:05:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:17 smithi110 ceph-mon[97633]: pgmap v3079: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T06:05:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:18.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:18.002414+0000) 2022-09-10T06:05:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:18.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:18.407531+0000) 2022-09-10T06:05:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:19.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:19.002610+0000) 2022-09-10T06:05:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:19.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:19.407709+0000) 2022-09-10T06:05:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:19 smithi104 ceph-mon[97044]: pgmap v3080: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:05:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:19 smithi110 ceph-mon[97633]: pgmap v3080: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:05:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:20.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:20.002871+0000) 2022-09-10T06:05:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:20.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:20.407947+0000) 2022-09-10T06:05:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:21.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:21.003072+0000) 2022-09-10T06:05:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:21.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:21.408172+0000) 2022-09-10T06:05:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:21 smithi104 ceph-mon[97044]: pgmap v3081: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:21 smithi110 ceph-mon[97633]: pgmap v3081: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:22.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:22.003317+0000) 2022-09-10T06:05:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:22.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:22.408403+0000) 2022-09-10T06:05:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:23.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:23.003454+0000) 2022-09-10T06:05:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:23.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:23.408661+0000) 2022-09-10T06:05:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:24.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:24.003603+0000) 2022-09-10T06:05:24.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:24 smithi104 ceph-mon[97044]: pgmap v3082: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:24.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:24.408947+0000) 2022-09-10T06:05:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:24 smithi110 ceph-mon[97633]: pgmap v3082: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:25.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:25.003833+0000) 2022-09-10T06:05:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:25.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:25.409197+0000) 2022-09-10T06:05:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:25 smithi104 ceph-mon[97044]: pgmap v3083: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:25 smithi110 ceph-mon[97633]: pgmap v3083: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:26.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:26.004085+0000) 2022-09-10T06:05:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:26.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:26.409485+0000) 2022-09-10T06:05:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:27.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:27.004328+0000) 2022-09-10T06:05:27.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:05:27.526+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:05:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:27.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:27.409666+0000) 2022-09-10T06:05:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:27 smithi104 ceph-mon[97044]: pgmap v3084: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:27 smithi110 ceph-mon[97633]: pgmap v3084: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:28.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:28.004504+0000) 2022-09-10T06:05:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:28.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:28.409922+0000) 2022-09-10T06:05:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:29.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:29.004723+0000) 2022-09-10T06:05:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:29.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:29.410179+0000) 2022-09-10T06:05:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:29 smithi104 ceph-mon[97044]: pgmap v3085: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:29 smithi110 ceph-mon[97633]: pgmap v3085: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:30.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:30.004911+0000) 2022-09-10T06:05:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:30.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:30.410375+0000) 2022-09-10T06:05:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:31.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:31.005174+0000) 2022-09-10T06:05:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:31.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:31.410592+0000) 2022-09-10T06:05:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:31 smithi104 ceph-mon[97044]: pgmap v3086: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:31 smithi110 ceph-mon[97633]: pgmap v3086: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:32.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:32.005367+0000) 2022-09-10T06:05:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:32.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:32.410739+0000) 2022-09-10T06:05:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:33.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:33.005525+0000) 2022-09-10T06:05:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:33.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:33.410953+0000) 2022-09-10T06:05:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:34.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:34.005684+0000) 2022-09-10T06:05:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:34 smithi104 ceph-mon[97044]: pgmap v3087: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:34.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:34.411210+0000) 2022-09-10T06:05:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:34 smithi110 ceph-mon[97633]: pgmap v3087: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:35.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:35.005882+0000) 2022-09-10T06:05:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:35.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:35.411470+0000) 2022-09-10T06:05:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:35 smithi104 ceph-mon[97044]: pgmap v3088: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:35 smithi110 ceph-mon[97633]: pgmap v3088: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:36.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:36.006077+0000) 2022-09-10T06:05:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:36.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:36.411690+0000) 2022-09-10T06:05:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:37.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:37.006298+0000) 2022-09-10T06:05:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:37.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:37.411913+0000) 2022-09-10T06:05:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:37 smithi104 ceph-mon[97044]: pgmap v3089: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:37 smithi110 ceph-mon[97633]: pgmap v3089: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:38.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:38.006501+0000) 2022-09-10T06:05:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:38.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:38.412162+0000) 2022-09-10T06:05:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:39.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:39.006720+0000) 2022-09-10T06:05:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:39.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:39.412387+0000) 2022-09-10T06:05:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:39 smithi104 ceph-mon[97044]: pgmap v3090: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:39 smithi110 ceph-mon[97633]: pgmap v3090: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:40.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:40.006974+0000) 2022-09-10T06:05:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:40.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:40.412584+0000) 2022-09-10T06:05:41.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:41.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:41.007188+0000) 2022-09-10T06:05:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:41.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:41.412843+0000) 2022-09-10T06:05:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:41 smithi104 ceph-mon[97044]: pgmap v3091: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:41 smithi110 ceph-mon[97633]: pgmap v3091: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:42.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:42.007318+0000) 2022-09-10T06:05:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:05:42.542+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:05:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:42.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:42.413141+0000) 2022-09-10T06:05:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:43.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:43.007552+0000) 2022-09-10T06:05:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:43.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:43.413365+0000) 2022-09-10T06:05:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:44.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:44.007785+0000) 2022-09-10T06:05:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:44 smithi104 ceph-mon[97044]: pgmap v3092: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:05:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:05:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:05:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:05:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:44.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:44.413603+0000) 2022-09-10T06:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:44 smithi110 ceph-mon[97633]: pgmap v3092: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:05:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:05:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:05:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:05:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:45.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:45.007960+0000) 2022-09-10T06:05:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:45.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:45.413819+0000) 2022-09-10T06:05:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:45 smithi104 ceph-mon[97044]: pgmap v3093: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:45 smithi110 ceph-mon[97633]: pgmap v3093: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:46.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:46.008191+0000) 2022-09-10T06:05:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:46.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:46.414027+0000) 2022-09-10T06:05:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:47.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:47.008395+0000) 2022-09-10T06:05:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:47.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:47.414270+0000) 2022-09-10T06:05:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:47 smithi104 ceph-mon[97044]: pgmap v3094: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:05:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:47 smithi110 ceph-mon[97633]: pgmap v3094: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:05:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:48.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:48.008626+0000) 2022-09-10T06:05:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:48.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:48.414535+0000) 2022-09-10T06:05:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:49.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:49.008868+0000) 2022-09-10T06:05:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:49.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:49.414741+0000) 2022-09-10T06:05:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:49 smithi104 ceph-mon[97044]: pgmap v3095: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:49 smithi110 ceph-mon[97633]: pgmap v3095: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:50.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:50.009138+0000) 2022-09-10T06:05:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:50.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:50.414980+0000) 2022-09-10T06:05:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:51.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:51.009278+0000) 2022-09-10T06:05:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:51.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:51.415213+0000) 2022-09-10T06:05:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:51 smithi104 ceph-mon[97044]: pgmap v3096: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:05:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:51 smithi110 ceph-mon[97633]: pgmap v3096: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:05:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:52.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:52.009496+0000) 2022-09-10T06:05:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:52.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:52.415376+0000) 2022-09-10T06:05:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:53.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:53.009633+0000) 2022-09-10T06:05:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:53.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:53.415616+0000) 2022-09-10T06:05:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:54.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:54.009839+0000) 2022-09-10T06:05:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:54 smithi104 ceph-mon[97044]: pgmap v3097: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:05:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:54.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:54.415870+0000) 2022-09-10T06:05:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:54 smithi110 ceph-mon[97633]: pgmap v3097: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:05:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:55.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:55.010136+0000) 2022-09-10T06:05:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:55.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:55.416035+0000) 2022-09-10T06:05:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:55 smithi104 ceph-mon[97044]: pgmap v3098: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:55 smithi110 ceph-mon[97633]: pgmap v3098: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:05:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:56.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:56.010413+0000) 2022-09-10T06:05:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:56.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:56.416274+0000) 2022-09-10T06:05:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:57.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:57.010601+0000) 2022-09-10T06:05:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:57.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:57.416431+0000) 2022-09-10T06:05:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:05:57.552+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:05:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:57 smithi104 ceph-mon[97044]: pgmap v3099: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:57 smithi110 ceph-mon[97633]: pgmap v3099: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:05:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:58.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:58.010850+0000) 2022-09-10T06:05:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:58.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:58.416669+0000) 2022-09-10T06:05:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:05:59.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:59.011074+0000) 2022-09-10T06:05:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:05:59.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:05:59.416955+0000) 2022-09-10T06:06:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:05:59 smithi104 ceph-mon[97044]: pgmap v3100: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:05:59 smithi110 ceph-mon[97633]: pgmap v3100: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:00.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:00.011319+0000) 2022-09-10T06:06:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:00.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:00.417099+0000) 2022-09-10T06:06:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:01.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:01.011502+0000) 2022-09-10T06:06:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:01.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:01.417270+0000) 2022-09-10T06:06:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:01 smithi104 ceph-mon[97044]: pgmap v3101: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:01 smithi110 ceph-mon[97633]: pgmap v3101: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:02.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:02.011684+0000) 2022-09-10T06:06:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:02.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:02.417471+0000) 2022-09-10T06:06:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:03.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:03.011907+0000) 2022-09-10T06:06:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:03.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:03.417704+0000) 2022-09-10T06:06:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:04.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:04.012177+0000) 2022-09-10T06:06:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:04 smithi104 ceph-mon[97044]: pgmap v3102: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:04.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:04.417967+0000) 2022-09-10T06:06:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:04 smithi110 ceph-mon[97633]: pgmap v3102: 129 pgs: 129 active+clean; 8.0 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:05.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:05.012347+0000) 2022-09-10T06:06:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:05.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:05.418173+0000) 2022-09-10T06:06:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:05 smithi104 ceph-mon[97044]: pgmap v3103: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:06:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:05 smithi110 ceph-mon[97633]: pgmap v3103: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:06:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:06.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:06.012501+0000) 2022-09-10T06:06:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:06.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:06.418385+0000) 2022-09-10T06:06:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:07.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:07.012767+0000) 2022-09-10T06:06:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:07.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:07.418638+0000) 2022-09-10T06:06:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:08.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:08.013015+0000) 2022-09-10T06:06:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:08 smithi104 ceph-mon[97044]: osdmap e276: 12 total, 12 up, 12 in 2022-09-10T06:06:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:08 smithi104 ceph-mon[97044]: pgmap v3105: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:06:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:08 smithi110 ceph-mon[97633]: osdmap e276: 12 total, 12 up, 12 in 2022-09-10T06:06:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:08 smithi110 ceph-mon[97633]: pgmap v3105: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:06:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:08.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:08.418869+0000) 2022-09-10T06:06:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:09.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:09.013206+0000) 2022-09-10T06:06:09.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:09.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:09.419016+0000) 2022-09-10T06:06:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:09 smithi104 ceph-mon[97044]: pgmap v3106: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:06:10.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:09 smithi110 ceph-mon[97633]: pgmap v3106: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:06:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:10.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:10.013409+0000) 2022-09-10T06:06:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:10.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:10.419213+0000) 2022-09-10T06:06:11.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:11.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:11.013703+0000) 2022-09-10T06:06:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:11.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:11.419424+0000) 2022-09-10T06:06:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:11 smithi104 ceph-mon[97044]: pgmap v3107: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:06:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:11 smithi110 ceph-mon[97633]: pgmap v3107: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:06:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:12.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:12.013915+0000) 2022-09-10T06:06:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:12.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:12.419640+0000) 2022-09-10T06:06:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:06:12.566+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:06:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:13.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:13.014178+0000) 2022-09-10T06:06:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:13.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:13.419868+0000) 2022-09-10T06:06:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:14.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:14.014321+0000) 2022-09-10T06:06:14.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:14 smithi104 ceph-mon[97044]: osdmap e277: 12 total, 12 up, 12 in 2022-09-10T06:06:14.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:14 smithi104 ceph-mon[97044]: pgmap v3109: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:06:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:14.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:14.420067+0000) 2022-09-10T06:06:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:14 smithi110 ceph-mon[97633]: osdmap e277: 12 total, 12 up, 12 in 2022-09-10T06:06:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:14 smithi110 ceph-mon[97633]: pgmap v3109: 129 pgs: 129 active+clean; 8.0 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:06:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:15.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:15.014458+0000) 2022-09-10T06:06:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:15.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:15.420269+0000) 2022-09-10T06:06:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:15 smithi104 ceph-mon[97044]: pgmap v3110: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T06:06:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:15 smithi110 ceph-mon[97633]: pgmap v3110: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T06:06:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:16.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:16.014640+0000) 2022-09-10T06:06:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:16.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:16.420493+0000) 2022-09-10T06:06:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:17.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:17.014843+0000) 2022-09-10T06:06:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:17.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:17.420732+0000) 2022-09-10T06:06:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:17 smithi104 ceph-mon[97044]: pgmap v3111: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:06:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:17 smithi110 ceph-mon[97633]: pgmap v3111: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:06:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:18.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:18.015032+0000) 2022-09-10T06:06:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:18.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:18.420977+0000) 2022-09-10T06:06:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:19.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:19.015190+0000) 2022-09-10T06:06:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:19.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:19.421164+0000) 2022-09-10T06:06:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:19 smithi104 ceph-mon[97044]: pgmap v3112: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:06:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:19 smithi110 ceph-mon[97633]: pgmap v3112: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:06:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:20.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:20.015455+0000) 2022-09-10T06:06:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:20.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:20.421299+0000) 2022-09-10T06:06:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:21.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:21.015649+0000) 2022-09-10T06:06:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:21.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:21.421548+0000) 2022-09-10T06:06:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:21 smithi104 ceph-mon[97044]: pgmap v3113: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:06:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:21 smithi110 ceph-mon[97633]: pgmap v3113: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:06:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:22.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:22.015872+0000) 2022-09-10T06:06:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:22.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:22.421771+0000) 2022-09-10T06:06:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:23.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:23.016128+0000) 2022-09-10T06:06:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:23.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:23.421998+0000) 2022-09-10T06:06:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:24.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:24.016387+0000) 2022-09-10T06:06:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:24 smithi104 ceph-mon[97044]: pgmap v3114: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:06:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:24.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:24.422257+0000) 2022-09-10T06:06:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:24 smithi110 ceph-mon[97633]: pgmap v3114: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:06:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:25.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:25.016619+0000) 2022-09-10T06:06:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:25.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:25.422433+0000) 2022-09-10T06:06:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:25 smithi104 ceph-mon[97044]: pgmap v3115: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:06:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:25 smithi110 ceph-mon[97633]: pgmap v3115: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:06:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:26.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:26.016896+0000) 2022-09-10T06:06:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:26.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:26.422704+0000) 2022-09-10T06:06:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:27.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:27.017140+0000) 2022-09-10T06:06:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:27.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:27.422880+0000) 2022-09-10T06:06:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:06:27.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:06:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:27 smithi104 ceph-mon[97044]: pgmap v3116: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:27 smithi110 ceph-mon[97633]: pgmap v3116: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:28.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:28.017345+0000) 2022-09-10T06:06:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:28.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:28.423062+0000) 2022-09-10T06:06:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:29.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:29.017521+0000) 2022-09-10T06:06:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:29.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:29.423245+0000) 2022-09-10T06:06:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:29 smithi104 ceph-mon[97044]: pgmap v3117: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:29 smithi110 ceph-mon[97633]: pgmap v3117: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:30.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:30.017736+0000) 2022-09-10T06:06:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:30.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:30.423413+0000) 2022-09-10T06:06:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:31.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:31.017929+0000) 2022-09-10T06:06:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:31.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:31.423638+0000) 2022-09-10T06:06:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:31 smithi104 ceph-mon[97044]: pgmap v3118: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:31 smithi110 ceph-mon[97633]: pgmap v3118: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:32.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:32.018155+0000) 2022-09-10T06:06:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:32.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:32.423846+0000) 2022-09-10T06:06:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:33.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:33.018359+0000) 2022-09-10T06:06:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:33.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:33.424070+0000) 2022-09-10T06:06:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:34.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:34.018602+0000) 2022-09-10T06:06:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:34 smithi104 ceph-mon[97044]: pgmap v3119: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:34.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:34.424361+0000) 2022-09-10T06:06:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:34 smithi110 ceph-mon[97633]: pgmap v3119: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:35.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:35.018790+0000) 2022-09-10T06:06:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:35.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:35.424549+0000) 2022-09-10T06:06:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:35 smithi104 ceph-mon[97044]: pgmap v3120: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:35 smithi110 ceph-mon[97633]: pgmap v3120: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:36.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:36.019060+0000) 2022-09-10T06:06:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:36.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:36.424732+0000) 2022-09-10T06:06:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:37.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:37.019325+0000) 2022-09-10T06:06:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:37.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:37.424977+0000) 2022-09-10T06:06:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:37 smithi104 ceph-mon[97044]: pgmap v3121: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:37 smithi110 ceph-mon[97633]: pgmap v3121: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:38.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:38.019503+0000) 2022-09-10T06:06:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:38.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:38.425199+0000) 2022-09-10T06:06:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:39.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:39.019742+0000) 2022-09-10T06:06:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:39.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:39.425377+0000) 2022-09-10T06:06:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:39 smithi104 ceph-mon[97044]: pgmap v3122: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:40.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:39 smithi110 ceph-mon[97633]: pgmap v3122: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:40.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:40.020050+0000) 2022-09-10T06:06:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:40.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:40.425593+0000) 2022-09-10T06:06:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:41.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:41.020256+0000) 2022-09-10T06:06:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:41.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:41.425792+0000) 2022-09-10T06:06:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:41 smithi104 ceph-mon[97044]: pgmap v3123: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:41 smithi110 ceph-mon[97633]: pgmap v3123: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:42.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:42.020422+0000) 2022-09-10T06:06:42.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:42.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:42.426005+0000) 2022-09-10T06:06:42.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:06:42.587+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:06:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:43.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:43.020593+0000) 2022-09-10T06:06:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:43.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:43.426209+0000) 2022-09-10T06:06:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:44.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:44.020749+0000) 2022-09-10T06:06:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:44 smithi104 ceph-mon[97044]: pgmap v3124: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:06:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:06:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:06:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:06:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:44.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:44.426508+0000) 2022-09-10T06:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:44 smithi110 ceph-mon[97633]: pgmap v3124: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:06:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:06:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:06:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:06:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:45.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:45.020986+0000) 2022-09-10T06:06:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:45.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:45.426728+0000) 2022-09-10T06:06:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:45 smithi104 ceph-mon[97044]: pgmap v3125: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:45 smithi110 ceph-mon[97633]: pgmap v3125: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:46.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:46.021168+0000) 2022-09-10T06:06:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:46.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:46.426988+0000) 2022-09-10T06:06:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:47.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:47.021372+0000) 2022-09-10T06:06:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:47.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:47.427238+0000) 2022-09-10T06:06:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:47 smithi104 ceph-mon[97044]: pgmap v3126: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:47 smithi110 ceph-mon[97633]: pgmap v3126: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:48.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:48.021560+0000) 2022-09-10T06:06:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:48.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:48.427477+0000) 2022-09-10T06:06:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:49.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:49.021805+0000) 2022-09-10T06:06:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:49.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:49.427723+0000) 2022-09-10T06:06:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:49 smithi104 ceph-mon[97044]: pgmap v3127: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:49 smithi110 ceph-mon[97633]: pgmap v3127: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:50.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:50.022037+0000) 2022-09-10T06:06:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:50.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:50.427970+0000) 2022-09-10T06:06:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:51.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:51.022225+0000) 2022-09-10T06:06:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:51.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:51.428173+0000) 2022-09-10T06:06:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:51 smithi104 ceph-mon[97044]: pgmap v3128: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:51 smithi110 ceph-mon[97633]: pgmap v3128: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:52.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:52.022361+0000) 2022-09-10T06:06:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:52.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:52.428414+0000) 2022-09-10T06:06:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:53.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:53.022530+0000) 2022-09-10T06:06:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:53.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:53.428602+0000) 2022-09-10T06:06:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:54.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:54.022758+0000) 2022-09-10T06:06:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:54 smithi104 ceph-mon[97044]: pgmap v3129: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:54.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:54.428817+0000) 2022-09-10T06:06:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:54 smithi110 ceph-mon[97633]: pgmap v3129: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:55.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:55.022946+0000) 2022-09-10T06:06:55.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:55.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:55.429005+0000) 2022-09-10T06:06:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:55 smithi104 ceph-mon[97044]: pgmap v3130: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:55 smithi110 ceph-mon[97633]: pgmap v3130: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:06:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:56.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:56.023196+0000) 2022-09-10T06:06:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:56.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:56.429248+0000) 2022-09-10T06:06:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:57.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:57.023446+0000) 2022-09-10T06:06:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:57.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:57.429483+0000) 2022-09-10T06:06:57.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:06:57.592+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:06:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:57 smithi104 ceph-mon[97044]: pgmap v3131: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:57 smithi110 ceph-mon[97633]: pgmap v3131: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:06:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:58.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:58.023700+0000) 2022-09-10T06:06:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:58.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:58.429733+0000) 2022-09-10T06:06:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:06:59.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:59.023968+0000) 2022-09-10T06:06:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:06:59.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:06:59.429978+0000) 2022-09-10T06:07:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:06:59 smithi104 ceph-mon[97044]: pgmap v3132: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:06:59 smithi110 ceph-mon[97633]: pgmap v3132: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:00.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:00.024269+0000) 2022-09-10T06:07:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:00.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:00.430167+0000) 2022-09-10T06:07:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:01.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:01.024521+0000) 2022-09-10T06:07:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:01.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:01.430387+0000) 2022-09-10T06:07:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:01 smithi104 ceph-mon[97044]: pgmap v3133: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:01 smithi110 ceph-mon[97633]: pgmap v3133: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:02.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:02.024721+0000) 2022-09-10T06:07:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:02.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:02.430666+0000) 2022-09-10T06:07:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:03.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:03.024948+0000) 2022-09-10T06:07:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:03.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:03.430880+0000) 2022-09-10T06:07:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:04.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:04.025205+0000) 2022-09-10T06:07:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:04 smithi104 ceph-mon[97044]: pgmap v3134: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:04.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:04.431145+0000) 2022-09-10T06:07:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:04 smithi110 ceph-mon[97633]: pgmap v3134: 129 pgs: 129 active+clean; 8.0 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:05.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:05.025476+0000) 2022-09-10T06:07:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:05.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:05.431274+0000) 2022-09-10T06:07:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:05 smithi104 ceph-mon[97044]: pgmap v3135: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:07:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:05 smithi110 ceph-mon[97633]: pgmap v3135: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:07:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:06.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:06.025633+0000) 2022-09-10T06:07:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:06.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:06.431450+0000) 2022-09-10T06:07:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:07.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:07.025852+0000) 2022-09-10T06:07:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:07.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:07.431721+0000) 2022-09-10T06:07:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:08.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:08.026125+0000) 2022-09-10T06:07:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:08 smithi104 ceph-mon[97044]: osdmap e278: 12 total, 12 up, 12 in 2022-09-10T06:07:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:08 smithi104 ceph-mon[97044]: pgmap v3137: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:07:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:08 smithi110 ceph-mon[97633]: osdmap e278: 12 total, 12 up, 12 in 2022-09-10T06:07:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:08 smithi110 ceph-mon[97633]: pgmap v3137: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:07:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:08.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:08.431983+0000) 2022-09-10T06:07:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:09.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:09.026255+0000) 2022-09-10T06:07:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:09.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:09.432232+0000) 2022-09-10T06:07:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:09 smithi104 ceph-mon[97044]: pgmap v3138: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:07:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:09 smithi110 ceph-mon[97633]: pgmap v3138: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:07:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:10.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:10.026481+0000) 2022-09-10T06:07:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:10.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:10.432500+0000) 2022-09-10T06:07:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:11.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:11.026741+0000) 2022-09-10T06:07:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:11.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:11.432718+0000) 2022-09-10T06:07:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:11 smithi104 ceph-mon[97044]: pgmap v3139: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:07:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:11 smithi110 ceph-mon[97633]: pgmap v3139: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:07:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:12.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:12.026954+0000) 2022-09-10T06:07:12.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:07:12.603+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:07:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:12.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:12.432916+0000) 2022-09-10T06:07:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:13.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:13.027199+0000) 2022-09-10T06:07:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:13.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:13.433157+0000) 2022-09-10T06:07:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:14.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:14.027451+0000) 2022-09-10T06:07:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:14 smithi104 ceph-mon[97044]: osdmap e279: 12 total, 12 up, 12 in 2022-09-10T06:07:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:14 smithi104 ceph-mon[97044]: pgmap v3141: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:07:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:14.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:14.433332+0000) 2022-09-10T06:07:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:14 smithi110 ceph-mon[97633]: osdmap e279: 12 total, 12 up, 12 in 2022-09-10T06:07:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:14 smithi110 ceph-mon[97633]: pgmap v3141: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:07:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:15.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:15.027675+0000) 2022-09-10T06:07:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:15.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:15.433502+0000) 2022-09-10T06:07:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:15 smithi104 ceph-mon[97044]: pgmap v3142: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T06:07:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:15 smithi110 ceph-mon[97633]: pgmap v3142: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T06:07:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:16.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:16.027934+0000) 2022-09-10T06:07:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:16.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:16.433732+0000) 2022-09-10T06:07:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:17.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:17.028106+0000) 2022-09-10T06:07:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:17.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:17.433953+0000) 2022-09-10T06:07:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:17 smithi104 ceph-mon[97044]: pgmap v3143: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:07:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:17 smithi110 ceph-mon[97633]: pgmap v3143: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:07:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:18.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:18.028315+0000) 2022-09-10T06:07:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:18.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:18.434135+0000) 2022-09-10T06:07:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:19.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:19.028541+0000) 2022-09-10T06:07:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:19.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:19.434305+0000) 2022-09-10T06:07:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:19 smithi104 ceph-mon[97044]: pgmap v3144: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:07:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:19 smithi110 ceph-mon[97633]: pgmap v3144: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:07:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:20.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:20.028806+0000) 2022-09-10T06:07:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:20.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:20.434539+0000) 2022-09-10T06:07:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:21.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:21.028952+0000) 2022-09-10T06:07:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:21.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:21.434774+0000) 2022-09-10T06:07:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:21 smithi104 ceph-mon[97044]: pgmap v3145: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:07:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:21 smithi110 ceph-mon[97633]: pgmap v3145: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:07:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:22.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:22.029192+0000) 2022-09-10T06:07:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:22.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:22.434986+0000) 2022-09-10T06:07:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:23.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:23.029477+0000) 2022-09-10T06:07:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:23.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:23.435179+0000) 2022-09-10T06:07:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:24.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:24.029668+0000) 2022-09-10T06:07:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:24 smithi104 ceph-mon[97044]: pgmap v3146: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T06:07:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:24.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:24.435409+0000) 2022-09-10T06:07:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:24 smithi110 ceph-mon[97633]: pgmap v3146: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T06:07:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:25.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:25.029874+0000) 2022-09-10T06:07:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:25.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:25.435634+0000) 2022-09-10T06:07:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:25 smithi104 ceph-mon[97044]: pgmap v3147: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:07:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:25 smithi110 ceph-mon[97633]: pgmap v3147: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:07:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:26.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:26.030158+0000) 2022-09-10T06:07:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:26.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:26.435872+0000) 2022-09-10T06:07:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:27.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:27.030409+0000) 2022-09-10T06:07:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:27.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:27.436120+0000) 2022-09-10T06:07:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:07:27.618+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:07:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:27 smithi104 ceph-mon[97044]: pgmap v3148: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:27 smithi110 ceph-mon[97633]: pgmap v3148: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:28.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:28.030659+0000) 2022-09-10T06:07:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:28.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:28.436269+0000) 2022-09-10T06:07:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:29.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:29.030882+0000) 2022-09-10T06:07:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:29.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:29.436468+0000) 2022-09-10T06:07:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:29 smithi104 ceph-mon[97044]: pgmap v3149: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:29 smithi110 ceph-mon[97633]: pgmap v3149: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:30.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:30.031120+0000) 2022-09-10T06:07:30.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:30.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:30.436691+0000) 2022-09-10T06:07:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:31.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:31.031338+0000) 2022-09-10T06:07:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:31.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:31.436918+0000) 2022-09-10T06:07:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:31 smithi104 ceph-mon[97044]: pgmap v3150: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:31 smithi110 ceph-mon[97633]: pgmap v3150: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:32.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:32.031513+0000) 2022-09-10T06:07:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:32.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:32.437176+0000) 2022-09-10T06:07:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:33.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:33.031723+0000) 2022-09-10T06:07:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:33.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:33.437355+0000) 2022-09-10T06:07:34.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:34.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:34.031944+0000) 2022-09-10T06:07:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:34 smithi104 ceph-mon[97044]: pgmap v3151: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:34.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:34.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:34.437586+0000) 2022-09-10T06:07:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:34 smithi110 ceph-mon[97633]: pgmap v3151: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:35.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:35.032160+0000) 2022-09-10T06:07:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:35.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:35.437792+0000) 2022-09-10T06:07:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:35 smithi104 ceph-mon[97044]: pgmap v3152: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:35 smithi110 ceph-mon[97633]: pgmap v3152: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:36.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:36.032378+0000) 2022-09-10T06:07:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:36.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:36.438028+0000) 2022-09-10T06:07:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:37.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:37.032626+0000) 2022-09-10T06:07:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:37.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:37.438217+0000) 2022-09-10T06:07:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:37 smithi104 ceph-mon[97044]: pgmap v3153: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:37 smithi110 ceph-mon[97633]: pgmap v3153: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:38.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:38.032896+0000) 2022-09-10T06:07:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:38.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:38.438442+0000) 2022-09-10T06:07:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:39.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:39.033082+0000) 2022-09-10T06:07:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:39.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:39.438675+0000) 2022-09-10T06:07:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:39 smithi104 ceph-mon[97044]: pgmap v3154: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:39 smithi110 ceph-mon[97633]: pgmap v3154: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:40.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:40.033346+0000) 2022-09-10T06:07:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:40.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:40.438941+0000) 2022-09-10T06:07:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:41.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:41.033573+0000) 2022-09-10T06:07:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:41.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:41.439226+0000) 2022-09-10T06:07:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:41 smithi104 ceph-mon[97044]: pgmap v3155: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:41 smithi110 ceph-mon[97633]: pgmap v3155: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:42.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:42.033788+0000) 2022-09-10T06:07:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:42.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:42.439510+0000) 2022-09-10T06:07:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:07:42.627+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:07:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:43.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:43.034040+0000) 2022-09-10T06:07:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:43.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:43.439772+0000) 2022-09-10T06:07:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:44.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:44.034310+0000) 2022-09-10T06:07:44.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:44 smithi104 ceph-mon[97044]: pgmap v3156: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:44.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:07:44.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:07:44.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:07:44.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:07:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:44.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:44.440002+0000) 2022-09-10T06:07:44.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:44 smithi110 ceph-mon[97633]: pgmap v3156: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:44.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:07:44.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:07:44.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:07:44.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:07:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:45.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:45.034522+0000) 2022-09-10T06:07:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:45.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:45.440266+0000) 2022-09-10T06:07:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:45 smithi104 ceph-mon[97044]: pgmap v3157: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:45 smithi110 ceph-mon[97633]: pgmap v3157: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:46.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:46.034754+0000) 2022-09-10T06:07:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:46.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:46.440453+0000) 2022-09-10T06:07:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:47.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:47.034970+0000) 2022-09-10T06:07:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:47.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:47.440709+0000) 2022-09-10T06:07:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:47 smithi104 ceph-mon[97044]: pgmap v3158: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:47 smithi110 ceph-mon[97633]: pgmap v3158: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:48.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:48.035201+0000) 2022-09-10T06:07:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:48.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:48.440881+0000) 2022-09-10T06:07:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:49.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:49.035364+0000) 2022-09-10T06:07:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:49.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:49.441160+0000) 2022-09-10T06:07:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:49 smithi104 ceph-mon[97044]: pgmap v3159: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:49 smithi110 ceph-mon[97633]: pgmap v3159: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:50.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:50.035582+0000) 2022-09-10T06:07:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:50.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:50.441361+0000) 2022-09-10T06:07:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:51.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:51.035773+0000) 2022-09-10T06:07:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:51.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:51.441555+0000) 2022-09-10T06:07:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:51 smithi104 ceph-mon[97044]: pgmap v3160: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:51 smithi110 ceph-mon[97633]: pgmap v3160: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:52.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:52.035903+0000) 2022-09-10T06:07:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:52.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:52.441735+0000) 2022-09-10T06:07:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:53.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:53.036066+0000) 2022-09-10T06:07:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:53.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:53.441970+0000) 2022-09-10T06:07:54.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:54.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:54.036283+0000) 2022-09-10T06:07:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:54 smithi104 ceph-mon[97044]: pgmap v3161: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:54.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:54.442217+0000) 2022-09-10T06:07:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:54 smithi110 ceph-mon[97633]: pgmap v3161: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:55.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:55.036505+0000) 2022-09-10T06:07:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:55.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:55.442371+0000) 2022-09-10T06:07:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:55 smithi104 ceph-mon[97044]: pgmap v3162: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:55 smithi110 ceph-mon[97633]: pgmap v3162: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:07:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:56.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:56.036736+0000) 2022-09-10T06:07:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:56.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:56.442559+0000) 2022-09-10T06:07:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:57.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:57.037041+0000) 2022-09-10T06:07:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:57.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:57.442762+0000) 2022-09-10T06:07:57.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:07:57.636+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:07:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:57 smithi104 ceph-mon[97044]: pgmap v3163: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:57 smithi110 ceph-mon[97633]: pgmap v3163: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:07:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:58.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:58.037299+0000) 2022-09-10T06:07:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:58.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:58.442952+0000) 2022-09-10T06:07:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:07:59.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:59.037484+0000) 2022-09-10T06:07:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:07:59.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:07:59.443244+0000) 2022-09-10T06:08:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:07:59 smithi104 ceph-mon[97044]: pgmap v3164: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:07:59 smithi110 ceph-mon[97633]: pgmap v3164: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:00.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:00.037728+0000) 2022-09-10T06:08:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:00.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:00.443511+0000) 2022-09-10T06:08:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:01.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:01.037999+0000) 2022-09-10T06:08:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:01.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:01.443729+0000) 2022-09-10T06:08:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:01 smithi104 ceph-mon[97044]: pgmap v3165: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:01 smithi110 ceph-mon[97633]: pgmap v3165: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:02.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:02.038209+0000) 2022-09-10T06:08:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:02.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:02.443900+0000) 2022-09-10T06:08:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:03.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:03.038408+0000) 2022-09-10T06:08:03.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:03.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:03.444152+0000) 2022-09-10T06:08:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:04.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:04.038636+0000) 2022-09-10T06:08:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:04 smithi104 ceph-mon[97044]: pgmap v3166: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:04.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:04.444386+0000) 2022-09-10T06:08:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:04 smithi110 ceph-mon[97633]: pgmap v3166: 129 pgs: 129 active+clean; 8.0 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:05.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:05.038806+0000) 2022-09-10T06:08:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:05.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:05.444650+0000) 2022-09-10T06:08:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:05 smithi104 ceph-mon[97044]: pgmap v3167: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:08:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:05 smithi110 ceph-mon[97633]: pgmap v3167: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:08:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:06.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:06.038961+0000) 2022-09-10T06:08:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:06.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:06.444894+0000) 2022-09-10T06:08:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:07.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:07.039231+0000) 2022-09-10T06:08:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:07.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:07.445154+0000) 2022-09-10T06:08:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:08.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:08.039427+0000) 2022-09-10T06:08:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:08 smithi104 ceph-mon[97044]: osdmap e280: 12 total, 12 up, 12 in 2022-09-10T06:08:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:08 smithi104 ceph-mon[97044]: pgmap v3169: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:08:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:08 smithi110 ceph-mon[97633]: osdmap e280: 12 total, 12 up, 12 in 2022-09-10T06:08:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:08 smithi110 ceph-mon[97633]: pgmap v3169: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:08:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:08.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:08.445375+0000) 2022-09-10T06:08:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:09.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:09.039582+0000) 2022-09-10T06:08:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:09.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:09.445611+0000) 2022-09-10T06:08:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:09 smithi104 ceph-mon[97044]: pgmap v3170: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:08:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:09 smithi110 ceph-mon[97633]: pgmap v3170: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:08:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:10.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:10.039816+0000) 2022-09-10T06:08:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:10.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:10.445833+0000) 2022-09-10T06:08:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:11.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:11.040050+0000) 2022-09-10T06:08:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:11.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:11.446023+0000) 2022-09-10T06:08:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:11 smithi104 ceph-mon[97044]: pgmap v3171: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:08:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:11 smithi110 ceph-mon[97633]: pgmap v3171: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:08:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:12.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:12.040327+0000) 2022-09-10T06:08:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:12.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:12.446237+0000) 2022-09-10T06:08:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:08:12.642+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:08:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:13.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:13.040593+0000) 2022-09-10T06:08:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:13.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:13.446472+0000) 2022-09-10T06:08:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:14.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:14.040847+0000) 2022-09-10T06:08:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:14 smithi104 ceph-mon[97044]: osdmap e281: 12 total, 12 up, 12 in 2022-09-10T06:08:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:14 smithi104 ceph-mon[97044]: pgmap v3173: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:08:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:14.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:14.446731+0000) 2022-09-10T06:08:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:14 smithi110 ceph-mon[97633]: osdmap e281: 12 total, 12 up, 12 in 2022-09-10T06:08:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:14 smithi110 ceph-mon[97633]: pgmap v3173: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:08:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:15.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:15.041138+0000) 2022-09-10T06:08:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:15.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:15.446900+0000) 2022-09-10T06:08:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:15 smithi104 ceph-mon[97044]: pgmap v3174: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T06:08:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:15 smithi110 ceph-mon[97633]: pgmap v3174: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T06:08:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:16.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:16.041323+0000) 2022-09-10T06:08:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:16.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:16.447167+0000) 2022-09-10T06:08:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:17.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:17.041542+0000) 2022-09-10T06:08:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:17.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:17.447395+0000) 2022-09-10T06:08:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:17 smithi104 ceph-mon[97044]: pgmap v3175: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:08:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:17 smithi110 ceph-mon[97633]: pgmap v3175: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:08:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:18.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:18.041790+0000) 2022-09-10T06:08:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:18.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:18.447633+0000) 2022-09-10T06:08:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:19.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:19.042006+0000) 2022-09-10T06:08:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:19.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:19.447815+0000) 2022-09-10T06:08:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:19 smithi104 ceph-mon[97044]: pgmap v3176: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:08:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:19 smithi110 ceph-mon[97633]: pgmap v3176: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:08:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:20.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:20.042267+0000) 2022-09-10T06:08:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:20.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:20.448036+0000) 2022-09-10T06:08:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:21.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:21.042393+0000) 2022-09-10T06:08:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:21.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:21.448282+0000) 2022-09-10T06:08:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:21 smithi104 ceph-mon[97044]: pgmap v3177: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:08:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:21 smithi110 ceph-mon[97633]: pgmap v3177: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:08:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:22.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:22.042617+0000) 2022-09-10T06:08:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:22.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:22.448488+0000) 2022-09-10T06:08:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:23.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:23.042821+0000) 2022-09-10T06:08:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:23.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:23.448672+0000) 2022-09-10T06:08:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:24.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:24.043064+0000) 2022-09-10T06:08:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:24 smithi104 ceph-mon[97044]: pgmap v3178: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T06:08:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:24.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:24.448859+0000) 2022-09-10T06:08:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:24 smithi110 ceph-mon[97633]: pgmap v3178: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T06:08:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:25.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:25.043304+0000) 2022-09-10T06:08:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:25.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:25.449077+0000) 2022-09-10T06:08:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:25 smithi104 ceph-mon[97044]: pgmap v3179: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:08:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:25 smithi110 ceph-mon[97633]: pgmap v3179: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:08:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:26.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:26.043548+0000) 2022-09-10T06:08:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:26.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:26.449326+0000) 2022-09-10T06:08:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:27.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:27.043688+0000) 2022-09-10T06:08:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:27.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:27.449552+0000) 2022-09-10T06:08:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:08:27.654+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:08:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:27 smithi104 ceph-mon[97044]: pgmap v3180: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:27 smithi110 ceph-mon[97633]: pgmap v3180: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:28.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:28.043879+0000) 2022-09-10T06:08:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:28.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:28.449759+0000) 2022-09-10T06:08:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:29.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:29.044021+0000) 2022-09-10T06:08:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:29.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:29.449973+0000) 2022-09-10T06:08:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:29 smithi104 ceph-mon[97044]: pgmap v3181: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:29 smithi110 ceph-mon[97633]: pgmap v3181: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:30.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:30.044209+0000) 2022-09-10T06:08:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:30.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:30.450207+0000) 2022-09-10T06:08:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:31.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:31.044432+0000) 2022-09-10T06:08:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:31.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:31.450398+0000) 2022-09-10T06:08:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:31 smithi104 ceph-mon[97044]: pgmap v3182: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:31 smithi110 ceph-mon[97633]: pgmap v3182: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:32.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:32.044661+0000) 2022-09-10T06:08:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:32.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:32.450611+0000) 2022-09-10T06:08:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:33.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:33.044866+0000) 2022-09-10T06:08:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:33.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:33.450838+0000) 2022-09-10T06:08:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:34.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:34.045130+0000) 2022-09-10T06:08:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:34 smithi104 ceph-mon[97044]: pgmap v3183: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:34.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:34.451141+0000) 2022-09-10T06:08:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:34 smithi110 ceph-mon[97633]: pgmap v3183: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:35.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:35.045391+0000) 2022-09-10T06:08:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:35.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:35.451369+0000) 2022-09-10T06:08:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:35 smithi104 ceph-mon[97044]: pgmap v3184: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:35 smithi110 ceph-mon[97633]: pgmap v3184: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:36.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:36.045568+0000) 2022-09-10T06:08:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:36.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:36.451580+0000) 2022-09-10T06:08:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:37.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:37.045750+0000) 2022-09-10T06:08:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:37.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:37.451791+0000) 2022-09-10T06:08:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:37 smithi104 ceph-mon[97044]: pgmap v3185: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:37 smithi110 ceph-mon[97633]: pgmap v3185: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:38.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:38.046025+0000) 2022-09-10T06:08:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:38.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:38.452055+0000) 2022-09-10T06:08:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:39.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:39.046242+0000) 2022-09-10T06:08:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:39.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:39.452267+0000) 2022-09-10T06:08:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:39 smithi104 ceph-mon[97044]: pgmap v3186: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:39 smithi110 ceph-mon[97633]: pgmap v3186: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:40.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:40.046459+0000) 2022-09-10T06:08:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:40.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:40.452520+0000) 2022-09-10T06:08:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:41.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:41.046713+0000) 2022-09-10T06:08:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:41.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:41.452735+0000) 2022-09-10T06:08:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:41 smithi104 ceph-mon[97044]: pgmap v3187: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:41 smithi110 ceph-mon[97633]: pgmap v3187: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:42.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:42.046984+0000) 2022-09-10T06:08:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:42.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:42.452957+0000) 2022-09-10T06:08:42.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:08:42.669+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:08:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:43.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:43.047271+0000) 2022-09-10T06:08:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:43.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:43.453199+0000) 2022-09-10T06:08:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:44.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:44.047383+0000) 2022-09-10T06:08:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:44 smithi104 ceph-mon[97044]: pgmap v3188: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:08:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:08:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:08:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:08:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:44.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:44.453457+0000) 2022-09-10T06:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:44 smithi110 ceph-mon[97633]: pgmap v3188: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:08:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:08:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:08:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:08:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:45.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:45.047606+0000) 2022-09-10T06:08:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:45.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:45.453708+0000) 2022-09-10T06:08:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:45 smithi104 ceph-mon[97044]: pgmap v3189: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:45 smithi110 ceph-mon[97633]: pgmap v3189: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:46.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:46.047805+0000) 2022-09-10T06:08:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:46.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:46.454046+0000) 2022-09-10T06:08:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:47.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:47.048071+0000) 2022-09-10T06:08:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:47.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:47.454279+0000) 2022-09-10T06:08:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:47 smithi104 ceph-mon[97044]: pgmap v3190: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:47 smithi110 ceph-mon[97633]: pgmap v3190: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:48.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:48.048378+0000) 2022-09-10T06:08:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:48.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:48.454499+0000) 2022-09-10T06:08:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:49.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:49.048617+0000) 2022-09-10T06:08:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:49.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:49.454686+0000) 2022-09-10T06:08:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:49 smithi104 ceph-mon[97044]: pgmap v3191: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:49 smithi110 ceph-mon[97633]: pgmap v3191: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:50.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:50.048867+0000) 2022-09-10T06:08:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:50.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:50.454904+0000) 2022-09-10T06:08:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:51.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:51.049170+0000) 2022-09-10T06:08:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:51.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:51.455173+0000) 2022-09-10T06:08:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:51 smithi104 ceph-mon[97044]: pgmap v3192: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:51 smithi110 ceph-mon[97633]: pgmap v3192: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:52.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:52.049441+0000) 2022-09-10T06:08:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:52.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:52.455384+0000) 2022-09-10T06:08:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:53.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:53.049690+0000) 2022-09-10T06:08:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:53.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:53.455576+0000) 2022-09-10T06:08:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:54.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:54.049978+0000) 2022-09-10T06:08:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:54 smithi104 ceph-mon[97044]: pgmap v3193: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:54.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:54.455783+0000) 2022-09-10T06:08:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:54 smithi110 ceph-mon[97633]: pgmap v3193: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:55.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:55.050191+0000) 2022-09-10T06:08:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:55.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:55.455995+0000) 2022-09-10T06:08:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:55 smithi104 ceph-mon[97044]: pgmap v3194: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:55 smithi110 ceph-mon[97633]: pgmap v3194: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:08:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:56.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:56.050404+0000) 2022-09-10T06:08:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:56.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:56.456253+0000) 2022-09-10T06:08:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:57.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:57.050646+0000) 2022-09-10T06:08:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:08:57.685+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:08:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:57.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:57.456415+0000) 2022-09-10T06:08:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:08:57 smithi104 ceph-mon[97044]: pgmap v3195: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:08:57 smithi110 ceph-mon[97633]: pgmap v3195: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:08:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:58.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:58.050790+0000) 2022-09-10T06:08:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:58.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:58.456649+0000) 2022-09-10T06:08:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:08:59.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:59.050994+0000) 2022-09-10T06:08:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:08:59.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:08:59.456886+0000) 2022-09-10T06:09:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:00.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:00.051234+0000) 2022-09-10T06:09:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:00.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:00.457048+0000) 2022-09-10T06:09:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:00 smithi104 ceph-mon[97044]: pgmap v3196: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:00 smithi110 ceph-mon[97633]: pgmap v3196: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:01.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:01.051434+0000) 2022-09-10T06:09:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:01.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:01.457261+0000) 2022-09-10T06:09:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:02.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:02.051671+0000) 2022-09-10T06:09:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:02.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:02.457453+0000) 2022-09-10T06:09:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:02 smithi104 ceph-mon[97044]: pgmap v3197: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:02 smithi110 ceph-mon[97633]: pgmap v3197: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:03.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:03.051942+0000) 2022-09-10T06:09:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:03.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:03.457618+0000) 2022-09-10T06:09:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:04.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:04.052192+0000) 2022-09-10T06:09:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:04.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:04.457856+0000) 2022-09-10T06:09:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:05.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:05.052339+0000) 2022-09-10T06:09:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:05 smithi104 ceph-mon[97044]: pgmap v3198: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:05.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:05.458086+0000) 2022-09-10T06:09:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:05 smithi110 ceph-mon[97633]: pgmap v3198: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:06.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:06.052492+0000) 2022-09-10T06:09:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:06.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:06.458371+0000) 2022-09-10T06:09:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:07.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:07.052699+0000) 2022-09-10T06:09:07.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:07 smithi104 ceph-mon[97044]: pgmap v3199: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:09:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:07.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:07.458514+0000) 2022-09-10T06:09:07.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:07 smithi110 ceph-mon[97633]: pgmap v3199: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:09:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:08.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:08.052952+0000) 2022-09-10T06:09:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:08 smithi104 ceph-mon[97044]: osdmap e282: 12 total, 12 up, 12 in 2022-09-10T06:09:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:08.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:08.458730+0000) 2022-09-10T06:09:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:08 smithi110 ceph-mon[97633]: osdmap e282: 12 total, 12 up, 12 in 2022-09-10T06:09:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:09.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:09.053184+0000) 2022-09-10T06:09:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:09 smithi104 ceph-mon[97044]: pgmap v3201: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:09:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:09.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:09.458972+0000) 2022-09-10T06:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:09 smithi110 ceph-mon[97633]: pgmap v3201: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:09:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:10.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:10.053446+0000) 2022-09-10T06:09:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:10.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:10.459222+0000) 2022-09-10T06:09:11.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:11.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:11.053642+0000) 2022-09-10T06:09:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:11 smithi104 ceph-mon[97044]: pgmap v3202: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:09:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:11.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:11.459435+0000) 2022-09-10T06:09:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:11 smithi110 ceph-mon[97633]: pgmap v3202: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:09:12.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:12.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:12.053794+0000) 2022-09-10T06:09:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:12.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:12.459698+0000) 2022-09-10T06:09:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:09:12.700+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:09:13.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:13.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:13.054000+0000) 2022-09-10T06:09:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:13 smithi104 ceph-mon[97044]: pgmap v3203: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:09:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:13 smithi104 ceph-mon[97044]: osdmap e283: 12 total, 12 up, 12 in 2022-09-10T06:09:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:13.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:13.459857+0000) 2022-09-10T06:09:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:13 smithi110 ceph-mon[97633]: pgmap v3203: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:09:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:13 smithi110 ceph-mon[97633]: osdmap e283: 12 total, 12 up, 12 in 2022-09-10T06:09:14.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:14.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:14.054205+0000) 2022-09-10T06:09:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:14.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:14.460046+0000) 2022-09-10T06:09:15.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:15.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:15.054374+0000) 2022-09-10T06:09:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:15 smithi104 ceph-mon[97044]: pgmap v3205: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:09:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:15.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:15.460229+0000) 2022-09-10T06:09:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:15 smithi110 ceph-mon[97633]: pgmap v3205: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:09:16.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:16.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:16.054574+0000) 2022-09-10T06:09:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:16.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:16.460509+0000) 2022-09-10T06:09:17.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:17.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:17.054838+0000) 2022-09-10T06:09:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:17 smithi104 ceph-mon[97044]: pgmap v3206: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T06:09:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:17.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:17.460653+0000) 2022-09-10T06:09:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:17 smithi110 ceph-mon[97633]: pgmap v3206: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T06:09:18.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:18.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:18.055051+0000) 2022-09-10T06:09:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:18.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:18.460830+0000) 2022-09-10T06:09:19.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:19.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:19.055304+0000) 2022-09-10T06:09:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:19.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:19.461057+0000) 2022-09-10T06:09:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:19 smithi110 ceph-mon[97633]: pgmap v3207: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:09:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:19 smithi104 ceph-mon[97044]: pgmap v3207: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:09:20.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:20.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:20.055582+0000) 2022-09-10T06:09:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:20.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:20.461254+0000) 2022-09-10T06:09:21.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:21.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:21.055795+0000) 2022-09-10T06:09:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:21.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:21.461514+0000) 2022-09-10T06:09:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:21 smithi110 ceph-mon[97633]: pgmap v3208: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:09:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:21 smithi104 ceph-mon[97044]: pgmap v3208: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:09:22.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:22.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:22.055936+0000) 2022-09-10T06:09:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:22.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:22.461702+0000) 2022-09-10T06:09:23.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:23.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:23.056145+0000) 2022-09-10T06:09:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:23.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:23.461921+0000) 2022-09-10T06:09:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:23 smithi110 ceph-mon[97633]: pgmap v3209: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:09:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:23 smithi104 ceph-mon[97044]: pgmap v3209: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:09:24.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:24.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:24.056348+0000) 2022-09-10T06:09:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:24.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:24.462139+0000) 2022-09-10T06:09:25.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:25.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:25.056563+0000) 2022-09-10T06:09:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:25.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:25.462297+0000) 2022-09-10T06:09:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:25 smithi110 ceph-mon[97633]: pgmap v3210: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:09:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:25 smithi104 ceph-mon[97044]: pgmap v3210: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T06:09:26.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:26.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:26.056789+0000) 2022-09-10T06:09:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:26.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:26.462532+0000) 2022-09-10T06:09:27.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:27.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:27.057036+0000) 2022-09-10T06:09:27.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:09:27.707+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:09:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:27.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:27.462756+0000) 2022-09-10T06:09:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:27 smithi110 ceph-mon[97633]: pgmap v3211: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:09:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:27 smithi104 ceph-mon[97044]: pgmap v3211: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:09:28.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:28.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:28.057259+0000) 2022-09-10T06:09:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:28.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:28.462957+0000) 2022-09-10T06:09:29.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:29.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:29.057459+0000) 2022-09-10T06:09:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:29.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:29.463136+0000) 2022-09-10T06:09:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:29 smithi110 ceph-mon[97633]: pgmap v3212: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:29 smithi104 ceph-mon[97044]: pgmap v3212: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:30.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:30.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:30.057627+0000) 2022-09-10T06:09:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:30.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:30.463370+0000) 2022-09-10T06:09:31.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:31.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:31.057802+0000) 2022-09-10T06:09:31.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:31.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:31.463534+0000) 2022-09-10T06:09:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:31 smithi110 ceph-mon[97633]: pgmap v3213: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:31 smithi104 ceph-mon[97044]: pgmap v3213: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:32.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:32.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:32.058063+0000) 2022-09-10T06:09:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:32.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:32.463748+0000) 2022-09-10T06:09:33.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:33.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:33.058298+0000) 2022-09-10T06:09:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:33.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:33.463910+0000) 2022-09-10T06:09:33.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:33 smithi110 ceph-mon[97633]: pgmap v3214: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:33 smithi104 ceph-mon[97044]: pgmap v3214: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:34.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:34.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:34.058535+0000) 2022-09-10T06:09:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:34.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:34.464169+0000) 2022-09-10T06:09:35.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:35.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:35.058804+0000) 2022-09-10T06:09:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:35.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:35.464341+0000) 2022-09-10T06:09:35.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:35 smithi110 ceph-mon[97633]: pgmap v3215: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:35 smithi104 ceph-mon[97044]: pgmap v3215: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:36.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:36.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:36.059033+0000) 2022-09-10T06:09:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:36.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:36.464577+0000) 2022-09-10T06:09:37.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:37.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:37.059273+0000) 2022-09-10T06:09:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:37.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:37.464752+0000) 2022-09-10T06:09:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:37 smithi110 ceph-mon[97633]: pgmap v3216: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:37 smithi104 ceph-mon[97044]: pgmap v3216: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:38.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:38.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:38.059464+0000) 2022-09-10T06:09:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:38.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:38.464926+0000) 2022-09-10T06:09:39.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:39.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:39.059628+0000) 2022-09-10T06:09:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:39.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:39.465213+0000) 2022-09-10T06:09:39.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:39 smithi110 ceph-mon[97633]: pgmap v3217: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:39 smithi104 ceph-mon[97044]: pgmap v3217: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:40.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:40.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:40.059842+0000) 2022-09-10T06:09:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:40.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:40.465438+0000) 2022-09-10T06:09:41.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:41.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:41.060072+0000) 2022-09-10T06:09:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:41.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:41.465642+0000) 2022-09-10T06:09:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:41 smithi110 ceph-mon[97633]: pgmap v3218: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:41 smithi104 ceph-mon[97044]: pgmap v3218: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:42.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:42.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:42.060236+0000) 2022-09-10T06:09:42.719 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:42.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:42.465824+0000) 2022-09-10T06:09:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:09:42.717+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:09:43.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:43.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:43.060431+0000) 2022-09-10T06:09:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:43.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:43.466010+0000) 2022-09-10T06:09:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:43 smithi110 ceph-mon[97633]: pgmap v3219: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:43 smithi104 ceph-mon[97044]: pgmap v3219: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:44.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:44.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:44.060658+0000) 2022-09-10T06:09:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:44.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:44.466247+0000) 2022-09-10T06:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:09:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:09:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:09:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:09:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:09:45.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:45.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:45.060863+0000) 2022-09-10T06:09:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:45.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:45.466410+0000) 2022-09-10T06:09:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:45 smithi110 ceph-mon[97633]: pgmap v3220: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:45 smithi104 ceph-mon[97044]: pgmap v3220: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:46.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:46.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:46.061088+0000) 2022-09-10T06:09:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:46.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:46.466725+0000) 2022-09-10T06:09:47.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:47.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:47.061350+0000) 2022-09-10T06:09:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:47.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:47.466929+0000) 2022-09-10T06:09:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:47 smithi110 ceph-mon[97633]: pgmap v3221: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:47 smithi104 ceph-mon[97044]: pgmap v3221: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:48.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:48.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:48.061580+0000) 2022-09-10T06:09:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:48.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:48.467147+0000) 2022-09-10T06:09:49.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:49.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:49.061743+0000) 2022-09-10T06:09:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:49.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:49.467374+0000) 2022-09-10T06:09:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:49 smithi110 ceph-mon[97633]: pgmap v3222: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:49 smithi104 ceph-mon[97044]: pgmap v3222: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:50.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:50.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:50.061991+0000) 2022-09-10T06:09:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:50.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:50.467585+0000) 2022-09-10T06:09:51.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:51.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:51.062254+0000) 2022-09-10T06:09:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:51.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:51.467835+0000) 2022-09-10T06:09:51.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:51 smithi110 ceph-mon[97633]: pgmap v3223: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:51 smithi104 ceph-mon[97044]: pgmap v3223: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:52.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:52.062427+0000) 2022-09-10T06:09:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:52.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:52.468066+0000) 2022-09-10T06:09:53.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:53.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:53.062687+0000) 2022-09-10T06:09:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:53.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:53.468299+0000) 2022-09-10T06:09:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:53 smithi110 ceph-mon[97633]: pgmap v3224: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:53 smithi104 ceph-mon[97044]: pgmap v3224: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:54.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:54.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:54.062899+0000) 2022-09-10T06:09:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:54.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:54.468575+0000) 2022-09-10T06:09:55.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:55.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:55.063177+0000) 2022-09-10T06:09:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:55.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:55.468793+0000) 2022-09-10T06:09:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:55 smithi110 ceph-mon[97633]: pgmap v3225: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:55 smithi104 ceph-mon[97044]: pgmap v3225: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:09:56.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:56.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:56.063436+0000) 2022-09-10T06:09:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:56.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:56.469028+0000) 2022-09-10T06:09:57.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:57.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:57.063599+0000) 2022-09-10T06:09:57.735 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:57.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:57.469208+0000) 2022-09-10T06:09:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:57 smithi110 ceph-mon[97633]: pgmap v3226: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:57 smithi104 ceph-mon[97044]: pgmap v3226: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:09:58.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:09:57.723+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:09:58.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:58.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:58.063828+0000) 2022-09-10T06:09:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:58.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:58.469384+0000) 2022-09-10T06:09:59.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:09:59.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:59.064050+0000) 2022-09-10T06:09:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:09:59.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:09:59.469553+0000) 2022-09-10T06:09:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:09:59 smithi110 ceph-mon[97633]: pgmap v3227: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:09:59 smithi104 ceph-mon[97044]: pgmap v3227: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:00.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:00.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:00.064172+0000) 2022-09-10T06:10:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:00.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:00.469786+0000) 2022-09-10T06:10:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:10:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:10:01.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:01.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:01.064446+0000) 2022-09-10T06:10:01.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:01 smithi104 ceph-mon[97044]: pgmap v3228: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:01.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:01.470044+0000) 2022-09-10T06:10:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:01 smithi110 ceph-mon[97633]: pgmap v3228: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:02.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:02.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:02.064641+0000) 2022-09-10T06:10:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:02.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:02.470330+0000) 2022-09-10T06:10:03.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:03.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:03.064901+0000) 2022-09-10T06:10:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:03.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:03.470480+0000) 2022-09-10T06:10:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:03 smithi110 ceph-mon[97633]: pgmap v3229: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:03 smithi104 ceph-mon[97044]: pgmap v3229: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:04.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:04.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:04.065104+0000) 2022-09-10T06:10:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:04.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:04.470672+0000) 2022-09-10T06:10:05.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:05.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:05.065328+0000) 2022-09-10T06:10:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:05.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:05.470862+0000) 2022-09-10T06:10:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:05 smithi110 ceph-mon[97633]: pgmap v3230: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:05 smithi104 ceph-mon[97044]: pgmap v3230: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:06.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:06.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:06.065509+0000) 2022-09-10T06:10:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:06.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:06.471123+0000) 2022-09-10T06:10:07.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:07.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:07.065737+0000) 2022-09-10T06:10:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:07.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:07.471373+0000) 2022-09-10T06:10:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:07 smithi110 ceph-mon[97633]: pgmap v3231: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:10:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:07 smithi104 ceph-mon[97044]: pgmap v3231: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:10:08.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:08.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:08.065959+0000) 2022-09-10T06:10:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:08.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:08.471568+0000) 2022-09-10T06:10:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:08 smithi110 ceph-mon[97633]: osdmap e284: 12 total, 12 up, 12 in 2022-09-10T06:10:09.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:08 smithi104 ceph-mon[97044]: osdmap e284: 12 total, 12 up, 12 in 2022-09-10T06:10:09.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:09.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:09.066155+0000) 2022-09-10T06:10:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:09.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:09.471788+0000) 2022-09-10T06:10:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:09 smithi110 ceph-mon[97633]: pgmap v3233: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:10:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:09 smithi104 ceph-mon[97044]: pgmap v3233: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:10:10.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:10.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:10.066424+0000) 2022-09-10T06:10:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:10.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:10.472055+0000) 2022-09-10T06:10:11.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:11.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:11.066689+0000) 2022-09-10T06:10:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:11.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:11.472313+0000) 2022-09-10T06:10:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:11 smithi110 ceph-mon[97633]: pgmap v3234: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:10:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:11 smithi104 ceph-mon[97044]: pgmap v3234: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:10:12.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:12.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:12.066965+0000) 2022-09-10T06:10:12.737 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:12.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:12.472470+0000) 2022-09-10T06:10:13.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:10:12.736+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:10:13.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:13.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:13.067209+0000) 2022-09-10T06:10:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:13.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:13.472640+0000) 2022-09-10T06:10:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:13 smithi110 ceph-mon[97633]: pgmap v3235: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:10:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:13 smithi110 ceph-mon[97633]: osdmap e285: 12 total, 12 up, 12 in 2022-09-10T06:10:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:13 smithi104 ceph-mon[97044]: pgmap v3235: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:10:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:13 smithi104 ceph-mon[97044]: osdmap e285: 12 total, 12 up, 12 in 2022-09-10T06:10:14.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:14.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:14.067439+0000) 2022-09-10T06:10:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:14.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:14.472931+0000) 2022-09-10T06:10:15.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:15.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:15.067657+0000) 2022-09-10T06:10:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:15.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:15.473195+0000) 2022-09-10T06:10:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:15 smithi110 ceph-mon[97633]: pgmap v3237: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:10:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:15 smithi104 ceph-mon[97044]: pgmap v3237: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:10:16.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:16.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:16.067898+0000) 2022-09-10T06:10:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:16.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:16.473393+0000) 2022-09-10T06:10:17.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:17.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:17.068160+0000) 2022-09-10T06:10:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:17.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:17.473573+0000) 2022-09-10T06:10:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:17 smithi110 ceph-mon[97633]: pgmap v3238: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T06:10:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:17 smithi104 ceph-mon[97044]: pgmap v3238: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T06:10:18.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:18.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:18.068449+0000) 2022-09-10T06:10:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:18.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:18.473855+0000) 2022-09-10T06:10:19.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:19.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:19.068634+0000) 2022-09-10T06:10:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:19.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:19.474041+0000) 2022-09-10T06:10:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:19 smithi110 ceph-mon[97633]: pgmap v3239: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:10:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:19 smithi104 ceph-mon[97044]: pgmap v3239: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:10:20.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:20.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:20.068900+0000) 2022-09-10T06:10:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:20.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:20.474236+0000) 2022-09-10T06:10:21.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:21.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:21.069193+0000) 2022-09-10T06:10:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:21.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:21.474429+0000) 2022-09-10T06:10:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:21 smithi110 ceph-mon[97633]: pgmap v3240: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:10:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:21 smithi104 ceph-mon[97044]: pgmap v3240: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:10:22.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:22.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:22.069392+0000) 2022-09-10T06:10:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:22.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:22.474638+0000) 2022-09-10T06:10:23.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:23.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:23.069608+0000) 2022-09-10T06:10:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:23.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:23.474906+0000) 2022-09-10T06:10:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:23 smithi110 ceph-mon[97633]: pgmap v3241: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:10:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:23 smithi104 ceph-mon[97044]: pgmap v3241: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:10:24.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:24.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:24.069789+0000) 2022-09-10T06:10:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:24.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:24.475196+0000) 2022-09-10T06:10:25.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:25.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:25.070061+0000) 2022-09-10T06:10:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:25.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:25.475385+0000) 2022-09-10T06:10:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:25 smithi104 ceph-mon[97044]: pgmap v3242: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:10:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:25 smithi110 ceph-mon[97633]: pgmap v3242: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:10:26.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:26.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:26.070338+0000) 2022-09-10T06:10:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:26.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:26.475604+0000) 2022-09-10T06:10:27.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:27.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:27.070600+0000) 2022-09-10T06:10:27.754 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:27.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:27.475823+0000) 2022-09-10T06:10:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:27 smithi104 ceph-mon[97044]: pgmap v3243: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:10:28.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:10:27.751+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:10:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:27 smithi110 ceph-mon[97633]: pgmap v3243: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:10:28.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:28.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:28.070883+0000) 2022-09-10T06:10:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:28.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:28.476065+0000) 2022-09-10T06:10:29.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:29.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:29.071065+0000) 2022-09-10T06:10:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:29.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:29.476287+0000) 2022-09-10T06:10:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:29 smithi104 ceph-mon[97044]: pgmap v3244: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:29 smithi110 ceph-mon[97633]: pgmap v3244: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:30.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:30.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:30.071223+0000) 2022-09-10T06:10:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:30.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:30.476520+0000) 2022-09-10T06:10:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:30 smithi104 ceph-mon[97044]: pgmap v3245: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:30 smithi110 ceph-mon[97633]: pgmap v3245: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:31.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:31.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:31.071430+0000) 2022-09-10T06:10:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:31.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:31.476747+0000) 2022-09-10T06:10:32.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:32.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:32.071682+0000) 2022-09-10T06:10:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:32.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:32.477013+0000) 2022-09-10T06:10:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:32 smithi104 ceph-mon[97044]: pgmap v3246: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:32 smithi110 ceph-mon[97633]: pgmap v3246: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:33.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:33.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:33.071900+0000) 2022-09-10T06:10:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:33.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:33.477237+0000) 2022-09-10T06:10:34.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:34.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:34.072133+0000) 2022-09-10T06:10:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:34.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:34.477501+0000) 2022-09-10T06:10:35.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:35.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:35.072352+0000) 2022-09-10T06:10:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:35 smithi104 ceph-mon[97044]: pgmap v3247: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:35.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:35.477639+0000) 2022-09-10T06:10:35.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:35 smithi110 ceph-mon[97633]: pgmap v3247: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:36.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:36.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:36.072546+0000) 2022-09-10T06:10:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:36.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:36.477892+0000) 2022-09-10T06:10:37.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:37.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:37.072789+0000) 2022-09-10T06:10:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:37.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:37.478084+0000) 2022-09-10T06:10:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:37 smithi110 ceph-mon[97633]: pgmap v3248: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:37 smithi104 ceph-mon[97044]: pgmap v3248: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:38.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:38.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:38.072951+0000) 2022-09-10T06:10:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:38.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:38.478315+0000) 2022-09-10T06:10:39.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:39.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:39.073152+0000) 2022-09-10T06:10:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:39.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:39.478508+0000) 2022-09-10T06:10:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:39 smithi110 ceph-mon[97633]: pgmap v3249: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:39 smithi104 ceph-mon[97044]: pgmap v3249: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:40.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:40.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:40.073337+0000) 2022-09-10T06:10:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:40.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:40.478700+0000) 2022-09-10T06:10:41.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:41.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:41.073472+0000) 2022-09-10T06:10:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:41.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:41.478930+0000) 2022-09-10T06:10:41.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:41 smithi110 ceph-mon[97633]: pgmap v3250: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:41 smithi104 ceph-mon[97044]: pgmap v3250: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:42.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:42.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:42.073728+0000) 2022-09-10T06:10:42.760 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:42.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:42.479137+0000) 2022-09-10T06:10:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:10:42.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:10:43.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:43.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:43.074012+0000) 2022-09-10T06:10:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:43.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:43.479343+0000) 2022-09-10T06:10:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:43 smithi110 ceph-mon[97633]: pgmap v3251: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:43 smithi104 ceph-mon[97044]: pgmap v3251: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:44.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:44.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:44.074251+0000) 2022-09-10T06:10:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:44.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:44.479562+0000) 2022-09-10T06:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:10:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:10:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:10:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:10:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:10:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:10:45.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:45.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:45.074486+0000) 2022-09-10T06:10:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:45.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:45.479790+0000) 2022-09-10T06:10:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:45 smithi110 ceph-mon[97633]: pgmap v3252: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:45 smithi104 ceph-mon[97044]: pgmap v3252: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:10:46.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:46.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:46.074779+0000) 2022-09-10T06:10:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:46.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:46.480091+0000) 2022-09-10T06:10:47.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:47.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:47.074991+0000) 2022-09-10T06:10:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:47.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:47.480373+0000) 2022-09-10T06:10:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:47 smithi110 ceph-mon[97633]: pgmap v3253: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:47 smithi104 ceph-mon[97044]: pgmap v3253: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:48.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:48.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:48.075167+0000) 2022-09-10T06:10:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:48.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:48.480637+0000) 2022-09-10T06:10:49.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:49.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:49.075396+0000) 2022-09-10T06:10:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:49.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:49.480820+0000) 2022-09-10T06:10:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:49 smithi110 ceph-mon[97633]: pgmap v3254: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:10:50.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:49 smithi104 ceph-mon[97044]: pgmap v3254: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:10:50.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:50.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:50.075647+0000) 2022-09-10T06:10:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:50.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:50.481034+0000) 2022-09-10T06:10:51.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:51.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:51.075879+0000) 2022-09-10T06:10:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:51.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:51.481241+0000) 2022-09-10T06:10:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:51 smithi110 ceph-mon[97633]: pgmap v3255: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:51 smithi104 ceph-mon[97044]: pgmap v3255: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:52.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:52.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:52.076071+0000) 2022-09-10T06:10:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:52.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:52.481439+0000) 2022-09-10T06:10:53.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:53.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:53.076309+0000) 2022-09-10T06:10:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:53.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:53.481664+0000) 2022-09-10T06:10:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:53 smithi110 ceph-mon[97633]: pgmap v3256: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:10:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:53 smithi104 ceph-mon[97044]: pgmap v3256: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:10:54.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:54.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:54.076554+0000) 2022-09-10T06:10:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:54.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:54.481932+0000) 2022-09-10T06:10:55.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:55.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:55.076761+0000) 2022-09-10T06:10:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:55.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:55.482177+0000) 2022-09-10T06:10:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:55 smithi110 ceph-mon[97633]: pgmap v3257: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:10:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:55 smithi104 ceph-mon[97044]: pgmap v3257: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:10:56.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:56.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:56.076953+0000) 2022-09-10T06:10:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:56.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:56.482390+0000) 2022-09-10T06:10:57.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:57.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:57.077153+0000) 2022-09-10T06:10:57.766 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:57.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:57.482524+0000) 2022-09-10T06:10:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:57 smithi110 ceph-mon[97633]: pgmap v3258: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:57 smithi104 ceph-mon[97044]: pgmap v3258: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:10:58.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:10:57.764+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:10:58.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:58.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:58.077362+0000) 2022-09-10T06:10:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:58.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:58.482701+0000) 2022-09-10T06:10:59.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:10:59.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:59.077601+0000) 2022-09-10T06:10:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:10:59.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:10:59.482933+0000) 2022-09-10T06:10:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:10:59 smithi110 ceph-mon[97633]: pgmap v3259: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:10:59 smithi104 ceph-mon[97044]: pgmap v3259: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:00.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:00.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:00.077878+0000) 2022-09-10T06:11:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:00.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:00.483145+0000) 2022-09-10T06:11:01.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:01.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:01.078134+0000) 2022-09-10T06:11:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:01.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:01.483350+0000) 2022-09-10T06:11:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:01 smithi110 ceph-mon[97633]: pgmap v3260: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:01 smithi104 ceph-mon[97044]: pgmap v3260: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:02.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:02.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:02.078257+0000) 2022-09-10T06:11:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:02.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:02.483495+0000) 2022-09-10T06:11:03.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:03.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:03.078486+0000) 2022-09-10T06:11:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:03.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:03.483689+0000) 2022-09-10T06:11:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:03 smithi110 ceph-mon[97633]: pgmap v3261: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:03 smithi104 ceph-mon[97044]: pgmap v3261: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:04.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:04.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:04.078715+0000) 2022-09-10T06:11:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:04.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:04.483917+0000) 2022-09-10T06:11:05.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:05.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:05.078963+0000) 2022-09-10T06:11:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:05.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:05.484201+0000) 2022-09-10T06:11:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:05 smithi110 ceph-mon[97633]: pgmap v3262: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:05 smithi104 ceph-mon[97044]: pgmap v3262: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:06.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:06.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:06.079231+0000) 2022-09-10T06:11:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:06.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:06.484455+0000) 2022-09-10T06:11:07.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:07.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:07.079480+0000) 2022-09-10T06:11:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:07.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:07.484680+0000) 2022-09-10T06:11:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:07 smithi110 ceph-mon[97633]: pgmap v3263: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:11:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:07 smithi104 ceph-mon[97044]: pgmap v3263: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:11:08.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:08.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:08.079683+0000) 2022-09-10T06:11:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:08.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:08.484925+0000) 2022-09-10T06:11:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:08 smithi110 ceph-mon[97633]: osdmap e286: 12 total, 12 up, 12 in 2022-09-10T06:11:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:08 smithi104 ceph-mon[97044]: osdmap e286: 12 total, 12 up, 12 in 2022-09-10T06:11:09.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:09.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:09.079841+0000) 2022-09-10T06:11:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:09.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:09.485198+0000) 2022-09-10T06:11:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:09 smithi110 ceph-mon[97633]: pgmap v3265: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:11:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:09 smithi104 ceph-mon[97044]: pgmap v3265: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:11:10.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:10.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:10.080040+0000) 2022-09-10T06:11:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:10.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:10.485407+0000) 2022-09-10T06:11:11.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:11.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:11.080210+0000) 2022-09-10T06:11:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:11.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:11.485691+0000) 2022-09-10T06:11:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:11 smithi110 ceph-mon[97633]: pgmap v3266: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:11:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:11 smithi104 ceph-mon[97044]: pgmap v3266: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:11:12.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:12.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:12.080472+0000) 2022-09-10T06:11:12.773 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:12.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:12.485858+0000) 2022-09-10T06:11:13.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:11:12.771+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:11:13.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:13.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:13.080738+0000) 2022-09-10T06:11:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:13.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:13.486041+0000) 2022-09-10T06:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:13 smithi110 ceph-mon[97633]: pgmap v3267: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:13 smithi110 ceph-mon[97633]: osdmap e287: 12 total, 12 up, 12 in 2022-09-10T06:11:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:13 smithi104 ceph-mon[97044]: pgmap v3267: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:11:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:13 smithi104 ceph-mon[97044]: osdmap e287: 12 total, 12 up, 12 in 2022-09-10T06:11:14.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:14.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:14.080989+0000) 2022-09-10T06:11:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:14.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:14.486195+0000) 2022-09-10T06:11:15.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:15.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:15.081217+0000) 2022-09-10T06:11:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:15.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:15.486427+0000) 2022-09-10T06:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:15 smithi110 ceph-mon[97633]: pgmap v3269: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:11:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:15 smithi104 ceph-mon[97044]: pgmap v3269: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:11:16.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:16.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:16.081456+0000) 2022-09-10T06:11:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:16.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:16.486605+0000) 2022-09-10T06:11:17.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:17.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:17.081621+0000) 2022-09-10T06:11:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:17.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:17.486813+0000) 2022-09-10T06:11:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:17 smithi110 ceph-mon[97633]: pgmap v3270: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T06:11:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:17 smithi104 ceph-mon[97044]: pgmap v3270: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T06:11:18.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:18.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:18.081875+0000) 2022-09-10T06:11:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:18.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:18.487042+0000) 2022-09-10T06:11:19.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:19.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:19.082137+0000) 2022-09-10T06:11:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:19.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:19.487273+0000) 2022-09-10T06:11:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:19 smithi110 ceph-mon[97633]: pgmap v3271: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:11:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:19 smithi104 ceph-mon[97044]: pgmap v3271: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:11:20.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:20.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:20.082343+0000) 2022-09-10T06:11:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:20.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:20.487530+0000) 2022-09-10T06:11:21.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:21.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:21.082533+0000) 2022-09-10T06:11:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:21.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:21.487782+0000) 2022-09-10T06:11:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:21 smithi110 ceph-mon[97633]: pgmap v3272: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:11:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:21 smithi104 ceph-mon[97044]: pgmap v3272: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:11:22.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:22.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:22.082685+0000) 2022-09-10T06:11:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:22.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:22.488024+0000) 2022-09-10T06:11:23.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:23.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:23.082926+0000) 2022-09-10T06:11:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:23.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:23.488240+0000) 2022-09-10T06:11:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:23 smithi110 ceph-mon[97633]: pgmap v3273: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:11:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:23 smithi104 ceph-mon[97044]: pgmap v3273: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:11:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:24.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:24.083211+0000) 2022-09-10T06:11:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:24.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:24.488488+0000) 2022-09-10T06:11:25.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:25.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:25.083421+0000) 2022-09-10T06:11:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:25.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:25.488717+0000) 2022-09-10T06:11:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:25 smithi110 ceph-mon[97633]: pgmap v3274: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:11:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:25 smithi104 ceph-mon[97044]: pgmap v3274: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:11:26.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:26.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:26.083557+0000) 2022-09-10T06:11:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:26.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:26.488970+0000) 2022-09-10T06:11:27.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:27.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:27.083712+0000) 2022-09-10T06:11:27.786 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:27.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:27.489236+0000) 2022-09-10T06:11:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:27 smithi110 ceph-mon[97633]: pgmap v3275: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:11:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:27 smithi104 ceph-mon[97044]: pgmap v3275: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:11:28.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:11:27.784+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:11:28.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:28.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:28.083941+0000) 2022-09-10T06:11:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:28.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:28.489420+0000) 2022-09-10T06:11:29.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:29.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:29.084124+0000) 2022-09-10T06:11:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:29.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:29.489638+0000) 2022-09-10T06:11:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:29 smithi110 ceph-mon[97633]: pgmap v3276: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:29 smithi104 ceph-mon[97044]: pgmap v3276: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:30.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:30.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:30.084364+0000) 2022-09-10T06:11:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:30.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:30.489817+0000) 2022-09-10T06:11:31.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:31.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:31.084598+0000) 2022-09-10T06:11:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:31.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:31.490023+0000) 2022-09-10T06:11:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:31 smithi110 ceph-mon[97633]: pgmap v3277: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:31 smithi104 ceph-mon[97044]: pgmap v3277: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:32.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:32.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:32.084896+0000) 2022-09-10T06:11:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:32.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:32.490236+0000) 2022-09-10T06:11:33.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:33.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:33.085136+0000) 2022-09-10T06:11:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:33.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:33.490393+0000) 2022-09-10T06:11:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:33 smithi110 ceph-mon[97633]: pgmap v3278: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:33 smithi104 ceph-mon[97044]: pgmap v3278: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:34.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:34.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:34.085325+0000) 2022-09-10T06:11:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:34.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:34.490636+0000) 2022-09-10T06:11:35.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:35.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:35.085479+0000) 2022-09-10T06:11:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:35.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:35.490865+0000) 2022-09-10T06:11:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:35 smithi110 ceph-mon[97633]: pgmap v3279: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:35 smithi104 ceph-mon[97044]: pgmap v3279: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:36.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:36.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:36.085707+0000) 2022-09-10T06:11:36.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:36.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:36.491185+0000) 2022-09-10T06:11:37.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:37.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:37.085965+0000) 2022-09-10T06:11:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:37.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:37.491357+0000) 2022-09-10T06:11:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:37 smithi110 ceph-mon[97633]: pgmap v3280: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:37 smithi104 ceph-mon[97044]: pgmap v3280: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:38.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:38.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:38.086214+0000) 2022-09-10T06:11:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:38.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:38.491554+0000) 2022-09-10T06:11:39.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:39.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:39.086420+0000) 2022-09-10T06:11:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:39.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:39.491762+0000) 2022-09-10T06:11:39.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:39 smithi110 ceph-mon[97633]: pgmap v3281: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:39 smithi104 ceph-mon[97044]: pgmap v3281: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:40.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:40.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:40.086653+0000) 2022-09-10T06:11:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:40.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:40.491966+0000) 2022-09-10T06:11:41.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:41.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:41.086912+0000) 2022-09-10T06:11:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:41.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:41.492163+0000) 2022-09-10T06:11:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:41 smithi110 ceph-mon[97633]: pgmap v3282: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:41 smithi104 ceph-mon[97044]: pgmap v3282: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:42.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:42.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:42.087162+0000) 2022-09-10T06:11:42.798 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:42.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:42.492368+0000) 2022-09-10T06:11:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:11:42.796+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:11:43.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:43.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:43.087374+0000) 2022-09-10T06:11:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:43.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:43.492510+0000) 2022-09-10T06:11:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:43 smithi104 ceph-mon[97044]: pgmap v3283: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:43 smithi110 ceph-mon[97633]: pgmap v3283: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:44.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:44.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:44.087619+0000) 2022-09-10T06:11:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:44.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:44.492754+0000) 2022-09-10T06:11:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:11:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:11:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:11:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:11:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:11:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:11:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:11:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:11:45.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:45.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:45.087859+0000) 2022-09-10T06:11:45.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:45.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:45.492932+0000) 2022-09-10T06:11:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:45 smithi104 ceph-mon[97044]: pgmap v3284: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:45 smithi110 ceph-mon[97633]: pgmap v3284: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:46.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:46.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:46.088077+0000) 2022-09-10T06:11:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:46.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:46.493181+0000) 2022-09-10T06:11:47.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:47.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:47.088327+0000) 2022-09-10T06:11:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:47.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:47.493352+0000) 2022-09-10T06:11:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:47 smithi104 ceph-mon[97044]: pgmap v3285: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:47 smithi110 ceph-mon[97633]: pgmap v3285: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:48.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:48.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:48.088582+0000) 2022-09-10T06:11:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:48.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:48.493550+0000) 2022-09-10T06:11:49.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:49.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:49.088827+0000) 2022-09-10T06:11:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:49.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:49.493767+0000) 2022-09-10T06:11:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:49 smithi104 ceph-mon[97044]: pgmap v3286: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:49 smithi110 ceph-mon[97633]: pgmap v3286: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:50.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:50.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:50.089108+0000) 2022-09-10T06:11:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:50.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:50.493955+0000) 2022-09-10T06:11:51.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:51.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:51.089362+0000) 2022-09-10T06:11:51.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:51.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:51.494148+0000) 2022-09-10T06:11:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:51 smithi104 ceph-mon[97044]: pgmap v3287: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:51 smithi110 ceph-mon[97633]: pgmap v3287: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:52.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:52.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:52.089564+0000) 2022-09-10T06:11:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:52.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:52.494368+0000) 2022-09-10T06:11:53.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:53.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:53.089791+0000) 2022-09-10T06:11:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:53.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:53.494617+0000) 2022-09-10T06:11:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:53 smithi104 ceph-mon[97044]: pgmap v3288: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:53 smithi110 ceph-mon[97633]: pgmap v3288: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:54.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:54.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:54.089954+0000) 2022-09-10T06:11:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:54.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:54.494832+0000) 2022-09-10T06:11:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:54 smithi104 ceph-mon[97044]: pgmap v3289: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:54 smithi110 ceph-mon[97633]: pgmap v3289: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:11:55.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:55.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:55.090159+0000) 2022-09-10T06:11:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:55.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:55.494964+0000) 2022-09-10T06:11:56.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:56.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:56.090347+0000) 2022-09-10T06:11:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:56.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:56.495193+0000) 2022-09-10T06:11:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:56 smithi104 ceph-mon[97044]: pgmap v3290: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:56 smithi110 ceph-mon[97633]: pgmap v3290: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:11:57.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:57.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:57.090524+0000) 2022-09-10T06:11:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:57.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:57.495437+0000) 2022-09-10T06:11:58.091 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:11:57.813+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:11:58.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:58.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:58.090784+0000) 2022-09-10T06:11:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:58.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:58.495667+0000) 2022-09-10T06:11:59.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:11:59.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:59.090978+0000) 2022-09-10T06:11:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:11:59.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:11:59.495850+0000) 2022-09-10T06:11:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:11:59 smithi110 ceph-mon[97633]: pgmap v3291: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:11:59 smithi104 ceph-mon[97044]: pgmap v3291: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:00.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:00.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:00.091247+0000) 2022-09-10T06:12:00.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:00.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:00.496135+0000) 2022-09-10T06:12:01.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:01.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:01.091481+0000) 2022-09-10T06:12:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:01.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:01.496314+0000) 2022-09-10T06:12:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:01 smithi110 ceph-mon[97633]: pgmap v3292: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:01 smithi104 ceph-mon[97044]: pgmap v3292: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:02.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:02.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:02.091727+0000) 2022-09-10T06:12:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:02.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:02.496497+0000) 2022-09-10T06:12:03.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:03.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:03.091994+0000) 2022-09-10T06:12:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:03.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:03.496728+0000) 2022-09-10T06:12:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:03 smithi110 ceph-mon[97633]: pgmap v3293: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:03 smithi104 ceph-mon[97044]: pgmap v3293: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:04.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:04.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:04.092254+0000) 2022-09-10T06:12:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:04.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:04.496998+0000) 2022-09-10T06:12:05.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:05.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:05.092467+0000) 2022-09-10T06:12:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:05.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:05.497266+0000) 2022-09-10T06:12:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:05 smithi110 ceph-mon[97633]: pgmap v3294: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:05 smithi104 ceph-mon[97044]: pgmap v3294: 129 pgs: 129 active+clean; 8.0 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:06.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:06.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:06.092696+0000) 2022-09-10T06:12:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:06.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:06.497490+0000) 2022-09-10T06:12:07.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:07.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:07.092935+0000) 2022-09-10T06:12:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:07.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:07.497719+0000) 2022-09-10T06:12:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:07 smithi110 ceph-mon[97633]: pgmap v3295: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:12:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:07 smithi104 ceph-mon[97044]: pgmap v3295: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:12:08.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:08.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:08.093208+0000) 2022-09-10T06:12:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:08.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:08.497996+0000) 2022-09-10T06:12:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:08 smithi110 ceph-mon[97633]: osdmap e288: 12 total, 12 up, 12 in 2022-09-10T06:12:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:08 smithi104 ceph-mon[97044]: osdmap e288: 12 total, 12 up, 12 in 2022-09-10T06:12:09.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:09.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:09.093418+0000) 2022-09-10T06:12:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:09.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:09.498228+0000) 2022-09-10T06:12:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:09 smithi110 ceph-mon[97633]: pgmap v3297: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:12:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:09 smithi104 ceph-mon[97044]: pgmap v3297: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:12:10.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:10.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:10.093648+0000) 2022-09-10T06:12:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:10.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:10.498429+0000) 2022-09-10T06:12:11.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:11.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:11.093856+0000) 2022-09-10T06:12:11.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:11.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:11.498554+0000) 2022-09-10T06:12:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:11 smithi110 ceph-mon[97633]: pgmap v3298: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:12:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:11 smithi104 ceph-mon[97044]: pgmap v3298: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:12:12.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:12.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:12.094123+0000) 2022-09-10T06:12:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:12.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:12.498776+0000) 2022-09-10T06:12:13.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:12:12.826+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:12:13.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:13.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:13.094311+0000) 2022-09-10T06:12:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:13.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:13.499045+0000) 2022-09-10T06:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:13 smithi110 ceph-mon[97633]: pgmap v3299: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:13 smithi110 ceph-mon[97633]: osdmap e289: 12 total, 12 up, 12 in 2022-09-10T06:12:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:13 smithi104 ceph-mon[97044]: pgmap v3299: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:12:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:13 smithi104 ceph-mon[97044]: osdmap e289: 12 total, 12 up, 12 in 2022-09-10T06:12:14.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:14.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:14.094485+0000) 2022-09-10T06:12:14.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:14.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:14.499291+0000) 2022-09-10T06:12:15.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:15.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:15.094694+0000) 2022-09-10T06:12:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:15.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:15.499512+0000) 2022-09-10T06:12:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:15 smithi110 ceph-mon[97633]: pgmap v3301: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:12:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:15 smithi104 ceph-mon[97044]: pgmap v3301: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:12:16.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:16.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:16.094941+0000) 2022-09-10T06:12:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:16.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:16.499704+0000) 2022-09-10T06:12:17.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:17.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:17.095188+0000) 2022-09-10T06:12:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:17.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:17.499896+0000) 2022-09-10T06:12:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:17 smithi110 ceph-mon[97633]: pgmap v3302: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:12:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:17 smithi104 ceph-mon[97044]: pgmap v3302: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:12:18.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:18.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:18.095387+0000) 2022-09-10T06:12:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:18.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:18.500151+0000) 2022-09-10T06:12:19.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:19.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:19.095595+0000) 2022-09-10T06:12:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:19.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:19.500360+0000) 2022-09-10T06:12:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:19 smithi110 ceph-mon[97633]: pgmap v3303: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:12:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:19 smithi104 ceph-mon[97044]: pgmap v3303: 129 pgs: 129 active+clean; 8.0 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:12:20.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:20.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:20.095765+0000) 2022-09-10T06:12:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:20.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:20.500634+0000) 2022-09-10T06:12:21.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:21.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:21.095937+0000) 2022-09-10T06:12:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:21.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:21.500900+0000) 2022-09-10T06:12:21.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:21 smithi110 ceph-mon[97633]: pgmap v3304: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:12:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:21 smithi104 ceph-mon[97044]: pgmap v3304: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:12:22.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:22.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:22.096144+0000) 2022-09-10T06:12:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:22.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:22.501141+0000) 2022-09-10T06:12:23.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:23.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:23.096404+0000) 2022-09-10T06:12:23.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:23.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:23.501338+0000) 2022-09-10T06:12:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:23 smithi110 ceph-mon[97633]: pgmap v3305: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:12:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:23 smithi104 ceph-mon[97044]: pgmap v3305: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:12:24.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:24.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:24.096604+0000) 2022-09-10T06:12:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:24.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:24.501613+0000) 2022-09-10T06:12:25.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:25.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:25.096827+0000) 2022-09-10T06:12:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:25.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:25.501880+0000) 2022-09-10T06:12:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:25 smithi110 ceph-mon[97633]: pgmap v3306: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:12:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:25 smithi104 ceph-mon[97044]: pgmap v3306: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:12:26.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:26.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:26.097024+0000) 2022-09-10T06:12:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:26.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:26.502051+0000) 2022-09-10T06:12:27.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:27.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:27.097218+0000) 2022-09-10T06:12:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:27.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:27.502161+0000) 2022-09-10T06:12:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:27 smithi110 ceph-mon[97633]: pgmap v3307: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:12:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:27 smithi104 ceph-mon[97044]: pgmap v3307: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:12:28.098 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:12:27.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:12:28.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:28.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:28.097446+0000) 2022-09-10T06:12:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:28.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:28.502397+0000) 2022-09-10T06:12:29.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:29.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:29.097691+0000) 2022-09-10T06:12:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:29.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:29.502660+0000) 2022-09-10T06:12:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:29 smithi110 ceph-mon[97633]: pgmap v3308: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:29 smithi104 ceph-mon[97044]: pgmap v3308: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:30.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:30.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:30.097945+0000) 2022-09-10T06:12:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:30.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:30.502931+0000) 2022-09-10T06:12:31.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:31.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:31.098192+0000) 2022-09-10T06:12:31.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:31.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:31.503205+0000) 2022-09-10T06:12:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:31 smithi110 ceph-mon[97633]: pgmap v3309: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:31 smithi104 ceph-mon[97044]: pgmap v3309: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:32.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:32.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:32.098423+0000) 2022-09-10T06:12:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:32.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:32.503397+0000) 2022-09-10T06:12:33.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:33.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:33.098648+0000) 2022-09-10T06:12:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:33.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:33.503626+0000) 2022-09-10T06:12:33.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:33 smithi110 ceph-mon[97633]: pgmap v3310: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:33 smithi104 ceph-mon[97044]: pgmap v3310: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:34.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:34.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:34.098892+0000) 2022-09-10T06:12:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:34.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:34.503856+0000) 2022-09-10T06:12:35.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:35.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:35.099137+0000) 2022-09-10T06:12:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:35.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:35.504153+0000) 2022-09-10T06:12:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:35 smithi110 ceph-mon[97633]: pgmap v3311: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:35 smithi104 ceph-mon[97044]: pgmap v3311: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:36.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:36.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:36.099323+0000) 2022-09-10T06:12:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:36.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:36.504401+0000) 2022-09-10T06:12:37.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:37.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:37.099552+0000) 2022-09-10T06:12:37.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:37.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:37.504676+0000) 2022-09-10T06:12:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:37 smithi110 ceph-mon[97633]: pgmap v3312: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:37 smithi104 ceph-mon[97044]: pgmap v3312: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:38.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:38.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:38.099786+0000) 2022-09-10T06:12:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:38.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:38.504930+0000) 2022-09-10T06:12:39.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:39.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:39.100006+0000) 2022-09-10T06:12:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:39.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:39.505149+0000) 2022-09-10T06:12:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:39 smithi110 ceph-mon[97633]: pgmap v3313: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:39 smithi104 ceph-mon[97044]: pgmap v3313: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:40.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:40.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:40.100272+0000) 2022-09-10T06:12:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:40.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:40.505317+0000) 2022-09-10T06:12:41.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:41.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:41.100531+0000) 2022-09-10T06:12:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:41.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:41.505523+0000) 2022-09-10T06:12:41.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:41 smithi110 ceph-mon[97633]: pgmap v3314: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:41 smithi104 ceph-mon[97044]: pgmap v3314: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:42.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:42.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:42.100822+0000) 2022-09-10T06:12:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:42.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:42.505771+0000) 2022-09-10T06:12:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:43.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:43.100993+0000) 2022-09-10T06:12:43.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:12:42.852+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:12:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:43.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:43.506002+0000) 2022-09-10T06:12:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:43 smithi110 ceph-mon[97633]: pgmap v3315: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:43 smithi104 ceph-mon[97044]: pgmap v3315: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:44.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:44.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:44.101173+0000) 2022-09-10T06:12:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:44.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:44.506249+0000) 2022-09-10T06:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:12:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:12:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:12:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:12:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:12:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:12:45.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:45.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:45.101368+0000) 2022-09-10T06:12:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:45.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:45.506482+0000) 2022-09-10T06:12:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:45 smithi110 ceph-mon[97633]: pgmap v3316: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:45 smithi104 ceph-mon[97044]: pgmap v3316: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:46.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:46.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:46.101606+0000) 2022-09-10T06:12:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:46.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:46.506658+0000) 2022-09-10T06:12:47.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:47.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:47.101885+0000) 2022-09-10T06:12:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:47.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:47.506858+0000) 2022-09-10T06:12:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:47 smithi110 ceph-mon[97633]: pgmap v3317: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:47 smithi104 ceph-mon[97044]: pgmap v3317: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:48.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:48.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:48.102142+0000) 2022-09-10T06:12:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:48.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:48.507078+0000) 2022-09-10T06:12:49.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:49.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:49.102340+0000) 2022-09-10T06:12:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:49.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:49.507302+0000) 2022-09-10T06:12:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:49 smithi110 ceph-mon[97633]: pgmap v3318: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:49 smithi104 ceph-mon[97044]: pgmap v3318: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:50.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:50.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:50.102614+0000) 2022-09-10T06:12:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:50.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:50.507536+0000) 2022-09-10T06:12:51.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:51.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:51.102755+0000) 2022-09-10T06:12:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:51.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:51.507716+0000) 2022-09-10T06:12:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:51 smithi110 ceph-mon[97633]: pgmap v3319: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:51 smithi104 ceph-mon[97044]: pgmap v3319: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:52.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:52.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:52.103008+0000) 2022-09-10T06:12:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:52.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:52.507914+0000) 2022-09-10T06:12:53.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:53.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:53.103265+0000) 2022-09-10T06:12:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:53.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:53.508184+0000) 2022-09-10T06:12:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:53 smithi104 ceph-mon[97044]: pgmap v3320: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:53 smithi110 ceph-mon[97633]: pgmap v3320: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:54.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:54.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:54.103482+0000) 2022-09-10T06:12:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:54.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:54.508420+0000) 2022-09-10T06:12:55.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:55.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:55.103737+0000) 2022-09-10T06:12:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:55.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:55.508683+0000) 2022-09-10T06:12:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:55 smithi104 ceph-mon[97044]: pgmap v3321: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:55 smithi110 ceph-mon[97633]: pgmap v3321: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:12:56.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:56.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:56.103956+0000) 2022-09-10T06:12:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:56.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:56.508896+0000) 2022-09-10T06:12:57.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:57.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:57.104142+0000) 2022-09-10T06:12:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:57.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:57.509082+0000) 2022-09-10T06:12:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:57 smithi104 ceph-mon[97044]: pgmap v3322: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:58.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:57 smithi110 ceph-mon[97633]: pgmap v3322: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:12:58.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:58.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:58.104380+0000) 2022-09-10T06:12:58.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:12:57.856+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:12:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:58.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:58.509266+0000) 2022-09-10T06:12:59.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:12:59.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:59.104589+0000) 2022-09-10T06:12:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:12:59.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:12:59.509498+0000) 2022-09-10T06:13:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:12:59 smithi104 ceph-mon[97044]: pgmap v3323: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:12:59 smithi110 ceph-mon[97633]: pgmap v3323: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:00.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:00.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:00.104788+0000) 2022-09-10T06:13:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:00.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:00.509703+0000) 2022-09-10T06:13:01.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:01.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:01.104953+0000) 2022-09-10T06:13:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:01.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:01.509919+0000) 2022-09-10T06:13:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:01 smithi104 ceph-mon[97044]: pgmap v3324: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:01 smithi110 ceph-mon[97633]: pgmap v3324: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:02.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:02.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:02.105163+0000) 2022-09-10T06:13:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:02.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:02.510087+0000) 2022-09-10T06:13:03.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:03.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:03.105371+0000) 2022-09-10T06:13:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:03.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:03.510259+0000) 2022-09-10T06:13:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:03 smithi104 ceph-mon[97044]: pgmap v3325: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:03 smithi110 ceph-mon[97633]: pgmap v3325: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:04.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:04.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:04.105544+0000) 2022-09-10T06:13:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:04.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:04.510506+0000) 2022-09-10T06:13:05.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:05.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:05.105797+0000) 2022-09-10T06:13:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:05.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:05.510804+0000) 2022-09-10T06:13:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:05 smithi104 ceph-mon[97044]: pgmap v3326: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:05 smithi110 ceph-mon[97633]: pgmap v3326: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:06.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:06.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:06.106038+0000) 2022-09-10T06:13:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:06.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:06.511019+0000) 2022-09-10T06:13:07.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:07.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:07.106276+0000) 2022-09-10T06:13:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:07.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:07.511297+0000) 2022-09-10T06:13:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:07 smithi104 ceph-mon[97044]: pgmap v3327: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:13:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:07 smithi110 ceph-mon[97633]: pgmap v3327: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:13:08.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:08.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:08.106473+0000) 2022-09-10T06:13:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:08.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:08.511575+0000) 2022-09-10T06:13:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:08 smithi104 ceph-mon[97044]: osdmap e290: 12 total, 12 up, 12 in 2022-09-10T06:13:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:08 smithi110 ceph-mon[97633]: osdmap e290: 12 total, 12 up, 12 in 2022-09-10T06:13:09.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:09.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:09.106707+0000) 2022-09-10T06:13:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:09.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:09.511827+0000) 2022-09-10T06:13:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:09 smithi104 ceph-mon[97044]: pgmap v3329: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:13:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:09 smithi110 ceph-mon[97633]: pgmap v3329: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:13:10.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:10.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:10.106956+0000) 2022-09-10T06:13:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:10.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:10.512124+0000) 2022-09-10T06:13:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:10 smithi104 ceph-mon[97044]: pgmap v3330: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:13:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:10 smithi110 ceph-mon[97633]: pgmap v3330: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:13:11.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:11.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:11.107173+0000) 2022-09-10T06:13:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:11.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:11.512279+0000) 2022-09-10T06:13:12.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:12.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:12.107351+0000) 2022-09-10T06:13:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:12.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:12.512546+0000) 2022-09-10T06:13:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:12 smithi104 ceph-mon[97044]: pgmap v3331: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:13:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:12 smithi110 ceph-mon[97633]: pgmap v3331: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:13:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:13.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:13.107516+0000) 2022-09-10T06:13:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:13:12.871+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:13:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:13.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:13.512750+0000) 2022-09-10T06:13:14.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:14.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:14.107681+0000) 2022-09-10T06:13:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:14.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:14.512964+0000) 2022-09-10T06:13:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:14 smithi110 ceph-mon[97633]: osdmap e291: 12 total, 12 up, 12 in 2022-09-10T06:13:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:14 smithi104 ceph-mon[97044]: osdmap e291: 12 total, 12 up, 12 in 2022-09-10T06:13:15.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:15.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:15.107966+0000) 2022-09-10T06:13:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:15.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:15.513207+0000) 2022-09-10T06:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:15 smithi110 ceph-mon[97633]: pgmap v3333: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:13:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:15 smithi104 ceph-mon[97044]: pgmap v3333: 129 pgs: 129 active+clean; 8.0 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:13:16.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:16.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:16.108230+0000) 2022-09-10T06:13:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:16.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:16.513474+0000) 2022-09-10T06:13:17.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:17.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:17.108458+0000) 2022-09-10T06:13:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:17.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:17.513672+0000) 2022-09-10T06:13:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:17 smithi110 ceph-mon[97633]: pgmap v3334: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T06:13:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:17 smithi104 ceph-mon[97044]: pgmap v3334: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T06:13:18.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:18.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:18.108607+0000) 2022-09-10T06:13:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:18.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:18.513919+0000) 2022-09-10T06:13:19.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:19.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:19.108767+0000) 2022-09-10T06:13:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:19.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:19.514194+0000) 2022-09-10T06:13:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:19 smithi110 ceph-mon[97633]: pgmap v3335: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:13:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:19 smithi104 ceph-mon[97044]: pgmap v3335: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:13:20.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:20.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:20.109050+0000) 2022-09-10T06:13:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:20.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:20.514447+0000) 2022-09-10T06:13:21.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:21.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:21.109261+0000) 2022-09-10T06:13:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:21.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:21.514676+0000) 2022-09-10T06:13:21.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:21 smithi110 ceph-mon[97633]: pgmap v3336: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:13:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:21 smithi104 ceph-mon[97044]: pgmap v3336: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:13:22.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:22.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:22.109496+0000) 2022-09-10T06:13:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:22.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:22.514945+0000) 2022-09-10T06:13:23.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:23.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:23.109669+0000) 2022-09-10T06:13:23.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:23.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:23.515191+0000) 2022-09-10T06:13:23.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:23 smithi110 ceph-mon[97633]: pgmap v3337: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:13:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:23 smithi104 ceph-mon[97044]: pgmap v3337: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:13:24.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:24.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:24.109833+0000) 2022-09-10T06:13:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:24.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:24.515427+0000) 2022-09-10T06:13:25.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:25.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:25.110062+0000) 2022-09-10T06:13:25.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:25.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:25.515603+0000) 2022-09-10T06:13:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:25 smithi110 ceph-mon[97633]: pgmap v3338: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:13:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:25 smithi104 ceph-mon[97044]: pgmap v3338: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:13:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:26.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:26.110321+0000) 2022-09-10T06:13:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:26.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:26.515814+0000) 2022-09-10T06:13:27.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:27.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:27.110510+0000) 2022-09-10T06:13:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:27.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:27.516017+0000) 2022-09-10T06:13:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:27 smithi110 ceph-mon[97633]: pgmap v3339: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:13:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:27 smithi104 ceph-mon[97044]: pgmap v3339: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:13:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:28.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:28.110701+0000) 2022-09-10T06:13:28.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:13:27.875+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:13:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:28.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:28.516227+0000) 2022-09-10T06:13:29.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:29.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:29.110924+0000) 2022-09-10T06:13:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:29.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:29.516444+0000) 2022-09-10T06:13:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:29 smithi110 ceph-mon[97633]: pgmap v3340: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:29 smithi104 ceph-mon[97044]: pgmap v3340: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:30.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:30.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:30.111054+0000) 2022-09-10T06:13:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:30.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:30.516659+0000) 2022-09-10T06:13:31.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:31.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:31.111233+0000) 2022-09-10T06:13:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:31.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:31.516884+0000) 2022-09-10T06:13:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:31 smithi110 ceph-mon[97633]: pgmap v3341: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:31 smithi104 ceph-mon[97044]: pgmap v3341: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:32.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:32.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:32.111429+0000) 2022-09-10T06:13:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:32.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:32.517036+0000) 2022-09-10T06:13:33.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:33.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:33.111660+0000) 2022-09-10T06:13:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:33.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:33.517270+0000) 2022-09-10T06:13:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:33 smithi110 ceph-mon[97633]: pgmap v3342: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:33 smithi104 ceph-mon[97044]: pgmap v3342: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:34.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:34.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:34.111865+0000) 2022-09-10T06:13:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:34.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:34.517507+0000) 2022-09-10T06:13:35.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:35.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:35.112136+0000) 2022-09-10T06:13:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:35.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:35.517765+0000) 2022-09-10T06:13:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:35 smithi110 ceph-mon[97633]: pgmap v3343: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:35 smithi104 ceph-mon[97044]: pgmap v3343: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:36.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:36.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:36.112405+0000) 2022-09-10T06:13:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:36.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:36.518011+0000) 2022-09-10T06:13:37.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:37.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:37.112638+0000) 2022-09-10T06:13:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:37.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:37.518242+0000) 2022-09-10T06:13:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:37 smithi110 ceph-mon[97633]: pgmap v3344: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:38.056 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:37 smithi104 ceph-mon[97044]: pgmap v3344: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:38.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:38.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:38.112896+0000) 2022-09-10T06:13:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:38.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:38.518465+0000) 2022-09-10T06:13:39.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:39.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:39.113041+0000) 2022-09-10T06:13:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:39.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:39.518698+0000) 2022-09-10T06:13:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:39 smithi110 ceph-mon[97633]: pgmap v3345: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:39 smithi104 ceph-mon[97044]: pgmap v3345: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:40.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:40.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:40.113284+0000) 2022-09-10T06:13:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:40.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:40.518929+0000) 2022-09-10T06:13:41.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:41.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:41.113502+0000) 2022-09-10T06:13:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:41.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:41.519163+0000) 2022-09-10T06:13:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:41 smithi110 ceph-mon[97633]: pgmap v3346: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:41 smithi104 ceph-mon[97044]: pgmap v3346: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:42.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:42.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:42.113670+0000) 2022-09-10T06:13:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:42.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:42.519397+0000) 2022-09-10T06:13:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:13:42.884+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:13:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:43.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:43.113895+0000) 2022-09-10T06:13:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:43.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:43.519626+0000) 2022-09-10T06:13:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:43 smithi110 ceph-mon[97633]: pgmap v3347: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:43 smithi104 ceph-mon[97044]: pgmap v3347: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:44.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:44.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:44.114153+0000) 2022-09-10T06:13:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:44.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:44.519817+0000) 2022-09-10T06:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:13:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:13:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:13:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:13:45.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:45.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:45.114363+0000) 2022-09-10T06:13:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:45.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:45.520057+0000) 2022-09-10T06:13:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:45 smithi110 ceph-mon[97633]: pgmap v3348: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:45 smithi104 ceph-mon[97044]: pgmap v3348: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:46.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:46.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:46.114611+0000) 2022-09-10T06:13:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:46.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:46.520327+0000) 2022-09-10T06:13:47.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:47.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:47.114868+0000) 2022-09-10T06:13:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:47.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:47.520557+0000) 2022-09-10T06:13:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:47 smithi110 ceph-mon[97633]: pgmap v3349: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:47 smithi104 ceph-mon[97044]: pgmap v3349: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:48.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:48.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:48.115079+0000) 2022-09-10T06:13:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:48.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:48.520856+0000) 2022-09-10T06:13:49.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:49.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:49.115384+0000) 2022-09-10T06:13:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:49.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:49.521080+0000) 2022-09-10T06:13:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:49 smithi110 ceph-mon[97633]: pgmap v3350: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:49 smithi104 ceph-mon[97044]: pgmap v3350: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:13:50.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:50.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:50.115591+0000) 2022-09-10T06:13:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:50.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:50.521219+0000) 2022-09-10T06:13:51.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:51.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:51.115780+0000) 2022-09-10T06:13:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:51.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:51.521463+0000) 2022-09-10T06:13:51.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:51 smithi110 ceph-mon[97633]: pgmap v3351: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:51 smithi104 ceph-mon[97044]: pgmap v3351: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:52.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:52.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:52.115975+0000) 2022-09-10T06:13:52.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:52.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:52.521733+0000) 2022-09-10T06:13:53.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:53.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:53.116219+0000) 2022-09-10T06:13:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:53.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:53.521972+0000) 2022-09-10T06:13:54.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:53 smithi104 ceph-mon[97044]: pgmap v3352: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:13:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:53 smithi110 ceph-mon[97633]: pgmap v3352: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:13:54.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:54.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:54.116405+0000) 2022-09-10T06:13:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:54.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:54.522183+0000) 2022-09-10T06:13:55.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:55.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:55.116591+0000) 2022-09-10T06:13:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:55.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:55.522410+0000) 2022-09-10T06:13:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:55 smithi104 ceph-mon[97044]: pgmap v3353: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:13:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:55 smithi110 ceph-mon[97633]: pgmap v3353: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:13:56.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:56.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:56.116796+0000) 2022-09-10T06:13:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:56.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:56.522685+0000) 2022-09-10T06:13:57.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:57.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:57.116973+0000) 2022-09-10T06:13:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:57.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:57.522843+0000) 2022-09-10T06:13:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:57 smithi104 ceph-mon[97044]: pgmap v3354: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:57 smithi110 ceph-mon[97633]: pgmap v3354: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:13:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:13:57.896+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:13:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:58.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:58.117205+0000) 2022-09-10T06:13:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:58.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:58.523064+0000) 2022-09-10T06:13:59.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:13:59.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:59.117448+0000) 2022-09-10T06:13:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:13:59.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:13:59.523322+0000) 2022-09-10T06:14:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:13:59 smithi104 ceph-mon[97044]: pgmap v3355: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:14:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:13:59 smithi110 ceph-mon[97633]: pgmap v3355: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:14:00.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:00.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:00.117680+0000) 2022-09-10T06:14:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:00.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:00.523528+0000) 2022-09-10T06:14:01.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:01.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:01.117816+0000) 2022-09-10T06:14:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:01.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:01.523758+0000) 2022-09-10T06:14:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:01 smithi104 ceph-mon[97044]: pgmap v3356: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:01 smithi110 ceph-mon[97633]: pgmap v3356: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:02.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:02.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:02.118011+0000) 2022-09-10T06:14:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:02.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:02.523901+0000) 2022-09-10T06:14:03.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:03.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:03.118227+0000) 2022-09-10T06:14:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:03.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:03.524143+0000) 2022-09-10T06:14:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:03 smithi104 ceph-mon[97044]: pgmap v3357: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:03 smithi110 ceph-mon[97633]: pgmap v3357: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:04.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:04.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:04.118447+0000) 2022-09-10T06:14:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:04.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:04.524372+0000) 2022-09-10T06:14:05.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:05.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:05.118704+0000) 2022-09-10T06:14:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:05.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:05.524615+0000) 2022-09-10T06:14:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:05 smithi104 ceph-mon[97044]: pgmap v3358: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:05 smithi110 ceph-mon[97633]: pgmap v3358: 129 pgs: 129 active+clean; 8.0 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:06.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:06.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:06.118870+0000) 2022-09-10T06:14:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:06.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:06.524859+0000) 2022-09-10T06:14:07.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:07.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:07.119123+0000) 2022-09-10T06:14:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:07.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:07.525043+0000) 2022-09-10T06:14:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:07 smithi104 ceph-mon[97044]: pgmap v3359: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:14:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:07 smithi110 ceph-mon[97633]: pgmap v3359: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:14:08.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:08.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:08.119366+0000) 2022-09-10T06:14:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:08.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:08.525187+0000) 2022-09-10T06:14:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:08 smithi104 ceph-mon[97044]: osdmap e292: 12 total, 12 up, 12 in 2022-09-10T06:14:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:08 smithi110 ceph-mon[97633]: osdmap e292: 12 total, 12 up, 12 in 2022-09-10T06:14:09.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:09.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:09.119503+0000) 2022-09-10T06:14:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:09.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:09.525315+0000) 2022-09-10T06:14:10.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:09 smithi104 ceph-mon[97044]: pgmap v3361: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:14:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:09 smithi110 ceph-mon[97633]: pgmap v3361: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:14:10.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:10.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:10.119711+0000) 2022-09-10T06:14:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:10.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:10.525518+0000) 2022-09-10T06:14:11.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:11.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:11.119967+0000) 2022-09-10T06:14:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:11.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:11.525732+0000) 2022-09-10T06:14:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:11 smithi104 ceph-mon[97044]: pgmap v3362: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:14:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:11 smithi110 ceph-mon[97633]: pgmap v3362: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:14:12.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:12.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:12.120189+0000) 2022-09-10T06:14:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:12.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:12.526026+0000) 2022-09-10T06:14:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:14:12.910+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:14:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:13.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:13.120415+0000) 2022-09-10T06:14:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:13.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:13.526236+0000) 2022-09-10T06:14:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:13 smithi104 ceph-mon[97044]: pgmap v3363: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:14:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:13 smithi104 ceph-mon[97044]: osdmap e293: 12 total, 12 up, 12 in 2022-09-10T06:14:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:13 smithi110 ceph-mon[97633]: pgmap v3363: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T06:14:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:13 smithi110 ceph-mon[97633]: osdmap e293: 12 total, 12 up, 12 in 2022-09-10T06:14:14.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:14.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:14.120617+0000) 2022-09-10T06:14:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:14.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:14.526447+0000) 2022-09-10T06:14:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:14 smithi104 ceph-mon[97044]: pgmap v3365: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:14:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:14 smithi110 ceph-mon[97633]: pgmap v3365: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:14:15.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:15.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:15.120820+0000) 2022-09-10T06:14:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:15.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:15.526640+0000) 2022-09-10T06:14:16.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:16.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:16.121087+0000) 2022-09-10T06:14:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:16.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:16.526849+0000) 2022-09-10T06:14:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:16 smithi110 ceph-mon[97633]: pgmap v3366: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T06:14:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:16 smithi104 ceph-mon[97044]: pgmap v3366: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T06:14:17.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:17.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:17.121294+0000) 2022-09-10T06:14:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:17.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:17.527071+0000) 2022-09-10T06:14:18.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:18.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:18.121477+0000) 2022-09-10T06:14:18.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:18.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:18.527341+0000) 2022-09-10T06:14:19.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:19.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:19.121727+0000) 2022-09-10T06:14:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:19.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:19.527489+0000) 2022-09-10T06:14:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:19 smithi110 ceph-mon[97633]: pgmap v3367: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:14:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:19 smithi104 ceph-mon[97044]: pgmap v3367: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:14:20.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:20.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:20.121959+0000) 2022-09-10T06:14:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:20.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:20.527679+0000) 2022-09-10T06:14:21.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:21.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:21.122128+0000) 2022-09-10T06:14:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:21.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:21.527898+0000) 2022-09-10T06:14:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:21 smithi110 ceph-mon[97633]: pgmap v3368: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:14:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:21 smithi104 ceph-mon[97044]: pgmap v3368: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:14:22.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:22.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:22.122314+0000) 2022-09-10T06:14:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:22.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:22.528181+0000) 2022-09-10T06:14:23.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:23.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:23.122468+0000) 2022-09-10T06:14:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:23.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:23.528337+0000) 2022-09-10T06:14:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:23 smithi110 ceph-mon[97633]: pgmap v3369: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:14:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:23 smithi104 ceph-mon[97044]: pgmap v3369: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:14:24.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:24.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:24.122672+0000) 2022-09-10T06:14:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:24.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:24.528503+0000) 2022-09-10T06:14:25.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:25.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:25.122919+0000) 2022-09-10T06:14:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:25.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:25.528689+0000) 2022-09-10T06:14:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:25 smithi110 ceph-mon[97633]: pgmap v3370: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:14:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:25 smithi104 ceph-mon[97044]: pgmap v3370: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:14:26.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:26.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:26.123193+0000) 2022-09-10T06:14:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:26.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:26.528892+0000) 2022-09-10T06:14:27.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:27.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:27.123395+0000) 2022-09-10T06:14:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:27.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:27.529183+0000) 2022-09-10T06:14:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:27 smithi110 ceph-mon[97633]: pgmap v3371: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:14:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:27 smithi104 ceph-mon[97044]: pgmap v3371: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:14:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:14:27.925+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:14:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:28.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:28.123623+0000) 2022-09-10T06:14:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:28.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:28.529390+0000) 2022-09-10T06:14:29.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:29.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:29.123814+0000) 2022-09-10T06:14:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:29.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:29.529636+0000) 2022-09-10T06:14:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:29 smithi110 ceph-mon[97633]: pgmap v3372: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:29 smithi104 ceph-mon[97044]: pgmap v3372: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:30.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:30.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:30.124043+0000) 2022-09-10T06:14:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:30.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:30.529887+0000) 2022-09-10T06:14:31.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:31.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:31.124224+0000) 2022-09-10T06:14:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:31.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:31.530139+0000) 2022-09-10T06:14:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:31 smithi110 ceph-mon[97633]: pgmap v3373: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:31 smithi104 ceph-mon[97044]: pgmap v3373: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:32.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:32.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:32.124420+0000) 2022-09-10T06:14:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:32.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:32.530352+0000) 2022-09-10T06:14:33.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:33.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:33.124670+0000) 2022-09-10T06:14:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:33.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:33.530562+0000) 2022-09-10T06:14:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:33 smithi110 ceph-mon[97633]: pgmap v3374: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:33 smithi104 ceph-mon[97044]: pgmap v3374: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:34.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:34.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:34.124926+0000) 2022-09-10T06:14:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:34.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:34.530774+0000) 2022-09-10T06:14:35.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:35.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:35.125186+0000) 2022-09-10T06:14:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:35.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:35.530948+0000) 2022-09-10T06:14:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:35 smithi110 ceph-mon[97633]: pgmap v3375: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:35 smithi104 ceph-mon[97044]: pgmap v3375: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:36.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:36.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:36.125369+0000) 2022-09-10T06:14:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:36.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:36.531193+0000) 2022-09-10T06:14:37.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:37.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:37.125558+0000) 2022-09-10T06:14:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:37.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:37.531423+0000) 2022-09-10T06:14:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:37 smithi110 ceph-mon[97633]: pgmap v3376: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:37 smithi104 ceph-mon[97044]: pgmap v3376: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:38.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:38.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:38.125857+0000) 2022-09-10T06:14:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:38.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:38.531621+0000) 2022-09-10T06:14:39.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:39.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:39.126092+0000) 2022-09-10T06:14:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:39.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:39.531850+0000) 2022-09-10T06:14:39.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:39 smithi110 ceph-mon[97633]: pgmap v3377: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:40.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:39 smithi104 ceph-mon[97044]: pgmap v3377: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:40.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:40.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:40.126386+0000) 2022-09-10T06:14:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:40.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:40.532066+0000) 2022-09-10T06:14:41.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:41.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:41.126642+0000) 2022-09-10T06:14:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:41.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:41.532286+0000) 2022-09-10T06:14:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:41 smithi110 ceph-mon[97633]: pgmap v3378: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:41 smithi104 ceph-mon[97044]: pgmap v3378: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:42.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:42.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:42.126896+0000) 2022-09-10T06:14:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:42.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:42.532504+0000) 2022-09-10T06:14:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:43.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:43.127050+0000) 2022-09-10T06:14:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:14:42.937+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:14:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:43.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:43.532722+0000) 2022-09-10T06:14:43.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:43 smithi110 ceph-mon[97633]: pgmap v3379: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:43 smithi104 ceph-mon[97044]: pgmap v3379: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:44.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:44.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:44.127255+0000) 2022-09-10T06:14:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:44.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:44.532981+0000) 2022-09-10T06:14:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:14:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:14:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:14:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:14:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:14:45.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:14:45.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:14:45.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:14:45.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:45.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:45.127455+0000) 2022-09-10T06:14:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:45.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:45.533189+0000) 2022-09-10T06:14:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:45 smithi110 ceph-mon[97633]: pgmap v3380: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:45 smithi104 ceph-mon[97044]: pgmap v3380: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:46.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:46.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:46.127686+0000) 2022-09-10T06:14:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:46.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:46.533390+0000) 2022-09-10T06:14:47.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:47.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:47.127911+0000) 2022-09-10T06:14:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:47.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:47.533534+0000) 2022-09-10T06:14:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:47 smithi110 ceph-mon[97633]: pgmap v3381: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:47 smithi104 ceph-mon[97044]: pgmap v3381: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:48.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:48.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:48.128169+0000) 2022-09-10T06:14:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:48.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:48.533752+0000) 2022-09-10T06:14:49.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:49.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:49.128391+0000) 2022-09-10T06:14:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:49.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:49.533983+0000) 2022-09-10T06:14:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:49 smithi110 ceph-mon[97633]: pgmap v3382: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:49 smithi104 ceph-mon[97044]: pgmap v3382: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:50.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:50.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:50.128629+0000) 2022-09-10T06:14:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:50.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:50.534214+0000) 2022-09-10T06:14:51.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:51.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:51.128804+0000) 2022-09-10T06:14:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:51.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:51.534417+0000) 2022-09-10T06:14:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:51 smithi110 ceph-mon[97633]: pgmap v3383: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:51 smithi104 ceph-mon[97044]: pgmap v3383: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:52.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:52.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:52.129034+0000) 2022-09-10T06:14:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:52.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:52.534661+0000) 2022-09-10T06:14:53.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:53.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:53.129235+0000) 2022-09-10T06:14:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:53.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:53.534888+0000) 2022-09-10T06:14:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:53 smithi104 ceph-mon[97044]: pgmap v3384: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:53 smithi110 ceph-mon[97633]: pgmap v3384: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:54.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:54.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:54.129359+0000) 2022-09-10T06:14:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:54.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:54.535172+0000) 2022-09-10T06:14:55.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:55.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:55.129534+0000) 2022-09-10T06:14:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:55.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:55.535425+0000) 2022-09-10T06:14:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:55 smithi104 ceph-mon[97044]: pgmap v3385: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:55 smithi110 ceph-mon[97633]: pgmap v3385: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:14:56.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:56.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:56.129761+0000) 2022-09-10T06:14:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:56.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:56.535717+0000) 2022-09-10T06:14:57.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:57.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:57.129959+0000) 2022-09-10T06:14:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:57.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:57.535906+0000) 2022-09-10T06:14:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:57 smithi104 ceph-mon[97044]: pgmap v3386: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:57 smithi110 ceph-mon[97633]: pgmap v3386: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:14:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:14:57.947+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:14:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:58.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:58.130079+0000) 2022-09-10T06:14:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:58.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:58.536072+0000) 2022-09-10T06:14:59.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:14:59.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:59.130313+0000) 2022-09-10T06:14:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:14:59.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:14:59.536351+0000) 2022-09-10T06:15:00.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:14:59 smithi104 ceph-mon[97044]: pgmap v3387: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:14:59 smithi110 ceph-mon[97633]: pgmap v3387: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:00.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:00.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:00.130477+0000) 2022-09-10T06:15:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:00.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:00.536587+0000) 2022-09-10T06:15:01.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:01.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:01.130636+0000) 2022-09-10T06:15:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:01.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:01.536805+0000) 2022-09-10T06:15:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:01 smithi104 ceph-mon[97044]: pgmap v3388: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:01 smithi110 ceph-mon[97633]: pgmap v3388: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:02.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:02.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:02.130809+0000) 2022-09-10T06:15:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:02.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:02.536953+0000) 2022-09-10T06:15:03.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:03.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:03.131024+0000) 2022-09-10T06:15:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:03.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:03.537153+0000) 2022-09-10T06:15:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:03 smithi104 ceph-mon[97044]: pgmap v3389: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:03 smithi110 ceph-mon[97633]: pgmap v3389: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:04.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:04.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:04.131256+0000) 2022-09-10T06:15:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:04.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:04.537415+0000) 2022-09-10T06:15:05.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:05.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:05.131543+0000) 2022-09-10T06:15:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:05.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:05.537548+0000) 2022-09-10T06:15:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:05 smithi104 ceph-mon[97044]: pgmap v3390: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:05 smithi110 ceph-mon[97633]: pgmap v3390: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:06.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:06.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:06.131738+0000) 2022-09-10T06:15:06.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:06.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:06.537768+0000) 2022-09-10T06:15:07.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:07.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:07.131941+0000) 2022-09-10T06:15:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:07.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:07.538015+0000) 2022-09-10T06:15:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:07 smithi104 ceph-mon[97044]: pgmap v3391: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:15:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:07 smithi110 ceph-mon[97633]: pgmap v3391: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:15:08.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:08.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:08.132198+0000) 2022-09-10T06:15:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:08.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:08.538197+0000) 2022-09-10T06:15:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:08 smithi104 ceph-mon[97044]: osdmap e294: 12 total, 12 up, 12 in 2022-09-10T06:15:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:08 smithi110 ceph-mon[97633]: osdmap e294: 12 total, 12 up, 12 in 2022-09-10T06:15:09.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:09.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:09.132361+0000) 2022-09-10T06:15:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:09.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:09.538418+0000) 2022-09-10T06:15:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:09 smithi104 ceph-mon[97044]: pgmap v3393: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:15:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:09 smithi110 ceph-mon[97633]: pgmap v3393: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:15:10.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:10.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:10.132650+0000) 2022-09-10T06:15:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:10.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:10.538614+0000) 2022-09-10T06:15:11.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:11.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:11.132812+0000) 2022-09-10T06:15:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:11.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:11.538829+0000) 2022-09-10T06:15:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:11 smithi104 ceph-mon[97044]: pgmap v3394: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:15:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:11 smithi110 ceph-mon[97633]: pgmap v3394: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:15:12.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:12.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:12.132973+0000) 2022-09-10T06:15:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:12.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:12.539055+0000) 2022-09-10T06:15:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:13.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:13.133133+0000) 2022-09-10T06:15:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:15:12.963+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:15:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:13.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:13.539208+0000) 2022-09-10T06:15:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:13 smithi104 ceph-mon[97044]: pgmap v3395: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:15:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:13 smithi104 ceph-mon[97044]: osdmap e295: 12 total, 12 up, 12 in 2022-09-10T06:15:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:13 smithi110 ceph-mon[97633]: pgmap v3395: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:15:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:13 smithi110 ceph-mon[97633]: osdmap e295: 12 total, 12 up, 12 in 2022-09-10T06:15:14.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:14.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:14.133291+0000) 2022-09-10T06:15:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:14.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:14.539442+0000) 2022-09-10T06:15:15.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:15.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:15.133492+0000) 2022-09-10T06:15:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:15.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:15.539664+0000) 2022-09-10T06:15:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:15 smithi104 ceph-mon[97044]: pgmap v3397: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:15:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:15 smithi110 ceph-mon[97633]: pgmap v3397: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:15:16.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:16.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:16.133720+0000) 2022-09-10T06:15:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:16.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:16.539914+0000) 2022-09-10T06:15:17.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:17.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:17.133966+0000) 2022-09-10T06:15:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:17.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:17.540159+0000) 2022-09-10T06:15:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:17 smithi104 ceph-mon[97044]: pgmap v3398: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T06:15:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:17 smithi110 ceph-mon[97633]: pgmap v3398: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T06:15:18.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:18.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:18.134209+0000) 2022-09-10T06:15:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:18.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:18.540404+0000) 2022-09-10T06:15:19.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:19.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:19.134358+0000) 2022-09-10T06:15:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:19.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:19.540591+0000) 2022-09-10T06:15:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:19 smithi104 ceph-mon[97044]: pgmap v3399: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:15:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:19 smithi110 ceph-mon[97633]: pgmap v3399: 129 pgs: 129 active+clean; 8.0 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:15:20.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:20.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:20.134572+0000) 2022-09-10T06:15:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:20.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:20.540834+0000) 2022-09-10T06:15:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:20 smithi104 ceph-mon[97044]: pgmap v3400: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:15:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:20 smithi110 ceph-mon[97633]: pgmap v3400: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:15:21.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:21.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:21.134777+0000) 2022-09-10T06:15:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:21.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:21.541017+0000) 2022-09-10T06:15:22.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:22.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:22.135013+0000) 2022-09-10T06:15:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:22.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:22.541181+0000) 2022-09-10T06:15:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:22 smithi110 ceph-mon[97633]: pgmap v3401: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:15:23.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:22 smithi104 ceph-mon[97044]: pgmap v3401: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:15:23.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:23.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:23.135248+0000) 2022-09-10T06:15:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:23.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:23.541360+0000) 2022-09-10T06:15:24.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:24.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:24.135494+0000) 2022-09-10T06:15:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:24.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:24.541621+0000) 2022-09-10T06:15:25.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:25.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:25.135729+0000) 2022-09-10T06:15:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:25.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:25.541876+0000) 2022-09-10T06:15:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:25 smithi110 ceph-mon[97633]: pgmap v3402: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:15:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:25 smithi104 ceph-mon[97044]: pgmap v3402: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:15:26.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:26.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:26.135910+0000) 2022-09-10T06:15:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:26.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:26.542089+0000) 2022-09-10T06:15:27.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:27.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:27.136189+0000) 2022-09-10T06:15:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:27.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:27.542358+0000) 2022-09-10T06:15:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:27 smithi110 ceph-mon[97633]: pgmap v3403: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:15:28.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:27 smithi104 ceph-mon[97044]: pgmap v3403: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:15:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:28.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:28.136348+0000) 2022-09-10T06:15:28.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:15:27.979+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:15:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:28.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:28.542627+0000) 2022-09-10T06:15:29.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:29.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:29.136588+0000) 2022-09-10T06:15:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:29.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:29.542901+0000) 2022-09-10T06:15:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:29 smithi110 ceph-mon[97633]: pgmap v3404: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:29 smithi104 ceph-mon[97044]: pgmap v3404: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:30.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:30.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:30.136847+0000) 2022-09-10T06:15:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:30.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:30.543070+0000) 2022-09-10T06:15:31.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:31.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:31.137051+0000) 2022-09-10T06:15:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:31.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:31.543304+0000) 2022-09-10T06:15:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:31 smithi110 ceph-mon[97633]: pgmap v3405: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:31 smithi104 ceph-mon[97044]: pgmap v3405: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:32.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:32.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:32.137223+0000) 2022-09-10T06:15:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:32.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:32.543573+0000) 2022-09-10T06:15:33.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:33.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:33.137472+0000) 2022-09-10T06:15:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:33.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:33.543830+0000) 2022-09-10T06:15:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:33 smithi110 ceph-mon[97633]: pgmap v3406: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:33 smithi104 ceph-mon[97044]: pgmap v3406: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:34.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:34.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:34.137732+0000) 2022-09-10T06:15:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:34.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:34.544151+0000) 2022-09-10T06:15:35.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:35.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:35.137977+0000) 2022-09-10T06:15:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:35.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:35.544423+0000) 2022-09-10T06:15:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:35 smithi110 ceph-mon[97633]: pgmap v3407: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:35 smithi104 ceph-mon[97044]: pgmap v3407: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:36.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:36.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:36.138224+0000) 2022-09-10T06:15:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:36.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:36.544705+0000) 2022-09-10T06:15:37.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:37.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:37.138465+0000) 2022-09-10T06:15:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:37.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:37.544930+0000) 2022-09-10T06:15:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:37 smithi110 ceph-mon[97633]: pgmap v3408: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:37 smithi104 ceph-mon[97044]: pgmap v3408: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:38.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:38.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:38.138707+0000) 2022-09-10T06:15:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:38.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:38.545223+0000) 2022-09-10T06:15:39.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:39.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:39.138925+0000) 2022-09-10T06:15:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:39.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:39.545436+0000) 2022-09-10T06:15:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:39 smithi110 ceph-mon[97633]: pgmap v3409: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:39 smithi104 ceph-mon[97044]: pgmap v3409: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:40.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:40.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:40.139190+0000) 2022-09-10T06:15:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:40.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:40.545695+0000) 2022-09-10T06:15:41.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:41.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:41.139403+0000) 2022-09-10T06:15:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:41.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:41.545888+0000) 2022-09-10T06:15:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:41 smithi110 ceph-mon[97633]: pgmap v3410: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:41 smithi104 ceph-mon[97044]: pgmap v3410: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:42.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:42.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:42.139651+0000) 2022-09-10T06:15:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:42.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:42.546149+0000) 2022-09-10T06:15:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:15:42.994+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:15:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:43.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:43.139891+0000) 2022-09-10T06:15:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:43.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:43.546397+0000) 2022-09-10T06:15:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:43 smithi110 ceph-mon[97633]: pgmap v3411: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:43 smithi104 ceph-mon[97044]: pgmap v3411: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:44.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:44.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:44.140126+0000) 2022-09-10T06:15:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:44.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:44.546655+0000) 2022-09-10T06:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:15:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:15:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:15:45.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:15:45.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:15:45.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:15:45.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:15:45.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:45.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:45.140352+0000) 2022-09-10T06:15:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:45.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:45.546922+0000) 2022-09-10T06:15:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:45 smithi104 ceph-mon[97044]: pgmap v3412: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:45 smithi110 ceph-mon[97633]: pgmap v3412: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:46.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:46.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:46.140572+0000) 2022-09-10T06:15:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:46.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:46.547169+0000) 2022-09-10T06:15:47.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:47.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:47.140710+0000) 2022-09-10T06:15:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:47.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:47.547392+0000) 2022-09-10T06:15:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:47 smithi104 ceph-mon[97044]: pgmap v3413: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:47 smithi110 ceph-mon[97633]: pgmap v3413: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:48.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:48.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:48.140889+0000) 2022-09-10T06:15:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:48.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:48.547633+0000) 2022-09-10T06:15:49.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:49.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:49.141080+0000) 2022-09-10T06:15:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:49.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:49.547861+0000) 2022-09-10T06:15:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:49 smithi104 ceph-mon[97044]: pgmap v3414: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:49 smithi110 ceph-mon[97633]: pgmap v3414: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:50.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:50.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:50.141364+0000) 2022-09-10T06:15:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:50.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:50.548016+0000) 2022-09-10T06:15:51.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:51.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:51.141552+0000) 2022-09-10T06:15:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:51.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:51.548173+0000) 2022-09-10T06:15:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:51 smithi104 ceph-mon[97044]: pgmap v3415: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:51 smithi110 ceph-mon[97633]: pgmap v3415: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:52.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:52.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:52.141744+0000) 2022-09-10T06:15:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:52.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:52.548399+0000) 2022-09-10T06:15:53.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:53.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:53.141916+0000) 2022-09-10T06:15:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:53.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:53.548634+0000) 2022-09-10T06:15:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:53 smithi104 ceph-mon[97044]: pgmap v3416: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:54.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:53 smithi110 ceph-mon[97633]: pgmap v3416: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:54.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:54.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:54.142198+0000) 2022-09-10T06:15:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:54.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:54.548917+0000) 2022-09-10T06:15:55.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:55.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:55.142477+0000) 2022-09-10T06:15:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:55.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:55.549145+0000) 2022-09-10T06:15:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:55 smithi104 ceph-mon[97044]: pgmap v3417: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:55 smithi110 ceph-mon[97633]: pgmap v3417: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:15:56.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:56.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:56.142746+0000) 2022-09-10T06:15:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:56.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:56.549332+0000) 2022-09-10T06:15:57.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:57.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:57.142933+0000) 2022-09-10T06:15:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:57.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:57.549527+0000) 2022-09-10T06:15:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:57 smithi104 ceph-mon[97044]: pgmap v3418: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:57 smithi110 ceph-mon[97633]: pgmap v3418: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:15:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:15:58.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:15:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:58.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:58.143202+0000) 2022-09-10T06:15:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:58.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:58.549753+0000) 2022-09-10T06:15:59.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:15:59.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:59.143396+0000) 2022-09-10T06:15:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:15:59.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:15:59.549941+0000) 2022-09-10T06:16:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:15:59 smithi104 ceph-mon[97044]: pgmap v3419: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:15:59 smithi110 ceph-mon[97633]: pgmap v3419: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:00.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:00.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:00.143653+0000) 2022-09-10T06:16:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:00.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:00.550183+0000) 2022-09-10T06:16:01.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:01.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:01.143920+0000) 2022-09-10T06:16:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:01.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:01.550389+0000) 2022-09-10T06:16:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:01 smithi104 ceph-mon[97044]: pgmap v3420: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:01 smithi110 ceph-mon[97633]: pgmap v3420: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:02.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:02.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:02.144190+0000) 2022-09-10T06:16:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:02.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:02.550611+0000) 2022-09-10T06:16:03.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:03.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:03.144445+0000) 2022-09-10T06:16:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:03.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:03.550854+0000) 2022-09-10T06:16:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:03 smithi104 ceph-mon[97044]: pgmap v3421: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:03 smithi110 ceph-mon[97633]: pgmap v3421: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:04.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:04.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:04.144706+0000) 2022-09-10T06:16:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:04.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:04.551127+0000) 2022-09-10T06:16:05.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:05.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:05.144944+0000) 2022-09-10T06:16:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:05.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:05.551355+0000) 2022-09-10T06:16:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:05 smithi104 ceph-mon[97044]: pgmap v3422: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:05 smithi110 ceph-mon[97633]: pgmap v3422: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:06.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:06.145212+0000) 2022-09-10T06:16:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:06.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:06.551599+0000) 2022-09-10T06:16:07.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:07.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:07.145368+0000) 2022-09-10T06:16:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:07.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:07.551849+0000) 2022-09-10T06:16:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:07 smithi104 ceph-mon[97044]: pgmap v3423: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:16:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:07 smithi110 ceph-mon[97633]: pgmap v3423: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:16:08.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:08.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:08.145572+0000) 2022-09-10T06:16:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:08.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:08.552085+0000) 2022-09-10T06:16:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:08 smithi104 ceph-mon[97044]: osdmap e296: 12 total, 12 up, 12 in 2022-09-10T06:16:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:08 smithi110 ceph-mon[97633]: osdmap e296: 12 total, 12 up, 12 in 2022-09-10T06:16:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:09.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:09.145795+0000) 2022-09-10T06:16:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:09.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:09.552363+0000) 2022-09-10T06:16:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:09 smithi104 ceph-mon[97044]: pgmap v3425: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:16:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:09 smithi110 ceph-mon[97633]: pgmap v3425: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:16:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:10.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:10.146127+0000) 2022-09-10T06:16:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:10.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:10.552593+0000) 2022-09-10T06:16:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:11.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:11.146341+0000) 2022-09-10T06:16:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:11.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:11.552803+0000) 2022-09-10T06:16:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:11 smithi104 ceph-mon[97044]: pgmap v3426: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:16:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:11 smithi110 ceph-mon[97633]: pgmap v3426: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:16:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:12.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:12.146515+0000) 2022-09-10T06:16:12.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:12.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:12.553029+0000) 2022-09-10T06:16:13.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:16:13.020+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:16:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:13.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:13.146757+0000) 2022-09-10T06:16:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:13.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:13.553235+0000) 2022-09-10T06:16:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:13 smithi104 ceph-mon[97044]: pgmap v3427: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:16:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:13 smithi104 ceph-mon[97044]: osdmap e297: 12 total, 12 up, 12 in 2022-09-10T06:16:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:13 smithi110 ceph-mon[97633]: pgmap v3427: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:16:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:13 smithi110 ceph-mon[97633]: osdmap e297: 12 total, 12 up, 12 in 2022-09-10T06:16:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:14.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:14.146924+0000) 2022-09-10T06:16:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:14.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:14.553425+0000) 2022-09-10T06:16:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:15.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:15.147197+0000) 2022-09-10T06:16:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:15 smithi104 ceph-mon[97044]: pgmap v3429: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:16:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:15.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:15.553622+0000) 2022-09-10T06:16:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:15 smithi110 ceph-mon[97633]: pgmap v3429: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:16:16.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:16.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:16.147399+0000) 2022-09-10T06:16:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:16.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:16.553817+0000) 2022-09-10T06:16:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:17.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:17.147578+0000) 2022-09-10T06:16:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:17 smithi104 ceph-mon[97044]: pgmap v3430: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T06:16:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:17.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:17.554001+0000) 2022-09-10T06:16:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:17 smithi110 ceph-mon[97633]: pgmap v3430: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T06:16:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:18.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:18.147856+0000) 2022-09-10T06:16:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:18.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:18.554160+0000) 2022-09-10T06:16:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:19.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:19.148053+0000) 2022-09-10T06:16:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:19 smithi104 ceph-mon[97044]: pgmap v3431: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:16:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:19.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:19.554301+0000) 2022-09-10T06:16:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:19 smithi110 ceph-mon[97633]: pgmap v3431: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:16:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:20.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:20.148274+0000) 2022-09-10T06:16:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:20 smithi104 ceph-mon[97044]: pgmap v3432: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T06:16:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:20.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:20.554495+0000) 2022-09-10T06:16:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:20 smithi110 ceph-mon[97633]: pgmap v3432: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T06:16:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:21.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:21.148436+0000) 2022-09-10T06:16:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:21.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:21.554674+0000) 2022-09-10T06:16:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:22.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:22.148616+0000) 2022-09-10T06:16:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:22.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:22.554860+0000) 2022-09-10T06:16:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:22 smithi110 ceph-mon[97633]: pgmap v3433: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T06:16:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:22 smithi104 ceph-mon[97044]: pgmap v3433: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T06:16:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:23.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:23.148786+0000) 2022-09-10T06:16:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:23.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:23.555132+0000) 2022-09-10T06:16:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:24.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:24.149039+0000) 2022-09-10T06:16:25.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:24.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:24.555353+0000) 2022-09-10T06:16:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:25.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:25.149313+0000) 2022-09-10T06:16:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:25 smithi110 ceph-mon[97633]: pgmap v3434: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:16:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:25 smithi104 ceph-mon[97044]: pgmap v3434: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:16:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:25.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:25.555539+0000) 2022-09-10T06:16:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:26.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:26.149492+0000) 2022-09-10T06:16:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:26.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:26.555785+0000) 2022-09-10T06:16:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:27.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:27.149755+0000) 2022-09-10T06:16:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:27 smithi110 ceph-mon[97633]: pgmap v3435: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:16:28.030 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:27.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:27.555998+0000) 2022-09-10T06:16:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:27 smithi104 ceph-mon[97044]: pgmap v3435: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:16:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:16:28.027+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:16:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:28.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:28.149951+0000) 2022-09-10T06:16:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:28.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:28.556257+0000) 2022-09-10T06:16:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:29.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:29.150172+0000) 2022-09-10T06:16:29.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:29 smithi110 ceph-mon[97633]: pgmap v3436: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:29 smithi104 ceph-mon[97044]: pgmap v3436: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:16:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:29.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:29.556471+0000) 2022-09-10T06:16:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:30.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:30.150342+0000) 2022-09-10T06:16:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:30.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:30.556663+0000) 2022-09-10T06:16:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:31.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:31.150515+0000) 2022-09-10T06:16:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:31 smithi110 ceph-mon[97633]: pgmap v3437: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:31 smithi104 ceph-mon[97044]: pgmap v3437: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:31.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:31.556855+0000) 2022-09-10T06:16:32.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:32.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:32.150694+0000) 2022-09-10T06:16:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:32.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:32.557123+0000) 2022-09-10T06:16:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:33.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:33.150904+0000) 2022-09-10T06:16:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:33 smithi110 ceph-mon[97633]: pgmap v3438: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:33 smithi104 ceph-mon[97044]: pgmap v3438: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:33.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:33.557308+0000) 2022-09-10T06:16:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:34.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:34.151067+0000) 2022-09-10T06:16:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:34.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:34.557472+0000) 2022-09-10T06:16:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:35.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:35.151259+0000) 2022-09-10T06:16:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:35 smithi110 ceph-mon[97633]: pgmap v3439: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:35 smithi104 ceph-mon[97044]: pgmap v3439: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:36.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:35.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:35.557670+0000) 2022-09-10T06:16:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:36.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:36.151493+0000) 2022-09-10T06:16:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:36.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:36.557918+0000) 2022-09-10T06:16:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:37.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:37.151696+0000) 2022-09-10T06:16:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:37 smithi104 ceph-mon[97044]: pgmap v3440: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:37.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:37.558164+0000) 2022-09-10T06:16:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:37 smithi110 ceph-mon[97633]: pgmap v3440: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:38.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:38.151865+0000) 2022-09-10T06:16:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:38.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:38.558337+0000) 2022-09-10T06:16:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:39.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:39.152071+0000) 2022-09-10T06:16:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:39 smithi104 ceph-mon[97044]: pgmap v3441: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:39.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:39.558484+0000) 2022-09-10T06:16:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:39 smithi110 ceph-mon[97633]: pgmap v3441: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:40.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:40.152365+0000) 2022-09-10T06:16:41.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:40.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:40.558679+0000) 2022-09-10T06:16:41.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:41.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:41.152534+0000) 2022-09-10T06:16:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:41 smithi104 ceph-mon[97044]: pgmap v3442: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:41.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:41.558866+0000) 2022-09-10T06:16:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:41 smithi110 ceph-mon[97633]: pgmap v3442: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:42.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:42.152749+0000) 2022-09-10T06:16:43.044 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:42.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:42.559114+0000) 2022-09-10T06:16:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:43.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:43.152995+0000) 2022-09-10T06:16:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:16:43.042+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:16:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:43 smithi104 ceph-mon[97044]: pgmap v3443: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:43.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:43.559312+0000) 2022-09-10T06:16:44.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:43 smithi110 ceph-mon[97633]: pgmap v3443: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:44.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:44.153287+0000) 2022-09-10T06:16:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:16:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:16:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:16:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:16:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:44.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:44.559529+0000) 2022-09-10T06:16:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:16:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:16:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:16:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:16:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:45.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:45.153439+0000) 2022-09-10T06:16:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:45 smithi104 ceph-mon[97044]: pgmap v3444: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:45.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:45.559728+0000) 2022-09-10T06:16:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:45 smithi110 ceph-mon[97633]: pgmap v3444: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:46.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:46.153686+0000) 2022-09-10T06:16:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:46.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:46.559955+0000) 2022-09-10T06:16:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:47.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:47.153926+0000) 2022-09-10T06:16:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:47 smithi104 ceph-mon[97044]: pgmap v3445: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:47.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:47.560223+0000) 2022-09-10T06:16:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:47 smithi110 ceph-mon[97633]: pgmap v3445: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:48.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:48.154206+0000) 2022-09-10T06:16:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:48.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:48.560362+0000) 2022-09-10T06:16:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:49.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:49.154445+0000) 2022-09-10T06:16:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:49 smithi104 ceph-mon[97044]: pgmap v3446: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:49.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:49.560508+0000) 2022-09-10T06:16:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:49 smithi110 ceph-mon[97633]: pgmap v3446: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:50.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:50.154706+0000) 2022-09-10T06:16:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:50.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:50.560712+0000) 2022-09-10T06:16:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:51.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:51.154924+0000) 2022-09-10T06:16:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:51 smithi104 ceph-mon[97044]: pgmap v3447: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:51.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:51.560990+0000) 2022-09-10T06:16:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:51 smithi110 ceph-mon[97633]: pgmap v3447: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:52.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:52.155164+0000) 2022-09-10T06:16:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:52.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:52.561240+0000) 2022-09-10T06:16:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:53.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:53.155411+0000) 2022-09-10T06:16:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:53 smithi104 ceph-mon[97044]: pgmap v3448: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:53.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:53.561456+0000) 2022-09-10T06:16:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:53 smithi110 ceph-mon[97633]: pgmap v3448: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:54.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:54.155601+0000) 2022-09-10T06:16:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:54.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:54.561767+0000) 2022-09-10T06:16:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:55.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:55.155780+0000) 2022-09-10T06:16:56.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:55 smithi104 ceph-mon[97044]: pgmap v3449: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:55.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:55.562048+0000) 2022-09-10T06:16:56.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:55 smithi110 ceph-mon[97633]: pgmap v3449: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:16:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:56.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:56.156017+0000) 2022-09-10T06:16:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:56.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:56.562242+0000) 2022-09-10T06:16:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:57.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:57.156265+0000) 2022-09-10T06:16:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:57 smithi104 ceph-mon[97044]: pgmap v3450: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:58.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:57.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:57.562385+0000) 2022-09-10T06:16:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:57 smithi110 ceph-mon[97633]: pgmap v3450: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:16:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:58.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:58.156478+0000) 2022-09-10T06:16:58.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:16:58.058+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:16:58.870 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:58.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:58.562572+0000) 2022-09-10T06:16:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:16:59.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:59.156695+0000) 2022-09-10T06:17:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:16:59 smithi104 ceph-mon[97044]: pgmap v3451: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:16:59.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:16:59.562837+0000) 2022-09-10T06:17:00.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:16:59 smithi110 ceph-mon[97633]: pgmap v3451: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:00.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:00.156983+0000) 2022-09-10T06:17:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:00.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:00.563035+0000) 2022-09-10T06:17:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:01.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:01.157226+0000) 2022-09-10T06:17:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:01 smithi104 ceph-mon[97044]: pgmap v3452: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:01.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:01.563265+0000) 2022-09-10T06:17:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:01 smithi110 ceph-mon[97633]: pgmap v3452: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:02.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:02.157491+0000) 2022-09-10T06:17:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:02.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:02.563434+0000) 2022-09-10T06:17:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:03.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:03.157735+0000) 2022-09-10T06:17:04.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:03 smithi104 ceph-mon[97044]: pgmap v3453: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:03.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:03.563688+0000) 2022-09-10T06:17:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:03 smithi110 ceph-mon[97633]: pgmap v3453: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:04.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:04.157967+0000) 2022-09-10T06:17:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:04.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:04.563957+0000) 2022-09-10T06:17:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:05.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:05.158236+0000) 2022-09-10T06:17:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:05 smithi104 ceph-mon[97044]: pgmap v3454: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:05.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:05.564223+0000) 2022-09-10T06:17:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:05 smithi110 ceph-mon[97633]: pgmap v3454: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:06.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:06.158404+0000) 2022-09-10T06:17:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:06.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:06.564483+0000) 2022-09-10T06:17:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:07.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:07.158621+0000) 2022-09-10T06:17:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:07 smithi104 ceph-mon[97044]: pgmap v3455: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:17:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:07.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:07.564716+0000) 2022-09-10T06:17:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:07 smithi110 ceph-mon[97633]: pgmap v3455: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:17:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:08.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:08.158860+0000) 2022-09-10T06:17:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:08 smithi104 ceph-mon[97044]: osdmap e298: 12 total, 12 up, 12 in 2022-09-10T06:17:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:08.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:08.564994+0000) 2022-09-10T06:17:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:08 smithi110 ceph-mon[97633]: osdmap e298: 12 total, 12 up, 12 in 2022-09-10T06:17:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:09.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:09.159135+0000) 2022-09-10T06:17:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:09 smithi104 ceph-mon[97044]: pgmap v3457: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:17:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:09.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:09.565263+0000) 2022-09-10T06:17:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:09 smithi110 ceph-mon[97633]: pgmap v3457: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:17:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:10.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:10.159405+0000) 2022-09-10T06:17:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:10.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:10.565502+0000) 2022-09-10T06:17:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:11.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:11.159655+0000) 2022-09-10T06:17:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:11 smithi104 ceph-mon[97044]: pgmap v3458: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:17:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:11.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:11.565737+0000) 2022-09-10T06:17:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:11 smithi110 ceph-mon[97633]: pgmap v3458: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:17:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:12.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:12.159919+0000) 2022-09-10T06:17:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:12 smithi104 ceph-mon[97044]: pgmap v3459: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:17:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:12.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:12.565941+0000) 2022-09-10T06:17:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:12 smithi110 ceph-mon[97633]: pgmap v3459: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:17:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:13.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:13.160191+0000) 2022-09-10T06:17:13.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:17:13.071+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:17:14.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:13.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:13.566138+0000) 2022-09-10T06:17:14.569 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:14.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:14.160367+0000) 2022-09-10T06:17:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:14 smithi110 ceph-mon[97633]: osdmap e299: 12 total, 12 up, 12 in 2022-09-10T06:17:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:14 smithi104 ceph-mon[97044]: osdmap e299: 12 total, 12 up, 12 in 2022-09-10T06:17:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:14.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:14.566387+0000) 2022-09-10T06:17:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:15.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:15.160607+0000) 2022-09-10T06:17:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:15 smithi104 ceph-mon[97044]: pgmap v3461: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:17:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:15.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:15.566611+0000) 2022-09-10T06:17:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:15 smithi110 ceph-mon[97633]: pgmap v3461: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:17:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:16.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:16.160851+0000) 2022-09-10T06:17:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:16.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:16.566854+0000) 2022-09-10T06:17:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:17.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:17.161089+0000) 2022-09-10T06:17:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:17 smithi104 ceph-mon[97044]: pgmap v3462: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T06:17:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:17.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:17.567149+0000) 2022-09-10T06:17:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:17 smithi110 ceph-mon[97633]: pgmap v3462: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T06:17:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:18.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:18.161394+0000) 2022-09-10T06:17:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:18.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:18.567277+0000) 2022-09-10T06:17:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:19.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:19.161627+0000) 2022-09-10T06:17:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:19 smithi104 ceph-mon[97044]: pgmap v3463: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:17:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:19.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:19.567459+0000) 2022-09-10T06:17:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:19 smithi110 ceph-mon[97633]: pgmap v3463: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:17:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:20.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:20.161850+0000) 2022-09-10T06:17:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:20.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:20.567652+0000) 2022-09-10T06:17:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:21.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:21.162077+0000) 2022-09-10T06:17:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:21 smithi104 ceph-mon[97044]: pgmap v3464: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:17:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:21.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:21.567897+0000) 2022-09-10T06:17:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:21 smithi110 ceph-mon[97633]: pgmap v3464: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:17:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:22.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:22.162298+0000) 2022-09-10T06:17:23.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:22.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:22.568181+0000) 2022-09-10T06:17:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:23.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:23.162502+0000) 2022-09-10T06:17:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:23 smithi104 ceph-mon[97044]: pgmap v3465: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:17:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:23.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:23.568385+0000) 2022-09-10T06:17:24.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:23 smithi110 ceph-mon[97633]: pgmap v3465: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:17:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:24.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:24.162705+0000) 2022-09-10T06:17:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:24.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:24.568633+0000) 2022-09-10T06:17:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:25.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:25.162908+0000) 2022-09-10T06:17:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:25 smithi104 ceph-mon[97044]: pgmap v3466: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:17:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:25.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:25.568847+0000) 2022-09-10T06:17:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:25 smithi110 ceph-mon[97633]: pgmap v3466: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:17:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:26.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:26.163148+0000) 2022-09-10T06:17:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:26.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:26.569079+0000) 2022-09-10T06:17:27.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:27.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:27.163343+0000) 2022-09-10T06:17:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:27 smithi104 ceph-mon[97044]: pgmap v3467: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:17:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:27.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:27.569277+0000) 2022-09-10T06:17:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:27 smithi110 ceph-mon[97633]: pgmap v3467: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:17:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:17:28.087+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:17:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:28.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:28.163573+0000) 2022-09-10T06:17:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:28.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:28.569485+0000) 2022-09-10T06:17:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:29.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:29.163798+0000) 2022-09-10T06:17:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:29 smithi104 ceph-mon[97044]: pgmap v3468: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:30.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:29.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:29.569728+0000) 2022-09-10T06:17:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:29 smithi110 ceph-mon[97633]: pgmap v3468: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:30.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:30.164044+0000) 2022-09-10T06:17:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:30.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:30.569934+0000) 2022-09-10T06:17:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:31.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:31.164196+0000) 2022-09-10T06:17:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:31 smithi104 ceph-mon[97044]: pgmap v3469: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:31.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:31.570221+0000) 2022-09-10T06:17:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:31 smithi110 ceph-mon[97633]: pgmap v3469: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:32.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:32.164424+0000) 2022-09-10T06:17:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:32.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:32.570475+0000) 2022-09-10T06:17:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:33.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:33.164627+0000) 2022-09-10T06:17:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:33 smithi104 ceph-mon[97044]: pgmap v3470: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:33.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:33.570717+0000) 2022-09-10T06:17:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:33 smithi110 ceph-mon[97633]: pgmap v3470: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:34.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:34.164886+0000) 2022-09-10T06:17:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:34.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:34.570966+0000) 2022-09-10T06:17:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:35.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:35.165088+0000) 2022-09-10T06:17:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:35 smithi104 ceph-mon[97044]: pgmap v3471: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:35.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:35.571132+0000) 2022-09-10T06:17:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:35 smithi110 ceph-mon[97633]: pgmap v3471: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:36.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:36.165296+0000) 2022-09-10T06:17:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:36.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:36.571333+0000) 2022-09-10T06:17:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:37.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:37.165526+0000) 2022-09-10T06:17:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:37 smithi104 ceph-mon[97044]: pgmap v3472: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:37.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:37.571529+0000) 2022-09-10T06:17:38.172 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:37 smithi110 ceph-mon[97633]: pgmap v3472: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:38.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:38.165701+0000) 2022-09-10T06:17:39.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:38.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:38.571726+0000) 2022-09-10T06:17:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:39.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:39.165988+0000) 2022-09-10T06:17:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:39 smithi104 ceph-mon[97044]: pgmap v3473: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:39.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:39.571960+0000) 2022-09-10T06:17:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:39 smithi110 ceph-mon[97633]: pgmap v3473: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:40.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:40.166135+0000) 2022-09-10T06:17:41.070 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:40.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:40.572155+0000) 2022-09-10T06:17:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:41.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:41.166306+0000) 2022-09-10T06:17:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:41 smithi104 ceph-mon[97044]: pgmap v3474: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:41.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:41.572296+0000) 2022-09-10T06:17:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:41 smithi110 ceph-mon[97633]: pgmap v3474: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:42.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:42.166521+0000) 2022-09-10T06:17:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:42.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:42.572462+0000) 2022-09-10T06:17:43.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:17:43.099+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:17:43.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:43.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:43.166718+0000) 2022-09-10T06:17:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:43 smithi104 ceph-mon[97044]: pgmap v3475: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:43.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:43.572629+0000) 2022-09-10T06:17:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:43 smithi110 ceph-mon[97633]: pgmap v3475: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:44.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:44.166891+0000) 2022-09-10T06:17:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:17:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:17:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:17:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:17:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:44.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:44.572853+0000) 2022-09-10T06:17:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:17:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:17:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:17:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:17:45.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:45.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:45.167151+0000) 2022-09-10T06:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:45 smithi104 ceph-mon[97044]: pgmap v3476: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:45.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:45.573013+0000) 2022-09-10T06:17:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:45 smithi110 ceph-mon[97633]: pgmap v3476: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:46.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:46.167406+0000) 2022-09-10T06:17:47.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:46.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:46.573211+0000) 2022-09-10T06:17:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:47.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:47.167623+0000) 2022-09-10T06:17:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:47 smithi104 ceph-mon[97044]: pgmap v3477: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:47.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:47.573403+0000) 2022-09-10T06:17:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:47 smithi110 ceph-mon[97633]: pgmap v3477: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:48.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:48.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:48.167803+0000) 2022-09-10T06:17:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:48.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:48.573616+0000) 2022-09-10T06:17:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:49.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:49.168067+0000) 2022-09-10T06:17:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:49 smithi104 ceph-mon[97044]: pgmap v3478: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:49.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:49.573865+0000) 2022-09-10T06:17:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:49 smithi110 ceph-mon[97633]: pgmap v3478: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:50.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:50.168298+0000) 2022-09-10T06:17:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:50.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:50.574089+0000) 2022-09-10T06:17:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:51.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:51.168504+0000) 2022-09-10T06:17:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:51 smithi104 ceph-mon[97044]: pgmap v3479: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:51.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:51.574332+0000) 2022-09-10T06:17:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:51 smithi110 ceph-mon[97633]: pgmap v3479: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:52.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:52.168713+0000) 2022-09-10T06:17:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:52.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:52.574476+0000) 2022-09-10T06:17:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:53.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:53.168909+0000) 2022-09-10T06:17:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:53 smithi104 ceph-mon[97044]: pgmap v3480: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:54.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:53.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:53.574637+0000) 2022-09-10T06:17:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:53 smithi110 ceph-mon[97633]: pgmap v3480: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:54.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:54.169166+0000) 2022-09-10T06:17:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:54.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:54.574898+0000) 2022-09-10T06:17:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:55.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:55.169374+0000) 2022-09-10T06:17:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:55 smithi104 ceph-mon[97044]: pgmap v3481: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:55.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:55.575179+0000) 2022-09-10T06:17:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:55 smithi110 ceph-mon[97633]: pgmap v3481: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:17:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:56.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:56.169521+0000) 2022-09-10T06:17:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:56.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:56.575343+0000) 2022-09-10T06:17:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:57.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:57.169708+0000) 2022-09-10T06:17:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:57 smithi104 ceph-mon[97044]: pgmap v3482: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:57.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:57.575483+0000) 2022-09-10T06:17:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:57 smithi110 ceph-mon[97633]: pgmap v3482: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:17:58.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:17:58.114+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:17:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:58.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:58.169948+0000) 2022-09-10T06:17:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:58.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:58.575639+0000) 2022-09-10T06:17:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:17:59.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:59.170202+0000) 2022-09-10T06:18:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:17:59 smithi104 ceph-mon[97044]: pgmap v3483: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:17:59.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:17:59.575874+0000) 2022-09-10T06:18:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:17:59 smithi110 ceph-mon[97633]: pgmap v3483: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:00.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:00.170394+0000) 2022-09-10T06:18:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:00.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:00.576162+0000) 2022-09-10T06:18:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:01.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:01.170581+0000) 2022-09-10T06:18:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:01 smithi104 ceph-mon[97044]: pgmap v3484: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:01.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:01.576289+0000) 2022-09-10T06:18:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:01 smithi110 ceph-mon[97633]: pgmap v3484: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:02.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:02.170813+0000) 2022-09-10T06:18:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:02 smithi104 ceph-mon[97044]: pgmap v3485: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:02.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:02.576510+0000) 2022-09-10T06:18:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:02 smithi110 ceph-mon[97633]: pgmap v3485: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:03.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:03.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:03.170978+0000) 2022-09-10T06:18:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:03.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:03.576761+0000) 2022-09-10T06:18:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:04.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:04.171197+0000) 2022-09-10T06:18:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:04.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:04.576962+0000) 2022-09-10T06:18:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:05.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:05.171431+0000) 2022-09-10T06:18:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:05 smithi104 ceph-mon[97044]: pgmap v3486: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:05.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:05.577198+0000) 2022-09-10T06:18:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:05 smithi110 ceph-mon[97633]: pgmap v3486: 129 pgs: 129 active+clean; 8.0 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:06.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:06.171686+0000) 2022-09-10T06:18:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:06.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:06.577322+0000) 2022-09-10T06:18:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:07.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:07.171891+0000) 2022-09-10T06:18:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:07 smithi104 ceph-mon[97044]: pgmap v3487: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:18:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:07.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:07.577576+0000) 2022-09-10T06:18:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:07 smithi110 ceph-mon[97633]: pgmap v3487: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:18:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:08.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:08.172051+0000) 2022-09-10T06:18:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:08 smithi104 ceph-mon[97044]: osdmap e300: 12 total, 12 up, 12 in 2022-09-10T06:18:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:08.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:08.577805+0000) 2022-09-10T06:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:08 smithi110 ceph-mon[97633]: osdmap e300: 12 total, 12 up, 12 in 2022-09-10T06:18:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:09.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:09.172249+0000) 2022-09-10T06:18:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:09 smithi104 ceph-mon[97044]: pgmap v3489: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:18:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:09.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:09.577980+0000) 2022-09-10T06:18:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:09 smithi110 ceph-mon[97633]: pgmap v3489: 129 pgs: 129 active+clean; 8.0 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:18:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:10.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:10.172420+0000) 2022-09-10T06:18:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:10.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:10.578183+0000) 2022-09-10T06:18:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:11.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:11.172578+0000) 2022-09-10T06:18:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:11 smithi104 ceph-mon[97044]: pgmap v3490: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:18:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:11.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:11.578427+0000) 2022-09-10T06:18:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:11 smithi110 ceph-mon[97633]: pgmap v3490: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:18:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:12.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:12.172707+0000) 2022-09-10T06:18:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:12.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:12.578667+0000) 2022-09-10T06:18:13.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:18:13.122+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:18:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:13.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:13.172879+0000) 2022-09-10T06:18:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:13 smithi104 ceph-mon[97044]: pgmap v3491: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:18:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:13 smithi104 ceph-mon[97044]: osdmap e301: 12 total, 12 up, 12 in 2022-09-10T06:18:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:13.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:13.578899+0000) 2022-09-10T06:18:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:13 smithi110 ceph-mon[97633]: pgmap v3491: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:18:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:13 smithi110 ceph-mon[97633]: osdmap e301: 12 total, 12 up, 12 in 2022-09-10T06:18:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:14.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:14.173196+0000) 2022-09-10T06:18:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:14.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:14.579201+0000) 2022-09-10T06:18:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:15.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:15.173370+0000) 2022-09-10T06:18:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:15 smithi104 ceph-mon[97044]: pgmap v3493: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:18:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:15.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:15.579393+0000) 2022-09-10T06:18:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:15 smithi110 ceph-mon[97633]: pgmap v3493: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:18:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:16.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:16.173518+0000) 2022-09-10T06:18:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:16.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:16.579587+0000) 2022-09-10T06:18:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:17.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:17.173665+0000) 2022-09-10T06:18:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:17 smithi104 ceph-mon[97044]: pgmap v3494: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:18:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:17.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:17.579818+0000) 2022-09-10T06:18:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:17 smithi110 ceph-mon[97633]: pgmap v3494: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:18:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:18.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:18.173824+0000) 2022-09-10T06:18:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:18.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:18.580071+0000) 2022-09-10T06:18:19.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:19.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:19.174080+0000) 2022-09-10T06:18:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:19 smithi104 ceph-mon[97044]: pgmap v3495: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:18:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:19.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:19.580348+0000) 2022-09-10T06:18:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:19 smithi110 ceph-mon[97633]: pgmap v3495: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:18:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:20.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:20.174273+0000) 2022-09-10T06:18:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:20.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:20.580583+0000) 2022-09-10T06:18:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:21.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:21.174516+0000) 2022-09-10T06:18:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:21 smithi104 ceph-mon[97044]: pgmap v3496: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:18:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:21.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:21.580798+0000) 2022-09-10T06:18:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:21 smithi110 ceph-mon[97633]: pgmap v3496: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:18:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:22.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:22.174737+0000) 2022-09-10T06:18:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:22.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:22.580976+0000) 2022-09-10T06:18:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:23.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:23.174960+0000) 2022-09-10T06:18:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:23 smithi104 ceph-mon[97044]: pgmap v3497: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:18:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:23.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:23.581200+0000) 2022-09-10T06:18:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:23 smithi110 ceph-mon[97633]: pgmap v3497: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:18:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:24.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:24.175070+0000) 2022-09-10T06:18:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:24.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:24.581399+0000) 2022-09-10T06:18:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:25.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:25.175240+0000) 2022-09-10T06:18:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:25 smithi104 ceph-mon[97044]: pgmap v3498: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:18:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:25.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:25.581576+0000) 2022-09-10T06:18:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:25 smithi110 ceph-mon[97633]: pgmap v3498: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:18:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:26.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:26.175451+0000) 2022-09-10T06:18:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:26.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:26.581833+0000) 2022-09-10T06:18:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:27.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:27.175651+0000) 2022-09-10T06:18:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:27 smithi104 ceph-mon[97044]: pgmap v3499: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:18:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:27.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:27.582073+0000) 2022-09-10T06:18:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:27 smithi110 ceph-mon[97633]: pgmap v3499: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:18:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:28.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:28.175918+0000) 2022-09-10T06:18:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:18:28.133+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:18:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:28.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:28.582342+0000) 2022-09-10T06:18:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:29.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:29.176124+0000) 2022-09-10T06:18:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:29.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:29.582591+0000) 2022-09-10T06:18:30.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:29 smithi104 ceph-mon[97044]: pgmap v3500: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:29 smithi110 ceph-mon[97633]: pgmap v3500: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:30.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:30.176360+0000) 2022-09-10T06:18:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:30.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:30.582822+0000) 2022-09-10T06:18:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:31.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:31.176553+0000) 2022-09-10T06:18:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:31 smithi104 ceph-mon[97044]: pgmap v3501: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:31.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:31.583058+0000) 2022-09-10T06:18:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:31 smithi110 ceph-mon[97633]: pgmap v3501: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:32.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:32.176769+0000) 2022-09-10T06:18:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:32.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:32.583290+0000) 2022-09-10T06:18:33.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:33.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:33.177048+0000) 2022-09-10T06:18:34.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:33 smithi104 ceph-mon[97044]: pgmap v3502: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:33.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:33.583550+0000) 2022-09-10T06:18:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:33 smithi110 ceph-mon[97633]: pgmap v3502: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:34.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:34.177240+0000) 2022-09-10T06:18:35.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:34.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:34.583820+0000) 2022-09-10T06:18:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:35.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:35.177432+0000) 2022-09-10T06:18:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:35 smithi104 ceph-mon[97044]: pgmap v3503: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:35.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:35.584076+0000) 2022-09-10T06:18:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:35 smithi110 ceph-mon[97633]: pgmap v3503: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:36.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:36.177660+0000) 2022-09-10T06:18:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:36.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:36.584319+0000) 2022-09-10T06:18:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:37.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:37.177907+0000) 2022-09-10T06:18:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:37 smithi104 ceph-mon[97044]: pgmap v3504: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:37.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:37.584483+0000) 2022-09-10T06:18:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:37 smithi110 ceph-mon[97633]: pgmap v3504: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:38.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:38.178148+0000) 2022-09-10T06:18:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:38.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:38.584680+0000) 2022-09-10T06:18:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:39.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:39.178377+0000) 2022-09-10T06:18:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:39 smithi104 ceph-mon[97044]: pgmap v3505: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:39.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:39.584889+0000) 2022-09-10T06:18:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:39 smithi110 ceph-mon[97633]: pgmap v3505: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:40.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:40.178537+0000) 2022-09-10T06:18:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:40.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:40.585174+0000) 2022-09-10T06:18:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:41.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:41.178723+0000) 2022-09-10T06:18:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:41 smithi104 ceph-mon[97044]: pgmap v3506: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:42.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:41.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:41.585401+0000) 2022-09-10T06:18:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:41 smithi110 ceph-mon[97633]: pgmap v3506: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:42.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:42.178941+0000) 2022-09-10T06:18:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:42.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:42.585658+0000) 2022-09-10T06:18:43.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:43.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:43.179193+0000) 2022-09-10T06:18:43.578 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:18:43.150+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:18:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:43 smithi104 ceph-mon[97044]: pgmap v3507: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:43.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:43.585859+0000) 2022-09-10T06:18:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:43 smithi110 ceph-mon[97633]: pgmap v3507: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:44.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:44.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:44.179408+0000) 2022-09-10T06:18:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:18:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:18:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:18:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:18:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:44.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:44.586136+0000) 2022-09-10T06:18:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:18:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:18:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:18:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:18:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:45.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:45.179651+0000) 2022-09-10T06:18:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:45 smithi104 ceph-mon[97044]: pgmap v3508: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:45.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:45.586335+0000) 2022-09-10T06:18:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:45 smithi110 ceph-mon[97633]: pgmap v3508: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:46.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:46.179854+0000) 2022-09-10T06:18:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:46 smithi104 ceph-mon[97044]: pgmap v3509: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:46.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:46.586520+0000) 2022-09-10T06:18:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:46 smithi110 ceph-mon[97633]: pgmap v3509: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:47.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:47.180017+0000) 2022-09-10T06:18:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:47.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:47.586658+0000) 2022-09-10T06:18:48.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:48.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:48.180244+0000) 2022-09-10T06:18:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:48.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:48.586833+0000) 2022-09-10T06:18:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:49.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:49.180488+0000) 2022-09-10T06:18:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:49 smithi104 ceph-mon[97044]: pgmap v3510: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:49.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:49.587079+0000) 2022-09-10T06:18:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:49 smithi110 ceph-mon[97633]: pgmap v3510: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:50.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:50.180733+0000) 2022-09-10T06:18:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:50.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:50.587264+0000) 2022-09-10T06:18:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:51.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:51.180977+0000) 2022-09-10T06:18:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:51 smithi104 ceph-mon[97044]: pgmap v3511: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:51.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:51.587450+0000) 2022-09-10T06:18:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:51 smithi110 ceph-mon[97633]: pgmap v3511: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:52.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:52.181172+0000) 2022-09-10T06:18:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:52.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:52.587681+0000) 2022-09-10T06:18:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:53.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:53.181354+0000) 2022-09-10T06:18:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:53 smithi104 ceph-mon[97044]: pgmap v3512: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:53.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:53.587904+0000) 2022-09-10T06:18:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:53 smithi110 ceph-mon[97633]: pgmap v3512: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:54.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:54.181610+0000) 2022-09-10T06:18:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:54.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:54.588159+0000) 2022-09-10T06:18:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:55.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:55.181824+0000) 2022-09-10T06:18:56.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:55 smithi104 ceph-mon[97044]: pgmap v3513: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:55.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:55.588396+0000) 2022-09-10T06:18:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:55 smithi110 ceph-mon[97633]: pgmap v3513: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:18:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:56.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:56.182044+0000) 2022-09-10T06:18:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:56.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:56.588620+0000) 2022-09-10T06:18:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:57.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:57.182324+0000) 2022-09-10T06:18:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:57.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:57.588831+0000) 2022-09-10T06:18:58.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:57 smithi104 ceph-mon[97044]: pgmap v3514: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:57 smithi110 ceph-mon[97633]: pgmap v3514: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:18:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:58.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:58.182519+0000) 2022-09-10T06:18:58.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:18:58.155+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:18:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:58.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:58.589022+0000) 2022-09-10T06:18:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:18:59.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:59.182746+0000) 2022-09-10T06:19:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:18:59 smithi104 ceph-mon[97044]: pgmap v3515: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:18:59.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:18:59.589255+0000) 2022-09-10T06:19:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:18:59 smithi110 ceph-mon[97633]: pgmap v3515: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:00.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:00.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:00.182977+0000) 2022-09-10T06:19:01.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:00.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:00.589500+0000) 2022-09-10T06:19:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:01.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:01.183216+0000) 2022-09-10T06:19:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:01 smithi104 ceph-mon[97044]: pgmap v3516: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:01.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:01.589722+0000) 2022-09-10T06:19:02.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:01 smithi110 ceph-mon[97633]: pgmap v3516: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:02.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:02.183363+0000) 2022-09-10T06:19:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:02.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:02.589972+0000) 2022-09-10T06:19:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:03.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:03.183572+0000) 2022-09-10T06:19:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:03 smithi104 ceph-mon[97044]: pgmap v3517: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:03.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:03.590196+0000) 2022-09-10T06:19:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:03 smithi110 ceph-mon[97633]: pgmap v3517: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:04.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:04.183793+0000) 2022-09-10T06:19:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:04.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:04.590375+0000) 2022-09-10T06:19:05.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:05.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:05.183990+0000) 2022-09-10T06:19:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:05 smithi104 ceph-mon[97044]: pgmap v3518: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:06.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:05.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:05.590616+0000) 2022-09-10T06:19:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:05 smithi110 ceph-mon[97633]: pgmap v3518: 129 pgs: 129 active+clean; 8.0 MiB data, 494 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:06.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:06.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:06.184214+0000) 2022-09-10T06:19:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:06.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:06.590859+0000) 2022-09-10T06:19:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:07.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:07.184425+0000) 2022-09-10T06:19:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:07 smithi104 ceph-mon[97044]: pgmap v3519: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:19:08.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:07.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:07.590999+0000) 2022-09-10T06:19:08.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:07 smithi110 ceph-mon[97633]: pgmap v3519: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:19:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:08.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:08.184585+0000) 2022-09-10T06:19:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:08 smithi104 ceph-mon[97044]: osdmap e302: 12 total, 12 up, 12 in 2022-09-10T06:19:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:08.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:08.591229+0000) 2022-09-10T06:19:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:08 smithi110 ceph-mon[97633]: osdmap e302: 12 total, 12 up, 12 in 2022-09-10T06:19:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:09.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:09.184767+0000) 2022-09-10T06:19:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:09 smithi104 ceph-mon[97044]: pgmap v3521: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:19:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:09.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:09.591466+0000) 2022-09-10T06:19:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:09 smithi110 ceph-mon[97633]: pgmap v3521: 129 pgs: 129 active+clean; 8.0 MiB data, 496 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:19:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:10.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:10.184950+0000) 2022-09-10T06:19:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:10.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:10.591734+0000) 2022-09-10T06:19:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:11.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:11.185251+0000) 2022-09-10T06:19:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:11 smithi104 ceph-mon[97044]: pgmap v3522: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:19:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:11.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:11.591899+0000) 2022-09-10T06:19:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:11 smithi110 ceph-mon[97633]: pgmap v3522: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:19:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:12.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:12.185451+0000) 2022-09-10T06:19:13.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:12.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:12.592131+0000) 2022-09-10T06:19:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:13.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:13.185614+0000) 2022-09-10T06:19:13.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:19:13.160+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:19:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:13 smithi104 ceph-mon[97044]: pgmap v3523: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:19:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:13 smithi104 ceph-mon[97044]: osdmap e303: 12 total, 12 up, 12 in 2022-09-10T06:19:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:13.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:13.592317+0000) 2022-09-10T06:19:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:13 smithi110 ceph-mon[97633]: pgmap v3523: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:19:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:13 smithi110 ceph-mon[97633]: osdmap e303: 12 total, 12 up, 12 in 2022-09-10T06:19:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:14.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:14.185789+0000) 2022-09-10T06:19:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:14.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:14.592557+0000) 2022-09-10T06:19:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:15.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:15.185950+0000) 2022-09-10T06:19:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:15 smithi104 ceph-mon[97044]: pgmap v3525: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:19:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:15.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:15.592694+0000) 2022-09-10T06:19:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:15 smithi110 ceph-mon[97633]: pgmap v3525: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:19:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:16.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:16.186185+0000) 2022-09-10T06:19:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:16.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:16.592906+0000) 2022-09-10T06:19:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:17.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:17.186474+0000) 2022-09-10T06:19:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:17 smithi104 ceph-mon[97044]: pgmap v3526: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:19:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:17.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:17.593155+0000) 2022-09-10T06:19:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:17 smithi110 ceph-mon[97633]: pgmap v3526: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:19:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:18.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:18.186747+0000) 2022-09-10T06:19:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:18.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:18.593369+0000) 2022-09-10T06:19:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:19.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:19.187003+0000) 2022-09-10T06:19:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:19 smithi104 ceph-mon[97044]: pgmap v3527: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:19:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:19.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:19.593586+0000) 2022-09-10T06:19:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:19 smithi110 ceph-mon[97633]: pgmap v3527: 129 pgs: 129 active+clean; 8.0 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:19:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:20.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:20.187222+0000) 2022-09-10T06:19:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:20.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:20.593772+0000) 2022-09-10T06:19:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:21.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:21.187466+0000) 2022-09-10T06:19:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:21 smithi104 ceph-mon[97044]: pgmap v3528: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:19:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:21.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:21.593976+0000) 2022-09-10T06:19:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:21 smithi110 ceph-mon[97633]: pgmap v3528: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:19:22.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:22.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:22.187729+0000) 2022-09-10T06:19:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:22.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:22.594183+0000) 2022-09-10T06:19:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:23.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:23.187887+0000) 2022-09-10T06:19:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:23 smithi104 ceph-mon[97044]: pgmap v3529: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:19:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:23.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:23.594434+0000) 2022-09-10T06:19:24.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:23 smithi110 ceph-mon[97633]: pgmap v3529: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:19:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:24.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:24.188049+0000) 2022-09-10T06:19:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:24.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:24.594673+0000) 2022-09-10T06:19:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:25.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:25.188274+0000) 2022-09-10T06:19:26.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:25 smithi104 ceph-mon[97044]: pgmap v3530: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:19:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:25.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:25.594903+0000) 2022-09-10T06:19:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:25 smithi110 ceph-mon[97633]: pgmap v3530: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:19:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:26.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:26.188448+0000) 2022-09-10T06:19:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:26.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:26.595176+0000) 2022-09-10T06:19:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:27.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:27.188693+0000) 2022-09-10T06:19:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:27 smithi104 ceph-mon[97044]: pgmap v3531: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:19:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:27.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:27.595309+0000) 2022-09-10T06:19:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:27 smithi110 ceph-mon[97633]: pgmap v3531: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:19:28.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:19:28.166+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:19:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:28.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:28.188880+0000) 2022-09-10T06:19:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:28.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:28.595577+0000) 2022-09-10T06:19:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:29.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:29.189084+0000) 2022-09-10T06:19:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:29 smithi104 ceph-mon[97044]: pgmap v3532: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:29.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:29.595850+0000) 2022-09-10T06:19:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:29 smithi110 ceph-mon[97633]: pgmap v3532: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:30.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:30.189305+0000) 2022-09-10T06:19:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:30.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:30.596159+0000) 2022-09-10T06:19:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:31.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:31.189539+0000) 2022-09-10T06:19:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:31 smithi104 ceph-mon[97044]: pgmap v3533: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:31.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:31.596381+0000) 2022-09-10T06:19:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:31 smithi110 ceph-mon[97633]: pgmap v3533: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:32.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:32.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:32.189750+0000) 2022-09-10T06:19:33.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:32.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:32.596552+0000) 2022-09-10T06:19:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:33.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:33.190002+0000) 2022-09-10T06:19:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:33 smithi104 ceph-mon[97044]: pgmap v3534: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:33.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:33.596796+0000) 2022-09-10T06:19:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:33 smithi110 ceph-mon[97633]: pgmap v3534: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:34.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:34.190236+0000) 2022-09-10T06:19:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:34.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:34.597062+0000) 2022-09-10T06:19:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:35.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:35.190415+0000) 2022-09-10T06:19:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:35 smithi104 ceph-mon[97044]: pgmap v3535: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:35.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:35.597296+0000) 2022-09-10T06:19:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:35 smithi110 ceph-mon[97633]: pgmap v3535: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:36.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:36.190586+0000) 2022-09-10T06:19:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:36.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:36.597524+0000) 2022-09-10T06:19:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:37.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:37.190827+0000) 2022-09-10T06:19:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:37 smithi104 ceph-mon[97044]: pgmap v3536: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:37.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:37.597803+0000) 2022-09-10T06:19:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:37 smithi110 ceph-mon[97633]: pgmap v3536: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:38.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:38.191019+0000) 2022-09-10T06:19:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:38.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:38.598063+0000) 2022-09-10T06:19:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:39.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:39.191253+0000) 2022-09-10T06:19:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:39 smithi104 ceph-mon[97044]: pgmap v3537: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:39.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:39.598270+0000) 2022-09-10T06:19:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:39 smithi110 ceph-mon[97633]: pgmap v3537: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:40.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:40.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:40.191476+0000) 2022-09-10T06:19:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:40.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:40.598524+0000) 2022-09-10T06:19:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:41.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:41.191624+0000) 2022-09-10T06:19:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:41 smithi104 ceph-mon[97044]: pgmap v3538: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:41.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:41.598787+0000) 2022-09-10T06:19:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:41 smithi110 ceph-mon[97633]: pgmap v3538: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:42.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:42.191865+0000) 2022-09-10T06:19:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:42.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:42.598990+0000) 2022-09-10T06:19:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:43.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:43.192088+0000) 2022-09-10T06:19:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:19:43.177+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:19:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:43 smithi104 ceph-mon[97044]: pgmap v3539: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:43.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:43.599209+0000) 2022-09-10T06:19:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:43 smithi110 ceph-mon[97633]: pgmap v3539: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:44.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:44.192311+0000) 2022-09-10T06:19:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:19:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:19:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:19:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:19:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:44.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:44.599493+0000) 2022-09-10T06:19:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:19:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:19:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:19:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:19:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:45.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:45.192493+0000) 2022-09-10T06:19:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:45 smithi104 ceph-mon[97044]: pgmap v3540: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:45.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:45.599687+0000) 2022-09-10T06:19:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:45 smithi110 ceph-mon[97633]: pgmap v3540: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:46.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:46.192671+0000) 2022-09-10T06:19:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:46.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:46.599926+0000) 2022-09-10T06:19:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:47.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:47.192873+0000) 2022-09-10T06:19:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:47 smithi104 ceph-mon[97044]: pgmap v3541: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:47.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:47.600151+0000) 2022-09-10T06:19:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:47 smithi110 ceph-mon[97633]: pgmap v3541: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:48.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:48.193083+0000) 2022-09-10T06:19:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:48.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:48.600407+0000) 2022-09-10T06:19:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:49.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:49.193376+0000) 2022-09-10T06:19:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:49 smithi104 ceph-mon[97044]: pgmap v3542: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:50.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:49.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:49.600640+0000) 2022-09-10T06:19:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:49 smithi110 ceph-mon[97633]: pgmap v3542: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:50.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:50.193640+0000) 2022-09-10T06:19:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:50.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:50.600882+0000) 2022-09-10T06:19:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:51.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:51.193886+0000) 2022-09-10T06:19:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:51 smithi104 ceph-mon[97044]: pgmap v3543: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:51.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:51.601181+0000) 2022-09-10T06:19:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:51 smithi110 ceph-mon[97633]: pgmap v3543: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:52.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:52.194180+0000) 2022-09-10T06:19:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:52.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:52.601455+0000) 2022-09-10T06:19:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:53.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:53.194341+0000) 2022-09-10T06:19:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:53 smithi104 ceph-mon[97044]: pgmap v3544: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:53.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:53.601713+0000) 2022-09-10T06:19:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:53 smithi110 ceph-mon[97633]: pgmap v3544: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:54.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:54.194505+0000) 2022-09-10T06:19:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:54.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:54.602009+0000) 2022-09-10T06:19:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:55.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:55.194751+0000) 2022-09-10T06:19:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:55 smithi104 ceph-mon[97044]: pgmap v3545: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:56.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:55.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:55.602228+0000) 2022-09-10T06:19:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:55 smithi110 ceph-mon[97633]: pgmap v3545: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:19:56.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:56.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:56.194935+0000) 2022-09-10T06:19:57.067 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:56.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:56.602456+0000) 2022-09-10T06:19:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:57.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:57.195124+0000) 2022-09-10T06:19:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:57 smithi104 ceph-mon[97044]: pgmap v3546: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:57.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:57.602703+0000) 2022-09-10T06:19:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:57 smithi110 ceph-mon[97633]: pgmap v3546: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:19:58.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:19:58.185+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:19:58.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:58.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:58.195349+0000) 2022-09-10T06:19:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:58.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:58.602920+0000) 2022-09-10T06:19:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:19:59.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:59.195543+0000) 2022-09-10T06:20:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:19:59 smithi104 ceph-mon[97044]: pgmap v3547: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:19:59.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:19:59.603184+0000) 2022-09-10T06:20:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:19:59 smithi110 ceph-mon[97633]: pgmap v3547: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:00.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:00.195746+0000) 2022-09-10T06:20:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:20:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:00.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:00.603329+0000) 2022-09-10T06:20:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:20:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:01.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:01.195983+0000) 2022-09-10T06:20:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:01 smithi104 ceph-mon[97044]: pgmap v3548: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:01.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:01.603512+0000) 2022-09-10T06:20:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:01 smithi110 ceph-mon[97633]: pgmap v3548: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:02.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:02.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:02.196209+0000) 2022-09-10T06:20:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:02.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:02.603700+0000) 2022-09-10T06:20:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:03.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:03.196442+0000) 2022-09-10T06:20:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:03 smithi104 ceph-mon[97044]: pgmap v3549: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:03.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:03.603939+0000) 2022-09-10T06:20:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:03 smithi110 ceph-mon[97633]: pgmap v3549: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:04.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:04.196723+0000) 2022-09-10T06:20:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:04.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:04.604220+0000) 2022-09-10T06:20:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:04 smithi110 ceph-mon[97633]: pgmap v3550: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:04 smithi104 ceph-mon[97044]: pgmap v3550: 129 pgs: 129 active+clean; 8.0 MiB data, 505 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:05.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:05.196951+0000) 2022-09-10T06:20:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:05.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:05.604400+0000) 2022-09-10T06:20:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:06.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:06.197197+0000) 2022-09-10T06:20:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:06.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:06.604616+0000) 2022-09-10T06:20:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:06 smithi104 ceph-mon[97044]: pgmap v3551: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:20:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:06 smithi110 ceph-mon[97633]: pgmap v3551: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:20:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:07.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:07.197383+0000) 2022-09-10T06:20:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:07.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:07.604846+0000) 2022-09-10T06:20:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:08.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:08.197627+0000) 2022-09-10T06:20:09.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:08.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:08.605076+0000) 2022-09-10T06:20:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:08 smithi104 ceph-mon[97044]: pgmap v3552: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:20:09.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:08 smithi104 ceph-mon[97044]: osdmap e304: 12 total, 12 up, 12 in 2022-09-10T06:20:09.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:08 smithi110 ceph-mon[97633]: pgmap v3552: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:20:09.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:08 smithi110 ceph-mon[97633]: osdmap e304: 12 total, 12 up, 12 in 2022-09-10T06:20:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:09.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:09.197893+0000) 2022-09-10T06:20:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:09.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:09.605314+0000) 2022-09-10T06:20:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:10.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:10.198169+0000) 2022-09-10T06:20:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:10.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:10.605532+0000) 2022-09-10T06:20:11.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:10 smithi104 ceph-mon[97044]: pgmap v3554: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:20:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:10 smithi110 ceph-mon[97633]: pgmap v3554: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:20:11.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:11.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:11.198400+0000) 2022-09-10T06:20:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:11.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:11.605673+0000) 2022-09-10T06:20:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:12.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:12.198565+0000) 2022-09-10T06:20:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:12.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:12.605912+0000) 2022-09-10T06:20:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:12 smithi104 ceph-mon[97044]: pgmap v3555: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:20:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:12 smithi110 ceph-mon[97633]: pgmap v3555: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:20:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:13.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:13.198778+0000) 2022-09-10T06:20:13.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:20:13.190+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:20:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:13.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:13.606142+0000) 2022-09-10T06:20:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:14.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:14.198964+0000) 2022-09-10T06:20:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:14 smithi104 ceph-mon[97044]: osdmap e305: 12 total, 12 up, 12 in 2022-09-10T06:20:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:14.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:14.606373+0000) 2022-09-10T06:20:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:14 smithi110 ceph-mon[97633]: osdmap e305: 12 total, 12 up, 12 in 2022-09-10T06:20:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:15.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:15.199216+0000) 2022-09-10T06:20:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:15 smithi104 ceph-mon[97044]: pgmap v3557: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:20:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:15.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:15.606620+0000) 2022-09-10T06:20:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:15 smithi110 ceph-mon[97633]: pgmap v3557: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:20:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:16.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:16.199447+0000) 2022-09-10T06:20:17.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:16.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:16.606849+0000) 2022-09-10T06:20:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:17.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:17.199737+0000) 2022-09-10T06:20:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:17 smithi104 ceph-mon[97044]: pgmap v3558: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:20:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:17.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:17.607031+0000) 2022-09-10T06:20:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:17 smithi110 ceph-mon[97633]: pgmap v3558: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:20:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:18.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:18.199930+0000) 2022-09-10T06:20:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:18.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:18.607250+0000) 2022-09-10T06:20:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:19.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:19.200184+0000) 2022-09-10T06:20:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:19 smithi104 ceph-mon[97044]: pgmap v3559: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:20:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:19.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:19.607507+0000) 2022-09-10T06:20:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:19 smithi110 ceph-mon[97633]: pgmap v3559: 129 pgs: 129 active+clean; 8.0 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:20:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:20.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:20.200453+0000) 2022-09-10T06:20:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:20.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:20.607730+0000) 2022-09-10T06:20:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:21.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:21.200688+0000) 2022-09-10T06:20:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:21 smithi104 ceph-mon[97044]: pgmap v3560: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:20:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:21.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:21.607966+0000) 2022-09-10T06:20:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:21 smithi110 ceph-mon[97633]: pgmap v3560: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:20:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:22.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:22.200938+0000) 2022-09-10T06:20:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:22.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:22.608199+0000) 2022-09-10T06:20:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:23.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:23.201157+0000) 2022-09-10T06:20:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:23 smithi104 ceph-mon[97044]: pgmap v3561: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:20:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:23.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:23.608420+0000) 2022-09-10T06:20:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:23 smithi110 ceph-mon[97633]: pgmap v3561: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:20:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:24.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:24.201347+0000) 2022-09-10T06:20:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:24.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:24.608685+0000) 2022-09-10T06:20:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:25.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:25.201504+0000) 2022-09-10T06:20:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:25 smithi104 ceph-mon[97044]: pgmap v3562: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:20:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:25.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:25.608856+0000) 2022-09-10T06:20:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:25 smithi110 ceph-mon[97633]: pgmap v3562: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:20:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:26.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:26.201738+0000) 2022-09-10T06:20:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:26.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:26.609059+0000) 2022-09-10T06:20:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:27.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:27.201985+0000) 2022-09-10T06:20:28.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:27 smithi104 ceph-mon[97044]: pgmap v3563: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:20:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:27.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:27.609290+0000) 2022-09-10T06:20:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:27 smithi110 ceph-mon[97633]: pgmap v3563: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:20:28.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:20:28.201+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:20:28.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:28.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:28.202227+0000) 2022-09-10T06:20:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:28.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:28.609442+0000) 2022-09-10T06:20:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:29.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:29.202470+0000) 2022-09-10T06:20:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:29 smithi104 ceph-mon[97044]: pgmap v3564: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:20:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:29.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:29.609703+0000) 2022-09-10T06:20:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:29 smithi110 ceph-mon[97633]: pgmap v3564: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:20:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:30.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:30.202722+0000) 2022-09-10T06:20:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:30.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:30.609884+0000) 2022-09-10T06:20:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:31.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:31.202900+0000) 2022-09-10T06:20:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:31 smithi104 ceph-mon[97044]: pgmap v3565: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:31.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:31.610028+0000) 2022-09-10T06:20:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:31 smithi110 ceph-mon[97633]: pgmap v3565: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:32.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:32.203134+0000) 2022-09-10T06:20:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:32.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:32.610262+0000) 2022-09-10T06:20:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:33.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:33.203327+0000) 2022-09-10T06:20:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:33 smithi104 ceph-mon[97044]: pgmap v3566: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:20:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:33.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:33.610484+0000) 2022-09-10T06:20:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:33 smithi110 ceph-mon[97633]: pgmap v3566: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:20:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:34.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:34.203544+0000) 2022-09-10T06:20:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:34.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:34.610739+0000) 2022-09-10T06:20:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:35.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:35.203715+0000) 2022-09-10T06:20:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:35 smithi104 ceph-mon[97044]: pgmap v3567: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:20:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:35.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:35.610951+0000) 2022-09-10T06:20:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:35 smithi110 ceph-mon[97633]: pgmap v3567: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:20:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:36.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:36.203914+0000) 2022-09-10T06:20:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:36.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:36.611205+0000) 2022-09-10T06:20:37.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:37.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:37.204192+0000) 2022-09-10T06:20:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:37 smithi104 ceph-mon[97044]: pgmap v3568: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:37.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:37.611410+0000) 2022-09-10T06:20:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:37 smithi110 ceph-mon[97633]: pgmap v3568: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:38.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:38.204395+0000) 2022-09-10T06:20:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:38.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:38.611637+0000) 2022-09-10T06:20:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:39.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:39.204631+0000) 2022-09-10T06:20:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:39 smithi104 ceph-mon[97044]: pgmap v3569: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:39.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:39.611833+0000) 2022-09-10T06:20:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:39 smithi110 ceph-mon[97633]: pgmap v3569: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:40.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:40.204901+0000) 2022-09-10T06:20:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:40.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:40.612060+0000) 2022-09-10T06:20:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:41.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:41.205187+0000) 2022-09-10T06:20:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:41 smithi104 ceph-mon[97044]: pgmap v3570: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:41.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:41.612282+0000) 2022-09-10T06:20:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:41 smithi110 ceph-mon[97633]: pgmap v3570: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:42.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:42.205424+0000) 2022-09-10T06:20:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:42.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:42.612445+0000) 2022-09-10T06:20:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:43.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:43.205658+0000) 2022-09-10T06:20:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:20:43.217+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:20:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:43 smithi104 ceph-mon[97044]: pgmap v3571: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:43.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:43.612649+0000) 2022-09-10T06:20:44.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:43 smithi110 ceph-mon[97633]: pgmap v3571: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:44.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:44.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:44.205915+0000) 2022-09-10T06:20:45.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:20:45.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:20:45.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:20:45.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:20:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:44.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:44.612846+0000) 2022-09-10T06:20:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:20:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:20:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:20:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:20:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:45.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:45.206124+0000) 2022-09-10T06:20:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:45 smithi104 ceph-mon[97044]: pgmap v3572: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:45.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:45.613090+0000) 2022-09-10T06:20:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:45 smithi110 ceph-mon[97633]: pgmap v3572: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:46.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:46.206280+0000) 2022-09-10T06:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:46 smithi104 ceph-mon[97044]: pgmap v3573: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:46.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:46.613362+0000) 2022-09-10T06:20:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:46 smithi110 ceph-mon[97633]: pgmap v3573: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:47.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:47.206504+0000) 2022-09-10T06:20:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:47.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:47.613556+0000) 2022-09-10T06:20:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:48.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:48.206736+0000) 2022-09-10T06:20:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:48.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:48.613817+0000) 2022-09-10T06:20:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:49.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:49.206988+0000) 2022-09-10T06:20:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:49 smithi104 ceph-mon[97044]: pgmap v3574: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:49.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:49.614024+0000) 2022-09-10T06:20:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:49 smithi110 ceph-mon[97633]: pgmap v3574: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:50.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:50.207190+0000) 2022-09-10T06:20:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:50.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:50.614240+0000) 2022-09-10T06:20:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:51.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:51.207400+0000) 2022-09-10T06:20:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:51 smithi104 ceph-mon[97044]: pgmap v3575: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:52.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:51.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:51.614490+0000) 2022-09-10T06:20:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:51 smithi110 ceph-mon[97633]: pgmap v3575: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:52.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:52.207637+0000) 2022-09-10T06:20:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:52.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:52.614758+0000) 2022-09-10T06:20:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:53.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:53.207908+0000) 2022-09-10T06:20:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:53 smithi104 ceph-mon[97044]: pgmap v3576: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:53.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:53.614930+0000) 2022-09-10T06:20:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:53 smithi110 ceph-mon[97633]: pgmap v3576: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:54.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:54.208086+0000) 2022-09-10T06:20:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:54.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:54.615151+0000) 2022-09-10T06:20:55.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:55.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:55.208285+0000) 2022-09-10T06:20:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:55 smithi104 ceph-mon[97044]: pgmap v3577: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:55.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:55.615320+0000) 2022-09-10T06:20:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:55 smithi110 ceph-mon[97633]: pgmap v3577: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:20:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:56.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:56.208535+0000) 2022-09-10T06:20:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:56.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:56.615496+0000) 2022-09-10T06:20:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:57.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:57.208771+0000) 2022-09-10T06:20:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:57 smithi104 ceph-mon[97044]: pgmap v3578: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:58.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:57.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:57.615729+0000) 2022-09-10T06:20:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:57 smithi110 ceph-mon[97633]: pgmap v3578: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:20:58.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:20:58.226+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:20:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:58.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:58.209008+0000) 2022-09-10T06:20:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:58.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:58.615908+0000) 2022-09-10T06:20:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:20:59.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:59.209205+0000) 2022-09-10T06:21:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:20:59 smithi104 ceph-mon[97044]: pgmap v3579: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:00.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:20:59.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:20:59.616156+0000) 2022-09-10T06:21:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:20:59 smithi110 ceph-mon[97633]: pgmap v3579: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:00.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:00.209413+0000) 2022-09-10T06:21:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:00.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:00.616296+0000) 2022-09-10T06:21:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:01.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:01.209600+0000) 2022-09-10T06:21:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:01 smithi104 ceph-mon[97044]: pgmap v3580: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:02.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:01.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:01.616510+0000) 2022-09-10T06:21:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:01 smithi110 ceph-mon[97633]: pgmap v3580: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:02.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:02.209776+0000) 2022-09-10T06:21:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:02.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:02.616765+0000) 2022-09-10T06:21:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:03.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:03.209956+0000) 2022-09-10T06:21:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:03 smithi104 ceph-mon[97044]: pgmap v3581: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:03.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:03.617008+0000) 2022-09-10T06:21:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:03 smithi110 ceph-mon[97633]: pgmap v3581: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:04.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:04.210206+0000) 2022-09-10T06:21:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:04.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:04.617211+0000) 2022-09-10T06:21:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:05.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:05.210431+0000) 2022-09-10T06:21:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:05 smithi104 ceph-mon[97044]: pgmap v3582: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:05.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:05.617439+0000) 2022-09-10T06:21:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:05 smithi110 ceph-mon[97633]: pgmap v3582: 129 pgs: 129 active+clean; 8.0 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:06.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:06.210605+0000) 2022-09-10T06:21:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:06.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:06.617623+0000) 2022-09-10T06:21:07.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:07.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:07.210816+0000) 2022-09-10T06:21:08.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:07 smithi104 ceph-mon[97044]: pgmap v3583: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:21:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:07.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:07.617899+0000) 2022-09-10T06:21:08.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:07 smithi110 ceph-mon[97633]: pgmap v3583: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:21:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:08.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:08.211024+0000) 2022-09-10T06:21:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:08 smithi104 ceph-mon[97044]: osdmap e306: 12 total, 12 up, 12 in 2022-09-10T06:21:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:08.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:08.618133+0000) 2022-09-10T06:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:08 smithi110 ceph-mon[97633]: osdmap e306: 12 total, 12 up, 12 in 2022-09-10T06:21:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:09.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:09.211260+0000) 2022-09-10T06:21:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:09 smithi104 ceph-mon[97044]: pgmap v3585: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:21:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:09.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:09.618352+0000) 2022-09-10T06:21:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:09 smithi110 ceph-mon[97633]: pgmap v3585: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:21:10.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:10.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:10.211527+0000) 2022-09-10T06:21:11.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:10.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:10.618612+0000) 2022-09-10T06:21:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:11.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:11.211754+0000) 2022-09-10T06:21:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:11 smithi104 ceph-mon[97044]: pgmap v3586: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:21:12.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:11.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:11.618870+0000) 2022-09-10T06:21:12.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:11 smithi110 ceph-mon[97633]: pgmap v3586: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:21:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:12.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:12.211949+0000) 2022-09-10T06:21:13.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:12.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:12.619156+0000) 2022-09-10T06:21:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:13.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:13.212195+0000) 2022-09-10T06:21:13.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:21:13.242+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:21:14.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:13 smithi104 ceph-mon[97044]: pgmap v3587: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:21:14.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:13 smithi104 ceph-mon[97044]: osdmap e307: 12 total, 12 up, 12 in 2022-09-10T06:21:14.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:13.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:13.619387+0000) 2022-09-10T06:21:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:13 smithi110 ceph-mon[97633]: pgmap v3587: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:21:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:13 smithi110 ceph-mon[97633]: osdmap e307: 12 total, 12 up, 12 in 2022-09-10T06:21:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:14.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:14.212364+0000) 2022-09-10T06:21:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:14.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:14.619648+0000) 2022-09-10T06:21:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:15.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:15.212616+0000) 2022-09-10T06:21:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:15 smithi104 ceph-mon[97044]: pgmap v3589: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:21:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:15.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:15.619892+0000) 2022-09-10T06:21:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:15 smithi110 ceph-mon[97633]: pgmap v3589: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:21:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:16.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:16.212858+0000) 2022-09-10T06:21:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:16 smithi104 ceph-mon[97044]: pgmap v3590: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:21:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:16.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:16.620177+0000) 2022-09-10T06:21:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:16 smithi110 ceph-mon[97633]: pgmap v3590: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:21:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:17.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:17.213031+0000) 2022-09-10T06:21:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:17.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:17.620323+0000) 2022-09-10T06:21:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:18.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:18.213268+0000) 2022-09-10T06:21:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:18.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:18.620444+0000) 2022-09-10T06:21:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:19.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:19.213468+0000) 2022-09-10T06:21:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:19 smithi104 ceph-mon[97044]: pgmap v3591: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:21:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:19.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:19.620711+0000) 2022-09-10T06:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:19 smithi110 ceph-mon[97633]: pgmap v3591: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:21:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:20.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:20.213702+0000) 2022-09-10T06:21:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:20.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:20.620935+0000) 2022-09-10T06:21:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:21.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:21.213889+0000) 2022-09-10T06:21:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:21 smithi104 ceph-mon[97044]: pgmap v3592: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:21:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:21.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:21.621165+0000) 2022-09-10T06:21:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:21 smithi110 ceph-mon[97633]: pgmap v3592: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:21:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:22.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:22.214120+0000) 2022-09-10T06:21:23.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:22.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:22.621363+0000) 2022-09-10T06:21:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:23.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:23.214368+0000) 2022-09-10T06:21:24.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:23 smithi104 ceph-mon[97044]: pgmap v3593: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:21:24.067 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:23.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:23.621540+0000) 2022-09-10T06:21:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:23 smithi110 ceph-mon[97633]: pgmap v3593: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:21:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:24.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:24.214610+0000) 2022-09-10T06:21:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:24.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:24.621764+0000) 2022-09-10T06:21:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:25.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:25.214929+0000) 2022-09-10T06:21:26.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:25 smithi104 ceph-mon[97044]: pgmap v3594: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:21:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:25.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:25.621927+0000) 2022-09-10T06:21:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:25 smithi110 ceph-mon[97633]: pgmap v3594: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:21:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:26.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:26.215197+0000) 2022-09-10T06:21:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:26.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:26.622189+0000) 2022-09-10T06:21:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:27.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:27.215415+0000) 2022-09-10T06:21:28.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:27 smithi104 ceph-mon[97044]: pgmap v3595: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:21:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:27.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:27.622379+0000) 2022-09-10T06:21:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:27 smithi110 ceph-mon[97633]: pgmap v3595: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:21:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:28.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:28.215653+0000) 2022-09-10T06:21:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:21:28.257+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:21:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:28.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:28.622576+0000) 2022-09-10T06:21:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:29.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:29.215866+0000) 2022-09-10T06:21:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:29 smithi104 ceph-mon[97044]: pgmap v3596: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:29.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:29.622821+0000) 2022-09-10T06:21:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:29 smithi110 ceph-mon[97633]: pgmap v3596: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:30.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:30.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:30.216126+0000) 2022-09-10T06:21:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:30.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:30.623044+0000) 2022-09-10T06:21:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:31.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:31.216338+0000) 2022-09-10T06:21:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:31 smithi104 ceph-mon[97044]: pgmap v3597: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:31.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:31.623335+0000) 2022-09-10T06:21:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:31 smithi110 ceph-mon[97633]: pgmap v3597: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:32.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:32.216495+0000) 2022-09-10T06:21:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:32.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:32.623566+0000) 2022-09-10T06:21:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:33.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:33.216726+0000) 2022-09-10T06:21:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:33 smithi104 ceph-mon[97044]: pgmap v3598: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:33.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:33.623788+0000) 2022-09-10T06:21:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:33 smithi110 ceph-mon[97633]: pgmap v3598: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:34.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:34.216924+0000) 2022-09-10T06:21:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:34.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:34.624059+0000) 2022-09-10T06:21:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:35.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:35.217175+0000) 2022-09-10T06:21:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:35 smithi104 ceph-mon[97044]: pgmap v3599: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:35.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:35.624260+0000) 2022-09-10T06:21:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:35 smithi110 ceph-mon[97633]: pgmap v3599: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:36.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:36.217349+0000) 2022-09-10T06:21:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:36.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:36.624496+0000) 2022-09-10T06:21:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:37.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:37.217549+0000) 2022-09-10T06:21:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:37 smithi104 ceph-mon[97044]: pgmap v3600: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:37.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:37.624723+0000) 2022-09-10T06:21:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:37 smithi110 ceph-mon[97633]: pgmap v3600: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:38.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:38.217803+0000) 2022-09-10T06:21:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:38.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:38.624957+0000) 2022-09-10T06:21:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:39.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:39.218041+0000) 2022-09-10T06:21:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:39 smithi104 ceph-mon[97044]: pgmap v3601: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:39.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:39.625202+0000) 2022-09-10T06:21:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:39 smithi110 ceph-mon[97633]: pgmap v3601: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:40.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:40.218181+0000) 2022-09-10T06:21:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:40.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:40.625444+0000) 2022-09-10T06:21:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:41.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:41.218395+0000) 2022-09-10T06:21:42.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:41 smithi104 ceph-mon[97044]: pgmap v3602: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:41.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:41.625716+0000) 2022-09-10T06:21:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:41 smithi110 ceph-mon[97633]: pgmap v3602: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:42.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:42.218608+0000) 2022-09-10T06:21:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:42.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:42.625905+0000) 2022-09-10T06:21:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:43.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:43.218822+0000) 2022-09-10T06:21:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:21:43.273+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:21:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:43 smithi104 ceph-mon[97044]: pgmap v3603: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:43.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:43.626159+0000) 2022-09-10T06:21:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:43 smithi110 ceph-mon[97633]: pgmap v3603: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:44.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:44.219077+0000) 2022-09-10T06:21:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:21:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:21:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:21:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:21:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:44.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:44.626418+0000) 2022-09-10T06:21:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:21:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:21:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:21:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:21:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:45.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:45.219321+0000) 2022-09-10T06:21:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:45 smithi104 ceph-mon[97044]: pgmap v3604: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:45.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:45.626660+0000) 2022-09-10T06:21:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:45 smithi110 ceph-mon[97633]: pgmap v3604: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:46.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:46.219557+0000) 2022-09-10T06:21:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:46.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:46.626913+0000) 2022-09-10T06:21:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:46 smithi110 ceph-mon[97633]: pgmap v3605: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:46 smithi104 ceph-mon[97044]: pgmap v3605: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:47.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:47.219784+0000) 2022-09-10T06:21:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:47.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:47.627162+0000) 2022-09-10T06:21:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:48.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:48.219972+0000) 2022-09-10T06:21:49.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:48.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:48.627378+0000) 2022-09-10T06:21:49.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:49.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:49.220170+0000) 2022-09-10T06:21:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:49 smithi104 ceph-mon[97044]: pgmap v3606: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:49.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:49.627598+0000) 2022-09-10T06:21:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:49 smithi110 ceph-mon[97633]: pgmap v3606: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:50.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:50.220365+0000) 2022-09-10T06:21:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:50.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:50.627827+0000) 2022-09-10T06:21:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:51.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:51.220625+0000) 2022-09-10T06:21:52.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:51 smithi104 ceph-mon[97044]: pgmap v3607: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:51.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:51.628053+0000) 2022-09-10T06:21:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:51 smithi110 ceph-mon[97633]: pgmap v3607: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:52.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:52.220836+0000) 2022-09-10T06:21:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:52.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:52.628271+0000) 2022-09-10T06:21:53.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:53.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:53.221047+0000) 2022-09-10T06:21:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:53 smithi104 ceph-mon[97044]: pgmap v3608: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:53.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:53.628426+0000) 2022-09-10T06:21:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:53 smithi110 ceph-mon[97633]: pgmap v3608: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:54.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:54.221258+0000) 2022-09-10T06:21:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:54.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:54.628653+0000) 2022-09-10T06:21:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:55.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:55.221461+0000) 2022-09-10T06:21:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:55 smithi104 ceph-mon[97044]: pgmap v3609: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:55.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:55.628874+0000) 2022-09-10T06:21:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:55 smithi110 ceph-mon[97633]: pgmap v3609: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:21:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:56.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:56.221662+0000) 2022-09-10T06:21:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:56.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:56.629140+0000) 2022-09-10T06:21:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:57.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:57.221841+0000) 2022-09-10T06:21:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:57 smithi104 ceph-mon[97044]: pgmap v3610: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:57.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:57.629281+0000) 2022-09-10T06:21:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:57 smithi110 ceph-mon[97633]: pgmap v3610: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:21:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:58.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:58.222073+0000) 2022-09-10T06:21:58.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:21:58.279+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:21:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:58.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:58.629494+0000) 2022-09-10T06:21:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:21:59.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:59.222321+0000) 2022-09-10T06:22:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:21:59 smithi104 ceph-mon[97044]: pgmap v3611: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:21:59.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:21:59.629753+0000) 2022-09-10T06:22:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:21:59 smithi110 ceph-mon[97633]: pgmap v3611: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:00.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:00.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:00.222571+0000) 2022-09-10T06:22:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:00.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:00.629928+0000) 2022-09-10T06:22:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:01.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:01.222817+0000) 2022-09-10T06:22:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:01 smithi104 ceph-mon[97044]: pgmap v3612: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:02.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:01.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:01.630158+0000) 2022-09-10T06:22:02.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:01 smithi110 ceph-mon[97633]: pgmap v3612: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:02.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:02.222999+0000) 2022-09-10T06:22:03.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:02.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:02.630357+0000) 2022-09-10T06:22:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:03.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:03.223230+0000) 2022-09-10T06:22:04.029 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:03.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:03.630546+0000) 2022-09-10T06:22:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:03 smithi104 ceph-mon[97044]: pgmap v3613: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:03 smithi110 ceph-mon[97633]: pgmap v3613: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:04.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:04.223456+0000) 2022-09-10T06:22:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:04.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:04.630790+0000) 2022-09-10T06:22:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:05.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:05.223643+0000) 2022-09-10T06:22:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:05 smithi104 ceph-mon[97044]: pgmap v3614: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:05.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:05.630981+0000) 2022-09-10T06:22:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:05 smithi110 ceph-mon[97633]: pgmap v3614: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:06.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:06.223904+0000) 2022-09-10T06:22:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:06.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:06.631198+0000) 2022-09-10T06:22:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:07.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:07.224150+0000) 2022-09-10T06:22:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:07 smithi104 ceph-mon[97044]: pgmap v3615: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:22:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:07.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:07.631378+0000) 2022-09-10T06:22:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:07 smithi110 ceph-mon[97633]: pgmap v3615: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:22:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:08.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:08.224383+0000) 2022-09-10T06:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:08 smithi104 ceph-mon[97044]: osdmap e308: 12 total, 12 up, 12 in 2022-09-10T06:22:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:08.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:08.631568+0000) 2022-09-10T06:22:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:08 smithi110 ceph-mon[97633]: osdmap e308: 12 total, 12 up, 12 in 2022-09-10T06:22:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:09.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:09.224633+0000) 2022-09-10T06:22:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:09 smithi104 ceph-mon[97044]: pgmap v3617: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:22:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:09.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:09.631818+0000) 2022-09-10T06:22:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:09 smithi110 ceph-mon[97633]: pgmap v3617: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:22:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:10.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:10.224916+0000) 2022-09-10T06:22:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:10 smithi104 ceph-mon[97044]: pgmap v3618: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:22:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:10.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:10.632079+0000) 2022-09-10T06:22:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:10 smithi110 ceph-mon[97633]: pgmap v3618: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:22:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:11.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:11.225170+0000) 2022-09-10T06:22:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:11.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:11.632352+0000) 2022-09-10T06:22:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:12.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:12.225335+0000) 2022-09-10T06:22:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:12.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:12.632618+0000) 2022-09-10T06:22:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:12 smithi104 ceph-mon[97044]: pgmap v3619: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:22:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:12 smithi110 ceph-mon[97633]: pgmap v3619: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:22:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:13.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:13.225554+0000) 2022-09-10T06:22:13.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:22:13.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:22:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:13.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:13.632843+0000) 2022-09-10T06:22:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:14.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:14.225796+0000) 2022-09-10T06:22:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:14 smithi104 ceph-mon[97044]: osdmap e309: 12 total, 12 up, 12 in 2022-09-10T06:22:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:14.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:14.633034+0000) 2022-09-10T06:22:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:14 smithi110 ceph-mon[97633]: osdmap e309: 12 total, 12 up, 12 in 2022-09-10T06:22:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:15.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:15.225972+0000) 2022-09-10T06:22:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:15 smithi104 ceph-mon[97044]: pgmap v3621: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:22:16.066 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:15.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:15.633231+0000) 2022-09-10T06:22:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:15 smithi110 ceph-mon[97633]: pgmap v3621: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:22:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:16.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:16.226149+0000) 2022-09-10T06:22:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:16.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:16.633466+0000) 2022-09-10T06:22:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:17.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:17.226333+0000) 2022-09-10T06:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:17 smithi104 ceph-mon[97044]: pgmap v3622: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:22:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:17.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:17.633644+0000) 2022-09-10T06:22:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:17 smithi110 ceph-mon[97633]: pgmap v3622: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:22:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:18.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:18.226553+0000) 2022-09-10T06:22:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:18.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:18.633884+0000) 2022-09-10T06:22:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:19.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:19.226772+0000) 2022-09-10T06:22:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:19 smithi104 ceph-mon[97044]: pgmap v3623: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:22:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:19.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:19.634039+0000) 2022-09-10T06:22:20.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:19 smithi110 ceph-mon[97633]: pgmap v3623: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:22:20.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:20.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:20.227015+0000) 2022-09-10T06:22:21.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:20.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:20.634268+0000) 2022-09-10T06:22:21.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:21.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:21.227256+0000) 2022-09-10T06:22:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:21 smithi104 ceph-mon[97044]: pgmap v3624: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:22:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:21.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:21.634451+0000) 2022-09-10T06:22:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:21 smithi110 ceph-mon[97633]: pgmap v3624: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:22:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:22.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:22.227415+0000) 2022-09-10T06:22:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:22.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:22.634688+0000) 2022-09-10T06:22:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:23.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:23.227630+0000) 2022-09-10T06:22:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:23 smithi104 ceph-mon[97044]: pgmap v3625: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:22:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:23.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:23.634857+0000) 2022-09-10T06:22:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:23 smithi110 ceph-mon[97633]: pgmap v3625: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:22:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:24.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:24.227772+0000) 2022-09-10T06:22:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:24.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:24.635053+0000) 2022-09-10T06:22:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:25.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:25.228008+0000) 2022-09-10T06:22:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:25 smithi104 ceph-mon[97044]: pgmap v3626: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:22:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:25.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:25.635234+0000) 2022-09-10T06:22:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:25 smithi110 ceph-mon[97633]: pgmap v3626: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:22:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:26.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:26.228186+0000) 2022-09-10T06:22:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:26.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:26.635487+0000) 2022-09-10T06:22:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:27.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:27.228391+0000) 2022-09-10T06:22:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:27 smithi104 ceph-mon[97044]: pgmap v3627: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:22:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:27.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:27.635656+0000) 2022-09-10T06:22:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:27 smithi110 ceph-mon[97633]: pgmap v3627: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:22:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:28.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:28.228628+0000) 2022-09-10T06:22:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:22:28.286+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:22:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:28.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:28.635911+0000) 2022-09-10T06:22:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:29.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:29.228818+0000) 2022-09-10T06:22:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:29 smithi104 ceph-mon[97044]: pgmap v3628: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:29.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:29.636174+0000) 2022-09-10T06:22:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:29 smithi110 ceph-mon[97633]: pgmap v3628: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:30.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:30.229077+0000) 2022-09-10T06:22:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:30.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:30.636406+0000) 2022-09-10T06:22:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:31.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:31.229337+0000) 2022-09-10T06:22:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:31 smithi104 ceph-mon[97044]: pgmap v3629: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:32.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:31.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:31.636562+0000) 2022-09-10T06:22:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:31 smithi110 ceph-mon[97633]: pgmap v3629: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:32.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:32.229528+0000) 2022-09-10T06:22:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:32.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:32.636819+0000) 2022-09-10T06:22:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:33.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:33.229714+0000) 2022-09-10T06:22:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:33 smithi104 ceph-mon[97044]: pgmap v3630: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:33.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:33.637067+0000) 2022-09-10T06:22:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:33 smithi110 ceph-mon[97633]: pgmap v3630: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:34.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:34.229983+0000) 2022-09-10T06:22:35.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:34.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:34.637297+0000) 2022-09-10T06:22:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:35.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:35.230214+0000) 2022-09-10T06:22:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:35 smithi104 ceph-mon[97044]: pgmap v3631: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:35.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:35.637524+0000) 2022-09-10T06:22:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:35 smithi110 ceph-mon[97633]: pgmap v3631: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:36.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:36.230357+0000) 2022-09-10T06:22:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:36.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:36.637755+0000) 2022-09-10T06:22:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:37.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:37.230579+0000) 2022-09-10T06:22:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:37 smithi104 ceph-mon[97044]: pgmap v3632: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:37.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:37.637996+0000) 2022-09-10T06:22:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:37 smithi110 ceph-mon[97633]: pgmap v3632: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:38.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:38.230802+0000) 2022-09-10T06:22:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:38.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:38.638207+0000) 2022-09-10T06:22:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:39.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:39.231087+0000) 2022-09-10T06:22:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:39 smithi104 ceph-mon[97044]: pgmap v3633: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:39.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:39.638419+0000) 2022-09-10T06:22:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:39 smithi110 ceph-mon[97633]: pgmap v3633: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:40.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:40.231366+0000) 2022-09-10T06:22:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:40.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:40.638587+0000) 2022-09-10T06:22:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:41.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:41.231574+0000) 2022-09-10T06:22:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:41 smithi104 ceph-mon[97044]: pgmap v3634: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:41.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:41.638730+0000) 2022-09-10T06:22:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:41 smithi110 ceph-mon[97633]: pgmap v3634: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:42.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:42.231821+0000) 2022-09-10T06:22:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:42.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:42.638924+0000) 2022-09-10T06:22:43.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:22:43.296+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:22:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:43.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:43.232048+0000) 2022-09-10T06:22:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:43 smithi104 ceph-mon[97044]: pgmap v3635: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:43.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:43.639162+0000) 2022-09-10T06:22:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:43 smithi110 ceph-mon[97633]: pgmap v3635: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:44.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:44.232289+0000) 2022-09-10T06:22:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:22:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:22:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:22:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:22:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:44.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:44.639399+0000) 2022-09-10T06:22:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:22:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:22:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:22:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:22:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:45.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:45.232525+0000) 2022-09-10T06:22:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:45 smithi104 ceph-mon[97044]: pgmap v3636: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:45.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:45.639610+0000) 2022-09-10T06:22:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:45 smithi110 ceph-mon[97633]: pgmap v3636: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:46.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:46.232714+0000) 2022-09-10T06:22:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:46.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:46.639834+0000) 2022-09-10T06:22:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:47.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:47.232899+0000) 2022-09-10T06:22:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:47 smithi104 ceph-mon[97044]: pgmap v3637: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:47.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:47.639993+0000) 2022-09-10T06:22:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:47 smithi110 ceph-mon[97633]: pgmap v3637: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:48.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:48.233166+0000) 2022-09-10T06:22:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:48.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:48.640192+0000) 2022-09-10T06:22:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:49.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:49.233428+0000) 2022-09-10T06:22:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:49 smithi104 ceph-mon[97044]: pgmap v3638: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:49.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:49.640441+0000) 2022-09-10T06:22:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:49 smithi110 ceph-mon[97633]: pgmap v3638: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:50.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:50.233699+0000) 2022-09-10T06:22:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:50.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:50.640683+0000) 2022-09-10T06:22:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:51.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:51.233913+0000) 2022-09-10T06:22:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:51 smithi104 ceph-mon[97044]: pgmap v3639: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:51.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:51.640944+0000) 2022-09-10T06:22:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:51 smithi110 ceph-mon[97633]: pgmap v3639: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:52.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:52.234151+0000) 2022-09-10T06:22:53.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:52.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:52.641153+0000) 2022-09-10T06:22:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:53.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:53.234424+0000) 2022-09-10T06:22:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:53 smithi104 ceph-mon[97044]: pgmap v3640: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:54.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:53.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:53.641363+0000) 2022-09-10T06:22:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:53 smithi110 ceph-mon[97633]: pgmap v3640: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:54.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:54.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:54.234682+0000) 2022-09-10T06:22:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:54.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:54.641529+0000) 2022-09-10T06:22:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:55.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:55.234933+0000) 2022-09-10T06:22:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:55 smithi104 ceph-mon[97044]: pgmap v3641: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:55.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:55.641784+0000) 2022-09-10T06:22:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:55 smithi110 ceph-mon[97633]: pgmap v3641: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:22:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:56.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:56.235207+0000) 2022-09-10T06:22:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:56.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:56.642035+0000) 2022-09-10T06:22:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:57.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:57.235475+0000) 2022-09-10T06:22:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:57 smithi104 ceph-mon[97044]: pgmap v3642: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:57.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:57.642214+0000) 2022-09-10T06:22:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:57 smithi110 ceph-mon[97633]: pgmap v3642: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:22:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:58.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:58.235711+0000) 2022-09-10T06:22:58.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:22:58.302+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:22:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:58.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:58.642361+0000) 2022-09-10T06:22:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:22:59.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:59.235947+0000) 2022-09-10T06:23:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:22:59 smithi104 ceph-mon[97044]: pgmap v3643: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:22:59.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:22:59.642576+0000) 2022-09-10T06:23:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:22:59 smithi110 ceph-mon[97633]: pgmap v3643: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:00.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:00.236146+0000) 2022-09-10T06:23:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:00.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:00.642821+0000) 2022-09-10T06:23:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:01.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:01.236357+0000) 2022-09-10T06:23:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:01 smithi104 ceph-mon[97044]: pgmap v3644: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:01.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:01.643069+0000) 2022-09-10T06:23:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:01 smithi110 ceph-mon[97633]: pgmap v3644: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:02.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:02.236550+0000) 2022-09-10T06:23:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:02.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:02.643313+0000) 2022-09-10T06:23:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:03.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:03.236739+0000) 2022-09-10T06:23:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:03 smithi104 ceph-mon[97044]: pgmap v3645: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:03.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:03.643550+0000) 2022-09-10T06:23:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:03 smithi110 ceph-mon[97633]: pgmap v3645: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:04.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:04.236992+0000) 2022-09-10T06:23:04.950 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:04.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:04.643799+0000) 2022-09-10T06:23:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:05.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:05.237269+0000) 2022-09-10T06:23:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:05 smithi104 ceph-mon[97044]: pgmap v3646: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:05.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:05.643987+0000) 2022-09-10T06:23:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:05 smithi110 ceph-mon[97633]: pgmap v3646: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:06.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:06.237404+0000) 2022-09-10T06:23:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:06.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:06.644200+0000) 2022-09-10T06:23:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:07.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:07.237645+0000) 2022-09-10T06:23:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:07 smithi104 ceph-mon[97044]: pgmap v3647: 129 pgs: 129 active+clean; 8.0 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:23:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:07.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:07.644383+0000) 2022-09-10T06:23:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:07 smithi110 ceph-mon[97633]: pgmap v3647: 129 pgs: 129 active+clean; 8.0 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:23:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:08.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:08.237900+0000) 2022-09-10T06:23:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:08 smithi104 ceph-mon[97044]: osdmap e310: 12 total, 12 up, 12 in 2022-09-10T06:23:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:08.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:08.644583+0000) 2022-09-10T06:23:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:08 smithi110 ceph-mon[97633]: osdmap e310: 12 total, 12 up, 12 in 2022-09-10T06:23:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:09.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:09.238127+0000) 2022-09-10T06:23:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:09 smithi104 ceph-mon[97044]: pgmap v3649: 129 pgs: 129 active+clean; 8.0 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:23:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:09.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:09.644797+0000) 2022-09-10T06:23:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:09 smithi110 ceph-mon[97633]: pgmap v3649: 129 pgs: 129 active+clean; 8.0 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:23:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:10.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:10.238428+0000) 2022-09-10T06:23:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:10.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:10.645049+0000) 2022-09-10T06:23:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:11.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:11.238564+0000) 2022-09-10T06:23:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:11 smithi104 ceph-mon[97044]: pgmap v3650: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:23:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:11.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:11.645356+0000) 2022-09-10T06:23:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:11 smithi110 ceph-mon[97633]: pgmap v3650: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:23:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:12.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:12.238803+0000) 2022-09-10T06:23:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:12.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:12.645586+0000) 2022-09-10T06:23:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:12 smithi110 ceph-mon[97633]: pgmap v3651: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:23:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:12 smithi104 ceph-mon[97044]: pgmap v3651: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:23:13.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:23:13.311+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:23:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:13.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:13.239027+0000) 2022-09-10T06:23:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:13.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:13.645798+0000) 2022-09-10T06:23:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:14.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:14.239197+0000) 2022-09-10T06:23:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:14 smithi104 ceph-mon[97044]: osdmap e311: 12 total, 12 up, 12 in 2022-09-10T06:23:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:14.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:14.645947+0000) 2022-09-10T06:23:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:14 smithi110 ceph-mon[97633]: osdmap e311: 12 total, 12 up, 12 in 2022-09-10T06:23:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:15.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:15.239411+0000) 2022-09-10T06:23:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:15 smithi104 ceph-mon[97044]: pgmap v3653: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:23:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:15.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:15.646124+0000) 2022-09-10T06:23:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:15 smithi110 ceph-mon[97633]: pgmap v3653: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:23:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:16.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:16.239659+0000) 2022-09-10T06:23:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:16.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:16.646275+0000) 2022-09-10T06:23:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:17.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:17.239858+0000) 2022-09-10T06:23:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:17 smithi104 ceph-mon[97044]: pgmap v3654: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T06:23:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:17.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:17.646483+0000) 2022-09-10T06:23:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:17 smithi110 ceph-mon[97633]: pgmap v3654: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T06:23:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:18.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:18.240032+0000) 2022-09-10T06:23:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:18.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:18.646706+0000) 2022-09-10T06:23:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:19.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:19.240269+0000) 2022-09-10T06:23:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:19 smithi104 ceph-mon[97044]: pgmap v3655: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:23:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:19.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:19.646946+0000) 2022-09-10T06:23:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:19 smithi110 ceph-mon[97633]: pgmap v3655: 129 pgs: 129 active+clean; 8.0 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:23:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:20.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:20.240532+0000) 2022-09-10T06:23:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:20.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:20.647191+0000) 2022-09-10T06:23:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:21.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:21.240809+0000) 2022-09-10T06:23:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:21 smithi104 ceph-mon[97044]: pgmap v3656: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:23:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:21.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:21.647445+0000) 2022-09-10T06:23:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:21 smithi110 ceph-mon[97633]: pgmap v3656: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:23:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:22.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:22.240969+0000) 2022-09-10T06:23:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:22.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:22.647680+0000) 2022-09-10T06:23:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:23.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:23.241182+0000) 2022-09-10T06:23:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:23 smithi104 ceph-mon[97044]: pgmap v3657: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:23:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:23.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:23.647912+0000) 2022-09-10T06:23:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:23 smithi110 ceph-mon[97633]: pgmap v3657: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:23:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:24.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:24.241344+0000) 2022-09-10T06:23:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:24.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:24.648144+0000) 2022-09-10T06:23:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:25.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:25.241580+0000) 2022-09-10T06:23:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:25 smithi104 ceph-mon[97044]: pgmap v3658: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:23:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:25.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:25.648322+0000) 2022-09-10T06:23:26.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:25 smithi110 ceph-mon[97633]: pgmap v3658: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:23:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:26.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:26.241768+0000) 2022-09-10T06:23:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:26.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:26.648548+0000) 2022-09-10T06:23:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:27.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:27.242042+0000) 2022-09-10T06:23:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:27 smithi104 ceph-mon[97044]: pgmap v3659: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:23:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:27.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:27.648784+0000) 2022-09-10T06:23:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:27 smithi110 ceph-mon[97633]: pgmap v3659: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:23:28.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:23:28.317+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:23:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:28.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:28.242244+0000) 2022-09-10T06:23:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:28.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:28.649010+0000) 2022-09-10T06:23:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:29.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:29.242436+0000) 2022-09-10T06:23:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:29 smithi104 ceph-mon[97044]: pgmap v3660: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:29.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:29.649204+0000) 2022-09-10T06:23:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:29 smithi110 ceph-mon[97633]: pgmap v3660: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:30.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:30.242580+0000) 2022-09-10T06:23:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:30.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:30.649387+0000) 2022-09-10T06:23:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:31.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:31.242809+0000) 2022-09-10T06:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:31 smithi104 ceph-mon[97044]: pgmap v3661: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:31.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:31.649568+0000) 2022-09-10T06:23:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:31 smithi110 ceph-mon[97633]: pgmap v3661: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:32.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:32.243059+0000) 2022-09-10T06:23:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:32.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:32.649800+0000) 2022-09-10T06:23:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:33.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:33.243243+0000) 2022-09-10T06:23:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:33 smithi104 ceph-mon[97044]: pgmap v3662: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:33.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:33.650024+0000) 2022-09-10T06:23:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:33 smithi110 ceph-mon[97633]: pgmap v3662: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:34.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:34.243427+0000) 2022-09-10T06:23:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:34.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:34.650241+0000) 2022-09-10T06:23:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:35.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:35.243700+0000) 2022-09-10T06:23:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:35 smithi104 ceph-mon[97044]: pgmap v3663: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:35.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:35.650458+0000) 2022-09-10T06:23:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:35 smithi110 ceph-mon[97633]: pgmap v3663: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:36.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:36.243950+0000) 2022-09-10T06:23:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:36.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:36.650677+0000) 2022-09-10T06:23:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:37.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:37.244223+0000) 2022-09-10T06:23:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:37 smithi104 ceph-mon[97044]: pgmap v3664: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:37.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:37.650915+0000) 2022-09-10T06:23:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:37 smithi110 ceph-mon[97633]: pgmap v3664: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:38.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:38.244426+0000) 2022-09-10T06:23:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:38.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:38.651181+0000) 2022-09-10T06:23:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:39.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:39.244605+0000) 2022-09-10T06:23:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:39 smithi104 ceph-mon[97044]: pgmap v3665: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:39.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:39.651381+0000) 2022-09-10T06:23:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:39 smithi110 ceph-mon[97633]: pgmap v3665: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:40.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:40.244837+0000) 2022-09-10T06:23:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:40.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:40.651595+0000) 2022-09-10T06:23:41.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:41.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:41.245146+0000) 2022-09-10T06:23:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:41 smithi104 ceph-mon[97044]: pgmap v3666: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:41.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:41.651783+0000) 2022-09-10T06:23:42.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:41 smithi110 ceph-mon[97633]: pgmap v3666: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:42.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:42.245413+0000) 2022-09-10T06:23:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:42.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:42.652021+0000) 2022-09-10T06:23:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:43.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:43.245689+0000) 2022-09-10T06:23:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:23:43.333+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:23:44.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:43 smithi104 ceph-mon[97044]: pgmap v3667: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:43.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:43.652224+0000) 2022-09-10T06:23:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:43 smithi110 ceph-mon[97633]: pgmap v3667: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:44.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:44.245888+0000) 2022-09-10T06:23:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:23:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:23:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:23:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:23:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:44.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:44.652443+0000) 2022-09-10T06:23:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:23:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:23:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:23:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:23:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:45.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:45.246146+0000) 2022-09-10T06:23:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:45 smithi104 ceph-mon[97044]: pgmap v3668: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:45.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:45.652641+0000) 2022-09-10T06:23:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:45 smithi110 ceph-mon[97633]: pgmap v3668: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:46.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:46.246313+0000) 2022-09-10T06:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:46 smithi104 ceph-mon[97044]: pgmap v3669: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:46.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:46.652880+0000) 2022-09-10T06:23:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:46 smithi110 ceph-mon[97633]: pgmap v3669: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:47.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:47.246526+0000) 2022-09-10T06:23:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:47.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:47.653064+0000) 2022-09-10T06:23:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:48.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:48.246748+0000) 2022-09-10T06:23:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:48.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:48.653300+0000) 2022-09-10T06:23:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:49.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:49.247028+0000) 2022-09-10T06:23:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:49 smithi104 ceph-mon[97044]: pgmap v3670: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:49.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:49.653477+0000) 2022-09-10T06:23:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:49 smithi110 ceph-mon[97633]: pgmap v3670: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:50.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:50.247331+0000) 2022-09-10T06:23:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:50.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:50.653691+0000) 2022-09-10T06:23:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:51.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:51.247565+0000) 2022-09-10T06:23:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:51 smithi104 ceph-mon[97044]: pgmap v3671: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:51.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:51.653904+0000) 2022-09-10T06:23:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:51 smithi110 ceph-mon[97633]: pgmap v3671: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:52.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:52.247809+0000) 2022-09-10T06:23:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:52.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:52.654141+0000) 2022-09-10T06:23:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:53.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:53.248025+0000) 2022-09-10T06:23:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:53 smithi104 ceph-mon[97044]: pgmap v3672: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:54.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:53.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:53.654349+0000) 2022-09-10T06:23:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:53 smithi110 ceph-mon[97633]: pgmap v3672: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:54.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:54.248236+0000) 2022-09-10T06:23:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:54.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:54.654576+0000) 2022-09-10T06:23:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:55.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:55.248483+0000) 2022-09-10T06:23:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:55 smithi104 ceph-mon[97044]: pgmap v3673: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:55.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:55.654836+0000) 2022-09-10T06:23:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:55 smithi110 ceph-mon[97633]: pgmap v3673: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:23:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:56.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:56.248698+0000) 2022-09-10T06:23:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:56.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:56.655008+0000) 2022-09-10T06:23:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:57.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:57.248927+0000) 2022-09-10T06:23:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:57 smithi104 ceph-mon[97044]: pgmap v3674: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:57.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:57.655172+0000) 2022-09-10T06:23:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:57 smithi110 ceph-mon[97633]: pgmap v3674: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:23:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:58.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:58.249178+0000) 2022-09-10T06:23:58.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:23:58.349+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:23:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:58.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:58.655381+0000) 2022-09-10T06:23:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:23:59.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:59.249421+0000) 2022-09-10T06:24:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:23:59 smithi104 ceph-mon[97044]: pgmap v3675: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:23:59.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:23:59.655633+0000) 2022-09-10T06:24:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:23:59 smithi110 ceph-mon[97633]: pgmap v3675: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:00.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:00.249667+0000) 2022-09-10T06:24:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:00.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:00.655882+0000) 2022-09-10T06:24:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:01.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:01.249903+0000) 2022-09-10T06:24:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:01 smithi104 ceph-mon[97044]: pgmap v3676: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:01.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:01.656142+0000) 2022-09-10T06:24:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:01 smithi110 ceph-mon[97633]: pgmap v3676: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:02.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:02.250148+0000) 2022-09-10T06:24:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:02.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:02.656378+0000) 2022-09-10T06:24:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:03.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:03.250276+0000) 2022-09-10T06:24:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:03 smithi104 ceph-mon[97044]: pgmap v3677: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:03.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:03.656542+0000) 2022-09-10T06:24:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:03 smithi110 ceph-mon[97633]: pgmap v3677: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:04.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:04.250480+0000) 2022-09-10T06:24:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:04.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:04.656727+0000) 2022-09-10T06:24:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:05.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:05.250689+0000) 2022-09-10T06:24:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:05 smithi104 ceph-mon[97044]: pgmap v3678: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:05.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:05.656915+0000) 2022-09-10T06:24:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:05 smithi110 ceph-mon[97633]: pgmap v3678: 129 pgs: 129 active+clean; 8.0 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:06.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:06.250946+0000) 2022-09-10T06:24:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:06.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:06.657195+0000) 2022-09-10T06:24:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:07.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:07.251209+0000) 2022-09-10T06:24:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:07 smithi104 ceph-mon[97044]: pgmap v3679: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:24:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:07.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:07.657401+0000) 2022-09-10T06:24:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:07 smithi110 ceph-mon[97633]: pgmap v3679: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:24:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:08.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:08.251437+0000) 2022-09-10T06:24:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:08 smithi104 ceph-mon[97044]: osdmap e312: 12 total, 12 up, 12 in 2022-09-10T06:24:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:08 smithi104 ceph-mon[97044]: pgmap v3681: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:24:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:08.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:08.657602+0000) 2022-09-10T06:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:08 smithi110 ceph-mon[97633]: osdmap e312: 12 total, 12 up, 12 in 2022-09-10T06:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:08 smithi110 ceph-mon[97633]: pgmap v3681: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:24:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:09.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:09.251629+0000) 2022-09-10T06:24:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:09.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:09.657813+0000) 2022-09-10T06:24:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:10.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:10.251905+0000) 2022-09-10T06:24:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:10.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:10.658123+0000) 2022-09-10T06:24:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:10 smithi104 ceph-mon[97044]: pgmap v3682: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:24:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:10 smithi110 ceph-mon[97633]: pgmap v3682: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:24:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:11.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:11.252171+0000) 2022-09-10T06:24:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:11.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:11.658289+0000) 2022-09-10T06:24:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:12.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:12.252355+0000) 2022-09-10T06:24:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:12.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:12.658473+0000) 2022-09-10T06:24:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:13 smithi104 ceph-mon[97044]: pgmap v3683: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:24:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:13 smithi110 ceph-mon[97633]: pgmap v3683: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:24:13.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:24:13.351+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:24:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:13.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:13.252602+0000) 2022-09-10T06:24:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:13.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:13.658686+0000) 2022-09-10T06:24:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:14.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:14.252884+0000) 2022-09-10T06:24:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:14 smithi104 ceph-mon[97044]: osdmap e313: 12 total, 12 up, 12 in 2022-09-10T06:24:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:14.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:14.658861+0000) 2022-09-10T06:24:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:14 smithi110 ceph-mon[97633]: osdmap e313: 12 total, 12 up, 12 in 2022-09-10T06:24:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:15.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:15.253167+0000) 2022-09-10T06:24:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:15 smithi104 ceph-mon[97044]: pgmap v3685: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:24:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:15.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:15.659007+0000) 2022-09-10T06:24:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:15 smithi110 ceph-mon[97633]: pgmap v3685: 129 pgs: 129 active+clean; 8.0 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:24:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:16.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:16.253360+0000) 2022-09-10T06:24:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:16.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:16.659246+0000) 2022-09-10T06:24:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:17.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:17.253545+0000) 2022-09-10T06:24:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:17 smithi104 ceph-mon[97044]: pgmap v3686: 129 pgs: 129 active+clean; 8.0 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:24:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:17.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:17.659426+0000) 2022-09-10T06:24:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:17 smithi110 ceph-mon[97633]: pgmap v3686: 129 pgs: 129 active+clean; 8.0 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T06:24:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:18.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:18.253757+0000) 2022-09-10T06:24:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:18.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:18.659650+0000) 2022-09-10T06:24:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:19.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:19.253969+0000) 2022-09-10T06:24:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:19 smithi104 ceph-mon[97044]: pgmap v3687: 129 pgs: 129 active+clean; 8.0 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:24:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:19.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:19.659889+0000) 2022-09-10T06:24:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:19 smithi110 ceph-mon[97633]: pgmap v3687: 129 pgs: 129 active+clean; 8.0 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:24:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:20.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:20.254221+0000) 2022-09-10T06:24:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:20.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:20.660081+0000) 2022-09-10T06:24:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:21.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:21.254548+0000) 2022-09-10T06:24:22.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:21 smithi104 ceph-mon[97044]: pgmap v3688: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:24:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:21.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:21.660262+0000) 2022-09-10T06:24:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:21 smithi110 ceph-mon[97633]: pgmap v3688: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:24:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:22.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:22.254769+0000) 2022-09-10T06:24:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:22.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:22.660496+0000) 2022-09-10T06:24:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:23.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:23.255030+0000) 2022-09-10T06:24:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:23 smithi104 ceph-mon[97044]: pgmap v3689: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:24:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:23.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:23.660715+0000) 2022-09-10T06:24:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:23 smithi110 ceph-mon[97633]: pgmap v3689: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:24:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:24.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:24.255265+0000) 2022-09-10T06:24:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:24.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:24.660987+0000) 2022-09-10T06:24:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:25.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:25.255498+0000) 2022-09-10T06:24:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:25 smithi104 ceph-mon[97044]: pgmap v3690: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:24:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:25.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:25.661265+0000) 2022-09-10T06:24:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:25 smithi110 ceph-mon[97633]: pgmap v3690: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:24:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:26.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:26.255736+0000) 2022-09-10T06:24:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:26 smithi104 ceph-mon[97044]: pgmap v3691: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:24:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:26.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:26.661484+0000) 2022-09-10T06:24:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:26 smithi110 ceph-mon[97633]: pgmap v3691: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:24:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:27.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:27.255984+0000) 2022-09-10T06:24:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:27.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:27.661722+0000) 2022-09-10T06:24:28.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:24:28.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:24:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:28.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:28.256209+0000) 2022-09-10T06:24:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:28.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:28.661963+0000) 2022-09-10T06:24:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:29.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:29.256509+0000) 2022-09-10T06:24:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:29 smithi104 ceph-mon[97044]: pgmap v3692: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:29.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:29.662244+0000) 2022-09-10T06:24:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:29 smithi110 ceph-mon[97633]: pgmap v3692: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:30.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:30.256693+0000) 2022-09-10T06:24:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:30.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:30.662469+0000) 2022-09-10T06:24:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:31.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:31.256849+0000) 2022-09-10T06:24:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:31 smithi104 ceph-mon[97044]: pgmap v3693: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:31.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:31.662638+0000) 2022-09-10T06:24:32.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:31 smithi110 ceph-mon[97633]: pgmap v3693: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:32.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:32.257052+0000) 2022-09-10T06:24:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:32.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:32.662830+0000) 2022-09-10T06:24:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:33.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:33.257358+0000) 2022-09-10T06:24:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:33 smithi104 ceph-mon[97044]: pgmap v3694: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:33.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:33.662980+0000) 2022-09-10T06:24:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:33 smithi110 ceph-mon[97633]: pgmap v3694: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:34.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:34.257555+0000) 2022-09-10T06:24:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:34.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:34.663170+0000) 2022-09-10T06:24:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:35.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:35.257797+0000) 2022-09-10T06:24:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:35 smithi104 ceph-mon[97044]: pgmap v3695: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:35.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:35.663319+0000) 2022-09-10T06:24:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:35 smithi110 ceph-mon[97633]: pgmap v3695: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:36.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:36.258000+0000) 2022-09-10T06:24:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:36.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:36.663504+0000) 2022-09-10T06:24:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:37.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:37.258187+0000) 2022-09-10T06:24:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:37 smithi104 ceph-mon[97044]: pgmap v3696: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:37.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:37.663759+0000) 2022-09-10T06:24:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:37 smithi110 ceph-mon[97633]: pgmap v3696: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:38.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:38.258407+0000) 2022-09-10T06:24:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:38.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:38.663983+0000) 2022-09-10T06:24:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:39.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:39.258651+0000) 2022-09-10T06:24:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:39 smithi104 ceph-mon[97044]: pgmap v3697: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:39.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:39.664172+0000) 2022-09-10T06:24:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:39 smithi110 ceph-mon[97633]: pgmap v3697: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:40.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:40.258911+0000) 2022-09-10T06:24:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:40.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:40.664427+0000) 2022-09-10T06:24:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:41.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:41.259165+0000) 2022-09-10T06:24:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:41 smithi104 ceph-mon[97044]: pgmap v3698: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:41.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:41.664596+0000) 2022-09-10T06:24:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:41 smithi110 ceph-mon[97633]: pgmap v3698: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:42.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:42.259344+0000) 2022-09-10T06:24:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:42.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:42.664826+0000) 2022-09-10T06:24:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:43.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:43.259556+0000) 2022-09-10T06:24:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:24:43.380+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:24:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:43 smithi104 ceph-mon[97044]: pgmap v3699: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:43.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:43.665002+0000) 2022-09-10T06:24:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:43 smithi110 ceph-mon[97633]: pgmap v3699: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:44.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:44.259812+0000) 2022-09-10T06:24:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:24:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:24:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:24:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:24:45.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:44 smithi104 ceph-mon[97044]: pgmap v3700: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:44.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:44.665221+0000) 2022-09-10T06:24:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:24:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:24:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:24:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:24:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:44 smithi110 ceph-mon[97633]: pgmap v3700: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:45.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:45.260127+0000) 2022-09-10T06:24:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:45.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:45.665373+0000) 2022-09-10T06:24:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:46.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:46.260287+0000) 2022-09-10T06:24:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:46.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:46.665576+0000) 2022-09-10T06:24:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:47 smithi104 ceph-mon[97044]: pgmap v3701: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:47 smithi110 ceph-mon[97633]: pgmap v3701: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:47.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:47.260484+0000) 2022-09-10T06:24:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:47.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:47.665800+0000) 2022-09-10T06:24:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:48.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:48.260642+0000) 2022-09-10T06:24:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:48.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:48.665993+0000) 2022-09-10T06:24:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:49.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:49.260768+0000) 2022-09-10T06:24:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:49 smithi104 ceph-mon[97044]: pgmap v3702: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:49.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:49.666206+0000) 2022-09-10T06:24:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:49 smithi110 ceph-mon[97633]: pgmap v3702: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:50.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:50.260907+0000) 2022-09-10T06:24:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:50.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:50.666417+0000) 2022-09-10T06:24:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:51.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:51.261122+0000) 2022-09-10T06:24:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:51 smithi104 ceph-mon[97044]: pgmap v3703: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:51.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:51.666598+0000) 2022-09-10T06:24:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:51 smithi110 ceph-mon[97633]: pgmap v3703: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:52.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:52.261336+0000) 2022-09-10T06:24:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:52.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:52.666790+0000) 2022-09-10T06:24:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:53.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:53.261571+0000) 2022-09-10T06:24:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:53 smithi104 ceph-mon[97044]: pgmap v3704: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:53.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:53.666952+0000) 2022-09-10T06:24:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:53 smithi110 ceph-mon[97633]: pgmap v3704: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:54.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:54.261814+0000) 2022-09-10T06:24:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:54.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:54.667106+0000) 2022-09-10T06:24:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:55.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:55.262034+0000) 2022-09-10T06:24:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:55 smithi104 ceph-mon[97044]: pgmap v3705: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:55.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:55.667311+0000) 2022-09-10T06:24:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:55 smithi110 ceph-mon[97633]: pgmap v3705: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:24:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:56.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:56.262232+0000) 2022-09-10T06:24:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:56.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:56.667584+0000) 2022-09-10T06:24:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:57.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:57.262402+0000) 2022-09-10T06:24:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:57 smithi104 ceph-mon[97044]: pgmap v3706: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:57.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:57.667890+0000) 2022-09-10T06:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:57 smithi110 ceph-mon[97633]: pgmap v3706: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:24:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:58.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:58.262611+0000) 2022-09-10T06:24:58.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:24:58.395+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:24:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:58.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:58.668158+0000) 2022-09-10T06:24:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:24:59.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:59.262841+0000) 2022-09-10T06:25:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:24:59 smithi104 ceph-mon[97044]: pgmap v3707: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:24:59.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:24:59.668429+0000) 2022-09-10T06:25:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:24:59 smithi110 ceph-mon[97633]: pgmap v3707: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:00.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:00.263062+0000) 2022-09-10T06:25:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:00.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:00.668683+0000) 2022-09-10T06:25:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:01.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:01.263280+0000) 2022-09-10T06:25:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:01 smithi104 ceph-mon[97044]: pgmap v3708: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:01.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:01.668907+0000) 2022-09-10T06:25:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:01 smithi110 ceph-mon[97633]: pgmap v3708: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:02.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:02.263443+0000) 2022-09-10T06:25:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:02.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:02.669104+0000) 2022-09-10T06:25:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:02 smithi110 ceph-mon[97633]: pgmap v3709: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:03.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:02 smithi104 ceph-mon[97044]: pgmap v3709: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:03.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:03.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:03.263701+0000) 2022-09-10T06:25:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:03.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:03.669324+0000) 2022-09-10T06:25:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:04.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:04.263932+0000) 2022-09-10T06:25:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:04.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:04.669604+0000) 2022-09-10T06:25:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:05.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:05.264204+0000) 2022-09-10T06:25:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:05 smithi104 ceph-mon[97044]: pgmap v3710: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:05.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:05.669853+0000) 2022-09-10T06:25:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:05 smithi110 ceph-mon[97633]: pgmap v3710: 129 pgs: 129 active+clean; 8.0 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:06.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:06.264414+0000) 2022-09-10T06:25:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:06.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:06.670122+0000) 2022-09-10T06:25:07.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:07.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:07.264597+0000) 2022-09-10T06:25:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:07 smithi104 ceph-mon[97044]: pgmap v3711: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:25:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:07.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:07.670322+0000) 2022-09-10T06:25:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:07 smithi110 ceph-mon[97633]: pgmap v3711: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:25:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:08.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:08.264817+0000) 2022-09-10T06:25:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:08 smithi104 ceph-mon[97044]: osdmap e314: 12 total, 12 up, 12 in 2022-09-10T06:25:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:08.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:08.670576+0000) 2022-09-10T06:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:08 smithi110 ceph-mon[97633]: osdmap e314: 12 total, 12 up, 12 in 2022-09-10T06:25:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:09.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:09.264984+0000) 2022-09-10T06:25:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:09 smithi104 ceph-mon[97044]: pgmap v3713: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:25:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:09.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:09.670769+0000) 2022-09-10T06:25:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:09 smithi110 ceph-mon[97633]: pgmap v3713: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:25:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:10.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:10.265227+0000) 2022-09-10T06:25:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:10.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:10.670994+0000) 2022-09-10T06:25:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:10 smithi110 ceph-mon[97633]: pgmap v3714: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:25:11.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:10 smithi104 ceph-mon[97044]: pgmap v3714: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:25:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:11.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:11.265475+0000) 2022-09-10T06:25:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:11.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:11.671202+0000) 2022-09-10T06:25:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:12.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:12.265646+0000) 2022-09-10T06:25:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:12.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:12.671393+0000) 2022-09-10T06:25:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:13 smithi104 ceph-mon[97044]: pgmap v3715: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:25:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:13 smithi110 ceph-mon[97633]: pgmap v3715: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:25:13.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:25:13.405+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:25:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:13.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:13.265789+0000) 2022-09-10T06:25:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:13.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:13.671623+0000) 2022-09-10T06:25:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:14.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:14.265991+0000) 2022-09-10T06:25:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:14 smithi104 ceph-mon[97044]: osdmap e315: 12 total, 12 up, 12 in 2022-09-10T06:25:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:14.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:14.671793+0000) 2022-09-10T06:25:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:14 smithi110 ceph-mon[97633]: osdmap e315: 12 total, 12 up, 12 in 2022-09-10T06:25:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:15.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:15.266214+0000) 2022-09-10T06:25:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:15 smithi104 ceph-mon[97044]: pgmap v3717: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:25:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:15.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:15.671963+0000) 2022-09-10T06:25:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:15 smithi110 ceph-mon[97633]: pgmap v3717: 129 pgs: 129 active+clean; 8.0 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:25:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:16.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:16.266456+0000) 2022-09-10T06:25:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:16.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:16.672183+0000) 2022-09-10T06:25:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:17.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:17.266655+0000) 2022-09-10T06:25:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:17 smithi104 ceph-mon[97044]: pgmap v3718: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T06:25:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:17.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:17.672416+0000) 2022-09-10T06:25:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:17 smithi110 ceph-mon[97633]: pgmap v3718: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T06:25:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:18.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:18.266907+0000) 2022-09-10T06:25:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:18.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:18.672652+0000) 2022-09-10T06:25:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:19.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:19.267173+0000) 2022-09-10T06:25:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:19 smithi104 ceph-mon[97044]: pgmap v3719: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:25:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:19.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:19.672901+0000) 2022-09-10T06:25:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:19 smithi110 ceph-mon[97633]: pgmap v3719: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:25:20.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:20.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:20.267392+0000) 2022-09-10T06:25:21.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:20.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:20.673126+0000) 2022-09-10T06:25:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:21.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:21.267575+0000) 2022-09-10T06:25:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:21 smithi104 ceph-mon[97044]: pgmap v3720: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:25:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:21.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:21.673340+0000) 2022-09-10T06:25:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:21 smithi110 ceph-mon[97633]: pgmap v3720: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:25:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:22.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:22.267772+0000) 2022-09-10T06:25:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:22.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:22.673579+0000) 2022-09-10T06:25:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:23.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:23.267941+0000) 2022-09-10T06:25:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:23 smithi104 ceph-mon[97044]: pgmap v3721: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:25:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:23.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:23.673842+0000) 2022-09-10T06:25:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:23 smithi110 ceph-mon[97633]: pgmap v3721: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:25:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:24.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:24.268184+0000) 2022-09-10T06:25:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:24.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:24.674152+0000) 2022-09-10T06:25:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:25.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:25.268414+0000) 2022-09-10T06:25:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:25 smithi104 ceph-mon[97044]: pgmap v3722: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:25:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:25.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:25.674375+0000) 2022-09-10T06:25:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:25 smithi110 ceph-mon[97633]: pgmap v3722: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:25:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:26.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:26.268555+0000) 2022-09-10T06:25:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:26.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:26.674612+0000) 2022-09-10T06:25:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:27.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:27.268783+0000) 2022-09-10T06:25:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:27 smithi104 ceph-mon[97044]: pgmap v3723: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:25:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:27.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:27.674832+0000) 2022-09-10T06:25:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:27 smithi110 ceph-mon[97633]: pgmap v3723: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:25:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:28.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:28.268975+0000) 2022-09-10T06:25:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:25:28.421+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:25:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:28.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:28.675013+0000) 2022-09-10T06:25:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:29.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:29.269174+0000) 2022-09-10T06:25:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:29 smithi104 ceph-mon[97044]: pgmap v3724: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:30.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:29.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:29.675250+0000) 2022-09-10T06:25:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:29 smithi110 ceph-mon[97633]: pgmap v3724: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:30.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:30.269397+0000) 2022-09-10T06:25:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:30.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:30.675477+0000) 2022-09-10T06:25:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:31.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:31.269549+0000) 2022-09-10T06:25:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:31 smithi104 ceph-mon[97044]: pgmap v3725: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:31.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:31.675706+0000) 2022-09-10T06:25:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:31 smithi110 ceph-mon[97633]: pgmap v3725: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:32.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:32.269812+0000) 2022-09-10T06:25:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:32.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:32.675978+0000) 2022-09-10T06:25:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:33.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:33.270061+0000) 2022-09-10T06:25:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:33 smithi104 ceph-mon[97044]: pgmap v3726: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:25:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:33.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:33.676231+0000) 2022-09-10T06:25:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:33 smithi110 ceph-mon[97633]: pgmap v3726: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:25:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:34.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:34.270309+0000) 2022-09-10T06:25:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:34.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:34.676500+0000) 2022-09-10T06:25:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:35.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:35.270562+0000) 2022-09-10T06:25:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:35 smithi104 ceph-mon[97044]: pgmap v3727: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:25:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:35.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:35.676767+0000) 2022-09-10T06:25:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:35 smithi110 ceph-mon[97633]: pgmap v3727: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:25:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:36.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:36.270834+0000) 2022-09-10T06:25:37.012 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:36.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:36.677007+0000) 2022-09-10T06:25:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:36 smithi110 ceph-mon[97633]: pgmap v3728: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:36 smithi104 ceph-mon[97044]: pgmap v3728: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:37.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:37.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:37.271066+0000) 2022-09-10T06:25:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:37.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:37.677202+0000) 2022-09-10T06:25:38.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:38.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:38.271274+0000) 2022-09-10T06:25:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:38.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:38.677417+0000) 2022-09-10T06:25:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:39.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:39.271443+0000) 2022-09-10T06:25:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:39 smithi104 ceph-mon[97044]: pgmap v3729: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:25:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:39.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:39.677592+0000) 2022-09-10T06:25:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:39 smithi110 ceph-mon[97633]: pgmap v3729: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:25:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:40.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:40.271689+0000) 2022-09-10T06:25:41.066 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:40.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:40.677757+0000) 2022-09-10T06:25:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:41.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:41.271960+0000) 2022-09-10T06:25:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:41 smithi104 ceph-mon[97044]: pgmap v3730: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:41.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:41.677957+0000) 2022-09-10T06:25:42.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:41 smithi110 ceph-mon[97633]: pgmap v3730: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:42.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:42.272243+0000) 2022-09-10T06:25:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:42.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:42.678204+0000) 2022-09-10T06:25:43.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:25:43.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:25:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:43.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:43.272507+0000) 2022-09-10T06:25:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:43 smithi104 ceph-mon[97044]: pgmap v3731: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:43.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:43.678360+0000) 2022-09-10T06:25:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:43 smithi110 ceph-mon[97633]: pgmap v3731: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:44.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:44.272669+0000) 2022-09-10T06:25:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:44.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:44.678625+0000) 2022-09-10T06:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:25:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:25:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:25:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:25:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:25:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:25:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:25:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:25:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:45.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:45.272904+0000) 2022-09-10T06:25:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:45.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:45.678893+0000) 2022-09-10T06:25:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:45 smithi110 ceph-mon[97633]: pgmap v3732: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:45 smithi104 ceph-mon[97044]: pgmap v3732: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:46.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:46.273210+0000) 2022-09-10T06:25:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:46.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:46.679177+0000) 2022-09-10T06:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:46 smithi110 ceph-mon[97633]: pgmap v3733: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:46 smithi104 ceph-mon[97044]: pgmap v3733: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:47.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:47.273478+0000) 2022-09-10T06:25:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:47.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:47.679397+0000) 2022-09-10T06:25:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:48.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:48.273752+0000) 2022-09-10T06:25:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:48.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:48.679640+0000) 2022-09-10T06:25:49.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:49.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:49.274013+0000) 2022-09-10T06:25:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:49 smithi104 ceph-mon[97044]: pgmap v3734: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:49.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:49.679875+0000) 2022-09-10T06:25:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:49 smithi110 ceph-mon[97633]: pgmap v3734: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:50.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:50.274308+0000) 2022-09-10T06:25:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:50.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:50.680079+0000) 2022-09-10T06:25:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:51.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:51.274540+0000) 2022-09-10T06:25:52.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:51 smithi104 ceph-mon[97044]: pgmap v3735: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:51.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:51.680324+0000) 2022-09-10T06:25:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:51 smithi110 ceph-mon[97633]: pgmap v3735: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:52.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:52.274795+0000) 2022-09-10T06:25:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:52.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:52.680564+0000) 2022-09-10T06:25:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:53.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:53.274972+0000) 2022-09-10T06:25:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:53.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:53.680787+0000) 2022-09-10T06:25:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:53 smithi110 ceph-mon[97633]: pgmap v3736: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:53 smithi104 ceph-mon[97044]: pgmap v3736: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:54.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:54.275214+0000) 2022-09-10T06:25:55.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:54.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:54.681077+0000) 2022-09-10T06:25:55.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:54 smithi110 ceph-mon[97633]: pgmap v3737: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:55.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:54 smithi104 ceph-mon[97044]: pgmap v3737: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:25:55.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:55.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:55.275404+0000) 2022-09-10T06:25:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:55.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:55.681288+0000) 2022-09-10T06:25:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:56.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:56.275640+0000) 2022-09-10T06:25:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:56.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:56.681543+0000) 2022-09-10T06:25:57.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:57 smithi104 ceph-mon[97044]: pgmap v3738: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:57 smithi110 ceph-mon[97633]: pgmap v3738: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:25:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:57.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:57.275916+0000) 2022-09-10T06:25:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:57.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:57.681736+0000) 2022-09-10T06:25:58.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:58.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:58.276169+0000) 2022-09-10T06:25:58.560 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:25:58.436+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:25:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:58.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:58.681937+0000) 2022-09-10T06:25:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:25:59.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:59.276450+0000) 2022-09-10T06:26:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:25:59 smithi104 ceph-mon[97044]: pgmap v3739: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:25:59.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:25:59.682159+0000) 2022-09-10T06:26:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:25:59 smithi110 ceph-mon[97633]: pgmap v3739: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:00.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:00.276715+0000) 2022-09-10T06:26:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:00.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:00.682299+0000) 2022-09-10T06:26:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:01.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:01.276970+0000) 2022-09-10T06:26:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:01.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:01.682552+0000) 2022-09-10T06:26:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:01 smithi110 ceph-mon[97633]: pgmap v3740: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:01 smithi104 ceph-mon[97044]: pgmap v3740: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:02.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:02.277231+0000) 2022-09-10T06:26:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:02.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:02.682781+0000) 2022-09-10T06:26:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:03.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:03.277451+0000) 2022-09-10T06:26:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:03.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:03.683012+0000) 2022-09-10T06:26:04.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:03 smithi110 ceph-mon[97633]: pgmap v3741: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:03 smithi104 ceph-mon[97044]: pgmap v3741: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:04.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:04.277727+0000) 2022-09-10T06:26:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:04.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:04.683162+0000) 2022-09-10T06:26:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:04 smithi110 ceph-mon[97633]: pgmap v3742: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:04 smithi104 ceph-mon[97044]: pgmap v3742: 129 pgs: 129 active+clean; 8.0 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:05.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:05.277974+0000) 2022-09-10T06:26:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:05.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:05.683356+0000) 2022-09-10T06:26:06.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:06.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:06.278217+0000) 2022-09-10T06:26:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:06.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:06.683571+0000) 2022-09-10T06:26:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:07 smithi104 ceph-mon[97044]: pgmap v3743: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:26:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:07 smithi110 ceph-mon[97633]: pgmap v3743: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:26:07.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:07.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:07.278460+0000) 2022-09-10T06:26:08.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:07.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:07.683801+0000) 2022-09-10T06:26:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:08.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:08.278727+0000) 2022-09-10T06:26:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:08.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:08.684017+0000) 2022-09-10T06:26:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:09 smithi104 ceph-mon[97044]: pgmap v3744: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:26:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:09 smithi104 ceph-mon[97044]: osdmap e316: 12 total, 12 up, 12 in 2022-09-10T06:26:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:09 smithi110 ceph-mon[97633]: pgmap v3744: 129 pgs: 129 active+clean; 8.0 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:26:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:09 smithi110 ceph-mon[97633]: osdmap e316: 12 total, 12 up, 12 in 2022-09-10T06:26:09.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:09.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:09.278957+0000) 2022-09-10T06:26:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:09.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:09.684251+0000) 2022-09-10T06:26:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:10.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:10.279145+0000) 2022-09-10T06:26:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:10.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:10.684440+0000) 2022-09-10T06:26:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:11 smithi104 ceph-mon[97044]: pgmap v3746: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:26:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:11 smithi110 ceph-mon[97633]: pgmap v3746: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:26:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:11.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:11.279341+0000) 2022-09-10T06:26:12.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:11.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:11.684642+0000) 2022-09-10T06:26:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:12.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:12.279621+0000) 2022-09-10T06:26:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:12.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:12.684873+0000) 2022-09-10T06:26:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:13 smithi110 ceph-mon[97633]: pgmap v3747: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:26:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:13 smithi104 ceph-mon[97044]: pgmap v3747: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:26:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:13.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:13.279841+0000) 2022-09-10T06:26:13.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:26:13.447+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:26:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:13.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:13.685164+0000) 2022-09-10T06:26:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:14.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:14.280079+0000) 2022-09-10T06:26:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:14.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:14.685377+0000) 2022-09-10T06:26:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:14 smithi110 ceph-mon[97633]: osdmap e317: 12 total, 12 up, 12 in 2022-09-10T06:26:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:14 smithi104 ceph-mon[97044]: osdmap e317: 12 total, 12 up, 12 in 2022-09-10T06:26:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:15.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:15.280328+0000) 2022-09-10T06:26:16.065 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:15.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:15.685622+0000) 2022-09-10T06:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:15 smithi110 ceph-mon[97633]: pgmap v3749: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:26:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:15 smithi104 ceph-mon[97044]: pgmap v3749: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:26:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:16.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:16.280582+0000) 2022-09-10T06:26:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:16.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:16.685850+0000) 2022-09-10T06:26:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:16 smithi110 ceph-mon[97633]: pgmap v3750: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:26:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:16 smithi104 ceph-mon[97044]: pgmap v3750: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:26:17.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:17.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:17.280874+0000) 2022-09-10T06:26:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:17.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:17.686166+0000) 2022-09-10T06:26:18.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:18.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:18.281175+0000) 2022-09-10T06:26:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:18.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:18.686397+0000) 2022-09-10T06:26:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:19.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:19.281376+0000) 2022-09-10T06:26:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:19.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:19.686606+0000) 2022-09-10T06:26:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:19 smithi110 ceph-mon[97633]: pgmap v3751: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:26:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:19 smithi104 ceph-mon[97044]: pgmap v3751: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:26:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:20.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:20.281567+0000) 2022-09-10T06:26:21.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:20.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:20.686865+0000) 2022-09-10T06:26:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:20 smithi110 ceph-mon[97633]: pgmap v3752: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:26:21.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:20 smithi104 ceph-mon[97044]: pgmap v3752: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:26:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:21.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:21.281743+0000) 2022-09-10T06:26:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:21.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:21.687159+0000) 2022-09-10T06:26:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:22.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:22.282044+0000) 2022-09-10T06:26:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:22.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:22.687306+0000) 2022-09-10T06:26:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:23 smithi104 ceph-mon[97044]: pgmap v3753: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:26:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:23 smithi110 ceph-mon[97633]: pgmap v3753: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:26:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:23.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:23.282306+0000) 2022-09-10T06:26:24.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:23.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:23.687458+0000) 2022-09-10T06:26:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:24.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:24.282528+0000) 2022-09-10T06:26:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:24.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:24.687592+0000) 2022-09-10T06:26:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:25.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:25.282782+0000) 2022-09-10T06:26:26.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:25.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:25.687797+0000) 2022-09-10T06:26:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:25 smithi110 ceph-mon[97633]: pgmap v3754: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:26:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:25 smithi104 ceph-mon[97044]: pgmap v3754: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:26:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:26.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:26.283062+0000) 2022-09-10T06:26:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:26.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:26.688010+0000) 2022-09-10T06:26:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:26 smithi110 ceph-mon[97633]: pgmap v3755: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:26:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:26 smithi104 ceph-mon[97044]: pgmap v3755: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:26:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:27.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:27.283378+0000) 2022-09-10T06:26:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:27.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:27.688208+0000) 2022-09-10T06:26:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:28.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:28.283623+0000) 2022-09-10T06:26:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:26:28.462+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:26:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:28.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:28.688375+0000) 2022-09-10T06:26:29.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:29.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:29.283820+0000) 2022-09-10T06:26:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:29.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:29.688600+0000) 2022-09-10T06:26:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:29 smithi110 ceph-mon[97633]: pgmap v3756: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:29 smithi104 ceph-mon[97044]: pgmap v3756: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:30.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:30.284046+0000) 2022-09-10T06:26:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:30.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:30.688806+0000) 2022-09-10T06:26:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:30 smithi110 ceph-mon[97633]: pgmap v3757: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:30 smithi104 ceph-mon[97044]: pgmap v3757: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:31.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:31.284278+0000) 2022-09-10T06:26:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:31.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:31.689040+0000) 2022-09-10T06:26:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:32.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:32.284441+0000) 2022-09-10T06:26:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:32.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:32.689352+0000) 2022-09-10T06:26:33.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:33 smithi104 ceph-mon[97044]: pgmap v3758: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:33 smithi110 ceph-mon[97633]: pgmap v3758: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:33.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:33.284688+0000) 2022-09-10T06:26:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:33.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:33.689539+0000) 2022-09-10T06:26:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:34.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:34.284915+0000) 2022-09-10T06:26:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:34.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:34.689792+0000) 2022-09-10T06:26:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:34 smithi110 ceph-mon[97633]: pgmap v3759: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:34 smithi104 ceph-mon[97044]: pgmap v3759: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:35.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:35.285171+0000) 2022-09-10T06:26:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:35.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:35.690046+0000) 2022-09-10T06:26:36.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:36.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:36.285373+0000) 2022-09-10T06:26:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:36.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:36.690241+0000) 2022-09-10T06:26:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:37 smithi104 ceph-mon[97044]: pgmap v3760: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:37 smithi110 ceph-mon[97633]: pgmap v3760: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:37.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:37.285543+0000) 2022-09-10T06:26:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:37.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:37.690391+0000) 2022-09-10T06:26:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:38.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:38.285726+0000) 2022-09-10T06:26:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:38.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:38.690552+0000) 2022-09-10T06:26:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:39 smithi110 ceph-mon[97633]: pgmap v3761: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:39 smithi104 ceph-mon[97044]: pgmap v3761: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:39.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:39.285966+0000) 2022-09-10T06:26:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:39.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:39.690686+0000) 2022-09-10T06:26:40.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:40.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:40.286222+0000) 2022-09-10T06:26:41.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:40.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:40.690868+0000) 2022-09-10T06:26:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:41 smithi110 ceph-mon[97633]: pgmap v3762: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:41 smithi104 ceph-mon[97044]: pgmap v3762: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:41.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:41.286483+0000) 2022-09-10T06:26:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:41.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:41.691041+0000) 2022-09-10T06:26:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:42.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:42.286736+0000) 2022-09-10T06:26:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:42.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:42.691277+0000) 2022-09-10T06:26:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:43 smithi110 ceph-mon[97633]: pgmap v3763: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:43 smithi104 ceph-mon[97044]: pgmap v3763: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:43.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:43.286944+0000) 2022-09-10T06:26:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:26:43.471+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:26:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:43.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:43.691540+0000) 2022-09-10T06:26:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:44.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:44.287171+0000) 2022-09-10T06:26:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:44.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:44.691787+0000) 2022-09-10T06:26:45.168 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:26:45.169 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:26:45.169 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:26:45.169 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:26:45.170 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:44 smithi110 ceph-mon[97633]: pgmap v3764: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:26:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:26:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:26:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:26:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:44 smithi104 ceph-mon[97044]: pgmap v3764: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:45.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:45.287396+0000) 2022-09-10T06:26:46.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:45.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:45.692001+0000) 2022-09-10T06:26:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:46.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:46.287672+0000) 2022-09-10T06:26:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:46.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:46.692257+0000) 2022-09-10T06:26:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:47 smithi110 ceph-mon[97633]: pgmap v3765: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:47 smithi104 ceph-mon[97044]: pgmap v3765: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:47.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:47.287926+0000) 2022-09-10T06:26:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:47.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:47.692456+0000) 2022-09-10T06:26:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:48.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:48.288137+0000) 2022-09-10T06:26:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:48.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:48.692721+0000) 2022-09-10T06:26:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:49 smithi110 ceph-mon[97633]: pgmap v3766: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:49 smithi104 ceph-mon[97044]: pgmap v3766: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:49.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:49.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:49.288352+0000) 2022-09-10T06:26:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:49.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:49.692948+0000) 2022-09-10T06:26:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:50.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:50.288537+0000) 2022-09-10T06:26:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:50.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:50.693182+0000) 2022-09-10T06:26:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:51 smithi110 ceph-mon[97633]: pgmap v3767: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:51 smithi104 ceph-mon[97044]: pgmap v3767: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:51.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:51.288778+0000) 2022-09-10T06:26:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:51.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:51.693336+0000) 2022-09-10T06:26:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:52.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:52.288949+0000) 2022-09-10T06:26:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:52.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:52.693556+0000) 2022-09-10T06:26:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:53 smithi110 ceph-mon[97633]: pgmap v3768: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:53 smithi104 ceph-mon[97044]: pgmap v3768: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:53.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:53.289184+0000) 2022-09-10T06:26:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:53.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:53.693751+0000) 2022-09-10T06:26:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:54.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:54.289399+0000) 2022-09-10T06:26:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:54.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:54.694007+0000) 2022-09-10T06:26:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:54 smithi110 ceph-mon[97633]: pgmap v3769: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:54 smithi104 ceph-mon[97044]: pgmap v3769: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:55.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:55.289540+0000) 2022-09-10T06:26:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:55.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:55.694241+0000) 2022-09-10T06:26:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:56.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:56.289686+0000) 2022-09-10T06:26:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:56.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:56.694433+0000) 2022-09-10T06:26:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:57 smithi110 ceph-mon[97633]: pgmap v3770: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:57 smithi104 ceph-mon[97044]: pgmap v3770: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:26:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:57.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:57.289887+0000) 2022-09-10T06:26:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:57.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:57.694597+0000) 2022-09-10T06:26:58.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:26:58.484+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:26:58.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:58.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:58.290139+0000) 2022-09-10T06:26:59.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:58.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:58.694763+0000) 2022-09-10T06:26:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:26:59 smithi110 ceph-mon[97633]: pgmap v3771: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:26:59 smithi104 ceph-mon[97044]: pgmap v3771: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:26:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:26:59.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:59.290414+0000) 2022-09-10T06:27:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:26:59.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:26:59.694954+0000) 2022-09-10T06:27:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:00.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:00.290676+0000) 2022-09-10T06:27:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:00.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:00.695152+0000) 2022-09-10T06:27:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:01 smithi110 ceph-mon[97633]: pgmap v3772: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:01 smithi104 ceph-mon[97044]: pgmap v3772: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:01.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:01.290870+0000) 2022-09-10T06:27:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:01.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:01.695350+0000) 2022-09-10T06:27:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:02.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:02.291056+0000) 2022-09-10T06:27:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:02.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:02.695496+0000) 2022-09-10T06:27:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:03 smithi110 ceph-mon[97633]: pgmap v3773: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:03 smithi104 ceph-mon[97044]: pgmap v3773: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:03.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:03.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:03.291309+0000) 2022-09-10T06:27:04.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:03.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:03.695693+0000) 2022-09-10T06:27:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:04.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:04.291576+0000) 2022-09-10T06:27:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:04.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:04.695945+0000) 2022-09-10T06:27:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:04 smithi110 ceph-mon[97633]: pgmap v3774: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:04 smithi104 ceph-mon[97044]: pgmap v3774: 129 pgs: 129 active+clean; 8.0 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:05.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:05.291748+0000) 2022-09-10T06:27:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:05.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:05.696155+0000) 2022-09-10T06:27:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:06.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:06.291949+0000) 2022-09-10T06:27:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:06.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:06.696388+0000) 2022-09-10T06:27:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:07 smithi110 ceph-mon[97633]: pgmap v3775: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:27:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:07 smithi104 ceph-mon[97044]: pgmap v3775: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:27:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:07.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:07.292200+0000) 2022-09-10T06:27:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:07.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:07.696666+0000) 2022-09-10T06:27:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:08.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:08.292445+0000) 2022-09-10T06:27:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:08.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:08.696918+0000) 2022-09-10T06:27:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:09 smithi110 ceph-mon[97633]: pgmap v3776: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:27:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:09 smithi110 ceph-mon[97633]: osdmap e318: 12 total, 12 up, 12 in 2022-09-10T06:27:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:09 smithi104 ceph-mon[97044]: pgmap v3776: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:27:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:09 smithi104 ceph-mon[97044]: osdmap e318: 12 total, 12 up, 12 in 2022-09-10T06:27:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:09.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:09.292649+0000) 2022-09-10T06:27:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:09.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:09.697172+0000) 2022-09-10T06:27:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:10.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:10.292901+0000) 2022-09-10T06:27:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:10.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:10.697370+0000) 2022-09-10T06:27:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:11 smithi110 ceph-mon[97633]: pgmap v3778: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:27:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:11 smithi104 ceph-mon[97044]: pgmap v3778: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:27:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:11.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:11.293170+0000) 2022-09-10T06:27:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:11.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:11.697628+0000) 2022-09-10T06:27:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:12.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:12.293407+0000) 2022-09-10T06:27:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:12.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:12.697837+0000) 2022-09-10T06:27:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:13 smithi110 ceph-mon[97633]: pgmap v3779: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:27:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:13 smithi104 ceph-mon[97044]: pgmap v3779: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:27:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:13.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:13.293608+0000) 2022-09-10T06:27:13.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:27:13.500+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:27:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:13.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:13.698050+0000) 2022-09-10T06:27:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:14.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:14.293862+0000) 2022-09-10T06:27:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:14.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:14.698309+0000) 2022-09-10T06:27:15.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:14 smithi110 ceph-mon[97633]: osdmap e319: 12 total, 12 up, 12 in 2022-09-10T06:27:15.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:14 smithi110 ceph-mon[97633]: pgmap v3781: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:27:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:14 smithi104 ceph-mon[97044]: osdmap e319: 12 total, 12 up, 12 in 2022-09-10T06:27:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:14 smithi104 ceph-mon[97044]: pgmap v3781: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:27:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:15.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:15.294136+0000) 2022-09-10T06:27:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:15.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:15.698569+0000) 2022-09-10T06:27:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:16.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:16.294364+0000) 2022-09-10T06:27:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:16.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:16.698796+0000) 2022-09-10T06:27:17.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:17 smithi110 ceph-mon[97633]: pgmap v3782: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:27:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:17 smithi104 ceph-mon[97044]: pgmap v3782: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:27:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:17.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:17.294593+0000) 2022-09-10T06:27:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:17.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:17.699037+0000) 2022-09-10T06:27:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:18.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:18.294891+0000) 2022-09-10T06:27:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:18.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:18.699200+0000) 2022-09-10T06:27:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:19 smithi110 ceph-mon[97633]: pgmap v3783: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:27:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:19 smithi104 ceph-mon[97044]: pgmap v3783: 129 pgs: 129 active+clean; 8.1 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:27:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:19.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:19.295129+0000) 2022-09-10T06:27:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:19.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:19.699346+0000) 2022-09-10T06:27:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:20.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:20.295283+0000) 2022-09-10T06:27:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:20.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:20.699588+0000) 2022-09-10T06:27:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:21 smithi110 ceph-mon[97633]: pgmap v3784: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:27:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:21 smithi104 ceph-mon[97044]: pgmap v3784: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:27:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:21.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:21.295471+0000) 2022-09-10T06:27:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:21.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:21.699813+0000) 2022-09-10T06:27:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:22.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:22.295630+0000) 2022-09-10T06:27:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:22.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:22.700050+0000) 2022-09-10T06:27:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:23 smithi110 ceph-mon[97633]: pgmap v3785: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:27:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:23 smithi104 ceph-mon[97044]: pgmap v3785: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:27:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:23.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:23.295839+0000) 2022-09-10T06:27:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:23.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:23.700267+0000) 2022-09-10T06:27:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:24.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:24.296045+0000) 2022-09-10T06:27:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:24.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:24.700508+0000) 2022-09-10T06:27:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:24 smithi110 ceph-mon[97633]: pgmap v3786: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:27:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:24 smithi104 ceph-mon[97044]: pgmap v3786: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:27:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:25.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:25.296246+0000) 2022-09-10T06:27:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:25.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:25.700639+0000) 2022-09-10T06:27:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:26.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:26.296505+0000) 2022-09-10T06:27:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:26.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:26.700822+0000) 2022-09-10T06:27:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:27 smithi110 ceph-mon[97633]: pgmap v3787: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:27:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:27 smithi104 ceph-mon[97044]: pgmap v3787: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:27:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:27.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:27.296784+0000) 2022-09-10T06:27:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:27.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:27.701017+0000) 2022-09-10T06:27:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:28.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:28.296987+0000) 2022-09-10T06:27:28.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:27:28.515+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:27:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:28.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:28.701257+0000) 2022-09-10T06:27:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:29 smithi110 ceph-mon[97633]: pgmap v3788: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:29 smithi104 ceph-mon[97044]: pgmap v3788: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:29.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:29.297159+0000) 2022-09-10T06:27:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:29.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:29.701506+0000) 2022-09-10T06:27:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:30.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:30.297386+0000) 2022-09-10T06:27:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:30.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:30.701740+0000) 2022-09-10T06:27:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:31 smithi110 ceph-mon[97633]: pgmap v3789: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:31 smithi104 ceph-mon[97044]: pgmap v3789: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:31.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:31.297651+0000) 2022-09-10T06:27:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:31.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:31.702031+0000) 2022-09-10T06:27:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:32.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:32.297876+0000) 2022-09-10T06:27:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:32.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:32.702343+0000) 2022-09-10T06:27:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:33 smithi110 ceph-mon[97633]: pgmap v3790: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:33 smithi104 ceph-mon[97044]: pgmap v3790: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:33.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:33.298143+0000) 2022-09-10T06:27:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:33.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:33.702539+0000) 2022-09-10T06:27:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:34.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:34.298349+0000) 2022-09-10T06:27:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:34.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:34.702784+0000) 2022-09-10T06:27:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:34 smithi110 ceph-mon[97633]: pgmap v3791: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:34 smithi104 ceph-mon[97044]: pgmap v3791: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:35.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:35.298615+0000) 2022-09-10T06:27:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:35.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:35.703044+0000) 2022-09-10T06:27:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:36.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:36.298873+0000) 2022-09-10T06:27:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:36.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:36.703294+0000) 2022-09-10T06:27:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:37 smithi110 ceph-mon[97633]: pgmap v3792: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:37 smithi104 ceph-mon[97044]: pgmap v3792: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:37.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:37.299153+0000) 2022-09-10T06:27:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:37.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:37.703514+0000) 2022-09-10T06:27:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:38.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:38.299335+0000) 2022-09-10T06:27:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:38.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:38.703751+0000) 2022-09-10T06:27:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:39 smithi110 ceph-mon[97633]: pgmap v3793: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:39 smithi104 ceph-mon[97044]: pgmap v3793: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:39.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:39.299553+0000) 2022-09-10T06:27:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:39.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:39.704038+0000) 2022-09-10T06:27:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:40.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:40.299816+0000) 2022-09-10T06:27:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:40.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:40.704288+0000) 2022-09-10T06:27:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:41 smithi110 ceph-mon[97633]: pgmap v3794: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:41 smithi104 ceph-mon[97044]: pgmap v3794: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:41.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:41.300045+0000) 2022-09-10T06:27:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:41.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:41.704444+0000) 2022-09-10T06:27:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:42.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:42.300302+0000) 2022-09-10T06:27:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:42.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:42.704650+0000) 2022-09-10T06:27:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:43 smithi110 ceph-mon[97633]: pgmap v3795: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:43 smithi104 ceph-mon[97044]: pgmap v3795: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:43.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:27:43.525+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:27:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:43.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:43.300443+0000) 2022-09-10T06:27:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:43.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:43.704899+0000) 2022-09-10T06:27:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:44.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:44.300657+0000) 2022-09-10T06:27:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:44.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:44.705194+0000) 2022-09-10T06:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:27:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:27:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:44 smithi110 ceph-mon[97633]: pgmap v3796: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:27:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:27:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:27:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:27:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:44 smithi104 ceph-mon[97044]: pgmap v3796: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:45.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:45.300901+0000) 2022-09-10T06:27:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:45.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:45.705435+0000) 2022-09-10T06:27:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:46.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:46.301129+0000) 2022-09-10T06:27:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:46.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:46.705667+0000) 2022-09-10T06:27:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:47 smithi110 ceph-mon[97633]: pgmap v3797: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:47 smithi104 ceph-mon[97044]: pgmap v3797: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:47.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:47.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:47.301270+0000) 2022-09-10T06:27:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:47.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:47.705950+0000) 2022-09-10T06:27:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:48.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:48.301507+0000) 2022-09-10T06:27:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:48.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:48.706206+0000) 2022-09-10T06:27:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:49 smithi110 ceph-mon[97633]: pgmap v3798: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:49 smithi104 ceph-mon[97044]: pgmap v3798: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:49.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:49.301690+0000) 2022-09-10T06:27:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:49.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:49.706450+0000) 2022-09-10T06:27:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:50.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:50.301977+0000) 2022-09-10T06:27:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:50.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:50.706650+0000) 2022-09-10T06:27:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:51 smithi110 ceph-mon[97633]: pgmap v3799: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:51 smithi104 ceph-mon[97044]: pgmap v3799: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:51.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:51.302225+0000) 2022-09-10T06:27:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:51.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:51.706905+0000) 2022-09-10T06:27:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:52.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:52.302428+0000) 2022-09-10T06:27:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:52.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:52.707145+0000) 2022-09-10T06:27:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:53 smithi110 ceph-mon[97633]: pgmap v3800: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:53 smithi104 ceph-mon[97044]: pgmap v3800: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:53.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:53.302650+0000) 2022-09-10T06:27:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:53.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:53.707318+0000) 2022-09-10T06:27:54.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:54.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:54.302865+0000) 2022-09-10T06:27:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:54.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:54.707544+0000) 2022-09-10T06:27:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:54 smithi110 ceph-mon[97633]: pgmap v3801: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:54 smithi104 ceph-mon[97044]: pgmap v3801: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:55.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:55.303049+0000) 2022-09-10T06:27:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:55.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:55.707742+0000) 2022-09-10T06:27:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:56.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:56.303232+0000) 2022-09-10T06:27:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:56.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:56.707961+0000) 2022-09-10T06:27:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:57 smithi110 ceph-mon[97633]: pgmap v3802: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:57.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:57 smithi104 ceph-mon[97044]: pgmap v3802: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:27:57.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:57.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:57.303496+0000) 2022-09-10T06:27:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:57.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:57.708207+0000) 2022-09-10T06:27:58.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:27:58.541+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:27:58.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:58.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:58.303675+0000) 2022-09-10T06:27:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:58.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:58.708373+0000) 2022-09-10T06:27:59.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:27:59 smithi110 ceph-mon[97633]: pgmap v3803: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:27:59 smithi104 ceph-mon[97044]: pgmap v3803: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:27:59.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:27:59.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:59.303875+0000) 2022-09-10T06:28:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:27:59.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:27:59.708619+0000) 2022-09-10T06:28:00.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:00.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:00.304152+0000) 2022-09-10T06:28:01.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:00.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:00.708876+0000) 2022-09-10T06:28:01.418 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:01 smithi110 ceph-mon[97633]: pgmap v3804: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:01.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:01 smithi104 ceph-mon[97044]: pgmap v3804: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:01.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:01.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:01.304323+0000) 2022-09-10T06:28:01.976 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:01.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:01.709129+0000) 2022-09-10T06:28:02.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:02.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:02.304490+0000) 2022-09-10T06:28:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:02.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:02.709372+0000) 2022-09-10T06:28:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:03 smithi110 ceph-mon[97633]: pgmap v3805: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:03 smithi104 ceph-mon[97044]: pgmap v3805: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:03.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:03.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:03.304670+0000) 2022-09-10T06:28:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:03.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:03.709654+0000) 2022-09-10T06:28:04.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:04.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:04.304898+0000) 2022-09-10T06:28:05.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:04.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:04.709929+0000) 2022-09-10T06:28:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:04 smithi110 ceph-mon[97633]: pgmap v3806: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:04 smithi104 ceph-mon[97044]: pgmap v3806: 129 pgs: 129 active+clean; 8.1 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:05.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:05.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:05.305162+0000) 2022-09-10T06:28:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:05.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:05.710146+0000) 2022-09-10T06:28:06.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:06.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:06.305428+0000) 2022-09-10T06:28:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:06.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:06.710318+0000) 2022-09-10T06:28:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:07 smithi110 ceph-mon[97633]: pgmap v3807: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:28:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:07 smithi104 ceph-mon[97044]: pgmap v3807: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:28:07.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:07.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:07.305640+0000) 2022-09-10T06:28:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:07.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:07.710547+0000) 2022-09-10T06:28:08.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:08.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:08.305783+0000) 2022-09-10T06:28:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:08.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:08.710801+0000) 2022-09-10T06:28:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:09 smithi110 ceph-mon[97633]: pgmap v3808: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:28:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:09 smithi110 ceph-mon[97633]: osdmap e320: 12 total, 12 up, 12 in 2022-09-10T06:28:09.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:09 smithi104 ceph-mon[97044]: pgmap v3808: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:28:09.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:09 smithi104 ceph-mon[97044]: osdmap e320: 12 total, 12 up, 12 in 2022-09-10T06:28:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:09.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:09.306015+0000) 2022-09-10T06:28:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:09.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:09.710968+0000) 2022-09-10T06:28:10.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:10.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:10.306309+0000) 2022-09-10T06:28:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:10.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:10.711198+0000) 2022-09-10T06:28:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:11 smithi110 ceph-mon[97633]: pgmap v3810: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:28:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:11 smithi104 ceph-mon[97044]: pgmap v3810: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:28:11.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:11.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:11.306579+0000) 2022-09-10T06:28:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:11.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:11.711426+0000) 2022-09-10T06:28:12.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:12.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:12.306795+0000) 2022-09-10T06:28:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:12.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:12.711656+0000) 2022-09-10T06:28:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:13 smithi110 ceph-mon[97633]: pgmap v3811: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:28:13.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:13 smithi104 ceph-mon[97044]: pgmap v3811: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:28:13.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:13.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:13.306974+0000) 2022-09-10T06:28:13.714 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:28:13.555+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:28:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:13.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:13.711903+0000) 2022-09-10T06:28:14.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:14.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:14.307155+0000) 2022-09-10T06:28:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:14.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:14.712157+0000) 2022-09-10T06:28:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:14 smithi110 ceph-mon[97633]: osdmap e321: 12 total, 12 up, 12 in 2022-09-10T06:28:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:14 smithi110 ceph-mon[97633]: pgmap v3813: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:28:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:14 smithi104 ceph-mon[97044]: osdmap e321: 12 total, 12 up, 12 in 2022-09-10T06:28:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:14 smithi104 ceph-mon[97044]: pgmap v3813: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:28:15.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:15.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:15.307442+0000) 2022-09-10T06:28:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:15.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:15.712308+0000) 2022-09-10T06:28:16.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:16.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:16.307651+0000) 2022-09-10T06:28:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:16.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:16.712523+0000) 2022-09-10T06:28:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:17 smithi110 ceph-mon[97633]: pgmap v3814: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:28:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:17 smithi104 ceph-mon[97044]: pgmap v3814: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:28:17.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:17.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:17.307902+0000) 2022-09-10T06:28:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:17.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:17.712725+0000) 2022-09-10T06:28:18.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:18.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:18.308212+0000) 2022-09-10T06:28:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:18.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:18.712971+0000) 2022-09-10T06:28:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:19 smithi110 ceph-mon[97633]: pgmap v3815: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:28:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:19 smithi104 ceph-mon[97044]: pgmap v3815: 129 pgs: 129 active+clean; 8.1 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:28:19.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:19.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:19.308361+0000) 2022-09-10T06:28:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:19.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:19.713259+0000) 2022-09-10T06:28:20.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:20.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:20.308527+0000) 2022-09-10T06:28:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:20.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:20.713490+0000) 2022-09-10T06:28:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:21 smithi110 ceph-mon[97633]: pgmap v3816: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:28:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:21 smithi104 ceph-mon[97044]: pgmap v3816: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:28:21.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:21.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:21.308782+0000) 2022-09-10T06:28:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:21.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:21.713690+0000) 2022-09-10T06:28:22.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:22.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:22.309076+0000) 2022-09-10T06:28:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:22.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:22.713907+0000) 2022-09-10T06:28:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:23 smithi110 ceph-mon[97633]: pgmap v3817: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:28:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:23 smithi104 ceph-mon[97044]: pgmap v3817: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:28:23.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:23.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:23.309319+0000) 2022-09-10T06:28:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:23.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:23.714188+0000) 2022-09-10T06:28:24.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:24.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:24.309500+0000) 2022-09-10T06:28:25.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:24.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:24.714362+0000) 2022-09-10T06:28:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:24 smithi110 ceph-mon[97633]: pgmap v3818: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:28:25.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:24 smithi104 ceph-mon[97044]: pgmap v3818: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:28:25.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:25.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:25.309739+0000) 2022-09-10T06:28:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:25.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:25.714566+0000) 2022-09-10T06:28:26.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:26.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:26.309968+0000) 2022-09-10T06:28:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:26.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:26.714778+0000) 2022-09-10T06:28:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:27 smithi110 ceph-mon[97633]: pgmap v3819: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:28:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:27 smithi104 ceph-mon[97044]: pgmap v3819: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:28:27.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:27.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:27.310190+0000) 2022-09-10T06:28:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:27.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:27.715031+0000) 2022-09-10T06:28:28.573 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:28.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:28.310400+0000) 2022-09-10T06:28:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:28.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:28.715272+0000) 2022-09-10T06:28:29.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:28:28.571+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:28:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:29 smithi110 ceph-mon[97633]: pgmap v3820: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:29 smithi104 ceph-mon[97044]: pgmap v3820: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:29.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:29.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:29.310596+0000) 2022-09-10T06:28:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:29.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:29.715476+0000) 2022-09-10T06:28:30.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:30.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:30.310827+0000) 2022-09-10T06:28:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:30.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:30.715711+0000) 2022-09-10T06:28:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:31 smithi110 ceph-mon[97633]: pgmap v3821: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:31 smithi104 ceph-mon[97044]: pgmap v3821: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:31.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:31.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:31.311044+0000) 2022-09-10T06:28:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:31.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:31.715953+0000) 2022-09-10T06:28:32.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:32.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:32.311333+0000) 2022-09-10T06:28:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:32.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:32.716189+0000) 2022-09-10T06:28:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:33 smithi110 ceph-mon[97633]: pgmap v3822: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:33 smithi104 ceph-mon[97044]: pgmap v3822: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:33.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:33.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:33.311564+0000) 2022-09-10T06:28:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:33.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:33.716422+0000) 2022-09-10T06:28:34.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:34.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:34.311753+0000) 2022-09-10T06:28:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:34.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:34.716595+0000) 2022-09-10T06:28:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:34 smithi110 ceph-mon[97633]: pgmap v3823: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:34 smithi104 ceph-mon[97044]: pgmap v3823: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:35.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:35.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:35.311923+0000) 2022-09-10T06:28:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:35.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:35.716835+0000) 2022-09-10T06:28:36.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:36.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:36.312165+0000) 2022-09-10T06:28:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:36.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:36.717123+0000) 2022-09-10T06:28:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:37 smithi110 ceph-mon[97633]: pgmap v3824: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:37 smithi104 ceph-mon[97044]: pgmap v3824: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:37.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:37.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:37.312338+0000) 2022-09-10T06:28:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:37.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:37.717339+0000) 2022-09-10T06:28:38.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:38.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:38.312600+0000) 2022-09-10T06:28:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:38.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:38.717569+0000) 2022-09-10T06:28:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:39 smithi110 ceph-mon[97633]: pgmap v3825: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:39 smithi104 ceph-mon[97044]: pgmap v3825: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:39.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:39.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:39.312815+0000) 2022-09-10T06:28:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:39.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:39.717735+0000) 2022-09-10T06:28:40.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:40.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:40.313056+0000) 2022-09-10T06:28:41.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:40.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:40.717952+0000) 2022-09-10T06:28:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:41 smithi110 ceph-mon[97633]: pgmap v3826: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:41 smithi104 ceph-mon[97044]: pgmap v3826: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:41.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:41.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:41.313371+0000) 2022-09-10T06:28:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:41.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:41.718191+0000) 2022-09-10T06:28:42.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:42.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:42.313610+0000) 2022-09-10T06:28:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:42.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:42.718444+0000) 2022-09-10T06:28:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:43 smithi110 ceph-mon[97633]: pgmap v3827: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:43 smithi104 ceph-mon[97044]: pgmap v3827: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:43.588 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:43.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:43.313808+0000) 2022-09-10T06:28:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:43.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:43.718722+0000) 2022-09-10T06:28:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:28:43.587+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:28:44.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:44.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:44.314078+0000) 2022-09-10T06:28:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:44.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:44.718994+0000) 2022-09-10T06:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:28:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:28:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:28:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:44 smithi110 ceph-mon[97633]: pgmap v3828: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:28:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:28:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:28:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:28:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:44 smithi104 ceph-mon[97044]: pgmap v3828: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:45.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:45.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:45.314329+0000) 2022-09-10T06:28:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:45.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:45.719195+0000) 2022-09-10T06:28:46.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:46.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:46.314493+0000) 2022-09-10T06:28:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:46.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:46.719397+0000) 2022-09-10T06:28:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:47 smithi110 ceph-mon[97633]: pgmap v3829: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:47 smithi104 ceph-mon[97044]: pgmap v3829: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:47.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:47.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:47.314724+0000) 2022-09-10T06:28:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:47.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:47.719585+0000) 2022-09-10T06:28:48.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:48.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:48.314890+0000) 2022-09-10T06:28:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:48.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:48.719742+0000) 2022-09-10T06:28:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:49 smithi110 ceph-mon[97633]: pgmap v3830: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:49 smithi104 ceph-mon[97044]: pgmap v3830: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:49.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:49.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:49.315073+0000) 2022-09-10T06:28:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:49.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:49.719921+0000) 2022-09-10T06:28:50.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:50.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:50.315328+0000) 2022-09-10T06:28:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:50.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:50.720134+0000) 2022-09-10T06:28:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:51 smithi110 ceph-mon[97633]: pgmap v3831: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:51 smithi104 ceph-mon[97044]: pgmap v3831: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:51.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:51.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:51.315527+0000) 2022-09-10T06:28:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:51.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:51.720388+0000) 2022-09-10T06:28:52.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:52.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:52.315738+0000) 2022-09-10T06:28:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:52.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:52.720655+0000) 2022-09-10T06:28:53.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:53 smithi110 ceph-mon[97633]: pgmap v3832: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:53.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:53 smithi104 ceph-mon[97044]: pgmap v3832: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:53.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:53.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:53.315985+0000) 2022-09-10T06:28:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:53.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:53.720889+0000) 2022-09-10T06:28:54.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:54.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:54.316245+0000) 2022-09-10T06:28:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:54.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:54.721080+0000) 2022-09-10T06:28:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:54 smithi110 ceph-mon[97633]: pgmap v3833: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:54 smithi104 ceph-mon[97044]: pgmap v3833: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:28:55.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:55.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:55.316465+0000) 2022-09-10T06:28:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:55.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:55.721300+0000) 2022-09-10T06:28:56.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:56.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:56.316673+0000) 2022-09-10T06:28:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:56.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:56.721511+0000) 2022-09-10T06:28:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:57 smithi110 ceph-mon[97633]: pgmap v3834: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:57 smithi104 ceph-mon[97044]: pgmap v3834: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:28:57.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:57.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:57.316908+0000) 2022-09-10T06:28:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:57.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:57.721729+0000) 2022-09-10T06:28:58.596 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:58.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:58.317207+0000) 2022-09-10T06:28:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:58.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:58.721940+0000) 2022-09-10T06:28:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:28:58.594+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:28:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:28:59 smithi110 ceph-mon[97633]: pgmap v3835: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:28:59 smithi104 ceph-mon[97044]: pgmap v3835: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:28:59.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:28:59.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:59.317362+0000) 2022-09-10T06:29:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:28:59.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:28:59.722192+0000) 2022-09-10T06:29:00.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:00.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:00.317503+0000) 2022-09-10T06:29:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:00.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:00.722422+0000) 2022-09-10T06:29:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:01 smithi110 ceph-mon[97633]: pgmap v3836: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:01.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:01 smithi104 ceph-mon[97044]: pgmap v3836: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:01.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:01.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:01.317635+0000) 2022-09-10T06:29:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:01.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:01.722665+0000) 2022-09-10T06:29:02.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:02.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:02.317837+0000) 2022-09-10T06:29:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:02.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:02.722856+0000) 2022-09-10T06:29:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:03 smithi110 ceph-mon[97633]: pgmap v3837: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:03 smithi104 ceph-mon[97044]: pgmap v3837: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:03.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:03.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:03.318136+0000) 2022-09-10T06:29:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:03.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:03.723022+0000) 2022-09-10T06:29:04.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:04.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:04.318330+0000) 2022-09-10T06:29:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:04.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:04.723259+0000) 2022-09-10T06:29:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:04 smithi110 ceph-mon[97633]: pgmap v3838: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:04 smithi104 ceph-mon[97044]: pgmap v3838: 129 pgs: 129 active+clean; 8.1 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:05.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:05.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:05.318532+0000) 2022-09-10T06:29:06.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:05.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:05.723513+0000) 2022-09-10T06:29:06.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:06.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:06.318771+0000) 2022-09-10T06:29:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:06.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:06.723697+0000) 2022-09-10T06:29:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:07 smithi110 ceph-mon[97633]: pgmap v3839: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:29:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:07 smithi104 ceph-mon[97044]: pgmap v3839: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:29:07.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:07.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:07.318998+0000) 2022-09-10T06:29:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:07.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:07.723881+0000) 2022-09-10T06:29:08.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:08.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:08.319237+0000) 2022-09-10T06:29:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:08.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:08.724142+0000) 2022-09-10T06:29:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:09 smithi110 ceph-mon[97633]: pgmap v3840: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:29:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:09 smithi110 ceph-mon[97633]: osdmap e322: 12 total, 12 up, 12 in 2022-09-10T06:29:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:09 smithi104 ceph-mon[97044]: pgmap v3840: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:29:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:09 smithi104 ceph-mon[97044]: osdmap e322: 12 total, 12 up, 12 in 2022-09-10T06:29:09.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:09.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:09.319495+0000) 2022-09-10T06:29:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:09.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:09.724359+0000) 2022-09-10T06:29:10.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:10.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:10.319652+0000) 2022-09-10T06:29:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:10.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:10.724660+0000) 2022-09-10T06:29:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:11 smithi110 ceph-mon[97633]: pgmap v3842: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:29:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:11 smithi104 ceph-mon[97044]: pgmap v3842: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:29:11.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:11.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:11.319890+0000) 2022-09-10T06:29:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:11.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:11.724867+0000) 2022-09-10T06:29:12.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:12.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:12.320197+0000) 2022-09-10T06:29:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:12.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:12.725125+0000) 2022-09-10T06:29:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:13 smithi110 ceph-mon[97633]: pgmap v3843: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:29:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:13 smithi104 ceph-mon[97044]: pgmap v3843: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:29:13.600 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:13.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:13.320428+0000) 2022-09-10T06:29:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:13.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:13.725273+0000) 2022-09-10T06:29:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:29:13.598+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:29:14.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:14.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:14.320617+0000) 2022-09-10T06:29:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:14.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:14.725525+0000) 2022-09-10T06:29:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:14 smithi110 ceph-mon[97633]: osdmap e323: 12 total, 12 up, 12 in 2022-09-10T06:29:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:14 smithi110 ceph-mon[97633]: pgmap v3845: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:29:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:14 smithi104 ceph-mon[97044]: osdmap e323: 12 total, 12 up, 12 in 2022-09-10T06:29:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:14 smithi104 ceph-mon[97044]: pgmap v3845: 129 pgs: 129 active+clean; 8.1 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:29:15.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:15.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:15.320819+0000) 2022-09-10T06:29:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:15.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:15.725761+0000) 2022-09-10T06:29:16.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:16.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:16.321050+0000) 2022-09-10T06:29:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:16.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:16.725973+0000) 2022-09-10T06:29:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:17 smithi110 ceph-mon[97633]: pgmap v3846: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:29:17.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:17 smithi104 ceph-mon[97044]: pgmap v3846: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:29:17.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:17.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:17.321255+0000) 2022-09-10T06:29:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:17.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:17.726213+0000) 2022-09-10T06:29:18.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:18.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:18.321432+0000) 2022-09-10T06:29:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:18.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:18.726425+0000) 2022-09-10T06:29:19.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:19 smithi110 ceph-mon[97633]: pgmap v3847: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:29:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:19 smithi104 ceph-mon[97044]: pgmap v3847: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:29:19.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:19.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:19.321650+0000) 2022-09-10T06:29:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:19.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:19.726615+0000) 2022-09-10T06:29:20.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:20.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:20.321911+0000) 2022-09-10T06:29:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:20.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:20.726841+0000) 2022-09-10T06:29:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:21 smithi110 ceph-mon[97633]: pgmap v3848: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:29:21.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:21 smithi104 ceph-mon[97044]: pgmap v3848: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:29:21.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:21.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:21.322149+0000) 2022-09-10T06:29:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:21.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:21.727075+0000) 2022-09-10T06:29:22.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:22.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:22.322348+0000) 2022-09-10T06:29:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:22.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:22.727276+0000) 2022-09-10T06:29:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:23 smithi110 ceph-mon[97633]: pgmap v3849: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:29:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:23 smithi104 ceph-mon[97044]: pgmap v3849: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:29:23.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:23.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:23.322594+0000) 2022-09-10T06:29:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:23.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:23.727464+0000) 2022-09-10T06:29:24.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:24.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:24.322840+0000) 2022-09-10T06:29:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:24.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:24.727621+0000) 2022-09-10T06:29:25.187 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:24 smithi110 ceph-mon[97633]: pgmap v3850: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:29:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:24 smithi104 ceph-mon[97044]: pgmap v3850: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T06:29:25.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:25.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:25.323045+0000) 2022-09-10T06:29:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:25.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:25.727745+0000) 2022-09-10T06:29:26.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:26.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:26.323261+0000) 2022-09-10T06:29:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:26.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:26.727907+0000) 2022-09-10T06:29:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:27 smithi110 ceph-mon[97633]: pgmap v3851: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:29:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:27 smithi104 ceph-mon[97044]: pgmap v3851: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:29:27.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:27.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:27.323456+0000) 2022-09-10T06:29:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:27.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:27.728125+0000) 2022-09-10T06:29:28.611 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:28.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:28.323630+0000) 2022-09-10T06:29:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:28.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:28.728306+0000) 2022-09-10T06:29:29.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:29:28.609+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:29:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:29 smithi110 ceph-mon[97633]: pgmap v3852: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:29 smithi104 ceph-mon[97044]: pgmap v3852: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:29.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:29.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:29.323890+0000) 2022-09-10T06:29:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:29.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:29.728511+0000) 2022-09-10T06:29:30.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:30.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:30.324121+0000) 2022-09-10T06:29:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:30.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:30.728746+0000) 2022-09-10T06:29:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:31 smithi110 ceph-mon[97633]: pgmap v3853: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:31 smithi104 ceph-mon[97044]: pgmap v3853: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:31.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:31.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:31.324302+0000) 2022-09-10T06:29:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:31.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:31.729045+0000) 2022-09-10T06:29:32.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:32.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:32.324478+0000) 2022-09-10T06:29:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:32.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:32.729258+0000) 2022-09-10T06:29:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:33 smithi110 ceph-mon[97633]: pgmap v3854: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:33 smithi104 ceph-mon[97044]: pgmap v3854: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:33.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:33.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:33.324721+0000) 2022-09-10T06:29:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:33.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:33.729470+0000) 2022-09-10T06:29:34.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:34.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:34.324978+0000) 2022-09-10T06:29:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:34.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:34.729719+0000) 2022-09-10T06:29:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:34 smithi110 ceph-mon[97633]: pgmap v3855: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:34 smithi104 ceph-mon[97044]: pgmap v3855: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:35.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:35.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:35.325196+0000) 2022-09-10T06:29:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:35.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:35.729957+0000) 2022-09-10T06:29:36.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:36.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:36.325446+0000) 2022-09-10T06:29:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:36.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:36.730143+0000) 2022-09-10T06:29:37.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:37 smithi110 ceph-mon[97633]: pgmap v3856: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:37 smithi104 ceph-mon[97044]: pgmap v3856: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:37.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:37.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:37.325698+0000) 2022-09-10T06:29:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:37.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:37.730362+0000) 2022-09-10T06:29:38.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:38.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:38.325930+0000) 2022-09-10T06:29:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:38.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:38.730578+0000) 2022-09-10T06:29:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:39 smithi110 ceph-mon[97633]: pgmap v3857: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:39 smithi104 ceph-mon[97044]: pgmap v3857: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:39.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:39.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:39.326152+0000) 2022-09-10T06:29:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:39.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:39.730712+0000) 2022-09-10T06:29:40.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:40.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:40.326371+0000) 2022-09-10T06:29:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:40.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:40.730882+0000) 2022-09-10T06:29:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:41 smithi110 ceph-mon[97633]: pgmap v3858: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:41 smithi104 ceph-mon[97044]: pgmap v3858: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:41.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:41.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:41.326599+0000) 2022-09-10T06:29:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:41.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:41.731120+0000) 2022-09-10T06:29:42.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:42.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:42.326782+0000) 2022-09-10T06:29:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:42.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:42.731322+0000) 2022-09-10T06:29:43.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:43 smithi110 ceph-mon[97633]: pgmap v3859: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:43 smithi104 ceph-mon[97044]: pgmap v3859: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:43.626 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:43.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:43.326996+0000) 2022-09-10T06:29:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:43.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:43.731555+0000) 2022-09-10T06:29:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:29:43.625+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:29:44.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:44.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:44.327209+0000) 2022-09-10T06:29:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:44.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:44.731811+0000) 2022-09-10T06:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:29:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:29:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:44 smithi110 ceph-mon[97633]: pgmap v3860: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:29:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:29:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:29:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:29:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:44 smithi104 ceph-mon[97044]: pgmap v3860: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:45.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:45.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:45.327382+0000) 2022-09-10T06:29:46.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:45.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:45.732075+0000) 2022-09-10T06:29:46.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:46.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:46.327619+0000) 2022-09-10T06:29:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:46.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:46.732344+0000) 2022-09-10T06:29:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:47 smithi110 ceph-mon[97633]: pgmap v3861: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:47 smithi104 ceph-mon[97044]: pgmap v3861: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:47.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:47.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:47.327847+0000) 2022-09-10T06:29:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:47.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:47.732624+0000) 2022-09-10T06:29:48.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:48.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:48.328082+0000) 2022-09-10T06:29:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:48.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:48.732896+0000) 2022-09-10T06:29:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:49 smithi110 ceph-mon[97633]: pgmap v3862: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:49 smithi104 ceph-mon[97044]: pgmap v3862: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:49.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:49.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:49.328357+0000) 2022-09-10T06:29:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:49.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:49.733073+0000) 2022-09-10T06:29:50.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:50.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:50.328559+0000) 2022-09-10T06:29:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:50.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:50.733195+0000) 2022-09-10T06:29:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:51 smithi110 ceph-mon[97633]: pgmap v3863: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:51 smithi104 ceph-mon[97044]: pgmap v3863: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:51.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:51.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:51.328798+0000) 2022-09-10T06:29:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:51.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:51.733450+0000) 2022-09-10T06:29:52.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:52.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:52.329017+0000) 2022-09-10T06:29:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:52.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:52.733695+0000) 2022-09-10T06:29:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:53 smithi110 ceph-mon[97633]: pgmap v3864: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:53 smithi104 ceph-mon[97044]: pgmap v3864: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:53.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:53.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:53.329260+0000) 2022-09-10T06:29:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:53.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:53.733916+0000) 2022-09-10T06:29:54.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:54.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:54.329485+0000) 2022-09-10T06:29:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:54.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:54.734148+0000) 2022-09-10T06:29:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:54 smithi110 ceph-mon[97633]: pgmap v3865: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:54 smithi104 ceph-mon[97044]: pgmap v3865: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:55.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:55.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:55.329650+0000) 2022-09-10T06:29:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:55.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:55.734388+0000) 2022-09-10T06:29:56.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:56.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:56.329872+0000) 2022-09-10T06:29:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:56.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:56.734598+0000) 2022-09-10T06:29:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:57 smithi110 ceph-mon[97633]: pgmap v3866: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:57 smithi104 ceph-mon[97044]: pgmap v3866: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:29:57.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:57.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:57.330169+0000) 2022-09-10T06:29:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:57.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:57.734783+0000) 2022-09-10T06:29:58.627 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:58.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:58.330394+0000) 2022-09-10T06:29:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:58.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:58.735019+0000) 2022-09-10T06:29:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:29:58.625+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:29:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:29:59 smithi110 ceph-mon[97633]: pgmap v3867: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:29:59 smithi104 ceph-mon[97044]: pgmap v3867: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:29:59.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:29:59.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:59.330664+0000) 2022-09-10T06:30:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:29:59.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:29:59.735247+0000) 2022-09-10T06:30:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:30:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:30:00.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:00.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:00.330924+0000) 2022-09-10T06:30:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:00.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:00.735397+0000) 2022-09-10T06:30:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:01 smithi110 ceph-mon[97633]: pgmap v3868: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:01.467 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:01 smithi104 ceph-mon[97044]: pgmap v3868: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:01.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:01.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:01.331197+0000) 2022-09-10T06:30:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:01.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:01.735618+0000) 2022-09-10T06:30:02.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:02.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:02.331458+0000) 2022-09-10T06:30:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:02.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:02.735868+0000) 2022-09-10T06:30:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:03 smithi110 ceph-mon[97633]: pgmap v3869: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:03 smithi104 ceph-mon[97044]: pgmap v3869: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:03.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:03.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:03.331725+0000) 2022-09-10T06:30:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:03.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:03.736056+0000) 2022-09-10T06:30:04.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:04.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:04.332001+0000) 2022-09-10T06:30:05.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:04.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:04.736294+0000) 2022-09-10T06:30:05.168 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:04 smithi110 ceph-mon[97633]: pgmap v3870: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:04 smithi104 ceph-mon[97044]: pgmap v3870: 129 pgs: 129 active+clean; 8.1 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:05.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:05.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:05.332217+0000) 2022-09-10T06:30:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:05.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:05.736439+0000) 2022-09-10T06:30:06.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:06.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:06.332423+0000) 2022-09-10T06:30:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:06.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:06.736660+0000) 2022-09-10T06:30:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:07 smithi110 ceph-mon[97633]: pgmap v3871: 129 pgs: 129 active+clean; 8.1 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:30:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:07 smithi104 ceph-mon[97044]: pgmap v3871: 129 pgs: 129 active+clean; 8.1 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:30:07.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:07.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:07.332646+0000) 2022-09-10T06:30:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:07.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:07.736931+0000) 2022-09-10T06:30:08.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:08.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:08.332882+0000) 2022-09-10T06:30:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:08.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:08.737185+0000) 2022-09-10T06:30:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:09 smithi110 ceph-mon[97633]: pgmap v3872: 129 pgs: 129 active+clean; 8.1 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:30:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:09 smithi110 ceph-mon[97633]: osdmap e324: 12 total, 12 up, 12 in 2022-09-10T06:30:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:09 smithi104 ceph-mon[97044]: pgmap v3872: 129 pgs: 129 active+clean; 8.1 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:30:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:09 smithi104 ceph-mon[97044]: osdmap e324: 12 total, 12 up, 12 in 2022-09-10T06:30:09.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:09.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:09.333109+0000) 2022-09-10T06:30:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:09.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:09.737335+0000) 2022-09-10T06:30:10.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:10.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:10.333270+0000) 2022-09-10T06:30:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:10.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:10.737527+0000) 2022-09-10T06:30:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:11 smithi110 ceph-mon[97633]: pgmap v3874: 129 pgs: 129 active+clean; 8.1 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:30:11.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:11 smithi104 ceph-mon[97044]: pgmap v3874: 129 pgs: 129 active+clean; 8.1 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:30:11.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:11.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:11.333464+0000) 2022-09-10T06:30:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:11.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:11.737799+0000) 2022-09-10T06:30:12.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:12.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:12.333700+0000) 2022-09-10T06:30:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:12.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:12.738006+0000) 2022-09-10T06:30:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:13 smithi110 ceph-mon[97633]: pgmap v3875: 129 pgs: 129 active+clean; 8.1 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:30:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:13 smithi104 ceph-mon[97044]: pgmap v3875: 129 pgs: 129 active+clean; 8.1 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:30:13.642 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:13.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:13.333965+0000) 2022-09-10T06:30:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:13.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:13.738222+0000) 2022-09-10T06:30:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:30:13.640+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:30:14.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:14.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:14.334181+0000) 2022-09-10T06:30:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:14.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:14.738452+0000) 2022-09-10T06:30:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:14 smithi110 ceph-mon[97633]: osdmap e325: 12 total, 12 up, 12 in 2022-09-10T06:30:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:14 smithi110 ceph-mon[97633]: pgmap v3877: 129 pgs: 129 active+clean; 8.1 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:30:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:14 smithi104 ceph-mon[97044]: osdmap e325: 12 total, 12 up, 12 in 2022-09-10T06:30:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:14 smithi104 ceph-mon[97044]: pgmap v3877: 129 pgs: 129 active+clean; 8.1 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:30:15.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:15.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:15.334438+0000) 2022-09-10T06:30:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:15.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:15.738660+0000) 2022-09-10T06:30:16.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:16.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:16.334720+0000) 2022-09-10T06:30:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:16.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:16.738834+0000) 2022-09-10T06:30:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:17 smithi110 ceph-mon[97633]: pgmap v3878: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:30:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:17 smithi104 ceph-mon[97044]: pgmap v3878: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:30:17.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:17.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:17.335002+0000) 2022-09-10T06:30:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:17.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:17.739057+0000) 2022-09-10T06:30:18.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:18.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:18.335304+0000) 2022-09-10T06:30:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:18.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:18.739274+0000) 2022-09-10T06:30:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:19 smithi110 ceph-mon[97633]: pgmap v3879: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:30:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:19 smithi104 ceph-mon[97044]: pgmap v3879: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:30:19.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:19.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:19.335596+0000) 2022-09-10T06:30:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:19.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:19.739506+0000) 2022-09-10T06:30:20.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:20.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:20.335857+0000) 2022-09-10T06:30:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:20.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:20.739680+0000) 2022-09-10T06:30:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:21 smithi110 ceph-mon[97633]: pgmap v3880: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:30:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:21 smithi104 ceph-mon[97044]: pgmap v3880: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:30:21.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:21.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:21.336175+0000) 2022-09-10T06:30:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:21.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:21.739958+0000) 2022-09-10T06:30:22.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:22.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:22.336443+0000) 2022-09-10T06:30:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:22.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:22.740168+0000) 2022-09-10T06:30:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:23 smithi110 ceph-mon[97633]: pgmap v3881: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:30:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:23 smithi104 ceph-mon[97044]: pgmap v3881: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:30:23.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:23.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:23.336699+0000) 2022-09-10T06:30:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:23.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:23.740410+0000) 2022-09-10T06:30:24.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:24.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:24.336990+0000) 2022-09-10T06:30:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:24.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:24.740619+0000) 2022-09-10T06:30:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:24 smithi110 ceph-mon[97633]: pgmap v3882: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:30:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:24 smithi104 ceph-mon[97044]: pgmap v3882: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:30:25.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:25.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:25.337176+0000) 2022-09-10T06:30:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:25.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:25.740833+0000) 2022-09-10T06:30:26.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:26.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:26.337364+0000) 2022-09-10T06:30:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:26.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:26.741047+0000) 2022-09-10T06:30:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:27 smithi110 ceph-mon[97633]: pgmap v3883: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:30:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:27 smithi104 ceph-mon[97044]: pgmap v3883: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:30:27.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:27.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:27.337585+0000) 2022-09-10T06:30:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:27.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:27.741302+0000) 2022-09-10T06:30:28.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:28.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:28.337790+0000) 2022-09-10T06:30:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:30:28.656+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:30:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:28.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:28.741511+0000) 2022-09-10T06:30:29.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:29 smithi110 ceph-mon[97633]: pgmap v3884: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:29 smithi104 ceph-mon[97044]: pgmap v3884: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:29.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:29.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:29.338010+0000) 2022-09-10T06:30:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:29.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:29.741718+0000) 2022-09-10T06:30:30.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:30.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:30.338224+0000) 2022-09-10T06:30:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:30.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:30.741885+0000) 2022-09-10T06:30:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:31 smithi110 ceph-mon[97633]: pgmap v3885: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:31 smithi104 ceph-mon[97044]: pgmap v3885: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:31.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:31.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:31.338443+0000) 2022-09-10T06:30:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:31.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:31.742060+0000) 2022-09-10T06:30:32.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:32.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:32.338632+0000) 2022-09-10T06:30:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:32.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:32.742278+0000) 2022-09-10T06:30:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:33 smithi110 ceph-mon[97633]: pgmap v3886: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:33 smithi104 ceph-mon[97044]: pgmap v3886: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:33.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:33.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:33.338812+0000) 2022-09-10T06:30:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:33.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:33.742524+0000) 2022-09-10T06:30:34.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:34.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:34.339022+0000) 2022-09-10T06:30:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:34.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:34.742672+0000) 2022-09-10T06:30:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:34 smithi110 ceph-mon[97633]: pgmap v3887: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:34 smithi104 ceph-mon[97044]: pgmap v3887: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:35.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:35.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:35.339242+0000) 2022-09-10T06:30:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:35.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:35.742894+0000) 2022-09-10T06:30:36.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:36.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:36.339391+0000) 2022-09-10T06:30:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:36.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:36.743165+0000) 2022-09-10T06:30:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:37 smithi110 ceph-mon[97633]: pgmap v3888: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:37 smithi104 ceph-mon[97044]: pgmap v3888: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:37.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:37.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:37.339602+0000) 2022-09-10T06:30:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:37.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:37.743322+0000) 2022-09-10T06:30:38.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:38.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:38.339827+0000) 2022-09-10T06:30:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:38.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:38.743461+0000) 2022-09-10T06:30:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:39 smithi110 ceph-mon[97633]: pgmap v3889: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:39 smithi104 ceph-mon[97044]: pgmap v3889: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:39.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:39.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:39.340041+0000) 2022-09-10T06:30:40.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:39.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:39.743690+0000) 2022-09-10T06:30:40.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:40.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:40.340233+0000) 2022-09-10T06:30:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:40.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:40.743949+0000) 2022-09-10T06:30:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:41 smithi110 ceph-mon[97633]: pgmap v3890: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:41.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:41 smithi104 ceph-mon[97044]: pgmap v3890: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:41.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:41.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:41.340377+0000) 2022-09-10T06:30:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:41.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:41.744187+0000) 2022-09-10T06:30:42.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:42.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:42.340603+0000) 2022-09-10T06:30:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:42.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:42.744348+0000) 2022-09-10T06:30:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:43 smithi110 ceph-mon[97633]: pgmap v3891: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:43 smithi104 ceph-mon[97044]: pgmap v3891: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:43.673 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:43.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:43.340819+0000) 2022-09-10T06:30:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:43.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:43.744575+0000) 2022-09-10T06:30:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:30:43.671+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:30:44.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:44.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:44.341052+0000) 2022-09-10T06:30:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:44.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:44.744793+0000) 2022-09-10T06:30:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:30:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:30:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:30:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:30:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:44 smithi110 ceph-mon[97633]: pgmap v3892: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:30:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:30:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:30:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:30:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:44 smithi104 ceph-mon[97044]: pgmap v3892: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:45.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:45.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:45.341299+0000) 2022-09-10T06:30:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:45.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:45.745049+0000) 2022-09-10T06:30:46.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:46.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:46.341500+0000) 2022-09-10T06:30:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:46.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:46.745263+0000) 2022-09-10T06:30:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:47 smithi110 ceph-mon[97633]: pgmap v3893: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:47 smithi104 ceph-mon[97044]: pgmap v3893: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:47.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:47.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:47.341705+0000) 2022-09-10T06:30:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:47.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:47.745511+0000) 2022-09-10T06:30:48.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:48.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:48.341937+0000) 2022-09-10T06:30:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:48.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:48.745742+0000) 2022-09-10T06:30:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:49 smithi110 ceph-mon[97633]: pgmap v3894: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:49 smithi104 ceph-mon[97044]: pgmap v3894: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:49.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:49.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:49.342221+0000) 2022-09-10T06:30:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:49.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:49.745941+0000) 2022-09-10T06:30:50.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:50.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:50.342463+0000) 2022-09-10T06:30:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:50.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:50.746070+0000) 2022-09-10T06:30:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:51 smithi110 ceph-mon[97633]: pgmap v3895: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:51 smithi104 ceph-mon[97044]: pgmap v3895: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:51.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:51.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:51.342631+0000) 2022-09-10T06:30:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:51.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:51.746230+0000) 2022-09-10T06:30:52.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:52.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:52.342916+0000) 2022-09-10T06:30:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:52.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:52.746422+0000) 2022-09-10T06:30:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:53 smithi110 ceph-mon[97633]: pgmap v3896: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:53 smithi104 ceph-mon[97044]: pgmap v3896: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:53.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:53.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:53.343187+0000) 2022-09-10T06:30:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:53.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:53.746629+0000) 2022-09-10T06:30:54.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:54.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:54.343354+0000) 2022-09-10T06:30:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:54.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:54.746866+0000) 2022-09-10T06:30:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:54 smithi110 ceph-mon[97633]: pgmap v3897: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:54 smithi104 ceph-mon[97044]: pgmap v3897: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:55.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:55.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:55.343521+0000) 2022-09-10T06:30:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:55.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:55.747156+0000) 2022-09-10T06:30:56.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:56.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:56.343708+0000) 2022-09-10T06:30:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:56.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:56.747391+0000) 2022-09-10T06:30:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:57 smithi110 ceph-mon[97633]: pgmap v3898: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:57 smithi104 ceph-mon[97044]: pgmap v3898: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:30:57.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:57.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:57.343964+0000) 2022-09-10T06:30:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:57.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:57.747639+0000) 2022-09-10T06:30:58.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:58.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:58.344213+0000) 2022-09-10T06:30:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:30:58.677+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:30:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:58.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:58.747893+0000) 2022-09-10T06:30:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:30:59 smithi110 ceph-mon[97633]: pgmap v3899: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:59.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:30:59 smithi104 ceph-mon[97044]: pgmap v3899: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:30:59.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:30:59.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:59.344449+0000) 2022-09-10T06:31:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:30:59.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:30:59.748148+0000) 2022-09-10T06:31:00.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:00.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:00.344691+0000) 2022-09-10T06:31:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:00.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:00.748340+0000) 2022-09-10T06:31:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:01 smithi110 ceph-mon[97633]: pgmap v3900: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:01 smithi104 ceph-mon[97044]: pgmap v3900: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:01.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:01.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:01.344941+0000) 2022-09-10T06:31:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:01.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:01.748636+0000) 2022-09-10T06:31:02.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:02.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:02.345276+0000) 2022-09-10T06:31:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:02.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:02.748786+0000) 2022-09-10T06:31:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:03 smithi110 ceph-mon[97633]: pgmap v3901: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:03 smithi104 ceph-mon[97044]: pgmap v3901: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:03.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:03.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:03.345443+0000) 2022-09-10T06:31:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:03.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:03.748983+0000) 2022-09-10T06:31:04.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:04.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:04.345704+0000) 2022-09-10T06:31:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:04.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:04.749171+0000) 2022-09-10T06:31:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:04 smithi110 ceph-mon[97633]: pgmap v3902: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:04 smithi104 ceph-mon[97044]: pgmap v3902: 129 pgs: 129 active+clean; 8.1 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:05.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:05.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:05.345924+0000) 2022-09-10T06:31:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:05.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:05.749325+0000) 2022-09-10T06:31:06.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:06.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:06.346144+0000) 2022-09-10T06:31:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:06.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:06.749519+0000) 2022-09-10T06:31:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:07 smithi110 ceph-mon[97633]: pgmap v3903: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:31:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:07 smithi104 ceph-mon[97044]: pgmap v3903: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:31:07.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:07.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:07.346337+0000) 2022-09-10T06:31:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:07.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:07.749762+0000) 2022-09-10T06:31:08.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:08.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:08.346523+0000) 2022-09-10T06:31:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:08.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:08.750019+0000) 2022-09-10T06:31:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:09 smithi110 ceph-mon[97633]: pgmap v3904: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:31:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:09 smithi110 ceph-mon[97633]: osdmap e326: 12 total, 12 up, 12 in 2022-09-10T06:31:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:09 smithi104 ceph-mon[97044]: pgmap v3904: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:31:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:09 smithi104 ceph-mon[97044]: osdmap e326: 12 total, 12 up, 12 in 2022-09-10T06:31:09.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:09.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:09.346747+0000) 2022-09-10T06:31:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:09.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:09.750227+0000) 2022-09-10T06:31:10.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:10.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:10.346952+0000) 2022-09-10T06:31:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:10.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:10.750451+0000) 2022-09-10T06:31:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:11 smithi110 ceph-mon[97633]: pgmap v3906: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:31:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:11 smithi104 ceph-mon[97044]: pgmap v3906: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:31:11.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:11.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:11.347175+0000) 2022-09-10T06:31:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:11.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:11.750694+0000) 2022-09-10T06:31:12.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:12.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:12.347385+0000) 2022-09-10T06:31:13.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:12.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:12.750863+0000) 2022-09-10T06:31:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:13 smithi110 ceph-mon[97633]: pgmap v3907: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:31:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:13 smithi104 ceph-mon[97044]: pgmap v3907: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:31:13.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:13.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:13.347564+0000) 2022-09-10T06:31:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:31:13.685+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:31:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:13.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:13.751065+0000) 2022-09-10T06:31:14.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:14.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:14.347801+0000) 2022-09-10T06:31:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:14.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:14.751277+0000) 2022-09-10T06:31:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:14 smithi110 ceph-mon[97633]: osdmap e327: 12 total, 12 up, 12 in 2022-09-10T06:31:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:14 smithi110 ceph-mon[97633]: pgmap v3909: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:31:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:14 smithi104 ceph-mon[97044]: osdmap e327: 12 total, 12 up, 12 in 2022-09-10T06:31:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:14 smithi104 ceph-mon[97044]: pgmap v3909: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:31:15.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:15.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:15.348048+0000) 2022-09-10T06:31:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:15.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:15.751479+0000) 2022-09-10T06:31:16.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:16.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:16.348226+0000) 2022-09-10T06:31:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:16.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:16.751660+0000) 2022-09-10T06:31:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:17 smithi110 ceph-mon[97633]: pgmap v3910: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:31:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:17 smithi104 ceph-mon[97044]: pgmap v3910: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:31:17.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:17.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:17.348466+0000) 2022-09-10T06:31:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:17.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:17.751835+0000) 2022-09-10T06:31:18.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:18.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:18.348634+0000) 2022-09-10T06:31:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:18.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:18.752070+0000) 2022-09-10T06:31:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:19 smithi110 ceph-mon[97633]: pgmap v3911: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:31:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:19 smithi104 ceph-mon[97044]: pgmap v3911: 129 pgs: 129 active+clean; 8.1 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:31:19.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:19.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:19.348873+0000) 2022-09-10T06:31:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:19.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:19.752337+0000) 2022-09-10T06:31:20.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:20.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:20.348993+0000) 2022-09-10T06:31:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:20.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:20.752539+0000) 2022-09-10T06:31:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:21 smithi110 ceph-mon[97633]: pgmap v3912: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:31:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:21 smithi104 ceph-mon[97044]: pgmap v3912: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:31:21.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:21.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:21.349236+0000) 2022-09-10T06:31:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:21.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:21.752686+0000) 2022-09-10T06:31:22.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:22.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:22.349492+0000) 2022-09-10T06:31:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:22.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:22.752908+0000) 2022-09-10T06:31:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:23 smithi110 ceph-mon[97633]: pgmap v3913: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:31:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:23 smithi104 ceph-mon[97044]: pgmap v3913: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:31:23.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:23.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:23.349686+0000) 2022-09-10T06:31:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:23.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:23.753150+0000) 2022-09-10T06:31:24.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:24.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:24.349951+0000) 2022-09-10T06:31:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:24.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:24.753391+0000) 2022-09-10T06:31:25.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:24 smithi110 ceph-mon[97633]: pgmap v3914: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:31:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:24 smithi104 ceph-mon[97044]: pgmap v3914: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:31:25.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:25.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:25.350196+0000) 2022-09-10T06:31:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:25.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:25.753627+0000) 2022-09-10T06:31:26.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:26.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:26.350445+0000) 2022-09-10T06:31:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:26.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:26.753910+0000) 2022-09-10T06:31:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:27 smithi110 ceph-mon[97633]: pgmap v3915: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:31:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:27 smithi104 ceph-mon[97044]: pgmap v3915: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:31:27.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:27.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:27.350663+0000) 2022-09-10T06:31:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:27.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:27.754183+0000) 2022-09-10T06:31:28.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:28.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:28.350860+0000) 2022-09-10T06:31:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:31:28.696+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:31:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:28.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:28.754361+0000) 2022-09-10T06:31:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:29 smithi110 ceph-mon[97633]: pgmap v3916: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:29 smithi104 ceph-mon[97044]: pgmap v3916: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:29.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:29.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:29.351134+0000) 2022-09-10T06:31:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:29.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:29.754539+0000) 2022-09-10T06:31:30.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:30.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:30.351372+0000) 2022-09-10T06:31:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:30.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:30.754725+0000) 2022-09-10T06:31:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:31 smithi110 ceph-mon[97633]: pgmap v3917: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:31 smithi104 ceph-mon[97044]: pgmap v3917: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:31.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:31.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:31.351551+0000) 2022-09-10T06:31:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:31.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:31.754951+0000) 2022-09-10T06:31:32.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:32.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:32.351792+0000) 2022-09-10T06:31:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:32.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:32.755203+0000) 2022-09-10T06:31:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:33 smithi110 ceph-mon[97633]: pgmap v3918: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:33 smithi104 ceph-mon[97044]: pgmap v3918: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:33.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:33.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:33.352026+0000) 2022-09-10T06:31:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:33.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:33.755445+0000) 2022-09-10T06:31:34.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:34.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:34.352264+0000) 2022-09-10T06:31:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:34.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:34.755609+0000) 2022-09-10T06:31:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:34 smithi110 ceph-mon[97633]: pgmap v3919: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:34 smithi104 ceph-mon[97044]: pgmap v3919: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:35.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:35.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:35.352513+0000) 2022-09-10T06:31:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:35.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:35.755837+0000) 2022-09-10T06:31:36.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:36.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:36.352691+0000) 2022-09-10T06:31:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:36.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:36.756084+0000) 2022-09-10T06:31:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:37 smithi110 ceph-mon[97633]: pgmap v3920: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:37 smithi104 ceph-mon[97044]: pgmap v3920: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:37.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:37.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:37.352895+0000) 2022-09-10T06:31:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:37.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:37.756351+0000) 2022-09-10T06:31:38.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:38.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:38.353074+0000) 2022-09-10T06:31:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:38.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:38.756597+0000) 2022-09-10T06:31:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:39 smithi110 ceph-mon[97633]: pgmap v3921: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:39 smithi104 ceph-mon[97044]: pgmap v3921: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:39.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:39.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:39.353284+0000) 2022-09-10T06:31:40.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:39.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:39.756778+0000) 2022-09-10T06:31:40.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:40.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:40.353497+0000) 2022-09-10T06:31:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:40.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:40.757009+0000) 2022-09-10T06:31:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:41 smithi110 ceph-mon[97633]: pgmap v3922: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:41 smithi104 ceph-mon[97044]: pgmap v3922: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:41.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:41.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:41.353741+0000) 2022-09-10T06:31:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:41.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:41.757231+0000) 2022-09-10T06:31:42.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:42.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:42.353982+0000) 2022-09-10T06:31:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:42.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:42.757456+0000) 2022-09-10T06:31:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:43 smithi110 ceph-mon[97633]: pgmap v3923: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:43 smithi104 ceph-mon[97044]: pgmap v3923: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:43.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:43.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:43.354249+0000) 2022-09-10T06:31:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:43.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:43.757667+0000) 2022-09-10T06:31:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:31:43.705+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:31:44.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:44.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:44.354488+0000) 2022-09-10T06:31:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:44.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:44.757859+0000) 2022-09-10T06:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:31:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:31:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:31:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:44 smithi110 ceph-mon[97633]: pgmap v3924: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:31:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:31:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:31:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:31:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:44 smithi104 ceph-mon[97044]: pgmap v3924: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:45.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:45.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:45.354688+0000) 2022-09-10T06:31:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:45.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:45.758091+0000) 2022-09-10T06:31:46.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:46.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:46.354875+0000) 2022-09-10T06:31:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:46.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:46.758253+0000) 2022-09-10T06:31:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:47 smithi110 ceph-mon[97633]: pgmap v3925: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:47 smithi104 ceph-mon[97044]: pgmap v3925: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:47.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:47.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:47.355073+0000) 2022-09-10T06:31:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:47.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:47.758378+0000) 2022-09-10T06:31:48.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:48.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:48.355276+0000) 2022-09-10T06:31:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:48.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:48.758584+0000) 2022-09-10T06:31:49.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:49 smithi110 ceph-mon[97633]: pgmap v3926: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:49 smithi104 ceph-mon[97044]: pgmap v3926: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:49.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:49.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:49.355425+0000) 2022-09-10T06:31:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:49.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:49.758802+0000) 2022-09-10T06:31:50.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:50.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:50.355663+0000) 2022-09-10T06:31:51.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:50.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:50.758978+0000) 2022-09-10T06:31:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:51 smithi110 ceph-mon[97633]: pgmap v3927: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:51 smithi104 ceph-mon[97044]: pgmap v3927: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:51.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:51.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:51.355918+0000) 2022-09-10T06:31:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:51.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:51.759162+0000) 2022-09-10T06:31:52.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:52.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:52.356164+0000) 2022-09-10T06:31:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:52.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:52.759398+0000) 2022-09-10T06:31:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:53 smithi110 ceph-mon[97633]: pgmap v3928: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:53 smithi104 ceph-mon[97044]: pgmap v3928: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:53.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:53.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:53.356382+0000) 2022-09-10T06:31:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:53.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:53.759556+0000) 2022-09-10T06:31:54.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:54.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:54.356601+0000) 2022-09-10T06:31:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:54.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:54.759786+0000) 2022-09-10T06:31:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:54 smithi110 ceph-mon[97633]: pgmap v3929: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:54 smithi104 ceph-mon[97044]: pgmap v3929: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:55.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:55.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:55.356806+0000) 2022-09-10T06:31:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:55.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:55.760058+0000) 2022-09-10T06:31:56.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:56.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:56.357004+0000) 2022-09-10T06:31:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:56.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:56.760350+0000) 2022-09-10T06:31:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:57 smithi110 ceph-mon[97633]: pgmap v3930: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:57 smithi104 ceph-mon[97044]: pgmap v3930: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:31:57.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:57.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:57.357180+0000) 2022-09-10T06:31:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:57.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:57.760584+0000) 2022-09-10T06:31:58.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:58.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:58.357433+0000) 2022-09-10T06:31:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:31:58.712+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:31:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:58.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:58.760786+0000) 2022-09-10T06:31:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:31:59 smithi110 ceph-mon[97633]: pgmap v3931: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:31:59 smithi104 ceph-mon[97044]: pgmap v3931: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:31:59.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:31:59.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:59.357691+0000) 2022-09-10T06:32:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:31:59.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:31:59.760979+0000) 2022-09-10T06:32:00.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:00.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:00.357915+0000) 2022-09-10T06:32:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:00.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:00.761153+0000) 2022-09-10T06:32:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:01 smithi110 ceph-mon[97633]: pgmap v3932: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:01 smithi104 ceph-mon[97044]: pgmap v3932: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:01.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:01.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:01.358181+0000) 2022-09-10T06:32:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:01.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:01.761375+0000) 2022-09-10T06:32:02.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:02.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:02.358407+0000) 2022-09-10T06:32:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:02.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:02.761598+0000) 2022-09-10T06:32:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:03 smithi104 ceph-mon[97044]: pgmap v3933: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:03 smithi110 ceph-mon[97633]: pgmap v3933: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:03.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:03.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:03.358661+0000) 2022-09-10T06:32:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:03.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:03.761894+0000) 2022-09-10T06:32:04.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:04.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:04.358909+0000) 2022-09-10T06:32:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:04.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:04.762188+0000) 2022-09-10T06:32:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:04 smithi110 ceph-mon[97633]: pgmap v3934: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:04 smithi104 ceph-mon[97044]: pgmap v3934: 129 pgs: 129 active+clean; 8.1 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:05.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:05.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:05.359150+0000) 2022-09-10T06:32:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:05.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:05.762419+0000) 2022-09-10T06:32:06.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:06.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:06.359394+0000) 2022-09-10T06:32:07.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:06.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:06.762576+0000) 2022-09-10T06:32:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:07 smithi110 ceph-mon[97633]: pgmap v3935: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:32:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:07 smithi104 ceph-mon[97044]: pgmap v3935: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:32:07.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:07.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:07.359637+0000) 2022-09-10T06:32:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:07.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:07.762855+0000) 2022-09-10T06:32:08.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:08.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:08.359907+0000) 2022-09-10T06:32:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:08.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:08.763162+0000) 2022-09-10T06:32:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:09 smithi104 ceph-mon[97044]: pgmap v3936: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:32:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:09 smithi104 ceph-mon[97044]: osdmap e328: 12 total, 12 up, 12 in 2022-09-10T06:32:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:09 smithi110 ceph-mon[97633]: pgmap v3936: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:32:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:09 smithi110 ceph-mon[97633]: osdmap e328: 12 total, 12 up, 12 in 2022-09-10T06:32:09.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:09.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:09.360164+0000) 2022-09-10T06:32:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:09.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:09.763415+0000) 2022-09-10T06:32:10.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:10.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:10.360412+0000) 2022-09-10T06:32:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:10.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:10.763650+0000) 2022-09-10T06:32:11.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:11 smithi104 ceph-mon[97044]: pgmap v3938: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:32:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:11 smithi110 ceph-mon[97633]: pgmap v3938: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:32:11.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:11.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:11.360614+0000) 2022-09-10T06:32:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:11.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:11.763938+0000) 2022-09-10T06:32:12.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:12.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:12.360798+0000) 2022-09-10T06:32:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:12.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:12.764156+0000) 2022-09-10T06:32:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:13 smithi104 ceph-mon[97044]: pgmap v3939: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:32:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:13 smithi110 ceph-mon[97633]: pgmap v3939: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:32:13.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:13.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:13.361010+0000) 2022-09-10T06:32:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:13.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:13.764286+0000) 2022-09-10T06:32:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:32:13.721+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:32:14.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:14.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:14.361210+0000) 2022-09-10T06:32:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:14.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:14.764534+0000) 2022-09-10T06:32:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:14 smithi110 ceph-mon[97633]: osdmap e329: 12 total, 12 up, 12 in 2022-09-10T06:32:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:14 smithi110 ceph-mon[97633]: pgmap v3941: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:32:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:14 smithi104 ceph-mon[97044]: osdmap e329: 12 total, 12 up, 12 in 2022-09-10T06:32:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:14 smithi104 ceph-mon[97044]: pgmap v3941: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:32:15.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:15.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:15.361443+0000) 2022-09-10T06:32:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:15.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:15.764799+0000) 2022-09-10T06:32:16.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:16.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:16.361660+0000) 2022-09-10T06:32:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:16.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:16.765001+0000) 2022-09-10T06:32:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:17 smithi110 ceph-mon[97633]: pgmap v3942: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:32:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:17 smithi104 ceph-mon[97044]: pgmap v3942: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:32:17.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:17.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:17.361872+0000) 2022-09-10T06:32:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:17.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:17.765222+0000) 2022-09-10T06:32:18.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:18.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:18.362162+0000) 2022-09-10T06:32:19.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:18.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:18.765420+0000) 2022-09-10T06:32:19.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:19 smithi104 ceph-mon[97044]: pgmap v3943: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:32:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:19 smithi110 ceph-mon[97633]: pgmap v3943: 129 pgs: 129 active+clean; 8.1 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T06:32:19.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:19.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:19.362364+0000) 2022-09-10T06:32:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:19.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:19.765663+0000) 2022-09-10T06:32:20.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:20.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:20.362562+0000) 2022-09-10T06:32:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:20.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:20.765878+0000) 2022-09-10T06:32:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:21 smithi104 ceph-mon[97044]: pgmap v3944: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:32:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:21 smithi110 ceph-mon[97633]: pgmap v3944: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:32:21.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:21.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:21.362748+0000) 2022-09-10T06:32:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:21.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:21.766090+0000) 2022-09-10T06:32:22.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:22.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:22.362958+0000) 2022-09-10T06:32:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:22.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:22.766353+0000) 2022-09-10T06:32:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:23 smithi104 ceph-mon[97044]: pgmap v3945: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:32:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:23 smithi110 ceph-mon[97633]: pgmap v3945: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:32:23.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:23.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:23.363212+0000) 2022-09-10T06:32:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:23.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:23.766598+0000) 2022-09-10T06:32:24.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:24.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:24.363426+0000) 2022-09-10T06:32:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:24.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:24.766874+0000) 2022-09-10T06:32:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:24 smithi110 ceph-mon[97633]: pgmap v3946: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:32:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:24 smithi104 ceph-mon[97044]: pgmap v3946: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:32:25.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:25.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:25.363674+0000) 2022-09-10T06:32:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:25.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:25.767143+0000) 2022-09-10T06:32:26.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:26.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:26.363897+0000) 2022-09-10T06:32:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:26.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:26.767368+0000) 2022-09-10T06:32:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:27 smithi104 ceph-mon[97044]: pgmap v3947: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:32:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:27 smithi110 ceph-mon[97633]: pgmap v3947: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:32:27.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:27.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:27.364186+0000) 2022-09-10T06:32:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:27.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:27.767645+0000) 2022-09-10T06:32:28.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:28.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:28.364344+0000) 2022-09-10T06:32:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:32:28.736+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:32:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:28.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:28.767864+0000) 2022-09-10T06:32:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:29 smithi104 ceph-mon[97044]: pgmap v3948: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:29 smithi110 ceph-mon[97633]: pgmap v3948: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:29.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:29.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:29.364577+0000) 2022-09-10T06:32:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:29.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:29.768006+0000) 2022-09-10T06:32:30.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:30.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:30.364851+0000) 2022-09-10T06:32:31.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:30.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:30.768246+0000) 2022-09-10T06:32:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:31 smithi104 ceph-mon[97044]: pgmap v3949: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:31 smithi110 ceph-mon[97633]: pgmap v3949: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:31.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:31.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:31.365194+0000) 2022-09-10T06:32:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:31.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:31.768514+0000) 2022-09-10T06:32:32.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:32.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:32.365410+0000) 2022-09-10T06:32:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:32.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:32.768777+0000) 2022-09-10T06:32:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:33 smithi104 ceph-mon[97044]: pgmap v3950: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:33 smithi110 ceph-mon[97633]: pgmap v3950: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:33.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:33.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:33.365713+0000) 2022-09-10T06:32:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:33.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:33.769019+0000) 2022-09-10T06:32:34.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:34.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:34.365890+0000) 2022-09-10T06:32:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:34.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:34.769231+0000) 2022-09-10T06:32:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:34 smithi110 ceph-mon[97633]: pgmap v3951: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:34 smithi104 ceph-mon[97044]: pgmap v3951: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:35.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:35.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:35.366074+0000) 2022-09-10T06:32:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:35.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:35.769534+0000) 2022-09-10T06:32:36.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:36.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:36.366277+0000) 2022-09-10T06:32:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:36.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:36.769779+0000) 2022-09-10T06:32:37.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:37 smithi104 ceph-mon[97044]: pgmap v3952: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:37.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:37 smithi110 ceph-mon[97633]: pgmap v3952: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:37.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:37.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:37.366462+0000) 2022-09-10T06:32:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:37.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:37.770008+0000) 2022-09-10T06:32:38.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:38.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:38.366693+0000) 2022-09-10T06:32:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:38.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:38.770232+0000) 2022-09-10T06:32:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:39 smithi104 ceph-mon[97044]: pgmap v3953: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:39 smithi110 ceph-mon[97633]: pgmap v3953: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:39.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:39.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:39.366942+0000) 2022-09-10T06:32:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:39.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:39.770454+0000) 2022-09-10T06:32:40.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:40.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:40.367195+0000) 2022-09-10T06:32:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:40.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:40.770655+0000) 2022-09-10T06:32:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:41 smithi104 ceph-mon[97044]: pgmap v3954: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:41 smithi110 ceph-mon[97633]: pgmap v3954: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:41.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:41.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:41.367366+0000) 2022-09-10T06:32:42.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:41.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:41.770876+0000) 2022-09-10T06:32:42.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:42.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:42.367548+0000) 2022-09-10T06:32:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:42.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:42.771149+0000) 2022-09-10T06:32:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:43 smithi104 ceph-mon[97044]: pgmap v3955: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:43 smithi110 ceph-mon[97633]: pgmap v3955: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:43.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:43.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:43.367779+0000) 2022-09-10T06:32:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:43.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:43.771398+0000) 2022-09-10T06:32:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:32:43.751+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:32:44.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:44.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:44.367958+0000) 2022-09-10T06:32:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:44.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:44.771613+0000) 2022-09-10T06:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:32:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:32:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:32:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:32:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:44 smithi110 ceph-mon[97633]: pgmap v3956: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:32:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:32:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:32:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:44 smithi104 ceph-mon[97044]: pgmap v3956: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:45.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:45.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:45.368156+0000) 2022-09-10T06:32:46.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:45.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:45.771831+0000) 2022-09-10T06:32:46.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:46.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:46.368380+0000) 2022-09-10T06:32:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:46.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:46.772069+0000) 2022-09-10T06:32:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:47 smithi104 ceph-mon[97044]: pgmap v3957: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:47 smithi110 ceph-mon[97633]: pgmap v3957: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:47.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:47.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:47.368613+0000) 2022-09-10T06:32:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:47.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:47.772382+0000) 2022-09-10T06:32:48.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:48.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:48.368840+0000) 2022-09-10T06:32:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:48.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:48.772590+0000) 2022-09-10T06:32:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:49 smithi104 ceph-mon[97044]: pgmap v3958: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:49 smithi110 ceph-mon[97633]: pgmap v3958: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:49.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:49.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:49.369144+0000) 2022-09-10T06:32:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:49.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:49.772758+0000) 2022-09-10T06:32:50.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:50.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:50.369325+0000) 2022-09-10T06:32:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:50.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:50.772984+0000) 2022-09-10T06:32:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:51 smithi104 ceph-mon[97044]: pgmap v3959: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:51 smithi110 ceph-mon[97633]: pgmap v3959: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:51.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:51.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:51.369544+0000) 2022-09-10T06:32:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:51.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:51.773265+0000) 2022-09-10T06:32:52.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:52.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:52.369779+0000) 2022-09-10T06:32:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:52.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:52.773441+0000) 2022-09-10T06:32:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:53 smithi104 ceph-mon[97044]: pgmap v3960: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:53 smithi110 ceph-mon[97633]: pgmap v3960: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:53.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:53.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:53.370036+0000) 2022-09-10T06:32:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:53.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:53.773653+0000) 2022-09-10T06:32:54.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:54.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:54.370288+0000) 2022-09-10T06:32:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:54.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:54.773932+0000) 2022-09-10T06:32:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:54 smithi104 ceph-mon[97044]: pgmap v3961: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:54 smithi110 ceph-mon[97633]: pgmap v3961: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:55.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:55.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:55.370516+0000) 2022-09-10T06:32:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:55.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:55.774186+0000) 2022-09-10T06:32:56.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:56.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:56.370718+0000) 2022-09-10T06:32:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:56.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:56.774428+0000) 2022-09-10T06:32:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:57 smithi104 ceph-mon[97044]: pgmap v3962: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:57 smithi110 ceph-mon[97633]: pgmap v3962: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:32:57.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:57.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:57.370966+0000) 2022-09-10T06:32:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:57.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:57.774582+0000) 2022-09-10T06:32:58.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:58.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:58.371231+0000) 2022-09-10T06:32:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:58.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:58.774818+0000) 2022-09-10T06:32:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:32:58.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:32:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:32:59 smithi104 ceph-mon[97044]: pgmap v3963: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:32:59 smithi110 ceph-mon[97633]: pgmap v3963: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:32:59.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:32:59.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:59.371449+0000) 2022-09-10T06:33:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:32:59.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:32:59.775013+0000) 2022-09-10T06:33:00.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:00.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:00.371672+0000) 2022-09-10T06:33:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:00.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:00.775234+0000) 2022-09-10T06:33:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:01 smithi104 ceph-mon[97044]: pgmap v3964: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:01 smithi110 ceph-mon[97633]: pgmap v3964: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:01.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:01.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:01.371918+0000) 2022-09-10T06:33:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:01.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:01.775503+0000) 2022-09-10T06:33:02.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:02.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:02.372175+0000) 2022-09-10T06:33:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:02.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:02.775746+0000) 2022-09-10T06:33:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:03 smithi104 ceph-mon[97044]: pgmap v3965: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:03 smithi110 ceph-mon[97633]: pgmap v3965: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:03.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:03.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:03.372368+0000) 2022-09-10T06:33:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:03.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:03.776027+0000) 2022-09-10T06:33:04.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:04.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:04.372518+0000) 2022-09-10T06:33:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:04.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:04.776333+0000) 2022-09-10T06:33:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:04 smithi104 ceph-mon[97044]: pgmap v3966: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:04 smithi110 ceph-mon[97633]: pgmap v3966: 129 pgs: 129 active+clean; 8.1 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:05.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:05.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:05.372735+0000) 2022-09-10T06:33:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:05.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:05.776577+0000) 2022-09-10T06:33:06.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:06.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:06.372974+0000) 2022-09-10T06:33:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:06.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:06.776793+0000) 2022-09-10T06:33:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:07 smithi104 ceph-mon[97044]: pgmap v3967: 129 pgs: 129 active+clean; 8.1 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:33:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:07 smithi110 ceph-mon[97633]: pgmap v3967: 129 pgs: 129 active+clean; 8.1 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:33:07.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:07.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:07.373224+0000) 2022-09-10T06:33:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:07.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:07.777061+0000) 2022-09-10T06:33:08.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:08.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:08.373409+0000) 2022-09-10T06:33:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:08.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:08.777338+0000) 2022-09-10T06:33:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:09 smithi104 ceph-mon[97044]: pgmap v3968: 129 pgs: 129 active+clean; 8.1 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:33:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:09 smithi104 ceph-mon[97044]: osdmap e330: 12 total, 12 up, 12 in 2022-09-10T06:33:09.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:09 smithi110 ceph-mon[97633]: pgmap v3968: 129 pgs: 129 active+clean; 8.1 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:33:09.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:09 smithi110 ceph-mon[97633]: osdmap e330: 12 total, 12 up, 12 in 2022-09-10T06:33:09.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:09.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:09.373551+0000) 2022-09-10T06:33:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:09.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:09.777512+0000) 2022-09-10T06:33:10.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:10.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:10.373790+0000) 2022-09-10T06:33:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:10.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:10.777708+0000) 2022-09-10T06:33:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:11 smithi104 ceph-mon[97044]: pgmap v3970: 129 pgs: 129 active+clean; 8.1 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:33:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:11 smithi110 ceph-mon[97633]: pgmap v3970: 129 pgs: 129 active+clean; 8.1 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:33:11.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:11.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:11.373996+0000) 2022-09-10T06:33:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:11.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:11.777933+0000) 2022-09-10T06:33:12.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:12.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:12.374136+0000) 2022-09-10T06:33:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:12.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:12.778132+0000) 2022-09-10T06:33:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:13 smithi104 ceph-mon[97044]: pgmap v3971: 129 pgs: 129 active+clean; 8.1 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:33:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:13 smithi110 ceph-mon[97633]: pgmap v3971: 129 pgs: 129 active+clean; 8.1 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:33:13.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:13.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:13.374351+0000) 2022-09-10T06:33:14.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:33:13.769+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:33:14.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:13.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:13.778323+0000) 2022-09-10T06:33:14.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:14.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:14.374586+0000) 2022-09-10T06:33:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:14.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:14.778540+0000) 2022-09-10T06:33:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:14 smithi104 ceph-mon[97044]: osdmap e331: 12 total, 12 up, 12 in 2022-09-10T06:33:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:14 smithi104 ceph-mon[97044]: pgmap v3973: 129 pgs: 129 active+clean; 8.1 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:33:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:14 smithi110 ceph-mon[97633]: osdmap e331: 12 total, 12 up, 12 in 2022-09-10T06:33:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:14 smithi110 ceph-mon[97633]: pgmap v3973: 129 pgs: 129 active+clean; 8.1 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:33:15.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:15.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:15.374808+0000) 2022-09-10T06:33:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:15.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:15.778780+0000) 2022-09-10T06:33:16.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:16.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:16.374983+0000) 2022-09-10T06:33:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:16.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:16.779016+0000) 2022-09-10T06:33:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:17 smithi104 ceph-mon[97044]: pgmap v3974: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:33:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:17 smithi110 ceph-mon[97633]: pgmap v3974: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:33:17.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:17.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:17.375199+0000) 2022-09-10T06:33:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:17.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:17.779205+0000) 2022-09-10T06:33:18.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:18.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:18.375504+0000) 2022-09-10T06:33:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:18.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:18.779431+0000) 2022-09-10T06:33:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:19 smithi104 ceph-mon[97044]: pgmap v3975: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:33:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:19 smithi110 ceph-mon[97633]: pgmap v3975: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:33:19.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:19.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:19.375765+0000) 2022-09-10T06:33:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:19.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:19.779607+0000) 2022-09-10T06:33:20.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:20.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:20.376033+0000) 2022-09-10T06:33:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:20.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:20.779857+0000) 2022-09-10T06:33:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:21 smithi104 ceph-mon[97044]: pgmap v3976: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:33:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:21 smithi110 ceph-mon[97633]: pgmap v3976: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:33:21.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:21.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:21.376260+0000) 2022-09-10T06:33:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:21.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:21.780087+0000) 2022-09-10T06:33:22.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:22.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:22.376490+0000) 2022-09-10T06:33:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:22.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:22.780281+0000) 2022-09-10T06:33:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:23 smithi104 ceph-mon[97044]: pgmap v3977: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:33:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:23 smithi110 ceph-mon[97633]: pgmap v3977: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:33:23.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:23.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:23.376735+0000) 2022-09-10T06:33:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:23.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:23.780503+0000) 2022-09-10T06:33:24.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:24.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:24.376920+0000) 2022-09-10T06:33:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:24.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:24.780731+0000) 2022-09-10T06:33:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:24 smithi104 ceph-mon[97044]: pgmap v3978: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:33:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:24 smithi110 ceph-mon[97633]: pgmap v3978: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:33:25.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:25.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:25.377131+0000) 2022-09-10T06:33:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:25.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:25.780959+0000) 2022-09-10T06:33:26.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:26.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:26.377368+0000) 2022-09-10T06:33:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:26.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:26.781159+0000) 2022-09-10T06:33:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:27 smithi104 ceph-mon[97044]: pgmap v3979: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:33:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:27 smithi110 ceph-mon[97633]: pgmap v3979: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:33:27.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:27.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:27.377593+0000) 2022-09-10T06:33:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:27.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:27.781375+0000) 2022-09-10T06:33:28.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:28.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:28.377815+0000) 2022-09-10T06:33:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:33:28.772+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:33:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:28.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:28.781573+0000) 2022-09-10T06:33:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:29 smithi104 ceph-mon[97044]: pgmap v3980: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:29 smithi110 ceph-mon[97633]: pgmap v3980: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:29.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:29.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:29.378037+0000) 2022-09-10T06:33:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:29.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:29.781802+0000) 2022-09-10T06:33:30.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:30.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:30.378259+0000) 2022-09-10T06:33:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:30.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:30.782044+0000) 2022-09-10T06:33:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:31 smithi104 ceph-mon[97044]: pgmap v3981: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:31 smithi110 ceph-mon[97633]: pgmap v3981: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:31.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:31.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:31.378472+0000) 2022-09-10T06:33:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:31.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:31.782266+0000) 2022-09-10T06:33:32.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:32.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:32.378742+0000) 2022-09-10T06:33:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:32.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:32.782493+0000) 2022-09-10T06:33:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:33 smithi104 ceph-mon[97044]: pgmap v3982: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:33 smithi110 ceph-mon[97633]: pgmap v3982: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:33.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:33.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:33.379013+0000) 2022-09-10T06:33:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:33.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:33.782755+0000) 2022-09-10T06:33:34.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:34.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:34.379302+0000) 2022-09-10T06:33:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:34.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:34.782960+0000) 2022-09-10T06:33:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:34 smithi104 ceph-mon[97044]: pgmap v3983: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:34 smithi110 ceph-mon[97633]: pgmap v3983: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:35.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:35.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:35.379522+0000) 2022-09-10T06:33:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:35.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:35.783162+0000) 2022-09-10T06:33:36.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:36.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:36.379705+0000) 2022-09-10T06:33:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:36.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:36.783379+0000) 2022-09-10T06:33:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:37 smithi104 ceph-mon[97044]: pgmap v3984: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:37 smithi110 ceph-mon[97633]: pgmap v3984: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:37.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:37.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:37.379918+0000) 2022-09-10T06:33:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:37.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:37.783611+0000) 2022-09-10T06:33:38.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:38.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:38.380189+0000) 2022-09-10T06:33:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:38.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:38.783811+0000) 2022-09-10T06:33:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:39 smithi104 ceph-mon[97044]: pgmap v3985: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:39 smithi110 ceph-mon[97633]: pgmap v3985: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:39.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:39.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:39.380461+0000) 2022-09-10T06:33:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:39.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:39.784070+0000) 2022-09-10T06:33:40.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:40.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:40.380663+0000) 2022-09-10T06:33:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:40.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:40.784290+0000) 2022-09-10T06:33:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:41 smithi104 ceph-mon[97044]: pgmap v3986: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:41 smithi110 ceph-mon[97633]: pgmap v3986: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:41.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:41.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:41.380841+0000) 2022-09-10T06:33:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:41.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:41.784497+0000) 2022-09-10T06:33:42.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:42.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:42.381086+0000) 2022-09-10T06:33:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:42.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:42.784671+0000) 2022-09-10T06:33:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:43 smithi104 ceph-mon[97044]: pgmap v3987: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:43 smithi110 ceph-mon[97633]: pgmap v3987: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:43.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:43.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:43.381365+0000) 2022-09-10T06:33:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:43.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:43.784886+0000) 2022-09-10T06:33:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:33:43.788+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:33:44.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:44.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:44.381623+0000) 2022-09-10T06:33:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:44.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:44.785172+0000) 2022-09-10T06:33:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:33:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:33:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:33:45.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:33:45.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:44 smithi104 ceph-mon[97044]: pgmap v3988: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:33:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:33:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:33:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:33:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:44 smithi110 ceph-mon[97633]: pgmap v3988: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:45.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:45.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:45.381853+0000) 2022-09-10T06:33:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:45.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:45.785382+0000) 2022-09-10T06:33:46.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:46.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:46.382154+0000) 2022-09-10T06:33:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:46.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:46.785514+0000) 2022-09-10T06:33:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:47 smithi104 ceph-mon[97044]: pgmap v3989: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:47 smithi110 ceph-mon[97633]: pgmap v3989: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:47.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:47.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:47.382405+0000) 2022-09-10T06:33:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:47.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:47.785723+0000) 2022-09-10T06:33:48.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:48.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:48.382675+0000) 2022-09-10T06:33:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:48.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:48.785958+0000) 2022-09-10T06:33:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:49 smithi104 ceph-mon[97044]: pgmap v3990: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:49 smithi110 ceph-mon[97633]: pgmap v3990: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:49.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:49.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:49.382931+0000) 2022-09-10T06:33:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:49.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:49.786158+0000) 2022-09-10T06:33:50.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:50.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:50.383235+0000) 2022-09-10T06:33:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:50.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:50.786347+0000) 2022-09-10T06:33:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:51 smithi104 ceph-mon[97044]: pgmap v3991: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:51 smithi110 ceph-mon[97633]: pgmap v3991: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:51.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:51.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:51.383502+0000) 2022-09-10T06:33:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:51.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:51.786490+0000) 2022-09-10T06:33:52.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:52.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:52.383759+0000) 2022-09-10T06:33:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:52.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:52.786737+0000) 2022-09-10T06:33:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:53 smithi104 ceph-mon[97044]: pgmap v3992: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:53 smithi110 ceph-mon[97633]: pgmap v3992: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:53.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:53.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:53.384014+0000) 2022-09-10T06:33:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:53.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:53.786990+0000) 2022-09-10T06:33:54.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:54.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:54.384280+0000) 2022-09-10T06:33:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:54.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:54.787242+0000) 2022-09-10T06:33:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:54 smithi104 ceph-mon[97044]: pgmap v3993: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:54 smithi110 ceph-mon[97633]: pgmap v3993: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:55.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:55.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:55.384480+0000) 2022-09-10T06:33:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:55.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:55.787454+0000) 2022-09-10T06:33:56.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:56.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:56.384636+0000) 2022-09-10T06:33:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:56.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:56.787696+0000) 2022-09-10T06:33:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:57 smithi104 ceph-mon[97044]: pgmap v3994: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:57 smithi110 ceph-mon[97633]: pgmap v3994: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:33:57.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:57.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:57.384903+0000) 2022-09-10T06:33:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:57.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:57.787977+0000) 2022-09-10T06:33:58.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:58.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:58.385224+0000) 2022-09-10T06:33:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:58.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:58.788185+0000) 2022-09-10T06:33:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:33:58.804+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:33:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:33:59 smithi104 ceph-mon[97044]: pgmap v3995: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:33:59 smithi110 ceph-mon[97633]: pgmap v3995: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:33:59.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:33:59.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:59.385498+0000) 2022-09-10T06:34:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:33:59.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:33:59.788388+0000) 2022-09-10T06:34:00.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:00.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:00.385683+0000) 2022-09-10T06:34:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:00.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:00.788527+0000) 2022-09-10T06:34:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:01 smithi104 ceph-mon[97044]: pgmap v3996: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:01 smithi110 ceph-mon[97633]: pgmap v3996: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:01.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:01.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:01.385869+0000) 2022-09-10T06:34:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:01.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:01.788744+0000) 2022-09-10T06:34:02.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:02.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:02.386131+0000) 2022-09-10T06:34:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:02.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:02.788980+0000) 2022-09-10T06:34:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:03 smithi104 ceph-mon[97044]: pgmap v3997: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:03 smithi110 ceph-mon[97633]: pgmap v3997: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:03.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:03.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:03.386398+0000) 2022-09-10T06:34:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:03.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:03.789202+0000) 2022-09-10T06:34:04.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:04.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:04.386677+0000) 2022-09-10T06:34:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:04.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:04.789471+0000) 2022-09-10T06:34:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:04 smithi104 ceph-mon[97044]: pgmap v3998: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:04 smithi110 ceph-mon[97633]: pgmap v3998: 129 pgs: 129 active+clean; 8.1 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:05.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:05.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:05.386954+0000) 2022-09-10T06:34:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:05.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:05.789751+0000) 2022-09-10T06:34:06.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:06.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:06.387255+0000) 2022-09-10T06:34:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:06.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:06.789946+0000) 2022-09-10T06:34:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:07 smithi104 ceph-mon[97044]: pgmap v3999: 129 pgs: 129 active+clean; 8.1 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:34:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:07 smithi110 ceph-mon[97633]: pgmap v3999: 129 pgs: 129 active+clean; 8.1 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:34:07.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:07.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:07.387467+0000) 2022-09-10T06:34:08.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:07.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:07.790213+0000) 2022-09-10T06:34:08.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:08.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:08.387719+0000) 2022-09-10T06:34:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:08.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:08.790419+0000) 2022-09-10T06:34:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:09 smithi104 ceph-mon[97044]: pgmap v4000: 129 pgs: 129 active+clean; 8.1 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:34:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:09 smithi104 ceph-mon[97044]: osdmap e332: 12 total, 12 up, 12 in 2022-09-10T06:34:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:09 smithi110 ceph-mon[97633]: pgmap v4000: 129 pgs: 129 active+clean; 8.1 MiB data, 549 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:34:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:09 smithi110 ceph-mon[97633]: osdmap e332: 12 total, 12 up, 12 in 2022-09-10T06:34:09.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:09.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:09.387912+0000) 2022-09-10T06:34:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:09.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:09.790657+0000) 2022-09-10T06:34:10.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:10.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:10.388079+0000) 2022-09-10T06:34:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:10.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:10.790890+0000) 2022-09-10T06:34:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:11 smithi104 ceph-mon[97044]: pgmap v4002: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:34:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:11 smithi110 ceph-mon[97633]: pgmap v4002: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:34:11.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:11.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:11.388311+0000) 2022-09-10T06:34:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:11.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:11.791155+0000) 2022-09-10T06:34:12.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:12.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:12.388468+0000) 2022-09-10T06:34:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:12.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:12.791367+0000) 2022-09-10T06:34:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:13 smithi104 ceph-mon[97044]: pgmap v4003: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:34:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:13 smithi110 ceph-mon[97633]: pgmap v4003: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:34:13.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:13.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:13.388763+0000) 2022-09-10T06:34:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:13.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:13.791630+0000) 2022-09-10T06:34:14.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:34:13.819+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:34:14.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:14.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:14.388966+0000) 2022-09-10T06:34:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:14.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:14.791846+0000) 2022-09-10T06:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:14 smithi104 ceph-mon[97044]: osdmap e333: 12 total, 12 up, 12 in 2022-09-10T06:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:14 smithi104 ceph-mon[97044]: pgmap v4005: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:34:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:14 smithi110 ceph-mon[97633]: osdmap e333: 12 total, 12 up, 12 in 2022-09-10T06:34:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:14 smithi110 ceph-mon[97633]: pgmap v4005: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:34:15.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:15.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:15.389163+0000) 2022-09-10T06:34:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:15.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:15.792081+0000) 2022-09-10T06:34:16.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:16.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:16.389400+0000) 2022-09-10T06:34:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:16.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:16.792309+0000) 2022-09-10T06:34:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:17 smithi104 ceph-mon[97044]: pgmap v4006: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:34:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:17 smithi110 ceph-mon[97633]: pgmap v4006: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:34:17.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:17.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:17.389613+0000) 2022-09-10T06:34:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:17.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:17.792554+0000) 2022-09-10T06:34:18.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:18.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:18.389879+0000) 2022-09-10T06:34:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:18.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:18.792767+0000) 2022-09-10T06:34:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:19 smithi104 ceph-mon[97044]: pgmap v4007: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:34:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:19 smithi110 ceph-mon[97633]: pgmap v4007: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T06:34:19.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:19.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:19.390139+0000) 2022-09-10T06:34:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:19.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:19.792990+0000) 2022-09-10T06:34:20.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:20.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:20.390379+0000) 2022-09-10T06:34:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:20.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:20.793168+0000) 2022-09-10T06:34:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:21 smithi104 ceph-mon[97044]: pgmap v4008: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:34:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:21 smithi110 ceph-mon[97633]: pgmap v4008: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:34:21.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:21.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:21.390555+0000) 2022-09-10T06:34:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:21.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:21.793376+0000) 2022-09-10T06:34:22.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:22.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:22.390771+0000) 2022-09-10T06:34:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:22.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:22.793612+0000) 2022-09-10T06:34:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:23 smithi104 ceph-mon[97044]: pgmap v4009: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:34:23.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:23 smithi110 ceph-mon[97633]: pgmap v4009: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:34:23.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:23.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:23.391019+0000) 2022-09-10T06:34:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:23.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:23.793866+0000) 2022-09-10T06:34:24.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:24.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:24.391260+0000) 2022-09-10T06:34:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:24.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:24.794078+0000) 2022-09-10T06:34:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:24 smithi104 ceph-mon[97044]: pgmap v4010: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:34:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:24 smithi110 ceph-mon[97633]: pgmap v4010: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:34:25.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:25.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:25.391439+0000) 2022-09-10T06:34:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:25.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:25.794354+0000) 2022-09-10T06:34:26.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:26.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:26.391632+0000) 2022-09-10T06:34:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:26.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:26.794592+0000) 2022-09-10T06:34:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:27 smithi104 ceph-mon[97044]: pgmap v4011: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:34:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:27 smithi110 ceph-mon[97633]: pgmap v4011: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:34:27.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:27.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:27.391874+0000) 2022-09-10T06:34:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:27.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:27.794824+0000) 2022-09-10T06:34:28.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:28.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:28.392189+0000) 2022-09-10T06:34:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:28.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:28.795014+0000) 2022-09-10T06:34:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:34:28.826+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:34:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:29 smithi104 ceph-mon[97044]: pgmap v4012: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:29 smithi110 ceph-mon[97633]: pgmap v4012: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:29.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:29.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:29.392438+0000) 2022-09-10T06:34:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:29.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:29.795218+0000) 2022-09-10T06:34:30.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:30.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:30.392623+0000) 2022-09-10T06:34:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:30.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:30.795408+0000) 2022-09-10T06:34:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:31 smithi104 ceph-mon[97044]: pgmap v4013: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:31 smithi110 ceph-mon[97633]: pgmap v4013: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:31.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:31.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:31.392868+0000) 2022-09-10T06:34:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:31.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:31.795569+0000) 2022-09-10T06:34:32.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:32.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:32.393146+0000) 2022-09-10T06:34:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:32.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:32.795746+0000) 2022-09-10T06:34:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:33 smithi104 ceph-mon[97044]: pgmap v4014: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:34:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:33 smithi110 ceph-mon[97633]: pgmap v4014: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:34:33.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:33.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:33.393326+0000) 2022-09-10T06:34:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:33.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:33.795970+0000) 2022-09-10T06:34:34.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:34.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:34.393553+0000) 2022-09-10T06:34:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:34.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:34.796276+0000) 2022-09-10T06:34:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:34 smithi104 ceph-mon[97044]: pgmap v4015: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:34:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:34 smithi110 ceph-mon[97633]: pgmap v4015: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:34:35.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:35.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:35.393776+0000) 2022-09-10T06:34:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:35.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:35.796567+0000) 2022-09-10T06:34:36.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:36.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:36.394027+0000) 2022-09-10T06:34:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:36.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:36.796845+0000) 2022-09-10T06:34:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:37 smithi104 ceph-mon[97044]: pgmap v4016: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:37 smithi110 ceph-mon[97633]: pgmap v4016: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:37.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:37.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:37.394246+0000) 2022-09-10T06:34:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:37.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:37.797145+0000) 2022-09-10T06:34:38.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:38.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:38.394502+0000) 2022-09-10T06:34:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:38.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:38.797396+0000) 2022-09-10T06:34:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:39 smithi104 ceph-mon[97044]: pgmap v4017: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:34:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:39 smithi110 ceph-mon[97633]: pgmap v4017: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:34:39.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:39.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:39.394762+0000) 2022-09-10T06:34:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:39.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:39.797666+0000) 2022-09-10T06:34:40.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:40.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:40.395047+0000) 2022-09-10T06:34:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:40.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:40.797908+0000) 2022-09-10T06:34:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:41 smithi104 ceph-mon[97044]: pgmap v4018: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:41 smithi110 ceph-mon[97633]: pgmap v4018: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:41.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:41.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:41.395302+0000) 2022-09-10T06:34:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:41.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:41.798152+0000) 2022-09-10T06:34:42.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:42.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:42.395524+0000) 2022-09-10T06:34:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:42.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:42.798369+0000) 2022-09-10T06:34:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:43 smithi104 ceph-mon[97044]: pgmap v4019: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:43 smithi110 ceph-mon[97633]: pgmap v4019: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:43.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:43.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:43.395779+0000) 2022-09-10T06:34:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:43.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:43.798607+0000) 2022-09-10T06:34:44.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:34:43.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:34:44.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:44.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:44.396063+0000) 2022-09-10T06:34:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:44.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:44.798778+0000) 2022-09-10T06:34:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:34:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:34:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:34:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:34:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:44 smithi104 ceph-mon[97044]: pgmap v4020: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:34:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:34:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:34:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:34:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:44 smithi110 ceph-mon[97633]: pgmap v4020: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:45.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:45.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:45.396264+0000) 2022-09-10T06:34:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:45.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:45.799021+0000) 2022-09-10T06:34:46.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:46.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:46.396526+0000) 2022-09-10T06:34:47.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:46.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:46.799223+0000) 2022-09-10T06:34:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:47 smithi104 ceph-mon[97044]: pgmap v4021: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:47 smithi110 ceph-mon[97633]: pgmap v4021: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:47.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:47.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:47.396812+0000) 2022-09-10T06:34:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:47.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:47.799371+0000) 2022-09-10T06:34:48.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:48.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:48.396981+0000) 2022-09-10T06:34:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:48.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:48.799588+0000) 2022-09-10T06:34:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:49 smithi104 ceph-mon[97044]: pgmap v4022: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:49 smithi110 ceph-mon[97633]: pgmap v4022: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:49.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:49.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:49.397166+0000) 2022-09-10T06:34:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:49.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:49.799840+0000) 2022-09-10T06:34:50.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:50.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:50.397406+0000) 2022-09-10T06:34:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:50.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:50.800062+0000) 2022-09-10T06:34:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:51 smithi104 ceph-mon[97044]: pgmap v4023: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:51 smithi110 ceph-mon[97633]: pgmap v4023: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:51.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:51.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:51.397622+0000) 2022-09-10T06:34:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:51.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:51.800203+0000) 2022-09-10T06:34:52.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:52.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:52.397788+0000) 2022-09-10T06:34:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:52.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:52.800421+0000) 2022-09-10T06:34:53.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:53 smithi104 ceph-mon[97044]: pgmap v4024: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:53 smithi110 ceph-mon[97633]: pgmap v4024: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:53.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:53.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:53.397992+0000) 2022-09-10T06:34:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:53.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:53.800649+0000) 2022-09-10T06:34:54.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:54.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:54.398192+0000) 2022-09-10T06:34:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:54.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:54.800876+0000) 2022-09-10T06:34:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:54 smithi104 ceph-mon[97044]: pgmap v4025: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:54 smithi110 ceph-mon[97633]: pgmap v4025: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:55.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:55.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:55.398369+0000) 2022-09-10T06:34:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:55.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:55.801134+0000) 2022-09-10T06:34:56.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:56.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:56.398542+0000) 2022-09-10T06:34:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:56.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:56.801311+0000) 2022-09-10T06:34:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:57 smithi104 ceph-mon[97044]: pgmap v4026: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:57 smithi110 ceph-mon[97633]: pgmap v4026: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:34:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:57.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:57.398782+0000) 2022-09-10T06:34:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:57.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:57.801583+0000) 2022-09-10T06:34:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:58.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:58.399057+0000) 2022-09-10T06:34:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:58.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:58.801814+0000) 2022-09-10T06:34:59.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:34:58.857+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:34:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:34:59 smithi104 ceph-mon[97044]: pgmap v4027: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:34:59 smithi110 ceph-mon[97633]: pgmap v4027: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:34:59.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:34:59.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:59.399342+0000) 2022-09-10T06:35:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:34:59.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:34:59.802031+0000) 2022-09-10T06:35:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:00.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:00.399607+0000) 2022-09-10T06:35:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:00.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:00.802165+0000) 2022-09-10T06:35:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:01 smithi104 ceph-mon[97044]: pgmap v4028: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:01 smithi110 ceph-mon[97633]: pgmap v4028: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:01.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:01.399775+0000) 2022-09-10T06:35:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:01.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:01.802379+0000) 2022-09-10T06:35:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:02.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:02.400030+0000) 2022-09-10T06:35:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:02.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:02.802636+0000) 2022-09-10T06:35:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:03 smithi104 ceph-mon[97044]: pgmap v4029: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:03 smithi110 ceph-mon[97633]: pgmap v4029: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:03.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:03.400278+0000) 2022-09-10T06:35:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:03.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:03.802902+0000) 2022-09-10T06:35:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:04.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:04.400445+0000) 2022-09-10T06:35:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:04.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:04.803180+0000) 2022-09-10T06:35:05.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:04 smithi104 ceph-mon[97044]: pgmap v4030: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:04 smithi110 ceph-mon[97633]: pgmap v4030: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:05.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:05.400583+0000) 2022-09-10T06:35:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:05.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:05.803427+0000) 2022-09-10T06:35:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:06.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:06.400767+0000) 2022-09-10T06:35:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:06.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:06.803693+0000) 2022-09-10T06:35:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:07 smithi104 ceph-mon[97044]: pgmap v4031: 129 pgs: 129 active+clean; 8.1 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:35:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:07 smithi110 ceph-mon[97633]: pgmap v4031: 129 pgs: 129 active+clean; 8.1 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:35:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:07.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:07.400984+0000) 2022-09-10T06:35:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:07.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:07.803987+0000) 2022-09-10T06:35:08.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:08 smithi104 ceph-mon[97044]: osdmap e334: 12 total, 12 up, 12 in 2022-09-10T06:35:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:08 smithi110 ceph-mon[97633]: osdmap e334: 12 total, 12 up, 12 in 2022-09-10T06:35:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:08.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:08.401194+0000) 2022-09-10T06:35:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:08.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:08.804149+0000) 2022-09-10T06:35:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:09 smithi104 ceph-mon[97044]: pgmap v4033: 129 pgs: 129 active+clean; 8.1 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:35:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:09 smithi110 ceph-mon[97633]: pgmap v4033: 129 pgs: 129 active+clean; 8.1 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:35:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:09.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:09.401451+0000) 2022-09-10T06:35:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:09.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:09.804286+0000) 2022-09-10T06:35:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:10.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:10.401649+0000) 2022-09-10T06:35:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:10.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:10.804448+0000) 2022-09-10T06:35:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:11 smithi104 ceph-mon[97044]: pgmap v4034: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:35:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:11 smithi110 ceph-mon[97633]: pgmap v4034: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:35:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:11.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:11.401826+0000) 2022-09-10T06:35:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:11.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:11.804597+0000) 2022-09-10T06:35:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:12.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:12.402003+0000) 2022-09-10T06:35:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:12.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:12.804768+0000) 2022-09-10T06:35:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:13 smithi104 ceph-mon[97044]: pgmap v4035: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:35:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:13 smithi110 ceph-mon[97633]: pgmap v4035: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:35:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:13.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:13.402171+0000) 2022-09-10T06:35:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:13.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:13.804963+0000) 2022-09-10T06:35:14.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:35:13.868+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:35:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:14.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:14.402345+0000) 2022-09-10T06:35:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:14 smithi104 ceph-mon[97044]: osdmap e335: 12 total, 12 up, 12 in 2022-09-10T06:35:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:14 smithi104 ceph-mon[97044]: pgmap v4037: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:35:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:14.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:14.805184+0000) 2022-09-10T06:35:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:14 smithi110 ceph-mon[97633]: osdmap e335: 12 total, 12 up, 12 in 2022-09-10T06:35:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:14 smithi110 ceph-mon[97633]: pgmap v4037: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:35:15.816 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:15.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:15.402473+0000) 2022-09-10T06:35:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:15.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:15.805379+0000) 2022-09-10T06:35:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:16.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:16.402665+0000) 2022-09-10T06:35:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:16.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:16.805562+0000) 2022-09-10T06:35:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:17 smithi104 ceph-mon[97044]: pgmap v4038: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:35:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:17 smithi110 ceph-mon[97633]: pgmap v4038: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:35:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:17.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:17.402877+0000) 2022-09-10T06:35:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:17.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:17.805757+0000) 2022-09-10T06:35:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:18.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:18.403159+0000) 2022-09-10T06:35:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:18.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:18.805942+0000) 2022-09-10T06:35:19.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:19 smithi104 ceph-mon[97044]: pgmap v4039: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:35:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:19 smithi110 ceph-mon[97633]: pgmap v4039: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:35:19.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:19.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:19.403388+0000) 2022-09-10T06:35:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:19.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:19.806160+0000) 2022-09-10T06:35:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:20.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:20.403518+0000) 2022-09-10T06:35:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:20.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:20.806321+0000) 2022-09-10T06:35:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:21 smithi104 ceph-mon[97044]: pgmap v4040: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:35:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:21 smithi110 ceph-mon[97633]: pgmap v4040: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:35:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:21.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:21.403752+0000) 2022-09-10T06:35:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:21.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:21.806490+0000) 2022-09-10T06:35:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:22.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:22.403995+0000) 2022-09-10T06:35:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:22.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:22.806675+0000) 2022-09-10T06:35:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:23 smithi104 ceph-mon[97044]: pgmap v4041: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:35:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:23 smithi110 ceph-mon[97633]: pgmap v4041: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:35:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:23.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:23.404187+0000) 2022-09-10T06:35:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:23.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:23.806902+0000) 2022-09-10T06:35:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:24.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:24.404423+0000) 2022-09-10T06:35:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:24 smithi104 ceph-mon[97044]: pgmap v4042: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:35:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:24.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:24.807137+0000) 2022-09-10T06:35:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:24 smithi110 ceph-mon[97633]: pgmap v4042: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:35:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:25.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:25.404602+0000) 2022-09-10T06:35:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:25.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:25.807310+0000) 2022-09-10T06:35:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:26.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:26.404800+0000) 2022-09-10T06:35:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:26.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:26.807480+0000) 2022-09-10T06:35:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:27 smithi104 ceph-mon[97044]: pgmap v4043: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:35:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:27 smithi110 ceph-mon[97633]: pgmap v4043: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:35:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:27.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:27.405018+0000) 2022-09-10T06:35:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:27.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:27.807664+0000) 2022-09-10T06:35:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:28.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:28.405257+0000) 2022-09-10T06:35:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:28.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:28.807850+0000) 2022-09-10T06:35:29.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:35:28.871+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:35:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:29 smithi104 ceph-mon[97044]: pgmap v4044: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:29 smithi110 ceph-mon[97633]: pgmap v4044: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:29.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:29.405465+0000) 2022-09-10T06:35:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:29.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:29.808044+0000) 2022-09-10T06:35:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:30.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:30.405646+0000) 2022-09-10T06:35:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:30.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:30.808170+0000) 2022-09-10T06:35:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:31 smithi104 ceph-mon[97044]: pgmap v4045: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:31 smithi110 ceph-mon[97633]: pgmap v4045: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:31.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:31.405785+0000) 2022-09-10T06:35:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:31.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:31.808355+0000) 2022-09-10T06:35:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:32.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:32.406005+0000) 2022-09-10T06:35:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:32.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:32.808527+0000) 2022-09-10T06:35:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:33 smithi104 ceph-mon[97044]: pgmap v4046: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:35:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:33 smithi110 ceph-mon[97633]: pgmap v4046: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:35:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:33.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:33.406200+0000) 2022-09-10T06:35:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:33.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:33.808658+0000) 2022-09-10T06:35:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:34.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:34.406380+0000) 2022-09-10T06:35:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:34 smithi104 ceph-mon[97044]: pgmap v4047: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:35:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:34.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:34.808842+0000) 2022-09-10T06:35:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:34 smithi110 ceph-mon[97633]: pgmap v4047: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:35:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:35.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:35.406592+0000) 2022-09-10T06:35:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:35.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:35.809058+0000) 2022-09-10T06:35:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:36.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:36.406851+0000) 2022-09-10T06:35:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:36.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:36.809256+0000) 2022-09-10T06:35:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:37 smithi104 ceph-mon[97044]: pgmap v4048: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:37.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:37 smithi110 ceph-mon[97633]: pgmap v4048: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:37.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:37.407149+0000) 2022-09-10T06:35:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:37.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:37.809453+0000) 2022-09-10T06:35:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:38.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:38.407368+0000) 2022-09-10T06:35:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:38.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:38.809656+0000) 2022-09-10T06:35:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:39 smithi104 ceph-mon[97044]: pgmap v4049: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:35:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:39 smithi110 ceph-mon[97633]: pgmap v4049: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:35:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:39.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:39.407584+0000) 2022-09-10T06:35:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:39.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:39.809865+0000) 2022-09-10T06:35:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:40.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:40.407798+0000) 2022-09-10T06:35:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:40.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:40.810044+0000) 2022-09-10T06:35:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:41 smithi104 ceph-mon[97044]: pgmap v4050: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:41 smithi110 ceph-mon[97633]: pgmap v4050: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:41.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:41.408070+0000) 2022-09-10T06:35:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:41.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:41.810218+0000) 2022-09-10T06:35:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:42.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:42.408287+0000) 2022-09-10T06:35:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:42.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:42.810340+0000) 2022-09-10T06:35:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:43 smithi104 ceph-mon[97044]: pgmap v4051: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:43 smithi110 ceph-mon[97633]: pgmap v4051: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:43.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:43.408428+0000) 2022-09-10T06:35:44.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:35:43.886+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:35:44.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:43.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:43.810537+0000) 2022-09-10T06:35:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:35:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:35:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:35:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:35:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:35:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:35:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:35:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:35:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:44.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:44.408624+0000) 2022-09-10T06:35:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:44.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:44.810782+0000) 2022-09-10T06:35:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:45 smithi104 ceph-mon[97044]: pgmap v4052: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:45.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:45 smithi110 ceph-mon[97633]: pgmap v4052: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:45.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:45.408852+0000) 2022-09-10T06:35:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:45.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:45.811018+0000) 2022-09-10T06:35:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:46.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:46.409080+0000) 2022-09-10T06:35:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:46.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:46.811206+0000) 2022-09-10T06:35:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:47 smithi104 ceph-mon[97044]: pgmap v4053: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:47 smithi110 ceph-mon[97633]: pgmap v4053: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:47.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:47.409365+0000) 2022-09-10T06:35:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:47.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:47.811438+0000) 2022-09-10T06:35:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:48.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:48.409530+0000) 2022-09-10T06:35:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:48.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:48.811637+0000) 2022-09-10T06:35:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:49 smithi104 ceph-mon[97044]: pgmap v4054: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:49 smithi110 ceph-mon[97633]: pgmap v4054: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:49.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:49.409769+0000) 2022-09-10T06:35:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:49.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:49.811853+0000) 2022-09-10T06:35:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:50.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:50.410023+0000) 2022-09-10T06:35:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:50.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:50.812013+0000) 2022-09-10T06:35:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:51 smithi104 ceph-mon[97044]: pgmap v4055: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:51 smithi110 ceph-mon[97633]: pgmap v4055: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:51.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:51.410267+0000) 2022-09-10T06:35:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:51.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:51.812207+0000) 2022-09-10T06:35:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:52.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:52.410428+0000) 2022-09-10T06:35:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:52.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:52.812395+0000) 2022-09-10T06:35:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:53 smithi104 ceph-mon[97044]: pgmap v4056: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:53 smithi110 ceph-mon[97633]: pgmap v4056: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:53.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:53.410626+0000) 2022-09-10T06:35:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:53.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:53.812657+0000) 2022-09-10T06:35:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:54.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:54.410863+0000) 2022-09-10T06:35:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:54 smithi104 ceph-mon[97044]: pgmap v4057: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:54.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:54.812879+0000) 2022-09-10T06:35:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:54 smithi110 ceph-mon[97633]: pgmap v4057: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:55.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:55.411050+0000) 2022-09-10T06:35:56.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:55.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:55.813077+0000) 2022-09-10T06:35:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:56.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:56.411235+0000) 2022-09-10T06:35:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:56.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:56.813342+0000) 2022-09-10T06:35:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:57 smithi104 ceph-mon[97044]: pgmap v4058: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:57 smithi110 ceph-mon[97633]: pgmap v4058: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:35:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:57.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:57.411468+0000) 2022-09-10T06:35:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:57.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:57.813608+0000) 2022-09-10T06:35:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:58.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:58.411665+0000) 2022-09-10T06:35:59.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:35:58.900+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:35:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:58.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:58.813813+0000) 2022-09-10T06:35:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:35:59 smithi104 ceph-mon[97044]: pgmap v4059: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:35:59 smithi110 ceph-mon[97633]: pgmap v4059: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:35:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:35:59.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:59.411897+0000) 2022-09-10T06:36:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:35:59.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:35:59.814134+0000) 2022-09-10T06:36:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:00.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:00.412140+0000) 2022-09-10T06:36:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:00.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:00.814286+0000) 2022-09-10T06:36:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:01 smithi104 ceph-mon[97044]: pgmap v4060: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:01 smithi110 ceph-mon[97633]: pgmap v4060: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:01.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:01.412324+0000) 2022-09-10T06:36:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:01.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:01.814498+0000) 2022-09-10T06:36:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:02.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:02.412503+0000) 2022-09-10T06:36:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:02.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:02.814740+0000) 2022-09-10T06:36:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:03 smithi104 ceph-mon[97044]: pgmap v4061: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:03 smithi110 ceph-mon[97633]: pgmap v4061: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:03.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:03.412749+0000) 2022-09-10T06:36:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:03.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:03.814966+0000) 2022-09-10T06:36:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:04.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:04.412948+0000) 2022-09-10T06:36:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:04 smithi104 ceph-mon[97044]: pgmap v4062: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:04.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:04.815248+0000) 2022-09-10T06:36:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:04 smithi110 ceph-mon[97633]: pgmap v4062: 129 pgs: 129 active+clean; 8.1 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:05.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:05.413141+0000) 2022-09-10T06:36:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:05.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:05.815432+0000) 2022-09-10T06:36:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:06.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:06.413271+0000) 2022-09-10T06:36:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:06.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:06.815629+0000) 2022-09-10T06:36:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:07 smithi104 ceph-mon[97044]: pgmap v4063: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:36:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:07 smithi110 ceph-mon[97633]: pgmap v4063: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:36:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:07.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:07.413475+0000) 2022-09-10T06:36:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:07.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:07.815880+0000) 2022-09-10T06:36:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:08 smithi104 ceph-mon[97044]: osdmap e336: 12 total, 12 up, 12 in 2022-09-10T06:36:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:08 smithi110 ceph-mon[97633]: osdmap e336: 12 total, 12 up, 12 in 2022-09-10T06:36:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:08.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:08.413725+0000) 2022-09-10T06:36:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:08.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:08.816157+0000) 2022-09-10T06:36:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:09 smithi104 ceph-mon[97044]: pgmap v4065: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:36:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:09 smithi110 ceph-mon[97633]: pgmap v4065: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:36:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:09.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:09.413975+0000) 2022-09-10T06:36:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:09.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:09.816304+0000) 2022-09-10T06:36:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:10.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:10.414172+0000) 2022-09-10T06:36:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:10.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:10.816551+0000) 2022-09-10T06:36:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:11 smithi104 ceph-mon[97044]: pgmap v4066: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:36:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:11 smithi110 ceph-mon[97633]: pgmap v4066: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:36:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:11.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:11.414404+0000) 2022-09-10T06:36:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:11.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:11.816813+0000) 2022-09-10T06:36:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:12.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:12.414671+0000) 2022-09-10T06:36:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:12.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:12.817008+0000) 2022-09-10T06:36:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:13 smithi104 ceph-mon[97044]: pgmap v4067: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:36:13.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:13 smithi110 ceph-mon[97633]: pgmap v4067: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:36:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:13.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:13.414918+0000) 2022-09-10T06:36:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:13.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:13.817209+0000) 2022-09-10T06:36:14.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:36:13.913+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:36:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:14.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:14.415194+0000) 2022-09-10T06:36:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:14 smithi104 ceph-mon[97044]: osdmap e337: 12 total, 12 up, 12 in 2022-09-10T06:36:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:14 smithi104 ceph-mon[97044]: pgmap v4069: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:36:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:14.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:14.817487+0000) 2022-09-10T06:36:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:14 smithi110 ceph-mon[97633]: osdmap e337: 12 total, 12 up, 12 in 2022-09-10T06:36:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:14 smithi110 ceph-mon[97633]: pgmap v4069: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:36:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:15.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:15.415431+0000) 2022-09-10T06:36:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:15.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:15.817649+0000) 2022-09-10T06:36:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:16.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:16.415606+0000) 2022-09-10T06:36:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:16.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:16.817889+0000) 2022-09-10T06:36:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:17 smithi104 ceph-mon[97044]: pgmap v4070: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:36:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:17 smithi110 ceph-mon[97633]: pgmap v4070: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:36:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:17.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:17.415842+0000) 2022-09-10T06:36:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:17.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:17.818132+0000) 2022-09-10T06:36:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:18.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:18.416080+0000) 2022-09-10T06:36:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:18.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:18.818356+0000) 2022-09-10T06:36:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:19 smithi104 ceph-mon[97044]: pgmap v4071: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:36:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:19 smithi110 ceph-mon[97633]: pgmap v4071: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:36:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:19.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:19.416330+0000) 2022-09-10T06:36:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:19.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:19.818609+0000) 2022-09-10T06:36:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:20.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:20.416536+0000) 2022-09-10T06:36:21.108 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:20.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:20.818760+0000) 2022-09-10T06:36:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:21 smithi104 ceph-mon[97044]: pgmap v4072: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:36:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:21 smithi110 ceph-mon[97633]: pgmap v4072: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:36:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:21.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:21.416793+0000) 2022-09-10T06:36:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:21.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:21.818965+0000) 2022-09-10T06:36:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:22.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:22.417020+0000) 2022-09-10T06:36:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:22.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:22.819211+0000) 2022-09-10T06:36:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:23 smithi104 ceph-mon[97044]: pgmap v4073: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:36:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:23 smithi110 ceph-mon[97633]: pgmap v4073: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:36:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:23.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:23.417232+0000) 2022-09-10T06:36:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:23.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:23.819470+0000) 2022-09-10T06:36:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:24.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:24.417452+0000) 2022-09-10T06:36:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:24 smithi104 ceph-mon[97044]: pgmap v4074: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:36:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:24.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:24.819785+0000) 2022-09-10T06:36:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:24 smithi110 ceph-mon[97633]: pgmap v4074: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:36:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:25.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:25.417580+0000) 2022-09-10T06:36:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:25.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:25.820002+0000) 2022-09-10T06:36:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:26.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:26.417789+0000) 2022-09-10T06:36:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:26.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:26.820189+0000) 2022-09-10T06:36:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:27 smithi104 ceph-mon[97044]: pgmap v4075: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:36:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:27 smithi110 ceph-mon[97633]: pgmap v4075: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:36:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:27.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:27.418020+0000) 2022-09-10T06:36:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:27.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:27.820402+0000) 2022-09-10T06:36:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:28.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:28.418228+0000) 2022-09-10T06:36:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:28.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:28.820661+0000) 2022-09-10T06:36:29.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:36:28.930+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:36:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:29 smithi104 ceph-mon[97044]: pgmap v4076: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:29 smithi110 ceph-mon[97633]: pgmap v4076: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:29.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:29.418512+0000) 2022-09-10T06:36:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:29.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:29.820895+0000) 2022-09-10T06:36:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:30.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:30.418638+0000) 2022-09-10T06:36:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:30.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:30.821150+0000) 2022-09-10T06:36:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:31 smithi104 ceph-mon[97044]: pgmap v4077: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:31 smithi110 ceph-mon[97633]: pgmap v4077: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:31.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:31.418862+0000) 2022-09-10T06:36:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:31.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:31.821311+0000) 2022-09-10T06:36:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:32.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:32.419017+0000) 2022-09-10T06:36:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:32.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:32.821506+0000) 2022-09-10T06:36:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:33 smithi104 ceph-mon[97044]: pgmap v4078: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:33 smithi110 ceph-mon[97633]: pgmap v4078: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:33.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:33.419238+0000) 2022-09-10T06:36:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:33.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:33.821752+0000) 2022-09-10T06:36:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:34.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:34.419480+0000) 2022-09-10T06:36:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:34 smithi104 ceph-mon[97044]: pgmap v4079: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:35.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:34.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:34.822013+0000) 2022-09-10T06:36:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:34 smithi110 ceph-mon[97633]: pgmap v4079: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:35.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:35.419745+0000) 2022-09-10T06:36:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:35.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:35.822212+0000) 2022-09-10T06:36:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:36.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:36.419982+0000) 2022-09-10T06:36:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:36.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:36.822405+0000) 2022-09-10T06:36:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:37 smithi104 ceph-mon[97044]: pgmap v4080: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:37 smithi110 ceph-mon[97633]: pgmap v4080: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:37.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:37.420199+0000) 2022-09-10T06:36:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:37.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:37.822594+0000) 2022-09-10T06:36:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:38.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:38.420382+0000) 2022-09-10T06:36:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:38.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:38.822847+0000) 2022-09-10T06:36:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:39 smithi104 ceph-mon[97044]: pgmap v4081: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:39 smithi110 ceph-mon[97633]: pgmap v4081: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:39.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:39.420597+0000) 2022-09-10T06:36:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:39.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:39.823157+0000) 2022-09-10T06:36:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:40.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:40.420832+0000) 2022-09-10T06:36:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:40.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:40.823321+0000) 2022-09-10T06:36:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:41 smithi104 ceph-mon[97044]: pgmap v4082: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:41 smithi110 ceph-mon[97633]: pgmap v4082: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:41.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:41.421038+0000) 2022-09-10T06:36:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:41.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:41.823496+0000) 2022-09-10T06:36:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:42.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:42.421275+0000) 2022-09-10T06:36:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:42.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:42.823650+0000) 2022-09-10T06:36:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:43 smithi104 ceph-mon[97044]: pgmap v4083: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:43 smithi110 ceph-mon[97633]: pgmap v4083: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:43.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:43.421504+0000) 2022-09-10T06:36:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:43.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:43.823902+0000) 2022-09-10T06:36:44.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:36:43.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:36:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:36:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:36:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:36:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:36:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:36:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:36:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:36:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:36:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:44.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:44.421678+0000) 2022-09-10T06:36:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:44.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:44.824187+0000) 2022-09-10T06:36:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:45 smithi104 ceph-mon[97044]: pgmap v4084: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:45 smithi110 ceph-mon[97633]: pgmap v4084: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:45.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:45.421913+0000) 2022-09-10T06:36:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:45.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:45.824376+0000) 2022-09-10T06:36:46.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:46.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:46.422078+0000) 2022-09-10T06:36:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:46.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:46.824546+0000) 2022-09-10T06:36:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:47 smithi104 ceph-mon[97044]: pgmap v4085: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:47 smithi110 ceph-mon[97633]: pgmap v4085: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:47.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:47.422357+0000) 2022-09-10T06:36:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:47.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:47.824791+0000) 2022-09-10T06:36:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:48.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:48.422495+0000) 2022-09-10T06:36:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:48.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:48.824995+0000) 2022-09-10T06:36:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:49 smithi104 ceph-mon[97044]: pgmap v4086: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:49 smithi110 ceph-mon[97633]: pgmap v4086: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:49.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:49.422638+0000) 2022-09-10T06:36:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:49.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:49.825164+0000) 2022-09-10T06:36:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:50.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:50.422843+0000) 2022-09-10T06:36:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:50.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:50.825357+0000) 2022-09-10T06:36:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:51 smithi104 ceph-mon[97044]: pgmap v4087: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:51 smithi110 ceph-mon[97633]: pgmap v4087: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:51.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:51.423039+0000) 2022-09-10T06:36:52.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:51.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:51.825544+0000) 2022-09-10T06:36:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:52.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:52.423239+0000) 2022-09-10T06:36:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:52.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:52.825690+0000) 2022-09-10T06:36:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:53 smithi104 ceph-mon[97044]: pgmap v4088: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:53 smithi110 ceph-mon[97633]: pgmap v4088: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:53.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:53.423499+0000) 2022-09-10T06:36:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:53.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:53.825930+0000) 2022-09-10T06:36:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:54.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:54.423746+0000) 2022-09-10T06:36:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:54 smithi104 ceph-mon[97044]: pgmap v4089: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:54.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:54.826184+0000) 2022-09-10T06:36:55.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:54 smithi110 ceph-mon[97633]: pgmap v4089: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:55.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:55.423998+0000) 2022-09-10T06:36:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:55.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:55.826387+0000) 2022-09-10T06:36:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:56.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:56.424150+0000) 2022-09-10T06:36:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:56.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:56.826582+0000) 2022-09-10T06:36:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:57 smithi104 ceph-mon[97044]: pgmap v4090: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:57 smithi110 ceph-mon[97633]: pgmap v4090: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:36:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:57.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:57.424388+0000) 2022-09-10T06:36:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:57.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:57.826783+0000) 2022-09-10T06:36:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:58.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:58.424564+0000) 2022-09-10T06:36:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:58.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:58.826992+0000) 2022-09-10T06:36:59.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:36:58.954+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:36:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:36:59 smithi104 ceph-mon[97044]: pgmap v4091: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:36:59 smithi110 ceph-mon[97633]: pgmap v4091: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:36:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:36:59.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:59.424757+0000) 2022-09-10T06:37:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:36:59.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:36:59.827266+0000) 2022-09-10T06:37:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:00.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:00.424973+0000) 2022-09-10T06:37:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:00.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:00.827475+0000) 2022-09-10T06:37:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:01 smithi104 ceph-mon[97044]: pgmap v4092: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:01 smithi110 ceph-mon[97633]: pgmap v4092: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:01.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:01.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:01.425188+0000) 2022-09-10T06:37:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:01.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:01.827645+0000) 2022-09-10T06:37:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:02.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:02.425375+0000) 2022-09-10T06:37:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:02.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:02.827863+0000) 2022-09-10T06:37:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:03 smithi104 ceph-mon[97044]: pgmap v4093: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:03 smithi110 ceph-mon[97633]: pgmap v4093: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:03.813 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:03.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:03.425655+0000) 2022-09-10T06:37:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:03.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:03.828126+0000) 2022-09-10T06:37:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:04.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:04.425838+0000) 2022-09-10T06:37:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:04 smithi104 ceph-mon[97044]: pgmap v4094: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:04.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:04.828319+0000) 2022-09-10T06:37:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:04 smithi110 ceph-mon[97633]: pgmap v4094: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:05.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:05.425962+0000) 2022-09-10T06:37:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:05.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:05.828550+0000) 2022-09-10T06:37:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:06.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:06.426203+0000) 2022-09-10T06:37:07.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:06.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:06.828773+0000) 2022-09-10T06:37:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:07 smithi104 ceph-mon[97044]: pgmap v4095: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:37:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:07 smithi110 ceph-mon[97633]: pgmap v4095: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:37:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:07.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:07.426458+0000) 2022-09-10T06:37:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:07.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:07.828971+0000) 2022-09-10T06:37:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:08 smithi104 ceph-mon[97044]: osdmap e338: 12 total, 12 up, 12 in 2022-09-10T06:37:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:08 smithi110 ceph-mon[97633]: osdmap e338: 12 total, 12 up, 12 in 2022-09-10T06:37:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:08.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:08.426635+0000) 2022-09-10T06:37:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:08.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:08.829199+0000) 2022-09-10T06:37:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:09 smithi104 ceph-mon[97044]: pgmap v4097: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:37:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:09 smithi110 ceph-mon[97633]: pgmap v4097: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:37:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:09.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:09.426877+0000) 2022-09-10T06:37:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:09.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:09.829348+0000) 2022-09-10T06:37:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:10.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:10.427170+0000) 2022-09-10T06:37:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:10.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:10.829540+0000) 2022-09-10T06:37:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:11 smithi104 ceph-mon[97044]: pgmap v4098: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:37:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:11 smithi110 ceph-mon[97633]: pgmap v4098: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:37:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:11.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:11.427403+0000) 2022-09-10T06:37:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:11.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:11.829776+0000) 2022-09-10T06:37:12.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:12.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:12.427567+0000) 2022-09-10T06:37:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:12.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:12.830018+0000) 2022-09-10T06:37:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:13 smithi104 ceph-mon[97044]: pgmap v4099: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:37:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:13 smithi110 ceph-mon[97633]: pgmap v4099: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:37:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:13.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:13.427779+0000) 2022-09-10T06:37:14.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:37:13.969+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:37:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:13.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:13.830186+0000) 2022-09-10T06:37:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:14.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:14.428038+0000) 2022-09-10T06:37:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:14 smithi104 ceph-mon[97044]: osdmap e339: 12 total, 12 up, 12 in 2022-09-10T06:37:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:14 smithi104 ceph-mon[97044]: pgmap v4101: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:37:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:14.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:14.830409+0000) 2022-09-10T06:37:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:14 smithi110 ceph-mon[97633]: osdmap e339: 12 total, 12 up, 12 in 2022-09-10T06:37:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:14 smithi110 ceph-mon[97633]: pgmap v4101: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:37:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:15.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:15.428173+0000) 2022-09-10T06:37:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:15.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:15.830635+0000) 2022-09-10T06:37:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:16.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:16.428365+0000) 2022-09-10T06:37:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:16.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:16.830890+0000) 2022-09-10T06:37:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:17 smithi104 ceph-mon[97044]: pgmap v4102: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:37:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:17 smithi110 ceph-mon[97633]: pgmap v4102: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:37:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:17.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:17.428615+0000) 2022-09-10T06:37:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:17.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:17.831154+0000) 2022-09-10T06:37:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:18.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:18.428902+0000) 2022-09-10T06:37:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:18.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:18.831293+0000) 2022-09-10T06:37:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:19 smithi104 ceph-mon[97044]: pgmap v4103: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:37:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:19 smithi110 ceph-mon[97633]: pgmap v4103: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:37:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:19.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:19.429175+0000) 2022-09-10T06:37:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:19.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:19.831526+0000) 2022-09-10T06:37:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:20.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:20.429326+0000) 2022-09-10T06:37:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:20.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:20.831776+0000) 2022-09-10T06:37:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:21 smithi104 ceph-mon[97044]: pgmap v4104: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:37:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:21 smithi110 ceph-mon[97633]: pgmap v4104: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:37:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:21.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:21.429485+0000) 2022-09-10T06:37:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:21.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:21.832000+0000) 2022-09-10T06:37:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:22.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:22.429736+0000) 2022-09-10T06:37:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:22.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:22.832252+0000) 2022-09-10T06:37:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:23 smithi104 ceph-mon[97044]: pgmap v4105: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:37:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:23 smithi110 ceph-mon[97633]: pgmap v4105: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:37:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:23.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:23.429956+0000) 2022-09-10T06:37:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:23.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:23.832381+0000) 2022-09-10T06:37:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:24.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:24.430207+0000) 2022-09-10T06:37:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:24 smithi104 ceph-mon[97044]: pgmap v4106: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:37:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:24.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:24.832598+0000) 2022-09-10T06:37:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:24 smithi110 ceph-mon[97633]: pgmap v4106: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:37:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:25.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:25.430360+0000) 2022-09-10T06:37:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:25.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:25.832868+0000) 2022-09-10T06:37:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:26.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:26.430539+0000) 2022-09-10T06:37:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:26.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:26.833085+0000) 2022-09-10T06:37:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:27 smithi104 ceph-mon[97044]: pgmap v4107: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:37:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:27 smithi110 ceph-mon[97633]: pgmap v4107: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:37:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:27.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:27.430707+0000) 2022-09-10T06:37:28.180 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:27.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:27.833333+0000) 2022-09-10T06:37:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:28.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:28.430846+0000) 2022-09-10T06:37:29.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:37:28.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:37:29.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:28.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:28.833604+0000) 2022-09-10T06:37:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:29 smithi104 ceph-mon[97044]: pgmap v4108: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:29 smithi110 ceph-mon[97633]: pgmap v4108: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:29.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:29.430984+0000) 2022-09-10T06:37:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:29.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:29.833795+0000) 2022-09-10T06:37:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:30.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:30.431214+0000) 2022-09-10T06:37:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:30.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:30.833993+0000) 2022-09-10T06:37:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:31 smithi104 ceph-mon[97044]: pgmap v4109: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T06:37:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:31 smithi110 ceph-mon[97633]: pgmap v4109: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T06:37:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:31.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:31.431337+0000) 2022-09-10T06:37:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:31.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:31.834203+0000) 2022-09-10T06:37:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:32.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:32.431529+0000) 2022-09-10T06:37:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:32.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:32.834424+0000) 2022-09-10T06:37:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:33 smithi104 ceph-mon[97044]: pgmap v4110: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:33 smithi110 ceph-mon[97633]: pgmap v4110: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:33.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:33.431684+0000) 2022-09-10T06:37:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:33.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:33.834594+0000) 2022-09-10T06:37:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:34.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:34.431874+0000) 2022-09-10T06:37:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:34 smithi104 ceph-mon[97044]: pgmap v4111: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:34.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:34.834794+0000) 2022-09-10T06:37:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:34 smithi110 ceph-mon[97633]: pgmap v4111: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:35.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:35.432175+0000) 2022-09-10T06:37:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:35.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:35.835006+0000) 2022-09-10T06:37:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:36.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:36.432394+0000) 2022-09-10T06:37:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:36.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:36.835221+0000) 2022-09-10T06:37:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:37 smithi104 ceph-mon[97044]: pgmap v4112: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:37 smithi110 ceph-mon[97633]: pgmap v4112: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:37.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:37.432683+0000) 2022-09-10T06:37:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:37.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:37.835400+0000) 2022-09-10T06:37:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:38.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:38.432962+0000) 2022-09-10T06:37:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:38.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:38.835594+0000) 2022-09-10T06:37:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:39 smithi104 ceph-mon[97044]: pgmap v4113: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:39 smithi110 ceph-mon[97633]: pgmap v4113: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:37:39.815 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:39.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:39.433260+0000) 2022-09-10T06:37:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:39.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:39.835840+0000) 2022-09-10T06:37:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:40.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:40.433499+0000) 2022-09-10T06:37:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:40.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:40.836064+0000) 2022-09-10T06:37:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:41 smithi104 ceph-mon[97044]: pgmap v4114: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:41 smithi110 ceph-mon[97633]: pgmap v4114: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:41.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:41.433773+0000) 2022-09-10T06:37:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:41.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:41.836332+0000) 2022-09-10T06:37:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:42.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:42.434058+0000) 2022-09-10T06:37:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:42.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:42.836573+0000) 2022-09-10T06:37:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:43 smithi104 ceph-mon[97044]: pgmap v4115: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:43 smithi110 ceph-mon[97633]: pgmap v4115: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:43.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:43.434286+0000) 2022-09-10T06:37:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:43.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:43.836787+0000) 2022-09-10T06:37:44.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:37:43.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:37:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:37:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:37:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:37:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:37:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:37:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:37:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:37:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:37:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:44.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:44.434425+0000) 2022-09-10T06:37:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:44.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:44.836975+0000) 2022-09-10T06:37:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:45 smithi104 ceph-mon[97044]: pgmap v4116: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:45 smithi110 ceph-mon[97633]: pgmap v4116: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:45.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:45.434681+0000) 2022-09-10T06:37:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:45.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:45.837247+0000) 2022-09-10T06:37:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:46.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:46.434977+0000) 2022-09-10T06:37:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:46.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:46.837447+0000) 2022-09-10T06:37:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:47 smithi104 ceph-mon[97044]: pgmap v4117: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:47 smithi110 ceph-mon[97633]: pgmap v4117: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:47.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:47.435177+0000) 2022-09-10T06:37:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:47.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:47.837712+0000) 2022-09-10T06:37:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:48.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:48.435388+0000) 2022-09-10T06:37:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:48.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:48.837846+0000) 2022-09-10T06:37:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:49 smithi104 ceph-mon[97044]: pgmap v4118: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:49 smithi110 ceph-mon[97633]: pgmap v4118: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:49.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:49.435603+0000) 2022-09-10T06:37:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:49.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:49.838070+0000) 2022-09-10T06:37:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:50.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:50.435808+0000) 2022-09-10T06:37:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:50.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:50.838250+0000) 2022-09-10T06:37:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:51 smithi104 ceph-mon[97044]: pgmap v4119: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:51 smithi110 ceph-mon[97633]: pgmap v4119: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:51.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:51.435995+0000) 2022-09-10T06:37:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:51.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:51.838488+0000) 2022-09-10T06:37:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:52.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:52.436207+0000) 2022-09-10T06:37:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:52.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:52.838711+0000) 2022-09-10T06:37:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:53 smithi104 ceph-mon[97044]: pgmap v4120: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:53 smithi110 ceph-mon[97633]: pgmap v4120: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:53.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:53.436349+0000) 2022-09-10T06:37:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:53.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:53.838928+0000) 2022-09-10T06:37:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:54.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:54.436499+0000) 2022-09-10T06:37:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:54 smithi104 ceph-mon[97044]: pgmap v4121: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:54.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:54.839154+0000) 2022-09-10T06:37:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:54 smithi110 ceph-mon[97633]: pgmap v4121: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:55.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:55.436662+0000) 2022-09-10T06:37:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:55.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:55.839304+0000) 2022-09-10T06:37:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:56.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:56.436904+0000) 2022-09-10T06:37:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:56.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:56.839514+0000) 2022-09-10T06:37:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:57 smithi104 ceph-mon[97044]: pgmap v4122: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:57 smithi110 ceph-mon[97633]: pgmap v4122: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:37:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:57.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:57.437173+0000) 2022-09-10T06:37:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:57.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:57.839725+0000) 2022-09-10T06:37:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:58.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:58.437349+0000) 2022-09-10T06:37:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:58.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:58.839892+0000) 2022-09-10T06:37:59.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:37:59.012+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:37:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:37:59 smithi104 ceph-mon[97044]: pgmap v4123: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:37:59 smithi110 ceph-mon[97633]: pgmap v4123: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:37:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:37:59.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:59.437531+0000) 2022-09-10T06:38:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:37:59.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:37:59.840151+0000) 2022-09-10T06:38:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:00.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:00.437738+0000) 2022-09-10T06:38:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:00.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:00.840331+0000) 2022-09-10T06:38:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:01 smithi104 ceph-mon[97044]: pgmap v4124: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:01 smithi110 ceph-mon[97633]: pgmap v4124: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:01.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:01.437980+0000) 2022-09-10T06:38:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:01.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:01.840519+0000) 2022-09-10T06:38:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:02.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:02.438215+0000) 2022-09-10T06:38:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:02.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:02.840703+0000) 2022-09-10T06:38:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:03 smithi104 ceph-mon[97044]: pgmap v4125: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:03 smithi110 ceph-mon[97633]: pgmap v4125: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:03.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:03.438412+0000) 2022-09-10T06:38:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:03.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:03.840896+0000) 2022-09-10T06:38:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:04.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:04.438640+0000) 2022-09-10T06:38:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:04 smithi104 ceph-mon[97044]: pgmap v4126: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:04.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:04.841174+0000) 2022-09-10T06:38:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:04 smithi110 ceph-mon[97633]: pgmap v4126: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:05.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:05.438837+0000) 2022-09-10T06:38:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:05.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:05.841361+0000) 2022-09-10T06:38:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:06.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:06.439078+0000) 2022-09-10T06:38:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:06.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:06.841597+0000) 2022-09-10T06:38:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:07 smithi104 ceph-mon[97044]: pgmap v4127: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:38:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:07 smithi110 ceph-mon[97633]: pgmap v4127: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:38:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:07.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:07.439315+0000) 2022-09-10T06:38:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:07.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:07.841853+0000) 2022-09-10T06:38:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:08 smithi104 ceph-mon[97044]: osdmap e340: 12 total, 12 up, 12 in 2022-09-10T06:38:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:08 smithi110 ceph-mon[97633]: osdmap e340: 12 total, 12 up, 12 in 2022-09-10T06:38:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:08.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:08.439467+0000) 2022-09-10T06:38:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:08.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:08.842080+0000) 2022-09-10T06:38:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:09 smithi104 ceph-mon[97044]: pgmap v4129: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:38:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:09 smithi110 ceph-mon[97633]: pgmap v4129: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:38:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:09.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:09.439658+0000) 2022-09-10T06:38:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:09.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:09.842259+0000) 2022-09-10T06:38:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:10.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:10.439945+0000) 2022-09-10T06:38:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:10.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:10.842407+0000) 2022-09-10T06:38:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:11 smithi104 ceph-mon[97044]: pgmap v4130: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:38:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:11 smithi110 ceph-mon[97633]: pgmap v4130: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:38:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:11.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:11.440119+0000) 2022-09-10T06:38:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:11.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:11.842602+0000) 2022-09-10T06:38:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:12.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:12.440345+0000) 2022-09-10T06:38:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:12.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:12.842826+0000) 2022-09-10T06:38:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:13 smithi104 ceph-mon[97044]: pgmap v4131: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:38:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:13 smithi110 ceph-mon[97633]: pgmap v4131: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:38:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:13.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:13.440616+0000) 2022-09-10T06:38:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:13.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:13.843047+0000) 2022-09-10T06:38:14.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:38:14.024+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:38:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:14.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:14.440853+0000) 2022-09-10T06:38:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:14 smithi104 ceph-mon[97044]: osdmap e341: 12 total, 12 up, 12 in 2022-09-10T06:38:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:14 smithi104 ceph-mon[97044]: pgmap v4133: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:38:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:14.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:14.843286+0000) 2022-09-10T06:38:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:14 smithi110 ceph-mon[97633]: osdmap e341: 12 total, 12 up, 12 in 2022-09-10T06:38:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:14 smithi110 ceph-mon[97633]: pgmap v4133: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:38:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:15.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:15.441114+0000) 2022-09-10T06:38:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:15.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:15.843491+0000) 2022-09-10T06:38:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:16.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:16.441340+0000) 2022-09-10T06:38:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:16.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:16.843731+0000) 2022-09-10T06:38:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:17 smithi104 ceph-mon[97044]: pgmap v4134: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:38:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:17 smithi110 ceph-mon[97633]: pgmap v4134: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:38:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:17.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:17.441499+0000) 2022-09-10T06:38:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:17.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:17.843975+0000) 2022-09-10T06:38:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:18.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:18.441756+0000) 2022-09-10T06:38:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:18.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:18.844249+0000) 2022-09-10T06:38:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:19 smithi104 ceph-mon[97044]: pgmap v4135: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:38:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:19 smithi110 ceph-mon[97633]: pgmap v4135: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:38:19.815 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:19.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:19.441982+0000) 2022-09-10T06:38:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:19.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:19.844475+0000) 2022-09-10T06:38:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:20.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:20.442253+0000) 2022-09-10T06:38:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:20.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:20.844620+0000) 2022-09-10T06:38:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:21 smithi104 ceph-mon[97044]: pgmap v4136: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:38:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:21 smithi110 ceph-mon[97633]: pgmap v4136: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:38:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:21.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:21.442427+0000) 2022-09-10T06:38:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:21.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:21.844874+0000) 2022-09-10T06:38:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:22.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:22.442613+0000) 2022-09-10T06:38:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:22.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:22.845075+0000) 2022-09-10T06:38:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:23 smithi104 ceph-mon[97044]: pgmap v4137: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:38:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:23 smithi110 ceph-mon[97633]: pgmap v4137: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:38:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:23.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:23.442809+0000) 2022-09-10T06:38:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:23.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:23.845216+0000) 2022-09-10T06:38:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:24.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:24.443304+0000) 2022-09-10T06:38:25.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:25 smithi104 ceph-mon[97044]: pgmap v4138: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:38:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:24.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:24.845410+0000) 2022-09-10T06:38:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:25 smithi110 ceph-mon[97633]: pgmap v4138: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:38:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:25.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:25.443556+0000) 2022-09-10T06:38:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:25.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:25.845614+0000) 2022-09-10T06:38:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:26.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:26.443755+0000) 2022-09-10T06:38:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:26.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:26.845822+0000) 2022-09-10T06:38:27.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:27 smithi104 ceph-mon[97044]: pgmap v4139: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:38:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:27 smithi110 ceph-mon[97633]: pgmap v4139: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:38:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:27.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:27.443926+0000) 2022-09-10T06:38:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:27.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:27.845978+0000) 2022-09-10T06:38:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:28.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:28.444122+0000) 2022-09-10T06:38:29.109 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:28.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:28.846232+0000) 2022-09-10T06:38:29.109 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:38:29.038+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:38:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:29 smithi104 ceph-mon[97044]: pgmap v4140: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:29 smithi110 ceph-mon[97633]: pgmap v4140: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:29.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:29.444331+0000) 2022-09-10T06:38:30.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:29.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:29.846424+0000) 2022-09-10T06:38:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:30.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:30.444558+0000) 2022-09-10T06:38:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:30.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:30.846661+0000) 2022-09-10T06:38:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:31 smithi104 ceph-mon[97044]: pgmap v4141: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:31 smithi110 ceph-mon[97633]: pgmap v4141: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:31.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:31.444817+0000) 2022-09-10T06:38:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:31.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:31.846877+0000) 2022-09-10T06:38:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:32.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:32.444975+0000) 2022-09-10T06:38:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:32.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:32.847170+0000) 2022-09-10T06:38:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:33 smithi104 ceph-mon[97044]: pgmap v4142: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:33 smithi110 ceph-mon[97633]: pgmap v4142: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:33.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:33.445215+0000) 2022-09-10T06:38:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:33.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:33.847355+0000) 2022-09-10T06:38:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:34.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:34.445425+0000) 2022-09-10T06:38:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:35 smithi104 ceph-mon[97044]: pgmap v4143: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:34.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:34.847511+0000) 2022-09-10T06:38:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:35 smithi110 ceph-mon[97633]: pgmap v4143: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:35.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:35.445685+0000) 2022-09-10T06:38:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:35.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:35.847733+0000) 2022-09-10T06:38:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:36.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:36.445919+0000) 2022-09-10T06:38:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:36.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:36.847989+0000) 2022-09-10T06:38:37.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:37 smithi104 ceph-mon[97044]: pgmap v4144: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:37 smithi110 ceph-mon[97633]: pgmap v4144: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:37.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:37.446177+0000) 2022-09-10T06:38:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:37.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:37.848266+0000) 2022-09-10T06:38:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:38.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:38.446413+0000) 2022-09-10T06:38:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:38.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:38.848438+0000) 2022-09-10T06:38:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:39 smithi104 ceph-mon[97044]: pgmap v4145: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:39 smithi110 ceph-mon[97633]: pgmap v4145: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:39.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:39.446652+0000) 2022-09-10T06:38:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:39.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:39.848653+0000) 2022-09-10T06:38:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:40.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:40.446894+0000) 2022-09-10T06:38:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:40.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:40.848846+0000) 2022-09-10T06:38:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:41 smithi104 ceph-mon[97044]: pgmap v4146: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:41 smithi110 ceph-mon[97633]: pgmap v4146: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:41.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:41.447124+0000) 2022-09-10T06:38:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:41.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:41.848971+0000) 2022-09-10T06:38:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:42.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:42.447296+0000) 2022-09-10T06:38:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:42.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:42.849146+0000) 2022-09-10T06:38:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:43 smithi104 ceph-mon[97044]: pgmap v4147: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:43 smithi110 ceph-mon[97633]: pgmap v4147: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:43.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:43.447490+0000) 2022-09-10T06:38:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:43.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:43.849355+0000) 2022-09-10T06:38:44.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:38:44.054+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:38:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:38:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:38:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:38:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:38:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:38:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:38:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:38:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:38:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:44.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:44.447692+0000) 2022-09-10T06:38:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:44.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:44.849601+0000) 2022-09-10T06:38:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:45 smithi104 ceph-mon[97044]: pgmap v4148: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:45 smithi110 ceph-mon[97633]: pgmap v4148: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:45.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:45.447845+0000) 2022-09-10T06:38:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:45.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:45.849839+0000) 2022-09-10T06:38:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:46.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:46.448006+0000) 2022-09-10T06:38:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:46.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:46.850087+0000) 2022-09-10T06:38:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:47 smithi104 ceph-mon[97044]: pgmap v4149: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:47 smithi110 ceph-mon[97633]: pgmap v4149: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:47.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:47.448222+0000) 2022-09-10T06:38:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:47.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:47.850292+0000) 2022-09-10T06:38:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:48.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:48.448421+0000) 2022-09-10T06:38:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:48.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:48.850463+0000) 2022-09-10T06:38:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:49 smithi110 ceph-mon[97633]: pgmap v4150: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:49 smithi104 ceph-mon[97044]: pgmap v4150: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:49.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:49.448676+0000) 2022-09-10T06:38:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:49.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:49.850646+0000) 2022-09-10T06:38:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:50.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:50.448912+0000) 2022-09-10T06:38:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:50.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:50.850803+0000) 2022-09-10T06:38:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:51 smithi110 ceph-mon[97633]: pgmap v4151: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:51 smithi104 ceph-mon[97044]: pgmap v4151: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:51.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:51.449203+0000) 2022-09-10T06:38:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:51.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:51.851051+0000) 2022-09-10T06:38:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:52.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:52.449370+0000) 2022-09-10T06:38:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:52.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:52.851292+0000) 2022-09-10T06:38:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:53 smithi110 ceph-mon[97633]: pgmap v4152: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:53 smithi104 ceph-mon[97044]: pgmap v4152: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:53.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:53.449520+0000) 2022-09-10T06:38:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:53.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:53.851477+0000) 2022-09-10T06:38:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:54.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:54.449724+0000) 2022-09-10T06:38:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:55 smithi104 ceph-mon[97044]: pgmap v4153: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:54.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:54.851698+0000) 2022-09-10T06:38:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:55 smithi110 ceph-mon[97633]: pgmap v4153: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:55.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:55.449941+0000) 2022-09-10T06:38:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:55.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:55.851940+0000) 2022-09-10T06:38:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:56.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:56.450199+0000) 2022-09-10T06:38:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:56.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:56.852215+0000) 2022-09-10T06:38:57.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:57 smithi104 ceph-mon[97044]: pgmap v4154: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:57 smithi110 ceph-mon[97633]: pgmap v4154: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:38:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:57.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:57.450445+0000) 2022-09-10T06:38:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:57.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:57.852434+0000) 2022-09-10T06:38:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:58.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:58.450708+0000) 2022-09-10T06:38:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:58.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:58.852626+0000) 2022-09-10T06:38:59.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:38:59.067+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:38:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:38:59 smithi104 ceph-mon[97044]: pgmap v4155: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:38:59 smithi110 ceph-mon[97633]: pgmap v4155: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:38:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:38:59.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:59.450914+0000) 2022-09-10T06:39:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:38:59.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:38:59.852878+0000) 2022-09-10T06:39:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:00.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:00.451157+0000) 2022-09-10T06:39:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:00.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:00.853138+0000) 2022-09-10T06:39:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:01 smithi104 ceph-mon[97044]: pgmap v4156: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:01 smithi110 ceph-mon[97633]: pgmap v4156: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:01.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:01.451393+0000) 2022-09-10T06:39:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:01.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:01.853386+0000) 2022-09-10T06:39:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:02.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:02.451547+0000) 2022-09-10T06:39:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:02.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:02.853649+0000) 2022-09-10T06:39:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:03 smithi104 ceph-mon[97044]: pgmap v4157: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:03 smithi110 ceph-mon[97633]: pgmap v4157: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:03.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:03.451788+0000) 2022-09-10T06:39:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:03.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:03.853841+0000) 2022-09-10T06:39:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:04.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:04.452024+0000) 2022-09-10T06:39:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:05 smithi104 ceph-mon[97044]: pgmap v4158: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:04.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:04.854080+0000) 2022-09-10T06:39:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:05 smithi110 ceph-mon[97633]: pgmap v4158: 129 pgs: 129 active+clean; 8.1 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:05.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:05.452225+0000) 2022-09-10T06:39:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:05.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:05.854320+0000) 2022-09-10T06:39:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:06.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:06.452466+0000) 2022-09-10T06:39:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:06.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:06.854549+0000) 2022-09-10T06:39:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:07 smithi110 ceph-mon[97633]: pgmap v4159: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:39:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:07 smithi104 ceph-mon[97044]: pgmap v4159: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:39:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:07.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:07.452711+0000) 2022-09-10T06:39:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:07.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:07.854787+0000) 2022-09-10T06:39:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:08 smithi110 ceph-mon[97633]: osdmap e342: 12 total, 12 up, 12 in 2022-09-10T06:39:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:08 smithi104 ceph-mon[97044]: osdmap e342: 12 total, 12 up, 12 in 2022-09-10T06:39:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:08.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:08.452863+0000) 2022-09-10T06:39:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:08.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:08.854984+0000) 2022-09-10T06:39:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:09 smithi110 ceph-mon[97633]: pgmap v4161: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:39:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:09 smithi104 ceph-mon[97044]: pgmap v4161: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:39:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:09.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:09.453067+0000) 2022-09-10T06:39:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:09.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:09.855197+0000) 2022-09-10T06:39:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:10.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:10.453342+0000) 2022-09-10T06:39:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:10.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:10.855368+0000) 2022-09-10T06:39:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:11 smithi110 ceph-mon[97633]: pgmap v4162: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:39:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:11 smithi104 ceph-mon[97044]: pgmap v4162: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:39:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:11.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:11.453633+0000) 2022-09-10T06:39:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:11.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:11.855504+0000) 2022-09-10T06:39:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:12.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:12.453816+0000) 2022-09-10T06:39:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:12.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:12.855778+0000) 2022-09-10T06:39:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:13 smithi110 ceph-mon[97633]: pgmap v4163: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:39:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:13 smithi104 ceph-mon[97044]: pgmap v4163: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:39:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:13.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:13.453994+0000) 2022-09-10T06:39:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:13.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:13.855917+0000) 2022-09-10T06:39:14.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:39:14.080+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:39:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:14.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:14.454272+0000) 2022-09-10T06:39:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:15 smithi104 ceph-mon[97044]: osdmap e343: 12 total, 12 up, 12 in 2022-09-10T06:39:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:15 smithi104 ceph-mon[97044]: pgmap v4165: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:39:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:14.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:14.856092+0000) 2022-09-10T06:39:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:15 smithi110 ceph-mon[97633]: osdmap e343: 12 total, 12 up, 12 in 2022-09-10T06:39:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:15 smithi110 ceph-mon[97633]: pgmap v4165: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:39:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:15.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:15.454424+0000) 2022-09-10T06:39:16.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:15.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:15.856325+0000) 2022-09-10T06:39:16.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:16.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:16.454650+0000) 2022-09-10T06:39:17.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:16.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:16.856601+0000) 2022-09-10T06:39:17.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:17 smithi104 ceph-mon[97044]: pgmap v4166: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:39:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:17 smithi110 ceph-mon[97633]: pgmap v4166: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:39:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:17.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:17.454910+0000) 2022-09-10T06:39:18.314 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:17.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:17.856847+0000) 2022-09-10T06:39:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:18.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:18.455132+0000) 2022-09-10T06:39:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:18.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:18.857016+0000) 2022-09-10T06:39:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:19 smithi110 ceph-mon[97633]: pgmap v4167: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:39:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:19 smithi104 ceph-mon[97044]: pgmap v4167: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:39:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:19.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:19.455325+0000) 2022-09-10T06:39:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:19.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:19.857211+0000) 2022-09-10T06:39:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:20.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:20.455494+0000) 2022-09-10T06:39:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:20.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:20.857364+0000) 2022-09-10T06:39:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:21 smithi110 ceph-mon[97633]: pgmap v4168: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:39:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:21 smithi104 ceph-mon[97044]: pgmap v4168: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:39:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:21.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:21.455697+0000) 2022-09-10T06:39:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:21.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:21.857590+0000) 2022-09-10T06:39:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:22.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:22.455935+0000) 2022-09-10T06:39:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:22.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:22.857799+0000) 2022-09-10T06:39:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:23 smithi110 ceph-mon[97633]: pgmap v4169: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:39:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:23 smithi104 ceph-mon[97044]: pgmap v4169: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:39:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:23.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:23.456190+0000) 2022-09-10T06:39:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:23.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:23.858017+0000) 2022-09-10T06:39:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:24.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:24.456436+0000) 2022-09-10T06:39:25.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:25 smithi104 ceph-mon[97044]: pgmap v4170: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:39:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:24.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:24.858267+0000) 2022-09-10T06:39:25.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:25 smithi110 ceph-mon[97633]: pgmap v4170: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:39:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:25.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:25.456660+0000) 2022-09-10T06:39:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:25.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:25.858548+0000) 2022-09-10T06:39:26.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:26.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:26.456841+0000) 2022-09-10T06:39:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:26.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:26.858762+0000) 2022-09-10T06:39:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:27 smithi110 ceph-mon[97633]: pgmap v4171: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:39:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:27 smithi104 ceph-mon[97044]: pgmap v4171: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:39:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:27.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:27.457063+0000) 2022-09-10T06:39:28.315 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:27.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:27.859049+0000) 2022-09-10T06:39:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:28.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:28.457224+0000) 2022-09-10T06:39:29.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:39:29.095+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:39:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:28.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:28.859311+0000) 2022-09-10T06:39:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:29 smithi110 ceph-mon[97633]: pgmap v4172: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:29 smithi104 ceph-mon[97044]: pgmap v4172: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:29.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:29.457445+0000) 2022-09-10T06:39:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:29.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:29.859493+0000) 2022-09-10T06:39:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:30.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:30.457636+0000) 2022-09-10T06:39:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:30.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:30.859712+0000) 2022-09-10T06:39:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:31 smithi110 ceph-mon[97633]: pgmap v4173: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:31 smithi104 ceph-mon[97044]: pgmap v4173: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:31.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:31.457829+0000) 2022-09-10T06:39:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:31.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:31.859939+0000) 2022-09-10T06:39:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:32.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:32.458078+0000) 2022-09-10T06:39:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:32.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:32.860197+0000) 2022-09-10T06:39:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:33 smithi110 ceph-mon[97633]: pgmap v4174: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:33 smithi104 ceph-mon[97044]: pgmap v4174: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:33.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:33.458287+0000) 2022-09-10T06:39:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:33.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:33.860423+0000) 2022-09-10T06:39:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:34.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:34.458569+0000) 2022-09-10T06:39:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:35 smithi104 ceph-mon[97044]: pgmap v4175: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:34.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:34.860673+0000) 2022-09-10T06:39:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:35 smithi110 ceph-mon[97633]: pgmap v4175: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:35.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:35.458856+0000) 2022-09-10T06:39:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:35.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:35.860952+0000) 2022-09-10T06:39:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:36.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:36.459142+0000) 2022-09-10T06:39:37.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:36.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:36.861198+0000) 2022-09-10T06:39:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:37 smithi110 ceph-mon[97633]: pgmap v4176: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:37 smithi104 ceph-mon[97044]: pgmap v4176: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:37.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:37.459429+0000) 2022-09-10T06:39:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:37.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:37.861414+0000) 2022-09-10T06:39:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:38.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:38.459653+0000) 2022-09-10T06:39:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:38.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:38.861622+0000) 2022-09-10T06:39:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:39 smithi110 ceph-mon[97633]: pgmap v4177: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:39 smithi104 ceph-mon[97044]: pgmap v4177: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:39.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:39.459919+0000) 2022-09-10T06:39:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:39.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:39.861846+0000) 2022-09-10T06:39:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:40.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:40.460139+0000) 2022-09-10T06:39:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:40.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:40.862071+0000) 2022-09-10T06:39:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:41 smithi110 ceph-mon[97633]: pgmap v4178: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:41 smithi104 ceph-mon[97044]: pgmap v4178: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:41.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:41.460265+0000) 2022-09-10T06:39:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:41.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:41.862354+0000) 2022-09-10T06:39:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:42.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:42.460532+0000) 2022-09-10T06:39:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:42.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:42.862556+0000) 2022-09-10T06:39:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:43 smithi110 ceph-mon[97633]: pgmap v4179: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:43 smithi104 ceph-mon[97044]: pgmap v4179: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:43.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:43.460731+0000) 2022-09-10T06:39:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:43.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:43.862751+0000) 2022-09-10T06:39:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:39:44.098+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:39:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:39:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:39:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:39:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:39:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:39:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:39:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:39:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:39:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:44.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:44.460967+0000) 2022-09-10T06:39:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:44.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:44.863006+0000) 2022-09-10T06:39:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:45 smithi110 ceph-mon[97633]: pgmap v4180: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:45 smithi104 ceph-mon[97044]: pgmap v4180: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:45.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:45.461204+0000) 2022-09-10T06:39:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:45.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:45.863276+0000) 2022-09-10T06:39:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:46.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:46.461432+0000) 2022-09-10T06:39:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:46.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:46.863495+0000) 2022-09-10T06:39:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:47 smithi110 ceph-mon[97633]: pgmap v4181: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:47 smithi104 ceph-mon[97044]: pgmap v4181: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:47.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:47.461568+0000) 2022-09-10T06:39:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:47.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:47.863743+0000) 2022-09-10T06:39:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:48.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:48.461770+0000) 2022-09-10T06:39:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:48.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:48.863982+0000) 2022-09-10T06:39:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:49 smithi110 ceph-mon[97633]: pgmap v4182: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:49 smithi104 ceph-mon[97044]: pgmap v4182: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:49.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:49.461964+0000) 2022-09-10T06:39:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:49.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:49.864213+0000) 2022-09-10T06:39:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:50.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:50.462220+0000) 2022-09-10T06:39:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:50.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:50.864355+0000) 2022-09-10T06:39:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:51 smithi110 ceph-mon[97633]: pgmap v4183: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:51.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:51 smithi104 ceph-mon[97044]: pgmap v4183: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:51.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:51.462520+0000) 2022-09-10T06:39:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:51.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:51.864567+0000) 2022-09-10T06:39:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:52.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:52.462753+0000) 2022-09-10T06:39:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:52.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:52.864795+0000) 2022-09-10T06:39:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:53 smithi110 ceph-mon[97633]: pgmap v4184: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:53 smithi104 ceph-mon[97044]: pgmap v4184: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:53.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:53.462961+0000) 2022-09-10T06:39:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:53.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:53.865019+0000) 2022-09-10T06:39:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:54.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:54.463178+0000) 2022-09-10T06:39:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:55 smithi104 ceph-mon[97044]: pgmap v4185: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:54.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:54.865267+0000) 2022-09-10T06:39:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:55 smithi110 ceph-mon[97633]: pgmap v4185: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:55.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:55.463325+0000) 2022-09-10T06:39:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:55.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:55.865540+0000) 2022-09-10T06:39:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:56.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:56.463508+0000) 2022-09-10T06:39:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:56.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:56.865779+0000) 2022-09-10T06:39:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:57 smithi110 ceph-mon[97633]: pgmap v4186: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:57 smithi104 ceph-mon[97044]: pgmap v4186: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:39:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:57.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:57.463702+0000) 2022-09-10T06:39:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:57.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:57.865986+0000) 2022-09-10T06:39:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:58.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:58.463953+0000) 2022-09-10T06:39:59.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:39:59.109+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:39:59.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:58.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:58.866204+0000) 2022-09-10T06:39:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:39:59 smithi110 ceph-mon[97633]: pgmap v4187: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:39:59 smithi104 ceph-mon[97044]: pgmap v4187: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:39:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:39:59.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:59.464218+0000) 2022-09-10T06:40:00.246 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:39:59.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:39:59.866426+0000) 2022-09-10T06:40:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:40:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:40:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:00.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:00.464364+0000) 2022-09-10T06:40:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:00.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:00.866631+0000) 2022-09-10T06:40:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:01 smithi110 ceph-mon[97633]: pgmap v4188: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:01 smithi104 ceph-mon[97044]: pgmap v4188: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:01.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:01.464629+0000) 2022-09-10T06:40:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:01.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:01.866803+0000) 2022-09-10T06:40:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:02.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:02.464898+0000) 2022-09-10T06:40:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:02.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:02.867007+0000) 2022-09-10T06:40:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:03 smithi110 ceph-mon[97633]: pgmap v4189: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:03 smithi104 ceph-mon[97044]: pgmap v4189: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:03.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:03.465176+0000) 2022-09-10T06:40:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:03.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:03.867148+0000) 2022-09-10T06:40:04.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:04.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:04.465376+0000) 2022-09-10T06:40:05.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:05 smithi104 ceph-mon[97044]: pgmap v4190: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:04.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:04.867371+0000) 2022-09-10T06:40:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:05 smithi110 ceph-mon[97633]: pgmap v4190: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:05.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:05.465579+0000) 2022-09-10T06:40:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:05.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:05.867623+0000) 2022-09-10T06:40:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:06.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:06.465740+0000) 2022-09-10T06:40:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:06.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:06.867881+0000) 2022-09-10T06:40:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:07 smithi110 ceph-mon[97633]: pgmap v4191: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:40:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:07 smithi104 ceph-mon[97044]: pgmap v4191: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:40:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:07.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:07.465940+0000) 2022-09-10T06:40:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:07.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:07.868188+0000) 2022-09-10T06:40:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:08 smithi110 ceph-mon[97633]: osdmap e344: 12 total, 12 up, 12 in 2022-09-10T06:40:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:08 smithi104 ceph-mon[97044]: osdmap e344: 12 total, 12 up, 12 in 2022-09-10T06:40:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:08.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:08.466135+0000) 2022-09-10T06:40:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:08.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:08.868438+0000) 2022-09-10T06:40:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:09 smithi110 ceph-mon[97633]: pgmap v4193: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:40:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:09 smithi104 ceph-mon[97044]: pgmap v4193: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:40:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:09.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:09.466373+0000) 2022-09-10T06:40:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:09.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:09.868632+0000) 2022-09-10T06:40:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:10.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:10.466559+0000) 2022-09-10T06:40:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:10.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:10.868834+0000) 2022-09-10T06:40:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:11 smithi110 ceph-mon[97633]: pgmap v4194: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:40:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:11 smithi104 ceph-mon[97044]: pgmap v4194: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:40:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:11.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:11.466754+0000) 2022-09-10T06:40:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:11.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:11.869010+0000) 2022-09-10T06:40:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:12.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:12.466987+0000) 2022-09-10T06:40:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:12.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:12.869269+0000) 2022-09-10T06:40:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:13 smithi110 ceph-mon[97633]: pgmap v4195: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:40:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:13 smithi104 ceph-mon[97044]: pgmap v4195: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:40:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:13.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:13.467248+0000) 2022-09-10T06:40:14.125 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:13.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:13.869510+0000) 2022-09-10T06:40:14.468 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:40:14.123+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:40:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:14.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:14.467499+0000) 2022-09-10T06:40:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:15 smithi104 ceph-mon[97044]: osdmap e345: 12 total, 12 up, 12 in 2022-09-10T06:40:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:15 smithi104 ceph-mon[97044]: pgmap v4197: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:40:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:14.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:14.869782+0000) 2022-09-10T06:40:15.418 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:15 smithi110 ceph-mon[97633]: osdmap e345: 12 total, 12 up, 12 in 2022-09-10T06:40:15.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:15 smithi110 ceph-mon[97633]: pgmap v4197: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:40:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:15.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:15.467664+0000) 2022-09-10T06:40:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:15.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:15.869996+0000) 2022-09-10T06:40:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:16.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:16.467822+0000) 2022-09-10T06:40:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:16.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:16.870241+0000) 2022-09-10T06:40:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:17 smithi110 ceph-mon[97633]: pgmap v4198: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:40:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:17 smithi104 ceph-mon[97044]: pgmap v4198: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:40:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:17.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:17.468015+0000) 2022-09-10T06:40:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:17.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:17.870474+0000) 2022-09-10T06:40:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:18.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:18.468261+0000) 2022-09-10T06:40:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:18.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:18.870642+0000) 2022-09-10T06:40:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:19 smithi110 ceph-mon[97633]: pgmap v4199: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:40:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:19 smithi104 ceph-mon[97044]: pgmap v4199: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:40:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:19.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:19.468482+0000) 2022-09-10T06:40:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:19.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:19.870856+0000) 2022-09-10T06:40:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:20.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:20.468695+0000) 2022-09-10T06:40:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:20.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:20.871083+0000) 2022-09-10T06:40:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:21 smithi110 ceph-mon[97633]: pgmap v4200: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:40:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:21 smithi104 ceph-mon[97044]: pgmap v4200: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:40:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:21.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:21.468922+0000) 2022-09-10T06:40:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:21.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:21.871354+0000) 2022-09-10T06:40:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:22.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:22.469117+0000) 2022-09-10T06:40:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:22.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:22.871574+0000) 2022-09-10T06:40:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:23 smithi110 ceph-mon[97633]: pgmap v4201: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:40:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:23 smithi104 ceph-mon[97044]: pgmap v4201: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:40:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:23.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:23.469373+0000) 2022-09-10T06:40:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:23.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:23.871836+0000) 2022-09-10T06:40:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:24.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:24.469633+0000) 2022-09-10T06:40:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:25 smithi104 ceph-mon[97044]: pgmap v4202: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:40:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:24.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:24.872128+0000) 2022-09-10T06:40:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:25 smithi110 ceph-mon[97633]: pgmap v4202: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:40:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:25.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:25.469809+0000) 2022-09-10T06:40:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:25.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:25.872304+0000) 2022-09-10T06:40:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:26.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:26.469983+0000) 2022-09-10T06:40:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:26.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:26.872543+0000) 2022-09-10T06:40:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:27 smithi110 ceph-mon[97633]: pgmap v4203: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:40:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:27 smithi104 ceph-mon[97044]: pgmap v4203: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:40:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:27.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:27.470196+0000) 2022-09-10T06:40:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:27.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:27.872743+0000) 2022-09-10T06:40:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:28.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:28.470422+0000) 2022-09-10T06:40:29.141 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:28.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:28.872971+0000) 2022-09-10T06:40:29.472 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:40:29.139+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:40:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:29 smithi110 ceph-mon[97633]: pgmap v4204: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:29 smithi104 ceph-mon[97044]: pgmap v4204: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:29.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:29.470624+0000) 2022-09-10T06:40:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:29.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:29.873166+0000) 2022-09-10T06:40:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:30.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:30.470887+0000) 2022-09-10T06:40:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:30.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:30.873405+0000) 2022-09-10T06:40:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:31 smithi110 ceph-mon[97633]: pgmap v4205: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:31 smithi104 ceph-mon[97044]: pgmap v4205: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:31.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:31.471156+0000) 2022-09-10T06:40:32.326 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:31.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:31.873619+0000) 2022-09-10T06:40:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:32.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:32.471345+0000) 2022-09-10T06:40:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:32.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:32.873857+0000) 2022-09-10T06:40:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:33 smithi110 ceph-mon[97633]: pgmap v4206: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:33 smithi104 ceph-mon[97044]: pgmap v4206: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:33.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:33.471543+0000) 2022-09-10T06:40:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:33.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:33.874069+0000) 2022-09-10T06:40:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:34.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:34.471746+0000) 2022-09-10T06:40:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:35 smithi104 ceph-mon[97044]: pgmap v4207: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:34.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:34.874362+0000) 2022-09-10T06:40:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:35 smithi110 ceph-mon[97633]: pgmap v4207: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:35.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:35.471906+0000) 2022-09-10T06:40:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:35.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:35.874616+0000) 2022-09-10T06:40:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:36.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:36.472056+0000) 2022-09-10T06:40:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:36.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:36.874803+0000) 2022-09-10T06:40:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:37 smithi110 ceph-mon[97633]: pgmap v4208: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:37 smithi104 ceph-mon[97044]: pgmap v4208: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:37.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:37.472289+0000) 2022-09-10T06:40:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:37.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:37.875000+0000) 2022-09-10T06:40:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:38.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:38.472540+0000) 2022-09-10T06:40:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:38.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:38.875169+0000) 2022-09-10T06:40:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:39 smithi110 ceph-mon[97633]: pgmap v4209: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:39 smithi104 ceph-mon[97044]: pgmap v4209: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:39.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:39.472798+0000) 2022-09-10T06:40:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:39.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:39.875354+0000) 2022-09-10T06:40:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:40.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:40.472952+0000) 2022-09-10T06:40:41.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:40.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:40.875532+0000) 2022-09-10T06:40:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:41 smithi110 ceph-mon[97633]: pgmap v4210: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:41 smithi104 ceph-mon[97044]: pgmap v4210: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:41.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:41.473218+0000) 2022-09-10T06:40:42.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:41.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:41.875779+0000) 2022-09-10T06:40:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:42.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:42.473467+0000) 2022-09-10T06:40:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:42.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:42.875944+0000) 2022-09-10T06:40:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:43 smithi110 ceph-mon[97633]: pgmap v4211: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:43 smithi104 ceph-mon[97044]: pgmap v4211: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:43.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:43.473684+0000) 2022-09-10T06:40:44.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:43.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:43.876134+0000) 2022-09-10T06:40:44.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:40:44.144+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:40:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:40:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:40:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:40:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:40:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:40:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:40:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:40:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:40:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:44.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:44.473888+0000) 2022-09-10T06:40:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:44.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:44.876334+0000) 2022-09-10T06:40:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:45 smithi110 ceph-mon[97633]: pgmap v4212: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:45 smithi104 ceph-mon[97044]: pgmap v4212: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:45.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:45.474163+0000) 2022-09-10T06:40:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:45.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:45.876477+0000) 2022-09-10T06:40:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:46.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:46.474348+0000) 2022-09-10T06:40:47.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:46.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:46.876698+0000) 2022-09-10T06:40:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:47 smithi110 ceph-mon[97633]: pgmap v4213: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:47 smithi104 ceph-mon[97044]: pgmap v4213: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:47.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:47.474511+0000) 2022-09-10T06:40:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:47.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:47.876870+0000) 2022-09-10T06:40:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:48.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:48.474667+0000) 2022-09-10T06:40:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:48.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:48.877016+0000) 2022-09-10T06:40:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:49 smithi110 ceph-mon[97633]: pgmap v4214: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:49 smithi104 ceph-mon[97044]: pgmap v4214: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:49.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:49.474835+0000) 2022-09-10T06:40:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:49.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:49.877241+0000) 2022-09-10T06:40:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:50.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:50.475062+0000) 2022-09-10T06:40:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:50.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:50.877439+0000) 2022-09-10T06:40:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:51 smithi110 ceph-mon[97633]: pgmap v4215: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:51 smithi104 ceph-mon[97044]: pgmap v4215: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:51.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:51.475324+0000) 2022-09-10T06:40:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:51.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:51.877567+0000) 2022-09-10T06:40:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:52.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:52.475597+0000) 2022-09-10T06:40:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:52.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:52.877790+0000) 2022-09-10T06:40:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:53 smithi110 ceph-mon[97633]: pgmap v4216: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:53 smithi104 ceph-mon[97044]: pgmap v4216: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:53.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:53.475841+0000) 2022-09-10T06:40:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:53.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:53.878060+0000) 2022-09-10T06:40:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:54.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:54.476039+0000) 2022-09-10T06:40:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:55 smithi104 ceph-mon[97044]: pgmap v4217: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:54.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:54.878343+0000) 2022-09-10T06:40:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:55 smithi110 ceph-mon[97633]: pgmap v4217: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:55.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:55.476260+0000) 2022-09-10T06:40:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:55.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:55.878559+0000) 2022-09-10T06:40:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:56.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:56.476422+0000) 2022-09-10T06:40:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:56.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:56.878792+0000) 2022-09-10T06:40:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:57 smithi110 ceph-mon[97633]: pgmap v4218: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:57 smithi104 ceph-mon[97044]: pgmap v4218: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:40:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:57.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:57.476663+0000) 2022-09-10T06:40:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:57.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:57.879039+0000) 2022-09-10T06:40:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:58.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:58.476871+0000) 2022-09-10T06:40:59.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:58.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:58.879305+0000) 2022-09-10T06:40:59.477 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:40:59.157+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:40:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:40:59 smithi110 ceph-mon[97633]: pgmap v4219: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:40:59 smithi104 ceph-mon[97044]: pgmap v4219: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:40:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:40:59.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:59.477148+0000) 2022-09-10T06:41:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:40:59.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:40:59.879566+0000) 2022-09-10T06:41:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:00.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:00.477346+0000) 2022-09-10T06:41:01.172 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:00.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:00.879786+0000) 2022-09-10T06:41:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:01 smithi110 ceph-mon[97633]: pgmap v4220: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:01 smithi104 ceph-mon[97044]: pgmap v4220: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:01.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:01.477570+0000) 2022-09-10T06:41:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:01.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:01.880056+0000) 2022-09-10T06:41:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:02.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:02.477779+0000) 2022-09-10T06:41:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:02.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:02.880346+0000) 2022-09-10T06:41:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:03 smithi110 ceph-mon[97633]: pgmap v4221: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:03 smithi104 ceph-mon[97044]: pgmap v4221: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:03.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:03.478042+0000) 2022-09-10T06:41:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:03.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:03.880509+0000) 2022-09-10T06:41:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:04.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:04.478241+0000) 2022-09-10T06:41:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:04.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:04.880758+0000) 2022-09-10T06:41:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:05 smithi110 ceph-mon[97633]: pgmap v4222: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:05 smithi104 ceph-mon[97044]: pgmap v4222: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:05.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:05.478470+0000) 2022-09-10T06:41:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:05.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:05.880996+0000) 2022-09-10T06:41:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:06.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:06.478694+0000) 2022-09-10T06:41:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:06.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:06.881229+0000) 2022-09-10T06:41:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:07 smithi110 ceph-mon[97633]: pgmap v4223: 129 pgs: 129 active+clean; 8.1 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:41:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:07 smithi104 ceph-mon[97044]: pgmap v4223: 129 pgs: 129 active+clean; 8.1 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:41:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:07.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:07.478863+0000) 2022-09-10T06:41:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:07.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:07.881493+0000) 2022-09-10T06:41:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:08 smithi110 ceph-mon[97633]: osdmap e346: 12 total, 12 up, 12 in 2022-09-10T06:41:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:08 smithi104 ceph-mon[97044]: osdmap e346: 12 total, 12 up, 12 in 2022-09-10T06:41:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:08.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:08.479078+0000) 2022-09-10T06:41:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:08.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:08.881719+0000) 2022-09-10T06:41:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:09 smithi110 ceph-mon[97633]: pgmap v4225: 129 pgs: 129 active+clean; 8.1 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:41:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:09 smithi104 ceph-mon[97044]: pgmap v4225: 129 pgs: 129 active+clean; 8.1 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:41:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:09.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:09.479311+0000) 2022-09-10T06:41:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:09.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:09.881964+0000) 2022-09-10T06:41:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:10.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:10.479540+0000) 2022-09-10T06:41:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:10.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:10.882212+0000) 2022-09-10T06:41:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:11 smithi110 ceph-mon[97633]: pgmap v4226: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:41:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:11 smithi104 ceph-mon[97044]: pgmap v4226: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:41:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:11.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:11.479800+0000) 2022-09-10T06:41:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:11.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:11.882446+0000) 2022-09-10T06:41:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:12.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:12.480003+0000) 2022-09-10T06:41:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:12.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:12.882677+0000) 2022-09-10T06:41:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:13 smithi110 ceph-mon[97633]: pgmap v4227: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:41:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:13 smithi104 ceph-mon[97044]: pgmap v4227: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:41:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:13.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:13.480247+0000) 2022-09-10T06:41:14.175 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:13.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:13.882901+0000) 2022-09-10T06:41:14.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:41:14.173+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:41:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:14.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:14.480428+0000) 2022-09-10T06:41:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:15 smithi104 ceph-mon[97044]: osdmap e347: 12 total, 12 up, 12 in 2022-09-10T06:41:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:15 smithi104 ceph-mon[97044]: pgmap v4229: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:41:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:14.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:14.883178+0000) 2022-09-10T06:41:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:15 smithi110 ceph-mon[97633]: osdmap e347: 12 total, 12 up, 12 in 2022-09-10T06:41:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:15 smithi110 ceph-mon[97633]: pgmap v4229: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:41:15.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:15.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:15.480604+0000) 2022-09-10T06:41:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:15.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:15.883338+0000) 2022-09-10T06:41:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:16.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:16.480756+0000) 2022-09-10T06:41:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:16.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:16.883535+0000) 2022-09-10T06:41:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:17 smithi110 ceph-mon[97633]: pgmap v4230: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:41:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:17 smithi104 ceph-mon[97044]: pgmap v4230: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:41:17.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:17.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:17.480979+0000) 2022-09-10T06:41:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:17.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:17.883769+0000) 2022-09-10T06:41:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:18.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:18.481188+0000) 2022-09-10T06:41:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:18.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:18.883997+0000) 2022-09-10T06:41:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:19 smithi110 ceph-mon[97633]: pgmap v4231: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:41:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:19 smithi104 ceph-mon[97044]: pgmap v4231: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:41:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:19.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:19.481365+0000) 2022-09-10T06:41:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:19.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:19.884183+0000) 2022-09-10T06:41:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:20.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:20.481547+0000) 2022-09-10T06:41:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:20.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:20.884433+0000) 2022-09-10T06:41:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:21 smithi110 ceph-mon[97633]: pgmap v4232: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:41:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:21 smithi104 ceph-mon[97044]: pgmap v4232: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:41:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:21.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:21.481743+0000) 2022-09-10T06:41:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:21.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:21.884657+0000) 2022-09-10T06:41:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:22.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:22.481916+0000) 2022-09-10T06:41:23.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:22.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:22.884798+0000) 2022-09-10T06:41:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:23 smithi110 ceph-mon[97633]: pgmap v4233: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:41:23.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:23 smithi104 ceph-mon[97044]: pgmap v4233: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:41:23.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:23.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:23.482161+0000) 2022-09-10T06:41:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:23.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:23.885003+0000) 2022-09-10T06:41:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:24.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:24.482383+0000) 2022-09-10T06:41:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:24.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:24.885250+0000) 2022-09-10T06:41:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:25 smithi110 ceph-mon[97633]: pgmap v4234: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:41:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:25 smithi104 ceph-mon[97044]: pgmap v4234: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:41:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:25.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:25.482662+0000) 2022-09-10T06:41:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:25.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:25.885437+0000) 2022-09-10T06:41:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:26.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:26.482830+0000) 2022-09-10T06:41:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:26.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:26.885648+0000) 2022-09-10T06:41:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:27 smithi110 ceph-mon[97633]: pgmap v4235: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:41:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:27 smithi104 ceph-mon[97044]: pgmap v4235: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:41:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:27.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:27.483008+0000) 2022-09-10T06:41:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:27.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:27.885833+0000) 2022-09-10T06:41:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:28.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:28.483247+0000) 2022-09-10T06:41:29.182 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:28.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:28.886044+0000) 2022-09-10T06:41:29.484 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:41:29.180+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:41:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:29 smithi110 ceph-mon[97633]: pgmap v4236: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:29 smithi104 ceph-mon[97044]: pgmap v4236: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:29.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:29.483508+0000) 2022-09-10T06:41:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:29.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:29.886324+0000) 2022-09-10T06:41:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:30.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:30.483703+0000) 2022-09-10T06:41:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:30.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:30.886548+0000) 2022-09-10T06:41:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:31 smithi110 ceph-mon[97633]: pgmap v4237: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:31 smithi104 ceph-mon[97044]: pgmap v4237: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:31.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:31.483954+0000) 2022-09-10T06:41:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:31.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:31.886764+0000) 2022-09-10T06:41:32.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:32.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:32.484154+0000) 2022-09-10T06:41:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:32.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:32.886927+0000) 2022-09-10T06:41:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:33 smithi110 ceph-mon[97633]: pgmap v4238: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:33 smithi104 ceph-mon[97044]: pgmap v4238: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:33.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:33.484383+0000) 2022-09-10T06:41:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:33.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:33.887188+0000) 2022-09-10T06:41:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:34.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:34.484581+0000) 2022-09-10T06:41:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:34.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:34.887382+0000) 2022-09-10T06:41:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:35 smithi110 ceph-mon[97633]: pgmap v4239: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:35 smithi104 ceph-mon[97044]: pgmap v4239: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:35.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:35.484764+0000) 2022-09-10T06:41:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:35.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:35.887510+0000) 2022-09-10T06:41:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:36.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:36.484941+0000) 2022-09-10T06:41:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:36.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:36.887735+0000) 2022-09-10T06:41:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:37 smithi110 ceph-mon[97633]: pgmap v4240: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:37 smithi104 ceph-mon[97044]: pgmap v4240: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:37.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:37.485187+0000) 2022-09-10T06:41:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:37.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:37.887976+0000) 2022-09-10T06:41:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:38.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:38.485364+0000) 2022-09-10T06:41:39.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:38.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:38.888229+0000) 2022-09-10T06:41:39.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:39 smithi110 ceph-mon[97633]: pgmap v4241: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:39 smithi104 ceph-mon[97044]: pgmap v4241: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:39.822 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:39.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:39.485535+0000) 2022-09-10T06:41:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:39.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:39.888520+0000) 2022-09-10T06:41:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:40.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:40.485759+0000) 2022-09-10T06:41:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:40.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:40.888758+0000) 2022-09-10T06:41:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:41 smithi110 ceph-mon[97633]: pgmap v4242: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:41 smithi104 ceph-mon[97044]: pgmap v4242: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:41.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:41.485982+0000) 2022-09-10T06:41:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:41.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:41.889007+0000) 2022-09-10T06:41:42.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:42.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:42.486180+0000) 2022-09-10T06:41:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:42.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:42.889221+0000) 2022-09-10T06:41:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:43 smithi110 ceph-mon[97633]: pgmap v4243: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:43 smithi104 ceph-mon[97044]: pgmap v4243: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:43.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:43.486378+0000) 2022-09-10T06:41:44.190 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:43.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:43.889397+0000) 2022-09-10T06:41:44.487 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:41:44.188+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:41:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:41:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:41:44.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:41:44.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:41:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:41:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:41:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:41:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:41:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:44.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:44.486622+0000) 2022-09-10T06:41:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:44.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:44.889629+0000) 2022-09-10T06:41:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:45 smithi110 ceph-mon[97633]: pgmap v4244: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:45 smithi104 ceph-mon[97044]: pgmap v4244: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:45.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:45.486780+0000) 2022-09-10T06:41:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:45.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:45.889859+0000) 2022-09-10T06:41:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:46.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:46.487005+0000) 2022-09-10T06:41:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:46.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:46.890119+0000) 2022-09-10T06:41:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:47 smithi110 ceph-mon[97633]: pgmap v4245: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:47 smithi104 ceph-mon[97044]: pgmap v4245: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:47.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:47.487230+0000) 2022-09-10T06:41:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:47.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:47.890271+0000) 2022-09-10T06:41:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:48.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:48.487468+0000) 2022-09-10T06:41:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:48.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:48.890534+0000) 2022-09-10T06:41:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:49 smithi110 ceph-mon[97633]: pgmap v4246: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:49 smithi104 ceph-mon[97044]: pgmap v4246: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:49.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:49.487711+0000) 2022-09-10T06:41:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:49.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:49.890741+0000) 2022-09-10T06:41:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:50.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:50.487953+0000) 2022-09-10T06:41:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:50.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:50.891036+0000) 2022-09-10T06:41:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:51 smithi110 ceph-mon[97633]: pgmap v4247: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:51 smithi104 ceph-mon[97044]: pgmap v4247: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:51.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:51.488138+0000) 2022-09-10T06:41:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:51.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:51.891247+0000) 2022-09-10T06:41:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:52.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:52.488307+0000) 2022-09-10T06:41:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:52.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:52.891458+0000) 2022-09-10T06:41:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:53 smithi110 ceph-mon[97633]: pgmap v4248: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:53 smithi104 ceph-mon[97044]: pgmap v4248: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:53.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:53.488529+0000) 2022-09-10T06:41:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:53.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:53.891701+0000) 2022-09-10T06:41:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:54.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:54.488717+0000) 2022-09-10T06:41:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:54.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:54.891949+0000) 2022-09-10T06:41:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:55 smithi110 ceph-mon[97633]: pgmap v4249: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:55 smithi104 ceph-mon[97044]: pgmap v4249: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:55.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:55.488917+0000) 2022-09-10T06:41:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:55.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:55.892202+0000) 2022-09-10T06:41:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:56.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:56.489164+0000) 2022-09-10T06:41:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:56.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:56.892406+0000) 2022-09-10T06:41:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:57 smithi110 ceph-mon[97633]: pgmap v4250: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:57 smithi104 ceph-mon[97044]: pgmap v4250: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:41:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:57.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:57.489384+0000) 2022-09-10T06:41:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:57.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:57.892553+0000) 2022-09-10T06:41:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:58.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:58.489617+0000) 2022-09-10T06:41:59.192 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:58.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:58.892717+0000) 2022-09-10T06:41:59.490 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:41:59.191+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:41:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:41:59 smithi110 ceph-mon[97633]: pgmap v4251: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:41:59 smithi104 ceph-mon[97044]: pgmap v4251: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:41:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:41:59.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:59.489873+0000) 2022-09-10T06:42:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:41:59.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:41:59.892941+0000) 2022-09-10T06:42:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:00.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:00.490050+0000) 2022-09-10T06:42:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:00.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:00.893207+0000) 2022-09-10T06:42:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:01 smithi110 ceph-mon[97633]: pgmap v4252: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:01 smithi104 ceph-mon[97044]: pgmap v4252: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:01.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:01.490281+0000) 2022-09-10T06:42:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:01.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:01.893439+0000) 2022-09-10T06:42:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:02.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:02.490502+0000) 2022-09-10T06:42:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:02.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:02.893621+0000) 2022-09-10T06:42:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:03 smithi110 ceph-mon[97633]: pgmap v4253: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:03 smithi104 ceph-mon[97044]: pgmap v4253: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:03.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:03.490658+0000) 2022-09-10T06:42:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:03.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:03.893864+0000) 2022-09-10T06:42:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:04.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:04.490855+0000) 2022-09-10T06:42:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:04.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:04.894143+0000) 2022-09-10T06:42:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:05 smithi110 ceph-mon[97633]: pgmap v4254: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:05 smithi104 ceph-mon[97044]: pgmap v4254: 129 pgs: 129 active+clean; 8.1 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:05.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:05.491087+0000) 2022-09-10T06:42:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:05.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:05.894313+0000) 2022-09-10T06:42:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:06.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:06.491320+0000) 2022-09-10T06:42:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:06.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:06.894453+0000) 2022-09-10T06:42:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:07 smithi110 ceph-mon[97633]: pgmap v4255: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:42:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:07 smithi104 ceph-mon[97044]: pgmap v4255: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:42:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:07.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:07.491534+0000) 2022-09-10T06:42:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:07.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:07.894651+0000) 2022-09-10T06:42:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:08 smithi110 ceph-mon[97633]: osdmap e348: 12 total, 12 up, 12 in 2022-09-10T06:42:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:08 smithi104 ceph-mon[97044]: osdmap e348: 12 total, 12 up, 12 in 2022-09-10T06:42:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:08.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:08.491767+0000) 2022-09-10T06:42:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:08.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:08.894785+0000) 2022-09-10T06:42:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:09 smithi110 ceph-mon[97633]: pgmap v4257: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:42:09.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:09 smithi104 ceph-mon[97044]: pgmap v4257: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:42:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:09.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:09.491959+0000) 2022-09-10T06:42:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:09.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:09.894970+0000) 2022-09-10T06:42:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:10.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:10.492152+0000) 2022-09-10T06:42:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:10.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:10.895138+0000) 2022-09-10T06:42:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:11 smithi110 ceph-mon[97633]: pgmap v4258: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:42:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:11 smithi104 ceph-mon[97044]: pgmap v4258: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:42:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:11.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:11.492357+0000) 2022-09-10T06:42:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:11.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:11.895319+0000) 2022-09-10T06:42:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:12.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:12.492508+0000) 2022-09-10T06:42:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:12.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:12.895475+0000) 2022-09-10T06:42:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:13 smithi110 ceph-mon[97633]: pgmap v4259: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:42:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:13 smithi104 ceph-mon[97044]: pgmap v4259: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:42:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:13.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:13.492723+0000) 2022-09-10T06:42:14.206 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:13.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:13.895608+0000) 2022-09-10T06:42:14.493 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:42:14.204+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:42:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:14.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:14.492878+0000) 2022-09-10T06:42:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:14.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:14.895812+0000) 2022-09-10T06:42:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:15 smithi110 ceph-mon[97633]: osdmap e349: 12 total, 12 up, 12 in 2022-09-10T06:42:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:15 smithi110 ceph-mon[97633]: pgmap v4261: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:42:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:15 smithi104 ceph-mon[97044]: osdmap e349: 12 total, 12 up, 12 in 2022-09-10T06:42:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:15 smithi104 ceph-mon[97044]: pgmap v4261: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:42:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:15.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:15.493180+0000) 2022-09-10T06:42:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:15.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:15.896035+0000) 2022-09-10T06:42:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:16.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:16.493367+0000) 2022-09-10T06:42:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:16.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:16.896229+0000) 2022-09-10T06:42:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:17 smithi110 ceph-mon[97633]: pgmap v4262: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:42:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:17 smithi104 ceph-mon[97044]: pgmap v4262: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:42:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:17.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:17.493506+0000) 2022-09-10T06:42:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:17.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:17.896446+0000) 2022-09-10T06:42:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:18.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:18.493747+0000) 2022-09-10T06:42:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:18.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:18.896649+0000) 2022-09-10T06:42:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:19 smithi110 ceph-mon[97633]: pgmap v4263: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:42:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:19 smithi104 ceph-mon[97044]: pgmap v4263: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:42:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:19.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:19.493995+0000) 2022-09-10T06:42:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:19.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:19.896872+0000) 2022-09-10T06:42:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:20.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:20.494192+0000) 2022-09-10T06:42:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:20.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:20.897065+0000) 2022-09-10T06:42:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:21 smithi110 ceph-mon[97633]: pgmap v4264: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:42:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:21 smithi104 ceph-mon[97044]: pgmap v4264: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:42:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:21.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:21.494421+0000) 2022-09-10T06:42:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:21.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:21.897271+0000) 2022-09-10T06:42:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:22.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:22.494644+0000) 2022-09-10T06:42:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:22.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:22.897519+0000) 2022-09-10T06:42:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:23 smithi110 ceph-mon[97633]: pgmap v4265: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:42:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:23 smithi104 ceph-mon[97044]: pgmap v4265: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:42:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:23.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:23.494869+0000) 2022-09-10T06:42:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:23.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:23.897770+0000) 2022-09-10T06:42:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:24.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:24.495011+0000) 2022-09-10T06:42:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:24.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:24.897984+0000) 2022-09-10T06:42:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:25 smithi110 ceph-mon[97633]: pgmap v4266: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:42:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:25 smithi104 ceph-mon[97044]: pgmap v4266: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:42:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:25.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:25.495234+0000) 2022-09-10T06:42:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:25.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:25.898166+0000) 2022-09-10T06:42:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:26.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:26.495495+0000) 2022-09-10T06:42:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:26.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:26.898374+0000) 2022-09-10T06:42:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:27 smithi110 ceph-mon[97633]: pgmap v4267: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:42:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:27 smithi104 ceph-mon[97044]: pgmap v4267: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:42:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:27.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:27.495705+0000) 2022-09-10T06:42:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:27.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:27.898597+0000) 2022-09-10T06:42:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:28.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:28.495925+0000) 2022-09-10T06:42:29.221 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:28.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:28.898850+0000) 2022-09-10T06:42:29.497 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:42:29.220+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:42:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:29 smithi110 ceph-mon[97633]: pgmap v4268: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:29 smithi104 ceph-mon[97044]: pgmap v4268: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:29.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:29.496191+0000) 2022-09-10T06:42:30.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:29.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:29.899193+0000) 2022-09-10T06:42:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:30.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:30.496448+0000) 2022-09-10T06:42:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:30.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:30.899415+0000) 2022-09-10T06:42:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:31 smithi110 ceph-mon[97633]: pgmap v4269: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:31 smithi104 ceph-mon[97044]: pgmap v4269: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:31.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:31.496718+0000) 2022-09-10T06:42:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:31.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:31.899679+0000) 2022-09-10T06:42:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:32.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:32.496876+0000) 2022-09-10T06:42:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:32.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:32.899924+0000) 2022-09-10T06:42:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:33 smithi110 ceph-mon[97633]: pgmap v4270: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:33 smithi104 ceph-mon[97044]: pgmap v4270: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:33.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:33.497060+0000) 2022-09-10T06:42:34.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:33.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:33.900176+0000) 2022-09-10T06:42:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:34.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:34.497320+0000) 2022-09-10T06:42:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:34.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:34.900368+0000) 2022-09-10T06:42:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:35 smithi110 ceph-mon[97633]: pgmap v4271: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:35 smithi104 ceph-mon[97044]: pgmap v4271: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:35.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:35.497549+0000) 2022-09-10T06:42:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:35.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:35.900544+0000) 2022-09-10T06:42:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:36.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:36.497700+0000) 2022-09-10T06:42:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:36.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:36.900718+0000) 2022-09-10T06:42:37.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:37 smithi110 ceph-mon[97633]: pgmap v4272: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:37 smithi104 ceph-mon[97044]: pgmap v4272: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:37.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:37.497886+0000) 2022-09-10T06:42:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:37.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:37.900944+0000) 2022-09-10T06:42:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:38.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:38.498143+0000) 2022-09-10T06:42:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:38.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:38.901191+0000) 2022-09-10T06:42:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:39 smithi110 ceph-mon[97633]: pgmap v4273: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:39 smithi104 ceph-mon[97044]: pgmap v4273: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:39.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:39.498397+0000) 2022-09-10T06:42:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:39.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:39.901369+0000) 2022-09-10T06:42:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:40.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:40.498610+0000) 2022-09-10T06:42:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:40.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:40.901603+0000) 2022-09-10T06:42:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:41 smithi110 ceph-mon[97633]: pgmap v4274: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:41 smithi104 ceph-mon[97044]: pgmap v4274: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:41.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:41.498838+0000) 2022-09-10T06:42:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:41.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:41.901846+0000) 2022-09-10T06:42:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:42.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:42.499133+0000) 2022-09-10T06:42:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:42.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:42.902061+0000) 2022-09-10T06:42:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:43 smithi110 ceph-mon[97633]: pgmap v4275: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:43 smithi104 ceph-mon[97044]: pgmap v4275: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:43.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:43.499329+0000) 2022-09-10T06:42:44.237 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:43.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:43.902268+0000) 2022-09-10T06:42:44.500 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:42:44.235+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:42:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:42:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:42:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:42:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:42:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:42:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:42:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:42:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:42:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:44.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:44.499478+0000) 2022-09-10T06:42:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:44.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:44.902541+0000) 2022-09-10T06:42:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:45 smithi110 ceph-mon[97633]: pgmap v4276: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:45 smithi104 ceph-mon[97044]: pgmap v4276: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:45.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:45.499723+0000) 2022-09-10T06:42:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:45.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:45.902759+0000) 2022-09-10T06:42:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:46.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:46.499991+0000) 2022-09-10T06:42:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:46.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:46.902883+0000) 2022-09-10T06:42:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:47 smithi110 ceph-mon[97633]: pgmap v4277: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:47 smithi104 ceph-mon[97044]: pgmap v4277: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:47.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:47.500263+0000) 2022-09-10T06:42:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:47.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:47.903119+0000) 2022-09-10T06:42:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:48.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:48.500451+0000) 2022-09-10T06:42:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:48.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:48.903343+0000) 2022-09-10T06:42:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:49 smithi110 ceph-mon[97633]: pgmap v4278: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:49 smithi104 ceph-mon[97044]: pgmap v4278: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:49.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:49.500628+0000) 2022-09-10T06:42:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:49.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:49.903520+0000) 2022-09-10T06:42:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:50.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:50.500835+0000) 2022-09-10T06:42:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:50.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:50.903708+0000) 2022-09-10T06:42:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:51 smithi110 ceph-mon[97633]: pgmap v4279: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:51 smithi104 ceph-mon[97044]: pgmap v4279: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:51.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:51.501119+0000) 2022-09-10T06:42:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:51.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:51.903895+0000) 2022-09-10T06:42:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:52.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:52.501329+0000) 2022-09-10T06:42:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:52.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:52.904148+0000) 2022-09-10T06:42:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:53 smithi110 ceph-mon[97633]: pgmap v4280: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:53 smithi104 ceph-mon[97044]: pgmap v4280: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:53.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:53.501533+0000) 2022-09-10T06:42:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:53.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:53.904415+0000) 2022-09-10T06:42:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:54.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:54.501760+0000) 2022-09-10T06:42:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:54.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:54.904571+0000) 2022-09-10T06:42:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:55 smithi110 ceph-mon[97633]: pgmap v4281: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:55 smithi104 ceph-mon[97044]: pgmap v4281: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:55.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:55.501949+0000) 2022-09-10T06:42:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:55.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:55.904791+0000) 2022-09-10T06:42:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:56.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:56.502187+0000) 2022-09-10T06:42:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:56.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:56.905006+0000) 2022-09-10T06:42:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:57 smithi110 ceph-mon[97633]: pgmap v4282: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:57 smithi104 ceph-mon[97044]: pgmap v4282: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:42:57.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:57.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:57.502479+0000) 2022-09-10T06:42:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:57.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:57.905286+0000) 2022-09-10T06:42:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:58.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:58.502710+0000) 2022-09-10T06:42:59.253 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:58.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:58.905427+0000) 2022-09-10T06:42:59.503 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:42:59.251+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:42:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:42:59 smithi110 ceph-mon[97633]: pgmap v4283: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:42:59 smithi104 ceph-mon[97044]: pgmap v4283: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:42:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:42:59.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:59.502969+0000) 2022-09-10T06:43:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:42:59.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:42:59.905647+0000) 2022-09-10T06:43:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:00.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:00.503112+0000) 2022-09-10T06:43:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:00.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:00.905816+0000) 2022-09-10T06:43:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:01 smithi110 ceph-mon[97633]: pgmap v4284: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:01 smithi104 ceph-mon[97044]: pgmap v4284: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:01.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:01.503323+0000) 2022-09-10T06:43:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:01.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:01.906053+0000) 2022-09-10T06:43:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:02.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:02.503580+0000) 2022-09-10T06:43:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:02.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:02.906319+0000) 2022-09-10T06:43:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:03 smithi110 ceph-mon[97633]: pgmap v4285: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:03 smithi104 ceph-mon[97044]: pgmap v4285: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:03.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:03.503764+0000) 2022-09-10T06:43:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:03.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:03.906611+0000) 2022-09-10T06:43:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:04.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:04.503954+0000) 2022-09-10T06:43:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:04.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:04.906854+0000) 2022-09-10T06:43:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:05 smithi110 ceph-mon[97633]: pgmap v4286: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:05 smithi104 ceph-mon[97044]: pgmap v4286: 129 pgs: 129 active+clean; 8.1 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:05.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:05.504160+0000) 2022-09-10T06:43:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:05.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:05.907082+0000) 2022-09-10T06:43:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:06.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:06.504370+0000) 2022-09-10T06:43:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:06.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:06.907338+0000) 2022-09-10T06:43:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:07 smithi110 ceph-mon[97633]: pgmap v4287: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:43:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:07 smithi104 ceph-mon[97044]: pgmap v4287: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:43:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:07.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:07.504595+0000) 2022-09-10T06:43:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:07.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:07.907658+0000) 2022-09-10T06:43:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:08 smithi110 ceph-mon[97633]: osdmap e350: 12 total, 12 up, 12 in 2022-09-10T06:43:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:08 smithi104 ceph-mon[97044]: osdmap e350: 12 total, 12 up, 12 in 2022-09-10T06:43:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:08.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:08.504855+0000) 2022-09-10T06:43:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:08.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:08.907838+0000) 2022-09-10T06:43:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:09 smithi110 ceph-mon[97633]: pgmap v4289: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:43:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:09 smithi104 ceph-mon[97044]: pgmap v4289: 129 pgs: 129 active+clean; 8.1 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:43:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:09.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:09.505167+0000) 2022-09-10T06:43:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:09.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:09.908052+0000) 2022-09-10T06:43:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:10.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:10.505384+0000) 2022-09-10T06:43:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:10.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:10.908260+0000) 2022-09-10T06:43:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:11 smithi110 ceph-mon[97633]: pgmap v4290: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:43:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:11 smithi104 ceph-mon[97044]: pgmap v4290: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:43:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:11.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:11.505593+0000) 2022-09-10T06:43:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:11.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:11.908519+0000) 2022-09-10T06:43:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:12.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:12.505730+0000) 2022-09-10T06:43:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:12.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:12.908788+0000) 2022-09-10T06:43:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:13 smithi110 ceph-mon[97633]: pgmap v4291: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:43:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:13 smithi104 ceph-mon[97044]: pgmap v4291: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:43:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:13.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:13.505941+0000) 2022-09-10T06:43:14.268 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:13.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:13.909035+0000) 2022-09-10T06:43:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:14.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:14.506190+0000) 2022-09-10T06:43:14.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:43:14.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:43:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:14.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:14.909348+0000) 2022-09-10T06:43:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:15 smithi110 ceph-mon[97633]: osdmap e351: 12 total, 12 up, 12 in 2022-09-10T06:43:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:15 smithi110 ceph-mon[97633]: pgmap v4293: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:43:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:15 smithi104 ceph-mon[97044]: osdmap e351: 12 total, 12 up, 12 in 2022-09-10T06:43:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:15 smithi104 ceph-mon[97044]: pgmap v4293: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T06:43:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:15.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:15.506437+0000) 2022-09-10T06:43:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:15.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:15.909605+0000) 2022-09-10T06:43:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:16.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:16.506628+0000) 2022-09-10T06:43:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:16.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:16.909855+0000) 2022-09-10T06:43:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:17 smithi110 ceph-mon[97633]: pgmap v4294: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:43:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:17 smithi104 ceph-mon[97044]: pgmap v4294: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:43:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:17.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:17.506850+0000) 2022-09-10T06:43:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:17.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:17.910064+0000) 2022-09-10T06:43:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:18.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:18.507154+0000) 2022-09-10T06:43:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:18.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:18.910264+0000) 2022-09-10T06:43:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:19 smithi110 ceph-mon[97633]: pgmap v4295: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:43:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:19 smithi104 ceph-mon[97044]: pgmap v4295: 129 pgs: 129 active+clean; 8.1 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:43:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:19.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:19.507380+0000) 2022-09-10T06:43:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:19.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:19.910461+0000) 2022-09-10T06:43:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:20.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:20.507617+0000) 2022-09-10T06:43:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:20.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:20.910633+0000) 2022-09-10T06:43:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:21 smithi110 ceph-mon[97633]: pgmap v4296: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:43:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:21 smithi104 ceph-mon[97044]: pgmap v4296: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:43:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:21.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:21.507911+0000) 2022-09-10T06:43:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:21.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:21.910881+0000) 2022-09-10T06:43:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:22.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:22.508184+0000) 2022-09-10T06:43:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:22.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:22.911002+0000) 2022-09-10T06:43:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:23 smithi110 ceph-mon[97633]: pgmap v4297: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:43:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:23 smithi104 ceph-mon[97044]: pgmap v4297: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:43:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:23.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:23.508344+0000) 2022-09-10T06:43:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:23.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:23.911177+0000) 2022-09-10T06:43:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:24.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:24.508565+0000) 2022-09-10T06:43:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:24.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:24.911365+0000) 2022-09-10T06:43:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:25 smithi110 ceph-mon[97633]: pgmap v4298: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:43:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:25 smithi104 ceph-mon[97044]: pgmap v4298: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T06:43:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:25.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:25.508782+0000) 2022-09-10T06:43:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:25.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:25.911571+0000) 2022-09-10T06:43:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:26.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:26.509002+0000) 2022-09-10T06:43:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:26.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:26.911791+0000) 2022-09-10T06:43:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:27 smithi110 ceph-mon[97633]: pgmap v4299: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:43:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:27 smithi104 ceph-mon[97044]: pgmap v4299: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T06:43:27.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:27.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:27.509211+0000) 2022-09-10T06:43:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:27.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:27.912036+0000) 2022-09-10T06:43:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:28.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:28.509434+0000) 2022-09-10T06:43:29.273 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:28.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:28.912265+0000) 2022-09-10T06:43:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:43:29.272+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:43:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:29.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:29.509660+0000) 2022-09-10T06:43:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:29 smithi110 ceph-mon[97633]: pgmap v4300: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:29 smithi104 ceph-mon[97044]: pgmap v4300: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:30.314 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:29.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:29.912517+0000) 2022-09-10T06:43:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:30.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:30.509902+0000) 2022-09-10T06:43:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:30.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:30.912677+0000) 2022-09-10T06:43:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:31 smithi110 ceph-mon[97633]: pgmap v4301: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:31 smithi104 ceph-mon[97044]: pgmap v4301: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:31.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:31.510065+0000) 2022-09-10T06:43:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:31.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:31.912793+0000) 2022-09-10T06:43:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:32.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:32.510376+0000) 2022-09-10T06:43:33.237 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:32.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:32.912963+0000) 2022-09-10T06:43:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:33 smithi110 ceph-mon[97633]: pgmap v4302: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:33 smithi104 ceph-mon[97044]: pgmap v4302: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:33.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:33.510637+0000) 2022-09-10T06:43:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:33.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:33.913188+0000) 2022-09-10T06:43:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:34.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:34.510793+0000) 2022-09-10T06:43:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:34.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:34.913429+0000) 2022-09-10T06:43:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:35 smithi110 ceph-mon[97633]: pgmap v4303: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:35 smithi104 ceph-mon[97044]: pgmap v4303: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:35.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:35.510974+0000) 2022-09-10T06:43:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:35.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:35.913673+0000) 2022-09-10T06:43:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:36.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:36.511199+0000) 2022-09-10T06:43:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:36.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:36.913899+0000) 2022-09-10T06:43:37.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:37 smithi110 ceph-mon[97633]: pgmap v4304: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:37 smithi104 ceph-mon[97044]: pgmap v4304: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:37.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:37.511363+0000) 2022-09-10T06:43:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:37.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:37.914124+0000) 2022-09-10T06:43:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:38.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:38.511487+0000) 2022-09-10T06:43:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:38.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:38.914354+0000) 2022-09-10T06:43:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:39 smithi110 ceph-mon[97633]: pgmap v4305: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:39 smithi104 ceph-mon[97044]: pgmap v4305: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:39.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:39.511709+0000) 2022-09-10T06:43:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:39.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:39.914520+0000) 2022-09-10T06:43:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:40.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:40.511906+0000) 2022-09-10T06:43:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:40.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:40.914733+0000) 2022-09-10T06:43:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:41 smithi110 ceph-mon[97633]: pgmap v4306: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:41 smithi104 ceph-mon[97044]: pgmap v4306: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:41.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:41.512176+0000) 2022-09-10T06:43:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:41.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:41.914952+0000) 2022-09-10T06:43:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:42.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:42.512412+0000) 2022-09-10T06:43:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:42.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:42.915233+0000) 2022-09-10T06:43:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:43 smithi110 ceph-mon[97633]: pgmap v4307: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:43 smithi104 ceph-mon[97044]: pgmap v4307: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:43.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:43.512573+0000) 2022-09-10T06:43:44.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:43.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:43.915399+0000) 2022-09-10T06:43:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:44.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:44.512797+0000) 2022-09-10T06:43:44.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:43:44.287+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:43:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:43:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:43:44.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:43:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:43:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:43:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:43:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:43:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:43:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:44.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:44.915664+0000) 2022-09-10T06:43:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:45 smithi110 ceph-mon[97633]: pgmap v4308: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:43:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:45 smithi104 ceph-mon[97044]: pgmap v4308: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:43:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:45.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:45.513064+0000) 2022-09-10T06:43:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:45.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:45.915882+0000) 2022-09-10T06:43:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:46.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:46.513274+0000) 2022-09-10T06:43:47.313 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:46.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:46.916119+0000) 2022-09-10T06:43:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:47 smithi110 ceph-mon[97633]: pgmap v4309: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:47 smithi104 ceph-mon[97044]: pgmap v4309: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:47.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:47.513524+0000) 2022-09-10T06:43:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:47.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:47.916383+0000) 2022-09-10T06:43:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:48.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:48.513727+0000) 2022-09-10T06:43:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:48.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:48.916610+0000) 2022-09-10T06:43:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:49 smithi110 ceph-mon[97633]: pgmap v4310: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:43:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:49 smithi104 ceph-mon[97044]: pgmap v4310: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:43:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:49.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:49.514008+0000) 2022-09-10T06:43:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:49.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:49.916833+0000) 2022-09-10T06:43:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:50.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:50.514237+0000) 2022-09-10T06:43:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:50.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:50.917079+0000) 2022-09-10T06:43:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:51 smithi110 ceph-mon[97633]: pgmap v4311: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:51 smithi104 ceph-mon[97044]: pgmap v4311: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:43:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:51.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:51.514399+0000) 2022-09-10T06:43:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:51.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:51.917221+0000) 2022-09-10T06:43:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:52.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:52.514600+0000) 2022-09-10T06:43:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:52.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:52.917467+0000) 2022-09-10T06:43:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:53 smithi110 ceph-mon[97633]: pgmap v4312: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:43:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:53 smithi104 ceph-mon[97044]: pgmap v4312: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:43:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:53.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:53.514774+0000) 2022-09-10T06:43:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:53.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:53.917659+0000) 2022-09-10T06:43:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:54.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:54.514977+0000) 2022-09-10T06:43:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:54.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:54.917824+0000) 2022-09-10T06:43:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:55 smithi110 ceph-mon[97633]: pgmap v4313: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:43:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:55 smithi104 ceph-mon[97044]: pgmap v4313: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:43:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:55.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:55.515191+0000) 2022-09-10T06:43:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:55.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:55.918026+0000) 2022-09-10T06:43:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:56.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:56.515446+0000) 2022-09-10T06:43:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:56.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:56.918223+0000) 2022-09-10T06:43:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:57 smithi110 ceph-mon[97633]: pgmap v4314: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:43:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:57 smithi104 ceph-mon[97044]: pgmap v4314: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:43:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:57.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:57.515640+0000) 2022-09-10T06:43:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:57.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:57.918470+0000) 2022-09-10T06:43:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:58.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:58.515868+0000) 2022-09-10T06:43:59.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:58.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:58.918640+0000) 2022-09-10T06:43:59.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:43:59.297+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:43:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:43:59.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:59.516162+0000) 2022-09-10T06:43:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:43:59 smithi110 ceph-mon[97633]: pgmap v4315: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:43:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:43:59 smithi104 ceph-mon[97044]: pgmap v4315: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:43:59.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:43:59.918815+0000) 2022-09-10T06:44:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:00.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:00.516351+0000) 2022-09-10T06:44:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:00.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:00.919068+0000) 2022-09-10T06:44:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:01 smithi110 ceph-mon[97633]: pgmap v4316: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:01 smithi104 ceph-mon[97044]: pgmap v4316: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:01.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:01.516598+0000) 2022-09-10T06:44:02.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:01.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:01.919300+0000) 2022-09-10T06:44:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:02.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:02.516871+0000) 2022-09-10T06:44:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:02.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:02.919424+0000) 2022-09-10T06:44:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:03 smithi110 ceph-mon[97633]: pgmap v4317: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:03 smithi104 ceph-mon[97044]: pgmap v4317: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:03.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:03.517023+0000) 2022-09-10T06:44:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:03.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:03.919660+0000) 2022-09-10T06:44:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:04.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:04.517252+0000) 2022-09-10T06:44:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:04.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:04.919939+0000) 2022-09-10T06:44:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:05 smithi110 ceph-mon[97633]: pgmap v4318: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T06:44:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:05 smithi104 ceph-mon[97044]: pgmap v4318: 129 pgs: 129 active+clean; 8.1 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T06:44:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:05.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:05.517493+0000) 2022-09-10T06:44:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:05.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:05.920176+0000) 2022-09-10T06:44:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:06.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:06.517642+0000) 2022-09-10T06:44:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:06.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:06.920357+0000) 2022-09-10T06:44:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:07 smithi110 ceph-mon[97633]: pgmap v4319: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T06:44:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:07 smithi104 ceph-mon[97044]: pgmap v4319: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T06:44:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:07.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:07.517886+0000) 2022-09-10T06:44:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:07.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:07.920578+0000) 2022-09-10T06:44:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:08 smithi110 ceph-mon[97633]: osdmap e352: 12 total, 12 up, 12 in 2022-09-10T06:44:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:08 smithi104 ceph-mon[97044]: osdmap e352: 12 total, 12 up, 12 in 2022-09-10T06:44:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:08.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:08.518167+0000) 2022-09-10T06:44:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:08.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:08.920795+0000) 2022-09-10T06:44:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:09 smithi110 ceph-mon[97633]: pgmap v4321: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:44:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:09 smithi104 ceph-mon[97044]: pgmap v4321: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:44:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:09.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:09.518328+0000) 2022-09-10T06:44:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:09.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:09.921015+0000) 2022-09-10T06:44:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:10.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:10.518553+0000) 2022-09-10T06:44:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:10.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:10.921208+0000) 2022-09-10T06:44:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:11 smithi110 ceph-mon[97633]: pgmap v4322: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:44:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:11 smithi104 ceph-mon[97044]: pgmap v4322: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:44:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:11.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:11.518763+0000) 2022-09-10T06:44:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:11.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:11.921395+0000) 2022-09-10T06:44:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:12.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:12.518971+0000) 2022-09-10T06:44:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:12.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:12.921638+0000) 2022-09-10T06:44:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:13 smithi110 ceph-mon[97633]: pgmap v4323: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:44:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:13 smithi104 ceph-mon[97044]: pgmap v4323: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:44:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:13.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:13.519206+0000) 2022-09-10T06:44:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:13.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:13.921794+0000) 2022-09-10T06:44:14.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:44:14.313+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:44:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:14.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:14.519419+0000) 2022-09-10T06:44:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:14.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:14.922020+0000) 2022-09-10T06:44:15.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:15 smithi110 ceph-mon[97633]: osdmap e353: 12 total, 12 up, 12 in 2022-09-10T06:44:15.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:15 smithi110 ceph-mon[97633]: pgmap v4325: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:44:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:15 smithi104 ceph-mon[97044]: osdmap e353: 12 total, 12 up, 12 in 2022-09-10T06:44:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:15 smithi104 ceph-mon[97044]: pgmap v4325: 129 pgs: 129 active+clean; 8.1 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:44:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:15.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:15.519688+0000) 2022-09-10T06:44:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:15.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:15.922252+0000) 2022-09-10T06:44:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:16.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:16.519865+0000) 2022-09-10T06:44:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:16.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:16.922395+0000) 2022-09-10T06:44:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:17 smithi110 ceph-mon[97633]: pgmap v4326: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:44:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:17 smithi104 ceph-mon[97044]: pgmap v4326: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:44:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:17.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:17.520080+0000) 2022-09-10T06:44:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:17.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:17.922608+0000) 2022-09-10T06:44:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:18.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:18.520325+0000) 2022-09-10T06:44:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:18.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:18.922749+0000) 2022-09-10T06:44:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:19 smithi110 ceph-mon[97633]: pgmap v4327: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:44:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:19 smithi104 ceph-mon[97044]: pgmap v4327: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:44:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:19.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:19.520573+0000) 2022-09-10T06:44:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:19.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:19.922896+0000) 2022-09-10T06:44:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:20.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:20.520814+0000) 2022-09-10T06:44:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:20.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:20.923058+0000) 2022-09-10T06:44:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:21 smithi110 ceph-mon[97633]: pgmap v4328: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:44:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:21 smithi104 ceph-mon[97044]: pgmap v4328: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:44:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:21.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:21.521068+0000) 2022-09-10T06:44:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:21.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:21.923268+0000) 2022-09-10T06:44:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:22.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:22.521276+0000) 2022-09-10T06:44:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:22.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:22.923506+0000) 2022-09-10T06:44:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:23 smithi110 ceph-mon[97633]: pgmap v4329: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:44:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:23 smithi104 ceph-mon[97044]: pgmap v4329: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:44:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:23.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:23.521443+0000) 2022-09-10T06:44:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:23.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:23.923705+0000) 2022-09-10T06:44:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:24.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:24.521732+0000) 2022-09-10T06:44:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:24.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:24.923955+0000) 2022-09-10T06:44:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:25 smithi110 ceph-mon[97633]: pgmap v4330: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 13 op/s 2022-09-10T06:44:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:25 smithi104 ceph-mon[97044]: pgmap v4330: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 13 op/s 2022-09-10T06:44:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:25.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:25.521966+0000) 2022-09-10T06:44:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:25.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:25.924185+0000) 2022-09-10T06:44:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:26.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:26.522230+0000) 2022-09-10T06:44:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:26.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:26.924352+0000) 2022-09-10T06:44:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:27 smithi110 ceph-mon[97633]: pgmap v4331: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:27 smithi104 ceph-mon[97044]: pgmap v4331: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:27.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:27.522416+0000) 2022-09-10T06:44:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:27.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:27.924494+0000) 2022-09-10T06:44:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:28.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:28.522622+0000) 2022-09-10T06:44:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:28.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:28.924731+0000) 2022-09-10T06:44:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:29 smithi110 ceph-mon[97633]: pgmap v4332: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:29 smithi104 ceph-mon[97044]: pgmap v4332: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:29.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:44:29.328+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:44:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:29.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:29.522865+0000) 2022-09-10T06:44:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:29.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:29.924910+0000) 2022-09-10T06:44:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:30.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:30.523170+0000) 2022-09-10T06:44:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:30.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:30.925108+0000) 2022-09-10T06:44:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:31 smithi110 ceph-mon[97633]: pgmap v4333: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:31 smithi104 ceph-mon[97044]: pgmap v4333: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:31.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:31.523417+0000) 2022-09-10T06:44:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:31.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:31.925284+0000) 2022-09-10T06:44:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:32.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:32.523643+0000) 2022-09-10T06:44:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:32.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:32.925472+0000) 2022-09-10T06:44:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:33 smithi110 ceph-mon[97633]: pgmap v4334: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:33 smithi104 ceph-mon[97044]: pgmap v4334: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:33.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:33.523805+0000) 2022-09-10T06:44:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:33.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:33.925710+0000) 2022-09-10T06:44:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:34.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:34.524043+0000) 2022-09-10T06:44:35.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:34.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:34.925948+0000) 2022-09-10T06:44:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:35 smithi110 ceph-mon[97633]: pgmap v4335: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:44:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:35 smithi104 ceph-mon[97044]: pgmap v4335: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:44:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:35.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:35.524256+0000) 2022-09-10T06:44:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:35.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:35.926084+0000) 2022-09-10T06:44:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:36.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:36.524474+0000) 2022-09-10T06:44:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:36.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:36.926345+0000) 2022-09-10T06:44:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:37 smithi110 ceph-mon[97633]: pgmap v4336: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:37 smithi104 ceph-mon[97044]: pgmap v4336: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:37.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:37.524699+0000) 2022-09-10T06:44:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:37.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:37.926537+0000) 2022-09-10T06:44:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:38.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:38.524905+0000) 2022-09-10T06:44:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:38.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:38.926757+0000) 2022-09-10T06:44:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:39 smithi110 ceph-mon[97633]: pgmap v4337: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:39 smithi104 ceph-mon[97044]: pgmap v4337: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:39.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:39.525196+0000) 2022-09-10T06:44:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:39.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:39.926966+0000) 2022-09-10T06:44:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:40.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:40.525389+0000) 2022-09-10T06:44:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:40.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:40.927172+0000) 2022-09-10T06:44:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:41 smithi110 ceph-mon[97633]: pgmap v4338: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:41 smithi104 ceph-mon[97044]: pgmap v4338: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:41.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:41.525535+0000) 2022-09-10T06:44:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:41.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:41.927409+0000) 2022-09-10T06:44:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:42.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:42.525715+0000) 2022-09-10T06:44:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:42.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:42.927611+0000) 2022-09-10T06:44:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:43 smithi110 ceph-mon[97633]: pgmap v4339: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:43 smithi104 ceph-mon[97044]: pgmap v4339: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:43.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:43.525869+0000) 2022-09-10T06:44:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:43.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:43.927802+0000) 2022-09-10T06:44:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:44:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:44:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:44:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:44:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:44:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:44:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:44:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:44:44.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:44:44.340+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:44:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:44.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:44.526021+0000) 2022-09-10T06:44:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:44.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:44.928051+0000) 2022-09-10T06:44:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:45 smithi104 ceph-mon[97044]: pgmap v4340: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:44:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:45.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:45.526229+0000) 2022-09-10T06:44:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:45 smithi110 ceph-mon[97633]: pgmap v4340: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:44:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:45.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:45.928217+0000) 2022-09-10T06:44:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:46.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:46.526480+0000) 2022-09-10T06:44:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:46.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:46.928405+0000) 2022-09-10T06:44:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:47 smithi104 ceph-mon[97044]: pgmap v4341: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:47.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:47.526701+0000) 2022-09-10T06:44:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:47 smithi110 ceph-mon[97633]: pgmap v4341: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:47.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:47.928604+0000) 2022-09-10T06:44:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:48.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:48.526958+0000) 2022-09-10T06:44:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:48.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:48.928807+0000) 2022-09-10T06:44:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:49 smithi104 ceph-mon[97044]: pgmap v4342: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:49.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:49.527209+0000) 2022-09-10T06:44:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:49 smithi110 ceph-mon[97633]: pgmap v4342: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:49.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:49.929022+0000) 2022-09-10T06:44:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:50.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:50.527424+0000) 2022-09-10T06:44:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:50.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:50.929237+0000) 2022-09-10T06:44:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:51 smithi104 ceph-mon[97044]: pgmap v4343: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:51.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:51.527681+0000) 2022-09-10T06:44:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:51 smithi110 ceph-mon[97633]: pgmap v4343: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:44:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:51.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:51.929497+0000) 2022-09-10T06:44:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:52.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:52.527856+0000) 2022-09-10T06:44:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:52.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:52.929746+0000) 2022-09-10T06:44:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:53 smithi104 ceph-mon[97044]: pgmap v4344: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:53.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:53.528027+0000) 2022-09-10T06:44:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:53 smithi110 ceph-mon[97633]: pgmap v4344: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:53.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:53.929885+0000) 2022-09-10T06:44:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:54.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:54.528298+0000) 2022-09-10T06:44:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:54.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:54.930077+0000) 2022-09-10T06:44:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:55 smithi110 ceph-mon[97633]: pgmap v4345: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:44:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:55 smithi104 ceph-mon[97044]: pgmap v4345: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:44:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:55.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:55.528530+0000) 2022-09-10T06:44:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:55.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:55.930279+0000) 2022-09-10T06:44:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:56.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:56.528797+0000) 2022-09-10T06:44:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:56.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:56.930553+0000) 2022-09-10T06:44:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:57 smithi110 ceph-mon[97633]: pgmap v4346: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:57 smithi104 ceph-mon[97044]: pgmap v4346: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:44:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:57.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:57.529019+0000) 2022-09-10T06:44:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:57.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:57.930816+0000) 2022-09-10T06:44:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:58.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:58.529214+0000) 2022-09-10T06:44:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:58.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:58.930966+0000) 2022-09-10T06:44:59.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:44:59 smithi110 ceph-mon[97633]: pgmap v4347: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:44:59 smithi104 ceph-mon[97044]: pgmap v4347: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:44:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:44:59.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:59.529398+0000) 2022-09-10T06:44:59.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:44:59.346+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:45:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:44:59.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:44:59.931205+0000) 2022-09-10T06:45:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:00.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:00.529618+0000) 2022-09-10T06:45:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:00.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:00.931363+0000) 2022-09-10T06:45:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:01 smithi104 ceph-mon[97044]: pgmap v4348: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:01.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:01.529866+0000) 2022-09-10T06:45:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:01 smithi110 ceph-mon[97633]: pgmap v4348: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:01.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:01.931553+0000) 2022-09-10T06:45:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:02.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:02.530069+0000) 2022-09-10T06:45:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:02.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:02.931776+0000) 2022-09-10T06:45:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:03 smithi104 ceph-mon[97044]: pgmap v4349: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:03.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:03.530304+0000) 2022-09-10T06:45:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:03 smithi110 ceph-mon[97633]: pgmap v4349: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:03.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:03.932022+0000) 2022-09-10T06:45:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:04.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:04.530556+0000) 2022-09-10T06:45:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:04.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:04.932267+0000) 2022-09-10T06:45:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:05 smithi110 ceph-mon[97633]: pgmap v4350: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T06:45:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:05 smithi104 ceph-mon[97044]: pgmap v4350: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T06:45:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:05.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:05.530772+0000) 2022-09-10T06:45:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:05.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:05.932536+0000) 2022-09-10T06:45:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:06.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:06.530995+0000) 2022-09-10T06:45:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:06.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:06.932769+0000) 2022-09-10T06:45:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:07 smithi110 ceph-mon[97633]: pgmap v4351: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T06:45:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:07 smithi104 ceph-mon[97044]: pgmap v4351: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T06:45:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:07.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:07.531207+0000) 2022-09-10T06:45:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:07.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:07.932983+0000) 2022-09-10T06:45:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:08 smithi104 ceph-mon[97044]: osdmap e354: 12 total, 12 up, 12 in 2022-09-10T06:45:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:08.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:08.531420+0000) 2022-09-10T06:45:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:08 smithi110 ceph-mon[97633]: osdmap e354: 12 total, 12 up, 12 in 2022-09-10T06:45:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:08.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:08.933210+0000) 2022-09-10T06:45:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:09 smithi104 ceph-mon[97044]: pgmap v4353: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:45:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:09.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:09.531633+0000) 2022-09-10T06:45:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:09 smithi110 ceph-mon[97633]: pgmap v4353: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:45:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:09.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:09.933459+0000) 2022-09-10T06:45:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:10.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:10.531865+0000) 2022-09-10T06:45:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:10.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:10.933612+0000) 2022-09-10T06:45:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:11 smithi104 ceph-mon[97044]: pgmap v4354: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:45:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:11.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:11.532143+0000) 2022-09-10T06:45:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:11 smithi110 ceph-mon[97633]: pgmap v4354: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:45:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:11.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:11.933831+0000) 2022-09-10T06:45:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:12.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:12.532348+0000) 2022-09-10T06:45:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:12.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:12.934032+0000) 2022-09-10T06:45:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:13 smithi104 ceph-mon[97044]: pgmap v4355: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:45:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:13.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:13.532485+0000) 2022-09-10T06:45:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:13 smithi110 ceph-mon[97633]: pgmap v4355: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:45:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:13.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:13.934278+0000) 2022-09-10T06:45:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:14.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:14.532655+0000) 2022-09-10T06:45:14.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:45:14.361+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:45:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:14.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:14.934523+0000) 2022-09-10T06:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:15 smithi110 ceph-mon[97633]: osdmap e355: 12 total, 12 up, 12 in 2022-09-10T06:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:15 smithi110 ceph-mon[97633]: pgmap v4357: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:45:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:15 smithi104 ceph-mon[97044]: osdmap e355: 12 total, 12 up, 12 in 2022-09-10T06:45:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:15 smithi104 ceph-mon[97044]: pgmap v4357: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:45:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:15.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:15.532880+0000) 2022-09-10T06:45:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:15.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:15.934770+0000) 2022-09-10T06:45:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:16.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:16.533066+0000) 2022-09-10T06:45:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:16.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:16.935016+0000) 2022-09-10T06:45:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:17 smithi104 ceph-mon[97044]: pgmap v4358: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:45:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:17.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:17.533273+0000) 2022-09-10T06:45:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:17 smithi110 ceph-mon[97633]: pgmap v4358: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:45:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:17.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:17.935268+0000) 2022-09-10T06:45:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:18.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:18.533458+0000) 2022-09-10T06:45:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:18.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:18.935507+0000) 2022-09-10T06:45:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:19 smithi104 ceph-mon[97044]: pgmap v4359: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:45:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:19.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:19.533667+0000) 2022-09-10T06:45:19.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:19 smithi110 ceph-mon[97633]: pgmap v4359: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:45:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:19.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:19.935695+0000) 2022-09-10T06:45:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:20.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:20.533908+0000) 2022-09-10T06:45:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:20.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:20.935835+0000) 2022-09-10T06:45:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:21 smithi104 ceph-mon[97044]: pgmap v4360: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:45:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:21.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:21.534071+0000) 2022-09-10T06:45:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:21 smithi110 ceph-mon[97633]: pgmap v4360: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:45:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:21.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:21.936061+0000) 2022-09-10T06:45:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:22.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:22.534290+0000) 2022-09-10T06:45:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:22.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:22.936273+0000) 2022-09-10T06:45:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:23 smithi104 ceph-mon[97044]: pgmap v4361: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:45:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:23.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:23.534534+0000) 2022-09-10T06:45:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:23 smithi110 ceph-mon[97633]: pgmap v4361: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:45:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:23.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:23.936443+0000) 2022-09-10T06:45:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:24.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:24.534779+0000) 2022-09-10T06:45:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:24.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:24.936683+0000) 2022-09-10T06:45:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:25 smithi110 ceph-mon[97633]: pgmap v4362: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 13 op/s 2022-09-10T06:45:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:25 smithi104 ceph-mon[97044]: pgmap v4362: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 13 op/s 2022-09-10T06:45:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:25.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:25.535012+0000) 2022-09-10T06:45:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:25.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:25.936876+0000) 2022-09-10T06:45:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:26.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:26.535259+0000) 2022-09-10T06:45:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:26.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:26.937168+0000) 2022-09-10T06:45:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:27 smithi104 ceph-mon[97044]: pgmap v4363: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:45:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:27.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:27.535477+0000) 2022-09-10T06:45:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:27 smithi110 ceph-mon[97633]: pgmap v4363: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:45:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:27.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:27.937310+0000) 2022-09-10T06:45:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:28.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:28.535705+0000) 2022-09-10T06:45:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:28.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:28.937541+0000) 2022-09-10T06:45:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:29 smithi104 ceph-mon[97044]: pgmap v4364: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:29.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:45:29.377+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:45:29.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:29.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:29.535922+0000) 2022-09-10T06:45:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:29 smithi110 ceph-mon[97633]: pgmap v4364: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:29.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:29.937718+0000) 2022-09-10T06:45:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:30.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:30.536161+0000) 2022-09-10T06:45:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:30.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:30.937907+0000) 2022-09-10T06:45:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:31 smithi104 ceph-mon[97044]: pgmap v4365: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:31.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:31.536318+0000) 2022-09-10T06:45:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:31 smithi110 ceph-mon[97633]: pgmap v4365: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:31.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:31.938089+0000) 2022-09-10T06:45:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:32.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:32.536571+0000) 2022-09-10T06:45:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:32.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:32.938319+0000) 2022-09-10T06:45:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:33 smithi104 ceph-mon[97044]: pgmap v4366: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:33.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:33.536846+0000) 2022-09-10T06:45:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:33 smithi110 ceph-mon[97633]: pgmap v4366: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:33.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:33.938520+0000) 2022-09-10T06:45:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:34.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:34.537117+0000) 2022-09-10T06:45:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:34.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:34.938767+0000) 2022-09-10T06:45:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:35 smithi110 ceph-mon[97633]: pgmap v4367: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:45:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:35 smithi104 ceph-mon[97044]: pgmap v4367: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:45:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:35.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:35.537348+0000) 2022-09-10T06:45:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:35.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:35.939066+0000) 2022-09-10T06:45:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:36.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:36.537566+0000) 2022-09-10T06:45:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:36.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:36.939218+0000) 2022-09-10T06:45:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:37 smithi104 ceph-mon[97044]: pgmap v4368: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:45:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:37.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:37.537799+0000) 2022-09-10T06:45:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:37 smithi110 ceph-mon[97633]: pgmap v4368: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:45:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:37.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:37.939389+0000) 2022-09-10T06:45:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:38.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:38.538039+0000) 2022-09-10T06:45:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:38.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:38.939607+0000) 2022-09-10T06:45:39.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:39 smithi104 ceph-mon[97044]: pgmap v4369: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:39.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:39.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:39.538288+0000) 2022-09-10T06:45:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:39 smithi110 ceph-mon[97633]: pgmap v4369: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:39.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:39.939810+0000) 2022-09-10T06:45:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:40.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:40.538490+0000) 2022-09-10T06:45:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:40.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:40.940046+0000) 2022-09-10T06:45:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:41 smithi104 ceph-mon[97044]: pgmap v4370: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:41.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:41.538653+0000) 2022-09-10T06:45:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:41 smithi110 ceph-mon[97633]: pgmap v4370: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:41.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:41.940245+0000) 2022-09-10T06:45:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:42.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:42.538912+0000) 2022-09-10T06:45:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:42.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:42.940463+0000) 2022-09-10T06:45:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:43 smithi104 ceph-mon[97044]: pgmap v4371: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:43.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:43.539173+0000) 2022-09-10T06:45:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:43 smithi110 ceph-mon[97633]: pgmap v4371: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:43.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:43.940674+0000) 2022-09-10T06:45:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:45:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:45:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:45:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:45:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:44.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:44.539353+0000) 2022-09-10T06:45:44.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:45:44.393+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:45:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:45:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:45:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:45:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:45:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:44.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:44.940913+0000) 2022-09-10T06:45:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:45 smithi104 ceph-mon[97044]: pgmap v4372: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:45:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:45.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:45.539598+0000) 2022-09-10T06:45:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:45 smithi110 ceph-mon[97633]: pgmap v4372: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:45:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:45.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:45.941128+0000) 2022-09-10T06:45:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:46.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:46.539842+0000) 2022-09-10T06:45:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:46.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:46.941364+0000) 2022-09-10T06:45:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:47 smithi104 ceph-mon[97044]: pgmap v4373: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T06:45:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:47.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:47.540064+0000) 2022-09-10T06:45:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:47 smithi110 ceph-mon[97633]: pgmap v4373: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T06:45:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:47.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:47.941595+0000) 2022-09-10T06:45:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:48.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:48.540260+0000) 2022-09-10T06:45:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:48.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:48.941875+0000) 2022-09-10T06:45:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:49 smithi104 ceph-mon[97044]: pgmap v4374: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:45:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:49.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:49.540488+0000) 2022-09-10T06:45:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:49 smithi110 ceph-mon[97633]: pgmap v4374: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:45:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:49.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:49.942146+0000) 2022-09-10T06:45:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:50.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:50.540689+0000) 2022-09-10T06:45:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:50.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:50.942386+0000) 2022-09-10T06:45:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:51 smithi104 ceph-mon[97044]: pgmap v4375: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:51.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:51.540947+0000) 2022-09-10T06:45:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:51 smithi110 ceph-mon[97633]: pgmap v4375: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:45:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:51.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:51.942621+0000) 2022-09-10T06:45:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:52.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:52.541161+0000) 2022-09-10T06:45:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:52.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:52.942832+0000) 2022-09-10T06:45:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:53 smithi104 ceph-mon[97044]: pgmap v4376: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:45:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:53.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:53.541400+0000) 2022-09-10T06:45:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:53 smithi110 ceph-mon[97633]: pgmap v4376: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:45:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:53.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:53.942995+0000) 2022-09-10T06:45:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:54.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:54.541628+0000) 2022-09-10T06:45:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:54.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:54.943226+0000) 2022-09-10T06:45:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:55 smithi110 ceph-mon[97633]: pgmap v4377: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:45:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:55 smithi104 ceph-mon[97044]: pgmap v4377: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:45:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:55.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:55.541903+0000) 2022-09-10T06:45:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:55.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:55.943458+0000) 2022-09-10T06:45:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:56.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:56.542186+0000) 2022-09-10T06:45:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:56.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:56.943666+0000) 2022-09-10T06:45:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:57 smithi104 ceph-mon[97044]: pgmap v4378: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:45:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:57.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:57.542394+0000) 2022-09-10T06:45:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:57 smithi110 ceph-mon[97633]: pgmap v4378: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:45:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:57.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:57.943951+0000) 2022-09-10T06:45:58.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:58.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:58.542565+0000) 2022-09-10T06:45:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:58.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:58.944148+0000) 2022-09-10T06:45:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:45:59 smithi104 ceph-mon[97044]: pgmap v4379: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:45:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:45:59.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:59.542791+0000) 2022-09-10T06:45:59.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:45:59.404+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:45:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:45:59 smithi110 ceph-mon[97633]: pgmap v4379: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:45:59.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:45:59.944304+0000) 2022-09-10T06:46:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:00.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:00.543065+0000) 2022-09-10T06:46:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:00.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:00.944503+0000) 2022-09-10T06:46:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:01 smithi104 ceph-mon[97044]: pgmap v4380: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:01.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:01.543354+0000) 2022-09-10T06:46:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:01 smithi110 ceph-mon[97633]: pgmap v4380: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:01.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:01.944725+0000) 2022-09-10T06:46:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:02.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:02.543496+0000) 2022-09-10T06:46:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:02.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:02.944968+0000) 2022-09-10T06:46:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:03 smithi104 ceph-mon[97044]: pgmap v4381: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:03.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:03.543644+0000) 2022-09-10T06:46:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:03 smithi110 ceph-mon[97633]: pgmap v4381: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:03.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:03.945240+0000) 2022-09-10T06:46:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:04.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:04.543849+0000) 2022-09-10T06:46:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:04.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:04.945395+0000) 2022-09-10T06:46:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:05 smithi110 ceph-mon[97633]: pgmap v4382: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T06:46:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:05 smithi104 ceph-mon[97044]: pgmap v4382: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T06:46:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:05.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:05.544126+0000) 2022-09-10T06:46:06.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:05.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:05.945632+0000) 2022-09-10T06:46:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:06.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:06.544296+0000) 2022-09-10T06:46:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:06.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:06.945869+0000) 2022-09-10T06:46:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:07 smithi104 ceph-mon[97044]: pgmap v4383: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T06:46:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:07.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:07.544443+0000) 2022-09-10T06:46:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:07 smithi110 ceph-mon[97633]: pgmap v4383: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T06:46:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:07.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:07.946168+0000) 2022-09-10T06:46:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:08 smithi104 ceph-mon[97044]: osdmap e356: 12 total, 12 up, 12 in 2022-09-10T06:46:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:08.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:08.544684+0000) 2022-09-10T06:46:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:08 smithi110 ceph-mon[97633]: osdmap e356: 12 total, 12 up, 12 in 2022-09-10T06:46:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:08.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:08.946376+0000) 2022-09-10T06:46:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:09 smithi104 ceph-mon[97044]: pgmap v4385: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:46:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:09.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:09.544938+0000) 2022-09-10T06:46:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:09 smithi110 ceph-mon[97633]: pgmap v4385: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:46:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:09.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:09.946602+0000) 2022-09-10T06:46:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:10.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:10.545137+0000) 2022-09-10T06:46:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:10.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:10.946833+0000) 2022-09-10T06:46:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:11 smithi104 ceph-mon[97044]: pgmap v4386: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:46:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:11.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:11.545397+0000) 2022-09-10T06:46:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:11 smithi110 ceph-mon[97633]: pgmap v4386: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:46:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:11.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:11.947056+0000) 2022-09-10T06:46:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:12.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:12.545592+0000) 2022-09-10T06:46:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:12.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:12.947334+0000) 2022-09-10T06:46:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:13 smithi104 ceph-mon[97044]: pgmap v4387: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:46:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:13.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:13.545774+0000) 2022-09-10T06:46:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:13 smithi110 ceph-mon[97633]: pgmap v4387: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:46:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:13.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:13.947517+0000) 2022-09-10T06:46:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:14.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:14.545979+0000) 2022-09-10T06:46:14.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:46:14.420+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:46:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:14.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:14.947745+0000) 2022-09-10T06:46:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:15 smithi110 ceph-mon[97633]: osdmap e357: 12 total, 12 up, 12 in 2022-09-10T06:46:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:15 smithi110 ceph-mon[97633]: pgmap v4389: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:46:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:15 smithi104 ceph-mon[97044]: osdmap e357: 12 total, 12 up, 12 in 2022-09-10T06:46:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:15 smithi104 ceph-mon[97044]: pgmap v4389: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:46:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:15.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:15.546173+0000) 2022-09-10T06:46:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:15.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:15.947978+0000) 2022-09-10T06:46:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:16.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:16.546383+0000) 2022-09-10T06:46:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:16.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:16.948191+0000) 2022-09-10T06:46:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:17 smithi104 ceph-mon[97044]: pgmap v4390: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:46:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:17.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:17.546583+0000) 2022-09-10T06:46:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:17 smithi110 ceph-mon[97633]: pgmap v4390: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:46:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:17.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:17.948366+0000) 2022-09-10T06:46:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:18.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:18.546795+0000) 2022-09-10T06:46:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:18.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:18.948566+0000) 2022-09-10T06:46:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:19 smithi104 ceph-mon[97044]: pgmap v4391: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:46:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:19.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:19.546953+0000) 2022-09-10T06:46:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:19 smithi110 ceph-mon[97633]: pgmap v4391: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:46:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:19.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:19.948760+0000) 2022-09-10T06:46:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:20.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:20.547195+0000) 2022-09-10T06:46:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:20.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:20.948993+0000) 2022-09-10T06:46:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:21 smithi104 ceph-mon[97044]: pgmap v4392: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:46:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:21.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:21.547410+0000) 2022-09-10T06:46:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:21 smithi110 ceph-mon[97633]: pgmap v4392: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:46:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:21.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:21.949237+0000) 2022-09-10T06:46:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:22.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:22.547623+0000) 2022-09-10T06:46:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:22.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:22.949473+0000) 2022-09-10T06:46:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:23 smithi104 ceph-mon[97044]: pgmap v4393: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:46:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:23.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:23.547774+0000) 2022-09-10T06:46:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:23 smithi110 ceph-mon[97633]: pgmap v4393: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:46:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:23.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:23.949702+0000) 2022-09-10T06:46:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:24.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:24.548017+0000) 2022-09-10T06:46:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:24.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:24.949853+0000) 2022-09-10T06:46:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:25 smithi110 ceph-mon[97633]: pgmap v4394: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 13 op/s 2022-09-10T06:46:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:25 smithi104 ceph-mon[97044]: pgmap v4394: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 13 op/s 2022-09-10T06:46:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:25.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:25.548272+0000) 2022-09-10T06:46:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:25.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:25.950126+0000) 2022-09-10T06:46:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:26.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:26.548471+0000) 2022-09-10T06:46:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:26.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:26.950334+0000) 2022-09-10T06:46:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:27 smithi104 ceph-mon[97044]: pgmap v4395: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:27.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:27.548688+0000) 2022-09-10T06:46:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:27 smithi110 ceph-mon[97633]: pgmap v4395: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:27.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:27.950607+0000) 2022-09-10T06:46:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:28.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:28.548902+0000) 2022-09-10T06:46:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:28.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:28.950855+0000) 2022-09-10T06:46:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:29 smithi104 ceph-mon[97044]: pgmap v4396: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:29.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:29.549147+0000) 2022-09-10T06:46:29.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:46:29.430+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:46:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:29 smithi110 ceph-mon[97633]: pgmap v4396: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:29.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:29.951127+0000) 2022-09-10T06:46:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:30.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:30.549352+0000) 2022-09-10T06:46:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:30.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:30.951322+0000) 2022-09-10T06:46:31.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:31 smithi104 ceph-mon[97044]: pgmap v4397: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:31.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:31.549598+0000) 2022-09-10T06:46:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:31 smithi110 ceph-mon[97633]: pgmap v4397: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:31.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:31.951525+0000) 2022-09-10T06:46:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:32.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:32.549848+0000) 2022-09-10T06:46:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:32.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:32.951778+0000) 2022-09-10T06:46:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:33 smithi104 ceph-mon[97044]: pgmap v4398: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:33.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:33.550063+0000) 2022-09-10T06:46:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:33 smithi110 ceph-mon[97633]: pgmap v4398: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:33.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:33.951992+0000) 2022-09-10T06:46:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:34.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:34.550300+0000) 2022-09-10T06:46:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:34.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:34.952154+0000) 2022-09-10T06:46:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:35 smithi110 ceph-mon[97633]: pgmap v4399: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:46:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:35 smithi104 ceph-mon[97044]: pgmap v4399: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:46:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:35.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:35.550451+0000) 2022-09-10T06:46:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:35.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:35.952344+0000) 2022-09-10T06:46:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:36.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:36.550611+0000) 2022-09-10T06:46:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:36.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:36.952589+0000) 2022-09-10T06:46:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:37 smithi104 ceph-mon[97044]: pgmap v4400: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:37.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:37.550787+0000) 2022-09-10T06:46:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:37 smithi110 ceph-mon[97633]: pgmap v4400: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:37.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:37.952864+0000) 2022-09-10T06:46:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:38.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:38.551039+0000) 2022-09-10T06:46:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:38.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:38.953132+0000) 2022-09-10T06:46:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:39 smithi104 ceph-mon[97044]: pgmap v4401: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:39.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:39.551321+0000) 2022-09-10T06:46:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:39 smithi110 ceph-mon[97633]: pgmap v4401: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:39.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:39.953415+0000) 2022-09-10T06:46:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:40.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:40.551550+0000) 2022-09-10T06:46:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:40.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:40.953600+0000) 2022-09-10T06:46:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:41 smithi104 ceph-mon[97044]: pgmap v4402: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:41.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:41.551749+0000) 2022-09-10T06:46:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:41 smithi110 ceph-mon[97633]: pgmap v4402: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:41.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:41.953842+0000) 2022-09-10T06:46:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:42.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:42.551967+0000) 2022-09-10T06:46:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:42.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:42.953995+0000) 2022-09-10T06:46:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:43 smithi104 ceph-mon[97044]: pgmap v4403: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:43.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:43.552185+0000) 2022-09-10T06:46:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:43 smithi110 ceph-mon[97633]: pgmap v4403: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:43.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:43.954244+0000) 2022-09-10T06:46:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:46:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:46:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:46:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:46:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:44.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:44.552323+0000) 2022-09-10T06:46:44.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:46:44.438+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:46:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:46:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:46:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:46:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:46:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:44.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:44.954520+0000) 2022-09-10T06:46:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:45 smithi104 ceph-mon[97044]: pgmap v4404: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:46:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:45.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:45.552480+0000) 2022-09-10T06:46:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:45 smithi110 ceph-mon[97633]: pgmap v4404: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:46:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:45.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:45.954817+0000) 2022-09-10T06:46:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:46.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:46.552666+0000) 2022-09-10T06:46:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:46.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:46.955033+0000) 2022-09-10T06:46:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:47 smithi104 ceph-mon[97044]: pgmap v4405: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:47.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:47.552888+0000) 2022-09-10T06:46:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:47 smithi110 ceph-mon[97633]: pgmap v4405: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:47.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:47.955254+0000) 2022-09-10T06:46:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:48.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:48.553165+0000) 2022-09-10T06:46:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:48.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:48.955494+0000) 2022-09-10T06:46:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:49 smithi104 ceph-mon[97044]: pgmap v4406: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:49.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:49.553379+0000) 2022-09-10T06:46:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:49 smithi110 ceph-mon[97633]: pgmap v4406: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:49.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:49.955663+0000) 2022-09-10T06:46:50.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:50.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:50.553554+0000) 2022-09-10T06:46:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:50.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:50.955885+0000) 2022-09-10T06:46:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:51 smithi104 ceph-mon[97044]: pgmap v4407: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:51 smithi110 ceph-mon[97633]: pgmap v4407: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:46:51.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:51.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:51.553754+0000) 2022-09-10T06:46:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:51.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:51.956212+0000) 2022-09-10T06:46:52.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:52.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:52.553949+0000) 2022-09-10T06:46:53.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:52.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:52.956394+0000) 2022-09-10T06:46:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:53 smithi104 ceph-mon[97044]: pgmap v4408: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:53 smithi110 ceph-mon[97633]: pgmap v4408: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:53.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:53.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:53.554129+0000) 2022-09-10T06:46:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:53.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:53.956571+0000) 2022-09-10T06:46:54.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:54.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:54.554251+0000) 2022-09-10T06:46:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:54.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:54.956718+0000) 2022-09-10T06:46:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:55 smithi110 ceph-mon[97633]: pgmap v4409: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:46:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:55 smithi104 ceph-mon[97044]: pgmap v4409: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T06:46:55.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:55.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:55.554445+0000) 2022-09-10T06:46:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:55.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:55.956954+0000) 2022-09-10T06:46:56.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:56.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:56.554649+0000) 2022-09-10T06:46:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:56.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:56.957171+0000) 2022-09-10T06:46:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:57 smithi104 ceph-mon[97044]: pgmap v4410: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:57 smithi110 ceph-mon[97633]: pgmap v4410: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T06:46:57.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:57.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:57.554787+0000) 2022-09-10T06:46:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:57.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:57.957417+0000) 2022-09-10T06:46:58.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:58.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:58.554983+0000) 2022-09-10T06:46:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:58.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:58.957600+0000) 2022-09-10T06:46:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:46:59 smithi104 ceph-mon[97044]: pgmap v4411: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:46:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:46:59.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:59.555224+0000) 2022-09-10T06:46:59.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:46:59.447+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:46:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:46:59 smithi110 ceph-mon[97633]: pgmap v4411: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:46:59.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:46:59.957787+0000) 2022-09-10T06:47:00.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:00.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:00.555395+0000) 2022-09-10T06:47:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:00.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:00.957925+0000) 2022-09-10T06:47:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:01 smithi104 ceph-mon[97044]: pgmap v4412: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:01 smithi110 ceph-mon[97633]: pgmap v4412: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:01.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:01.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:01.555605+0000) 2022-09-10T06:47:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:01.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:01.958147+0000) 2022-09-10T06:47:02.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:02.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:02.555887+0000) 2022-09-10T06:47:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:02.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:02.958364+0000) 2022-09-10T06:47:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:03 smithi104 ceph-mon[97044]: pgmap v4413: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:03 smithi110 ceph-mon[97633]: pgmap v4413: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:03.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:03.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:03.556086+0000) 2022-09-10T06:47:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:03.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:03.958601+0000) 2022-09-10T06:47:04.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:04.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:04.556366+0000) 2022-09-10T06:47:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:04.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:04.958843+0000) 2022-09-10T06:47:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:05 smithi110 ceph-mon[97633]: pgmap v4414: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T06:47:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:05 smithi104 ceph-mon[97044]: pgmap v4414: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T06:47:05.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:05.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:05.556618+0000) 2022-09-10T06:47:06.229 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:05.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:05.959019+0000) 2022-09-10T06:47:06.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:06.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:06.556841+0000) 2022-09-10T06:47:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:06.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:06.959238+0000) 2022-09-10T06:47:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:07 smithi104 ceph-mon[97044]: pgmap v4415: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:47:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:07 smithi110 ceph-mon[97633]: pgmap v4415: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:47:07.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:07.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:07.557054+0000) 2022-09-10T06:47:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:07.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:07.959474+0000) 2022-09-10T06:47:08.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:08 smithi104 ceph-mon[97044]: osdmap e358: 12 total, 12 up, 12 in 2022-09-10T06:47:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:08 smithi110 ceph-mon[97633]: osdmap e358: 12 total, 12 up, 12 in 2022-09-10T06:47:08.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:08.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:08.557269+0000) 2022-09-10T06:47:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:08.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:08.959635+0000) 2022-09-10T06:47:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:09 smithi104 ceph-mon[97044]: pgmap v4417: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:47:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:09 smithi110 ceph-mon[97633]: pgmap v4417: 129 pgs: 129 active+clean; 8.1 MiB data, 579 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:47:09.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:09.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:09.557486+0000) 2022-09-10T06:47:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:09.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:09.959839+0000) 2022-09-10T06:47:10.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:10.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:10.557753+0000) 2022-09-10T06:47:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:10.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:10.960055+0000) 2022-09-10T06:47:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:11 smithi104 ceph-mon[97044]: pgmap v4418: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:47:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:11 smithi110 ceph-mon[97633]: pgmap v4418: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:47:11.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:11.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:11.557979+0000) 2022-09-10T06:47:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:11.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:11.960322+0000) 2022-09-10T06:47:12.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:12.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:12.558266+0000) 2022-09-10T06:47:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:12.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:12.960548+0000) 2022-09-10T06:47:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:13 smithi104 ceph-mon[97044]: pgmap v4419: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:47:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:13 smithi110 ceph-mon[97633]: pgmap v4419: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:47:13.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:13.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:13.558437+0000) 2022-09-10T06:47:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:13.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:13.960666+0000) 2022-09-10T06:47:14.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:47:14.451+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:47:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:14.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:14.558648+0000) 2022-09-10T06:47:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:14.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:14.960891+0000) 2022-09-10T06:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:15 smithi110 ceph-mon[97633]: osdmap e359: 12 total, 12 up, 12 in 2022-09-10T06:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:15 smithi110 ceph-mon[97633]: pgmap v4421: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:47:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:15 smithi104 ceph-mon[97044]: osdmap e359: 12 total, 12 up, 12 in 2022-09-10T06:47:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:15 smithi104 ceph-mon[97044]: pgmap v4421: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T06:47:15.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:15.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:15.558821+0000) 2022-09-10T06:47:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:15.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:15.961195+0000) 2022-09-10T06:47:16.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:16.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:16.558954+0000) 2022-09-10T06:47:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:16.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:16.961442+0000) 2022-09-10T06:47:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:17 smithi104 ceph-mon[97044]: pgmap v4422: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:47:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:17 smithi110 ceph-mon[97633]: pgmap v4422: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:47:17.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:17.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:17.559160+0000) 2022-09-10T06:47:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:17.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:17.961652+0000) 2022-09-10T06:47:18.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:18.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:18.559315+0000) 2022-09-10T06:47:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:18.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:18.961912+0000) 2022-09-10T06:47:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:19 smithi104 ceph-mon[97044]: pgmap v4423: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:47:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:19 smithi110 ceph-mon[97633]: pgmap v4423: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:47:19.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:19.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:19.559455+0000) 2022-09-10T06:47:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:19.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:19.962177+0000) 2022-09-10T06:47:20.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:20.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:20.559595+0000) 2022-09-10T06:47:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:20.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:20.962424+0000) 2022-09-10T06:47:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:21 smithi104 ceph-mon[97044]: pgmap v4424: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:47:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:21 smithi110 ceph-mon[97633]: pgmap v4424: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:47:21.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:21.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:21.559813+0000) 2022-09-10T06:47:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:21.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:21.962634+0000) 2022-09-10T06:47:22.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:22.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:22.559939+0000) 2022-09-10T06:47:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:22.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:22.962874+0000) 2022-09-10T06:47:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:23 smithi104 ceph-mon[97044]: pgmap v4425: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:47:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:23 smithi110 ceph-mon[97633]: pgmap v4425: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:47:23.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:23.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:23.560182+0000) 2022-09-10T06:47:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:23.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:23.963128+0000) 2022-09-10T06:47:24.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:24.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:24.560357+0000) 2022-09-10T06:47:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:24.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:24.963313+0000) 2022-09-10T06:47:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:25 smithi110 ceph-mon[97633]: pgmap v4426: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:47:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:25 smithi104 ceph-mon[97044]: pgmap v4426: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:47:25.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:25.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:25.560557+0000) 2022-09-10T06:47:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:25.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:25.963570+0000) 2022-09-10T06:47:26.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:26.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:26.560732+0000) 2022-09-10T06:47:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:26.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:26.963813+0000) 2022-09-10T06:47:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:27 smithi104 ceph-mon[97044]: pgmap v4427: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T06:47:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:27 smithi110 ceph-mon[97633]: pgmap v4427: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T06:47:27.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:27.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:27.560965+0000) 2022-09-10T06:47:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:27.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:27.963976+0000) 2022-09-10T06:47:28.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:28.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:28.561137+0000) 2022-09-10T06:47:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:28.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:28.964184+0000) 2022-09-10T06:47:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:29 smithi104 ceph-mon[97044]: pgmap v4428: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T06:47:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:29.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:29.561357+0000) 2022-09-10T06:47:29.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:47:29.462+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:47:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:29 smithi110 ceph-mon[97633]: pgmap v4428: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T06:47:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:29.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:29.964376+0000) 2022-09-10T06:47:30.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:30.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:30.561587+0000) 2022-09-10T06:47:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:30.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:30.964639+0000) 2022-09-10T06:47:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:31 smithi104 ceph-mon[97044]: pgmap v4429: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T06:47:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:31 smithi110 ceph-mon[97633]: pgmap v4429: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T06:47:31.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:31.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:31.561751+0000) 2022-09-10T06:47:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:31.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:31.964845+0000) 2022-09-10T06:47:32.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:32.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:32.561968+0000) 2022-09-10T06:47:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:32.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:32.965080+0000) 2022-09-10T06:47:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:33 smithi104 ceph-mon[97044]: pgmap v4430: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T06:47:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:33 smithi110 ceph-mon[97633]: pgmap v4430: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T06:47:33.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:33.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:33.562193+0000) 2022-09-10T06:47:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:33.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:33.965387+0000) 2022-09-10T06:47:34.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:34.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:34.562397+0000) 2022-09-10T06:47:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:34.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:34.965663+0000) 2022-09-10T06:47:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:35 smithi110 ceph-mon[97633]: pgmap v4431: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T06:47:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:35 smithi104 ceph-mon[97044]: pgmap v4431: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T06:47:35.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:35.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:35.562604+0000) 2022-09-10T06:47:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:35.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:35.965888+0000) 2022-09-10T06:47:36.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:36.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:36.562845+0000) 2022-09-10T06:47:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:36.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:36.966149+0000) 2022-09-10T06:47:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:37 smithi104 ceph-mon[97044]: pgmap v4432: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:37 smithi110 ceph-mon[97633]: pgmap v4432: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:37.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:37.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:37.563001+0000) 2022-09-10T06:47:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:37.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:37.966365+0000) 2022-09-10T06:47:38.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:38.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:38.563250+0000) 2022-09-10T06:47:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:38.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:38.966526+0000) 2022-09-10T06:47:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:39 smithi104 ceph-mon[97044]: pgmap v4433: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:39 smithi110 ceph-mon[97633]: pgmap v4433: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:39.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:39.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:39.563453+0000) 2022-09-10T06:47:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:39.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:39.966664+0000) 2022-09-10T06:47:40.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:40.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:40.563587+0000) 2022-09-10T06:47:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:40.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:40.966836+0000) 2022-09-10T06:47:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:41 smithi104 ceph-mon[97044]: pgmap v4434: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:41 smithi110 ceph-mon[97633]: pgmap v4434: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:41.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:41.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:41.563767+0000) 2022-09-10T06:47:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:41.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:41.967025+0000) 2022-09-10T06:47:42.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:42.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:42.563941+0000) 2022-09-10T06:47:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:42.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:42.967161+0000) 2022-09-10T06:47:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:43 smithi104 ceph-mon[97044]: pgmap v4435: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:43 smithi110 ceph-mon[97633]: pgmap v4435: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:43.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:43.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:43.564069+0000) 2022-09-10T06:47:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:43.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:43.967309+0000) 2022-09-10T06:47:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:47:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:47:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:47:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:47:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:44.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:44.564328+0000) 2022-09-10T06:47:44.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:47:44.478+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:47:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:47:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:47:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:47:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:47:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:44.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:44.967518+0000) 2022-09-10T06:47:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:45 smithi104 ceph-mon[97044]: pgmap v4436: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:45 smithi110 ceph-mon[97633]: pgmap v4436: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:45.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:45.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:45.564552+0000) 2022-09-10T06:47:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:45.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:45.967687+0000) 2022-09-10T06:47:46.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:46.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:46.564812+0000) 2022-09-10T06:47:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:46.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:46.967875+0000) 2022-09-10T06:47:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:47 smithi104 ceph-mon[97044]: pgmap v4437: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:47:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:47 smithi110 ceph-mon[97633]: pgmap v4437: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:47:47.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:47.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:47.565007+0000) 2022-09-10T06:47:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:47.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:47.968006+0000) 2022-09-10T06:47:48.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:48.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:48.565193+0000) 2022-09-10T06:47:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:48.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:48.968154+0000) 2022-09-10T06:47:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:49 smithi104 ceph-mon[97044]: pgmap v4438: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:47:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:49 smithi110 ceph-mon[97633]: pgmap v4438: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:47:49.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:49.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:49.565460+0000) 2022-09-10T06:47:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:49.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:49.968279+0000) 2022-09-10T06:47:50.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:50.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:50.565653+0000) 2022-09-10T06:47:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:50.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:50.968423+0000) 2022-09-10T06:47:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:51 smithi104 ceph-mon[97044]: pgmap v4439: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:51 smithi110 ceph-mon[97633]: pgmap v4439: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:51.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:51.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:51.565854+0000) 2022-09-10T06:47:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:51.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:51.968591+0000) 2022-09-10T06:47:52.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:52.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:52.566013+0000) 2022-09-10T06:47:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:52.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:52.968770+0000) 2022-09-10T06:47:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:53 smithi104 ceph-mon[97044]: pgmap v4440: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:47:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:53 smithi110 ceph-mon[97633]: pgmap v4440: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:47:53.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:53.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:53.566265+0000) 2022-09-10T06:47:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:53.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:53.968951+0000) 2022-09-10T06:47:54.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:54.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:54.566516+0000) 2022-09-10T06:47:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:54.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:54.969085+0000) 2022-09-10T06:47:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:55 smithi110 ceph-mon[97633]: pgmap v4441: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:55 smithi104 ceph-mon[97044]: pgmap v4441: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:47:55.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:55.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:55.566779+0000) 2022-09-10T06:47:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:55.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:55.969214+0000) 2022-09-10T06:47:56.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:56.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:56.566942+0000) 2022-09-10T06:47:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:56.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:56.969339+0000) 2022-09-10T06:47:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:57 smithi104 ceph-mon[97044]: pgmap v4442: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:57 smithi110 ceph-mon[97633]: pgmap v4442: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:57.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:57.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:57.567170+0000) 2022-09-10T06:47:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:57.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:57.969559+0000) 2022-09-10T06:47:58.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:58.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:58.567388+0000) 2022-09-10T06:47:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:58.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:58.969719+0000) 2022-09-10T06:47:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:47:59 smithi104 ceph-mon[97044]: pgmap v4443: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:47:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:47:59.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:59.567558+0000) 2022-09-10T06:47:59.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:47:59.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:47:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:47:59 smithi110 ceph-mon[97633]: pgmap v4443: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:00.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:47:59.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:47:59.969899+0000) 2022-09-10T06:48:00.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:00.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:00.567732+0000) 2022-09-10T06:48:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:00.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:00.970056+0000) 2022-09-10T06:48:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:01 smithi104 ceph-mon[97044]: pgmap v4444: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:01 smithi110 ceph-mon[97633]: pgmap v4444: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:01.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:01.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:01.567963+0000) 2022-09-10T06:48:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:01.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:01.970201+0000) 2022-09-10T06:48:02.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:02.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:02.568228+0000) 2022-09-10T06:48:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:02.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:02.970328+0000) 2022-09-10T06:48:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:03 smithi104 ceph-mon[97044]: pgmap v4445: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:03 smithi110 ceph-mon[97633]: pgmap v4445: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:03.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:03.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:03.568449+0000) 2022-09-10T06:48:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:03.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:03.970450+0000) 2022-09-10T06:48:04.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:04.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:04.568651+0000) 2022-09-10T06:48:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:04.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:04.970594+0000) 2022-09-10T06:48:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:05 smithi110 ceph-mon[97633]: pgmap v4446: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:48:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:05 smithi104 ceph-mon[97044]: pgmap v4446: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:48:05.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:05.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:05.568815+0000) 2022-09-10T06:48:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:05.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:05.970761+0000) 2022-09-10T06:48:06.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:06.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:06.568988+0000) 2022-09-10T06:48:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:06.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:06.970907+0000) 2022-09-10T06:48:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:07 smithi104 ceph-mon[97044]: pgmap v4447: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:48:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:07 smithi110 ceph-mon[97633]: pgmap v4447: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:48:07.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:07.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:07.569184+0000) 2022-09-10T06:48:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:07.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:07.971049+0000) 2022-09-10T06:48:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:08 smithi104 ceph-mon[97044]: osdmap e360: 12 total, 12 up, 12 in 2022-09-10T06:48:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:08 smithi110 ceph-mon[97633]: osdmap e360: 12 total, 12 up, 12 in 2022-09-10T06:48:08.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:08.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:08.569428+0000) 2022-09-10T06:48:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:08.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:08.971150+0000) 2022-09-10T06:48:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:09 smithi104 ceph-mon[97044]: pgmap v4449: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:48:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:09 smithi110 ceph-mon[97633]: pgmap v4449: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:48:09.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:09.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:09.569620+0000) 2022-09-10T06:48:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:09.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:09.971269+0000) 2022-09-10T06:48:10.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:10.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:10.569860+0000) 2022-09-10T06:48:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:10.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:10.971402+0000) 2022-09-10T06:48:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:11 smithi104 ceph-mon[97044]: pgmap v4450: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:48:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:11 smithi110 ceph-mon[97633]: pgmap v4450: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:48:11.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:11.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:11.570084+0000) 2022-09-10T06:48:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:11.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:11.971587+0000) 2022-09-10T06:48:12.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:12.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:12.570345+0000) 2022-09-10T06:48:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:12.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:12.971799+0000) 2022-09-10T06:48:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:13 smithi104 ceph-mon[97044]: pgmap v4451: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:48:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:13 smithi110 ceph-mon[97633]: pgmap v4451: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:48:13.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:13.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:13.570504+0000) 2022-09-10T06:48:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:13.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:13.971952+0000) 2022-09-10T06:48:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:14.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:14.570759+0000) 2022-09-10T06:48:14.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:48:14.502+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:48:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:14.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:14.972079+0000) 2022-09-10T06:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:15 smithi110 ceph-mon[97633]: osdmap e361: 12 total, 12 up, 12 in 2022-09-10T06:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:15 smithi110 ceph-mon[97633]: pgmap v4453: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:15 smithi104 ceph-mon[97044]: osdmap e361: 12 total, 12 up, 12 in 2022-09-10T06:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:15 smithi104 ceph-mon[97044]: pgmap v4453: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:48:15.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:15.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:15.570951+0000) 2022-09-10T06:48:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:15.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:15.972211+0000) 2022-09-10T06:48:16.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:16.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:16.571215+0000) 2022-09-10T06:48:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:16.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:16.972383+0000) 2022-09-10T06:48:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:17 smithi104 ceph-mon[97044]: pgmap v4454: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:48:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:17 smithi110 ceph-mon[97633]: pgmap v4454: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:48:17.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:17.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:17.571417+0000) 2022-09-10T06:48:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:17.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:17.972542+0000) 2022-09-10T06:48:18.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:18.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:18.571649+0000) 2022-09-10T06:48:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:18.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:18.972664+0000) 2022-09-10T06:48:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:19 smithi104 ceph-mon[97044]: pgmap v4455: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:48:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:19 smithi110 ceph-mon[97633]: pgmap v4455: 129 pgs: 129 active+clean; 8.1 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:48:19.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:19.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:19.571909+0000) 2022-09-10T06:48:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:19.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:19.972891+0000) 2022-09-10T06:48:20.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:20.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:20.572200+0000) 2022-09-10T06:48:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:20.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:20.973079+0000) 2022-09-10T06:48:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:21 smithi104 ceph-mon[97044]: pgmap v4456: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:48:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:21 smithi110 ceph-mon[97633]: pgmap v4456: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:48:21.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:21.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:21.572458+0000) 2022-09-10T06:48:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:21.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:21.973348+0000) 2022-09-10T06:48:22.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:22.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:22.572675+0000) 2022-09-10T06:48:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:22.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:22.973540+0000) 2022-09-10T06:48:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:23 smithi104 ceph-mon[97044]: pgmap v4457: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:48:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:23 smithi110 ceph-mon[97633]: pgmap v4457: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:48:23.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:23.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:23.572912+0000) 2022-09-10T06:48:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:23.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:23.973741+0000) 2022-09-10T06:48:24.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:24.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:24.573214+0000) 2022-09-10T06:48:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:24.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:24.974010+0000) 2022-09-10T06:48:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:25 smithi104 ceph-mon[97044]: pgmap v4458: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:48:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:25 smithi110 ceph-mon[97633]: pgmap v4458: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:48:25.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:25.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:25.573428+0000) 2022-09-10T06:48:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:25.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:25.974266+0000) 2022-09-10T06:48:26.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:26.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:26.573626+0000) 2022-09-10T06:48:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:26.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:26.974537+0000) 2022-09-10T06:48:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:27 smithi104 ceph-mon[97044]: pgmap v4459: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:27 smithi110 ceph-mon[97633]: pgmap v4459: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:27.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:27.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:27.573862+0000) 2022-09-10T06:48:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:27.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:27.974728+0000) 2022-09-10T06:48:28.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:28.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:28.574087+0000) 2022-09-10T06:48:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:28.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:28.974897+0000) 2022-09-10T06:48:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:29 smithi104 ceph-mon[97044]: pgmap v4460: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:29.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:29.574374+0000) 2022-09-10T06:48:29.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:48:29.513+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:48:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:29 smithi110 ceph-mon[97633]: pgmap v4460: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:29.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:29.975050+0000) 2022-09-10T06:48:30.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:30.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:30.574648+0000) 2022-09-10T06:48:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:30.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:30.975217+0000) 2022-09-10T06:48:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:31 smithi104 ceph-mon[97044]: pgmap v4461: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:31 smithi110 ceph-mon[97633]: pgmap v4461: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:31.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:31.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:31.574940+0000) 2022-09-10T06:48:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:31.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:31.975457+0000) 2022-09-10T06:48:32.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:32.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:32.575155+0000) 2022-09-10T06:48:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:32.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:32.975706+0000) 2022-09-10T06:48:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:33 smithi104 ceph-mon[97044]: pgmap v4462: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:33 smithi110 ceph-mon[97633]: pgmap v4462: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:33.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:33.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:33.575405+0000) 2022-09-10T06:48:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:33.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:33.975941+0000) 2022-09-10T06:48:34.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:34.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:34.575660+0000) 2022-09-10T06:48:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:34.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:34.976206+0000) 2022-09-10T06:48:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:35 smithi104 ceph-mon[97044]: pgmap v4463: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:35 smithi110 ceph-mon[97633]: pgmap v4463: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:35.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:35.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:35.575885+0000) 2022-09-10T06:48:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:35.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:35.976373+0000) 2022-09-10T06:48:36.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:36.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:36.576118+0000) 2022-09-10T06:48:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:36.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:36.976609+0000) 2022-09-10T06:48:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:37 smithi104 ceph-mon[97044]: pgmap v4464: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:37.876 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:37.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:37.576398+0000) 2022-09-10T06:48:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:37 smithi110 ceph-mon[97633]: pgmap v4464: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:37.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:37.976861+0000) 2022-09-10T06:48:38.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:38.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:38.576651+0000) 2022-09-10T06:48:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:38.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:38.977019+0000) 2022-09-10T06:48:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:39 smithi104 ceph-mon[97044]: pgmap v4465: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:39 smithi110 ceph-mon[97633]: pgmap v4465: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:39.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:39.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:39.576843+0000) 2022-09-10T06:48:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:39.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:39.977275+0000) 2022-09-10T06:48:40.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:40.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:40.577069+0000) 2022-09-10T06:48:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:40.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:40.977509+0000) 2022-09-10T06:48:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:41 smithi104 ceph-mon[97044]: pgmap v4466: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:41 smithi110 ceph-mon[97633]: pgmap v4466: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:41.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:41.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:41.577386+0000) 2022-09-10T06:48:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:41.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:41.977694+0000) 2022-09-10T06:48:42.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:42.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:42.577571+0000) 2022-09-10T06:48:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:42.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:42.977906+0000) 2022-09-10T06:48:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:43 smithi104 ceph-mon[97044]: pgmap v4467: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:43 smithi110 ceph-mon[97633]: pgmap v4467: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:43.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:43.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:43.577842+0000) 2022-09-10T06:48:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:43.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:43.978156+0000) 2022-09-10T06:48:44.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:48:44.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:48:44.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:48:44.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:48:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:44.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:44.578066+0000) 2022-09-10T06:48:44.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:48:44.520+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:48:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:48:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:48:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:48:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:48:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:44.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:44.978377+0000) 2022-09-10T06:48:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:45 smithi104 ceph-mon[97044]: pgmap v4468: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:45 smithi110 ceph-mon[97633]: pgmap v4468: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:45.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:45.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:45.578247+0000) 2022-09-10T06:48:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:45.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:45.978538+0000) 2022-09-10T06:48:46.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:46.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:46.578427+0000) 2022-09-10T06:48:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:46.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:46.978739+0000) 2022-09-10T06:48:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:47 smithi104 ceph-mon[97044]: pgmap v4469: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:47 smithi110 ceph-mon[97633]: pgmap v4469: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:47.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:47.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:47.578691+0000) 2022-09-10T06:48:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:47.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:47.978979+0000) 2022-09-10T06:48:48.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:48.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:48.578938+0000) 2022-09-10T06:48:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:48.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:48.979205+0000) 2022-09-10T06:48:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:49 smithi104 ceph-mon[97044]: pgmap v4470: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:49 smithi110 ceph-mon[97633]: pgmap v4470: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:49.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:49.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:49.579171+0000) 2022-09-10T06:48:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:49.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:49.979462+0000) 2022-09-10T06:48:50.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:50.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:50.579370+0000) 2022-09-10T06:48:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:50.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:50.979666+0000) 2022-09-10T06:48:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:51 smithi104 ceph-mon[97044]: pgmap v4471: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:51 smithi110 ceph-mon[97633]: pgmap v4471: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:51.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:51.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:51.579538+0000) 2022-09-10T06:48:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:51.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:51.979852+0000) 2022-09-10T06:48:52.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:52.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:52.579784+0000) 2022-09-10T06:48:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:52.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:52.980063+0000) 2022-09-10T06:48:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:53 smithi104 ceph-mon[97044]: pgmap v4472: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:53 smithi110 ceph-mon[97633]: pgmap v4472: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:53.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:53.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:53.580007+0000) 2022-09-10T06:48:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:53.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:53.980363+0000) 2022-09-10T06:48:54.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:54.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:54.580162+0000) 2022-09-10T06:48:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:54.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:54.980621+0000) 2022-09-10T06:48:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:55 smithi104 ceph-mon[97044]: pgmap v4473: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:55 smithi110 ceph-mon[97633]: pgmap v4473: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:48:55.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:55.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:55.580331+0000) 2022-09-10T06:48:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:55.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:55.980869+0000) 2022-09-10T06:48:56.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:56.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:56.580548+0000) 2022-09-10T06:48:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:56.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:56.981076+0000) 2022-09-10T06:48:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:57 smithi104 ceph-mon[97044]: pgmap v4474: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:57 smithi110 ceph-mon[97633]: pgmap v4474: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:57.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:57.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:57.580796+0000) 2022-09-10T06:48:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:57.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:57.981314+0000) 2022-09-10T06:48:58.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:58.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:58.581039+0000) 2022-09-10T06:48:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:58.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:58.981565+0000) 2022-09-10T06:48:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:48:59 smithi104 ceph-mon[97044]: pgmap v4475: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:48:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:48:59.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:59.581257+0000) 2022-09-10T06:48:59.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:48:59.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:48:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:48:59 smithi110 ceph-mon[97633]: pgmap v4475: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:48:59.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:48:59.981772+0000) 2022-09-10T06:49:00.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:00.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:00.581499+0000) 2022-09-10T06:49:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:00.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:00.982232+0000) 2022-09-10T06:49:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:01 smithi104 ceph-mon[97044]: pgmap v4476: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:01 smithi110 ceph-mon[97633]: pgmap v4476: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:01.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:01.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:01.581729+0000) 2022-09-10T06:49:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:01.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:01.982474+0000) 2022-09-10T06:49:02.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:02.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:02.581935+0000) 2022-09-10T06:49:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:02.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:02.982715+0000) 2022-09-10T06:49:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:03 smithi104 ceph-mon[97044]: pgmap v4477: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:03 smithi110 ceph-mon[97633]: pgmap v4477: 129 pgs: 129 active+clean; 8.1 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:03.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:03.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:03.582202+0000) 2022-09-10T06:49:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:03.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:03.982867+0000) 2022-09-10T06:49:04.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:04.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:04.582407+0000) 2022-09-10T06:49:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:04.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:04.983067+0000) 2022-09-10T06:49:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:05 smithi104 ceph-mon[97044]: pgmap v4478: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:49:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:05 smithi110 ceph-mon[97633]: pgmap v4478: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:49:05.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:05.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:05.582601+0000) 2022-09-10T06:49:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:05.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:05.983207+0000) 2022-09-10T06:49:06.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:06.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:06.582791+0000) 2022-09-10T06:49:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:06.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:06.983396+0000) 2022-09-10T06:49:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:07 smithi104 ceph-mon[97044]: pgmap v4479: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:49:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:07 smithi110 ceph-mon[97633]: pgmap v4479: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:49:07.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:07.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:07.583029+0000) 2022-09-10T06:49:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:07.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:07.983649+0000) 2022-09-10T06:49:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:08 smithi104 ceph-mon[97044]: osdmap e362: 12 total, 12 up, 12 in 2022-09-10T06:49:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:08 smithi110 ceph-mon[97633]: osdmap e362: 12 total, 12 up, 12 in 2022-09-10T06:49:08.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:08.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:08.583253+0000) 2022-09-10T06:49:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:08.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:08.983828+0000) 2022-09-10T06:49:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:09 smithi104 ceph-mon[97044]: pgmap v4481: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:49:09.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:09 smithi110 ceph-mon[97633]: pgmap v4481: 129 pgs: 129 active+clean; 8.1 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:49:09.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:09.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:09.583541+0000) 2022-09-10T06:49:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:09.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:09.984061+0000) 2022-09-10T06:49:10.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:10.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:10.583805+0000) 2022-09-10T06:49:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:10.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:10.984269+0000) 2022-09-10T06:49:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:11 smithi104 ceph-mon[97044]: pgmap v4482: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:49:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:11 smithi110 ceph-mon[97633]: pgmap v4482: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:49:11.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:11.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:11.584046+0000) 2022-09-10T06:49:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:11.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:11.984522+0000) 2022-09-10T06:49:12.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:12.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:12.584309+0000) 2022-09-10T06:49:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:12.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:12.984797+0000) 2022-09-10T06:49:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:13 smithi104 ceph-mon[97044]: pgmap v4483: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:49:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:13 smithi110 ceph-mon[97633]: pgmap v4483: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:49:13.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:13.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:13.584554+0000) 2022-09-10T06:49:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:13.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:13.984995+0000) 2022-09-10T06:49:14.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:49:14.551+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:49:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:14.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:14.584775+0000) 2022-09-10T06:49:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:14.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:14.985224+0000) 2022-09-10T06:49:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:15 smithi104 ceph-mon[97044]: osdmap e363: 12 total, 12 up, 12 in 2022-09-10T06:49:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:15 smithi104 ceph-mon[97044]: pgmap v4485: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:49:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:15 smithi110 ceph-mon[97633]: osdmap e363: 12 total, 12 up, 12 in 2022-09-10T06:49:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:15 smithi110 ceph-mon[97633]: pgmap v4485: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:49:15.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:15.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:15.584929+0000) 2022-09-10T06:49:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:15.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:15.985486+0000) 2022-09-10T06:49:16.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:16.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:16.585167+0000) 2022-09-10T06:49:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:16.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:16.985721+0000) 2022-09-10T06:49:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:17 smithi104 ceph-mon[97044]: pgmap v4486: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:49:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:17 smithi110 ceph-mon[97633]: pgmap v4486: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:49:17.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:17.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:17.585342+0000) 2022-09-10T06:49:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:17.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:17.985972+0000) 2022-09-10T06:49:18.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:18.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:18.585592+0000) 2022-09-10T06:49:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:18.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:18.986214+0000) 2022-09-10T06:49:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:19 smithi104 ceph-mon[97044]: pgmap v4487: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:49:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:19 smithi110 ceph-mon[97633]: pgmap v4487: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:49:19.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:19.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:19.585831+0000) 2022-09-10T06:49:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:19.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:19.986465+0000) 2022-09-10T06:49:20.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:20.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:20.585971+0000) 2022-09-10T06:49:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:20.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:20.986683+0000) 2022-09-10T06:49:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:21 smithi104 ceph-mon[97044]: pgmap v4488: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:49:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:21 smithi110 ceph-mon[97633]: pgmap v4488: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:49:21.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:21.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:21.586159+0000) 2022-09-10T06:49:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:21.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:21.986893+0000) 2022-09-10T06:49:22.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:22.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:22.586402+0000) 2022-09-10T06:49:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:22.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:22.987134+0000) 2022-09-10T06:49:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:23 smithi104 ceph-mon[97044]: pgmap v4489: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:49:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:23 smithi110 ceph-mon[97633]: pgmap v4489: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:49:23.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:23.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:23.586671+0000) 2022-09-10T06:49:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:23.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:23.987419+0000) 2022-09-10T06:49:24.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:24.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:24.586848+0000) 2022-09-10T06:49:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:24.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:24.987672+0000) 2022-09-10T06:49:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:25 smithi104 ceph-mon[97044]: pgmap v4490: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:49:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:25 smithi110 ceph-mon[97633]: pgmap v4490: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:49:25.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:25.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:25.587001+0000) 2022-09-10T06:49:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:25.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:25.987808+0000) 2022-09-10T06:49:26.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:26.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:26.587240+0000) 2022-09-10T06:49:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:26.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:26.988021+0000) 2022-09-10T06:49:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:27 smithi104 ceph-mon[97044]: pgmap v4491: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:27 smithi110 ceph-mon[97633]: pgmap v4491: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:27.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:27.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:27.587460+0000) 2022-09-10T06:49:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:27.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:27.988204+0000) 2022-09-10T06:49:28.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:28.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:28.587699+0000) 2022-09-10T06:49:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:28.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:28.988427+0000) 2022-09-10T06:49:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:29 smithi104 ceph-mon[97044]: pgmap v4492: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:29 smithi110 ceph-mon[97633]: pgmap v4492: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:29.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:29.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:29.587995+0000) 2022-09-10T06:49:29.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:49:29.558+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:49:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:29.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:29.988668+0000) 2022-09-10T06:49:30.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:30.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:30.588270+0000) 2022-09-10T06:49:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:30.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:30.988829+0000) 2022-09-10T06:49:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:31 smithi104 ceph-mon[97044]: pgmap v4493: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:31 smithi110 ceph-mon[97633]: pgmap v4493: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:31.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:31.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:31.588483+0000) 2022-09-10T06:49:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:31.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:31.989014+0000) 2022-09-10T06:49:32.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:32.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:32.588650+0000) 2022-09-10T06:49:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:32.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:32.989237+0000) 2022-09-10T06:49:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:33 smithi104 ceph-mon[97044]: pgmap v4494: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:33 smithi110 ceph-mon[97633]: pgmap v4494: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:33.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:33.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:33.588936+0000) 2022-09-10T06:49:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:33.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:33.989442+0000) 2022-09-10T06:49:34.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:34.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:34.589206+0000) 2022-09-10T06:49:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:34.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:34.989646+0000) 2022-09-10T06:49:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:35 smithi104 ceph-mon[97044]: pgmap v4495: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:35 smithi110 ceph-mon[97633]: pgmap v4495: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:35.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:35.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:35.589390+0000) 2022-09-10T06:49:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:35.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:35.989853+0000) 2022-09-10T06:49:36.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:36.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:36.589638+0000) 2022-09-10T06:49:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:36.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:36.990086+0000) 2022-09-10T06:49:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:37 smithi104 ceph-mon[97044]: pgmap v4496: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:37 smithi110 ceph-mon[97633]: pgmap v4496: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:37.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:37.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:37.589893+0000) 2022-09-10T06:49:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:37.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:37.990348+0000) 2022-09-10T06:49:38.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:38.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:38.590199+0000) 2022-09-10T06:49:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:38.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:38.990558+0000) 2022-09-10T06:49:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:39 smithi104 ceph-mon[97044]: pgmap v4497: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:39 smithi110 ceph-mon[97633]: pgmap v4497: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:39.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:39.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:39.590472+0000) 2022-09-10T06:49:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:39.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:39.990806+0000) 2022-09-10T06:49:40.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:40.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:40.590724+0000) 2022-09-10T06:49:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:40.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:40.991053+0000) 2022-09-10T06:49:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:41 smithi104 ceph-mon[97044]: pgmap v4498: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:41 smithi110 ceph-mon[97633]: pgmap v4498: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:41.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:41.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:41.590949+0000) 2022-09-10T06:49:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:41.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:41.991246+0000) 2022-09-10T06:49:42.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:42.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:42.591220+0000) 2022-09-10T06:49:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:42.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:42.991408+0000) 2022-09-10T06:49:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:43 smithi104 ceph-mon[97044]: pgmap v4499: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:43 smithi110 ceph-mon[97633]: pgmap v4499: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:43.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:43.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:43.591494+0000) 2022-09-10T06:49:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:43.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:43.991601+0000) 2022-09-10T06:49:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:49:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:49:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:49:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:49:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:49:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:49:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:49:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:49:44.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:44.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:44.591671+0000) 2022-09-10T06:49:44.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:49:44.563+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:49:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:44.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:44.991860+0000) 2022-09-10T06:49:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:45 smithi104 ceph-mon[97044]: pgmap v4500: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:45 smithi110 ceph-mon[97633]: pgmap v4500: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:45.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:45.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:45.591911+0000) 2022-09-10T06:49:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:45.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:45.992149+0000) 2022-09-10T06:49:46.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:46.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:46.592194+0000) 2022-09-10T06:49:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:46.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:46.992283+0000) 2022-09-10T06:49:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:47 smithi104 ceph-mon[97044]: pgmap v4501: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:47 smithi110 ceph-mon[97633]: pgmap v4501: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:47.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:47.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:47.592422+0000) 2022-09-10T06:49:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:47.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:47.992439+0000) 2022-09-10T06:49:48.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:48.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:48.592622+0000) 2022-09-10T06:49:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:48.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:48.992625+0000) 2022-09-10T06:49:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:49 smithi104 ceph-mon[97044]: pgmap v4502: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:49 smithi110 ceph-mon[97633]: pgmap v4502: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:49.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:49.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:49.592883+0000) 2022-09-10T06:49:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:49.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:49.992822+0000) 2022-09-10T06:49:50.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:50.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:50.593128+0000) 2022-09-10T06:49:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:50.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:50.993049+0000) 2022-09-10T06:49:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:51 smithi104 ceph-mon[97044]: pgmap v4503: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:51 smithi110 ceph-mon[97633]: pgmap v4503: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:51.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:51.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:51.593307+0000) 2022-09-10T06:49:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:51.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:51.993315+0000) 2022-09-10T06:49:52.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:52.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:52.593539+0000) 2022-09-10T06:49:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:52.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:52.993493+0000) 2022-09-10T06:49:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:53 smithi104 ceph-mon[97044]: pgmap v4504: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:53 smithi110 ceph-mon[97633]: pgmap v4504: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:53.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:53.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:53.593787+0000) 2022-09-10T06:49:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:53.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:53.993726+0000) 2022-09-10T06:49:54.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:54.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:54.594055+0000) 2022-09-10T06:49:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:54.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:54.993979+0000) 2022-09-10T06:49:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:55 smithi104 ceph-mon[97044]: pgmap v4505: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:55 smithi110 ceph-mon[97633]: pgmap v4505: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:49:55.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:55.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:55.594276+0000) 2022-09-10T06:49:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:55.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:55.994209+0000) 2022-09-10T06:49:56.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:56.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:56.594539+0000) 2022-09-10T06:49:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:56.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:56.994337+0000) 2022-09-10T06:49:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:57 smithi104 ceph-mon[97044]: pgmap v4506: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:57 smithi110 ceph-mon[97633]: pgmap v4506: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:57.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:57.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:57.594751+0000) 2022-09-10T06:49:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:57.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:57.994579+0000) 2022-09-10T06:49:58.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:58.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:58.594977+0000) 2022-09-10T06:49:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:58.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:58.994815+0000) 2022-09-10T06:49:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:49:59 smithi104 ceph-mon[97044]: pgmap v4507: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:49:59 smithi110 ceph-mon[97633]: pgmap v4507: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:49:59.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:49:59.572+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:49:59.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:49:59.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:59.595235+0000) 2022-09-10T06:50:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:49:59.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:49:59.995035+0000) 2022-09-10T06:50:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:50:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T06:50:00.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:00.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:00.595462+0000) 2022-09-10T06:50:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:00.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:00.995242+0000) 2022-09-10T06:50:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:01 smithi104 ceph-mon[97044]: pgmap v4508: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:01 smithi110 ceph-mon[97633]: pgmap v4508: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:01.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:01.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:01.595698+0000) 2022-09-10T06:50:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:01.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:01.995476+0000) 2022-09-10T06:50:02.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:02.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:02.595971+0000) 2022-09-10T06:50:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:02.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:02.995669+0000) 2022-09-10T06:50:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:03 smithi104 ceph-mon[97044]: pgmap v4509: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:03 smithi110 ceph-mon[97633]: pgmap v4509: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:03.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:03.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:03.596173+0000) 2022-09-10T06:50:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:03.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:03.995865+0000) 2022-09-10T06:50:04.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:04.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:04.596455+0000) 2022-09-10T06:50:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:04.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:04.996164+0000) 2022-09-10T06:50:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:05 smithi104 ceph-mon[97044]: pgmap v4510: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:50:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:05 smithi110 ceph-mon[97633]: pgmap v4510: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:50:05.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:05.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:05.596656+0000) 2022-09-10T06:50:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:05.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:05.996283+0000) 2022-09-10T06:50:06.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:06.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:06.596916+0000) 2022-09-10T06:50:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:06.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:06.996398+0000) 2022-09-10T06:50:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:07 smithi104 ceph-mon[97044]: pgmap v4511: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:50:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:07 smithi110 ceph-mon[97633]: pgmap v4511: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:50:07.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:07.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:07.597241+0000) 2022-09-10T06:50:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:07.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:07.996644+0000) 2022-09-10T06:50:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:08 smithi104 ceph-mon[97044]: osdmap e364: 12 total, 12 up, 12 in 2022-09-10T06:50:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:08 smithi110 ceph-mon[97633]: osdmap e364: 12 total, 12 up, 12 in 2022-09-10T06:50:08.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:08.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:08.597513+0000) 2022-09-10T06:50:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:08.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:08.996861+0000) 2022-09-10T06:50:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:09 smithi104 ceph-mon[97044]: pgmap v4513: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:50:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:09 smithi110 ceph-mon[97633]: pgmap v4513: 129 pgs: 129 active+clean; 8.1 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:50:09.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:09.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:09.597770+0000) 2022-09-10T06:50:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:09.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:09.997142+0000) 2022-09-10T06:50:10.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:10.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:10.597967+0000) 2022-09-10T06:50:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:10.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:10.997288+0000) 2022-09-10T06:50:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:11 smithi104 ceph-mon[97044]: pgmap v4514: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:50:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:11 smithi110 ceph-mon[97633]: pgmap v4514: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:50:11.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:11.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:11.598114+0000) 2022-09-10T06:50:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:11.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:11.997508+0000) 2022-09-10T06:50:12.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:12.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:12.598316+0000) 2022-09-10T06:50:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:12.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:12.997725+0000) 2022-09-10T06:50:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:13 smithi104 ceph-mon[97044]: pgmap v4515: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:50:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:13 smithi110 ceph-mon[97633]: pgmap v4515: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:50:13.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:13.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:13.598517+0000) 2022-09-10T06:50:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:13.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:13.997948+0000) 2022-09-10T06:50:14.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:14.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:14.598746+0000) 2022-09-10T06:50:14.999 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:50:14.587+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:50:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:14.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:14.998178+0000) 2022-09-10T06:50:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:15 smithi104 ceph-mon[97044]: osdmap e365: 12 total, 12 up, 12 in 2022-09-10T06:50:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:15 smithi104 ceph-mon[97044]: pgmap v4517: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:50:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:15 smithi110 ceph-mon[97633]: osdmap e365: 12 total, 12 up, 12 in 2022-09-10T06:50:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:15 smithi110 ceph-mon[97633]: pgmap v4517: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:50:16.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:15.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:15.598928+0000) 2022-09-10T06:50:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:15.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:15.998368+0000) 2022-09-10T06:50:16.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:16.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:16.599113+0000) 2022-09-10T06:50:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:16.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:16.998607+0000) 2022-09-10T06:50:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:17 smithi104 ceph-mon[97044]: pgmap v4518: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:50:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:17 smithi110 ceph-mon[97633]: pgmap v4518: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:50:17.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:17.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:17.599330+0000) 2022-09-10T06:50:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:17.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:17.998760+0000) 2022-09-10T06:50:18.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:18.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:18.599597+0000) 2022-09-10T06:50:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:18.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:18.999002+0000) 2022-09-10T06:50:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:19 smithi104 ceph-mon[97044]: pgmap v4519: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:50:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:19 smithi110 ceph-mon[97633]: pgmap v4519: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:50:20.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:19.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:19.599842+0000) 2022-09-10T06:50:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:19.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:19.999252+0000) 2022-09-10T06:50:21.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:20.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:20.600079+0000) 2022-09-10T06:50:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:20.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:20.999449+0000) 2022-09-10T06:50:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:21 smithi104 ceph-mon[97044]: pgmap v4520: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:50:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:21 smithi110 ceph-mon[97633]: pgmap v4520: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:50:22.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:21.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:21.600258+0000) 2022-09-10T06:50:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:21.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:21.999741+0000) 2022-09-10T06:50:23.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:22.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:22.600431+0000) 2022-09-10T06:50:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:22.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:22.999977+0000) 2022-09-10T06:50:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:23 smithi104 ceph-mon[97044]: pgmap v4521: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:50:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:23 smithi110 ceph-mon[97633]: pgmap v4521: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:50:24.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:23.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:23.600668+0000) 2022-09-10T06:50:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:23.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:24.000172+0000) 2022-09-10T06:50:25.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:24.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:24.600887+0000) 2022-09-10T06:50:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:24.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:25.000393+0000) 2022-09-10T06:50:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:25 smithi104 ceph-mon[97044]: pgmap v4522: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:50:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:25 smithi110 ceph-mon[97633]: pgmap v4522: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:50:26.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:25.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:25.601088+0000) 2022-09-10T06:50:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:25.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:26.000610+0000) 2022-09-10T06:50:27.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:26.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:26.601357+0000) 2022-09-10T06:50:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:27.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:27.000850+0000) 2022-09-10T06:50:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:27 smithi104 ceph-mon[97044]: pgmap v4523: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:27 smithi110 ceph-mon[97633]: pgmap v4523: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:28.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:27.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:27.601598+0000) 2022-09-10T06:50:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:28.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:28.001102+0000) 2022-09-10T06:50:29.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:28.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:28.601793+0000) 2022-09-10T06:50:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:29.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:29.001311+0000) 2022-09-10T06:50:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:29 smithi104 ceph-mon[97044]: pgmap v4524: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:29 smithi110 ceph-mon[97633]: pgmap v4524: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:30.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:29.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:29.602049+0000) 2022-09-10T06:50:30.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:50:29.588+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:50:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:30.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:30.001560+0000) 2022-09-10T06:50:31.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:30.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:30.602270+0000) 2022-09-10T06:50:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:31.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:31.001796+0000) 2022-09-10T06:50:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:31 smithi104 ceph-mon[97044]: pgmap v4525: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:31 smithi110 ceph-mon[97633]: pgmap v4525: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:32.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:31.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:31.602400+0000) 2022-09-10T06:50:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:32.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:32.002059+0000) 2022-09-10T06:50:33.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:32.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:32.602588+0000) 2022-09-10T06:50:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:33.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:33.002327+0000) 2022-09-10T06:50:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:33 smithi104 ceph-mon[97044]: pgmap v4526: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:33 smithi110 ceph-mon[97633]: pgmap v4526: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:34.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:33.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:33.602801+0000) 2022-09-10T06:50:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:34.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:34.002532+0000) 2022-09-10T06:50:35.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:34.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:34.603061+0000) 2022-09-10T06:50:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:35.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:35.002800+0000) 2022-09-10T06:50:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:35 smithi104 ceph-mon[97044]: pgmap v4527: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:35 smithi110 ceph-mon[97633]: pgmap v4527: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:36.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:35.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:35.603331+0000) 2022-09-10T06:50:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:36.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:36.003019+0000) 2022-09-10T06:50:37.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:36.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:36.603557+0000) 2022-09-10T06:50:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:37.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:37.003208+0000) 2022-09-10T06:50:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:37 smithi104 ceph-mon[97044]: pgmap v4528: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:37 smithi110 ceph-mon[97633]: pgmap v4528: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:38.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:37.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:37.603839+0000) 2022-09-10T06:50:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:38.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:38.003389+0000) 2022-09-10T06:50:39.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:38.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:38.604050+0000) 2022-09-10T06:50:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:39.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:39.003630+0000) 2022-09-10T06:50:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:39 smithi104 ceph-mon[97044]: pgmap v4529: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:39 smithi110 ceph-mon[97633]: pgmap v4529: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:40.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:39.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:39.604285+0000) 2022-09-10T06:50:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:40.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:40.003903+0000) 2022-09-10T06:50:41.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:40.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:40.604470+0000) 2022-09-10T06:50:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:41.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:41.004202+0000) 2022-09-10T06:50:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:41 smithi104 ceph-mon[97044]: pgmap v4530: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:41 smithi110 ceph-mon[97633]: pgmap v4530: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:42.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:41.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:41.604712+0000) 2022-09-10T06:50:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:42.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:42.004403+0000) 2022-09-10T06:50:43.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:42.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:42.604963+0000) 2022-09-10T06:50:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:43.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:43.004561+0000) 2022-09-10T06:50:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:43 smithi104 ceph-mon[97044]: pgmap v4531: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:43 smithi110 ceph-mon[97633]: pgmap v4531: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:44.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:43.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:43.605247+0000) 2022-09-10T06:50:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:44.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:44.004801+0000) 2022-09-10T06:50:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:50:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:50:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:50:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:50:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:50:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:50:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:50:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:50:45.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:44.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:44.605425+0000) 2022-09-10T06:50:45.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:50:44.603+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:50:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:45.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:45.005034+0000) 2022-09-10T06:50:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:45 smithi104 ceph-mon[97044]: pgmap v4532: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:45 smithi110 ceph-mon[97633]: pgmap v4532: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:46.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:45.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:45.605603+0000) 2022-09-10T06:50:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:46.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:46.005259+0000) 2022-09-10T06:50:47.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:46.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:46.605855+0000) 2022-09-10T06:50:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:47.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:47.005429+0000) 2022-09-10T06:50:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:47 smithi104 ceph-mon[97044]: pgmap v4533: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:47 smithi110 ceph-mon[97633]: pgmap v4533: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:48.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:47.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:47.606065+0000) 2022-09-10T06:50:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:48.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:48.005682+0000) 2022-09-10T06:50:49.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:48.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:48.606339+0000) 2022-09-10T06:50:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:49.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:49.005937+0000) 2022-09-10T06:50:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:49 smithi104 ceph-mon[97044]: pgmap v4534: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:49 smithi110 ceph-mon[97633]: pgmap v4534: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:50.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:49.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:49.606532+0000) 2022-09-10T06:50:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:50.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:50.006215+0000) 2022-09-10T06:50:51.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:50.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:50.606711+0000) 2022-09-10T06:50:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:51.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:51.006405+0000) 2022-09-10T06:50:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:51 smithi104 ceph-mon[97044]: pgmap v4535: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:51 smithi110 ceph-mon[97633]: pgmap v4535: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:52.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:51.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:51.606899+0000) 2022-09-10T06:50:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:52.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:52.006657+0000) 2022-09-10T06:50:53.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:52.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:52.607121+0000) 2022-09-10T06:50:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:53.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:53.006840+0000) 2022-09-10T06:50:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:53 smithi104 ceph-mon[97044]: pgmap v4536: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:53 smithi110 ceph-mon[97633]: pgmap v4536: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:54.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:53.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:53.607370+0000) 2022-09-10T06:50:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:54.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:54.007016+0000) 2022-09-10T06:50:55.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:54.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:54.607610+0000) 2022-09-10T06:50:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:55.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:55.007217+0000) 2022-09-10T06:50:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:55 smithi104 ceph-mon[97044]: pgmap v4537: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:55 smithi110 ceph-mon[97633]: pgmap v4537: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:50:56.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:55.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:55.607813+0000) 2022-09-10T06:50:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:56.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:56.007387+0000) 2022-09-10T06:50:57.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:56.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:56.608027+0000) 2022-09-10T06:50:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:57.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:57.007574+0000) 2022-09-10T06:50:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:57 smithi104 ceph-mon[97044]: pgmap v4538: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:57 smithi110 ceph-mon[97633]: pgmap v4538: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:58.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:57.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:57.608187+0000) 2022-09-10T06:50:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:58.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:58.007835+0000) 2022-09-10T06:50:59.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:58.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:58.608426+0000) 2022-09-10T06:50:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:50:59.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:59.008020+0000) 2022-09-10T06:50:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:50:59 smithi104 ceph-mon[97044]: pgmap v4539: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:50:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:50:59 smithi110 ceph-mon[97633]: pgmap v4539: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:00.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:50:59.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:50:59.608685+0000) 2022-09-10T06:51:00.009 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:50:59.617+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:51:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:00.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:00.008221+0000) 2022-09-10T06:51:01.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:00.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:00.608926+0000) 2022-09-10T06:51:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:01.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:01.008431+0000) 2022-09-10T06:51:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:01 smithi104 ceph-mon[97044]: pgmap v4540: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:01 smithi110 ceph-mon[97633]: pgmap v4540: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:02.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:01.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:01.609188+0000) 2022-09-10T06:51:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:02.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:02.008652+0000) 2022-09-10T06:51:03.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:02.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:02.609452+0000) 2022-09-10T06:51:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:03.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:03.008808+0000) 2022-09-10T06:51:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:03 smithi104 ceph-mon[97044]: pgmap v4541: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:03 smithi110 ceph-mon[97633]: pgmap v4541: 129 pgs: 129 active+clean; 8.1 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:04.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:03.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:03.609647+0000) 2022-09-10T06:51:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:04.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:04.009026+0000) 2022-09-10T06:51:05.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:04.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:04.609890+0000) 2022-09-10T06:51:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:05.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:05.009290+0000) 2022-09-10T06:51:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:05 smithi104 ceph-mon[97044]: pgmap v4542: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:51:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:05 smithi110 ceph-mon[97633]: pgmap v4542: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:51:06.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:05.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:05.610130+0000) 2022-09-10T06:51:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:06.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:06.009465+0000) 2022-09-10T06:51:07.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:06.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:06.610384+0000) 2022-09-10T06:51:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:07.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:07.009605+0000) 2022-09-10T06:51:07.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:07 smithi104 ceph-mon[97044]: pgmap v4543: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:51:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:07 smithi110 ceph-mon[97633]: pgmap v4543: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:51:08.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:07.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:07.610545+0000) 2022-09-10T06:51:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:08.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:08.009732+0000) 2022-09-10T06:51:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:08 smithi104 ceph-mon[97044]: osdmap e366: 12 total, 12 up, 12 in 2022-09-10T06:51:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:08 smithi110 ceph-mon[97633]: osdmap e366: 12 total, 12 up, 12 in 2022-09-10T06:51:09.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:08.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:08.610745+0000) 2022-09-10T06:51:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:09.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:09.009856+0000) 2022-09-10T06:51:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:09 smithi104 ceph-mon[97044]: pgmap v4545: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:51:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:09 smithi110 ceph-mon[97633]: pgmap v4545: 129 pgs: 129 active+clean; 8.1 MiB data, 580 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:51:10.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:09.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:09.610914+0000) 2022-09-10T06:51:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:10.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:10.009989+0000) 2022-09-10T06:51:11.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:10.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:10.611161+0000) 2022-09-10T06:51:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:11.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:11.010121+0000) 2022-09-10T06:51:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:11 smithi104 ceph-mon[97044]: pgmap v4546: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:51:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:11 smithi110 ceph-mon[97633]: pgmap v4546: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:51:12.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:11.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:11.611410+0000) 2022-09-10T06:51:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:12.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:12.010245+0000) 2022-09-10T06:51:13.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:12.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:12.611616+0000) 2022-09-10T06:51:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:13.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:13.010387+0000) 2022-09-10T06:51:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:13 smithi104 ceph-mon[97044]: pgmap v4547: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:51:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:13 smithi110 ceph-mon[97633]: pgmap v4547: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:51:14.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:13.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:13.611804+0000) 2022-09-10T06:51:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:14.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:14.010508+0000) 2022-09-10T06:51:15.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:14.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:14.612045+0000) 2022-09-10T06:51:15.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:51:14.632+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:51:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:15.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:15.010631+0000) 2022-09-10T06:51:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:15 smithi104 ceph-mon[97044]: osdmap e367: 12 total, 12 up, 12 in 2022-09-10T06:51:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:15 smithi104 ceph-mon[97044]: pgmap v4549: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:51:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:15 smithi110 ceph-mon[97633]: osdmap e367: 12 total, 12 up, 12 in 2022-09-10T06:51:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:15 smithi110 ceph-mon[97633]: pgmap v4549: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:51:16.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:15.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:15.612258+0000) 2022-09-10T06:51:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:16.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:16.010750+0000) 2022-09-10T06:51:17.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:16.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:16.612505+0000) 2022-09-10T06:51:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:17.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:17.010890+0000) 2022-09-10T06:51:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:17 smithi104 ceph-mon[97044]: pgmap v4550: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:51:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:17 smithi110 ceph-mon[97633]: pgmap v4550: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:51:18.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:17.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:17.612710+0000) 2022-09-10T06:51:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:18.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:18.011016+0000) 2022-09-10T06:51:19.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:18.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:18.612973+0000) 2022-09-10T06:51:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:19.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:19.011138+0000) 2022-09-10T06:51:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:19 smithi104 ceph-mon[97044]: pgmap v4551: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:51:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:19 smithi110 ceph-mon[97633]: pgmap v4551: 129 pgs: 129 active+clean; 8.1 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:51:20.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:19.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:19.613223+0000) 2022-09-10T06:51:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:20.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:20.011266+0000) 2022-09-10T06:51:21.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:20.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:20.613436+0000) 2022-09-10T06:51:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:21.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:21.011394+0000) 2022-09-10T06:51:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:21 smithi104 ceph-mon[97044]: pgmap v4552: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:51:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:21 smithi110 ceph-mon[97633]: pgmap v4552: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:51:22.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:21.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:21.613682+0000) 2022-09-10T06:51:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:22.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:22.011586+0000) 2022-09-10T06:51:23.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:22.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:22.613881+0000) 2022-09-10T06:51:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:23.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:23.011798+0000) 2022-09-10T06:51:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:23 smithi104 ceph-mon[97044]: pgmap v4553: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:51:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:23 smithi110 ceph-mon[97633]: pgmap v4553: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:51:24.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:23.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:23.614147+0000) 2022-09-10T06:51:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:24.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:24.012004+0000) 2022-09-10T06:51:25.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:24.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:24.614404+0000) 2022-09-10T06:51:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:25.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:25.012168+0000) 2022-09-10T06:51:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:25 smithi104 ceph-mon[97044]: pgmap v4554: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:51:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:25 smithi110 ceph-mon[97633]: pgmap v4554: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:51:26.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:25.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:25.614630+0000) 2022-09-10T06:51:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:26.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:26.012288+0000) 2022-09-10T06:51:27.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:26.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:26.614853+0000) 2022-09-10T06:51:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:27.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:27.012413+0000) 2022-09-10T06:51:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:27 smithi104 ceph-mon[97044]: pgmap v4555: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:27 smithi110 ceph-mon[97633]: pgmap v4555: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:28.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:27.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:27.615122+0000) 2022-09-10T06:51:28.313 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:28.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:28.012610+0000) 2022-09-10T06:51:29.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:28.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:28.615296+0000) 2022-09-10T06:51:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:29.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:29.012832+0000) 2022-09-10T06:51:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:29 smithi104 ceph-mon[97044]: pgmap v4556: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:29 smithi110 ceph-mon[97633]: pgmap v4556: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:30.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:29.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:29.615536+0000) 2022-09-10T06:51:30.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:51:29.637+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:51:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:30.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:30.012975+0000) 2022-09-10T06:51:31.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:30.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:30.615805+0000) 2022-09-10T06:51:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:31.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:31.013108+0000) 2022-09-10T06:51:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:31 smithi104 ceph-mon[97044]: pgmap v4557: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:31 smithi110 ceph-mon[97633]: pgmap v4557: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:32.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:31.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:31.616049+0000) 2022-09-10T06:51:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:32.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:32.013280+0000) 2022-09-10T06:51:33.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:32.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:32.616237+0000) 2022-09-10T06:51:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:33.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:33.013440+0000) 2022-09-10T06:51:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:33 smithi104 ceph-mon[97044]: pgmap v4558: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:33 smithi110 ceph-mon[97633]: pgmap v4558: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:34.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:33.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:33.616375+0000) 2022-09-10T06:51:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:34.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:34.013569+0000) 2022-09-10T06:51:35.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:34.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:34.616613+0000) 2022-09-10T06:51:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:35.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:35.013715+0000) 2022-09-10T06:51:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:35 smithi104 ceph-mon[97044]: pgmap v4559: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:35 smithi110 ceph-mon[97633]: pgmap v4559: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:36.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:35.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:35.616857+0000) 2022-09-10T06:51:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:36.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:36.013854+0000) 2022-09-10T06:51:37.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:36.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:36.617135+0000) 2022-09-10T06:51:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:37.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:37.013982+0000) 2022-09-10T06:51:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:37 smithi104 ceph-mon[97044]: pgmap v4560: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:37 smithi110 ceph-mon[97633]: pgmap v4560: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:38.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:37.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:37.617337+0000) 2022-09-10T06:51:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:38.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:38.014123+0000) 2022-09-10T06:51:39.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:38.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:38.617549+0000) 2022-09-10T06:51:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:39.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:39.014244+0000) 2022-09-10T06:51:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:39 smithi104 ceph-mon[97044]: pgmap v4561: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:39 smithi110 ceph-mon[97633]: pgmap v4561: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:40.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:39.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:39.617787+0000) 2022-09-10T06:51:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:40.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:40.014381+0000) 2022-09-10T06:51:41.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:40.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:40.618065+0000) 2022-09-10T06:51:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:41.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:41.014512+0000) 2022-09-10T06:51:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:41 smithi104 ceph-mon[97044]: pgmap v4562: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:41 smithi110 ceph-mon[97633]: pgmap v4562: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:42.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:41.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:41.618329+0000) 2022-09-10T06:51:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:42.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:42.014641+0000) 2022-09-10T06:51:43.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:42.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:42.618551+0000) 2022-09-10T06:51:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:43.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:43.014830+0000) 2022-09-10T06:51:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:43 smithi104 ceph-mon[97044]: pgmap v4563: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:43 smithi110 ceph-mon[97633]: pgmap v4563: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:44.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:43.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:43.618825+0000) 2022-09-10T06:51:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:44.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:44.014999+0000) 2022-09-10T06:51:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:51:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:51:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:51:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:51:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:51:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:51:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:51:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:51:45.016 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:51:44.646+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:51:45.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:44.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:44.618997+0000) 2022-09-10T06:51:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:45.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:45.015204+0000) 2022-09-10T06:51:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:45 smithi104 ceph-mon[97044]: pgmap v4564: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:45 smithi110 ceph-mon[97633]: pgmap v4564: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:46.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:45.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:45.619232+0000) 2022-09-10T06:51:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:46.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:46.015343+0000) 2022-09-10T06:51:47.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:46.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:46.619475+0000) 2022-09-10T06:51:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:47.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:47.015496+0000) 2022-09-10T06:51:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:47 smithi104 ceph-mon[97044]: pgmap v4565: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:47 smithi110 ceph-mon[97633]: pgmap v4565: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:48.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:47.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:47.619763+0000) 2022-09-10T06:51:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:48.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:48.015657+0000) 2022-09-10T06:51:49.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:48.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:48.619996+0000) 2022-09-10T06:51:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:49.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:49.015798+0000) 2022-09-10T06:51:49.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:49 smithi110 ceph-mon[97633]: pgmap v4566: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:50.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:49.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:49.620244+0000) 2022-09-10T06:51:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:49 smithi104 ceph-mon[97044]: pgmap v4566: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:50.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:50.015935+0000) 2022-09-10T06:51:51.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:50.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:50.620457+0000) 2022-09-10T06:51:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:51.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:51.016065+0000) 2022-09-10T06:51:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:51 smithi110 ceph-mon[97633]: pgmap v4567: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:52.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:51.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:51.620607+0000) 2022-09-10T06:51:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:51 smithi104 ceph-mon[97044]: pgmap v4567: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:52.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:52.016171+0000) 2022-09-10T06:51:53.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:52.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:52.620791+0000) 2022-09-10T06:51:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:53.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:53.016308+0000) 2022-09-10T06:51:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:53 smithi110 ceph-mon[97633]: pgmap v4568: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:54.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:53.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:53.620970+0000) 2022-09-10T06:51:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:53 smithi104 ceph-mon[97044]: pgmap v4568: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:54.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:54.016495+0000) 2022-09-10T06:51:55.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:54.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:54.621202+0000) 2022-09-10T06:51:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:55.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:55.016666+0000) 2022-09-10T06:51:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:55 smithi104 ceph-mon[97044]: pgmap v4569: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:55 smithi110 ceph-mon[97633]: pgmap v4569: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:51:56.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:55.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:55.621454+0000) 2022-09-10T06:51:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:56.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:56.016838+0000) 2022-09-10T06:51:57.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:56.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:56.621729+0000) 2022-09-10T06:51:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:57.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:57.016970+0000) 2022-09-10T06:51:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:57 smithi104 ceph-mon[97044]: pgmap v4570: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:57 smithi110 ceph-mon[97633]: pgmap v4570: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:58.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:57.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:57.622021+0000) 2022-09-10T06:51:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:58.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:58.017121+0000) 2022-09-10T06:51:59.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:58.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:58.622286+0000) 2022-09-10T06:51:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:51:59.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:59.017278+0000) 2022-09-10T06:51:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:51:59 smithi104 ceph-mon[97044]: pgmap v4571: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:51:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:51:59 smithi110 ceph-mon[97633]: pgmap v4571: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:00.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:51:59.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:51:59.622539+0000) 2022-09-10T06:52:00.019 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:51:59.662+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:52:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:00.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:00.017410+0000) 2022-09-10T06:52:01.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:00.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:00.622782+0000) 2022-09-10T06:52:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:01.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:01.017580+0000) 2022-09-10T06:52:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:01 smithi104 ceph-mon[97044]: pgmap v4572: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:01 smithi110 ceph-mon[97633]: pgmap v4572: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:02.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:01.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:01.622965+0000) 2022-09-10T06:52:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:02.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:02.017716+0000) 2022-09-10T06:52:03.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:02.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:02.623153+0000) 2022-09-10T06:52:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:03.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:03.017837+0000) 2022-09-10T06:52:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:03 smithi104 ceph-mon[97044]: pgmap v4573: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:52:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:03 smithi110 ceph-mon[97633]: pgmap v4573: 129 pgs: 129 active+clean; 8.1 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:52:04.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:03.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:03.623385+0000) 2022-09-10T06:52:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:04.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:04.017985+0000) 2022-09-10T06:52:05.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:04.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:04.623678+0000) 2022-09-10T06:52:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:05.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:05.018151+0000) 2022-09-10T06:52:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:05 smithi104 ceph-mon[97044]: pgmap v4574: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:52:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:05 smithi110 ceph-mon[97633]: pgmap v4574: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:52:06.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:05.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:05.623966+0000) 2022-09-10T06:52:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:06.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:06.018267+0000) 2022-09-10T06:52:07.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:06.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:06.624233+0000) 2022-09-10T06:52:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:07.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:07.018383+0000) 2022-09-10T06:52:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:07 smithi110 ceph-mon[97633]: pgmap v4575: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:52:08.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:07.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:07.624438+0000) 2022-09-10T06:52:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:07 smithi104 ceph-mon[97044]: pgmap v4575: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:52:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:08.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:08.018507+0000) 2022-09-10T06:52:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:08 smithi110 ceph-mon[97633]: osdmap e368: 12 total, 12 up, 12 in 2022-09-10T06:52:09.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:08.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:08.624689+0000) 2022-09-10T06:52:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:08 smithi104 ceph-mon[97044]: osdmap e368: 12 total, 12 up, 12 in 2022-09-10T06:52:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:09.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:09.018647+0000) 2022-09-10T06:52:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:09 smithi110 ceph-mon[97633]: pgmap v4577: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:52:10.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:09.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:09.624967+0000) 2022-09-10T06:52:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:09 smithi104 ceph-mon[97044]: pgmap v4577: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:52:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:10.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:10.018798+0000) 2022-09-10T06:52:11.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:10.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:10.625242+0000) 2022-09-10T06:52:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:11.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:11.018909+0000) 2022-09-10T06:52:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:11 smithi110 ceph-mon[97633]: pgmap v4578: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:52:12.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:11.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:11.625523+0000) 2022-09-10T06:52:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:11 smithi104 ceph-mon[97044]: pgmap v4578: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:52:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:12.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:12.019056+0000) 2022-09-10T06:52:13.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:12.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:12.625792+0000) 2022-09-10T06:52:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:13.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:13.019192+0000) 2022-09-10T06:52:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:13 smithi110 ceph-mon[97633]: pgmap v4579: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:52:14.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:13.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:13.626027+0000) 2022-09-10T06:52:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:13 smithi104 ceph-mon[97044]: pgmap v4579: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:52:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:14.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:14.019334+0000) 2022-09-10T06:52:15.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:14.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:14.626206+0000) 2022-09-10T06:52:15.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:52:14.671+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:52:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:15.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:15.019594+0000) 2022-09-10T06:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:15 smithi104 ceph-mon[97044]: osdmap e369: 12 total, 12 up, 12 in 2022-09-10T06:52:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:15 smithi104 ceph-mon[97044]: pgmap v4581: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:52:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:15 smithi110 ceph-mon[97633]: osdmap e369: 12 total, 12 up, 12 in 2022-09-10T06:52:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:15 smithi110 ceph-mon[97633]: pgmap v4581: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:52:16.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:15.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:15.626396+0000) 2022-09-10T06:52:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:16.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:16.019723+0000) 2022-09-10T06:52:17.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:16.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:16.626569+0000) 2022-09-10T06:52:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:17.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:17.019847+0000) 2022-09-10T06:52:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:17 smithi104 ceph-mon[97044]: pgmap v4582: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:52:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:17 smithi110 ceph-mon[97633]: pgmap v4582: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:52:18.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:17.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:17.626751+0000) 2022-09-10T06:52:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:18.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:18.019974+0000) 2022-09-10T06:52:19.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:18.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:18.626944+0000) 2022-09-10T06:52:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:19.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:19.020090+0000) 2022-09-10T06:52:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:19 smithi110 ceph-mon[97633]: pgmap v4583: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:52:20.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:19.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:19.627144+0000) 2022-09-10T06:52:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:19 smithi104 ceph-mon[97044]: pgmap v4583: 129 pgs: 129 active+clean; 8.1 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:52:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:20.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:20.020258+0000) 2022-09-10T06:52:21.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:20.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:20.627417+0000) 2022-09-10T06:52:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:21.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:21.020445+0000) 2022-09-10T06:52:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:21 smithi110 ceph-mon[97633]: pgmap v4584: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:52:22.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:21.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:21.627656+0000) 2022-09-10T06:52:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:21 smithi104 ceph-mon[97044]: pgmap v4584: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:52:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:22.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:22.020576+0000) 2022-09-10T06:52:23.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:22.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:22.627846+0000) 2022-09-10T06:52:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:23.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:23.020696+0000) 2022-09-10T06:52:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:23 smithi110 ceph-mon[97633]: pgmap v4585: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:52:24.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:23.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:23.628049+0000) 2022-09-10T06:52:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:23 smithi104 ceph-mon[97044]: pgmap v4585: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:52:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:24.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:24.020835+0000) 2022-09-10T06:52:25.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:24.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:24.628335+0000) 2022-09-10T06:52:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:25.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:25.020994+0000) 2022-09-10T06:52:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:25 smithi104 ceph-mon[97044]: pgmap v4586: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:52:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:25 smithi110 ceph-mon[97633]: pgmap v4586: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:52:26.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:25.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:25.628533+0000) 2022-09-10T06:52:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:26.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:26.021152+0000) 2022-09-10T06:52:27.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:26.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:26.628725+0000) 2022-09-10T06:52:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:27.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:27.021303+0000) 2022-09-10T06:52:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:27 smithi110 ceph-mon[97633]: pgmap v4587: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:28.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:27.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:27.628960+0000) 2022-09-10T06:52:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:27 smithi104 ceph-mon[97044]: pgmap v4587: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:28.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:28.021523+0000) 2022-09-10T06:52:29.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:28.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:28.629160+0000) 2022-09-10T06:52:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:29.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:29.021683+0000) 2022-09-10T06:52:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:29 smithi110 ceph-mon[97633]: pgmap v4588: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:30.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:29.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:29.629375+0000) 2022-09-10T06:52:30.024 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:52:29.683+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:52:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:29 smithi104 ceph-mon[97044]: pgmap v4588: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:30.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:30.021810+0000) 2022-09-10T06:52:31.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:30.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:30.629584+0000) 2022-09-10T06:52:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:31.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:31.021941+0000) 2022-09-10T06:52:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:31 smithi110 ceph-mon[97633]: pgmap v4589: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:32.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:31.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:31.629819+0000) 2022-09-10T06:52:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:31 smithi104 ceph-mon[97044]: pgmap v4589: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:32.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:32.022073+0000) 2022-09-10T06:52:33.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:32.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:32.630054+0000) 2022-09-10T06:52:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:33.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:33.022228+0000) 2022-09-10T06:52:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:33 smithi110 ceph-mon[97633]: pgmap v4590: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:34.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:33.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:33.630311+0000) 2022-09-10T06:52:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:33 smithi104 ceph-mon[97044]: pgmap v4590: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:34.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:34.022377+0000) 2022-09-10T06:52:35.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:34.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:34.630567+0000) 2022-09-10T06:52:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:35.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:35.022551+0000) 2022-09-10T06:52:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:35 smithi104 ceph-mon[97044]: pgmap v4591: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:35 smithi110 ceph-mon[97633]: pgmap v4591: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:36.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:35.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:35.630749+0000) 2022-09-10T06:52:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:36.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:36.022683+0000) 2022-09-10T06:52:37.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:36.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:36.631008+0000) 2022-09-10T06:52:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:37.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:37.022836+0000) 2022-09-10T06:52:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:37 smithi110 ceph-mon[97633]: pgmap v4592: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:38.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:37.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:37.631267+0000) 2022-09-10T06:52:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:37 smithi104 ceph-mon[97044]: pgmap v4592: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:38.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:38.023000+0000) 2022-09-10T06:52:39.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:38.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:38.631496+0000) 2022-09-10T06:52:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:39.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:39.023141+0000) 2022-09-10T06:52:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:39 smithi110 ceph-mon[97633]: pgmap v4593: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:40.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:39.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:39.631698+0000) 2022-09-10T06:52:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:39 smithi104 ceph-mon[97044]: pgmap v4593: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:40.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:40.023278+0000) 2022-09-10T06:52:41.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:40.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:40.631918+0000) 2022-09-10T06:52:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:41.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:41.023422+0000) 2022-09-10T06:52:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:41 smithi110 ceph-mon[97633]: pgmap v4594: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:42.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:41.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:41.632182+0000) 2022-09-10T06:52:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:41 smithi104 ceph-mon[97044]: pgmap v4594: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:42.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:42.023558+0000) 2022-09-10T06:52:43.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:42.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:42.632365+0000) 2022-09-10T06:52:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:43.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:43.023684+0000) 2022-09-10T06:52:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:43 smithi110 ceph-mon[97633]: pgmap v4595: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:44.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:43.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:43.632570+0000) 2022-09-10T06:52:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:43 smithi104 ceph-mon[97044]: pgmap v4595: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:44.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:44.023881+0000) 2022-09-10T06:52:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:52:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:52:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:52:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:52:45.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:44.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:44.632729+0000) 2022-09-10T06:52:45.026 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:52:44.694+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:52:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:52:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:52:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:52:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:52:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:45.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:45.024031+0000) 2022-09-10T06:52:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:45 smithi110 ceph-mon[97633]: pgmap v4596: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:46.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:45.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:45.632965+0000) 2022-09-10T06:52:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:45 smithi104 ceph-mon[97044]: pgmap v4596: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:46.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:46.024164+0000) 2022-09-10T06:52:47.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:46.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:46.633193+0000) 2022-09-10T06:52:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:47.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:47.024281+0000) 2022-09-10T06:52:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:47 smithi110 ceph-mon[97633]: pgmap v4597: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:48.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:47.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:47.633418+0000) 2022-09-10T06:52:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:47 smithi104 ceph-mon[97044]: pgmap v4597: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:48.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:48.024411+0000) 2022-09-10T06:52:49.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:48.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:48.633610+0000) 2022-09-10T06:52:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:49.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:49.024552+0000) 2022-09-10T06:52:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:49 smithi110 ceph-mon[97633]: pgmap v4598: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:50.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:49.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:49.633780+0000) 2022-09-10T06:52:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:49 smithi104 ceph-mon[97044]: pgmap v4598: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:50.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:50.024716+0000) 2022-09-10T06:52:51.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:50.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:50.634135+0000) 2022-09-10T06:52:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:51.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:51.024829+0000) 2022-09-10T06:52:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:51 smithi110 ceph-mon[97633]: pgmap v4599: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:52.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:51.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:51.634361+0000) 2022-09-10T06:52:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:51 smithi104 ceph-mon[97044]: pgmap v4599: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:52.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:52.024952+0000) 2022-09-10T06:52:53.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:52.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:52.634569+0000) 2022-09-10T06:52:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:53.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:53.025160+0000) 2022-09-10T06:52:53.915 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:53 smithi110 ceph-mon[97633]: pgmap v4600: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:54.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:53.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:53.634794+0000) 2022-09-10T06:52:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:53 smithi104 ceph-mon[97044]: pgmap v4600: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:54.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:54.025303+0000) 2022-09-10T06:52:55.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:54.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:54.635061+0000) 2022-09-10T06:52:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:55.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:55.025489+0000) 2022-09-10T06:52:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:55 smithi104 ceph-mon[97044]: pgmap v4601: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:55 smithi110 ceph-mon[97633]: pgmap v4601: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:52:56.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:55.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:55.635353+0000) 2022-09-10T06:52:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:56.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:56.025636+0000) 2022-09-10T06:52:57.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:56.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:56.635627+0000) 2022-09-10T06:52:57.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:57.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:57.025773+0000) 2022-09-10T06:52:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:57 smithi110 ceph-mon[97633]: pgmap v4602: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:58.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:57.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:57.635869+0000) 2022-09-10T06:52:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:57 smithi104 ceph-mon[97044]: pgmap v4602: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:52:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:58.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:58.025906+0000) 2022-09-10T06:52:59.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:58.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:58.636144+0000) 2022-09-10T06:52:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:52:59.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:59.026076+0000) 2022-09-10T06:52:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:52:59 smithi110 ceph-mon[97633]: pgmap v4603: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:00.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:52:59.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:52:59.636394+0000) 2022-09-10T06:53:00.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:52:59.710+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:53:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:52:59 smithi104 ceph-mon[97044]: pgmap v4603: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:00.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:00.026228+0000) 2022-09-10T06:53:01.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:00.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:00.636583+0000) 2022-09-10T06:53:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:01.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:01.026371+0000) 2022-09-10T06:53:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:01 smithi110 ceph-mon[97633]: pgmap v4604: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:02.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:01.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:01.636826+0000) 2022-09-10T06:53:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:01 smithi104 ceph-mon[97044]: pgmap v4604: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:02.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:02.026514+0000) 2022-09-10T06:53:03.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:02.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:02.637066+0000) 2022-09-10T06:53:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:03.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:03.026666+0000) 2022-09-10T06:53:03.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:03 smithi110 ceph-mon[97633]: pgmap v4605: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:04.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:03.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:03.637366+0000) 2022-09-10T06:53:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:03 smithi104 ceph-mon[97044]: pgmap v4605: 129 pgs: 129 active+clean; 8.1 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:04.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:04.026807+0000) 2022-09-10T06:53:05.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:04.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:04.637640+0000) 2022-09-10T06:53:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:05.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:05.026951+0000) 2022-09-10T06:53:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:05 smithi104 ceph-mon[97044]: pgmap v4606: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:53:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:05 smithi110 ceph-mon[97633]: pgmap v4606: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:53:06.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:05.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:05.637836+0000) 2022-09-10T06:53:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:06.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:06.027077+0000) 2022-09-10T06:53:07.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:06.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:06.638108+0000) 2022-09-10T06:53:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:07.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:07.027227+0000) 2022-09-10T06:53:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:07 smithi110 ceph-mon[97633]: pgmap v4607: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:53:08.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:07.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:07.638335+0000) 2022-09-10T06:53:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:07 smithi104 ceph-mon[97044]: pgmap v4607: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:53:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:08.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:08.027425+0000) 2022-09-10T06:53:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:08 smithi110 ceph-mon[97633]: osdmap e370: 12 total, 12 up, 12 in 2022-09-10T06:53:09.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:08.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:08.638572+0000) 2022-09-10T06:53:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:08 smithi104 ceph-mon[97044]: osdmap e370: 12 total, 12 up, 12 in 2022-09-10T06:53:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:09.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:09.027636+0000) 2022-09-10T06:53:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:09 smithi110 ceph-mon[97633]: pgmap v4609: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:53:10.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:09.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:09.638779+0000) 2022-09-10T06:53:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:09 smithi104 ceph-mon[97044]: pgmap v4609: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:53:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:10.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:10.027767+0000) 2022-09-10T06:53:11.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:10.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:10.638954+0000) 2022-09-10T06:53:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:11.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:11.027884+0000) 2022-09-10T06:53:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:11 smithi110 ceph-mon[97633]: pgmap v4610: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:53:12.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:11.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:11.639205+0000) 2022-09-10T06:53:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:11 smithi104 ceph-mon[97044]: pgmap v4610: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:53:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:12.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:12.028010+0000) 2022-09-10T06:53:13.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:12.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:12.639380+0000) 2022-09-10T06:53:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:13.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:13.028224+0000) 2022-09-10T06:53:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:13 smithi110 ceph-mon[97633]: pgmap v4611: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:53:14.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:13.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:13.639608+0000) 2022-09-10T06:53:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:13 smithi104 ceph-mon[97044]: pgmap v4611: 129 pgs: 129 active+clean; 8.1 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:53:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:14.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:14.028353+0000) 2022-09-10T06:53:15.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:14.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:14.639832+0000) 2022-09-10T06:53:15.030 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:53:14.726+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:53:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:15.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:15.028480+0000) 2022-09-10T06:53:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:15 smithi104 ceph-mon[97044]: osdmap e371: 12 total, 12 up, 12 in 2022-09-10T06:53:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:15 smithi104 ceph-mon[97044]: pgmap v4613: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:53:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:15 smithi110 ceph-mon[97633]: osdmap e371: 12 total, 12 up, 12 in 2022-09-10T06:53:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:15 smithi110 ceph-mon[97633]: pgmap v4613: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:53:16.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:15.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:15.640053+0000) 2022-09-10T06:53:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:16.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:16.028609+0000) 2022-09-10T06:53:17.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:16.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:16.640251+0000) 2022-09-10T06:53:17.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:17.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:17.028736+0000) 2022-09-10T06:53:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:17 smithi110 ceph-mon[97633]: pgmap v4614: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:53:18.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:17.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:17.640396+0000) 2022-09-10T06:53:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:17 smithi104 ceph-mon[97044]: pgmap v4614: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:53:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:18.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:18.028861+0000) 2022-09-10T06:53:19.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:18.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:18.640625+0000) 2022-09-10T06:53:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:19.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:19.028982+0000) 2022-09-10T06:53:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:19 smithi110 ceph-mon[97633]: pgmap v4615: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:53:20.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:19.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:19.640853+0000) 2022-09-10T06:53:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:19 smithi104 ceph-mon[97044]: pgmap v4615: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:53:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:20.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:20.029120+0000) 2022-09-10T06:53:21.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:20.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:20.641138+0000) 2022-09-10T06:53:21.313 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:21.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:21.029249+0000) 2022-09-10T06:53:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:21 smithi110 ceph-mon[97633]: pgmap v4616: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:53:22.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:21.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:21.641358+0000) 2022-09-10T06:53:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:21 smithi104 ceph-mon[97044]: pgmap v4616: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:53:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:22.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:22.029425+0000) 2022-09-10T06:53:23.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:22.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:22.641627+0000) 2022-09-10T06:53:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:23.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:23.029541+0000) 2022-09-10T06:53:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:23 smithi110 ceph-mon[97633]: pgmap v4617: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:53:24.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:23.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:23.641767+0000) 2022-09-10T06:53:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:23 smithi104 ceph-mon[97044]: pgmap v4617: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:53:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:24.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:24.029671+0000) 2022-09-10T06:53:25.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:24.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:24.641996+0000) 2022-09-10T06:53:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:25.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:25.029873+0000) 2022-09-10T06:53:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:25 smithi104 ceph-mon[97044]: pgmap v4618: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:53:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:25 smithi110 ceph-mon[97633]: pgmap v4618: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:53:26.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:25.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:25.642273+0000) 2022-09-10T06:53:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:26.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:26.030035+0000) 2022-09-10T06:53:27.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:26.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:26.642494+0000) 2022-09-10T06:53:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:27.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:27.030154+0000) 2022-09-10T06:53:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:27 smithi110 ceph-mon[97633]: pgmap v4619: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:28.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:27.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:27.642745+0000) 2022-09-10T06:53:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:27 smithi104 ceph-mon[97044]: pgmap v4619: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:28.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:28.030277+0000) 2022-09-10T06:53:29.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:28.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:28.642970+0000) 2022-09-10T06:53:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:29.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:29.030404+0000) 2022-09-10T06:53:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:29 smithi110 ceph-mon[97633]: pgmap v4620: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:30.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:29.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:29.643198+0000) 2022-09-10T06:53:30.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:53:29.737+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:53:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:29 smithi104 ceph-mon[97044]: pgmap v4620: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:30.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:30.030533+0000) 2022-09-10T06:53:31.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:30.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:30.643402+0000) 2022-09-10T06:53:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:31.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:31.030677+0000) 2022-09-10T06:53:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:31 smithi110 ceph-mon[97633]: pgmap v4621: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:32.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:31.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:31.643615+0000) 2022-09-10T06:53:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:31 smithi104 ceph-mon[97044]: pgmap v4621: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:32.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:32.030810+0000) 2022-09-10T06:53:33.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:32.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:32.643867+0000) 2022-09-10T06:53:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:33.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:33.031117+0000) 2022-09-10T06:53:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:33 smithi110 ceph-mon[97633]: pgmap v4622: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:34.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:33.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:33.644077+0000) 2022-09-10T06:53:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:33 smithi104 ceph-mon[97044]: pgmap v4622: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:34.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:34.031427+0000) 2022-09-10T06:53:35.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:34.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:34.644343+0000) 2022-09-10T06:53:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:35.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:35.031581+0000) 2022-09-10T06:53:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:35 smithi104 ceph-mon[97044]: pgmap v4623: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:35 smithi110 ceph-mon[97633]: pgmap v4623: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:36.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:35.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:35.644570+0000) 2022-09-10T06:53:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:36.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:36.031715+0000) 2022-09-10T06:53:37.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:36.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:36.644712+0000) 2022-09-10T06:53:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:37.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:37.031951+0000) 2022-09-10T06:53:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:37 smithi110 ceph-mon[97633]: pgmap v4624: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:38.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:37.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:37.644906+0000) 2022-09-10T06:53:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:37 smithi104 ceph-mon[97044]: pgmap v4624: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:38.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:38.032081+0000) 2022-09-10T06:53:39.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:38.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:38.645159+0000) 2022-09-10T06:53:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:39.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:39.032224+0000) 2022-09-10T06:53:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:39 smithi110 ceph-mon[97633]: pgmap v4625: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:40.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:39.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:39.645345+0000) 2022-09-10T06:53:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:39 smithi104 ceph-mon[97044]: pgmap v4625: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:40.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:40.032359+0000) 2022-09-10T06:53:41.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:40.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:40.645552+0000) 2022-09-10T06:53:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:41.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:41.032478+0000) 2022-09-10T06:53:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:41 smithi110 ceph-mon[97633]: pgmap v4626: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:42.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:41.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:41.645758+0000) 2022-09-10T06:53:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:41 smithi104 ceph-mon[97044]: pgmap v4626: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:42.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:42.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:42.032602+0000) 2022-09-10T06:53:43.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:42.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:42.645942+0000) 2022-09-10T06:53:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:43.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:43.032720+0000) 2022-09-10T06:53:43.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:43 smithi110 ceph-mon[97633]: pgmap v4627: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:44.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:43.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:43.646150+0000) 2022-09-10T06:53:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:43 smithi104 ceph-mon[97044]: pgmap v4627: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:44.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:44.032840+0000) 2022-09-10T06:53:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:53:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:53:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:53:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:53:45.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:44.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:44.646368+0000) 2022-09-10T06:53:45.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:53:44.752+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:53:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:53:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:53:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:53:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:53:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:45.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:45.033020+0000) 2022-09-10T06:53:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:45 smithi110 ceph-mon[97633]: pgmap v4628: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:46.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:45.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:45.646537+0000) 2022-09-10T06:53:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:45 smithi104 ceph-mon[97044]: pgmap v4628: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:46.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:46.033194+0000) 2022-09-10T06:53:47.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:46.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:46.646767+0000) 2022-09-10T06:53:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:47.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:47.033309+0000) 2022-09-10T06:53:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:47 smithi110 ceph-mon[97633]: pgmap v4629: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:48.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:47.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:47.647012+0000) 2022-09-10T06:53:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:47 smithi104 ceph-mon[97044]: pgmap v4629: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:48.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:48.033442+0000) 2022-09-10T06:53:49.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:48.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:48.647231+0000) 2022-09-10T06:53:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:49.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:49.033587+0000) 2022-09-10T06:53:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:49 smithi110 ceph-mon[97633]: pgmap v4630: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:50.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:49.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:49.647522+0000) 2022-09-10T06:53:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:49 smithi104 ceph-mon[97044]: pgmap v4630: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:50.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:50.033793+0000) 2022-09-10T06:53:51.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:50.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:50.647732+0000) 2022-09-10T06:53:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:51.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:51.033977+0000) 2022-09-10T06:53:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:51 smithi110 ceph-mon[97633]: pgmap v4631: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:52.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:51.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:51.647969+0000) 2022-09-10T06:53:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:51 smithi104 ceph-mon[97044]: pgmap v4631: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:52.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:52.034211+0000) 2022-09-10T06:53:53.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:52.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:52.648178+0000) 2022-09-10T06:53:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:53.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:53.034379+0000) 2022-09-10T06:53:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:53 smithi110 ceph-mon[97633]: pgmap v4632: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:54.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:53.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:53.648438+0000) 2022-09-10T06:53:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:53 smithi104 ceph-mon[97044]: pgmap v4632: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:54.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:54.034519+0000) 2022-09-10T06:53:55.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:54.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:54.648652+0000) 2022-09-10T06:53:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:55.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:55.034640+0000) 2022-09-10T06:53:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:55 smithi104 ceph-mon[97044]: pgmap v4633: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:55 smithi110 ceph-mon[97633]: pgmap v4633: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:53:56.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:55.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:55.648815+0000) 2022-09-10T06:53:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:56.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:56.034764+0000) 2022-09-10T06:53:57.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:56.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:56.649065+0000) 2022-09-10T06:53:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:57.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:57.034903+0000) 2022-09-10T06:53:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:57 smithi110 ceph-mon[97633]: pgmap v4634: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:58.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:57.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:57.649270+0000) 2022-09-10T06:53:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:57 smithi104 ceph-mon[97044]: pgmap v4634: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:53:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:58.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:58.035059+0000) 2022-09-10T06:53:59.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:58.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:58.649463+0000) 2022-09-10T06:53:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:53:59.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:59.035174+0000) 2022-09-10T06:53:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:53:59 smithi110 ceph-mon[97633]: pgmap v4635: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:00.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:53:59.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:53:59.649712+0000) 2022-09-10T06:54:00.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:53:59.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:54:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:53:59 smithi104 ceph-mon[97044]: pgmap v4635: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:00.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:00.035309+0000) 2022-09-10T06:54:01.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:00.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:00.649932+0000) 2022-09-10T06:54:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:01.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:01.035431+0000) 2022-09-10T06:54:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:01 smithi110 ceph-mon[97633]: pgmap v4636: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:02.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:01.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:01.650124+0000) 2022-09-10T06:54:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:01 smithi104 ceph-mon[97044]: pgmap v4636: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:02.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:02.035604+0000) 2022-09-10T06:54:03.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:02.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:02.650284+0000) 2022-09-10T06:54:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:03.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:03.035740+0000) 2022-09-10T06:54:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:03 smithi110 ceph-mon[97633]: pgmap v4637: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:04.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:03.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:03.650536+0000) 2022-09-10T06:54:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:03 smithi104 ceph-mon[97044]: pgmap v4637: 129 pgs: 129 active+clean; 8.1 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:04.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:04.035876+0000) 2022-09-10T06:54:05.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:04.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:04.650792+0000) 2022-09-10T06:54:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:05.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:05.036039+0000) 2022-09-10T06:54:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:05 smithi104 ceph-mon[97044]: pgmap v4638: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:54:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:05 smithi110 ceph-mon[97633]: pgmap v4638: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:54:06.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:05.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:05.651012+0000) 2022-09-10T06:54:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:06.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:06.036165+0000) 2022-09-10T06:54:07.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:06.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:06.651248+0000) 2022-09-10T06:54:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:07.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:07.036299+0000) 2022-09-10T06:54:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:07 smithi110 ceph-mon[97633]: pgmap v4639: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:54:08.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:07.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:07.651442+0000) 2022-09-10T06:54:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:07 smithi104 ceph-mon[97044]: pgmap v4639: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:54:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:08.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:08.036432+0000) 2022-09-10T06:54:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:08 smithi110 ceph-mon[97633]: osdmap e372: 12 total, 12 up, 12 in 2022-09-10T06:54:09.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:08.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:08.651612+0000) 2022-09-10T06:54:09.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:08 smithi104 ceph-mon[97044]: osdmap e372: 12 total, 12 up, 12 in 2022-09-10T06:54:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:09.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:09.036567+0000) 2022-09-10T06:54:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:09 smithi110 ceph-mon[97633]: pgmap v4641: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:54:10.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:09.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:09.651866+0000) 2022-09-10T06:54:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:09 smithi104 ceph-mon[97044]: pgmap v4641: 129 pgs: 129 active+clean; 8.1 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:54:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:10.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:10.036706+0000) 2022-09-10T06:54:11.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:10.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:10.652146+0000) 2022-09-10T06:54:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:11.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:11.036833+0000) 2022-09-10T06:54:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:11 smithi110 ceph-mon[97633]: pgmap v4642: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:54:12.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:11.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:11.652336+0000) 2022-09-10T06:54:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:11 smithi104 ceph-mon[97044]: pgmap v4642: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:54:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:12.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:12.036966+0000) 2022-09-10T06:54:13.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:12.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:12.652518+0000) 2022-09-10T06:54:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:13.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:13.037122+0000) 2022-09-10T06:54:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:13 smithi110 ceph-mon[97633]: pgmap v4643: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:54:14.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:13.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:13.652810+0000) 2022-09-10T06:54:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:13 smithi104 ceph-mon[97044]: pgmap v4643: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:54:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:14.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:14.037263+0000) 2022-09-10T06:54:15.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:14.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:14.653057+0000) 2022-09-10T06:54:15.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:54:14.765+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:54:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:15.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:15.037447+0000) 2022-09-10T06:54:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:15 smithi104 ceph-mon[97044]: osdmap e373: 12 total, 12 up, 12 in 2022-09-10T06:54:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:15 smithi104 ceph-mon[97044]: pgmap v4645: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:54:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:15 smithi110 ceph-mon[97633]: osdmap e373: 12 total, 12 up, 12 in 2022-09-10T06:54:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:15 smithi110 ceph-mon[97633]: pgmap v4645: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:54:16.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:15.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:15.653344+0000) 2022-09-10T06:54:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:16.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:16.037598+0000) 2022-09-10T06:54:17.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:16.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:16.653575+0000) 2022-09-10T06:54:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:17.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:17.037734+0000) 2022-09-10T06:54:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:17 smithi110 ceph-mon[97633]: pgmap v4646: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:54:18.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:17.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:17.653814+0000) 2022-09-10T06:54:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:17 smithi104 ceph-mon[97044]: pgmap v4646: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:54:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:18.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:18.037874+0000) 2022-09-10T06:54:19.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:18.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:18.654009+0000) 2022-09-10T06:54:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:19.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:19.038002+0000) 2022-09-10T06:54:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:19 smithi110 ceph-mon[97633]: pgmap v4647: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:54:20.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:19.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:19.654217+0000) 2022-09-10T06:54:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:19 smithi104 ceph-mon[97044]: pgmap v4647: 129 pgs: 129 active+clean; 8.1 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:54:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:20.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:20.038155+0000) 2022-09-10T06:54:21.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:20.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:20.654444+0000) 2022-09-10T06:54:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:21.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:21.038291+0000) 2022-09-10T06:54:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:21 smithi110 ceph-mon[97633]: pgmap v4648: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:54:22.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:21.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:21.654673+0000) 2022-09-10T06:54:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:21 smithi104 ceph-mon[97044]: pgmap v4648: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:54:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:22.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:22.038426+0000) 2022-09-10T06:54:23.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:22.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:22.654866+0000) 2022-09-10T06:54:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:23.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:23.038553+0000) 2022-09-10T06:54:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:23 smithi110 ceph-mon[97633]: pgmap v4649: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:54:24.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:23.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:23.655141+0000) 2022-09-10T06:54:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:23 smithi104 ceph-mon[97044]: pgmap v4649: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:54:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:24.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:24.038691+0000) 2022-09-10T06:54:25.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:24.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:24.655326+0000) 2022-09-10T06:54:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:25.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:25.038841+0000) 2022-09-10T06:54:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:25 smithi104 ceph-mon[97044]: pgmap v4650: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:54:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:25 smithi110 ceph-mon[97633]: pgmap v4650: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:54:26.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:25.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:25.655523+0000) 2022-09-10T06:54:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:26.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:26.038960+0000) 2022-09-10T06:54:27.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:26.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:26.655717+0000) 2022-09-10T06:54:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:27.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:27.039086+0000) 2022-09-10T06:54:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:27 smithi110 ceph-mon[97633]: pgmap v4651: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:28.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:27.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:27.655910+0000) 2022-09-10T06:54:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:27 smithi104 ceph-mon[97044]: pgmap v4651: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:28.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:28.039277+0000) 2022-09-10T06:54:29.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:28.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:28.656081+0000) 2022-09-10T06:54:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:29.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:29.039403+0000) 2022-09-10T06:54:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:29 smithi110 ceph-mon[97633]: pgmap v4652: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:30.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:29.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:29.656324+0000) 2022-09-10T06:54:30.041 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:54:29.781+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:54:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:29 smithi104 ceph-mon[97044]: pgmap v4652: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:30.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:30.039533+0000) 2022-09-10T06:54:31.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:30.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:30.656553+0000) 2022-09-10T06:54:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:31.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:31.039656+0000) 2022-09-10T06:54:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:31 smithi110 ceph-mon[97633]: pgmap v4653: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:32.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:31.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:31.656750+0000) 2022-09-10T06:54:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:31 smithi104 ceph-mon[97044]: pgmap v4653: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:32.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:32.039882+0000) 2022-09-10T06:54:33.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:32.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:32.656938+0000) 2022-09-10T06:54:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:33.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:33.040025+0000) 2022-09-10T06:54:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:33 smithi110 ceph-mon[97633]: pgmap v4654: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:34.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:33.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:33.657173+0000) 2022-09-10T06:54:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:33 smithi104 ceph-mon[97044]: pgmap v4654: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:34.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:34.040202+0000) 2022-09-10T06:54:35.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:34.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:34.657422+0000) 2022-09-10T06:54:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:35.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:35.040427+0000) 2022-09-10T06:54:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:35 smithi104 ceph-mon[97044]: pgmap v4655: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:35 smithi110 ceph-mon[97633]: pgmap v4655: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:36.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:35.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:35.657683+0000) 2022-09-10T06:54:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:36.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:36.040559+0000) 2022-09-10T06:54:37.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:36.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:36.657834+0000) 2022-09-10T06:54:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:37.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:37.040699+0000) 2022-09-10T06:54:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:37 smithi110 ceph-mon[97633]: pgmap v4656: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:38.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:37.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:37.658037+0000) 2022-09-10T06:54:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:37 smithi104 ceph-mon[97044]: pgmap v4656: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:38.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:38.040839+0000) 2022-09-10T06:54:39.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:38.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:38.658267+0000) 2022-09-10T06:54:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:39.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:39.040995+0000) 2022-09-10T06:54:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:39 smithi110 ceph-mon[97633]: pgmap v4657: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:40.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:39.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:39.658482+0000) 2022-09-10T06:54:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:39 smithi104 ceph-mon[97044]: pgmap v4657: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:40.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:40.041150+0000) 2022-09-10T06:54:41.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:40.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:40.658726+0000) 2022-09-10T06:54:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:41.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:41.041277+0000) 2022-09-10T06:54:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:41 smithi110 ceph-mon[97633]: pgmap v4658: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:42.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:41.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:41.658965+0000) 2022-09-10T06:54:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:41 smithi104 ceph-mon[97044]: pgmap v4658: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:42.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:42.041418+0000) 2022-09-10T06:54:43.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:42.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:42.659177+0000) 2022-09-10T06:54:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:43.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:43.041552+0000) 2022-09-10T06:54:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:43 smithi110 ceph-mon[97633]: pgmap v4659: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:43.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:43.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:43.659400+0000) 2022-09-10T06:54:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:43 smithi104 ceph-mon[97044]: pgmap v4659: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:44.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:44.041694+0000) 2022-09-10T06:54:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:54:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:54:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:54:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:54:45.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:44.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:44.659565+0000) 2022-09-10T06:54:45.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:54:44.796+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:54:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:54:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:54:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:54:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:54:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:45.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:45.041880+0000) 2022-09-10T06:54:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:45 smithi110 ceph-mon[97633]: pgmap v4660: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:46.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:45.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:45.659794+0000) 2022-09-10T06:54:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:45 smithi104 ceph-mon[97044]: pgmap v4660: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:46.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:46.042004+0000) 2022-09-10T06:54:47.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:46.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:46.660037+0000) 2022-09-10T06:54:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:47.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:47.042134+0000) 2022-09-10T06:54:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:47 smithi110 ceph-mon[97633]: pgmap v4661: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:48.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:47.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:47.660294+0000) 2022-09-10T06:54:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:47 smithi104 ceph-mon[97044]: pgmap v4661: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:48.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:48.042259+0000) 2022-09-10T06:54:49.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:48.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:48.660501+0000) 2022-09-10T06:54:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:49.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:49.042386+0000) 2022-09-10T06:54:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:49 smithi110 ceph-mon[97633]: pgmap v4662: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:50.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:49.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:49.660724+0000) 2022-09-10T06:54:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:49 smithi104 ceph-mon[97044]: pgmap v4662: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:50.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:50.042516+0000) 2022-09-10T06:54:51.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:50.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:50.660961+0000) 2022-09-10T06:54:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:51.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:51.042694+0000) 2022-09-10T06:54:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:51 smithi110 ceph-mon[97633]: pgmap v4663: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:52.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:51.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:51.661181+0000) 2022-09-10T06:54:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:51 smithi104 ceph-mon[97044]: pgmap v4663: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:52.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:52.042869+0000) 2022-09-10T06:54:53.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:52.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:52.661398+0000) 2022-09-10T06:54:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:53.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:53.043043+0000) 2022-09-10T06:54:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:53 smithi110 ceph-mon[97633]: pgmap v4664: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:54.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:53.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:53.661601+0000) 2022-09-10T06:54:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:53 smithi104 ceph-mon[97044]: pgmap v4664: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:54.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:54.043156+0000) 2022-09-10T06:54:55.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:54.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:54.661792+0000) 2022-09-10T06:54:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:55.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:55.043302+0000) 2022-09-10T06:54:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:55 smithi104 ceph-mon[97044]: pgmap v4665: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:55 smithi110 ceph-mon[97633]: pgmap v4665: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:54:56.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:55.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:55.662059+0000) 2022-09-10T06:54:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:56.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:56.043424+0000) 2022-09-10T06:54:57.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:56.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:56.662263+0000) 2022-09-10T06:54:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:57.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:57.043610+0000) 2022-09-10T06:54:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:57 smithi110 ceph-mon[97633]: pgmap v4666: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:58.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:57.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:57.662486+0000) 2022-09-10T06:54:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:57 smithi104 ceph-mon[97044]: pgmap v4666: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:54:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:58.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:58.043751+0000) 2022-09-10T06:54:59.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:58.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:58.662715+0000) 2022-09-10T06:54:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:54:59.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:59.043913+0000) 2022-09-10T06:54:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:54:59 smithi110 ceph-mon[97633]: pgmap v4667: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:00.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:54:59.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:54:59.662966+0000) 2022-09-10T06:55:00.046 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:54:59.810+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:55:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:54:59 smithi104 ceph-mon[97044]: pgmap v4667: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:00.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:00.044091+0000) 2022-09-10T06:55:01.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:00.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:00.663134+0000) 2022-09-10T06:55:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:01.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:01.044278+0000) 2022-09-10T06:55:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:01 smithi110 ceph-mon[97633]: pgmap v4668: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:02.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:01.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:01.663270+0000) 2022-09-10T06:55:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:01 smithi104 ceph-mon[97044]: pgmap v4668: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:02.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:02.044416+0000) 2022-09-10T06:55:03.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:02.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:02.663500+0000) 2022-09-10T06:55:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:03.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:03.044586+0000) 2022-09-10T06:55:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:03 smithi110 ceph-mon[97633]: pgmap v4669: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:04.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:03.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:03.663728+0000) 2022-09-10T06:55:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:03 smithi104 ceph-mon[97044]: pgmap v4669: 129 pgs: 129 active+clean; 8.1 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:04.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:04.044781+0000) 2022-09-10T06:55:05.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:04.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:04.663935+0000) 2022-09-10T06:55:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:05.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:05.044970+0000) 2022-09-10T06:55:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:05 smithi104 ceph-mon[97044]: pgmap v4670: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:55:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:05 smithi110 ceph-mon[97633]: pgmap v4670: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:55:06.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:05.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:05.664192+0000) 2022-09-10T06:55:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:06.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:06.045165+0000) 2022-09-10T06:55:07.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:06.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:06.664401+0000) 2022-09-10T06:55:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:07.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:07.045378+0000) 2022-09-10T06:55:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:07 smithi110 ceph-mon[97633]: pgmap v4671: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:55:08.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:07.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:07.664672+0000) 2022-09-10T06:55:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:07 smithi104 ceph-mon[97044]: pgmap v4671: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:55:08.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:08.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:08.045516+0000) 2022-09-10T06:55:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:08 smithi110 ceph-mon[97633]: osdmap e374: 12 total, 12 up, 12 in 2022-09-10T06:55:09.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:08.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:08.664903+0000) 2022-09-10T06:55:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:08 smithi104 ceph-mon[97044]: osdmap e374: 12 total, 12 up, 12 in 2022-09-10T06:55:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:09.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:09.045640+0000) 2022-09-10T06:55:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:09 smithi110 ceph-mon[97633]: pgmap v4673: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:55:10.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:09.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:09.665182+0000) 2022-09-10T06:55:10.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:09 smithi104 ceph-mon[97044]: pgmap v4673: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:55:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:10.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:10.045772+0000) 2022-09-10T06:55:11.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:10.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:10.665427+0000) 2022-09-10T06:55:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:11.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:11.045899+0000) 2022-09-10T06:55:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:11 smithi110 ceph-mon[97633]: pgmap v4674: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:55:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:11 smithi104 ceph-mon[97044]: pgmap v4674: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:55:12.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:11.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:11.665663+0000) 2022-09-10T06:55:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:12.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:12.046063+0000) 2022-09-10T06:55:13.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:12.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:12.665896+0000) 2022-09-10T06:55:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:13.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:13.046238+0000) 2022-09-10T06:55:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:13 smithi110 ceph-mon[97633]: pgmap v4675: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:55:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:13 smithi104 ceph-mon[97044]: pgmap v4675: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:55:14.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:13.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:13.666176+0000) 2022-09-10T06:55:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:14.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:14.046362+0000) 2022-09-10T06:55:15.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:14.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:14.666379+0000) 2022-09-10T06:55:15.048 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:55:14.823+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:55:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:15.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:15.046516+0000) 2022-09-10T06:55:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:15 smithi104 ceph-mon[97044]: osdmap e375: 12 total, 12 up, 12 in 2022-09-10T06:55:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:15 smithi104 ceph-mon[97044]: pgmap v4677: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:55:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:15 smithi110 ceph-mon[97633]: osdmap e375: 12 total, 12 up, 12 in 2022-09-10T06:55:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:15 smithi110 ceph-mon[97633]: pgmap v4677: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:55:16.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:15.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:15.666600+0000) 2022-09-10T06:55:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:16.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:16.046649+0000) 2022-09-10T06:55:17.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:16.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:16.666823+0000) 2022-09-10T06:55:17.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:17.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:17.046791+0000) 2022-09-10T06:55:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:17 smithi110 ceph-mon[97633]: pgmap v4678: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:55:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:17 smithi104 ceph-mon[97044]: pgmap v4678: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:55:18.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:17.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:17.667016+0000) 2022-09-10T06:55:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:18.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:18.046934+0000) 2022-09-10T06:55:19.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:18.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:18.667256+0000) 2022-09-10T06:55:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:19.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:19.047062+0000) 2022-09-10T06:55:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:19 smithi110 ceph-mon[97633]: pgmap v4679: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:55:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:19 smithi104 ceph-mon[97044]: pgmap v4679: 129 pgs: 129 active+clean; 8.2 MiB data, 565 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:55:20.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:19.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:19.667510+0000) 2022-09-10T06:55:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:20.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:20.047203+0000) 2022-09-10T06:55:21.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:20.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:20.667792+0000) 2022-09-10T06:55:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:21.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:21.047379+0000) 2022-09-10T06:55:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:21 smithi110 ceph-mon[97633]: pgmap v4680: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:55:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:21 smithi104 ceph-mon[97044]: pgmap v4680: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:55:22.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:21.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:21.668054+0000) 2022-09-10T06:55:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:22.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:22.047506+0000) 2022-09-10T06:55:23.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:22.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:22.668295+0000) 2022-09-10T06:55:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:23.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:23.047655+0000) 2022-09-10T06:55:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:23 smithi110 ceph-mon[97633]: pgmap v4681: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:55:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:23 smithi104 ceph-mon[97044]: pgmap v4681: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:55:24.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:23.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:23.668541+0000) 2022-09-10T06:55:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:24.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:24.047848+0000) 2022-09-10T06:55:25.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:24.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:24.668800+0000) 2022-09-10T06:55:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:25.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:25.048037+0000) 2022-09-10T06:55:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:25 smithi104 ceph-mon[97044]: pgmap v4682: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:55:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:25 smithi110 ceph-mon[97633]: pgmap v4682: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:55:26.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:25.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:25.669080+0000) 2022-09-10T06:55:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:26.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:26.048151+0000) 2022-09-10T06:55:27.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:26.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:26.669323+0000) 2022-09-10T06:55:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:27.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:27.048279+0000) 2022-09-10T06:55:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:27 smithi110 ceph-mon[97633]: pgmap v4683: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:27 smithi104 ceph-mon[97044]: pgmap v4683: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:28.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:27.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:27.669590+0000) 2022-09-10T06:55:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:28.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:28.048406+0000) 2022-09-10T06:55:29.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:28.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:28.669838+0000) 2022-09-10T06:55:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:29.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:29.048527+0000) 2022-09-10T06:55:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:29 smithi110 ceph-mon[97633]: pgmap v4684: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:29 smithi104 ceph-mon[97044]: pgmap v4684: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:30.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:29.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:29.670073+0000) 2022-09-10T06:55:30.049 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:55:29.835+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:55:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:30.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:30.048654+0000) 2022-09-10T06:55:31.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:30.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:30.670348+0000) 2022-09-10T06:55:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:31.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:31.048770+0000) 2022-09-10T06:55:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:31 smithi110 ceph-mon[97633]: pgmap v4685: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:31 smithi104 ceph-mon[97044]: pgmap v4685: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:32.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:31.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:31.670587+0000) 2022-09-10T06:55:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:32.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:32.048900+0000) 2022-09-10T06:55:33.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:32.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:32.670791+0000) 2022-09-10T06:55:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:33.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:33.049032+0000) 2022-09-10T06:55:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:33 smithi110 ceph-mon[97633]: pgmap v4686: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:33 smithi104 ceph-mon[97044]: pgmap v4686: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:34.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:33.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:33.671036+0000) 2022-09-10T06:55:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:34.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:34.049154+0000) 2022-09-10T06:55:35.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:34.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:34.671286+0000) 2022-09-10T06:55:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:35.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:35.049288+0000) 2022-09-10T06:55:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:35 smithi104 ceph-mon[97044]: pgmap v4687: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:35 smithi110 ceph-mon[97633]: pgmap v4687: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:36.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:35.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:35.671497+0000) 2022-09-10T06:55:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:36.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:36.049420+0000) 2022-09-10T06:55:37.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:36.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:36.671733+0000) 2022-09-10T06:55:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:37.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:37.049553+0000) 2022-09-10T06:55:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:37 smithi110 ceph-mon[97633]: pgmap v4688: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:37 smithi104 ceph-mon[97044]: pgmap v4688: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:38.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:37.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:37.672003+0000) 2022-09-10T06:55:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:38.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:38.049684+0000) 2022-09-10T06:55:39.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:38.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:38.672250+0000) 2022-09-10T06:55:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:39.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:39.049850+0000) 2022-09-10T06:55:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:39 smithi110 ceph-mon[97633]: pgmap v4689: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:39 smithi104 ceph-mon[97044]: pgmap v4689: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:40.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:39.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:39.672414+0000) 2022-09-10T06:55:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:40.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:40.049991+0000) 2022-09-10T06:55:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:40.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:40.672657+0000) 2022-09-10T06:55:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:41.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:41.050122+0000) 2022-09-10T06:55:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:41 smithi110 ceph-mon[97633]: pgmap v4690: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:41 smithi104 ceph-mon[97044]: pgmap v4690: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:42.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:41.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:41.672847+0000) 2022-09-10T06:55:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:42.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:42.050254+0000) 2022-09-10T06:55:43.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:42.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:42.673149+0000) 2022-09-10T06:55:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:43.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:43.050393+0000) 2022-09-10T06:55:43.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:43 smithi110 ceph-mon[97633]: pgmap v4691: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:44.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:43 smithi104 ceph-mon[97044]: pgmap v4691: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:44.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:43.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:43.673414+0000) 2022-09-10T06:55:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:44.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:44.050571+0000) 2022-09-10T06:55:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:55:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:55:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:55:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:55:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:55:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:55:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:55:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:55:45.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:44.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:44.673666+0000) 2022-09-10T06:55:45.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:55:44.844+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:55:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:45.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:45.050728+0000) 2022-09-10T06:55:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:45 smithi110 ceph-mon[97633]: pgmap v4692: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:45 smithi104 ceph-mon[97044]: pgmap v4692: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:46.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:45.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:45.673862+0000) 2022-09-10T06:55:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:46.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:46.050864+0000) 2022-09-10T06:55:47.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:46.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:46.674035+0000) 2022-09-10T06:55:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:47.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:47.050984+0000) 2022-09-10T06:55:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:47 smithi110 ceph-mon[97633]: pgmap v4693: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:47 smithi104 ceph-mon[97044]: pgmap v4693: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:48.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:47.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:47.674278+0000) 2022-09-10T06:55:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:48.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:48.051137+0000) 2022-09-10T06:55:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:48.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:48.674463+0000) 2022-09-10T06:55:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:49.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:49.051273+0000) 2022-09-10T06:55:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:49 smithi110 ceph-mon[97633]: pgmap v4694: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:49 smithi104 ceph-mon[97044]: pgmap v4694: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:50.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:49.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:49.674729+0000) 2022-09-10T06:55:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:50.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:50.051425+0000) 2022-09-10T06:55:51.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:50.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:50.675040+0000) 2022-09-10T06:55:51.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:51.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:51.051554+0000) 2022-09-10T06:55:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:51 smithi110 ceph-mon[97633]: pgmap v4695: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:51 smithi104 ceph-mon[97044]: pgmap v4695: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:52.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:51.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:51.675276+0000) 2022-09-10T06:55:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:52.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:52.051741+0000) 2022-09-10T06:55:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:52.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:52.675468+0000) 2022-09-10T06:55:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:53.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:53.051900+0000) 2022-09-10T06:55:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:53 smithi110 ceph-mon[97633]: pgmap v4696: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:53 smithi104 ceph-mon[97044]: pgmap v4696: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:54.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:53.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:53.675708+0000) 2022-09-10T06:55:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:54.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:54.052087+0000) 2022-09-10T06:55:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:54.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:54.675931+0000) 2022-09-10T06:55:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:55.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:55.052237+0000) 2022-09-10T06:55:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:55 smithi104 ceph-mon[97044]: pgmap v4697: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:55 smithi110 ceph-mon[97633]: pgmap v4697: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:55:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:55.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:55.676218+0000) 2022-09-10T06:55:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:56.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:56.052357+0000) 2022-09-10T06:55:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:56.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:56.676367+0000) 2022-09-10T06:55:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:57.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:57.052507+0000) 2022-09-10T06:55:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:57 smithi110 ceph-mon[97633]: pgmap v4698: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:57 smithi104 ceph-mon[97044]: pgmap v4698: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:55:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:57.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:57.676543+0000) 2022-09-10T06:55:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:58.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:58.052706+0000) 2022-09-10T06:55:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:58.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:58.676732+0000) 2022-09-10T06:55:59.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:55:59.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:59.052901+0000) 2022-09-10T06:55:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:55:59 smithi110 ceph-mon[97633]: pgmap v4699: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:55:59 smithi104 ceph-mon[97044]: pgmap v4699: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:55:59.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:55:59.676905+0000) 2022-09-10T06:56:00.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:55:59.851+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:56:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:00.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:00.053050+0000) 2022-09-10T06:56:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:00.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:00.677184+0000) 2022-09-10T06:56:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:01.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:01.053184+0000) 2022-09-10T06:56:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:01 smithi110 ceph-mon[97633]: pgmap v4700: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:01 smithi104 ceph-mon[97044]: pgmap v4700: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:01.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:01.677479+0000) 2022-09-10T06:56:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:02.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:02.053358+0000) 2022-09-10T06:56:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:02.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:02.677769+0000) 2022-09-10T06:56:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:03.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:03.053491+0000) 2022-09-10T06:56:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:03 smithi110 ceph-mon[97633]: pgmap v4701: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:03 smithi104 ceph-mon[97044]: pgmap v4701: 129 pgs: 129 active+clean; 8.2 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:03.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:03.678025+0000) 2022-09-10T06:56:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:04.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:04.053617+0000) 2022-09-10T06:56:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:04.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:04.678294+0000) 2022-09-10T06:56:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:05 smithi104 ceph-mon[97044]: pgmap v4702: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:56:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:05.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:05.053848+0000) 2022-09-10T06:56:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:05 smithi110 ceph-mon[97633]: pgmap v4702: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:56:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:05.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:05.678510+0000) 2022-09-10T06:56:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:06.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:06.053981+0000) 2022-09-10T06:56:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:06.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:06.678696+0000) 2022-09-10T06:56:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:07.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:07.054167+0000) 2022-09-10T06:56:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:07 smithi110 ceph-mon[97633]: pgmap v4703: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:56:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:07 smithi104 ceph-mon[97044]: pgmap v4703: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:56:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:07.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:07.678851+0000) 2022-09-10T06:56:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:08.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:08.054294+0000) 2022-09-10T06:56:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:08 smithi110 ceph-mon[97633]: osdmap e376: 12 total, 12 up, 12 in 2022-09-10T06:56:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:08 smithi104 ceph-mon[97044]: osdmap e376: 12 total, 12 up, 12 in 2022-09-10T06:56:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:08.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:08.679084+0000) 2022-09-10T06:56:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:09.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:09.054421+0000) 2022-09-10T06:56:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:09 smithi110 ceph-mon[97633]: pgmap v4705: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:56:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:09 smithi104 ceph-mon[97044]: pgmap v4705: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:56:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:09.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:09.679339+0000) 2022-09-10T06:56:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:10.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:10.054549+0000) 2022-09-10T06:56:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:10.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:10.679613+0000) 2022-09-10T06:56:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:11.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:11.054658+0000) 2022-09-10T06:56:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:11 smithi110 ceph-mon[97633]: pgmap v4706: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:56:12.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:11 smithi104 ceph-mon[97044]: pgmap v4706: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:56:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:11.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:11.679875+0000) 2022-09-10T06:56:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:12.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:12.054766+0000) 2022-09-10T06:56:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:12.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:12.680079+0000) 2022-09-10T06:56:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:13.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:13.054895+0000) 2022-09-10T06:56:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:13 smithi110 ceph-mon[97633]: pgmap v4707: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:56:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:13 smithi104 ceph-mon[97044]: pgmap v4707: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:56:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:13.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:13.680244+0000) 2022-09-10T06:56:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:14.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:14.055023+0000) 2022-09-10T06:56:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:14.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:14.680469+0000) 2022-09-10T06:56:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:56:14.858+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:56:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:15 smithi104 ceph-mon[97044]: osdmap e377: 12 total, 12 up, 12 in 2022-09-10T06:56:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:15 smithi104 ceph-mon[97044]: pgmap v4709: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:56:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:15.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:15.055162+0000) 2022-09-10T06:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:15 smithi110 ceph-mon[97633]: osdmap e377: 12 total, 12 up, 12 in 2022-09-10T06:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:15 smithi110 ceph-mon[97633]: pgmap v4709: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:56:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:15.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:15.680696+0000) 2022-09-10T06:56:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:16.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:16.055291+0000) 2022-09-10T06:56:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:16.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:16.680893+0000) 2022-09-10T06:56:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:17.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:17.055428+0000) 2022-09-10T06:56:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:17 smithi110 ceph-mon[97633]: pgmap v4710: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:56:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:17 smithi104 ceph-mon[97044]: pgmap v4710: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:56:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:17.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:17.681166+0000) 2022-09-10T06:56:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:18.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:18.055568+0000) 2022-09-10T06:56:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:18.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:18.681342+0000) 2022-09-10T06:56:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:19.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:19.055684+0000) 2022-09-10T06:56:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:19 smithi110 ceph-mon[97633]: pgmap v4711: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:56:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:19 smithi104 ceph-mon[97044]: pgmap v4711: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:56:20.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:19.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:19.681584+0000) 2022-09-10T06:56:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:20.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:20.055811+0000) 2022-09-10T06:56:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:20.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:20.681814+0000) 2022-09-10T06:56:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:21.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:21.055937+0000) 2022-09-10T06:56:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:21 smithi110 ceph-mon[97633]: pgmap v4712: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:56:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:21 smithi104 ceph-mon[97044]: pgmap v4712: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:56:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:21.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:21.682040+0000) 2022-09-10T06:56:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:22.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:22.056053+0000) 2022-09-10T06:56:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:22.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:22.682302+0000) 2022-09-10T06:56:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:23.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:23.056158+0000) 2022-09-10T06:56:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:23 smithi110 ceph-mon[97633]: pgmap v4713: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:56:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:23 smithi104 ceph-mon[97044]: pgmap v4713: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:56:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:23.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:23.682483+0000) 2022-09-10T06:56:24.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:24.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:24.056277+0000) 2022-09-10T06:56:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:24.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:24.682721+0000) 2022-09-10T06:56:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:25 smithi104 ceph-mon[97044]: pgmap v4714: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:56:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:25.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:25.056412+0000) 2022-09-10T06:56:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:25 smithi110 ceph-mon[97633]: pgmap v4714: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:56:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:25.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:25.682928+0000) 2022-09-10T06:56:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:26.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:26.056530+0000) 2022-09-10T06:56:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:26.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:26.683177+0000) 2022-09-10T06:56:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:27.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:27.056673+0000) 2022-09-10T06:56:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:27 smithi110 ceph-mon[97633]: pgmap v4715: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:27 smithi104 ceph-mon[97044]: pgmap v4715: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:27.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:27.683443+0000) 2022-09-10T06:56:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:28.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:28.056810+0000) 2022-09-10T06:56:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:28.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:28.683672+0000) 2022-09-10T06:56:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:29.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:29.056947+0000) 2022-09-10T06:56:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:29 smithi110 ceph-mon[97633]: pgmap v4716: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:29 smithi104 ceph-mon[97044]: pgmap v4716: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:30.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:29.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:29.683919+0000) 2022-09-10T06:56:30.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:56:29.874+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:56:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:30.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:30.057114+0000) 2022-09-10T06:56:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:30.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:30.684214+0000) 2022-09-10T06:56:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:31.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:31.057246+0000) 2022-09-10T06:56:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:31 smithi110 ceph-mon[97633]: pgmap v4717: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:31 smithi104 ceph-mon[97044]: pgmap v4717: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:31.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:31.684384+0000) 2022-09-10T06:56:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:32.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:32.057394+0000) 2022-09-10T06:56:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:32.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:32.684556+0000) 2022-09-10T06:56:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:33.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:33.057566+0000) 2022-09-10T06:56:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:33 smithi110 ceph-mon[97633]: pgmap v4718: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:33 smithi104 ceph-mon[97044]: pgmap v4718: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:33.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:33.684769+0000) 2022-09-10T06:56:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:34.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:34.057730+0000) 2022-09-10T06:56:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:34.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:34.685011+0000) 2022-09-10T06:56:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:35 smithi104 ceph-mon[97044]: pgmap v4719: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:35.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:35.057880+0000) 2022-09-10T06:56:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:35 smithi110 ceph-mon[97633]: pgmap v4719: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:35.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:35.685264+0000) 2022-09-10T06:56:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:36.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:36.058017+0000) 2022-09-10T06:56:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:36.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:36.685529+0000) 2022-09-10T06:56:37.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:37.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:37.058163+0000) 2022-09-10T06:56:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:37 smithi110 ceph-mon[97633]: pgmap v4720: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:37 smithi104 ceph-mon[97044]: pgmap v4720: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:37.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:37.685781+0000) 2022-09-10T06:56:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:38.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:38.058297+0000) 2022-09-10T06:56:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:38.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:38.686007+0000) 2022-09-10T06:56:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:39.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:39.058385+0000) 2022-09-10T06:56:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:39 smithi110 ceph-mon[97633]: pgmap v4721: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:39 smithi104 ceph-mon[97044]: pgmap v4721: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:39.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:39.686229+0000) 2022-09-10T06:56:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:40.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:40.058529+0000) 2022-09-10T06:56:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:40.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:40.686438+0000) 2022-09-10T06:56:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:41.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:41.058642+0000) 2022-09-10T06:56:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:41 smithi110 ceph-mon[97633]: pgmap v4722: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:41 smithi104 ceph-mon[97044]: pgmap v4722: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:42.073 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:41.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:41.686652+0000) 2022-09-10T06:56:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:42.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:42.058782+0000) 2022-09-10T06:56:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:42.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:42.686838+0000) 2022-09-10T06:56:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:43.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:43.058897+0000) 2022-09-10T06:56:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:43 smithi110 ceph-mon[97633]: pgmap v4723: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:43 smithi104 ceph-mon[97044]: pgmap v4723: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:43.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:43.687068+0000) 2022-09-10T06:56:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:44.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:44.059123+0000) 2022-09-10T06:56:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:56:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:56:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:56:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:56:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:56:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:56:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:56:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:56:45.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:56:44.880+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:56:45.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:44.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:44.687358+0000) 2022-09-10T06:56:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:45.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:45.059326+0000) 2022-09-10T06:56:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:45 smithi110 ceph-mon[97633]: pgmap v4724: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:45 smithi104 ceph-mon[97044]: pgmap v4724: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:45.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:45.687647+0000) 2022-09-10T06:56:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:46.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:46.059458+0000) 2022-09-10T06:56:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:46.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:46.687927+0000) 2022-09-10T06:56:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:47.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:47.059628+0000) 2022-09-10T06:56:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:47 smithi110 ceph-mon[97633]: pgmap v4725: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:47 smithi104 ceph-mon[97044]: pgmap v4725: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:47.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:47.688171+0000) 2022-09-10T06:56:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:48.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:48.059808+0000) 2022-09-10T06:56:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:48.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:48.688415+0000) 2022-09-10T06:56:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:49.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:49.059948+0000) 2022-09-10T06:56:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:49 smithi110 ceph-mon[97633]: pgmap v4726: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:49 smithi104 ceph-mon[97044]: pgmap v4726: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:49.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:49.688575+0000) 2022-09-10T06:56:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:50.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:50.060082+0000) 2022-09-10T06:56:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:50.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:50.688804+0000) 2022-09-10T06:56:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:51.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:51.060251+0000) 2022-09-10T06:56:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:51 smithi110 ceph-mon[97633]: pgmap v4727: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:51 smithi104 ceph-mon[97044]: pgmap v4727: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:51.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:51.688987+0000) 2022-09-10T06:56:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:52.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:52.060384+0000) 2022-09-10T06:56:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:52.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:52.689147+0000) 2022-09-10T06:56:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:53.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:53.060535+0000) 2022-09-10T06:56:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:53 smithi110 ceph-mon[97633]: pgmap v4728: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:53 smithi104 ceph-mon[97044]: pgmap v4728: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:54.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:53.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:53.689374+0000) 2022-09-10T06:56:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:54.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:54.060722+0000) 2022-09-10T06:56:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:54.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:54.689620+0000) 2022-09-10T06:56:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:55.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:55.060923+0000) 2022-09-10T06:56:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:55 smithi110 ceph-mon[97633]: pgmap v4729: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:55 smithi104 ceph-mon[97044]: pgmap v4729: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:56:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:55.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:55.689891+0000) 2022-09-10T06:56:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:56.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:56.061200+0000) 2022-09-10T06:56:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:56.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:56.690108+0000) 2022-09-10T06:56:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:57.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:57.061360+0000) 2022-09-10T06:56:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:57 smithi110 ceph-mon[97633]: pgmap v4730: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:57 smithi104 ceph-mon[97044]: pgmap v4730: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:56:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:57.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:57.690318+0000) 2022-09-10T06:56:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:58.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:58.061556+0000) 2022-09-10T06:56:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:58.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:58.690559+0000) 2022-09-10T06:56:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:56:59.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:59.061706+0000) 2022-09-10T06:56:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:56:59 smithi110 ceph-mon[97633]: pgmap v4731: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:56:59 smithi104 ceph-mon[97044]: pgmap v4731: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:56:59.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:56:59.690857+0000) 2022-09-10T06:57:00.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:56:59.895+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:57:00.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:00.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:00.061918+0000) 2022-09-10T06:57:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:00.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:00.691158+0000) 2022-09-10T06:57:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:01.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:01.062205+0000) 2022-09-10T06:57:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:01 smithi110 ceph-mon[97633]: pgmap v4732: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:01 smithi104 ceph-mon[97044]: pgmap v4732: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:02.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:01.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:01.691377+0000) 2022-09-10T06:57:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:02.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:02.062417+0000) 2022-09-10T06:57:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:02.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:02.691597+0000) 2022-09-10T06:57:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:03.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:03.062569+0000) 2022-09-10T06:57:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:03 smithi110 ceph-mon[97633]: pgmap v4733: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:03 smithi104 ceph-mon[97044]: pgmap v4733: 129 pgs: 129 active+clean; 8.2 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:03.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:03.691863+0000) 2022-09-10T06:57:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:04.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:04.062704+0000) 2022-09-10T06:57:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:04.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:04.692166+0000) 2022-09-10T06:57:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:05.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:05.062972+0000) 2022-09-10T06:57:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:05 smithi110 ceph-mon[97633]: pgmap v4734: 129 pgs: 129 active+clean; 8.2 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:57:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:05 smithi104 ceph-mon[97044]: pgmap v4734: 129 pgs: 129 active+clean; 8.2 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:57:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:05.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:05.692379+0000) 2022-09-10T06:57:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:06.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:06.063203+0000) 2022-09-10T06:57:07.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:06.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:06.692615+0000) 2022-09-10T06:57:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:07.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:07.063422+0000) 2022-09-10T06:57:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:07 smithi110 ceph-mon[97633]: pgmap v4735: 129 pgs: 129 active+clean; 8.2 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:57:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:07 smithi104 ceph-mon[97044]: pgmap v4735: 129 pgs: 129 active+clean; 8.2 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:57:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:07.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:07.692845+0000) 2022-09-10T06:57:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:08.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:08.063687+0000) 2022-09-10T06:57:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:08 smithi110 ceph-mon[97633]: osdmap e378: 12 total, 12 up, 12 in 2022-09-10T06:57:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:08 smithi104 ceph-mon[97044]: osdmap e378: 12 total, 12 up, 12 in 2022-09-10T06:57:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:08.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:08.693129+0000) 2022-09-10T06:57:09.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:09.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:09.063964+0000) 2022-09-10T06:57:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:09 smithi110 ceph-mon[97633]: pgmap v4737: 129 pgs: 129 active+clean; 8.2 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:57:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:09 smithi104 ceph-mon[97044]: pgmap v4737: 129 pgs: 129 active+clean; 8.2 MiB data, 569 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:57:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:09.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:09.693450+0000) 2022-09-10T06:57:10.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:10.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:10.064202+0000) 2022-09-10T06:57:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:10.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:10.693705+0000) 2022-09-10T06:57:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:11.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:11.064412+0000) 2022-09-10T06:57:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:11 smithi110 ceph-mon[97633]: pgmap v4738: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:57:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:11 smithi104 ceph-mon[97044]: pgmap v4738: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:57:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:11.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:11.693928+0000) 2022-09-10T06:57:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:12.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:12.064594+0000) 2022-09-10T06:57:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:12.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:12.694199+0000) 2022-09-10T06:57:13.573 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:13.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:13.064767+0000) 2022-09-10T06:57:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:13 smithi110 ceph-mon[97633]: pgmap v4739: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:57:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:13 smithi104 ceph-mon[97044]: pgmap v4739: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:57:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:13.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:13.694452+0000) 2022-09-10T06:57:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:14.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:14.064942+0000) 2022-09-10T06:57:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:14.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:14.694639+0000) 2022-09-10T06:57:15.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:57:14.908+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:57:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:15.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:15.065193+0000) 2022-09-10T06:57:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:15 smithi110 ceph-mon[97633]: osdmap e379: 12 total, 12 up, 12 in 2022-09-10T06:57:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:15 smithi110 ceph-mon[97633]: pgmap v4741: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:57:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:15 smithi104 ceph-mon[97044]: osdmap e379: 12 total, 12 up, 12 in 2022-09-10T06:57:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:15 smithi104 ceph-mon[97044]: pgmap v4741: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:57:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:15.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:15.694807+0000) 2022-09-10T06:57:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:16.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:16.065409+0000) 2022-09-10T06:57:17.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:16.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:16.695005+0000) 2022-09-10T06:57:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:17.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:17.065589+0000) 2022-09-10T06:57:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:17 smithi110 ceph-mon[97633]: pgmap v4742: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:57:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:17 smithi104 ceph-mon[97044]: pgmap v4742: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:57:18.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:17.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:17.695183+0000) 2022-09-10T06:57:18.563 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:18.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:18.065821+0000) 2022-09-10T06:57:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:18.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:18.695376+0000) 2022-09-10T06:57:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:19.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:19.066058+0000) 2022-09-10T06:57:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:19 smithi110 ceph-mon[97633]: pgmap v4743: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:57:20.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:19 smithi104 ceph-mon[97044]: pgmap v4743: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:57:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:19.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:19.695645+0000) 2022-09-10T06:57:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:20.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:20.066285+0000) 2022-09-10T06:57:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:20.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:20.695924+0000) 2022-09-10T06:57:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:21.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:21.066506+0000) 2022-09-10T06:57:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:21 smithi110 ceph-mon[97633]: pgmap v4744: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:57:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:21 smithi104 ceph-mon[97044]: pgmap v4744: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:57:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:21.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:21.696154+0000) 2022-09-10T06:57:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:22.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:22.066676+0000) 2022-09-10T06:57:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:22.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:22.696327+0000) 2022-09-10T06:57:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:23.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:23.066846+0000) 2022-09-10T06:57:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:23 smithi110 ceph-mon[97633]: pgmap v4745: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:57:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:23 smithi104 ceph-mon[97044]: pgmap v4745: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:57:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:23.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:23.696527+0000) 2022-09-10T06:57:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:24.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:24.067052+0000) 2022-09-10T06:57:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:24.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:24.696793+0000) 2022-09-10T06:57:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:25.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:25.067302+0000) 2022-09-10T06:57:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:25 smithi110 ceph-mon[97633]: pgmap v4746: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:57:25.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:25 smithi104 ceph-mon[97044]: pgmap v4746: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:57:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:25.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:25.697015+0000) 2022-09-10T06:57:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:26.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:26.067489+0000) 2022-09-10T06:57:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:26.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:26.697269+0000) 2022-09-10T06:57:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:27.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:27.067691+0000) 2022-09-10T06:57:27.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:27 smithi110 ceph-mon[97633]: pgmap v4747: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:27 smithi104 ceph-mon[97044]: pgmap v4747: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:27.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:27.697554+0000) 2022-09-10T06:57:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:28.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:28.067898+0000) 2022-09-10T06:57:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:28.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:28.697775+0000) 2022-09-10T06:57:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:29.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:29.068139+0000) 2022-09-10T06:57:29.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:29 smithi110 ceph-mon[97633]: pgmap v4748: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:29 smithi104 ceph-mon[97044]: pgmap v4748: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:29.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:29.697981+0000) 2022-09-10T06:57:30.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:57:29.924+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:57:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:30.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:30.068337+0000) 2022-09-10T06:57:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:30.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:30.698180+0000) 2022-09-10T06:57:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:31.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:31.068514+0000) 2022-09-10T06:57:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:31 smithi110 ceph-mon[97633]: pgmap v4749: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:31 smithi104 ceph-mon[97044]: pgmap v4749: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:31.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:31.698414+0000) 2022-09-10T06:57:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:32.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:32.068692+0000) 2022-09-10T06:57:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:32.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:32.698694+0000) 2022-09-10T06:57:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:33.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:33.068907+0000) 2022-09-10T06:57:33.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:33 smithi110 ceph-mon[97633]: pgmap v4750: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:33 smithi104 ceph-mon[97044]: pgmap v4750: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:33.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:33.698921+0000) 2022-09-10T06:57:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:34.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:34.069186+0000) 2022-09-10T06:57:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:34.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:34.699215+0000) 2022-09-10T06:57:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:35.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:35.069369+0000) 2022-09-10T06:57:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:35 smithi110 ceph-mon[97633]: pgmap v4751: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:35 smithi104 ceph-mon[97044]: pgmap v4751: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:35.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:35.699478+0000) 2022-09-10T06:57:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:36.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:36.069617+0000) 2022-09-10T06:57:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:36.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:36.699706+0000) 2022-09-10T06:57:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:37.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:37.069857+0000) 2022-09-10T06:57:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:37 smithi110 ceph-mon[97633]: pgmap v4752: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:37 smithi104 ceph-mon[97044]: pgmap v4752: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:37.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:37.699968+0000) 2022-09-10T06:57:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:38.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:38.070142+0000) 2022-09-10T06:57:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:38.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:38.700247+0000) 2022-09-10T06:57:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:39.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:39.070334+0000) 2022-09-10T06:57:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:39 smithi110 ceph-mon[97633]: pgmap v4753: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:39 smithi104 ceph-mon[97044]: pgmap v4753: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:39.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:39.700450+0000) 2022-09-10T06:57:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:40.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:40.070538+0000) 2022-09-10T06:57:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:40.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:40.700721+0000) 2022-09-10T06:57:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:41.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:41.070732+0000) 2022-09-10T06:57:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:41 smithi110 ceph-mon[97633]: pgmap v4754: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:41 smithi104 ceph-mon[97044]: pgmap v4754: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:41.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:41.700937+0000) 2022-09-10T06:57:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:42.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:42.070990+0000) 2022-09-10T06:57:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:42.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:42.701223+0000) 2022-09-10T06:57:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:43.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:43.071219+0000) 2022-09-10T06:57:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:43 smithi110 ceph-mon[97633]: pgmap v4755: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:57:44.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:43 smithi104 ceph-mon[97044]: pgmap v4755: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:57:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:43.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:43.701383+0000) 2022-09-10T06:57:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:44.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:44.071401+0000) 2022-09-10T06:57:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:57:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:57:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:57:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:57:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:57:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:57:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:57:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:57:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:44.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:44.701555+0000) 2022-09-10T06:57:45.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:57:44.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:57:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:45.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:45.071639+0000) 2022-09-10T06:57:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:45 smithi110 ceph-mon[97633]: pgmap v4756: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:45 smithi104 ceph-mon[97044]: pgmap v4756: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:45.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:45.701798+0000) 2022-09-10T06:57:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:46.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:46.071852+0000) 2022-09-10T06:57:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:46.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:46.701994+0000) 2022-09-10T06:57:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:47.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:47.072067+0000) 2022-09-10T06:57:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:47 smithi104 ceph-mon[97044]: pgmap v4757: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:57:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:47.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:47.702250+0000) 2022-09-10T06:57:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:47 smithi110 ceph-mon[97633]: pgmap v4757: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:57:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:48.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:48.072329+0000) 2022-09-10T06:57:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:48.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:48.702459+0000) 2022-09-10T06:57:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:49.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:49.072530+0000) 2022-09-10T06:57:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:49 smithi104 ceph-mon[97044]: pgmap v4758: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:57:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:49.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:49.702736+0000) 2022-09-10T06:57:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:49 smithi110 ceph-mon[97633]: pgmap v4758: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T06:57:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:50.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:50.072757+0000) 2022-09-10T06:57:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:50.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:50.702937+0000) 2022-09-10T06:57:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:51.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:51.073009+0000) 2022-09-10T06:57:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:51 smithi104 ceph-mon[97044]: pgmap v4759: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:51.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:51.703178+0000) 2022-09-10T06:57:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:51 smithi110 ceph-mon[97633]: pgmap v4759: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:52.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:52.073261+0000) 2022-09-10T06:57:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:52.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:52.703418+0000) 2022-09-10T06:57:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:53.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:53.073437+0000) 2022-09-10T06:57:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:53 smithi104 ceph-mon[97044]: pgmap v4760: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:53.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:53.703673+0000) 2022-09-10T06:57:54.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:53 smithi110 ceph-mon[97633]: pgmap v4760: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:54.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:54.073645+0000) 2022-09-10T06:57:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:54.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:54.703928+0000) 2022-09-10T06:57:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:55.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:55.073788+0000) 2022-09-10T06:57:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:55 smithi110 ceph-mon[97633]: pgmap v4761: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:55 smithi104 ceph-mon[97044]: pgmap v4761: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:57:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:55.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:55.704202+0000) 2022-09-10T06:57:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:56.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:56.073997+0000) 2022-09-10T06:57:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:56.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:56.704344+0000) 2022-09-10T06:57:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:57.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:57.074209+0000) 2022-09-10T06:57:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:57 smithi110 ceph-mon[97633]: pgmap v4762: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:57 smithi104 ceph-mon[97044]: pgmap v4762: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:57.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:57.704586+0000) 2022-09-10T06:57:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:58.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:58.074376+0000) 2022-09-10T06:57:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:58.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:58.704851+0000) 2022-09-10T06:57:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:57:59.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:59.074523+0000) 2022-09-10T06:57:59.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:57:59 smithi110 ceph-mon[97633]: pgmap v4763: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:57:59.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:57:59.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:57:59.705075+0000) 2022-09-10T06:58:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:57:59 smithi104 ceph-mon[97044]: pgmap v4763: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:57:59.955+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:58:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:00.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:00.074753+0000) 2022-09-10T06:58:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:00.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:00.705401+0000) 2022-09-10T06:58:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:01.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:01.075053+0000) 2022-09-10T06:58:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:01 smithi104 ceph-mon[97044]: pgmap v4764: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:01.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:01.705683+0000) 2022-09-10T06:58:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:01 smithi110 ceph-mon[97633]: pgmap v4764: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:02.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:02.075309+0000) 2022-09-10T06:58:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:02.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:02.705964+0000) 2022-09-10T06:58:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:03.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:03.075520+0000) 2022-09-10T06:58:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:03 smithi104 ceph-mon[97044]: pgmap v4765: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:03.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:03.706247+0000) 2022-09-10T06:58:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:03 smithi110 ceph-mon[97633]: pgmap v4765: 129 pgs: 129 active+clean; 8.2 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:04.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:04.075704+0000) 2022-09-10T06:58:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:04.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:04.706518+0000) 2022-09-10T06:58:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:05.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:05.075867+0000) 2022-09-10T06:58:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:05 smithi110 ceph-mon[97633]: pgmap v4766: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:58:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:05 smithi104 ceph-mon[97044]: pgmap v4766: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:58:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:05.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:05.706728+0000) 2022-09-10T06:58:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:06.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:06.076037+0000) 2022-09-10T06:58:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:06.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:06.706945+0000) 2022-09-10T06:58:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:07.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:07.076265+0000) 2022-09-10T06:58:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:07 smithi110 ceph-mon[97633]: pgmap v4767: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:58:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:07 smithi104 ceph-mon[97044]: pgmap v4767: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T06:58:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:07.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:07.707190+0000) 2022-09-10T06:58:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:08.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:08.076478+0000) 2022-09-10T06:58:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:08 smithi104 ceph-mon[97044]: osdmap e380: 12 total, 12 up, 12 in 2022-09-10T06:58:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:08.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:08.707441+0000) 2022-09-10T06:58:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:08 smithi110 ceph-mon[97633]: osdmap e380: 12 total, 12 up, 12 in 2022-09-10T06:58:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:09.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:09.076684+0000) 2022-09-10T06:58:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:09 smithi104 ceph-mon[97044]: pgmap v4769: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:58:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:09.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:09.707714+0000) 2022-09-10T06:58:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:09 smithi110 ceph-mon[97633]: pgmap v4769: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:58:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:10.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:10.076895+0000) 2022-09-10T06:58:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:10.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:10.707990+0000) 2022-09-10T06:58:11.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:11.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:11.077151+0000) 2022-09-10T06:58:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:11 smithi104 ceph-mon[97044]: pgmap v4770: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:58:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:11.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:11.708198+0000) 2022-09-10T06:58:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:11 smithi110 ceph-mon[97633]: pgmap v4770: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:58:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:12.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:12.077316+0000) 2022-09-10T06:58:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:12.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:12.708369+0000) 2022-09-10T06:58:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:13.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:13.077517+0000) 2022-09-10T06:58:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:13 smithi104 ceph-mon[97044]: pgmap v4771: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:58:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:13.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:13.708533+0000) 2022-09-10T06:58:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:13 smithi110 ceph-mon[97633]: pgmap v4771: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:58:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:14.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:14.077693+0000) 2022-09-10T06:58:14.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:14.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:14.708780+0000) 2022-09-10T06:58:15.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:58:14.971+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:58:15.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:15.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:15.077856+0000) 2022-09-10T06:58:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:15 smithi110 ceph-mon[97633]: osdmap e381: 12 total, 12 up, 12 in 2022-09-10T06:58:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:15 smithi110 ceph-mon[97633]: pgmap v4773: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:58:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:15 smithi104 ceph-mon[97044]: osdmap e381: 12 total, 12 up, 12 in 2022-09-10T06:58:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:15 smithi104 ceph-mon[97044]: pgmap v4773: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:58:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:15.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:15.709041+0000) 2022-09-10T06:58:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:16.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:16.078087+0000) 2022-09-10T06:58:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:16.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:16.709260+0000) 2022-09-10T06:58:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:17.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:17.078347+0000) 2022-09-10T06:58:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:17 smithi104 ceph-mon[97044]: pgmap v4774: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:58:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:17.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:17.709501+0000) 2022-09-10T06:58:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:17 smithi110 ceph-mon[97633]: pgmap v4774: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T06:58:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:18.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:18.078525+0000) 2022-09-10T06:58:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:18.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:18.709668+0000) 2022-09-10T06:58:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:19.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:19.078691+0000) 2022-09-10T06:58:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:19 smithi104 ceph-mon[97044]: pgmap v4775: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:58:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:19.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:19.709926+0000) 2022-09-10T06:58:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:19 smithi110 ceph-mon[97633]: pgmap v4775: 129 pgs: 129 active+clean; 8.2 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:58:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:20.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:20.078910+0000) 2022-09-10T06:58:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:20.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:20.710193+0000) 2022-09-10T06:58:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:21.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:21.079181+0000) 2022-09-10T06:58:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:21 smithi104 ceph-mon[97044]: pgmap v4776: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:58:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:21.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:21.710447+0000) 2022-09-10T06:58:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:21 smithi110 ceph-mon[97633]: pgmap v4776: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:58:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:22.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:22.079379+0000) 2022-09-10T06:58:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:22.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:22.710650+0000) 2022-09-10T06:58:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:23.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:23.079548+0000) 2022-09-10T06:58:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:23 smithi104 ceph-mon[97044]: pgmap v4777: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:58:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:23.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:23.710881+0000) 2022-09-10T06:58:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:23 smithi110 ceph-mon[97633]: pgmap v4777: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:58:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:24.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:24.079742+0000) 2022-09-10T06:58:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:24.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:24.711089+0000) 2022-09-10T06:58:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:25.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:25.079927+0000) 2022-09-10T06:58:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:25 smithi110 ceph-mon[97633]: pgmap v4778: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:58:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:25 smithi104 ceph-mon[97044]: pgmap v4778: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T06:58:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:25.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:25.711351+0000) 2022-09-10T06:58:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:26.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:26.080156+0000) 2022-09-10T06:58:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:26.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:26.711537+0000) 2022-09-10T06:58:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:27.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:27.080394+0000) 2022-09-10T06:58:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:27 smithi104 ceph-mon[97044]: pgmap v4779: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:27.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:27.711717+0000) 2022-09-10T06:58:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:27 smithi110 ceph-mon[97633]: pgmap v4779: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:28.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:28.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:28.080644+0000) 2022-09-10T06:58:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:28.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:28.711944+0000) 2022-09-10T06:58:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:29.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:29.080844+0000) 2022-09-10T06:58:29.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:29.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:29.712126+0000) 2022-09-10T06:58:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:29 smithi104 ceph-mon[97044]: pgmap v4780: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:29 smithi110 ceph-mon[97633]: pgmap v4780: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:58:29.973+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:58:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:30.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:30.081049+0000) 2022-09-10T06:58:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:30.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:30.712344+0000) 2022-09-10T06:58:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:31.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:31.081278+0000) 2022-09-10T06:58:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:31 smithi104 ceph-mon[97044]: pgmap v4781: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:31.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:31.712590+0000) 2022-09-10T06:58:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:31 smithi110 ceph-mon[97633]: pgmap v4781: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:32.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:32.081495+0000) 2022-09-10T06:58:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:32.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:32.712789+0000) 2022-09-10T06:58:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:33.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:33.081664+0000) 2022-09-10T06:58:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:33 smithi104 ceph-mon[97044]: pgmap v4782: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:34.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:33.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:33.713025+0000) 2022-09-10T06:58:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:33 smithi110 ceph-mon[97633]: pgmap v4782: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:34.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:34.081894+0000) 2022-09-10T06:58:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:34.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:34.713224+0000) 2022-09-10T06:58:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:35.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:35.082080+0000) 2022-09-10T06:58:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:35 smithi110 ceph-mon[97633]: pgmap v4783: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:35 smithi104 ceph-mon[97044]: pgmap v4783: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:35.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:35.713451+0000) 2022-09-10T06:58:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:36.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:36.082267+0000) 2022-09-10T06:58:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:36.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:36.713699+0000) 2022-09-10T06:58:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:37.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:37.082424+0000) 2022-09-10T06:58:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:37 smithi104 ceph-mon[97044]: pgmap v4784: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:37.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:37.713914+0000) 2022-09-10T06:58:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:37 smithi110 ceph-mon[97633]: pgmap v4784: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:38.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:38.082682+0000) 2022-09-10T06:58:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:38.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:38.714149+0000) 2022-09-10T06:58:39.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:39.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:39.082934+0000) 2022-09-10T06:58:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:39 smithi104 ceph-mon[97044]: pgmap v4785: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:39.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:39.714358+0000) 2022-09-10T06:58:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:39 smithi110 ceph-mon[97633]: pgmap v4785: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:40.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:40.083147+0000) 2022-09-10T06:58:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:40.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:40.714597+0000) 2022-09-10T06:58:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:41.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:41.083367+0000) 2022-09-10T06:58:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:41 smithi104 ceph-mon[97044]: pgmap v4786: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:41.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:41.714874+0000) 2022-09-10T06:58:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:41 smithi110 ceph-mon[97633]: pgmap v4786: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:42.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:42.083593+0000) 2022-09-10T06:58:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:42.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:42.715127+0000) 2022-09-10T06:58:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:43.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:43.083780+0000) 2022-09-10T06:58:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:43 smithi104 ceph-mon[97044]: pgmap v4787: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:43.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:43.715306+0000) 2022-09-10T06:58:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:43 smithi110 ceph-mon[97633]: pgmap v4787: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:44.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:44.083951+0000) 2022-09-10T06:58:44.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:44.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:44.715536+0000) 2022-09-10T06:58:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:58:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:58:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:58:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:58:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:58:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:58:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:58:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:58:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:45.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:45.084173+0000) 2022-09-10T06:58:45.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:58:44.980+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:58:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:45 smithi104 ceph-mon[97044]: pgmap v4788: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:45.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:45.715837+0000) 2022-09-10T06:58:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:45 smithi110 ceph-mon[97633]: pgmap v4788: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:46.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:46.084371+0000) 2022-09-10T06:58:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:46.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:46.716124+0000) 2022-09-10T06:58:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:47.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:47.084587+0000) 2022-09-10T06:58:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:47 smithi104 ceph-mon[97044]: pgmap v4789: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:47.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:47.716288+0000) 2022-09-10T06:58:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:47 smithi110 ceph-mon[97633]: pgmap v4789: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:48.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:48.084827+0000) 2022-09-10T06:58:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:48.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:48.716471+0000) 2022-09-10T06:58:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:49.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:49.084985+0000) 2022-09-10T06:58:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:49 smithi104 ceph-mon[97044]: pgmap v4790: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:49.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:49.716769+0000) 2022-09-10T06:58:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:49 smithi110 ceph-mon[97633]: pgmap v4790: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:50.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:50.085202+0000) 2022-09-10T06:58:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:50.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:50.716957+0000) 2022-09-10T06:58:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:51.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:51.085452+0000) 2022-09-10T06:58:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:51 smithi104 ceph-mon[97044]: pgmap v4791: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:51.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:51.717216+0000) 2022-09-10T06:58:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:51 smithi110 ceph-mon[97633]: pgmap v4791: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:52.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:52.085672+0000) 2022-09-10T06:58:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:52.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:52.717470+0000) 2022-09-10T06:58:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:53.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:53.085937+0000) 2022-09-10T06:58:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:53 smithi104 ceph-mon[97044]: pgmap v4792: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:53.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:53.717691+0000) 2022-09-10T06:58:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:53 smithi110 ceph-mon[97633]: pgmap v4792: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:54.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:54.086209+0000) 2022-09-10T06:58:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:54.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:54.717926+0000) 2022-09-10T06:58:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:55.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:55.086404+0000) 2022-09-10T06:58:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:55 smithi110 ceph-mon[97633]: pgmap v4793: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:55 smithi104 ceph-mon[97044]: pgmap v4793: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:58:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:55.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:55.718160+0000) 2022-09-10T06:58:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:56.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:56.086549+0000) 2022-09-10T06:58:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:56.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:56.718327+0000) 2022-09-10T06:58:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:57.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:57.086755+0000) 2022-09-10T06:58:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:57 smithi104 ceph-mon[97044]: pgmap v4794: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:57.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:57.718579+0000) 2022-09-10T06:58:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:57 smithi110 ceph-mon[97633]: pgmap v4794: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:58:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:58.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:58.086986+0000) 2022-09-10T06:58:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:58.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:58.718804+0000) 2022-09-10T06:58:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:58:59.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:59.087159+0000) 2022-09-10T06:59:00.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:58:59.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:58:59.719055+0000) 2022-09-10T06:59:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:58:59 smithi104 ceph-mon[97044]: pgmap v4795: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:58:59 smithi110 ceph-mon[97633]: pgmap v4795: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:58:59.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:59:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:00.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:00.087377+0000) 2022-09-10T06:59:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:00.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:00.719309+0000) 2022-09-10T06:59:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:01.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:01.087531+0000) 2022-09-10T06:59:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:01 smithi104 ceph-mon[97044]: pgmap v4796: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:01.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:01.719557+0000) 2022-09-10T06:59:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:01 smithi110 ceph-mon[97633]: pgmap v4796: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:02.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:02.087720+0000) 2022-09-10T06:59:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:02.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:02.719735+0000) 2022-09-10T06:59:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:03.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:03.087970+0000) 2022-09-10T06:59:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:03 smithi104 ceph-mon[97044]: pgmap v4797: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:03.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:03.719987+0000) 2022-09-10T06:59:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:03 smithi110 ceph-mon[97633]: pgmap v4797: 129 pgs: 129 active+clean; 8.2 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:04.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:04.088197+0000) 2022-09-10T06:59:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:04.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:04.720183+0000) 2022-09-10T06:59:05.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:05.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:05.088433+0000) 2022-09-10T06:59:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:05 smithi110 ceph-mon[97633]: pgmap v4798: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:59:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:05 smithi104 ceph-mon[97044]: pgmap v4798: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T06:59:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:05.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:05.720416+0000) 2022-09-10T06:59:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:06.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:06.088608+0000) 2022-09-10T06:59:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:06.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:06.720670+0000) 2022-09-10T06:59:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:07.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:07.088812+0000) 2022-09-10T06:59:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:07 smithi104 ceph-mon[97044]: pgmap v4799: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T06:59:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:07.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:07.720935+0000) 2022-09-10T06:59:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:07 smithi110 ceph-mon[97633]: pgmap v4799: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T06:59:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:08.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:08.089000+0000) 2022-09-10T06:59:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:08 smithi104 ceph-mon[97044]: osdmap e382: 12 total, 12 up, 12 in 2022-09-10T06:59:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:08.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:08.721062+0000) 2022-09-10T06:59:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:08 smithi110 ceph-mon[97633]: osdmap e382: 12 total, 12 up, 12 in 2022-09-10T06:59:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:09.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:09.089180+0000) 2022-09-10T06:59:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:09 smithi104 ceph-mon[97044]: pgmap v4801: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:59:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:09.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:09.721316+0000) 2022-09-10T06:59:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:09 smithi110 ceph-mon[97633]: pgmap v4801: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:59:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:10.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:10.089429+0000) 2022-09-10T06:59:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:10.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:10.721537+0000) 2022-09-10T06:59:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:11.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:11.089635+0000) 2022-09-10T06:59:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:11 smithi104 ceph-mon[97044]: pgmap v4802: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:59:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:11.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:11.721775+0000) 2022-09-10T06:59:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:11 smithi110 ceph-mon[97633]: pgmap v4802: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:59:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:12.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:12.089866+0000) 2022-09-10T06:59:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:12.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:12.721922+0000) 2022-09-10T06:59:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:13.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:13.090082+0000) 2022-09-10T06:59:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:13 smithi104 ceph-mon[97044]: pgmap v4803: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:59:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:13.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:13.722185+0000) 2022-09-10T06:59:14.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:13 smithi110 ceph-mon[97633]: pgmap v4803: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T06:59:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:14.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:14.090355+0000) 2022-09-10T06:59:15.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:14.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:14.722442+0000) 2022-09-10T06:59:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:59:15.004+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:59:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:15.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:15.090557+0000) 2022-09-10T06:59:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:15 smithi110 ceph-mon[97633]: osdmap e383: 12 total, 12 up, 12 in 2022-09-10T06:59:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:15 smithi110 ceph-mon[97633]: pgmap v4805: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:59:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:15 smithi104 ceph-mon[97044]: osdmap e383: 12 total, 12 up, 12 in 2022-09-10T06:59:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:15 smithi104 ceph-mon[97044]: pgmap v4805: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T06:59:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:15.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:15.722623+0000) 2022-09-10T06:59:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:16.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:16.090780+0000) 2022-09-10T06:59:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:16.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:16.722818+0000) 2022-09-10T06:59:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:17.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:17.091026+0000) 2022-09-10T06:59:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:17 smithi104 ceph-mon[97044]: pgmap v4806: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:59:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:17.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:17.723043+0000) 2022-09-10T06:59:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:17 smithi110 ceph-mon[97633]: pgmap v4806: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T06:59:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:18.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:18.091252+0000) 2022-09-10T06:59:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:18.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:18.723249+0000) 2022-09-10T06:59:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:19.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:19.091495+0000) 2022-09-10T06:59:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:19 smithi104 ceph-mon[97044]: pgmap v4807: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:59:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:19.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:19.723517+0000) 2022-09-10T06:59:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:19 smithi110 ceph-mon[97633]: pgmap v4807: 129 pgs: 129 active+clean; 8.2 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T06:59:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:20.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:20.091749+0000) 2022-09-10T06:59:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:20.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:20.723729+0000) 2022-09-10T06:59:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:21.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:21.091989+0000) 2022-09-10T06:59:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:21 smithi104 ceph-mon[97044]: pgmap v4808: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:59:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:21.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:21.723956+0000) 2022-09-10T06:59:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:21 smithi110 ceph-mon[97633]: pgmap v4808: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:59:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:22.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:22.092232+0000) 2022-09-10T06:59:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:22.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:22.724170+0000) 2022-09-10T06:59:23.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:23.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:23.092471+0000) 2022-09-10T06:59:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:23 smithi104 ceph-mon[97044]: pgmap v4809: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:59:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:23.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:23.724399+0000) 2022-09-10T06:59:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:23 smithi110 ceph-mon[97633]: pgmap v4809: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T06:59:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:24.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:24.092674+0000) 2022-09-10T06:59:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:24.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:24.724604+0000) 2022-09-10T06:59:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:25.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:25.092939+0000) 2022-09-10T06:59:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:25 smithi110 ceph-mon[97633]: pgmap v4810: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T06:59:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:25 smithi104 ceph-mon[97044]: pgmap v4810: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T06:59:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:25.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:25.724822+0000) 2022-09-10T06:59:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:26.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:26.093174+0000) 2022-09-10T06:59:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:26.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:26.725129+0000) 2022-09-10T06:59:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:27.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:27.093439+0000) 2022-09-10T06:59:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:27 smithi104 ceph-mon[97044]: pgmap v4811: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:27.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:27.725345+0000) 2022-09-10T06:59:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:27 smithi110 ceph-mon[97633]: pgmap v4811: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:28.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:28.093710+0000) 2022-09-10T06:59:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:28.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:28.725603+0000) 2022-09-10T06:59:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:29.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:29.093934+0000) 2022-09-10T06:59:30.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:29.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:29.725828+0000) 2022-09-10T06:59:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:29 smithi104 ceph-mon[97044]: pgmap v4812: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:29 smithi110 ceph-mon[97633]: pgmap v4812: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:30.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:30.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:30.094140+0000) 2022-09-10T06:59:30.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:59:30.018+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:59:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:30.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:30.726062+0000) 2022-09-10T06:59:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:31.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:31.094346+0000) 2022-09-10T06:59:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:31 smithi104 ceph-mon[97044]: pgmap v4813: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:31.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:31.726368+0000) 2022-09-10T06:59:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:31 smithi110 ceph-mon[97633]: pgmap v4813: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:32.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:32.094494+0000) 2022-09-10T06:59:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:32.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:32.726559+0000) 2022-09-10T06:59:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:33.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:33.094679+0000) 2022-09-10T06:59:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:33 smithi104 ceph-mon[97044]: pgmap v4814: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:33.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:33.726791+0000) 2022-09-10T06:59:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:33 smithi110 ceph-mon[97633]: pgmap v4814: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:34.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:34.094881+0000) 2022-09-10T06:59:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:34.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:34.727041+0000) 2022-09-10T06:59:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:35.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:35.095184+0000) 2022-09-10T06:59:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:35 smithi110 ceph-mon[97633]: pgmap v4815: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:35 smithi104 ceph-mon[97044]: pgmap v4815: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:35.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:35.727244+0000) 2022-09-10T06:59:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:36.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:36.095380+0000) 2022-09-10T06:59:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:36.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:36.727436+0000) 2022-09-10T06:59:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:37.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:37.095614+0000) 2022-09-10T06:59:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:37 smithi104 ceph-mon[97044]: pgmap v4816: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:37.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:37.727652+0000) 2022-09-10T06:59:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:37 smithi110 ceph-mon[97633]: pgmap v4816: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:38.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:38.095845+0000) 2022-09-10T06:59:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:38.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:38.727864+0000) 2022-09-10T06:59:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:39.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:39.096081+0000) 2022-09-10T06:59:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:39 smithi104 ceph-mon[97044]: pgmap v4817: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:39.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:39.728034+0000) 2022-09-10T06:59:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:39 smithi110 ceph-mon[97633]: pgmap v4817: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:40.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:40.096306+0000) 2022-09-10T06:59:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:40.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:40.728252+0000) 2022-09-10T06:59:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:41.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:41.096545+0000) 2022-09-10T06:59:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:41 smithi104 ceph-mon[97044]: pgmap v4818: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:41.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:41.728497+0000) 2022-09-10T06:59:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:41 smithi110 ceph-mon[97633]: pgmap v4818: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:42.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:42.096725+0000) 2022-09-10T06:59:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:42.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:42.728745+0000) 2022-09-10T06:59:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:43.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:43.096953+0000) 2022-09-10T06:59:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:43 smithi104 ceph-mon[97044]: pgmap v4819: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:43.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:43.728980+0000) 2022-09-10T06:59:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:43 smithi110 ceph-mon[97633]: pgmap v4819: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:44.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:44.097218+0000) 2022-09-10T06:59:45.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:44.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:44.729203+0000) 2022-09-10T06:59:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:59:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:59:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:59:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:59:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:59:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:59:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T06:59:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T06:59:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:45.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:45.097433+0000) 2022-09-10T06:59:45.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 06:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T06:59:45.033+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T06:59:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:45 smithi104 ceph-mon[97044]: pgmap v4820: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:45.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:45.729450+0000) 2022-09-10T06:59:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:45 smithi110 ceph-mon[97633]: pgmap v4820: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:46.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:46.097666+0000) 2022-09-10T06:59:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:46.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:46.729673+0000) 2022-09-10T06:59:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:47.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:47.097814+0000) 2022-09-10T06:59:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:47 smithi104 ceph-mon[97044]: pgmap v4821: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:47.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:47.729889+0000) 2022-09-10T06:59:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:47 smithi110 ceph-mon[97633]: pgmap v4821: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:48.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:48.098034+0000) 2022-09-10T06:59:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:48.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:48.730198+0000) 2022-09-10T06:59:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:49.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:49.098234+0000) 2022-09-10T06:59:50.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:49 smithi104 ceph-mon[97044]: pgmap v4822: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:49.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:49.730382+0000) 2022-09-10T06:59:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:49 smithi110 ceph-mon[97633]: pgmap v4822: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:50.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:50.098436+0000) 2022-09-10T06:59:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:50.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:50.730642+0000) 2022-09-10T06:59:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:51.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:51.098673+0000) 2022-09-10T06:59:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:51 smithi104 ceph-mon[97044]: pgmap v4823: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:51.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:51.730874+0000) 2022-09-10T06:59:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:51 smithi110 ceph-mon[97633]: pgmap v4823: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:52.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:52.098944+0000) 2022-09-10T06:59:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:52.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:52.731017+0000) 2022-09-10T06:59:53.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:53.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:53.099156+0000) 2022-09-10T06:59:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:53 smithi104 ceph-mon[97044]: pgmap v4824: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:53.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:53.731241+0000) 2022-09-10T06:59:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:53 smithi110 ceph-mon[97633]: pgmap v4824: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:54.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:54.099338+0000) 2022-09-10T06:59:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:54.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:54.731430+0000) 2022-09-10T06:59:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:55.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:55.099590+0000) 2022-09-10T06:59:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:55 smithi110 ceph-mon[97633]: pgmap v4825: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:55 smithi104 ceph-mon[97044]: pgmap v4825: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T06:59:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:55.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:55.731650+0000) 2022-09-10T06:59:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:56.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:56.099821+0000) 2022-09-10T06:59:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:56.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:56.731818+0000) 2022-09-10T06:59:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:57.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:57.100034+0000) 2022-09-10T06:59:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:57 smithi104 ceph-mon[97044]: pgmap v4826: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:57.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:57.731986+0000) 2022-09-10T06:59:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:57 smithi110 ceph-mon[97633]: pgmap v4826: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T06:59:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:58.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:58.100246+0000) 2022-09-10T06:59:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:58.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:58.732238+0000) 2022-09-10T06:59:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 06:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T06:59:59.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:59.100437+0000) 2022-09-10T07:00:00.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 06:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T06:59:59.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T05:59:59.732510+0000) 2022-09-10T07:00:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 06:59:59 smithi104 ceph-mon[97044]: pgmap v4827: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 06:59:59 smithi110 ceph-mon[97633]: pgmap v4827: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:00.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:00.100618+0000) 2022-09-10T07:00:00.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:00:00.044+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:00:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:00:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:00.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:00.732836+0000) 2022-09-10T07:00:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:00:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:01.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:01.100837+0000) 2022-09-10T07:00:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:01 smithi104 ceph-mon[97044]: pgmap v4828: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:01.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:01.733053+0000) 2022-09-10T07:00:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:01 smithi110 ceph-mon[97633]: pgmap v4828: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:02.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:02.101136+0000) 2022-09-10T07:00:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:02.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:02.733281+0000) 2022-09-10T07:00:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:03.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:03.101321+0000) 2022-09-10T07:00:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:03 smithi104 ceph-mon[97044]: pgmap v4829: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:03.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:03.733478+0000) 2022-09-10T07:00:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:03 smithi110 ceph-mon[97633]: pgmap v4829: 129 pgs: 129 active+clean; 8.2 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:04.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:04.101524+0000) 2022-09-10T07:00:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:04.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:04.733715+0000) 2022-09-10T07:00:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:05.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:05.101774+0000) 2022-09-10T07:00:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:05 smithi110 ceph-mon[97633]: pgmap v4830: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:00:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:05 smithi104 ceph-mon[97044]: pgmap v4830: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:00:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:05.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:05.733914+0000) 2022-09-10T07:00:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:06.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:06.102021+0000) 2022-09-10T07:00:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:06.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:06.734192+0000) 2022-09-10T07:00:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:07.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:07.102176+0000) 2022-09-10T07:00:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:07 smithi104 ceph-mon[97044]: pgmap v4831: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:00:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:07.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:07.734426+0000) 2022-09-10T07:00:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:07 smithi110 ceph-mon[97633]: pgmap v4831: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:00:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:08.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:08.102427+0000) 2022-09-10T07:00:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:08 smithi104 ceph-mon[97044]: osdmap e384: 12 total, 12 up, 12 in 2022-09-10T07:00:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:08.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:08.734642+0000) 2022-09-10T07:00:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:08 smithi110 ceph-mon[97633]: osdmap e384: 12 total, 12 up, 12 in 2022-09-10T07:00:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:09.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:09.102676+0000) 2022-09-10T07:00:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:09 smithi104 ceph-mon[97044]: pgmap v4833: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:00:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:09.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:09.734866+0000) 2022-09-10T07:00:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:09 smithi110 ceph-mon[97633]: pgmap v4833: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:00:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:10.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:10.102908+0000) 2022-09-10T07:00:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:10.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:10.735070+0000) 2022-09-10T07:00:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:11.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:11.103091+0000) 2022-09-10T07:00:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:11 smithi104 ceph-mon[97044]: pgmap v4834: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:00:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:11.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:11.735306+0000) 2022-09-10T07:00:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:11 smithi110 ceph-mon[97633]: pgmap v4834: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:00:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:12.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:12.103287+0000) 2022-09-10T07:00:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:12.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:12.735535+0000) 2022-09-10T07:00:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:13.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:13.103539+0000) 2022-09-10T07:00:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:13 smithi104 ceph-mon[97044]: pgmap v4835: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:00:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:13.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:13.735841+0000) 2022-09-10T07:00:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:13 smithi110 ceph-mon[97633]: pgmap v4835: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:00:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:14.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:14.103792+0000) 2022-09-10T07:00:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:14.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:14.736080+0000) 2022-09-10T07:00:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:15.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:15.103987+0000) 2022-09-10T07:00:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:00:15.053+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:00:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:15 smithi110 ceph-mon[97633]: osdmap e385: 12 total, 12 up, 12 in 2022-09-10T07:00:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:15 smithi110 ceph-mon[97633]: pgmap v4837: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:00:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:15 smithi104 ceph-mon[97044]: osdmap e385: 12 total, 12 up, 12 in 2022-09-10T07:00:15.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:15 smithi104 ceph-mon[97044]: pgmap v4837: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:00:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:15.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:15.736317+0000) 2022-09-10T07:00:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:16.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:16.104201+0000) 2022-09-10T07:00:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:16.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:16.736501+0000) 2022-09-10T07:00:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:17.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:17.104357+0000) 2022-09-10T07:00:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:17 smithi104 ceph-mon[97044]: pgmap v4838: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:00:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:17.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:17.736705+0000) 2022-09-10T07:00:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:17 smithi110 ceph-mon[97633]: pgmap v4838: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:00:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:18.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:18.104590+0000) 2022-09-10T07:00:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:18.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:18.736873+0000) 2022-09-10T07:00:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:19.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:19.104817+0000) 2022-09-10T07:00:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:19 smithi104 ceph-mon[97044]: pgmap v4839: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:00:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:19.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:19.737069+0000) 2022-09-10T07:00:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:19 smithi110 ceph-mon[97633]: pgmap v4839: 129 pgs: 129 active+clean; 8.2 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:00:20.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:20.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:20.105005+0000) 2022-09-10T07:00:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:20.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:20.737363+0000) 2022-09-10T07:00:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:21.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:21.105233+0000) 2022-09-10T07:00:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:21 smithi104 ceph-mon[97044]: pgmap v4840: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:00:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:21.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:21.737613+0000) 2022-09-10T07:00:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:21 smithi110 ceph-mon[97633]: pgmap v4840: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:00:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:22.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:22.105490+0000) 2022-09-10T07:00:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:22.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:22.737786+0000) 2022-09-10T07:00:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:23.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:23.105729+0000) 2022-09-10T07:00:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:23 smithi104 ceph-mon[97044]: pgmap v4841: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:00:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:23.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:23.738003+0000) 2022-09-10T07:00:24.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:23 smithi110 ceph-mon[97633]: pgmap v4841: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:00:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:24.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:24.105945+0000) 2022-09-10T07:00:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:24.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:24.738224+0000) 2022-09-10T07:00:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:25.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:25.106205+0000) 2022-09-10T07:00:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:25 smithi110 ceph-mon[97633]: pgmap v4842: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T07:00:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:25 smithi104 ceph-mon[97044]: pgmap v4842: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T07:00:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:25.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:25.738472+0000) 2022-09-10T07:00:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:26.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:26.106406+0000) 2022-09-10T07:00:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:26.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:26.738702+0000) 2022-09-10T07:00:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:27.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:27.106674+0000) 2022-09-10T07:00:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:27 smithi104 ceph-mon[97044]: pgmap v4843: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:27.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:27.738925+0000) 2022-09-10T07:00:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:27 smithi110 ceph-mon[97633]: pgmap v4843: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:28.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:28.106889+0000) 2022-09-10T07:00:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:28.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:28.739083+0000) 2022-09-10T07:00:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:29.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:29.107203+0000) 2022-09-10T07:00:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:29 smithi104 ceph-mon[97044]: pgmap v4844: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:29.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:29.739340+0000) 2022-09-10T07:00:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:29 smithi110 ceph-mon[97633]: pgmap v4844: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:00:30.069+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:00:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:30.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:30.107482+0000) 2022-09-10T07:00:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:30.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:30.739544+0000) 2022-09-10T07:00:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:31.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:31.107691+0000) 2022-09-10T07:00:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:31 smithi104 ceph-mon[97044]: pgmap v4845: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:31.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:31.739716+0000) 2022-09-10T07:00:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:31 smithi110 ceph-mon[97633]: pgmap v4845: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:32.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:32.107921+0000) 2022-09-10T07:00:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:32.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:32.739948+0000) 2022-09-10T07:00:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:33.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:33.108181+0000) 2022-09-10T07:00:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:33 smithi104 ceph-mon[97044]: pgmap v4846: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:33.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:33.740190+0000) 2022-09-10T07:00:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:33 smithi110 ceph-mon[97633]: pgmap v4846: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:34.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:34.108436+0000) 2022-09-10T07:00:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:34.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:34.740422+0000) 2022-09-10T07:00:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:35.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:35.108695+0000) 2022-09-10T07:00:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:35 smithi110 ceph-mon[97633]: pgmap v4847: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:35 smithi104 ceph-mon[97044]: pgmap v4847: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:35.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:35.740613+0000) 2022-09-10T07:00:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:36.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:36.108891+0000) 2022-09-10T07:00:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:36.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:36.740884+0000) 2022-09-10T07:00:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:37.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:37.109157+0000) 2022-09-10T07:00:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:37 smithi104 ceph-mon[97044]: pgmap v4848: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:37.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:37.741058+0000) 2022-09-10T07:00:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:37 smithi110 ceph-mon[97633]: pgmap v4848: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:38.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:38.109427+0000) 2022-09-10T07:00:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:38.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:38.741242+0000) 2022-09-10T07:00:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:39.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:39.109716+0000) 2022-09-10T07:00:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:39 smithi104 ceph-mon[97044]: pgmap v4849: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:39.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:39.741476+0000) 2022-09-10T07:00:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:39 smithi110 ceph-mon[97633]: pgmap v4849: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:40.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:40.110003+0000) 2022-09-10T07:00:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:40.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:40.741648+0000) 2022-09-10T07:00:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:41.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:41.110252+0000) 2022-09-10T07:00:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:41 smithi104 ceph-mon[97044]: pgmap v4850: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:41.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:41.741808+0000) 2022-09-10T07:00:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:41 smithi110 ceph-mon[97633]: pgmap v4850: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:42.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:42.110421+0000) 2022-09-10T07:00:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:42.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:42.742065+0000) 2022-09-10T07:00:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:43.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:43.110679+0000) 2022-09-10T07:00:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:43 smithi104 ceph-mon[97044]: pgmap v4851: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:43.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:43.742319+0000) 2022-09-10T07:00:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:43 smithi110 ceph-mon[97633]: pgmap v4851: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:44.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:44.110936+0000) 2022-09-10T07:00:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:00:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:00:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:00:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:00:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:44.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:44.742554+0000) 2022-09-10T07:00:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:00:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:00:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:00:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:00:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:45.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:45.111141+0000) 2022-09-10T07:00:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:00:45.075+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:00:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:45 smithi104 ceph-mon[97044]: pgmap v4852: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:45.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:45.742757+0000) 2022-09-10T07:00:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:45 smithi110 ceph-mon[97633]: pgmap v4852: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:46.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:46.111337+0000) 2022-09-10T07:00:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:46.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:46.742922+0000) 2022-09-10T07:00:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:47.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:47.111591+0000) 2022-09-10T07:00:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:47 smithi104 ceph-mon[97044]: pgmap v4853: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:47.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:47.743158+0000) 2022-09-10T07:00:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:47 smithi110 ceph-mon[97633]: pgmap v4853: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:48.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:48.111839+0000) 2022-09-10T07:00:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:48.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:48.743365+0000) 2022-09-10T07:00:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:49.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:49.111982+0000) 2022-09-10T07:00:50.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:49.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:49.743599+0000) 2022-09-10T07:00:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:49 smithi104 ceph-mon[97044]: pgmap v4854: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:49 smithi110 ceph-mon[97633]: pgmap v4854: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:00:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:50.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:50.112229+0000) 2022-09-10T07:00:51.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:50.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:50.743828+0000) 2022-09-10T07:00:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:51.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:51.112474+0000) 2022-09-10T07:00:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:51 smithi104 ceph-mon[97044]: pgmap v4855: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:51.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:51.744070+0000) 2022-09-10T07:00:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:51 smithi110 ceph-mon[97633]: pgmap v4855: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:52.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:52.112677+0000) 2022-09-10T07:00:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:52.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:52.744303+0000) 2022-09-10T07:00:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:53.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:53.112914+0000) 2022-09-10T07:00:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:53 smithi104 ceph-mon[97044]: pgmap v4856: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:00:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:53.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:53.744552+0000) 2022-09-10T07:00:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:53 smithi110 ceph-mon[97633]: pgmap v4856: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:00:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:54.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:54.113146+0000) 2022-09-10T07:00:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:54.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:54.744807+0000) 2022-09-10T07:00:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:55.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:55.113363+0000) 2022-09-10T07:00:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:55 smithi110 ceph-mon[97633]: pgmap v4857: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:55 smithi104 ceph-mon[97044]: pgmap v4857: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:00:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:55.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:55.745058+0000) 2022-09-10T07:00:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:56.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:56.113612+0000) 2022-09-10T07:00:57.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:56.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:56.745340+0000) 2022-09-10T07:00:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:57.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:57.113895+0000) 2022-09-10T07:00:58.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:57 smithi104 ceph-mon[97044]: pgmap v4858: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:00:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:57.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:57.745594+0000) 2022-09-10T07:00:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:57 smithi110 ceph-mon[97633]: pgmap v4858: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:00:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:58.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:58.114068+0000) 2022-09-10T07:00:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:58.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:58.745819+0000) 2022-09-10T07:00:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:00:59.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:59.114262+0000) 2022-09-10T07:01:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:00:59 smithi104 ceph-mon[97044]: pgmap v4859: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:01:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:00:59.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:00:59.745979+0000) 2022-09-10T07:01:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:00:59 smithi110 ceph-mon[97633]: pgmap v4859: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:01:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:00.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:00.114487+0000) 2022-09-10T07:01:00.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:01:00.091+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:01:01.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:00.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:00.746206+0000) 2022-09-10T07:01:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:01.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:01.114737+0000) 2022-09-10T07:01:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:01 smithi104 ceph-mon[97044]: pgmap v4860: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:01.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:01.746399+0000) 2022-09-10T07:01:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:01 smithi110 ceph-mon[97633]: pgmap v4860: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:02.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:02.114878+0000) 2022-09-10T07:01:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:02.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:02.746582+0000) 2022-09-10T07:01:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:03.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:03.115125+0000) 2022-09-10T07:01:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:03 smithi104 ceph-mon[97044]: pgmap v4861: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:03.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:03.746823+0000) 2022-09-10T07:01:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:03 smithi110 ceph-mon[97633]: pgmap v4861: 129 pgs: 129 active+clean; 8.2 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:04.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:04.115379+0000) 2022-09-10T07:01:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:04.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:04.747050+0000) 2022-09-10T07:01:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:05.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:05.115615+0000) 2022-09-10T07:01:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:05 smithi110 ceph-mon[97633]: pgmap v4862: 129 pgs: 129 active+clean; 8.2 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:01:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:05 smithi104 ceph-mon[97044]: pgmap v4862: 129 pgs: 129 active+clean; 8.2 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:01:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:05.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:05.747297+0000) 2022-09-10T07:01:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:06.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:06.115868+0000) 2022-09-10T07:01:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:06.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:06.747496+0000) 2022-09-10T07:01:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:07.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:07.115997+0000) 2022-09-10T07:01:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:07 smithi104 ceph-mon[97044]: pgmap v4863: 129 pgs: 129 active+clean; 8.2 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:01:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:07.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:07.747749+0000) 2022-09-10T07:01:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:07 smithi110 ceph-mon[97633]: pgmap v4863: 129 pgs: 129 active+clean; 8.2 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:01:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:08.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:08.116225+0000) 2022-09-10T07:01:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:08 smithi104 ceph-mon[97044]: osdmap e386: 12 total, 12 up, 12 in 2022-09-10T07:01:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:08.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:08.747941+0000) 2022-09-10T07:01:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:08 smithi110 ceph-mon[97633]: osdmap e386: 12 total, 12 up, 12 in 2022-09-10T07:01:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:09.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:09.116458+0000) 2022-09-10T07:01:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:09 smithi104 ceph-mon[97044]: pgmap v4865: 129 pgs: 129 active+clean; 8.2 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:01:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:09.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:09.748200+0000) 2022-09-10T07:01:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:09 smithi110 ceph-mon[97633]: pgmap v4865: 129 pgs: 129 active+clean; 8.2 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:01:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:10.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:10.116600+0000) 2022-09-10T07:01:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:10.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:10.748456+0000) 2022-09-10T07:01:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:11.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:11.116817+0000) 2022-09-10T07:01:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:11 smithi104 ceph-mon[97044]: pgmap v4866: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:01:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:11.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:11.748645+0000) 2022-09-10T07:01:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:11 smithi110 ceph-mon[97633]: pgmap v4866: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:01:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:12.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:12.117009+0000) 2022-09-10T07:01:13.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:12.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:12.748841+0000) 2022-09-10T07:01:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:13.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:13.117256+0000) 2022-09-10T07:01:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:13 smithi104 ceph-mon[97044]: pgmap v4867: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:01:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:13.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:13.749014+0000) 2022-09-10T07:01:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:13 smithi110 ceph-mon[97633]: pgmap v4867: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:01:14.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:14.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:14.117482+0000) 2022-09-10T07:01:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:14.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:14.749247+0000) 2022-09-10T07:01:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:15.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:15.117741+0000) 2022-09-10T07:01:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:01:15.094+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:01:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:15 smithi110 ceph-mon[97633]: osdmap e387: 12 total, 12 up, 12 in 2022-09-10T07:01:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:15 smithi110 ceph-mon[97633]: pgmap v4869: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:01:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:15 smithi104 ceph-mon[97044]: osdmap e387: 12 total, 12 up, 12 in 2022-09-10T07:01:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:15 smithi104 ceph-mon[97044]: pgmap v4869: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:01:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:15.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:15.749441+0000) 2022-09-10T07:01:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:16.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:16.117992+0000) 2022-09-10T07:01:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:16.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:16.749650+0000) 2022-09-10T07:01:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:17.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:17.118207+0000) 2022-09-10T07:01:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:17 smithi104 ceph-mon[97044]: pgmap v4870: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:01:18.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:17.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:17.749900+0000) 2022-09-10T07:01:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:17 smithi110 ceph-mon[97633]: pgmap v4870: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:01:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:18.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:18.118409+0000) 2022-09-10T07:01:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:18.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:18.750185+0000) 2022-09-10T07:01:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:19.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:19.118552+0000) 2022-09-10T07:01:20.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:19 smithi104 ceph-mon[97044]: pgmap v4871: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:01:20.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:19.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:19.750445+0000) 2022-09-10T07:01:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:19 smithi110 ceph-mon[97633]: pgmap v4871: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:01:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:20.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:20.118711+0000) 2022-09-10T07:01:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:20.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:20.750624+0000) 2022-09-10T07:01:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:21.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:21.118953+0000) 2022-09-10T07:01:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:21 smithi104 ceph-mon[97044]: pgmap v4872: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:01:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:21.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:21.750843+0000) 2022-09-10T07:01:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:21 smithi110 ceph-mon[97633]: pgmap v4872: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:01:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:22.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:22.119167+0000) 2022-09-10T07:01:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:22.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:22.751115+0000) 2022-09-10T07:01:23.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:23.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:23.119346+0000) 2022-09-10T07:01:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:23 smithi104 ceph-mon[97044]: pgmap v4873: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:01:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:23.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:23.751351+0000) 2022-09-10T07:01:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:23 smithi110 ceph-mon[97633]: pgmap v4873: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:01:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:24.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:24.119558+0000) 2022-09-10T07:01:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:24.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:24.751559+0000) 2022-09-10T07:01:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:25.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:25.119766+0000) 2022-09-10T07:01:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:25 smithi110 ceph-mon[97633]: pgmap v4874: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T07:01:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:25 smithi104 ceph-mon[97044]: pgmap v4874: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T07:01:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:25.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:25.751783+0000) 2022-09-10T07:01:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:26.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:26.119954+0000) 2022-09-10T07:01:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:26.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:26.752052+0000) 2022-09-10T07:01:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:27.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:27.120138+0000) 2022-09-10T07:01:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:27 smithi104 ceph-mon[97044]: pgmap v4875: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:27.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:27.752307+0000) 2022-09-10T07:01:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:27 smithi110 ceph-mon[97633]: pgmap v4875: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:28.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:28.120366+0000) 2022-09-10T07:01:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:28.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:28.752518+0000) 2022-09-10T07:01:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:29.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:29.120619+0000) 2022-09-10T07:01:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:29 smithi104 ceph-mon[97044]: pgmap v4876: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:29.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:29.752765+0000) 2022-09-10T07:01:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:29 smithi110 ceph-mon[97633]: pgmap v4876: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:01:30.100+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:01:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:30.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:30.120844+0000) 2022-09-10T07:01:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:30.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:30.752922+0000) 2022-09-10T07:01:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:31.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:31.121035+0000) 2022-09-10T07:01:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:31 smithi104 ceph-mon[97044]: pgmap v4877: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:31.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:31.753086+0000) 2022-09-10T07:01:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:31 smithi110 ceph-mon[97633]: pgmap v4877: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:32.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:32.121232+0000) 2022-09-10T07:01:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:32.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:32.753350+0000) 2022-09-10T07:01:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:33.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:33.121459+0000) 2022-09-10T07:01:34.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:33 smithi104 ceph-mon[97044]: pgmap v4878: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:33.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:33.753543+0000) 2022-09-10T07:01:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:33 smithi110 ceph-mon[97633]: pgmap v4878: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:34.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:34.121697+0000) 2022-09-10T07:01:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:34.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:34.753770+0000) 2022-09-10T07:01:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:35.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:35.121916+0000) 2022-09-10T07:01:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:35 smithi110 ceph-mon[97633]: pgmap v4879: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:35 smithi104 ceph-mon[97044]: pgmap v4879: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:35.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:35.754012+0000) 2022-09-10T07:01:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:36.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:36.122144+0000) 2022-09-10T07:01:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:36.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:36.754235+0000) 2022-09-10T07:01:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:37.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:37.122376+0000) 2022-09-10T07:01:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:37 smithi104 ceph-mon[97044]: pgmap v4880: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:37.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:37.754476+0000) 2022-09-10T07:01:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:37 smithi110 ceph-mon[97633]: pgmap v4880: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:38.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:38.122618+0000) 2022-09-10T07:01:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:38.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:38.754658+0000) 2022-09-10T07:01:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:39.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:39.122821+0000) 2022-09-10T07:01:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:39 smithi104 ceph-mon[97044]: pgmap v4881: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:39.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:39.754898+0000) 2022-09-10T07:01:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:39 smithi110 ceph-mon[97633]: pgmap v4881: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:40.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:40.123052+0000) 2022-09-10T07:01:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:40.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:40.755138+0000) 2022-09-10T07:01:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:41.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:41.123255+0000) 2022-09-10T07:01:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:41 smithi104 ceph-mon[97044]: pgmap v4882: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:41.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:41.755350+0000) 2022-09-10T07:01:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:41 smithi110 ceph-mon[97633]: pgmap v4882: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:42.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:42.123429+0000) 2022-09-10T07:01:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:42.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:42.755551+0000) 2022-09-10T07:01:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:43.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:43.123635+0000) 2022-09-10T07:01:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:43 smithi104 ceph-mon[97044]: pgmap v4883: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:43.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:43.755768+0000) 2022-09-10T07:01:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:43 smithi110 ceph-mon[97633]: pgmap v4883: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:44.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:44.123813+0000) 2022-09-10T07:01:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:01:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:01:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:01:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:01:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:44.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:44.755963+0000) 2022-09-10T07:01:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:01:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:01:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:01:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:01:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:45.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:45.124049+0000) 2022-09-10T07:01:45.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:01:45.112+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:01:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:45 smithi104 ceph-mon[97044]: pgmap v4884: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:45.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:45.756189+0000) 2022-09-10T07:01:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:45 smithi110 ceph-mon[97633]: pgmap v4884: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:46.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:46.124249+0000) 2022-09-10T07:01:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:46.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:46.756342+0000) 2022-09-10T07:01:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:47.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:47.124405+0000) 2022-09-10T07:01:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:47 smithi104 ceph-mon[97044]: pgmap v4885: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:47.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:47.756508+0000) 2022-09-10T07:01:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:47 smithi110 ceph-mon[97633]: pgmap v4885: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:48.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:48.124659+0000) 2022-09-10T07:01:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:48.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:48.756653+0000) 2022-09-10T07:01:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:49.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:49.124830+0000) 2022-09-10T07:01:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:49 smithi104 ceph-mon[97044]: pgmap v4886: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:49.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:49.756905+0000) 2022-09-10T07:01:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:49 smithi110 ceph-mon[97633]: pgmap v4886: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:50.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:50.125030+0000) 2022-09-10T07:01:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:50.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:50.757165+0000) 2022-09-10T07:01:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:51.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:51.125255+0000) 2022-09-10T07:01:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:51 smithi104 ceph-mon[97044]: pgmap v4887: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:51.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:51.757447+0000) 2022-09-10T07:01:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:51 smithi110 ceph-mon[97633]: pgmap v4887: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:52.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:52.125541+0000) 2022-09-10T07:01:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:52.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:52.757699+0000) 2022-09-10T07:01:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:53.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:53.125762+0000) 2022-09-10T07:01:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:53 smithi104 ceph-mon[97044]: pgmap v4888: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:53.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:53.757916+0000) 2022-09-10T07:01:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:53 smithi110 ceph-mon[97633]: pgmap v4888: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:54.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:54.125992+0000) 2022-09-10T07:01:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:54.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:54.758190+0000) 2022-09-10T07:01:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:55.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:55.126192+0000) 2022-09-10T07:01:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:55 smithi110 ceph-mon[97633]: pgmap v4889: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:55 smithi104 ceph-mon[97044]: pgmap v4889: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:01:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:55.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:55.758447+0000) 2022-09-10T07:01:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:56.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:56.126412+0000) 2022-09-10T07:01:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:56.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:56.758683+0000) 2022-09-10T07:01:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:57.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:57.126675+0000) 2022-09-10T07:01:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:57 smithi104 ceph-mon[97044]: pgmap v4890: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:57.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:57.758882+0000) 2022-09-10T07:01:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:57 smithi110 ceph-mon[97633]: pgmap v4890: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:01:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:58.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:58.126888+0000) 2022-09-10T07:01:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:58.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:58.759117+0000) 2022-09-10T07:01:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:01:59.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:59.127149+0000) 2022-09-10T07:02:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:01:59 smithi104 ceph-mon[97044]: pgmap v4891: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:01:59.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:01:59.759371+0000) 2022-09-10T07:02:00.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:01:59 smithi110 ceph-mon[97633]: pgmap v4891: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:02:00.126+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:02:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:00.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:00.127351+0000) 2022-09-10T07:02:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:00.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:00.759584+0000) 2022-09-10T07:02:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:01.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:01.127561+0000) 2022-09-10T07:02:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:01 smithi104 ceph-mon[97044]: pgmap v4892: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:01.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:01.759850+0000) 2022-09-10T07:02:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:01 smithi110 ceph-mon[97633]: pgmap v4892: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:02.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:02.127768+0000) 2022-09-10T07:02:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:02.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:02.760152+0000) 2022-09-10T07:02:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:03.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:03.127981+0000) 2022-09-10T07:02:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:03 smithi104 ceph-mon[97044]: pgmap v4893: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:03.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:03.760328+0000) 2022-09-10T07:02:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:03 smithi110 ceph-mon[97633]: pgmap v4893: 129 pgs: 129 active+clean; 8.2 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:04.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:04.128230+0000) 2022-09-10T07:02:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:04.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:04.760562+0000) 2022-09-10T07:02:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:05.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:05.128450+0000) 2022-09-10T07:02:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:05 smithi110 ceph-mon[97633]: pgmap v4894: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:02:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:05 smithi104 ceph-mon[97044]: pgmap v4894: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:02:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:05.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:05.760840+0000) 2022-09-10T07:02:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:06.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:06.128622+0000) 2022-09-10T07:02:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:06.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:06.761083+0000) 2022-09-10T07:02:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:07.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:07.128849+0000) 2022-09-10T07:02:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:07 smithi104 ceph-mon[97044]: pgmap v4895: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:02:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:07.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:07.761293+0000) 2022-09-10T07:02:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:07 smithi110 ceph-mon[97633]: pgmap v4895: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:02:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:08.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:08.129031+0000) 2022-09-10T07:02:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:08 smithi104 ceph-mon[97044]: osdmap e388: 12 total, 12 up, 12 in 2022-09-10T07:02:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:08.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:08.761543+0000) 2022-09-10T07:02:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:08 smithi110 ceph-mon[97633]: osdmap e388: 12 total, 12 up, 12 in 2022-09-10T07:02:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:09.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:09.129258+0000) 2022-09-10T07:02:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:09 smithi104 ceph-mon[97044]: pgmap v4897: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:02:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:09.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:09.761742+0000) 2022-09-10T07:02:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:09 smithi110 ceph-mon[97633]: pgmap v4897: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:02:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:10.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:10.129393+0000) 2022-09-10T07:02:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:10.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:10.761991+0000) 2022-09-10T07:02:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:11.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:11.129570+0000) 2022-09-10T07:02:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:11 smithi104 ceph-mon[97044]: pgmap v4898: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:02:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:11.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:11.762225+0000) 2022-09-10T07:02:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:11 smithi110 ceph-mon[97633]: pgmap v4898: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:02:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:12.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:12.129843+0000) 2022-09-10T07:02:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:12.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:12.762434+0000) 2022-09-10T07:02:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:13.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:13.129987+0000) 2022-09-10T07:02:14.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:13 smithi104 ceph-mon[97044]: pgmap v4899: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:02:14.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:13.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:13.762567+0000) 2022-09-10T07:02:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:13 smithi110 ceph-mon[97633]: pgmap v4899: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:02:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:14.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:14.130176+0000) 2022-09-10T07:02:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:14.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:14.762784+0000) 2022-09-10T07:02:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:15.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:15.130362+0000) 2022-09-10T07:02:15.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:02:15.142+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:02:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:15 smithi110 ceph-mon[97633]: osdmap e389: 12 total, 12 up, 12 in 2022-09-10T07:02:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:15 smithi110 ceph-mon[97633]: pgmap v4901: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:02:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:15 smithi104 ceph-mon[97044]: osdmap e389: 12 total, 12 up, 12 in 2022-09-10T07:02:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:15 smithi104 ceph-mon[97044]: pgmap v4901: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:02:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:15.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:15.763017+0000) 2022-09-10T07:02:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:16.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:16.130553+0000) 2022-09-10T07:02:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:16.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:16.763204+0000) 2022-09-10T07:02:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:17.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:17.130750+0000) 2022-09-10T07:02:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:17 smithi104 ceph-mon[97044]: pgmap v4902: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:02:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:17.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:17.763430+0000) 2022-09-10T07:02:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:17 smithi110 ceph-mon[97633]: pgmap v4902: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:02:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:18.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:18.130976+0000) 2022-09-10T07:02:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:18.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:18.763678+0000) 2022-09-10T07:02:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:19.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:19.131224+0000) 2022-09-10T07:02:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:19 smithi104 ceph-mon[97044]: pgmap v4903: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:02:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:19.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:19.763913+0000) 2022-09-10T07:02:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:19 smithi110 ceph-mon[97633]: pgmap v4903: 129 pgs: 129 active+clean; 8.2 MiB data, 573 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:02:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:20.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:20.131414+0000) 2022-09-10T07:02:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:20.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:20.764174+0000) 2022-09-10T07:02:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:21.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:21.131573+0000) 2022-09-10T07:02:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:21 smithi104 ceph-mon[97044]: pgmap v4904: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:02:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:21.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:21.764343+0000) 2022-09-10T07:02:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:21 smithi110 ceph-mon[97633]: pgmap v4904: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:02:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:22.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:22.131781+0000) 2022-09-10T07:02:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:22.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:22.764521+0000) 2022-09-10T07:02:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:23.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:23.132010+0000) 2022-09-10T07:02:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:23 smithi104 ceph-mon[97044]: pgmap v4905: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:02:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:23.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:23.764740+0000) 2022-09-10T07:02:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:23 smithi110 ceph-mon[97633]: pgmap v4905: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:02:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:24.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:24.132266+0000) 2022-09-10T07:02:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:24.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:24.764993+0000) 2022-09-10T07:02:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:25.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:25.132500+0000) 2022-09-10T07:02:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:25 smithi110 ceph-mon[97633]: pgmap v4906: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T07:02:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:25 smithi104 ceph-mon[97044]: pgmap v4906: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 12 op/s 2022-09-10T07:02:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:25.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:25.765172+0000) 2022-09-10T07:02:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:26.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:26.132621+0000) 2022-09-10T07:02:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:26.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:26.765396+0000) 2022-09-10T07:02:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:27.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:27.132844+0000) 2022-09-10T07:02:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:27 smithi104 ceph-mon[97044]: pgmap v4907: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:27.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:27.765614+0000) 2022-09-10T07:02:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:27 smithi110 ceph-mon[97633]: pgmap v4907: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:28.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:28.133138+0000) 2022-09-10T07:02:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:28.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:28.765846+0000) 2022-09-10T07:02:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:29.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:29.133277+0000) 2022-09-10T07:02:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:29 smithi104 ceph-mon[97044]: pgmap v4908: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:29.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:29.766079+0000) 2022-09-10T07:02:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:29 smithi110 ceph-mon[97633]: pgmap v4908: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:30.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:30.133493+0000) 2022-09-10T07:02:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:02:30.147+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:02:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:30.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:30.766343+0000) 2022-09-10T07:02:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:31.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:31.133727+0000) 2022-09-10T07:02:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:31 smithi104 ceph-mon[97044]: pgmap v4909: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:31.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:31.766579+0000) 2022-09-10T07:02:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:31 smithi110 ceph-mon[97633]: pgmap v4909: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:32.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:32.133967+0000) 2022-09-10T07:02:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:32.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:32.766806+0000) 2022-09-10T07:02:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:33.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:33.134178+0000) 2022-09-10T07:02:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:33 smithi104 ceph-mon[97044]: pgmap v4910: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:33.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:33.767031+0000) 2022-09-10T07:02:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:33 smithi110 ceph-mon[97633]: pgmap v4910: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:34.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:34.134309+0000) 2022-09-10T07:02:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:34.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:34.767293+0000) 2022-09-10T07:02:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:35.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:35.134546+0000) 2022-09-10T07:02:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:35 smithi110 ceph-mon[97633]: pgmap v4911: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:35 smithi104 ceph-mon[97044]: pgmap v4911: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:35.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:35.767540+0000) 2022-09-10T07:02:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:36.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:36.134818+0000) 2022-09-10T07:02:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:36.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:36.767797+0000) 2022-09-10T07:02:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:37.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:37.135011+0000) 2022-09-10T07:02:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:37 smithi104 ceph-mon[97044]: pgmap v4912: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:37.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:37.768016+0000) 2022-09-10T07:02:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:37 smithi110 ceph-mon[97633]: pgmap v4912: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:38.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:38.135208+0000) 2022-09-10T07:02:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:38.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:38.768234+0000) 2022-09-10T07:02:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:39.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:39.135338+0000) 2022-09-10T07:02:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:39 smithi104 ceph-mon[97044]: pgmap v4913: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:39.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:39.768468+0000) 2022-09-10T07:02:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:39 smithi110 ceph-mon[97633]: pgmap v4913: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:40.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:40.135530+0000) 2022-09-10T07:02:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:40.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:40.768713+0000) 2022-09-10T07:02:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:41.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:41.135669+0000) 2022-09-10T07:02:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:41 smithi104 ceph-mon[97044]: pgmap v4914: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:41.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:41.768923+0000) 2022-09-10T07:02:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:41 smithi110 ceph-mon[97633]: pgmap v4914: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:42.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:42.135873+0000) 2022-09-10T07:02:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:42.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:42.769162+0000) 2022-09-10T07:02:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:43.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:43.136027+0000) 2022-09-10T07:02:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:43 smithi104 ceph-mon[97044]: pgmap v4915: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:43.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:43.769376+0000) 2022-09-10T07:02:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:43 smithi110 ceph-mon[97633]: pgmap v4915: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:44.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:44.136259+0000) 2022-09-10T07:02:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:02:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:02:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:02:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:02:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:44.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:44.769565+0000) 2022-09-10T07:02:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:02:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:02:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:02:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:02:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:45.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:45.136491+0000) 2022-09-10T07:02:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:02:45.158+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:02:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:45 smithi104 ceph-mon[97044]: pgmap v4916: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:45.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:45.769778+0000) 2022-09-10T07:02:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:45 smithi110 ceph-mon[97633]: pgmap v4916: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:46.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:46.136748+0000) 2022-09-10T07:02:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:46.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:46.770009+0000) 2022-09-10T07:02:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:47.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:47.136934+0000) 2022-09-10T07:02:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:47 smithi104 ceph-mon[97044]: pgmap v4917: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:47.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:47.770225+0000) 2022-09-10T07:02:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:47 smithi110 ceph-mon[97633]: pgmap v4917: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:48.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:48.137208+0000) 2022-09-10T07:02:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:48.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:48.770498+0000) 2022-09-10T07:02:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:49.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:49.137464+0000) 2022-09-10T07:02:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:49 smithi104 ceph-mon[97044]: pgmap v4918: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:49.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:49.770679+0000) 2022-09-10T07:02:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:49 smithi110 ceph-mon[97633]: pgmap v4918: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:50.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:50.137695+0000) 2022-09-10T07:02:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:50.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:50.770933+0000) 2022-09-10T07:02:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:51.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:51.137969+0000) 2022-09-10T07:02:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:51 smithi104 ceph-mon[97044]: pgmap v4919: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:51.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:51.771147+0000) 2022-09-10T07:02:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:51 smithi110 ceph-mon[97633]: pgmap v4919: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:52.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:52.138192+0000) 2022-09-10T07:02:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:52.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:52.771329+0000) 2022-09-10T07:02:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:53.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:53.138381+0000) 2022-09-10T07:02:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:53 smithi104 ceph-mon[97044]: pgmap v4920: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:53.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:53.771523+0000) 2022-09-10T07:02:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:53 smithi110 ceph-mon[97633]: pgmap v4920: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:54.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:54.138607+0000) 2022-09-10T07:02:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:54.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:54.771726+0000) 2022-09-10T07:02:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:55.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:55.138786+0000) 2022-09-10T07:02:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:55 smithi110 ceph-mon[97633]: pgmap v4921: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:55 smithi104 ceph-mon[97044]: pgmap v4921: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:02:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:55.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:55.771881+0000) 2022-09-10T07:02:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:56.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:56.139017+0000) 2022-09-10T07:02:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:56.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:56.772053+0000) 2022-09-10T07:02:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:57.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:57.139250+0000) 2022-09-10T07:02:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:57 smithi104 ceph-mon[97044]: pgmap v4922: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:57.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:57.772242+0000) 2022-09-10T07:02:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:57 smithi110 ceph-mon[97633]: pgmap v4922: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:02:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:58.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:58.139514+0000) 2022-09-10T07:02:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:58.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:58.772458+0000) 2022-09-10T07:02:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:02:59.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:59.139684+0000) 2022-09-10T07:03:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:02:59 smithi104 ceph-mon[97044]: pgmap v4923: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:00.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:02:59.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:02:59.772622+0000) 2022-09-10T07:03:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:02:59 smithi110 ceph-mon[97633]: pgmap v4923: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:03:00.163+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:03:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:00.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:00.139932+0000) 2022-09-10T07:03:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:00.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:00.772794+0000) 2022-09-10T07:03:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:01.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:01.140167+0000) 2022-09-10T07:03:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:01 smithi104 ceph-mon[97044]: pgmap v4924: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:01.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:01.772979+0000) 2022-09-10T07:03:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:01 smithi110 ceph-mon[97633]: pgmap v4924: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:02.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:02.140360+0000) 2022-09-10T07:03:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:02.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:02.773159+0000) 2022-09-10T07:03:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:03.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:03.140588+0000) 2022-09-10T07:03:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:03 smithi104 ceph-mon[97044]: pgmap v4925: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:03.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:03.773381+0000) 2022-09-10T07:03:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:03 smithi110 ceph-mon[97633]: pgmap v4925: 129 pgs: 129 active+clean; 8.2 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:04.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:04.140837+0000) 2022-09-10T07:03:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:04.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:04.773588+0000) 2022-09-10T07:03:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:05.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:05.141118+0000) 2022-09-10T07:03:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:05 smithi110 ceph-mon[97633]: pgmap v4926: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:03:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:05 smithi104 ceph-mon[97044]: pgmap v4926: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:03:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:05.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:05.773795+0000) 2022-09-10T07:03:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:06.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:06.141362+0000) 2022-09-10T07:03:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:06.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:06.774049+0000) 2022-09-10T07:03:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:07.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:07.141562+0000) 2022-09-10T07:03:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:07 smithi104 ceph-mon[97044]: pgmap v4927: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:03:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:07.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:07.774393+0000) 2022-09-10T07:03:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:07 smithi110 ceph-mon[97633]: pgmap v4927: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:03:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:08.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:08.141799+0000) 2022-09-10T07:03:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:08 smithi104 ceph-mon[97044]: osdmap e390: 12 total, 12 up, 12 in 2022-09-10T07:03:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:08.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:08.774691+0000) 2022-09-10T07:03:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:08 smithi110 ceph-mon[97633]: osdmap e390: 12 total, 12 up, 12 in 2022-09-10T07:03:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:09.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:09.142024+0000) 2022-09-10T07:03:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:09 smithi104 ceph-mon[97044]: pgmap v4929: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:03:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:09.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:09.774907+0000) 2022-09-10T07:03:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:09 smithi110 ceph-mon[97633]: pgmap v4929: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:03:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:10.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:10.142253+0000) 2022-09-10T07:03:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:10.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:10.775229+0000) 2022-09-10T07:03:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:11.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:11.142477+0000) 2022-09-10T07:03:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:11 smithi104 ceph-mon[97044]: pgmap v4930: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:03:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:11.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:11.775429+0000) 2022-09-10T07:03:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:11 smithi110 ceph-mon[97633]: pgmap v4930: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:03:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:12.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:12.142730+0000) 2022-09-10T07:03:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:12.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:12.775636+0000) 2022-09-10T07:03:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:13.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:13.142980+0000) 2022-09-10T07:03:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:13 smithi104 ceph-mon[97044]: pgmap v4931: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:03:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:13.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:13.775773+0000) 2022-09-10T07:03:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:13 smithi110 ceph-mon[97633]: pgmap v4931: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:03:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:14.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:14.143201+0000) 2022-09-10T07:03:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:14.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:14.776017+0000) 2022-09-10T07:03:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:03:15.173+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:03:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:15.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:15.143349+0000) 2022-09-10T07:03:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:15 smithi110 ceph-mon[97633]: osdmap e391: 12 total, 12 up, 12 in 2022-09-10T07:03:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:15 smithi110 ceph-mon[97633]: pgmap v4933: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:03:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:15 smithi104 ceph-mon[97044]: osdmap e391: 12 total, 12 up, 12 in 2022-09-10T07:03:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:15 smithi104 ceph-mon[97044]: pgmap v4933: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:03:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:15.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:15.776243+0000) 2022-09-10T07:03:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:16.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:16.143563+0000) 2022-09-10T07:03:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:16.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:16.776421+0000) 2022-09-10T07:03:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:17.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:17.143793+0000) 2022-09-10T07:03:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:17 smithi104 ceph-mon[97044]: pgmap v4934: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:03:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:17.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:17.776653+0000) 2022-09-10T07:03:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:17 smithi110 ceph-mon[97633]: pgmap v4934: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:03:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:18.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:18.143983+0000) 2022-09-10T07:03:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:18.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:18.776913+0000) 2022-09-10T07:03:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:19.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:19.144165+0000) 2022-09-10T07:03:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:19 smithi104 ceph-mon[97044]: pgmap v4935: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:03:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:19.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:19.777118+0000) 2022-09-10T07:03:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:19 smithi110 ceph-mon[97633]: pgmap v4935: 129 pgs: 129 active+clean; 8.2 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:03:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:20.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:20.144372+0000) 2022-09-10T07:03:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:20.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:20.777348+0000) 2022-09-10T07:03:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:21.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:21.144586+0000) 2022-09-10T07:03:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:21 smithi104 ceph-mon[97044]: pgmap v4936: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:03:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:21.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:21.777479+0000) 2022-09-10T07:03:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:21 smithi110 ceph-mon[97633]: pgmap v4936: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:03:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:22.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:22.144832+0000) 2022-09-10T07:03:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:22.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:22.777724+0000) 2022-09-10T07:03:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:23.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:23.145049+0000) 2022-09-10T07:03:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:23 smithi104 ceph-mon[97044]: pgmap v4937: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:03:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:23.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:23.777980+0000) 2022-09-10T07:03:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:23 smithi110 ceph-mon[97633]: pgmap v4937: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:03:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:24.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:24.145270+0000) 2022-09-10T07:03:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:24.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:24.778231+0000) 2022-09-10T07:03:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:25.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:25.145510+0000) 2022-09-10T07:03:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:25 smithi110 ceph-mon[97633]: pgmap v4938: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T07:03:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:25 smithi104 ceph-mon[97044]: pgmap v4938: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T07:03:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:25.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:25.778423+0000) 2022-09-10T07:03:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:26.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:26.145754+0000) 2022-09-10T07:03:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:26.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:26.778565+0000) 2022-09-10T07:03:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:27.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:27.146035+0000) 2022-09-10T07:03:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:27 smithi104 ceph-mon[97044]: pgmap v4939: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:27.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:27.778784+0000) 2022-09-10T07:03:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:27 smithi110 ceph-mon[97633]: pgmap v4939: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:28.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:28.146276+0000) 2022-09-10T07:03:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:28.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:28.779047+0000) 2022-09-10T07:03:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:29.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:29.146492+0000) 2022-09-10T07:03:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:29 smithi104 ceph-mon[97044]: pgmap v4940: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:29.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:29.779313+0000) 2022-09-10T07:03:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:29 smithi110 ceph-mon[97633]: pgmap v4940: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:30.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:30.146733+0000) 2022-09-10T07:03:30.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:03:30.188+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:03:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:30.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:30.779616+0000) 2022-09-10T07:03:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:31.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:31.146984+0000) 2022-09-10T07:03:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:31 smithi104 ceph-mon[97044]: pgmap v4941: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:31.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:31.779826+0000) 2022-09-10T07:03:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:31 smithi110 ceph-mon[97633]: pgmap v4941: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:32.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:32.147206+0000) 2022-09-10T07:03:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:32.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:32.780041+0000) 2022-09-10T07:03:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:33.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:33.147440+0000) 2022-09-10T07:03:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:33 smithi104 ceph-mon[97044]: pgmap v4942: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:33.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:33.780305+0000) 2022-09-10T07:03:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:33 smithi110 ceph-mon[97633]: pgmap v4942: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:34.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:34.147692+0000) 2022-09-10T07:03:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:34.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:34.780573+0000) 2022-09-10T07:03:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:35.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:35.147903+0000) 2022-09-10T07:03:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:35 smithi110 ceph-mon[97633]: pgmap v4943: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:35 smithi104 ceph-mon[97044]: pgmap v4943: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:35.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:35.780833+0000) 2022-09-10T07:03:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:36.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:36.148121+0000) 2022-09-10T07:03:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:36.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:36.781112+0000) 2022-09-10T07:03:37.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:37.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:37.148365+0000) 2022-09-10T07:03:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:37 smithi104 ceph-mon[97044]: pgmap v4944: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:37.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:37.781344+0000) 2022-09-10T07:03:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:37 smithi110 ceph-mon[97633]: pgmap v4944: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:38.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:38.148640+0000) 2022-09-10T07:03:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:38.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:38.781513+0000) 2022-09-10T07:03:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:39.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:39.148871+0000) 2022-09-10T07:03:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:39 smithi104 ceph-mon[97044]: pgmap v4945: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:39.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:39.781762+0000) 2022-09-10T07:03:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:39 smithi110 ceph-mon[97633]: pgmap v4945: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:40.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:40.149124+0000) 2022-09-10T07:03:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:40.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:40.782036+0000) 2022-09-10T07:03:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:41.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:41.149415+0000) 2022-09-10T07:03:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:41 smithi104 ceph-mon[97044]: pgmap v4946: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:41.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:41.782265+0000) 2022-09-10T07:03:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:41 smithi110 ceph-mon[97633]: pgmap v4946: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:42.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:42.149633+0000) 2022-09-10T07:03:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:42.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:42.782511+0000) 2022-09-10T07:03:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:43.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:43.149850+0000) 2022-09-10T07:03:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:43 smithi104 ceph-mon[97044]: pgmap v4947: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:43.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:43.782714+0000) 2022-09-10T07:03:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:43 smithi110 ceph-mon[97633]: pgmap v4947: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:44.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:44.150119+0000) 2022-09-10T07:03:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:03:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:03:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:03:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:03:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:44.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:44.783002+0000) 2022-09-10T07:03:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:03:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:03:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:03:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:03:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:45.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:45.150397+0000) 2022-09-10T07:03:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:03:45.197+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:03:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:45 smithi104 ceph-mon[97044]: pgmap v4948: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:45.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:45.783274+0000) 2022-09-10T07:03:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:45 smithi110 ceph-mon[97633]: pgmap v4948: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:46.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:46.150638+0000) 2022-09-10T07:03:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:46.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:46.783488+0000) 2022-09-10T07:03:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:47.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:47.150894+0000) 2022-09-10T07:03:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:47 smithi104 ceph-mon[97044]: pgmap v4949: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:47.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:47.783769+0000) 2022-09-10T07:03:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:47 smithi110 ceph-mon[97633]: pgmap v4949: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:48.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:48.151078+0000) 2022-09-10T07:03:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:48.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:48.784002+0000) 2022-09-10T07:03:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:49.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:49.151312+0000) 2022-09-10T07:03:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:49 smithi104 ceph-mon[97044]: pgmap v4950: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:49.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:49.784264+0000) 2022-09-10T07:03:50.170 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:49 smithi110 ceph-mon[97633]: pgmap v4950: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:50.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:50.151501+0000) 2022-09-10T07:03:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:50.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:50.784470+0000) 2022-09-10T07:03:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:51.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:51.151657+0000) 2022-09-10T07:03:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:51 smithi104 ceph-mon[97044]: pgmap v4951: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:51.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:51.784632+0000) 2022-09-10T07:03:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:51 smithi110 ceph-mon[97633]: pgmap v4951: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:52.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:52.151896+0000) 2022-09-10T07:03:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:52.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:52.784858+0000) 2022-09-10T07:03:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:53.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:53.152090+0000) 2022-09-10T07:03:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:53 smithi104 ceph-mon[97044]: pgmap v4952: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:53.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:53.785146+0000) 2022-09-10T07:03:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:53 smithi110 ceph-mon[97633]: pgmap v4952: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:54.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:54.152341+0000) 2022-09-10T07:03:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:54.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:54.785313+0000) 2022-09-10T07:03:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:55.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:55.152590+0000) 2022-09-10T07:03:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:55 smithi104 ceph-mon[97044]: pgmap v4953: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:55 smithi110 ceph-mon[97633]: pgmap v4953: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:03:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:55.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:55.785474+0000) 2022-09-10T07:03:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:56.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:56.152844+0000) 2022-09-10T07:03:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:56.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:56.785609+0000) 2022-09-10T07:03:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:57.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:57.153168+0000) 2022-09-10T07:03:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:57 smithi104 ceph-mon[97044]: pgmap v4954: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:57.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:57.785747+0000) 2022-09-10T07:03:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:57 smithi110 ceph-mon[97633]: pgmap v4954: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:03:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:58.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:58.153381+0000) 2022-09-10T07:03:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:58.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:58.785904+0000) 2022-09-10T07:03:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:03:59.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:59.153606+0000) 2022-09-10T07:04:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:03:59 smithi104 ceph-mon[97044]: pgmap v4955: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:03:59.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:03:59.786157+0000) 2022-09-10T07:04:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:03:59 smithi110 ceph-mon[97633]: pgmap v4955: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:00.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:00.153819+0000) 2022-09-10T07:04:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:04:00.210+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:04:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:00.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:00.786306+0000) 2022-09-10T07:04:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:01.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:01.154070+0000) 2022-09-10T07:04:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:01 smithi104 ceph-mon[97044]: pgmap v4956: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:01.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:01.786498+0000) 2022-09-10T07:04:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:01 smithi110 ceph-mon[97633]: pgmap v4956: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:02.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:02.154325+0000) 2022-09-10T07:04:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:02.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:02.786659+0000) 2022-09-10T07:04:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:03.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:03.154539+0000) 2022-09-10T07:04:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:03 smithi104 ceph-mon[97044]: pgmap v4957: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:03.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:03.786868+0000) 2022-09-10T07:04:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:03 smithi110 ceph-mon[97633]: pgmap v4957: 129 pgs: 129 active+clean; 8.2 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:04.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:04.154812+0000) 2022-09-10T07:04:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:04.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:04.787052+0000) 2022-09-10T07:04:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:05.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:05.155024+0000) 2022-09-10T07:04:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:05 smithi104 ceph-mon[97044]: pgmap v4958: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:04:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:05 smithi110 ceph-mon[97633]: pgmap v4958: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:04:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:05.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:05.787261+0000) 2022-09-10T07:04:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:06.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:06.155230+0000) 2022-09-10T07:04:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:06.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:06.787472+0000) 2022-09-10T07:04:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:07.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:07.155498+0000) 2022-09-10T07:04:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:07 smithi104 ceph-mon[97044]: pgmap v4959: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:04:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:07.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:07.787646+0000) 2022-09-10T07:04:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:07 smithi110 ceph-mon[97633]: pgmap v4959: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:04:08.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:08.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:08.155721+0000) 2022-09-10T07:04:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:08 smithi104 ceph-mon[97044]: osdmap e392: 12 total, 12 up, 12 in 2022-09-10T07:04:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:08.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:08.787864+0000) 2022-09-10T07:04:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:08 smithi110 ceph-mon[97633]: osdmap e392: 12 total, 12 up, 12 in 2022-09-10T07:04:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:09.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:09.155985+0000) 2022-09-10T07:04:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:09 smithi104 ceph-mon[97044]: pgmap v4961: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:04:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:09.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:09.788001+0000) 2022-09-10T07:04:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:09 smithi110 ceph-mon[97633]: pgmap v4961: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:04:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:10.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:10.156203+0000) 2022-09-10T07:04:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:10.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:10.788202+0000) 2022-09-10T07:04:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:11.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:11.156384+0000) 2022-09-10T07:04:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:11 smithi104 ceph-mon[97044]: pgmap v4962: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:04:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:11.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:11.788368+0000) 2022-09-10T07:04:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:11 smithi110 ceph-mon[97633]: pgmap v4962: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:04:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:12.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:12.156575+0000) 2022-09-10T07:04:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:12.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:12.788541+0000) 2022-09-10T07:04:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:13.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:13.156838+0000) 2022-09-10T07:04:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:13 smithi104 ceph-mon[97044]: pgmap v4963: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:04:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:13.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:13.788665+0000) 2022-09-10T07:04:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:13 smithi110 ceph-mon[97633]: pgmap v4963: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:04:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:14.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:14.157064+0000) 2022-09-10T07:04:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:14.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:14.788870+0000) 2022-09-10T07:04:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:15.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:15.157380+0000) 2022-09-10T07:04:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:04:15.225+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:04:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:15 smithi104 ceph-mon[97044]: osdmap e393: 12 total, 12 up, 12 in 2022-09-10T07:04:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:15 smithi104 ceph-mon[97044]: pgmap v4965: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:04:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:15 smithi110 ceph-mon[97633]: osdmap e393: 12 total, 12 up, 12 in 2022-09-10T07:04:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:15 smithi110 ceph-mon[97633]: pgmap v4965: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:04:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:15.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:15.789123+0000) 2022-09-10T07:04:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:16.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:16.157657+0000) 2022-09-10T07:04:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:16.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:16.789360+0000) 2022-09-10T07:04:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:17.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:17.157857+0000) 2022-09-10T07:04:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:17 smithi104 ceph-mon[97044]: pgmap v4966: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:04:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:17.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:17.789581+0000) 2022-09-10T07:04:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:17 smithi110 ceph-mon[97633]: pgmap v4966: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:04:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:18.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:18.158127+0000) 2022-09-10T07:04:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:18.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:18.789756+0000) 2022-09-10T07:04:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:19.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:19.158278+0000) 2022-09-10T07:04:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:19 smithi104 ceph-mon[97044]: pgmap v4967: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:04:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:19.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:19.789943+0000) 2022-09-10T07:04:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:19 smithi110 ceph-mon[97633]: pgmap v4967: 129 pgs: 129 active+clean; 8.2 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:04:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:20.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:20.158492+0000) 2022-09-10T07:04:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:20.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:20.790160+0000) 2022-09-10T07:04:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:21.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:21.158733+0000) 2022-09-10T07:04:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:21 smithi104 ceph-mon[97044]: pgmap v4968: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:04:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:21.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:21.790337+0000) 2022-09-10T07:04:22.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:21 smithi110 ceph-mon[97633]: pgmap v4968: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:04:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:22.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:22.158974+0000) 2022-09-10T07:04:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:22.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:22.790460+0000) 2022-09-10T07:04:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:23.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:23.159265+0000) 2022-09-10T07:04:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:23 smithi104 ceph-mon[97044]: pgmap v4969: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:04:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:23.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:23.790605+0000) 2022-09-10T07:04:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:23 smithi110 ceph-mon[97633]: pgmap v4969: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:04:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:24.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:24.159507+0000) 2022-09-10T07:04:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:24.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:24.790873+0000) 2022-09-10T07:04:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:25.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:25.159703+0000) 2022-09-10T07:04:25.742 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:25 smithi104 ceph-mon[97044]: pgmap v4970: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:04:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:25 smithi110 ceph-mon[97633]: pgmap v4970: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:04:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:25.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:25.791032+0000) 2022-09-10T07:04:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:26.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:26.159916+0000) 2022-09-10T07:04:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:26.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:26.791256+0000) 2022-09-10T07:04:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:27.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:27.160114+0000) 2022-09-10T07:04:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:27 smithi104 ceph-mon[97044]: pgmap v4971: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:27.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:27.791455+0000) 2022-09-10T07:04:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:27 smithi110 ceph-mon[97633]: pgmap v4971: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:28.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:28.160326+0000) 2022-09-10T07:04:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:28.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:28.791607+0000) 2022-09-10T07:04:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:29.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:29.160547+0000) 2022-09-10T07:04:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:29 smithi104 ceph-mon[97044]: pgmap v4972: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:29.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:29.791862+0000) 2022-09-10T07:04:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:29 smithi110 ceph-mon[97633]: pgmap v4972: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:30.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:30.160710+0000) 2022-09-10T07:04:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:04:30.230+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:04:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:30.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:30.792037+0000) 2022-09-10T07:04:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:31.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:31.160945+0000) 2022-09-10T07:04:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:31 smithi104 ceph-mon[97044]: pgmap v4973: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:31.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:31.792268+0000) 2022-09-10T07:04:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:31 smithi110 ceph-mon[97633]: pgmap v4973: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:32.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:32.161153+0000) 2022-09-10T07:04:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:32.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:32.792558+0000) 2022-09-10T07:04:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:33.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:33.161412+0000) 2022-09-10T07:04:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:33 smithi104 ceph-mon[97044]: pgmap v4974: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:33.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:33.792859+0000) 2022-09-10T07:04:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:33 smithi110 ceph-mon[97633]: pgmap v4974: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:34.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:34.161659+0000) 2022-09-10T07:04:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:34.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:34.793091+0000) 2022-09-10T07:04:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:35.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:35.161926+0000) 2022-09-10T07:04:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:35 smithi104 ceph-mon[97044]: pgmap v4975: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:35 smithi110 ceph-mon[97633]: pgmap v4975: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:35.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:35.793366+0000) 2022-09-10T07:04:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:36.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:36.162188+0000) 2022-09-10T07:04:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:36.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:36.793568+0000) 2022-09-10T07:04:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:37.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:37.162381+0000) 2022-09-10T07:04:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:37 smithi104 ceph-mon[97044]: pgmap v4976: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:37.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:37.793689+0000) 2022-09-10T07:04:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:37 smithi110 ceph-mon[97633]: pgmap v4976: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:38.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:38.162651+0000) 2022-09-10T07:04:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:38.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:38.793913+0000) 2022-09-10T07:04:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:39.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:39.162885+0000) 2022-09-10T07:04:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:39 smithi104 ceph-mon[97044]: pgmap v4977: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:39.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:39.794191+0000) 2022-09-10T07:04:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:39 smithi110 ceph-mon[97633]: pgmap v4977: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:40.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:40.163063+0000) 2022-09-10T07:04:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:40.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:40.794381+0000) 2022-09-10T07:04:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:41.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:41.163299+0000) 2022-09-10T07:04:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:41 smithi104 ceph-mon[97044]: pgmap v4978: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:41.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:41.794550+0000) 2022-09-10T07:04:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:41 smithi110 ceph-mon[97633]: pgmap v4978: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:42.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:42.163527+0000) 2022-09-10T07:04:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:42.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:42.794745+0000) 2022-09-10T07:04:43.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:43.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:43.163752+0000) 2022-09-10T07:04:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:43 smithi104 ceph-mon[97044]: pgmap v4979: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:43.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:43.794958+0000) 2022-09-10T07:04:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:43 smithi110 ceph-mon[97633]: pgmap v4979: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:44.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:44.163919+0000) 2022-09-10T07:04:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:04:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:04:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:04:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:04:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:44.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:44.795158+0000) 2022-09-10T07:04:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:04:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:04:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:04:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:04:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:45.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:45.164156+0000) 2022-09-10T07:04:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:04:45.243+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:04:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:45.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:45.795343+0000) 2022-09-10T07:04:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:45 smithi110 ceph-mon[97633]: pgmap v4980: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:45 smithi104 ceph-mon[97044]: pgmap v4980: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:46.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:46.164413+0000) 2022-09-10T07:04:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:46.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:46.795530+0000) 2022-09-10T07:04:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:47.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:47.164576+0000) 2022-09-10T07:04:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:47.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:47.795669+0000) 2022-09-10T07:04:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:47 smithi110 ceph-mon[97633]: pgmap v4981: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:47 smithi104 ceph-mon[97044]: pgmap v4981: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:48.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:48.164870+0000) 2022-09-10T07:04:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:48.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:48.795889+0000) 2022-09-10T07:04:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:49.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:49.165155+0000) 2022-09-10T07:04:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:49.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:49.796176+0000) 2022-09-10T07:04:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:49 smithi110 ceph-mon[97633]: pgmap v4982: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:49 smithi104 ceph-mon[97044]: pgmap v4982: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:50.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:50.165413+0000) 2022-09-10T07:04:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:50.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:50.796371+0000) 2022-09-10T07:04:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:51.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:51.165654+0000) 2022-09-10T07:04:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:51.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:51.796603+0000) 2022-09-10T07:04:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:51 smithi110 ceph-mon[97633]: pgmap v4983: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:51 smithi104 ceph-mon[97044]: pgmap v4983: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:52.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:52.165906+0000) 2022-09-10T07:04:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:52.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:52.796824+0000) 2022-09-10T07:04:53.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:53.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:53.166175+0000) 2022-09-10T07:04:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:53.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:53.797026+0000) 2022-09-10T07:04:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:53 smithi110 ceph-mon[97633]: pgmap v4984: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:53 smithi104 ceph-mon[97044]: pgmap v4984: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:54.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:54.166407+0000) 2022-09-10T07:04:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:54.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:54.797206+0000) 2022-09-10T07:04:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:55.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:55.166566+0000) 2022-09-10T07:04:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:55 smithi104 ceph-mon[97044]: pgmap v4985: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:55 smithi110 ceph-mon[97633]: pgmap v4985: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:04:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:55.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:55.797401+0000) 2022-09-10T07:04:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:56.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:56.166796+0000) 2022-09-10T07:04:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:56.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:56.797526+0000) 2022-09-10T07:04:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:57.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:57.166985+0000) 2022-09-10T07:04:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:57 smithi104 ceph-mon[97044]: pgmap v4986: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:57.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:57.797779+0000) 2022-09-10T07:04:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:57 smithi110 ceph-mon[97633]: pgmap v4986: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:04:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:58.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:58.167227+0000) 2022-09-10T07:04:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:58.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:58.798020+0000) 2022-09-10T07:04:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:04:59.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:59.167420+0000) 2022-09-10T07:05:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:04:59 smithi104 ceph-mon[97044]: pgmap v4987: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:04:59.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:04:59.798190+0000) 2022-09-10T07:05:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:04:59 smithi110 ceph-mon[97633]: pgmap v4987: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:00.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:00.167619+0000) 2022-09-10T07:05:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:05:00.252+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:05:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:00.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:00.798365+0000) 2022-09-10T07:05:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:01.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:01.167942+0000) 2022-09-10T07:05:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:01 smithi104 ceph-mon[97044]: pgmap v4988: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:01.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:01.798490+0000) 2022-09-10T07:05:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:01 smithi110 ceph-mon[97633]: pgmap v4988: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:02.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:02.168184+0000) 2022-09-10T07:05:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:02.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:02.798709+0000) 2022-09-10T07:05:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:03.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:03.168351+0000) 2022-09-10T07:05:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:03.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:03.798993+0000) 2022-09-10T07:05:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:03 smithi110 ceph-mon[97633]: pgmap v4989: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:03 smithi104 ceph-mon[97044]: pgmap v4989: 129 pgs: 129 active+clean; 8.2 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:04.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:04.168489+0000) 2022-09-10T07:05:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:04.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:04.799190+0000) 2022-09-10T07:05:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:05.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:05.168755+0000) 2022-09-10T07:05:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:05 smithi104 ceph-mon[97044]: pgmap v4990: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T07:05:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:05 smithi110 ceph-mon[97633]: pgmap v4990: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T07:05:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:05.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:05.799356+0000) 2022-09-10T07:05:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:06.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:06.169002+0000) 2022-09-10T07:05:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:06.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:06.799623+0000) 2022-09-10T07:05:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:07.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:07.169242+0000) 2022-09-10T07:05:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:07 smithi104 ceph-mon[97044]: pgmap v4991: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T07:05:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:07.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:07.799875+0000) 2022-09-10T07:05:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:07 smithi110 ceph-mon[97633]: pgmap v4991: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T07:05:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:08.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:08.169443+0000) 2022-09-10T07:05:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:08.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:08.800124+0000) 2022-09-10T07:05:09.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:09.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:09.169679+0000) 2022-09-10T07:05:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:09 smithi104 ceph-mon[97044]: pgmap v4992: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T07:05:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:09.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:09.800353+0000) 2022-09-10T07:05:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:09 smithi110 ceph-mon[97633]: pgmap v4992: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2022-09-10T07:05:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:10.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:10.169948+0000) 2022-09-10T07:05:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:10.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:10.800556+0000) 2022-09-10T07:05:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:11.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:11.170199+0000) 2022-09-10T07:05:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:11.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:11.800810+0000) 2022-09-10T07:05:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:11 smithi110 ceph-mon[97633]: pgmap v4993: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T07:05:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:11 smithi104 ceph-mon[97044]: pgmap v4993: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T07:05:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:12.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:12.170388+0000) 2022-09-10T07:05:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:12.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:12.801041+0000) 2022-09-10T07:05:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:13.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:13.170619+0000) 2022-09-10T07:05:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:13.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:13.801330+0000) 2022-09-10T07:05:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:13 smithi110 ceph-mon[97633]: pgmap v4994: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T07:05:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:13 smithi104 ceph-mon[97044]: pgmap v4994: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T07:05:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:14.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:14.170874+0000) 2022-09-10T07:05:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:14.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:14.801554+0000) 2022-09-10T07:05:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:15.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:15.171114+0000) 2022-09-10T07:05:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:05:15.259+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:05:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:15 smithi104 ceph-mon[97044]: pgmap v4995: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T07:05:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:15 smithi110 ceph-mon[97633]: pgmap v4995: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 25 op/s 2022-09-10T07:05:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:15.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:15.801773+0000) 2022-09-10T07:05:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:16.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:16.171363+0000) 2022-09-10T07:05:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:16.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:16.802017+0000) 2022-09-10T07:05:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:17.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:17.171607+0000) 2022-09-10T07:05:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:17 smithi104 ceph-mon[97044]: pgmap v4996: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T07:05:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:17.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:17.802269+0000) 2022-09-10T07:05:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:17 smithi110 ceph-mon[97633]: pgmap v4996: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T07:05:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:18.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:18.171837+0000) 2022-09-10T07:05:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:18.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:18.802512+0000) 2022-09-10T07:05:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:19.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:19.172055+0000) 2022-09-10T07:05:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:19.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:19.802661+0000) 2022-09-10T07:05:20.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:19 smithi110 ceph-mon[97633]: pgmap v4997: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T07:05:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:19 smithi104 ceph-mon[97044]: pgmap v4997: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T07:05:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:20.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:20.172266+0000) 2022-09-10T07:05:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:20.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:20.802921+0000) 2022-09-10T07:05:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:21.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:21.172508+0000) 2022-09-10T07:05:22.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:21.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:21.803173+0000) 2022-09-10T07:05:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:21 smithi110 ceph-mon[97633]: pgmap v4998: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T07:05:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:21 smithi104 ceph-mon[97044]: pgmap v4998: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T07:05:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:22.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:22.172763+0000) 2022-09-10T07:05:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:22.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:22.803348+0000) 2022-09-10T07:05:23.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:23.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:23.172988+0000) 2022-09-10T07:05:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:23 smithi110 ceph-mon[97633]: pgmap v4999: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:24.174 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:23.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:23.803588+0000) 2022-09-10T07:05:24.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:23 smithi104 ceph-mon[97044]: pgmap v4999: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:24.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:24.173238+0000) 2022-09-10T07:05:25.175 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:24.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:24.803767+0000) 2022-09-10T07:05:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:25.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:25.173432+0000) 2022-09-10T07:05:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:25 smithi104 ceph-mon[97044]: pgmap v5000: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:25 smithi110 ceph-mon[97633]: pgmap v5000: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:26.174 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:25.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:25.803990+0000) 2022-09-10T07:05:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:26.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:26.173579+0000) 2022-09-10T07:05:27.174 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:26.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:26.804229+0000) 2022-09-10T07:05:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:27.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:27.173694+0000) 2022-09-10T07:05:28.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:27 smithi110 ceph-mon[97633]: pgmap v5001: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:28.177 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:27.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:27.804432+0000) 2022-09-10T07:05:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:27 smithi104 ceph-mon[97044]: pgmap v5001: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:28.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:28.173925+0000) 2022-09-10T07:05:29.175 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:28.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:28.804631+0000) 2022-09-10T07:05:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:29.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:29.174159+0000) 2022-09-10T07:05:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:29 smithi110 ceph-mon[97633]: pgmap v5002: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:30.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:29.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:29.804814+0000) 2022-09-10T07:05:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:29 smithi104 ceph-mon[97044]: pgmap v5002: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:30.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:30.174367+0000) 2022-09-10T07:05:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:05:30.264+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:05:31.175 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:30.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:30.805041+0000) 2022-09-10T07:05:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:31.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:31.174577+0000) 2022-09-10T07:05:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:31 smithi110 ceph-mon[97633]: pgmap v5003: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:32.175 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:31.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:31.805291+0000) 2022-09-10T07:05:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:31 smithi104 ceph-mon[97044]: pgmap v5003: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:32.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:32.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:32.174703+0000) 2022-09-10T07:05:33.176 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:32.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:32.805488+0000) 2022-09-10T07:05:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:33.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:33.174943+0000) 2022-09-10T07:05:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:33 smithi110 ceph-mon[97633]: pgmap v5004: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:34.176 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:33.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:33.805743+0000) 2022-09-10T07:05:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:33 smithi104 ceph-mon[97044]: pgmap v5004: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:34.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:34.175235+0000) 2022-09-10T07:05:35.176 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:34.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:34.805931+0000) 2022-09-10T07:05:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:35.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:35.175478+0000) 2022-09-10T07:05:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:35 smithi104 ceph-mon[97044]: pgmap v5005: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:35 smithi110 ceph-mon[97633]: pgmap v5005: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:36.176 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:35.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:35.806085+0000) 2022-09-10T07:05:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:36.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:36.175699+0000) 2022-09-10T07:05:37.176 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:36.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:36.806255+0000) 2022-09-10T07:05:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:37.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:37.175955+0000) 2022-09-10T07:05:38.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:37.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:37.806427+0000) 2022-09-10T07:05:38.180 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:37 smithi110 ceph-mon[97633]: pgmap v5006: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:37 smithi104 ceph-mon[97044]: pgmap v5006: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:38.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:38.176180+0000) 2022-09-10T07:05:39.177 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:38.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:38.806575+0000) 2022-09-10T07:05:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:39.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:39.176346+0000) 2022-09-10T07:05:40.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:39 smithi110 ceph-mon[97633]: pgmap v5007: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:40.177 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:39.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:39.806763+0000) 2022-09-10T07:05:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:39 smithi104 ceph-mon[97044]: pgmap v5007: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:40.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:40.176586+0000) 2022-09-10T07:05:41.177 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:40.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:40.807016+0000) 2022-09-10T07:05:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:41.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:41.176806+0000) 2022-09-10T07:05:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:41 smithi110 ceph-mon[97633]: pgmap v5008: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:42.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:41.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:41.807288+0000) 2022-09-10T07:05:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:41 smithi104 ceph-mon[97044]: pgmap v5008: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:42.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:42.177064+0000) 2022-09-10T07:05:43.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:42.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:42.807526+0000) 2022-09-10T07:05:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:43.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:43.177250+0000) 2022-09-10T07:05:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:43 smithi110 ceph-mon[97633]: pgmap v5009: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:44.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:43.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:43.807763+0000) 2022-09-10T07:05:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:43 smithi104 ceph-mon[97044]: pgmap v5009: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:44.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:44.177507+0000) 2022-09-10T07:05:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:05:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:05:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:05:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:05:45.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:44.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:44.807992+0000) 2022-09-10T07:05:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:05:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:05:45.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:05:45.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:05:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:45.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:45.177743+0000) 2022-09-10T07:05:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:05:45.277+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:05:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:45 smithi110 ceph-mon[97633]: pgmap v5010: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:46.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:45.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:45.808254+0000) 2022-09-10T07:05:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:45 smithi104 ceph-mon[97044]: pgmap v5010: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:46.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:46.177985+0000) 2022-09-10T07:05:47.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:46.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:46.808489+0000) 2022-09-10T07:05:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:47.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:47.178179+0000) 2022-09-10T07:05:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:47 smithi110 ceph-mon[97633]: pgmap v5011: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:48.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:47.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:47.808824+0000) 2022-09-10T07:05:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:47 smithi104 ceph-mon[97044]: pgmap v5011: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:48.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:48.178370+0000) 2022-09-10T07:05:49.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:48.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:48.809090+0000) 2022-09-10T07:05:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:49.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:49.178628+0000) 2022-09-10T07:05:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:49 smithi110 ceph-mon[97633]: pgmap v5012: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:50.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:49.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:49.809342+0000) 2022-09-10T07:05:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:49 smithi104 ceph-mon[97044]: pgmap v5012: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:50.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:50.178789+0000) 2022-09-10T07:05:51.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:50.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:50.809557+0000) 2022-09-10T07:05:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:51.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:51.179043+0000) 2022-09-10T07:05:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:51 smithi110 ceph-mon[97633]: pgmap v5013: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:52.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:51.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:51.809813+0000) 2022-09-10T07:05:52.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:51 smithi104 ceph-mon[97044]: pgmap v5013: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:52.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:52.179336+0000) 2022-09-10T07:05:53.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:52.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:52.810024+0000) 2022-09-10T07:05:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:53.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:53.179603+0000) 2022-09-10T07:05:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:53 smithi110 ceph-mon[97633]: pgmap v5014: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:54.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:53.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:53.810192+0000) 2022-09-10T07:05:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:53 smithi104 ceph-mon[97044]: pgmap v5014: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:54.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:54.179827+0000) 2022-09-10T07:05:55.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:54.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:54.810355+0000) 2022-09-10T07:05:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:55.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:55.180075+0000) 2022-09-10T07:05:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:55 smithi104 ceph-mon[97044]: pgmap v5015: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:55 smithi110 ceph-mon[97633]: pgmap v5015: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:05:56.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:55.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:55.810534+0000) 2022-09-10T07:05:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:56.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:56.180366+0000) 2022-09-10T07:05:57.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:56.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:56.810690+0000) 2022-09-10T07:05:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:57.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:57.180586+0000) 2022-09-10T07:05:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:57 smithi110 ceph-mon[97633]: pgmap v5016: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:58.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:57.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:57.810942+0000) 2022-09-10T07:05:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:57 smithi104 ceph-mon[97044]: pgmap v5016: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:05:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:58.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:58.180809+0000) 2022-09-10T07:05:59.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:58.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:58.811140+0000) 2022-09-10T07:05:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:05:59.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:59.181062+0000) 2022-09-10T07:06:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:05:59 smithi110 ceph-mon[97633]: pgmap v5017: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:00.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:05:59.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:05:59.811366+0000) 2022-09-10T07:06:00.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:05:59 smithi104 ceph-mon[97044]: pgmap v5017: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:00.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:00.181275+0000) 2022-09-10T07:06:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:06:00.292+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:06:01.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:00.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:00.811537+0000) 2022-09-10T07:06:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:01.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:01.181526+0000) 2022-09-10T07:06:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:01 smithi110 ceph-mon[97633]: pgmap v5018: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:02.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:01.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:01.811755+0000) 2022-09-10T07:06:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:01 smithi104 ceph-mon[97044]: pgmap v5018: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:02.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:02.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:02.181748+0000) 2022-09-10T07:06:03.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:02.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:02.811979+0000) 2022-09-10T07:06:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:03.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:03.181925+0000) 2022-09-10T07:06:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:03 smithi110 ceph-mon[97633]: pgmap v5019: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:04.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:03.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:03.812176+0000) 2022-09-10T07:06:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:03 smithi104 ceph-mon[97044]: pgmap v5019: 129 pgs: 129 active+clean; 8.2 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:04.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:04.182208+0000) 2022-09-10T07:06:05.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:04.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:04.812376+0000) 2022-09-10T07:06:05.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:05.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:05.182499+0000) 2022-09-10T07:06:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:05 smithi104 ceph-mon[97044]: pgmap v5020: 129 pgs: 129 active+clean; 8.2 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:06:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:05 smithi110 ceph-mon[97633]: pgmap v5020: 129 pgs: 129 active+clean; 8.2 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:06:06.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:05.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:05.812629+0000) 2022-09-10T07:06:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:06.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:06.182784+0000) 2022-09-10T07:06:07.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:06.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:06.812804+0000) 2022-09-10T07:06:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:07.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:07.182974+0000) 2022-09-10T07:06:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:07 smithi110 ceph-mon[97633]: pgmap v5021: 129 pgs: 129 active+clean; 8.2 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:06:08.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:07.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:07.812984+0000) 2022-09-10T07:06:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:07 smithi104 ceph-mon[97044]: pgmap v5021: 129 pgs: 129 active+clean; 8.2 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:06:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:08.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:08.183192+0000) 2022-09-10T07:06:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:08 smithi110 ceph-mon[97633]: osdmap e394: 12 total, 12 up, 12 in 2022-09-10T07:06:09.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:08.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:08.813195+0000) 2022-09-10T07:06:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:08 smithi104 ceph-mon[97044]: osdmap e394: 12 total, 12 up, 12 in 2022-09-10T07:06:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:09.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:09.183436+0000) 2022-09-10T07:06:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:09 smithi110 ceph-mon[97633]: pgmap v5023: 129 pgs: 129 active+clean; 8.2 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:06:10.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:09.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:09.813431+0000) 2022-09-10T07:06:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:09 smithi104 ceph-mon[97044]: pgmap v5023: 129 pgs: 129 active+clean; 8.2 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:06:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:10.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:10.183723+0000) 2022-09-10T07:06:11.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:10.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:10.813663+0000) 2022-09-10T07:06:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:11.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:11.183998+0000) 2022-09-10T07:06:12.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:11 smithi110 ceph-mon[97633]: pgmap v5024: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:06:12.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:11.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:11.813921+0000) 2022-09-10T07:06:12.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:11 smithi104 ceph-mon[97044]: pgmap v5024: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:06:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:12.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:12.184217+0000) 2022-09-10T07:06:13.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:12.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:12.814186+0000) 2022-09-10T07:06:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:13.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:13.184479+0000) 2022-09-10T07:06:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:13 smithi110 ceph-mon[97633]: pgmap v5025: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:06:14.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:13.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:13.814363+0000) 2022-09-10T07:06:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:13 smithi104 ceph-mon[97044]: pgmap v5025: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:06:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:14.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:14.184699+0000) 2022-09-10T07:06:15.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:14.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:14.814498+0000) 2022-09-10T07:06:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:15.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:15.184886+0000) 2022-09-10T07:06:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:06:15.308+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:06:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:15 smithi104 ceph-mon[97044]: osdmap e395: 12 total, 12 up, 12 in 2022-09-10T07:06:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:15 smithi104 ceph-mon[97044]: pgmap v5027: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:06:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:15 smithi110 ceph-mon[97633]: osdmap e395: 12 total, 12 up, 12 in 2022-09-10T07:06:15.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:15 smithi110 ceph-mon[97633]: pgmap v5027: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:06:16.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:15.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:15.814739+0000) 2022-09-10T07:06:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:16.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:16.185180+0000) 2022-09-10T07:06:17.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:16.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:16.814961+0000) 2022-09-10T07:06:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:17.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:17.185400+0000) 2022-09-10T07:06:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:17 smithi110 ceph-mon[97633]: pgmap v5028: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:06:18.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:17.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:17.815224+0000) 2022-09-10T07:06:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:17 smithi104 ceph-mon[97044]: pgmap v5028: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:06:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:18.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:18.185667+0000) 2022-09-10T07:06:19.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:18.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:18.815374+0000) 2022-09-10T07:06:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:19.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:19.185859+0000) 2022-09-10T07:06:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:19 smithi110 ceph-mon[97633]: pgmap v5029: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:06:20.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:19.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:19.815641+0000) 2022-09-10T07:06:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:19 smithi104 ceph-mon[97044]: pgmap v5029: 129 pgs: 129 active+clean; 8.2 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:06:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:20.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:20.186059+0000) 2022-09-10T07:06:21.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:20.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:20.815843+0000) 2022-09-10T07:06:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:21.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:21.186315+0000) 2022-09-10T07:06:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:21 smithi110 ceph-mon[97633]: pgmap v5030: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:06:22.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:21.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:21.816022+0000) 2022-09-10T07:06:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:21 smithi104 ceph-mon[97044]: pgmap v5030: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:06:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:22.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:22.186475+0000) 2022-09-10T07:06:23.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:22.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:22.816248+0000) 2022-09-10T07:06:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:23.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:23.186671+0000) 2022-09-10T07:06:24.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:23 smithi110 ceph-mon[97633]: pgmap v5031: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:06:24.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:23.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:23.816505+0000) 2022-09-10T07:06:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:23 smithi104 ceph-mon[97044]: pgmap v5031: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:06:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:24.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:24.186920+0000) 2022-09-10T07:06:25.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:24.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:24.816706+0000) 2022-09-10T07:06:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:25.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:25.187134+0000) 2022-09-10T07:06:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:25 smithi104 ceph-mon[97044]: pgmap v5032: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:06:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:25 smithi110 ceph-mon[97633]: pgmap v5032: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:06:26.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:25.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:25.816884+0000) 2022-09-10T07:06:26.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:26.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:26.187396+0000) 2022-09-10T07:06:27.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:26.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:26.817172+0000) 2022-09-10T07:06:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:27.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:27.187671+0000) 2022-09-10T07:06:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:27 smithi110 ceph-mon[97633]: pgmap v5033: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:28.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:27.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:27.817403+0000) 2022-09-10T07:06:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:27 smithi104 ceph-mon[97044]: pgmap v5033: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:28.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:28.187953+0000) 2022-09-10T07:06:29.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:28.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:28.817568+0000) 2022-09-10T07:06:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:29.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:29.188222+0000) 2022-09-10T07:06:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:29 smithi110 ceph-mon[97633]: pgmap v5034: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:30.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:29.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:29.817818+0000) 2022-09-10T07:06:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:29 smithi104 ceph-mon[97044]: pgmap v5034: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:30.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:30.188533+0000) 2022-09-10T07:06:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:06:30.323+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:06:31.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:30.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:30.818118+0000) 2022-09-10T07:06:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:31.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:31.188732+0000) 2022-09-10T07:06:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:31 smithi110 ceph-mon[97633]: pgmap v5035: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:32.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:31.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:31.818378+0000) 2022-09-10T07:06:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:31 smithi104 ceph-mon[97044]: pgmap v5035: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:32.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:32.188971+0000) 2022-09-10T07:06:33.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:32.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:32.818579+0000) 2022-09-10T07:06:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:33.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:33.189244+0000) 2022-09-10T07:06:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:33 smithi110 ceph-mon[97633]: pgmap v5036: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:34.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:33.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:33.818795+0000) 2022-09-10T07:06:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:33 smithi104 ceph-mon[97044]: pgmap v5036: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:34.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:34.189500+0000) 2022-09-10T07:06:35.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:34.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:34.819044+0000) 2022-09-10T07:06:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:35.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:35.189778+0000) 2022-09-10T07:06:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:35 smithi104 ceph-mon[97044]: pgmap v5037: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:35 smithi110 ceph-mon[97633]: pgmap v5037: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:36.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:35.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:35.819270+0000) 2022-09-10T07:06:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:36.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:36.189993+0000) 2022-09-10T07:06:37.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:36.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:36.819540+0000) 2022-09-10T07:06:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:37.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:37.190254+0000) 2022-09-10T07:06:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:37 smithi110 ceph-mon[97633]: pgmap v5038: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:38.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:37.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:37.819719+0000) 2022-09-10T07:06:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:37 smithi104 ceph-mon[97044]: pgmap v5038: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:38.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:38.190518+0000) 2022-09-10T07:06:39.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:38.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:38.819929+0000) 2022-09-10T07:06:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:39.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:39.190781+0000) 2022-09-10T07:06:40.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:39 smithi110 ceph-mon[97633]: pgmap v5039: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:40.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:39.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:39.820160+0000) 2022-09-10T07:06:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:39 smithi104 ceph-mon[97044]: pgmap v5039: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:40.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:40.190952+0000) 2022-09-10T07:06:41.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:40.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:40.820331+0000) 2022-09-10T07:06:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:41.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:41.191265+0000) 2022-09-10T07:06:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:41 smithi110 ceph-mon[97633]: pgmap v5040: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:42.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:41.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:41.820525+0000) 2022-09-10T07:06:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:41 smithi104 ceph-mon[97044]: pgmap v5040: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:42.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:42.191497+0000) 2022-09-10T07:06:43.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:42.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:42.820781+0000) 2022-09-10T07:06:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:43.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:43.191672+0000) 2022-09-10T07:06:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:43 smithi110 ceph-mon[97633]: pgmap v5041: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:44.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:43.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:43.821001+0000) 2022-09-10T07:06:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:43 smithi104 ceph-mon[97044]: pgmap v5041: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:44.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:44.191908+0000) 2022-09-10T07:06:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:06:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:06:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:06:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:06:45.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:44.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:44.821235+0000) 2022-09-10T07:06:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:06:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:06:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:06:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:06:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:45.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:45.192189+0000) 2022-09-10T07:06:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:06:45.328+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:06:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:45 smithi110 ceph-mon[97633]: pgmap v5042: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:46.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:45.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:45.821454+0000) 2022-09-10T07:06:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:45 smithi104 ceph-mon[97044]: pgmap v5042: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:46.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:46.192450+0000) 2022-09-10T07:06:47.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:46.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:46.821670+0000) 2022-09-10T07:06:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:47.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:47.192687+0000) 2022-09-10T07:06:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:47 smithi110 ceph-mon[97633]: pgmap v5043: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:48.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:47.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:47.821896+0000) 2022-09-10T07:06:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:47 smithi104 ceph-mon[97044]: pgmap v5043: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:48.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:48.193017+0000) 2022-09-10T07:06:49.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:48.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:48.822159+0000) 2022-09-10T07:06:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:49.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:49.193281+0000) 2022-09-10T07:06:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:49 smithi110 ceph-mon[97633]: pgmap v5044: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:50.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:49.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:49.822342+0000) 2022-09-10T07:06:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:49 smithi104 ceph-mon[97044]: pgmap v5044: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:50.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:50.193478+0000) 2022-09-10T07:06:51.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:50.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:50.822575+0000) 2022-09-10T07:06:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:51.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:51.193694+0000) 2022-09-10T07:06:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:51 smithi110 ceph-mon[97633]: pgmap v5045: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:52.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:51.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:51.822845+0000) 2022-09-10T07:06:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:51 smithi104 ceph-mon[97044]: pgmap v5045: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:52.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:52.193885+0000) 2022-09-10T07:06:53.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:52.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:52.823042+0000) 2022-09-10T07:06:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:53.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:53.194125+0000) 2022-09-10T07:06:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:53 smithi110 ceph-mon[97633]: pgmap v5046: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:54.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:53.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:53.823324+0000) 2022-09-10T07:06:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:53 smithi104 ceph-mon[97044]: pgmap v5046: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:54.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:54.194336+0000) 2022-09-10T07:06:55.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:54.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:54.823554+0000) 2022-09-10T07:06:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:55.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:55.194567+0000) 2022-09-10T07:06:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:55 smithi104 ceph-mon[97044]: pgmap v5047: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:55 smithi110 ceph-mon[97633]: pgmap v5047: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:06:56.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:55.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:55.823691+0000) 2022-09-10T07:06:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:56.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:56.194729+0000) 2022-09-10T07:06:57.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:56.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:56.823829+0000) 2022-09-10T07:06:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:57.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:57.194985+0000) 2022-09-10T07:06:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:57 smithi110 ceph-mon[97633]: pgmap v5048: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:58.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:57.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:57.824035+0000) 2022-09-10T07:06:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:57 smithi104 ceph-mon[97044]: pgmap v5048: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:06:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:58.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:58.195262+0000) 2022-09-10T07:06:59.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:58.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:58.824250+0000) 2022-09-10T07:06:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:06:59.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:59.195396+0000) 2022-09-10T07:07:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:06:59 smithi110 ceph-mon[97633]: pgmap v5049: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:00.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:06:59.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:06:59.824482+0000) 2022-09-10T07:07:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:06:59 smithi104 ceph-mon[97044]: pgmap v5049: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:00.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:00.195636+0000) 2022-09-10T07:07:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:07:00.336+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:07:01.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:00.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:00.824629+0000) 2022-09-10T07:07:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:01.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:01.195804+0000) 2022-09-10T07:07:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:01 smithi110 ceph-mon[97633]: pgmap v5050: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:02.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:01.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:01.824855+0000) 2022-09-10T07:07:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:01 smithi104 ceph-mon[97044]: pgmap v5050: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:02.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:02.196006+0000) 2022-09-10T07:07:03.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:02.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:02.825158+0000) 2022-09-10T07:07:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:03.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:03.196213+0000) 2022-09-10T07:07:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:03 smithi110 ceph-mon[97633]: pgmap v5051: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:04.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:03.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:03.825368+0000) 2022-09-10T07:07:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:03 smithi104 ceph-mon[97044]: pgmap v5051: 129 pgs: 129 active+clean; 8.2 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:04.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:04.196479+0000) 2022-09-10T07:07:05.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:04.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:04.825633+0000) 2022-09-10T07:07:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:05.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:05.196686+0000) 2022-09-10T07:07:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:05 smithi104 ceph-mon[97044]: pgmap v5052: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:07:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:05 smithi110 ceph-mon[97633]: pgmap v5052: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:07:06.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:05.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:05.825856+0000) 2022-09-10T07:07:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:06.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:06.196936+0000) 2022-09-10T07:07:07.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:06.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:06.826048+0000) 2022-09-10T07:07:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:07.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:07.197184+0000) 2022-09-10T07:07:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:07 smithi110 ceph-mon[97633]: pgmap v5053: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:07:08.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:07.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:07.826233+0000) 2022-09-10T07:07:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:07 smithi104 ceph-mon[97044]: pgmap v5053: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:07:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:08.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:08.197463+0000) 2022-09-10T07:07:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:08 smithi110 ceph-mon[97633]: osdmap e396: 12 total, 12 up, 12 in 2022-09-10T07:07:09.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:08.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:08.826466+0000) 2022-09-10T07:07:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:08 smithi104 ceph-mon[97044]: osdmap e396: 12 total, 12 up, 12 in 2022-09-10T07:07:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:09.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:09.197691+0000) 2022-09-10T07:07:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:09 smithi110 ceph-mon[97633]: pgmap v5055: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:07:10.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:09.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:09.826618+0000) 2022-09-10T07:07:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:09 smithi104 ceph-mon[97044]: pgmap v5055: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:07:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:10.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:10.197914+0000) 2022-09-10T07:07:11.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:10.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:10.826786+0000) 2022-09-10T07:07:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:11.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:11.198165+0000) 2022-09-10T07:07:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:11 smithi110 ceph-mon[97633]: pgmap v5056: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:07:12.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:11.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:11.826988+0000) 2022-09-10T07:07:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:11 smithi104 ceph-mon[97044]: pgmap v5056: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:07:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:12.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:12.198407+0000) 2022-09-10T07:07:13.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:12.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:12.827203+0000) 2022-09-10T07:07:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:13.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:13.198652+0000) 2022-09-10T07:07:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:13 smithi110 ceph-mon[97633]: pgmap v5057: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:07:14.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:13.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:13.827410+0000) 2022-09-10T07:07:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:13 smithi104 ceph-mon[97044]: pgmap v5057: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:07:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:14.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:14.198895+0000) 2022-09-10T07:07:15.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:14.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:14.827605+0000) 2022-09-10T07:07:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:15.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:15.199081+0000) 2022-09-10T07:07:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:07:15.340+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:07:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:15 smithi104 ceph-mon[97044]: osdmap e397: 12 total, 12 up, 12 in 2022-09-10T07:07:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:15 smithi104 ceph-mon[97044]: pgmap v5059: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:07:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:15 smithi110 ceph-mon[97633]: osdmap e397: 12 total, 12 up, 12 in 2022-09-10T07:07:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:15 smithi110 ceph-mon[97633]: pgmap v5059: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:07:16.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:15.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:15.827796+0000) 2022-09-10T07:07:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:16.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:16.199321+0000) 2022-09-10T07:07:17.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:16.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:16.828015+0000) 2022-09-10T07:07:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:17.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:17.199460+0000) 2022-09-10T07:07:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:17 smithi110 ceph-mon[97633]: pgmap v5060: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:07:18.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:17.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:17.828276+0000) 2022-09-10T07:07:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:17 smithi104 ceph-mon[97044]: pgmap v5060: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:07:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:18.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:18.199639+0000) 2022-09-10T07:07:19.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:18.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:18.828460+0000) 2022-09-10T07:07:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:19.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:19.199767+0000) 2022-09-10T07:07:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:19 smithi110 ceph-mon[97633]: pgmap v5061: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:07:20.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:19.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:19.828628+0000) 2022-09-10T07:07:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:19 smithi104 ceph-mon[97044]: pgmap v5061: 129 pgs: 129 active+clean; 8.2 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:07:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:20.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:20.200000+0000) 2022-09-10T07:07:21.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:20.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:20.828830+0000) 2022-09-10T07:07:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:21.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:21.200162+0000) 2022-09-10T07:07:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:21 smithi110 ceph-mon[97633]: pgmap v5062: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:07:22.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:21.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:21.828962+0000) 2022-09-10T07:07:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:21 smithi104 ceph-mon[97044]: pgmap v5062: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:07:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:22.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:22.200353+0000) 2022-09-10T07:07:23.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:22.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:22.829179+0000) 2022-09-10T07:07:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:23.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:23.200513+0000) 2022-09-10T07:07:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:23 smithi110 ceph-mon[97633]: pgmap v5063: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:07:24.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:23.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:23.829423+0000) 2022-09-10T07:07:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:23 smithi104 ceph-mon[97044]: pgmap v5063: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:07:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:24.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:24.200799+0000) 2022-09-10T07:07:25.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:24.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:24.829620+0000) 2022-09-10T07:07:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:25.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:25.201043+0000) 2022-09-10T07:07:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:25 smithi104 ceph-mon[97044]: pgmap v5064: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:07:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:25 smithi110 ceph-mon[97633]: pgmap v5064: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:07:26.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:25.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:25.829779+0000) 2022-09-10T07:07:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:26.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:26.201312+0000) 2022-09-10T07:07:27.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:26.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:26.830010+0000) 2022-09-10T07:07:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:27.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:27.201606+0000) 2022-09-10T07:07:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:27 smithi110 ceph-mon[97633]: pgmap v5065: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:28.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:27.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:27.830217+0000) 2022-09-10T07:07:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:27 smithi104 ceph-mon[97044]: pgmap v5065: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:28.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:28.201820+0000) 2022-09-10T07:07:29.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:28.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:28.830431+0000) 2022-09-10T07:07:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:29.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:29.202026+0000) 2022-09-10T07:07:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:29 smithi110 ceph-mon[97633]: pgmap v5066: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:30.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:29.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:29.830610+0000) 2022-09-10T07:07:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:29 smithi104 ceph-mon[97044]: pgmap v5066: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:30.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:30.202223+0000) 2022-09-10T07:07:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:07:30.346+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:07:31.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:30.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:30.830868+0000) 2022-09-10T07:07:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:31.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:31.202406+0000) 2022-09-10T07:07:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:31 smithi110 ceph-mon[97633]: pgmap v5067: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:32.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:31.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:31.831065+0000) 2022-09-10T07:07:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:31 smithi104 ceph-mon[97044]: pgmap v5067: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:32.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:32.202658+0000) 2022-09-10T07:07:33.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:32.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:32.831318+0000) 2022-09-10T07:07:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:33.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:33.202895+0000) 2022-09-10T07:07:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:33 smithi110 ceph-mon[97633]: pgmap v5068: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:34.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:33.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:33.831581+0000) 2022-09-10T07:07:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:33 smithi104 ceph-mon[97044]: pgmap v5068: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:34.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:34.203127+0000) 2022-09-10T07:07:35.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:34.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:34.831845+0000) 2022-09-10T07:07:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:35.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:35.203332+0000) 2022-09-10T07:07:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:35 smithi104 ceph-mon[97044]: pgmap v5069: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:35.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:35 smithi110 ceph-mon[97633]: pgmap v5069: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:36.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:35.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:35.832125+0000) 2022-09-10T07:07:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:36.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:36.203566+0000) 2022-09-10T07:07:37.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:36.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:36.832321+0000) 2022-09-10T07:07:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:37.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:37.203773+0000) 2022-09-10T07:07:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:37 smithi110 ceph-mon[97633]: pgmap v5070: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:38.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:37.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:37.832560+0000) 2022-09-10T07:07:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:37 smithi104 ceph-mon[97044]: pgmap v5070: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:38.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:38.204027+0000) 2022-09-10T07:07:39.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:38.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:38.832781+0000) 2022-09-10T07:07:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:39.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:39.204173+0000) 2022-09-10T07:07:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:39 smithi110 ceph-mon[97633]: pgmap v5071: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:40.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:39.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:39.833008+0000) 2022-09-10T07:07:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:39 smithi104 ceph-mon[97044]: pgmap v5071: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:40.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:40.204399+0000) 2022-09-10T07:07:41.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:40.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:40.833253+0000) 2022-09-10T07:07:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:41.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:41.204577+0000) 2022-09-10T07:07:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:41 smithi110 ceph-mon[97633]: pgmap v5072: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:42.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:41.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:41.833501+0000) 2022-09-10T07:07:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:41 smithi104 ceph-mon[97044]: pgmap v5072: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:42.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:42.204797+0000) 2022-09-10T07:07:43.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:42.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:42.833708+0000) 2022-09-10T07:07:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:43.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:43.205058+0000) 2022-09-10T07:07:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:43 smithi110 ceph-mon[97633]: pgmap v5073: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:44.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:43.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:43.833956+0000) 2022-09-10T07:07:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:43 smithi104 ceph-mon[97044]: pgmap v5073: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:44.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:44.205344+0000) 2022-09-10T07:07:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:07:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:07:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:07:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:07:45.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:44.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:44.834224+0000) 2022-09-10T07:07:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:07:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:07:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:07:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:07:45.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:07:45.354+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:07:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:45.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:45.205509+0000) 2022-09-10T07:07:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:45 smithi110 ceph-mon[97633]: pgmap v5074: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:46.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:45.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:45.834419+0000) 2022-09-10T07:07:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:45 smithi104 ceph-mon[97044]: pgmap v5074: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:46.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:46.205722+0000) 2022-09-10T07:07:47.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:46.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:46.834650+0000) 2022-09-10T07:07:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:47.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:47.205927+0000) 2022-09-10T07:07:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:47 smithi110 ceph-mon[97633]: pgmap v5075: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:48.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:47.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:47.834864+0000) 2022-09-10T07:07:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:47 smithi104 ceph-mon[97044]: pgmap v5075: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:48.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:48.206202+0000) 2022-09-10T07:07:49.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:48.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:48.835129+0000) 2022-09-10T07:07:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:49.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:49.206455+0000) 2022-09-10T07:07:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:49 smithi110 ceph-mon[97633]: pgmap v5076: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:50.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:49.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:49.835325+0000) 2022-09-10T07:07:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:49 smithi104 ceph-mon[97044]: pgmap v5076: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:50.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:50.206675+0000) 2022-09-10T07:07:51.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:50.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:50.835560+0000) 2022-09-10T07:07:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:51.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:51.206870+0000) 2022-09-10T07:07:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:51 smithi110 ceph-mon[97633]: pgmap v5077: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:52.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:51.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:51.835832+0000) 2022-09-10T07:07:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:51 smithi104 ceph-mon[97044]: pgmap v5077: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:52.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:52.207152+0000) 2022-09-10T07:07:53.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:52.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:52.836017+0000) 2022-09-10T07:07:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:53.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:53.207381+0000) 2022-09-10T07:07:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:53 smithi110 ceph-mon[97633]: pgmap v5078: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:54.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:53.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:53.836197+0000) 2022-09-10T07:07:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:53 smithi104 ceph-mon[97044]: pgmap v5078: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:54.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:54.207669+0000) 2022-09-10T07:07:55.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:54.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:54.836394+0000) 2022-09-10T07:07:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:55.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:55.207923+0000) 2022-09-10T07:07:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:55 smithi104 ceph-mon[97044]: pgmap v5079: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:55 smithi110 ceph-mon[97633]: pgmap v5079: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:07:56.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:55.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:55.836566+0000) 2022-09-10T07:07:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:56.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:56.208186+0000) 2022-09-10T07:07:57.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:56.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:56.836775+0000) 2022-09-10T07:07:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:57.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:57.208351+0000) 2022-09-10T07:07:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:57 smithi110 ceph-mon[97633]: pgmap v5080: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:58.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:57.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:57.836919+0000) 2022-09-10T07:07:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:57 smithi104 ceph-mon[97044]: pgmap v5080: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:07:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:58.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:58.208533+0000) 2022-09-10T07:07:59.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:58.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:58.837123+0000) 2022-09-10T07:07:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:07:59.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:59.208717+0000) 2022-09-10T07:08:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:07:59 smithi110 ceph-mon[97633]: pgmap v5081: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:00.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:07:59.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:07:59.837374+0000) 2022-09-10T07:08:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:07:59 smithi104 ceph-mon[97044]: pgmap v5081: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:00.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:00.208897+0000) 2022-09-10T07:08:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:08:00.366+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:08:01.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:00.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:00.837568+0000) 2022-09-10T07:08:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:01.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:01.209153+0000) 2022-09-10T07:08:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:01 smithi110 ceph-mon[97633]: pgmap v5082: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:02.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:01.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:01.837758+0000) 2022-09-10T07:08:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:01 smithi104 ceph-mon[97044]: pgmap v5082: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:02.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:02.209373+0000) 2022-09-10T07:08:03.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:02.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:02.837938+0000) 2022-09-10T07:08:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:03.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:03.209540+0000) 2022-09-10T07:08:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:03 smithi110 ceph-mon[97633]: pgmap v5083: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:04.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:03.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:03.838158+0000) 2022-09-10T07:08:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:03 smithi104 ceph-mon[97044]: pgmap v5083: 129 pgs: 129 active+clean; 8.2 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:04.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:04.209759+0000) 2022-09-10T07:08:05.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:04.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:04.838333+0000) 2022-09-10T07:08:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:05.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:05.209936+0000) 2022-09-10T07:08:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:05 smithi104 ceph-mon[97044]: pgmap v5084: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:08:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:05 smithi110 ceph-mon[97633]: pgmap v5084: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:08:06.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:05.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:05.838535+0000) 2022-09-10T07:08:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:06.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:06.210195+0000) 2022-09-10T07:08:07.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:06.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:06.838714+0000) 2022-09-10T07:08:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:07.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:07.210367+0000) 2022-09-10T07:08:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:07 smithi110 ceph-mon[97633]: pgmap v5085: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:08:08.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:07.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:07.838890+0000) 2022-09-10T07:08:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:07 smithi104 ceph-mon[97044]: pgmap v5085: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:08:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:08.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:08.210545+0000) 2022-09-10T07:08:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:08 smithi110 ceph-mon[97633]: osdmap e398: 12 total, 12 up, 12 in 2022-09-10T07:08:09.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:08.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:08.839132+0000) 2022-09-10T07:08:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:08 smithi104 ceph-mon[97044]: osdmap e398: 12 total, 12 up, 12 in 2022-09-10T07:08:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:09.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:09.210709+0000) 2022-09-10T07:08:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:09 smithi110 ceph-mon[97633]: pgmap v5087: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:08:10.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:09.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:09.839259+0000) 2022-09-10T07:08:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:09 smithi104 ceph-mon[97044]: pgmap v5087: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:08:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:10.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:10.210956+0000) 2022-09-10T07:08:11.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:10.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:10.839443+0000) 2022-09-10T07:08:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:11.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:11.211171+0000) 2022-09-10T07:08:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:11 smithi110 ceph-mon[97633]: pgmap v5088: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:08:12.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:11.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:11.839717+0000) 2022-09-10T07:08:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:11 smithi104 ceph-mon[97044]: pgmap v5088: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:08:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:12.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:12.211305+0000) 2022-09-10T07:08:13.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:12.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:12.839942+0000) 2022-09-10T07:08:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:13.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:13.211510+0000) 2022-09-10T07:08:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:13 smithi110 ceph-mon[97633]: pgmap v5089: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:08:14.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:13.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:13.840162+0000) 2022-09-10T07:08:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:13 smithi104 ceph-mon[97044]: pgmap v5089: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:08:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:14.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:14.211732+0000) 2022-09-10T07:08:15.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:14.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:14.840392+0000) 2022-09-10T07:08:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:15.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:15.211968+0000) 2022-09-10T07:08:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:08:15.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:08:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:15 smithi104 ceph-mon[97044]: osdmap e399: 12 total, 12 up, 12 in 2022-09-10T07:08:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:15 smithi104 ceph-mon[97044]: pgmap v5091: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:08:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:15 smithi110 ceph-mon[97633]: osdmap e399: 12 total, 12 up, 12 in 2022-09-10T07:08:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:15 smithi110 ceph-mon[97633]: pgmap v5091: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:08:16.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:15.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:15.840556+0000) 2022-09-10T07:08:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:16.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:16.212177+0000) 2022-09-10T07:08:17.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:16.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:16.840767+0000) 2022-09-10T07:08:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:17.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:17.212365+0000) 2022-09-10T07:08:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:17 smithi110 ceph-mon[97633]: pgmap v5092: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:08:18.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:17.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:17.841002+0000) 2022-09-10T07:08:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:17 smithi104 ceph-mon[97044]: pgmap v5092: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:08:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:18.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:18.212613+0000) 2022-09-10T07:08:19.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:18.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:18.841211+0000) 2022-09-10T07:08:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:19.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:19.212766+0000) 2022-09-10T07:08:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:19 smithi110 ceph-mon[97633]: pgmap v5093: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:08:20.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:19.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:19.841414+0000) 2022-09-10T07:08:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:19 smithi104 ceph-mon[97044]: pgmap v5093: 129 pgs: 129 active+clean; 8.2 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:08:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:20.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:20.213021+0000) 2022-09-10T07:08:21.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:20.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:20.841620+0000) 2022-09-10T07:08:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:21.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:21.213270+0000) 2022-09-10T07:08:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:21 smithi110 ceph-mon[97633]: pgmap v5094: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:08:22.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:21.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:21.841822+0000) 2022-09-10T07:08:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:21 smithi104 ceph-mon[97044]: pgmap v5094: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:08:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:22.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:22.213570+0000) 2022-09-10T07:08:23.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:22.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:22.842010+0000) 2022-09-10T07:08:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:23.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:23.213708+0000) 2022-09-10T07:08:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:23 smithi110 ceph-mon[97633]: pgmap v5095: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:08:24.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:23.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:23.842198+0000) 2022-09-10T07:08:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:23 smithi104 ceph-mon[97044]: pgmap v5095: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:08:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:24.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:24.213880+0000) 2022-09-10T07:08:25.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:24.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:24.842504+0000) 2022-09-10T07:08:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:25.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:25.214190+0000) 2022-09-10T07:08:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:25 smithi104 ceph-mon[97044]: pgmap v5096: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:08:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:25 smithi110 ceph-mon[97633]: pgmap v5096: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:08:26.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:25.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:25.842681+0000) 2022-09-10T07:08:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:26.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:26.214428+0000) 2022-09-10T07:08:27.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:26.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:26.842827+0000) 2022-09-10T07:08:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:27.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:27.214614+0000) 2022-09-10T07:08:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:27 smithi110 ceph-mon[97633]: pgmap v5097: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:28.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:27.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:27.843053+0000) 2022-09-10T07:08:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:27 smithi104 ceph-mon[97044]: pgmap v5097: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:28.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:28.214824+0000) 2022-09-10T07:08:29.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:28.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:28.843224+0000) 2022-09-10T07:08:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:29.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:29.215024+0000) 2022-09-10T07:08:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:29 smithi110 ceph-mon[97633]: pgmap v5098: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:30.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:29.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:29.843430+0000) 2022-09-10T07:08:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:29 smithi104 ceph-mon[97044]: pgmap v5098: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:30.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:30.215237+0000) 2022-09-10T07:08:30.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:08:30.384+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:08:31.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:30.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:30.843594+0000) 2022-09-10T07:08:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:31.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:31.215487+0000) 2022-09-10T07:08:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:31 smithi110 ceph-mon[97633]: pgmap v5099: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:32.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:31.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:31.843769+0000) 2022-09-10T07:08:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:31 smithi104 ceph-mon[97044]: pgmap v5099: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:32.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:32.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:32.215742+0000) 2022-09-10T07:08:33.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:32.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:32.843958+0000) 2022-09-10T07:08:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:33.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:33.215998+0000) 2022-09-10T07:08:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:33 smithi110 ceph-mon[97633]: pgmap v5100: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:34.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:33.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:33.844160+0000) 2022-09-10T07:08:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:33 smithi104 ceph-mon[97044]: pgmap v5100: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:34.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:34.216201+0000) 2022-09-10T07:08:35.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:34.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:34.844400+0000) 2022-09-10T07:08:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:35.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:35.216401+0000) 2022-09-10T07:08:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:35 smithi104 ceph-mon[97044]: pgmap v5101: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:35 smithi110 ceph-mon[97633]: pgmap v5101: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:36.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:35.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:35.844647+0000) 2022-09-10T07:08:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:36.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:36.216535+0000) 2022-09-10T07:08:37.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:36.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:36.844849+0000) 2022-09-10T07:08:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:37.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:37.216784+0000) 2022-09-10T07:08:38.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:37 smithi110 ceph-mon[97633]: pgmap v5102: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:38.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:37.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:37.845076+0000) 2022-09-10T07:08:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:37 smithi104 ceph-mon[97044]: pgmap v5102: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:38.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:38.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:38.217013+0000) 2022-09-10T07:08:39.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:38.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:38.845328+0000) 2022-09-10T07:08:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:39.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:39.217262+0000) 2022-09-10T07:08:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:39 smithi110 ceph-mon[97633]: pgmap v5103: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:40.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:39.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:39.845567+0000) 2022-09-10T07:08:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:39 smithi104 ceph-mon[97044]: pgmap v5103: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:40.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:40.217513+0000) 2022-09-10T07:08:41.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:40.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:40.845845+0000) 2022-09-10T07:08:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:41.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:41.217756+0000) 2022-09-10T07:08:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:41 smithi110 ceph-mon[97633]: pgmap v5104: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:42.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:41.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:41.846086+0000) 2022-09-10T07:08:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:41 smithi104 ceph-mon[97044]: pgmap v5104: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:42.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:42.217959+0000) 2022-09-10T07:08:43.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:42.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:42.846318+0000) 2022-09-10T07:08:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:43.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:43.218208+0000) 2022-09-10T07:08:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:43 smithi110 ceph-mon[97633]: pgmap v5105: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:44.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:43.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:43.846537+0000) 2022-09-10T07:08:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:43 smithi104 ceph-mon[97044]: pgmap v5105: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:44.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:44.218493+0000) 2022-09-10T07:08:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:08:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:08:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:08:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:08:45.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:44.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:44.846746+0000) 2022-09-10T07:08:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:08:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:08:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:08:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:08:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:45.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:45.218745+0000) 2022-09-10T07:08:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:08:45.394+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:08:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:45 smithi110 ceph-mon[97633]: pgmap v5106: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:46.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:45.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:45.846923+0000) 2022-09-10T07:08:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:45 smithi104 ceph-mon[97044]: pgmap v5106: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:46.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:46.219008+0000) 2022-09-10T07:08:47.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:46.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:46.847174+0000) 2022-09-10T07:08:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:47.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:47.219251+0000) 2022-09-10T07:08:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:47 smithi110 ceph-mon[97633]: pgmap v5107: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:08:48.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:47.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:47.847370+0000) 2022-09-10T07:08:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:47 smithi104 ceph-mon[97044]: pgmap v5107: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:08:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:48.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:48.219415+0000) 2022-09-10T07:08:49.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:48.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:48.847538+0000) 2022-09-10T07:08:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:49.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:49.219582+0000) 2022-09-10T07:08:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:49 smithi110 ceph-mon[97633]: pgmap v5108: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:08:50.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:49.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:49.847795+0000) 2022-09-10T07:08:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:49 smithi104 ceph-mon[97044]: pgmap v5108: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:08:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:50.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:50.219749+0000) 2022-09-10T07:08:51.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:50.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:50.848056+0000) 2022-09-10T07:08:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:51.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:51.219980+0000) 2022-09-10T07:08:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:51 smithi110 ceph-mon[97633]: pgmap v5109: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:52.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:51.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:51.848343+0000) 2022-09-10T07:08:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:51 smithi104 ceph-mon[97044]: pgmap v5109: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:52.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:52.220155+0000) 2022-09-10T07:08:53.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:52.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:52.848574+0000) 2022-09-10T07:08:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:53.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:53.220359+0000) 2022-09-10T07:08:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:53 smithi110 ceph-mon[97633]: pgmap v5110: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:08:54.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:53.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:53.848771+0000) 2022-09-10T07:08:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:53 smithi104 ceph-mon[97044]: pgmap v5110: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:08:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:54.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:54.220524+0000) 2022-09-10T07:08:55.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:54.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:54.849037+0000) 2022-09-10T07:08:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:55.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:55.220735+0000) 2022-09-10T07:08:55.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:55 smithi104 ceph-mon[97044]: pgmap v5111: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:55 smithi110 ceph-mon[97633]: pgmap v5111: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:08:56.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:55.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:55.849267+0000) 2022-09-10T07:08:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:56.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:56.220947+0000) 2022-09-10T07:08:57.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:56.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:56.849446+0000) 2022-09-10T07:08:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:57.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:57.221176+0000) 2022-09-10T07:08:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:57 smithi110 ceph-mon[97633]: pgmap v5112: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:58.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:57.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:57.849606+0000) 2022-09-10T07:08:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:57 smithi104 ceph-mon[97044]: pgmap v5112: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:08:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:58.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:58.221391+0000) 2022-09-10T07:08:59.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:58.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:58.849777+0000) 2022-09-10T07:08:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:08:59.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:59.221630+0000) 2022-09-10T07:09:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:08:59 smithi110 ceph-mon[97633]: pgmap v5113: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:00.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:08:59.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:08:59.849998+0000) 2022-09-10T07:09:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:08:59 smithi104 ceph-mon[97044]: pgmap v5113: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:09:00.406+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:09:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:00.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:00.221838+0000) 2022-09-10T07:09:01.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:00.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:00.850183+0000) 2022-09-10T07:09:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:01.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:01.222034+0000) 2022-09-10T07:09:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:01 smithi110 ceph-mon[97633]: pgmap v5114: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:02.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:01.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:01.850499+0000) 2022-09-10T07:09:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:01 smithi104 ceph-mon[97044]: pgmap v5114: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:02.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:02.222278+0000) 2022-09-10T07:09:03.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:02.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:02.850771+0000) 2022-09-10T07:09:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:03.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:03.222517+0000) 2022-09-10T07:09:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:03 smithi110 ceph-mon[97633]: pgmap v5115: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:04.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:03.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:03.850991+0000) 2022-09-10T07:09:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:03 smithi104 ceph-mon[97044]: pgmap v5115: 129 pgs: 129 active+clean; 8.2 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:04.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:04.222724+0000) 2022-09-10T07:09:05.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:04.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:04.851229+0000) 2022-09-10T07:09:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:05.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:05.222979+0000) 2022-09-10T07:09:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:05 smithi104 ceph-mon[97044]: pgmap v5116: 129 pgs: 129 active+clean; 8.2 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:09:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:05 smithi110 ceph-mon[97633]: pgmap v5116: 129 pgs: 129 active+clean; 8.2 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:09:06.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:05.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:05.851394+0000) 2022-09-10T07:09:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:06.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:06.223159+0000) 2022-09-10T07:09:07.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:06.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:06.851634+0000) 2022-09-10T07:09:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:07.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:07.223335+0000) 2022-09-10T07:09:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:07 smithi110 ceph-mon[97633]: pgmap v5117: 129 pgs: 129 active+clean; 8.2 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:09:08.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:07.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:07.851787+0000) 2022-09-10T07:09:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:07 smithi104 ceph-mon[97044]: pgmap v5117: 129 pgs: 129 active+clean; 8.2 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:09:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:08.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:08.223596+0000) 2022-09-10T07:09:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:08 smithi110 ceph-mon[97633]: osdmap e400: 12 total, 12 up, 12 in 2022-09-10T07:09:09.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:08.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:08.851992+0000) 2022-09-10T07:09:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:08 smithi104 ceph-mon[97044]: osdmap e400: 12 total, 12 up, 12 in 2022-09-10T07:09:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:09.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:09.223784+0000) 2022-09-10T07:09:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:09 smithi110 ceph-mon[97633]: pgmap v5119: 129 pgs: 129 active+clean; 8.2 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:09:10.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:09.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:09.852237+0000) 2022-09-10T07:09:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:09 smithi104 ceph-mon[97044]: pgmap v5119: 129 pgs: 129 active+clean; 8.2 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:09:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:10.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:10.223966+0000) 2022-09-10T07:09:11.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:10.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:10.852512+0000) 2022-09-10T07:09:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:11.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:11.224219+0000) 2022-09-10T07:09:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:11 smithi110 ceph-mon[97633]: pgmap v5120: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:09:12.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:11.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:11.852691+0000) 2022-09-10T07:09:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:11 smithi104 ceph-mon[97044]: pgmap v5120: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:09:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:12.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:12.224417+0000) 2022-09-10T07:09:13.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:12.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:12.852872+0000) 2022-09-10T07:09:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:13.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:13.224643+0000) 2022-09-10T07:09:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:13 smithi110 ceph-mon[97633]: pgmap v5121: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:09:14.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:13.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:13.853057+0000) 2022-09-10T07:09:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:13 smithi104 ceph-mon[97044]: pgmap v5121: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:09:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:14.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:14.224876+0000) 2022-09-10T07:09:15.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:14.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:14.853323+0000) 2022-09-10T07:09:15.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:09:15.422+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:09:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:15.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:15.225177+0000) 2022-09-10T07:09:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:15 smithi104 ceph-mon[97044]: osdmap e401: 12 total, 12 up, 12 in 2022-09-10T07:09:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:15 smithi104 ceph-mon[97044]: pgmap v5123: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:09:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:15 smithi110 ceph-mon[97633]: osdmap e401: 12 total, 12 up, 12 in 2022-09-10T07:09:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:15 smithi110 ceph-mon[97633]: pgmap v5123: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:09:16.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:15.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:15.853529+0000) 2022-09-10T07:09:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:16.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:16.225381+0000) 2022-09-10T07:09:17.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:16.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:16.853697+0000) 2022-09-10T07:09:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:17.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:17.225539+0000) 2022-09-10T07:09:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:17 smithi110 ceph-mon[97633]: pgmap v5124: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:09:18.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:17.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:17.853863+0000) 2022-09-10T07:09:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:17 smithi104 ceph-mon[97044]: pgmap v5124: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:09:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:18.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:18.225703+0000) 2022-09-10T07:09:19.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:18.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:18.854036+0000) 2022-09-10T07:09:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:19.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:19.225956+0000) 2022-09-10T07:09:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:19 smithi110 ceph-mon[97633]: pgmap v5125: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:09:20.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:19.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:19.854236+0000) 2022-09-10T07:09:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:19 smithi104 ceph-mon[97044]: pgmap v5125: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:09:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:20.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:20.226228+0000) 2022-09-10T07:09:21.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:20.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:20.854422+0000) 2022-09-10T07:09:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:21.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:21.226481+0000) 2022-09-10T07:09:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:21 smithi110 ceph-mon[97633]: pgmap v5126: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:09:22.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:21.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:21.854658+0000) 2022-09-10T07:09:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:21 smithi104 ceph-mon[97044]: pgmap v5126: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:09:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:22.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:22.226737+0000) 2022-09-10T07:09:23.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:22.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:22.854887+0000) 2022-09-10T07:09:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:23.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:23.226981+0000) 2022-09-10T07:09:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:23 smithi110 ceph-mon[97633]: pgmap v5127: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:09:24.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:23.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:23.855114+0000) 2022-09-10T07:09:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:23 smithi104 ceph-mon[97044]: pgmap v5127: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:09:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:24.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:24.227243+0000) 2022-09-10T07:09:25.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:24.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:24.855268+0000) 2022-09-10T07:09:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:25.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:25.227477+0000) 2022-09-10T07:09:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:25 smithi104 ceph-mon[97044]: pgmap v5128: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:09:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:25 smithi110 ceph-mon[97633]: pgmap v5128: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:09:26.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:25.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:25.855473+0000) 2022-09-10T07:09:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:26.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:26.227674+0000) 2022-09-10T07:09:27.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:26.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:26.855628+0000) 2022-09-10T07:09:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:27.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:27.227876+0000) 2022-09-10T07:09:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:27 smithi110 ceph-mon[97633]: pgmap v5129: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:28.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:27.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:27.855824+0000) 2022-09-10T07:09:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:27 smithi104 ceph-mon[97044]: pgmap v5129: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:28.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:28.228120+0000) 2022-09-10T07:09:29.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:28.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:28.856027+0000) 2022-09-10T07:09:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:29.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:29.228297+0000) 2022-09-10T07:09:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:29 smithi110 ceph-mon[97633]: pgmap v5130: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:30.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:29.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:29.856252+0000) 2022-09-10T07:09:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:29 smithi104 ceph-mon[97044]: pgmap v5130: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:09:30.435+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:09:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:30.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:30.228465+0000) 2022-09-10T07:09:31.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:30.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:30.856375+0000) 2022-09-10T07:09:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:31.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:31.228636+0000) 2022-09-10T07:09:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:31 smithi110 ceph-mon[97633]: pgmap v5131: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:32.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:31.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:31.856520+0000) 2022-09-10T07:09:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:31 smithi104 ceph-mon[97044]: pgmap v5131: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:32.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:32.228854+0000) 2022-09-10T07:09:33.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:32.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:32.856719+0000) 2022-09-10T07:09:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:33.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:33.229132+0000) 2022-09-10T07:09:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:33 smithi110 ceph-mon[97633]: pgmap v5132: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:34.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:33.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:33.856904+0000) 2022-09-10T07:09:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:33 smithi104 ceph-mon[97044]: pgmap v5132: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:34.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:34.229335+0000) 2022-09-10T07:09:35.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:34.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:34.857123+0000) 2022-09-10T07:09:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:35.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:35.229546+0000) 2022-09-10T07:09:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:35 smithi104 ceph-mon[97044]: pgmap v5133: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:35 smithi110 ceph-mon[97633]: pgmap v5133: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:36.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:35.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:35.857288+0000) 2022-09-10T07:09:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:36.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:36.229757+0000) 2022-09-10T07:09:37.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:36.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:36.857486+0000) 2022-09-10T07:09:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:37.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:37.229953+0000) 2022-09-10T07:09:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:37 smithi110 ceph-mon[97633]: pgmap v5134: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:38.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:37.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:37.857623+0000) 2022-09-10T07:09:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:37 smithi104 ceph-mon[97044]: pgmap v5134: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:38.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:38.230187+0000) 2022-09-10T07:09:39.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:38.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:38.857865+0000) 2022-09-10T07:09:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:39.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:39.230357+0000) 2022-09-10T07:09:40.172 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:39 smithi110 ceph-mon[97633]: pgmap v5135: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:40.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:39.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:39.858048+0000) 2022-09-10T07:09:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:39 smithi104 ceph-mon[97044]: pgmap v5135: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:40.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:40.230603+0000) 2022-09-10T07:09:41.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:40.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:40.858221+0000) 2022-09-10T07:09:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:41.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:41.230811+0000) 2022-09-10T07:09:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:41 smithi110 ceph-mon[97633]: pgmap v5136: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:42.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:41.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:41.858375+0000) 2022-09-10T07:09:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:41 smithi104 ceph-mon[97044]: pgmap v5136: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:42.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:42.231034+0000) 2022-09-10T07:09:43.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:42.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:42.858515+0000) 2022-09-10T07:09:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:43.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:43.231280+0000) 2022-09-10T07:09:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:43 smithi110 ceph-mon[97633]: pgmap v5137: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:44.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:43.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:43.858658+0000) 2022-09-10T07:09:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:43 smithi104 ceph-mon[97044]: pgmap v5137: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:44.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:44.231510+0000) 2022-09-10T07:09:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:09:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:09:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:09:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:09:45.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:44.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:44.858802+0000) 2022-09-10T07:09:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:09:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:09:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:09:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:09:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:45.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:45.231774+0000) 2022-09-10T07:09:45.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:09:45.443+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:09:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:45 smithi110 ceph-mon[97633]: pgmap v5138: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:46.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:45.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:45.859051+0000) 2022-09-10T07:09:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:45 smithi104 ceph-mon[97044]: pgmap v5138: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:46.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:46.231950+0000) 2022-09-10T07:09:47.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:46.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:46.859187+0000) 2022-09-10T07:09:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:47.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:47.232191+0000) 2022-09-10T07:09:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:47 smithi110 ceph-mon[97633]: pgmap v5139: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:09:48.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:47.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:47.859377+0000) 2022-09-10T07:09:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:47 smithi104 ceph-mon[97044]: pgmap v5139: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:09:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:48.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:48.232403+0000) 2022-09-10T07:09:49.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:48.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:48.859569+0000) 2022-09-10T07:09:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:49.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:49.232600+0000) 2022-09-10T07:09:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:49 smithi110 ceph-mon[97633]: pgmap v5140: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:09:50.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:49.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:49.859784+0000) 2022-09-10T07:09:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:49 smithi104 ceph-mon[97044]: pgmap v5140: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:09:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:50.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:50.232829+0000) 2022-09-10T07:09:51.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:50.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:50.859983+0000) 2022-09-10T07:09:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:51.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:51.233001+0000) 2022-09-10T07:09:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:51 smithi110 ceph-mon[97633]: pgmap v5141: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:52.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:51.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:51.860261+0000) 2022-09-10T07:09:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:51 smithi104 ceph-mon[97044]: pgmap v5141: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:52.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:52.233239+0000) 2022-09-10T07:09:53.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:52.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:52.860506+0000) 2022-09-10T07:09:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:53.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:53.233460+0000) 2022-09-10T07:09:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:53 smithi110 ceph-mon[97633]: pgmap v5142: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:09:54.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:53.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:53.860747+0000) 2022-09-10T07:09:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:53 smithi104 ceph-mon[97044]: pgmap v5142: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:09:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:54.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:54.233693+0000) 2022-09-10T07:09:55.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:54.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:54.860949+0000) 2022-09-10T07:09:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:55.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:55.233854+0000) 2022-09-10T07:09:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:55 smithi104 ceph-mon[97044]: pgmap v5143: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:55 smithi110 ceph-mon[97633]: pgmap v5143: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:09:56.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:55.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:55.861199+0000) 2022-09-10T07:09:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:56.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:56.234011+0000) 2022-09-10T07:09:57.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:56.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:56.861384+0000) 2022-09-10T07:09:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:57.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:57.234235+0000) 2022-09-10T07:09:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:57 smithi110 ceph-mon[97633]: pgmap v5144: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:58.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:57.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:57.861569+0000) 2022-09-10T07:09:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:57 smithi104 ceph-mon[97044]: pgmap v5144: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:09:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:58.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:58.234476+0000) 2022-09-10T07:09:59.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:58.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:58.861765+0000) 2022-09-10T07:09:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:09:59.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:59.234730+0000) 2022-09-10T07:10:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:09:59 smithi110 ceph-mon[97633]: pgmap v5145: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:00.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:09:59.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:09:59.861994+0000) 2022-09-10T07:10:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:09:59 smithi104 ceph-mon[97044]: pgmap v5145: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:00.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:00.234915+0000) 2022-09-10T07:10:00.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:10:00.460+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:10:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:10:01.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:00.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:00.862198+0000) 2022-09-10T07:10:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:10:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:01.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:01.235125+0000) 2022-09-10T07:10:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:01 smithi110 ceph-mon[97633]: pgmap v5146: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:02.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:01.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:01.862384+0000) 2022-09-10T07:10:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:01 smithi104 ceph-mon[97044]: pgmap v5146: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:02.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:02.235298+0000) 2022-09-10T07:10:03.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:02.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:02.862624+0000) 2022-09-10T07:10:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:03.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:03.235540+0000) 2022-09-10T07:10:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:03 smithi110 ceph-mon[97633]: pgmap v5147: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:04.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:03.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:03.862801+0000) 2022-09-10T07:10:04.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:03 smithi104 ceph-mon[97044]: pgmap v5147: 129 pgs: 129 active+clean; 8.2 MiB data, 603 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:04.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:04.235787+0000) 2022-09-10T07:10:05.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:04.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:04.863009+0000) 2022-09-10T07:10:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:05.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:05.236069+0000) 2022-09-10T07:10:05.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:05 smithi104 ceph-mon[97044]: pgmap v5148: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:10:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:05 smithi110 ceph-mon[97633]: pgmap v5148: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:10:06.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:05.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:05.863259+0000) 2022-09-10T07:10:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:06.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:06.236349+0000) 2022-09-10T07:10:07.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:06.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:06.863453+0000) 2022-09-10T07:10:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:07.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:07.236545+0000) 2022-09-10T07:10:08.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:07 smithi110 ceph-mon[97633]: pgmap v5149: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:10:08.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:07.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:07.863610+0000) 2022-09-10T07:10:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:07 smithi104 ceph-mon[97044]: pgmap v5149: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:10:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:08.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:08.236727+0000) 2022-09-10T07:10:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:08 smithi110 ceph-mon[97633]: osdmap e402: 12 total, 12 up, 12 in 2022-09-10T07:10:09.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:08.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:08.863772+0000) 2022-09-10T07:10:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:08 smithi104 ceph-mon[97044]: osdmap e402: 12 total, 12 up, 12 in 2022-09-10T07:10:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:09.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:09.236949+0000) 2022-09-10T07:10:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:09 smithi110 ceph-mon[97633]: pgmap v5151: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:10:10.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:09.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:09.863960+0000) 2022-09-10T07:10:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:09 smithi104 ceph-mon[97044]: pgmap v5151: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:10:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:10.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:10.237180+0000) 2022-09-10T07:10:11.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:10.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:10.864164+0000) 2022-09-10T07:10:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:11.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:11.237397+0000) 2022-09-10T07:10:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:11 smithi110 ceph-mon[97633]: pgmap v5152: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:10:12.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:11.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:11.864345+0000) 2022-09-10T07:10:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:11 smithi104 ceph-mon[97044]: pgmap v5152: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:10:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:12.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:12.237594+0000) 2022-09-10T07:10:13.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:12.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:12.864546+0000) 2022-09-10T07:10:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:13.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:13.237867+0000) 2022-09-10T07:10:14.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:13.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:13.864777+0000) 2022-09-10T07:10:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:13 smithi104 ceph-mon[97044]: pgmap v5153: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:10:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:13 smithi110 ceph-mon[97633]: pgmap v5153: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:10:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:14.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:14.238141+0000) 2022-09-10T07:10:15.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:14.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:14.864981+0000) 2022-09-10T07:10:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:15.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:15.238382+0000) 2022-09-10T07:10:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:10:15.474+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:10:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:15 smithi104 ceph-mon[97044]: osdmap e403: 12 total, 12 up, 12 in 2022-09-10T07:10:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:15 smithi104 ceph-mon[97044]: pgmap v5155: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:10:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:15 smithi110 ceph-mon[97633]: osdmap e403: 12 total, 12 up, 12 in 2022-09-10T07:10:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:15 smithi110 ceph-mon[97633]: pgmap v5155: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:10:16.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:15.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:15.865182+0000) 2022-09-10T07:10:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:16.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:16.238578+0000) 2022-09-10T07:10:17.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:16.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:16.865360+0000) 2022-09-10T07:10:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:17.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:17.238802+0000) 2022-09-10T07:10:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:17 smithi110 ceph-mon[97633]: pgmap v5156: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:10:18.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:17.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:17.865603+0000) 2022-09-10T07:10:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:17 smithi104 ceph-mon[97044]: pgmap v5156: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:10:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:18.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:18.239032+0000) 2022-09-10T07:10:19.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:18.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:18.865826+0000) 2022-09-10T07:10:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:19.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:19.239268+0000) 2022-09-10T07:10:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:19 smithi110 ceph-mon[97633]: pgmap v5157: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:10:20.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:19.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:19.866068+0000) 2022-09-10T07:10:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:19 smithi104 ceph-mon[97044]: pgmap v5157: 129 pgs: 129 active+clean; 8.2 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:10:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:20.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:20.239531+0000) 2022-09-10T07:10:21.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:20.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:20.866308+0000) 2022-09-10T07:10:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:21.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:21.239782+0000) 2022-09-10T07:10:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:21 smithi110 ceph-mon[97633]: pgmap v5158: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:10:22.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:21.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:21.866563+0000) 2022-09-10T07:10:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:21 smithi104 ceph-mon[97044]: pgmap v5158: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:10:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:22.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:22.240038+0000) 2022-09-10T07:10:23.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:22.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:22.866733+0000) 2022-09-10T07:10:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:23.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:23.240202+0000) 2022-09-10T07:10:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:23 smithi110 ceph-mon[97633]: pgmap v5159: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:10:24.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:23.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:23.866911+0000) 2022-09-10T07:10:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:23 smithi104 ceph-mon[97044]: pgmap v5159: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:10:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:24.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:24.240420+0000) 2022-09-10T07:10:25.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:24.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:24.867190+0000) 2022-09-10T07:10:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:25.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:25.240701+0000) 2022-09-10T07:10:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:25 smithi104 ceph-mon[97044]: pgmap v5160: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:10:25.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:25 smithi110 ceph-mon[97633]: pgmap v5160: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:10:26.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:25.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:25.867420+0000) 2022-09-10T07:10:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:26.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:26.240969+0000) 2022-09-10T07:10:27.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:26.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:26.867650+0000) 2022-09-10T07:10:27.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:27.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:27.241229+0000) 2022-09-10T07:10:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:27 smithi110 ceph-mon[97633]: pgmap v5161: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:28.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:27.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:27.867884+0000) 2022-09-10T07:10:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:27 smithi104 ceph-mon[97044]: pgmap v5161: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:28.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:28.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:28.241478+0000) 2022-09-10T07:10:29.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:28.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:28.868125+0000) 2022-09-10T07:10:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:29.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:29.241726+0000) 2022-09-10T07:10:30.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:29 smithi110 ceph-mon[97633]: pgmap v5162: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:30.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:29.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:29.868339+0000) 2022-09-10T07:10:30.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:29 smithi104 ceph-mon[97044]: pgmap v5162: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:30.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:30.242071+0000) 2022-09-10T07:10:30.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:10:30.490+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:10:31.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:30.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:30.868525+0000) 2022-09-10T07:10:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:31.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:31.242325+0000) 2022-09-10T07:10:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:31 smithi110 ceph-mon[97633]: pgmap v5163: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:32.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:31.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:31.868688+0000) 2022-09-10T07:10:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:31 smithi104 ceph-mon[97044]: pgmap v5163: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:32.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:32.242551+0000) 2022-09-10T07:10:33.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:32.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:32.868911+0000) 2022-09-10T07:10:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:33.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:33.242796+0000) 2022-09-10T07:10:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:33 smithi110 ceph-mon[97633]: pgmap v5164: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:34.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:33.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:33.869091+0000) 2022-09-10T07:10:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:33 smithi104 ceph-mon[97044]: pgmap v5164: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:34.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:34.242980+0000) 2022-09-10T07:10:35.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:34.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:34.869286+0000) 2022-09-10T07:10:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:35.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:35.243229+0000) 2022-09-10T07:10:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:35 smithi104 ceph-mon[97044]: pgmap v5165: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:35 smithi110 ceph-mon[97633]: pgmap v5165: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:36.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:35.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:35.869506+0000) 2022-09-10T07:10:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:36.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:36.243449+0000) 2022-09-10T07:10:37.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:36.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:36.869686+0000) 2022-09-10T07:10:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:37.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:37.243623+0000) 2022-09-10T07:10:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:37 smithi110 ceph-mon[97633]: pgmap v5166: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:38.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:37.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:37.869897+0000) 2022-09-10T07:10:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:37 smithi104 ceph-mon[97044]: pgmap v5166: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:38.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:38.243808+0000) 2022-09-10T07:10:39.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:38.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:38.870089+0000) 2022-09-10T07:10:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:39.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:39.244030+0000) 2022-09-10T07:10:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:39 smithi110 ceph-mon[97633]: pgmap v5167: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:40.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:39.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:39.870308+0000) 2022-09-10T07:10:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:39 smithi104 ceph-mon[97044]: pgmap v5167: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:40.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:40.244280+0000) 2022-09-10T07:10:41.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:40.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:40.870501+0000) 2022-09-10T07:10:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:41.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:41.244529+0000) 2022-09-10T07:10:42.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:41 smithi110 ceph-mon[97633]: pgmap v5168: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:42.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:41.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:41.870682+0000) 2022-09-10T07:10:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:41 smithi104 ceph-mon[97044]: pgmap v5168: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:42.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:42.244678+0000) 2022-09-10T07:10:43.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:42.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:42.870910+0000) 2022-09-10T07:10:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:43.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:43.244897+0000) 2022-09-10T07:10:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:43 smithi110 ceph-mon[97633]: pgmap v5169: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:44.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:43.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:43.871089+0000) 2022-09-10T07:10:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:43 smithi104 ceph-mon[97044]: pgmap v5169: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:44.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:44.245180+0000) 2022-09-10T07:10:45.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:44.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:44.871352+0000) 2022-09-10T07:10:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:10:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:10:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:10:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:10:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:10:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:10:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:10:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:10:45.504 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:45.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:45.245403+0000) 2022-09-10T07:10:45.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:10:45.502+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:10:46.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:45.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:45.871593+0000) 2022-09-10T07:10:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:45 smithi104 ceph-mon[97044]: pgmap v5170: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:45 smithi110 ceph-mon[97633]: pgmap v5170: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:46.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:46.245608+0000) 2022-09-10T07:10:47.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:46.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:46.871832+0000) 2022-09-10T07:10:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:47.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:47.245777+0000) 2022-09-10T07:10:48.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:47.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:47.871984+0000) 2022-09-10T07:10:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:47 smithi104 ceph-mon[97044]: pgmap v5171: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:47 smithi110 ceph-mon[97633]: pgmap v5171: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:48.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:48.246003+0000) 2022-09-10T07:10:49.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:48.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:48.872178+0000) 2022-09-10T07:10:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:49.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:49.246231+0000) 2022-09-10T07:10:50.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:49.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:49.872363+0000) 2022-09-10T07:10:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:49 smithi104 ceph-mon[97044]: pgmap v5172: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:49 smithi110 ceph-mon[97633]: pgmap v5172: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:50.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:50.246409+0000) 2022-09-10T07:10:51.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:50.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:50.872576+0000) 2022-09-10T07:10:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:51.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:51.246528+0000) 2022-09-10T07:10:52.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:51.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:51.872707+0000) 2022-09-10T07:10:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:51 smithi104 ceph-mon[97044]: pgmap v5173: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:51 smithi110 ceph-mon[97633]: pgmap v5173: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:52.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:52.246737+0000) 2022-09-10T07:10:53.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:52.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:52.872917+0000) 2022-09-10T07:10:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:53.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:53.246939+0000) 2022-09-10T07:10:54.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:53.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:53.873181+0000) 2022-09-10T07:10:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:53 smithi104 ceph-mon[97044]: pgmap v5174: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:53 smithi110 ceph-mon[97633]: pgmap v5174: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:54.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:54.247195+0000) 2022-09-10T07:10:55.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:54.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:54.873412+0000) 2022-09-10T07:10:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:55.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:55.247428+0000) 2022-09-10T07:10:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:55 smithi104 ceph-mon[97044]: pgmap v5175: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:55 smithi110 ceph-mon[97633]: pgmap v5175: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:10:56.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:55.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:55.873694+0000) 2022-09-10T07:10:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:56.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:56.247678+0000) 2022-09-10T07:10:57.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:56.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:56.873890+0000) 2022-09-10T07:10:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:57.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:57.247908+0000) 2022-09-10T07:10:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:57 smithi110 ceph-mon[97633]: pgmap v5176: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:58.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:57.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:57.874171+0000) 2022-09-10T07:10:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:57 smithi104 ceph-mon[97044]: pgmap v5176: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:10:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:58.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:58.248160+0000) 2022-09-10T07:10:59.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:58.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:58.874391+0000) 2022-09-10T07:10:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:10:59.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:59.248287+0000) 2022-09-10T07:11:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:10:59 smithi110 ceph-mon[97633]: pgmap v5177: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:00.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:10:59.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:10:59.874627+0000) 2022-09-10T07:11:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:10:59 smithi104 ceph-mon[97044]: pgmap v5177: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:00.511 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:00.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:00.248459+0000) 2022-09-10T07:11:00.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:11:00.509+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:11:01.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:00.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:00.874857+0000) 2022-09-10T07:11:01.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:01.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:01.248718+0000) 2022-09-10T07:11:02.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:01.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:01.875174+0000) 2022-09-10T07:11:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:01 smithi104 ceph-mon[97044]: pgmap v5178: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:01 smithi110 ceph-mon[97633]: pgmap v5178: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:02.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:02.248849+0000) 2022-09-10T07:11:03.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:02.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:02.875359+0000) 2022-09-10T07:11:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:03.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:03.249050+0000) 2022-09-10T07:11:04.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:03.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:03.875593+0000) 2022-09-10T07:11:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:03 smithi104 ceph-mon[97044]: pgmap v5179: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:03 smithi110 ceph-mon[97633]: pgmap v5179: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:04.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:04.249281+0000) 2022-09-10T07:11:05.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:04.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:04.875870+0000) 2022-09-10T07:11:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:05.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:05.249548+0000) 2022-09-10T07:11:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:05 smithi104 ceph-mon[97044]: pgmap v5180: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:11:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:05 smithi110 ceph-mon[97633]: pgmap v5180: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:11:06.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:05.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:05.876077+0000) 2022-09-10T07:11:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:06.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:06.249762+0000) 2022-09-10T07:11:07.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:06.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:06.876266+0000) 2022-09-10T07:11:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:07.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:07.249984+0000) 2022-09-10T07:11:08.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:07.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:07.876430+0000) 2022-09-10T07:11:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:07 smithi104 ceph-mon[97044]: pgmap v5181: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:11:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:07 smithi110 ceph-mon[97633]: pgmap v5181: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:11:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:08.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:08.250254+0000) 2022-09-10T07:11:09.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:08.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:08.876641+0000) 2022-09-10T07:11:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:08 smithi104 ceph-mon[97044]: osdmap e404: 12 total, 12 up, 12 in 2022-09-10T07:11:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:08 smithi110 ceph-mon[97633]: osdmap e404: 12 total, 12 up, 12 in 2022-09-10T07:11:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:09.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:09.250505+0000) 2022-09-10T07:11:10.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:09.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:09.876820+0000) 2022-09-10T07:11:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:09 smithi104 ceph-mon[97044]: pgmap v5183: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:11:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:09 smithi110 ceph-mon[97633]: pgmap v5183: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:11:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:10.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:10.250774+0000) 2022-09-10T07:11:11.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:10.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:10.876973+0000) 2022-09-10T07:11:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:11.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:11.250942+0000) 2022-09-10T07:11:12.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:11.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:11.877142+0000) 2022-09-10T07:11:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:11 smithi104 ceph-mon[97044]: pgmap v5184: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:11:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:11 smithi110 ceph-mon[97633]: pgmap v5184: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:11:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:12.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:12.251158+0000) 2022-09-10T07:11:13.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:12.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:12.877359+0000) 2022-09-10T07:11:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:13.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:13.251372+0000) 2022-09-10T07:11:14.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:13.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:13.877537+0000) 2022-09-10T07:11:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:13 smithi104 ceph-mon[97044]: pgmap v5185: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:11:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:13 smithi110 ceph-mon[97633]: pgmap v5185: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:11:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:14.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:14.251587+0000) 2022-09-10T07:11:15.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:14.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:14.877759+0000) 2022-09-10T07:11:15.525 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:15.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:15.251833+0000) 2022-09-10T07:11:15.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:15 smithi104 ceph-mon[97044]: osdmap e405: 12 total, 12 up, 12 in 2022-09-10T07:11:15.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:15 smithi104 ceph-mon[97044]: pgmap v5187: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:11:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:11:15.523+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:15 smithi110 ceph-mon[97633]: osdmap e405: 12 total, 12 up, 12 in 2022-09-10T07:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:15 smithi110 ceph-mon[97633]: pgmap v5187: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:11:16.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:15.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:15.877885+0000) 2022-09-10T07:11:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:16.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:16.252055+0000) 2022-09-10T07:11:17.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:16.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:16.878048+0000) 2022-09-10T07:11:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:17.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:17.252305+0000) 2022-09-10T07:11:18.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:17.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:17.878184+0000) 2022-09-10T07:11:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:17 smithi104 ceph-mon[97044]: pgmap v5188: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:11:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:17 smithi110 ceph-mon[97633]: pgmap v5188: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:11:18.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:18.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:18.252487+0000) 2022-09-10T07:11:19.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:18.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:18.878309+0000) 2022-09-10T07:11:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:19.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:19.252699+0000) 2022-09-10T07:11:20.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:19.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:19.878450+0000) 2022-09-10T07:11:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:19 smithi104 ceph-mon[97044]: pgmap v5189: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:11:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:19 smithi110 ceph-mon[97633]: pgmap v5189: 129 pgs: 129 active+clean; 8.2 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:11:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:20.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:20.252954+0000) 2022-09-10T07:11:21.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:20.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:20.878649+0000) 2022-09-10T07:11:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:21.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:21.253197+0000) 2022-09-10T07:11:22.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:21.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:21.878805+0000) 2022-09-10T07:11:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:21 smithi104 ceph-mon[97044]: pgmap v5190: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:11:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:21 smithi110 ceph-mon[97633]: pgmap v5190: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:11:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:22.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:22.253430+0000) 2022-09-10T07:11:23.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:22.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:22.879010+0000) 2022-09-10T07:11:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:23.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:23.253624+0000) 2022-09-10T07:11:24.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:23.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:23.879224+0000) 2022-09-10T07:11:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:23 smithi104 ceph-mon[97044]: pgmap v5191: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:11:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:23 smithi110 ceph-mon[97633]: pgmap v5191: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:11:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:24.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:24.253852+0000) 2022-09-10T07:11:25.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:24.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:24.879467+0000) 2022-09-10T07:11:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:25.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:25.254140+0000) 2022-09-10T07:11:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:25 smithi104 ceph-mon[97044]: pgmap v5192: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:11:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:25 smithi110 ceph-mon[97633]: pgmap v5192: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:11:26.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:25.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:25.879676+0000) 2022-09-10T07:11:26.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:26.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:26.254330+0000) 2022-09-10T07:11:27.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:26.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:26.879866+0000) 2022-09-10T07:11:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:27.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:27.254464+0000) 2022-09-10T07:11:28.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:27.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:27.880055+0000) 2022-09-10T07:11:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:27 smithi104 ceph-mon[97044]: pgmap v5193: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:27 smithi110 ceph-mon[97633]: pgmap v5193: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:28.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:28.254636+0000) 2022-09-10T07:11:29.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:28.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:28.880263+0000) 2022-09-10T07:11:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:29.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:29.254938+0000) 2022-09-10T07:11:30.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:29.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:29.880501+0000) 2022-09-10T07:11:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:29 smithi104 ceph-mon[97044]: pgmap v5194: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:29 smithi110 ceph-mon[97633]: pgmap v5194: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:30.540 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:30.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:30.255205+0000) 2022-09-10T07:11:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:11:30.539+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:11:31.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:30.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:30.880747+0000) 2022-09-10T07:11:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:31.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:31.255349+0000) 2022-09-10T07:11:32.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:31.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:31.880971+0000) 2022-09-10T07:11:32.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:31 smithi104 ceph-mon[97044]: pgmap v5195: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:31 smithi110 ceph-mon[97633]: pgmap v5195: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:32.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:32.255587+0000) 2022-09-10T07:11:33.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:32.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:32.881143+0000) 2022-09-10T07:11:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:33.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:33.255827+0000) 2022-09-10T07:11:34.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:33.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:33.881355+0000) 2022-09-10T07:11:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:33 smithi104 ceph-mon[97044]: pgmap v5196: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:33 smithi110 ceph-mon[97633]: pgmap v5196: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:34.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:34.256004+0000) 2022-09-10T07:11:35.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:34.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:34.881563+0000) 2022-09-10T07:11:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:35.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:35.256233+0000) 2022-09-10T07:11:35.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:35 smithi104 ceph-mon[97044]: pgmap v5197: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:35 smithi110 ceph-mon[97633]: pgmap v5197: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:36.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:35.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:35.881785+0000) 2022-09-10T07:11:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:36.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:36.256466+0000) 2022-09-10T07:11:37.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:36.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:36.882007+0000) 2022-09-10T07:11:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:37.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:37.256740+0000) 2022-09-10T07:11:38.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:37.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:37.882188+0000) 2022-09-10T07:11:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:37 smithi104 ceph-mon[97044]: pgmap v5198: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:38.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:37 smithi110 ceph-mon[97633]: pgmap v5198: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:38.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:38.256980+0000) 2022-09-10T07:11:39.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:38.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:38.882416+0000) 2022-09-10T07:11:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:39.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:39.257207+0000) 2022-09-10T07:11:40.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:39.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:39.882643+0000) 2022-09-10T07:11:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:39 smithi104 ceph-mon[97044]: pgmap v5199: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:39 smithi110 ceph-mon[97633]: pgmap v5199: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:40.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:40.257423+0000) 2022-09-10T07:11:41.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:40.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:40.882851+0000) 2022-09-10T07:11:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:41.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:41.257652+0000) 2022-09-10T07:11:42.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:41.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:41.883068+0000) 2022-09-10T07:11:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:41 smithi104 ceph-mon[97044]: pgmap v5200: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T07:11:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:41 smithi110 ceph-mon[97633]: pgmap v5200: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T07:11:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:42.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:42.257879+0000) 2022-09-10T07:11:43.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:42.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:42.883248+0000) 2022-09-10T07:11:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:43.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:43.258142+0000) 2022-09-10T07:11:44.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:43.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:43.883421+0000) 2022-09-10T07:11:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:43 smithi104 ceph-mon[97044]: pgmap v5201: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:43 smithi110 ceph-mon[97633]: pgmap v5201: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:44.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:44.258352+0000) 2022-09-10T07:11:45.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:44.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:44.883689+0000) 2022-09-10T07:11:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:11:45.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:11:45.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:11:45.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:11:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:11:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:11:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:11:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:11:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:45.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:45.258627+0000) 2022-09-10T07:11:45.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:11:45.554+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:11:46.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:45.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:45.883854+0000) 2022-09-10T07:11:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:45 smithi104 ceph-mon[97044]: pgmap v5202: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:45 smithi110 ceph-mon[97633]: pgmap v5202: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:46.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:46.258895+0000) 2022-09-10T07:11:47.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:46.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:46.884080+0000) 2022-09-10T07:11:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:47.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:47.259194+0000) 2022-09-10T07:11:48.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:47.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:47.884353+0000) 2022-09-10T07:11:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:47 smithi104 ceph-mon[97044]: pgmap v5203: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:47 smithi110 ceph-mon[97633]: pgmap v5203: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:48.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:48.259400+0000) 2022-09-10T07:11:49.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:48.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:48.884506+0000) 2022-09-10T07:11:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:49.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:49.259579+0000) 2022-09-10T07:11:50.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:49.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:49.884706+0000) 2022-09-10T07:11:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:49 smithi104 ceph-mon[97044]: pgmap v5204: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:49 smithi110 ceph-mon[97633]: pgmap v5204: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:11:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:50.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:50.259798+0000) 2022-09-10T07:11:51.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:50.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:50.884939+0000) 2022-09-10T07:11:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:51.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:51.260023+0000) 2022-09-10T07:11:52.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:51.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:51.885193+0000) 2022-09-10T07:11:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:51 smithi104 ceph-mon[97044]: pgmap v5205: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:51 smithi110 ceph-mon[97633]: pgmap v5205: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:52.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:52.260252+0000) 2022-09-10T07:11:53.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:52.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:52.885445+0000) 2022-09-10T07:11:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:53.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:53.260508+0000) 2022-09-10T07:11:54.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:53.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:53.885605+0000) 2022-09-10T07:11:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:53 smithi104 ceph-mon[97044]: pgmap v5206: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:53 smithi110 ceph-mon[97633]: pgmap v5206: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:54.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:54.260688+0000) 2022-09-10T07:11:55.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:54.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:54.885809+0000) 2022-09-10T07:11:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:55.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:55.260915+0000) 2022-09-10T07:11:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:55 smithi104 ceph-mon[97044]: pgmap v5207: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:55 smithi110 ceph-mon[97633]: pgmap v5207: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:11:56.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:55.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:55.886036+0000) 2022-09-10T07:11:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:56.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:56.261162+0000) 2022-09-10T07:11:57.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:56.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:56.886214+0000) 2022-09-10T07:11:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:57.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:57.261356+0000) 2022-09-10T07:11:58.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:57.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:57.886444+0000) 2022-09-10T07:11:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:57 smithi104 ceph-mon[97044]: pgmap v5208: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:57 smithi110 ceph-mon[97633]: pgmap v5208: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:11:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:58.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:58.261567+0000) 2022-09-10T07:11:59.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:58.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:58.886695+0000) 2022-09-10T07:11:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:11:59.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:59.261764+0000) 2022-09-10T07:12:00.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:11:59.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:11:59.886966+0000) 2022-09-10T07:12:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:11:59 smithi104 ceph-mon[97044]: pgmap v5209: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:11:59 smithi110 ceph-mon[97633]: pgmap v5209: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:00.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:00.261903+0000) 2022-09-10T07:12:00.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:12:00.570+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:12:01.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:00.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:00.887239+0000) 2022-09-10T07:12:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:01.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:01.262086+0000) 2022-09-10T07:12:02.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:01.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:01.887478+0000) 2022-09-10T07:12:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:01 smithi104 ceph-mon[97044]: pgmap v5210: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:01 smithi110 ceph-mon[97633]: pgmap v5210: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:02.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:02.262311+0000) 2022-09-10T07:12:03.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:02.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:02.887684+0000) 2022-09-10T07:12:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:03.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:03.262442+0000) 2022-09-10T07:12:04.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:03.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:03.887941+0000) 2022-09-10T07:12:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:03 smithi104 ceph-mon[97044]: pgmap v5211: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:03 smithi110 ceph-mon[97633]: pgmap v5211: 129 pgs: 129 active+clean; 8.2 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:04.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:04.262639+0000) 2022-09-10T07:12:05.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:04.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:04.888194+0000) 2022-09-10T07:12:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:05.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:05.262846+0000) 2022-09-10T07:12:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:05 smithi104 ceph-mon[97044]: pgmap v5212: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:12:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:05 smithi110 ceph-mon[97633]: pgmap v5212: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:12:06.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:05.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:05.888448+0000) 2022-09-10T07:12:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:06.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:06.263080+0000) 2022-09-10T07:12:07.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:06.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:06.888696+0000) 2022-09-10T07:12:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:07.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:07.263391+0000) 2022-09-10T07:12:08.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:07.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:07.888834+0000) 2022-09-10T07:12:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:07 smithi104 ceph-mon[97044]: pgmap v5213: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:12:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:07 smithi110 ceph-mon[97633]: pgmap v5213: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:12:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:08.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:08.263666+0000) 2022-09-10T07:12:09.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:08.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:08.889041+0000) 2022-09-10T07:12:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:08 smithi104 ceph-mon[97044]: osdmap e406: 12 total, 12 up, 12 in 2022-09-10T07:12:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:08 smithi110 ceph-mon[97633]: osdmap e406: 12 total, 12 up, 12 in 2022-09-10T07:12:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:09.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:09.263833+0000) 2022-09-10T07:12:10.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:09.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:09.889308+0000) 2022-09-10T07:12:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:09 smithi104 ceph-mon[97044]: pgmap v5215: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:12:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:09 smithi110 ceph-mon[97633]: pgmap v5215: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:12:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:10.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:10.264073+0000) 2022-09-10T07:12:11.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:10.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:10.889444+0000) 2022-09-10T07:12:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:11.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:11.264375+0000) 2022-09-10T07:12:12.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:11.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:11.889635+0000) 2022-09-10T07:12:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:11 smithi104 ceph-mon[97044]: pgmap v5216: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:12:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:11 smithi110 ceph-mon[97633]: pgmap v5216: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:12:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:12.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:12.264638+0000) 2022-09-10T07:12:13.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:12.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:12.889805+0000) 2022-09-10T07:12:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:13.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:13.264908+0000) 2022-09-10T07:12:14.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:13.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:13.890035+0000) 2022-09-10T07:12:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:13 smithi104 ceph-mon[97044]: pgmap v5217: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:12:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:13 smithi110 ceph-mon[97633]: pgmap v5217: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:12:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:14.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:14.265195+0000) 2022-09-10T07:12:15.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:14.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:14.890188+0000) 2022-09-10T07:12:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:15.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:15.265410+0000) 2022-09-10T07:12:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:15 smithi104 ceph-mon[97044]: osdmap e407: 12 total, 12 up, 12 in 2022-09-10T07:12:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:15 smithi104 ceph-mon[97044]: pgmap v5219: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:12:15.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:12:15.585+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:12:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:15 smithi110 ceph-mon[97633]: osdmap e407: 12 total, 12 up, 12 in 2022-09-10T07:12:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:15 smithi110 ceph-mon[97633]: pgmap v5219: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:12:16.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:15.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:15.890357+0000) 2022-09-10T07:12:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:16.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:16.265545+0000) 2022-09-10T07:12:17.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:16.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:16.890585+0000) 2022-09-10T07:12:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:17.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:17.265739+0000) 2022-09-10T07:12:18.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:17.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:17.890748+0000) 2022-09-10T07:12:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:17 smithi104 ceph-mon[97044]: pgmap v5220: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:12:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:17 smithi110 ceph-mon[97633]: pgmap v5220: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:12:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:18.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:18.265975+0000) 2022-09-10T07:12:19.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:18.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:18.890951+0000) 2022-09-10T07:12:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:19.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:19.266141+0000) 2022-09-10T07:12:20.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:19.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:19.891209+0000) 2022-09-10T07:12:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:19 smithi104 ceph-mon[97044]: pgmap v5221: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:12:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:19 smithi110 ceph-mon[97633]: pgmap v5221: 129 pgs: 129 active+clean; 8.2 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:12:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:20.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:20.266369+0000) 2022-09-10T07:12:21.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:20.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:20.891442+0000) 2022-09-10T07:12:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:21.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:21.266610+0000) 2022-09-10T07:12:22.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:21.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:21.891645+0000) 2022-09-10T07:12:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:21 smithi104 ceph-mon[97044]: pgmap v5222: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:12:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:21 smithi110 ceph-mon[97633]: pgmap v5222: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:12:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:22.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:22.266854+0000) 2022-09-10T07:12:23.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:22.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:22.891896+0000) 2022-09-10T07:12:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:23.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:23.267137+0000) 2022-09-10T07:12:24.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:23.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:23.892172+0000) 2022-09-10T07:12:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:23 smithi104 ceph-mon[97044]: pgmap v5223: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:12:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:23 smithi110 ceph-mon[97633]: pgmap v5223: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:12:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:24.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:24.267394+0000) 2022-09-10T07:12:25.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:24.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:24.892336+0000) 2022-09-10T07:12:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:25.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:25.267661+0000) 2022-09-10T07:12:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:25 smithi110 ceph-mon[97633]: pgmap v5224: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:12:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:25 smithi104 ceph-mon[97044]: pgmap v5224: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:12:26.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:25.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:25.892464+0000) 2022-09-10T07:12:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:26.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:26.267915+0000) 2022-09-10T07:12:27.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:26.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:26.892644+0000) 2022-09-10T07:12:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:27.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:27.268159+0000) 2022-09-10T07:12:28.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:27.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:27.892891+0000) 2022-09-10T07:12:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:27 smithi104 ceph-mon[97044]: pgmap v5225: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:27 smithi110 ceph-mon[97633]: pgmap v5225: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:28.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:28.268386+0000) 2022-09-10T07:12:29.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:28.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:28.893159+0000) 2022-09-10T07:12:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:29.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:29.268615+0000) 2022-09-10T07:12:30.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:29.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:29.893383+0000) 2022-09-10T07:12:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:29 smithi104 ceph-mon[97044]: pgmap v5226: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:29 smithi110 ceph-mon[97633]: pgmap v5226: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:30.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:30.268804+0000) 2022-09-10T07:12:30.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:12:30.601+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:12:31.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:30.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:30.893661+0000) 2022-09-10T07:12:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:31.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:31.269075+0000) 2022-09-10T07:12:32.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:31.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:31.893864+0000) 2022-09-10T07:12:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:31 smithi104 ceph-mon[97044]: pgmap v5227: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:31 smithi110 ceph-mon[97633]: pgmap v5227: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:32.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:32.269391+0000) 2022-09-10T07:12:33.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:32.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:32.894023+0000) 2022-09-10T07:12:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:33.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:33.269656+0000) 2022-09-10T07:12:34.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:33.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:33.894276+0000) 2022-09-10T07:12:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:33 smithi104 ceph-mon[97044]: pgmap v5228: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:33 smithi110 ceph-mon[97633]: pgmap v5228: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:34.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:34.269919+0000) 2022-09-10T07:12:35.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:34.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:34.894501+0000) 2022-09-10T07:12:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:35.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:35.270132+0000) 2022-09-10T07:12:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:35 smithi110 ceph-mon[97633]: pgmap v5229: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:35 smithi104 ceph-mon[97044]: pgmap v5229: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:36.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:35.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:35.894685+0000) 2022-09-10T07:12:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:36.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:36.270386+0000) 2022-09-10T07:12:37.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:36.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:36.894924+0000) 2022-09-10T07:12:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:37.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:37.270625+0000) 2022-09-10T07:12:38.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:37.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:37.895149+0000) 2022-09-10T07:12:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:37 smithi104 ceph-mon[97044]: pgmap v5230: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:37 smithi110 ceph-mon[97633]: pgmap v5230: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:38.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:38.270838+0000) 2022-09-10T07:12:39.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:38.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:38.895325+0000) 2022-09-10T07:12:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:39.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:39.271085+0000) 2022-09-10T07:12:40.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:39.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:39.895512+0000) 2022-09-10T07:12:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:39 smithi104 ceph-mon[97044]: pgmap v5231: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:39 smithi110 ceph-mon[97633]: pgmap v5231: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:40.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:40.271311+0000) 2022-09-10T07:12:41.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:40.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:40.895786+0000) 2022-09-10T07:12:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:41.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:41.271448+0000) 2022-09-10T07:12:42.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:41.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:41.896015+0000) 2022-09-10T07:12:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:41 smithi104 ceph-mon[97044]: pgmap v5232: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:41 smithi110 ceph-mon[97633]: pgmap v5232: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:42.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:42.271621+0000) 2022-09-10T07:12:43.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:42.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:42.896200+0000) 2022-09-10T07:12:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:43.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:43.271802+0000) 2022-09-10T07:12:44.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:43.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:43.896372+0000) 2022-09-10T07:12:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:43 smithi104 ceph-mon[97044]: pgmap v5233: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:43 smithi110 ceph-mon[97633]: pgmap v5233: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:44.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:44.272049+0000) 2022-09-10T07:12:45.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:44.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:44.896652+0000) 2022-09-10T07:12:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:12:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:12:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:12:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:12:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:12:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:12:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:12:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:12:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:45.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:45.272196+0000) 2022-09-10T07:12:45.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:12:45.615+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:12:46.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:45.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:45.896888+0000) 2022-09-10T07:12:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:45 smithi104 ceph-mon[97044]: pgmap v5234: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:45 smithi110 ceph-mon[97633]: pgmap v5234: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:46.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:46.272456+0000) 2022-09-10T07:12:47.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:46.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:46.897174+0000) 2022-09-10T07:12:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:47.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:47.272687+0000) 2022-09-10T07:12:48.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:47.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:47.897344+0000) 2022-09-10T07:12:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:47 smithi104 ceph-mon[97044]: pgmap v5235: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:47 smithi110 ceph-mon[97633]: pgmap v5235: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:48.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:48.272900+0000) 2022-09-10T07:12:49.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:48.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:48.897550+0000) 2022-09-10T07:12:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:49.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:49.273082+0000) 2022-09-10T07:12:50.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:49.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:49.897803+0000) 2022-09-10T07:12:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:49 smithi104 ceph-mon[97044]: pgmap v5236: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:49 smithi110 ceph-mon[97633]: pgmap v5236: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:50.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:50.273318+0000) 2022-09-10T07:12:51.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:50.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:50.897993+0000) 2022-09-10T07:12:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:51.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:51.273555+0000) 2022-09-10T07:12:52.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:51.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:51.898230+0000) 2022-09-10T07:12:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:51 smithi104 ceph-mon[97044]: pgmap v5237: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:51 smithi110 ceph-mon[97633]: pgmap v5237: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:52.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:52.273811+0000) 2022-09-10T07:12:53.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:52.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:52.898405+0000) 2022-09-10T07:12:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:53.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:53.274050+0000) 2022-09-10T07:12:54.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:53.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:53.898588+0000) 2022-09-10T07:12:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:53 smithi104 ceph-mon[97044]: pgmap v5238: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:53 smithi110 ceph-mon[97633]: pgmap v5238: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:54.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:54.274258+0000) 2022-09-10T07:12:55.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:54.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:54.898820+0000) 2022-09-10T07:12:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:55.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:55.274553+0000) 2022-09-10T07:12:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:55 smithi110 ceph-mon[97633]: pgmap v5239: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:55 smithi104 ceph-mon[97044]: pgmap v5239: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:12:56.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:55.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:55.899042+0000) 2022-09-10T07:12:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:56.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:56.274782+0000) 2022-09-10T07:12:57.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:56.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:56.899266+0000) 2022-09-10T07:12:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:57.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:57.274919+0000) 2022-09-10T07:12:58.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:57.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:57.899479+0000) 2022-09-10T07:12:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:57 smithi104 ceph-mon[97044]: pgmap v5240: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:57 smithi110 ceph-mon[97633]: pgmap v5240: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:12:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:58.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:58.275187+0000) 2022-09-10T07:12:59.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:58.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:58.899692+0000) 2022-09-10T07:12:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:12:59.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:59.275428+0000) 2022-09-10T07:13:00.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:12:59.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:12:59.899879+0000) 2022-09-10T07:13:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:12:59 smithi104 ceph-mon[97044]: pgmap v5241: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:12:59 smithi110 ceph-mon[97633]: pgmap v5241: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:00.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:00.275699+0000) 2022-09-10T07:13:00.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:13:00.627+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:13:01.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:00.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:00.900064+0000) 2022-09-10T07:13:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:01.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:01.275927+0000) 2022-09-10T07:13:02.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:01.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:01.900239+0000) 2022-09-10T07:13:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:01 smithi104 ceph-mon[97044]: pgmap v5242: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:01 smithi110 ceph-mon[97633]: pgmap v5242: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:02.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:02.276196+0000) 2022-09-10T07:13:03.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:02.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:02.900450+0000) 2022-09-10T07:13:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:03.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:03.276390+0000) 2022-09-10T07:13:04.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:03.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:03.900665+0000) 2022-09-10T07:13:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:03 smithi104 ceph-mon[97044]: pgmap v5243: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:03 smithi110 ceph-mon[97633]: pgmap v5243: 129 pgs: 129 active+clean; 8.2 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:04.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:04.276619+0000) 2022-09-10T07:13:05.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:04.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:04.900796+0000) 2022-09-10T07:13:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:05.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:05.276794+0000) 2022-09-10T07:13:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:05 smithi110 ceph-mon[97633]: pgmap v5244: 129 pgs: 129 active+clean; 8.2 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:13:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:05 smithi104 ceph-mon[97044]: pgmap v5244: 129 pgs: 129 active+clean; 8.2 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:13:06.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:05.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:05.901026+0000) 2022-09-10T07:13:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:06.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:06.277039+0000) 2022-09-10T07:13:07.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:06.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:06.901195+0000) 2022-09-10T07:13:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:07.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:07.277212+0000) 2022-09-10T07:13:08.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:07.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:07.901352+0000) 2022-09-10T07:13:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:07 smithi104 ceph-mon[97044]: pgmap v5245: 129 pgs: 129 active+clean; 8.2 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:13:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:07 smithi110 ceph-mon[97633]: pgmap v5245: 129 pgs: 129 active+clean; 8.2 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:13:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:08.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:08.277467+0000) 2022-09-10T07:13:09.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:08.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:08.901514+0000) 2022-09-10T07:13:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:08 smithi104 ceph-mon[97044]: osdmap e408: 12 total, 12 up, 12 in 2022-09-10T07:13:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:08 smithi110 ceph-mon[97633]: osdmap e408: 12 total, 12 up, 12 in 2022-09-10T07:13:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:09.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:09.277682+0000) 2022-09-10T07:13:10.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:09.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:09.901683+0000) 2022-09-10T07:13:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:09 smithi104 ceph-mon[97044]: pgmap v5247: 129 pgs: 129 active+clean; 8.2 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:13:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:09 smithi110 ceph-mon[97633]: pgmap v5247: 129 pgs: 129 active+clean; 8.2 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:13:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:10.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:10.277964+0000) 2022-09-10T07:13:11.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:10.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:10.901972+0000) 2022-09-10T07:13:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:11.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:11.278191+0000) 2022-09-10T07:13:12.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:11.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:11.902210+0000) 2022-09-10T07:13:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:11 smithi104 ceph-mon[97044]: pgmap v5248: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:13:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:11 smithi110 ceph-mon[97633]: pgmap v5248: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:13:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:12.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:12.278446+0000) 2022-09-10T07:13:13.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:12.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:12.902486+0000) 2022-09-10T07:13:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:13.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:13.278624+0000) 2022-09-10T07:13:14.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:13.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:13.902707+0000) 2022-09-10T07:13:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:13 smithi104 ceph-mon[97044]: pgmap v5249: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:13:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:13 smithi110 ceph-mon[97633]: pgmap v5249: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:13:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:14.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:14.278865+0000) 2022-09-10T07:13:15.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:14.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:14.902848+0000) 2022-09-10T07:13:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:15.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:15.279184+0000) 2022-09-10T07:13:15.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:13:15.634+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:15 smithi110 ceph-mon[97633]: osdmap e409: 12 total, 12 up, 12 in 2022-09-10T07:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:15 smithi110 ceph-mon[97633]: pgmap v5251: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:13:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:15 smithi104 ceph-mon[97044]: osdmap e409: 12 total, 12 up, 12 in 2022-09-10T07:13:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:15 smithi104 ceph-mon[97044]: pgmap v5251: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:13:16.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:15.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:15.903058+0000) 2022-09-10T07:13:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:16.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:16.279410+0000) 2022-09-10T07:13:17.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:16.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:16.903196+0000) 2022-09-10T07:13:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:17.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:17.279565+0000) 2022-09-10T07:13:18.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:17.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:17.903406+0000) 2022-09-10T07:13:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:17 smithi104 ceph-mon[97044]: pgmap v5252: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:13:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:17 smithi110 ceph-mon[97633]: pgmap v5252: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:13:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:18.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:18.279834+0000) 2022-09-10T07:13:19.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:18.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:18.903631+0000) 2022-09-10T07:13:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:19.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:19.280131+0000) 2022-09-10T07:13:20.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:19.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:19.903829+0000) 2022-09-10T07:13:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:19 smithi104 ceph-mon[97044]: pgmap v5253: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:13:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:19 smithi110 ceph-mon[97633]: pgmap v5253: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:13:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:20.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:20.280393+0000) 2022-09-10T07:13:21.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:20.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:20.903996+0000) 2022-09-10T07:13:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:21.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:21.280605+0000) 2022-09-10T07:13:22.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:21.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:21.904229+0000) 2022-09-10T07:13:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:21 smithi104 ceph-mon[97044]: pgmap v5254: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:13:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:21 smithi110 ceph-mon[97633]: pgmap v5254: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:13:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:22.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:22.280836+0000) 2022-09-10T07:13:23.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:22.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:22.904390+0000) 2022-09-10T07:13:23.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:23.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:23.281062+0000) 2022-09-10T07:13:24.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:23.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:23.904676+0000) 2022-09-10T07:13:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:23 smithi104 ceph-mon[97044]: pgmap v5255: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:13:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:23 smithi110 ceph-mon[97633]: pgmap v5255: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:13:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:24.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:24.281323+0000) 2022-09-10T07:13:25.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:24.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:24.904875+0000) 2022-09-10T07:13:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:25.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:25.281544+0000) 2022-09-10T07:13:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:25 smithi110 ceph-mon[97633]: pgmap v5256: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:13:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:25 smithi104 ceph-mon[97044]: pgmap v5256: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:13:26.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:25.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:25.905046+0000) 2022-09-10T07:13:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:26.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:26.281775+0000) 2022-09-10T07:13:27.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:26.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:26.905308+0000) 2022-09-10T07:13:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:27.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:27.282026+0000) 2022-09-10T07:13:28.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:27.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:27.905522+0000) 2022-09-10T07:13:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:27 smithi104 ceph-mon[97044]: pgmap v5257: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:27 smithi110 ceph-mon[97633]: pgmap v5257: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:28.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:28.282268+0000) 2022-09-10T07:13:29.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:28.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:28.905717+0000) 2022-09-10T07:13:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:29.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:29.282485+0000) 2022-09-10T07:13:30.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:29.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:29.905925+0000) 2022-09-10T07:13:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:29 smithi104 ceph-mon[97044]: pgmap v5258: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:29 smithi110 ceph-mon[97633]: pgmap v5258: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:30.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:30.282707+0000) 2022-09-10T07:13:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:13:30.646+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:13:31.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:30.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:30.906227+0000) 2022-09-10T07:13:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:31.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:31.282962+0000) 2022-09-10T07:13:32.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:31.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:31.906481+0000) 2022-09-10T07:13:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:31 smithi104 ceph-mon[97044]: pgmap v5259: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:31 smithi110 ceph-mon[97633]: pgmap v5259: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:32.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:32.283227+0000) 2022-09-10T07:13:33.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:32.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:32.906743+0000) 2022-09-10T07:13:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:33.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:33.283468+0000) 2022-09-10T07:13:34.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:33.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:33.907007+0000) 2022-09-10T07:13:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:33 smithi104 ceph-mon[97044]: pgmap v5260: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:33 smithi110 ceph-mon[97633]: pgmap v5260: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:34.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:34.283697+0000) 2022-09-10T07:13:35.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:34.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:34.907257+0000) 2022-09-10T07:13:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:35.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:35.283973+0000) 2022-09-10T07:13:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:35 smithi110 ceph-mon[97633]: pgmap v5261: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:35 smithi104 ceph-mon[97044]: pgmap v5261: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:36.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:35.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:35.907436+0000) 2022-09-10T07:13:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:36.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:36.284191+0000) 2022-09-10T07:13:37.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:36.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:36.907617+0000) 2022-09-10T07:13:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:37.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:37.284370+0000) 2022-09-10T07:13:38.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:37.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:37.907781+0000) 2022-09-10T07:13:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:37 smithi104 ceph-mon[97044]: pgmap v5262: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:37 smithi110 ceph-mon[97633]: pgmap v5262: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:38.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:38.284617+0000) 2022-09-10T07:13:39.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:38.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:38.907989+0000) 2022-09-10T07:13:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:39.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:39.284843+0000) 2022-09-10T07:13:40.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:39.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:39.908192+0000) 2022-09-10T07:13:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:39 smithi104 ceph-mon[97044]: pgmap v5263: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:39 smithi110 ceph-mon[97633]: pgmap v5263: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:40.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:40.285141+0000) 2022-09-10T07:13:41.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:40.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:40.908380+0000) 2022-09-10T07:13:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:41.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:41.285322+0000) 2022-09-10T07:13:42.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:41.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:41.908536+0000) 2022-09-10T07:13:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:41 smithi104 ceph-mon[97044]: pgmap v5264: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:41 smithi110 ceph-mon[97633]: pgmap v5264: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:42.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:42.285557+0000) 2022-09-10T07:13:43.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:42.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:42.908791+0000) 2022-09-10T07:13:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:43.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:43.285754+0000) 2022-09-10T07:13:44.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:43.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:43.908992+0000) 2022-09-10T07:13:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:43 smithi104 ceph-mon[97044]: pgmap v5265: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:43 smithi110 ceph-mon[97633]: pgmap v5265: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:44.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:44.285931+0000) 2022-09-10T07:13:45.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:44.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:44.909217+0000) 2022-09-10T07:13:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:13:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:13:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:13:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:13:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:13:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:13:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:13:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:13:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:45.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:45.286198+0000) 2022-09-10T07:13:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:45.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:45.909410+0000) 2022-09-10T07:13:46.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:13:45.662+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:13:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:45 smithi104 ceph-mon[97044]: pgmap v5266: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:45 smithi110 ceph-mon[97633]: pgmap v5266: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:46.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:46.286427+0000) 2022-09-10T07:13:47.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:46.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:46.909592+0000) 2022-09-10T07:13:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:47.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:47.286671+0000) 2022-09-10T07:13:48.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:47.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:47.909793+0000) 2022-09-10T07:13:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:47 smithi104 ceph-mon[97044]: pgmap v5267: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:47 smithi110 ceph-mon[97633]: pgmap v5267: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:48.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:48.286874+0000) 2022-09-10T07:13:49.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:48.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:48.910025+0000) 2022-09-10T07:13:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:49.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:49.287154+0000) 2022-09-10T07:13:50.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:49.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:49.910229+0000) 2022-09-10T07:13:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:49 smithi104 ceph-mon[97044]: pgmap v5268: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:49 smithi110 ceph-mon[97633]: pgmap v5268: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:50.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:50.287327+0000) 2022-09-10T07:13:51.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:50.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:50.910391+0000) 2022-09-10T07:13:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:51.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:51.287490+0000) 2022-09-10T07:13:52.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:51.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:51.910644+0000) 2022-09-10T07:13:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:51 smithi104 ceph-mon[97044]: pgmap v5269: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:51 smithi110 ceph-mon[97633]: pgmap v5269: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:52.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:52.287677+0000) 2022-09-10T07:13:53.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:52.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:52.910905+0000) 2022-09-10T07:13:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:53.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:53.287869+0000) 2022-09-10T07:13:54.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:53.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:53.911169+0000) 2022-09-10T07:13:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:53 smithi104 ceph-mon[97044]: pgmap v5270: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:53 smithi110 ceph-mon[97633]: pgmap v5270: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:54.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:54.288028+0000) 2022-09-10T07:13:55.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:54.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:54.911409+0000) 2022-09-10T07:13:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:55.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:55.288229+0000) 2022-09-10T07:13:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:55 smithi110 ceph-mon[97633]: pgmap v5271: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:55 smithi104 ceph-mon[97044]: pgmap v5271: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:13:56.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:55.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:55.911605+0000) 2022-09-10T07:13:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:56.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:56.288458+0000) 2022-09-10T07:13:57.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:56.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:56.911790+0000) 2022-09-10T07:13:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:57.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:57.288656+0000) 2022-09-10T07:13:58.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:57.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:57.911974+0000) 2022-09-10T07:13:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:57 smithi104 ceph-mon[97044]: pgmap v5272: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:57 smithi110 ceph-mon[97633]: pgmap v5272: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:13:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:58.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:58.288877+0000) 2022-09-10T07:13:59.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:58.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:58.912201+0000) 2022-09-10T07:13:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:13:59.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:59.289169+0000) 2022-09-10T07:14:00.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:13:59.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:13:59.912387+0000) 2022-09-10T07:14:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:13:59 smithi104 ceph-mon[97044]: pgmap v5273: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:13:59 smithi110 ceph-mon[97633]: pgmap v5273: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:00.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:00.289412+0000) 2022-09-10T07:14:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:14:00.678+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:14:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:00.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:00.912526+0000) 2022-09-10T07:14:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:01.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:01.289598+0000) 2022-09-10T07:14:02.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:01.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:01.912757+0000) 2022-09-10T07:14:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:01 smithi104 ceph-mon[97044]: pgmap v5274: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:01 smithi110 ceph-mon[97633]: pgmap v5274: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:02.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:02.289809+0000) 2022-09-10T07:14:03.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:02.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:02.912968+0000) 2022-09-10T07:14:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:03.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:03.290051+0000) 2022-09-10T07:14:04.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:03.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:03.913167+0000) 2022-09-10T07:14:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:03 smithi104 ceph-mon[97044]: pgmap v5275: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:03 smithi110 ceph-mon[97633]: pgmap v5275: 129 pgs: 129 active+clean; 8.2 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:04.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:04.290272+0000) 2022-09-10T07:14:05.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:04.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:04.913368+0000) 2022-09-10T07:14:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:05.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:05.290483+0000) 2022-09-10T07:14:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:05 smithi110 ceph-mon[97633]: pgmap v5276: 129 pgs: 129 active+clean; 8.2 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:14:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:05 smithi104 ceph-mon[97044]: pgmap v5276: 129 pgs: 129 active+clean; 8.2 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:14:06.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:05.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:05.913542+0000) 2022-09-10T07:14:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:06.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:06.290726+0000) 2022-09-10T07:14:07.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:06.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:06.913700+0000) 2022-09-10T07:14:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:07.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:07.290919+0000) 2022-09-10T07:14:08.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:07.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:07.913952+0000) 2022-09-10T07:14:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:07 smithi104 ceph-mon[97044]: pgmap v5277: 129 pgs: 129 active+clean; 8.2 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:14:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:07 smithi110 ceph-mon[97633]: pgmap v5277: 129 pgs: 129 active+clean; 8.2 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:14:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:08.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:08.291162+0000) 2022-09-10T07:14:09.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:08.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:08.914201+0000) 2022-09-10T07:14:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:08 smithi104 ceph-mon[97044]: osdmap e410: 12 total, 12 up, 12 in 2022-09-10T07:14:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:08 smithi110 ceph-mon[97633]: osdmap e410: 12 total, 12 up, 12 in 2022-09-10T07:14:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:09.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:09.291413+0000) 2022-09-10T07:14:10.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:09.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:09.914450+0000) 2022-09-10T07:14:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:09 smithi104 ceph-mon[97044]: pgmap v5279: 129 pgs: 129 active+clean; 8.2 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:14:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:09 smithi110 ceph-mon[97633]: pgmap v5279: 129 pgs: 129 active+clean; 8.2 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:14:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:10.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:10.291647+0000) 2022-09-10T07:14:11.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:10.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:10.914714+0000) 2022-09-10T07:14:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:11.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:11.291849+0000) 2022-09-10T07:14:12.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:11.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:11.914944+0000) 2022-09-10T07:14:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:11 smithi104 ceph-mon[97044]: pgmap v5280: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:14:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:11 smithi110 ceph-mon[97633]: pgmap v5280: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:14:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:12.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:12.292043+0000) 2022-09-10T07:14:13.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:12.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:12.915174+0000) 2022-09-10T07:14:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:13.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:13.292311+0000) 2022-09-10T07:14:14.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:13.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:13.915381+0000) 2022-09-10T07:14:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:13 smithi104 ceph-mon[97044]: pgmap v5281: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:14:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:13 smithi110 ceph-mon[97633]: pgmap v5281: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:14:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:14.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:14.292524+0000) 2022-09-10T07:14:15.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:14.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:14.915579+0000) 2022-09-10T07:14:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:15.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:15.292790+0000) 2022-09-10T07:14:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:15 smithi110 ceph-mon[97633]: osdmap e411: 12 total, 12 up, 12 in 2022-09-10T07:14:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:15 smithi110 ceph-mon[97633]: pgmap v5283: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:14:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:15 smithi104 ceph-mon[97044]: osdmap e411: 12 total, 12 up, 12 in 2022-09-10T07:14:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:15 smithi104 ceph-mon[97044]: pgmap v5283: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:14:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:15.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:15.915817+0000) 2022-09-10T07:14:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:14:15.693+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:14:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:16.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:16.292998+0000) 2022-09-10T07:14:17.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:16.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:16.916007+0000) 2022-09-10T07:14:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:17.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:17.293230+0000) 2022-09-10T07:14:18.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:17.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:17.916193+0000) 2022-09-10T07:14:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:17 smithi104 ceph-mon[97044]: pgmap v5284: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:14:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:17 smithi110 ceph-mon[97633]: pgmap v5284: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:14:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:18.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:18.293445+0000) 2022-09-10T07:14:19.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:18.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:18.916446+0000) 2022-09-10T07:14:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:19.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:19.293714+0000) 2022-09-10T07:14:20.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:19.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:19.916659+0000) 2022-09-10T07:14:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:19 smithi104 ceph-mon[97044]: pgmap v5285: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:14:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:19 smithi110 ceph-mon[97633]: pgmap v5285: 129 pgs: 129 active+clean; 8.2 MiB data, 618 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:14:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:20.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:20.293892+0000) 2022-09-10T07:14:21.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:20.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:20.916871+0000) 2022-09-10T07:14:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:21.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:21.294146+0000) 2022-09-10T07:14:22.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:21.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:21.917171+0000) 2022-09-10T07:14:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:21 smithi104 ceph-mon[97044]: pgmap v5286: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:14:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:21 smithi110 ceph-mon[97633]: pgmap v5286: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:14:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:22.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:22.294335+0000) 2022-09-10T07:14:23.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:22.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:22.917436+0000) 2022-09-10T07:14:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:23.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:23.294589+0000) 2022-09-10T07:14:24.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:23.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:23.917578+0000) 2022-09-10T07:14:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:23 smithi104 ceph-mon[97044]: pgmap v5287: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:14:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:23 smithi110 ceph-mon[97633]: pgmap v5287: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:14:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:24.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:24.294804+0000) 2022-09-10T07:14:25.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:24.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:24.917806+0000) 2022-09-10T07:14:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:25.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:25.295047+0000) 2022-09-10T07:14:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:25 smithi110 ceph-mon[97633]: pgmap v5288: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:14:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:25 smithi104 ceph-mon[97044]: pgmap v5288: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:14:26.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:25.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:25.918054+0000) 2022-09-10T07:14:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:26.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:26.295359+0000) 2022-09-10T07:14:27.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:26.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:26.918266+0000) 2022-09-10T07:14:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:27.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:27.295624+0000) 2022-09-10T07:14:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:27 smithi104 ceph-mon[97044]: pgmap v5289: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:28.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:27.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:27.918487+0000) 2022-09-10T07:14:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:27 smithi110 ceph-mon[97633]: pgmap v5289: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:28.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:28.295871+0000) 2022-09-10T07:14:29.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:28.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:28.918683+0000) 2022-09-10T07:14:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:29.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:29.296032+0000) 2022-09-10T07:14:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:29 smithi104 ceph-mon[97044]: pgmap v5290: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:30.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:29.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:29.918824+0000) 2022-09-10T07:14:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:29 smithi110 ceph-mon[97633]: pgmap v5290: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:30.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:30.296192+0000) 2022-09-10T07:14:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:14:30.708+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:14:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:30.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:30.919022+0000) 2022-09-10T07:14:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:31.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:31.296412+0000) 2022-09-10T07:14:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:31 smithi104 ceph-mon[97044]: pgmap v5291: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:32.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:31.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:31.919266+0000) 2022-09-10T07:14:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:31 smithi110 ceph-mon[97633]: pgmap v5291: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:32.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:32.296633+0000) 2022-09-10T07:14:33.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:32.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:32.919487+0000) 2022-09-10T07:14:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:33.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:33.296896+0000) 2022-09-10T07:14:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:33 smithi104 ceph-mon[97044]: pgmap v5292: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:34.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:33.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:33.919655+0000) 2022-09-10T07:14:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:33 smithi110 ceph-mon[97633]: pgmap v5292: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:34.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:34.297215+0000) 2022-09-10T07:14:35.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:34.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:34.919856+0000) 2022-09-10T07:14:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:35.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:35.297476+0000) 2022-09-10T07:14:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:35 smithi110 ceph-mon[97633]: pgmap v5293: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:35 smithi104 ceph-mon[97044]: pgmap v5293: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:36.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:35.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:35.920016+0000) 2022-09-10T07:14:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:36.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:36.297747+0000) 2022-09-10T07:14:37.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:36.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:36.920180+0000) 2022-09-10T07:14:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:37.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:37.297910+0000) 2022-09-10T07:14:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:37 smithi104 ceph-mon[97044]: pgmap v5294: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:38.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:37.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:37.920438+0000) 2022-09-10T07:14:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:37 smithi110 ceph-mon[97633]: pgmap v5294: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:38.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:38.298073+0000) 2022-09-10T07:14:39.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:38.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:38.920571+0000) 2022-09-10T07:14:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:39.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:39.298222+0000) 2022-09-10T07:14:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:39 smithi104 ceph-mon[97044]: pgmap v5295: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:40.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:39.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:39.920740+0000) 2022-09-10T07:14:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:39 smithi110 ceph-mon[97633]: pgmap v5295: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:40.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:40.298492+0000) 2022-09-10T07:14:41.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:40.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:40.920916+0000) 2022-09-10T07:14:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:41.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:41.298710+0000) 2022-09-10T07:14:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:41 smithi104 ceph-mon[97044]: pgmap v5296: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:42.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:41.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:41.921147+0000) 2022-09-10T07:14:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:41 smithi110 ceph-mon[97633]: pgmap v5296: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:42.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:42.298952+0000) 2022-09-10T07:14:43.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:42.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:42.921320+0000) 2022-09-10T07:14:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:43.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:43.299192+0000) 2022-09-10T07:14:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:43 smithi104 ceph-mon[97044]: pgmap v5297: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:44.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:43.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:43.921488+0000) 2022-09-10T07:14:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:43 smithi110 ceph-mon[97633]: pgmap v5297: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:44.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:44.299359+0000) 2022-09-10T07:14:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:14:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:14:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:14:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:14:45.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:44.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:44.921672+0000) 2022-09-10T07:14:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:14:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:14:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:14:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:14:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:45.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:45.299515+0000) 2022-09-10T07:14:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:45.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:45.921892+0000) 2022-09-10T07:14:46.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:14:45.711+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:14:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:45 smithi104 ceph-mon[97044]: pgmap v5298: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:45 smithi110 ceph-mon[97633]: pgmap v5298: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:46.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:46.299722+0000) 2022-09-10T07:14:47.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:46.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:46.922147+0000) 2022-09-10T07:14:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:47.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:47.299952+0000) 2022-09-10T07:14:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:47 smithi104 ceph-mon[97044]: pgmap v5299: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:48.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:47.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:47.922325+0000) 2022-09-10T07:14:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:47 smithi110 ceph-mon[97633]: pgmap v5299: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:48.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:48.300227+0000) 2022-09-10T07:14:49.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:48.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:48.922531+0000) 2022-09-10T07:14:49.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:49.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:49.300438+0000) 2022-09-10T07:14:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:49 smithi104 ceph-mon[97044]: pgmap v5300: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:50.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:49.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:49.922684+0000) 2022-09-10T07:14:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:49 smithi110 ceph-mon[97633]: pgmap v5300: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:50.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:50.300589+0000) 2022-09-10T07:14:51.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:50.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:50.922962+0000) 2022-09-10T07:14:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:51.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:51.300808+0000) 2022-09-10T07:14:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:51 smithi104 ceph-mon[97044]: pgmap v5301: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:52.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:51.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:51.923176+0000) 2022-09-10T07:14:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:51 smithi110 ceph-mon[97633]: pgmap v5301: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:52.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:52.301028+0000) 2022-09-10T07:14:53.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:52.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:52.923402+0000) 2022-09-10T07:14:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:53.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:53.301286+0000) 2022-09-10T07:14:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:53 smithi104 ceph-mon[97044]: pgmap v5302: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:54.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:53.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:53.923564+0000) 2022-09-10T07:14:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:53 smithi110 ceph-mon[97633]: pgmap v5302: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:54.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:54.301473+0000) 2022-09-10T07:14:55.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:54.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:54.923772+0000) 2022-09-10T07:14:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:55.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:55.301710+0000) 2022-09-10T07:14:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:55 smithi110 ceph-mon[97633]: pgmap v5303: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:55 smithi104 ceph-mon[97044]: pgmap v5303: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:14:56.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:55.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:55.924042+0000) 2022-09-10T07:14:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:56.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:56.302000+0000) 2022-09-10T07:14:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:56.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:56.924241+0000) 2022-09-10T07:14:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:57.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:57.302196+0000) 2022-09-10T07:14:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:57 smithi104 ceph-mon[97044]: pgmap v5304: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:57.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:57.924512+0000) 2022-09-10T07:14:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:57 smithi110 ceph-mon[97633]: pgmap v5304: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:14:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:58.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:58.302449+0000) 2022-09-10T07:14:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:58.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:58.924690+0000) 2022-09-10T07:14:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:14:59.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:59.302668+0000) 2022-09-10T07:15:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:14:59 smithi104 ceph-mon[97044]: pgmap v5305: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:14:59.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:14:59.924943+0000) 2022-09-10T07:15:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:14:59 smithi110 ceph-mon[97633]: pgmap v5305: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:00.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:00.302888+0000) 2022-09-10T07:15:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:00.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:00.925196+0000) 2022-09-10T07:15:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:15:00.722+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:15:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:01.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:01.303193+0000) 2022-09-10T07:15:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:01 smithi104 ceph-mon[97044]: pgmap v5306: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:01.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:01.925357+0000) 2022-09-10T07:15:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:01 smithi110 ceph-mon[97633]: pgmap v5306: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:02.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:02.303423+0000) 2022-09-10T07:15:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:02.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:02.925625+0000) 2022-09-10T07:15:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:03.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:03.303651+0000) 2022-09-10T07:15:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:03 smithi104 ceph-mon[97044]: pgmap v5307: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:03.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:03.925848+0000) 2022-09-10T07:15:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:03 smithi110 ceph-mon[97633]: pgmap v5307: 129 pgs: 129 active+clean; 8.2 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:04.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:04.304044+0000) 2022-09-10T07:15:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:04.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:04.926066+0000) 2022-09-10T07:15:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:05.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:05.304165+0000) 2022-09-10T07:15:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:05 smithi110 ceph-mon[97633]: pgmap v5308: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:15:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:05 smithi104 ceph-mon[97044]: pgmap v5308: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:15:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:05.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:05.926225+0000) 2022-09-10T07:15:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:06.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:06.304294+0000) 2022-09-10T07:15:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:06.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:06.926407+0000) 2022-09-10T07:15:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:07.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:07.304454+0000) 2022-09-10T07:15:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:07 smithi104 ceph-mon[97044]: pgmap v5309: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:15:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:07.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:07.926645+0000) 2022-09-10T07:15:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:07 smithi110 ceph-mon[97633]: pgmap v5309: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:15:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:08.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:08.304597+0000) 2022-09-10T07:15:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:08 smithi104 ceph-mon[97044]: osdmap e412: 12 total, 12 up, 12 in 2022-09-10T07:15:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:08.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:08.926886+0000) 2022-09-10T07:15:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:08 smithi110 ceph-mon[97633]: osdmap e412: 12 total, 12 up, 12 in 2022-09-10T07:15:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:09.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:09.304812+0000) 2022-09-10T07:15:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:09 smithi104 ceph-mon[97044]: pgmap v5311: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:15:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:09.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:09.927090+0000) 2022-09-10T07:15:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:09 smithi110 ceph-mon[97633]: pgmap v5311: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:15:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:10.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:10.304990+0000) 2022-09-10T07:15:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:10.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:10.927315+0000) 2022-09-10T07:15:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:11.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:11.305197+0000) 2022-09-10T07:15:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:11 smithi104 ceph-mon[97044]: pgmap v5312: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:15:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:11.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:11.927494+0000) 2022-09-10T07:15:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:11 smithi110 ceph-mon[97633]: pgmap v5312: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:15:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:12.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:12.305364+0000) 2022-09-10T07:15:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:12.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:12.927697+0000) 2022-09-10T07:15:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:13.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:13.305528+0000) 2022-09-10T07:15:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:14 smithi104 ceph-mon[97044]: pgmap v5313: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:15:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:13.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:13.927910+0000) 2022-09-10T07:15:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:13 smithi110 ceph-mon[97633]: pgmap v5313: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:15:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:14.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:14.305660+0000) 2022-09-10T07:15:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:14.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:14.928177+0000) 2022-09-10T07:15:15.726 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:15.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:15.305835+0000) 2022-09-10T07:15:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:15 smithi110 ceph-mon[97633]: osdmap e413: 12 total, 12 up, 12 in 2022-09-10T07:15:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:15 smithi110 ceph-mon[97633]: pgmap v5315: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:15:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:15 smithi104 ceph-mon[97044]: osdmap e413: 12 total, 12 up, 12 in 2022-09-10T07:15:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:15 smithi104 ceph-mon[97044]: pgmap v5315: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:15:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:15:15.724+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:15:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:15.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:15.928355+0000) 2022-09-10T07:15:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:16.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:16.306012+0000) 2022-09-10T07:15:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:16.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:16.928610+0000) 2022-09-10T07:15:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:17.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:17.306224+0000) 2022-09-10T07:15:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:17 smithi104 ceph-mon[97044]: pgmap v5316: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:15:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:17.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:17.928835+0000) 2022-09-10T07:15:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:17 smithi110 ceph-mon[97633]: pgmap v5316: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:15:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:18.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:18.306354+0000) 2022-09-10T07:15:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:18.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:18.929069+0000) 2022-09-10T07:15:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:19.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:19.306501+0000) 2022-09-10T07:15:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:19 smithi104 ceph-mon[97044]: pgmap v5317: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:15:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:19.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:19.929337+0000) 2022-09-10T07:15:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:19 smithi110 ceph-mon[97633]: pgmap v5317: 129 pgs: 129 active+clean; 8.2 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:15:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:20.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:20.306630+0000) 2022-09-10T07:15:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:20.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:20.929598+0000) 2022-09-10T07:15:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:21.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:21.306829+0000) 2022-09-10T07:15:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:21 smithi104 ceph-mon[97044]: pgmap v5318: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:15:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:21.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:21.929827+0000) 2022-09-10T07:15:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:21 smithi110 ceph-mon[97633]: pgmap v5318: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:15:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:22.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:22.307035+0000) 2022-09-10T07:15:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:22.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:22.930034+0000) 2022-09-10T07:15:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:23.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:23.307235+0000) 2022-09-10T07:15:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:23 smithi104 ceph-mon[97044]: pgmap v5319: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:15:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:23.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:23.930275+0000) 2022-09-10T07:15:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:23 smithi110 ceph-mon[97633]: pgmap v5319: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:15:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:24.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:24.307411+0000) 2022-09-10T07:15:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:24.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:24.930490+0000) 2022-09-10T07:15:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:25.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:25.307651+0000) 2022-09-10T07:15:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:25 smithi110 ceph-mon[97633]: pgmap v5320: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:15:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:25 smithi104 ceph-mon[97044]: pgmap v5320: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:15:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:25.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:25.930756+0000) 2022-09-10T07:15:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:26.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:26.307878+0000) 2022-09-10T07:15:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:26.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:26.930943+0000) 2022-09-10T07:15:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:27.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:27.308079+0000) 2022-09-10T07:15:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:27 smithi104 ceph-mon[97044]: pgmap v5321: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:27.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:27.931204+0000) 2022-09-10T07:15:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:27 smithi110 ceph-mon[97633]: pgmap v5321: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:28.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:28.308222+0000) 2022-09-10T07:15:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:28.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:28.931395+0000) 2022-09-10T07:15:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:29.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:29.308396+0000) 2022-09-10T07:15:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:29 smithi104 ceph-mon[97044]: pgmap v5322: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:29.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:29.931616+0000) 2022-09-10T07:15:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:29 smithi110 ceph-mon[97633]: pgmap v5322: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:30.734 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:30.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:30.308564+0000) 2022-09-10T07:15:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:30.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:30.931849+0000) 2022-09-10T07:15:31.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:15:30.732+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:15:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:31.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:31.308757+0000) 2022-09-10T07:15:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:31 smithi104 ceph-mon[97044]: pgmap v5323: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:31.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:31.932143+0000) 2022-09-10T07:15:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:31 smithi110 ceph-mon[97633]: pgmap v5323: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:32.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:32.308957+0000) 2022-09-10T07:15:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:32.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:32.932292+0000) 2022-09-10T07:15:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:33.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:33.309180+0000) 2022-09-10T07:15:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:33 smithi104 ceph-mon[97044]: pgmap v5324: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:33.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:33.932520+0000) 2022-09-10T07:15:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:33 smithi110 ceph-mon[97633]: pgmap v5324: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:34.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:34.309354+0000) 2022-09-10T07:15:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:34.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:34.932757+0000) 2022-09-10T07:15:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:35.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:35.309552+0000) 2022-09-10T07:15:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:35 smithi110 ceph-mon[97633]: pgmap v5325: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:35 smithi104 ceph-mon[97044]: pgmap v5325: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:35.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:35.932958+0000) 2022-09-10T07:15:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:36.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:36.309805+0000) 2022-09-10T07:15:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:36.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:36.933202+0000) 2022-09-10T07:15:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:37.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:37.310049+0000) 2022-09-10T07:15:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:37 smithi104 ceph-mon[97044]: pgmap v5326: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:37.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:37.933448+0000) 2022-09-10T07:15:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:37 smithi110 ceph-mon[97633]: pgmap v5326: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:38.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:38.310242+0000) 2022-09-10T07:15:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:38.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:38.933720+0000) 2022-09-10T07:15:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:39.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:39.310417+0000) 2022-09-10T07:15:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:39 smithi104 ceph-mon[97044]: pgmap v5327: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:39.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:39.933939+0000) 2022-09-10T07:15:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:39 smithi110 ceph-mon[97633]: pgmap v5327: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:40.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:40.310610+0000) 2022-09-10T07:15:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:40.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:40.934133+0000) 2022-09-10T07:15:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:41.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:41.310866+0000) 2022-09-10T07:15:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:41 smithi104 ceph-mon[97044]: pgmap v5328: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:41.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:41.934337+0000) 2022-09-10T07:15:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:41 smithi110 ceph-mon[97633]: pgmap v5328: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:42.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:42.311124+0000) 2022-09-10T07:15:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:42.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:42.934572+0000) 2022-09-10T07:15:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:43.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:43.311293+0000) 2022-09-10T07:15:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:44 smithi104 ceph-mon[97044]: pgmap v5329: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:43.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:43.934796+0000) 2022-09-10T07:15:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:44 smithi110 ceph-mon[97633]: pgmap v5329: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:44.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:44.311468+0000) 2022-09-10T07:15:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:15:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:15:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:15:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:15:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:44.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:44.935010+0000) 2022-09-10T07:15:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:15:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:15:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:15:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:15:45.749 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:45.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:45.311640+0000) 2022-09-10T07:15:46.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:15:45.748+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:15:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:45.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:45.935199+0000) 2022-09-10T07:15:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:46 smithi104 ceph-mon[97044]: pgmap v5330: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:46 smithi110 ceph-mon[97633]: pgmap v5330: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:46.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:46.311877+0000) 2022-09-10T07:15:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:46.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:46.935369+0000) 2022-09-10T07:15:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:47.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:47.312193+0000) 2022-09-10T07:15:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:48 smithi104 ceph-mon[97044]: pgmap v5331: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:47.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:47.935553+0000) 2022-09-10T07:15:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:48 smithi110 ceph-mon[97633]: pgmap v5331: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:48.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:48.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:48.312411+0000) 2022-09-10T07:15:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:48.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:48.935783+0000) 2022-09-10T07:15:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:49.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:49.312708+0000) 2022-09-10T07:15:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:50 smithi104 ceph-mon[97044]: pgmap v5332: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:49.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:49.936025+0000) 2022-09-10T07:15:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:50 smithi110 ceph-mon[97633]: pgmap v5332: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:15:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:50.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:50.312992+0000) 2022-09-10T07:15:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:50.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:50.936264+0000) 2022-09-10T07:15:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:51.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:51.313276+0000) 2022-09-10T07:15:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:52 smithi104 ceph-mon[97044]: pgmap v5333: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:51.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:51.936439+0000) 2022-09-10T07:15:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:52 smithi110 ceph-mon[97633]: pgmap v5333: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:52.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:52.313531+0000) 2022-09-10T07:15:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:52.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:52.936581+0000) 2022-09-10T07:15:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:53.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:53.313749+0000) 2022-09-10T07:15:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:54 smithi104 ceph-mon[97044]: pgmap v5334: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:15:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:53.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:53.936714+0000) 2022-09-10T07:15:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:54 smithi110 ceph-mon[97633]: pgmap v5334: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:15:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:54.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:54.314014+0000) 2022-09-10T07:15:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:54.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:54.936934+0000) 2022-09-10T07:15:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:55.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:55.314222+0000) 2022-09-10T07:15:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:55 smithi110 ceph-mon[97633]: pgmap v5335: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:55 smithi104 ceph-mon[97044]: pgmap v5335: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:15:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:55.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:55.937137+0000) 2022-09-10T07:15:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:56.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:56.314449+0000) 2022-09-10T07:15:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:56.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:56.937284+0000) 2022-09-10T07:15:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:57.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:57.314668+0000) 2022-09-10T07:15:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:15:57 smithi104 ceph-mon[97044]: pgmap v5336: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:15:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:57.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:57.937502+0000) 2022-09-10T07:15:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:15:57 smithi110 ceph-mon[97633]: pgmap v5336: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:15:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:58.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:58.314951+0000) 2022-09-10T07:15:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:58.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:58.937721+0000) 2022-09-10T07:15:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:15:59.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:59.315220+0000) 2022-09-10T07:16:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:00 smithi104 ceph-mon[97044]: pgmap v5337: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:16:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:15:59.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:15:59.937931+0000) 2022-09-10T07:16:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:00 smithi110 ceph-mon[97633]: pgmap v5337: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:16:00.759 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:00.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:00.315489+0000) 2022-09-10T07:16:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:00.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:00.938216+0000) 2022-09-10T07:16:01.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:16:00.758+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:16:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:01.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:01.315709+0000) 2022-09-10T07:16:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:02 smithi104 ceph-mon[97044]: pgmap v5338: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:01.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:01.938455+0000) 2022-09-10T07:16:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:02 smithi110 ceph-mon[97633]: pgmap v5338: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:02.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:02.315990+0000) 2022-09-10T07:16:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:02.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:02.938709+0000) 2022-09-10T07:16:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:03.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:03.316271+0000) 2022-09-10T07:16:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:04 smithi104 ceph-mon[97044]: pgmap v5339: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:03.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:03.938901+0000) 2022-09-10T07:16:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:04 smithi110 ceph-mon[97633]: pgmap v5339: 129 pgs: 129 active+clean; 8.2 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:04.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:04.316532+0000) 2022-09-10T07:16:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:04.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:04.939166+0000) 2022-09-10T07:16:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:05.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:05.316844+0000) 2022-09-10T07:16:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:05 smithi110 ceph-mon[97633]: pgmap v5340: 129 pgs: 129 active+clean; 8.2 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:16:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:05 smithi104 ceph-mon[97044]: pgmap v5340: 129 pgs: 129 active+clean; 8.2 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:16:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:05.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:05.939361+0000) 2022-09-10T07:16:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:06.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:06.317111+0000) 2022-09-10T07:16:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:06.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:06.939579+0000) 2022-09-10T07:16:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:07.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:07.317414+0000) 2022-09-10T07:16:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:08 smithi104 ceph-mon[97044]: pgmap v5341: 129 pgs: 129 active+clean; 8.2 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:16:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:07.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:07.939822+0000) 2022-09-10T07:16:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:08 smithi110 ceph-mon[97633]: pgmap v5341: 129 pgs: 129 active+clean; 8.2 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:16:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:08.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:08.317636+0000) 2022-09-10T07:16:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:09 smithi104 ceph-mon[97044]: osdmap e414: 12 total, 12 up, 12 in 2022-09-10T07:16:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:08.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:08.940070+0000) 2022-09-10T07:16:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:09 smithi110 ceph-mon[97633]: osdmap e414: 12 total, 12 up, 12 in 2022-09-10T07:16:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:09.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:09.317855+0000) 2022-09-10T07:16:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:10 smithi104 ceph-mon[97044]: pgmap v5343: 129 pgs: 129 active+clean; 8.2 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:16:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:09.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:09.940364+0000) 2022-09-10T07:16:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:10 smithi110 ceph-mon[97633]: pgmap v5343: 129 pgs: 129 active+clean; 8.2 MiB data, 622 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:16:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:10.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:10.318146+0000) 2022-09-10T07:16:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:10.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:10.940589+0000) 2022-09-10T07:16:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:11.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:11.318285+0000) 2022-09-10T07:16:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:12 smithi104 ceph-mon[97044]: pgmap v5344: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:16:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:11.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:11.940785+0000) 2022-09-10T07:16:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:12 smithi110 ceph-mon[97633]: pgmap v5344: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:16:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:12.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:12.318442+0000) 2022-09-10T07:16:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:12.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:12.941043+0000) 2022-09-10T07:16:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:13.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:13.318685+0000) 2022-09-10T07:16:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:14 smithi104 ceph-mon[97044]: pgmap v5345: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:16:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:13.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:13.941270+0000) 2022-09-10T07:16:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:14 smithi110 ceph-mon[97633]: pgmap v5345: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:16:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:14.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:14.318976+0000) 2022-09-10T07:16:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:14.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:14.941445+0000) 2022-09-10T07:16:15.772 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:15.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:15.319258+0000) 2022-09-10T07:16:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:15 smithi110 ceph-mon[97633]: osdmap e415: 12 total, 12 up, 12 in 2022-09-10T07:16:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:15 smithi110 ceph-mon[97633]: pgmap v5347: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:16:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:15 smithi104 ceph-mon[97044]: osdmap e415: 12 total, 12 up, 12 in 2022-09-10T07:16:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:15 smithi104 ceph-mon[97044]: pgmap v5347: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:16:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:15.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:15.941660+0000) 2022-09-10T07:16:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:16:15.770+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:16:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:16.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:16.319423+0000) 2022-09-10T07:16:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:16.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:16.941821+0000) 2022-09-10T07:16:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:17.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:17.319561+0000) 2022-09-10T07:16:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:18 smithi104 ceph-mon[97044]: pgmap v5348: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:16:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:17.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:17.942049+0000) 2022-09-10T07:16:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:18 smithi110 ceph-mon[97633]: pgmap v5348: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:16:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:18.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:18.319824+0000) 2022-09-10T07:16:19.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:18.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:18.942309+0000) 2022-09-10T07:16:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:19.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:19.320069+0000) 2022-09-10T07:16:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:20 smithi104 ceph-mon[97044]: pgmap v5349: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:16:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:19.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:19.942553+0000) 2022-09-10T07:16:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:20 smithi110 ceph-mon[97633]: pgmap v5349: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:16:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:20.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:20.320276+0000) 2022-09-10T07:16:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:20.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:20.942784+0000) 2022-09-10T07:16:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:21.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:21.320514+0000) 2022-09-10T07:16:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:22 smithi104 ceph-mon[97044]: pgmap v5350: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:16:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:21.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:21.943023+0000) 2022-09-10T07:16:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:22 smithi110 ceph-mon[97633]: pgmap v5350: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:16:22.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:22.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:22.320759+0000) 2022-09-10T07:16:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:22.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:22.943202+0000) 2022-09-10T07:16:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:23.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:23.321015+0000) 2022-09-10T07:16:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:24 smithi104 ceph-mon[97044]: pgmap v5351: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:16:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:23.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:23.943375+0000) 2022-09-10T07:16:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:24 smithi110 ceph-mon[97633]: pgmap v5351: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:16:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:24.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:24.321218+0000) 2022-09-10T07:16:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:24.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:24.943572+0000) 2022-09-10T07:16:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:25.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:25.321374+0000) 2022-09-10T07:16:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:25 smithi110 ceph-mon[97633]: pgmap v5352: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:16:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:25 smithi104 ceph-mon[97044]: pgmap v5352: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:16:26.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:25.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:25.943839+0000) 2022-09-10T07:16:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:26.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:26.321601+0000) 2022-09-10T07:16:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:26.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:26.944008+0000) 2022-09-10T07:16:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:27.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:27.321808+0000) 2022-09-10T07:16:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:28 smithi104 ceph-mon[97044]: pgmap v5353: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:27.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:27.944225+0000) 2022-09-10T07:16:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:28 smithi110 ceph-mon[97633]: pgmap v5353: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:28.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:28.322004+0000) 2022-09-10T07:16:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:28.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:28.944479+0000) 2022-09-10T07:16:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:29.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:29.322250+0000) 2022-09-10T07:16:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:30 smithi104 ceph-mon[97044]: pgmap v5354: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:29.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:29.944663+0000) 2022-09-10T07:16:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:30 smithi110 ceph-mon[97633]: pgmap v5354: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:30.776 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:30.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:30.322435+0000) 2022-09-10T07:16:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:16:30.774+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:16:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:30.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:30.944869+0000) 2022-09-10T07:16:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:31.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:31.322707+0000) 2022-09-10T07:16:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:32 smithi104 ceph-mon[97044]: pgmap v5355: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:31.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:31.945046+0000) 2022-09-10T07:16:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:32 smithi110 ceph-mon[97633]: pgmap v5355: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:32.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:32.322923+0000) 2022-09-10T07:16:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:32.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:32.945268+0000) 2022-09-10T07:16:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:33.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:33.323187+0000) 2022-09-10T07:16:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:34 smithi104 ceph-mon[97044]: pgmap v5356: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:33.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:33.945485+0000) 2022-09-10T07:16:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:34 smithi110 ceph-mon[97633]: pgmap v5356: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:34.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:34.323426+0000) 2022-09-10T07:16:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:34.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:34.945655+0000) 2022-09-10T07:16:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:35.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:35.323651+0000) 2022-09-10T07:16:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:35 smithi110 ceph-mon[97633]: pgmap v5357: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:35 smithi104 ceph-mon[97044]: pgmap v5357: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:35.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:35.945868+0000) 2022-09-10T07:16:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:36.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:36.323884+0000) 2022-09-10T07:16:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:36.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:36.946084+0000) 2022-09-10T07:16:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:37.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:37.324061+0000) 2022-09-10T07:16:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:38 smithi104 ceph-mon[97044]: pgmap v5358: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:37.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:37.946397+0000) 2022-09-10T07:16:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:38 smithi110 ceph-mon[97633]: pgmap v5358: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:38.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:38.324194+0000) 2022-09-10T07:16:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:38.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:38.946650+0000) 2022-09-10T07:16:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:39.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:39.324436+0000) 2022-09-10T07:16:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:40 smithi104 ceph-mon[97044]: pgmap v5359: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:39.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:39.946929+0000) 2022-09-10T07:16:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:40 smithi110 ceph-mon[97633]: pgmap v5359: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:40.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:40.324637+0000) 2022-09-10T07:16:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:40.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:40.947210+0000) 2022-09-10T07:16:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:41.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:41.324886+0000) 2022-09-10T07:16:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:42 smithi104 ceph-mon[97044]: pgmap v5360: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:41.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:41.947421+0000) 2022-09-10T07:16:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:42 smithi110 ceph-mon[97633]: pgmap v5360: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:42.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:42.325162+0000) 2022-09-10T07:16:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:42.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:42.947654+0000) 2022-09-10T07:16:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:43.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:43.325283+0000) 2022-09-10T07:16:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:44 smithi104 ceph-mon[97044]: pgmap v5361: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:43.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:43.947936+0000) 2022-09-10T07:16:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:44 smithi110 ceph-mon[97633]: pgmap v5361: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:44.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:44.325419+0000) 2022-09-10T07:16:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:16:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:16:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:16:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:16:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:44.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:44.948153+0000) 2022-09-10T07:16:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:16:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:16:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:16:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:16:45.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:45.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:45.325644+0000) 2022-09-10T07:16:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:45.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:45.948384+0000) 2022-09-10T07:16:46.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:16:45.788+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:16:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:46 smithi104 ceph-mon[97044]: pgmap v5362: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:46 smithi110 ceph-mon[97633]: pgmap v5362: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:46.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:46.325831+0000) 2022-09-10T07:16:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:46.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:46.948583+0000) 2022-09-10T07:16:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:47.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:47.325996+0000) 2022-09-10T07:16:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:48 smithi104 ceph-mon[97044]: pgmap v5363: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:47.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:47.948763+0000) 2022-09-10T07:16:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:48 smithi110 ceph-mon[97633]: pgmap v5363: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:48.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:48.326249+0000) 2022-09-10T07:16:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:48.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:48.949007+0000) 2022-09-10T07:16:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:49.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:49.326494+0000) 2022-09-10T07:16:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:50 smithi104 ceph-mon[97044]: pgmap v5364: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:49.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:49.949270+0000) 2022-09-10T07:16:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:50 smithi110 ceph-mon[97633]: pgmap v5364: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:50.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:50.326697+0000) 2022-09-10T07:16:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:50.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:50.949544+0000) 2022-09-10T07:16:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:51.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:51.326912+0000) 2022-09-10T07:16:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:52 smithi104 ceph-mon[97044]: pgmap v5365: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:51.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:51.949766+0000) 2022-09-10T07:16:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:52 smithi110 ceph-mon[97633]: pgmap v5365: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:52.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:52.327201+0000) 2022-09-10T07:16:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:52.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:52.949995+0000) 2022-09-10T07:16:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:53.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:53.327433+0000) 2022-09-10T07:16:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:54 smithi104 ceph-mon[97044]: pgmap v5366: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:53.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:53.950222+0000) 2022-09-10T07:16:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:54 smithi110 ceph-mon[97633]: pgmap v5366: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:54.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:54.327578+0000) 2022-09-10T07:16:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:54.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:54.950473+0000) 2022-09-10T07:16:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:55.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:55.327798+0000) 2022-09-10T07:16:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:55 smithi110 ceph-mon[97633]: pgmap v5367: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:55 smithi104 ceph-mon[97044]: pgmap v5367: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:16:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:55.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:55.950659+0000) 2022-09-10T07:16:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:56.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:56.328018+0000) 2022-09-10T07:16:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:56.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:56.950908+0000) 2022-09-10T07:16:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:57.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:57.328255+0000) 2022-09-10T07:16:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:16:58 smithi104 ceph-mon[97044]: pgmap v5368: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:57.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:57.951183+0000) 2022-09-10T07:16:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:16:58 smithi110 ceph-mon[97633]: pgmap v5368: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:16:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:58.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:58.328513+0000) 2022-09-10T07:16:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:58.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:58.951402+0000) 2022-09-10T07:16:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:16:59.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:59.328696+0000) 2022-09-10T07:17:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:00 smithi104 ceph-mon[97044]: pgmap v5369: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:16:59.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:16:59.951642+0000) 2022-09-10T07:17:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:00 smithi110 ceph-mon[97633]: pgmap v5369: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:00.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:00.328918+0000) 2022-09-10T07:17:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:00.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:00.951811+0000) 2022-09-10T07:17:01.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:17:00.803+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:17:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:01.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:01.329115+0000) 2022-09-10T07:17:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:02 smithi104 ceph-mon[97044]: pgmap v5370: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:01.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:01.952060+0000) 2022-09-10T07:17:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:02 smithi110 ceph-mon[97633]: pgmap v5370: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:02.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:02.329351+0000) 2022-09-10T07:17:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:02.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:02.952234+0000) 2022-09-10T07:17:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:03.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:03.329494+0000) 2022-09-10T07:17:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:04 smithi104 ceph-mon[97044]: pgmap v5371: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:03.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:03.952380+0000) 2022-09-10T07:17:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:04 smithi110 ceph-mon[97633]: pgmap v5371: 129 pgs: 129 active+clean; 8.2 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:04.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:04.329708+0000) 2022-09-10T07:17:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:04.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:04.952636+0000) 2022-09-10T07:17:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:05.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:05.329963+0000) 2022-09-10T07:17:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:05 smithi110 ceph-mon[97633]: pgmap v5372: 129 pgs: 129 active+clean; 8.2 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:17:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:05 smithi104 ceph-mon[97044]: pgmap v5372: 129 pgs: 129 active+clean; 8.2 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:17:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:05.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:05.952837+0000) 2022-09-10T07:17:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:06.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:06.330201+0000) 2022-09-10T07:17:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:06.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:06.953064+0000) 2022-09-10T07:17:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:07.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:07.330387+0000) 2022-09-10T07:17:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:08 smithi104 ceph-mon[97044]: pgmap v5373: 129 pgs: 129 active+clean; 8.2 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:17:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:07.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:07.953166+0000) 2022-09-10T07:17:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:08 smithi110 ceph-mon[97633]: pgmap v5373: 129 pgs: 129 active+clean; 8.2 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:17:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:08.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:08.330568+0000) 2022-09-10T07:17:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:09 smithi104 ceph-mon[97044]: osdmap e416: 12 total, 12 up, 12 in 2022-09-10T07:17:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:08.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:08.953407+0000) 2022-09-10T07:17:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:09 smithi110 ceph-mon[97633]: osdmap e416: 12 total, 12 up, 12 in 2022-09-10T07:17:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:09.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:09.330792+0000) 2022-09-10T07:17:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:10 smithi104 ceph-mon[97044]: pgmap v5375: 129 pgs: 129 active+clean; 8.2 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:17:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:09.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:09.953642+0000) 2022-09-10T07:17:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:10 smithi110 ceph-mon[97633]: pgmap v5375: 129 pgs: 129 active+clean; 8.2 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:17:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:10.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:10.331020+0000) 2022-09-10T07:17:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:10.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:10.953854+0000) 2022-09-10T07:17:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:11.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:11.331248+0000) 2022-09-10T07:17:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:12 smithi104 ceph-mon[97044]: pgmap v5376: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:17:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:11.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:11.954071+0000) 2022-09-10T07:17:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:12 smithi110 ceph-mon[97633]: pgmap v5376: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:17:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:12.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:12.331427+0000) 2022-09-10T07:17:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:12.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:12.954345+0000) 2022-09-10T07:17:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:13.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:13.331651+0000) 2022-09-10T07:17:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:14 smithi104 ceph-mon[97044]: pgmap v5377: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:17:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:13.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:13.954513+0000) 2022-09-10T07:17:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:14 smithi110 ceph-mon[97633]: pgmap v5377: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:17:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:14.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:14.331908+0000) 2022-09-10T07:17:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:14.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:14.954661+0000) 2022-09-10T07:17:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:15.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:15.332154+0000) 2022-09-10T07:17:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:15 smithi110 ceph-mon[97633]: osdmap e417: 12 total, 12 up, 12 in 2022-09-10T07:17:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:15 smithi110 ceph-mon[97633]: pgmap v5379: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:17:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:15 smithi104 ceph-mon[97044]: osdmap e417: 12 total, 12 up, 12 in 2022-09-10T07:17:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:15 smithi104 ceph-mon[97044]: pgmap v5379: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:17:16.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:17:15.818+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:17:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:15.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:15.954836+0000) 2022-09-10T07:17:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:16.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:16.332333+0000) 2022-09-10T07:17:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:16.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:16.955041+0000) 2022-09-10T07:17:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:17.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:17.332502+0000) 2022-09-10T07:17:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:18 smithi104 ceph-mon[97044]: pgmap v5380: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:17:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:17.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:17.955286+0000) 2022-09-10T07:17:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:18 smithi110 ceph-mon[97633]: pgmap v5380: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:17:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:18.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:18.332646+0000) 2022-09-10T07:17:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:18.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:18.955579+0000) 2022-09-10T07:17:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:19.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:19.332847+0000) 2022-09-10T07:17:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:20 smithi104 ceph-mon[97044]: pgmap v5381: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:17:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:19.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:19.955848+0000) 2022-09-10T07:17:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:20 smithi110 ceph-mon[97633]: pgmap v5381: 129 pgs: 129 active+clean; 8.2 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:17:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:20.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:20.333064+0000) 2022-09-10T07:17:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:20.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:20.956149+0000) 2022-09-10T07:17:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:21.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:21.333328+0000) 2022-09-10T07:17:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:22 smithi104 ceph-mon[97044]: pgmap v5382: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:17:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:21.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:21.956389+0000) 2022-09-10T07:17:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:22 smithi110 ceph-mon[97633]: pgmap v5382: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:17:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:22.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:22.333576+0000) 2022-09-10T07:17:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:22.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:22.956609+0000) 2022-09-10T07:17:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:23.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:23.333823+0000) 2022-09-10T07:17:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:24 smithi104 ceph-mon[97044]: pgmap v5383: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:17:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:23.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:23.956902+0000) 2022-09-10T07:17:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:24 smithi110 ceph-mon[97633]: pgmap v5383: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:17:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:24.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:24.333965+0000) 2022-09-10T07:17:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:24.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:24.957191+0000) 2022-09-10T07:17:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:25.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:25.334193+0000) 2022-09-10T07:17:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:25 smithi110 ceph-mon[97633]: pgmap v5384: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:17:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:25 smithi104 ceph-mon[97044]: pgmap v5384: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:17:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:25.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:25.957482+0000) 2022-09-10T07:17:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:26.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:26.334463+0000) 2022-09-10T07:17:27.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:26.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:26.957740+0000) 2022-09-10T07:17:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:27.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:27.334717+0000) 2022-09-10T07:17:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:28 smithi104 ceph-mon[97044]: pgmap v5385: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:27.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:27.957955+0000) 2022-09-10T07:17:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:28 smithi110 ceph-mon[97633]: pgmap v5385: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:28.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:28.334984+0000) 2022-09-10T07:17:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:28.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:28.958220+0000) 2022-09-10T07:17:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:29.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:29.335182+0000) 2022-09-10T07:17:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:30 smithi104 ceph-mon[97044]: pgmap v5386: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:29.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:29.958445+0000) 2022-09-10T07:17:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:30 smithi110 ceph-mon[97633]: pgmap v5386: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:30.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:30.335352+0000) 2022-09-10T07:17:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:17:30.834+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:17:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:30.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:30.958718+0000) 2022-09-10T07:17:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:31.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:31.335595+0000) 2022-09-10T07:17:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:32 smithi104 ceph-mon[97044]: pgmap v5387: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:31.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:31.958942+0000) 2022-09-10T07:17:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:32 smithi110 ceph-mon[97633]: pgmap v5387: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:32.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:32.335800+0000) 2022-09-10T07:17:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:32.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:32.959158+0000) 2022-09-10T07:17:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:33.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:33.336062+0000) 2022-09-10T07:17:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:34 smithi104 ceph-mon[97044]: pgmap v5388: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:33.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:33.959319+0000) 2022-09-10T07:17:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:34 smithi110 ceph-mon[97633]: pgmap v5388: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:34.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:34.336302+0000) 2022-09-10T07:17:35.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:34.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:34.959493+0000) 2022-09-10T07:17:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:35.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:35.336503+0000) 2022-09-10T07:17:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:35 smithi110 ceph-mon[97633]: pgmap v5389: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:35 smithi104 ceph-mon[97044]: pgmap v5389: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:35.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:35.959703+0000) 2022-09-10T07:17:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:36.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:36.336669+0000) 2022-09-10T07:17:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:36.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:36.959942+0000) 2022-09-10T07:17:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:37.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:37.336867+0000) 2022-09-10T07:17:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:38 smithi104 ceph-mon[97044]: pgmap v5390: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:37.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:37.960197+0000) 2022-09-10T07:17:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:38 smithi110 ceph-mon[97633]: pgmap v5390: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:38.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:38.337048+0000) 2022-09-10T07:17:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:38.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:38.960490+0000) 2022-09-10T07:17:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:39.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:39.337238+0000) 2022-09-10T07:17:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:40 smithi104 ceph-mon[97044]: pgmap v5391: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:39.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:39.960689+0000) 2022-09-10T07:17:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:40 smithi110 ceph-mon[97633]: pgmap v5391: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:40.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:40.337505+0000) 2022-09-10T07:17:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:40.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:40.960946+0000) 2022-09-10T07:17:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:41.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:41.337713+0000) 2022-09-10T07:17:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:42 smithi104 ceph-mon[97044]: pgmap v5392: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:41.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:41.961191+0000) 2022-09-10T07:17:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:42 smithi110 ceph-mon[97633]: pgmap v5392: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:42.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:42.337959+0000) 2022-09-10T07:17:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:42.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:42.961358+0000) 2022-09-10T07:17:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:43.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:43.338146+0000) 2022-09-10T07:17:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:44 smithi104 ceph-mon[97044]: pgmap v5393: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:43.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:43.961609+0000) 2022-09-10T07:17:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:44 smithi110 ceph-mon[97633]: pgmap v5393: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:44.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:44.338379+0000) 2022-09-10T07:17:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:17:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:17:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:17:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:17:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:44.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:44.961813+0000) 2022-09-10T07:17:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:17:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:17:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:17:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:17:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:45.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:45.338628+0000) 2022-09-10T07:17:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:46 smithi104 ceph-mon[97044]: pgmap v5394: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:45.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:45.962002+0000) 2022-09-10T07:17:46.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:17:45.839+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:17:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:46 smithi110 ceph-mon[97633]: pgmap v5394: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:46.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:46.338873+0000) 2022-09-10T07:17:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:46.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:46.962214+0000) 2022-09-10T07:17:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:47.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:47.339078+0000) 2022-09-10T07:17:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:48 smithi104 ceph-mon[97044]: pgmap v5395: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:47.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:47.962465+0000) 2022-09-10T07:17:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:48 smithi110 ceph-mon[97633]: pgmap v5395: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:48.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:48.339298+0000) 2022-09-10T07:17:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:48.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:48.962631+0000) 2022-09-10T07:17:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:49.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:49.339561+0000) 2022-09-10T07:17:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:49.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:49.962775+0000) 2022-09-10T07:17:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:50 smithi110 ceph-mon[97633]: pgmap v5396: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:50 smithi104 ceph-mon[97044]: pgmap v5396: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:50.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:50.339820+0000) 2022-09-10T07:17:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:50.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:50.962963+0000) 2022-09-10T07:17:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:51.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:51.340043+0000) 2022-09-10T07:17:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:51.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:51.963210+0000) 2022-09-10T07:17:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:52 smithi110 ceph-mon[97633]: pgmap v5397: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:52 smithi104 ceph-mon[97044]: pgmap v5397: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:52.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:52.340269+0000) 2022-09-10T07:17:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:52.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:52.963406+0000) 2022-09-10T07:17:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:53.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:53.340499+0000) 2022-09-10T07:17:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:53.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:53.963654+0000) 2022-09-10T07:17:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:54 smithi110 ceph-mon[97633]: pgmap v5398: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:54 smithi104 ceph-mon[97044]: pgmap v5398: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:17:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:54.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:54.340670+0000) 2022-09-10T07:17:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:54.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:54.963933+0000) 2022-09-10T07:17:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:55.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:55.340845+0000) 2022-09-10T07:17:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:55 smithi110 ceph-mon[97633]: pgmap v5399: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:55 smithi104 ceph-mon[97044]: pgmap v5399: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:17:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:55.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:55.964199+0000) 2022-09-10T07:17:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:56.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:56.341049+0000) 2022-09-10T07:17:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:56.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:56.964374+0000) 2022-09-10T07:17:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:57.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:57.341267+0000) 2022-09-10T07:17:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:17:58 smithi104 ceph-mon[97044]: pgmap v5400: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:57.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:57.964542+0000) 2022-09-10T07:17:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:17:58 smithi110 ceph-mon[97633]: pgmap v5400: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:17:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:58.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:58.341517+0000) 2022-09-10T07:17:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:58.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:58.964723+0000) 2022-09-10T07:17:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:17:59.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:59.341708+0000) 2022-09-10T07:18:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:00 smithi104 ceph-mon[97044]: pgmap v5401: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:17:59.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:17:59.964922+0000) 2022-09-10T07:18:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:00 smithi110 ceph-mon[97633]: pgmap v5401: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:00.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:00.341869+0000) 2022-09-10T07:18:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:00.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:00.965153+0000) 2022-09-10T07:18:01.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:18:00.854+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:18:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:01.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:01.342090+0000) 2022-09-10T07:18:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:02 smithi104 ceph-mon[97044]: pgmap v5402: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:01.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:01.965371+0000) 2022-09-10T07:18:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:02 smithi110 ceph-mon[97633]: pgmap v5402: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:02.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:02.342330+0000) 2022-09-10T07:18:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:02.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:02.965643+0000) 2022-09-10T07:18:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:03.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:03.342557+0000) 2022-09-10T07:18:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:04 smithi104 ceph-mon[97044]: pgmap v5403: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:03.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:03.965839+0000) 2022-09-10T07:18:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:04 smithi110 ceph-mon[97633]: pgmap v5403: 129 pgs: 129 active+clean; 8.2 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:04.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:04.342733+0000) 2022-09-10T07:18:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:04.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:04.966058+0000) 2022-09-10T07:18:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:05.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:05.342974+0000) 2022-09-10T07:18:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:05 smithi110 ceph-mon[97633]: pgmap v5404: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:18:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:05 smithi104 ceph-mon[97044]: pgmap v5404: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:18:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:05.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:05.966336+0000) 2022-09-10T07:18:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:06.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:06.343234+0000) 2022-09-10T07:18:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:06.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:06.966540+0000) 2022-09-10T07:18:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:07.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:07.343456+0000) 2022-09-10T07:18:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:07.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:07.966788+0000) 2022-09-10T07:18:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:08 smithi110 ceph-mon[97633]: pgmap v5405: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:18:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:08 smithi104 ceph-mon[97044]: pgmap v5405: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:18:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:08.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:08.343670+0000) 2022-09-10T07:18:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:08.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:08.966979+0000) 2022-09-10T07:18:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:09 smithi110 ceph-mon[97633]: osdmap e418: 12 total, 12 up, 12 in 2022-09-10T07:18:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:09 smithi104 ceph-mon[97044]: osdmap e418: 12 total, 12 up, 12 in 2022-09-10T07:18:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:09.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:09.343928+0000) 2022-09-10T07:18:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:09.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:09.967207+0000) 2022-09-10T07:18:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:10 smithi110 ceph-mon[97633]: pgmap v5407: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:18:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:10 smithi104 ceph-mon[97044]: pgmap v5407: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:18:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:10.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:10.344177+0000) 2022-09-10T07:18:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:10.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:10.967416+0000) 2022-09-10T07:18:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:11.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:11.344359+0000) 2022-09-10T07:18:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:11.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:11.967651+0000) 2022-09-10T07:18:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:12 smithi110 ceph-mon[97633]: pgmap v5408: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:18:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:12 smithi104 ceph-mon[97044]: pgmap v5408: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:18:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:12.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:12.344517+0000) 2022-09-10T07:18:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:12.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:12.967831+0000) 2022-09-10T07:18:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:13.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:13.344711+0000) 2022-09-10T07:18:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:13.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:13.968052+0000) 2022-09-10T07:18:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:14 smithi110 ceph-mon[97633]: pgmap v5409: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:18:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:14 smithi104 ceph-mon[97044]: pgmap v5409: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:18:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:14.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:14.344949+0000) 2022-09-10T07:18:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:14.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:14.968271+0000) 2022-09-10T07:18:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:15.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:15.345194+0000) 2022-09-10T07:18:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:15 smithi110 ceph-mon[97633]: osdmap e419: 12 total, 12 up, 12 in 2022-09-10T07:18:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:15 smithi110 ceph-mon[97633]: pgmap v5411: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:18:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:15 smithi104 ceph-mon[97044]: osdmap e419: 12 total, 12 up, 12 in 2022-09-10T07:18:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:15 smithi104 ceph-mon[97044]: pgmap v5411: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:18:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:15.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:15.968472+0000) 2022-09-10T07:18:16.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:18:15.860+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:18:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:16.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:16.345413+0000) 2022-09-10T07:18:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:16.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:16.968733+0000) 2022-09-10T07:18:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:17.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:17.345608+0000) 2022-09-10T07:18:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:18 smithi104 ceph-mon[97044]: pgmap v5412: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:18:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:17.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:17.968932+0000) 2022-09-10T07:18:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:18 smithi110 ceph-mon[97633]: pgmap v5412: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:18:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:18.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:18.345796+0000) 2022-09-10T07:18:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:18.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:18.969090+0000) 2022-09-10T07:18:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:19.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:19.346023+0000) 2022-09-10T07:18:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:20 smithi104 ceph-mon[97044]: pgmap v5413: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:18:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:19.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:19.969299+0000) 2022-09-10T07:18:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:20 smithi110 ceph-mon[97633]: pgmap v5413: 129 pgs: 129 active+clean; 8.2 MiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:18:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:20.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:20.346246+0000) 2022-09-10T07:18:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:20.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:20.969548+0000) 2022-09-10T07:18:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:21.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:21.346536+0000) 2022-09-10T07:18:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:21.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:21.969808+0000) 2022-09-10T07:18:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:22 smithi110 ceph-mon[97633]: pgmap v5414: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:18:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:22 smithi104 ceph-mon[97044]: pgmap v5414: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:18:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:22.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:22.346796+0000) 2022-09-10T07:18:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:22.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:22.969985+0000) 2022-09-10T07:18:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:23.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:23.347001+0000) 2022-09-10T07:18:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:23.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:23.970246+0000) 2022-09-10T07:18:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:24 smithi110 ceph-mon[97633]: pgmap v5415: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:18:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:24 smithi104 ceph-mon[97044]: pgmap v5415: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:18:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:24.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:24.347213+0000) 2022-09-10T07:18:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:24.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:24.970472+0000) 2022-09-10T07:18:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:25.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:25.347408+0000) 2022-09-10T07:18:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:25 smithi110 ceph-mon[97633]: pgmap v5416: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:18:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:25 smithi104 ceph-mon[97044]: pgmap v5416: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:18:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:25.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:25.970719+0000) 2022-09-10T07:18:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:26.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:26.347582+0000) 2022-09-10T07:18:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:26.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:26.970949+0000) 2022-09-10T07:18:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:27.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:27.347748+0000) 2022-09-10T07:18:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:27.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:27.971188+0000) 2022-09-10T07:18:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:28 smithi110 ceph-mon[97633]: pgmap v5417: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:28 smithi104 ceph-mon[97044]: pgmap v5417: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:28.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:28.347910+0000) 2022-09-10T07:18:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:28.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:28.971453+0000) 2022-09-10T07:18:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:29.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:29.348080+0000) 2022-09-10T07:18:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:29.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:29.971705+0000) 2022-09-10T07:18:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:30 smithi110 ceph-mon[97633]: pgmap v5418: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:30 smithi104 ceph-mon[97044]: pgmap v5418: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:30.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:30.348282+0000) 2022-09-10T07:18:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:30.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:30.971976+0000) 2022-09-10T07:18:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:18:30.870+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:18:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:31.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:31.348525+0000) 2022-09-10T07:18:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:31.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:31.972252+0000) 2022-09-10T07:18:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:32 smithi110 ceph-mon[97633]: pgmap v5419: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:32 smithi104 ceph-mon[97044]: pgmap v5419: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:32.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:32.348721+0000) 2022-09-10T07:18:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:32.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:32.972517+0000) 2022-09-10T07:18:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:33.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:33.349004+0000) 2022-09-10T07:18:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:33.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:33.972797+0000) 2022-09-10T07:18:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:34 smithi110 ceph-mon[97633]: pgmap v5420: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:34 smithi104 ceph-mon[97044]: pgmap v5420: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:34.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:34.349239+0000) 2022-09-10T07:18:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:34.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:34.973061+0000) 2022-09-10T07:18:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:35.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:35.349485+0000) 2022-09-10T07:18:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:35 smithi110 ceph-mon[97633]: pgmap v5421: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:36.044 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:35 smithi104 ceph-mon[97044]: pgmap v5421: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:35.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:35.973246+0000) 2022-09-10T07:18:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:36.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:36.349726+0000) 2022-09-10T07:18:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:36.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:36.973413+0000) 2022-09-10T07:18:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:37.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:37.349981+0000) 2022-09-10T07:18:38.255 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:38 smithi104 ceph-mon[97044]: pgmap v5422: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:37.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:37.973604+0000) 2022-09-10T07:18:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:38 smithi110 ceph-mon[97633]: pgmap v5422: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:38.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:38.350232+0000) 2022-09-10T07:18:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:38.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:38.973776+0000) 2022-09-10T07:18:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:39.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:39.350468+0000) 2022-09-10T07:18:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:40 smithi104 ceph-mon[97044]: pgmap v5423: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:39.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:39.973986+0000) 2022-09-10T07:18:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:40 smithi110 ceph-mon[97633]: pgmap v5423: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:40.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:40.350723+0000) 2022-09-10T07:18:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:40.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:40.974232+0000) 2022-09-10T07:18:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:41.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:41.350976+0000) 2022-09-10T07:18:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:41.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:41.974432+0000) 2022-09-10T07:18:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:42 smithi110 ceph-mon[97633]: pgmap v5424: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:42 smithi104 ceph-mon[97044]: pgmap v5424: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:42.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:42.351210+0000) 2022-09-10T07:18:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:42.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:42.974671+0000) 2022-09-10T07:18:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:43.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:43.351411+0000) 2022-09-10T07:18:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:43.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:43.974963+0000) 2022-09-10T07:18:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:44 smithi110 ceph-mon[97633]: pgmap v5425: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:44 smithi104 ceph-mon[97044]: pgmap v5425: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:44.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:44.351675+0000) 2022-09-10T07:18:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:44.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:44.975208+0000) 2022-09-10T07:18:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:18:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:18:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:18:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:18:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:18:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:18:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:18:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:18:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:45.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:45.351952+0000) 2022-09-10T07:18:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:45.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:45.975407+0000) 2022-09-10T07:18:46.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:18:45.886+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:18:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:46 smithi110 ceph-mon[97633]: pgmap v5426: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:46 smithi104 ceph-mon[97044]: pgmap v5426: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:46.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:46.352204+0000) 2022-09-10T07:18:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:46.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:46.975661+0000) 2022-09-10T07:18:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:47.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:47.352425+0000) 2022-09-10T07:18:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:47.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:47.975818+0000) 2022-09-10T07:18:48.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:48 smithi104 ceph-mon[97044]: pgmap v5427: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:48 smithi110 ceph-mon[97633]: pgmap v5427: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:48.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:48.352633+0000) 2022-09-10T07:18:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:48.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:48.976034+0000) 2022-09-10T07:18:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:49.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:49.352874+0000) 2022-09-10T07:18:50.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:49.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:49.976229+0000) 2022-09-10T07:18:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:50 smithi110 ceph-mon[97633]: pgmap v5428: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:50 smithi104 ceph-mon[97044]: pgmap v5428: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:50.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:50.353128+0000) 2022-09-10T07:18:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:50.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:50.976395+0000) 2022-09-10T07:18:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:51.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:51.353347+0000) 2022-09-10T07:18:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:51.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:51.976591+0000) 2022-09-10T07:18:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:52 smithi110 ceph-mon[97633]: pgmap v5429: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:52 smithi104 ceph-mon[97044]: pgmap v5429: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:52.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:52.353608+0000) 2022-09-10T07:18:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:52.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:52.976829+0000) 2022-09-10T07:18:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:53.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:53.353851+0000) 2022-09-10T07:18:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:53.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:53.977059+0000) 2022-09-10T07:18:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:54 smithi110 ceph-mon[97633]: pgmap v5430: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:54 smithi104 ceph-mon[97044]: pgmap v5430: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:54.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:54.354032+0000) 2022-09-10T07:18:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:54.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:54.977309+0000) 2022-09-10T07:18:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:55.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:55.354220+0000) 2022-09-10T07:18:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:55 smithi110 ceph-mon[97633]: pgmap v5431: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:55 smithi104 ceph-mon[97044]: pgmap v5431: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:18:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:55.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:55.977467+0000) 2022-09-10T07:18:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:56.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:56.354450+0000) 2022-09-10T07:18:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:56.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:56.977685+0000) 2022-09-10T07:18:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:57.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:57.354691+0000) 2022-09-10T07:18:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:57.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:57.977924+0000) 2022-09-10T07:18:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:18:58 smithi110 ceph-mon[97633]: pgmap v5432: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:18:58 smithi104 ceph-mon[97044]: pgmap v5432: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:18:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:58.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:58.354909+0000) 2022-09-10T07:18:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:58.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:58.978241+0000) 2022-09-10T07:18:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:18:59.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:59.355211+0000) 2022-09-10T07:19:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:18:59.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:18:59.978420+0000) 2022-09-10T07:19:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:00 smithi110 ceph-mon[97633]: pgmap v5433: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:00 smithi104 ceph-mon[97044]: pgmap v5433: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:00.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:00.355478+0000) 2022-09-10T07:19:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:00.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:00.978588+0000) 2022-09-10T07:19:01.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:19:00.901+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:19:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:01.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:01.355688+0000) 2022-09-10T07:19:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:01.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:01.978820+0000) 2022-09-10T07:19:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:02 smithi110 ceph-mon[97633]: pgmap v5434: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:02 smithi104 ceph-mon[97044]: pgmap v5434: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:02.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:02.355853+0000) 2022-09-10T07:19:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:02.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:02.979065+0000) 2022-09-10T07:19:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:03.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:03.356036+0000) 2022-09-10T07:19:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:03.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:03.979307+0000) 2022-09-10T07:19:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:04 smithi110 ceph-mon[97633]: pgmap v5435: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:04 smithi104 ceph-mon[97044]: pgmap v5435: 129 pgs: 129 active+clean; 8.2 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:04.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:04.356192+0000) 2022-09-10T07:19:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:04.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:04.979555+0000) 2022-09-10T07:19:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:05.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:05.356379+0000) 2022-09-10T07:19:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:05 smithi110 ceph-mon[97633]: pgmap v5436: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:19:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:05 smithi104 ceph-mon[97044]: pgmap v5436: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:19:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:05.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:05.979739+0000) 2022-09-10T07:19:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:06.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:06.356596+0000) 2022-09-10T07:19:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:06.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:06.979868+0000) 2022-09-10T07:19:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:07.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:07.356849+0000) 2022-09-10T07:19:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:07.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:07.980088+0000) 2022-09-10T07:19:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:08 smithi110 ceph-mon[97633]: pgmap v5437: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:19:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:08 smithi104 ceph-mon[97044]: pgmap v5437: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:19:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:08.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:08.357027+0000) 2022-09-10T07:19:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:08.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:08.980269+0000) 2022-09-10T07:19:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:09 smithi110 ceph-mon[97633]: osdmap e420: 12 total, 12 up, 12 in 2022-09-10T07:19:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:09 smithi104 ceph-mon[97044]: osdmap e420: 12 total, 12 up, 12 in 2022-09-10T07:19:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:09.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:09.357182+0000) 2022-09-10T07:19:10.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:09.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:09.980486+0000) 2022-09-10T07:19:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:10 smithi110 ceph-mon[97633]: pgmap v5439: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:19:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:10 smithi104 ceph-mon[97044]: pgmap v5439: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:19:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:10.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:10.357335+0000) 2022-09-10T07:19:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:10.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:10.980678+0000) 2022-09-10T07:19:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:11.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:11.357521+0000) 2022-09-10T07:19:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:11.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:11.980928+0000) 2022-09-10T07:19:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:12 smithi110 ceph-mon[97633]: pgmap v5440: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:19:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:12 smithi104 ceph-mon[97044]: pgmap v5440: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:19:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:12.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:12.357710+0000) 2022-09-10T07:19:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:12.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:12.981204+0000) 2022-09-10T07:19:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:13.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:13.357914+0000) 2022-09-10T07:19:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:13.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:13.981420+0000) 2022-09-10T07:19:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:14 smithi110 ceph-mon[97633]: pgmap v5441: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:19:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:14 smithi104 ceph-mon[97044]: pgmap v5441: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:19:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:14.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:14.358054+0000) 2022-09-10T07:19:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:14.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:14.981618+0000) 2022-09-10T07:19:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:15.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:15.358246+0000) 2022-09-10T07:19:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:15 smithi110 ceph-mon[97633]: osdmap e421: 12 total, 12 up, 12 in 2022-09-10T07:19:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:15 smithi110 ceph-mon[97633]: pgmap v5443: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:19:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:15 smithi104 ceph-mon[97044]: osdmap e421: 12 total, 12 up, 12 in 2022-09-10T07:19:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:15 smithi104 ceph-mon[97044]: pgmap v5443: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:19:16.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:19:15.914+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:19:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:15.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:15.981834+0000) 2022-09-10T07:19:16.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:16.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:16.358476+0000) 2022-09-10T07:19:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:16.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:16.982072+0000) 2022-09-10T07:19:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:17.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:17.358655+0000) 2022-09-10T07:19:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:17.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:17.982350+0000) 2022-09-10T07:19:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:18 smithi110 ceph-mon[97633]: pgmap v5444: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:19:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:18 smithi104 ceph-mon[97044]: pgmap v5444: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:19:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:18.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:18.358907+0000) 2022-09-10T07:19:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:18.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:18.982549+0000) 2022-09-10T07:19:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:19.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:19.359211+0000) 2022-09-10T07:19:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:19.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:19.982787+0000) 2022-09-10T07:19:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:20 smithi110 ceph-mon[97633]: pgmap v5445: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:19:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:20 smithi104 ceph-mon[97044]: pgmap v5445: 129 pgs: 129 active+clean; 8.2 MiB data, 629 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:19:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:20.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:20.359433+0000) 2022-09-10T07:19:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:20.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:20.983031+0000) 2022-09-10T07:19:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:21.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:21.359675+0000) 2022-09-10T07:19:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:21.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:21.983251+0000) 2022-09-10T07:19:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:22 smithi110 ceph-mon[97633]: pgmap v5446: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:19:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:22 smithi104 ceph-mon[97044]: pgmap v5446: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:19:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:22.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:22.359920+0000) 2022-09-10T07:19:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:22.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:22.983407+0000) 2022-09-10T07:19:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:23.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:23.360203+0000) 2022-09-10T07:19:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:23.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:23.983585+0000) 2022-09-10T07:19:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:24 smithi110 ceph-mon[97633]: pgmap v5447: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:19:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:24 smithi104 ceph-mon[97044]: pgmap v5447: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:19:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:24.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:24.360455+0000) 2022-09-10T07:19:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:24.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:24.983839+0000) 2022-09-10T07:19:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:25.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:25.360692+0000) 2022-09-10T07:19:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:25 smithi104 ceph-mon[97044]: pgmap v5448: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:19:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:25 smithi110 ceph-mon[97633]: pgmap v5448: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:19:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:25.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:25.984071+0000) 2022-09-10T07:19:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:26.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:26.360861+0000) 2022-09-10T07:19:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:26.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:26.984348+0000) 2022-09-10T07:19:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:27.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:27.361068+0000) 2022-09-10T07:19:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:27.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:27.984571+0000) 2022-09-10T07:19:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:28 smithi110 ceph-mon[97633]: pgmap v5449: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:28 smithi104 ceph-mon[97044]: pgmap v5449: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:28.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:28.361272+0000) 2022-09-10T07:19:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:28.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:28.984802+0000) 2022-09-10T07:19:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:29.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:29.361441+0000) 2022-09-10T07:19:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:29.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:29.985045+0000) 2022-09-10T07:19:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:30 smithi110 ceph-mon[97633]: pgmap v5450: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:30 smithi104 ceph-mon[97044]: pgmap v5450: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:30.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:30.361614+0000) 2022-09-10T07:19:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:30.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:30.985158+0000) 2022-09-10T07:19:31.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:19:30.930+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:19:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:31.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:31.361854+0000) 2022-09-10T07:19:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:31.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:31.985354+0000) 2022-09-10T07:19:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:32 smithi110 ceph-mon[97633]: pgmap v5451: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:32 smithi104 ceph-mon[97044]: pgmap v5451: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:32.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:32.362112+0000) 2022-09-10T07:19:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:32.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:32.985589+0000) 2022-09-10T07:19:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:33.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:33.362349+0000) 2022-09-10T07:19:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:33.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:33.985860+0000) 2022-09-10T07:19:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:34 smithi110 ceph-mon[97633]: pgmap v5452: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:34 smithi104 ceph-mon[97044]: pgmap v5452: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:34.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:34.362489+0000) 2022-09-10T07:19:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:34.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:34.986077+0000) 2022-09-10T07:19:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:35.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:35.362672+0000) 2022-09-10T07:19:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:35 smithi104 ceph-mon[97044]: pgmap v5453: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:35 smithi110 ceph-mon[97633]: pgmap v5453: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:35.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:35.986296+0000) 2022-09-10T07:19:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:36.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:36.362917+0000) 2022-09-10T07:19:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:36.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:36.986460+0000) 2022-09-10T07:19:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:37.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:37.363172+0000) 2022-09-10T07:19:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:37.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:37.986722+0000) 2022-09-10T07:19:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:38 smithi110 ceph-mon[97633]: pgmap v5454: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:38 smithi104 ceph-mon[97044]: pgmap v5454: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:38.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:38.363380+0000) 2022-09-10T07:19:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:38.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:38.986934+0000) 2022-09-10T07:19:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:39.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:39.363519+0000) 2022-09-10T07:19:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:39.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:39.987178+0000) 2022-09-10T07:19:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:40 smithi110 ceph-mon[97633]: pgmap v5455: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:40 smithi104 ceph-mon[97044]: pgmap v5455: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:40.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:40.363678+0000) 2022-09-10T07:19:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:40.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:40.987317+0000) 2022-09-10T07:19:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:41.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:41.363891+0000) 2022-09-10T07:19:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:41.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:41.987540+0000) 2022-09-10T07:19:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:42 smithi110 ceph-mon[97633]: pgmap v5456: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:42 smithi104 ceph-mon[97044]: pgmap v5456: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:42.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:42.364176+0000) 2022-09-10T07:19:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:42.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:42.987770+0000) 2022-09-10T07:19:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:43.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:43.364399+0000) 2022-09-10T07:19:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:43.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:43.987966+0000) 2022-09-10T07:19:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:44 smithi110 ceph-mon[97633]: pgmap v5457: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:44 smithi104 ceph-mon[97044]: pgmap v5457: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:44.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:44.364586+0000) 2022-09-10T07:19:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:44.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:44.988214+0000) 2022-09-10T07:19:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:19:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:19:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:19:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:19:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:19:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:19:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:19:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:19:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:45.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:45.364739+0000) 2022-09-10T07:19:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:45.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:45.988432+0000) 2022-09-10T07:19:46.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:19:45.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:19:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:46 smithi110 ceph-mon[97633]: pgmap v5458: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:46 smithi104 ceph-mon[97044]: pgmap v5458: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:46.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:46.364967+0000) 2022-09-10T07:19:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:46.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:46.988604+0000) 2022-09-10T07:19:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:47.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:47.365203+0000) 2022-09-10T07:19:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:47.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:47.988830+0000) 2022-09-10T07:19:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:48 smithi110 ceph-mon[97633]: pgmap v5459: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:48 smithi104 ceph-mon[97044]: pgmap v5459: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:48.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:48.365412+0000) 2022-09-10T07:19:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:48.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:48.989064+0000) 2022-09-10T07:19:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:49.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:49.365611+0000) 2022-09-10T07:19:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:49.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:49.989252+0000) 2022-09-10T07:19:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:50 smithi110 ceph-mon[97633]: pgmap v5460: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:50 smithi104 ceph-mon[97044]: pgmap v5460: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:50.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:50.365880+0000) 2022-09-10T07:19:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:50.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:50.989429+0000) 2022-09-10T07:19:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:51.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:51.366047+0000) 2022-09-10T07:19:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:51.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:51.989601+0000) 2022-09-10T07:19:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:52 smithi110 ceph-mon[97633]: pgmap v5461: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:52 smithi104 ceph-mon[97044]: pgmap v5461: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:52.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:52.366271+0000) 2022-09-10T07:19:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:52.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:52.989863+0000) 2022-09-10T07:19:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:53.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:53.366442+0000) 2022-09-10T07:19:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:53.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:53.990129+0000) 2022-09-10T07:19:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:54 smithi110 ceph-mon[97633]: pgmap v5462: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:54 smithi104 ceph-mon[97044]: pgmap v5462: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:54.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:54.366673+0000) 2022-09-10T07:19:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:54.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:54.990378+0000) 2022-09-10T07:19:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:55.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:55.366874+0000) 2022-09-10T07:19:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:55 smithi104 ceph-mon[97044]: pgmap v5463: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:55 smithi110 ceph-mon[97633]: pgmap v5463: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:19:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:55.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:55.990616+0000) 2022-09-10T07:19:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:56.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:56.367155+0000) 2022-09-10T07:19:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:56.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:56.990839+0000) 2022-09-10T07:19:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:57.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:57.367389+0000) 2022-09-10T07:19:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:57.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:57.991082+0000) 2022-09-10T07:19:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:19:58 smithi110 ceph-mon[97633]: pgmap v5464: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:19:58 smithi104 ceph-mon[97044]: pgmap v5464: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:19:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:58.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:58.367652+0000) 2022-09-10T07:19:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:58.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:58.991378+0000) 2022-09-10T07:19:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:19:59.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:59.367849+0000) 2022-09-10T07:20:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:19:59.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:19:59.991602+0000) 2022-09-10T07:20:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:00 smithi110 ceph-mon[97633]: pgmap v5465: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:20:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:00 smithi104 ceph-mon[97044]: pgmap v5465: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:20:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:00.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:00.368075+0000) 2022-09-10T07:20:01.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:20:00.950+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:20:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:00.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:00.991816+0000) 2022-09-10T07:20:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:01.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:01.368303+0000) 2022-09-10T07:20:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:01.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:01.991995+0000) 2022-09-10T07:20:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:02 smithi110 ceph-mon[97633]: pgmap v5466: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:02 smithi104 ceph-mon[97044]: pgmap v5466: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:02.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:02.368532+0000) 2022-09-10T07:20:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:02.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:02.992207+0000) 2022-09-10T07:20:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:03.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:03.368754+0000) 2022-09-10T07:20:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:03.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:03.992387+0000) 2022-09-10T07:20:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:04 smithi110 ceph-mon[97633]: pgmap v5467: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:04 smithi104 ceph-mon[97044]: pgmap v5467: 129 pgs: 129 active+clean; 8.2 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:04.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:04.368995+0000) 2022-09-10T07:20:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:04.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:04.992624+0000) 2022-09-10T07:20:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:05.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:05.369236+0000) 2022-09-10T07:20:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:05 smithi104 ceph-mon[97044]: pgmap v5468: 129 pgs: 129 active+clean; 8.2 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:20:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:05 smithi110 ceph-mon[97633]: pgmap v5468: 129 pgs: 129 active+clean; 8.2 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:20:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:05.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:05.992868+0000) 2022-09-10T07:20:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:06.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:06.369467+0000) 2022-09-10T07:20:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:06.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:06.993089+0000) 2022-09-10T07:20:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:07.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:07.369646+0000) 2022-09-10T07:20:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:07.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:07.993280+0000) 2022-09-10T07:20:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:08 smithi110 ceph-mon[97633]: pgmap v5469: 129 pgs: 129 active+clean; 8.2 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:20:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:08 smithi104 ceph-mon[97044]: pgmap v5469: 129 pgs: 129 active+clean; 8.2 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:20:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:08.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:08.369842+0000) 2022-09-10T07:20:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:08.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:08.993470+0000) 2022-09-10T07:20:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:09 smithi110 ceph-mon[97633]: osdmap e422: 12 total, 12 up, 12 in 2022-09-10T07:20:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:09 smithi104 ceph-mon[97044]: osdmap e422: 12 total, 12 up, 12 in 2022-09-10T07:20:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:09.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:09.370064+0000) 2022-09-10T07:20:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:09.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:09.993680+0000) 2022-09-10T07:20:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:10 smithi110 ceph-mon[97633]: pgmap v5471: 129 pgs: 129 active+clean; 8.2 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:20:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:10 smithi104 ceph-mon[97044]: pgmap v5471: 129 pgs: 129 active+clean; 8.2 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:20:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:10.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:10.370350+0000) 2022-09-10T07:20:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:10.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:10.993914+0000) 2022-09-10T07:20:11.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:11.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:11.370605+0000) 2022-09-10T07:20:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:11.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:11.994181+0000) 2022-09-10T07:20:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:12 smithi110 ceph-mon[97633]: pgmap v5472: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:20:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:12 smithi104 ceph-mon[97044]: pgmap v5472: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:20:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:12.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:12.370801+0000) 2022-09-10T07:20:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:12.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:12.994405+0000) 2022-09-10T07:20:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:13.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:13.371021+0000) 2022-09-10T07:20:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:13.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:13.994652+0000) 2022-09-10T07:20:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:14 smithi110 ceph-mon[97633]: pgmap v5473: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:20:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:14 smithi104 ceph-mon[97044]: pgmap v5473: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:20:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:14.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:14.371241+0000) 2022-09-10T07:20:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:14.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:14.994871+0000) 2022-09-10T07:20:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:15.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:15.371482+0000) 2022-09-10T07:20:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:15 smithi104 ceph-mon[97044]: osdmap e423: 12 total, 12 up, 12 in 2022-09-10T07:20:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:15 smithi104 ceph-mon[97044]: pgmap v5475: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:20:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:15 smithi110 ceph-mon[97633]: osdmap e423: 12 total, 12 up, 12 in 2022-09-10T07:20:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:15 smithi110 ceph-mon[97633]: pgmap v5475: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:20:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:15.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:15.995087+0000) 2022-09-10T07:20:16.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:20:15.957+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:20:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:16.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:16.371706+0000) 2022-09-10T07:20:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:16.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:16.995340+0000) 2022-09-10T07:20:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:17.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:17.371980+0000) 2022-09-10T07:20:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:17.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:17.995523+0000) 2022-09-10T07:20:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:18 smithi110 ceph-mon[97633]: pgmap v5476: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:20:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:18 smithi104 ceph-mon[97044]: pgmap v5476: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:20:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:18.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:18.372196+0000) 2022-09-10T07:20:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:18.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:18.995778+0000) 2022-09-10T07:20:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:19.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:19.372367+0000) 2022-09-10T07:20:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:19.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:19.995976+0000) 2022-09-10T07:20:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:20 smithi110 ceph-mon[97633]: pgmap v5477: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:20:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:20 smithi104 ceph-mon[97044]: pgmap v5477: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:20:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:20.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:20.372599+0000) 2022-09-10T07:20:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:20.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:20.996223+0000) 2022-09-10T07:20:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:21.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:21.372718+0000) 2022-09-10T07:20:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:21.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:21.996487+0000) 2022-09-10T07:20:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:22 smithi110 ceph-mon[97633]: pgmap v5478: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:20:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:22 smithi104 ceph-mon[97044]: pgmap v5478: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:20:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:22.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:22.372909+0000) 2022-09-10T07:20:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:22.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:22.996709+0000) 2022-09-10T07:20:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:23.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:23.373157+0000) 2022-09-10T07:20:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:23.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:23.996982+0000) 2022-09-10T07:20:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:24 smithi110 ceph-mon[97633]: pgmap v5479: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:20:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:24 smithi104 ceph-mon[97044]: pgmap v5479: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:20:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:24.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:24.373327+0000) 2022-09-10T07:20:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:24.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:24.997191+0000) 2022-09-10T07:20:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:25.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:25.373566+0000) 2022-09-10T07:20:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:25 smithi104 ceph-mon[97044]: pgmap v5480: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:20:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:25 smithi110 ceph-mon[97633]: pgmap v5480: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:20:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:25.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:25.997456+0000) 2022-09-10T07:20:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:26.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:26.373833+0000) 2022-09-10T07:20:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:26.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:26.997726+0000) 2022-09-10T07:20:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:27.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:27.374084+0000) 2022-09-10T07:20:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:27.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:27.997983+0000) 2022-09-10T07:20:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:28 smithi110 ceph-mon[97633]: pgmap v5481: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:28 smithi104 ceph-mon[97044]: pgmap v5481: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:28.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:28.374274+0000) 2022-09-10T07:20:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:28.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:28.998201+0000) 2022-09-10T07:20:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:29.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:29.374449+0000) 2022-09-10T07:20:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:29.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:29.998390+0000) 2022-09-10T07:20:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:30 smithi110 ceph-mon[97633]: pgmap v5482: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:30 smithi104 ceph-mon[97044]: pgmap v5482: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:30.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:30.374615+0000) 2022-09-10T07:20:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:20:30.971+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:20:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:30.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:30.998656+0000) 2022-09-10T07:20:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:31.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:31.374826+0000) 2022-09-10T07:20:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:31.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:31.998883+0000) 2022-09-10T07:20:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:32 smithi110 ceph-mon[97633]: pgmap v5483: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:32 smithi104 ceph-mon[97044]: pgmap v5483: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:32.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:32.375079+0000) 2022-09-10T07:20:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:32.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:32.999077+0000) 2022-09-10T07:20:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:33.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:33.375329+0000) 2022-09-10T07:20:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:33.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:33.999376+0000) 2022-09-10T07:20:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:34 smithi110 ceph-mon[97633]: pgmap v5484: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:34 smithi104 ceph-mon[97044]: pgmap v5484: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:34.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:34.375533+0000) 2022-09-10T07:20:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:34.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:34.999610+0000) 2022-09-10T07:20:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:35.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:35.375667+0000) 2022-09-10T07:20:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:35 smithi104 ceph-mon[97044]: pgmap v5485: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:35 smithi110 ceph-mon[97633]: pgmap v5485: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:35.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:35.999875+0000) 2022-09-10T07:20:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:36.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:36.375912+0000) 2022-09-10T07:20:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:36.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:37.000167+0000) 2022-09-10T07:20:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:37.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:37.376126+0000) 2022-09-10T07:20:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:37.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:38.000440+0000) 2022-09-10T07:20:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:38 smithi110 ceph-mon[97633]: pgmap v5486: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:38 smithi104 ceph-mon[97044]: pgmap v5486: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:38.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:38.376330+0000) 2022-09-10T07:20:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:38.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:39.000599+0000) 2022-09-10T07:20:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:39.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:39.376576+0000) 2022-09-10T07:20:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:40.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:40.000824+0000) 2022-09-10T07:20:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:40 smithi110 ceph-mon[97633]: pgmap v5487: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:40 smithi104 ceph-mon[97044]: pgmap v5487: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:40.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:40.376834+0000) 2022-09-10T07:20:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:41.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:41.001064+0000) 2022-09-10T07:20:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:41.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:41.377056+0000) 2022-09-10T07:20:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:42.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:42.001295+0000) 2022-09-10T07:20:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:42 smithi110 ceph-mon[97633]: pgmap v5488: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:42 smithi104 ceph-mon[97044]: pgmap v5488: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:42.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:42.377270+0000) 2022-09-10T07:20:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:43.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:43.001511+0000) 2022-09-10T07:20:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:43.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:43.377532+0000) 2022-09-10T07:20:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:44.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:44.001750+0000) 2022-09-10T07:20:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:44 smithi110 ceph-mon[97633]: pgmap v5489: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:44 smithi104 ceph-mon[97044]: pgmap v5489: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:44.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:44.377776+0000) 2022-09-10T07:20:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:45.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:45.001965+0000) 2022-09-10T07:20:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:20:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:20:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:20:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:20:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:20:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:20:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:20:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:20:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:45.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:45.377963+0000) 2022-09-10T07:20:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:46.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:46.002249+0000) 2022-09-10T07:20:46.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:20:45.985+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:20:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:46 smithi110 ceph-mon[97633]: pgmap v5490: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:46 smithi104 ceph-mon[97044]: pgmap v5490: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:46.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:46.378147+0000) 2022-09-10T07:20:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:47.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:47.002446+0000) 2022-09-10T07:20:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:47.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:47.378359+0000) 2022-09-10T07:20:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:48.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:48.002692+0000) 2022-09-10T07:20:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:48 smithi110 ceph-mon[97633]: pgmap v5491: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:48 smithi104 ceph-mon[97044]: pgmap v5491: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:48.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:48.378558+0000) 2022-09-10T07:20:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:49.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:49.002956+0000) 2022-09-10T07:20:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:49.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:49.378760+0000) 2022-09-10T07:20:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:50.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:50.003181+0000) 2022-09-10T07:20:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:50 smithi110 ceph-mon[97633]: pgmap v5492: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:50 smithi104 ceph-mon[97044]: pgmap v5492: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:50.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:50.378982+0000) 2022-09-10T07:20:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:51.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:51.003377+0000) 2022-09-10T07:20:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:51.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:51.379191+0000) 2022-09-10T07:20:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:52.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:52.003559+0000) 2022-09-10T07:20:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:52 smithi110 ceph-mon[97633]: pgmap v5493: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:52 smithi104 ceph-mon[97044]: pgmap v5493: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:52.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:52.379418+0000) 2022-09-10T07:20:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:53.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:53.003742+0000) 2022-09-10T07:20:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:53.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:53.379581+0000) 2022-09-10T07:20:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:54.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:54.003993+0000) 2022-09-10T07:20:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:54 smithi110 ceph-mon[97633]: pgmap v5494: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:54 smithi104 ceph-mon[97044]: pgmap v5494: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:54.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:54.379745+0000) 2022-09-10T07:20:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:55.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:55.004191+0000) 2022-09-10T07:20:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:55.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:55.379962+0000) 2022-09-10T07:20:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:55 smithi104 ceph-mon[97044]: pgmap v5495: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:55 smithi110 ceph-mon[97633]: pgmap v5495: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:20:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:56.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:56.004422+0000) 2022-09-10T07:20:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:56.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:56.380172+0000) 2022-09-10T07:20:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:57.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:57.004653+0000) 2022-09-10T07:20:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:57.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:57.380403+0000) 2022-09-10T07:20:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:58.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:58.004924+0000) 2022-09-10T07:20:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:20:58 smithi110 ceph-mon[97633]: pgmap v5496: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:20:58 smithi104 ceph-mon[97044]: pgmap v5496: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:20:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:58.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:58.380595+0000) 2022-09-10T07:20:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:20:59.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:59.005134+0000) 2022-09-10T07:20:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:20:59.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:20:59.380804+0000) 2022-09-10T07:21:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:00.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:00.005379+0000) 2022-09-10T07:21:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:00 smithi110 ceph-mon[97633]: pgmap v5497: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:00 smithi104 ceph-mon[97044]: pgmap v5497: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:00.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:00.380980+0000) 2022-09-10T07:21:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:01.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:01.005586+0000) 2022-09-10T07:21:01.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:21:00.995+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:21:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:01.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:01.381204+0000) 2022-09-10T07:21:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:02.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:02.005805+0000) 2022-09-10T07:21:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:02 smithi110 ceph-mon[97633]: pgmap v5498: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:02 smithi104 ceph-mon[97044]: pgmap v5498: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:02.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:02.381423+0000) 2022-09-10T07:21:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:03.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:03.006024+0000) 2022-09-10T07:21:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:03.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:03.381563+0000) 2022-09-10T07:21:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:04.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:04.006247+0000) 2022-09-10T07:21:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:04 smithi110 ceph-mon[97633]: pgmap v5499: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:04 smithi104 ceph-mon[97044]: pgmap v5499: 129 pgs: 129 active+clean; 8.2 MiB data, 632 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:04.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:04.381797+0000) 2022-09-10T07:21:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:05.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:05.006443+0000) 2022-09-10T07:21:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:05.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:05.382049+0000) 2022-09-10T07:21:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:05 smithi104 ceph-mon[97044]: pgmap v5500: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:21:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:05 smithi110 ceph-mon[97633]: pgmap v5500: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:21:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:06.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:06.006663+0000) 2022-09-10T07:21:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:06.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:06.382263+0000) 2022-09-10T07:21:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:07.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:07.006849+0000) 2022-09-10T07:21:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:07.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:07.382519+0000) 2022-09-10T07:21:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:08.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:08.007050+0000) 2022-09-10T07:21:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:08 smithi110 ceph-mon[97633]: pgmap v5501: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:21:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:08 smithi104 ceph-mon[97044]: pgmap v5501: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:21:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:08.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:08.382754+0000) 2022-09-10T07:21:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:09.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:09.007213+0000) 2022-09-10T07:21:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:09 smithi110 ceph-mon[97633]: osdmap e424: 12 total, 12 up, 12 in 2022-09-10T07:21:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:09 smithi104 ceph-mon[97044]: osdmap e424: 12 total, 12 up, 12 in 2022-09-10T07:21:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:09.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:09.382990+0000) 2022-09-10T07:21:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:10.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:10.007488+0000) 2022-09-10T07:21:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:10 smithi110 ceph-mon[97633]: pgmap v5503: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:21:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:10 smithi104 ceph-mon[97044]: pgmap v5503: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:21:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:10.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:10.383143+0000) 2022-09-10T07:21:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:11.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:11.007719+0000) 2022-09-10T07:21:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:11.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:11.383315+0000) 2022-09-10T07:21:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:12.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:12.007971+0000) 2022-09-10T07:21:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:12 smithi110 ceph-mon[97633]: pgmap v5504: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:21:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:12 smithi104 ceph-mon[97044]: pgmap v5504: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:21:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:12.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:12.383477+0000) 2022-09-10T07:21:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:13.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:13.008220+0000) 2022-09-10T07:21:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:13.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:13.383663+0000) 2022-09-10T07:21:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:14.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:14.008449+0000) 2022-09-10T07:21:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:14 smithi110 ceph-mon[97633]: pgmap v5505: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:21:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:14 smithi104 ceph-mon[97044]: pgmap v5505: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:21:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:14.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:14.383877+0000) 2022-09-10T07:21:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:15.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:15.008690+0000) 2022-09-10T07:21:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:15.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:15.384164+0000) 2022-09-10T07:21:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:15 smithi104 ceph-mon[97044]: osdmap e425: 12 total, 12 up, 12 in 2022-09-10T07:21:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:15 smithi104 ceph-mon[97044]: pgmap v5507: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:21:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:15 smithi110 ceph-mon[97633]: osdmap e425: 12 total, 12 up, 12 in 2022-09-10T07:21:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:15 smithi110 ceph-mon[97633]: pgmap v5507: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:21:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:16.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:16.008956+0000) 2022-09-10T07:21:16.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:21:16.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:21:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:16.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:16.384344+0000) 2022-09-10T07:21:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:17.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:17.009199+0000) 2022-09-10T07:21:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:17.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:17.384525+0000) 2022-09-10T07:21:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:18.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:18.009440+0000) 2022-09-10T07:21:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:18 smithi110 ceph-mon[97633]: pgmap v5508: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:21:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:18 smithi104 ceph-mon[97044]: pgmap v5508: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:21:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:18.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:18.384693+0000) 2022-09-10T07:21:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:19.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:19.009712+0000) 2022-09-10T07:21:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:19.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:19.384949+0000) 2022-09-10T07:21:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:20.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:20.009963+0000) 2022-09-10T07:21:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:20 smithi110 ceph-mon[97633]: pgmap v5509: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:21:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:20 smithi104 ceph-mon[97044]: pgmap v5509: 129 pgs: 129 active+clean; 8.2 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:21:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:20.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:20.385199+0000) 2022-09-10T07:21:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:21.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:21.010186+0000) 2022-09-10T07:21:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:21.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:21.385392+0000) 2022-09-10T07:21:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:22.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:22.010380+0000) 2022-09-10T07:21:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:22 smithi110 ceph-mon[97633]: pgmap v5510: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T07:21:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:22 smithi104 ceph-mon[97044]: pgmap v5510: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T07:21:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:22.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:22.385601+0000) 2022-09-10T07:21:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:23.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:23.010623+0000) 2022-09-10T07:21:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:23.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:23.385901+0000) 2022-09-10T07:21:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:24.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:24.010842+0000) 2022-09-10T07:21:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:24 smithi110 ceph-mon[97633]: pgmap v5511: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T07:21:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:24 smithi104 ceph-mon[97044]: pgmap v5511: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T07:21:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:24.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:24.386131+0000) 2022-09-10T07:21:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:25.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:25.011056+0000) 2022-09-10T07:21:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:25.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:25.386320+0000) 2022-09-10T07:21:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:25 smithi104 ceph-mon[97044]: pgmap v5512: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T07:21:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:25 smithi110 ceph-mon[97633]: pgmap v5512: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T07:21:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:26.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:26.011259+0000) 2022-09-10T07:21:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:26.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:26.386523+0000) 2022-09-10T07:21:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:27.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:27.011473+0000) 2022-09-10T07:21:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:27.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:27.386770+0000) 2022-09-10T07:21:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:28.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:28.011695+0000) 2022-09-10T07:21:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:28 smithi110 ceph-mon[97633]: pgmap v5513: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T07:21:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:28 smithi104 ceph-mon[97044]: pgmap v5513: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T07:21:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:28.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:28.387015+0000) 2022-09-10T07:21:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:29.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:29.011901+0000) 2022-09-10T07:21:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:29.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:29.387209+0000) 2022-09-10T07:21:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:30.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:30.012181+0000) 2022-09-10T07:21:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:30 smithi110 ceph-mon[97633]: pgmap v5514: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T07:21:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:30 smithi104 ceph-mon[97044]: pgmap v5514: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T07:21:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:30.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:30.387512+0000) 2022-09-10T07:21:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:31.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:31.012367+0000) 2022-09-10T07:21:31.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:21:31.020+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:21:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:31.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:31.387786+0000) 2022-09-10T07:21:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:32.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:32.012585+0000) 2022-09-10T07:21:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:32 smithi110 ceph-mon[97633]: pgmap v5515: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T07:21:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:32 smithi104 ceph-mon[97044]: pgmap v5515: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T07:21:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:32.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:32.388030+0000) 2022-09-10T07:21:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:33.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:33.012826+0000) 2022-09-10T07:21:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:33.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:33.388240+0000) 2022-09-10T07:21:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:34.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:34.013091+0000) 2022-09-10T07:21:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:34 smithi110 ceph-mon[97633]: pgmap v5516: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:21:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:34 smithi104 ceph-mon[97044]: pgmap v5516: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:21:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:34.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:34.388419+0000) 2022-09-10T07:21:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:35.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:35.013378+0000) 2022-09-10T07:21:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:35.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:35.388649+0000) 2022-09-10T07:21:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:35 smithi104 ceph-mon[97044]: pgmap v5517: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:35 smithi110 ceph-mon[97633]: pgmap v5517: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:36.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:36.013592+0000) 2022-09-10T07:21:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:36.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:36.388879+0000) 2022-09-10T07:21:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:37.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:37.013871+0000) 2022-09-10T07:21:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:37.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:37.389122+0000) 2022-09-10T07:21:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:38.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:38.014088+0000) 2022-09-10T07:21:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:38 smithi110 ceph-mon[97633]: pgmap v5518: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:38 smithi104 ceph-mon[97044]: pgmap v5518: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:38.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:38.389394+0000) 2022-09-10T07:21:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:39.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:39.014316+0000) 2022-09-10T07:21:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:39.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:39.389647+0000) 2022-09-10T07:21:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:40.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:40.014519+0000) 2022-09-10T07:21:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:40 smithi110 ceph-mon[97633]: pgmap v5519: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:40 smithi104 ceph-mon[97044]: pgmap v5519: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:40.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:40.389904+0000) 2022-09-10T07:21:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:41.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:41.014753+0000) 2022-09-10T07:21:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:41.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:41.390150+0000) 2022-09-10T07:21:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:42.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:42.014994+0000) 2022-09-10T07:21:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:42 smithi110 ceph-mon[97633]: pgmap v5520: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:42 smithi104 ceph-mon[97044]: pgmap v5520: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:42.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:42.390349+0000) 2022-09-10T07:21:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:43.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:43.015162+0000) 2022-09-10T07:21:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:43.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:43.390598+0000) 2022-09-10T07:21:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:44.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:44.015344+0000) 2022-09-10T07:21:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:44 smithi110 ceph-mon[97633]: pgmap v5521: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:44 smithi104 ceph-mon[97044]: pgmap v5521: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:44.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:44.390848+0000) 2022-09-10T07:21:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:45.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:45.015591+0000) 2022-09-10T07:21:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:21:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:21:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:21:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:21:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:21:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:21:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:21:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:21:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:45.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:45.391160+0000) 2022-09-10T07:21:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:46.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:46.015854+0000) 2022-09-10T07:21:46.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:21:46.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:21:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:46 smithi110 ceph-mon[97633]: pgmap v5522: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:46 smithi104 ceph-mon[97044]: pgmap v5522: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:46.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:46.391327+0000) 2022-09-10T07:21:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:47.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:47.016124+0000) 2022-09-10T07:21:47.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:47.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:47.391568+0000) 2022-09-10T07:21:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:48.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:48.016395+0000) 2022-09-10T07:21:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:48 smithi110 ceph-mon[97633]: pgmap v5523: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:48 smithi104 ceph-mon[97044]: pgmap v5523: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:48.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:48.391765+0000) 2022-09-10T07:21:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:49.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:49.016609+0000) 2022-09-10T07:21:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:49.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:49.392025+0000) 2022-09-10T07:21:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:50.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:50.016753+0000) 2022-09-10T07:21:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:50 smithi110 ceph-mon[97633]: pgmap v5524: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:50 smithi104 ceph-mon[97044]: pgmap v5524: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:50.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:50.392280+0000) 2022-09-10T07:21:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:51.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:51.016934+0000) 2022-09-10T07:21:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:51.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:51.392502+0000) 2022-09-10T07:21:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:52.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:52.017157+0000) 2022-09-10T07:21:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:52 smithi110 ceph-mon[97633]: pgmap v5525: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:52 smithi104 ceph-mon[97044]: pgmap v5525: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:52.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:52.392688+0000) 2022-09-10T07:21:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:53.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:53.017401+0000) 2022-09-10T07:21:53.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:53.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:53.392929+0000) 2022-09-10T07:21:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:54.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:54.017563+0000) 2022-09-10T07:21:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:54 smithi110 ceph-mon[97633]: pgmap v5526: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:54 smithi104 ceph-mon[97044]: pgmap v5526: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:54.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:54.393157+0000) 2022-09-10T07:21:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:55.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:55.017789+0000) 2022-09-10T07:21:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:55.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:55.393454+0000) 2022-09-10T07:21:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:55 smithi104 ceph-mon[97044]: pgmap v5527: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:55 smithi110 ceph-mon[97633]: pgmap v5527: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:21:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:56.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:56.017983+0000) 2022-09-10T07:21:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:56.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:56.393653+0000) 2022-09-10T07:21:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:57.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:57.018265+0000) 2022-09-10T07:21:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:57.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:57.393876+0000) 2022-09-10T07:21:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:58.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:58.018487+0000) 2022-09-10T07:21:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:21:58 smithi110 ceph-mon[97633]: pgmap v5528: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:21:58 smithi104 ceph-mon[97044]: pgmap v5528: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:21:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:58.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:58.394185+0000) 2022-09-10T07:21:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:21:59.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:59.018744+0000) 2022-09-10T07:21:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:21:59.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:21:59.394344+0000) 2022-09-10T07:22:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:00.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:00.018994+0000) 2022-09-10T07:22:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:00 smithi110 ceph-mon[97633]: pgmap v5529: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:00 smithi104 ceph-mon[97044]: pgmap v5529: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:00.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:00.394478+0000) 2022-09-10T07:22:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:01.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:01.019265+0000) 2022-09-10T07:22:01.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:22:01.052+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:22:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:01.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:01.394703+0000) 2022-09-10T07:22:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:02.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:02.019461+0000) 2022-09-10T07:22:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:02 smithi110 ceph-mon[97633]: pgmap v5530: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:02 smithi104 ceph-mon[97044]: pgmap v5530: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:02.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:02.394925+0000) 2022-09-10T07:22:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:03.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:03.019646+0000) 2022-09-10T07:22:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:03.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:03.395198+0000) 2022-09-10T07:22:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:04.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:04.019850+0000) 2022-09-10T07:22:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:04 smithi110 ceph-mon[97633]: pgmap v5531: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:04 smithi104 ceph-mon[97044]: pgmap v5531: 129 pgs: 129 active+clean; 8.2 MiB data, 638 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:04.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:04.395441+0000) 2022-09-10T07:22:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:05.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:05.020088+0000) 2022-09-10T07:22:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:05.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:05.395710+0000) 2022-09-10T07:22:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:05 smithi104 ceph-mon[97044]: pgmap v5532: 129 pgs: 129 active+clean; 8.3 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:22:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:05 smithi110 ceph-mon[97633]: pgmap v5532: 129 pgs: 129 active+clean; 8.3 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:22:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:06.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:06.020281+0000) 2022-09-10T07:22:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:06.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:06.395911+0000) 2022-09-10T07:22:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:07.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:07.020490+0000) 2022-09-10T07:22:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:07.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:07.396195+0000) 2022-09-10T07:22:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:08.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:08.020739+0000) 2022-09-10T07:22:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:08 smithi110 ceph-mon[97633]: pgmap v5533: 129 pgs: 129 active+clean; 8.3 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:22:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:08 smithi104 ceph-mon[97044]: pgmap v5533: 129 pgs: 129 active+clean; 8.3 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:22:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:08.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:08.396429+0000) 2022-09-10T07:22:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:09.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:09.020920+0000) 2022-09-10T07:22:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:09 smithi110 ceph-mon[97633]: osdmap e426: 12 total, 12 up, 12 in 2022-09-10T07:22:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:09 smithi104 ceph-mon[97044]: osdmap e426: 12 total, 12 up, 12 in 2022-09-10T07:22:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:09.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:09.396649+0000) 2022-09-10T07:22:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:10.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:10.021068+0000) 2022-09-10T07:22:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:10 smithi110 ceph-mon[97633]: pgmap v5535: 129 pgs: 129 active+clean; 8.3 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:22:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:10 smithi104 ceph-mon[97044]: pgmap v5535: 129 pgs: 129 active+clean; 8.3 MiB data, 640 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:22:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:10.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:10.396881+0000) 2022-09-10T07:22:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:11.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:11.021293+0000) 2022-09-10T07:22:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:11.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:11.397125+0000) 2022-09-10T07:22:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:12.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:12.021504+0000) 2022-09-10T07:22:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:12 smithi110 ceph-mon[97633]: pgmap v5536: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:22:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:12 smithi104 ceph-mon[97044]: pgmap v5536: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:22:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:12.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:12.397299+0000) 2022-09-10T07:22:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:13.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:13.021746+0000) 2022-09-10T07:22:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:13.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:13.397541+0000) 2022-09-10T07:22:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:14.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:14.022027+0000) 2022-09-10T07:22:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:14 smithi110 ceph-mon[97633]: pgmap v5537: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:22:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:14 smithi104 ceph-mon[97044]: pgmap v5537: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:22:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:14.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:14.397742+0000) 2022-09-10T07:22:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:15.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:15.022252+0000) 2022-09-10T07:22:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:15.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:15.398011+0000) 2022-09-10T07:22:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:15 smithi104 ceph-mon[97044]: osdmap e427: 12 total, 12 up, 12 in 2022-09-10T07:22:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:15 smithi104 ceph-mon[97044]: pgmap v5539: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:22:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:15 smithi110 ceph-mon[97633]: osdmap e427: 12 total, 12 up, 12 in 2022-09-10T07:22:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:15 smithi110 ceph-mon[97633]: pgmap v5539: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:22:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:16.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:16.022502+0000) 2022-09-10T07:22:16.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:22:16.066+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:22:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:16.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:16.398229+0000) 2022-09-10T07:22:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:17.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:17.022748+0000) 2022-09-10T07:22:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:17.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:17.398461+0000) 2022-09-10T07:22:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:18.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:18.022991+0000) 2022-09-10T07:22:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:18 smithi110 ceph-mon[97633]: pgmap v5540: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:22:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:18 smithi104 ceph-mon[97044]: pgmap v5540: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:22:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:18.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:18.398699+0000) 2022-09-10T07:22:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:19.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:19.023188+0000) 2022-09-10T07:22:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:19.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:19.398952+0000) 2022-09-10T07:22:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:20.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:20.023364+0000) 2022-09-10T07:22:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:20 smithi110 ceph-mon[97633]: pgmap v5541: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:22:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:20 smithi104 ceph-mon[97044]: pgmap v5541: 129 pgs: 129 active+clean; 8.3 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:22:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:20.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:20.399163+0000) 2022-09-10T07:22:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:21.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:21.023595+0000) 2022-09-10T07:22:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:21.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:21.399308+0000) 2022-09-10T07:22:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:22.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:22.023852+0000) 2022-09-10T07:22:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:22 smithi110 ceph-mon[97633]: pgmap v5542: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:22:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:22 smithi104 ceph-mon[97044]: pgmap v5542: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:22:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:22.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:22.399525+0000) 2022-09-10T07:22:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:23.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:23.024195+0000) 2022-09-10T07:22:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:23.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:23.399684+0000) 2022-09-10T07:22:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:24.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:24.024333+0000) 2022-09-10T07:22:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:24 smithi110 ceph-mon[97633]: pgmap v5543: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:22:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:24 smithi104 ceph-mon[97044]: pgmap v5543: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:22:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:24.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:24.399916+0000) 2022-09-10T07:22:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:25.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:25.024547+0000) 2022-09-10T07:22:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:25.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:25.400180+0000) 2022-09-10T07:22:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:25 smithi104 ceph-mon[97044]: pgmap v5544: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:22:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:25 smithi110 ceph-mon[97633]: pgmap v5544: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:22:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:26.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:26.024799+0000) 2022-09-10T07:22:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:26.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:26.400368+0000) 2022-09-10T07:22:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:27.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:27.025063+0000) 2022-09-10T07:22:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:27.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:27.400585+0000) 2022-09-10T07:22:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:28.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:28.025354+0000) 2022-09-10T07:22:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:28 smithi110 ceph-mon[97633]: pgmap v5545: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:28 smithi104 ceph-mon[97044]: pgmap v5545: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:28.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:28.400816+0000) 2022-09-10T07:22:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:29.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:29.025559+0000) 2022-09-10T07:22:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:29.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:29.401092+0000) 2022-09-10T07:22:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:30.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:30.025802+0000) 2022-09-10T07:22:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:30 smithi110 ceph-mon[97633]: pgmap v5546: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:30 smithi104 ceph-mon[97044]: pgmap v5546: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:30.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:30.401355+0000) 2022-09-10T07:22:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:22:31.081+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:22:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:31.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:31.026065+0000) 2022-09-10T07:22:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:31.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:31.401585+0000) 2022-09-10T07:22:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:32.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:32.026302+0000) 2022-09-10T07:22:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:32 smithi110 ceph-mon[97633]: pgmap v5547: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:32 smithi104 ceph-mon[97044]: pgmap v5547: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:32.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:32.401840+0000) 2022-09-10T07:22:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:33.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:33.026514+0000) 2022-09-10T07:22:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:33.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:33.402018+0000) 2022-09-10T07:22:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:34.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:34.026770+0000) 2022-09-10T07:22:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:34 smithi110 ceph-mon[97633]: pgmap v5548: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:34 smithi104 ceph-mon[97044]: pgmap v5548: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:34.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:34.402261+0000) 2022-09-10T07:22:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:35.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:35.026997+0000) 2022-09-10T07:22:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:35.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:35.402589+0000) 2022-09-10T07:22:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:35 smithi104 ceph-mon[97044]: pgmap v5549: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:35 smithi110 ceph-mon[97633]: pgmap v5549: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:36.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:36.027253+0000) 2022-09-10T07:22:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:36.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:36.402719+0000) 2022-09-10T07:22:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:37.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:37.027457+0000) 2022-09-10T07:22:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:37.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:37.402918+0000) 2022-09-10T07:22:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:38.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:38.027659+0000) 2022-09-10T07:22:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:38 smithi110 ceph-mon[97633]: pgmap v5550: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:38 smithi104 ceph-mon[97044]: pgmap v5550: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:38.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:38.403181+0000) 2022-09-10T07:22:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:39.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:39.027910+0000) 2022-09-10T07:22:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:39.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:39.403384+0000) 2022-09-10T07:22:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:40.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:40.028125+0000) 2022-09-10T07:22:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:40 smithi110 ceph-mon[97633]: pgmap v5551: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:40 smithi104 ceph-mon[97044]: pgmap v5551: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:40.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:40.403621+0000) 2022-09-10T07:22:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:41.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:41.028359+0000) 2022-09-10T07:22:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:41.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:41.403847+0000) 2022-09-10T07:22:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:42.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:42.028527+0000) 2022-09-10T07:22:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:42 smithi110 ceph-mon[97633]: pgmap v5552: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:42 smithi104 ceph-mon[97044]: pgmap v5552: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:42.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:42.404077+0000) 2022-09-10T07:22:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:43.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:43.028673+0000) 2022-09-10T07:22:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:43.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:43.404342+0000) 2022-09-10T07:22:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:44.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:44.028859+0000) 2022-09-10T07:22:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:44 smithi110 ceph-mon[97633]: pgmap v5553: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:44 smithi104 ceph-mon[97044]: pgmap v5553: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:44.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:44.404597+0000) 2022-09-10T07:22:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:45.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:45.029125+0000) 2022-09-10T07:22:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:22:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:22:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:22:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:22:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:22:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:22:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:22:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:22:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:45.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:45.404851+0000) 2022-09-10T07:22:46.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:22:46.087+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:22:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:46.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:46.029297+0000) 2022-09-10T07:22:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:46 smithi110 ceph-mon[97633]: pgmap v5554: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:46 smithi104 ceph-mon[97044]: pgmap v5554: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:46.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:46.405123+0000) 2022-09-10T07:22:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:47.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:47.029567+0000) 2022-09-10T07:22:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:47.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:47.405340+0000) 2022-09-10T07:22:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:48.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:48.029760+0000) 2022-09-10T07:22:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:48 smithi110 ceph-mon[97633]: pgmap v5555: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:48 smithi104 ceph-mon[97044]: pgmap v5555: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:48.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:48.405555+0000) 2022-09-10T07:22:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:49.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:49.029989+0000) 2022-09-10T07:22:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:49.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:49.405755+0000) 2022-09-10T07:22:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:50.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:50.030173+0000) 2022-09-10T07:22:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:50 smithi110 ceph-mon[97633]: pgmap v5556: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:50 smithi104 ceph-mon[97044]: pgmap v5556: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:50.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:50.406078+0000) 2022-09-10T07:22:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:51.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:51.030408+0000) 2022-09-10T07:22:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:51.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:51.406253+0000) 2022-09-10T07:22:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:52.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:52.030612+0000) 2022-09-10T07:22:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:52 smithi110 ceph-mon[97633]: pgmap v5557: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:52 smithi104 ceph-mon[97044]: pgmap v5557: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:52.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:52.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:52.406509+0000) 2022-09-10T07:22:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:53.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:53.030842+0000) 2022-09-10T07:22:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:53.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:53.406714+0000) 2022-09-10T07:22:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:54.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:54.031079+0000) 2022-09-10T07:22:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:54 smithi110 ceph-mon[97633]: pgmap v5558: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:54 smithi104 ceph-mon[97044]: pgmap v5558: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:54.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:54.406847+0000) 2022-09-10T07:22:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:55.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:55.031311+0000) 2022-09-10T07:22:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:55.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:55.407073+0000) 2022-09-10T07:22:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:55 smithi104 ceph-mon[97044]: pgmap v5559: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:55 smithi110 ceph-mon[97633]: pgmap v5559: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:22:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:56.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:56.031544+0000) 2022-09-10T07:22:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:56.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:56.407346+0000) 2022-09-10T07:22:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:57.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:57.031737+0000) 2022-09-10T07:22:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:57.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:57.407603+0000) 2022-09-10T07:22:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:58.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:58.031969+0000) 2022-09-10T07:22:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:22:58 smithi110 ceph-mon[97633]: pgmap v5560: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:22:58 smithi104 ceph-mon[97044]: pgmap v5560: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:22:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:58.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:58.407919+0000) 2022-09-10T07:22:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:22:59.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:59.032187+0000) 2022-09-10T07:22:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:22:59.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:22:59.408076+0000) 2022-09-10T07:23:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:00.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:00.032393+0000) 2022-09-10T07:23:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:00 smithi110 ceph-mon[97633]: pgmap v5561: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:00 smithi104 ceph-mon[97044]: pgmap v5561: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:00.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:00.408274+0000) 2022-09-10T07:23:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:01.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:01.032595+0000) 2022-09-10T07:23:01.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:23:01.103+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:23:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:01.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:01.408464+0000) 2022-09-10T07:23:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:02.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:02.032850+0000) 2022-09-10T07:23:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:02 smithi110 ceph-mon[97633]: pgmap v5562: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:02 smithi104 ceph-mon[97044]: pgmap v5562: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:02.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:02.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:02.408646+0000) 2022-09-10T07:23:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:03.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:03.033083+0000) 2022-09-10T07:23:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:03.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:03.408876+0000) 2022-09-10T07:23:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:04.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:04.033299+0000) 2022-09-10T07:23:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:04 smithi110 ceph-mon[97633]: pgmap v5563: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:04 smithi104 ceph-mon[97044]: pgmap v5563: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:04.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:04.409072+0000) 2022-09-10T07:23:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:05.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:05.033517+0000) 2022-09-10T07:23:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:05.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:05.409351+0000) 2022-09-10T07:23:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:05 smithi104 ceph-mon[97044]: pgmap v5564: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:23:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:05 smithi110 ceph-mon[97633]: pgmap v5564: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:23:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:06.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:06.033741+0000) 2022-09-10T07:23:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:06.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:06.409531+0000) 2022-09-10T07:23:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:07.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:07.033967+0000) 2022-09-10T07:23:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:07.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:07.409697+0000) 2022-09-10T07:23:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:08.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:08.034243+0000) 2022-09-10T07:23:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:08 smithi110 ceph-mon[97633]: pgmap v5565: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:23:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:08 smithi104 ceph-mon[97044]: pgmap v5565: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:23:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:08.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:08.409919+0000) 2022-09-10T07:23:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:09.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:09.034472+0000) 2022-09-10T07:23:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:09 smithi110 ceph-mon[97633]: osdmap e428: 12 total, 12 up, 12 in 2022-09-10T07:23:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:09 smithi104 ceph-mon[97044]: osdmap e428: 12 total, 12 up, 12 in 2022-09-10T07:23:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:09.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:09.410178+0000) 2022-09-10T07:23:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:10.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:10.034720+0000) 2022-09-10T07:23:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:10 smithi110 ceph-mon[97633]: pgmap v5567: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:23:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:10 smithi104 ceph-mon[97044]: pgmap v5567: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:23:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:10.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:10.410452+0000) 2022-09-10T07:23:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:11.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:11.034982+0000) 2022-09-10T07:23:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:11.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:11.410587+0000) 2022-09-10T07:23:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:12.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:12.035242+0000) 2022-09-10T07:23:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:12 smithi110 ceph-mon[97633]: pgmap v5568: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:23:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:12 smithi104 ceph-mon[97044]: pgmap v5568: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:23:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:12.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:12.410840+0000) 2022-09-10T07:23:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:13.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:13.035473+0000) 2022-09-10T07:23:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:13.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:13.411071+0000) 2022-09-10T07:23:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:14.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:14.035700+0000) 2022-09-10T07:23:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:14 smithi110 ceph-mon[97633]: pgmap v5569: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:23:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:14 smithi104 ceph-mon[97044]: pgmap v5569: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:23:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:14.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:14.411331+0000) 2022-09-10T07:23:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:15.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:15.035954+0000) 2022-09-10T07:23:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:15.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:15.411486+0000) 2022-09-10T07:23:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:15 smithi104 ceph-mon[97044]: osdmap e429: 12 total, 12 up, 12 in 2022-09-10T07:23:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:15 smithi104 ceph-mon[97044]: pgmap v5571: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:23:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:15 smithi110 ceph-mon[97633]: osdmap e429: 12 total, 12 up, 12 in 2022-09-10T07:23:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:15 smithi110 ceph-mon[97633]: pgmap v5571: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:23:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:16.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:16.036212+0000) 2022-09-10T07:23:16.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:23:16.118+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:23:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:16.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:16.411736+0000) 2022-09-10T07:23:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:17.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:17.036407+0000) 2022-09-10T07:23:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:17.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:17.411984+0000) 2022-09-10T07:23:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:18.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:18.036639+0000) 2022-09-10T07:23:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:18 smithi110 ceph-mon[97633]: pgmap v5572: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:23:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:18 smithi104 ceph-mon[97044]: pgmap v5572: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:23:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:18.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:18.412205+0000) 2022-09-10T07:23:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:19.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:19.036906+0000) 2022-09-10T07:23:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:19.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:19.412345+0000) 2022-09-10T07:23:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:20.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:20.037183+0000) 2022-09-10T07:23:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:20 smithi110 ceph-mon[97633]: pgmap v5573: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:23:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:20 smithi104 ceph-mon[97044]: pgmap v5573: 129 pgs: 129 active+clean; 8.3 MiB data, 650 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:23:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:20.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:20.412498+0000) 2022-09-10T07:23:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:21.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:21.037406+0000) 2022-09-10T07:23:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:21.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:21.412729+0000) 2022-09-10T07:23:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:22.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:22.037636+0000) 2022-09-10T07:23:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:22 smithi110 ceph-mon[97633]: pgmap v5574: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:23:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:22 smithi104 ceph-mon[97044]: pgmap v5574: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:23:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:22.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:22.412904+0000) 2022-09-10T07:23:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:23.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:23.037842+0000) 2022-09-10T07:23:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:23.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:23.413089+0000) 2022-09-10T07:23:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:24.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:24.038056+0000) 2022-09-10T07:23:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:24 smithi110 ceph-mon[97633]: pgmap v5575: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:23:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:24 smithi104 ceph-mon[97044]: pgmap v5575: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:23:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:24.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:24.413257+0000) 2022-09-10T07:23:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:25.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:25.038347+0000) 2022-09-10T07:23:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:25.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:25.413433+0000) 2022-09-10T07:23:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:25 smithi104 ceph-mon[97044]: pgmap v5576: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:23:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:25 smithi110 ceph-mon[97633]: pgmap v5576: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:23:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:26.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:26.038558+0000) 2022-09-10T07:23:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:26.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:26.413646+0000) 2022-09-10T07:23:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:27.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:27.038760+0000) 2022-09-10T07:23:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:27.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:27.413861+0000) 2022-09-10T07:23:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:28.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:28.038903+0000) 2022-09-10T07:23:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:28 smithi110 ceph-mon[97633]: pgmap v5577: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:28 smithi104 ceph-mon[97044]: pgmap v5577: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:28.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:28.414077+0000) 2022-09-10T07:23:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:29.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:29.039174+0000) 2022-09-10T07:23:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:29.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:29.414351+0000) 2022-09-10T07:23:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:30.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:30.039393+0000) 2022-09-10T07:23:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:30 smithi110 ceph-mon[97633]: pgmap v5578: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:30 smithi104 ceph-mon[97044]: pgmap v5578: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:30.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:30.414591+0000) 2022-09-10T07:23:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:31.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:31.039621+0000) 2022-09-10T07:23:31.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:23:31.132+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:23:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:31.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:31.414811+0000) 2022-09-10T07:23:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:32.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:32.039791+0000) 2022-09-10T07:23:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:32 smithi110 ceph-mon[97633]: pgmap v5579: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:32.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:32 smithi104 ceph-mon[97044]: pgmap v5579: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:32.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:32.415010+0000) 2022-09-10T07:23:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:33.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:33.039967+0000) 2022-09-10T07:23:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:33.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:33.415173+0000) 2022-09-10T07:23:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:34.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:34.040234+0000) 2022-09-10T07:23:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:34 smithi110 ceph-mon[97633]: pgmap v5580: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:34 smithi104 ceph-mon[97044]: pgmap v5580: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:34.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:34.415330+0000) 2022-09-10T07:23:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:35.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:35.040440+0000) 2022-09-10T07:23:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:35.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:35.415575+0000) 2022-09-10T07:23:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:35 smithi104 ceph-mon[97044]: pgmap v5581: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:35 smithi110 ceph-mon[97633]: pgmap v5581: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:36.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:36.040705+0000) 2022-09-10T07:23:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:36.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:36.415797+0000) 2022-09-10T07:23:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:37.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:37.040969+0000) 2022-09-10T07:23:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:37.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:37.416022+0000) 2022-09-10T07:23:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:38.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:38.041214+0000) 2022-09-10T07:23:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:38 smithi110 ceph-mon[97633]: pgmap v5582: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:38 smithi104 ceph-mon[97044]: pgmap v5582: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:38.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:38.416282+0000) 2022-09-10T07:23:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:39.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:39.041435+0000) 2022-09-10T07:23:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:39.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:39.416496+0000) 2022-09-10T07:23:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:40.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:40.041650+0000) 2022-09-10T07:23:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:40 smithi110 ceph-mon[97633]: pgmap v5583: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:40 smithi104 ceph-mon[97044]: pgmap v5583: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:40.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:40.416698+0000) 2022-09-10T07:23:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:41.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:41.041893+0000) 2022-09-10T07:23:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:41.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:41.416875+0000) 2022-09-10T07:23:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:42.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:42.042196+0000) 2022-09-10T07:23:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:42 smithi110 ceph-mon[97633]: pgmap v5584: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:42 smithi104 ceph-mon[97044]: pgmap v5584: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:42.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:42.417130+0000) 2022-09-10T07:23:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:43.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:43.042379+0000) 2022-09-10T07:23:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:43.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:43.417321+0000) 2022-09-10T07:23:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:44.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:44.042541+0000) 2022-09-10T07:23:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:44 smithi110 ceph-mon[97633]: pgmap v5585: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:44 smithi104 ceph-mon[97044]: pgmap v5585: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:44.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:44.417483+0000) 2022-09-10T07:23:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:45.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:45.042770+0000) 2022-09-10T07:23:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:23:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:23:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:23:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:23:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:23:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:23:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:23:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:23:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:45.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:45.417729+0000) 2022-09-10T07:23:46.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:23:46.134+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:23:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:46.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:46.042913+0000) 2022-09-10T07:23:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:46 smithi104 ceph-mon[97044]: pgmap v5586: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:46 smithi110 ceph-mon[97633]: pgmap v5586: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:46.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:46.417914+0000) 2022-09-10T07:23:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:47.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:47.043121+0000) 2022-09-10T07:23:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:47.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:47.418167+0000) 2022-09-10T07:23:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:48.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:48.043354+0000) 2022-09-10T07:23:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:48 smithi104 ceph-mon[97044]: pgmap v5587: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:48 smithi110 ceph-mon[97633]: pgmap v5587: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:48.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:48.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:48.418287+0000) 2022-09-10T07:23:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:49.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:49.043546+0000) 2022-09-10T07:23:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:49.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:49.418512+0000) 2022-09-10T07:23:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:50.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:50.043722+0000) 2022-09-10T07:23:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:50 smithi104 ceph-mon[97044]: pgmap v5588: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:50 smithi110 ceph-mon[97633]: pgmap v5588: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:50.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:50.418755+0000) 2022-09-10T07:23:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:51.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:51.043939+0000) 2022-09-10T07:23:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:51.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:51.418999+0000) 2022-09-10T07:23:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:52.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:52.044189+0000) 2022-09-10T07:23:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:52 smithi104 ceph-mon[97044]: pgmap v5589: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:52 smithi110 ceph-mon[97633]: pgmap v5589: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:52.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:52.419245+0000) 2022-09-10T07:23:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:53.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:53.044438+0000) 2022-09-10T07:23:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:53.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:53.419433+0000) 2022-09-10T07:23:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:54.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:54.044678+0000) 2022-09-10T07:23:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:54 smithi104 ceph-mon[97044]: pgmap v5590: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:54 smithi110 ceph-mon[97633]: pgmap v5590: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:54.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:54.419637+0000) 2022-09-10T07:23:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:55.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:55.044938+0000) 2022-09-10T07:23:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:55.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:55.419831+0000) 2022-09-10T07:23:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:55 smithi104 ceph-mon[97044]: pgmap v5591: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:55 smithi110 ceph-mon[97633]: pgmap v5591: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:23:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:56.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:56.045203+0000) 2022-09-10T07:23:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:56.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:56.420057+0000) 2022-09-10T07:23:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:57.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:57.045409+0000) 2022-09-10T07:23:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:57.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:57.420323+0000) 2022-09-10T07:23:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:58.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:58.045595+0000) 2022-09-10T07:23:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:23:58 smithi110 ceph-mon[97633]: pgmap v5592: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:23:58 smithi104 ceph-mon[97044]: pgmap v5592: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:23:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:58.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:58.420521+0000) 2022-09-10T07:23:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:23:59.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:59.045790+0000) 2022-09-10T07:23:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:23:59.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:23:59.420670+0000) 2022-09-10T07:24:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:00.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:00.045973+0000) 2022-09-10T07:24:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:00 smithi110 ceph-mon[97633]: pgmap v5593: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:00 smithi104 ceph-mon[97044]: pgmap v5593: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:00.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:00.420836+0000) 2022-09-10T07:24:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:01.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:01.046214+0000) 2022-09-10T07:24:01.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:24:01.145+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:24:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:01.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:01.421075+0000) 2022-09-10T07:24:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:02.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:02.046393+0000) 2022-09-10T07:24:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:02 smithi104 ceph-mon[97044]: pgmap v5594: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:02 smithi110 ceph-mon[97633]: pgmap v5594: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:02.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:02.421348+0000) 2022-09-10T07:24:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:03.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:03.046630+0000) 2022-09-10T07:24:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:03.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:03.421572+0000) 2022-09-10T07:24:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:04.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:04.046846+0000) 2022-09-10T07:24:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:04 smithi104 ceph-mon[97044]: pgmap v5595: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:04 smithi110 ceph-mon[97633]: pgmap v5595: 129 pgs: 129 active+clean; 8.3 MiB data, 651 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:04.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:04.421823+0000) 2022-09-10T07:24:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:05.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:05.047156+0000) 2022-09-10T07:24:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:05.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:05.422011+0000) 2022-09-10T07:24:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:05 smithi104 ceph-mon[97044]: pgmap v5596: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:24:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:05 smithi110 ceph-mon[97633]: pgmap v5596: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:24:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:06.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:06.047380+0000) 2022-09-10T07:24:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:06.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:06.422273+0000) 2022-09-10T07:24:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:07.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:07.047538+0000) 2022-09-10T07:24:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:07.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:07.422518+0000) 2022-09-10T07:24:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:08.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:08.047744+0000) 2022-09-10T07:24:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:08 smithi104 ceph-mon[97044]: pgmap v5597: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:24:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:08 smithi110 ceph-mon[97633]: pgmap v5597: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:24:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:08.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:08.422697+0000) 2022-09-10T07:24:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:09.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:09.047956+0000) 2022-09-10T07:24:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:09 smithi104 ceph-mon[97044]: osdmap e430: 12 total, 12 up, 12 in 2022-09-10T07:24:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:09 smithi110 ceph-mon[97633]: osdmap e430: 12 total, 12 up, 12 in 2022-09-10T07:24:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:09.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:09.422943+0000) 2022-09-10T07:24:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:10.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:10.048168+0000) 2022-09-10T07:24:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:10 smithi104 ceph-mon[97044]: pgmap v5599: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:24:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:10 smithi110 ceph-mon[97633]: pgmap v5599: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:24:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:10.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:10.423203+0000) 2022-09-10T07:24:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:11.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:11.048350+0000) 2022-09-10T07:24:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:11.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:11.423398+0000) 2022-09-10T07:24:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:12.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:12.048548+0000) 2022-09-10T07:24:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:12 smithi104 ceph-mon[97044]: pgmap v5600: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:24:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:12 smithi110 ceph-mon[97633]: pgmap v5600: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:24:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:12.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:12.423667+0000) 2022-09-10T07:24:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:13.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:13.048746+0000) 2022-09-10T07:24:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:13.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:13.423946+0000) 2022-09-10T07:24:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:14.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:14.048988+0000) 2022-09-10T07:24:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:14 smithi104 ceph-mon[97044]: pgmap v5601: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:24:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:14 smithi110 ceph-mon[97633]: pgmap v5601: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:24:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:14.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:14.424198+0000) 2022-09-10T07:24:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:15.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:15.049261+0000) 2022-09-10T07:24:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:15.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:15.424404+0000) 2022-09-10T07:24:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:15 smithi104 ceph-mon[97044]: osdmap e431: 12 total, 12 up, 12 in 2022-09-10T07:24:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:15 smithi104 ceph-mon[97044]: pgmap v5603: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:24:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:15 smithi110 ceph-mon[97633]: osdmap e431: 12 total, 12 up, 12 in 2022-09-10T07:24:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:15 smithi110 ceph-mon[97633]: pgmap v5603: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:24:16.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:24:16.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:24:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:16.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:16.049517+0000) 2022-09-10T07:24:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:16.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:16.424634+0000) 2022-09-10T07:24:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:17.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:17.049768+0000) 2022-09-10T07:24:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:17.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:17.424887+0000) 2022-09-10T07:24:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:18.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:18.049968+0000) 2022-09-10T07:24:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:18 smithi104 ceph-mon[97044]: pgmap v5604: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:24:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:18 smithi110 ceph-mon[97633]: pgmap v5604: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:24:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:18.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:18.425141+0000) 2022-09-10T07:24:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:19.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:19.050204+0000) 2022-09-10T07:24:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:19.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:19.425351+0000) 2022-09-10T07:24:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:20.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:20.050481+0000) 2022-09-10T07:24:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:20 smithi104 ceph-mon[97044]: pgmap v5605: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:24:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:20 smithi110 ceph-mon[97633]: pgmap v5605: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:24:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:20.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:20.425533+0000) 2022-09-10T07:24:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:21.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:21.050723+0000) 2022-09-10T07:24:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:21.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:21.425802+0000) 2022-09-10T07:24:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:22.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:22.050943+0000) 2022-09-10T07:24:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:22 smithi104 ceph-mon[97044]: pgmap v5606: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:24:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:22 smithi110 ceph-mon[97633]: pgmap v5606: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:24:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:22.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:22.426058+0000) 2022-09-10T07:24:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:23.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:23.051243+0000) 2022-09-10T07:24:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:23.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:23.426331+0000) 2022-09-10T07:24:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:24.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:24.051424+0000) 2022-09-10T07:24:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:24 smithi104 ceph-mon[97044]: pgmap v5607: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:24:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:24 smithi110 ceph-mon[97633]: pgmap v5607: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:24:24.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:24.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:24.426580+0000) 2022-09-10T07:24:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:25.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:25.051638+0000) 2022-09-10T07:24:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:25.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:25.426834+0000) 2022-09-10T07:24:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:25 smithi104 ceph-mon[97044]: pgmap v5608: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:24:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:25 smithi110 ceph-mon[97633]: pgmap v5608: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:24:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:26.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:26.051884+0000) 2022-09-10T07:24:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:26.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:26.427075+0000) 2022-09-10T07:24:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:27.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:27.052169+0000) 2022-09-10T07:24:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:27.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:27.427332+0000) 2022-09-10T07:24:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:28.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:28.052361+0000) 2022-09-10T07:24:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:28 smithi104 ceph-mon[97044]: pgmap v5609: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:28 smithi110 ceph-mon[97633]: pgmap v5609: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:28.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:28.427494+0000) 2022-09-10T07:24:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:29.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:29.052616+0000) 2022-09-10T07:24:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:29.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:29.427739+0000) 2022-09-10T07:24:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:30.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:30.052894+0000) 2022-09-10T07:24:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:30 smithi104 ceph-mon[97044]: pgmap v5610: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:30 smithi110 ceph-mon[97633]: pgmap v5610: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:30.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:30.427941+0000) 2022-09-10T07:24:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:24:31.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:24:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:31.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:31.053201+0000) 2022-09-10T07:24:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:31.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:31.428190+0000) 2022-09-10T07:24:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:32.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:32.053367+0000) 2022-09-10T07:24:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:32 smithi104 ceph-mon[97044]: pgmap v5611: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:32 smithi110 ceph-mon[97633]: pgmap v5611: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:32.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:32.428409+0000) 2022-09-10T07:24:33.429 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:33.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:33.053609+0000) 2022-09-10T07:24:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:33.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:33.428636+0000) 2022-09-10T07:24:34.429 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:34.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:34.053846+0000) 2022-09-10T07:24:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:34 smithi104 ceph-mon[97044]: pgmap v5612: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:34 smithi110 ceph-mon[97633]: pgmap v5612: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:34.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:34.428829+0000) 2022-09-10T07:24:35.429 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:35.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:35.054068+0000) 2022-09-10T07:24:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:35.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:35.429046+0000) 2022-09-10T07:24:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:35 smithi104 ceph-mon[97044]: pgmap v5613: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:35 smithi110 ceph-mon[97633]: pgmap v5613: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:36.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:36.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:36.054308+0000) 2022-09-10T07:24:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:36.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:36.429277+0000) 2022-09-10T07:24:37.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:37.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:37.054535+0000) 2022-09-10T07:24:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:37.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:37.429471+0000) 2022-09-10T07:24:38.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:38.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:38.054773+0000) 2022-09-10T07:24:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:38 smithi104 ceph-mon[97044]: pgmap v5614: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:38 smithi110 ceph-mon[97633]: pgmap v5614: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:38.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:38.429718+0000) 2022-09-10T07:24:39.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:39.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:39.054999+0000) 2022-09-10T07:24:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:39.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:39.430015+0000) 2022-09-10T07:24:40.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:40.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:40.055232+0000) 2022-09-10T07:24:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:40 smithi104 ceph-mon[97044]: pgmap v5615: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:40 smithi110 ceph-mon[97633]: pgmap v5615: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:40.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:40.430214+0000) 2022-09-10T07:24:41.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:41.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:41.055481+0000) 2022-09-10T07:24:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:41.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:41.430423+0000) 2022-09-10T07:24:42.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:42.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:42.055725+0000) 2022-09-10T07:24:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:42 smithi104 ceph-mon[97044]: pgmap v5616: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:42 smithi110 ceph-mon[97633]: pgmap v5616: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:42.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:42.430583+0000) 2022-09-10T07:24:43.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:43.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:43.055950+0000) 2022-09-10T07:24:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:43.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:43.430761+0000) 2022-09-10T07:24:44.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:44.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:44.056193+0000) 2022-09-10T07:24:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:44 smithi104 ceph-mon[97044]: pgmap v5617: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:44 smithi110 ceph-mon[97633]: pgmap v5617: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:44.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:44.431006+0000) 2022-09-10T07:24:45.432 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:45.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:45.056394+0000) 2022-09-10T07:24:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:24:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:24:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:24:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:24:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:24:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:24:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:24:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:24:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:45.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:45.431274+0000) 2022-09-10T07:24:46.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:46.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:46.056627+0000) 2022-09-10T07:24:46.434 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:24:46.166+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:24:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:46 smithi104 ceph-mon[97044]: pgmap v5618: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:46 smithi110 ceph-mon[97633]: pgmap v5618: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:46.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:46.431502+0000) 2022-09-10T07:24:47.432 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:47.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:47.056879+0000) 2022-09-10T07:24:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:47.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:47.431721+0000) 2022-09-10T07:24:48.432 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:48.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:48.057136+0000) 2022-09-10T07:24:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:48 smithi104 ceph-mon[97044]: pgmap v5619: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:48 smithi110 ceph-mon[97633]: pgmap v5619: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:48.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:48.431963+0000) 2022-09-10T07:24:49.432 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:49.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:49.057309+0000) 2022-09-10T07:24:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:49.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:49.432237+0000) 2022-09-10T07:24:50.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:50.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:50.057555+0000) 2022-09-10T07:24:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:50 smithi104 ceph-mon[97044]: pgmap v5620: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:50 smithi110 ceph-mon[97633]: pgmap v5620: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:50.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:50.432467+0000) 2022-09-10T07:24:51.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:51.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:51.057771+0000) 2022-09-10T07:24:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:51.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:51.432631+0000) 2022-09-10T07:24:52.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:52.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:52.058021+0000) 2022-09-10T07:24:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:52 smithi104 ceph-mon[97044]: pgmap v5621: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:52 smithi110 ceph-mon[97633]: pgmap v5621: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:52.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:52.432837+0000) 2022-09-10T07:24:53.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:53.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:53.058284+0000) 2022-09-10T07:24:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:53.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:53.433068+0000) 2022-09-10T07:24:54.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:54.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:54.058502+0000) 2022-09-10T07:24:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:54 smithi104 ceph-mon[97044]: pgmap v5622: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:54 smithi110 ceph-mon[97633]: pgmap v5622: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:54.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:54.433294+0000) 2022-09-10T07:24:55.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:55.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:55.058700+0000) 2022-09-10T07:24:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:55.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:55.433498+0000) 2022-09-10T07:24:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:55 smithi104 ceph-mon[97044]: pgmap v5623: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:55 smithi110 ceph-mon[97633]: pgmap v5623: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:24:56.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:56.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:56.058872+0000) 2022-09-10T07:24:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:56.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:56.433653+0000) 2022-09-10T07:24:57.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:57.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:57.059166+0000) 2022-09-10T07:24:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:57.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:57.433895+0000) 2022-09-10T07:24:58.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:58.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:58.059303+0000) 2022-09-10T07:24:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:24:58 smithi104 ceph-mon[97044]: pgmap v5624: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:24:58 smithi110 ceph-mon[97633]: pgmap v5624: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:24:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:58.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:58.434134+0000) 2022-09-10T07:24:59.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:24:59.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:59.059523+0000) 2022-09-10T07:24:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:24:59.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:24:59.434350+0000) 2022-09-10T07:25:00.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:00.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:00.059760+0000) 2022-09-10T07:25:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:00 smithi104 ceph-mon[97044]: pgmap v5625: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:00 smithi110 ceph-mon[97633]: pgmap v5625: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:00.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:00.434509+0000) 2022-09-10T07:25:01.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:01.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:01.059997+0000) 2022-09-10T07:25:01.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:25:01.179+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:25:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:01.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:01.434751+0000) 2022-09-10T07:25:02.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:02.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:02.060204+0000) 2022-09-10T07:25:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:02 smithi104 ceph-mon[97044]: pgmap v5626: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:02 smithi110 ceph-mon[97633]: pgmap v5626: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:02.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:02.434919+0000) 2022-09-10T07:25:03.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:03.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:03.060422+0000) 2022-09-10T07:25:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:03.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:03.435046+0000) 2022-09-10T07:25:04.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:04.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:04.060608+0000) 2022-09-10T07:25:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:04 smithi104 ceph-mon[97044]: pgmap v5627: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:04 smithi110 ceph-mon[97633]: pgmap v5627: 129 pgs: 129 active+clean; 8.3 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:04.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:04.435209+0000) 2022-09-10T07:25:05.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:05.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:05.060826+0000) 2022-09-10T07:25:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:05.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:05.435497+0000) 2022-09-10T07:25:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:05 smithi104 ceph-mon[97044]: pgmap v5628: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:25:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:05 smithi110 ceph-mon[97633]: pgmap v5628: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:25:06.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:06.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:06.061062+0000) 2022-09-10T07:25:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:06.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:06.435696+0000) 2022-09-10T07:25:07.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:07.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:07.061222+0000) 2022-09-10T07:25:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:07.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:07.435867+0000) 2022-09-10T07:25:08.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:08.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:08.061401+0000) 2022-09-10T07:25:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:08 smithi104 ceph-mon[97044]: pgmap v5629: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:25:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:08 smithi110 ceph-mon[97633]: pgmap v5629: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:25:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:08.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:08.436040+0000) 2022-09-10T07:25:09.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:09.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:09.061591+0000) 2022-09-10T07:25:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:09 smithi104 ceph-mon[97044]: osdmap e432: 12 total, 12 up, 12 in 2022-09-10T07:25:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:09 smithi110 ceph-mon[97633]: osdmap e432: 12 total, 12 up, 12 in 2022-09-10T07:25:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:09.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:09.436288+0000) 2022-09-10T07:25:10.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:10.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:10.061764+0000) 2022-09-10T07:25:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:10 smithi104 ceph-mon[97044]: pgmap v5631: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:25:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:10 smithi110 ceph-mon[97633]: pgmap v5631: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:25:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:10.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:10.436450+0000) 2022-09-10T07:25:11.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:11.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:11.062017+0000) 2022-09-10T07:25:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:11.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:11.436679+0000) 2022-09-10T07:25:12.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:12.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:12.062267+0000) 2022-09-10T07:25:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:12 smithi104 ceph-mon[97044]: pgmap v5632: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:25:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:12 smithi110 ceph-mon[97633]: pgmap v5632: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:25:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:12.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:12.436856+0000) 2022-09-10T07:25:13.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:13.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:13.062508+0000) 2022-09-10T07:25:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:13.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:13.437090+0000) 2022-09-10T07:25:14.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:14.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:14.062749+0000) 2022-09-10T07:25:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:14 smithi104 ceph-mon[97044]: pgmap v5633: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:25:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:14 smithi110 ceph-mon[97633]: pgmap v5633: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:25:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:14.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:14.437305+0000) 2022-09-10T07:25:15.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:15.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:15.062989+0000) 2022-09-10T07:25:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:15.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:15.437460+0000) 2022-09-10T07:25:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:15 smithi104 ceph-mon[97044]: osdmap e433: 12 total, 12 up, 12 in 2022-09-10T07:25:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:15 smithi104 ceph-mon[97044]: pgmap v5635: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:25:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:15 smithi110 ceph-mon[97633]: osdmap e433: 12 total, 12 up, 12 in 2022-09-10T07:25:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:15 smithi110 ceph-mon[97633]: pgmap v5635: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:25:16.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:16.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:16.063172+0000) 2022-09-10T07:25:16.439 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:25:16.195+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:25:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:16.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:16.437707+0000) 2022-09-10T07:25:17.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:17.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:17.063369+0000) 2022-09-10T07:25:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:17.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:17.437992+0000) 2022-09-10T07:25:18.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:18.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:18.063563+0000) 2022-09-10T07:25:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:18 smithi104 ceph-mon[97044]: pgmap v5636: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:25:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:18 smithi110 ceph-mon[97633]: pgmap v5636: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T07:25:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:18.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:18.438270+0000) 2022-09-10T07:25:19.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:19.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:19.063779+0000) 2022-09-10T07:25:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:19.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:19.438484+0000) 2022-09-10T07:25:20.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:20.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:20.063994+0000) 2022-09-10T07:25:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:20 smithi104 ceph-mon[97044]: pgmap v5637: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:25:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:20 smithi110 ceph-mon[97633]: pgmap v5637: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:25:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:20.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:20.438707+0000) 2022-09-10T07:25:21.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:21.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:21.064206+0000) 2022-09-10T07:25:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:21.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:21.438859+0000) 2022-09-10T07:25:22.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:22.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:22.064424+0000) 2022-09-10T07:25:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:22 smithi104 ceph-mon[97044]: pgmap v5638: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:25:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:22 smithi110 ceph-mon[97633]: pgmap v5638: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:25:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:22.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:22.439138+0000) 2022-09-10T07:25:23.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:23.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:23.064681+0000) 2022-09-10T07:25:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:23.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:23.439334+0000) 2022-09-10T07:25:24.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:24.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:24.064878+0000) 2022-09-10T07:25:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:24 smithi104 ceph-mon[97044]: pgmap v5639: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:25:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:24 smithi110 ceph-mon[97633]: pgmap v5639: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:25:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:24.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:24.439560+0000) 2022-09-10T07:25:25.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:25.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:25.065125+0000) 2022-09-10T07:25:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:25.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:25.439801+0000) 2022-09-10T07:25:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:25 smithi104 ceph-mon[97044]: pgmap v5640: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:25:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:25 smithi110 ceph-mon[97633]: pgmap v5640: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:25:26.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:26.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:26.065370+0000) 2022-09-10T07:25:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:26.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:26.440055+0000) 2022-09-10T07:25:27.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:27.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:27.065637+0000) 2022-09-10T07:25:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:27.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:27.440261+0000) 2022-09-10T07:25:28.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:28.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:28.065864+0000) 2022-09-10T07:25:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:28 smithi104 ceph-mon[97044]: pgmap v5641: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:28 smithi110 ceph-mon[97633]: pgmap v5641: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:28.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:28.440451+0000) 2022-09-10T07:25:29.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:29.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:29.066082+0000) 2022-09-10T07:25:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:29.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:29.440645+0000) 2022-09-10T07:25:30.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:30.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:30.066353+0000) 2022-09-10T07:25:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:30 smithi104 ceph-mon[97044]: pgmap v5642: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:30 smithi110 ceph-mon[97633]: pgmap v5642: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:30.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:30.440854+0000) 2022-09-10T07:25:31.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:31.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:31.066563+0000) 2022-09-10T07:25:31.442 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:25:31.197+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:25:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:31.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:31.441036+0000) 2022-09-10T07:25:32.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:32.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:32.066756+0000) 2022-09-10T07:25:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:32 smithi104 ceph-mon[97044]: pgmap v5643: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:32 smithi110 ceph-mon[97633]: pgmap v5643: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:32.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:32.441268+0000) 2022-09-10T07:25:33.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:33.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:33.066979+0000) 2022-09-10T07:25:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:33.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:33.441524+0000) 2022-09-10T07:25:34.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:34.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:34.067196+0000) 2022-09-10T07:25:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:34 smithi104 ceph-mon[97044]: pgmap v5644: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:25:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:34 smithi110 ceph-mon[97633]: pgmap v5644: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:25:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:34.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:34.441682+0000) 2022-09-10T07:25:35.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:35.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:35.067448+0000) 2022-09-10T07:25:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:35.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:35.441887+0000) 2022-09-10T07:25:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:35 smithi104 ceph-mon[97044]: pgmap v5645: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:35 smithi110 ceph-mon[97633]: pgmap v5645: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:36.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:36.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:36.067752+0000) 2022-09-10T07:25:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:36.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:36.442156+0000) 2022-09-10T07:25:37.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:37.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:37.067986+0000) 2022-09-10T07:25:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:37.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:37.442383+0000) 2022-09-10T07:25:38.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:38.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:38.068189+0000) 2022-09-10T07:25:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:38 smithi104 ceph-mon[97044]: pgmap v5646: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:25:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:38 smithi110 ceph-mon[97633]: pgmap v5646: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:25:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:38.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:38.442645+0000) 2022-09-10T07:25:39.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:39.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:39.068464+0000) 2022-09-10T07:25:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:39.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:39.442820+0000) 2022-09-10T07:25:40.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:40.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:40.068679+0000) 2022-09-10T07:25:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:40 smithi104 ceph-mon[97044]: pgmap v5647: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:25:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:40 smithi110 ceph-mon[97633]: pgmap v5647: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:25:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:40.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:40.442995+0000) 2022-09-10T07:25:41.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:41.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:41.068930+0000) 2022-09-10T07:25:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:41.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:41.443238+0000) 2022-09-10T07:25:42.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:42.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:42.069198+0000) 2022-09-10T07:25:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:42 smithi104 ceph-mon[97044]: pgmap v5648: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:42 smithi110 ceph-mon[97633]: pgmap v5648: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:42.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:42.443488+0000) 2022-09-10T07:25:43.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:43.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:43.069443+0000) 2022-09-10T07:25:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:43.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:43.443740+0000) 2022-09-10T07:25:44.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:44.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:44.069705+0000) 2022-09-10T07:25:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:44 smithi104 ceph-mon[97044]: pgmap v5649: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:44 smithi110 ceph-mon[97633]: pgmap v5649: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:44.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:44.443921+0000) 2022-09-10T07:25:45.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:45.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:45.069938+0000) 2022-09-10T07:25:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:25:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:25:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:25:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:25:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:25:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:25:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:25:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:25:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:45.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:45.444123+0000) 2022-09-10T07:25:46.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:46.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:46.070190+0000) 2022-09-10T07:25:46.445 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:25:46.213+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:25:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:46 smithi104 ceph-mon[97044]: pgmap v5650: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:46 smithi110 ceph-mon[97633]: pgmap v5650: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:46.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:46.444304+0000) 2022-09-10T07:25:47.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:47.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:47.070323+0000) 2022-09-10T07:25:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:47.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:47.444476+0000) 2022-09-10T07:25:48.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:48.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:48.070548+0000) 2022-09-10T07:25:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:48 smithi104 ceph-mon[97044]: pgmap v5651: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:48 smithi110 ceph-mon[97633]: pgmap v5651: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:48.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:48.444668+0000) 2022-09-10T07:25:49.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:49.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:49.070786+0000) 2022-09-10T07:25:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:49.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:49.444831+0000) 2022-09-10T07:25:50.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:50.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:50.070998+0000) 2022-09-10T07:25:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:50 smithi104 ceph-mon[97044]: pgmap v5652: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:50 smithi110 ceph-mon[97633]: pgmap v5652: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:50.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:50.445006+0000) 2022-09-10T07:25:51.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:51.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:51.071241+0000) 2022-09-10T07:25:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:51.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:51.445209+0000) 2022-09-10T07:25:52.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:52.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:52.071470+0000) 2022-09-10T07:25:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:52 smithi104 ceph-mon[97044]: pgmap v5653: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:52 smithi110 ceph-mon[97633]: pgmap v5653: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:52.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:52.445395+0000) 2022-09-10T07:25:53.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:53.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:53.071648+0000) 2022-09-10T07:25:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:53.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:53.445670+0000) 2022-09-10T07:25:54.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:54.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:54.071839+0000) 2022-09-10T07:25:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:54 smithi104 ceph-mon[97044]: pgmap v5654: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:54 smithi110 ceph-mon[97633]: pgmap v5654: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:54.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:54.445892+0000) 2022-09-10T07:25:55.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:55.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:55.072056+0000) 2022-09-10T07:25:55.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:55.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:55.446082+0000) 2022-09-10T07:25:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:55 smithi104 ceph-mon[97044]: pgmap v5655: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:55 smithi110 ceph-mon[97633]: pgmap v5655: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:25:56.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:56.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:56.072293+0000) 2022-09-10T07:25:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:56.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:56.446332+0000) 2022-09-10T07:25:57.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:57.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:57.072497+0000) 2022-09-10T07:25:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:57.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:57.446488+0000) 2022-09-10T07:25:58.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:58.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:58.072737+0000) 2022-09-10T07:25:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:25:58 smithi104 ceph-mon[97044]: pgmap v5656: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:25:58 smithi110 ceph-mon[97633]: pgmap v5656: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:25:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:58.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:58.446690+0000) 2022-09-10T07:25:59.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:25:59.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:59.072958+0000) 2022-09-10T07:25:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:25:59.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:25:59.446882+0000) 2022-09-10T07:26:00.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:00.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:00.073185+0000) 2022-09-10T07:26:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:00 smithi104 ceph-mon[97044]: pgmap v5657: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:00 smithi110 ceph-mon[97633]: pgmap v5657: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:00.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:00.447051+0000) 2022-09-10T07:26:01.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:01.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:01.073429+0000) 2022-09-10T07:26:01.448 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:26:01.227+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:26:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:01.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:01.447231+0000) 2022-09-10T07:26:02.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:02.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:02.073623+0000) 2022-09-10T07:26:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:02 smithi104 ceph-mon[97044]: pgmap v5658: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:02 smithi110 ceph-mon[97633]: pgmap v5658: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:02.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:02.447435+0000) 2022-09-10T07:26:03.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:03.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:03.073883+0000) 2022-09-10T07:26:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:03.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:03.447654+0000) 2022-09-10T07:26:04.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:04.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:04.074124+0000) 2022-09-10T07:26:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:04 smithi104 ceph-mon[97044]: pgmap v5659: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:04 smithi110 ceph-mon[97633]: pgmap v5659: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:04.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:04.447859+0000) 2022-09-10T07:26:05.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:05.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:05.074361+0000) 2022-09-10T07:26:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:05.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:05.448049+0000) 2022-09-10T07:26:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:05 smithi104 ceph-mon[97044]: pgmap v5660: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:26:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:05 smithi110 ceph-mon[97633]: pgmap v5660: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:26:06.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:06.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:06.074619+0000) 2022-09-10T07:26:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:06.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:06.448213+0000) 2022-09-10T07:26:07.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:07.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:07.074851+0000) 2022-09-10T07:26:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:07.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:07.448461+0000) 2022-09-10T07:26:08.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:08.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:08.075139+0000) 2022-09-10T07:26:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:08 smithi104 ceph-mon[97044]: pgmap v5661: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:26:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:08 smithi110 ceph-mon[97633]: pgmap v5661: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:26:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:08.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:08.448621+0000) 2022-09-10T07:26:09.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:09.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:09.075320+0000) 2022-09-10T07:26:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:09 smithi104 ceph-mon[97044]: osdmap e434: 12 total, 12 up, 12 in 2022-09-10T07:26:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:09 smithi110 ceph-mon[97633]: osdmap e434: 12 total, 12 up, 12 in 2022-09-10T07:26:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:09.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:09.448800+0000) 2022-09-10T07:26:10.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:10.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:10.075552+0000) 2022-09-10T07:26:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:10 smithi104 ceph-mon[97044]: pgmap v5663: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:26:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:10 smithi110 ceph-mon[97633]: pgmap v5663: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:26:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:10.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:10.448976+0000) 2022-09-10T07:26:11.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:11.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:11.075825+0000) 2022-09-10T07:26:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:11.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:11.449178+0000) 2022-09-10T07:26:12.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:12.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:12.076062+0000) 2022-09-10T07:26:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:12 smithi104 ceph-mon[97044]: pgmap v5664: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:26:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:12 smithi110 ceph-mon[97633]: pgmap v5664: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:26:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:12.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:12.449350+0000) 2022-09-10T07:26:13.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:13.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:13.076341+0000) 2022-09-10T07:26:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:13.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:13.449596+0000) 2022-09-10T07:26:14.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:14.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:14.076621+0000) 2022-09-10T07:26:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:14 smithi104 ceph-mon[97044]: pgmap v5665: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:26:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:14 smithi110 ceph-mon[97633]: pgmap v5665: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:26:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:14.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:14.449797+0000) 2022-09-10T07:26:15.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:15.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:15.076892+0000) 2022-09-10T07:26:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:15.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:15.450005+0000) 2022-09-10T07:26:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:15 smithi104 ceph-mon[97044]: osdmap e435: 12 total, 12 up, 12 in 2022-09-10T07:26:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:15 smithi104 ceph-mon[97044]: pgmap v5667: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:15 smithi110 ceph-mon[97633]: osdmap e435: 12 total, 12 up, 12 in 2022-09-10T07:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:15 smithi110 ceph-mon[97633]: pgmap v5667: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:26:16.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:16.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:16.077180+0000) 2022-09-10T07:26:16.451 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:26:16.243+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:26:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:16.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:16.450181+0000) 2022-09-10T07:26:17.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:17.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:17.077371+0000) 2022-09-10T07:26:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:17.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:17.450352+0000) 2022-09-10T07:26:18.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:18.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:18.077592+0000) 2022-09-10T07:26:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:18 smithi104 ceph-mon[97044]: pgmap v5668: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:26:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:18 smithi110 ceph-mon[97633]: pgmap v5668: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:26:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:18.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:18.450564+0000) 2022-09-10T07:26:19.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:19.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:19.077850+0000) 2022-09-10T07:26:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:19.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:19.450813+0000) 2022-09-10T07:26:20.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:20.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:20.078007+0000) 2022-09-10T07:26:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:20 smithi104 ceph-mon[97044]: pgmap v5669: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:26:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:20 smithi110 ceph-mon[97633]: pgmap v5669: 129 pgs: 129 active+clean; 8.3 MiB data, 645 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:26:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:20.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:20.450983+0000) 2022-09-10T07:26:21.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:21.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:21.078211+0000) 2022-09-10T07:26:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:21.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:21.451180+0000) 2022-09-10T07:26:22.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:22.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:22.078388+0000) 2022-09-10T07:26:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:22 smithi104 ceph-mon[97044]: pgmap v5670: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:26:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:22 smithi110 ceph-mon[97633]: pgmap v5670: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:26:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:22.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:22.451306+0000) 2022-09-10T07:26:23.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:23.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:23.078605+0000) 2022-09-10T07:26:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:23.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:23.451566+0000) 2022-09-10T07:26:24.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:24.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:24.078830+0000) 2022-09-10T07:26:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:24 smithi104 ceph-mon[97044]: pgmap v5671: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:26:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:24 smithi110 ceph-mon[97633]: pgmap v5671: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:26:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:24.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:24.451775+0000) 2022-09-10T07:26:25.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:25.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:25.079109+0000) 2022-09-10T07:26:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:25.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:25.451943+0000) 2022-09-10T07:26:26.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:25 smithi104 ceph-mon[97044]: pgmap v5672: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:26:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:25 smithi110 ceph-mon[97633]: pgmap v5672: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:26:26.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:26.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:26.079330+0000) 2022-09-10T07:26:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:26.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:26.452201+0000) 2022-09-10T07:26:27.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:27.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:27.079516+0000) 2022-09-10T07:26:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:27.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:27.452435+0000) 2022-09-10T07:26:28.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:28.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:28.079679+0000) 2022-09-10T07:26:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:28 smithi104 ceph-mon[97044]: pgmap v5673: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:28 smithi110 ceph-mon[97633]: pgmap v5673: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:28.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:28.452618+0000) 2022-09-10T07:26:29.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:29.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:29.079921+0000) 2022-09-10T07:26:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:29.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:29.452862+0000) 2022-09-10T07:26:30.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:30.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:30.080178+0000) 2022-09-10T07:26:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:30 smithi104 ceph-mon[97044]: pgmap v5674: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:30 smithi110 ceph-mon[97633]: pgmap v5674: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:30.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:30.453066+0000) 2022-09-10T07:26:31.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:31.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:31.080370+0000) 2022-09-10T07:26:31.454 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:26:31.257+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:26:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:31.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:31.453279+0000) 2022-09-10T07:26:32.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:32.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:32.080548+0000) 2022-09-10T07:26:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:32 smithi104 ceph-mon[97044]: pgmap v5675: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:32 smithi110 ceph-mon[97633]: pgmap v5675: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:32.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:32.453451+0000) 2022-09-10T07:26:33.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:33.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:33.080774+0000) 2022-09-10T07:26:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:33.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:33.453699+0000) 2022-09-10T07:26:34.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:34.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:34.081010+0000) 2022-09-10T07:26:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:34 smithi104 ceph-mon[97044]: pgmap v5676: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:34 smithi110 ceph-mon[97633]: pgmap v5676: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:34.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:34.453915+0000) 2022-09-10T07:26:35.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:35.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:35.081203+0000) 2022-09-10T07:26:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:35.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:35.454082+0000) 2022-09-10T07:26:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:35 smithi104 ceph-mon[97044]: pgmap v5677: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:35 smithi110 ceph-mon[97633]: pgmap v5677: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:36.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:36.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:36.081472+0000) 2022-09-10T07:26:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:36.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:36.454333+0000) 2022-09-10T07:26:37.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:37.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:37.081660+0000) 2022-09-10T07:26:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:37.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:37.454529+0000) 2022-09-10T07:26:38.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:38.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:38.081918+0000) 2022-09-10T07:26:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:38 smithi104 ceph-mon[97044]: pgmap v5678: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:38 smithi110 ceph-mon[97633]: pgmap v5678: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:38.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:38.454702+0000) 2022-09-10T07:26:39.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:39.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:39.082158+0000) 2022-09-10T07:26:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:39.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:39.454902+0000) 2022-09-10T07:26:40.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:40.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:40.082406+0000) 2022-09-10T07:26:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:40 smithi104 ceph-mon[97044]: pgmap v5679: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:40 smithi110 ceph-mon[97633]: pgmap v5679: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:40.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:40.455179+0000) 2022-09-10T07:26:41.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:41.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:41.082576+0000) 2022-09-10T07:26:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:41.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:41.455380+0000) 2022-09-10T07:26:42.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:42.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:42.082768+0000) 2022-09-10T07:26:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:42 smithi104 ceph-mon[97044]: pgmap v5680: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:42 smithi110 ceph-mon[97633]: pgmap v5680: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:42.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:42.455611+0000) 2022-09-10T07:26:43.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:43.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:43.083014+0000) 2022-09-10T07:26:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:43.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:43.455804+0000) 2022-09-10T07:26:44.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:44.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:44.083289+0000) 2022-09-10T07:26:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:44 smithi104 ceph-mon[97044]: pgmap v5681: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:44 smithi110 ceph-mon[97633]: pgmap v5681: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:44.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:44.456034+0000) 2022-09-10T07:26:45.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:45.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:45.083552+0000) 2022-09-10T07:26:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:26:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:26:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:26:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:26:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:26:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:26:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:26:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:26:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:45.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:45.456277+0000) 2022-09-10T07:26:46.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:46.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:46.083758+0000) 2022-09-10T07:26:46.457 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:26:46.273+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:26:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:46 smithi104 ceph-mon[97044]: pgmap v5682: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:46 smithi110 ceph-mon[97633]: pgmap v5682: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:46.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:46.456491+0000) 2022-09-10T07:26:47.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:47.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:47.084005+0000) 2022-09-10T07:26:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:47.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:47.456679+0000) 2022-09-10T07:26:48.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:48.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:48.084262+0000) 2022-09-10T07:26:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:48 smithi104 ceph-mon[97044]: pgmap v5683: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:26:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:48 smithi110 ceph-mon[97633]: pgmap v5683: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:26:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:48.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:48.456840+0000) 2022-09-10T07:26:49.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:49.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:49.084425+0000) 2022-09-10T07:26:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:49.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:49.457015+0000) 2022-09-10T07:26:50.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:50.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:50.084615+0000) 2022-09-10T07:26:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:50 smithi104 ceph-mon[97044]: pgmap v5684: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:26:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:50 smithi110 ceph-mon[97633]: pgmap v5684: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:26:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:50.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:50.457249+0000) 2022-09-10T07:26:51.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:51.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:51.084883+0000) 2022-09-10T07:26:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:51.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:51.457450+0000) 2022-09-10T07:26:52.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:52.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:52.085169+0000) 2022-09-10T07:26:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:52 smithi104 ceph-mon[97044]: pgmap v5685: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:52 smithi110 ceph-mon[97633]: pgmap v5685: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:52.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:52.457671+0000) 2022-09-10T07:26:53.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:53.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:53.085316+0000) 2022-09-10T07:26:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:53.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:53.457932+0000) 2022-09-10T07:26:54.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:54.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:54.085496+0000) 2022-09-10T07:26:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:54 smithi104 ceph-mon[97044]: pgmap v5686: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:26:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:54 smithi110 ceph-mon[97633]: pgmap v5686: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:26:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:54.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:54.458126+0000) 2022-09-10T07:26:55.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:55.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:55.085744+0000) 2022-09-10T07:26:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:55.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:55.458285+0000) 2022-09-10T07:26:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:55 smithi104 ceph-mon[97044]: pgmap v5687: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:55 smithi110 ceph-mon[97633]: pgmap v5687: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:26:56.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:56.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:56.085923+0000) 2022-09-10T07:26:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:56.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:56.458431+0000) 2022-09-10T07:26:57.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:57.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:57.086200+0000) 2022-09-10T07:26:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:57.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:57.458638+0000) 2022-09-10T07:26:58.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:58.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:58.086453+0000) 2022-09-10T07:26:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:26:58 smithi104 ceph-mon[97044]: pgmap v5688: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:26:58 smithi110 ceph-mon[97633]: pgmap v5688: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:26:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:58.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:58.458842+0000) 2022-09-10T07:26:59.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:26:59.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:59.086594+0000) 2022-09-10T07:26:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:26:59.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:26:59.459065+0000) 2022-09-10T07:27:00.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:00.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:00.086788+0000) 2022-09-10T07:27:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:00 smithi104 ceph-mon[97044]: pgmap v5689: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:00 smithi110 ceph-mon[97633]: pgmap v5689: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:00.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:00.459366+0000) 2022-09-10T07:27:01.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:01.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:01.086959+0000) 2022-09-10T07:27:01.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:27:01.288+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:27:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:01.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:01.459575+0000) 2022-09-10T07:27:02.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:02.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:02.087190+0000) 2022-09-10T07:27:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:02 smithi104 ceph-mon[97044]: pgmap v5690: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:02 smithi110 ceph-mon[97633]: pgmap v5690: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:02.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:02.459800+0000) 2022-09-10T07:27:03.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:03.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:03.087392+0000) 2022-09-10T07:27:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:03.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:03.459990+0000) 2022-09-10T07:27:04.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:04.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:04.087571+0000) 2022-09-10T07:27:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:04 smithi104 ceph-mon[97044]: pgmap v5691: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:04 smithi110 ceph-mon[97633]: pgmap v5691: 129 pgs: 129 active+clean; 8.3 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:04.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:04.460230+0000) 2022-09-10T07:27:05.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:05.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:05.087772+0000) 2022-09-10T07:27:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:05.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:05.460437+0000) 2022-09-10T07:27:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:05 smithi104 ceph-mon[97044]: pgmap v5692: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:27:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:05 smithi110 ceph-mon[97633]: pgmap v5692: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:27:06.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:06.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:06.087990+0000) 2022-09-10T07:27:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:06.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:06.460598+0000) 2022-09-10T07:27:07.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:07.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:07.088196+0000) 2022-09-10T07:27:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:07.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:07.460738+0000) 2022-09-10T07:27:08.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:08.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:08.088425+0000) 2022-09-10T07:27:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:08 smithi104 ceph-mon[97044]: pgmap v5693: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:27:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:08 smithi110 ceph-mon[97633]: pgmap v5693: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:27:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:08.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:08.460916+0000) 2022-09-10T07:27:09.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:09.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:09.088681+0000) 2022-09-10T07:27:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:09 smithi104 ceph-mon[97044]: osdmap e436: 12 total, 12 up, 12 in 2022-09-10T07:27:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:09 smithi110 ceph-mon[97633]: osdmap e436: 12 total, 12 up, 12 in 2022-09-10T07:27:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:09.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:09.461111+0000) 2022-09-10T07:27:10.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:10.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:10.088918+0000) 2022-09-10T07:27:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:10 smithi104 ceph-mon[97044]: pgmap v5695: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:27:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:10 smithi110 ceph-mon[97633]: pgmap v5695: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:27:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:10.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:10.461290+0000) 2022-09-10T07:27:11.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:11.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:11.089204+0000) 2022-09-10T07:27:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:11.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:11.461464+0000) 2022-09-10T07:27:12.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:12.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:12.089445+0000) 2022-09-10T07:27:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:12 smithi104 ceph-mon[97044]: pgmap v5696: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:27:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:12 smithi110 ceph-mon[97633]: pgmap v5696: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:27:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:12.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:12.461669+0000) 2022-09-10T07:27:13.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:13.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:13.089709+0000) 2022-09-10T07:27:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:13.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:13.461874+0000) 2022-09-10T07:27:14.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:14.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:14.089953+0000) 2022-09-10T07:27:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:14 smithi104 ceph-mon[97044]: pgmap v5697: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:27:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:14 smithi110 ceph-mon[97633]: pgmap v5697: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:27:14.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:14.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:14.462053+0000) 2022-09-10T07:27:15.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:15.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:15.090168+0000) 2022-09-10T07:27:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:15.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:15.462271+0000) 2022-09-10T07:27:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:15 smithi104 ceph-mon[97044]: osdmap e437: 12 total, 12 up, 12 in 2022-09-10T07:27:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:15 smithi104 ceph-mon[97044]: pgmap v5699: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:27:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:15 smithi110 ceph-mon[97633]: osdmap e437: 12 total, 12 up, 12 in 2022-09-10T07:27:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:15 smithi110 ceph-mon[97633]: pgmap v5699: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:27:16.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:16.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:16.090357+0000) 2022-09-10T07:27:16.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:27:16.304+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:27:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:16.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:16.462430+0000) 2022-09-10T07:27:17.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:17.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:17.090558+0000) 2022-09-10T07:27:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:17.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:17.462635+0000) 2022-09-10T07:27:18.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:18.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:18.090811+0000) 2022-09-10T07:27:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:18 smithi104 ceph-mon[97044]: pgmap v5700: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:27:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:18 smithi110 ceph-mon[97633]: pgmap v5700: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:27:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:18.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:18.462817+0000) 2022-09-10T07:27:19.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:19.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:19.091021+0000) 2022-09-10T07:27:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:19.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:19.462963+0000) 2022-09-10T07:27:20.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:20.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:20.091254+0000) 2022-09-10T07:27:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:20 smithi104 ceph-mon[97044]: pgmap v5701: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:27:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:20 smithi110 ceph-mon[97633]: pgmap v5701: 129 pgs: 129 active+clean; 8.3 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:27:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:20.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:20.463172+0000) 2022-09-10T07:27:21.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:21.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:21.091471+0000) 2022-09-10T07:27:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:21.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:21.463384+0000) 2022-09-10T07:27:22.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:22.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:22.091608+0000) 2022-09-10T07:27:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:22 smithi104 ceph-mon[97044]: pgmap v5702: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:27:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:22 smithi110 ceph-mon[97633]: pgmap v5702: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:27:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:22.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:22.463583+0000) 2022-09-10T07:27:23.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:23.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:23.091808+0000) 2022-09-10T07:27:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:23.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:23.463774+0000) 2022-09-10T07:27:24.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:24.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:24.091991+0000) 2022-09-10T07:27:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:24 smithi104 ceph-mon[97044]: pgmap v5703: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:27:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:24 smithi110 ceph-mon[97633]: pgmap v5703: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:27:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:24.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:24.463930+0000) 2022-09-10T07:27:25.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:25.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:25.092252+0000) 2022-09-10T07:27:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:25.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:25.464191+0000) 2022-09-10T07:27:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:25 smithi104 ceph-mon[97044]: pgmap v5704: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:27:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:25 smithi110 ceph-mon[97633]: pgmap v5704: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:27:26.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:26.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:26.092434+0000) 2022-09-10T07:27:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:26.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:26.464374+0000) 2022-09-10T07:27:27.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:27.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:27.092697+0000) 2022-09-10T07:27:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:27.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:27.464626+0000) 2022-09-10T07:27:28.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:28.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:28.092965+0000) 2022-09-10T07:27:28.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:28 smithi104 ceph-mon[97044]: pgmap v5705: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:28 smithi110 ceph-mon[97633]: pgmap v5705: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:28.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:28.464838+0000) 2022-09-10T07:27:29.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:29.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:29.093228+0000) 2022-09-10T07:27:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:29.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:29.465020+0000) 2022-09-10T07:27:30.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:30.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:30.093458+0000) 2022-09-10T07:27:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:30 smithi104 ceph-mon[97044]: pgmap v5706: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:30 smithi110 ceph-mon[97633]: pgmap v5706: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:30.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:30.465229+0000) 2022-09-10T07:27:31.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:31.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:31.093640+0000) 2022-09-10T07:27:31.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:27:31.319+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:27:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:31.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:31.465478+0000) 2022-09-10T07:27:32.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:32.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:32.093887+0000) 2022-09-10T07:27:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:32 smithi104 ceph-mon[97044]: pgmap v5707: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:32 smithi110 ceph-mon[97633]: pgmap v5707: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:32.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:32.465727+0000) 2022-09-10T07:27:33.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:33.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:33.094043+0000) 2022-09-10T07:27:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:33.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:33.465927+0000) 2022-09-10T07:27:34.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:34.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:34.094261+0000) 2022-09-10T07:27:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:34 smithi104 ceph-mon[97044]: pgmap v5708: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:34 smithi110 ceph-mon[97633]: pgmap v5708: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:34.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:34.466091+0000) 2022-09-10T07:27:35.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:35.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:35.094444+0000) 2022-09-10T07:27:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:35.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:35.466303+0000) 2022-09-10T07:27:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:35 smithi104 ceph-mon[97044]: pgmap v5709: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:35 smithi110 ceph-mon[97633]: pgmap v5709: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:36.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:36.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:36.094634+0000) 2022-09-10T07:27:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:36.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:36.466502+0000) 2022-09-10T07:27:37.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:37.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:37.094877+0000) 2022-09-10T07:27:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:37.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:37.466742+0000) 2022-09-10T07:27:38.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:38.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:38.095165+0000) 2022-09-10T07:27:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:38 smithi104 ceph-mon[97044]: pgmap v5710: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:38 smithi110 ceph-mon[97633]: pgmap v5710: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:38.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:38.466964+0000) 2022-09-10T07:27:39.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:39.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:39.095324+0000) 2022-09-10T07:27:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:39.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:39.467133+0000) 2022-09-10T07:27:40.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:40.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:40.095574+0000) 2022-09-10T07:27:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:40 smithi104 ceph-mon[97044]: pgmap v5711: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:40 smithi110 ceph-mon[97633]: pgmap v5711: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:40.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:40.467299+0000) 2022-09-10T07:27:41.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:41.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:41.095743+0000) 2022-09-10T07:27:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:41.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:41.467501+0000) 2022-09-10T07:27:42.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:42.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:42.095993+0000) 2022-09-10T07:27:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:42 smithi104 ceph-mon[97044]: pgmap v5712: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:42.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:42 smithi110 ceph-mon[97633]: pgmap v5712: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:42.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:42.467718+0000) 2022-09-10T07:27:43.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:43.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:43.096189+0000) 2022-09-10T07:27:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:43.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:43.467964+0000) 2022-09-10T07:27:44.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:44.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:44.096394+0000) 2022-09-10T07:27:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:44 smithi104 ceph-mon[97044]: pgmap v5713: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:44 smithi110 ceph-mon[97633]: pgmap v5713: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:44.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:44.468163+0000) 2022-09-10T07:27:45.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:45.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:45.096558+0000) 2022-09-10T07:27:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:27:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:27:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:27:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:27:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:27:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:27:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:27:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:27:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:45.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:45.468328+0000) 2022-09-10T07:27:46.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:46.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:46.096779+0000) 2022-09-10T07:27:46.469 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:27:46.335+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:27:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:46 smithi104 ceph-mon[97044]: pgmap v5714: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:46 smithi110 ceph-mon[97633]: pgmap v5714: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:46.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:46.468537+0000) 2022-09-10T07:27:47.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:47.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:47.097034+0000) 2022-09-10T07:27:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:47.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:47.468760+0000) 2022-09-10T07:27:48.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:48.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:48.097282+0000) 2022-09-10T07:27:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:48 smithi104 ceph-mon[97044]: pgmap v5715: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:48 smithi110 ceph-mon[97633]: pgmap v5715: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:48.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:48.468970+0000) 2022-09-10T07:27:49.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:49.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:49.097515+0000) 2022-09-10T07:27:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:49.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:49.469186+0000) 2022-09-10T07:27:50.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:50.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:50.097711+0000) 2022-09-10T07:27:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:50 smithi104 ceph-mon[97044]: pgmap v5716: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:50 smithi110 ceph-mon[97633]: pgmap v5716: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:50.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:50.469416+0000) 2022-09-10T07:27:51.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:51.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:51.097957+0000) 2022-09-10T07:27:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:51.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:51.469653+0000) 2022-09-10T07:27:52.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:52.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:52.098215+0000) 2022-09-10T07:27:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:52 smithi104 ceph-mon[97044]: pgmap v5717: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:52 smithi110 ceph-mon[97633]: pgmap v5717: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:52.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:52.469913+0000) 2022-09-10T07:27:53.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:53.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:53.098501+0000) 2022-09-10T07:27:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:53.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:53.470191+0000) 2022-09-10T07:27:54.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:54.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:54.098676+0000) 2022-09-10T07:27:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:54 smithi104 ceph-mon[97044]: pgmap v5718: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:54 smithi110 ceph-mon[97633]: pgmap v5718: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:54.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:54.470409+0000) 2022-09-10T07:27:55.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:55.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:55.098861+0000) 2022-09-10T07:27:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:55.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:55.470668+0000) 2022-09-10T07:27:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:55 smithi110 ceph-mon[97633]: pgmap v5719: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:55 smithi104 ceph-mon[97044]: pgmap v5719: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:27:56.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:56.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:56.099084+0000) 2022-09-10T07:27:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:56.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:56.470957+0000) 2022-09-10T07:27:57.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:57.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:57.099383+0000) 2022-09-10T07:27:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:57.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:57.471218+0000) 2022-09-10T07:27:58.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:58.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:58.099643+0000) 2022-09-10T07:27:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:27:58 smithi104 ceph-mon[97044]: pgmap v5720: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:27:58 smithi110 ceph-mon[97633]: pgmap v5720: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:27:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:58.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:58.471470+0000) 2022-09-10T07:27:59.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:27:59.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:59.099841+0000) 2022-09-10T07:27:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:27:59.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:27:59.471651+0000) 2022-09-10T07:28:00.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:00.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:00.100049+0000) 2022-09-10T07:28:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:00 smithi104 ceph-mon[97044]: pgmap v5721: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:00 smithi110 ceph-mon[97633]: pgmap v5721: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:00.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:00.471782+0000) 2022-09-10T07:28:01.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:01.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:01.100272+0000) 2022-09-10T07:28:01.473 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:28:01.347+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:28:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:01.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:01.472035+0000) 2022-09-10T07:28:02.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:02.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:02.100409+0000) 2022-09-10T07:28:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:02 smithi104 ceph-mon[97044]: pgmap v5722: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:02 smithi110 ceph-mon[97633]: pgmap v5722: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:02.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:02.472266+0000) 2022-09-10T07:28:03.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:03.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:03.100655+0000) 2022-09-10T07:28:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:03.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:03.472476+0000) 2022-09-10T07:28:04.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:04.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:04.100878+0000) 2022-09-10T07:28:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:04 smithi104 ceph-mon[97044]: pgmap v5723: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:04 smithi110 ceph-mon[97633]: pgmap v5723: 129 pgs: 129 active+clean; 8.3 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:04.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:04.472718+0000) 2022-09-10T07:28:05.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:05.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:05.101188+0000) 2022-09-10T07:28:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:05.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:05.472995+0000) 2022-09-10T07:28:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:05 smithi110 ceph-mon[97633]: pgmap v5724: 129 pgs: 129 active+clean; 8.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:28:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:05 smithi104 ceph-mon[97044]: pgmap v5724: 129 pgs: 129 active+clean; 8.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:28:06.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:06.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:06.101324+0000) 2022-09-10T07:28:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:06.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:06.473238+0000) 2022-09-10T07:28:07.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:07.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:07.101575+0000) 2022-09-10T07:28:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:07.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:07.473497+0000) 2022-09-10T07:28:08.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:08.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:08.101800+0000) 2022-09-10T07:28:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:08 smithi104 ceph-mon[97044]: pgmap v5725: 129 pgs: 129 active+clean; 8.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:28:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:08 smithi110 ceph-mon[97633]: pgmap v5725: 129 pgs: 129 active+clean; 8.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:28:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:08.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:08.473788+0000) 2022-09-10T07:28:09.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:09.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:09.102002+0000) 2022-09-10T07:28:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:09 smithi104 ceph-mon[97044]: osdmap e438: 12 total, 12 up, 12 in 2022-09-10T07:28:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:09 smithi110 ceph-mon[97633]: osdmap e438: 12 total, 12 up, 12 in 2022-09-10T07:28:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:09.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:09.474048+0000) 2022-09-10T07:28:10.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:10.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:10.102202+0000) 2022-09-10T07:28:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:10 smithi104 ceph-mon[97044]: pgmap v5727: 129 pgs: 129 active+clean; 8.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:28:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:10 smithi110 ceph-mon[97633]: pgmap v5727: 129 pgs: 129 active+clean; 8.3 MiB data, 653 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:28:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:10.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:10.474283+0000) 2022-09-10T07:28:11.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:11.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:11.102432+0000) 2022-09-10T07:28:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:11.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:11.474417+0000) 2022-09-10T07:28:12.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:12.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:12.102593+0000) 2022-09-10T07:28:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:12 smithi104 ceph-mon[97044]: pgmap v5728: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:28:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:12 smithi110 ceph-mon[97633]: pgmap v5728: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:28:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:12.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:12.474620+0000) 2022-09-10T07:28:13.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:13.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:13.102839+0000) 2022-09-10T07:28:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:13.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:13.474883+0000) 2022-09-10T07:28:14.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:14.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:14.103016+0000) 2022-09-10T07:28:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:14 smithi104 ceph-mon[97044]: pgmap v5729: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:28:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:14 smithi110 ceph-mon[97633]: pgmap v5729: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:28:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:14.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:14.475192+0000) 2022-09-10T07:28:15.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:15.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:15.103263+0000) 2022-09-10T07:28:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:15.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:15.475426+0000) 2022-09-10T07:28:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:15 smithi110 ceph-mon[97633]: osdmap e439: 12 total, 12 up, 12 in 2022-09-10T07:28:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:15 smithi110 ceph-mon[97633]: pgmap v5731: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:28:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:15 smithi104 ceph-mon[97044]: osdmap e439: 12 total, 12 up, 12 in 2022-09-10T07:28:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:15 smithi104 ceph-mon[97044]: pgmap v5731: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:28:16.360 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:16.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:16.103517+0000) 2022-09-10T07:28:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:16.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:16.475688+0000) 2022-09-10T07:28:16.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:28:16.359+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:28:17.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:17.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:17.103713+0000) 2022-09-10T07:28:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:17.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:17.475845+0000) 2022-09-10T07:28:18.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:18.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:18.103990+0000) 2022-09-10T07:28:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:18 smithi104 ceph-mon[97044]: pgmap v5732: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:28:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:18 smithi110 ceph-mon[97633]: pgmap v5732: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:28:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:18.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:18.476060+0000) 2022-09-10T07:28:19.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:19.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:19.104251+0000) 2022-09-10T07:28:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:19.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:19.476349+0000) 2022-09-10T07:28:20.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:20.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:20.104505+0000) 2022-09-10T07:28:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:20 smithi104 ceph-mon[97044]: pgmap v5733: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:28:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:20 smithi110 ceph-mon[97633]: pgmap v5733: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:28:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:20.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:20.476635+0000) 2022-09-10T07:28:21.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:21.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:21.104703+0000) 2022-09-10T07:28:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:21.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:21.476815+0000) 2022-09-10T07:28:22.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:22.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:22.104939+0000) 2022-09-10T07:28:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:22 smithi104 ceph-mon[97044]: pgmap v5734: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:28:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:22 smithi110 ceph-mon[97633]: pgmap v5734: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:28:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:22.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:22.477051+0000) 2022-09-10T07:28:23.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:23.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:23.105197+0000) 2022-09-10T07:28:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:23.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:23.477264+0000) 2022-09-10T07:28:24.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:24.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:24.105446+0000) 2022-09-10T07:28:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:24 smithi104 ceph-mon[97044]: pgmap v5735: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:28:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:24 smithi110 ceph-mon[97633]: pgmap v5735: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:28:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:24.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:24.477511+0000) 2022-09-10T07:28:25.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:25.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:25.105701+0000) 2022-09-10T07:28:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:25.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:25.477722+0000) 2022-09-10T07:28:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:25 smithi110 ceph-mon[97633]: pgmap v5736: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T07:28:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:25 smithi104 ceph-mon[97044]: pgmap v5736: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T07:28:26.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:26.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:26.105921+0000) 2022-09-10T07:28:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:26.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:26.477950+0000) 2022-09-10T07:28:27.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:27.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:27.106183+0000) 2022-09-10T07:28:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:27.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:27.478197+0000) 2022-09-10T07:28:28.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:28.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:28.106385+0000) 2022-09-10T07:28:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:28 smithi104 ceph-mon[97044]: pgmap v5737: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:28:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:28 smithi110 ceph-mon[97633]: pgmap v5737: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:28:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:28.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:28.478489+0000) 2022-09-10T07:28:29.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:29.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:29.106504+0000) 2022-09-10T07:28:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:29.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:29.478702+0000) 2022-09-10T07:28:30.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:30.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:30.106757+0000) 2022-09-10T07:28:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:30 smithi104 ceph-mon[97044]: pgmap v5738: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:28:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:30 smithi110 ceph-mon[97633]: pgmap v5738: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:28:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:30.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:30.478876+0000) 2022-09-10T07:28:31.371 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:31.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:31.107122+0000) 2022-09-10T07:28:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:31.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:31.479170+0000) 2022-09-10T07:28:31.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:28:31.369+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:28:32.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:32.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:32.107378+0000) 2022-09-10T07:28:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:32 smithi104 ceph-mon[97044]: pgmap v5739: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:32 smithi110 ceph-mon[97633]: pgmap v5739: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:32.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:32.479441+0000) 2022-09-10T07:28:33.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:33.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:33.107597+0000) 2022-09-10T07:28:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:33.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:33.479636+0000) 2022-09-10T07:28:34.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:34.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:34.107842+0000) 2022-09-10T07:28:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:34 smithi104 ceph-mon[97044]: pgmap v5740: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:34 smithi110 ceph-mon[97633]: pgmap v5740: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:34.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:34.479872+0000) 2022-09-10T07:28:35.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:35.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:35.108120+0000) 2022-09-10T07:28:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:35.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:35.480178+0000) 2022-09-10T07:28:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:35 smithi110 ceph-mon[97633]: pgmap v5741: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:35 smithi104 ceph-mon[97044]: pgmap v5741: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:36.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:36.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:36.108311+0000) 2022-09-10T07:28:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:36.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:36.480392+0000) 2022-09-10T07:28:37.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:37.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:37.108481+0000) 2022-09-10T07:28:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:37.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:37.480527+0000) 2022-09-10T07:28:38.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:38.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:38.108705+0000) 2022-09-10T07:28:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:38 smithi104 ceph-mon[97044]: pgmap v5742: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:38 smithi110 ceph-mon[97633]: pgmap v5742: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:38.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:38.480734+0000) 2022-09-10T07:28:39.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:39.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:39.108927+0000) 2022-09-10T07:28:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:39.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:39.480930+0000) 2022-09-10T07:28:40.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:40.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:40.109222+0000) 2022-09-10T07:28:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:40 smithi104 ceph-mon[97044]: pgmap v5743: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:40 smithi110 ceph-mon[97633]: pgmap v5743: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:40.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:40.481164+0000) 2022-09-10T07:28:41.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:41.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:41.109441+0000) 2022-09-10T07:28:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:41.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:41.481352+0000) 2022-09-10T07:28:42.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:42.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:42.109683+0000) 2022-09-10T07:28:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:42 smithi104 ceph-mon[97044]: pgmap v5744: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:42 smithi110 ceph-mon[97633]: pgmap v5744: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:42.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:42.481585+0000) 2022-09-10T07:28:43.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:43.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:43.109946+0000) 2022-09-10T07:28:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:43.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:43.481874+0000) 2022-09-10T07:28:44.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:44.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:44.110221+0000) 2022-09-10T07:28:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:44 smithi104 ceph-mon[97044]: pgmap v5745: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:28:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:28:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:44 smithi110 ceph-mon[97633]: pgmap v5745: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:28:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:28:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:28:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:28:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:44.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:44.482173+0000) 2022-09-10T07:28:45.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:45.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:45.110492+0000) 2022-09-10T07:28:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:45.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:45.482394+0000) 2022-09-10T07:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:45 smithi110 ceph-mon[97633]: pgmap v5746: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:45 smithi104 ceph-mon[97044]: pgmap v5746: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:46.373 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:46.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:46.110677+0000) 2022-09-10T07:28:46.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:28:46.371+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:28:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:46.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:46.482670+0000) 2022-09-10T07:28:47.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:47.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:47.110932+0000) 2022-09-10T07:28:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:47.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:47.482895+0000) 2022-09-10T07:28:48.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:48.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:48.111206+0000) 2022-09-10T07:28:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:48 smithi104 ceph-mon[97044]: pgmap v5747: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:48 smithi110 ceph-mon[97633]: pgmap v5747: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:48.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:48.483204+0000) 2022-09-10T07:28:49.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:49.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:49.111502+0000) 2022-09-10T07:28:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:49.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:49.483385+0000) 2022-09-10T07:28:50.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:50.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:50.111796+0000) 2022-09-10T07:28:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:50 smithi104 ceph-mon[97044]: pgmap v5748: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:50 smithi110 ceph-mon[97633]: pgmap v5748: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:50.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:50.483547+0000) 2022-09-10T07:28:51.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:51.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:51.112029+0000) 2022-09-10T07:28:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:51.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:51.483785+0000) 2022-09-10T07:28:52.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:52.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:52.112275+0000) 2022-09-10T07:28:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:52 smithi104 ceph-mon[97044]: pgmap v5749: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:52 smithi110 ceph-mon[97633]: pgmap v5749: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:52.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:52.483979+0000) 2022-09-10T07:28:53.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:53.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:53.112537+0000) 2022-09-10T07:28:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:53.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:53.484192+0000) 2022-09-10T07:28:54.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:54.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:54.112759+0000) 2022-09-10T07:28:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:54 smithi104 ceph-mon[97044]: pgmap v5750: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:54 smithi110 ceph-mon[97633]: pgmap v5750: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:54.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:54.484339+0000) 2022-09-10T07:28:55.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:55.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:55.112999+0000) 2022-09-10T07:28:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:55.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:55.484502+0000) 2022-09-10T07:28:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:55 smithi110 ceph-mon[97633]: pgmap v5751: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:55 smithi104 ceph-mon[97044]: pgmap v5751: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:28:56.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:56.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:56.113266+0000) 2022-09-10T07:28:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:56.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:56.484664+0000) 2022-09-10T07:28:57.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:57.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:57.113549+0000) 2022-09-10T07:28:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:57.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:57.484827+0000) 2022-09-10T07:28:58.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:58.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:58.113745+0000) 2022-09-10T07:28:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:28:58 smithi104 ceph-mon[97044]: pgmap v5752: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:28:58 smithi110 ceph-mon[97633]: pgmap v5752: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:28:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:58.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:58.485068+0000) 2022-09-10T07:28:59.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:28:59.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:59.113971+0000) 2022-09-10T07:28:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:28:59.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:28:59.485246+0000) 2022-09-10T07:29:00.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:00.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:00.114236+0000) 2022-09-10T07:29:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:00 smithi104 ceph-mon[97044]: pgmap v5753: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:00 smithi110 ceph-mon[97633]: pgmap v5753: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:00.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:00.485417+0000) 2022-09-10T07:29:01.388 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:01.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:01.114445+0000) 2022-09-10T07:29:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:01.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:01.485592+0000) 2022-09-10T07:29:01.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:29:01.386+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:29:02.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:02.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:02.114656+0000) 2022-09-10T07:29:02.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:02 smithi104 ceph-mon[97044]: pgmap v5754: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:02 smithi110 ceph-mon[97633]: pgmap v5754: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:02.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:02.485838+0000) 2022-09-10T07:29:03.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:03.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:03.114930+0000) 2022-09-10T07:29:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:03.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:03.486012+0000) 2022-09-10T07:29:04.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:04.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:04.115137+0000) 2022-09-10T07:29:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:04 smithi104 ceph-mon[97044]: pgmap v5755: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:04 smithi110 ceph-mon[97633]: pgmap v5755: 129 pgs: 129 active+clean; 8.3 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:04.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:04.486267+0000) 2022-09-10T07:29:05.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:05.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:05.115362+0000) 2022-09-10T07:29:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:05.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:05.486444+0000) 2022-09-10T07:29:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:05 smithi110 ceph-mon[97633]: pgmap v5756: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:29:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:05 smithi104 ceph-mon[97044]: pgmap v5756: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:29:06.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:06.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:06.115586+0000) 2022-09-10T07:29:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:06.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:06.486660+0000) 2022-09-10T07:29:07.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:07.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:07.115842+0000) 2022-09-10T07:29:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:07.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:07.486789+0000) 2022-09-10T07:29:08.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:08.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:08.116082+0000) 2022-09-10T07:29:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:08 smithi104 ceph-mon[97044]: pgmap v5757: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:29:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:08 smithi110 ceph-mon[97633]: pgmap v5757: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:29:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:08.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:08.486935+0000) 2022-09-10T07:29:09.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:09.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:09.116306+0000) 2022-09-10T07:29:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:09 smithi104 ceph-mon[97044]: osdmap e440: 12 total, 12 up, 12 in 2022-09-10T07:29:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:09 smithi110 ceph-mon[97633]: osdmap e440: 12 total, 12 up, 12 in 2022-09-10T07:29:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:09.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:09.487126+0000) 2022-09-10T07:29:10.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:10.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:10.116468+0000) 2022-09-10T07:29:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:10 smithi104 ceph-mon[97044]: pgmap v5759: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:29:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:10 smithi110 ceph-mon[97633]: pgmap v5759: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:29:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:10.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:10.487386+0000) 2022-09-10T07:29:11.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:11.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:11.116723+0000) 2022-09-10T07:29:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:11.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:11.487633+0000) 2022-09-10T07:29:12.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:12.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:12.116938+0000) 2022-09-10T07:29:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:12 smithi104 ceph-mon[97044]: pgmap v5760: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:29:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:12 smithi110 ceph-mon[97633]: pgmap v5760: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:29:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:12.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:12.487823+0000) 2022-09-10T07:29:13.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:13.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:13.117182+0000) 2022-09-10T07:29:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:13.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:13.488050+0000) 2022-09-10T07:29:14.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:14.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:14.117428+0000) 2022-09-10T07:29:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:14 smithi104 ceph-mon[97044]: pgmap v5761: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:29:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:14 smithi110 ceph-mon[97633]: pgmap v5761: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:29:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:14.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:14.488323+0000) 2022-09-10T07:29:15.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:15.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:15.117705+0000) 2022-09-10T07:29:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:15.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:15.488542+0000) 2022-09-10T07:29:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:15 smithi110 ceph-mon[97633]: osdmap e441: 12 total, 12 up, 12 in 2022-09-10T07:29:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:15 smithi110 ceph-mon[97633]: pgmap v5763: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:29:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:15 smithi104 ceph-mon[97044]: osdmap e441: 12 total, 12 up, 12 in 2022-09-10T07:29:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:15 smithi104 ceph-mon[97044]: pgmap v5763: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:29:16.403 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:16.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:16.117949+0000) 2022-09-10T07:29:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:16.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:16.488741+0000) 2022-09-10T07:29:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:29:16.403+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:29:17.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:17.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:17.118194+0000) 2022-09-10T07:29:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:17.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:17.488942+0000) 2022-09-10T07:29:18.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:18.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:18.118394+0000) 2022-09-10T07:29:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:18 smithi104 ceph-mon[97044]: pgmap v5764: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:29:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:18 smithi110 ceph-mon[97633]: pgmap v5764: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T07:29:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:18.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:18.489147+0000) 2022-09-10T07:29:19.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:19.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:19.118629+0000) 2022-09-10T07:29:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:19.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:19.489391+0000) 2022-09-10T07:29:20.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:20.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:20.118872+0000) 2022-09-10T07:29:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:20 smithi104 ceph-mon[97044]: pgmap v5765: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:29:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:20 smithi110 ceph-mon[97633]: pgmap v5765: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:29:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:20.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:20.489563+0000) 2022-09-10T07:29:21.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:21.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:21.119204+0000) 2022-09-10T07:29:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:21.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:21.489787+0000) 2022-09-10T07:29:22.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:22.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:22.119459+0000) 2022-09-10T07:29:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:22 smithi104 ceph-mon[97044]: pgmap v5766: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:29:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:22 smithi110 ceph-mon[97633]: pgmap v5766: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:29:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:22.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:22.489991+0000) 2022-09-10T07:29:23.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:23.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:23.119744+0000) 2022-09-10T07:29:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:23.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:23.490236+0000) 2022-09-10T07:29:24.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:24.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:24.120000+0000) 2022-09-10T07:29:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:24 smithi104 ceph-mon[97044]: pgmap v5767: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:29:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:24 smithi110 ceph-mon[97633]: pgmap v5767: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:29:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:24.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:24.490457+0000) 2022-09-10T07:29:25.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:25.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:25.120211+0000) 2022-09-10T07:29:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:25.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:25.490647+0000) 2022-09-10T07:29:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:25 smithi110 ceph-mon[97633]: pgmap v5768: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:29:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:25 smithi104 ceph-mon[97044]: pgmap v5768: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:29:26.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:26.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:26.120431+0000) 2022-09-10T07:29:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:26.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:26.490766+0000) 2022-09-10T07:29:27.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:27.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:27.120660+0000) 2022-09-10T07:29:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:27.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:27.490972+0000) 2022-09-10T07:29:28.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:28.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:28.120884+0000) 2022-09-10T07:29:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:28 smithi104 ceph-mon[97044]: pgmap v5769: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:28 smithi110 ceph-mon[97633]: pgmap v5769: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:28.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:28.491191+0000) 2022-09-10T07:29:29.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:29.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:29.121135+0000) 2022-09-10T07:29:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:29.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:29.491431+0000) 2022-09-10T07:29:30.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:30.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:30.121355+0000) 2022-09-10T07:29:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:30 smithi104 ceph-mon[97044]: pgmap v5770: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:30 smithi110 ceph-mon[97633]: pgmap v5770: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:30.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:30.491655+0000) 2022-09-10T07:29:31.405 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:31.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:31.121581+0000) 2022-09-10T07:29:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:31.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:31.491904+0000) 2022-09-10T07:29:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:29:31.403+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:29:32.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:32.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:32.121826+0000) 2022-09-10T07:29:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:32 smithi104 ceph-mon[97044]: pgmap v5771: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:32 smithi110 ceph-mon[97633]: pgmap v5771: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:32.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:32.492140+0000) 2022-09-10T07:29:33.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:33.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:33.121990+0000) 2022-09-10T07:29:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:33.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:33.492326+0000) 2022-09-10T07:29:34.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:34.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:34.122245+0000) 2022-09-10T07:29:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:34 smithi104 ceph-mon[97044]: pgmap v5772: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:34 smithi110 ceph-mon[97633]: pgmap v5772: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:34.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:34.492583+0000) 2022-09-10T07:29:35.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:35.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:35.122490+0000) 2022-09-10T07:29:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:35.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:35.492819+0000) 2022-09-10T07:29:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:35 smithi110 ceph-mon[97633]: pgmap v5773: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:35 smithi104 ceph-mon[97044]: pgmap v5773: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:36.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:36.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:36.122717+0000) 2022-09-10T07:29:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:36.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:36.493153+0000) 2022-09-10T07:29:37.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:37.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:37.122956+0000) 2022-09-10T07:29:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:37.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:37.493348+0000) 2022-09-10T07:29:38.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:38.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:38.123192+0000) 2022-09-10T07:29:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:38 smithi104 ceph-mon[97044]: pgmap v5774: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:38 smithi110 ceph-mon[97633]: pgmap v5774: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:38.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:38.493555+0000) 2022-09-10T07:29:39.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:39.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:39.123412+0000) 2022-09-10T07:29:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:39.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:39.493790+0000) 2022-09-10T07:29:40.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:40.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:40.123570+0000) 2022-09-10T07:29:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:40 smithi104 ceph-mon[97044]: pgmap v5775: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:40 smithi110 ceph-mon[97633]: pgmap v5775: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:40.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:40.494042+0000) 2022-09-10T07:29:41.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:41.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:41.123819+0000) 2022-09-10T07:29:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:41.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:41.494292+0000) 2022-09-10T07:29:42.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:42.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:42.124062+0000) 2022-09-10T07:29:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:42 smithi104 ceph-mon[97044]: pgmap v5776: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:42 smithi110 ceph-mon[97633]: pgmap v5776: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:42.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:42.494438+0000) 2022-09-10T07:29:43.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:43.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:43.124309+0000) 2022-09-10T07:29:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:43.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:43.494668+0000) 2022-09-10T07:29:44.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:44.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:44.124447+0000) 2022-09-10T07:29:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:44 smithi104 ceph-mon[97044]: pgmap v5777: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:29:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:29:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:44 smithi110 ceph-mon[97633]: pgmap v5777: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:29:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:29:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:44.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:44.494928+0000) 2022-09-10T07:29:45.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:45.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:45.124634+0000) 2022-09-10T07:29:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:45.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:45.495208+0000) 2022-09-10T07:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:45 smithi110 ceph-mon[97633]: pgmap v5778: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:45 smithi104 ceph-mon[97044]: pgmap v5778: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:46.421 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:46.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:46.124814+0000) 2022-09-10T07:29:46.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:29:46.419+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:29:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:46.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:46.495380+0000) 2022-09-10T07:29:47.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:47.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:47.124962+0000) 2022-09-10T07:29:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:47.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:47.495630+0000) 2022-09-10T07:29:48.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:48.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:48.125156+0000) 2022-09-10T07:29:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:48 smithi104 ceph-mon[97044]: pgmap v5779: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:48 smithi110 ceph-mon[97633]: pgmap v5779: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:48.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:48.495836+0000) 2022-09-10T07:29:49.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:49.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:49.125282+0000) 2022-09-10T07:29:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:49.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:49.496076+0000) 2022-09-10T07:29:50.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:50.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:50.125456+0000) 2022-09-10T07:29:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:50 smithi104 ceph-mon[97044]: pgmap v5780: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:50 smithi110 ceph-mon[97633]: pgmap v5780: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:50.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:50.496245+0000) 2022-09-10T07:29:51.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:51.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:51.125668+0000) 2022-09-10T07:29:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:51.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:51.496493+0000) 2022-09-10T07:29:52.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:52.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:52.125861+0000) 2022-09-10T07:29:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:52 smithi104 ceph-mon[97044]: pgmap v5781: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:52 smithi110 ceph-mon[97633]: pgmap v5781: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:52.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:52.496695+0000) 2022-09-10T07:29:53.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:53.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:53.126125+0000) 2022-09-10T07:29:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:53.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:53.496934+0000) 2022-09-10T07:29:54.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:54.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:54.126389+0000) 2022-09-10T07:29:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:54 smithi104 ceph-mon[97044]: pgmap v5782: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:54 smithi110 ceph-mon[97633]: pgmap v5782: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:54.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:54.497222+0000) 2022-09-10T07:29:55.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:55.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:55.126611+0000) 2022-09-10T07:29:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:55.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:55.497470+0000) 2022-09-10T07:29:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:55 smithi110 ceph-mon[97633]: pgmap v5783: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:55 smithi104 ceph-mon[97044]: pgmap v5783: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:29:56.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:56.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:56.126846+0000) 2022-09-10T07:29:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:56.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:56.497627+0000) 2022-09-10T07:29:57.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:57.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:57.127152+0000) 2022-09-10T07:29:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:57.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:57.497781+0000) 2022-09-10T07:29:58.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:58.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:58.127387+0000) 2022-09-10T07:29:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:29:58 smithi104 ceph-mon[97044]: pgmap v5784: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:29:58 smithi110 ceph-mon[97633]: pgmap v5784: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:29:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:58.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:58.498011+0000) 2022-09-10T07:29:59.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:29:59.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:59.127649+0000) 2022-09-10T07:29:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:29:59.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:29:59.498301+0000) 2022-09-10T07:30:00.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:00.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:00.127871+0000) 2022-09-10T07:30:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:00 smithi104 ceph-mon[97044]: pgmap v5785: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:30:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:00 smithi110 ceph-mon[97633]: pgmap v5785: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:30:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:00.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:00.498498+0000) 2022-09-10T07:30:01.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:01.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:01.128085+0000) 2022-09-10T07:30:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:01.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:01.498723+0000) 2022-09-10T07:30:01.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:30:01.430+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:30:02.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:02.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:02.128393+0000) 2022-09-10T07:30:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:02 smithi104 ceph-mon[97044]: pgmap v5786: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:02 smithi110 ceph-mon[97633]: pgmap v5786: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:02.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:02.498954+0000) 2022-09-10T07:30:03.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:03.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:03.128635+0000) 2022-09-10T07:30:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:03.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:03.499242+0000) 2022-09-10T07:30:04.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:04.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:04.128869+0000) 2022-09-10T07:30:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:04 smithi104 ceph-mon[97044]: pgmap v5787: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:04 smithi110 ceph-mon[97633]: pgmap v5787: 129 pgs: 129 active+clean; 8.3 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:04.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:04.499405+0000) 2022-09-10T07:30:05.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:05.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:05.129178+0000) 2022-09-10T07:30:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:05.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:05.499660+0000) 2022-09-10T07:30:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:05 smithi110 ceph-mon[97633]: pgmap v5788: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:30:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:05 smithi104 ceph-mon[97044]: pgmap v5788: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:30:06.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:06.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:06.129442+0000) 2022-09-10T07:30:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:06.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:06.499896+0000) 2022-09-10T07:30:07.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:07.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:07.129705+0000) 2022-09-10T07:30:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:07.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:07.500213+0000) 2022-09-10T07:30:08.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:08.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:08.130001+0000) 2022-09-10T07:30:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:08 smithi110 ceph-mon[97633]: pgmap v5789: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:30:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:08 smithi110 ceph-mon[97633]: osdmap e442: 12 total, 12 up, 12 in 2022-09-10T07:30:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:08 smithi104 ceph-mon[97044]: pgmap v5789: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:30:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:08 smithi104 ceph-mon[97044]: osdmap e442: 12 total, 12 up, 12 in 2022-09-10T07:30:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:08.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:08.500436+0000) 2022-09-10T07:30:09.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:09.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:09.130284+0000) 2022-09-10T07:30:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:09.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:09.500717+0000) 2022-09-10T07:30:10.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:10.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:10.130503+0000) 2022-09-10T07:30:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:10 smithi110 ceph-mon[97633]: pgmap v5791: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:30:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:10 smithi104 ceph-mon[97044]: pgmap v5791: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:30:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:10.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:10.500969+0000) 2022-09-10T07:30:11.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:11.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:11.130772+0000) 2022-09-10T07:30:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:11.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:11.501226+0000) 2022-09-10T07:30:12.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:12.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:12.131036+0000) 2022-09-10T07:30:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:12 smithi110 ceph-mon[97633]: pgmap v5792: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:30:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:12 smithi104 ceph-mon[97044]: pgmap v5792: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:30:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:12.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:12.501471+0000) 2022-09-10T07:30:13.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:13.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:13.131262+0000) 2022-09-10T07:30:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:13.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:13.501746+0000) 2022-09-10T07:30:14.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:14.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:14.131457+0000) 2022-09-10T07:30:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:14 smithi110 ceph-mon[97633]: pgmap v5793: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:30:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:14 smithi104 ceph-mon[97044]: pgmap v5793: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:30:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:14.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:14.501947+0000) 2022-09-10T07:30:15.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:15.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:15.131602+0000) 2022-09-10T07:30:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:15.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:15.502175+0000) 2022-09-10T07:30:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:15 smithi110 ceph-mon[97633]: osdmap e443: 12 total, 12 up, 12 in 2022-09-10T07:30:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:15 smithi110 ceph-mon[97633]: pgmap v5795: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:30:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:15 smithi104 ceph-mon[97044]: osdmap e443: 12 total, 12 up, 12 in 2022-09-10T07:30:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:15 smithi104 ceph-mon[97044]: pgmap v5795: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:30:16.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:16.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:16.131844+0000) 2022-09-10T07:30:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:16.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:16.502386+0000) 2022-09-10T07:30:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:30:16.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:30:17.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:17.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:17.132138+0000) 2022-09-10T07:30:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:17.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:17.502577+0000) 2022-09-10T07:30:18.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:18.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:18.132399+0000) 2022-09-10T07:30:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:18 smithi104 ceph-mon[97044]: pgmap v5796: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T07:30:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:18 smithi110 ceph-mon[97633]: pgmap v5796: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T07:30:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:18.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:18.502821+0000) 2022-09-10T07:30:19.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:19.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:19.132661+0000) 2022-09-10T07:30:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:19.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:19.502993+0000) 2022-09-10T07:30:20.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:20.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:20.132801+0000) 2022-09-10T07:30:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:20 smithi104 ceph-mon[97044]: pgmap v5797: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:30:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:20 smithi110 ceph-mon[97633]: pgmap v5797: 129 pgs: 129 active+clean; 8.3 MiB data, 658 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:30:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:20.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:20.503221+0000) 2022-09-10T07:30:21.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:21.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:21.133042+0000) 2022-09-10T07:30:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:21.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:21.503480+0000) 2022-09-10T07:30:22.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:22.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:22.133341+0000) 2022-09-10T07:30:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:22 smithi104 ceph-mon[97044]: pgmap v5798: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:30:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:22 smithi110 ceph-mon[97633]: pgmap v5798: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:30:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:22.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:22.503698+0000) 2022-09-10T07:30:23.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:23.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:23.133558+0000) 2022-09-10T07:30:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:23.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:23.503907+0000) 2022-09-10T07:30:24.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:24.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:24.133765+0000) 2022-09-10T07:30:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:24 smithi104 ceph-mon[97044]: pgmap v5799: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:30:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:24 smithi110 ceph-mon[97633]: pgmap v5799: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:30:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:24.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:24.504187+0000) 2022-09-10T07:30:25.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:25.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:25.134029+0000) 2022-09-10T07:30:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:25.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:25.504364+0000) 2022-09-10T07:30:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:25 smithi110 ceph-mon[97633]: pgmap v5800: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:30:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:25 smithi104 ceph-mon[97044]: pgmap v5800: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:30:26.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:26.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:26.134293+0000) 2022-09-10T07:30:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:26.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:26.504586+0000) 2022-09-10T07:30:27.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:27.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:27.134490+0000) 2022-09-10T07:30:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:27.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:27.504785+0000) 2022-09-10T07:30:28.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:28.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:28.134736+0000) 2022-09-10T07:30:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:28 smithi104 ceph-mon[97044]: pgmap v5801: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:28 smithi110 ceph-mon[97633]: pgmap v5801: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:28.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:28.505076+0000) 2022-09-10T07:30:29.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:29.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:29.135011+0000) 2022-09-10T07:30:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:29.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:29.505393+0000) 2022-09-10T07:30:30.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:30.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:30.135240+0000) 2022-09-10T07:30:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:30 smithi104 ceph-mon[97044]: pgmap v5802: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:30 smithi110 ceph-mon[97633]: pgmap v5802: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:30.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:30.505660+0000) 2022-09-10T07:30:31.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:31.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:31.135495+0000) 2022-09-10T07:30:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:31.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:31.505817+0000) 2022-09-10T07:30:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:30:31.446+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:30:32.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:32.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:32.135703+0000) 2022-09-10T07:30:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:32 smithi104 ceph-mon[97044]: pgmap v5803: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:32 smithi110 ceph-mon[97633]: pgmap v5803: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:32.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:32.505992+0000) 2022-09-10T07:30:33.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:33.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:33.135961+0000) 2022-09-10T07:30:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:33.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:33.506192+0000) 2022-09-10T07:30:34.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:34.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:34.136130+0000) 2022-09-10T07:30:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:34 smithi104 ceph-mon[97044]: pgmap v5804: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:34 smithi110 ceph-mon[97633]: pgmap v5804: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:34.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:34.506480+0000) 2022-09-10T07:30:35.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:35.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:35.136298+0000) 2022-09-10T07:30:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:35.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:35.506757+0000) 2022-09-10T07:30:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:35 smithi110 ceph-mon[97633]: pgmap v5805: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:35 smithi104 ceph-mon[97044]: pgmap v5805: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:36.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:36.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:36.136540+0000) 2022-09-10T07:30:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:36.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:36.507021+0000) 2022-09-10T07:30:37.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:37.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:37.136719+0000) 2022-09-10T07:30:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:37.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:37.507286+0000) 2022-09-10T07:30:38.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:38.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:38.136929+0000) 2022-09-10T07:30:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:38 smithi104 ceph-mon[97044]: pgmap v5806: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:38 smithi110 ceph-mon[97633]: pgmap v5806: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:38.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:38.507556+0000) 2022-09-10T07:30:39.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:39.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:39.137222+0000) 2022-09-10T07:30:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:39.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:39.507793+0000) 2022-09-10T07:30:40.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:40.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:40.137433+0000) 2022-09-10T07:30:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:40 smithi104 ceph-mon[97044]: pgmap v5807: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:40 smithi110 ceph-mon[97633]: pgmap v5807: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:40.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:40.508029+0000) 2022-09-10T07:30:41.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:41.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:41.137635+0000) 2022-09-10T07:30:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:41.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:41.508193+0000) 2022-09-10T07:30:42.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:42.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:42.137811+0000) 2022-09-10T07:30:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:42 smithi104 ceph-mon[97044]: pgmap v5808: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:42 smithi110 ceph-mon[97633]: pgmap v5808: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:42.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:42.508371+0000) 2022-09-10T07:30:43.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:43.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:43.138000+0000) 2022-09-10T07:30:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:43.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:43.508632+0000) 2022-09-10T07:30:44.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:44.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:44.138237+0000) 2022-09-10T07:30:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:44 smithi104 ceph-mon[97044]: pgmap v5809: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:30:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:30:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:44 smithi110 ceph-mon[97633]: pgmap v5809: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:30:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:30:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:30:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:30:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:44.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:44.508829+0000) 2022-09-10T07:30:45.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:45.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:45.138458+0000) 2022-09-10T07:30:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:45.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:45.509072+0000) 2022-09-10T07:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:45 smithi110 ceph-mon[97633]: pgmap v5810: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:45 smithi104 ceph-mon[97044]: pgmap v5810: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:46.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:46.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:46.138644+0000) 2022-09-10T07:30:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:46.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:46.509240+0000) 2022-09-10T07:30:46.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:30:46.461+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:30:47.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:47.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:47.138840+0000) 2022-09-10T07:30:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:47.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:47.509471+0000) 2022-09-10T07:30:48.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:48.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:48.139032+0000) 2022-09-10T07:30:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:48 smithi104 ceph-mon[97044]: pgmap v5811: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:48 smithi110 ceph-mon[97633]: pgmap v5811: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:48.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:48.509732+0000) 2022-09-10T07:30:49.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:49.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:49.139263+0000) 2022-09-10T07:30:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:49.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:49.509995+0000) 2022-09-10T07:30:50.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:50.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:50.139508+0000) 2022-09-10T07:30:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:50 smithi104 ceph-mon[97044]: pgmap v5812: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:50 smithi110 ceph-mon[97633]: pgmap v5812: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:50.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:50.510261+0000) 2022-09-10T07:30:51.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:51.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:51.139772+0000) 2022-09-10T07:30:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:51.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:51.510510+0000) 2022-09-10T07:30:52.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:52.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:52.139920+0000) 2022-09-10T07:30:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:52 smithi104 ceph-mon[97044]: pgmap v5813: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:52 smithi110 ceph-mon[97633]: pgmap v5813: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:52.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:52.510739+0000) 2022-09-10T07:30:53.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:53.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:53.140151+0000) 2022-09-10T07:30:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:53.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:53.510999+0000) 2022-09-10T07:30:54.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:54.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:54.140322+0000) 2022-09-10T07:30:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:54 smithi104 ceph-mon[97044]: pgmap v5814: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:54 smithi110 ceph-mon[97633]: pgmap v5814: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:54.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:54.511162+0000) 2022-09-10T07:30:55.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:55.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:55.140578+0000) 2022-09-10T07:30:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:55.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:55.511375+0000) 2022-09-10T07:30:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:55 smithi110 ceph-mon[97633]: pgmap v5815: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:55 smithi104 ceph-mon[97044]: pgmap v5815: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:30:56.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:56.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:56.140833+0000) 2022-09-10T07:30:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:56.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:56.511620+0000) 2022-09-10T07:30:57.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:57.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:57.141137+0000) 2022-09-10T07:30:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:57.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:57.511876+0000) 2022-09-10T07:30:58.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:58.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:58.141373+0000) 2022-09-10T07:30:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:30:58 smithi104 ceph-mon[97044]: pgmap v5816: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:30:58 smithi110 ceph-mon[97633]: pgmap v5816: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:30:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:58.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:58.512040+0000) 2022-09-10T07:30:59.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:30:59.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:59.141556+0000) 2022-09-10T07:30:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:30:59.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:30:59.512278+0000) 2022-09-10T07:31:00.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:00.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:00.141704+0000) 2022-09-10T07:31:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:00 smithi104 ceph-mon[97044]: pgmap v5817: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:00 smithi110 ceph-mon[97633]: pgmap v5817: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:00.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:00.512490+0000) 2022-09-10T07:31:01.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:01.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:01.141876+0000) 2022-09-10T07:31:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:01.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:01.512706+0000) 2022-09-10T07:31:01.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:31:01.473+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:31:02.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:02.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:02.142086+0000) 2022-09-10T07:31:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:02 smithi104 ceph-mon[97044]: pgmap v5818: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:02 smithi110 ceph-mon[97633]: pgmap v5818: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:02.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:02.512942+0000) 2022-09-10T07:31:03.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:03.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:03.142369+0000) 2022-09-10T07:31:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:03.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:03.513109+0000) 2022-09-10T07:31:04.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:04.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:04.142625+0000) 2022-09-10T07:31:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:04 smithi110 ceph-mon[97633]: pgmap v5819: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:04 smithi104 ceph-mon[97044]: pgmap v5819: 129 pgs: 129 active+clean; 8.3 MiB data, 659 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:04.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:04.513357+0000) 2022-09-10T07:31:05.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:05.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:05.142885+0000) 2022-09-10T07:31:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:05.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:05.513605+0000) 2022-09-10T07:31:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:05 smithi110 ceph-mon[97633]: pgmap v5820: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:31:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:05 smithi104 ceph-mon[97044]: pgmap v5820: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:31:06.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:06.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:06.143176+0000) 2022-09-10T07:31:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:06.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:06.513827+0000) 2022-09-10T07:31:07.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:07.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:07.143396+0000) 2022-09-10T07:31:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:07.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:07.514019+0000) 2022-09-10T07:31:08.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:08.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:08.143601+0000) 2022-09-10T07:31:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:08 smithi110 ceph-mon[97633]: pgmap v5821: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:31:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:08 smithi110 ceph-mon[97633]: osdmap e444: 12 total, 12 up, 12 in 2022-09-10T07:31:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:08 smithi104 ceph-mon[97044]: pgmap v5821: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:31:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:08 smithi104 ceph-mon[97044]: osdmap e444: 12 total, 12 up, 12 in 2022-09-10T07:31:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:08.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:08.514232+0000) 2022-09-10T07:31:09.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:09.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:09.143822+0000) 2022-09-10T07:31:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:09.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:09.514466+0000) 2022-09-10T07:31:10.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:10.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:10.144044+0000) 2022-09-10T07:31:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:10 smithi110 ceph-mon[97633]: pgmap v5823: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:31:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:10 smithi104 ceph-mon[97044]: pgmap v5823: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:31:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:10.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:10.514736+0000) 2022-09-10T07:31:11.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:11.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:11.144244+0000) 2022-09-10T07:31:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:11.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:11.514971+0000) 2022-09-10T07:31:12.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:12.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:12.144449+0000) 2022-09-10T07:31:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:12 smithi110 ceph-mon[97633]: pgmap v5824: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:31:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:12 smithi104 ceph-mon[97044]: pgmap v5824: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:31:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:12.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:12.515200+0000) 2022-09-10T07:31:13.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:13.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:13.144694+0000) 2022-09-10T07:31:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:13.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:13.515442+0000) 2022-09-10T07:31:14.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:14.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:14.144896+0000) 2022-09-10T07:31:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:14 smithi110 ceph-mon[97633]: pgmap v5825: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:31:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:14 smithi104 ceph-mon[97044]: pgmap v5825: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:31:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:14.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:14.515647+0000) 2022-09-10T07:31:15.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:15.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:15.145172+0000) 2022-09-10T07:31:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:15.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:15.515881+0000) 2022-09-10T07:31:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:15 smithi110 ceph-mon[97633]: osdmap e445: 12 total, 12 up, 12 in 2022-09-10T07:31:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:15 smithi110 ceph-mon[97633]: pgmap v5827: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:31:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:15 smithi104 ceph-mon[97044]: osdmap e445: 12 total, 12 up, 12 in 2022-09-10T07:31:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:15 smithi104 ceph-mon[97044]: pgmap v5827: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:31:16.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:16.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:16.145345+0000) 2022-09-10T07:31:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:16.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:16.516092+0000) 2022-09-10T07:31:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:31:16.489+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:31:17.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:17.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:17.145531+0000) 2022-09-10T07:31:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:17.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:17.516351+0000) 2022-09-10T07:31:18.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:18.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:18.145771+0000) 2022-09-10T07:31:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:18 smithi110 ceph-mon[97633]: pgmap v5828: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T07:31:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:18 smithi104 ceph-mon[97044]: pgmap v5828: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T07:31:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:18.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:18.516548+0000) 2022-09-10T07:31:19.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:19.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:19.146021+0000) 2022-09-10T07:31:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:19.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:19.516732+0000) 2022-09-10T07:31:20.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:20.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:20.146263+0000) 2022-09-10T07:31:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:20 smithi110 ceph-mon[97633]: pgmap v5829: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:31:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:20 smithi104 ceph-mon[97044]: pgmap v5829: 129 pgs: 129 active+clean; 8.3 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:31:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:20.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:20.516876+0000) 2022-09-10T07:31:21.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:21.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:21.146474+0000) 2022-09-10T07:31:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:21.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:21.517160+0000) 2022-09-10T07:31:22.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:22.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:22.146705+0000) 2022-09-10T07:31:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:22 smithi110 ceph-mon[97633]: pgmap v5830: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:31:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:22 smithi104 ceph-mon[97044]: pgmap v5830: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:31:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:22.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:22.517396+0000) 2022-09-10T07:31:23.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:23.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:23.146919+0000) 2022-09-10T07:31:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:23.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:23.517557+0000) 2022-09-10T07:31:24.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:24.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:24.147189+0000) 2022-09-10T07:31:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:24 smithi110 ceph-mon[97633]: pgmap v5831: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:31:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:24 smithi104 ceph-mon[97044]: pgmap v5831: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:31:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:24.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:24.517779+0000) 2022-09-10T07:31:25.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:25.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:25.147390+0000) 2022-09-10T07:31:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:25.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:25.518040+0000) 2022-09-10T07:31:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:25 smithi110 ceph-mon[97633]: pgmap v5832: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:31:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:25 smithi104 ceph-mon[97044]: pgmap v5832: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:31:26.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:26.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:26.147638+0000) 2022-09-10T07:31:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:26.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:26.518239+0000) 2022-09-10T07:31:27.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:27.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:27.147877+0000) 2022-09-10T07:31:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:27.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:27.518397+0000) 2022-09-10T07:31:28.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:28.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:28.148019+0000) 2022-09-10T07:31:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:28 smithi110 ceph-mon[97633]: pgmap v5833: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:28 smithi104 ceph-mon[97044]: pgmap v5833: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:28.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:28.518625+0000) 2022-09-10T07:31:29.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:29.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:29.148203+0000) 2022-09-10T07:31:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:29.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:29.518834+0000) 2022-09-10T07:31:30.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:30.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:30.148404+0000) 2022-09-10T07:31:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:30 smithi110 ceph-mon[97633]: pgmap v5834: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:30 smithi104 ceph-mon[97044]: pgmap v5834: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:30.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:30.519082+0000) 2022-09-10T07:31:31.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:31.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:31.148696+0000) 2022-09-10T07:31:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:31.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:31.519225+0000) 2022-09-10T07:31:31.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:31:31.504+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:31:32.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:32.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:32.148890+0000) 2022-09-10T07:31:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:32 smithi110 ceph-mon[97633]: pgmap v5835: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:32 smithi104 ceph-mon[97044]: pgmap v5835: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:32.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:32.519407+0000) 2022-09-10T07:31:33.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:33.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:33.149083+0000) 2022-09-10T07:31:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:33.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:33.519593+0000) 2022-09-10T07:31:34.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:34.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:34.149294+0000) 2022-09-10T07:31:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:34 smithi110 ceph-mon[97633]: pgmap v5836: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:34 smithi104 ceph-mon[97044]: pgmap v5836: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:34.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:34.519828+0000) 2022-09-10T07:31:35.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:35.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:35.149530+0000) 2022-09-10T07:31:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:35.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:35.520038+0000) 2022-09-10T07:31:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:35 smithi110 ceph-mon[97633]: pgmap v5837: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:35 smithi104 ceph-mon[97044]: pgmap v5837: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:36.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:36.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:36.149807+0000) 2022-09-10T07:31:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:36.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:36.520270+0000) 2022-09-10T07:31:37.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:37.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:37.150082+0000) 2022-09-10T07:31:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:37.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:37.520545+0000) 2022-09-10T07:31:38.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:38.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:38.150316+0000) 2022-09-10T07:31:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:38 smithi110 ceph-mon[97633]: pgmap v5838: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:38 smithi104 ceph-mon[97044]: pgmap v5838: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:38.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:38.520747+0000) 2022-09-10T07:31:39.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:39.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:39.150561+0000) 2022-09-10T07:31:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:39.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:39.521002+0000) 2022-09-10T07:31:40.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:40.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:40.150738+0000) 2022-09-10T07:31:40.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:40 smithi110 ceph-mon[97633]: pgmap v5839: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:40 smithi104 ceph-mon[97044]: pgmap v5839: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:40.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:40.521232+0000) 2022-09-10T07:31:41.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:41.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:41.150924+0000) 2022-09-10T07:31:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:41.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:41.521440+0000) 2022-09-10T07:31:42.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:42.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:42.151201+0000) 2022-09-10T07:31:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:42 smithi110 ceph-mon[97633]: pgmap v5840: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:42 smithi104 ceph-mon[97044]: pgmap v5840: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:42.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:42.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:42.521566+0000) 2022-09-10T07:31:43.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:43.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:43.151468+0000) 2022-09-10T07:31:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:43.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:43.521802+0000) 2022-09-10T07:31:44.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:44.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:44.151712+0000) 2022-09-10T07:31:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:44 smithi110 ceph-mon[97633]: pgmap v5841: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:31:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:31:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:31:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:31:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:44 smithi104 ceph-mon[97044]: pgmap v5841: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:31:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:31:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:31:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:31:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:44.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:44.522032+0000) 2022-09-10T07:31:45.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:45.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:45.151940+0000) 2022-09-10T07:31:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:45.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:45.522263+0000) 2022-09-10T07:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:45 smithi110 ceph-mon[97633]: pgmap v5842: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:45 smithi104 ceph-mon[97044]: pgmap v5842: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:46.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:46.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:46.152082+0000) 2022-09-10T07:31:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:46.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:46.522441+0000) 2022-09-10T07:31:46.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:31:46.520+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:31:47.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:47.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:47.152281+0000) 2022-09-10T07:31:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:47.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:47.522664+0000) 2022-09-10T07:31:48.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:48.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:48.152462+0000) 2022-09-10T07:31:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:48 smithi110 ceph-mon[97633]: pgmap v5843: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:48 smithi104 ceph-mon[97044]: pgmap v5843: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:48.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:48.522916+0000) 2022-09-10T07:31:49.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:49.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:49.152645+0000) 2022-09-10T07:31:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:49.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:49.523158+0000) 2022-09-10T07:31:50.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:50.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:50.152879+0000) 2022-09-10T07:31:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:50 smithi110 ceph-mon[97633]: pgmap v5844: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:50 smithi104 ceph-mon[97044]: pgmap v5844: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:50.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:50.523401+0000) 2022-09-10T07:31:51.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:51.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:51.153056+0000) 2022-09-10T07:31:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:51.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:51.523647+0000) 2022-09-10T07:31:52.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:52.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:52.153309+0000) 2022-09-10T07:31:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:52 smithi110 ceph-mon[97633]: pgmap v5845: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:52 smithi104 ceph-mon[97044]: pgmap v5845: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:52.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:52.523954+0000) 2022-09-10T07:31:53.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:53.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:53.153475+0000) 2022-09-10T07:31:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:53.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:53.524212+0000) 2022-09-10T07:31:54.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:54.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:54.153741+0000) 2022-09-10T07:31:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:54 smithi110 ceph-mon[97633]: pgmap v5846: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:54 smithi104 ceph-mon[97044]: pgmap v5846: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:54.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:54.524440+0000) 2022-09-10T07:31:55.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:55.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:55.153984+0000) 2022-09-10T07:31:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:55.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:55.524648+0000) 2022-09-10T07:31:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:55 smithi110 ceph-mon[97633]: pgmap v5847: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:55 smithi104 ceph-mon[97044]: pgmap v5847: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:31:56.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:56.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:56.154239+0000) 2022-09-10T07:31:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:56.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:56.524903+0000) 2022-09-10T07:31:57.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:57.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:57.154505+0000) 2022-09-10T07:31:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:57.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:57.525119+0000) 2022-09-10T07:31:58.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:58.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:58.154657+0000) 2022-09-10T07:31:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:31:58 smithi110 ceph-mon[97633]: pgmap v5848: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:31:58 smithi104 ceph-mon[97044]: pgmap v5848: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:31:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:58.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:58.525338+0000) 2022-09-10T07:31:59.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:31:59.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:59.154823+0000) 2022-09-10T07:31:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:31:59.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:31:59.525568+0000) 2022-09-10T07:32:00.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:00.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:00.155066+0000) 2022-09-10T07:32:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:00 smithi110 ceph-mon[97633]: pgmap v5849: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:00 smithi104 ceph-mon[97044]: pgmap v5849: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:00.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:00.525806+0000) 2022-09-10T07:32:01.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:01.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:01.155306+0000) 2022-09-10T07:32:01.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:32:01.522+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:32:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:01.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:01.526031+0000) 2022-09-10T07:32:02.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:02.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:02.155579+0000) 2022-09-10T07:32:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:02 smithi110 ceph-mon[97633]: pgmap v5850: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:02 smithi104 ceph-mon[97044]: pgmap v5850: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:02.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:02.526284+0000) 2022-09-10T07:32:03.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:03.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:03.155740+0000) 2022-09-10T07:32:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:03.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:03.526555+0000) 2022-09-10T07:32:04.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:04.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:04.156010+0000) 2022-09-10T07:32:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:04 smithi110 ceph-mon[97633]: pgmap v5851: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:04 smithi104 ceph-mon[97044]: pgmap v5851: 129 pgs: 129 active+clean; 8.3 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:04.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:04.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:04.526756+0000) 2022-09-10T07:32:05.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:05.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:05.156292+0000) 2022-09-10T07:32:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:05.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:05.526990+0000) 2022-09-10T07:32:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:05 smithi110 ceph-mon[97633]: pgmap v5852: 129 pgs: 129 active+clean; 8.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:32:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:05 smithi104 ceph-mon[97044]: pgmap v5852: 129 pgs: 129 active+clean; 8.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:32:06.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:06.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:06.156574+0000) 2022-09-10T07:32:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:06.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:06.527267+0000) 2022-09-10T07:32:07.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:07.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:07.156852+0000) 2022-09-10T07:32:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:07.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:07.527560+0000) 2022-09-10T07:32:08.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:08.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:08.157148+0000) 2022-09-10T07:32:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:08 smithi110 ceph-mon[97633]: pgmap v5853: 129 pgs: 129 active+clean; 8.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:32:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:08 smithi110 ceph-mon[97633]: osdmap e446: 12 total, 12 up, 12 in 2022-09-10T07:32:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:08 smithi104 ceph-mon[97044]: pgmap v5853: 129 pgs: 129 active+clean; 8.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:32:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:08 smithi104 ceph-mon[97044]: osdmap e446: 12 total, 12 up, 12 in 2022-09-10T07:32:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:08.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:08.527763+0000) 2022-09-10T07:32:09.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:09.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:09.157410+0000) 2022-09-10T07:32:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:09.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:09.528006+0000) 2022-09-10T07:32:10.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:10.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:10.157679+0000) 2022-09-10T07:32:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:10 smithi110 ceph-mon[97633]: pgmap v5855: 129 pgs: 129 active+clean; 8.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:32:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:10 smithi104 ceph-mon[97044]: pgmap v5855: 129 pgs: 129 active+clean; 8.3 MiB data, 670 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:32:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:10.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:10.528242+0000) 2022-09-10T07:32:11.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:11.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:11.157965+0000) 2022-09-10T07:32:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:11.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:11.528439+0000) 2022-09-10T07:32:12.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:12.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:12.158242+0000) 2022-09-10T07:32:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:12 smithi110 ceph-mon[97633]: pgmap v5856: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:32:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:12 smithi104 ceph-mon[97044]: pgmap v5856: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:32:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:12.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:12.528659+0000) 2022-09-10T07:32:13.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:13.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:13.158521+0000) 2022-09-10T07:32:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:13.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:13.528890+0000) 2022-09-10T07:32:14.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:14.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:14.158780+0000) 2022-09-10T07:32:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:14 smithi110 ceph-mon[97633]: pgmap v5857: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:32:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:14 smithi104 ceph-mon[97044]: pgmap v5857: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:32:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:14.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:14.529126+0000) 2022-09-10T07:32:15.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:15.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:15.159028+0000) 2022-09-10T07:32:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:15.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:15.529382+0000) 2022-09-10T07:32:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:15 smithi110 ceph-mon[97633]: osdmap e447: 12 total, 12 up, 12 in 2022-09-10T07:32:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:15 smithi110 ceph-mon[97633]: pgmap v5859: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:32:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:15 smithi104 ceph-mon[97044]: osdmap e447: 12 total, 12 up, 12 in 2022-09-10T07:32:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:15 smithi104 ceph-mon[97044]: pgmap v5859: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:32:16.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:16.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:16.159235+0000) 2022-09-10T07:32:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:16.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:16.529625+0000) 2022-09-10T07:32:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:32:16.537+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:32:17.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:17.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:17.159458+0000) 2022-09-10T07:32:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:17.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:17.529906+0000) 2022-09-10T07:32:18.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:18.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:18.159687+0000) 2022-09-10T07:32:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:18 smithi110 ceph-mon[97633]: pgmap v5860: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T07:32:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:18 smithi104 ceph-mon[97044]: pgmap v5860: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T07:32:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:18.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:18.530210+0000) 2022-09-10T07:32:19.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:19.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:19.159917+0000) 2022-09-10T07:32:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:19.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:19.530427+0000) 2022-09-10T07:32:20.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:20.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:20.160129+0000) 2022-09-10T07:32:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:20 smithi110 ceph-mon[97633]: pgmap v5861: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:32:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:20 smithi104 ceph-mon[97044]: pgmap v5861: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:32:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:20.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:20.530640+0000) 2022-09-10T07:32:21.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:21.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:21.160390+0000) 2022-09-10T07:32:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:21.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:21.530901+0000) 2022-09-10T07:32:22.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:22.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:22.160652+0000) 2022-09-10T07:32:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:22 smithi110 ceph-mon[97633]: pgmap v5862: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:32:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:22 smithi104 ceph-mon[97044]: pgmap v5862: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:32:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:22.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:22.531209+0000) 2022-09-10T07:32:23.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:23.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:23.160916+0000) 2022-09-10T07:32:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:23.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:23.531427+0000) 2022-09-10T07:32:24.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:24.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:24.161047+0000) 2022-09-10T07:32:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:24 smithi110 ceph-mon[97633]: pgmap v5863: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:32:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:24 smithi104 ceph-mon[97044]: pgmap v5863: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:32:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:24.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:24.531665+0000) 2022-09-10T07:32:25.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:25.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:25.161239+0000) 2022-09-10T07:32:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:25.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:25.531938+0000) 2022-09-10T07:32:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:25 smithi110 ceph-mon[97633]: pgmap v5864: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:32:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:25 smithi104 ceph-mon[97044]: pgmap v5864: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:32:26.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:26.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:26.161502+0000) 2022-09-10T07:32:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:26.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:26.532120+0000) 2022-09-10T07:32:27.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:27.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:27.161800+0000) 2022-09-10T07:32:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:27.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:27.532289+0000) 2022-09-10T07:32:28.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:28.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:28.162031+0000) 2022-09-10T07:32:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:28 smithi110 ceph-mon[97633]: pgmap v5865: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:28 smithi104 ceph-mon[97044]: pgmap v5865: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:28.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:28.532444+0000) 2022-09-10T07:32:29.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:29.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:29.162267+0000) 2022-09-10T07:32:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:29.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:29.532616+0000) 2022-09-10T07:32:30.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:30.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:30.162460+0000) 2022-09-10T07:32:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:30 smithi110 ceph-mon[97633]: pgmap v5866: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:30 smithi104 ceph-mon[97044]: pgmap v5866: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:30.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:30.532805+0000) 2022-09-10T07:32:31.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:31.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:31.162652+0000) 2022-09-10T07:32:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:31.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:31.533035+0000) 2022-09-10T07:32:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:32:31.548+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:32:32.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:32.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:32.162914+0000) 2022-09-10T07:32:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:32 smithi110 ceph-mon[97633]: pgmap v5867: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:32 smithi104 ceph-mon[97044]: pgmap v5867: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:32.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:32.533292+0000) 2022-09-10T07:32:33.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:33.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:33.163176+0000) 2022-09-10T07:32:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:33.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:33.533535+0000) 2022-09-10T07:32:34.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:34.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:34.163422+0000) 2022-09-10T07:32:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:34 smithi110 ceph-mon[97633]: pgmap v5868: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:34 smithi104 ceph-mon[97044]: pgmap v5868: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:34.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:34.533801+0000) 2022-09-10T07:32:35.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:35.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:35.163583+0000) 2022-09-10T07:32:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:35.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:35.534076+0000) 2022-09-10T07:32:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:35 smithi110 ceph-mon[97633]: pgmap v5869: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:35 smithi104 ceph-mon[97044]: pgmap v5869: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:36.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:36.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:36.163833+0000) 2022-09-10T07:32:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:36.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:36.534313+0000) 2022-09-10T07:32:37.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:37.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:37.164089+0000) 2022-09-10T07:32:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:37.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:37.534485+0000) 2022-09-10T07:32:38.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:38.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:38.164295+0000) 2022-09-10T07:32:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:38 smithi110 ceph-mon[97633]: pgmap v5870: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:38 smithi104 ceph-mon[97044]: pgmap v5870: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:38.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:38.534719+0000) 2022-09-10T07:32:39.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:39.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:39.164523+0000) 2022-09-10T07:32:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:39.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:39.534961+0000) 2022-09-10T07:32:40.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:40.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:40.164796+0000) 2022-09-10T07:32:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:40 smithi110 ceph-mon[97633]: pgmap v5871: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:40 smithi104 ceph-mon[97044]: pgmap v5871: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:40.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:40.535189+0000) 2022-09-10T07:32:41.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:41.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:41.165068+0000) 2022-09-10T07:32:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:41.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:41.535430+0000) 2022-09-10T07:32:42.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:42.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:42.165173+0000) 2022-09-10T07:32:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:42 smithi110 ceph-mon[97633]: pgmap v5872: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:42 smithi104 ceph-mon[97044]: pgmap v5872: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:42.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:42.535602+0000) 2022-09-10T07:32:43.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:43.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:43.165361+0000) 2022-09-10T07:32:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:43.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:43.535834+0000) 2022-09-10T07:32:44.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:44.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:44.165599+0000) 2022-09-10T07:32:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:44 smithi110 ceph-mon[97633]: pgmap v5873: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:32:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:32:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:32:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:32:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:32:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:44 smithi104 ceph-mon[97044]: pgmap v5873: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:32:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:32:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:32:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:32:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:32:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:44.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:44.536157+0000) 2022-09-10T07:32:45.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:45.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:45.165795+0000) 2022-09-10T07:32:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:45.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:45.536330+0000) 2022-09-10T07:32:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:45 smithi110 ceph-mon[97633]: pgmap v5874: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:45 smithi104 ceph-mon[97044]: pgmap v5874: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:46.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:46.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:46.166020+0000) 2022-09-10T07:32:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:46.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:46.536524+0000) 2022-09-10T07:32:46.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:32:46.564+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:32:47.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:47.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:47.166244+0000) 2022-09-10T07:32:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:47.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:47.536782+0000) 2022-09-10T07:32:48.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:48.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:48.166484+0000) 2022-09-10T07:32:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:48 smithi110 ceph-mon[97633]: pgmap v5875: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:32:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:48 smithi104 ceph-mon[97044]: pgmap v5875: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:32:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:48.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:48.537007+0000) 2022-09-10T07:32:49.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:49.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:49.166663+0000) 2022-09-10T07:32:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:49.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:49.537291+0000) 2022-09-10T07:32:50.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:50.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:50.166853+0000) 2022-09-10T07:32:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:50 smithi110 ceph-mon[97633]: pgmap v5876: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:32:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:50 smithi104 ceph-mon[97044]: pgmap v5876: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:32:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:50.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:50.537510+0000) 2022-09-10T07:32:51.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:51.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:51.167089+0000) 2022-09-10T07:32:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:51.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:51.537631+0000) 2022-09-10T07:32:52.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:52.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:52.167309+0000) 2022-09-10T07:32:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:52 smithi110 ceph-mon[97633]: pgmap v5877: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:52 smithi104 ceph-mon[97044]: pgmap v5877: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:52.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:52.537880+0000) 2022-09-10T07:32:53.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:53.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:53.167482+0000) 2022-09-10T07:32:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:53.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:53.538143+0000) 2022-09-10T07:32:54.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:54.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:54.167694+0000) 2022-09-10T07:32:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:54 smithi110 ceph-mon[97633]: pgmap v5878: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:54 smithi104 ceph-mon[97044]: pgmap v5878: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:54.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:54.538356+0000) 2022-09-10T07:32:55.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:55.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:55.167933+0000) 2022-09-10T07:32:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:55.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:55.538574+0000) 2022-09-10T07:32:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:55 smithi110 ceph-mon[97633]: pgmap v5879: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:55 smithi104 ceph-mon[97044]: pgmap v5879: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:32:56.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:56.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:56.168210+0000) 2022-09-10T07:32:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:56.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:56.538760+0000) 2022-09-10T07:32:57.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:57.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:57.168385+0000) 2022-09-10T07:32:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:57.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:57.538962+0000) 2022-09-10T07:32:58.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:58.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:58.168504+0000) 2022-09-10T07:32:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:32:58 smithi110 ceph-mon[97633]: pgmap v5880: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:32:58 smithi104 ceph-mon[97044]: pgmap v5880: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:32:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:58.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:58.539216+0000) 2022-09-10T07:32:59.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:32:59.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:59.168659+0000) 2022-09-10T07:32:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:32:59.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:32:59.539496+0000) 2022-09-10T07:33:00.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:00.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:00.168835+0000) 2022-09-10T07:33:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:00 smithi110 ceph-mon[97633]: pgmap v5881: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:00 smithi104 ceph-mon[97044]: pgmap v5881: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:00.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:00.539649+0000) 2022-09-10T07:33:01.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:01.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:01.169083+0000) 2022-09-10T07:33:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:01.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:01.539836+0000) 2022-09-10T07:33:01.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:33:01.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:33:02.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:02.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:02.169362+0000) 2022-09-10T07:33:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:02 smithi110 ceph-mon[97633]: pgmap v5882: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:02 smithi104 ceph-mon[97044]: pgmap v5882: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:02.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:02.540089+0000) 2022-09-10T07:33:03.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:03.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:03.169628+0000) 2022-09-10T07:33:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:03.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:03.540341+0000) 2022-09-10T07:33:04.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:04.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:04.169871+0000) 2022-09-10T07:33:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:04 smithi110 ceph-mon[97633]: pgmap v5883: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:04 smithi104 ceph-mon[97044]: pgmap v5883: 129 pgs: 129 active+clean; 8.3 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:04.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:04.540586+0000) 2022-09-10T07:33:05.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:05.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:05.170168+0000) 2022-09-10T07:33:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:05.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:05.540896+0000) 2022-09-10T07:33:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:05 smithi110 ceph-mon[97633]: pgmap v5884: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:33:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:05 smithi104 ceph-mon[97044]: pgmap v5884: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:33:06.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:06.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:06.170367+0000) 2022-09-10T07:33:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:06.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:06.541065+0000) 2022-09-10T07:33:07.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:07.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:07.170605+0000) 2022-09-10T07:33:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:07.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:07.541237+0000) 2022-09-10T07:33:08.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:08.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:08.170788+0000) 2022-09-10T07:33:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:08 smithi110 ceph-mon[97633]: pgmap v5885: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:33:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:08 smithi110 ceph-mon[97633]: osdmap e448: 12 total, 12 up, 12 in 2022-09-10T07:33:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:08 smithi104 ceph-mon[97044]: pgmap v5885: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:33:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:08 smithi104 ceph-mon[97044]: osdmap e448: 12 total, 12 up, 12 in 2022-09-10T07:33:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:08.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:08.541405+0000) 2022-09-10T07:33:09.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:09.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:09.171042+0000) 2022-09-10T07:33:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:09.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:09.541569+0000) 2022-09-10T07:33:10.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:10.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:10.171276+0000) 2022-09-10T07:33:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:10 smithi110 ceph-mon[97633]: pgmap v5887: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:33:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:10 smithi104 ceph-mon[97044]: pgmap v5887: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:33:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:10.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:10.541744+0000) 2022-09-10T07:33:11.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:11.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:11.171485+0000) 2022-09-10T07:33:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:11.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:11.541948+0000) 2022-09-10T07:33:12.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:12.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:12.171747+0000) 2022-09-10T07:33:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:12 smithi110 ceph-mon[97633]: pgmap v5888: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:33:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:12 smithi104 ceph-mon[97044]: pgmap v5888: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:33:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:12.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:12.542155+0000) 2022-09-10T07:33:13.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:13.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:13.171952+0000) 2022-09-10T07:33:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:13.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:13.542364+0000) 2022-09-10T07:33:14.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:14.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:14.172188+0000) 2022-09-10T07:33:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:14 smithi110 ceph-mon[97633]: pgmap v5889: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:33:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:14 smithi104 ceph-mon[97044]: pgmap v5889: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:33:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:14.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:14.542630+0000) 2022-09-10T07:33:15.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:15.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:15.172402+0000) 2022-09-10T07:33:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:15.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:15.542822+0000) 2022-09-10T07:33:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:15 smithi110 ceph-mon[97633]: osdmap e449: 12 total, 12 up, 12 in 2022-09-10T07:33:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:15 smithi110 ceph-mon[97633]: pgmap v5891: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:33:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:15 smithi104 ceph-mon[97044]: osdmap e449: 12 total, 12 up, 12 in 2022-09-10T07:33:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:15 smithi104 ceph-mon[97044]: pgmap v5891: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:33:16.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:16.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:16.172636+0000) 2022-09-10T07:33:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:16.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:16.543003+0000) 2022-09-10T07:33:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:33:16.594+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:33:17.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:17.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:17.172867+0000) 2022-09-10T07:33:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:17.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:17.543290+0000) 2022-09-10T07:33:18.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:18.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:18.173150+0000) 2022-09-10T07:33:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:18 smithi110 ceph-mon[97633]: pgmap v5892: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T07:33:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:18 smithi104 ceph-mon[97044]: pgmap v5892: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T07:33:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:18.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:18.543444+0000) 2022-09-10T07:33:19.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:19.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:19.173335+0000) 2022-09-10T07:33:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:19.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:19.543670+0000) 2022-09-10T07:33:20.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:20.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:20.173612+0000) 2022-09-10T07:33:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:20 smithi110 ceph-mon[97633]: pgmap v5893: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:33:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:20 smithi104 ceph-mon[97044]: pgmap v5893: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:33:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:20.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:20.543897+0000) 2022-09-10T07:33:21.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:21.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:21.173837+0000) 2022-09-10T07:33:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:21.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:21.544140+0000) 2022-09-10T07:33:22.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:22.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:22.174070+0000) 2022-09-10T07:33:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:22 smithi110 ceph-mon[97633]: pgmap v5894: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:33:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:22 smithi104 ceph-mon[97044]: pgmap v5894: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:33:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:22.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:22.544352+0000) 2022-09-10T07:33:23.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:23.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:23.174269+0000) 2022-09-10T07:33:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:23.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:23.544580+0000) 2022-09-10T07:33:24.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:24.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:24.174446+0000) 2022-09-10T07:33:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:24 smithi110 ceph-mon[97633]: pgmap v5895: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:33:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:24 smithi104 ceph-mon[97044]: pgmap v5895: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:33:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:24.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:24.544743+0000) 2022-09-10T07:33:25.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:25.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:25.174666+0000) 2022-09-10T07:33:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:25.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:25.544937+0000) 2022-09-10T07:33:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:25 smithi110 ceph-mon[97633]: pgmap v5896: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:33:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:25 smithi104 ceph-mon[97044]: pgmap v5896: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:33:26.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:26.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:26.174920+0000) 2022-09-10T07:33:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:26.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:26.545198+0000) 2022-09-10T07:33:27.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:27.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:27.175168+0000) 2022-09-10T07:33:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:27.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:27.545422+0000) 2022-09-10T07:33:28.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:28.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:28.175329+0000) 2022-09-10T07:33:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:28 smithi110 ceph-mon[97633]: pgmap v5897: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:28 smithi104 ceph-mon[97044]: pgmap v5897: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:28.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:28.545665+0000) 2022-09-10T07:33:29.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:29.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:29.175585+0000) 2022-09-10T07:33:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:29.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:29.545857+0000) 2022-09-10T07:33:30.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:30.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:30.175812+0000) 2022-09-10T07:33:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:30 smithi110 ceph-mon[97633]: pgmap v5898: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T07:33:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:30 smithi104 ceph-mon[97044]: pgmap v5898: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T07:33:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:30.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:30.546076+0000) 2022-09-10T07:33:31.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:31.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:31.175999+0000) 2022-09-10T07:33:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:33:31.599+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:33:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:31.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:31.546255+0000) 2022-09-10T07:33:32.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:32.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:32.176193+0000) 2022-09-10T07:33:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:32 smithi110 ceph-mon[97633]: pgmap v5899: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:32 smithi104 ceph-mon[97044]: pgmap v5899: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:32.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:32.546449+0000) 2022-09-10T07:33:33.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:33.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:33.176405+0000) 2022-09-10T07:33:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:33.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:33.546689+0000) 2022-09-10T07:33:34.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:34.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:34.176660+0000) 2022-09-10T07:33:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:34 smithi110 ceph-mon[97633]: pgmap v5900: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:34 smithi104 ceph-mon[97044]: pgmap v5900: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:34.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:34.546894+0000) 2022-09-10T07:33:35.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:35.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:35.176915+0000) 2022-09-10T07:33:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:35.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:35.547135+0000) 2022-09-10T07:33:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:35 smithi110 ceph-mon[97633]: pgmap v5901: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:35 smithi104 ceph-mon[97044]: pgmap v5901: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:36.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:36.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:36.177196+0000) 2022-09-10T07:33:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:36.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:36.547320+0000) 2022-09-10T07:33:37.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:37.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:37.177410+0000) 2022-09-10T07:33:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:37.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:37.547519+0000) 2022-09-10T07:33:38.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:38.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:38.177694+0000) 2022-09-10T07:33:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:38 smithi110 ceph-mon[97633]: pgmap v5902: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:38 smithi104 ceph-mon[97044]: pgmap v5902: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:38.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:38.547741+0000) 2022-09-10T07:33:39.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:39.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:39.177944+0000) 2022-09-10T07:33:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:39.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:39.547932+0000) 2022-09-10T07:33:40.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:40.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:40.178213+0000) 2022-09-10T07:33:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:40 smithi110 ceph-mon[97633]: pgmap v5903: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T07:33:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:40 smithi104 ceph-mon[97044]: pgmap v5903: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T07:33:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:40.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:40.548167+0000) 2022-09-10T07:33:41.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:41.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:41.178446+0000) 2022-09-10T07:33:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:41.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:41.548456+0000) 2022-09-10T07:33:42.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:42.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:42.178638+0000) 2022-09-10T07:33:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:42 smithi110 ceph-mon[97633]: pgmap v5904: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T07:33:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:42 smithi104 ceph-mon[97044]: pgmap v5904: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T07:33:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:42.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:42.548630+0000) 2022-09-10T07:33:43.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:43.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:43.178872+0000) 2022-09-10T07:33:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:43.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:43.548850+0000) 2022-09-10T07:33:44.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:44.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:44.179089+0000) 2022-09-10T07:33:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:44 smithi110 ceph-mon[97633]: pgmap v5905: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:33:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:33:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:33:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:33:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:44 smithi104 ceph-mon[97044]: pgmap v5905: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:33:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:33:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:33:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:33:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:44.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:44.549018+0000) 2022-09-10T07:33:45.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:45.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:45.179326+0000) 2022-09-10T07:33:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:45.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:45.549232+0000) 2022-09-10T07:33:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:45 smithi110 ceph-mon[97633]: pgmap v5906: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:45 smithi104 ceph-mon[97044]: pgmap v5906: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:46.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:46.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:46.179558+0000) 2022-09-10T07:33:46.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:33:46.614+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:33:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:46.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:46.549476+0000) 2022-09-10T07:33:47.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:47.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:47.179809+0000) 2022-09-10T07:33:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:47.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:47.549694+0000) 2022-09-10T07:33:48.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:48.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:48.180038+0000) 2022-09-10T07:33:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:48 smithi110 ceph-mon[97633]: pgmap v5907: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:48 smithi104 ceph-mon[97044]: pgmap v5907: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:48.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:48.549947+0000) 2022-09-10T07:33:49.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:49.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:49.180248+0000) 2022-09-10T07:33:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:49.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:49.550177+0000) 2022-09-10T07:33:50.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:50.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:50.180409+0000) 2022-09-10T07:33:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:50 smithi110 ceph-mon[97633]: pgmap v5908: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T07:33:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:50 smithi104 ceph-mon[97044]: pgmap v5908: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T07:33:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:50.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:50.550430+0000) 2022-09-10T07:33:51.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:51.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:51.180662+0000) 2022-09-10T07:33:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:51.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:51.550653+0000) 2022-09-10T07:33:52.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:52.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:52.180909+0000) 2022-09-10T07:33:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:52 smithi110 ceph-mon[97633]: pgmap v5909: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T07:33:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:52 smithi104 ceph-mon[97044]: pgmap v5909: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T07:33:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:52.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:52.550887+0000) 2022-09-10T07:33:53.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:53.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:53.181119+0000) 2022-09-10T07:33:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:53.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:53.551194+0000) 2022-09-10T07:33:54.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:54.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:54.181330+0000) 2022-09-10T07:33:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:54 smithi110 ceph-mon[97633]: pgmap v5910: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:54 smithi104 ceph-mon[97044]: pgmap v5910: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:54.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:54.551437+0000) 2022-09-10T07:33:55.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:55.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:55.181582+0000) 2022-09-10T07:33:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:55.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:55.551613+0000) 2022-09-10T07:33:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:55 smithi110 ceph-mon[97633]: pgmap v5911: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:55 smithi104 ceph-mon[97044]: pgmap v5911: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:33:56.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:56.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:56.181845+0000) 2022-09-10T07:33:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:56.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:56.551871+0000) 2022-09-10T07:33:57.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:57.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:57.182060+0000) 2022-09-10T07:33:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:57.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:57.552150+0000) 2022-09-10T07:33:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:58.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:58.182251+0000) 2022-09-10T07:33:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:33:58 smithi110 ceph-mon[97633]: pgmap v5912: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:33:58 smithi104 ceph-mon[97044]: pgmap v5912: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:33:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:58.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:58.552402+0000) 2022-09-10T07:33:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:33:59.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:59.182482+0000) 2022-09-10T07:33:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:33:59.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:33:59.552597+0000) 2022-09-10T07:34:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:00.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:00.182773+0000) 2022-09-10T07:34:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:00 smithi110 ceph-mon[97633]: pgmap v5913: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T07:34:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:00 smithi104 ceph-mon[97044]: pgmap v5913: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T07:34:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:00.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:00.552777+0000) 2022-09-10T07:34:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:01.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:01.183001+0000) 2022-09-10T07:34:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:01.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:01.553021+0000) 2022-09-10T07:34:01.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:34:01.625+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:34:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:02.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:02.183288+0000) 2022-09-10T07:34:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:02 smithi110 ceph-mon[97633]: pgmap v5914: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T07:34:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:02 smithi104 ceph-mon[97044]: pgmap v5914: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T07:34:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:02.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:02.553280+0000) 2022-09-10T07:34:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:03.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:03.183478+0000) 2022-09-10T07:34:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:03.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:03.553551+0000) 2022-09-10T07:34:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:04.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:04.183643+0000) 2022-09-10T07:34:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:04 smithi110 ceph-mon[97633]: pgmap v5915: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:04 smithi104 ceph-mon[97044]: pgmap v5915: 129 pgs: 129 active+clean; 8.3 MiB data, 677 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:04.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:04.553837+0000) 2022-09-10T07:34:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:05.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:05.183895+0000) 2022-09-10T07:34:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:05.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:05.553973+0000) 2022-09-10T07:34:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:05 smithi110 ceph-mon[97633]: pgmap v5916: 129 pgs: 129 active+clean; 8.3 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:34:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:05 smithi104 ceph-mon[97044]: pgmap v5916: 129 pgs: 129 active+clean; 8.3 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:34:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:06.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:06.184142+0000) 2022-09-10T07:34:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:06.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:06.554112+0000) 2022-09-10T07:34:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:07.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:07.184353+0000) 2022-09-10T07:34:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:07.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:07.554249+0000) 2022-09-10T07:34:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:08.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:08.184606+0000) 2022-09-10T07:34:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:08 smithi110 ceph-mon[97633]: pgmap v5917: 129 pgs: 129 active+clean; 8.3 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:34:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:08 smithi110 ceph-mon[97633]: osdmap e450: 12 total, 12 up, 12 in 2022-09-10T07:34:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:08 smithi104 ceph-mon[97044]: pgmap v5917: 129 pgs: 129 active+clean; 8.3 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:34:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:08 smithi104 ceph-mon[97044]: osdmap e450: 12 total, 12 up, 12 in 2022-09-10T07:34:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:08.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:08.554415+0000) 2022-09-10T07:34:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:09.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:09.184780+0000) 2022-09-10T07:34:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:09.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:09.554585+0000) 2022-09-10T07:34:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:10.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:10.185043+0000) 2022-09-10T07:34:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:10 smithi110 ceph-mon[97633]: pgmap v5919: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:34:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:10 smithi104 ceph-mon[97044]: pgmap v5919: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:34:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:10.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:10.554751+0000) 2022-09-10T07:34:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:11.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:11.185269+0000) 2022-09-10T07:34:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:11.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:11.554952+0000) 2022-09-10T07:34:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:12.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:12.185505+0000) 2022-09-10T07:34:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:12 smithi110 ceph-mon[97633]: pgmap v5920: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:34:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:12 smithi104 ceph-mon[97044]: pgmap v5920: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:34:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:12.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:12.555128+0000) 2022-09-10T07:34:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:13.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:13.185743+0000) 2022-09-10T07:34:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:13.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:13.555254+0000) 2022-09-10T07:34:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:14.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:14.186022+0000) 2022-09-10T07:34:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:14 smithi110 ceph-mon[97633]: pgmap v5921: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:34:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:14 smithi104 ceph-mon[97044]: pgmap v5921: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:34:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:14.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:14.555413+0000) 2022-09-10T07:34:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:15.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:15.186199+0000) 2022-09-10T07:34:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:15.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:15.555597+0000) 2022-09-10T07:34:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:15 smithi110 ceph-mon[97633]: osdmap e451: 12 total, 12 up, 12 in 2022-09-10T07:34:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:15 smithi110 ceph-mon[97633]: pgmap v5923: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:15 smithi104 ceph-mon[97044]: osdmap e451: 12 total, 12 up, 12 in 2022-09-10T07:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:15 smithi104 ceph-mon[97044]: pgmap v5923: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:34:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:16.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:16.186447+0000) 2022-09-10T07:34:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:16.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:16.555796+0000) 2022-09-10T07:34:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:34:16.639+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:34:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:17.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:17.186662+0000) 2022-09-10T07:34:18.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:17.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:17.556022+0000) 2022-09-10T07:34:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:18.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:18.186898+0000) 2022-09-10T07:34:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:18 smithi110 ceph-mon[97633]: pgmap v5924: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T07:34:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:18 smithi104 ceph-mon[97044]: pgmap v5924: 129 pgs: 129 active+clean; 8.3 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T07:34:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:18.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:18.556220+0000) 2022-09-10T07:34:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:19.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:19.187117+0000) 2022-09-10T07:34:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:19.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:19.556393+0000) 2022-09-10T07:34:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:20.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:20.187404+0000) 2022-09-10T07:34:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:20 smithi110 ceph-mon[97633]: pgmap v5925: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:34:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:20 smithi104 ceph-mon[97044]: pgmap v5925: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:34:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:20.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:20.556576+0000) 2022-09-10T07:34:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:21.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:21.187628+0000) 2022-09-10T07:34:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:21.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:21.556755+0000) 2022-09-10T07:34:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:22.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:22.187864+0000) 2022-09-10T07:34:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:22 smithi110 ceph-mon[97633]: pgmap v5926: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:34:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:22 smithi104 ceph-mon[97044]: pgmap v5926: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:34:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:22.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:22.556974+0000) 2022-09-10T07:34:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:23.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:23.188151+0000) 2022-09-10T07:34:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:23.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:23.557204+0000) 2022-09-10T07:34:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:24.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:24.188343+0000) 2022-09-10T07:34:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:24 smithi110 ceph-mon[97633]: pgmap v5927: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:34:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:24 smithi104 ceph-mon[97044]: pgmap v5927: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:34:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:24.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:24.557388+0000) 2022-09-10T07:34:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:25.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:25.188555+0000) 2022-09-10T07:34:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:25.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:25.557550+0000) 2022-09-10T07:34:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:25 smithi110 ceph-mon[97633]: pgmap v5928: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:34:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:25 smithi104 ceph-mon[97044]: pgmap v5928: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:34:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:26.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:26.188794+0000) 2022-09-10T07:34:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:26.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:26.557687+0000) 2022-09-10T07:34:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:27.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:27.189016+0000) 2022-09-10T07:34:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:27.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:27.557848+0000) 2022-09-10T07:34:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:28.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:28.189202+0000) 2022-09-10T07:34:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:28 smithi110 ceph-mon[97633]: pgmap v5929: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:28 smithi104 ceph-mon[97044]: pgmap v5929: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:28.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:28.558003+0000) 2022-09-10T07:34:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:29.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:29.189408+0000) 2022-09-10T07:34:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:29.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:29.558194+0000) 2022-09-10T07:34:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:30.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:30.189588+0000) 2022-09-10T07:34:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:30 smithi110 ceph-mon[97633]: pgmap v5930: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T07:34:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:30 smithi104 ceph-mon[97044]: pgmap v5930: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T07:34:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:30.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:30.558319+0000) 2022-09-10T07:34:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:31.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:31.189772+0000) 2022-09-10T07:34:32.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:34:31.653+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:34:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:31.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:31.558481+0000) 2022-09-10T07:34:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:32.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:32.190007+0000) 2022-09-10T07:34:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:32 smithi110 ceph-mon[97633]: pgmap v5931: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T07:34:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:32 smithi104 ceph-mon[97044]: pgmap v5931: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T07:34:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:32.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:32.558679+0000) 2022-09-10T07:34:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:33.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:33.190270+0000) 2022-09-10T07:34:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:33.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:33.558857+0000) 2022-09-10T07:34:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:34.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:34.190459+0000) 2022-09-10T07:34:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:34 smithi110 ceph-mon[97633]: pgmap v5932: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:34 smithi104 ceph-mon[97044]: pgmap v5932: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:34.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:34.559023+0000) 2022-09-10T07:34:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:35.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:35.190721+0000) 2022-09-10T07:34:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:35.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:35.559255+0000) 2022-09-10T07:34:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:35 smithi104 ceph-mon[97044]: pgmap v5933: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:34:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:35 smithi110 ceph-mon[97633]: pgmap v5933: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:34:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:36.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:36.190996+0000) 2022-09-10T07:34:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:36.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:36.559432+0000) 2022-09-10T07:34:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:37.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:37.191256+0000) 2022-09-10T07:34:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:37.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:37.559631+0000) 2022-09-10T07:34:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:38.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:38.191462+0000) 2022-09-10T07:34:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:38 smithi110 ceph-mon[97633]: pgmap v5934: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:38 smithi104 ceph-mon[97044]: pgmap v5934: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:38.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:38.559857+0000) 2022-09-10T07:34:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:39.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:39.191705+0000) 2022-09-10T07:34:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:39.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:39.560038+0000) 2022-09-10T07:34:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:40.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:40.191965+0000) 2022-09-10T07:34:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:40 smithi110 ceph-mon[97633]: pgmap v5935: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:34:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:40 smithi104 ceph-mon[97044]: pgmap v5935: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:34:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:40.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:40.560195+0000) 2022-09-10T07:34:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:41.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:41.192153+0000) 2022-09-10T07:34:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:41.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:41.560371+0000) 2022-09-10T07:34:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:42.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:42.192354+0000) 2022-09-10T07:34:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:42 smithi110 ceph-mon[97633]: pgmap v5936: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T07:34:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:42 smithi104 ceph-mon[97044]: pgmap v5936: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T07:34:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:42.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:42.560502+0000) 2022-09-10T07:34:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:43.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:43.192538+0000) 2022-09-10T07:34:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:43.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:43.560632+0000) 2022-09-10T07:34:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:44.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:44.192830+0000) 2022-09-10T07:34:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:44 smithi110 ceph-mon[97633]: pgmap v5937: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:34:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:34:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:34:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:34:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:44 smithi104 ceph-mon[97044]: pgmap v5937: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:34:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:34:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:34:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:34:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:44.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:44.560890+0000) 2022-09-10T07:34:45.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:45.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:45.193028+0000) 2022-09-10T07:34:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:45.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:45.561154+0000) 2022-09-10T07:34:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:45 smithi104 ceph-mon[97044]: pgmap v5938: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:34:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:45 smithi110 ceph-mon[97633]: pgmap v5938: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:34:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:46.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:46.193243+0000) 2022-09-10T07:34:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:46.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:46.561348+0000) 2022-09-10T07:34:47.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:34:46.662+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:34:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:47.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:47.193467+0000) 2022-09-10T07:34:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:47.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:47.561618+0000) 2022-09-10T07:34:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:48.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:48.193711+0000) 2022-09-10T07:34:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:48 smithi110 ceph-mon[97633]: pgmap v5939: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:48 smithi104 ceph-mon[97044]: pgmap v5939: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:48.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:48.561829+0000) 2022-09-10T07:34:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:49.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:49.193938+0000) 2022-09-10T07:34:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:49.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:49.562088+0000) 2022-09-10T07:34:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:50.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:50.194167+0000) 2022-09-10T07:34:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:50 smithi110 ceph-mon[97633]: pgmap v5940: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:34:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:50 smithi104 ceph-mon[97044]: pgmap v5940: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:34:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:50.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:50.562385+0000) 2022-09-10T07:34:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:51.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:51.194386+0000) 2022-09-10T07:34:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:51.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:51.562537+0000) 2022-09-10T07:34:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:52.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:52.194611+0000) 2022-09-10T07:34:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:52 smithi110 ceph-mon[97633]: pgmap v5941: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:34:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:52 smithi104 ceph-mon[97044]: pgmap v5941: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:34:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:52.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:52.562701+0000) 2022-09-10T07:34:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:53.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:53.194819+0000) 2022-09-10T07:34:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:53.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:53.562868+0000) 2022-09-10T07:34:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:54.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:54.195066+0000) 2022-09-10T07:34:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:54 smithi110 ceph-mon[97633]: pgmap v5942: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:54 smithi104 ceph-mon[97044]: pgmap v5942: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:54.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:54.563115+0000) 2022-09-10T07:34:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:55.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:55.195278+0000) 2022-09-10T07:34:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:55.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:55.563325+0000) 2022-09-10T07:34:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:55 smithi104 ceph-mon[97044]: pgmap v5943: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:34:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:55 smithi110 ceph-mon[97633]: pgmap v5943: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:34:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:56.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:56.195530+0000) 2022-09-10T07:34:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:56.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:56.563468+0000) 2022-09-10T07:34:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:57.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:57.195802+0000) 2022-09-10T07:34:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:57.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:57.563697+0000) 2022-09-10T07:34:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:58.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:58.196061+0000) 2022-09-10T07:34:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:34:58 smithi110 ceph-mon[97633]: pgmap v5944: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:34:58 smithi104 ceph-mon[97044]: pgmap v5944: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:34:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:58.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:58.563988+0000) 2022-09-10T07:34:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:34:59.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:59.196348+0000) 2022-09-10T07:35:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:34:59.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:34:59.564275+0000) 2022-09-10T07:35:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:00.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:00.196556+0000) 2022-09-10T07:35:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:00 smithi110 ceph-mon[97633]: pgmap v5945: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:00 smithi104 ceph-mon[97044]: pgmap v5945: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:00.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:00.564424+0000) 2022-09-10T07:35:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:01.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:01.196744+0000) 2022-09-10T07:35:02.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:35:01.678+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:35:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:01.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:01.564592+0000) 2022-09-10T07:35:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:02.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:02.196994+0000) 2022-09-10T07:35:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:02 smithi110 ceph-mon[97633]: pgmap v5946: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:02 smithi104 ceph-mon[97044]: pgmap v5946: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:02.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:02.564775+0000) 2022-09-10T07:35:03.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:03.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:03.197240+0000) 2022-09-10T07:35:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:03.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:03.564951+0000) 2022-09-10T07:35:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:04.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:04.197523+0000) 2022-09-10T07:35:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:04 smithi110 ceph-mon[97633]: pgmap v5947: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:04 smithi104 ceph-mon[97044]: pgmap v5947: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:04.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:04.565151+0000) 2022-09-10T07:35:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:05.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:05.197748+0000) 2022-09-10T07:35:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:05.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:05.565354+0000) 2022-09-10T07:35:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:05 smithi104 ceph-mon[97044]: pgmap v5948: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:35:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:05 smithi110 ceph-mon[97633]: pgmap v5948: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:35:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:06.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:06.197982+0000) 2022-09-10T07:35:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:06.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:06.565557+0000) 2022-09-10T07:35:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:07.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:07.198156+0000) 2022-09-10T07:35:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:07.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:07.565823+0000) 2022-09-10T07:35:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:08.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:08.198303+0000) 2022-09-10T07:35:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:08 smithi110 ceph-mon[97633]: pgmap v5949: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:35:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:08 smithi110 ceph-mon[97633]: osdmap e452: 12 total, 12 up, 12 in 2022-09-10T07:35:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:08 smithi104 ceph-mon[97044]: pgmap v5949: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:35:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:08 smithi104 ceph-mon[97044]: osdmap e452: 12 total, 12 up, 12 in 2022-09-10T07:35:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:08.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:08.566047+0000) 2022-09-10T07:35:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:09.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:09.198545+0000) 2022-09-10T07:35:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:09.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:09.566283+0000) 2022-09-10T07:35:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:10.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:10.198793+0000) 2022-09-10T07:35:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:10 smithi110 ceph-mon[97633]: pgmap v5951: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:35:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:10 smithi104 ceph-mon[97044]: pgmap v5951: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:35:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:10.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:10.566498+0000) 2022-09-10T07:35:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:11.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:11.199049+0000) 2022-09-10T07:35:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:11.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:11.566688+0000) 2022-09-10T07:35:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:12.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:12.199270+0000) 2022-09-10T07:35:12.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:12 smithi110 ceph-mon[97633]: pgmap v5952: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:35:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:12 smithi104 ceph-mon[97044]: pgmap v5952: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:35:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:12.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:12.566890+0000) 2022-09-10T07:35:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:13.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:13.199499+0000) 2022-09-10T07:35:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:13.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:13.567080+0000) 2022-09-10T07:35:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:14.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:14.199699+0000) 2022-09-10T07:35:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:14 smithi110 ceph-mon[97633]: pgmap v5953: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:35:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:14 smithi104 ceph-mon[97044]: pgmap v5953: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:35:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:14.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:14.567268+0000) 2022-09-10T07:35:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:15.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:15.199843+0000) 2022-09-10T07:35:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:15.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:15.567466+0000) 2022-09-10T07:35:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:15 smithi104 ceph-mon[97044]: osdmap e453: 12 total, 12 up, 12 in 2022-09-10T07:35:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:15 smithi104 ceph-mon[97044]: pgmap v5955: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:35:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:15 smithi110 ceph-mon[97633]: osdmap e453: 12 total, 12 up, 12 in 2022-09-10T07:35:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:15 smithi110 ceph-mon[97633]: pgmap v5955: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:35:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:16.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:16.200151+0000) 2022-09-10T07:35:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:16.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:16.567695+0000) 2022-09-10T07:35:17.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:35:16.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:35:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:17.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:17.200315+0000) 2022-09-10T07:35:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:17.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:17.567977+0000) 2022-09-10T07:35:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:18.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:18.200572+0000) 2022-09-10T07:35:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:18 smithi110 ceph-mon[97633]: pgmap v5956: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T07:35:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:18 smithi104 ceph-mon[97044]: pgmap v5956: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T07:35:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:18.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:18.568180+0000) 2022-09-10T07:35:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:19.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:19.200829+0000) 2022-09-10T07:35:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:19.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:19.568431+0000) 2022-09-10T07:35:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:20.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:20.201120+0000) 2022-09-10T07:35:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:20 smithi110 ceph-mon[97633]: pgmap v5957: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:35:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:20 smithi104 ceph-mon[97044]: pgmap v5957: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:35:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:20.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:20.568637+0000) 2022-09-10T07:35:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:21.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:21.201357+0000) 2022-09-10T07:35:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:21.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:21.568870+0000) 2022-09-10T07:35:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:22.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:22.201591+0000) 2022-09-10T07:35:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:22 smithi110 ceph-mon[97633]: pgmap v5958: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:35:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:22 smithi104 ceph-mon[97044]: pgmap v5958: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:35:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:22.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:22.569126+0000) 2022-09-10T07:35:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:23.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:23.201814+0000) 2022-09-10T07:35:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:23.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:23.569369+0000) 2022-09-10T07:35:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:24.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:24.202056+0000) 2022-09-10T07:35:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:24 smithi110 ceph-mon[97633]: pgmap v5959: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:35:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:24 smithi104 ceph-mon[97044]: pgmap v5959: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:35:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:24.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:24.569550+0000) 2022-09-10T07:35:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:25.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:25.202315+0000) 2022-09-10T07:35:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:25.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:25.569793+0000) 2022-09-10T07:35:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:25 smithi104 ceph-mon[97044]: pgmap v5960: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:35:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:25 smithi110 ceph-mon[97633]: pgmap v5960: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:35:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:26.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:26.202552+0000) 2022-09-10T07:35:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:26.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:26.570055+0000) 2022-09-10T07:35:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:27.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:27.202800+0000) 2022-09-10T07:35:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:27.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:27.570314+0000) 2022-09-10T07:35:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:28.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:28.203066+0000) 2022-09-10T07:35:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:28 smithi110 ceph-mon[97633]: pgmap v5961: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:28 smithi104 ceph-mon[97044]: pgmap v5961: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:28.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:28.570474+0000) 2022-09-10T07:35:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:29.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:29.203375+0000) 2022-09-10T07:35:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:29.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:29.570742+0000) 2022-09-10T07:35:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:30.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:30.203655+0000) 2022-09-10T07:35:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:30 smithi110 ceph-mon[97633]: pgmap v5962: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:30 smithi104 ceph-mon[97044]: pgmap v5962: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:30.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:30.570975+0000) 2022-09-10T07:35:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:31.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:31.203922+0000) 2022-09-10T07:35:32.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:35:31.699+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:35:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:31.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:31.571210+0000) 2022-09-10T07:35:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:32.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:32.204130+0000) 2022-09-10T07:35:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:32 smithi110 ceph-mon[97633]: pgmap v5963: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:32 smithi104 ceph-mon[97044]: pgmap v5963: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:32.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:32.571451+0000) 2022-09-10T07:35:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:33.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:33.204385+0000) 2022-09-10T07:35:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:33.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:33.571629+0000) 2022-09-10T07:35:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:34.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:34.204670+0000) 2022-09-10T07:35:34.681 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:34 smithi110 ceph-mon[97633]: pgmap v5964: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:34 smithi104 ceph-mon[97044]: pgmap v5964: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:34.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:34.571865+0000) 2022-09-10T07:35:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:35.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:35.204866+0000) 2022-09-10T07:35:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:35.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:35.572114+0000) 2022-09-10T07:35:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:35 smithi104 ceph-mon[97044]: pgmap v5965: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:35:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:35 smithi110 ceph-mon[97633]: pgmap v5965: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:35:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:36.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:36.205086+0000) 2022-09-10T07:35:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:36.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:36.572319+0000) 2022-09-10T07:35:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:37.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:37.205368+0000) 2022-09-10T07:35:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:37.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:37.572505+0000) 2022-09-10T07:35:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:38.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:38.205570+0000) 2022-09-10T07:35:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:38 smithi110 ceph-mon[97633]: pgmap v5966: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:38 smithi104 ceph-mon[97044]: pgmap v5966: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:38.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:38.572658+0000) 2022-09-10T07:35:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:39.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:39.205784+0000) 2022-09-10T07:35:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:39.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:39.572878+0000) 2022-09-10T07:35:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:40.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:40.206032+0000) 2022-09-10T07:35:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:40 smithi110 ceph-mon[97633]: pgmap v5967: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:40 smithi104 ceph-mon[97044]: pgmap v5967: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:40.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:40.573125+0000) 2022-09-10T07:35:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:41.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:41.206186+0000) 2022-09-10T07:35:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:41.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:41.573323+0000) 2022-09-10T07:35:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:42.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:42.206409+0000) 2022-09-10T07:35:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:42 smithi110 ceph-mon[97633]: pgmap v5968: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:42 smithi104 ceph-mon[97044]: pgmap v5968: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:42.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:42.573526+0000) 2022-09-10T07:35:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:43.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:43.206566+0000) 2022-09-10T07:35:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:43.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:43.573701+0000) 2022-09-10T07:35:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:44.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:44.206779+0000) 2022-09-10T07:35:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:44 smithi110 ceph-mon[97633]: pgmap v5969: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:35:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:35:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:35:44.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:35:44.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:44 smithi104 ceph-mon[97044]: pgmap v5969: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:44.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:35:44.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:35:44.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:35:44.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:35:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:44.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:44.573976+0000) 2022-09-10T07:35:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:45.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:45.206984+0000) 2022-09-10T07:35:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:45.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:45.574219+0000) 2022-09-10T07:35:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:45 smithi104 ceph-mon[97044]: pgmap v5970: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:35:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:45 smithi110 ceph-mon[97633]: pgmap v5970: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:35:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:46.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:46.207220+0000) 2022-09-10T07:35:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:46.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:46.574473+0000) 2022-09-10T07:35:47.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:35:46.705+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:35:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:47.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:47.207487+0000) 2022-09-10T07:35:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:47.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:47.574707+0000) 2022-09-10T07:35:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:48.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:48.207767+0000) 2022-09-10T07:35:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:48 smithi110 ceph-mon[97633]: pgmap v5971: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:48 smithi104 ceph-mon[97044]: pgmap v5971: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:48.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:48.574933+0000) 2022-09-10T07:35:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:49.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:49.207988+0000) 2022-09-10T07:35:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:49.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:49.575195+0000) 2022-09-10T07:35:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:50.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:50.208246+0000) 2022-09-10T07:35:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:50 smithi110 ceph-mon[97633]: pgmap v5972: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:50 smithi104 ceph-mon[97044]: pgmap v5972: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:35:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:50.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:50.575467+0000) 2022-09-10T07:35:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:51.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:51.208455+0000) 2022-09-10T07:35:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:51.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:51.575738+0000) 2022-09-10T07:35:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:52.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:52.208642+0000) 2022-09-10T07:35:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:52 smithi110 ceph-mon[97633]: pgmap v5973: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:52 smithi104 ceph-mon[97044]: pgmap v5973: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:35:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:52.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:52.575968+0000) 2022-09-10T07:35:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:53.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:53.208885+0000) 2022-09-10T07:35:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:53.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:53.576220+0000) 2022-09-10T07:35:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:54.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:54.209158+0000) 2022-09-10T07:35:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:54 smithi110 ceph-mon[97633]: pgmap v5974: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:54 smithi104 ceph-mon[97044]: pgmap v5974: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:35:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:54.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:54.576389+0000) 2022-09-10T07:35:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:55.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:55.209380+0000) 2022-09-10T07:35:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:55.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:55.576581+0000) 2022-09-10T07:35:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:55 smithi104 ceph-mon[97044]: pgmap v5975: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:35:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:55 smithi110 ceph-mon[97633]: pgmap v5975: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:35:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:56.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:56.209601+0000) 2022-09-10T07:35:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:56.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:56.576762+0000) 2022-09-10T07:35:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:57.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:57.209845+0000) 2022-09-10T07:35:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:57.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:57.577034+0000) 2022-09-10T07:35:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:58.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:58.210052+0000) 2022-09-10T07:35:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:35:58 smithi110 ceph-mon[97633]: pgmap v5976: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:35:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:35:58 smithi104 ceph-mon[97044]: pgmap v5976: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:35:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:58.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:58.577256+0000) 2022-09-10T07:35:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:35:59.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:59.210184+0000) 2022-09-10T07:36:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:35:59.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:35:59.577383+0000) 2022-09-10T07:36:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:00.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:00.210375+0000) 2022-09-10T07:36:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:00 smithi110 ceph-mon[97633]: pgmap v5977: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2022-09-10T07:36:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:00 smithi104 ceph-mon[97044]: pgmap v5977: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2022-09-10T07:36:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:00.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:00.577546+0000) 2022-09-10T07:36:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:01.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:01.210597+0000) 2022-09-10T07:36:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:01.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:01.577768+0000) 2022-09-10T07:36:02.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:36:01.722+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:36:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:02.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:02.210820+0000) 2022-09-10T07:36:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:02 smithi110 ceph-mon[97633]: pgmap v5978: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T07:36:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:02 smithi104 ceph-mon[97044]: pgmap v5978: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T07:36:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:02.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:02.577934+0000) 2022-09-10T07:36:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:03.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:03.211051+0000) 2022-09-10T07:36:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:03.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:03.578166+0000) 2022-09-10T07:36:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:04.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:04.211334+0000) 2022-09-10T07:36:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:04 smithi110 ceph-mon[97633]: pgmap v5979: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:36:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:04 smithi104 ceph-mon[97044]: pgmap v5979: 129 pgs: 129 active+clean; 8.3 MiB data, 686 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:36:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:04.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:04.578365+0000) 2022-09-10T07:36:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:05.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:05.211591+0000) 2022-09-10T07:36:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:05.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:05.578573+0000) 2022-09-10T07:36:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:05 smithi104 ceph-mon[97044]: pgmap v5980: 129 pgs: 129 active+clean; 8.3 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:36:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:05 smithi110 ceph-mon[97633]: pgmap v5980: 129 pgs: 129 active+clean; 8.3 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:36:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:06.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:06.211816+0000) 2022-09-10T07:36:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:06.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:06.578828+0000) 2022-09-10T07:36:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:07.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:07.212051+0000) 2022-09-10T07:36:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:07.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:07.579009+0000) 2022-09-10T07:36:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:08.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:08.212238+0000) 2022-09-10T07:36:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:08 smithi110 ceph-mon[97633]: osdmap e454: 12 total, 12 up, 12 in 2022-09-10T07:36:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:08 smithi110 ceph-mon[97633]: pgmap v5982: 129 pgs: 129 active+clean; 8.3 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:08 smithi104 ceph-mon[97044]: osdmap e454: 12 total, 12 up, 12 in 2022-09-10T07:36:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:08 smithi104 ceph-mon[97044]: pgmap v5982: 129 pgs: 129 active+clean; 8.3 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:08.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:08.579163+0000) 2022-09-10T07:36:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:09.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:09.212427+0000) 2022-09-10T07:36:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:09.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:09.579354+0000) 2022-09-10T07:36:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:10.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:10.212668+0000) 2022-09-10T07:36:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:10 smithi110 ceph-mon[97633]: pgmap v5983: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:10 smithi104 ceph-mon[97044]: pgmap v5983: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:10.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:10.579502+0000) 2022-09-10T07:36:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:11.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:11.212886+0000) 2022-09-10T07:36:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:11.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:11.579641+0000) 2022-09-10T07:36:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:12.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:12.213142+0000) 2022-09-10T07:36:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:12 smithi110 ceph-mon[97633]: pgmap v5984: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:12 smithi104 ceph-mon[97044]: pgmap v5984: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:12.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:12.579784+0000) 2022-09-10T07:36:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:13.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:13.213288+0000) 2022-09-10T07:36:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:13.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:13.579964+0000) 2022-09-10T07:36:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:14.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:14.213515+0000) 2022-09-10T07:36:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:14 smithi110 ceph-mon[97633]: pgmap v5985: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:14 smithi104 ceph-mon[97044]: pgmap v5985: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:36:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:14.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:14.580200+0000) 2022-09-10T07:36:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:15.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:15.213714+0000) 2022-09-10T07:36:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:15.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:15.580399+0000) 2022-09-10T07:36:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:15 smithi104 ceph-mon[97044]: osdmap e455: 12 total, 12 up, 12 in 2022-09-10T07:36:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:15 smithi104 ceph-mon[97044]: pgmap v5987: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:36:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:15 smithi110 ceph-mon[97633]: osdmap e455: 12 total, 12 up, 12 in 2022-09-10T07:36:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:15 smithi110 ceph-mon[97633]: pgmap v5987: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:36:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:16.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:16.213965+0000) 2022-09-10T07:36:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:36:16.734+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:36:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:16.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:16.580600+0000) 2022-09-10T07:36:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:17.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:17.214184+0000) 2022-09-10T07:36:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:17.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:17.580773+0000) 2022-09-10T07:36:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:18.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:18.214412+0000) 2022-09-10T07:36:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:18 smithi110 ceph-mon[97633]: pgmap v5988: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:36:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:18 smithi104 ceph-mon[97044]: pgmap v5988: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:36:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:18.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:18.580914+0000) 2022-09-10T07:36:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:19.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:19.214642+0000) 2022-09-10T07:36:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:19.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:19.581159+0000) 2022-09-10T07:36:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:20.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:20.214949+0000) 2022-09-10T07:36:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:20 smithi110 ceph-mon[97633]: pgmap v5989: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:36:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:20 smithi104 ceph-mon[97044]: pgmap v5989: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:36:21.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:20.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:20.581417+0000) 2022-09-10T07:36:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:21.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:21.215143+0000) 2022-09-10T07:36:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:21.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:21.581622+0000) 2022-09-10T07:36:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:22.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:22.215342+0000) 2022-09-10T07:36:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:22 smithi110 ceph-mon[97633]: pgmap v5990: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:36:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:22 smithi104 ceph-mon[97044]: pgmap v5990: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:36:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:22.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:22.581800+0000) 2022-09-10T07:36:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:23.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:23.215549+0000) 2022-09-10T07:36:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:23.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:23.581982+0000) 2022-09-10T07:36:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:24.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:24.215777+0000) 2022-09-10T07:36:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:24 smithi110 ceph-mon[97633]: pgmap v5991: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:36:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:24 smithi104 ceph-mon[97044]: pgmap v5991: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:36:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:24.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:24.582234+0000) 2022-09-10T07:36:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:25.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:25.215992+0000) 2022-09-10T07:36:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:25.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:25.582424+0000) 2022-09-10T07:36:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:25 smithi104 ceph-mon[97044]: pgmap v5992: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:36:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:25 smithi110 ceph-mon[97633]: pgmap v5992: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:36:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:26.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:26.216202+0000) 2022-09-10T07:36:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:26.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:26.582714+0000) 2022-09-10T07:36:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:27.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:27.216409+0000) 2022-09-10T07:36:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:27.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:27.582957+0000) 2022-09-10T07:36:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:28.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:28.216580+0000) 2022-09-10T07:36:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:28 smithi110 ceph-mon[97633]: pgmap v5993: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:28 smithi104 ceph-mon[97044]: pgmap v5993: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:28.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:28.583174+0000) 2022-09-10T07:36:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:29.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:29.216706+0000) 2022-09-10T07:36:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:29.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:29.583380+0000) 2022-09-10T07:36:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:30.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:30.216884+0000) 2022-09-10T07:36:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:30 smithi110 ceph-mon[97633]: pgmap v5994: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:30 smithi104 ceph-mon[97044]: pgmap v5994: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:30.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:30.583604+0000) 2022-09-10T07:36:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:31.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:31.217069+0000) 2022-09-10T07:36:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:31.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:31.583806+0000) 2022-09-10T07:36:32.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:36:31.750+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:36:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:32.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:32.217322+0000) 2022-09-10T07:36:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:32 smithi110 ceph-mon[97633]: pgmap v5995: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:36:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:32 smithi104 ceph-mon[97044]: pgmap v5995: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:36:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:32.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:32.584278+0000) 2022-09-10T07:36:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:33.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:33.217513+0000) 2022-09-10T07:36:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:33 smithi104 ceph-mon[97044]: pgmap v5996: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:33 smithi110 ceph-mon[97633]: pgmap v5996: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:33.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:33.584515+0000) 2022-09-10T07:36:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:34.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:34.217752+0000) 2022-09-10T07:36:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:34.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:34.584774+0000) 2022-09-10T07:36:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:35.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:35.218022+0000) 2022-09-10T07:36:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:35.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:35.585023+0000) 2022-09-10T07:36:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:35 smithi104 ceph-mon[97044]: pgmap v5997: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:36:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:35 smithi110 ceph-mon[97633]: pgmap v5997: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:36:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:36.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:36.218230+0000) 2022-09-10T07:36:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:36.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:36.585205+0000) 2022-09-10T07:36:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:37.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:37.218461+0000) 2022-09-10T07:36:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:37.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:37.585395+0000) 2022-09-10T07:36:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:38.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:38.218658+0000) 2022-09-10T07:36:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:38 smithi110 ceph-mon[97633]: pgmap v5998: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:38 smithi104 ceph-mon[97044]: pgmap v5998: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:38.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:38.585518+0000) 2022-09-10T07:36:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:39.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:39.218863+0000) 2022-09-10T07:36:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:39.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:39.585743+0000) 2022-09-10T07:36:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:40.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:40.219092+0000) 2022-09-10T07:36:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:40 smithi110 ceph-mon[97633]: pgmap v5999: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:40 smithi104 ceph-mon[97044]: pgmap v5999: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:40.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:40.585921+0000) 2022-09-10T07:36:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:41.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:41.219389+0000) 2022-09-10T07:36:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:41 smithi104 ceph-mon[97044]: pgmap v6000: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:36:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:41 smithi110 ceph-mon[97633]: pgmap v6000: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:36:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:41.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:41.586173+0000) 2022-09-10T07:36:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:42.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:42.219596+0000) 2022-09-10T07:36:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:42.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:42.586349+0000) 2022-09-10T07:36:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:43.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:43.219743+0000) 2022-09-10T07:36:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:43.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:43.586625+0000) 2022-09-10T07:36:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:44.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:44.219928+0000) 2022-09-10T07:36:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:44 smithi110 ceph-mon[97633]: pgmap v6001: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:36:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:36:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:36:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:36:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:44 smithi104 ceph-mon[97044]: pgmap v6001: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:36:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:36:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:36:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:36:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:44.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:44.586902+0000) 2022-09-10T07:36:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:45.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:45.220188+0000) 2022-09-10T07:36:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:45.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:45.587191+0000) 2022-09-10T07:36:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:45 smithi104 ceph-mon[97044]: pgmap v6002: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:36:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:45 smithi110 ceph-mon[97633]: pgmap v6002: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:36:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:46.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:46.220400+0000) 2022-09-10T07:36:47.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:36:46.763+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:36:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:46.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:46.587406+0000) 2022-09-10T07:36:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:47.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:47.220592+0000) 2022-09-10T07:36:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:47.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:47.587587+0000) 2022-09-10T07:36:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:48.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:48.220826+0000) 2022-09-10T07:36:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:48 smithi110 ceph-mon[97633]: pgmap v6003: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:48 smithi104 ceph-mon[97044]: pgmap v6003: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:48.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:48.587756+0000) 2022-09-10T07:36:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:49.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:49.221077+0000) 2022-09-10T07:36:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:49.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:49.587970+0000) 2022-09-10T07:36:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:50.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:50.221347+0000) 2022-09-10T07:36:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:50 smithi110 ceph-mon[97633]: pgmap v6004: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:50 smithi104 ceph-mon[97044]: pgmap v6004: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:50.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:50.588183+0000) 2022-09-10T07:36:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:51.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:51.221550+0000) 2022-09-10T07:36:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:51 smithi104 ceph-mon[97044]: pgmap v6005: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:36:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:51 smithi110 ceph-mon[97633]: pgmap v6005: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:36:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:51.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:51.588406+0000) 2022-09-10T07:36:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:52.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:52.221764+0000) 2022-09-10T07:36:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:52.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:52.588654+0000) 2022-09-10T07:36:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:53.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:53.221948+0000) 2022-09-10T07:36:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:53.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:53.588864+0000) 2022-09-10T07:36:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:54.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:54.222188+0000) 2022-09-10T07:36:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:54 smithi110 ceph-mon[97633]: pgmap v6006: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:54 smithi104 ceph-mon[97044]: pgmap v6006: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:54.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:54.589162+0000) 2022-09-10T07:36:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:55.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:55.222421+0000) 2022-09-10T07:36:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:55.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:55.589401+0000) 2022-09-10T07:36:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:55 smithi104 ceph-mon[97044]: pgmap v6007: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:36:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:55 smithi110 ceph-mon[97633]: pgmap v6007: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:36:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:56.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:56.222626+0000) 2022-09-10T07:36:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:56.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:56.589586+0000) 2022-09-10T07:36:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:57.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:57.222838+0000) 2022-09-10T07:36:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:57.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:57.589789+0000) 2022-09-10T07:36:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:58.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:58.223073+0000) 2022-09-10T07:36:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:58 smithi104 ceph-mon[97044]: pgmap v6008: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:58 smithi110 ceph-mon[97633]: pgmap v6008: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:36:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:58.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:58.590014+0000) 2022-09-10T07:36:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:36:59.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:59.223355+0000) 2022-09-10T07:36:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:36:59 smithi104 ceph-mon[97044]: pgmap v6009: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:36:59.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:36:59 smithi110 ceph-mon[97633]: pgmap v6009: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:36:59.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:36:59.590266+0000) 2022-09-10T07:37:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:00.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:00.223616+0000) 2022-09-10T07:37:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:00.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:00.590521+0000) 2022-09-10T07:37:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:01.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:01.223829+0000) 2022-09-10T07:37:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:01.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:01.590786+0000) 2022-09-10T07:37:02.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:37:01.767+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:37:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:02.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:02.224082+0000) 2022-09-10T07:37:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:02 smithi104 ceph-mon[97044]: pgmap v6010: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:02 smithi110 ceph-mon[97633]: pgmap v6010: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:02.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:02.590986+0000) 2022-09-10T07:37:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:03.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:03.224335+0000) 2022-09-10T07:37:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:03 smithi104 ceph-mon[97044]: pgmap v6011: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:03 smithi110 ceph-mon[97633]: pgmap v6011: 129 pgs: 129 active+clean; 8.3 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:03.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:03.591178+0000) 2022-09-10T07:37:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:04.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:04.224557+0000) 2022-09-10T07:37:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:04.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:04.591325+0000) 2022-09-10T07:37:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:05.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:05.224732+0000) 2022-09-10T07:37:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:05.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:05.591581+0000) 2022-09-10T07:37:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:05 smithi104 ceph-mon[97044]: pgmap v6012: 129 pgs: 129 active+clean; 8.3 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:37:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:05 smithi110 ceph-mon[97633]: pgmap v6012: 129 pgs: 129 active+clean; 8.3 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:37:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:06.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:06.224956+0000) 2022-09-10T07:37:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:06.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:06.591780+0000) 2022-09-10T07:37:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:07.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:07.225202+0000) 2022-09-10T07:37:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:07.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:07.592025+0000) 2022-09-10T07:37:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:08.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:08.225432+0000) 2022-09-10T07:37:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:08 smithi110 ceph-mon[97633]: osdmap e456: 12 total, 12 up, 12 in 2022-09-10T07:37:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:08 smithi110 ceph-mon[97633]: pgmap v6014: 129 pgs: 129 active+clean; 8.3 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:08 smithi104 ceph-mon[97044]: osdmap e456: 12 total, 12 up, 12 in 2022-09-10T07:37:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:08 smithi104 ceph-mon[97044]: pgmap v6014: 129 pgs: 129 active+clean; 8.3 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:08.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:08.592258+0000) 2022-09-10T07:37:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:09.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:09.225678+0000) 2022-09-10T07:37:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:09 smithi104 ceph-mon[97044]: pgmap v6015: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:09 smithi110 ceph-mon[97633]: pgmap v6015: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:09.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:09.592515+0000) 2022-09-10T07:37:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:10.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:10.225934+0000) 2022-09-10T07:37:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:10.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:10.592769+0000) 2022-09-10T07:37:11.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:11.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:11.226205+0000) 2022-09-10T07:37:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:11 smithi104 ceph-mon[97044]: pgmap v6016: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:11 smithi110 ceph-mon[97633]: pgmap v6016: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:11.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:11.593055+0000) 2022-09-10T07:37:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:12.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:12.226386+0000) 2022-09-10T07:37:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:12.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:12.593285+0000) 2022-09-10T07:37:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:13.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:13.226620+0000) 2022-09-10T07:37:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:13 smithi104 ceph-mon[97044]: pgmap v6017: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:13 smithi110 ceph-mon[97633]: pgmap v6017: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:37:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:13.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:13.593469+0000) 2022-09-10T07:37:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:14.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:14.226835+0000) 2022-09-10T07:37:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:14.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:14.593705+0000) 2022-09-10T07:37:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:15.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:15.227088+0000) 2022-09-10T07:37:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:15.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:15.593940+0000) 2022-09-10T07:37:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:15 smithi104 ceph-mon[97044]: osdmap e457: 12 total, 12 up, 12 in 2022-09-10T07:37:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:15 smithi104 ceph-mon[97044]: pgmap v6019: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:37:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:15 smithi110 ceph-mon[97633]: osdmap e457: 12 total, 12 up, 12 in 2022-09-10T07:37:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:15 smithi110 ceph-mon[97633]: pgmap v6019: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:37:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:16.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:16.227339+0000) 2022-09-10T07:37:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:16.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:16.594161+0000) 2022-09-10T07:37:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:37:16.781+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:37:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:17.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:17.227595+0000) 2022-09-10T07:37:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:17 smithi104 ceph-mon[97044]: pgmap v6020: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:37:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:17 smithi110 ceph-mon[97633]: pgmap v6020: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:37:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:17.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:17.594337+0000) 2022-09-10T07:37:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:18.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:18.227818+0000) 2022-09-10T07:37:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:18.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:18.594543+0000) 2022-09-10T07:37:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:19.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:19.228067+0000) 2022-09-10T07:37:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:19 smithi104 ceph-mon[97044]: pgmap v6021: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:37:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:19 smithi110 ceph-mon[97633]: pgmap v6021: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:37:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:19.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:19.594739+0000) 2022-09-10T07:37:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:20.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:20.228299+0000) 2022-09-10T07:37:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:20.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:20.594870+0000) 2022-09-10T07:37:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:21.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:21.228548+0000) 2022-09-10T07:37:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:21 smithi104 ceph-mon[97044]: pgmap v6022: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:37:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:21 smithi110 ceph-mon[97633]: pgmap v6022: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:37:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:21.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:21.595074+0000) 2022-09-10T07:37:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:22.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:22.228771+0000) 2022-09-10T07:37:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:22.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:22.595326+0000) 2022-09-10T07:37:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:23.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:23.229021+0000) 2022-09-10T07:37:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:23 smithi104 ceph-mon[97044]: pgmap v6023: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:37:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:23 smithi110 ceph-mon[97633]: pgmap v6023: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:37:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:23.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:23.595520+0000) 2022-09-10T07:37:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:24.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:24.229224+0000) 2022-09-10T07:37:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:24.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:24.595793+0000) 2022-09-10T07:37:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:25.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:25.229410+0000) 2022-09-10T07:37:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:25.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:25.596061+0000) 2022-09-10T07:37:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:25 smithi104 ceph-mon[97044]: pgmap v6024: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:37:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:25 smithi110 ceph-mon[97633]: pgmap v6024: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:37:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:26.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:26.229650+0000) 2022-09-10T07:37:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:26.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:26.596291+0000) 2022-09-10T07:37:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:27.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:27.229832+0000) 2022-09-10T07:37:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:27 smithi104 ceph-mon[97044]: pgmap v6025: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:27.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:27 smithi110 ceph-mon[97633]: pgmap v6025: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:27.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:27.596462+0000) 2022-09-10T07:37:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:28.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:28.230015+0000) 2022-09-10T07:37:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:28.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:28.596656+0000) 2022-09-10T07:37:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:29.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:29.230243+0000) 2022-09-10T07:37:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:29 smithi104 ceph-mon[97044]: pgmap v6026: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:29 smithi110 ceph-mon[97633]: pgmap v6026: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:29.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:29.596924+0000) 2022-09-10T07:37:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:30.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:30.230485+0000) 2022-09-10T07:37:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:30.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:30.597134+0000) 2022-09-10T07:37:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:31.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:31.230720+0000) 2022-09-10T07:37:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:31 smithi104 ceph-mon[97044]: pgmap v6027: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:31 smithi110 ceph-mon[97633]: pgmap v6027: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:31.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:31.597335+0000) 2022-09-10T07:37:32.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:37:31.789+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:37:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:32.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:32.230969+0000) 2022-09-10T07:37:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:32.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:32.597580+0000) 2022-09-10T07:37:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:33.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:33.231218+0000) 2022-09-10T07:37:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:33 smithi104 ceph-mon[97044]: pgmap v6028: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:33 smithi110 ceph-mon[97633]: pgmap v6028: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:33.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:33.597818+0000) 2022-09-10T07:37:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:34.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:34.231449+0000) 2022-09-10T07:37:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:34.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:34.598020+0000) 2022-09-10T07:37:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:35.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:35.231701+0000) 2022-09-10T07:37:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:35.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:35.598245+0000) 2022-09-10T07:37:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:35 smithi104 ceph-mon[97044]: pgmap v6029: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:37:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:35 smithi110 ceph-mon[97633]: pgmap v6029: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:37:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:36.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:36.231942+0000) 2022-09-10T07:37:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:36.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:36.598407+0000) 2022-09-10T07:37:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:37.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:37.232201+0000) 2022-09-10T07:37:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:37 smithi104 ceph-mon[97044]: pgmap v6030: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:37 smithi110 ceph-mon[97633]: pgmap v6030: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:37.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:37.598657+0000) 2022-09-10T07:37:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:38.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:38.232432+0000) 2022-09-10T07:37:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:38.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:38.598825+0000) 2022-09-10T07:37:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:39.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:39.232650+0000) 2022-09-10T07:37:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:39 smithi104 ceph-mon[97044]: pgmap v6031: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:39 smithi110 ceph-mon[97633]: pgmap v6031: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:39.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:39.599068+0000) 2022-09-10T07:37:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:40.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:40.232902+0000) 2022-09-10T07:37:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:40.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:40.599316+0000) 2022-09-10T07:37:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:41.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:41.233159+0000) 2022-09-10T07:37:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:41 smithi104 ceph-mon[97044]: pgmap v6032: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:41 smithi110 ceph-mon[97633]: pgmap v6032: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:41.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:41.599563+0000) 2022-09-10T07:37:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:42.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:42.233417+0000) 2022-09-10T07:37:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:42.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:42.599762+0000) 2022-09-10T07:37:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:43.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:43.233601+0000) 2022-09-10T07:37:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:43 smithi104 ceph-mon[97044]: pgmap v6033: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:43 smithi110 ceph-mon[97633]: pgmap v6033: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:43.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:43.599989+0000) 2022-09-10T07:37:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:44.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:44.233841+0000) 2022-09-10T07:37:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:37:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:37:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:37:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:37:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:37:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:37:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:37:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:37:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:44.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:44.600197+0000) 2022-09-10T07:37:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:45.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:45.234147+0000) 2022-09-10T07:37:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:45.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:45.600392+0000) 2022-09-10T07:37:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:45 smithi104 ceph-mon[97044]: pgmap v6034: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:37:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:45 smithi110 ceph-mon[97633]: pgmap v6034: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:37:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:46.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:46.234400+0000) 2022-09-10T07:37:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:46.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:46.600527+0000) 2022-09-10T07:37:47.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:37:46.802+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:37:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:47.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:47.234628+0000) 2022-09-10T07:37:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:47 smithi104 ceph-mon[97044]: pgmap v6035: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:47 smithi110 ceph-mon[97633]: pgmap v6035: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:47.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:47.600687+0000) 2022-09-10T07:37:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:48.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:48.234798+0000) 2022-09-10T07:37:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:48.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:48.600883+0000) 2022-09-10T07:37:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:49.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:49.235032+0000) 2022-09-10T07:37:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:49 smithi104 ceph-mon[97044]: pgmap v6036: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:49 smithi110 ceph-mon[97633]: pgmap v6036: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:49.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:49.601067+0000) 2022-09-10T07:37:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:50.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:50.235263+0000) 2022-09-10T07:37:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:50.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:50.601246+0000) 2022-09-10T07:37:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:51.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:51.235493+0000) 2022-09-10T07:37:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:51 smithi104 ceph-mon[97044]: pgmap v6037: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:51 smithi110 ceph-mon[97633]: pgmap v6037: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:37:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:51.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:51.601408+0000) 2022-09-10T07:37:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:52.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:52.235704+0000) 2022-09-10T07:37:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:52.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:52.601561+0000) 2022-09-10T07:37:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:53.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:53.235923+0000) 2022-09-10T07:37:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:53 smithi104 ceph-mon[97044]: pgmap v6038: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:53 smithi110 ceph-mon[97633]: pgmap v6038: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:53.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:53.601772+0000) 2022-09-10T07:37:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:54.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:54.236184+0000) 2022-09-10T07:37:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:54.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:54.601943+0000) 2022-09-10T07:37:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:55.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:55.236378+0000) 2022-09-10T07:37:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:55.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:55.602244+0000) 2022-09-10T07:37:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:55 smithi104 ceph-mon[97044]: pgmap v6039: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:37:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:55 smithi110 ceph-mon[97633]: pgmap v6039: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:37:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:56.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:56.236574+0000) 2022-09-10T07:37:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:56.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:56.602465+0000) 2022-09-10T07:37:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:57.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:57.236783+0000) 2022-09-10T07:37:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:57 smithi104 ceph-mon[97044]: pgmap v6040: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:57 smithi110 ceph-mon[97633]: pgmap v6040: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:37:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:57.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:57.602642+0000) 2022-09-10T07:37:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:58.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:58.236999+0000) 2022-09-10T07:37:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:58.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:58.602833+0000) 2022-09-10T07:37:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:37:59.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:59.237224+0000) 2022-09-10T07:37:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:37:59 smithi104 ceph-mon[97044]: pgmap v6041: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:37:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:37:59 smithi110 ceph-mon[97633]: pgmap v6041: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:37:59.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:37:59.603125+0000) 2022-09-10T07:38:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:00.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:00.237418+0000) 2022-09-10T07:38:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:00.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:00.603375+0000) 2022-09-10T07:38:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:01.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:01.237663+0000) 2022-09-10T07:38:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:01 smithi104 ceph-mon[97044]: pgmap v6042: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:01 smithi110 ceph-mon[97633]: pgmap v6042: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:02.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:38:01.814+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:38:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:01.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:01.603626+0000) 2022-09-10T07:38:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:02.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:02.237854+0000) 2022-09-10T07:38:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:02.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:02.603886+0000) 2022-09-10T07:38:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:03.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:03.238088+0000) 2022-09-10T07:38:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:03 smithi104 ceph-mon[97044]: pgmap v6043: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:03 smithi110 ceph-mon[97633]: pgmap v6043: 129 pgs: 129 active+clean; 8.3 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:03.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:03.604020+0000) 2022-09-10T07:38:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:04.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:04.238354+0000) 2022-09-10T07:38:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:04.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:04.604195+0000) 2022-09-10T07:38:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:05.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:05.238569+0000) 2022-09-10T07:38:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:05.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:05.604415+0000) 2022-09-10T07:38:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:05 smithi104 ceph-mon[97044]: pgmap v6044: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:38:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:05 smithi110 ceph-mon[97633]: pgmap v6044: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:38:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:06.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:06.238759+0000) 2022-09-10T07:38:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:06.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:06.604641+0000) 2022-09-10T07:38:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:07.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:07.239000+0000) 2022-09-10T07:38:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:07.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:07.604904+0000) 2022-09-10T07:38:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:08.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:08.239236+0000) 2022-09-10T07:38:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:08 smithi110 ceph-mon[97633]: osdmap e458: 12 total, 12 up, 12 in 2022-09-10T07:38:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:08 smithi110 ceph-mon[97633]: pgmap v6046: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:08 smithi104 ceph-mon[97044]: osdmap e458: 12 total, 12 up, 12 in 2022-09-10T07:38:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:08 smithi104 ceph-mon[97044]: pgmap v6046: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:08.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:08.605173+0000) 2022-09-10T07:38:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:09.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:09.239427+0000) 2022-09-10T07:38:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:09 smithi104 ceph-mon[97044]: pgmap v6047: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:09 smithi110 ceph-mon[97633]: pgmap v6047: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:09.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:09.605313+0000) 2022-09-10T07:38:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:10.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:10.239688+0000) 2022-09-10T07:38:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:10.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:10.605566+0000) 2022-09-10T07:38:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:11.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:11.239885+0000) 2022-09-10T07:38:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:11 smithi104 ceph-mon[97044]: pgmap v6048: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:11 smithi110 ceph-mon[97633]: pgmap v6048: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:11.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:11.605802+0000) 2022-09-10T07:38:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:12.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:12.240154+0000) 2022-09-10T07:38:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:12.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:12.606048+0000) 2022-09-10T07:38:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:13.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:13.240351+0000) 2022-09-10T07:38:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:13 smithi104 ceph-mon[97044]: pgmap v6049: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:13 smithi110 ceph-mon[97633]: pgmap v6049: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:38:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:13.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:13.606276+0000) 2022-09-10T07:38:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:14.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:14.240524+0000) 2022-09-10T07:38:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:14.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:14.606562+0000) 2022-09-10T07:38:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:15.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:15.240688+0000) 2022-09-10T07:38:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:15.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:15.606805+0000) 2022-09-10T07:38:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:15 smithi104 ceph-mon[97044]: osdmap e459: 12 total, 12 up, 12 in 2022-09-10T07:38:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:15 smithi104 ceph-mon[97044]: pgmap v6051: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T07:38:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:15 smithi110 ceph-mon[97633]: osdmap e459: 12 total, 12 up, 12 in 2022-09-10T07:38:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:15 smithi110 ceph-mon[97633]: pgmap v6051: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T07:38:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:16.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:16.240915+0000) 2022-09-10T07:38:17.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:38:16.827+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:38:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:16.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:16.607008+0000) 2022-09-10T07:38:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:17.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:17.241139+0000) 2022-09-10T07:38:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:17 smithi104 ceph-mon[97044]: pgmap v6052: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:38:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:17 smithi110 ceph-mon[97633]: pgmap v6052: 129 pgs: 129 active+clean; 8.3 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:38:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:17.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:17.607187+0000) 2022-09-10T07:38:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:18.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:18.241367+0000) 2022-09-10T07:38:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:18.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:18.607403+0000) 2022-09-10T07:38:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:19.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:19.241578+0000) 2022-09-10T07:38:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:19 smithi104 ceph-mon[97044]: pgmap v6053: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:38:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:19 smithi110 ceph-mon[97633]: pgmap v6053: 129 pgs: 129 active+clean; 8.3 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:38:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:19.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:19.607663+0000) 2022-09-10T07:38:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:20.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:20.241801+0000) 2022-09-10T07:38:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:20.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:20.607851+0000) 2022-09-10T07:38:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:21.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:21.242028+0000) 2022-09-10T07:38:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:21 smithi104 ceph-mon[97044]: pgmap v6054: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:38:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:21 smithi110 ceph-mon[97633]: pgmap v6054: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:38:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:21.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:21.608057+0000) 2022-09-10T07:38:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:22.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:22.242252+0000) 2022-09-10T07:38:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:22.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:22.608233+0000) 2022-09-10T07:38:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:23.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:23.242490+0000) 2022-09-10T07:38:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:23 smithi104 ceph-mon[97044]: pgmap v6055: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:38:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:23 smithi110 ceph-mon[97633]: pgmap v6055: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:38:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:23.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:23.608457+0000) 2022-09-10T07:38:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:24.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:24.242716+0000) 2022-09-10T07:38:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:24.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:24.608725+0000) 2022-09-10T07:38:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:25.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:25.242920+0000) 2022-09-10T07:38:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:25.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:25.608933+0000) 2022-09-10T07:38:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:25 smithi104 ceph-mon[97044]: pgmap v6056: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:38:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:25 smithi110 ceph-mon[97633]: pgmap v6056: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:38:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:26.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:26.243083+0000) 2022-09-10T07:38:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:26.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:26.609184+0000) 2022-09-10T07:38:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:27.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:27.243321+0000) 2022-09-10T07:38:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:27 smithi104 ceph-mon[97044]: pgmap v6057: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:27 smithi110 ceph-mon[97633]: pgmap v6057: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:27.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:27.609377+0000) 2022-09-10T07:38:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:28.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:28.243595+0000) 2022-09-10T07:38:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:28.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:28.609597+0000) 2022-09-10T07:38:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:29.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:29.243765+0000) 2022-09-10T07:38:29.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:29 smithi104 ceph-mon[97044]: pgmap v6058: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:29 smithi110 ceph-mon[97633]: pgmap v6058: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:29.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:29.609827+0000) 2022-09-10T07:38:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:30.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:30.243992+0000) 2022-09-10T07:38:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:30.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:30.610031+0000) 2022-09-10T07:38:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:31.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:31.244263+0000) 2022-09-10T07:38:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:31 smithi104 ceph-mon[97044]: pgmap v6059: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:31 smithi110 ceph-mon[97633]: pgmap v6059: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:31.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:31.610214+0000) 2022-09-10T07:38:32.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:38:31.841+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:38:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:32.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:32.244481+0000) 2022-09-10T07:38:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:32.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:32.610460+0000) 2022-09-10T07:38:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:33.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:33.244703+0000) 2022-09-10T07:38:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:33 smithi104 ceph-mon[97044]: pgmap v6060: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:33 smithi110 ceph-mon[97633]: pgmap v6060: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:33.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:33.610637+0000) 2022-09-10T07:38:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:34.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:34.244909+0000) 2022-09-10T07:38:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:34.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:34.610888+0000) 2022-09-10T07:38:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:35.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:35.245198+0000) 2022-09-10T07:38:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:35.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:35.611163+0000) 2022-09-10T07:38:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:35 smithi104 ceph-mon[97044]: pgmap v6061: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:38:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:35 smithi110 ceph-mon[97633]: pgmap v6061: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:38:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:36.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:36.245448+0000) 2022-09-10T07:38:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:36.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:36.611327+0000) 2022-09-10T07:38:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:37.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:37.245717+0000) 2022-09-10T07:38:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:37 smithi104 ceph-mon[97044]: pgmap v6062: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:37 smithi110 ceph-mon[97633]: pgmap v6062: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:37.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:37.611534+0000) 2022-09-10T07:38:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:38.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:38.245927+0000) 2022-09-10T07:38:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:38.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:38.611782+0000) 2022-09-10T07:38:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:39.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:39.246198+0000) 2022-09-10T07:38:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:39 smithi104 ceph-mon[97044]: pgmap v6063: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:39 smithi110 ceph-mon[97633]: pgmap v6063: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:39.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:39.612024+0000) 2022-09-10T07:38:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:40.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:40.246376+0000) 2022-09-10T07:38:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:40.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:40.612339+0000) 2022-09-10T07:38:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:41.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:41.246634+0000) 2022-09-10T07:38:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:41 smithi104 ceph-mon[97044]: pgmap v6064: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:41 smithi110 ceph-mon[97633]: pgmap v6064: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:41.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:41.612578+0000) 2022-09-10T07:38:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:42.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:42.246898+0000) 2022-09-10T07:38:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:42.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:42.612791+0000) 2022-09-10T07:38:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:43.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:43.247170+0000) 2022-09-10T07:38:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:43 smithi104 ceph-mon[97044]: pgmap v6065: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:43 smithi110 ceph-mon[97633]: pgmap v6065: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:43.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:43.613029+0000) 2022-09-10T07:38:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:44.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:44.247386+0000) 2022-09-10T07:38:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:38:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:38:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:38:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:38:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:38:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:38:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:38:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:38:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:44.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:44.613293+0000) 2022-09-10T07:38:45.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:45.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:45.247606+0000) 2022-09-10T07:38:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:45.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:45.613512+0000) 2022-09-10T07:38:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:45 smithi104 ceph-mon[97044]: pgmap v6066: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:38:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:45 smithi110 ceph-mon[97633]: pgmap v6066: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:38:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:46.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:46.247797+0000) 2022-09-10T07:38:47.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:38:46.852+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:38:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:46.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:46.613706+0000) 2022-09-10T07:38:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:47.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:47.248019+0000) 2022-09-10T07:38:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:47 smithi104 ceph-mon[97044]: pgmap v6067: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:47 smithi110 ceph-mon[97633]: pgmap v6067: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:47.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:47.613911+0000) 2022-09-10T07:38:48.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:48.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:48.248308+0000) 2022-09-10T07:38:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:48.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:48.614050+0000) 2022-09-10T07:38:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:49.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:49.248533+0000) 2022-09-10T07:38:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:49 smithi104 ceph-mon[97044]: pgmap v6068: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:49 smithi110 ceph-mon[97633]: pgmap v6068: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:49.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:49.614287+0000) 2022-09-10T07:38:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:50.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:50.248785+0000) 2022-09-10T07:38:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:50.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:50.614467+0000) 2022-09-10T07:38:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:51.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:51.249039+0000) 2022-09-10T07:38:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:51 smithi104 ceph-mon[97044]: pgmap v6069: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:51 smithi110 ceph-mon[97633]: pgmap v6069: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:38:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:51.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:51.614674+0000) 2022-09-10T07:38:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:52.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:52.249263+0000) 2022-09-10T07:38:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:52.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:52.614933+0000) 2022-09-10T07:38:53.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:53.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:53.249471+0000) 2022-09-10T07:38:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:53 smithi104 ceph-mon[97044]: pgmap v6070: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:53 smithi110 ceph-mon[97633]: pgmap v6070: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:53.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:53.615197+0000) 2022-09-10T07:38:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:54.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:54.249714+0000) 2022-09-10T07:38:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:54.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:54.615427+0000) 2022-09-10T07:38:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:55.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:55.249980+0000) 2022-09-10T07:38:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:55.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:55.615654+0000) 2022-09-10T07:38:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:55 smithi104 ceph-mon[97044]: pgmap v6071: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:38:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:55 smithi110 ceph-mon[97633]: pgmap v6071: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:38:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:56.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:56.250233+0000) 2022-09-10T07:38:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:56.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:56.615800+0000) 2022-09-10T07:38:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:57.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:57.250413+0000) 2022-09-10T07:38:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:57 smithi104 ceph-mon[97044]: pgmap v6072: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:57 smithi110 ceph-mon[97633]: pgmap v6072: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:38:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:57.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:57.616042+0000) 2022-09-10T07:38:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:58.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:58.250641+0000) 2022-09-10T07:38:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:58.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:58.616209+0000) 2022-09-10T07:38:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:38:59.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:59.250853+0000) 2022-09-10T07:38:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:38:59 smithi104 ceph-mon[97044]: pgmap v6073: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:38:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:38:59 smithi110 ceph-mon[97633]: pgmap v6073: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:38:59.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:38:59.616394+0000) 2022-09-10T07:39:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:00.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:00.251055+0000) 2022-09-10T07:39:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:00.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:00.616637+0000) 2022-09-10T07:39:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:01.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:01.251315+0000) 2022-09-10T07:39:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:01 smithi104 ceph-mon[97044]: pgmap v6074: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:01.867 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:01.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:01.616869+0000) 2022-09-10T07:39:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:01 smithi110 ceph-mon[97633]: pgmap v6074: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:02.252 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:39:01.866+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:39:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:02.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:02.251561+0000) 2022-09-10T07:39:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:02.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:02.617158+0000) 2022-09-10T07:39:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:03.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:03.251734+0000) 2022-09-10T07:39:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:03 smithi104 ceph-mon[97044]: pgmap v6075: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:03 smithi110 ceph-mon[97633]: pgmap v6075: 129 pgs: 129 active+clean; 8.3 MiB data, 689 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:03.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:03.617295+0000) 2022-09-10T07:39:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:04.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:04.251959+0000) 2022-09-10T07:39:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:04.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:04.617467+0000) 2022-09-10T07:39:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:05.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:05.252209+0000) 2022-09-10T07:39:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:05.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:05.617640+0000) 2022-09-10T07:39:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:05 smithi104 ceph-mon[97044]: pgmap v6076: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:39:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:05 smithi110 ceph-mon[97633]: pgmap v6076: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:39:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:06.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:06.252435+0000) 2022-09-10T07:39:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:06.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:06.617841+0000) 2022-09-10T07:39:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:07.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:07.252618+0000) 2022-09-10T07:39:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:07.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:07.618021+0000) 2022-09-10T07:39:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:08.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:08.252855+0000) 2022-09-10T07:39:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:08 smithi110 ceph-mon[97633]: osdmap e460: 12 total, 12 up, 12 in 2022-09-10T07:39:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:08 smithi110 ceph-mon[97633]: pgmap v6078: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:08 smithi104 ceph-mon[97044]: osdmap e460: 12 total, 12 up, 12 in 2022-09-10T07:39:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:08 smithi104 ceph-mon[97044]: pgmap v6078: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:08.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:08.618198+0000) 2022-09-10T07:39:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:09.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:09.253121+0000) 2022-09-10T07:39:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:09 smithi104 ceph-mon[97044]: pgmap v6079: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:09 smithi110 ceph-mon[97633]: pgmap v6079: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:09.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:09.618466+0000) 2022-09-10T07:39:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:10.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:10.253305+0000) 2022-09-10T07:39:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:10.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:10.618676+0000) 2022-09-10T07:39:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:11.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:11.253513+0000) 2022-09-10T07:39:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:11 smithi104 ceph-mon[97044]: pgmap v6080: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:11 smithi110 ceph-mon[97633]: pgmap v6080: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:11.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:11.618890+0000) 2022-09-10T07:39:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:12.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:12.253741+0000) 2022-09-10T07:39:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:12.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:12.619155+0000) 2022-09-10T07:39:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:13.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:13.254046+0000) 2022-09-10T07:39:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:13 smithi104 ceph-mon[97044]: pgmap v6081: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:13 smithi110 ceph-mon[97633]: pgmap v6081: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:39:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:13.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:13.619380+0000) 2022-09-10T07:39:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:14.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:14.254212+0000) 2022-09-10T07:39:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:14.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:14.619617+0000) 2022-09-10T07:39:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:15.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:15.254429+0000) 2022-09-10T07:39:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:15.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:15.619845+0000) 2022-09-10T07:39:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:15 smithi104 ceph-mon[97044]: osdmap e461: 12 total, 12 up, 12 in 2022-09-10T07:39:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:15 smithi104 ceph-mon[97044]: pgmap v6083: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T07:39:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:15 smithi110 ceph-mon[97633]: osdmap e461: 12 total, 12 up, 12 in 2022-09-10T07:39:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:15 smithi110 ceph-mon[97633]: pgmap v6083: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T07:39:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:16.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:16.254655+0000) 2022-09-10T07:39:16.877 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:16.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:16.620034+0000) 2022-09-10T07:39:17.255 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:39:16.875+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:39:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:17.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:17.254938+0000) 2022-09-10T07:39:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:17 smithi104 ceph-mon[97044]: pgmap v6084: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:39:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:17 smithi110 ceph-mon[97633]: pgmap v6084: 129 pgs: 129 active+clean; 8.3 MiB data, 690 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:39:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:17.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:17.620279+0000) 2022-09-10T07:39:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:18.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:18.255135+0000) 2022-09-10T07:39:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:18.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:18.620490+0000) 2022-09-10T07:39:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:19.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:19.255331+0000) 2022-09-10T07:39:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:19 smithi104 ceph-mon[97044]: pgmap v6085: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:39:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:19 smithi110 ceph-mon[97633]: pgmap v6085: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:39:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:19.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:19.620752+0000) 2022-09-10T07:39:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:20.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:20.255492+0000) 2022-09-10T07:39:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:20.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:20.620970+0000) 2022-09-10T07:39:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:21.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:21.255749+0000) 2022-09-10T07:39:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:21 smithi104 ceph-mon[97044]: pgmap v6086: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:39:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:21 smithi110 ceph-mon[97633]: pgmap v6086: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:39:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:21.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:21.621166+0000) 2022-09-10T07:39:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:22.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:22.256023+0000) 2022-09-10T07:39:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:22.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:22.621398+0000) 2022-09-10T07:39:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:23.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:23.256237+0000) 2022-09-10T07:39:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:23 smithi104 ceph-mon[97044]: pgmap v6087: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:39:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:23 smithi110 ceph-mon[97633]: pgmap v6087: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:39:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:23.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:23.621612+0000) 2022-09-10T07:39:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:24.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:24.256448+0000) 2022-09-10T07:39:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:24.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:24.621856+0000) 2022-09-10T07:39:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:25.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:25.256709+0000) 2022-09-10T07:39:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:25.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:25.622119+0000) 2022-09-10T07:39:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:25 smithi104 ceph-mon[97044]: pgmap v6088: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:39:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:25 smithi110 ceph-mon[97633]: pgmap v6088: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:39:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:26.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:26.256938+0000) 2022-09-10T07:39:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:26.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:26.622301+0000) 2022-09-10T07:39:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:27.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:27.257204+0000) 2022-09-10T07:39:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:27 smithi104 ceph-mon[97044]: pgmap v6089: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:27 smithi110 ceph-mon[97633]: pgmap v6089: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:27.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:27.622496+0000) 2022-09-10T07:39:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:28.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:28.257430+0000) 2022-09-10T07:39:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:28.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:28.622714+0000) 2022-09-10T07:39:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:29.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:29.257634+0000) 2022-09-10T07:39:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:29 smithi104 ceph-mon[97044]: pgmap v6090: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:29.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:29 smithi110 ceph-mon[97633]: pgmap v6090: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:29.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:29.622931+0000) 2022-09-10T07:39:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:30.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:30.257852+0000) 2022-09-10T07:39:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:30.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:30.623216+0000) 2022-09-10T07:39:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:31.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:31.258080+0000) 2022-09-10T07:39:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:31 smithi104 ceph-mon[97044]: pgmap v6091: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:31.886 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:31.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:31.623441+0000) 2022-09-10T07:39:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:31 smithi110 ceph-mon[97633]: pgmap v6091: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:32.259 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:39:31.884+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:39:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:32.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:32.258329+0000) 2022-09-10T07:39:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:32.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:32.623702+0000) 2022-09-10T07:39:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:33.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:33.258614+0000) 2022-09-10T07:39:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:33 smithi104 ceph-mon[97044]: pgmap v6092: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:33 smithi110 ceph-mon[97633]: pgmap v6092: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:33.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:33.623919+0000) 2022-09-10T07:39:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:34.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:34.258839+0000) 2022-09-10T07:39:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:34.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:34.624179+0000) 2022-09-10T07:39:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:35.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:35.259041+0000) 2022-09-10T07:39:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:35.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:35.624405+0000) 2022-09-10T07:39:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:35 smithi104 ceph-mon[97044]: pgmap v6093: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:39:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:35 smithi110 ceph-mon[97633]: pgmap v6093: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:39:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:36.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:36.259325+0000) 2022-09-10T07:39:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:36.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:36.624639+0000) 2022-09-10T07:39:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:37.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:37.259627+0000) 2022-09-10T07:39:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:37 smithi104 ceph-mon[97044]: pgmap v6094: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:37 smithi110 ceph-mon[97633]: pgmap v6094: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:37.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:37.624885+0000) 2022-09-10T07:39:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:38.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:38.259812+0000) 2022-09-10T07:39:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:38.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:38.625142+0000) 2022-09-10T07:39:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:39.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:39.260071+0000) 2022-09-10T07:39:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:39 smithi104 ceph-mon[97044]: pgmap v6095: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:39 smithi110 ceph-mon[97633]: pgmap v6095: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:39.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:39.625368+0000) 2022-09-10T07:39:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:40.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:40.260285+0000) 2022-09-10T07:39:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:40.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:40.625558+0000) 2022-09-10T07:39:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:41.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:41.260491+0000) 2022-09-10T07:39:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:41 smithi104 ceph-mon[97044]: pgmap v6096: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:41 smithi110 ceph-mon[97633]: pgmap v6096: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:41.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:41.625757+0000) 2022-09-10T07:39:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:42.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:42.260712+0000) 2022-09-10T07:39:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:42.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:42.625958+0000) 2022-09-10T07:39:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:43.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:43.260909+0000) 2022-09-10T07:39:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:43 smithi104 ceph-mon[97044]: pgmap v6097: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:43 smithi110 ceph-mon[97633]: pgmap v6097: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:43.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:43.626198+0000) 2022-09-10T07:39:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:44.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:44.261167+0000) 2022-09-10T07:39:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:39:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:39:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:39:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:39:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:39:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:39:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:39:44.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:39:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:44.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:44.626395+0000) 2022-09-10T07:39:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:45.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:45.261357+0000) 2022-09-10T07:39:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:45.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:45.626544+0000) 2022-09-10T07:39:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:45 smithi104 ceph-mon[97044]: pgmap v6098: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:39:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:45 smithi110 ceph-mon[97633]: pgmap v6098: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:39:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:46.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:46.261530+0000) 2022-09-10T07:39:46.902 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:46.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:46.626775+0000) 2022-09-10T07:39:47.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:39:46.900+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:39:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:47.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:47.261816+0000) 2022-09-10T07:39:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:47 smithi104 ceph-mon[97044]: pgmap v6099: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:47 smithi110 ceph-mon[97633]: pgmap v6099: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:47.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:47.626991+0000) 2022-09-10T07:39:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:48.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:48.262042+0000) 2022-09-10T07:39:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:48.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:48.627237+0000) 2022-09-10T07:39:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:49.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:49.262263+0000) 2022-09-10T07:39:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:49 smithi104 ceph-mon[97044]: pgmap v6100: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:49 smithi110 ceph-mon[97633]: pgmap v6100: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:49.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:49.627377+0000) 2022-09-10T07:39:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:50.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:50.262458+0000) 2022-09-10T07:39:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:50.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:50.627611+0000) 2022-09-10T07:39:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:51.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:51.262667+0000) 2022-09-10T07:39:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:51 smithi104 ceph-mon[97044]: pgmap v6101: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:51 smithi110 ceph-mon[97633]: pgmap v6101: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:39:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:51.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:51.627805+0000) 2022-09-10T07:39:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:52.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:52.262952+0000) 2022-09-10T07:39:53.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:52.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:52.627985+0000) 2022-09-10T07:39:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:53.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:53.263195+0000) 2022-09-10T07:39:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:53 smithi104 ceph-mon[97044]: pgmap v6102: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:53 smithi110 ceph-mon[97633]: pgmap v6102: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:53.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:53.628197+0000) 2022-09-10T07:39:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:54.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:54.263431+0000) 2022-09-10T07:39:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:54.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:54.628422+0000) 2022-09-10T07:39:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:55.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:55.263677+0000) 2022-09-10T07:39:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:55.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:55.628657+0000) 2022-09-10T07:39:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:55 smithi104 ceph-mon[97044]: pgmap v6103: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:39:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:55 smithi110 ceph-mon[97633]: pgmap v6103: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:39:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:56.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:56.263910+0000) 2022-09-10T07:39:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:56.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:56.628829+0000) 2022-09-10T07:39:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:57.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:57.264165+0000) 2022-09-10T07:39:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:57 smithi104 ceph-mon[97044]: pgmap v6104: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:57 smithi110 ceph-mon[97633]: pgmap v6104: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:39:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:57.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:57.629085+0000) 2022-09-10T07:39:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:58.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:58.264382+0000) 2022-09-10T07:39:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:58.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:58.629274+0000) 2022-09-10T07:39:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:39:59.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:59.264597+0000) 2022-09-10T07:39:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:39:59 smithi104 ceph-mon[97044]: pgmap v6105: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:39:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:39:59 smithi110 ceph-mon[97633]: pgmap v6105: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:39:59.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:39:59.629527+0000) 2022-09-10T07:40:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:00.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:00.264817+0000) 2022-09-10T07:40:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:00.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:00.629750+0000) 2022-09-10T07:40:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:40:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:40:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:01.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:01.265056+0000) 2022-09-10T07:40:01.916 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:01.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:01.629964+0000) 2022-09-10T07:40:02.266 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:40:01.914+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:40:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:01 smithi104 ceph-mon[97044]: pgmap v6106: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:01 smithi110 ceph-mon[97633]: pgmap v6106: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:02.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:02.265343+0000) 2022-09-10T07:40:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:02.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:02.630200+0000) 2022-09-10T07:40:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:03.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:03.265577+0000) 2022-09-10T07:40:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:03 smithi104 ceph-mon[97044]: pgmap v6107: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:03 smithi110 ceph-mon[97633]: pgmap v6107: 129 pgs: 129 active+clean; 8.3 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:03.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:03.630425+0000) 2022-09-10T07:40:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:04.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:04.265811+0000) 2022-09-10T07:40:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:04.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:04.630627+0000) 2022-09-10T07:40:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:05.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:05.266024+0000) 2022-09-10T07:40:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:05.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:05.630862+0000) 2022-09-10T07:40:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:06 smithi104 ceph-mon[97044]: pgmap v6108: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:40:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:06 smithi110 ceph-mon[97633]: pgmap v6108: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:40:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:06.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:06.266272+0000) 2022-09-10T07:40:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:06.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:06.631176+0000) 2022-09-10T07:40:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:07.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:07.266519+0000) 2022-09-10T07:40:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:07.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:07.631348+0000) 2022-09-10T07:40:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:08.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:08.266816+0000) 2022-09-10T07:40:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:08 smithi110 ceph-mon[97633]: osdmap e462: 12 total, 12 up, 12 in 2022-09-10T07:40:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:08 smithi110 ceph-mon[97633]: pgmap v6110: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:08 smithi104 ceph-mon[97044]: osdmap e462: 12 total, 12 up, 12 in 2022-09-10T07:40:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:08 smithi104 ceph-mon[97044]: pgmap v6110: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:08.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:08.631580+0000) 2022-09-10T07:40:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:09.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:09.267044+0000) 2022-09-10T07:40:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:09 smithi104 ceph-mon[97044]: pgmap v6111: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:09 smithi110 ceph-mon[97633]: pgmap v6111: 129 pgs: 129 active+clean; 8.3 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:09.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:09.631829+0000) 2022-09-10T07:40:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:10.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:10.267317+0000) 2022-09-10T07:40:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:10.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:10.632064+0000) 2022-09-10T07:40:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:11.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:11.267556+0000) 2022-09-10T07:40:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:11 smithi104 ceph-mon[97044]: pgmap v6112: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:11 smithi110 ceph-mon[97633]: pgmap v6112: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:11.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:11.632277+0000) 2022-09-10T07:40:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:12.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:12.267771+0000) 2022-09-10T07:40:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:12.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:12.632500+0000) 2022-09-10T07:40:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:13.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:13.267998+0000) 2022-09-10T07:40:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:13 smithi104 ceph-mon[97044]: pgmap v6113: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:13 smithi110 ceph-mon[97633]: pgmap v6113: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:40:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:13.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:13.632773+0000) 2022-09-10T07:40:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:14.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:14.268268+0000) 2022-09-10T07:40:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:14.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:14.633017+0000) 2022-09-10T07:40:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:15.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:15.268515+0000) 2022-09-10T07:40:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:15.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:15.633254+0000) 2022-09-10T07:40:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:15 smithi104 ceph-mon[97044]: osdmap e463: 12 total, 12 up, 12 in 2022-09-10T07:40:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:15 smithi104 ceph-mon[97044]: pgmap v6115: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T07:40:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:15 smithi110 ceph-mon[97633]: osdmap e463: 12 total, 12 up, 12 in 2022-09-10T07:40:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:15 smithi110 ceph-mon[97633]: pgmap v6115: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T07:40:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:16.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:16.268680+0000) 2022-09-10T07:40:16.932 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:16.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:16.633487+0000) 2022-09-10T07:40:17.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:40:16.930+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:40:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:17.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:17.268909+0000) 2022-09-10T07:40:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:17 smithi104 ceph-mon[97044]: pgmap v6116: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:40:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:17 smithi110 ceph-mon[97633]: pgmap v6116: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:40:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:17.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:17.633655+0000) 2022-09-10T07:40:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:18.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:18.269184+0000) 2022-09-10T07:40:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:18.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:18.633871+0000) 2022-09-10T07:40:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:19.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:19.269402+0000) 2022-09-10T07:40:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:19 smithi104 ceph-mon[97044]: pgmap v6117: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:40:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:19 smithi110 ceph-mon[97633]: pgmap v6117: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:40:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:19.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:19.634142+0000) 2022-09-10T07:40:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:20.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:20.269568+0000) 2022-09-10T07:40:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:20.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:20.634381+0000) 2022-09-10T07:40:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:21.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:21.269769+0000) 2022-09-10T07:40:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:21 smithi104 ceph-mon[97044]: pgmap v6118: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:40:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:21 smithi110 ceph-mon[97633]: pgmap v6118: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:40:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:21.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:21.634608+0000) 2022-09-10T07:40:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:22.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:22.270007+0000) 2022-09-10T07:40:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:22.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:22.634836+0000) 2022-09-10T07:40:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:23.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:23.270152+0000) 2022-09-10T07:40:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:23 smithi104 ceph-mon[97044]: pgmap v6119: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:40:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:23 smithi110 ceph-mon[97633]: pgmap v6119: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:40:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:23.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:23.635029+0000) 2022-09-10T07:40:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:24.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:24.270382+0000) 2022-09-10T07:40:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:24.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:24.635298+0000) 2022-09-10T07:40:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:25.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:25.270625+0000) 2022-09-10T07:40:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:25.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:25.635515+0000) 2022-09-10T07:40:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:26 smithi104 ceph-mon[97044]: pgmap v6120: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:40:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:26 smithi110 ceph-mon[97633]: pgmap v6120: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:40:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:26.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:26.270838+0000) 2022-09-10T07:40:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:26.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:26.635755+0000) 2022-09-10T07:40:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:27.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:27.271074+0000) 2022-09-10T07:40:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:27 smithi104 ceph-mon[97044]: pgmap v6121: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:27 smithi110 ceph-mon[97633]: pgmap v6121: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:27.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:27.635982+0000) 2022-09-10T07:40:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:28.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:28.271336+0000) 2022-09-10T07:40:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:28.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:28.636233+0000) 2022-09-10T07:40:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:29.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:29.271591+0000) 2022-09-10T07:40:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:29 smithi104 ceph-mon[97044]: pgmap v6122: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:29 smithi110 ceph-mon[97633]: pgmap v6122: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:29.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:29.636435+0000) 2022-09-10T07:40:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:30.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:30.271797+0000) 2022-09-10T07:40:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:30.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:30.636675+0000) 2022-09-10T07:40:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:31.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:31.272052+0000) 2022-09-10T07:40:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:31 smithi104 ceph-mon[97044]: pgmap v6123: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:31 smithi110 ceph-mon[97633]: pgmap v6123: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:31.947 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:31.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:31.636885+0000) 2022-09-10T07:40:32.273 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:40:31.946+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:40:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:32.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:32.272354+0000) 2022-09-10T07:40:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:32.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:32.637085+0000) 2022-09-10T07:40:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:33.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:33.272618+0000) 2022-09-10T07:40:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:33 smithi104 ceph-mon[97044]: pgmap v6124: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:33 smithi110 ceph-mon[97633]: pgmap v6124: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:33.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:33.637321+0000) 2022-09-10T07:40:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:34.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:34.272876+0000) 2022-09-10T07:40:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:34.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:34.637522+0000) 2022-09-10T07:40:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:35.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:35.273116+0000) 2022-09-10T07:40:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:35.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:35.637741+0000) 2022-09-10T07:40:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:36 smithi104 ceph-mon[97044]: pgmap v6125: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:40:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:36 smithi110 ceph-mon[97633]: pgmap v6125: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:40:36.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:36.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:36.273411+0000) 2022-09-10T07:40:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:36.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:36.637927+0000) 2022-09-10T07:40:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:37.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:37.273685+0000) 2022-09-10T07:40:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:37 smithi104 ceph-mon[97044]: pgmap v6126: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:37 smithi110 ceph-mon[97633]: pgmap v6126: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:37.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:37.638163+0000) 2022-09-10T07:40:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:38.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:38.273940+0000) 2022-09-10T07:40:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:38.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:38.638351+0000) 2022-09-10T07:40:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:39.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:39.274177+0000) 2022-09-10T07:40:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:39 smithi104 ceph-mon[97044]: pgmap v6127: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:39 smithi110 ceph-mon[97633]: pgmap v6127: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:39.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:39.638586+0000) 2022-09-10T07:40:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:40.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:40.274446+0000) 2022-09-10T07:40:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:40.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:40.638733+0000) 2022-09-10T07:40:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:41.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:41.274678+0000) 2022-09-10T07:40:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:41 smithi104 ceph-mon[97044]: pgmap v6128: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:41 smithi110 ceph-mon[97633]: pgmap v6128: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:41.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:41.638962+0000) 2022-09-10T07:40:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:42.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:42.274933+0000) 2022-09-10T07:40:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:42.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:42.639211+0000) 2022-09-10T07:40:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:43.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:43.275192+0000) 2022-09-10T07:40:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:43 smithi104 ceph-mon[97044]: pgmap v6129: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:43 smithi110 ceph-mon[97633]: pgmap v6129: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:43.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:43.639386+0000) 2022-09-10T07:40:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:44.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:44.275415+0000) 2022-09-10T07:40:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:40:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:40:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:40:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:40:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:40:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:40:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:40:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:40:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:44.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:44.639636+0000) 2022-09-10T07:40:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:45.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:45.275700+0000) 2022-09-10T07:40:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:45.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:45.639847+0000) 2022-09-10T07:40:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:46 smithi104 ceph-mon[97044]: pgmap v6130: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:40:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:46 smithi110 ceph-mon[97633]: pgmap v6130: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:40:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:46.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:46.275966+0000) 2022-09-10T07:40:46.958 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:46.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:46.640014+0000) 2022-09-10T07:40:47.277 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:40:46.956+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:40:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:47.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:47.276260+0000) 2022-09-10T07:40:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:47 smithi104 ceph-mon[97044]: pgmap v6131: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:47 smithi110 ceph-mon[97633]: pgmap v6131: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:47.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:47.640199+0000) 2022-09-10T07:40:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:48.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:48.276524+0000) 2022-09-10T07:40:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:48.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:48.640359+0000) 2022-09-10T07:40:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:49.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:49.276785+0000) 2022-09-10T07:40:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:49 smithi104 ceph-mon[97044]: pgmap v6132: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:49 smithi110 ceph-mon[97633]: pgmap v6132: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:49.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:49.640534+0000) 2022-09-10T07:40:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:50.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:50.277011+0000) 2022-09-10T07:40:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:50.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:50.640741+0000) 2022-09-10T07:40:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:51.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:51.277263+0000) 2022-09-10T07:40:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:51 smithi104 ceph-mon[97044]: pgmap v6133: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:51 smithi110 ceph-mon[97633]: pgmap v6133: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:40:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:51.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:51.641002+0000) 2022-09-10T07:40:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:52.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:52.277496+0000) 2022-09-10T07:40:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:52.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:52.641200+0000) 2022-09-10T07:40:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:53.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:53.277718+0000) 2022-09-10T07:40:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:53 smithi104 ceph-mon[97044]: pgmap v6134: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:53 smithi110 ceph-mon[97633]: pgmap v6134: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:53.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:53.641376+0000) 2022-09-10T07:40:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:54.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:54.277954+0000) 2022-09-10T07:40:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:54.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:54.641619+0000) 2022-09-10T07:40:55.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:55.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:55.278161+0000) 2022-09-10T07:40:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:55.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:55.641877+0000) 2022-09-10T07:40:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:56 smithi104 ceph-mon[97044]: pgmap v6135: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:40:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:56 smithi110 ceph-mon[97633]: pgmap v6135: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:40:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:56.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:56.278415+0000) 2022-09-10T07:40:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:56.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:56.642054+0000) 2022-09-10T07:40:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:57.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:57.278651+0000) 2022-09-10T07:40:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:57 smithi104 ceph-mon[97044]: pgmap v6136: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:57 smithi110 ceph-mon[97633]: pgmap v6136: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:40:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:57.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:57.642276+0000) 2022-09-10T07:40:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:58.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:58.278842+0000) 2022-09-10T07:40:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:58.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:58.642450+0000) 2022-09-10T07:40:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:40:59.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:59.279123+0000) 2022-09-10T07:40:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:40:59 smithi104 ceph-mon[97044]: pgmap v6137: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:40:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:40:59 smithi110 ceph-mon[97633]: pgmap v6137: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:41:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:40:59.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:40:59.642690+0000) 2022-09-10T07:41:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:00.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:00.279374+0000) 2022-09-10T07:41:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:00.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:00.642949+0000) 2022-09-10T07:41:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:01.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:01.279582+0000) 2022-09-10T07:41:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:01 smithi104 ceph-mon[97044]: pgmap v6138: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:01 smithi110 ceph-mon[97633]: pgmap v6138: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:01.970 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:01.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:01.643172+0000) 2022-09-10T07:41:02.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:41:01.968+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:41:02.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:02.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:02.279865+0000) 2022-09-10T07:41:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:02.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:02.643422+0000) 2022-09-10T07:41:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:03.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:03.280074+0000) 2022-09-10T07:41:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:03 smithi104 ceph-mon[97044]: pgmap v6139: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:03 smithi110 ceph-mon[97633]: pgmap v6139: 129 pgs: 129 active+clean; 8.3 MiB data, 693 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:03.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:03.643622+0000) 2022-09-10T07:41:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:04.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:04.280346+0000) 2022-09-10T07:41:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:04.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:04.643853+0000) 2022-09-10T07:41:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:05.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:05.280611+0000) 2022-09-10T07:41:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:05.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:05.644087+0000) 2022-09-10T07:41:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:06 smithi104 ceph-mon[97044]: pgmap v6140: 129 pgs: 129 active+clean; 8.3 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:41:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:06 smithi110 ceph-mon[97633]: pgmap v6140: 129 pgs: 129 active+clean; 8.3 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:41:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:06.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:06.280855+0000) 2022-09-10T07:41:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:06.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:06.644374+0000) 2022-09-10T07:41:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:07.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:07.281058+0000) 2022-09-10T07:41:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:07.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:07.644609+0000) 2022-09-10T07:41:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:08.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:08.281239+0000) 2022-09-10T07:41:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:08 smithi110 ceph-mon[97633]: osdmap e464: 12 total, 12 up, 12 in 2022-09-10T07:41:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:08 smithi110 ceph-mon[97633]: pgmap v6142: 129 pgs: 129 active+clean; 8.3 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:08 smithi104 ceph-mon[97044]: osdmap e464: 12 total, 12 up, 12 in 2022-09-10T07:41:08.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:08 smithi104 ceph-mon[97044]: pgmap v6142: 129 pgs: 129 active+clean; 8.3 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:08.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:08.644884+0000) 2022-09-10T07:41:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:09.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:09.281449+0000) 2022-09-10T07:41:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:09 smithi104 ceph-mon[97044]: pgmap v6143: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:09 smithi110 ceph-mon[97633]: pgmap v6143: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:10.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:09.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:09.645143+0000) 2022-09-10T07:41:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:10.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:10.281733+0000) 2022-09-10T07:41:11.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:10.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:10.645394+0000) 2022-09-10T07:41:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:11.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:11.281960+0000) 2022-09-10T07:41:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:11 smithi104 ceph-mon[97044]: pgmap v6144: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:11 smithi110 ceph-mon[97633]: pgmap v6144: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:11.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:11.645629+0000) 2022-09-10T07:41:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:12.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:12.282228+0000) 2022-09-10T07:41:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:12.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:12.645852+0000) 2022-09-10T07:41:13.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:13.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:13.282422+0000) 2022-09-10T07:41:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:13 smithi104 ceph-mon[97044]: pgmap v6145: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:13 smithi110 ceph-mon[97633]: pgmap v6145: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:41:14.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:13.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:13.646090+0000) 2022-09-10T07:41:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:14.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:14.282666+0000) 2022-09-10T07:41:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:14.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:14.646348+0000) 2022-09-10T07:41:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:15.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:15.282871+0000) 2022-09-10T07:41:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:15.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:15.646597+0000) 2022-09-10T07:41:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:16 smithi104 ceph-mon[97044]: osdmap e465: 12 total, 12 up, 12 in 2022-09-10T07:41:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:16 smithi104 ceph-mon[97044]: pgmap v6147: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T07:41:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:16 smithi110 ceph-mon[97633]: osdmap e465: 12 total, 12 up, 12 in 2022-09-10T07:41:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:16 smithi110 ceph-mon[97633]: pgmap v6147: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T07:41:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:16.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:16.283074+0000) 2022-09-10T07:41:16.986 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:16.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:16.646822+0000) 2022-09-10T07:41:17.284 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:41:16.979+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:41:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:17.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:17.283350+0000) 2022-09-10T07:41:17.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:17 smithi104 ceph-mon[97044]: pgmap v6148: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:41:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:17 smithi110 ceph-mon[97633]: pgmap v6148: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:41:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:17.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:17.647070+0000) 2022-09-10T07:41:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:18.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:18.283554+0000) 2022-09-10T07:41:19.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:18.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:18.647263+0000) 2022-09-10T07:41:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:19.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:19.283714+0000) 2022-09-10T07:41:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:19 smithi104 ceph-mon[97044]: pgmap v6149: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:41:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:19 smithi110 ceph-mon[97633]: pgmap v6149: 129 pgs: 129 active+clean; 8.3 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:41:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:19.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:19.647401+0000) 2022-09-10T07:41:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:20.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:20.283892+0000) 2022-09-10T07:41:21.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:20.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:20.647627+0000) 2022-09-10T07:41:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:21.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:21.284108+0000) 2022-09-10T07:41:21.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:21 smithi104 ceph-mon[97044]: pgmap v6150: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:41:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:21 smithi110 ceph-mon[97633]: pgmap v6150: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:41:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:21.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:21.647808+0000) 2022-09-10T07:41:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:22.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:22.284302+0000) 2022-09-10T07:41:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:22.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:22.648051+0000) 2022-09-10T07:41:23.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:23.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:23.284484+0000) 2022-09-10T07:41:23.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:23 smithi104 ceph-mon[97044]: pgmap v6151: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:41:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:23 smithi110 ceph-mon[97633]: pgmap v6151: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:41:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:23.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:23.648247+0000) 2022-09-10T07:41:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:24.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:24.284626+0000) 2022-09-10T07:41:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:24.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:24.648413+0000) 2022-09-10T07:41:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:25.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:25.284774+0000) 2022-09-10T07:41:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:25.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:25.648563+0000) 2022-09-10T07:41:26.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:26 smithi104 ceph-mon[97044]: pgmap v6152: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:41:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:26 smithi110 ceph-mon[97633]: pgmap v6152: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:41:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:26.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:26.284943+0000) 2022-09-10T07:41:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:26.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:26.648755+0000) 2022-09-10T07:41:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:27.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:27.285178+0000) 2022-09-10T07:41:27.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:27 smithi104 ceph-mon[97044]: pgmap v6153: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:27.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:27 smithi110 ceph-mon[97633]: pgmap v6153: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:27.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:27.648980+0000) 2022-09-10T07:41:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:28.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:28.285326+0000) 2022-09-10T07:41:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:28.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:28.649229+0000) 2022-09-10T07:41:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:29.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:29.285531+0000) 2022-09-10T07:41:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:29 smithi104 ceph-mon[97044]: pgmap v6154: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:41:29.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:29 smithi110 ceph-mon[97633]: pgmap v6154: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:41:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:29.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:29.649483+0000) 2022-09-10T07:41:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:30.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:30.285737+0000) 2022-09-10T07:41:31.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:30.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:30.649667+0000) 2022-09-10T07:41:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:31.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:31.285931+0000) 2022-09-10T07:41:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:31 smithi104 ceph-mon[97044]: pgmap v6155: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:31.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:31 smithi110 ceph-mon[97633]: pgmap v6155: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:31.996 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:31.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:31.649907+0000) 2022-09-10T07:41:32.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:41:31.995+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:41:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:32.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:32.286084+0000) 2022-09-10T07:41:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:32.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:32.650228+0000) 2022-09-10T07:41:33.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:33.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:33.286266+0000) 2022-09-10T07:41:33.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:33 smithi104 ceph-mon[97044]: pgmap v6156: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:33 smithi110 ceph-mon[97633]: pgmap v6156: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:33.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:33.650414+0000) 2022-09-10T07:41:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:34.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:34.286513+0000) 2022-09-10T07:41:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:34.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:34.650608+0000) 2022-09-10T07:41:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:35.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:35.286714+0000) 2022-09-10T07:41:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:35.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:35.650819+0000) 2022-09-10T07:41:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:36 smithi104 ceph-mon[97044]: pgmap v6157: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:36 smithi110 ceph-mon[97633]: pgmap v6157: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:36.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:36.286969+0000) 2022-09-10T07:41:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:36.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:36.650997+0000) 2022-09-10T07:41:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:37.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:37.287176+0000) 2022-09-10T07:41:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:37 smithi104 ceph-mon[97044]: pgmap v6158: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:37 smithi110 ceph-mon[97633]: pgmap v6158: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:37.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:37.651244+0000) 2022-09-10T07:41:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:38.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:38.287424+0000) 2022-09-10T07:41:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:38.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:38.651410+0000) 2022-09-10T07:41:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:39.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:39.287644+0000) 2022-09-10T07:41:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:39 smithi104 ceph-mon[97044]: pgmap v6159: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:41:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:39 smithi110 ceph-mon[97633]: pgmap v6159: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T07:41:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:39.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:39.651641+0000) 2022-09-10T07:41:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:40.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:40.287804+0000) 2022-09-10T07:41:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:40.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:40.651874+0000) 2022-09-10T07:41:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:41.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:41.287995+0000) 2022-09-10T07:41:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:41 smithi104 ceph-mon[97044]: pgmap v6160: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:41.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:41 smithi110 ceph-mon[97633]: pgmap v6160: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:41.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:41.652041+0000) 2022-09-10T07:41:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:42.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:42.288149+0000) 2022-09-10T07:41:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:42.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:42.652298+0000) 2022-09-10T07:41:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:43.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:43.288352+0000) 2022-09-10T07:41:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:43 smithi104 ceph-mon[97044]: pgmap v6161: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:43 smithi110 ceph-mon[97633]: pgmap v6161: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:43.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:43.652448+0000) 2022-09-10T07:41:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:44.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:44.288553+0000) 2022-09-10T07:41:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:41:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:41:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:41:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:41:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:41:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:41:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:41:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:41:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:44.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:44.652641+0000) 2022-09-10T07:41:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:45.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:45.288816+0000) 2022-09-10T07:41:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:45.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:45.652827+0000) 2022-09-10T07:41:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:46 smithi104 ceph-mon[97044]: pgmap v6162: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:46 smithi110 ceph-mon[97633]: pgmap v6162: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:46.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:46.288991+0000) 2022-09-10T07:41:47.009 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:46.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:46.653118+0000) 2022-09-10T07:41:47.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:41:47.008+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:41:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:47.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:47.289202+0000) 2022-09-10T07:41:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:47 smithi104 ceph-mon[97044]: pgmap v6163: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:47 smithi110 ceph-mon[97633]: pgmap v6163: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:47.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:47.653262+0000) 2022-09-10T07:41:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:48.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:48.289366+0000) 2022-09-10T07:41:49.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:48.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:48.653462+0000) 2022-09-10T07:41:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:49.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:49.289551+0000) 2022-09-10T07:41:50.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:49.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:49.653704+0000) 2022-09-10T07:41:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:50.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:50.289759+0000) 2022-09-10T07:41:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:50.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:50.653981+0000) 2022-09-10T07:41:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:51 smithi104 ceph-mon[97044]: pgmap v6164: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:51 smithi110 ceph-mon[97633]: pgmap v6164: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:51.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:51.289964+0000) 2022-09-10T07:41:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:51.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:51.654244+0000) 2022-09-10T07:41:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:52.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:52.290140+0000) 2022-09-10T07:41:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:52.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:52.654504+0000) 2022-09-10T07:41:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:53 smithi104 ceph-mon[97044]: pgmap v6165: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:53 smithi110 ceph-mon[97633]: pgmap v6165: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T07:41:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:53.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:53.290348+0000) 2022-09-10T07:41:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:53.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:53.654750+0000) 2022-09-10T07:41:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:54.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:54.290489+0000) 2022-09-10T07:41:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:54.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:54.654992+0000) 2022-09-10T07:41:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:55 smithi104 ceph-mon[97044]: pgmap v6166: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:55 smithi110 ceph-mon[97633]: pgmap v6166: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:55.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:55.290711+0000) 2022-09-10T07:41:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:55.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:55.655222+0000) 2022-09-10T07:41:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:56.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:56.290935+0000) 2022-09-10T07:41:57.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:56.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:56.655375+0000) 2022-09-10T07:41:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:57 smithi104 ceph-mon[97044]: pgmap v6167: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:57 smithi110 ceph-mon[97633]: pgmap v6167: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:41:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:57.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:57.291185+0000) 2022-09-10T07:41:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:57.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:57.655633+0000) 2022-09-10T07:41:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:58.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:58.291425+0000) 2022-09-10T07:41:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:58.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:58.655944+0000) 2022-09-10T07:41:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:41:59 smithi104 ceph-mon[97044]: pgmap v6168: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:41:59 smithi110 ceph-mon[97633]: pgmap v6168: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:41:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:41:59.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:59.291544+0000) 2022-09-10T07:42:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:41:59.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:41:59.656227+0000) 2022-09-10T07:42:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:00.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:00.291682+0000) 2022-09-10T07:42:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:00.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:00.656449+0000) 2022-09-10T07:42:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:01 smithi104 ceph-mon[97044]: pgmap v6169: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:01 smithi110 ceph-mon[97633]: pgmap v6169: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:01.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:01.291916+0000) 2022-09-10T07:42:02.018 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:01.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:01.656627+0000) 2022-09-10T07:42:02.294 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:42:02.016+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:42:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:02.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:02.292092+0000) 2022-09-10T07:42:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:02.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:02.656822+0000) 2022-09-10T07:42:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:03 smithi104 ceph-mon[97044]: pgmap v6170: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:03 smithi110 ceph-mon[97633]: pgmap v6170: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:03.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:03.292282+0000) 2022-09-10T07:42:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:03.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:03.657052+0000) 2022-09-10T07:42:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:04.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:04.292519+0000) 2022-09-10T07:42:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:04.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:04.657336+0000) 2022-09-10T07:42:05.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:05 smithi104 ceph-mon[97044]: pgmap v6171: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:05 smithi110 ceph-mon[97633]: pgmap v6171: 129 pgs: 129 active+clean; 8.3 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:05.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:05.292725+0000) 2022-09-10T07:42:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:05.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:05.657571+0000) 2022-09-10T07:42:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:06.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:06.292926+0000) 2022-09-10T07:42:07.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:06.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:06.657808+0000) 2022-09-10T07:42:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:07 smithi110 ceph-mon[97633]: pgmap v6172: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:42:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:07 smithi104 ceph-mon[97044]: pgmap v6172: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:42:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:07.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:07.293200+0000) 2022-09-10T07:42:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:07.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:07.658052+0000) 2022-09-10T07:42:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:08.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:08.293440+0000) 2022-09-10T07:42:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:08.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:08.658275+0000) 2022-09-10T07:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:09 smithi110 ceph-mon[97633]: pgmap v6173: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:09 smithi110 ceph-mon[97633]: osdmap e466: 12 total, 12 up, 12 in 2022-09-10T07:42:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:09 smithi104 ceph-mon[97044]: pgmap v6173: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:42:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:09 smithi104 ceph-mon[97044]: osdmap e466: 12 total, 12 up, 12 in 2022-09-10T07:42:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:09.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:09.293597+0000) 2022-09-10T07:42:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:09.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:09.658513+0000) 2022-09-10T07:42:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:10.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:10.293724+0000) 2022-09-10T07:42:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:10.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:10.658710+0000) 2022-09-10T07:42:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:11 smithi110 ceph-mon[97633]: pgmap v6175: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:42:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:11 smithi104 ceph-mon[97044]: pgmap v6175: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:42:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:11.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:11.293961+0000) 2022-09-10T07:42:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:11.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:11.658869+0000) 2022-09-10T07:42:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:12.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:12.294175+0000) 2022-09-10T07:42:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:12.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:12.659003+0000) 2022-09-10T07:42:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:13 smithi110 ceph-mon[97633]: pgmap v6176: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:42:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:13 smithi104 ceph-mon[97044]: pgmap v6176: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:42:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:13.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:13.294389+0000) 2022-09-10T07:42:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:13.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:13.659209+0000) 2022-09-10T07:42:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:14.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:14.294559+0000) 2022-09-10T07:42:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:14.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:14.659394+0000) 2022-09-10T07:42:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:15 smithi110 ceph-mon[97633]: pgmap v6177: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:42:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:15 smithi110 ceph-mon[97633]: osdmap e467: 12 total, 12 up, 12 in 2022-09-10T07:42:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:15 smithi104 ceph-mon[97044]: pgmap v6177: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:42:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:15 smithi104 ceph-mon[97044]: osdmap e467: 12 total, 12 up, 12 in 2022-09-10T07:42:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:15.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:15.294721+0000) 2022-09-10T07:42:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:15.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:15.659652+0000) 2022-09-10T07:42:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:16.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:16.294890+0000) 2022-09-10T07:42:17.034 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:16.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:16.659849+0000) 2022-09-10T07:42:17.295 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:42:17.030+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:42:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:17 smithi110 ceph-mon[97633]: pgmap v6179: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:42:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:17 smithi104 ceph-mon[97044]: pgmap v6179: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:42:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:17.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:17.295071+0000) 2022-09-10T07:42:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:17.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:17.660055+0000) 2022-09-10T07:42:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:18.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:18.295323+0000) 2022-09-10T07:42:19.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:18.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:18.660284+0000) 2022-09-10T07:42:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:19 smithi110 ceph-mon[97633]: pgmap v6180: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T07:42:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:19 smithi104 ceph-mon[97044]: pgmap v6180: 129 pgs: 129 active+clean; 8.3 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T07:42:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:19.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:19.295523+0000) 2022-09-10T07:42:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:19.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:19.660469+0000) 2022-09-10T07:42:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:20.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:20.295789+0000) 2022-09-10T07:42:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:20.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:20.660638+0000) 2022-09-10T07:42:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:21 smithi110 ceph-mon[97633]: pgmap v6181: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:42:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:21 smithi104 ceph-mon[97044]: pgmap v6181: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:42:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:21.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:21.295974+0000) 2022-09-10T07:42:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:21.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:21.660879+0000) 2022-09-10T07:42:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:22.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:22.296229+0000) 2022-09-10T07:42:23.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:22.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:22.661124+0000) 2022-09-10T07:42:23.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:23 smithi110 ceph-mon[97633]: pgmap v6182: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:42:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:23 smithi104 ceph-mon[97044]: pgmap v6182: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:42:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:23.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:23.296456+0000) 2022-09-10T07:42:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:23.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:23.661316+0000) 2022-09-10T07:42:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:24.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:24.296644+0000) 2022-09-10T07:42:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:24.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:24.661555+0000) 2022-09-10T07:42:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:25 smithi110 ceph-mon[97633]: pgmap v6183: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:42:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:25 smithi104 ceph-mon[97044]: pgmap v6183: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:42:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:25.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:25.296824+0000) 2022-09-10T07:42:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:25.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:25.661798+0000) 2022-09-10T07:42:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:26.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:26.297028+0000) 2022-09-10T07:42:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:26.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:26.661958+0000) 2022-09-10T07:42:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:27 smithi110 ceph-mon[97633]: pgmap v6184: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:42:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:27 smithi104 ceph-mon[97044]: pgmap v6184: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:42:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:27.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:27.297187+0000) 2022-09-10T07:42:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:27.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:27.662198+0000) 2022-09-10T07:42:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:28.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:28.297425+0000) 2022-09-10T07:42:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:28.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:28.662402+0000) 2022-09-10T07:42:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:29 smithi110 ceph-mon[97633]: pgmap v6185: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:29 smithi104 ceph-mon[97044]: pgmap v6185: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:29.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:29.297612+0000) 2022-09-10T07:42:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:29.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:29.662684+0000) 2022-09-10T07:42:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:30.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:30.297810+0000) 2022-09-10T07:42:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:30.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:30.662907+0000) 2022-09-10T07:42:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:31 smithi110 ceph-mon[97633]: pgmap v6186: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:31 smithi104 ceph-mon[97044]: pgmap v6186: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:31.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:31.298061+0000) 2022-09-10T07:42:32.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:31.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:31.663126+0000) 2022-09-10T07:42:32.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:42:32.045+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:42:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:32.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:32.298299+0000) 2022-09-10T07:42:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:32.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:32.663280+0000) 2022-09-10T07:42:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:33 smithi110 ceph-mon[97633]: pgmap v6187: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:33 smithi104 ceph-mon[97044]: pgmap v6187: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:33.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:33.298529+0000) 2022-09-10T07:42:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:33.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:33.663483+0000) 2022-09-10T07:42:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:34.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:34.298746+0000) 2022-09-10T07:42:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:34.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:34.663732+0000) 2022-09-10T07:42:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:35 smithi110 ceph-mon[97633]: pgmap v6188: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:35 smithi104 ceph-mon[97044]: pgmap v6188: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:35.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:35.299006+0000) 2022-09-10T07:42:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:35.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:35.663932+0000) 2022-09-10T07:42:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:36.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:36.299201+0000) 2022-09-10T07:42:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:36.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:36.664189+0000) 2022-09-10T07:42:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:37 smithi110 ceph-mon[97633]: pgmap v6189: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:37 smithi104 ceph-mon[97044]: pgmap v6189: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:37.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:37.299372+0000) 2022-09-10T07:42:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:37.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:37.664467+0000) 2022-09-10T07:42:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:38.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:38.299570+0000) 2022-09-10T07:42:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:38.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:38.664660+0000) 2022-09-10T07:42:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:39 smithi110 ceph-mon[97633]: pgmap v6190: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:39 smithi104 ceph-mon[97044]: pgmap v6190: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:39.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:39.299761+0000) 2022-09-10T07:42:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:39.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:39.664899+0000) 2022-09-10T07:42:40.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:40.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:40.300013+0000) 2022-09-10T07:42:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:40.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:40.665073+0000) 2022-09-10T07:42:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:41 smithi110 ceph-mon[97633]: pgmap v6191: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:41 smithi104 ceph-mon[97044]: pgmap v6191: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:41.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:41.300249+0000) 2022-09-10T07:42:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:41.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:41.665339+0000) 2022-09-10T07:42:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:42.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:42.300447+0000) 2022-09-10T07:42:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:42.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:42.665590+0000) 2022-09-10T07:42:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:43 smithi110 ceph-mon[97633]: pgmap v6192: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:43 smithi104 ceph-mon[97044]: pgmap v6192: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:43.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:43.300675+0000) 2022-09-10T07:42:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:43.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:43.665830+0000) 2022-09-10T07:42:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:44.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:44.300905+0000) 2022-09-10T07:42:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:44.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:44.666076+0000) 2022-09-10T07:42:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:45 smithi110 ceph-mon[97633]: pgmap v6193: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:42:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:42:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:42:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:42:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:45 smithi104 ceph-mon[97044]: pgmap v6193: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:42:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:42:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:42:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:42:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:45.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:45.301210+0000) 2022-09-10T07:42:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:45.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:45.666353+0000) 2022-09-10T07:42:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:46.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:46.301390+0000) 2022-09-10T07:42:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:46.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:46.666602+0000) 2022-09-10T07:42:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:47 smithi110 ceph-mon[97633]: pgmap v6194: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:47 smithi104 ceph-mon[97044]: pgmap v6194: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:47.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:47.301548+0000) 2022-09-10T07:42:47.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:42:47.061+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:42:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:47.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:47.666874+0000) 2022-09-10T07:42:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:48.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:48.301691+0000) 2022-09-10T07:42:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:48.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:48.667078+0000) 2022-09-10T07:42:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:49 smithi110 ceph-mon[97633]: pgmap v6195: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:49 smithi104 ceph-mon[97044]: pgmap v6195: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:49.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:49.301856+0000) 2022-09-10T07:42:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:49.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:49.667320+0000) 2022-09-10T07:42:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:50.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:50.302077+0000) 2022-09-10T07:42:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:50.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:50.667505+0000) 2022-09-10T07:42:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:51 smithi110 ceph-mon[97633]: pgmap v6196: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:51 smithi104 ceph-mon[97044]: pgmap v6196: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:51.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:51.302372+0000) 2022-09-10T07:42:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:51.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:51.667752+0000) 2022-09-10T07:42:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:52.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:52.302595+0000) 2022-09-10T07:42:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:52.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:52.667947+0000) 2022-09-10T07:42:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:53 smithi110 ceph-mon[97633]: pgmap v6197: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:53 smithi104 ceph-mon[97044]: pgmap v6197: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:53.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:53.302808+0000) 2022-09-10T07:42:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:53.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:53.668162+0000) 2022-09-10T07:42:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:54.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:54.302978+0000) 2022-09-10T07:42:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:54.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:54.668405+0000) 2022-09-10T07:42:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:55 smithi110 ceph-mon[97633]: pgmap v6198: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:55 smithi104 ceph-mon[97044]: pgmap v6198: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:55.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:55.303176+0000) 2022-09-10T07:42:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:55.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:55.668643+0000) 2022-09-10T07:42:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:56.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:56.303375+0000) 2022-09-10T07:42:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:56.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:56.668840+0000) 2022-09-10T07:42:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:57 smithi110 ceph-mon[97633]: pgmap v6199: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:57 smithi104 ceph-mon[97044]: pgmap v6199: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:42:57.669 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:57.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:57.303558+0000) 2022-09-10T07:42:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:57.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:57.669021+0000) 2022-09-10T07:42:58.669 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:58.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:58.303798+0000) 2022-09-10T07:42:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:58.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:58.669206+0000) 2022-09-10T07:42:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:42:59 smithi110 ceph-mon[97633]: pgmap v6200: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:42:59 smithi104 ceph-mon[97044]: pgmap v6200: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:42:59.669 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:42:59.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:59.303998+0000) 2022-09-10T07:43:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:42:59.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:42:59.669398+0000) 2022-09-10T07:43:00.670 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:00.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:00.304186+0000) 2022-09-10T07:43:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:00.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:00.669545+0000) 2022-09-10T07:43:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:01 smithi110 ceph-mon[97633]: pgmap v6201: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:01 smithi104 ceph-mon[97044]: pgmap v6201: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:01.670 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:01.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:01.304431+0000) 2022-09-10T07:43:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:01.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:01.669727+0000) 2022-09-10T07:43:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:02.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:02.304644+0000) 2022-09-10T07:43:02.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:43:02.076+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:43:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:02.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:02.669921+0000) 2022-09-10T07:43:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:03 smithi110 ceph-mon[97633]: pgmap v6202: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:03 smithi104 ceph-mon[97044]: pgmap v6202: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:03.671 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:03.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:03.304815+0000) 2022-09-10T07:43:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:03.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:03.670187+0000) 2022-09-10T07:43:04.671 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:04.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:04.305041+0000) 2022-09-10T07:43:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:04.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:04.670430+0000) 2022-09-10T07:43:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:05 smithi110 ceph-mon[97633]: pgmap v6203: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:05 smithi104 ceph-mon[97044]: pgmap v6203: 129 pgs: 129 active+clean; 8.3 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:05.671 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:05.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:05.305275+0000) 2022-09-10T07:43:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:05.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:05.670602+0000) 2022-09-10T07:43:06.672 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:06.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:06.305486+0000) 2022-09-10T07:43:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:06.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:06.670785+0000) 2022-09-10T07:43:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:07 smithi110 ceph-mon[97633]: pgmap v6204: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:43:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:07 smithi104 ceph-mon[97044]: pgmap v6204: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:43:07.671 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:07.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:07.305649+0000) 2022-09-10T07:43:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:07.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:07.671016+0000) 2022-09-10T07:43:08.672 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:08.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:08.305827+0000) 2022-09-10T07:43:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:08.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:08.671241+0000) 2022-09-10T07:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:09 smithi110 ceph-mon[97633]: pgmap v6205: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:09 smithi110 ceph-mon[97633]: osdmap e468: 12 total, 12 up, 12 in 2022-09-10T07:43:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:09 smithi104 ceph-mon[97044]: pgmap v6205: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:43:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:09 smithi104 ceph-mon[97044]: osdmap e468: 12 total, 12 up, 12 in 2022-09-10T07:43:09.672 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:09.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:09.306007+0000) 2022-09-10T07:43:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:09.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:09.671476+0000) 2022-09-10T07:43:10.672 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:10.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:10.306171+0000) 2022-09-10T07:43:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:10.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:10.671655+0000) 2022-09-10T07:43:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:11 smithi110 ceph-mon[97633]: pgmap v6207: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:43:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:11 smithi104 ceph-mon[97044]: pgmap v6207: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:43:11.672 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:11.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:11.306361+0000) 2022-09-10T07:43:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:11.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:11.671847+0000) 2022-09-10T07:43:12.672 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:12.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:12.306540+0000) 2022-09-10T07:43:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:12.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:12.672086+0000) 2022-09-10T07:43:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:13 smithi110 ceph-mon[97633]: pgmap v6208: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:43:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:13 smithi104 ceph-mon[97044]: pgmap v6208: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:43:13.673 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:13.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:13.306695+0000) 2022-09-10T07:43:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:13.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:13.672334+0000) 2022-09-10T07:43:14.673 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:14.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:14.306895+0000) 2022-09-10T07:43:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:14.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:14.672566+0000) 2022-09-10T07:43:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:15 smithi110 ceph-mon[97633]: pgmap v6209: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:43:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:15 smithi110 ceph-mon[97633]: osdmap e469: 12 total, 12 up, 12 in 2022-09-10T07:43:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:15 smithi104 ceph-mon[97044]: pgmap v6209: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:43:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:15 smithi104 ceph-mon[97044]: osdmap e469: 12 total, 12 up, 12 in 2022-09-10T07:43:15.673 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:15.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:15.307075+0000) 2022-09-10T07:43:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:15.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:15.672777+0000) 2022-09-10T07:43:16.673 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:16.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:16.307269+0000) 2022-09-10T07:43:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:16.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:16.672999+0000) 2022-09-10T07:43:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:17 smithi110 ceph-mon[97633]: pgmap v6211: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:43:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:17 smithi104 ceph-mon[97044]: pgmap v6211: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:43:17.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:43:17.077+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:43:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:17.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:17.307437+0000) 2022-09-10T07:43:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:17.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:17.673206+0000) 2022-09-10T07:43:18.674 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:18.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:18.307616+0000) 2022-09-10T07:43:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:18.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:18.673345+0000) 2022-09-10T07:43:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:19 smithi110 ceph-mon[97633]: pgmap v6212: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T07:43:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:19 smithi104 ceph-mon[97044]: pgmap v6212: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T07:43:19.674 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:19.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:19.307810+0000) 2022-09-10T07:43:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:19.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:19.673571+0000) 2022-09-10T07:43:20.674 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:20.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:20.308061+0000) 2022-09-10T07:43:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:20.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:20.673753+0000) 2022-09-10T07:43:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:21 smithi104 ceph-mon[97044]: pgmap v6213: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:43:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:21 smithi110 ceph-mon[97633]: pgmap v6213: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:43:21.674 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:21.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:21.308293+0000) 2022-09-10T07:43:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:21.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:21.673963+0000) 2022-09-10T07:43:22.674 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:22.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:22.308472+0000) 2022-09-10T07:43:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:22.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:22.674186+0000) 2022-09-10T07:43:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:23 smithi104 ceph-mon[97044]: pgmap v6214: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:43:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:23 smithi110 ceph-mon[97633]: pgmap v6214: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:43:23.674 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:23.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:23.308648+0000) 2022-09-10T07:43:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:23.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:23.674357+0000) 2022-09-10T07:43:24.675 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:24.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:24.308867+0000) 2022-09-10T07:43:25.065 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:24.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:24.674577+0000) 2022-09-10T07:43:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:25 smithi104 ceph-mon[97044]: pgmap v6215: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:43:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:25 smithi110 ceph-mon[97633]: pgmap v6215: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:43:25.675 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:25.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:25.309149+0000) 2022-09-10T07:43:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:25.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:25.674769+0000) 2022-09-10T07:43:26.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:26.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:26.309382+0000) 2022-09-10T07:43:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:26.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:26.674981+0000) 2022-09-10T07:43:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:27 smithi104 ceph-mon[97044]: pgmap v6216: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:43:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:27 smithi110 ceph-mon[97633]: pgmap v6216: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:43:27.675 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:27.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:27.309543+0000) 2022-09-10T07:43:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:27.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:27.675245+0000) 2022-09-10T07:43:28.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:28.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:28.309773+0000) 2022-09-10T07:43:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:28.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:28.675417+0000) 2022-09-10T07:43:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:29 smithi104 ceph-mon[97044]: pgmap v6217: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:29 smithi110 ceph-mon[97633]: pgmap v6217: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:29.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:29.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:29.310031+0000) 2022-09-10T07:43:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:29.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:29.675629+0000) 2022-09-10T07:43:30.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:30.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:30.310247+0000) 2022-09-10T07:43:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:30.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:30.675862+0000) 2022-09-10T07:43:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:31 smithi104 ceph-mon[97044]: pgmap v6218: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:31 smithi110 ceph-mon[97633]: pgmap v6218: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:31.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:31.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:31.310435+0000) 2022-09-10T07:43:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:31.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:31.676017+0000) 2022-09-10T07:43:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:32.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:32.310703+0000) 2022-09-10T07:43:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:43:32.093+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:43:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:32.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:32.676233+0000) 2022-09-10T07:43:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:33 smithi104 ceph-mon[97044]: pgmap v6219: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:33 smithi110 ceph-mon[97633]: pgmap v6219: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:33.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:33.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:33.310955+0000) 2022-09-10T07:43:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:33.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:33.676429+0000) 2022-09-10T07:43:34.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:34.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:34.311160+0000) 2022-09-10T07:43:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:34.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:34.676580+0000) 2022-09-10T07:43:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:35 smithi104 ceph-mon[97044]: pgmap v6220: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:35 smithi110 ceph-mon[97633]: pgmap v6220: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:35.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:35.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:35.311370+0000) 2022-09-10T07:43:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:35.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:35.676821+0000) 2022-09-10T07:43:36.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:36.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:36.311623+0000) 2022-09-10T07:43:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:36.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:36.677070+0000) 2022-09-10T07:43:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:37 smithi104 ceph-mon[97044]: pgmap v6221: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:37 smithi110 ceph-mon[97633]: pgmap v6221: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:37.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:37.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:37.311821+0000) 2022-09-10T07:43:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:37.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:37.677377+0000) 2022-09-10T07:43:38.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:38.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:38.312080+0000) 2022-09-10T07:43:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:38.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:38.677554+0000) 2022-09-10T07:43:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:39 smithi104 ceph-mon[97044]: pgmap v6222: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:39 smithi110 ceph-mon[97633]: pgmap v6222: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:39.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:39.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:39.312282+0000) 2022-09-10T07:43:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:39.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:39.677807+0000) 2022-09-10T07:43:40.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:40.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:40.312481+0000) 2022-09-10T07:43:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:40.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:40.677968+0000) 2022-09-10T07:43:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:41 smithi104 ceph-mon[97044]: pgmap v6223: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:41 smithi110 ceph-mon[97633]: pgmap v6223: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:43:41.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:41.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:41.312676+0000) 2022-09-10T07:43:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:41.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:41.678188+0000) 2022-09-10T07:43:42.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:42.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:42.312847+0000) 2022-09-10T07:43:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:42.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:42.678435+0000) 2022-09-10T07:43:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:43 smithi104 ceph-mon[97044]: pgmap v6224: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:43 smithi110 ceph-mon[97633]: pgmap v6224: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:43.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:43.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:43.312991+0000) 2022-09-10T07:43:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:43.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:43.678654+0000) 2022-09-10T07:43:44.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:44.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:44.313148+0000) 2022-09-10T07:43:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:44.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:44.678887+0000) 2022-09-10T07:43:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:45 smithi104 ceph-mon[97044]: pgmap v6225: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:43:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:43:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:43:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:43:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:45 smithi110 ceph-mon[97633]: pgmap v6225: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:43:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:43:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:43:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:43:45.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:45.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:45.313322+0000) 2022-09-10T07:43:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:45.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:45.679192+0000) 2022-09-10T07:43:46.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:46.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:46.313485+0000) 2022-09-10T07:43:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:46.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:46.679402+0000) 2022-09-10T07:43:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:47 smithi104 ceph-mon[97044]: pgmap v6226: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T07:43:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:47.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:47.313638+0000) 2022-09-10T07:43:47.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:43:47.094+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:43:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:47 smithi110 ceph-mon[97633]: pgmap v6226: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T07:43:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:47.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:47.679555+0000) 2022-09-10T07:43:48.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:48.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:48.313852+0000) 2022-09-10T07:43:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:48.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:48.679769+0000) 2022-09-10T07:43:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:49 smithi104 ceph-mon[97044]: pgmap v6227: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T07:43:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:49 smithi110 ceph-mon[97633]: pgmap v6227: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T07:43:49.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:49.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:49.314061+0000) 2022-09-10T07:43:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:49.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:49.679962+0000) 2022-09-10T07:43:50.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:50.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:50.314232+0000) 2022-09-10T07:43:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:50.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:50.680178+0000) 2022-09-10T07:43:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:51 smithi104 ceph-mon[97044]: pgmap v6228: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T07:43:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:51 smithi110 ceph-mon[97633]: pgmap v6228: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T07:43:51.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:51.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:51.314458+0000) 2022-09-10T07:43:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:51.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:51.680376+0000) 2022-09-10T07:43:52.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:52.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:52.314634+0000) 2022-09-10T07:43:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:52.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:52.680572+0000) 2022-09-10T07:43:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:53 smithi104 ceph-mon[97044]: pgmap v6229: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T07:43:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:53 smithi110 ceph-mon[97633]: pgmap v6229: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T07:43:53.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:53.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:53.314804+0000) 2022-09-10T07:43:54.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:53.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:53.680780+0000) 2022-09-10T07:43:54.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:54.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:54.315045+0000) 2022-09-10T07:43:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:54.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:54.681023+0000) 2022-09-10T07:43:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:55 smithi104 ceph-mon[97044]: pgmap v6230: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T07:43:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:55 smithi110 ceph-mon[97633]: pgmap v6230: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T07:43:55.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:55.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:55.315314+0000) 2022-09-10T07:43:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:55.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:55.681290+0000) 2022-09-10T07:43:56.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:56.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:56.315473+0000) 2022-09-10T07:43:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:56.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:56.681492+0000) 2022-09-10T07:43:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:57 smithi104 ceph-mon[97044]: pgmap v6231: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T07:43:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:57 smithi110 ceph-mon[97633]: pgmap v6231: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T07:43:57.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:57.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:57.315614+0000) 2022-09-10T07:43:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:57.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:57.681680+0000) 2022-09-10T07:43:58.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:58.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:58.315784+0000) 2022-09-10T07:43:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:58.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:58.681878+0000) 2022-09-10T07:43:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:43:59 smithi104 ceph-mon[97044]: pgmap v6232: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:43:59 smithi110 ceph-mon[97633]: pgmap v6232: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:43:59.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:43:59.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:59.316033+0000) 2022-09-10T07:44:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:43:59.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:43:59.682080+0000) 2022-09-10T07:44:00.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:00.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:00.316157+0000) 2022-09-10T07:44:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:00.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:00.682332+0000) 2022-09-10T07:44:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:01 smithi104 ceph-mon[97044]: pgmap v6233: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:01 smithi110 ceph-mon[97633]: pgmap v6233: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:01.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:01.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:01.316293+0000) 2022-09-10T07:44:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:01.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:01.682573+0000) 2022-09-10T07:44:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:44:02.110+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:44:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:02.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:02.316457+0000) 2022-09-10T07:44:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:02.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:02.682766+0000) 2022-09-10T07:44:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:03 smithi104 ceph-mon[97044]: pgmap v6234: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:03 smithi110 ceph-mon[97633]: pgmap v6234: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:03.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:03.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:03.316659+0000) 2022-09-10T07:44:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:03.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:03.682922+0000) 2022-09-10T07:44:04.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:04.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:04.316878+0000) 2022-09-10T07:44:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:04.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:04.683178+0000) 2022-09-10T07:44:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:05 smithi104 ceph-mon[97044]: pgmap v6235: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:05 smithi110 ceph-mon[97633]: pgmap v6235: 129 pgs: 129 active+clean; 8.3 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:05.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:05.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:05.317064+0000) 2022-09-10T07:44:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:05.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:05.683420+0000) 2022-09-10T07:44:06.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:06.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:06.317264+0000) 2022-09-10T07:44:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:06.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:06.683620+0000) 2022-09-10T07:44:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:07 smithi104 ceph-mon[97044]: pgmap v6236: 129 pgs: 129 active+clean; 8.3 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:44:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:07 smithi110 ceph-mon[97633]: pgmap v6236: 129 pgs: 129 active+clean; 8.3 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:44:07.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:07.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:07.317461+0000) 2022-09-10T07:44:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:07.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:07.683790+0000) 2022-09-10T07:44:08.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:08.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:08.317664+0000) 2022-09-10T07:44:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:08.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:08.684020+0000) 2022-09-10T07:44:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:09 smithi104 ceph-mon[97044]: pgmap v6237: 129 pgs: 129 active+clean; 8.3 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:44:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:09 smithi104 ceph-mon[97044]: osdmap e470: 12 total, 12 up, 12 in 2022-09-10T07:44:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:09 smithi110 ceph-mon[97633]: pgmap v6237: 129 pgs: 129 active+clean; 8.3 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:44:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:09 smithi110 ceph-mon[97633]: osdmap e470: 12 total, 12 up, 12 in 2022-09-10T07:44:09.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:09.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:09.317909+0000) 2022-09-10T07:44:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:09.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:09.684271+0000) 2022-09-10T07:44:10.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:10.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:10.318180+0000) 2022-09-10T07:44:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:10.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:10.684521+0000) 2022-09-10T07:44:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:11 smithi104 ceph-mon[97044]: pgmap v6239: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:44:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:11 smithi110 ceph-mon[97633]: pgmap v6239: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:44:11.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:11.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:11.318346+0000) 2022-09-10T07:44:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:11.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:11.684846+0000) 2022-09-10T07:44:12.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:12.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:12.318547+0000) 2022-09-10T07:44:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:12.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:12.685163+0000) 2022-09-10T07:44:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:13 smithi104 ceph-mon[97044]: pgmap v6240: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:44:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:13 smithi110 ceph-mon[97633]: pgmap v6240: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:44:13.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:13.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:13.318700+0000) 2022-09-10T07:44:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:13.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:13.685349+0000) 2022-09-10T07:44:14.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:14.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:14.318856+0000) 2022-09-10T07:44:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:14.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:14.685641+0000) 2022-09-10T07:44:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:15 smithi104 ceph-mon[97044]: pgmap v6241: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:44:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:15 smithi104 ceph-mon[97044]: osdmap e471: 12 total, 12 up, 12 in 2022-09-10T07:44:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:15 smithi110 ceph-mon[97633]: pgmap v6241: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:44:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:15 smithi110 ceph-mon[97633]: osdmap e471: 12 total, 12 up, 12 in 2022-09-10T07:44:15.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:15.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:15.319125+0000) 2022-09-10T07:44:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:15.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:15.685887+0000) 2022-09-10T07:44:16.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:16.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:16.319357+0000) 2022-09-10T07:44:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:16.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:16.686084+0000) 2022-09-10T07:44:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:17 smithi104 ceph-mon[97044]: pgmap v6243: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:44:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:17.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:17.319567+0000) 2022-09-10T07:44:17.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:44:17.125+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:44:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:17 smithi110 ceph-mon[97633]: pgmap v6243: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:44:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:17.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:17.686303+0000) 2022-09-10T07:44:18.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:18.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:18.319821+0000) 2022-09-10T07:44:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:18.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:18.686510+0000) 2022-09-10T07:44:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:19 smithi104 ceph-mon[97044]: pgmap v6244: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T07:44:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:19 smithi110 ceph-mon[97633]: pgmap v6244: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T07:44:19.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:19.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:19.320006+0000) 2022-09-10T07:44:20.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:19.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:19.686733+0000) 2022-09-10T07:44:20.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:20.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:20.320260+0000) 2022-09-10T07:44:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:20.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:20.686981+0000) 2022-09-10T07:44:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:21 smithi104 ceph-mon[97044]: pgmap v6245: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:44:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:21 smithi110 ceph-mon[97633]: pgmap v6245: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:44:21.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:21.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:21.320527+0000) 2022-09-10T07:44:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:21.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:21.687236+0000) 2022-09-10T07:44:22.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:22.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:22.320731+0000) 2022-09-10T07:44:23.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:22.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:22.687450+0000) 2022-09-10T07:44:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:23 smithi104 ceph-mon[97044]: pgmap v6246: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:44:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:23 smithi110 ceph-mon[97633]: pgmap v6246: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:44:23.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:23.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:23.320940+0000) 2022-09-10T07:44:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:23.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:23.687652+0000) 2022-09-10T07:44:24.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:24.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:24.321202+0000) 2022-09-10T07:44:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:24.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:24.687838+0000) 2022-09-10T07:44:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:25 smithi104 ceph-mon[97044]: pgmap v6247: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:44:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:25 smithi110 ceph-mon[97633]: pgmap v6247: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:44:25.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:25.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:25.321413+0000) 2022-09-10T07:44:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:25.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:25.688031+0000) 2022-09-10T07:44:26.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:26.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:26.321631+0000) 2022-09-10T07:44:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:26.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:26.688204+0000) 2022-09-10T07:44:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:27 smithi104 ceph-mon[97044]: pgmap v6248: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:44:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:27 smithi110 ceph-mon[97633]: pgmap v6248: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:44:27.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:27.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:27.321865+0000) 2022-09-10T07:44:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:27.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:27.688395+0000) 2022-09-10T07:44:28.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:28.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:28.322049+0000) 2022-09-10T07:44:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:28.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:28.688618+0000) 2022-09-10T07:44:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:29 smithi104 ceph-mon[97044]: pgmap v6249: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:29 smithi110 ceph-mon[97633]: pgmap v6249: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:29.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:29.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:29.322288+0000) 2022-09-10T07:44:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:29.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:29.688848+0000) 2022-09-10T07:44:30.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:30.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:30.322433+0000) 2022-09-10T07:44:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:30.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:30.689054+0000) 2022-09-10T07:44:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:31 smithi104 ceph-mon[97044]: pgmap v6250: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:31 smithi110 ceph-mon[97633]: pgmap v6250: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:31.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:31.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:31.322593+0000) 2022-09-10T07:44:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:31.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:31.689274+0000) 2022-09-10T07:44:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:32.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:32.322835+0000) 2022-09-10T07:44:32.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:44:32.140+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:44:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:32.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:32.689528+0000) 2022-09-10T07:44:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:33 smithi104 ceph-mon[97044]: pgmap v6251: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:44:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:33 smithi110 ceph-mon[97633]: pgmap v6251: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:44:33.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:33.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:33.323039+0000) 2022-09-10T07:44:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:33.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:33.689817+0000) 2022-09-10T07:44:34.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:34.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:34.323275+0000) 2022-09-10T07:44:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:34.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:34.690057+0000) 2022-09-10T07:44:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:35 smithi104 ceph-mon[97044]: pgmap v6252: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:44:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:35 smithi110 ceph-mon[97633]: pgmap v6252: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:44:35.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:35.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:35.323428+0000) 2022-09-10T07:44:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:35.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:35.690319+0000) 2022-09-10T07:44:36.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:36.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:36.323656+0000) 2022-09-10T07:44:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:36.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:36.690585+0000) 2022-09-10T07:44:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:37 smithi104 ceph-mon[97044]: pgmap v6253: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:37 smithi110 ceph-mon[97633]: pgmap v6253: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:37.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:37.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:37.323905+0000) 2022-09-10T07:44:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:37.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:37.690764+0000) 2022-09-10T07:44:38.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:38.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:38.324124+0000) 2022-09-10T07:44:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:38.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:38.690979+0000) 2022-09-10T07:44:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:39 smithi104 ceph-mon[97044]: pgmap v6254: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:44:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:39 smithi110 ceph-mon[97633]: pgmap v6254: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:44:39.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:39.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:39.324376+0000) 2022-09-10T07:44:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:39.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:39.691207+0000) 2022-09-10T07:44:40.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:40.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:40.324502+0000) 2022-09-10T07:44:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:40.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:40.691438+0000) 2022-09-10T07:44:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:41 smithi104 ceph-mon[97044]: pgmap v6255: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:41 smithi110 ceph-mon[97633]: pgmap v6255: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:41.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:41.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:41.324765+0000) 2022-09-10T07:44:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:41.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:41.691606+0000) 2022-09-10T07:44:42.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:42.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:42.325017+0000) 2022-09-10T07:44:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:42.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:42.691799+0000) 2022-09-10T07:44:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:43 smithi104 ceph-mon[97044]: pgmap v6256: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:43 smithi110 ceph-mon[97633]: pgmap v6256: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:43.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:43.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:43.325217+0000) 2022-09-10T07:44:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:43.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:43.692019+0000) 2022-09-10T07:44:44.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:44.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:44.325361+0000) 2022-09-10T07:44:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:44.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:44.692237+0000) 2022-09-10T07:44:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:45 smithi104 ceph-mon[97044]: pgmap v6257: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:44:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:44:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:44:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:44:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:45 smithi110 ceph-mon[97633]: pgmap v6257: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:44:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:44:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:44:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:44:45.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:45.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:45.325619+0000) 2022-09-10T07:44:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:45.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:45.692425+0000) 2022-09-10T07:44:46.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:46.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:46.325866+0000) 2022-09-10T07:44:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:46.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:46.692671+0000) 2022-09-10T07:44:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:47 smithi104 ceph-mon[97044]: pgmap v6258: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:47.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:47.326076+0000) 2022-09-10T07:44:47.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:44:47.156+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:44:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:47 smithi110 ceph-mon[97633]: pgmap v6258: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:47.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:47.692858+0000) 2022-09-10T07:44:48.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:48.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:48.326322+0000) 2022-09-10T07:44:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:48.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:48.693084+0000) 2022-09-10T07:44:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:49 smithi104 ceph-mon[97044]: pgmap v6259: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:49 smithi110 ceph-mon[97633]: pgmap v6259: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:49.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:49.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:49.326564+0000) 2022-09-10T07:44:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:49.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:49.693373+0000) 2022-09-10T07:44:50.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:50.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:50.326700+0000) 2022-09-10T07:44:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:50.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:50.693621+0000) 2022-09-10T07:44:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:51 smithi104 ceph-mon[97044]: pgmap v6260: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:51 smithi110 ceph-mon[97633]: pgmap v6260: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:51.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:51.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:51.326905+0000) 2022-09-10T07:44:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:51.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:51.693870+0000) 2022-09-10T07:44:52.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:52.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:52.327064+0000) 2022-09-10T07:44:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:52.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:52.694075+0000) 2022-09-10T07:44:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:53 smithi104 ceph-mon[97044]: pgmap v6261: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:53 smithi110 ceph-mon[97633]: pgmap v6261: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:53.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:53.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:53.327238+0000) 2022-09-10T07:44:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:53.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:53.694328+0000) 2022-09-10T07:44:54.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:54.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:54.327469+0000) 2022-09-10T07:44:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:54.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:54.694534+0000) 2022-09-10T07:44:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:55 smithi104 ceph-mon[97044]: pgmap v6262: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:55 smithi110 ceph-mon[97633]: pgmap v6262: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:55.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:55.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:55.327653+0000) 2022-09-10T07:44:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:55.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:55.694754+0000) 2022-09-10T07:44:56.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:56.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:56.327878+0000) 2022-09-10T07:44:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:56.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:56.695007+0000) 2022-09-10T07:44:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:57 smithi110 ceph-mon[97633]: pgmap v6263: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:57.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:57.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:57.328109+0000) 2022-09-10T07:44:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:57 smithi104 ceph-mon[97044]: pgmap v6263: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:44:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:57.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:57.695239+0000) 2022-09-10T07:44:58.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:58.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:58.328283+0000) 2022-09-10T07:44:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:58.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:58.695425+0000) 2022-09-10T07:44:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:44:59 smithi110 ceph-mon[97633]: pgmap v6264: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:44:59.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:44:59.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:59.328523+0000) 2022-09-10T07:44:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:44:59 smithi104 ceph-mon[97044]: pgmap v6264: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:44:59.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:44:59.695662+0000) 2022-09-10T07:45:00.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:00.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:00.328696+0000) 2022-09-10T07:45:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:00.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:00.695816+0000) 2022-09-10T07:45:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:01 smithi110 ceph-mon[97633]: pgmap v6265: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:01.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:01.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:01.328892+0000) 2022-09-10T07:45:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:01 smithi104 ceph-mon[97044]: pgmap v6265: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:01.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:01.696055+0000) 2022-09-10T07:45:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:02.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:02.329019+0000) 2022-09-10T07:45:02.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:45:02.164+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:45:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:02.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:02.696371+0000) 2022-09-10T07:45:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:03 smithi110 ceph-mon[97633]: pgmap v6266: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:03.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:03.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:03.329267+0000) 2022-09-10T07:45:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:03 smithi104 ceph-mon[97044]: pgmap v6266: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:03.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:03.696638+0000) 2022-09-10T07:45:04.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:04.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:04.329470+0000) 2022-09-10T07:45:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:04.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:04.696843+0000) 2022-09-10T07:45:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:05 smithi110 ceph-mon[97633]: pgmap v6267: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:05.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:05.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:05.329689+0000) 2022-09-10T07:45:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:05 smithi104 ceph-mon[97044]: pgmap v6267: 129 pgs: 129 active+clean; 8.3 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:05.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:05.697041+0000) 2022-09-10T07:45:06.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:06.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:06.329931+0000) 2022-09-10T07:45:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:06.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:06.697280+0000) 2022-09-10T07:45:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:07 smithi110 ceph-mon[97633]: pgmap v6268: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:45:07.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:07.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:07.330191+0000) 2022-09-10T07:45:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:07 smithi104 ceph-mon[97044]: pgmap v6268: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:45:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:07.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:07.697507+0000) 2022-09-10T07:45:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:08 smithi110 ceph-mon[97633]: osdmap e472: 12 total, 12 up, 12 in 2022-09-10T07:45:08.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:08.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:08.330447+0000) 2022-09-10T07:45:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:08 smithi104 ceph-mon[97044]: osdmap e472: 12 total, 12 up, 12 in 2022-09-10T07:45:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:08.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:08.697755+0000) 2022-09-10T07:45:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:09 smithi110 ceph-mon[97633]: pgmap v6270: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:09.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:09.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:09.330710+0000) 2022-09-10T07:45:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:09 smithi104 ceph-mon[97044]: pgmap v6270: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:09.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:09.697993+0000) 2022-09-10T07:45:10.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:10.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:10.330956+0000) 2022-09-10T07:45:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:10.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:10.698243+0000) 2022-09-10T07:45:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:11 smithi110 ceph-mon[97633]: pgmap v6271: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:11.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:11.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:11.331246+0000) 2022-09-10T07:45:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:11 smithi104 ceph-mon[97044]: pgmap v6271: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:11.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:11.698419+0000) 2022-09-10T07:45:12.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:12.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:12.331471+0000) 2022-09-10T07:45:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:12.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:12.698642+0000) 2022-09-10T07:45:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:13 smithi110 ceph-mon[97633]: pgmap v6272: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:13.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:13.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:13.331734+0000) 2022-09-10T07:45:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:13 smithi104 ceph-mon[97044]: pgmap v6272: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:13.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:13.698903+0000) 2022-09-10T07:45:14.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:14.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:14.331965+0000) 2022-09-10T07:45:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:14.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:14.699138+0000) 2022-09-10T07:45:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:15 smithi110 ceph-mon[97633]: pgmap v6273: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:15 smithi110 ceph-mon[97633]: osdmap e473: 12 total, 12 up, 12 in 2022-09-10T07:45:15.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:15.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:15.332226+0000) 2022-09-10T07:45:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:15 smithi104 ceph-mon[97044]: pgmap v6273: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:45:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:15 smithi104 ceph-mon[97044]: osdmap e473: 12 total, 12 up, 12 in 2022-09-10T07:45:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:15.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:15.699361+0000) 2022-09-10T07:45:16.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:16.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:16.332431+0000) 2022-09-10T07:45:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:16.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:16.699610+0000) 2022-09-10T07:45:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:17.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:17.332653+0000) 2022-09-10T07:45:17.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:45:17.179+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:45:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:17 smithi110 ceph-mon[97633]: pgmap v6275: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T07:45:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:17 smithi104 ceph-mon[97044]: pgmap v6275: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T07:45:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:17.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:17.699855+0000) 2022-09-10T07:45:18.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:18.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:18.332922+0000) 2022-09-10T07:45:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:18.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:18.700137+0000) 2022-09-10T07:45:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:19 smithi110 ceph-mon[97633]: pgmap v6276: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:45:19.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:19.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:19.333139+0000) 2022-09-10T07:45:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:19 smithi104 ceph-mon[97044]: pgmap v6276: 129 pgs: 129 active+clean; 8.3 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:45:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:19.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:19.700341+0000) 2022-09-10T07:45:20.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:20.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:20.333316+0000) 2022-09-10T07:45:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:20.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:20.700493+0000) 2022-09-10T07:45:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:21 smithi110 ceph-mon[97633]: pgmap v6277: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:45:21.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:21.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:21.333535+0000) 2022-09-10T07:45:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:21 smithi104 ceph-mon[97044]: pgmap v6277: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:45:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:21.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:21.700657+0000) 2022-09-10T07:45:22.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:22.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:22.333758+0000) 2022-09-10T07:45:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:22.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:22.700922+0000) 2022-09-10T07:45:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:23 smithi110 ceph-mon[97633]: pgmap v6278: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:45:23.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:23.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:23.333974+0000) 2022-09-10T07:45:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:23 smithi104 ceph-mon[97044]: pgmap v6278: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:45:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:23.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:23.701207+0000) 2022-09-10T07:45:24.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:24.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:24.334220+0000) 2022-09-10T07:45:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:24.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:24.701453+0000) 2022-09-10T07:45:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:25 smithi110 ceph-mon[97633]: pgmap v6279: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:45:25.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:25.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:25.334467+0000) 2022-09-10T07:45:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:25 smithi104 ceph-mon[97044]: pgmap v6279: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:45:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:25.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:25.701668+0000) 2022-09-10T07:45:26.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:26.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:26.334749+0000) 2022-09-10T07:45:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:26.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:26.701907+0000) 2022-09-10T07:45:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:27 smithi110 ceph-mon[97633]: pgmap v6280: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:45:27.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:27.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:27.334943+0000) 2022-09-10T07:45:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:27 smithi104 ceph-mon[97044]: pgmap v6280: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:45:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:27.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:27.702132+0000) 2022-09-10T07:45:28.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:28.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:28.335175+0000) 2022-09-10T07:45:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:28.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:28.702349+0000) 2022-09-10T07:45:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:29 smithi110 ceph-mon[97633]: pgmap v6281: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:29.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:29.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:29.335350+0000) 2022-09-10T07:45:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:29 smithi104 ceph-mon[97044]: pgmap v6281: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:29.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:29.702598+0000) 2022-09-10T07:45:30.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:30.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:30.335595+0000) 2022-09-10T07:45:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:30.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:30.702851+0000) 2022-09-10T07:45:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:31 smithi110 ceph-mon[97633]: pgmap v6282: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:31.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:31.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:31.335863+0000) 2022-09-10T07:45:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:31 smithi104 ceph-mon[97044]: pgmap v6282: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:31.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:31.703046+0000) 2022-09-10T07:45:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:45:32.193+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:45:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:32.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:32.336075+0000) 2022-09-10T07:45:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:32.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:32.703291+0000) 2022-09-10T07:45:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:33 smithi110 ceph-mon[97633]: pgmap v6283: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:33.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:33.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:33.336316+0000) 2022-09-10T07:45:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:33 smithi104 ceph-mon[97044]: pgmap v6283: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:33.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:33.703490+0000) 2022-09-10T07:45:34.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:34.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:34.336559+0000) 2022-09-10T07:45:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:34.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:34.703685+0000) 2022-09-10T07:45:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:35 smithi110 ceph-mon[97633]: pgmap v6284: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:35.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:35.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:35.336818+0000) 2022-09-10T07:45:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:35 smithi104 ceph-mon[97044]: pgmap v6284: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:35.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:35.703901+0000) 2022-09-10T07:45:36.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:36.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:36.337058+0000) 2022-09-10T07:45:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:36.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:36.704194+0000) 2022-09-10T07:45:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:37 smithi110 ceph-mon[97633]: pgmap v6285: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:37.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:37.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:37.337251+0000) 2022-09-10T07:45:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:37 smithi104 ceph-mon[97044]: pgmap v6285: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:37.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:37.704445+0000) 2022-09-10T07:45:38.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:38.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:38.337458+0000) 2022-09-10T07:45:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:38.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:38.704752+0000) 2022-09-10T07:45:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:39 smithi110 ceph-mon[97633]: pgmap v6286: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:39.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:39.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:39.337679+0000) 2022-09-10T07:45:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:39 smithi104 ceph-mon[97044]: pgmap v6286: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:39.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:39.705017+0000) 2022-09-10T07:45:40.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:40.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:40.337925+0000) 2022-09-10T07:45:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:40.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:40.705206+0000) 2022-09-10T07:45:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:41 smithi110 ceph-mon[97633]: pgmap v6287: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:41.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:41.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:41.338198+0000) 2022-09-10T07:45:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:41 smithi104 ceph-mon[97044]: pgmap v6287: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:41.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:41.705486+0000) 2022-09-10T07:45:42.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:42.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:42.338446+0000) 2022-09-10T07:45:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:42.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:42.705755+0000) 2022-09-10T07:45:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:43 smithi110 ceph-mon[97633]: pgmap v6288: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:43.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:43.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:43.338644+0000) 2022-09-10T07:45:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:43 smithi104 ceph-mon[97044]: pgmap v6288: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:43.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:43.706032+0000) 2022-09-10T07:45:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:45:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:45:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:45:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:45:44.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:44.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:44.338804+0000) 2022-09-10T07:45:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:45:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:45:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:45:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:45:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:44.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:44.706326+0000) 2022-09-10T07:45:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:45 smithi110 ceph-mon[97633]: pgmap v6289: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:45.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:45.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:45.339037+0000) 2022-09-10T07:45:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:45 smithi104 ceph-mon[97044]: pgmap v6289: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:45.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:45.706597+0000) 2022-09-10T07:45:46.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:46.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:46.339279+0000) 2022-09-10T07:45:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:46.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:46.706885+0000) 2022-09-10T07:45:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:47.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:47.339475+0000) 2022-09-10T07:45:47.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:45:47.198+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:45:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:47 smithi110 ceph-mon[97633]: pgmap v6290: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:47 smithi104 ceph-mon[97044]: pgmap v6290: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:47.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:47.707150+0000) 2022-09-10T07:45:48.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:48.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:48.339702+0000) 2022-09-10T07:45:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:48.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:48.707332+0000) 2022-09-10T07:45:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:49 smithi110 ceph-mon[97633]: pgmap v6291: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:49.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:49.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:49.339942+0000) 2022-09-10T07:45:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:49 smithi104 ceph-mon[97044]: pgmap v6291: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:49.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:49.707574+0000) 2022-09-10T07:45:50.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:50.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:50.340190+0000) 2022-09-10T07:45:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:50.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:50.707831+0000) 2022-09-10T07:45:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:51 smithi110 ceph-mon[97633]: pgmap v6292: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:51.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:51.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:51.340484+0000) 2022-09-10T07:45:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:51 smithi104 ceph-mon[97044]: pgmap v6292: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:51.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:51.708065+0000) 2022-09-10T07:45:52.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:52.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:52.340679+0000) 2022-09-10T07:45:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:52.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:52.708304+0000) 2022-09-10T07:45:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:53 smithi110 ceph-mon[97633]: pgmap v6293: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:53.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:53.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:53.340945+0000) 2022-09-10T07:45:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:53 smithi104 ceph-mon[97044]: pgmap v6293: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:53.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:53.708565+0000) 2022-09-10T07:45:54.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:54.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:54.341219+0000) 2022-09-10T07:45:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:54.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:54.708838+0000) 2022-09-10T07:45:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:55 smithi110 ceph-mon[97633]: pgmap v6294: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:55.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:55.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:55.341468+0000) 2022-09-10T07:45:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:55 smithi104 ceph-mon[97044]: pgmap v6294: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:55.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:55.709153+0000) 2022-09-10T07:45:56.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:56.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:56.341667+0000) 2022-09-10T07:45:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:56.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:56.709301+0000) 2022-09-10T07:45:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:57 smithi110 ceph-mon[97633]: pgmap v6295: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:57.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:57.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:57.341908+0000) 2022-09-10T07:45:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:57 smithi104 ceph-mon[97044]: pgmap v6295: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:45:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:57.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:57.709444+0000) 2022-09-10T07:45:58.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:58.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:58.342146+0000) 2022-09-10T07:45:59.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:58.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:58.709675+0000) 2022-09-10T07:45:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:45:59 smithi110 ceph-mon[97633]: pgmap v6296: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:45:59.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:45:59.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:59.342397+0000) 2022-09-10T07:45:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:45:59 smithi104 ceph-mon[97044]: pgmap v6296: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:45:59.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:45:59.709929+0000) 2022-09-10T07:46:00.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:00.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:00.342663+0000) 2022-09-10T07:46:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:00.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:00.710073+0000) 2022-09-10T07:46:01.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:01.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:01.342894+0000) 2022-09-10T07:46:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:01 smithi104 ceph-mon[97044]: pgmap v6297: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:01 smithi110 ceph-mon[97633]: pgmap v6297: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:01.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:01.710289+0000) 2022-09-10T07:46:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:46:02.200+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:46:02.565 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:02.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:02.343133+0000) 2022-09-10T07:46:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:02.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:02.710452+0000) 2022-09-10T07:46:03.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:03.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:03.343352+0000) 2022-09-10T07:46:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:03 smithi104 ceph-mon[97044]: pgmap v6298: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:03 smithi110 ceph-mon[97633]: pgmap v6298: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:03.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:03.710675+0000) 2022-09-10T07:46:04.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:04.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:04.343592+0000) 2022-09-10T07:46:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:04.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:04.710882+0000) 2022-09-10T07:46:05.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:05.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:05.343852+0000) 2022-09-10T07:46:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:05 smithi104 ceph-mon[97044]: pgmap v6299: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:05 smithi110 ceph-mon[97633]: pgmap v6299: 129 pgs: 129 active+clean; 8.3 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:05.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:05.711127+0000) 2022-09-10T07:46:06.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:06.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:06.344119+0000) 2022-09-10T07:46:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:06.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:06.711275+0000) 2022-09-10T07:46:07.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:07.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:07.344331+0000) 2022-09-10T07:46:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:07 smithi104 ceph-mon[97044]: pgmap v6300: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:46:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:07 smithi110 ceph-mon[97633]: pgmap v6300: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:46:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:07.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:07.711429+0000) 2022-09-10T07:46:08.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:08.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:08.344570+0000) 2022-09-10T07:46:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:08 smithi104 ceph-mon[97044]: osdmap e474: 12 total, 12 up, 12 in 2022-09-10T07:46:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:08 smithi110 ceph-mon[97633]: osdmap e474: 12 total, 12 up, 12 in 2022-09-10T07:46:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:08.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:08.711656+0000) 2022-09-10T07:46:09.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:09.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:09.344806+0000) 2022-09-10T07:46:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:09 smithi104 ceph-mon[97044]: pgmap v6302: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:09 smithi110 ceph-mon[97633]: pgmap v6302: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:09.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:09.711873+0000) 2022-09-10T07:46:10.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:10.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:10.345039+0000) 2022-09-10T07:46:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:10.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:10.712068+0000) 2022-09-10T07:46:11.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:11.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:11.345293+0000) 2022-09-10T07:46:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:11 smithi104 ceph-mon[97044]: pgmap v6303: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:11 smithi110 ceph-mon[97633]: pgmap v6303: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:11.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:11.712304+0000) 2022-09-10T07:46:12.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:12.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:12.345532+0000) 2022-09-10T07:46:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:12.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:12.712483+0000) 2022-09-10T07:46:13.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:13.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:13.345709+0000) 2022-09-10T07:46:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:13 smithi104 ceph-mon[97044]: pgmap v6304: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:13 smithi110 ceph-mon[97633]: pgmap v6304: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:13.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:13.712695+0000) 2022-09-10T07:46:14.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:14.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:14.345849+0000) 2022-09-10T07:46:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:14.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:14.712906+0000) 2022-09-10T07:46:15.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:15.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:15.346043+0000) 2022-09-10T07:46:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:15 smithi104 ceph-mon[97044]: pgmap v6305: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:15 smithi104 ceph-mon[97044]: osdmap e475: 12 total, 12 up, 12 in 2022-09-10T07:46:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:15 smithi110 ceph-mon[97633]: pgmap v6305: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:46:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:15 smithi110 ceph-mon[97633]: osdmap e475: 12 total, 12 up, 12 in 2022-09-10T07:46:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:15.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:15.713079+0000) 2022-09-10T07:46:16.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:16.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:16.346281+0000) 2022-09-10T07:46:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:16.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:16.713292+0000) 2022-09-10T07:46:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:17.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:17.346510+0000) 2022-09-10T07:46:17.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:46:17.200+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:46:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:17 smithi104 ceph-mon[97044]: pgmap v6307: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T07:46:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:17 smithi110 ceph-mon[97633]: pgmap v6307: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T07:46:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:17.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:17.713546+0000) 2022-09-10T07:46:18.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:18.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:18.346753+0000) 2022-09-10T07:46:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:18.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:18.713760+0000) 2022-09-10T07:46:19.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:19.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:19.347022+0000) 2022-09-10T07:46:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:19 smithi104 ceph-mon[97044]: pgmap v6308: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:46:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:19 smithi110 ceph-mon[97633]: pgmap v6308: 129 pgs: 129 active+clean; 8.3 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:46:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:19.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:19.713965+0000) 2022-09-10T07:46:20.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:20.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:20.347244+0000) 2022-09-10T07:46:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:20.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:20.714188+0000) 2022-09-10T07:46:21.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:21.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:21.347446+0000) 2022-09-10T07:46:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:21 smithi104 ceph-mon[97044]: pgmap v6309: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:46:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:21 smithi110 ceph-mon[97633]: pgmap v6309: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:46:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:21.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:21.714342+0000) 2022-09-10T07:46:22.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:22.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:22.347586+0000) 2022-09-10T07:46:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:22.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:22.714520+0000) 2022-09-10T07:46:23.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:23.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:23.347730+0000) 2022-09-10T07:46:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:23 smithi104 ceph-mon[97044]: pgmap v6310: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:46:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:23 smithi110 ceph-mon[97633]: pgmap v6310: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:46:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:23.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:23.714758+0000) 2022-09-10T07:46:24.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:24.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:24.347946+0000) 2022-09-10T07:46:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:24.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:24.714984+0000) 2022-09-10T07:46:25.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:25.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:25.348206+0000) 2022-09-10T07:46:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:25 smithi104 ceph-mon[97044]: pgmap v6311: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:46:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:25 smithi110 ceph-mon[97633]: pgmap v6311: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:46:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:25.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:25.715175+0000) 2022-09-10T07:46:26.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:26.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:26.348480+0000) 2022-09-10T07:46:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:26.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:26.715407+0000) 2022-09-10T07:46:27.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:27.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:27.348645+0000) 2022-09-10T07:46:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:27 smithi104 ceph-mon[97044]: pgmap v6312: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:46:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:27 smithi110 ceph-mon[97633]: pgmap v6312: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:46:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:27.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:27.715543+0000) 2022-09-10T07:46:28.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:28.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:28.348850+0000) 2022-09-10T07:46:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:28.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:28.715807+0000) 2022-09-10T07:46:29.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:29.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:29.349142+0000) 2022-09-10T07:46:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:29 smithi104 ceph-mon[97044]: pgmap v6313: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:29.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:29 smithi110 ceph-mon[97633]: pgmap v6313: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:29.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:29.716008+0000) 2022-09-10T07:46:30.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:30.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:30.349373+0000) 2022-09-10T07:46:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:30.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:30.716180+0000) 2022-09-10T07:46:31.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:31.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:31.349561+0000) 2022-09-10T07:46:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:31 smithi104 ceph-mon[97044]: pgmap v6314: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:31 smithi110 ceph-mon[97633]: pgmap v6314: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:31.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:31.716435+0000) 2022-09-10T07:46:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:32.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:32.349762+0000) 2022-09-10T07:46:32.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:46:32.209+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:46:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:32.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:32.716644+0000) 2022-09-10T07:46:33.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:33.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:33.350008+0000) 2022-09-10T07:46:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:33 smithi104 ceph-mon[97044]: pgmap v6315: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:33 smithi110 ceph-mon[97633]: pgmap v6315: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:33.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:33.716888+0000) 2022-09-10T07:46:34.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:34.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:34.350201+0000) 2022-09-10T07:46:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:34.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:34.717188+0000) 2022-09-10T07:46:35.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:35.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:35.350349+0000) 2022-09-10T07:46:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:35 smithi104 ceph-mon[97044]: pgmap v6316: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:35 smithi110 ceph-mon[97633]: pgmap v6316: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:35.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:35.717437+0000) 2022-09-10T07:46:36.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:36.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:36.350532+0000) 2022-09-10T07:46:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:36.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:36.717634+0000) 2022-09-10T07:46:37.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:37.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:37.350764+0000) 2022-09-10T07:46:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:37 smithi104 ceph-mon[97044]: pgmap v6317: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:37 smithi110 ceph-mon[97633]: pgmap v6317: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:37.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:37.717865+0000) 2022-09-10T07:46:38.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:38.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:38.350967+0000) 2022-09-10T07:46:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:38.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:38.718136+0000) 2022-09-10T07:46:39.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:39.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:39.351219+0000) 2022-09-10T07:46:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:39 smithi104 ceph-mon[97044]: pgmap v6318: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:39 smithi110 ceph-mon[97633]: pgmap v6318: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:39.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:39.718392+0000) 2022-09-10T07:46:40.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:40.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:40.351408+0000) 2022-09-10T07:46:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:40.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:40.718583+0000) 2022-09-10T07:46:41.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:41.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:41.351640+0000) 2022-09-10T07:46:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:41 smithi104 ceph-mon[97044]: pgmap v6319: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:41 smithi110 ceph-mon[97633]: pgmap v6319: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:41.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:41.718740+0000) 2022-09-10T07:46:42.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:42.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:42.351886+0000) 2022-09-10T07:46:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:42.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:42.718885+0000) 2022-09-10T07:46:43.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:43.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:43.352085+0000) 2022-09-10T07:46:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:43 smithi104 ceph-mon[97044]: pgmap v6320: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:43 smithi110 ceph-mon[97633]: pgmap v6320: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:43.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:43.719112+0000) 2022-09-10T07:46:44.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:44.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:44.352325+0000) 2022-09-10T07:46:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:46:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:46:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:46:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:46:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:46:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:46:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:46:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:46:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:44.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:44.719261+0000) 2022-09-10T07:46:45.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:45.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:45.352446+0000) 2022-09-10T07:46:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:45 smithi104 ceph-mon[97044]: pgmap v6321: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:45 smithi110 ceph-mon[97633]: pgmap v6321: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:45.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:45.719475+0000) 2022-09-10T07:46:46.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:46.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:46.352641+0000) 2022-09-10T07:46:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:46.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:46.719722+0000) 2022-09-10T07:46:47.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:46:47.216+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:46:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:47.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:47.352839+0000) 2022-09-10T07:46:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:47 smithi104 ceph-mon[97044]: pgmap v6322: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:47 smithi110 ceph-mon[97633]: pgmap v6322: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:47.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:47.719883+0000) 2022-09-10T07:46:48.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:48.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:48.353044+0000) 2022-09-10T07:46:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:48.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:48.720139+0000) 2022-09-10T07:46:49.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:49.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:49.353279+0000) 2022-09-10T07:46:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:49 smithi104 ceph-mon[97044]: pgmap v6323: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:49 smithi110 ceph-mon[97633]: pgmap v6323: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:49.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:49.720301+0000) 2022-09-10T07:46:50.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:50.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:50.353440+0000) 2022-09-10T07:46:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:50.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:50.720482+0000) 2022-09-10T07:46:51.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:51.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:51.353658+0000) 2022-09-10T07:46:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:51 smithi104 ceph-mon[97044]: pgmap v6324: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:51 smithi110 ceph-mon[97633]: pgmap v6324: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:51.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:51.720659+0000) 2022-09-10T07:46:52.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:52.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:52.353888+0000) 2022-09-10T07:46:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:52 smithi104 ceph-mon[97044]: pgmap v6325: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:52 smithi110 ceph-mon[97633]: pgmap v6325: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:52.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:52.720813+0000) 2022-09-10T07:46:53.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:53.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:53.354071+0000) 2022-09-10T07:46:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:53.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:53.720978+0000) 2022-09-10T07:46:54.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:54.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:54.354284+0000) 2022-09-10T07:46:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:54 smithi110 ceph-mon[97633]: pgmap v6326: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:54 smithi104 ceph-mon[97044]: pgmap v6326: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:54.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:54.721189+0000) 2022-09-10T07:46:55.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:55.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:55.354522+0000) 2022-09-10T07:46:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:55.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:55.721464+0000) 2022-09-10T07:46:56.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:56.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:56.354700+0000) 2022-09-10T07:46:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:56.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:56.721719+0000) 2022-09-10T07:46:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:57 smithi110 ceph-mon[97633]: pgmap v6327: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:57 smithi104 ceph-mon[97044]: pgmap v6327: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:46:57.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:57.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:57.354952+0000) 2022-09-10T07:46:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:57.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:57.721945+0000) 2022-09-10T07:46:58.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:58.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:58.355199+0000) 2022-09-10T07:46:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:58.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:58.722135+0000) 2022-09-10T07:46:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:46:59 smithi110 ceph-mon[97633]: pgmap v6328: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:46:59 smithi104 ceph-mon[97044]: pgmap v6328: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:46:59.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:46:59.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:59.355471+0000) 2022-09-10T07:47:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:46:59.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:46:59.722372+0000) 2022-09-10T07:47:00.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:00.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:00.355652+0000) 2022-09-10T07:47:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:00.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:00.722560+0000) 2022-09-10T07:47:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:01 smithi110 ceph-mon[97633]: pgmap v6329: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:01 smithi104 ceph-mon[97044]: pgmap v6329: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:01.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:01.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:01.355839+0000) 2022-09-10T07:47:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:01.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:01.722721+0000) 2022-09-10T07:47:02.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:47:02.224+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:47:02.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:02.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:02.356054+0000) 2022-09-10T07:47:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:02.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:02.722897+0000) 2022-09-10T07:47:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:03 smithi110 ceph-mon[97633]: pgmap v6330: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:03 smithi104 ceph-mon[97044]: pgmap v6330: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:03.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:03.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:03.356248+0000) 2022-09-10T07:47:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:03.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:03.723018+0000) 2022-09-10T07:47:04.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:04.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:04.356444+0000) 2022-09-10T07:47:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:04.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:04.723246+0000) 2022-09-10T07:47:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:05 smithi110 ceph-mon[97633]: pgmap v6331: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:05 smithi104 ceph-mon[97044]: pgmap v6331: 129 pgs: 129 active+clean; 8.3 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:05.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:05.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:05.356686+0000) 2022-09-10T07:47:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:05.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:05.723428+0000) 2022-09-10T07:47:06.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:06.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:06.356946+0000) 2022-09-10T07:47:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:06.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:06.723611+0000) 2022-09-10T07:47:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:07 smithi110 ceph-mon[97633]: pgmap v6332: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:47:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:07 smithi104 ceph-mon[97044]: pgmap v6332: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:47:07.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:07.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:07.357171+0000) 2022-09-10T07:47:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:07.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:07.723844+0000) 2022-09-10T07:47:08.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:08.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:08.357349+0000) 2022-09-10T07:47:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:08.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:08.724053+0000) 2022-09-10T07:47:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:09 smithi110 ceph-mon[97633]: pgmap v6333: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:47:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:09 smithi110 ceph-mon[97633]: osdmap e476: 12 total, 12 up, 12 in 2022-09-10T07:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:09 smithi104 ceph-mon[97044]: pgmap v6333: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:09 smithi104 ceph-mon[97044]: osdmap e476: 12 total, 12 up, 12 in 2022-09-10T07:47:09.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:09.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:09.357578+0000) 2022-09-10T07:47:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:09.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:09.724275+0000) 2022-09-10T07:47:10.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:10.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:10.357839+0000) 2022-09-10T07:47:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:10.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:10.724438+0000) 2022-09-10T07:47:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:11 smithi110 ceph-mon[97633]: pgmap v6335: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:47:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:11 smithi104 ceph-mon[97044]: pgmap v6335: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:47:11.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:11.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:11.358133+0000) 2022-09-10T07:47:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:11.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:11.724660+0000) 2022-09-10T07:47:12.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:12.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:12.358383+0000) 2022-09-10T07:47:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:12.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:12.724900+0000) 2022-09-10T07:47:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:13 smithi110 ceph-mon[97633]: pgmap v6336: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:47:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:13 smithi104 ceph-mon[97044]: pgmap v6336: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:47:13.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:13.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:13.358567+0000) 2022-09-10T07:47:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:13.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:13.725140+0000) 2022-09-10T07:47:14.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:14.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:14.358750+0000) 2022-09-10T07:47:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:14.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:14.725326+0000) 2022-09-10T07:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:15 smithi110 ceph-mon[97633]: pgmap v6337: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:15 smithi110 ceph-mon[97633]: osdmap e477: 12 total, 12 up, 12 in 2022-09-10T07:47:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:15 smithi104 ceph-mon[97044]: pgmap v6337: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:47:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:15 smithi104 ceph-mon[97044]: osdmap e477: 12 total, 12 up, 12 in 2022-09-10T07:47:15.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:15.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:15.358966+0000) 2022-09-10T07:47:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:15.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:15.725530+0000) 2022-09-10T07:47:16.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:16.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:16.359202+0000) 2022-09-10T07:47:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:16.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:16.725754+0000) 2022-09-10T07:47:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:17 smithi110 ceph-mon[97633]: pgmap v6339: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:47:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:17 smithi104 ceph-mon[97044]: pgmap v6339: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:47:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:17.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:17.359407+0000) 2022-09-10T07:47:17.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:47:17.232+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:47:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:17.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:17.726006+0000) 2022-09-10T07:47:18.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:18.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:18.359636+0000) 2022-09-10T07:47:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:18.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:18.726203+0000) 2022-09-10T07:47:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:19 smithi110 ceph-mon[97633]: pgmap v6340: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T07:47:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:19 smithi104 ceph-mon[97044]: pgmap v6340: 129 pgs: 129 active+clean; 8.3 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T07:47:19.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:19.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:19.359854+0000) 2022-09-10T07:47:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:19.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:19.726485+0000) 2022-09-10T07:47:20.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:20.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:20.360136+0000) 2022-09-10T07:47:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:20.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:20.726696+0000) 2022-09-10T07:47:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:21 smithi110 ceph-mon[97633]: pgmap v6341: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:47:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:21 smithi104 ceph-mon[97044]: pgmap v6341: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:47:21.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:21.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:21.360381+0000) 2022-09-10T07:47:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:21.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:21.726924+0000) 2022-09-10T07:47:22.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:22.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:22.360513+0000) 2022-09-10T07:47:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:22.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:22.727156+0000) 2022-09-10T07:47:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:23 smithi110 ceph-mon[97633]: pgmap v6342: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:47:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:23 smithi104 ceph-mon[97044]: pgmap v6342: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:47:23.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:23.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:23.360747+0000) 2022-09-10T07:47:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:23.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:23.727369+0000) 2022-09-10T07:47:24.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:24.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:24.360983+0000) 2022-09-10T07:47:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:24.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:24.727504+0000) 2022-09-10T07:47:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:25 smithi110 ceph-mon[97633]: pgmap v6343: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:47:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:25 smithi104 ceph-mon[97044]: pgmap v6343: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:47:25.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:25.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:25.361212+0000) 2022-09-10T07:47:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:25.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:25.727752+0000) 2022-09-10T07:47:26.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:26.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:26.361421+0000) 2022-09-10T07:47:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:26.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:26.728025+0000) 2022-09-10T07:47:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:27 smithi110 ceph-mon[97633]: pgmap v6344: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:47:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:27 smithi104 ceph-mon[97044]: pgmap v6344: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:47:27.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:27.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:27.361558+0000) 2022-09-10T07:47:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:27.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:27.728266+0000) 2022-09-10T07:47:28.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:28.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:28.361769+0000) 2022-09-10T07:47:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:28.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:28.728480+0000) 2022-09-10T07:47:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:29 smithi110 ceph-mon[97633]: pgmap v6345: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:29 smithi104 ceph-mon[97044]: pgmap v6345: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:29.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:29.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:29.361980+0000) 2022-09-10T07:47:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:29.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:29.728662+0000) 2022-09-10T07:47:30.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:30.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:30.362158+0000) 2022-09-10T07:47:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:30.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:30.728871+0000) 2022-09-10T07:47:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:31 smithi110 ceph-mon[97633]: pgmap v6346: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:31 smithi104 ceph-mon[97044]: pgmap v6346: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:31.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:31.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:31.362405+0000) 2022-09-10T07:47:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:31.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:31.729044+0000) 2022-09-10T07:47:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:32.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:32.362644+0000) 2022-09-10T07:47:32.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:47:32.247+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:47:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:32.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:32.729243+0000) 2022-09-10T07:47:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:33 smithi104 ceph-mon[97044]: pgmap v6347: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:33 smithi110 ceph-mon[97633]: pgmap v6347: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:33.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:33.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:33.362899+0000) 2022-09-10T07:47:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:33.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:33.729397+0000) 2022-09-10T07:47:34.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:34.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:34.363133+0000) 2022-09-10T07:47:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:34.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:34.729644+0000) 2022-09-10T07:47:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:35 smithi104 ceph-mon[97044]: pgmap v6348: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:35 smithi110 ceph-mon[97633]: pgmap v6348: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:35.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:35.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:35.363383+0000) 2022-09-10T07:47:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:35.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:35.729783+0000) 2022-09-10T07:47:36.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:36.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:36.363584+0000) 2022-09-10T07:47:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:36.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:36.730028+0000) 2022-09-10T07:47:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:37 smithi104 ceph-mon[97044]: pgmap v6349: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:37 smithi110 ceph-mon[97633]: pgmap v6349: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:37.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:37.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:37.363844+0000) 2022-09-10T07:47:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:37.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:37.730276+0000) 2022-09-10T07:47:38.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:38.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:38.364075+0000) 2022-09-10T07:47:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:38.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:38.730517+0000) 2022-09-10T07:47:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:39 smithi104 ceph-mon[97044]: pgmap v6350: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:47:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:39 smithi110 ceph-mon[97633]: pgmap v6350: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:47:39.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:39.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:39.364258+0000) 2022-09-10T07:47:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:39.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:39.730779+0000) 2022-09-10T07:47:40.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:40.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:40.364453+0000) 2022-09-10T07:47:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:40.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:40.731016+0000) 2022-09-10T07:47:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:41 smithi104 ceph-mon[97044]: pgmap v6351: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:41 smithi110 ceph-mon[97633]: pgmap v6351: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:41.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:41.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:41.364676+0000) 2022-09-10T07:47:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:41.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:41.731279+0000) 2022-09-10T07:47:42.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:42.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:42.364840+0000) 2022-09-10T07:47:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:42.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:42.731563+0000) 2022-09-10T07:47:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:43 smithi104 ceph-mon[97044]: pgmap v6352: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:47:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:43 smithi110 ceph-mon[97633]: pgmap v6352: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:47:43.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:43.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:43.365074+0000) 2022-09-10T07:47:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:43.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:43.731711+0000) 2022-09-10T07:47:44.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:44.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:44.365306+0000) 2022-09-10T07:47:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:44.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:44.731877+0000) 2022-09-10T07:47:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:45 smithi104 ceph-mon[97044]: pgmap v6353: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:47:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:47:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:47:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:47:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:47:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:45 smithi110 ceph-mon[97633]: pgmap v6353: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:47:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:47:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:47:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:47:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:47:45.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:45.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:45.365554+0000) 2022-09-10T07:47:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:45.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:45.732206+0000) 2022-09-10T07:47:46.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:46.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:46.365792+0000) 2022-09-10T07:47:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:46.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:46.732417+0000) 2022-09-10T07:47:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:47 smithi104 ceph-mon[97044]: pgmap v6354: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:47.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:47:47.264+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:47:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:47.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:47.366019+0000) 2022-09-10T07:47:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:47 smithi110 ceph-mon[97633]: pgmap v6354: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:47.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:47.732688+0000) 2022-09-10T07:47:48.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:48.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:48.366232+0000) 2022-09-10T07:47:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:48.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:48.732865+0000) 2022-09-10T07:47:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:49 smithi104 ceph-mon[97044]: pgmap v6355: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:49 smithi110 ceph-mon[97633]: pgmap v6355: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:49.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:49.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:49.366412+0000) 2022-09-10T07:47:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:49.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:49.733141+0000) 2022-09-10T07:47:50.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:50.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:50.366677+0000) 2022-09-10T07:47:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:50.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:50.733318+0000) 2022-09-10T07:47:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:51 smithi104 ceph-mon[97044]: pgmap v6356: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:51 smithi110 ceph-mon[97633]: pgmap v6356: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:51.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:51.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:51.366906+0000) 2022-09-10T07:47:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:51.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:51.733546+0000) 2022-09-10T07:47:52.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:52.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:52.367164+0000) 2022-09-10T07:47:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:52.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:52.733807+0000) 2022-09-10T07:47:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:53 smithi104 ceph-mon[97044]: pgmap v6357: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:53 smithi110 ceph-mon[97633]: pgmap v6357: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:53.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:53.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:53.367409+0000) 2022-09-10T07:47:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:53.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:53.733940+0000) 2022-09-10T07:47:54.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:54.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:54.367648+0000) 2022-09-10T07:47:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:54.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:54.734188+0000) 2022-09-10T07:47:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:55 smithi104 ceph-mon[97044]: pgmap v6358: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:55 smithi110 ceph-mon[97633]: pgmap v6358: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:55.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:55.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:55.367887+0000) 2022-09-10T07:47:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:55.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:55.734368+0000) 2022-09-10T07:47:56.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:56.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:56.368080+0000) 2022-09-10T07:47:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:56.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:56.734577+0000) 2022-09-10T07:47:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:57 smithi104 ceph-mon[97044]: pgmap v6359: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:57 smithi110 ceph-mon[97633]: pgmap v6359: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:47:57.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:57.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:57.368336+0000) 2022-09-10T07:47:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:57.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:57.734836+0000) 2022-09-10T07:47:58.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:58.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:58.368581+0000) 2022-09-10T07:47:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:58.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:58.735017+0000) 2022-09-10T07:47:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:47:59 smithi104 ceph-mon[97044]: pgmap v6360: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:47:59 smithi110 ceph-mon[97633]: pgmap v6360: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:47:59.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:47:59.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:59.368771+0000) 2022-09-10T07:48:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:47:59.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:47:59.735207+0000) 2022-09-10T07:48:00.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:00.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:00.368947+0000) 2022-09-10T07:48:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:00.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:00.735356+0000) 2022-09-10T07:48:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:01 smithi104 ceph-mon[97044]: pgmap v6361: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:01 smithi110 ceph-mon[97633]: pgmap v6361: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:01.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:01.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:01.369250+0000) 2022-09-10T07:48:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:01.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:01.735571+0000) 2022-09-10T07:48:02.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:48:02.279+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:48:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:02.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:02.369477+0000) 2022-09-10T07:48:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:02.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:02.735760+0000) 2022-09-10T07:48:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:03 smithi104 ceph-mon[97044]: pgmap v6362: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:03 smithi110 ceph-mon[97633]: pgmap v6362: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:03.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:03.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:03.369658+0000) 2022-09-10T07:48:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:03.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:03.735951+0000) 2022-09-10T07:48:04.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:04.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:04.369889+0000) 2022-09-10T07:48:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:04.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:04.736187+0000) 2022-09-10T07:48:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:05 smithi104 ceph-mon[97044]: pgmap v6363: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:05 smithi110 ceph-mon[97633]: pgmap v6363: 129 pgs: 129 active+clean; 8.3 MiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:05.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:05.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:05.370168+0000) 2022-09-10T07:48:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:05.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:05.736393+0000) 2022-09-10T07:48:06.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:06.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:06.370349+0000) 2022-09-10T07:48:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:06.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:06.736676+0000) 2022-09-10T07:48:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:07 smithi104 ceph-mon[97044]: pgmap v6364: 129 pgs: 129 active+clean; 8.3 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:48:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:07 smithi110 ceph-mon[97633]: pgmap v6364: 129 pgs: 129 active+clean; 8.3 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:48:07.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:07.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:07.370585+0000) 2022-09-10T07:48:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:07.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:07.736929+0000) 2022-09-10T07:48:08.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:08.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:08.370834+0000) 2022-09-10T07:48:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:08.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:08.737114+0000) 2022-09-10T07:48:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:09 smithi104 ceph-mon[97044]: pgmap v6365: 129 pgs: 129 active+clean; 8.3 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:48:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:09 smithi104 ceph-mon[97044]: osdmap e478: 12 total, 12 up, 12 in 2022-09-10T07:48:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:09 smithi110 ceph-mon[97633]: pgmap v6365: 129 pgs: 129 active+clean; 8.3 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:48:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:09 smithi110 ceph-mon[97633]: osdmap e478: 12 total, 12 up, 12 in 2022-09-10T07:48:09.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:09.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:09.371082+0000) 2022-09-10T07:48:09.994 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:09.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:09.737265+0000) 2022-09-10T07:48:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:10.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:10.371297+0000) 2022-09-10T07:48:11.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:10.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:10.737419+0000) 2022-09-10T07:48:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:11 smithi104 ceph-mon[97044]: pgmap v6367: 129 pgs: 129 active+clean; 8.3 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:48:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:11 smithi110 ceph-mon[97633]: pgmap v6367: 129 pgs: 129 active+clean; 8.3 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:48:11.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:11.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:11.371459+0000) 2022-09-10T07:48:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:11.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:11.737593+0000) 2022-09-10T07:48:12.668 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:12.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:12.371585+0000) 2022-09-10T07:48:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:12.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:12.737739+0000) 2022-09-10T07:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:13 smithi104 ceph-mon[97044]: pgmap v6368: 129 pgs: 129 active+clean; 8.3 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:48:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:13 smithi110 ceph-mon[97633]: pgmap v6368: 129 pgs: 129 active+clean; 8.3 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:48:13.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:13.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:13.371708+0000) 2022-09-10T07:48:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:13.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:13.737963+0000) 2022-09-10T07:48:14.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:14.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:14.371898+0000) 2022-09-10T07:48:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:14.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:14.738156+0000) 2022-09-10T07:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:15 smithi104 ceph-mon[97044]: pgmap v6369: 129 pgs: 129 active+clean; 8.3 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:15 smithi104 ceph-mon[97044]: osdmap e479: 12 total, 12 up, 12 in 2022-09-10T07:48:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:15 smithi110 ceph-mon[97633]: pgmap v6369: 129 pgs: 129 active+clean; 8.3 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:48:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:15 smithi110 ceph-mon[97633]: osdmap e479: 12 total, 12 up, 12 in 2022-09-10T07:48:15.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:15.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:15.372158+0000) 2022-09-10T07:48:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:15.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:15.738383+0000) 2022-09-10T07:48:16.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:16.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:16.372386+0000) 2022-09-10T07:48:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:16.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:16.738594+0000) 2022-09-10T07:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:17 smithi104 ceph-mon[97044]: pgmap v6371: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:48:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:17.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:17.372654+0000) 2022-09-10T07:48:17.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:48:17.288+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:48:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:17 smithi110 ceph-mon[97633]: pgmap v6371: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:48:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:17.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:17.738791+0000) 2022-09-10T07:48:18.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:18.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:18.372912+0000) 2022-09-10T07:48:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:18.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:18.738979+0000) 2022-09-10T07:48:19.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:19 smithi104 ceph-mon[97044]: pgmap v6372: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T07:48:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:19 smithi110 ceph-mon[97633]: pgmap v6372: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T07:48:19.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:19.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:19.373179+0000) 2022-09-10T07:48:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:19.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:19.739176+0000) 2022-09-10T07:48:20.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:20.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:20.373396+0000) 2022-09-10T07:48:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:20.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:20.739335+0000) 2022-09-10T07:48:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:21 smithi104 ceph-mon[97044]: pgmap v6373: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:48:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:21 smithi110 ceph-mon[97633]: pgmap v6373: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:48:21.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:21.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:21.373677+0000) 2022-09-10T07:48:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:21.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:21.739572+0000) 2022-09-10T07:48:22.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:22.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:22.373871+0000) 2022-09-10T07:48:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:22.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:22.739747+0000) 2022-09-10T07:48:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:23 smithi104 ceph-mon[97044]: pgmap v6374: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:48:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:23 smithi110 ceph-mon[97633]: pgmap v6374: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:48:23.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:23.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:23.374064+0000) 2022-09-10T07:48:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:23.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:23.739961+0000) 2022-09-10T07:48:24.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:24.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:24.374324+0000) 2022-09-10T07:48:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:24.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:24.740187+0000) 2022-09-10T07:48:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:25 smithi104 ceph-mon[97044]: pgmap v6375: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:48:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:25 smithi110 ceph-mon[97633]: pgmap v6375: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:48:25.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:25.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:25.374563+0000) 2022-09-10T07:48:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:25.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:25.740372+0000) 2022-09-10T07:48:26.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:26.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:26.374749+0000) 2022-09-10T07:48:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:26.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:26.740598+0000) 2022-09-10T07:48:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:27 smithi104 ceph-mon[97044]: pgmap v6376: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:48:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:27 smithi110 ceph-mon[97633]: pgmap v6376: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:48:27.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:27.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:27.374927+0000) 2022-09-10T07:48:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:27.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:27.740804+0000) 2022-09-10T07:48:28.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:28.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:28.375171+0000) 2022-09-10T07:48:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:28.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:28.741050+0000) 2022-09-10T07:48:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:29 smithi104 ceph-mon[97044]: pgmap v6377: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:29.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:29 smithi110 ceph-mon[97633]: pgmap v6377: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:29.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:29.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:29.375421+0000) 2022-09-10T07:48:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:29.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:29.741292+0000) 2022-09-10T07:48:30.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:30.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:30.375687+0000) 2022-09-10T07:48:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:30.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:30.741533+0000) 2022-09-10T07:48:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:31 smithi104 ceph-mon[97044]: pgmap v6378: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:31 smithi110 ceph-mon[97633]: pgmap v6378: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:31.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:31.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:31.375955+0000) 2022-09-10T07:48:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:31.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:31.741769+0000) 2022-09-10T07:48:32.742 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:48:32.304+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:48:32.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:32.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:32.376220+0000) 2022-09-10T07:48:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:32.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:32.741973+0000) 2022-09-10T07:48:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:33 smithi104 ceph-mon[97044]: pgmap v6379: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:33 smithi110 ceph-mon[97633]: pgmap v6379: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:33.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:33.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:33.376416+0000) 2022-09-10T07:48:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:33.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:33.742226+0000) 2022-09-10T07:48:34.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:34.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:34.376662+0000) 2022-09-10T07:48:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:34.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:34.742371+0000) 2022-09-10T07:48:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:35 smithi104 ceph-mon[97044]: pgmap v6380: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:35 smithi110 ceph-mon[97633]: pgmap v6380: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:35.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:35.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:35.376865+0000) 2022-09-10T07:48:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:35.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:35.742616+0000) 2022-09-10T07:48:36.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:36.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:36.377074+0000) 2022-09-10T07:48:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:36.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:36.742870+0000) 2022-09-10T07:48:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:37 smithi104 ceph-mon[97044]: pgmap v6381: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:37 smithi110 ceph-mon[97633]: pgmap v6381: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:37.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:37.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:37.377284+0000) 2022-09-10T07:48:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:37.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:37.743166+0000) 2022-09-10T07:48:38.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:38.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:38.377493+0000) 2022-09-10T07:48:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:38.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:38.743334+0000) 2022-09-10T07:48:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:39 smithi104 ceph-mon[97044]: pgmap v6382: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:39 smithi110 ceph-mon[97633]: pgmap v6382: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:39.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:39.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:39.377727+0000) 2022-09-10T07:48:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:39.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:39.743484+0000) 2022-09-10T07:48:40.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:40.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:40.377991+0000) 2022-09-10T07:48:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:40.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:40.743665+0000) 2022-09-10T07:48:41.352 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:41 smithi110 ceph-mon[97633]: pgmap v6383: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:41 smithi104 ceph-mon[97044]: pgmap v6383: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:41.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:41.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:41.378198+0000) 2022-09-10T07:48:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:41.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:41.743855+0000) 2022-09-10T07:48:42.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:42.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:42.378394+0000) 2022-09-10T07:48:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:42.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:42.744057+0000) 2022-09-10T07:48:43.480 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:43 smithi110 ceph-mon[97633]: pgmap v6384: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:43 smithi104 ceph-mon[97044]: pgmap v6384: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:43.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:43.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:43.378568+0000) 2022-09-10T07:48:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:43.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:43.744268+0000) 2022-09-10T07:48:44.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:44.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:44.378803+0000) 2022-09-10T07:48:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:44.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:44.744441+0000) 2022-09-10T07:48:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:45 smithi104 ceph-mon[97044]: pgmap v6385: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:48:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:48:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:48:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:48:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:45 smithi110 ceph-mon[97633]: pgmap v6385: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:48:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:48:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:48:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:48:45.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:45.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:45.379080+0000) 2022-09-10T07:48:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:45.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:45.744587+0000) 2022-09-10T07:48:46.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:46.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:46.379337+0000) 2022-09-10T07:48:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:46.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:46.744801+0000) 2022-09-10T07:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:47 smithi104 ceph-mon[97044]: pgmap v6386: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:47 smithi110 ceph-mon[97633]: pgmap v6386: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:47.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:47.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:47.379478+0000) 2022-09-10T07:48:47.747 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:48:47.315+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:48:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:47.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:47.745027+0000) 2022-09-10T07:48:48.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:48.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:48.379670+0000) 2022-09-10T07:48:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:48.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:48.745252+0000) 2022-09-10T07:48:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:49 smithi104 ceph-mon[97044]: pgmap v6387: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:49 smithi110 ceph-mon[97633]: pgmap v6387: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:49.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:49.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:49.379891+0000) 2022-09-10T07:48:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:49.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:49.745502+0000) 2022-09-10T07:48:50.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:50.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:50.380089+0000) 2022-09-10T07:48:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:50.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:50.745706+0000) 2022-09-10T07:48:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:51 smithi104 ceph-mon[97044]: pgmap v6388: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:51 smithi110 ceph-mon[97633]: pgmap v6388: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:51.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:51.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:51.380324+0000) 2022-09-10T07:48:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:51.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:51.745940+0000) 2022-09-10T07:48:52.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:52.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:52.380583+0000) 2022-09-10T07:48:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:52.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:52.746162+0000) 2022-09-10T07:48:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:53 smithi104 ceph-mon[97044]: pgmap v6389: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:53 smithi110 ceph-mon[97633]: pgmap v6389: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:53.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:53.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:53.380833+0000) 2022-09-10T07:48:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:53.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:53.746348+0000) 2022-09-10T07:48:54.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:54.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:54.381036+0000) 2022-09-10T07:48:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:54.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:54.746537+0000) 2022-09-10T07:48:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:55 smithi104 ceph-mon[97044]: pgmap v6390: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:55 smithi110 ceph-mon[97633]: pgmap v6390: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:55.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:55.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:55.381257+0000) 2022-09-10T07:48:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:55.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:55.746739+0000) 2022-09-10T07:48:56.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:56.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:56.381474+0000) 2022-09-10T07:48:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:56.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:56.746875+0000) 2022-09-10T07:48:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:57 smithi104 ceph-mon[97044]: pgmap v6391: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:57 smithi110 ceph-mon[97633]: pgmap v6391: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:48:57.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:57.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:57.381647+0000) 2022-09-10T07:48:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:57.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:57.747131+0000) 2022-09-10T07:48:58.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:58.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:58.381854+0000) 2022-09-10T07:48:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:58.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:58.747325+0000) 2022-09-10T07:48:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:48:59 smithi104 ceph-mon[97044]: pgmap v6392: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:48:59 smithi110 ceph-mon[97633]: pgmap v6392: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:48:59.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:48:59.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:59.382068+0000) 2022-09-10T07:49:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:48:59.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:48:59.747522+0000) 2022-09-10T07:49:00.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:00.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:00.382314+0000) 2022-09-10T07:49:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:00.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:00.747732+0000) 2022-09-10T07:49:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:01 smithi104 ceph-mon[97044]: pgmap v6393: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:01 smithi110 ceph-mon[97633]: pgmap v6393: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:01.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:01.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:01.382601+0000) 2022-09-10T07:49:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:01.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:01.747979+0000) 2022-09-10T07:49:02.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:02.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:02.382818+0000) 2022-09-10T07:49:02.749 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:49:02.317+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:49:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:02.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:02.748235+0000) 2022-09-10T07:49:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:03 smithi104 ceph-mon[97044]: pgmap v6394: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:03 smithi110 ceph-mon[97633]: pgmap v6394: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:03.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:03.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:03.383035+0000) 2022-09-10T07:49:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:03.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:03.748465+0000) 2022-09-10T07:49:04.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:04.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:04.383260+0000) 2022-09-10T07:49:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:04.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:04.748715+0000) 2022-09-10T07:49:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:05 smithi110 ceph-mon[97633]: pgmap v6395: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:05.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:05.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:05.383407+0000) 2022-09-10T07:49:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:05 smithi104 ceph-mon[97044]: pgmap v6395: 129 pgs: 129 active+clean; 8.3 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:05.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:05.748979+0000) 2022-09-10T07:49:06.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:06.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:06.383633+0000) 2022-09-10T07:49:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:06.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:06.749208+0000) 2022-09-10T07:49:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:07 smithi110 ceph-mon[97633]: pgmap v6396: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:49:07.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:07.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:07.383876+0000) 2022-09-10T07:49:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:07 smithi104 ceph-mon[97044]: pgmap v6396: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:49:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:07.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:07.749369+0000) 2022-09-10T07:49:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:08 smithi110 ceph-mon[97633]: osdmap e480: 12 total, 12 up, 12 in 2022-09-10T07:49:08.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:08.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:08.384126+0000) 2022-09-10T07:49:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:08 smithi104 ceph-mon[97044]: osdmap e480: 12 total, 12 up, 12 in 2022-09-10T07:49:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:08.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:08.749541+0000) 2022-09-10T07:49:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:09 smithi110 ceph-mon[97633]: pgmap v6398: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:09.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:09.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:09.384297+0000) 2022-09-10T07:49:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:09 smithi104 ceph-mon[97044]: pgmap v6398: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:09.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:09.749724+0000) 2022-09-10T07:49:10.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:10.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:10.384464+0000) 2022-09-10T07:49:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:10.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:10.749972+0000) 2022-09-10T07:49:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:11 smithi110 ceph-mon[97633]: pgmap v6399: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:11.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:11.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:11.384700+0000) 2022-09-10T07:49:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:11 smithi104 ceph-mon[97044]: pgmap v6399: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:11.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:11.750190+0000) 2022-09-10T07:49:12.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:12.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:12.384959+0000) 2022-09-10T07:49:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:12.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:12.750405+0000) 2022-09-10T07:49:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:13 smithi110 ceph-mon[97633]: pgmap v6400: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:13.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:13.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:13.385228+0000) 2022-09-10T07:49:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:13 smithi104 ceph-mon[97044]: pgmap v6400: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:13.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:13.750596+0000) 2022-09-10T07:49:14.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:14.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:14.385486+0000) 2022-09-10T07:49:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:14.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:14.750820+0000) 2022-09-10T07:49:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:15 smithi110 ceph-mon[97633]: pgmap v6401: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:15 smithi110 ceph-mon[97633]: osdmap e481: 12 total, 12 up, 12 in 2022-09-10T07:49:15.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:15.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:15.385747+0000) 2022-09-10T07:49:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:15 smithi104 ceph-mon[97044]: pgmap v6401: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:49:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:15 smithi104 ceph-mon[97044]: osdmap e481: 12 total, 12 up, 12 in 2022-09-10T07:49:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:15.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:15.751088+0000) 2022-09-10T07:49:16.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:16.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:16.385930+0000) 2022-09-10T07:49:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:16.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:16.751305+0000) 2022-09-10T07:49:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:17 smithi110 ceph-mon[97633]: pgmap v6403: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T07:49:17.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:17.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:17.386202+0000) 2022-09-10T07:49:17.752 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:49:17.322+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:49:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:17 smithi104 ceph-mon[97044]: pgmap v6403: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T07:49:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:17.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:17.751532+0000) 2022-09-10T07:49:18.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:18.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:18.386415+0000) 2022-09-10T07:49:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:18.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:18.751760+0000) 2022-09-10T07:49:19.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:19 smithi110 ceph-mon[97633]: pgmap v6404: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:49:19.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:19.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:19.386680+0000) 2022-09-10T07:49:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:19 smithi104 ceph-mon[97044]: pgmap v6404: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:49:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:19.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:19.751989+0000) 2022-09-10T07:49:20.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:20.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:20.386967+0000) 2022-09-10T07:49:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:20.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:20.752204+0000) 2022-09-10T07:49:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:21 smithi110 ceph-mon[97633]: pgmap v6405: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:49:21.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:21.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:21.387226+0000) 2022-09-10T07:49:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:21 smithi104 ceph-mon[97044]: pgmap v6405: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:49:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:21.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:21.752397+0000) 2022-09-10T07:49:22.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:22.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:22.387429+0000) 2022-09-10T07:49:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:22.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:22.752572+0000) 2022-09-10T07:49:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:23 smithi110 ceph-mon[97633]: pgmap v6406: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:49:23.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:23.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:23.387605+0000) 2022-09-10T07:49:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:23 smithi104 ceph-mon[97044]: pgmap v6406: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:49:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:23.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:23.752807+0000) 2022-09-10T07:49:24.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:24.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:24.387857+0000) 2022-09-10T07:49:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:24.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:24.753033+0000) 2022-09-10T07:49:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:25 smithi110 ceph-mon[97633]: pgmap v6407: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:49:25.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:25.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:25.388083+0000) 2022-09-10T07:49:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:25 smithi104 ceph-mon[97044]: pgmap v6407: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:49:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:25.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:25.753277+0000) 2022-09-10T07:49:26.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:26.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:26.388358+0000) 2022-09-10T07:49:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:26.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:26.753502+0000) 2022-09-10T07:49:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:27 smithi110 ceph-mon[97633]: pgmap v6408: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:49:27.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:27.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:27.388560+0000) 2022-09-10T07:49:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:27 smithi104 ceph-mon[97044]: pgmap v6408: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:49:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:27.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:27.753689+0000) 2022-09-10T07:49:28.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:28.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:28.388758+0000) 2022-09-10T07:49:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:28.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:28.753896+0000) 2022-09-10T07:49:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:29 smithi110 ceph-mon[97633]: pgmap v6409: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:29.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:29.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:29.389047+0000) 2022-09-10T07:49:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:29 smithi104 ceph-mon[97044]: pgmap v6409: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:29.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:29.754168+0000) 2022-09-10T07:49:30.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:30.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:30.389358+0000) 2022-09-10T07:49:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:30.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:30.754400+0000) 2022-09-10T07:49:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:31 smithi110 ceph-mon[97633]: pgmap v6410: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:31.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:31.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:31.389569+0000) 2022-09-10T07:49:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:31 smithi104 ceph-mon[97044]: pgmap v6410: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:31.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:31.754616+0000) 2022-09-10T07:49:32.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:32.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:32.389726+0000) 2022-09-10T07:49:32.756 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:49:32.338+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:49:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:32.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:32.754857+0000) 2022-09-10T07:49:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:33 smithi110 ceph-mon[97633]: pgmap v6411: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:33.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:33.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:33.389978+0000) 2022-09-10T07:49:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:33 smithi104 ceph-mon[97044]: pgmap v6411: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:33.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:33.755124+0000) 2022-09-10T07:49:34.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:34.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:34.390238+0000) 2022-09-10T07:49:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:34.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:34.755352+0000) 2022-09-10T07:49:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:35 smithi110 ceph-mon[97633]: pgmap v6412: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:35.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:35.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:35.390446+0000) 2022-09-10T07:49:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:35 smithi104 ceph-mon[97044]: pgmap v6412: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:35.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:35.755605+0000) 2022-09-10T07:49:36.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:36.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:36.390640+0000) 2022-09-10T07:49:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:36.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:36.755828+0000) 2022-09-10T07:49:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:37 smithi110 ceph-mon[97633]: pgmap v6413: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:37.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:37.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:37.390912+0000) 2022-09-10T07:49:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:37 smithi104 ceph-mon[97044]: pgmap v6413: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:37.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:37.756034+0000) 2022-09-10T07:49:38.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:38.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:38.391116+0000) 2022-09-10T07:49:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:38.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:38.756241+0000) 2022-09-10T07:49:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:39 smithi110 ceph-mon[97633]: pgmap v6414: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:39.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:39.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:39.391388+0000) 2022-09-10T07:49:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:39 smithi104 ceph-mon[97044]: pgmap v6414: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:39.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:39.756472+0000) 2022-09-10T07:49:40.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:40.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:40.391554+0000) 2022-09-10T07:49:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:40.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:40.756697+0000) 2022-09-10T07:49:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:41 smithi110 ceph-mon[97633]: pgmap v6415: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:41.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:41.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:41.391816+0000) 2022-09-10T07:49:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:41 smithi104 ceph-mon[97044]: pgmap v6415: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:41.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:41.756915+0000) 2022-09-10T07:49:42.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:42.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:42.392019+0000) 2022-09-10T07:49:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:42.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:42.757151+0000) 2022-09-10T07:49:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:43 smithi110 ceph-mon[97633]: pgmap v6416: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:43.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:43.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:43.392282+0000) 2022-09-10T07:49:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:43 smithi104 ceph-mon[97044]: pgmap v6416: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:43.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:43.757362+0000) 2022-09-10T07:49:44.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:44.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:44.392518+0000) 2022-09-10T07:49:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:49:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:49:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:49:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:49:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:49:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:49:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:49:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:49:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:44.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:44.757595+0000) 2022-09-10T07:49:45.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:45.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:45.392803+0000) 2022-09-10T07:49:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:45 smithi104 ceph-mon[97044]: pgmap v6417: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:45 smithi110 ceph-mon[97633]: pgmap v6417: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:45.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:45.757834+0000) 2022-09-10T07:49:46.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:46.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:46.393078+0000) 2022-09-10T07:49:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:46.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:46.758039+0000) 2022-09-10T07:49:47.758 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:49:47.353+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:49:47.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:47.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:47.393336+0000) 2022-09-10T07:49:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:47 smithi104 ceph-mon[97044]: pgmap v6418: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:47 smithi110 ceph-mon[97633]: pgmap v6418: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:47.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:47.758215+0000) 2022-09-10T07:49:48.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:48.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:48.393565+0000) 2022-09-10T07:49:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:48.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:48.758469+0000) 2022-09-10T07:49:49.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:49.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:49.393794+0000) 2022-09-10T07:49:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:49 smithi104 ceph-mon[97044]: pgmap v6419: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:49 smithi110 ceph-mon[97633]: pgmap v6419: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:49.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:49.758711+0000) 2022-09-10T07:49:50.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:50.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:50.393965+0000) 2022-09-10T07:49:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:50.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:50.758938+0000) 2022-09-10T07:49:51.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:51.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:51.394195+0000) 2022-09-10T07:49:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:51 smithi104 ceph-mon[97044]: pgmap v6420: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:51 smithi110 ceph-mon[97633]: pgmap v6420: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:51.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:51.759191+0000) 2022-09-10T07:49:52.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:52.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:52.394443+0000) 2022-09-10T07:49:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:52.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:52.759432+0000) 2022-09-10T07:49:53.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:53.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:53.394644+0000) 2022-09-10T07:49:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:53 smithi104 ceph-mon[97044]: pgmap v6421: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:53 smithi110 ceph-mon[97633]: pgmap v6421: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:53.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:53.759656+0000) 2022-09-10T07:49:54.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:54.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:54.394882+0000) 2022-09-10T07:49:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:54.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:54.759912+0000) 2022-09-10T07:49:55.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:55.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:55.395085+0000) 2022-09-10T07:49:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:55 smithi104 ceph-mon[97044]: pgmap v6422: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:55 smithi110 ceph-mon[97633]: pgmap v6422: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:55.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:55.760170+0000) 2022-09-10T07:49:56.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:56.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:56.395319+0000) 2022-09-10T07:49:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:56.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:56.760403+0000) 2022-09-10T07:49:57.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:57.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:57.395526+0000) 2022-09-10T07:49:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:57 smithi104 ceph-mon[97044]: pgmap v6423: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:57 smithi110 ceph-mon[97633]: pgmap v6423: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:49:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:57.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:57.760612+0000) 2022-09-10T07:49:58.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:58.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:58.395742+0000) 2022-09-10T07:49:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:58.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:58.760797+0000) 2022-09-10T07:49:59.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:49:59.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:59.395917+0000) 2022-09-10T07:49:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:49:59 smithi104 ceph-mon[97044]: pgmap v6424: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:49:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:49:59 smithi110 ceph-mon[97633]: pgmap v6424: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:49:59.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:49:59.760997+0000) 2022-09-10T07:50:00.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:00.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:00.396069+0000) 2022-09-10T07:50:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:50:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T07:50:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:00.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:00.761144+0000) 2022-09-10T07:50:01.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:01.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:01.396373+0000) 2022-09-10T07:50:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:01 smithi104 ceph-mon[97044]: pgmap v6425: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:01 smithi110 ceph-mon[97633]: pgmap v6425: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:01.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:01.761353+0000) 2022-09-10T07:50:02.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:02.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:02.396640+0000) 2022-09-10T07:50:02.762 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:50:02.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:50:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:02.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:02.761604+0000) 2022-09-10T07:50:03.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:03.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:03.396873+0000) 2022-09-10T07:50:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:03 smithi104 ceph-mon[97044]: pgmap v6426: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:03 smithi110 ceph-mon[97633]: pgmap v6426: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:03.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:03.761847+0000) 2022-09-10T07:50:04.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:04.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:04.397092+0000) 2022-09-10T07:50:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:04.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:04.762137+0000) 2022-09-10T07:50:05.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:05.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:05.397358+0000) 2022-09-10T07:50:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:05 smithi104 ceph-mon[97044]: pgmap v6427: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:05 smithi110 ceph-mon[97633]: pgmap v6427: 129 pgs: 129 active+clean; 8.4 MiB data, 709 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:05.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:05.762377+0000) 2022-09-10T07:50:06.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:06.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:06.397560+0000) 2022-09-10T07:50:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:06.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:06.762631+0000) 2022-09-10T07:50:07.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:07.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:07.397791+0000) 2022-09-10T07:50:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:07 smithi104 ceph-mon[97044]: pgmap v6428: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:50:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:07 smithi110 ceph-mon[97633]: pgmap v6428: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:50:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:07.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:07.762866+0000) 2022-09-10T07:50:08.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:08.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:08.398042+0000) 2022-09-10T07:50:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:08 smithi104 ceph-mon[97044]: osdmap e482: 12 total, 12 up, 12 in 2022-09-10T07:50:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:08 smithi110 ceph-mon[97633]: osdmap e482: 12 total, 12 up, 12 in 2022-09-10T07:50:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:08.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:08.763076+0000) 2022-09-10T07:50:09.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:09.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:09.398276+0000) 2022-09-10T07:50:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:09 smithi104 ceph-mon[97044]: pgmap v6430: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:09 smithi110 ceph-mon[97633]: pgmap v6430: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:09.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:09.763338+0000) 2022-09-10T07:50:10.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:10.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:10.398453+0000) 2022-09-10T07:50:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:10.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:10.763560+0000) 2022-09-10T07:50:11.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:11.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:11.398670+0000) 2022-09-10T07:50:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:11 smithi104 ceph-mon[97044]: pgmap v6431: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:11 smithi110 ceph-mon[97633]: pgmap v6431: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:11.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:11.763715+0000) 2022-09-10T07:50:12.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:12.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:12.398892+0000) 2022-09-10T07:50:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:12.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:12.763948+0000) 2022-09-10T07:50:13.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:13.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:13.399088+0000) 2022-09-10T07:50:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:13 smithi104 ceph-mon[97044]: pgmap v6432: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:13 smithi110 ceph-mon[97633]: pgmap v6432: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:13.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:13.764152+0000) 2022-09-10T07:50:14.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:14.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:14.399348+0000) 2022-09-10T07:50:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:14.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:14.764370+0000) 2022-09-10T07:50:15.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:15.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:15.399592+0000) 2022-09-10T07:50:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:15 smithi104 ceph-mon[97044]: pgmap v6433: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:15 smithi104 ceph-mon[97044]: osdmap e483: 12 total, 12 up, 12 in 2022-09-10T07:50:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:15 smithi110 ceph-mon[97633]: pgmap v6433: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:50:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:15 smithi110 ceph-mon[97633]: osdmap e483: 12 total, 12 up, 12 in 2022-09-10T07:50:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:15.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:15.764622+0000) 2022-09-10T07:50:16.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:16.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:16.399792+0000) 2022-09-10T07:50:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:16.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:16.764885+0000) 2022-09-10T07:50:17.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:17.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:17.399976+0000) 2022-09-10T07:50:17.766 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:50:17.369+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:50:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:17 smithi104 ceph-mon[97044]: pgmap v6435: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T07:50:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:17 smithi110 ceph-mon[97633]: pgmap v6435: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T07:50:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:17.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:17.765089+0000) 2022-09-10T07:50:18.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:18.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:18.400171+0000) 2022-09-10T07:50:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:18.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:18.765261+0000) 2022-09-10T07:50:19.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:19.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:19.400391+0000) 2022-09-10T07:50:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:19 smithi104 ceph-mon[97044]: pgmap v6436: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:50:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:19 smithi110 ceph-mon[97633]: pgmap v6436: 129 pgs: 129 active+clean; 8.4 MiB data, 711 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:50:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:19.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:19.765523+0000) 2022-09-10T07:50:20.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:20.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:20.400660+0000) 2022-09-10T07:50:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:20.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:20.765650+0000) 2022-09-10T07:50:21.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:21.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:21.400918+0000) 2022-09-10T07:50:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:21 smithi104 ceph-mon[97044]: pgmap v6437: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:50:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:21 smithi110 ceph-mon[97633]: pgmap v6437: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:50:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:21.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:21.765866+0000) 2022-09-10T07:50:22.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:22.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:22.401210+0000) 2022-09-10T07:50:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:22.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:22.766085+0000) 2022-09-10T07:50:23.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:23.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:23.401470+0000) 2022-09-10T07:50:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:23 smithi104 ceph-mon[97044]: pgmap v6438: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:50:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:23 smithi110 ceph-mon[97633]: pgmap v6438: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:50:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:23.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:23.766271+0000) 2022-09-10T07:50:24.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:24.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:24.401706+0000) 2022-09-10T07:50:25.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:24.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:24.766472+0000) 2022-09-10T07:50:25.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:25.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:25.401974+0000) 2022-09-10T07:50:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:25 smithi104 ceph-mon[97044]: pgmap v6439: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:50:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:25 smithi110 ceph-mon[97633]: pgmap v6439: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:50:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:25.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:25.766659+0000) 2022-09-10T07:50:26.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:26.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:26.402129+0000) 2022-09-10T07:50:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:26.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:26.766893+0000) 2022-09-10T07:50:27.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:27.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:27.402353+0000) 2022-09-10T07:50:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:27 smithi104 ceph-mon[97044]: pgmap v6440: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:50:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:27 smithi110 ceph-mon[97633]: pgmap v6440: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:50:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:27.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:27.767160+0000) 2022-09-10T07:50:28.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:28.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:28.402632+0000) 2022-09-10T07:50:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:28.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:28.767314+0000) 2022-09-10T07:50:29.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:29.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:29.402824+0000) 2022-09-10T07:50:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:29 smithi104 ceph-mon[97044]: pgmap v6441: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:29 smithi110 ceph-mon[97633]: pgmap v6441: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:29.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:29.767603+0000) 2022-09-10T07:50:30.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:30.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:30.403068+0000) 2022-09-10T07:50:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:30.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:30.767901+0000) 2022-09-10T07:50:31.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:31.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:31.403353+0000) 2022-09-10T07:50:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:31 smithi104 ceph-mon[97044]: pgmap v6442: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:31 smithi110 ceph-mon[97633]: pgmap v6442: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:31.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:31.768180+0000) 2022-09-10T07:50:32.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:32.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:32.403587+0000) 2022-09-10T07:50:32.769 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:50:32.373+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:50:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:32.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:32.768432+0000) 2022-09-10T07:50:33.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:33.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:33.403836+0000) 2022-09-10T07:50:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:33 smithi104 ceph-mon[97044]: pgmap v6443: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:33 smithi110 ceph-mon[97633]: pgmap v6443: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:33.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:33.768636+0000) 2022-09-10T07:50:34.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:34.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:34.404091+0000) 2022-09-10T07:50:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:34.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:34.768918+0000) 2022-09-10T07:50:35.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:35.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:35.404300+0000) 2022-09-10T07:50:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:35 smithi104 ceph-mon[97044]: pgmap v6444: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:35 smithi110 ceph-mon[97633]: pgmap v6444: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:35.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:35.769157+0000) 2022-09-10T07:50:36.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:36.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:36.404512+0000) 2022-09-10T07:50:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:36.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:36.769400+0000) 2022-09-10T07:50:37.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:37.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:37.404684+0000) 2022-09-10T07:50:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:37 smithi104 ceph-mon[97044]: pgmap v6445: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:37 smithi110 ceph-mon[97633]: pgmap v6445: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:37.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:37.769560+0000) 2022-09-10T07:50:38.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:38.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:38.404913+0000) 2022-09-10T07:50:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:38.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:38.769794+0000) 2022-09-10T07:50:39.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:39.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:39.405198+0000) 2022-09-10T07:50:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:39 smithi104 ceph-mon[97044]: pgmap v6446: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:39 smithi110 ceph-mon[97633]: pgmap v6446: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:39.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:39.769949+0000) 2022-09-10T07:50:40.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:40.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:40.405417+0000) 2022-09-10T07:50:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:40.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:40.770085+0000) 2022-09-10T07:50:41.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:41.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:41.405702+0000) 2022-09-10T07:50:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:41 smithi104 ceph-mon[97044]: pgmap v6447: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:41 smithi110 ceph-mon[97633]: pgmap v6447: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:41.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:41.770355+0000) 2022-09-10T07:50:42.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:42.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:42.405875+0000) 2022-09-10T07:50:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:42.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:42.770592+0000) 2022-09-10T07:50:43.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:43.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:43.406144+0000) 2022-09-10T07:50:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:43 smithi104 ceph-mon[97044]: pgmap v6448: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:43 smithi110 ceph-mon[97633]: pgmap v6448: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:43.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:43.770756+0000) 2022-09-10T07:50:44.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:44.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:44.406378+0000) 2022-09-10T07:50:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:50:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:50:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:50:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:50:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:50:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:50:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:50:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:50:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:44.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:44.770963+0000) 2022-09-10T07:50:45.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:45.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:45.406652+0000) 2022-09-10T07:50:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:45 smithi104 ceph-mon[97044]: pgmap v6449: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:45 smithi110 ceph-mon[97633]: pgmap v6449: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:45.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:45.771224+0000) 2022-09-10T07:50:46.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:46.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:46.406918+0000) 2022-09-10T07:50:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:46.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:46.771456+0000) 2022-09-10T07:50:47.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:47.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:47.407201+0000) 2022-09-10T07:50:47.773 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:50:47.384+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:50:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:47 smithi104 ceph-mon[97044]: pgmap v6450: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:47 smithi110 ceph-mon[97633]: pgmap v6450: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:47.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:47.771646+0000) 2022-09-10T07:50:48.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:48.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:48.407422+0000) 2022-09-10T07:50:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:48.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:48.771826+0000) 2022-09-10T07:50:49.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:49.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:49.407564+0000) 2022-09-10T07:50:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:49 smithi104 ceph-mon[97044]: pgmap v6451: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:49 smithi110 ceph-mon[97633]: pgmap v6451: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:49.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:49.772023+0000) 2022-09-10T07:50:50.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:50.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:50.407810+0000) 2022-09-10T07:50:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:50.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:50.772280+0000) 2022-09-10T07:50:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:51 smithi104 ceph-mon[97044]: pgmap v6452: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:51 smithi110 ceph-mon[97633]: pgmap v6452: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:51.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:51.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:51.408060+0000) 2022-09-10T07:50:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:51.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:51.772525+0000) 2022-09-10T07:50:52.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:52.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:52.408348+0000) 2022-09-10T07:50:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:52.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:52.772776+0000) 2022-09-10T07:50:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:53 smithi104 ceph-mon[97044]: pgmap v6453: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:53 smithi110 ceph-mon[97633]: pgmap v6453: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:53.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:53.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:53.408579+0000) 2022-09-10T07:50:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:53.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:53.772995+0000) 2022-09-10T07:50:54.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:54.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:54.408777+0000) 2022-09-10T07:50:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:54.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:54.773224+0000) 2022-09-10T07:50:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:55 smithi104 ceph-mon[97044]: pgmap v6454: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:55 smithi110 ceph-mon[97633]: pgmap v6454: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:55.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:55.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:55.409010+0000) 2022-09-10T07:50:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:55.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:55.773391+0000) 2022-09-10T07:50:56.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:56.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:56.409238+0000) 2022-09-10T07:50:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:56.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:56.773616+0000) 2022-09-10T07:50:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:57 smithi104 ceph-mon[97044]: pgmap v6455: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:57 smithi110 ceph-mon[97633]: pgmap v6455: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:50:57.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:57.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:57.409509+0000) 2022-09-10T07:50:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:57.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:57.773836+0000) 2022-09-10T07:50:58.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:58.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:58.409698+0000) 2022-09-10T07:50:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:58.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:58.774084+0000) 2022-09-10T07:50:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:50:59 smithi104 ceph-mon[97044]: pgmap v6456: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:50:59 smithi110 ceph-mon[97633]: pgmap v6456: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:50:59.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:50:59.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:59.409943+0000) 2022-09-10T07:51:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:50:59.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:50:59.774307+0000) 2022-09-10T07:51:00.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:00.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:00.410133+0000) 2022-09-10T07:51:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:00.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:00.774523+0000) 2022-09-10T07:51:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:01 smithi104 ceph-mon[97044]: pgmap v6457: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:01 smithi110 ceph-mon[97633]: pgmap v6457: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:01.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:01.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:01.410297+0000) 2022-09-10T07:51:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:01.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:01.774695+0000) 2022-09-10T07:51:02.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:02.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:02.410549+0000) 2022-09-10T07:51:02.776 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:51:02.397+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:51:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:02.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:02.774952+0000) 2022-09-10T07:51:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:03 smithi104 ceph-mon[97044]: pgmap v6458: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:03 smithi110 ceph-mon[97633]: pgmap v6458: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:03.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:03.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:03.410772+0000) 2022-09-10T07:51:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:03.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:03.775174+0000) 2022-09-10T07:51:04.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:04.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:04.410976+0000) 2022-09-10T07:51:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:04.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:04.775430+0000) 2022-09-10T07:51:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:05 smithi104 ceph-mon[97044]: pgmap v6459: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:05 smithi110 ceph-mon[97633]: pgmap v6459: 129 pgs: 129 active+clean; 8.4 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:05.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:05.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:05.411174+0000) 2022-09-10T07:51:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:05.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:05.775650+0000) 2022-09-10T07:51:06.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:06.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:06.411390+0000) 2022-09-10T07:51:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:06.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:06.775841+0000) 2022-09-10T07:51:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:07 smithi104 ceph-mon[97044]: pgmap v6460: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:51:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:07 smithi110 ceph-mon[97633]: pgmap v6460: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:51:07.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:07.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:07.411582+0000) 2022-09-10T07:51:08.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:07.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:07.776077+0000) 2022-09-10T07:51:08.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:08.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:08.411747+0000) 2022-09-10T07:51:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:08.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:08.776346+0000) 2022-09-10T07:51:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:09 smithi104 ceph-mon[97044]: pgmap v6461: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:51:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:09 smithi104 ceph-mon[97044]: osdmap e484: 12 total, 12 up, 12 in 2022-09-10T07:51:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:09 smithi110 ceph-mon[97633]: pgmap v6461: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:51:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:09 smithi110 ceph-mon[97633]: osdmap e484: 12 total, 12 up, 12 in 2022-09-10T07:51:09.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:09.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:09.411888+0000) 2022-09-10T07:51:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:09.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:09.776564+0000) 2022-09-10T07:51:10.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:10.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:10.412135+0000) 2022-09-10T07:51:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:10.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:10.776741+0000) 2022-09-10T07:51:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:11 smithi104 ceph-mon[97044]: pgmap v6463: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:51:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:11 smithi110 ceph-mon[97633]: pgmap v6463: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:51:11.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:11.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:11.412299+0000) 2022-09-10T07:51:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:11.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:11.776989+0000) 2022-09-10T07:51:12.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:12.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:12.412531+0000) 2022-09-10T07:51:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:12.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:12.777201+0000) 2022-09-10T07:51:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:13 smithi104 ceph-mon[97044]: pgmap v6464: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:51:13.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:13 smithi110 ceph-mon[97633]: pgmap v6464: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:51:13.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:13.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:13.412774+0000) 2022-09-10T07:51:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:13.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:13.777415+0000) 2022-09-10T07:51:14.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:14.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:14.412917+0000) 2022-09-10T07:51:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:14.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:14.777622+0000) 2022-09-10T07:51:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:15 smithi104 ceph-mon[97044]: pgmap v6465: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:51:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:15 smithi104 ceph-mon[97044]: osdmap e485: 12 total, 12 up, 12 in 2022-09-10T07:51:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:15 smithi110 ceph-mon[97633]: pgmap v6465: 129 pgs: 129 active+clean; 8.4 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:51:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:15 smithi110 ceph-mon[97633]: osdmap e485: 12 total, 12 up, 12 in 2022-09-10T07:51:15.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:15.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:15.413085+0000) 2022-09-10T07:51:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:15.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:15.777865+0000) 2022-09-10T07:51:16.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:16.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:16.413339+0000) 2022-09-10T07:51:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:16.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:16.778125+0000) 2022-09-10T07:51:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:17 smithi104 ceph-mon[97044]: pgmap v6467: 129 pgs: 129 active+clean; 8.4 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:51:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:17 smithi110 ceph-mon[97633]: pgmap v6467: 129 pgs: 129 active+clean; 8.4 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:51:17.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:17.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:17.413533+0000) 2022-09-10T07:51:17.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:51:17.410+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:51:18.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:17.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:17.778346+0000) 2022-09-10T07:51:18.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:18.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:18.413785+0000) 2022-09-10T07:51:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:18.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:18.778554+0000) 2022-09-10T07:51:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:19 smithi104 ceph-mon[97044]: pgmap v6468: 129 pgs: 129 active+clean; 8.4 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T07:51:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:19 smithi110 ceph-mon[97633]: pgmap v6468: 129 pgs: 129 active+clean; 8.4 MiB data, 717 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T07:51:19.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:19.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:19.414003+0000) 2022-09-10T07:51:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:19.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:19.778818+0000) 2022-09-10T07:51:20.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:20.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:20.414231+0000) 2022-09-10T07:51:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:20.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:20.779083+0000) 2022-09-10T07:51:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:21 smithi104 ceph-mon[97044]: pgmap v6469: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:51:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:21 smithi110 ceph-mon[97633]: pgmap v6469: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:51:21.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:21.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:21.414444+0000) 2022-09-10T07:51:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:21.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:21.779329+0000) 2022-09-10T07:51:22.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:22.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:22.414659+0000) 2022-09-10T07:51:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:22.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:22.779547+0000) 2022-09-10T07:51:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:23 smithi104 ceph-mon[97044]: pgmap v6470: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:51:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:23 smithi110 ceph-mon[97633]: pgmap v6470: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:51:23.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:23.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:23.414937+0000) 2022-09-10T07:51:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:23.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:23.779778+0000) 2022-09-10T07:51:24.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:24.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:24.415192+0000) 2022-09-10T07:51:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:24.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:24.779999+0000) 2022-09-10T07:51:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:25 smithi104 ceph-mon[97044]: pgmap v6471: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:51:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:25 smithi110 ceph-mon[97633]: pgmap v6471: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:51:25.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:25.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:25.415417+0000) 2022-09-10T07:51:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:25.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:25.780246+0000) 2022-09-10T07:51:26.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:26.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:26.415608+0000) 2022-09-10T07:51:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:26.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:26.780478+0000) 2022-09-10T07:51:27.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:27 smithi104 ceph-mon[97044]: pgmap v6472: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:51:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:27 smithi110 ceph-mon[97633]: pgmap v6472: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:51:27.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:27.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:27.415866+0000) 2022-09-10T07:51:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:27.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:27.780680+0000) 2022-09-10T07:51:28.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:28.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:28.416174+0000) 2022-09-10T07:51:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:28.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:28.780927+0000) 2022-09-10T07:51:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:29 smithi104 ceph-mon[97044]: pgmap v6473: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:29 smithi110 ceph-mon[97633]: pgmap v6473: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:29.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:29.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:29.416359+0000) 2022-09-10T07:51:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:29.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:29.781064+0000) 2022-09-10T07:51:30.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:30.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:30.416586+0000) 2022-09-10T07:51:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:30.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:30.781291+0000) 2022-09-10T07:51:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:31 smithi104 ceph-mon[97044]: pgmap v6474: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:31 smithi110 ceph-mon[97633]: pgmap v6474: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:31.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:31.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:31.416834+0000) 2022-09-10T07:51:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:31.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:31.781467+0000) 2022-09-10T07:51:32.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:32.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:32.417014+0000) 2022-09-10T07:51:32.782 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:51:32.415+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:51:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:32.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:32.781685+0000) 2022-09-10T07:51:33.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:33 smithi104 ceph-mon[97044]: pgmap v6475: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:33 smithi110 ceph-mon[97633]: pgmap v6475: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:33.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:33.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:33.417207+0000) 2022-09-10T07:51:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:33.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:33.781928+0000) 2022-09-10T07:51:34.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:34.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:34.417442+0000) 2022-09-10T07:51:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:34.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:34.782209+0000) 2022-09-10T07:51:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:35 smithi104 ceph-mon[97044]: pgmap v6476: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:35 smithi110 ceph-mon[97633]: pgmap v6476: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:35.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:35.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:35.417674+0000) 2022-09-10T07:51:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:35.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:35.782455+0000) 2022-09-10T07:51:36.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:36.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:36.417881+0000) 2022-09-10T07:51:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:36.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:36.782642+0000) 2022-09-10T07:51:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:37 smithi104 ceph-mon[97044]: pgmap v6477: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:37 smithi110 ceph-mon[97633]: pgmap v6477: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:37.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:37.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:37.418079+0000) 2022-09-10T07:51:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:37.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:37.782829+0000) 2022-09-10T07:51:38.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:38.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:38.418333+0000) 2022-09-10T07:51:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:38.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:38.783071+0000) 2022-09-10T07:51:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:39 smithi104 ceph-mon[97044]: pgmap v6478: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:39 smithi110 ceph-mon[97633]: pgmap v6478: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:39.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:39.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:39.418577+0000) 2022-09-10T07:51:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:39.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:39.783328+0000) 2022-09-10T07:51:40.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:40.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:40.418829+0000) 2022-09-10T07:51:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:40.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:40.783575+0000) 2022-09-10T07:51:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:41 smithi104 ceph-mon[97044]: pgmap v6479: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:41 smithi110 ceph-mon[97633]: pgmap v6479: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:41.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:41.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:41.419071+0000) 2022-09-10T07:51:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:41.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:41.783797+0000) 2022-09-10T07:51:42.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:42.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:42.419232+0000) 2022-09-10T07:51:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:42.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:42.783999+0000) 2022-09-10T07:51:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:43 smithi104 ceph-mon[97044]: pgmap v6480: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:43 smithi110 ceph-mon[97633]: pgmap v6480: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:43.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:43.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:43.419435+0000) 2022-09-10T07:51:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:43.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:43.784220+0000) 2022-09-10T07:51:44.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:44.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:44.419618+0000) 2022-09-10T07:51:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:44.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:44.784432+0000) 2022-09-10T07:51:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:45 smithi104 ceph-mon[97044]: pgmap v6481: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:51:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:51:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:51:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:51:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:45 smithi110 ceph-mon[97633]: pgmap v6481: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:51:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:51:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:51:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:51:45.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:45.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:45.419797+0000) 2022-09-10T07:51:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:45.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:45.784582+0000) 2022-09-10T07:51:46.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:46.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:46.419971+0000) 2022-09-10T07:51:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:46.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:46.784835+0000) 2022-09-10T07:51:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:47 smithi104 ceph-mon[97044]: pgmap v6482: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:47 smithi110 ceph-mon[97633]: pgmap v6482: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:47.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:47.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:47.420222+0000) 2022-09-10T07:51:47.787 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:51:47.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:51:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:47.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:47.785084+0000) 2022-09-10T07:51:48.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:48.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:48.420407+0000) 2022-09-10T07:51:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:48.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:48.785292+0000) 2022-09-10T07:51:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:49 smithi104 ceph-mon[97044]: pgmap v6483: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:49 smithi110 ceph-mon[97633]: pgmap v6483: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:49.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:49.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:49.420632+0000) 2022-09-10T07:51:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:49.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:49.785454+0000) 2022-09-10T07:51:50.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:50.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:50.420860+0000) 2022-09-10T07:51:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:50.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:50.785670+0000) 2022-09-10T07:51:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:51 smithi104 ceph-mon[97044]: pgmap v6484: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:51 smithi110 ceph-mon[97633]: pgmap v6484: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:51.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:51.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:51.421092+0000) 2022-09-10T07:51:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:51.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:51.785888+0000) 2022-09-10T07:51:52.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:52.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:52.421354+0000) 2022-09-10T07:51:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:52.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:52.786153+0000) 2022-09-10T07:51:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:53 smithi104 ceph-mon[97044]: pgmap v6485: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:53 smithi110 ceph-mon[97633]: pgmap v6485: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:53.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:53.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:53.421608+0000) 2022-09-10T07:51:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:53.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:53.786348+0000) 2022-09-10T07:51:54.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:54.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:54.421772+0000) 2022-09-10T07:51:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:54.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:54.786574+0000) 2022-09-10T07:51:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:55 smithi104 ceph-mon[97044]: pgmap v6486: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:55 smithi110 ceph-mon[97633]: pgmap v6486: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:55.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:55.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:55.421980+0000) 2022-09-10T07:51:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:55.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:55.786810+0000) 2022-09-10T07:51:56.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:56.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:56.422178+0000) 2022-09-10T07:51:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:56.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:56.787029+0000) 2022-09-10T07:51:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:57 smithi104 ceph-mon[97044]: pgmap v6487: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:57 smithi110 ceph-mon[97633]: pgmap v6487: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:51:57.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:57.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:57.422460+0000) 2022-09-10T07:51:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:57.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:57.787287+0000) 2022-09-10T07:51:58.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:58.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:58.422664+0000) 2022-09-10T07:51:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:58.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:58.787468+0000) 2022-09-10T07:51:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:51:59 smithi104 ceph-mon[97044]: pgmap v6488: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:51:59 smithi110 ceph-mon[97633]: pgmap v6488: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:51:59.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:51:59.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:59.422899+0000) 2022-09-10T07:52:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:51:59.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:51:59.787714+0000) 2022-09-10T07:52:00.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:00.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:00.423091+0000) 2022-09-10T07:52:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:00.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:00.787901+0000) 2022-09-10T07:52:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:01 smithi104 ceph-mon[97044]: pgmap v6489: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:01 smithi110 ceph-mon[97633]: pgmap v6489: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:01.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:01.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:01.423321+0000) 2022-09-10T07:52:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:01.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:01.788059+0000) 2022-09-10T07:52:02.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:02.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:02.423552+0000) 2022-09-10T07:52:02.789 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:52:02.436+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:52:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:02.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:02.788285+0000) 2022-09-10T07:52:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:03 smithi104 ceph-mon[97044]: pgmap v6490: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:03 smithi110 ceph-mon[97633]: pgmap v6490: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:03.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:03.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:03.423784+0000) 2022-09-10T07:52:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:03.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:03.788463+0000) 2022-09-10T07:52:04.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:04.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:04.423999+0000) 2022-09-10T07:52:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:04.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:04.788688+0000) 2022-09-10T07:52:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:05 smithi104 ceph-mon[97044]: pgmap v6491: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:05 smithi110 ceph-mon[97633]: pgmap v6491: 129 pgs: 129 active+clean; 8.4 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:05.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:05.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:05.424186+0000) 2022-09-10T07:52:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:05.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:05.788868+0000) 2022-09-10T07:52:06.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:06.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:06.424369+0000) 2022-09-10T07:52:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:06.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:06.789118+0000) 2022-09-10T07:52:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:07 smithi104 ceph-mon[97044]: pgmap v6492: 129 pgs: 129 active+clean; 8.4 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:52:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:07 smithi110 ceph-mon[97633]: pgmap v6492: 129 pgs: 129 active+clean; 8.4 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:52:07.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:07.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:07.424549+0000) 2022-09-10T07:52:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:07.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:07.789302+0000) 2022-09-10T07:52:08.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:08.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:08.424788+0000) 2022-09-10T07:52:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:08.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:08.789478+0000) 2022-09-10T07:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:09 smithi104 ceph-mon[97044]: pgmap v6493: 129 pgs: 129 active+clean; 8.4 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:09 smithi104 ceph-mon[97044]: osdmap e486: 12 total, 12 up, 12 in 2022-09-10T07:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:09 smithi110 ceph-mon[97633]: pgmap v6493: 129 pgs: 129 active+clean; 8.4 MiB data, 719 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T07:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:09 smithi110 ceph-mon[97633]: osdmap e486: 12 total, 12 up, 12 in 2022-09-10T07:52:09.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:09.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:09.425064+0000) 2022-09-10T07:52:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:09.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:09.789717+0000) 2022-09-10T07:52:10.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:10.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:10.425306+0000) 2022-09-10T07:52:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:10.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:10.789946+0000) 2022-09-10T07:52:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:11 smithi104 ceph-mon[97044]: pgmap v6495: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:52:11.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:11 smithi110 ceph-mon[97633]: pgmap v6495: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:52:11.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:11.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:11.425483+0000) 2022-09-10T07:52:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:11.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:11.790198+0000) 2022-09-10T07:52:12.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:12.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:12.425709+0000) 2022-09-10T07:52:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:12.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:12.790400+0000) 2022-09-10T07:52:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:13 smithi104 ceph-mon[97044]: pgmap v6496: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:52:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:13 smithi110 ceph-mon[97633]: pgmap v6496: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:52:13.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:13.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:13.425914+0000) 2022-09-10T07:52:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:13.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:13.790620+0000) 2022-09-10T07:52:14.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:14.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:14.426160+0000) 2022-09-10T07:52:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:14.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:14.790875+0000) 2022-09-10T07:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:15 smithi104 ceph-mon[97044]: pgmap v6497: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:15 smithi104 ceph-mon[97044]: osdmap e487: 12 total, 12 up, 12 in 2022-09-10T07:52:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:15 smithi110 ceph-mon[97633]: pgmap v6497: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:52:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:15 smithi110 ceph-mon[97633]: osdmap e487: 12 total, 12 up, 12 in 2022-09-10T07:52:15.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:15.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:15.426393+0000) 2022-09-10T07:52:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:15.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:15.791188+0000) 2022-09-10T07:52:16.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:16.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:16.426559+0000) 2022-09-10T07:52:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:16.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:16.791328+0000) 2022-09-10T07:52:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:17 smithi110 ceph-mon[97633]: pgmap v6499: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:52:17.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:17.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:17.426772+0000) 2022-09-10T07:52:17.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:52:17.443+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:52:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:17 smithi104 ceph-mon[97044]: pgmap v6499: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T07:52:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:17.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:17.791500+0000) 2022-09-10T07:52:18.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:18.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:18.426944+0000) 2022-09-10T07:52:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:18.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:18.791681+0000) 2022-09-10T07:52:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:19 smithi110 ceph-mon[97633]: pgmap v6500: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T07:52:19.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:19.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:19.427199+0000) 2022-09-10T07:52:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:19 smithi104 ceph-mon[97044]: pgmap v6500: 129 pgs: 129 active+clean; 8.4 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T07:52:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:19.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:19.791910+0000) 2022-09-10T07:52:20.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:20.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:20.427462+0000) 2022-09-10T07:52:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:20.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:20.792188+0000) 2022-09-10T07:52:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:21 smithi110 ceph-mon[97633]: pgmap v6501: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:52:21.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:21.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:21.427731+0000) 2022-09-10T07:52:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:21 smithi104 ceph-mon[97044]: pgmap v6501: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:52:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:21.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:21.792391+0000) 2022-09-10T07:52:22.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:22.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:22.427952+0000) 2022-09-10T07:52:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:22.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:22.792639+0000) 2022-09-10T07:52:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:23 smithi110 ceph-mon[97633]: pgmap v6502: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:52:23.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:23.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:23.428181+0000) 2022-09-10T07:52:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:23 smithi104 ceph-mon[97044]: pgmap v6502: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:52:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:23.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:23.792880+0000) 2022-09-10T07:52:24.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:24.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:24.428400+0000) 2022-09-10T07:52:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:24.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:24.793174+0000) 2022-09-10T07:52:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:25 smithi110 ceph-mon[97633]: pgmap v6503: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:52:25.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:25.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:25.428625+0000) 2022-09-10T07:52:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:25 smithi104 ceph-mon[97044]: pgmap v6503: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:52:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:25.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:25.793383+0000) 2022-09-10T07:52:26.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:26.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:26.428877+0000) 2022-09-10T07:52:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:26.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:26.793563+0000) 2022-09-10T07:52:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:27 smithi110 ceph-mon[97633]: pgmap v6504: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:52:27.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:27.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:27.429150+0000) 2022-09-10T07:52:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:27 smithi104 ceph-mon[97044]: pgmap v6504: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:52:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:27.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:27.793801+0000) 2022-09-10T07:52:28.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:28.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:28.429403+0000) 2022-09-10T07:52:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:28.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:28.793991+0000) 2022-09-10T07:52:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:29 smithi110 ceph-mon[97633]: pgmap v6505: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:29.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:29.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:29.429593+0000) 2022-09-10T07:52:29.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:29 smithi104 ceph-mon[97044]: pgmap v6505: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:29.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:29.794213+0000) 2022-09-10T07:52:30.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:30.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:30.429840+0000) 2022-09-10T07:52:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:30.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:30.794439+0000) 2022-09-10T07:52:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:31 smithi110 ceph-mon[97633]: pgmap v6506: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:31.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:31.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:31.430122+0000) 2022-09-10T07:52:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:31 smithi104 ceph-mon[97044]: pgmap v6506: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:31.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:31.794669+0000) 2022-09-10T07:52:32.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:32.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:32.430324+0000) 2022-09-10T07:52:32.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:52:32.456+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:52:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:32.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:32.794840+0000) 2022-09-10T07:52:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:33 smithi110 ceph-mon[97633]: pgmap v6507: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:52:33.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:33.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:33.430583+0000) 2022-09-10T07:52:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:33 smithi104 ceph-mon[97044]: pgmap v6507: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:52:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:33.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:33.795040+0000) 2022-09-10T07:52:34.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:34.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:34.430795+0000) 2022-09-10T07:52:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:34.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:34.795222+0000) 2022-09-10T07:52:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:35 smithi110 ceph-mon[97633]: pgmap v6508: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:52:35.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:35.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:35.431021+0000) 2022-09-10T07:52:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:35 smithi104 ceph-mon[97044]: pgmap v6508: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:52:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:35.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:35.795494+0000) 2022-09-10T07:52:36.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:36.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:36.431186+0000) 2022-09-10T07:52:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:36.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:36.795731+0000) 2022-09-10T07:52:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:37 smithi110 ceph-mon[97633]: pgmap v6509: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:37 smithi104 ceph-mon[97044]: pgmap v6509: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:37.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:37.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:37.431354+0000) 2022-09-10T07:52:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:37.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:37.795975+0000) 2022-09-10T07:52:38.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:38.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:38.431606+0000) 2022-09-10T07:52:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:38.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:38.796217+0000) 2022-09-10T07:52:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:39 smithi110 ceph-mon[97633]: pgmap v6510: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:52:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:39 smithi104 ceph-mon[97044]: pgmap v6510: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:52:39.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:39.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:39.431818+0000) 2022-09-10T07:52:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:39.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:39.796474+0000) 2022-09-10T07:52:40.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:40.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:40.431997+0000) 2022-09-10T07:52:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:40.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:40.796665+0000) 2022-09-10T07:52:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:41 smithi110 ceph-mon[97633]: pgmap v6511: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:41 smithi104 ceph-mon[97044]: pgmap v6511: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:41.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:41.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:41.432260+0000) 2022-09-10T07:52:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:41.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:41.796898+0000) 2022-09-10T07:52:42.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:42.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:42.432426+0000) 2022-09-10T07:52:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:42.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:42.797173+0000) 2022-09-10T07:52:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:43 smithi110 ceph-mon[97633]: pgmap v6512: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:43 smithi104 ceph-mon[97044]: pgmap v6512: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:43.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:43.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:43.432665+0000) 2022-09-10T07:52:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:43.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:43.797337+0000) 2022-09-10T07:52:44.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:44.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:44.432916+0000) 2022-09-10T07:52:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:44.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:44.797544+0000) 2022-09-10T07:52:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:45 smithi110 ceph-mon[97633]: pgmap v6513: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:52:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:52:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:52:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:52:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:45 smithi104 ceph-mon[97044]: pgmap v6513: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:52:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:52:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:52:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:52:45.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:45.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:45.433142+0000) 2022-09-10T07:52:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:45.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:45.797743+0000) 2022-09-10T07:52:46.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:46.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:46.433349+0000) 2022-09-10T07:52:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:46.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:46.797968+0000) 2022-09-10T07:52:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:47 smithi110 ceph-mon[97633]: pgmap v6514: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:47 smithi104 ceph-mon[97044]: pgmap v6514: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:47.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:47.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:47.433577+0000) 2022-09-10T07:52:47.799 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:52:47.468+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:52:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:47.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:47.798214+0000) 2022-09-10T07:52:48.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:48.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:48.433716+0000) 2022-09-10T07:52:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:48.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:48.798449+0000) 2022-09-10T07:52:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:49 smithi110 ceph-mon[97633]: pgmap v6515: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:49 smithi104 ceph-mon[97044]: pgmap v6515: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:49.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:49.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:49.433879+0000) 2022-09-10T07:52:50.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:49.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:49.798675+0000) 2022-09-10T07:52:50.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:50.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:50.434147+0000) 2022-09-10T07:52:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:50.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:50.798926+0000) 2022-09-10T07:52:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:51 smithi110 ceph-mon[97633]: pgmap v6516: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:51 smithi104 ceph-mon[97044]: pgmap v6516: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:51.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:51.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:51.434311+0000) 2022-09-10T07:52:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:51.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:51.799127+0000) 2022-09-10T07:52:52.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:52.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:52.434529+0000) 2022-09-10T07:52:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:52.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:52.799305+0000) 2022-09-10T07:52:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:53 smithi110 ceph-mon[97633]: pgmap v6517: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:53 smithi104 ceph-mon[97044]: pgmap v6517: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:53.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:53.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:53.434761+0000) 2022-09-10T07:52:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:53.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:53.799533+0000) 2022-09-10T07:52:54.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:54.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:54.434942+0000) 2022-09-10T07:52:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:54.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:54.799739+0000) 2022-09-10T07:52:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:55 smithi110 ceph-mon[97633]: pgmap v6518: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:55 smithi104 ceph-mon[97044]: pgmap v6518: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:55.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:55.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:55.435194+0000) 2022-09-10T07:52:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:55.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:55.800022+0000) 2022-09-10T07:52:56.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:56.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:56.435403+0000) 2022-09-10T07:52:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:56.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:56.800250+0000) 2022-09-10T07:52:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:57 smithi110 ceph-mon[97633]: pgmap v6519: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:57 smithi104 ceph-mon[97044]: pgmap v6519: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:52:57.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:57.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:57.435657+0000) 2022-09-10T07:52:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:57.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:57.800500+0000) 2022-09-10T07:52:58.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:58.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:58.435915+0000) 2022-09-10T07:52:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:58.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:58.800731+0000) 2022-09-10T07:52:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:52:59 smithi110 ceph-mon[97633]: pgmap v6520: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:52:59 smithi104 ceph-mon[97044]: pgmap v6520: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:52:59.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:52:59.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:59.436144+0000) 2022-09-10T07:53:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:52:59.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:52:59.800985+0000) 2022-09-10T07:53:00.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:00.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:00.436268+0000) 2022-09-10T07:53:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:00.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:00.801197+0000) 2022-09-10T07:53:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:01 smithi110 ceph-mon[97633]: pgmap v6521: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:01 smithi104 ceph-mon[97044]: pgmap v6521: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:01.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:01.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:01.436481+0000) 2022-09-10T07:53:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:01.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:01.801327+0000) 2022-09-10T07:53:02.802 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:53:02.482+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:53:02.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:02.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:02.436707+0000) 2022-09-10T07:53:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:02.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:02.801488+0000) 2022-09-10T07:53:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:03 smithi110 ceph-mon[97633]: pgmap v6522: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:03 smithi104 ceph-mon[97044]: pgmap v6522: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:03.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:03.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:03.436866+0000) 2022-09-10T07:53:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:03.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:03.801664+0000) 2022-09-10T07:53:04.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:04.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:04.437079+0000) 2022-09-10T07:53:05.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:04.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:04.801907+0000) 2022-09-10T07:53:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:05 smithi110 ceph-mon[97633]: pgmap v6523: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:05 smithi104 ceph-mon[97044]: pgmap v6523: 129 pgs: 129 active+clean; 8.4 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:05.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:05.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:05.437293+0000) 2022-09-10T07:53:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:05.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:05.802178+0000) 2022-09-10T07:53:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:06.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:06.437469+0000) 2022-09-10T07:53:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:06.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:06.802330+0000) 2022-09-10T07:53:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:07 smithi110 ceph-mon[97633]: pgmap v6524: 129 pgs: 129 active+clean; 8.4 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:53:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:07 smithi104 ceph-mon[97044]: pgmap v6524: 129 pgs: 129 active+clean; 8.4 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:53:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:07.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:07.438334+0000) 2022-09-10T07:53:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:07.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:07.802580+0000) 2022-09-10T07:53:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:08 smithi110 ceph-mon[97633]: osdmap e488: 12 total, 12 up, 12 in 2022-09-10T07:53:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:08 smithi104 ceph-mon[97044]: osdmap e488: 12 total, 12 up, 12 in 2022-09-10T07:53:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:08.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:08.438558+0000) 2022-09-10T07:53:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:08.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:08.802791+0000) 2022-09-10T07:53:09.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:09 smithi110 ceph-mon[97633]: pgmap v6526: 129 pgs: 129 active+clean; 8.4 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:09 smithi104 ceph-mon[97044]: pgmap v6526: 129 pgs: 129 active+clean; 8.4 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:09.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:09.438825+0000) 2022-09-10T07:53:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:09.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:09.803008+0000) 2022-09-10T07:53:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:10.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:10.439050+0000) 2022-09-10T07:53:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:10.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:10.803214+0000) 2022-09-10T07:53:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:11 smithi110 ceph-mon[97633]: pgmap v6527: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:11 smithi104 ceph-mon[97044]: pgmap v6527: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:11.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:11.439260+0000) 2022-09-10T07:53:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:11.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:11.803457+0000) 2022-09-10T07:53:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:12.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:12.439484+0000) 2022-09-10T07:53:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:12.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:12.803724+0000) 2022-09-10T07:53:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:13 smithi110 ceph-mon[97633]: pgmap v6528: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:13 smithi104 ceph-mon[97044]: pgmap v6528: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:13.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:13.439695+0000) 2022-09-10T07:53:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:13.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:13.803878+0000) 2022-09-10T07:53:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:14.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:14.439933+0000) 2022-09-10T07:53:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:14.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:14.804009+0000) 2022-09-10T07:53:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:15 smithi110 ceph-mon[97633]: pgmap v6529: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:15 smithi110 ceph-mon[97633]: osdmap e489: 12 total, 12 up, 12 in 2022-09-10T07:53:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:15 smithi104 ceph-mon[97044]: pgmap v6529: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:53:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:15 smithi104 ceph-mon[97044]: osdmap e489: 12 total, 12 up, 12 in 2022-09-10T07:53:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:15.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:15.440220+0000) 2022-09-10T07:53:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:15.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:15.804171+0000) 2022-09-10T07:53:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:16.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:16.440471+0000) 2022-09-10T07:53:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:16.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:16.804334+0000) 2022-09-10T07:53:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:17 smithi110 ceph-mon[97633]: pgmap v6531: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T07:53:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:17 smithi104 ceph-mon[97044]: pgmap v6531: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T07:53:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:53:17.489+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:53:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:17.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:17.440648+0000) 2022-09-10T07:53:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:17.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:17.804475+0000) 2022-09-10T07:53:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:18.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:18.440893+0000) 2022-09-10T07:53:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:18.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:18.804603+0000) 2022-09-10T07:53:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:19 smithi110 ceph-mon[97633]: pgmap v6532: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:53:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:19 smithi104 ceph-mon[97044]: pgmap v6532: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:53:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:19.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:19.441177+0000) 2022-09-10T07:53:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:19.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:19.804790+0000) 2022-09-10T07:53:20.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:20.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:20.441372+0000) 2022-09-10T07:53:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:20.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:20.805001+0000) 2022-09-10T07:53:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:21 smithi110 ceph-mon[97633]: pgmap v6533: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:53:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:21 smithi104 ceph-mon[97044]: pgmap v6533: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:53:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:21.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:21.441546+0000) 2022-09-10T07:53:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:21.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:21.805156+0000) 2022-09-10T07:53:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:22.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:22.441792+0000) 2022-09-10T07:53:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:22.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:22.805336+0000) 2022-09-10T07:53:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:23 smithi110 ceph-mon[97633]: pgmap v6534: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:53:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:23 smithi104 ceph-mon[97044]: pgmap v6534: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:53:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:23.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:23.442049+0000) 2022-09-10T07:53:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:23.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:23.805535+0000) 2022-09-10T07:53:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:24.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:24.442166+0000) 2022-09-10T07:53:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:24.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:24.805739+0000) 2022-09-10T07:53:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:25 smithi110 ceph-mon[97633]: pgmap v6535: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:53:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:25 smithi104 ceph-mon[97044]: pgmap v6535: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:53:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:25.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:25.442307+0000) 2022-09-10T07:53:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:25.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:25.805959+0000) 2022-09-10T07:53:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:26.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:26.442502+0000) 2022-09-10T07:53:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:26.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:26.806217+0000) 2022-09-10T07:53:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:27 smithi104 ceph-mon[97044]: pgmap v6536: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:53:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:27.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:27.442710+0000) 2022-09-10T07:53:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:27 smithi110 ceph-mon[97633]: pgmap v6536: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:53:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:27.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:27.806399+0000) 2022-09-10T07:53:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:28.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:28.442962+0000) 2022-09-10T07:53:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:28.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:28.806555+0000) 2022-09-10T07:53:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:29 smithi104 ceph-mon[97044]: pgmap v6537: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:29.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:29.443198+0000) 2022-09-10T07:53:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:29 smithi110 ceph-mon[97633]: pgmap v6537: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:29.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:29.806715+0000) 2022-09-10T07:53:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:30.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:30.443427+0000) 2022-09-10T07:53:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:30.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:30.806890+0000) 2022-09-10T07:53:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:31 smithi104 ceph-mon[97044]: pgmap v6538: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:31.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:31.443661+0000) 2022-09-10T07:53:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:31 smithi110 ceph-mon[97633]: pgmap v6538: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:31.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:31.807081+0000) 2022-09-10T07:53:32.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:32.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:32.443879+0000) 2022-09-10T07:53:32.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:53:32.491+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:53:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:32.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:32.807262+0000) 2022-09-10T07:53:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:33 smithi104 ceph-mon[97044]: pgmap v6539: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:33.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:33.444168+0000) 2022-09-10T07:53:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:33 smithi110 ceph-mon[97633]: pgmap v6539: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:33.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:33.807397+0000) 2022-09-10T07:53:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:34.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:34.444391+0000) 2022-09-10T07:53:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:34.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:34.807572+0000) 2022-09-10T07:53:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:35 smithi104 ceph-mon[97044]: pgmap v6540: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:35.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:35.444607+0000) 2022-09-10T07:53:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:35 smithi110 ceph-mon[97633]: pgmap v6540: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:35.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:35.807825+0000) 2022-09-10T07:53:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:36.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:36.444882+0000) 2022-09-10T07:53:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:36.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:36.808160+0000) 2022-09-10T07:53:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:37 smithi104 ceph-mon[97044]: pgmap v6541: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:37.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:37.445130+0000) 2022-09-10T07:53:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:37 smithi110 ceph-mon[97633]: pgmap v6541: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:37.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:37.808299+0000) 2022-09-10T07:53:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:38.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:38.445384+0000) 2022-09-10T07:53:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:38.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:38.808460+0000) 2022-09-10T07:53:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:39 smithi104 ceph-mon[97044]: pgmap v6542: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:53:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:39.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:39.445551+0000) 2022-09-10T07:53:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:39 smithi110 ceph-mon[97633]: pgmap v6542: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:53:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:39.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:39.808656+0000) 2022-09-10T07:53:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:40.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:40.445777+0000) 2022-09-10T07:53:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:40.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:40.808900+0000) 2022-09-10T07:53:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:41 smithi104 ceph-mon[97044]: pgmap v6543: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:41.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:41.446043+0000) 2022-09-10T07:53:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:41 smithi110 ceph-mon[97633]: pgmap v6543: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:41.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:41.809163+0000) 2022-09-10T07:53:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:42.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:42.446281+0000) 2022-09-10T07:53:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:42.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:42.809404+0000) 2022-09-10T07:53:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:43 smithi104 ceph-mon[97044]: pgmap v6544: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:53:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:43.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:43.446525+0000) 2022-09-10T07:53:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:43 smithi110 ceph-mon[97633]: pgmap v6544: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:53:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:43.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:43.809561+0000) 2022-09-10T07:53:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:53:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:53:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:53:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:53:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:44.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:44.446775+0000) 2022-09-10T07:53:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:53:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:53:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:53:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:53:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:44.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:44.809854+0000) 2022-09-10T07:53:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:45 smithi104 ceph-mon[97044]: pgmap v6545: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:53:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:45.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:45.446965+0000) 2022-09-10T07:53:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:45 smithi110 ceph-mon[97633]: pgmap v6545: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:53:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:45.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:45.810035+0000) 2022-09-10T07:53:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:46.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:46.447278+0000) 2022-09-10T07:53:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:46.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:46.810265+0000) 2022-09-10T07:53:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:47 smithi104 ceph-mon[97044]: pgmap v6546: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:47.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:47.447544+0000) 2022-09-10T07:53:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:53:47.506+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:53:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:47 smithi110 ceph-mon[97633]: pgmap v6546: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:47.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:47.810496+0000) 2022-09-10T07:53:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:48.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:48.447794+0000) 2022-09-10T07:53:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:48.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:48.810715+0000) 2022-09-10T07:53:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:49 smithi104 ceph-mon[97044]: pgmap v6547: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:49.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:49.447946+0000) 2022-09-10T07:53:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:49 smithi110 ceph-mon[97633]: pgmap v6547: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:49.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:49.810943+0000) 2022-09-10T07:53:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:50.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:50.448158+0000) 2022-09-10T07:53:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:50.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:50.811220+0000) 2022-09-10T07:53:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:51 smithi104 ceph-mon[97044]: pgmap v6548: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:51.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:51.448388+0000) 2022-09-10T07:53:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:51 smithi110 ceph-mon[97633]: pgmap v6548: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:51.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:51.811416+0000) 2022-09-10T07:53:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:52.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:52.448526+0000) 2022-09-10T07:53:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:52.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:52.811582+0000) 2022-09-10T07:53:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:53 smithi104 ceph-mon[97044]: pgmap v6549: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:53.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:53.448748+0000) 2022-09-10T07:53:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:53 smithi110 ceph-mon[97633]: pgmap v6549: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:53.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:53.811863+0000) 2022-09-10T07:53:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:54.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:54.448995+0000) 2022-09-10T07:53:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:54.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:54.812056+0000) 2022-09-10T07:53:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:55 smithi104 ceph-mon[97044]: pgmap v6550: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:55.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:55.449190+0000) 2022-09-10T07:53:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:55 smithi110 ceph-mon[97633]: pgmap v6550: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:55.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:55.812236+0000) 2022-09-10T07:53:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:56.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:56.449413+0000) 2022-09-10T07:53:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:56.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:56.812473+0000) 2022-09-10T07:53:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:57 smithi104 ceph-mon[97044]: pgmap v6551: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:57.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:57.449568+0000) 2022-09-10T07:53:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:57 smithi110 ceph-mon[97633]: pgmap v6551: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:53:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:57.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:57.812630+0000) 2022-09-10T07:53:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:58.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:58.449713+0000) 2022-09-10T07:53:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:58.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:58.812875+0000) 2022-09-10T07:53:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:53:59 smithi104 ceph-mon[97044]: pgmap v6552: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:53:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:53:59.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:59.449900+0000) 2022-09-10T07:53:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:53:59 smithi110 ceph-mon[97633]: pgmap v6552: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:53:59.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:53:59.813177+0000) 2022-09-10T07:54:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:00.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:00.450020+0000) 2022-09-10T07:54:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:00.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:00.813425+0000) 2022-09-10T07:54:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:01 smithi104 ceph-mon[97044]: pgmap v6553: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:01.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:01.450220+0000) 2022-09-10T07:54:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:01 smithi110 ceph-mon[97633]: pgmap v6553: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:01.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:01.813703+0000) 2022-09-10T07:54:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:54:02.522+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:54:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:02.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:02.450471+0000) 2022-09-10T07:54:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:02.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:02.813870+0000) 2022-09-10T07:54:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:03 smithi104 ceph-mon[97044]: pgmap v6554: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:03.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:03.450663+0000) 2022-09-10T07:54:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:03 smithi110 ceph-mon[97633]: pgmap v6554: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:03.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:03.814154+0000) 2022-09-10T07:54:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:04.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:04.450774+0000) 2022-09-10T07:54:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:04.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:04.814336+0000) 2022-09-10T07:54:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:05 smithi104 ceph-mon[97044]: pgmap v6555: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:05.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:05.450957+0000) 2022-09-10T07:54:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:05 smithi110 ceph-mon[97633]: pgmap v6555: 129 pgs: 129 active+clean; 8.4 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:05.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:05.814530+0000) 2022-09-10T07:54:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:06.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:06.451207+0000) 2022-09-10T07:54:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:06.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:06.814729+0000) 2022-09-10T07:54:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:07 smithi104 ceph-mon[97044]: pgmap v6556: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:54:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:07.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:07.451423+0000) 2022-09-10T07:54:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:07 smithi110 ceph-mon[97633]: pgmap v6556: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:54:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:07.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:07.815025+0000) 2022-09-10T07:54:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:08 smithi104 ceph-mon[97044]: osdmap e490: 12 total, 12 up, 12 in 2022-09-10T07:54:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:08.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:08.451564+0000) 2022-09-10T07:54:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:08 smithi110 ceph-mon[97633]: osdmap e490: 12 total, 12 up, 12 in 2022-09-10T07:54:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:08.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:08.815220+0000) 2022-09-10T07:54:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:09 smithi104 ceph-mon[97044]: pgmap v6558: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:09.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:09.451825+0000) 2022-09-10T07:54:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:09 smithi110 ceph-mon[97633]: pgmap v6558: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:09.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:09.815420+0000) 2022-09-10T07:54:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:10.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:10.452020+0000) 2022-09-10T07:54:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:10.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:10.815679+0000) 2022-09-10T07:54:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:11 smithi104 ceph-mon[97044]: pgmap v6559: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:11.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:11.452251+0000) 2022-09-10T07:54:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:11 smithi110 ceph-mon[97633]: pgmap v6559: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:11.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:11.815878+0000) 2022-09-10T07:54:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:12.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:12.452442+0000) 2022-09-10T07:54:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:12.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:12.816134+0000) 2022-09-10T07:54:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:13 smithi104 ceph-mon[97044]: pgmap v6560: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:13.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:13.452689+0000) 2022-09-10T07:54:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:13 smithi110 ceph-mon[97633]: pgmap v6560: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:13.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:13.816338+0000) 2022-09-10T07:54:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:14.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:14.452819+0000) 2022-09-10T07:54:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:14.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:14.816613+0000) 2022-09-10T07:54:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:15 smithi104 ceph-mon[97044]: pgmap v6561: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:15 smithi104 ceph-mon[97044]: osdmap e491: 12 total, 12 up, 12 in 2022-09-10T07:54:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:15.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:15.453057+0000) 2022-09-10T07:54:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:15 smithi110 ceph-mon[97633]: pgmap v6561: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:54:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:15 smithi110 ceph-mon[97633]: osdmap e491: 12 total, 12 up, 12 in 2022-09-10T07:54:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:15.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:15.816835+0000) 2022-09-10T07:54:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:16.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:16.453323+0000) 2022-09-10T07:54:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:16.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:16.817091+0000) 2022-09-10T07:54:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:17 smithi104 ceph-mon[97044]: pgmap v6563: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T07:54:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:17.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:17.453543+0000) 2022-09-10T07:54:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:54:17.536+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:54:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:17 smithi110 ceph-mon[97633]: pgmap v6563: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T07:54:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:17.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:17.817312+0000) 2022-09-10T07:54:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:18.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:18.453716+0000) 2022-09-10T07:54:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:18.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:18.817507+0000) 2022-09-10T07:54:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:19 smithi104 ceph-mon[97044]: pgmap v6564: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:54:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:19.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:19.453934+0000) 2022-09-10T07:54:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:19 smithi110 ceph-mon[97633]: pgmap v6564: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:54:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:19.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:19.817766+0000) 2022-09-10T07:54:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:20.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:20.454162+0000) 2022-09-10T07:54:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:20.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:20.817993+0000) 2022-09-10T07:54:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:21 smithi104 ceph-mon[97044]: pgmap v6565: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:54:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:21.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:21.454359+0000) 2022-09-10T07:54:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:21 smithi110 ceph-mon[97633]: pgmap v6565: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:54:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:21.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:21.818212+0000) 2022-09-10T07:54:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:22.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:22.454585+0000) 2022-09-10T07:54:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:22.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:22.818424+0000) 2022-09-10T07:54:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:23 smithi104 ceph-mon[97044]: pgmap v6566: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:54:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:23.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:23.454751+0000) 2022-09-10T07:54:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:23 smithi110 ceph-mon[97633]: pgmap v6566: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:54:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:23.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:23.818628+0000) 2022-09-10T07:54:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:24.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:24.454975+0000) 2022-09-10T07:54:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:24.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:24.818839+0000) 2022-09-10T07:54:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:25 smithi104 ceph-mon[97044]: pgmap v6567: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:54:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:25.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:25.455206+0000) 2022-09-10T07:54:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:25 smithi110 ceph-mon[97633]: pgmap v6567: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:54:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:25.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:25.819082+0000) 2022-09-10T07:54:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:26.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:26.455423+0000) 2022-09-10T07:54:27.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:26.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:26.819371+0000) 2022-09-10T07:54:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:27 smithi104 ceph-mon[97044]: pgmap v6568: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:54:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:27.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:27.455669+0000) 2022-09-10T07:54:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:27 smithi110 ceph-mon[97633]: pgmap v6568: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:54:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:27.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:27.819617+0000) 2022-09-10T07:54:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:28.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:28.455924+0000) 2022-09-10T07:54:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:28.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:28.819848+0000) 2022-09-10T07:54:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:29 smithi104 ceph-mon[97044]: pgmap v6569: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:29.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:29.456184+0000) 2022-09-10T07:54:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:29 smithi110 ceph-mon[97633]: pgmap v6569: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:29.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:29.820040+0000) 2022-09-10T07:54:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:30.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:30.456342+0000) 2022-09-10T07:54:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:30.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:30.820273+0000) 2022-09-10T07:54:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:31 smithi104 ceph-mon[97044]: pgmap v6570: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:31.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:31.456589+0000) 2022-09-10T07:54:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:31 smithi110 ceph-mon[97633]: pgmap v6570: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:31.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:31.820489+0000) 2022-09-10T07:54:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:32.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:32.456786+0000) 2022-09-10T07:54:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:54:32.551+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:54:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:32.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:32.820677+0000) 2022-09-10T07:54:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:33 smithi104 ceph-mon[97044]: pgmap v6571: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:33.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:33.457055+0000) 2022-09-10T07:54:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:33 smithi110 ceph-mon[97633]: pgmap v6571: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:33.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:33.820899+0000) 2022-09-10T07:54:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:34.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:34.457303+0000) 2022-09-10T07:54:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:34.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:34.821184+0000) 2022-09-10T07:54:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:35 smithi104 ceph-mon[97044]: pgmap v6572: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:35.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:35.457501+0000) 2022-09-10T07:54:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:35 smithi110 ceph-mon[97633]: pgmap v6572: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:35.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:35.821393+0000) 2022-09-10T07:54:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:36.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:36.457669+0000) 2022-09-10T07:54:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:36.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:36.821630+0000) 2022-09-10T07:54:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:37 smithi104 ceph-mon[97044]: pgmap v6573: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:37.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:37.457901+0000) 2022-09-10T07:54:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:37 smithi110 ceph-mon[97633]: pgmap v6573: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:37.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:37.821840+0000) 2022-09-10T07:54:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:38.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:38.458176+0000) 2022-09-10T07:54:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:38.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:38.822032+0000) 2022-09-10T07:54:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:39 smithi104 ceph-mon[97044]: pgmap v6574: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:39.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:39.458397+0000) 2022-09-10T07:54:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:39 smithi110 ceph-mon[97633]: pgmap v6574: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:39.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:39.822258+0000) 2022-09-10T07:54:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:40.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:40.458621+0000) 2022-09-10T07:54:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:40.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:40.822424+0000) 2022-09-10T07:54:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:41 smithi104 ceph-mon[97044]: pgmap v6575: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:41.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:41.458915+0000) 2022-09-10T07:54:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:41 smithi110 ceph-mon[97633]: pgmap v6575: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:41.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:41.822635+0000) 2022-09-10T07:54:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:42.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:42.459200+0000) 2022-09-10T07:54:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:42.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:42.822863+0000) 2022-09-10T07:54:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:43 smithi104 ceph-mon[97044]: pgmap v6576: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:43.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:43.459458+0000) 2022-09-10T07:54:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:43 smithi110 ceph-mon[97633]: pgmap v6576: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:43.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:43.823142+0000) 2022-09-10T07:54:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:54:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:54:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:54:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:54:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:44.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:44.459629+0000) 2022-09-10T07:54:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:54:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:54:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:54:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:54:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:44.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:44.823369+0000) 2022-09-10T07:54:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:45 smithi104 ceph-mon[97044]: pgmap v6577: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:45.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:45.459839+0000) 2022-09-10T07:54:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:45 smithi110 ceph-mon[97633]: pgmap v6577: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:45.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:45.823591+0000) 2022-09-10T07:54:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:46.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:46.460072+0000) 2022-09-10T07:54:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:46.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:46.823792+0000) 2022-09-10T07:54:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:47 smithi104 ceph-mon[97044]: pgmap v6578: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:47.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:47.460261+0000) 2022-09-10T07:54:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:54:47.555+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:54:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:47 smithi110 ceph-mon[97633]: pgmap v6578: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:47.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:47.824006+0000) 2022-09-10T07:54:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:48.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:48.460485+0000) 2022-09-10T07:54:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:48.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:48.824213+0000) 2022-09-10T07:54:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:49 smithi104 ceph-mon[97044]: pgmap v6579: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:49.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:49.460784+0000) 2022-09-10T07:54:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:49 smithi110 ceph-mon[97633]: pgmap v6579: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:49.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:49.824454+0000) 2022-09-10T07:54:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:50.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:50.460997+0000) 2022-09-10T07:54:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:50.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:50.824684+0000) 2022-09-10T07:54:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:51 smithi104 ceph-mon[97044]: pgmap v6580: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:51.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:51.461224+0000) 2022-09-10T07:54:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:51 smithi110 ceph-mon[97633]: pgmap v6580: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:51.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:51.824931+0000) 2022-09-10T07:54:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:52.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:52.461372+0000) 2022-09-10T07:54:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:52.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:52.825141+0000) 2022-09-10T07:54:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:53.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:53.461602+0000) 2022-09-10T07:54:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:53 smithi110 ceph-mon[97633]: pgmap v6581: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:53 smithi104 ceph-mon[97044]: pgmap v6581: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:53.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:53.825297+0000) 2022-09-10T07:54:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:54.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:54.461816+0000) 2022-09-10T07:54:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:54.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:54.825470+0000) 2022-09-10T07:54:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:55.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:55.462077+0000) 2022-09-10T07:54:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:55 smithi110 ceph-mon[97633]: pgmap v6582: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:55 smithi104 ceph-mon[97044]: pgmap v6582: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:54:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:55.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:55.825703+0000) 2022-09-10T07:54:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:56.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:56.462348+0000) 2022-09-10T07:54:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:56.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:56.825905+0000) 2022-09-10T07:54:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:57.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:57.462557+0000) 2022-09-10T07:54:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:57 smithi110 ceph-mon[97633]: pgmap v6583: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:57 smithi104 ceph-mon[97044]: pgmap v6583: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:54:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:57.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:57.826083+0000) 2022-09-10T07:54:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:58.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:58.462792+0000) 2022-09-10T07:54:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:58.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:58.826350+0000) 2022-09-10T07:54:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:54:59.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:59.463042+0000) 2022-09-10T07:54:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:54:59 smithi110 ceph-mon[97633]: pgmap v6584: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:54:59 smithi104 ceph-mon[97044]: pgmap v6584: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:54:59.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:54:59.826544+0000) 2022-09-10T07:55:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:00.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:00.463271+0000) 2022-09-10T07:55:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:00.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:00.826784+0000) 2022-09-10T07:55:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:01.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:01.463515+0000) 2022-09-10T07:55:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:01 smithi110 ceph-mon[97633]: pgmap v6585: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:01 smithi104 ceph-mon[97044]: pgmap v6585: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:01.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:01.826948+0000) 2022-09-10T07:55:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:02.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:02.463750+0000) 2022-09-10T07:55:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:55:02.562+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:55:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:02.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:02.827144+0000) 2022-09-10T07:55:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:03.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:03.463981+0000) 2022-09-10T07:55:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:03 smithi110 ceph-mon[97633]: pgmap v6586: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:03 smithi104 ceph-mon[97044]: pgmap v6586: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:03.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:03.827348+0000) 2022-09-10T07:55:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:04.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:04.464193+0000) 2022-09-10T07:55:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:04.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:04.827607+0000) 2022-09-10T07:55:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:05.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:05.464411+0000) 2022-09-10T07:55:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:05 smithi110 ceph-mon[97633]: pgmap v6587: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:05 smithi104 ceph-mon[97044]: pgmap v6587: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:05.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:05.827846+0000) 2022-09-10T07:55:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:06.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:06.464645+0000) 2022-09-10T07:55:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:06 smithi110 ceph-mon[97633]: pgmap v6588: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:55:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:06 smithi104 ceph-mon[97044]: pgmap v6588: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:55:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:06.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:06.828083+0000) 2022-09-10T07:55:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:07.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:07.464883+0000) 2022-09-10T07:55:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:07.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:07.828274+0000) 2022-09-10T07:55:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:08.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:08.465079+0000) 2022-09-10T07:55:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:08 smithi110 ceph-mon[97633]: osdmap e492: 12 total, 12 up, 12 in 2022-09-10T07:55:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:08 smithi110 ceph-mon[97633]: pgmap v6590: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:55:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:08 smithi104 ceph-mon[97044]: osdmap e492: 12 total, 12 up, 12 in 2022-09-10T07:55:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:08 smithi104 ceph-mon[97044]: pgmap v6590: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:55:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:08.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:08.828420+0000) 2022-09-10T07:55:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:09.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:09.465350+0000) 2022-09-10T07:55:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:09.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:09.828596+0000) 2022-09-10T07:55:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:10.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:10.465594+0000) 2022-09-10T07:55:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:10.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:10.828802+0000) 2022-09-10T07:55:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:11 smithi104 ceph-mon[97044]: pgmap v6591: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:55:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:11 smithi104 ceph-mon[97044]: osdmap e493: 12 total, 12 up, 12 in 2022-09-10T07:55:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:11 smithi110 ceph-mon[97633]: pgmap v6591: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:55:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:11 smithi110 ceph-mon[97633]: osdmap e493: 12 total, 12 up, 12 in 2022-09-10T07:55:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:11.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:11.465829+0000) 2022-09-10T07:55:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:11.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:11.829009+0000) 2022-09-10T07:55:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:12.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:12.466048+0000) 2022-09-10T07:55:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:12.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:12.829263+0000) 2022-09-10T07:55:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:13 smithi104 ceph-mon[97044]: pgmap v6593: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T07:55:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:13 smithi110 ceph-mon[97633]: pgmap v6593: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T07:55:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:13.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:13.466350+0000) 2022-09-10T07:55:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:13.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:13.829497+0000) 2022-09-10T07:55:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:14.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:14.466496+0000) 2022-09-10T07:55:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:14.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:14.829715+0000) 2022-09-10T07:55:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:15 smithi104 ceph-mon[97044]: pgmap v6594: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T07:55:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:15 smithi110 ceph-mon[97633]: pgmap v6594: 129 pgs: 129 active+clean; 8.4 MiB data, 734 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T07:55:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:15.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:15.466742+0000) 2022-09-10T07:55:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:15.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:15.829932+0000) 2022-09-10T07:55:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:16.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:16.466965+0000) 2022-09-10T07:55:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:16.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:16.830059+0000) 2022-09-10T07:55:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:17 smithi104 ceph-mon[97044]: pgmap v6595: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T07:55:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:17 smithi110 ceph-mon[97633]: pgmap v6595: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T07:55:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:17.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:17.467250+0000) 2022-09-10T07:55:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:55:17.569+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:55:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:17.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:17.830268+0000) 2022-09-10T07:55:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:18.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:18.467504+0000) 2022-09-10T07:55:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:18.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:18.830467+0000) 2022-09-10T07:55:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:19 smithi104 ceph-mon[97044]: pgmap v6596: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:55:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:19 smithi110 ceph-mon[97633]: pgmap v6596: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:55:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:19.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:19.467710+0000) 2022-09-10T07:55:20.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:19.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:19.830633+0000) 2022-09-10T07:55:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:20.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:20.467904+0000) 2022-09-10T07:55:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:20.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:20.830849+0000) 2022-09-10T07:55:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:21 smithi104 ceph-mon[97044]: pgmap v6597: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:55:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:21 smithi110 ceph-mon[97633]: pgmap v6597: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:55:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:21.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:21.468175+0000) 2022-09-10T07:55:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:21.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:21.831120+0000) 2022-09-10T07:55:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:22.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:22.468378+0000) 2022-09-10T07:55:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:22.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:22.831265+0000) 2022-09-10T07:55:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:23 smithi104 ceph-mon[97044]: pgmap v6598: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T07:55:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:23 smithi110 ceph-mon[97633]: pgmap v6598: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T07:55:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:23.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:23.468619+0000) 2022-09-10T07:55:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:23.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:23.831442+0000) 2022-09-10T07:55:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:24.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:24.468820+0000) 2022-09-10T07:55:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:24.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:24.831673+0000) 2022-09-10T07:55:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:25 smithi104 ceph-mon[97044]: pgmap v6599: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T07:55:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:25 smithi110 ceph-mon[97633]: pgmap v6599: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T07:55:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:25.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:25.469035+0000) 2022-09-10T07:55:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:25.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:25.831843+0000) 2022-09-10T07:55:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:26.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:26.469234+0000) 2022-09-10T07:55:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:26.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:26.832070+0000) 2022-09-10T07:55:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:27 smithi104 ceph-mon[97044]: pgmap v6600: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T07:55:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:27 smithi110 ceph-mon[97633]: pgmap v6600: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T07:55:27.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:27.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:27.469459+0000) 2022-09-10T07:55:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:27.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:27.832275+0000) 2022-09-10T07:55:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:28.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:28.469648+0000) 2022-09-10T07:55:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:28.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:28.832463+0000) 2022-09-10T07:55:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:29 smithi104 ceph-mon[97044]: pgmap v6601: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:29 smithi110 ceph-mon[97633]: pgmap v6601: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:29.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:29.469859+0000) 2022-09-10T07:55:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:29.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:29.832708+0000) 2022-09-10T07:55:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:30.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:30.470065+0000) 2022-09-10T07:55:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:30.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:30.832916+0000) 2022-09-10T07:55:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:31 smithi104 ceph-mon[97044]: pgmap v6602: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:31 smithi110 ceph-mon[97633]: pgmap v6602: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:31.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:31.470395+0000) 2022-09-10T07:55:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:31.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:31.833045+0000) 2022-09-10T07:55:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:32.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:32.470580+0000) 2022-09-10T07:55:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:55:32.577+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:55:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:32.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:32.833240+0000) 2022-09-10T07:55:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:33 smithi104 ceph-mon[97044]: pgmap v6603: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:33 smithi110 ceph-mon[97633]: pgmap v6603: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:33.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:33.470802+0000) 2022-09-10T07:55:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:33.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:33.833442+0000) 2022-09-10T07:55:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:34.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:34.471076+0000) 2022-09-10T07:55:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:34.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:34.833646+0000) 2022-09-10T07:55:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:35 smithi104 ceph-mon[97044]: pgmap v6604: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:35 smithi110 ceph-mon[97633]: pgmap v6604: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:35.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:35.471344+0000) 2022-09-10T07:55:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:35.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:35.833904+0000) 2022-09-10T07:55:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:36.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:36.471572+0000) 2022-09-10T07:55:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:36.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:36.834154+0000) 2022-09-10T07:55:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:37 smithi104 ceph-mon[97044]: pgmap v6605: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:37 smithi110 ceph-mon[97633]: pgmap v6605: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:37.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:37.471854+0000) 2022-09-10T07:55:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:37.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:37.834387+0000) 2022-09-10T07:55:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:38.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:38.472124+0000) 2022-09-10T07:55:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:38.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:38.834624+0000) 2022-09-10T07:55:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:39 smithi104 ceph-mon[97044]: pgmap v6606: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:39 smithi110 ceph-mon[97633]: pgmap v6606: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:39.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:39.472352+0000) 2022-09-10T07:55:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:39.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:39.834890+0000) 2022-09-10T07:55:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:40.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:40.472567+0000) 2022-09-10T07:55:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:40.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:40.835167+0000) 2022-09-10T07:55:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:41 smithi104 ceph-mon[97044]: pgmap v6607: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:41 smithi110 ceph-mon[97633]: pgmap v6607: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:41.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:41.472913+0000) 2022-09-10T07:55:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:41.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:41.835430+0000) 2022-09-10T07:55:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:42.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:42.473179+0000) 2022-09-10T07:55:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:42.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:42.835714+0000) 2022-09-10T07:55:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:43 smithi104 ceph-mon[97044]: pgmap v6608: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:43 smithi110 ceph-mon[97633]: pgmap v6608: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:43.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:43.473404+0000) 2022-09-10T07:55:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:43.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:43.835924+0000) 2022-09-10T07:55:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:44.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:44.473639+0000) 2022-09-10T07:55:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:44.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:44.836226+0000) 2022-09-10T07:55:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:45 smithi104 ceph-mon[97044]: pgmap v6609: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:55:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:55:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:55:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:55:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:45 smithi110 ceph-mon[97633]: pgmap v6609: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:55:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:55:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:55:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:55:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:45.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:45.473938+0000) 2022-09-10T07:55:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:45.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:45.836434+0000) 2022-09-10T07:55:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:46.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:46.474190+0000) 2022-09-10T07:55:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:46.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:46.836667+0000) 2022-09-10T07:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:47 smithi104 ceph-mon[97044]: pgmap v6610: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:47 smithi110 ceph-mon[97633]: pgmap v6610: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:47.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:47.474418+0000) 2022-09-10T07:55:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:55:47.589+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:55:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:47.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:47.836930+0000) 2022-09-10T07:55:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:48.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:48.474595+0000) 2022-09-10T07:55:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:48.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:48.837194+0000) 2022-09-10T07:55:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:49 smithi104 ceph-mon[97044]: pgmap v6611: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:49 smithi110 ceph-mon[97633]: pgmap v6611: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:49.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:49.474751+0000) 2022-09-10T07:55:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:49.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:49.837392+0000) 2022-09-10T07:55:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:50.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:50.474904+0000) 2022-09-10T07:55:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:50.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:50.837569+0000) 2022-09-10T07:55:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:51 smithi104 ceph-mon[97044]: pgmap v6612: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:51 smithi110 ceph-mon[97633]: pgmap v6612: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:51.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:51.475174+0000) 2022-09-10T07:55:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:51.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:51.837828+0000) 2022-09-10T07:55:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:52.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:52.475354+0000) 2022-09-10T07:55:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:52.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:52.838109+0000) 2022-09-10T07:55:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:53 smithi110 ceph-mon[97633]: pgmap v6613: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:53 smithi104 ceph-mon[97044]: pgmap v6613: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:53.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:53.475576+0000) 2022-09-10T07:55:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:53.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:53.838276+0000) 2022-09-10T07:55:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:54.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:54.475793+0000) 2022-09-10T07:55:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:54.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:54.838464+0000) 2022-09-10T07:55:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:55 smithi110 ceph-mon[97633]: pgmap v6614: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:55 smithi104 ceph-mon[97044]: pgmap v6614: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:55:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:55.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:55.476018+0000) 2022-09-10T07:55:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:55.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:55.838664+0000) 2022-09-10T07:55:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:56.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:56.476232+0000) 2022-09-10T07:55:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:56.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:56.838880+0000) 2022-09-10T07:55:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:57 smithi110 ceph-mon[97633]: pgmap v6615: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:57 smithi104 ceph-mon[97044]: pgmap v6615: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:55:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:57.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:57.476487+0000) 2022-09-10T07:55:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:57.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:57.839178+0000) 2022-09-10T07:55:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:58.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:58.476745+0000) 2022-09-10T07:55:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:58.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:58.839352+0000) 2022-09-10T07:55:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:55:59 smithi110 ceph-mon[97633]: pgmap v6616: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:55:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:55:59 smithi104 ceph-mon[97044]: pgmap v6616: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:55:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:55:59.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:59.477004+0000) 2022-09-10T07:56:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:55:59.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:55:59.839567+0000) 2022-09-10T07:56:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:00.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:00.477182+0000) 2022-09-10T07:56:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:00.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:00.839817+0000) 2022-09-10T07:56:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:01 smithi110 ceph-mon[97633]: pgmap v6617: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:01 smithi104 ceph-mon[97044]: pgmap v6617: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:01.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:01.477437+0000) 2022-09-10T07:56:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:01.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:01.840024+0000) 2022-09-10T07:56:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:02.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:02.477667+0000) 2022-09-10T07:56:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:56:02.605+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:56:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:02.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:02.840186+0000) 2022-09-10T07:56:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:03 smithi110 ceph-mon[97633]: pgmap v6618: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:56:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:03 smithi104 ceph-mon[97044]: pgmap v6618: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:56:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:03.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:03.477901+0000) 2022-09-10T07:56:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:03.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:03.840423+0000) 2022-09-10T07:56:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:04.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:04.478079+0000) 2022-09-10T07:56:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:04.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:04.840622+0000) 2022-09-10T07:56:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:05 smithi110 ceph-mon[97633]: pgmap v6619: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:56:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:05 smithi104 ceph-mon[97044]: pgmap v6619: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T07:56:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:05.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:05.478238+0000) 2022-09-10T07:56:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:05.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:05.840834+0000) 2022-09-10T07:56:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:06.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:06.478449+0000) 2022-09-10T07:56:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:06.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:06.840975+0000) 2022-09-10T07:56:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:07 smithi110 ceph-mon[97633]: pgmap v6620: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:56:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:07 smithi104 ceph-mon[97044]: pgmap v6620: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:56:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:07.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:07.478680+0000) 2022-09-10T07:56:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:07.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:07.841233+0000) 2022-09-10T07:56:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:08 smithi110 ceph-mon[97633]: osdmap e494: 12 total, 12 up, 12 in 2022-09-10T07:56:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:08 smithi104 ceph-mon[97044]: osdmap e494: 12 total, 12 up, 12 in 2022-09-10T07:56:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:08.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:08.478897+0000) 2022-09-10T07:56:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:08.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:08.841482+0000) 2022-09-10T07:56:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:09 smithi110 ceph-mon[97633]: pgmap v6622: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:09 smithi104 ceph-mon[97044]: pgmap v6622: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:09.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:09.479177+0000) 2022-09-10T07:56:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:09.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:09.841680+0000) 2022-09-10T07:56:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:10.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:10.479356+0000) 2022-09-10T07:56:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:10.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:10.841843+0000) 2022-09-10T07:56:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:11 smithi110 ceph-mon[97633]: pgmap v6623: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:11 smithi104 ceph-mon[97044]: pgmap v6623: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:11.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:11.479549+0000) 2022-09-10T07:56:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:11.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:11.842029+0000) 2022-09-10T07:56:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:12.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:12.479757+0000) 2022-09-10T07:56:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:12.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:12.842296+0000) 2022-09-10T07:56:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:13 smithi110 ceph-mon[97633]: pgmap v6624: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:13 smithi104 ceph-mon[97044]: pgmap v6624: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:13.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:13.479999+0000) 2022-09-10T07:56:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:13.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:13.842510+0000) 2022-09-10T07:56:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:14.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:14.480261+0000) 2022-09-10T07:56:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:14.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:14.842727+0000) 2022-09-10T07:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:15 smithi110 ceph-mon[97633]: pgmap v6625: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:15 smithi110 ceph-mon[97633]: osdmap e495: 12 total, 12 up, 12 in 2022-09-10T07:56:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:15 smithi104 ceph-mon[97044]: pgmap v6625: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:56:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:15 smithi104 ceph-mon[97044]: osdmap e495: 12 total, 12 up, 12 in 2022-09-10T07:56:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:15.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:15.480416+0000) 2022-09-10T07:56:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:15.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:15.842972+0000) 2022-09-10T07:56:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:16.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:16.480578+0000) 2022-09-10T07:56:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:16.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:16.843200+0000) 2022-09-10T07:56:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:17 smithi110 ceph-mon[97633]: pgmap v6627: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T07:56:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:17 smithi104 ceph-mon[97044]: pgmap v6627: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T07:56:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:17.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:17.480810+0000) 2022-09-10T07:56:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:56:17.620+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:56:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:17.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:17.843362+0000) 2022-09-10T07:56:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:18.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:18.481036+0000) 2022-09-10T07:56:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:18.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:18.843587+0000) 2022-09-10T07:56:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:19 smithi110 ceph-mon[97633]: pgmap v6628: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:56:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:19 smithi104 ceph-mon[97044]: pgmap v6628: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:56:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:19.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:19.481231+0000) 2022-09-10T07:56:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:19.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:19.843798+0000) 2022-09-10T07:56:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:20.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:20.481445+0000) 2022-09-10T07:56:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:20.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:20.844072+0000) 2022-09-10T07:56:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:21 smithi110 ceph-mon[97633]: pgmap v6629: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:56:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:21 smithi104 ceph-mon[97044]: pgmap v6629: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:56:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:21.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:21.481632+0000) 2022-09-10T07:56:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:21.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:21.844330+0000) 2022-09-10T07:56:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:22.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:22.481839+0000) 2022-09-10T07:56:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:22.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:22.844512+0000) 2022-09-10T07:56:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:23 smithi110 ceph-mon[97633]: pgmap v6630: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:56:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:23 smithi104 ceph-mon[97044]: pgmap v6630: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:56:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:23.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:23.482074+0000) 2022-09-10T07:56:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:23.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:23.844740+0000) 2022-09-10T07:56:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:24.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:24.482295+0000) 2022-09-10T07:56:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:24.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:24.845011+0000) 2022-09-10T07:56:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:25 smithi110 ceph-mon[97633]: pgmap v6631: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:56:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:25 smithi104 ceph-mon[97044]: pgmap v6631: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:56:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:25.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:25.482544+0000) 2022-09-10T07:56:26.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:25.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:25.845199+0000) 2022-09-10T07:56:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:26.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:26.482779+0000) 2022-09-10T07:56:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:26.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:26.845436+0000) 2022-09-10T07:56:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:27 smithi110 ceph-mon[97633]: pgmap v6632: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:56:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:27 smithi104 ceph-mon[97044]: pgmap v6632: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:56:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:27.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:27.483017+0000) 2022-09-10T07:56:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:27.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:27.845689+0000) 2022-09-10T07:56:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:28.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:28.483252+0000) 2022-09-10T07:56:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:28.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:28.845937+0000) 2022-09-10T07:56:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:29 smithi110 ceph-mon[97633]: pgmap v6633: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:29 smithi104 ceph-mon[97044]: pgmap v6633: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:29.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:29.483466+0000) 2022-09-10T07:56:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:29.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:29.846180+0000) 2022-09-10T07:56:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:30.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:30.483699+0000) 2022-09-10T07:56:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:30.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:30.846416+0000) 2022-09-10T07:56:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:31 smithi110 ceph-mon[97633]: pgmap v6634: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:31 smithi104 ceph-mon[97044]: pgmap v6634: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:31.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:31.483974+0000) 2022-09-10T07:56:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:31.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:31.846644+0000) 2022-09-10T07:56:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:32.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:32.484223+0000) 2022-09-10T07:56:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:56:32.635+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:56:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:32.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:32.846815+0000) 2022-09-10T07:56:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:33 smithi110 ceph-mon[97633]: pgmap v6635: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:33 smithi104 ceph-mon[97044]: pgmap v6635: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:33.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:33.484490+0000) 2022-09-10T07:56:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:33.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:33.847031+0000) 2022-09-10T07:56:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:34.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:34.484699+0000) 2022-09-10T07:56:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:34.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:34.847271+0000) 2022-09-10T07:56:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:35 smithi110 ceph-mon[97633]: pgmap v6636: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:35 smithi104 ceph-mon[97044]: pgmap v6636: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:35.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:35.484931+0000) 2022-09-10T07:56:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:35.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:35.847535+0000) 2022-09-10T07:56:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:36.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:36.485172+0000) 2022-09-10T07:56:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:36.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:36.847810+0000) 2022-09-10T07:56:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:37 smithi110 ceph-mon[97633]: pgmap v6637: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:37 smithi104 ceph-mon[97044]: pgmap v6637: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:37.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:37.485457+0000) 2022-09-10T07:56:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:37.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:37.848077+0000) 2022-09-10T07:56:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:38.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:38.485665+0000) 2022-09-10T07:56:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:38.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:38.848358+0000) 2022-09-10T07:56:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:39 smithi110 ceph-mon[97633]: pgmap v6638: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:39 smithi104 ceph-mon[97044]: pgmap v6638: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:39.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:39.485905+0000) 2022-09-10T07:56:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:39.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:39.848507+0000) 2022-09-10T07:56:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:40.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:40.486089+0000) 2022-09-10T07:56:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:40.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:40.848666+0000) 2022-09-10T07:56:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:41 smithi110 ceph-mon[97633]: pgmap v6639: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:41 smithi104 ceph-mon[97044]: pgmap v6639: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:41.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:41.486358+0000) 2022-09-10T07:56:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:41.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:41.848887+0000) 2022-09-10T07:56:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:42.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:42.486621+0000) 2022-09-10T07:56:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:42.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:42.849091+0000) 2022-09-10T07:56:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:43 smithi110 ceph-mon[97633]: pgmap v6640: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:43 smithi104 ceph-mon[97044]: pgmap v6640: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:43.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:43.486863+0000) 2022-09-10T07:56:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:43.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:43.849241+0000) 2022-09-10T07:56:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:44.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:44.487049+0000) 2022-09-10T07:56:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:44.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:44.849475+0000) 2022-09-10T07:56:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:45 smithi110 ceph-mon[97633]: pgmap v6641: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:56:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:56:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:56:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:56:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:45 smithi104 ceph-mon[97044]: pgmap v6641: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:56:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:56:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:56:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:56:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:45.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:45.487314+0000) 2022-09-10T07:56:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:45.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:45.849657+0000) 2022-09-10T07:56:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:46.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:46.487561+0000) 2022-09-10T07:56:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:46.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:46.849835+0000) 2022-09-10T07:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:47 smithi110 ceph-mon[97633]: pgmap v6642: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:47 smithi104 ceph-mon[97044]: pgmap v6642: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:47.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:47.487789+0000) 2022-09-10T07:56:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:56:47.644+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:56:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:47.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:47.850127+0000) 2022-09-10T07:56:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:48.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:48.487972+0000) 2022-09-10T07:56:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:48.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:48.850408+0000) 2022-09-10T07:56:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:49 smithi110 ceph-mon[97633]: pgmap v6643: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:49 smithi104 ceph-mon[97044]: pgmap v6643: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:49.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:49.488205+0000) 2022-09-10T07:56:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:49.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:49.850601+0000) 2022-09-10T07:56:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:50.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:50.488417+0000) 2022-09-10T07:56:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:50.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:50.850730+0000) 2022-09-10T07:56:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:51 smithi110 ceph-mon[97633]: pgmap v6644: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:51 smithi104 ceph-mon[97044]: pgmap v6644: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:51.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:51.488678+0000) 2022-09-10T07:56:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:51.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:51.850944+0000) 2022-09-10T07:56:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:52.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:52.488909+0000) 2022-09-10T07:56:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:52.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:52.851213+0000) 2022-09-10T07:56:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:53 smithi110 ceph-mon[97633]: pgmap v6645: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:53 smithi104 ceph-mon[97044]: pgmap v6645: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:53.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:53.489197+0000) 2022-09-10T07:56:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:53.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:53.851389+0000) 2022-09-10T07:56:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:54.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:54.489470+0000) 2022-09-10T07:56:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:54.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:54.851623+0000) 2022-09-10T07:56:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:55 smithi110 ceph-mon[97633]: pgmap v6646: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:55 smithi104 ceph-mon[97044]: pgmap v6646: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:55.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:55.489668+0000) 2022-09-10T07:56:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:55.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:55.851885+0000) 2022-09-10T07:56:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:56.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:56.489918+0000) 2022-09-10T07:56:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:56.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:56.852199+0000) 2022-09-10T07:56:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:57 smithi110 ceph-mon[97633]: pgmap v6647: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:57 smithi104 ceph-mon[97044]: pgmap v6647: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:56:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:57.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:57.490185+0000) 2022-09-10T07:56:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:57.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:57.852364+0000) 2022-09-10T07:56:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:58.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:58.490351+0000) 2022-09-10T07:56:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:58.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:58.852558+0000) 2022-09-10T07:56:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:56:59 smithi110 ceph-mon[97633]: pgmap v6648: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:56:59 smithi104 ceph-mon[97044]: pgmap v6648: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:56:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:56:59.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:59.490511+0000) 2022-09-10T07:57:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:56:59.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:56:59.852696+0000) 2022-09-10T07:57:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:00.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:00.490771+0000) 2022-09-10T07:57:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:00.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:00.852844+0000) 2022-09-10T07:57:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:01 smithi110 ceph-mon[97633]: pgmap v6649: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:01 smithi104 ceph-mon[97044]: pgmap v6649: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:01.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:01.490939+0000) 2022-09-10T07:57:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:01.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:01.853067+0000) 2022-09-10T07:57:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:02.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:02.491115+0000) 2022-09-10T07:57:02.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:57:02.648+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:57:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:02.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:02.853262+0000) 2022-09-10T07:57:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:03 smithi110 ceph-mon[97633]: pgmap v6650: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:03 smithi104 ceph-mon[97044]: pgmap v6650: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:03.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:03.491354+0000) 2022-09-10T07:57:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:03.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:03.853435+0000) 2022-09-10T07:57:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:04.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:04.491527+0000) 2022-09-10T07:57:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:04.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:04.853686+0000) 2022-09-10T07:57:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:05 smithi104 ceph-mon[97044]: pgmap v6651: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:05.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:05.491699+0000) 2022-09-10T07:57:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:05 smithi110 ceph-mon[97633]: pgmap v6651: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:05.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:05.853922+0000) 2022-09-10T07:57:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:06.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:06.491839+0000) 2022-09-10T07:57:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:06.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:06.854198+0000) 2022-09-10T07:57:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:07 smithi104 ceph-mon[97044]: pgmap v6652: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:57:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:07.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:07.492070+0000) 2022-09-10T07:57:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:07 smithi110 ceph-mon[97633]: pgmap v6652: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:57:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:07.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:07.854431+0000) 2022-09-10T07:57:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:08 smithi104 ceph-mon[97044]: osdmap e496: 12 total, 12 up, 12 in 2022-09-10T07:57:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:08.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:08.492316+0000) 2022-09-10T07:57:08.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:08 smithi110 ceph-mon[97633]: osdmap e496: 12 total, 12 up, 12 in 2022-09-10T07:57:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:08.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:08.854666+0000) 2022-09-10T07:57:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:09 smithi104 ceph-mon[97044]: pgmap v6654: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:09.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:09.492540+0000) 2022-09-10T07:57:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:09 smithi110 ceph-mon[97633]: pgmap v6654: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:09.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:09.854873+0000) 2022-09-10T07:57:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:10.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:10.492750+0000) 2022-09-10T07:57:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:10.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:10.855081+0000) 2022-09-10T07:57:11.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:11 smithi104 ceph-mon[97044]: pgmap v6655: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:11.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:11.493008+0000) 2022-09-10T07:57:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:11 smithi110 ceph-mon[97633]: pgmap v6655: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:11.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:11.855266+0000) 2022-09-10T07:57:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:12.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:12.493228+0000) 2022-09-10T07:57:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:12.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:12.855476+0000) 2022-09-10T07:57:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:13 smithi104 ceph-mon[97044]: pgmap v6656: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:13.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:13.493430+0000) 2022-09-10T07:57:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:13 smithi110 ceph-mon[97633]: pgmap v6656: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:13.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:13.855648+0000) 2022-09-10T07:57:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:14.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:14.493599+0000) 2022-09-10T07:57:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:14.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:14.855860+0000) 2022-09-10T07:57:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:15 smithi104 ceph-mon[97044]: pgmap v6657: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:15 smithi104 ceph-mon[97044]: osdmap e497: 12 total, 12 up, 12 in 2022-09-10T07:57:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:15.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:15.493799+0000) 2022-09-10T07:57:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:15 smithi110 ceph-mon[97633]: pgmap v6657: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:57:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:15 smithi110 ceph-mon[97633]: osdmap e497: 12 total, 12 up, 12 in 2022-09-10T07:57:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:15.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:15.856183+0000) 2022-09-10T07:57:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:16.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:16.494036+0000) 2022-09-10T07:57:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:16.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:16.856428+0000) 2022-09-10T07:57:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:17 smithi104 ceph-mon[97044]: pgmap v6659: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T07:57:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:17.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:17.494223+0000) 2022-09-10T07:57:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:57:17.657+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:57:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:17 smithi110 ceph-mon[97633]: pgmap v6659: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T07:57:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:17.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:17.856681+0000) 2022-09-10T07:57:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:18.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:18.494443+0000) 2022-09-10T07:57:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:18.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:18.856886+0000) 2022-09-10T07:57:19.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:19 smithi104 ceph-mon[97044]: pgmap v6660: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:57:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:19.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:19.494674+0000) 2022-09-10T07:57:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:19 smithi110 ceph-mon[97633]: pgmap v6660: 129 pgs: 129 active+clean; 8.4 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:57:20.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:19.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:19.857112+0000) 2022-09-10T07:57:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:20.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:20.494899+0000) 2022-09-10T07:57:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:20.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:20.857327+0000) 2022-09-10T07:57:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:21 smithi104 ceph-mon[97044]: pgmap v6661: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:57:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:21.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:21.495165+0000) 2022-09-10T07:57:21.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:21 smithi110 ceph-mon[97633]: pgmap v6661: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:57:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:21.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:21.857535+0000) 2022-09-10T07:57:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:22.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:22.495333+0000) 2022-09-10T07:57:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:22.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:22.857698+0000) 2022-09-10T07:57:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:23 smithi104 ceph-mon[97044]: pgmap v6662: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:57:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:23.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:23.495511+0000) 2022-09-10T07:57:23.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:23 smithi110 ceph-mon[97633]: pgmap v6662: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:57:24.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:23.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:23.857924+0000) 2022-09-10T07:57:24.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:24.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:24.495683+0000) 2022-09-10T07:57:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:24.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:24.858178+0000) 2022-09-10T07:57:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:25 smithi104 ceph-mon[97044]: pgmap v6663: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:57:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:25.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:25.495915+0000) 2022-09-10T07:57:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:25 smithi110 ceph-mon[97633]: pgmap v6663: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:57:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:25.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:25.858415+0000) 2022-09-10T07:57:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:26.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:26.496150+0000) 2022-09-10T07:57:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:26.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:26.858630+0000) 2022-09-10T07:57:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:27 smithi104 ceph-mon[97044]: pgmap v6664: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:57:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:27.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:27.496324+0000) 2022-09-10T07:57:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:27 smithi110 ceph-mon[97633]: pgmap v6664: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:57:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:27.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:27.858781+0000) 2022-09-10T07:57:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:28.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:28.496523+0000) 2022-09-10T07:57:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:28.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:28.858975+0000) 2022-09-10T07:57:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:29 smithi104 ceph-mon[97044]: pgmap v6665: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:29.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:29.496709+0000) 2022-09-10T07:57:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:29 smithi110 ceph-mon[97633]: pgmap v6665: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:29.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:29.859207+0000) 2022-09-10T07:57:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:30.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:30.496955+0000) 2022-09-10T07:57:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:30.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:30.859474+0000) 2022-09-10T07:57:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:31 smithi104 ceph-mon[97044]: pgmap v6666: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:31.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:31.497216+0000) 2022-09-10T07:57:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:31 smithi110 ceph-mon[97633]: pgmap v6666: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:31.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:31.859708+0000) 2022-09-10T07:57:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:57:32.658+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:57:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:32.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:32.497395+0000) 2022-09-10T07:57:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:32.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:32.859922+0000) 2022-09-10T07:57:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:33 smithi104 ceph-mon[97044]: pgmap v6667: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:33.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:33.497668+0000) 2022-09-10T07:57:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:33 smithi110 ceph-mon[97633]: pgmap v6667: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:33.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:33.860165+0000) 2022-09-10T07:57:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:34.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:34.497863+0000) 2022-09-10T07:57:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:34.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:34.860402+0000) 2022-09-10T07:57:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:35 smithi104 ceph-mon[97044]: pgmap v6668: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:35.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:35.498064+0000) 2022-09-10T07:57:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:35 smithi110 ceph-mon[97633]: pgmap v6668: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:35.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:35.860636+0000) 2022-09-10T07:57:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:36.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:36.498245+0000) 2022-09-10T07:57:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:36.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:36.860889+0000) 2022-09-10T07:57:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:37 smithi104 ceph-mon[97044]: pgmap v6669: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:37.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:37.498404+0000) 2022-09-10T07:57:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:37 smithi110 ceph-mon[97633]: pgmap v6669: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:37.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:37.861090+0000) 2022-09-10T07:57:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:38.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:38.498602+0000) 2022-09-10T07:57:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:38.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:38.861326+0000) 2022-09-10T07:57:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:39 smithi104 ceph-mon[97044]: pgmap v6670: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:39.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:39.498851+0000) 2022-09-10T07:57:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:39 smithi110 ceph-mon[97633]: pgmap v6670: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:39.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:39.861470+0000) 2022-09-10T07:57:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:40.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:40.499042+0000) 2022-09-10T07:57:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:40.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:40.861630+0000) 2022-09-10T07:57:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:41 smithi104 ceph-mon[97044]: pgmap v6671: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:41.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:41.499280+0000) 2022-09-10T07:57:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:41 smithi110 ceph-mon[97633]: pgmap v6671: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:41.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:41.861840+0000) 2022-09-10T07:57:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:42.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:42.499494+0000) 2022-09-10T07:57:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:42.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:42.862035+0000) 2022-09-10T07:57:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:43 smithi104 ceph-mon[97044]: pgmap v6672: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:43.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:43.499747+0000) 2022-09-10T07:57:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:43 smithi110 ceph-mon[97633]: pgmap v6672: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:43.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:43.862277+0000) 2022-09-10T07:57:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:57:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:57:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:44.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:44.499951+0000) 2022-09-10T07:57:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:57:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:57:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:57:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:57:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:44.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:44.862478+0000) 2022-09-10T07:57:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:45 smithi104 ceph-mon[97044]: pgmap v6673: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:45.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:45.500190+0000) 2022-09-10T07:57:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:45 smithi110 ceph-mon[97633]: pgmap v6673: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:45.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:45.862726+0000) 2022-09-10T07:57:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:46.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:46.500443+0000) 2022-09-10T07:57:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:46.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:46.862962+0000) 2022-09-10T07:57:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:47 smithi104 ceph-mon[97044]: pgmap v6674: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:47.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:57:47.674+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:57:47.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:47.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:47.500652+0000) 2022-09-10T07:57:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:47 smithi110 ceph-mon[97633]: pgmap v6674: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:47.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:47.863193+0000) 2022-09-10T07:57:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:48.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:48.500881+0000) 2022-09-10T07:57:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:48.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:48.863363+0000) 2022-09-10T07:57:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:49 smithi104 ceph-mon[97044]: pgmap v6675: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:49.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:49.501145+0000) 2022-09-10T07:57:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:49 smithi110 ceph-mon[97633]: pgmap v6675: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:49.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:49.863592+0000) 2022-09-10T07:57:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:50.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:50.501361+0000) 2022-09-10T07:57:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:50.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:50.863827+0000) 2022-09-10T07:57:51.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:51 smithi104 ceph-mon[97044]: pgmap v6676: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:51.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:51.501585+0000) 2022-09-10T07:57:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:51 smithi110 ceph-mon[97633]: pgmap v6676: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:51.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:51.864069+0000) 2022-09-10T07:57:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:52.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:52.501797+0000) 2022-09-10T07:57:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:52.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:52.864280+0000) 2022-09-10T07:57:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:53 smithi104 ceph-mon[97044]: pgmap v6677: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:53.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:53.502079+0000) 2022-09-10T07:57:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:53 smithi110 ceph-mon[97633]: pgmap v6677: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:53.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:53.864529+0000) 2022-09-10T07:57:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:54.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:54.502271+0000) 2022-09-10T07:57:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:54.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:54.864751+0000) 2022-09-10T07:57:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:55 smithi104 ceph-mon[97044]: pgmap v6678: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:55.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:55.502536+0000) 2022-09-10T07:57:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:55 smithi110 ceph-mon[97633]: pgmap v6678: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:55.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:55.864937+0000) 2022-09-10T07:57:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:56.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:56.502750+0000) 2022-09-10T07:57:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:56.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:56.865151+0000) 2022-09-10T07:57:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:57 smithi104 ceph-mon[97044]: pgmap v6679: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:57.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:57.502975+0000) 2022-09-10T07:57:57.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:57 smithi110 ceph-mon[97633]: pgmap v6679: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:57:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:57.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:57.865383+0000) 2022-09-10T07:57:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:58.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:58.503178+0000) 2022-09-10T07:57:59.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:58.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:58.865623+0000) 2022-09-10T07:57:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:57:59 smithi104 ceph-mon[97044]: pgmap v6680: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:57:59.818 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:57:59.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:59.503419+0000) 2022-09-10T07:57:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:57:59 smithi110 ceph-mon[97633]: pgmap v6680: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:00.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:57:59.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:57:59.865829+0000) 2022-09-10T07:58:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:00.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:00.503651+0000) 2022-09-10T07:58:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:00.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:00.866021+0000) 2022-09-10T07:58:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:01 smithi104 ceph-mon[97044]: pgmap v6681: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:01.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:01.503888+0000) 2022-09-10T07:58:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:01 smithi110 ceph-mon[97633]: pgmap v6681: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:01.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:01.866235+0000) 2022-09-10T07:58:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:58:02.676+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:58:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:02.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:02.504063+0000) 2022-09-10T07:58:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:02.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:02.866425+0000) 2022-09-10T07:58:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:03 smithi104 ceph-mon[97044]: pgmap v6682: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:03.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:03.504339+0000) 2022-09-10T07:58:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:03 smithi110 ceph-mon[97633]: pgmap v6682: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:03.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:03.866662+0000) 2022-09-10T07:58:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:04.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:04.504507+0000) 2022-09-10T07:58:05.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:04.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:04.866894+0000) 2022-09-10T07:58:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:05 smithi104 ceph-mon[97044]: pgmap v6683: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:05.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:05.504737+0000) 2022-09-10T07:58:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:05 smithi110 ceph-mon[97633]: pgmap v6683: 129 pgs: 129 active+clean; 8.4 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:05.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:05.867165+0000) 2022-09-10T07:58:06.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:06.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:06.504988+0000) 2022-09-10T07:58:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:06.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:06.867385+0000) 2022-09-10T07:58:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:07 smithi104 ceph-mon[97044]: pgmap v6684: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:58:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:07.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:07.505213+0000) 2022-09-10T07:58:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:07 smithi110 ceph-mon[97633]: pgmap v6684: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:58:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:07.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:07.867558+0000) 2022-09-10T07:58:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:08 smithi104 ceph-mon[97044]: osdmap e498: 12 total, 12 up, 12 in 2022-09-10T07:58:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:08.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:08.505448+0000) 2022-09-10T07:58:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:08 smithi110 ceph-mon[97633]: osdmap e498: 12 total, 12 up, 12 in 2022-09-10T07:58:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:08.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:08.867785+0000) 2022-09-10T07:58:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:09 smithi104 ceph-mon[97044]: pgmap v6686: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:09.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:09.505645+0000) 2022-09-10T07:58:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:09 smithi110 ceph-mon[97633]: pgmap v6686: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:09.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:09.868050+0000) 2022-09-10T07:58:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:10.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:10.505909+0000) 2022-09-10T07:58:11.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:10.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:10.868327+0000) 2022-09-10T07:58:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:11 smithi104 ceph-mon[97044]: pgmap v6687: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:11.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:11.506201+0000) 2022-09-10T07:58:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:11 smithi110 ceph-mon[97633]: pgmap v6687: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:11.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:11.868572+0000) 2022-09-10T07:58:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:12.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:12.506425+0000) 2022-09-10T07:58:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:12.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:12.868802+0000) 2022-09-10T07:58:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:13 smithi104 ceph-mon[97044]: pgmap v6688: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:13.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:13.506648+0000) 2022-09-10T07:58:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:13 smithi110 ceph-mon[97633]: pgmap v6688: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:13.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:13.869013+0000) 2022-09-10T07:58:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:14.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:14.506917+0000) 2022-09-10T07:58:15.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:14.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:14.869265+0000) 2022-09-10T07:58:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:15 smithi104 ceph-mon[97044]: pgmap v6689: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:15 smithi104 ceph-mon[97044]: osdmap e499: 12 total, 12 up, 12 in 2022-09-10T07:58:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:15.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:15.507172+0000) 2022-09-10T07:58:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:15 smithi110 ceph-mon[97633]: pgmap v6689: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:58:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:15 smithi110 ceph-mon[97633]: osdmap e499: 12 total, 12 up, 12 in 2022-09-10T07:58:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:15.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:15.869562+0000) 2022-09-10T07:58:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:16.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:16.507376+0000) 2022-09-10T07:58:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:16.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:16.869850+0000) 2022-09-10T07:58:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:17 smithi104 ceph-mon[97044]: pgmap v6691: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T07:58:17.823 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:17.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:17.507623+0000) 2022-09-10T07:58:17.824 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:58:17.690+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:58:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:17 smithi110 ceph-mon[97633]: pgmap v6691: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T07:58:18.313 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:17.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:17.870142+0000) 2022-09-10T07:58:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:18.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:18.507868+0000) 2022-09-10T07:58:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:18.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:18.870365+0000) 2022-09-10T07:58:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:19 smithi104 ceph-mon[97044]: pgmap v6692: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:58:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:19.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:19.508149+0000) 2022-09-10T07:58:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:19 smithi110 ceph-mon[97633]: pgmap v6692: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:58:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:19.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:19.870605+0000) 2022-09-10T07:58:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:20.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:20.508302+0000) 2022-09-10T07:58:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:20.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:20.870888+0000) 2022-09-10T07:58:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:21 smithi104 ceph-mon[97044]: pgmap v6693: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:58:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:21.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:21.508464+0000) 2022-09-10T07:58:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:21 smithi110 ceph-mon[97633]: pgmap v6693: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:58:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:21.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:21.871133+0000) 2022-09-10T07:58:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:22.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:22.508631+0000) 2022-09-10T07:58:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:22.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:22.871403+0000) 2022-09-10T07:58:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:23.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:23.508878+0000) 2022-09-10T07:58:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:23 smithi110 ceph-mon[97633]: pgmap v6694: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:58:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:23 smithi104 ceph-mon[97044]: pgmap v6694: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:58:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:23.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:23.871642+0000) 2022-09-10T07:58:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:24.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:24.509141+0000) 2022-09-10T07:58:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:24.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:24.871785+0000) 2022-09-10T07:58:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:25.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:25.509326+0000) 2022-09-10T07:58:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:25 smithi110 ceph-mon[97633]: pgmap v6695: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:58:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:25 smithi104 ceph-mon[97044]: pgmap v6695: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:58:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:25.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:25.871994+0000) 2022-09-10T07:58:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:26.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:26.509552+0000) 2022-09-10T07:58:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:26.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:26.872228+0000) 2022-09-10T07:58:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:27.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:27.509795+0000) 2022-09-10T07:58:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:27 smithi110 ceph-mon[97633]: pgmap v6696: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:58:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:27 smithi104 ceph-mon[97044]: pgmap v6696: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:58:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:27.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:27.872402+0000) 2022-09-10T07:58:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:28.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:28.510029+0000) 2022-09-10T07:58:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:28.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:28.872652+0000) 2022-09-10T07:58:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:29.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:29.510286+0000) 2022-09-10T07:58:29.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:29 smithi110 ceph-mon[97633]: pgmap v6697: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:29 smithi104 ceph-mon[97044]: pgmap v6697: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:29.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:29.872896+0000) 2022-09-10T07:58:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:30.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:30.510521+0000) 2022-09-10T07:58:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:30.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:30.873181+0000) 2022-09-10T07:58:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:31.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:31.510745+0000) 2022-09-10T07:58:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:31 smithi110 ceph-mon[97633]: pgmap v6698: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:31 smithi104 ceph-mon[97044]: pgmap v6698: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:31.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:31.873394+0000) 2022-09-10T07:58:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:32.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:32.510995+0000) 2022-09-10T07:58:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:58:32.705+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:58:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:32.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:32.873634+0000) 2022-09-10T07:58:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:33.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:33.511269+0000) 2022-09-10T07:58:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:33 smithi110 ceph-mon[97633]: pgmap v6699: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:33 smithi104 ceph-mon[97044]: pgmap v6699: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:33.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:33.873819+0000) 2022-09-10T07:58:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:34.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:34.511452+0000) 2022-09-10T07:58:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:34.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:34.874035+0000) 2022-09-10T07:58:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:35.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:35.511632+0000) 2022-09-10T07:58:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:35 smithi110 ceph-mon[97633]: pgmap v6700: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:35 smithi104 ceph-mon[97044]: pgmap v6700: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:35.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:35.874322+0000) 2022-09-10T07:58:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:36.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:36.511868+0000) 2022-09-10T07:58:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:36.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:36.874533+0000) 2022-09-10T07:58:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:37.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:37.512139+0000) 2022-09-10T07:58:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:37 smithi110 ceph-mon[97633]: pgmap v6701: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:37 smithi104 ceph-mon[97044]: pgmap v6701: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:37.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:37.874773+0000) 2022-09-10T07:58:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:38.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:38.512391+0000) 2022-09-10T07:58:39.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:38.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:38.874984+0000) 2022-09-10T07:58:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:39.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:39.512541+0000) 2022-09-10T07:58:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:39 smithi110 ceph-mon[97633]: pgmap v6702: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:39 smithi104 ceph-mon[97044]: pgmap v6702: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:39.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:39.875162+0000) 2022-09-10T07:58:40.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:40.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:40.512796+0000) 2022-09-10T07:58:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:40.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:40.875407+0000) 2022-09-10T07:58:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:41.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:41.512979+0000) 2022-09-10T07:58:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:41 smithi110 ceph-mon[97633]: pgmap v6703: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:41 smithi104 ceph-mon[97044]: pgmap v6703: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:41.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:41.875608+0000) 2022-09-10T07:58:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:42.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:42.513156+0000) 2022-09-10T07:58:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:42.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:42.875818+0000) 2022-09-10T07:58:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:43.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:43.513342+0000) 2022-09-10T07:58:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:43 smithi110 ceph-mon[97633]: pgmap v6704: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:43 smithi104 ceph-mon[97044]: pgmap v6704: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:43.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:43.876009+0000) 2022-09-10T07:58:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:44.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:44.513560+0000) 2022-09-10T07:58:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:58:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:58:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:58:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:58:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:58:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:58:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:58:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:58:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:44.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:44.876265+0000) 2022-09-10T07:58:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:45.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:45.513817+0000) 2022-09-10T07:58:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:45 smithi110 ceph-mon[97633]: pgmap v6705: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:45 smithi104 ceph-mon[97044]: pgmap v6705: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:45.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:45.876472+0000) 2022-09-10T07:58:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:46.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:46.514069+0000) 2022-09-10T07:58:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:46.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:46.876696+0000) 2022-09-10T07:58:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:47.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:47.514373+0000) 2022-09-10T07:58:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:58:47.714+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:58:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:47 smithi110 ceph-mon[97633]: pgmap v6706: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:47 smithi104 ceph-mon[97044]: pgmap v6706: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:47.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:47.876873+0000) 2022-09-10T07:58:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:48.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:48.514618+0000) 2022-09-10T07:58:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:48.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:48.877132+0000) 2022-09-10T07:58:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:49.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:49.514790+0000) 2022-09-10T07:58:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:49 smithi110 ceph-mon[97633]: pgmap v6707: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:49 smithi104 ceph-mon[97044]: pgmap v6707: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:49.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:49.877306+0000) 2022-09-10T07:58:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:50.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:50.514986+0000) 2022-09-10T07:58:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:50.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:50.877491+0000) 2022-09-10T07:58:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:51.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:51.515240+0000) 2022-09-10T07:58:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:51 smithi110 ceph-mon[97633]: pgmap v6708: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:51 smithi104 ceph-mon[97044]: pgmap v6708: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:51.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:51.877685+0000) 2022-09-10T07:58:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:52.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:52.515440+0000) 2022-09-10T07:58:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:52.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:52.877900+0000) 2022-09-10T07:58:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:53.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:53.515680+0000) 2022-09-10T07:58:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:53 smithi110 ceph-mon[97633]: pgmap v6709: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:53 smithi104 ceph-mon[97044]: pgmap v6709: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:53.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:53.878139+0000) 2022-09-10T07:58:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:54.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:54.515913+0000) 2022-09-10T07:58:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:54.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:54.878382+0000) 2022-09-10T07:58:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:55.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:55.516189+0000) 2022-09-10T07:58:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:55 smithi110 ceph-mon[97633]: pgmap v6710: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:55 smithi104 ceph-mon[97044]: pgmap v6710: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:55.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:55.878607+0000) 2022-09-10T07:58:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:56.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:56.516386+0000) 2022-09-10T07:58:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:56 smithi110 ceph-mon[97633]: pgmap v6711: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:56 smithi104 ceph-mon[97044]: pgmap v6711: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:58:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:56.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:56.878804+0000) 2022-09-10T07:58:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:57.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:57.516561+0000) 2022-09-10T07:58:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:57.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:57.879035+0000) 2022-09-10T07:58:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:58.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:58.516796+0000) 2022-09-10T07:58:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:58:58 smithi110 ceph-mon[97633]: pgmap v6712: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:58:58 smithi104 ceph-mon[97044]: pgmap v6712: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:58:59.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:58.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:58.879241+0000) 2022-09-10T07:58:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:58:59.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:59.516991+0000) 2022-09-10T07:59:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:58:59.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:58:59.879422+0000) 2022-09-10T07:59:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:00.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:00.517229+0000) 2022-09-10T07:59:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:00.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:00.879643+0000) 2022-09-10T07:59:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:01 smithi110 ceph-mon[97633]: pgmap v6713: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:01 smithi104 ceph-mon[97044]: pgmap v6713: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:01.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:01.517394+0000) 2022-09-10T07:59:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:01.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:01.879855+0000) 2022-09-10T07:59:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:02.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:02.517573+0000) 2022-09-10T07:59:02.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:59:02.723+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:59:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:02.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:02.880048+0000) 2022-09-10T07:59:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:03 smithi110 ceph-mon[97633]: pgmap v6714: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:03 smithi104 ceph-mon[97044]: pgmap v6714: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:03.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:03.517852+0000) 2022-09-10T07:59:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:03.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:03.880346+0000) 2022-09-10T07:59:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:04.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:04.518088+0000) 2022-09-10T07:59:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:04.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:04.880521+0000) 2022-09-10T07:59:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:05 smithi110 ceph-mon[97633]: pgmap v6715: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:05 smithi104 ceph-mon[97044]: pgmap v6715: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:05.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:05.518358+0000) 2022-09-10T07:59:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:05.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:05.880684+0000) 2022-09-10T07:59:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:06.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:06.518541+0000) 2022-09-10T07:59:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:06.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:06.880805+0000) 2022-09-10T07:59:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:07 smithi110 ceph-mon[97633]: pgmap v6716: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:59:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:07 smithi104 ceph-mon[97044]: pgmap v6716: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T07:59:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:07.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:07.518827+0000) 2022-09-10T07:59:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:07.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:07.881034+0000) 2022-09-10T07:59:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:08 smithi110 ceph-mon[97633]: osdmap e500: 12 total, 12 up, 12 in 2022-09-10T07:59:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:08 smithi104 ceph-mon[97044]: osdmap e500: 12 total, 12 up, 12 in 2022-09-10T07:59:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:08.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:08.519050+0000) 2022-09-10T07:59:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:08.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:08.881223+0000) 2022-09-10T07:59:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:09 smithi110 ceph-mon[97633]: pgmap v6718: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:09 smithi104 ceph-mon[97044]: pgmap v6718: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T07:59:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:09.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:09.519229+0000) 2022-09-10T07:59:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:09.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:09.881405+0000) 2022-09-10T07:59:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:10.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:10.519400+0000) 2022-09-10T07:59:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:10.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:10.881631+0000) 2022-09-10T07:59:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:11 smithi110 ceph-mon[97633]: pgmap v6719: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:59:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:11 smithi104 ceph-mon[97044]: pgmap v6719: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:59:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:11.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:11.519662+0000) 2022-09-10T07:59:12.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:11.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:11.881891+0000) 2022-09-10T07:59:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:12.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:12.519913+0000) 2022-09-10T07:59:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:12.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:12.882154+0000) 2022-09-10T07:59:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:13 smithi110 ceph-mon[97633]: pgmap v6720: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:59:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:13 smithi104 ceph-mon[97044]: pgmap v6720: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:59:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:13.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:13.520189+0000) 2022-09-10T07:59:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:13.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:13.882354+0000) 2022-09-10T07:59:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:14.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:14.520392+0000) 2022-09-10T07:59:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:14.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:14.882630+0000) 2022-09-10T07:59:15.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:15 smithi110 ceph-mon[97633]: pgmap v6721: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:59:15.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:15 smithi110 ceph-mon[97633]: osdmap e501: 12 total, 12 up, 12 in 2022-09-10T07:59:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:15 smithi104 ceph-mon[97044]: pgmap v6721: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T07:59:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:15 smithi104 ceph-mon[97044]: osdmap e501: 12 total, 12 up, 12 in 2022-09-10T07:59:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:15.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:15.520610+0000) 2022-09-10T07:59:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:15.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:15.882893+0000) 2022-09-10T07:59:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:16.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:16.520799+0000) 2022-09-10T07:59:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:16.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:16.883169+0000) 2022-09-10T07:59:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:17 smithi110 ceph-mon[97633]: pgmap v6723: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T07:59:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:17 smithi104 ceph-mon[97044]: pgmap v6723: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T07:59:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:17.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:17.521036+0000) 2022-09-10T07:59:17.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:59:17.739+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:59:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:17.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:17.883413+0000) 2022-09-10T07:59:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:18.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:18.521197+0000) 2022-09-10T07:59:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:18.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:18.883674+0000) 2022-09-10T07:59:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:19 smithi110 ceph-mon[97633]: pgmap v6724: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:59:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:19 smithi104 ceph-mon[97044]: pgmap v6724: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T07:59:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:19.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:19.521361+0000) 2022-09-10T07:59:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:19.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:19.883892+0000) 2022-09-10T07:59:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:20.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:20.521562+0000) 2022-09-10T07:59:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:20.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:20.884130+0000) 2022-09-10T07:59:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:21 smithi110 ceph-mon[97633]: pgmap v6725: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:59:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:21 smithi104 ceph-mon[97044]: pgmap v6725: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:59:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:21.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:21.521769+0000) 2022-09-10T07:59:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:21.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:21.884362+0000) 2022-09-10T07:59:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:22.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:22.522010+0000) 2022-09-10T07:59:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:22.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:22.884561+0000) 2022-09-10T07:59:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:23 smithi110 ceph-mon[97633]: pgmap v6726: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:59:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:23 smithi104 ceph-mon[97044]: pgmap v6726: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:59:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:23.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:23.522240+0000) 2022-09-10T07:59:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:23.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:23.884783+0000) 2022-09-10T07:59:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:24.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:24.522438+0000) 2022-09-10T07:59:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:24.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:24.885013+0000) 2022-09-10T07:59:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:25 smithi110 ceph-mon[97633]: pgmap v6727: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:59:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:25 smithi104 ceph-mon[97044]: pgmap v6727: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T07:59:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:25.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:25.522702+0000) 2022-09-10T07:59:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:25.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:25.885218+0000) 2022-09-10T07:59:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:26.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:26.522949+0000) 2022-09-10T07:59:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:26.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:26.885445+0000) 2022-09-10T07:59:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:27 smithi110 ceph-mon[97633]: pgmap v6728: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:59:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:27 smithi104 ceph-mon[97044]: pgmap v6728: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T07:59:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:27.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:27.523202+0000) 2022-09-10T07:59:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:27.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:27.885637+0000) 2022-09-10T07:59:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:28.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:28.523412+0000) 2022-09-10T07:59:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:28.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:28.885896+0000) 2022-09-10T07:59:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:29 smithi110 ceph-mon[97633]: pgmap v6729: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:29 smithi104 ceph-mon[97044]: pgmap v6729: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:29.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:29.523615+0000) 2022-09-10T07:59:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:29.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:29.886129+0000) 2022-09-10T07:59:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:30.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:30.523846+0000) 2022-09-10T07:59:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:30.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:30.886349+0000) 2022-09-10T07:59:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:31 smithi110 ceph-mon[97633]: pgmap v6730: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:31 smithi104 ceph-mon[97044]: pgmap v6730: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:31.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:31.524084+0000) 2022-09-10T07:59:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:31.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:31.886524+0000) 2022-09-10T07:59:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:59:32.754+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:59:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:32.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:32.524284+0000) 2022-09-10T07:59:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:32.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:32.886681+0000) 2022-09-10T07:59:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:33 smithi104 ceph-mon[97044]: pgmap v6731: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:33.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:33.524556+0000) 2022-09-10T07:59:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:33 smithi110 ceph-mon[97633]: pgmap v6731: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:33.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:33.886922+0000) 2022-09-10T07:59:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:34.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:34.524739+0000) 2022-09-10T07:59:35.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:34.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:34.887197+0000) 2022-09-10T07:59:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:35 smithi104 ceph-mon[97044]: pgmap v6732: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:35.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:35.524975+0000) 2022-09-10T07:59:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:35 smithi110 ceph-mon[97633]: pgmap v6732: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:35.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:35.887372+0000) 2022-09-10T07:59:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:36.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:36.525205+0000) 2022-09-10T07:59:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:36.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:36.887631+0000) 2022-09-10T07:59:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:37 smithi104 ceph-mon[97044]: pgmap v6733: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:37.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:37.525462+0000) 2022-09-10T07:59:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:37 smithi110 ceph-mon[97633]: pgmap v6733: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:37.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:37.887773+0000) 2022-09-10T07:59:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:38.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:38.525694+0000) 2022-09-10T07:59:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:38.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:38.887974+0000) 2022-09-10T07:59:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:39 smithi104 ceph-mon[97044]: pgmap v6734: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:39.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:39.525891+0000) 2022-09-10T07:59:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:39 smithi110 ceph-mon[97633]: pgmap v6734: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:39.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:39.888213+0000) 2022-09-10T07:59:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:40.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:40.526061+0000) 2022-09-10T07:59:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:40.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:40.888469+0000) 2022-09-10T07:59:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:41 smithi104 ceph-mon[97044]: pgmap v6735: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:41.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:41.526331+0000) 2022-09-10T07:59:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:41 smithi110 ceph-mon[97633]: pgmap v6735: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:41.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:41.888711+0000) 2022-09-10T07:59:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:42.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:42.526580+0000) 2022-09-10T07:59:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:42.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:42.888925+0000) 2022-09-10T07:59:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:43 smithi104 ceph-mon[97044]: pgmap v6736: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:43.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:43.526771+0000) 2022-09-10T07:59:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:43 smithi110 ceph-mon[97633]: pgmap v6736: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:43.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:43.889148+0000) 2022-09-10T07:59:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:59:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:59:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:59:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:44.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:44.527021+0000) 2022-09-10T07:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T07:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T07:59:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:44.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:44.889289+0000) 2022-09-10T07:59:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:45 smithi104 ceph-mon[97044]: pgmap v6737: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:45.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:45.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:45.527294+0000) 2022-09-10T07:59:45.915 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:45 smithi110 ceph-mon[97633]: pgmap v6737: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:45.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:45.889510+0000) 2022-09-10T07:59:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:46.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:46.527484+0000) 2022-09-10T07:59:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:46.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:46.889798+0000) 2022-09-10T07:59:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:47 smithi104 ceph-mon[97044]: pgmap v6738: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:47.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:47.527642+0000) 2022-09-10T07:59:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 07:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T07:59:47.766+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T07:59:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:47 smithi110 ceph-mon[97633]: pgmap v6738: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:47.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:47.890037+0000) 2022-09-10T07:59:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:48.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:48.527862+0000) 2022-09-10T07:59:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:48.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:48.890283+0000) 2022-09-10T07:59:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:49 smithi104 ceph-mon[97044]: pgmap v6739: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:49.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:49.528088+0000) 2022-09-10T07:59:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:49 smithi110 ceph-mon[97633]: pgmap v6739: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:49.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:49.890510+0000) 2022-09-10T07:59:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:50.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:50.528348+0000) 2022-09-10T07:59:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:50.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:50.890776+0000) 2022-09-10T07:59:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:51 smithi104 ceph-mon[97044]: pgmap v6740: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:51.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:51.528610+0000) 2022-09-10T07:59:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:51 smithi110 ceph-mon[97633]: pgmap v6740: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:51.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:51.891044+0000) 2022-09-10T07:59:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:52.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:52.528801+0000) 2022-09-10T07:59:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:52.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:52.891219+0000) 2022-09-10T07:59:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:53 smithi104 ceph-mon[97044]: pgmap v6741: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:53.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:53.529008+0000) 2022-09-10T07:59:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:53 smithi110 ceph-mon[97633]: pgmap v6741: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:53.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:53.891442+0000) 2022-09-10T07:59:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:54.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:54.529265+0000) 2022-09-10T07:59:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:54.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:54.891710+0000) 2022-09-10T07:59:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:55 smithi104 ceph-mon[97044]: pgmap v6742: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:55.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:55.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:55.529485+0000) 2022-09-10T07:59:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:55 smithi110 ceph-mon[97633]: pgmap v6742: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:55.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:55.891950+0000) 2022-09-10T07:59:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:56.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:56.529734+0000) 2022-09-10T07:59:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:56.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:56.892244+0000) 2022-09-10T07:59:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:57 smithi104 ceph-mon[97044]: pgmap v6743: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:57.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:57.529989+0000) 2022-09-10T07:59:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:57 smithi110 ceph-mon[97633]: pgmap v6743: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T07:59:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:57.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:57.892462+0000) 2022-09-10T07:59:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:58.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:58.530272+0000) 2022-09-10T07:59:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:58.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:58.892615+0000) 2022-09-10T07:59:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 07:59:59 smithi104 ceph-mon[97044]: pgmap v6744: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T07:59:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 07:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T07:59:59.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:59.530504+0000) 2022-09-10T07:59:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 07:59:59 smithi110 ceph-mon[97633]: pgmap v6744: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 07:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T07:59:59.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T06:59:59.892843+0000) 2022-09-10T08:00:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:00:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:00.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:00.530696+0000) 2022-09-10T08:00:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:00:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:00.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:00.893086+0000) 2022-09-10T08:00:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:01 smithi104 ceph-mon[97044]: pgmap v6745: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:01.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:01.530952+0000) 2022-09-10T08:00:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:01 smithi110 ceph-mon[97633]: pgmap v6745: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:01.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:01.893283+0000) 2022-09-10T08:00:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:00:02.774+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:00:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:02.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:02.531200+0000) 2022-09-10T08:00:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:02.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:02.893507+0000) 2022-09-10T08:00:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:03 smithi104 ceph-mon[97044]: pgmap v6746: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:03.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:03.531458+0000) 2022-09-10T08:00:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:03 smithi110 ceph-mon[97633]: pgmap v6746: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:03.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:03.893648+0000) 2022-09-10T08:00:04.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:04.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:04.531646+0000) 2022-09-10T08:00:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:04.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:04.893875+0000) 2022-09-10T08:00:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:05 smithi104 ceph-mon[97044]: pgmap v6747: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:05.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:05.531808+0000) 2022-09-10T08:00:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:05 smithi110 ceph-mon[97633]: pgmap v6747: 129 pgs: 129 active+clean; 8.4 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:05.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:05.894092+0000) 2022-09-10T08:00:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:06.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:06.532085+0000) 2022-09-10T08:00:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:06.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:06.894313+0000) 2022-09-10T08:00:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:07 smithi104 ceph-mon[97044]: pgmap v6748: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:00:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:07.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:07.532302+0000) 2022-09-10T08:00:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:07 smithi110 ceph-mon[97633]: pgmap v6748: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:00:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:07.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:07.894568+0000) 2022-09-10T08:00:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:08 smithi104 ceph-mon[97044]: osdmap e502: 12 total, 12 up, 12 in 2022-09-10T08:00:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:08.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:08.532540+0000) 2022-09-10T08:00:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:08 smithi110 ceph-mon[97633]: osdmap e502: 12 total, 12 up, 12 in 2022-09-10T08:00:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:08.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:08.894814+0000) 2022-09-10T08:00:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:09 smithi104 ceph-mon[97044]: pgmap v6750: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:09.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:09.532795+0000) 2022-09-10T08:00:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:09 smithi110 ceph-mon[97633]: pgmap v6750: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:09.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:09.895058+0000) 2022-09-10T08:00:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:10.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:10.533041+0000) 2022-09-10T08:00:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:10.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:10.895248+0000) 2022-09-10T08:00:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:11 smithi104 ceph-mon[97044]: pgmap v6751: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:11.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:11.533350+0000) 2022-09-10T08:00:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:11 smithi110 ceph-mon[97633]: pgmap v6751: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:11.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:11.895413+0000) 2022-09-10T08:00:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:12.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:12.533559+0000) 2022-09-10T08:00:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:12.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:12.895682+0000) 2022-09-10T08:00:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:13 smithi104 ceph-mon[97044]: pgmap v6752: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:13.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:13.533801+0000) 2022-09-10T08:00:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:13 smithi110 ceph-mon[97633]: pgmap v6752: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:13.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:13.895878+0000) 2022-09-10T08:00:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:14.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:14.533966+0000) 2022-09-10T08:00:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:14.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:14.896066+0000) 2022-09-10T08:00:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:15 smithi104 ceph-mon[97044]: pgmap v6753: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:15 smithi104 ceph-mon[97044]: osdmap e503: 12 total, 12 up, 12 in 2022-09-10T08:00:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:15.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:15.534214+0000) 2022-09-10T08:00:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:15 smithi110 ceph-mon[97633]: pgmap v6753: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:00:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:15 smithi110 ceph-mon[97633]: osdmap e503: 12 total, 12 up, 12 in 2022-09-10T08:00:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:15.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:15.896309+0000) 2022-09-10T08:00:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:16.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:16.534381+0000) 2022-09-10T08:00:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:16.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:16.896510+0000) 2022-09-10T08:00:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:17 smithi104 ceph-mon[97044]: pgmap v6755: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:00:17.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:00:17.780+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:00:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:17.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:17.534588+0000) 2022-09-10T08:00:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:17 smithi110 ceph-mon[97633]: pgmap v6755: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:00:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:17.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:17.896775+0000) 2022-09-10T08:00:18.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:18.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:18.534841+0000) 2022-09-10T08:00:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:18.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:18.897022+0000) 2022-09-10T08:00:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:19 smithi104 ceph-mon[97044]: pgmap v6756: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:00:19.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:19.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:19.535089+0000) 2022-09-10T08:00:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:19 smithi110 ceph-mon[97633]: pgmap v6756: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:00:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:19.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:19.897238+0000) 2022-09-10T08:00:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:20.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:20.535329+0000) 2022-09-10T08:00:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:20.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:20.897463+0000) 2022-09-10T08:00:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:21 smithi104 ceph-mon[97044]: pgmap v6757: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:00:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:21.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:21.535616+0000) 2022-09-10T08:00:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:21 smithi110 ceph-mon[97633]: pgmap v6757: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:00:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:21.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:21.897663+0000) 2022-09-10T08:00:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:22.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:22.535870+0000) 2022-09-10T08:00:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:22.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:22.897918+0000) 2022-09-10T08:00:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:23 smithi104 ceph-mon[97044]: pgmap v6758: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:00:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:23.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:23.536078+0000) 2022-09-10T08:00:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:23 smithi110 ceph-mon[97633]: pgmap v6758: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:00:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:23.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:23.898199+0000) 2022-09-10T08:00:24.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:24.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:24.536319+0000) 2022-09-10T08:00:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:24.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:24.898493+0000) 2022-09-10T08:00:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:25 smithi104 ceph-mon[97044]: pgmap v6759: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:00:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:25.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:25.536598+0000) 2022-09-10T08:00:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:25 smithi110 ceph-mon[97633]: pgmap v6759: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:00:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:25.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:25.898684+0000) 2022-09-10T08:00:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:26.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:26.536867+0000) 2022-09-10T08:00:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:26.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:26.898918+0000) 2022-09-10T08:00:27.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:27 smithi104 ceph-mon[97044]: pgmap v6760: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:00:27.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:27.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:27.537091+0000) 2022-09-10T08:00:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:27 smithi110 ceph-mon[97633]: pgmap v6760: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:00:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:27.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:27.899139+0000) 2022-09-10T08:00:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:28.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:28.537236+0000) 2022-09-10T08:00:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:28.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:28.899321+0000) 2022-09-10T08:00:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:29 smithi104 ceph-mon[97044]: pgmap v6761: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:29.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:29.537486+0000) 2022-09-10T08:00:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:29 smithi110 ceph-mon[97633]: pgmap v6761: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:29.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:29.899539+0000) 2022-09-10T08:00:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:30.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:30.537709+0000) 2022-09-10T08:00:31.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:30.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:30.899709+0000) 2022-09-10T08:00:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:31 smithi104 ceph-mon[97044]: pgmap v6762: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:31.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:31.537962+0000) 2022-09-10T08:00:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:31 smithi110 ceph-mon[97633]: pgmap v6762: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:31.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:31.899906+0000) 2022-09-10T08:00:32.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:32.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:32.538184+0000) 2022-09-10T08:00:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:32.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:32.900149+0000) 2022-09-10T08:00:33.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:00:32.795+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:00:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:33 smithi104 ceph-mon[97044]: pgmap v6763: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:33.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:33.538369+0000) 2022-09-10T08:00:33.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:33 smithi110 ceph-mon[97633]: pgmap v6763: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:33.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:33.900313+0000) 2022-09-10T08:00:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:34.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:34.538512+0000) 2022-09-10T08:00:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:34.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:34.900570+0000) 2022-09-10T08:00:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:35 smithi104 ceph-mon[97044]: pgmap v6764: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:35.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:35.538752+0000) 2022-09-10T08:00:35.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:35 smithi110 ceph-mon[97633]: pgmap v6764: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:35.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:35.900815+0000) 2022-09-10T08:00:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:36.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:36.539003+0000) 2022-09-10T08:00:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:36.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:36.901057+0000) 2022-09-10T08:00:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:37 smithi104 ceph-mon[97044]: pgmap v6765: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:37.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:37.539233+0000) 2022-09-10T08:00:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:37 smithi110 ceph-mon[97633]: pgmap v6765: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:37.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:37.901296+0000) 2022-09-10T08:00:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:38.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:38.539477+0000) 2022-09-10T08:00:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:38.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:38.901563+0000) 2022-09-10T08:00:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:39 smithi104 ceph-mon[97044]: pgmap v6766: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:39.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:39.539665+0000) 2022-09-10T08:00:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:39 smithi110 ceph-mon[97633]: pgmap v6766: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:39.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:39.901815+0000) 2022-09-10T08:00:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:40.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:40.539842+0000) 2022-09-10T08:00:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:40.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:40.902003+0000) 2022-09-10T08:00:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:41 smithi104 ceph-mon[97044]: pgmap v6767: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:41.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:41.540121+0000) 2022-09-10T08:00:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:41 smithi110 ceph-mon[97633]: pgmap v6767: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:41.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:41.902164+0000) 2022-09-10T08:00:42.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:42.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:42.540386+0000) 2022-09-10T08:00:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:42.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:42.902378+0000) 2022-09-10T08:00:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:43 smithi104 ceph-mon[97044]: pgmap v6768: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:43.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:43.540657+0000) 2022-09-10T08:00:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:43 smithi110 ceph-mon[97633]: pgmap v6768: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:43.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:43.902576+0000) 2022-09-10T08:00:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:00:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:00:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:00:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:00:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:44.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:44.540842+0000) 2022-09-10T08:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:00:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:00:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:44.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:44.902790+0000) 2022-09-10T08:00:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:45 smithi104 ceph-mon[97044]: pgmap v6769: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:45.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:45.541086+0000) 2022-09-10T08:00:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:45 smithi110 ceph-mon[97633]: pgmap v6769: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:45.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:45.903060+0000) 2022-09-10T08:00:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:46.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:46.541333+0000) 2022-09-10T08:00:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:46.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:46.903332+0000) 2022-09-10T08:00:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:47 smithi104 ceph-mon[97044]: pgmap v6770: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:47.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:47.541587+0000) 2022-09-10T08:00:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:47 smithi110 ceph-mon[97633]: pgmap v6770: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:47.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:47.903544+0000) 2022-09-10T08:00:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:00:47.810+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:00:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:48.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:48.541825+0000) 2022-09-10T08:00:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:48.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:48.903751+0000) 2022-09-10T08:00:49.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:49 smithi104 ceph-mon[97044]: pgmap v6771: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:49.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:49.542124+0000) 2022-09-10T08:00:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:49 smithi110 ceph-mon[97633]: pgmap v6771: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:49.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:49.903955+0000) 2022-09-10T08:00:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:50.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:50.542325+0000) 2022-09-10T08:00:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:50.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:50.904162+0000) 2022-09-10T08:00:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:51 smithi104 ceph-mon[97044]: pgmap v6772: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:51.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:51.542570+0000) 2022-09-10T08:00:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:51 smithi110 ceph-mon[97633]: pgmap v6772: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:51.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:51.904424+0000) 2022-09-10T08:00:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:52.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:52.542734+0000) 2022-09-10T08:00:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:52.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:52.904632+0000) 2022-09-10T08:00:53.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:53 smithi104 ceph-mon[97044]: pgmap v6773: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:53.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:53.542981+0000) 2022-09-10T08:00:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:53 smithi110 ceph-mon[97633]: pgmap v6773: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:53.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:53.904896+0000) 2022-09-10T08:00:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:54.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:54.543235+0000) 2022-09-10T08:00:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:54.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:54.905186+0000) 2022-09-10T08:00:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:55.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:55.543485+0000) 2022-09-10T08:00:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:55 smithi110 ceph-mon[97633]: pgmap v6774: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:55 smithi104 ceph-mon[97044]: pgmap v6774: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:00:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:55.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:55.905423+0000) 2022-09-10T08:00:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:56.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:56.543707+0000) 2022-09-10T08:00:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:56.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:56.905678+0000) 2022-09-10T08:00:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:57.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:57.543996+0000) 2022-09-10T08:00:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:57 smithi110 ceph-mon[97633]: pgmap v6775: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:57 smithi104 ceph-mon[97044]: pgmap v6775: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:00:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:57.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:57.905839+0000) 2022-09-10T08:00:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:58.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:58.544211+0000) 2022-09-10T08:00:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:58.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:58.906039+0000) 2022-09-10T08:00:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:00:59.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:59.544364+0000) 2022-09-10T08:00:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:00:59 smithi110 ceph-mon[97633]: pgmap v6776: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:00:59 smithi104 ceph-mon[97044]: pgmap v6776: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:00:59.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:00:59.906308+0000) 2022-09-10T08:01:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:00.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:00.544536+0000) 2022-09-10T08:01:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:00.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:00.906584+0000) 2022-09-10T08:01:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:01.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:01.544811+0000) 2022-09-10T08:01:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:01 smithi110 ceph-mon[97633]: pgmap v6777: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:01 smithi104 ceph-mon[97044]: pgmap v6777: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:01.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:01.906747+0000) 2022-09-10T08:01:02.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:02.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:02.545071+0000) 2022-09-10T08:01:03.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:02.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:02.906892+0000) 2022-09-10T08:01:03.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:01:02.825+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:01:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:03.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:03.545310+0000) 2022-09-10T08:01:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:03 smithi110 ceph-mon[97633]: pgmap v6778: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:03 smithi104 ceph-mon[97044]: pgmap v6778: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:03.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:03.907168+0000) 2022-09-10T08:01:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:04.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:04.545541+0000) 2022-09-10T08:01:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:04.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:04.907386+0000) 2022-09-10T08:01:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:05.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:05.545741+0000) 2022-09-10T08:01:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:05 smithi110 ceph-mon[97633]: pgmap v6779: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:05 smithi104 ceph-mon[97044]: pgmap v6779: 129 pgs: 129 active+clean; 8.4 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:05.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:05.907580+0000) 2022-09-10T08:01:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:06.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:06.545965+0000) 2022-09-10T08:01:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:06.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:06.907805+0000) 2022-09-10T08:01:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:07.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:07.546220+0000) 2022-09-10T08:01:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:07 smithi110 ceph-mon[97633]: pgmap v6780: 129 pgs: 129 active+clean; 8.4 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:01:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:07 smithi104 ceph-mon[97044]: pgmap v6780: 129 pgs: 129 active+clean; 8.4 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:01:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:07.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:07.908000+0000) 2022-09-10T08:01:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:08.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:08.546407+0000) 2022-09-10T08:01:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:08 smithi110 ceph-mon[97633]: osdmap e504: 12 total, 12 up, 12 in 2022-09-10T08:01:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:08 smithi104 ceph-mon[97044]: osdmap e504: 12 total, 12 up, 12 in 2022-09-10T08:01:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:08.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:08.908197+0000) 2022-09-10T08:01:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:09.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:09.546661+0000) 2022-09-10T08:01:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:09 smithi110 ceph-mon[97633]: pgmap v6782: 129 pgs: 129 active+clean; 8.4 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:09 smithi104 ceph-mon[97044]: pgmap v6782: 129 pgs: 129 active+clean; 8.4 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:09.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:09.908409+0000) 2022-09-10T08:01:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:10.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:10.546889+0000) 2022-09-10T08:01:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:10.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:10.908627+0000) 2022-09-10T08:01:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:11.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:11.547196+0000) 2022-09-10T08:01:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:11 smithi110 ceph-mon[97633]: pgmap v6783: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:11 smithi104 ceph-mon[97044]: pgmap v6783: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:11.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:11.908818+0000) 2022-09-10T08:01:12.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:12.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:12.547425+0000) 2022-09-10T08:01:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:12.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:12.909021+0000) 2022-09-10T08:01:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:13.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:13.547643+0000) 2022-09-10T08:01:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:13 smithi110 ceph-mon[97633]: pgmap v6784: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:13 smithi104 ceph-mon[97044]: pgmap v6784: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:13.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:13.909253+0000) 2022-09-10T08:01:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:14.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:14.547869+0000) 2022-09-10T08:01:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:14.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:14.909498+0000) 2022-09-10T08:01:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:15.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:15.548069+0000) 2022-09-10T08:01:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:15 smithi110 ceph-mon[97633]: pgmap v6785: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:15 smithi110 ceph-mon[97633]: osdmap e505: 12 total, 12 up, 12 in 2022-09-10T08:01:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:15 smithi104 ceph-mon[97044]: pgmap v6785: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:01:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:15 smithi104 ceph-mon[97044]: osdmap e505: 12 total, 12 up, 12 in 2022-09-10T08:01:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:15.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:15.909704+0000) 2022-09-10T08:01:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:16.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:16.548232+0000) 2022-09-10T08:01:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:16.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:16.909910+0000) 2022-09-10T08:01:17.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:17.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:17.548392+0000) 2022-09-10T08:01:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:17 smithi110 ceph-mon[97633]: pgmap v6787: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T08:01:18.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:17 smithi104 ceph-mon[97044]: pgmap v6787: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T08:01:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:17.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:17.910133+0000) 2022-09-10T08:01:18.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:01:17.841+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:01:18.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:18.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:18.548582+0000) 2022-09-10T08:01:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:18.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:18.910348+0000) 2022-09-10T08:01:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:19.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:19.548833+0000) 2022-09-10T08:01:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:19 smithi110 ceph-mon[97633]: pgmap v6788: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:01:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:19 smithi104 ceph-mon[97044]: pgmap v6788: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:01:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:19.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:19.910522+0000) 2022-09-10T08:01:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:20.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:20.549058+0000) 2022-09-10T08:01:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:20.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:20.910659+0000) 2022-09-10T08:01:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:21.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:21.549315+0000) 2022-09-10T08:01:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:21 smithi110 ceph-mon[97633]: pgmap v6789: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:01:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:21 smithi104 ceph-mon[97044]: pgmap v6789: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:01:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:21.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:21.910887+0000) 2022-09-10T08:01:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:22.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:22.549508+0000) 2022-09-10T08:01:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:22.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:22.911182+0000) 2022-09-10T08:01:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:23.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:23.549778+0000) 2022-09-10T08:01:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:23 smithi110 ceph-mon[97633]: pgmap v6790: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:01:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:23 smithi104 ceph-mon[97044]: pgmap v6790: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:01:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:23.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:23.911417+0000) 2022-09-10T08:01:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:24.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:24.550014+0000) 2022-09-10T08:01:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:24.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:24.911635+0000) 2022-09-10T08:01:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:25.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:25.550267+0000) 2022-09-10T08:01:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:25 smithi110 ceph-mon[97633]: pgmap v6791: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:01:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:25 smithi104 ceph-mon[97044]: pgmap v6791: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:01:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:25.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:25.911901+0000) 2022-09-10T08:01:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:26.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:26.550398+0000) 2022-09-10T08:01:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:26.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:26.912188+0000) 2022-09-10T08:01:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:27.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:27.550606+0000) 2022-09-10T08:01:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:27 smithi110 ceph-mon[97633]: pgmap v6792: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:01:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:27 smithi104 ceph-mon[97044]: pgmap v6792: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:01:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:27.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:27.912391+0000) 2022-09-10T08:01:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:28.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:28.550827+0000) 2022-09-10T08:01:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:28.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:28.912616+0000) 2022-09-10T08:01:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:29.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:29.551053+0000) 2022-09-10T08:01:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:29 smithi110 ceph-mon[97633]: pgmap v6793: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:29 smithi104 ceph-mon[97044]: pgmap v6793: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:29.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:29.912900+0000) 2022-09-10T08:01:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:30.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:30.551171+0000) 2022-09-10T08:01:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:30.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:30.913168+0000) 2022-09-10T08:01:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:31.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:31.551325+0000) 2022-09-10T08:01:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:31 smithi110 ceph-mon[97633]: pgmap v6794: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:31 smithi104 ceph-mon[97044]: pgmap v6794: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:31.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:31.913336+0000) 2022-09-10T08:01:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:32.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:32.551506+0000) 2022-09-10T08:01:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:32.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:32.913611+0000) 2022-09-10T08:01:33.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:01:32.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:01:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:33.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:33.551724+0000) 2022-09-10T08:01:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:33 smithi110 ceph-mon[97633]: pgmap v6795: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:33 smithi104 ceph-mon[97044]: pgmap v6795: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:33.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:33.913806+0000) 2022-09-10T08:01:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:34.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:34.551947+0000) 2022-09-10T08:01:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:34.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:34.914065+0000) 2022-09-10T08:01:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:35.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:35.552197+0000) 2022-09-10T08:01:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:35 smithi110 ceph-mon[97633]: pgmap v6796: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:36.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:35 smithi104 ceph-mon[97044]: pgmap v6796: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:35.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:35.914234+0000) 2022-09-10T08:01:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:36.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:36.552362+0000) 2022-09-10T08:01:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:36.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:36.914458+0000) 2022-09-10T08:01:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:37.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:37.552526+0000) 2022-09-10T08:01:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:37 smithi110 ceph-mon[97633]: pgmap v6797: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:37 smithi104 ceph-mon[97044]: pgmap v6797: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:37.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:37.914706+0000) 2022-09-10T08:01:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:38.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:38.552663+0000) 2022-09-10T08:01:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:38.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:38.914888+0000) 2022-09-10T08:01:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:39.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:39.552903+0000) 2022-09-10T08:01:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:39 smithi110 ceph-mon[97633]: pgmap v6798: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:01:40.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:39 smithi104 ceph-mon[97044]: pgmap v6798: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:01:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:39.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:39.915086+0000) 2022-09-10T08:01:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:40.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:40.553175+0000) 2022-09-10T08:01:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:40.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:40.915285+0000) 2022-09-10T08:01:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:41.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:41.553340+0000) 2022-09-10T08:01:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:41 smithi110 ceph-mon[97633]: pgmap v6799: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:41 smithi104 ceph-mon[97044]: pgmap v6799: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:41.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:41.915461+0000) 2022-09-10T08:01:42.916 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:42.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:42.553526+0000) 2022-09-10T08:01:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:42 smithi104 ceph-mon[97044]: pgmap v6800: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:01:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:42 smithi110 ceph-mon[97633]: pgmap v6800: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:01:43.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:42.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:42.915684+0000) 2022-09-10T08:01:43.916 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:43.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:43.553789+0000) 2022-09-10T08:01:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:43.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:43.915828+0000) 2022-09-10T08:01:44.916 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:44.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:44.554005+0000) 2022-09-10T08:01:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:44 smithi110 ceph-mon[97633]: pgmap v6801: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:01:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:01:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:01:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:01:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:01:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:44 smithi104 ceph-mon[97044]: pgmap v6801: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:01:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:01:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:01:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:01:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:01:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:44.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:44.916073+0000) 2022-09-10T08:01:45.917 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:45.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:45.554289+0000) 2022-09-10T08:01:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:45.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:45.916374+0000) 2022-09-10T08:01:46.917 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:46.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:46.554530+0000) 2022-09-10T08:01:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:46.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:46.916612+0000) 2022-09-10T08:01:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:47 smithi110 ceph-mon[97633]: pgmap v6802: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:47 smithi104 ceph-mon[97044]: pgmap v6802: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:47.858 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:47.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:47.554780+0000) 2022-09-10T08:01:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:47.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:47.916848+0000) 2022-09-10T08:01:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:01:47.856+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:01:48.917 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:48.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:48.555003+0000) 2022-09-10T08:01:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:48.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:48.917144+0000) 2022-09-10T08:01:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:49 smithi110 ceph-mon[97633]: pgmap v6803: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:49 smithi104 ceph-mon[97044]: pgmap v6803: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:49.919 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:49.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:49.555256+0000) 2022-09-10T08:01:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:49.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:49.917359+0000) 2022-09-10T08:01:50.918 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:50.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:50.555477+0000) 2022-09-10T08:01:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:50.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:50.917606+0000) 2022-09-10T08:01:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:51 smithi110 ceph-mon[97633]: pgmap v6804: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:51 smithi104 ceph-mon[97044]: pgmap v6804: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:51.918 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:51.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:51.555672+0000) 2022-09-10T08:01:52.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:51.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:51.917749+0000) 2022-09-10T08:01:52.918 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:52.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:52.555871+0000) 2022-09-10T08:01:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:52.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:52.917918+0000) 2022-09-10T08:01:53.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:53 smithi110 ceph-mon[97633]: pgmap v6805: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:53 smithi104 ceph-mon[97044]: pgmap v6805: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:53.918 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:53.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:53.556081+0000) 2022-09-10T08:01:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:53.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:53.918195+0000) 2022-09-10T08:01:54.918 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:54.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:54.556256+0000) 2022-09-10T08:01:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:54.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:54.918441+0000) 2022-09-10T08:01:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:55 smithi110 ceph-mon[97633]: pgmap v6806: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:55 smithi104 ceph-mon[97044]: pgmap v6806: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:55.919 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:55.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:55.556401+0000) 2022-09-10T08:01:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:55.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:55.918681+0000) 2022-09-10T08:01:56.919 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:56.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:56.556556+0000) 2022-09-10T08:01:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:56.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:56.918844+0000) 2022-09-10T08:01:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:57 smithi110 ceph-mon[97633]: pgmap v6807: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:57 smithi104 ceph-mon[97044]: pgmap v6807: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:01:57.919 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:57.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:57.556737+0000) 2022-09-10T08:01:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:57.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:57.919085+0000) 2022-09-10T08:01:58.919 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:58.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:58.556924+0000) 2022-09-10T08:01:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:58.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:58.919279+0000) 2022-09-10T08:01:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:01:59 smithi110 ceph-mon[97633]: pgmap v6808: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:01:59 smithi104 ceph-mon[97044]: pgmap v6808: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:01:59.920 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:01:59.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:59.557081+0000) 2022-09-10T08:02:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:01:59.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:01:59.919414+0000) 2022-09-10T08:02:00.920 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:00.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:00.557280+0000) 2022-09-10T08:02:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:00.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:00.919542+0000) 2022-09-10T08:02:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:01 smithi110 ceph-mon[97633]: pgmap v6809: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:01 smithi104 ceph-mon[97044]: pgmap v6809: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:01.920 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:01.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:01.557531+0000) 2022-09-10T08:02:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:01.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:01.919768+0000) 2022-09-10T08:02:02.873 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:02.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:02.557695+0000) 2022-09-10T08:02:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:02.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:02.920030+0000) 2022-09-10T08:02:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:02:02.872+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:02:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:03 smithi110 ceph-mon[97633]: pgmap v6810: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:03 smithi104 ceph-mon[97044]: pgmap v6810: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:03.920 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:03.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:03.557899+0000) 2022-09-10T08:02:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:03.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:03.920269+0000) 2022-09-10T08:02:04.923 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:04.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:04.558115+0000) 2022-09-10T08:02:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:04.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:04.920517+0000) 2022-09-10T08:02:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:05 smithi110 ceph-mon[97633]: pgmap v6811: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:05 smithi104 ceph-mon[97044]: pgmap v6811: 129 pgs: 129 active+clean; 8.4 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:05.921 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:05.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:05.558357+0000) 2022-09-10T08:02:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:05.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:05.920745+0000) 2022-09-10T08:02:06.921 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:06.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:06.558547+0000) 2022-09-10T08:02:07.221 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:06.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:06.920977+0000) 2022-09-10T08:02:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:07 smithi110 ceph-mon[97633]: pgmap v6812: 129 pgs: 129 active+clean; 8.4 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:02:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:07 smithi104 ceph-mon[97044]: pgmap v6812: 129 pgs: 129 active+clean; 8.4 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:02:07.921 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:07.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:07.558784+0000) 2022-09-10T08:02:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:07.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:07.921177+0000) 2022-09-10T08:02:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:08 smithi110 ceph-mon[97633]: osdmap e506: 12 total, 12 up, 12 in 2022-09-10T08:02:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:08 smithi104 ceph-mon[97044]: osdmap e506: 12 total, 12 up, 12 in 2022-09-10T08:02:08.921 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:08.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:08.558966+0000) 2022-09-10T08:02:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:08.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:08.921331+0000) 2022-09-10T08:02:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:09 smithi110 ceph-mon[97633]: pgmap v6814: 129 pgs: 129 active+clean; 8.4 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:09 smithi104 ceph-mon[97044]: pgmap v6814: 129 pgs: 129 active+clean; 8.4 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:09.922 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:09.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:09.559226+0000) 2022-09-10T08:02:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:09.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:09.921458+0000) 2022-09-10T08:02:10.922 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:10.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:10.559431+0000) 2022-09-10T08:02:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:10.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:10.921671+0000) 2022-09-10T08:02:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:11 smithi110 ceph-mon[97633]: pgmap v6815: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:11.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:11 smithi104 ceph-mon[97044]: pgmap v6815: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:11.922 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:11.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:11.559660+0000) 2022-09-10T08:02:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:11.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:11.921822+0000) 2022-09-10T08:02:12.922 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:12.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:12.559914+0000) 2022-09-10T08:02:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:12.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:12.922052+0000) 2022-09-10T08:02:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:13 smithi110 ceph-mon[97633]: pgmap v6816: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:13 smithi104 ceph-mon[97044]: pgmap v6816: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:13.923 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:13.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:13.560181+0000) 2022-09-10T08:02:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:13.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:13.922319+0000) 2022-09-10T08:02:14.923 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:14.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:14.560429+0000) 2022-09-10T08:02:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:14.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:14.922520+0000) 2022-09-10T08:02:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:15 smithi110 ceph-mon[97633]: pgmap v6817: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:15 smithi110 ceph-mon[97633]: osdmap e507: 12 total, 12 up, 12 in 2022-09-10T08:02:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:15 smithi104 ceph-mon[97044]: pgmap v6817: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:02:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:15 smithi104 ceph-mon[97044]: osdmap e507: 12 total, 12 up, 12 in 2022-09-10T08:02:15.923 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:15.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:15.560661+0000) 2022-09-10T08:02:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:15.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:15.922713+0000) 2022-09-10T08:02:16.923 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:16.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:16.560875+0000) 2022-09-10T08:02:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:16.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:16.922937+0000) 2022-09-10T08:02:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:17 smithi110 ceph-mon[97633]: pgmap v6819: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:02:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:17 smithi104 ceph-mon[97044]: pgmap v6819: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:02:17.877 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:17.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:17.561047+0000) 2022-09-10T08:02:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:17.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:17.923157+0000) 2022-09-10T08:02:18.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:02:17.876+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:02:18.924 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:18.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:18.561270+0000) 2022-09-10T08:02:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:18.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:18.923399+0000) 2022-09-10T08:02:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:19 smithi110 ceph-mon[97633]: pgmap v6820: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:02:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:19 smithi104 ceph-mon[97044]: pgmap v6820: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:02:19.924 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:19.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:19.561506+0000) 2022-09-10T08:02:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:19.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:19.923590+0000) 2022-09-10T08:02:20.924 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:20.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:20.561707+0000) 2022-09-10T08:02:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:20.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:20.923822+0000) 2022-09-10T08:02:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:21 smithi110 ceph-mon[97633]: pgmap v6821: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:02:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:21 smithi104 ceph-mon[97044]: pgmap v6821: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:02:21.924 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:21.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:21.561891+0000) 2022-09-10T08:02:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:21.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:21.924135+0000) 2022-09-10T08:02:22.924 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:22.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:22.562039+0000) 2022-09-10T08:02:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:22.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:22.924294+0000) 2022-09-10T08:02:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:23 smithi110 ceph-mon[97633]: pgmap v6822: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:02:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:23 smithi104 ceph-mon[97044]: pgmap v6822: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:02:23.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:23.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:23.562250+0000) 2022-09-10T08:02:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:23.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:23.924512+0000) 2022-09-10T08:02:24.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:24.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:24.562494+0000) 2022-09-10T08:02:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:24.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:24.924761+0000) 2022-09-10T08:02:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:25 smithi104 ceph-mon[97044]: pgmap v6823: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:02:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:25 smithi110 ceph-mon[97633]: pgmap v6823: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:02:25.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:25.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:25.562762+0000) 2022-09-10T08:02:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:25.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:25.924988+0000) 2022-09-10T08:02:26.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:26.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:26.562945+0000) 2022-09-10T08:02:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:26.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:26.925227+0000) 2022-09-10T08:02:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:27 smithi104 ceph-mon[97044]: pgmap v6824: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:02:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:27 smithi110 ceph-mon[97633]: pgmap v6824: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:02:27.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:27.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:27.563200+0000) 2022-09-10T08:02:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:27.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:27.925379+0000) 2022-09-10T08:02:28.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:28.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:28.563410+0000) 2022-09-10T08:02:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:28.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:28.925578+0000) 2022-09-10T08:02:29.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:29 smithi104 ceph-mon[97044]: pgmap v6825: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:29 smithi110 ceph-mon[97633]: pgmap v6825: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:29.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:29.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:29.563556+0000) 2022-09-10T08:02:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:29.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:29.925798+0000) 2022-09-10T08:02:30.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:30.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:30.563759+0000) 2022-09-10T08:02:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:30.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:30.926004+0000) 2022-09-10T08:02:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:31 smithi104 ceph-mon[97044]: pgmap v6826: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:31 smithi110 ceph-mon[97633]: pgmap v6826: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:31.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:31.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:31.564025+0000) 2022-09-10T08:02:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:31.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:31.926179+0000) 2022-09-10T08:02:32.893 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:32.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:32.564238+0000) 2022-09-10T08:02:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:32.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:32.926400+0000) 2022-09-10T08:02:33.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:02:32.891+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:02:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:33 smithi104 ceph-mon[97044]: pgmap v6827: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:33 smithi110 ceph-mon[97633]: pgmap v6827: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:33.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:33.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:33.564459+0000) 2022-09-10T08:02:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:33.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:33.926559+0000) 2022-09-10T08:02:34.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:34.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:34.564722+0000) 2022-09-10T08:02:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:34.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:34.926774+0000) 2022-09-10T08:02:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:35 smithi104 ceph-mon[97044]: pgmap v6828: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:35 smithi110 ceph-mon[97633]: pgmap v6828: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:35.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:35.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:35.564975+0000) 2022-09-10T08:02:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:35.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:35.926968+0000) 2022-09-10T08:02:36.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:36.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:36.565173+0000) 2022-09-10T08:02:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:36.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:36.927200+0000) 2022-09-10T08:02:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:37 smithi104 ceph-mon[97044]: pgmap v6829: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:37 smithi110 ceph-mon[97633]: pgmap v6829: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:37.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:37.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:37.565427+0000) 2022-09-10T08:02:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:37.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:37.927449+0000) 2022-09-10T08:02:38.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:38.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:38.565644+0000) 2022-09-10T08:02:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:38.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:38.927641+0000) 2022-09-10T08:02:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:39 smithi104 ceph-mon[97044]: pgmap v6830: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:39 smithi110 ceph-mon[97633]: pgmap v6830: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:39.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:39.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:39.565901+0000) 2022-09-10T08:02:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:39.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:39.927862+0000) 2022-09-10T08:02:40.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:40.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:40.566142+0000) 2022-09-10T08:02:41.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:40.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:40.928090+0000) 2022-09-10T08:02:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:41 smithi104 ceph-mon[97044]: pgmap v6831: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:41 smithi110 ceph-mon[97633]: pgmap v6831: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:41.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:41.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:41.566402+0000) 2022-09-10T08:02:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:41.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:41.928399+0000) 2022-09-10T08:02:42.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:42.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:42.566585+0000) 2022-09-10T08:02:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:42.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:42.928631+0000) 2022-09-10T08:02:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:43 smithi104 ceph-mon[97044]: pgmap v6832: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:43 smithi110 ceph-mon[97633]: pgmap v6832: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:43.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:43.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:43.566757+0000) 2022-09-10T08:02:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:43.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:43.928822+0000) 2022-09-10T08:02:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:02:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:02:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:02:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:02:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:02:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:02:44.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:44.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:44.566996+0000) 2022-09-10T08:02:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:44.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:44.929042+0000) 2022-09-10T08:02:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:45 smithi104 ceph-mon[97044]: pgmap v6833: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:45 smithi110 ceph-mon[97633]: pgmap v6833: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:45.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:45.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:45.567252+0000) 2022-09-10T08:02:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:45.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:45.929317+0000) 2022-09-10T08:02:46.930 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:46.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:46.567502+0000) 2022-09-10T08:02:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:46.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:46.929587+0000) 2022-09-10T08:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:47 smithi104 ceph-mon[97044]: pgmap v6834: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:47.901 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:47.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:47.567745+0000) 2022-09-10T08:02:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:47 smithi110 ceph-mon[97633]: pgmap v6834: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:47.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:47.929744+0000) 2022-09-10T08:02:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:02:47.900+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:02:48.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:48.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:48.567913+0000) 2022-09-10T08:02:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:48.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:48.930005+0000) 2022-09-10T08:02:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:49 smithi104 ceph-mon[97044]: pgmap v6835: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:49 smithi110 ceph-mon[97633]: pgmap v6835: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:49.930 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:49.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:49.568123+0000) 2022-09-10T08:02:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:49.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:49.930203+0000) 2022-09-10T08:02:50.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:50.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:50.568269+0000) 2022-09-10T08:02:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:50.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:50.930465+0000) 2022-09-10T08:02:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:51 smithi104 ceph-mon[97044]: pgmap v6836: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:51 smithi110 ceph-mon[97633]: pgmap v6836: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:51.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:51.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:51.568487+0000) 2022-09-10T08:02:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:51.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:51.930696+0000) 2022-09-10T08:02:52.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:52.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:52.568686+0000) 2022-09-10T08:02:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:52.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:52.930895+0000) 2022-09-10T08:02:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:53 smithi104 ceph-mon[97044]: pgmap v6837: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:53 smithi110 ceph-mon[97633]: pgmap v6837: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:53.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:53.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:53.568920+0000) 2022-09-10T08:02:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:53.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:53.931080+0000) 2022-09-10T08:02:54.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:54.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:54.569169+0000) 2022-09-10T08:02:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:54.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:54.931356+0000) 2022-09-10T08:02:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:55 smithi104 ceph-mon[97044]: pgmap v6838: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:55 smithi110 ceph-mon[97633]: pgmap v6838: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:55.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:55.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:55.569399+0000) 2022-09-10T08:02:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:55.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:55.931551+0000) 2022-09-10T08:02:56.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:56.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:56.569616+0000) 2022-09-10T08:02:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:56.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:56.931713+0000) 2022-09-10T08:02:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:57 smithi104 ceph-mon[97044]: pgmap v6839: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:57 smithi110 ceph-mon[97633]: pgmap v6839: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:02:57.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:57.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:57.569869+0000) 2022-09-10T08:02:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:57.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:57.931867+0000) 2022-09-10T08:02:58.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:58.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:58.570069+0000) 2022-09-10T08:02:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:58.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:58.932169+0000) 2022-09-10T08:02:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:02:59 smithi104 ceph-mon[97044]: pgmap v6840: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:02:59 smithi110 ceph-mon[97633]: pgmap v6840: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:02:59.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:02:59.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:59.570273+0000) 2022-09-10T08:03:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:02:59.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:02:59.932436+0000) 2022-09-10T08:03:00.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:00.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:00.570430+0000) 2022-09-10T08:03:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:00.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:00.932631+0000) 2022-09-10T08:03:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:01 smithi104 ceph-mon[97044]: pgmap v6841: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:01 smithi110 ceph-mon[97633]: pgmap v6841: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:01.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:01.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:01.570567+0000) 2022-09-10T08:03:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:01.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:01.932829+0000) 2022-09-10T08:03:02.917 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:02.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:02.570781+0000) 2022-09-10T08:03:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:03:02.915+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:03:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:02.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:02.933075+0000) 2022-09-10T08:03:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:03 smithi104 ceph-mon[97044]: pgmap v6842: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:03 smithi110 ceph-mon[97633]: pgmap v6842: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:03.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:03.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:03.570982+0000) 2022-09-10T08:03:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:03.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:03.933316+0000) 2022-09-10T08:03:04.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:04.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:04.571182+0000) 2022-09-10T08:03:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:04.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:04.933486+0000) 2022-09-10T08:03:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:05 smithi104 ceph-mon[97044]: pgmap v6843: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:05 smithi110 ceph-mon[97633]: pgmap v6843: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:05.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:05.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:05.571384+0000) 2022-09-10T08:03:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:05.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:05.933732+0000) 2022-09-10T08:03:06.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:06.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:06.571631+0000) 2022-09-10T08:03:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:06.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:06.933979+0000) 2022-09-10T08:03:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:07 smithi104 ceph-mon[97044]: pgmap v6844: 129 pgs: 129 active+clean; 8.4 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:03:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:07 smithi110 ceph-mon[97633]: pgmap v6844: 129 pgs: 129 active+clean; 8.4 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:03:07.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:07.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:07.571791+0000) 2022-09-10T08:03:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:07.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:07.934169+0000) 2022-09-10T08:03:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:08 smithi104 ceph-mon[97044]: osdmap e508: 12 total, 12 up, 12 in 2022-09-10T08:03:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:08 smithi110 ceph-mon[97633]: osdmap e508: 12 total, 12 up, 12 in 2022-09-10T08:03:08.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:08.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:08.571964+0000) 2022-09-10T08:03:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:08.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:08.934368+0000) 2022-09-10T08:03:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:09 smithi104 ceph-mon[97044]: pgmap v6846: 129 pgs: 129 active+clean; 8.4 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:09 smithi110 ceph-mon[97633]: pgmap v6846: 129 pgs: 129 active+clean; 8.4 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:09.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:09.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:09.572178+0000) 2022-09-10T08:03:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:09.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:09.934644+0000) 2022-09-10T08:03:10.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:10.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:10.572390+0000) 2022-09-10T08:03:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:10.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:10.934876+0000) 2022-09-10T08:03:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:11 smithi104 ceph-mon[97044]: pgmap v6847: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:11 smithi110 ceph-mon[97633]: pgmap v6847: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:11.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:11.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:11.572598+0000) 2022-09-10T08:03:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:11.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:11.935133+0000) 2022-09-10T08:03:12.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:12.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:12.572834+0000) 2022-09-10T08:03:13.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:12.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:12.935313+0000) 2022-09-10T08:03:13.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:13 smithi104 ceph-mon[97044]: pgmap v6848: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:13 smithi110 ceph-mon[97633]: pgmap v6848: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:13.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:13.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:13.573107+0000) 2022-09-10T08:03:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:13.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:13.935463+0000) 2022-09-10T08:03:14.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:14.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:14.573315+0000) 2022-09-10T08:03:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:14.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:14.935695+0000) 2022-09-10T08:03:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:15 smithi104 ceph-mon[97044]: pgmap v6849: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:15 smithi104 ceph-mon[97044]: osdmap e509: 12 total, 12 up, 12 in 2022-09-10T08:03:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:15 smithi110 ceph-mon[97633]: pgmap v6849: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:03:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:15 smithi110 ceph-mon[97633]: osdmap e509: 12 total, 12 up, 12 in 2022-09-10T08:03:15.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:15.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:15.573481+0000) 2022-09-10T08:03:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:15.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:15.935926+0000) 2022-09-10T08:03:16.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:16.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:16.573670+0000) 2022-09-10T08:03:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:16.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:16.936132+0000) 2022-09-10T08:03:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:17 smithi104 ceph-mon[97044]: pgmap v6851: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:03:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:17 smithi110 ceph-mon[97633]: pgmap v6851: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:03:17.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:17.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:17.573907+0000) 2022-09-10T08:03:18.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:03:17.930+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:03:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:17.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:17.936306+0000) 2022-09-10T08:03:18.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:18.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:18.574159+0000) 2022-09-10T08:03:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:18.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:18.936533+0000) 2022-09-10T08:03:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:19 smithi104 ceph-mon[97044]: pgmap v6852: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:03:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:19 smithi110 ceph-mon[97633]: pgmap v6852: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:03:19.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:19.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:19.574339+0000) 2022-09-10T08:03:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:19.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:19.936727+0000) 2022-09-10T08:03:20.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:20.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:20.574540+0000) 2022-09-10T08:03:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:20.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:20.936906+0000) 2022-09-10T08:03:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:21 smithi104 ceph-mon[97044]: pgmap v6853: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:03:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:21 smithi110 ceph-mon[97633]: pgmap v6853: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:03:21.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:21.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:21.574752+0000) 2022-09-10T08:03:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:21.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:21.937168+0000) 2022-09-10T08:03:22.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:22.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:22.574979+0000) 2022-09-10T08:03:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:22.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:22.937359+0000) 2022-09-10T08:03:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:23 smithi104 ceph-mon[97044]: pgmap v6854: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:03:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:23 smithi110 ceph-mon[97633]: pgmap v6854: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:03:23.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:23.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:23.575171+0000) 2022-09-10T08:03:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:23.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:23.937613+0000) 2022-09-10T08:03:24.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:24.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:24.575391+0000) 2022-09-10T08:03:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:24.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:24.937849+0000) 2022-09-10T08:03:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:25 smithi104 ceph-mon[97044]: pgmap v6855: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:03:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:25 smithi110 ceph-mon[97633]: pgmap v6855: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:03:25.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:25.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:25.575582+0000) 2022-09-10T08:03:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:25.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:25.938081+0000) 2022-09-10T08:03:26.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:26.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:26.575812+0000) 2022-09-10T08:03:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:26.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:26.938343+0000) 2022-09-10T08:03:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:27 smithi104 ceph-mon[97044]: pgmap v6856: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:03:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:27 smithi110 ceph-mon[97633]: pgmap v6856: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:03:27.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:27.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:27.576019+0000) 2022-09-10T08:03:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:27.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:27.938531+0000) 2022-09-10T08:03:28.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:28.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:28.576193+0000) 2022-09-10T08:03:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:28.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:28.938724+0000) 2022-09-10T08:03:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:29 smithi104 ceph-mon[97044]: pgmap v6857: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:29 smithi110 ceph-mon[97633]: pgmap v6857: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:29.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:29.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:29.576460+0000) 2022-09-10T08:03:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:29.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:29.938925+0000) 2022-09-10T08:03:30.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:30.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:30.576678+0000) 2022-09-10T08:03:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:30.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:30.939169+0000) 2022-09-10T08:03:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:31 smithi104 ceph-mon[97044]: pgmap v6858: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:31 smithi110 ceph-mon[97633]: pgmap v6858: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:31.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:31.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:31.576917+0000) 2022-09-10T08:03:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:31.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:31.939330+0000) 2022-09-10T08:03:32.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:32.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:32.577161+0000) 2022-09-10T08:03:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:32.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:32.939524+0000) 2022-09-10T08:03:33.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:03:32.945+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:03:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:33 smithi104 ceph-mon[97044]: pgmap v6859: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:33 smithi110 ceph-mon[97633]: pgmap v6859: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:33.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:33.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:33.577419+0000) 2022-09-10T08:03:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:33.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:33.939718+0000) 2022-09-10T08:03:34.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:34.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:34.577641+0000) 2022-09-10T08:03:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:34.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:34.939982+0000) 2022-09-10T08:03:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:35 smithi104 ceph-mon[97044]: pgmap v6860: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:35 smithi110 ceph-mon[97633]: pgmap v6860: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:35.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:35.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:35.577890+0000) 2022-09-10T08:03:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:35.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:35.940210+0000) 2022-09-10T08:03:36.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:36.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:36.578059+0000) 2022-09-10T08:03:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:36.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:36.940413+0000) 2022-09-10T08:03:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:37 smithi104 ceph-mon[97044]: pgmap v6861: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:37 smithi110 ceph-mon[97633]: pgmap v6861: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:37.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:37.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:37.578274+0000) 2022-09-10T08:03:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:37.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:37.940632+0000) 2022-09-10T08:03:38.868 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:38.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:38.578453+0000) 2022-09-10T08:03:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:38.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:38.940816+0000) 2022-09-10T08:03:39.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:39 smithi104 ceph-mon[97044]: pgmap v6862: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:39 smithi110 ceph-mon[97633]: pgmap v6862: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:39.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:39.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:39.578704+0000) 2022-09-10T08:03:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:39.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:39.941066+0000) 2022-09-10T08:03:40.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:40.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:40.578873+0000) 2022-09-10T08:03:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:40.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:40.941346+0000) 2022-09-10T08:03:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:41 smithi104 ceph-mon[97044]: pgmap v6863: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:41 smithi110 ceph-mon[97633]: pgmap v6863: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:41.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:41.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:41.579180+0000) 2022-09-10T08:03:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:41.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:41.941600+0000) 2022-09-10T08:03:42.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:42.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:42.579353+0000) 2022-09-10T08:03:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:42.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:42.941841+0000) 2022-09-10T08:03:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:43 smithi104 ceph-mon[97044]: pgmap v6864: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:43 smithi110 ceph-mon[97633]: pgmap v6864: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:43.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:43.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:43.579579+0000) 2022-09-10T08:03:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:43.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:43.942045+0000) 2022-09-10T08:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:03:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:03:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:03:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:03:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:03:44.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:44.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:44.579745+0000) 2022-09-10T08:03:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:44.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:44.942270+0000) 2022-09-10T08:03:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:45 smithi104 ceph-mon[97044]: pgmap v6865: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:45 smithi110 ceph-mon[97633]: pgmap v6865: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:45.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:45.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:45.579960+0000) 2022-09-10T08:03:46.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:45.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:45.942482+0000) 2022-09-10T08:03:46.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:46.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:46.580220+0000) 2022-09-10T08:03:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:46.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:46.942729+0000) 2022-09-10T08:03:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:47 smithi104 ceph-mon[97044]: pgmap v6866: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:47 smithi110 ceph-mon[97633]: pgmap v6866: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:47.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:47.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:47.580408+0000) 2022-09-10T08:03:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:47.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:47.942980+0000) 2022-09-10T08:03:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:03:47.962+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:03:48.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:48.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:48.580613+0000) 2022-09-10T08:03:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:48.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:48.943169+0000) 2022-09-10T08:03:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:49 smithi110 ceph-mon[97633]: pgmap v6867: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:49.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:49.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:49.580850+0000) 2022-09-10T08:03:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:49 smithi104 ceph-mon[97044]: pgmap v6867: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:03:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:49.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:49.943349+0000) 2022-09-10T08:03:50.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:50.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:50.581136+0000) 2022-09-10T08:03:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:50.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:50.943544+0000) 2022-09-10T08:03:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:51 smithi110 ceph-mon[97633]: pgmap v6868: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:51.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:51.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:51.581327+0000) 2022-09-10T08:03:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:51 smithi104 ceph-mon[97044]: pgmap v6868: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:51.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:51.943729+0000) 2022-09-10T08:03:52.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:52.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:52.581544+0000) 2022-09-10T08:03:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:52.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:52.943965+0000) 2022-09-10T08:03:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:53 smithi110 ceph-mon[97633]: pgmap v6869: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:03:53.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:53.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:53.581720+0000) 2022-09-10T08:03:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:53 smithi104 ceph-mon[97044]: pgmap v6869: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:03:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:53.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:53.944209+0000) 2022-09-10T08:03:54.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:54.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:54.581946+0000) 2022-09-10T08:03:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:54.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:54.944463+0000) 2022-09-10T08:03:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:55 smithi110 ceph-mon[97633]: pgmap v6870: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:03:55.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:55.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:55.582140+0000) 2022-09-10T08:03:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:55 smithi104 ceph-mon[97044]: pgmap v6870: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:03:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:55.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:55.944707+0000) 2022-09-10T08:03:56.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:56.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:56.582358+0000) 2022-09-10T08:03:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:56.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:56.944938+0000) 2022-09-10T08:03:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:57 smithi110 ceph-mon[97633]: pgmap v6871: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:57.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:57.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:57.582585+0000) 2022-09-10T08:03:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:57 smithi104 ceph-mon[97044]: pgmap v6871: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:03:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:57.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:57.945185+0000) 2022-09-10T08:03:58.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:58.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:58.582840+0000) 2022-09-10T08:03:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:58.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:58.945378+0000) 2022-09-10T08:03:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:03:59 smithi110 ceph-mon[97633]: pgmap v6872: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:03:59.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:03:59.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:59.583152+0000) 2022-09-10T08:04:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:03:59 smithi104 ceph-mon[97044]: pgmap v6872: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:04:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:03:59.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:03:59.945576+0000) 2022-09-10T08:04:00.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:00.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:00.583292+0000) 2022-09-10T08:04:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:00.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:00.945773+0000) 2022-09-10T08:04:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:01 smithi110 ceph-mon[97633]: pgmap v6873: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:01.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:01.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:01.583476+0000) 2022-09-10T08:04:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:01 smithi104 ceph-mon[97044]: pgmap v6873: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:01.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:01.945990+0000) 2022-09-10T08:04:02.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:02.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:02.583671+0000) 2022-09-10T08:04:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:02.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:02.946173+0000) 2022-09-10T08:04:03.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:04:02.976+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:04:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:03 smithi110 ceph-mon[97633]: pgmap v6874: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:03.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:03.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:03.583815+0000) 2022-09-10T08:04:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:03 smithi104 ceph-mon[97044]: pgmap v6874: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:03.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:03.946367+0000) 2022-09-10T08:04:04.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:04.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:04.584030+0000) 2022-09-10T08:04:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:04.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:04.946610+0000) 2022-09-10T08:04:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:05 smithi110 ceph-mon[97633]: pgmap v6875: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:05.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:05.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:05.584253+0000) 2022-09-10T08:04:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:05 smithi104 ceph-mon[97044]: pgmap v6875: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:05.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:05.946857+0000) 2022-09-10T08:04:06.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:06.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:06.584504+0000) 2022-09-10T08:04:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:06.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:06.947071+0000) 2022-09-10T08:04:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:07 smithi110 ceph-mon[97633]: pgmap v6876: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:04:07.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:07.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:07.584683+0000) 2022-09-10T08:04:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:07 smithi104 ceph-mon[97044]: pgmap v6876: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:04:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:07.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:07.947363+0000) 2022-09-10T08:04:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:08 smithi110 ceph-mon[97633]: osdmap e510: 12 total, 12 up, 12 in 2022-09-10T08:04:08.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:08.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:08.584897+0000) 2022-09-10T08:04:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:08 smithi104 ceph-mon[97044]: osdmap e510: 12 total, 12 up, 12 in 2022-09-10T08:04:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:08.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:08.947622+0000) 2022-09-10T08:04:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:09 smithi110 ceph-mon[97633]: pgmap v6878: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:09.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:09.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:09.585184+0000) 2022-09-10T08:04:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:09 smithi104 ceph-mon[97044]: pgmap v6878: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:09.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:09.947856+0000) 2022-09-10T08:04:10.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:10.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:10.585365+0000) 2022-09-10T08:04:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:10.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:10.948079+0000) 2022-09-10T08:04:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:11 smithi110 ceph-mon[97633]: pgmap v6879: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:11.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:11.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:11.585552+0000) 2022-09-10T08:04:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:11 smithi104 ceph-mon[97044]: pgmap v6879: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:11.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:11.948357+0000) 2022-09-10T08:04:12.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:12.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:12.585717+0000) 2022-09-10T08:04:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:12.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:12.948626+0000) 2022-09-10T08:04:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:13 smithi110 ceph-mon[97633]: pgmap v6880: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:13.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:13.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:13.585909+0000) 2022-09-10T08:04:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:13 smithi104 ceph-mon[97044]: pgmap v6880: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:13.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:13.948833+0000) 2022-09-10T08:04:14.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:14.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:14.586196+0000) 2022-09-10T08:04:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:14.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:14.949016+0000) 2022-09-10T08:04:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:15 smithi110 ceph-mon[97633]: pgmap v6881: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:15 smithi110 ceph-mon[97633]: osdmap e511: 12 total, 12 up, 12 in 2022-09-10T08:04:15.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:15.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:15.586356+0000) 2022-09-10T08:04:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:15 smithi104 ceph-mon[97044]: pgmap v6881: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:04:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:15 smithi104 ceph-mon[97044]: osdmap e511: 12 total, 12 up, 12 in 2022-09-10T08:04:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:15.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:15.949258+0000) 2022-09-10T08:04:16.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:16.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:16.586519+0000) 2022-09-10T08:04:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:16.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:16.949437+0000) 2022-09-10T08:04:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:17 smithi110 ceph-mon[97633]: pgmap v6883: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:04:17.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:17.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:17.586733+0000) 2022-09-10T08:04:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:17 smithi104 ceph-mon[97044]: pgmap v6883: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:04:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:17.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:17.949646+0000) 2022-09-10T08:04:18.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:04:17.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:04:18.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:18.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:18.586954+0000) 2022-09-10T08:04:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:18.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:18.949890+0000) 2022-09-10T08:04:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:19 smithi110 ceph-mon[97633]: pgmap v6884: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:04:19.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:19.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:19.587197+0000) 2022-09-10T08:04:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:19 smithi104 ceph-mon[97044]: pgmap v6884: 129 pgs: 129 active+clean; 8.4 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:04:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:19.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:19.950172+0000) 2022-09-10T08:04:20.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:20.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:20.587380+0000) 2022-09-10T08:04:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:20.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:20.950367+0000) 2022-09-10T08:04:21.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:21 smithi110 ceph-mon[97633]: pgmap v6885: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:04:21.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:21.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:21.587583+0000) 2022-09-10T08:04:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:21 smithi104 ceph-mon[97044]: pgmap v6885: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:04:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:21.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:21.950587+0000) 2022-09-10T08:04:22.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:22.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:22.587762+0000) 2022-09-10T08:04:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:22.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:22.950828+0000) 2022-09-10T08:04:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:23 smithi110 ceph-mon[97633]: pgmap v6886: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:04:23.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:23.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:23.587982+0000) 2022-09-10T08:04:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:23 smithi104 ceph-mon[97044]: pgmap v6886: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:04:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:23.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:23.951057+0000) 2022-09-10T08:04:24.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:24.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:24.588156+0000) 2022-09-10T08:04:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:24.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:24.951263+0000) 2022-09-10T08:04:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:25 smithi110 ceph-mon[97633]: pgmap v6887: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:04:25.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:25.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:25.588373+0000) 2022-09-10T08:04:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:25 smithi104 ceph-mon[97044]: pgmap v6887: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:04:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:25.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:25.951506+0000) 2022-09-10T08:04:26.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:26.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:26.588620+0000) 2022-09-10T08:04:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:26.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:26.951651+0000) 2022-09-10T08:04:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:27 smithi110 ceph-mon[97633]: pgmap v6888: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:04:27.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:27.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:27.588828+0000) 2022-09-10T08:04:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:27 smithi104 ceph-mon[97044]: pgmap v6888: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:04:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:27.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:27.951875+0000) 2022-09-10T08:04:28.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:28.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:28.589046+0000) 2022-09-10T08:04:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:28.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:28.952151+0000) 2022-09-10T08:04:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:29 smithi110 ceph-mon[97633]: pgmap v6889: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:29.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:29.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:29.589328+0000) 2022-09-10T08:04:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:29 smithi104 ceph-mon[97044]: pgmap v6889: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:29.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:29.952374+0000) 2022-09-10T08:04:30.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:30.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:30.589586+0000) 2022-09-10T08:04:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:30.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:30.952611+0000) 2022-09-10T08:04:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:31 smithi110 ceph-mon[97633]: pgmap v6890: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:31.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:31.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:31.589849+0000) 2022-09-10T08:04:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:31 smithi104 ceph-mon[97044]: pgmap v6890: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:31.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:31.952801+0000) 2022-09-10T08:04:32.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:32.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:32.590121+0000) 2022-09-10T08:04:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:32.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:32.952991+0000) 2022-09-10T08:04:33.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:04:33.006+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:04:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:33 smithi110 ceph-mon[97633]: pgmap v6891: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:33.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:33.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:33.590285+0000) 2022-09-10T08:04:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:33 smithi104 ceph-mon[97044]: pgmap v6891: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:33.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:33.953235+0000) 2022-09-10T08:04:34.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:34.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:34.590451+0000) 2022-09-10T08:04:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:34.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:34.953463+0000) 2022-09-10T08:04:35.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:35 smithi110 ceph-mon[97633]: pgmap v6892: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:35.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:35.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:35.590673+0000) 2022-09-10T08:04:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:35 smithi104 ceph-mon[97044]: pgmap v6892: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:35.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:35.953717+0000) 2022-09-10T08:04:36.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:36.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:36.590872+0000) 2022-09-10T08:04:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:36.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:36.953974+0000) 2022-09-10T08:04:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:37 smithi110 ceph-mon[97633]: pgmap v6893: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:37.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:37.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:37.591083+0000) 2022-09-10T08:04:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:37 smithi104 ceph-mon[97044]: pgmap v6893: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:37.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:37.954240+0000) 2022-09-10T08:04:38.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:38.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:38.591307+0000) 2022-09-10T08:04:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:38.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:38.954483+0000) 2022-09-10T08:04:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:39 smithi110 ceph-mon[97633]: pgmap v6894: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:39.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:39.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:39.591544+0000) 2022-09-10T08:04:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:39 smithi104 ceph-mon[97044]: pgmap v6894: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:39.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:39.954748+0000) 2022-09-10T08:04:40.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:40.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:40.591669+0000) 2022-09-10T08:04:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:40.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:40.954979+0000) 2022-09-10T08:04:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:41 smithi110 ceph-mon[97633]: pgmap v6895: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:41.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:41.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:41.591841+0000) 2022-09-10T08:04:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:41 smithi104 ceph-mon[97044]: pgmap v6895: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:41.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:41.955207+0000) 2022-09-10T08:04:42.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:42.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:42.592072+0000) 2022-09-10T08:04:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:42.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:42.955414+0000) 2022-09-10T08:04:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:43 smithi110 ceph-mon[97633]: pgmap v6896: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:43.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:43.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:43.592282+0000) 2022-09-10T08:04:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:43 smithi104 ceph-mon[97044]: pgmap v6896: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:43.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:43.955639+0000) 2022-09-10T08:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:04:44.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:44.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:44.592519+0000) 2022-09-10T08:04:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:04:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:04:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:04:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:04:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:44.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:44.955884+0000) 2022-09-10T08:04:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:45 smithi110 ceph-mon[97633]: pgmap v6897: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:45.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:45.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:45.592717+0000) 2022-09-10T08:04:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:45 smithi104 ceph-mon[97044]: pgmap v6897: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:45.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:45.956156+0000) 2022-09-10T08:04:46.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:46.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:46.592969+0000) 2022-09-10T08:04:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:46.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:46.956366+0000) 2022-09-10T08:04:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:47 smithi110 ceph-mon[97633]: pgmap v6898: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:47.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:47.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:47.593139+0000) 2022-09-10T08:04:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:47 smithi104 ceph-mon[97044]: pgmap v6898: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:47.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:47.956617+0000) 2022-09-10T08:04:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:04:48.021+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:04:48.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:48.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:48.593339+0000) 2022-09-10T08:04:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:48.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:48.956842+0000) 2022-09-10T08:04:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:49 smithi110 ceph-mon[97633]: pgmap v6899: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:49.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:49.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:49.593610+0000) 2022-09-10T08:04:50.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:49 smithi104 ceph-mon[97044]: pgmap v6899: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:49.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:49.957070+0000) 2022-09-10T08:04:50.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:50.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:50.593806+0000) 2022-09-10T08:04:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:50.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:50.957310+0000) 2022-09-10T08:04:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:51 smithi110 ceph-mon[97633]: pgmap v6900: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:51 smithi104 ceph-mon[97044]: pgmap v6900: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:51.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:51.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:51.594011+0000) 2022-09-10T08:04:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:51.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:51.957484+0000) 2022-09-10T08:04:52.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:52.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:52.594220+0000) 2022-09-10T08:04:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:52.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:52.957665+0000) 2022-09-10T08:04:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:53 smithi110 ceph-mon[97633]: pgmap v6901: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:53 smithi104 ceph-mon[97044]: pgmap v6901: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:53.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:53.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:53.594495+0000) 2022-09-10T08:04:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:53.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:53.957906+0000) 2022-09-10T08:04:54.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:54.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:54.594717+0000) 2022-09-10T08:04:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:54.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:54.958172+0000) 2022-09-10T08:04:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:55 smithi110 ceph-mon[97633]: pgmap v6902: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:55 smithi104 ceph-mon[97044]: pgmap v6902: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:55.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:55.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:55.594924+0000) 2022-09-10T08:04:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:55.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:55.958376+0000) 2022-09-10T08:04:56.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:56.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:56.595153+0000) 2022-09-10T08:04:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:56.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:56.958543+0000) 2022-09-10T08:04:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:57 smithi110 ceph-mon[97633]: pgmap v6903: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:57 smithi104 ceph-mon[97044]: pgmap v6903: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:04:57.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:57.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:57.595375+0000) 2022-09-10T08:04:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:57.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:57.958711+0000) 2022-09-10T08:04:58.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:58.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:58.595632+0000) 2022-09-10T08:04:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:58.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:58.958942+0000) 2022-09-10T08:04:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:04:59 smithi110 ceph-mon[97633]: pgmap v6904: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:04:59 smithi104 ceph-mon[97044]: pgmap v6904: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:04:59.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:04:59.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:59.595846+0000) 2022-09-10T08:05:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:04:59.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:04:59.959213+0000) 2022-09-10T08:05:00.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:00.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:00.596047+0000) 2022-09-10T08:05:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:00.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:00.959371+0000) 2022-09-10T08:05:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:01 smithi104 ceph-mon[97044]: pgmap v6905: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:01 smithi110 ceph-mon[97633]: pgmap v6905: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:01.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:01.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:01.596290+0000) 2022-09-10T08:05:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:01.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:01.959570+0000) 2022-09-10T08:05:02.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:02.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:02.596472+0000) 2022-09-10T08:05:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:02.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:02.959825+0000) 2022-09-10T08:05:03.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:05:03.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:05:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:03 smithi104 ceph-mon[97044]: pgmap v6906: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:03 smithi110 ceph-mon[97633]: pgmap v6906: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:03.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:03.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:03.596651+0000) 2022-09-10T08:05:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:03.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:03.959988+0000) 2022-09-10T08:05:04.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:04.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:04.596886+0000) 2022-09-10T08:05:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:04.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:04.960230+0000) 2022-09-10T08:05:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:05 smithi104 ceph-mon[97044]: pgmap v6907: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:05 smithi110 ceph-mon[97633]: pgmap v6907: 129 pgs: 129 active+clean; 8.4 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:05.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:05.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:05.597088+0000) 2022-09-10T08:05:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:05.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:05.960472+0000) 2022-09-10T08:05:06.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:06.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:06.597299+0000) 2022-09-10T08:05:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:06.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:06.960592+0000) 2022-09-10T08:05:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:07 smithi104 ceph-mon[97044]: pgmap v6908: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:05:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:07 smithi110 ceph-mon[97633]: pgmap v6908: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:05:07.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:07.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:07.597480+0000) 2022-09-10T08:05:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:07.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:07.960804+0000) 2022-09-10T08:05:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:08 smithi104 ceph-mon[97044]: osdmap e512: 12 total, 12 up, 12 in 2022-09-10T08:05:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:08 smithi110 ceph-mon[97633]: osdmap e512: 12 total, 12 up, 12 in 2022-09-10T08:05:08.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:08.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:08.597642+0000) 2022-09-10T08:05:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:08.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:08.960996+0000) 2022-09-10T08:05:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:09 smithi104 ceph-mon[97044]: pgmap v6910: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:09 smithi110 ceph-mon[97633]: pgmap v6910: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:09.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:09.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:09.597824+0000) 2022-09-10T08:05:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:09.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:09.961286+0000) 2022-09-10T08:05:10.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:10.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:10.598026+0000) 2022-09-10T08:05:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:10.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:10.961469+0000) 2022-09-10T08:05:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:11 smithi104 ceph-mon[97044]: pgmap v6911: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:11 smithi110 ceph-mon[97633]: pgmap v6911: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:11.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:11.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:11.598266+0000) 2022-09-10T08:05:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:11.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:11.961649+0000) 2022-09-10T08:05:12.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:12.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:12.598497+0000) 2022-09-10T08:05:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:12.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:12.961881+0000) 2022-09-10T08:05:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:13 smithi104 ceph-mon[97044]: pgmap v6912: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:13 smithi110 ceph-mon[97633]: pgmap v6912: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:13.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:13.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:13.598673+0000) 2022-09-10T08:05:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:13.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:13.962133+0000) 2022-09-10T08:05:14.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:14.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:14.598864+0000) 2022-09-10T08:05:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:14.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:14.962286+0000) 2022-09-10T08:05:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:15 smithi104 ceph-mon[97044]: pgmap v6913: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:15 smithi104 ceph-mon[97044]: osdmap e513: 12 total, 12 up, 12 in 2022-09-10T08:05:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:15 smithi110 ceph-mon[97633]: pgmap v6913: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:05:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:15 smithi110 ceph-mon[97633]: osdmap e513: 12 total, 12 up, 12 in 2022-09-10T08:05:15.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:15.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:15.599092+0000) 2022-09-10T08:05:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:15.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:15.962586+0000) 2022-09-10T08:05:16.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:16.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:16.599335+0000) 2022-09-10T08:05:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:16.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:16.962834+0000) 2022-09-10T08:05:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:17 smithi104 ceph-mon[97044]: pgmap v6915: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:05:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:17 smithi110 ceph-mon[97633]: pgmap v6915: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:05:17.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:17.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:17.599580+0000) 2022-09-10T08:05:18.326 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:17.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:17.963126+0000) 2022-09-10T08:05:18.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:05:18.045+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:05:18.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:18.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:18.599813+0000) 2022-09-10T08:05:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:18.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:18.963254+0000) 2022-09-10T08:05:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:19 smithi104 ceph-mon[97044]: pgmap v6916: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:05:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:19 smithi110 ceph-mon[97633]: pgmap v6916: 129 pgs: 129 active+clean; 8.4 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:05:19.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:19.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:19.600060+0000) 2022-09-10T08:05:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:19.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:19.963482+0000) 2022-09-10T08:05:20.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:20.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:20.600291+0000) 2022-09-10T08:05:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:20.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:20.963704+0000) 2022-09-10T08:05:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:21 smithi104 ceph-mon[97044]: pgmap v6917: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:05:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:21 smithi110 ceph-mon[97633]: pgmap v6917: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:05:21.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:21.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:21.600464+0000) 2022-09-10T08:05:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:21.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:21.963929+0000) 2022-09-10T08:05:22.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:22.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:22.600650+0000) 2022-09-10T08:05:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:22.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:22.964147+0000) 2022-09-10T08:05:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:23 smithi104 ceph-mon[97044]: pgmap v6918: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:05:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:23 smithi110 ceph-mon[97633]: pgmap v6918: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:05:23.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:23.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:23.600903+0000) 2022-09-10T08:05:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:23.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:23.964286+0000) 2022-09-10T08:05:24.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:24.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:24.601124+0000) 2022-09-10T08:05:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:24.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:24.964471+0000) 2022-09-10T08:05:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:25 smithi104 ceph-mon[97044]: pgmap v6919: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:05:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:25 smithi110 ceph-mon[97633]: pgmap v6919: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:05:25.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:25.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:25.601361+0000) 2022-09-10T08:05:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:25.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:25.964761+0000) 2022-09-10T08:05:26.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:26.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:26.601639+0000) 2022-09-10T08:05:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:26.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:26.965015+0000) 2022-09-10T08:05:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:27 smithi104 ceph-mon[97044]: pgmap v6920: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:05:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:27 smithi110 ceph-mon[97633]: pgmap v6920: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:05:27.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:27.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:27.601850+0000) 2022-09-10T08:05:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:27.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:27.965282+0000) 2022-09-10T08:05:28.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:28.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:28.602068+0000) 2022-09-10T08:05:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:28.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:28.965464+0000) 2022-09-10T08:05:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:29 smithi104 ceph-mon[97044]: pgmap v6921: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:29 smithi110 ceph-mon[97633]: pgmap v6921: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:29.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:29.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:29.602370+0000) 2022-09-10T08:05:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:29.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:29.965724+0000) 2022-09-10T08:05:30.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:30.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:30.602589+0000) 2022-09-10T08:05:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:30.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:30.965926+0000) 2022-09-10T08:05:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:31 smithi104 ceph-mon[97044]: pgmap v6922: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:31 smithi110 ceph-mon[97633]: pgmap v6922: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:31.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:31.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:31.602863+0000) 2022-09-10T08:05:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:31.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:31.966184+0000) 2022-09-10T08:05:32.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:32.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:32.603048+0000) 2022-09-10T08:05:33.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:05:33.061+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:05:33.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:32.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:32.966448+0000) 2022-09-10T08:05:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:33 smithi104 ceph-mon[97044]: pgmap v6923: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:33 smithi110 ceph-mon[97633]: pgmap v6923: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:33.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:33.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:33.603187+0000) 2022-09-10T08:05:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:33.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:33.966702+0000) 2022-09-10T08:05:34.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:34.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:34.603357+0000) 2022-09-10T08:05:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:34.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:34.966903+0000) 2022-09-10T08:05:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:35 smithi104 ceph-mon[97044]: pgmap v6924: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:35 smithi110 ceph-mon[97633]: pgmap v6924: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:35.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:35.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:35.603589+0000) 2022-09-10T08:05:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:35.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:35.967185+0000) 2022-09-10T08:05:36.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:36.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:36.603817+0000) 2022-09-10T08:05:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:36.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:36.967349+0000) 2022-09-10T08:05:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:37 smithi104 ceph-mon[97044]: pgmap v6925: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:37 smithi110 ceph-mon[97633]: pgmap v6925: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:37.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:37.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:37.604036+0000) 2022-09-10T08:05:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:37.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:37.967517+0000) 2022-09-10T08:05:38.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:38.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:38.604253+0000) 2022-09-10T08:05:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:38.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:38.967748+0000) 2022-09-10T08:05:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:39 smithi104 ceph-mon[97044]: pgmap v6926: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:39 smithi110 ceph-mon[97633]: pgmap v6926: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:39.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:39.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:39.604536+0000) 2022-09-10T08:05:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:39.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:39.968008+0000) 2022-09-10T08:05:40.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:40.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:40.604796+0000) 2022-09-10T08:05:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:40.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:40.968285+0000) 2022-09-10T08:05:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:41 smithi104 ceph-mon[97044]: pgmap v6927: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:41 smithi110 ceph-mon[97633]: pgmap v6927: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:41.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:41.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:41.605045+0000) 2022-09-10T08:05:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:41.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:41.968496+0000) 2022-09-10T08:05:42.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:42.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:42.605246+0000) 2022-09-10T08:05:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:42.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:42.968700+0000) 2022-09-10T08:05:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:43 smithi104 ceph-mon[97044]: pgmap v6928: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:43 smithi110 ceph-mon[97633]: pgmap v6928: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:43.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:43.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:43.605410+0000) 2022-09-10T08:05:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:43.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:43.968869+0000) 2022-09-10T08:05:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:05:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:05:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:05:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:05:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:05:44.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:44.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:44.605606+0000) 2022-09-10T08:05:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:44.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:44.969073+0000) 2022-09-10T08:05:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:45 smithi104 ceph-mon[97044]: pgmap v6929: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:45 smithi110 ceph-mon[97633]: pgmap v6929: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:45.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:45.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:45.605870+0000) 2022-09-10T08:05:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:45.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:45.969293+0000) 2022-09-10T08:05:46.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:46.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:46.606162+0000) 2022-09-10T08:05:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:46.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:46.969452+0000) 2022-09-10T08:05:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:47 smithi104 ceph-mon[97044]: pgmap v6930: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:47 smithi110 ceph-mon[97633]: pgmap v6930: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:47.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:47.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:47.606384+0000) 2022-09-10T08:05:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:47.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:47.969653+0000) 2022-09-10T08:05:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:05:48.071+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:05:48.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:48.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:48.606566+0000) 2022-09-10T08:05:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:48.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:48.969806+0000) 2022-09-10T08:05:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:49 smithi104 ceph-mon[97044]: pgmap v6931: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:49 smithi110 ceph-mon[97633]: pgmap v6931: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:49.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:49.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:49.606827+0000) 2022-09-10T08:05:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:49.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:49.970049+0000) 2022-09-10T08:05:50.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:50.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:50.606976+0000) 2022-09-10T08:05:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:50.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:50.970261+0000) 2022-09-10T08:05:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:51 smithi104 ceph-mon[97044]: pgmap v6932: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:51 smithi110 ceph-mon[97633]: pgmap v6932: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:51.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:51.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:51.607218+0000) 2022-09-10T08:05:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:51.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:51.970510+0000) 2022-09-10T08:05:52.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:52.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:52.607445+0000) 2022-09-10T08:05:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:52.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:52.970738+0000) 2022-09-10T08:05:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:53 smithi104 ceph-mon[97044]: pgmap v6933: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:53 smithi110 ceph-mon[97633]: pgmap v6933: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:53.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:53.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:53.607666+0000) 2022-09-10T08:05:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:53.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:53.970944+0000) 2022-09-10T08:05:54.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:54.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:54.607938+0000) 2022-09-10T08:05:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:54.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:54.971175+0000) 2022-09-10T08:05:55.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:55 smithi104 ceph-mon[97044]: pgmap v6934: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:55.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:55 smithi110 ceph-mon[97633]: pgmap v6934: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:55.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:55.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:55.608182+0000) 2022-09-10T08:05:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:55.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:55.971418+0000) 2022-09-10T08:05:56.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:56.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:56.608381+0000) 2022-09-10T08:05:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:56.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:56.971611+0000) 2022-09-10T08:05:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:57 smithi104 ceph-mon[97044]: pgmap v6935: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:57 smithi110 ceph-mon[97633]: pgmap v6935: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:05:57.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:57.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:57.608558+0000) 2022-09-10T08:05:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:57.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:57.971820+0000) 2022-09-10T08:05:58.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:58.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:58.608799+0000) 2022-09-10T08:05:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:58.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:58.972066+0000) 2022-09-10T08:05:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:05:59 smithi104 ceph-mon[97044]: pgmap v6936: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:05:59 smithi110 ceph-mon[97633]: pgmap v6936: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:05:59.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:05:59.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:59.608997+0000) 2022-09-10T08:06:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:05:59.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:05:59.972272+0000) 2022-09-10T08:06:00.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:00.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:00.609187+0000) 2022-09-10T08:06:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:00.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:00.972481+0000) 2022-09-10T08:06:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:01 smithi104 ceph-mon[97044]: pgmap v6937: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:01 smithi110 ceph-mon[97633]: pgmap v6937: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:01.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:01.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:01.609519+0000) 2022-09-10T08:06:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:01.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:01.972727+0000) 2022-09-10T08:06:02.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:02.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:02.609753+0000) 2022-09-10T08:06:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:06:03.087+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:06:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:02.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:02.972880+0000) 2022-09-10T08:06:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:03 smithi104 ceph-mon[97044]: pgmap v6938: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:03 smithi110 ceph-mon[97633]: pgmap v6938: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:03.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:03.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:03.609940+0000) 2022-09-10T08:06:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:03.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:03.973067+0000) 2022-09-10T08:06:04.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:04.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:04.610186+0000) 2022-09-10T08:06:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:04.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:04.973207+0000) 2022-09-10T08:06:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:05 smithi104 ceph-mon[97044]: pgmap v6939: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:05 smithi110 ceph-mon[97633]: pgmap v6939: 129 pgs: 129 active+clean; 8.4 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:05.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:05.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:05.610399+0000) 2022-09-10T08:06:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:05.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:05.973396+0000) 2022-09-10T08:06:06.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:06.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:06.610663+0000) 2022-09-10T08:06:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:06.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:06.973645+0000) 2022-09-10T08:06:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:07 smithi104 ceph-mon[97044]: pgmap v6940: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:06:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:07 smithi110 ceph-mon[97633]: pgmap v6940: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:06:07.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:07.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:07.610847+0000) 2022-09-10T08:06:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:07.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:07.973867+0000) 2022-09-10T08:06:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:08 smithi104 ceph-mon[97044]: osdmap e514: 12 total, 12 up, 12 in 2022-09-10T08:06:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:08 smithi110 ceph-mon[97633]: osdmap e514: 12 total, 12 up, 12 in 2022-09-10T08:06:08.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:08.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:08.611030+0000) 2022-09-10T08:06:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:08.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:08.974068+0000) 2022-09-10T08:06:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:09 smithi104 ceph-mon[97044]: pgmap v6942: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:09 smithi110 ceph-mon[97633]: pgmap v6942: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:09.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:09.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:09.611284+0000) 2022-09-10T08:06:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:09.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:09.974305+0000) 2022-09-10T08:06:10.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:10.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:10.611462+0000) 2022-09-10T08:06:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:10.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:10.974482+0000) 2022-09-10T08:06:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:11 smithi104 ceph-mon[97044]: pgmap v6943: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:11 smithi110 ceph-mon[97633]: pgmap v6943: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:11.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:11.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:11.611616+0000) 2022-09-10T08:06:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:11.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:11.974702+0000) 2022-09-10T08:06:12.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:12.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:12.611794+0000) 2022-09-10T08:06:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:12.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:12.974966+0000) 2022-09-10T08:06:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:13 smithi104 ceph-mon[97044]: pgmap v6944: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:13 smithi110 ceph-mon[97633]: pgmap v6944: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:13.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:13.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:13.611993+0000) 2022-09-10T08:06:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:13.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:13.975223+0000) 2022-09-10T08:06:14.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:14.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:14.612219+0000) 2022-09-10T08:06:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:14.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:14.975439+0000) 2022-09-10T08:06:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:15 smithi110 ceph-mon[97633]: pgmap v6945: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:15 smithi110 ceph-mon[97633]: osdmap e515: 12 total, 12 up, 12 in 2022-09-10T08:06:15.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:15.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:15.612465+0000) 2022-09-10T08:06:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:15 smithi104 ceph-mon[97044]: pgmap v6945: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:06:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:15 smithi104 ceph-mon[97044]: osdmap e515: 12 total, 12 up, 12 in 2022-09-10T08:06:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:15.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:15.975726+0000) 2022-09-10T08:06:16.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:16.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:16.612639+0000) 2022-09-10T08:06:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:16.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:16.975950+0000) 2022-09-10T08:06:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:17 smithi110 ceph-mon[97633]: pgmap v6947: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:06:17.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:17.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:17.612802+0000) 2022-09-10T08:06:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:17 smithi104 ceph-mon[97044]: pgmap v6947: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:06:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:17.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:17.976138+0000) 2022-09-10T08:06:18.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:06:18.098+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:06:18.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:18.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:18.612931+0000) 2022-09-10T08:06:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:18.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:18.976353+0000) 2022-09-10T08:06:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:19 smithi110 ceph-mon[97633]: pgmap v6948: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:06:19.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:19.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:19.613065+0000) 2022-09-10T08:06:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:19 smithi104 ceph-mon[97044]: pgmap v6948: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:06:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:19.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:19.976625+0000) 2022-09-10T08:06:20.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:20.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:20.613202+0000) 2022-09-10T08:06:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:20.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:20.976874+0000) 2022-09-10T08:06:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:21 smithi110 ceph-mon[97633]: pgmap v6949: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:06:21.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:21.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:21.613371+0000) 2022-09-10T08:06:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:21 smithi104 ceph-mon[97044]: pgmap v6949: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:06:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:21.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:21.977056+0000) 2022-09-10T08:06:22.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:22.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:22.613536+0000) 2022-09-10T08:06:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:22.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:22.977300+0000) 2022-09-10T08:06:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:23 smithi110 ceph-mon[97633]: pgmap v6950: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:06:23.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:23.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:23.613697+0000) 2022-09-10T08:06:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:23 smithi104 ceph-mon[97044]: pgmap v6950: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:06:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:23.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:23.977557+0000) 2022-09-10T08:06:24.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:24.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:24.613949+0000) 2022-09-10T08:06:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:24.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:24.977800+0000) 2022-09-10T08:06:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:25 smithi110 ceph-mon[97633]: pgmap v6951: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:06:25.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:25.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:25.614077+0000) 2022-09-10T08:06:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:25 smithi104 ceph-mon[97044]: pgmap v6951: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:06:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:25.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:25.978060+0000) 2022-09-10T08:06:26.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:26.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:26.614236+0000) 2022-09-10T08:06:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:26.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:26.978306+0000) 2022-09-10T08:06:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:27 smithi110 ceph-mon[97633]: pgmap v6952: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:06:27.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:27.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:27.614441+0000) 2022-09-10T08:06:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:27 smithi104 ceph-mon[97044]: pgmap v6952: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:06:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:27.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:27.978545+0000) 2022-09-10T08:06:28.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:28.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:28.614561+0000) 2022-09-10T08:06:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:28.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:28.978781+0000) 2022-09-10T08:06:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:29 smithi110 ceph-mon[97633]: pgmap v6953: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:29.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:29.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:29.614725+0000) 2022-09-10T08:06:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:29 smithi104 ceph-mon[97044]: pgmap v6953: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:29.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:29.979029+0000) 2022-09-10T08:06:30.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:30.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:30.614887+0000) 2022-09-10T08:06:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:30.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:30.979234+0000) 2022-09-10T08:06:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:31 smithi110 ceph-mon[97633]: pgmap v6954: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:31.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:31.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:31.615063+0000) 2022-09-10T08:06:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:31 smithi104 ceph-mon[97044]: pgmap v6954: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:31.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:31.979423+0000) 2022-09-10T08:06:32.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:32.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:32.615248+0000) 2022-09-10T08:06:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:32.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:32.979601+0000) 2022-09-10T08:06:33.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:06:33.099+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:06:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:33 smithi110 ceph-mon[97633]: pgmap v6955: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:33.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:33.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:33.615509+0000) 2022-09-10T08:06:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:33 smithi104 ceph-mon[97044]: pgmap v6955: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:33.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:33.979896+0000) 2022-09-10T08:06:34.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:34.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:34.615709+0000) 2022-09-10T08:06:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:34.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:34.980081+0000) 2022-09-10T08:06:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:35 smithi110 ceph-mon[97633]: pgmap v6956: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:35.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:35.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:35.615852+0000) 2022-09-10T08:06:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:35 smithi104 ceph-mon[97044]: pgmap v6956: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:35.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:35.980329+0000) 2022-09-10T08:06:36.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:36.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:36.616040+0000) 2022-09-10T08:06:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:36.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:36.980631+0000) 2022-09-10T08:06:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:37 smithi110 ceph-mon[97633]: pgmap v6957: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:37.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:37.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:37.616227+0000) 2022-09-10T08:06:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:37 smithi104 ceph-mon[97044]: pgmap v6957: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:37.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:37.980820+0000) 2022-09-10T08:06:38.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:38.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:38.616486+0000) 2022-09-10T08:06:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:38.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:38.981052+0000) 2022-09-10T08:06:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:39 smithi110 ceph-mon[97633]: pgmap v6958: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:39.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:39.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:39.616694+0000) 2022-09-10T08:06:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:39 smithi104 ceph-mon[97044]: pgmap v6958: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:39.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:39.981281+0000) 2022-09-10T08:06:40.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:40.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:40.616893+0000) 2022-09-10T08:06:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:40.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:40.981518+0000) 2022-09-10T08:06:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:41 smithi110 ceph-mon[97633]: pgmap v6959: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:41.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:41.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:41.617135+0000) 2022-09-10T08:06:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:41 smithi104 ceph-mon[97044]: pgmap v6959: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:41.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:41.981704+0000) 2022-09-10T08:06:42.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:42.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:42.617321+0000) 2022-09-10T08:06:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:42.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:42.981939+0000) 2022-09-10T08:06:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:43 smithi110 ceph-mon[97633]: pgmap v6960: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:43.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:43.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:43.617482+0000) 2022-09-10T08:06:44.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:43 smithi104 ceph-mon[97044]: pgmap v6960: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:43.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:43.982205+0000) 2022-09-10T08:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:06:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:06:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:06:44.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:44.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:44.617677+0000) 2022-09-10T08:06:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:06:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:06:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:06:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:06:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:44.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:44.982474+0000) 2022-09-10T08:06:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:45 smithi110 ceph-mon[97633]: pgmap v6961: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:45.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:45.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:45.617877+0000) 2022-09-10T08:06:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:45 smithi104 ceph-mon[97044]: pgmap v6961: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:45.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:45.982688+0000) 2022-09-10T08:06:46.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:46.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:46.618171+0000) 2022-09-10T08:06:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:46.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:46.982886+0000) 2022-09-10T08:06:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:47 smithi110 ceph-mon[97633]: pgmap v6962: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:47.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:47.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:47.618351+0000) 2022-09-10T08:06:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:47 smithi104 ceph-mon[97044]: pgmap v6962: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:06:48.109+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:06:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:47.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:47.983109+0000) 2022-09-10T08:06:48.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:48.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:48.618538+0000) 2022-09-10T08:06:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:48.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:48.983322+0000) 2022-09-10T08:06:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:49 smithi110 ceph-mon[97633]: pgmap v6963: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:49.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:49.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:49.618800+0000) 2022-09-10T08:06:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:49 smithi104 ceph-mon[97044]: pgmap v6963: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:49.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:49.983507+0000) 2022-09-10T08:06:50.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:50.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:50.618995+0000) 2022-09-10T08:06:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:50.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:50.983704+0000) 2022-09-10T08:06:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:51 smithi110 ceph-mon[97633]: pgmap v6964: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:51.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:51.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:51.619255+0000) 2022-09-10T08:06:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:51 smithi104 ceph-mon[97044]: pgmap v6964: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:51.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:51.983895+0000) 2022-09-10T08:06:52.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:52.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:52.619471+0000) 2022-09-10T08:06:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:52.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:52.984131+0000) 2022-09-10T08:06:53.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:53 smithi110 ceph-mon[97633]: pgmap v6965: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:53.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:53.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:53.619732+0000) 2022-09-10T08:06:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:53 smithi104 ceph-mon[97044]: pgmap v6965: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:53.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:53.984307+0000) 2022-09-10T08:06:54.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:54.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:54.619964+0000) 2022-09-10T08:06:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:54.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:54.984435+0000) 2022-09-10T08:06:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:55 smithi110 ceph-mon[97633]: pgmap v6966: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:55.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:55.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:55.620159+0000) 2022-09-10T08:06:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:55 smithi104 ceph-mon[97044]: pgmap v6966: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:55.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:55.984650+0000) 2022-09-10T08:06:56.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:56.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:56.620301+0000) 2022-09-10T08:06:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:56.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:56.984851+0000) 2022-09-10T08:06:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:57 smithi110 ceph-mon[97633]: pgmap v6967: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:57.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:57.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:57.620499+0000) 2022-09-10T08:06:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:57 smithi104 ceph-mon[97044]: pgmap v6967: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:06:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:57.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:57.985126+0000) 2022-09-10T08:06:58.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:58.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:58.620744+0000) 2022-09-10T08:06:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:58.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:58.985252+0000) 2022-09-10T08:06:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:06:59 smithi110 ceph-mon[97633]: pgmap v6968: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:06:59.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:06:59.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:59.620875+0000) 2022-09-10T08:07:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:06:59 smithi104 ceph-mon[97044]: pgmap v6968: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:06:59.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:06:59.985411+0000) 2022-09-10T08:07:00.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:00.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:00.621077+0000) 2022-09-10T08:07:01.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:00.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:00.985550+0000) 2022-09-10T08:07:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:01 smithi110 ceph-mon[97633]: pgmap v6969: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:01.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:01.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:01.621330+0000) 2022-09-10T08:07:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:01 smithi104 ceph-mon[97044]: pgmap v6969: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:01.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:01.985714+0000) 2022-09-10T08:07:02.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:02.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:02.621599+0000) 2022-09-10T08:07:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:02.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:02.985870+0000) 2022-09-10T08:07:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:07:03.111+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:07:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:03 smithi110 ceph-mon[97633]: pgmap v6970: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:07:03.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:03.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:03.621878+0000) 2022-09-10T08:07:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:03 smithi104 ceph-mon[97044]: pgmap v6970: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:07:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:03.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:03.986039+0000) 2022-09-10T08:07:04.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:04.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:04.622051+0000) 2022-09-10T08:07:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:04.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:04.986193+0000) 2022-09-10T08:07:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:05 smithi110 ceph-mon[97633]: pgmap v6971: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:07:05.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:05.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:05.622248+0000) 2022-09-10T08:07:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:05 smithi104 ceph-mon[97044]: pgmap v6971: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:07:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:05.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:05.986488+0000) 2022-09-10T08:07:06.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:06.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:06.622466+0000) 2022-09-10T08:07:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:06.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:06.986661+0000) 2022-09-10T08:07:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:07 smithi110 ceph-mon[97633]: pgmap v6972: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:07:07.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:07.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:07.622619+0000) 2022-09-10T08:07:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:07 smithi104 ceph-mon[97044]: pgmap v6972: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:07:08.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:07.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:07.986867+0000) 2022-09-10T08:07:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:08 smithi110 ceph-mon[97633]: osdmap e516: 12 total, 12 up, 12 in 2022-09-10T08:07:08.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:08.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:08.622802+0000) 2022-09-10T08:07:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:08 smithi104 ceph-mon[97044]: osdmap e516: 12 total, 12 up, 12 in 2022-09-10T08:07:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:08.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:08.987037+0000) 2022-09-10T08:07:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:09 smithi110 ceph-mon[97633]: pgmap v6974: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:09.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:09.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:09.623067+0000) 2022-09-10T08:07:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:09 smithi104 ceph-mon[97044]: pgmap v6974: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:09.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:09.987203+0000) 2022-09-10T08:07:10.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:10.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:10.623349+0000) 2022-09-10T08:07:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:10.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:10.987382+0000) 2022-09-10T08:07:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:11 smithi110 ceph-mon[97633]: pgmap v6975: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:11.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:11.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:11.623554+0000) 2022-09-10T08:07:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:11 smithi104 ceph-mon[97044]: pgmap v6975: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:11.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:11.987623+0000) 2022-09-10T08:07:12.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:12.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:12.623728+0000) 2022-09-10T08:07:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:12.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:12.987820+0000) 2022-09-10T08:07:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:13 smithi110 ceph-mon[97633]: pgmap v6976: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:13.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:13.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:13.623960+0000) 2022-09-10T08:07:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:13 smithi104 ceph-mon[97044]: pgmap v6976: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:13.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:13.988041+0000) 2022-09-10T08:07:14.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:14.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:14.624213+0000) 2022-09-10T08:07:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:14.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:14.988332+0000) 2022-09-10T08:07:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:15 smithi110 ceph-mon[97633]: pgmap v6977: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:15 smithi110 ceph-mon[97633]: osdmap e517: 12 total, 12 up, 12 in 2022-09-10T08:07:15.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:15.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:15.624446+0000) 2022-09-10T08:07:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:15 smithi104 ceph-mon[97044]: pgmap v6977: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:07:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:15 smithi104 ceph-mon[97044]: osdmap e517: 12 total, 12 up, 12 in 2022-09-10T08:07:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:15.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:15.988559+0000) 2022-09-10T08:07:16.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:16.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:16.624634+0000) 2022-09-10T08:07:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:16.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:16.988797+0000) 2022-09-10T08:07:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:17 smithi110 ceph-mon[97633]: pgmap v6979: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:07:17.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:17.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:17.624796+0000) 2022-09-10T08:07:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:17 smithi104 ceph-mon[97044]: pgmap v6979: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:07:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:17.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:17.988962+0000) 2022-09-10T08:07:18.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:07:18.125+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:07:18.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:18.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:18.625044+0000) 2022-09-10T08:07:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:18.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:18.989198+0000) 2022-09-10T08:07:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:19 smithi110 ceph-mon[97633]: pgmap v6980: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:07:19.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:19.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:19.625289+0000) 2022-09-10T08:07:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:19 smithi104 ceph-mon[97044]: pgmap v6980: 129 pgs: 129 active+clean; 8.4 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:07:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:19.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:19.989394+0000) 2022-09-10T08:07:20.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:20.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:20.625528+0000) 2022-09-10T08:07:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:20.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:20.989583+0000) 2022-09-10T08:07:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:21 smithi110 ceph-mon[97633]: pgmap v6981: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:07:21.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:21.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:21.625786+0000) 2022-09-10T08:07:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:21 smithi104 ceph-mon[97044]: pgmap v6981: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:07:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:21.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:21.989795+0000) 2022-09-10T08:07:22.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:22.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:22.626066+0000) 2022-09-10T08:07:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:22.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:22.990012+0000) 2022-09-10T08:07:23.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:23 smithi110 ceph-mon[97633]: pgmap v6982: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:07:23.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:23.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:23.626254+0000) 2022-09-10T08:07:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:23 smithi104 ceph-mon[97044]: pgmap v6982: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:07:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:23.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:23.990189+0000) 2022-09-10T08:07:24.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:24.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:24.626478+0000) 2022-09-10T08:07:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:24.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:24.990419+0000) 2022-09-10T08:07:25.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:25.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:25.626722+0000) 2022-09-10T08:07:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:25 smithi104 ceph-mon[97044]: pgmap v6983: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:07:26.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:25 smithi110 ceph-mon[97633]: pgmap v6983: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:07:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:25.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:25.990714+0000) 2022-09-10T08:07:26.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:26.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:26.626951+0000) 2022-09-10T08:07:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:26.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:26.990908+0000) 2022-09-10T08:07:27.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:27.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:27.627149+0000) 2022-09-10T08:07:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:27 smithi104 ceph-mon[97044]: pgmap v6984: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:07:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:27 smithi110 ceph-mon[97633]: pgmap v6984: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:07:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:27.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:27.991155+0000) 2022-09-10T08:07:28.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:28.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:28.627357+0000) 2022-09-10T08:07:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:28.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:28.991359+0000) 2022-09-10T08:07:29.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:29.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:29.627542+0000) 2022-09-10T08:07:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:29 smithi104 ceph-mon[97044]: pgmap v6985: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:29 smithi110 ceph-mon[97633]: pgmap v6985: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:29.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:29.991524+0000) 2022-09-10T08:07:30.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:30.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:30.627683+0000) 2022-09-10T08:07:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:30.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:30.991709+0000) 2022-09-10T08:07:31.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:31.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:31.627894+0000) 2022-09-10T08:07:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:31 smithi104 ceph-mon[97044]: pgmap v6986: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:31 smithi110 ceph-mon[97633]: pgmap v6986: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:31.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:31.991851+0000) 2022-09-10T08:07:32.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:32.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:32.628119+0000) 2022-09-10T08:07:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:32.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:32.992144+0000) 2022-09-10T08:07:33.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:07:33.133+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:07:33.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:33.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:33.628343+0000) 2022-09-10T08:07:34.056 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:33 smithi104 ceph-mon[97044]: pgmap v6987: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:33 smithi110 ceph-mon[97633]: pgmap v6987: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:33.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:33.992349+0000) 2022-09-10T08:07:34.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:34.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:34.628597+0000) 2022-09-10T08:07:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:34.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:34.992592+0000) 2022-09-10T08:07:35.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:35.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:35.628778+0000) 2022-09-10T08:07:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:35 smithi104 ceph-mon[97044]: pgmap v6988: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:35 smithi110 ceph-mon[97633]: pgmap v6988: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:35.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:35.992902+0000) 2022-09-10T08:07:36.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:36.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:36.629016+0000) 2022-09-10T08:07:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:36 smithi104 ceph-mon[97044]: pgmap v6989: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:37.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:36 smithi110 ceph-mon[97633]: pgmap v6989: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:36.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:36.993121+0000) 2022-09-10T08:07:37.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:37.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:37.629272+0000) 2022-09-10T08:07:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:37.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:37.993344+0000) 2022-09-10T08:07:38.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:38.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:38.629541+0000) 2022-09-10T08:07:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:38 smithi104 ceph-mon[97044]: pgmap v6990: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:38.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:38.993568+0000) 2022-09-10T08:07:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:38 smithi110 ceph-mon[97633]: pgmap v6990: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:39.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:39.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:39.629797+0000) 2022-09-10T08:07:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:39.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:39.993784+0000) 2022-09-10T08:07:40.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:40.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:40.629944+0000) 2022-09-10T08:07:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:40.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:40.993982+0000) 2022-09-10T08:07:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:41 smithi104 ceph-mon[97044]: pgmap v6991: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:41 smithi110 ceph-mon[97633]: pgmap v6991: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:41.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:41.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:41.630197+0000) 2022-09-10T08:07:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:41.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:41.994185+0000) 2022-09-10T08:07:42.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:42.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:42.630449+0000) 2022-09-10T08:07:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:42.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:42.994331+0000) 2022-09-10T08:07:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:43 smithi104 ceph-mon[97044]: pgmap v6992: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:43 smithi110 ceph-mon[97633]: pgmap v6992: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:43.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:43.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:43.630699+0000) 2022-09-10T08:07:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:43.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:43.994528+0000) 2022-09-10T08:07:44.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:44.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:44.630960+0000) 2022-09-10T08:07:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:44.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:44.994795+0000) 2022-09-10T08:07:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:45 smithi104 ceph-mon[97044]: pgmap v6993: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:07:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:07:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:07:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:07:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:45 smithi110 ceph-mon[97633]: pgmap v6993: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:07:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:07:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:07:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:07:45.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:45.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:45.631191+0000) 2022-09-10T08:07:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:45.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:45.995080+0000) 2022-09-10T08:07:46.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:46.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:46.631399+0000) 2022-09-10T08:07:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:46.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:46.995291+0000) 2022-09-10T08:07:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:47 smithi104 ceph-mon[97044]: pgmap v6994: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:47 smithi110 ceph-mon[97633]: pgmap v6994: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:47.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:47.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:47.631577+0000) 2022-09-10T08:07:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:47.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:47.995425+0000) 2022-09-10T08:07:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:07:48.136+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:07:48.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:48.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:48.631799+0000) 2022-09-10T08:07:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:48.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:48.995647+0000) 2022-09-10T08:07:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:49 smithi104 ceph-mon[97044]: pgmap v6995: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:49 smithi110 ceph-mon[97633]: pgmap v6995: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:49.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:49.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:49.632001+0000) 2022-09-10T08:07:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:49.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:49.995931+0000) 2022-09-10T08:07:50.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:50.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:50.632256+0000) 2022-09-10T08:07:51.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:50.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:50.996151+0000) 2022-09-10T08:07:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:51 smithi104 ceph-mon[97044]: pgmap v6996: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:51 smithi110 ceph-mon[97633]: pgmap v6996: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:51.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:51.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:51.632501+0000) 2022-09-10T08:07:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:51.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:51.996365+0000) 2022-09-10T08:07:52.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:52.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:52.632708+0000) 2022-09-10T08:07:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:52.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:52.996579+0000) 2022-09-10T08:07:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:53 smithi104 ceph-mon[97044]: pgmap v6997: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:53 smithi110 ceph-mon[97633]: pgmap v6997: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:53.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:53.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:53.632941+0000) 2022-09-10T08:07:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:53.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:53.996764+0000) 2022-09-10T08:07:54.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:54.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:54.633192+0000) 2022-09-10T08:07:55.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:54.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:54.996928+0000) 2022-09-10T08:07:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:55 smithi104 ceph-mon[97044]: pgmap v6998: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:55 smithi110 ceph-mon[97633]: pgmap v6998: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:55.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:55.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:55.633434+0000) 2022-09-10T08:07:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:55.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:55.997208+0000) 2022-09-10T08:07:56.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:56.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:56.633604+0000) 2022-09-10T08:07:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:56.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:56.997384+0000) 2022-09-10T08:07:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:57 smithi104 ceph-mon[97044]: pgmap v6999: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:57 smithi110 ceph-mon[97633]: pgmap v6999: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:07:57.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:57.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:57.633832+0000) 2022-09-10T08:07:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:57.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:57.997636+0000) 2022-09-10T08:07:58.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:58.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:58.634090+0000) 2022-09-10T08:07:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:58.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:58.997852+0000) 2022-09-10T08:07:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:07:59 smithi104 ceph-mon[97044]: pgmap v7000: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:07:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:07:59 smithi110 ceph-mon[97633]: pgmap v7000: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:00.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:07:59.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:59.634369+0000) 2022-09-10T08:08:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:07:59.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:07:59.998082+0000) 2022-09-10T08:08:00.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:00.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:00.634586+0000) 2022-09-10T08:08:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:00.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:00.998244+0000) 2022-09-10T08:08:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:01 smithi104 ceph-mon[97044]: pgmap v7001: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:01 smithi110 ceph-mon[97633]: pgmap v7001: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:01.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:01.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:01.634814+0000) 2022-09-10T08:08:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:01.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:01.998472+0000) 2022-09-10T08:08:02.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:02.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:02.635028+0000) 2022-09-10T08:08:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:02.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:02.998700+0000) 2022-09-10T08:08:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:08:03.151+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:08:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:03 smithi104 ceph-mon[97044]: pgmap v7002: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:03 smithi110 ceph-mon[97633]: pgmap v7002: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:03.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:03.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:03.635234+0000) 2022-09-10T08:08:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:03.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:03.998950+0000) 2022-09-10T08:08:04.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:04.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:04.635397+0000) 2022-09-10T08:08:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:04.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:04.999177+0000) 2022-09-10T08:08:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:05 smithi104 ceph-mon[97044]: pgmap v7003: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:05 smithi110 ceph-mon[97633]: pgmap v7003: 129 pgs: 129 active+clean; 8.4 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:05.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:05.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:05.635649+0000) 2022-09-10T08:08:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:05.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:05.999348+0000) 2022-09-10T08:08:07.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:06.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:06.635906+0000) 2022-09-10T08:08:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:06.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:06.999518+0000) 2022-09-10T08:08:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:07 smithi104 ceph-mon[97044]: pgmap v7004: 129 pgs: 129 active+clean; 8.4 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:08:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:07 smithi110 ceph-mon[97633]: pgmap v7004: 129 pgs: 129 active+clean; 8.4 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:08:08.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:07.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:07.636185+0000) 2022-09-10T08:08:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:07.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:07.999698+0000) 2022-09-10T08:08:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:08 smithi104 ceph-mon[97044]: osdmap e518: 12 total, 12 up, 12 in 2022-09-10T08:08:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:08 smithi110 ceph-mon[97633]: osdmap e518: 12 total, 12 up, 12 in 2022-09-10T08:08:09.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:08.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:08.636366+0000) 2022-09-10T08:08:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:08.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:08.999927+0000) 2022-09-10T08:08:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:09 smithi104 ceph-mon[97044]: pgmap v7006: 129 pgs: 129 active+clean; 8.4 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:09 smithi110 ceph-mon[97633]: pgmap v7006: 129 pgs: 129 active+clean; 8.4 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:10.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:09.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:09.636618+0000) 2022-09-10T08:08:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:09.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:10.000210+0000) 2022-09-10T08:08:11.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:10.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:10.636862+0000) 2022-09-10T08:08:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:10.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:11.000394+0000) 2022-09-10T08:08:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:11 smithi104 ceph-mon[97044]: pgmap v7007: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:11 smithi110 ceph-mon[97633]: pgmap v7007: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:12.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:11.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:11.637177+0000) 2022-09-10T08:08:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:11.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:12.000557+0000) 2022-09-10T08:08:13.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:12.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:12.637344+0000) 2022-09-10T08:08:13.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:12.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:13.000790+0000) 2022-09-10T08:08:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:13 smithi104 ceph-mon[97044]: pgmap v7008: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:13 smithi110 ceph-mon[97633]: pgmap v7008: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:14.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:13.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:13.637526+0000) 2022-09-10T08:08:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:13.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:14.000955+0000) 2022-09-10T08:08:15.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:14.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:14.637712+0000) 2022-09-10T08:08:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:15.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:15.001143+0000) 2022-09-10T08:08:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:15 smithi104 ceph-mon[97044]: pgmap v7009: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:15 smithi104 ceph-mon[97044]: osdmap e519: 12 total, 12 up, 12 in 2022-09-10T08:08:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:15 smithi110 ceph-mon[97633]: pgmap v7009: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:08:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:15 smithi110 ceph-mon[97633]: osdmap e519: 12 total, 12 up, 12 in 2022-09-10T08:08:16.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:15.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:15.637959+0000) 2022-09-10T08:08:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:16.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:16.001396+0000) 2022-09-10T08:08:17.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:16.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:16.638196+0000) 2022-09-10T08:08:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:17.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:17.001615+0000) 2022-09-10T08:08:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:17 smithi104 ceph-mon[97044]: pgmap v7011: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:08:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:17 smithi110 ceph-mon[97633]: pgmap v7011: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:08:18.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:17.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:17.638441+0000) 2022-09-10T08:08:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:18.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:18.001863+0000) 2022-09-10T08:08:18.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:08:18.158+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:08:19.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:18.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:18.638644+0000) 2022-09-10T08:08:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:19.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:19.002047+0000) 2022-09-10T08:08:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:19 smithi104 ceph-mon[97044]: pgmap v7012: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:08:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:19 smithi110 ceph-mon[97633]: pgmap v7012: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:08:20.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:19.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:19.638895+0000) 2022-09-10T08:08:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:20.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:20.002210+0000) 2022-09-10T08:08:21.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:20.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:20.639157+0000) 2022-09-10T08:08:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:21.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:21.002424+0000) 2022-09-10T08:08:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:21 smithi104 ceph-mon[97044]: pgmap v7013: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:08:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:21 smithi110 ceph-mon[97633]: pgmap v7013: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:08:22.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:21.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:21.639393+0000) 2022-09-10T08:08:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:22.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:22.002642+0000) 2022-09-10T08:08:23.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:22.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:22.639672+0000) 2022-09-10T08:08:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:23.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:23.002842+0000) 2022-09-10T08:08:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:23 smithi104 ceph-mon[97044]: pgmap v7014: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:08:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:23 smithi110 ceph-mon[97633]: pgmap v7014: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:08:24.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:23.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:23.639910+0000) 2022-09-10T08:08:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:24.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:24.003009+0000) 2022-09-10T08:08:25.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:24.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:24.640082+0000) 2022-09-10T08:08:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:25.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:25.003249+0000) 2022-09-10T08:08:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:25 smithi104 ceph-mon[97044]: pgmap v7015: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:08:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:25 smithi110 ceph-mon[97633]: pgmap v7015: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:08:26.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:25.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:25.640317+0000) 2022-09-10T08:08:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:26.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:26.003438+0000) 2022-09-10T08:08:27.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:26.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:26.640564+0000) 2022-09-10T08:08:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:27.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:27.003649+0000) 2022-09-10T08:08:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:27 smithi104 ceph-mon[97044]: pgmap v7016: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:08:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:27 smithi110 ceph-mon[97633]: pgmap v7016: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:08:28.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:27.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:27.640804+0000) 2022-09-10T08:08:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:28.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:28.003838+0000) 2022-09-10T08:08:29.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:28.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:28.640988+0000) 2022-09-10T08:08:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:29.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:29.004021+0000) 2022-09-10T08:08:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:29 smithi104 ceph-mon[97044]: pgmap v7017: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:08:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:29 smithi110 ceph-mon[97633]: pgmap v7017: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:08:30.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:29.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:29.641198+0000) 2022-09-10T08:08:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:30.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:30.004211+0000) 2022-09-10T08:08:31.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:30.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:30.641410+0000) 2022-09-10T08:08:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:31.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:31.004420+0000) 2022-09-10T08:08:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:31 smithi104 ceph-mon[97044]: pgmap v7018: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:31 smithi110 ceph-mon[97633]: pgmap v7018: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:32.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:31.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:31.641611+0000) 2022-09-10T08:08:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:32.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:32.004614+0000) 2022-09-10T08:08:33.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:32.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:32.641796+0000) 2022-09-10T08:08:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:33.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:33.004873+0000) 2022-09-10T08:08:33.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:08:33.172+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:08:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:33 smithi104 ceph-mon[97044]: pgmap v7019: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:08:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:33 smithi110 ceph-mon[97633]: pgmap v7019: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:08:34.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:33.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:33.642052+0000) 2022-09-10T08:08:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:34.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:34.005061+0000) 2022-09-10T08:08:35.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:34.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:34.642228+0000) 2022-09-10T08:08:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:35.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:35.005224+0000) 2022-09-10T08:08:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:35 smithi104 ceph-mon[97044]: pgmap v7020: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:08:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:35 smithi110 ceph-mon[97633]: pgmap v7020: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:08:36.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:35.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:35.642466+0000) 2022-09-10T08:08:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:36.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:36.005415+0000) 2022-09-10T08:08:37.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:36.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:36.642633+0000) 2022-09-10T08:08:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:37.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:37.005642+0000) 2022-09-10T08:08:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:37 smithi104 ceph-mon[97044]: pgmap v7021: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:37 smithi110 ceph-mon[97633]: pgmap v7021: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:38.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:37.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:37.642846+0000) 2022-09-10T08:08:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:38.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:38.005862+0000) 2022-09-10T08:08:39.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:38.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:38.643014+0000) 2022-09-10T08:08:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:39.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:39.006075+0000) 2022-09-10T08:08:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:39 smithi104 ceph-mon[97044]: pgmap v7022: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:39 smithi110 ceph-mon[97633]: pgmap v7022: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:40.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:39.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:39.643217+0000) 2022-09-10T08:08:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:40.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:40.006282+0000) 2022-09-10T08:08:41.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:40.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:40.643414+0000) 2022-09-10T08:08:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:41.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:41.006459+0000) 2022-09-10T08:08:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:41 smithi104 ceph-mon[97044]: pgmap v7023: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:41 smithi110 ceph-mon[97633]: pgmap v7023: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:42.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:41.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:41.643681+0000) 2022-09-10T08:08:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:42.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:42.006672+0000) 2022-09-10T08:08:43.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:42.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:42.643926+0000) 2022-09-10T08:08:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:43.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:43.006898+0000) 2022-09-10T08:08:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:43 smithi104 ceph-mon[97044]: pgmap v7024: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:43 smithi110 ceph-mon[97633]: pgmap v7024: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:44.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:43.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:43.644198+0000) 2022-09-10T08:08:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:44.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:44.007160+0000) 2022-09-10T08:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:08:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:08:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:08:45.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:44.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:44.644328+0000) 2022-09-10T08:08:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:08:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:08:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:08:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:08:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:45.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:45.007342+0000) 2022-09-10T08:08:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:45 smithi110 ceph-mon[97633]: pgmap v7025: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:46.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:45.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:45.644481+0000) 2022-09-10T08:08:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:45 smithi104 ceph-mon[97044]: pgmap v7025: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:46.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:46.007581+0000) 2022-09-10T08:08:47.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:46.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:46.644700+0000) 2022-09-10T08:08:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:47.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:47.007760+0000) 2022-09-10T08:08:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:47 smithi110 ceph-mon[97633]: pgmap v7026: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:48.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:47.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:47.644890+0000) 2022-09-10T08:08:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:47 smithi104 ceph-mon[97044]: pgmap v7026: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:48.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:48.007959+0000) 2022-09-10T08:08:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:08:48.188+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:08:49.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:48.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:48.645058+0000) 2022-09-10T08:08:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:49.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:49.008205+0000) 2022-09-10T08:08:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:49 smithi110 ceph-mon[97633]: pgmap v7027: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:50.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:49.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:49.645240+0000) 2022-09-10T08:08:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:49 smithi104 ceph-mon[97044]: pgmap v7027: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:50.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:50.008429+0000) 2022-09-10T08:08:51.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:50.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:50.645415+0000) 2022-09-10T08:08:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:51.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:51.008624+0000) 2022-09-10T08:08:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:51 smithi110 ceph-mon[97633]: pgmap v7028: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:52.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:51.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:51.645681+0000) 2022-09-10T08:08:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:51 smithi104 ceph-mon[97044]: pgmap v7028: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:52.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:52.008827+0000) 2022-09-10T08:08:53.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:52.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:52.645863+0000) 2022-09-10T08:08:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:53.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:53.009140+0000) 2022-09-10T08:08:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:53 smithi110 ceph-mon[97633]: pgmap v7029: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:54.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:53.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:53.646155+0000) 2022-09-10T08:08:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:53 smithi104 ceph-mon[97044]: pgmap v7029: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:54.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:54.009389+0000) 2022-09-10T08:08:55.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:54.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:54.646384+0000) 2022-09-10T08:08:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:55.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:55.009647+0000) 2022-09-10T08:08:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:55 smithi110 ceph-mon[97633]: pgmap v7030: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:56.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:55.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:55.646594+0000) 2022-09-10T08:08:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:55 smithi104 ceph-mon[97044]: pgmap v7030: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:08:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:56.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:56.009921+0000) 2022-09-10T08:08:57.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:56.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:56.646719+0000) 2022-09-10T08:08:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:57.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:57.010145+0000) 2022-09-10T08:08:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:57 smithi110 ceph-mon[97633]: pgmap v7031: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:58.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:57.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:57.646960+0000) 2022-09-10T08:08:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:57 smithi104 ceph-mon[97044]: pgmap v7031: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:08:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:58.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:58.010372+0000) 2022-09-10T08:08:59.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:58.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:58.647224+0000) 2022-09-10T08:08:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:08:59.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:59.010607+0000) 2022-09-10T08:08:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:08:59 smithi110 ceph-mon[97633]: pgmap v7032: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:00.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:08:59.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:08:59.647410+0000) 2022-09-10T08:09:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:08:59 smithi104 ceph-mon[97044]: pgmap v7032: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:00.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:00.010837+0000) 2022-09-10T08:09:01.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:00.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:00.647606+0000) 2022-09-10T08:09:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:01.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:01.011071+0000) 2022-09-10T08:09:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:01 smithi110 ceph-mon[97633]: pgmap v7033: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:02.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:01.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:01.647886+0000) 2022-09-10T08:09:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:01 smithi104 ceph-mon[97044]: pgmap v7033: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:02.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:02.011343+0000) 2022-09-10T08:09:03.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:02.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:02.648167+0000) 2022-09-10T08:09:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:03.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:03.011451+0000) 2022-09-10T08:09:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:09:03.193+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:09:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:03 smithi110 ceph-mon[97633]: pgmap v7034: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:04.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:03.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:03.648411+0000) 2022-09-10T08:09:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:03 smithi104 ceph-mon[97044]: pgmap v7034: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:04.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:04.011625+0000) 2022-09-10T08:09:05.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:04.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:04.648599+0000) 2022-09-10T08:09:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:05.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:05.011810+0000) 2022-09-10T08:09:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:05 smithi110 ceph-mon[97633]: pgmap v7035: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:06.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:05.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:05.648825+0000) 2022-09-10T08:09:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:05 smithi104 ceph-mon[97044]: pgmap v7035: 129 pgs: 129 active+clean; 8.4 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:06.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:06.011980+0000) 2022-09-10T08:09:07.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:06.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:06.649132+0000) 2022-09-10T08:09:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:07.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:07.012179+0000) 2022-09-10T08:09:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:07 smithi110 ceph-mon[97633]: pgmap v7036: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:09:08.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:07.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:07.649355+0000) 2022-09-10T08:09:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:07 smithi104 ceph-mon[97044]: pgmap v7036: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:09:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:08.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:08.012403+0000) 2022-09-10T08:09:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:08 smithi110 ceph-mon[97633]: osdmap e520: 12 total, 12 up, 12 in 2022-09-10T08:09:09.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:08.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:08.649586+0000) 2022-09-10T08:09:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:08 smithi104 ceph-mon[97044]: osdmap e520: 12 total, 12 up, 12 in 2022-09-10T08:09:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:09.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:09.012566+0000) 2022-09-10T08:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:09 smithi110 ceph-mon[97633]: pgmap v7038: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:10.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:09.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:09.649853+0000) 2022-09-10T08:09:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:09 smithi104 ceph-mon[97044]: pgmap v7038: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:10.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:10.012808+0000) 2022-09-10T08:09:11.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:10.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:10.650084+0000) 2022-09-10T08:09:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:11.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:11.013089+0000) 2022-09-10T08:09:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:11 smithi110 ceph-mon[97633]: pgmap v7039: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:12.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:11.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:11.650352+0000) 2022-09-10T08:09:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:11 smithi104 ceph-mon[97044]: pgmap v7039: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:12.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:12.013367+0000) 2022-09-10T08:09:13.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:12.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:12.650509+0000) 2022-09-10T08:09:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:13.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:13.013563+0000) 2022-09-10T08:09:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:13 smithi110 ceph-mon[97633]: pgmap v7040: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:14.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:13.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:13.650727+0000) 2022-09-10T08:09:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:13 smithi104 ceph-mon[97044]: pgmap v7040: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:14.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:14.013728+0000) 2022-09-10T08:09:15.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:14.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:14.650965+0000) 2022-09-10T08:09:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:15.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:15.013919+0000) 2022-09-10T08:09:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:15 smithi110 ceph-mon[97633]: pgmap v7041: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:15 smithi110 ceph-mon[97633]: osdmap e521: 12 total, 12 up, 12 in 2022-09-10T08:09:16.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:15.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:15.651209+0000) 2022-09-10T08:09:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:15 smithi104 ceph-mon[97044]: pgmap v7041: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:09:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:15 smithi104 ceph-mon[97044]: osdmap e521: 12 total, 12 up, 12 in 2022-09-10T08:09:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:16.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:16.014135+0000) 2022-09-10T08:09:17.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:16.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:16.651385+0000) 2022-09-10T08:09:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:17.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:17.014317+0000) 2022-09-10T08:09:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:17 smithi110 ceph-mon[97633]: pgmap v7043: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:09:18.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:17.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:17.651620+0000) 2022-09-10T08:09:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:17 smithi104 ceph-mon[97044]: pgmap v7043: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:09:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:18.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:18.014487+0000) 2022-09-10T08:09:18.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:09:18.209+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:09:19.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:18.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:18.651856+0000) 2022-09-10T08:09:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:19.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:19.014670+0000) 2022-09-10T08:09:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:19 smithi110 ceph-mon[97633]: pgmap v7044: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:09:20.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:19.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:19.652068+0000) 2022-09-10T08:09:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:19 smithi104 ceph-mon[97044]: pgmap v7044: 129 pgs: 129 active+clean; 8.4 MiB data, 754 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:09:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:20.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:20.014930+0000) 2022-09-10T08:09:21.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:20.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:20.652229+0000) 2022-09-10T08:09:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:21.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:21.015157+0000) 2022-09-10T08:09:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:21 smithi110 ceph-mon[97633]: pgmap v7045: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:09:22.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:21.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:21.652432+0000) 2022-09-10T08:09:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:21 smithi104 ceph-mon[97044]: pgmap v7045: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:09:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:22.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:22.015375+0000) 2022-09-10T08:09:23.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:22.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:22.652647+0000) 2022-09-10T08:09:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:23.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:23.015558+0000) 2022-09-10T08:09:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:23 smithi110 ceph-mon[97633]: pgmap v7046: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:09:24.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:23.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:23.652889+0000) 2022-09-10T08:09:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:23 smithi104 ceph-mon[97044]: pgmap v7046: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:09:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:24.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:24.015745+0000) 2022-09-10T08:09:25.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:24.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:24.653130+0000) 2022-09-10T08:09:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:25.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:25.015980+0000) 2022-09-10T08:09:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:25 smithi110 ceph-mon[97633]: pgmap v7047: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:09:26.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:25.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:25.653356+0000) 2022-09-10T08:09:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:25 smithi104 ceph-mon[97044]: pgmap v7047: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:09:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:26.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:26.016290+0000) 2022-09-10T08:09:27.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:26.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:26.653601+0000) 2022-09-10T08:09:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:27.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:27.016492+0000) 2022-09-10T08:09:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:27 smithi110 ceph-mon[97633]: pgmap v7048: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:09:28.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:27.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:27.653864+0000) 2022-09-10T08:09:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:27 smithi104 ceph-mon[97044]: pgmap v7048: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:09:28.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:28.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:28.016661+0000) 2022-09-10T08:09:29.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:28.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:28.654001+0000) 2022-09-10T08:09:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:29.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:29.016848+0000) 2022-09-10T08:09:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:29 smithi110 ceph-mon[97633]: pgmap v7049: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:09:30.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:29.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:29.654222+0000) 2022-09-10T08:09:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:29 smithi104 ceph-mon[97044]: pgmap v7049: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:09:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:30.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:30.017049+0000) 2022-09-10T08:09:31.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:30.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:30.654484+0000) 2022-09-10T08:09:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:31.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:31.017288+0000) 2022-09-10T08:09:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:31 smithi110 ceph-mon[97633]: pgmap v7050: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:32.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:31.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:31.654778+0000) 2022-09-10T08:09:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:31 smithi104 ceph-mon[97044]: pgmap v7050: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:32.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:32.017536+0000) 2022-09-10T08:09:33.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:32.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:32.654994+0000) 2022-09-10T08:09:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:33.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:33.017721+0000) 2022-09-10T08:09:33.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:09:33.224+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:09:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:33 smithi110 ceph-mon[97633]: pgmap v7051: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:34.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:33.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:33.655162+0000) 2022-09-10T08:09:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:33 smithi104 ceph-mon[97044]: pgmap v7051: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:34.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:34.017974+0000) 2022-09-10T08:09:35.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:34.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:34.655411+0000) 2022-09-10T08:09:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:35.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:35.018186+0000) 2022-09-10T08:09:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:35 smithi110 ceph-mon[97633]: pgmap v7052: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:36.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:35.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:35.655600+0000) 2022-09-10T08:09:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:35 smithi104 ceph-mon[97044]: pgmap v7052: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:36.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:36.018404+0000) 2022-09-10T08:09:37.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:36.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:36.655834+0000) 2022-09-10T08:09:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:37.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:37.018682+0000) 2022-09-10T08:09:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:37 smithi110 ceph-mon[97633]: pgmap v7053: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:38.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:37.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:37.656069+0000) 2022-09-10T08:09:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:37 smithi104 ceph-mon[97044]: pgmap v7053: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:38.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:38.018913+0000) 2022-09-10T08:09:39.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:38.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:38.656334+0000) 2022-09-10T08:09:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:39.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:39.019065+0000) 2022-09-10T08:09:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:39 smithi110 ceph-mon[97633]: pgmap v7054: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:40.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:39.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:39.656528+0000) 2022-09-10T08:09:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:39 smithi104 ceph-mon[97044]: pgmap v7054: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:40.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:40.019303+0000) 2022-09-10T08:09:41.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:40.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:40.656726+0000) 2022-09-10T08:09:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:41.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:41.019522+0000) 2022-09-10T08:09:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:41 smithi110 ceph-mon[97633]: pgmap v7055: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:42.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:41.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:41.656957+0000) 2022-09-10T08:09:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:41 smithi104 ceph-mon[97044]: pgmap v7055: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:42.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:42.019705+0000) 2022-09-10T08:09:43.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:42.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:42.657214+0000) 2022-09-10T08:09:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:43.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:43.019947+0000) 2022-09-10T08:09:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:43 smithi110 ceph-mon[97633]: pgmap v7056: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:44.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:43.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:43.657392+0000) 2022-09-10T08:09:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:43 smithi104 ceph-mon[97044]: pgmap v7056: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:44.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:44.020208+0000) 2022-09-10T08:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:09:44.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:44.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:44.657513+0000) 2022-09-10T08:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:09:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:09:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:45.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:45.020447+0000) 2022-09-10T08:09:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:45 smithi110 ceph-mon[97633]: pgmap v7057: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:46.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:45.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:45.657681+0000) 2022-09-10T08:09:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:45 smithi104 ceph-mon[97044]: pgmap v7057: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:46.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:46.020704+0000) 2022-09-10T08:09:47.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:46.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:46.657922+0000) 2022-09-10T08:09:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:47.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:47.020911+0000) 2022-09-10T08:09:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:47 smithi110 ceph-mon[97633]: pgmap v7058: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:48.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:47.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:47.658203+0000) 2022-09-10T08:09:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:47 smithi104 ceph-mon[97044]: pgmap v7058: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:48.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:48.021141+0000) 2022-09-10T08:09:48.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:09:48.227+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:09:49.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:48.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:48.658407+0000) 2022-09-10T08:09:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:49.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:49.021325+0000) 2022-09-10T08:09:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:49 smithi110 ceph-mon[97633]: pgmap v7059: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:50.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:49.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:49.658602+0000) 2022-09-10T08:09:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:49 smithi104 ceph-mon[97044]: pgmap v7059: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:50.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:50.021555+0000) 2022-09-10T08:09:51.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:50.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:50.658771+0000) 2022-09-10T08:09:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:51.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:51.021732+0000) 2022-09-10T08:09:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:51 smithi110 ceph-mon[97633]: pgmap v7060: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:52.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:51.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:51.659073+0000) 2022-09-10T08:09:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:51 smithi104 ceph-mon[97044]: pgmap v7060: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:52.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:52.021948+0000) 2022-09-10T08:09:53.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:52.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:52.659313+0000) 2022-09-10T08:09:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:53.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:53.022199+0000) 2022-09-10T08:09:54.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:53.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:53.659551+0000) 2022-09-10T08:09:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:53 smithi104 ceph-mon[97044]: pgmap v7061: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:53 smithi110 ceph-mon[97633]: pgmap v7061: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:54.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:54.022399+0000) 2022-09-10T08:09:55.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:54.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:54.659733+0000) 2022-09-10T08:09:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:55.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:55.022572+0000) 2022-09-10T08:09:56.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:55.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:55.659966+0000) 2022-09-10T08:09:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:55 smithi104 ceph-mon[97044]: pgmap v7062: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:55 smithi110 ceph-mon[97633]: pgmap v7062: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:09:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:56.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:56.022837+0000) 2022-09-10T08:09:57.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:56.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:56.660219+0000) 2022-09-10T08:09:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:57.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:57.023068+0000) 2022-09-10T08:09:58.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:57.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:57.660466+0000) 2022-09-10T08:09:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:57 smithi104 ceph-mon[97044]: pgmap v7063: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:57 smithi110 ceph-mon[97633]: pgmap v7063: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:09:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:58.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:58.023360+0000) 2022-09-10T08:09:59.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:58.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:58.660734+0000) 2022-09-10T08:09:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:09:59.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:59.023586+0000) 2022-09-10T08:10:00.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:09:59.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:09:59.660919+0000) 2022-09-10T08:10:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:09:59 smithi104 ceph-mon[97044]: pgmap v7064: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:09:59 smithi110 ceph-mon[97633]: pgmap v7064: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:00.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:00.023809+0000) 2022-09-10T08:10:01.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:00.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:00.661166+0000) 2022-09-10T08:10:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:10:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:10:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:01.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:01.024028+0000) 2022-09-10T08:10:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:01 smithi104 ceph-mon[97044]: pgmap v7065: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:02.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:01.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:01.661438+0000) 2022-09-10T08:10:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:01 smithi110 ceph-mon[97633]: pgmap v7065: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:02.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:02.024185+0000) 2022-09-10T08:10:03.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:02.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:02.661621+0000) 2022-09-10T08:10:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:03.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:03.024360+0000) 2022-09-10T08:10:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:10:03.233+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:10:04.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:03.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:03.661833+0000) 2022-09-10T08:10:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:03 smithi104 ceph-mon[97044]: pgmap v7066: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:03 smithi110 ceph-mon[97633]: pgmap v7066: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:04.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:04.024547+0000) 2022-09-10T08:10:05.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:04.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:04.662044+0000) 2022-09-10T08:10:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:05.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:05.024762+0000) 2022-09-10T08:10:06.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:05.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:05.662205+0000) 2022-09-10T08:10:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:05 smithi104 ceph-mon[97044]: pgmap v7067: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:05 smithi110 ceph-mon[97633]: pgmap v7067: 129 pgs: 129 active+clean; 8.4 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:06.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:06.025018+0000) 2022-09-10T08:10:07.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:06.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:06.662433+0000) 2022-09-10T08:10:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:07.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:07.025231+0000) 2022-09-10T08:10:08.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:07.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:07.662643+0000) 2022-09-10T08:10:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:07 smithi104 ceph-mon[97044]: pgmap v7068: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:10:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:07 smithi110 ceph-mon[97633]: pgmap v7068: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:10:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:08.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:08.025411+0000) 2022-09-10T08:10:09.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:08.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:08.662899+0000) 2022-09-10T08:10:09.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:08 smithi104 ceph-mon[97044]: osdmap e522: 12 total, 12 up, 12 in 2022-09-10T08:10:09.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:08 smithi110 ceph-mon[97633]: osdmap e522: 12 total, 12 up, 12 in 2022-09-10T08:10:09.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:09.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:09.025596+0000) 2022-09-10T08:10:10.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:09.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:09.663174+0000) 2022-09-10T08:10:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:09 smithi104 ceph-mon[97044]: pgmap v7070: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:09 smithi110 ceph-mon[97633]: pgmap v7070: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:10.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:10.025831+0000) 2022-09-10T08:10:11.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:10.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:10.663334+0000) 2022-09-10T08:10:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:11.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:11.026004+0000) 2022-09-10T08:10:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:11 smithi104 ceph-mon[97044]: pgmap v7071: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:11 smithi110 ceph-mon[97633]: pgmap v7071: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:12.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:11.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:11.663615+0000) 2022-09-10T08:10:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:12.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:12.026269+0000) 2022-09-10T08:10:13.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:12.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:12.663866+0000) 2022-09-10T08:10:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:13.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:13.026478+0000) 2022-09-10T08:10:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:13 smithi104 ceph-mon[97044]: pgmap v7072: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:13 smithi110 ceph-mon[97633]: pgmap v7072: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:14.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:13.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:13.664172+0000) 2022-09-10T08:10:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:14.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:14.026670+0000) 2022-09-10T08:10:15.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:14.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:14.664365+0000) 2022-09-10T08:10:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:15.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:15.026844+0000) 2022-09-10T08:10:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:15 smithi104 ceph-mon[97044]: pgmap v7073: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:15 smithi104 ceph-mon[97044]: osdmap e523: 12 total, 12 up, 12 in 2022-09-10T08:10:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:15 smithi110 ceph-mon[97633]: pgmap v7073: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:10:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:15 smithi110 ceph-mon[97633]: osdmap e523: 12 total, 12 up, 12 in 2022-09-10T08:10:16.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:15.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:15.664637+0000) 2022-09-10T08:10:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:16.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:16.027080+0000) 2022-09-10T08:10:17.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:16.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:16.664816+0000) 2022-09-10T08:10:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:17.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:17.027336+0000) 2022-09-10T08:10:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:17 smithi104 ceph-mon[97044]: pgmap v7075: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:10:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:17 smithi110 ceph-mon[97633]: pgmap v7075: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T08:10:18.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:17.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:17.665038+0000) 2022-09-10T08:10:18.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:10:18.244+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:10:18.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:18.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:18.027588+0000) 2022-09-10T08:10:19.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:18.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:18.665282+0000) 2022-09-10T08:10:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:19.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:19.027780+0000) 2022-09-10T08:10:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:19 smithi104 ceph-mon[97044]: pgmap v7076: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:10:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:19 smithi110 ceph-mon[97633]: pgmap v7076: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:10:20.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:19.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:19.665423+0000) 2022-09-10T08:10:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:20.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:20.028025+0000) 2022-09-10T08:10:21.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:20.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:20.665644+0000) 2022-09-10T08:10:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:21.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:21.028276+0000) 2022-09-10T08:10:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:21 smithi104 ceph-mon[97044]: pgmap v7077: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:10:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:21 smithi110 ceph-mon[97633]: pgmap v7077: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:10:22.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:21.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:21.665936+0000) 2022-09-10T08:10:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:22.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:22.028523+0000) 2022-09-10T08:10:23.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:22.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:22.666205+0000) 2022-09-10T08:10:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:23.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:23.028787+0000) 2022-09-10T08:10:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:23 smithi104 ceph-mon[97044]: pgmap v7078: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:10:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:23 smithi110 ceph-mon[97633]: pgmap v7078: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:10:24.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:23.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:23.666436+0000) 2022-09-10T08:10:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:24.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:24.029035+0000) 2022-09-10T08:10:25.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:24.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:24.666665+0000) 2022-09-10T08:10:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:25.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:25.029233+0000) 2022-09-10T08:10:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:25 smithi104 ceph-mon[97044]: pgmap v7079: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:10:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:25 smithi110 ceph-mon[97633]: pgmap v7079: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:10:26.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:25.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:25.666940+0000) 2022-09-10T08:10:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:26.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:26.029383+0000) 2022-09-10T08:10:27.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:26.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:26.667208+0000) 2022-09-10T08:10:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:27.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:27.029550+0000) 2022-09-10T08:10:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:27 smithi104 ceph-mon[97044]: pgmap v7080: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:10:27.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:27 smithi110 ceph-mon[97633]: pgmap v7080: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:10:28.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:27.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:27.667430+0000) 2022-09-10T08:10:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:28.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:28.029733+0000) 2022-09-10T08:10:29.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:28.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:28.667659+0000) 2022-09-10T08:10:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:29.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:29.029977+0000) 2022-09-10T08:10:29.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:29 smithi104 ceph-mon[97044]: pgmap v7081: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:29 smithi110 ceph-mon[97633]: pgmap v7081: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:30.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:29.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:29.667834+0000) 2022-09-10T08:10:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:30.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:30.030224+0000) 2022-09-10T08:10:31.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:30.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:30.668081+0000) 2022-09-10T08:10:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:31.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:31.030416+0000) 2022-09-10T08:10:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:31 smithi104 ceph-mon[97044]: pgmap v7082: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:31 smithi110 ceph-mon[97633]: pgmap v7082: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:32.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:31.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:31.668354+0000) 2022-09-10T08:10:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:32.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:32.030591+0000) 2022-09-10T08:10:33.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:32.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:32.668574+0000) 2022-09-10T08:10:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:33.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:33.030784+0000) 2022-09-10T08:10:33.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:10:33.254+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:10:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:33 smithi104 ceph-mon[97044]: pgmap v7083: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:33 smithi110 ceph-mon[97633]: pgmap v7083: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:34.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:33.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:33.668825+0000) 2022-09-10T08:10:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:34.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:34.031032+0000) 2022-09-10T08:10:35.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:34.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:34.669009+0000) 2022-09-10T08:10:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:35.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:35.031203+0000) 2022-09-10T08:10:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:35 smithi104 ceph-mon[97044]: pgmap v7084: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:35 smithi110 ceph-mon[97633]: pgmap v7084: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:36.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:35.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:35.669258+0000) 2022-09-10T08:10:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:36.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:36.031355+0000) 2022-09-10T08:10:37.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:36.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:36.669442+0000) 2022-09-10T08:10:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:37.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:37.031505+0000) 2022-09-10T08:10:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:37 smithi104 ceph-mon[97044]: pgmap v7085: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:37 smithi110 ceph-mon[97633]: pgmap v7085: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:38.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:37.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:37.669700+0000) 2022-09-10T08:10:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:38.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:38.031688+0000) 2022-09-10T08:10:39.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:38.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:38.669875+0000) 2022-09-10T08:10:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:39.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:39.031815+0000) 2022-09-10T08:10:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:39 smithi104 ceph-mon[97044]: pgmap v7086: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:10:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:39 smithi110 ceph-mon[97633]: pgmap v7086: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:10:40.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:39.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:39.670047+0000) 2022-09-10T08:10:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:40.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:40.032046+0000) 2022-09-10T08:10:41.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:40.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:40.670225+0000) 2022-09-10T08:10:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:41.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:41.032273+0000) 2022-09-10T08:10:41.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:41 smithi104 ceph-mon[97044]: pgmap v7087: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:41.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:41 smithi110 ceph-mon[97633]: pgmap v7087: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:42.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:41.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:41.670405+0000) 2022-09-10T08:10:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:42.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:42.032495+0000) 2022-09-10T08:10:43.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:42.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:42.670544+0000) 2022-09-10T08:10:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:43.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:43.032682+0000) 2022-09-10T08:10:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:43 smithi104 ceph-mon[97044]: pgmap v7088: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:10:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:43 smithi110 ceph-mon[97633]: pgmap v7088: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:10:44.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:43.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:43.670786+0000) 2022-09-10T08:10:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:44.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:44.032884+0000) 2022-09-10T08:10:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:10:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:10:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:10:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:10:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:10:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:10:45.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:44.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:44.670918+0000) 2022-09-10T08:10:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:45.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:45.033079+0000) 2022-09-10T08:10:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:45 smithi104 ceph-mon[97044]: pgmap v7089: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:10:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:45 smithi110 ceph-mon[97633]: pgmap v7089: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:10:46.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:45.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:45.671174+0000) 2022-09-10T08:10:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:46.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:46.033369+0000) 2022-09-10T08:10:47.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:46.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:46.671404+0000) 2022-09-10T08:10:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:47.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:47.033572+0000) 2022-09-10T08:10:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:47 smithi104 ceph-mon[97044]: pgmap v7090: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:47 smithi110 ceph-mon[97633]: pgmap v7090: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:48.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:47.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:47.671637+0000) 2022-09-10T08:10:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:48.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:48.033805+0000) 2022-09-10T08:10:48.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:10:48.265+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:10:49.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:48.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:48.671869+0000) 2022-09-10T08:10:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:49.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:49.034032+0000) 2022-09-10T08:10:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:49 smithi104 ceph-mon[97044]: pgmap v7091: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:49 smithi110 ceph-mon[97633]: pgmap v7091: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:50.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:49.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:49.672062+0000) 2022-09-10T08:10:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:50.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:50.034222+0000) 2022-09-10T08:10:51.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:50.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:50.672219+0000) 2022-09-10T08:10:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:51.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:51.034431+0000) 2022-09-10T08:10:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:51 smithi104 ceph-mon[97044]: pgmap v7092: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:51 smithi110 ceph-mon[97633]: pgmap v7092: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:52.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:51.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:51.672455+0000) 2022-09-10T08:10:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:52.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:52.034651+0000) 2022-09-10T08:10:53.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:52.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:52.672628+0000) 2022-09-10T08:10:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:53.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:53.034913+0000) 2022-09-10T08:10:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:53 smithi110 ceph-mon[97633]: pgmap v7093: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:54.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:53.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:53.672890+0000) 2022-09-10T08:10:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:53 smithi104 ceph-mon[97044]: pgmap v7093: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:54.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:54.035188+0000) 2022-09-10T08:10:55.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:54.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:54.673067+0000) 2022-09-10T08:10:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:55.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:55.035425+0000) 2022-09-10T08:10:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:55 smithi110 ceph-mon[97633]: pgmap v7094: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:56.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:55.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:55.673299+0000) 2022-09-10T08:10:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:55 smithi104 ceph-mon[97044]: pgmap v7094: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:10:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:56.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:56.035615+0000) 2022-09-10T08:10:57.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:56.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:56.673461+0000) 2022-09-10T08:10:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:57.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:57.035869+0000) 2022-09-10T08:10:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:57 smithi110 ceph-mon[97633]: pgmap v7095: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:58.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:57.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:57.673603+0000) 2022-09-10T08:10:58.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:57 smithi104 ceph-mon[97044]: pgmap v7095: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:10:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:58.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:58.036127+0000) 2022-09-10T08:10:59.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:58.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:58.673801+0000) 2022-09-10T08:10:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:10:59.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:59.036346+0000) 2022-09-10T08:10:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:10:59 smithi110 ceph-mon[97633]: pgmap v7096: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:00.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:10:59.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:10:59.673996+0000) 2022-09-10T08:11:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:10:59 smithi104 ceph-mon[97044]: pgmap v7096: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:00.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:00.036591+0000) 2022-09-10T08:11:01.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:00.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:00.674228+0000) 2022-09-10T08:11:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:01.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:01.036770+0000) 2022-09-10T08:11:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:01 smithi110 ceph-mon[97633]: pgmap v7097: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:02.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:01.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:01.674430+0000) 2022-09-10T08:11:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:01 smithi104 ceph-mon[97044]: pgmap v7097: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:02.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:02.036985+0000) 2022-09-10T08:11:03.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:02.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:02.674639+0000) 2022-09-10T08:11:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:03.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:03.037185+0000) 2022-09-10T08:11:03.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:11:03.275+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:11:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:03 smithi110 ceph-mon[97633]: pgmap v7098: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:04.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:03.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:03.674864+0000) 2022-09-10T08:11:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:03 smithi104 ceph-mon[97044]: pgmap v7098: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:04.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:04.037439+0000) 2022-09-10T08:11:05.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:04.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:04.675021+0000) 2022-09-10T08:11:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:05.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:05.037667+0000) 2022-09-10T08:11:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:05 smithi110 ceph-mon[97633]: pgmap v7099: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:06.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:05.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:05.675186+0000) 2022-09-10T08:11:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:05 smithi104 ceph-mon[97044]: pgmap v7099: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:06.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:06.037898+0000) 2022-09-10T08:11:07.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:06.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:06.675375+0000) 2022-09-10T08:11:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:07.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:07.038191+0000) 2022-09-10T08:11:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:07 smithi110 ceph-mon[97633]: pgmap v7100: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:11:08.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:07.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:07.675555+0000) 2022-09-10T08:11:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:07 smithi104 ceph-mon[97044]: pgmap v7100: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:11:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:08.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:08.038440+0000) 2022-09-10T08:11:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:08 smithi110 ceph-mon[97633]: osdmap e524: 12 total, 12 up, 12 in 2022-09-10T08:11:09.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:08.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:08.675729+0000) 2022-09-10T08:11:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:08 smithi104 ceph-mon[97044]: osdmap e524: 12 total, 12 up, 12 in 2022-09-10T08:11:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:09.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:09.038690+0000) 2022-09-10T08:11:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:09 smithi110 ceph-mon[97633]: pgmap v7102: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:10.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:09.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:09.675980+0000) 2022-09-10T08:11:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:09 smithi104 ceph-mon[97044]: pgmap v7102: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:10.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:10.038906+0000) 2022-09-10T08:11:11.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:10.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:10.676197+0000) 2022-09-10T08:11:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:11.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:11.039176+0000) 2022-09-10T08:11:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:11 smithi110 ceph-mon[97633]: pgmap v7103: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:12.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:11.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:11.676480+0000) 2022-09-10T08:11:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:11 smithi104 ceph-mon[97044]: pgmap v7103: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:12.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:12.039482+0000) 2022-09-10T08:11:13.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:12.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:12.676720+0000) 2022-09-10T08:11:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:13.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:13.039721+0000) 2022-09-10T08:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:13 smithi110 ceph-mon[97633]: pgmap v7104: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:14.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:13.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:13.676885+0000) 2022-09-10T08:11:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:13 smithi104 ceph-mon[97044]: pgmap v7104: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:14.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:14.039965+0000) 2022-09-10T08:11:15.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:14.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:14.677176+0000) 2022-09-10T08:11:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:15.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:15.040143+0000) 2022-09-10T08:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:15 smithi110 ceph-mon[97633]: pgmap v7105: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:15 smithi110 ceph-mon[97633]: osdmap e525: 12 total, 12 up, 12 in 2022-09-10T08:11:16.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:15.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:15.677368+0000) 2022-09-10T08:11:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:15 smithi104 ceph-mon[97044]: pgmap v7105: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:11:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:15 smithi104 ceph-mon[97044]: osdmap e525: 12 total, 12 up, 12 in 2022-09-10T08:11:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:16.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:16.040341+0000) 2022-09-10T08:11:17.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:16.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:16.677531+0000) 2022-09-10T08:11:17.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:17.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:17.040532+0000) 2022-09-10T08:11:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:17 smithi110 ceph-mon[97633]: pgmap v7107: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:11:18.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:17.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:17.677731+0000) 2022-09-10T08:11:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:17 smithi104 ceph-mon[97044]: pgmap v7107: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:11:18.293 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:18.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:18.040778+0000) 2022-09-10T08:11:18.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:11:18.291+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:11:19.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:18.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:18.678002+0000) 2022-09-10T08:11:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:19.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:19.041045+0000) 2022-09-10T08:11:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:19 smithi110 ceph-mon[97633]: pgmap v7108: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:11:20.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:19.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:19.678212+0000) 2022-09-10T08:11:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:19 smithi104 ceph-mon[97044]: pgmap v7108: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:11:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:20.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:20.041324+0000) 2022-09-10T08:11:21.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:20.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:20.678432+0000) 2022-09-10T08:11:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:21.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:21.041503+0000) 2022-09-10T08:11:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:21 smithi110 ceph-mon[97633]: pgmap v7109: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:11:22.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:21.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:21.678761+0000) 2022-09-10T08:11:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:21 smithi104 ceph-mon[97044]: pgmap v7109: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:11:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:22.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:22.041733+0000) 2022-09-10T08:11:23.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:22.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:22.678953+0000) 2022-09-10T08:11:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:23.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:23.041958+0000) 2022-09-10T08:11:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:23 smithi110 ceph-mon[97633]: pgmap v7110: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:11:24.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:23.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:23.679205+0000) 2022-09-10T08:11:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:23 smithi104 ceph-mon[97044]: pgmap v7110: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:11:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:24.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:24.042191+0000) 2022-09-10T08:11:25.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:24.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:24.679390+0000) 2022-09-10T08:11:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:25.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:25.042332+0000) 2022-09-10T08:11:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:25 smithi110 ceph-mon[97633]: pgmap v7111: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:11:26.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:25.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:25.679653+0000) 2022-09-10T08:11:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:25 smithi104 ceph-mon[97044]: pgmap v7111: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:11:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:26.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:26.042571+0000) 2022-09-10T08:11:27.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:26.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:26.679879+0000) 2022-09-10T08:11:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:27.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:27.042827+0000) 2022-09-10T08:11:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:27 smithi110 ceph-mon[97633]: pgmap v7112: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:11:28.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:27.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:27.680090+0000) 2022-09-10T08:11:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:27 smithi104 ceph-mon[97044]: pgmap v7112: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:11:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:28.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:28.043083+0000) 2022-09-10T08:11:29.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:28.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:28.680300+0000) 2022-09-10T08:11:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:29.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:29.043297+0000) 2022-09-10T08:11:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:29 smithi110 ceph-mon[97633]: pgmap v7113: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:30.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:29.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:29.680420+0000) 2022-09-10T08:11:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:29 smithi104 ceph-mon[97044]: pgmap v7113: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:30.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:30.043502+0000) 2022-09-10T08:11:31.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:30.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:30.680650+0000) 2022-09-10T08:11:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:31.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:31.043743+0000) 2022-09-10T08:11:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:31 smithi110 ceph-mon[97633]: pgmap v7114: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:32.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:31.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:31.680893+0000) 2022-09-10T08:11:32.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:31 smithi104 ceph-mon[97044]: pgmap v7114: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:32.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:32.043988+0000) 2022-09-10T08:11:33.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:32.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:32.681129+0000) 2022-09-10T08:11:33.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:33.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:33.044230+0000) 2022-09-10T08:11:33.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:11:33.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:11:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:33 smithi110 ceph-mon[97633]: pgmap v7115: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:34.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:33.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:33.681375+0000) 2022-09-10T08:11:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:33 smithi104 ceph-mon[97044]: pgmap v7115: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:34.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:34.044406+0000) 2022-09-10T08:11:35.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:34.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:34.681629+0000) 2022-09-10T08:11:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:35.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:35.044570+0000) 2022-09-10T08:11:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:35 smithi110 ceph-mon[97633]: pgmap v7116: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:36.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:35.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:35.681848+0000) 2022-09-10T08:11:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:35 smithi104 ceph-mon[97044]: pgmap v7116: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:36.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:36.044847+0000) 2022-09-10T08:11:37.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:36.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:36.682041+0000) 2022-09-10T08:11:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:37.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:37.045044+0000) 2022-09-10T08:11:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:37 smithi110 ceph-mon[97633]: pgmap v7117: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:38.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:37.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:37.682286+0000) 2022-09-10T08:11:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:37 smithi104 ceph-mon[97044]: pgmap v7117: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:38.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:38.045265+0000) 2022-09-10T08:11:39.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:38.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:38.682530+0000) 2022-09-10T08:11:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:39.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:39.045512+0000) 2022-09-10T08:11:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:39 smithi110 ceph-mon[97633]: pgmap v7118: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:39 smithi104 ceph-mon[97044]: pgmap v7118: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:40.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:39.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:39.682782+0000) 2022-09-10T08:11:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:40.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:40.045723+0000) 2022-09-10T08:11:41.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:40.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:40.682996+0000) 2022-09-10T08:11:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:41.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:41.045929+0000) 2022-09-10T08:11:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:41 smithi110 ceph-mon[97633]: pgmap v7119: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:41 smithi104 ceph-mon[97044]: pgmap v7119: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:42.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:41.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:41.683344+0000) 2022-09-10T08:11:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:42.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:42.046069+0000) 2022-09-10T08:11:43.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:42.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:42.683483+0000) 2022-09-10T08:11:43.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:43.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:43.046301+0000) 2022-09-10T08:11:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:43 smithi110 ceph-mon[97633]: pgmap v7120: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:43 smithi104 ceph-mon[97044]: pgmap v7120: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:44.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:43.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:43.683731+0000) 2022-09-10T08:11:44.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:44.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:44.046551+0000) 2022-09-10T08:11:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:11:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:11:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:11:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:11:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:11:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:11:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:11:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:11:45.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:44.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:44.683873+0000) 2022-09-10T08:11:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:45.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:45.046731+0000) 2022-09-10T08:11:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:45 smithi110 ceph-mon[97633]: pgmap v7121: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:45 smithi104 ceph-mon[97044]: pgmap v7121: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:46.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:45.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:45.684090+0000) 2022-09-10T08:11:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:46.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:46.046955+0000) 2022-09-10T08:11:47.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:46.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:46.684330+0000) 2022-09-10T08:11:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:47.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:47.047228+0000) 2022-09-10T08:11:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:47 smithi110 ceph-mon[97633]: pgmap v7122: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:47 smithi104 ceph-mon[97044]: pgmap v7122: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:48.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:47.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:47.684465+0000) 2022-09-10T08:11:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:48.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:48.047470+0000) 2022-09-10T08:11:48.685 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:11:48.316+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:11:49.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:48.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:48.684643+0000) 2022-09-10T08:11:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:49.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:49.047719+0000) 2022-09-10T08:11:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:49 smithi110 ceph-mon[97633]: pgmap v7123: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:49 smithi104 ceph-mon[97044]: pgmap v7123: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:50.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:49.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:49.684807+0000) 2022-09-10T08:11:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:50.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:50.047945+0000) 2022-09-10T08:11:51.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:50.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:50.685042+0000) 2022-09-10T08:11:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:51.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:51.048166+0000) 2022-09-10T08:11:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:51 smithi110 ceph-mon[97633]: pgmap v7124: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:51 smithi104 ceph-mon[97044]: pgmap v7124: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:52.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:51.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:51.685298+0000) 2022-09-10T08:11:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:52.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:52.048307+0000) 2022-09-10T08:11:53.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:52.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:52.685567+0000) 2022-09-10T08:11:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:53.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:53.048538+0000) 2022-09-10T08:11:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:53 smithi110 ceph-mon[97633]: pgmap v7125: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:53 smithi104 ceph-mon[97044]: pgmap v7125: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:54.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:53.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:53.685779+0000) 2022-09-10T08:11:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:54.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:54.048807+0000) 2022-09-10T08:11:55.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:54.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:54.686042+0000) 2022-09-10T08:11:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:55.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:55.049077+0000) 2022-09-10T08:11:55.921 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:55 smithi110 ceph-mon[97633]: pgmap v7126: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:55 smithi104 ceph-mon[97044]: pgmap v7126: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:11:56.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:55.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:55.686258+0000) 2022-09-10T08:11:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:56.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:56.049372+0000) 2022-09-10T08:11:57.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:56.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:56.686520+0000) 2022-09-10T08:11:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:57.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:57.049574+0000) 2022-09-10T08:11:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:57 smithi110 ceph-mon[97633]: pgmap v7127: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:57 smithi104 ceph-mon[97044]: pgmap v7127: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:11:58.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:57.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:57.686713+0000) 2022-09-10T08:11:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:58.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:58.049828+0000) 2022-09-10T08:11:59.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:58.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:58.686938+0000) 2022-09-10T08:11:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:11:59.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:59.050005+0000) 2022-09-10T08:11:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:11:59 smithi110 ceph-mon[97633]: pgmap v7128: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:11:59 smithi104 ceph-mon[97044]: pgmap v7128: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:00.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:11:59.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:11:59.687248+0000) 2022-09-10T08:12:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:00.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:00.050225+0000) 2022-09-10T08:12:01.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:00.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:00.687426+0000) 2022-09-10T08:12:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:01.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:01.050440+0000) 2022-09-10T08:12:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:01 smithi110 ceph-mon[97633]: pgmap v7129: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:01 smithi104 ceph-mon[97044]: pgmap v7129: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:02.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:01.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:01.687565+0000) 2022-09-10T08:12:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:02.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:02.050667+0000) 2022-09-10T08:12:03.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:02.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:02.687790+0000) 2022-09-10T08:12:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:03.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:03.050864+0000) 2022-09-10T08:12:03.688 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:12:03.331+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:12:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:03 smithi104 ceph-mon[97044]: pgmap v7130: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:04.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:03.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:03.687994+0000) 2022-09-10T08:12:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:03 smithi110 ceph-mon[97633]: pgmap v7130: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:04.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:04.051058+0000) 2022-09-10T08:12:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:04.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:04.688171+0000) 2022-09-10T08:12:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:05.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:05.051246+0000) 2022-09-10T08:12:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:05 smithi104 ceph-mon[97044]: pgmap v7131: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:06.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:05.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:05.688496+0000) 2022-09-10T08:12:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:05 smithi110 ceph-mon[97633]: pgmap v7131: 129 pgs: 129 active+clean; 8.4 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:06.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:06.051510+0000) 2022-09-10T08:12:07.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:06.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:06.688784+0000) 2022-09-10T08:12:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:07.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:07.051732+0000) 2022-09-10T08:12:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:07 smithi104 ceph-mon[97044]: pgmap v7132: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:12:08.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:07.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:07.689046+0000) 2022-09-10T08:12:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:07 smithi110 ceph-mon[97633]: pgmap v7132: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:12:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:08.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:08.051945+0000) 2022-09-10T08:12:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:08 smithi104 ceph-mon[97044]: osdmap e526: 12 total, 12 up, 12 in 2022-09-10T08:12:09.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:08.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:08.689243+0000) 2022-09-10T08:12:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:08 smithi110 ceph-mon[97633]: osdmap e526: 12 total, 12 up, 12 in 2022-09-10T08:12:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:09.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:09.052255+0000) 2022-09-10T08:12:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:09 smithi104 ceph-mon[97044]: pgmap v7134: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:10.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:09.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:09.689419+0000) 2022-09-10T08:12:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:09 smithi110 ceph-mon[97633]: pgmap v7134: 129 pgs: 129 active+clean; 8.4 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:10.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:10.052527+0000) 2022-09-10T08:12:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:10.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:10.689664+0000) 2022-09-10T08:12:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:11.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:11.052793+0000) 2022-09-10T08:12:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:11 smithi104 ceph-mon[97044]: pgmap v7135: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:11.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:11.689840+0000) 2022-09-10T08:12:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:11 smithi110 ceph-mon[97633]: pgmap v7135: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:12.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:12.053021+0000) 2022-09-10T08:12:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:12.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:12.690005+0000) 2022-09-10T08:12:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:13.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:13.053248+0000) 2022-09-10T08:12:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:13 smithi104 ceph-mon[97044]: pgmap v7136: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:13.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:13.690235+0000) 2022-09-10T08:12:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:13 smithi110 ceph-mon[97633]: pgmap v7136: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:14.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:14.053446+0000) 2022-09-10T08:12:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:14.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:14.690444+0000) 2022-09-10T08:12:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:15.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:15.053597+0000) 2022-09-10T08:12:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:15 smithi104 ceph-mon[97044]: pgmap v7137: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:15 smithi104 ceph-mon[97044]: osdmap e527: 12 total, 12 up, 12 in 2022-09-10T08:12:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:15.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:15.690626+0000) 2022-09-10T08:12:16.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:15 smithi110 ceph-mon[97633]: pgmap v7137: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:12:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:15 smithi110 ceph-mon[97633]: osdmap e527: 12 total, 12 up, 12 in 2022-09-10T08:12:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:16.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:16.053898+0000) 2022-09-10T08:12:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:16.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:16.690870+0000) 2022-09-10T08:12:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:17.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:17.054035+0000) 2022-09-10T08:12:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:17 smithi104 ceph-mon[97044]: pgmap v7139: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T08:12:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:17.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:17.691068+0000) 2022-09-10T08:12:18.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:17 smithi110 ceph-mon[97633]: pgmap v7139: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T08:12:18.338 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:18.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:18.054166+0000) 2022-09-10T08:12:18.692 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:12:18.335+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:12:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:18.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:18.691336+0000) 2022-09-10T08:12:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:19.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:19.054304+0000) 2022-09-10T08:12:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:19 smithi104 ceph-mon[97044]: pgmap v7140: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:12:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:19.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:19.691629+0000) 2022-09-10T08:12:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:19 smithi110 ceph-mon[97633]: pgmap v7140: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:12:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:20.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:20.054440+0000) 2022-09-10T08:12:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:20.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:20.691854+0000) 2022-09-10T08:12:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:21.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:21.054607+0000) 2022-09-10T08:12:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:21 smithi104 ceph-mon[97044]: pgmap v7141: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:12:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:21.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:21.692126+0000) 2022-09-10T08:12:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:21 smithi110 ceph-mon[97633]: pgmap v7141: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:12:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:22.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:22.054829+0000) 2022-09-10T08:12:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:22.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:22.692287+0000) 2022-09-10T08:12:23.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:23.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:23.054989+0000) 2022-09-10T08:12:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:23 smithi104 ceph-mon[97044]: pgmap v7142: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:12:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:23.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:23.692483+0000) 2022-09-10T08:12:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:23 smithi110 ceph-mon[97633]: pgmap v7142: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:12:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:24.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:24.055203+0000) 2022-09-10T08:12:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:24.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:24.692710+0000) 2022-09-10T08:12:25.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:25.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:25.055331+0000) 2022-09-10T08:12:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:25 smithi104 ceph-mon[97044]: pgmap v7143: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:12:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:25.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:25.692984+0000) 2022-09-10T08:12:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:25 smithi110 ceph-mon[97633]: pgmap v7143: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:12:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:26.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:26.055510+0000) 2022-09-10T08:12:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:26.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:26.693226+0000) 2022-09-10T08:12:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:27.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:27.055692+0000) 2022-09-10T08:12:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:27 smithi104 ceph-mon[97044]: pgmap v7144: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:12:28.062 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:27.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:27.693442+0000) 2022-09-10T08:12:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:27 smithi110 ceph-mon[97633]: pgmap v7144: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:12:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:28.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:28.055911+0000) 2022-09-10T08:12:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:28.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:28.693606+0000) 2022-09-10T08:12:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:29.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:29.056164+0000) 2022-09-10T08:12:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:29 smithi104 ceph-mon[97044]: pgmap v7145: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:29.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:29.693885+0000) 2022-09-10T08:12:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:29 smithi110 ceph-mon[97633]: pgmap v7145: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:30.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:30.056296+0000) 2022-09-10T08:12:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:30.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:30.694153+0000) 2022-09-10T08:12:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:31.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:31.056482+0000) 2022-09-10T08:12:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:31 smithi104 ceph-mon[97044]: pgmap v7146: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:31.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:31.694384+0000) 2022-09-10T08:12:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:31 smithi110 ceph-mon[97633]: pgmap v7146: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:32.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:32.056684+0000) 2022-09-10T08:12:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:32.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:32.694662+0000) 2022-09-10T08:12:33.340 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:33.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:33.056901+0000) 2022-09-10T08:12:33.695 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:12:33.338+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:12:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:33 smithi104 ceph-mon[97044]: pgmap v7147: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:33.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:33.694853+0000) 2022-09-10T08:12:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:33 smithi110 ceph-mon[97633]: pgmap v7147: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:34.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:34.057089+0000) 2022-09-10T08:12:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:34 smithi104 ceph-mon[97044]: pgmap v7148: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:34.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:34.694979+0000) 2022-09-10T08:12:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:34 smithi110 ceph-mon[97633]: pgmap v7148: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:35.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:35.057194+0000) 2022-09-10T08:12:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:35.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:35.695132+0000) 2022-09-10T08:12:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:36.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:36.057327+0000) 2022-09-10T08:12:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:36.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:36.695335+0000) 2022-09-10T08:12:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:37.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:37.057496+0000) 2022-09-10T08:12:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:37 smithi104 ceph-mon[97044]: pgmap v7149: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:37 smithi110 ceph-mon[97633]: pgmap v7149: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:37.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:37.695608+0000) 2022-09-10T08:12:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:38.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:38.057683+0000) 2022-09-10T08:12:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:38.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:38.695856+0000) 2022-09-10T08:12:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:39.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:39.057893+0000) 2022-09-10T08:12:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:39 smithi104 ceph-mon[97044]: pgmap v7150: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:39 smithi110 ceph-mon[97633]: pgmap v7150: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:39.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:39.696149+0000) 2022-09-10T08:12:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:40.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:40.058140+0000) 2022-09-10T08:12:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:40.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:40.696293+0000) 2022-09-10T08:12:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:41.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:41.058327+0000) 2022-09-10T08:12:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:41 smithi104 ceph-mon[97044]: pgmap v7151: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:41 smithi110 ceph-mon[97633]: pgmap v7151: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:41.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:41.696515+0000) 2022-09-10T08:12:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:42.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:42.058453+0000) 2022-09-10T08:12:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:42.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:42.696717+0000) 2022-09-10T08:12:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:43.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:43.058612+0000) 2022-09-10T08:12:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:43 smithi104 ceph-mon[97044]: pgmap v7152: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:43 smithi110 ceph-mon[97633]: pgmap v7152: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:43.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:43.696955+0000) 2022-09-10T08:12:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:44.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:44.058835+0000) 2022-09-10T08:12:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:12:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:12:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:12:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:12:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:12:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:12:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:12:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:44.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:44.697178+0000) 2022-09-10T08:12:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:45.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:45.059026+0000) 2022-09-10T08:12:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:45 smithi104 ceph-mon[97044]: pgmap v7153: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:45 smithi110 ceph-mon[97633]: pgmap v7153: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:45.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:45.697357+0000) 2022-09-10T08:12:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:46.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:46.059192+0000) 2022-09-10T08:12:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:46.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:46.697622+0000) 2022-09-10T08:12:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:47.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:47.059313+0000) 2022-09-10T08:12:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:47 smithi104 ceph-mon[97044]: pgmap v7154: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:47.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:47 smithi110 ceph-mon[97633]: pgmap v7154: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:47.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:47.697865+0000) 2022-09-10T08:12:48.352 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:48.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:48.059481+0000) 2022-09-10T08:12:48.698 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:12:48.351+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:12:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:48.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:48.698023+0000) 2022-09-10T08:12:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:49.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:49.059655+0000) 2022-09-10T08:12:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:49 smithi104 ceph-mon[97044]: pgmap v7155: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:49 smithi110 ceph-mon[97633]: pgmap v7155: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:49.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:49.698199+0000) 2022-09-10T08:12:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:50.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:50.059878+0000) 2022-09-10T08:12:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:50.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:50.698455+0000) 2022-09-10T08:12:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:51.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:51.060144+0000) 2022-09-10T08:12:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:51 smithi110 ceph-mon[97633]: pgmap v7156: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:51 smithi104 ceph-mon[97044]: pgmap v7156: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:51.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:51.698657+0000) 2022-09-10T08:12:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:52.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:52.060356+0000) 2022-09-10T08:12:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:52.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:52.698888+0000) 2022-09-10T08:12:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:53.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:53.060547+0000) 2022-09-10T08:12:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:53 smithi110 ceph-mon[97633]: pgmap v7157: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:53 smithi104 ceph-mon[97044]: pgmap v7157: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:53.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:53.699120+0000) 2022-09-10T08:12:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:54.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:54.060739+0000) 2022-09-10T08:12:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:54.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:54.699344+0000) 2022-09-10T08:12:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:55.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:55.060953+0000) 2022-09-10T08:12:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:55 smithi110 ceph-mon[97633]: pgmap v7158: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:55 smithi104 ceph-mon[97044]: pgmap v7158: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:12:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:55.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:55.699515+0000) 2022-09-10T08:12:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:56.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:56.061196+0000) 2022-09-10T08:12:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:56.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:56.699714+0000) 2022-09-10T08:12:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:57.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:57.061370+0000) 2022-09-10T08:12:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:57 smithi110 ceph-mon[97633]: pgmap v7159: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:57 smithi104 ceph-mon[97044]: pgmap v7159: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:12:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:57.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:57.699943+0000) 2022-09-10T08:12:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:58.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:58.061572+0000) 2022-09-10T08:12:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:58.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:58.700192+0000) 2022-09-10T08:12:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:12:59.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:59.061801+0000) 2022-09-10T08:12:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:12:59 smithi110 ceph-mon[97633]: pgmap v7160: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:12:59 smithi104 ceph-mon[97044]: pgmap v7160: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:12:59.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:12:59.700413+0000) 2022-09-10T08:13:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:00.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:00.062054+0000) 2022-09-10T08:13:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:00.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:00.700571+0000) 2022-09-10T08:13:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:01.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:01.062241+0000) 2022-09-10T08:13:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:01 smithi110 ceph-mon[97633]: pgmap v7161: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:01 smithi104 ceph-mon[97044]: pgmap v7161: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:01.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:01.700738+0000) 2022-09-10T08:13:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:02.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:02.062368+0000) 2022-09-10T08:13:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:02.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:02.700942+0000) 2022-09-10T08:13:03.356 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:03.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:03.062567+0000) 2022-09-10T08:13:03.710 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:13:03.355+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:13:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:03 smithi110 ceph-mon[97633]: pgmap v7162: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:03 smithi104 ceph-mon[97044]: pgmap v7162: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:03.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:03.701157+0000) 2022-09-10T08:13:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:04.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:04.062814+0000) 2022-09-10T08:13:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:04.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:04.701344+0000) 2022-09-10T08:13:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:05.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:05.063061+0000) 2022-09-10T08:13:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:05 smithi110 ceph-mon[97633]: pgmap v7163: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:05 smithi104 ceph-mon[97044]: pgmap v7163: 129 pgs: 129 active+clean; 8.4 MiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:05.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:05.701573+0000) 2022-09-10T08:13:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:06.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:06.063301+0000) 2022-09-10T08:13:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:06.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:06.701801+0000) 2022-09-10T08:13:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:07.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:07.063546+0000) 2022-09-10T08:13:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:07 smithi110 ceph-mon[97633]: pgmap v7164: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:13:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:07 smithi104 ceph-mon[97044]: pgmap v7164: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:13:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:07.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:07.702033+0000) 2022-09-10T08:13:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:08.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:08.063718+0000) 2022-09-10T08:13:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:08 smithi110 ceph-mon[97633]: osdmap e528: 12 total, 12 up, 12 in 2022-09-10T08:13:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:08 smithi104 ceph-mon[97044]: osdmap e528: 12 total, 12 up, 12 in 2022-09-10T08:13:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:08.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:08.702250+0000) 2022-09-10T08:13:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:09.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:09.063962+0000) 2022-09-10T08:13:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:09 smithi110 ceph-mon[97633]: pgmap v7166: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:09 smithi104 ceph-mon[97044]: pgmap v7166: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:09.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:09.702510+0000) 2022-09-10T08:13:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:10.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:10.064206+0000) 2022-09-10T08:13:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:10.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:10.702717+0000) 2022-09-10T08:13:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:11.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:11.064429+0000) 2022-09-10T08:13:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:11 smithi110 ceph-mon[97633]: pgmap v7167: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:11 smithi104 ceph-mon[97044]: pgmap v7167: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:11.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:11.702931+0000) 2022-09-10T08:13:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:12.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:12.064594+0000) 2022-09-10T08:13:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:12.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:12.703197+0000) 2022-09-10T08:13:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:13.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:13.064771+0000) 2022-09-10T08:13:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:13 smithi110 ceph-mon[97633]: pgmap v7168: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:13 smithi104 ceph-mon[97044]: pgmap v7168: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:13.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:13.703375+0000) 2022-09-10T08:13:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:14.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:14.064992+0000) 2022-09-10T08:13:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:14.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:14.703520+0000) 2022-09-10T08:13:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:15.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:15.065235+0000) 2022-09-10T08:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:15 smithi110 ceph-mon[97633]: pgmap v7169: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:15 smithi110 ceph-mon[97633]: osdmap e529: 12 total, 12 up, 12 in 2022-09-10T08:13:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:15 smithi104 ceph-mon[97044]: pgmap v7169: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:13:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:15 smithi104 ceph-mon[97044]: osdmap e529: 12 total, 12 up, 12 in 2022-09-10T08:13:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:15.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:15.703693+0000) 2022-09-10T08:13:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:16.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:16.065541+0000) 2022-09-10T08:13:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:16.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:16.703921+0000) 2022-09-10T08:13:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:17.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:17.065784+0000) 2022-09-10T08:13:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:17 smithi110 ceph-mon[97633]: pgmap v7171: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:13:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:17 smithi104 ceph-mon[97044]: pgmap v7171: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:13:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:17.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:17.704193+0000) 2022-09-10T08:13:18.371 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:18.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:18.066031+0000) 2022-09-10T08:13:18.705 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:13:18.370+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:13:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:18.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:18.704412+0000) 2022-09-10T08:13:19.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:19.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:19.066218+0000) 2022-09-10T08:13:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:19 smithi110 ceph-mon[97633]: pgmap v7172: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:13:20.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:19 smithi104 ceph-mon[97044]: pgmap v7172: 129 pgs: 129 active+clean; 8.4 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:13:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:19.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:19.704634+0000) 2022-09-10T08:13:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:20.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:20.066379+0000) 2022-09-10T08:13:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:20.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:20.704847+0000) 2022-09-10T08:13:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:21.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:21.066565+0000) 2022-09-10T08:13:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:21 smithi110 ceph-mon[97633]: pgmap v7173: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:13:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:21 smithi104 ceph-mon[97044]: pgmap v7173: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:13:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:21.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:21.705085+0000) 2022-09-10T08:13:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:22.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:22.066744+0000) 2022-09-10T08:13:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:22.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:22.705351+0000) 2022-09-10T08:13:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:23.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:23.066919+0000) 2022-09-10T08:13:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:23 smithi110 ceph-mon[97633]: pgmap v7174: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:13:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:23 smithi104 ceph-mon[97044]: pgmap v7174: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:13:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:23.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:23.705526+0000) 2022-09-10T08:13:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:24.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:24.067125+0000) 2022-09-10T08:13:25.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:24.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:24.705798+0000) 2022-09-10T08:13:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:25.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:25.067343+0000) 2022-09-10T08:13:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:25 smithi110 ceph-mon[97633]: pgmap v7175: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:13:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:25 smithi104 ceph-mon[97044]: pgmap v7175: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:13:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:25.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:25.706043+0000) 2022-09-10T08:13:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:26.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:26.067575+0000) 2022-09-10T08:13:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:26.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:26.706250+0000) 2022-09-10T08:13:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:27.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:27.067726+0000) 2022-09-10T08:13:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:27 smithi110 ceph-mon[97633]: pgmap v7176: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:13:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:27 smithi104 ceph-mon[97044]: pgmap v7176: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:13:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:27.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:27.706431+0000) 2022-09-10T08:13:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:28.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:28.067993+0000) 2022-09-10T08:13:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:28.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:28.706584+0000) 2022-09-10T08:13:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:29.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:29.068221+0000) 2022-09-10T08:13:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:29 smithi110 ceph-mon[97633]: pgmap v7177: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:29 smithi104 ceph-mon[97044]: pgmap v7177: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:29.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:29.706727+0000) 2022-09-10T08:13:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:30.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:30.068459+0000) 2022-09-10T08:13:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:30.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:30.706875+0000) 2022-09-10T08:13:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:31.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:31.068686+0000) 2022-09-10T08:13:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:31 smithi110 ceph-mon[97633]: pgmap v7178: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:31 smithi104 ceph-mon[97044]: pgmap v7178: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:31.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:31.707052+0000) 2022-09-10T08:13:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:32.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:32.068940+0000) 2022-09-10T08:13:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:32.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:32.707247+0000) 2022-09-10T08:13:33.373 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:33.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:33.069160+0000) 2022-09-10T08:13:33.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:13:33.371+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:13:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:33 smithi110 ceph-mon[97633]: pgmap v7179: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:33 smithi104 ceph-mon[97044]: pgmap v7179: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:33.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:33.707396+0000) 2022-09-10T08:13:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:34.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:34.069353+0000) 2022-09-10T08:13:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:34.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:34.707580+0000) 2022-09-10T08:13:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:35.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:35.069546+0000) 2022-09-10T08:13:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:35 smithi110 ceph-mon[97633]: pgmap v7180: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:35 smithi104 ceph-mon[97044]: pgmap v7180: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:35.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:35.707707+0000) 2022-09-10T08:13:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:36.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:36.069829+0000) 2022-09-10T08:13:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:36.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:36.707855+0000) 2022-09-10T08:13:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:37.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:37.070048+0000) 2022-09-10T08:13:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:37 smithi110 ceph-mon[97633]: pgmap v7181: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:37 smithi104 ceph-mon[97044]: pgmap v7181: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:37.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:37.707976+0000) 2022-09-10T08:13:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:38.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:38.070268+0000) 2022-09-10T08:13:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:38.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:38.708129+0000) 2022-09-10T08:13:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:39.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:39.070412+0000) 2022-09-10T08:13:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:39 smithi110 ceph-mon[97633]: pgmap v7182: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:39 smithi104 ceph-mon[97044]: pgmap v7182: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:39.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:39.708256+0000) 2022-09-10T08:13:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:40.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:40.070615+0000) 2022-09-10T08:13:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:40.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:40.708410+0000) 2022-09-10T08:13:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:41.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:41.070866+0000) 2022-09-10T08:13:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:41 smithi110 ceph-mon[97633]: pgmap v7183: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:41 smithi104 ceph-mon[97044]: pgmap v7183: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:41.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:41.708552+0000) 2022-09-10T08:13:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:42.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:42.071070+0000) 2022-09-10T08:13:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:42.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:42.708719+0000) 2022-09-10T08:13:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:43.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:43.071300+0000) 2022-09-10T08:13:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:43 smithi110 ceph-mon[97633]: pgmap v7184: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:43 smithi104 ceph-mon[97044]: pgmap v7184: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:43.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:43.708867+0000) 2022-09-10T08:13:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:44.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:44.071449+0000) 2022-09-10T08:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:13:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:13:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:13:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:13:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:44.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:44.709046+0000) 2022-09-10T08:13:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:45.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:45.071657+0000) 2022-09-10T08:13:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:45 smithi110 ceph-mon[97633]: pgmap v7185: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:45 smithi104 ceph-mon[97044]: pgmap v7185: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:45.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:45.709242+0000) 2022-09-10T08:13:46.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:46.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:46.071892+0000) 2022-09-10T08:13:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:46.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:46.709391+0000) 2022-09-10T08:13:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:47.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:47.072134+0000) 2022-09-10T08:13:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:47 smithi110 ceph-mon[97633]: pgmap v7186: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:47 smithi104 ceph-mon[97044]: pgmap v7186: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:47.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:47.709530+0000) 2022-09-10T08:13:48.381 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:48.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:48.072364+0000) 2022-09-10T08:13:48.710 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:13:48.380+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:13:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:48.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:48.709649+0000) 2022-09-10T08:13:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:49.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:49.072523+0000) 2022-09-10T08:13:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:49 smithi110 ceph-mon[97633]: pgmap v7187: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:49 smithi104 ceph-mon[97044]: pgmap v7187: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:49.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:49.709796+0000) 2022-09-10T08:13:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:50.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:50.072681+0000) 2022-09-10T08:13:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:50.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:50.709930+0000) 2022-09-10T08:13:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:51.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:51.072952+0000) 2022-09-10T08:13:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:51 smithi110 ceph-mon[97633]: pgmap v7188: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:51 smithi104 ceph-mon[97044]: pgmap v7188: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:51.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:51.710108+0000) 2022-09-10T08:13:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:52.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:52.073196+0000) 2022-09-10T08:13:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:52.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:52.710278+0000) 2022-09-10T08:13:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:53.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:53.073424+0000) 2022-09-10T08:13:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:53 smithi110 ceph-mon[97633]: pgmap v7189: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:53 smithi104 ceph-mon[97044]: pgmap v7189: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:53.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:53.710422+0000) 2022-09-10T08:13:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:54.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:54.073608+0000) 2022-09-10T08:13:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:54.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:54.710554+0000) 2022-09-10T08:13:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:55.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:55.073844+0000) 2022-09-10T08:13:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:55 smithi110 ceph-mon[97633]: pgmap v7190: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:55 smithi104 ceph-mon[97044]: pgmap v7190: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:13:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:55.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:55.710722+0000) 2022-09-10T08:13:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:56.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:56.074107+0000) 2022-09-10T08:13:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:56.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:56.710928+0000) 2022-09-10T08:13:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:57.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:57.074269+0000) 2022-09-10T08:13:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:57 smithi110 ceph-mon[97633]: pgmap v7191: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:57 smithi104 ceph-mon[97044]: pgmap v7191: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:13:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:57.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:57.711077+0000) 2022-09-10T08:13:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:58.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:58.074413+0000) 2022-09-10T08:13:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:58.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:58.711224+0000) 2022-09-10T08:13:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:13:59.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:59.074544+0000) 2022-09-10T08:14:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:13:59 smithi104 ceph-mon[97044]: pgmap v7192: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:13:59.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:13:59.711367+0000) 2022-09-10T08:14:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:13:59 smithi110 ceph-mon[97633]: pgmap v7192: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:00.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:00.074779+0000) 2022-09-10T08:14:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:00.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:00.711513+0000) 2022-09-10T08:14:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:01.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:01.075018+0000) 2022-09-10T08:14:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:01 smithi104 ceph-mon[97044]: pgmap v7193: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:01.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:01.711656+0000) 2022-09-10T08:14:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:01 smithi110 ceph-mon[97633]: pgmap v7193: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:02.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:02.075264+0000) 2022-09-10T08:14:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:02.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:02.711891+0000) 2022-09-10T08:14:03.394 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:03.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:03.075470+0000) 2022-09-10T08:14:03.712 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:14:03.392+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:14:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:03 smithi104 ceph-mon[97044]: pgmap v7194: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:03.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:03.712060+0000) 2022-09-10T08:14:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:03 smithi110 ceph-mon[97633]: pgmap v7194: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:04.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:04.075657+0000) 2022-09-10T08:14:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:04.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:04.712261+0000) 2022-09-10T08:14:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:05.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:05.075859+0000) 2022-09-10T08:14:06.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:05.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:05.712450+0000) 2022-09-10T08:14:06.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:05 smithi104 ceph-mon[97044]: pgmap v7195: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:05 smithi110 ceph-mon[97633]: pgmap v7195: 129 pgs: 129 active+clean; 8.4 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:06.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:06.076146+0000) 2022-09-10T08:14:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:06.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:06.712617+0000) 2022-09-10T08:14:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:07.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:07.076307+0000) 2022-09-10T08:14:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:07 smithi104 ceph-mon[97044]: pgmap v7196: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:14:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:07.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:07.712778+0000) 2022-09-10T08:14:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:07 smithi110 ceph-mon[97633]: pgmap v7196: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:14:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:08.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:08.076493+0000) 2022-09-10T08:14:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:08 smithi104 ceph-mon[97044]: osdmap e530: 12 total, 12 up, 12 in 2022-09-10T08:14:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:08.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:08.712933+0000) 2022-09-10T08:14:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:08 smithi110 ceph-mon[97633]: osdmap e530: 12 total, 12 up, 12 in 2022-09-10T08:14:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:09.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:09.076751+0000) 2022-09-10T08:14:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:09 smithi104 ceph-mon[97044]: pgmap v7198: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:09.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:09.713121+0000) 2022-09-10T08:14:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:09 smithi110 ceph-mon[97633]: pgmap v7198: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:10.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:10.076986+0000) 2022-09-10T08:14:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:10.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:10.713294+0000) 2022-09-10T08:14:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:11.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:11.077230+0000) 2022-09-10T08:14:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:11 smithi104 ceph-mon[97044]: pgmap v7199: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:11.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:11.713431+0000) 2022-09-10T08:14:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:11 smithi110 ceph-mon[97633]: pgmap v7199: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:12.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:12.077394+0000) 2022-09-10T08:14:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:12.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:12.713564+0000) 2022-09-10T08:14:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:13.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:13.077653+0000) 2022-09-10T08:14:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:13 smithi104 ceph-mon[97044]: pgmap v7200: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:13.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:13.713742+0000) 2022-09-10T08:14:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:13 smithi110 ceph-mon[97633]: pgmap v7200: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:14.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:14.077846+0000) 2022-09-10T08:14:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:14.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:14.713907+0000) 2022-09-10T08:14:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:15.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:15.078040+0000) 2022-09-10T08:14:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:15 smithi104 ceph-mon[97044]: pgmap v7201: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:15 smithi104 ceph-mon[97044]: osdmap e531: 12 total, 12 up, 12 in 2022-09-10T08:14:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:15.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:15.714171+0000) 2022-09-10T08:14:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:15 smithi110 ceph-mon[97633]: pgmap v7201: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:14:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:15 smithi110 ceph-mon[97633]: osdmap e531: 12 total, 12 up, 12 in 2022-09-10T08:14:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:16.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:16.078295+0000) 2022-09-10T08:14:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:16.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:16.714321+0000) 2022-09-10T08:14:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:17.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:17.078473+0000) 2022-09-10T08:14:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:17 smithi104 ceph-mon[97044]: pgmap v7203: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:14:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:17.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:17.714548+0000) 2022-09-10T08:14:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:17 smithi110 ceph-mon[97633]: pgmap v7203: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:14:18.405 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:18.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:18.078703+0000) 2022-09-10T08:14:18.715 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:14:18.404+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:14:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:18.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:18.714759+0000) 2022-09-10T08:14:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:19.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:19.078923+0000) 2022-09-10T08:14:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:19 smithi104 ceph-mon[97044]: pgmap v7204: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:14:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:19.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:19.714987+0000) 2022-09-10T08:14:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:19 smithi110 ceph-mon[97633]: pgmap v7204: 129 pgs: 129 active+clean; 8.4 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:14:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:20.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:20.079192+0000) 2022-09-10T08:14:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:20.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:20.715236+0000) 2022-09-10T08:14:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:21.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:21.079405+0000) 2022-09-10T08:14:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:21 smithi104 ceph-mon[97044]: pgmap v7205: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:14:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:21.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:21.715474+0000) 2022-09-10T08:14:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:21 smithi110 ceph-mon[97633]: pgmap v7205: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:14:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:22.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:22.079620+0000) 2022-09-10T08:14:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:22.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:22.715695+0000) 2022-09-10T08:14:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:23.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:23.079854+0000) 2022-09-10T08:14:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:23 smithi104 ceph-mon[97044]: pgmap v7206: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:14:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:23.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:23.715873+0000) 2022-09-10T08:14:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:23 smithi110 ceph-mon[97633]: pgmap v7206: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:14:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:24.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:24.080079+0000) 2022-09-10T08:14:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:24.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:24.716058+0000) 2022-09-10T08:14:25.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:25.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:25.080355+0000) 2022-09-10T08:14:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:25 smithi104 ceph-mon[97044]: pgmap v7207: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:14:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:25.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:25.716239+0000) 2022-09-10T08:14:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:25 smithi110 ceph-mon[97633]: pgmap v7207: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:14:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:26.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:26.080589+0000) 2022-09-10T08:14:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:26.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:26.716456+0000) 2022-09-10T08:14:27.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:27.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:27.080831+0000) 2022-09-10T08:14:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:27 smithi104 ceph-mon[97044]: pgmap v7208: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:14:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:27.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:27.716695+0000) 2022-09-10T08:14:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:27 smithi110 ceph-mon[97633]: pgmap v7208: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:14:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:28.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:28.081117+0000) 2022-09-10T08:14:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:28.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:28.716895+0000) 2022-09-10T08:14:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:29.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:29.081337+0000) 2022-09-10T08:14:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:29 smithi104 ceph-mon[97044]: pgmap v7209: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:29.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:29.717110+0000) 2022-09-10T08:14:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:29 smithi110 ceph-mon[97633]: pgmap v7209: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:30.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:30.081585+0000) 2022-09-10T08:14:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:30.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:30.717360+0000) 2022-09-10T08:14:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:31.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:31.081798+0000) 2022-09-10T08:14:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:31 smithi104 ceph-mon[97044]: pgmap v7210: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:31.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:31.717608+0000) 2022-09-10T08:14:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:31 smithi110 ceph-mon[97633]: pgmap v7210: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:32.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:32.081969+0000) 2022-09-10T08:14:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:32.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:32.717884+0000) 2022-09-10T08:14:33.418 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:33.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:33.082163+0000) 2022-09-10T08:14:33.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:14:33.416+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:14:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:33 smithi104 ceph-mon[97044]: pgmap v7211: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:33.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:33.718081+0000) 2022-09-10T08:14:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:33 smithi110 ceph-mon[97633]: pgmap v7211: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:34.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:34.082322+0000) 2022-09-10T08:14:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:34.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:34.718333+0000) 2022-09-10T08:14:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:35.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:35.082511+0000) 2022-09-10T08:14:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:35 smithi104 ceph-mon[97044]: pgmap v7212: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:35.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:35.718599+0000) 2022-09-10T08:14:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:35 smithi110 ceph-mon[97633]: pgmap v7212: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:36.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:36.082798+0000) 2022-09-10T08:14:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:36.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:36.718843+0000) 2022-09-10T08:14:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:37.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:37.083075+0000) 2022-09-10T08:14:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:37 smithi104 ceph-mon[97044]: pgmap v7213: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:37.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:37.719064+0000) 2022-09-10T08:14:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:37 smithi110 ceph-mon[97633]: pgmap v7213: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:38.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:38.083358+0000) 2022-09-10T08:14:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:38.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:38.719330+0000) 2022-09-10T08:14:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:39.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:39.083508+0000) 2022-09-10T08:14:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:39 smithi104 ceph-mon[97044]: pgmap v7214: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:39.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:39.719546+0000) 2022-09-10T08:14:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:39 smithi110 ceph-mon[97633]: pgmap v7214: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:40.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:40.083736+0000) 2022-09-10T08:14:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:40.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:40.719751+0000) 2022-09-10T08:14:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:41.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:41.083928+0000) 2022-09-10T08:14:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:41 smithi104 ceph-mon[97044]: pgmap v7215: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:41.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:41.719989+0000) 2022-09-10T08:14:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:41 smithi110 ceph-mon[97633]: pgmap v7215: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:42.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:42.084174+0000) 2022-09-10T08:14:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:42.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:42.720267+0000) 2022-09-10T08:14:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:43.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:43.084330+0000) 2022-09-10T08:14:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:43 smithi104 ceph-mon[97044]: pgmap v7216: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:43.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:43.720543+0000) 2022-09-10T08:14:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:43 smithi110 ceph-mon[97633]: pgmap v7216: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:44.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:44.084543+0000) 2022-09-10T08:14:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:14:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:14:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:14:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:14:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:44.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:44.720779+0000) 2022-09-10T08:14:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:14:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:14:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:14:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:14:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:45.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:45.084671+0000) 2022-09-10T08:14:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:45 smithi104 ceph-mon[97044]: pgmap v7217: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:45.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:45.721049+0000) 2022-09-10T08:14:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:45 smithi110 ceph-mon[97633]: pgmap v7217: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:46.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:46.084842+0000) 2022-09-10T08:14:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:46 smithi104 ceph-mon[97044]: pgmap v7218: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:46.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:46.721347+0000) 2022-09-10T08:14:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:46 smithi110 ceph-mon[97633]: pgmap v7218: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:47.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:47.085029+0000) 2022-09-10T08:14:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:47.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:47.721624+0000) 2022-09-10T08:14:48.433 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:48.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:48.085247+0000) 2022-09-10T08:14:48.722 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:14:48.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:14:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:48.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:48.721779+0000) 2022-09-10T08:14:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:49 smithi110 ceph-mon[97633]: pgmap v7219: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:49 smithi104 ceph-mon[97044]: pgmap v7219: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:49.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:49.085449+0000) 2022-09-10T08:14:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:49.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:49.721941+0000) 2022-09-10T08:14:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:50.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:50.085653+0000) 2022-09-10T08:14:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:50.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:50.722192+0000) 2022-09-10T08:14:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:51.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:51.085801+0000) 2022-09-10T08:14:51.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:51 smithi110 ceph-mon[97633]: pgmap v7220: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:51 smithi104 ceph-mon[97044]: pgmap v7220: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:51.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:51.722350+0000) 2022-09-10T08:14:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:52.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:52.086064+0000) 2022-09-10T08:14:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:52.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:52.722566+0000) 2022-09-10T08:14:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:53.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:53.086311+0000) 2022-09-10T08:14:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:53 smithi110 ceph-mon[97633]: pgmap v7221: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:53 smithi104 ceph-mon[97044]: pgmap v7221: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:53.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:53.722797+0000) 2022-09-10T08:14:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:54.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:54.086542+0000) 2022-09-10T08:14:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:54.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:54.723009+0000) 2022-09-10T08:14:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:55.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:55.086750+0000) 2022-09-10T08:14:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:55 smithi110 ceph-mon[97633]: pgmap v7222: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:55 smithi104 ceph-mon[97044]: pgmap v7222: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:14:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:55.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:55.723210+0000) 2022-09-10T08:14:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:56.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:56.087008+0000) 2022-09-10T08:14:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:56.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:56.723406+0000) 2022-09-10T08:14:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:57.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:57.087239+0000) 2022-09-10T08:14:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:57 smithi110 ceph-mon[97633]: pgmap v7223: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:57 smithi104 ceph-mon[97044]: pgmap v7223: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:14:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:57.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:57.723605+0000) 2022-09-10T08:14:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:58.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:58.087446+0000) 2022-09-10T08:14:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:58.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:58.723831+0000) 2022-09-10T08:14:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:14:59.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:59.087662+0000) 2022-09-10T08:14:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:14:59 smithi110 ceph-mon[97633]: pgmap v7224: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:14:59 smithi104 ceph-mon[97044]: pgmap v7224: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:14:59.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:14:59.724066+0000) 2022-09-10T08:15:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:00.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:00.087825+0000) 2022-09-10T08:15:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:00.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:00.724286+0000) 2022-09-10T08:15:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:01.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:01.087975+0000) 2022-09-10T08:15:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:01 smithi110 ceph-mon[97633]: pgmap v7225: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:01 smithi104 ceph-mon[97044]: pgmap v7225: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:01.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:01.724500+0000) 2022-09-10T08:15:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:02.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:02.088209+0000) 2022-09-10T08:15:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:02.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:02.724772+0000) 2022-09-10T08:15:03.443 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:03.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:03.088396+0000) 2022-09-10T08:15:03.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:15:03.441+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:15:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:03 smithi110 ceph-mon[97633]: pgmap v7226: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:03 smithi104 ceph-mon[97044]: pgmap v7226: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:03.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:03.725020+0000) 2022-09-10T08:15:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:04.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:04.088538+0000) 2022-09-10T08:15:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:04.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:04.725300+0000) 2022-09-10T08:15:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:05.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:05.088726+0000) 2022-09-10T08:15:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:05 smithi110 ceph-mon[97633]: pgmap v7227: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:05 smithi104 ceph-mon[97044]: pgmap v7227: 129 pgs: 129 active+clean; 8.4 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:05.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:05.725440+0000) 2022-09-10T08:15:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:06.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:06.088926+0000) 2022-09-10T08:15:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:06.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:06.725673+0000) 2022-09-10T08:15:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:07.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:07.089183+0000) 2022-09-10T08:15:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:07 smithi110 ceph-mon[97633]: pgmap v7228: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:15:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:07 smithi104 ceph-mon[97044]: pgmap v7228: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:15:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:07.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:07.725850+0000) 2022-09-10T08:15:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:08.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:08.089353+0000) 2022-09-10T08:15:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:08 smithi110 ceph-mon[97633]: osdmap e532: 12 total, 12 up, 12 in 2022-09-10T08:15:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:08 smithi104 ceph-mon[97044]: osdmap e532: 12 total, 12 up, 12 in 2022-09-10T08:15:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:08.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:08.726147+0000) 2022-09-10T08:15:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:09.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:09.089561+0000) 2022-09-10T08:15:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:09 smithi110 ceph-mon[97633]: pgmap v7230: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:09 smithi104 ceph-mon[97044]: pgmap v7230: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:09.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:09.726381+0000) 2022-09-10T08:15:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:10.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:10.089784+0000) 2022-09-10T08:15:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:10.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:10.726583+0000) 2022-09-10T08:15:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:11.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:11.090025+0000) 2022-09-10T08:15:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:11 smithi110 ceph-mon[97633]: pgmap v7231: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:11 smithi104 ceph-mon[97044]: pgmap v7231: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:12.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:11.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:11.726796+0000) 2022-09-10T08:15:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:12.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:12.090282+0000) 2022-09-10T08:15:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:12.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:12.726989+0000) 2022-09-10T08:15:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:13.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:13.090506+0000) 2022-09-10T08:15:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:13 smithi110 ceph-mon[97633]: pgmap v7232: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:13 smithi104 ceph-mon[97044]: pgmap v7232: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:13.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:13.727239+0000) 2022-09-10T08:15:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:14.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:14.090688+0000) 2022-09-10T08:15:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:14.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:14.727440+0000) 2022-09-10T08:15:15.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:15.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:15.090900+0000) 2022-09-10T08:15:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:15 smithi110 ceph-mon[97633]: pgmap v7233: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:15 smithi110 ceph-mon[97633]: osdmap e533: 12 total, 12 up, 12 in 2022-09-10T08:15:16.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:15 smithi104 ceph-mon[97044]: pgmap v7233: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:15:16.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:15 smithi104 ceph-mon[97044]: osdmap e533: 12 total, 12 up, 12 in 2022-09-10T08:15:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:15.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:15.727613+0000) 2022-09-10T08:15:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:16.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:16.091143+0000) 2022-09-10T08:15:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:16.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:16.727800+0000) 2022-09-10T08:15:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:17.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:17.091355+0000) 2022-09-10T08:15:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:17 smithi110 ceph-mon[97633]: pgmap v7235: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:15:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:17 smithi104 ceph-mon[97044]: pgmap v7235: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:15:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:17.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:17.727995+0000) 2022-09-10T08:15:18.458 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:18.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:18.091570+0000) 2022-09-10T08:15:18.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:15:18.456+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:15:19.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:18.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:18.728233+0000) 2022-09-10T08:15:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:19.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:19.091797+0000) 2022-09-10T08:15:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:19 smithi110 ceph-mon[97633]: pgmap v7236: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:15:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:19 smithi104 ceph-mon[97044]: pgmap v7236: 129 pgs: 129 active+clean; 8.4 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:15:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:19.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:19.728433+0000) 2022-09-10T08:15:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:20.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:20.092024+0000) 2022-09-10T08:15:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:20.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:20.728652+0000) 2022-09-10T08:15:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:21.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:21.092253+0000) 2022-09-10T08:15:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:21 smithi110 ceph-mon[97633]: pgmap v7237: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:15:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:21 smithi104 ceph-mon[97044]: pgmap v7237: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:15:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:21.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:21.728915+0000) 2022-09-10T08:15:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:22.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:22.092462+0000) 2022-09-10T08:15:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:22.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:22.729214+0000) 2022-09-10T08:15:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:23.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:23.092682+0000) 2022-09-10T08:15:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:23 smithi110 ceph-mon[97633]: pgmap v7238: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:15:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:23 smithi104 ceph-mon[97044]: pgmap v7238: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:15:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:23.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:23.729457+0000) 2022-09-10T08:15:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:24.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:24.092916+0000) 2022-09-10T08:15:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:24.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:24.729639+0000) 2022-09-10T08:15:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:25.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:25.093162+0000) 2022-09-10T08:15:25.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:25 smithi110 ceph-mon[97633]: pgmap v7239: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:15:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:25 smithi104 ceph-mon[97044]: pgmap v7239: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:15:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:25.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:25.729856+0000) 2022-09-10T08:15:26.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:26.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:26.093400+0000) 2022-09-10T08:15:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:26.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:26.730129+0000) 2022-09-10T08:15:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:27.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:27.093578+0000) 2022-09-10T08:15:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:27 smithi110 ceph-mon[97633]: pgmap v7240: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:15:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:27 smithi104 ceph-mon[97044]: pgmap v7240: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:15:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:27.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:27.730399+0000) 2022-09-10T08:15:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:28.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:28.093854+0000) 2022-09-10T08:15:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:28.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:28.730659+0000) 2022-09-10T08:15:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:29.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:29.094070+0000) 2022-09-10T08:15:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:29 smithi110 ceph-mon[97633]: pgmap v7241: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:29 smithi104 ceph-mon[97044]: pgmap v7241: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:29.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:29.730911+0000) 2022-09-10T08:15:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:30.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:30.094277+0000) 2022-09-10T08:15:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:30.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:30.731207+0000) 2022-09-10T08:15:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:31.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:31.094448+0000) 2022-09-10T08:15:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:31 smithi110 ceph-mon[97633]: pgmap v7242: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:31 smithi104 ceph-mon[97044]: pgmap v7242: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:31.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:31.731457+0000) 2022-09-10T08:15:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:32.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:32.094691+0000) 2022-09-10T08:15:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:32.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:32.731670+0000) 2022-09-10T08:15:33.473 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:33.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:33.094909+0000) 2022-09-10T08:15:33.734 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:15:33.473+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:15:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:33 smithi110 ceph-mon[97633]: pgmap v7243: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:33 smithi104 ceph-mon[97044]: pgmap v7243: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:33.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:33.731896+0000) 2022-09-10T08:15:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:34.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:34.095180+0000) 2022-09-10T08:15:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:34.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:34.732144+0000) 2022-09-10T08:15:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:35.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:35.095360+0000) 2022-09-10T08:15:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:35 smithi110 ceph-mon[97633]: pgmap v7244: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:35 smithi104 ceph-mon[97044]: pgmap v7244: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:35.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:35.732371+0000) 2022-09-10T08:15:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:36.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:36.095592+0000) 2022-09-10T08:15:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:36.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:36.732624+0000) 2022-09-10T08:15:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:37.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:37.095801+0000) 2022-09-10T08:15:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:37 smithi110 ceph-mon[97633]: pgmap v7245: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:37 smithi104 ceph-mon[97044]: pgmap v7245: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:37.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:37.732750+0000) 2022-09-10T08:15:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:38.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:38.096045+0000) 2022-09-10T08:15:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:38.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:38.732995+0000) 2022-09-10T08:15:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:39.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:39.096293+0000) 2022-09-10T08:15:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:39 smithi110 ceph-mon[97633]: pgmap v7246: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:39 smithi104 ceph-mon[97044]: pgmap v7246: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:39.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:39.733279+0000) 2022-09-10T08:15:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:40.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:40.096539+0000) 2022-09-10T08:15:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:40.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:40.733533+0000) 2022-09-10T08:15:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:41.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:41.096771+0000) 2022-09-10T08:15:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:41 smithi110 ceph-mon[97633]: pgmap v7247: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:41 smithi104 ceph-mon[97044]: pgmap v7247: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:41.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:41.733707+0000) 2022-09-10T08:15:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:42.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:42.096968+0000) 2022-09-10T08:15:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:42.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:42.733922+0000) 2022-09-10T08:15:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:43.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:43.097177+0000) 2022-09-10T08:15:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:43 smithi110 ceph-mon[97633]: pgmap v7248: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:43 smithi104 ceph-mon[97044]: pgmap v7248: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:43.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:43.734155+0000) 2022-09-10T08:15:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:44.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:44.097318+0000) 2022-09-10T08:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:15:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:15:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:15:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:15:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:15:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:15:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:44.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:44.734288+0000) 2022-09-10T08:15:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:45.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:45.097564+0000) 2022-09-10T08:15:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:45 smithi110 ceph-mon[97633]: pgmap v7249: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:45 smithi104 ceph-mon[97044]: pgmap v7249: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:45.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:45.734546+0000) 2022-09-10T08:15:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:46.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:46.097758+0000) 2022-09-10T08:15:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:46.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:46.734786+0000) 2022-09-10T08:15:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:47.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:47.097946+0000) 2022-09-10T08:15:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:47 smithi104 ceph-mon[97044]: pgmap v7250: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:47.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:47.734981+0000) 2022-09-10T08:15:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:47 smithi110 ceph-mon[97633]: pgmap v7250: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:48.476 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:48.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:48.098192+0000) 2022-09-10T08:15:48.735 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:15:48.474+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:15:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:48.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:48.735241+0000) 2022-09-10T08:15:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:49.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:49.098416+0000) 2022-09-10T08:15:50.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:49.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:49.735548+0000) 2022-09-10T08:15:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:49 smithi104 ceph-mon[97044]: pgmap v7251: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:49 smithi110 ceph-mon[97633]: pgmap v7251: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:50.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:50.098660+0000) 2022-09-10T08:15:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:50.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:50.735844+0000) 2022-09-10T08:15:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:51.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:51.098865+0000) 2022-09-10T08:15:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:51 smithi104 ceph-mon[97044]: pgmap v7252: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:51.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:51.736157+0000) 2022-09-10T08:15:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:51 smithi110 ceph-mon[97633]: pgmap v7252: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:52.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:52.099068+0000) 2022-09-10T08:15:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:52.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:52.736412+0000) 2022-09-10T08:15:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:53.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:53.099290+0000) 2022-09-10T08:15:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:53 smithi104 ceph-mon[97044]: pgmap v7253: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:53.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:53.736621+0000) 2022-09-10T08:15:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:53 smithi110 ceph-mon[97633]: pgmap v7253: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:54.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:54.099460+0000) 2022-09-10T08:15:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:54.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:54.736762+0000) 2022-09-10T08:15:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:55.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:55.099667+0000) 2022-09-10T08:15:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:55 smithi104 ceph-mon[97044]: pgmap v7254: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:55.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:55.737009+0000) 2022-09-10T08:15:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:55 smithi110 ceph-mon[97633]: pgmap v7254: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:15:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:56.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:56.099861+0000) 2022-09-10T08:15:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:56.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:56.737187+0000) 2022-09-10T08:15:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:57.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:57.099990+0000) 2022-09-10T08:15:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:57 smithi104 ceph-mon[97044]: pgmap v7255: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:57.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:57.737394+0000) 2022-09-10T08:15:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:57 smithi110 ceph-mon[97633]: pgmap v7255: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:15:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:58.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:58.100220+0000) 2022-09-10T08:15:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:58.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:58.737565+0000) 2022-09-10T08:15:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:15:59.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:59.100459+0000) 2022-09-10T08:16:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:15:59 smithi104 ceph-mon[97044]: pgmap v7256: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:15:59.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:15:59.737800+0000) 2022-09-10T08:16:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:15:59 smithi110 ceph-mon[97633]: pgmap v7256: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:00.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:00.100636+0000) 2022-09-10T08:16:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:00.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:00.738072+0000) 2022-09-10T08:16:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:01.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:01.100793+0000) 2022-09-10T08:16:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:01 smithi104 ceph-mon[97044]: pgmap v7257: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:01.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:01.738399+0000) 2022-09-10T08:16:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:01 smithi110 ceph-mon[97633]: pgmap v7257: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:02.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:02.100915+0000) 2022-09-10T08:16:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:02.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:02.738642+0000) 2022-09-10T08:16:03.480 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:03.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:03.101168+0000) 2022-09-10T08:16:03.739 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:16:03.477+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:16:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:03 smithi104 ceph-mon[97044]: pgmap v7258: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:03.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:03.738803+0000) 2022-09-10T08:16:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:03 smithi110 ceph-mon[97633]: pgmap v7258: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:04.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:04.101409+0000) 2022-09-10T08:16:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:04.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:04.738996+0000) 2022-09-10T08:16:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:05.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:05.101633+0000) 2022-09-10T08:16:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:05 smithi104 ceph-mon[97044]: pgmap v7259: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:05.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:05.739247+0000) 2022-09-10T08:16:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:05 smithi110 ceph-mon[97633]: pgmap v7259: 129 pgs: 129 active+clean; 8.4 MiB data, 748 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:06.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:06.101920+0000) 2022-09-10T08:16:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:06.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:06.739441+0000) 2022-09-10T08:16:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:07.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:07.102188+0000) 2022-09-10T08:16:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:07 smithi104 ceph-mon[97044]: pgmap v7260: 129 pgs: 129 active+clean; 8.5 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:16:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:07.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:07.739661+0000) 2022-09-10T08:16:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:07 smithi110 ceph-mon[97633]: pgmap v7260: 129 pgs: 129 active+clean; 8.5 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:16:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:08.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:08.102423+0000) 2022-09-10T08:16:09.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:08 smithi104 ceph-mon[97044]: osdmap e534: 12 total, 12 up, 12 in 2022-09-10T08:16:09.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:08.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:08.739835+0000) 2022-09-10T08:16:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:08 smithi110 ceph-mon[97633]: osdmap e534: 12 total, 12 up, 12 in 2022-09-10T08:16:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:09.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:09.102584+0000) 2022-09-10T08:16:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:09 smithi104 ceph-mon[97044]: pgmap v7262: 129 pgs: 129 active+clean; 8.5 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:09.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:09.740154+0000) 2022-09-10T08:16:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:09 smithi110 ceph-mon[97633]: pgmap v7262: 129 pgs: 129 active+clean; 8.5 MiB data, 749 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:10.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:10.102855+0000) 2022-09-10T08:16:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:10.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:10.740342+0000) 2022-09-10T08:16:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:11.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:11.103070+0000) 2022-09-10T08:16:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:11 smithi104 ceph-mon[97044]: pgmap v7263: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:11.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:11.740619+0000) 2022-09-10T08:16:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:11 smithi110 ceph-mon[97633]: pgmap v7263: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:12.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:12.103278+0000) 2022-09-10T08:16:13.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:12.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:12.740820+0000) 2022-09-10T08:16:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:13.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:13.103555+0000) 2022-09-10T08:16:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:13 smithi104 ceph-mon[97044]: pgmap v7264: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:13.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:13.740985+0000) 2022-09-10T08:16:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:13 smithi110 ceph-mon[97633]: pgmap v7264: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:14.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:14.103784+0000) 2022-09-10T08:16:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:14.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:14.741179+0000) 2022-09-10T08:16:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:15.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:15.104000+0000) 2022-09-10T08:16:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:15 smithi104 ceph-mon[97044]: pgmap v7265: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:15 smithi104 ceph-mon[97044]: osdmap e535: 12 total, 12 up, 12 in 2022-09-10T08:16:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:15.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:15.741375+0000) 2022-09-10T08:16:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:15 smithi110 ceph-mon[97633]: pgmap v7265: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:16:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:15 smithi110 ceph-mon[97633]: osdmap e535: 12 total, 12 up, 12 in 2022-09-10T08:16:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:16.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:16.104240+0000) 2022-09-10T08:16:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:16.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:16.741578+0000) 2022-09-10T08:16:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:17.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:17.104491+0000) 2022-09-10T08:16:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:17 smithi104 ceph-mon[97044]: pgmap v7267: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T08:16:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:17.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:17.741865+0000) 2022-09-10T08:16:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:17 smithi110 ceph-mon[97633]: pgmap v7267: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T08:16:18.483 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:18.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:18.104672+0000) 2022-09-10T08:16:18.743 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:16:18.482+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:16:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:18.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:18.742164+0000) 2022-09-10T08:16:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:19.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:19.104940+0000) 2022-09-10T08:16:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:19 smithi104 ceph-mon[97044]: pgmap v7268: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:16:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:19.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:19.742388+0000) 2022-09-10T08:16:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:19 smithi110 ceph-mon[97633]: pgmap v7268: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:16:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:20.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:20.105186+0000) 2022-09-10T08:16:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:20.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:20.742591+0000) 2022-09-10T08:16:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:21.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:21.105309+0000) 2022-09-10T08:16:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:21 smithi104 ceph-mon[97044]: pgmap v7269: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:16:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:21.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:21.742802+0000) 2022-09-10T08:16:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:21 smithi110 ceph-mon[97633]: pgmap v7269: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:16:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:22.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:22.105464+0000) 2022-09-10T08:16:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:22.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:22.742942+0000) 2022-09-10T08:16:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:23.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:23.105623+0000) 2022-09-10T08:16:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:23 smithi104 ceph-mon[97044]: pgmap v7270: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:16:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:23.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:23.743205+0000) 2022-09-10T08:16:24.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:23 smithi110 ceph-mon[97633]: pgmap v7270: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:16:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:24.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:24.105847+0000) 2022-09-10T08:16:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:24.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:24.743478+0000) 2022-09-10T08:16:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:25.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:25.106047+0000) 2022-09-10T08:16:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:25 smithi104 ceph-mon[97044]: pgmap v7271: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:16:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:25.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:25.743737+0000) 2022-09-10T08:16:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:25 smithi110 ceph-mon[97633]: pgmap v7271: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:16:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:26.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:26.106349+0000) 2022-09-10T08:16:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:26.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:26.744033+0000) 2022-09-10T08:16:27.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:27.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:27.106555+0000) 2022-09-10T08:16:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:27 smithi104 ceph-mon[97044]: pgmap v7272: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:16:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:27.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:27.744276+0000) 2022-09-10T08:16:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:27 smithi110 ceph-mon[97633]: pgmap v7272: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:16:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:28.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:28.106771+0000) 2022-09-10T08:16:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:28.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:28.744437+0000) 2022-09-10T08:16:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:29.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:29.106990+0000) 2022-09-10T08:16:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:29 smithi104 ceph-mon[97044]: pgmap v7273: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:29.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:29.744585+0000) 2022-09-10T08:16:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:29 smithi110 ceph-mon[97633]: pgmap v7273: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:30.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:30.107257+0000) 2022-09-10T08:16:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:30.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:30.744821+0000) 2022-09-10T08:16:31.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:31.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:31.107489+0000) 2022-09-10T08:16:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:31 smithi104 ceph-mon[97044]: pgmap v7274: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:31.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:31.745122+0000) 2022-09-10T08:16:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:31 smithi110 ceph-mon[97633]: pgmap v7274: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:32.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:32.107645+0000) 2022-09-10T08:16:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:32.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:32.745327+0000) 2022-09-10T08:16:33.490 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:33.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:33.107910+0000) 2022-09-10T08:16:33.746 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:16:33.488+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:16:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:33 smithi104 ceph-mon[97044]: pgmap v7275: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:33.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:33.745596+0000) 2022-09-10T08:16:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:33 smithi110 ceph-mon[97633]: pgmap v7275: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:34.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:34.108204+0000) 2022-09-10T08:16:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:34.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:34.745873+0000) 2022-09-10T08:16:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:35.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:35.108416+0000) 2022-09-10T08:16:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:35 smithi104 ceph-mon[97044]: pgmap v7276: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:35.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:35.746122+0000) 2022-09-10T08:16:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:35 smithi110 ceph-mon[97633]: pgmap v7276: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:36.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:36.108648+0000) 2022-09-10T08:16:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:36.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:36.746269+0000) 2022-09-10T08:16:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:37.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:37.108827+0000) 2022-09-10T08:16:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:37 smithi104 ceph-mon[97044]: pgmap v7277: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:37.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:37.746503+0000) 2022-09-10T08:16:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:37 smithi110 ceph-mon[97633]: pgmap v7277: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:38.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:38.108948+0000) 2022-09-10T08:16:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:38.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:38.746690+0000) 2022-09-10T08:16:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:39.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:39.109186+0000) 2022-09-10T08:16:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:39 smithi104 ceph-mon[97044]: pgmap v7278: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:40.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:39.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:39.746894+0000) 2022-09-10T08:16:40.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:39 smithi110 ceph-mon[97633]: pgmap v7278: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:40.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:40.109374+0000) 2022-09-10T08:16:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:40.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:40.747012+0000) 2022-09-10T08:16:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:41.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:41.109515+0000) 2022-09-10T08:16:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:41 smithi104 ceph-mon[97044]: pgmap v7279: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:41.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:41.747163+0000) 2022-09-10T08:16:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:41 smithi110 ceph-mon[97633]: pgmap v7279: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:42.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:42.109701+0000) 2022-09-10T08:16:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:42.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:42.747302+0000) 2022-09-10T08:16:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:43.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:43.109936+0000) 2022-09-10T08:16:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:43 smithi104 ceph-mon[97044]: pgmap v7280: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:43.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:43.747473+0000) 2022-09-10T08:16:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:43 smithi110 ceph-mon[97633]: pgmap v7280: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:44.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:44.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:44.110152+0000) 2022-09-10T08:16:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:16:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:16:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:16:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:16:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:44.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:44.747631+0000) 2022-09-10T08:16:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:16:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:16:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:16:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:16:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:45.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:45.110363+0000) 2022-09-10T08:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:45 smithi104 ceph-mon[97044]: pgmap v7281: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:45.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:45.747774+0000) 2022-09-10T08:16:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:45 smithi110 ceph-mon[97633]: pgmap v7281: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:46.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:46.110508+0000) 2022-09-10T08:16:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:46 smithi104 ceph-mon[97044]: pgmap v7282: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:46.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:46.747915+0000) 2022-09-10T08:16:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:46 smithi110 ceph-mon[97633]: pgmap v7282: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:47.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:47.110762+0000) 2022-09-10T08:16:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:47.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:47.748092+0000) 2022-09-10T08:16:48.501 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:48.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:48.110925+0000) 2022-09-10T08:16:48.812 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:48.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:48.748259+0000) 2022-09-10T08:16:48.813 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:16:48.499+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:16:49.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:49 smithi104 ceph-mon[97044]: pgmap v7283: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:49.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:49.111200+0000) 2022-09-10T08:16:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:49 smithi110 ceph-mon[97633]: pgmap v7283: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:49.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:49.748397+0000) 2022-09-10T08:16:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:50.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:50.111357+0000) 2022-09-10T08:16:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:50.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:50.748519+0000) 2022-09-10T08:16:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:51.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:51.111521+0000) 2022-09-10T08:16:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:51 smithi110 ceph-mon[97633]: pgmap v7284: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:51 smithi104 ceph-mon[97044]: pgmap v7284: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:51.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:51.748659+0000) 2022-09-10T08:16:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:52.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:52.111737+0000) 2022-09-10T08:16:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:52.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:52.748812+0000) 2022-09-10T08:16:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:53.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:53.111981+0000) 2022-09-10T08:16:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:53 smithi110 ceph-mon[97633]: pgmap v7285: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:53 smithi104 ceph-mon[97044]: pgmap v7285: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:53.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:53.748954+0000) 2022-09-10T08:16:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:54.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:54.112226+0000) 2022-09-10T08:16:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:54.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:54.749082+0000) 2022-09-10T08:16:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:55.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:55.112441+0000) 2022-09-10T08:16:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:55 smithi110 ceph-mon[97633]: pgmap v7286: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:55 smithi104 ceph-mon[97044]: pgmap v7286: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:16:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:55.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:55.749226+0000) 2022-09-10T08:16:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:56.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:56.112619+0000) 2022-09-10T08:16:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:56.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:56.749406+0000) 2022-09-10T08:16:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:57.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:57.112783+0000) 2022-09-10T08:16:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:57 smithi110 ceph-mon[97633]: pgmap v7287: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:57 smithi104 ceph-mon[97044]: pgmap v7287: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:16:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:57.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:57.749552+0000) 2022-09-10T08:16:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:58.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:58.113022+0000) 2022-09-10T08:16:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:58.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:58.749766+0000) 2022-09-10T08:16:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:16:59.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:59.113246+0000) 2022-09-10T08:16:59.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:16:59 smithi110 ceph-mon[97633]: pgmap v7288: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:16:59 smithi104 ceph-mon[97044]: pgmap v7288: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:16:59.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:16:59.749934+0000) 2022-09-10T08:17:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:00.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:00.113372+0000) 2022-09-10T08:17:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:00.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:00.750065+0000) 2022-09-10T08:17:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:01.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:01.113610+0000) 2022-09-10T08:17:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:01 smithi110 ceph-mon[97633]: pgmap v7289: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:01 smithi104 ceph-mon[97044]: pgmap v7289: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:01.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:01.750230+0000) 2022-09-10T08:17:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:02.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:02.113894+0000) 2022-09-10T08:17:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:02.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:02.750396+0000) 2022-09-10T08:17:03.511 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:03.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:03.114087+0000) 2022-09-10T08:17:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:17:03.509+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:17:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:03.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:03.750533+0000) 2022-09-10T08:17:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:03 smithi110 ceph-mon[97633]: pgmap v7290: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:03 smithi104 ceph-mon[97044]: pgmap v7290: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:04.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:04.114284+0000) 2022-09-10T08:17:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:04.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:04.750670+0000) 2022-09-10T08:17:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:05.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:05.114469+0000) 2022-09-10T08:17:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:05 smithi110 ceph-mon[97633]: pgmap v7291: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:05 smithi104 ceph-mon[97044]: pgmap v7291: 129 pgs: 129 active+clean; 8.5 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:05.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:05.750857+0000) 2022-09-10T08:17:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:06.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:06.114703+0000) 2022-09-10T08:17:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:06.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:06.751027+0000) 2022-09-10T08:17:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:07.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:07.114890+0000) 2022-09-10T08:17:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:07 smithi110 ceph-mon[97633]: pgmap v7292: 129 pgs: 129 active+clean; 8.5 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:17:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:07 smithi104 ceph-mon[97044]: pgmap v7292: 129 pgs: 129 active+clean; 8.5 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:17:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:07.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:07.751172+0000) 2022-09-10T08:17:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:08.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:08.115039+0000) 2022-09-10T08:17:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:08 smithi110 ceph-mon[97633]: osdmap e536: 12 total, 12 up, 12 in 2022-09-10T08:17:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:08 smithi104 ceph-mon[97044]: osdmap e536: 12 total, 12 up, 12 in 2022-09-10T08:17:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:08.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:08.751411+0000) 2022-09-10T08:17:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:09.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:09.115283+0000) 2022-09-10T08:17:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:09 smithi110 ceph-mon[97633]: pgmap v7294: 129 pgs: 129 active+clean; 8.5 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:09 smithi104 ceph-mon[97044]: pgmap v7294: 129 pgs: 129 active+clean; 8.5 MiB data, 752 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:09.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:09.751554+0000) 2022-09-10T08:17:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:10.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:10.115511+0000) 2022-09-10T08:17:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:10.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:10.751681+0000) 2022-09-10T08:17:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:11.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:11.115681+0000) 2022-09-10T08:17:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:11 smithi110 ceph-mon[97633]: pgmap v7295: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:11 smithi104 ceph-mon[97044]: pgmap v7295: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:11.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:11.751817+0000) 2022-09-10T08:17:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:12.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:12.115960+0000) 2022-09-10T08:17:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:12.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:12.751940+0000) 2022-09-10T08:17:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:13.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:13.116215+0000) 2022-09-10T08:17:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:13 smithi110 ceph-mon[97633]: pgmap v7296: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:13 smithi104 ceph-mon[97044]: pgmap v7296: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:13.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:13.752115+0000) 2022-09-10T08:17:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:14.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:14.116404+0000) 2022-09-10T08:17:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:14.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:14.752232+0000) 2022-09-10T08:17:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:15.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:15.116614+0000) 2022-09-10T08:17:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:15 smithi110 ceph-mon[97633]: pgmap v7297: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:15 smithi110 ceph-mon[97633]: osdmap e537: 12 total, 12 up, 12 in 2022-09-10T08:17:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:15 smithi104 ceph-mon[97044]: pgmap v7297: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:17:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:15 smithi104 ceph-mon[97044]: osdmap e537: 12 total, 12 up, 12 in 2022-09-10T08:17:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:15.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:15.752357+0000) 2022-09-10T08:17:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:16.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:16.116891+0000) 2022-09-10T08:17:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:16.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:16.752515+0000) 2022-09-10T08:17:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:17.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:17.117078+0000) 2022-09-10T08:17:17.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:17 smithi110 ceph-mon[97633]: pgmap v7299: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:17:18.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:17 smithi104 ceph-mon[97044]: pgmap v7299: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:17:18.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:17.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:17.752657+0000) 2022-09-10T08:17:18.517 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:18.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:18.117328+0000) 2022-09-10T08:17:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:17:18.515+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:17:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:18.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:18.752783+0000) 2022-09-10T08:17:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:19.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:19.117513+0000) 2022-09-10T08:17:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:19 smithi110 ceph-mon[97633]: pgmap v7300: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:17:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:19 smithi104 ceph-mon[97044]: pgmap v7300: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:17:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:19.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:19.752914+0000) 2022-09-10T08:17:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:20.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:20.117711+0000) 2022-09-10T08:17:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:20.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:20.753035+0000) 2022-09-10T08:17:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:21.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:21.117893+0000) 2022-09-10T08:17:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:21 smithi110 ceph-mon[97633]: pgmap v7301: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:17:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:21 smithi104 ceph-mon[97044]: pgmap v7301: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:17:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:21.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:21.753164+0000) 2022-09-10T08:17:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:22.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:22.118143+0000) 2022-09-10T08:17:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:22.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:22.753296+0000) 2022-09-10T08:17:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:23.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:23.118369+0000) 2022-09-10T08:17:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:23 smithi110 ceph-mon[97633]: pgmap v7302: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:17:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:23 smithi104 ceph-mon[97044]: pgmap v7302: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:17:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:23.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:23.753428+0000) 2022-09-10T08:17:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:24.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:24.118606+0000) 2022-09-10T08:17:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:24.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:24.753557+0000) 2022-09-10T08:17:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:25.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:25.118823+0000) 2022-09-10T08:17:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:25 smithi110 ceph-mon[97633]: pgmap v7303: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:17:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:25 smithi104 ceph-mon[97044]: pgmap v7303: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:17:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:25.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:25.753706+0000) 2022-09-10T08:17:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:26.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:26.119057+0000) 2022-09-10T08:17:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:26.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:26.753841+0000) 2022-09-10T08:17:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:27.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:27.119259+0000) 2022-09-10T08:17:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:27 smithi104 ceph-mon[97044]: pgmap v7304: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:17:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:27.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:27.753993+0000) 2022-09-10T08:17:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:27 smithi110 ceph-mon[97633]: pgmap v7304: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:17:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:28.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:28.119517+0000) 2022-09-10T08:17:29.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:28.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:28.754151+0000) 2022-09-10T08:17:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:29.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:29.119746+0000) 2022-09-10T08:17:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:29 smithi104 ceph-mon[97044]: pgmap v7305: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:29.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:29.754317+0000) 2022-09-10T08:17:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:29 smithi110 ceph-mon[97633]: pgmap v7305: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:30.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:30.119977+0000) 2022-09-10T08:17:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:30.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:30.754480+0000) 2022-09-10T08:17:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:31.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:31.120169+0000) 2022-09-10T08:17:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:31 smithi104 ceph-mon[97044]: pgmap v7306: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:32.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:31.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:31.754625+0000) 2022-09-10T08:17:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:31 smithi110 ceph-mon[97633]: pgmap v7306: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:32.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:32.120364+0000) 2022-09-10T08:17:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:32.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:32.754801+0000) 2022-09-10T08:17:33.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:33.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:33.120591+0000) 2022-09-10T08:17:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:17:33.525+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:17:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:33.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:33.754940+0000) 2022-09-10T08:17:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:33 smithi104 ceph-mon[97044]: pgmap v7307: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:33 smithi110 ceph-mon[97633]: pgmap v7307: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:34.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:34.120804+0000) 2022-09-10T08:17:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:34.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:34.755073+0000) 2022-09-10T08:17:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:35.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:35.121055+0000) 2022-09-10T08:17:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:35 smithi104 ceph-mon[97044]: pgmap v7308: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:35.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:35.755226+0000) 2022-09-10T08:17:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:35 smithi110 ceph-mon[97633]: pgmap v7308: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:36.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:36.121349+0000) 2022-09-10T08:17:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:36.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:36.755356+0000) 2022-09-10T08:17:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:37.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:37.121602+0000) 2022-09-10T08:17:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:37 smithi104 ceph-mon[97044]: pgmap v7309: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:37.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:37.755488+0000) 2022-09-10T08:17:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:37 smithi110 ceph-mon[97633]: pgmap v7309: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:38.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:38.121811+0000) 2022-09-10T08:17:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:38.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:38.755619+0000) 2022-09-10T08:17:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:39.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:39.122009+0000) 2022-09-10T08:17:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:39 smithi104 ceph-mon[97044]: pgmap v7310: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:39.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:39.755747+0000) 2022-09-10T08:17:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:39 smithi110 ceph-mon[97633]: pgmap v7310: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:40.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:40.122267+0000) 2022-09-10T08:17:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:40.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:40.755866+0000) 2022-09-10T08:17:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:41.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:41.122446+0000) 2022-09-10T08:17:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:41 smithi104 ceph-mon[97044]: pgmap v7311: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:41.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:41.756000+0000) 2022-09-10T08:17:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:41 smithi110 ceph-mon[97633]: pgmap v7311: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:42.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:42.122650+0000) 2022-09-10T08:17:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:42.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:42.756148+0000) 2022-09-10T08:17:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:43.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:43.122905+0000) 2022-09-10T08:17:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:43 smithi104 ceph-mon[97044]: pgmap v7312: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:17:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:43.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:43.756306+0000) 2022-09-10T08:17:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:43 smithi110 ceph-mon[97633]: pgmap v7312: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:17:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:44.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:44.123194+0000) 2022-09-10T08:17:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:17:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:17:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:17:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:17:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:44.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:44.756421+0000) 2022-09-10T08:17:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:17:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:17:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:17:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:17:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:45.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:45.123449+0000) 2022-09-10T08:17:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:45 smithi104 ceph-mon[97044]: pgmap v7313: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:17:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:45.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:45.756555+0000) 2022-09-10T08:17:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:45 smithi110 ceph-mon[97633]: pgmap v7313: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:17:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:46.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:46.123635+0000) 2022-09-10T08:17:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:46.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:46.756685+0000) 2022-09-10T08:17:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:47.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:47.123837+0000) 2022-09-10T08:17:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:47 smithi104 ceph-mon[97044]: pgmap v7314: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:47.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:47.756822+0000) 2022-09-10T08:17:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:47 smithi110 ceph-mon[97633]: pgmap v7314: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:48.536 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:48.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:48.124044+0000) 2022-09-10T08:17:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:17:48.534+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:17:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:48.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:48.756944+0000) 2022-09-10T08:17:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:49.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:49.124284+0000) 2022-09-10T08:17:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:49 smithi104 ceph-mon[97044]: pgmap v7315: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:17:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:49.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:49.757083+0000) 2022-09-10T08:17:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:49 smithi110 ceph-mon[97633]: pgmap v7315: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:17:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:50.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:50.124451+0000) 2022-09-10T08:17:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:50.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:50.757228+0000) 2022-09-10T08:17:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:51.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:51.124641+0000) 2022-09-10T08:17:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:51 smithi104 ceph-mon[97044]: pgmap v7316: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:51.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:51.757363+0000) 2022-09-10T08:17:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:51 smithi110 ceph-mon[97633]: pgmap v7316: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:52.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:52.124955+0000) 2022-09-10T08:17:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:52.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:52.757488+0000) 2022-09-10T08:17:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:53.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:53.125173+0000) 2022-09-10T08:17:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:53 smithi104 ceph-mon[97044]: pgmap v7317: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:53.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:53.757614+0000) 2022-09-10T08:17:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:53 smithi110 ceph-mon[97633]: pgmap v7317: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:54.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:54.125429+0000) 2022-09-10T08:17:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:54.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:54.757738+0000) 2022-09-10T08:17:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:55.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:55.125694+0000) 2022-09-10T08:17:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:55 smithi104 ceph-mon[97044]: pgmap v7318: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:55.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:55.757881+0000) 2022-09-10T08:17:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:55 smithi110 ceph-mon[97633]: pgmap v7318: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:17:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:56.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:56.125892+0000) 2022-09-10T08:17:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:56.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:56.758004+0000) 2022-09-10T08:17:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:57.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:57.126190+0000) 2022-09-10T08:17:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:57 smithi104 ceph-mon[97044]: pgmap v7319: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:57.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:57.758160+0000) 2022-09-10T08:17:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:57 smithi110 ceph-mon[97633]: pgmap v7319: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:17:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:58.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:58.126418+0000) 2022-09-10T08:17:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:58.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:58.758298+0000) 2022-09-10T08:17:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:17:59.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:59.126630+0000) 2022-09-10T08:18:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:17:59 smithi104 ceph-mon[97044]: pgmap v7320: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:17:59.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:17:59.758430+0000) 2022-09-10T08:18:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:17:59 smithi110 ceph-mon[97633]: pgmap v7320: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:00.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:00.126870+0000) 2022-09-10T08:18:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:00.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:00.758558+0000) 2022-09-10T08:18:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:01.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:01.127140+0000) 2022-09-10T08:18:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:01 smithi104 ceph-mon[97044]: pgmap v7321: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:01.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:01.758705+0000) 2022-09-10T08:18:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:01 smithi110 ceph-mon[97633]: pgmap v7321: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:02.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:02.127375+0000) 2022-09-10T08:18:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:02.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:02.758843+0000) 2022-09-10T08:18:03.549 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:03.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:03.127580+0000) 2022-09-10T08:18:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:03.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:03.758990+0000) 2022-09-10T08:18:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:18:03.548+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:18:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:03 smithi104 ceph-mon[97044]: pgmap v7322: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:03 smithi110 ceph-mon[97633]: pgmap v7322: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:04.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:04.127800+0000) 2022-09-10T08:18:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:04.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:04.759154+0000) 2022-09-10T08:18:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:05.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:05.128077+0000) 2022-09-10T08:18:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:05 smithi104 ceph-mon[97044]: pgmap v7323: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:05.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:05.759291+0000) 2022-09-10T08:18:06.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:05 smithi110 ceph-mon[97633]: pgmap v7323: 129 pgs: 129 active+clean; 8.5 MiB data, 756 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:06.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:06.128354+0000) 2022-09-10T08:18:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:06.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:06.759440+0000) 2022-09-10T08:18:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:07.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:07.128577+0000) 2022-09-10T08:18:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:07 smithi104 ceph-mon[97044]: pgmap v7324: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:18:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:07.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:07.759575+0000) 2022-09-10T08:18:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:07 smithi110 ceph-mon[97633]: pgmap v7324: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:18:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:08.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:08.128789+0000) 2022-09-10T08:18:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:08 smithi104 ceph-mon[97044]: osdmap e538: 12 total, 12 up, 12 in 2022-09-10T08:18:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:08.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:08.759761+0000) 2022-09-10T08:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:08 smithi110 ceph-mon[97633]: osdmap e538: 12 total, 12 up, 12 in 2022-09-10T08:18:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:09.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:09.129022+0000) 2022-09-10T08:18:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:09 smithi104 ceph-mon[97044]: pgmap v7326: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:09.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:09.759935+0000) 2022-09-10T08:18:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:09 smithi110 ceph-mon[97633]: pgmap v7326: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:10.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:10.129209+0000) 2022-09-10T08:18:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:10.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:10.760156+0000) 2022-09-10T08:18:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:11.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:11.129434+0000) 2022-09-10T08:18:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:11 smithi104 ceph-mon[97044]: pgmap v7327: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:11.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:11.760342+0000) 2022-09-10T08:18:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:11 smithi110 ceph-mon[97633]: pgmap v7327: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:12.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:12.129678+0000) 2022-09-10T08:18:13.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:12.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:12.760467+0000) 2022-09-10T08:18:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:13.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:13.129923+0000) 2022-09-10T08:18:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:13 smithi104 ceph-mon[97044]: pgmap v7328: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:13.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:13.760593+0000) 2022-09-10T08:18:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:13 smithi110 ceph-mon[97633]: pgmap v7328: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:14.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:14.130197+0000) 2022-09-10T08:18:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:14.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:14.760712+0000) 2022-09-10T08:18:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:15.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:15.130407+0000) 2022-09-10T08:18:16.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:15 smithi104 ceph-mon[97044]: pgmap v7329: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:16.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:15 smithi104 ceph-mon[97044]: osdmap e539: 12 total, 12 up, 12 in 2022-09-10T08:18:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:15.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:15.760873+0000) 2022-09-10T08:18:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:15 smithi110 ceph-mon[97633]: pgmap v7329: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:18:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:15 smithi110 ceph-mon[97633]: osdmap e539: 12 total, 12 up, 12 in 2022-09-10T08:18:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:16.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:16.130591+0000) 2022-09-10T08:18:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:16.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:16.761013+0000) 2022-09-10T08:18:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:17.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:17.130868+0000) 2022-09-10T08:18:18.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:17.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:17.761169+0000) 2022-09-10T08:18:18.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:17 smithi104 ceph-mon[97044]: pgmap v7331: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T08:18:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:17 smithi110 ceph-mon[97633]: pgmap v7331: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T08:18:18.552 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:18.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:18.131046+0000) 2022-09-10T08:18:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:18:18.551+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:18:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:18.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:18.761304+0000) 2022-09-10T08:18:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:19.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:19.131189+0000) 2022-09-10T08:18:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:19 smithi104 ceph-mon[97044]: pgmap v7332: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:18:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:19.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:19.761448+0000) 2022-09-10T08:18:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:19 smithi110 ceph-mon[97633]: pgmap v7332: 129 pgs: 129 active+clean; 8.5 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:18:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:20.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:20.131394+0000) 2022-09-10T08:18:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:20.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:20.761588+0000) 2022-09-10T08:18:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:21.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:21.131649+0000) 2022-09-10T08:18:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:21 smithi104 ceph-mon[97044]: pgmap v7333: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:18:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:21.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:21.761778+0000) 2022-09-10T08:18:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:21 smithi110 ceph-mon[97633]: pgmap v7333: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:18:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:22.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:22.131893+0000) 2022-09-10T08:18:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:22.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:22.761926+0000) 2022-09-10T08:18:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:23.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:23.132077+0000) 2022-09-10T08:18:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:23 smithi104 ceph-mon[97044]: pgmap v7334: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:18:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:23.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:23.762061+0000) 2022-09-10T08:18:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:23 smithi110 ceph-mon[97633]: pgmap v7334: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:18:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:24.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:24.132299+0000) 2022-09-10T08:18:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:24.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:24.762200+0000) 2022-09-10T08:18:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:25.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:25.132555+0000) 2022-09-10T08:18:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:25 smithi104 ceph-mon[97044]: pgmap v7335: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:18:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:25.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:25.762358+0000) 2022-09-10T08:18:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:25 smithi110 ceph-mon[97633]: pgmap v7335: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:18:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:26.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:26.132771+0000) 2022-09-10T08:18:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:26.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:26.762501+0000) 2022-09-10T08:18:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:27.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:27.132993+0000) 2022-09-10T08:18:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:27 smithi104 ceph-mon[97044]: pgmap v7336: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:18:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:27.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:27.762626+0000) 2022-09-10T08:18:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:27 smithi110 ceph-mon[97633]: pgmap v7336: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T08:18:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:28.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:28.133231+0000) 2022-09-10T08:18:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:28.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:28.762752+0000) 2022-09-10T08:18:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:29.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:29.133442+0000) 2022-09-10T08:18:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:29 smithi104 ceph-mon[97044]: pgmap v7337: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:29.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:29.762876+0000) 2022-09-10T08:18:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:29 smithi110 ceph-mon[97633]: pgmap v7337: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:30.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:30.133711+0000) 2022-09-10T08:18:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:30.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:30.763015+0000) 2022-09-10T08:18:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:31.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:31.133935+0000) 2022-09-10T08:18:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:31 smithi104 ceph-mon[97044]: pgmap v7338: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:31.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:31.763162+0000) 2022-09-10T08:18:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:31 smithi110 ceph-mon[97633]: pgmap v7338: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:32.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:32.134129+0000) 2022-09-10T08:18:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:32 smithi104 ceph-mon[97044]: pgmap v7339: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:32.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:32.763388+0000) 2022-09-10T08:18:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:32 smithi110 ceph-mon[97633]: pgmap v7339: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:33.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:33.134375+0000) 2022-09-10T08:18:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:33.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:33.763592+0000) 2022-09-10T08:18:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:18:33.566+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:18:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:34.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:34.134646+0000) 2022-09-10T08:18:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:34.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:34.763750+0000) 2022-09-10T08:18:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:35 smithi104 ceph-mon[97044]: pgmap v7340: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:35.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:35.134834+0000) 2022-09-10T08:18:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:35 smithi110 ceph-mon[97633]: pgmap v7340: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:35.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:35.763893+0000) 2022-09-10T08:18:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:36.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:36.135059+0000) 2022-09-10T08:18:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:36.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:36.764060+0000) 2022-09-10T08:18:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:37.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:37.135289+0000) 2022-09-10T08:18:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:37 smithi110 ceph-mon[97633]: pgmap v7341: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:37 smithi104 ceph-mon[97044]: pgmap v7341: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:37.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:37.764165+0000) 2022-09-10T08:18:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:38.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:38.135516+0000) 2022-09-10T08:18:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:38.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:38.764294+0000) 2022-09-10T08:18:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:39.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:39.135727+0000) 2022-09-10T08:18:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:39 smithi110 ceph-mon[97633]: pgmap v7342: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:39 smithi104 ceph-mon[97044]: pgmap v7342: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:39.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:39.764420+0000) 2022-09-10T08:18:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:40.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:40.135862+0000) 2022-09-10T08:18:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:40.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:40.764558+0000) 2022-09-10T08:18:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:41.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:41.136123+0000) 2022-09-10T08:18:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:41 smithi110 ceph-mon[97633]: pgmap v7343: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:41 smithi104 ceph-mon[97044]: pgmap v7343: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:41.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:41.764689+0000) 2022-09-10T08:18:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:42.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:42.136313+0000) 2022-09-10T08:18:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:42.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:42.764892+0000) 2022-09-10T08:18:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:43.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:43.136503+0000) 2022-09-10T08:18:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:43 smithi110 ceph-mon[97633]: pgmap v7344: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:43 smithi104 ceph-mon[97044]: pgmap v7344: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:43.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:43.765075+0000) 2022-09-10T08:18:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:44.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:44.136735+0000) 2022-09-10T08:18:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:18:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:18:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:18:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:18:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:18:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:18:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:18:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:18:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:44.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:44.765251+0000) 2022-09-10T08:18:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:45.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:45.136884+0000) 2022-09-10T08:18:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:45 smithi110 ceph-mon[97633]: pgmap v7345: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:45 smithi104 ceph-mon[97044]: pgmap v7345: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:45.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:45.765394+0000) 2022-09-10T08:18:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:46.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:46.137140+0000) 2022-09-10T08:18:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:46.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:46.765526+0000) 2022-09-10T08:18:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:47.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:47.137355+0000) 2022-09-10T08:18:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:47 smithi110 ceph-mon[97633]: pgmap v7346: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:47 smithi104 ceph-mon[97044]: pgmap v7346: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:47.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:47.765651+0000) 2022-09-10T08:18:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:48.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:48.137523+0000) 2022-09-10T08:18:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:48.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:48.765784+0000) 2022-09-10T08:18:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:18:48.581+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:18:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:49.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:49.137729+0000) 2022-09-10T08:18:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:49 smithi110 ceph-mon[97633]: pgmap v7347: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:49 smithi104 ceph-mon[97044]: pgmap v7347: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:18:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:49.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:49.765911+0000) 2022-09-10T08:18:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:50.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:50.137961+0000) 2022-09-10T08:18:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:50.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:50.766035+0000) 2022-09-10T08:18:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:51.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:51.138148+0000) 2022-09-10T08:18:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:51 smithi110 ceph-mon[97633]: pgmap v7348: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:51 smithi104 ceph-mon[97044]: pgmap v7348: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:51.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:51.766189+0000) 2022-09-10T08:18:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:52.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:52.138368+0000) 2022-09-10T08:18:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:52.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:52.766346+0000) 2022-09-10T08:18:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:53.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:53.138615+0000) 2022-09-10T08:18:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:53 smithi110 ceph-mon[97633]: pgmap v7349: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:53 smithi104 ceph-mon[97044]: pgmap v7349: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:53.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:53.766483+0000) 2022-09-10T08:18:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:54.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:54.138866+0000) 2022-09-10T08:18:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:54.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:54.766680+0000) 2022-09-10T08:18:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:55.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:55.139053+0000) 2022-09-10T08:18:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:55 smithi110 ceph-mon[97633]: pgmap v7350: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:55 smithi104 ceph-mon[97044]: pgmap v7350: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:18:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:55.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:55.766853+0000) 2022-09-10T08:18:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:56.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:56.139334+0000) 2022-09-10T08:18:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:56.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:56.767058+0000) 2022-09-10T08:18:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:57.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:57.139521+0000) 2022-09-10T08:18:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:57 smithi104 ceph-mon[97044]: pgmap v7351: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:57.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:57.767173+0000) 2022-09-10T08:18:58.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:57 smithi110 ceph-mon[97633]: pgmap v7351: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:18:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:58.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:58.139746+0000) 2022-09-10T08:18:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:58.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:58.767312+0000) 2022-09-10T08:18:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:18:59.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:59.139978+0000) 2022-09-10T08:19:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:18:59 smithi104 ceph-mon[97044]: pgmap v7352: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:18:59.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:18:59.767438+0000) 2022-09-10T08:19:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:18:59 smithi110 ceph-mon[97633]: pgmap v7352: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:00.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:00.140229+0000) 2022-09-10T08:19:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:00.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:00.767561+0000) 2022-09-10T08:19:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:01.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:01.140409+0000) 2022-09-10T08:19:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:01 smithi104 ceph-mon[97044]: pgmap v7353: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:01.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:01.767702+0000) 2022-09-10T08:19:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:01 smithi110 ceph-mon[97633]: pgmap v7353: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:02.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:02.140608+0000) 2022-09-10T08:19:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:02.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:02.767838+0000) 2022-09-10T08:19:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:03.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:03.140812+0000) 2022-09-10T08:19:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:03 smithi104 ceph-mon[97044]: pgmap v7354: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:03.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:03.768040+0000) 2022-09-10T08:19:04.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:19:03.586+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:19:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:03 smithi110 ceph-mon[97633]: pgmap v7354: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:04.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:04.141001+0000) 2022-09-10T08:19:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:04.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:04.768169+0000) 2022-09-10T08:19:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:05.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:05.141274+0000) 2022-09-10T08:19:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:05 smithi104 ceph-mon[97044]: pgmap v7355: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:05.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:05.768308+0000) 2022-09-10T08:19:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:05 smithi110 ceph-mon[97633]: pgmap v7355: 129 pgs: 129 active+clean; 8.5 MiB data, 759 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:06.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:06.141484+0000) 2022-09-10T08:19:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:06.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:06.768448+0000) 2022-09-10T08:19:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:07.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:07.141649+0000) 2022-09-10T08:19:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:07 smithi104 ceph-mon[97044]: pgmap v7356: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:19:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:07.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:07.768589+0000) 2022-09-10T08:19:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:07 smithi110 ceph-mon[97633]: pgmap v7356: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:19:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:08.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:08.141862+0000) 2022-09-10T08:19:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:08 smithi104 ceph-mon[97044]: osdmap e540: 12 total, 12 up, 12 in 2022-09-10T08:19:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:08.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:08.768713+0000) 2022-09-10T08:19:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:08 smithi110 ceph-mon[97633]: osdmap e540: 12 total, 12 up, 12 in 2022-09-10T08:19:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:09.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:09.142087+0000) 2022-09-10T08:19:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:09 smithi104 ceph-mon[97044]: pgmap v7358: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:09.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:09.768844+0000) 2022-09-10T08:19:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:09 smithi110 ceph-mon[97633]: pgmap v7358: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:10.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:10.142361+0000) 2022-09-10T08:19:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:10.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:10.768971+0000) 2022-09-10T08:19:11.564 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:11.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:11.142570+0000) 2022-09-10T08:19:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:11 smithi104 ceph-mon[97044]: pgmap v7359: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:11.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:11.769124+0000) 2022-09-10T08:19:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:11 smithi110 ceph-mon[97633]: pgmap v7359: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:12.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:12.142833+0000) 2022-09-10T08:19:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:12.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:12.769246+0000) 2022-09-10T08:19:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:13.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:13.143063+0000) 2022-09-10T08:19:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:13 smithi104 ceph-mon[97044]: pgmap v7360: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:13.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:13.769371+0000) 2022-09-10T08:19:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:13 smithi110 ceph-mon[97633]: pgmap v7360: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:14.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:14.143343+0000) 2022-09-10T08:19:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:14.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:14.769498+0000) 2022-09-10T08:19:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:15.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:15.143500+0000) 2022-09-10T08:19:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:15 smithi104 ceph-mon[97044]: pgmap v7361: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:15 smithi104 ceph-mon[97044]: osdmap e541: 12 total, 12 up, 12 in 2022-09-10T08:19:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:15.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:15.769640+0000) 2022-09-10T08:19:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:15 smithi110 ceph-mon[97633]: pgmap v7361: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:19:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:15 smithi110 ceph-mon[97633]: osdmap e541: 12 total, 12 up, 12 in 2022-09-10T08:19:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:16.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:16.143657+0000) 2022-09-10T08:19:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:16.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:16.769761+0000) 2022-09-10T08:19:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:17.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:17.143879+0000) 2022-09-10T08:19:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:17 smithi104 ceph-mon[97044]: pgmap v7363: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:19:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:17.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:17.769897+0000) 2022-09-10T08:19:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:17 smithi110 ceph-mon[97633]: pgmap v7363: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:19:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:18.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:18.144143+0000) 2022-09-10T08:19:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:19:18.601+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:19:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:18.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:18.770033+0000) 2022-09-10T08:19:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:19.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:19.144378+0000) 2022-09-10T08:19:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:19 smithi104 ceph-mon[97044]: pgmap v7364: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:19:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:19.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:19.770167+0000) 2022-09-10T08:19:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:19 smithi110 ceph-mon[97633]: pgmap v7364: 129 pgs: 129 active+clean; 8.5 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:19:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:20.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:20.144593+0000) 2022-09-10T08:19:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:20.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:20.770296+0000) 2022-09-10T08:19:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:21.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:21.144822+0000) 2022-09-10T08:19:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:21 smithi104 ceph-mon[97044]: pgmap v7365: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:19:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:21.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:21.770479+0000) 2022-09-10T08:19:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:21 smithi110 ceph-mon[97633]: pgmap v7365: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:19:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:22.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:22.145061+0000) 2022-09-10T08:19:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:22.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:22.770607+0000) 2022-09-10T08:19:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:23.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:23.145337+0000) 2022-09-10T08:19:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:23 smithi104 ceph-mon[97044]: pgmap v7366: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:19:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:23.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:23.770726+0000) 2022-09-10T08:19:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:23 smithi110 ceph-mon[97633]: pgmap v7366: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:19:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:24.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:24.145539+0000) 2022-09-10T08:19:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:24.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:24.770869+0000) 2022-09-10T08:19:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:25.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:25.145775+0000) 2022-09-10T08:19:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:25 smithi104 ceph-mon[97044]: pgmap v7367: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:19:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:25.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:25.771042+0000) 2022-09-10T08:19:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:25 smithi110 ceph-mon[97633]: pgmap v7367: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:19:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:26.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:26.145977+0000) 2022-09-10T08:19:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:26.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:26.771166+0000) 2022-09-10T08:19:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:27.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:27.146193+0000) 2022-09-10T08:19:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:27 smithi104 ceph-mon[97044]: pgmap v7368: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:19:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:27.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:27.771313+0000) 2022-09-10T08:19:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:27 smithi110 ceph-mon[97633]: pgmap v7368: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:19:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:28.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:28.146380+0000) 2022-09-10T08:19:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:28.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:28.771463+0000) 2022-09-10T08:19:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:29.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:29.146633+0000) 2022-09-10T08:19:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:29 smithi104 ceph-mon[97044]: pgmap v7369: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:29.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:29.771602+0000) 2022-09-10T08:19:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:29 smithi110 ceph-mon[97633]: pgmap v7369: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:30.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:30.146807+0000) 2022-09-10T08:19:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:30.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:30.771810+0000) 2022-09-10T08:19:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:31.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:31.147006+0000) 2022-09-10T08:19:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:31 smithi104 ceph-mon[97044]: pgmap v7370: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:31.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:31.771944+0000) 2022-09-10T08:19:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:31 smithi110 ceph-mon[97633]: pgmap v7370: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:32.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:32.147241+0000) 2022-09-10T08:19:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:32.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:32.772061+0000) 2022-09-10T08:19:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:33.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:33.147375+0000) 2022-09-10T08:19:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:33 smithi104 ceph-mon[97044]: pgmap v7371: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:33.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:33.772174+0000) 2022-09-10T08:19:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:19:33.607+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:19:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:33 smithi110 ceph-mon[97633]: pgmap v7371: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:34.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:34.147566+0000) 2022-09-10T08:19:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:34.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:34.772323+0000) 2022-09-10T08:19:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:35.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:35.147722+0000) 2022-09-10T08:19:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:35 smithi104 ceph-mon[97044]: pgmap v7372: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:35.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:35.772511+0000) 2022-09-10T08:19:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:35 smithi110 ceph-mon[97633]: pgmap v7372: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:36.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:36.147979+0000) 2022-09-10T08:19:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:36.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:36.772706+0000) 2022-09-10T08:19:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:37.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:37.148226+0000) 2022-09-10T08:19:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:37 smithi104 ceph-mon[97044]: pgmap v7373: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:37.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:37.772881+0000) 2022-09-10T08:19:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:37 smithi110 ceph-mon[97633]: pgmap v7373: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:38.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:38.148354+0000) 2022-09-10T08:19:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:38.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:38.773055+0000) 2022-09-10T08:19:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:39.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:39.148559+0000) 2022-09-10T08:19:40.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:39 smithi104 ceph-mon[97044]: pgmap v7374: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:39.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:39.773173+0000) 2022-09-10T08:19:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:39 smithi110 ceph-mon[97633]: pgmap v7374: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:40.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:40.148753+0000) 2022-09-10T08:19:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:40.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:40.773334+0000) 2022-09-10T08:19:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:41.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:41.148987+0000) 2022-09-10T08:19:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:41 smithi104 ceph-mon[97044]: pgmap v7375: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:41.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:41.773476+0000) 2022-09-10T08:19:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:41 smithi110 ceph-mon[97633]: pgmap v7375: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:42.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:42.149246+0000) 2022-09-10T08:19:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:42.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:42.773605+0000) 2022-09-10T08:19:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:43.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:43.149446+0000) 2022-09-10T08:19:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:43 smithi104 ceph-mon[97044]: pgmap v7376: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:43.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:43.773752+0000) 2022-09-10T08:19:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:43 smithi110 ceph-mon[97633]: pgmap v7376: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:44.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:44.149716+0000) 2022-09-10T08:19:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:19:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:19:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:19:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:19:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:44.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:44.773865+0000) 2022-09-10T08:19:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:19:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:19:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:19:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:19:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:45.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:45.149873+0000) 2022-09-10T08:19:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:45 smithi104 ceph-mon[97044]: pgmap v7377: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:45.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:45.773997+0000) 2022-09-10T08:19:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:45 smithi110 ceph-mon[97633]: pgmap v7377: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:46.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:46.150020+0000) 2022-09-10T08:19:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:46.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:46.774135+0000) 2022-09-10T08:19:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:47.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:47.150230+0000) 2022-09-10T08:19:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:47 smithi104 ceph-mon[97044]: pgmap v7378: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:47.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:47.774260+0000) 2022-09-10T08:19:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:47 smithi110 ceph-mon[97633]: pgmap v7378: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:48.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:48.150391+0000) 2022-09-10T08:19:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:19:48.622+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:19:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:48.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:48.774377+0000) 2022-09-10T08:19:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:49.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:49.150593+0000) 2022-09-10T08:19:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:49 smithi104 ceph-mon[97044]: pgmap v7379: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:49.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:49.774513+0000) 2022-09-10T08:19:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:49 smithi110 ceph-mon[97633]: pgmap v7379: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:50.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:50.150832+0000) 2022-09-10T08:19:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:50.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:50.774675+0000) 2022-09-10T08:19:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:51.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:51.150989+0000) 2022-09-10T08:19:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:51 smithi104 ceph-mon[97044]: pgmap v7380: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:51.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:51.774828+0000) 2022-09-10T08:19:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:51 smithi110 ceph-mon[97633]: pgmap v7380: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:52.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:52.151238+0000) 2022-09-10T08:19:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:52.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:52.774955+0000) 2022-09-10T08:19:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:53.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:53.151380+0000) 2022-09-10T08:19:54.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:53 smithi104 ceph-mon[97044]: pgmap v7381: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:53.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:53.775092+0000) 2022-09-10T08:19:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:53 smithi110 ceph-mon[97633]: pgmap v7381: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:54.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:54.151543+0000) 2022-09-10T08:19:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:54.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:54.775258+0000) 2022-09-10T08:19:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:55.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:55.151761+0000) 2022-09-10T08:19:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:55 smithi104 ceph-mon[97044]: pgmap v7382: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:55.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:55.775434+0000) 2022-09-10T08:19:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:55 smithi110 ceph-mon[97633]: pgmap v7382: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:19:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:56.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:56.151910+0000) 2022-09-10T08:19:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:56.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:56.775607+0000) 2022-09-10T08:19:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:57.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:57.152143+0000) 2022-09-10T08:19:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:57 smithi104 ceph-mon[97044]: pgmap v7383: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:57.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:57.775736+0000) 2022-09-10T08:19:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:57 smithi110 ceph-mon[97633]: pgmap v7383: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:19:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:58.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:58.152351+0000) 2022-09-10T08:19:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:58.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:58.775877+0000) 2022-09-10T08:19:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:19:59.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:59.152631+0000) 2022-09-10T08:20:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:19:59 smithi104 ceph-mon[97044]: pgmap v7384: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:19:59.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:19:59.776035+0000) 2022-09-10T08:20:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:19:59 smithi110 ceph-mon[97633]: pgmap v7384: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:00.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:00.152822+0000) 2022-09-10T08:20:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:20:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:00.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:00.776150+0000) 2022-09-10T08:20:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:20:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:01.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:01.153050+0000) 2022-09-10T08:20:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:01 smithi104 ceph-mon[97044]: pgmap v7385: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:01.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:01.776277+0000) 2022-09-10T08:20:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:01 smithi110 ceph-mon[97633]: pgmap v7385: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:02.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:02.153228+0000) 2022-09-10T08:20:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:02.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:02.776449+0000) 2022-09-10T08:20:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:03.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:03.153377+0000) 2022-09-10T08:20:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:03 smithi104 ceph-mon[97044]: pgmap v7386: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:04.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:20:03.638+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:20:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:03.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:03.776569+0000) 2022-09-10T08:20:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:03 smithi110 ceph-mon[97633]: pgmap v7386: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:04.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:04.153534+0000) 2022-09-10T08:20:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:04.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:04.776695+0000) 2022-09-10T08:20:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:05.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:05.153677+0000) 2022-09-10T08:20:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:05 smithi104 ceph-mon[97044]: pgmap v7387: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:05.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:05.776835+0000) 2022-09-10T08:20:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:05 smithi110 ceph-mon[97633]: pgmap v7387: 129 pgs: 129 active+clean; 8.5 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:06.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:06.153843+0000) 2022-09-10T08:20:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:06 smithi104 ceph-mon[97044]: pgmap v7388: 129 pgs: 129 active+clean; 8.5 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:20:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:06.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:06.776961+0000) 2022-09-10T08:20:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:06 smithi110 ceph-mon[97633]: pgmap v7388: 129 pgs: 129 active+clean; 8.5 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:20:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:07.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:07.154062+0000) 2022-09-10T08:20:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:07.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:07.777087+0000) 2022-09-10T08:20:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:08.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:08.154290+0000) 2022-09-10T08:20:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:08 smithi104 ceph-mon[97044]: osdmap e542: 12 total, 12 up, 12 in 2022-09-10T08:20:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:08 smithi104 ceph-mon[97044]: pgmap v7390: 129 pgs: 129 active+clean; 8.5 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:20:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:08.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:08.777309+0000) 2022-09-10T08:20:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:08 smithi110 ceph-mon[97633]: osdmap e542: 12 total, 12 up, 12 in 2022-09-10T08:20:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:08 smithi110 ceph-mon[97633]: pgmap v7390: 129 pgs: 129 active+clean; 8.5 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:20:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:09.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:09.154465+0000) 2022-09-10T08:20:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:09.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:09.777488+0000) 2022-09-10T08:20:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:10.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:10.154611+0000) 2022-09-10T08:20:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:10.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:10.777644+0000) 2022-09-10T08:20:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:11.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:11.154774+0000) 2022-09-10T08:20:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:11 smithi110 ceph-mon[97633]: pgmap v7391: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:20:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:11 smithi110 ceph-mon[97633]: osdmap e543: 12 total, 12 up, 12 in 2022-09-10T08:20:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:11 smithi104 ceph-mon[97044]: pgmap v7391: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:20:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:11 smithi104 ceph-mon[97044]: osdmap e543: 12 total, 12 up, 12 in 2022-09-10T08:20:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:11.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:11.777778+0000) 2022-09-10T08:20:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:12.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:12.154924+0000) 2022-09-10T08:20:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:12.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:12.777909+0000) 2022-09-10T08:20:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:13.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:13.155147+0000) 2022-09-10T08:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:13 smithi110 ceph-mon[97633]: pgmap v7393: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T08:20:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:13 smithi104 ceph-mon[97044]: pgmap v7393: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T08:20:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:13.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:13.778072+0000) 2022-09-10T08:20:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:14.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:14.155377+0000) 2022-09-10T08:20:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:14.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:14.778288+0000) 2022-09-10T08:20:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:15.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:15.155536+0000) 2022-09-10T08:20:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:15 smithi110 ceph-mon[97633]: pgmap v7394: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:20:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:15 smithi104 ceph-mon[97044]: pgmap v7394: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:20:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:15.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:15.778438+0000) 2022-09-10T08:20:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:16.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:16.155724+0000) 2022-09-10T08:20:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:16.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:16.778566+0000) 2022-09-10T08:20:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:17.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:17.155969+0000) 2022-09-10T08:20:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:17 smithi110 ceph-mon[97633]: pgmap v7395: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:20:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:17 smithi104 ceph-mon[97044]: pgmap v7395: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T08:20:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:17.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:17.778763+0000) 2022-09-10T08:20:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:18.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:18.156232+0000) 2022-09-10T08:20:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:18.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:18.778948+0000) 2022-09-10T08:20:19.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:20:18.650+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:20:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:19.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:19.156438+0000) 2022-09-10T08:20:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:19 smithi104 ceph-mon[97044]: pgmap v7396: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:20:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:19.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:19.779148+0000) 2022-09-10T08:20:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:19 smithi110 ceph-mon[97633]: pgmap v7396: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:20:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:20.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:20.156619+0000) 2022-09-10T08:20:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:20.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:20.779281+0000) 2022-09-10T08:20:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:21.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:21.156817+0000) 2022-09-10T08:20:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:21 smithi104 ceph-mon[97044]: pgmap v7397: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:20:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:21.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:21.779407+0000) 2022-09-10T08:20:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:21 smithi110 ceph-mon[97633]: pgmap v7397: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:20:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:22.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:22.157030+0000) 2022-09-10T08:20:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:22.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:22.779533+0000) 2022-09-10T08:20:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:23.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:23.157194+0000) 2022-09-10T08:20:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:23 smithi104 ceph-mon[97044]: pgmap v7398: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:20:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:23.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:23.779655+0000) 2022-09-10T08:20:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:23 smithi110 ceph-mon[97633]: pgmap v7398: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:20:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:24.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:24.157353+0000) 2022-09-10T08:20:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:24.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:24.779810+0000) 2022-09-10T08:20:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:25.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:25.157546+0000) 2022-09-10T08:20:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:25 smithi104 ceph-mon[97044]: pgmap v7399: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T08:20:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:25.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:25.779958+0000) 2022-09-10T08:20:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:25 smithi110 ceph-mon[97633]: pgmap v7399: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T08:20:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:26.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:26.157784+0000) 2022-09-10T08:20:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:26.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:26.780113+0000) 2022-09-10T08:20:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:27.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:27.157975+0000) 2022-09-10T08:20:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:27 smithi104 ceph-mon[97044]: pgmap v7400: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:20:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:27.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:27.780247+0000) 2022-09-10T08:20:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:27 smithi110 ceph-mon[97633]: pgmap v7400: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:20:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:28.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:28.158237+0000) 2022-09-10T08:20:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:28.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:28.780357+0000) 2022-09-10T08:20:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:29.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:29.158489+0000) 2022-09-10T08:20:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:29 smithi104 ceph-mon[97044]: pgmap v7401: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:29.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:29.780481+0000) 2022-09-10T08:20:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:29 smithi110 ceph-mon[97633]: pgmap v7401: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:30.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:30.158722+0000) 2022-09-10T08:20:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:30.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:30.780630+0000) 2022-09-10T08:20:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:31.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:31.158978+0000) 2022-09-10T08:20:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:31 smithi104 ceph-mon[97044]: pgmap v7402: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:31.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:31.780769+0000) 2022-09-10T08:20:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:31 smithi110 ceph-mon[97633]: pgmap v7402: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:32.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:32.159264+0000) 2022-09-10T08:20:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:32.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:32.780950+0000) 2022-09-10T08:20:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:33.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:33.159504+0000) 2022-09-10T08:20:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:33 smithi104 ceph-mon[97044]: pgmap v7403: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:20:33.664+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:20:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:33.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:33.781129+0000) 2022-09-10T08:20:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:33 smithi110 ceph-mon[97633]: pgmap v7403: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:34.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:34.159731+0000) 2022-09-10T08:20:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:34.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:34.781282+0000) 2022-09-10T08:20:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:35.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:35.159953+0000) 2022-09-10T08:20:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:35 smithi104 ceph-mon[97044]: pgmap v7404: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:35.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:35.781481+0000) 2022-09-10T08:20:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:35 smithi110 ceph-mon[97633]: pgmap v7404: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:36.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:36.160152+0000) 2022-09-10T08:20:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:36.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:36.781635+0000) 2022-09-10T08:20:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:37.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:37.160384+0000) 2022-09-10T08:20:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:37 smithi104 ceph-mon[97044]: pgmap v7405: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:37.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:37.781791+0000) 2022-09-10T08:20:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:37 smithi110 ceph-mon[97633]: pgmap v7405: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:38.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:38.160649+0000) 2022-09-10T08:20:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:38.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:38.781927+0000) 2022-09-10T08:20:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:39.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:39.160841+0000) 2022-09-10T08:20:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:39 smithi104 ceph-mon[97044]: pgmap v7406: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:39.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:39.782059+0000) 2022-09-10T08:20:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:39 smithi110 ceph-mon[97633]: pgmap v7406: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:40.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:40.161008+0000) 2022-09-10T08:20:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:40.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:40.782169+0000) 2022-09-10T08:20:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:41.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:41.161189+0000) 2022-09-10T08:20:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:41 smithi104 ceph-mon[97044]: pgmap v7407: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:41.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:41.782344+0000) 2022-09-10T08:20:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:41 smithi110 ceph-mon[97633]: pgmap v7407: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:42.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:42.161401+0000) 2022-09-10T08:20:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:42.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:42.782474+0000) 2022-09-10T08:20:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:43.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:43.161575+0000) 2022-09-10T08:20:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:43 smithi104 ceph-mon[97044]: pgmap v7408: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:20:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:43.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:43.782603+0000) 2022-09-10T08:20:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:43 smithi110 ceph-mon[97633]: pgmap v7408: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:20:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:44.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:44.161808+0000) 2022-09-10T08:20:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:20:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:20:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:20:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:20:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:44.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:44.782811+0000) 2022-09-10T08:20:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:20:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:20:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:20:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:20:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:45.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:45.162008+0000) 2022-09-10T08:20:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:45 smithi104 ceph-mon[97044]: pgmap v7409: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:20:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:45.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:45.782946+0000) 2022-09-10T08:20:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:45 smithi110 ceph-mon[97633]: pgmap v7409: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:20:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:46.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:46.162258+0000) 2022-09-10T08:20:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:46.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:46.783108+0000) 2022-09-10T08:20:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:47.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:47.162461+0000) 2022-09-10T08:20:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:47 smithi104 ceph-mon[97044]: pgmap v7410: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:47.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:47.783254+0000) 2022-09-10T08:20:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:47 smithi110 ceph-mon[97633]: pgmap v7410: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:48.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:48.162676+0000) 2022-09-10T08:20:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:48.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:48.783382+0000) 2022-09-10T08:20:49.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:20:48.679+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:20:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:49.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:49.162888+0000) 2022-09-10T08:20:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:49 smithi104 ceph-mon[97044]: pgmap v7411: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:20:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:49.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:49.783557+0000) 2022-09-10T08:20:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:49 smithi110 ceph-mon[97633]: pgmap v7411: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:20:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:50.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:50.163177+0000) 2022-09-10T08:20:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:50.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:50.783693+0000) 2022-09-10T08:20:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:51.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:51.163343+0000) 2022-09-10T08:20:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:51 smithi104 ceph-mon[97044]: pgmap v7412: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:51.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:51.783882+0000) 2022-09-10T08:20:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:51 smithi110 ceph-mon[97633]: pgmap v7412: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:52.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:52.163548+0000) 2022-09-10T08:20:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:52.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:52.784053+0000) 2022-09-10T08:20:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:53.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:53.163769+0000) 2022-09-10T08:20:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:53 smithi104 ceph-mon[97044]: pgmap v7413: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:53.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:53.784215+0000) 2022-09-10T08:20:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:53 smithi110 ceph-mon[97633]: pgmap v7413: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:54.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:54.163966+0000) 2022-09-10T08:20:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:54.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:54.784352+0000) 2022-09-10T08:20:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:55.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:55.164175+0000) 2022-09-10T08:20:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:55 smithi104 ceph-mon[97044]: pgmap v7414: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:55.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:55.784506+0000) 2022-09-10T08:20:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:55 smithi110 ceph-mon[97633]: pgmap v7414: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:20:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:56.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:56.164395+0000) 2022-09-10T08:20:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:56.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:56.784631+0000) 2022-09-10T08:20:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:57.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:57.164595+0000) 2022-09-10T08:20:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:57 smithi104 ceph-mon[97044]: pgmap v7415: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:57.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:57.784788+0000) 2022-09-10T08:20:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:57 smithi110 ceph-mon[97633]: pgmap v7415: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:20:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:58.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:58.164838+0000) 2022-09-10T08:20:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:58.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:58.784919+0000) 2022-09-10T08:20:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:20:59.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:59.165042+0000) 2022-09-10T08:21:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:20:59 smithi104 ceph-mon[97044]: pgmap v7416: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:20:59.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:20:59.785046+0000) 2022-09-10T08:21:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:20:59 smithi110 ceph-mon[97633]: pgmap v7416: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:00.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:00.165214+0000) 2022-09-10T08:21:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:00.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:00.785147+0000) 2022-09-10T08:21:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:01.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:01.165381+0000) 2022-09-10T08:21:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:01 smithi104 ceph-mon[97044]: pgmap v7417: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:01.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:01.785294+0000) 2022-09-10T08:21:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:01 smithi110 ceph-mon[97633]: pgmap v7417: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:02.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:02.165590+0000) 2022-09-10T08:21:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:02.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:02.785423+0000) 2022-09-10T08:21:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:03.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:03.165775+0000) 2022-09-10T08:21:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:03 smithi104 ceph-mon[97044]: pgmap v7418: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:03.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:03.785550+0000) 2022-09-10T08:21:04.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:21:03.693+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:21:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:03 smithi110 ceph-mon[97633]: pgmap v7418: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:04.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:04.165999+0000) 2022-09-10T08:21:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:04.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:04.785684+0000) 2022-09-10T08:21:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:05.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:05.166232+0000) 2022-09-10T08:21:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:05 smithi104 ceph-mon[97044]: pgmap v7419: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:05.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:05.785811+0000) 2022-09-10T08:21:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:05 smithi110 ceph-mon[97633]: pgmap v7419: 129 pgs: 129 active+clean; 8.5 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:06.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:06.166516+0000) 2022-09-10T08:21:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:06.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:06.785929+0000) 2022-09-10T08:21:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:07.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:07.166757+0000) 2022-09-10T08:21:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:07 smithi104 ceph-mon[97044]: pgmap v7420: 129 pgs: 129 active+clean; 8.5 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:21:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:07.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:07.786115+0000) 2022-09-10T08:21:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:07 smithi110 ceph-mon[97633]: pgmap v7420: 129 pgs: 129 active+clean; 8.5 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:21:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:08.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:08.167015+0000) 2022-09-10T08:21:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:08 smithi104 ceph-mon[97044]: osdmap e544: 12 total, 12 up, 12 in 2022-09-10T08:21:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:08.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:08.786266+0000) 2022-09-10T08:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:08 smithi110 ceph-mon[97633]: osdmap e544: 12 total, 12 up, 12 in 2022-09-10T08:21:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:09.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:09.167278+0000) 2022-09-10T08:21:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:09 smithi104 ceph-mon[97044]: pgmap v7422: 129 pgs: 129 active+clean; 8.5 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:21:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:09.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:09.786425+0000) 2022-09-10T08:21:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:09 smithi110 ceph-mon[97633]: pgmap v7422: 129 pgs: 129 active+clean; 8.5 MiB data, 765 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:21:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:10.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:10.167476+0000) 2022-09-10T08:21:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:10.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:10.786567+0000) 2022-09-10T08:21:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:11.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:11.167725+0000) 2022-09-10T08:21:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:11 smithi104 ceph-mon[97044]: pgmap v7423: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:21:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:11.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:11.786711+0000) 2022-09-10T08:21:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:11 smithi110 ceph-mon[97633]: pgmap v7423: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:21:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:12.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:12.167976+0000) 2022-09-10T08:21:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:12.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:12.786842+0000) 2022-09-10T08:21:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:13.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:13.168206+0000) 2022-09-10T08:21:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:13 smithi104 ceph-mon[97044]: pgmap v7424: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:21:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:13.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:13.786966+0000) 2022-09-10T08:21:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:13 smithi110 ceph-mon[97633]: pgmap v7424: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:21:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:14.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:14.168358+0000) 2022-09-10T08:21:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:14.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:14.787106+0000) 2022-09-10T08:21:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:15.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:15.168485+0000) 2022-09-10T08:21:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:15 smithi104 ceph-mon[97044]: pgmap v7425: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:21:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:15 smithi104 ceph-mon[97044]: osdmap e545: 12 total, 12 up, 12 in 2022-09-10T08:21:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:15.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:15.787299+0000) 2022-09-10T08:21:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:15 smithi110 ceph-mon[97633]: pgmap v7425: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:21:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:15 smithi110 ceph-mon[97633]: osdmap e545: 12 total, 12 up, 12 in 2022-09-10T08:21:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:16.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:16.168710+0000) 2022-09-10T08:21:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:16.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:16.787418+0000) 2022-09-10T08:21:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:17.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:17.168872+0000) 2022-09-10T08:21:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:17 smithi104 ceph-mon[97044]: pgmap v7427: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:21:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:17.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:17.787548+0000) 2022-09-10T08:21:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:17 smithi110 ceph-mon[97633]: pgmap v7427: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:21:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:18.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:18.169084+0000) 2022-09-10T08:21:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:21:18.707+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:21:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:18.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:18.787681+0000) 2022-09-10T08:21:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:19.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:19.169359+0000) 2022-09-10T08:21:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:19 smithi104 ceph-mon[97044]: pgmap v7428: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:21:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:19.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:19.787827+0000) 2022-09-10T08:21:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:19 smithi110 ceph-mon[97633]: pgmap v7428: 129 pgs: 129 active+clean; 8.5 MiB data, 769 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:21:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:20.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:20.169606+0000) 2022-09-10T08:21:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:20.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:20.787958+0000) 2022-09-10T08:21:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:21.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:21.169860+0000) 2022-09-10T08:21:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:21 smithi104 ceph-mon[97044]: pgmap v7429: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:21:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:21.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:21.788112+0000) 2022-09-10T08:21:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:21 smithi110 ceph-mon[97633]: pgmap v7429: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:21:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:22.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:22.170110+0000) 2022-09-10T08:21:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:22.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:22.788275+0000) 2022-09-10T08:21:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:23.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:23.170340+0000) 2022-09-10T08:21:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:23 smithi104 ceph-mon[97044]: pgmap v7430: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:21:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:23.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:23.788420+0000) 2022-09-10T08:21:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:23 smithi110 ceph-mon[97633]: pgmap v7430: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:21:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:24.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:24.170558+0000) 2022-09-10T08:21:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:24.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:24.788545+0000) 2022-09-10T08:21:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:25.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:25.170790+0000) 2022-09-10T08:21:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:25 smithi104 ceph-mon[97044]: pgmap v7431: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:21:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:25.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:25.788676+0000) 2022-09-10T08:21:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:25 smithi110 ceph-mon[97633]: pgmap v7431: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:21:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:26.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:26.170977+0000) 2022-09-10T08:21:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:26.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:26.788804+0000) 2022-09-10T08:21:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:27.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:27.171173+0000) 2022-09-10T08:21:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:27 smithi104 ceph-mon[97044]: pgmap v7432: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:21:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:27.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:27.788936+0000) 2022-09-10T08:21:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:27 smithi110 ceph-mon[97633]: pgmap v7432: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:21:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:28.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:28.171428+0000) 2022-09-10T08:21:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:28.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:28.789063+0000) 2022-09-10T08:21:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:29.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:29.171584+0000) 2022-09-10T08:21:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:29 smithi104 ceph-mon[97044]: pgmap v7433: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:29.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:29.789162+0000) 2022-09-10T08:21:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:29 smithi110 ceph-mon[97633]: pgmap v7433: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:30.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:30.171869+0000) 2022-09-10T08:21:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:30.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:30.789295+0000) 2022-09-10T08:21:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:31.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:31.172080+0000) 2022-09-10T08:21:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:31 smithi104 ceph-mon[97044]: pgmap v7434: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:31.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:31.789424+0000) 2022-09-10T08:21:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:31 smithi110 ceph-mon[97633]: pgmap v7434: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:32.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:32.172306+0000) 2022-09-10T08:21:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:32.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:32.789539+0000) 2022-09-10T08:21:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:33.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:33.172489+0000) 2022-09-10T08:21:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:33 smithi104 ceph-mon[97044]: pgmap v7435: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:21:33.710+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:21:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:33.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:33.789678+0000) 2022-09-10T08:21:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:33 smithi110 ceph-mon[97633]: pgmap v7435: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:34.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:34.172729+0000) 2022-09-10T08:21:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:34.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:34.789811+0000) 2022-09-10T08:21:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:35.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:35.172981+0000) 2022-09-10T08:21:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:35 smithi104 ceph-mon[97044]: pgmap v7436: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:35.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:35.790015+0000) 2022-09-10T08:21:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:35 smithi110 ceph-mon[97633]: pgmap v7436: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:36.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:36.173199+0000) 2022-09-10T08:21:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:36.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:36.790160+0000) 2022-09-10T08:21:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:37.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:37.173359+0000) 2022-09-10T08:21:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:37 smithi104 ceph-mon[97044]: pgmap v7437: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:37.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:37.790285+0000) 2022-09-10T08:21:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:37 smithi110 ceph-mon[97633]: pgmap v7437: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:38.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:38.173628+0000) 2022-09-10T08:21:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:38.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:38.790415+0000) 2022-09-10T08:21:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:39.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:39.173776+0000) 2022-09-10T08:21:40.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:39 smithi104 ceph-mon[97044]: pgmap v7438: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:39.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:39.790590+0000) 2022-09-10T08:21:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:39 smithi110 ceph-mon[97633]: pgmap v7438: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:40.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:40.174040+0000) 2022-09-10T08:21:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:40.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:40.790786+0000) 2022-09-10T08:21:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:41.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:41.174298+0000) 2022-09-10T08:21:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:41 smithi104 ceph-mon[97044]: pgmap v7439: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:41.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:41.790939+0000) 2022-09-10T08:21:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:41 smithi110 ceph-mon[97633]: pgmap v7439: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:42.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:42.174464+0000) 2022-09-10T08:21:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:42.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:42.791064+0000) 2022-09-10T08:21:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:43.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:43.174684+0000) 2022-09-10T08:21:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:43 smithi104 ceph-mon[97044]: pgmap v7440: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:43.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:43.791191+0000) 2022-09-10T08:21:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:43 smithi110 ceph-mon[97633]: pgmap v7440: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:44.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:44.174926+0000) 2022-09-10T08:21:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:44.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:44.791372+0000) 2022-09-10T08:21:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:21:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:21:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:21:45.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:21:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:21:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:21:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:21:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:21:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:45.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:45.175152+0000) 2022-09-10T08:21:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:45.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:45.791572+0000) 2022-09-10T08:21:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:45 smithi110 ceph-mon[97633]: pgmap v7441: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:45 smithi104 ceph-mon[97044]: pgmap v7441: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:46.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:46.175407+0000) 2022-09-10T08:21:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:46.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:46.791808+0000) 2022-09-10T08:21:47.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:46 smithi110 ceph-mon[97633]: pgmap v7442: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:46 smithi104 ceph-mon[97044]: pgmap v7442: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:47.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:47.175568+0000) 2022-09-10T08:21:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:47.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:47.791982+0000) 2022-09-10T08:21:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:48.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:48.175822+0000) 2022-09-10T08:21:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:21:48.715+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:21:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:48.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:48.792108+0000) 2022-09-10T08:21:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:49 smithi104 ceph-mon[97044]: pgmap v7443: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:49.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:49.176076+0000) 2022-09-10T08:21:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:49 smithi110 ceph-mon[97633]: pgmap v7443: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:49.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:49.792273+0000) 2022-09-10T08:21:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:50.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:50.176351+0000) 2022-09-10T08:21:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:50.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:50.792398+0000) 2022-09-10T08:21:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:51.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:51.176559+0000) 2022-09-10T08:21:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:51 smithi104 ceph-mon[97044]: pgmap v7444: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:51.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:51.792610+0000) 2022-09-10T08:21:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:51 smithi110 ceph-mon[97633]: pgmap v7444: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:52.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:52.176822+0000) 2022-09-10T08:21:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:52.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:52.792752+0000) 2022-09-10T08:21:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:53.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:53.177047+0000) 2022-09-10T08:21:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:53 smithi104 ceph-mon[97044]: pgmap v7445: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:53.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:53.792922+0000) 2022-09-10T08:21:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:53 smithi110 ceph-mon[97633]: pgmap v7445: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:54.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:54.177227+0000) 2022-09-10T08:21:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:54.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:54.793134+0000) 2022-09-10T08:21:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:55.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:55.177480+0000) 2022-09-10T08:21:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:55 smithi104 ceph-mon[97044]: pgmap v7446: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:55.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:55.793278+0000) 2022-09-10T08:21:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:55 smithi110 ceph-mon[97633]: pgmap v7446: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:21:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:56.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:56.177664+0000) 2022-09-10T08:21:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:56.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:56.793459+0000) 2022-09-10T08:21:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:57.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:57.177877+0000) 2022-09-10T08:21:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:57 smithi104 ceph-mon[97044]: pgmap v7447: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:57.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:57.793615+0000) 2022-09-10T08:21:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:57 smithi110 ceph-mon[97633]: pgmap v7447: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:21:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:58.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:58.178133+0000) 2022-09-10T08:21:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:58.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:58.793842+0000) 2022-09-10T08:21:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:21:59.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:59.178360+0000) 2022-09-10T08:22:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:21:59 smithi104 ceph-mon[97044]: pgmap v7448: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:21:59.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:21:59.793978+0000) 2022-09-10T08:22:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:21:59 smithi110 ceph-mon[97633]: pgmap v7448: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:00.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:00.178566+0000) 2022-09-10T08:22:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:00.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:00.794090+0000) 2022-09-10T08:22:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:01.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:01.178755+0000) 2022-09-10T08:22:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:01 smithi104 ceph-mon[97044]: pgmap v7449: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:01.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:01.794225+0000) 2022-09-10T08:22:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:01 smithi110 ceph-mon[97633]: pgmap v7449: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:02.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:02.178982+0000) 2022-09-10T08:22:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:02.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:02.794404+0000) 2022-09-10T08:22:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:03.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:03.179173+0000) 2022-09-10T08:22:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:03 smithi104 ceph-mon[97044]: pgmap v7450: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:03.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:03.794550+0000) 2022-09-10T08:22:04.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:22:03.731+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:22:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:03 smithi110 ceph-mon[97633]: pgmap v7450: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:04.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:04.179420+0000) 2022-09-10T08:22:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:04.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:04.794666+0000) 2022-09-10T08:22:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:05.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:05.179635+0000) 2022-09-10T08:22:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:05 smithi104 ceph-mon[97044]: pgmap v7451: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:05.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:05.794801+0000) 2022-09-10T08:22:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:05 smithi110 ceph-mon[97633]: pgmap v7451: 129 pgs: 129 active+clean; 8.5 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:06.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:06.179845+0000) 2022-09-10T08:22:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:06.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:06.794927+0000) 2022-09-10T08:22:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:07.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:07.180032+0000) 2022-09-10T08:22:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:07 smithi104 ceph-mon[97044]: pgmap v7452: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:22:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:07.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:07.795054+0000) 2022-09-10T08:22:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:07 smithi110 ceph-mon[97633]: pgmap v7452: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:22:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:08.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:08.180226+0000) 2022-09-10T08:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:08 smithi104 ceph-mon[97044]: osdmap e546: 12 total, 12 up, 12 in 2022-09-10T08:22:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:08.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:08.795157+0000) 2022-09-10T08:22:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:08 smithi110 ceph-mon[97633]: osdmap e546: 12 total, 12 up, 12 in 2022-09-10T08:22:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:09.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:09.180470+0000) 2022-09-10T08:22:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:09 smithi104 ceph-mon[97044]: pgmap v7454: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:22:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:09.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:09.795286+0000) 2022-09-10T08:22:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:09 smithi110 ceph-mon[97633]: pgmap v7454: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:22:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:10.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:10.180719+0000) 2022-09-10T08:22:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:10.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:10.795416+0000) 2022-09-10T08:22:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:11.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:11.180885+0000) 2022-09-10T08:22:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:11 smithi104 ceph-mon[97044]: pgmap v7455: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:22:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:11.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:11.795581+0000) 2022-09-10T08:22:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:11 smithi110 ceph-mon[97633]: pgmap v7455: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:22:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:12.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:12.181043+0000) 2022-09-10T08:22:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:12.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:12.795711+0000) 2022-09-10T08:22:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:13.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:13.181294+0000) 2022-09-10T08:22:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:13 smithi104 ceph-mon[97044]: pgmap v7456: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:22:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:13.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:13.795834+0000) 2022-09-10T08:22:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:13 smithi110 ceph-mon[97633]: pgmap v7456: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:22:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:14.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:14.181517+0000) 2022-09-10T08:22:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:14.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:14.795957+0000) 2022-09-10T08:22:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:15.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:15.181729+0000) 2022-09-10T08:22:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:15 smithi104 ceph-mon[97044]: pgmap v7457: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:22:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:15 smithi104 ceph-mon[97044]: osdmap e547: 12 total, 12 up, 12 in 2022-09-10T08:22:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:15.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:15.796084+0000) 2022-09-10T08:22:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:15 smithi110 ceph-mon[97633]: pgmap v7457: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:22:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:15 smithi110 ceph-mon[97633]: osdmap e547: 12 total, 12 up, 12 in 2022-09-10T08:22:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:16.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:16.181962+0000) 2022-09-10T08:22:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:16.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:16.796218+0000) 2022-09-10T08:22:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:17.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:17.182225+0000) 2022-09-10T08:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:17 smithi104 ceph-mon[97044]: pgmap v7459: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:22:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:17.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:17.796342+0000) 2022-09-10T08:22:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:17 smithi110 ceph-mon[97633]: pgmap v7459: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:22:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:18.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:18.182390+0000) 2022-09-10T08:22:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:22:18.741+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:22:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:18.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:18.796499+0000) 2022-09-10T08:22:19.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:19.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:19.182536+0000) 2022-09-10T08:22:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:19 smithi104 ceph-mon[97044]: pgmap v7460: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:22:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:19.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:19.796655+0000) 2022-09-10T08:22:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:19 smithi110 ceph-mon[97633]: pgmap v7460: 129 pgs: 129 active+clean; 8.5 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:22:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:20.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:20.182726+0000) 2022-09-10T08:22:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:20.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:20.796786+0000) 2022-09-10T08:22:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:21.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:21.182920+0000) 2022-09-10T08:22:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:21 smithi104 ceph-mon[97044]: pgmap v7461: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:22:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:21.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:21.796931+0000) 2022-09-10T08:22:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:21 smithi110 ceph-mon[97633]: pgmap v7461: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:22:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:22.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:22.183189+0000) 2022-09-10T08:22:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:22.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:22.797078+0000) 2022-09-10T08:22:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:23.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:23.183470+0000) 2022-09-10T08:22:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:23 smithi104 ceph-mon[97044]: pgmap v7462: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:22:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:23.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:23.797223+0000) 2022-09-10T08:22:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:23 smithi110 ceph-mon[97633]: pgmap v7462: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:22:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:24.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:24.183737+0000) 2022-09-10T08:22:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:24.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:24.797357+0000) 2022-09-10T08:22:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:25.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:25.183993+0000) 2022-09-10T08:22:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:25 smithi104 ceph-mon[97044]: pgmap v7463: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:22:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:25.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:25.797489+0000) 2022-09-10T08:22:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:25 smithi110 ceph-mon[97633]: pgmap v7463: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:22:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:26.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:26.184262+0000) 2022-09-10T08:22:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:26.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:26.797639+0000) 2022-09-10T08:22:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:27.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:27.184409+0000) 2022-09-10T08:22:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:27 smithi104 ceph-mon[97044]: pgmap v7464: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:22:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:27.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:27.797768+0000) 2022-09-10T08:22:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:27 smithi110 ceph-mon[97633]: pgmap v7464: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:22:28.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:28.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:28.184646+0000) 2022-09-10T08:22:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:28.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:28.797882+0000) 2022-09-10T08:22:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:29.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:29.184898+0000) 2022-09-10T08:22:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:29 smithi104 ceph-mon[97044]: pgmap v7465: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:29.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:29.798020+0000) 2022-09-10T08:22:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:29 smithi110 ceph-mon[97633]: pgmap v7465: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:30.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:30.185129+0000) 2022-09-10T08:22:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:30.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:30.798217+0000) 2022-09-10T08:22:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:31.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:31.185316+0000) 2022-09-10T08:22:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:31 smithi104 ceph-mon[97044]: pgmap v7466: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:31.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:31.798366+0000) 2022-09-10T08:22:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:31 smithi110 ceph-mon[97633]: pgmap v7466: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:32.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:32.185569+0000) 2022-09-10T08:22:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:32.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:32.798500+0000) 2022-09-10T08:22:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:33.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:33.185781+0000) 2022-09-10T08:22:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:33 smithi104 ceph-mon[97044]: pgmap v7467: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:33.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:33.798745+0000) 2022-09-10T08:22:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:22:33.754+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:22:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:33 smithi110 ceph-mon[97633]: pgmap v7467: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:34.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:34.185986+0000) 2022-09-10T08:22:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:34.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:34.798894+0000) 2022-09-10T08:22:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:35.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:35.186136+0000) 2022-09-10T08:22:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:35 smithi104 ceph-mon[97044]: pgmap v7468: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:35.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:35.799028+0000) 2022-09-10T08:22:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:35 smithi110 ceph-mon[97633]: pgmap v7468: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:36.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:36.186374+0000) 2022-09-10T08:22:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:36.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:36.799162+0000) 2022-09-10T08:22:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:37.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:37.186653+0000) 2022-09-10T08:22:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:37 smithi104 ceph-mon[97044]: pgmap v7469: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:37.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:37.799303+0000) 2022-09-10T08:22:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:37 smithi110 ceph-mon[97633]: pgmap v7469: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:38.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:38.186944+0000) 2022-09-10T08:22:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:38.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:38.799442+0000) 2022-09-10T08:22:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:39.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:39.187178+0000) 2022-09-10T08:22:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:39 smithi104 ceph-mon[97044]: pgmap v7470: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:39.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:39.799582+0000) 2022-09-10T08:22:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:39 smithi110 ceph-mon[97633]: pgmap v7470: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:40.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:40.187408+0000) 2022-09-10T08:22:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:40.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:40.799711+0000) 2022-09-10T08:22:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:41.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:41.187579+0000) 2022-09-10T08:22:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:41 smithi104 ceph-mon[97044]: pgmap v7471: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:41.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:41.799840+0000) 2022-09-10T08:22:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:41 smithi110 ceph-mon[97633]: pgmap v7471: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:42.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:42.187724+0000) 2022-09-10T08:22:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:42.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:42.799969+0000) 2022-09-10T08:22:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:43.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:43.187896+0000) 2022-09-10T08:22:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:43 smithi104 ceph-mon[97044]: pgmap v7472: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:43.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:43.800137+0000) 2022-09-10T08:22:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:43 smithi110 ceph-mon[97633]: pgmap v7472: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:44.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:44.188077+0000) 2022-09-10T08:22:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:22:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:22:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:22:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:22:45.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:44.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:44.800260+0000) 2022-09-10T08:22:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:22:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:22:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:22:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:22:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:45.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:45.188304+0000) 2022-09-10T08:22:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:45 smithi104 ceph-mon[97044]: pgmap v7473: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:45.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:45.800388+0000) 2022-09-10T08:22:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:45 smithi110 ceph-mon[97633]: pgmap v7473: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:46.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:46.188525+0000) 2022-09-10T08:22:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:46.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:46.800540+0000) 2022-09-10T08:22:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:47.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:47.188739+0000) 2022-09-10T08:22:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:47 smithi104 ceph-mon[97044]: pgmap v7474: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:47.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:47.800681+0000) 2022-09-10T08:22:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:47 smithi110 ceph-mon[97633]: pgmap v7474: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:48.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:48.188962+0000) 2022-09-10T08:22:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:48.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:48.800825+0000) 2022-09-10T08:22:49.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:22:48.760+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:22:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:49.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:49.189188+0000) 2022-09-10T08:22:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:49 smithi104 ceph-mon[97044]: pgmap v7475: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:49.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:49.800973+0000) 2022-09-10T08:22:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:49 smithi110 ceph-mon[97633]: pgmap v7475: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:50.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:50.189395+0000) 2022-09-10T08:22:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:50.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:50.801124+0000) 2022-09-10T08:22:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:51.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:51.189574+0000) 2022-09-10T08:22:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:51 smithi104 ceph-mon[97044]: pgmap v7476: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:51.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:51.801285+0000) 2022-09-10T08:22:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:51 smithi110 ceph-mon[97633]: pgmap v7476: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:52.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:52.189699+0000) 2022-09-10T08:22:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:52.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:52.801409+0000) 2022-09-10T08:22:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:53.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:53.189930+0000) 2022-09-10T08:22:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:53 smithi104 ceph-mon[97044]: pgmap v7477: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:53.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:53.801534+0000) 2022-09-10T08:22:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:53 smithi110 ceph-mon[97633]: pgmap v7477: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:54.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:54.190168+0000) 2022-09-10T08:22:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:54.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:54.801686+0000) 2022-09-10T08:22:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:55.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:55.190377+0000) 2022-09-10T08:22:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:55 smithi104 ceph-mon[97044]: pgmap v7478: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:55.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:55.801818+0000) 2022-09-10T08:22:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:55 smithi110 ceph-mon[97633]: pgmap v7478: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:22:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:56.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:56.190615+0000) 2022-09-10T08:22:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:56.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:56.801990+0000) 2022-09-10T08:22:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:57.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:57.190824+0000) 2022-09-10T08:22:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:57 smithi104 ceph-mon[97044]: pgmap v7479: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:57.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:57.802163+0000) 2022-09-10T08:22:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:57 smithi110 ceph-mon[97633]: pgmap v7479: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:22:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:58.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:58.191015+0000) 2022-09-10T08:22:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:58.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:58.802330+0000) 2022-09-10T08:22:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:22:59.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:59.191226+0000) 2022-09-10T08:23:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:22:59 smithi104 ceph-mon[97044]: pgmap v7480: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:22:59.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:22:59.802475+0000) 2022-09-10T08:23:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:22:59 smithi110 ceph-mon[97633]: pgmap v7480: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:00.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:00.191428+0000) 2022-09-10T08:23:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:00.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:00.802594+0000) 2022-09-10T08:23:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:01.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:01.191668+0000) 2022-09-10T08:23:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:01 smithi104 ceph-mon[97044]: pgmap v7481: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:01.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:01.802767+0000) 2022-09-10T08:23:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:01 smithi110 ceph-mon[97633]: pgmap v7481: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:02.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:02.191868+0000) 2022-09-10T08:23:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:02.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:02.802896+0000) 2022-09-10T08:23:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:03.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:03.192147+0000) 2022-09-10T08:23:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:03 smithi104 ceph-mon[97044]: pgmap v7482: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:04.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:23:03.762+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:23:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:03.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:03.803026+0000) 2022-09-10T08:23:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:03 smithi110 ceph-mon[97633]: pgmap v7482: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:04.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:04.192323+0000) 2022-09-10T08:23:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:04.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:04.803155+0000) 2022-09-10T08:23:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:05.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:05.192603+0000) 2022-09-10T08:23:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:05.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:05.803297+0000) 2022-09-10T08:23:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:05 smithi110 ceph-mon[97633]: pgmap v7483: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:05 smithi104 ceph-mon[97044]: pgmap v7483: 129 pgs: 129 active+clean; 8.5 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:06.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:06.192790+0000) 2022-09-10T08:23:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:06.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:06.803421+0000) 2022-09-10T08:23:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:07.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:07.192986+0000) 2022-09-10T08:23:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:07 smithi110 ceph-mon[97633]: pgmap v7484: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:23:08.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:07.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:07.803555+0000) 2022-09-10T08:23:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:07 smithi104 ceph-mon[97044]: pgmap v7484: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:23:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:08.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:08.193227+0000) 2022-09-10T08:23:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:08 smithi110 ceph-mon[97633]: osdmap e548: 12 total, 12 up, 12 in 2022-09-10T08:23:09.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:08.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:08.803678+0000) 2022-09-10T08:23:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:08 smithi104 ceph-mon[97044]: osdmap e548: 12 total, 12 up, 12 in 2022-09-10T08:23:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:09.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:09.193447+0000) 2022-09-10T08:23:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:09 smithi110 ceph-mon[97633]: pgmap v7486: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:23:10.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:09.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:09.803867+0000) 2022-09-10T08:23:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:09 smithi104 ceph-mon[97044]: pgmap v7486: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:23:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:10.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:10.193663+0000) 2022-09-10T08:23:11.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:10.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:10.804000+0000) 2022-09-10T08:23:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:11.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:11.193858+0000) 2022-09-10T08:23:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:11 smithi104 ceph-mon[97044]: pgmap v7487: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:23:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:11 smithi110 ceph-mon[97633]: pgmap v7487: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:23:12.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:11.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:11.804166+0000) 2022-09-10T08:23:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:12.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:12.194128+0000) 2022-09-10T08:23:13.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:12.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:12.804356+0000) 2022-09-10T08:23:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:13.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:13.194342+0000) 2022-09-10T08:23:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:13 smithi104 ceph-mon[97044]: pgmap v7488: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:23:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:13 smithi110 ceph-mon[97633]: pgmap v7488: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:23:14.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:13.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:13.804487+0000) 2022-09-10T08:23:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:14.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:14.194592+0000) 2022-09-10T08:23:15.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:14.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:14.804617+0000) 2022-09-10T08:23:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:15.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:15.194790+0000) 2022-09-10T08:23:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:15 smithi104 ceph-mon[97044]: pgmap v7489: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:23:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:15 smithi104 ceph-mon[97044]: osdmap e549: 12 total, 12 up, 12 in 2022-09-10T08:23:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:15 smithi110 ceph-mon[97633]: pgmap v7489: 129 pgs: 129 active+clean; 8.5 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:23:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:15 smithi110 ceph-mon[97633]: osdmap e549: 12 total, 12 up, 12 in 2022-09-10T08:23:16.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:15.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:15.804780+0000) 2022-09-10T08:23:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:16.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:16.195009+0000) 2022-09-10T08:23:17.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:16.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:16.804928+0000) 2022-09-10T08:23:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:17.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:17.195212+0000) 2022-09-10T08:23:18.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:17 smithi104 ceph-mon[97044]: pgmap v7491: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:23:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:17 smithi110 ceph-mon[97633]: pgmap v7491: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T08:23:18.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:17.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:17.805115+0000) 2022-09-10T08:23:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:18.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:18.195436+0000) 2022-09-10T08:23:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:23:18.776+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:23:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:18.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:18.805297+0000) 2022-09-10T08:23:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:19.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:19.195606+0000) 2022-09-10T08:23:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:19 smithi104 ceph-mon[97044]: pgmap v7492: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:23:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:19 smithi110 ceph-mon[97633]: pgmap v7492: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:23:20.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:19.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:19.805444+0000) 2022-09-10T08:23:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:20.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:20.195836+0000) 2022-09-10T08:23:21.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:20.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:20.805604+0000) 2022-09-10T08:23:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:21.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:21.196041+0000) 2022-09-10T08:23:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:21 smithi104 ceph-mon[97044]: pgmap v7493: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:23:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:21 smithi110 ceph-mon[97633]: pgmap v7493: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:23:22.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:21.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:21.805752+0000) 2022-09-10T08:23:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:22.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:22.196194+0000) 2022-09-10T08:23:23.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:22.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:22.805879+0000) 2022-09-10T08:23:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:23.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:23.196395+0000) 2022-09-10T08:23:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:23 smithi104 ceph-mon[97044]: pgmap v7494: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:23:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:23 smithi110 ceph-mon[97633]: pgmap v7494: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:23:24.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:23.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:23.806020+0000) 2022-09-10T08:23:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:24.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:24.196554+0000) 2022-09-10T08:23:25.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:24.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:24.806151+0000) 2022-09-10T08:23:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:25.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:25.196809+0000) 2022-09-10T08:23:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:25 smithi104 ceph-mon[97044]: pgmap v7495: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:23:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:25 smithi110 ceph-mon[97633]: pgmap v7495: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:23:26.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:25.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:25.806273+0000) 2022-09-10T08:23:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:26.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:26.197091+0000) 2022-09-10T08:23:27.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:26.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:26.806400+0000) 2022-09-10T08:23:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:27.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:27.197332+0000) 2022-09-10T08:23:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:27 smithi104 ceph-mon[97044]: pgmap v7496: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:23:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:27 smithi110 ceph-mon[97633]: pgmap v7496: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:23:28.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:27.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:27.806528+0000) 2022-09-10T08:23:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:28.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:28.197598+0000) 2022-09-10T08:23:29.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:28.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:28.806656+0000) 2022-09-10T08:23:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:29.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:29.197844+0000) 2022-09-10T08:23:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:29 smithi104 ceph-mon[97044]: pgmap v7497: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:29 smithi110 ceph-mon[97633]: pgmap v7497: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:30.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:29.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:29.806806+0000) 2022-09-10T08:23:30.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:30.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:30.198089+0000) 2022-09-10T08:23:31.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:30.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:30.806923+0000) 2022-09-10T08:23:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:31.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:31.198360+0000) 2022-09-10T08:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:31 smithi104 ceph-mon[97044]: pgmap v7498: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:32.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:31 smithi110 ceph-mon[97633]: pgmap v7498: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:32.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:31.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:31.807076+0000) 2022-09-10T08:23:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:32.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:32.198588+0000) 2022-09-10T08:23:33.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:32.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:32.807200+0000) 2022-09-10T08:23:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:33.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:33.198718+0000) 2022-09-10T08:23:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:33 smithi104 ceph-mon[97044]: pgmap v7499: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:33.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:33.807347+0000) 2022-09-10T08:23:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:23:33.780+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:23:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:33 smithi110 ceph-mon[97633]: pgmap v7499: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:34.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:34.198966+0000) 2022-09-10T08:23:35.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:34.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:34.807543+0000) 2022-09-10T08:23:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:35.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:35.199199+0000) 2022-09-10T08:23:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:35 smithi104 ceph-mon[97044]: pgmap v7500: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:35 smithi110 ceph-mon[97633]: pgmap v7500: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:36.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:35.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:35.807687+0000) 2022-09-10T08:23:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:36.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:36.199413+0000) 2022-09-10T08:23:37.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:36.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:36.807820+0000) 2022-09-10T08:23:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:37.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:37.199683+0000) 2022-09-10T08:23:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:37 smithi104 ceph-mon[97044]: pgmap v7501: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:37 smithi110 ceph-mon[97633]: pgmap v7501: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:38.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:37.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:37.807954+0000) 2022-09-10T08:23:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:38.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:38.199964+0000) 2022-09-10T08:23:39.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:38.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:38.808157+0000) 2022-09-10T08:23:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:39.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:39.200176+0000) 2022-09-10T08:23:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:39 smithi104 ceph-mon[97044]: pgmap v7502: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:40.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:39 smithi110 ceph-mon[97633]: pgmap v7502: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:40.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:39.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:39.808317+0000) 2022-09-10T08:23:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:40.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:40.200448+0000) 2022-09-10T08:23:41.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:40.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:40.808448+0000) 2022-09-10T08:23:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:41.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:41.200647+0000) 2022-09-10T08:23:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:41 smithi104 ceph-mon[97044]: pgmap v7503: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:41 smithi110 ceph-mon[97633]: pgmap v7503: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:42.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:41.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:41.808605+0000) 2022-09-10T08:23:42.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:42.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:42.200873+0000) 2022-09-10T08:23:43.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:42.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:42.808749+0000) 2022-09-10T08:23:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:43.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:43.201128+0000) 2022-09-10T08:23:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:43 smithi104 ceph-mon[97044]: pgmap v7504: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:43 smithi110 ceph-mon[97633]: pgmap v7504: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:44.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:43.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:43.808941+0000) 2022-09-10T08:23:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:44.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:44.201358+0000) 2022-09-10T08:23:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:23:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:23:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:23:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:23:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:23:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:23:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:23:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:23:45.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:44.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:44.809081+0000) 2022-09-10T08:23:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:45.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:45.201613+0000) 2022-09-10T08:23:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:45 smithi104 ceph-mon[97044]: pgmap v7505: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:45 smithi110 ceph-mon[97633]: pgmap v7505: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:46.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:45.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:45.809244+0000) 2022-09-10T08:23:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:46.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:46.201852+0000) 2022-09-10T08:23:47.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:46.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:46.809376+0000) 2022-09-10T08:23:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:47.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:47.202065+0000) 2022-09-10T08:23:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:47 smithi104 ceph-mon[97044]: pgmap v7506: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:47 smithi110 ceph-mon[97633]: pgmap v7506: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:48.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:47.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:47.809497+0000) 2022-09-10T08:23:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:48.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:48.202270+0000) 2022-09-10T08:23:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:48.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:48.809615+0000) 2022-09-10T08:23:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:23:48.789+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:23:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:49.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:49.202552+0000) 2022-09-10T08:23:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:49 smithi104 ceph-mon[97044]: pgmap v7507: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:49 smithi110 ceph-mon[97633]: pgmap v7507: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:50.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:49.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:49.809741+0000) 2022-09-10T08:23:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:50.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:50.202775+0000) 2022-09-10T08:23:51.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:50.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:50.809872+0000) 2022-09-10T08:23:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:51.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:51.202996+0000) 2022-09-10T08:23:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:51 smithi104 ceph-mon[97044]: pgmap v7508: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:51 smithi110 ceph-mon[97633]: pgmap v7508: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:52.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:51.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:51.810049+0000) 2022-09-10T08:23:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:52.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:52.203253+0000) 2022-09-10T08:23:53.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:52.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:52.810157+0000) 2022-09-10T08:23:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:53.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:53.203499+0000) 2022-09-10T08:23:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:53 smithi104 ceph-mon[97044]: pgmap v7509: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:53 smithi110 ceph-mon[97633]: pgmap v7509: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:54.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:53.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:53.810282+0000) 2022-09-10T08:23:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:54.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:54.203707+0000) 2022-09-10T08:23:55.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:54.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:54.810490+0000) 2022-09-10T08:23:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:55.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:55.203929+0000) 2022-09-10T08:23:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:55 smithi104 ceph-mon[97044]: pgmap v7510: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:55 smithi110 ceph-mon[97633]: pgmap v7510: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:23:56.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:55.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:55.810642+0000) 2022-09-10T08:23:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:56.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:56.204172+0000) 2022-09-10T08:23:57.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:56.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:56.810760+0000) 2022-09-10T08:23:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:57.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:57.204381+0000) 2022-09-10T08:23:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:57 smithi104 ceph-mon[97044]: pgmap v7511: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:57 smithi110 ceph-mon[97633]: pgmap v7511: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:23:58.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:57.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:57.810906+0000) 2022-09-10T08:23:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:58.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:58.204638+0000) 2022-09-10T08:23:59.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:58.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:58.811038+0000) 2022-09-10T08:23:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:23:59.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:59.204845+0000) 2022-09-10T08:24:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:23:59 smithi104 ceph-mon[97044]: pgmap v7512: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:23:59 smithi110 ceph-mon[97633]: pgmap v7512: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:00.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:23:59.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:23:59.811163+0000) 2022-09-10T08:24:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:00.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:00.205073+0000) 2022-09-10T08:24:01.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:00.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:00.811286+0000) 2022-09-10T08:24:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:01.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:01.205293+0000) 2022-09-10T08:24:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:01 smithi104 ceph-mon[97044]: pgmap v7513: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:01 smithi110 ceph-mon[97633]: pgmap v7513: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:02.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:01.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:01.811452+0000) 2022-09-10T08:24:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:02.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:02.205547+0000) 2022-09-10T08:24:03.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:02.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:02.811578+0000) 2022-09-10T08:24:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:03.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:03.205796+0000) 2022-09-10T08:24:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:03 smithi104 ceph-mon[97044]: pgmap v7514: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:03 smithi110 ceph-mon[97633]: pgmap v7514: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:04.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:03.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:03.811709+0000) 2022-09-10T08:24:04.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:24:03.805+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:24:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:04.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:04.205976+0000) 2022-09-10T08:24:05.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:04.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:04.811837+0000) 2022-09-10T08:24:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:05.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:05.206241+0000) 2022-09-10T08:24:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:05 smithi104 ceph-mon[97044]: pgmap v7515: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:05 smithi110 ceph-mon[97633]: pgmap v7515: 129 pgs: 129 active+clean; 8.5 MiB data, 774 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:06.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:05.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:05.811974+0000) 2022-09-10T08:24:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:06.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:06.206462+0000) 2022-09-10T08:24:07.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:06.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:06.812118+0000) 2022-09-10T08:24:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:07.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:07.206650+0000) 2022-09-10T08:24:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:07 smithi104 ceph-mon[97044]: pgmap v7516: 129 pgs: 129 active+clean; 8.5 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:24:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:07 smithi110 ceph-mon[97633]: pgmap v7516: 129 pgs: 129 active+clean; 8.5 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:24:08.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:07.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:07.812339+0000) 2022-09-10T08:24:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:08.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:08.206838+0000) 2022-09-10T08:24:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:08 smithi104 ceph-mon[97044]: osdmap e550: 12 total, 12 up, 12 in 2022-09-10T08:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:08 smithi110 ceph-mon[97633]: osdmap e550: 12 total, 12 up, 12 in 2022-09-10T08:24:09.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:08.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:08.812493+0000) 2022-09-10T08:24:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:09.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:09.207038+0000) 2022-09-10T08:24:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:09 smithi104 ceph-mon[97044]: pgmap v7518: 129 pgs: 129 active+clean; 8.5 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:24:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:09 smithi110 ceph-mon[97633]: pgmap v7518: 129 pgs: 129 active+clean; 8.5 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:24:10.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:09.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:09.812657+0000) 2022-09-10T08:24:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:10.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:10.207281+0000) 2022-09-10T08:24:11.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:10.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:10.812785+0000) 2022-09-10T08:24:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:11.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:11.207494+0000) 2022-09-10T08:24:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:11 smithi104 ceph-mon[97044]: pgmap v7519: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:24:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:11 smithi110 ceph-mon[97633]: pgmap v7519: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:24:12.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:11.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:11.812951+0000) 2022-09-10T08:24:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:12.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:12.207783+0000) 2022-09-10T08:24:13.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:12.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:12.813114+0000) 2022-09-10T08:24:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:13.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:13.208029+0000) 2022-09-10T08:24:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:13 smithi110 ceph-mon[97633]: pgmap v7520: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:24:14.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:13.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:13.813246+0000) 2022-09-10T08:24:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:13 smithi104 ceph-mon[97044]: pgmap v7520: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:24:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:14.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:14.208301+0000) 2022-09-10T08:24:15.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:14.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:14.813384+0000) 2022-09-10T08:24:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:15.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:15.208493+0000) 2022-09-10T08:24:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:15 smithi110 ceph-mon[97633]: pgmap v7521: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:24:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:15 smithi110 ceph-mon[97633]: osdmap e551: 12 total, 12 up, 12 in 2022-09-10T08:24:16.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:15.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:15.813521+0000) 2022-09-10T08:24:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:15 smithi104 ceph-mon[97044]: pgmap v7521: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:24:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:15 smithi104 ceph-mon[97044]: osdmap e551: 12 total, 12 up, 12 in 2022-09-10T08:24:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:16.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:16.208741+0000) 2022-09-10T08:24:17.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:16.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:16.813675+0000) 2022-09-10T08:24:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:17.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:17.208881+0000) 2022-09-10T08:24:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:17 smithi110 ceph-mon[97633]: pgmap v7523: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:24:18.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:17.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:17.813805+0000) 2022-09-10T08:24:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:17 smithi104 ceph-mon[97044]: pgmap v7523: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:24:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:18.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:18.209087+0000) 2022-09-10T08:24:19.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:18.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:18.813971+0000) 2022-09-10T08:24:19.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:24:18.811+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:24:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:19.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:19.209312+0000) 2022-09-10T08:24:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:19 smithi110 ceph-mon[97633]: pgmap v7524: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:24:20.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:19.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:19.814108+0000) 2022-09-10T08:24:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:19 smithi104 ceph-mon[97044]: pgmap v7524: 129 pgs: 129 active+clean; 8.5 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:24:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:20.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:20.209547+0000) 2022-09-10T08:24:21.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:20.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:20.814219+0000) 2022-09-10T08:24:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:21.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:21.209769+0000) 2022-09-10T08:24:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:21 smithi110 ceph-mon[97633]: pgmap v7525: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:24:22.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:21.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:21.814376+0000) 2022-09-10T08:24:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:21 smithi104 ceph-mon[97044]: pgmap v7525: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:24:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:22.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:22.209926+0000) 2022-09-10T08:24:23.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:22.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:22.814501+0000) 2022-09-10T08:24:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:23.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:23.210056+0000) 2022-09-10T08:24:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:23 smithi110 ceph-mon[97633]: pgmap v7526: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:24:24.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:23.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:23.814633+0000) 2022-09-10T08:24:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:23 smithi104 ceph-mon[97044]: pgmap v7526: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:24:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:24.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:24.210282+0000) 2022-09-10T08:24:25.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:24.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:24.814823+0000) 2022-09-10T08:24:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:25.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:25.210503+0000) 2022-09-10T08:24:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:25 smithi110 ceph-mon[97633]: pgmap v7527: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:24:26.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:25.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:25.814952+0000) 2022-09-10T08:24:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:25 smithi104 ceph-mon[97044]: pgmap v7527: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:24:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:26.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:26.210692+0000) 2022-09-10T08:24:27.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:26.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:26.815091+0000) 2022-09-10T08:24:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:27.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:27.210946+0000) 2022-09-10T08:24:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:27 smithi110 ceph-mon[97633]: pgmap v7528: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:24:28.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:27.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:27.815247+0000) 2022-09-10T08:24:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:27 smithi104 ceph-mon[97044]: pgmap v7528: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:24:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:28.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:28.211158+0000) 2022-09-10T08:24:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:28 smithi110 ceph-mon[97633]: pgmap v7529: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:29.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:28.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:28.815414+0000) 2022-09-10T08:24:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:28 smithi104 ceph-mon[97044]: pgmap v7529: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:29.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:29.211358+0000) 2022-09-10T08:24:30.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:29.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:29.815562+0000) 2022-09-10T08:24:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:30.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:30.211617+0000) 2022-09-10T08:24:31.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:30.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:30.815684+0000) 2022-09-10T08:24:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:31.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:31.211889+0000) 2022-09-10T08:24:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:31 smithi104 ceph-mon[97044]: pgmap v7530: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:31 smithi110 ceph-mon[97633]: pgmap v7530: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:32.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:31.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:31.815836+0000) 2022-09-10T08:24:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:32.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:32.212179+0000) 2022-09-10T08:24:33.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:32.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:32.815960+0000) 2022-09-10T08:24:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:33.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:33.212361+0000) 2022-09-10T08:24:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:33 smithi104 ceph-mon[97044]: pgmap v7531: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:33 smithi110 ceph-mon[97633]: pgmap v7531: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:34.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:33.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:33.816086+0000) 2022-09-10T08:24:34.214 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:24:33.824+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:24:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:34.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:34.212556+0000) 2022-09-10T08:24:35.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:34.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:34.816255+0000) 2022-09-10T08:24:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:35.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:35.212758+0000) 2022-09-10T08:24:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:35 smithi104 ceph-mon[97044]: pgmap v7532: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:35 smithi110 ceph-mon[97633]: pgmap v7532: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:36.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:35.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:35.816393+0000) 2022-09-10T08:24:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:36.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:36.212983+0000) 2022-09-10T08:24:37.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:36.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:36.816589+0000) 2022-09-10T08:24:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:37.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:37.213282+0000) 2022-09-10T08:24:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:37 smithi104 ceph-mon[97044]: pgmap v7533: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:37 smithi110 ceph-mon[97633]: pgmap v7533: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:38.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:37.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:37.816751+0000) 2022-09-10T08:24:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:38.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:38.213561+0000) 2022-09-10T08:24:39.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:38.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:38.816975+0000) 2022-09-10T08:24:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:39.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:39.213727+0000) 2022-09-10T08:24:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:39 smithi104 ceph-mon[97044]: pgmap v7534: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:39 smithi110 ceph-mon[97633]: pgmap v7534: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:40.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:39.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:39.817190+0000) 2022-09-10T08:24:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:40.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:40.213913+0000) 2022-09-10T08:24:41.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:40.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:40.817349+0000) 2022-09-10T08:24:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:41.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:41.214150+0000) 2022-09-10T08:24:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:41 smithi104 ceph-mon[97044]: pgmap v7535: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:41 smithi110 ceph-mon[97633]: pgmap v7535: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:42.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:41.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:41.817492+0000) 2022-09-10T08:24:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:42.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:42.214304+0000) 2022-09-10T08:24:43.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:42.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:42.817630+0000) 2022-09-10T08:24:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:43.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:43.214537+0000) 2022-09-10T08:24:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:43 smithi104 ceph-mon[97044]: pgmap v7536: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:43 smithi110 ceph-mon[97633]: pgmap v7536: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:44.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:43.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:43.817884+0000) 2022-09-10T08:24:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:44.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:44.214792+0000) 2022-09-10T08:24:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:24:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:24:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:24:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:24:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:24:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:24:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:24:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:24:45.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:44.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:44.818006+0000) 2022-09-10T08:24:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:45.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:45.214981+0000) 2022-09-10T08:24:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:45 smithi104 ceph-mon[97044]: pgmap v7537: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:45 smithi110 ceph-mon[97633]: pgmap v7537: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:46.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:45.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:45.818154+0000) 2022-09-10T08:24:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:46.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:46.215265+0000) 2022-09-10T08:24:47.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:46.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:46.818280+0000) 2022-09-10T08:24:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:47.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:47.215483+0000) 2022-09-10T08:24:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:47 smithi104 ceph-mon[97044]: pgmap v7538: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:47 smithi110 ceph-mon[97633]: pgmap v7538: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:48.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:47.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:47.818407+0000) 2022-09-10T08:24:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:48.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:48.215741+0000) 2022-09-10T08:24:49.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:48.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:48.818540+0000) 2022-09-10T08:24:49.217 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:24:48.834+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:24:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:49.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:49.215978+0000) 2022-09-10T08:24:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:49 smithi104 ceph-mon[97044]: pgmap v7539: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:49 smithi110 ceph-mon[97633]: pgmap v7539: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:50.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:49.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:49.818663+0000) 2022-09-10T08:24:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:50.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:50.216185+0000) 2022-09-10T08:24:51.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:50.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:50.818766+0000) 2022-09-10T08:24:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:51.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:51.216364+0000) 2022-09-10T08:24:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:51 smithi104 ceph-mon[97044]: pgmap v7540: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:51 smithi110 ceph-mon[97633]: pgmap v7540: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:52.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:51.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:51.818961+0000) 2022-09-10T08:24:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:52.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:52.216570+0000) 2022-09-10T08:24:53.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:52.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:52.819083+0000) 2022-09-10T08:24:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:53.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:53.216715+0000) 2022-09-10T08:24:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:53 smithi104 ceph-mon[97044]: pgmap v7541: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:53 smithi110 ceph-mon[97633]: pgmap v7541: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:54.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:53.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:53.819294+0000) 2022-09-10T08:24:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:54.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:54.216932+0000) 2022-09-10T08:24:55.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:54.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:54.819500+0000) 2022-09-10T08:24:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:55.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:55.217125+0000) 2022-09-10T08:24:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:55 smithi104 ceph-mon[97044]: pgmap v7542: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:55 smithi110 ceph-mon[97633]: pgmap v7542: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:24:56.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:55.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:55.819644+0000) 2022-09-10T08:24:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:56.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:56.217343+0000) 2022-09-10T08:24:57.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:56.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:56.819766+0000) 2022-09-10T08:24:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:57.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:57.217607+0000) 2022-09-10T08:24:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:57 smithi104 ceph-mon[97044]: pgmap v7543: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:57 smithi110 ceph-mon[97633]: pgmap v7543: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:24:58.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:57.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:57.819895+0000) 2022-09-10T08:24:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:58.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:58.217877+0000) 2022-09-10T08:24:59.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:58.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:58.820037+0000) 2022-09-10T08:24:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:24:59.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:59.218154+0000) 2022-09-10T08:25:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:24:59 smithi104 ceph-mon[97044]: pgmap v7544: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:24:59 smithi110 ceph-mon[97633]: pgmap v7544: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:00.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:24:59.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:24:59.820167+0000) 2022-09-10T08:25:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:00.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:00.218368+0000) 2022-09-10T08:25:01.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:00.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:00.820294+0000) 2022-09-10T08:25:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:01.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:01.218585+0000) 2022-09-10T08:25:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:01 smithi104 ceph-mon[97044]: pgmap v7545: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:01 smithi110 ceph-mon[97633]: pgmap v7545: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:02.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:01.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:01.820464+0000) 2022-09-10T08:25:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:02.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:02.218843+0000) 2022-09-10T08:25:03.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:02.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:02.820668+0000) 2022-09-10T08:25:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:03.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:03.219082+0000) 2022-09-10T08:25:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:03 smithi104 ceph-mon[97044]: pgmap v7546: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:03 smithi110 ceph-mon[97633]: pgmap v7546: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:04.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:25:03.843+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:25:04.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:03.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:03.820821+0000) 2022-09-10T08:25:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:04.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:04.219280+0000) 2022-09-10T08:25:05.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:04.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:04.820941+0000) 2022-09-10T08:25:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:05.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:05.219502+0000) 2022-09-10T08:25:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:05 smithi104 ceph-mon[97044]: pgmap v7547: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:05 smithi110 ceph-mon[97633]: pgmap v7547: 129 pgs: 129 active+clean; 8.5 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:06.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:05.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:05.821075+0000) 2022-09-10T08:25:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:06.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:06.219687+0000) 2022-09-10T08:25:07.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:06.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:06.821224+0000) 2022-09-10T08:25:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:07.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:07.219886+0000) 2022-09-10T08:25:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:07 smithi104 ceph-mon[97044]: pgmap v7548: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:25:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:07 smithi110 ceph-mon[97633]: pgmap v7548: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:25:08.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:07.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:07.821372+0000) 2022-09-10T08:25:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:08.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:08.220207+0000) 2022-09-10T08:25:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:08 smithi104 ceph-mon[97044]: osdmap e552: 12 total, 12 up, 12 in 2022-09-10T08:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:08 smithi110 ceph-mon[97633]: osdmap e552: 12 total, 12 up, 12 in 2022-09-10T08:25:09.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:08.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:08.821497+0000) 2022-09-10T08:25:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:09.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:09.220436+0000) 2022-09-10T08:25:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:09 smithi104 ceph-mon[97044]: pgmap v7550: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:09 smithi110 ceph-mon[97633]: pgmap v7550: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:10.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:09.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:09.821629+0000) 2022-09-10T08:25:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:10.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:10.220617+0000) 2022-09-10T08:25:11.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:10.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:10.821784+0000) 2022-09-10T08:25:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:11.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:11.220879+0000) 2022-09-10T08:25:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:11 smithi104 ceph-mon[97044]: pgmap v7551: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:11 smithi110 ceph-mon[97633]: pgmap v7551: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:12.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:11.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:11.821925+0000) 2022-09-10T08:25:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:12.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:12.221076+0000) 2022-09-10T08:25:13.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:12.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:12.822050+0000) 2022-09-10T08:25:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:13.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:13.221351+0000) 2022-09-10T08:25:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:13 smithi104 ceph-mon[97044]: pgmap v7552: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:13 smithi110 ceph-mon[97633]: pgmap v7552: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:14.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:13.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:13.822164+0000) 2022-09-10T08:25:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:14.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:14.221535+0000) 2022-09-10T08:25:15.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:14.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:14.822270+0000) 2022-09-10T08:25:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:15.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:15.221751+0000) 2022-09-10T08:25:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:15 smithi104 ceph-mon[97044]: pgmap v7553: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:15 smithi104 ceph-mon[97044]: osdmap e553: 12 total, 12 up, 12 in 2022-09-10T08:25:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:15 smithi110 ceph-mon[97633]: pgmap v7553: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:25:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:15 smithi110 ceph-mon[97633]: osdmap e553: 12 total, 12 up, 12 in 2022-09-10T08:25:16.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:15.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:15.822414+0000) 2022-09-10T08:25:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:16.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:16.221969+0000) 2022-09-10T08:25:17.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:16.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:16.822526+0000) 2022-09-10T08:25:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:17.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:17.222221+0000) 2022-09-10T08:25:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:17 smithi110 ceph-mon[97633]: pgmap v7555: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:25:18.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:17.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:17.822645+0000) 2022-09-10T08:25:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:17 smithi104 ceph-mon[97044]: pgmap v7555: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T08:25:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:18.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:18.222426+0000) 2022-09-10T08:25:19.223 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:25:18.849+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:25:19.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:18.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:18.822825+0000) 2022-09-10T08:25:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:19.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:19.222646+0000) 2022-09-10T08:25:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:19 smithi110 ceph-mon[97633]: pgmap v7556: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:25:20.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:19.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:19.823014+0000) 2022-09-10T08:25:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:19 smithi104 ceph-mon[97044]: pgmap v7556: 129 pgs: 129 active+clean; 8.5 MiB data, 782 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:25:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:20.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:20.222817+0000) 2022-09-10T08:25:21.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:20.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:20.823170+0000) 2022-09-10T08:25:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:21.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:21.223021+0000) 2022-09-10T08:25:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:21 smithi110 ceph-mon[97633]: pgmap v7557: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:25:22.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:21.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:21.823322+0000) 2022-09-10T08:25:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:21 smithi104 ceph-mon[97044]: pgmap v7557: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:25:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:22.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:22.223267+0000) 2022-09-10T08:25:23.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:22.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:22.823494+0000) 2022-09-10T08:25:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:23.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:23.223471+0000) 2022-09-10T08:25:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:23 smithi110 ceph-mon[97633]: pgmap v7558: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:25:24.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:23.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:23.823699+0000) 2022-09-10T08:25:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:23 smithi104 ceph-mon[97044]: pgmap v7558: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:25:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:24.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:24.223646+0000) 2022-09-10T08:25:25.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:24.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:24.823838+0000) 2022-09-10T08:25:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:25.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:25.223909+0000) 2022-09-10T08:25:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:25 smithi110 ceph-mon[97633]: pgmap v7559: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:25:26.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:25.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:25.823964+0000) 2022-09-10T08:25:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:25 smithi104 ceph-mon[97044]: pgmap v7559: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:25:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:26.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:26.224182+0000) 2022-09-10T08:25:27.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:26.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:26.824127+0000) 2022-09-10T08:25:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:27.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:27.224366+0000) 2022-09-10T08:25:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:27 smithi110 ceph-mon[97633]: pgmap v7560: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:25:28.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:27.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:27.824259+0000) 2022-09-10T08:25:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:27 smithi104 ceph-mon[97044]: pgmap v7560: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:25:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:28.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:28.224605+0000) 2022-09-10T08:25:29.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:28.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:28.824410+0000) 2022-09-10T08:25:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:29.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:29.224853+0000) 2022-09-10T08:25:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:29 smithi110 ceph-mon[97633]: pgmap v7561: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:30.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:29.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:29.824538+0000) 2022-09-10T08:25:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:29 smithi104 ceph-mon[97044]: pgmap v7561: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:30.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:30.225046+0000) 2022-09-10T08:25:31.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:30.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:30.824669+0000) 2022-09-10T08:25:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:31.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:31.225252+0000) 2022-09-10T08:25:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:31 smithi110 ceph-mon[97633]: pgmap v7562: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:32.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:31.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:31.824825+0000) 2022-09-10T08:25:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:31 smithi104 ceph-mon[97044]: pgmap v7562: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:32.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:32.225502+0000) 2022-09-10T08:25:33.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:32.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:32.824988+0000) 2022-09-10T08:25:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:33.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:33.225707+0000) 2022-09-10T08:25:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:33 smithi110 ceph-mon[97633]: pgmap v7563: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:34.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:33.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:33.825137+0000) 2022-09-10T08:25:34.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:25:33.866+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:25:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:33 smithi104 ceph-mon[97044]: pgmap v7563: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:34.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:34.225845+0000) 2022-09-10T08:25:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:34 smithi110 ceph-mon[97633]: pgmap v7564: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:35.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:34.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:34.825273+0000) 2022-09-10T08:25:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:34 smithi104 ceph-mon[97044]: pgmap v7564: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:35.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:35.226151+0000) 2022-09-10T08:25:36.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:35.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:35.825411+0000) 2022-09-10T08:25:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:36.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:36.226380+0000) 2022-09-10T08:25:37.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:36.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:36.825547+0000) 2022-09-10T08:25:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:37.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:37.226556+0000) 2022-09-10T08:25:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:37 smithi104 ceph-mon[97044]: pgmap v7565: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:37 smithi110 ceph-mon[97633]: pgmap v7565: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:38.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:37.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:37.825692+0000) 2022-09-10T08:25:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:38.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:38.226784+0000) 2022-09-10T08:25:39.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:38.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:38.825833+0000) 2022-09-10T08:25:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:39.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:39.227025+0000) 2022-09-10T08:25:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:39 smithi104 ceph-mon[97044]: pgmap v7566: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:39 smithi110 ceph-mon[97633]: pgmap v7566: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:40.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:39.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:39.825986+0000) 2022-09-10T08:25:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:40.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:40.227225+0000) 2022-09-10T08:25:41.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:40.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:40.826128+0000) 2022-09-10T08:25:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:41.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:41.227473+0000) 2022-09-10T08:25:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:41 smithi104 ceph-mon[97044]: pgmap v7567: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:41 smithi110 ceph-mon[97633]: pgmap v7567: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:42.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:41.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:41.826344+0000) 2022-09-10T08:25:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:42.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:42.227723+0000) 2022-09-10T08:25:43.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:42.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:42.826516+0000) 2022-09-10T08:25:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:43.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:43.227914+0000) 2022-09-10T08:25:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:43 smithi104 ceph-mon[97044]: pgmap v7568: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:43 smithi110 ceph-mon[97633]: pgmap v7568: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:44.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:43.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:43.826689+0000) 2022-09-10T08:25:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:44.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:44.228147+0000) 2022-09-10T08:25:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:25:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:25:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:25:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:25:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:25:45.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:44.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:44.826890+0000) 2022-09-10T08:25:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:45.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:45.228356+0000) 2022-09-10T08:25:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:45 smithi104 ceph-mon[97044]: pgmap v7569: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:45 smithi110 ceph-mon[97633]: pgmap v7569: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:46.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:45.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:45.827061+0000) 2022-09-10T08:25:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:46.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:46.228542+0000) 2022-09-10T08:25:47.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:46.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:46.827172+0000) 2022-09-10T08:25:47.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:47.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:47.228767+0000) 2022-09-10T08:25:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:47 smithi104 ceph-mon[97044]: pgmap v7570: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:47 smithi110 ceph-mon[97633]: pgmap v7570: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:48.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:47.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:47.827314+0000) 2022-09-10T08:25:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:48.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:48.229045+0000) 2022-09-10T08:25:49.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:48.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:48.827515+0000) 2022-09-10T08:25:49.230 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:25:48.881+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:25:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:49.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:49.229335+0000) 2022-09-10T08:25:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:49 smithi104 ceph-mon[97044]: pgmap v7571: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:49 smithi110 ceph-mon[97633]: pgmap v7571: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:50.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:49.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:49.827649+0000) 2022-09-10T08:25:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:50.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:50.229609+0000) 2022-09-10T08:25:51.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:50.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:50.827833+0000) 2022-09-10T08:25:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:51.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:51.229815+0000) 2022-09-10T08:25:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:51 smithi104 ceph-mon[97044]: pgmap v7572: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:51 smithi110 ceph-mon[97633]: pgmap v7572: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:52.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:51.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:51.828058+0000) 2022-09-10T08:25:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:52.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:52.229996+0000) 2022-09-10T08:25:53.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:52.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:52.828170+0000) 2022-09-10T08:25:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:53.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:53.230205+0000) 2022-09-10T08:25:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:53 smithi104 ceph-mon[97044]: pgmap v7573: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:53 smithi110 ceph-mon[97633]: pgmap v7573: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:54.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:53.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:53.828306+0000) 2022-09-10T08:25:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:54.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:54.230398+0000) 2022-09-10T08:25:55.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:54.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:54.828436+0000) 2022-09-10T08:25:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:55.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:55.230576+0000) 2022-09-10T08:25:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:55 smithi104 ceph-mon[97044]: pgmap v7574: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:55 smithi110 ceph-mon[97633]: pgmap v7574: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:25:56.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:55.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:55.828570+0000) 2022-09-10T08:25:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:56.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:56.230812+0000) 2022-09-10T08:25:57.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:56.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:56.828746+0000) 2022-09-10T08:25:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:57.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:57.231053+0000) 2022-09-10T08:25:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:57 smithi104 ceph-mon[97044]: pgmap v7575: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:57 smithi110 ceph-mon[97633]: pgmap v7575: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:25:58.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:57.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:57.828901+0000) 2022-09-10T08:25:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:58.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:58.231308+0000) 2022-09-10T08:25:59.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:58.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:58.829088+0000) 2022-09-10T08:25:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:25:59.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:59.231569+0000) 2022-09-10T08:26:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:25:59 smithi104 ceph-mon[97044]: pgmap v7576: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:25:59 smithi110 ceph-mon[97633]: pgmap v7576: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:00.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:25:59.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:25:59.829290+0000) 2022-09-10T08:26:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:00.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:00.231704+0000) 2022-09-10T08:26:01.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:00.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:00.829417+0000) 2022-09-10T08:26:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:01.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:01.231910+0000) 2022-09-10T08:26:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:01 smithi104 ceph-mon[97044]: pgmap v7577: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:01 smithi110 ceph-mon[97633]: pgmap v7577: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:02.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:01.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:01.829547+0000) 2022-09-10T08:26:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:02.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:02.232145+0000) 2022-09-10T08:26:03.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:02.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:02.829667+0000) 2022-09-10T08:26:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:03.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:03.232384+0000) 2022-09-10T08:26:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:03 smithi104 ceph-mon[97044]: pgmap v7578: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:03 smithi110 ceph-mon[97633]: pgmap v7578: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:04.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:03.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:03.829795+0000) 2022-09-10T08:26:04.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:26:03.896+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:26:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:04.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:04.232546+0000) 2022-09-10T08:26:05.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:04.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:04.829955+0000) 2022-09-10T08:26:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:05.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:05.232727+0000) 2022-09-10T08:26:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:05 smithi104 ceph-mon[97044]: pgmap v7579: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:05 smithi110 ceph-mon[97633]: pgmap v7579: 129 pgs: 129 active+clean; 8.5 MiB data, 783 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:06.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:05.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:05.830124+0000) 2022-09-10T08:26:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:06.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:06.232932+0000) 2022-09-10T08:26:07.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:06.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:06.830250+0000) 2022-09-10T08:26:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:07.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:07.233185+0000) 2022-09-10T08:26:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:07 smithi104 ceph-mon[97044]: pgmap v7580: 129 pgs: 129 active+clean; 8.5 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:26:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:07 smithi110 ceph-mon[97633]: pgmap v7580: 129 pgs: 129 active+clean; 8.5 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:26:08.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:07.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:07.830396+0000) 2022-09-10T08:26:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:08.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:08.233358+0000) 2022-09-10T08:26:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:08 smithi104 ceph-mon[97044]: osdmap e554: 12 total, 12 up, 12 in 2022-09-10T08:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:08 smithi110 ceph-mon[97633]: osdmap e554: 12 total, 12 up, 12 in 2022-09-10T08:26:09.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:08.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:08.830522+0000) 2022-09-10T08:26:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:09.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:09.233530+0000) 2022-09-10T08:26:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:09 smithi110 ceph-mon[97633]: pgmap v7582: 129 pgs: 129 active+clean; 8.5 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:10.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:09.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:09.830680+0000) 2022-09-10T08:26:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:09 smithi104 ceph-mon[97044]: pgmap v7582: 129 pgs: 129 active+clean; 8.5 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:10.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:10.233765+0000) 2022-09-10T08:26:11.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:10.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:10.830765+0000) 2022-09-10T08:26:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:11.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:11.233912+0000) 2022-09-10T08:26:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:11 smithi110 ceph-mon[97633]: pgmap v7583: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:12.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:11.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:11.830898+0000) 2022-09-10T08:26:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:11 smithi104 ceph-mon[97044]: pgmap v7583: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:12.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:12.234166+0000) 2022-09-10T08:26:13.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:12.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:12.831067+0000) 2022-09-10T08:26:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:13.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:13.234378+0000) 2022-09-10T08:26:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:13 smithi110 ceph-mon[97633]: pgmap v7584: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:14.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:13.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:13.831163+0000) 2022-09-10T08:26:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:13 smithi104 ceph-mon[97044]: pgmap v7584: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:14.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:14.234597+0000) 2022-09-10T08:26:15.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:14.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:14.831296+0000) 2022-09-10T08:26:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:15.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:15.234775+0000) 2022-09-10T08:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:15 smithi110 ceph-mon[97633]: pgmap v7585: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:15 smithi110 ceph-mon[97633]: osdmap e555: 12 total, 12 up, 12 in 2022-09-10T08:26:16.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:15.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:15.831436+0000) 2022-09-10T08:26:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:15 smithi104 ceph-mon[97044]: pgmap v7585: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:26:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:15 smithi104 ceph-mon[97044]: osdmap e555: 12 total, 12 up, 12 in 2022-09-10T08:26:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:16.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:16.234992+0000) 2022-09-10T08:26:17.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:16.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:16.831604+0000) 2022-09-10T08:26:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:17.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:17.235270+0000) 2022-09-10T08:26:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:17 smithi110 ceph-mon[97633]: pgmap v7587: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:26:18.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:17.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:17.831751+0000) 2022-09-10T08:26:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:17 smithi104 ceph-mon[97044]: pgmap v7587: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:26:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:18.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:18.235452+0000) 2022-09-10T08:26:19.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:26:18.911+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:26:19.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:18.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:18.831927+0000) 2022-09-10T08:26:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:19.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:19.235610+0000) 2022-09-10T08:26:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:19 smithi110 ceph-mon[97633]: pgmap v7588: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:26:20.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:19.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:19.832083+0000) 2022-09-10T08:26:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:19 smithi104 ceph-mon[97044]: pgmap v7588: 129 pgs: 129 active+clean; 8.5 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:26:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:20.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:20.235819+0000) 2022-09-10T08:26:21.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:20.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:20.832221+0000) 2022-09-10T08:26:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:21.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:21.236014+0000) 2022-09-10T08:26:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:21 smithi110 ceph-mon[97633]: pgmap v7589: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:26:22.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:21.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:21.832401+0000) 2022-09-10T08:26:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:21 smithi104 ceph-mon[97044]: pgmap v7589: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:26:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:22.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:22.236233+0000) 2022-09-10T08:26:23.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:22.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:22.832556+0000) 2022-09-10T08:26:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:23.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:23.236473+0000) 2022-09-10T08:26:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:23 smithi110 ceph-mon[97633]: pgmap v7590: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:26:24.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:23.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:23.832681+0000) 2022-09-10T08:26:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:23 smithi104 ceph-mon[97044]: pgmap v7590: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:26:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:24.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:24.236729+0000) 2022-09-10T08:26:25.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:24.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:24.832805+0000) 2022-09-10T08:26:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:25.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:25.236942+0000) 2022-09-10T08:26:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:25 smithi110 ceph-mon[97633]: pgmap v7591: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:26:26.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:25.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:25.832932+0000) 2022-09-10T08:26:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:25 smithi104 ceph-mon[97044]: pgmap v7591: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:26:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:26.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:26.237173+0000) 2022-09-10T08:26:27.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:26.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:26.833174+0000) 2022-09-10T08:26:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:27.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:27.237382+0000) 2022-09-10T08:26:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:27 smithi110 ceph-mon[97633]: pgmap v7592: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:26:28.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:27.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:27.833342+0000) 2022-09-10T08:26:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:27 smithi104 ceph-mon[97044]: pgmap v7592: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:26:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:28.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:28.237607+0000) 2022-09-10T08:26:29.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:28.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:28.833461+0000) 2022-09-10T08:26:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:29.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:29.237849+0000) 2022-09-10T08:26:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:29 smithi110 ceph-mon[97633]: pgmap v7593: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:30.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:29.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:29.833602+0000) 2022-09-10T08:26:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:29 smithi104 ceph-mon[97044]: pgmap v7593: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:30.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:30.238052+0000) 2022-09-10T08:26:31.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:30.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:30.833747+0000) 2022-09-10T08:26:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:31.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:31.238230+0000) 2022-09-10T08:26:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:31 smithi110 ceph-mon[97633]: pgmap v7594: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:32.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:31.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:31.833912+0000) 2022-09-10T08:26:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:31 smithi104 ceph-mon[97044]: pgmap v7594: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:32.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:32.238442+0000) 2022-09-10T08:26:33.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:32.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:32.834056+0000) 2022-09-10T08:26:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:33.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:33.238673+0000) 2022-09-10T08:26:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:33 smithi110 ceph-mon[97633]: pgmap v7595: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:34.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:33.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:33.834171+0000) 2022-09-10T08:26:34.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:26:33.919+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:26:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:33 smithi104 ceph-mon[97044]: pgmap v7595: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:34.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:34.238963+0000) 2022-09-10T08:26:35.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:34.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:34.834318+0000) 2022-09-10T08:26:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:35.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:35.239279+0000) 2022-09-10T08:26:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:35 smithi110 ceph-mon[97633]: pgmap v7596: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:36.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:35.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:35.834474+0000) 2022-09-10T08:26:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:35 smithi104 ceph-mon[97044]: pgmap v7596: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:36.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:36.239475+0000) 2022-09-10T08:26:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:36 smithi110 ceph-mon[97633]: pgmap v7597: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:37.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:36.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:36.834594+0000) 2022-09-10T08:26:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:36 smithi104 ceph-mon[97044]: pgmap v7597: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:37.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:37.239627+0000) 2022-09-10T08:26:38.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:37.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:37.834729+0000) 2022-09-10T08:26:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:38.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:38.239783+0000) 2022-09-10T08:26:39.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:38.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:38.834914+0000) 2022-09-10T08:26:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:39.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:39.239944+0000) 2022-09-10T08:26:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:39 smithi110 ceph-mon[97633]: pgmap v7598: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:39 smithi104 ceph-mon[97044]: pgmap v7598: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:40.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:39.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:39.835060+0000) 2022-09-10T08:26:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:40.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:40.240162+0000) 2022-09-10T08:26:41.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:40.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:40.835159+0000) 2022-09-10T08:26:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:41.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:41.240355+0000) 2022-09-10T08:26:42.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:41 smithi104 ceph-mon[97044]: pgmap v7599: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:41 smithi110 ceph-mon[97633]: pgmap v7599: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:42.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:41.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:41.835328+0000) 2022-09-10T08:26:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:42.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:42.240551+0000) 2022-09-10T08:26:43.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:42.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:42.835517+0000) 2022-09-10T08:26:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:43.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:43.240704+0000) 2022-09-10T08:26:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:43 smithi104 ceph-mon[97044]: pgmap v7600: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:43 smithi110 ceph-mon[97633]: pgmap v7600: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:44.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:43.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:43.835697+0000) 2022-09-10T08:26:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:44.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:44.240934+0000) 2022-09-10T08:26:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:26:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:26:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:26:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:26:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:26:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:26:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:26:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:26:45.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:44.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:44.835846+0000) 2022-09-10T08:26:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:45.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:45.241213+0000) 2022-09-10T08:26:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:45 smithi104 ceph-mon[97044]: pgmap v7601: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:45 smithi110 ceph-mon[97633]: pgmap v7601: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:26:46.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:45.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:45.836019+0000) 2022-09-10T08:26:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:46.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:46.241440+0000) 2022-09-10T08:26:47.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:46.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:46.836159+0000) 2022-09-10T08:26:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:47.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:47.241666+0000) 2022-09-10T08:26:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:47 smithi104 ceph-mon[97044]: pgmap v7602: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:47 smithi110 ceph-mon[97633]: pgmap v7602: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:48.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:47.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:47.836281+0000) 2022-09-10T08:26:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:48.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:48.241864+0000) 2022-09-10T08:26:49.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:48.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:48.836408+0000) 2022-09-10T08:26:49.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:26:48.923+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:26:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:49.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:49.242153+0000) 2022-09-10T08:26:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:49 smithi104 ceph-mon[97044]: pgmap v7603: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:26:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:49 smithi110 ceph-mon[97633]: pgmap v7603: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:26:50.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:49.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:49.836534+0000) 2022-09-10T08:26:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:50.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:50.242300+0000) 2022-09-10T08:26:51.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:50.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:50.836657+0000) 2022-09-10T08:26:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:51.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:51.242501+0000) 2022-09-10T08:26:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:51 smithi104 ceph-mon[97044]: pgmap v7604: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:51 smithi110 ceph-mon[97633]: pgmap v7604: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:52.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:51.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:51.836845+0000) 2022-09-10T08:26:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:52.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:52.242698+0000) 2022-09-10T08:26:53.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:52.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:52.836958+0000) 2022-09-10T08:26:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:53.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:53.242834+0000) 2022-09-10T08:26:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:53 smithi104 ceph-mon[97044]: pgmap v7605: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:26:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:53 smithi110 ceph-mon[97633]: pgmap v7605: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:26:54.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:53.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:53.837124+0000) 2022-09-10T08:26:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:54.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:54.243011+0000) 2022-09-10T08:26:55.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:54.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:54.837334+0000) 2022-09-10T08:26:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:55.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:55.243204+0000) 2022-09-10T08:26:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:55 smithi104 ceph-mon[97044]: pgmap v7606: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:26:56.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:55 smithi110 ceph-mon[97633]: pgmap v7606: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:26:56.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:55.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:55.837533+0000) 2022-09-10T08:26:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:56.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:56.243450+0000) 2022-09-10T08:26:57.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:56.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:56.837686+0000) 2022-09-10T08:26:57.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:57.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:57.243665+0000) 2022-09-10T08:26:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:57 smithi104 ceph-mon[97044]: pgmap v7607: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:57 smithi110 ceph-mon[97633]: pgmap v7607: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:26:58.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:57.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:57.837809+0000) 2022-09-10T08:26:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:58.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:58.243912+0000) 2022-09-10T08:26:59.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:58.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:58.837993+0000) 2022-09-10T08:26:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:26:59.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:59.244173+0000) 2022-09-10T08:27:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:26:59 smithi104 ceph-mon[97044]: pgmap v7608: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:26:59 smithi110 ceph-mon[97633]: pgmap v7608: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:00.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:26:59.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:26:59.838133+0000) 2022-09-10T08:27:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:00.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:00.244317+0000) 2022-09-10T08:27:01.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:00.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:00.838258+0000) 2022-09-10T08:27:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:01.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:01.244506+0000) 2022-09-10T08:27:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:01 smithi104 ceph-mon[97044]: pgmap v7609: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:01 smithi110 ceph-mon[97633]: pgmap v7609: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:02.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:01.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:01.838395+0000) 2022-09-10T08:27:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:02.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:02.244704+0000) 2022-09-10T08:27:03.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:02.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:02.838527+0000) 2022-09-10T08:27:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:03.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:03.244950+0000) 2022-09-10T08:27:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:03 smithi104 ceph-mon[97044]: pgmap v7610: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:03 smithi110 ceph-mon[97633]: pgmap v7610: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:04.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:03.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:03.838664+0000) 2022-09-10T08:27:04.246 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:27:03.938+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:27:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:04.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:04.245207+0000) 2022-09-10T08:27:05.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:04.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:04.838797+0000) 2022-09-10T08:27:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:05.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:05.245472+0000) 2022-09-10T08:27:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:05 smithi104 ceph-mon[97044]: pgmap v7611: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:05 smithi110 ceph-mon[97633]: pgmap v7611: 129 pgs: 129 active+clean; 8.5 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:06.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:05.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:05.838964+0000) 2022-09-10T08:27:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:06.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:06.245757+0000) 2022-09-10T08:27:07.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:06.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:06.839125+0000) 2022-09-10T08:27:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:07.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:07.245910+0000) 2022-09-10T08:27:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:07 smithi104 ceph-mon[97044]: pgmap v7612: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:27:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:07 smithi110 ceph-mon[97633]: pgmap v7612: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:27:08.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:07.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:07.839262+0000) 2022-09-10T08:27:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:08.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:08.246180+0000) 2022-09-10T08:27:09.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:08 smithi110 ceph-mon[97633]: osdmap e556: 12 total, 12 up, 12 in 2022-09-10T08:27:09.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:08.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:08.839415+0000) 2022-09-10T08:27:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:08 smithi104 ceph-mon[97044]: osdmap e556: 12 total, 12 up, 12 in 2022-09-10T08:27:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:09.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:09.246440+0000) 2022-09-10T08:27:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:09 smithi110 ceph-mon[97633]: pgmap v7614: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:10.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:09.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:09.839551+0000) 2022-09-10T08:27:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:09 smithi104 ceph-mon[97044]: pgmap v7614: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:10.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:10.246600+0000) 2022-09-10T08:27:11.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:10.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:10.839687+0000) 2022-09-10T08:27:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:11.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:11.246727+0000) 2022-09-10T08:27:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:11 smithi110 ceph-mon[97633]: pgmap v7615: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:12.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:11.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:11.839819+0000) 2022-09-10T08:27:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:11 smithi104 ceph-mon[97044]: pgmap v7615: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:12.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:12.246970+0000) 2022-09-10T08:27:13.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:12.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:12.839986+0000) 2022-09-10T08:27:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:13.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:13.247234+0000) 2022-09-10T08:27:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:13 smithi110 ceph-mon[97633]: pgmap v7616: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:14.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:13.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:13.840128+0000) 2022-09-10T08:27:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:13 smithi104 ceph-mon[97044]: pgmap v7616: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:14.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:14.247431+0000) 2022-09-10T08:27:15.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:14.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:14.840299+0000) 2022-09-10T08:27:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:15.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:15.247604+0000) 2022-09-10T08:27:16.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:15 smithi110 ceph-mon[97633]: pgmap v7617: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:16.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:15 smithi110 ceph-mon[97633]: osdmap e557: 12 total, 12 up, 12 in 2022-09-10T08:27:16.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:15.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:15.840454+0000) 2022-09-10T08:27:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:15 smithi104 ceph-mon[97044]: pgmap v7617: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:27:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:15 smithi104 ceph-mon[97044]: osdmap e557: 12 total, 12 up, 12 in 2022-09-10T08:27:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:16.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:16.247907+0000) 2022-09-10T08:27:17.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:16.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:16.840603+0000) 2022-09-10T08:27:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:17.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:17.248152+0000) 2022-09-10T08:27:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:17 smithi110 ceph-mon[97633]: pgmap v7619: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:27:18.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:17.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:17.840740+0000) 2022-09-10T08:27:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:17 smithi104 ceph-mon[97044]: pgmap v7619: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:27:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:18.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:18.248383+0000) 2022-09-10T08:27:19.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:18.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:18.840879+0000) 2022-09-10T08:27:19.250 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:27:18.952+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:27:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:19.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:19.248598+0000) 2022-09-10T08:27:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:19 smithi110 ceph-mon[97633]: pgmap v7620: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:27:20.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:19.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:19.841034+0000) 2022-09-10T08:27:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:19 smithi104 ceph-mon[97044]: pgmap v7620: 129 pgs: 129 active+clean; 8.5 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:27:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:20.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:20.248855+0000) 2022-09-10T08:27:21.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:20.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:20.841196+0000) 2022-09-10T08:27:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:21.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:21.248993+0000) 2022-09-10T08:27:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:21 smithi110 ceph-mon[97633]: pgmap v7621: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:27:22.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:21.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:21.841338+0000) 2022-09-10T08:27:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:21 smithi104 ceph-mon[97044]: pgmap v7621: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:27:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:22.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:22.249186+0000) 2022-09-10T08:27:23.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:22.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:22.841471+0000) 2022-09-10T08:27:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:23.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:23.249378+0000) 2022-09-10T08:27:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:23 smithi110 ceph-mon[97633]: pgmap v7622: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:27:24.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:23.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:23.841612+0000) 2022-09-10T08:27:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:23 smithi104 ceph-mon[97044]: pgmap v7622: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:27:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:24.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:24.249653+0000) 2022-09-10T08:27:25.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:24.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:24.841751+0000) 2022-09-10T08:27:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:25.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:25.249815+0000) 2022-09-10T08:27:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:25 smithi110 ceph-mon[97633]: pgmap v7623: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:27:26.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:25.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:25.841882+0000) 2022-09-10T08:27:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:25 smithi104 ceph-mon[97044]: pgmap v7623: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:27:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:26.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:26.250010+0000) 2022-09-10T08:27:27.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:26.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:26.842017+0000) 2022-09-10T08:27:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:27.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:27.250224+0000) 2022-09-10T08:27:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:27 smithi110 ceph-mon[97633]: pgmap v7624: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:27:28.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:27.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:27.842153+0000) 2022-09-10T08:27:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:27 smithi104 ceph-mon[97044]: pgmap v7624: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:27:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:28.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:28.250480+0000) 2022-09-10T08:27:29.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:28.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:28.842298+0000) 2022-09-10T08:27:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:29.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:29.250615+0000) 2022-09-10T08:27:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:29 smithi110 ceph-mon[97633]: pgmap v7625: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:30.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:29.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:29.842448+0000) 2022-09-10T08:27:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:29 smithi104 ceph-mon[97044]: pgmap v7625: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:30.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:30.250810+0000) 2022-09-10T08:27:31.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:30.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:30.842686+0000) 2022-09-10T08:27:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:31.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:31.251013+0000) 2022-09-10T08:27:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:31 smithi110 ceph-mon[97633]: pgmap v7626: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:32.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:31.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:31.842849+0000) 2022-09-10T08:27:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:31 smithi104 ceph-mon[97044]: pgmap v7626: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:32.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:32.251267+0000) 2022-09-10T08:27:33.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:32.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:32.842958+0000) 2022-09-10T08:27:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:33.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:33.251450+0000) 2022-09-10T08:27:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:33 smithi110 ceph-mon[97633]: pgmap v7627: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:34.252 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:27:33.963+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:27:34.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:33.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:33.843183+0000) 2022-09-10T08:27:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:33 smithi104 ceph-mon[97044]: pgmap v7627: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:34.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:34.251710+0000) 2022-09-10T08:27:35.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:34.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:34.843348+0000) 2022-09-10T08:27:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:35.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:35.251983+0000) 2022-09-10T08:27:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:35 smithi110 ceph-mon[97633]: pgmap v7628: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:36.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:35.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:35.843486+0000) 2022-09-10T08:27:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:35 smithi104 ceph-mon[97044]: pgmap v7628: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:36.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:36.252193+0000) 2022-09-10T08:27:37.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:36.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:36.843705+0000) 2022-09-10T08:27:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:37.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:37.252448+0000) 2022-09-10T08:27:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:37 smithi110 ceph-mon[97633]: pgmap v7629: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:38.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:37.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:37.843916+0000) 2022-09-10T08:27:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:37 smithi104 ceph-mon[97044]: pgmap v7629: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:38.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:38.252662+0000) 2022-09-10T08:27:39.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:38 smithi110 ceph-mon[97633]: pgmap v7630: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:39.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:38.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:38.844079+0000) 2022-09-10T08:27:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:38 smithi104 ceph-mon[97044]: pgmap v7630: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:39.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:39.252922+0000) 2022-09-10T08:27:40.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:39.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:39.844237+0000) 2022-09-10T08:27:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:40.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:40.253222+0000) 2022-09-10T08:27:41.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:40.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:40.844361+0000) 2022-09-10T08:27:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:41.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:41.253379+0000) 2022-09-10T08:27:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:41 smithi104 ceph-mon[97044]: pgmap v7631: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:41 smithi110 ceph-mon[97633]: pgmap v7631: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:42.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:41.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:41.844531+0000) 2022-09-10T08:27:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:42.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:42.253524+0000) 2022-09-10T08:27:43.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:42.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:42.844694+0000) 2022-09-10T08:27:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:43.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:43.253696+0000) 2022-09-10T08:27:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:43 smithi104 ceph-mon[97044]: pgmap v7632: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:43 smithi110 ceph-mon[97633]: pgmap v7632: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:44.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:43.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:43.844842+0000) 2022-09-10T08:27:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:44.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:44.253839+0000) 2022-09-10T08:27:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:27:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:27:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:27:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:27:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:27:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:27:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:27:45.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:44.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:44.844976+0000) 2022-09-10T08:27:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:45.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:45.253958+0000) 2022-09-10T08:27:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:45 smithi104 ceph-mon[97044]: pgmap v7633: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:45 smithi110 ceph-mon[97633]: pgmap v7633: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:46.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:45.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:45.845150+0000) 2022-09-10T08:27:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:46.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:46.254147+0000) 2022-09-10T08:27:47.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:46.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:46.845308+0000) 2022-09-10T08:27:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:47.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:47.254321+0000) 2022-09-10T08:27:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:47 smithi104 ceph-mon[97044]: pgmap v7634: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:47 smithi110 ceph-mon[97633]: pgmap v7634: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:48.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:47.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:47.845428+0000) 2022-09-10T08:27:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:48.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:48.254444+0000) 2022-09-10T08:27:49.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:48.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:48.845548+0000) 2022-09-10T08:27:49.256 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:27:48.976+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:27:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:49.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:49.254568+0000) 2022-09-10T08:27:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:49 smithi104 ceph-mon[97044]: pgmap v7635: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:49 smithi110 ceph-mon[97633]: pgmap v7635: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:50.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:49.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:49.845680+0000) 2022-09-10T08:27:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:50.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:50.254693+0000) 2022-09-10T08:27:51.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:50.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:50.845818+0000) 2022-09-10T08:27:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:51.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:51.254802+0000) 2022-09-10T08:27:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:51 smithi104 ceph-mon[97044]: pgmap v7636: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:51 smithi110 ceph-mon[97633]: pgmap v7636: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:52.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:51.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:51.845962+0000) 2022-09-10T08:27:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:52.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:52.254930+0000) 2022-09-10T08:27:53.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:52.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:52.846078+0000) 2022-09-10T08:27:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:53.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:53.255129+0000) 2022-09-10T08:27:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:53 smithi104 ceph-mon[97044]: pgmap v7637: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:53 smithi110 ceph-mon[97633]: pgmap v7637: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:54.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:53.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:53.846213+0000) 2022-09-10T08:27:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:54.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:54.255276+0000) 2022-09-10T08:27:55.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:54.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:54.846344+0000) 2022-09-10T08:27:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:55.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:55.255439+0000) 2022-09-10T08:27:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:55 smithi110 ceph-mon[97633]: pgmap v7638: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:56.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:55.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:55.846480+0000) 2022-09-10T08:27:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:55 smithi104 ceph-mon[97044]: pgmap v7638: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:27:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:56.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:56.255615+0000) 2022-09-10T08:27:57.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:56.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:56.846700+0000) 2022-09-10T08:27:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:57.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:57.255739+0000) 2022-09-10T08:27:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:57 smithi110 ceph-mon[97633]: pgmap v7639: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:58.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:57.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:57.846905+0000) 2022-09-10T08:27:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:57 smithi104 ceph-mon[97044]: pgmap v7639: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:27:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:58.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:58.255900+0000) 2022-09-10T08:27:59.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:58.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:58.847073+0000) 2022-09-10T08:27:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:27:59.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:59.256033+0000) 2022-09-10T08:28:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:27:59 smithi110 ceph-mon[97633]: pgmap v7640: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:00.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:27:59.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:27:59.847172+0000) 2022-09-10T08:28:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:27:59 smithi104 ceph-mon[97044]: pgmap v7640: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:00.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:00.256200+0000) 2022-09-10T08:28:01.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:00.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:00.847380+0000) 2022-09-10T08:28:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:01.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:01.256324+0000) 2022-09-10T08:28:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:01 smithi110 ceph-mon[97633]: pgmap v7641: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:02.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:01.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:01.847558+0000) 2022-09-10T08:28:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:01 smithi104 ceph-mon[97044]: pgmap v7641: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:02.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:02.256491+0000) 2022-09-10T08:28:03.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:02.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:02.847702+0000) 2022-09-10T08:28:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:03.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:03.256641+0000) 2022-09-10T08:28:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:03 smithi110 ceph-mon[97633]: pgmap v7642: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:04.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:03.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:03.847831+0000) 2022-09-10T08:28:04.258 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:28:03.992+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:28:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:03 smithi104 ceph-mon[97044]: pgmap v7642: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:04.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:04.256882+0000) 2022-09-10T08:28:05.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:04.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:04.848010+0000) 2022-09-10T08:28:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:05.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:05.257137+0000) 2022-09-10T08:28:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:05 smithi110 ceph-mon[97633]: pgmap v7643: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:06.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:05.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:05.848234+0000) 2022-09-10T08:28:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:05 smithi104 ceph-mon[97044]: pgmap v7643: 129 pgs: 129 active+clean; 8.5 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:06.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:06.257290+0000) 2022-09-10T08:28:07.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:06.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:06.848368+0000) 2022-09-10T08:28:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:07.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:07.257394+0000) 2022-09-10T08:28:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:07 smithi110 ceph-mon[97633]: pgmap v7644: 129 pgs: 129 active+clean; 8.5 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:28:08.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:07.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:07.848520+0000) 2022-09-10T08:28:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:07 smithi104 ceph-mon[97044]: pgmap v7644: 129 pgs: 129 active+clean; 8.5 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:28:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:08.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:08.257538+0000) 2022-09-10T08:28:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:08 smithi110 ceph-mon[97633]: osdmap e558: 12 total, 12 up, 12 in 2022-09-10T08:28:09.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:08.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:08.848640+0000) 2022-09-10T08:28:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:08 smithi104 ceph-mon[97044]: osdmap e558: 12 total, 12 up, 12 in 2022-09-10T08:28:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:09.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:09.257732+0000) 2022-09-10T08:28:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:09 smithi110 ceph-mon[97633]: pgmap v7646: 129 pgs: 129 active+clean; 8.5 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:10.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:09.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:09.848799+0000) 2022-09-10T08:28:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:09 smithi104 ceph-mon[97044]: pgmap v7646: 129 pgs: 129 active+clean; 8.5 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:10.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:10.257875+0000) 2022-09-10T08:28:11.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:10.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:10.848973+0000) 2022-09-10T08:28:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:11.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:11.258022+0000) 2022-09-10T08:28:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:11 smithi110 ceph-mon[97633]: pgmap v7647: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:12.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:11.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:11.849108+0000) 2022-09-10T08:28:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:11 smithi104 ceph-mon[97044]: pgmap v7647: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:12.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:12.258155+0000) 2022-09-10T08:28:13.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:12.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:12.849222+0000) 2022-09-10T08:28:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:13.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:13.258378+0000) 2022-09-10T08:28:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:13 smithi110 ceph-mon[97633]: pgmap v7648: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:14.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:13.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:13.849390+0000) 2022-09-10T08:28:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:13 smithi104 ceph-mon[97044]: pgmap v7648: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:14.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:14.258522+0000) 2022-09-10T08:28:15.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:14.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:14.849516+0000) 2022-09-10T08:28:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:15.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:15.258726+0000) 2022-09-10T08:28:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:15 smithi110 ceph-mon[97633]: pgmap v7649: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:15 smithi110 ceph-mon[97633]: osdmap e559: 12 total, 12 up, 12 in 2022-09-10T08:28:16.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:15.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:15.849645+0000) 2022-09-10T08:28:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:15 smithi104 ceph-mon[97044]: pgmap v7649: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:28:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:15 smithi104 ceph-mon[97044]: osdmap e559: 12 total, 12 up, 12 in 2022-09-10T08:28:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:16.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:16.258968+0000) 2022-09-10T08:28:17.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:16.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:16.849806+0000) 2022-09-10T08:28:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:17.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:17.259227+0000) 2022-09-10T08:28:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:17 smithi110 ceph-mon[97633]: pgmap v7651: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:28:18.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:17.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:17.849927+0000) 2022-09-10T08:28:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:17 smithi104 ceph-mon[97044]: pgmap v7651: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:28:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:18.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:18.259414+0000) 2022-09-10T08:28:19.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:18.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:18.850070+0000) 2022-09-10T08:28:19.261 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:28:18.999+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:28:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:19.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:19.259644+0000) 2022-09-10T08:28:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:19 smithi110 ceph-mon[97633]: pgmap v7652: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:28:20.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:19.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:19.850257+0000) 2022-09-10T08:28:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:19 smithi104 ceph-mon[97044]: pgmap v7652: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:28:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:20.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:20.259931+0000) 2022-09-10T08:28:21.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:20.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:20.850400+0000) 2022-09-10T08:28:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:21.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:21.260191+0000) 2022-09-10T08:28:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:21 smithi110 ceph-mon[97633]: pgmap v7653: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:28:22.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:21.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:21.850541+0000) 2022-09-10T08:28:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:21 smithi104 ceph-mon[97044]: pgmap v7653: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:28:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:22.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:22.260361+0000) 2022-09-10T08:28:23.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:22.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:22.850667+0000) 2022-09-10T08:28:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:23.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:23.260562+0000) 2022-09-10T08:28:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:23 smithi110 ceph-mon[97633]: pgmap v7654: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:28:24.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:23.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:23.850815+0000) 2022-09-10T08:28:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:23 smithi104 ceph-mon[97044]: pgmap v7654: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:28:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:24.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:24.260794+0000) 2022-09-10T08:28:25.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:24.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:24.850955+0000) 2022-09-10T08:28:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:25.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:25.260970+0000) 2022-09-10T08:28:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:25 smithi110 ceph-mon[97633]: pgmap v7655: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:28:26.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:25.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:25.851078+0000) 2022-09-10T08:28:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:25 smithi104 ceph-mon[97044]: pgmap v7655: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:28:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:26.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:26.261229+0000) 2022-09-10T08:28:27.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:26.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:26.851159+0000) 2022-09-10T08:28:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:27.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:27.261475+0000) 2022-09-10T08:28:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:27 smithi110 ceph-mon[97633]: pgmap v7656: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:28:28.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:27.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:27.851328+0000) 2022-09-10T08:28:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:27 smithi104 ceph-mon[97044]: pgmap v7656: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:28:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:28.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:28.261730+0000) 2022-09-10T08:28:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:28 smithi110 ceph-mon[97633]: pgmap v7657: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:29.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:28.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:28.851518+0000) 2022-09-10T08:28:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:28 smithi104 ceph-mon[97044]: pgmap v7657: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:29.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:29.261980+0000) 2022-09-10T08:28:30.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:29.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:29.851681+0000) 2022-09-10T08:28:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:30.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:30.262173+0000) 2022-09-10T08:28:31.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:30.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:30.851821+0000) 2022-09-10T08:28:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:31.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:31.262355+0000) 2022-09-10T08:28:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:31 smithi104 ceph-mon[97044]: pgmap v7658: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:31 smithi110 ceph-mon[97633]: pgmap v7658: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:32.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:31.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:31.851983+0000) 2022-09-10T08:28:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:32.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:32.262584+0000) 2022-09-10T08:28:33.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:32.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:32.852122+0000) 2022-09-10T08:28:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:33.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:33.262824+0000) 2022-09-10T08:28:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:33 smithi104 ceph-mon[97044]: pgmap v7659: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:33 smithi110 ceph-mon[97633]: pgmap v7659: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:34.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:33.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:33.852258+0000) 2022-09-10T08:28:34.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:28:34.014+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:28:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:34.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:34.263070+0000) 2022-09-10T08:28:35.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:34.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:34.852404+0000) 2022-09-10T08:28:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:35.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:35.263265+0000) 2022-09-10T08:28:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:35 smithi104 ceph-mon[97044]: pgmap v7660: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:35 smithi110 ceph-mon[97633]: pgmap v7660: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:36.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:35.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:35.852538+0000) 2022-09-10T08:28:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:36.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:36.263501+0000) 2022-09-10T08:28:37.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:36.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:36.852714+0000) 2022-09-10T08:28:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:37.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:37.263684+0000) 2022-09-10T08:28:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:37 smithi104 ceph-mon[97044]: pgmap v7661: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:37 smithi110 ceph-mon[97633]: pgmap v7661: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:38.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:37.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:37.852869+0000) 2022-09-10T08:28:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:38.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:38.263879+0000) 2022-09-10T08:28:39.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:38.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:38.853021+0000) 2022-09-10T08:28:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:39.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:39.264072+0000) 2022-09-10T08:28:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:39 smithi104 ceph-mon[97044]: pgmap v7662: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:39 smithi110 ceph-mon[97633]: pgmap v7662: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:40.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:39.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:39.853158+0000) 2022-09-10T08:28:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:40.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:40.264314+0000) 2022-09-10T08:28:41.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:40.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:40.853282+0000) 2022-09-10T08:28:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:41.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:41.264521+0000) 2022-09-10T08:28:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:41 smithi110 ceph-mon[97633]: pgmap v7663: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:42.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:41.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:41.853412+0000) 2022-09-10T08:28:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:41 smithi104 ceph-mon[97044]: pgmap v7663: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:42.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:42.264744+0000) 2022-09-10T08:28:43.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:42.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:42.853530+0000) 2022-09-10T08:28:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:43.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:43.265000+0000) 2022-09-10T08:28:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:43 smithi110 ceph-mon[97633]: pgmap v7664: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:44.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:43.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:43.853668+0000) 2022-09-10T08:28:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:43 smithi104 ceph-mon[97044]: pgmap v7664: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:28:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:44.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:44.265179+0000) 2022-09-10T08:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:28:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:28:45.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:44.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:44.853786+0000) 2022-09-10T08:28:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:28:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:28:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:28:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:28:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:45.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:45.265389+0000) 2022-09-10T08:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:45 smithi110 ceph-mon[97633]: pgmap v7665: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T08:28:46.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:45.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:45.853911+0000) 2022-09-10T08:28:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:45 smithi104 ceph-mon[97044]: pgmap v7665: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T08:28:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:46.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:46.265556+0000) 2022-09-10T08:28:47.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:46.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:46.854036+0000) 2022-09-10T08:28:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:47.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:47.265730+0000) 2022-09-10T08:28:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:47 smithi110 ceph-mon[97633]: pgmap v7666: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:48.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:47.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:47.854164+0000) 2022-09-10T08:28:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:47 smithi104 ceph-mon[97044]: pgmap v7666: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:28:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:48.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:48.265922+0000) 2022-09-10T08:28:49.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:48.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:48.854287+0000) 2022-09-10T08:28:49.268 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:28:49.018+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:28:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:49.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:49.266154+0000) 2022-09-10T08:28:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:49 smithi110 ceph-mon[97633]: pgmap v7667: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:28:50.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:49.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:49.854412+0000) 2022-09-10T08:28:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:49 smithi104 ceph-mon[97044]: pgmap v7667: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:28:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:50.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:50.266323+0000) 2022-09-10T08:28:51.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:50.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:50.854536+0000) 2022-09-10T08:28:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:51.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:51.266591+0000) 2022-09-10T08:28:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:51 smithi110 ceph-mon[97633]: pgmap v7668: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T08:28:52.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:51.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:51.854670+0000) 2022-09-10T08:28:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:51 smithi104 ceph-mon[97044]: pgmap v7668: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T08:28:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:52.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:52.266788+0000) 2022-09-10T08:28:53.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:52.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:52.854784+0000) 2022-09-10T08:28:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:53.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:53.266948+0000) 2022-09-10T08:28:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:53 smithi110 ceph-mon[97633]: pgmap v7669: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:28:54.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:53.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:53.854908+0000) 2022-09-10T08:28:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:53 smithi104 ceph-mon[97044]: pgmap v7669: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:28:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:54.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:54.267160+0000) 2022-09-10T08:28:55.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:54.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:54.855035+0000) 2022-09-10T08:28:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:55.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:55.267303+0000) 2022-09-10T08:28:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:55 smithi110 ceph-mon[97633]: pgmap v7670: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T08:28:56.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:55.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:55.855180+0000) 2022-09-10T08:28:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:55 smithi104 ceph-mon[97044]: pgmap v7670: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T08:28:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:56.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:56.267481+0000) 2022-09-10T08:28:57.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:56.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:56.855302+0000) 2022-09-10T08:28:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:57.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:57.267683+0000) 2022-09-10T08:28:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:57 smithi110 ceph-mon[97633]: pgmap v7671: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2022-09-10T08:28:58.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:57.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:57.855424+0000) 2022-09-10T08:28:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:57 smithi104 ceph-mon[97044]: pgmap v7671: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2022-09-10T08:28:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:58.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:58.267865+0000) 2022-09-10T08:28:59.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:58.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:58.855549+0000) 2022-09-10T08:28:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:28:59.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:59.268008+0000) 2022-09-10T08:29:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:28:59 smithi110 ceph-mon[97633]: pgmap v7672: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T08:29:00.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:28:59.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:28:59.855688+0000) 2022-09-10T08:29:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:28:59 smithi104 ceph-mon[97044]: pgmap v7672: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T08:29:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:00.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:00.268205+0000) 2022-09-10T08:29:01.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:00.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:00.855844+0000) 2022-09-10T08:29:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:01.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:01.268461+0000) 2022-09-10T08:29:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:01 smithi110 ceph-mon[97633]: pgmap v7673: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T08:29:02.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:01.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:01.855983+0000) 2022-09-10T08:29:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:01 smithi104 ceph-mon[97044]: pgmap v7673: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T08:29:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:02.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:02.268708+0000) 2022-09-10T08:29:03.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:02.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:02.856136+0000) 2022-09-10T08:29:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:03.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:03.268872+0000) 2022-09-10T08:29:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:03 smithi110 ceph-mon[97633]: pgmap v7674: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:04.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:03.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:03.856265+0000) 2022-09-10T08:29:04.270 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:29:04.033+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:29:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:03 smithi104 ceph-mon[97044]: pgmap v7674: 129 pgs: 129 active+clean; 8.5 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:04.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:04.269038+0000) 2022-09-10T08:29:05.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:04.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:04.856402+0000) 2022-09-10T08:29:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:05.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:05.269219+0000) 2022-09-10T08:29:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:05 smithi110 ceph-mon[97633]: pgmap v7675: 129 pgs: 129 active+clean; 8.5 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2022-09-10T08:29:06.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:05.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:05.856543+0000) 2022-09-10T08:29:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:05 smithi104 ceph-mon[97044]: pgmap v7675: 129 pgs: 129 active+clean; 8.5 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 KiB/s wr, 13 op/s 2022-09-10T08:29:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:06.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:06.269439+0000) 2022-09-10T08:29:07.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:06.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:06.856665+0000) 2022-09-10T08:29:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:07.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:07.269671+0000) 2022-09-10T08:29:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:07 smithi110 ceph-mon[97633]: pgmap v7676: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T08:29:08.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:07.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:07.856807+0000) 2022-09-10T08:29:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:07 smithi104 ceph-mon[97044]: pgmap v7676: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T08:29:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:08.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:08.269860+0000) 2022-09-10T08:29:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:08 smithi110 ceph-mon[97633]: osdmap e560: 12 total, 12 up, 12 in 2022-09-10T08:29:09.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:08.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:08.856929+0000) 2022-09-10T08:29:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:08 smithi104 ceph-mon[97044]: osdmap e560: 12 total, 12 up, 12 in 2022-09-10T08:29:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:09.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:09.270123+0000) 2022-09-10T08:29:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:09 smithi110 ceph-mon[97633]: pgmap v7678: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:29:10.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:09.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:09.857055+0000) 2022-09-10T08:29:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:09 smithi104 ceph-mon[97044]: pgmap v7678: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:29:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:10.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:10.270334+0000) 2022-09-10T08:29:11.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:10.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:10.857176+0000) 2022-09-10T08:29:11.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:11.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:11.270578+0000) 2022-09-10T08:29:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:11 smithi110 ceph-mon[97633]: pgmap v7679: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:29:12.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:11.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:11.857308+0000) 2022-09-10T08:29:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:11 smithi104 ceph-mon[97044]: pgmap v7679: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:29:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:12.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:12.270838+0000) 2022-09-10T08:29:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:12 smithi110 ceph-mon[97633]: pgmap v7680: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:29:13.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:12.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:12.857440+0000) 2022-09-10T08:29:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:12 smithi104 ceph-mon[97044]: pgmap v7680: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:29:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:13.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:13.271083+0000) 2022-09-10T08:29:14.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:13.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:13.857566+0000) 2022-09-10T08:29:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:14.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:14.271323+0000) 2022-09-10T08:29:15.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:14.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:14.857700+0000) 2022-09-10T08:29:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:15.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:15.271476+0000) 2022-09-10T08:29:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:15 smithi104 ceph-mon[97044]: pgmap v7681: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 25 op/s 2022-09-10T08:29:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:15 smithi110 ceph-mon[97633]: pgmap v7681: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 25 op/s 2022-09-10T08:29:16.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:15.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:15.857827+0000) 2022-09-10T08:29:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:16.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:16.271722+0000) 2022-09-10T08:29:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:16 smithi104 ceph-mon[97044]: osdmap e561: 12 total, 12 up, 12 in 2022-09-10T08:29:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:16 smithi110 ceph-mon[97633]: osdmap e561: 12 total, 12 up, 12 in 2022-09-10T08:29:17.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:16.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:16.857942+0000) 2022-09-10T08:29:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:17.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:17.271876+0000) 2022-09-10T08:29:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:17 smithi110 ceph-mon[97633]: pgmap v7683: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:29:18.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:17.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:17.858131+0000) 2022-09-10T08:29:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:17 smithi104 ceph-mon[97044]: pgmap v7683: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:29:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:18.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:18.272041+0000) 2022-09-10T08:29:19.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:18.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:18.858283+0000) 2022-09-10T08:29:19.273 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:29:19.049+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:29:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:19.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:19.272260+0000) 2022-09-10T08:29:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:19 smithi110 ceph-mon[97633]: pgmap v7684: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:29:20.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:19.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:19.858446+0000) 2022-09-10T08:29:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:19 smithi104 ceph-mon[97044]: pgmap v7684: 129 pgs: 129 active+clean; 8.5 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:29:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:20.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:20.272471+0000) 2022-09-10T08:29:21.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:20.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:20.858591+0000) 2022-09-10T08:29:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:21.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:21.272688+0000) 2022-09-10T08:29:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:21 smithi110 ceph-mon[97633]: pgmap v7685: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:29:22.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:21.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:21.858730+0000) 2022-09-10T08:29:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:21 smithi104 ceph-mon[97044]: pgmap v7685: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:29:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:22.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:22.272886+0000) 2022-09-10T08:29:23.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:22.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:22.858858+0000) 2022-09-10T08:29:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:23.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:23.273074+0000) 2022-09-10T08:29:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:23 smithi110 ceph-mon[97633]: pgmap v7686: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:29:24.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:23.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:23.858982+0000) 2022-09-10T08:29:24.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:23 smithi104 ceph-mon[97044]: pgmap v7686: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:29:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:24.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:24.273252+0000) 2022-09-10T08:29:25.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:24.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:24.859107+0000) 2022-09-10T08:29:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:25.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:25.273426+0000) 2022-09-10T08:29:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:25 smithi110 ceph-mon[97633]: pgmap v7687: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:29:26.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:25.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:25.859242+0000) 2022-09-10T08:29:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:25 smithi104 ceph-mon[97044]: pgmap v7687: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:29:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:26.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:26.273677+0000) 2022-09-10T08:29:27.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:26.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:26.859375+0000) 2022-09-10T08:29:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:27.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:27.273852+0000) 2022-09-10T08:29:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:27 smithi110 ceph-mon[97633]: pgmap v7688: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:29:28.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:27.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:27.859528+0000) 2022-09-10T08:29:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:27 smithi104 ceph-mon[97044]: pgmap v7688: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:29:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:28.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:28.274080+0000) 2022-09-10T08:29:29.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:28.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:28.859715+0000) 2022-09-10T08:29:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:29.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:29.274219+0000) 2022-09-10T08:29:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:29 smithi110 ceph-mon[97633]: pgmap v7689: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:30.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:29.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:29.859879+0000) 2022-09-10T08:29:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:29 smithi104 ceph-mon[97044]: pgmap v7689: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:30.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:30.274394+0000) 2022-09-10T08:29:31.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:30.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:30.860020+0000) 2022-09-10T08:29:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:31.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:31.274632+0000) 2022-09-10T08:29:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:31 smithi110 ceph-mon[97633]: pgmap v7690: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:29:32.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:31.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:31.860179+0000) 2022-09-10T08:29:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:31 smithi104 ceph-mon[97044]: pgmap v7690: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:29:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:32.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:32.274898+0000) 2022-09-10T08:29:33.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:32.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:32.860315+0000) 2022-09-10T08:29:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:33.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:33.275061+0000) 2022-09-10T08:29:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:33 smithi110 ceph-mon[97633]: pgmap v7691: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:34.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:33.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:33.860448+0000) 2022-09-10T08:29:34.276 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:29:34.061+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:29:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:33 smithi104 ceph-mon[97044]: pgmap v7691: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:34.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:34.275254+0000) 2022-09-10T08:29:35.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:34.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:34.860588+0000) 2022-09-10T08:29:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:35.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:35.275416+0000) 2022-09-10T08:29:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:35 smithi110 ceph-mon[97633]: pgmap v7692: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T08:29:36.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:35.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:35.860739+0000) 2022-09-10T08:29:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:35 smithi104 ceph-mon[97044]: pgmap v7692: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T08:29:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:36.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:36.275686+0000) 2022-09-10T08:29:37.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:36.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:36.860879+0000) 2022-09-10T08:29:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:37.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:37.275923+0000) 2022-09-10T08:29:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:37 smithi110 ceph-mon[97633]: pgmap v7693: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T08:29:38.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:37.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:37.861015+0000) 2022-09-10T08:29:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:37 smithi104 ceph-mon[97044]: pgmap v7693: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T08:29:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:38.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:38.276203+0000) 2022-09-10T08:29:39.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:38.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:38.861164+0000) 2022-09-10T08:29:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:39.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:39.276364+0000) 2022-09-10T08:29:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:39 smithi110 ceph-mon[97633]: pgmap v7694: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:40.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:39.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:39.861307+0000) 2022-09-10T08:29:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:39 smithi104 ceph-mon[97044]: pgmap v7694: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:40.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:40.276538+0000) 2022-09-10T08:29:41.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:40.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:40.861449+0000) 2022-09-10T08:29:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:41.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:41.276693+0000) 2022-09-10T08:29:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:41 smithi110 ceph-mon[97633]: pgmap v7695: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:29:42.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:41.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:41.861603+0000) 2022-09-10T08:29:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:41 smithi104 ceph-mon[97044]: pgmap v7695: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:29:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:42.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:42.276919+0000) 2022-09-10T08:29:43.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:42.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:42.861788+0000) 2022-09-10T08:29:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:43.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:43.277120+0000) 2022-09-10T08:29:44.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:43 smithi110 ceph-mon[97633]: pgmap v7696: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:44.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:43.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:43.861937+0000) 2022-09-10T08:29:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:43 smithi104 ceph-mon[97044]: pgmap v7696: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:44.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:44.277353+0000) 2022-09-10T08:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:29:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:29:45.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:44.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:44.862070+0000) 2022-09-10T08:29:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:29:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:29:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:29:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:29:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:45.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:45.277555+0000) 2022-09-10T08:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:45 smithi110 ceph-mon[97633]: pgmap v7697: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T08:29:46.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:45.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:45.862178+0000) 2022-09-10T08:29:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:45 smithi104 ceph-mon[97044]: pgmap v7697: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T08:29:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:46.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:46.277779+0000) 2022-09-10T08:29:47.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:46.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:46.862313+0000) 2022-09-10T08:29:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:47.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:47.278012+0000) 2022-09-10T08:29:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:47 smithi110 ceph-mon[97633]: pgmap v7698: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T08:29:48.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:47.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:47.862453+0000) 2022-09-10T08:29:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:47 smithi104 ceph-mon[97044]: pgmap v7698: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T08:29:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:48.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:48.278262+0000) 2022-09-10T08:29:49.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:48.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:48.862569+0000) 2022-09-10T08:29:49.279 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:29:49.072+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:29:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:49.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:49.278508+0000) 2022-09-10T08:29:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:49 smithi110 ceph-mon[97633]: pgmap v7699: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:50.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:49.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:49.862694+0000) 2022-09-10T08:29:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:49 smithi104 ceph-mon[97044]: pgmap v7699: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:50.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:50.278691+0000) 2022-09-10T08:29:51.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:50.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:50.862822+0000) 2022-09-10T08:29:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:51.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:51.278944+0000) 2022-09-10T08:29:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:51 smithi110 ceph-mon[97633]: pgmap v7700: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:29:52.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:51.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:51.862956+0000) 2022-09-10T08:29:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:51 smithi104 ceph-mon[97044]: pgmap v7700: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:29:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:52.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:52.279203+0000) 2022-09-10T08:29:53.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:52.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:52.863076+0000) 2022-09-10T08:29:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:53.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:53.279436+0000) 2022-09-10T08:29:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:53 smithi110 ceph-mon[97633]: pgmap v7701: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:54.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:53.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:53.863194+0000) 2022-09-10T08:29:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:53 smithi104 ceph-mon[97044]: pgmap v7701: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:29:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:54.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:54.279592+0000) 2022-09-10T08:29:55.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:54.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:54.863315+0000) 2022-09-10T08:29:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:55.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:55.279841+0000) 2022-09-10T08:29:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:55 smithi110 ceph-mon[97633]: pgmap v7702: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T08:29:56.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:55.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:55.863441+0000) 2022-09-10T08:29:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:55 smithi104 ceph-mon[97044]: pgmap v7702: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T08:29:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:56.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:56.280083+0000) 2022-09-10T08:29:57.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:56.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:56.863554+0000) 2022-09-10T08:29:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:57.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:57.280345+0000) 2022-09-10T08:29:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:57 smithi110 ceph-mon[97633]: pgmap v7703: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T08:29:58.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:57.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:57.863682+0000) 2022-09-10T08:29:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:57 smithi104 ceph-mon[97044]: pgmap v7703: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T08:29:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:58.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:58.280537+0000) 2022-09-10T08:29:59.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:58.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:58.863860+0000) 2022-09-10T08:29:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:29:59.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:59.280736+0000) 2022-09-10T08:30:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:29:59 smithi110 ceph-mon[97633]: pgmap v7704: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:00.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:29:59.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:29:59.864011+0000) 2022-09-10T08:30:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:29:59 smithi104 ceph-mon[97044]: pgmap v7704: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:00.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:00.280955+0000) 2022-09-10T08:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:00 smithi110 ceph-mon[97633]: pgmap v7705: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:01.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:00.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:00.864177+0000) 2022-09-10T08:30:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:30:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:00 smithi104 ceph-mon[97044]: pgmap v7705: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:01.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:01.281235+0000) 2022-09-10T08:30:02.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:01.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:01.864362+0000) 2022-09-10T08:30:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:02.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:02.281445+0000) 2022-09-10T08:30:03.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:02.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:02.864495+0000) 2022-09-10T08:30:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:03.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:03.281666+0000) 2022-09-10T08:30:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:03 smithi104 ceph-mon[97044]: pgmap v7706: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:03 smithi110 ceph-mon[97633]: pgmap v7706: 129 pgs: 129 active+clean; 8.5 MiB data, 800 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:04.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:03.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:03.864631+0000) 2022-09-10T08:30:04.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:30:04.086+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:30:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:04.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:04.281910+0000) 2022-09-10T08:30:05.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:04.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:04.864747+0000) 2022-09-10T08:30:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:05.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:05.282183+0000) 2022-09-10T08:30:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:05 smithi104 ceph-mon[97044]: pgmap v7707: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:30:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:05 smithi110 ceph-mon[97633]: pgmap v7707: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:30:06.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:05.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:05.864901+0000) 2022-09-10T08:30:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:06.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:06.282423+0000) 2022-09-10T08:30:07.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:06.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:06.865062+0000) 2022-09-10T08:30:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:07.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:07.282657+0000) 2022-09-10T08:30:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:07 smithi110 ceph-mon[97633]: pgmap v7708: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T08:30:08.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:07.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:07.865195+0000) 2022-09-10T08:30:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:07 smithi104 ceph-mon[97044]: pgmap v7708: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T08:30:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:08.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:08.282917+0000) 2022-09-10T08:30:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:08 smithi110 ceph-mon[97633]: osdmap e562: 12 total, 12 up, 12 in 2022-09-10T08:30:09.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:08.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:08.865317+0000) 2022-09-10T08:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:08 smithi104 ceph-mon[97044]: osdmap e562: 12 total, 12 up, 12 in 2022-09-10T08:30:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:09.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:09.283177+0000) 2022-09-10T08:30:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:09 smithi110 ceph-mon[97633]: pgmap v7710: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:30:10.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:09.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:09.865522+0000) 2022-09-10T08:30:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:09 smithi104 ceph-mon[97044]: pgmap v7710: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:30:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:10.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:10.283427+0000) 2022-09-10T08:30:11.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:10.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:10.865652+0000) 2022-09-10T08:30:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:11.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:11.283643+0000) 2022-09-10T08:30:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:11 smithi110 ceph-mon[97633]: pgmap v7711: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:30:12.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:11.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:11.865826+0000) 2022-09-10T08:30:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:11 smithi104 ceph-mon[97044]: pgmap v7711: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:30:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:12.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:12.283870+0000) 2022-09-10T08:30:13.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:12.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:12.865947+0000) 2022-09-10T08:30:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:13.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:13.284147+0000) 2022-09-10T08:30:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:13 smithi110 ceph-mon[97633]: pgmap v7712: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:30:14.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:13.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:13.866124+0000) 2022-09-10T08:30:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:13 smithi104 ceph-mon[97044]: pgmap v7712: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:30:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:14.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:14.284338+0000) 2022-09-10T08:30:15.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:14.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:14.866283+0000) 2022-09-10T08:30:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:15.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:15.284557+0000) 2022-09-10T08:30:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:15 smithi110 ceph-mon[97633]: pgmap v7713: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:30:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:15 smithi110 ceph-mon[97633]: osdmap e563: 12 total, 12 up, 12 in 2022-09-10T08:30:16.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:15.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:15.866479+0000) 2022-09-10T08:30:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:15 smithi104 ceph-mon[97044]: pgmap v7713: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:30:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:15 smithi104 ceph-mon[97044]: osdmap e563: 12 total, 12 up, 12 in 2022-09-10T08:30:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:16.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:16.284778+0000) 2022-09-10T08:30:17.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:16.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:16.866607+0000) 2022-09-10T08:30:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:17.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:17.284942+0000) 2022-09-10T08:30:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:17 smithi110 ceph-mon[97633]: pgmap v7715: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:30:18.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:17.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:17.866781+0000) 2022-09-10T08:30:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:17 smithi104 ceph-mon[97044]: pgmap v7715: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:30:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:18.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:18.285234+0000) 2022-09-10T08:30:19.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:18.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:18.866981+0000) 2022-09-10T08:30:19.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:30:19.098+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:30:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:19.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:19.285470+0000) 2022-09-10T08:30:20.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:19 smithi110 ceph-mon[97633]: pgmap v7716: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:30:20.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:19.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:19.867157+0000) 2022-09-10T08:30:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:19 smithi104 ceph-mon[97044]: pgmap v7716: 129 pgs: 129 active+clean; 8.5 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:30:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:20.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:20.285707+0000) 2022-09-10T08:30:21.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:20.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:20.867282+0000) 2022-09-10T08:30:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:21.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:21.285981+0000) 2022-09-10T08:30:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:21 smithi110 ceph-mon[97633]: pgmap v7717: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:30:22.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:21.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:21.867421+0000) 2022-09-10T08:30:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:21 smithi104 ceph-mon[97044]: pgmap v7717: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:30:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:22.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:22.286239+0000) 2022-09-10T08:30:23.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:22.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:22.867562+0000) 2022-09-10T08:30:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:23.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:23.286471+0000) 2022-09-10T08:30:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:23 smithi110 ceph-mon[97633]: pgmap v7718: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:30:24.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:23.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:23.867698+0000) 2022-09-10T08:30:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:23 smithi104 ceph-mon[97044]: pgmap v7718: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:30:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:24.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:24.286732+0000) 2022-09-10T08:30:25.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:24.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:24.867829+0000) 2022-09-10T08:30:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:25.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:25.286971+0000) 2022-09-10T08:30:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:25 smithi110 ceph-mon[97633]: pgmap v7719: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:30:26.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:25.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:25.867956+0000) 2022-09-10T08:30:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:25 smithi104 ceph-mon[97044]: pgmap v7719: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:30:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:26.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:26.287194+0000) 2022-09-10T08:30:27.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:26.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:26.868081+0000) 2022-09-10T08:30:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:27.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:27.287413+0000) 2022-09-10T08:30:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:27 smithi110 ceph-mon[97633]: pgmap v7720: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:30:28.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:27.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:27.868231+0000) 2022-09-10T08:30:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:27 smithi104 ceph-mon[97044]: pgmap v7720: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:30:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:28.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:28.287660+0000) 2022-09-10T08:30:29.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:28.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:28.868358+0000) 2022-09-10T08:30:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:29.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:29.287926+0000) 2022-09-10T08:30:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:29 smithi110 ceph-mon[97633]: pgmap v7721: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:30.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:29.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:29.868491+0000) 2022-09-10T08:30:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:29 smithi104 ceph-mon[97044]: pgmap v7721: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:30.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:30.288188+0000) 2022-09-10T08:30:31.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:30.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:30.868626+0000) 2022-09-10T08:30:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:31.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:31.288348+0000) 2022-09-10T08:30:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:31 smithi110 ceph-mon[97633]: pgmap v7722: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:32.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:31.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:31.868766+0000) 2022-09-10T08:30:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:31 smithi104 ceph-mon[97044]: pgmap v7722: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:32.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:32.288585+0000) 2022-09-10T08:30:33.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:32.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:32.868890+0000) 2022-09-10T08:30:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:33.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:33.288816+0000) 2022-09-10T08:30:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:33 smithi110 ceph-mon[97633]: pgmap v7723: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:34.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:33.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:33.869025+0000) 2022-09-10T08:30:34.290 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:30:34.114+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:30:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:33 smithi104 ceph-mon[97044]: pgmap v7723: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:34.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:34.289045+0000) 2022-09-10T08:30:35.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:34.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:34.869165+0000) 2022-09-10T08:30:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:35.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:35.289259+0000) 2022-09-10T08:30:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:35 smithi110 ceph-mon[97633]: pgmap v7724: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:36.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:35.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:35.869282+0000) 2022-09-10T08:30:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:35 smithi104 ceph-mon[97044]: pgmap v7724: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:36.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:36.289524+0000) 2022-09-10T08:30:37.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:36.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:36.869433+0000) 2022-09-10T08:30:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:37.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:37.289720+0000) 2022-09-10T08:30:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:37 smithi110 ceph-mon[97633]: pgmap v7725: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:38.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:37.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:37.869566+0000) 2022-09-10T08:30:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:37 smithi104 ceph-mon[97044]: pgmap v7725: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:38.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:38.289959+0000) 2022-09-10T08:30:39.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:38.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:38.869704+0000) 2022-09-10T08:30:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:39.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:39.290159+0000) 2022-09-10T08:30:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:39 smithi110 ceph-mon[97633]: pgmap v7726: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:40.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:39.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:39.869850+0000) 2022-09-10T08:30:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:39 smithi104 ceph-mon[97044]: pgmap v7726: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:40.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:40.290288+0000) 2022-09-10T08:30:41.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:40.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:40.869985+0000) 2022-09-10T08:30:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:41.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:41.290530+0000) 2022-09-10T08:30:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:41 smithi110 ceph-mon[97633]: pgmap v7727: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:42.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:41.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:41.870152+0000) 2022-09-10T08:30:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:41 smithi104 ceph-mon[97044]: pgmap v7727: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:42.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:42.290709+0000) 2022-09-10T08:30:43.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:42.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:42.870351+0000) 2022-09-10T08:30:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:43.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:43.290852+0000) 2022-09-10T08:30:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:43 smithi110 ceph-mon[97633]: pgmap v7728: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:44.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:43.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:43.870511+0000) 2022-09-10T08:30:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:43 smithi104 ceph-mon[97044]: pgmap v7728: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:44.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:44.291032+0000) 2022-09-10T08:30:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:30:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:30:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:30:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:30:45.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:44.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:44.870632+0000) 2022-09-10T08:30:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:30:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:30:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:30:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:30:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:45.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:45.291195+0000) 2022-09-10T08:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:45 smithi110 ceph-mon[97633]: pgmap v7729: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:46.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:45.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:45.870764+0000) 2022-09-10T08:30:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:45 smithi104 ceph-mon[97044]: pgmap v7729: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:46.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:46.291470+0000) 2022-09-10T08:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:46 smithi110 ceph-mon[97633]: pgmap v7730: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:47.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:46.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:46.870936+0000) 2022-09-10T08:30:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:46 smithi104 ceph-mon[97044]: pgmap v7730: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:47.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:47.291658+0000) 2022-09-10T08:30:48.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:47.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:47.871112+0000) 2022-09-10T08:30:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:48.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:48.291892+0000) 2022-09-10T08:30:49.128 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:48.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:48.871275+0000) 2022-09-10T08:30:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:49.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:49.292091+0000) 2022-09-10T08:30:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:30:49.126+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:30:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:49 smithi104 ceph-mon[97044]: pgmap v7731: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:49 smithi110 ceph-mon[97633]: pgmap v7731: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:50.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:49.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:49.871419+0000) 2022-09-10T08:30:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:50.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:50.292352+0000) 2022-09-10T08:30:51.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:50.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:50.871540+0000) 2022-09-10T08:30:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:51.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:51.292553+0000) 2022-09-10T08:30:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:51 smithi110 ceph-mon[97633]: pgmap v7732: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:52.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:51.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:51.871679+0000) 2022-09-10T08:30:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:51 smithi104 ceph-mon[97044]: pgmap v7732: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:52.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:52.292806+0000) 2022-09-10T08:30:53.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:52.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:52.871797+0000) 2022-09-10T08:30:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:53.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:53.293064+0000) 2022-09-10T08:30:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:53 smithi110 ceph-mon[97633]: pgmap v7733: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:54.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:53.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:53.871939+0000) 2022-09-10T08:30:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:53 smithi104 ceph-mon[97044]: pgmap v7733: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:54.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:54.293273+0000) 2022-09-10T08:30:55.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:54.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:54.872067+0000) 2022-09-10T08:30:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:55.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:55.293467+0000) 2022-09-10T08:30:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:55 smithi110 ceph-mon[97633]: pgmap v7734: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:56.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:55.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:55.872161+0000) 2022-09-10T08:30:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:55 smithi104 ceph-mon[97044]: pgmap v7734: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:30:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:56.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:56.293649+0000) 2022-09-10T08:30:57.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:56.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:56.872297+0000) 2022-09-10T08:30:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:57.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:57.293859+0000) 2022-09-10T08:30:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:57 smithi110 ceph-mon[97633]: pgmap v7735: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:58.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:57.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:57.872486+0000) 2022-09-10T08:30:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:57 smithi104 ceph-mon[97044]: pgmap v7735: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:30:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:58.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:58.294155+0000) 2022-09-10T08:30:59.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:58.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:58.872644+0000) 2022-09-10T08:30:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:30:59.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:59.294393+0000) 2022-09-10T08:31:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:30:59 smithi110 ceph-mon[97633]: pgmap v7736: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:00.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:30:59.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:30:59.872783+0000) 2022-09-10T08:31:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:30:59 smithi104 ceph-mon[97044]: pgmap v7736: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:00.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:00.294602+0000) 2022-09-10T08:31:01.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:00.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:00.872923+0000) 2022-09-10T08:31:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:01.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:01.294829+0000) 2022-09-10T08:31:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:01 smithi110 ceph-mon[97633]: pgmap v7737: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:02.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:01.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:01.873077+0000) 2022-09-10T08:31:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:01 smithi104 ceph-mon[97044]: pgmap v7737: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:02.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:02.295120+0000) 2022-09-10T08:31:03.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:02.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:02.873159+0000) 2022-09-10T08:31:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:03.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:03.295308+0000) 2022-09-10T08:31:04.140 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:03.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:03.873283+0000) 2022-09-10T08:31:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:03 smithi110 ceph-mon[97633]: pgmap v7738: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:03 smithi104 ceph-mon[97044]: pgmap v7738: 129 pgs: 129 active+clean; 8.5 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:04.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:04.295538+0000) 2022-09-10T08:31:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:31:04.138+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:31:05.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:04.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:04.873410+0000) 2022-09-10T08:31:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:05.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:05.295750+0000) 2022-09-10T08:31:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:05 smithi110 ceph-mon[97633]: pgmap v7739: 129 pgs: 129 active+clean; 8.5 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:31:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:05 smithi104 ceph-mon[97044]: pgmap v7739: 129 pgs: 129 active+clean; 8.5 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:31:06.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:05.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:05.873536+0000) 2022-09-10T08:31:06.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:06.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:06.295985+0000) 2022-09-10T08:31:07.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:06.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:06.873723+0000) 2022-09-10T08:31:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:07.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:07.296179+0000) 2022-09-10T08:31:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:07 smithi110 ceph-mon[97633]: pgmap v7740: 129 pgs: 129 active+clean; 8.5 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:31:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:07 smithi104 ceph-mon[97044]: pgmap v7740: 129 pgs: 129 active+clean; 8.5 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:31:08.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:07.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:07.873880+0000) 2022-09-10T08:31:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:08.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:08.296412+0000) 2022-09-10T08:31:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:08 smithi110 ceph-mon[97633]: osdmap e564: 12 total, 12 up, 12 in 2022-09-10T08:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:08 smithi104 ceph-mon[97044]: osdmap e564: 12 total, 12 up, 12 in 2022-09-10T08:31:09.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:08.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:08.874089+0000) 2022-09-10T08:31:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:09.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:09.296618+0000) 2022-09-10T08:31:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:09 smithi110 ceph-mon[97633]: pgmap v7742: 129 pgs: 129 active+clean; 8.5 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:31:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:09 smithi104 ceph-mon[97044]: pgmap v7742: 129 pgs: 129 active+clean; 8.5 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:31:10.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:09.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:09.874315+0000) 2022-09-10T08:31:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:10.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:10.296861+0000) 2022-09-10T08:31:11.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:10.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:10.874460+0000) 2022-09-10T08:31:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:11.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:11.297031+0000) 2022-09-10T08:31:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:11 smithi110 ceph-mon[97633]: pgmap v7743: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:31:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:11 smithi104 ceph-mon[97044]: pgmap v7743: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:31:12.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:11.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:11.874595+0000) 2022-09-10T08:31:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:12.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:12.297283+0000) 2022-09-10T08:31:13.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:12.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:12.874723+0000) 2022-09-10T08:31:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:13.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:13.297505+0000) 2022-09-10T08:31:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:13 smithi110 ceph-mon[97633]: pgmap v7744: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:31:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:13 smithi104 ceph-mon[97044]: pgmap v7744: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:31:14.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:13.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:13.874862+0000) 2022-09-10T08:31:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:14.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:14.297754+0000) 2022-09-10T08:31:15.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:14.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:14.875010+0000) 2022-09-10T08:31:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:15.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:15.297986+0000) 2022-09-10T08:31:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:15 smithi110 ceph-mon[97633]: pgmap v7745: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:31:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:15 smithi110 ceph-mon[97633]: osdmap e565: 12 total, 12 up, 12 in 2022-09-10T08:31:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:15 smithi104 ceph-mon[97044]: pgmap v7745: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:31:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:15 smithi104 ceph-mon[97044]: osdmap e565: 12 total, 12 up, 12 in 2022-09-10T08:31:16.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:15.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:15.875165+0000) 2022-09-10T08:31:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:16.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:16.298264+0000) 2022-09-10T08:31:17.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:16.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:16.875350+0000) 2022-09-10T08:31:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:17.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:17.298499+0000) 2022-09-10T08:31:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:17 smithi110 ceph-mon[97633]: pgmap v7747: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:31:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:17 smithi104 ceph-mon[97044]: pgmap v7747: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:31:18.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:17.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:17.875567+0000) 2022-09-10T08:31:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:18.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:18.298731+0000) 2022-09-10T08:31:19.156 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:18.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:18.875727+0000) 2022-09-10T08:31:19.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:31:19.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:31:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:19.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:19.298974+0000) 2022-09-10T08:31:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:19 smithi110 ceph-mon[97633]: pgmap v7748: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:31:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:19 smithi104 ceph-mon[97044]: pgmap v7748: 129 pgs: 129 active+clean; 8.5 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:31:20.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:19.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:19.875862+0000) 2022-09-10T08:31:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:20.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:20.299192+0000) 2022-09-10T08:31:21.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:20.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:20.876052+0000) 2022-09-10T08:31:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:21.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:21.299426+0000) 2022-09-10T08:31:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:21 smithi110 ceph-mon[97633]: pgmap v7749: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:31:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:21 smithi104 ceph-mon[97044]: pgmap v7749: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:31:22.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:21.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:21.876247+0000) 2022-09-10T08:31:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:22.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:22.299650+0000) 2022-09-10T08:31:23.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:22.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:22.876422+0000) 2022-09-10T08:31:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:23.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:23.299842+0000) 2022-09-10T08:31:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:23 smithi110 ceph-mon[97633]: pgmap v7750: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:31:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:23 smithi104 ceph-mon[97044]: pgmap v7750: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:31:24.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:23.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:23.876562+0000) 2022-09-10T08:31:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:24.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:24.300074+0000) 2022-09-10T08:31:25.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:24.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:24.876685+0000) 2022-09-10T08:31:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:25.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:25.300328+0000) 2022-09-10T08:31:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:25 smithi110 ceph-mon[97633]: pgmap v7751: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:31:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:25 smithi104 ceph-mon[97044]: pgmap v7751: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:31:26.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:25.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:25.876839+0000) 2022-09-10T08:31:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:26.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:26.300495+0000) 2022-09-10T08:31:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:26 smithi104 ceph-mon[97044]: pgmap v7752: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:31:27.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:26.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:26.876979+0000) 2022-09-10T08:31:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:26 smithi110 ceph-mon[97633]: pgmap v7752: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:31:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:27.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:27.300713+0000) 2022-09-10T08:31:28.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:27.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:27.877118+0000) 2022-09-10T08:31:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:28.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:28.300932+0000) 2022-09-10T08:31:29.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:28.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:28.877267+0000) 2022-09-10T08:31:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:29.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:29.301196+0000) 2022-09-10T08:31:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:29 smithi104 ceph-mon[97044]: pgmap v7753: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:29 smithi110 ceph-mon[97633]: pgmap v7753: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:30.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:29.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:29.877394+0000) 2022-09-10T08:31:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:30.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:30.301435+0000) 2022-09-10T08:31:31.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:30.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:30.877523+0000) 2022-09-10T08:31:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:31.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:31.301717+0000) 2022-09-10T08:31:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:31 smithi110 ceph-mon[97633]: pgmap v7754: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:31 smithi104 ceph-mon[97044]: pgmap v7754: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:32.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:31.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:31.877664+0000) 2022-09-10T08:31:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:32.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:32.301926+0000) 2022-09-10T08:31:33.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:32.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:32.877793+0000) 2022-09-10T08:31:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:33.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:33.302202+0000) 2022-09-10T08:31:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:33 smithi110 ceph-mon[97633]: pgmap v7755: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:34.165 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:33.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:33.877930+0000) 2022-09-10T08:31:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:33 smithi104 ceph-mon[97044]: pgmap v7755: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:34.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:34.302384+0000) 2022-09-10T08:31:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:31:34.164+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:31:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:34.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:34.878074+0000) 2022-09-10T08:31:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:35.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:35.302574+0000) 2022-09-10T08:31:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:35 smithi110 ceph-mon[97633]: pgmap v7756: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:35 smithi104 ceph-mon[97044]: pgmap v7756: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:35.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:35.878282+0000) 2022-09-10T08:31:36.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:36.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:36.302827+0000) 2022-09-10T08:31:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:36.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:36.878438+0000) 2022-09-10T08:31:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:37.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:37.303071+0000) 2022-09-10T08:31:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:37 smithi110 ceph-mon[97633]: pgmap v7757: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:37 smithi104 ceph-mon[97044]: pgmap v7757: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:37.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:37.878577+0000) 2022-09-10T08:31:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:38.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:38.303332+0000) 2022-09-10T08:31:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:38.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:38.878784+0000) 2022-09-10T08:31:39.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:39.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:39.303554+0000) 2022-09-10T08:31:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:39 smithi110 ceph-mon[97633]: pgmap v7758: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:39 smithi104 ceph-mon[97044]: pgmap v7758: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:39.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:39.878916+0000) 2022-09-10T08:31:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:40.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:40.303768+0000) 2022-09-10T08:31:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:40.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:40.879128+0000) 2022-09-10T08:31:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:41.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:41.303942+0000) 2022-09-10T08:31:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:41 smithi110 ceph-mon[97633]: pgmap v7759: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:41 smithi104 ceph-mon[97044]: pgmap v7759: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:41.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:41.879289+0000) 2022-09-10T08:31:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:42.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:42.304087+0000) 2022-09-10T08:31:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:42.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:42.879414+0000) 2022-09-10T08:31:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:43.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:43.304276+0000) 2022-09-10T08:31:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:43 smithi110 ceph-mon[97633]: pgmap v7760: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:43 smithi104 ceph-mon[97044]: pgmap v7760: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:43.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:43.879548+0000) 2022-09-10T08:31:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:44.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:44.304433+0000) 2022-09-10T08:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:31:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:31:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:31:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:31:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:31:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:31:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:31:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:44.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:44.879674+0000) 2022-09-10T08:31:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:45.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:45.304601+0000) 2022-09-10T08:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:45 smithi110 ceph-mon[97633]: pgmap v7761: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:45 smithi104 ceph-mon[97044]: pgmap v7761: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:45.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:45.879790+0000) 2022-09-10T08:31:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:46.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:46.304766+0000) 2022-09-10T08:31:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:46.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:46.879924+0000) 2022-09-10T08:31:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:47.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:47.304945+0000) 2022-09-10T08:31:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:47 smithi110 ceph-mon[97633]: pgmap v7762: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:47 smithi104 ceph-mon[97044]: pgmap v7762: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:47.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:47.880053+0000) 2022-09-10T08:31:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:48.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:48.305148+0000) 2022-09-10T08:31:49.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:48.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:48.880183+0000) 2022-09-10T08:31:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:31:49.179+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:31:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:49.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:49.305331+0000) 2022-09-10T08:31:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:49 smithi110 ceph-mon[97633]: pgmap v7763: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:49 smithi104 ceph-mon[97044]: pgmap v7763: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:49.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:49.880397+0000) 2022-09-10T08:31:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:50.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:50.305528+0000) 2022-09-10T08:31:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:50.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:50.880533+0000) 2022-09-10T08:31:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:51.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:51.305702+0000) 2022-09-10T08:31:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:51 smithi110 ceph-mon[97633]: pgmap v7764: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:51 smithi104 ceph-mon[97044]: pgmap v7764: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:51.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:51.880703+0000) 2022-09-10T08:31:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:52.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:52.305869+0000) 2022-09-10T08:31:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:52.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:52.880838+0000) 2022-09-10T08:31:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:53.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:53.306051+0000) 2022-09-10T08:31:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:53 smithi110 ceph-mon[97633]: pgmap v7765: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:53 smithi104 ceph-mon[97044]: pgmap v7765: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:53.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:53.881020+0000) 2022-09-10T08:31:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:54.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:54.306162+0000) 2022-09-10T08:31:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:54.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:54.881169+0000) 2022-09-10T08:31:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:55.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:55.306321+0000) 2022-09-10T08:31:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:55 smithi110 ceph-mon[97633]: pgmap v7766: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:55 smithi104 ceph-mon[97044]: pgmap v7766: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:31:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:55.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:55.881303+0000) 2022-09-10T08:31:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:56.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:56.306506+0000) 2022-09-10T08:31:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:56.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:56.881423+0000) 2022-09-10T08:31:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:57.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:57.306697+0000) 2022-09-10T08:31:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:57 smithi110 ceph-mon[97633]: pgmap v7767: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:57 smithi104 ceph-mon[97044]: pgmap v7767: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:31:58.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:57.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:57.881590+0000) 2022-09-10T08:31:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:58.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:58.306887+0000) 2022-09-10T08:31:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:58.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:58.881721+0000) 2022-09-10T08:31:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:31:59.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:59.307071+0000) 2022-09-10T08:32:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:31:59 smithi110 ceph-mon[97633]: pgmap v7768: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:31:59 smithi104 ceph-mon[97044]: pgmap v7768: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:31:59.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:31:59.881879+0000) 2022-09-10T08:32:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:00.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:00.307286+0000) 2022-09-10T08:32:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:00.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:00.882004+0000) 2022-09-10T08:32:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:01.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:01.307449+0000) 2022-09-10T08:32:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:01 smithi110 ceph-mon[97633]: pgmap v7769: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:01 smithi104 ceph-mon[97044]: pgmap v7769: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:01.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:01.882155+0000) 2022-09-10T08:32:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:02.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:02.307568+0000) 2022-09-10T08:32:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:02.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:02.882290+0000) 2022-09-10T08:32:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:03.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:03.307721+0000) 2022-09-10T08:32:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:03 smithi110 ceph-mon[97633]: pgmap v7770: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:04.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:03.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:03.882420+0000) 2022-09-10T08:32:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:03 smithi104 ceph-mon[97044]: pgmap v7770: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:04.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:04.307955+0000) 2022-09-10T08:32:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:32:04.192+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:32:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:04.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:04.882542+0000) 2022-09-10T08:32:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:05.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:05.308170+0000) 2022-09-10T08:32:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:05 smithi110 ceph-mon[97633]: pgmap v7771: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:32:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:05 smithi104 ceph-mon[97044]: pgmap v7771: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:32:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:05.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:05.882670+0000) 2022-09-10T08:32:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:06.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:06.308399+0000) 2022-09-10T08:32:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:06 smithi110 ceph-mon[97633]: pgmap v7772: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:32:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:06 smithi104 ceph-mon[97044]: pgmap v7772: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:32:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:06.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:06.882806+0000) 2022-09-10T08:32:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:07.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:07.308576+0000) 2022-09-10T08:32:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:07.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:07.882947+0000) 2022-09-10T08:32:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:08.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:08.308811+0000) 2022-09-10T08:32:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:08 smithi110 ceph-mon[97633]: osdmap e566: 12 total, 12 up, 12 in 2022-09-10T08:32:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:08 smithi110 ceph-mon[97633]: pgmap v7774: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:08 smithi104 ceph-mon[97044]: osdmap e566: 12 total, 12 up, 12 in 2022-09-10T08:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:08 smithi104 ceph-mon[97044]: pgmap v7774: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:32:09.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:08.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:08.883155+0000) 2022-09-10T08:32:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:09.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:09.308991+0000) 2022-09-10T08:32:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:09.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:09.883329+0000) 2022-09-10T08:32:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:10.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:10.309206+0000) 2022-09-10T08:32:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:10.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:10.883471+0000) 2022-09-10T08:32:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:11.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:11.309344+0000) 2022-09-10T08:32:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:11 smithi110 ceph-mon[97633]: pgmap v7775: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:32:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:11 smithi110 ceph-mon[97633]: osdmap e567: 12 total, 12 up, 12 in 2022-09-10T08:32:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:11 smithi104 ceph-mon[97044]: pgmap v7775: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:32:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:11 smithi104 ceph-mon[97044]: osdmap e567: 12 total, 12 up, 12 in 2022-09-10T08:32:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:11.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:11.883623+0000) 2022-09-10T08:32:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:12.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:12.309509+0000) 2022-09-10T08:32:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:12.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:12.883756+0000) 2022-09-10T08:32:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:13.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:13.309739+0000) 2022-09-10T08:32:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:13 smithi110 ceph-mon[97633]: pgmap v7777: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:32:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:13 smithi104 ceph-mon[97044]: pgmap v7777: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:32:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:13.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:13.883900+0000) 2022-09-10T08:32:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:14.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:14.309892+0000) 2022-09-10T08:32:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:14.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:14.884041+0000) 2022-09-10T08:32:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:15.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:15.310012+0000) 2022-09-10T08:32:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:15 smithi110 ceph-mon[97633]: pgmap v7778: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:32:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:15 smithi104 ceph-mon[97044]: pgmap v7778: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:32:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:15.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:15.884153+0000) 2022-09-10T08:32:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:16.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:16.310178+0000) 2022-09-10T08:32:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:16.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:16.884284+0000) 2022-09-10T08:32:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:17.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:17.310343+0000) 2022-09-10T08:32:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:17 smithi110 ceph-mon[97633]: pgmap v7779: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:32:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:17 smithi104 ceph-mon[97044]: pgmap v7779: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:32:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:17.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:17.884473+0000) 2022-09-10T08:32:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:18.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:18.310484+0000) 2022-09-10T08:32:19.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:18.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:18.884590+0000) 2022-09-10T08:32:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:19.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:19.310672+0000) 2022-09-10T08:32:19.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:32:19.208+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:32:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:19 smithi110 ceph-mon[97633]: pgmap v7780: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:32:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:19 smithi104 ceph-mon[97044]: pgmap v7780: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:32:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:19.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:19.884716+0000) 2022-09-10T08:32:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:20.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:20.310852+0000) 2022-09-10T08:32:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:20.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:20.884836+0000) 2022-09-10T08:32:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:21.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:21.311052+0000) 2022-09-10T08:32:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:21 smithi110 ceph-mon[97633]: pgmap v7781: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:32:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:21 smithi104 ceph-mon[97044]: pgmap v7781: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:32:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:21.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:21.884970+0000) 2022-09-10T08:32:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:22.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:22.311166+0000) 2022-09-10T08:32:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:22.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:22.885104+0000) 2022-09-10T08:32:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:23.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:23.311365+0000) 2022-09-10T08:32:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:23 smithi110 ceph-mon[97633]: pgmap v7782: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:32:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:23 smithi104 ceph-mon[97044]: pgmap v7782: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:32:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:23.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:23.885293+0000) 2022-09-10T08:32:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:24.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:24.311557+0000) 2022-09-10T08:32:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:24.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:24.885454+0000) 2022-09-10T08:32:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:25.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:25.311749+0000) 2022-09-10T08:32:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:25 smithi110 ceph-mon[97633]: pgmap v7783: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:32:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:25 smithi104 ceph-mon[97044]: pgmap v7783: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:32:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:25.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:25.885608+0000) 2022-09-10T08:32:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:26.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:26.312044+0000) 2022-09-10T08:32:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:26.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:26.885727+0000) 2022-09-10T08:32:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:27.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:27.312341+0000) 2022-09-10T08:32:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:27 smithi110 ceph-mon[97633]: pgmap v7784: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:27 smithi104 ceph-mon[97044]: pgmap v7784: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:27.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:27.885864+0000) 2022-09-10T08:32:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:28.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:28.312640+0000) 2022-09-10T08:32:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:28.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:28.885990+0000) 2022-09-10T08:32:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:29.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:29.312812+0000) 2022-09-10T08:32:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:29 smithi110 ceph-mon[97633]: pgmap v7785: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:29 smithi104 ceph-mon[97044]: pgmap v7785: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:29.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:29.886201+0000) 2022-09-10T08:32:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:30.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:30.313008+0000) 2022-09-10T08:32:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:30.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:30.886339+0000) 2022-09-10T08:32:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:31.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:31.313234+0000) 2022-09-10T08:32:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:31 smithi110 ceph-mon[97633]: pgmap v7786: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:31 smithi104 ceph-mon[97044]: pgmap v7786: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:31.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:31.886478+0000) 2022-09-10T08:32:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:32.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:32.313422+0000) 2022-09-10T08:32:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:32.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:32.886606+0000) 2022-09-10T08:32:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:33.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:33.313650+0000) 2022-09-10T08:32:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:33 smithi110 ceph-mon[97633]: pgmap v7787: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:34.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:33.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:33.886786+0000) 2022-09-10T08:32:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:33 smithi104 ceph-mon[97044]: pgmap v7787: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:34.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:34.313902+0000) 2022-09-10T08:32:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:32:34.223+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:32:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:34.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:34.887000+0000) 2022-09-10T08:32:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:35.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:35.314146+0000) 2022-09-10T08:32:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:35 smithi110 ceph-mon[97633]: pgmap v7788: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:35 smithi104 ceph-mon[97044]: pgmap v7788: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:35.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:35.887143+0000) 2022-09-10T08:32:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:36.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:36.314376+0000) 2022-09-10T08:32:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:36.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:36.887258+0000) 2022-09-10T08:32:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:37.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:37.314591+0000) 2022-09-10T08:32:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:37 smithi110 ceph-mon[97633]: pgmap v7789: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:37 smithi104 ceph-mon[97044]: pgmap v7789: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:37.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:37.887396+0000) 2022-09-10T08:32:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:38.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:38.314764+0000) 2022-09-10T08:32:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:38.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:38.887540+0000) 2022-09-10T08:32:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:39.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:39.315018+0000) 2022-09-10T08:32:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:39 smithi110 ceph-mon[97633]: pgmap v7790: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:39 smithi104 ceph-mon[97044]: pgmap v7790: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:39.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:39.887679+0000) 2022-09-10T08:32:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:40.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:40.315240+0000) 2022-09-10T08:32:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:40.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:40.887813+0000) 2022-09-10T08:32:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:41.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:41.315425+0000) 2022-09-10T08:32:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:41 smithi110 ceph-mon[97633]: pgmap v7791: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:42.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:41 smithi104 ceph-mon[97044]: pgmap v7791: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:41.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:41.887962+0000) 2022-09-10T08:32:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:42.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:42.315615+0000) 2022-09-10T08:32:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:42.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:42.888125+0000) 2022-09-10T08:32:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:43.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:43.315787+0000) 2022-09-10T08:32:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:43 smithi110 ceph-mon[97633]: pgmap v7792: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:43 smithi104 ceph-mon[97044]: pgmap v7792: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:43.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:43.888253+0000) 2022-09-10T08:32:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:44.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:44.316034+0000) 2022-09-10T08:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:32:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:32:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:32:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:32:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:44.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:44.888374+0000) 2022-09-10T08:32:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:45.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:45.316235+0000) 2022-09-10T08:32:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:45 smithi110 ceph-mon[97633]: pgmap v7793: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:45 smithi104 ceph-mon[97044]: pgmap v7793: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:45.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:45.888492+0000) 2022-09-10T08:32:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:46.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:46.316454+0000) 2022-09-10T08:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:46 smithi110 ceph-mon[97633]: pgmap v7794: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:46 smithi104 ceph-mon[97044]: pgmap v7794: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:46.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:46.888613+0000) 2022-09-10T08:32:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:47.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:47.316686+0000) 2022-09-10T08:32:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:47.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:47.888755+0000) 2022-09-10T08:32:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:48.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:48.316881+0000) 2022-09-10T08:32:49.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:48.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:48.888889+0000) 2022-09-10T08:32:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:32:49.238+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:32:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:49.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:49.317043+0000) 2022-09-10T08:32:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:49 smithi104 ceph-mon[97044]: pgmap v7795: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:49 smithi110 ceph-mon[97633]: pgmap v7795: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:49.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:49.889069+0000) 2022-09-10T08:32:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:50.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:50.317269+0000) 2022-09-10T08:32:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:50.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:50.889223+0000) 2022-09-10T08:32:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:51.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:51.317505+0000) 2022-09-10T08:32:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:51 smithi110 ceph-mon[97633]: pgmap v7796: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:51 smithi104 ceph-mon[97044]: pgmap v7796: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:51.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:51.889380+0000) 2022-09-10T08:32:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:52.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:52.317712+0000) 2022-09-10T08:32:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:52.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:52.889499+0000) 2022-09-10T08:32:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:53.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:53.317957+0000) 2022-09-10T08:32:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:53 smithi110 ceph-mon[97633]: pgmap v7797: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:53 smithi104 ceph-mon[97044]: pgmap v7797: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:53.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:53.889621+0000) 2022-09-10T08:32:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:54.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:54.318175+0000) 2022-09-10T08:32:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:54.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:54.889791+0000) 2022-09-10T08:32:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:55.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:55.318394+0000) 2022-09-10T08:32:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:55 smithi110 ceph-mon[97633]: pgmap v7798: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:55 smithi104 ceph-mon[97044]: pgmap v7798: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:32:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:55.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:55.889963+0000) 2022-09-10T08:32:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:56.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:56.318578+0000) 2022-09-10T08:32:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:56.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:56.890116+0000) 2022-09-10T08:32:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:57.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:57.318769+0000) 2022-09-10T08:32:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:57 smithi110 ceph-mon[97633]: pgmap v7799: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:57 smithi104 ceph-mon[97044]: pgmap v7799: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:32:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:57.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:57.890258+0000) 2022-09-10T08:32:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:58.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:58.318940+0000) 2022-09-10T08:32:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:58.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:58.890420+0000) 2022-09-10T08:32:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:32:59.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:59.319209+0000) 2022-09-10T08:33:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:32:59 smithi110 ceph-mon[97633]: pgmap v7800: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:32:59 smithi104 ceph-mon[97044]: pgmap v7800: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:32:59.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:32:59.890566+0000) 2022-09-10T08:33:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:00.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:00.319426+0000) 2022-09-10T08:33:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:00.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:00.890687+0000) 2022-09-10T08:33:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:01.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:01.319668+0000) 2022-09-10T08:33:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:01 smithi110 ceph-mon[97633]: pgmap v7801: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:01 smithi104 ceph-mon[97044]: pgmap v7801: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:01.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:01.890828+0000) 2022-09-10T08:33:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:02.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:02.319871+0000) 2022-09-10T08:33:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:02.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:02.890952+0000) 2022-09-10T08:33:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:03.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:03.320048+0000) 2022-09-10T08:33:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:03 smithi110 ceph-mon[97633]: pgmap v7802: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:04.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:03.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:03.891081+0000) 2022-09-10T08:33:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:03 smithi104 ceph-mon[97044]: pgmap v7802: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:04.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:04.320282+0000) 2022-09-10T08:33:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:33:04.244+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:33:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:04.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:04.891219+0000) 2022-09-10T08:33:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:05.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:05.320423+0000) 2022-09-10T08:33:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:05 smithi110 ceph-mon[97633]: pgmap v7803: 129 pgs: 129 active+clean; 8.5 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:33:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:05 smithi104 ceph-mon[97044]: pgmap v7803: 129 pgs: 129 active+clean; 8.5 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:33:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:05.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:05.891340+0000) 2022-09-10T08:33:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:06.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:06.320649+0000) 2022-09-10T08:33:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:06.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:06.891491+0000) 2022-09-10T08:33:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:07.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:07.320822+0000) 2022-09-10T08:33:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:07 smithi110 ceph-mon[97633]: pgmap v7804: 129 pgs: 129 active+clean; 8.5 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:33:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:07 smithi104 ceph-mon[97044]: pgmap v7804: 129 pgs: 129 active+clean; 8.5 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:33:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:07.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:07.891622+0000) 2022-09-10T08:33:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:08.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:08.321056+0000) 2022-09-10T08:33:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:08 smithi110 ceph-mon[97633]: osdmap e568: 12 total, 12 up, 12 in 2022-09-10T08:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:08 smithi104 ceph-mon[97044]: osdmap e568: 12 total, 12 up, 12 in 2022-09-10T08:33:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:08.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:08.891737+0000) 2022-09-10T08:33:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:09.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:09.321286+0000) 2022-09-10T08:33:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:09 smithi110 ceph-mon[97633]: pgmap v7806: 129 pgs: 129 active+clean; 8.5 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:33:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:09 smithi104 ceph-mon[97044]: pgmap v7806: 129 pgs: 129 active+clean; 8.5 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:33:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:09.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:09.891864+0000) 2022-09-10T08:33:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:10.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:10.321538+0000) 2022-09-10T08:33:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:10.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:10.892062+0000) 2022-09-10T08:33:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:11.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:11.321742+0000) 2022-09-10T08:33:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:11 smithi110 ceph-mon[97633]: pgmap v7807: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:33:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:11 smithi104 ceph-mon[97044]: pgmap v7807: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:33:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:11.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:11.892202+0000) 2022-09-10T08:33:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:12.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:12.321919+0000) 2022-09-10T08:33:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:12.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:12.892332+0000) 2022-09-10T08:33:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:13.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:13.322167+0000) 2022-09-10T08:33:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:13 smithi110 ceph-mon[97633]: pgmap v7808: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:33:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:13 smithi104 ceph-mon[97044]: pgmap v7808: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:33:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:13.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:13.892450+0000) 2022-09-10T08:33:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:14.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:14.322393+0000) 2022-09-10T08:33:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:14 smithi110 ceph-mon[97633]: pgmap v7809: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:33:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:14 smithi104 ceph-mon[97044]: pgmap v7809: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:33:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:14.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:14.892584+0000) 2022-09-10T08:33:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:15.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:15.322584+0000) 2022-09-10T08:33:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:15.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:15.892711+0000) 2022-09-10T08:33:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:16.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:16.322770+0000) 2022-09-10T08:33:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:16 smithi110 ceph-mon[97633]: osdmap e569: 12 total, 12 up, 12 in 2022-09-10T08:33:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:16 smithi104 ceph-mon[97044]: osdmap e569: 12 total, 12 up, 12 in 2022-09-10T08:33:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:16.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:16.892834+0000) 2022-09-10T08:33:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:17.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:17.322955+0000) 2022-09-10T08:33:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:17 smithi110 ceph-mon[97633]: pgmap v7811: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:33:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:17 smithi104 ceph-mon[97044]: pgmap v7811: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:33:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:17.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:17.892961+0000) 2022-09-10T08:33:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:18.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:18.323177+0000) 2022-09-10T08:33:19.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:18.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:18.893110+0000) 2022-09-10T08:33:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:19.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:19.323387+0000) 2022-09-10T08:33:19.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:33:19.256+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:33:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:19 smithi110 ceph-mon[97633]: pgmap v7812: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:33:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:19 smithi104 ceph-mon[97044]: pgmap v7812: 129 pgs: 129 active+clean; 8.5 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:33:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:19.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:19.893265+0000) 2022-09-10T08:33:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:20.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:20.323584+0000) 2022-09-10T08:33:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:20.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:20.893389+0000) 2022-09-10T08:33:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:21.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:21.323807+0000) 2022-09-10T08:33:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:21 smithi110 ceph-mon[97633]: pgmap v7813: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:33:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:21 smithi104 ceph-mon[97044]: pgmap v7813: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:33:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:21.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:21.893525+0000) 2022-09-10T08:33:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:22.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:22.323970+0000) 2022-09-10T08:33:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:22.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:22.893666+0000) 2022-09-10T08:33:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:23.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:23.324170+0000) 2022-09-10T08:33:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:23 smithi110 ceph-mon[97633]: pgmap v7814: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:33:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:23 smithi104 ceph-mon[97044]: pgmap v7814: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:33:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:23.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:23.893784+0000) 2022-09-10T08:33:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:24.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:24.324350+0000) 2022-09-10T08:33:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:24.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:24.893901+0000) 2022-09-10T08:33:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:25.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:25.324499+0000) 2022-09-10T08:33:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:25 smithi110 ceph-mon[97633]: pgmap v7815: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:33:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:25 smithi104 ceph-mon[97044]: pgmap v7815: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:33:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:25.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:25.894030+0000) 2022-09-10T08:33:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:26.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:26.324734+0000) 2022-09-10T08:33:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:26.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:26.894166+0000) 2022-09-10T08:33:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:27.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:27.324929+0000) 2022-09-10T08:33:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:27 smithi110 ceph-mon[97633]: pgmap v7816: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:33:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:27 smithi104 ceph-mon[97044]: pgmap v7816: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:33:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:27.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:27.894295+0000) 2022-09-10T08:33:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:28.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:28.325215+0000) 2022-09-10T08:33:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:28.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:28.894424+0000) 2022-09-10T08:33:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:29.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:29.325357+0000) 2022-09-10T08:33:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:29 smithi110 ceph-mon[97633]: pgmap v7817: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:29 smithi104 ceph-mon[97044]: pgmap v7817: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:29.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:29.894557+0000) 2022-09-10T08:33:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:30.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:30.325515+0000) 2022-09-10T08:33:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:30.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:30.894675+0000) 2022-09-10T08:33:31.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:31.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:31.325662+0000) 2022-09-10T08:33:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:31 smithi110 ceph-mon[97633]: pgmap v7818: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:31 smithi104 ceph-mon[97044]: pgmap v7818: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:31.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:31.894867+0000) 2022-09-10T08:33:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:32.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:32.325946+0000) 2022-09-10T08:33:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:32.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:32.895009+0000) 2022-09-10T08:33:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:33.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:33.326199+0000) 2022-09-10T08:33:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:33 smithi110 ceph-mon[97633]: pgmap v7819: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:34.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:33.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:33.895145+0000) 2022-09-10T08:33:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:33 smithi104 ceph-mon[97044]: pgmap v7819: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:34.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:34.326403+0000) 2022-09-10T08:33:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:33:34.263+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:33:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:34.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:34.895281+0000) 2022-09-10T08:33:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:35.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:35.326674+0000) 2022-09-10T08:33:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:35 smithi110 ceph-mon[97633]: pgmap v7820: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:35 smithi104 ceph-mon[97044]: pgmap v7820: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:35.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:35.895454+0000) 2022-09-10T08:33:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:36.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:36.326943+0000) 2022-09-10T08:33:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:36.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:36.895592+0000) 2022-09-10T08:33:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:37.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:37.327151+0000) 2022-09-10T08:33:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:37 smithi110 ceph-mon[97633]: pgmap v7821: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:37 smithi104 ceph-mon[97044]: pgmap v7821: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:37.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:37.895779+0000) 2022-09-10T08:33:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:38.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:38.327318+0000) 2022-09-10T08:33:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:38 smithi104 ceph-mon[97044]: pgmap v7822: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:38.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:38.895935+0000) 2022-09-10T08:33:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:38 smithi110 ceph-mon[97633]: pgmap v7822: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:39.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:39.327517+0000) 2022-09-10T08:33:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:39.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:39.896070+0000) 2022-09-10T08:33:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:40.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:40.327765+0000) 2022-09-10T08:33:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:40.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:40.896162+0000) 2022-09-10T08:33:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:41.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:41.328014+0000) 2022-09-10T08:33:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:41 smithi110 ceph-mon[97633]: pgmap v7823: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:41 smithi104 ceph-mon[97044]: pgmap v7823: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:41.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:41.896330+0000) 2022-09-10T08:33:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:42.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:42.328184+0000) 2022-09-10T08:33:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:42.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:42.896496+0000) 2022-09-10T08:33:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:43.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:43.328374+0000) 2022-09-10T08:33:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:43 smithi110 ceph-mon[97633]: pgmap v7824: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:43 smithi104 ceph-mon[97044]: pgmap v7824: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:43.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:43.896630+0000) 2022-09-10T08:33:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:44.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:44.328581+0000) 2022-09-10T08:33:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:33:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:33:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:33:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:33:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:33:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:33:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:33:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:33:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:44.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:44.896764+0000) 2022-09-10T08:33:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:45.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:45.328744+0000) 2022-09-10T08:33:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:45 smithi110 ceph-mon[97633]: pgmap v7825: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:45 smithi104 ceph-mon[97044]: pgmap v7825: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:45.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:45.896901+0000) 2022-09-10T08:33:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:46.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:46.328936+0000) 2022-09-10T08:33:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:46.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:46.897049+0000) 2022-09-10T08:33:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:47.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:47.329160+0000) 2022-09-10T08:33:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:47 smithi110 ceph-mon[97633]: pgmap v7826: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:47 smithi104 ceph-mon[97044]: pgmap v7826: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:47.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:47.897165+0000) 2022-09-10T08:33:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:48.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:48.329390+0000) 2022-09-10T08:33:49.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:48.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:48.897318+0000) 2022-09-10T08:33:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:49.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:49.329703+0000) 2022-09-10T08:33:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:33:49.279+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:33:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:49 smithi110 ceph-mon[97633]: pgmap v7827: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:49 smithi104 ceph-mon[97044]: pgmap v7827: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:49.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:49.897445+0000) 2022-09-10T08:33:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:50.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:50.329895+0000) 2022-09-10T08:33:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:50.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:50.897568+0000) 2022-09-10T08:33:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:51.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:51.330113+0000) 2022-09-10T08:33:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:51 smithi110 ceph-mon[97633]: pgmap v7828: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:51 smithi104 ceph-mon[97044]: pgmap v7828: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:51.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:51.897715+0000) 2022-09-10T08:33:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:52.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:52.330342+0000) 2022-09-10T08:33:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:52.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:52.897850+0000) 2022-09-10T08:33:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:53.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:53.330497+0000) 2022-09-10T08:33:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:53 smithi110 ceph-mon[97633]: pgmap v7829: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:53 smithi104 ceph-mon[97044]: pgmap v7829: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:53.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:53.897970+0000) 2022-09-10T08:33:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:54.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:54.330695+0000) 2022-09-10T08:33:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:54.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:54.898195+0000) 2022-09-10T08:33:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:55.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:55.330865+0000) 2022-09-10T08:33:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:55 smithi110 ceph-mon[97633]: pgmap v7830: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:55 smithi104 ceph-mon[97044]: pgmap v7830: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:33:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:55.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:55.898357+0000) 2022-09-10T08:33:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:56.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:56.331057+0000) 2022-09-10T08:33:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:56.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:56.898485+0000) 2022-09-10T08:33:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:57.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:57.331306+0000) 2022-09-10T08:33:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:57 smithi110 ceph-mon[97633]: pgmap v7831: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:57 smithi104 ceph-mon[97044]: pgmap v7831: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:33:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:57.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:57.898620+0000) 2022-09-10T08:33:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:58.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:58.331481+0000) 2022-09-10T08:33:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:58.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:58.898757+0000) 2022-09-10T08:33:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:33:59.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:59.331639+0000) 2022-09-10T08:34:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:33:59 smithi110 ceph-mon[97633]: pgmap v7832: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:33:59 smithi104 ceph-mon[97044]: pgmap v7832: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:33:59.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:33:59.898892+0000) 2022-09-10T08:34:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:00.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:00.331793+0000) 2022-09-10T08:34:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:00.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:00.899053+0000) 2022-09-10T08:34:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:01.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:01.331971+0000) 2022-09-10T08:34:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:01 smithi110 ceph-mon[97633]: pgmap v7833: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:01 smithi104 ceph-mon[97044]: pgmap v7833: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:01.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:01.899161+0000) 2022-09-10T08:34:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:02.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:02.332193+0000) 2022-09-10T08:34:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:02.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:02.899300+0000) 2022-09-10T08:34:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:03.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:03.332389+0000) 2022-09-10T08:34:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:03 smithi110 ceph-mon[97633]: pgmap v7834: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:04.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:03.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:03.899472+0000) 2022-09-10T08:34:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:03 smithi104 ceph-mon[97044]: pgmap v7834: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:04.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:04.332591+0000) 2022-09-10T08:34:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:34:04.284+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:34:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:04 smithi110 ceph-mon[97633]: pgmap v7835: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:34:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:04 smithi104 ceph-mon[97044]: pgmap v7835: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:34:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:04.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:04.899603+0000) 2022-09-10T08:34:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:05.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:05.332816+0000) 2022-09-10T08:34:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:05.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:05.899741+0000) 2022-09-10T08:34:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:06.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:06.333048+0000) 2022-09-10T08:34:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:06.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:06.899884+0000) 2022-09-10T08:34:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:07.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:07.333269+0000) 2022-09-10T08:34:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:07 smithi110 ceph-mon[97633]: pgmap v7836: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:34:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:07 smithi104 ceph-mon[97044]: pgmap v7836: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:34:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:07.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:07.900040+0000) 2022-09-10T08:34:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:08.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:08.333471+0000) 2022-09-10T08:34:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:08 smithi110 ceph-mon[97633]: osdmap e570: 12 total, 12 up, 12 in 2022-09-10T08:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:08 smithi104 ceph-mon[97044]: osdmap e570: 12 total, 12 up, 12 in 2022-09-10T08:34:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:08.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:08.900171+0000) 2022-09-10T08:34:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:09.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:09.333680+0000) 2022-09-10T08:34:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:09 smithi110 ceph-mon[97633]: pgmap v7838: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:34:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:09 smithi104 ceph-mon[97044]: pgmap v7838: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:34:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:09.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:09.900296+0000) 2022-09-10T08:34:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:10.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:10.333910+0000) 2022-09-10T08:34:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:10.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:10.900421+0000) 2022-09-10T08:34:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:11.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:11.334182+0000) 2022-09-10T08:34:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:11 smithi110 ceph-mon[97633]: pgmap v7839: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:34:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:11 smithi104 ceph-mon[97044]: pgmap v7839: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:34:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:11.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:11.900587+0000) 2022-09-10T08:34:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:12.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:12.334403+0000) 2022-09-10T08:34:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:12.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:12.900705+0000) 2022-09-10T08:34:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:13.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:13.334631+0000) 2022-09-10T08:34:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:13 smithi110 ceph-mon[97633]: pgmap v7840: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:34:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:13 smithi104 ceph-mon[97044]: pgmap v7840: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:34:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:13.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:13.900823+0000) 2022-09-10T08:34:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:14.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:14.334770+0000) 2022-09-10T08:34:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:14.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:14.900949+0000) 2022-09-10T08:34:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:15.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:15.335023+0000) 2022-09-10T08:34:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:15 smithi110 ceph-mon[97633]: pgmap v7841: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:34:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:15 smithi110 ceph-mon[97633]: osdmap e571: 12 total, 12 up, 12 in 2022-09-10T08:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:15 smithi104 ceph-mon[97044]: pgmap v7841: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:34:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:15 smithi104 ceph-mon[97044]: osdmap e571: 12 total, 12 up, 12 in 2022-09-10T08:34:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:15.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:15.901273+0000) 2022-09-10T08:34:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:16.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:16.335307+0000) 2022-09-10T08:34:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:16.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:16.901421+0000) 2022-09-10T08:34:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:17.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:17.335502+0000) 2022-09-10T08:34:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:17 smithi110 ceph-mon[97633]: pgmap v7843: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:34:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:17 smithi104 ceph-mon[97044]: pgmap v7843: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:34:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:17.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:17.901559+0000) 2022-09-10T08:34:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:18.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:18.335654+0000) 2022-09-10T08:34:19.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:18.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:18.901694+0000) 2022-09-10T08:34:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:19.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:19.335886+0000) 2022-09-10T08:34:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:34:19.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:34:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:19 smithi110 ceph-mon[97633]: pgmap v7844: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:34:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:19 smithi104 ceph-mon[97044]: pgmap v7844: 129 pgs: 129 active+clean; 8.5 MiB data, 834 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:34:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:19.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:19.901865+0000) 2022-09-10T08:34:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:20.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:20.336173+0000) 2022-09-10T08:34:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:20.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:20.902048+0000) 2022-09-10T08:34:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:21.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:21.336419+0000) 2022-09-10T08:34:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:21 smithi110 ceph-mon[97633]: pgmap v7845: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:34:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:21 smithi104 ceph-mon[97044]: pgmap v7845: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:34:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:21.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:21.902209+0000) 2022-09-10T08:34:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:22.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:22.336653+0000) 2022-09-10T08:34:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:22 smithi104 ceph-mon[97044]: pgmap v7846: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:34:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:22.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:22.902358+0000) 2022-09-10T08:34:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:22 smithi110 ceph-mon[97633]: pgmap v7846: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:34:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:23.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:23.336881+0000) 2022-09-10T08:34:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:23.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:23.902481+0000) 2022-09-10T08:34:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:24.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:24.337113+0000) 2022-09-10T08:34:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:24.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:24.902602+0000) 2022-09-10T08:34:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:25 smithi104 ceph-mon[97044]: pgmap v7847: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:34:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:25.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:25.337316+0000) 2022-09-10T08:34:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:25 smithi110 ceph-mon[97633]: pgmap v7847: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T08:34:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:25.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:25.902782+0000) 2022-09-10T08:34:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:26.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:26.337559+0000) 2022-09-10T08:34:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:26.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:26.903007+0000) 2022-09-10T08:34:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:27.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:27.337784+0000) 2022-09-10T08:34:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:27 smithi110 ceph-mon[97633]: pgmap v7848: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:34:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:27 smithi104 ceph-mon[97044]: pgmap v7848: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T08:34:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:27.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:27.903176+0000) 2022-09-10T08:34:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:28.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:28.338023+0000) 2022-09-10T08:34:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:28.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:28.903292+0000) 2022-09-10T08:34:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:29.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:29.338236+0000) 2022-09-10T08:34:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:29 smithi110 ceph-mon[97633]: pgmap v7849: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:29 smithi104 ceph-mon[97044]: pgmap v7849: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:30.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:29.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:29.903436+0000) 2022-09-10T08:34:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:30.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:30.338424+0000) 2022-09-10T08:34:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:30.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:30.903560+0000) 2022-09-10T08:34:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:31.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:31.338632+0000) 2022-09-10T08:34:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:31 smithi110 ceph-mon[97633]: pgmap v7850: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:31 smithi104 ceph-mon[97044]: pgmap v7850: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:31.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:31.903691+0000) 2022-09-10T08:34:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:32.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:32.338924+0000) 2022-09-10T08:34:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:32.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:32.903836+0000) 2022-09-10T08:34:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:33.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:33.339201+0000) 2022-09-10T08:34:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:33 smithi110 ceph-mon[97633]: pgmap v7851: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:33 smithi104 ceph-mon[97044]: pgmap v7851: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:33.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:33.903975+0000) 2022-09-10T08:34:34.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:34:34.316+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:34:34.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:34.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:34.339362+0000) 2022-09-10T08:34:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:34.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:34.904123+0000) 2022-09-10T08:34:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:35.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:35.339575+0000) 2022-09-10T08:34:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:35 smithi110 ceph-mon[97633]: pgmap v7852: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:35 smithi104 ceph-mon[97044]: pgmap v7852: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:35.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:35.904255+0000) 2022-09-10T08:34:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:36.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:36.339738+0000) 2022-09-10T08:34:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:36.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:36.904385+0000) 2022-09-10T08:34:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:37.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:37.339991+0000) 2022-09-10T08:34:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:37 smithi110 ceph-mon[97633]: pgmap v7853: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:37 smithi104 ceph-mon[97044]: pgmap v7853: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:37.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:37.904546+0000) 2022-09-10T08:34:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:38.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:38.340198+0000) 2022-09-10T08:34:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:38.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:38.904687+0000) 2022-09-10T08:34:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:39.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:39.340396+0000) 2022-09-10T08:34:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:39 smithi110 ceph-mon[97633]: pgmap v7854: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:39 smithi104 ceph-mon[97044]: pgmap v7854: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:39.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:39.904832+0000) 2022-09-10T08:34:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:40.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:40.340610+0000) 2022-09-10T08:34:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:40.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:40.904956+0000) 2022-09-10T08:34:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:41.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:41.340839+0000) 2022-09-10T08:34:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:41 smithi110 ceph-mon[97633]: pgmap v7855: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:41 smithi104 ceph-mon[97044]: pgmap v7855: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:41.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:41.905090+0000) 2022-09-10T08:34:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:42.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:42.341068+0000) 2022-09-10T08:34:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:42.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:42.905227+0000) 2022-09-10T08:34:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:43.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:43.341336+0000) 2022-09-10T08:34:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:43 smithi110 ceph-mon[97633]: pgmap v7856: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:43 smithi104 ceph-mon[97044]: pgmap v7856: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:43.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:43.905351+0000) 2022-09-10T08:34:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:44.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:44.341532+0000) 2022-09-10T08:34:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:34:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:34:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:34:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:34:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:44.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:44.905487+0000) 2022-09-10T08:34:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:34:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:34:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:34:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:34:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:45.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:45.341702+0000) 2022-09-10T08:34:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:45 smithi104 ceph-mon[97044]: pgmap v7857: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:45.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:45.905623+0000) 2022-09-10T08:34:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:45 smithi110 ceph-mon[97633]: pgmap v7857: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:46.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:46.341931+0000) 2022-09-10T08:34:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:46 smithi104 ceph-mon[97044]: pgmap v7858: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:46.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:46.905753+0000) 2022-09-10T08:34:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:46 smithi110 ceph-mon[97633]: pgmap v7858: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:47.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:47.342164+0000) 2022-09-10T08:34:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:47.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:47.905875+0000) 2022-09-10T08:34:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:48.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:48.342390+0000) 2022-09-10T08:34:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:48.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:48.906000+0000) 2022-09-10T08:34:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:49 smithi104 ceph-mon[97044]: pgmap v7859: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:49.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:34:49.331+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:34:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:49.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:49.342574+0000) 2022-09-10T08:34:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:49 smithi110 ceph-mon[97633]: pgmap v7859: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:34:50.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:49.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:49.906140+0000) 2022-09-10T08:34:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:50.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:50.342785+0000) 2022-09-10T08:34:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:50.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:50.906264+0000) 2022-09-10T08:34:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:51.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:51.343043+0000) 2022-09-10T08:34:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:51 smithi110 ceph-mon[97633]: pgmap v7860: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:51 smithi104 ceph-mon[97044]: pgmap v7860: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:51.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:51.906396+0000) 2022-09-10T08:34:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:52.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:52.343300+0000) 2022-09-10T08:34:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:52.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:52.906518+0000) 2022-09-10T08:34:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:53.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:53.343525+0000) 2022-09-10T08:34:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:53 smithi110 ceph-mon[97633]: pgmap v7861: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:34:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:53 smithi104 ceph-mon[97044]: pgmap v7861: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:34:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:53.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:53.906640+0000) 2022-09-10T08:34:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:54.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:54.343744+0000) 2022-09-10T08:34:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:54.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:54.906771+0000) 2022-09-10T08:34:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:55.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:55.343942+0000) 2022-09-10T08:34:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:55 smithi110 ceph-mon[97633]: pgmap v7862: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:55 smithi104 ceph-mon[97044]: pgmap v7862: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:34:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:55.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:55.906896+0000) 2022-09-10T08:34:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:56.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:56.344260+0000) 2022-09-10T08:34:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:56.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:56.907022+0000) 2022-09-10T08:34:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:57.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:57.344481+0000) 2022-09-10T08:34:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:57 smithi110 ceph-mon[97633]: pgmap v7863: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:34:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:57 smithi104 ceph-mon[97044]: pgmap v7863: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:34:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:57.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:57.907150+0000) 2022-09-10T08:34:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:58.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:58.344717+0000) 2022-09-10T08:34:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:58.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:58.907293+0000) 2022-09-10T08:34:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:34:59.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:59.344937+0000) 2022-09-10T08:35:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:34:59 smithi110 ceph-mon[97633]: pgmap v7864: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:35:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:34:59 smithi104 ceph-mon[97044]: pgmap v7864: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:35:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:34:59.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:34:59.907425+0000) 2022-09-10T08:35:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:00.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:00.345159+0000) 2022-09-10T08:35:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:00.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:00.907540+0000) 2022-09-10T08:35:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:01.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:01.345391+0000) 2022-09-10T08:35:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:01 smithi110 ceph-mon[97633]: pgmap v7865: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:01 smithi104 ceph-mon[97044]: pgmap v7865: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:01.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:01.907682+0000) 2022-09-10T08:35:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:02.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:02.345641+0000) 2022-09-10T08:35:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:02.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:02.907834+0000) 2022-09-10T08:35:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:03.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:03.345892+0000) 2022-09-10T08:35:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:03 smithi110 ceph-mon[97633]: pgmap v7866: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:03 smithi104 ceph-mon[97044]: pgmap v7866: 129 pgs: 129 active+clean; 8.5 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:03.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:03.908042+0000) 2022-09-10T08:35:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:04.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:04.346130+0000) 2022-09-10T08:35:04.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:35:04.347+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:35:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:04.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:04.908171+0000) 2022-09-10T08:35:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:05.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:05.346337+0000) 2022-09-10T08:35:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:05 smithi110 ceph-mon[97633]: pgmap v7867: 129 pgs: 129 active+clean; 8.5 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:35:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:05 smithi104 ceph-mon[97044]: pgmap v7867: 129 pgs: 129 active+clean; 8.5 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:35:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:05.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:05.908350+0000) 2022-09-10T08:35:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:06.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:06.346567+0000) 2022-09-10T08:35:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:06.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:06.908493+0000) 2022-09-10T08:35:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:07.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:07.346733+0000) 2022-09-10T08:35:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:07 smithi110 ceph-mon[97633]: pgmap v7868: 129 pgs: 129 active+clean; 8.5 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:35:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:07 smithi104 ceph-mon[97044]: pgmap v7868: 129 pgs: 129 active+clean; 8.5 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:35:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:07.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:07.908633+0000) 2022-09-10T08:35:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:08.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:08.346959+0000) 2022-09-10T08:35:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:08 smithi104 ceph-mon[97044]: osdmap e572: 12 total, 12 up, 12 in 2022-09-10T08:35:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:08 smithi104 ceph-mon[97044]: pgmap v7870: 129 pgs: 129 active+clean; 8.5 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:35:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:08.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:08.908764+0000) 2022-09-10T08:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:08 smithi110 ceph-mon[97633]: osdmap e572: 12 total, 12 up, 12 in 2022-09-10T08:35:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:08 smithi110 ceph-mon[97633]: pgmap v7870: 129 pgs: 129 active+clean; 8.5 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:35:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:09.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:09.347208+0000) 2022-09-10T08:35:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:09.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:09.908890+0000) 2022-09-10T08:35:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:10.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:10.347431+0000) 2022-09-10T08:35:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:10.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:10.909018+0000) 2022-09-10T08:35:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:11.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:11.347663+0000) 2022-09-10T08:35:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:11 smithi110 ceph-mon[97633]: pgmap v7871: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:35:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:11 smithi110 ceph-mon[97633]: osdmap e573: 12 total, 12 up, 12 in 2022-09-10T08:35:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:11 smithi104 ceph-mon[97044]: pgmap v7871: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:35:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:11 smithi104 ceph-mon[97044]: osdmap e573: 12 total, 12 up, 12 in 2022-09-10T08:35:12.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:11.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:11.909163+0000) 2022-09-10T08:35:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:12.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:12.347893+0000) 2022-09-10T08:35:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:12.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:12.909332+0000) 2022-09-10T08:35:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:13.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:13.348159+0000) 2022-09-10T08:35:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:13 smithi110 ceph-mon[97633]: pgmap v7873: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:35:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:13 smithi104 ceph-mon[97044]: pgmap v7873: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:35:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:13.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:13.909493+0000) 2022-09-10T08:35:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:14.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:14.348411+0000) 2022-09-10T08:35:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:14.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:14.909637+0000) 2022-09-10T08:35:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:15.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:15.348599+0000) 2022-09-10T08:35:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:15 smithi110 ceph-mon[97633]: pgmap v7874: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:35:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:15 smithi104 ceph-mon[97044]: pgmap v7874: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:35:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:15.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:15.909771+0000) 2022-09-10T08:35:16.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:16.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:16.348852+0000) 2022-09-10T08:35:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:16.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:16.909910+0000) 2022-09-10T08:35:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:17.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:17.349124+0000) 2022-09-10T08:35:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:17 smithi110 ceph-mon[97633]: pgmap v7875: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:35:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:17 smithi104 ceph-mon[97044]: pgmap v7875: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:35:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:17.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:17.910082+0000) 2022-09-10T08:35:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:18.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:18.349289+0000) 2022-09-10T08:35:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:18.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:18.910269+0000) 2022-09-10T08:35:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:19.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:19.349532+0000) 2022-09-10T08:35:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:35:19.362+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:35:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:19 smithi110 ceph-mon[97633]: pgmap v7876: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:35:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:19 smithi104 ceph-mon[97044]: pgmap v7876: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:35:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:19.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:19.910450+0000) 2022-09-10T08:35:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:20.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:20.349794+0000) 2022-09-10T08:35:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:20.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:20.910593+0000) 2022-09-10T08:35:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:21.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:21.349995+0000) 2022-09-10T08:35:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:21 smithi104 ceph-mon[97044]: pgmap v7877: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:35:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:21.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:21.910739+0000) 2022-09-10T08:35:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:21 smithi110 ceph-mon[97633]: pgmap v7877: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:35:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:22.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:22.350233+0000) 2022-09-10T08:35:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:22.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:22.910928+0000) 2022-09-10T08:35:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:23.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:23.350489+0000) 2022-09-10T08:35:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:23 smithi104 ceph-mon[97044]: pgmap v7878: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:35:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:23.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:23.911106+0000) 2022-09-10T08:35:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:23 smithi110 ceph-mon[97633]: pgmap v7878: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:35:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:24.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:24.350738+0000) 2022-09-10T08:35:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:24 smithi104 ceph-mon[97044]: pgmap v7879: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:35:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:24.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:24.911241+0000) 2022-09-10T08:35:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:24 smithi110 ceph-mon[97633]: pgmap v7879: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:35:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:25.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:25.350918+0000) 2022-09-10T08:35:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:25.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:25.911386+0000) 2022-09-10T08:35:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:26.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:26.351132+0000) 2022-09-10T08:35:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:26.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:26.911525+0000) 2022-09-10T08:35:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:27.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:27.351284+0000) 2022-09-10T08:35:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:27 smithi110 ceph-mon[97633]: pgmap v7880: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:27 smithi104 ceph-mon[97044]: pgmap v7880: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:27.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:27.911656+0000) 2022-09-10T08:35:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:28.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:28.351496+0000) 2022-09-10T08:35:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:28.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:28.911788+0000) 2022-09-10T08:35:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:29.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:29.351662+0000) 2022-09-10T08:35:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:29 smithi110 ceph-mon[97633]: pgmap v7881: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:29 smithi104 ceph-mon[97044]: pgmap v7881: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:29.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:29.911918+0000) 2022-09-10T08:35:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:30.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:30.351901+0000) 2022-09-10T08:35:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:30.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:30.912051+0000) 2022-09-10T08:35:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:31.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:31.352176+0000) 2022-09-10T08:35:32.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:31 smithi110 ceph-mon[97633]: pgmap v7882: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:31 smithi104 ceph-mon[97044]: pgmap v7882: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:31.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:31.912184+0000) 2022-09-10T08:35:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:32.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:32.352452+0000) 2022-09-10T08:35:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:32.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:32.912360+0000) 2022-09-10T08:35:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:33.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:33.352626+0000) 2022-09-10T08:35:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:33 smithi104 ceph-mon[97044]: pgmap v7883: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:33.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:33.912482+0000) 2022-09-10T08:35:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:33 smithi110 ceph-mon[97633]: pgmap v7883: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:34.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:34.352816+0000) 2022-09-10T08:35:34.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:35:34.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:35:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:34.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:34.912608+0000) 2022-09-10T08:35:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:35.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:35.353040+0000) 2022-09-10T08:35:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:35 smithi104 ceph-mon[97044]: pgmap v7884: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:35.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:35.912744+0000) 2022-09-10T08:35:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:35 smithi110 ceph-mon[97633]: pgmap v7884: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:36.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:36.353296+0000) 2022-09-10T08:35:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:36.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:36.912910+0000) 2022-09-10T08:35:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:37.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:37.353535+0000) 2022-09-10T08:35:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:37 smithi104 ceph-mon[97044]: pgmap v7885: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:37.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:37.913163+0000) 2022-09-10T08:35:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:37 smithi110 ceph-mon[97633]: pgmap v7885: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:38.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:38.353745+0000) 2022-09-10T08:35:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:38 smithi104 ceph-mon[97044]: pgmap v7886: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:38.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:38.913296+0000) 2022-09-10T08:35:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:38 smithi110 ceph-mon[97633]: pgmap v7886: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:39.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:39.353998+0000) 2022-09-10T08:35:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:39.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:39.913423+0000) 2022-09-10T08:35:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:40.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:40.354259+0000) 2022-09-10T08:35:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:40.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:40.913542+0000) 2022-09-10T08:35:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:41.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:41.354484+0000) 2022-09-10T08:35:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:41 smithi110 ceph-mon[97633]: pgmap v7887: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:41 smithi104 ceph-mon[97044]: pgmap v7887: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:41.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:41.913673+0000) 2022-09-10T08:35:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:42.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:42.354696+0000) 2022-09-10T08:35:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:42.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:42.913793+0000) 2022-09-10T08:35:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:43.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:43.354848+0000) 2022-09-10T08:35:44.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:43 smithi110 ceph-mon[97633]: pgmap v7888: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:43 smithi104 ceph-mon[97044]: pgmap v7888: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:43.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:43.913921+0000) 2022-09-10T08:35:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:44.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:44.355046+0000) 2022-09-10T08:35:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:35:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:35:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:35:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:35:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:44.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:44.914055+0000) 2022-09-10T08:35:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:35:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:35:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:35:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:35:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:45.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:45.355242+0000) 2022-09-10T08:35:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:45 smithi104 ceph-mon[97044]: pgmap v7889: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:45.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:45.914238+0000) 2022-09-10T08:35:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:45 smithi110 ceph-mon[97633]: pgmap v7889: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:46.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:46.355380+0000) 2022-09-10T08:35:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:46.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:46.914417+0000) 2022-09-10T08:35:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:47.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:47.355513+0000) 2022-09-10T08:35:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:47 smithi104 ceph-mon[97044]: pgmap v7890: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:47.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:47.914535+0000) 2022-09-10T08:35:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:47 smithi110 ceph-mon[97633]: pgmap v7890: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:48.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:48.355679+0000) 2022-09-10T08:35:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:48.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:48.914654+0000) 2022-09-10T08:35:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:49.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:49.355939+0000) 2022-09-10T08:35:49.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:35:49.374+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:35:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:49 smithi104 ceph-mon[97044]: pgmap v7891: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:49.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:49.914779+0000) 2022-09-10T08:35:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:49 smithi110 ceph-mon[97633]: pgmap v7891: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:50.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:50.356176+0000) 2022-09-10T08:35:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:50.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:50.914972+0000) 2022-09-10T08:35:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:51.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:51.356382+0000) 2022-09-10T08:35:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:51 smithi104 ceph-mon[97044]: pgmap v7892: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:51.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:51.915131+0000) 2022-09-10T08:35:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:51 smithi110 ceph-mon[97633]: pgmap v7892: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:52.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:52.356576+0000) 2022-09-10T08:35:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:52 smithi104 ceph-mon[97044]: pgmap v7893: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:52.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:52.915257+0000) 2022-09-10T08:35:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:52 smithi110 ceph-mon[97633]: pgmap v7893: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:53.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:53.356777+0000) 2022-09-10T08:35:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:53.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:53.915379+0000) 2022-09-10T08:35:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:54.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:54.357039+0000) 2022-09-10T08:35:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:54.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:54.915501+0000) 2022-09-10T08:35:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:55 smithi104 ceph-mon[97044]: pgmap v7894: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:55.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:55.357251+0000) 2022-09-10T08:35:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:55 smithi110 ceph-mon[97633]: pgmap v7894: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:35:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:55.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:55.915641+0000) 2022-09-10T08:35:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:56.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:56.357525+0000) 2022-09-10T08:35:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:56.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:56.915768+0000) 2022-09-10T08:35:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:57.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:57.357688+0000) 2022-09-10T08:35:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:57 smithi110 ceph-mon[97633]: pgmap v7895: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:57 smithi104 ceph-mon[97044]: pgmap v7895: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:35:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:57.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:57.915932+0000) 2022-09-10T08:35:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:58.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:58.357903+0000) 2022-09-10T08:35:59.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:58.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:58.916075+0000) 2022-09-10T08:35:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:35:59.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:59.358046+0000) 2022-09-10T08:36:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:35:59 smithi104 ceph-mon[97044]: pgmap v7896: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:35:59.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:35:59.916266+0000) 2022-09-10T08:36:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:35:59 smithi110 ceph-mon[97633]: pgmap v7896: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:00.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:00.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:00.358155+0000) 2022-09-10T08:36:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:00.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:00.916423+0000) 2022-09-10T08:36:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:01.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:01.358329+0000) 2022-09-10T08:36:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:01 smithi104 ceph-mon[97044]: pgmap v7897: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:01.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:01.916605+0000) 2022-09-10T08:36:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:01 smithi110 ceph-mon[97633]: pgmap v7897: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:02.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:02.358455+0000) 2022-09-10T08:36:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:02.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:02.916785+0000) 2022-09-10T08:36:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:03.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:03.358653+0000) 2022-09-10T08:36:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:03 smithi104 ceph-mon[97044]: pgmap v7898: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:03.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:03.916916+0000) 2022-09-10T08:36:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:03 smithi110 ceph-mon[97633]: pgmap v7898: 129 pgs: 129 active+clean; 8.5 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:04.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:04.358851+0000) 2022-09-10T08:36:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:36:04.385+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:36:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:04.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:04.917041+0000) 2022-09-10T08:36:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:05.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:05.359069+0000) 2022-09-10T08:36:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:05 smithi104 ceph-mon[97044]: pgmap v7899: 129 pgs: 129 active+clean; 8.5 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:36:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:05.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:05.917179+0000) 2022-09-10T08:36:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:05 smithi110 ceph-mon[97633]: pgmap v7899: 129 pgs: 129 active+clean; 8.5 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:36:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:06.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:06.359387+0000) 2022-09-10T08:36:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:06 smithi104 ceph-mon[97044]: pgmap v7900: 129 pgs: 129 active+clean; 8.5 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:36:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:06.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:06.917313+0000) 2022-09-10T08:36:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:06 smithi110 ceph-mon[97633]: pgmap v7900: 129 pgs: 129 active+clean; 8.5 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:36:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:07.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:07.359557+0000) 2022-09-10T08:36:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:07.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:07.917444+0000) 2022-09-10T08:36:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:08.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:08.359775+0000) 2022-09-10T08:36:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:08 smithi104 ceph-mon[97044]: osdmap e574: 12 total, 12 up, 12 in 2022-09-10T08:36:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:08 smithi104 ceph-mon[97044]: pgmap v7902: 129 pgs: 129 active+clean; 8.5 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:36:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:08.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:08.917587+0000) 2022-09-10T08:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:08 smithi110 ceph-mon[97633]: osdmap e574: 12 total, 12 up, 12 in 2022-09-10T08:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:08 smithi110 ceph-mon[97633]: pgmap v7902: 129 pgs: 129 active+clean; 8.5 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:36:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:09.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:09.360000+0000) 2022-09-10T08:36:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:09.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:09.917723+0000) 2022-09-10T08:36:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:10.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:10.360269+0000) 2022-09-10T08:36:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:10.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:10.917899+0000) 2022-09-10T08:36:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:11.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:11.360495+0000) 2022-09-10T08:36:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:11 smithi104 ceph-mon[97044]: pgmap v7903: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:36:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:11 smithi104 ceph-mon[97044]: osdmap e575: 12 total, 12 up, 12 in 2022-09-10T08:36:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:11.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:11.918057+0000) 2022-09-10T08:36:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:11 smithi110 ceph-mon[97633]: pgmap v7903: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:36:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:11 smithi110 ceph-mon[97633]: osdmap e575: 12 total, 12 up, 12 in 2022-09-10T08:36:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:12.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:12.360619+0000) 2022-09-10T08:36:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:12.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:12.918168+0000) 2022-09-10T08:36:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:13.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:13.360806+0000) 2022-09-10T08:36:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:13 smithi104 ceph-mon[97044]: pgmap v7905: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:36:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:13.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:13.918303+0000) 2022-09-10T08:36:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:13 smithi110 ceph-mon[97633]: pgmap v7905: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:36:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:14.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:14.360967+0000) 2022-09-10T08:36:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:14.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:14.918439+0000) 2022-09-10T08:36:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:15.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:15.361176+0000) 2022-09-10T08:36:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:15 smithi104 ceph-mon[97044]: pgmap v7906: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:36:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:15.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:15.918565+0000) 2022-09-10T08:36:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:15 smithi110 ceph-mon[97633]: pgmap v7906: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:36:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:16.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:16.361341+0000) 2022-09-10T08:36:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:16.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:16.918692+0000) 2022-09-10T08:36:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:17.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:17.361537+0000) 2022-09-10T08:36:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:17 smithi104 ceph-mon[97044]: pgmap v7907: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:36:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:17.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:17.918811+0000) 2022-09-10T08:36:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:17 smithi110 ceph-mon[97633]: pgmap v7907: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T08:36:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:18.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:18.361703+0000) 2022-09-10T08:36:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:18.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:18.918933+0000) 2022-09-10T08:36:19.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:36:19.394+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:36:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:19.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:19.361929+0000) 2022-09-10T08:36:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:19 smithi104 ceph-mon[97044]: pgmap v7908: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:36:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:19.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:19.919066+0000) 2022-09-10T08:36:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:19 smithi110 ceph-mon[97633]: pgmap v7908: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:36:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:20.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:20.362183+0000) 2022-09-10T08:36:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:20.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:20.919154+0000) 2022-09-10T08:36:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:21.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:21.362374+0000) 2022-09-10T08:36:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:21 smithi104 ceph-mon[97044]: pgmap v7909: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:36:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:21.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:21.919319+0000) 2022-09-10T08:36:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:21 smithi110 ceph-mon[97633]: pgmap v7909: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:36:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:22.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:22.362602+0000) 2022-09-10T08:36:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:22 smithi104 ceph-mon[97044]: pgmap v7910: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:36:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:22.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:22.919453+0000) 2022-09-10T08:36:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:22 smithi110 ceph-mon[97633]: pgmap v7910: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:36:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:23.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:23.362801+0000) 2022-09-10T08:36:24.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:23.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:23.919597+0000) 2022-09-10T08:36:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:24.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:24.363129+0000) 2022-09-10T08:36:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:24.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:24.919819+0000) 2022-09-10T08:36:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:25.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:25.363336+0000) 2022-09-10T08:36:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:25 smithi110 ceph-mon[97633]: pgmap v7911: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:36:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:25 smithi104 ceph-mon[97044]: pgmap v7911: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:36:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:25.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:25.919967+0000) 2022-09-10T08:36:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:26.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:26.363544+0000) 2022-09-10T08:36:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:26.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:26.920144+0000) 2022-09-10T08:36:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:27.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:27.363718+0000) 2022-09-10T08:36:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:27 smithi104 ceph-mon[97044]: pgmap v7912: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:27.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:27.920269+0000) 2022-09-10T08:36:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:27 smithi110 ceph-mon[97633]: pgmap v7912: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:28.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:28.363963+0000) 2022-09-10T08:36:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:28.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:28.920430+0000) 2022-09-10T08:36:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:29.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:29.364209+0000) 2022-09-10T08:36:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:29 smithi104 ceph-mon[97044]: pgmap v7913: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:29.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:29.920658+0000) 2022-09-10T08:36:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:29 smithi110 ceph-mon[97633]: pgmap v7913: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:30.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:30.364387+0000) 2022-09-10T08:36:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:30.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:30.920862+0000) 2022-09-10T08:36:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:31.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:31.364532+0000) 2022-09-10T08:36:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:31 smithi104 ceph-mon[97044]: pgmap v7914: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:31.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:31.921016+0000) 2022-09-10T08:36:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:31 smithi110 ceph-mon[97633]: pgmap v7914: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:32.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:32.364739+0000) 2022-09-10T08:36:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:32 smithi104 ceph-mon[97044]: pgmap v7915: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:32.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:32.921148+0000) 2022-09-10T08:36:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:32 smithi110 ceph-mon[97633]: pgmap v7915: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:33.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:33.364968+0000) 2022-09-10T08:36:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:33.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:33.921274+0000) 2022-09-10T08:36:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:34.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:34.365240+0000) 2022-09-10T08:36:34.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:36:34.402+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:36:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:34.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:34.921405+0000) 2022-09-10T08:36:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:35.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:35.365437+0000) 2022-09-10T08:36:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:35 smithi110 ceph-mon[97633]: pgmap v7916: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:35 smithi104 ceph-mon[97044]: pgmap v7916: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:35.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:35.921531+0000) 2022-09-10T08:36:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:36.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:36.365669+0000) 2022-09-10T08:36:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:36.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:36.921648+0000) 2022-09-10T08:36:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:37.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:37.365887+0000) 2022-09-10T08:36:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:37 smithi104 ceph-mon[97044]: pgmap v7917: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:37.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:37.921800+0000) 2022-09-10T08:36:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:37 smithi110 ceph-mon[97633]: pgmap v7917: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:38.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:38.366168+0000) 2022-09-10T08:36:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:38.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:38.921944+0000) 2022-09-10T08:36:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:39.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:39.366422+0000) 2022-09-10T08:36:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:39 smithi104 ceph-mon[97044]: pgmap v7918: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:39.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:39.922130+0000) 2022-09-10T08:36:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:39 smithi110 ceph-mon[97633]: pgmap v7918: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:40.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:40.366659+0000) 2022-09-10T08:36:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:40.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:40.922318+0000) 2022-09-10T08:36:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:41.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:41.366938+0000) 2022-09-10T08:36:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:41 smithi104 ceph-mon[97044]: pgmap v7919: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:41.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:41.922447+0000) 2022-09-10T08:36:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:41 smithi110 ceph-mon[97633]: pgmap v7919: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:42.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:42.367148+0000) 2022-09-10T08:36:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:42.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:42.922574+0000) 2022-09-10T08:36:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:43.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:43.367309+0000) 2022-09-10T08:36:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:43 smithi104 ceph-mon[97044]: pgmap v7920: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:43.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:43.922711+0000) 2022-09-10T08:36:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:43 smithi110 ceph-mon[97633]: pgmap v7920: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:44.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:44.367547+0000) 2022-09-10T08:36:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:36:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:36:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:36:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:36:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:44.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:44.922850+0000) 2022-09-10T08:36:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:36:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:36:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:36:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:36:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:45.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:45.367818+0000) 2022-09-10T08:36:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:45 smithi104 ceph-mon[97044]: pgmap v7921: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:45.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:45.922988+0000) 2022-09-10T08:36:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:45 smithi110 ceph-mon[97633]: pgmap v7921: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:46.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:46.368120+0000) 2022-09-10T08:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:46 smithi104 ceph-mon[97044]: pgmap v7922: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:46.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:46.923132+0000) 2022-09-10T08:36:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:46 smithi110 ceph-mon[97633]: pgmap v7922: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:47.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:47.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:47.368412+0000) 2022-09-10T08:36:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:47.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:47.923260+0000) 2022-09-10T08:36:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:48.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:48.368631+0000) 2022-09-10T08:36:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:48.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:48.923383+0000) 2022-09-10T08:36:49.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:36:49.412+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:36:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:49.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:49.368766+0000) 2022-09-10T08:36:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:49 smithi110 ceph-mon[97633]: pgmap v7923: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:49 smithi104 ceph-mon[97044]: pgmap v7923: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:49.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:49.923521+0000) 2022-09-10T08:36:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:50.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:50.369019+0000) 2022-09-10T08:36:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:50.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:50.923693+0000) 2022-09-10T08:36:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:51.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:51.369267+0000) 2022-09-10T08:36:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:51 smithi104 ceph-mon[97044]: pgmap v7924: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:51.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:51.923831+0000) 2022-09-10T08:36:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:51 smithi110 ceph-mon[97633]: pgmap v7924: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:52.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:52.369489+0000) 2022-09-10T08:36:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:52.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:52.923943+0000) 2022-09-10T08:36:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:53.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:53.369717+0000) 2022-09-10T08:36:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:53 smithi104 ceph-mon[97044]: pgmap v7925: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:54.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:53.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:53.924067+0000) 2022-09-10T08:36:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:53 smithi110 ceph-mon[97633]: pgmap v7925: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:54.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:54.369979+0000) 2022-09-10T08:36:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:54.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:54.924169+0000) 2022-09-10T08:36:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:55.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:55.370226+0000) 2022-09-10T08:36:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:55 smithi104 ceph-mon[97044]: pgmap v7926: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:55.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:55.924303+0000) 2022-09-10T08:36:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:55 smithi110 ceph-mon[97633]: pgmap v7926: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:36:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:56.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:56.370462+0000) 2022-09-10T08:36:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:56.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:56.924422+0000) 2022-09-10T08:36:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:57.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:57.370645+0000) 2022-09-10T08:36:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:57 smithi104 ceph-mon[97044]: pgmap v7927: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:57.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:57.924551+0000) 2022-09-10T08:36:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:57 smithi110 ceph-mon[97633]: pgmap v7927: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:58.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:58.370859+0000) 2022-09-10T08:36:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:36:58 smithi104 ceph-mon[97044]: pgmap v7928: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:58.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:58.924716+0000) 2022-09-10T08:36:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:36:58 smithi110 ceph-mon[97633]: pgmap v7928: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:36:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:36:59.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:59.371040+0000) 2022-09-10T08:37:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:36:59.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:36:59.924851+0000) 2022-09-10T08:37:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:00.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:00.371184+0000) 2022-09-10T08:37:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:00.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:00.924964+0000) 2022-09-10T08:37:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:01.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:01.371417+0000) 2022-09-10T08:37:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:01 smithi104 ceph-mon[97044]: pgmap v7929: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:01.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:01.925088+0000) 2022-09-10T08:37:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:01 smithi110 ceph-mon[97633]: pgmap v7929: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:02.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:02.371645+0000) 2022-09-10T08:37:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:02.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:02.925234+0000) 2022-09-10T08:37:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:03.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:03.371875+0000) 2022-09-10T08:37:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:03 smithi104 ceph-mon[97044]: pgmap v7930: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:03.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:03.925412+0000) 2022-09-10T08:37:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:03 smithi110 ceph-mon[97633]: pgmap v7930: 129 pgs: 129 active+clean; 8.5 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:04.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:37:04.427+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:37:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:04.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:04.372041+0000) 2022-09-10T08:37:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:04.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:04.925542+0000) 2022-09-10T08:37:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:05.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:05.372221+0000) 2022-09-10T08:37:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:05 smithi104 ceph-mon[97044]: pgmap v7931: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:37:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:05.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:05.925674+0000) 2022-09-10T08:37:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:05 smithi110 ceph-mon[97633]: pgmap v7931: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:37:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:06.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:06.372399+0000) 2022-09-10T08:37:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:06.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:06.925799+0000) 2022-09-10T08:37:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:07.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:07.372588+0000) 2022-09-10T08:37:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:07 smithi104 ceph-mon[97044]: pgmap v7932: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:37:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:07.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:07.925969+0000) 2022-09-10T08:37:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:07 smithi110 ceph-mon[97633]: pgmap v7932: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:37:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:08.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:08.372815+0000) 2022-09-10T08:37:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:08 smithi104 ceph-mon[97044]: osdmap e576: 12 total, 12 up, 12 in 2022-09-10T08:37:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:08 smithi104 ceph-mon[97044]: pgmap v7934: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:37:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:08.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:08.926128+0000) 2022-09-10T08:37:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:08 smithi110 ceph-mon[97633]: osdmap e576: 12 total, 12 up, 12 in 2022-09-10T08:37:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:08 smithi110 ceph-mon[97633]: pgmap v7934: 129 pgs: 129 active+clean; 8.5 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:37:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:09.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:09.372980+0000) 2022-09-10T08:37:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:09.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:09.926295+0000) 2022-09-10T08:37:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:10.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:10.373229+0000) 2022-09-10T08:37:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:10.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:10.926440+0000) 2022-09-10T08:37:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:11.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:11.373465+0000) 2022-09-10T08:37:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:11 smithi104 ceph-mon[97044]: pgmap v7935: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:37:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:11 smithi104 ceph-mon[97044]: osdmap e577: 12 total, 12 up, 12 in 2022-09-10T08:37:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:11.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:11.926612+0000) 2022-09-10T08:37:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:11 smithi110 ceph-mon[97633]: pgmap v7935: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:37:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:11 smithi110 ceph-mon[97633]: osdmap e577: 12 total, 12 up, 12 in 2022-09-10T08:37:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:12.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:12.373692+0000) 2022-09-10T08:37:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:12.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:12.926740+0000) 2022-09-10T08:37:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:13.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:13.373935+0000) 2022-09-10T08:37:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:13 smithi104 ceph-mon[97044]: pgmap v7937: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:37:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:13.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:13.926873+0000) 2022-09-10T08:37:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:13 smithi110 ceph-mon[97633]: pgmap v7937: 129 pgs: 129 active+clean; 8.5 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:37:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:14.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:14.374146+0000) 2022-09-10T08:37:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:14.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:14.927037+0000) 2022-09-10T08:37:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:15.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:15.374376+0000) 2022-09-10T08:37:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:15 smithi104 ceph-mon[97044]: pgmap v7938: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:37:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:15.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:15.927170+0000) 2022-09-10T08:37:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:15 smithi110 ceph-mon[97633]: pgmap v7938: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:37:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:16.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:16.374633+0000) 2022-09-10T08:37:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:16 smithi104 ceph-mon[97044]: pgmap v7939: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:37:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:16.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:16.927289+0000) 2022-09-10T08:37:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:16 smithi110 ceph-mon[97633]: pgmap v7939: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:37:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:17.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:17.374869+0000) 2022-09-10T08:37:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:17.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:17.927419+0000) 2022-09-10T08:37:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:18.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:18.375125+0000) 2022-09-10T08:37:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:18.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:18.927559+0000) 2022-09-10T08:37:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:19.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:19.375333+0000) 2022-09-10T08:37:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:37:19.443+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:37:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:19 smithi110 ceph-mon[97633]: pgmap v7940: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:37:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:19 smithi104 ceph-mon[97044]: pgmap v7940: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:37:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:19.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:19.927692+0000) 2022-09-10T08:37:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:20.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:20.375536+0000) 2022-09-10T08:37:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:20.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:20.927832+0000) 2022-09-10T08:37:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:21.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:21.375753+0000) 2022-09-10T08:37:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:21 smithi104 ceph-mon[97044]: pgmap v7941: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:37:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:21.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:21.928021+0000) 2022-09-10T08:37:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:21 smithi110 ceph-mon[97633]: pgmap v7941: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:37:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:22.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:22.375889+0000) 2022-09-10T08:37:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:22.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:22.928162+0000) 2022-09-10T08:37:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:23.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:23.376091+0000) 2022-09-10T08:37:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:23 smithi104 ceph-mon[97044]: pgmap v7942: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:37:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:23.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:23.928307+0000) 2022-09-10T08:37:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:23 smithi110 ceph-mon[97633]: pgmap v7942: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:37:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:24.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:24.376336+0000) 2022-09-10T08:37:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:24.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:24.928452+0000) 2022-09-10T08:37:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:25.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:25.376585+0000) 2022-09-10T08:37:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:25 smithi104 ceph-mon[97044]: pgmap v7943: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:37:26.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:25.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:25.928581+0000) 2022-09-10T08:37:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:25 smithi110 ceph-mon[97633]: pgmap v7943: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:37:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:26.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:26.376804+0000) 2022-09-10T08:37:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:26 smithi104 ceph-mon[97044]: pgmap v7944: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:26.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:26.928723+0000) 2022-09-10T08:37:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:26 smithi110 ceph-mon[97633]: pgmap v7944: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:27.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:27.377032+0000) 2022-09-10T08:37:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:27.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:27.928861+0000) 2022-09-10T08:37:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:28.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:28.377217+0000) 2022-09-10T08:37:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:28.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:28.929076+0000) 2022-09-10T08:37:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:29.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:29.377393+0000) 2022-09-10T08:37:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:29 smithi110 ceph-mon[97633]: pgmap v7945: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:29 smithi104 ceph-mon[97044]: pgmap v7945: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:29.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:29.929244+0000) 2022-09-10T08:37:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:30.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:30.377606+0000) 2022-09-10T08:37:31.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:30.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:30.929365+0000) 2022-09-10T08:37:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:31.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:31.377841+0000) 2022-09-10T08:37:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:31 smithi104 ceph-mon[97044]: pgmap v7946: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:31.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:31.929495+0000) 2022-09-10T08:37:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:31 smithi110 ceph-mon[97633]: pgmap v7946: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:32.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:32.378066+0000) 2022-09-10T08:37:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:32.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:32.929650+0000) 2022-09-10T08:37:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:33.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:33.378299+0000) 2022-09-10T08:37:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:33 smithi104 ceph-mon[97044]: pgmap v7947: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:33.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:33.929773+0000) 2022-09-10T08:37:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:33 smithi110 ceph-mon[97633]: pgmap v7947: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:37:34.458+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:37:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:34.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:34.378534+0000) 2022-09-10T08:37:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:34 smithi104 ceph-mon[97044]: pgmap v7948: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:34.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:34.929915+0000) 2022-09-10T08:37:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:34 smithi110 ceph-mon[97633]: pgmap v7948: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:35.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:35.378778+0000) 2022-09-10T08:37:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:35.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:35.930056+0000) 2022-09-10T08:37:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:36.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:36.379056+0000) 2022-09-10T08:37:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:36.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:36.930213+0000) 2022-09-10T08:37:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:37.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:37.379278+0000) 2022-09-10T08:37:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:37 smithi104 ceph-mon[97044]: pgmap v7949: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:38.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:37.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:37.930349+0000) 2022-09-10T08:37:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:37 smithi110 ceph-mon[97633]: pgmap v7949: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:38.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:38.379511+0000) 2022-09-10T08:37:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:38.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:38.930533+0000) 2022-09-10T08:37:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:39.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:39.379701+0000) 2022-09-10T08:37:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:39 smithi104 ceph-mon[97044]: pgmap v7950: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:39.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:39.930671+0000) 2022-09-10T08:37:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:39 smithi110 ceph-mon[97633]: pgmap v7950: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:40.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:40.379957+0000) 2022-09-10T08:37:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:40.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:40.930810+0000) 2022-09-10T08:37:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:41.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:41.380219+0000) 2022-09-10T08:37:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:41 smithi104 ceph-mon[97044]: pgmap v7951: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:41.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:41.930949+0000) 2022-09-10T08:37:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:41 smithi110 ceph-mon[97633]: pgmap v7951: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:42.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:42.380386+0000) 2022-09-10T08:37:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:42.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:42.931089+0000) 2022-09-10T08:37:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:43.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:43.380571+0000) 2022-09-10T08:37:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:43 smithi104 ceph-mon[97044]: pgmap v7952: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:43.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:43.931238+0000) 2022-09-10T08:37:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:43 smithi110 ceph-mon[97633]: pgmap v7952: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:44.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:44.380810+0000) 2022-09-10T08:37:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:44 smithi104 ceph-mon[97044]: pgmap v7953: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:37:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:37:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:37:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:37:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:44.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:44.931419+0000) 2022-09-10T08:37:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:44 smithi110 ceph-mon[97633]: pgmap v7953: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:37:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:37:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:37:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:37:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:45.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:45.381030+0000) 2022-09-10T08:37:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:45.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:45.931557+0000) 2022-09-10T08:37:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:46.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:46.381310+0000) 2022-09-10T08:37:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:46.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:46.931694+0000) 2022-09-10T08:37:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:47.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:47.381506+0000) 2022-09-10T08:37:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:47 smithi104 ceph-mon[97044]: pgmap v7954: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:47.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:47.931905+0000) 2022-09-10T08:37:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:47 smithi110 ceph-mon[97633]: pgmap v7954: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:48.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:48.381695+0000) 2022-09-10T08:37:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:48 smithi104 ceph-mon[97044]: pgmap v7955: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:48.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:48.932116+0000) 2022-09-10T08:37:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:48 smithi110 ceph-mon[97633]: pgmap v7955: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:49.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:37:49.472+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:37:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:49.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:49.381856+0000) 2022-09-10T08:37:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:49.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:49.932311+0000) 2022-09-10T08:37:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:50.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:50.382159+0000) 2022-09-10T08:37:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:50.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:50.932479+0000) 2022-09-10T08:37:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:51.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:51.382343+0000) 2022-09-10T08:37:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:51 smithi104 ceph-mon[97044]: pgmap v7956: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:51.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:51.932624+0000) 2022-09-10T08:37:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:51 smithi110 ceph-mon[97633]: pgmap v7956: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:52.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:52.382582+0000) 2022-09-10T08:37:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:52.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:52.932790+0000) 2022-09-10T08:37:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:53.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:53.382830+0000) 2022-09-10T08:37:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:53 smithi104 ceph-mon[97044]: pgmap v7957: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:53.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:53.932919+0000) 2022-09-10T08:37:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:53 smithi110 ceph-mon[97633]: pgmap v7957: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:54.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:54.383079+0000) 2022-09-10T08:37:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:54 smithi104 ceph-mon[97044]: pgmap v7958: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:54.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:54.933045+0000) 2022-09-10T08:37:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:54 smithi110 ceph-mon[97633]: pgmap v7958: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:37:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:55.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:55.383239+0000) 2022-09-10T08:37:56.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:55.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:55.933169+0000) 2022-09-10T08:37:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:56.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:56.383425+0000) 2022-09-10T08:37:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:56.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:56.933296+0000) 2022-09-10T08:37:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:57.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:57.383698+0000) 2022-09-10T08:37:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:57 smithi104 ceph-mon[97044]: pgmap v7959: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:57.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:57.933430+0000) 2022-09-10T08:37:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:57 smithi110 ceph-mon[97633]: pgmap v7959: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:58.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:58.383940+0000) 2022-09-10T08:37:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:37:58 smithi104 ceph-mon[97044]: pgmap v7960: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:58.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:58.933571+0000) 2022-09-10T08:37:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:37:58 smithi110 ceph-mon[97633]: pgmap v7960: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:37:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:37:59.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:59.384191+0000) 2022-09-10T08:38:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:37:59.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:37:59.933695+0000) 2022-09-10T08:38:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:00.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:00.384420+0000) 2022-09-10T08:38:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:00.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:00.933815+0000) 2022-09-10T08:38:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:01.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:01.384601+0000) 2022-09-10T08:38:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:01 smithi104 ceph-mon[97044]: pgmap v7961: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:01.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:01.933953+0000) 2022-09-10T08:38:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:01 smithi110 ceph-mon[97633]: pgmap v7961: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:02.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:02.384824+0000) 2022-09-10T08:38:03.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:02 smithi104 ceph-mon[97044]: pgmap v7962: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:02.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:02.934110+0000) 2022-09-10T08:38:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:02 smithi110 ceph-mon[97633]: pgmap v7962: 129 pgs: 129 active+clean; 8.5 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:03.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:03.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:03.385039+0000) 2022-09-10T08:38:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:03.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:03.934299+0000) 2022-09-10T08:38:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:38:04.480+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:38:04.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:04.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:04.385321+0000) 2022-09-10T08:38:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:04.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:04.934492+0000) 2022-09-10T08:38:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:05.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:05.385489+0000) 2022-09-10T08:38:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:05 smithi110 ceph-mon[97633]: pgmap v7963: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:38:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:05 smithi104 ceph-mon[97044]: pgmap v7963: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:38:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:05.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:05.934639+0000) 2022-09-10T08:38:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:06.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:06.385737+0000) 2022-09-10T08:38:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:06 smithi104 ceph-mon[97044]: pgmap v7964: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:38:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:06.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:06.934782+0000) 2022-09-10T08:38:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:06 smithi110 ceph-mon[97633]: pgmap v7964: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:38:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:07.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:07.385888+0000) 2022-09-10T08:38:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:07.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:07.934942+0000) 2022-09-10T08:38:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:08.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:08.386157+0000) 2022-09-10T08:38:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:08 smithi104 ceph-mon[97044]: osdmap e578: 12 total, 12 up, 12 in 2022-09-10T08:38:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:08 smithi104 ceph-mon[97044]: pgmap v7966: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:38:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:08.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:08.935120+0000) 2022-09-10T08:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:08 smithi110 ceph-mon[97633]: osdmap e578: 12 total, 12 up, 12 in 2022-09-10T08:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:08 smithi110 ceph-mon[97633]: pgmap v7966: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:38:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:09.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:09.386402+0000) 2022-09-10T08:38:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:09.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:09.935301+0000) 2022-09-10T08:38:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:10.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:10.386615+0000) 2022-09-10T08:38:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:10.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:10.935450+0000) 2022-09-10T08:38:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:11.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:11.386859+0000) 2022-09-10T08:38:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:11 smithi104 ceph-mon[97044]: pgmap v7967: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:38:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:11 smithi104 ceph-mon[97044]: osdmap e579: 12 total, 12 up, 12 in 2022-09-10T08:38:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:11.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:11.935597+0000) 2022-09-10T08:38:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:11 smithi110 ceph-mon[97633]: pgmap v7967: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:38:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:11 smithi110 ceph-mon[97633]: osdmap e579: 12 total, 12 up, 12 in 2022-09-10T08:38:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:12.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:12.387159+0000) 2022-09-10T08:38:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:12 smithi104 ceph-mon[97044]: pgmap v7969: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:38:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:12.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:12.935729+0000) 2022-09-10T08:38:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:12 smithi110 ceph-mon[97633]: pgmap v7969: 129 pgs: 129 active+clean; 8.5 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:38:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:13.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:13.387423+0000) 2022-09-10T08:38:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:13.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:13.935870+0000) 2022-09-10T08:38:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:14.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:14.387647+0000) 2022-09-10T08:38:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:14.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:14.936065+0000) 2022-09-10T08:38:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:15.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:15.387846+0000) 2022-09-10T08:38:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:15 smithi110 ceph-mon[97633]: pgmap v7970: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:38:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:15 smithi104 ceph-mon[97044]: pgmap v7970: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:38:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:15.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:15.936192+0000) 2022-09-10T08:38:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:16.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:16.388175+0000) 2022-09-10T08:38:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:16 smithi104 ceph-mon[97044]: pgmap v7971: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:38:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:16.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:16.936336+0000) 2022-09-10T08:38:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:16 smithi110 ceph-mon[97633]: pgmap v7971: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T08:38:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:17.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:17.388408+0000) 2022-09-10T08:38:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:17.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:17.936539+0000) 2022-09-10T08:38:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:18.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:18.388650+0000) 2022-09-10T08:38:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:18.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:18.936678+0000) 2022-09-10T08:38:19.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:38:19.484+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:38:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:19.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:19.388905+0000) 2022-09-10T08:38:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:19 smithi110 ceph-mon[97633]: pgmap v7972: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:38:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:19 smithi104 ceph-mon[97044]: pgmap v7972: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:38:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:19.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:19.936810+0000) 2022-09-10T08:38:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:20.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:20.389202+0000) 2022-09-10T08:38:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:20.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:20.936947+0000) 2022-09-10T08:38:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:21.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:21.389415+0000) 2022-09-10T08:38:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:21 smithi110 ceph-mon[97633]: pgmap v7973: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:38:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:21 smithi104 ceph-mon[97044]: pgmap v7973: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:38:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:21.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:21.937090+0000) 2022-09-10T08:38:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:22.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:22.389574+0000) 2022-09-10T08:38:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:22.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:22.937246+0000) 2022-09-10T08:38:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:23.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:23.389824+0000) 2022-09-10T08:38:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:23 smithi110 ceph-mon[97633]: pgmap v7974: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:38:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:23 smithi104 ceph-mon[97044]: pgmap v7974: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:38:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:23.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:23.937387+0000) 2022-09-10T08:38:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:24.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:24.390001+0000) 2022-09-10T08:38:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:24.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:24.937523+0000) 2022-09-10T08:38:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:25.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:25.390214+0000) 2022-09-10T08:38:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:25 smithi110 ceph-mon[97633]: pgmap v7975: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:38:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:25 smithi104 ceph-mon[97044]: pgmap v7975: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:38:26.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:25.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:25.937654+0000) 2022-09-10T08:38:26.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:26.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:26.390373+0000) 2022-09-10T08:38:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:26 smithi104 ceph-mon[97044]: pgmap v7976: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:38:27.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:26.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:26.937774+0000) 2022-09-10T08:38:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:26 smithi110 ceph-mon[97633]: pgmap v7976: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:38:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:27.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:27.390550+0000) 2022-09-10T08:38:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:27.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:27.937936+0000) 2022-09-10T08:38:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:28.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:28.390735+0000) 2022-09-10T08:38:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:28.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:28.938051+0000) 2022-09-10T08:38:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:29.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:29.390978+0000) 2022-09-10T08:38:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:29 smithi110 ceph-mon[97633]: pgmap v7977: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:38:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:29 smithi104 ceph-mon[97044]: pgmap v7977: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:38:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:29.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:29.938172+0000) 2022-09-10T08:38:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:30.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:30.391225+0000) 2022-09-10T08:38:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:30.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:30.938347+0000) 2022-09-10T08:38:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:31.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:31.391477+0000) 2022-09-10T08:38:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:31 smithi110 ceph-mon[97633]: pgmap v7978: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:31 smithi104 ceph-mon[97044]: pgmap v7978: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:31.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:31.938498+0000) 2022-09-10T08:38:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:32.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:32.391739+0000) 2022-09-10T08:38:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:32.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:32.938615+0000) 2022-09-10T08:38:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:33.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:33.391986+0000) 2022-09-10T08:38:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:33 smithi110 ceph-mon[97633]: pgmap v7979: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:33 smithi104 ceph-mon[97044]: pgmap v7979: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:33.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:33.938740+0000) 2022-09-10T08:38:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:38:34.498+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:38:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:34.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:34.392271+0000) 2022-09-10T08:38:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:34.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:34.938876+0000) 2022-09-10T08:38:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:35.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:35.392444+0000) 2022-09-10T08:38:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:35 smithi110 ceph-mon[97633]: pgmap v7980: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:35 smithi104 ceph-mon[97044]: pgmap v7980: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:35.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:35.939013+0000) 2022-09-10T08:38:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:36.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:36.392629+0000) 2022-09-10T08:38:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:36 smithi104 ceph-mon[97044]: pgmap v7981: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:36.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:36.939144+0000) 2022-09-10T08:38:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:36 smithi110 ceph-mon[97633]: pgmap v7981: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:37.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:37.392874+0000) 2022-09-10T08:38:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:37.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:37.939270+0000) 2022-09-10T08:38:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:38.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:38.393054+0000) 2022-09-10T08:38:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:38.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:38.939415+0000) 2022-09-10T08:38:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:39.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:39.393184+0000) 2022-09-10T08:38:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:39 smithi110 ceph-mon[97633]: pgmap v7982: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:39 smithi104 ceph-mon[97044]: pgmap v7982: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:39.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:39.939571+0000) 2022-09-10T08:38:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:40.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:40.393346+0000) 2022-09-10T08:38:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:40.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:40.939760+0000) 2022-09-10T08:38:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:41.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:41.393525+0000) 2022-09-10T08:38:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:41 smithi110 ceph-mon[97633]: pgmap v7983: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:41 smithi104 ceph-mon[97044]: pgmap v7983: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:41.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:41.939935+0000) 2022-09-10T08:38:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:42.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:42.393758+0000) 2022-09-10T08:38:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:42.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:42.940060+0000) 2022-09-10T08:38:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:43.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:43.393983+0000) 2022-09-10T08:38:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:43 smithi110 ceph-mon[97633]: pgmap v7984: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:43 smithi104 ceph-mon[97044]: pgmap v7984: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:43.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:43.940177+0000) 2022-09-10T08:38:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:44.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:44.394211+0000) 2022-09-10T08:38:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:44.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:44.940438+0000) 2022-09-10T08:38:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:45.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:45.394451+0000) 2022-09-10T08:38:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:45 smithi110 ceph-mon[97633]: pgmap v7985: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:38:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:38:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:38:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:38:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:45 smithi104 ceph-mon[97044]: pgmap v7985: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:38:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:38:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:38:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:38:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:45.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:45.940620+0000) 2022-09-10T08:38:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:46.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:46.394701+0000) 2022-09-10T08:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:46 smithi104 ceph-mon[97044]: pgmap v7986: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:46.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:46.940756+0000) 2022-09-10T08:38:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:46 smithi110 ceph-mon[97633]: pgmap v7986: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:47.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:47.394962+0000) 2022-09-10T08:38:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:47.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:47.940931+0000) 2022-09-10T08:38:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:48.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:48.395215+0000) 2022-09-10T08:38:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:48.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:48.941072+0000) 2022-09-10T08:38:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:49.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:49.395358+0000) 2022-09-10T08:38:49.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:38:49.513+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:38:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:49 smithi110 ceph-mon[97633]: pgmap v7987: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:49 smithi104 ceph-mon[97044]: pgmap v7987: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:49.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:49.941178+0000) 2022-09-10T08:38:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:50.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:50.395545+0000) 2022-09-10T08:38:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:50.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:50.941314+0000) 2022-09-10T08:38:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:51.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:51.395766+0000) 2022-09-10T08:38:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:51 smithi110 ceph-mon[97633]: pgmap v7988: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:51 smithi104 ceph-mon[97044]: pgmap v7988: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:51.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:51.941541+0000) 2022-09-10T08:38:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:52.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:52.396022+0000) 2022-09-10T08:38:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:52.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:52.941694+0000) 2022-09-10T08:38:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:53.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:53.396218+0000) 2022-09-10T08:38:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:53 smithi110 ceph-mon[97633]: pgmap v7989: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:53 smithi104 ceph-mon[97044]: pgmap v7989: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:53.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:53.941857+0000) 2022-09-10T08:38:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:54.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:54.396466+0000) 2022-09-10T08:38:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:54.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:54.942017+0000) 2022-09-10T08:38:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:55.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:55.396737+0000) 2022-09-10T08:38:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:55 smithi110 ceph-mon[97633]: pgmap v7990: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:55 smithi104 ceph-mon[97044]: pgmap v7990: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:38:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:55.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:55.942163+0000) 2022-09-10T08:38:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:56.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:56.397024+0000) 2022-09-10T08:38:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:56 smithi104 ceph-mon[97044]: pgmap v7991: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:56.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:56.942281+0000) 2022-09-10T08:38:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:56 smithi110 ceph-mon[97633]: pgmap v7991: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:38:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:57.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:57.397264+0000) 2022-09-10T08:38:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:57.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:57.942415+0000) 2022-09-10T08:38:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:58.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:58.397509+0000) 2022-09-10T08:38:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:58.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:58.942543+0000) 2022-09-10T08:38:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:38:59.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:59.397778+0000) 2022-09-10T08:38:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:38:59 smithi110 ceph-mon[97633]: pgmap v7992: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:38:59 smithi104 ceph-mon[97044]: pgmap v7992: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:38:59.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:38:59.942674+0000) 2022-09-10T08:39:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:00.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:00.397985+0000) 2022-09-10T08:39:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:00.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:00.942797+0000) 2022-09-10T08:39:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:01.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:01.398225+0000) 2022-09-10T08:39:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:01 smithi110 ceph-mon[97633]: pgmap v7993: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:01 smithi104 ceph-mon[97044]: pgmap v7993: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:01.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:01.942988+0000) 2022-09-10T08:39:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:02.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:02.398484+0000) 2022-09-10T08:39:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:02.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:02.943240+0000) 2022-09-10T08:39:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:03.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:03.398711+0000) 2022-09-10T08:39:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:03 smithi110 ceph-mon[97633]: pgmap v7994: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:03 smithi104 ceph-mon[97044]: pgmap v7994: 129 pgs: 129 active+clean; 8.5 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:03.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:03.943396+0000) 2022-09-10T08:39:04.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:39:04.520+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:39:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:04.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:04.398965+0000) 2022-09-10T08:39:05.317 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:04.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:04.943525+0000) 2022-09-10T08:39:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:05.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:05.399162+0000) 2022-09-10T08:39:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:05 smithi110 ceph-mon[97633]: pgmap v7995: 129 pgs: 129 active+clean; 8.5 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:39:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:05 smithi104 ceph-mon[97044]: pgmap v7995: 129 pgs: 129 active+clean; 8.5 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:39:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:05.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:05.943658+0000) 2022-09-10T08:39:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:06.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:06.399370+0000) 2022-09-10T08:39:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:06 smithi104 ceph-mon[97044]: pgmap v7996: 129 pgs: 129 active+clean; 8.5 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:39:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:06.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:06.943786+0000) 2022-09-10T08:39:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:06 smithi110 ceph-mon[97633]: pgmap v7996: 129 pgs: 129 active+clean; 8.5 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:39:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:07.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:07.399584+0000) 2022-09-10T08:39:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:07.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:07.943961+0000) 2022-09-10T08:39:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:08.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:08.399791+0000) 2022-09-10T08:39:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:08 smithi104 ceph-mon[97044]: osdmap e580: 12 total, 12 up, 12 in 2022-09-10T08:39:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:08 smithi104 ceph-mon[97044]: pgmap v7998: 129 pgs: 129 active+clean; 8.5 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:39:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:08.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:08.944141+0000) 2022-09-10T08:39:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:08 smithi110 ceph-mon[97633]: osdmap e580: 12 total, 12 up, 12 in 2022-09-10T08:39:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:08 smithi110 ceph-mon[97633]: pgmap v7998: 129 pgs: 129 active+clean; 8.5 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:39:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:09.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:09.400005+0000) 2022-09-10T08:39:10.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:09.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:09.944275+0000) 2022-09-10T08:39:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:10.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:10.400237+0000) 2022-09-10T08:39:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:10.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:10.944423+0000) 2022-09-10T08:39:11.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:11.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:11.400489+0000) 2022-09-10T08:39:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:11 smithi110 ceph-mon[97633]: pgmap v7999: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:39:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:11 smithi110 ceph-mon[97633]: osdmap e581: 12 total, 12 up, 12 in 2022-09-10T08:39:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:11 smithi104 ceph-mon[97044]: pgmap v7999: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:39:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:11 smithi104 ceph-mon[97044]: osdmap e581: 12 total, 12 up, 12 in 2022-09-10T08:39:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:11.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:11.944573+0000) 2022-09-10T08:39:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:12.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:12.400683+0000) 2022-09-10T08:39:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:12.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:12.944719+0000) 2022-09-10T08:39:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:13.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:13.400870+0000) 2022-09-10T08:39:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:13 smithi110 ceph-mon[97633]: pgmap v8001: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:39:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:13 smithi104 ceph-mon[97044]: pgmap v8001: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:39:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:13.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:13.944852+0000) 2022-09-10T08:39:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:14.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:14.401092+0000) 2022-09-10T08:39:15.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:14.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:14.944983+0000) 2022-09-10T08:39:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:15.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:15.401320+0000) 2022-09-10T08:39:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:15 smithi110 ceph-mon[97633]: pgmap v8002: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:39:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:15 smithi104 ceph-mon[97044]: pgmap v8002: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:39:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:15.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:15.945121+0000) 2022-09-10T08:39:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:16.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:16.401478+0000) 2022-09-10T08:39:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:16 smithi104 ceph-mon[97044]: pgmap v8003: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:39:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:16.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:16.945245+0000) 2022-09-10T08:39:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:16 smithi110 ceph-mon[97633]: pgmap v8003: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:39:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:17.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:17.401628+0000) 2022-09-10T08:39:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:17.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:17.945384+0000) 2022-09-10T08:39:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:18.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:18.401846+0000) 2022-09-10T08:39:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:18.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:18.945571+0000) 2022-09-10T08:39:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:19.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:19.402083+0000) 2022-09-10T08:39:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:39:19.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:39:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:19 smithi110 ceph-mon[97633]: pgmap v8004: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:39:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:19 smithi104 ceph-mon[97044]: pgmap v8004: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:39:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:19.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:19.945712+0000) 2022-09-10T08:39:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:20.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:20.402263+0000) 2022-09-10T08:39:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:20.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:20.945838+0000) 2022-09-10T08:39:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:21.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:21.402457+0000) 2022-09-10T08:39:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:21 smithi110 ceph-mon[97633]: pgmap v8005: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:39:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:21 smithi104 ceph-mon[97044]: pgmap v8005: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:39:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:21.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:21.945985+0000) 2022-09-10T08:39:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:22.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:22.402709+0000) 2022-09-10T08:39:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:22.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:22.946146+0000) 2022-09-10T08:39:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:23.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:23.402860+0000) 2022-09-10T08:39:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:23 smithi110 ceph-mon[97633]: pgmap v8006: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:39:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:23 smithi104 ceph-mon[97044]: pgmap v8006: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:39:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:23.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:23.946292+0000) 2022-09-10T08:39:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:24.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:24.403143+0000) 2022-09-10T08:39:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:24.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:24.946427+0000) 2022-09-10T08:39:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:25.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:25.403358+0000) 2022-09-10T08:39:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:25 smithi110 ceph-mon[97633]: pgmap v8007: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:39:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:25 smithi104 ceph-mon[97044]: pgmap v8007: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:39:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:25.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:25.946571+0000) 2022-09-10T08:39:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:26.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:26.403603+0000) 2022-09-10T08:39:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:26 smithi104 ceph-mon[97044]: pgmap v8008: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:26.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:26.946705+0000) 2022-09-10T08:39:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:26 smithi110 ceph-mon[97633]: pgmap v8008: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:27.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:27.403854+0000) 2022-09-10T08:39:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:27.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:27.946836+0000) 2022-09-10T08:39:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:28.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:28.404047+0000) 2022-09-10T08:39:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:28.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:28.946976+0000) 2022-09-10T08:39:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:29.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:29.404232+0000) 2022-09-10T08:39:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:29 smithi110 ceph-mon[97633]: pgmap v8009: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:29 smithi104 ceph-mon[97044]: pgmap v8009: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:29.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:29.947119+0000) 2022-09-10T08:39:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:30.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:30.404458+0000) 2022-09-10T08:39:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:30.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:30.947285+0000) 2022-09-10T08:39:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:31.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:31.404605+0000) 2022-09-10T08:39:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:31 smithi110 ceph-mon[97633]: pgmap v8010: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:31 smithi104 ceph-mon[97044]: pgmap v8010: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:31.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:31.947441+0000) 2022-09-10T08:39:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:32.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:32.404835+0000) 2022-09-10T08:39:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:32.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:32.947569+0000) 2022-09-10T08:39:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:33.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:33.405159+0000) 2022-09-10T08:39:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:33 smithi110 ceph-mon[97633]: pgmap v8011: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:33 smithi104 ceph-mon[97044]: pgmap v8011: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:33.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:33.947696+0000) 2022-09-10T08:39:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:34.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:34.405400+0000) 2022-09-10T08:39:34.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:39:34.551+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:39:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:34.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:34.947829+0000) 2022-09-10T08:39:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:35.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:35.405639+0000) 2022-09-10T08:39:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:35 smithi110 ceph-mon[97633]: pgmap v8012: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:35 smithi104 ceph-mon[97044]: pgmap v8012: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:35.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:35.947960+0000) 2022-09-10T08:39:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:36.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:36.405881+0000) 2022-09-10T08:39:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:36 smithi104 ceph-mon[97044]: pgmap v8013: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:36.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:36.948086+0000) 2022-09-10T08:39:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:36 smithi110 ceph-mon[97633]: pgmap v8013: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:37.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:37.406135+0000) 2022-09-10T08:39:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:37.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:37.948231+0000) 2022-09-10T08:39:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:38.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:38.406299+0000) 2022-09-10T08:39:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:38.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:38.948379+0000) 2022-09-10T08:39:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:39.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:39.406498+0000) 2022-09-10T08:39:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:39 smithi110 ceph-mon[97633]: pgmap v8014: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:39 smithi104 ceph-mon[97044]: pgmap v8014: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:39.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:39.948556+0000) 2022-09-10T08:39:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:40.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:40.406662+0000) 2022-09-10T08:39:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:40.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:40.948682+0000) 2022-09-10T08:39:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:41.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:41.406806+0000) 2022-09-10T08:39:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:41 smithi110 ceph-mon[97633]: pgmap v8015: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:41 smithi104 ceph-mon[97044]: pgmap v8015: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:41.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:41.948826+0000) 2022-09-10T08:39:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:42.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:42.407049+0000) 2022-09-10T08:39:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:42.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:42.948967+0000) 2022-09-10T08:39:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:43.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:43.407241+0000) 2022-09-10T08:39:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:43 smithi110 ceph-mon[97633]: pgmap v8016: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:43 smithi104 ceph-mon[97044]: pgmap v8016: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:43.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:43.949113+0000) 2022-09-10T08:39:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:44.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:44.407484+0000) 2022-09-10T08:39:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:44.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:44.949310+0000) 2022-09-10T08:39:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:45.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:45.407732+0000) 2022-09-10T08:39:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:45 smithi110 ceph-mon[97633]: pgmap v8017: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:45.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:39:45.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:39:45.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:39:45.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:39:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:45 smithi104 ceph-mon[97044]: pgmap v8017: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:39:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:39:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:39:46.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:39:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:45.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:45.949438+0000) 2022-09-10T08:39:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:46.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:46.408007+0000) 2022-09-10T08:39:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:46 smithi104 ceph-mon[97044]: pgmap v8018: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:46.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:46.949563+0000) 2022-09-10T08:39:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:46 smithi110 ceph-mon[97633]: pgmap v8018: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:47.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:47.408273+0000) 2022-09-10T08:39:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:47.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:47.949707+0000) 2022-09-10T08:39:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:48.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:48.408541+0000) 2022-09-10T08:39:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:48.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:48.949857+0000) 2022-09-10T08:39:49.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:39:49.563+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:39:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:49.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:49.408728+0000) 2022-09-10T08:39:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:49 smithi110 ceph-mon[97633]: pgmap v8019: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:49 smithi104 ceph-mon[97044]: pgmap v8019: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:49.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:49.950051+0000) 2022-09-10T08:39:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:50.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:50.408954+0000) 2022-09-10T08:39:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:50.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:50.950162+0000) 2022-09-10T08:39:51.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:51.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:51.409177+0000) 2022-09-10T08:39:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:51 smithi110 ceph-mon[97633]: pgmap v8020: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:51 smithi104 ceph-mon[97044]: pgmap v8020: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:51.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:51.950298+0000) 2022-09-10T08:39:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:52.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:52.409411+0000) 2022-09-10T08:39:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:52.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:52.950425+0000) 2022-09-10T08:39:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:53.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:53.409656+0000) 2022-09-10T08:39:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:53 smithi110 ceph-mon[97633]: pgmap v8021: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:53 smithi104 ceph-mon[97044]: pgmap v8021: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:53.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:53.950561+0000) 2022-09-10T08:39:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:54.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:54.409868+0000) 2022-09-10T08:39:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:54.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:54.950698+0000) 2022-09-10T08:39:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:55.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:55.410026+0000) 2022-09-10T08:39:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:55 smithi110 ceph-mon[97633]: pgmap v8022: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:55 smithi104 ceph-mon[97044]: pgmap v8022: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:39:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:55.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:55.950880+0000) 2022-09-10T08:39:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:56.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:56.410276+0000) 2022-09-10T08:39:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:56 smithi104 ceph-mon[97044]: pgmap v8023: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:56.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:56.951037+0000) 2022-09-10T08:39:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:56 smithi110 ceph-mon[97633]: pgmap v8023: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:39:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:57.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:57.410498+0000) 2022-09-10T08:39:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:57.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:57.951170+0000) 2022-09-10T08:39:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:58.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:58.410760+0000) 2022-09-10T08:39:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:58.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:58.951317+0000) 2022-09-10T08:39:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:39:59.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:59.410983+0000) 2022-09-10T08:39:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:39:59 smithi110 ceph-mon[97633]: pgmap v8024: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:39:59 smithi104 ceph-mon[97044]: pgmap v8024: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:39:59.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:39:59.951534+0000) 2022-09-10T08:40:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:00.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:00.411197+0000) 2022-09-10T08:40:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:40:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:40:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:00.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:00.951667+0000) 2022-09-10T08:40:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:01 smithi104 ceph-mon[97044]: pgmap v8025: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:01.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:01.411399+0000) 2022-09-10T08:40:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:01 smithi110 ceph-mon[97633]: pgmap v8025: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:01.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:01.951834+0000) 2022-09-10T08:40:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:02.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:02.411642+0000) 2022-09-10T08:40:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:02.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:02.951998+0000) 2022-09-10T08:40:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:03.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:03.411876+0000) 2022-09-10T08:40:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:03 smithi110 ceph-mon[97633]: pgmap v8026: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:03 smithi104 ceph-mon[97044]: pgmap v8026: 129 pgs: 129 active+clean; 8.5 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:03.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:03.952158+0000) 2022-09-10T08:40:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:04.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:04.412131+0000) 2022-09-10T08:40:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:40:04.574+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:40:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:04.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:04.952282+0000) 2022-09-10T08:40:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:05.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:05.412334+0000) 2022-09-10T08:40:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:05 smithi110 ceph-mon[97633]: pgmap v8027: 129 pgs: 129 active+clean; 8.5 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:40:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:05 smithi104 ceph-mon[97044]: pgmap v8027: 129 pgs: 129 active+clean; 8.5 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:40:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:05.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:05.952442+0000) 2022-09-10T08:40:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:06.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:06.412534+0000) 2022-09-10T08:40:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:06 smithi104 ceph-mon[97044]: pgmap v8028: 129 pgs: 129 active+clean; 8.5 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:40:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:06.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:06.952576+0000) 2022-09-10T08:40:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:06 smithi110 ceph-mon[97633]: pgmap v8028: 129 pgs: 129 active+clean; 8.5 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:40:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:07.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:07.412767+0000) 2022-09-10T08:40:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:07.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:07.952708+0000) 2022-09-10T08:40:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:08.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:08.412976+0000) 2022-09-10T08:40:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:08 smithi104 ceph-mon[97044]: osdmap e582: 12 total, 12 up, 12 in 2022-09-10T08:40:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:08 smithi104 ceph-mon[97044]: pgmap v8030: 129 pgs: 129 active+clean; 8.5 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:40:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:08.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:08.952848+0000) 2022-09-10T08:40:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:08 smithi110 ceph-mon[97633]: osdmap e582: 12 total, 12 up, 12 in 2022-09-10T08:40:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:08 smithi110 ceph-mon[97633]: pgmap v8030: 129 pgs: 129 active+clean; 8.5 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:40:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:09.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:09.413171+0000) 2022-09-10T08:40:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:09.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:09.952974+0000) 2022-09-10T08:40:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:10.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:10.413351+0000) 2022-09-10T08:40:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:10.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:10.953126+0000) 2022-09-10T08:40:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:11.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:11.413540+0000) 2022-09-10T08:40:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:11 smithi110 ceph-mon[97633]: pgmap v8031: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:40:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:11 smithi110 ceph-mon[97633]: osdmap e583: 12 total, 12 up, 12 in 2022-09-10T08:40:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:11 smithi104 ceph-mon[97044]: pgmap v8031: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:40:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:11 smithi104 ceph-mon[97044]: osdmap e583: 12 total, 12 up, 12 in 2022-09-10T08:40:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:11.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:11.953302+0000) 2022-09-10T08:40:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:12.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:12.413758+0000) 2022-09-10T08:40:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:12.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:12.953432+0000) 2022-09-10T08:40:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:13.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:13.414007+0000) 2022-09-10T08:40:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:13 smithi110 ceph-mon[97633]: pgmap v8033: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:40:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:13 smithi104 ceph-mon[97044]: pgmap v8033: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:40:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:13.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:13.953561+0000) 2022-09-10T08:40:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:14.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:14.414224+0000) 2022-09-10T08:40:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:14.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:14.953702+0000) 2022-09-10T08:40:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:15.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:15.414416+0000) 2022-09-10T08:40:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:15 smithi110 ceph-mon[97633]: pgmap v8034: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:40:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:15 smithi104 ceph-mon[97044]: pgmap v8034: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:40:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:15.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:15.953850+0000) 2022-09-10T08:40:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:16.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:16.414655+0000) 2022-09-10T08:40:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:16 smithi104 ceph-mon[97044]: pgmap v8035: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:40:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:16.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:16.953984+0000) 2022-09-10T08:40:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:16 smithi110 ceph-mon[97633]: pgmap v8035: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:40:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:17.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:17.414947+0000) 2022-09-10T08:40:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:17.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:17.954140+0000) 2022-09-10T08:40:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:18.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:18.415191+0000) 2022-09-10T08:40:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:18.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:18.954267+0000) 2022-09-10T08:40:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:19.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:19.415394+0000) 2022-09-10T08:40:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:40:19.582+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:40:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:19 smithi110 ceph-mon[97633]: pgmap v8036: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:40:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:19 smithi104 ceph-mon[97044]: pgmap v8036: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:40:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:19.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:19.954401+0000) 2022-09-10T08:40:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:20.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:20.415592+0000) 2022-09-10T08:40:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:20.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:20.954524+0000) 2022-09-10T08:40:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:21.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:21.415877+0000) 2022-09-10T08:40:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:21 smithi110 ceph-mon[97633]: pgmap v8037: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:40:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:21 smithi104 ceph-mon[97044]: pgmap v8037: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:40:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:21.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:21.954665+0000) 2022-09-10T08:40:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:22.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:22.416131+0000) 2022-09-10T08:40:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:22.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:22.954824+0000) 2022-09-10T08:40:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:23.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:23.416315+0000) 2022-09-10T08:40:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:23 smithi110 ceph-mon[97633]: pgmap v8038: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:40:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:23 smithi104 ceph-mon[97044]: pgmap v8038: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:40:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:23.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:23.954959+0000) 2022-09-10T08:40:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:24.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:24.416590+0000) 2022-09-10T08:40:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:24.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:24.955113+0000) 2022-09-10T08:40:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:25.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:25.416823+0000) 2022-09-10T08:40:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:25 smithi110 ceph-mon[97633]: pgmap v8039: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:40:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:25 smithi104 ceph-mon[97044]: pgmap v8039: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:40:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:25.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:25.955251+0000) 2022-09-10T08:40:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:26.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:26.417082+0000) 2022-09-10T08:40:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:26 smithi104 ceph-mon[97044]: pgmap v8040: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:26.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:26.955384+0000) 2022-09-10T08:40:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:26 smithi110 ceph-mon[97633]: pgmap v8040: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:27.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:27.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:27.417338+0000) 2022-09-10T08:40:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:27.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:27.955576+0000) 2022-09-10T08:40:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:28.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:28.417565+0000) 2022-09-10T08:40:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:28.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:28.955710+0000) 2022-09-10T08:40:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:29.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:29.417789+0000) 2022-09-10T08:40:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:29 smithi110 ceph-mon[97633]: pgmap v8041: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:29 smithi104 ceph-mon[97044]: pgmap v8041: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:29.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:29.955873+0000) 2022-09-10T08:40:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:30.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:30.417977+0000) 2022-09-10T08:40:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:30.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:30.955992+0000) 2022-09-10T08:40:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:31.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:31.418224+0000) 2022-09-10T08:40:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:31 smithi110 ceph-mon[97633]: pgmap v8042: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:31 smithi104 ceph-mon[97044]: pgmap v8042: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:31.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:31.956153+0000) 2022-09-10T08:40:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:32.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:32.418466+0000) 2022-09-10T08:40:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:32.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:32.956279+0000) 2022-09-10T08:40:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:33.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:33.418699+0000) 2022-09-10T08:40:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:33 smithi110 ceph-mon[97633]: pgmap v8043: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:33 smithi104 ceph-mon[97044]: pgmap v8043: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:33.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:33.956404+0000) 2022-09-10T08:40:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:34.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:34.418927+0000) 2022-09-10T08:40:34.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:40:34.596+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:40:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:34.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:34.956538+0000) 2022-09-10T08:40:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:35.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:35.419146+0000) 2022-09-10T08:40:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:35 smithi110 ceph-mon[97633]: pgmap v8044: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:35 smithi104 ceph-mon[97044]: pgmap v8044: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:35.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:35.956676+0000) 2022-09-10T08:40:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:36.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:36.419331+0000) 2022-09-10T08:40:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:36 smithi104 ceph-mon[97044]: pgmap v8045: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:36.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:36.956874+0000) 2022-09-10T08:40:37.317 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:36 smithi110 ceph-mon[97633]: pgmap v8045: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:37.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:37.419567+0000) 2022-09-10T08:40:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:37.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:37.957018+0000) 2022-09-10T08:40:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:38.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:38.419858+0000) 2022-09-10T08:40:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:38.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:38.957234+0000) 2022-09-10T08:40:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:39.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:39.420086+0000) 2022-09-10T08:40:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:39 smithi110 ceph-mon[97633]: pgmap v8046: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:39 smithi104 ceph-mon[97044]: pgmap v8046: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:39.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:39.957382+0000) 2022-09-10T08:40:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:40.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:40.420270+0000) 2022-09-10T08:40:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:40.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:40.957534+0000) 2022-09-10T08:40:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:41.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:41.420488+0000) 2022-09-10T08:40:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:41 smithi110 ceph-mon[97633]: pgmap v8047: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:41 smithi104 ceph-mon[97044]: pgmap v8047: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:42.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:41.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:41.957684+0000) 2022-09-10T08:40:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:42.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:42.420723+0000) 2022-09-10T08:40:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:42.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:42.957820+0000) 2022-09-10T08:40:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:43.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:43.420935+0000) 2022-09-10T08:40:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:43 smithi110 ceph-mon[97633]: pgmap v8048: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:43 smithi104 ceph-mon[97044]: pgmap v8048: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:43.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:43.958020+0000) 2022-09-10T08:40:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:44.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:44.421172+0000) 2022-09-10T08:40:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:44.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:44.958166+0000) 2022-09-10T08:40:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:45.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:45.421403+0000) 2022-09-10T08:40:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:45 smithi110 ceph-mon[97633]: pgmap v8049: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:40:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:40:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:40:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:40:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:45 smithi104 ceph-mon[97044]: pgmap v8049: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:40:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:40:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:40:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:40:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:45.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:45.958295+0000) 2022-09-10T08:40:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:46.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:46.421677+0000) 2022-09-10T08:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:46 smithi104 ceph-mon[97044]: pgmap v8050: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:46.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:46.958439+0000) 2022-09-10T08:40:47.363 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:46 smithi110 ceph-mon[97633]: pgmap v8050: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:47.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:47.421858+0000) 2022-09-10T08:40:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:47.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:47.958576+0000) 2022-09-10T08:40:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:48.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:48.422038+0000) 2022-09-10T08:40:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:48.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:48.958785+0000) 2022-09-10T08:40:49.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:40:49.608+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:40:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:49.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:49.422287+0000) 2022-09-10T08:40:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:49 smithi110 ceph-mon[97633]: pgmap v8051: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:49 smithi104 ceph-mon[97044]: pgmap v8051: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:49.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:49.958928+0000) 2022-09-10T08:40:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:50.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:50.422480+0000) 2022-09-10T08:40:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:50.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:50.959074+0000) 2022-09-10T08:40:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:51.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:51.422672+0000) 2022-09-10T08:40:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:51 smithi110 ceph-mon[97633]: pgmap v8052: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:51 smithi104 ceph-mon[97044]: pgmap v8052: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:51.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:51.959245+0000) 2022-09-10T08:40:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:52.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:52.422851+0000) 2022-09-10T08:40:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:52.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:52.959379+0000) 2022-09-10T08:40:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:53.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:53.423077+0000) 2022-09-10T08:40:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:53 smithi110 ceph-mon[97633]: pgmap v8053: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:53 smithi104 ceph-mon[97044]: pgmap v8053: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:53.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:53.959540+0000) 2022-09-10T08:40:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:54.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:54.423262+0000) 2022-09-10T08:40:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:54.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:54.959667+0000) 2022-09-10T08:40:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:55.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:55.423500+0000) 2022-09-10T08:40:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:55 smithi110 ceph-mon[97633]: pgmap v8054: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:55 smithi104 ceph-mon[97044]: pgmap v8054: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:40:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:55.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:55.959791+0000) 2022-09-10T08:40:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:56.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:56.423668+0000) 2022-09-10T08:40:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:56 smithi104 ceph-mon[97044]: pgmap v8055: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:56.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:56.959917+0000) 2022-09-10T08:40:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:56 smithi110 ceph-mon[97633]: pgmap v8055: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:40:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:57.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:57.423844+0000) 2022-09-10T08:40:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:57.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:57.960062+0000) 2022-09-10T08:40:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:58.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:58.424088+0000) 2022-09-10T08:40:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:58.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:58.960172+0000) 2022-09-10T08:40:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:40:59.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:59.424296+0000) 2022-09-10T08:40:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:40:59 smithi110 ceph-mon[97633]: pgmap v8056: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:40:59 smithi104 ceph-mon[97044]: pgmap v8056: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:40:59.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:40:59.960312+0000) 2022-09-10T08:41:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:00.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:00.424483+0000) 2022-09-10T08:41:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:00.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:00.960442+0000) 2022-09-10T08:41:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:01.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:01.424671+0000) 2022-09-10T08:41:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:01 smithi110 ceph-mon[97633]: pgmap v8057: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:01 smithi104 ceph-mon[97044]: pgmap v8057: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:01.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:01.960594+0000) 2022-09-10T08:41:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:02.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:02.424880+0000) 2022-09-10T08:41:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:02.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:02.960730+0000) 2022-09-10T08:41:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:03.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:03.425081+0000) 2022-09-10T08:41:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:03 smithi110 ceph-mon[97633]: pgmap v8058: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:03 smithi104 ceph-mon[97044]: pgmap v8058: 129 pgs: 129 active+clean; 8.5 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:03.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:03.960853+0000) 2022-09-10T08:41:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:04.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:04.425313+0000) 2022-09-10T08:41:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:41:04.609+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:41:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:04.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:04.960994+0000) 2022-09-10T08:41:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:05.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:05.425559+0000) 2022-09-10T08:41:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:05 smithi110 ceph-mon[97633]: pgmap v8059: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:41:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:05 smithi104 ceph-mon[97044]: pgmap v8059: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:41:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:05.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:05.961183+0000) 2022-09-10T08:41:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:06.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:06.425821+0000) 2022-09-10T08:41:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:06 smithi104 ceph-mon[97044]: pgmap v8060: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:41:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:06.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:06.961313+0000) 2022-09-10T08:41:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:06 smithi110 ceph-mon[97633]: pgmap v8060: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:41:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:07.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:07.426027+0000) 2022-09-10T08:41:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:07.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:07.961446+0000) 2022-09-10T08:41:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:08.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:08.426262+0000) 2022-09-10T08:41:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:08 smithi104 ceph-mon[97044]: osdmap e584: 12 total, 12 up, 12 in 2022-09-10T08:41:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:08 smithi104 ceph-mon[97044]: pgmap v8062: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:41:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:08.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:08.961561+0000) 2022-09-10T08:41:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:08 smithi110 ceph-mon[97633]: osdmap e584: 12 total, 12 up, 12 in 2022-09-10T08:41:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:08 smithi110 ceph-mon[97633]: pgmap v8062: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:41:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:09.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:09.426424+0000) 2022-09-10T08:41:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:09.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:09.961687+0000) 2022-09-10T08:41:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:10.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:10.426622+0000) 2022-09-10T08:41:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:10.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:10.961811+0000) 2022-09-10T08:41:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:11.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:11.426850+0000) 2022-09-10T08:41:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:11 smithi110 ceph-mon[97633]: pgmap v8063: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:41:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:11 smithi110 ceph-mon[97633]: osdmap e585: 12 total, 12 up, 12 in 2022-09-10T08:41:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:11 smithi104 ceph-mon[97044]: pgmap v8063: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:41:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:11 smithi104 ceph-mon[97044]: osdmap e585: 12 total, 12 up, 12 in 2022-09-10T08:41:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:11.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:11.961947+0000) 2022-09-10T08:41:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:12.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:12.427080+0000) 2022-09-10T08:41:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:12.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:12.962071+0000) 2022-09-10T08:41:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:13.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:13.427354+0000) 2022-09-10T08:41:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:13 smithi110 ceph-mon[97633]: pgmap v8065: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:41:14.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:13 smithi104 ceph-mon[97044]: pgmap v8065: 129 pgs: 129 active+clean; 8.5 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:41:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:13.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:13.962165+0000) 2022-09-10T08:41:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:14.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:14.427537+0000) 2022-09-10T08:41:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:14.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:14.962295+0000) 2022-09-10T08:41:15.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:15.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:15.427753+0000) 2022-09-10T08:41:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:15 smithi110 ceph-mon[97633]: pgmap v8066: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:41:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:15 smithi104 ceph-mon[97044]: pgmap v8066: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:41:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:15.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:15.962412+0000) 2022-09-10T08:41:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:16.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:16.428020+0000) 2022-09-10T08:41:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:16 smithi104 ceph-mon[97044]: pgmap v8067: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:41:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:16.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:16.962521+0000) 2022-09-10T08:41:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:16 smithi110 ceph-mon[97633]: pgmap v8067: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:41:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:17.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:17.428262+0000) 2022-09-10T08:41:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:17.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:17.962647+0000) 2022-09-10T08:41:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:18.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:18.428478+0000) 2022-09-10T08:41:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:18.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:18.962766+0000) 2022-09-10T08:41:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:19.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:19.428622+0000) 2022-09-10T08:41:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:41:19.618+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:41:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:19 smithi110 ceph-mon[97633]: pgmap v8068: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:41:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:19 smithi104 ceph-mon[97044]: pgmap v8068: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:41:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:19.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:19.962894+0000) 2022-09-10T08:41:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:20.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:20.428878+0000) 2022-09-10T08:41:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:20.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:20.963037+0000) 2022-09-10T08:41:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:21.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:21.429150+0000) 2022-09-10T08:41:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:21 smithi110 ceph-mon[97633]: pgmap v8069: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T08:41:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:21 smithi104 ceph-mon[97044]: pgmap v8069: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T08:41:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:21.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:21.963165+0000) 2022-09-10T08:41:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:22.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:22.429396+0000) 2022-09-10T08:41:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:22.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:22.963290+0000) 2022-09-10T08:41:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:23.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:23.429589+0000) 2022-09-10T08:41:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:23 smithi110 ceph-mon[97633]: pgmap v8070: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:41:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:23 smithi104 ceph-mon[97044]: pgmap v8070: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:41:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:23.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:23.963411+0000) 2022-09-10T08:41:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:24.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:24.429790+0000) 2022-09-10T08:41:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:24.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:24.963528+0000) 2022-09-10T08:41:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:25.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:25.429983+0000) 2022-09-10T08:41:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:25 smithi110 ceph-mon[97633]: pgmap v8071: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:41:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:25 smithi104 ceph-mon[97044]: pgmap v8071: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:41:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:25.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:25.963653+0000) 2022-09-10T08:41:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:26.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:26.430265+0000) 2022-09-10T08:41:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:26 smithi104 ceph-mon[97044]: pgmap v8072: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T08:41:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:26.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:26.963781+0000) 2022-09-10T08:41:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:26 smithi110 ceph-mon[97633]: pgmap v8072: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T08:41:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:27.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:27.430434+0000) 2022-09-10T08:41:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:27.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:27.963929+0000) 2022-09-10T08:41:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:28.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:28.430695+0000) 2022-09-10T08:41:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:28.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:28.964055+0000) 2022-09-10T08:41:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:29.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:29.430918+0000) 2022-09-10T08:41:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:29 smithi110 ceph-mon[97633]: pgmap v8073: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T08:41:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:29 smithi104 ceph-mon[97044]: pgmap v8073: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T08:41:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:29.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:29.964175+0000) 2022-09-10T08:41:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:30.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:30.431173+0000) 2022-09-10T08:41:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:30.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:30.964300+0000) 2022-09-10T08:41:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:31.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:31.431417+0000) 2022-09-10T08:41:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:31 smithi110 ceph-mon[97633]: pgmap v8074: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T08:41:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:31 smithi104 ceph-mon[97044]: pgmap v8074: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T08:41:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:31.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:31.964437+0000) 2022-09-10T08:41:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:32.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:32.431576+0000) 2022-09-10T08:41:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:32.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:32.964627+0000) 2022-09-10T08:41:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:33.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:33.431755+0000) 2022-09-10T08:41:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:33 smithi110 ceph-mon[97633]: pgmap v8075: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:33 smithi104 ceph-mon[97044]: pgmap v8075: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:33.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:33.964840+0000) 2022-09-10T08:41:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:41:34.633+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:41:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:34.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:34.431922+0000) 2022-09-10T08:41:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:34.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:34.965125+0000) 2022-09-10T08:41:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:35.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:35.432085+0000) 2022-09-10T08:41:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:35 smithi110 ceph-mon[97633]: pgmap v8076: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:35 smithi104 ceph-mon[97044]: pgmap v8076: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:35.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:35.965261+0000) 2022-09-10T08:41:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:36.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:36.432262+0000) 2022-09-10T08:41:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:36 smithi104 ceph-mon[97044]: pgmap v8077: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:36.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:36.965391+0000) 2022-09-10T08:41:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:36 smithi110 ceph-mon[97633]: pgmap v8077: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:37.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:37.432476+0000) 2022-09-10T08:41:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:37.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:37.965521+0000) 2022-09-10T08:41:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:38.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:38.432717+0000) 2022-09-10T08:41:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:38.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:38.965646+0000) 2022-09-10T08:41:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:39.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:39.432900+0000) 2022-09-10T08:41:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:39 smithi110 ceph-mon[97633]: pgmap v8078: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:39 smithi104 ceph-mon[97044]: pgmap v8078: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:39.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:39.965803+0000) 2022-09-10T08:41:40.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:40.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:40.433150+0000) 2022-09-10T08:41:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:40.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:40.965924+0000) 2022-09-10T08:41:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:41.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:41.433418+0000) 2022-09-10T08:41:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:41 smithi110 ceph-mon[97633]: pgmap v8079: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:41 smithi104 ceph-mon[97044]: pgmap v8079: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:41.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:41.966064+0000) 2022-09-10T08:41:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:42.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:42.433594+0000) 2022-09-10T08:41:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:42.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:42.966170+0000) 2022-09-10T08:41:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:43.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:43.433825+0000) 2022-09-10T08:41:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:43 smithi110 ceph-mon[97633]: pgmap v8080: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:43 smithi104 ceph-mon[97044]: pgmap v8080: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:43.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:43.966308+0000) 2022-09-10T08:41:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:44.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:44.434015+0000) 2022-09-10T08:41:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:41:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:41:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:41:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:41:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:41:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:41:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:41:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:41:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:44.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:44.966457+0000) 2022-09-10T08:41:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:45.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:45.434199+0000) 2022-09-10T08:41:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:45 smithi104 ceph-mon[97044]: pgmap v8081: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:45 smithi110 ceph-mon[97633]: pgmap v8081: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:45.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:45.966599+0000) 2022-09-10T08:41:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:46.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:46.434418+0000) 2022-09-10T08:41:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:47 smithi104 ceph-mon[97044]: pgmap v8082: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:46.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:46.966750+0000) 2022-09-10T08:41:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:46 smithi110 ceph-mon[97633]: pgmap v8082: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:47.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:47.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:47.434619+0000) 2022-09-10T08:41:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:47.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:47.966894+0000) 2022-09-10T08:41:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:48.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:48.434881+0000) 2022-09-10T08:41:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:48.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:48.967026+0000) 2022-09-10T08:41:49.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:41:49.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:41:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:49.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:49.435159+0000) 2022-09-10T08:41:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:49 smithi110 ceph-mon[97633]: pgmap v8083: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:49 smithi104 ceph-mon[97044]: pgmap v8083: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:49.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:49.967161+0000) 2022-09-10T08:41:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:50.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:50.435331+0000) 2022-09-10T08:41:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:50.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:50.967290+0000) 2022-09-10T08:41:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:51.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:51.435454+0000) 2022-09-10T08:41:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:51 smithi110 ceph-mon[97633]: pgmap v8084: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:51 smithi104 ceph-mon[97044]: pgmap v8084: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:51.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:51.967444+0000) 2022-09-10T08:41:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:52.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:52.435680+0000) 2022-09-10T08:41:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:52.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:52.967586+0000) 2022-09-10T08:41:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:53.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:53.435927+0000) 2022-09-10T08:41:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:53 smithi104 ceph-mon[97044]: pgmap v8085: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:53 smithi110 ceph-mon[97633]: pgmap v8085: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:53.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:53.967710+0000) 2022-09-10T08:41:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:54.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:54.436207+0000) 2022-09-10T08:41:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:54.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:54.967834+0000) 2022-09-10T08:41:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:55.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:55.436381+0000) 2022-09-10T08:41:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:55 smithi104 ceph-mon[97044]: pgmap v8086: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:55 smithi110 ceph-mon[97633]: pgmap v8086: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:41:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:55.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:55.968004+0000) 2022-09-10T08:41:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:56.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:56.436592+0000) 2022-09-10T08:41:57.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:56.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:56.968159+0000) 2022-09-10T08:41:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:56 smithi104 ceph-mon[97044]: pgmap v8087: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:56 smithi110 ceph-mon[97633]: pgmap v8087: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:41:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:57.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:57.436839+0000) 2022-09-10T08:41:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:57.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:57.968296+0000) 2022-09-10T08:41:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:58.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:58.437067+0000) 2022-09-10T08:41:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:58.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:58.968431+0000) 2022-09-10T08:41:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:41:59.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:59.437310+0000) 2022-09-10T08:41:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:41:59 smithi110 ceph-mon[97633]: pgmap v8088: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:41:59 smithi104 ceph-mon[97044]: pgmap v8088: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:41:59.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:41:59.968625+0000) 2022-09-10T08:42:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:00.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:00.437500+0000) 2022-09-10T08:42:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:00.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:00.968788+0000) 2022-09-10T08:42:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:01.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:01.437670+0000) 2022-09-10T08:42:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:01 smithi104 ceph-mon[97044]: pgmap v8089: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:01 smithi110 ceph-mon[97633]: pgmap v8089: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:01.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:01.968978+0000) 2022-09-10T08:42:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:02.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:02.437933+0000) 2022-09-10T08:42:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:02.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:02.969136+0000) 2022-09-10T08:42:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:03.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:03.438071+0000) 2022-09-10T08:42:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:03 smithi104 ceph-mon[97044]: pgmap v8090: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:03 smithi110 ceph-mon[97633]: pgmap v8090: 129 pgs: 129 active+clean; 8.5 MiB data, 811 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:03.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:03.969264+0000) 2022-09-10T08:42:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:04.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:04.438235+0000) 2022-09-10T08:42:04.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:42:04.659+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:42:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:04.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:04.969397+0000) 2022-09-10T08:42:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:05.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:05.438445+0000) 2022-09-10T08:42:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:05 smithi104 ceph-mon[97044]: pgmap v8091: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:42:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:05 smithi110 ceph-mon[97633]: pgmap v8091: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:42:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:05.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:05.969531+0000) 2022-09-10T08:42:06.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:06.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:06.438665+0000) 2022-09-10T08:42:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:07 smithi104 ceph-mon[97044]: pgmap v8092: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:42:07.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:06.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:06.969664+0000) 2022-09-10T08:42:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:07 smithi110 ceph-mon[97633]: pgmap v8092: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:42:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:07.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:07.438883+0000) 2022-09-10T08:42:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:07.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:07.969798+0000) 2022-09-10T08:42:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:08.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:08.439194+0000) 2022-09-10T08:42:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:09 smithi104 ceph-mon[97044]: osdmap e586: 12 total, 12 up, 12 in 2022-09-10T08:42:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:09 smithi104 ceph-mon[97044]: pgmap v8094: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:42:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:08.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:08.969929+0000) 2022-09-10T08:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:09 smithi110 ceph-mon[97633]: osdmap e586: 12 total, 12 up, 12 in 2022-09-10T08:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:09 smithi110 ceph-mon[97633]: pgmap v8094: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:42:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:09.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:09.439417+0000) 2022-09-10T08:42:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:09.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:09.970058+0000) 2022-09-10T08:42:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:10.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:10.439644+0000) 2022-09-10T08:42:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:10.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:10.970214+0000) 2022-09-10T08:42:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:11.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:11.439861+0000) 2022-09-10T08:42:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:11 smithi104 ceph-mon[97044]: pgmap v8095: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:42:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:11 smithi104 ceph-mon[97044]: osdmap e587: 12 total, 12 up, 12 in 2022-09-10T08:42:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:11 smithi110 ceph-mon[97633]: pgmap v8095: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:42:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:11 smithi110 ceph-mon[97633]: osdmap e587: 12 total, 12 up, 12 in 2022-09-10T08:42:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:11.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:11.970366+0000) 2022-09-10T08:42:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:12.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:12.440050+0000) 2022-09-10T08:42:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:12.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:12.970485+0000) 2022-09-10T08:42:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:13.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:13.440162+0000) 2022-09-10T08:42:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:13 smithi104 ceph-mon[97044]: pgmap v8097: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:42:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:13 smithi110 ceph-mon[97633]: pgmap v8097: 129 pgs: 129 active+clean; 8.6 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:42:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:13.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:13.970615+0000) 2022-09-10T08:42:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:14.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:14.440373+0000) 2022-09-10T08:42:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:14.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:14.970748+0000) 2022-09-10T08:42:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:15.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:15.440591+0000) 2022-09-10T08:42:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:15 smithi104 ceph-mon[97044]: pgmap v8098: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:42:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:15 smithi110 ceph-mon[97633]: pgmap v8098: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:42:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:15.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:15.970867+0000) 2022-09-10T08:42:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:16.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:16.440874+0000) 2022-09-10T08:42:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:17 smithi104 ceph-mon[97044]: pgmap v8099: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:42:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:16.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:16.971002+0000) 2022-09-10T08:42:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:17 smithi110 ceph-mon[97633]: pgmap v8099: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T08:42:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:17.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:17.441077+0000) 2022-09-10T08:42:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:17.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:17.971245+0000) 2022-09-10T08:42:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:18.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:18.441325+0000) 2022-09-10T08:42:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:18.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:18.971383+0000) 2022-09-10T08:42:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:19.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:19.441496+0000) 2022-09-10T08:42:19.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:42:19.674+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:42:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:19 smithi104 ceph-mon[97044]: pgmap v8100: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:42:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:19 smithi110 ceph-mon[97633]: pgmap v8100: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:42:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:19.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:19.971530+0000) 2022-09-10T08:42:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:20.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:20.441681+0000) 2022-09-10T08:42:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:20.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:20.971667+0000) 2022-09-10T08:42:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:21.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:21.441922+0000) 2022-09-10T08:42:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:21 smithi104 ceph-mon[97044]: pgmap v8101: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T08:42:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:21 smithi110 ceph-mon[97633]: pgmap v8101: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T08:42:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:21.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:21.971820+0000) 2022-09-10T08:42:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:22.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:22.442152+0000) 2022-09-10T08:42:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:22.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:22.971954+0000) 2022-09-10T08:42:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:23.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:23.442300+0000) 2022-09-10T08:42:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:23 smithi104 ceph-mon[97044]: pgmap v8102: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:42:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:23 smithi110 ceph-mon[97633]: pgmap v8102: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:42:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:23.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:23.972146+0000) 2022-09-10T08:42:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:24.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:24.442557+0000) 2022-09-10T08:42:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:24.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:24.972274+0000) 2022-09-10T08:42:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:25.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:25.442796+0000) 2022-09-10T08:42:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:25 smithi104 ceph-mon[97044]: pgmap v8103: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:42:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:25 smithi110 ceph-mon[97633]: pgmap v8103: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:42:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:25.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:25.972406+0000) 2022-09-10T08:42:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:26.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:26.443026+0000) 2022-09-10T08:42:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:27 smithi104 ceph-mon[97044]: pgmap v8104: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:42:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:26.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:26.972532+0000) 2022-09-10T08:42:27.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:27 smithi110 ceph-mon[97633]: pgmap v8104: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:42:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:27.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:27.443299+0000) 2022-09-10T08:42:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:27.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:27.972655+0000) 2022-09-10T08:42:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:28.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:28.443490+0000) 2022-09-10T08:42:29.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:28.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:28.972775+0000) 2022-09-10T08:42:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:29.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:29.443692+0000) 2022-09-10T08:42:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:29 smithi104 ceph-mon[97044]: pgmap v8105: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:42:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:29 smithi110 ceph-mon[97633]: pgmap v8105: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:42:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:29.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:29.972901+0000) 2022-09-10T08:42:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:30.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:30.443805+0000) 2022-09-10T08:42:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:30.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:30.973015+0000) 2022-09-10T08:42:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:31.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:31.444018+0000) 2022-09-10T08:42:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:31 smithi104 ceph-mon[97044]: pgmap v8106: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:31 smithi110 ceph-mon[97633]: pgmap v8106: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:31.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:31.973191+0000) 2022-09-10T08:42:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:32.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:32.444265+0000) 2022-09-10T08:42:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:32.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:32.973373+0000) 2022-09-10T08:42:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:33.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:33.444482+0000) 2022-09-10T08:42:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:33 smithi104 ceph-mon[97044]: pgmap v8107: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:33 smithi110 ceph-mon[97633]: pgmap v8107: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:33.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:33.973549+0000) 2022-09-10T08:42:34.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:42:34.681+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:42:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:34.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:34.444711+0000) 2022-09-10T08:42:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:34.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:34.973682+0000) 2022-09-10T08:42:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:35.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:35.444874+0000) 2022-09-10T08:42:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:35 smithi104 ceph-mon[97044]: pgmap v8108: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:35 smithi110 ceph-mon[97633]: pgmap v8108: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:35.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:35.973840+0000) 2022-09-10T08:42:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:36.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:36.445191+0000) 2022-09-10T08:42:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:37 smithi104 ceph-mon[97044]: pgmap v8109: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:36.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:36.973965+0000) 2022-09-10T08:42:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:37 smithi110 ceph-mon[97633]: pgmap v8109: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:37.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:37.445485+0000) 2022-09-10T08:42:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:37.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:37.974092+0000) 2022-09-10T08:42:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:38.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:38.445750+0000) 2022-09-10T08:42:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:38.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:38.974242+0000) 2022-09-10T08:42:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:39.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:39.445973+0000) 2022-09-10T08:42:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:39 smithi104 ceph-mon[97044]: pgmap v8110: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:39 smithi110 ceph-mon[97633]: pgmap v8110: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:39.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:39.974379+0000) 2022-09-10T08:42:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:40.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:40.446196+0000) 2022-09-10T08:42:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:40.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:40.974523+0000) 2022-09-10T08:42:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:41.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:41.446399+0000) 2022-09-10T08:42:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:41 smithi104 ceph-mon[97044]: pgmap v8111: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:41 smithi110 ceph-mon[97633]: pgmap v8111: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:41.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:41.974656+0000) 2022-09-10T08:42:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:42.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:42.446592+0000) 2022-09-10T08:42:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:42.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:42.974783+0000) 2022-09-10T08:42:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:43.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:43.446803+0000) 2022-09-10T08:42:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:43 smithi104 ceph-mon[97044]: pgmap v8112: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:43 smithi110 ceph-mon[97633]: pgmap v8112: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:43.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:43.974972+0000) 2022-09-10T08:42:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:44.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:44.447032+0000) 2022-09-10T08:42:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:42:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:42:45.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:42:45.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:42:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:42:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:42:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:42:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:42:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:44.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:44.975183+0000) 2022-09-10T08:42:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:45.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:45.447157+0000) 2022-09-10T08:42:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:45 smithi104 ceph-mon[97044]: pgmap v8113: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:45 smithi110 ceph-mon[97633]: pgmap v8113: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:45.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:45.975361+0000) 2022-09-10T08:42:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:46.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:46.447370+0000) 2022-09-10T08:42:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:47 smithi104 ceph-mon[97044]: pgmap v8114: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:46.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:46.975544+0000) 2022-09-10T08:42:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:47 smithi110 ceph-mon[97633]: pgmap v8114: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:47.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:47.447583+0000) 2022-09-10T08:42:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:47.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:47.975682+0000) 2022-09-10T08:42:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:48.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:48.447845+0000) 2022-09-10T08:42:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:48.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:48.975820+0000) 2022-09-10T08:42:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:49.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:49.448110+0000) 2022-09-10T08:42:49.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:42:49.692+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:42:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:49 smithi104 ceph-mon[97044]: pgmap v8115: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:49 smithi110 ceph-mon[97633]: pgmap v8115: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:49.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:49.975957+0000) 2022-09-10T08:42:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:50.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:50.448363+0000) 2022-09-10T08:42:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:50.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:50.976090+0000) 2022-09-10T08:42:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:51.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:51.448514+0000) 2022-09-10T08:42:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:51 smithi104 ceph-mon[97044]: pgmap v8116: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:51 smithi110 ceph-mon[97633]: pgmap v8116: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:51.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:51.976259+0000) 2022-09-10T08:42:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:52.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:52.448795+0000) 2022-09-10T08:42:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:52.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:52.976381+0000) 2022-09-10T08:42:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:53.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:53.449072+0000) 2022-09-10T08:42:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:53 smithi104 ceph-mon[97044]: pgmap v8117: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:53 smithi110 ceph-mon[97633]: pgmap v8117: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:53.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:53.976514+0000) 2022-09-10T08:42:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:54.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:54.449377+0000) 2022-09-10T08:42:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:54.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:54.976645+0000) 2022-09-10T08:42:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:55.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:55.449576+0000) 2022-09-10T08:42:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:55 smithi104 ceph-mon[97044]: pgmap v8118: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:55 smithi110 ceph-mon[97633]: pgmap v8118: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:42:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:55.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:55.976781+0000) 2022-09-10T08:42:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:56.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:56.449839+0000) 2022-09-10T08:42:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:57 smithi104 ceph-mon[97044]: pgmap v8119: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:56.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:56.976912+0000) 2022-09-10T08:42:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:57 smithi110 ceph-mon[97633]: pgmap v8119: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:42:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:57.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:57.450019+0000) 2022-09-10T08:42:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:57.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:57.977042+0000) 2022-09-10T08:42:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:58.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:58.450274+0000) 2022-09-10T08:42:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:58.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:58.977158+0000) 2022-09-10T08:42:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:42:59.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:59.450511+0000) 2022-09-10T08:43:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:42:59 smithi104 ceph-mon[97044]: pgmap v8120: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:42:59 smithi110 ceph-mon[97633]: pgmap v8120: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:42:59.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:42:59.977326+0000) 2022-09-10T08:43:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:00.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:00.450685+0000) 2022-09-10T08:43:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:00.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:00.977477+0000) 2022-09-10T08:43:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:01.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:01.450938+0000) 2022-09-10T08:43:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:01 smithi104 ceph-mon[97044]: pgmap v8121: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:01 smithi110 ceph-mon[97633]: pgmap v8121: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:01.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:01.977615+0000) 2022-09-10T08:43:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:02.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:02.451208+0000) 2022-09-10T08:43:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:02.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:02.977740+0000) 2022-09-10T08:43:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:03.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:03.451441+0000) 2022-09-10T08:43:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:03 smithi104 ceph-mon[97044]: pgmap v8122: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:03 smithi110 ceph-mon[97633]: pgmap v8122: 129 pgs: 129 active+clean; 8.6 MiB data, 813 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:03.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:03.977944+0000) 2022-09-10T08:43:04.709 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:04.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:04.451659+0000) 2022-09-10T08:43:04.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:43:04.708+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:43:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:04.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:04.978128+0000) 2022-09-10T08:43:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:05.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:05.451896+0000) 2022-09-10T08:43:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:05 smithi104 ceph-mon[97044]: pgmap v8123: 129 pgs: 129 active+clean; 8.6 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:43:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:05 smithi110 ceph-mon[97633]: pgmap v8123: 129 pgs: 129 active+clean; 8.6 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:43:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:05.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:05.978265+0000) 2022-09-10T08:43:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:06.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:06.452166+0000) 2022-09-10T08:43:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:07 smithi104 ceph-mon[97044]: pgmap v8124: 129 pgs: 129 active+clean; 8.6 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:43:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:06.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:06.978401+0000) 2022-09-10T08:43:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:07 smithi110 ceph-mon[97633]: pgmap v8124: 129 pgs: 129 active+clean; 8.6 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:43:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:07.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:07.452329+0000) 2022-09-10T08:43:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:07.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:07.978598+0000) 2022-09-10T08:43:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:08.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:08.452581+0000) 2022-09-10T08:43:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:09 smithi104 ceph-mon[97044]: osdmap e588: 12 total, 12 up, 12 in 2022-09-10T08:43:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:09 smithi104 ceph-mon[97044]: pgmap v8126: 129 pgs: 129 active+clean; 8.6 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:43:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:08.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:08.978755+0000) 2022-09-10T08:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:09 smithi110 ceph-mon[97633]: osdmap e588: 12 total, 12 up, 12 in 2022-09-10T08:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:09 smithi110 ceph-mon[97633]: pgmap v8126: 129 pgs: 129 active+clean; 8.6 MiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:43:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:09.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:09.452785+0000) 2022-09-10T08:43:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:09.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:09.978897+0000) 2022-09-10T08:43:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:10.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:10.453041+0000) 2022-09-10T08:43:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:10.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:10.979023+0000) 2022-09-10T08:43:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:11.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:11.453186+0000) 2022-09-10T08:43:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:11 smithi104 ceph-mon[97044]: pgmap v8127: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:43:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:11 smithi104 ceph-mon[97044]: osdmap e589: 12 total, 12 up, 12 in 2022-09-10T08:43:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:11 smithi110 ceph-mon[97633]: pgmap v8127: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:43:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:11 smithi110 ceph-mon[97633]: osdmap e589: 12 total, 12 up, 12 in 2022-09-10T08:43:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:11.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:11.979185+0000) 2022-09-10T08:43:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:12.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:12.453406+0000) 2022-09-10T08:43:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:12.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:12.979328+0000) 2022-09-10T08:43:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:13.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:13.453609+0000) 2022-09-10T08:43:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:13 smithi104 ceph-mon[97044]: pgmap v8129: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:43:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:13 smithi110 ceph-mon[97633]: pgmap v8129: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:43:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:13.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:13.979464+0000) 2022-09-10T08:43:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:14.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:14.453865+0000) 2022-09-10T08:43:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:14.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:14.979703+0000) 2022-09-10T08:43:15.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:15.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:15.454033+0000) 2022-09-10T08:43:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:15 smithi104 ceph-mon[97044]: pgmap v8130: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:43:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:15 smithi110 ceph-mon[97633]: pgmap v8130: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:43:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:15.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:15.979886+0000) 2022-09-10T08:43:16.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:16.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:16.454336+0000) 2022-09-10T08:43:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:17 smithi104 ceph-mon[97044]: pgmap v8131: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:43:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:16.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:16.980027+0000) 2022-09-10T08:43:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:17 smithi110 ceph-mon[97633]: pgmap v8131: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:43:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:17.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:17.454590+0000) 2022-09-10T08:43:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:17.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:17.980207+0000) 2022-09-10T08:43:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:18.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:18.454821+0000) 2022-09-10T08:43:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:18.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:18.980339+0000) 2022-09-10T08:43:19.725 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:19.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:19.455051+0000) 2022-09-10T08:43:19.981 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:43:19.723+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:43:20.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:19 smithi104 ceph-mon[97044]: pgmap v8132: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:43:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:19 smithi110 ceph-mon[97633]: pgmap v8132: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:43:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:19.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:19.980466+0000) 2022-09-10T08:43:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:20.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:20.455271+0000) 2022-09-10T08:43:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:20.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:20.980599+0000) 2022-09-10T08:43:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:21.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:21.455517+0000) 2022-09-10T08:43:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:21 smithi104 ceph-mon[97044]: pgmap v8133: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:43:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:21 smithi110 ceph-mon[97633]: pgmap v8133: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:43:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:21.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:21.980732+0000) 2022-09-10T08:43:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:22.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:22.455744+0000) 2022-09-10T08:43:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:22.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:22.980881+0000) 2022-09-10T08:43:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:23.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:23.455864+0000) 2022-09-10T08:43:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:23 smithi104 ceph-mon[97044]: pgmap v8134: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:43:24.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:23 smithi110 ceph-mon[97633]: pgmap v8134: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:43:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:23.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:23.981025+0000) 2022-09-10T08:43:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:24.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:24.456139+0000) 2022-09-10T08:43:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:24.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:24.981164+0000) 2022-09-10T08:43:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:25.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:25.456302+0000) 2022-09-10T08:43:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:25 smithi104 ceph-mon[97044]: pgmap v8135: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:43:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:25 smithi110 ceph-mon[97633]: pgmap v8135: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:43:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:25.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:25.981301+0000) 2022-09-10T08:43:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:26.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:26.456543+0000) 2022-09-10T08:43:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:27 smithi104 ceph-mon[97044]: pgmap v8136: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:26.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:26.981477+0000) 2022-09-10T08:43:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:27 smithi110 ceph-mon[97633]: pgmap v8136: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:27.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:27.456782+0000) 2022-09-10T08:43:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:27.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:27.981667+0000) 2022-09-10T08:43:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:28.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:28.457051+0000) 2022-09-10T08:43:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:28.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:28.981858+0000) 2022-09-10T08:43:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:29.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:29.457238+0000) 2022-09-10T08:43:30.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:29 smithi104 ceph-mon[97044]: pgmap v8137: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:29 smithi110 ceph-mon[97633]: pgmap v8137: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:29.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:29.982004+0000) 2022-09-10T08:43:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:30.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:30.457395+0000) 2022-09-10T08:43:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:30.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:30.982133+0000) 2022-09-10T08:43:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:31.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:31.457538+0000) 2022-09-10T08:43:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:31 smithi104 ceph-mon[97044]: pgmap v8138: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:31 smithi110 ceph-mon[97633]: pgmap v8138: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:31.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:31.982263+0000) 2022-09-10T08:43:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:32.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:32.457726+0000) 2022-09-10T08:43:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:32.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:32.982396+0000) 2022-09-10T08:43:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:33.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:33.457957+0000) 2022-09-10T08:43:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:33 smithi104 ceph-mon[97044]: pgmap v8139: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:33 smithi110 ceph-mon[97633]: pgmap v8139: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:33.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:33.982531+0000) 2022-09-10T08:43:34.740 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:34.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:34.458209+0000) 2022-09-10T08:43:34.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:43:34.738+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:43:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:34.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:34.982751+0000) 2022-09-10T08:43:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:35.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:35.458393+0000) 2022-09-10T08:43:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:35 smithi104 ceph-mon[97044]: pgmap v8140: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:35 smithi110 ceph-mon[97633]: pgmap v8140: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:35.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:35.982950+0000) 2022-09-10T08:43:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:36.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:36.458633+0000) 2022-09-10T08:43:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:37 smithi104 ceph-mon[97044]: pgmap v8141: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:36.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:36.983138+0000) 2022-09-10T08:43:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:37 smithi110 ceph-mon[97633]: pgmap v8141: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:37.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:37.458813+0000) 2022-09-10T08:43:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:37.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:37.983263+0000) 2022-09-10T08:43:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:38.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:38.459007+0000) 2022-09-10T08:43:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:38.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:38.983404+0000) 2022-09-10T08:43:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:39.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:39.459172+0000) 2022-09-10T08:43:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:39 smithi104 ceph-mon[97044]: pgmap v8142: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:39 smithi110 ceph-mon[97633]: pgmap v8142: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:39.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:39.983542+0000) 2022-09-10T08:43:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:40.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:40.459413+0000) 2022-09-10T08:43:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:40.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:40.983684+0000) 2022-09-10T08:43:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:41.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:41.459633+0000) 2022-09-10T08:43:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:41 smithi104 ceph-mon[97044]: pgmap v8143: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:41 smithi110 ceph-mon[97633]: pgmap v8143: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:41.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:41.983888+0000) 2022-09-10T08:43:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:42.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:42.459847+0000) 2022-09-10T08:43:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:42.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:42.984084+0000) 2022-09-10T08:43:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:43.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:43.460071+0000) 2022-09-10T08:43:44.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:43 smithi104 ceph-mon[97044]: pgmap v8144: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:43 smithi110 ceph-mon[97633]: pgmap v8144: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:43.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:43.984232+0000) 2022-09-10T08:43:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:44.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:44.460284+0000) 2022-09-10T08:43:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:43:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:43:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:43:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:43:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:43:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:43:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:43:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:43:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:44.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:44.984373+0000) 2022-09-10T08:43:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:45.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:45.460539+0000) 2022-09-10T08:43:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:45 smithi104 ceph-mon[97044]: pgmap v8145: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:45 smithi110 ceph-mon[97633]: pgmap v8145: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:45.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:45.984518+0000) 2022-09-10T08:43:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:46.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:46.460773+0000) 2022-09-10T08:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:47 smithi104 ceph-mon[97044]: pgmap v8146: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:43:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:46.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:46.984656+0000) 2022-09-10T08:43:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:47 smithi110 ceph-mon[97633]: pgmap v8146: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:43:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:47.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:47.460998+0000) 2022-09-10T08:43:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:47.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:47.984786+0000) 2022-09-10T08:43:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:48.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:48.461254+0000) 2022-09-10T08:43:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:48.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:48.984912+0000) 2022-09-10T08:43:49.747 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:49.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:49.461542+0000) 2022-09-10T08:43:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:49 smithi104 ceph-mon[97044]: pgmap v8147: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:43:50.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:43:49.745+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:43:50.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:49.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:49.985046+0000) 2022-09-10T08:43:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:49 smithi110 ceph-mon[97633]: pgmap v8147: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:43:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:50.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:50.461735+0000) 2022-09-10T08:43:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:50.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:50.985188+0000) 2022-09-10T08:43:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:51.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:51.461913+0000) 2022-09-10T08:43:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:51 smithi104 ceph-mon[97044]: pgmap v8148: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:51 smithi110 ceph-mon[97633]: pgmap v8148: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:51.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:51.985339+0000) 2022-09-10T08:43:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:52.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:52.462169+0000) 2022-09-10T08:43:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:52.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:52.985481+0000) 2022-09-10T08:43:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:53.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:53.462379+0000) 2022-09-10T08:43:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:53 smithi104 ceph-mon[97044]: pgmap v8149: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:43:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:53 smithi110 ceph-mon[97633]: pgmap v8149: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:43:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:53.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:53.985649+0000) 2022-09-10T08:43:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:54.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:54.462610+0000) 2022-09-10T08:43:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:54.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:54.985773+0000) 2022-09-10T08:43:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:55.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:55.462827+0000) 2022-09-10T08:43:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:55 smithi104 ceph-mon[97044]: pgmap v8150: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:55 smithi110 ceph-mon[97633]: pgmap v8150: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:43:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:55.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:55.985902+0000) 2022-09-10T08:43:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:56.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:56.463047+0000) 2022-09-10T08:43:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:57 smithi104 ceph-mon[97044]: pgmap v8151: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:56.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:56.986023+0000) 2022-09-10T08:43:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:57 smithi110 ceph-mon[97633]: pgmap v8151: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:43:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:57.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:57.463157+0000) 2022-09-10T08:43:58.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:57.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:57.986178+0000) 2022-09-10T08:43:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:58.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:58.463335+0000) 2022-09-10T08:43:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:58.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:58.986336+0000) 2022-09-10T08:43:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:43:59.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:59.463531+0000) 2022-09-10T08:44:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:43:59 smithi104 ceph-mon[97044]: pgmap v8152: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:43:59 smithi110 ceph-mon[97633]: pgmap v8152: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:43:59.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:43:59.986468+0000) 2022-09-10T08:44:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:00.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:00.463692+0000) 2022-09-10T08:44:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:00.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:00.986585+0000) 2022-09-10T08:44:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:01.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:01.463915+0000) 2022-09-10T08:44:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:01 smithi104 ceph-mon[97044]: pgmap v8153: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:01 smithi110 ceph-mon[97633]: pgmap v8153: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:01.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:01.986768+0000) 2022-09-10T08:44:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:02.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:02.464206+0000) 2022-09-10T08:44:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:02.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:02.986922+0000) 2022-09-10T08:44:03.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:03.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:03.464383+0000) 2022-09-10T08:44:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:03 smithi104 ceph-mon[97044]: pgmap v8154: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:03 smithi110 ceph-mon[97633]: pgmap v8154: 129 pgs: 129 active+clean; 8.6 MiB data, 815 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:03.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:03.987066+0000) 2022-09-10T08:44:04.749 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:04.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:04.464596+0000) 2022-09-10T08:44:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:04.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:04.987191+0000) 2022-09-10T08:44:05.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:44:04.747+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:44:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:05.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:05.464738+0000) 2022-09-10T08:44:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:05 smithi104 ceph-mon[97044]: pgmap v8155: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:44:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:05 smithi110 ceph-mon[97633]: pgmap v8155: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:44:06.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:05.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:05.987338+0000) 2022-09-10T08:44:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:06.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:06.464954+0000) 2022-09-10T08:44:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:07 smithi104 ceph-mon[97044]: pgmap v8156: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:44:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:06.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:06.987485+0000) 2022-09-10T08:44:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:07 smithi110 ceph-mon[97633]: pgmap v8156: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:44:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:07.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:07.465197+0000) 2022-09-10T08:44:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:07.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:07.987674+0000) 2022-09-10T08:44:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:08.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:08.465398+0000) 2022-09-10T08:44:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:09 smithi104 ceph-mon[97044]: osdmap e590: 12 total, 12 up, 12 in 2022-09-10T08:44:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:09 smithi104 ceph-mon[97044]: pgmap v8158: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:44:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:08.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:08.987844+0000) 2022-09-10T08:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:09 smithi110 ceph-mon[97633]: osdmap e590: 12 total, 12 up, 12 in 2022-09-10T08:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:09 smithi110 ceph-mon[97633]: pgmap v8158: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:44:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:09.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:09.465623+0000) 2022-09-10T08:44:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:09.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:09.987982+0000) 2022-09-10T08:44:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:10.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:10.465883+0000) 2022-09-10T08:44:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:10.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:10.988135+0000) 2022-09-10T08:44:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:11.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:11.466157+0000) 2022-09-10T08:44:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:11 smithi104 ceph-mon[97044]: pgmap v8159: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:44:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:11 smithi104 ceph-mon[97044]: osdmap e591: 12 total, 12 up, 12 in 2022-09-10T08:44:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:11 smithi110 ceph-mon[97633]: pgmap v8159: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:44:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:11 smithi110 ceph-mon[97633]: osdmap e591: 12 total, 12 up, 12 in 2022-09-10T08:44:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:11.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:11.988298+0000) 2022-09-10T08:44:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:12.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:12.466377+0000) 2022-09-10T08:44:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:12.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:12.988418+0000) 2022-09-10T08:44:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:13.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:13.466566+0000) 2022-09-10T08:44:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:13 smithi104 ceph-mon[97044]: pgmap v8161: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:44:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:13 smithi110 ceph-mon[97633]: pgmap v8161: 129 pgs: 129 active+clean; 8.6 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:44:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:13.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:13.988547+0000) 2022-09-10T08:44:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:14.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:14.466789+0000) 2022-09-10T08:44:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:14.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:14.988686+0000) 2022-09-10T08:44:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:15.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:15.466992+0000) 2022-09-10T08:44:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:15 smithi104 ceph-mon[97044]: pgmap v8162: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:44:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:15 smithi110 ceph-mon[97633]: pgmap v8162: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:44:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:15.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:15.988822+0000) 2022-09-10T08:44:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:16.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:16.467246+0000) 2022-09-10T08:44:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:17 smithi104 ceph-mon[97044]: pgmap v8163: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:44:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:16.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:16.988959+0000) 2022-09-10T08:44:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:17 smithi110 ceph-mon[97633]: pgmap v8163: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:44:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:17.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:17.467511+0000) 2022-09-10T08:44:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:17.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:17.989136+0000) 2022-09-10T08:44:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:18.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:18.467707+0000) 2022-09-10T08:44:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:18.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:18.989314+0000) 2022-09-10T08:44:19.753 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:19.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:19.467995+0000) 2022-09-10T08:44:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:19 smithi104 ceph-mon[97044]: pgmap v8164: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:44:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:44:19.750+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:44:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:19.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:19.989509+0000) 2022-09-10T08:44:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:19 smithi110 ceph-mon[97633]: pgmap v8164: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:44:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:20.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:20.468224+0000) 2022-09-10T08:44:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:20.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:20.989636+0000) 2022-09-10T08:44:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:21.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:21.468416+0000) 2022-09-10T08:44:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:21 smithi104 ceph-mon[97044]: pgmap v8165: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:44:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:21 smithi110 ceph-mon[97633]: pgmap v8165: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:44:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:21.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:21.989784+0000) 2022-09-10T08:44:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:22.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:22.468652+0000) 2022-09-10T08:44:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:22.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:22.989981+0000) 2022-09-10T08:44:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:23.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:23.468847+0000) 2022-09-10T08:44:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:23 smithi104 ceph-mon[97044]: pgmap v8166: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:44:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:23 smithi110 ceph-mon[97633]: pgmap v8166: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:44:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:23.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:23.990130+0000) 2022-09-10T08:44:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:24.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:24.469022+0000) 2022-09-10T08:44:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:24.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:24.990266+0000) 2022-09-10T08:44:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:25.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:25.469272+0000) 2022-09-10T08:44:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:25 smithi104 ceph-mon[97044]: pgmap v8167: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:44:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:25 smithi110 ceph-mon[97633]: pgmap v8167: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:44:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:25.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:25.990428+0000) 2022-09-10T08:44:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:26.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:26.469495+0000) 2022-09-10T08:44:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:27 smithi104 ceph-mon[97044]: pgmap v8168: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:26.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:26.990568+0000) 2022-09-10T08:44:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:27 smithi110 ceph-mon[97633]: pgmap v8168: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:27.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:27.469765+0000) 2022-09-10T08:44:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:27.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:27.990700+0000) 2022-09-10T08:44:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:28.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:28.469974+0000) 2022-09-10T08:44:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:28.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:28.990829+0000) 2022-09-10T08:44:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:29.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:29.470200+0000) 2022-09-10T08:44:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:29 smithi104 ceph-mon[97044]: pgmap v8169: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:29 smithi110 ceph-mon[97633]: pgmap v8169: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:29.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:29.990966+0000) 2022-09-10T08:44:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:30.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:30.470375+0000) 2022-09-10T08:44:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:30.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:30.991131+0000) 2022-09-10T08:44:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:31.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:31.470616+0000) 2022-09-10T08:44:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:31 smithi104 ceph-mon[97044]: pgmap v8170: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:31 smithi110 ceph-mon[97633]: pgmap v8170: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:31.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:31.991263+0000) 2022-09-10T08:44:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:32.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:32.470866+0000) 2022-09-10T08:44:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:32.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:32.991384+0000) 2022-09-10T08:44:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:33.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:33.471046+0000) 2022-09-10T08:44:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:33 smithi104 ceph-mon[97044]: pgmap v8171: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:33 smithi110 ceph-mon[97633]: pgmap v8171: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:33.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:33.991519+0000) 2022-09-10T08:44:34.767 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:34.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:34.471251+0000) 2022-09-10T08:44:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:34.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:34.991676+0000) 2022-09-10T08:44:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:44:34.766+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:44:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:35.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:35.471443+0000) 2022-09-10T08:44:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:35 smithi104 ceph-mon[97044]: pgmap v8172: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:35 smithi110 ceph-mon[97633]: pgmap v8172: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:35.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:35.991853+0000) 2022-09-10T08:44:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:36.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:36.471662+0000) 2022-09-10T08:44:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:37 smithi104 ceph-mon[97044]: pgmap v8173: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:36.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:36.992033+0000) 2022-09-10T08:44:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:37 smithi110 ceph-mon[97633]: pgmap v8173: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:37.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:37.471883+0000) 2022-09-10T08:44:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:37.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:37.992193+0000) 2022-09-10T08:44:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:38.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:38.472188+0000) 2022-09-10T08:44:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:38.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:38.992356+0000) 2022-09-10T08:44:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:39.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:39.472350+0000) 2022-09-10T08:44:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:39 smithi104 ceph-mon[97044]: pgmap v8174: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:39 smithi110 ceph-mon[97633]: pgmap v8174: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:39.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:39.992524+0000) 2022-09-10T08:44:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:40.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:40.472520+0000) 2022-09-10T08:44:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:40.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:40.992710+0000) 2022-09-10T08:44:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:41.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:41.472682+0000) 2022-09-10T08:44:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:41 smithi104 ceph-mon[97044]: pgmap v8175: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:41 smithi110 ceph-mon[97633]: pgmap v8175: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:41.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:41.992885+0000) 2022-09-10T08:44:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:42.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:42.472957+0000) 2022-09-10T08:44:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:42.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:42.993010+0000) 2022-09-10T08:44:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:43.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:43.473153+0000) 2022-09-10T08:44:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:43 smithi104 ceph-mon[97044]: pgmap v8176: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:43 smithi110 ceph-mon[97633]: pgmap v8176: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:43.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:43.993163+0000) 2022-09-10T08:44:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:44.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:44.473380+0000) 2022-09-10T08:44:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:44:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:44:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:44:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:44:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:44:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:44:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:44:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:44:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:44.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:44.993296+0000) 2022-09-10T08:44:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:45.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:45.473610+0000) 2022-09-10T08:44:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:45 smithi104 ceph-mon[97044]: pgmap v8177: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:45 smithi110 ceph-mon[97633]: pgmap v8177: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:45.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:45.993409+0000) 2022-09-10T08:44:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:46.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:46.473812+0000) 2022-09-10T08:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:47 smithi104 ceph-mon[97044]: pgmap v8178: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:46.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:46.993549+0000) 2022-09-10T08:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:47 smithi110 ceph-mon[97633]: pgmap v8178: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:47.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:47.474012+0000) 2022-09-10T08:44:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:47.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:47.993688+0000) 2022-09-10T08:44:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:48.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:48.474243+0000) 2022-09-10T08:44:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:48.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:48.993852+0000) 2022-09-10T08:44:49.778 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:49.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:49.474422+0000) 2022-09-10T08:44:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:49 smithi104 ceph-mon[97044]: pgmap v8179: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:49.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:49.994075+0000) 2022-09-10T08:44:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:44:49.777+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:44:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:49 smithi110 ceph-mon[97633]: pgmap v8179: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:50.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:50.474673+0000) 2022-09-10T08:44:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:50.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:50.994268+0000) 2022-09-10T08:44:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:51.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:51.474839+0000) 2022-09-10T08:44:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:51 smithi104 ceph-mon[97044]: pgmap v8180: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:51 smithi110 ceph-mon[97633]: pgmap v8180: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:51.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:51.994411+0000) 2022-09-10T08:44:52.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:52.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:52.475026+0000) 2022-09-10T08:44:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:52.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:52.994583+0000) 2022-09-10T08:44:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:53.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:53.475212+0000) 2022-09-10T08:44:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:53 smithi104 ceph-mon[97044]: pgmap v8181: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:53 smithi110 ceph-mon[97633]: pgmap v8181: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:53.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:53.994736+0000) 2022-09-10T08:44:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:54.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:54.475536+0000) 2022-09-10T08:44:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:54.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:54.994881+0000) 2022-09-10T08:44:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:55.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:55.475797+0000) 2022-09-10T08:44:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:55 smithi104 ceph-mon[97044]: pgmap v8182: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:55 smithi110 ceph-mon[97633]: pgmap v8182: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:44:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:55.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:55.995011+0000) 2022-09-10T08:44:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:56.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:56.476001+0000) 2022-09-10T08:44:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:56.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:56.995148+0000) 2022-09-10T08:44:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:57 smithi110 ceph-mon[97633]: pgmap v8183: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:57 smithi104 ceph-mon[97044]: pgmap v8183: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:44:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:57.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:57.476257+0000) 2022-09-10T08:44:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:57.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:57.995310+0000) 2022-09-10T08:44:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:58.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:58.476455+0000) 2022-09-10T08:44:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:58.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:58.995455+0000) 2022-09-10T08:44:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:44:59.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:59.476724+0000) 2022-09-10T08:45:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:44:59 smithi104 ceph-mon[97044]: pgmap v8184: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:44:59 smithi110 ceph-mon[97633]: pgmap v8184: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:44:59.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:44:59.995596+0000) 2022-09-10T08:45:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:00.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:00.476989+0000) 2022-09-10T08:45:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:00.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:00.995701+0000) 2022-09-10T08:45:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:01.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:01.477213+0000) 2022-09-10T08:45:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:01 smithi104 ceph-mon[97044]: pgmap v8185: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:01 smithi110 ceph-mon[97633]: pgmap v8185: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:01.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:01.995891+0000) 2022-09-10T08:45:02.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:02.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:02.477384+0000) 2022-09-10T08:45:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:02.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:02.996087+0000) 2022-09-10T08:45:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:03.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:03.477629+0000) 2022-09-10T08:45:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:03 smithi104 ceph-mon[97044]: pgmap v8186: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:03 smithi110 ceph-mon[97633]: pgmap v8186: 129 pgs: 129 active+clean; 8.6 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:03.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:03.996223+0000) 2022-09-10T08:45:04.794 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:04.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:04.477832+0000) 2022-09-10T08:45:05.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:45:04.792+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:45:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:04.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:04.996349+0000) 2022-09-10T08:45:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:05.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:05.478050+0000) 2022-09-10T08:45:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:05 smithi104 ceph-mon[97044]: pgmap v8187: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:45:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:05 smithi110 ceph-mon[97633]: pgmap v8187: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:45:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:05.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:05.996509+0000) 2022-09-10T08:45:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:06.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:06.478335+0000) 2022-09-10T08:45:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:06.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:06.996651+0000) 2022-09-10T08:45:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:07 smithi110 ceph-mon[97633]: pgmap v8188: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:45:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:07 smithi104 ceph-mon[97044]: pgmap v8188: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:45:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:07.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:07.478530+0000) 2022-09-10T08:45:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:07.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:07.996776+0000) 2022-09-10T08:45:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:08.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:08.478740+0000) 2022-09-10T08:45:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:08.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:08.996963+0000) 2022-09-10T08:45:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:09 smithi110 ceph-mon[97633]: osdmap e592: 12 total, 12 up, 12 in 2022-09-10T08:45:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:09 smithi110 ceph-mon[97633]: pgmap v8190: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:09 smithi104 ceph-mon[97044]: osdmap e592: 12 total, 12 up, 12 in 2022-09-10T08:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:09 smithi104 ceph-mon[97044]: pgmap v8190: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:45:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:09.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:09.478907+0000) 2022-09-10T08:45:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:09.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:09.997151+0000) 2022-09-10T08:45:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:10.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:10.479137+0000) 2022-09-10T08:45:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:10.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:10.997281+0000) 2022-09-10T08:45:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:11.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:11.479300+0000) 2022-09-10T08:45:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:11 smithi104 ceph-mon[97044]: pgmap v8191: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:45:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:11 smithi104 ceph-mon[97044]: osdmap e593: 12 total, 12 up, 12 in 2022-09-10T08:45:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:11 smithi110 ceph-mon[97633]: pgmap v8191: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:45:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:11 smithi110 ceph-mon[97633]: osdmap e593: 12 total, 12 up, 12 in 2022-09-10T08:45:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:11.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:11.997441+0000) 2022-09-10T08:45:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:12.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:12.479553+0000) 2022-09-10T08:45:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:12.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:12.997586+0000) 2022-09-10T08:45:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:13.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:13.479683+0000) 2022-09-10T08:45:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:13 smithi104 ceph-mon[97044]: pgmap v8193: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:45:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:13 smithi110 ceph-mon[97633]: pgmap v8193: 129 pgs: 129 active+clean; 8.6 MiB data, 819 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:45:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:13.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:13.997722+0000) 2022-09-10T08:45:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:14.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:14.479925+0000) 2022-09-10T08:45:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:14.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:14.997863+0000) 2022-09-10T08:45:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:15.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:15.480179+0000) 2022-09-10T08:45:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:15 smithi104 ceph-mon[97044]: pgmap v8194: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:45:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:15 smithi110 ceph-mon[97633]: pgmap v8194: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:45:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:15.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:15.997988+0000) 2022-09-10T08:45:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:16.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:16.480489+0000) 2022-09-10T08:45:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:16.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:16.998138+0000) 2022-09-10T08:45:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:17 smithi110 ceph-mon[97633]: pgmap v8195: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:45:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:17 smithi104 ceph-mon[97044]: pgmap v8195: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:45:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:17.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:17.480662+0000) 2022-09-10T08:45:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:17.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:17.998274+0000) 2022-09-10T08:45:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:18.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:18.480868+0000) 2022-09-10T08:45:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:18.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:18.998413+0000) 2022-09-10T08:45:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:19.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:19.481034+0000) 2022-09-10T08:45:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:19 smithi104 ceph-mon[97044]: pgmap v8196: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:45:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:19 smithi110 ceph-mon[97633]: pgmap v8196: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:45:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:19.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:19.998580+0000) 2022-09-10T08:45:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:45:19.807+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:45:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:20.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:20.481256+0000) 2022-09-10T08:45:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:20.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:20.998705+0000) 2022-09-10T08:45:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:21.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:21.481456+0000) 2022-09-10T08:45:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:21 smithi104 ceph-mon[97044]: pgmap v8197: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:45:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:21 smithi110 ceph-mon[97633]: pgmap v8197: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:45:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:21.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:21.998837+0000) 2022-09-10T08:45:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:22.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:22.481653+0000) 2022-09-10T08:45:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:22.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:22.998962+0000) 2022-09-10T08:45:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:23.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:23.481890+0000) 2022-09-10T08:45:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:23 smithi104 ceph-mon[97044]: pgmap v8198: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:45:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:23 smithi110 ceph-mon[97633]: pgmap v8198: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:45:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:23.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:23.999088+0000) 2022-09-10T08:45:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:24.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:24.482076+0000) 2022-09-10T08:45:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:24.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:24.999245+0000) 2022-09-10T08:45:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:25.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:25.482356+0000) 2022-09-10T08:45:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:25 smithi104 ceph-mon[97044]: pgmap v8199: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:45:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:25 smithi110 ceph-mon[97633]: pgmap v8199: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:45:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:25.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:25.999378+0000) 2022-09-10T08:45:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:26.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:26.482611+0000) 2022-09-10T08:45:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:26.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:26.999515+0000) 2022-09-10T08:45:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:27 smithi110 ceph-mon[97633]: pgmap v8200: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:27 smithi104 ceph-mon[97044]: pgmap v8200: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:27.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:27.482849+0000) 2022-09-10T08:45:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:27.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:27.999644+0000) 2022-09-10T08:45:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:28.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:28.483088+0000) 2022-09-10T08:45:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:28.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:28.999775+0000) 2022-09-10T08:45:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:29.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:29.483346+0000) 2022-09-10T08:45:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:29 smithi104 ceph-mon[97044]: pgmap v8201: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:29 smithi110 ceph-mon[97633]: pgmap v8201: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:29.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:29.999907+0000) 2022-09-10T08:45:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:30.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:30.483598+0000) 2022-09-10T08:45:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:30.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:31.000037+0000) 2022-09-10T08:45:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:31.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:31.483799+0000) 2022-09-10T08:45:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:31 smithi104 ceph-mon[97044]: pgmap v8202: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:31 smithi110 ceph-mon[97633]: pgmap v8202: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:31.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:32.000167+0000) 2022-09-10T08:45:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:32.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:32.484047+0000) 2022-09-10T08:45:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:32.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:33.000294+0000) 2022-09-10T08:45:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:33.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:33.484257+0000) 2022-09-10T08:45:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:33 smithi104 ceph-mon[97044]: pgmap v8203: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:33 smithi110 ceph-mon[97633]: pgmap v8203: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:33.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:34.000432+0000) 2022-09-10T08:45:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:34.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:34.484411+0000) 2022-09-10T08:45:35.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:45:34.823+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:45:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:35.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:35.000572+0000) 2022-09-10T08:45:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:35.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:35.484556+0000) 2022-09-10T08:45:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:35 smithi104 ceph-mon[97044]: pgmap v8204: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:35 smithi110 ceph-mon[97633]: pgmap v8204: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:36.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:36.000735+0000) 2022-09-10T08:45:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:36.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:36.484810+0000) 2022-09-10T08:45:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:37.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:37.000885+0000) 2022-09-10T08:45:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:37 smithi110 ceph-mon[97633]: pgmap v8205: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:37 smithi104 ceph-mon[97044]: pgmap v8205: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:37.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:37.484940+0000) 2022-09-10T08:45:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:38.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:38.001060+0000) 2022-09-10T08:45:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:38.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:38.485165+0000) 2022-09-10T08:45:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:39.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:39.001170+0000) 2022-09-10T08:45:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:39.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:39.485329+0000) 2022-09-10T08:45:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:39 smithi104 ceph-mon[97044]: pgmap v8206: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:39 smithi110 ceph-mon[97633]: pgmap v8206: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:40.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:40.001306+0000) 2022-09-10T08:45:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:40.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:40.485511+0000) 2022-09-10T08:45:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:41.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:41.001422+0000) 2022-09-10T08:45:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:41.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:41.485763+0000) 2022-09-10T08:45:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:41 smithi104 ceph-mon[97044]: pgmap v8207: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:41 smithi110 ceph-mon[97633]: pgmap v8207: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:42.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:42.001560+0000) 2022-09-10T08:45:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:42.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:42.485986+0000) 2022-09-10T08:45:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:43.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:43.001692+0000) 2022-09-10T08:45:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:43.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:43.486177+0000) 2022-09-10T08:45:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:43 smithi104 ceph-mon[97044]: pgmap v8208: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:43 smithi110 ceph-mon[97633]: pgmap v8208: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:44.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:44.001829+0000) 2022-09-10T08:45:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:44.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:44.486417+0000) 2022-09-10T08:45:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:45:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:45:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:45:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:45:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:45:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:45:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:45:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:45:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:45.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:45.001960+0000) 2022-09-10T08:45:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:45.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:45.486596+0000) 2022-09-10T08:45:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:45 smithi104 ceph-mon[97044]: pgmap v8209: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:45 smithi110 ceph-mon[97633]: pgmap v8209: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:46.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:46.002108+0000) 2022-09-10T08:45:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:46.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:46.486836+0000) 2022-09-10T08:45:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:47.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:47.002239+0000) 2022-09-10T08:45:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:47 smithi110 ceph-mon[97633]: pgmap v8210: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:47 smithi104 ceph-mon[97044]: pgmap v8210: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:47.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:47.487124+0000) 2022-09-10T08:45:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:48.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:48.002364+0000) 2022-09-10T08:45:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:48.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:48.487328+0000) 2022-09-10T08:45:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:49.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:49.002551+0000) 2022-09-10T08:45:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:49.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:49.487582+0000) 2022-09-10T08:45:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:49 smithi104 ceph-mon[97044]: pgmap v8211: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:49 smithi110 ceph-mon[97633]: pgmap v8211: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:50.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:45:49.837+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:45:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:50.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:50.002725+0000) 2022-09-10T08:45:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:50.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:50.487806+0000) 2022-09-10T08:45:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:51.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:51.002851+0000) 2022-09-10T08:45:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:51.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:51.488043+0000) 2022-09-10T08:45:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:51 smithi104 ceph-mon[97044]: pgmap v8212: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:51 smithi110 ceph-mon[97633]: pgmap v8212: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:52.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:52.002985+0000) 2022-09-10T08:45:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:52.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:52.488227+0000) 2022-09-10T08:45:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:53.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:53.003128+0000) 2022-09-10T08:45:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:53.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:53.488429+0000) 2022-09-10T08:45:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:53 smithi104 ceph-mon[97044]: pgmap v8213: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:53 smithi110 ceph-mon[97633]: pgmap v8213: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:54.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:54.003261+0000) 2022-09-10T08:45:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:54.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:54.488606+0000) 2022-09-10T08:45:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:55.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:55.003402+0000) 2022-09-10T08:45:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:55.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:55.488851+0000) 2022-09-10T08:45:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:55 smithi104 ceph-mon[97044]: pgmap v8214: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:55 smithi110 ceph-mon[97633]: pgmap v8214: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:45:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:56.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:56.003542+0000) 2022-09-10T08:45:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:56.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:56.489123+0000) 2022-09-10T08:45:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:57.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:57.003664+0000) 2022-09-10T08:45:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:57 smithi110 ceph-mon[97633]: pgmap v8215: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:57 smithi104 ceph-mon[97044]: pgmap v8215: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:45:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:57.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:57.489337+0000) 2022-09-10T08:45:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:58.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:58.003832+0000) 2022-09-10T08:45:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:58.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:58.489550+0000) 2022-09-10T08:45:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:45:59.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:59.003966+0000) 2022-09-10T08:45:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:45:59.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:45:59.489795+0000) 2022-09-10T08:46:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:45:59 smithi104 ceph-mon[97044]: pgmap v8216: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:45:59 smithi110 ceph-mon[97633]: pgmap v8216: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:00.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:00.004117+0000) 2022-09-10T08:46:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:00.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:00.489952+0000) 2022-09-10T08:46:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:01.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:01.004229+0000) 2022-09-10T08:46:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:01.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:01.490147+0000) 2022-09-10T08:46:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:01 smithi104 ceph-mon[97044]: pgmap v8217: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:01 smithi110 ceph-mon[97633]: pgmap v8217: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:02.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:02.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:02.004367+0000) 2022-09-10T08:46:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:02.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:02.490324+0000) 2022-09-10T08:46:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:03.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:03.004517+0000) 2022-09-10T08:46:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:03.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:03.490479+0000) 2022-09-10T08:46:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:03 smithi104 ceph-mon[97044]: pgmap v8218: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:03 smithi110 ceph-mon[97633]: pgmap v8218: 129 pgs: 129 active+clean; 8.6 MiB data, 820 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:04.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:04.004652+0000) 2022-09-10T08:46:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:04.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:04.490716+0000) 2022-09-10T08:46:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:05.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:05.004787+0000) 2022-09-10T08:46:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:46:04.841+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:46:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:05.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:05.490964+0000) 2022-09-10T08:46:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:05 smithi104 ceph-mon[97044]: pgmap v8219: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:46:06.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:05 smithi110 ceph-mon[97633]: pgmap v8219: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:46:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:06.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:06.004917+0000) 2022-09-10T08:46:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:06.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:06.491171+0000) 2022-09-10T08:46:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:07.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:07.005058+0000) 2022-09-10T08:46:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:07 smithi110 ceph-mon[97633]: pgmap v8220: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:46:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:07 smithi104 ceph-mon[97044]: pgmap v8220: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:46:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:07.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:07.491354+0000) 2022-09-10T08:46:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:08.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:08.005188+0000) 2022-09-10T08:46:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:08.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:08.491565+0000) 2022-09-10T08:46:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:09.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:09.005345+0000) 2022-09-10T08:46:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:09 smithi110 ceph-mon[97633]: osdmap e594: 12 total, 12 up, 12 in 2022-09-10T08:46:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:09 smithi110 ceph-mon[97633]: pgmap v8222: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:46:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:09 smithi104 ceph-mon[97044]: osdmap e594: 12 total, 12 up, 12 in 2022-09-10T08:46:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:09 smithi104 ceph-mon[97044]: pgmap v8222: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:46:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:09.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:09.491792+0000) 2022-09-10T08:46:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:10.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:10.005480+0000) 2022-09-10T08:46:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:10.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:10.491997+0000) 2022-09-10T08:46:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:11.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:11.005597+0000) 2022-09-10T08:46:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:11.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:11.492245+0000) 2022-09-10T08:46:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:11 smithi104 ceph-mon[97044]: pgmap v8223: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:46:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:11 smithi104 ceph-mon[97044]: osdmap e595: 12 total, 12 up, 12 in 2022-09-10T08:46:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:11 smithi110 ceph-mon[97633]: pgmap v8223: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:46:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:11 smithi110 ceph-mon[97633]: osdmap e595: 12 total, 12 up, 12 in 2022-09-10T08:46:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:12.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:12.005740+0000) 2022-09-10T08:46:12.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:12.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:12.492486+0000) 2022-09-10T08:46:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:13.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:13.005869+0000) 2022-09-10T08:46:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:13.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:13.492734+0000) 2022-09-10T08:46:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:13 smithi104 ceph-mon[97044]: pgmap v8225: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:46:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:13 smithi110 ceph-mon[97633]: pgmap v8225: 129 pgs: 129 active+clean; 8.6 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:46:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:14.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:14.006044+0000) 2022-09-10T08:46:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:14.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:14.492982+0000) 2022-09-10T08:46:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:15.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:15.006246+0000) 2022-09-10T08:46:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:15.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:15.493239+0000) 2022-09-10T08:46:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:15 smithi104 ceph-mon[97044]: pgmap v8226: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:46:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:15 smithi110 ceph-mon[97633]: pgmap v8226: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:46:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:16.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:16.006371+0000) 2022-09-10T08:46:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:16.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:16.493465+0000) 2022-09-10T08:46:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:17.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:17.006502+0000) 2022-09-10T08:46:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:17 smithi110 ceph-mon[97633]: pgmap v8227: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:46:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:17 smithi104 ceph-mon[97044]: pgmap v8227: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:46:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:17.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:17.493738+0000) 2022-09-10T08:46:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:18.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:18.006639+0000) 2022-09-10T08:46:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:18.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:18.493915+0000) 2022-09-10T08:46:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:19.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:19.006777+0000) 2022-09-10T08:46:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:19.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:19.494183+0000) 2022-09-10T08:46:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:19 smithi104 ceph-mon[97044]: pgmap v8228: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:46:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:19 smithi110 ceph-mon[97633]: pgmap v8228: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:46:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:20.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:20.006923+0000) 2022-09-10T08:46:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:46:19.852+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:46:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:20.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:20.494408+0000) 2022-09-10T08:46:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:21.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:21.007042+0000) 2022-09-10T08:46:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:21.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:21.494657+0000) 2022-09-10T08:46:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:21 smithi104 ceph-mon[97044]: pgmap v8229: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:46:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:21 smithi110 ceph-mon[97633]: pgmap v8229: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:46:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:22.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:22.007185+0000) 2022-09-10T08:46:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:22.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:22.494878+0000) 2022-09-10T08:46:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:23.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:23.007364+0000) 2022-09-10T08:46:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:23.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:23.495066+0000) 2022-09-10T08:46:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:23 smithi104 ceph-mon[97044]: pgmap v8230: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:46:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:23 smithi110 ceph-mon[97633]: pgmap v8230: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:46:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:24.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:24.007488+0000) 2022-09-10T08:46:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:24.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:24.495255+0000) 2022-09-10T08:46:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:25.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:25.007611+0000) 2022-09-10T08:46:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:25.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:25.495459+0000) 2022-09-10T08:46:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:25 smithi104 ceph-mon[97044]: pgmap v8231: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:46:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:25 smithi110 ceph-mon[97633]: pgmap v8231: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:46:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:26.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:26.007769+0000) 2022-09-10T08:46:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:26.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:26.495639+0000) 2022-09-10T08:46:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:27.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:27.007905+0000) 2022-09-10T08:46:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:27 smithi110 ceph-mon[97633]: pgmap v8232: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:27 smithi104 ceph-mon[97044]: pgmap v8232: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:27.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:27.495813+0000) 2022-09-10T08:46:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:28.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:28.008036+0000) 2022-09-10T08:46:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:28.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:28.496063+0000) 2022-09-10T08:46:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:29.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:29.008173+0000) 2022-09-10T08:46:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:29.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:29.496280+0000) 2022-09-10T08:46:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:29 smithi104 ceph-mon[97044]: pgmap v8233: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:29 smithi110 ceph-mon[97633]: pgmap v8233: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:30.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:30.008312+0000) 2022-09-10T08:46:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:30.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:30.496524+0000) 2022-09-10T08:46:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:31.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:31.008463+0000) 2022-09-10T08:46:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:31.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:31.496764+0000) 2022-09-10T08:46:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:31 smithi104 ceph-mon[97044]: pgmap v8234: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:31 smithi110 ceph-mon[97633]: pgmap v8234: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:32.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:32.008594+0000) 2022-09-10T08:46:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:32.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:32.496977+0000) 2022-09-10T08:46:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:33.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:33.008782+0000) 2022-09-10T08:46:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:33.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:33.497235+0000) 2022-09-10T08:46:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:33 smithi104 ceph-mon[97044]: pgmap v8235: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:33 smithi110 ceph-mon[97633]: pgmap v8235: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:34.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:34.008963+0000) 2022-09-10T08:46:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:34.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:34.497485+0000) 2022-09-10T08:46:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:35.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:35.009175+0000) 2022-09-10T08:46:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:46:34.857+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:46:35.817 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:35.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:35.497694+0000) 2022-09-10T08:46:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:35 smithi104 ceph-mon[97044]: pgmap v8236: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:35 smithi110 ceph-mon[97633]: pgmap v8236: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:36.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:36.009293+0000) 2022-09-10T08:46:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:36.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:36.497928+0000) 2022-09-10T08:46:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:37.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:37.009406+0000) 2022-09-10T08:46:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:37 smithi110 ceph-mon[97633]: pgmap v8237: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:37 smithi104 ceph-mon[97044]: pgmap v8237: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:37.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:37.498194+0000) 2022-09-10T08:46:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:38.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:38.009536+0000) 2022-09-10T08:46:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:38.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:38.498416+0000) 2022-09-10T08:46:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:39.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:39.009662+0000) 2022-09-10T08:46:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:39.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:39.498715+0000) 2022-09-10T08:46:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:39 smithi104 ceph-mon[97044]: pgmap v8238: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:39 smithi110 ceph-mon[97633]: pgmap v8238: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:40.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:40.009789+0000) 2022-09-10T08:46:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:40.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:40.498966+0000) 2022-09-10T08:46:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:41.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:41.009911+0000) 2022-09-10T08:46:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:41.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:41.499233+0000) 2022-09-10T08:46:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:41 smithi104 ceph-mon[97044]: pgmap v8239: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:41 smithi110 ceph-mon[97633]: pgmap v8239: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:42.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:42.010049+0000) 2022-09-10T08:46:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:42.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:42.499477+0000) 2022-09-10T08:46:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:43.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:43.010253+0000) 2022-09-10T08:46:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:43.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:43.499636+0000) 2022-09-10T08:46:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:43 smithi104 ceph-mon[97044]: pgmap v8240: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:43 smithi110 ceph-mon[97633]: pgmap v8240: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:44.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:44.010390+0000) 2022-09-10T08:46:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:44.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:44.499869+0000) 2022-09-10T08:46:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:46:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:46:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:46:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:46:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:46:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:46:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:46:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:46:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:45.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:45.010528+0000) 2022-09-10T08:46:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:45.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:45.500144+0000) 2022-09-10T08:46:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:45 smithi104 ceph-mon[97044]: pgmap v8241: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:45 smithi110 ceph-mon[97633]: pgmap v8241: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:46.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:46.010709+0000) 2022-09-10T08:46:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:46.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:46.500429+0000) 2022-09-10T08:46:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:47.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:47.010857+0000) 2022-09-10T08:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:47 smithi110 ceph-mon[97633]: pgmap v8242: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:47 smithi104 ceph-mon[97044]: pgmap v8242: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:47.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:47.500661+0000) 2022-09-10T08:46:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:48.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:48.010982+0000) 2022-09-10T08:46:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:48.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:48.500872+0000) 2022-09-10T08:46:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:49.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:49.011249+0000) 2022-09-10T08:46:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:49.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:49.501139+0000) 2022-09-10T08:46:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:49 smithi104 ceph-mon[97044]: pgmap v8243: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:49 smithi110 ceph-mon[97633]: pgmap v8243: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:50.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:50.011410+0000) 2022-09-10T08:46:50.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:46:49.872+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:46:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:50.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:50.501417+0000) 2022-09-10T08:46:51.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:51.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:51.011543+0000) 2022-09-10T08:46:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:51.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:51.501668+0000) 2022-09-10T08:46:52.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:51 smithi104 ceph-mon[97044]: pgmap v8244: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:51 smithi110 ceph-mon[97633]: pgmap v8244: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:52.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:52.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:52.011692+0000) 2022-09-10T08:46:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:52.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:52.501848+0000) 2022-09-10T08:46:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:53.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:53.011812+0000) 2022-09-10T08:46:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:53.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:53.502090+0000) 2022-09-10T08:46:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:53 smithi104 ceph-mon[97044]: pgmap v8245: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:53 smithi110 ceph-mon[97633]: pgmap v8245: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:54.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:54.011937+0000) 2022-09-10T08:46:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:54.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:54.502375+0000) 2022-09-10T08:46:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:55.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:55.012063+0000) 2022-09-10T08:46:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:55.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:55.502629+0000) 2022-09-10T08:46:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:55 smithi104 ceph-mon[97044]: pgmap v8246: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:55 smithi110 ceph-mon[97633]: pgmap v8246: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:46:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:56.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:56.012251+0000) 2022-09-10T08:46:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:56.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:56.502884+0000) 2022-09-10T08:46:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:57.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:57.012488+0000) 2022-09-10T08:46:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:57 smithi110 ceph-mon[97633]: pgmap v8247: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:57 smithi104 ceph-mon[97044]: pgmap v8247: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:46:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:57.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:57.503087+0000) 2022-09-10T08:46:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:58.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:58.012610+0000) 2022-09-10T08:46:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:58.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:58.503337+0000) 2022-09-10T08:46:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:46:59.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:59.012734+0000) 2022-09-10T08:46:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:46:59.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:46:59.503531+0000) 2022-09-10T08:47:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:46:59 smithi104 ceph-mon[97044]: pgmap v8248: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:46:59 smithi110 ceph-mon[97633]: pgmap v8248: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:00.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:00.012877+0000) 2022-09-10T08:47:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:00.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:00.503727+0000) 2022-09-10T08:47:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:01.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:01.012986+0000) 2022-09-10T08:47:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:01.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:01.503935+0000) 2022-09-10T08:47:02.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:01 smithi104 ceph-mon[97044]: pgmap v8249: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:01 smithi110 ceph-mon[97633]: pgmap v8249: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:02.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:02.013136+0000) 2022-09-10T08:47:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:02.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:02.504161+0000) 2022-09-10T08:47:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:03.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:03.013259+0000) 2022-09-10T08:47:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:03.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:03.504390+0000) 2022-09-10T08:47:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:03 smithi104 ceph-mon[97044]: pgmap v8250: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:03 smithi110 ceph-mon[97633]: pgmap v8250: 129 pgs: 129 active+clean; 8.6 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:04.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:04.013437+0000) 2022-09-10T08:47:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:04.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:04.504562+0000) 2022-09-10T08:47:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:05.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:05.013593+0000) 2022-09-10T08:47:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:47:04.886+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:47:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:05.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:05.504776+0000) 2022-09-10T08:47:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:05 smithi104 ceph-mon[97044]: pgmap v8251: 129 pgs: 129 active+clean; 8.6 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:47:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:05 smithi110 ceph-mon[97633]: pgmap v8251: 129 pgs: 129 active+clean; 8.6 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:47:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:06.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:06.013721+0000) 2022-09-10T08:47:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:06.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:06.505004+0000) 2022-09-10T08:47:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:07.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:07.013852+0000) 2022-09-10T08:47:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:07 smithi110 ceph-mon[97633]: pgmap v8252: 129 pgs: 129 active+clean; 8.6 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:47:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:07 smithi104 ceph-mon[97044]: pgmap v8252: 129 pgs: 129 active+clean; 8.6 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:47:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:07.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:07.505216+0000) 2022-09-10T08:47:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:08.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:08.013986+0000) 2022-09-10T08:47:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:08.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:08.505484+0000) 2022-09-10T08:47:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:09.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:09.014138+0000) 2022-09-10T08:47:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:09 smithi110 ceph-mon[97633]: osdmap e596: 12 total, 12 up, 12 in 2022-09-10T08:47:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:09 smithi110 ceph-mon[97633]: pgmap v8254: 129 pgs: 129 active+clean; 8.6 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:09 smithi104 ceph-mon[97044]: osdmap e596: 12 total, 12 up, 12 in 2022-09-10T08:47:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:09 smithi104 ceph-mon[97044]: pgmap v8254: 129 pgs: 129 active+clean; 8.6 MiB data, 823 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:47:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:09.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:09.505665+0000) 2022-09-10T08:47:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:10.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:10.014280+0000) 2022-09-10T08:47:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:10.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:10.505848+0000) 2022-09-10T08:47:11.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:11.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:11.014417+0000) 2022-09-10T08:47:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:11.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:11.506125+0000) 2022-09-10T08:47:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:11 smithi104 ceph-mon[97044]: pgmap v8255: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:47:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:11 smithi104 ceph-mon[97044]: osdmap e597: 12 total, 12 up, 12 in 2022-09-10T08:47:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:11 smithi110 ceph-mon[97633]: pgmap v8255: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:47:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:11 smithi110 ceph-mon[97633]: osdmap e597: 12 total, 12 up, 12 in 2022-09-10T08:47:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:12.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:12.014566+0000) 2022-09-10T08:47:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:12.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:12.506350+0000) 2022-09-10T08:47:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:13.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:13.014764+0000) 2022-09-10T08:47:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:13.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:13.506535+0000) 2022-09-10T08:47:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:13 smithi104 ceph-mon[97044]: pgmap v8257: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:47:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:13 smithi110 ceph-mon[97633]: pgmap v8257: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:47:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:14.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:14.014927+0000) 2022-09-10T08:47:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:14.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:14.506765+0000) 2022-09-10T08:47:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:15.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:15.015061+0000) 2022-09-10T08:47:15.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:15.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:15.506987+0000) 2022-09-10T08:47:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:15 smithi104 ceph-mon[97044]: pgmap v8258: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:47:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:15 smithi110 ceph-mon[97633]: pgmap v8258: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:47:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:16.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:16.015230+0000) 2022-09-10T08:47:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:16.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:16.507219+0000) 2022-09-10T08:47:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:17.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:17.015375+0000) 2022-09-10T08:47:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:17 smithi110 ceph-mon[97633]: pgmap v8259: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:47:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:17 smithi104 ceph-mon[97044]: pgmap v8259: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:47:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:17.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:17.507485+0000) 2022-09-10T08:47:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:18.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:18.015505+0000) 2022-09-10T08:47:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:18.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:18.507668+0000) 2022-09-10T08:47:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:19.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:19.015674+0000) 2022-09-10T08:47:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:19.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:19.507909+0000) 2022-09-10T08:47:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:19 smithi104 ceph-mon[97044]: pgmap v8260: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:47:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:19 smithi110 ceph-mon[97633]: pgmap v8260: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:47:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:47:19.900+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:47:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:20.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:20.015824+0000) 2022-09-10T08:47:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:20.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:20.508152+0000) 2022-09-10T08:47:21.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:21.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:21.015997+0000) 2022-09-10T08:47:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:21.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:21.508399+0000) 2022-09-10T08:47:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:21 smithi104 ceph-mon[97044]: pgmap v8261: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:47:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:21 smithi110 ceph-mon[97633]: pgmap v8261: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:47:22.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:22.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:22.016161+0000) 2022-09-10T08:47:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:22.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:22.508605+0000) 2022-09-10T08:47:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:23.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:23.016340+0000) 2022-09-10T08:47:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:23.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:23.508833+0000) 2022-09-10T08:47:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:23 smithi104 ceph-mon[97044]: pgmap v8262: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:47:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:23 smithi110 ceph-mon[97633]: pgmap v8262: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:47:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:24.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:24.016505+0000) 2022-09-10T08:47:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:24.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:24.509133+0000) 2022-09-10T08:47:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:25.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:25.016637+0000) 2022-09-10T08:47:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:25.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:25.509413+0000) 2022-09-10T08:47:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:25 smithi104 ceph-mon[97044]: pgmap v8263: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:47:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:25 smithi110 ceph-mon[97633]: pgmap v8263: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:47:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:26.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:26.016816+0000) 2022-09-10T08:47:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:26.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:26.509647+0000) 2022-09-10T08:47:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:27.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:27.016990+0000) 2022-09-10T08:47:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:27 smithi110 ceph-mon[97633]: pgmap v8264: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:27 smithi104 ceph-mon[97044]: pgmap v8264: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:27.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:27.509888+0000) 2022-09-10T08:47:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:28.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:28.017161+0000) 2022-09-10T08:47:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:28.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:28.510148+0000) 2022-09-10T08:47:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:29.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:29.017317+0000) 2022-09-10T08:47:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:29.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:29.510394+0000) 2022-09-10T08:47:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:29 smithi104 ceph-mon[97044]: pgmap v8265: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:29 smithi110 ceph-mon[97633]: pgmap v8265: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:30.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:30.017447+0000) 2022-09-10T08:47:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:30.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:30.510660+0000) 2022-09-10T08:47:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:31.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:31.017573+0000) 2022-09-10T08:47:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:31.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:31.510861+0000) 2022-09-10T08:47:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:31 smithi104 ceph-mon[97044]: pgmap v8266: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:31 smithi110 ceph-mon[97633]: pgmap v8266: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:32.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:32.017708+0000) 2022-09-10T08:47:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:32.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:32.511130+0000) 2022-09-10T08:47:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:33.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:33.017869+0000) 2022-09-10T08:47:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:33.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:33.511358+0000) 2022-09-10T08:47:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:33 smithi104 ceph-mon[97044]: pgmap v8267: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:33 smithi110 ceph-mon[97633]: pgmap v8267: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:34.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:34.018046+0000) 2022-09-10T08:47:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:34.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:34.511602+0000) 2022-09-10T08:47:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:35.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:35.018199+0000) 2022-09-10T08:47:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:47:34.900+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:47:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:35.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:35.511782+0000) 2022-09-10T08:47:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:35 smithi104 ceph-mon[97044]: pgmap v8268: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:35 smithi110 ceph-mon[97633]: pgmap v8268: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:36.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:36.018331+0000) 2022-09-10T08:47:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:36.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:36.512018+0000) 2022-09-10T08:47:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:37.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:37.018454+0000) 2022-09-10T08:47:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:37 smithi110 ceph-mon[97633]: pgmap v8269: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:37 smithi104 ceph-mon[97044]: pgmap v8269: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:37.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:37.512244+0000) 2022-09-10T08:47:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:38.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:38.018596+0000) 2022-09-10T08:47:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:38.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:38.512498+0000) 2022-09-10T08:47:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:39.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:39.018730+0000) 2022-09-10T08:47:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:39.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:39.512711+0000) 2022-09-10T08:47:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:39 smithi104 ceph-mon[97044]: pgmap v8270: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:39 smithi110 ceph-mon[97633]: pgmap v8270: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:40.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:40.018867+0000) 2022-09-10T08:47:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:40.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:40.512961+0000) 2022-09-10T08:47:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:41.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:41.019002+0000) 2022-09-10T08:47:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:41.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:41.513228+0000) 2022-09-10T08:47:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:41 smithi104 ceph-mon[97044]: pgmap v8271: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:41 smithi110 ceph-mon[97633]: pgmap v8271: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:42.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:42.019207+0000) 2022-09-10T08:47:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:42.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:42.513475+0000) 2022-09-10T08:47:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:43.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:43.019344+0000) 2022-09-10T08:47:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:43.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:43.513708+0000) 2022-09-10T08:47:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:43 smithi104 ceph-mon[97044]: pgmap v8272: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:43 smithi110 ceph-mon[97633]: pgmap v8272: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:44.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:44.019534+0000) 2022-09-10T08:47:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:44.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:44.513949+0000) 2022-09-10T08:47:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:47:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:47:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:47:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:47:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:47:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:47:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:47:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:47:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:45.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:45.019668+0000) 2022-09-10T08:47:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:45.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:45.514203+0000) 2022-09-10T08:47:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:45 smithi104 ceph-mon[97044]: pgmap v8273: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:45 smithi110 ceph-mon[97633]: pgmap v8273: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:46.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:46.019789+0000) 2022-09-10T08:47:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:46.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:46.514455+0000) 2022-09-10T08:47:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:47.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:47.019914+0000) 2022-09-10T08:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:47 smithi110 ceph-mon[97633]: pgmap v8274: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:47 smithi104 ceph-mon[97044]: pgmap v8274: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:47.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:47.514599+0000) 2022-09-10T08:47:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:48.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:48.020057+0000) 2022-09-10T08:47:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:48.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:48.514764+0000) 2022-09-10T08:47:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:49.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:49.020249+0000) 2022-09-10T08:47:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:49.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:49.514975+0000) 2022-09-10T08:47:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:49 smithi104 ceph-mon[97044]: pgmap v8275: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:49 smithi110 ceph-mon[97633]: pgmap v8275: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:50.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:50.020384+0000) 2022-09-10T08:47:50.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:47:49.914+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:47:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:50.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:50.515143+0000) 2022-09-10T08:47:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:51.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:51.020511+0000) 2022-09-10T08:47:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:51.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:51.515340+0000) 2022-09-10T08:47:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:51 smithi104 ceph-mon[97044]: pgmap v8276: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:51 smithi110 ceph-mon[97633]: pgmap v8276: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:52.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:52.020641+0000) 2022-09-10T08:47:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:52.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:52.515525+0000) 2022-09-10T08:47:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:53.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:53.020781+0000) 2022-09-10T08:47:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:53.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:53.515799+0000) 2022-09-10T08:47:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:53 smithi104 ceph-mon[97044]: pgmap v8277: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:53 smithi110 ceph-mon[97633]: pgmap v8277: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:54.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:54.020910+0000) 2022-09-10T08:47:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:54.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:54.516022+0000) 2022-09-10T08:47:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:55.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:55.021036+0000) 2022-09-10T08:47:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:55.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:55.516214+0000) 2022-09-10T08:47:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:55 smithi104 ceph-mon[97044]: pgmap v8278: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:55 smithi110 ceph-mon[97633]: pgmap v8278: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:47:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:56.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:56.021173+0000) 2022-09-10T08:47:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:56.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:56.516473+0000) 2022-09-10T08:47:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:57.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:57.021305+0000) 2022-09-10T08:47:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:57 smithi110 ceph-mon[97633]: pgmap v8279: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:57 smithi104 ceph-mon[97044]: pgmap v8279: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:47:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:57.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:57.516688+0000) 2022-09-10T08:47:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:58.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:58.021447+0000) 2022-09-10T08:47:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:58.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:58.516904+0000) 2022-09-10T08:47:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:47:59.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:59.021624+0000) 2022-09-10T08:47:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:47:59.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:47:59.517080+0000) 2022-09-10T08:48:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:47:59 smithi104 ceph-mon[97044]: pgmap v8280: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:47:59 smithi110 ceph-mon[97633]: pgmap v8280: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:00.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:00.021767+0000) 2022-09-10T08:48:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:00.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:00.517359+0000) 2022-09-10T08:48:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:01.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:01.021879+0000) 2022-09-10T08:48:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:01.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:01.517569+0000) 2022-09-10T08:48:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:01 smithi104 ceph-mon[97044]: pgmap v8281: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:01 smithi110 ceph-mon[97633]: pgmap v8281: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:02.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:02.022023+0000) 2022-09-10T08:48:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:02.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:02.517798+0000) 2022-09-10T08:48:03.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:03.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:03.022169+0000) 2022-09-10T08:48:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:03.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:03.518005+0000) 2022-09-10T08:48:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:03 smithi104 ceph-mon[97044]: pgmap v8282: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:03 smithi110 ceph-mon[97633]: pgmap v8282: 129 pgs: 129 active+clean; 8.6 MiB data, 824 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:04.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:04.022353+0000) 2022-09-10T08:48:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:04.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:04.518188+0000) 2022-09-10T08:48:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:05.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:05.022489+0000) 2022-09-10T08:48:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:48:04.924+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:48:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:05.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:05.518402+0000) 2022-09-10T08:48:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:05 smithi104 ceph-mon[97044]: pgmap v8283: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:48:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:05 smithi110 ceph-mon[97633]: pgmap v8283: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:48:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:06.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:06.022584+0000) 2022-09-10T08:48:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:06.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:06.518544+0000) 2022-09-10T08:48:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:07.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:07.022730+0000) 2022-09-10T08:48:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:07 smithi110 ceph-mon[97633]: pgmap v8284: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:48:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:07 smithi104 ceph-mon[97044]: pgmap v8284: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:48:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:07.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:07.518783+0000) 2022-09-10T08:48:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:08.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:08.022946+0000) 2022-09-10T08:48:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:08.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:08.519036+0000) 2022-09-10T08:48:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:09.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:09.023112+0000) 2022-09-10T08:48:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:09 smithi110 ceph-mon[97633]: osdmap e598: 12 total, 12 up, 12 in 2022-09-10T08:48:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:09 smithi110 ceph-mon[97633]: pgmap v8286: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:09 smithi104 ceph-mon[97044]: osdmap e598: 12 total, 12 up, 12 in 2022-09-10T08:48:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:09 smithi104 ceph-mon[97044]: pgmap v8286: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:48:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:09.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:09.519255+0000) 2022-09-10T08:48:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:10.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:10.023241+0000) 2022-09-10T08:48:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:10.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:10.519489+0000) 2022-09-10T08:48:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:11.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:11.023415+0000) 2022-09-10T08:48:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:11.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:11.519665+0000) 2022-09-10T08:48:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:11 smithi104 ceph-mon[97044]: pgmap v8287: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:48:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:11 smithi104 ceph-mon[97044]: osdmap e599: 12 total, 12 up, 12 in 2022-09-10T08:48:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:11 smithi110 ceph-mon[97633]: pgmap v8287: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:48:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:11 smithi110 ceph-mon[97633]: osdmap e599: 12 total, 12 up, 12 in 2022-09-10T08:48:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:12.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:12.023553+0000) 2022-09-10T08:48:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:12.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:12.519848+0000) 2022-09-10T08:48:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:13.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:13.023675+0000) 2022-09-10T08:48:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:13.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:13.520042+0000) 2022-09-10T08:48:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:13 smithi104 ceph-mon[97044]: pgmap v8289: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:48:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:13 smithi110 ceph-mon[97633]: pgmap v8289: 129 pgs: 129 active+clean; 8.6 MiB data, 830 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:48:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:14.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:14.023804+0000) 2022-09-10T08:48:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:14.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:14.520274+0000) 2022-09-10T08:48:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:15.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:15.023942+0000) 2022-09-10T08:48:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:15.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:15.520436+0000) 2022-09-10T08:48:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:15 smithi104 ceph-mon[97044]: pgmap v8290: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:48:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:15 smithi110 ceph-mon[97633]: pgmap v8290: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:48:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:16.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:16.024074+0000) 2022-09-10T08:48:16.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:16.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:16.520655+0000) 2022-09-10T08:48:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:17.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:17.024170+0000) 2022-09-10T08:48:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:17 smithi110 ceph-mon[97633]: pgmap v8291: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:17 smithi104 ceph-mon[97044]: pgmap v8291: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:48:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:17.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:17.520886+0000) 2022-09-10T08:48:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:18.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:18.024298+0000) 2022-09-10T08:48:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:18.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:18.521016+0000) 2022-09-10T08:48:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:19.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:19.024443+0000) 2022-09-10T08:48:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:19.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:19.521235+0000) 2022-09-10T08:48:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:19 smithi104 ceph-mon[97044]: pgmap v8292: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:48:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:19 smithi110 ceph-mon[97633]: pgmap v8292: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:48:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:48:19.940+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:48:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:20.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:20.024582+0000) 2022-09-10T08:48:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:20.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:20.521454+0000) 2022-09-10T08:48:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:21.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:21.024715+0000) 2022-09-10T08:48:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:21.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:21.521649+0000) 2022-09-10T08:48:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:21 smithi104 ceph-mon[97044]: pgmap v8293: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:48:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:21 smithi110 ceph-mon[97633]: pgmap v8293: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:48:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:22.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:22.024855+0000) 2022-09-10T08:48:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:22.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:22.521835+0000) 2022-09-10T08:48:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:23.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:23.024981+0000) 2022-09-10T08:48:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:23.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:23.522062+0000) 2022-09-10T08:48:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:23 smithi104 ceph-mon[97044]: pgmap v8294: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:48:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:23 smithi110 ceph-mon[97633]: pgmap v8294: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T08:48:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:24.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:24.025170+0000) 2022-09-10T08:48:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:24.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:24.522401+0000) 2022-09-10T08:48:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:25.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:25.025301+0000) 2022-09-10T08:48:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:25.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:25.522585+0000) 2022-09-10T08:48:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:25 smithi104 ceph-mon[97044]: pgmap v8295: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:48:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:25 smithi110 ceph-mon[97633]: pgmap v8295: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:48:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:26.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:26.025442+0000) 2022-09-10T08:48:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:26.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:26.522838+0000) 2022-09-10T08:48:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:27.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:27.025570+0000) 2022-09-10T08:48:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:27 smithi110 ceph-mon[97633]: pgmap v8296: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:27 smithi104 ceph-mon[97044]: pgmap v8296: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:27.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:27.523015+0000) 2022-09-10T08:48:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:28.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:28.025712+0000) 2022-09-10T08:48:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:28.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:28.523191+0000) 2022-09-10T08:48:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:29.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:29.025871+0000) 2022-09-10T08:48:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:29.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:29.523371+0000) 2022-09-10T08:48:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:29 smithi104 ceph-mon[97044]: pgmap v8297: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:29 smithi110 ceph-mon[97633]: pgmap v8297: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:30.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:30.026001+0000) 2022-09-10T08:48:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:30.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:30.523619+0000) 2022-09-10T08:48:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:31.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:31.026128+0000) 2022-09-10T08:48:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:31.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:31.523838+0000) 2022-09-10T08:48:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:31 smithi104 ceph-mon[97044]: pgmap v8298: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:31 smithi110 ceph-mon[97633]: pgmap v8298: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:32.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:32.026266+0000) 2022-09-10T08:48:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:32.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:32.524081+0000) 2022-09-10T08:48:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:33.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:33.026389+0000) 2022-09-10T08:48:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:33.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:33.524317+0000) 2022-09-10T08:48:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:33 smithi104 ceph-mon[97044]: pgmap v8299: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:33 smithi110 ceph-mon[97633]: pgmap v8299: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:34.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:34.026507+0000) 2022-09-10T08:48:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:34.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:34.524535+0000) 2022-09-10T08:48:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:35.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:35.026669+0000) 2022-09-10T08:48:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:48:34.949+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:48:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:35.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:35.524744+0000) 2022-09-10T08:48:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:35 smithi110 ceph-mon[97633]: pgmap v8300: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:35 smithi104 ceph-mon[97044]: pgmap v8300: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:36.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:36.026856+0000) 2022-09-10T08:48:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:36.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:36.524974+0000) 2022-09-10T08:48:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:37.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:37.026986+0000) 2022-09-10T08:48:37.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:37 smithi110 ceph-mon[97633]: pgmap v8301: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:37 smithi104 ceph-mon[97044]: pgmap v8301: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:37.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:37.525201+0000) 2022-09-10T08:48:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:38.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:38.027129+0000) 2022-09-10T08:48:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:38.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:38.525431+0000) 2022-09-10T08:48:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:39.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:39.027259+0000) 2022-09-10T08:48:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:39.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:39.525657+0000) 2022-09-10T08:48:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:39 smithi104 ceph-mon[97044]: pgmap v8302: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:39 smithi110 ceph-mon[97633]: pgmap v8302: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:40.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:40.027409+0000) 2022-09-10T08:48:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:40.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:40.525823+0000) 2022-09-10T08:48:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:41.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:41.027529+0000) 2022-09-10T08:48:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:41.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:41.526050+0000) 2022-09-10T08:48:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:41 smithi104 ceph-mon[97044]: pgmap v8303: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:41 smithi110 ceph-mon[97633]: pgmap v8303: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:42.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:42.027661+0000) 2022-09-10T08:48:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:42.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:42.526325+0000) 2022-09-10T08:48:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:43.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:43.027786+0000) 2022-09-10T08:48:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:43.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:43.526493+0000) 2022-09-10T08:48:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:43 smithi104 ceph-mon[97044]: pgmap v8304: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:43 smithi110 ceph-mon[97633]: pgmap v8304: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:44.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:44.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:44.027911+0000) 2022-09-10T08:48:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:44.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:44.526662+0000) 2022-09-10T08:48:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:48:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:48:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:48:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:48:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:48:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:48:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:48:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:48:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:45.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:45.028031+0000) 2022-09-10T08:48:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:45.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:45.526886+0000) 2022-09-10T08:48:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:45 smithi110 ceph-mon[97633]: pgmap v8305: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:45 smithi104 ceph-mon[97044]: pgmap v8305: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:46.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:46.028154+0000) 2022-09-10T08:48:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:46.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:46.527413+0000) 2022-09-10T08:48:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:47.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:47.028281+0000) 2022-09-10T08:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:47 smithi110 ceph-mon[97633]: pgmap v8306: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:47 smithi104 ceph-mon[97044]: pgmap v8306: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:47.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:47.527576+0000) 2022-09-10T08:48:48.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:48.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:48.028416+0000) 2022-09-10T08:48:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:48.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:48.527825+0000) 2022-09-10T08:48:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:49.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:49.028541+0000) 2022-09-10T08:48:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:49.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:49.528046+0000) 2022-09-10T08:48:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:49 smithi104 ceph-mon[97044]: pgmap v8307: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:49 smithi110 ceph-mon[97633]: pgmap v8307: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:50.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:48:49.963+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:48:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:50.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:50.028668+0000) 2022-09-10T08:48:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:50.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:50.528217+0000) 2022-09-10T08:48:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:51.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:51.028796+0000) 2022-09-10T08:48:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:51.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:51.528419+0000) 2022-09-10T08:48:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:51 smithi104 ceph-mon[97044]: pgmap v8308: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:51 smithi110 ceph-mon[97633]: pgmap v8308: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:52.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:52.028942+0000) 2022-09-10T08:48:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:52.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:52.528807+0000) 2022-09-10T08:48:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:53.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:53.029066+0000) 2022-09-10T08:48:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:53.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:53.529048+0000) 2022-09-10T08:48:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:53 smithi110 ceph-mon[97633]: pgmap v8309: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:53 smithi104 ceph-mon[97044]: pgmap v8309: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:54.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:54.029194+0000) 2022-09-10T08:48:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:54.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:54.529183+0000) 2022-09-10T08:48:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:55.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:55.029332+0000) 2022-09-10T08:48:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:55.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:55.529390+0000) 2022-09-10T08:48:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:55 smithi110 ceph-mon[97633]: pgmap v8310: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:55 smithi104 ceph-mon[97044]: pgmap v8310: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:48:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:56.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:56.029458+0000) 2022-09-10T08:48:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:56.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:56.529651+0000) 2022-09-10T08:48:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:57.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:57.029601+0000) 2022-09-10T08:48:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:57 smithi110 ceph-mon[97633]: pgmap v8311: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:57 smithi104 ceph-mon[97044]: pgmap v8311: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:48:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:57.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:57.529907+0000) 2022-09-10T08:48:58.316 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:58.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:58.029789+0000) 2022-09-10T08:48:58.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:58.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:58.530072+0000) 2022-09-10T08:48:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:48:59.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:59.029930+0000) 2022-09-10T08:48:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:48:59.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:48:59.530310+0000) 2022-09-10T08:49:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:48:59 smithi104 ceph-mon[97044]: pgmap v8312: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:48:59 smithi110 ceph-mon[97633]: pgmap v8312: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:00.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:00.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:00.030074+0000) 2022-09-10T08:49:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:00.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:00.530586+0000) 2022-09-10T08:49:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:01.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:01.030202+0000) 2022-09-10T08:49:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:01.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:01.530822+0000) 2022-09-10T08:49:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:01 smithi110 ceph-mon[97633]: pgmap v8313: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:01 smithi104 ceph-mon[97044]: pgmap v8313: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:02.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:02.030350+0000) 2022-09-10T08:49:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:02.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:02.531051+0000) 2022-09-10T08:49:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:03.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:03.030488+0000) 2022-09-10T08:49:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:03.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:03.531303+0000) 2022-09-10T08:49:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:03 smithi110 ceph-mon[97633]: pgmap v8314: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:03 smithi104 ceph-mon[97044]: pgmap v8314: 129 pgs: 129 active+clean; 8.6 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:04.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:04.030621+0000) 2022-09-10T08:49:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:04.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:04.531551+0000) 2022-09-10T08:49:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:05.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:05.030761+0000) 2022-09-10T08:49:05.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:49:04.976+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:49:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:05.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:05.531696+0000) 2022-09-10T08:49:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:05 smithi110 ceph-mon[97633]: pgmap v8315: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:49:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:05 smithi104 ceph-mon[97044]: pgmap v8315: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:49:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:06.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:06.030895+0000) 2022-09-10T08:49:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:06.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:06.531956+0000) 2022-09-10T08:49:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:07.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:07.031059+0000) 2022-09-10T08:49:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:07 smithi110 ceph-mon[97633]: pgmap v8316: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:49:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:07 smithi104 ceph-mon[97044]: pgmap v8316: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:49:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:07.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:07.532211+0000) 2022-09-10T08:49:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:08.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:08.031173+0000) 2022-09-10T08:49:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:08.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:08.532410+0000) 2022-09-10T08:49:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:09.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:09.031310+0000) 2022-09-10T08:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:09 smithi110 ceph-mon[97633]: osdmap e600: 12 total, 12 up, 12 in 2022-09-10T08:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:09 smithi110 ceph-mon[97633]: pgmap v8318: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:49:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:09 smithi104 ceph-mon[97044]: osdmap e600: 12 total, 12 up, 12 in 2022-09-10T08:49:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:09 smithi104 ceph-mon[97044]: pgmap v8318: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:49:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:09.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:09.532632+0000) 2022-09-10T08:49:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:10.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:10.031437+0000) 2022-09-10T08:49:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:10.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:10.532898+0000) 2022-09-10T08:49:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:11.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:11.031562+0000) 2022-09-10T08:49:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:11.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:11.533089+0000) 2022-09-10T08:49:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:11 smithi110 ceph-mon[97633]: pgmap v8319: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:49:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:11 smithi110 ceph-mon[97633]: osdmap e601: 12 total, 12 up, 12 in 2022-09-10T08:49:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:11 smithi104 ceph-mon[97044]: pgmap v8319: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:49:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:11 smithi104 ceph-mon[97044]: osdmap e601: 12 total, 12 up, 12 in 2022-09-10T08:49:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:12.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:12.031709+0000) 2022-09-10T08:49:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:12.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:12.533410+0000) 2022-09-10T08:49:13.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:13.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:13.031904+0000) 2022-09-10T08:49:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:13.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:13.533571+0000) 2022-09-10T08:49:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:13 smithi110 ceph-mon[97633]: pgmap v8321: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:49:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:13 smithi104 ceph-mon[97044]: pgmap v8321: 129 pgs: 129 active+clean; 8.6 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:49:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:14.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:14.032043+0000) 2022-09-10T08:49:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:14.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:14.533750+0000) 2022-09-10T08:49:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:15.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:15.032288+0000) 2022-09-10T08:49:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:15.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:15.533986+0000) 2022-09-10T08:49:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:15 smithi110 ceph-mon[97633]: pgmap v8322: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:49:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:15 smithi104 ceph-mon[97044]: pgmap v8322: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:49:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:16.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:16.032420+0000) 2022-09-10T08:49:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:16.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:16.534198+0000) 2022-09-10T08:49:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:17.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:17.032624+0000) 2022-09-10T08:49:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:17 smithi110 ceph-mon[97633]: pgmap v8323: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 237 B/s wr, 11 op/s 2022-09-10T08:49:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:17 smithi104 ceph-mon[97044]: pgmap v8323: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 237 B/s wr, 11 op/s 2022-09-10T08:49:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:17.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:17.534399+0000) 2022-09-10T08:49:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:18.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:18.032769+0000) 2022-09-10T08:49:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:18.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:18.534587+0000) 2022-09-10T08:49:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:19.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:19.032905+0000) 2022-09-10T08:49:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:19.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:19.534811+0000) 2022-09-10T08:49:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:19 smithi110 ceph-mon[97633]: pgmap v8324: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:49:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:19 smithi104 ceph-mon[97044]: pgmap v8324: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:49:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:20.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:20.033050+0000) 2022-09-10T08:49:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:49:19.980+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:49:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:20.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:20.534997+0000) 2022-09-10T08:49:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:21.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:21.033163+0000) 2022-09-10T08:49:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:21.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:21.535235+0000) 2022-09-10T08:49:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:21 smithi110 ceph-mon[97633]: pgmap v8325: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:49:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:21 smithi104 ceph-mon[97044]: pgmap v8325: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:49:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:22.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:22.033318+0000) 2022-09-10T08:49:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:22.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:22.535428+0000) 2022-09-10T08:49:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:23.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:23.033475+0000) 2022-09-10T08:49:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:23.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:23.535621+0000) 2022-09-10T08:49:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:23 smithi110 ceph-mon[97633]: pgmap v8326: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:49:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:23 smithi104 ceph-mon[97044]: pgmap v8326: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:49:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:24.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:24.033623+0000) 2022-09-10T08:49:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:24.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:24.535805+0000) 2022-09-10T08:49:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:25.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:25.033757+0000) 2022-09-10T08:49:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:25.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:25.536088+0000) 2022-09-10T08:49:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:25 smithi110 ceph-mon[97633]: pgmap v8327: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:49:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:25 smithi104 ceph-mon[97044]: pgmap v8327: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:49:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:26.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:26.033929+0000) 2022-09-10T08:49:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:26.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:26.536356+0000) 2022-09-10T08:49:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:27.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:27.034074+0000) 2022-09-10T08:49:27.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:27 smithi110 ceph-mon[97633]: pgmap v8328: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:27 smithi104 ceph-mon[97044]: pgmap v8328: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:27.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:27.536549+0000) 2022-09-10T08:49:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:28.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:28.034235+0000) 2022-09-10T08:49:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:28.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:28.536805+0000) 2022-09-10T08:49:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:29.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:29.034367+0000) 2022-09-10T08:49:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:29.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:29.537070+0000) 2022-09-10T08:49:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:29 smithi110 ceph-mon[97633]: pgmap v8329: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:29 smithi104 ceph-mon[97044]: pgmap v8329: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:30.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:30.034524+0000) 2022-09-10T08:49:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:30.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:30.537294+0000) 2022-09-10T08:49:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:31.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:31.034647+0000) 2022-09-10T08:49:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:31.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:31.537510+0000) 2022-09-10T08:49:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:31 smithi110 ceph-mon[97633]: pgmap v8330: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:31 smithi104 ceph-mon[97044]: pgmap v8330: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:32.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:32.034800+0000) 2022-09-10T08:49:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:32.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:32.537736+0000) 2022-09-10T08:49:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:33.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:33.034930+0000) 2022-09-10T08:49:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:33.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:33.537962+0000) 2022-09-10T08:49:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:33 smithi110 ceph-mon[97633]: pgmap v8331: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:33 smithi104 ceph-mon[97044]: pgmap v8331: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:34.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:34.035145+0000) 2022-09-10T08:49:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:34.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:34.538216+0000) 2022-09-10T08:49:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:35.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:35.035384+0000) 2022-09-10T08:49:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:49:34.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:49:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:35.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:35.538410+0000) 2022-09-10T08:49:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:35 smithi110 ceph-mon[97633]: pgmap v8332: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:35 smithi104 ceph-mon[97044]: pgmap v8332: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:36.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:36.035553+0000) 2022-09-10T08:49:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:36.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:36.538681+0000) 2022-09-10T08:49:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:37.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:37.035691+0000) 2022-09-10T08:49:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:37 smithi110 ceph-mon[97633]: pgmap v8333: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:37 smithi104 ceph-mon[97044]: pgmap v8333: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:37.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:37.538937+0000) 2022-09-10T08:49:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:38.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:38.035827+0000) 2022-09-10T08:49:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:38.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:38.539192+0000) 2022-09-10T08:49:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:39.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:39.035953+0000) 2022-09-10T08:49:39.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:39.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:39.539396+0000) 2022-09-10T08:49:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:39 smithi110 ceph-mon[97633]: pgmap v8334: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:39 smithi104 ceph-mon[97044]: pgmap v8334: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:40.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:40.036087+0000) 2022-09-10T08:49:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:40.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:40.539659+0000) 2022-09-10T08:49:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:41.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:41.036229+0000) 2022-09-10T08:49:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:41.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:41.539820+0000) 2022-09-10T08:49:42.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:41 smithi110 ceph-mon[97633]: pgmap v8335: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:41 smithi104 ceph-mon[97044]: pgmap v8335: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:42.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:42.036409+0000) 2022-09-10T08:49:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:42.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:42.540070+0000) 2022-09-10T08:49:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:43.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:43.036543+0000) 2022-09-10T08:49:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:43.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:43.540354+0000) 2022-09-10T08:49:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:43 smithi110 ceph-mon[97633]: pgmap v8336: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:43 smithi104 ceph-mon[97044]: pgmap v8336: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:44.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:44.036668+0000) 2022-09-10T08:49:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:44.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:44.540594+0000) 2022-09-10T08:49:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:49:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:49:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:49:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:49:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:49:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:49:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:49:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:49:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:45.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:45.036801+0000) 2022-09-10T08:49:45.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:45.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:45.540833+0000) 2022-09-10T08:49:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:45 smithi110 ceph-mon[97633]: pgmap v8337: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:45 smithi104 ceph-mon[97044]: pgmap v8337: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:46.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:46.036971+0000) 2022-09-10T08:49:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:46.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:46.541071+0000) 2022-09-10T08:49:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:47.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:47.037120+0000) 2022-09-10T08:49:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:47 smithi110 ceph-mon[97633]: pgmap v8338: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:47 smithi104 ceph-mon[97044]: pgmap v8338: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:47.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:47.541347+0000) 2022-09-10T08:49:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:48.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:48.037269+0000) 2022-09-10T08:49:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:48.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:48.541525+0000) 2022-09-10T08:49:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:49.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:49.037393+0000) 2022-09-10T08:49:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:49.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:49.541691+0000) 2022-09-10T08:49:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:49 smithi110 ceph-mon[97633]: pgmap v8339: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:49 smithi104 ceph-mon[97044]: pgmap v8339: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:50.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:49:50.003+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:49:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:50.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:50.037567+0000) 2022-09-10T08:49:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:50.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:50.541911+0000) 2022-09-10T08:49:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:51.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:51.037698+0000) 2022-09-10T08:49:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:51.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:51.542169+0000) 2022-09-10T08:49:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:51 smithi110 ceph-mon[97633]: pgmap v8340: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:51 smithi104 ceph-mon[97044]: pgmap v8340: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:52.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:52.037838+0000) 2022-09-10T08:49:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:52.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:52.542318+0000) 2022-09-10T08:49:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:53.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:53.038042+0000) 2022-09-10T08:49:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:53.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:53.542522+0000) 2022-09-10T08:49:54.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:53 smithi110 ceph-mon[97633]: pgmap v8341: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:53 smithi104 ceph-mon[97044]: pgmap v8341: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:54.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:54.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:54.038191+0000) 2022-09-10T08:49:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:54.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:54.542717+0000) 2022-09-10T08:49:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:55.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:55.038323+0000) 2022-09-10T08:49:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:55.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:55.542905+0000) 2022-09-10T08:49:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:55 smithi110 ceph-mon[97633]: pgmap v8342: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:55 smithi104 ceph-mon[97044]: pgmap v8342: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:49:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:56.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:56.038478+0000) 2022-09-10T08:49:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:56.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:56.543107+0000) 2022-09-10T08:49:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:57.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:57.038604+0000) 2022-09-10T08:49:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:57 smithi110 ceph-mon[97633]: pgmap v8343: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:57 smithi104 ceph-mon[97044]: pgmap v8343: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:49:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:57.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:57.543273+0000) 2022-09-10T08:49:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:58.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:58.038740+0000) 2022-09-10T08:49:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:58.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:58.543475+0000) 2022-09-10T08:49:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:49:59.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:59.038878+0000) 2022-09-10T08:49:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:49:59.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:49:59.543637+0000) 2022-09-10T08:50:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:49:59 smithi110 ceph-mon[97633]: pgmap v8344: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:49:59 smithi104 ceph-mon[97044]: pgmap v8344: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:00.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:00.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:00.039012+0000) 2022-09-10T08:50:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:00.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:00.543867+0000) 2022-09-10T08:50:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:50:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T08:50:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:01.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:01.039149+0000) 2022-09-10T08:50:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:01.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:01.544019+0000) 2022-09-10T08:50:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:01 smithi110 ceph-mon[97633]: pgmap v8345: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:01 smithi104 ceph-mon[97044]: pgmap v8345: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:02.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:02.039285+0000) 2022-09-10T08:50:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:02.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:02.544185+0000) 2022-09-10T08:50:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:03.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:03.039415+0000) 2022-09-10T08:50:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:03.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:03.544368+0000) 2022-09-10T08:50:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:03 smithi110 ceph-mon[97633]: pgmap v8346: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:03 smithi104 ceph-mon[97044]: pgmap v8346: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:04.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:04.039534+0000) 2022-09-10T08:50:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:04.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:04.544585+0000) 2022-09-10T08:50:05.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:50:05.018+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:50:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:05.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:05.039652+0000) 2022-09-10T08:50:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:05.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:05.544788+0000) 2022-09-10T08:50:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:05 smithi110 ceph-mon[97633]: pgmap v8347: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:50:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:05 smithi104 ceph-mon[97044]: pgmap v8347: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:50:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:06.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:06.039803+0000) 2022-09-10T08:50:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:06.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:06.544973+0000) 2022-09-10T08:50:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:07.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:07.039926+0000) 2022-09-10T08:50:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:07 smithi110 ceph-mon[97633]: pgmap v8348: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:50:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:07 smithi104 ceph-mon[97044]: pgmap v8348: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:50:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:07.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:07.545203+0000) 2022-09-10T08:50:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:08.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:08.040055+0000) 2022-09-10T08:50:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:08.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:08.545412+0000) 2022-09-10T08:50:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:09.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:09.040159+0000) 2022-09-10T08:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:09 smithi110 ceph-mon[97633]: osdmap e602: 12 total, 12 up, 12 in 2022-09-10T08:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:09 smithi110 ceph-mon[97633]: pgmap v8350: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:50:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:09 smithi104 ceph-mon[97044]: osdmap e602: 12 total, 12 up, 12 in 2022-09-10T08:50:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:09 smithi104 ceph-mon[97044]: pgmap v8350: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:50:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:09.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:09.545633+0000) 2022-09-10T08:50:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:10.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:10.040288+0000) 2022-09-10T08:50:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:10.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:10.545887+0000) 2022-09-10T08:50:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:11.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:11.040417+0000) 2022-09-10T08:50:11.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:11.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:11.546181+0000) 2022-09-10T08:50:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:11 smithi110 ceph-mon[97633]: pgmap v8351: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:50:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:11 smithi110 ceph-mon[97633]: osdmap e603: 12 total, 12 up, 12 in 2022-09-10T08:50:12.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:11 smithi104 ceph-mon[97044]: pgmap v8351: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:50:12.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:11 smithi104 ceph-mon[97044]: osdmap e603: 12 total, 12 up, 12 in 2022-09-10T08:50:12.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:12.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:12.040573+0000) 2022-09-10T08:50:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:12.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:12.546418+0000) 2022-09-10T08:50:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:13.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:13.040701+0000) 2022-09-10T08:50:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:13.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:13.546643+0000) 2022-09-10T08:50:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:13 smithi110 ceph-mon[97633]: pgmap v8353: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:50:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:13 smithi104 ceph-mon[97044]: pgmap v8353: 129 pgs: 129 active+clean; 8.6 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:50:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:14.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:14.040837+0000) 2022-09-10T08:50:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:14.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:14.546896+0000) 2022-09-10T08:50:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:15.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:15.040984+0000) 2022-09-10T08:50:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:15.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:15.547059+0000) 2022-09-10T08:50:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:15 smithi110 ceph-mon[97633]: pgmap v8354: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:50:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:15 smithi104 ceph-mon[97044]: pgmap v8354: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:50:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:16.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:16.041140+0000) 2022-09-10T08:50:16.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:16.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:16.547312+0000) 2022-09-10T08:50:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:17.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:17.041327+0000) 2022-09-10T08:50:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:17 smithi110 ceph-mon[97633]: pgmap v8355: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:50:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:17 smithi104 ceph-mon[97044]: pgmap v8355: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:50:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:17.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:17.547497+0000) 2022-09-10T08:50:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:18.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:18.041462+0000) 2022-09-10T08:50:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:18.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:18.547690+0000) 2022-09-10T08:50:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:19.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:19.041607+0000) 2022-09-10T08:50:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:19.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:19.547959+0000) 2022-09-10T08:50:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:19 smithi110 ceph-mon[97633]: pgmap v8356: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:50:20.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:19 smithi104 ceph-mon[97044]: pgmap v8356: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:50:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:50:20.029+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:50:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:20.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:20.041754+0000) 2022-09-10T08:50:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:20.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:20.548204+0000) 2022-09-10T08:50:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:21.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:21.041874+0000) 2022-09-10T08:50:21.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:21.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:21.548368+0000) 2022-09-10T08:50:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:21 smithi110 ceph-mon[97633]: pgmap v8357: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:50:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:21 smithi104 ceph-mon[97044]: pgmap v8357: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:50:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:22.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:22.042041+0000) 2022-09-10T08:50:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:22.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:22.548554+0000) 2022-09-10T08:50:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:23.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:23.042165+0000) 2022-09-10T08:50:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:23.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:23.548758+0000) 2022-09-10T08:50:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:23 smithi110 ceph-mon[97633]: pgmap v8358: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:50:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:23 smithi104 ceph-mon[97044]: pgmap v8358: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:50:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:24.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:24.042318+0000) 2022-09-10T08:50:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:24.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:24.549002+0000) 2022-09-10T08:50:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:25.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:25.042450+0000) 2022-09-10T08:50:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:25.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:25.549182+0000) 2022-09-10T08:50:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:25 smithi110 ceph-mon[97633]: pgmap v8359: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:50:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:25 smithi104 ceph-mon[97044]: pgmap v8359: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:50:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:26.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:26.042588+0000) 2022-09-10T08:50:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:26.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:26.549449+0000) 2022-09-10T08:50:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:27.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:27.042772+0000) 2022-09-10T08:50:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:27 smithi110 ceph-mon[97633]: pgmap v8360: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:50:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:27 smithi104 ceph-mon[97044]: pgmap v8360: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:50:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:27.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:27.549698+0000) 2022-09-10T08:50:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:28.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:28.042912+0000) 2022-09-10T08:50:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:28.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:28.549878+0000) 2022-09-10T08:50:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:29.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:29.043073+0000) 2022-09-10T08:50:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:29.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:29.550079+0000) 2022-09-10T08:50:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:29 smithi110 ceph-mon[97633]: pgmap v8361: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:50:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:29 smithi104 ceph-mon[97044]: pgmap v8361: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:50:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:30.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:30.043226+0000) 2022-09-10T08:50:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:30.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:30.550321+0000) 2022-09-10T08:50:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:31.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:31.043434+0000) 2022-09-10T08:50:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:31.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:31.550542+0000) 2022-09-10T08:50:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:31 smithi110 ceph-mon[97633]: pgmap v8362: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:31 smithi104 ceph-mon[97044]: pgmap v8362: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:32.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:32.043578+0000) 2022-09-10T08:50:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:32.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:32.550744+0000) 2022-09-10T08:50:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:33.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:33.043702+0000) 2022-09-10T08:50:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:33.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:33.550994+0000) 2022-09-10T08:50:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:33 smithi110 ceph-mon[97633]: pgmap v8363: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:33 smithi104 ceph-mon[97044]: pgmap v8363: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:34.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:34.043831+0000) 2022-09-10T08:50:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:34.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:34.551222+0000) 2022-09-10T08:50:35.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:35.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:35.043968+0000) 2022-09-10T08:50:35.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:50:35.040+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:50:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:35.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:35.551478+0000) 2022-09-10T08:50:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:35 smithi110 ceph-mon[97633]: pgmap v8364: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:35 smithi104 ceph-mon[97044]: pgmap v8364: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:36.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:36.044118+0000) 2022-09-10T08:50:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:36.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:36.551716+0000) 2022-09-10T08:50:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:37.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:37.044245+0000) 2022-09-10T08:50:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:37 smithi110 ceph-mon[97633]: pgmap v8365: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:37 smithi104 ceph-mon[97044]: pgmap v8365: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:37.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:37.551915+0000) 2022-09-10T08:50:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:38.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:38.044382+0000) 2022-09-10T08:50:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:38.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:38.552162+0000) 2022-09-10T08:50:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:39.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:39.044519+0000) 2022-09-10T08:50:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:39.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:39.552331+0000) 2022-09-10T08:50:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:39 smithi110 ceph-mon[97633]: pgmap v8366: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:39 smithi104 ceph-mon[97044]: pgmap v8366: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:40.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:40.044655+0000) 2022-09-10T08:50:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:40.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:40.552512+0000) 2022-09-10T08:50:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:41.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:41.044791+0000) 2022-09-10T08:50:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:41.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:41.552733+0000) 2022-09-10T08:50:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:41 smithi110 ceph-mon[97633]: pgmap v8367: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:41 smithi104 ceph-mon[97044]: pgmap v8367: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:42.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:42.044960+0000) 2022-09-10T08:50:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:42.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:42.552877+0000) 2022-09-10T08:50:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:43.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:43.045108+0000) 2022-09-10T08:50:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:43.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:43.553028+0000) 2022-09-10T08:50:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:43 smithi110 ceph-mon[97633]: pgmap v8368: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:44.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:43 smithi104 ceph-mon[97044]: pgmap v8368: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:44.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:44.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:44.045273+0000) 2022-09-10T08:50:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:44.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:44.553244+0000) 2022-09-10T08:50:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:50:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:50:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:50:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:50:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:50:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:50:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:50:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:50:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:45.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:45.045411+0000) 2022-09-10T08:50:46.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:45.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:45.553500+0000) 2022-09-10T08:50:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:45 smithi110 ceph-mon[97633]: pgmap v8369: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:45 smithi104 ceph-mon[97044]: pgmap v8369: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:46.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:46.045658+0000) 2022-09-10T08:50:47.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:46.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:46.553782+0000) 2022-09-10T08:50:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:47.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:47.045803+0000) 2022-09-10T08:50:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:47 smithi110 ceph-mon[97633]: pgmap v8370: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:47 smithi104 ceph-mon[97044]: pgmap v8370: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:48.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:47.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:47.553956+0000) 2022-09-10T08:50:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:48.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:48.046025+0000) 2022-09-10T08:50:49.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:48.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:48.554193+0000) 2022-09-10T08:50:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:49.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:49.046155+0000) 2022-09-10T08:50:50.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:49.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:49.554381+0000) 2022-09-10T08:50:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:49 smithi110 ceph-mon[97633]: pgmap v8371: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:49 smithi104 ceph-mon[97044]: pgmap v8371: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:50.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:50.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:50.046281+0000) 2022-09-10T08:50:50.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:50:50.055+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:50:51.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:50.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:50.554606+0000) 2022-09-10T08:50:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:51.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:51.046413+0000) 2022-09-10T08:50:52.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:51.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:51.554850+0000) 2022-09-10T08:50:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:51 smithi110 ceph-mon[97633]: pgmap v8372: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:51 smithi104 ceph-mon[97044]: pgmap v8372: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:52.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:52.046556+0000) 2022-09-10T08:50:53.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:52.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:52.555083+0000) 2022-09-10T08:50:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:53.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:53.046682+0000) 2022-09-10T08:50:54.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:53.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:53.555279+0000) 2022-09-10T08:50:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:53 smithi110 ceph-mon[97633]: pgmap v8373: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:53 smithi104 ceph-mon[97044]: pgmap v8373: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:54.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:54.046875+0000) 2022-09-10T08:50:55.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:54.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:54.555532+0000) 2022-09-10T08:50:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:55.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:55.047009+0000) 2022-09-10T08:50:56.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:55.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:55.555661+0000) 2022-09-10T08:50:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:55 smithi110 ceph-mon[97633]: pgmap v8374: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:55 smithi104 ceph-mon[97044]: pgmap v8374: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:50:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:56.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:56.047206+0000) 2022-09-10T08:50:57.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:56.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:56.555913+0000) 2022-09-10T08:50:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:57.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:57.047336+0000) 2022-09-10T08:50:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:57 smithi110 ceph-mon[97633]: pgmap v8375: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:57 smithi104 ceph-mon[97044]: pgmap v8375: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:50:58.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:57.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:57.556147+0000) 2022-09-10T08:50:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:58.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:58.047478+0000) 2022-09-10T08:50:59.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:58.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:58.556325+0000) 2022-09-10T08:50:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:50:59.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:59.047614+0000) 2022-09-10T08:51:00.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:50:59.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:50:59.556548+0000) 2022-09-10T08:51:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:50:59 smithi110 ceph-mon[97633]: pgmap v8376: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:50:59 smithi104 ceph-mon[97044]: pgmap v8376: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:00.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:00.047765+0000) 2022-09-10T08:51:01.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:00.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:00.556719+0000) 2022-09-10T08:51:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:01.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:01.047894+0000) 2022-09-10T08:51:02.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:01.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:01.556939+0000) 2022-09-10T08:51:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:01 smithi110 ceph-mon[97633]: pgmap v8377: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:01 smithi104 ceph-mon[97044]: pgmap v8377: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:02.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:02.048030+0000) 2022-09-10T08:51:03.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:02.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:02.557194+0000) 2022-09-10T08:51:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:03.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:03.048181+0000) 2022-09-10T08:51:04.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:03.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:03.557330+0000) 2022-09-10T08:51:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:03 smithi110 ceph-mon[97633]: pgmap v8378: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:03 smithi104 ceph-mon[97044]: pgmap v8378: 129 pgs: 129 active+clean; 8.6 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:04.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:04.048314+0000) 2022-09-10T08:51:05.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:04.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:04.557576+0000) 2022-09-10T08:51:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:51:05.066+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:51:05.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:05.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:05.048460+0000) 2022-09-10T08:51:06.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:05.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:05.557824+0000) 2022-09-10T08:51:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:05 smithi110 ceph-mon[97633]: pgmap v8379: 129 pgs: 129 active+clean; 8.6 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:51:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:05 smithi104 ceph-mon[97044]: pgmap v8379: 129 pgs: 129 active+clean; 8.6 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:51:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:06.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:06.048681+0000) 2022-09-10T08:51:07.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:06.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:06.558043+0000) 2022-09-10T08:51:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:07.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:07.048915+0000) 2022-09-10T08:51:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:07 smithi110 ceph-mon[97633]: pgmap v8380: 129 pgs: 129 active+clean; 8.6 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:51:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:07 smithi104 ceph-mon[97044]: pgmap v8380: 129 pgs: 129 active+clean; 8.6 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:51:08.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:07.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:07.558218+0000) 2022-09-10T08:51:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:08.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:08.049079+0000) 2022-09-10T08:51:09.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:08.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:08.558448+0000) 2022-09-10T08:51:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:09.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:09.049216+0000) 2022-09-10T08:51:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:09 smithi110 ceph-mon[97633]: osdmap e604: 12 total, 12 up, 12 in 2022-09-10T08:51:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:09 smithi110 ceph-mon[97633]: pgmap v8382: 129 pgs: 129 active+clean; 8.6 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:51:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:09 smithi104 ceph-mon[97044]: osdmap e604: 12 total, 12 up, 12 in 2022-09-10T08:51:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:09 smithi104 ceph-mon[97044]: pgmap v8382: 129 pgs: 129 active+clean; 8.6 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:51:10.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:09.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:09.558630+0000) 2022-09-10T08:51:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:10.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:10.049350+0000) 2022-09-10T08:51:11.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:10.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:10.558874+0000) 2022-09-10T08:51:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:11.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:11.049482+0000) 2022-09-10T08:51:12.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:11.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:11.559077+0000) 2022-09-10T08:51:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:11 smithi110 ceph-mon[97633]: pgmap v8383: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:51:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:11 smithi110 ceph-mon[97633]: osdmap e605: 12 total, 12 up, 12 in 2022-09-10T08:51:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:11 smithi104 ceph-mon[97044]: pgmap v8383: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:51:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:11 smithi104 ceph-mon[97044]: osdmap e605: 12 total, 12 up, 12 in 2022-09-10T08:51:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:12.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:12.049635+0000) 2022-09-10T08:51:13.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:12.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:12.559339+0000) 2022-09-10T08:51:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:13.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:13.049768+0000) 2022-09-10T08:51:14.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:13.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:13.559599+0000) 2022-09-10T08:51:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:13 smithi110 ceph-mon[97633]: pgmap v8385: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:51:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:13 smithi104 ceph-mon[97044]: pgmap v8385: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:51:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:14.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:14.049894+0000) 2022-09-10T08:51:15.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:14.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:14.559817+0000) 2022-09-10T08:51:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:15.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:15.050031+0000) 2022-09-10T08:51:16.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:15.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:15.560074+0000) 2022-09-10T08:51:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:15 smithi110 ceph-mon[97633]: pgmap v8386: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:51:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:15 smithi104 ceph-mon[97044]: pgmap v8386: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:51:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:16.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:16.050198+0000) 2022-09-10T08:51:17.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:16.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:16.560328+0000) 2022-09-10T08:51:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:17.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:17.050367+0000) 2022-09-10T08:51:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:17 smithi110 ceph-mon[97633]: pgmap v8387: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:51:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:17 smithi104 ceph-mon[97044]: pgmap v8387: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:51:18.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:17.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:17.560572+0000) 2022-09-10T08:51:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:18.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:18.050523+0000) 2022-09-10T08:51:19.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:18.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:18.560767+0000) 2022-09-10T08:51:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:19.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:19.050652+0000) 2022-09-10T08:51:20.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:19.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:19.561016+0000) 2022-09-10T08:51:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:19 smithi110 ceph-mon[97633]: pgmap v8388: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:51:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:19 smithi104 ceph-mon[97044]: pgmap v8388: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:51:20.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:51:20.079+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:51:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:20.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:20.050780+0000) 2022-09-10T08:51:21.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:20.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:20.561211+0000) 2022-09-10T08:51:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:21.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:21.050912+0000) 2022-09-10T08:51:22.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:21.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:21.561456+0000) 2022-09-10T08:51:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:21 smithi110 ceph-mon[97633]: pgmap v8389: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:51:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:21 smithi104 ceph-mon[97044]: pgmap v8389: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:51:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:22.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:22.051047+0000) 2022-09-10T08:51:23.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:22.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:22.561657+0000) 2022-09-10T08:51:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:23.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:23.051190+0000) 2022-09-10T08:51:24.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:23.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:23.561827+0000) 2022-09-10T08:51:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:23 smithi110 ceph-mon[97633]: pgmap v8390: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:51:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:23 smithi104 ceph-mon[97044]: pgmap v8390: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:51:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:24.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:24.051359+0000) 2022-09-10T08:51:25.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:24.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:24.562034+0000) 2022-09-10T08:51:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:25.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:25.051532+0000) 2022-09-10T08:51:26.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:25.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:25.562277+0000) 2022-09-10T08:51:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:25 smithi110 ceph-mon[97633]: pgmap v8391: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:51:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:25 smithi104 ceph-mon[97044]: pgmap v8391: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:51:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:26.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:26.051682+0000) 2022-09-10T08:51:27.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:26.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:26.562538+0000) 2022-09-10T08:51:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:27.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:27.051812+0000) 2022-09-10T08:51:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:27 smithi110 ceph-mon[97633]: pgmap v8392: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:27 smithi104 ceph-mon[97044]: pgmap v8392: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:28.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:27.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:27.562787+0000) 2022-09-10T08:51:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:28.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:28.051943+0000) 2022-09-10T08:51:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:28.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:28.562994+0000) 2022-09-10T08:51:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:29.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:29.052077+0000) 2022-09-10T08:51:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:29.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:29.563222+0000) 2022-09-10T08:51:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:29 smithi110 ceph-mon[97633]: pgmap v8393: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:29 smithi104 ceph-mon[97044]: pgmap v8393: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:30.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:30.052227+0000) 2022-09-10T08:51:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:30.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:30.563359+0000) 2022-09-10T08:51:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:31.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:31.052363+0000) 2022-09-10T08:51:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:31.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:31.563516+0000) 2022-09-10T08:51:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:31 smithi110 ceph-mon[97633]: pgmap v8394: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:31 smithi104 ceph-mon[97044]: pgmap v8394: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:32.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:32.052497+0000) 2022-09-10T08:51:33.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:32.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:32.563758+0000) 2022-09-10T08:51:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:33.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:33.052608+0000) 2022-09-10T08:51:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:33.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:33.563988+0000) 2022-09-10T08:51:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:33 smithi110 ceph-mon[97633]: pgmap v8395: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:33 smithi104 ceph-mon[97044]: pgmap v8395: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:34.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:34.052730+0000) 2022-09-10T08:51:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:34.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:34.564242+0000) 2022-09-10T08:51:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:35.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:35.052857+0000) 2022-09-10T08:51:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:51:35.088+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:51:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:35.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:35.564437+0000) 2022-09-10T08:51:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:35 smithi110 ceph-mon[97633]: pgmap v8396: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:35 smithi104 ceph-mon[97044]: pgmap v8396: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:36.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:36.053007+0000) 2022-09-10T08:51:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:36.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:36.564629+0000) 2022-09-10T08:51:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:37.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:37.053154+0000) 2022-09-10T08:51:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:37 smithi110 ceph-mon[97633]: pgmap v8397: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:37 smithi104 ceph-mon[97044]: pgmap v8397: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:37.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:37.564863+0000) 2022-09-10T08:51:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:38.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:38.053299+0000) 2022-09-10T08:51:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:38.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:38.565135+0000) 2022-09-10T08:51:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:39.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:39.053432+0000) 2022-09-10T08:51:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:39.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:39.565321+0000) 2022-09-10T08:51:40.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:39 smithi110 ceph-mon[97633]: pgmap v8398: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:39 smithi104 ceph-mon[97044]: pgmap v8398: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:40.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:40.053568+0000) 2022-09-10T08:51:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:40.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:40.565574+0000) 2022-09-10T08:51:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:41.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:41.053765+0000) 2022-09-10T08:51:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:41.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:41.565797+0000) 2022-09-10T08:51:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:41 smithi110 ceph-mon[97633]: pgmap v8399: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:41 smithi104 ceph-mon[97044]: pgmap v8399: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:42.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:42.053917+0000) 2022-09-10T08:51:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:42.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:42.565956+0000) 2022-09-10T08:51:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:43.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:43.054089+0000) 2022-09-10T08:51:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:43.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:43.566148+0000) 2022-09-10T08:51:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:43 smithi110 ceph-mon[97633]: pgmap v8400: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:43 smithi104 ceph-mon[97044]: pgmap v8400: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:44.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:44.054249+0000) 2022-09-10T08:51:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:44.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:44.566395+0000) 2022-09-10T08:51:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:51:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:51:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:51:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:51:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:51:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:51:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:51:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:51:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:45.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:45.054387+0000) 2022-09-10T08:51:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:45.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:45.566586+0000) 2022-09-10T08:51:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:45 smithi110 ceph-mon[97633]: pgmap v8401: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:45 smithi104 ceph-mon[97044]: pgmap v8401: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:46.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:46.054521+0000) 2022-09-10T08:51:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:46.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:46.566778+0000) 2022-09-10T08:51:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:47 smithi104 ceph-mon[97044]: pgmap v8402: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:47.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:47.054644+0000) 2022-09-10T08:51:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:47 smithi110 ceph-mon[97633]: pgmap v8402: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:47.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:47.566915+0000) 2022-09-10T08:51:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:48.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:48.054765+0000) 2022-09-10T08:51:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:48.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:48.567179+0000) 2022-09-10T08:51:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:49.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:49.054880+0000) 2022-09-10T08:51:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:49.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:49.567412+0000) 2022-09-10T08:51:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:49 smithi110 ceph-mon[97633]: pgmap v8403: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:49 smithi104 ceph-mon[97044]: pgmap v8403: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:50.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:51:50.089+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:51:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:50.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:50.055011+0000) 2022-09-10T08:51:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:50.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:50.567612+0000) 2022-09-10T08:51:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:51.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:51.055152+0000) 2022-09-10T08:51:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:51.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:51.567829+0000) 2022-09-10T08:51:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:51 smithi110 ceph-mon[97633]: pgmap v8404: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:52.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:51 smithi104 ceph-mon[97044]: pgmap v8404: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:51:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:52.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:52.055297+0000) 2022-09-10T08:51:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:52.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:52.568088+0000) 2022-09-10T08:51:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:53.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:53.055425+0000) 2022-09-10T08:51:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:53.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:53.568338+0000) 2022-09-10T08:51:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:53 smithi110 ceph-mon[97633]: pgmap v8405: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:53 smithi104 ceph-mon[97044]: pgmap v8405: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:51:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:54.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:54.055555+0000) 2022-09-10T08:51:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:54.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:54.568544+0000) 2022-09-10T08:51:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:55.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:55.055687+0000) 2022-09-10T08:51:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:55.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:55.568793+0000) 2022-09-10T08:51:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:55 smithi110 ceph-mon[97633]: pgmap v8406: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T08:51:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:55 smithi104 ceph-mon[97044]: pgmap v8406: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T08:51:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:56.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:56.055820+0000) 2022-09-10T08:51:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:56.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:56.569068+0000) 2022-09-10T08:51:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:57 smithi110 ceph-mon[97633]: pgmap v8407: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:51:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:57 smithi104 ceph-mon[97044]: pgmap v8407: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:51:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:57.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:57.055939+0000) 2022-09-10T08:51:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:57.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:57.569291+0000) 2022-09-10T08:51:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:58.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:58.056049+0000) 2022-09-10T08:51:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:58.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:58.569559+0000) 2022-09-10T08:51:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:51:59.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:59.056158+0000) 2022-09-10T08:52:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:51:59.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:51:59.569836+0000) 2022-09-10T08:52:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:51:59 smithi110 ceph-mon[97633]: pgmap v8408: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:52:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:51:59 smithi104 ceph-mon[97044]: pgmap v8408: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:52:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:00.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:00.056284+0000) 2022-09-10T08:52:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:00.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:00.570048+0000) 2022-09-10T08:52:01.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:01.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:01.056404+0000) 2022-09-10T08:52:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:01.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:01.570279+0000) 2022-09-10T08:52:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:01 smithi110 ceph-mon[97633]: pgmap v8409: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T08:52:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:01 smithi104 ceph-mon[97044]: pgmap v8409: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T08:52:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:02.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:02.056530+0000) 2022-09-10T08:52:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:02.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:02.570487+0000) 2022-09-10T08:52:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:03.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:03.056651+0000) 2022-09-10T08:52:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:03.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:03.570738+0000) 2022-09-10T08:52:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:03 smithi110 ceph-mon[97633]: pgmap v8410: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:52:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:03 smithi104 ceph-mon[97044]: pgmap v8410: 129 pgs: 129 active+clean; 8.6 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T08:52:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:04.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:04.056774+0000) 2022-09-10T08:52:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:04.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:04.570990+0000) 2022-09-10T08:52:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:52:05.099+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:52:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:05.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:05.056898+0000) 2022-09-10T08:52:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:05.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:05.571237+0000) 2022-09-10T08:52:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:05 smithi110 ceph-mon[97633]: pgmap v8411: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T08:52:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:05 smithi104 ceph-mon[97044]: pgmap v8411: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T08:52:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:06.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:06.057024+0000) 2022-09-10T08:52:07.029 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:06.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:06.571421+0000) 2022-09-10T08:52:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:07 smithi104 ceph-mon[97044]: pgmap v8412: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:52:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:07.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:07.057159+0000) 2022-09-10T08:52:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:07 smithi110 ceph-mon[97633]: pgmap v8412: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:52:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:07.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:07.571619+0000) 2022-09-10T08:52:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:08.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:08.057282+0000) 2022-09-10T08:52:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:08.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:08.571791+0000) 2022-09-10T08:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:09 smithi104 ceph-mon[97044]: osdmap e606: 12 total, 12 up, 12 in 2022-09-10T08:52:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:09 smithi104 ceph-mon[97044]: pgmap v8414: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:52:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:09.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:09.057405+0000) 2022-09-10T08:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:09 smithi110 ceph-mon[97633]: osdmap e606: 12 total, 12 up, 12 in 2022-09-10T08:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:09 smithi110 ceph-mon[97633]: pgmap v8414: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:52:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:09.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:09.571970+0000) 2022-09-10T08:52:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:10.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:10.057532+0000) 2022-09-10T08:52:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:10.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:10.572181+0000) 2022-09-10T08:52:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:11.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:11.057655+0000) 2022-09-10T08:52:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:11.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:11.572459+0000) 2022-09-10T08:52:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:11 smithi110 ceph-mon[97633]: pgmap v8415: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:52:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:11 smithi110 ceph-mon[97633]: osdmap e607: 12 total, 12 up, 12 in 2022-09-10T08:52:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:11 smithi104 ceph-mon[97044]: pgmap v8415: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:52:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:11 smithi104 ceph-mon[97044]: osdmap e607: 12 total, 12 up, 12 in 2022-09-10T08:52:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:12.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:12.057790+0000) 2022-09-10T08:52:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:12.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:12.572681+0000) 2022-09-10T08:52:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:13.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:13.057912+0000) 2022-09-10T08:52:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:13.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:13.572950+0000) 2022-09-10T08:52:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:13 smithi110 ceph-mon[97633]: pgmap v8417: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:52:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:13 smithi104 ceph-mon[97044]: pgmap v8417: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:52:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:14.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:14.058076+0000) 2022-09-10T08:52:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:14.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:14.573189+0000) 2022-09-10T08:52:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:15.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:15.058291+0000) 2022-09-10T08:52:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:15.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:15.573387+0000) 2022-09-10T08:52:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:15 smithi110 ceph-mon[97633]: pgmap v8418: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:52:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:15 smithi104 ceph-mon[97044]: pgmap v8418: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:52:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:16.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:16.058431+0000) 2022-09-10T08:52:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:16.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:16.573600+0000) 2022-09-10T08:52:17.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:17 smithi104 ceph-mon[97044]: pgmap v8419: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:52:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:17.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:17.058554+0000) 2022-09-10T08:52:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:17 smithi110 ceph-mon[97633]: pgmap v8419: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T08:52:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:17.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:17.573783+0000) 2022-09-10T08:52:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:18.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:18.058695+0000) 2022-09-10T08:52:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:18.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:18.574031+0000) 2022-09-10T08:52:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:19.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:19.058820+0000) 2022-09-10T08:52:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:19.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:19.574296+0000) 2022-09-10T08:52:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:19 smithi110 ceph-mon[97633]: pgmap v8420: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:52:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:19 smithi104 ceph-mon[97044]: pgmap v8420: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:52:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:20.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:20.058943+0000) 2022-09-10T08:52:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:52:20.116+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:52:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:20.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:20.574529+0000) 2022-09-10T08:52:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:21.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:21.059127+0000) 2022-09-10T08:52:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:21.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:21.574800+0000) 2022-09-10T08:52:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:21 smithi110 ceph-mon[97633]: pgmap v8421: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:52:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:21 smithi104 ceph-mon[97044]: pgmap v8421: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:52:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:22.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:22.059322+0000) 2022-09-10T08:52:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:22.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:22.575069+0000) 2022-09-10T08:52:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:23.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:23.059463+0000) 2022-09-10T08:52:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:23.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:23.575312+0000) 2022-09-10T08:52:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:23 smithi110 ceph-mon[97633]: pgmap v8422: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:52:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:23 smithi104 ceph-mon[97044]: pgmap v8422: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:52:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:24.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:24.059644+0000) 2022-09-10T08:52:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:24.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:24.575534+0000) 2022-09-10T08:52:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:25.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:25.059820+0000) 2022-09-10T08:52:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:25.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:25.575717+0000) 2022-09-10T08:52:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:25 smithi110 ceph-mon[97633]: pgmap v8423: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:52:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:25 smithi104 ceph-mon[97044]: pgmap v8423: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:52:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:26.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:26.059993+0000) 2022-09-10T08:52:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:26.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:26.575959+0000) 2022-09-10T08:52:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:27 smithi104 ceph-mon[97044]: pgmap v8424: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:27.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:27.060235+0000) 2022-09-10T08:52:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:27 smithi110 ceph-mon[97633]: pgmap v8424: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:27.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:27.576169+0000) 2022-09-10T08:52:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:28.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:28.060398+0000) 2022-09-10T08:52:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:28.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:28.576401+0000) 2022-09-10T08:52:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:29.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:29.060528+0000) 2022-09-10T08:52:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:29.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:29.576644+0000) 2022-09-10T08:52:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:29 smithi110 ceph-mon[97633]: pgmap v8425: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:29 smithi104 ceph-mon[97044]: pgmap v8425: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:30.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:30.060703+0000) 2022-09-10T08:52:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:30.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:30.576842+0000) 2022-09-10T08:52:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:31.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:31.060843+0000) 2022-09-10T08:52:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:31.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:31.577043+0000) 2022-09-10T08:52:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:31 smithi110 ceph-mon[97633]: pgmap v8426: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:31 smithi104 ceph-mon[97044]: pgmap v8426: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:32.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:32.061022+0000) 2022-09-10T08:52:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:32.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:32.577262+0000) 2022-09-10T08:52:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:33.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:33.061186+0000) 2022-09-10T08:52:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:33.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:33.577506+0000) 2022-09-10T08:52:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:33 smithi110 ceph-mon[97633]: pgmap v8427: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:33 smithi104 ceph-mon[97044]: pgmap v8427: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:34.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:34.061360+0000) 2022-09-10T08:52:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:34.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:34.577734+0000) 2022-09-10T08:52:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:35.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:35.061534+0000) 2022-09-10T08:52:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:52:35.125+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:52:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:35.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:35.577955+0000) 2022-09-10T08:52:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:35 smithi110 ceph-mon[97633]: pgmap v8428: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:35 smithi104 ceph-mon[97044]: pgmap v8428: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:36.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:36.061746+0000) 2022-09-10T08:52:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:36.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:36.578205+0000) 2022-09-10T08:52:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:37 smithi104 ceph-mon[97044]: pgmap v8429: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:37.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:37.061940+0000) 2022-09-10T08:52:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:37 smithi110 ceph-mon[97633]: pgmap v8429: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:37.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:37.578393+0000) 2022-09-10T08:52:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:38.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:38.062140+0000) 2022-09-10T08:52:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:38.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:38.578643+0000) 2022-09-10T08:52:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:39.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:39.062313+0000) 2022-09-10T08:52:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:39.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:39.578877+0000) 2022-09-10T08:52:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:39 smithi110 ceph-mon[97633]: pgmap v8430: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:39 smithi104 ceph-mon[97044]: pgmap v8430: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:40.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:40.062498+0000) 2022-09-10T08:52:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:40.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:40.579087+0000) 2022-09-10T08:52:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:41.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:41.062687+0000) 2022-09-10T08:52:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:41.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:41.579274+0000) 2022-09-10T08:52:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:41 smithi110 ceph-mon[97633]: pgmap v8431: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:41 smithi104 ceph-mon[97044]: pgmap v8431: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:42.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:42.062899+0000) 2022-09-10T08:52:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:42.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:42.579529+0000) 2022-09-10T08:52:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:43.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:43.063068+0000) 2022-09-10T08:52:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:43.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:43.579740+0000) 2022-09-10T08:52:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:43 smithi110 ceph-mon[97633]: pgmap v8432: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:43 smithi104 ceph-mon[97044]: pgmap v8432: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:44.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:44.063310+0000) 2022-09-10T08:52:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:44.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:44.579902+0000) 2022-09-10T08:52:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:52:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:52:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:52:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:52:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:52:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:52:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:52:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:52:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:45.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:45.063526+0000) 2022-09-10T08:52:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:45.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:45.580187+0000) 2022-09-10T08:52:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:45 smithi110 ceph-mon[97633]: pgmap v8433: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:45 smithi104 ceph-mon[97044]: pgmap v8433: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:46.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:46.063716+0000) 2022-09-10T08:52:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:46.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:46.580371+0000) 2022-09-10T08:52:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:47 smithi104 ceph-mon[97044]: pgmap v8434: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:47.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:47.063883+0000) 2022-09-10T08:52:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:47 smithi110 ceph-mon[97633]: pgmap v8434: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:47.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:47.580602+0000) 2022-09-10T08:52:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:48.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:48.064089+0000) 2022-09-10T08:52:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:48.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:48.580819+0000) 2022-09-10T08:52:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:49.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:49.064270+0000) 2022-09-10T08:52:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:49.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:49.580994+0000) 2022-09-10T08:52:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:49 smithi110 ceph-mon[97633]: pgmap v8435: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:49 smithi104 ceph-mon[97044]: pgmap v8435: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:50.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:50.064484+0000) 2022-09-10T08:52:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:52:50.136+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:52:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:50.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:50.581204+0000) 2022-09-10T08:52:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:51.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:51.064677+0000) 2022-09-10T08:52:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:51.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:51.581424+0000) 2022-09-10T08:52:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:51 smithi110 ceph-mon[97633]: pgmap v8436: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:51 smithi104 ceph-mon[97044]: pgmap v8436: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:52.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:52.064954+0000) 2022-09-10T08:52:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:52.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:52.581613+0000) 2022-09-10T08:52:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:53.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:53.065118+0000) 2022-09-10T08:52:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:53.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:53.581824+0000) 2022-09-10T08:52:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:53 smithi110 ceph-mon[97633]: pgmap v8437: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:53 smithi104 ceph-mon[97044]: pgmap v8437: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:54.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:54.065351+0000) 2022-09-10T08:52:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:54.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:54.582020+0000) 2022-09-10T08:52:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:55.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:55.065555+0000) 2022-09-10T08:52:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:55.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:55.582289+0000) 2022-09-10T08:52:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:55 smithi110 ceph-mon[97633]: pgmap v8438: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:55 smithi104 ceph-mon[97044]: pgmap v8438: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:52:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:56.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:56.065757+0000) 2022-09-10T08:52:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:56.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:56.582492+0000) 2022-09-10T08:52:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:57 smithi104 ceph-mon[97044]: pgmap v8439: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:57.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:57.065930+0000) 2022-09-10T08:52:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:57 smithi110 ceph-mon[97633]: pgmap v8439: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:52:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:57.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:57.582691+0000) 2022-09-10T08:52:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:58.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:58.066159+0000) 2022-09-10T08:52:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:58.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:58.582850+0000) 2022-09-10T08:52:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:52:59.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:59.066328+0000) 2022-09-10T08:53:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:52:59.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:52:59.583060+0000) 2022-09-10T08:53:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:52:59 smithi110 ceph-mon[97633]: pgmap v8440: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:52:59 smithi104 ceph-mon[97044]: pgmap v8440: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:00.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:00.066512+0000) 2022-09-10T08:53:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:00.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:00.583264+0000) 2022-09-10T08:53:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:01.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:01.066744+0000) 2022-09-10T08:53:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:01.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:01.583471+0000) 2022-09-10T08:53:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:01 smithi110 ceph-mon[97633]: pgmap v8441: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:01 smithi104 ceph-mon[97044]: pgmap v8441: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:02.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:02.066945+0000) 2022-09-10T08:53:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:02.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:02.583675+0000) 2022-09-10T08:53:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:03.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:03.067177+0000) 2022-09-10T08:53:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:03.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:03.583951+0000) 2022-09-10T08:53:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:03 smithi110 ceph-mon[97633]: pgmap v8442: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:03 smithi104 ceph-mon[97044]: pgmap v8442: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:04.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:04.067389+0000) 2022-09-10T08:53:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:04.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:04.584180+0000) 2022-09-10T08:53:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:05.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:05.067584+0000) 2022-09-10T08:53:05.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:53:05.143+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:53:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:05.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:05.584420+0000) 2022-09-10T08:53:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:05 smithi110 ceph-mon[97633]: pgmap v8443: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:53:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:05 smithi104 ceph-mon[97044]: pgmap v8443: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:53:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:06.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:06.067795+0000) 2022-09-10T08:53:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:06.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:06.584608+0000) 2022-09-10T08:53:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:07 smithi104 ceph-mon[97044]: pgmap v8444: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:53:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:07.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:07.068059+0000) 2022-09-10T08:53:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:07 smithi110 ceph-mon[97633]: pgmap v8444: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:53:07.957 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:07.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:07.584764+0000) 2022-09-10T08:53:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:08.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:08.068311+0000) 2022-09-10T08:53:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:08.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:08.584992+0000) 2022-09-10T08:53:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:09 smithi104 ceph-mon[97044]: osdmap e608: 12 total, 12 up, 12 in 2022-09-10T08:53:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:09 smithi104 ceph-mon[97044]: pgmap v8446: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:53:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:09.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:09.068530+0000) 2022-09-10T08:53:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:09 smithi110 ceph-mon[97633]: osdmap e608: 12 total, 12 up, 12 in 2022-09-10T08:53:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:09 smithi110 ceph-mon[97633]: pgmap v8446: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:53:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:09.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:09.585192+0000) 2022-09-10T08:53:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:10.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:10.068764+0000) 2022-09-10T08:53:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:10.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:10.585384+0000) 2022-09-10T08:53:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:11.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:11.069034+0000) 2022-09-10T08:53:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:11.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:11.585581+0000) 2022-09-10T08:53:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:11 smithi110 ceph-mon[97633]: pgmap v8447: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:53:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:11 smithi110 ceph-mon[97633]: osdmap e609: 12 total, 12 up, 12 in 2022-09-10T08:53:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:11 smithi104 ceph-mon[97044]: pgmap v8447: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:53:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:11 smithi104 ceph-mon[97044]: osdmap e609: 12 total, 12 up, 12 in 2022-09-10T08:53:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:12.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:12.069278+0000) 2022-09-10T08:53:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:12.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:12.585796+0000) 2022-09-10T08:53:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:13.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:13.069526+0000) 2022-09-10T08:53:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:13.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:13.586059+0000) 2022-09-10T08:53:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:13 smithi110 ceph-mon[97633]: pgmap v8449: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:53:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:13 smithi104 ceph-mon[97044]: pgmap v8449: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:53:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:14.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:14.069780+0000) 2022-09-10T08:53:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:14.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:14.586270+0000) 2022-09-10T08:53:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:15.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:15.070003+0000) 2022-09-10T08:53:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:15.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:15.586469+0000) 2022-09-10T08:53:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:15 smithi110 ceph-mon[97633]: pgmap v8450: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:53:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:15 smithi104 ceph-mon[97044]: pgmap v8450: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:53:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:16.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:16.070260+0000) 2022-09-10T08:53:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:16.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:16.586684+0000) 2022-09-10T08:53:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:17 smithi104 ceph-mon[97044]: pgmap v8451: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:53:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:17.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:17.070436+0000) 2022-09-10T08:53:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:17 smithi110 ceph-mon[97633]: pgmap v8451: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:53:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:17.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:17.586936+0000) 2022-09-10T08:53:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:18.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:18.070702+0000) 2022-09-10T08:53:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:18.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:18.587164+0000) 2022-09-10T08:53:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:19.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:19.070979+0000) 2022-09-10T08:53:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:19.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:19.587337+0000) 2022-09-10T08:53:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:19 smithi110 ceph-mon[97633]: pgmap v8452: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:53:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:19 smithi104 ceph-mon[97044]: pgmap v8452: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:53:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:20.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:20.071255+0000) 2022-09-10T08:53:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:53:20.157+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:53:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:20.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:20.587502+0000) 2022-09-10T08:53:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:21.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:21.071512+0000) 2022-09-10T08:53:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:21.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:21.587713+0000) 2022-09-10T08:53:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:21 smithi110 ceph-mon[97633]: pgmap v8453: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:53:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:21 smithi104 ceph-mon[97044]: pgmap v8453: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:53:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:22.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:22.071718+0000) 2022-09-10T08:53:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:22.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:22.587898+0000) 2022-09-10T08:53:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:23.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:23.071962+0000) 2022-09-10T08:53:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:23.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:23.588120+0000) 2022-09-10T08:53:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:23 smithi110 ceph-mon[97633]: pgmap v8454: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:53:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:23 smithi104 ceph-mon[97044]: pgmap v8454: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:53:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:24.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:24.072198+0000) 2022-09-10T08:53:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:24.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:24.588261+0000) 2022-09-10T08:53:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:25.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:25.072409+0000) 2022-09-10T08:53:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:25.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:25.588474+0000) 2022-09-10T08:53:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:25 smithi110 ceph-mon[97633]: pgmap v8455: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:53:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:25 smithi104 ceph-mon[97044]: pgmap v8455: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:53:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:26.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:26.072587+0000) 2022-09-10T08:53:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:26.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:26.588702+0000) 2022-09-10T08:53:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:27 smithi104 ceph-mon[97044]: pgmap v8456: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:27.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:27.072839+0000) 2022-09-10T08:53:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:27 smithi110 ceph-mon[97633]: pgmap v8456: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:27.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:27.588851+0000) 2022-09-10T08:53:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:28.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:28.073131+0000) 2022-09-10T08:53:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:28.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:28.589056+0000) 2022-09-10T08:53:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:29.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:29.073337+0000) 2022-09-10T08:53:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:29.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:29.589183+0000) 2022-09-10T08:53:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:29 smithi110 ceph-mon[97633]: pgmap v8457: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:29 smithi104 ceph-mon[97044]: pgmap v8457: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:30.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:30.073569+0000) 2022-09-10T08:53:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:30.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:30.589392+0000) 2022-09-10T08:53:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:31.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:31.073813+0000) 2022-09-10T08:53:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:31.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:31.589616+0000) 2022-09-10T08:53:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:31 smithi110 ceph-mon[97633]: pgmap v8458: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:31 smithi104 ceph-mon[97044]: pgmap v8458: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:32.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:32.073984+0000) 2022-09-10T08:53:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:32.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:32.589837+0000) 2022-09-10T08:53:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:33.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:33.074210+0000) 2022-09-10T08:53:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:33.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:33.589999+0000) 2022-09-10T08:53:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:33 smithi110 ceph-mon[97633]: pgmap v8459: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:33 smithi104 ceph-mon[97044]: pgmap v8459: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:34.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:34.074424+0000) 2022-09-10T08:53:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:34.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:34.590210+0000) 2022-09-10T08:53:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:35.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:35.074629+0000) 2022-09-10T08:53:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:53:35.173+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:53:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:35.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:35.590400+0000) 2022-09-10T08:53:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:35 smithi110 ceph-mon[97633]: pgmap v8460: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:35 smithi104 ceph-mon[97044]: pgmap v8460: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:36.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:36.074767+0000) 2022-09-10T08:53:37.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:36.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:36.590588+0000) 2022-09-10T08:53:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:37 smithi104 ceph-mon[97044]: pgmap v8461: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:37.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:37.074994+0000) 2022-09-10T08:53:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:37 smithi110 ceph-mon[97633]: pgmap v8461: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:37.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:37.590840+0000) 2022-09-10T08:53:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:38.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:38.075244+0000) 2022-09-10T08:53:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:38.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:38.591049+0000) 2022-09-10T08:53:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:39.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:39.075447+0000) 2022-09-10T08:53:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:39.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:39.591262+0000) 2022-09-10T08:53:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:39 smithi110 ceph-mon[97633]: pgmap v8462: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:39 smithi104 ceph-mon[97044]: pgmap v8462: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:40.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:40.075648+0000) 2022-09-10T08:53:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:40.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:40.591447+0000) 2022-09-10T08:53:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:41.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:41.075834+0000) 2022-09-10T08:53:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:41.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:41.591649+0000) 2022-09-10T08:53:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:41 smithi110 ceph-mon[97633]: pgmap v8463: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:41 smithi104 ceph-mon[97044]: pgmap v8463: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:42.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:42.076124+0000) 2022-09-10T08:53:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:42.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:42.591917+0000) 2022-09-10T08:53:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:43.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:43.076302+0000) 2022-09-10T08:53:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:43.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:43.592168+0000) 2022-09-10T08:53:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:43 smithi110 ceph-mon[97633]: pgmap v8464: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:43 smithi104 ceph-mon[97044]: pgmap v8464: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:44.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:44.076509+0000) 2022-09-10T08:53:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:44.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:44.592368+0000) 2022-09-10T08:53:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:53:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:53:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:53:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:53:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:53:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:53:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:53:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:53:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:45.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:45.076700+0000) 2022-09-10T08:53:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:45.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:45.592616+0000) 2022-09-10T08:53:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:45 smithi110 ceph-mon[97633]: pgmap v8465: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:45 smithi104 ceph-mon[97044]: pgmap v8465: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:46.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:46.076881+0000) 2022-09-10T08:53:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:46.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:46.592869+0000) 2022-09-10T08:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:47 smithi104 ceph-mon[97044]: pgmap v8466: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:47.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:47.077159+0000) 2022-09-10T08:53:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:47 smithi110 ceph-mon[97633]: pgmap v8466: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:47.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:47.593133+0000) 2022-09-10T08:53:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:48.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:48.077343+0000) 2022-09-10T08:53:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:48.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:48.593398+0000) 2022-09-10T08:53:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:49.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:49.077511+0000) 2022-09-10T08:53:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:49.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:49.593631+0000) 2022-09-10T08:53:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:49 smithi110 ceph-mon[97633]: pgmap v8467: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:49 smithi104 ceph-mon[97044]: pgmap v8467: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:50.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:50.077713+0000) 2022-09-10T08:53:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:53:50.185+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:53:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:50.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:50.593884+0000) 2022-09-10T08:53:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:51.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:51.077927+0000) 2022-09-10T08:53:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:51.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:51.594162+0000) 2022-09-10T08:53:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:51 smithi110 ceph-mon[97633]: pgmap v8468: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:51 smithi104 ceph-mon[97044]: pgmap v8468: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:52.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:52.078197+0000) 2022-09-10T08:53:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:52.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:52.594411+0000) 2022-09-10T08:53:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:53.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:53.078445+0000) 2022-09-10T08:53:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:53.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:53.594611+0000) 2022-09-10T08:53:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:53 smithi110 ceph-mon[97633]: pgmap v8469: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:53 smithi104 ceph-mon[97044]: pgmap v8469: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:54.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:54.078621+0000) 2022-09-10T08:53:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:54.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:54.594873+0000) 2022-09-10T08:53:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:55.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:55.078852+0000) 2022-09-10T08:53:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:55.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:55.595083+0000) 2022-09-10T08:53:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:55 smithi110 ceph-mon[97633]: pgmap v8470: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:55 smithi104 ceph-mon[97044]: pgmap v8470: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:53:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:56.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:56.079091+0000) 2022-09-10T08:53:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:56.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:56.595396+0000) 2022-09-10T08:53:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:57 smithi104 ceph-mon[97044]: pgmap v8471: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:57.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:57.079349+0000) 2022-09-10T08:53:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:57 smithi110 ceph-mon[97633]: pgmap v8471: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:53:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:57.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:57.595627+0000) 2022-09-10T08:53:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:58.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:58.079551+0000) 2022-09-10T08:53:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:58.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:58.595828+0000) 2022-09-10T08:53:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:53:59.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:59.079715+0000) 2022-09-10T08:54:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:53:59.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:53:59.596055+0000) 2022-09-10T08:54:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:53:59 smithi110 ceph-mon[97633]: pgmap v8472: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:53:59 smithi104 ceph-mon[97044]: pgmap v8472: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:00.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:00.079946+0000) 2022-09-10T08:54:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:00.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:00.596217+0000) 2022-09-10T08:54:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:01.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:01.080188+0000) 2022-09-10T08:54:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:01.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:01.596422+0000) 2022-09-10T08:54:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:01 smithi110 ceph-mon[97633]: pgmap v8473: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:01 smithi104 ceph-mon[97044]: pgmap v8473: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:02.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:02.080441+0000) 2022-09-10T08:54:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:02.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:02.596643+0000) 2022-09-10T08:54:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:03.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:03.080681+0000) 2022-09-10T08:54:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:03.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:03.596903+0000) 2022-09-10T08:54:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:03 smithi110 ceph-mon[97633]: pgmap v8474: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:03 smithi104 ceph-mon[97044]: pgmap v8474: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:04.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:04.080923+0000) 2022-09-10T08:54:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:04.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:04.597149+0000) 2022-09-10T08:54:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:05.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:05.081156+0000) 2022-09-10T08:54:05.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:54:05.189+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:54:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:05.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:05.597372+0000) 2022-09-10T08:54:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:05 smithi110 ceph-mon[97633]: pgmap v8475: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:54:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:05 smithi104 ceph-mon[97044]: pgmap v8475: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:54:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:06.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:06.081409+0000) 2022-09-10T08:54:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:06.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:06.597639+0000) 2022-09-10T08:54:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:07 smithi104 ceph-mon[97044]: pgmap v8476: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:54:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:07.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:07.081627+0000) 2022-09-10T08:54:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:07 smithi110 ceph-mon[97633]: pgmap v8476: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:54:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:07.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:07.597868+0000) 2022-09-10T08:54:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:08.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:08.081839+0000) 2022-09-10T08:54:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:08.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:08.598054+0000) 2022-09-10T08:54:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:09 smithi104 ceph-mon[97044]: osdmap e610: 12 total, 12 up, 12 in 2022-09-10T08:54:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:09 smithi104 ceph-mon[97044]: pgmap v8478: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:54:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:09.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:09.082119+0000) 2022-09-10T08:54:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:09 smithi110 ceph-mon[97633]: osdmap e610: 12 total, 12 up, 12 in 2022-09-10T08:54:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:09 smithi110 ceph-mon[97633]: pgmap v8478: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:54:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:09.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:09.598303+0000) 2022-09-10T08:54:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:10.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:10.082306+0000) 2022-09-10T08:54:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:10.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:10.598560+0000) 2022-09-10T08:54:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:11.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:11.082455+0000) 2022-09-10T08:54:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:11.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:11.598812+0000) 2022-09-10T08:54:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:11 smithi110 ceph-mon[97633]: pgmap v8479: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:54:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:11 smithi110 ceph-mon[97633]: osdmap e611: 12 total, 12 up, 12 in 2022-09-10T08:54:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:11 smithi104 ceph-mon[97044]: pgmap v8479: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:54:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:11 smithi104 ceph-mon[97044]: osdmap e611: 12 total, 12 up, 12 in 2022-09-10T08:54:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:12.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:12.082715+0000) 2022-09-10T08:54:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:12.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:12.599060+0000) 2022-09-10T08:54:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:13.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:13.082980+0000) 2022-09-10T08:54:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:13.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:13.599308+0000) 2022-09-10T08:54:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:13 smithi110 ceph-mon[97633]: pgmap v8481: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:54:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:13 smithi104 ceph-mon[97044]: pgmap v8481: 129 pgs: 129 active+clean; 8.6 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:54:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:14.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:14.083212+0000) 2022-09-10T08:54:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:14.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:14.599590+0000) 2022-09-10T08:54:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:15.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:15.083452+0000) 2022-09-10T08:54:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:15.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:15.599796+0000) 2022-09-10T08:54:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:15 smithi110 ceph-mon[97633]: pgmap v8482: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:54:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:15 smithi104 ceph-mon[97044]: pgmap v8482: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:54:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:16.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:16.083606+0000) 2022-09-10T08:54:17.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:16.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:16.600050+0000) 2022-09-10T08:54:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:17 smithi104 ceph-mon[97044]: pgmap v8483: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:54:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:17.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:17.083804+0000) 2022-09-10T08:54:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:17 smithi110 ceph-mon[97633]: pgmap v8483: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:54:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:17.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:17.600274+0000) 2022-09-10T08:54:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:18.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:18.084003+0000) 2022-09-10T08:54:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:18.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:18.600497+0000) 2022-09-10T08:54:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:19.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:19.084202+0000) 2022-09-10T08:54:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:19.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:19.600669+0000) 2022-09-10T08:54:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:19 smithi110 ceph-mon[97633]: pgmap v8484: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:54:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:19 smithi104 ceph-mon[97044]: pgmap v8484: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:54:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:20.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:20.084346+0000) 2022-09-10T08:54:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:54:20.205+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:54:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:20.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:20.600934+0000) 2022-09-10T08:54:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:21.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:21.084552+0000) 2022-09-10T08:54:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:21.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:21.601131+0000) 2022-09-10T08:54:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:21 smithi110 ceph-mon[97633]: pgmap v8485: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:54:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:21 smithi104 ceph-mon[97044]: pgmap v8485: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:54:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:22.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:22.084836+0000) 2022-09-10T08:54:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:22.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:22.601281+0000) 2022-09-10T08:54:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:23.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:23.085070+0000) 2022-09-10T08:54:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:23.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:23.601404+0000) 2022-09-10T08:54:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:23 smithi110 ceph-mon[97633]: pgmap v8486: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:54:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:23 smithi104 ceph-mon[97044]: pgmap v8486: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:54:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:24.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:24.085259+0000) 2022-09-10T08:54:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:24.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:24.601578+0000) 2022-09-10T08:54:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:25.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:25.085457+0000) 2022-09-10T08:54:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:25.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:25.601798+0000) 2022-09-10T08:54:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:25 smithi104 ceph-mon[97044]: pgmap v8487: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:54:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:25 smithi110 ceph-mon[97633]: pgmap v8487: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:54:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:26.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:26.085635+0000) 2022-09-10T08:54:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:26.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:26.601996+0000) 2022-09-10T08:54:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:27 smithi104 ceph-mon[97044]: pgmap v8488: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:27.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:27.085784+0000) 2022-09-10T08:54:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:27 smithi110 ceph-mon[97633]: pgmap v8488: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:27.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:27.602201+0000) 2022-09-10T08:54:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:28.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:28.086008+0000) 2022-09-10T08:54:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:28.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:28.602489+0000) 2022-09-10T08:54:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:29.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:29.086164+0000) 2022-09-10T08:54:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:29.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:29.602744+0000) 2022-09-10T08:54:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:29 smithi110 ceph-mon[97633]: pgmap v8489: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:29 smithi104 ceph-mon[97044]: pgmap v8489: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:30.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:30.086393+0000) 2022-09-10T08:54:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:30.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:30.602911+0000) 2022-09-10T08:54:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:31.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:31.086592+0000) 2022-09-10T08:54:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:31.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:31.603092+0000) 2022-09-10T08:54:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:31 smithi110 ceph-mon[97633]: pgmap v8490: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:31 smithi104 ceph-mon[97044]: pgmap v8490: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:32.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:32.086792+0000) 2022-09-10T08:54:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:32.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:32.603275+0000) 2022-09-10T08:54:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:33.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:33.087049+0000) 2022-09-10T08:54:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:33.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:33.603441+0000) 2022-09-10T08:54:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:33 smithi104 ceph-mon[97044]: pgmap v8491: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:33 smithi110 ceph-mon[97633]: pgmap v8491: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:34.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:34.087323+0000) 2022-09-10T08:54:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:34.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:34.603688+0000) 2022-09-10T08:54:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:35.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:35.087473+0000) 2022-09-10T08:54:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:54:35.220+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:54:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:35.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:35.603900+0000) 2022-09-10T08:54:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:35 smithi104 ceph-mon[97044]: pgmap v8492: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:35 smithi110 ceph-mon[97633]: pgmap v8492: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:36.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:36.087719+0000) 2022-09-10T08:54:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:36.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:36.604172+0000) 2022-09-10T08:54:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:37 smithi104 ceph-mon[97044]: pgmap v8493: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:37.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:37.087948+0000) 2022-09-10T08:54:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:37 smithi110 ceph-mon[97633]: pgmap v8493: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:37.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:37.604307+0000) 2022-09-10T08:54:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:38.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:38.088238+0000) 2022-09-10T08:54:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:38.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:38.604468+0000) 2022-09-10T08:54:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:39.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:39.088383+0000) 2022-09-10T08:54:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:39.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:39.604711+0000) 2022-09-10T08:54:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:39 smithi110 ceph-mon[97633]: pgmap v8494: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:39 smithi104 ceph-mon[97044]: pgmap v8494: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:40.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:40.088623+0000) 2022-09-10T08:54:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:40.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:40.604955+0000) 2022-09-10T08:54:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:41.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:41.088867+0000) 2022-09-10T08:54:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:41.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:41.605205+0000) 2022-09-10T08:54:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:41 smithi104 ceph-mon[97044]: pgmap v8495: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:41 smithi110 ceph-mon[97633]: pgmap v8495: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:42.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:42.089075+0000) 2022-09-10T08:54:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:42.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:42.605426+0000) 2022-09-10T08:54:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:43.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:43.089357+0000) 2022-09-10T08:54:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:43.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:43.605618+0000) 2022-09-10T08:54:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:43 smithi104 ceph-mon[97044]: pgmap v8496: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:43 smithi110 ceph-mon[97633]: pgmap v8496: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:44.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:44.089570+0000) 2022-09-10T08:54:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:44.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:44.605819+0000) 2022-09-10T08:54:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:54:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:54:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:54:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:54:45.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:54:45.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:54:45.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:54:45.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:54:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:45.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:45.089794+0000) 2022-09-10T08:54:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:45.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:45.606047+0000) 2022-09-10T08:54:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:45 smithi104 ceph-mon[97044]: pgmap v8497: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:45 smithi110 ceph-mon[97633]: pgmap v8497: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:46.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:46.090009+0000) 2022-09-10T08:54:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:46.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:46.606276+0000) 2022-09-10T08:54:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:47 smithi104 ceph-mon[97044]: pgmap v8498: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:47.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:47.090185+0000) 2022-09-10T08:54:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:47 smithi110 ceph-mon[97633]: pgmap v8498: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:47.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:47.606507+0000) 2022-09-10T08:54:48.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:48.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:48.090359+0000) 2022-09-10T08:54:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:48.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:48.606751+0000) 2022-09-10T08:54:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:49.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:49.090525+0000) 2022-09-10T08:54:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:49.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:49.606994+0000) 2022-09-10T08:54:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:49 smithi104 ceph-mon[97044]: pgmap v8499: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:49 smithi110 ceph-mon[97633]: pgmap v8499: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:50.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:50.090722+0000) 2022-09-10T08:54:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:54:50.236+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:54:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:50.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:50.607230+0000) 2022-09-10T08:54:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:51.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:51.090974+0000) 2022-09-10T08:54:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:51.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:51.607486+0000) 2022-09-10T08:54:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:51 smithi104 ceph-mon[97044]: pgmap v8500: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:51 smithi110 ceph-mon[97633]: pgmap v8500: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:52.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:52.091261+0000) 2022-09-10T08:54:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:52.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:52.607667+0000) 2022-09-10T08:54:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:53.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:53.091485+0000) 2022-09-10T08:54:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:53.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:53.607930+0000) 2022-09-10T08:54:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:53 smithi104 ceph-mon[97044]: pgmap v8501: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:53 smithi110 ceph-mon[97633]: pgmap v8501: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:54.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:54.091663+0000) 2022-09-10T08:54:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:54.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:54.608207+0000) 2022-09-10T08:54:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:55.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:55.091886+0000) 2022-09-10T08:54:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:55.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:55.608372+0000) 2022-09-10T08:54:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:55 smithi104 ceph-mon[97044]: pgmap v8502: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:55 smithi110 ceph-mon[97633]: pgmap v8502: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:54:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:56.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:56.092148+0000) 2022-09-10T08:54:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:56.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:56.608615+0000) 2022-09-10T08:54:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:57 smithi104 ceph-mon[97044]: pgmap v8503: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:57.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:57.092374+0000) 2022-09-10T08:54:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:57 smithi110 ceph-mon[97633]: pgmap v8503: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:54:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:57.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:57.608805+0000) 2022-09-10T08:54:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:58.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:58.092569+0000) 2022-09-10T08:54:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:58.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:58.609055+0000) 2022-09-10T08:54:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:54:59.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:59.092819+0000) 2022-09-10T08:55:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:54:59.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:54:59.609313+0000) 2022-09-10T08:55:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:54:59 smithi104 ceph-mon[97044]: pgmap v8504: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:54:59 smithi110 ceph-mon[97633]: pgmap v8504: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:00.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:00.093040+0000) 2022-09-10T08:55:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:00.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:00.609466+0000) 2022-09-10T08:55:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:01.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:01.093218+0000) 2022-09-10T08:55:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:01.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:01.609620+0000) 2022-09-10T08:55:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:01 smithi104 ceph-mon[97044]: pgmap v8505: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:01 smithi110 ceph-mon[97633]: pgmap v8505: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:02.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:02.093485+0000) 2022-09-10T08:55:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:02.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:02.609851+0000) 2022-09-10T08:55:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:03.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:03.093710+0000) 2022-09-10T08:55:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:03.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:03.610124+0000) 2022-09-10T08:55:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:03 smithi104 ceph-mon[97044]: pgmap v8506: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:03 smithi110 ceph-mon[97633]: pgmap v8506: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:04.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:04.093954+0000) 2022-09-10T08:55:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:04.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:04.610301+0000) 2022-09-10T08:55:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:55:05.246+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:55:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:05.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:05.094253+0000) 2022-09-10T08:55:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:05.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:05.610502+0000) 2022-09-10T08:55:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:05 smithi104 ceph-mon[97044]: pgmap v8507: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:55:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:05 smithi110 ceph-mon[97633]: pgmap v8507: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:55:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:06.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:06.094450+0000) 2022-09-10T08:55:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:06.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:06.610730+0000) 2022-09-10T08:55:07.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:07 smithi104 ceph-mon[97044]: pgmap v8508: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:55:07.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:07.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:07.094644+0000) 2022-09-10T08:55:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:07 smithi110 ceph-mon[97633]: pgmap v8508: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:55:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:07.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:07.610946+0000) 2022-09-10T08:55:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:08.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:08.094817+0000) 2022-09-10T08:55:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:08.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:08.611192+0000) 2022-09-10T08:55:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:09 smithi104 ceph-mon[97044]: osdmap e612: 12 total, 12 up, 12 in 2022-09-10T08:55:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:09 smithi104 ceph-mon[97044]: pgmap v8510: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:55:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:09.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:09.095015+0000) 2022-09-10T08:55:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:09 smithi110 ceph-mon[97633]: osdmap e612: 12 total, 12 up, 12 in 2022-09-10T08:55:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:09 smithi110 ceph-mon[97633]: pgmap v8510: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:55:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:09.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:09.611388+0000) 2022-09-10T08:55:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:10.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:10.095248+0000) 2022-09-10T08:55:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:10.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:10.611637+0000) 2022-09-10T08:55:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:11.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:11.095504+0000) 2022-09-10T08:55:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:11.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:11.611869+0000) 2022-09-10T08:55:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:11 smithi104 ceph-mon[97044]: pgmap v8511: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:55:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:11 smithi104 ceph-mon[97044]: osdmap e613: 12 total, 12 up, 12 in 2022-09-10T08:55:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:11 smithi110 ceph-mon[97633]: pgmap v8511: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:55:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:11 smithi110 ceph-mon[97633]: osdmap e613: 12 total, 12 up, 12 in 2022-09-10T08:55:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:12.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:12.095765+0000) 2022-09-10T08:55:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:12.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:12.612071+0000) 2022-09-10T08:55:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:13.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:13.095972+0000) 2022-09-10T08:55:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:13.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:13.612278+0000) 2022-09-10T08:55:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:13 smithi104 ceph-mon[97044]: pgmap v8513: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:55:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:13 smithi110 ceph-mon[97633]: pgmap v8513: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:55:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:14.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:14.096226+0000) 2022-09-10T08:55:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:14.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:14.612489+0000) 2022-09-10T08:55:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:15.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:15.096491+0000) 2022-09-10T08:55:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:15.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:15.612701+0000) 2022-09-10T08:55:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:15 smithi104 ceph-mon[97044]: pgmap v8514: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:55:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:15 smithi110 ceph-mon[97633]: pgmap v8514: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:55:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:16.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:16.096651+0000) 2022-09-10T08:55:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:16.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:16.612975+0000) 2022-09-10T08:55:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:17 smithi104 ceph-mon[97044]: pgmap v8515: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:55:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:17.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:17.096846+0000) 2022-09-10T08:55:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:17 smithi110 ceph-mon[97633]: pgmap v8515: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:55:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:17.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:17.613173+0000) 2022-09-10T08:55:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:18.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:18.097037+0000) 2022-09-10T08:55:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:18.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:18.613322+0000) 2022-09-10T08:55:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:19.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:19.097260+0000) 2022-09-10T08:55:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:19.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:19.613549+0000) 2022-09-10T08:55:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:19 smithi104 ceph-mon[97044]: pgmap v8516: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:55:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:19 smithi110 ceph-mon[97633]: pgmap v8516: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:55:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:20.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:20.097522+0000) 2022-09-10T08:55:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:55:20.256+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:55:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:20.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:20.613817+0000) 2022-09-10T08:55:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:21.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:21.097778+0000) 2022-09-10T08:55:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:21.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:21.614049+0000) 2022-09-10T08:55:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:21 smithi104 ceph-mon[97044]: pgmap v8517: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:55:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:21 smithi110 ceph-mon[97633]: pgmap v8517: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:55:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:22.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:22.098081+0000) 2022-09-10T08:55:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:22.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:22.614261+0000) 2022-09-10T08:55:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:23.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:23.098338+0000) 2022-09-10T08:55:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:23.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:23.614446+0000) 2022-09-10T08:55:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:23 smithi104 ceph-mon[97044]: pgmap v8518: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:55:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:23 smithi110 ceph-mon[97633]: pgmap v8518: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:55:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:24.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:24.098622+0000) 2022-09-10T08:55:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:24.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:24.614590+0000) 2022-09-10T08:55:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:25.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:25.098760+0000) 2022-09-10T08:55:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:25.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:25.614800+0000) 2022-09-10T08:55:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:25 smithi104 ceph-mon[97044]: pgmap v8519: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:55:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:25 smithi110 ceph-mon[97633]: pgmap v8519: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:55:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:26.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:26.098988+0000) 2022-09-10T08:55:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:26.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:26.615018+0000) 2022-09-10T08:55:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:27 smithi104 ceph-mon[97044]: pgmap v8520: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:27.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:27.099191+0000) 2022-09-10T08:55:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:27 smithi110 ceph-mon[97633]: pgmap v8520: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:27.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:27.615185+0000) 2022-09-10T08:55:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:28.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:28.099348+0000) 2022-09-10T08:55:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:28.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:28.615358+0000) 2022-09-10T08:55:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:29.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:29.099594+0000) 2022-09-10T08:55:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:29.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:29.615604+0000) 2022-09-10T08:55:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:29 smithi104 ceph-mon[97044]: pgmap v8521: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:29 smithi110 ceph-mon[97633]: pgmap v8521: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:30.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:30.099858+0000) 2022-09-10T08:55:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:30.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:30.615780+0000) 2022-09-10T08:55:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:31.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:31.100120+0000) 2022-09-10T08:55:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:31.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:31.616007+0000) 2022-09-10T08:55:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:31 smithi104 ceph-mon[97044]: pgmap v8522: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:31 smithi110 ceph-mon[97633]: pgmap v8522: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:32.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:32.100335+0000) 2022-09-10T08:55:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:32.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:32.616227+0000) 2022-09-10T08:55:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:33.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:33.100562+0000) 2022-09-10T08:55:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:33.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:33.616500+0000) 2022-09-10T08:55:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:33 smithi104 ceph-mon[97044]: pgmap v8523: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:33 smithi110 ceph-mon[97633]: pgmap v8523: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:34.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:34.100803+0000) 2022-09-10T08:55:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:34.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:34.616713+0000) 2022-09-10T08:55:35.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:55:35.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:55:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:35.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:35.101073+0000) 2022-09-10T08:55:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:35.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:35.616973+0000) 2022-09-10T08:55:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:35 smithi104 ceph-mon[97044]: pgmap v8524: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:35 smithi110 ceph-mon[97633]: pgmap v8524: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:36.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:36.101368+0000) 2022-09-10T08:55:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:36.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:36.617202+0000) 2022-09-10T08:55:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:37 smithi104 ceph-mon[97044]: pgmap v8525: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:37.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:37.101618+0000) 2022-09-10T08:55:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:37 smithi110 ceph-mon[97633]: pgmap v8525: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:37.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:37.617415+0000) 2022-09-10T08:55:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:38.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:38.101826+0000) 2022-09-10T08:55:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:38.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:38.617672+0000) 2022-09-10T08:55:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:39.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:39.102053+0000) 2022-09-10T08:55:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:39.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:39.617893+0000) 2022-09-10T08:55:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:39 smithi104 ceph-mon[97044]: pgmap v8526: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:39 smithi110 ceph-mon[97633]: pgmap v8526: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:40.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:40.102291+0000) 2022-09-10T08:55:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:40.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:40.618173+0000) 2022-09-10T08:55:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:41.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:41.102543+0000) 2022-09-10T08:55:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:41.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:41.618387+0000) 2022-09-10T08:55:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:41 smithi104 ceph-mon[97044]: pgmap v8527: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:41 smithi110 ceph-mon[97633]: pgmap v8527: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:42.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:42.102728+0000) 2022-09-10T08:55:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:42.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:42.618609+0000) 2022-09-10T08:55:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:43.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:43.102987+0000) 2022-09-10T08:55:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:43.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:43.618865+0000) 2022-09-10T08:55:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:43 smithi104 ceph-mon[97044]: pgmap v8528: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:43 smithi110 ceph-mon[97633]: pgmap v8528: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:44.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:44.103247+0000) 2022-09-10T08:55:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:44.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:44.619036+0000) 2022-09-10T08:55:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:55:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:55:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:55:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:55:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:55:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:55:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:55:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:55:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:45.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:45.103513+0000) 2022-09-10T08:55:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:45.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:45.619211+0000) 2022-09-10T08:55:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:45 smithi104 ceph-mon[97044]: pgmap v8529: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:45 smithi110 ceph-mon[97633]: pgmap v8529: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:46.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:46.103708+0000) 2022-09-10T08:55:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:46.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:46.619391+0000) 2022-09-10T08:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:47 smithi104 ceph-mon[97044]: pgmap v8530: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:47.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:47.103967+0000) 2022-09-10T08:55:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:47 smithi110 ceph-mon[97633]: pgmap v8530: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:47.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:47.619627+0000) 2022-09-10T08:55:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:48.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:48.104203+0000) 2022-09-10T08:55:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:48.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:48.619767+0000) 2022-09-10T08:55:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:49.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:49.104398+0000) 2022-09-10T08:55:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:49.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:49.619981+0000) 2022-09-10T08:55:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:49 smithi104 ceph-mon[97044]: pgmap v8531: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:49 smithi110 ceph-mon[97633]: pgmap v8531: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:50.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:50.104601+0000) 2022-09-10T08:55:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:55:50.274+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:55:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:50.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:50.620213+0000) 2022-09-10T08:55:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:51.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:51.104860+0000) 2022-09-10T08:55:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:51.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:51.620436+0000) 2022-09-10T08:55:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:51 smithi104 ceph-mon[97044]: pgmap v8532: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:51 smithi110 ceph-mon[97633]: pgmap v8532: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:52.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:52.105174+0000) 2022-09-10T08:55:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:52.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:52.620684+0000) 2022-09-10T08:55:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:53.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:53.105422+0000) 2022-09-10T08:55:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:53.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:53.620860+0000) 2022-09-10T08:55:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:53 smithi104 ceph-mon[97044]: pgmap v8533: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:53 smithi110 ceph-mon[97633]: pgmap v8533: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:54.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:54.105665+0000) 2022-09-10T08:55:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:54.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:54.621134+0000) 2022-09-10T08:55:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:55.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:55.105924+0000) 2022-09-10T08:55:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:55.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:55.621360+0000) 2022-09-10T08:55:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:55 smithi104 ceph-mon[97044]: pgmap v8534: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:55 smithi110 ceph-mon[97633]: pgmap v8534: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:55:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:56.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:56.106191+0000) 2022-09-10T08:55:57.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:56.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:56.621570+0000) 2022-09-10T08:55:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:57 smithi104 ceph-mon[97044]: pgmap v8535: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:57.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:57.106485+0000) 2022-09-10T08:55:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:57 smithi110 ceph-mon[97633]: pgmap v8535: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:55:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:57.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:57.621755+0000) 2022-09-10T08:55:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:58.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:58.106705+0000) 2022-09-10T08:55:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:58.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:58.621986+0000) 2022-09-10T08:55:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:55:59.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:59.106906+0000) 2022-09-10T08:56:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:55:59.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:55:59.622214+0000) 2022-09-10T08:56:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:55:59 smithi104 ceph-mon[97044]: pgmap v8536: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:55:59 smithi110 ceph-mon[97633]: pgmap v8536: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:00.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:00.107165+0000) 2022-09-10T08:56:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:00.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:00.622443+0000) 2022-09-10T08:56:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:01.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:01.107339+0000) 2022-09-10T08:56:02.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:01.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:01.622668+0000) 2022-09-10T08:56:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:01 smithi104 ceph-mon[97044]: pgmap v8537: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:01 smithi110 ceph-mon[97633]: pgmap v8537: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:02.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:02.107581+0000) 2022-09-10T08:56:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:02.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:02.622873+0000) 2022-09-10T08:56:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:03.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:03.107838+0000) 2022-09-10T08:56:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:03.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:03.623071+0000) 2022-09-10T08:56:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:03 smithi104 ceph-mon[97044]: pgmap v8538: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:03 smithi110 ceph-mon[97633]: pgmap v8538: 129 pgs: 129 active+clean; 8.6 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:04.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:04.108092+0000) 2022-09-10T08:56:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:04.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:04.623321+0000) 2022-09-10T08:56:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:56:05.290+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:56:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:05.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:05.108270+0000) 2022-09-10T08:56:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:05.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:05.623467+0000) 2022-09-10T08:56:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:05 smithi104 ceph-mon[97044]: pgmap v8539: 129 pgs: 129 active+clean; 8.6 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:56:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:05 smithi110 ceph-mon[97633]: pgmap v8539: 129 pgs: 129 active+clean; 8.6 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:56:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:06.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:06.108445+0000) 2022-09-10T08:56:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:06.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:06.623666+0000) 2022-09-10T08:56:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:07 smithi104 ceph-mon[97044]: pgmap v8540: 129 pgs: 129 active+clean; 8.6 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:56:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:07.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:07.108669+0000) 2022-09-10T08:56:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:07 smithi110 ceph-mon[97633]: pgmap v8540: 129 pgs: 129 active+clean; 8.6 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:56:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:07.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:07.623911+0000) 2022-09-10T08:56:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:08.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:08.108908+0000) 2022-09-10T08:56:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:08.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:08.624117+0000) 2022-09-10T08:56:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:09 smithi104 ceph-mon[97044]: osdmap e614: 12 total, 12 up, 12 in 2022-09-10T08:56:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:09 smithi104 ceph-mon[97044]: pgmap v8542: 129 pgs: 129 active+clean; 8.6 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:56:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:09.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:09.109129+0000) 2022-09-10T08:56:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:09 smithi110 ceph-mon[97633]: osdmap e614: 12 total, 12 up, 12 in 2022-09-10T08:56:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:09 smithi110 ceph-mon[97633]: pgmap v8542: 129 pgs: 129 active+clean; 8.6 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:56:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:09.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:09.624340+0000) 2022-09-10T08:56:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:10.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:10.109297+0000) 2022-09-10T08:56:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:10.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:10.624605+0000) 2022-09-10T08:56:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:11.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:11.109521+0000) 2022-09-10T08:56:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:11.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:11.624881+0000) 2022-09-10T08:56:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:11 smithi104 ceph-mon[97044]: pgmap v8543: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:56:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:11 smithi104 ceph-mon[97044]: osdmap e615: 12 total, 12 up, 12 in 2022-09-10T08:56:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:11 smithi110 ceph-mon[97633]: pgmap v8543: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:56:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:11 smithi110 ceph-mon[97633]: osdmap e615: 12 total, 12 up, 12 in 2022-09-10T08:56:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:12.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:12.109785+0000) 2022-09-10T08:56:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:12.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:12.625158+0000) 2022-09-10T08:56:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:13.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:13.110015+0000) 2022-09-10T08:56:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:13.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:13.625399+0000) 2022-09-10T08:56:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:13 smithi104 ceph-mon[97044]: pgmap v8545: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:56:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:13 smithi110 ceph-mon[97633]: pgmap v8545: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:56:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:14.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:14.110318+0000) 2022-09-10T08:56:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:14.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:14.625567+0000) 2022-09-10T08:56:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:15.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:15.110580+0000) 2022-09-10T08:56:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:15.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:15.625781+0000) 2022-09-10T08:56:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:15 smithi104 ceph-mon[97044]: pgmap v8546: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:56:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:15 smithi110 ceph-mon[97633]: pgmap v8546: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:56:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:16.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:16.110854+0000) 2022-09-10T08:56:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:16.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:16.626037+0000) 2022-09-10T08:56:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:17 smithi104 ceph-mon[97044]: pgmap v8547: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:56:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:17.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:17.111059+0000) 2022-09-10T08:56:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:17 smithi110 ceph-mon[97633]: pgmap v8547: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:56:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:17.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:17.626290+0000) 2022-09-10T08:56:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:18.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:18.111285+0000) 2022-09-10T08:56:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:18.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:18.626550+0000) 2022-09-10T08:56:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:19.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:19.111512+0000) 2022-09-10T08:56:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:19.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:19.626816+0000) 2022-09-10T08:56:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:19 smithi104 ceph-mon[97044]: pgmap v8548: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:56:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:19 smithi110 ceph-mon[97633]: pgmap v8548: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:56:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:20.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:20.111736+0000) 2022-09-10T08:56:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:56:20.306+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:56:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:20.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:20.627076+0000) 2022-09-10T08:56:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:21.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:21.111973+0000) 2022-09-10T08:56:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:21.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:21.627255+0000) 2022-09-10T08:56:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:21 smithi104 ceph-mon[97044]: pgmap v8549: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:56:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:21 smithi110 ceph-mon[97633]: pgmap v8549: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:56:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:22.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:22.112271+0000) 2022-09-10T08:56:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:22.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:22.627418+0000) 2022-09-10T08:56:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:23.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:23.112513+0000) 2022-09-10T08:56:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:23.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:23.627561+0000) 2022-09-10T08:56:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:23 smithi104 ceph-mon[97044]: pgmap v8550: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:56:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:23 smithi110 ceph-mon[97633]: pgmap v8550: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:56:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:24.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:24.112776+0000) 2022-09-10T08:56:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:24.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:24.627696+0000) 2022-09-10T08:56:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:25.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:25.113015+0000) 2022-09-10T08:56:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:25.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:25.627906+0000) 2022-09-10T08:56:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:25 smithi104 ceph-mon[97044]: pgmap v8551: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:56:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:25 smithi110 ceph-mon[97633]: pgmap v8551: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:56:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:26.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:26.113185+0000) 2022-09-10T08:56:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:26.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:26.628083+0000) 2022-09-10T08:56:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:27 smithi104 ceph-mon[97044]: pgmap v8552: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:27.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:27.113402+0000) 2022-09-10T08:56:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:27 smithi110 ceph-mon[97633]: pgmap v8552: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:28.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:27.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:27.628278+0000) 2022-09-10T08:56:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:28.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:28.113610+0000) 2022-09-10T08:56:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:28.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:28.628455+0000) 2022-09-10T08:56:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:29.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:29.113800+0000) 2022-09-10T08:56:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:29.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:29.628687+0000) 2022-09-10T08:56:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:29 smithi104 ceph-mon[97044]: pgmap v8553: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:29 smithi110 ceph-mon[97633]: pgmap v8553: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:30.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:30.113976+0000) 2022-09-10T08:56:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:30.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:30.628892+0000) 2022-09-10T08:56:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:31.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:31.114179+0000) 2022-09-10T08:56:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:31.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:31.629150+0000) 2022-09-10T08:56:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:31 smithi104 ceph-mon[97044]: pgmap v8554: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:32.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:31 smithi110 ceph-mon[97633]: pgmap v8554: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:32.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:32.114438+0000) 2022-09-10T08:56:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:32.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:32.629383+0000) 2022-09-10T08:56:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:33.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:33.114629+0000) 2022-09-10T08:56:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:33.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:33.629614+0000) 2022-09-10T08:56:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:33 smithi104 ceph-mon[97044]: pgmap v8555: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:33 smithi110 ceph-mon[97633]: pgmap v8555: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:34.565 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:34.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:34.114842+0000) 2022-09-10T08:56:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:34.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:34.629848+0000) 2022-09-10T08:56:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:35.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:35.115080+0000) 2022-09-10T08:56:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:56:35.317+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:56:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:35.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:35.630066+0000) 2022-09-10T08:56:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:35 smithi104 ceph-mon[97044]: pgmap v8556: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:35 smithi110 ceph-mon[97633]: pgmap v8556: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:36.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:36.115365+0000) 2022-09-10T08:56:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:36.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:36.630275+0000) 2022-09-10T08:56:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:37 smithi104 ceph-mon[97044]: pgmap v8557: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:37.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:37.115583+0000) 2022-09-10T08:56:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:37 smithi110 ceph-mon[97633]: pgmap v8557: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:37.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:37.630437+0000) 2022-09-10T08:56:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:38.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:38.115823+0000) 2022-09-10T08:56:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:38.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:38.630670+0000) 2022-09-10T08:56:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:39.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:39.116107+0000) 2022-09-10T08:56:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:39.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:39.630881+0000) 2022-09-10T08:56:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:39 smithi104 ceph-mon[97044]: pgmap v8558: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:39 smithi110 ceph-mon[97633]: pgmap v8558: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:40.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:40.116273+0000) 2022-09-10T08:56:40.949 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:40.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:40.631124+0000) 2022-09-10T08:56:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:41.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:41.116511+0000) 2022-09-10T08:56:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:41.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:41.631340+0000) 2022-09-10T08:56:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:41 smithi104 ceph-mon[97044]: pgmap v8559: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:41 smithi110 ceph-mon[97633]: pgmap v8559: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:42.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:42.116741+0000) 2022-09-10T08:56:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:42.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:42.631549+0000) 2022-09-10T08:56:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:43.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:43.116913+0000) 2022-09-10T08:56:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:43.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:43.631740+0000) 2022-09-10T08:56:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:43 smithi104 ceph-mon[97044]: pgmap v8560: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:43 smithi110 ceph-mon[97633]: pgmap v8560: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:44.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:44.117132+0000) 2022-09-10T08:56:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:44.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:44.631965+0000) 2022-09-10T08:56:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:56:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:56:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:56:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:56:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:56:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:56:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:56:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:56:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:45.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:45.117313+0000) 2022-09-10T08:56:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:45.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:45.632166+0000) 2022-09-10T08:56:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:45 smithi104 ceph-mon[97044]: pgmap v8561: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:45 smithi110 ceph-mon[97633]: pgmap v8561: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:46.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:46.117521+0000) 2022-09-10T08:56:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:46.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:46.632400+0000) 2022-09-10T08:56:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:47 smithi104 ceph-mon[97044]: pgmap v8562: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:47.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:47.117717+0000) 2022-09-10T08:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:47 smithi110 ceph-mon[97633]: pgmap v8562: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:47.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:47.632645+0000) 2022-09-10T08:56:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:48.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:48.117868+0000) 2022-09-10T08:56:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:48.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:48.632844+0000) 2022-09-10T08:56:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:49.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:49.118059+0000) 2022-09-10T08:56:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:49.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:49.633079+0000) 2022-09-10T08:56:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:49 smithi104 ceph-mon[97044]: pgmap v8563: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:49 smithi110 ceph-mon[97633]: pgmap v8563: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:50.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:50.118324+0000) 2022-09-10T08:56:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:56:50.332+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:56:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:50.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:50.633358+0000) 2022-09-10T08:56:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:51.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:51.118524+0000) 2022-09-10T08:56:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:51.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:51.633602+0000) 2022-09-10T08:56:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:51 smithi104 ceph-mon[97044]: pgmap v8564: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:51 smithi110 ceph-mon[97633]: pgmap v8564: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:52.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:52.118745+0000) 2022-09-10T08:56:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:52.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:52.633758+0000) 2022-09-10T08:56:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:53.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:53.119020+0000) 2022-09-10T08:56:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:53.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:53.633980+0000) 2022-09-10T08:56:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:53 smithi104 ceph-mon[97044]: pgmap v8565: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:53 smithi110 ceph-mon[97633]: pgmap v8565: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:54.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:54.119170+0000) 2022-09-10T08:56:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:54.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:54.634243+0000) 2022-09-10T08:56:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:55.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:55.119450+0000) 2022-09-10T08:56:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:55.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:55.634486+0000) 2022-09-10T08:56:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:55 smithi104 ceph-mon[97044]: pgmap v8566: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:55 smithi110 ceph-mon[97633]: pgmap v8566: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:56:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:56.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:56.119754+0000) 2022-09-10T08:56:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:56.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:56.634745+0000) 2022-09-10T08:56:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:57 smithi104 ceph-mon[97044]: pgmap v8567: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:57.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:57.119947+0000) 2022-09-10T08:56:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:57 smithi110 ceph-mon[97633]: pgmap v8567: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:56:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:57.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:57.634997+0000) 2022-09-10T08:56:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:58.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:58.120224+0000) 2022-09-10T08:56:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:58.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:58.635250+0000) 2022-09-10T08:56:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:56:59.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:59.120442+0000) 2022-09-10T08:57:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:56:59.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:56:59.635474+0000) 2022-09-10T08:57:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:56:59 smithi104 ceph-mon[97044]: pgmap v8568: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:56:59 smithi110 ceph-mon[97633]: pgmap v8568: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:00.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:00.120658+0000) 2022-09-10T08:57:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:00.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:00.635668+0000) 2022-09-10T08:57:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:01.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:01.120865+0000) 2022-09-10T08:57:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:01.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:01.635902+0000) 2022-09-10T08:57:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:01 smithi104 ceph-mon[97044]: pgmap v8569: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:01 smithi110 ceph-mon[97633]: pgmap v8569: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:02.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:02.121068+0000) 2022-09-10T08:57:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:02.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:02.636139+0000) 2022-09-10T08:57:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:03.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:03.121323+0000) 2022-09-10T08:57:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:03.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:03.636337+0000) 2022-09-10T08:57:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:03 smithi104 ceph-mon[97044]: pgmap v8570: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:03 smithi110 ceph-mon[97633]: pgmap v8570: 129 pgs: 129 active+clean; 8.6 MiB data, 860 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:04.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:04.121519+0000) 2022-09-10T08:57:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:04.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:04.636546+0000) 2022-09-10T08:57:05.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:57:05.333+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:57:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:05.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:05.121701+0000) 2022-09-10T08:57:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:05.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:05.636792+0000) 2022-09-10T08:57:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:05 smithi104 ceph-mon[97044]: pgmap v8571: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:57:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:05 smithi110 ceph-mon[97633]: pgmap v8571: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:57:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:06.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:06.121945+0000) 2022-09-10T08:57:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:06.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:06.637056+0000) 2022-09-10T08:57:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:07 smithi104 ceph-mon[97044]: pgmap v8572: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:57:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:07.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:07.122166+0000) 2022-09-10T08:57:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:07 smithi110 ceph-mon[97633]: pgmap v8572: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:57:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:07.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:07.637333+0000) 2022-09-10T08:57:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:08.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:08.122417+0000) 2022-09-10T08:57:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:08.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:08.637547+0000) 2022-09-10T08:57:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:09 smithi104 ceph-mon[97044]: osdmap e616: 12 total, 12 up, 12 in 2022-09-10T08:57:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:09 smithi104 ceph-mon[97044]: pgmap v8574: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:57:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:09.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:09.122580+0000) 2022-09-10T08:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:09 smithi110 ceph-mon[97633]: osdmap e616: 12 total, 12 up, 12 in 2022-09-10T08:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:09 smithi110 ceph-mon[97633]: pgmap v8574: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:57:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:09.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:09.637808+0000) 2022-09-10T08:57:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:10.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:10.122728+0000) 2022-09-10T08:57:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:10.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:10.638008+0000) 2022-09-10T08:57:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:11.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:11.122957+0000) 2022-09-10T08:57:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:11.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:11.638260+0000) 2022-09-10T08:57:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:11 smithi104 ceph-mon[97044]: pgmap v8575: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:57:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:11 smithi104 ceph-mon[97044]: osdmap e617: 12 total, 12 up, 12 in 2022-09-10T08:57:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:11 smithi110 ceph-mon[97633]: pgmap v8575: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:57:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:11 smithi110 ceph-mon[97633]: osdmap e617: 12 total, 12 up, 12 in 2022-09-10T08:57:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:12.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:12.123222+0000) 2022-09-10T08:57:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:12.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:12.638494+0000) 2022-09-10T08:57:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:13.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:13.123460+0000) 2022-09-10T08:57:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:13.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:13.638739+0000) 2022-09-10T08:57:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:13 smithi104 ceph-mon[97044]: pgmap v8577: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:57:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:13 smithi110 ceph-mon[97633]: pgmap v8577: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:57:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:14.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:14.123640+0000) 2022-09-10T08:57:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:14.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:14.638984+0000) 2022-09-10T08:57:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:15.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:15.123846+0000) 2022-09-10T08:57:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:15.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:15.639236+0000) 2022-09-10T08:57:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:15 smithi104 ceph-mon[97044]: pgmap v8578: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:57:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:15 smithi110 ceph-mon[97633]: pgmap v8578: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:57:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:16.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:16.124060+0000) 2022-09-10T08:57:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:16.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:16.639463+0000) 2022-09-10T08:57:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:17 smithi104 ceph-mon[97044]: pgmap v8579: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:57:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:17.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:17.124279+0000) 2022-09-10T08:57:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:17 smithi110 ceph-mon[97633]: pgmap v8579: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T08:57:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:17.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:17.639730+0000) 2022-09-10T08:57:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:18.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:18.124554+0000) 2022-09-10T08:57:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:18.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:18.639915+0000) 2022-09-10T08:57:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:19.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:19.124690+0000) 2022-09-10T08:57:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:19.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:19.640192+0000) 2022-09-10T08:57:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:19 smithi104 ceph-mon[97044]: pgmap v8580: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:57:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:19 smithi110 ceph-mon[97633]: pgmap v8580: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:57:20.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:57:20.343+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:57:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:20.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:20.124886+0000) 2022-09-10T08:57:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:20.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:20.640430+0000) 2022-09-10T08:57:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:21.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:21.125117+0000) 2022-09-10T08:57:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:21.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:21.640653+0000) 2022-09-10T08:57:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:21 smithi104 ceph-mon[97044]: pgmap v8581: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:57:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:21 smithi110 ceph-mon[97633]: pgmap v8581: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:57:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:22.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:22.125335+0000) 2022-09-10T08:57:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:22.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:22.640828+0000) 2022-09-10T08:57:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:23.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:23.125563+0000) 2022-09-10T08:57:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:23.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:23.641054+0000) 2022-09-10T08:57:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:23 smithi104 ceph-mon[97044]: pgmap v8582: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:57:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:23 smithi110 ceph-mon[97633]: pgmap v8582: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:57:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:24.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:24.125855+0000) 2022-09-10T08:57:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:24.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:24.641266+0000) 2022-09-10T08:57:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:25.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:25.126053+0000) 2022-09-10T08:57:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:25.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:25.641489+0000) 2022-09-10T08:57:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:25 smithi104 ceph-mon[97044]: pgmap v8583: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:57:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:25 smithi110 ceph-mon[97633]: pgmap v8583: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:57:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:26.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:26.126216+0000) 2022-09-10T08:57:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:26.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:26.641649+0000) 2022-09-10T08:57:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:27 smithi104 ceph-mon[97044]: pgmap v8584: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:27.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:27.126433+0000) 2022-09-10T08:57:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:27 smithi110 ceph-mon[97633]: pgmap v8584: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:27.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:27.641834+0000) 2022-09-10T08:57:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:28.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:28.126683+0000) 2022-09-10T08:57:29.066 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:28.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:28.641969+0000) 2022-09-10T08:57:29.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:29.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:29.126906+0000) 2022-09-10T08:57:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:29.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:29.642151+0000) 2022-09-10T08:57:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:29 smithi104 ceph-mon[97044]: pgmap v8585: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:29 smithi110 ceph-mon[97633]: pgmap v8585: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:30.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:30.127090+0000) 2022-09-10T08:57:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:30.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:30.642407+0000) 2022-09-10T08:57:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:31.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:31.127364+0000) 2022-09-10T08:57:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:31.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:31.642602+0000) 2022-09-10T08:57:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:31 smithi104 ceph-mon[97044]: pgmap v8586: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:31 smithi110 ceph-mon[97633]: pgmap v8586: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:32.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:32.127625+0000) 2022-09-10T08:57:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:32.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:32.642783+0000) 2022-09-10T08:57:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:33.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:33.127863+0000) 2022-09-10T08:57:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:33.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:33.643014+0000) 2022-09-10T08:57:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:33 smithi104 ceph-mon[97044]: pgmap v8587: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:33 smithi110 ceph-mon[97633]: pgmap v8587: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:34.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:34.128160+0000) 2022-09-10T08:57:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:34.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:34.643273+0000) 2022-09-10T08:57:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:57:35.348+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:57:35.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:35.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:35.128324+0000) 2022-09-10T08:57:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:35.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:35.643500+0000) 2022-09-10T08:57:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:35 smithi104 ceph-mon[97044]: pgmap v8588: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:35 smithi110 ceph-mon[97633]: pgmap v8588: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:36.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:36.128559+0000) 2022-09-10T08:57:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:36.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:36.643720+0000) 2022-09-10T08:57:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:37 smithi104 ceph-mon[97044]: pgmap v8589: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:37.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:37.128780+0000) 2022-09-10T08:57:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:37 smithi110 ceph-mon[97633]: pgmap v8589: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:37.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:37.643889+0000) 2022-09-10T08:57:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:38.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:38.128997+0000) 2022-09-10T08:57:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:38.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:38.644170+0000) 2022-09-10T08:57:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:39.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:39.129281+0000) 2022-09-10T08:57:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:39.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:39.644297+0000) 2022-09-10T08:57:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:39 smithi104 ceph-mon[97044]: pgmap v8590: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:39 smithi110 ceph-mon[97633]: pgmap v8590: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:40.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:40.129438+0000) 2022-09-10T08:57:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:40.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:40.644545+0000) 2022-09-10T08:57:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:41.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:41.129649+0000) 2022-09-10T08:57:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:41.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:41.644743+0000) 2022-09-10T08:57:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:41 smithi104 ceph-mon[97044]: pgmap v8591: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:41 smithi110 ceph-mon[97633]: pgmap v8591: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:42.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:42.129883+0000) 2022-09-10T08:57:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:42.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:42.644985+0000) 2022-09-10T08:57:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:43.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:43.130071+0000) 2022-09-10T08:57:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:43.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:43.645175+0000) 2022-09-10T08:57:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:43 smithi104 ceph-mon[97044]: pgmap v8592: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:43 smithi110 ceph-mon[97633]: pgmap v8592: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:44.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:44.130212+0000) 2022-09-10T08:57:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:44.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:44.645432+0000) 2022-09-10T08:57:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:57:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:57:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:57:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:57:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:57:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:57:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:57:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:57:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:45.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:45.130428+0000) 2022-09-10T08:57:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:45.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:45.645605+0000) 2022-09-10T08:57:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:45 smithi104 ceph-mon[97044]: pgmap v8593: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:45 smithi110 ceph-mon[97633]: pgmap v8593: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:46.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:46.130673+0000) 2022-09-10T08:57:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:46.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:46.645809+0000) 2022-09-10T08:57:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:47 smithi104 ceph-mon[97044]: pgmap v8594: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:47.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:47.130940+0000) 2022-09-10T08:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:47 smithi110 ceph-mon[97633]: pgmap v8594: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:47.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:47.645947+0000) 2022-09-10T08:57:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:48.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:48.131217+0000) 2022-09-10T08:57:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:48.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:48.646154+0000) 2022-09-10T08:57:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:49.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:49.131488+0000) 2022-09-10T08:57:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:49.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:49.646399+0000) 2022-09-10T08:57:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:49 smithi104 ceph-mon[97044]: pgmap v8595: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:49 smithi110 ceph-mon[97633]: pgmap v8595: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:50.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:50.131766+0000) 2022-09-10T08:57:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:57:50.363+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:57:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:50.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:50.646660+0000) 2022-09-10T08:57:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:51.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:51.132017+0000) 2022-09-10T08:57:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:51.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:51.646826+0000) 2022-09-10T08:57:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:51 smithi104 ceph-mon[97044]: pgmap v8596: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:51 smithi110 ceph-mon[97633]: pgmap v8596: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:52.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:52.132283+0000) 2022-09-10T08:57:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:52.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:52.646973+0000) 2022-09-10T08:57:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:53.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:53.132559+0000) 2022-09-10T08:57:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:53.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:53.647150+0000) 2022-09-10T08:57:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:53 smithi104 ceph-mon[97044]: pgmap v8597: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:53 smithi110 ceph-mon[97633]: pgmap v8597: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:54.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:54.132765+0000) 2022-09-10T08:57:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:54.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:54.647345+0000) 2022-09-10T08:57:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:55.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:55.132911+0000) 2022-09-10T08:57:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:55.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:55.647541+0000) 2022-09-10T08:57:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:55 smithi104 ceph-mon[97044]: pgmap v8598: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:55 smithi110 ceph-mon[97633]: pgmap v8598: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:57:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:56.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:56.133192+0000) 2022-09-10T08:57:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:56.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:56.647719+0000) 2022-09-10T08:57:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:57 smithi104 ceph-mon[97044]: pgmap v8599: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:57.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:57.133415+0000) 2022-09-10T08:57:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:57 smithi110 ceph-mon[97633]: pgmap v8599: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:57:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:57.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:57.647948+0000) 2022-09-10T08:57:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:58.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:58.133637+0000) 2022-09-10T08:57:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:58.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:58.648164+0000) 2022-09-10T08:57:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:57:59.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:59.133781+0000) 2022-09-10T08:58:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:57:59.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:57:59.648412+0000) 2022-09-10T08:58:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:57:59 smithi104 ceph-mon[97044]: pgmap v8600: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:57:59 smithi110 ceph-mon[97633]: pgmap v8600: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:00.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:00.134008+0000) 2022-09-10T08:58:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:00.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:00.648606+0000) 2022-09-10T08:58:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:01.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:01.134176+0000) 2022-09-10T08:58:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:01.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:01.648857+0000) 2022-09-10T08:58:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:01 smithi104 ceph-mon[97044]: pgmap v8601: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:01 smithi110 ceph-mon[97633]: pgmap v8601: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:02.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:02.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:02.134330+0000) 2022-09-10T08:58:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:02.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:02.648976+0000) 2022-09-10T08:58:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:03.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:03.134563+0000) 2022-09-10T08:58:04.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:03.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:03.649175+0000) 2022-09-10T08:58:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:03 smithi104 ceph-mon[97044]: pgmap v8602: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:03 smithi110 ceph-mon[97633]: pgmap v8602: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:04.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:04.134822+0000) 2022-09-10T08:58:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:04.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:04.649387+0000) 2022-09-10T08:58:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:05.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:05.135059+0000) 2022-09-10T08:58:05.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:58:05.378+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:58:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:05.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:05.649570+0000) 2022-09-10T08:58:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:05 smithi104 ceph-mon[97044]: pgmap v8603: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:58:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:05 smithi110 ceph-mon[97633]: pgmap v8603: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:58:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:06.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:06.135270+0000) 2022-09-10T08:58:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:06.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:06.649779+0000) 2022-09-10T08:58:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:07 smithi104 ceph-mon[97044]: pgmap v8604: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:58:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:07.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:07.135477+0000) 2022-09-10T08:58:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:07 smithi110 ceph-mon[97633]: pgmap v8604: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:58:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:07.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:07.649978+0000) 2022-09-10T08:58:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:08.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:08.135699+0000) 2022-09-10T08:58:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:08.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:08.650242+0000) 2022-09-10T08:58:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:09 smithi104 ceph-mon[97044]: osdmap e618: 12 total, 12 up, 12 in 2022-09-10T08:58:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:09 smithi104 ceph-mon[97044]: pgmap v8606: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:58:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:09.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:09.135940+0000) 2022-09-10T08:58:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:09 smithi110 ceph-mon[97633]: osdmap e618: 12 total, 12 up, 12 in 2022-09-10T08:58:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:09 smithi110 ceph-mon[97633]: pgmap v8606: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:58:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:09.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:09.650460+0000) 2022-09-10T08:58:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:10.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:10.136194+0000) 2022-09-10T08:58:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:10.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:10.650634+0000) 2022-09-10T08:58:11.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:11.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:11.136406+0000) 2022-09-10T08:58:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:11.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:11.650841+0000) 2022-09-10T08:58:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:11 smithi104 ceph-mon[97044]: pgmap v8607: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:58:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:11 smithi104 ceph-mon[97044]: osdmap e619: 12 total, 12 up, 12 in 2022-09-10T08:58:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:11 smithi110 ceph-mon[97633]: pgmap v8607: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T08:58:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:11 smithi110 ceph-mon[97633]: osdmap e619: 12 total, 12 up, 12 in 2022-09-10T08:58:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:12.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:12.136617+0000) 2022-09-10T08:58:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:12.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:12.651024+0000) 2022-09-10T08:58:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:13.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:13.136833+0000) 2022-09-10T08:58:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:13.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:13.651272+0000) 2022-09-10T08:58:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:13 smithi104 ceph-mon[97044]: pgmap v8609: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:58:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:13 smithi110 ceph-mon[97633]: pgmap v8609: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:58:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:14.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:14.137088+0000) 2022-09-10T08:58:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:14.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:14.651479+0000) 2022-09-10T08:58:15.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:15.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:15.137344+0000) 2022-09-10T08:58:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:15.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:15.651646+0000) 2022-09-10T08:58:16.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:15 smithi104 ceph-mon[97044]: pgmap v8610: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:58:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:15 smithi110 ceph-mon[97633]: pgmap v8610: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:58:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:16.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:16.137511+0000) 2022-09-10T08:58:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:16.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:16.651862+0000) 2022-09-10T08:58:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:17 smithi104 ceph-mon[97044]: pgmap v8611: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2022-09-10T08:58:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:17.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:17.137782+0000) 2022-09-10T08:58:17.669 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:17 smithi110 ceph-mon[97633]: pgmap v8611: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2022-09-10T08:58:18.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:17.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:17.652083+0000) 2022-09-10T08:58:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:18.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:18.137937+0000) 2022-09-10T08:58:19.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:18.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:18.652341+0000) 2022-09-10T08:58:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:19.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:19.138177+0000) 2022-09-10T08:58:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:19.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:19.652564+0000) 2022-09-10T08:58:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:19 smithi104 ceph-mon[97044]: pgmap v8612: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:58:20.393 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:20.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:20.138407+0000) 2022-09-10T08:58:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:19 smithi110 ceph-mon[97633]: pgmap v8612: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:58:20.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:58:20.391+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:58:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:20.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:20.652776+0000) 2022-09-10T08:58:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:21.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:21.138634+0000) 2022-09-10T08:58:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:21.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:21.652988+0000) 2022-09-10T08:58:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:21 smithi104 ceph-mon[97044]: pgmap v8613: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:58:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:21 smithi110 ceph-mon[97633]: pgmap v8613: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:58:22.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:22.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:22.138844+0000) 2022-09-10T08:58:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:22.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:22.653215+0000) 2022-09-10T08:58:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:23.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:23.139086+0000) 2022-09-10T08:58:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:23.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:23.653378+0000) 2022-09-10T08:58:24.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:23 smithi104 ceph-mon[97044]: pgmap v8614: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:58:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:23 smithi110 ceph-mon[97633]: pgmap v8614: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:58:24.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:24.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:24.139376+0000) 2022-09-10T08:58:25.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:24.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:24.653645+0000) 2022-09-10T08:58:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:25.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:25.139575+0000) 2022-09-10T08:58:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:25.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:25.653859+0000) 2022-09-10T08:58:26.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:25 smithi104 ceph-mon[97044]: pgmap v8615: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:58:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:25 smithi110 ceph-mon[97633]: pgmap v8615: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:58:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:26.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:26.139746+0000) 2022-09-10T08:58:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:26.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:26.654034+0000) 2022-09-10T08:58:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:27 smithi104 ceph-mon[97044]: pgmap v8616: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:27.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:27.139973+0000) 2022-09-10T08:58:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:27 smithi110 ceph-mon[97633]: pgmap v8616: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:27.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:27.654197+0000) 2022-09-10T08:58:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:28.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:28.140167+0000) 2022-09-10T08:58:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:28.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:28.654409+0000) 2022-09-10T08:58:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:29.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:29.140376+0000) 2022-09-10T08:58:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:29.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:29.654610+0000) 2022-09-10T08:58:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:29 smithi104 ceph-mon[97044]: pgmap v8617: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:29 smithi110 ceph-mon[97633]: pgmap v8617: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:30.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:30.140633+0000) 2022-09-10T08:58:31.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:30.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:30.654839+0000) 2022-09-10T08:58:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:31.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:31.140862+0000) 2022-09-10T08:58:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:31.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:31.655049+0000) 2022-09-10T08:58:32.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:31 smithi104 ceph-mon[97044]: pgmap v8618: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:31 smithi110 ceph-mon[97633]: pgmap v8618: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:32.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:32.141141+0000) 2022-09-10T08:58:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:32.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:32.655309+0000) 2022-09-10T08:58:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:33.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:33.141323+0000) 2022-09-10T08:58:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:33.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:33.655537+0000) 2022-09-10T08:58:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:33 smithi104 ceph-mon[97044]: pgmap v8619: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:33 smithi110 ceph-mon[97633]: pgmap v8619: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:34.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:34.141553+0000) 2022-09-10T08:58:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:34.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:34.655714+0000) 2022-09-10T08:58:35.407 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:35.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:35.141766+0000) 2022-09-10T08:58:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:35.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:35.655892+0000) 2022-09-10T08:58:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:58:35.405+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:58:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:35 smithi104 ceph-mon[97044]: pgmap v8620: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:35 smithi110 ceph-mon[97633]: pgmap v8620: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:36.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:36.142015+0000) 2022-09-10T08:58:37.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:36.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:36.656175+0000) 2022-09-10T08:58:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:37 smithi104 ceph-mon[97044]: pgmap v8621: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:37.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:37.142243+0000) 2022-09-10T08:58:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:37 smithi110 ceph-mon[97633]: pgmap v8621: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:37.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:37.656373+0000) 2022-09-10T08:58:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:38.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:38.142414+0000) 2022-09-10T08:58:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:38.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:38.656559+0000) 2022-09-10T08:58:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:39.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:39.142645+0000) 2022-09-10T08:58:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:39.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:39.656795+0000) 2022-09-10T08:58:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:39 smithi104 ceph-mon[97044]: pgmap v8622: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:39 smithi110 ceph-mon[97633]: pgmap v8622: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:40.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:40.142905+0000) 2022-09-10T08:58:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:40.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:40.657091+0000) 2022-09-10T08:58:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:41.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:41.143118+0000) 2022-09-10T08:58:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:41.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:41.657355+0000) 2022-09-10T08:58:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:41 smithi104 ceph-mon[97044]: pgmap v8623: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:41 smithi110 ceph-mon[97633]: pgmap v8623: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:42.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:42.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:42.143294+0000) 2022-09-10T08:58:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:42.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:42.657582+0000) 2022-09-10T08:58:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:43.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:43.143566+0000) 2022-09-10T08:58:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:43.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:43.657837+0000) 2022-09-10T08:58:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:43 smithi104 ceph-mon[97044]: pgmap v8624: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:43 smithi110 ceph-mon[97633]: pgmap v8624: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:44.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:44.143817+0000) 2022-09-10T08:58:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:44.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:44.658123+0000) 2022-09-10T08:58:45.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:58:45.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:58:45.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:58:45.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:58:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:58:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:58:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:58:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:58:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:45.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:45.144048+0000) 2022-09-10T08:58:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:45.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:45.658356+0000) 2022-09-10T08:58:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:45 smithi104 ceph-mon[97044]: pgmap v8625: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:45 smithi110 ceph-mon[97633]: pgmap v8625: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:46.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:46.144262+0000) 2022-09-10T08:58:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:46.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:46.658591+0000) 2022-09-10T08:58:47.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:47 smithi104 ceph-mon[97044]: pgmap v8626: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:58:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:47.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:47.144504+0000) 2022-09-10T08:58:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:47 smithi110 ceph-mon[97633]: pgmap v8626: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:58:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:47.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:47.658837+0000) 2022-09-10T08:58:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:48.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:48.144744+0000) 2022-09-10T08:58:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:48.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:48.659084+0000) 2022-09-10T08:58:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:49.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:49.144966+0000) 2022-09-10T08:58:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:49.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:49.659293+0000) 2022-09-10T08:58:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:49 smithi104 ceph-mon[97044]: pgmap v8627: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:58:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:49 smithi110 ceph-mon[97633]: pgmap v8627: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:58:50.421 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:50.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:50.145165+0000) 2022-09-10T08:58:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:50.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:50.659510+0000) 2022-09-10T08:58:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:58:50.419+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:58:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:51.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:51.145325+0000) 2022-09-10T08:58:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:51.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:51.659753+0000) 2022-09-10T08:58:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:51 smithi104 ceph-mon[97044]: pgmap v8628: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:51 smithi110 ceph-mon[97633]: pgmap v8628: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:52.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:52.145562+0000) 2022-09-10T08:58:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:52.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:52.659995+0000) 2022-09-10T08:58:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:53.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:53.145775+0000) 2022-09-10T08:58:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:53.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:53.660248+0000) 2022-09-10T08:58:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:53 smithi104 ceph-mon[97044]: pgmap v8629: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:58:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:53 smithi110 ceph-mon[97633]: pgmap v8629: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T08:58:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:54.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:54.145963+0000) 2022-09-10T08:58:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:54.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:54.660476+0000) 2022-09-10T08:58:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:55.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:55.146198+0000) 2022-09-10T08:58:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:55.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:55.660746+0000) 2022-09-10T08:58:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:55 smithi104 ceph-mon[97044]: pgmap v8630: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:55 smithi110 ceph-mon[97633]: pgmap v8630: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:58:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:56.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:56.146379+0000) 2022-09-10T08:58:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:56.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:56.661006+0000) 2022-09-10T08:58:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:57 smithi104 ceph-mon[97044]: pgmap v8631: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:57.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:57.146564+0000) 2022-09-10T08:58:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:57 smithi110 ceph-mon[97633]: pgmap v8631: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:58:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:57.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:57.661258+0000) 2022-09-10T08:58:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:58.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:58.146831+0000) 2022-09-10T08:58:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:58.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:58.661466+0000) 2022-09-10T08:58:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:58:59.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:59.147046+0000) 2022-09-10T08:59:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:58:59.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:58:59.661670+0000) 2022-09-10T08:59:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:58:59 smithi104 ceph-mon[97044]: pgmap v8632: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:58:59 smithi110 ceph-mon[97633]: pgmap v8632: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:00.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:00.147157+0000) 2022-09-10T08:59:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:00.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:00.661920+0000) 2022-09-10T08:59:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:01.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:01.147371+0000) 2022-09-10T08:59:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:01.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:01.662170+0000) 2022-09-10T08:59:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:01 smithi104 ceph-mon[97044]: pgmap v8633: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:01 smithi110 ceph-mon[97633]: pgmap v8633: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:02.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:02.147626+0000) 2022-09-10T08:59:03.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:02.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:02.662399+0000) 2022-09-10T08:59:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:03.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:03.147816+0000) 2022-09-10T08:59:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:03.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:03.662643+0000) 2022-09-10T08:59:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:03 smithi104 ceph-mon[97044]: pgmap v8634: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:03 smithi110 ceph-mon[97633]: pgmap v8634: 129 pgs: 129 active+clean; 8.6 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:04.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:04.148058+0000) 2022-09-10T08:59:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:04.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:04.662836+0000) 2022-09-10T08:59:05.427 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:05.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:05.148338+0000) 2022-09-10T08:59:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:05.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:05.663071+0000) 2022-09-10T08:59:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:59:05.425+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:59:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:05 smithi104 ceph-mon[97044]: pgmap v8635: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:59:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:05 smithi110 ceph-mon[97633]: pgmap v8635: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T08:59:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:06.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:06.148511+0000) 2022-09-10T08:59:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:06.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:06.663334+0000) 2022-09-10T08:59:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:07 smithi104 ceph-mon[97044]: pgmap v8636: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:59:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:07.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:07.148720+0000) 2022-09-10T08:59:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:07 smithi110 ceph-mon[97633]: pgmap v8636: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T08:59:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:07.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:07.663558+0000) 2022-09-10T08:59:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:08.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:08.148947+0000) 2022-09-10T08:59:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:08.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:08.663758+0000) 2022-09-10T08:59:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:09 smithi104 ceph-mon[97044]: osdmap e620: 12 total, 12 up, 12 in 2022-09-10T08:59:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:09 smithi104 ceph-mon[97044]: pgmap v8638: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:59:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:09.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:09.149219+0000) 2022-09-10T08:59:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:09 smithi110 ceph-mon[97633]: osdmap e620: 12 total, 12 up, 12 in 2022-09-10T08:59:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:09 smithi110 ceph-mon[97633]: pgmap v8638: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:59:10.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:09.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:09.663949+0000) 2022-09-10T08:59:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:10.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:10.149451+0000) 2022-09-10T08:59:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:10.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:10.664140+0000) 2022-09-10T08:59:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:11.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:11.149655+0000) 2022-09-10T08:59:12.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:11.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:11.664359+0000) 2022-09-10T08:59:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:11 smithi104 ceph-mon[97044]: pgmap v8639: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:59:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:11 smithi104 ceph-mon[97044]: osdmap e621: 12 total, 12 up, 12 in 2022-09-10T08:59:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:11 smithi110 ceph-mon[97633]: pgmap v8639: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T08:59:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:11 smithi110 ceph-mon[97633]: osdmap e621: 12 total, 12 up, 12 in 2022-09-10T08:59:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:12.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:12.149821+0000) 2022-09-10T08:59:13.012 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:12.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:12.664599+0000) 2022-09-10T08:59:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:13.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:13.149965+0000) 2022-09-10T08:59:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:13.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:13.664826+0000) 2022-09-10T08:59:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:13 smithi104 ceph-mon[97044]: pgmap v8641: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:59:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:13 smithi110 ceph-mon[97633]: pgmap v8641: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T08:59:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:14.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:14.150212+0000) 2022-09-10T08:59:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:14.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:14.664993+0000) 2022-09-10T08:59:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:15.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:15.150435+0000) 2022-09-10T08:59:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:15.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:15.665262+0000) 2022-09-10T08:59:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:16 smithi104 ceph-mon[97044]: pgmap v8642: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:59:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:16 smithi110 ceph-mon[97633]: pgmap v8642: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T08:59:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:16.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:16.150689+0000) 2022-09-10T08:59:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:16.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:16.665513+0000) 2022-09-10T08:59:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:17 smithi104 ceph-mon[97044]: pgmap v8643: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2022-09-10T08:59:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:17.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:17.150945+0000) 2022-09-10T08:59:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:17 smithi110 ceph-mon[97633]: pgmap v8643: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2022-09-10T08:59:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:17.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:17.665712+0000) 2022-09-10T08:59:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:18.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:18.151197+0000) 2022-09-10T08:59:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:18.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:18.665861+0000) 2022-09-10T08:59:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:19.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:19.151439+0000) 2022-09-10T08:59:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:19.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:19.666075+0000) 2022-09-10T08:59:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:19 smithi104 ceph-mon[97044]: pgmap v8644: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:59:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:19 smithi110 ceph-mon[97633]: pgmap v8644: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T08:59:20.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:20.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:20.151653+0000) 2022-09-10T08:59:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:20.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:20.666239+0000) 2022-09-10T08:59:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:59:20.437+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:59:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:21.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:21.151847+0000) 2022-09-10T08:59:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:21.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:21.666436+0000) 2022-09-10T08:59:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:22 smithi104 ceph-mon[97044]: pgmap v8645: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:59:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:22 smithi110 ceph-mon[97633]: pgmap v8645: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T08:59:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:22.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:22.152056+0000) 2022-09-10T08:59:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:22.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:22.666611+0000) 2022-09-10T08:59:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:23.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:23.152269+0000) 2022-09-10T08:59:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:23.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:23.666845+0000) 2022-09-10T08:59:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:24 smithi104 ceph-mon[97044]: pgmap v8646: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:59:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:24 smithi110 ceph-mon[97633]: pgmap v8646: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T08:59:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:24.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:24.152496+0000) 2022-09-10T08:59:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:24.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:24.667074+0000) 2022-09-10T08:59:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:25.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:25.152657+0000) 2022-09-10T08:59:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:25.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:25.667275+0000) 2022-09-10T08:59:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:26 smithi104 ceph-mon[97044]: pgmap v8647: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:59:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:26 smithi110 ceph-mon[97633]: pgmap v8647: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T08:59:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:26.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:26.152842+0000) 2022-09-10T08:59:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:26.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:26.667445+0000) 2022-09-10T08:59:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:27 smithi104 ceph-mon[97044]: pgmap v8648: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:27.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:27.153073+0000) 2022-09-10T08:59:27.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:27 smithi110 ceph-mon[97633]: pgmap v8648: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:27.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:27.667618+0000) 2022-09-10T08:59:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:28.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:28.153329+0000) 2022-09-10T08:59:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:28.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:28.667879+0000) 2022-09-10T08:59:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:29.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:29.153540+0000) 2022-09-10T08:59:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:29.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:29.668141+0000) 2022-09-10T08:59:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:30 smithi104 ceph-mon[97044]: pgmap v8649: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:30 smithi110 ceph-mon[97633]: pgmap v8649: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:30.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:30.153766+0000) 2022-09-10T08:59:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:30.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:30.668382+0000) 2022-09-10T08:59:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:31.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:31.153978+0000) 2022-09-10T08:59:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:31.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:31.668645+0000) 2022-09-10T08:59:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:32 smithi104 ceph-mon[97044]: pgmap v8650: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:32 smithi110 ceph-mon[97633]: pgmap v8650: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:32.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:32.154133+0000) 2022-09-10T08:59:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:32.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:32.668894+0000) 2022-09-10T08:59:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:33.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:33.154293+0000) 2022-09-10T08:59:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:33.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:33.669157+0000) 2022-09-10T08:59:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:34 smithi104 ceph-mon[97044]: pgmap v8651: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:34 smithi110 ceph-mon[97633]: pgmap v8651: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:34.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:34.154479+0000) 2022-09-10T08:59:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:34.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:34.669409+0000) 2022-09-10T08:59:35.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:35.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:35.154723+0000) 2022-09-10T08:59:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:35.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:35.669600+0000) 2022-09-10T08:59:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:59:35.453+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:59:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:36 smithi104 ceph-mon[97044]: pgmap v8652: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:36 smithi110 ceph-mon[97633]: pgmap v8652: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:36.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:36.154960+0000) 2022-09-10T08:59:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:36.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:36.669822+0000) 2022-09-10T08:59:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:37 smithi104 ceph-mon[97044]: pgmap v8653: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:37.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:37.155108+0000) 2022-09-10T08:59:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:37 smithi110 ceph-mon[97633]: pgmap v8653: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:37.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:37.670067+0000) 2022-09-10T08:59:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:38.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:38.155288+0000) 2022-09-10T08:59:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:38.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:38.670315+0000) 2022-09-10T08:59:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:39.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:39.155518+0000) 2022-09-10T08:59:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:39.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:39.670572+0000) 2022-09-10T08:59:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:40 smithi104 ceph-mon[97044]: pgmap v8654: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:40 smithi110 ceph-mon[97633]: pgmap v8654: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:40.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:40.155759+0000) 2022-09-10T08:59:41.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:40.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:40.670804+0000) 2022-09-10T08:59:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:41.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:41.156018+0000) 2022-09-10T08:59:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:41.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:41.670974+0000) 2022-09-10T08:59:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:42 smithi104 ceph-mon[97044]: pgmap v8655: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:42 smithi110 ceph-mon[97633]: pgmap v8655: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:42.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:42.156271+0000) 2022-09-10T08:59:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:42.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:42.671182+0000) 2022-09-10T08:59:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:43.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:43.156407+0000) 2022-09-10T08:59:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:43.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:43.671441+0000) 2022-09-10T08:59:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:44 smithi104 ceph-mon[97044]: pgmap v8656: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:44 smithi110 ceph-mon[97633]: pgmap v8656: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:44.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:44.156599+0000) 2022-09-10T08:59:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:44.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:44.671614+0000) 2022-09-10T08:59:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:59:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:59:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:59:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:59:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:59:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:59:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T08:59:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T08:59:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:45.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:45.156825+0000) 2022-09-10T08:59:46.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:45.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:45.671848+0000) 2022-09-10T08:59:46.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:46 smithi104 ceph-mon[97044]: pgmap v8657: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:46 smithi110 ceph-mon[97633]: pgmap v8657: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:46.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:46.157001+0000) 2022-09-10T08:59:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:46.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:46.672124+0000) 2022-09-10T08:59:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:47 smithi104 ceph-mon[97044]: pgmap v8658: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:47.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:47.157190+0000) 2022-09-10T08:59:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:47 smithi110 ceph-mon[97633]: pgmap v8658: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:47.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:47.672346+0000) 2022-09-10T08:59:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:48.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:48.157314+0000) 2022-09-10T08:59:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:48.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:48.672494+0000) 2022-09-10T08:59:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:49.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:49.157509+0000) 2022-09-10T08:59:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:49.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:49.672747+0000) 2022-09-10T08:59:50.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:50 smithi104 ceph-mon[97044]: pgmap v8659: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:50 smithi110 ceph-mon[97633]: pgmap v8659: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:50.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:50.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:50.157714+0000) 2022-09-10T08:59:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:50.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:50.672917+0000) 2022-09-10T08:59:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 08:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T08:59:50.462+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T08:59:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:51.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:51.157921+0000) 2022-09-10T08:59:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:51.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:51.673158+0000) 2022-09-10T08:59:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:52 smithi104 ceph-mon[97044]: pgmap v8660: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:52 smithi110 ceph-mon[97633]: pgmap v8660: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:52.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:52.158163+0000) 2022-09-10T08:59:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:52.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:52.673348+0000) 2022-09-10T08:59:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:53.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:53.158390+0000) 2022-09-10T08:59:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:53.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:53.673628+0000) 2022-09-10T08:59:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:54 smithi104 ceph-mon[97044]: pgmap v8661: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:54 smithi110 ceph-mon[97633]: pgmap v8661: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:54.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:54.158610+0000) 2022-09-10T08:59:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:54.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:54.673793+0000) 2022-09-10T08:59:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:55.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:55.158812+0000) 2022-09-10T08:59:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:55.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:55.673916+0000) 2022-09-10T08:59:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:56 smithi104 ceph-mon[97044]: pgmap v8662: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:56 smithi110 ceph-mon[97633]: pgmap v8662: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T08:59:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:56.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:56.159043+0000) 2022-09-10T08:59:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:56.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:56.674129+0000) 2022-09-10T08:59:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 08:59:57 smithi104 ceph-mon[97044]: pgmap v8663: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:57.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:57.159270+0000) 2022-09-10T08:59:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 08:59:57 smithi110 ceph-mon[97633]: pgmap v8663: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T08:59:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:57.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:57.674356+0000) 2022-09-10T08:59:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:58.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:58.159469+0000) 2022-09-10T08:59:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:58.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:58.674558+0000) 2022-09-10T08:59:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 08:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T08:59:59.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:59.159718+0000) 2022-09-10T09:00:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 08:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T08:59:59.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T07:59:59.674793+0000) 2022-09-10T09:00:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:00 smithi104 ceph-mon[97044]: pgmap v8664: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:00:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:00 smithi110 ceph-mon[97633]: pgmap v8664: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:00:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:00.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:00.159886+0000) 2022-09-10T09:00:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:00.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:00.674937+0000) 2022-09-10T09:00:01.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:01.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:01.160150+0000) 2022-09-10T09:00:02.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:01.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:01.675181+0000) 2022-09-10T09:00:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:02 smithi104 ceph-mon[97044]: pgmap v8665: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:02 smithi110 ceph-mon[97633]: pgmap v8665: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:02.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:02.160315+0000) 2022-09-10T09:00:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:02.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:02.675414+0000) 2022-09-10T09:00:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:03.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:03.160556+0000) 2022-09-10T09:00:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:03.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:03.675642+0000) 2022-09-10T09:00:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:04 smithi104 ceph-mon[97044]: pgmap v8666: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:04 smithi110 ceph-mon[97633]: pgmap v8666: 129 pgs: 129 active+clean; 8.6 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:04.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:04.160806+0000) 2022-09-10T09:00:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:04.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:04.675826+0000) 2022-09-10T09:00:05.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:05.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:05.161052+0000) 2022-09-10T09:00:05.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:00:05.477+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:00:05.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:05.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:05.676061+0000) 2022-09-10T09:00:06.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:06 smithi104 ceph-mon[97044]: pgmap v8667: 129 pgs: 129 active+clean; 8.6 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:00:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:06 smithi110 ceph-mon[97633]: pgmap v8667: 129 pgs: 129 active+clean; 8.6 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:00:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:06.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:06.161312+0000) 2022-09-10T09:00:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:06.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:06.676300+0000) 2022-09-10T09:00:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:07 smithi104 ceph-mon[97044]: pgmap v8668: 129 pgs: 129 active+clean; 8.6 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:00:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:07.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:07.161546+0000) 2022-09-10T09:00:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:07 smithi110 ceph-mon[97633]: pgmap v8668: 129 pgs: 129 active+clean; 8.6 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:00:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:07.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:07.676474+0000) 2022-09-10T09:00:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:08.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:08.161813+0000) 2022-09-10T09:00:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:08.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:08.676656+0000) 2022-09-10T09:00:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:09 smithi104 ceph-mon[97044]: osdmap e622: 12 total, 12 up, 12 in 2022-09-10T09:00:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:09 smithi104 ceph-mon[97044]: pgmap v8670: 129 pgs: 129 active+clean; 8.6 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:00:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:09.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:09.162089+0000) 2022-09-10T09:00:09.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:09 smithi110 ceph-mon[97633]: osdmap e622: 12 total, 12 up, 12 in 2022-09-10T09:00:09.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:09 smithi110 ceph-mon[97633]: pgmap v8670: 129 pgs: 129 active+clean; 8.6 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:00:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:09.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:09.676914+0000) 2022-09-10T09:00:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:10.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:10.162321+0000) 2022-09-10T09:00:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:10.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:10.677193+0000) 2022-09-10T09:00:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:11.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:11.162500+0000) 2022-09-10T09:00:12.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:11.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:11.677374+0000) 2022-09-10T09:00:12.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:12 smithi104 ceph-mon[97044]: pgmap v8671: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:00:12.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:12 smithi104 ceph-mon[97044]: osdmap e623: 12 total, 12 up, 12 in 2022-09-10T09:00:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:12 smithi110 ceph-mon[97633]: pgmap v8671: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:00:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:12 smithi110 ceph-mon[97633]: osdmap e623: 12 total, 12 up, 12 in 2022-09-10T09:00:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:12.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:12.162698+0000) 2022-09-10T09:00:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:12.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:12.677682+0000) 2022-09-10T09:00:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:13.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:13.162886+0000) 2022-09-10T09:00:13.941 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:13.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:13.677942+0000) 2022-09-10T09:00:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:14 smithi104 ceph-mon[97044]: pgmap v8673: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:00:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:14 smithi110 ceph-mon[97633]: pgmap v8673: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:00:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:14.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:14.163157+0000) 2022-09-10T09:00:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:14.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:14.678231+0000) 2022-09-10T09:00:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:15.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:15.163376+0000) 2022-09-10T09:00:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:15.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:15.678530+0000) 2022-09-10T09:00:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:16 smithi104 ceph-mon[97044]: pgmap v8674: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:00:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:16 smithi110 ceph-mon[97633]: pgmap v8674: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:00:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:16.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:16.163612+0000) 2022-09-10T09:00:17.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:16.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:16.678814+0000) 2022-09-10T09:00:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:17 smithi104 ceph-mon[97044]: pgmap v8675: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2022-09-10T09:00:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:17.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:17.163874+0000) 2022-09-10T09:00:17.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:17 smithi110 ceph-mon[97633]: pgmap v8675: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 11 op/s 2022-09-10T09:00:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:17.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:17.679087+0000) 2022-09-10T09:00:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:18.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:18.164123+0000) 2022-09-10T09:00:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:18.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:18.679352+0000) 2022-09-10T09:00:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:19.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:19.164372+0000) 2022-09-10T09:00:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:19.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:19.679570+0000) 2022-09-10T09:00:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:20 smithi104 ceph-mon[97044]: pgmap v8676: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:00:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:20 smithi110 ceph-mon[97633]: pgmap v8676: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:00:20.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:20.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:20.164567+0000) 2022-09-10T09:00:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:20.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:20.679731+0000) 2022-09-10T09:00:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:00:20.493+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:00:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:21.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:21.164758+0000) 2022-09-10T09:00:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:21.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:21.679945+0000) 2022-09-10T09:00:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:22 smithi104 ceph-mon[97044]: pgmap v8677: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:00:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:22 smithi110 ceph-mon[97633]: pgmap v8677: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:00:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:22.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:22.165004+0000) 2022-09-10T09:00:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:22.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:22.680179+0000) 2022-09-10T09:00:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:23.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:23.165223+0000) 2022-09-10T09:00:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:23.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:23.680332+0000) 2022-09-10T09:00:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:24 smithi104 ceph-mon[97044]: pgmap v8678: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:00:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:24 smithi110 ceph-mon[97633]: pgmap v8678: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:00:24.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:24.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:24.165388+0000) 2022-09-10T09:00:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:24.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:24.680596+0000) 2022-09-10T09:00:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:25.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:25.165574+0000) 2022-09-10T09:00:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:25.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:25.680863+0000) 2022-09-10T09:00:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:26 smithi104 ceph-mon[97044]: pgmap v8679: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:00:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:26 smithi110 ceph-mon[97633]: pgmap v8679: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:00:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:26.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:26.165742+0000) 2022-09-10T09:00:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:26.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:26.681164+0000) 2022-09-10T09:00:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:27 smithi104 ceph-mon[97044]: pgmap v8680: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:00:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:27.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:27.165961+0000) 2022-09-10T09:00:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:27 smithi110 ceph-mon[97633]: pgmap v8680: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:00:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:27.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:27.681301+0000) 2022-09-10T09:00:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:28.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:28.166178+0000) 2022-09-10T09:00:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:28.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:28.681554+0000) 2022-09-10T09:00:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:29.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:29.166412+0000) 2022-09-10T09:00:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:29.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:29.681741+0000) 2022-09-10T09:00:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:30 smithi104 ceph-mon[97044]: pgmap v8681: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:00:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:30 smithi110 ceph-mon[97633]: pgmap v8681: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:00:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:30.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:30.166661+0000) 2022-09-10T09:00:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:30.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:30.681965+0000) 2022-09-10T09:00:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:31.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:31.166904+0000) 2022-09-10T09:00:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:31.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:31.682157+0000) 2022-09-10T09:00:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:32 smithi104 ceph-mon[97044]: pgmap v8682: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:32 smithi110 ceph-mon[97633]: pgmap v8682: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:32.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:32.167188+0000) 2022-09-10T09:00:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:32.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:32.682432+0000) 2022-09-10T09:00:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:33.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:33.167392+0000) 2022-09-10T09:00:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:33.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:33.682699+0000) 2022-09-10T09:00:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:34 smithi104 ceph-mon[97044]: pgmap v8683: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:00:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:34 smithi110 ceph-mon[97633]: pgmap v8683: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:00:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:34.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:34.167599+0000) 2022-09-10T09:00:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:34.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:34.682999+0000) 2022-09-10T09:00:35.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:35.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:35.167837+0000) 2022-09-10T09:00:35.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:00:35.503+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:00:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:35.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:35.683281+0000) 2022-09-10T09:00:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:36 smithi104 ceph-mon[97044]: pgmap v8684: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:36 smithi110 ceph-mon[97633]: pgmap v8684: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:36.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:36.168131+0000) 2022-09-10T09:00:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:36.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:36.683492+0000) 2022-09-10T09:00:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:37 smithi104 ceph-mon[97044]: pgmap v8685: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:37.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:37.168347+0000) 2022-09-10T09:00:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:37 smithi110 ceph-mon[97633]: pgmap v8685: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:37.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:37.683712+0000) 2022-09-10T09:00:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:38.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:38.168477+0000) 2022-09-10T09:00:39.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:38.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:38.683913+0000) 2022-09-10T09:00:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:39.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:39.168727+0000) 2022-09-10T09:00:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:39.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:39.684186+0000) 2022-09-10T09:00:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:40 smithi104 ceph-mon[97044]: pgmap v8686: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:40 smithi110 ceph-mon[97633]: pgmap v8686: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:40.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:40.168920+0000) 2022-09-10T09:00:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:40.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:40.684364+0000) 2022-09-10T09:00:41.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:41.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:41.169134+0000) 2022-09-10T09:00:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:41.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:41.684600+0000) 2022-09-10T09:00:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:42 smithi104 ceph-mon[97044]: pgmap v8687: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:42 smithi110 ceph-mon[97633]: pgmap v8687: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:42.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:42.169382+0000) 2022-09-10T09:00:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:42.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:42.684794+0000) 2022-09-10T09:00:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:43.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:43.169643+0000) 2022-09-10T09:00:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:43.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:43.684971+0000) 2022-09-10T09:00:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:44 smithi104 ceph-mon[97044]: pgmap v8688: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:44 smithi110 ceph-mon[97633]: pgmap v8688: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:44.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:44.169819+0000) 2022-09-10T09:00:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:44.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:44.685185+0000) 2022-09-10T09:00:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:00:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:00:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:00:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:00:45.421 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:00:45.422 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:00:45.422 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:00:45.422 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:00:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:45.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:45.170016+0000) 2022-09-10T09:00:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:45.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:45.685414+0000) 2022-09-10T09:00:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:46 smithi104 ceph-mon[97044]: pgmap v8689: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:46 smithi110 ceph-mon[97633]: pgmap v8689: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:46.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:46.170159+0000) 2022-09-10T09:00:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:46.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:46.685655+0000) 2022-09-10T09:00:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:47 smithi104 ceph-mon[97044]: pgmap v8690: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:47.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:47.170372+0000) 2022-09-10T09:00:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:47 smithi110 ceph-mon[97633]: pgmap v8690: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:47.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:47.685892+0000) 2022-09-10T09:00:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:48.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:48.170634+0000) 2022-09-10T09:00:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:48.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:48.686188+0000) 2022-09-10T09:00:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:49.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:49.170899+0000) 2022-09-10T09:00:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:49.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:49.686431+0000) 2022-09-10T09:00:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:50 smithi104 ceph-mon[97044]: pgmap v8691: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:50 smithi110 ceph-mon[97633]: pgmap v8691: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:50.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:50.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:50.171150+0000) 2022-09-10T09:00:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:50.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:50.686676+0000) 2022-09-10T09:00:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:00:50.507+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:00:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:51.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:51.171348+0000) 2022-09-10T09:00:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:51.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:51.686935+0000) 2022-09-10T09:00:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:52 smithi104 ceph-mon[97044]: pgmap v8692: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:52 smithi110 ceph-mon[97633]: pgmap v8692: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:52.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:52.171564+0000) 2022-09-10T09:00:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:52.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:52.687182+0000) 2022-09-10T09:00:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:53.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:53.171796+0000) 2022-09-10T09:00:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:53.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:53.687394+0000) 2022-09-10T09:00:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:54 smithi104 ceph-mon[97044]: pgmap v8693: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:54 smithi110 ceph-mon[97633]: pgmap v8693: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:54.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:54.172055+0000) 2022-09-10T09:00:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:54.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:54.687633+0000) 2022-09-10T09:00:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:55.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:55.172166+0000) 2022-09-10T09:00:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:55.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:55.687820+0000) 2022-09-10T09:00:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:56 smithi104 ceph-mon[97044]: pgmap v8694: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:56 smithi110 ceph-mon[97633]: pgmap v8694: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:00:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:56.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:56.172348+0000) 2022-09-10T09:00:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:56.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:56.688024+0000) 2022-09-10T09:00:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:00:57 smithi104 ceph-mon[97044]: pgmap v8695: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:57.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:57.172533+0000) 2022-09-10T09:00:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:00:57 smithi110 ceph-mon[97633]: pgmap v8695: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:00:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:57.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:57.688267+0000) 2022-09-10T09:00:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:58.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:58.172802+0000) 2022-09-10T09:00:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:58.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:58.688473+0000) 2022-09-10T09:00:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:00:59.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:59.173020+0000) 2022-09-10T09:01:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:00:59.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:00:59.688777+0000) 2022-09-10T09:01:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:00 smithi104 ceph-mon[97044]: pgmap v8696: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:00 smithi110 ceph-mon[97633]: pgmap v8696: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:00.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:00.173239+0000) 2022-09-10T09:01:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:00.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:00.688961+0000) 2022-09-10T09:01:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:01.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:01.173460+0000) 2022-09-10T09:01:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:01.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:01.689190+0000) 2022-09-10T09:01:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:02 smithi104 ceph-mon[97044]: pgmap v8697: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:02 smithi110 ceph-mon[97633]: pgmap v8697: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:02.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:02.173649+0000) 2022-09-10T09:01:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:02.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:02.689391+0000) 2022-09-10T09:01:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:03.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:03.173859+0000) 2022-09-10T09:01:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:03.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:03.689610+0000) 2022-09-10T09:01:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:04 smithi104 ceph-mon[97044]: pgmap v8698: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:04 smithi110 ceph-mon[97633]: pgmap v8698: 129 pgs: 129 active+clean; 8.6 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:04.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:04.174054+0000) 2022-09-10T09:01:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:04.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:04.689869+0000) 2022-09-10T09:01:05.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:05.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:05.174263+0000) 2022-09-10T09:01:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:05.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:05.690126+0000) 2022-09-10T09:01:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:01:05.513+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:01:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:06 smithi104 ceph-mon[97044]: pgmap v8699: 129 pgs: 129 active+clean; 8.6 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:01:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:06 smithi110 ceph-mon[97633]: pgmap v8699: 129 pgs: 129 active+clean; 8.6 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:01:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:06.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:06.174468+0000) 2022-09-10T09:01:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:06.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:06.690387+0000) 2022-09-10T09:01:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:07.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:07.174654+0000) 2022-09-10T09:01:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:07 smithi110 ceph-mon[97633]: pgmap v8700: 129 pgs: 129 active+clean; 8.6 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:01:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:07 smithi104 ceph-mon[97044]: pgmap v8700: 129 pgs: 129 active+clean; 8.6 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:01:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:07.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:07.690564+0000) 2022-09-10T09:01:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:08.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:08.174845+0000) 2022-09-10T09:01:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:08.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:08.690826+0000) 2022-09-10T09:01:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:09.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:09.175075+0000) 2022-09-10T09:01:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:09 smithi110 ceph-mon[97633]: osdmap e624: 12 total, 12 up, 12 in 2022-09-10T09:01:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:09 smithi110 ceph-mon[97633]: pgmap v8702: 129 pgs: 129 active+clean; 8.6 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:01:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:09 smithi104 ceph-mon[97044]: osdmap e624: 12 total, 12 up, 12 in 2022-09-10T09:01:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:09 smithi104 ceph-mon[97044]: pgmap v8702: 129 pgs: 129 active+clean; 8.6 MiB data, 843 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:01:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:09.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:09.691049+0000) 2022-09-10T09:01:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:10.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:10.175297+0000) 2022-09-10T09:01:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:10.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:10.691278+0000) 2022-09-10T09:01:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:11.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:11.175514+0000) 2022-09-10T09:01:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:11.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:11.691509+0000) 2022-09-10T09:01:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:12 smithi104 ceph-mon[97044]: pgmap v8703: 129 pgs: 129 active+clean; 8.6 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:01:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:12 smithi104 ceph-mon[97044]: osdmap e625: 12 total, 12 up, 12 in 2022-09-10T09:01:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:12 smithi110 ceph-mon[97633]: pgmap v8703: 129 pgs: 129 active+clean; 8.6 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:01:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:12 smithi110 ceph-mon[97633]: osdmap e625: 12 total, 12 up, 12 in 2022-09-10T09:01:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:12.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:12.175782+0000) 2022-09-10T09:01:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:12.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:12.691744+0000) 2022-09-10T09:01:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:13.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:13.176030+0000) 2022-09-10T09:01:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:13.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:13.691959+0000) 2022-09-10T09:01:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:14 smithi104 ceph-mon[97044]: pgmap v8705: 129 pgs: 129 active+clean; 8.6 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:01:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:14 smithi110 ceph-mon[97633]: pgmap v8705: 129 pgs: 129 active+clean; 8.6 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:01:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:14.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:14.176320+0000) 2022-09-10T09:01:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:14.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:14.692192+0000) 2022-09-10T09:01:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:15.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:15.176524+0000) 2022-09-10T09:01:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:15.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:15.692378+0000) 2022-09-10T09:01:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:16 smithi104 ceph-mon[97044]: pgmap v8706: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:01:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:16 smithi110 ceph-mon[97633]: pgmap v8706: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:01:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:16.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:16.176705+0000) 2022-09-10T09:01:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:16.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:16.692624+0000) 2022-09-10T09:01:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:17.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:17.176896+0000) 2022-09-10T09:01:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:17 smithi110 ceph-mon[97633]: pgmap v8707: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:01:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:17 smithi104 ceph-mon[97044]: pgmap v8707: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:01:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:17.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:17.692874+0000) 2022-09-10T09:01:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:18.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:18.177069+0000) 2022-09-10T09:01:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:18.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:18.693141+0000) 2022-09-10T09:01:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:19.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:19.177240+0000) 2022-09-10T09:01:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:19.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:19.693368+0000) 2022-09-10T09:01:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:20 smithi104 ceph-mon[97044]: pgmap v8708: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:01:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:20 smithi110 ceph-mon[97633]: pgmap v8708: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:01:20.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:20.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:20.177483+0000) 2022-09-10T09:01:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:20.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:20.693561+0000) 2022-09-10T09:01:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:01:20.520+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:01:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:21.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:21.177734+0000) 2022-09-10T09:01:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:21.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:21.693760+0000) 2022-09-10T09:01:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:22 smithi104 ceph-mon[97044]: pgmap v8709: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:01:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:22 smithi110 ceph-mon[97633]: pgmap v8709: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:01:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:22.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:22.178006+0000) 2022-09-10T09:01:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:22.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:22.693975+0000) 2022-09-10T09:01:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:23.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:23.178265+0000) 2022-09-10T09:01:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:23.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:23.694174+0000) 2022-09-10T09:01:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:24 smithi104 ceph-mon[97044]: pgmap v8710: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:01:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:24 smithi110 ceph-mon[97633]: pgmap v8710: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:01:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:24.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:24.178529+0000) 2022-09-10T09:01:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:24.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:24.694316+0000) 2022-09-10T09:01:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:25.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:25.178803+0000) 2022-09-10T09:01:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:25.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:25.694547+0000) 2022-09-10T09:01:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:26 smithi110 ceph-mon[97633]: pgmap v8711: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:01:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:26 smithi104 ceph-mon[97044]: pgmap v8711: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:01:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:26.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:26.179029+0000) 2022-09-10T09:01:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:26.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:26.694837+0000) 2022-09-10T09:01:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:27.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:27.179254+0000) 2022-09-10T09:01:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:27 smithi110 ceph-mon[97633]: pgmap v8712: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:27 smithi104 ceph-mon[97044]: pgmap v8712: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:27.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:27.695015+0000) 2022-09-10T09:01:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:28.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:28.179498+0000) 2022-09-10T09:01:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:28.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:28.695198+0000) 2022-09-10T09:01:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:29.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:29.179770+0000) 2022-09-10T09:01:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:29.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:29.695441+0000) 2022-09-10T09:01:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:30 smithi104 ceph-mon[97044]: pgmap v8713: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:30 smithi110 ceph-mon[97633]: pgmap v8713: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:30.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:30.179962+0000) 2022-09-10T09:01:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:30.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:30.695644+0000) 2022-09-10T09:01:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:31.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:31.180215+0000) 2022-09-10T09:01:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:31.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:31.695838+0000) 2022-09-10T09:01:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:32 smithi104 ceph-mon[97044]: pgmap v8714: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:32 smithi110 ceph-mon[97633]: pgmap v8714: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:32.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:32.180416+0000) 2022-09-10T09:01:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:32.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:32.696085+0000) 2022-09-10T09:01:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:33.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:33.180644+0000) 2022-09-10T09:01:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:33.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:33.696274+0000) 2022-09-10T09:01:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:34 smithi110 ceph-mon[97633]: pgmap v8715: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:34 smithi104 ceph-mon[97044]: pgmap v8715: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:34.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:34.180914+0000) 2022-09-10T09:01:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:34.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:34.696458+0000) 2022-09-10T09:01:35.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:35.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:35.181200+0000) 2022-09-10T09:01:35.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:01:35.531+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:01:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:35.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:35.696684+0000) 2022-09-10T09:01:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:36 smithi110 ceph-mon[97633]: pgmap v8716: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:36 smithi104 ceph-mon[97044]: pgmap v8716: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:36.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:36.181410+0000) 2022-09-10T09:01:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:36.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:36.696980+0000) 2022-09-10T09:01:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:37.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:37.181607+0000) 2022-09-10T09:01:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:37 smithi110 ceph-mon[97633]: pgmap v8717: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:37 smithi104 ceph-mon[97044]: pgmap v8717: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:37.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:37.697188+0000) 2022-09-10T09:01:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:38.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:38.181800+0000) 2022-09-10T09:01:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:38.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:38.697367+0000) 2022-09-10T09:01:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:39.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:39.182073+0000) 2022-09-10T09:01:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:39.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:39.697587+0000) 2022-09-10T09:01:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:40 smithi104 ceph-mon[97044]: pgmap v8718: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:40 smithi110 ceph-mon[97633]: pgmap v8718: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:40.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:40.182260+0000) 2022-09-10T09:01:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:40.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:40.697799+0000) 2022-09-10T09:01:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:41.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:41.182430+0000) 2022-09-10T09:01:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:41.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:41.697994+0000) 2022-09-10T09:01:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:42 smithi110 ceph-mon[97633]: pgmap v8719: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:42 smithi104 ceph-mon[97044]: pgmap v8719: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:42.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:42.182617+0000) 2022-09-10T09:01:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:42.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:42.698170+0000) 2022-09-10T09:01:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:43.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:43.182800+0000) 2022-09-10T09:01:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:43.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:43.698366+0000) 2022-09-10T09:01:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:44 smithi110 ceph-mon[97633]: pgmap v8720: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:44 smithi104 ceph-mon[97044]: pgmap v8720: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:44.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:44.183022+0000) 2022-09-10T09:01:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:44.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:44.698565+0000) 2022-09-10T09:01:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:01:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:01:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:01:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:01:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:01:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:01:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:01:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:01:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:45.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:45.183238+0000) 2022-09-10T09:01:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:45.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:45.698753+0000) 2022-09-10T09:01:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:46 smithi110 ceph-mon[97633]: pgmap v8721: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:46 smithi104 ceph-mon[97044]: pgmap v8721: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:46.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:46.183413+0000) 2022-09-10T09:01:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:46.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:46.698988+0000) 2022-09-10T09:01:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:47.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:47.183644+0000) 2022-09-10T09:01:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:47 smithi110 ceph-mon[97633]: pgmap v8722: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:47 smithi104 ceph-mon[97044]: pgmap v8722: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:47.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:47.699275+0000) 2022-09-10T09:01:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:48.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:48.183878+0000) 2022-09-10T09:01:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:48.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:48.699482+0000) 2022-09-10T09:01:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:49.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:49.184165+0000) 2022-09-10T09:01:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:49.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:49.699716+0000) 2022-09-10T09:01:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:50 smithi110 ceph-mon[97633]: pgmap v8723: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:50.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:50.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:50.184393+0000) 2022-09-10T09:01:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:50 smithi104 ceph-mon[97044]: pgmap v8723: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:50.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:50.699968+0000) 2022-09-10T09:01:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:01:50.541+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:01:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:51.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:51.184603+0000) 2022-09-10T09:01:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:51.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:51.700201+0000) 2022-09-10T09:01:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:52 smithi110 ceph-mon[97633]: pgmap v8724: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:52 smithi104 ceph-mon[97044]: pgmap v8724: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:52.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:52.184882+0000) 2022-09-10T09:01:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:52.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:52.700377+0000) 2022-09-10T09:01:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:53.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:53.185138+0000) 2022-09-10T09:01:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:53.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:53.700593+0000) 2022-09-10T09:01:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:54 smithi110 ceph-mon[97633]: pgmap v8725: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:54 smithi104 ceph-mon[97044]: pgmap v8725: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:54.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:54.185322+0000) 2022-09-10T09:01:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:54.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:54.700847+0000) 2022-09-10T09:01:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:55.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:55.185509+0000) 2022-09-10T09:01:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:55.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:55.701026+0000) 2022-09-10T09:01:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:56 smithi110 ceph-mon[97633]: pgmap v8726: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:56 smithi104 ceph-mon[97044]: pgmap v8726: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:01:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:56.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:56.185725+0000) 2022-09-10T09:01:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:56.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:56.701319+0000) 2022-09-10T09:01:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:57.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:57.185964+0000) 2022-09-10T09:01:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:01:57 smithi110 ceph-mon[97633]: pgmap v8727: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:01:57 smithi104 ceph-mon[97044]: pgmap v8727: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:01:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:57.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:57.701539+0000) 2022-09-10T09:01:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:58.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:58.186114+0000) 2022-09-10T09:01:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:58.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:58.701766+0000) 2022-09-10T09:01:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:01:59.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:59.186313+0000) 2022-09-10T09:02:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:01:59.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:01:59.701948+0000) 2022-09-10T09:02:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:00 smithi110 ceph-mon[97633]: pgmap v8728: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:00 smithi104 ceph-mon[97044]: pgmap v8728: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:00.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:00.186557+0000) 2022-09-10T09:02:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:00.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:00.702187+0000) 2022-09-10T09:02:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:01.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:01.186712+0000) 2022-09-10T09:02:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:01.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:01.702428+0000) 2022-09-10T09:02:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:02 smithi110 ceph-mon[97633]: pgmap v8729: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:02 smithi104 ceph-mon[97044]: pgmap v8729: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:02.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:02.186917+0000) 2022-09-10T09:02:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:02.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:02.702657+0000) 2022-09-10T09:02:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:03.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:03.187204+0000) 2022-09-10T09:02:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:03.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:03.702854+0000) 2022-09-10T09:02:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:04 smithi110 ceph-mon[97633]: pgmap v8730: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:04 smithi104 ceph-mon[97044]: pgmap v8730: 129 pgs: 129 active+clean; 8.6 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:04.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:04.187449+0000) 2022-09-10T09:02:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:04.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:04.703127+0000) 2022-09-10T09:02:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:05.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:05.187681+0000) 2022-09-10T09:02:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:05.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:05.703337+0000) 2022-09-10T09:02:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:02:05.556+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:02:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:06 smithi110 ceph-mon[97633]: pgmap v8731: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:02:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:06 smithi104 ceph-mon[97044]: pgmap v8731: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:02:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:06.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:06.187902+0000) 2022-09-10T09:02:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:06.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:06.703553+0000) 2022-09-10T09:02:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:07.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:07.188177+0000) 2022-09-10T09:02:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:07 smithi110 ceph-mon[97633]: pgmap v8732: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:02:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:07 smithi104 ceph-mon[97044]: pgmap v8732: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:02:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:07.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:07.703783+0000) 2022-09-10T09:02:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:08.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:08.188356+0000) 2022-09-10T09:02:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:08.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:08.703973+0000) 2022-09-10T09:02:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:09.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:09.188615+0000) 2022-09-10T09:02:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:09 smithi110 ceph-mon[97633]: osdmap e626: 12 total, 12 up, 12 in 2022-09-10T09:02:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:09 smithi110 ceph-mon[97633]: pgmap v8734: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:02:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:09 smithi104 ceph-mon[97044]: osdmap e626: 12 total, 12 up, 12 in 2022-09-10T09:02:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:09 smithi104 ceph-mon[97044]: pgmap v8734: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:02:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:09.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:09.704178+0000) 2022-09-10T09:02:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:10.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:10.188807+0000) 2022-09-10T09:02:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:10.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:10.704313+0000) 2022-09-10T09:02:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:11.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:11.188961+0000) 2022-09-10T09:02:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:11.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:11.704547+0000) 2022-09-10T09:02:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:12 smithi110 ceph-mon[97633]: pgmap v8735: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:02:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:12 smithi110 ceph-mon[97633]: osdmap e627: 12 total, 12 up, 12 in 2022-09-10T09:02:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:12 smithi104 ceph-mon[97044]: pgmap v8735: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:02:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:12 smithi104 ceph-mon[97044]: osdmap e627: 12 total, 12 up, 12 in 2022-09-10T09:02:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:12.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:12.189229+0000) 2022-09-10T09:02:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:12.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:12.704712+0000) 2022-09-10T09:02:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:13.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:13.189500+0000) 2022-09-10T09:02:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:13.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:13.704874+0000) 2022-09-10T09:02:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:14 smithi110 ceph-mon[97633]: pgmap v8737: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:02:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:14 smithi104 ceph-mon[97044]: pgmap v8737: 129 pgs: 129 active+clean; 8.6 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:02:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:14.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:14.189741+0000) 2022-09-10T09:02:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:14.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:14.705052+0000) 2022-09-10T09:02:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:15.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:15.190002+0000) 2022-09-10T09:02:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:15.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:15.705183+0000) 2022-09-10T09:02:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:16 smithi110 ceph-mon[97633]: pgmap v8738: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:02:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:16 smithi104 ceph-mon[97044]: pgmap v8738: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:02:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:16.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:16.190252+0000) 2022-09-10T09:02:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:16.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:16.705358+0000) 2022-09-10T09:02:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:17.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:17.190479+0000) 2022-09-10T09:02:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:17 smithi110 ceph-mon[97633]: pgmap v8739: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:02:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:17 smithi104 ceph-mon[97044]: pgmap v8739: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:02:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:17.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:17.705536+0000) 2022-09-10T09:02:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:18.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:18.190744+0000) 2022-09-10T09:02:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:18.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:18.705713+0000) 2022-09-10T09:02:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:19.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:19.191030+0000) 2022-09-10T09:02:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:19.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:19.705907+0000) 2022-09-10T09:02:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:20 smithi110 ceph-mon[97633]: pgmap v8740: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:02:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:20 smithi104 ceph-mon[97044]: pgmap v8740: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:02:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:20.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:20.191238+0000) 2022-09-10T09:02:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:20.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:20.706158+0000) 2022-09-10T09:02:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:02:20.569+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:02:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:21.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:21.191383+0000) 2022-09-10T09:02:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:21.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:21.706334+0000) 2022-09-10T09:02:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:22 smithi110 ceph-mon[97633]: pgmap v8741: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:02:22.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:22 smithi104 ceph-mon[97044]: pgmap v8741: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:02:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:22.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:22.191651+0000) 2022-09-10T09:02:23.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:22.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:22.706587+0000) 2022-09-10T09:02:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:23.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:23.191896+0000) 2022-09-10T09:02:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:23.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:23.706825+0000) 2022-09-10T09:02:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:24 smithi110 ceph-mon[97633]: pgmap v8742: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:02:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:24 smithi104 ceph-mon[97044]: pgmap v8742: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:02:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:24.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:24.192134+0000) 2022-09-10T09:02:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:24.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:24.707028+0000) 2022-09-10T09:02:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:25.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:25.192379+0000) 2022-09-10T09:02:26.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:25.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:25.707204+0000) 2022-09-10T09:02:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:26 smithi110 ceph-mon[97633]: pgmap v8743: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:02:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:26 smithi104 ceph-mon[97044]: pgmap v8743: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:02:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:26.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:26.192595+0000) 2022-09-10T09:02:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:26.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:26.707378+0000) 2022-09-10T09:02:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:27.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:27.192856+0000) 2022-09-10T09:02:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:27 smithi110 ceph-mon[97633]: pgmap v8744: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:27 smithi104 ceph-mon[97044]: pgmap v8744: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:27.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:27.707531+0000) 2022-09-10T09:02:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:28.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:28.192999+0000) 2022-09-10T09:02:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:28.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:28.707696+0000) 2022-09-10T09:02:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:29.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:29.193127+0000) 2022-09-10T09:02:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:29.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:29.707965+0000) 2022-09-10T09:02:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:30 smithi110 ceph-mon[97633]: pgmap v8745: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:30 smithi104 ceph-mon[97044]: pgmap v8745: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:30.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:30.193263+0000) 2022-09-10T09:02:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:30.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:30.708208+0000) 2022-09-10T09:02:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:31.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:31.193501+0000) 2022-09-10T09:02:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:31.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:31.708415+0000) 2022-09-10T09:02:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:32 smithi110 ceph-mon[97633]: pgmap v8746: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:32 smithi104 ceph-mon[97044]: pgmap v8746: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:32.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:32.193669+0000) 2022-09-10T09:02:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:32.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:32.708618+0000) 2022-09-10T09:02:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:33.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:33.193894+0000) 2022-09-10T09:02:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:33.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:33.708776+0000) 2022-09-10T09:02:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:34 smithi110 ceph-mon[97633]: pgmap v8747: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:34 smithi104 ceph-mon[97044]: pgmap v8747: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:34.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:34.194139+0000) 2022-09-10T09:02:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:34.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:34.708964+0000) 2022-09-10T09:02:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:35.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:35.194263+0000) 2022-09-10T09:02:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:35.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:35.709130+0000) 2022-09-10T09:02:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:02:35.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:02:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:36 smithi110 ceph-mon[97633]: pgmap v8748: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:36 smithi104 ceph-mon[97044]: pgmap v8748: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:36.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:36.194484+0000) 2022-09-10T09:02:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:36.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:36.709342+0000) 2022-09-10T09:02:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:37.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:37.194719+0000) 2022-09-10T09:02:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:37 smithi110 ceph-mon[97633]: pgmap v8749: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:37 smithi104 ceph-mon[97044]: pgmap v8749: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:37.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:37.709507+0000) 2022-09-10T09:02:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:38.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:38.194894+0000) 2022-09-10T09:02:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:38.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:38.709763+0000) 2022-09-10T09:02:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:39.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:39.195140+0000) 2022-09-10T09:02:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:39.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:39.709962+0000) 2022-09-10T09:02:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:40 smithi110 ceph-mon[97633]: pgmap v8750: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:40 smithi104 ceph-mon[97044]: pgmap v8750: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:40.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:40.195305+0000) 2022-09-10T09:02:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:40.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:40.710179+0000) 2022-09-10T09:02:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:41.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:41.195509+0000) 2022-09-10T09:02:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:41.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:41.710415+0000) 2022-09-10T09:02:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:42 smithi110 ceph-mon[97633]: pgmap v8751: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:42 smithi104 ceph-mon[97044]: pgmap v8751: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:42.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:42.195696+0000) 2022-09-10T09:02:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:42.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:42.710606+0000) 2022-09-10T09:02:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:43.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:43.195861+0000) 2022-09-10T09:02:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:43.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:43.710870+0000) 2022-09-10T09:02:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:44 smithi110 ceph-mon[97633]: pgmap v8752: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:44 smithi104 ceph-mon[97044]: pgmap v8752: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:44.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:44.196129+0000) 2022-09-10T09:02:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:44.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:44.711022+0000) 2022-09-10T09:02:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:02:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:02:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:02:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:02:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:02:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:02:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:02:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:02:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:45.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:45.196330+0000) 2022-09-10T09:02:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:45.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:45.711252+0000) 2022-09-10T09:02:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:46 smithi110 ceph-mon[97633]: pgmap v8753: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:46 smithi104 ceph-mon[97044]: pgmap v8753: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:46.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:46.196510+0000) 2022-09-10T09:02:47.029 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:46.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:46.711494+0000) 2022-09-10T09:02:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:47.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:47.196717+0000) 2022-09-10T09:02:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:47 smithi110 ceph-mon[97633]: pgmap v8754: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:47 smithi104 ceph-mon[97044]: pgmap v8754: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:47.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:47.711733+0000) 2022-09-10T09:02:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:48.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:48.196886+0000) 2022-09-10T09:02:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:48.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:48.711920+0000) 2022-09-10T09:02:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:49.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:49.197108+0000) 2022-09-10T09:02:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:49.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:49.712160+0000) 2022-09-10T09:02:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:50 smithi110 ceph-mon[97633]: pgmap v8755: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:50 smithi104 ceph-mon[97044]: pgmap v8755: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:50.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:50.197345+0000) 2022-09-10T09:02:51.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:02:50.595+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:02:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:50.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:50.712391+0000) 2022-09-10T09:02:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:51.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:51.197539+0000) 2022-09-10T09:02:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:51.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:51.712548+0000) 2022-09-10T09:02:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:52 smithi110 ceph-mon[97633]: pgmap v8756: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:52.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:52 smithi104 ceph-mon[97044]: pgmap v8756: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:52.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:52.197792+0000) 2022-09-10T09:02:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:52.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:52.712739+0000) 2022-09-10T09:02:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:53.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:53.198065+0000) 2022-09-10T09:02:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:53.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:53.712970+0000) 2022-09-10T09:02:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:54 smithi110 ceph-mon[97633]: pgmap v8757: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:54 smithi104 ceph-mon[97044]: pgmap v8757: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:54.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:54.198386+0000) 2022-09-10T09:02:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:54.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:54.713199+0000) 2022-09-10T09:02:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:55.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:55.198657+0000) 2022-09-10T09:02:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:55.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:55.713464+0000) 2022-09-10T09:02:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:56 smithi110 ceph-mon[97633]: pgmap v8758: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:56 smithi104 ceph-mon[97044]: pgmap v8758: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:02:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:56.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:56.198951+0000) 2022-09-10T09:02:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:56.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:56.713700+0000) 2022-09-10T09:02:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:57.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:57.199156+0000) 2022-09-10T09:02:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:02:57 smithi110 ceph-mon[97633]: pgmap v8759: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:02:57 smithi104 ceph-mon[97044]: pgmap v8759: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:02:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:57.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:57.713938+0000) 2022-09-10T09:02:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:58.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:58.199325+0000) 2022-09-10T09:02:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:58.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:58.714212+0000) 2022-09-10T09:02:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:02:59.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:59.199595+0000) 2022-09-10T09:03:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:02:59.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:02:59.714477+0000) 2022-09-10T09:03:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:00 smithi110 ceph-mon[97633]: pgmap v8760: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:00 smithi104 ceph-mon[97044]: pgmap v8760: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:00.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:00.199787+0000) 2022-09-10T09:03:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:00.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:00.714718+0000) 2022-09-10T09:03:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:01.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:01.200022+0000) 2022-09-10T09:03:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:01.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:01.714952+0000) 2022-09-10T09:03:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:02 smithi110 ceph-mon[97633]: pgmap v8761: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:02 smithi104 ceph-mon[97044]: pgmap v8761: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:02.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:02.200321+0000) 2022-09-10T09:03:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:02.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:02.715174+0000) 2022-09-10T09:03:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:03.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:03.200599+0000) 2022-09-10T09:03:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:03.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:03.715409+0000) 2022-09-10T09:03:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:04 smithi110 ceph-mon[97633]: pgmap v8762: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:04 smithi104 ceph-mon[97044]: pgmap v8762: 129 pgs: 129 active+clean; 8.6 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:04.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:04.200796+0000) 2022-09-10T09:03:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:04.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:04.715673+0000) 2022-09-10T09:03:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:05.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:05.201025+0000) 2022-09-10T09:03:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:05.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:05.715870+0000) 2022-09-10T09:03:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:03:05.602+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:03:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:06 smithi110 ceph-mon[97633]: pgmap v8763: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:03:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:06 smithi104 ceph-mon[97044]: pgmap v8763: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:03:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:06.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:06.201216+0000) 2022-09-10T09:03:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:06.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:06.716106+0000) 2022-09-10T09:03:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:07.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:07.201453+0000) 2022-09-10T09:03:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:07 smithi110 ceph-mon[97633]: pgmap v8764: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:03:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:07 smithi104 ceph-mon[97044]: pgmap v8764: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:03:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:07.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:07.716341+0000) 2022-09-10T09:03:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:08.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:08.201719+0000) 2022-09-10T09:03:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:08.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:08.716526+0000) 2022-09-10T09:03:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:09.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:09.201980+0000) 2022-09-10T09:03:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:09 smithi110 ceph-mon[97633]: osdmap e628: 12 total, 12 up, 12 in 2022-09-10T09:03:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:09 smithi110 ceph-mon[97633]: pgmap v8766: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:03:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:09 smithi104 ceph-mon[97044]: osdmap e628: 12 total, 12 up, 12 in 2022-09-10T09:03:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:09 smithi104 ceph-mon[97044]: pgmap v8766: 129 pgs: 129 active+clean; 8.6 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:03:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:09.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:09.716774+0000) 2022-09-10T09:03:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:10.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:10.202299+0000) 2022-09-10T09:03:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:10.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:10.717013+0000) 2022-09-10T09:03:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:11.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:11.202514+0000) 2022-09-10T09:03:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:11.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:11.717217+0000) 2022-09-10T09:03:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:12 smithi110 ceph-mon[97633]: pgmap v8767: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:03:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:12 smithi110 ceph-mon[97633]: osdmap e629: 12 total, 12 up, 12 in 2022-09-10T09:03:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:12 smithi104 ceph-mon[97044]: pgmap v8767: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:03:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:12 smithi104 ceph-mon[97044]: osdmap e629: 12 total, 12 up, 12 in 2022-09-10T09:03:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:12.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:12.202814+0000) 2022-09-10T09:03:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:12.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:12.717438+0000) 2022-09-10T09:03:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:13.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:13.203069+0000) 2022-09-10T09:03:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:13.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:13.717640+0000) 2022-09-10T09:03:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:14 smithi110 ceph-mon[97633]: pgmap v8769: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:03:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:14 smithi104 ceph-mon[97044]: pgmap v8769: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:03:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:14.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:14.203302+0000) 2022-09-10T09:03:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:14.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:14.717845+0000) 2022-09-10T09:03:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:15.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:15.203461+0000) 2022-09-10T09:03:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:15.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:15.718003+0000) 2022-09-10T09:03:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:16 smithi110 ceph-mon[97633]: pgmap v8770: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:03:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:16 smithi104 ceph-mon[97044]: pgmap v8770: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:03:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:16.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:16.203643+0000) 2022-09-10T09:03:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:16.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:16.718168+0000) 2022-09-10T09:03:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:17.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:17.203855+0000) 2022-09-10T09:03:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:17 smithi110 ceph-mon[97633]: pgmap v8771: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:03:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:17 smithi104 ceph-mon[97044]: pgmap v8771: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:03:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:17.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:17.718377+0000) 2022-09-10T09:03:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:18.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:18.204081+0000) 2022-09-10T09:03:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:18.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:18.718566+0000) 2022-09-10T09:03:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:19.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:19.204375+0000) 2022-09-10T09:03:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:19.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:19.718768+0000) 2022-09-10T09:03:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:20 smithi110 ceph-mon[97633]: pgmap v8772: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:03:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:20 smithi104 ceph-mon[97044]: pgmap v8772: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:03:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:20.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:20.204636+0000) 2022-09-10T09:03:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:20.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:20.718999+0000) 2022-09-10T09:03:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:03:20.608+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:03:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:21.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:21.204894+0000) 2022-09-10T09:03:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:21.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:21.719225+0000) 2022-09-10T09:03:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:22 smithi110 ceph-mon[97633]: pgmap v8773: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:03:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:22 smithi104 ceph-mon[97044]: pgmap v8773: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:03:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:22.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:22.205227+0000) 2022-09-10T09:03:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:22.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:22.719417+0000) 2022-09-10T09:03:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:23.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:23.205451+0000) 2022-09-10T09:03:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:23.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:23.719587+0000) 2022-09-10T09:03:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:24 smithi110 ceph-mon[97633]: pgmap v8774: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:03:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:24 smithi104 ceph-mon[97044]: pgmap v8774: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:03:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:24.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:24.205685+0000) 2022-09-10T09:03:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:24.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:24.719754+0000) 2022-09-10T09:03:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:25.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:25.205918+0000) 2022-09-10T09:03:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:25.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:25.719940+0000) 2022-09-10T09:03:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:26 smithi110 ceph-mon[97633]: pgmap v8775: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:03:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:26 smithi104 ceph-mon[97044]: pgmap v8775: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:03:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:26.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:26.206201+0000) 2022-09-10T09:03:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:26.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:26.720200+0000) 2022-09-10T09:03:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:27.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:27.206419+0000) 2022-09-10T09:03:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:27 smithi110 ceph-mon[97633]: pgmap v8776: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:03:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:27 smithi104 ceph-mon[97044]: pgmap v8776: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:03:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:27.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:27.720403+0000) 2022-09-10T09:03:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:28.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:28.206673+0000) 2022-09-10T09:03:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:28.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:28.720629+0000) 2022-09-10T09:03:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:29.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:29.206931+0000) 2022-09-10T09:03:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:29.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:29.720821+0000) 2022-09-10T09:03:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:30 smithi110 ceph-mon[97633]: pgmap v8777: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:03:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:30 smithi104 ceph-mon[97044]: pgmap v8777: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:03:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:30.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:30.207144+0000) 2022-09-10T09:03:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:30.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:30.720989+0000) 2022-09-10T09:03:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:31.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:31.207308+0000) 2022-09-10T09:03:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:31.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:31.721167+0000) 2022-09-10T09:03:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:32 smithi110 ceph-mon[97633]: pgmap v8778: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:03:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:32 smithi104 ceph-mon[97044]: pgmap v8778: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:03:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:32.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:32.207545+0000) 2022-09-10T09:03:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:32.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:32.721406+0000) 2022-09-10T09:03:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:33.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:33.207818+0000) 2022-09-10T09:03:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:33.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:33.721652+0000) 2022-09-10T09:03:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:34 smithi110 ceph-mon[97633]: pgmap v8779: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:03:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:34 smithi104 ceph-mon[97044]: pgmap v8779: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:03:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:34.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:34.208133+0000) 2022-09-10T09:03:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:34.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:34.721883+0000) 2022-09-10T09:03:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:35.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:35.208329+0000) 2022-09-10T09:03:36.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:03:35.615+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:03:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:35.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:35.722166+0000) 2022-09-10T09:03:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:36 smithi110 ceph-mon[97633]: pgmap v8780: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:03:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:36 smithi104 ceph-mon[97044]: pgmap v8780: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:03:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:36.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:36.208586+0000) 2022-09-10T09:03:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:36.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:36.722378+0000) 2022-09-10T09:03:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:37.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:37.208779+0000) 2022-09-10T09:03:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:37 smithi110 ceph-mon[97633]: pgmap v8781: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:37 smithi104 ceph-mon[97044]: pgmap v8781: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:37.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:37.722537+0000) 2022-09-10T09:03:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:38.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:38.209042+0000) 2022-09-10T09:03:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:38.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:38.722806+0000) 2022-09-10T09:03:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:39.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:39.209306+0000) 2022-09-10T09:03:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:39.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:39.723048+0000) 2022-09-10T09:03:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:40 smithi110 ceph-mon[97633]: pgmap v8782: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:40 smithi104 ceph-mon[97044]: pgmap v8782: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:40.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:40.209470+0000) 2022-09-10T09:03:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:40.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:40.723232+0000) 2022-09-10T09:03:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:41.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:41.209684+0000) 2022-09-10T09:03:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:41.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:41.723483+0000) 2022-09-10T09:03:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:42 smithi110 ceph-mon[97633]: pgmap v8783: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:42 smithi104 ceph-mon[97044]: pgmap v8783: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:42.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:42.209879+0000) 2022-09-10T09:03:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:42.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:42.723645+0000) 2022-09-10T09:03:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:43.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:43.210115+0000) 2022-09-10T09:03:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:43.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:43.723775+0000) 2022-09-10T09:03:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:44 smithi110 ceph-mon[97633]: pgmap v8784: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:44 smithi104 ceph-mon[97044]: pgmap v8784: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:44.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:44.210338+0000) 2022-09-10T09:03:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:44.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:44.723923+0000) 2022-09-10T09:03:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:03:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:03:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:03:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:03:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:03:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:03:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:03:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:03:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:45.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:45.210511+0000) 2022-09-10T09:03:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:45.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:45.724217+0000) 2022-09-10T09:03:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:46 smithi110 ceph-mon[97633]: pgmap v8785: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:46 smithi104 ceph-mon[97044]: pgmap v8785: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:46.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:46.210721+0000) 2022-09-10T09:03:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:46.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:46.724507+0000) 2022-09-10T09:03:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:47.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:47.210902+0000) 2022-09-10T09:03:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:47 smithi110 ceph-mon[97633]: pgmap v8786: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:47 smithi104 ceph-mon[97044]: pgmap v8786: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:47.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:47.724771+0000) 2022-09-10T09:03:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:48.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:48.211148+0000) 2022-09-10T09:03:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:48.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:48.724960+0000) 2022-09-10T09:03:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:49.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:49.211405+0000) 2022-09-10T09:03:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:49.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:49.725218+0000) 2022-09-10T09:03:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:50 smithi110 ceph-mon[97633]: pgmap v8787: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:50 smithi104 ceph-mon[97044]: pgmap v8787: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:50.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:50.211581+0000) 2022-09-10T09:03:51.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:03:50.628+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:03:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:50.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:50.725384+0000) 2022-09-10T09:03:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:51.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:51.211718+0000) 2022-09-10T09:03:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:51.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:51.725574+0000) 2022-09-10T09:03:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:52 smithi110 ceph-mon[97633]: pgmap v8788: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:52 smithi104 ceph-mon[97044]: pgmap v8788: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:52.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:52.211888+0000) 2022-09-10T09:03:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:52.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:52.725821+0000) 2022-09-10T09:03:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:53.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:53.212062+0000) 2022-09-10T09:03:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:53.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:53.726091+0000) 2022-09-10T09:03:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:54 smithi110 ceph-mon[97633]: pgmap v8789: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:54 smithi104 ceph-mon[97044]: pgmap v8789: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:54.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:54.212276+0000) 2022-09-10T09:03:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:54.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:54.726320+0000) 2022-09-10T09:03:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:55.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:55.212454+0000) 2022-09-10T09:03:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:55.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:55.726592+0000) 2022-09-10T09:03:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:56 smithi110 ceph-mon[97633]: pgmap v8790: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:56 smithi104 ceph-mon[97044]: pgmap v8790: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:03:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:56.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:56.212665+0000) 2022-09-10T09:03:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:56.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:56.726803+0000) 2022-09-10T09:03:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:57.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:57.212906+0000) 2022-09-10T09:03:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:03:57 smithi110 ceph-mon[97633]: pgmap v8791: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:03:57 smithi104 ceph-mon[97044]: pgmap v8791: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:03:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:57.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:57.727019+0000) 2022-09-10T09:03:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:58.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:58.213163+0000) 2022-09-10T09:03:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:58.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:58.727295+0000) 2022-09-10T09:03:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:03:59.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:59.213337+0000) 2022-09-10T09:04:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:03:59.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:03:59.727557+0000) 2022-09-10T09:04:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:00 smithi110 ceph-mon[97633]: pgmap v8792: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:00 smithi104 ceph-mon[97044]: pgmap v8792: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:00.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:00.213490+0000) 2022-09-10T09:04:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:00.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:00.727714+0000) 2022-09-10T09:04:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:01.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:01.213717+0000) 2022-09-10T09:04:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:01.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:01.727900+0000) 2022-09-10T09:04:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:02 smithi110 ceph-mon[97633]: pgmap v8793: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:02 smithi104 ceph-mon[97044]: pgmap v8793: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:02.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:02.213907+0000) 2022-09-10T09:04:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:02.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:02.728123+0000) 2022-09-10T09:04:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:03.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:03.214151+0000) 2022-09-10T09:04:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:03.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:03.728340+0000) 2022-09-10T09:04:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:04 smithi110 ceph-mon[97633]: pgmap v8794: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:04 smithi104 ceph-mon[97044]: pgmap v8794: 129 pgs: 129 active+clean; 8.6 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:04.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:04.214359+0000) 2022-09-10T09:04:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:04.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:04.728599+0000) 2022-09-10T09:04:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:05.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:05.214608+0000) 2022-09-10T09:04:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:05.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:05.728866+0000) 2022-09-10T09:04:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:04:05.641+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:04:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:06 smithi110 ceph-mon[97633]: pgmap v8795: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:04:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:06 smithi104 ceph-mon[97044]: pgmap v8795: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:04:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:06.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:06.214861+0000) 2022-09-10T09:04:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:06.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:06.729164+0000) 2022-09-10T09:04:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:07.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:07.215130+0000) 2022-09-10T09:04:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:07 smithi110 ceph-mon[97633]: pgmap v8796: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:04:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:07 smithi104 ceph-mon[97044]: pgmap v8796: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:04:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:07.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:07.729345+0000) 2022-09-10T09:04:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:08.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:08.215306+0000) 2022-09-10T09:04:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:08.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:08.729521+0000) 2022-09-10T09:04:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:09.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:09.215513+0000) 2022-09-10T09:04:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:09 smithi110 ceph-mon[97633]: osdmap e630: 12 total, 12 up, 12 in 2022-09-10T09:04:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:09 smithi110 ceph-mon[97633]: pgmap v8798: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:04:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:09 smithi104 ceph-mon[97044]: osdmap e630: 12 total, 12 up, 12 in 2022-09-10T09:04:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:09 smithi104 ceph-mon[97044]: pgmap v8798: 129 pgs: 129 active+clean; 8.6 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:04:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:09.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:09.729754+0000) 2022-09-10T09:04:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:10.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:10.215771+0000) 2022-09-10T09:04:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:10.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:10.729930+0000) 2022-09-10T09:04:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:11.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:11.215976+0000) 2022-09-10T09:04:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:11.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:11.730203+0000) 2022-09-10T09:04:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:12 smithi110 ceph-mon[97633]: pgmap v8799: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:04:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:12 smithi110 ceph-mon[97633]: osdmap e631: 12 total, 12 up, 12 in 2022-09-10T09:04:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:12 smithi104 ceph-mon[97044]: pgmap v8799: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:04:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:12 smithi104 ceph-mon[97044]: osdmap e631: 12 total, 12 up, 12 in 2022-09-10T09:04:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:12.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:12.216217+0000) 2022-09-10T09:04:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:12.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:12.730421+0000) 2022-09-10T09:04:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:13.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:13.216477+0000) 2022-09-10T09:04:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:13.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:13.730639+0000) 2022-09-10T09:04:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:14 smithi110 ceph-mon[97633]: pgmap v8801: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:04:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:14 smithi104 ceph-mon[97044]: pgmap v8801: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:04:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:14.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:14.216707+0000) 2022-09-10T09:04:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:14.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:14.730914+0000) 2022-09-10T09:04:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:15.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:15.216866+0000) 2022-09-10T09:04:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:15.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:15.731174+0000) 2022-09-10T09:04:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:16 smithi110 ceph-mon[97633]: pgmap v8802: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:04:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:16 smithi104 ceph-mon[97044]: pgmap v8802: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:04:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:16.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:16.217022+0000) 2022-09-10T09:04:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:16.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:16.731417+0000) 2022-09-10T09:04:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:17.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:17.217226+0000) 2022-09-10T09:04:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:17 smithi110 ceph-mon[97633]: pgmap v8803: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:04:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:17 smithi104 ceph-mon[97044]: pgmap v8803: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:04:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:17.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:17.731668+0000) 2022-09-10T09:04:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:18.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:18.217464+0000) 2022-09-10T09:04:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:18.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:18.731838+0000) 2022-09-10T09:04:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:19.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:19.217610+0000) 2022-09-10T09:04:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:19.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:19.732012+0000) 2022-09-10T09:04:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:20 smithi110 ceph-mon[97633]: pgmap v8804: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:04:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:20 smithi104 ceph-mon[97044]: pgmap v8804: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:04:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:20.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:20.217841+0000) 2022-09-10T09:04:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:04:20.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:04:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:20.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:20.732270+0000) 2022-09-10T09:04:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:21.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:21.218127+0000) 2022-09-10T09:04:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:21.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:21.732480+0000) 2022-09-10T09:04:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:22 smithi110 ceph-mon[97633]: pgmap v8805: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:04:22.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:22 smithi104 ceph-mon[97044]: pgmap v8805: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:04:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:22.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:22.218274+0000) 2022-09-10T09:04:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:22.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:22.732659+0000) 2022-09-10T09:04:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:23.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:23.218409+0000) 2022-09-10T09:04:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:23.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:23.732816+0000) 2022-09-10T09:04:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:24 smithi110 ceph-mon[97633]: pgmap v8806: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:04:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:24 smithi104 ceph-mon[97044]: pgmap v8806: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:04:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:24.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:24.218620+0000) 2022-09-10T09:04:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:24.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:24.733019+0000) 2022-09-10T09:04:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:25.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:25.218893+0000) 2022-09-10T09:04:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:25.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:25.733215+0000) 2022-09-10T09:04:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:26 smithi110 ceph-mon[97633]: pgmap v8807: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:04:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:26 smithi104 ceph-mon[97044]: pgmap v8807: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:04:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:26.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:26.219127+0000) 2022-09-10T09:04:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:26.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:26.733469+0000) 2022-09-10T09:04:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:27.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:27.219356+0000) 2022-09-10T09:04:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:27 smithi110 ceph-mon[97633]: pgmap v8808: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:27 smithi104 ceph-mon[97044]: pgmap v8808: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:27.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:27.733728+0000) 2022-09-10T09:04:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:28.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:28.219601+0000) 2022-09-10T09:04:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:28.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:28.734007+0000) 2022-09-10T09:04:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:29.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:29.219809+0000) 2022-09-10T09:04:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:29.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:29.734218+0000) 2022-09-10T09:04:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:30 smithi110 ceph-mon[97633]: pgmap v8809: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:30 smithi104 ceph-mon[97044]: pgmap v8809: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:30.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:30.220045+0000) 2022-09-10T09:04:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:30.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:30.734451+0000) 2022-09-10T09:04:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:31.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:31.220251+0000) 2022-09-10T09:04:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:31.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:31.734664+0000) 2022-09-10T09:04:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:32 smithi110 ceph-mon[97633]: pgmap v8810: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:32 smithi104 ceph-mon[97044]: pgmap v8810: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:32.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:32.220519+0000) 2022-09-10T09:04:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:32.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:32.734870+0000) 2022-09-10T09:04:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:33.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:33.220712+0000) 2022-09-10T09:04:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:33.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:33.735087+0000) 2022-09-10T09:04:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:34 smithi110 ceph-mon[97633]: pgmap v8811: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:34 smithi104 ceph-mon[97044]: pgmap v8811: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:34.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:34.220916+0000) 2022-09-10T09:04:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:34.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:34.735293+0000) 2022-09-10T09:04:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:35.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:35.221172+0000) 2022-09-10T09:04:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:04:35.646+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:04:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:35.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:35.735493+0000) 2022-09-10T09:04:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:36 smithi110 ceph-mon[97633]: pgmap v8812: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:36 smithi104 ceph-mon[97044]: pgmap v8812: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:36.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:36.221384+0000) 2022-09-10T09:04:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:36.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:36.735630+0000) 2022-09-10T09:04:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:37.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:37.221599+0000) 2022-09-10T09:04:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:37 smithi110 ceph-mon[97633]: pgmap v8813: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:37 smithi104 ceph-mon[97044]: pgmap v8813: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:37.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:37.735758+0000) 2022-09-10T09:04:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:38.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:38.221838+0000) 2022-09-10T09:04:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:38.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:38.735968+0000) 2022-09-10T09:04:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:39.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:39.222033+0000) 2022-09-10T09:04:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:39.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:39.736202+0000) 2022-09-10T09:04:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:40 smithi110 ceph-mon[97633]: pgmap v8814: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:40 smithi104 ceph-mon[97044]: pgmap v8814: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:40.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:40.222281+0000) 2022-09-10T09:04:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:40.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:40.736496+0000) 2022-09-10T09:04:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:41.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:41.222554+0000) 2022-09-10T09:04:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:41.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:41.736687+0000) 2022-09-10T09:04:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:42 smithi110 ceph-mon[97633]: pgmap v8815: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:42 smithi104 ceph-mon[97044]: pgmap v8815: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:42.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:42.222837+0000) 2022-09-10T09:04:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:42.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:42.736940+0000) 2022-09-10T09:04:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:43.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:43.223131+0000) 2022-09-10T09:04:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:43.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:43.737168+0000) 2022-09-10T09:04:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:44 smithi110 ceph-mon[97633]: pgmap v8816: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:44 smithi104 ceph-mon[97044]: pgmap v8816: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:44.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:44.223262+0000) 2022-09-10T09:04:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:44.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:44.737378+0000) 2022-09-10T09:04:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:04:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:04:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:04:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:04:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:04:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:04:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:04:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:04:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:45.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:45.223525+0000) 2022-09-10T09:04:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:45.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:45.737629+0000) 2022-09-10T09:04:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:46 smithi110 ceph-mon[97633]: pgmap v8817: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:46 smithi104 ceph-mon[97044]: pgmap v8817: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:46.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:46.223744+0000) 2022-09-10T09:04:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:46.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:46.737892+0000) 2022-09-10T09:04:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:47.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:47.223994+0000) 2022-09-10T09:04:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:47 smithi110 ceph-mon[97633]: pgmap v8818: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:47 smithi104 ceph-mon[97044]: pgmap v8818: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:47.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:47.738178+0000) 2022-09-10T09:04:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:48.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:48.224285+0000) 2022-09-10T09:04:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:48.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:48.738300+0000) 2022-09-10T09:04:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:49.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:49.224574+0000) 2022-09-10T09:04:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:49.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:49.738534+0000) 2022-09-10T09:04:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:50 smithi110 ceph-mon[97633]: pgmap v8819: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:50 smithi104 ceph-mon[97044]: pgmap v8819: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:50.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:50.224800+0000) 2022-09-10T09:04:51.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:04:50.662+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:04:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:50.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:50.738776+0000) 2022-09-10T09:04:51.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:51.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:51.225041+0000) 2022-09-10T09:04:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:51.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:51.739012+0000) 2022-09-10T09:04:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:52 smithi110 ceph-mon[97633]: pgmap v8820: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:52 smithi104 ceph-mon[97044]: pgmap v8820: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:52.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:52.225367+0000) 2022-09-10T09:04:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:52.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:52.739262+0000) 2022-09-10T09:04:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:53.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:53.225609+0000) 2022-09-10T09:04:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:53.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:53.739505+0000) 2022-09-10T09:04:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:54 smithi110 ceph-mon[97633]: pgmap v8821: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:54 smithi104 ceph-mon[97044]: pgmap v8821: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:54.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:54.225835+0000) 2022-09-10T09:04:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:54.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:54.739741+0000) 2022-09-10T09:04:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:55.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:55.226050+0000) 2022-09-10T09:04:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:55.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:55.739970+0000) 2022-09-10T09:04:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:56 smithi110 ceph-mon[97633]: pgmap v8822: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:56 smithi104 ceph-mon[97044]: pgmap v8822: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:04:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:56.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:56.226282+0000) 2022-09-10T09:04:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:56.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:56.740240+0000) 2022-09-10T09:04:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:57.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:57.226440+0000) 2022-09-10T09:04:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:04:57 smithi110 ceph-mon[97633]: pgmap v8823: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:04:57 smithi104 ceph-mon[97044]: pgmap v8823: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:04:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:57.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:57.740514+0000) 2022-09-10T09:04:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:58.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:58.226703+0000) 2022-09-10T09:04:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:58.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:58.740740+0000) 2022-09-10T09:04:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:04:59.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:59.226957+0000) 2022-09-10T09:05:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:04:59.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:04:59.741009+0000) 2022-09-10T09:05:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:00 smithi110 ceph-mon[97633]: pgmap v8824: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:00 smithi104 ceph-mon[97044]: pgmap v8824: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:00.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:00.227262+0000) 2022-09-10T09:05:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:00.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:00.741236+0000) 2022-09-10T09:05:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:01.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:01.227536+0000) 2022-09-10T09:05:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:01.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:01.741488+0000) 2022-09-10T09:05:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:02 smithi110 ceph-mon[97633]: pgmap v8825: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:02 smithi104 ceph-mon[97044]: pgmap v8825: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:02.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:02.227763+0000) 2022-09-10T09:05:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:02.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:02.741703+0000) 2022-09-10T09:05:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:03.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:03.228029+0000) 2022-09-10T09:05:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:03.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:03.741905+0000) 2022-09-10T09:05:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:04 smithi110 ceph-mon[97633]: pgmap v8826: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:04 smithi104 ceph-mon[97044]: pgmap v8826: 129 pgs: 129 active+clean; 8.6 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:04.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:04.228281+0000) 2022-09-10T09:05:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:04.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:04.742125+0000) 2022-09-10T09:05:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:05.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:05.228546+0000) 2022-09-10T09:05:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:05.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:05.742335+0000) 2022-09-10T09:05:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:05:05.675+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:05:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:06 smithi110 ceph-mon[97633]: pgmap v8827: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:05:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:06 smithi104 ceph-mon[97044]: pgmap v8827: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:05:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:06.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:06.228762+0000) 2022-09-10T09:05:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:06.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:06.742601+0000) 2022-09-10T09:05:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:07.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:07.228954+0000) 2022-09-10T09:05:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:07 smithi110 ceph-mon[97633]: pgmap v8828: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:05:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:07 smithi104 ceph-mon[97044]: pgmap v8828: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:05:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:07.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:07.742747+0000) 2022-09-10T09:05:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:08.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:08.229218+0000) 2022-09-10T09:05:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:08.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:08.742955+0000) 2022-09-10T09:05:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:09.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:09.229420+0000) 2022-09-10T09:05:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:09 smithi110 ceph-mon[97633]: osdmap e632: 12 total, 12 up, 12 in 2022-09-10T09:05:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:09 smithi110 ceph-mon[97633]: pgmap v8830: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:05:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:09 smithi104 ceph-mon[97044]: osdmap e632: 12 total, 12 up, 12 in 2022-09-10T09:05:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:09 smithi104 ceph-mon[97044]: pgmap v8830: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:05:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:09.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:09.743124+0000) 2022-09-10T09:05:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:10.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:10.229661+0000) 2022-09-10T09:05:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:10.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:10.743384+0000) 2022-09-10T09:05:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:11.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:11.229942+0000) 2022-09-10T09:05:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:11.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:11.743668+0000) 2022-09-10T09:05:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:12 smithi110 ceph-mon[97633]: pgmap v8831: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:05:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:12 smithi110 ceph-mon[97633]: osdmap e633: 12 total, 12 up, 12 in 2022-09-10T09:05:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:12 smithi104 ceph-mon[97044]: pgmap v8831: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:05:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:12 smithi104 ceph-mon[97044]: osdmap e633: 12 total, 12 up, 12 in 2022-09-10T09:05:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:12.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:12.230263+0000) 2022-09-10T09:05:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:12.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:12.743838+0000) 2022-09-10T09:05:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:13.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:13.230550+0000) 2022-09-10T09:05:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:13.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:13.744089+0000) 2022-09-10T09:05:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:14 smithi110 ceph-mon[97633]: pgmap v8833: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:05:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:14 smithi104 ceph-mon[97044]: pgmap v8833: 129 pgs: 129 active+clean; 8.6 MiB data, 856 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:05:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:14.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:14.230852+0000) 2022-09-10T09:05:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:14.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:14.744337+0000) 2022-09-10T09:05:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:15.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:15.231147+0000) 2022-09-10T09:05:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:15.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:15.744629+0000) 2022-09-10T09:05:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:16 smithi110 ceph-mon[97633]: pgmap v8834: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:05:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:16 smithi104 ceph-mon[97044]: pgmap v8834: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:05:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:16.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:16.231384+0000) 2022-09-10T09:05:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:16.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:16.744841+0000) 2022-09-10T09:05:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:17.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:17.231573+0000) 2022-09-10T09:05:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:17 smithi110 ceph-mon[97633]: pgmap v8835: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:05:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:17 smithi104 ceph-mon[97044]: pgmap v8835: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:05:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:17.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:17.745034+0000) 2022-09-10T09:05:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:18.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:18.231793+0000) 2022-09-10T09:05:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:18.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:18.745248+0000) 2022-09-10T09:05:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:19.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:19.231983+0000) 2022-09-10T09:05:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:19.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:19.745450+0000) 2022-09-10T09:05:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:20 smithi110 ceph-mon[97633]: pgmap v8836: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:05:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:20 smithi104 ceph-mon[97044]: pgmap v8836: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:05:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:20.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:20.232246+0000) 2022-09-10T09:05:21.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:20.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:20.745649+0000) 2022-09-10T09:05:21.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:05:20.690+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:05:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:21.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:21.232408+0000) 2022-09-10T09:05:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:21.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:21.745785+0000) 2022-09-10T09:05:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:22 smithi110 ceph-mon[97633]: pgmap v8837: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:05:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:22 smithi104 ceph-mon[97044]: pgmap v8837: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:05:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:22.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:22.232654+0000) 2022-09-10T09:05:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:22.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:22.746021+0000) 2022-09-10T09:05:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:23.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:23.232920+0000) 2022-09-10T09:05:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:23.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:23.746194+0000) 2022-09-10T09:05:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:24 smithi110 ceph-mon[97633]: pgmap v8838: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:05:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:24 smithi104 ceph-mon[97044]: pgmap v8838: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:05:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:24.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:24.233193+0000) 2022-09-10T09:05:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:24.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:24.746430+0000) 2022-09-10T09:05:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:25.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:25.233402+0000) 2022-09-10T09:05:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:25.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:25.746664+0000) 2022-09-10T09:05:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:26 smithi110 ceph-mon[97633]: pgmap v8839: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:05:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:26 smithi104 ceph-mon[97044]: pgmap v8839: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:05:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:26.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:26.233605+0000) 2022-09-10T09:05:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:26.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:26.746867+0000) 2022-09-10T09:05:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:27.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:27.233821+0000) 2022-09-10T09:05:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:27 smithi110 ceph-mon[97633]: pgmap v8840: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:27 smithi104 ceph-mon[97044]: pgmap v8840: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:27.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:27.747066+0000) 2022-09-10T09:05:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:28.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:28.234034+0000) 2022-09-10T09:05:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:28.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:28.747281+0000) 2022-09-10T09:05:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:29.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:29.234225+0000) 2022-09-10T09:05:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:29.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:29.747455+0000) 2022-09-10T09:05:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:30 smithi110 ceph-mon[97633]: pgmap v8841: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:30 smithi104 ceph-mon[97044]: pgmap v8841: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:30.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:30.234407+0000) 2022-09-10T09:05:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:30.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:30.747650+0000) 2022-09-10T09:05:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:31.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:31.234602+0000) 2022-09-10T09:05:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:31.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:31.747874+0000) 2022-09-10T09:05:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:32 smithi110 ceph-mon[97633]: pgmap v8842: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:32 smithi104 ceph-mon[97044]: pgmap v8842: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:32.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:32.234802+0000) 2022-09-10T09:05:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:32.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:32.748091+0000) 2022-09-10T09:05:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:33.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:33.235089+0000) 2022-09-10T09:05:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:33.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:33.748334+0000) 2022-09-10T09:05:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:34 smithi110 ceph-mon[97633]: pgmap v8843: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:34 smithi104 ceph-mon[97044]: pgmap v8843: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:34.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:34.235349+0000) 2022-09-10T09:05:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:34.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:34.748565+0000) 2022-09-10T09:05:35.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:35.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:35.235644+0000) 2022-09-10T09:05:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:35.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:35.748795+0000) 2022-09-10T09:05:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:05:35.703+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:05:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:36 smithi110 ceph-mon[97633]: pgmap v8844: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:36 smithi104 ceph-mon[97044]: pgmap v8844: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:36.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:36.235844+0000) 2022-09-10T09:05:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:36.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:36.748973+0000) 2022-09-10T09:05:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:37.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:37.236040+0000) 2022-09-10T09:05:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:37 smithi110 ceph-mon[97633]: pgmap v8845: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:37 smithi104 ceph-mon[97044]: pgmap v8845: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:37.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:37.749222+0000) 2022-09-10T09:05:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:38.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:38.236244+0000) 2022-09-10T09:05:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:38.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:38.749403+0000) 2022-09-10T09:05:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:39.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:39.236382+0000) 2022-09-10T09:05:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:39.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:39.749648+0000) 2022-09-10T09:05:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:40 smithi110 ceph-mon[97633]: pgmap v8846: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:40 smithi104 ceph-mon[97044]: pgmap v8846: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:40.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:40.236555+0000) 2022-09-10T09:05:41.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:40.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:40.749833+0000) 2022-09-10T09:05:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:41.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:41.236744+0000) 2022-09-10T09:05:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:41.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:41.750055+0000) 2022-09-10T09:05:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:42 smithi110 ceph-mon[97633]: pgmap v8847: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:42 smithi104 ceph-mon[97044]: pgmap v8847: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:42.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:42.236940+0000) 2022-09-10T09:05:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:42.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:42.750192+0000) 2022-09-10T09:05:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:43.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:43.237166+0000) 2022-09-10T09:05:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:43.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:43.750405+0000) 2022-09-10T09:05:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:44 smithi110 ceph-mon[97633]: pgmap v8848: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:44 smithi104 ceph-mon[97044]: pgmap v8848: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:44.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:44.237326+0000) 2022-09-10T09:05:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:44.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:44.750669+0000) 2022-09-10T09:05:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:05:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:05:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:05:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:05:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:05:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:05:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:05:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:05:45.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:45.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:45.237564+0000) 2022-09-10T09:05:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:45.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:45.750829+0000) 2022-09-10T09:05:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:46 smithi110 ceph-mon[97633]: pgmap v8849: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:46 smithi104 ceph-mon[97044]: pgmap v8849: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:46.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:46.237762+0000) 2022-09-10T09:05:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:46.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:46.751066+0000) 2022-09-10T09:05:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:47.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:47.237930+0000) 2022-09-10T09:05:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:47 smithi110 ceph-mon[97633]: pgmap v8850: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:47 smithi104 ceph-mon[97044]: pgmap v8850: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:47.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:47.751262+0000) 2022-09-10T09:05:48.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:48.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:48.238176+0000) 2022-09-10T09:05:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:48.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:48.751518+0000) 2022-09-10T09:05:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:49.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:49.238347+0000) 2022-09-10T09:05:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:49.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:49.751718+0000) 2022-09-10T09:05:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:50 smithi110 ceph-mon[97633]: pgmap v8851: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:50 smithi104 ceph-mon[97044]: pgmap v8851: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:50.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:50.238516+0000) 2022-09-10T09:05:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:50.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:50.751972+0000) 2022-09-10T09:05:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:05:50.718+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:05:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:51.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:51.238751+0000) 2022-09-10T09:05:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:51.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:51.752136+0000) 2022-09-10T09:05:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:52 smithi110 ceph-mon[97633]: pgmap v8852: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:52 smithi104 ceph-mon[97044]: pgmap v8852: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:52.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:52.239011+0000) 2022-09-10T09:05:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:52.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:52.752369+0000) 2022-09-10T09:05:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:53.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:53.239250+0000) 2022-09-10T09:05:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:53.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:53.752595+0000) 2022-09-10T09:05:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:54 smithi110 ceph-mon[97633]: pgmap v8853: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:54 smithi104 ceph-mon[97044]: pgmap v8853: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:54.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:54.239424+0000) 2022-09-10T09:05:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:54.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:54.752845+0000) 2022-09-10T09:05:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:55.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:55.239589+0000) 2022-09-10T09:05:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:55.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:55.753080+0000) 2022-09-10T09:05:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:56 smithi110 ceph-mon[97633]: pgmap v8854: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:56 smithi104 ceph-mon[97044]: pgmap v8854: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:05:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:56.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:56.239757+0000) 2022-09-10T09:05:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:56.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:56.753419+0000) 2022-09-10T09:05:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:57.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:57.239928+0000) 2022-09-10T09:05:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:05:57 smithi110 ceph-mon[97633]: pgmap v8855: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:05:57 smithi104 ceph-mon[97044]: pgmap v8855: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:05:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:57.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:57.753681+0000) 2022-09-10T09:05:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:58.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:58.240155+0000) 2022-09-10T09:05:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:58.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:58.753876+0000) 2022-09-10T09:05:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:05:59.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:59.240303+0000) 2022-09-10T09:06:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:05:59.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:05:59.754119+0000) 2022-09-10T09:06:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:00 smithi110 ceph-mon[97633]: pgmap v8856: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:00 smithi104 ceph-mon[97044]: pgmap v8856: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:00.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:00.240492+0000) 2022-09-10T09:06:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:00.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:00.754352+0000) 2022-09-10T09:06:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:01.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:01.240698+0000) 2022-09-10T09:06:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:01.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:01.754507+0000) 2022-09-10T09:06:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:02 smithi110 ceph-mon[97633]: pgmap v8857: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:02 smithi104 ceph-mon[97044]: pgmap v8857: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:02.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:02.240925+0000) 2022-09-10T09:06:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:02.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:02.754730+0000) 2022-09-10T09:06:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:03.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:03.241190+0000) 2022-09-10T09:06:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:03.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:03.754933+0000) 2022-09-10T09:06:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:04 smithi110 ceph-mon[97633]: pgmap v8858: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:04 smithi104 ceph-mon[97044]: pgmap v8858: 129 pgs: 129 active+clean; 8.6 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:04.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:04.241430+0000) 2022-09-10T09:06:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:04.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:04.755147+0000) 2022-09-10T09:06:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:05.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:05.241647+0000) 2022-09-10T09:06:06.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:06:05.733+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:06:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:05.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:05.755408+0000) 2022-09-10T09:06:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:06 smithi110 ceph-mon[97633]: pgmap v8859: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:06:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:06 smithi104 ceph-mon[97044]: pgmap v8859: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:06:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:06.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:06.241834+0000) 2022-09-10T09:06:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:06.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:06.755644+0000) 2022-09-10T09:06:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:07.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:07.242042+0000) 2022-09-10T09:06:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:07 smithi110 ceph-mon[97633]: pgmap v8860: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:06:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:07 smithi104 ceph-mon[97044]: pgmap v8860: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:06:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:07.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:07.755850+0000) 2022-09-10T09:06:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:08.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:08.242290+0000) 2022-09-10T09:06:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:08.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:08.756018+0000) 2022-09-10T09:06:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:09.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:09.242535+0000) 2022-09-10T09:06:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:09 smithi110 ceph-mon[97633]: osdmap e634: 12 total, 12 up, 12 in 2022-09-10T09:06:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:09 smithi110 ceph-mon[97633]: pgmap v8862: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:06:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:09 smithi104 ceph-mon[97044]: osdmap e634: 12 total, 12 up, 12 in 2022-09-10T09:06:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:09 smithi104 ceph-mon[97044]: pgmap v8862: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:06:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:09.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:09.756156+0000) 2022-09-10T09:06:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:10.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:10.242776+0000) 2022-09-10T09:06:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:10.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:10.756365+0000) 2022-09-10T09:06:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:11.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:11.243015+0000) 2022-09-10T09:06:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:11.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:11.756549+0000) 2022-09-10T09:06:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:12 smithi110 ceph-mon[97633]: pgmap v8863: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:06:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:12 smithi110 ceph-mon[97633]: osdmap e635: 12 total, 12 up, 12 in 2022-09-10T09:06:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:12 smithi104 ceph-mon[97044]: pgmap v8863: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:06:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:12 smithi104 ceph-mon[97044]: osdmap e635: 12 total, 12 up, 12 in 2022-09-10T09:06:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:12.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:12.243281+0000) 2022-09-10T09:06:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:12.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:12.756709+0000) 2022-09-10T09:06:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:13.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:13.243488+0000) 2022-09-10T09:06:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:13.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:13.756855+0000) 2022-09-10T09:06:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:14 smithi110 ceph-mon[97633]: pgmap v8865: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:06:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:14 smithi104 ceph-mon[97044]: pgmap v8865: 129 pgs: 129 active+clean; 8.6 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:06:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:14.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:14.243671+0000) 2022-09-10T09:06:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:14.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:14.757076+0000) 2022-09-10T09:06:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:15.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:15.243801+0000) 2022-09-10T09:06:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:15.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:15.757335+0000) 2022-09-10T09:06:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:16 smithi110 ceph-mon[97633]: pgmap v8866: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:06:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:16 smithi104 ceph-mon[97044]: pgmap v8866: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:06:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:16.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:16.244008+0000) 2022-09-10T09:06:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:16.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:16.757582+0000) 2022-09-10T09:06:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:17.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:17.244254+0000) 2022-09-10T09:06:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:17 smithi110 ceph-mon[97633]: pgmap v8867: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:06:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:17 smithi104 ceph-mon[97044]: pgmap v8867: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:06:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:17.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:17.757786+0000) 2022-09-10T09:06:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:18.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:18.244472+0000) 2022-09-10T09:06:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:18.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:18.758030+0000) 2022-09-10T09:06:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:19.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:19.244700+0000) 2022-09-10T09:06:20.021 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:19.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:19.758282+0000) 2022-09-10T09:06:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:20 smithi110 ceph-mon[97633]: pgmap v8868: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:06:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:20 smithi104 ceph-mon[97044]: pgmap v8868: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:06:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:20.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:20.244944+0000) 2022-09-10T09:06:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:20.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:20.758555+0000) 2022-09-10T09:06:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:06:20.745+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:06:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:21.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:21.245154+0000) 2022-09-10T09:06:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:21.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:21.758769+0000) 2022-09-10T09:06:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:22 smithi110 ceph-mon[97633]: pgmap v8869: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:06:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:22 smithi104 ceph-mon[97044]: pgmap v8869: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:06:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:22.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:22.245422+0000) 2022-09-10T09:06:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:22.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:22.758979+0000) 2022-09-10T09:06:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:23.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:23.245656+0000) 2022-09-10T09:06:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:23.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:23.759242+0000) 2022-09-10T09:06:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:24 smithi110 ceph-mon[97633]: pgmap v8870: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:06:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:24 smithi104 ceph-mon[97044]: pgmap v8870: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:06:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:24.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:24.245899+0000) 2022-09-10T09:06:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:24.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:24.759428+0000) 2022-09-10T09:06:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:25.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:25.246129+0000) 2022-09-10T09:06:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:25.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:25.759657+0000) 2022-09-10T09:06:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:26 smithi110 ceph-mon[97633]: pgmap v8871: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:06:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:26 smithi104 ceph-mon[97044]: pgmap v8871: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:06:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:26.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:26.246393+0000) 2022-09-10T09:06:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:26.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:26.759840+0000) 2022-09-10T09:06:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:27.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:27.246569+0000) 2022-09-10T09:06:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:27 smithi110 ceph-mon[97633]: pgmap v8872: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:27 smithi104 ceph-mon[97044]: pgmap v8872: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:27.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:27.760015+0000) 2022-09-10T09:06:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:28.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:28.246744+0000) 2022-09-10T09:06:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:28.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:28.760229+0000) 2022-09-10T09:06:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:29.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:29.246930+0000) 2022-09-10T09:06:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:29.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:29.760460+0000) 2022-09-10T09:06:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:30 smithi110 ceph-mon[97633]: pgmap v8873: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:30 smithi104 ceph-mon[97044]: pgmap v8873: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:30.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:30.247183+0000) 2022-09-10T09:06:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:30.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:30.760715+0000) 2022-09-10T09:06:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:31.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:31.247427+0000) 2022-09-10T09:06:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:31.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:31.760974+0000) 2022-09-10T09:06:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:32 smithi110 ceph-mon[97633]: pgmap v8874: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:32 smithi104 ceph-mon[97044]: pgmap v8874: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:32.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:32.247648+0000) 2022-09-10T09:06:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:32.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:32.761225+0000) 2022-09-10T09:06:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:33.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:33.247807+0000) 2022-09-10T09:06:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:33.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:33.761365+0000) 2022-09-10T09:06:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:34 smithi110 ceph-mon[97633]: pgmap v8875: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:34 smithi104 ceph-mon[97044]: pgmap v8875: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:34.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:34.247968+0000) 2022-09-10T09:06:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:34.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:34.761507+0000) 2022-09-10T09:06:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:35.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:35.248190+0000) 2022-09-10T09:06:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:35.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:35.761704+0000) 2022-09-10T09:06:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:06:35.759+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:06:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:36 smithi110 ceph-mon[97633]: pgmap v8876: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:36 smithi104 ceph-mon[97044]: pgmap v8876: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:36.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:36.248389+0000) 2022-09-10T09:06:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:36.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:36.761921+0000) 2022-09-10T09:06:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:37.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:37.248646+0000) 2022-09-10T09:06:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:37 smithi110 ceph-mon[97633]: pgmap v8877: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:37 smithi104 ceph-mon[97044]: pgmap v8877: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:37.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:37.762195+0000) 2022-09-10T09:06:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:38.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:38.248867+0000) 2022-09-10T09:06:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:38.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:38.762477+0000) 2022-09-10T09:06:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:39.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:39.249084+0000) 2022-09-10T09:06:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:39.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:39.762655+0000) 2022-09-10T09:06:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:40 smithi110 ceph-mon[97633]: pgmap v8878: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:40 smithi104 ceph-mon[97044]: pgmap v8878: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:40.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:40.249319+0000) 2022-09-10T09:06:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:40.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:40.762911+0000) 2022-09-10T09:06:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:41.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:41.249533+0000) 2022-09-10T09:06:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:41.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:41.763191+0000) 2022-09-10T09:06:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:42 smithi110 ceph-mon[97633]: pgmap v8879: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:42 smithi104 ceph-mon[97044]: pgmap v8879: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:42.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:42.249713+0000) 2022-09-10T09:06:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:42.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:42.763357+0000) 2022-09-10T09:06:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:43.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:43.249935+0000) 2022-09-10T09:06:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:43.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:43.763578+0000) 2022-09-10T09:06:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:44 smithi110 ceph-mon[97633]: pgmap v8880: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:44 smithi104 ceph-mon[97044]: pgmap v8880: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:44.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:44.250126+0000) 2022-09-10T09:06:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:44.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:44.763847+0000) 2022-09-10T09:06:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:06:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:06:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:06:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:06:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:06:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:06:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:06:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:06:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:45.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:45.250330+0000) 2022-09-10T09:06:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:45.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:45.764042+0000) 2022-09-10T09:06:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:46 smithi110 ceph-mon[97633]: pgmap v8881: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:46 smithi104 ceph-mon[97044]: pgmap v8881: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:46.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:46.250540+0000) 2022-09-10T09:06:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:46.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:46.764280+0000) 2022-09-10T09:06:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:47.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:47.250711+0000) 2022-09-10T09:06:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:47 smithi110 ceph-mon[97633]: pgmap v8882: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:47 smithi104 ceph-mon[97044]: pgmap v8882: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:47.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:47.764516+0000) 2022-09-10T09:06:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:48.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:48.250926+0000) 2022-09-10T09:06:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:48.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:48.764732+0000) 2022-09-10T09:06:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:49.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:49.251215+0000) 2022-09-10T09:06:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:49.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:49.764993+0000) 2022-09-10T09:06:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:50 smithi110 ceph-mon[97633]: pgmap v8883: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:50 smithi104 ceph-mon[97044]: pgmap v8883: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:50.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:50.251418+0000) 2022-09-10T09:06:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:50.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:50.765203+0000) 2022-09-10T09:06:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:06:50.773+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:06:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:51.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:51.251656+0000) 2022-09-10T09:06:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:51.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:51.765369+0000) 2022-09-10T09:06:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:52 smithi110 ceph-mon[97633]: pgmap v8884: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:52.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:52 smithi104 ceph-mon[97044]: pgmap v8884: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:52.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:52.251925+0000) 2022-09-10T09:06:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:52.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:52.765584+0000) 2022-09-10T09:06:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:53.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:53.252172+0000) 2022-09-10T09:06:54.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:53.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:53.765748+0000) 2022-09-10T09:06:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:54 smithi110 ceph-mon[97633]: pgmap v8885: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:54 smithi104 ceph-mon[97044]: pgmap v8885: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:54.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:54.252340+0000) 2022-09-10T09:06:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:54.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:54.766018+0000) 2022-09-10T09:06:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:55.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:55.252531+0000) 2022-09-10T09:06:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:55.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:55.766278+0000) 2022-09-10T09:06:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:56 smithi110 ceph-mon[97633]: pgmap v8886: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:56 smithi104 ceph-mon[97044]: pgmap v8886: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:06:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:56.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:56.252759+0000) 2022-09-10T09:06:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:56.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:56.766563+0000) 2022-09-10T09:06:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:57.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:57.253041+0000) 2022-09-10T09:06:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:06:57 smithi110 ceph-mon[97633]: pgmap v8887: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:06:57 smithi104 ceph-mon[97044]: pgmap v8887: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:06:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:57.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:57.766764+0000) 2022-09-10T09:06:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:58.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:58.253273+0000) 2022-09-10T09:06:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:58.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:58.767009+0000) 2022-09-10T09:06:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:06:59.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:59.253499+0000) 2022-09-10T09:07:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:06:59.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:06:59.767200+0000) 2022-09-10T09:07:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:00 smithi110 ceph-mon[97633]: pgmap v8888: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:00 smithi104 ceph-mon[97044]: pgmap v8888: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:00.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:00.253706+0000) 2022-09-10T09:07:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:00.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:00.767451+0000) 2022-09-10T09:07:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:01.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:01.253912+0000) 2022-09-10T09:07:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:01.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:01.767733+0000) 2022-09-10T09:07:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:02 smithi110 ceph-mon[97633]: pgmap v8889: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:02 smithi104 ceph-mon[97044]: pgmap v8889: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:02.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:02.254190+0000) 2022-09-10T09:07:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:02.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:02.767961+0000) 2022-09-10T09:07:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:03.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:03.254479+0000) 2022-09-10T09:07:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:03.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:03.768153+0000) 2022-09-10T09:07:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:04 smithi110 ceph-mon[97633]: pgmap v8890: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:04 smithi104 ceph-mon[97044]: pgmap v8890: 129 pgs: 129 active+clean; 8.6 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:04.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:04.254939+0000) 2022-09-10T09:07:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:04.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:04.768335+0000) 2022-09-10T09:07:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:05.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:05.255224+0000) 2022-09-10T09:07:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:05.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:05.768497+0000) 2022-09-10T09:07:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:07:05.786+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:07:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:06 smithi110 ceph-mon[97633]: pgmap v8891: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:07:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:06 smithi104 ceph-mon[97044]: pgmap v8891: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:07:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:06.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:06.255427+0000) 2022-09-10T09:07:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:06.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:06.768670+0000) 2022-09-10T09:07:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:07.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:07.255612+0000) 2022-09-10T09:07:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:07 smithi110 ceph-mon[97633]: pgmap v8892: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:07:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:07 smithi104 ceph-mon[97044]: pgmap v8892: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:07:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:07.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:07.768888+0000) 2022-09-10T09:07:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:08.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:08.255851+0000) 2022-09-10T09:07:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:08.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:08.769136+0000) 2022-09-10T09:07:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:09.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:09.256012+0000) 2022-09-10T09:07:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:09 smithi110 ceph-mon[97633]: osdmap e636: 12 total, 12 up, 12 in 2022-09-10T09:07:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:09 smithi110 ceph-mon[97633]: pgmap v8894: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:07:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:09 smithi104 ceph-mon[97044]: osdmap e636: 12 total, 12 up, 12 in 2022-09-10T09:07:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:09 smithi104 ceph-mon[97044]: pgmap v8894: 129 pgs: 129 active+clean; 8.6 MiB data, 864 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:07:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:09.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:09.769369+0000) 2022-09-10T09:07:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:10.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:10.256270+0000) 2022-09-10T09:07:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:10.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:10.769628+0000) 2022-09-10T09:07:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:11.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:11.256488+0000) 2022-09-10T09:07:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:11.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:11.769835+0000) 2022-09-10T09:07:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:12 smithi110 ceph-mon[97633]: pgmap v8895: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:07:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:12 smithi110 ceph-mon[97633]: osdmap e637: 12 total, 12 up, 12 in 2022-09-10T09:07:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:12 smithi104 ceph-mon[97044]: pgmap v8895: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:07:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:12 smithi104 ceph-mon[97044]: osdmap e637: 12 total, 12 up, 12 in 2022-09-10T09:07:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:12.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:12.256652+0000) 2022-09-10T09:07:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:12.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:12.769970+0000) 2022-09-10T09:07:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:13.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:13.256899+0000) 2022-09-10T09:07:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:13.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:13.770137+0000) 2022-09-10T09:07:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:14 smithi110 ceph-mon[97633]: pgmap v8897: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:07:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:14 smithi104 ceph-mon[97044]: pgmap v8897: 129 pgs: 129 active+clean; 8.6 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:07:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:14.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:14.257153+0000) 2022-09-10T09:07:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:14.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:14.770271+0000) 2022-09-10T09:07:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:15.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:15.257372+0000) 2022-09-10T09:07:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:15.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:15.770455+0000) 2022-09-10T09:07:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:16 smithi104 ceph-mon[97044]: pgmap v8898: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:07:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:16.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:16.257647+0000) 2022-09-10T09:07:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:16 smithi110 ceph-mon[97633]: pgmap v8898: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:07:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:16.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:16.770711+0000) 2022-09-10T09:07:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:17.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:17.257893+0000) 2022-09-10T09:07:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:17 smithi110 ceph-mon[97633]: pgmap v8899: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:07:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:17 smithi104 ceph-mon[97044]: pgmap v8899: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 235 B/s wr, 11 op/s 2022-09-10T09:07:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:17.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:17.770892+0000) 2022-09-10T09:07:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:18.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:18.258089+0000) 2022-09-10T09:07:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:18.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:18.771144+0000) 2022-09-10T09:07:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:19.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:19.258308+0000) 2022-09-10T09:07:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:19.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:19.771397+0000) 2022-09-10T09:07:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:20 smithi110 ceph-mon[97633]: pgmap v8900: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:07:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:20 smithi104 ceph-mon[97044]: pgmap v8900: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:07:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:20.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:20.258493+0000) 2022-09-10T09:07:21.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:07:20.801+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:07:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:20.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:20.771592+0000) 2022-09-10T09:07:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:21.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:21.258655+0000) 2022-09-10T09:07:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:21.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:21.771859+0000) 2022-09-10T09:07:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:22 smithi110 ceph-mon[97633]: pgmap v8901: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:07:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:22 smithi104 ceph-mon[97044]: pgmap v8901: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:07:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:22.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:22.258871+0000) 2022-09-10T09:07:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:22.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:22.772080+0000) 2022-09-10T09:07:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:23.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:23.259137+0000) 2022-09-10T09:07:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:23.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:23.772270+0000) 2022-09-10T09:07:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:24 smithi104 ceph-mon[97044]: pgmap v8902: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:07:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:24.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:24.259318+0000) 2022-09-10T09:07:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:24 smithi110 ceph-mon[97633]: pgmap v8902: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:07:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:24.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:24.772531+0000) 2022-09-10T09:07:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:25.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:25.259505+0000) 2022-09-10T09:07:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:25.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:25.772738+0000) 2022-09-10T09:07:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:26 smithi104 ceph-mon[97044]: pgmap v8903: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:07:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:26.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:26.259747+0000) 2022-09-10T09:07:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:26 smithi110 ceph-mon[97633]: pgmap v8903: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:07:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:26.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:26.772980+0000) 2022-09-10T09:07:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:27.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:27.259959+0000) 2022-09-10T09:07:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:27 smithi110 ceph-mon[97633]: pgmap v8904: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:27 smithi104 ceph-mon[97044]: pgmap v8904: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:27.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:27.773179+0000) 2022-09-10T09:07:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:28.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:28.260259+0000) 2022-09-10T09:07:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:28.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:28.773334+0000) 2022-09-10T09:07:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:29.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:29.260430+0000) 2022-09-10T09:07:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:29.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:29.773570+0000) 2022-09-10T09:07:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:30 smithi104 ceph-mon[97044]: pgmap v8905: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:30.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:30.260665+0000) 2022-09-10T09:07:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:30 smithi110 ceph-mon[97633]: pgmap v8905: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:30.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:30.773733+0000) 2022-09-10T09:07:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:31.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:31.260915+0000) 2022-09-10T09:07:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:31.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:31.773948+0000) 2022-09-10T09:07:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:32 smithi104 ceph-mon[97044]: pgmap v8906: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:32.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:32.261177+0000) 2022-09-10T09:07:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:32 smithi110 ceph-mon[97633]: pgmap v8906: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:32.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:32.774136+0000) 2022-09-10T09:07:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:33.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:33.261443+0000) 2022-09-10T09:07:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:33.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:33.774312+0000) 2022-09-10T09:07:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:34 smithi104 ceph-mon[97044]: pgmap v8907: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:34.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:34.261712+0000) 2022-09-10T09:07:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:34 smithi110 ceph-mon[97633]: pgmap v8907: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:34.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:34.774542+0000) 2022-09-10T09:07:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:35.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:35.261908+0000) 2022-09-10T09:07:36.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:07:35.812+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:07:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:35.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:35.774829+0000) 2022-09-10T09:07:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:36 smithi104 ceph-mon[97044]: pgmap v8908: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:36.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:36.262186+0000) 2022-09-10T09:07:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:36 smithi110 ceph-mon[97633]: pgmap v8908: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:36.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:36.775054+0000) 2022-09-10T09:07:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:37.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:37.262433+0000) 2022-09-10T09:07:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:37 smithi110 ceph-mon[97633]: pgmap v8909: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:37.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:37 smithi104 ceph-mon[97044]: pgmap v8909: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:37.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:37.775280+0000) 2022-09-10T09:07:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:38.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:38.262653+0000) 2022-09-10T09:07:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:38.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:38.775499+0000) 2022-09-10T09:07:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:39.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:39.262895+0000) 2022-09-10T09:07:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:39.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:39.775675+0000) 2022-09-10T09:07:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:40 smithi104 ceph-mon[97044]: pgmap v8910: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:40.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:40.263171+0000) 2022-09-10T09:07:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:40 smithi110 ceph-mon[97633]: pgmap v8910: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:40.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:40.775899+0000) 2022-09-10T09:07:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:41.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:41.263377+0000) 2022-09-10T09:07:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:41.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:41.776078+0000) 2022-09-10T09:07:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:42 smithi104 ceph-mon[97044]: pgmap v8911: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:42.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:42.263616+0000) 2022-09-10T09:07:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:42 smithi110 ceph-mon[97633]: pgmap v8911: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:42.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:42.776322+0000) 2022-09-10T09:07:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:43.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:43.263887+0000) 2022-09-10T09:07:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:43.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:43.776530+0000) 2022-09-10T09:07:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:44 smithi104 ceph-mon[97044]: pgmap v8912: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:44.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:44.264188+0000) 2022-09-10T09:07:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:44 smithi110 ceph-mon[97633]: pgmap v8912: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:44.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:44.776721+0000) 2022-09-10T09:07:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:07:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:07:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:07:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:07:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:45.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:45.264462+0000) 2022-09-10T09:07:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:07:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:07:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:07:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:07:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:45.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:45.776903+0000) 2022-09-10T09:07:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:46 smithi104 ceph-mon[97044]: pgmap v8913: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:46.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:46.264687+0000) 2022-09-10T09:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:46 smithi110 ceph-mon[97633]: pgmap v8913: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:46.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:46.777171+0000) 2022-09-10T09:07:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:47.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:47.264969+0000) 2022-09-10T09:07:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:47 smithi110 ceph-mon[97633]: pgmap v8914: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:47 smithi104 ceph-mon[97044]: pgmap v8914: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:47.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:47.777414+0000) 2022-09-10T09:07:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:48.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:48.265222+0000) 2022-09-10T09:07:49.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:48.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:48.777624+0000) 2022-09-10T09:07:49.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:49.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:49.265449+0000) 2022-09-10T09:07:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:49.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:49.777904+0000) 2022-09-10T09:07:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:50 smithi104 ceph-mon[97044]: pgmap v8915: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:50.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:50.265717+0000) 2022-09-10T09:07:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:50 smithi110 ceph-mon[97633]: pgmap v8915: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:50.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:50.778181+0000) 2022-09-10T09:07:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:07:50.827+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:07:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:51.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:51.265921+0000) 2022-09-10T09:07:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:51.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:51.778326+0000) 2022-09-10T09:07:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:52 smithi104 ceph-mon[97044]: pgmap v8916: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:52.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:52.266154+0000) 2022-09-10T09:07:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:52 smithi110 ceph-mon[97633]: pgmap v8916: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:52.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:52.778527+0000) 2022-09-10T09:07:53.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:53.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:53.266389+0000) 2022-09-10T09:07:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:53.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:53.778765+0000) 2022-09-10T09:07:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:54 smithi104 ceph-mon[97044]: pgmap v8917: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:54.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:54.266608+0000) 2022-09-10T09:07:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:54 smithi110 ceph-mon[97633]: pgmap v8917: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:54.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:54.778954+0000) 2022-09-10T09:07:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:55.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:55.266843+0000) 2022-09-10T09:07:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:55.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:55.779199+0000) 2022-09-10T09:07:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:56 smithi104 ceph-mon[97044]: pgmap v8918: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:56.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:56.267130+0000) 2022-09-10T09:07:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:56 smithi110 ceph-mon[97633]: pgmap v8918: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:07:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:56.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:56.779429+0000) 2022-09-10T09:07:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:57.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:57.267341+0000) 2022-09-10T09:07:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:07:57 smithi104 ceph-mon[97044]: pgmap v8919: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:07:57 smithi110 ceph-mon[97633]: pgmap v8919: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:07:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:57.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:57.779685+0000) 2022-09-10T09:07:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:58.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:58.267571+0000) 2022-09-10T09:07:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:58.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:58.779928+0000) 2022-09-10T09:07:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:07:59.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:59.267803+0000) 2022-09-10T09:08:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:07:59.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:07:59.780189+0000) 2022-09-10T09:08:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:00 smithi104 ceph-mon[97044]: pgmap v8920: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:00.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:00.267948+0000) 2022-09-10T09:08:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:00 smithi110 ceph-mon[97633]: pgmap v8920: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:00.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:00.780442+0000) 2022-09-10T09:08:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:01.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:01.268227+0000) 2022-09-10T09:08:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:01.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:01.780607+0000) 2022-09-10T09:08:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:02 smithi104 ceph-mon[97044]: pgmap v8921: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:02.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:02.268454+0000) 2022-09-10T09:08:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:02 smithi110 ceph-mon[97633]: pgmap v8921: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:02.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:02.780795+0000) 2022-09-10T09:08:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:03.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:03.268725+0000) 2022-09-10T09:08:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:03.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:03.781020+0000) 2022-09-10T09:08:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:04 smithi104 ceph-mon[97044]: pgmap v8922: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:04.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:04.268887+0000) 2022-09-10T09:08:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:04 smithi110 ceph-mon[97633]: pgmap v8922: 129 pgs: 129 active+clean; 8.6 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:04.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:04.781215+0000) 2022-09-10T09:08:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:05.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:05.269071+0000) 2022-09-10T09:08:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:05.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:05.781458+0000) 2022-09-10T09:08:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:08:05.838+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:08:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:06 smithi104 ceph-mon[97044]: pgmap v8923: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:08:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:06.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:06.269313+0000) 2022-09-10T09:08:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:06 smithi110 ceph-mon[97633]: pgmap v8923: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:08:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:06.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:06.781656+0000) 2022-09-10T09:08:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:07.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:07.269569+0000) 2022-09-10T09:08:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:07 smithi104 ceph-mon[97044]: pgmap v8924: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:08:07.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:07 smithi110 ceph-mon[97633]: pgmap v8924: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:08:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:07.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:07.781890+0000) 2022-09-10T09:08:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:08.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:08.269831+0000) 2022-09-10T09:08:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:08 smithi104 ceph-mon[97044]: osdmap e638: 12 total, 12 up, 12 in 2022-09-10T09:08:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:08 smithi110 ceph-mon[97633]: osdmap e638: 12 total, 12 up, 12 in 2022-09-10T09:08:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:08.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:08.782086+0000) 2022-09-10T09:08:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:09.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:09.270032+0000) 2022-09-10T09:08:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:09 smithi104 ceph-mon[97044]: pgmap v8926: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:09 smithi110 ceph-mon[97633]: pgmap v8926: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:08:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:09.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:09.782273+0000) 2022-09-10T09:08:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:10.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:10.270275+0000) 2022-09-10T09:08:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:10.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:10.782466+0000) 2022-09-10T09:08:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:11.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:11.270530+0000) 2022-09-10T09:08:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:11.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:11.782700+0000) 2022-09-10T09:08:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:12 smithi104 ceph-mon[97044]: pgmap v8927: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:08:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:12 smithi104 ceph-mon[97044]: osdmap e639: 12 total, 12 up, 12 in 2022-09-10T09:08:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:12.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:12.270735+0000) 2022-09-10T09:08:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:12 smithi110 ceph-mon[97633]: pgmap v8927: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:08:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:12 smithi110 ceph-mon[97633]: osdmap e639: 12 total, 12 up, 12 in 2022-09-10T09:08:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:12.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:12.782961+0000) 2022-09-10T09:08:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:13.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:13.270968+0000) 2022-09-10T09:08:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:13.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:13.783172+0000) 2022-09-10T09:08:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:14 smithi104 ceph-mon[97044]: pgmap v8929: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:08:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:14.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:14.271215+0000) 2022-09-10T09:08:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:14 smithi110 ceph-mon[97633]: pgmap v8929: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:08:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:14.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:14.783344+0000) 2022-09-10T09:08:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:15.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:15.271450+0000) 2022-09-10T09:08:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:15.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:15.783542+0000) 2022-09-10T09:08:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:16 smithi104 ceph-mon[97044]: pgmap v8930: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:08:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:16.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:16.271674+0000) 2022-09-10T09:08:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:16 smithi110 ceph-mon[97633]: pgmap v8930: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:08:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:16.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:16.783716+0000) 2022-09-10T09:08:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:17.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:17.271890+0000) 2022-09-10T09:08:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:17 smithi104 ceph-mon[97044]: pgmap v8931: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T09:08:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:17 smithi110 ceph-mon[97633]: pgmap v8931: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T09:08:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:17.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:17.783932+0000) 2022-09-10T09:08:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:18.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:18.272158+0000) 2022-09-10T09:08:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:18.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:18.784142+0000) 2022-09-10T09:08:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:19.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:19.272398+0000) 2022-09-10T09:08:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:19.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:19.784325+0000) 2022-09-10T09:08:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:20 smithi104 ceph-mon[97044]: pgmap v8932: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:08:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:20.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:20.272661+0000) 2022-09-10T09:08:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:20 smithi110 ceph-mon[97633]: pgmap v8932: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:08:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:20.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:20.784567+0000) 2022-09-10T09:08:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:08:20.851+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:08:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:21.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:21.272926+0000) 2022-09-10T09:08:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:21.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:21.784837+0000) 2022-09-10T09:08:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:22 smithi104 ceph-mon[97044]: pgmap v8933: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:08:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:22.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:22.273171+0000) 2022-09-10T09:08:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:22 smithi110 ceph-mon[97633]: pgmap v8933: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:08:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:22.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:22.785037+0000) 2022-09-10T09:08:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:23.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:23.273391+0000) 2022-09-10T09:08:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:23.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:23.785245+0000) 2022-09-10T09:08:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:24 smithi104 ceph-mon[97044]: pgmap v8934: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:08:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:24.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:24.273636+0000) 2022-09-10T09:08:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:24 smithi110 ceph-mon[97633]: pgmap v8934: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:08:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:24.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:24.785419+0000) 2022-09-10T09:08:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:25.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:25.273862+0000) 2022-09-10T09:08:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:25.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:25.785681+0000) 2022-09-10T09:08:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:26 smithi104 ceph-mon[97044]: pgmap v8935: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:08:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:26.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:26.274058+0000) 2022-09-10T09:08:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:26 smithi110 ceph-mon[97633]: pgmap v8935: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:08:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:26.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:26.785828+0000) 2022-09-10T09:08:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:27.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:27.274272+0000) 2022-09-10T09:08:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:27 smithi104 ceph-mon[97044]: pgmap v8936: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:27 smithi110 ceph-mon[97633]: pgmap v8936: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:27.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:27.786065+0000) 2022-09-10T09:08:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:28.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:28.274472+0000) 2022-09-10T09:08:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:28.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:28.786287+0000) 2022-09-10T09:08:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:29.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:29.274715+0000) 2022-09-10T09:08:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:29.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:29.786503+0000) 2022-09-10T09:08:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:30 smithi104 ceph-mon[97044]: pgmap v8937: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:30.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:30.274947+0000) 2022-09-10T09:08:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:30 smithi110 ceph-mon[97633]: pgmap v8937: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:30.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:30.786754+0000) 2022-09-10T09:08:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:31.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:31.275202+0000) 2022-09-10T09:08:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:31.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:31.786955+0000) 2022-09-10T09:08:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:32 smithi104 ceph-mon[97044]: pgmap v8938: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:32.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:32.275393+0000) 2022-09-10T09:08:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:32 smithi110 ceph-mon[97633]: pgmap v8938: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:32.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:32.787179+0000) 2022-09-10T09:08:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:33.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:33.275604+0000) 2022-09-10T09:08:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:33.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:33.787396+0000) 2022-09-10T09:08:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:34 smithi104 ceph-mon[97044]: pgmap v8939: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:34.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:34.275860+0000) 2022-09-10T09:08:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:34 smithi110 ceph-mon[97633]: pgmap v8939: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:34.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:34.787610+0000) 2022-09-10T09:08:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:35.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:35.276142+0000) 2022-09-10T09:08:36.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:35.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:35.787822+0000) 2022-09-10T09:08:36.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:08:35.859+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:08:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:36 smithi104 ceph-mon[97044]: pgmap v8940: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:36.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:36.276327+0000) 2022-09-10T09:08:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:36 smithi110 ceph-mon[97633]: pgmap v8940: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:36.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:36.788063+0000) 2022-09-10T09:08:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:37.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:37.276593+0000) 2022-09-10T09:08:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:37 smithi104 ceph-mon[97044]: pgmap v8941: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:37 smithi110 ceph-mon[97633]: pgmap v8941: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:37.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:37.788332+0000) 2022-09-10T09:08:38.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:38.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:38.276788+0000) 2022-09-10T09:08:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:38.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:38.788544+0000) 2022-09-10T09:08:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:39.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:39.277046+0000) 2022-09-10T09:08:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:39.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:39.788721+0000) 2022-09-10T09:08:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:40 smithi104 ceph-mon[97044]: pgmap v8942: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:40.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:40.277239+0000) 2022-09-10T09:08:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:40 smithi110 ceph-mon[97633]: pgmap v8942: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:40.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:40.788977+0000) 2022-09-10T09:08:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:41.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:41.277467+0000) 2022-09-10T09:08:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:41.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:41.789210+0000) 2022-09-10T09:08:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:42 smithi104 ceph-mon[97044]: pgmap v8943: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:42.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:42.277676+0000) 2022-09-10T09:08:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:42 smithi110 ceph-mon[97633]: pgmap v8943: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:42.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:42.789461+0000) 2022-09-10T09:08:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:43.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:43.277888+0000) 2022-09-10T09:08:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:43.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:43.789716+0000) 2022-09-10T09:08:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:44 smithi104 ceph-mon[97044]: pgmap v8944: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:44.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:44.278081+0000) 2022-09-10T09:08:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:44 smithi110 ceph-mon[97633]: pgmap v8944: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:44.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:44.789976+0000) 2022-09-10T09:08:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:08:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:08:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:08:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:08:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:45.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:45.278319+0000) 2022-09-10T09:08:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:08:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:08:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:08:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:08:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:45.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:45.790240+0000) 2022-09-10T09:08:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:46 smithi104 ceph-mon[97044]: pgmap v8945: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:46.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:46.278624+0000) 2022-09-10T09:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:46 smithi110 ceph-mon[97633]: pgmap v8945: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:46.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:46.790528+0000) 2022-09-10T09:08:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:47.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:47.278805+0000) 2022-09-10T09:08:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:47 smithi104 ceph-mon[97044]: pgmap v8946: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:47 smithi110 ceph-mon[97633]: pgmap v8946: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:47.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:47.790701+0000) 2022-09-10T09:08:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:48.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:48.279131+0000) 2022-09-10T09:08:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:48.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:48.790830+0000) 2022-09-10T09:08:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:49.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:49.279311+0000) 2022-09-10T09:08:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:49.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:49.791052+0000) 2022-09-10T09:08:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:50 smithi104 ceph-mon[97044]: pgmap v8947: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:50.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:50.279556+0000) 2022-09-10T09:08:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:50 smithi110 ceph-mon[97633]: pgmap v8947: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:50.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:50.791306+0000) 2022-09-10T09:08:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:08:50.861+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:08:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:51.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:51.279798+0000) 2022-09-10T09:08:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:51.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:51.791504+0000) 2022-09-10T09:08:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:52 smithi104 ceph-mon[97044]: pgmap v8948: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:52.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:52.280130+0000) 2022-09-10T09:08:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:52 smithi110 ceph-mon[97633]: pgmap v8948: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:52.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:52.791751+0000) 2022-09-10T09:08:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:53.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:53.280372+0000) 2022-09-10T09:08:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:53.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:53.791898+0000) 2022-09-10T09:08:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:54 smithi104 ceph-mon[97044]: pgmap v8949: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:54.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:54.280666+0000) 2022-09-10T09:08:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:54 smithi110 ceph-mon[97633]: pgmap v8949: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:54.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:54.792054+0000) 2022-09-10T09:08:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:55.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:55.280916+0000) 2022-09-10T09:08:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:55.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:55.792265+0000) 2022-09-10T09:08:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:56 smithi104 ceph-mon[97044]: pgmap v8950: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:56.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:56.281195+0000) 2022-09-10T09:08:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:56 smithi110 ceph-mon[97633]: pgmap v8950: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:08:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:56.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:56.792434+0000) 2022-09-10T09:08:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:57.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:57.281367+0000) 2022-09-10T09:08:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:08:57 smithi104 ceph-mon[97044]: pgmap v8951: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:08:57 smithi110 ceph-mon[97633]: pgmap v8951: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:08:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:57.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:57.792669+0000) 2022-09-10T09:08:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:58.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:58.281552+0000) 2022-09-10T09:08:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:58.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:58.792863+0000) 2022-09-10T09:08:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:08:59.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:59.281770+0000) 2022-09-10T09:09:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:08:59.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:08:59.793084+0000) 2022-09-10T09:09:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:00 smithi104 ceph-mon[97044]: pgmap v8952: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:00.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:00.281946+0000) 2022-09-10T09:09:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:00 smithi110 ceph-mon[97633]: pgmap v8952: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:00.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:00.793221+0000) 2022-09-10T09:09:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:01.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:01.282142+0000) 2022-09-10T09:09:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:01.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:01.793447+0000) 2022-09-10T09:09:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:02 smithi104 ceph-mon[97044]: pgmap v8953: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:02.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:02.282330+0000) 2022-09-10T09:09:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:02 smithi110 ceph-mon[97633]: pgmap v8953: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:02.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:02.793729+0000) 2022-09-10T09:09:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:03.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:03.282505+0000) 2022-09-10T09:09:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:03.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:03.793900+0000) 2022-09-10T09:09:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:04 smithi104 ceph-mon[97044]: pgmap v8954: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:04.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:04.282705+0000) 2022-09-10T09:09:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:04 smithi110 ceph-mon[97633]: pgmap v8954: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:04.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:04.794062+0000) 2022-09-10T09:09:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:05.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:05.282910+0000) 2022-09-10T09:09:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:05.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:05.794293+0000) 2022-09-10T09:09:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:09:05.878+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:09:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:06 smithi104 ceph-mon[97044]: pgmap v8955: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:09:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:06.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:06.283188+0000) 2022-09-10T09:09:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:06 smithi110 ceph-mon[97633]: pgmap v8955: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:09:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:06.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:06.794522+0000) 2022-09-10T09:09:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:07.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:07.283409+0000) 2022-09-10T09:09:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:07 smithi104 ceph-mon[97044]: pgmap v8956: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:09:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:07 smithi110 ceph-mon[97633]: pgmap v8956: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:09:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:07.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:07.794749+0000) 2022-09-10T09:09:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:08.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:08.283638+0000) 2022-09-10T09:09:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:08 smithi104 ceph-mon[97044]: osdmap e640: 12 total, 12 up, 12 in 2022-09-10T09:09:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:08 smithi110 ceph-mon[97633]: osdmap e640: 12 total, 12 up, 12 in 2022-09-10T09:09:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:08.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:08.794936+0000) 2022-09-10T09:09:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:09.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:09.283909+0000) 2022-09-10T09:09:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:09 smithi104 ceph-mon[97044]: pgmap v8958: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:09 smithi110 ceph-mon[97633]: pgmap v8958: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:09:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:09.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:09.795187+0000) 2022-09-10T09:09:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:10.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:10.284159+0000) 2022-09-10T09:09:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:10.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:10.795358+0000) 2022-09-10T09:09:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:11.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:11.284408+0000) 2022-09-10T09:09:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:11.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:11.795599+0000) 2022-09-10T09:09:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:12 smithi104 ceph-mon[97044]: pgmap v8959: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:09:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:12 smithi104 ceph-mon[97044]: osdmap e641: 12 total, 12 up, 12 in 2022-09-10T09:09:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:12.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:12.284614+0000) 2022-09-10T09:09:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:12 smithi110 ceph-mon[97633]: pgmap v8959: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:09:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:12 smithi110 ceph-mon[97633]: osdmap e641: 12 total, 12 up, 12 in 2022-09-10T09:09:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:12.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:12.795859+0000) 2022-09-10T09:09:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:13.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:13.284832+0000) 2022-09-10T09:09:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:13.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:13.796142+0000) 2022-09-10T09:09:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:14 smithi104 ceph-mon[97044]: pgmap v8961: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:09:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:14.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:14.285032+0000) 2022-09-10T09:09:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:14 smithi110 ceph-mon[97633]: pgmap v8961: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:09:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:14.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:14.796290+0000) 2022-09-10T09:09:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:15.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:15.285230+0000) 2022-09-10T09:09:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:15.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:15.796450+0000) 2022-09-10T09:09:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:16 smithi104 ceph-mon[97044]: pgmap v8962: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:09:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:16.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:16.285437+0000) 2022-09-10T09:09:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:16 smithi110 ceph-mon[97633]: pgmap v8962: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:09:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:16.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:16.796626+0000) 2022-09-10T09:09:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:17.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:17.285633+0000) 2022-09-10T09:09:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:17 smithi104 ceph-mon[97044]: pgmap v8963: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:09:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:17 smithi110 ceph-mon[97633]: pgmap v8963: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:09:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:17.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:17.796837+0000) 2022-09-10T09:09:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:18.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:18.285890+0000) 2022-09-10T09:09:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:18.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:18.797064+0000) 2022-09-10T09:09:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:19.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:19.286185+0000) 2022-09-10T09:09:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:19.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:19.797306+0000) 2022-09-10T09:09:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:20 smithi104 ceph-mon[97044]: pgmap v8964: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:09:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:20.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:20.286370+0000) 2022-09-10T09:09:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:20 smithi110 ceph-mon[97633]: pgmap v8964: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:09:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:20.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:20.797473+0000) 2022-09-10T09:09:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:09:20.892+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:09:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:21.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:21.286605+0000) 2022-09-10T09:09:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:21.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:21.797647+0000) 2022-09-10T09:09:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:22 smithi104 ceph-mon[97044]: pgmap v8965: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:09:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:22.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:22.286833+0000) 2022-09-10T09:09:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:22 smithi110 ceph-mon[97633]: pgmap v8965: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:09:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:22.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:22.797826+0000) 2022-09-10T09:09:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:23.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:23.287154+0000) 2022-09-10T09:09:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:23.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:23.797964+0000) 2022-09-10T09:09:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:24 smithi104 ceph-mon[97044]: pgmap v8966: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:09:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:24.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:24.287369+0000) 2022-09-10T09:09:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:24 smithi110 ceph-mon[97633]: pgmap v8966: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:09:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:24.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:24.798207+0000) 2022-09-10T09:09:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:25.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:25.287596+0000) 2022-09-10T09:09:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:25.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:25.798456+0000) 2022-09-10T09:09:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:26 smithi104 ceph-mon[97044]: pgmap v8967: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:09:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:26.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:26.287778+0000) 2022-09-10T09:09:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:26 smithi110 ceph-mon[97633]: pgmap v8967: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:09:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:26.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:26.798715+0000) 2022-09-10T09:09:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:27.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:27.288000+0000) 2022-09-10T09:09:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:27 smithi104 ceph-mon[97044]: pgmap v8968: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:27 smithi110 ceph-mon[97633]: pgmap v8968: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:27.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:27.798966+0000) 2022-09-10T09:09:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:28.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:28.288273+0000) 2022-09-10T09:09:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:28.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:28.799148+0000) 2022-09-10T09:09:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:29.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:29.288539+0000) 2022-09-10T09:09:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:29.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:29.799412+0000) 2022-09-10T09:09:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:30 smithi104 ceph-mon[97044]: pgmap v8969: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:30.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:30.288806+0000) 2022-09-10T09:09:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:30 smithi110 ceph-mon[97633]: pgmap v8969: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:30.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:30.799610+0000) 2022-09-10T09:09:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:31.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:31.289019+0000) 2022-09-10T09:09:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:31.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:31.799840+0000) 2022-09-10T09:09:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:32 smithi104 ceph-mon[97044]: pgmap v8970: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:32.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:32.289252+0000) 2022-09-10T09:09:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:32 smithi110 ceph-mon[97633]: pgmap v8970: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:32.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:32.800148+0000) 2022-09-10T09:09:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:33.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:33.289488+0000) 2022-09-10T09:09:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:33.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:33.800367+0000) 2022-09-10T09:09:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:34 smithi104 ceph-mon[97044]: pgmap v8971: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:34.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:34.289745+0000) 2022-09-10T09:09:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:34 smithi110 ceph-mon[97633]: pgmap v8971: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:34.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:34.800617+0000) 2022-09-10T09:09:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:35.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:35.289999+0000) 2022-09-10T09:09:36.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:09:35.908+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:09:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:35.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:35.800821+0000) 2022-09-10T09:09:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:36 smithi104 ceph-mon[97044]: pgmap v8972: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:36.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:36.290210+0000) 2022-09-10T09:09:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:36 smithi110 ceph-mon[97633]: pgmap v8972: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:36.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:36.801088+0000) 2022-09-10T09:09:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:37.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:37.290442+0000) 2022-09-10T09:09:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:37 smithi104 ceph-mon[97044]: pgmap v8973: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:37 smithi110 ceph-mon[97633]: pgmap v8973: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:37.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:37.801369+0000) 2022-09-10T09:09:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:38.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:38.290658+0000) 2022-09-10T09:09:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:38.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:38.801549+0000) 2022-09-10T09:09:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:39.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:39.290874+0000) 2022-09-10T09:09:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:39.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:39.801758+0000) 2022-09-10T09:09:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:40 smithi104 ceph-mon[97044]: pgmap v8974: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:40.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:40.291081+0000) 2022-09-10T09:09:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:40 smithi110 ceph-mon[97633]: pgmap v8974: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:40.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:40.802037+0000) 2022-09-10T09:09:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:41.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:41.291309+0000) 2022-09-10T09:09:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:41.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:41.802236+0000) 2022-09-10T09:09:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:42 smithi104 ceph-mon[97044]: pgmap v8975: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:42.568 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:42.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:42.291587+0000) 2022-09-10T09:09:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:42 smithi110 ceph-mon[97633]: pgmap v8975: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:42.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:42.802477+0000) 2022-09-10T09:09:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:43.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:43.291850+0000) 2022-09-10T09:09:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:43.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:43.802696+0000) 2022-09-10T09:09:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:44 smithi104 ceph-mon[97044]: pgmap v8976: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:44.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:44.292150+0000) 2022-09-10T09:09:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:44 smithi110 ceph-mon[97633]: pgmap v8976: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:44.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:44.802919+0000) 2022-09-10T09:09:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:09:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:09:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:09:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:09:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:45.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:45.292431+0000) 2022-09-10T09:09:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:09:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:09:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:09:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:09:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:45.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:45.803179+0000) 2022-09-10T09:09:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:46 smithi104 ceph-mon[97044]: pgmap v8977: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:46.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:46.292631+0000) 2022-09-10T09:09:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:46 smithi110 ceph-mon[97633]: pgmap v8977: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:46.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:46.803346+0000) 2022-09-10T09:09:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:47.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:47.292852+0000) 2022-09-10T09:09:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:47 smithi104 ceph-mon[97044]: pgmap v8978: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:47 smithi110 ceph-mon[97633]: pgmap v8978: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:48.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:47.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:47.803555+0000) 2022-09-10T09:09:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:48.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:48.293138+0000) 2022-09-10T09:09:49.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:48.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:48.803778+0000) 2022-09-10T09:09:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:49.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:49.293335+0000) 2022-09-10T09:09:50.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:49.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:49.803986+0000) 2022-09-10T09:09:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:50 smithi104 ceph-mon[97044]: pgmap v8979: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:50.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:50.293582+0000) 2022-09-10T09:09:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:50 smithi110 ceph-mon[97633]: pgmap v8979: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:51.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:50.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:50.804189+0000) 2022-09-10T09:09:51.295 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:09:50.913+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:09:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:51.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:51.293826+0000) 2022-09-10T09:09:52.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:51.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:51.804327+0000) 2022-09-10T09:09:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:52 smithi104 ceph-mon[97044]: pgmap v8980: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:52.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:52.294071+0000) 2022-09-10T09:09:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:52 smithi110 ceph-mon[97633]: pgmap v8980: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:53.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:52.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:52.804537+0000) 2022-09-10T09:09:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:53.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:53.294293+0000) 2022-09-10T09:09:54.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:53.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:53.804731+0000) 2022-09-10T09:09:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:54 smithi104 ceph-mon[97044]: pgmap v8981: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:54.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:54.294520+0000) 2022-09-10T09:09:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:54 smithi110 ceph-mon[97633]: pgmap v8981: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:55.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:54.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:54.804980+0000) 2022-09-10T09:09:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:55.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:55.294716+0000) 2022-09-10T09:09:56.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:55.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:55.805187+0000) 2022-09-10T09:09:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:56 smithi104 ceph-mon[97044]: pgmap v8982: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:56.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:56.294947+0000) 2022-09-10T09:09:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:56 smithi110 ceph-mon[97633]: pgmap v8982: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:09:57.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:56.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:56.805372+0000) 2022-09-10T09:09:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:57.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:57.295211+0000) 2022-09-10T09:09:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:09:57 smithi104 ceph-mon[97044]: pgmap v8983: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:09:57 smithi110 ceph-mon[97633]: pgmap v8983: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:09:58.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:57.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:57.805520+0000) 2022-09-10T09:09:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:58.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:58.295418+0000) 2022-09-10T09:09:59.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:58.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:58.805744+0000) 2022-09-10T09:09:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:09:59.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:59.295624+0000) 2022-09-10T09:10:00.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:09:59.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:09:59.805990+0000) 2022-09-10T09:10:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:00 smithi104 ceph-mon[97044]: pgmap v8984: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:10:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:00.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:00.295806+0000) 2022-09-10T09:10:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:00 smithi110 ceph-mon[97633]: pgmap v8984: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:10:01.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:00.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:00.806244+0000) 2022-09-10T09:10:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:01.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:01.296002+0000) 2022-09-10T09:10:02.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:01.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:01.806462+0000) 2022-09-10T09:10:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:02 smithi104 ceph-mon[97044]: pgmap v8985: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:02.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:02.296253+0000) 2022-09-10T09:10:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:02 smithi110 ceph-mon[97633]: pgmap v8985: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:03.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:02.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:02.806626+0000) 2022-09-10T09:10:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:03.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:03.296499+0000) 2022-09-10T09:10:04.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:03.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:03.806794+0000) 2022-09-10T09:10:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:04 smithi104 ceph-mon[97044]: pgmap v8986: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:04.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:04.296653+0000) 2022-09-10T09:10:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:04 smithi110 ceph-mon[97633]: pgmap v8986: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:05.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:04.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:04.806979+0000) 2022-09-10T09:10:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:05.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:05.296863+0000) 2022-09-10T09:10:06.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:05.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:05.807199+0000) 2022-09-10T09:10:06.298 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:10:05.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:10:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:06 smithi104 ceph-mon[97044]: pgmap v8987: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:10:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:06.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:06.297127+0000) 2022-09-10T09:10:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:06 smithi110 ceph-mon[97633]: pgmap v8987: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:10:07.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:06.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:06.807450+0000) 2022-09-10T09:10:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:07.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:07.297357+0000) 2022-09-10T09:10:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:07 smithi104 ceph-mon[97044]: pgmap v8988: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:10:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:07 smithi110 ceph-mon[97633]: pgmap v8988: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:10:08.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:07.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:07.807616+0000) 2022-09-10T09:10:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:08.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:08.297612+0000) 2022-09-10T09:10:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:08 smithi104 ceph-mon[97044]: osdmap e642: 12 total, 12 up, 12 in 2022-09-10T09:10:08.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:08 smithi110 ceph-mon[97633]: osdmap e642: 12 total, 12 up, 12 in 2022-09-10T09:10:09.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:08.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:08.807801+0000) 2022-09-10T09:10:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:09.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:09.297840+0000) 2022-09-10T09:10:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:09 smithi104 ceph-mon[97044]: pgmap v8990: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:10:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:09 smithi110 ceph-mon[97633]: pgmap v8990: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:10:10.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:09.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:09.808027+0000) 2022-09-10T09:10:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:10.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:10.298091+0000) 2022-09-10T09:10:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:10.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:10.808302+0000) 2022-09-10T09:10:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:11.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:11.298356+0000) 2022-09-10T09:10:12.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:11.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:11.808579+0000) 2022-09-10T09:10:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:12 smithi104 ceph-mon[97044]: pgmap v8991: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:10:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:12 smithi104 ceph-mon[97044]: osdmap e643: 12 total, 12 up, 12 in 2022-09-10T09:10:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:12.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:12.298559+0000) 2022-09-10T09:10:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:12 smithi110 ceph-mon[97633]: pgmap v8991: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:10:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:12 smithi110 ceph-mon[97633]: osdmap e643: 12 total, 12 up, 12 in 2022-09-10T09:10:13.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:12.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:12.808794+0000) 2022-09-10T09:10:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:13.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:13.298752+0000) 2022-09-10T09:10:14.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:13.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:13.809050+0000) 2022-09-10T09:10:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:14 smithi104 ceph-mon[97044]: pgmap v8993: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:10:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:14.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:14.298922+0000) 2022-09-10T09:10:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:14 smithi110 ceph-mon[97633]: pgmap v8993: 129 pgs: 129 active+clean; 8.7 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:10:15.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:14.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:14.809296+0000) 2022-09-10T09:10:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:15.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:15.299129+0000) 2022-09-10T09:10:16.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:15.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:15.809503+0000) 2022-09-10T09:10:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:16 smithi104 ceph-mon[97044]: pgmap v8994: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:10:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:16.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:16.299272+0000) 2022-09-10T09:10:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:16 smithi110 ceph-mon[97633]: pgmap v8994: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:10:17.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:16.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:16.809753+0000) 2022-09-10T09:10:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:17.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:17.299460+0000) 2022-09-10T09:10:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:17 smithi104 ceph-mon[97044]: pgmap v8995: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:10:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:17 smithi110 ceph-mon[97633]: pgmap v8995: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:10:18.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:17.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:17.810002+0000) 2022-09-10T09:10:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:18.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:18.299688+0000) 2022-09-10T09:10:19.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:18.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:18.810241+0000) 2022-09-10T09:10:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:19.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:19.299892+0000) 2022-09-10T09:10:20.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:19.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:19.810507+0000) 2022-09-10T09:10:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:20 smithi104 ceph-mon[97044]: pgmap v8996: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:10:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:20.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:20.300119+0000) 2022-09-10T09:10:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:20 smithi110 ceph-mon[97633]: pgmap v8996: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:10:21.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:20.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:20.810693+0000) 2022-09-10T09:10:21.302 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:10:20.940+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:10:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:21.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:21.300383+0000) 2022-09-10T09:10:22.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:21.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:21.810920+0000) 2022-09-10T09:10:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:22 smithi104 ceph-mon[97044]: pgmap v8997: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:10:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:22.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:22.300636+0000) 2022-09-10T09:10:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:22 smithi110 ceph-mon[97633]: pgmap v8997: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:10:23.221 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:22.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:22.811190+0000) 2022-09-10T09:10:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:23.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:23.300871+0000) 2022-09-10T09:10:24.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:23.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:23.811437+0000) 2022-09-10T09:10:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:24 smithi104 ceph-mon[97044]: pgmap v8998: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:10:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:24.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:24.301077+0000) 2022-09-10T09:10:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:24 smithi110 ceph-mon[97633]: pgmap v8998: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:10:25.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:24.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:24.811576+0000) 2022-09-10T09:10:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:25 smithi104 ceph-mon[97044]: pgmap v8999: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:10:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:25.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:25.301271+0000) 2022-09-10T09:10:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:25 smithi110 ceph-mon[97633]: pgmap v8999: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:10:26.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:25.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:25.811813+0000) 2022-09-10T09:10:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:26.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:26.301476+0000) 2022-09-10T09:10:27.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:26.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:26.811995+0000) 2022-09-10T09:10:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:27.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:27.301730+0000) 2022-09-10T09:10:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:27 smithi104 ceph-mon[97044]: pgmap v9000: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:27 smithi110 ceph-mon[97633]: pgmap v9000: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:27.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:27.812237+0000) 2022-09-10T09:10:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:28.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:28.301932+0000) 2022-09-10T09:10:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:28.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:28.812402+0000) 2022-09-10T09:10:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:29.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:29.302128+0000) 2022-09-10T09:10:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:29.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:29.812563+0000) 2022-09-10T09:10:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:30 smithi104 ceph-mon[97044]: pgmap v9001: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:30.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:30.302314+0000) 2022-09-10T09:10:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:30 smithi110 ceph-mon[97633]: pgmap v9001: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:30.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:30.812811+0000) 2022-09-10T09:10:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:31.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:31.302554+0000) 2022-09-10T09:10:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:31.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:31.813049+0000) 2022-09-10T09:10:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:32 smithi104 ceph-mon[97044]: pgmap v9002: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:32.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:32.302794+0000) 2022-09-10T09:10:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:32 smithi110 ceph-mon[97633]: pgmap v9002: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:32.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:32.813241+0000) 2022-09-10T09:10:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:33 smithi104 ceph-mon[97044]: pgmap v9003: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:33.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:33.303070+0000) 2022-09-10T09:10:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:33 smithi110 ceph-mon[97633]: pgmap v9003: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:33.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:33.813388+0000) 2022-09-10T09:10:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:34.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:34.303248+0000) 2022-09-10T09:10:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:34.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:34.813578+0000) 2022-09-10T09:10:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:35.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:35.303487+0000) 2022-09-10T09:10:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:35.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:35.813751+0000) 2022-09-10T09:10:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:10:35.955+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:10:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:36 smithi104 ceph-mon[97044]: pgmap v9004: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:10:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:36 smithi110 ceph-mon[97633]: pgmap v9004: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:10:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:36.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:36.303737+0000) 2022-09-10T09:10:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:36.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:36.813984+0000) 2022-09-10T09:10:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:37 smithi104 ceph-mon[97044]: pgmap v9005: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:10:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:37.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:37.303896+0000) 2022-09-10T09:10:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:37 smithi110 ceph-mon[97633]: pgmap v9005: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:10:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:37.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:37.814171+0000) 2022-09-10T09:10:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:38.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:38.304065+0000) 2022-09-10T09:10:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:38.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:38.814347+0000) 2022-09-10T09:10:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:39.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:39.304171+0000) 2022-09-10T09:10:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:39.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:39.814580+0000) 2022-09-10T09:10:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:40 smithi104 ceph-mon[97044]: pgmap v9006: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:10:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:40 smithi110 ceph-mon[97633]: pgmap v9006: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:10:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:40.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:40.304312+0000) 2022-09-10T09:10:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:40.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:40.814739+0000) 2022-09-10T09:10:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:41.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:41.304447+0000) 2022-09-10T09:10:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:41.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:41.814905+0000) 2022-09-10T09:10:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:42 smithi104 ceph-mon[97044]: pgmap v9007: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:10:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:42 smithi110 ceph-mon[97633]: pgmap v9007: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:10:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:42.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:42.304665+0000) 2022-09-10T09:10:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:42.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:42.815178+0000) 2022-09-10T09:10:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:43 smithi104 ceph-mon[97044]: pgmap v9008: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:10:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:43 smithi110 ceph-mon[97633]: pgmap v9008: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:10:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:43.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:43.304878+0000) 2022-09-10T09:10:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:43.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:43.815428+0000) 2022-09-10T09:10:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:44.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:44.305062+0000) 2022-09-10T09:10:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:10:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:10:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:10:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:10:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:10:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:10:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:10:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:10:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:44.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:44.815587+0000) 2022-09-10T09:10:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:45.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:45.305263+0000) 2022-09-10T09:10:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:45 smithi110 ceph-mon[97633]: pgmap v9009: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:10:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:45 smithi104 ceph-mon[97044]: pgmap v9009: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:10:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:45.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:45.815795+0000) 2022-09-10T09:10:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:46.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:46.305493+0000) 2022-09-10T09:10:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:46.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:46.816049+0000) 2022-09-10T09:10:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:47 smithi104 ceph-mon[97044]: pgmap v9010: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:47.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:47.305669+0000) 2022-09-10T09:10:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:47 smithi110 ceph-mon[97633]: pgmap v9010: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:47.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:47.816258+0000) 2022-09-10T09:10:48.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:48.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:48.305811+0000) 2022-09-10T09:10:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:48.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:48.816465+0000) 2022-09-10T09:10:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:49.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:49.306037+0000) 2022-09-10T09:10:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:49.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:49.816652+0000) 2022-09-10T09:10:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:50 smithi104 ceph-mon[97044]: pgmap v9011: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:50 smithi110 ceph-mon[97633]: pgmap v9011: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:50.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:50.306266+0000) 2022-09-10T09:10:51.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:10:50.970+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:10:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:50.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:50.816893+0000) 2022-09-10T09:10:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:51 smithi104 ceph-mon[97044]: pgmap v9012: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:51 smithi110 ceph-mon[97633]: pgmap v9012: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:51.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:51.306486+0000) 2022-09-10T09:10:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:51.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:51.817215+0000) 2022-09-10T09:10:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:52.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:52.306720+0000) 2022-09-10T09:10:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:52.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:52.817437+0000) 2022-09-10T09:10:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:53.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:53.306978+0000) 2022-09-10T09:10:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:53.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:53.817576+0000) 2022-09-10T09:10:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:54 smithi104 ceph-mon[97044]: pgmap v9013: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:54 smithi110 ceph-mon[97633]: pgmap v9013: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:54.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:54.307242+0000) 2022-09-10T09:10:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:54.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:54.817793+0000) 2022-09-10T09:10:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:55 smithi104 ceph-mon[97044]: pgmap v9014: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:55 smithi110 ceph-mon[97633]: pgmap v9014: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:10:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:55.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:55.307479+0000) 2022-09-10T09:10:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:55.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:55.818027+0000) 2022-09-10T09:10:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:56.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:56.307645+0000) 2022-09-10T09:10:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:56.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:56.818233+0000) 2022-09-10T09:10:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:57 smithi104 ceph-mon[97044]: pgmap v9015: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:57.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:57.307807+0000) 2022-09-10T09:10:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:57 smithi110 ceph-mon[97633]: pgmap v9015: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:57.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:57.818470+0000) 2022-09-10T09:10:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:58.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:58.308042+0000) 2022-09-10T09:10:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:58.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:58.818682+0000) 2022-09-10T09:10:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:10:59 smithi104 ceph-mon[97044]: pgmap v9016: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:10:59 smithi110 ceph-mon[97633]: pgmap v9016: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:10:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:10:59.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:59.308219+0000) 2022-09-10T09:11:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:10:59.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:10:59.818915+0000) 2022-09-10T09:11:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:00.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:00.308416+0000) 2022-09-10T09:11:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:00.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:00.819149+0000) 2022-09-10T09:11:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:01 smithi104 ceph-mon[97044]: pgmap v9017: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:01 smithi110 ceph-mon[97633]: pgmap v9017: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:01.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:01.308572+0000) 2022-09-10T09:11:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:01.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:01.819348+0000) 2022-09-10T09:11:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:02.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:02.308782+0000) 2022-09-10T09:11:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:02.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:02.819545+0000) 2022-09-10T09:11:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:03 smithi104 ceph-mon[97044]: pgmap v9018: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:03 smithi110 ceph-mon[97633]: pgmap v9018: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:03.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:03.309037+0000) 2022-09-10T09:11:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:03.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:03.819790+0000) 2022-09-10T09:11:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:04.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:04.309222+0000) 2022-09-10T09:11:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:04.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:04.820025+0000) 2022-09-10T09:11:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:05 smithi104 ceph-mon[97044]: pgmap v9019: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:11:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:05 smithi110 ceph-mon[97633]: pgmap v9019: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:11:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:05.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:05.309414+0000) 2022-09-10T09:11:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:05.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:05.820252+0000) 2022-09-10T09:11:06.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:11:05.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:11:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:06.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:06.309580+0000) 2022-09-10T09:11:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:06.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:06.820506+0000) 2022-09-10T09:11:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:07 smithi104 ceph-mon[97044]: pgmap v9020: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:11:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:07.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:07.309752+0000) 2022-09-10T09:11:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:07 smithi110 ceph-mon[97633]: pgmap v9020: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:11:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:07.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:07.820686+0000) 2022-09-10T09:11:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:08 smithi104 ceph-mon[97044]: osdmap e644: 12 total, 12 up, 12 in 2022-09-10T09:11:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:08.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:08.309988+0000) 2022-09-10T09:11:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:08 smithi110 ceph-mon[97633]: osdmap e644: 12 total, 12 up, 12 in 2022-09-10T09:11:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:08.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:08.820905+0000) 2022-09-10T09:11:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:09 smithi104 ceph-mon[97044]: pgmap v9022: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:11:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:09.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:09.310256+0000) 2022-09-10T09:11:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:09 smithi110 ceph-mon[97633]: pgmap v9022: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:11:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:09.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:09.821164+0000) 2022-09-10T09:11:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:10.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:10.310442+0000) 2022-09-10T09:11:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:10.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:10.821333+0000) 2022-09-10T09:11:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:11 smithi104 ceph-mon[97044]: pgmap v9023: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:11:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:11 smithi110 ceph-mon[97633]: pgmap v9023: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:11:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:11.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:11.310658+0000) 2022-09-10T09:11:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:11.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:11.821576+0000) 2022-09-10T09:11:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:12 smithi104 ceph-mon[97044]: osdmap e645: 12 total, 12 up, 12 in 2022-09-10T09:11:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:12.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:12.310868+0000) 2022-09-10T09:11:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:12 smithi110 ceph-mon[97633]: osdmap e645: 12 total, 12 up, 12 in 2022-09-10T09:11:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:12.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:12.821878+0000) 2022-09-10T09:11:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:13 smithi104 ceph-mon[97044]: pgmap v9025: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:11:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:13.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:13.311115+0000) 2022-09-10T09:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:13 smithi110 ceph-mon[97633]: pgmap v9025: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:11:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:13.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:13.822062+0000) 2022-09-10T09:11:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:14.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:14.311336+0000) 2022-09-10T09:11:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:14.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:14.822266+0000) 2022-09-10T09:11:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:15 smithi104 ceph-mon[97044]: pgmap v9026: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:11:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:15 smithi110 ceph-mon[97633]: pgmap v9026: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:11:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:15.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:15.311554+0000) 2022-09-10T09:11:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:15.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:15.822452+0000) 2022-09-10T09:11:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:16.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:16.311779+0000) 2022-09-10T09:11:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:16.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:16.822685+0000) 2022-09-10T09:11:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:17 smithi104 ceph-mon[97044]: pgmap v9027: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:11:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:17.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:17.311998+0000) 2022-09-10T09:11:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:17 smithi110 ceph-mon[97633]: pgmap v9027: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:11:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:17.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:17.822887+0000) 2022-09-10T09:11:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:18.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:18.312223+0000) 2022-09-10T09:11:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:18.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:18.823062+0000) 2022-09-10T09:11:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:19 smithi104 ceph-mon[97044]: pgmap v9028: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:11:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:19 smithi110 ceph-mon[97633]: pgmap v9028: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:11:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:19.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:19.312461+0000) 2022-09-10T09:11:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:19.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:19.823232+0000) 2022-09-10T09:11:20.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:20.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:20.312661+0000) 2022-09-10T09:11:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:20.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:20.823371+0000) 2022-09-10T09:11:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:11:20.985+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:11:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:21 smithi104 ceph-mon[97044]: pgmap v9029: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:11:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:21 smithi110 ceph-mon[97633]: pgmap v9029: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:11:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:21.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:21.312921+0000) 2022-09-10T09:11:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:21.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:21.823628+0000) 2022-09-10T09:11:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:22.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:22.313199+0000) 2022-09-10T09:11:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:22.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:22.823937+0000) 2022-09-10T09:11:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:23 smithi104 ceph-mon[97044]: pgmap v9030: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:11:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:23 smithi110 ceph-mon[97633]: pgmap v9030: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:11:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:23.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:23.313385+0000) 2022-09-10T09:11:24.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:23.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:23.824115+0000) 2022-09-10T09:11:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:24.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:24.313576+0000) 2022-09-10T09:11:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:24.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:24.824319+0000) 2022-09-10T09:11:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:25 smithi104 ceph-mon[97044]: pgmap v9031: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:11:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:25 smithi110 ceph-mon[97633]: pgmap v9031: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:11:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:25.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:25.313708+0000) 2022-09-10T09:11:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:25.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:25.824513+0000) 2022-09-10T09:11:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:26.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:26.313938+0000) 2022-09-10T09:11:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:26.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:26.824696+0000) 2022-09-10T09:11:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:27 smithi104 ceph-mon[97044]: pgmap v9032: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:27.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:27.314204+0000) 2022-09-10T09:11:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:27 smithi110 ceph-mon[97633]: pgmap v9032: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:27.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:27.824836+0000) 2022-09-10T09:11:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:28.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:28.314405+0000) 2022-09-10T09:11:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:28.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:28.824993+0000) 2022-09-10T09:11:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:29 smithi104 ceph-mon[97044]: pgmap v9033: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:29 smithi110 ceph-mon[97633]: pgmap v9033: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:29.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:29.314554+0000) 2022-09-10T09:11:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:29.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:29.825188+0000) 2022-09-10T09:11:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:30.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:30.314785+0000) 2022-09-10T09:11:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:30.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:30.825333+0000) 2022-09-10T09:11:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:31 smithi104 ceph-mon[97044]: pgmap v9034: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:31 smithi110 ceph-mon[97633]: pgmap v9034: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:31.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:31.314977+0000) 2022-09-10T09:11:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:31.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:31.825494+0000) 2022-09-10T09:11:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:32.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:32.315193+0000) 2022-09-10T09:11:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:32.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:32.825728+0000) 2022-09-10T09:11:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:33 smithi104 ceph-mon[97044]: pgmap v9035: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:33 smithi110 ceph-mon[97633]: pgmap v9035: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:33.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:33.315345+0000) 2022-09-10T09:11:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:33.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:33.825893+0000) 2022-09-10T09:11:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:34.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:34.315513+0000) 2022-09-10T09:11:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:34.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:34.826164+0000) 2022-09-10T09:11:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:35 smithi104 ceph-mon[97044]: pgmap v9036: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:35 smithi110 ceph-mon[97633]: pgmap v9036: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:35.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:35.315721+0000) 2022-09-10T09:11:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:35.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:35.826362+0000) 2022-09-10T09:11:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:11:35.989+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:11:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:36.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:36.315969+0000) 2022-09-10T09:11:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:36.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:36.826520+0000) 2022-09-10T09:11:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:37 smithi104 ceph-mon[97044]: pgmap v9037: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:37.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:37.316212+0000) 2022-09-10T09:11:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:37 smithi110 ceph-mon[97633]: pgmap v9037: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:37.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:37.826717+0000) 2022-09-10T09:11:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:38.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:38.316433+0000) 2022-09-10T09:11:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:38.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:38.826915+0000) 2022-09-10T09:11:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:39 smithi104 ceph-mon[97044]: pgmap v9038: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:39 smithi110 ceph-mon[97633]: pgmap v9038: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:39.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:39.316611+0000) 2022-09-10T09:11:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:39.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:39.827175+0000) 2022-09-10T09:11:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:40.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:40.316786+0000) 2022-09-10T09:11:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:40.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:40.827336+0000) 2022-09-10T09:11:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:41 smithi104 ceph-mon[97044]: pgmap v9039: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:41 smithi110 ceph-mon[97633]: pgmap v9039: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:41.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:41.317028+0000) 2022-09-10T09:11:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:41.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:41.827560+0000) 2022-09-10T09:11:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:42.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:42.317308+0000) 2022-09-10T09:11:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:42.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:42.827721+0000) 2022-09-10T09:11:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:43 smithi104 ceph-mon[97044]: pgmap v9040: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:43 smithi110 ceph-mon[97633]: pgmap v9040: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:43.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:43.317531+0000) 2022-09-10T09:11:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:43.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:43.827945+0000) 2022-09-10T09:11:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:44.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:44.317752+0000) 2022-09-10T09:11:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:11:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:11:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:11:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:11:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:11:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:11:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:11:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:11:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:44.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:44.828148+0000) 2022-09-10T09:11:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:45.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:45.318010+0000) 2022-09-10T09:11:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:45 smithi110 ceph-mon[97633]: pgmap v9041: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:45 smithi104 ceph-mon[97044]: pgmap v9041: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:45.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:45.828382+0000) 2022-09-10T09:11:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:46.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:46.318254+0000) 2022-09-10T09:11:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:46.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:46.828689+0000) 2022-09-10T09:11:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:47 smithi104 ceph-mon[97044]: pgmap v9042: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:47.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:47.318456+0000) 2022-09-10T09:11:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:47 smithi110 ceph-mon[97633]: pgmap v9042: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:47.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:47.828889+0000) 2022-09-10T09:11:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:48.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:48.318701+0000) 2022-09-10T09:11:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:48.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:48.829051+0000) 2022-09-10T09:11:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:49 smithi104 ceph-mon[97044]: pgmap v9043: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:49 smithi110 ceph-mon[97633]: pgmap v9043: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:49.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:49.318950+0000) 2022-09-10T09:11:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:49.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:49.829287+0000) 2022-09-10T09:11:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:50.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:50.319158+0000) 2022-09-10T09:11:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:50.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:50.829534+0000) 2022-09-10T09:11:51.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:11:50.997+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:11:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:51 smithi104 ceph-mon[97044]: pgmap v9044: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:51 smithi110 ceph-mon[97633]: pgmap v9044: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:51.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:51.319370+0000) 2022-09-10T09:11:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:51.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:51.829782+0000) 2022-09-10T09:11:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:52.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:52.319606+0000) 2022-09-10T09:11:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:52.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:52.830031+0000) 2022-09-10T09:11:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:53 smithi104 ceph-mon[97044]: pgmap v9045: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:53 smithi110 ceph-mon[97633]: pgmap v9045: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:53.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:53.319855+0000) 2022-09-10T09:11:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:53.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:53.830203+0000) 2022-09-10T09:11:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:54.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:54.320071+0000) 2022-09-10T09:11:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:54.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:54.830431+0000) 2022-09-10T09:11:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:55 smithi104 ceph-mon[97044]: pgmap v9046: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:55 smithi110 ceph-mon[97633]: pgmap v9046: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:11:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:55.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:55.320292+0000) 2022-09-10T09:11:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:55.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:55.830714+0000) 2022-09-10T09:11:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:56.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:56.320502+0000) 2022-09-10T09:11:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:56.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:56.830975+0000) 2022-09-10T09:11:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:57 smithi104 ceph-mon[97044]: pgmap v9047: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:57.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:57.320760+0000) 2022-09-10T09:11:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:57 smithi110 ceph-mon[97633]: pgmap v9047: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:57.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:57.831224+0000) 2022-09-10T09:11:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:58.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:58.320990+0000) 2022-09-10T09:11:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:58.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:58.831476+0000) 2022-09-10T09:11:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:11:59 smithi104 ceph-mon[97044]: pgmap v9048: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:11:59 smithi110 ceph-mon[97633]: pgmap v9048: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:11:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:11:59.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:59.321230+0000) 2022-09-10T09:12:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:11:59.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:11:59.831723+0000) 2022-09-10T09:12:00.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:00.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:00.321490+0000) 2022-09-10T09:12:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:00.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:00.831902+0000) 2022-09-10T09:12:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:01 smithi104 ceph-mon[97044]: pgmap v9049: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:01 smithi110 ceph-mon[97633]: pgmap v9049: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:01.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:01.321681+0000) 2022-09-10T09:12:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:01.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:01.832190+0000) 2022-09-10T09:12:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:02.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:02.321907+0000) 2022-09-10T09:12:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:02.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:02.832383+0000) 2022-09-10T09:12:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:03 smithi104 ceph-mon[97044]: pgmap v9050: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:03 smithi110 ceph-mon[97633]: pgmap v9050: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:03.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:03.322141+0000) 2022-09-10T09:12:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:03.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:03.832638+0000) 2022-09-10T09:12:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:04.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:04.322389+0000) 2022-09-10T09:12:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:04.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:04.832827+0000) 2022-09-10T09:12:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:05 smithi104 ceph-mon[97044]: pgmap v9051: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:12:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:05 smithi110 ceph-mon[97633]: pgmap v9051: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:12:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:05.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:05.322658+0000) 2022-09-10T09:12:06.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:12:06.009+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:12:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:05.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:05.833064+0000) 2022-09-10T09:12:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:06.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:06.322908+0000) 2022-09-10T09:12:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:06.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:06.833277+0000) 2022-09-10T09:12:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:07 smithi104 ceph-mon[97044]: pgmap v9052: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:12:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:07.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:07.323195+0000) 2022-09-10T09:12:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:07 smithi110 ceph-mon[97633]: pgmap v9052: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:12:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:07.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:07.833586+0000) 2022-09-10T09:12:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:08 smithi104 ceph-mon[97044]: osdmap e646: 12 total, 12 up, 12 in 2022-09-10T09:12:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:08.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:08.323408+0000) 2022-09-10T09:12:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:08 smithi110 ceph-mon[97633]: osdmap e646: 12 total, 12 up, 12 in 2022-09-10T09:12:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:08.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:08.833734+0000) 2022-09-10T09:12:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:09 smithi104 ceph-mon[97044]: pgmap v9054: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:12:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:09.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:09.323643+0000) 2022-09-10T09:12:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:09 smithi110 ceph-mon[97633]: pgmap v9054: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:12:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:09.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:09.833995+0000) 2022-09-10T09:12:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:10.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:10.323775+0000) 2022-09-10T09:12:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:10.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:10.834226+0000) 2022-09-10T09:12:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:11 smithi104 ceph-mon[97044]: pgmap v9055: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:12:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:11 smithi110 ceph-mon[97633]: pgmap v9055: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:12:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:11.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:11.323965+0000) 2022-09-10T09:12:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:11.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:11.834440+0000) 2022-09-10T09:12:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:12 smithi104 ceph-mon[97044]: osdmap e647: 12 total, 12 up, 12 in 2022-09-10T09:12:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:12.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:12.324220+0000) 2022-09-10T09:12:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:12 smithi110 ceph-mon[97633]: osdmap e647: 12 total, 12 up, 12 in 2022-09-10T09:12:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:12.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:12.834666+0000) 2022-09-10T09:12:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:13 smithi104 ceph-mon[97044]: pgmap v9057: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:12:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:13.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:13.324432+0000) 2022-09-10T09:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:13 smithi110 ceph-mon[97633]: pgmap v9057: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:12:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:13.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:13.834921+0000) 2022-09-10T09:12:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:14.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:14.324570+0000) 2022-09-10T09:12:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:14.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:14.835219+0000) 2022-09-10T09:12:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:15 smithi104 ceph-mon[97044]: pgmap v9058: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:12:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:15 smithi110 ceph-mon[97633]: pgmap v9058: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:12:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:15.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:15.324787+0000) 2022-09-10T09:12:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:15.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:15.835467+0000) 2022-09-10T09:12:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:16.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:16.325010+0000) 2022-09-10T09:12:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:16.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:16.835598+0000) 2022-09-10T09:12:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:17 smithi104 ceph-mon[97044]: pgmap v9059: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:12:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:17.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:17.325156+0000) 2022-09-10T09:12:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:17 smithi110 ceph-mon[97633]: pgmap v9059: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:12:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:17.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:17.835808+0000) 2022-09-10T09:12:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:18.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:18.325423+0000) 2022-09-10T09:12:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:18.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:18.836022+0000) 2022-09-10T09:12:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:19 smithi104 ceph-mon[97044]: pgmap v9060: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:12:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:19 smithi110 ceph-mon[97633]: pgmap v9060: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:12:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:19.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:19.325558+0000) 2022-09-10T09:12:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:19.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:19.836267+0000) 2022-09-10T09:12:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:20.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:20.325719+0000) 2022-09-10T09:12:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:20.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:20.836511+0000) 2022-09-10T09:12:21.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:12:21.025+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:12:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:21 smithi104 ceph-mon[97044]: pgmap v9061: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:12:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:21 smithi110 ceph-mon[97633]: pgmap v9061: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:12:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:21.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:21.325912+0000) 2022-09-10T09:12:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:21.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:21.836775+0000) 2022-09-10T09:12:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:22.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:22.326091+0000) 2022-09-10T09:12:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:22.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:22.836979+0000) 2022-09-10T09:12:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:23 smithi104 ceph-mon[97044]: pgmap v9062: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:12:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:23 smithi110 ceph-mon[97633]: pgmap v9062: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:12:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:23.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:23.326318+0000) 2022-09-10T09:12:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:23.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:23.837188+0000) 2022-09-10T09:12:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:24.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:24.326565+0000) 2022-09-10T09:12:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:24.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:24.837386+0000) 2022-09-10T09:12:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:25 smithi104 ceph-mon[97044]: pgmap v9063: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:12:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:25 smithi110 ceph-mon[97633]: pgmap v9063: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:12:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:25.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:25.326805+0000) 2022-09-10T09:12:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:25.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:25.837569+0000) 2022-09-10T09:12:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:26.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:26.327045+0000) 2022-09-10T09:12:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:26.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:26.837823+0000) 2022-09-10T09:12:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:27 smithi104 ceph-mon[97044]: pgmap v9064: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:27.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:27.327314+0000) 2022-09-10T09:12:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:27 smithi110 ceph-mon[97633]: pgmap v9064: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:27.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:27.838033+0000) 2022-09-10T09:12:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:28.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:28.327481+0000) 2022-09-10T09:12:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:28.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:28.838286+0000) 2022-09-10T09:12:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:29 smithi104 ceph-mon[97044]: pgmap v9065: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:29 smithi110 ceph-mon[97633]: pgmap v9065: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:29.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:29.327703+0000) 2022-09-10T09:12:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:29.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:29.838566+0000) 2022-09-10T09:12:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:30.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:30.327977+0000) 2022-09-10T09:12:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:30.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:30.838782+0000) 2022-09-10T09:12:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:31 smithi104 ceph-mon[97044]: pgmap v9066: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:31 smithi110 ceph-mon[97633]: pgmap v9066: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:31.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:31.328236+0000) 2022-09-10T09:12:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:31.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:31.839024+0000) 2022-09-10T09:12:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:32.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:32.328412+0000) 2022-09-10T09:12:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:32.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:32.839316+0000) 2022-09-10T09:12:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:33 smithi104 ceph-mon[97044]: pgmap v9067: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:33 smithi110 ceph-mon[97633]: pgmap v9067: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:33.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:33.328603+0000) 2022-09-10T09:12:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:33.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:33.839521+0000) 2022-09-10T09:12:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:34.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:34.328776+0000) 2022-09-10T09:12:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:34.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:34.839794+0000) 2022-09-10T09:12:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:35 smithi104 ceph-mon[97044]: pgmap v9068: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:35 smithi110 ceph-mon[97633]: pgmap v9068: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:35.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:35.328987+0000) 2022-09-10T09:12:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:12:36.036+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:12:36.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:35.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:35.839958+0000) 2022-09-10T09:12:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:36.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:36.329162+0000) 2022-09-10T09:12:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:36.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:36.840089+0000) 2022-09-10T09:12:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:37 smithi104 ceph-mon[97044]: pgmap v9069: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:37.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:37.329319+0000) 2022-09-10T09:12:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:37 smithi110 ceph-mon[97633]: pgmap v9069: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:37.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:37.840239+0000) 2022-09-10T09:12:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:38.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:38.329528+0000) 2022-09-10T09:12:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:38.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:38.840467+0000) 2022-09-10T09:12:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:39 smithi104 ceph-mon[97044]: pgmap v9070: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:39 smithi110 ceph-mon[97633]: pgmap v9070: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:39.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:39.329676+0000) 2022-09-10T09:12:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:39.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:39.840687+0000) 2022-09-10T09:12:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:40.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:40.329894+0000) 2022-09-10T09:12:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:40.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:40.840874+0000) 2022-09-10T09:12:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:41 smithi104 ceph-mon[97044]: pgmap v9071: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:41 smithi110 ceph-mon[97633]: pgmap v9071: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:41.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:41.330117+0000) 2022-09-10T09:12:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:41.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:41.841160+0000) 2022-09-10T09:12:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:42.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:42.330356+0000) 2022-09-10T09:12:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:42.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:42.841282+0000) 2022-09-10T09:12:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:43 smithi104 ceph-mon[97044]: pgmap v9072: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:43 smithi110 ceph-mon[97633]: pgmap v9072: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:43.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:43.330549+0000) 2022-09-10T09:12:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:43.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:43.841446+0000) 2022-09-10T09:12:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:44.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:44.330817+0000) 2022-09-10T09:12:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:12:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:12:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:12:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:12:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:12:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:12:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:12:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:12:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:44.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:44.841628+0000) 2022-09-10T09:12:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:45.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:45.331055+0000) 2022-09-10T09:12:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:45 smithi110 ceph-mon[97633]: pgmap v9073: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:45 smithi104 ceph-mon[97044]: pgmap v9073: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:45.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:45.841818+0000) 2022-09-10T09:12:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:46.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:46.331340+0000) 2022-09-10T09:12:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:46.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:46.842016+0000) 2022-09-10T09:12:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:47 smithi104 ceph-mon[97044]: pgmap v9074: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:47.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:47.331534+0000) 2022-09-10T09:12:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:47 smithi110 ceph-mon[97633]: pgmap v9074: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:47.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:47.842216+0000) 2022-09-10T09:12:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:48.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:48.331790+0000) 2022-09-10T09:12:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:48.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:48.842389+0000) 2022-09-10T09:12:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:49 smithi104 ceph-mon[97044]: pgmap v9075: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:49 smithi110 ceph-mon[97633]: pgmap v9075: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:49.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:49.331985+0000) 2022-09-10T09:12:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:49.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:49.842607+0000) 2022-09-10T09:12:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:50.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:50.332164+0000) 2022-09-10T09:12:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:50.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:50.842847+0000) 2022-09-10T09:12:51.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:12:51.052+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:12:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:51 smithi104 ceph-mon[97044]: pgmap v9076: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:51 smithi110 ceph-mon[97633]: pgmap v9076: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:51.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:51.332390+0000) 2022-09-10T09:12:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:51.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:51.843049+0000) 2022-09-10T09:12:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:52.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:52.332603+0000) 2022-09-10T09:12:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:52.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:52.843269+0000) 2022-09-10T09:12:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:53 smithi104 ceph-mon[97044]: pgmap v9077: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:53 smithi110 ceph-mon[97633]: pgmap v9077: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:53.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:53.332820+0000) 2022-09-10T09:12:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:53.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:53.843462+0000) 2022-09-10T09:12:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:54.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:54.333014+0000) 2022-09-10T09:12:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:54.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:54.843714+0000) 2022-09-10T09:12:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:55 smithi104 ceph-mon[97044]: pgmap v9078: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:55 smithi110 ceph-mon[97633]: pgmap v9078: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:12:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:55.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:55.333274+0000) 2022-09-10T09:12:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:55.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:55.843846+0000) 2022-09-10T09:12:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:56.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:56.333476+0000) 2022-09-10T09:12:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:56.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:56.844037+0000) 2022-09-10T09:12:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:57 smithi104 ceph-mon[97044]: pgmap v9079: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:57.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:57.333658+0000) 2022-09-10T09:12:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:57 smithi110 ceph-mon[97633]: pgmap v9079: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:57.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:57.844224+0000) 2022-09-10T09:12:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:58.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:58.333887+0000) 2022-09-10T09:12:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:58.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:58.844346+0000) 2022-09-10T09:12:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:12:59 smithi104 ceph-mon[97044]: pgmap v9080: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:12:59 smithi110 ceph-mon[97633]: pgmap v9080: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:12:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:12:59.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:59.334137+0000) 2022-09-10T09:13:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:12:59.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:12:59.844473+0000) 2022-09-10T09:13:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:00.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:00.334277+0000) 2022-09-10T09:13:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:00.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:00.844630+0000) 2022-09-10T09:13:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:01 smithi104 ceph-mon[97044]: pgmap v9081: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:01 smithi110 ceph-mon[97633]: pgmap v9081: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:01.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:01.334483+0000) 2022-09-10T09:13:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:01.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:01.844757+0000) 2022-09-10T09:13:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:02.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:02.334706+0000) 2022-09-10T09:13:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:02.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:02.844941+0000) 2022-09-10T09:13:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:03 smithi104 ceph-mon[97044]: pgmap v9082: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:03 smithi110 ceph-mon[97633]: pgmap v9082: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:03.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:03.334884+0000) 2022-09-10T09:13:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:03.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:03.845168+0000) 2022-09-10T09:13:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:04.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:04.335140+0000) 2022-09-10T09:13:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:04.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:04.845336+0000) 2022-09-10T09:13:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:05 smithi104 ceph-mon[97044]: pgmap v9083: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:13:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:05 smithi110 ceph-mon[97633]: pgmap v9083: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:13:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:05.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:05.335321+0000) 2022-09-10T09:13:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:05.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:05.845551+0000) 2022-09-10T09:13:06.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:13:06.060+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:13:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:06.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:06.335542+0000) 2022-09-10T09:13:07.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:06.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:06.845806+0000) 2022-09-10T09:13:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:07 smithi104 ceph-mon[97044]: pgmap v9084: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:13:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:07.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:07.335765+0000) 2022-09-10T09:13:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:07 smithi110 ceph-mon[97633]: pgmap v9084: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:13:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:07.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:07.846082+0000) 2022-09-10T09:13:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:08 smithi104 ceph-mon[97044]: osdmap e648: 12 total, 12 up, 12 in 2022-09-10T09:13:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:08.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:08.335990+0000) 2022-09-10T09:13:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:08 smithi110 ceph-mon[97633]: osdmap e648: 12 total, 12 up, 12 in 2022-09-10T09:13:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:08.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:08.846346+0000) 2022-09-10T09:13:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:09 smithi104 ceph-mon[97044]: pgmap v9086: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:13:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:09.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:09.336227+0000) 2022-09-10T09:13:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:09 smithi110 ceph-mon[97633]: pgmap v9086: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:13:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:09.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:09.846548+0000) 2022-09-10T09:13:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:10.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:10.336455+0000) 2022-09-10T09:13:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:10.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:10.846783+0000) 2022-09-10T09:13:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:11 smithi104 ceph-mon[97044]: pgmap v9087: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:13:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:11 smithi110 ceph-mon[97633]: pgmap v9087: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:13:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:11.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:11.336598+0000) 2022-09-10T09:13:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:11.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:11.846992+0000) 2022-09-10T09:13:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:12 smithi104 ceph-mon[97044]: osdmap e649: 12 total, 12 up, 12 in 2022-09-10T09:13:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:12.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:12.336836+0000) 2022-09-10T09:13:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:12 smithi110 ceph-mon[97633]: osdmap e649: 12 total, 12 up, 12 in 2022-09-10T09:13:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:12.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:12.847226+0000) 2022-09-10T09:13:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:13 smithi104 ceph-mon[97044]: pgmap v9089: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:13:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:13.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:13.337063+0000) 2022-09-10T09:13:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:13 smithi110 ceph-mon[97633]: pgmap v9089: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:13:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:13.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:13.847411+0000) 2022-09-10T09:13:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:14.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:14.337293+0000) 2022-09-10T09:13:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:14.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:14.847667+0000) 2022-09-10T09:13:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:15 smithi104 ceph-mon[97044]: pgmap v9090: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:13:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:15 smithi110 ceph-mon[97633]: pgmap v9090: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:13:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:15.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:15.337415+0000) 2022-09-10T09:13:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:15.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:15.847919+0000) 2022-09-10T09:13:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:16.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:16.337596+0000) 2022-09-10T09:13:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:16.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:16.848059+0000) 2022-09-10T09:13:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:17 smithi104 ceph-mon[97044]: pgmap v9091: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:13:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:17.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:17.337745+0000) 2022-09-10T09:13:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:17 smithi110 ceph-mon[97633]: pgmap v9091: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:13:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:17.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:17.848275+0000) 2022-09-10T09:13:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:18.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:18.337867+0000) 2022-09-10T09:13:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:18.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:18.848509+0000) 2022-09-10T09:13:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:19 smithi104 ceph-mon[97044]: pgmap v9092: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:13:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:19 smithi110 ceph-mon[97633]: pgmap v9092: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:13:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:19.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:19.337992+0000) 2022-09-10T09:13:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:19.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:19.848742+0000) 2022-09-10T09:13:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:20.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:20.338220+0000) 2022-09-10T09:13:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:20.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:20.848978+0000) 2022-09-10T09:13:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:13:21.068+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:13:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:21 smithi104 ceph-mon[97044]: pgmap v9093: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:13:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:21 smithi110 ceph-mon[97633]: pgmap v9093: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:13:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:21.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:21.338420+0000) 2022-09-10T09:13:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:21.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:21.849229+0000) 2022-09-10T09:13:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:22.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:22.338621+0000) 2022-09-10T09:13:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:22.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:22.849435+0000) 2022-09-10T09:13:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:23 smithi104 ceph-mon[97044]: pgmap v9094: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:13:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:23 smithi110 ceph-mon[97633]: pgmap v9094: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:13:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:23.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:23.338761+0000) 2022-09-10T09:13:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:23.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:23.849680+0000) 2022-09-10T09:13:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:24.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:24.338979+0000) 2022-09-10T09:13:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:24.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:24.849934+0000) 2022-09-10T09:13:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:25 smithi104 ceph-mon[97044]: pgmap v9095: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:13:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:25 smithi110 ceph-mon[97633]: pgmap v9095: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:13:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:25.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:25.339201+0000) 2022-09-10T09:13:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:25.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:25.850204+0000) 2022-09-10T09:13:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:26.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:26.339429+0000) 2022-09-10T09:13:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:26.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:26.850453+0000) 2022-09-10T09:13:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:27 smithi104 ceph-mon[97044]: pgmap v9096: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:27.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:27.339696+0000) 2022-09-10T09:13:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:27 smithi110 ceph-mon[97633]: pgmap v9096: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:27.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:27.850720+0000) 2022-09-10T09:13:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:28.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:28.339941+0000) 2022-09-10T09:13:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:28.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:28.850955+0000) 2022-09-10T09:13:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:29 smithi104 ceph-mon[97044]: pgmap v9097: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:29 smithi110 ceph-mon[97633]: pgmap v9097: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:29.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:29.340151+0000) 2022-09-10T09:13:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:29.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:29.851189+0000) 2022-09-10T09:13:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:30.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:30.340366+0000) 2022-09-10T09:13:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:30.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:30.851449+0000) 2022-09-10T09:13:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:31 smithi110 ceph-mon[97633]: pgmap v9098: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:31 smithi104 ceph-mon[97044]: pgmap v9098: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:31.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:31.340535+0000) 2022-09-10T09:13:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:31.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:31.851657+0000) 2022-09-10T09:13:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:32.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:32.340783+0000) 2022-09-10T09:13:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:32.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:32.851906+0000) 2022-09-10T09:13:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:33 smithi110 ceph-mon[97633]: pgmap v9099: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:33 smithi104 ceph-mon[97044]: pgmap v9099: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:33.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:33.340959+0000) 2022-09-10T09:13:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:33.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:33.852119+0000) 2022-09-10T09:13:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:34.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:34.341220+0000) 2022-09-10T09:13:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:34.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:34.852393+0000) 2022-09-10T09:13:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:35 smithi110 ceph-mon[97633]: pgmap v9100: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:35 smithi104 ceph-mon[97044]: pgmap v9100: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:35.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:35.341417+0000) 2022-09-10T09:13:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:35.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:35.852551+0000) 2022-09-10T09:13:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:13:36.079+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:13:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:36.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:36.341644+0000) 2022-09-10T09:13:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:36.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:36.852809+0000) 2022-09-10T09:13:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:37 smithi104 ceph-mon[97044]: pgmap v9101: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:37.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:37.341872+0000) 2022-09-10T09:13:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:37 smithi110 ceph-mon[97633]: pgmap v9101: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:37.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:37.853065+0000) 2022-09-10T09:13:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:38.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:38.342083+0000) 2022-09-10T09:13:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:38.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:38.853351+0000) 2022-09-10T09:13:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:39 smithi110 ceph-mon[97633]: pgmap v9102: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:39 smithi104 ceph-mon[97044]: pgmap v9102: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:39.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:39.342356+0000) 2022-09-10T09:13:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:39.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:39.853611+0000) 2022-09-10T09:13:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:40.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:40.342523+0000) 2022-09-10T09:13:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:40.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:40.853865+0000) 2022-09-10T09:13:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:41 smithi110 ceph-mon[97633]: pgmap v9103: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:41 smithi104 ceph-mon[97044]: pgmap v9103: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:13:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:41.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:41.342698+0000) 2022-09-10T09:13:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:41.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:41.854176+0000) 2022-09-10T09:13:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:42.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:42.342969+0000) 2022-09-10T09:13:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:42.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:42.854437+0000) 2022-09-10T09:13:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:43 smithi110 ceph-mon[97633]: pgmap v9104: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:43 smithi104 ceph-mon[97044]: pgmap v9104: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:43.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:43.343221+0000) 2022-09-10T09:13:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:43.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:43.854696+0000) 2022-09-10T09:13:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:44.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:44.343432+0000) 2022-09-10T09:13:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:13:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:13:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:13:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:13:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:13:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:13:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:13:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:13:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:44.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:44.854889+0000) 2022-09-10T09:13:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:45.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:45.343681+0000) 2022-09-10T09:13:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:45 smithi110 ceph-mon[97633]: pgmap v9105: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:13:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:45 smithi104 ceph-mon[97044]: pgmap v9105: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:13:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:45.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:45.855177+0000) 2022-09-10T09:13:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:46.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:46.343938+0000) 2022-09-10T09:13:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:46.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:46.855442+0000) 2022-09-10T09:13:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:47 smithi104 ceph-mon[97044]: pgmap v9106: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T09:13:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:47.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:47.344058+0000) 2022-09-10T09:13:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:47 smithi110 ceph-mon[97633]: pgmap v9106: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T09:13:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:47.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:47.855645+0000) 2022-09-10T09:13:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:48.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:48.344260+0000) 2022-09-10T09:13:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:48.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:48.855894+0000) 2022-09-10T09:13:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:49 smithi110 ceph-mon[97633]: pgmap v9107: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T09:13:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:49 smithi104 ceph-mon[97044]: pgmap v9107: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T09:13:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:49.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:49.344547+0000) 2022-09-10T09:13:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:49.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:49.856189+0000) 2022-09-10T09:13:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:50.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:50.344770+0000) 2022-09-10T09:13:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:50.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:50.856443+0000) 2022-09-10T09:13:51.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:13:51.088+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:13:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:51 smithi110 ceph-mon[97633]: pgmap v9108: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:13:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:51 smithi104 ceph-mon[97044]: pgmap v9108: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:13:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:51.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:51.345040+0000) 2022-09-10T09:13:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:51.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:51.856672+0000) 2022-09-10T09:13:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:52.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:52.345307+0000) 2022-09-10T09:13:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:52.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:52.856871+0000) 2022-09-10T09:13:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:53 smithi110 ceph-mon[97633]: pgmap v9109: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T09:13:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:53 smithi104 ceph-mon[97044]: pgmap v9109: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T09:13:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:53.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:53.345501+0000) 2022-09-10T09:13:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:53.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:53.857135+0000) 2022-09-10T09:13:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:54.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:54.345658+0000) 2022-09-10T09:13:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:54.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:54.857265+0000) 2022-09-10T09:13:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:55 smithi110 ceph-mon[97633]: pgmap v9110: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:13:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:55 smithi104 ceph-mon[97044]: pgmap v9110: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:13:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:55.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:55.345920+0000) 2022-09-10T09:13:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:55.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:55.857385+0000) 2022-09-10T09:13:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:56.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:56.346154+0000) 2022-09-10T09:13:57.237 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:56.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:56.857624+0000) 2022-09-10T09:13:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:57 smithi104 ceph-mon[97044]: pgmap v9111: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:57.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:57.346350+0000) 2022-09-10T09:13:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:57 smithi110 ceph-mon[97633]: pgmap v9111: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:57.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:57.857883+0000) 2022-09-10T09:13:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:58.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:58.346561+0000) 2022-09-10T09:13:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:58.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:58.858078+0000) 2022-09-10T09:13:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:13:59 smithi110 ceph-mon[97633]: pgmap v9112: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:13:59 smithi104 ceph-mon[97044]: pgmap v9112: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:13:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:13:59.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:59.346726+0000) 2022-09-10T09:14:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:13:59.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:13:59.858328+0000) 2022-09-10T09:14:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:00.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:00.346948+0000) 2022-09-10T09:14:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:00.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:00.858584+0000) 2022-09-10T09:14:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:01 smithi110 ceph-mon[97633]: pgmap v9113: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:01 smithi104 ceph-mon[97044]: pgmap v9113: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:01.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:01.347185+0000) 2022-09-10T09:14:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:01.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:01.858836+0000) 2022-09-10T09:14:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:02.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:02.347425+0000) 2022-09-10T09:14:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:02.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:02.859102+0000) 2022-09-10T09:14:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:03 smithi110 ceph-mon[97633]: pgmap v9114: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:03 smithi104 ceph-mon[97044]: pgmap v9114: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:03.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:03.347673+0000) 2022-09-10T09:14:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:03.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:03.859287+0000) 2022-09-10T09:14:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:04.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:04.347922+0000) 2022-09-10T09:14:05.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:04.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:04.859552+0000) 2022-09-10T09:14:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:05 smithi110 ceph-mon[97633]: pgmap v9115: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:14:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:05 smithi104 ceph-mon[97044]: pgmap v9115: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:14:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:05.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:05.348176+0000) 2022-09-10T09:14:06.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:14:06.105+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:14:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:05.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:05.859772+0000) 2022-09-10T09:14:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:06.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:06.348360+0000) 2022-09-10T09:14:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:06.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:06.859961+0000) 2022-09-10T09:14:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:07 smithi104 ceph-mon[97044]: pgmap v9116: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:14:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:07.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:07.348547+0000) 2022-09-10T09:14:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:07 smithi110 ceph-mon[97633]: pgmap v9116: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:14:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:07.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:07.860181+0000) 2022-09-10T09:14:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:08 smithi104 ceph-mon[97044]: osdmap e650: 12 total, 12 up, 12 in 2022-09-10T09:14:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:08.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:08.348693+0000) 2022-09-10T09:14:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:08 smithi110 ceph-mon[97633]: osdmap e650: 12 total, 12 up, 12 in 2022-09-10T09:14:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:08.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:08.860392+0000) 2022-09-10T09:14:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:09 smithi104 ceph-mon[97044]: pgmap v9118: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:14:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:09.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:09.348861+0000) 2022-09-10T09:14:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:09 smithi110 ceph-mon[97633]: pgmap v9118: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:14:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:09.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:09.860570+0000) 2022-09-10T09:14:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:10.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:10.349061+0000) 2022-09-10T09:14:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:10.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:10.860755+0000) 2022-09-10T09:14:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:11 smithi110 ceph-mon[97633]: pgmap v9119: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:14:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:11 smithi104 ceph-mon[97044]: pgmap v9119: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:14:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:11.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:11.349242+0000) 2022-09-10T09:14:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:11.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:11.860926+0000) 2022-09-10T09:14:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:12 smithi104 ceph-mon[97044]: osdmap e651: 12 total, 12 up, 12 in 2022-09-10T09:14:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:12.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:12.349478+0000) 2022-09-10T09:14:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:12 smithi110 ceph-mon[97633]: osdmap e651: 12 total, 12 up, 12 in 2022-09-10T09:14:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:12.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:12.861162+0000) 2022-09-10T09:14:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:13 smithi104 ceph-mon[97044]: pgmap v9121: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:14:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:13.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:13.349720+0000) 2022-09-10T09:14:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:13 smithi110 ceph-mon[97633]: pgmap v9121: 129 pgs: 129 active+clean; 8.7 MiB data, 868 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:14:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:13.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:13.861414+0000) 2022-09-10T09:14:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:14.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:14.349962+0000) 2022-09-10T09:14:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:14.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:14.861657+0000) 2022-09-10T09:14:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:15 smithi110 ceph-mon[97633]: pgmap v9122: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:14:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:15 smithi104 ceph-mon[97044]: pgmap v9122: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:14:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:15.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:15.350190+0000) 2022-09-10T09:14:16.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:15.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:15.861877+0000) 2022-09-10T09:14:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:16.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:16.350442+0000) 2022-09-10T09:14:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:16.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:16.862112+0000) 2022-09-10T09:14:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:17 smithi104 ceph-mon[97044]: pgmap v9123: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:14:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:17.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:17.350707+0000) 2022-09-10T09:14:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:17 smithi110 ceph-mon[97633]: pgmap v9123: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:14:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:17.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:17.862334+0000) 2022-09-10T09:14:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:18.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:18.350906+0000) 2022-09-10T09:14:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:18.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:18.862550+0000) 2022-09-10T09:14:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:19 smithi110 ceph-mon[97633]: pgmap v9124: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:14:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:19 smithi104 ceph-mon[97044]: pgmap v9124: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:14:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:19.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:19.351204+0000) 2022-09-10T09:14:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:19.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:19.862743+0000) 2022-09-10T09:14:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:20.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:20.351436+0000) 2022-09-10T09:14:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:20.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:20.862937+0000) 2022-09-10T09:14:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:14:21.111+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:14:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:21 smithi110 ceph-mon[97633]: pgmap v9125: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:14:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:21 smithi104 ceph-mon[97044]: pgmap v9125: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:14:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:21.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:21.351635+0000) 2022-09-10T09:14:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:21.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:21.863194+0000) 2022-09-10T09:14:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:22.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:22.351821+0000) 2022-09-10T09:14:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:22.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:22.863430+0000) 2022-09-10T09:14:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:23 smithi110 ceph-mon[97633]: pgmap v9126: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:14:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:23 smithi104 ceph-mon[97044]: pgmap v9126: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:14:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:23.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:23.352043+0000) 2022-09-10T09:14:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:23.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:23.863667+0000) 2022-09-10T09:14:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:24.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:24.352289+0000) 2022-09-10T09:14:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:24.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:24.863893+0000) 2022-09-10T09:14:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:25 smithi110 ceph-mon[97633]: pgmap v9127: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:14:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:25 smithi104 ceph-mon[97044]: pgmap v9127: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:14:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:25.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:25.352533+0000) 2022-09-10T09:14:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:25.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:25.864143+0000) 2022-09-10T09:14:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:26.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:26.352789+0000) 2022-09-10T09:14:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:26.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:26.864335+0000) 2022-09-10T09:14:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:27 smithi104 ceph-mon[97044]: pgmap v9128: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:27.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:27.353039+0000) 2022-09-10T09:14:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:27 smithi110 ceph-mon[97633]: pgmap v9128: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:27.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:27.864543+0000) 2022-09-10T09:14:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:28.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:28.353233+0000) 2022-09-10T09:14:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:28.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:28.864693+0000) 2022-09-10T09:14:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:29 smithi110 ceph-mon[97633]: pgmap v9129: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:29 smithi104 ceph-mon[97044]: pgmap v9129: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:29.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:29.353449+0000) 2022-09-10T09:14:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:29.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:29.864858+0000) 2022-09-10T09:14:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:30.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:30.353649+0000) 2022-09-10T09:14:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:30.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:30.865086+0000) 2022-09-10T09:14:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:31 smithi110 ceph-mon[97633]: pgmap v9130: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:31 smithi104 ceph-mon[97044]: pgmap v9130: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:31.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:31.353854+0000) 2022-09-10T09:14:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:31.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:31.865288+0000) 2022-09-10T09:14:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:32.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:32.354085+0000) 2022-09-10T09:14:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:32.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:32.865510+0000) 2022-09-10T09:14:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:33 smithi110 ceph-mon[97633]: pgmap v9131: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:33 smithi104 ceph-mon[97044]: pgmap v9131: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:33.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:33.354334+0000) 2022-09-10T09:14:34.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:33.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:33.865774+0000) 2022-09-10T09:14:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:34.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:34.354556+0000) 2022-09-10T09:14:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:34.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:34.866004+0000) 2022-09-10T09:14:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:35 smithi110 ceph-mon[97633]: pgmap v9132: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:35 smithi104 ceph-mon[97044]: pgmap v9132: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:35.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:35.354737+0000) 2022-09-10T09:14:36.124 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:35.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:35.866248+0000) 2022-09-10T09:14:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:36.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:36.354992+0000) 2022-09-10T09:14:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:14:36.122+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:14:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:36.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:36.866459+0000) 2022-09-10T09:14:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:37 smithi104 ceph-mon[97044]: pgmap v9133: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:37.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:37.355230+0000) 2022-09-10T09:14:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:37 smithi110 ceph-mon[97633]: pgmap v9133: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:37.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:37.866667+0000) 2022-09-10T09:14:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:38.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:38.355453+0000) 2022-09-10T09:14:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:38.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:38.866822+0000) 2022-09-10T09:14:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:39 smithi110 ceph-mon[97633]: pgmap v9134: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:39 smithi104 ceph-mon[97044]: pgmap v9134: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:39.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:39.355737+0000) 2022-09-10T09:14:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:39.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:39.867075+0000) 2022-09-10T09:14:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:40.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:40.355921+0000) 2022-09-10T09:14:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:40.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:40.867276+0000) 2022-09-10T09:14:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:41 smithi110 ceph-mon[97633]: pgmap v9135: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:41 smithi104 ceph-mon[97044]: pgmap v9135: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:41.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:41.356084+0000) 2022-09-10T09:14:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:41.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:41.867456+0000) 2022-09-10T09:14:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:42.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:42.356315+0000) 2022-09-10T09:14:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:42.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:42.867689+0000) 2022-09-10T09:14:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:43 smithi110 ceph-mon[97633]: pgmap v9136: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:43 smithi104 ceph-mon[97044]: pgmap v9136: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:43.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:43.356483+0000) 2022-09-10T09:14:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:43.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:43.867829+0000) 2022-09-10T09:14:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:44.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:44.356716+0000) 2022-09-10T09:14:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:14:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:14:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:14:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:14:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:14:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:14:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:14:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:14:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:44.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:44.868075+0000) 2022-09-10T09:14:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:45.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:45.356951+0000) 2022-09-10T09:14:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:45 smithi110 ceph-mon[97633]: pgmap v9137: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:45 smithi104 ceph-mon[97044]: pgmap v9137: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:45.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:45.868301+0000) 2022-09-10T09:14:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:46.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:46.357213+0000) 2022-09-10T09:14:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:46.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:46.868494+0000) 2022-09-10T09:14:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:47 smithi104 ceph-mon[97044]: pgmap v9138: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:47.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:47.357383+0000) 2022-09-10T09:14:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:47 smithi110 ceph-mon[97633]: pgmap v9138: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:47.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:47.868701+0000) 2022-09-10T09:14:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:48.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:48.357585+0000) 2022-09-10T09:14:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:48.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:48.868925+0000) 2022-09-10T09:14:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:49 smithi110 ceph-mon[97633]: pgmap v9139: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:49 smithi104 ceph-mon[97044]: pgmap v9139: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:49.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:49.357862+0000) 2022-09-10T09:14:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:49.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:49.869149+0000) 2022-09-10T09:14:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:50.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:50.358087+0000) 2022-09-10T09:14:51.134 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:50.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:50.869291+0000) 2022-09-10T09:14:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:51.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:51.358315+0000) 2022-09-10T09:14:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:14:51.132+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:14:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:51 smithi110 ceph-mon[97633]: pgmap v9140: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:51 smithi104 ceph-mon[97044]: pgmap v9140: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:51.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:51.869500+0000) 2022-09-10T09:14:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:52.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:52.358555+0000) 2022-09-10T09:14:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:52.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:52.869748+0000) 2022-09-10T09:14:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:53 smithi110 ceph-mon[97633]: pgmap v9141: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:53 smithi104 ceph-mon[97044]: pgmap v9141: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:53.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:53.358817+0000) 2022-09-10T09:14:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:53.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:53.870012+0000) 2022-09-10T09:14:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:54.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:54.359089+0000) 2022-09-10T09:14:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:54.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:54.870193+0000) 2022-09-10T09:14:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:55 smithi110 ceph-mon[97633]: pgmap v9142: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:55 smithi104 ceph-mon[97044]: pgmap v9142: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:14:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:55.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:55.359371+0000) 2022-09-10T09:14:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:55.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:55.870380+0000) 2022-09-10T09:14:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:56.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:56.359619+0000) 2022-09-10T09:14:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:56.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:56.870622+0000) 2022-09-10T09:14:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:57 smithi104 ceph-mon[97044]: pgmap v9143: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:57.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:57.359835+0000) 2022-09-10T09:14:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:57 smithi110 ceph-mon[97633]: pgmap v9143: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:58.165 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:57.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:57.870850+0000) 2022-09-10T09:14:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:58.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:58.360077+0000) 2022-09-10T09:14:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:58.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:58.871046+0000) 2022-09-10T09:14:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:14:59 smithi110 ceph-mon[97633]: pgmap v9144: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:14:59 smithi104 ceph-mon[97044]: pgmap v9144: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:14:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:14:59.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:59.360366+0000) 2022-09-10T09:15:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:14:59.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:14:59.871331+0000) 2022-09-10T09:15:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:00.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:00.360583+0000) 2022-09-10T09:15:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:00.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:00.871554+0000) 2022-09-10T09:15:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:01 smithi110 ceph-mon[97633]: pgmap v9145: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:01 smithi104 ceph-mon[97044]: pgmap v9145: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:01.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:01.360759+0000) 2022-09-10T09:15:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:01.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:01.871771+0000) 2022-09-10T09:15:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:02.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:02.361024+0000) 2022-09-10T09:15:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:02.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:02.872033+0000) 2022-09-10T09:15:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:03 smithi110 ceph-mon[97633]: pgmap v9146: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:03 smithi104 ceph-mon[97044]: pgmap v9146: 129 pgs: 129 active+clean; 8.7 MiB data, 869 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:03.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:03.361239+0000) 2022-09-10T09:15:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:03.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:03.872157+0000) 2022-09-10T09:15:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:04.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:04.361475+0000) 2022-09-10T09:15:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:04.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:04.872348+0000) 2022-09-10T09:15:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:05 smithi110 ceph-mon[97633]: pgmap v9147: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:15:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:05 smithi104 ceph-mon[97044]: pgmap v9147: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:15:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:05.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:05.361776+0000) 2022-09-10T09:15:06.137 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:05.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:05.872568+0000) 2022-09-10T09:15:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:06.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:06.362038+0000) 2022-09-10T09:15:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:15:06.135+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:15:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:06.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:06.872846+0000) 2022-09-10T09:15:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:07 smithi104 ceph-mon[97044]: pgmap v9148: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:15:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:07.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:07.362314+0000) 2022-09-10T09:15:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:07 smithi110 ceph-mon[97633]: pgmap v9148: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:15:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:07.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:07.873043+0000) 2022-09-10T09:15:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:08 smithi104 ceph-mon[97044]: osdmap e652: 12 total, 12 up, 12 in 2022-09-10T09:15:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:08.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:08.362557+0000) 2022-09-10T09:15:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:08 smithi110 ceph-mon[97633]: osdmap e652: 12 total, 12 up, 12 in 2022-09-10T09:15:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:08.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:08.873210+0000) 2022-09-10T09:15:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:09 smithi104 ceph-mon[97044]: pgmap v9150: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:15:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:09.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:09.362702+0000) 2022-09-10T09:15:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:09 smithi110 ceph-mon[97633]: pgmap v9150: 129 pgs: 129 active+clean; 8.7 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:15:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:09.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:09.873413+0000) 2022-09-10T09:15:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:10.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:10.362897+0000) 2022-09-10T09:15:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:10.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:10.873598+0000) 2022-09-10T09:15:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:11 smithi110 ceph-mon[97633]: pgmap v9151: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:15:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:11 smithi104 ceph-mon[97044]: pgmap v9151: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:15:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:11.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:11.363091+0000) 2022-09-10T09:15:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:11.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:11.873866+0000) 2022-09-10T09:15:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:12 smithi104 ceph-mon[97044]: osdmap e653: 12 total, 12 up, 12 in 2022-09-10T09:15:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:12.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:12.363368+0000) 2022-09-10T09:15:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:12 smithi110 ceph-mon[97633]: osdmap e653: 12 total, 12 up, 12 in 2022-09-10T09:15:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:12.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:12.874163+0000) 2022-09-10T09:15:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:13 smithi104 ceph-mon[97044]: pgmap v9153: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:15:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:13.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:13.363544+0000) 2022-09-10T09:15:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:13 smithi110 ceph-mon[97633]: pgmap v9153: 129 pgs: 129 active+clean; 8.7 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:15:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:13.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:13.874316+0000) 2022-09-10T09:15:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:14.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:14.363711+0000) 2022-09-10T09:15:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:14.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:14.874507+0000) 2022-09-10T09:15:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:15 smithi110 ceph-mon[97633]: pgmap v9154: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:15:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:15 smithi104 ceph-mon[97044]: pgmap v9154: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:15:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:15.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:15.363973+0000) 2022-09-10T09:15:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:15.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:15.874684+0000) 2022-09-10T09:15:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:16.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:16.364206+0000) 2022-09-10T09:15:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:16.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:16.874885+0000) 2022-09-10T09:15:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:17 smithi104 ceph-mon[97044]: pgmap v9155: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:15:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:17.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:17.364410+0000) 2022-09-10T09:15:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:17 smithi110 ceph-mon[97633]: pgmap v9155: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:15:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:17.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:17.875138+0000) 2022-09-10T09:15:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:18.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:18.364630+0000) 2022-09-10T09:15:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:18.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:18.875277+0000) 2022-09-10T09:15:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:19 smithi110 ceph-mon[97633]: pgmap v9156: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:15:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:19 smithi104 ceph-mon[97044]: pgmap v9156: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:15:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:19.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:19.364815+0000) 2022-09-10T09:15:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:19.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:19.875496+0000) 2022-09-10T09:15:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:20.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:20.365048+0000) 2022-09-10T09:15:21.139 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:20.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:20.875752+0000) 2022-09-10T09:15:21.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:15:21.137+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:15:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:21.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:21.365282+0000) 2022-09-10T09:15:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:21 smithi110 ceph-mon[97633]: pgmap v9157: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:15:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:21 smithi104 ceph-mon[97044]: pgmap v9157: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:15:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:21.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:21.875981+0000) 2022-09-10T09:15:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:22.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:22.365513+0000) 2022-09-10T09:15:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:22.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:22.876219+0000) 2022-09-10T09:15:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:23 smithi110 ceph-mon[97633]: pgmap v9158: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:15:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:23 smithi104 ceph-mon[97044]: pgmap v9158: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T09:15:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:23.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:23.365651+0000) 2022-09-10T09:15:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:23.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:23.876394+0000) 2022-09-10T09:15:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:24.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:24.365875+0000) 2022-09-10T09:15:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:24.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:24.876631+0000) 2022-09-10T09:15:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:25 smithi110 ceph-mon[97633]: pgmap v9159: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:15:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:25 smithi104 ceph-mon[97044]: pgmap v9159: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:15:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:25.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:25.366124+0000) 2022-09-10T09:15:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:25.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:25.876887+0000) 2022-09-10T09:15:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:26.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:26.366370+0000) 2022-09-10T09:15:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:26.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:26.877163+0000) 2022-09-10T09:15:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:27 smithi104 ceph-mon[97044]: pgmap v9160: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:27.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:27.366574+0000) 2022-09-10T09:15:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:27 smithi110 ceph-mon[97633]: pgmap v9160: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:27.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:27.877370+0000) 2022-09-10T09:15:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:28.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:28.366755+0000) 2022-09-10T09:15:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:28.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:28.877598+0000) 2022-09-10T09:15:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:29 smithi110 ceph-mon[97633]: pgmap v9161: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:29 smithi104 ceph-mon[97044]: pgmap v9161: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:29.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:29.366932+0000) 2022-09-10T09:15:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:29.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:29.877842+0000) 2022-09-10T09:15:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:30.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:30.367176+0000) 2022-09-10T09:15:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:30.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:30.878033+0000) 2022-09-10T09:15:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:31 smithi110 ceph-mon[97633]: pgmap v9162: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:31 smithi104 ceph-mon[97044]: pgmap v9162: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:31.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:31.367391+0000) 2022-09-10T09:15:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:31.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:31.878284+0000) 2022-09-10T09:15:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:32.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:32.367672+0000) 2022-09-10T09:15:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:32.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:32.878529+0000) 2022-09-10T09:15:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:33 smithi110 ceph-mon[97633]: pgmap v9163: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:33 smithi104 ceph-mon[97044]: pgmap v9163: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:33.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:33.367918+0000) 2022-09-10T09:15:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:33.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:33.878782+0000) 2022-09-10T09:15:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:34.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:34.368072+0000) 2022-09-10T09:15:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:34.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:34.879013+0000) 2022-09-10T09:15:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:35.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:35.368353+0000) 2022-09-10T09:15:36.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:35.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:35.879241+0000) 2022-09-10T09:15:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:36.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:36.368551+0000) 2022-09-10T09:15:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:15:36.142+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:15:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:36 smithi110 ceph-mon[97633]: pgmap v9164: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:36 smithi104 ceph-mon[97044]: pgmap v9164: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:36.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:36.879467+0000) 2022-09-10T09:15:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:37.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:37.368731+0000) 2022-09-10T09:15:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:37.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:37.879649+0000) 2022-09-10T09:15:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:38 smithi104 ceph-mon[97044]: pgmap v9165: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:38.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:38.368906+0000) 2022-09-10T09:15:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:38 smithi110 ceph-mon[97633]: pgmap v9165: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:38.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:38.879879+0000) 2022-09-10T09:15:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:39.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:39.369070+0000) 2022-09-10T09:15:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:39.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:39.880173+0000) 2022-09-10T09:15:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:40 smithi104 ceph-mon[97044]: pgmap v9166: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:40.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:40.369289+0000) 2022-09-10T09:15:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:40 smithi110 ceph-mon[97633]: pgmap v9166: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:40.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:40.880313+0000) 2022-09-10T09:15:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:41.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:41.369489+0000) 2022-09-10T09:15:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:41.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:41.880548+0000) 2022-09-10T09:15:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:42 smithi104 ceph-mon[97044]: pgmap v9167: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:42.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:42.369786+0000) 2022-09-10T09:15:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:42 smithi110 ceph-mon[97633]: pgmap v9167: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:42.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:42.880768+0000) 2022-09-10T09:15:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:43.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:43.369977+0000) 2022-09-10T09:15:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:43.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:43.880944+0000) 2022-09-10T09:15:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:44 smithi104 ceph-mon[97044]: pgmap v9168: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:44.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:44.370163+0000) 2022-09-10T09:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:44 smithi110 ceph-mon[97633]: pgmap v9168: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:44.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:44.881203+0000) 2022-09-10T09:15:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:15:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:15:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:15:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:15:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:45.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:45.370382+0000) 2022-09-10T09:15:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:15:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:15:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:45.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:45.881428+0000) 2022-09-10T09:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:46 smithi104 ceph-mon[97044]: pgmap v9169: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:46.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:46.370601+0000) 2022-09-10T09:15:46.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:46 smithi110 ceph-mon[97633]: pgmap v9169: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:46.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:46.881704+0000) 2022-09-10T09:15:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:47.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:47.370856+0000) 2022-09-10T09:15:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:47.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:47.881930+0000) 2022-09-10T09:15:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:48 smithi104 ceph-mon[97044]: pgmap v9170: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:48.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:48.371073+0000) 2022-09-10T09:15:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:48 smithi110 ceph-mon[97633]: pgmap v9170: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:48.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:48.882140+0000) 2022-09-10T09:15:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:49.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:49.371192+0000) 2022-09-10T09:15:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:49.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:49.882397+0000) 2022-09-10T09:15:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:50 smithi104 ceph-mon[97044]: pgmap v9171: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:50.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:50.371401+0000) 2022-09-10T09:15:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:50 smithi110 ceph-mon[97633]: pgmap v9171: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:51.151 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:50.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:50.882597+0000) 2022-09-10T09:15:51.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:15:51.150+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:15:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:51.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:51.371631+0000) 2022-09-10T09:15:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:51.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:51.882741+0000) 2022-09-10T09:15:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:52.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:52.371842+0000) 2022-09-10T09:15:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:52 smithi110 ceph-mon[97633]: pgmap v9172: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:52 smithi104 ceph-mon[97044]: pgmap v9172: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:52.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:52.882990+0000) 2022-09-10T09:15:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:53.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:53.372072+0000) 2022-09-10T09:15:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:53.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:53.883235+0000) 2022-09-10T09:15:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:54.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:54.372307+0000) 2022-09-10T09:15:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:54 smithi110 ceph-mon[97633]: pgmap v9173: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:54 smithi104 ceph-mon[97044]: pgmap v9173: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:54.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:54.883407+0000) 2022-09-10T09:15:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:55.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:55.372484+0000) 2022-09-10T09:15:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:55.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:55.883587+0000) 2022-09-10T09:15:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:56.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:56.372680+0000) 2022-09-10T09:15:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:56 smithi110 ceph-mon[97633]: pgmap v9174: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:56 smithi104 ceph-mon[97044]: pgmap v9174: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:15:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:56.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:56.883775+0000) 2022-09-10T09:15:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:57.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:57.372883+0000) 2022-09-10T09:15:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:57.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:57.883933+0000) 2022-09-10T09:15:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:58.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:58.373036+0000) 2022-09-10T09:15:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:15:58 smithi110 ceph-mon[97633]: pgmap v9175: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:15:58 smithi104 ceph-mon[97044]: pgmap v9175: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:15:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:58.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:58.884134+0000) 2022-09-10T09:15:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:15:59.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:59.373266+0000) 2022-09-10T09:16:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:15:59.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:15:59.884292+0000) 2022-09-10T09:16:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:00.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:00.373395+0000) 2022-09-10T09:16:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:00 smithi110 ceph-mon[97633]: pgmap v9176: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:00 smithi104 ceph-mon[97044]: pgmap v9176: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:00.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:00.884450+0000) 2022-09-10T09:16:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:01.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:01.373608+0000) 2022-09-10T09:16:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:01.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:01.884666+0000) 2022-09-10T09:16:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:02.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:02.373860+0000) 2022-09-10T09:16:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:02 smithi110 ceph-mon[97633]: pgmap v9177: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:02 smithi104 ceph-mon[97044]: pgmap v9177: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:02.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:02.884923+0000) 2022-09-10T09:16:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:03.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:03.374136+0000) 2022-09-10T09:16:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:03.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:03.885135+0000) 2022-09-10T09:16:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:04.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:04.374356+0000) 2022-09-10T09:16:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:04 smithi110 ceph-mon[97633]: pgmap v9178: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:04 smithi104 ceph-mon[97044]: pgmap v9178: 129 pgs: 129 active+clean; 8.7 MiB data, 875 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:04.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:04.885304+0000) 2022-09-10T09:16:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:05.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:05.374559+0000) 2022-09-10T09:16:06.166 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:05.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:05.885549+0000) 2022-09-10T09:16:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:06.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:06.374783+0000) 2022-09-10T09:16:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:16:06.164+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:16:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:06 smithi110 ceph-mon[97633]: pgmap v9179: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:16:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:06 smithi104 ceph-mon[97044]: pgmap v9179: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:16:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:06.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:06.885830+0000) 2022-09-10T09:16:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:07.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:07.374995+0000) 2022-09-10T09:16:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:07.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:07.886025+0000) 2022-09-10T09:16:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:08.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:08.375192+0000) 2022-09-10T09:16:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:08 smithi110 ceph-mon[97633]: pgmap v9180: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:16:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:08 smithi110 ceph-mon[97633]: osdmap e654: 12 total, 12 up, 12 in 2022-09-10T09:16:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:08 smithi104 ceph-mon[97044]: pgmap v9180: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:16:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:08 smithi104 ceph-mon[97044]: osdmap e654: 12 total, 12 up, 12 in 2022-09-10T09:16:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:08.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:08.886279+0000) 2022-09-10T09:16:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:09.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:09.375382+0000) 2022-09-10T09:16:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:09.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:09.886498+0000) 2022-09-10T09:16:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:10.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:10.375634+0000) 2022-09-10T09:16:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:10 smithi110 ceph-mon[97633]: pgmap v9182: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:16:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:10 smithi104 ceph-mon[97044]: pgmap v9182: 129 pgs: 129 active+clean; 8.7 MiB data, 876 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:16:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:10.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:10.886718+0000) 2022-09-10T09:16:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:11.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:11.375837+0000) 2022-09-10T09:16:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:11.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:11.886962+0000) 2022-09-10T09:16:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:12.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:12.376036+0000) 2022-09-10T09:16:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:12 smithi110 ceph-mon[97633]: pgmap v9183: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:16:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:12 smithi104 ceph-mon[97044]: pgmap v9183: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:16:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:12.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:12.887172+0000) 2022-09-10T09:16:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:13.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:13.376261+0000) 2022-09-10T09:16:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:13.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:13.887300+0000) 2022-09-10T09:16:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:14.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:14.376479+0000) 2022-09-10T09:16:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:14 smithi110 ceph-mon[97633]: pgmap v9184: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:16:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:14 smithi104 ceph-mon[97044]: pgmap v9184: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:16:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:14.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:14.887475+0000) 2022-09-10T09:16:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:15.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:15.376708+0000) 2022-09-10T09:16:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:15.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:15.887674+0000) 2022-09-10T09:16:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:16.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:16.376865+0000) 2022-09-10T09:16:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:16 smithi110 ceph-mon[97633]: pgmap v9185: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:16:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:16 smithi110 ceph-mon[97633]: osdmap e655: 12 total, 12 up, 12 in 2022-09-10T09:16:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:16 smithi104 ceph-mon[97044]: pgmap v9185: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:16:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:16 smithi104 ceph-mon[97044]: osdmap e655: 12 total, 12 up, 12 in 2022-09-10T09:16:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:16.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:16.887850+0000) 2022-09-10T09:16:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:17.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:17.377143+0000) 2022-09-10T09:16:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:17.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:17.888063+0000) 2022-09-10T09:16:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:18.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:18.377370+0000) 2022-09-10T09:16:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:18 smithi110 ceph-mon[97633]: pgmap v9187: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T09:16:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:18 smithi104 ceph-mon[97044]: pgmap v9187: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T09:16:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:18.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:18.888293+0000) 2022-09-10T09:16:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:19.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:19.377616+0000) 2022-09-10T09:16:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:19.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:19.888529+0000) 2022-09-10T09:16:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:20.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:20.377887+0000) 2022-09-10T09:16:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:20 smithi110 ceph-mon[97633]: pgmap v9188: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:16:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:20 smithi104 ceph-mon[97044]: pgmap v9188: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:16:21.176 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:20.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:20.888760+0000) 2022-09-10T09:16:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:21.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:21.378125+0000) 2022-09-10T09:16:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:16:21.174+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:16:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:21.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:21.888983+0000) 2022-09-10T09:16:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:22.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:22.378348+0000) 2022-09-10T09:16:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:22 smithi110 ceph-mon[97633]: pgmap v9189: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:16:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:22 smithi104 ceph-mon[97044]: pgmap v9189: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:16:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:22.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:22.889152+0000) 2022-09-10T09:16:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:23.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:23.378596+0000) 2022-09-10T09:16:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:23.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:23.889395+0000) 2022-09-10T09:16:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:24.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:24.378834+0000) 2022-09-10T09:16:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:24 smithi110 ceph-mon[97633]: pgmap v9190: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:16:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:24 smithi104 ceph-mon[97044]: pgmap v9190: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:16:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:24.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:24.889562+0000) 2022-09-10T09:16:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:25.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:25.379021+0000) 2022-09-10T09:16:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:25.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:25.889800+0000) 2022-09-10T09:16:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:26.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:26.379286+0000) 2022-09-10T09:16:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:26 smithi110 ceph-mon[97633]: pgmap v9191: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:16:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:26 smithi104 ceph-mon[97044]: pgmap v9191: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:16:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:26.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:26.890079+0000) 2022-09-10T09:16:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:27.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:27.379562+0000) 2022-09-10T09:16:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:27.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:27.890351+0000) 2022-09-10T09:16:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:28.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:28.379749+0000) 2022-09-10T09:16:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:28 smithi110 ceph-mon[97633]: pgmap v9192: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T09:16:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:28 smithi104 ceph-mon[97044]: pgmap v9192: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T09:16:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:28.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:28.890610+0000) 2022-09-10T09:16:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:29.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:29.379962+0000) 2022-09-10T09:16:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:29.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:29.890863+0000) 2022-09-10T09:16:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:30.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:30.380155+0000) 2022-09-10T09:16:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:30 smithi110 ceph-mon[97633]: pgmap v9193: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:16:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:30 smithi104 ceph-mon[97044]: pgmap v9193: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:16:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:30.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:30.891153+0000) 2022-09-10T09:16:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:31.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:31.380394+0000) 2022-09-10T09:16:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:31.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:31.891399+0000) 2022-09-10T09:16:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:32.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:32.380588+0000) 2022-09-10T09:16:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:32 smithi110 ceph-mon[97633]: pgmap v9194: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:32 smithi104 ceph-mon[97044]: pgmap v9194: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:32.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:32.891622+0000) 2022-09-10T09:16:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:33.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:33.380833+0000) 2022-09-10T09:16:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:33.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:33.891786+0000) 2022-09-10T09:16:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:34.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:34.381011+0000) 2022-09-10T09:16:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:34 smithi110 ceph-mon[97633]: pgmap v9195: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:34 smithi104 ceph-mon[97044]: pgmap v9195: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:34.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:34.892017+0000) 2022-09-10T09:16:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:35.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:35.381238+0000) 2022-09-10T09:16:36.186 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:35.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:35.892241+0000) 2022-09-10T09:16:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:36.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:36.381498+0000) 2022-09-10T09:16:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:16:36.185+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:16:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:36 smithi110 ceph-mon[97633]: pgmap v9196: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:36 smithi104 ceph-mon[97044]: pgmap v9196: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:36.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:36.892489+0000) 2022-09-10T09:16:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:37.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:37.381747+0000) 2022-09-10T09:16:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:37.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:37.892754+0000) 2022-09-10T09:16:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:38.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:38.381951+0000) 2022-09-10T09:16:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:38 smithi110 ceph-mon[97633]: pgmap v9197: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:38 smithi104 ceph-mon[97044]: pgmap v9197: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:38.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:38.892943+0000) 2022-09-10T09:16:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:39.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:39.382167+0000) 2022-09-10T09:16:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:39.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:39.893091+0000) 2022-09-10T09:16:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:40.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:40.382398+0000) 2022-09-10T09:16:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:40 smithi110 ceph-mon[97633]: pgmap v9198: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:16:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:40 smithi104 ceph-mon[97044]: pgmap v9198: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:16:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:40.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:40.893301+0000) 2022-09-10T09:16:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:41.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:41.382574+0000) 2022-09-10T09:16:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:41.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:41.893540+0000) 2022-09-10T09:16:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:42.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:42.382841+0000) 2022-09-10T09:16:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:42 smithi110 ceph-mon[97633]: pgmap v9199: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:16:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:42 smithi104 ceph-mon[97044]: pgmap v9199: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:16:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:42.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:42.893712+0000) 2022-09-10T09:16:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:43.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:43.383041+0000) 2022-09-10T09:16:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:43.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:43.893959+0000) 2022-09-10T09:16:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:44.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:44.383218+0000) 2022-09-10T09:16:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:44 smithi110 ceph-mon[97633]: pgmap v9200: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:44 smithi104 ceph-mon[97044]: pgmap v9200: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:44.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:44.894212+0000) 2022-09-10T09:16:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:45.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:45.383410+0000) 2022-09-10T09:16:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:16:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:16:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:45.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:45.894459+0000) 2022-09-10T09:16:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:46.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:46.383607+0000) 2022-09-10T09:16:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:46 smithi110 ceph-mon[97633]: pgmap v9201: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:46 smithi104 ceph-mon[97044]: pgmap v9201: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:46.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:46.894672+0000) 2022-09-10T09:16:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:47.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:47.383801+0000) 2022-09-10T09:16:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:47.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:47.894872+0000) 2022-09-10T09:16:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:48.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:48.384069+0000) 2022-09-10T09:16:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:48 smithi110 ceph-mon[97633]: pgmap v9202: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:48 smithi104 ceph-mon[97044]: pgmap v9202: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:48.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:48.895130+0000) 2022-09-10T09:16:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:49.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:49.384333+0000) 2022-09-10T09:16:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:49.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:49.895351+0000) 2022-09-10T09:16:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:50.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:50.384516+0000) 2022-09-10T09:16:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:50 smithi110 ceph-mon[97633]: pgmap v9203: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:16:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:50 smithi104 ceph-mon[97044]: pgmap v9203: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:16:51.202 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:50.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:50.895560+0000) 2022-09-10T09:16:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:51.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:51.384753+0000) 2022-09-10T09:16:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:16:51.200+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:16:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:51.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:51.895835+0000) 2022-09-10T09:16:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:52.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:52.385020+0000) 2022-09-10T09:16:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:52 smithi110 ceph-mon[97633]: pgmap v9204: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:16:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:52 smithi104 ceph-mon[97044]: pgmap v9204: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:16:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:52.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:52.896016+0000) 2022-09-10T09:16:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:53.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:53.385203+0000) 2022-09-10T09:16:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:53.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:53.896247+0000) 2022-09-10T09:16:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:54.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:54.385347+0000) 2022-09-10T09:16:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:54 smithi110 ceph-mon[97633]: pgmap v9205: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:54 smithi104 ceph-mon[97044]: pgmap v9205: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:54.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:54.896478+0000) 2022-09-10T09:16:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:55.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:55.385487+0000) 2022-09-10T09:16:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:55.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:55.896707+0000) 2022-09-10T09:16:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:56.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:56.385655+0000) 2022-09-10T09:16:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:56 smithi110 ceph-mon[97633]: pgmap v9206: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:56 smithi104 ceph-mon[97044]: pgmap v9206: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:16:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:56.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:56.896900+0000) 2022-09-10T09:16:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:57.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:57.385914+0000) 2022-09-10T09:16:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:57.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:57.897121+0000) 2022-09-10T09:16:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:58.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:58.386193+0000) 2022-09-10T09:16:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:16:58 smithi110 ceph-mon[97633]: pgmap v9207: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:16:58 smithi104 ceph-mon[97044]: pgmap v9207: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:16:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:58.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:58.897273+0000) 2022-09-10T09:16:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:16:59.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:59.386389+0000) 2022-09-10T09:17:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:16:59.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:16:59.897491+0000) 2022-09-10T09:17:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:00.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:00.386548+0000) 2022-09-10T09:17:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:00 smithi110 ceph-mon[97633]: pgmap v9208: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:17:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:00 smithi104 ceph-mon[97044]: pgmap v9208: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:17:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:00.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:00.897655+0000) 2022-09-10T09:17:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:01.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:01.386771+0000) 2022-09-10T09:17:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:01.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:01.897873+0000) 2022-09-10T09:17:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:02.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:02.387007+0000) 2022-09-10T09:17:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:02 smithi110 ceph-mon[97633]: pgmap v9209: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:17:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:02 smithi104 ceph-mon[97044]: pgmap v9209: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:17:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:02.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:02.898184+0000) 2022-09-10T09:17:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:03.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:03.387198+0000) 2022-09-10T09:17:04.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:03.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:03.898405+0000) 2022-09-10T09:17:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:04.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:04.387400+0000) 2022-09-10T09:17:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:04 smithi110 ceph-mon[97633]: pgmap v9210: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:04 smithi104 ceph-mon[97044]: pgmap v9210: 129 pgs: 129 active+clean; 8.7 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:04.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:04.898635+0000) 2022-09-10T09:17:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:05.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:05.387643+0000) 2022-09-10T09:17:06.208 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:05.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:05.898881+0000) 2022-09-10T09:17:06.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:17:06.206+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:17:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:06.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:06.387909+0000) 2022-09-10T09:17:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:06 smithi104 ceph-mon[97044]: pgmap v9211: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:17:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:06 smithi110 ceph-mon[97633]: pgmap v9211: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:17:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:06.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:06.899161+0000) 2022-09-10T09:17:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:07.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:07.388202+0000) 2022-09-10T09:17:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:07.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:07.899400+0000) 2022-09-10T09:17:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:08.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:08.388427+0000) 2022-09-10T09:17:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:08 smithi104 ceph-mon[97044]: pgmap v9212: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:17:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:08 smithi104 ceph-mon[97044]: osdmap e656: 12 total, 12 up, 12 in 2022-09-10T09:17:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:08 smithi110 ceph-mon[97633]: pgmap v9212: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:17:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:08 smithi110 ceph-mon[97633]: osdmap e656: 12 total, 12 up, 12 in 2022-09-10T09:17:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:08.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:08.899623+0000) 2022-09-10T09:17:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:09.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:09.388654+0000) 2022-09-10T09:17:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:09.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:09.899873+0000) 2022-09-10T09:17:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:10.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:10.388899+0000) 2022-09-10T09:17:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:10 smithi104 ceph-mon[97044]: pgmap v9214: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 29 op/s 2022-09-10T09:17:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:10 smithi110 ceph-mon[97633]: pgmap v9214: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 29 op/s 2022-09-10T09:17:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:10.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:10.900148+0000) 2022-09-10T09:17:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:11.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:11.389217+0000) 2022-09-10T09:17:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:11.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:11.900383+0000) 2022-09-10T09:17:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:12.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:12.389454+0000) 2022-09-10T09:17:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:12 smithi104 ceph-mon[97044]: pgmap v9215: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:17:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:12 smithi110 ceph-mon[97633]: pgmap v9215: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:17:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:12.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:12.900624+0000) 2022-09-10T09:17:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:13.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:13.389681+0000) 2022-09-10T09:17:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:13.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:13.900796+0000) 2022-09-10T09:17:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:14.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:14.389929+0000) 2022-09-10T09:17:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:14 smithi104 ceph-mon[97044]: pgmap v9216: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:17:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:14 smithi110 ceph-mon[97633]: pgmap v9216: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:17:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:14.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:14.900971+0000) 2022-09-10T09:17:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:15.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:15.390153+0000) 2022-09-10T09:17:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:15.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:15.901217+0000) 2022-09-10T09:17:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:16.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:16.390369+0000) 2022-09-10T09:17:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:16 smithi104 ceph-mon[97044]: pgmap v9217: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:17:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:16 smithi104 ceph-mon[97044]: osdmap e657: 12 total, 12 up, 12 in 2022-09-10T09:17:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:16 smithi110 ceph-mon[97633]: pgmap v9217: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:17:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:16 smithi110 ceph-mon[97633]: osdmap e657: 12 total, 12 up, 12 in 2022-09-10T09:17:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:16.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:16.901437+0000) 2022-09-10T09:17:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:17.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:17.390603+0000) 2022-09-10T09:17:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:17.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:17.901647+0000) 2022-09-10T09:17:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:18.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:18.390866+0000) 2022-09-10T09:17:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:18 smithi104 ceph-mon[97044]: pgmap v9219: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T09:17:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:18 smithi110 ceph-mon[97633]: pgmap v9219: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T09:17:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:18.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:18.901890+0000) 2022-09-10T09:17:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:19.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:19.391086+0000) 2022-09-10T09:17:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:19.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:19.902140+0000) 2022-09-10T09:17:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:20.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:20.391304+0000) 2022-09-10T09:17:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:20 smithi104 ceph-mon[97044]: pgmap v9220: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:17:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:20 smithi110 ceph-mon[97633]: pgmap v9220: 129 pgs: 129 active+clean; 8.7 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:17:21.223 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:20.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:20.902356+0000) 2022-09-10T09:17:21.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:17:21.221+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:17:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:21.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:21.391534+0000) 2022-09-10T09:17:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:21.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:21.902527+0000) 2022-09-10T09:17:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:22.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:22.391806+0000) 2022-09-10T09:17:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:22 smithi104 ceph-mon[97044]: pgmap v9221: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:17:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:22 smithi110 ceph-mon[97633]: pgmap v9221: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:17:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:22.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:22.902720+0000) 2022-09-10T09:17:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:23.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:23.392067+0000) 2022-09-10T09:17:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:23.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:23.902908+0000) 2022-09-10T09:17:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:24.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:24.392380+0000) 2022-09-10T09:17:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:24 smithi104 ceph-mon[97044]: pgmap v9222: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:17:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:24 smithi110 ceph-mon[97633]: pgmap v9222: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:17:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:24.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:24.903089+0000) 2022-09-10T09:17:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:25.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:25.392588+0000) 2022-09-10T09:17:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:25.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:25.903316+0000) 2022-09-10T09:17:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:26.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:26.392859+0000) 2022-09-10T09:17:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:26 smithi104 ceph-mon[97044]: pgmap v9223: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:17:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:26 smithi110 ceph-mon[97633]: pgmap v9223: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:17:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:26.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:26.903575+0000) 2022-09-10T09:17:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:27.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:27.393053+0000) 2022-09-10T09:17:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:27.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:27.903834+0000) 2022-09-10T09:17:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:28.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:28.393285+0000) 2022-09-10T09:17:29.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:28 smithi104 ceph-mon[97044]: pgmap v9224: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T09:17:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:28 smithi110 ceph-mon[97633]: pgmap v9224: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T09:17:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:28.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:28.904084+0000) 2022-09-10T09:17:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:29.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:29.393578+0000) 2022-09-10T09:17:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:29.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:29.904307+0000) 2022-09-10T09:17:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:30.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:30.393875+0000) 2022-09-10T09:17:31.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:30 smithi104 ceph-mon[97044]: pgmap v9225: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:17:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:30 smithi110 ceph-mon[97633]: pgmap v9225: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T09:17:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:30.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:30.904567+0000) 2022-09-10T09:17:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:31.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:31.394064+0000) 2022-09-10T09:17:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:31.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:31.904794+0000) 2022-09-10T09:17:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:32.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:32.394301+0000) 2022-09-10T09:17:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:32 smithi104 ceph-mon[97044]: pgmap v9226: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:17:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:32 smithi110 ceph-mon[97633]: pgmap v9226: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:17:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:32.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:32.905019+0000) 2022-09-10T09:17:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:33.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:33.394495+0000) 2022-09-10T09:17:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:33.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:33.905257+0000) 2022-09-10T09:17:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:34.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:34.394672+0000) 2022-09-10T09:17:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:34 smithi104 ceph-mon[97044]: pgmap v9227: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:34 smithi110 ceph-mon[97633]: pgmap v9227: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:34.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:34.905493+0000) 2022-09-10T09:17:35.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:35.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:35.394915+0000) 2022-09-10T09:17:36.238 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:35.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:35.905768+0000) 2022-09-10T09:17:36.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:17:36.236+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:17:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:36.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:36.395209+0000) 2022-09-10T09:17:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:36 smithi104 ceph-mon[97044]: pgmap v9228: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:17:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:36 smithi110 ceph-mon[97633]: pgmap v9228: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:17:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:36.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:36.906058+0000) 2022-09-10T09:17:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:37.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:37.395370+0000) 2022-09-10T09:17:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:37.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:37.906295+0000) 2022-09-10T09:17:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:38.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:38.395594+0000) 2022-09-10T09:17:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:38 smithi104 ceph-mon[97044]: pgmap v9229: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:38 smithi110 ceph-mon[97633]: pgmap v9229: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:38.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:38.906569+0000) 2022-09-10T09:17:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:39.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:39.395762+0000) 2022-09-10T09:17:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:39.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:39.906834+0000) 2022-09-10T09:17:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:40.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:40.396008+0000) 2022-09-10T09:17:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:40 smithi104 ceph-mon[97044]: pgmap v9230: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:17:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:40 smithi110 ceph-mon[97633]: pgmap v9230: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:17:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:40.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:40.907125+0000) 2022-09-10T09:17:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:41.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:41.396151+0000) 2022-09-10T09:17:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:41.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:41.907419+0000) 2022-09-10T09:17:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:42.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:42.396333+0000) 2022-09-10T09:17:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:42 smithi104 ceph-mon[97044]: pgmap v9231: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T09:17:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:42 smithi110 ceph-mon[97633]: pgmap v9231: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T09:17:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:42.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:42.907673+0000) 2022-09-10T09:17:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:43.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:43.396521+0000) 2022-09-10T09:17:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:43.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:43.907911+0000) 2022-09-10T09:17:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:44.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:44.396790+0000) 2022-09-10T09:17:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:44 smithi104 ceph-mon[97044]: pgmap v9232: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:17:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:44 smithi110 ceph-mon[97633]: pgmap v9232: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:17:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:44.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:44.908174+0000) 2022-09-10T09:17:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:45.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:45.396991+0000) 2022-09-10T09:17:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:17:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:17:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:17:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:17:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:17:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:45.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:45.908317+0000) 2022-09-10T09:17:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:46.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:46.397199+0000) 2022-09-10T09:17:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:46 smithi104 ceph-mon[97044]: pgmap v9233: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:17:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:46 smithi110 ceph-mon[97633]: pgmap v9233: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:17:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:46.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:46.908543+0000) 2022-09-10T09:17:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:47.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:47.397427+0000) 2022-09-10T09:17:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:47.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:47.908678+0000) 2022-09-10T09:17:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:48.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:48.397623+0000) 2022-09-10T09:17:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:48 smithi104 ceph-mon[97044]: pgmap v9234: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:17:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:48 smithi110 ceph-mon[97633]: pgmap v9234: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:17:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:48.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:48.908900+0000) 2022-09-10T09:17:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:49.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:49.397816+0000) 2022-09-10T09:17:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:49.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:49.909154+0000) 2022-09-10T09:17:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:50.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:50.398075+0000) 2022-09-10T09:17:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:50 smithi104 ceph-mon[97044]: pgmap v9235: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:17:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:50 smithi110 ceph-mon[97633]: pgmap v9235: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:17:51.253 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:50.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:50.909338+0000) 2022-09-10T09:17:51.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:17:51.251+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:17:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:51.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:51.398376+0000) 2022-09-10T09:17:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:51.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:51.909497+0000) 2022-09-10T09:17:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:52.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:52.398602+0000) 2022-09-10T09:17:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:52 smithi104 ceph-mon[97044]: pgmap v9236: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:17:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:52 smithi110 ceph-mon[97633]: pgmap v9236: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:17:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:52.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:52.909698+0000) 2022-09-10T09:17:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:53.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:53.398865+0000) 2022-09-10T09:17:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:53.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:53.909921+0000) 2022-09-10T09:17:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:54.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:54.399121+0000) 2022-09-10T09:17:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:54 smithi104 ceph-mon[97044]: pgmap v9237: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:54 smithi110 ceph-mon[97633]: pgmap v9237: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:54.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:54.910086+0000) 2022-09-10T09:17:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:55.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:55.399269+0000) 2022-09-10T09:17:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:55.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:55.910269+0000) 2022-09-10T09:17:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:56.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:56.399437+0000) 2022-09-10T09:17:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:56 smithi104 ceph-mon[97044]: pgmap v9238: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:17:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:56 smithi110 ceph-mon[97633]: pgmap v9238: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:17:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:56.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:56.910472+0000) 2022-09-10T09:17:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:57.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:57.399610+0000) 2022-09-10T09:17:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:57.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:57.910694+0000) 2022-09-10T09:17:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:58.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:58.399793+0000) 2022-09-10T09:17:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:17:58 smithi104 ceph-mon[97044]: pgmap v9239: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:17:58 smithi110 ceph-mon[97633]: pgmap v9239: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:17:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:58.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:58.910882+0000) 2022-09-10T09:17:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:17:59.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:59.400048+0000) 2022-09-10T09:18:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:17:59.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:17:59.911160+0000) 2022-09-10T09:18:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:00.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:00.400299+0000) 2022-09-10T09:18:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:00 smithi104 ceph-mon[97044]: pgmap v9240: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:00 smithi110 ceph-mon[97633]: pgmap v9240: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:00.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:00.911393+0000) 2022-09-10T09:18:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:01.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:01.400526+0000) 2022-09-10T09:18:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:01.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:01.911632+0000) 2022-09-10T09:18:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:02.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:02.400749+0000) 2022-09-10T09:18:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:02 smithi104 ceph-mon[97044]: pgmap v9241: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:02 smithi110 ceph-mon[97633]: pgmap v9241: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:02.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:02.911894+0000) 2022-09-10T09:18:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:03.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:03.400996+0000) 2022-09-10T09:18:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:03.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:03.912171+0000) 2022-09-10T09:18:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:04.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:04.401263+0000) 2022-09-10T09:18:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:04 smithi104 ceph-mon[97044]: pgmap v9242: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:04 smithi110 ceph-mon[97633]: pgmap v9242: 129 pgs: 129 active+clean; 8.7 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:04.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:04.912416+0000) 2022-09-10T09:18:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:05.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:05.401504+0000) 2022-09-10T09:18:06.269 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:05.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:05.912665+0000) 2022-09-10T09:18:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:06.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:06.401749+0000) 2022-09-10T09:18:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:18:06.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:18:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:06 smithi104 ceph-mon[97044]: pgmap v9243: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:18:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:06 smithi110 ceph-mon[97633]: pgmap v9243: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:18:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:06.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:06.912905+0000) 2022-09-10T09:18:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:07.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:07.401976+0000) 2022-09-10T09:18:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:07.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:07.913181+0000) 2022-09-10T09:18:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:08.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:08.402157+0000) 2022-09-10T09:18:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:08 smithi104 ceph-mon[97044]: pgmap v9244: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:18:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:08 smithi104 ceph-mon[97044]: osdmap e658: 12 total, 12 up, 12 in 2022-09-10T09:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:08 smithi110 ceph-mon[97633]: pgmap v9244: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:18:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:08 smithi110 ceph-mon[97633]: osdmap e658: 12 total, 12 up, 12 in 2022-09-10T09:18:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:08.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:08.913403+0000) 2022-09-10T09:18:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:09.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:09.402328+0000) 2022-09-10T09:18:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:09.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:09.913631+0000) 2022-09-10T09:18:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:10.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:10.402475+0000) 2022-09-10T09:18:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:10 smithi104 ceph-mon[97044]: pgmap v9246: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:18:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:10 smithi110 ceph-mon[97633]: pgmap v9246: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:18:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:10.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:10.913812+0000) 2022-09-10T09:18:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:11.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:11.402659+0000) 2022-09-10T09:18:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:11.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:11.914061+0000) 2022-09-10T09:18:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:12.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:12.402854+0000) 2022-09-10T09:18:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:12 smithi104 ceph-mon[97044]: pgmap v9247: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:18:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:12 smithi110 ceph-mon[97633]: pgmap v9247: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:18:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:12.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:12.914316+0000) 2022-09-10T09:18:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:13.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:13.403132+0000) 2022-09-10T09:18:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:13.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:13.914540+0000) 2022-09-10T09:18:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:14.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:14.403325+0000) 2022-09-10T09:18:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:14 smithi104 ceph-mon[97044]: pgmap v9248: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:18:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:14 smithi110 ceph-mon[97633]: pgmap v9248: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:18:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:14.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:14.914762+0000) 2022-09-10T09:18:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:15.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:15.403512+0000) 2022-09-10T09:18:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:15.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:15.914993+0000) 2022-09-10T09:18:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:16.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:16.403791+0000) 2022-09-10T09:18:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:16 smithi104 ceph-mon[97044]: pgmap v9249: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:18:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:16 smithi104 ceph-mon[97044]: osdmap e659: 12 total, 12 up, 12 in 2022-09-10T09:18:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:16 smithi110 ceph-mon[97633]: pgmap v9249: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:18:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:16 smithi110 ceph-mon[97633]: osdmap e659: 12 total, 12 up, 12 in 2022-09-10T09:18:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:16.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:16.915193+0000) 2022-09-10T09:18:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:17.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:17.404062+0000) 2022-09-10T09:18:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:17.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:17.915393+0000) 2022-09-10T09:18:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:18.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:18.404362+0000) 2022-09-10T09:18:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:18 smithi104 ceph-mon[97044]: pgmap v9251: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T09:18:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:18 smithi110 ceph-mon[97633]: pgmap v9251: 129 pgs: 129 active+clean; 8.7 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T09:18:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:18.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:18.915631+0000) 2022-09-10T09:18:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:19.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:19.404527+0000) 2022-09-10T09:18:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:19.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:19.915853+0000) 2022-09-10T09:18:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:20.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:20.404762+0000) 2022-09-10T09:18:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:20 smithi104 ceph-mon[97044]: pgmap v9252: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:18:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:20 smithi110 ceph-mon[97633]: pgmap v9252: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:18:21.278 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:20.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:20.916079+0000) 2022-09-10T09:18:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:21.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:21.404981+0000) 2022-09-10T09:18:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:18:21.276+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:18:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:21.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:21.916306+0000) 2022-09-10T09:18:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:22.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:22.405195+0000) 2022-09-10T09:18:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:22 smithi104 ceph-mon[97044]: pgmap v9253: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:18:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:22 smithi110 ceph-mon[97633]: pgmap v9253: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:18:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:22.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:22.916551+0000) 2022-09-10T09:18:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:23.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:23.405429+0000) 2022-09-10T09:18:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:23.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:23.916769+0000) 2022-09-10T09:18:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:24.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:24.405599+0000) 2022-09-10T09:18:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:24 smithi104 ceph-mon[97044]: pgmap v9254: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:18:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:24 smithi110 ceph-mon[97633]: pgmap v9254: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:18:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:24.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:24.917035+0000) 2022-09-10T09:18:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:25.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:25.405813+0000) 2022-09-10T09:18:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:25.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:25.917211+0000) 2022-09-10T09:18:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:26.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:26.406006+0000) 2022-09-10T09:18:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:26 smithi104 ceph-mon[97044]: pgmap v9255: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:18:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:26 smithi110 ceph-mon[97633]: pgmap v9255: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:18:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:26.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:26.917422+0000) 2022-09-10T09:18:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:27.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:27.406240+0000) 2022-09-10T09:18:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:27.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:27.917628+0000) 2022-09-10T09:18:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:28.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:28.406377+0000) 2022-09-10T09:18:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:28 smithi104 ceph-mon[97044]: pgmap v9256: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:18:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:28 smithi110 ceph-mon[97633]: pgmap v9256: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:18:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:28.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:28.917849+0000) 2022-09-10T09:18:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:29.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:29.406608+0000) 2022-09-10T09:18:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:29.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:29.918014+0000) 2022-09-10T09:18:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:30.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:30.406845+0000) 2022-09-10T09:18:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:30 smithi110 ceph-mon[97633]: pgmap v9257: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:30 smithi104 ceph-mon[97044]: pgmap v9257: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:30.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:30.918245+0000) 2022-09-10T09:18:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:31.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:31.406987+0000) 2022-09-10T09:18:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:31.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:31.918493+0000) 2022-09-10T09:18:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:32.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:32.407200+0000) 2022-09-10T09:18:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:32 smithi110 ceph-mon[97633]: pgmap v9258: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:33.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:32 smithi104 ceph-mon[97044]: pgmap v9258: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:33.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:32.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:32.918725+0000) 2022-09-10T09:18:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:33.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:33.407405+0000) 2022-09-10T09:18:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:33.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:33.918903+0000) 2022-09-10T09:18:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:34.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:34.407588+0000) 2022-09-10T09:18:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:34 smithi110 ceph-mon[97633]: pgmap v9259: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:34 smithi104 ceph-mon[97044]: pgmap v9259: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:34.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:34.919138+0000) 2022-09-10T09:18:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:35.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:35.407809+0000) 2022-09-10T09:18:36.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:35.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:35.919380+0000) 2022-09-10T09:18:36.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:18:36.286+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:18:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:36.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:36.408059+0000) 2022-09-10T09:18:37.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:36 smithi104 ceph-mon[97044]: pgmap v9260: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:18:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:36 smithi110 ceph-mon[97633]: pgmap v9260: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:18:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:36.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:36.919624+0000) 2022-09-10T09:18:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:37.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:37.408256+0000) 2022-09-10T09:18:38.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:37.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:37.919893+0000) 2022-09-10T09:18:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:38.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:38.408453+0000) 2022-09-10T09:18:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:38 smithi110 ceph-mon[97633]: pgmap v9261: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:38 smithi104 ceph-mon[97044]: pgmap v9261: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:38.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:38.920143+0000) 2022-09-10T09:18:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:39.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:39.408658+0000) 2022-09-10T09:18:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:39.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:39.920319+0000) 2022-09-10T09:18:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:40.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:40.408871+0000) 2022-09-10T09:18:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:40 smithi110 ceph-mon[97633]: pgmap v9262: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:40 smithi104 ceph-mon[97044]: pgmap v9262: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:40.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:40.920563+0000) 2022-09-10T09:18:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:41.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:41.409029+0000) 2022-09-10T09:18:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:41.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:41.920791+0000) 2022-09-10T09:18:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:42.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:42.409246+0000) 2022-09-10T09:18:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:42 smithi110 ceph-mon[97633]: pgmap v9263: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:42 smithi104 ceph-mon[97044]: pgmap v9263: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:42.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:42.921062+0000) 2022-09-10T09:18:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:43.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:43.409488+0000) 2022-09-10T09:18:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:43.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:43.921330+0000) 2022-09-10T09:18:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:44.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:44.409692+0000) 2022-09-10T09:18:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:44 smithi110 ceph-mon[97633]: pgmap v9264: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:44 smithi104 ceph-mon[97044]: pgmap v9264: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:44.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:44.921474+0000) 2022-09-10T09:18:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:45.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:45.409903+0000) 2022-09-10T09:18:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:18:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:18:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:18:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:18:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:18:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:18:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:18:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:18:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:45.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:45.921678+0000) 2022-09-10T09:18:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:46.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:46.410178+0000) 2022-09-10T09:18:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:46 smithi110 ceph-mon[97633]: pgmap v9265: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:18:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:46 smithi104 ceph-mon[97044]: pgmap v9265: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:18:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:46.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:46.921818+0000) 2022-09-10T09:18:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:47.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:47.410424+0000) 2022-09-10T09:18:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:47.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:47.922045+0000) 2022-09-10T09:18:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:48.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:48.410691+0000) 2022-09-10T09:18:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:48 smithi110 ceph-mon[97633]: pgmap v9266: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:48 smithi104 ceph-mon[97044]: pgmap v9266: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:48.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:48.922285+0000) 2022-09-10T09:18:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:49.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:49.410959+0000) 2022-09-10T09:18:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:49.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:49.922539+0000) 2022-09-10T09:18:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:50.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:50.411221+0000) 2022-09-10T09:18:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:50 smithi110 ceph-mon[97633]: pgmap v9267: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:50 smithi104 ceph-mon[97044]: pgmap v9267: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:18:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:50.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:50.922742+0000) 2022-09-10T09:18:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:51.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:51.411461+0000) 2022-09-10T09:18:51.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:18:51.302+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:18:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:51.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:51.922917+0000) 2022-09-10T09:18:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:52.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:52.411697+0000) 2022-09-10T09:18:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:52 smithi110 ceph-mon[97633]: pgmap v9268: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:52 smithi104 ceph-mon[97044]: pgmap v9268: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:18:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:52.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:52.923204+0000) 2022-09-10T09:18:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:53.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:53.411985+0000) 2022-09-10T09:18:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:53.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:53.923390+0000) 2022-09-10T09:18:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:54.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:54.412212+0000) 2022-09-10T09:18:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:54 smithi110 ceph-mon[97633]: pgmap v9269: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:54 smithi104 ceph-mon[97044]: pgmap v9269: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:54.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:54.923630+0000) 2022-09-10T09:18:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:55.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:55.412499+0000) 2022-09-10T09:18:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:55.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:55.923942+0000) 2022-09-10T09:18:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:56.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:56.412708+0000) 2022-09-10T09:18:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:56 smithi110 ceph-mon[97633]: pgmap v9270: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:18:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:56 smithi104 ceph-mon[97044]: pgmap v9270: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:18:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:56.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:56.924250+0000) 2022-09-10T09:18:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:57.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:57.412882+0000) 2022-09-10T09:18:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:57.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:57.924443+0000) 2022-09-10T09:18:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:58.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:58.413033+0000) 2022-09-10T09:18:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:18:58 smithi110 ceph-mon[97633]: pgmap v9271: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:18:58 smithi104 ceph-mon[97044]: pgmap v9271: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:18:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:58.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:58.924572+0000) 2022-09-10T09:18:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:18:59.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:59.413258+0000) 2022-09-10T09:19:00.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:18:59.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:18:59.924793+0000) 2022-09-10T09:19:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:00.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:00.413515+0000) 2022-09-10T09:19:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:00 smithi110 ceph-mon[97633]: pgmap v9272: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:00 smithi104 ceph-mon[97044]: pgmap v9272: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:00.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:00.924979+0000) 2022-09-10T09:19:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:01.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:01.413792+0000) 2022-09-10T09:19:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:01.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:01.925220+0000) 2022-09-10T09:19:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:02.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:02.413979+0000) 2022-09-10T09:19:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:02 smithi110 ceph-mon[97633]: pgmap v9273: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:02 smithi104 ceph-mon[97044]: pgmap v9273: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:02.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:02.925391+0000) 2022-09-10T09:19:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:03.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:03.414228+0000) 2022-09-10T09:19:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:03.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:03.925544+0000) 2022-09-10T09:19:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:04.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:04.414419+0000) 2022-09-10T09:19:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:04 smithi110 ceph-mon[97633]: pgmap v9274: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:04 smithi104 ceph-mon[97044]: pgmap v9274: 129 pgs: 129 active+clean; 8.7 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:04.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:04.925751+0000) 2022-09-10T09:19:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:05.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:05.414597+0000) 2022-09-10T09:19:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:05.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:05.926000+0000) 2022-09-10T09:19:06.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:19:06.318+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:19:06.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:06.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:06.414819+0000) 2022-09-10T09:19:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:06 smithi110 ceph-mon[97633]: pgmap v9275: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:19:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:06 smithi104 ceph-mon[97044]: pgmap v9275: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:19:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:06.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:06.926191+0000) 2022-09-10T09:19:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:07.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:07.415061+0000) 2022-09-10T09:19:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:07.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:07.926420+0000) 2022-09-10T09:19:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:08.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:08.415340+0000) 2022-09-10T09:19:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:08 smithi110 ceph-mon[97633]: pgmap v9276: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:19:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:08 smithi110 ceph-mon[97633]: osdmap e660: 12 total, 12 up, 12 in 2022-09-10T09:19:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:08 smithi104 ceph-mon[97044]: pgmap v9276: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:19:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:08 smithi104 ceph-mon[97044]: osdmap e660: 12 total, 12 up, 12 in 2022-09-10T09:19:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:08.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:08.926647+0000) 2022-09-10T09:19:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:09.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:09.415540+0000) 2022-09-10T09:19:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:09.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:09.926884+0000) 2022-09-10T09:19:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:10.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:10.415785+0000) 2022-09-10T09:19:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:10 smithi110 ceph-mon[97633]: pgmap v9278: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:19:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:10 smithi104 ceph-mon[97044]: pgmap v9278: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:19:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:10.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:10.927084+0000) 2022-09-10T09:19:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:11.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:11.416023+0000) 2022-09-10T09:19:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:11.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:11.927314+0000) 2022-09-10T09:19:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:12.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:12.416271+0000) 2022-09-10T09:19:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:12 smithi110 ceph-mon[97633]: pgmap v9279: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:19:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:12 smithi104 ceph-mon[97044]: pgmap v9279: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:19:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:12.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:12.927621+0000) 2022-09-10T09:19:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:13.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:13.416520+0000) 2022-09-10T09:19:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:13.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:13.927835+0000) 2022-09-10T09:19:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:14.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:14.416770+0000) 2022-09-10T09:19:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:14 smithi110 ceph-mon[97633]: pgmap v9280: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:19:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:14 smithi104 ceph-mon[97044]: pgmap v9280: 129 pgs: 129 active+clean; 8.7 MiB data, 883 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:19:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:14.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:14.928010+0000) 2022-09-10T09:19:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:15.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:15.416964+0000) 2022-09-10T09:19:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:15.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:15.928221+0000) 2022-09-10T09:19:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:16.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:16.417203+0000) 2022-09-10T09:19:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:16 smithi110 ceph-mon[97633]: pgmap v9281: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:19:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:16 smithi110 ceph-mon[97633]: osdmap e661: 12 total, 12 up, 12 in 2022-09-10T09:19:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:16 smithi104 ceph-mon[97044]: pgmap v9281: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:19:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:16 smithi104 ceph-mon[97044]: osdmap e661: 12 total, 12 up, 12 in 2022-09-10T09:19:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:16.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:16.928468+0000) 2022-09-10T09:19:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:17.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:17.417444+0000) 2022-09-10T09:19:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:17.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:17.928648+0000) 2022-09-10T09:19:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:18.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:18.417614+0000) 2022-09-10T09:19:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:18 smithi110 ceph-mon[97633]: pgmap v9283: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 216 B/s wr, 10 op/s 2022-09-10T09:19:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:18 smithi104 ceph-mon[97044]: pgmap v9283: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 216 B/s wr, 10 op/s 2022-09-10T09:19:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:18.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:18.928863+0000) 2022-09-10T09:19:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:19.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:19.417778+0000) 2022-09-10T09:19:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:19.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:19.929138+0000) 2022-09-10T09:19:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:20.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:20.418004+0000) 2022-09-10T09:19:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:20 smithi110 ceph-mon[97633]: pgmap v9284: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:19:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:20 smithi104 ceph-mon[97044]: pgmap v9284: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:19:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:20.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:20.929322+0000) 2022-09-10T09:19:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:21.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:21.418204+0000) 2022-09-10T09:19:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:19:21.323+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:19:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:21.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:21.929480+0000) 2022-09-10T09:19:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:22.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:22.418493+0000) 2022-09-10T09:19:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:22 smithi110 ceph-mon[97633]: pgmap v9285: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:19:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:22 smithi104 ceph-mon[97044]: pgmap v9285: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:19:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:22.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:22.929743+0000) 2022-09-10T09:19:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:23.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:23.418680+0000) 2022-09-10T09:19:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:23.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:23.929942+0000) 2022-09-10T09:19:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:24.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:24.418912+0000) 2022-09-10T09:19:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:24 smithi110 ceph-mon[97633]: pgmap v9286: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:19:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:24 smithi104 ceph-mon[97044]: pgmap v9286: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:19:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:24.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:24.930174+0000) 2022-09-10T09:19:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:25.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:25.419171+0000) 2022-09-10T09:19:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:25.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:25.930408+0000) 2022-09-10T09:19:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:26.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:26.419395+0000) 2022-09-10T09:19:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:26 smithi110 ceph-mon[97633]: pgmap v9287: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:19:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:26 smithi104 ceph-mon[97044]: pgmap v9287: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:19:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:26.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:26.930715+0000) 2022-09-10T09:19:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:27.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:27.419554+0000) 2022-09-10T09:19:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:27.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:27.930967+0000) 2022-09-10T09:19:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:28.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:28.419716+0000) 2022-09-10T09:19:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:28 smithi110 ceph-mon[97633]: pgmap v9288: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:19:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:28 smithi104 ceph-mon[97044]: pgmap v9288: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:19:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:28.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:28.931164+0000) 2022-09-10T09:19:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:29.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:29.419961+0000) 2022-09-10T09:19:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:29.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:29.931306+0000) 2022-09-10T09:19:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:30.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:30.420177+0000) 2022-09-10T09:19:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:30 smithi110 ceph-mon[97633]: pgmap v9289: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:30 smithi104 ceph-mon[97044]: pgmap v9289: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:30.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:30.931475+0000) 2022-09-10T09:19:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:31.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:31.420396+0000) 2022-09-10T09:19:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:31.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:31.931617+0000) 2022-09-10T09:19:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:32.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:32.420569+0000) 2022-09-10T09:19:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:32 smithi110 ceph-mon[97633]: pgmap v9290: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:32 smithi104 ceph-mon[97044]: pgmap v9290: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:32.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:32.931812+0000) 2022-09-10T09:19:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:33.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:33.420802+0000) 2022-09-10T09:19:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:33.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:33.932040+0000) 2022-09-10T09:19:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:34.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:34.421047+0000) 2022-09-10T09:19:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:34 smithi110 ceph-mon[97633]: pgmap v9291: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:34 smithi104 ceph-mon[97044]: pgmap v9291: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:34.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:34.932230+0000) 2022-09-10T09:19:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:35.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:35.421241+0000) 2022-09-10T09:19:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:35.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:35.932477+0000) 2022-09-10T09:19:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:36.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:36.421488+0000) 2022-09-10T09:19:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:19:36.337+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:19:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:36 smithi110 ceph-mon[97633]: pgmap v9292: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:19:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:36 smithi104 ceph-mon[97044]: pgmap v9292: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:19:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:36.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:36.932674+0000) 2022-09-10T09:19:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:37.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:37.421754+0000) 2022-09-10T09:19:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:37.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:37.932817+0000) 2022-09-10T09:19:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:38.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:38.421981+0000) 2022-09-10T09:19:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:38 smithi110 ceph-mon[97633]: pgmap v9293: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:38 smithi104 ceph-mon[97044]: pgmap v9293: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:38.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:38.933023+0000) 2022-09-10T09:19:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:39.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:39.422134+0000) 2022-09-10T09:19:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:39.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:39.933275+0000) 2022-09-10T09:19:40.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:40.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:40.422301+0000) 2022-09-10T09:19:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:40 smithi104 ceph-mon[97044]: pgmap v9294: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:40.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:40.933510+0000) 2022-09-10T09:19:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:40 smithi110 ceph-mon[97633]: pgmap v9294: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:41.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:41.422505+0000) 2022-09-10T09:19:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:41.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:41.933737+0000) 2022-09-10T09:19:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:42.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:42.422663+0000) 2022-09-10T09:19:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:42 smithi104 ceph-mon[97044]: pgmap v9295: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:42.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:42.933992+0000) 2022-09-10T09:19:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:42 smithi110 ceph-mon[97633]: pgmap v9295: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:43.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:43.422879+0000) 2022-09-10T09:19:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:43.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:43.934196+0000) 2022-09-10T09:19:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:44.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:44.423115+0000) 2022-09-10T09:19:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:44 smithi104 ceph-mon[97044]: pgmap v9296: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:19:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:19:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:19:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:44 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:19:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:44.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:44.934494+0000) 2022-09-10T09:19:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:44 smithi110 ceph-mon[97633]: pgmap v9296: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:19:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:19:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:19:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:44 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:19:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:45.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:45.423360+0000) 2022-09-10T09:19:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:45.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:45.934689+0000) 2022-09-10T09:19:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:46.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:46.423498+0000) 2022-09-10T09:19:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:46 smithi104 ceph-mon[97044]: pgmap v9297: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:19:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:46.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:46.934927+0000) 2022-09-10T09:19:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:46 smithi110 ceph-mon[97633]: pgmap v9297: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:19:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:47.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:47.423673+0000) 2022-09-10T09:19:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:47.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:47.935155+0000) 2022-09-10T09:19:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:48.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:48.423943+0000) 2022-09-10T09:19:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:48 smithi104 ceph-mon[97044]: pgmap v9298: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:48.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:48.935324+0000) 2022-09-10T09:19:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:48 smithi110 ceph-mon[97633]: pgmap v9298: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:49.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:49.424127+0000) 2022-09-10T09:19:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:49.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:49.935576+0000) 2022-09-10T09:19:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:50.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:50.424360+0000) 2022-09-10T09:19:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:50 smithi104 ceph-mon[97044]: pgmap v9299: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:50.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:50.935822+0000) 2022-09-10T09:19:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:50 smithi110 ceph-mon[97633]: pgmap v9299: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:19:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:19:51.349+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:19:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:51.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:51.424560+0000) 2022-09-10T09:19:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:51.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:51.935959+0000) 2022-09-10T09:19:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:52.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:52.424758+0000) 2022-09-10T09:19:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:52 smithi104 ceph-mon[97044]: pgmap v9300: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:52.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:52.936157+0000) 2022-09-10T09:19:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:52 smithi110 ceph-mon[97633]: pgmap v9300: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:19:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:53.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:53.425005+0000) 2022-09-10T09:19:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:53.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:53.936368+0000) 2022-09-10T09:19:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:54.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:54.425197+0000) 2022-09-10T09:19:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:54 smithi104 ceph-mon[97044]: pgmap v9301: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:54.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:54.936628+0000) 2022-09-10T09:19:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:54 smithi110 ceph-mon[97633]: pgmap v9301: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:55.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:55.425428+0000) 2022-09-10T09:19:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:55.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:55.936869+0000) 2022-09-10T09:19:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:56.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:56.425617+0000) 2022-09-10T09:19:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:56 smithi104 ceph-mon[97044]: pgmap v9302: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:19:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:56.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:56.937137+0000) 2022-09-10T09:19:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:56 smithi110 ceph-mon[97633]: pgmap v9302: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:19:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:57.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:57.425810+0000) 2022-09-10T09:19:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:57.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:57.937402+0000) 2022-09-10T09:19:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:58.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:58.426071+0000) 2022-09-10T09:19:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:19:58 smithi104 ceph-mon[97044]: pgmap v9303: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:58.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:58.937622+0000) 2022-09-10T09:19:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:19:58 smithi110 ceph-mon[97633]: pgmap v9303: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:19:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:19:59.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:59.426282+0000) 2022-09-10T09:20:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:19:59.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:19:59.937762+0000) 2022-09-10T09:20:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:00.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:00.426462+0000) 2022-09-10T09:20:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:00 smithi104 ceph-mon[97044]: pgmap v9304: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:01.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:20:01.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:00.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:00.937922+0000) 2022-09-10T09:20:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:00 smithi110 ceph-mon[97633]: pgmap v9304: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:20:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:01.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:01.426668+0000) 2022-09-10T09:20:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:01.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:01.938140+0000) 2022-09-10T09:20:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:02.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:02.426883+0000) 2022-09-10T09:20:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:02 smithi104 ceph-mon[97044]: pgmap v9305: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:02.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:02.938268+0000) 2022-09-10T09:20:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:02 smithi110 ceph-mon[97633]: pgmap v9305: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:03.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:03.427076+0000) 2022-09-10T09:20:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:03.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:03.938449+0000) 2022-09-10T09:20:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:04.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:04.427252+0000) 2022-09-10T09:20:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:04 smithi104 ceph-mon[97044]: pgmap v9306: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:04.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:04.938698+0000) 2022-09-10T09:20:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:04 smithi110 ceph-mon[97633]: pgmap v9306: 129 pgs: 129 active+clean; 8.7 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:05.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:05.427416+0000) 2022-09-10T09:20:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:05.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:05.938957+0000) 2022-09-10T09:20:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:20:06.361+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:20:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:06.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:06.427592+0000) 2022-09-10T09:20:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:06 smithi104 ceph-mon[97044]: pgmap v9307: 129 pgs: 129 active+clean; 8.7 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:20:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:06.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:06.939160+0000) 2022-09-10T09:20:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:06 smithi110 ceph-mon[97633]: pgmap v9307: 129 pgs: 129 active+clean; 8.7 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:20:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:07.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:07.427835+0000) 2022-09-10T09:20:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:07.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:07.939371+0000) 2022-09-10T09:20:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:08.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:08.428059+0000) 2022-09-10T09:20:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:08 smithi104 ceph-mon[97044]: pgmap v9308: 129 pgs: 129 active+clean; 8.7 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:20:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:08 smithi104 ceph-mon[97044]: osdmap e662: 12 total, 12 up, 12 in 2022-09-10T09:20:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:08.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:08.939573+0000) 2022-09-10T09:20:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:08 smithi110 ceph-mon[97633]: pgmap v9308: 129 pgs: 129 active+clean; 8.7 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:20:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:08 smithi110 ceph-mon[97633]: osdmap e662: 12 total, 12 up, 12 in 2022-09-10T09:20:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:09.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:09.428309+0000) 2022-09-10T09:20:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:09.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:09.939740+0000) 2022-09-10T09:20:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:10.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:10.428510+0000) 2022-09-10T09:20:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:10 smithi104 ceph-mon[97044]: pgmap v9310: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:20:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:10.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:10.939919+0000) 2022-09-10T09:20:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:10 smithi110 ceph-mon[97633]: pgmap v9310: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:20:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:11.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:11.428709+0000) 2022-09-10T09:20:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:11.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:11.940180+0000) 2022-09-10T09:20:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:12.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:12.428997+0000) 2022-09-10T09:20:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:12 smithi104 ceph-mon[97044]: pgmap v9311: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:20:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:12.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:12.940371+0000) 2022-09-10T09:20:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:12 smithi110 ceph-mon[97633]: pgmap v9311: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:20:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:13.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:13.429216+0000) 2022-09-10T09:20:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:13.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:13.940517+0000) 2022-09-10T09:20:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:14.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:14.429403+0000) 2022-09-10T09:20:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:14 smithi104 ceph-mon[97044]: pgmap v9312: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:20:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:14.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:14.940801+0000) 2022-09-10T09:20:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:14 smithi110 ceph-mon[97633]: pgmap v9312: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:20:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:15.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:15.429633+0000) 2022-09-10T09:20:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:15.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:15.941025+0000) 2022-09-10T09:20:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:16.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:16.429891+0000) 2022-09-10T09:20:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:16 smithi104 ceph-mon[97044]: pgmap v9313: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:20:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:16 smithi104 ceph-mon[97044]: osdmap e663: 12 total, 12 up, 12 in 2022-09-10T09:20:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:16.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:16.941268+0000) 2022-09-10T09:20:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:16 smithi110 ceph-mon[97633]: pgmap v9313: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:20:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:16 smithi110 ceph-mon[97633]: osdmap e663: 12 total, 12 up, 12 in 2022-09-10T09:20:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:17.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:17.430133+0000) 2022-09-10T09:20:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:17.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:17.941491+0000) 2022-09-10T09:20:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:18.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:18.430349+0000) 2022-09-10T09:20:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:18 smithi104 ceph-mon[97044]: pgmap v9315: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T09:20:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:18.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:18.941686+0000) 2022-09-10T09:20:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:18 smithi110 ceph-mon[97633]: pgmap v9315: 129 pgs: 129 active+clean; 8.7 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T09:20:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:19.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:19.430494+0000) 2022-09-10T09:20:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:19.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:19.941964+0000) 2022-09-10T09:20:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:20.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:20.430713+0000) 2022-09-10T09:20:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:20 smithi104 ceph-mon[97044]: pgmap v9316: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:20:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:20.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:20.942225+0000) 2022-09-10T09:20:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:20 smithi110 ceph-mon[97633]: pgmap v9316: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:20:21.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:20:21.367+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:20:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:21.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:21.431000+0000) 2022-09-10T09:20:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:21.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:21.942506+0000) 2022-09-10T09:20:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:22.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:22.431212+0000) 2022-09-10T09:20:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:22 smithi104 ceph-mon[97044]: pgmap v9317: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:20:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:22.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:22.942797+0000) 2022-09-10T09:20:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:22 smithi110 ceph-mon[97633]: pgmap v9317: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:20:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:23.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:23.431468+0000) 2022-09-10T09:20:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:23.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:23.943032+0000) 2022-09-10T09:20:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:24.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:24.431670+0000) 2022-09-10T09:20:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:24 smithi104 ceph-mon[97044]: pgmap v9318: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:20:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:24.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:24.943172+0000) 2022-09-10T09:20:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:24 smithi110 ceph-mon[97633]: pgmap v9318: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:20:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:25.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:25.431877+0000) 2022-09-10T09:20:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:25.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:25.943393+0000) 2022-09-10T09:20:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:26.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:26.432130+0000) 2022-09-10T09:20:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:26 smithi104 ceph-mon[97044]: pgmap v9319: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:20:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:26.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:26.943572+0000) 2022-09-10T09:20:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:26 smithi110 ceph-mon[97633]: pgmap v9319: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:20:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:27.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:27.432315+0000) 2022-09-10T09:20:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:27.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:27.943814+0000) 2022-09-10T09:20:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:28.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:28.432586+0000) 2022-09-10T09:20:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:28 smithi104 ceph-mon[97044]: pgmap v9320: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:20:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:28.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:28.944073+0000) 2022-09-10T09:20:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:28 smithi110 ceph-mon[97633]: pgmap v9320: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:20:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:29.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:29.432814+0000) 2022-09-10T09:20:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:29.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:29.944341+0000) 2022-09-10T09:20:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:30.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:30.433041+0000) 2022-09-10T09:20:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:30 smithi104 ceph-mon[97044]: pgmap v9321: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:30.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:30.944547+0000) 2022-09-10T09:20:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:30 smithi110 ceph-mon[97633]: pgmap v9321: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:31.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:31.433236+0000) 2022-09-10T09:20:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:31.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:31.944773+0000) 2022-09-10T09:20:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:32.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:32.433437+0000) 2022-09-10T09:20:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:32 smithi104 ceph-mon[97044]: pgmap v9322: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:32.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:32.944961+0000) 2022-09-10T09:20:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:32 smithi110 ceph-mon[97633]: pgmap v9322: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:33.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:33.433695+0000) 2022-09-10T09:20:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:33.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:33.945155+0000) 2022-09-10T09:20:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:34.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:34.433960+0000) 2022-09-10T09:20:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:34 smithi104 ceph-mon[97044]: pgmap v9323: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:34.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:34.945383+0000) 2022-09-10T09:20:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:34 smithi110 ceph-mon[97633]: pgmap v9323: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:35.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:35.434171+0000) 2022-09-10T09:20:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:35.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:35.945621+0000) 2022-09-10T09:20:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:36.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:36.434407+0000) 2022-09-10T09:20:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:20:36.381+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:20:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:36 smithi104 ceph-mon[97044]: pgmap v9324: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:20:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:36.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:36.945854+0000) 2022-09-10T09:20:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:36 smithi110 ceph-mon[97633]: pgmap v9324: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:20:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:37.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:37.434613+0000) 2022-09-10T09:20:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:37.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:37.946171+0000) 2022-09-10T09:20:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:38.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:38.434836+0000) 2022-09-10T09:20:39.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:38 smithi104 ceph-mon[97044]: pgmap v9325: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:38.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:38.946364+0000) 2022-09-10T09:20:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:38 smithi110 ceph-mon[97633]: pgmap v9325: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:39.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:39.435069+0000) 2022-09-10T09:20:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:39.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:39.946566+0000) 2022-09-10T09:20:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:40.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:40.435286+0000) 2022-09-10T09:20:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:41 smithi104 ceph-mon[97044]: pgmap v9326: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:40.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:40.946790+0000) 2022-09-10T09:20:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:41 smithi110 ceph-mon[97633]: pgmap v9326: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:41.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:41.435528+0000) 2022-09-10T09:20:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:41.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:41.947000+0000) 2022-09-10T09:20:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:42.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:42.435817+0000) 2022-09-10T09:20:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:43 smithi104 ceph-mon[97044]: pgmap v9327: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:42.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:42.947209+0000) 2022-09-10T09:20:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:43 smithi110 ceph-mon[97633]: pgmap v9327: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:43.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:43.436048+0000) 2022-09-10T09:20:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:43.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:43.947454+0000) 2022-09-10T09:20:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:44.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:44.436282+0000) 2022-09-10T09:20:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:45 smithi104 ceph-mon[97044]: pgmap v9328: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:20:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:20:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:20:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:20:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:44.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:44.947621+0000) 2022-09-10T09:20:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:45 smithi110 ceph-mon[97633]: pgmap v9328: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:20:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:20:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:20:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:20:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:45.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:45.436502+0000) 2022-09-10T09:20:46.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:45.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:45.947789+0000) 2022-09-10T09:20:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:46.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:46.436686+0000) 2022-09-10T09:20:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:47 smithi104 ceph-mon[97044]: pgmap v9329: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:20:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:46.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:46.948051+0000) 2022-09-10T09:20:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:47 smithi110 ceph-mon[97633]: pgmap v9329: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:20:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:47.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:47.436944+0000) 2022-09-10T09:20:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:47.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:47.948232+0000) 2022-09-10T09:20:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:48.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:48.437210+0000) 2022-09-10T09:20:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:49 smithi104 ceph-mon[97044]: pgmap v9330: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:48.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:48.948407+0000) 2022-09-10T09:20:49.422 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:49 smithi110 ceph-mon[97633]: pgmap v9330: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:49.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:49.437466+0000) 2022-09-10T09:20:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:49.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:49.948639+0000) 2022-09-10T09:20:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:50.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:50.437671+0000) 2022-09-10T09:20:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:51 smithi104 ceph-mon[97044]: pgmap v9331: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:50.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:50.948897+0000) 2022-09-10T09:20:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:51 smithi110 ceph-mon[97633]: pgmap v9331: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:20:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:20:51.397+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:20:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:51.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:51.437893+0000) 2022-09-10T09:20:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:51.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:51.949154+0000) 2022-09-10T09:20:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:52.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:52.438183+0000) 2022-09-10T09:20:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:53 smithi104 ceph-mon[97044]: pgmap v9332: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:52.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:52.949379+0000) 2022-09-10T09:20:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:53 smithi110 ceph-mon[97633]: pgmap v9332: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:20:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:53.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:53.438395+0000) 2022-09-10T09:20:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:53.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:53.949619+0000) 2022-09-10T09:20:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:54.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:54.438617+0000) 2022-09-10T09:20:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:55 smithi104 ceph-mon[97044]: pgmap v9333: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:54.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:54.949863+0000) 2022-09-10T09:20:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:55 smithi110 ceph-mon[97633]: pgmap v9333: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:55.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:55.438859+0000) 2022-09-10T09:20:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:55.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:55.950148+0000) 2022-09-10T09:20:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:56.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:56.439119+0000) 2022-09-10T09:20:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:57 smithi104 ceph-mon[97044]: pgmap v9334: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:20:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:56.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:56.950371+0000) 2022-09-10T09:20:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:57 smithi110 ceph-mon[97633]: pgmap v9334: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:20:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:57.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:57.439347+0000) 2022-09-10T09:20:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:57.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:57.950535+0000) 2022-09-10T09:20:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:58.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:58.439493+0000) 2022-09-10T09:20:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:20:59 smithi104 ceph-mon[97044]: pgmap v9335: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:58.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:58.950760+0000) 2022-09-10T09:20:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:20:59 smithi110 ceph-mon[97633]: pgmap v9335: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:20:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:20:59.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:59.439652+0000) 2022-09-10T09:21:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:20:59.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:20:59.951032+0000) 2022-09-10T09:21:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:00.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:00.439847+0000) 2022-09-10T09:21:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:01 smithi104 ceph-mon[97044]: pgmap v9336: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:00.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:00.951148+0000) 2022-09-10T09:21:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:01 smithi110 ceph-mon[97633]: pgmap v9336: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:01.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:01.440086+0000) 2022-09-10T09:21:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:01.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:01.951354+0000) 2022-09-10T09:21:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:02.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:02.440348+0000) 2022-09-10T09:21:03.220 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:02.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:02.951612+0000) 2022-09-10T09:21:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:03 smithi104 ceph-mon[97044]: pgmap v9337: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:03 smithi110 ceph-mon[97633]: pgmap v9337: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:03.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:03.440558+0000) 2022-09-10T09:21:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:03.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:03.951837+0000) 2022-09-10T09:21:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:04.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:04.440823+0000) 2022-09-10T09:21:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:05 smithi104 ceph-mon[97044]: pgmap v9338: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:04.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:04.952071+0000) 2022-09-10T09:21:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:05 smithi110 ceph-mon[97633]: pgmap v9338: 129 pgs: 129 active+clean; 8.7 MiB data, 894 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:05.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:05.441016+0000) 2022-09-10T09:21:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:05.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:05.952334+0000) 2022-09-10T09:21:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:06.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:06.441261+0000) 2022-09-10T09:21:06.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:21:06.406+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:21:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:07 smithi104 ceph-mon[97044]: pgmap v9339: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:21:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:06.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:06.952534+0000) 2022-09-10T09:21:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:07 smithi110 ceph-mon[97633]: pgmap v9339: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:21:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:07.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:07.441406+0000) 2022-09-10T09:21:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:07.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:07.952765+0000) 2022-09-10T09:21:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:08.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:08.441642+0000) 2022-09-10T09:21:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:08.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:08.952976+0000) 2022-09-10T09:21:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:09 smithi110 ceph-mon[97633]: pgmap v9340: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:21:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:09 smithi110 ceph-mon[97633]: osdmap e664: 12 total, 12 up, 12 in 2022-09-10T09:21:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:09 smithi104 ceph-mon[97044]: pgmap v9340: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:21:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:09 smithi104 ceph-mon[97044]: osdmap e664: 12 total, 12 up, 12 in 2022-09-10T09:21:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:09.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:09.441854+0000) 2022-09-10T09:21:10.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:09.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:09.953213+0000) 2022-09-10T09:21:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:10.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:10.442071+0000) 2022-09-10T09:21:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:10.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:10.953464+0000) 2022-09-10T09:21:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:11 smithi110 ceph-mon[97633]: pgmap v9342: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:21:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:11 smithi104 ceph-mon[97044]: pgmap v9342: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:21:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:11.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:11.442274+0000) 2022-09-10T09:21:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:11.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:11.953667+0000) 2022-09-10T09:21:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:12.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:12.442522+0000) 2022-09-10T09:21:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:12.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:12.953874+0000) 2022-09-10T09:21:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:13 smithi110 ceph-mon[97633]: pgmap v9343: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:21:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:13 smithi104 ceph-mon[97044]: pgmap v9343: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:21:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:13.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:13.442743+0000) 2022-09-10T09:21:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:13.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:13.954128+0000) 2022-09-10T09:21:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:14.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:14.442992+0000) 2022-09-10T09:21:15.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:14.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:14.954382+0000) 2022-09-10T09:21:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:15 smithi110 ceph-mon[97633]: pgmap v9344: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:21:16.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:15.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:15.443213+0000) 2022-09-10T09:21:16.538 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:15 smithi104 ceph-mon[97044]: pgmap v9344: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:21:16.539 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:15.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:15.954671+0000) 2022-09-10T09:21:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:16.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:16.443416+0000) 2022-09-10T09:21:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:16.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:16.954921+0000) 2022-09-10T09:21:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:17 smithi110 ceph-mon[97633]: pgmap v9345: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:21:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:17 smithi110 ceph-mon[97633]: osdmap e665: 12 total, 12 up, 12 in 2022-09-10T09:21:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:17 smithi104 ceph-mon[97044]: pgmap v9345: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:21:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:17 smithi104 ceph-mon[97044]: osdmap e665: 12 total, 12 up, 12 in 2022-09-10T09:21:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:17.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:17.443595+0000) 2022-09-10T09:21:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:17.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:17.955194+0000) 2022-09-10T09:21:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:18.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:18.443777+0000) 2022-09-10T09:21:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:18.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:18.955357+0000) 2022-09-10T09:21:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:19 smithi110 ceph-mon[97633]: pgmap v9347: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T09:21:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:19 smithi104 ceph-mon[97044]: pgmap v9347: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T09:21:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:19.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:19.443952+0000) 2022-09-10T09:21:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:19.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:19.955567+0000) 2022-09-10T09:21:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:20.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:20.444197+0000) 2022-09-10T09:21:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:20.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:20.955764+0000) 2022-09-10T09:21:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:21 smithi110 ceph-mon[97633]: pgmap v9348: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:21:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:21 smithi104 ceph-mon[97044]: pgmap v9348: 129 pgs: 129 active+clean; 8.7 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:21:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:21.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:21.444341+0000) 2022-09-10T09:21:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:21:21.408+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:21:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:21.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:21.955991+0000) 2022-09-10T09:21:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:22.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:22.444551+0000) 2022-09-10T09:21:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:22.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:22.956214+0000) 2022-09-10T09:21:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:23 smithi110 ceph-mon[97633]: pgmap v9349: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:21:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:23 smithi104 ceph-mon[97044]: pgmap v9349: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:21:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:23.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:23.444784+0000) 2022-09-10T09:21:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:23.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:23.956456+0000) 2022-09-10T09:21:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:24.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:24.445043+0000) 2022-09-10T09:21:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:24.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:24.956698+0000) 2022-09-10T09:21:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:25 smithi110 ceph-mon[97633]: pgmap v9350: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:21:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:25 smithi104 ceph-mon[97044]: pgmap v9350: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:21:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:25.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:25.445242+0000) 2022-09-10T09:21:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:25.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:25.956982+0000) 2022-09-10T09:21:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:26.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:26.445443+0000) 2022-09-10T09:21:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:26.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:26.957209+0000) 2022-09-10T09:21:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:27 smithi110 ceph-mon[97633]: pgmap v9351: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:21:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:27 smithi104 ceph-mon[97044]: pgmap v9351: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:21:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:27.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:27.445645+0000) 2022-09-10T09:21:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:27.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:27.957464+0000) 2022-09-10T09:21:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:28.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:28.445962+0000) 2022-09-10T09:21:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:28.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:28.957606+0000) 2022-09-10T09:21:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:29 smithi110 ceph-mon[97633]: pgmap v9352: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:21:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:29 smithi104 ceph-mon[97044]: pgmap v9352: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:21:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:29.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:29.446211+0000) 2022-09-10T09:21:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:29.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:29.957843+0000) 2022-09-10T09:21:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:30.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:30.446406+0000) 2022-09-10T09:21:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:30.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:30.958123+0000) 2022-09-10T09:21:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:31 smithi110 ceph-mon[97633]: pgmap v9353: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:31 smithi104 ceph-mon[97044]: pgmap v9353: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:31.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:31.446651+0000) 2022-09-10T09:21:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:31.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:31.958400+0000) 2022-09-10T09:21:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:32.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:32.446889+0000) 2022-09-10T09:21:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:32.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:32.958686+0000) 2022-09-10T09:21:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:33 smithi110 ceph-mon[97633]: pgmap v9354: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:33 smithi104 ceph-mon[97044]: pgmap v9354: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:33.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:33.447161+0000) 2022-09-10T09:21:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:33.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:33.958905+0000) 2022-09-10T09:21:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:34.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:34.447332+0000) 2022-09-10T09:21:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:34.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:34.959165+0000) 2022-09-10T09:21:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:35 smithi110 ceph-mon[97633]: pgmap v9355: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:35 smithi104 ceph-mon[97044]: pgmap v9355: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:35.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:35.447530+0000) 2022-09-10T09:21:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:35.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:35.959298+0000) 2022-09-10T09:21:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:36.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:36.447720+0000) 2022-09-10T09:21:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:21:36.422+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:21:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:36.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:36.959521+0000) 2022-09-10T09:21:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:37 smithi110 ceph-mon[97633]: pgmap v9356: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:21:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:37 smithi104 ceph-mon[97044]: pgmap v9356: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:21:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:37.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:37.447955+0000) 2022-09-10T09:21:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:37.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:37.959812+0000) 2022-09-10T09:21:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:38.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:38.448210+0000) 2022-09-10T09:21:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:38.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:38.959976+0000) 2022-09-10T09:21:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:39 smithi110 ceph-mon[97633]: pgmap v9357: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:39 smithi104 ceph-mon[97044]: pgmap v9357: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:39.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:39.448370+0000) 2022-09-10T09:21:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:39.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:39.960232+0000) 2022-09-10T09:21:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:40.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:40.448599+0000) 2022-09-10T09:21:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:40.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:40.960475+0000) 2022-09-10T09:21:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:41 smithi110 ceph-mon[97633]: pgmap v9358: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:41 smithi104 ceph-mon[97044]: pgmap v9358: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:41.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:41.448858+0000) 2022-09-10T09:21:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:41.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:41.960666+0000) 2022-09-10T09:21:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:42.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:42.449168+0000) 2022-09-10T09:21:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:42.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:42.960908+0000) 2022-09-10T09:21:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:43 smithi110 ceph-mon[97633]: pgmap v9359: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:43 smithi104 ceph-mon[97044]: pgmap v9359: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:43.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:43.449395+0000) 2022-09-10T09:21:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:43.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:43.961119+0000) 2022-09-10T09:21:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:44.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:44.449598+0000) 2022-09-10T09:21:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:44.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:44.961345+0000) 2022-09-10T09:21:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:45 smithi110 ceph-mon[97633]: pgmap v9360: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:21:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:21:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:21:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:21:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:45 smithi104 ceph-mon[97044]: pgmap v9360: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:21:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:21:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:21:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:21:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:45.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:45.449857+0000) 2022-09-10T09:21:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:45.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:45.961601+0000) 2022-09-10T09:21:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:46.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:46.450140+0000) 2022-09-10T09:21:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:46.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:46.961901+0000) 2022-09-10T09:21:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:47 smithi110 ceph-mon[97633]: pgmap v9361: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:21:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:47 smithi104 ceph-mon[97044]: pgmap v9361: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:21:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:47.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:47.450313+0000) 2022-09-10T09:21:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:47.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:47.962147+0000) 2022-09-10T09:21:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:48.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:48.450552+0000) 2022-09-10T09:21:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:48.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:48.962434+0000) 2022-09-10T09:21:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:49 smithi110 ceph-mon[97633]: pgmap v9362: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:49 smithi104 ceph-mon[97044]: pgmap v9362: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:49.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:49.450776+0000) 2022-09-10T09:21:50.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:49.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:49.962599+0000) 2022-09-10T09:21:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:50.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:50.450998+0000) 2022-09-10T09:21:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:50.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:50.962871+0000) 2022-09-10T09:21:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:51 smithi110 ceph-mon[97633]: pgmap v9363: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:51 smithi104 ceph-mon[97044]: pgmap v9363: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:21:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:21:51.437+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:21:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:51.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:51.451272+0000) 2022-09-10T09:21:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:51.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:51.963184+0000) 2022-09-10T09:21:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:52.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:52.451522+0000) 2022-09-10T09:21:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:52.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:52.963402+0000) 2022-09-10T09:21:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:53 smithi110 ceph-mon[97633]: pgmap v9364: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:53 smithi104 ceph-mon[97044]: pgmap v9364: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:21:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:53.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:53.451729+0000) 2022-09-10T09:21:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:53.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:53.963624+0000) 2022-09-10T09:21:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:54.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:54.451973+0000) 2022-09-10T09:21:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:54.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:54.963857+0000) 2022-09-10T09:21:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:55 smithi110 ceph-mon[97633]: pgmap v9365: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:55 smithi104 ceph-mon[97044]: pgmap v9365: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:55.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:55.452161+0000) 2022-09-10T09:21:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:55.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:55.964156+0000) 2022-09-10T09:21:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:56.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:56.452360+0000) 2022-09-10T09:21:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:56.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:56.964344+0000) 2022-09-10T09:21:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:57 smithi110 ceph-mon[97633]: pgmap v9366: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:21:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:57 smithi104 ceph-mon[97044]: pgmap v9366: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:21:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:57.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:57.452593+0000) 2022-09-10T09:21:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:57.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:57.964539+0000) 2022-09-10T09:21:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:58.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:58.452858+0000) 2022-09-10T09:21:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:58.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:58.964733+0000) 2022-09-10T09:21:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:21:59 smithi110 ceph-mon[97633]: pgmap v9367: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:21:59 smithi104 ceph-mon[97044]: pgmap v9367: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:21:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:21:59.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:59.453087+0000) 2022-09-10T09:22:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:21:59.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:21:59.964957+0000) 2022-09-10T09:22:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:00.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:00.453382+0000) 2022-09-10T09:22:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:00.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:00.965166+0000) 2022-09-10T09:22:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:01 smithi110 ceph-mon[97633]: pgmap v9368: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:01 smithi104 ceph-mon[97044]: pgmap v9368: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:01.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:01.453571+0000) 2022-09-10T09:22:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:01.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:01.965323+0000) 2022-09-10T09:22:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:02.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:02.453782+0000) 2022-09-10T09:22:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:02.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:02.965558+0000) 2022-09-10T09:22:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:03 smithi110 ceph-mon[97633]: pgmap v9369: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:03 smithi104 ceph-mon[97044]: pgmap v9369: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:03.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:03.453988+0000) 2022-09-10T09:22:04.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:03.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:03.965752+0000) 2022-09-10T09:22:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:04.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:04.454170+0000) 2022-09-10T09:22:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:04.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:04.965987+0000) 2022-09-10T09:22:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:05 smithi110 ceph-mon[97633]: pgmap v9370: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:05 smithi104 ceph-mon[97044]: pgmap v9370: 129 pgs: 129 active+clean; 8.7 MiB data, 897 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:05.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:05.454381+0000) 2022-09-10T09:22:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:05.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:05.966212+0000) 2022-09-10T09:22:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:06.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:06.454549+0000) 2022-09-10T09:22:06.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:22:06.448+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:22:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:06.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:06.966451+0000) 2022-09-10T09:22:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:07 smithi110 ceph-mon[97633]: pgmap v9371: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:22:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:07 smithi104 ceph-mon[97044]: pgmap v9371: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:22:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:07.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:07.454759+0000) 2022-09-10T09:22:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:07.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:07.966656+0000) 2022-09-10T09:22:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:08.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:08.455018+0000) 2022-09-10T09:22:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:08.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:08.966857+0000) 2022-09-10T09:22:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:09 smithi110 ceph-mon[97633]: pgmap v9372: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:22:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:09 smithi110 ceph-mon[97633]: osdmap e666: 12 total, 12 up, 12 in 2022-09-10T09:22:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:09 smithi104 ceph-mon[97044]: pgmap v9372: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:22:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:09 smithi104 ceph-mon[97044]: osdmap e666: 12 total, 12 up, 12 in 2022-09-10T09:22:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:09.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:09.455311+0000) 2022-09-10T09:22:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:09.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:09.967077+0000) 2022-09-10T09:22:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:10.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:10.455514+0000) 2022-09-10T09:22:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:10.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:10.967330+0000) 2022-09-10T09:22:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:11 smithi110 ceph-mon[97633]: pgmap v9374: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:22:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:11 smithi104 ceph-mon[97044]: pgmap v9374: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:22:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:11.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:11.455694+0000) 2022-09-10T09:22:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:11.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:11.967506+0000) 2022-09-10T09:22:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:12.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:12.455943+0000) 2022-09-10T09:22:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:12.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:12.967716+0000) 2022-09-10T09:22:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:13 smithi110 ceph-mon[97633]: pgmap v9375: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:22:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:13 smithi104 ceph-mon[97044]: pgmap v9375: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:22:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:13.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:13.456164+0000) 2022-09-10T09:22:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:13.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:13.967968+0000) 2022-09-10T09:22:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:14.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:14.456417+0000) 2022-09-10T09:22:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:14.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:14.968207+0000) 2022-09-10T09:22:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:15 smithi110 ceph-mon[97633]: pgmap v9376: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:22:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:15 smithi104 ceph-mon[97044]: pgmap v9376: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:22:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:15.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:15.456653+0000) 2022-09-10T09:22:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:15.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:15.968359+0000) 2022-09-10T09:22:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:16.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:16.456871+0000) 2022-09-10T09:22:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:16.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:16.968554+0000) 2022-09-10T09:22:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:17 smithi110 ceph-mon[97633]: pgmap v9377: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:22:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:17 smithi110 ceph-mon[97633]: osdmap e667: 12 total, 12 up, 12 in 2022-09-10T09:22:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:17 smithi104 ceph-mon[97044]: pgmap v9377: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:22:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:17 smithi104 ceph-mon[97044]: osdmap e667: 12 total, 12 up, 12 in 2022-09-10T09:22:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:17.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:17.457143+0000) 2022-09-10T09:22:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:17.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:17.968705+0000) 2022-09-10T09:22:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:18.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:18.457358+0000) 2022-09-10T09:22:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:18.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:18.968865+0000) 2022-09-10T09:22:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:19 smithi104 ceph-mon[97044]: pgmap v9379: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T09:22:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:19 smithi110 ceph-mon[97633]: pgmap v9379: 129 pgs: 129 active+clean; 8.7 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T09:22:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:19.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:19.457575+0000) 2022-09-10T09:22:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:19.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:19.969047+0000) 2022-09-10T09:22:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:20.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:20.457840+0000) 2022-09-10T09:22:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:20.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:20.969277+0000) 2022-09-10T09:22:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:21 smithi104 ceph-mon[97044]: pgmap v9380: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:22:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:21 smithi110 ceph-mon[97633]: pgmap v9380: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:22:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:21.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:21.458036+0000) 2022-09-10T09:22:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:22:21.464+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:22:22.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:21.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:21.969527+0000) 2022-09-10T09:22:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:22.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:22.458284+0000) 2022-09-10T09:22:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:22.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:22.969767+0000) 2022-09-10T09:22:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:23 smithi104 ceph-mon[97044]: pgmap v9381: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:22:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:23 smithi110 ceph-mon[97633]: pgmap v9381: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:22:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:23.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:23.458465+0000) 2022-09-10T09:22:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:23.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:23.969994+0000) 2022-09-10T09:22:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:24.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:24.458591+0000) 2022-09-10T09:22:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:24.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:24.970198+0000) 2022-09-10T09:22:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:25 smithi104 ceph-mon[97044]: pgmap v9382: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:22:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:25 smithi110 ceph-mon[97633]: pgmap v9382: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:22:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:25.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:25.458810+0000) 2022-09-10T09:22:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:25.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:25.970390+0000) 2022-09-10T09:22:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:26.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:26.459051+0000) 2022-09-10T09:22:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:26.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:26.970587+0000) 2022-09-10T09:22:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:27 smithi104 ceph-mon[97044]: pgmap v9383: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:22:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:27 smithi110 ceph-mon[97633]: pgmap v9383: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:22:27.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:27.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:27.459282+0000) 2022-09-10T09:22:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:27.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:27.970801+0000) 2022-09-10T09:22:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:28.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:28.459519+0000) 2022-09-10T09:22:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:28.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:28.971034+0000) 2022-09-10T09:22:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:29 smithi104 ceph-mon[97044]: pgmap v9384: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:22:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:29 smithi110 ceph-mon[97633]: pgmap v9384: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:22:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:29.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:29.459702+0000) 2022-09-10T09:22:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:29.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:29.971288+0000) 2022-09-10T09:22:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:30.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:30.459903+0000) 2022-09-10T09:22:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:30.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:30.971544+0000) 2022-09-10T09:22:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:31 smithi104 ceph-mon[97044]: pgmap v9385: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:31 smithi110 ceph-mon[97633]: pgmap v9385: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:31.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:31.460083+0000) 2022-09-10T09:22:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:31.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:31.971746+0000) 2022-09-10T09:22:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:32.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:32.460269+0000) 2022-09-10T09:22:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:32.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:32.971962+0000) 2022-09-10T09:22:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:33 smithi104 ceph-mon[97044]: pgmap v9386: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:33 smithi110 ceph-mon[97633]: pgmap v9386: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:33.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:33.460453+0000) 2022-09-10T09:22:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:33.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:33.972201+0000) 2022-09-10T09:22:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:34.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:34.460687+0000) 2022-09-10T09:22:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:34.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:34.972363+0000) 2022-09-10T09:22:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:35 smithi104 ceph-mon[97044]: pgmap v9387: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:35 smithi110 ceph-mon[97633]: pgmap v9387: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:35.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:35.460896+0000) 2022-09-10T09:22:36.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:35.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:35.972546+0000) 2022-09-10T09:22:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:22:36.480+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:22:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:36.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:36.461090+0000) 2022-09-10T09:22:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:36.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:36.972721+0000) 2022-09-10T09:22:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:37 smithi104 ceph-mon[97044]: pgmap v9388: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:22:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:37 smithi110 ceph-mon[97633]: pgmap v9388: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:22:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:37.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:37.461348+0000) 2022-09-10T09:22:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:37.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:37.972970+0000) 2022-09-10T09:22:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:38.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:38.461579+0000) 2022-09-10T09:22:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:38.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:38.973198+0000) 2022-09-10T09:22:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:39 smithi104 ceph-mon[97044]: pgmap v9389: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:39 smithi110 ceph-mon[97633]: pgmap v9389: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:39.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:39.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:39.461793+0000) 2022-09-10T09:22:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:39.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:39.973429+0000) 2022-09-10T09:22:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:40.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:40.462050+0000) 2022-09-10T09:22:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:40.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:40.973569+0000) 2022-09-10T09:22:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:41 smithi104 ceph-mon[97044]: pgmap v9390: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:41 smithi110 ceph-mon[97633]: pgmap v9390: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:41.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:41.462294+0000) 2022-09-10T09:22:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:41.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:41.973804+0000) 2022-09-10T09:22:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:42.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:42.462489+0000) 2022-09-10T09:22:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:42.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:42.973996+0000) 2022-09-10T09:22:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:43 smithi104 ceph-mon[97044]: pgmap v9391: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:43 smithi110 ceph-mon[97633]: pgmap v9391: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:43.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:43.462682+0000) 2022-09-10T09:22:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:43.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:43.974238+0000) 2022-09-10T09:22:44.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:44.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:44.462902+0000) 2022-09-10T09:22:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:44.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:44.974421+0000) 2022-09-10T09:22:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:45 smithi104 ceph-mon[97044]: pgmap v9392: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:22:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:22:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:22:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:22:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:45 smithi110 ceph-mon[97633]: pgmap v9392: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:22:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:22:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:22:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:22:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:45.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:45.463077+0000) 2022-09-10T09:22:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:45.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:45.974652+0000) 2022-09-10T09:22:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:46.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:46.463285+0000) 2022-09-10T09:22:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:46.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:46.974931+0000) 2022-09-10T09:22:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:47 smithi104 ceph-mon[97044]: pgmap v9393: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:22:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:47 smithi110 ceph-mon[97633]: pgmap v9393: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:22:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:47.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:47.463467+0000) 2022-09-10T09:22:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:47.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:47.975212+0000) 2022-09-10T09:22:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:48.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:48.463695+0000) 2022-09-10T09:22:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:48.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:48.975438+0000) 2022-09-10T09:22:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:49 smithi104 ceph-mon[97044]: pgmap v9394: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:49 smithi110 ceph-mon[97633]: pgmap v9394: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:49.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:49.463961+0000) 2022-09-10T09:22:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:49.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:49.975662+0000) 2022-09-10T09:22:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:50.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:50.464177+0000) 2022-09-10T09:22:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:50.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:50.975860+0000) 2022-09-10T09:22:51.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:51 smithi104 ceph-mon[97044]: pgmap v9395: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:51.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:51 smithi110 ceph-mon[97633]: pgmap v9395: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:22:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:51.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:51.464371+0000) 2022-09-10T09:22:51.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:22:51.489+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:22:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:51.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:51.976061+0000) 2022-09-10T09:22:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:52.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:52.464619+0000) 2022-09-10T09:22:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:52.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:52.976254+0000) 2022-09-10T09:22:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:53 smithi104 ceph-mon[97044]: pgmap v9396: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:53 smithi110 ceph-mon[97633]: pgmap v9396: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:22:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:53.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:53.464851+0000) 2022-09-10T09:22:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:53.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:53.976474+0000) 2022-09-10T09:22:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:54.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:54.465112+0000) 2022-09-10T09:22:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:54.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:54.976706+0000) 2022-09-10T09:22:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:55 smithi104 ceph-mon[97044]: pgmap v9397: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:55 smithi110 ceph-mon[97633]: pgmap v9397: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:55.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:55.465347+0000) 2022-09-10T09:22:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:55.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:55.976902+0000) 2022-09-10T09:22:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:56.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:56.465629+0000) 2022-09-10T09:22:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:56.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:56.977153+0000) 2022-09-10T09:22:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:57 smithi104 ceph-mon[97044]: pgmap v9398: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:22:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:57 smithi110 ceph-mon[97633]: pgmap v9398: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:22:57.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:57.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:57.465850+0000) 2022-09-10T09:22:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:57.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:57.977338+0000) 2022-09-10T09:22:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:58.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:58.466131+0000) 2022-09-10T09:22:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:58.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:58.977508+0000) 2022-09-10T09:22:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:22:59 smithi104 ceph-mon[97044]: pgmap v9399: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:22:59 smithi110 ceph-mon[97633]: pgmap v9399: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:22:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:22:59.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:59.466317+0000) 2022-09-10T09:23:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:22:59.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:22:59.977676+0000) 2022-09-10T09:23:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:00.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:00.466477+0000) 2022-09-10T09:23:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:00.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:00.977812+0000) 2022-09-10T09:23:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:01 smithi104 ceph-mon[97044]: pgmap v9400: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:23:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:01 smithi110 ceph-mon[97633]: pgmap v9400: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:23:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:01.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:01.466746+0000) 2022-09-10T09:23:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:01.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:01.978023+0000) 2022-09-10T09:23:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:02.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:02.467016+0000) 2022-09-10T09:23:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:02.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:02.978308+0000) 2022-09-10T09:23:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:03 smithi104 ceph-mon[97044]: pgmap v9401: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:23:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:03 smithi110 ceph-mon[97633]: pgmap v9401: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:23:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:03.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:03.467226+0000) 2022-09-10T09:23:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:03.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:03.978536+0000) 2022-09-10T09:23:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:04.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:04.467410+0000) 2022-09-10T09:23:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:04.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:04.978736+0000) 2022-09-10T09:23:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:05 smithi104 ceph-mon[97044]: pgmap v9402: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:05 smithi110 ceph-mon[97633]: pgmap v9402: 129 pgs: 129 active+clean; 8.7 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:05.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:05.467619+0000) 2022-09-10T09:23:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:05.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:05.978960+0000) 2022-09-10T09:23:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:06.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:06.467861+0000) 2022-09-10T09:23:06.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:23:06.502+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:23:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:06.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:06.979226+0000) 2022-09-10T09:23:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:07 smithi104 ceph-mon[97044]: pgmap v9403: 129 pgs: 129 active+clean; 8.7 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:23:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:07 smithi110 ceph-mon[97633]: pgmap v9403: 129 pgs: 129 active+clean; 8.7 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:23:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:07.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:07.468045+0000) 2022-09-10T09:23:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:07.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:07.979500+0000) 2022-09-10T09:23:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:08.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:08.468449+0000) 2022-09-10T09:23:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:08.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:08.979718+0000) 2022-09-10T09:23:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:09 smithi104 ceph-mon[97044]: pgmap v9404: 129 pgs: 129 active+clean; 8.7 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:23:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:09 smithi104 ceph-mon[97044]: osdmap e668: 12 total, 12 up, 12 in 2022-09-10T09:23:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:09 smithi110 ceph-mon[97633]: pgmap v9404: 129 pgs: 129 active+clean; 8.7 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:23:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:09 smithi110 ceph-mon[97633]: osdmap e668: 12 total, 12 up, 12 in 2022-09-10T09:23:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:09.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:09.468585+0000) 2022-09-10T09:23:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:09.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:09.979961+0000) 2022-09-10T09:23:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:10.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:10.468759+0000) 2022-09-10T09:23:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:10.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:10.980200+0000) 2022-09-10T09:23:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:11 smithi104 ceph-mon[97044]: pgmap v9406: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:23:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:11 smithi110 ceph-mon[97633]: pgmap v9406: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:23:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:11.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:11.468997+0000) 2022-09-10T09:23:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:11.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:11.980448+0000) 2022-09-10T09:23:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:12.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:12.469237+0000) 2022-09-10T09:23:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:12.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:12.980623+0000) 2022-09-10T09:23:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:13 smithi104 ceph-mon[97044]: pgmap v9407: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:23:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:13 smithi110 ceph-mon[97633]: pgmap v9407: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:23:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:13.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:13.469399+0000) 2022-09-10T09:23:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:13.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:13.980871+0000) 2022-09-10T09:23:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:14.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:14.469626+0000) 2022-09-10T09:23:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:14.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:14.981066+0000) 2022-09-10T09:23:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:15 smithi104 ceph-mon[97044]: pgmap v9408: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:23:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:15 smithi110 ceph-mon[97633]: pgmap v9408: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:23:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:15.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:15.469805+0000) 2022-09-10T09:23:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:15.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:15.981339+0000) 2022-09-10T09:23:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:16.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:16.470033+0000) 2022-09-10T09:23:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:16.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:16.981558+0000) 2022-09-10T09:23:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:17 smithi104 ceph-mon[97044]: pgmap v9409: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:23:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:17 smithi104 ceph-mon[97044]: osdmap e669: 12 total, 12 up, 12 in 2022-09-10T09:23:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:17 smithi110 ceph-mon[97633]: pgmap v9409: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:23:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:17 smithi110 ceph-mon[97633]: osdmap e669: 12 total, 12 up, 12 in 2022-09-10T09:23:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:17.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:17.470289+0000) 2022-09-10T09:23:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:17.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:17.981745+0000) 2022-09-10T09:23:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:18.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:18.470468+0000) 2022-09-10T09:23:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:18.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:18.981983+0000) 2022-09-10T09:23:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:19 smithi104 ceph-mon[97044]: pgmap v9411: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T09:23:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:19 smithi110 ceph-mon[97633]: pgmap v9411: 129 pgs: 129 active+clean; 8.7 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T09:23:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:19.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:19.470677+0000) 2022-09-10T09:23:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:19.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:19.982183+0000) 2022-09-10T09:23:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:20.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:20.470883+0000) 2022-09-10T09:23:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:20.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:20.982332+0000) 2022-09-10T09:23:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:21 smithi104 ceph-mon[97044]: pgmap v9412: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:23:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:21 smithi110 ceph-mon[97633]: pgmap v9412: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:23:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:21.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:21.471087+0000) 2022-09-10T09:23:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:23:21.508+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:23:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:21.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:21.982528+0000) 2022-09-10T09:23:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:22.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:22.471352+0000) 2022-09-10T09:23:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:22.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:22.982729+0000) 2022-09-10T09:23:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:23 smithi104 ceph-mon[97044]: pgmap v9413: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:23:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:23 smithi110 ceph-mon[97633]: pgmap v9413: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:23:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:23.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:23.471614+0000) 2022-09-10T09:23:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:23.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:23.982925+0000) 2022-09-10T09:23:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:24.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:24.471770+0000) 2022-09-10T09:23:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:24.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:24.983185+0000) 2022-09-10T09:23:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:25 smithi104 ceph-mon[97044]: pgmap v9414: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:23:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:25 smithi110 ceph-mon[97633]: pgmap v9414: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:23:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:25.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:25.471987+0000) 2022-09-10T09:23:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:25.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:25.983329+0000) 2022-09-10T09:23:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:26.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:26.472229+0000) 2022-09-10T09:23:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:26.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:26.983532+0000) 2022-09-10T09:23:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:27 smithi104 ceph-mon[97044]: pgmap v9415: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T09:23:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:27 smithi110 ceph-mon[97633]: pgmap v9415: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T09:23:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:27.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:27.472499+0000) 2022-09-10T09:23:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:27.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:27.983730+0000) 2022-09-10T09:23:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:28.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:28.472779+0000) 2022-09-10T09:23:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:28.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:28.983966+0000) 2022-09-10T09:23:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:29 smithi104 ceph-mon[97044]: pgmap v9416: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T09:23:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:29 smithi110 ceph-mon[97633]: pgmap v9416: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T09:23:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:29.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:29.473046+0000) 2022-09-10T09:23:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:29.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:29.984216+0000) 2022-09-10T09:23:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:30.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:30.473240+0000) 2022-09-10T09:23:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:30.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:30.984393+0000) 2022-09-10T09:23:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:31 smithi104 ceph-mon[97044]: pgmap v9417: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:23:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:31 smithi110 ceph-mon[97633]: pgmap v9417: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:23:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:31.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:31.473473+0000) 2022-09-10T09:23:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:31.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:31.984585+0000) 2022-09-10T09:23:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:32.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:32.473626+0000) 2022-09-10T09:23:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:32.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:32.984774+0000) 2022-09-10T09:23:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:33 smithi104 ceph-mon[97044]: pgmap v9418: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-10T09:23:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:33 smithi110 ceph-mon[97633]: pgmap v9418: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-10T09:23:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:33.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:33.473890+0000) 2022-09-10T09:23:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:33.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:33.984917+0000) 2022-09-10T09:23:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:34.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:34.474170+0000) 2022-09-10T09:23:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:34.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:34.985188+0000) 2022-09-10T09:23:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:35 smithi104 ceph-mon[97044]: pgmap v9419: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:23:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:35 smithi110 ceph-mon[97633]: pgmap v9419: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:23:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:35.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:35.474336+0000) 2022-09-10T09:23:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:35.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:35.985384+0000) 2022-09-10T09:23:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:23:36.522+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:23:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:36.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:36.474584+0000) 2022-09-10T09:23:37.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:36.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:36.985591+0000) 2022-09-10T09:23:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:37 smithi104 ceph-mon[97044]: pgmap v9420: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:23:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:37 smithi110 ceph-mon[97633]: pgmap v9420: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:23:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:37.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:37.474793+0000) 2022-09-10T09:23:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:37.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:37.985770+0000) 2022-09-10T09:23:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:38.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:38.475035+0000) 2022-09-10T09:23:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:38.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:38.986011+0000) 2022-09-10T09:23:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:39 smithi104 ceph-mon[97044]: pgmap v9421: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:23:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:39 smithi110 ceph-mon[97633]: pgmap v9421: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:23:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:39.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:39.475305+0000) 2022-09-10T09:23:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:39.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:39.986171+0000) 2022-09-10T09:23:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:40.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:40.475531+0000) 2022-09-10T09:23:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:40.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:40.986419+0000) 2022-09-10T09:23:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:41 smithi104 ceph-mon[97044]: pgmap v9422: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:23:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:41 smithi110 ceph-mon[97633]: pgmap v9422: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T09:23:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:41.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:41.475749+0000) 2022-09-10T09:23:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:41.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:41.986648+0000) 2022-09-10T09:23:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:42.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:42.475960+0000) 2022-09-10T09:23:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:42.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:42.986834+0000) 2022-09-10T09:23:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:43 smithi104 ceph-mon[97044]: pgmap v9423: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:23:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:43 smithi110 ceph-mon[97633]: pgmap v9423: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:23:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:43.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:43.476181+0000) 2022-09-10T09:23:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:43.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:43.987043+0000) 2022-09-10T09:23:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:44.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:44.476369+0000) 2022-09-10T09:23:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:44.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:44.987276+0000) 2022-09-10T09:23:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:45 smithi104 ceph-mon[97044]: pgmap v9424: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:23:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:23:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:23:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:23:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:45 smithi110 ceph-mon[97633]: pgmap v9424: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:23:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:23:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:23:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:23:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:45.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:45.476609+0000) 2022-09-10T09:23:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:45.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:45.987438+0000) 2022-09-10T09:23:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:46.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:46.476841+0000) 2022-09-10T09:23:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:46.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:46.987697+0000) 2022-09-10T09:23:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:47 smithi104 ceph-mon[97044]: pgmap v9425: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:23:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:47 smithi110 ceph-mon[97633]: pgmap v9425: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:23:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:47.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:47.477075+0000) 2022-09-10T09:23:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:47.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:47.987875+0000) 2022-09-10T09:23:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:48.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:48.477235+0000) 2022-09-10T09:23:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:48.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:48.988117+0000) 2022-09-10T09:23:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:49 smithi110 ceph-mon[97633]: pgmap v9426: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:49 smithi104 ceph-mon[97044]: pgmap v9426: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:49.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:49.477367+0000) 2022-09-10T09:23:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:49.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:49.988362+0000) 2022-09-10T09:23:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:50.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:50.477526+0000) 2022-09-10T09:23:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:50.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:50.988567+0000) 2022-09-10T09:23:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:51 smithi110 ceph-mon[97633]: pgmap v9427: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:23:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:51 smithi104 ceph-mon[97044]: pgmap v9427: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:23:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:23:51.524+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:23:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:51.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:51.477802+0000) 2022-09-10T09:23:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:51.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:51.988744+0000) 2022-09-10T09:23:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:52.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:52.478002+0000) 2022-09-10T09:23:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:52.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:52.988995+0000) 2022-09-10T09:23:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:53 smithi110 ceph-mon[97633]: pgmap v9428: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:23:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:53 smithi104 ceph-mon[97044]: pgmap v9428: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:23:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:53.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:53.478212+0000) 2022-09-10T09:23:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:53.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:53.989171+0000) 2022-09-10T09:23:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:54.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:54.478454+0000) 2022-09-10T09:23:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:54.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:54.989351+0000) 2022-09-10T09:23:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:55 smithi110 ceph-mon[97633]: pgmap v9429: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:55 smithi104 ceph-mon[97044]: pgmap v9429: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:55.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:55.478633+0000) 2022-09-10T09:23:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:55.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:55.989562+0000) 2022-09-10T09:23:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:56.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:56.478859+0000) 2022-09-10T09:23:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:56.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:56.989710+0000) 2022-09-10T09:23:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:57 smithi110 ceph-mon[97633]: pgmap v9430: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:23:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:57 smithi104 ceph-mon[97044]: pgmap v9430: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:23:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:57.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:57.479148+0000) 2022-09-10T09:23:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:57.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:57.989980+0000) 2022-09-10T09:23:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:58.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:58.479310+0000) 2022-09-10T09:23:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:58.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:58.990212+0000) 2022-09-10T09:23:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:23:59 smithi110 ceph-mon[97633]: pgmap v9431: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:23:59 smithi104 ceph-mon[97044]: pgmap v9431: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:23:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:23:59.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:59.479437+0000) 2022-09-10T09:24:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:23:59.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:23:59.990441+0000) 2022-09-10T09:24:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:00.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:00.479575+0000) 2022-09-10T09:24:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:00.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:00.990667+0000) 2022-09-10T09:24:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:01 smithi110 ceph-mon[97633]: pgmap v9432: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:24:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:01 smithi104 ceph-mon[97044]: pgmap v9432: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:24:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:01.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:01.479797+0000) 2022-09-10T09:24:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:01.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:01.990900+0000) 2022-09-10T09:24:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:02.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:02.480002+0000) 2022-09-10T09:24:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:02.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:02.991081+0000) 2022-09-10T09:24:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:03 smithi110 ceph-mon[97633]: pgmap v9433: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:03 smithi104 ceph-mon[97044]: pgmap v9433: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:03.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:03.480171+0000) 2022-09-10T09:24:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:03.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:03.991298+0000) 2022-09-10T09:24:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:04.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:04.480365+0000) 2022-09-10T09:24:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:04.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:04.991512+0000) 2022-09-10T09:24:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:05 smithi110 ceph-mon[97633]: pgmap v9434: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:05 smithi104 ceph-mon[97044]: pgmap v9434: 129 pgs: 129 active+clean; 8.7 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:05.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:05.480633+0000) 2022-09-10T09:24:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:05.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:05.991676+0000) 2022-09-10T09:24:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:24:06.536+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:24:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:06.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:06.480870+0000) 2022-09-10T09:24:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:06.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:06.991934+0000) 2022-09-10T09:24:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:07 smithi110 ceph-mon[97633]: pgmap v9435: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:24:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:07 smithi104 ceph-mon[97044]: pgmap v9435: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:24:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:07.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:07.481068+0000) 2022-09-10T09:24:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:07.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:07.992179+0000) 2022-09-10T09:24:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:08.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:08.481322+0000) 2022-09-10T09:24:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:08.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:08.992408+0000) 2022-09-10T09:24:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:09 smithi110 ceph-mon[97633]: pgmap v9436: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:24:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:09 smithi110 ceph-mon[97633]: osdmap e670: 12 total, 12 up, 12 in 2022-09-10T09:24:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:09 smithi104 ceph-mon[97044]: pgmap v9436: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:24:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:09 smithi104 ceph-mon[97044]: osdmap e670: 12 total, 12 up, 12 in 2022-09-10T09:24:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:09.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:09.481583+0000) 2022-09-10T09:24:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:09.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:09.992637+0000) 2022-09-10T09:24:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:10.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:10.481759+0000) 2022-09-10T09:24:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:10.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:10.992854+0000) 2022-09-10T09:24:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:11 smithi110 ceph-mon[97633]: pgmap v9438: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:24:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:11 smithi104 ceph-mon[97044]: pgmap v9438: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:24:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:11.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:11.481968+0000) 2022-09-10T09:24:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:11.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:11.993129+0000) 2022-09-10T09:24:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:12.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:12.482188+0000) 2022-09-10T09:24:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:12 smithi110 ceph-mon[97633]: pgmap v9439: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:24:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:12 smithi104 ceph-mon[97044]: pgmap v9439: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:24:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:12.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:12.993360+0000) 2022-09-10T09:24:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:13.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:13.482361+0000) 2022-09-10T09:24:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:13.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:13.993578+0000) 2022-09-10T09:24:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:14.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:14.482538+0000) 2022-09-10T09:24:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:14 smithi104 ceph-mon[97044]: pgmap v9440: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:24:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:14 smithi110 ceph-mon[97633]: pgmap v9440: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:24:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:14.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:14.993787+0000) 2022-09-10T09:24:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:15.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:15.482802+0000) 2022-09-10T09:24:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:15.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:15.993966+0000) 2022-09-10T09:24:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:16.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:16.482986+0000) 2022-09-10T09:24:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:16 smithi104 ceph-mon[97044]: pgmap v9441: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:24:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:16 smithi104 ceph-mon[97044]: osdmap e671: 12 total, 12 up, 12 in 2022-09-10T09:24:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:16 smithi110 ceph-mon[97633]: pgmap v9441: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:24:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:16 smithi110 ceph-mon[97633]: osdmap e671: 12 total, 12 up, 12 in 2022-09-10T09:24:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:16.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:16.994230+0000) 2022-09-10T09:24:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:17.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:17.483210+0000) 2022-09-10T09:24:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:17.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:17.994429+0000) 2022-09-10T09:24:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:18.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:18.483426+0000) 2022-09-10T09:24:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:18 smithi104 ceph-mon[97044]: pgmap v9443: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 225 B/s wr, 11 op/s 2022-09-10T09:24:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:18 smithi110 ceph-mon[97633]: pgmap v9443: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 225 B/s wr, 11 op/s 2022-09-10T09:24:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:18.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:18.994603+0000) 2022-09-10T09:24:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:19.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:19.483684+0000) 2022-09-10T09:24:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:19.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:19.994828+0000) 2022-09-10T09:24:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:20.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:20.483882+0000) 2022-09-10T09:24:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:20 smithi104 ceph-mon[97044]: pgmap v9444: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:24:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:20 smithi110 ceph-mon[97633]: pgmap v9444: 129 pgs: 129 active+clean; 8.7 MiB data, 911 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:24:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:20.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:20.995142+0000) 2022-09-10T09:24:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:24:21.541+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:24:21.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:21.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:21.484207+0000) 2022-09-10T09:24:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:21.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:21.995379+0000) 2022-09-10T09:24:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:22.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:22.484452+0000) 2022-09-10T09:24:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:22 smithi104 ceph-mon[97044]: pgmap v9445: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:24:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:22 smithi110 ceph-mon[97633]: pgmap v9445: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:24:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:22.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:22.995568+0000) 2022-09-10T09:24:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:23.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:23.484652+0000) 2022-09-10T09:24:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:23.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:23.995709+0000) 2022-09-10T09:24:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:24.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:24.484895+0000) 2022-09-10T09:24:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:24 smithi104 ceph-mon[97044]: pgmap v9446: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:24:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:24 smithi110 ceph-mon[97633]: pgmap v9446: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:24:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:24.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:24.995924+0000) 2022-09-10T09:24:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:25.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:25.485158+0000) 2022-09-10T09:24:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:25.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:25.996189+0000) 2022-09-10T09:24:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:26.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:26.485374+0000) 2022-09-10T09:24:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:26 smithi104 ceph-mon[97044]: pgmap v9447: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:24:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:26 smithi110 ceph-mon[97633]: pgmap v9447: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:24:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:26.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:26.996381+0000) 2022-09-10T09:24:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:27.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:27.485594+0000) 2022-09-10T09:24:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:27.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:27.996572+0000) 2022-09-10T09:24:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:28.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:28.485773+0000) 2022-09-10T09:24:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:28 smithi104 ceph-mon[97044]: pgmap v9448: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:24:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:28 smithi110 ceph-mon[97633]: pgmap v9448: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:24:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:28.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:28.996739+0000) 2022-09-10T09:24:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:29.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:29.485987+0000) 2022-09-10T09:24:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:29.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:29.996988+0000) 2022-09-10T09:24:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:30.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:30.486141+0000) 2022-09-10T09:24:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:30 smithi104 ceph-mon[97044]: pgmap v9449: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:24:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:30 smithi110 ceph-mon[97633]: pgmap v9449: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:24:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:30.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:30.997167+0000) 2022-09-10T09:24:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:31.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:31.486277+0000) 2022-09-10T09:24:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:31.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:31.997361+0000) 2022-09-10T09:24:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:32.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:32.486474+0000) 2022-09-10T09:24:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:32 smithi104 ceph-mon[97044]: pgmap v9450: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:32 smithi110 ceph-mon[97633]: pgmap v9450: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:32.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:32.997539+0000) 2022-09-10T09:24:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:33.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:33.486677+0000) 2022-09-10T09:24:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:33.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:33.997718+0000) 2022-09-10T09:24:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:34.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:34.486910+0000) 2022-09-10T09:24:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:34 smithi104 ceph-mon[97044]: pgmap v9451: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:34 smithi110 ceph-mon[97633]: pgmap v9451: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:34.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:34.997884+0000) 2022-09-10T09:24:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:35.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:35.487174+0000) 2022-09-10T09:24:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:35.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:35.998123+0000) 2022-09-10T09:24:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:24:36.556+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:24:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:36.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:36.487388+0000) 2022-09-10T09:24:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:36 smithi104 ceph-mon[97044]: pgmap v9452: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:36 smithi110 ceph-mon[97633]: pgmap v9452: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:36.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:36.998356+0000) 2022-09-10T09:24:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:37.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:37.487614+0000) 2022-09-10T09:24:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:37.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:37.998582+0000) 2022-09-10T09:24:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:38.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:38.487862+0000) 2022-09-10T09:24:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:38 smithi104 ceph-mon[97044]: pgmap v9453: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:38 smithi110 ceph-mon[97633]: pgmap v9453: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:38.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:38.998742+0000) 2022-09-10T09:24:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:39.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:39.488130+0000) 2022-09-10T09:24:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:39.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:39.999023+0000) 2022-09-10T09:24:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:40.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:40.488359+0000) 2022-09-10T09:24:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:40 smithi104 ceph-mon[97044]: pgmap v9454: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:24:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:40 smithi110 ceph-mon[97633]: pgmap v9454: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T09:24:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:40.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:40.999200+0000) 2022-09-10T09:24:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:41.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:41.488538+0000) 2022-09-10T09:24:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:41.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:41.999446+0000) 2022-09-10T09:24:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:42.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:42.488745+0000) 2022-09-10T09:24:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:42 smithi104 ceph-mon[97044]: pgmap v9455: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:42 smithi110 ceph-mon[97633]: pgmap v9455: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:42.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:42.999695+0000) 2022-09-10T09:24:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:43.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:43.488993+0000) 2022-09-10T09:24:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:43.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:43.999898+0000) 2022-09-10T09:24:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:44.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:44.489173+0000) 2022-09-10T09:24:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:44 smithi104 ceph-mon[97044]: pgmap v9456: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:44 smithi110 ceph-mon[97633]: pgmap v9456: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:44.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:45.000087+0000) 2022-09-10T09:24:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:45.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:45.489396+0000) 2022-09-10T09:24:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:24:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:24:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:24:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:24:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:24:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:45.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:46.000306+0000) 2022-09-10T09:24:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:46.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:46.489606+0000) 2022-09-10T09:24:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:46 smithi104 ceph-mon[97044]: pgmap v9457: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:46 smithi110 ceph-mon[97633]: pgmap v9457: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:47.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:47.000573+0000) 2022-09-10T09:24:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:47.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:47.489865+0000) 2022-09-10T09:24:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:48.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:48.000762+0000) 2022-09-10T09:24:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:48.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:48.490138+0000) 2022-09-10T09:24:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:48 smithi104 ceph-mon[97044]: pgmap v9458: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:48 smithi110 ceph-mon[97633]: pgmap v9458: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:48.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:49.001011+0000) 2022-09-10T09:24:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:49.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:49.490363+0000) 2022-09-10T09:24:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:50.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:50.001173+0000) 2022-09-10T09:24:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:50.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:50.490613+0000) 2022-09-10T09:24:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:50 smithi104 ceph-mon[97044]: pgmap v9459: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:50 smithi110 ceph-mon[97633]: pgmap v9459: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:51.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:51.001431+0000) 2022-09-10T09:24:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:51.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:51.490815+0000) 2022-09-10T09:24:51.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:24:51.572+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:24:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:52.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:52.001702+0000) 2022-09-10T09:24:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:52.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:52.491064+0000) 2022-09-10T09:24:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:52 smithi104 ceph-mon[97044]: pgmap v9460: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:52 smithi110 ceph-mon[97633]: pgmap v9460: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T09:24:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:53.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:53.001928+0000) 2022-09-10T09:24:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:53.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:53.491316+0000) 2022-09-10T09:24:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:54.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:54.002153+0000) 2022-09-10T09:24:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:54.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:54.491528+0000) 2022-09-10T09:24:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:54 smithi104 ceph-mon[97044]: pgmap v9461: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:54 smithi110 ceph-mon[97633]: pgmap v9461: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:55.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:55.002327+0000) 2022-09-10T09:24:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:55.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:55.491774+0000) 2022-09-10T09:24:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:56.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:56.002580+0000) 2022-09-10T09:24:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:56.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:56.492008+0000) 2022-09-10T09:24:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:56 smithi104 ceph-mon[97044]: pgmap v9462: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:56 smithi110 ceph-mon[97633]: pgmap v9462: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:24:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:57.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:57.002889+0000) 2022-09-10T09:24:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:57.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:57.492216+0000) 2022-09-10T09:24:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:58.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:58.003117+0000) 2022-09-10T09:24:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:58.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:58.492425+0000) 2022-09-10T09:24:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:24:58 smithi104 ceph-mon[97044]: pgmap v9463: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:24:58 smithi110 ceph-mon[97633]: pgmap v9463: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:24:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:24:59.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:59.003289+0000) 2022-09-10T09:24:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:24:59.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:24:59.492648+0000) 2022-09-10T09:25:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:00.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:00.003501+0000) 2022-09-10T09:25:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:00.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:00.492804+0000) 2022-09-10T09:25:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:00 smithi104 ceph-mon[97044]: pgmap v9464: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:00 smithi110 ceph-mon[97633]: pgmap v9464: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:01.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:01.003663+0000) 2022-09-10T09:25:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:01.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:01.493008+0000) 2022-09-10T09:25:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:02.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:02.003925+0000) 2022-09-10T09:25:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:02.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:02.493258+0000) 2022-09-10T09:25:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:02 smithi104 ceph-mon[97044]: pgmap v9465: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:02 smithi110 ceph-mon[97633]: pgmap v9465: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:03.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:03.004135+0000) 2022-09-10T09:25:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:03.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:03.493484+0000) 2022-09-10T09:25:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:04.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:04.004341+0000) 2022-09-10T09:25:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:04.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:04.493685+0000) 2022-09-10T09:25:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:04 smithi104 ceph-mon[97044]: pgmap v9466: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:04 smithi110 ceph-mon[97633]: pgmap v9466: 129 pgs: 129 active+clean; 8.7 MiB data, 912 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:05.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:05.004521+0000) 2022-09-10T09:25:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:05.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:05.493878+0000) 2022-09-10T09:25:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:06.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:06.004699+0000) 2022-09-10T09:25:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:06.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:06.494054+0000) 2022-09-10T09:25:06.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:25:06.575+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:25:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:06 smithi104 ceph-mon[97044]: pgmap v9467: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:25:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:06 smithi110 ceph-mon[97633]: pgmap v9467: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:25:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:07.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:07.004947+0000) 2022-09-10T09:25:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:07.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:07.494212+0000) 2022-09-10T09:25:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:08.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:08.005151+0000) 2022-09-10T09:25:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:08.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:08.494434+0000) 2022-09-10T09:25:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:08 smithi104 ceph-mon[97044]: pgmap v9468: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:25:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:08 smithi104 ceph-mon[97044]: osdmap e672: 12 total, 12 up, 12 in 2022-09-10T09:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:08 smithi110 ceph-mon[97633]: pgmap v9468: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:08 smithi110 ceph-mon[97633]: osdmap e672: 12 total, 12 up, 12 in 2022-09-10T09:25:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:09.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:09.005365+0000) 2022-09-10T09:25:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:09.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:09.494629+0000) 2022-09-10T09:25:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:10.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:10.005541+0000) 2022-09-10T09:25:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:10.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:10.494855+0000) 2022-09-10T09:25:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:10 smithi104 ceph-mon[97044]: pgmap v9470: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:25:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:10 smithi110 ceph-mon[97633]: pgmap v9470: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:25:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:11.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:11.005711+0000) 2022-09-10T09:25:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:11.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:11.495055+0000) 2022-09-10T09:25:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:12.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:12.005964+0000) 2022-09-10T09:25:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:12.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:12.495280+0000) 2022-09-10T09:25:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:12 smithi104 ceph-mon[97044]: pgmap v9471: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:25:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:12 smithi110 ceph-mon[97633]: pgmap v9471: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:25:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:13.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:13.006212+0000) 2022-09-10T09:25:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:13.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:13.495470+0000) 2022-09-10T09:25:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:14.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:14.006419+0000) 2022-09-10T09:25:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:14.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:14.495608+0000) 2022-09-10T09:25:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:14 smithi104 ceph-mon[97044]: pgmap v9472: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:25:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:14 smithi110 ceph-mon[97633]: pgmap v9472: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:25:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:15.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:15.006657+0000) 2022-09-10T09:25:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:15.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:15.495810+0000) 2022-09-10T09:25:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:16.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:16.006908+0000) 2022-09-10T09:25:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:16.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:16.496001+0000) 2022-09-10T09:25:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:16 smithi104 ceph-mon[97044]: pgmap v9473: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:25:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:16 smithi104 ceph-mon[97044]: osdmap e673: 12 total, 12 up, 12 in 2022-09-10T09:25:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:16 smithi110 ceph-mon[97633]: pgmap v9473: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:25:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:16 smithi110 ceph-mon[97633]: osdmap e673: 12 total, 12 up, 12 in 2022-09-10T09:25:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:17.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:17.007189+0000) 2022-09-10T09:25:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:17.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:17.496231+0000) 2022-09-10T09:25:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:18.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:18.007370+0000) 2022-09-10T09:25:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:18.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:18.496391+0000) 2022-09-10T09:25:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:18 smithi104 ceph-mon[97044]: pgmap v9475: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T09:25:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:18 smithi110 ceph-mon[97633]: pgmap v9475: 129 pgs: 129 active+clean; 8.7 MiB data, 917 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T09:25:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:19.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:19.007612+0000) 2022-09-10T09:25:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:19.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:19.496647+0000) 2022-09-10T09:25:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:20.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:20.007765+0000) 2022-09-10T09:25:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:20.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:20.496856+0000) 2022-09-10T09:25:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:20 smithi104 ceph-mon[97044]: pgmap v9476: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:25:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:20 smithi110 ceph-mon[97633]: pgmap v9476: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:25:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:21.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:21.008008+0000) 2022-09-10T09:25:21.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:25:21.590+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:25:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:21.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:21.497015+0000) 2022-09-10T09:25:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:22.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:22.008153+0000) 2022-09-10T09:25:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:22.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:22.497249+0000) 2022-09-10T09:25:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:22 smithi104 ceph-mon[97044]: pgmap v9477: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:25:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:22 smithi110 ceph-mon[97633]: pgmap v9477: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:25:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:23.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:23.008368+0000) 2022-09-10T09:25:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:23.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:23.497500+0000) 2022-09-10T09:25:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:24.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:24.008563+0000) 2022-09-10T09:25:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:24.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:24.497707+0000) 2022-09-10T09:25:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:24 smithi104 ceph-mon[97044]: pgmap v9478: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:25:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:24 smithi110 ceph-mon[97633]: pgmap v9478: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:25:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:25.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:25.008816+0000) 2022-09-10T09:25:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:25.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:25.497952+0000) 2022-09-10T09:25:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:26.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:26.009043+0000) 2022-09-10T09:25:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:26.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:26.498251+0000) 2022-09-10T09:25:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:26 smithi104 ceph-mon[97044]: pgmap v9479: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:25:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:26 smithi110 ceph-mon[97633]: pgmap v9479: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:25:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:27.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:27.009190+0000) 2022-09-10T09:25:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:27.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:27.498509+0000) 2022-09-10T09:25:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:28.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:28.009368+0000) 2022-09-10T09:25:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:28.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:28.498699+0000) 2022-09-10T09:25:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:28 smithi104 ceph-mon[97044]: pgmap v9480: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:25:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:28 smithi110 ceph-mon[97633]: pgmap v9480: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:25:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:29.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:29.009528+0000) 2022-09-10T09:25:29.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:29.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:29.498928+0000) 2022-09-10T09:25:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:30.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:30.009759+0000) 2022-09-10T09:25:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:30.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:30.499154+0000) 2022-09-10T09:25:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:30 smithi104 ceph-mon[97044]: pgmap v9481: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:30 smithi110 ceph-mon[97633]: pgmap v9481: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:31.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:31.009973+0000) 2022-09-10T09:25:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:31.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:31.499339+0000) 2022-09-10T09:25:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:32.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:32.010211+0000) 2022-09-10T09:25:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:32.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:32.499574+0000) 2022-09-10T09:25:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:32 smithi104 ceph-mon[97044]: pgmap v9482: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:32 smithi110 ceph-mon[97633]: pgmap v9482: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:33.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:33.010433+0000) 2022-09-10T09:25:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:33.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:33.499824+0000) 2022-09-10T09:25:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:34.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:34.010657+0000) 2022-09-10T09:25:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:34.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:34.500074+0000) 2022-09-10T09:25:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:34 smithi104 ceph-mon[97044]: pgmap v9483: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:34 smithi110 ceph-mon[97633]: pgmap v9483: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:35.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:35.010844+0000) 2022-09-10T09:25:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:35.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:35.500355+0000) 2022-09-10T09:25:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:36.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:36.011081+0000) 2022-09-10T09:25:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:36.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:36.500619+0000) 2022-09-10T09:25:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:25:36.596+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:25:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:36 smithi110 ceph-mon[97633]: pgmap v9484: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:36 smithi104 ceph-mon[97044]: pgmap v9484: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:37.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:37.011360+0000) 2022-09-10T09:25:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:37.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:37.500847+0000) 2022-09-10T09:25:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:38.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:38.011552+0000) 2022-09-10T09:25:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:38.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:38.501068+0000) 2022-09-10T09:25:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:38 smithi110 ceph-mon[97633]: pgmap v9485: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:38 smithi104 ceph-mon[97044]: pgmap v9485: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:39.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:39.011708+0000) 2022-09-10T09:25:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:39.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:39.501275+0000) 2022-09-10T09:25:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:40.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:40.011981+0000) 2022-09-10T09:25:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:40.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:40.501491+0000) 2022-09-10T09:25:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:40 smithi110 ceph-mon[97633]: pgmap v9486: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:40 smithi104 ceph-mon[97044]: pgmap v9486: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:41.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:41.012245+0000) 2022-09-10T09:25:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:41.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:41.501615+0000) 2022-09-10T09:25:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:42.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:42.012487+0000) 2022-09-10T09:25:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:42.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:42.501835+0000) 2022-09-10T09:25:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:42 smithi110 ceph-mon[97633]: pgmap v9487: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:42 smithi104 ceph-mon[97044]: pgmap v9487: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:43.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:43.012689+0000) 2022-09-10T09:25:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:43.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:43.502078+0000) 2022-09-10T09:25:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:44.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:44.012906+0000) 2022-09-10T09:25:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:44.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:44.502317+0000) 2022-09-10T09:25:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:44 smithi110 ceph-mon[97633]: pgmap v9488: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:44 smithi104 ceph-mon[97044]: pgmap v9488: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:45.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:45.013150+0000) 2022-09-10T09:25:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:45.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:45.502556+0000) 2022-09-10T09:25:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:25:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:25:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:25:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:25:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:25:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:25:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:25:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:25:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:46.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:46.013380+0000) 2022-09-10T09:25:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:46.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:46.502755+0000) 2022-09-10T09:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:46 smithi110 ceph-mon[97633]: pgmap v9489: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:46 smithi104 ceph-mon[97044]: pgmap v9489: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:47.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:47.013620+0000) 2022-09-10T09:25:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:47.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:47.503006+0000) 2022-09-10T09:25:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:48.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:48.013876+0000) 2022-09-10T09:25:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:48.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:48.503204+0000) 2022-09-10T09:25:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:48 smithi110 ceph-mon[97633]: pgmap v9490: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:48 smithi104 ceph-mon[97044]: pgmap v9490: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:49.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:49.014139+0000) 2022-09-10T09:25:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:49.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:49.503432+0000) 2022-09-10T09:25:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:50.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:50.014339+0000) 2022-09-10T09:25:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:50.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:50.503627+0000) 2022-09-10T09:25:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:50 smithi110 ceph-mon[97633]: pgmap v9491: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:50 smithi104 ceph-mon[97044]: pgmap v9491: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:51.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:51.014530+0000) 2022-09-10T09:25:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:51.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:51.503887+0000) 2022-09-10T09:25:51.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:25:51.611+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:25:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:52.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:52.014793+0000) 2022-09-10T09:25:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:52.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:52.504209+0000) 2022-09-10T09:25:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:52 smithi110 ceph-mon[97633]: pgmap v9492: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:52 smithi104 ceph-mon[97044]: pgmap v9492: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:53.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:53.014982+0000) 2022-09-10T09:25:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:53.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:53.504436+0000) 2022-09-10T09:25:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:54.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:54.015209+0000) 2022-09-10T09:25:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:54.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:54.504649+0000) 2022-09-10T09:25:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:54 smithi110 ceph-mon[97633]: pgmap v9493: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:54 smithi104 ceph-mon[97044]: pgmap v9493: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:55.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:55.015466+0000) 2022-09-10T09:25:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:55.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:55.504893+0000) 2022-09-10T09:25:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:56.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:56.015712+0000) 2022-09-10T09:25:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:56.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:56.505183+0000) 2022-09-10T09:25:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:56 smithi110 ceph-mon[97633]: pgmap v9494: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:56 smithi104 ceph-mon[97044]: pgmap v9494: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:25:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:57.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:57.015896+0000) 2022-09-10T09:25:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:57.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:57.505378+0000) 2022-09-10T09:25:58.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:58.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:58.016156+0000) 2022-09-10T09:25:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:58.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:58.505593+0000) 2022-09-10T09:25:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:25:58 smithi110 ceph-mon[97633]: pgmap v9495: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:25:58 smithi104 ceph-mon[97044]: pgmap v9495: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:25:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:25:59.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:59.016380+0000) 2022-09-10T09:25:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:25:59.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:25:59.505857+0000) 2022-09-10T09:26:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:00.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:00.016607+0000) 2022-09-10T09:26:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:00.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:00.506047+0000) 2022-09-10T09:26:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:00 smithi110 ceph-mon[97633]: pgmap v9496: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:00 smithi104 ceph-mon[97044]: pgmap v9496: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:01.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:01.016869+0000) 2022-09-10T09:26:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:01.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:01.506277+0000) 2022-09-10T09:26:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:02.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:02.017008+0000) 2022-09-10T09:26:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:02.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:02.506523+0000) 2022-09-10T09:26:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:02 smithi110 ceph-mon[97633]: pgmap v9497: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:02 smithi104 ceph-mon[97044]: pgmap v9497: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:03.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:03.017215+0000) 2022-09-10T09:26:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:03.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:03.506734+0000) 2022-09-10T09:26:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:04.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:04.017444+0000) 2022-09-10T09:26:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:04.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:04.507006+0000) 2022-09-10T09:26:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:04 smithi110 ceph-mon[97633]: pgmap v9498: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:04 smithi104 ceph-mon[97044]: pgmap v9498: 129 pgs: 129 active+clean; 8.7 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:05.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:05.017687+0000) 2022-09-10T09:26:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:05.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:05.507247+0000) 2022-09-10T09:26:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:06.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:06.017902+0000) 2022-09-10T09:26:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:26:06.617+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:26:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:06.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:06.507451+0000) 2022-09-10T09:26:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:06 smithi110 ceph-mon[97633]: pgmap v9499: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:26:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:06 smithi104 ceph-mon[97044]: pgmap v9499: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:26:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:07.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:07.018208+0000) 2022-09-10T09:26:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:07.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:07.507662+0000) 2022-09-10T09:26:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:08.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:08.018392+0000) 2022-09-10T09:26:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:08.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:08.507963+0000) 2022-09-10T09:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:08 smithi110 ceph-mon[97633]: pgmap v9500: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:08 smithi110 ceph-mon[97633]: osdmap e674: 12 total, 12 up, 12 in 2022-09-10T09:26:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:08 smithi104 ceph-mon[97044]: pgmap v9500: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:26:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:08 smithi104 ceph-mon[97044]: osdmap e674: 12 total, 12 up, 12 in 2022-09-10T09:26:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:09.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:09.018645+0000) 2022-09-10T09:26:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:09.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:09.508209+0000) 2022-09-10T09:26:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:10.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:10.018826+0000) 2022-09-10T09:26:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:10.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:10.508376+0000) 2022-09-10T09:26:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:10 smithi110 ceph-mon[97633]: pgmap v9502: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:26:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:10 smithi104 ceph-mon[97044]: pgmap v9502: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:26:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:11.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:11.019108+0000) 2022-09-10T09:26:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:11.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:11.508548+0000) 2022-09-10T09:26:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:12.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:12.019388+0000) 2022-09-10T09:26:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:12.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:12.508728+0000) 2022-09-10T09:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:12 smithi110 ceph-mon[97633]: pgmap v9503: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:26:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:12 smithi104 ceph-mon[97044]: pgmap v9503: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:26:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:13.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:13.019573+0000) 2022-09-10T09:26:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:13.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:13.508957+0000) 2022-09-10T09:26:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:14.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:14.019774+0000) 2022-09-10T09:26:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:14.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:14.509190+0000) 2022-09-10T09:26:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:14 smithi110 ceph-mon[97633]: pgmap v9504: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:26:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:14 smithi104 ceph-mon[97044]: pgmap v9504: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:26:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:15.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:15.019991+0000) 2022-09-10T09:26:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:15.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:15.509412+0000) 2022-09-10T09:26:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:16.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:16.020168+0000) 2022-09-10T09:26:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:16.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:16.509567+0000) 2022-09-10T09:26:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:16 smithi110 ceph-mon[97633]: pgmap v9505: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:26:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:16 smithi110 ceph-mon[97633]: osdmap e675: 12 total, 12 up, 12 in 2022-09-10T09:26:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:16 smithi104 ceph-mon[97044]: pgmap v9505: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T09:26:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:16 smithi104 ceph-mon[97044]: osdmap e675: 12 total, 12 up, 12 in 2022-09-10T09:26:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:17.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:17.020437+0000) 2022-09-10T09:26:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:17.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:17.509793+0000) 2022-09-10T09:26:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:18.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:18.020626+0000) 2022-09-10T09:26:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:18.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:18.510045+0000) 2022-09-10T09:26:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:18 smithi110 ceph-mon[97633]: pgmap v9507: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T09:26:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:18 smithi104 ceph-mon[97044]: pgmap v9507: 129 pgs: 129 active+clean; 8.7 MiB data, 919 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T09:26:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:19.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:19.020759+0000) 2022-09-10T09:26:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:19.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:19.510271+0000) 2022-09-10T09:26:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:20.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:20.020940+0000) 2022-09-10T09:26:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:20.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:20.510490+0000) 2022-09-10T09:26:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:20 smithi110 ceph-mon[97633]: pgmap v9508: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:26:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:20 smithi104 ceph-mon[97044]: pgmap v9508: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:26:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:21.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:21.021203+0000) 2022-09-10T09:26:21.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:26:21.630+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:26:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:21.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:21.510738+0000) 2022-09-10T09:26:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:22.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:22.021419+0000) 2022-09-10T09:26:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:22.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:22.510971+0000) 2022-09-10T09:26:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:22 smithi110 ceph-mon[97633]: pgmap v9509: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:26:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:22 smithi104 ceph-mon[97044]: pgmap v9509: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:26:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:23.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:23.021650+0000) 2022-09-10T09:26:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:23.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:23.511253+0000) 2022-09-10T09:26:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:24.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:24.021900+0000) 2022-09-10T09:26:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:24.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:24.511485+0000) 2022-09-10T09:26:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:24 smithi110 ceph-mon[97633]: pgmap v9510: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:26:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:24 smithi104 ceph-mon[97044]: pgmap v9510: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:26:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:25.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:25.022087+0000) 2022-09-10T09:26:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:25.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:25.511648+0000) 2022-09-10T09:26:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:26.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:26.022270+0000) 2022-09-10T09:26:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:26.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:26.511868+0000) 2022-09-10T09:26:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:26 smithi110 ceph-mon[97633]: pgmap v9511: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:26:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:26 smithi104 ceph-mon[97044]: pgmap v9511: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T09:26:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:27.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:27.022441+0000) 2022-09-10T09:26:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:27.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:27.512180+0000) 2022-09-10T09:26:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:28.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:28.022673+0000) 2022-09-10T09:26:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:28.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:28.512365+0000) 2022-09-10T09:26:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:28 smithi110 ceph-mon[97633]: pgmap v9512: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:26:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:28 smithi104 ceph-mon[97044]: pgmap v9512: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T09:26:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:29.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:29.022919+0000) 2022-09-10T09:26:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:29.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:29.512518+0000) 2022-09-10T09:26:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:30.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:30.023122+0000) 2022-09-10T09:26:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:30.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:30.512706+0000) 2022-09-10T09:26:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:30 smithi110 ceph-mon[97633]: pgmap v9513: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:30 smithi104 ceph-mon[97044]: pgmap v9513: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:31.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:31.023357+0000) 2022-09-10T09:26:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:31.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:31.512864+0000) 2022-09-10T09:26:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:32.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:32.023556+0000) 2022-09-10T09:26:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:32.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:32.513080+0000) 2022-09-10T09:26:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:32 smithi110 ceph-mon[97633]: pgmap v9514: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:32 smithi104 ceph-mon[97044]: pgmap v9514: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:33.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:33.023807+0000) 2022-09-10T09:26:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:33.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:33.513349+0000) 2022-09-10T09:26:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:34.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:34.024053+0000) 2022-09-10T09:26:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:34.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:34.513543+0000) 2022-09-10T09:26:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:34 smithi110 ceph-mon[97633]: pgmap v9515: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:34 smithi104 ceph-mon[97044]: pgmap v9515: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:35.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:35.024249+0000) 2022-09-10T09:26:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:35.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:35.513772+0000) 2022-09-10T09:26:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:36.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:36.024430+0000) 2022-09-10T09:26:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:36.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:36.514020+0000) 2022-09-10T09:26:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:26:36.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:26:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:36 smithi110 ceph-mon[97633]: pgmap v9516: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:36 smithi104 ceph-mon[97044]: pgmap v9516: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:37.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:37.024666+0000) 2022-09-10T09:26:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:37.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:37.514271+0000) 2022-09-10T09:26:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:38.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:38.024877+0000) 2022-09-10T09:26:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:38.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:38.514520+0000) 2022-09-10T09:26:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:38 smithi110 ceph-mon[97633]: pgmap v9517: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:38 smithi104 ceph-mon[97044]: pgmap v9517: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:39.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:39.025137+0000) 2022-09-10T09:26:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:39.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:39.514716+0000) 2022-09-10T09:26:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:40.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:40.025334+0000) 2022-09-10T09:26:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:40.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:40.514868+0000) 2022-09-10T09:26:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:40 smithi110 ceph-mon[97633]: pgmap v9518: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:40 smithi104 ceph-mon[97044]: pgmap v9518: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:41.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:41.025585+0000) 2022-09-10T09:26:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:41.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:41.515092+0000) 2022-09-10T09:26:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:42.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:42.025837+0000) 2022-09-10T09:26:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:42.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:42.515419+0000) 2022-09-10T09:26:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:42 smithi110 ceph-mon[97633]: pgmap v9519: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:42 smithi104 ceph-mon[97044]: pgmap v9519: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:43.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:43.026019+0000) 2022-09-10T09:26:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:43.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:43.515617+0000) 2022-09-10T09:26:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:44.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:44.026178+0000) 2022-09-10T09:26:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:44.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:44.515890+0000) 2022-09-10T09:26:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:44 smithi110 ceph-mon[97633]: pgmap v9520: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:44 smithi104 ceph-mon[97044]: pgmap v9520: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:45.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:45.026430+0000) 2022-09-10T09:26:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:45.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:45.516070+0000) 2022-09-10T09:26:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:26:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:26:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:26:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:26:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:46.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:46.026690+0000) 2022-09-10T09:26:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:26:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:26:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:26:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:26:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:46.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:46.516228+0000) 2022-09-10T09:26:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:46 smithi104 ceph-mon[97044]: pgmap v9521: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:47.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:47.026909+0000) 2022-09-10T09:26:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:46 smithi110 ceph-mon[97633]: pgmap v9521: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:47.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:47.516458+0000) 2022-09-10T09:26:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:48.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:48.027178+0000) 2022-09-10T09:26:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:48.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:48.516627+0000) 2022-09-10T09:26:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:48 smithi104 ceph-mon[97044]: pgmap v9522: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:26:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:49.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:49.027394+0000) 2022-09-10T09:26:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:48 smithi110 ceph-mon[97633]: pgmap v9522: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:26:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:49.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:49.516829+0000) 2022-09-10T09:26:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:50.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:50.027581+0000) 2022-09-10T09:26:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:50.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:50.516986+0000) 2022-09-10T09:26:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:50 smithi104 ceph-mon[97044]: pgmap v9523: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:51.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:51.027777+0000) 2022-09-10T09:26:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:50 smithi110 ceph-mon[97633]: pgmap v9523: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:51.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:51.517167+0000) 2022-09-10T09:26:51.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:26:51.658+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:26:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:52.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:52.028021+0000) 2022-09-10T09:26:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:52.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:52.517398+0000) 2022-09-10T09:26:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:52 smithi104 ceph-mon[97044]: pgmap v9524: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:26:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:53.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:53.028243+0000) 2022-09-10T09:26:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:52 smithi110 ceph-mon[97633]: pgmap v9524: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:26:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:53.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:53.517585+0000) 2022-09-10T09:26:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:54.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:54.028421+0000) 2022-09-10T09:26:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:54.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:54.517762+0000) 2022-09-10T09:26:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:54 smithi104 ceph-mon[97044]: pgmap v9525: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:26:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:55.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:55.028671+0000) 2022-09-10T09:26:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:54 smithi110 ceph-mon[97633]: pgmap v9525: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:26:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:55.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:55.517997+0000) 2022-09-10T09:26:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:56.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:56.028850+0000) 2022-09-10T09:26:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:56.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:56.518153+0000) 2022-09-10T09:26:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:56 smithi104 ceph-mon[97044]: pgmap v9526: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:57.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:57.029123+0000) 2022-09-10T09:26:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:56 smithi110 ceph-mon[97633]: pgmap v9526: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:26:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:57.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:57.518411+0000) 2022-09-10T09:26:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:58.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:58.029260+0000) 2022-09-10T09:26:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:58.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:58.518685+0000) 2022-09-10T09:26:59.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:26:58 smithi104 ceph-mon[97044]: pgmap v9527: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:26:59.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:59.029394+0000) 2022-09-10T09:26:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:26:58 smithi110 ceph-mon[97633]: pgmap v9527: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:26:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:26:59.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:26:59.518919+0000) 2022-09-10T09:27:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:00.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:00.029606+0000) 2022-09-10T09:27:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:00.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:00.519165+0000) 2022-09-10T09:27:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:00 smithi104 ceph-mon[97044]: pgmap v9528: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:01.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:01.029820+0000) 2022-09-10T09:27:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:00 smithi110 ceph-mon[97633]: pgmap v9528: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:01.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:01.519365+0000) 2022-09-10T09:27:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:02.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:02.030046+0000) 2022-09-10T09:27:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:02.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:02.519601+0000) 2022-09-10T09:27:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:02 smithi104 ceph-mon[97044]: pgmap v9529: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:03.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:03.030278+0000) 2022-09-10T09:27:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:02 smithi110 ceph-mon[97633]: pgmap v9529: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:03.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:03.519762+0000) 2022-09-10T09:27:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:04.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:04.030521+0000) 2022-09-10T09:27:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:04.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:04.520011+0000) 2022-09-10T09:27:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:04 smithi104 ceph-mon[97044]: pgmap v9530: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:05.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:05.030706+0000) 2022-09-10T09:27:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:04 smithi110 ceph-mon[97633]: pgmap v9530: 129 pgs: 129 active+clean; 8.7 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:05.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:05.520243+0000) 2022-09-10T09:27:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:06.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:06.030921+0000) 2022-09-10T09:27:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:06.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:06.520440+0000) 2022-09-10T09:27:06.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:27:06.673+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:27:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:06 smithi104 ceph-mon[97044]: pgmap v9531: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:27:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:07.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:07.031169+0000) 2022-09-10T09:27:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:06 smithi110 ceph-mon[97633]: pgmap v9531: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:27:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:07.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:07.520611+0000) 2022-09-10T09:27:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:08.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:08.031290+0000) 2022-09-10T09:27:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:08.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:08.520858+0000) 2022-09-10T09:27:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:08 smithi104 ceph-mon[97044]: pgmap v9532: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:27:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:08 smithi104 ceph-mon[97044]: osdmap e676: 12 total, 12 up, 12 in 2022-09-10T09:27:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:09.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:09.031495+0000) 2022-09-10T09:27:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:08 smithi110 ceph-mon[97633]: pgmap v9532: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:27:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:08 smithi110 ceph-mon[97633]: osdmap e676: 12 total, 12 up, 12 in 2022-09-10T09:27:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:09.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:09.521061+0000) 2022-09-10T09:27:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:10.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:10.031755+0000) 2022-09-10T09:27:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:10.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:10.521350+0000) 2022-09-10T09:27:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:10 smithi104 ceph-mon[97044]: pgmap v9534: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:27:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:11.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:11.032004+0000) 2022-09-10T09:27:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:10 smithi110 ceph-mon[97633]: pgmap v9534: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:27:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:11.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:11.521584+0000) 2022-09-10T09:27:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:12.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:12.032240+0000) 2022-09-10T09:27:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:12.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:12.521816+0000) 2022-09-10T09:27:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:12 smithi104 ceph-mon[97044]: pgmap v9535: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:27:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:12 smithi104 ceph-mon[97044]: osdmap e677: 12 total, 12 up, 12 in 2022-09-10T09:27:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:12 smithi110 ceph-mon[97633]: pgmap v9535: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:27:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:12 smithi110 ceph-mon[97633]: osdmap e677: 12 total, 12 up, 12 in 2022-09-10T09:27:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:13.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:13.032475+0000) 2022-09-10T09:27:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:13.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:13.522031+0000) 2022-09-10T09:27:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:14.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:14.032711+0000) 2022-09-10T09:27:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:14.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:14.522249+0000) 2022-09-10T09:27:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:14 smithi104 ceph-mon[97044]: pgmap v9537: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:27:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:14 smithi110 ceph-mon[97633]: pgmap v9537: 129 pgs: 129 active+clean; 8.7 MiB data, 921 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:27:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:15.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:15.032950+0000) 2022-09-10T09:27:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:15.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:15.522458+0000) 2022-09-10T09:27:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:16.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:16.033148+0000) 2022-09-10T09:27:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:16.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:16.522645+0000) 2022-09-10T09:27:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:16 smithi104 ceph-mon[97044]: pgmap v9538: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:27:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:16 smithi110 ceph-mon[97633]: pgmap v9538: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:27:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:17.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:17.033394+0000) 2022-09-10T09:27:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:17.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:17.522867+0000) 2022-09-10T09:27:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:18.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:18.033604+0000) 2022-09-10T09:27:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:18.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:18.523194+0000) 2022-09-10T09:27:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:18 smithi104 ceph-mon[97044]: pgmap v9539: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T09:27:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:18 smithi110 ceph-mon[97633]: pgmap v9539: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T09:27:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:19.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:19.033815+0000) 2022-09-10T09:27:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:19.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:19.523412+0000) 2022-09-10T09:27:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:20.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:20.034033+0000) 2022-09-10T09:27:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:20.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:20.523641+0000) 2022-09-10T09:27:21.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:20 smithi104 ceph-mon[97044]: pgmap v9540: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:27:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:20 smithi110 ceph-mon[97633]: pgmap v9540: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:27:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:21.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:21.034276+0000) 2022-09-10T09:27:21.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:27:21.677+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:27:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:21.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:21.523903+0000) 2022-09-10T09:27:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:22.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:22.034507+0000) 2022-09-10T09:27:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:22.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:22.524203+0000) 2022-09-10T09:27:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:22 smithi104 ceph-mon[97044]: pgmap v9541: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:27:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:22 smithi110 ceph-mon[97633]: pgmap v9541: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:27:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:23.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:23.034718+0000) 2022-09-10T09:27:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:23.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:23.524391+0000) 2022-09-10T09:27:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:24.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:24.034930+0000) 2022-09-10T09:27:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:24.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:24.524603+0000) 2022-09-10T09:27:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:24 smithi104 ceph-mon[97044]: pgmap v9542: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:27:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:24 smithi110 ceph-mon[97633]: pgmap v9542: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:27:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:25.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:25.035185+0000) 2022-09-10T09:27:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:25.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:25.524824+0000) 2022-09-10T09:27:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:26.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:26.035416+0000) 2022-09-10T09:27:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:26.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:26.525081+0000) 2022-09-10T09:27:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:26 smithi104 ceph-mon[97044]: pgmap v9543: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:27:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:26 smithi110 ceph-mon[97633]: pgmap v9543: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:27:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:27.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:27.035683+0000) 2022-09-10T09:27:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:27.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:27.525333+0000) 2022-09-10T09:27:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:28.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:28.035977+0000) 2022-09-10T09:27:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:28.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:28.525585+0000) 2022-09-10T09:27:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:28 smithi104 ceph-mon[97044]: pgmap v9544: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:28 smithi110 ceph-mon[97633]: pgmap v9544: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:29.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:29.036212+0000) 2022-09-10T09:27:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:29.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:29.525866+0000) 2022-09-10T09:27:30.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:30.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:30.036432+0000) 2022-09-10T09:27:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:30.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:30.526086+0000) 2022-09-10T09:27:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:30 smithi104 ceph-mon[97044]: pgmap v9545: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:30 smithi110 ceph-mon[97633]: pgmap v9545: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:31.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:31.036626+0000) 2022-09-10T09:27:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:31.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:31.526362+0000) 2022-09-10T09:27:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:32.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:32.036861+0000) 2022-09-10T09:27:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:32.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:32.526645+0000) 2022-09-10T09:27:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:32 smithi104 ceph-mon[97044]: pgmap v9546: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:32 smithi110 ceph-mon[97633]: pgmap v9546: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:33.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:33.037060+0000) 2022-09-10T09:27:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:33.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:33.526892+0000) 2022-09-10T09:27:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:34.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:34.037319+0000) 2022-09-10T09:27:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:34.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:34.527176+0000) 2022-09-10T09:27:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:34 smithi104 ceph-mon[97044]: pgmap v9547: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:34 smithi110 ceph-mon[97633]: pgmap v9547: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:35.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:35.037591+0000) 2022-09-10T09:27:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:35.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:35.527338+0000) 2022-09-10T09:27:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:36.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:36.037859+0000) 2022-09-10T09:27:36.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:36.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:36.527496+0000) 2022-09-10T09:27:36.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:27:36.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:27:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:36 smithi104 ceph-mon[97044]: pgmap v9548: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:36 smithi110 ceph-mon[97633]: pgmap v9548: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:37.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:37.038085+0000) 2022-09-10T09:27:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:37.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:37.527734+0000) 2022-09-10T09:27:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:38.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:38.038295+0000) 2022-09-10T09:27:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:38.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:38.527986+0000) 2022-09-10T09:27:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:38 smithi104 ceph-mon[97044]: pgmap v9549: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:38 smithi110 ceph-mon[97633]: pgmap v9549: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:39.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:39.038499+0000) 2022-09-10T09:27:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:39.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:39.528236+0000) 2022-09-10T09:27:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:40.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:40.038668+0000) 2022-09-10T09:27:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:40.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:40.528456+0000) 2022-09-10T09:27:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:40 smithi104 ceph-mon[97044]: pgmap v9550: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:40 smithi110 ceph-mon[97633]: pgmap v9550: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:41.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:41.038934+0000) 2022-09-10T09:27:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:41.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:41.528657+0000) 2022-09-10T09:27:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:42.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:42.039196+0000) 2022-09-10T09:27:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:42.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:42.528925+0000) 2022-09-10T09:27:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:42 smithi104 ceph-mon[97044]: pgmap v9551: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:42 smithi110 ceph-mon[97633]: pgmap v9551: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:43.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:43.039394+0000) 2022-09-10T09:27:43.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:43.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:43.529139+0000) 2022-09-10T09:27:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:44.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:44.039563+0000) 2022-09-10T09:27:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:44.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:44.529352+0000) 2022-09-10T09:27:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:44 smithi104 ceph-mon[97044]: pgmap v9552: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:44 smithi110 ceph-mon[97633]: pgmap v9552: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:45.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:45.039765+0000) 2022-09-10T09:27:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:45.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:45.529533+0000) 2022-09-10T09:27:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:27:46.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:27:46.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:27:46.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:27:46.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:27:46.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:46.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:46.039994+0000) 2022-09-10T09:27:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:46.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:46.529712+0000) 2022-09-10T09:27:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:46 smithi104 ceph-mon[97044]: pgmap v9553: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:46 smithi110 ceph-mon[97633]: pgmap v9553: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:47.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:47.040225+0000) 2022-09-10T09:27:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:47.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:47.529943+0000) 2022-09-10T09:27:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:48.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:48.040479+0000) 2022-09-10T09:27:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:48.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:48.530075+0000) 2022-09-10T09:27:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:48 smithi104 ceph-mon[97044]: pgmap v9554: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:48 smithi110 ceph-mon[97633]: pgmap v9554: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:27:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:49.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:49.040626+0000) 2022-09-10T09:27:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:49.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:49.530261+0000) 2022-09-10T09:27:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:50.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:50.040859+0000) 2022-09-10T09:27:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:50.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:50.530475+0000) 2022-09-10T09:27:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:50 smithi104 ceph-mon[97044]: pgmap v9555: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:50 smithi110 ceph-mon[97633]: pgmap v9555: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:51.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:51.041134+0000) 2022-09-10T09:27:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:51.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:51.530633+0000) 2022-09-10T09:27:51.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:27:51.687+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:27:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:52.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:52.041369+0000) 2022-09-10T09:27:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:52.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:52.530846+0000) 2022-09-10T09:27:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:52 smithi104 ceph-mon[97044]: pgmap v9556: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:27:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:52 smithi110 ceph-mon[97633]: pgmap v9556: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:27:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:53.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:53.041628+0000) 2022-09-10T09:27:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:53.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:53.531024+0000) 2022-09-10T09:27:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:54.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:54.041853+0000) 2022-09-10T09:27:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:54.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:54.531302+0000) 2022-09-10T09:27:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:54 smithi104 ceph-mon[97044]: pgmap v9557: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:27:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:54 smithi110 ceph-mon[97633]: pgmap v9557: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:27:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:55.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:55.042125+0000) 2022-09-10T09:27:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:55.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:55.531530+0000) 2022-09-10T09:27:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:56.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:56.042346+0000) 2022-09-10T09:27:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:56.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:56.531723+0000) 2022-09-10T09:27:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:56 smithi104 ceph-mon[97044]: pgmap v9558: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:56 smithi110 ceph-mon[97633]: pgmap v9558: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:27:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:57.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:57.042535+0000) 2022-09-10T09:27:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:57.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:57.531886+0000) 2022-09-10T09:27:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:58.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:58.042735+0000) 2022-09-10T09:27:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:58.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:58.532181+0000) 2022-09-10T09:27:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:27:58 smithi104 ceph-mon[97044]: pgmap v9559: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:27:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:27:58 smithi110 ceph-mon[97633]: pgmap v9559: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:27:59.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:27:59.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:59.042979+0000) 2022-09-10T09:27:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:27:59.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:27:59.532353+0000) 2022-09-10T09:28:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:00.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:00.043205+0000) 2022-09-10T09:28:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:00.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:00.532536+0000) 2022-09-10T09:28:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:00 smithi104 ceph-mon[97044]: pgmap v9560: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:00 smithi110 ceph-mon[97633]: pgmap v9560: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:01.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:01.043393+0000) 2022-09-10T09:28:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:01.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:01.532710+0000) 2022-09-10T09:28:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:02.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:02.043588+0000) 2022-09-10T09:28:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:02.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:02.532936+0000) 2022-09-10T09:28:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:02 smithi104 ceph-mon[97044]: pgmap v9561: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:02 smithi110 ceph-mon[97633]: pgmap v9561: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:03.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:03.043847+0000) 2022-09-10T09:28:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:03.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:03.533198+0000) 2022-09-10T09:28:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:04.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:04.044014+0000) 2022-09-10T09:28:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:04.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:04.533447+0000) 2022-09-10T09:28:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:04 smithi104 ceph-mon[97044]: pgmap v9562: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:04 smithi110 ceph-mon[97633]: pgmap v9562: 129 pgs: 129 active+clean; 8.7 MiB data, 922 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:05.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:05.044270+0000) 2022-09-10T09:28:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:05.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:05.533606+0000) 2022-09-10T09:28:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:06.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:06.044493+0000) 2022-09-10T09:28:06.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:28:06.700+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:28:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:06.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:06.533812+0000) 2022-09-10T09:28:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:06 smithi104 ceph-mon[97044]: pgmap v9563: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:28:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:06 smithi110 ceph-mon[97633]: pgmap v9563: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:28:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:07.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:07.044731+0000) 2022-09-10T09:28:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:07.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:07.534010+0000) 2022-09-10T09:28:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:08.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:08.045011+0000) 2022-09-10T09:28:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:08.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:08.534263+0000) 2022-09-10T09:28:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:08 smithi104 ceph-mon[97044]: pgmap v9564: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:28:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:08 smithi104 ceph-mon[97044]: osdmap e678: 12 total, 12 up, 12 in 2022-09-10T09:28:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:08 smithi110 ceph-mon[97633]: pgmap v9564: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:28:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:08 smithi110 ceph-mon[97633]: osdmap e678: 12 total, 12 up, 12 in 2022-09-10T09:28:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:09.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:09.045278+0000) 2022-09-10T09:28:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:09.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:09.534437+0000) 2022-09-10T09:28:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:10.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:10.045549+0000) 2022-09-10T09:28:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:10.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:10.534653+0000) 2022-09-10T09:28:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:10 smithi110 ceph-mon[97633]: pgmap v9566: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:28:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:10 smithi104 ceph-mon[97044]: pgmap v9566: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:28:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:11.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:11.045711+0000) 2022-09-10T09:28:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:11.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:11.534877+0000) 2022-09-10T09:28:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:12.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:12.045884+0000) 2022-09-10T09:28:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:12.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:12.535199+0000) 2022-09-10T09:28:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:12 smithi104 ceph-mon[97044]: pgmap v9567: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:28:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:12 smithi104 ceph-mon[97044]: osdmap e679: 12 total, 12 up, 12 in 2022-09-10T09:28:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:12 smithi110 ceph-mon[97633]: pgmap v9567: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:28:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:12 smithi110 ceph-mon[97633]: osdmap e679: 12 total, 12 up, 12 in 2022-09-10T09:28:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:13.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:13.046059+0000) 2022-09-10T09:28:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:13.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:13.535448+0000) 2022-09-10T09:28:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:14.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:14.046276+0000) 2022-09-10T09:28:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:14.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:14.535680+0000) 2022-09-10T09:28:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:14 smithi104 ceph-mon[97044]: pgmap v9569: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:28:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:14 smithi110 ceph-mon[97633]: pgmap v9569: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:28:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:15.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:15.046505+0000) 2022-09-10T09:28:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:15.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:15.535908+0000) 2022-09-10T09:28:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:16.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:16.046683+0000) 2022-09-10T09:28:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:16.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:16.536114+0000) 2022-09-10T09:28:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:16 smithi104 ceph-mon[97044]: pgmap v9570: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:28:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:16 smithi110 ceph-mon[97633]: pgmap v9570: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:28:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:17.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:17.046942+0000) 2022-09-10T09:28:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:17.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:17.536367+0000) 2022-09-10T09:28:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:18.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:18.047180+0000) 2022-09-10T09:28:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:18.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:18.536564+0000) 2022-09-10T09:28:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:18 smithi104 ceph-mon[97044]: pgmap v9571: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:28:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:18 smithi110 ceph-mon[97633]: pgmap v9571: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:28:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:19.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:19.047411+0000) 2022-09-10T09:28:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:19.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:19.536715+0000) 2022-09-10T09:28:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:20.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:20.047655+0000) 2022-09-10T09:28:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:20.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:20.536889+0000) 2022-09-10T09:28:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:20 smithi104 ceph-mon[97044]: pgmap v9572: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:28:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:20 smithi110 ceph-mon[97633]: pgmap v9572: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:28:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:21.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:21.047904+0000) 2022-09-10T09:28:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:21.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:21.537196+0000) 2022-09-10T09:28:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:28:21.712+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:28:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:22.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:22.048052+0000) 2022-09-10T09:28:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:22.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:22.537439+0000) 2022-09-10T09:28:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:22 smithi104 ceph-mon[97044]: pgmap v9573: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:28:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:22 smithi110 ceph-mon[97633]: pgmap v9573: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:28:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:23.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:23.048260+0000) 2022-09-10T09:28:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:23.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:23.537681+0000) 2022-09-10T09:28:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:24.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:24.048488+0000) 2022-09-10T09:28:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:24.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:24.537912+0000) 2022-09-10T09:28:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:24 smithi104 ceph-mon[97044]: pgmap v9574: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:28:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:24 smithi110 ceph-mon[97633]: pgmap v9574: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:28:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:25.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:25.048658+0000) 2022-09-10T09:28:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:25.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:25.538136+0000) 2022-09-10T09:28:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:26.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:26.048831+0000) 2022-09-10T09:28:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:26.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:26.538337+0000) 2022-09-10T09:28:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:26 smithi104 ceph-mon[97044]: pgmap v9575: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:28:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:26 smithi110 ceph-mon[97633]: pgmap v9575: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:28:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:27.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:27.049044+0000) 2022-09-10T09:28:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:27.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:27.538535+0000) 2022-09-10T09:28:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:28.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:28.049311+0000) 2022-09-10T09:28:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:28.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:28.538758+0000) 2022-09-10T09:28:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:28 smithi104 ceph-mon[97044]: pgmap v9576: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:28 smithi110 ceph-mon[97633]: pgmap v9576: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:29.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:29.049525+0000) 2022-09-10T09:28:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:29.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:29.539045+0000) 2022-09-10T09:28:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:30.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:30.049703+0000) 2022-09-10T09:28:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:30.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:30.539299+0000) 2022-09-10T09:28:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:30 smithi104 ceph-mon[97044]: pgmap v9577: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:30 smithi110 ceph-mon[97633]: pgmap v9577: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:31.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:31.049918+0000) 2022-09-10T09:28:31.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:31.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:31.539543+0000) 2022-09-10T09:28:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:32.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:32.050187+0000) 2022-09-10T09:28:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:32.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:32.539836+0000) 2022-09-10T09:28:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:32 smithi104 ceph-mon[97044]: pgmap v9578: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:32 smithi110 ceph-mon[97633]: pgmap v9578: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:33.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:33.050415+0000) 2022-09-10T09:28:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:33.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:33.540052+0000) 2022-09-10T09:28:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:34.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:34.050607+0000) 2022-09-10T09:28:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:34.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:34.540276+0000) 2022-09-10T09:28:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:34 smithi104 ceph-mon[97044]: pgmap v9579: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:34 smithi110 ceph-mon[97633]: pgmap v9579: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:35.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:35.050820+0000) 2022-09-10T09:28:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:35.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:35.540526+0000) 2022-09-10T09:28:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:36.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:36.051124+0000) 2022-09-10T09:28:36.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:28:36.719+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:28:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:36.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:36.540703+0000) 2022-09-10T09:28:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:36 smithi104 ceph-mon[97044]: pgmap v9580: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:36 smithi110 ceph-mon[97633]: pgmap v9580: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:37.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:37.051308+0000) 2022-09-10T09:28:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:37.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:37.540922+0000) 2022-09-10T09:28:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:38.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:38.051453+0000) 2022-09-10T09:28:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:38.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:38.541171+0000) 2022-09-10T09:28:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:38 smithi104 ceph-mon[97044]: pgmap v9581: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:38 smithi110 ceph-mon[97633]: pgmap v9581: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:39.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:39.051635+0000) 2022-09-10T09:28:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:39.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:39.541350+0000) 2022-09-10T09:28:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:40.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:40.051878+0000) 2022-09-10T09:28:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:40.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:40.541549+0000) 2022-09-10T09:28:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:40 smithi104 ceph-mon[97044]: pgmap v9582: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:40 smithi110 ceph-mon[97633]: pgmap v9582: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:41.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:41.052133+0000) 2022-09-10T09:28:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:41.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:41.541734+0000) 2022-09-10T09:28:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:42.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:42.052362+0000) 2022-09-10T09:28:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:42.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:42.541992+0000) 2022-09-10T09:28:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:42 smithi104 ceph-mon[97044]: pgmap v9583: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:42 smithi110 ceph-mon[97633]: pgmap v9583: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:43.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:43.052540+0000) 2022-09-10T09:28:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:43.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:43.542229+0000) 2022-09-10T09:28:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:44.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:44.052708+0000) 2022-09-10T09:28:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:44.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:44.542434+0000) 2022-09-10T09:28:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:44 smithi104 ceph-mon[97044]: pgmap v9584: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:44 smithi110 ceph-mon[97633]: pgmap v9584: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:45.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:45.052937+0000) 2022-09-10T09:28:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:45.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:45.542685+0000) 2022-09-10T09:28:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:28:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:46.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:46.053221+0000) 2022-09-10T09:28:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:46.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:46.542944+0000) 2022-09-10T09:28:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:46 smithi104 ceph-mon[97044]: pgmap v9585: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:46 smithi110 ceph-mon[97633]: pgmap v9585: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:47.544 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:47.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:47.053454+0000) 2022-09-10T09:28:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:47.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:47.543198+0000) 2022-09-10T09:28:48.544 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:48.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:48.053702+0000) 2022-09-10T09:28:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:48.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:48.543413+0000) 2022-09-10T09:28:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:48 smithi104 ceph-mon[97044]: pgmap v9586: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:28:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:48 smithi110 ceph-mon[97633]: pgmap v9586: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:28:49.544 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:49.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:49.053928+0000) 2022-09-10T09:28:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:49.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:49.543587+0000) 2022-09-10T09:28:50.544 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:50.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:50.054138+0000) 2022-09-10T09:28:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:50.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:50.543783+0000) 2022-09-10T09:28:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:50 smithi104 ceph-mon[97044]: pgmap v9587: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:50 smithi110 ceph-mon[97633]: pgmap v9587: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:51.544 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:51.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:51.054288+0000) 2022-09-10T09:28:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:51.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:51.543996+0000) 2022-09-10T09:28:51.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:28:51.735+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:28:52.545 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:52.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:52.054447+0000) 2022-09-10T09:28:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:52.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:52.544232+0000) 2022-09-10T09:28:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:52 smithi104 ceph-mon[97044]: pgmap v9588: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:28:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:52 smithi110 ceph-mon[97633]: pgmap v9588: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:28:53.545 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:53.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:53.054676+0000) 2022-09-10T09:28:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:53.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:53.544454+0000) 2022-09-10T09:28:54.545 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:54.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:54.054916+0000) 2022-09-10T09:28:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:54.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:54.544636+0000) 2022-09-10T09:28:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:54 smithi104 ceph-mon[97044]: pgmap v9589: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:28:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:54 smithi110 ceph-mon[97633]: pgmap v9589: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:28:55.545 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:55.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:55.055172+0000) 2022-09-10T09:28:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:55.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:55.544814+0000) 2022-09-10T09:28:56.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:56.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:56.055387+0000) 2022-09-10T09:28:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:56.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:56.545045+0000) 2022-09-10T09:28:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:56 smithi104 ceph-mon[97044]: pgmap v9590: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:56 smithi110 ceph-mon[97633]: pgmap v9590: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:28:57.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:57.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:57.055679+0000) 2022-09-10T09:28:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:57.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:57.545325+0000) 2022-09-10T09:28:58.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:58.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:58.055911+0000) 2022-09-10T09:28:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:58.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:58.545587+0000) 2022-09-10T09:28:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:28:58 smithi110 ceph-mon[97633]: pgmap v9591: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:28:58 smithi104 ceph-mon[97044]: pgmap v9591: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:28:59.547 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:28:59.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:59.056125+0000) 2022-09-10T09:28:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:28:59.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:28:59.545832+0000) 2022-09-10T09:29:00.547 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:00.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:00.056353+0000) 2022-09-10T09:29:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:00.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:00.546088+0000) 2022-09-10T09:29:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:00 smithi110 ceph-mon[97633]: pgmap v9592: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:00 smithi104 ceph-mon[97044]: pgmap v9592: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:01.547 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:01.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:01.056503+0000) 2022-09-10T09:29:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:01.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:01.546336+0000) 2022-09-10T09:29:02.547 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:02.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:02.056669+0000) 2022-09-10T09:29:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:02.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:02.546575+0000) 2022-09-10T09:29:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:02 smithi110 ceph-mon[97633]: pgmap v9593: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:03.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:02 smithi104 ceph-mon[97044]: pgmap v9593: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:03.547 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:03.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:03.056917+0000) 2022-09-10T09:29:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:03.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:03.546749+0000) 2022-09-10T09:29:04.550 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:04.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:04.057139+0000) 2022-09-10T09:29:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:04.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:04.546949+0000) 2022-09-10T09:29:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:04 smithi110 ceph-mon[97633]: pgmap v9594: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:04 smithi104 ceph-mon[97044]: pgmap v9594: 129 pgs: 129 active+clean; 8.7 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:05.548 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:05.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:05.057289+0000) 2022-09-10T09:29:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:05.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:05.547194+0000) 2022-09-10T09:29:06.548 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:06.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:06.057526+0000) 2022-09-10T09:29:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:06.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:06.547423+0000) 2022-09-10T09:29:06.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:29:06.750+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:29:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:06 smithi110 ceph-mon[97633]: pgmap v9595: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:29:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:06 smithi104 ceph-mon[97044]: pgmap v9595: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:29:07.548 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:07.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:07.057791+0000) 2022-09-10T09:29:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:07.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:07.547565+0000) 2022-09-10T09:29:08.548 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:08.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:08.058000+0000) 2022-09-10T09:29:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:08.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:08.547781+0000) 2022-09-10T09:29:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:08 smithi110 ceph-mon[97633]: pgmap v9596: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:29:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:08 smithi110 ceph-mon[97633]: osdmap e680: 12 total, 12 up, 12 in 2022-09-10T09:29:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:08 smithi104 ceph-mon[97044]: pgmap v9596: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:29:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:08 smithi104 ceph-mon[97044]: osdmap e680: 12 total, 12 up, 12 in 2022-09-10T09:29:09.549 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:09.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:09.058204+0000) 2022-09-10T09:29:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:09.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:09.548059+0000) 2022-09-10T09:29:10.549 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:10.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:10.058385+0000) 2022-09-10T09:29:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:10.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:10.548338+0000) 2022-09-10T09:29:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:10 smithi110 ceph-mon[97633]: pgmap v9598: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:29:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:10 smithi104 ceph-mon[97044]: pgmap v9598: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:29:11.549 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:11.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:11.058615+0000) 2022-09-10T09:29:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:11.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:11.548498+0000) 2022-09-10T09:29:12.549 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:12.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:12.058836+0000) 2022-09-10T09:29:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:12.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:12.548754+0000) 2022-09-10T09:29:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:12 smithi104 ceph-mon[97044]: pgmap v9599: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:29:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:12 smithi104 ceph-mon[97044]: osdmap e681: 12 total, 12 up, 12 in 2022-09-10T09:29:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:12 smithi110 ceph-mon[97633]: pgmap v9599: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:29:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:12 smithi110 ceph-mon[97633]: osdmap e681: 12 total, 12 up, 12 in 2022-09-10T09:29:13.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:13.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:13.059061+0000) 2022-09-10T09:29:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:13.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:13.548945+0000) 2022-09-10T09:29:14.550 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:14.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:14.059322+0000) 2022-09-10T09:29:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:14.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:14.549201+0000) 2022-09-10T09:29:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:14 smithi104 ceph-mon[97044]: pgmap v9601: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:29:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:14 smithi110 ceph-mon[97633]: pgmap v9601: 129 pgs: 129 active+clean; 8.7 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:29:15.550 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:15.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:15.059499+0000) 2022-09-10T09:29:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:15.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:15.549364+0000) 2022-09-10T09:29:16.550 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:16.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:16.059766+0000) 2022-09-10T09:29:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:16.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:16.549483+0000) 2022-09-10T09:29:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:16 smithi104 ceph-mon[97044]: pgmap v9602: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:29:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:16 smithi110 ceph-mon[97633]: pgmap v9602: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:29:17.550 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:17.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:17.060012+0000) 2022-09-10T09:29:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:17.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:17.549700+0000) 2022-09-10T09:29:18.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:18.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:18.060226+0000) 2022-09-10T09:29:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:18.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:18.549909+0000) 2022-09-10T09:29:19.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:18 smithi104 ceph-mon[97044]: pgmap v9603: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:29:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:18 smithi110 ceph-mon[97633]: pgmap v9603: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:29:19.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:19.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:19.060437+0000) 2022-09-10T09:29:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:19.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:19.550139+0000) 2022-09-10T09:29:20.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:20.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:20.060690+0000) 2022-09-10T09:29:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:20.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:20.550366+0000) 2022-09-10T09:29:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:20 smithi104 ceph-mon[97044]: pgmap v9604: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:29:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:20 smithi110 ceph-mon[97633]: pgmap v9604: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:29:21.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:21.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:21.060924+0000) 2022-09-10T09:29:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:21.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:21.550542+0000) 2022-09-10T09:29:21.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:29:21.766+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:29:22.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:22.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:22.061193+0000) 2022-09-10T09:29:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:22.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:22.550781+0000) 2022-09-10T09:29:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:22 smithi104 ceph-mon[97044]: pgmap v9605: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:29:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:22 smithi110 ceph-mon[97633]: pgmap v9605: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:29:23.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:23.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:23.061419+0000) 2022-09-10T09:29:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:23.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:23.551008+0000) 2022-09-10T09:29:24.552 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:24.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:24.061658+0000) 2022-09-10T09:29:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:24.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:24.551283+0000) 2022-09-10T09:29:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:24 smithi104 ceph-mon[97044]: pgmap v9606: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:29:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:24 smithi110 ceph-mon[97633]: pgmap v9606: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:29:25.552 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:25.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:25.061911+0000) 2022-09-10T09:29:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:25.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:25.551540+0000) 2022-09-10T09:29:26.552 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:26.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:26.062179+0000) 2022-09-10T09:29:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:26.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:26.551791+0000) 2022-09-10T09:29:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:26 smithi104 ceph-mon[97044]: pgmap v9607: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:29:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:26 smithi110 ceph-mon[97633]: pgmap v9607: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:29:27.552 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:27.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:27.062414+0000) 2022-09-10T09:29:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:27.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:27.551999+0000) 2022-09-10T09:29:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:28.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:28.062642+0000) 2022-09-10T09:29:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:28.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:28.552224+0000) 2022-09-10T09:29:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:28 smithi104 ceph-mon[97044]: pgmap v9608: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:28 smithi110 ceph-mon[97633]: pgmap v9608: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:29.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:29.062903+0000) 2022-09-10T09:29:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:29.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:29.552484+0000) 2022-09-10T09:29:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:30.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:30.063077+0000) 2022-09-10T09:29:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:30.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:30.552683+0000) 2022-09-10T09:29:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:30 smithi104 ceph-mon[97044]: pgmap v9609: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:30 smithi110 ceph-mon[97633]: pgmap v9609: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:31.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:31.063313+0000) 2022-09-10T09:29:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:31.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:31.552860+0000) 2022-09-10T09:29:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:32.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:32.063517+0000) 2022-09-10T09:29:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:32.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:32.553155+0000) 2022-09-10T09:29:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:32 smithi104 ceph-mon[97044]: pgmap v9610: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:32 smithi110 ceph-mon[97633]: pgmap v9610: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:33.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:33.063690+0000) 2022-09-10T09:29:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:33.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:33.553377+0000) 2022-09-10T09:29:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:34.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:34.063866+0000) 2022-09-10T09:29:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:34 smithi104 ceph-mon[97044]: pgmap v9611: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:34.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:34.553568+0000) 2022-09-10T09:29:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:34 smithi110 ceph-mon[97633]: pgmap v9611: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:35.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:35.064122+0000) 2022-09-10T09:29:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:35.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:35.553933+0000) 2022-09-10T09:29:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:36.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:36.064287+0000) 2022-09-10T09:29:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:36 smithi104 ceph-mon[97044]: pgmap v9612: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:36.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:36.554115+0000) 2022-09-10T09:29:37.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:29:36.781+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:29:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:36 smithi110 ceph-mon[97633]: pgmap v9612: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:37.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:37.064508+0000) 2022-09-10T09:29:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:37.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:37.554250+0000) 2022-09-10T09:29:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:38.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:38.064648+0000) 2022-09-10T09:29:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:38 smithi104 ceph-mon[97044]: pgmap v9613: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:38.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:38.554371+0000) 2022-09-10T09:29:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:38 smithi110 ceph-mon[97633]: pgmap v9613: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:39.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:39.064839+0000) 2022-09-10T09:29:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:39.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:39.554495+0000) 2022-09-10T09:29:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:40.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:40.065083+0000) 2022-09-10T09:29:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:40 smithi104 ceph-mon[97044]: pgmap v9614: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:40.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:40.554768+0000) 2022-09-10T09:29:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:40 smithi110 ceph-mon[97633]: pgmap v9614: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:41.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:41.065364+0000) 2022-09-10T09:29:41.876 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:41.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:41.554998+0000) 2022-09-10T09:29:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:42.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:42.065618+0000) 2022-09-10T09:29:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:42 smithi104 ceph-mon[97044]: pgmap v9615: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:42.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:42.555247+0000) 2022-09-10T09:29:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:42 smithi110 ceph-mon[97633]: pgmap v9615: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:43.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:43.065852+0000) 2022-09-10T09:29:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:43.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:43.555392+0000) 2022-09-10T09:29:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:44.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:44.066034+0000) 2022-09-10T09:29:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:44 smithi104 ceph-mon[97044]: pgmap v9616: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:44.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:44.555565+0000) 2022-09-10T09:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:44 smithi110 ceph-mon[97633]: pgmap v9616: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:45.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:45.066247+0000) 2022-09-10T09:29:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:29:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:29:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:29:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:45.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:45.555756+0000) 2022-09-10T09:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:29:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:46.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:46.066499+0000) 2022-09-10T09:29:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:46 smithi104 ceph-mon[97044]: pgmap v9617: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:46.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:46.555937+0000) 2022-09-10T09:29:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:46 smithi110 ceph-mon[97633]: pgmap v9617: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:47.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:47.066702+0000) 2022-09-10T09:29:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:47.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:47.556160+0000) 2022-09-10T09:29:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:48.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:48.066928+0000) 2022-09-10T09:29:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:48 smithi104 ceph-mon[97044]: pgmap v9618: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:48.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:48.556371+0000) 2022-09-10T09:29:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:48 smithi110 ceph-mon[97633]: pgmap v9618: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:49.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:49.067199+0000) 2022-09-10T09:29:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:49.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:49.556517+0000) 2022-09-10T09:29:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:50.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:50.067434+0000) 2022-09-10T09:29:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:50.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:50.556646+0000) 2022-09-10T09:29:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:50 smithi110 ceph-mon[97633]: pgmap v9619: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:50 smithi104 ceph-mon[97044]: pgmap v9619: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:51.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:51.067594+0000) 2022-09-10T09:29:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:51.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:51.556854+0000) 2022-09-10T09:29:52.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:29:51.797+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:29:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:52.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:52.067835+0000) 2022-09-10T09:29:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:52.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:52.557146+0000) 2022-09-10T09:29:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:52 smithi110 ceph-mon[97633]: pgmap v9620: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:52 smithi104 ceph-mon[97044]: pgmap v9620: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:53.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:53.068149+0000) 2022-09-10T09:29:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:53.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:53.557295+0000) 2022-09-10T09:29:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:54.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:54.068389+0000) 2022-09-10T09:29:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:54.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:54.557460+0000) 2022-09-10T09:29:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:54 smithi110 ceph-mon[97633]: pgmap v9621: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:54 smithi104 ceph-mon[97044]: pgmap v9621: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:55.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:55.068650+0000) 2022-09-10T09:29:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:55.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:55.557652+0000) 2022-09-10T09:29:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:56.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:56.068877+0000) 2022-09-10T09:29:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:56.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:56.557851+0000) 2022-09-10T09:29:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:56 smithi110 ceph-mon[97633]: pgmap v9622: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:56 smithi104 ceph-mon[97044]: pgmap v9622: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:29:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:57.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:57.069157+0000) 2022-09-10T09:29:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:57.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:57.558060+0000) 2022-09-10T09:29:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:58.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:58.069389+0000) 2022-09-10T09:29:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:58.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:58.558240+0000) 2022-09-10T09:29:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:29:58 smithi110 ceph-mon[97633]: pgmap v9623: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:29:58 smithi104 ceph-mon[97044]: pgmap v9623: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:29:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:29:59.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:59.069594+0000) 2022-09-10T09:30:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:29:59.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:29:59.558395+0000) 2022-09-10T09:30:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:00.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:00.069753+0000) 2022-09-10T09:30:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:00.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:00.558512+0000) 2022-09-10T09:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:00 smithi110 ceph-mon[97633]: pgmap v9624: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:30:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:00 smithi104 ceph-mon[97044]: pgmap v9624: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:30:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:01.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:01.069918+0000) 2022-09-10T09:30:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:01.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:01.558652+0000) 2022-09-10T09:30:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:02.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:02.070187+0000) 2022-09-10T09:30:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:02.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:02.558844+0000) 2022-09-10T09:30:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:02 smithi110 ceph-mon[97633]: pgmap v9625: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:02 smithi104 ceph-mon[97044]: pgmap v9625: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:03.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:03.070390+0000) 2022-09-10T09:30:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:03.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:03.559026+0000) 2022-09-10T09:30:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:04.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:04.070685+0000) 2022-09-10T09:30:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:04.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:04.559201+0000) 2022-09-10T09:30:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:04 smithi110 ceph-mon[97633]: pgmap v9626: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:04 smithi104 ceph-mon[97044]: pgmap v9626: 129 pgs: 129 active+clean; 8.7 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:05.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:05.070919+0000) 2022-09-10T09:30:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:05.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:05.559369+0000) 2022-09-10T09:30:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:06.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:06.071153+0000) 2022-09-10T09:30:06.814 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:06.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:06.559529+0000) 2022-09-10T09:30:07.072 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:30:06.813+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:30:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:06 smithi110 ceph-mon[97633]: pgmap v9627: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:30:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:06 smithi104 ceph-mon[97044]: pgmap v9627: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:30:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:07.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:07.071349+0000) 2022-09-10T09:30:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:07.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:07.559709+0000) 2022-09-10T09:30:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:08.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:08.071484+0000) 2022-09-10T09:30:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:08.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:08.559848+0000) 2022-09-10T09:30:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:08 smithi110 ceph-mon[97633]: pgmap v9628: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:30:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:08 smithi110 ceph-mon[97633]: osdmap e682: 12 total, 12 up, 12 in 2022-09-10T09:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:08 smithi104 ceph-mon[97044]: pgmap v9628: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:30:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:08 smithi104 ceph-mon[97044]: osdmap e682: 12 total, 12 up, 12 in 2022-09-10T09:30:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:09.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:09.071667+0000) 2022-09-10T09:30:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:09.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:09.560029+0000) 2022-09-10T09:30:10.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:10.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:10.071884+0000) 2022-09-10T09:30:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:10.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:10.560159+0000) 2022-09-10T09:30:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:10 smithi110 ceph-mon[97633]: pgmap v9630: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:30:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:10 smithi104 ceph-mon[97044]: pgmap v9630: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:30:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:11.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:11.072138+0000) 2022-09-10T09:30:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:11.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:11.560339+0000) 2022-09-10T09:30:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:12.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:12.072355+0000) 2022-09-10T09:30:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:12 smithi104 ceph-mon[97044]: pgmap v9631: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:30:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:12 smithi104 ceph-mon[97044]: osdmap e683: 12 total, 12 up, 12 in 2022-09-10T09:30:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:12.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:12.560470+0000) 2022-09-10T09:30:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:12 smithi110 ceph-mon[97633]: pgmap v9631: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:30:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:12 smithi110 ceph-mon[97633]: osdmap e683: 12 total, 12 up, 12 in 2022-09-10T09:30:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:13.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:13.072608+0000) 2022-09-10T09:30:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:13.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:13.560655+0000) 2022-09-10T09:30:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:14.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:14.072839+0000) 2022-09-10T09:30:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:14 smithi104 ceph-mon[97044]: pgmap v9633: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:30:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:14.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:14.560810+0000) 2022-09-10T09:30:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:14 smithi110 ceph-mon[97633]: pgmap v9633: 129 pgs: 129 active+clean; 8.7 MiB data, 928 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:30:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:15.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:15.073126+0000) 2022-09-10T09:30:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:15.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:15.560989+0000) 2022-09-10T09:30:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:16.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:16.073361+0000) 2022-09-10T09:30:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:16 smithi104 ceph-mon[97044]: pgmap v9634: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:30:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:16.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:16.561193+0000) 2022-09-10T09:30:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:16 smithi110 ceph-mon[97633]: pgmap v9634: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:30:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:17.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:17.073565+0000) 2022-09-10T09:30:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:17.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:17.561336+0000) 2022-09-10T09:30:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:18.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:18.073753+0000) 2022-09-10T09:30:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:18 smithi104 ceph-mon[97044]: pgmap v9635: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:30:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:18.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:18.561527+0000) 2022-09-10T09:30:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:18 smithi110 ceph-mon[97633]: pgmap v9635: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:30:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:19.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:19.073985+0000) 2022-09-10T09:30:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:19.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:19.561684+0000) 2022-09-10T09:30:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:20.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:20.074200+0000) 2022-09-10T09:30:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:20 smithi104 ceph-mon[97044]: pgmap v9636: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:30:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:20.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:20.561819+0000) 2022-09-10T09:30:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:20 smithi110 ceph-mon[97633]: pgmap v9636: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:30:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:21.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:21.074404+0000) 2022-09-10T09:30:21.829 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:21.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:21.561954+0000) 2022-09-10T09:30:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:22.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:22.074605+0000) 2022-09-10T09:30:22.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:30:21.827+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:30:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:22 smithi104 ceph-mon[97044]: pgmap v9637: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:30:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:22.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:22.562198+0000) 2022-09-10T09:30:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:22 smithi110 ceph-mon[97633]: pgmap v9637: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:30:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:23.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:23.074833+0000) 2022-09-10T09:30:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:23.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:23.562331+0000) 2022-09-10T09:30:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:24.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:24.075061+0000) 2022-09-10T09:30:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:24 smithi104 ceph-mon[97044]: pgmap v9638: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:30:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:24.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:24.562604+0000) 2022-09-10T09:30:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:24 smithi110 ceph-mon[97633]: pgmap v9638: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:30:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:25.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:25.075296+0000) 2022-09-10T09:30:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:25.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:25.562810+0000) 2022-09-10T09:30:26.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:26.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:26.075522+0000) 2022-09-10T09:30:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:26 smithi104 ceph-mon[97044]: pgmap v9639: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:30:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:26.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:26.563013+0000) 2022-09-10T09:30:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:26 smithi110 ceph-mon[97633]: pgmap v9639: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:30:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:27.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:27.075721+0000) 2022-09-10T09:30:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:27.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:27.563200+0000) 2022-09-10T09:30:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:28.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:28.075982+0000) 2022-09-10T09:30:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:28 smithi104 ceph-mon[97044]: pgmap v9640: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:28.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:28.563411+0000) 2022-09-10T09:30:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:28 smithi110 ceph-mon[97633]: pgmap v9640: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:29.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:29.076210+0000) 2022-09-10T09:30:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:29.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:29.563655+0000) 2022-09-10T09:30:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:30.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:30.076444+0000) 2022-09-10T09:30:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:30 smithi104 ceph-mon[97044]: pgmap v9641: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:30.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:30.563879+0000) 2022-09-10T09:30:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:30 smithi110 ceph-mon[97633]: pgmap v9641: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:31.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:31.076660+0000) 2022-09-10T09:30:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:31.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:31.564046+0000) 2022-09-10T09:30:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:32.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:32.076776+0000) 2022-09-10T09:30:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:32 smithi104 ceph-mon[97044]: pgmap v9642: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:32.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:32.564263+0000) 2022-09-10T09:30:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:32 smithi110 ceph-mon[97633]: pgmap v9642: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:33.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:33.076895+0000) 2022-09-10T09:30:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:33.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:33.564460+0000) 2022-09-10T09:30:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:34.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:34.077157+0000) 2022-09-10T09:30:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:34 smithi104 ceph-mon[97044]: pgmap v9643: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:34.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:34.564694+0000) 2022-09-10T09:30:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:34 smithi110 ceph-mon[97633]: pgmap v9643: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:35.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:35.077323+0000) 2022-09-10T09:30:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:35.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:35.564937+0000) 2022-09-10T09:30:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:36.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:36.077526+0000) 2022-09-10T09:30:36.832 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:36.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:36.565193+0000) 2022-09-10T09:30:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:36 smithi104 ceph-mon[97044]: pgmap v9644: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:36 smithi110 ceph-mon[97633]: pgmap v9644: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:37.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:37.077796+0000) 2022-09-10T09:30:37.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:30:36.830+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:30:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:37.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:37.565383+0000) 2022-09-10T09:30:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:38.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:38.078022+0000) 2022-09-10T09:30:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:38.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:38.565562+0000) 2022-09-10T09:30:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:38 smithi110 ceph-mon[97633]: pgmap v9645: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:38 smithi104 ceph-mon[97044]: pgmap v9645: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:39.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:39.078219+0000) 2022-09-10T09:30:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:39.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:39.565825+0000) 2022-09-10T09:30:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:40.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:40.078435+0000) 2022-09-10T09:30:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:40.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:40.566073+0000) 2022-09-10T09:30:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:40 smithi110 ceph-mon[97633]: pgmap v9646: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:40 smithi104 ceph-mon[97044]: pgmap v9646: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:41.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:41.078649+0000) 2022-09-10T09:30:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:41.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:41.566260+0000) 2022-09-10T09:30:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:42.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:42.078892+0000) 2022-09-10T09:30:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:42.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:42.566431+0000) 2022-09-10T09:30:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:42 smithi110 ceph-mon[97633]: pgmap v9647: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:42 smithi104 ceph-mon[97044]: pgmap v9647: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:43.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:43.079176+0000) 2022-09-10T09:30:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:43.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:43.566653+0000) 2022-09-10T09:30:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:44.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:44.079358+0000) 2022-09-10T09:30:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:44.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:44.566884+0000) 2022-09-10T09:30:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:44 smithi110 ceph-mon[97633]: pgmap v9648: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:44 smithi104 ceph-mon[97044]: pgmap v9648: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:45.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:45.079611+0000) 2022-09-10T09:30:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:45.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:45.567150+0000) 2022-09-10T09:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:30:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:30:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:30:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:30:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:30:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:46.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:46.079836+0000) 2022-09-10T09:30:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:46.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:46.567315+0000) 2022-09-10T09:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:46 smithi110 ceph-mon[97633]: pgmap v9649: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:46 smithi104 ceph-mon[97044]: pgmap v9649: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:47.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:47.080084+0000) 2022-09-10T09:30:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:47.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:47.567543+0000) 2022-09-10T09:30:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:48.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:48.080293+0000) 2022-09-10T09:30:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:48.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:48.567759+0000) 2022-09-10T09:30:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:48 smithi110 ceph-mon[97633]: pgmap v9650: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:48 smithi104 ceph-mon[97044]: pgmap v9650: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:49.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:49.080559+0000) 2022-09-10T09:30:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:49.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:49.567965+0000) 2022-09-10T09:30:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:50.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:50.080777+0000) 2022-09-10T09:30:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:50.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:50.568213+0000) 2022-09-10T09:30:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:50 smithi110 ceph-mon[97633]: pgmap v9651: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:50 smithi104 ceph-mon[97044]: pgmap v9651: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:51.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:51.081016+0000) 2022-09-10T09:30:51.847 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:51.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:51.568463+0000) 2022-09-10T09:30:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:52.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:52.081270+0000) 2022-09-10T09:30:52.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:30:51.845+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:30:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:52.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:52.568707+0000) 2022-09-10T09:30:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:52 smithi110 ceph-mon[97633]: pgmap v9652: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:52 smithi104 ceph-mon[97044]: pgmap v9652: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:53.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:53.081495+0000) 2022-09-10T09:30:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:53.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:53.568918+0000) 2022-09-10T09:30:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:54.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:54.081686+0000) 2022-09-10T09:30:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:54.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:54.569192+0000) 2022-09-10T09:30:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:54 smithi110 ceph-mon[97633]: pgmap v9653: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:54 smithi104 ceph-mon[97044]: pgmap v9653: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:55.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:55.081914+0000) 2022-09-10T09:30:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:55.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:55.569403+0000) 2022-09-10T09:30:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:56.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:56.082151+0000) 2022-09-10T09:30:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:56.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:56.569660+0000) 2022-09-10T09:30:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:56 smithi110 ceph-mon[97633]: pgmap v9654: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:56 smithi104 ceph-mon[97044]: pgmap v9654: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:30:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:57.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:57.082428+0000) 2022-09-10T09:30:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:57.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:57.569836+0000) 2022-09-10T09:30:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:58.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:58.082631+0000) 2022-09-10T09:30:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:58.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:58.570126+0000) 2022-09-10T09:30:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:30:58 smithi110 ceph-mon[97633]: pgmap v9655: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:30:58 smithi104 ceph-mon[97044]: pgmap v9655: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:30:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:30:59.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:59.082800+0000) 2022-09-10T09:31:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:30:59.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:30:59.570270+0000) 2022-09-10T09:31:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:00.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:00.083090+0000) 2022-09-10T09:31:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:00.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:00.570424+0000) 2022-09-10T09:31:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:00 smithi110 ceph-mon[97633]: pgmap v9656: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:00 smithi104 ceph-mon[97044]: pgmap v9656: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:01.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:01.083297+0000) 2022-09-10T09:31:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:01.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:01.570620+0000) 2022-09-10T09:31:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:02.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:02.083498+0000) 2022-09-10T09:31:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:02.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:02.570787+0000) 2022-09-10T09:31:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:02 smithi110 ceph-mon[97633]: pgmap v9657: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:02 smithi104 ceph-mon[97044]: pgmap v9657: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:03.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:03.083719+0000) 2022-09-10T09:31:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:03.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:03.571007+0000) 2022-09-10T09:31:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:04.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:04.083977+0000) 2022-09-10T09:31:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:04.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:04.571288+0000) 2022-09-10T09:31:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:04 smithi110 ceph-mon[97633]: pgmap v9658: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:04 smithi104 ceph-mon[97044]: pgmap v9658: 129 pgs: 129 active+clean; 8.7 MiB data, 929 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:05.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:05.084173+0000) 2022-09-10T09:31:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:05.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:05.571516+0000) 2022-09-10T09:31:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:06.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:06.084357+0000) 2022-09-10T09:31:06.857 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:06.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:06.571763+0000) 2022-09-10T09:31:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:06 smithi110 ceph-mon[97633]: pgmap v9659: 129 pgs: 129 active+clean; 8.7 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:31:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:06 smithi104 ceph-mon[97044]: pgmap v9659: 129 pgs: 129 active+clean; 8.7 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:31:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:07.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:07.084587+0000) 2022-09-10T09:31:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:31:06.855+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:31:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:07.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:07.571990+0000) 2022-09-10T09:31:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:08.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:08.084787+0000) 2022-09-10T09:31:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:08.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:08.572189+0000) 2022-09-10T09:31:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:08 smithi110 ceph-mon[97633]: pgmap v9660: 129 pgs: 129 active+clean; 8.7 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:31:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:08 smithi110 ceph-mon[97633]: osdmap e684: 12 total, 12 up, 12 in 2022-09-10T09:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:08 smithi104 ceph-mon[97044]: pgmap v9660: 129 pgs: 129 active+clean; 8.7 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:31:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:08 smithi104 ceph-mon[97044]: osdmap e684: 12 total, 12 up, 12 in 2022-09-10T09:31:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:09.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:09.084968+0000) 2022-09-10T09:31:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:09.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:09.572348+0000) 2022-09-10T09:31:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:10.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:10.085188+0000) 2022-09-10T09:31:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:10.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:10.572574+0000) 2022-09-10T09:31:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:10 smithi110 ceph-mon[97633]: pgmap v9662: 129 pgs: 129 active+clean; 8.7 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:31:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:10 smithi104 ceph-mon[97044]: pgmap v9662: 129 pgs: 129 active+clean; 8.7 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:31:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:11.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:11.085455+0000) 2022-09-10T09:31:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:11.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:11.572793+0000) 2022-09-10T09:31:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:12.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:12.085743+0000) 2022-09-10T09:31:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:12 smithi104 ceph-mon[97044]: pgmap v9663: 129 pgs: 129 active+clean; 8.7 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:31:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:12 smithi104 ceph-mon[97044]: osdmap e685: 12 total, 12 up, 12 in 2022-09-10T09:31:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:12.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:12.573022+0000) 2022-09-10T09:31:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:12 smithi110 ceph-mon[97633]: pgmap v9663: 129 pgs: 129 active+clean; 8.7 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:31:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:12 smithi110 ceph-mon[97633]: osdmap e685: 12 total, 12 up, 12 in 2022-09-10T09:31:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:13.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:13.085972+0000) 2022-09-10T09:31:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:13.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:13.573272+0000) 2022-09-10T09:31:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:14.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:14.086189+0000) 2022-09-10T09:31:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:14 smithi104 ceph-mon[97044]: pgmap v9665: 129 pgs: 129 active+clean; 8.7 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:31:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:14.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:14.573475+0000) 2022-09-10T09:31:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:14 smithi110 ceph-mon[97633]: pgmap v9665: 129 pgs: 129 active+clean; 8.7 MiB data, 938 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:31:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:15.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:15.086376+0000) 2022-09-10T09:31:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:15.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:15.573737+0000) 2022-09-10T09:31:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:16.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:16.086537+0000) 2022-09-10T09:31:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:16 smithi104 ceph-mon[97044]: pgmap v9666: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:31:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:16.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:16.574000+0000) 2022-09-10T09:31:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:16 smithi110 ceph-mon[97633]: pgmap v9666: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:31:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:17.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:17.086754+0000) 2022-09-10T09:31:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:17.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:17.574255+0000) 2022-09-10T09:31:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:18.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:18.086995+0000) 2022-09-10T09:31:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:18 smithi104 ceph-mon[97044]: pgmap v9667: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:31:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:18.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:18.574501+0000) 2022-09-10T09:31:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:18 smithi110 ceph-mon[97633]: pgmap v9667: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:31:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:19.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:19.087261+0000) 2022-09-10T09:31:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:19.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:19.574706+0000) 2022-09-10T09:31:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:20.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:20.087478+0000) 2022-09-10T09:31:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:20 smithi104 ceph-mon[97044]: pgmap v9668: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:31:21.062 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:20.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:20.574918+0000) 2022-09-10T09:31:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:20 smithi110 ceph-mon[97633]: pgmap v9668: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:31:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:21.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:21.087710+0000) 2022-09-10T09:31:21.867 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:21.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:21.575168+0000) 2022-09-10T09:31:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:22.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:22.087955+0000) 2022-09-10T09:31:22.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:31:21.865+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:31:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:22 smithi104 ceph-mon[97044]: pgmap v9669: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:31:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:22.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:22.575368+0000) 2022-09-10T09:31:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:22 smithi110 ceph-mon[97633]: pgmap v9669: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:31:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:23.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:23.088158+0000) 2022-09-10T09:31:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:23.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:23.575631+0000) 2022-09-10T09:31:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:24.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:24.088338+0000) 2022-09-10T09:31:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:24 smithi104 ceph-mon[97044]: pgmap v9670: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:31:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:24.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:24.575799+0000) 2022-09-10T09:31:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:24 smithi110 ceph-mon[97633]: pgmap v9670: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:31:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:25.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:25.088550+0000) 2022-09-10T09:31:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:25.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:25.576011+0000) 2022-09-10T09:31:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:26.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:26.088723+0000) 2022-09-10T09:31:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:26 smithi104 ceph-mon[97044]: pgmap v9671: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:31:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:26.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:26.576209+0000) 2022-09-10T09:31:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:26 smithi110 ceph-mon[97633]: pgmap v9671: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:31:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:27.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:27.088960+0000) 2022-09-10T09:31:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:27.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:27.576380+0000) 2022-09-10T09:31:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:28.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:28.089165+0000) 2022-09-10T09:31:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:28 smithi104 ceph-mon[97044]: pgmap v9672: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:28.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:28.576591+0000) 2022-09-10T09:31:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:28 smithi110 ceph-mon[97633]: pgmap v9672: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:29.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:29.089397+0000) 2022-09-10T09:31:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:29.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:29.576826+0000) 2022-09-10T09:31:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:30.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:30.089583+0000) 2022-09-10T09:31:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:30 smithi104 ceph-mon[97044]: pgmap v9673: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:30.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:30.577048+0000) 2022-09-10T09:31:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:30 smithi110 ceph-mon[97633]: pgmap v9673: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:31.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:31.089755+0000) 2022-09-10T09:31:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:31.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:31.577156+0000) 2022-09-10T09:31:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:32.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:32.089894+0000) 2022-09-10T09:31:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:32.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:32.577376+0000) 2022-09-10T09:31:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:32 smithi110 ceph-mon[97633]: pgmap v9674: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:32 smithi104 ceph-mon[97044]: pgmap v9674: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:33.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:33.090152+0000) 2022-09-10T09:31:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:33.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:33.577538+0000) 2022-09-10T09:31:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:34.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:34.090327+0000) 2022-09-10T09:31:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:34.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:34.577777+0000) 2022-09-10T09:31:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:34 smithi110 ceph-mon[97633]: pgmap v9675: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:34 smithi104 ceph-mon[97044]: pgmap v9675: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:35.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:35.090539+0000) 2022-09-10T09:31:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:35.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:35.578048+0000) 2022-09-10T09:31:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:36.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:36.090746+0000) 2022-09-10T09:31:36.883 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:36.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:36.578263+0000) 2022-09-10T09:31:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:36 smithi110 ceph-mon[97633]: pgmap v9676: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:36 smithi104 ceph-mon[97044]: pgmap v9676: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:37.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:37.090921+0000) 2022-09-10T09:31:37.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:31:36.881+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:31:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:37.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:37.578430+0000) 2022-09-10T09:31:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:38.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:38.091089+0000) 2022-09-10T09:31:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:38.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:38.578675+0000) 2022-09-10T09:31:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:38 smithi110 ceph-mon[97633]: pgmap v9677: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:38 smithi104 ceph-mon[97044]: pgmap v9677: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:39.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:39.091333+0000) 2022-09-10T09:31:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:39.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:39.578875+0000) 2022-09-10T09:31:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:40.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:40.091509+0000) 2022-09-10T09:31:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:40.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:40.579167+0000) 2022-09-10T09:31:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:40 smithi110 ceph-mon[97633]: pgmap v9678: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:40 smithi104 ceph-mon[97044]: pgmap v9678: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:41.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:41.091724+0000) 2022-09-10T09:31:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:41.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:41.579314+0000) 2022-09-10T09:31:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:42.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:42.091950+0000) 2022-09-10T09:31:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:42.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:42.579563+0000) 2022-09-10T09:31:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:42 smithi110 ceph-mon[97633]: pgmap v9679: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:42 smithi104 ceph-mon[97044]: pgmap v9679: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:43.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:43.092151+0000) 2022-09-10T09:31:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:43.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:43.579787+0000) 2022-09-10T09:31:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:44.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:44.092281+0000) 2022-09-10T09:31:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:44.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:44.580030+0000) 2022-09-10T09:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:44 smithi110 ceph-mon[97633]: pgmap v9680: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:44 smithi104 ceph-mon[97044]: pgmap v9680: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:45.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:45.092426+0000) 2022-09-10T09:31:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:45.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:45.580252+0000) 2022-09-10T09:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:31:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:31:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:31:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:46.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:46.092618+0000) 2022-09-10T09:31:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:46.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:46.580503+0000) 2022-09-10T09:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:46 smithi110 ceph-mon[97633]: pgmap v9681: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:46 smithi104 ceph-mon[97044]: pgmap v9681: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:47.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:47.092845+0000) 2022-09-10T09:31:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:47.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:47.580721+0000) 2022-09-10T09:31:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:48.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:48.093051+0000) 2022-09-10T09:31:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:48.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:48.580926+0000) 2022-09-10T09:31:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:48 smithi110 ceph-mon[97633]: pgmap v9682: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:48 smithi104 ceph-mon[97044]: pgmap v9682: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:49.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:49.093359+0000) 2022-09-10T09:31:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:49.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:49.581130+0000) 2022-09-10T09:31:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:50.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:50.093619+0000) 2022-09-10T09:31:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:50.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:50.581349+0000) 2022-09-10T09:31:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:50 smithi110 ceph-mon[97633]: pgmap v9683: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:50 smithi104 ceph-mon[97044]: pgmap v9683: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:51.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:51.093876+0000) 2022-09-10T09:31:51.892 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:51.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:51.581575+0000) 2022-09-10T09:31:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:52.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:52.094133+0000) 2022-09-10T09:31:52.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:31:51.891+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:31:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:52.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:52.581798+0000) 2022-09-10T09:31:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:52 smithi110 ceph-mon[97633]: pgmap v9684: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:52 smithi104 ceph-mon[97044]: pgmap v9684: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:53.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:53.094338+0000) 2022-09-10T09:31:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:53.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:53.582016+0000) 2022-09-10T09:31:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:54.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:54.094535+0000) 2022-09-10T09:31:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:54.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:54.582211+0000) 2022-09-10T09:31:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:54 smithi110 ceph-mon[97633]: pgmap v9685: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:54 smithi104 ceph-mon[97044]: pgmap v9685: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:55.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:55.094773+0000) 2022-09-10T09:31:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:55.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:55.582417+0000) 2022-09-10T09:31:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:56.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:56.095000+0000) 2022-09-10T09:31:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:56.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:56.582643+0000) 2022-09-10T09:31:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:56 smithi110 ceph-mon[97633]: pgmap v9686: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:56 smithi104 ceph-mon[97044]: pgmap v9686: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:31:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:57.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:57.095267+0000) 2022-09-10T09:31:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:57.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:57.582814+0000) 2022-09-10T09:31:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:58.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:58.095449+0000) 2022-09-10T09:31:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:58.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:58.583065+0000) 2022-09-10T09:31:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:31:58 smithi110 ceph-mon[97633]: pgmap v9687: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:31:58 smithi104 ceph-mon[97044]: pgmap v9687: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:31:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:31:59.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:59.095717+0000) 2022-09-10T09:32:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:31:59.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:31:59.583233+0000) 2022-09-10T09:32:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:00.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:00.095913+0000) 2022-09-10T09:32:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:00.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:00.583395+0000) 2022-09-10T09:32:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:00 smithi110 ceph-mon[97633]: pgmap v9688: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:00 smithi104 ceph-mon[97044]: pgmap v9688: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:01.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:01.096091+0000) 2022-09-10T09:32:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:01.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:01.583517+0000) 2022-09-10T09:32:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:02.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:02.096357+0000) 2022-09-10T09:32:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:02.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:02.583646+0000) 2022-09-10T09:32:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:02 smithi110 ceph-mon[97633]: pgmap v9689: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:02 smithi104 ceph-mon[97044]: pgmap v9689: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:03.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:03.096606+0000) 2022-09-10T09:32:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:03.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:03.583822+0000) 2022-09-10T09:32:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:04.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:04.096815+0000) 2022-09-10T09:32:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:04.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:04.583981+0000) 2022-09-10T09:32:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:04 smithi110 ceph-mon[97633]: pgmap v9690: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:04 smithi104 ceph-mon[97044]: pgmap v9690: 129 pgs: 129 active+clean; 8.7 MiB data, 943 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:05.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:05.097018+0000) 2022-09-10T09:32:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:05.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:05.584199+0000) 2022-09-10T09:32:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:06.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:06.097215+0000) 2022-09-10T09:32:06.906 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:06.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:06.584439+0000) 2022-09-10T09:32:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:06 smithi110 ceph-mon[97633]: pgmap v9691: 129 pgs: 129 active+clean; 8.7 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:32:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:06 smithi104 ceph-mon[97044]: pgmap v9691: 129 pgs: 129 active+clean; 8.7 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:32:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:32:06.904+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:32:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:07.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:07.097391+0000) 2022-09-10T09:32:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:07.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:07.584643+0000) 2022-09-10T09:32:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:08.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:08.097616+0000) 2022-09-10T09:32:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:08.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:08.584878+0000) 2022-09-10T09:32:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:08 smithi110 ceph-mon[97633]: pgmap v9692: 129 pgs: 129 active+clean; 8.7 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:32:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:08 smithi110 ceph-mon[97633]: osdmap e686: 12 total, 12 up, 12 in 2022-09-10T09:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:08 smithi104 ceph-mon[97044]: pgmap v9692: 129 pgs: 129 active+clean; 8.7 MiB data, 944 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:32:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:08 smithi104 ceph-mon[97044]: osdmap e686: 12 total, 12 up, 12 in 2022-09-10T09:32:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:09.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:09.097831+0000) 2022-09-10T09:32:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:09.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:09.585118+0000) 2022-09-10T09:32:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:10.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:10.098050+0000) 2022-09-10T09:32:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:10.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:10.585305+0000) 2022-09-10T09:32:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:10 smithi110 ceph-mon[97633]: pgmap v9694: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:32:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:10 smithi104 ceph-mon[97044]: pgmap v9694: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:32:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:11.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:11.098347+0000) 2022-09-10T09:32:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:11.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:11.585449+0000) 2022-09-10T09:32:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:12.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:12.098601+0000) 2022-09-10T09:32:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:12 smithi104 ceph-mon[97044]: pgmap v9695: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:32:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:12 smithi104 ceph-mon[97044]: osdmap e687: 12 total, 12 up, 12 in 2022-09-10T09:32:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:12.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:12.585683+0000) 2022-09-10T09:32:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:12 smithi110 ceph-mon[97633]: pgmap v9695: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:32:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:12 smithi110 ceph-mon[97633]: osdmap e687: 12 total, 12 up, 12 in 2022-09-10T09:32:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:13.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:13.098845+0000) 2022-09-10T09:32:13.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:13.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:13.585928+0000) 2022-09-10T09:32:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:14.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:14.099043+0000) 2022-09-10T09:32:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:14 smithi104 ceph-mon[97044]: pgmap v9697: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:32:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:14.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:14.586175+0000) 2022-09-10T09:32:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:14 smithi110 ceph-mon[97633]: pgmap v9697: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:32:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:15.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:15.099282+0000) 2022-09-10T09:32:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:15.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:15.586315+0000) 2022-09-10T09:32:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:16.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:16.099482+0000) 2022-09-10T09:32:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:16 smithi104 ceph-mon[97044]: pgmap v9698: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:32:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:16.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:16.586474+0000) 2022-09-10T09:32:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:16 smithi110 ceph-mon[97633]: pgmap v9698: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:32:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:17.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:17.099708+0000) 2022-09-10T09:32:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:17.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:17.586601+0000) 2022-09-10T09:32:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:18.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:18.099890+0000) 2022-09-10T09:32:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:18 smithi104 ceph-mon[97044]: pgmap v9699: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:32:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:18.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:18.586754+0000) 2022-09-10T09:32:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:18 smithi110 ceph-mon[97633]: pgmap v9699: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:32:19.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:19.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:19.100125+0000) 2022-09-10T09:32:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:19.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:19.586926+0000) 2022-09-10T09:32:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:20.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:20.100345+0000) 2022-09-10T09:32:21.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:20.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:20.587158+0000) 2022-09-10T09:32:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:20 smithi110 ceph-mon[97633]: pgmap v9700: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:32:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:20 smithi104 ceph-mon[97044]: pgmap v9700: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:32:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:21.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:21.100532+0000) 2022-09-10T09:32:21.922 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:21.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:21.587395+0000) 2022-09-10T09:32:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:22.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:22.100812+0000) 2022-09-10T09:32:22.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:32:21.920+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:32:23.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:22.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:22.587595+0000) 2022-09-10T09:32:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:22 smithi110 ceph-mon[97633]: pgmap v9701: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:32:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:22 smithi104 ceph-mon[97044]: pgmap v9701: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:32:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:23.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:23.101061+0000) 2022-09-10T09:32:24.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:23.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:23.587822+0000) 2022-09-10T09:32:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:24.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:24.101316+0000) 2022-09-10T09:32:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:24.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:24.588045+0000) 2022-09-10T09:32:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:24 smithi110 ceph-mon[97633]: pgmap v9702: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:32:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:24 smithi104 ceph-mon[97044]: pgmap v9702: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:32:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:25.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:25.101507+0000) 2022-09-10T09:32:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:25.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:25.588304+0000) 2022-09-10T09:32:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:26.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:26.101658+0000) 2022-09-10T09:32:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:26.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:26.588483+0000) 2022-09-10T09:32:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:26 smithi110 ceph-mon[97633]: pgmap v9703: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:32:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:26 smithi104 ceph-mon[97044]: pgmap v9703: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:32:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:27.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:27.101885+0000) 2022-09-10T09:32:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:27.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:27.588726+0000) 2022-09-10T09:32:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:28.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:28.102162+0000) 2022-09-10T09:32:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:28.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:28.588982+0000) 2022-09-10T09:32:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:28 smithi110 ceph-mon[97633]: pgmap v9704: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:28 smithi104 ceph-mon[97044]: pgmap v9704: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:29.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:29.102401+0000) 2022-09-10T09:32:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:29.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:29.589187+0000) 2022-09-10T09:32:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:30.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:30.102598+0000) 2022-09-10T09:32:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:30.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:30.589440+0000) 2022-09-10T09:32:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:30 smithi110 ceph-mon[97633]: pgmap v9705: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:30 smithi104 ceph-mon[97044]: pgmap v9705: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:31.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:31.102866+0000) 2022-09-10T09:32:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:31.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:31.589647+0000) 2022-09-10T09:32:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:32.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:32.103146+0000) 2022-09-10T09:32:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:32.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:32.589914+0000) 2022-09-10T09:32:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:32 smithi110 ceph-mon[97633]: pgmap v9706: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:32 smithi104 ceph-mon[97044]: pgmap v9706: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:33.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:33.103366+0000) 2022-09-10T09:32:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:33.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:33.590191+0000) 2022-09-10T09:32:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:34.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:34.103588+0000) 2022-09-10T09:32:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:34.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:34.590421+0000) 2022-09-10T09:32:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:34 smithi110 ceph-mon[97633]: pgmap v9707: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:34 smithi104 ceph-mon[97044]: pgmap v9707: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:35.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:35.103834+0000) 2022-09-10T09:32:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:35.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:35.590576+0000) 2022-09-10T09:32:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:36.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:36.104024+0000) 2022-09-10T09:32:36.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:36.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:36.590799+0000) 2022-09-10T09:32:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:36 smithi110 ceph-mon[97633]: pgmap v9708: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:36 smithi104 ceph-mon[97044]: pgmap v9708: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:37.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:37.104304+0000) 2022-09-10T09:32:37.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:32:36.934+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:32:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:37.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:37.591040+0000) 2022-09-10T09:32:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:38.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:38.104525+0000) 2022-09-10T09:32:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:38.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:38.591251+0000) 2022-09-10T09:32:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:38 smithi110 ceph-mon[97633]: pgmap v9709: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:38 smithi104 ceph-mon[97044]: pgmap v9709: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:39.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:39.104753+0000) 2022-09-10T09:32:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:39.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:39.591439+0000) 2022-09-10T09:32:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:40.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:40.104966+0000) 2022-09-10T09:32:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:40.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:40.591681+0000) 2022-09-10T09:32:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:40 smithi110 ceph-mon[97633]: pgmap v9710: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:40 smithi104 ceph-mon[97044]: pgmap v9710: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:41.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:41.105207+0000) 2022-09-10T09:32:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:41.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:41.591900+0000) 2022-09-10T09:32:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:42.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:42.105470+0000) 2022-09-10T09:32:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:42.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:42.592206+0000) 2022-09-10T09:32:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:42 smithi110 ceph-mon[97633]: pgmap v9711: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:42 smithi104 ceph-mon[97044]: pgmap v9711: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:43.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:43.105723+0000) 2022-09-10T09:32:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:43.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:43.592408+0000) 2022-09-10T09:32:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:44.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:44.105971+0000) 2022-09-10T09:32:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:44.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:44.592605+0000) 2022-09-10T09:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:44 smithi110 ceph-mon[97633]: pgmap v9712: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:44 smithi104 ceph-mon[97044]: pgmap v9712: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:45.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:45.106228+0000) 2022-09-10T09:32:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:45.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:45.592835+0000) 2022-09-10T09:32:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:32:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:32:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:32:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:32:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:46.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:46.106416+0000) 2022-09-10T09:32:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:46.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:46.593086+0000) 2022-09-10T09:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:46 smithi110 ceph-mon[97633]: pgmap v9713: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:46 smithi104 ceph-mon[97044]: pgmap v9713: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:47.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:47.106667+0000) 2022-09-10T09:32:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:47.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:47.593276+0000) 2022-09-10T09:32:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:48.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:48.106911+0000) 2022-09-10T09:32:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:48.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:48.593458+0000) 2022-09-10T09:32:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:48 smithi110 ceph-mon[97633]: pgmap v9714: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:48 smithi104 ceph-mon[97044]: pgmap v9714: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:49.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:49.107185+0000) 2022-09-10T09:32:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:49.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:49.593676+0000) 2022-09-10T09:32:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:50.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:50.107416+0000) 2022-09-10T09:32:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:50.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:50.593918+0000) 2022-09-10T09:32:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:50 smithi110 ceph-mon[97633]: pgmap v9715: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:50 smithi104 ceph-mon[97044]: pgmap v9715: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:51.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:51.107666+0000) 2022-09-10T09:32:51.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:51.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:51.594086+0000) 2022-09-10T09:32:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:52.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:52.107889+0000) 2022-09-10T09:32:52.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:32:51.949+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:32:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:52.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:52.594272+0000) 2022-09-10T09:32:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:52 smithi110 ceph-mon[97633]: pgmap v9716: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:52 smithi104 ceph-mon[97044]: pgmap v9716: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:53.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:53.108135+0000) 2022-09-10T09:32:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:53.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:53.594436+0000) 2022-09-10T09:32:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:54.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:54.108354+0000) 2022-09-10T09:32:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:54.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:54.594640+0000) 2022-09-10T09:32:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:54 smithi110 ceph-mon[97633]: pgmap v9717: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:54 smithi104 ceph-mon[97044]: pgmap v9717: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:55.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:55.108537+0000) 2022-09-10T09:32:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:55.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:55.594827+0000) 2022-09-10T09:32:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:56.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:56.108808+0000) 2022-09-10T09:32:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:56.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:56.595043+0000) 2022-09-10T09:32:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:56 smithi110 ceph-mon[97633]: pgmap v9718: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:56 smithi104 ceph-mon[97044]: pgmap v9718: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:32:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:57.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:57.109182+0000) 2022-09-10T09:32:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:57.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:57.595259+0000) 2022-09-10T09:32:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:58.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:58.109333+0000) 2022-09-10T09:32:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:58.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:58.595461+0000) 2022-09-10T09:32:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:32:58 smithi110 ceph-mon[97633]: pgmap v9719: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:32:58 smithi104 ceph-mon[97044]: pgmap v9719: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:32:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:32:59.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:59.109543+0000) 2022-09-10T09:33:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:32:59.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:32:59.595732+0000) 2022-09-10T09:33:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:00.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:00.109788+0000) 2022-09-10T09:33:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:00.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:00.595892+0000) 2022-09-10T09:33:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:00 smithi110 ceph-mon[97633]: pgmap v9720: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:00 smithi104 ceph-mon[97044]: pgmap v9720: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:01.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:01.110000+0000) 2022-09-10T09:33:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:01.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:01.596056+0000) 2022-09-10T09:33:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:02.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:02.110221+0000) 2022-09-10T09:33:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:02.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:02.596340+0000) 2022-09-10T09:33:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:02 smithi110 ceph-mon[97633]: pgmap v9721: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:02 smithi104 ceph-mon[97044]: pgmap v9721: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:03.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:03.110444+0000) 2022-09-10T09:33:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:03.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:03.596509+0000) 2022-09-10T09:33:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:04.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:04.110685+0000) 2022-09-10T09:33:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:04.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:04.596703+0000) 2022-09-10T09:33:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:04 smithi110 ceph-mon[97633]: pgmap v9722: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:04 smithi104 ceph-mon[97044]: pgmap v9722: 129 pgs: 129 active+clean; 8.7 MiB data, 945 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:05.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:05.110902+0000) 2022-09-10T09:33:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:05.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:05.596927+0000) 2022-09-10T09:33:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:06.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:06.111066+0000) 2022-09-10T09:33:06.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:06.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:06.597109+0000) 2022-09-10T09:33:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:06 smithi110 ceph-mon[97633]: pgmap v9723: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:33:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:06 smithi104 ceph-mon[97044]: pgmap v9723: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:33:07.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:33:06.963+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:33:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:07.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:07.111312+0000) 2022-09-10T09:33:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:07.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:07.597328+0000) 2022-09-10T09:33:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:08.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:08.111523+0000) 2022-09-10T09:33:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:08.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:08.597577+0000) 2022-09-10T09:33:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:08 smithi110 ceph-mon[97633]: pgmap v9724: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:33:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:08 smithi110 ceph-mon[97633]: osdmap e688: 12 total, 12 up, 12 in 2022-09-10T09:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:08 smithi104 ceph-mon[97044]: pgmap v9724: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:08 smithi104 ceph-mon[97044]: osdmap e688: 12 total, 12 up, 12 in 2022-09-10T09:33:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:09.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:09.111768+0000) 2022-09-10T09:33:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:09.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:09.597806+0000) 2022-09-10T09:33:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:10.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:10.111945+0000) 2022-09-10T09:33:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:10.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:10.598038+0000) 2022-09-10T09:33:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:10 smithi110 ceph-mon[97633]: pgmap v9726: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:33:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:10 smithi104 ceph-mon[97044]: pgmap v9726: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:33:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:11.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:11.112176+0000) 2022-09-10T09:33:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:11.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:11.598246+0000) 2022-09-10T09:33:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:12.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:12.112333+0000) 2022-09-10T09:33:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:12.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:12.598442+0000) 2022-09-10T09:33:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:12 smithi110 ceph-mon[97633]: pgmap v9727: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:33:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:12 smithi110 ceph-mon[97633]: osdmap e689: 12 total, 12 up, 12 in 2022-09-10T09:33:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:12 smithi104 ceph-mon[97044]: pgmap v9727: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:33:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:12 smithi104 ceph-mon[97044]: osdmap e689: 12 total, 12 up, 12 in 2022-09-10T09:33:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:13.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:13.112594+0000) 2022-09-10T09:33:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:13.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:13.598614+0000) 2022-09-10T09:33:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:14.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:14.112827+0000) 2022-09-10T09:33:14.868 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:14.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:14.598791+0000) 2022-09-10T09:33:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:14 smithi110 ceph-mon[97633]: pgmap v9729: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:33:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:14 smithi104 ceph-mon[97044]: pgmap v9729: 129 pgs: 129 active+clean; 8.7 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:33:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:15.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:15.113062+0000) 2022-09-10T09:33:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:15.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:15.599023+0000) 2022-09-10T09:33:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:16.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:16.113270+0000) 2022-09-10T09:33:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:16.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:16.599236+0000) 2022-09-10T09:33:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:16 smithi110 ceph-mon[97633]: pgmap v9730: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:33:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:16 smithi104 ceph-mon[97044]: pgmap v9730: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:33:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:17.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:17.113539+0000) 2022-09-10T09:33:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:17.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:17.599426+0000) 2022-09-10T09:33:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:18.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:18.113723+0000) 2022-09-10T09:33:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:18.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:18.599643+0000) 2022-09-10T09:33:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:18 smithi110 ceph-mon[97633]: pgmap v9731: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:33:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:18 smithi104 ceph-mon[97044]: pgmap v9731: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:33:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:19.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:19.113900+0000) 2022-09-10T09:33:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:19.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:19.599887+0000) 2022-09-10T09:33:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:20.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:20.114118+0000) 2022-09-10T09:33:21.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:20.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:20.600162+0000) 2022-09-10T09:33:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:20 smithi110 ceph-mon[97633]: pgmap v9732: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:33:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:20 smithi104 ceph-mon[97044]: pgmap v9732: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:33:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:21.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:21.114355+0000) 2022-09-10T09:33:21.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:21.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:21.600300+0000) 2022-09-10T09:33:22.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:22.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:22.114606+0000) 2022-09-10T09:33:22.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:33:21.979+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:33:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:22.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:22.600461+0000) 2022-09-10T09:33:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:22 smithi110 ceph-mon[97633]: pgmap v9733: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:33:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:22 smithi104 ceph-mon[97044]: pgmap v9733: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:33:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:23.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:23.114836+0000) 2022-09-10T09:33:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:23.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:23.600624+0000) 2022-09-10T09:33:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:24.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:24.115128+0000) 2022-09-10T09:33:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:24.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:24.600890+0000) 2022-09-10T09:33:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:24 smithi110 ceph-mon[97633]: pgmap v9734: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:33:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:24 smithi104 ceph-mon[97044]: pgmap v9734: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:33:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:25.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:25.115355+0000) 2022-09-10T09:33:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:25.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:25.601170+0000) 2022-09-10T09:33:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:26.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:26.115543+0000) 2022-09-10T09:33:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:26.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:26.601364+0000) 2022-09-10T09:33:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:26 smithi110 ceph-mon[97633]: pgmap v9735: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:33:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:26 smithi104 ceph-mon[97044]: pgmap v9735: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:33:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:27.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:27.115782+0000) 2022-09-10T09:33:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:27.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:27.601566+0000) 2022-09-10T09:33:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:28.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:28.116019+0000) 2022-09-10T09:33:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:28.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:28.601810+0000) 2022-09-10T09:33:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:28 smithi110 ceph-mon[97633]: pgmap v9736: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:28 smithi104 ceph-mon[97044]: pgmap v9736: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:29.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:29.116294+0000) 2022-09-10T09:33:30.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:29.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:29.602036+0000) 2022-09-10T09:33:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:30.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:30.116533+0000) 2022-09-10T09:33:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:30.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:30.602251+0000) 2022-09-10T09:33:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:30 smithi110 ceph-mon[97633]: pgmap v9737: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:30 smithi104 ceph-mon[97044]: pgmap v9737: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:31.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:31.116692+0000) 2022-09-10T09:33:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:31.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:31.602438+0000) 2022-09-10T09:33:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:32.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:32.116897+0000) 2022-09-10T09:33:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:32.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:32.602696+0000) 2022-09-10T09:33:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:32 smithi110 ceph-mon[97633]: pgmap v9738: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:32 smithi104 ceph-mon[97044]: pgmap v9738: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:33.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:33.117075+0000) 2022-09-10T09:33:34.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:33.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:33.602915+0000) 2022-09-10T09:33:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:34.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:34.117319+0000) 2022-09-10T09:33:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:34.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:34.603178+0000) 2022-09-10T09:33:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:34 smithi110 ceph-mon[97633]: pgmap v9739: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:34 smithi104 ceph-mon[97044]: pgmap v9739: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:35.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:35.117572+0000) 2022-09-10T09:33:36.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:35.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:35.603385+0000) 2022-09-10T09:33:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:36.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:36.117818+0000) 2022-09-10T09:33:36.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:36.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:36.603607+0000) 2022-09-10T09:33:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:36 smithi110 ceph-mon[97633]: pgmap v9740: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:36 smithi104 ceph-mon[97044]: pgmap v9740: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:37.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:37.118034+0000) 2022-09-10T09:33:37.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:33:36.994+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:33:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:37.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:37.603837+0000) 2022-09-10T09:33:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:38.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:38.118236+0000) 2022-09-10T09:33:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:38.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:38.604024+0000) 2022-09-10T09:33:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:38 smithi110 ceph-mon[97633]: pgmap v9741: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:38 smithi104 ceph-mon[97044]: pgmap v9741: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:39.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:39.118436+0000) 2022-09-10T09:33:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:39.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:39.604272+0000) 2022-09-10T09:33:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:40.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:40.118682+0000) 2022-09-10T09:33:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:40.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:40.604502+0000) 2022-09-10T09:33:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:40 smithi110 ceph-mon[97633]: pgmap v9742: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:40 smithi104 ceph-mon[97044]: pgmap v9742: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:33:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:41.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:41.118857+0000) 2022-09-10T09:33:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:41.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:41.604741+0000) 2022-09-10T09:33:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:42.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:42.119125+0000) 2022-09-10T09:33:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:42.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:42.605000+0000) 2022-09-10T09:33:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:42 smithi110 ceph-mon[97633]: pgmap v9743: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:42 smithi104 ceph-mon[97044]: pgmap v9743: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:43.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:43.119370+0000) 2022-09-10T09:33:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:43.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:43.605271+0000) 2022-09-10T09:33:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:44.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:44.119588+0000) 2022-09-10T09:33:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:44.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:44.605428+0000) 2022-09-10T09:33:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:44 smithi110 ceph-mon[97633]: pgmap v9744: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:44 smithi104 ceph-mon[97044]: pgmap v9744: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:45.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:45.119852+0000) 2022-09-10T09:33:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:45.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:45.605657+0000) 2022-09-10T09:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:33:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:33:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:33:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:33:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:33:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:33:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:33:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:33:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:46.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:46.120034+0000) 2022-09-10T09:33:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:46.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:46.605873+0000) 2022-09-10T09:33:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:46 smithi110 ceph-mon[97633]: pgmap v9745: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:33:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:46 smithi104 ceph-mon[97044]: pgmap v9745: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:33:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:47.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:47.120272+0000) 2022-09-10T09:33:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:47.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:47.606088+0000) 2022-09-10T09:33:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:48.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:48.120396+0000) 2022-09-10T09:33:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:48.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:48.606275+0000) 2022-09-10T09:33:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:48 smithi110 ceph-mon[97633]: pgmap v9746: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:33:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:48 smithi104 ceph-mon[97044]: pgmap v9746: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:33:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:49.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:49.120602+0000) 2022-09-10T09:33:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:49.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:49.606497+0000) 2022-09-10T09:33:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:50.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:50.120829+0000) 2022-09-10T09:33:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:50.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:50.606693+0000) 2022-09-10T09:33:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:50 smithi110 ceph-mon[97633]: pgmap v9747: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:33:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:50 smithi104 ceph-mon[97044]: pgmap v9747: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:33:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:51.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:51.121035+0000) 2022-09-10T09:33:52.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:51.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:51.606917+0000) 2022-09-10T09:33:52.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:33:52.010+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:33:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:52.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:52.121219+0000) 2022-09-10T09:33:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:52.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:52.607134+0000) 2022-09-10T09:33:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:52 smithi110 ceph-mon[97633]: pgmap v9748: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:33:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:52 smithi104 ceph-mon[97044]: pgmap v9748: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:33:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:53.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:53.121456+0000) 2022-09-10T09:33:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:53.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:53.607341+0000) 2022-09-10T09:33:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:54.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:54.121694+0000) 2022-09-10T09:33:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:54.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:54.607545+0000) 2022-09-10T09:33:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:54 smithi110 ceph-mon[97633]: pgmap v9749: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:33:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:54 smithi104 ceph-mon[97044]: pgmap v9749: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T09:33:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:55.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:55.121877+0000) 2022-09-10T09:33:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:55.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:55.607690+0000) 2022-09-10T09:33:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:56.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:56.122125+0000) 2022-09-10T09:33:57.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:56.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:56.607925+0000) 2022-09-10T09:33:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:56 smithi110 ceph-mon[97633]: pgmap v9750: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:33:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:56 smithi104 ceph-mon[97044]: pgmap v9750: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T09:33:57.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:57.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:57.122390+0000) 2022-09-10T09:33:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:57.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:57.608199+0000) 2022-09-10T09:33:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:58.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:58.122622+0000) 2022-09-10T09:33:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:58.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:58.608419+0000) 2022-09-10T09:33:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:33:58 smithi110 ceph-mon[97633]: pgmap v9751: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:33:58 smithi104 ceph-mon[97044]: pgmap v9751: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:33:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:33:59.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:59.122847+0000) 2022-09-10T09:34:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:33:59.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:33:59.608564+0000) 2022-09-10T09:34:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:00.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:00.123147+0000) 2022-09-10T09:34:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:00.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:00.608726+0000) 2022-09-10T09:34:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:00 smithi110 ceph-mon[97633]: pgmap v9752: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:00 smithi104 ceph-mon[97044]: pgmap v9752: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:01.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:01.123317+0000) 2022-09-10T09:34:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:01.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:01.608910+0000) 2022-09-10T09:34:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:02.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:02.123465+0000) 2022-09-10T09:34:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:02.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:02.609144+0000) 2022-09-10T09:34:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:02 smithi110 ceph-mon[97633]: pgmap v9753: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:02 smithi104 ceph-mon[97044]: pgmap v9753: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:03.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:03.123655+0000) 2022-09-10T09:34:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:03.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:03.609369+0000) 2022-09-10T09:34:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:04.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:04.123902+0000) 2022-09-10T09:34:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:04.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:04.609618+0000) 2022-09-10T09:34:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:04 smithi110 ceph-mon[97633]: pgmap v9754: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:04 smithi104 ceph-mon[97044]: pgmap v9754: 129 pgs: 129 active+clean; 8.7 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:05.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:05.124190+0000) 2022-09-10T09:34:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:05.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:05.609844+0000) 2022-09-10T09:34:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:06.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:06.124414+0000) 2022-09-10T09:34:07.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:06.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:06.610089+0000) 2022-09-10T09:34:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:06 smithi110 ceph-mon[97633]: pgmap v9755: 129 pgs: 129 active+clean; 8.8 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:34:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:06 smithi104 ceph-mon[97044]: pgmap v9755: 129 pgs: 129 active+clean; 8.8 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:34:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:07.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:07.124611+0000) 2022-09-10T09:34:07.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:34:07.016+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:34:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:07.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:07.610363+0000) 2022-09-10T09:34:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:08.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:08.124879+0000) 2022-09-10T09:34:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:08.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:08.610612+0000) 2022-09-10T09:34:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:08 smithi110 ceph-mon[97633]: pgmap v9756: 129 pgs: 129 active+clean; 8.8 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:34:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:08 smithi110 ceph-mon[97633]: osdmap e690: 12 total, 12 up, 12 in 2022-09-10T09:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:08 smithi104 ceph-mon[97044]: pgmap v9756: 129 pgs: 129 active+clean; 8.8 MiB data, 949 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:34:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:08 smithi104 ceph-mon[97044]: osdmap e690: 12 total, 12 up, 12 in 2022-09-10T09:34:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:09.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:09.125184+0000) 2022-09-10T09:34:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:09.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:09.610789+0000) 2022-09-10T09:34:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:10.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:10.125411+0000) 2022-09-10T09:34:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:10.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:10.611026+0000) 2022-09-10T09:34:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:10 smithi110 ceph-mon[97633]: pgmap v9758: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:34:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:10 smithi104 ceph-mon[97044]: pgmap v9758: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:34:11.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:11.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:11.125649+0000) 2022-09-10T09:34:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:11.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:11.611238+0000) 2022-09-10T09:34:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:12.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:12.125852+0000) 2022-09-10T09:34:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:12.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:12.611489+0000) 2022-09-10T09:34:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:12 smithi110 ceph-mon[97633]: pgmap v9759: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:34:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:12 smithi110 ceph-mon[97633]: osdmap e691: 12 total, 12 up, 12 in 2022-09-10T09:34:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:12 smithi104 ceph-mon[97044]: pgmap v9759: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:34:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:12 smithi104 ceph-mon[97044]: osdmap e691: 12 total, 12 up, 12 in 2022-09-10T09:34:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:13.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:13.126055+0000) 2022-09-10T09:34:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:13.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:13.611790+0000) 2022-09-10T09:34:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:14.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:14.126248+0000) 2022-09-10T09:34:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:14.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:14.612061+0000) 2022-09-10T09:34:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:14 smithi110 ceph-mon[97633]: pgmap v9761: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:14 smithi104 ceph-mon[97044]: pgmap v9761: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:34:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:15.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:15.126492+0000) 2022-09-10T09:34:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:15.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:15.612316+0000) 2022-09-10T09:34:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:16.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:16.126735+0000) 2022-09-10T09:34:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:16.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:16.612595+0000) 2022-09-10T09:34:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:16 smithi110 ceph-mon[97633]: pgmap v9762: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:34:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:16 smithi104 ceph-mon[97044]: pgmap v9762: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:34:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:17.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:17.127023+0000) 2022-09-10T09:34:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:17.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:17.612821+0000) 2022-09-10T09:34:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:18.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:18.127296+0000) 2022-09-10T09:34:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:18.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:18.613020+0000) 2022-09-10T09:34:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:18 smithi110 ceph-mon[97633]: pgmap v9763: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:34:19.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:18 smithi104 ceph-mon[97044]: pgmap v9763: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:34:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:19.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:19.127481+0000) 2022-09-10T09:34:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:19.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:19.613231+0000) 2022-09-10T09:34:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:20.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:20.127776+0000) 2022-09-10T09:34:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:20.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:20.613468+0000) 2022-09-10T09:34:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:20 smithi110 ceph-mon[97633]: pgmap v9764: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:34:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:20 smithi104 ceph-mon[97044]: pgmap v9764: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:34:21.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:21.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:21.127936+0000) 2022-09-10T09:34:22.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:21.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:21.613680+0000) 2022-09-10T09:34:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:22.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:22.128128+0000) 2022-09-10T09:34:22.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:34:22.032+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:34:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:22.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:22.613888+0000) 2022-09-10T09:34:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:22 smithi110 ceph-mon[97633]: pgmap v9765: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:34:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:22 smithi104 ceph-mon[97044]: pgmap v9765: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:34:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:23.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:23.128372+0000) 2022-09-10T09:34:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:23.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:23.614132+0000) 2022-09-10T09:34:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:24.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:24.128638+0000) 2022-09-10T09:34:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:24.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:24.614326+0000) 2022-09-10T09:34:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:24 smithi110 ceph-mon[97633]: pgmap v9766: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:34:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:24 smithi104 ceph-mon[97044]: pgmap v9766: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:34:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:25.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:25.128906+0000) 2022-09-10T09:34:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:25.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:25.614574+0000) 2022-09-10T09:34:26.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:26.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:26.129199+0000) 2022-09-10T09:34:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:26.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:26.614803+0000) 2022-09-10T09:34:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:26 smithi110 ceph-mon[97633]: pgmap v9767: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:34:27.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:26 smithi104 ceph-mon[97044]: pgmap v9767: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:34:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:27.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:27.129443+0000) 2022-09-10T09:34:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:27.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:27.615018+0000) 2022-09-10T09:34:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:28.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:28.129700+0000) 2022-09-10T09:34:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:28.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:28.615260+0000) 2022-09-10T09:34:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:28 smithi110 ceph-mon[97633]: pgmap v9768: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:28 smithi104 ceph-mon[97044]: pgmap v9768: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:29.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:29.129879+0000) 2022-09-10T09:34:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:29.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:29.615473+0000) 2022-09-10T09:34:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:30.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:30.130091+0000) 2022-09-10T09:34:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:30.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:30.615737+0000) 2022-09-10T09:34:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:30 smithi110 ceph-mon[97633]: pgmap v9769: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:30 smithi104 ceph-mon[97044]: pgmap v9769: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:31.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:31.130291+0000) 2022-09-10T09:34:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:31.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:31.615962+0000) 2022-09-10T09:34:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:32.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:32.130489+0000) 2022-09-10T09:34:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:32.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:32.616201+0000) 2022-09-10T09:34:33.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:32 smithi110 ceph-mon[97633]: pgmap v9770: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:32 smithi104 ceph-mon[97044]: pgmap v9770: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:33.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:33.130721+0000) 2022-09-10T09:34:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:33.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:33.616416+0000) 2022-09-10T09:34:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:34.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:34.130904+0000) 2022-09-10T09:34:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:34.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:34.616616+0000) 2022-09-10T09:34:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:34 smithi110 ceph-mon[97633]: pgmap v9771: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:34 smithi104 ceph-mon[97044]: pgmap v9771: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:35.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:35.131156+0000) 2022-09-10T09:34:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:35.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:35.616841+0000) 2022-09-10T09:34:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:36.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:36.131366+0000) 2022-09-10T09:34:37.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:36.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:36.617151+0000) 2022-09-10T09:34:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:36 smithi110 ceph-mon[97633]: pgmap v9772: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:36 smithi104 ceph-mon[97044]: pgmap v9772: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:37.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:37.131494+0000) 2022-09-10T09:34:37.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:34:37.038+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:34:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:37.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:37.617338+0000) 2022-09-10T09:34:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:38.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:38.131699+0000) 2022-09-10T09:34:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:38.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:38.617579+0000) 2022-09-10T09:34:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:38 smithi110 ceph-mon[97633]: pgmap v9773: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:38 smithi104 ceph-mon[97044]: pgmap v9773: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:39.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:39.131917+0000) 2022-09-10T09:34:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:39.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:39.617775+0000) 2022-09-10T09:34:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:40.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:40.132200+0000) 2022-09-10T09:34:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:40.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:40.618022+0000) 2022-09-10T09:34:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:40 smithi110 ceph-mon[97633]: pgmap v9774: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:40 smithi104 ceph-mon[97044]: pgmap v9774: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:41.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:41.132441+0000) 2022-09-10T09:34:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:41.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:41.618251+0000) 2022-09-10T09:34:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:42.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:42.132660+0000) 2022-09-10T09:34:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:42.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:42.618417+0000) 2022-09-10T09:34:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:42 smithi110 ceph-mon[97633]: pgmap v9775: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:42 smithi104 ceph-mon[97044]: pgmap v9775: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:43.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:43.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:43.132950+0000) 2022-09-10T09:34:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:43.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:43.618592+0000) 2022-09-10T09:34:44.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:44.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:44.133141+0000) 2022-09-10T09:34:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:44.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:44.618769+0000) 2022-09-10T09:34:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:44 smithi110 ceph-mon[97633]: pgmap v9776: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:44 smithi104 ceph-mon[97044]: pgmap v9776: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:45.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:45.133289+0000) 2022-09-10T09:34:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:45.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:45.618989+0000) 2022-09-10T09:34:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:34:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:34:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:34:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:34:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:34:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:34:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:46.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:46.133524+0000) 2022-09-10T09:34:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:46.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:46.619228+0000) 2022-09-10T09:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:46 smithi110 ceph-mon[97633]: pgmap v9777: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:46 smithi104 ceph-mon[97044]: pgmap v9777: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:47.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:47.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:47.133816+0000) 2022-09-10T09:34:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:47.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:47.619488+0000) 2022-09-10T09:34:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:48.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:48.134074+0000) 2022-09-10T09:34:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:48.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:48.619686+0000) 2022-09-10T09:34:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:48 smithi110 ceph-mon[97633]: pgmap v9778: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:34:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:48 smithi104 ceph-mon[97044]: pgmap v9778: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:34:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:49.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:49.134292+0000) 2022-09-10T09:34:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:49.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:49.619943+0000) 2022-09-10T09:34:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:50.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:50.134549+0000) 2022-09-10T09:34:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:50.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:50.620173+0000) 2022-09-10T09:34:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:50 smithi110 ceph-mon[97633]: pgmap v9779: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:50 smithi104 ceph-mon[97044]: pgmap v9779: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:51.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:51.134746+0000) 2022-09-10T09:34:52.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:51.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:51.620315+0000) 2022-09-10T09:34:52.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:34:52.040+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:34:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:52.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:52.135009+0000) 2022-09-10T09:34:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:52.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:52.620489+0000) 2022-09-10T09:34:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:52 smithi110 ceph-mon[97633]: pgmap v9780: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:34:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:52 smithi104 ceph-mon[97044]: pgmap v9780: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:34:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:53.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:53.135249+0000) 2022-09-10T09:34:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:53.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:53.620702+0000) 2022-09-10T09:34:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:54.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:54.135477+0000) 2022-09-10T09:34:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:54.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:54.620914+0000) 2022-09-10T09:34:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:54 smithi110 ceph-mon[97633]: pgmap v9781: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:34:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:54 smithi104 ceph-mon[97044]: pgmap v9781: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:34:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:55.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:55.135755+0000) 2022-09-10T09:34:56.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:55.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:55.621165+0000) 2022-09-10T09:34:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:56.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:56.136015+0000) 2022-09-10T09:34:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:56.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:56.621350+0000) 2022-09-10T09:34:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:56 smithi110 ceph-mon[97633]: pgmap v9782: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:56 smithi104 ceph-mon[97044]: pgmap v9782: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:34:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:57.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:57.136284+0000) 2022-09-10T09:34:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:57.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:57.621475+0000) 2022-09-10T09:34:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:58.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:58.136483+0000) 2022-09-10T09:34:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:58.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:58.621726+0000) 2022-09-10T09:34:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:34:58 smithi110 ceph-mon[97633]: pgmap v9783: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:34:58 smithi104 ceph-mon[97044]: pgmap v9783: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:34:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:34:59.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:59.136677+0000) 2022-09-10T09:35:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:34:59.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:34:59.621950+0000) 2022-09-10T09:35:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:00.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:00.136841+0000) 2022-09-10T09:35:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:00.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:00.622132+0000) 2022-09-10T09:35:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:00 smithi110 ceph-mon[97633]: pgmap v9784: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:00 smithi104 ceph-mon[97044]: pgmap v9784: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:01.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:01.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:01.137057+0000) 2022-09-10T09:35:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:01.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:01.622321+0000) 2022-09-10T09:35:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:02.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:02.137160+0000) 2022-09-10T09:35:03.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:02.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:02.622574+0000) 2022-09-10T09:35:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:02 smithi110 ceph-mon[97633]: pgmap v9785: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:02 smithi104 ceph-mon[97044]: pgmap v9785: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:03.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:03.137404+0000) 2022-09-10T09:35:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:03.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:03.622802+0000) 2022-09-10T09:35:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:04.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:04.137640+0000) 2022-09-10T09:35:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:04.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:04.623058+0000) 2022-09-10T09:35:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:04 smithi110 ceph-mon[97633]: pgmap v9786: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:04 smithi104 ceph-mon[97044]: pgmap v9786: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:05.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:05.137863+0000) 2022-09-10T09:35:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:05.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:05.623310+0000) 2022-09-10T09:35:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:06.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:06.138019+0000) 2022-09-10T09:35:07.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:06.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:06.623573+0000) 2022-09-10T09:35:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:06 smithi110 ceph-mon[97633]: pgmap v9787: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:35:07.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:06 smithi104 ceph-mon[97044]: pgmap v9787: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:35:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:07.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:07.138235+0000) 2022-09-10T09:35:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:35:07.056+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:35:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:07.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:07.623818+0000) 2022-09-10T09:35:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:08.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:08.138428+0000) 2022-09-10T09:35:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:08.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:08.624036+0000) 2022-09-10T09:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:08 smithi104 ceph-mon[97044]: pgmap v9788: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:08 smithi104 ceph-mon[97044]: osdmap e692: 12 total, 12 up, 12 in 2022-09-10T09:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:08 smithi110 ceph-mon[97633]: pgmap v9788: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:08 smithi110 ceph-mon[97633]: osdmap e692: 12 total, 12 up, 12 in 2022-09-10T09:35:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:09.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:09.138648+0000) 2022-09-10T09:35:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:09.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:09.624212+0000) 2022-09-10T09:35:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:10.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:10.138885+0000) 2022-09-10T09:35:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:10.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:10.624411+0000) 2022-09-10T09:35:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:10 smithi104 ceph-mon[97044]: pgmap v9790: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:35:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:10 smithi110 ceph-mon[97633]: pgmap v9790: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:35:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:11.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:11.139205+0000) 2022-09-10T09:35:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:11.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:11.624615+0000) 2022-09-10T09:35:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:12.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:12.139368+0000) 2022-09-10T09:35:13.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:12.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:12.624827+0000) 2022-09-10T09:35:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:12 smithi110 ceph-mon[97633]: pgmap v9791: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:35:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:12 smithi110 ceph-mon[97633]: osdmap e693: 12 total, 12 up, 12 in 2022-09-10T09:35:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:12 smithi104 ceph-mon[97044]: pgmap v9791: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:35:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:12 smithi104 ceph-mon[97044]: osdmap e693: 12 total, 12 up, 12 in 2022-09-10T09:35:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:13.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:13.139634+0000) 2022-09-10T09:35:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:13.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:13.625044+0000) 2022-09-10T09:35:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:14.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:14.139773+0000) 2022-09-10T09:35:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:14.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:14.625309+0000) 2022-09-10T09:35:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:14 smithi110 ceph-mon[97633]: pgmap v9793: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:35:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:14 smithi104 ceph-mon[97044]: pgmap v9793: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:35:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:15.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:15.139971+0000) 2022-09-10T09:35:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:15.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:15.625564+0000) 2022-09-10T09:35:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:16.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:16.140230+0000) 2022-09-10T09:35:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:16.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:16.625793+0000) 2022-09-10T09:35:17.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:16 smithi110 ceph-mon[97633]: pgmap v9794: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:35:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:16 smithi104 ceph-mon[97044]: pgmap v9794: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:35:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:17.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:17.140432+0000) 2022-09-10T09:35:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:17.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:17.626064+0000) 2022-09-10T09:35:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:18.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:18.140650+0000) 2022-09-10T09:35:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:18.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:18.626350+0000) 2022-09-10T09:35:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:18 smithi110 ceph-mon[97633]: pgmap v9795: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:35:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:18 smithi104 ceph-mon[97044]: pgmap v9795: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:35:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:19.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:19.140908+0000) 2022-09-10T09:35:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:19.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:19.626534+0000) 2022-09-10T09:35:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:20.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:20.141205+0000) 2022-09-10T09:35:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:20.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:20.626779+0000) 2022-09-10T09:35:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:20 smithi110 ceph-mon[97633]: pgmap v9796: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:35:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:20 smithi104 ceph-mon[97044]: pgmap v9796: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:35:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:21.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:21.141456+0000) 2022-09-10T09:35:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:21.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:21.627014+0000) 2022-09-10T09:35:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:35:22.071+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:35:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:22.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:22.141667+0000) 2022-09-10T09:35:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:22.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:22.627224+0000) 2022-09-10T09:35:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:22 smithi110 ceph-mon[97633]: pgmap v9797: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:35:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:22 smithi104 ceph-mon[97044]: pgmap v9797: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:35:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:23.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:23.141907+0000) 2022-09-10T09:35:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:23.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:23.627457+0000) 2022-09-10T09:35:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:24.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:24.142209+0000) 2022-09-10T09:35:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:24.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:24.627716+0000) 2022-09-10T09:35:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:24 smithi110 ceph-mon[97633]: pgmap v9798: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:35:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:24 smithi104 ceph-mon[97044]: pgmap v9798: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:35:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:25.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:25.142367+0000) 2022-09-10T09:35:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:25.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:25.627939+0000) 2022-09-10T09:35:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:26.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:26.142540+0000) 2022-09-10T09:35:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:26.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:26.628211+0000) 2022-09-10T09:35:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:26 smithi110 ceph-mon[97633]: pgmap v9799: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:35:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:26 smithi104 ceph-mon[97044]: pgmap v9799: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:35:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:27.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:27.142799+0000) 2022-09-10T09:35:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:27.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:27.628407+0000) 2022-09-10T09:35:28.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:28.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:28.143030+0000) 2022-09-10T09:35:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:28.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:28.628593+0000) 2022-09-10T09:35:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:28 smithi110 ceph-mon[97633]: pgmap v9800: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:28 smithi104 ceph-mon[97044]: pgmap v9800: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:29.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:29.143249+0000) 2022-09-10T09:35:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:29.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:29.628760+0000) 2022-09-10T09:35:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:30.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:30.143517+0000) 2022-09-10T09:35:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:30.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:30.628990+0000) 2022-09-10T09:35:31.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:30 smithi110 ceph-mon[97633]: pgmap v9801: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:30 smithi104 ceph-mon[97044]: pgmap v9801: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:31.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:31.143716+0000) 2022-09-10T09:35:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:31.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:31.629231+0000) 2022-09-10T09:35:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:32.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:32.143948+0000) 2022-09-10T09:35:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:32.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:32.629494+0000) 2022-09-10T09:35:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:32 smithi110 ceph-mon[97633]: pgmap v9802: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:32 smithi104 ceph-mon[97044]: pgmap v9802: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:33.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:33.144197+0000) 2022-09-10T09:35:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:33.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:33.629741+0000) 2022-09-10T09:35:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:34.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:34.144358+0000) 2022-09-10T09:35:35.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:34.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:34.629981+0000) 2022-09-10T09:35:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:34 smithi110 ceph-mon[97633]: pgmap v9803: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:34 smithi104 ceph-mon[97044]: pgmap v9803: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:35.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:35.144539+0000) 2022-09-10T09:35:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:35.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:35.630203+0000) 2022-09-10T09:35:36.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:36.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:36.144796+0000) 2022-09-10T09:35:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:36.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:36.630400+0000) 2022-09-10T09:35:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:36 smithi110 ceph-mon[97633]: pgmap v9804: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:36 smithi104 ceph-mon[97044]: pgmap v9804: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:37.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:37.145078+0000) 2022-09-10T09:35:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:35:37.084+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:35:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:37.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:37.630655+0000) 2022-09-10T09:35:38.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:38.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:38.145317+0000) 2022-09-10T09:35:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:38.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:38.630894+0000) 2022-09-10T09:35:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:38 smithi110 ceph-mon[97633]: pgmap v9805: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:38 smithi104 ceph-mon[97044]: pgmap v9805: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:39.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:39.145563+0000) 2022-09-10T09:35:40.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:39.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:39.631197+0000) 2022-09-10T09:35:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:40.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:40.145821+0000) 2022-09-10T09:35:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:40.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:40.631342+0000) 2022-09-10T09:35:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:40 smithi110 ceph-mon[97633]: pgmap v9806: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:40 smithi104 ceph-mon[97044]: pgmap v9806: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:41.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:41.146036+0000) 2022-09-10T09:35:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:41.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:41.631504+0000) 2022-09-10T09:35:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:42.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:42.146278+0000) 2022-09-10T09:35:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:42.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:42.631645+0000) 2022-09-10T09:35:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:42 smithi110 ceph-mon[97633]: pgmap v9807: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:42 smithi104 ceph-mon[97044]: pgmap v9807: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:43.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:43.146546+0000) 2022-09-10T09:35:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:43.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:43.631851+0000) 2022-09-10T09:35:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:44.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:44.146734+0000) 2022-09-10T09:35:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:44.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:44.632131+0000) 2022-09-10T09:35:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:44 smithi110 ceph-mon[97633]: pgmap v9808: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:44 smithi104 ceph-mon[97044]: pgmap v9808: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:45.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:45.146963+0000) 2022-09-10T09:35:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:45.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:45.632309+0000) 2022-09-10T09:35:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:35:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:35:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:35:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:46.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:46.147211+0000) 2022-09-10T09:35:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:46.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:46.632495+0000) 2022-09-10T09:35:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:46 smithi110 ceph-mon[97633]: pgmap v9809: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:46 smithi104 ceph-mon[97044]: pgmap v9809: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:47.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:47.147420+0000) 2022-09-10T09:35:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:47.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:47.632697+0000) 2022-09-10T09:35:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:48.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:48.147659+0000) 2022-09-10T09:35:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:48.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:48.632885+0000) 2022-09-10T09:35:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:48 smithi110 ceph-mon[97633]: pgmap v9810: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:49.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:48 smithi104 ceph-mon[97044]: pgmap v9810: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:49.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:49.147918+0000) 2022-09-10T09:35:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:49.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:49.633061+0000) 2022-09-10T09:35:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:50.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:50.148187+0000) 2022-09-10T09:35:51.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:50.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:50.633230+0000) 2022-09-10T09:35:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:50 smithi110 ceph-mon[97633]: pgmap v9811: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:50 smithi104 ceph-mon[97044]: pgmap v9811: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:51.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:51.148403+0000) 2022-09-10T09:35:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:51.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:51.633393+0000) 2022-09-10T09:35:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:52.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:52.148610+0000) 2022-09-10T09:35:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:35:52.096+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:35:53.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:52.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:52.633530+0000) 2022-09-10T09:35:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:52 smithi110 ceph-mon[97633]: pgmap v9812: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:52 smithi104 ceph-mon[97044]: pgmap v9812: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:53.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:53.148762+0000) 2022-09-10T09:35:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:53.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:53.633743+0000) 2022-09-10T09:35:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:54.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:54.148990+0000) 2022-09-10T09:35:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:54.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:54.633973+0000) 2022-09-10T09:35:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:54 smithi110 ceph-mon[97633]: pgmap v9813: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:54 smithi104 ceph-mon[97044]: pgmap v9813: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:55.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:55.149208+0000) 2022-09-10T09:35:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:55.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:55.634203+0000) 2022-09-10T09:35:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:56.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:56.149357+0000) 2022-09-10T09:35:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:56.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:56.634373+0000) 2022-09-10T09:35:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:56 smithi110 ceph-mon[97633]: pgmap v9814: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:56 smithi104 ceph-mon[97044]: pgmap v9814: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:35:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:57.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:57.149542+0000) 2022-09-10T09:35:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:57.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:57.634578+0000) 2022-09-10T09:35:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:58.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:58.149772+0000) 2022-09-10T09:35:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:58.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:58.634747+0000) 2022-09-10T09:35:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:35:58 smithi110 ceph-mon[97633]: pgmap v9815: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:35:58 smithi104 ceph-mon[97044]: pgmap v9815: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:35:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:35:59.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:59.150027+0000) 2022-09-10T09:36:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:35:59.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:35:59.634936+0000) 2022-09-10T09:36:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:00.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:00.150239+0000) 2022-09-10T09:36:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:00.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:00.635198+0000) 2022-09-10T09:36:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:00 smithi104 ceph-mon[97044]: pgmap v9816: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:00 smithi110 ceph-mon[97633]: pgmap v9816: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:01.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:01.150458+0000) 2022-09-10T09:36:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:01.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:01.635409+0000) 2022-09-10T09:36:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:02.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:02.150629+0000) 2022-09-10T09:36:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:02.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:02.635629+0000) 2022-09-10T09:36:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:02 smithi104 ceph-mon[97044]: pgmap v9817: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:02 smithi110 ceph-mon[97633]: pgmap v9817: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:03.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:03.150877+0000) 2022-09-10T09:36:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:03.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:03.635853+0000) 2022-09-10T09:36:04.569 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:04.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:04.151037+0000) 2022-09-10T09:36:05.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:04.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:04.636066+0000) 2022-09-10T09:36:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:04 smithi104 ceph-mon[97044]: pgmap v9818: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:05.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:04 smithi110 ceph-mon[97633]: pgmap v9818: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:05.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:05.151331+0000) 2022-09-10T09:36:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:05.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:05.636281+0000) 2022-09-10T09:36:06.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:06.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:06.151513+0000) 2022-09-10T09:36:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:06.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:06.636517+0000) 2022-09-10T09:36:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:06 smithi104 ceph-mon[97044]: pgmap v9819: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:36:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:06 smithi110 ceph-mon[97633]: pgmap v9819: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:36:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:07.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:07.151731+0000) 2022-09-10T09:36:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:36:07.099+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:36:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:07.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:07.636726+0000) 2022-09-10T09:36:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:08.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:08.151936+0000) 2022-09-10T09:36:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:08.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:08.636960+0000) 2022-09-10T09:36:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:08 smithi104 ceph-mon[97044]: pgmap v9820: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:36:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:08 smithi104 ceph-mon[97044]: osdmap e694: 12 total, 12 up, 12 in 2022-09-10T09:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:08 smithi110 ceph-mon[97633]: pgmap v9820: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:36:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:08 smithi110 ceph-mon[97633]: osdmap e694: 12 total, 12 up, 12 in 2022-09-10T09:36:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:09.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:09.152156+0000) 2022-09-10T09:36:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:09.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:09.637200+0000) 2022-09-10T09:36:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:10.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:10.152314+0000) 2022-09-10T09:36:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:10.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:10.637402+0000) 2022-09-10T09:36:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:10 smithi104 ceph-mon[97044]: pgmap v9822: 129 pgs: 129 active+clean; 8.8 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:36:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:10 smithi110 ceph-mon[97633]: pgmap v9822: 129 pgs: 129 active+clean; 8.8 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:36:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:11.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:11.152544+0000) 2022-09-10T09:36:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:11.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:11.637560+0000) 2022-09-10T09:36:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:12.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:12.152782+0000) 2022-09-10T09:36:13.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:12.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:12.637810+0000) 2022-09-10T09:36:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:12 smithi110 ceph-mon[97633]: pgmap v9823: 129 pgs: 129 active+clean; 8.8 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:36:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:12 smithi110 ceph-mon[97633]: osdmap e695: 12 total, 12 up, 12 in 2022-09-10T09:36:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:12 smithi104 ceph-mon[97044]: pgmap v9823: 129 pgs: 129 active+clean; 8.8 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:36:13.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:12 smithi104 ceph-mon[97044]: osdmap e695: 12 total, 12 up, 12 in 2022-09-10T09:36:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:13.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:13.153027+0000) 2022-09-10T09:36:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:13.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:13.638044+0000) 2022-09-10T09:36:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:14.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:14.153244+0000) 2022-09-10T09:36:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:14.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:14.638286+0000) 2022-09-10T09:36:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:14 smithi110 ceph-mon[97633]: pgmap v9825: 129 pgs: 129 active+clean; 8.8 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:36:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:14 smithi104 ceph-mon[97044]: pgmap v9825: 129 pgs: 129 active+clean; 8.8 MiB data, 958 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:36:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:15.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:15.153481+0000) 2022-09-10T09:36:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:15.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:15.638495+0000) 2022-09-10T09:36:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:16.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:16.153656+0000) 2022-09-10T09:36:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:16.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:16.638741+0000) 2022-09-10T09:36:17.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:16 smithi110 ceph-mon[97633]: pgmap v9826: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:36:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:16 smithi104 ceph-mon[97044]: pgmap v9826: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:36:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:17.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:17.153912+0000) 2022-09-10T09:36:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:17.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:17.638918+0000) 2022-09-10T09:36:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:18.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:18.154202+0000) 2022-09-10T09:36:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:18.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:18.639127+0000) 2022-09-10T09:36:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:18 smithi110 ceph-mon[97633]: pgmap v9827: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:36:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:18 smithi104 ceph-mon[97044]: pgmap v9827: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:36:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:19.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:19.154372+0000) 2022-09-10T09:36:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:19.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:19.639323+0000) 2022-09-10T09:36:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:20.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:20.154579+0000) 2022-09-10T09:36:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:20.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:20.639512+0000) 2022-09-10T09:36:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:20 smithi110 ceph-mon[97633]: pgmap v9828: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:36:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:20 smithi104 ceph-mon[97044]: pgmap v9828: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:36:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:21.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:21.154838+0000) 2022-09-10T09:36:22.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:21.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:21.639762+0000) 2022-09-10T09:36:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:36:22.114+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:36:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:22.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:22.155121+0000) 2022-09-10T09:36:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:22.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:22.640011+0000) 2022-09-10T09:36:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:22 smithi110 ceph-mon[97633]: pgmap v9829: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:36:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:22 smithi104 ceph-mon[97044]: pgmap v9829: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:36:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:23.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:23.155344+0000) 2022-09-10T09:36:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:23.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:23.640192+0000) 2022-09-10T09:36:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:24.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:24.155523+0000) 2022-09-10T09:36:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:24.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:24.640362+0000) 2022-09-10T09:36:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:24 smithi110 ceph-mon[97633]: pgmap v9830: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:36:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:24 smithi104 ceph-mon[97044]: pgmap v9830: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:36:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:25.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:25.155781+0000) 2022-09-10T09:36:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:25.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:25.640558+0000) 2022-09-10T09:36:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:26.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:26.156046+0000) 2022-09-10T09:36:27.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:26.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:26.640801+0000) 2022-09-10T09:36:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:26 smithi110 ceph-mon[97633]: pgmap v9831: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:36:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:26 smithi104 ceph-mon[97044]: pgmap v9831: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:36:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:27.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:27.156190+0000) 2022-09-10T09:36:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:27.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:27.640982+0000) 2022-09-10T09:36:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:28.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:28.156417+0000) 2022-09-10T09:36:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:28.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:28.641204+0000) 2022-09-10T09:36:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:28 smithi110 ceph-mon[97633]: pgmap v9832: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:28 smithi104 ceph-mon[97044]: pgmap v9832: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:29.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:29.156656+0000) 2022-09-10T09:36:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:29.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:29.641446+0000) 2022-09-10T09:36:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:30.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:30.156880+0000) 2022-09-10T09:36:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:30.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:30.641717+0000) 2022-09-10T09:36:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:30 smithi110 ceph-mon[97633]: pgmap v9833: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:30 smithi104 ceph-mon[97044]: pgmap v9833: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:31.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:31.157024+0000) 2022-09-10T09:36:32.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:31.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:31.641965+0000) 2022-09-10T09:36:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:32.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:32.157209+0000) 2022-09-10T09:36:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:32.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:32.642229+0000) 2022-09-10T09:36:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:32 smithi110 ceph-mon[97633]: pgmap v9834: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:33.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:32 smithi104 ceph-mon[97044]: pgmap v9834: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:33.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:33.157465+0000) 2022-09-10T09:36:34.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:33.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:33.642391+0000) 2022-09-10T09:36:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:34.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:34.157650+0000) 2022-09-10T09:36:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:34.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:34.642564+0000) 2022-09-10T09:36:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:34 smithi110 ceph-mon[97633]: pgmap v9835: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:35.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:34 smithi104 ceph-mon[97044]: pgmap v9835: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:35.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:35.157879+0000) 2022-09-10T09:36:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:35.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:35.642738+0000) 2022-09-10T09:36:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:36.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:36.158180+0000) 2022-09-10T09:36:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:36.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:36.642984+0000) 2022-09-10T09:36:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:36 smithi110 ceph-mon[97633]: pgmap v9836: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:37.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:36 smithi104 ceph-mon[97044]: pgmap v9836: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:37.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:37.158416+0000) 2022-09-10T09:36:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:36:37.116+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:36:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:37.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:37.643163+0000) 2022-09-10T09:36:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:38.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:38.158627+0000) 2022-09-10T09:36:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:38.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:38.643369+0000) 2022-09-10T09:36:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:38 smithi110 ceph-mon[97633]: pgmap v9837: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:38 smithi104 ceph-mon[97044]: pgmap v9837: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:39.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:39.158874+0000) 2022-09-10T09:36:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:39.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:39.643610+0000) 2022-09-10T09:36:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:40.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:40.159150+0000) 2022-09-10T09:36:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:40.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:40.643885+0000) 2022-09-10T09:36:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:40 smithi110 ceph-mon[97633]: pgmap v9838: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:40 smithi104 ceph-mon[97044]: pgmap v9838: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:41.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:41.159356+0000) 2022-09-10T09:36:42.064 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:41.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:41.644133+0000) 2022-09-10T09:36:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:42.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:42.159536+0000) 2022-09-10T09:36:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:42.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:42.644269+0000) 2022-09-10T09:36:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:42 smithi110 ceph-mon[97633]: pgmap v9839: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:42 smithi104 ceph-mon[97044]: pgmap v9839: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:43.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:43.159795+0000) 2022-09-10T09:36:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:43.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:43.644431+0000) 2022-09-10T09:36:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:44.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:44.160030+0000) 2022-09-10T09:36:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:44.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:44.644678+0000) 2022-09-10T09:36:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:44 smithi110 ceph-mon[97633]: pgmap v9840: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:44 smithi104 ceph-mon[97044]: pgmap v9840: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:45.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:45.160290+0000) 2022-09-10T09:36:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:45.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:45.644917+0000) 2022-09-10T09:36:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:36:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:36:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:36:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:36:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:46.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:46.160586+0000) 2022-09-10T09:36:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:46.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:46.645171+0000) 2022-09-10T09:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:46 smithi110 ceph-mon[97633]: pgmap v9841: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:46 smithi104 ceph-mon[97044]: pgmap v9841: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:47.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:47.160885+0000) 2022-09-10T09:36:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:47.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:47.645362+0000) 2022-09-10T09:36:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:48.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:48.161170+0000) 2022-09-10T09:36:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:48.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:48.645524+0000) 2022-09-10T09:36:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:48 smithi110 ceph-mon[97633]: pgmap v9842: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:48 smithi104 ceph-mon[97044]: pgmap v9842: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:49.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:49.161352+0000) 2022-09-10T09:36:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:49.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:49.645768+0000) 2022-09-10T09:36:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:50.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:50.161621+0000) 2022-09-10T09:36:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:50.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:50.645981+0000) 2022-09-10T09:36:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:50 smithi104 ceph-mon[97044]: pgmap v9843: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:50 smithi110 ceph-mon[97633]: pgmap v9843: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:51.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:51.161877+0000) 2022-09-10T09:36:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:51.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:51.646142+0000) 2022-09-10T09:36:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:52.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:52.162142+0000) 2022-09-10T09:36:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:36:52.120+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:36:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:52.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:52.646309+0000) 2022-09-10T09:36:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:52 smithi104 ceph-mon[97044]: pgmap v9844: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:52 smithi110 ceph-mon[97633]: pgmap v9844: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:53.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:53.162383+0000) 2022-09-10T09:36:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:53.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:53.646500+0000) 2022-09-10T09:36:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:54.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:54.162547+0000) 2022-09-10T09:36:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:54.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:54.646747+0000) 2022-09-10T09:36:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:54 smithi104 ceph-mon[97044]: pgmap v9845: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:54 smithi110 ceph-mon[97633]: pgmap v9845: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:55.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:55.162806+0000) 2022-09-10T09:36:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:55.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:55.646994+0000) 2022-09-10T09:36:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:56.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:56.163039+0000) 2022-09-10T09:36:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:56.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:56.647207+0000) 2022-09-10T09:36:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:56 smithi104 ceph-mon[97044]: pgmap v9846: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:56 smithi110 ceph-mon[97633]: pgmap v9846: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:36:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:57.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:57.163274+0000) 2022-09-10T09:36:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:57.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:57.647433+0000) 2022-09-10T09:36:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:58.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:58.163504+0000) 2022-09-10T09:36:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:58.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:58.647604+0000) 2022-09-10T09:36:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:36:58 smithi104 ceph-mon[97044]: pgmap v9847: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:36:58 smithi110 ceph-mon[97633]: pgmap v9847: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:36:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:36:59.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:59.163625+0000) 2022-09-10T09:37:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:36:59.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:36:59.647802+0000) 2022-09-10T09:37:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:00.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:00.163789+0000) 2022-09-10T09:37:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:00.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:00.648029+0000) 2022-09-10T09:37:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:00 smithi104 ceph-mon[97044]: pgmap v9848: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:00 smithi110 ceph-mon[97633]: pgmap v9848: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:01.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:01.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:01.164052+0000) 2022-09-10T09:37:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:01.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:01.648216+0000) 2022-09-10T09:37:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:02.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:02.164238+0000) 2022-09-10T09:37:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:02.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:02.648438+0000) 2022-09-10T09:37:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:02 smithi104 ceph-mon[97044]: pgmap v9849: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:02 smithi110 ceph-mon[97633]: pgmap v9849: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:03.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:03.164412+0000) 2022-09-10T09:37:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:03.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:03.648616+0000) 2022-09-10T09:37:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:04.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:04.164631+0000) 2022-09-10T09:37:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:04.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:04.648891+0000) 2022-09-10T09:37:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:04 smithi104 ceph-mon[97044]: pgmap v9850: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:04 smithi110 ceph-mon[97633]: pgmap v9850: 129 pgs: 129 active+clean; 8.8 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:05.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:05.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:05.164922+0000) 2022-09-10T09:37:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:05.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:05.649224+0000) 2022-09-10T09:37:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:06.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:06.165218+0000) 2022-09-10T09:37:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:06.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:06.649481+0000) 2022-09-10T09:37:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:06 smithi104 ceph-mon[97044]: pgmap v9851: 129 pgs: 129 active+clean; 8.8 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:37:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:06 smithi110 ceph-mon[97633]: pgmap v9851: 129 pgs: 129 active+clean; 8.8 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:37:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:37:07.135+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:37:07.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:07.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:07.165416+0000) 2022-09-10T09:37:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:07.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:07.649648+0000) 2022-09-10T09:37:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:08.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:08.165603+0000) 2022-09-10T09:37:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:08.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:08.649843+0000) 2022-09-10T09:37:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:08 smithi104 ceph-mon[97044]: pgmap v9852: 129 pgs: 129 active+clean; 8.8 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:37:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:08 smithi104 ceph-mon[97044]: osdmap e696: 12 total, 12 up, 12 in 2022-09-10T09:37:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:08 smithi110 ceph-mon[97633]: pgmap v9852: 129 pgs: 129 active+clean; 8.8 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:37:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:08 smithi110 ceph-mon[97633]: osdmap e696: 12 total, 12 up, 12 in 2022-09-10T09:37:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:09.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:09.165820+0000) 2022-09-10T09:37:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:09.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:09.650044+0000) 2022-09-10T09:37:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:10.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:10.166076+0000) 2022-09-10T09:37:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:10.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:10.650176+0000) 2022-09-10T09:37:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:10 smithi104 ceph-mon[97044]: pgmap v9854: 129 pgs: 129 active+clean; 8.8 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:37:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:10 smithi110 ceph-mon[97633]: pgmap v9854: 129 pgs: 129 active+clean; 8.8 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:37:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:11.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:11.166319+0000) 2022-09-10T09:37:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:11.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:11.650403+0000) 2022-09-10T09:37:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:12.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:12.166532+0000) 2022-09-10T09:37:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:12.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:12.650645+0000) 2022-09-10T09:37:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:12 smithi110 ceph-mon[97633]: pgmap v9855: 129 pgs: 129 active+clean; 8.8 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:37:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:12 smithi110 ceph-mon[97633]: osdmap e697: 12 total, 12 up, 12 in 2022-09-10T09:37:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:12 smithi104 ceph-mon[97044]: pgmap v9855: 129 pgs: 129 active+clean; 8.8 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:37:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:12 smithi104 ceph-mon[97044]: osdmap e697: 12 total, 12 up, 12 in 2022-09-10T09:37:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:13.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:13.166743+0000) 2022-09-10T09:37:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:13.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:13.650827+0000) 2022-09-10T09:37:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:14.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:14.166934+0000) 2022-09-10T09:37:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:14.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:14.651029+0000) 2022-09-10T09:37:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:14 smithi110 ceph-mon[97633]: pgmap v9857: 129 pgs: 129 active+clean; 8.8 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:37:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:14 smithi104 ceph-mon[97044]: pgmap v9857: 129 pgs: 129 active+clean; 8.8 MiB data, 952 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:37:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:15.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:15.167164+0000) 2022-09-10T09:37:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:15.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:15.651162+0000) 2022-09-10T09:37:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:16.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:16.167375+0000) 2022-09-10T09:37:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:16.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:16.651388+0000) 2022-09-10T09:37:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:16 smithi110 ceph-mon[97633]: pgmap v9858: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:37:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:16 smithi104 ceph-mon[97044]: pgmap v9858: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:37:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:17.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:17.167636+0000) 2022-09-10T09:37:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:17.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:17.651552+0000) 2022-09-10T09:37:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:18.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:18.167863+0000) 2022-09-10T09:37:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:18.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:18.651790+0000) 2022-09-10T09:37:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:18 smithi110 ceph-mon[97633]: pgmap v9859: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:37:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:18 smithi104 ceph-mon[97044]: pgmap v9859: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:37:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:19.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:19.168083+0000) 2022-09-10T09:37:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:19.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:19.651998+0000) 2022-09-10T09:37:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:20.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:20.168335+0000) 2022-09-10T09:37:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:20.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:20.652220+0000) 2022-09-10T09:37:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:20 smithi110 ceph-mon[97633]: pgmap v9860: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:37:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:20 smithi104 ceph-mon[97044]: pgmap v9860: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:37:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:21.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:21.168582+0000) 2022-09-10T09:37:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:21.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:21.652460+0000) 2022-09-10T09:37:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:37:22.147+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:37:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:22.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:22.168802+0000) 2022-09-10T09:37:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:22.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:22.652666+0000) 2022-09-10T09:37:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:22 smithi110 ceph-mon[97633]: pgmap v9861: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:37:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:22 smithi104 ceph-mon[97044]: pgmap v9861: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:37:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:23.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:23.168967+0000) 2022-09-10T09:37:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:23.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:23.652864+0000) 2022-09-10T09:37:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:24.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:24.169230+0000) 2022-09-10T09:37:25.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:24.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:24.653180+0000) 2022-09-10T09:37:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:24 smithi110 ceph-mon[97633]: pgmap v9862: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:37:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:24 smithi104 ceph-mon[97044]: pgmap v9862: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:37:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:25.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:25.169459+0000) 2022-09-10T09:37:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:25.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:25.653414+0000) 2022-09-10T09:37:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:26.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:26.169665+0000) 2022-09-10T09:37:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:26.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:26.653604+0000) 2022-09-10T09:37:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:26 smithi110 ceph-mon[97633]: pgmap v9863: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:37:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:26 smithi104 ceph-mon[97044]: pgmap v9863: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:37:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:27.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:27.169870+0000) 2022-09-10T09:37:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:27.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:27.653820+0000) 2022-09-10T09:37:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:28.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:28.170165+0000) 2022-09-10T09:37:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:28.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:28.654038+0000) 2022-09-10T09:37:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:28 smithi110 ceph-mon[97633]: pgmap v9864: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:28 smithi104 ceph-mon[97044]: pgmap v9864: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:29.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:29.170328+0000) 2022-09-10T09:37:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:29.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:29.654357+0000) 2022-09-10T09:37:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:30.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:30.170505+0000) 2022-09-10T09:37:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:30.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:30.654507+0000) 2022-09-10T09:37:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:30 smithi110 ceph-mon[97633]: pgmap v9865: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:30 smithi104 ceph-mon[97044]: pgmap v9865: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:31.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:31.170674+0000) 2022-09-10T09:37:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:31.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:31.654752+0000) 2022-09-10T09:37:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:32.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:32.170903+0000) 2022-09-10T09:37:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:32.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:32.655042+0000) 2022-09-10T09:37:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:32 smithi110 ceph-mon[97633]: pgmap v9866: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:32 smithi104 ceph-mon[97044]: pgmap v9866: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:33.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:33.171070+0000) 2022-09-10T09:37:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:33.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:33.655235+0000) 2022-09-10T09:37:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:34.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:34.171332+0000) 2022-09-10T09:37:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:34.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:34.655485+0000) 2022-09-10T09:37:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:34 smithi110 ceph-mon[97633]: pgmap v9867: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:34 smithi104 ceph-mon[97044]: pgmap v9867: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:35.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:35.171565+0000) 2022-09-10T09:37:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:35.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:35.655707+0000) 2022-09-10T09:37:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:36.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:36.171800+0000) 2022-09-10T09:37:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:36.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:36.655874+0000) 2022-09-10T09:37:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:36 smithi110 ceph-mon[97633]: pgmap v9868: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:36 smithi104 ceph-mon[97044]: pgmap v9868: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:37:37.161+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:37:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:37.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:37.172043+0000) 2022-09-10T09:37:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:37.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:37.656184+0000) 2022-09-10T09:37:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:38.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:38.172321+0000) 2022-09-10T09:37:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:38.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:38.656410+0000) 2022-09-10T09:37:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:38 smithi104 ceph-mon[97044]: pgmap v9869: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:37:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:38 smithi110 ceph-mon[97633]: pgmap v9869: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:37:39.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:39.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:39.172508+0000) 2022-09-10T09:37:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:39.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:39.656617+0000) 2022-09-10T09:37:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:40.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:40.172722+0000) 2022-09-10T09:37:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:40.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:40.656867+0000) 2022-09-10T09:37:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:40 smithi104 ceph-mon[97044]: pgmap v9870: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:40 smithi110 ceph-mon[97633]: pgmap v9870: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:41.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:41.172984+0000) 2022-09-10T09:37:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:41.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:41.657078+0000) 2022-09-10T09:37:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:42.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:42.173246+0000) 2022-09-10T09:37:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:42.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:42.657286+0000) 2022-09-10T09:37:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:42 smithi104 ceph-mon[97044]: pgmap v9871: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:37:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:42 smithi110 ceph-mon[97633]: pgmap v9871: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:37:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:43.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:43.173528+0000) 2022-09-10T09:37:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:43.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:43.657422+0000) 2022-09-10T09:37:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:44.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:44.173731+0000) 2022-09-10T09:37:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:44.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:44.657599+0000) 2022-09-10T09:37:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:44 smithi104 ceph-mon[97044]: pgmap v9872: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:37:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:44 smithi110 ceph-mon[97633]: pgmap v9872: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:37:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:45.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:45.173942+0000) 2022-09-10T09:37:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:45.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:45.657784+0000) 2022-09-10T09:37:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:37:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:37:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:37:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:37:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:37:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:46.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:46.174161+0000) 2022-09-10T09:37:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:46.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:46.658013+0000) 2022-09-10T09:37:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:46 smithi104 ceph-mon[97044]: pgmap v9873: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:46 smithi110 ceph-mon[97633]: pgmap v9873: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:47.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:47.174368+0000) 2022-09-10T09:37:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:47.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:47.658200+0000) 2022-09-10T09:37:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:48.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:48.174630+0000) 2022-09-10T09:37:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:48.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:48.658328+0000) 2022-09-10T09:37:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:48 smithi104 ceph-mon[97044]: pgmap v9874: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:48 smithi110 ceph-mon[97633]: pgmap v9874: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:49.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:49.174891+0000) 2022-09-10T09:37:50.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:49.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:49.658580+0000) 2022-09-10T09:37:50.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:50.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:50.175031+0000) 2022-09-10T09:37:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:50.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:50.658772+0000) 2022-09-10T09:37:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:50 smithi104 ceph-mon[97044]: pgmap v9875: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:50 smithi110 ceph-mon[97633]: pgmap v9875: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:51.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:51.175307+0000) 2022-09-10T09:37:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:51.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:51.658926+0000) 2022-09-10T09:37:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:52.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:52.175564+0000) 2022-09-10T09:37:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:37:52.176+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:37:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:52.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:52.659080+0000) 2022-09-10T09:37:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:52 smithi104 ceph-mon[97044]: pgmap v9876: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:52 smithi110 ceph-mon[97633]: pgmap v9876: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:53.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:53.175775+0000) 2022-09-10T09:37:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:53.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:53.659262+0000) 2022-09-10T09:37:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:54.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:54.176049+0000) 2022-09-10T09:37:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:54.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:54.659500+0000) 2022-09-10T09:37:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:54 smithi104 ceph-mon[97044]: pgmap v9877: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:54 smithi110 ceph-mon[97633]: pgmap v9877: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:55.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:55.176275+0000) 2022-09-10T09:37:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:55.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:55.659663+0000) 2022-09-10T09:37:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:56.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:56.176470+0000) 2022-09-10T09:37:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:56.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:56.659863+0000) 2022-09-10T09:37:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:56 smithi104 ceph-mon[97044]: pgmap v9878: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:56 smithi110 ceph-mon[97633]: pgmap v9878: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:37:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:57.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:57.176622+0000) 2022-09-10T09:37:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:57.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:57.660135+0000) 2022-09-10T09:37:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:58.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:58.176839+0000) 2022-09-10T09:37:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:58.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:58.660312+0000) 2022-09-10T09:37:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:37:58 smithi104 ceph-mon[97044]: pgmap v9879: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:37:58 smithi110 ceph-mon[97633]: pgmap v9879: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:37:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:37:59.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:59.177008+0000) 2022-09-10T09:38:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:37:59.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:37:59.660485+0000) 2022-09-10T09:38:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:00.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:00.177132+0000) 2022-09-10T09:38:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:00.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:00.660721+0000) 2022-09-10T09:38:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:00 smithi104 ceph-mon[97044]: pgmap v9880: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:00 smithi110 ceph-mon[97633]: pgmap v9880: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:01.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:01.177356+0000) 2022-09-10T09:38:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:01.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:01.660932+0000) 2022-09-10T09:38:02.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:02.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:02.177572+0000) 2022-09-10T09:38:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:02.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:02.661193+0000) 2022-09-10T09:38:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:02 smithi104 ceph-mon[97044]: pgmap v9881: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:02 smithi110 ceph-mon[97633]: pgmap v9881: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:03.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:03.177701+0000) 2022-09-10T09:38:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:03.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:03.661401+0000) 2022-09-10T09:38:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:04.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:04.177921+0000) 2022-09-10T09:38:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:04.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:04.661650+0000) 2022-09-10T09:38:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:04 smithi104 ceph-mon[97044]: pgmap v9882: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:04 smithi110 ceph-mon[97633]: pgmap v9882: 129 pgs: 129 active+clean; 8.8 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:05.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:05.178152+0000) 2022-09-10T09:38:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:05.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:05.661836+0000) 2022-09-10T09:38:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:06.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:06.178325+0000) 2022-09-10T09:38:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:06.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:06.662044+0000) 2022-09-10T09:38:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:06 smithi104 ceph-mon[97044]: pgmap v9883: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:38:07.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:06 smithi110 ceph-mon[97633]: pgmap v9883: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:38:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:07.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:07.178564+0000) 2022-09-10T09:38:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:38:07.183+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:38:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:07.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:07.662266+0000) 2022-09-10T09:38:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:08.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:08.178827+0000) 2022-09-10T09:38:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:08.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:08.662449+0000) 2022-09-10T09:38:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:08 smithi104 ceph-mon[97044]: pgmap v9884: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:38:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:08 smithi104 ceph-mon[97044]: osdmap e698: 12 total, 12 up, 12 in 2022-09-10T09:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:08 smithi110 ceph-mon[97633]: pgmap v9884: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:08 smithi110 ceph-mon[97633]: osdmap e698: 12 total, 12 up, 12 in 2022-09-10T09:38:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:09.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:09.179013+0000) 2022-09-10T09:38:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:09.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:09.662701+0000) 2022-09-10T09:38:10.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:10.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:10.179254+0000) 2022-09-10T09:38:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:10.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:10.662878+0000) 2022-09-10T09:38:11.306 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:10 smithi104 ceph-mon[97044]: pgmap v9886: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:38:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:10 smithi110 ceph-mon[97633]: pgmap v9886: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:38:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:11.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:11.179504+0000) 2022-09-10T09:38:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:11.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:11.663072+0000) 2022-09-10T09:38:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:12.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:12.179766+0000) 2022-09-10T09:38:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:12.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:12.663323+0000) 2022-09-10T09:38:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:12 smithi110 ceph-mon[97633]: pgmap v9887: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:38:13.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:12 smithi110 ceph-mon[97633]: osdmap e699: 12 total, 12 up, 12 in 2022-09-10T09:38:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:12 smithi104 ceph-mon[97044]: pgmap v9887: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:38:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:12 smithi104 ceph-mon[97044]: osdmap e699: 12 total, 12 up, 12 in 2022-09-10T09:38:13.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:13.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:13.180038+0000) 2022-09-10T09:38:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:13.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:13.663572+0000) 2022-09-10T09:38:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:14.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:14.180318+0000) 2022-09-10T09:38:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:14.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:14.663818+0000) 2022-09-10T09:38:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:14 smithi110 ceph-mon[97633]: pgmap v9889: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:38:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:14 smithi104 ceph-mon[97044]: pgmap v9889: 129 pgs: 129 active+clean; 8.8 MiB data, 954 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:38:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:15.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:15.180562+0000) 2022-09-10T09:38:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:15.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:15.663967+0000) 2022-09-10T09:38:16.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:16.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:16.180749+0000) 2022-09-10T09:38:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:16.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:16.664172+0000) 2022-09-10T09:38:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:16 smithi110 ceph-mon[97633]: pgmap v9890: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:38:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:16 smithi104 ceph-mon[97044]: pgmap v9890: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:38:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:17.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:17.180970+0000) 2022-09-10T09:38:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:17.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:17.664361+0000) 2022-09-10T09:38:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:18.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:18.181211+0000) 2022-09-10T09:38:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:18.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:18.664649+0000) 2022-09-10T09:38:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:18 smithi110 ceph-mon[97633]: pgmap v9891: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:38:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:18 smithi104 ceph-mon[97044]: pgmap v9891: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:38:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:19.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:19.181415+0000) 2022-09-10T09:38:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:19.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:19.664818+0000) 2022-09-10T09:38:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:20.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:20.181577+0000) 2022-09-10T09:38:21.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:20.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:20.665049+0000) 2022-09-10T09:38:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:20 smithi110 ceph-mon[97633]: pgmap v9892: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:38:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:20 smithi104 ceph-mon[97044]: pgmap v9892: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:38:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:21.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:21.181822+0000) 2022-09-10T09:38:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:21.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:21.665334+0000) 2022-09-10T09:38:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:22.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:22.182047+0000) 2022-09-10T09:38:22.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:38:22.199+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:38:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:22.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:22.665540+0000) 2022-09-10T09:38:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:22 smithi110 ceph-mon[97633]: pgmap v9893: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:38:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:22 smithi104 ceph-mon[97044]: pgmap v9893: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:38:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:23.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:23.182238+0000) 2022-09-10T09:38:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:23.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:23.665709+0000) 2022-09-10T09:38:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:24.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:24.182479+0000) 2022-09-10T09:38:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:24.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:24.665908+0000) 2022-09-10T09:38:25.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:24 smithi110 ceph-mon[97633]: pgmap v9894: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:38:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:24 smithi104 ceph-mon[97044]: pgmap v9894: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:38:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:25.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:25.182674+0000) 2022-09-10T09:38:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:25.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:25.666201+0000) 2022-09-10T09:38:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:26.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:26.182913+0000) 2022-09-10T09:38:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:26.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:26.666379+0000) 2022-09-10T09:38:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:26 smithi104 ceph-mon[97044]: pgmap v9895: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:38:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:26 smithi110 ceph-mon[97633]: pgmap v9895: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:38:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:27.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:27.183190+0000) 2022-09-10T09:38:28.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:27.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:27.666619+0000) 2022-09-10T09:38:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:28.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:28.183460+0000) 2022-09-10T09:38:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:28.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:28.666926+0000) 2022-09-10T09:38:29.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:28 smithi104 ceph-mon[97044]: pgmap v9896: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:29.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:28 smithi110 ceph-mon[97633]: pgmap v9896: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:29.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:29.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:29.183673+0000) 2022-09-10T09:38:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:29.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:29.667091+0000) 2022-09-10T09:38:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:30.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:30.183857+0000) 2022-09-10T09:38:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:30.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:30.667317+0000) 2022-09-10T09:38:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:30 smithi104 ceph-mon[97044]: pgmap v9897: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:30 smithi110 ceph-mon[97633]: pgmap v9897: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:31.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:31.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:31.184080+0000) 2022-09-10T09:38:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:31.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:31.667518+0000) 2022-09-10T09:38:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:32.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:32.184324+0000) 2022-09-10T09:38:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:32.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:32.667742+0000) 2022-09-10T09:38:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:32 smithi104 ceph-mon[97044]: pgmap v9898: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:32 smithi110 ceph-mon[97633]: pgmap v9898: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:33.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:33.184557+0000) 2022-09-10T09:38:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:33.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:33.667991+0000) 2022-09-10T09:38:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:34.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:34.184842+0000) 2022-09-10T09:38:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:34.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:34.668195+0000) 2022-09-10T09:38:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:34 smithi104 ceph-mon[97044]: pgmap v9899: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:34 smithi110 ceph-mon[97633]: pgmap v9899: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:35.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:35.185143+0000) 2022-09-10T09:38:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:35.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:35.668448+0000) 2022-09-10T09:38:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:36.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:36.185396+0000) 2022-09-10T09:38:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:36.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:36.668682+0000) 2022-09-10T09:38:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:36 smithi104 ceph-mon[97044]: pgmap v9900: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:36 smithi110 ceph-mon[97633]: pgmap v9900: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:38:37.205+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:38:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:37.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:37.185680+0000) 2022-09-10T09:38:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:37.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:37.668886+0000) 2022-09-10T09:38:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:38.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:38.185896+0000) 2022-09-10T09:38:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:38.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:38.669068+0000) 2022-09-10T09:38:39.306 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:38 smithi104 ceph-mon[97044]: pgmap v9901: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:38 smithi110 ceph-mon[97633]: pgmap v9901: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:39.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:39.186079+0000) 2022-09-10T09:38:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:39.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:39.669292+0000) 2022-09-10T09:38:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:40.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:40.186375+0000) 2022-09-10T09:38:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:40.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:40.669487+0000) 2022-09-10T09:38:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:40 smithi104 ceph-mon[97044]: pgmap v9902: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:40 smithi110 ceph-mon[97633]: pgmap v9902: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:41.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:41.186637+0000) 2022-09-10T09:38:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:41.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:41.669697+0000) 2022-09-10T09:38:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:42.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:42.186817+0000) 2022-09-10T09:38:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:42.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:42.669865+0000) 2022-09-10T09:38:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:42 smithi104 ceph-mon[97044]: pgmap v9903: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:42 smithi110 ceph-mon[97633]: pgmap v9903: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:43.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:43.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:43.187033+0000) 2022-09-10T09:38:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:43.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:43.670159+0000) 2022-09-10T09:38:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:44.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:44.187279+0000) 2022-09-10T09:38:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:44.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:44.670355+0000) 2022-09-10T09:38:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:44 smithi104 ceph-mon[97044]: pgmap v9904: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:44 smithi110 ceph-mon[97633]: pgmap v9904: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:45.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:45.187484+0000) 2022-09-10T09:38:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:45.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:45.670605+0000) 2022-09-10T09:38:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:38:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:38:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:38:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:38:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:38:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:46.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:46.187746+0000) 2022-09-10T09:38:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:46.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:46.670876+0000) 2022-09-10T09:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:46 smithi104 ceph-mon[97044]: pgmap v9905: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:46 smithi110 ceph-mon[97633]: pgmap v9905: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:47.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:47.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:47.187954+0000) 2022-09-10T09:38:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:47.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:47.671172+0000) 2022-09-10T09:38:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:48.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:48.188136+0000) 2022-09-10T09:38:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:48.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:48.671366+0000) 2022-09-10T09:38:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:48 smithi104 ceph-mon[97044]: pgmap v9906: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:48 smithi110 ceph-mon[97633]: pgmap v9906: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:49.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:49.188368+0000) 2022-09-10T09:38:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:49.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:49.671566+0000) 2022-09-10T09:38:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:50.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:50.188574+0000) 2022-09-10T09:38:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:50.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:50.671776+0000) 2022-09-10T09:38:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:50 smithi104 ceph-mon[97044]: pgmap v9907: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:50 smithi110 ceph-mon[97633]: pgmap v9907: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:51.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:51.188805+0000) 2022-09-10T09:38:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:51.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:51.671985+0000) 2022-09-10T09:38:52.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:52.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:52.189059+0000) 2022-09-10T09:38:52.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:38:52.209+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:38:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:52.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:52.672233+0000) 2022-09-10T09:38:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:52 smithi104 ceph-mon[97044]: pgmap v9908: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:52 smithi110 ceph-mon[97633]: pgmap v9908: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:53.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:53.189366+0000) 2022-09-10T09:38:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:53.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:53.672478+0000) 2022-09-10T09:38:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:54.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:54.189589+0000) 2022-09-10T09:38:55.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:54.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:54.672619+0000) 2022-09-10T09:38:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:54 smithi104 ceph-mon[97044]: pgmap v9909: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:54 smithi110 ceph-mon[97633]: pgmap v9909: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:55.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:55.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:55.189834+0000) 2022-09-10T09:38:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:55.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:55.672829+0000) 2022-09-10T09:38:56.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:56.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:56.190085+0000) 2022-09-10T09:38:57.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:56.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:56.673052+0000) 2022-09-10T09:38:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:56 smithi104 ceph-mon[97044]: pgmap v9910: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:56 smithi110 ceph-mon[97633]: pgmap v9910: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:38:57.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:57.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:57.190387+0000) 2022-09-10T09:38:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:57.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:57.673293+0000) 2022-09-10T09:38:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:58.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:58.190620+0000) 2022-09-10T09:38:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:58.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:58.673539+0000) 2022-09-10T09:38:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:38:58 smithi104 ceph-mon[97044]: pgmap v9911: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:38:58 smithi110 ceph-mon[97633]: pgmap v9911: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:38:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:38:59.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:59.190853+0000) 2022-09-10T09:39:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:38:59.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:38:59.673748+0000) 2022-09-10T09:39:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:00.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:00.191127+0000) 2022-09-10T09:39:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:00.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:00.673947+0000) 2022-09-10T09:39:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:00 smithi104 ceph-mon[97044]: pgmap v9912: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:00 smithi110 ceph-mon[97633]: pgmap v9912: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:01.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:01.191331+0000) 2022-09-10T09:39:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:01.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:01.674196+0000) 2022-09-10T09:39:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:02.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:02.191553+0000) 2022-09-10T09:39:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:02.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:02.674329+0000) 2022-09-10T09:39:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:02 smithi104 ceph-mon[97044]: pgmap v9913: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:02 smithi110 ceph-mon[97633]: pgmap v9913: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:03.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:03.191738+0000) 2022-09-10T09:39:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:03.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:03.674584+0000) 2022-09-10T09:39:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:04.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:04.192016+0000) 2022-09-10T09:39:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:04.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:04.674777+0000) 2022-09-10T09:39:05.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:04 smithi104 ceph-mon[97044]: pgmap v9914: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:04 smithi110 ceph-mon[97633]: pgmap v9914: 129 pgs: 129 active+clean; 8.8 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:05.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:05.192158+0000) 2022-09-10T09:39:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:05.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:05.674987+0000) 2022-09-10T09:39:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:06.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:06.192391+0000) 2022-09-10T09:39:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:06.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:06.675159+0000) 2022-09-10T09:39:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:06 smithi104 ceph-mon[97044]: pgmap v9915: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:39:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:06 smithi110 ceph-mon[97633]: pgmap v9915: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:39:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:07.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:07.192642+0000) 2022-09-10T09:39:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:39:07.210+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:39:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:07.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:07.675306+0000) 2022-09-10T09:39:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:08.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:08.192862+0000) 2022-09-10T09:39:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:08.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:08.675542+0000) 2022-09-10T09:39:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:08 smithi104 ceph-mon[97044]: pgmap v9916: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:39:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:08 smithi104 ceph-mon[97044]: osdmap e700: 12 total, 12 up, 12 in 2022-09-10T09:39:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:08 smithi110 ceph-mon[97633]: pgmap v9916: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:39:09.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:08 smithi110 ceph-mon[97633]: osdmap e700: 12 total, 12 up, 12 in 2022-09-10T09:39:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:09.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:09.193112+0000) 2022-09-10T09:39:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:09.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:09.675737+0000) 2022-09-10T09:39:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:10.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:10.193360+0000) 2022-09-10T09:39:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:10.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:10.675915+0000) 2022-09-10T09:39:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:10 smithi104 ceph-mon[97044]: pgmap v9918: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:39:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:10 smithi110 ceph-mon[97633]: pgmap v9918: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:39:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:11.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:11.193558+0000) 2022-09-10T09:39:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:11.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:11.676192+0000) 2022-09-10T09:39:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:12.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:12.193756+0000) 2022-09-10T09:39:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:12.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:12.676348+0000) 2022-09-10T09:39:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:12 smithi110 ceph-mon[97633]: pgmap v9919: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:39:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:12 smithi110 ceph-mon[97633]: osdmap e701: 12 total, 12 up, 12 in 2022-09-10T09:39:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:12 smithi104 ceph-mon[97044]: pgmap v9919: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:39:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:12 smithi104 ceph-mon[97044]: osdmap e701: 12 total, 12 up, 12 in 2022-09-10T09:39:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:13.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:13.193994+0000) 2022-09-10T09:39:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:13.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:13.676532+0000) 2022-09-10T09:39:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:14.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:14.194212+0000) 2022-09-10T09:39:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:14.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:14.676770+0000) 2022-09-10T09:39:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:14 smithi110 ceph-mon[97633]: pgmap v9921: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:39:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:14 smithi104 ceph-mon[97044]: pgmap v9921: 129 pgs: 129 active+clean; 8.8 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:39:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:15.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:15.194449+0000) 2022-09-10T09:39:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:15.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:15.676971+0000) 2022-09-10T09:39:16.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:16.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:16.194611+0000) 2022-09-10T09:39:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:16.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:16.677100+0000) 2022-09-10T09:39:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:16 smithi110 ceph-mon[97633]: pgmap v9922: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:39:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:16 smithi104 ceph-mon[97044]: pgmap v9922: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:39:17.497 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:17.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:17.194754+0000) 2022-09-10T09:39:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:17.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:17.677246+0000) 2022-09-10T09:39:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:18.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:18.194887+0000) 2022-09-10T09:39:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:18.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:18.677387+0000) 2022-09-10T09:39:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:18 smithi104 ceph-mon[97044]: pgmap v9923: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:39:19.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:18 smithi110 ceph-mon[97633]: pgmap v9923: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:39:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:19.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:19.195041+0000) 2022-09-10T09:39:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:19.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:19.677520+0000) 2022-09-10T09:39:20.523 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:20.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:20.195201+0000) 2022-09-10T09:39:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:20.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:20.677675+0000) 2022-09-10T09:39:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:20 smithi104 ceph-mon[97044]: pgmap v9924: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:39:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:20 smithi110 ceph-mon[97633]: pgmap v9924: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:39:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:21.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:21.195348+0000) 2022-09-10T09:39:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:21.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:21.677824+0000) 2022-09-10T09:39:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:39:22.226+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:39:22.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:22.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:22.195470+0000) 2022-09-10T09:39:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:22.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:22.677944+0000) 2022-09-10T09:39:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:22 smithi104 ceph-mon[97044]: pgmap v9925: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:39:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:22 smithi110 ceph-mon[97633]: pgmap v9925: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:39:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:23.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:23.195637+0000) 2022-09-10T09:39:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:23.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:23.678086+0000) 2022-09-10T09:39:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:24.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:24.195741+0000) 2022-09-10T09:39:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:24.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:24.678210+0000) 2022-09-10T09:39:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:24 smithi104 ceph-mon[97044]: pgmap v9926: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:39:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:24 smithi110 ceph-mon[97633]: pgmap v9926: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:39:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:25.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:25.195907+0000) 2022-09-10T09:39:26.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:25.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:25.678365+0000) 2022-09-10T09:39:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:26.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:26.196019+0000) 2022-09-10T09:39:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:26.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:26.678511+0000) 2022-09-10T09:39:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:26 smithi104 ceph-mon[97044]: pgmap v9927: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:39:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:26 smithi110 ceph-mon[97633]: pgmap v9927: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:39:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:27.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:27.196193+0000) 2022-09-10T09:39:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:27.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:27.678664+0000) 2022-09-10T09:39:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:28.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:28.196347+0000) 2022-09-10T09:39:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:28.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:28.678774+0000) 2022-09-10T09:39:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:28 smithi104 ceph-mon[97044]: pgmap v9928: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:39:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:28 smithi110 ceph-mon[97633]: pgmap v9928: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:39:29.533 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:29.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:29.196502+0000) 2022-09-10T09:39:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:29.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:29.678928+0000) 2022-09-10T09:39:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:30.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:30.196633+0000) 2022-09-10T09:39:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:30.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:30.679067+0000) 2022-09-10T09:39:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:30 smithi104 ceph-mon[97044]: pgmap v9929: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:30 smithi110 ceph-mon[97633]: pgmap v9929: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:31.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:31.196788+0000) 2022-09-10T09:39:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:31.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:31.679232+0000) 2022-09-10T09:39:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:32.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:32.196900+0000) 2022-09-10T09:39:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:32.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:32.679377+0000) 2022-09-10T09:39:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:32 smithi104 ceph-mon[97044]: pgmap v9930: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:39:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:32 smithi110 ceph-mon[97633]: pgmap v9930: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:39:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:33.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:33.197050+0000) 2022-09-10T09:39:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:33.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:33.679533+0000) 2022-09-10T09:39:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:34.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:34.197205+0000) 2022-09-10T09:39:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:34.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:34.679705+0000) 2022-09-10T09:39:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:34 smithi104 ceph-mon[97044]: pgmap v9931: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:39:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:34 smithi110 ceph-mon[97633]: pgmap v9931: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:39:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:35.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:35.197329+0000) 2022-09-10T09:39:36.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:35.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:35.679860+0000) 2022-09-10T09:39:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:36.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:36.197487+0000) 2022-09-10T09:39:37.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:36.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:36.680013+0000) 2022-09-10T09:39:37.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:39:37.241+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:39:37.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:36 smithi104 ceph-mon[97044]: pgmap v9932: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:37.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:37.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:37.197647+0000) 2022-09-10T09:39:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:36 smithi110 ceph-mon[97633]: pgmap v9932: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:37.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:37.680167+0000) 2022-09-10T09:39:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:38.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:38.197802+0000) 2022-09-10T09:39:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:38.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:38.680324+0000) 2022-09-10T09:39:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:38 smithi104 ceph-mon[97044]: pgmap v9933: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:38 smithi110 ceph-mon[97633]: pgmap v9933: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:39.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:39.197930+0000) 2022-09-10T09:39:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:39.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:39.680471+0000) 2022-09-10T09:39:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:40.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:40.198091+0000) 2022-09-10T09:39:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:40.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:40.680616+0000) 2022-09-10T09:39:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:40 smithi104 ceph-mon[97044]: pgmap v9934: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:40 smithi110 ceph-mon[97633]: pgmap v9934: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:41.542 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:41.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:41.198244+0000) 2022-09-10T09:39:41.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:41.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:41.680782+0000) 2022-09-10T09:39:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:42.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:42.198493+0000) 2022-09-10T09:39:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:42.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:42.680994+0000) 2022-09-10T09:39:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:42 smithi104 ceph-mon[97044]: pgmap v9935: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:42 smithi110 ceph-mon[97633]: pgmap v9935: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:43.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:43.198636+0000) 2022-09-10T09:39:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:43.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:43.681177+0000) 2022-09-10T09:39:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:44.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:44.198777+0000) 2022-09-10T09:39:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:44.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:44.681316+0000) 2022-09-10T09:39:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:44 smithi104 ceph-mon[97044]: pgmap v9936: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:44 smithi110 ceph-mon[97633]: pgmap v9936: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:45.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:45.199025+0000) 2022-09-10T09:39:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:45.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:45.681504+0000) 2022-09-10T09:39:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:39:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:39:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:39:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:39:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:39:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:39:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:39:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:39:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:46.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:46.199270+0000) 2022-09-10T09:39:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:46.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:46.681728+0000) 2022-09-10T09:39:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:46 smithi104 ceph-mon[97044]: pgmap v9937: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:46 smithi110 ceph-mon[97633]: pgmap v9937: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:47.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:47.199454+0000) 2022-09-10T09:39:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:47.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:47.681975+0000) 2022-09-10T09:39:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:48.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:48.199637+0000) 2022-09-10T09:39:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:48.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:48.682147+0000) 2022-09-10T09:39:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:48 smithi104 ceph-mon[97044]: pgmap v9938: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:48 smithi110 ceph-mon[97633]: pgmap v9938: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:49.454 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:49.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:49.199824+0000) 2022-09-10T09:39:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:49.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:49.682396+0000) 2022-09-10T09:39:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:50.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:50.200017+0000) 2022-09-10T09:39:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:50.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:50.682591+0000) 2022-09-10T09:39:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:50 smithi104 ceph-mon[97044]: pgmap v9939: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:50 smithi110 ceph-mon[97633]: pgmap v9939: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:51.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:51.200142+0000) 2022-09-10T09:39:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:51.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:51.682841+0000) 2022-09-10T09:39:52.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:39:52.251+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:39:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:52.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:52.200322+0000) 2022-09-10T09:39:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:52.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:52.683119+0000) 2022-09-10T09:39:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:52 smithi104 ceph-mon[97044]: pgmap v9940: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:52 smithi110 ceph-mon[97633]: pgmap v9940: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:53.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:53.200535+0000) 2022-09-10T09:39:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:53.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:53.683320+0000) 2022-09-10T09:39:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:54.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:54.200774+0000) 2022-09-10T09:39:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:54.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:54.683598+0000) 2022-09-10T09:39:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:54 smithi104 ceph-mon[97044]: pgmap v9941: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:54 smithi110 ceph-mon[97633]: pgmap v9941: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:55.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:55.201017+0000) 2022-09-10T09:39:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:55.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:55.683821+0000) 2022-09-10T09:39:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:56.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:56.201324+0000) 2022-09-10T09:39:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:56.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:56.684082+0000) 2022-09-10T09:39:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:56 smithi104 ceph-mon[97044]: pgmap v9942: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:56 smithi110 ceph-mon[97633]: pgmap v9942: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:39:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:57.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:57.201615+0000) 2022-09-10T09:39:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:57.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:57.684307+0000) 2022-09-10T09:39:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:58.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:58.201872+0000) 2022-09-10T09:39:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:58.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:58.684464+0000) 2022-09-10T09:39:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:39:58 smithi104 ceph-mon[97044]: pgmap v9943: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:39:58 smithi110 ceph-mon[97633]: pgmap v9943: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:39:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:39:59.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:59.202109+0000) 2022-09-10T09:40:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:39:59.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:39:59.684620+0000) 2022-09-10T09:40:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:00.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:00.202337+0000) 2022-09-10T09:40:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:00.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:00.684824+0000) 2022-09-10T09:40:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:00 smithi104 ceph-mon[97044]: pgmap v9944: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:40:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:00 smithi110 ceph-mon[97633]: pgmap v9944: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:40:01.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:01.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:01.202619+0000) 2022-09-10T09:40:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:01.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:01.685000+0000) 2022-09-10T09:40:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:02.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:02.202844+0000) 2022-09-10T09:40:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:02.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:02.685227+0000) 2022-09-10T09:40:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:02 smithi104 ceph-mon[97044]: pgmap v9945: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:02 smithi110 ceph-mon[97633]: pgmap v9945: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:03.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:03.203061+0000) 2022-09-10T09:40:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:03.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:03.685391+0000) 2022-09-10T09:40:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:04.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:04.203324+0000) 2022-09-10T09:40:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:04.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:04.685599+0000) 2022-09-10T09:40:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:04 smithi104 ceph-mon[97044]: pgmap v9946: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:05.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:04 smithi110 ceph-mon[97633]: pgmap v9946: 129 pgs: 129 active+clean; 8.8 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:05.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:05.203507+0000) 2022-09-10T09:40:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:05.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:05.685817+0000) 2022-09-10T09:40:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:06.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:06.203733+0000) 2022-09-10T09:40:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:06.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:06.686015+0000) 2022-09-10T09:40:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:06 smithi104 ceph-mon[97044]: pgmap v9947: 129 pgs: 129 active+clean; 8.8 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:40:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:06 smithi110 ceph-mon[97633]: pgmap v9947: 129 pgs: 129 active+clean; 8.8 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:40:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:07.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:07.203904+0000) 2022-09-10T09:40:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:40:07.264+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:40:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:07.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:07.686260+0000) 2022-09-10T09:40:08.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:08.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:08.204153+0000) 2022-09-10T09:40:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:08.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:08.686489+0000) 2022-09-10T09:40:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:08 smithi104 ceph-mon[97044]: pgmap v9948: 129 pgs: 129 active+clean; 8.8 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:40:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:08 smithi104 ceph-mon[97044]: osdmap e702: 12 total, 12 up, 12 in 2022-09-10T09:40:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:08 smithi110 ceph-mon[97633]: pgmap v9948: 129 pgs: 129 active+clean; 8.8 MiB data, 962 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:40:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:08 smithi110 ceph-mon[97633]: osdmap e702: 12 total, 12 up, 12 in 2022-09-10T09:40:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:09.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:09.204298+0000) 2022-09-10T09:40:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:09.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:09.686729+0000) 2022-09-10T09:40:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:10.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:10.204441+0000) 2022-09-10T09:40:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:10.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:10.686978+0000) 2022-09-10T09:40:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:10 smithi104 ceph-mon[97044]: pgmap v9950: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:40:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:10 smithi110 ceph-mon[97633]: pgmap v9950: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:40:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:11.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:11.204657+0000) 2022-09-10T09:40:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:11.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:11.687162+0000) 2022-09-10T09:40:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:12.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:12.204935+0000) 2022-09-10T09:40:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:12.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:12.687355+0000) 2022-09-10T09:40:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:12 smithi104 ceph-mon[97044]: pgmap v9951: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:40:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:12 smithi104 ceph-mon[97044]: osdmap e703: 12 total, 12 up, 12 in 2022-09-10T09:40:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:12 smithi110 ceph-mon[97633]: pgmap v9951: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:40:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:12 smithi110 ceph-mon[97633]: osdmap e703: 12 total, 12 up, 12 in 2022-09-10T09:40:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:13.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:13.205172+0000) 2022-09-10T09:40:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:13.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:13.687580+0000) 2022-09-10T09:40:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:14.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:14.205291+0000) 2022-09-10T09:40:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:14.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:14.687842+0000) 2022-09-10T09:40:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:14 smithi104 ceph-mon[97044]: pgmap v9953: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:40:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:14 smithi110 ceph-mon[97633]: pgmap v9953: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:40:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:15.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:15.205517+0000) 2022-09-10T09:40:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:15.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:15.688080+0000) 2022-09-10T09:40:16.564 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:16.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:16.205721+0000) 2022-09-10T09:40:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:16.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:16.688352+0000) 2022-09-10T09:40:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:16 smithi104 ceph-mon[97044]: pgmap v9954: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:40:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:16 smithi110 ceph-mon[97633]: pgmap v9954: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:40:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:17.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:17.205983+0000) 2022-09-10T09:40:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:17.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:17.688490+0000) 2022-09-10T09:40:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:18.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:18.206235+0000) 2022-09-10T09:40:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:18.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:18.688730+0000) 2022-09-10T09:40:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:18 smithi104 ceph-mon[97044]: pgmap v9955: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:40:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:18 smithi110 ceph-mon[97633]: pgmap v9955: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:40:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:19.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:19.206413+0000) 2022-09-10T09:40:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:19.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:19.688980+0000) 2022-09-10T09:40:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:20.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:20.206587+0000) 2022-09-10T09:40:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:20.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:20.689223+0000) 2022-09-10T09:40:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:20 smithi104 ceph-mon[97044]: pgmap v9956: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:40:21.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:20 smithi110 ceph-mon[97633]: pgmap v9956: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:40:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:21.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:21.206846+0000) 2022-09-10T09:40:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:21.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:21.689433+0000) 2022-09-10T09:40:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:40:22.279+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:40:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:22.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:22.207078+0000) 2022-09-10T09:40:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:22.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:22.689608+0000) 2022-09-10T09:40:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:22 smithi104 ceph-mon[97044]: pgmap v9957: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:40:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:22 smithi110 ceph-mon[97633]: pgmap v9957: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:40:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:23.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:23.207321+0000) 2022-09-10T09:40:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:23.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:23.689840+0000) 2022-09-10T09:40:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:24.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:24.207567+0000) 2022-09-10T09:40:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:24.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:24.690128+0000) 2022-09-10T09:40:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:24 smithi104 ceph-mon[97044]: pgmap v9958: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:40:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:24 smithi110 ceph-mon[97633]: pgmap v9958: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:40:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:25.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:25.207856+0000) 2022-09-10T09:40:26.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:25.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:25.690339+0000) 2022-09-10T09:40:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:26.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:26.208134+0000) 2022-09-10T09:40:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:26.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:26.690565+0000) 2022-09-10T09:40:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:26 smithi104 ceph-mon[97044]: pgmap v9959: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:40:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:26 smithi110 ceph-mon[97633]: pgmap v9959: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:40:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:27.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:27.208333+0000) 2022-09-10T09:40:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:27.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:27.690810+0000) 2022-09-10T09:40:28.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:28.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:28.208532+0000) 2022-09-10T09:40:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:28.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:28.691055+0000) 2022-09-10T09:40:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:28 smithi104 ceph-mon[97044]: pgmap v9960: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:29.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:28 smithi110 ceph-mon[97633]: pgmap v9960: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:29.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:29.208699+0000) 2022-09-10T09:40:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:29.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:29.691240+0000) 2022-09-10T09:40:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:30.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:30.208920+0000) 2022-09-10T09:40:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:30.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:30.691482+0000) 2022-09-10T09:40:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:30 smithi104 ceph-mon[97044]: pgmap v9961: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:30 smithi110 ceph-mon[97633]: pgmap v9961: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:31.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:31.209178+0000) 2022-09-10T09:40:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:31.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:31.691706+0000) 2022-09-10T09:40:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:32.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:32.209428+0000) 2022-09-10T09:40:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:32.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:32.691963+0000) 2022-09-10T09:40:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:32 smithi104 ceph-mon[97044]: pgmap v9962: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:32 smithi110 ceph-mon[97633]: pgmap v9962: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:33.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:33.209684+0000) 2022-09-10T09:40:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:33.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:33.692182+0000) 2022-09-10T09:40:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:34.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:34.209874+0000) 2022-09-10T09:40:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:34.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:34.692395+0000) 2022-09-10T09:40:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:34 smithi104 ceph-mon[97044]: pgmap v9963: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:35.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:34 smithi110 ceph-mon[97633]: pgmap v9963: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:35.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:35.210032+0000) 2022-09-10T09:40:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:35.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:35.692552+0000) 2022-09-10T09:40:36.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:36.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:36.210269+0000) 2022-09-10T09:40:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:36.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:36.692770+0000) 2022-09-10T09:40:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:36 smithi104 ceph-mon[97044]: pgmap v9964: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:36 smithi110 ceph-mon[97633]: pgmap v9964: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:37.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:37.210500+0000) 2022-09-10T09:40:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:40:37.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:40:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:37.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:37.693009+0000) 2022-09-10T09:40:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:38.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:38.210748+0000) 2022-09-10T09:40:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:38.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:38.693276+0000) 2022-09-10T09:40:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:38 smithi104 ceph-mon[97044]: pgmap v9965: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:39.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:38 smithi110 ceph-mon[97633]: pgmap v9965: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:39.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:39.210901+0000) 2022-09-10T09:40:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:39.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:39.693469+0000) 2022-09-10T09:40:40.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:40.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:40.211084+0000) 2022-09-10T09:40:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:40.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:40.693567+0000) 2022-09-10T09:40:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:40 smithi104 ceph-mon[97044]: pgmap v9966: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:40 smithi110 ceph-mon[97633]: pgmap v9966: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:41.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:41.211226+0000) 2022-09-10T09:40:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:41.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:41.693789+0000) 2022-09-10T09:40:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:42.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:42.211441+0000) 2022-09-10T09:40:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:42.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:42.694011+0000) 2022-09-10T09:40:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:42 smithi104 ceph-mon[97044]: pgmap v9967: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:42 smithi110 ceph-mon[97633]: pgmap v9967: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:43.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:43.211683+0000) 2022-09-10T09:40:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:43.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:43.694248+0000) 2022-09-10T09:40:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:44.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:44.211854+0000) 2022-09-10T09:40:45.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:44.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:44.694474+0000) 2022-09-10T09:40:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:44 smithi104 ceph-mon[97044]: pgmap v9968: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:44 smithi110 ceph-mon[97633]: pgmap v9968: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:45.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:45.212056+0000) 2022-09-10T09:40:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:45.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:45.694670+0000) 2022-09-10T09:40:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:40:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:40:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:46.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:46.212233+0000) 2022-09-10T09:40:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:46.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:46.694880+0000) 2022-09-10T09:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:46 smithi104 ceph-mon[97044]: pgmap v9969: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:46 smithi110 ceph-mon[97633]: pgmap v9969: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:47.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:47.212448+0000) 2022-09-10T09:40:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:47.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:47.695079+0000) 2022-09-10T09:40:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:48.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:48.212662+0000) 2022-09-10T09:40:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:48.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:48.695276+0000) 2022-09-10T09:40:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:48 smithi104 ceph-mon[97044]: pgmap v9970: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:48 smithi110 ceph-mon[97633]: pgmap v9970: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:49.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:49.212841+0000) 2022-09-10T09:40:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:49.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:49.695530+0000) 2022-09-10T09:40:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:50.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:50.213038+0000) 2022-09-10T09:40:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:50.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:50.695718+0000) 2022-09-10T09:40:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:50 smithi104 ceph-mon[97044]: pgmap v9971: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:50 smithi110 ceph-mon[97633]: pgmap v9971: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:51.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:51.213263+0000) 2022-09-10T09:40:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:51.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:51.695917+0000) 2022-09-10T09:40:52.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:52.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:52.213517+0000) 2022-09-10T09:40:52.559 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:40:52.294+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:40:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:52.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:52.696166+0000) 2022-09-10T09:40:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:52 smithi104 ceph-mon[97044]: pgmap v9972: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:52 smithi110 ceph-mon[97633]: pgmap v9972: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:53.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:53.213733+0000) 2022-09-10T09:40:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:53.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:53.696290+0000) 2022-09-10T09:40:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:54.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:54.213968+0000) 2022-09-10T09:40:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:54.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:54.696508+0000) 2022-09-10T09:40:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:54 smithi104 ceph-mon[97044]: pgmap v9973: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:54 smithi110 ceph-mon[97633]: pgmap v9973: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:55.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:55.214161+0000) 2022-09-10T09:40:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:55.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:55.696727+0000) 2022-09-10T09:40:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:56.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:56.214363+0000) 2022-09-10T09:40:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:56.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:56.696989+0000) 2022-09-10T09:40:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:56 smithi104 ceph-mon[97044]: pgmap v9974: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:56 smithi110 ceph-mon[97633]: pgmap v9974: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:40:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:57.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:57.214536+0000) 2022-09-10T09:40:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:57.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:57.697186+0000) 2022-09-10T09:40:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:58.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:58.214777+0000) 2022-09-10T09:40:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:58.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:58.697410+0000) 2022-09-10T09:40:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:40:58 smithi104 ceph-mon[97044]: pgmap v9975: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:40:58 smithi110 ceph-mon[97633]: pgmap v9975: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:40:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:40:59.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:59.214977+0000) 2022-09-10T09:41:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:40:59.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:40:59.697611+0000) 2022-09-10T09:41:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:00.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:00.215224+0000) 2022-09-10T09:41:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:00.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:00.697897+0000) 2022-09-10T09:41:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:00 smithi104 ceph-mon[97044]: pgmap v9976: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:00 smithi110 ceph-mon[97633]: pgmap v9976: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:01.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:01.215403+0000) 2022-09-10T09:41:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:01.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:01.698130+0000) 2022-09-10T09:41:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:02.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:02.215592+0000) 2022-09-10T09:41:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:02.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:02.698352+0000) 2022-09-10T09:41:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:02 smithi104 ceph-mon[97044]: pgmap v9977: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:02 smithi110 ceph-mon[97633]: pgmap v9977: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:03.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:03.215771+0000) 2022-09-10T09:41:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:03.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:03.698590+0000) 2022-09-10T09:41:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:04.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:04.215933+0000) 2022-09-10T09:41:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:04.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:04.698848+0000) 2022-09-10T09:41:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:04 smithi104 ceph-mon[97044]: pgmap v9978: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:04 smithi110 ceph-mon[97633]: pgmap v9978: 129 pgs: 129 active+clean; 8.8 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:05.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:05.216169+0000) 2022-09-10T09:41:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:05.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:05.699122+0000) 2022-09-10T09:41:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:06.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:06.216400+0000) 2022-09-10T09:41:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:06.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:06.699372+0000) 2022-09-10T09:41:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:07 smithi104 ceph-mon[97044]: pgmap v9979: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:41:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:06 smithi110 ceph-mon[97633]: pgmap v9979: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:41:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:07.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:07.216583+0000) 2022-09-10T09:41:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:41:07.310+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:41:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:07.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:07.699541+0000) 2022-09-10T09:41:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:08.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:08.216772+0000) 2022-09-10T09:41:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:08.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:08.699699+0000) 2022-09-10T09:41:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:09 smithi104 ceph-mon[97044]: pgmap v9980: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:41:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:09 smithi104 ceph-mon[97044]: osdmap e704: 12 total, 12 up, 12 in 2022-09-10T09:41:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:09 smithi110 ceph-mon[97633]: pgmap v9980: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:41:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:09 smithi110 ceph-mon[97633]: osdmap e704: 12 total, 12 up, 12 in 2022-09-10T09:41:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:09.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:09.217032+0000) 2022-09-10T09:41:10.062 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:09.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:09.699916+0000) 2022-09-10T09:41:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:10.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:10.217278+0000) 2022-09-10T09:41:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:10.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:10.700156+0000) 2022-09-10T09:41:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:11 smithi104 ceph-mon[97044]: pgmap v9982: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:41:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:11 smithi110 ceph-mon[97633]: pgmap v9982: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:41:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:11.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:11.217493+0000) 2022-09-10T09:41:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:11.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:11.700278+0000) 2022-09-10T09:41:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:12.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:12.217687+0000) 2022-09-10T09:41:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:12.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:12.700519+0000) 2022-09-10T09:41:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:12 smithi104 ceph-mon[97044]: pgmap v9983: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:41:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:12 smithi104 ceph-mon[97044]: osdmap e705: 12 total, 12 up, 12 in 2022-09-10T09:41:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:12 smithi110 ceph-mon[97633]: pgmap v9983: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:41:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:12 smithi110 ceph-mon[97633]: osdmap e705: 12 total, 12 up, 12 in 2022-09-10T09:41:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:13.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:13.217917+0000) 2022-09-10T09:41:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:13.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:13.700731+0000) 2022-09-10T09:41:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:14.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:14.218085+0000) 2022-09-10T09:41:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:14.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:14.700886+0000) 2022-09-10T09:41:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:14 smithi104 ceph-mon[97044]: pgmap v9985: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:41:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:14 smithi110 ceph-mon[97633]: pgmap v9985: 129 pgs: 129 active+clean; 8.8 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:41:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:15.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:15.218282+0000) 2022-09-10T09:41:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:15.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:15.701053+0000) 2022-09-10T09:41:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:16.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:16.218470+0000) 2022-09-10T09:41:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:16.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:16.701265+0000) 2022-09-10T09:41:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:16 smithi104 ceph-mon[97044]: pgmap v9986: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:41:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:16 smithi110 ceph-mon[97633]: pgmap v9986: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:41:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:17.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:17.218730+0000) 2022-09-10T09:41:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:17.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:17.701409+0000) 2022-09-10T09:41:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:18.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:18.218946+0000) 2022-09-10T09:41:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:18.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:18.701629+0000) 2022-09-10T09:41:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:18 smithi104 ceph-mon[97044]: pgmap v9987: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:41:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:18 smithi110 ceph-mon[97633]: pgmap v9987: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:41:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:19.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:19.219128+0000) 2022-09-10T09:41:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:19.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:19.701823+0000) 2022-09-10T09:41:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:20.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:20.219351+0000) 2022-09-10T09:41:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:20.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:20.701991+0000) 2022-09-10T09:41:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:20 smithi104 ceph-mon[97044]: pgmap v9988: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:41:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:20 smithi110 ceph-mon[97633]: pgmap v9988: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:41:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:21.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:21.219594+0000) 2022-09-10T09:41:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:21.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:21.702165+0000) 2022-09-10T09:41:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:22.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:22.219765+0000) 2022-09-10T09:41:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:41:22.326+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:41:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:22.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:22.702398+0000) 2022-09-10T09:41:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:22 smithi104 ceph-mon[97044]: pgmap v9989: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:41:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:22 smithi110 ceph-mon[97633]: pgmap v9989: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:41:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:23.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:23.220018+0000) 2022-09-10T09:41:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:23.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:23.702648+0000) 2022-09-10T09:41:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:24.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:24.220237+0000) 2022-09-10T09:41:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:24.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:24.702911+0000) 2022-09-10T09:41:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:24 smithi104 ceph-mon[97044]: pgmap v9990: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:41:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:24 smithi110 ceph-mon[97633]: pgmap v9990: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:41:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:25.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:25.220513+0000) 2022-09-10T09:41:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:25.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:25.703139+0000) 2022-09-10T09:41:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:26.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:26.220717+0000) 2022-09-10T09:41:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:26.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:26.703351+0000) 2022-09-10T09:41:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:26 smithi104 ceph-mon[97044]: pgmap v9991: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:41:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:26 smithi110 ceph-mon[97633]: pgmap v9991: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:41:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:27.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:27.220917+0000) 2022-09-10T09:41:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:27.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:27.703615+0000) 2022-09-10T09:41:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:28.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:28.221202+0000) 2022-09-10T09:41:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:28.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:28.703869+0000) 2022-09-10T09:41:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:28 smithi104 ceph-mon[97044]: pgmap v9992: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:28 smithi110 ceph-mon[97633]: pgmap v9992: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:29.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:29.221346+0000) 2022-09-10T09:41:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:29.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:29.704154+0000) 2022-09-10T09:41:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:30.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:30.221548+0000) 2022-09-10T09:41:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:30.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:30.704352+0000) 2022-09-10T09:41:31.306 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:30 smithi104 ceph-mon[97044]: pgmap v9993: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:30 smithi110 ceph-mon[97633]: pgmap v9993: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:31.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:31.221718+0000) 2022-09-10T09:41:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:31.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:31.704515+0000) 2022-09-10T09:41:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:32.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:32.221995+0000) 2022-09-10T09:41:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:32.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:32.704748+0000) 2022-09-10T09:41:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:32 smithi104 ceph-mon[97044]: pgmap v9994: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:32 smithi110 ceph-mon[97633]: pgmap v9994: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:33.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:33.222233+0000) 2022-09-10T09:41:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:33.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:33.704978+0000) 2022-09-10T09:41:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:34.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:34.222439+0000) 2022-09-10T09:41:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:34.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:34.705250+0000) 2022-09-10T09:41:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:34 smithi104 ceph-mon[97044]: pgmap v9995: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:34 smithi110 ceph-mon[97633]: pgmap v9995: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:35.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:35.222700+0000) 2022-09-10T09:41:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:35.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:35.705495+0000) 2022-09-10T09:41:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:36.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:36.222915+0000) 2022-09-10T09:41:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:36.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:36.705760+0000) 2022-09-10T09:41:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:36 smithi104 ceph-mon[97044]: pgmap v9996: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:36 smithi110 ceph-mon[97633]: pgmap v9996: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:41:37.332+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:41:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:37.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:37.223221+0000) 2022-09-10T09:41:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:37.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:37.706011+0000) 2022-09-10T09:41:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:38.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:38.223435+0000) 2022-09-10T09:41:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:38.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:38.706268+0000) 2022-09-10T09:41:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:38 smithi104 ceph-mon[97044]: pgmap v9997: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:38 smithi110 ceph-mon[97633]: pgmap v9997: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:39.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:39.223631+0000) 2022-09-10T09:41:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:39.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:39.706435+0000) 2022-09-10T09:41:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:40.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:40.223875+0000) 2022-09-10T09:41:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:40.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:40.706654+0000) 2022-09-10T09:41:41.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:40 smithi104 ceph-mon[97044]: pgmap v9998: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:41.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:40 smithi110 ceph-mon[97633]: pgmap v9998: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:41.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:41.224127+0000) 2022-09-10T09:41:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:41.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:41.706890+0000) 2022-09-10T09:41:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:42.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:42.224346+0000) 2022-09-10T09:41:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:42.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:42.707181+0000) 2022-09-10T09:41:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:42 smithi104 ceph-mon[97044]: pgmap v9999: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:42 smithi110 ceph-mon[97633]: pgmap v9999: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:43.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:43.224533+0000) 2022-09-10T09:41:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:43.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:43.707401+0000) 2022-09-10T09:41:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:44.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:44.224785+0000) 2022-09-10T09:41:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:44.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:44.707625+0000) 2022-09-10T09:41:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:44 smithi104 ceph-mon[97044]: pgmap v10000: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:44 smithi110 ceph-mon[97633]: pgmap v10000: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:45.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:45.225040+0000) 2022-09-10T09:41:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:45.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:45.707764+0000) 2022-09-10T09:41:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:41:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:41:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:41:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:41:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:46.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:46.225280+0000) 2022-09-10T09:41:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:46.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:46.707967+0000) 2022-09-10T09:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:46 smithi104 ceph-mon[97044]: pgmap v10001: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:46 smithi110 ceph-mon[97633]: pgmap v10001: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:47.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:47.225535+0000) 2022-09-10T09:41:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:47.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:47.708215+0000) 2022-09-10T09:41:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:48.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:48.225813+0000) 2022-09-10T09:41:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:48.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:48.708398+0000) 2022-09-10T09:41:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:48 smithi104 ceph-mon[97044]: pgmap v10002: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:48 smithi110 ceph-mon[97633]: pgmap v10002: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:49.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:49.226065+0000) 2022-09-10T09:41:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:49.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:49.708606+0000) 2022-09-10T09:41:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:50.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:50.226270+0000) 2022-09-10T09:41:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:50.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:50.708813+0000) 2022-09-10T09:41:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:50 smithi104 ceph-mon[97044]: pgmap v10003: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:50 smithi110 ceph-mon[97633]: pgmap v10003: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:51.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:51.226475+0000) 2022-09-10T09:41:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:51.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:51.709025+0000) 2022-09-10T09:41:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:52.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:52.226743+0000) 2022-09-10T09:41:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:41:52.338+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:41:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:52.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:52.709288+0000) 2022-09-10T09:41:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:53 smithi104 ceph-mon[97044]: pgmap v10004: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:53 smithi110 ceph-mon[97633]: pgmap v10004: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:53.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:53.226950+0000) 2022-09-10T09:41:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:53.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:53.709494+0000) 2022-09-10T09:41:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:54.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:54.227184+0000) 2022-09-10T09:41:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:54.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:54.709707+0000) 2022-09-10T09:41:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:55 smithi104 ceph-mon[97044]: pgmap v10005: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:55 smithi110 ceph-mon[97633]: pgmap v10005: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:55.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:55.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:55.227299+0000) 2022-09-10T09:41:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:55.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:55.709968+0000) 2022-09-10T09:41:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:56.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:56.227482+0000) 2022-09-10T09:41:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:56.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:56.710196+0000) 2022-09-10T09:41:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:57 smithi104 ceph-mon[97044]: pgmap v10006: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:57 smithi110 ceph-mon[97633]: pgmap v10006: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:41:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:57.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:57.227710+0000) 2022-09-10T09:41:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:57.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:57.710466+0000) 2022-09-10T09:41:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:58.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:58.227909+0000) 2022-09-10T09:41:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:58.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:58.710752+0000) 2022-09-10T09:41:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:41:59 smithi104 ceph-mon[97044]: pgmap v10007: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:41:59 smithi110 ceph-mon[97633]: pgmap v10007: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:41:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:41:59.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:59.228198+0000) 2022-09-10T09:42:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:41:59.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:41:59.710991+0000) 2022-09-10T09:42:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:00.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:00.228416+0000) 2022-09-10T09:42:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:00.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:00.711144+0000) 2022-09-10T09:42:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:01 smithi104 ceph-mon[97044]: pgmap v10008: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:01 smithi110 ceph-mon[97633]: pgmap v10008: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:01.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:01.228562+0000) 2022-09-10T09:42:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:01.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:01.711343+0000) 2022-09-10T09:42:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:02.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:02.228738+0000) 2022-09-10T09:42:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:02.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:02.711592+0000) 2022-09-10T09:42:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:03 smithi104 ceph-mon[97044]: pgmap v10009: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:03 smithi110 ceph-mon[97633]: pgmap v10009: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:03.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:03.228965+0000) 2022-09-10T09:42:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:03.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:03.711846+0000) 2022-09-10T09:42:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:04.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:04.229192+0000) 2022-09-10T09:42:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:04.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:04.712066+0000) 2022-09-10T09:42:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:05 smithi104 ceph-mon[97044]: pgmap v10010: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:05 smithi110 ceph-mon[97633]: pgmap v10010: 129 pgs: 129 active+clean; 8.8 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:05.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:05.229377+0000) 2022-09-10T09:42:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:05.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:05.712325+0000) 2022-09-10T09:42:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:06.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:06.229622+0000) 2022-09-10T09:42:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:06.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:06.712544+0000) 2022-09-10T09:42:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:07 smithi104 ceph-mon[97044]: pgmap v10011: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:42:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:07 smithi110 ceph-mon[97633]: pgmap v10011: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:42:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:07.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:07.229848+0000) 2022-09-10T09:42:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:42:07.340+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:42:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:07.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:07.712699+0000) 2022-09-10T09:42:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:08.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:08.230122+0000) 2022-09-10T09:42:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:08.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:08.712967+0000) 2022-09-10T09:42:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:09 smithi104 ceph-mon[97044]: pgmap v10012: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:42:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:09 smithi104 ceph-mon[97044]: osdmap e706: 12 total, 12 up, 12 in 2022-09-10T09:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:09 smithi110 ceph-mon[97633]: pgmap v10012: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:09 smithi110 ceph-mon[97633]: osdmap e706: 12 total, 12 up, 12 in 2022-09-10T09:42:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:09.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:09.230358+0000) 2022-09-10T09:42:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:09.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:09.713214+0000) 2022-09-10T09:42:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:10.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:10.230588+0000) 2022-09-10T09:42:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:10.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:10.713371+0000) 2022-09-10T09:42:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:11 smithi104 ceph-mon[97044]: pgmap v10014: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:42:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:11 smithi110 ceph-mon[97633]: pgmap v10014: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:42:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:11.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:11.230775+0000) 2022-09-10T09:42:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:11.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:11.713619+0000) 2022-09-10T09:42:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:12.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:12.230995+0000) 2022-09-10T09:42:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:12.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:12.713881+0000) 2022-09-10T09:42:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:12 smithi104 ceph-mon[97044]: pgmap v10015: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:42:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:12 smithi104 ceph-mon[97044]: osdmap e707: 12 total, 12 up, 12 in 2022-09-10T09:42:13.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:12 smithi110 ceph-mon[97633]: pgmap v10015: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:42:13.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:12 smithi110 ceph-mon[97633]: osdmap e707: 12 total, 12 up, 12 in 2022-09-10T09:42:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:13.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:13.231209+0000) 2022-09-10T09:42:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:13.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:13.714160+0000) 2022-09-10T09:42:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:14.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:14.231445+0000) 2022-09-10T09:42:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:14.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:14.714328+0000) 2022-09-10T09:42:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:14 smithi104 ceph-mon[97044]: pgmap v10017: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:42:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:14 smithi110 ceph-mon[97633]: pgmap v10017: 129 pgs: 129 active+clean; 8.8 MiB data, 971 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:42:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:15.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:15.231668+0000) 2022-09-10T09:42:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:15.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:15.714459+0000) 2022-09-10T09:42:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:16.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:16.231873+0000) 2022-09-10T09:42:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:16.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:16.714665+0000) 2022-09-10T09:42:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:16 smithi104 ceph-mon[97044]: pgmap v10018: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:42:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:16 smithi110 ceph-mon[97633]: pgmap v10018: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:42:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:17.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:17.232071+0000) 2022-09-10T09:42:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:17.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:17.714920+0000) 2022-09-10T09:42:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:18.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:18.232262+0000) 2022-09-10T09:42:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:18.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:18.715080+0000) 2022-09-10T09:42:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:18 smithi104 ceph-mon[97044]: pgmap v10019: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:42:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:18 smithi110 ceph-mon[97633]: pgmap v10019: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T09:42:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:19.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:19.232528+0000) 2022-09-10T09:42:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:19.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:19.715290+0000) 2022-09-10T09:42:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:20.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:20.232736+0000) 2022-09-10T09:42:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:20.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:20.715528+0000) 2022-09-10T09:42:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:20 smithi104 ceph-mon[97044]: pgmap v10020: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:42:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:20 smithi110 ceph-mon[97633]: pgmap v10020: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:42:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:21.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:21.232986+0000) 2022-09-10T09:42:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:21.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:21.715689+0000) 2022-09-10T09:42:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:22.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:22.233254+0000) 2022-09-10T09:42:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:42:22.346+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:42:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:22.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:22.715940+0000) 2022-09-10T09:42:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:22 smithi104 ceph-mon[97044]: pgmap v10021: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:42:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:22 smithi110 ceph-mon[97633]: pgmap v10021: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:42:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:23.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:23.233457+0000) 2022-09-10T09:42:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:23.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:23.716159+0000) 2022-09-10T09:42:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:24.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:24.233689+0000) 2022-09-10T09:42:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:24.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:24.716429+0000) 2022-09-10T09:42:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:24 smithi104 ceph-mon[97044]: pgmap v10022: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:42:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:24 smithi110 ceph-mon[97633]: pgmap v10022: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:42:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:25.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:25.233911+0000) 2022-09-10T09:42:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:25.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:25.716628+0000) 2022-09-10T09:42:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:26.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:26.234159+0000) 2022-09-10T09:42:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:26.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:26.716901+0000) 2022-09-10T09:42:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:26 smithi104 ceph-mon[97044]: pgmap v10023: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:42:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:26 smithi110 ceph-mon[97633]: pgmap v10023: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:42:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:27.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:27.234396+0000) 2022-09-10T09:42:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:27.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:27.717137+0000) 2022-09-10T09:42:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:28.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:28.234624+0000) 2022-09-10T09:42:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:28.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:28.717273+0000) 2022-09-10T09:42:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:28 smithi104 ceph-mon[97044]: pgmap v10024: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:28 smithi110 ceph-mon[97633]: pgmap v10024: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:29.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:29.234831+0000) 2022-09-10T09:42:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:29.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:29.717457+0000) 2022-09-10T09:42:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:30.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:30.234999+0000) 2022-09-10T09:42:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:30.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:30.717747+0000) 2022-09-10T09:42:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:30 smithi104 ceph-mon[97044]: pgmap v10025: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:30 smithi110 ceph-mon[97633]: pgmap v10025: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:31.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:31.235173+0000) 2022-09-10T09:42:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:31.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:31.717939+0000) 2022-09-10T09:42:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:32.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:32.235366+0000) 2022-09-10T09:42:33.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:32.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:32.718286+0000) 2022-09-10T09:42:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:32 smithi104 ceph-mon[97044]: pgmap v10026: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:32 smithi110 ceph-mon[97633]: pgmap v10026: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:33.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:33.235540+0000) 2022-09-10T09:42:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:33.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:33.718520+0000) 2022-09-10T09:42:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:34.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:34.235804+0000) 2022-09-10T09:42:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:34.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:34.718798+0000) 2022-09-10T09:42:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:34 smithi104 ceph-mon[97044]: pgmap v10027: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:34 smithi110 ceph-mon[97633]: pgmap v10027: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:35.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:35.236008+0000) 2022-09-10T09:42:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:35.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:35.718987+0000) 2022-09-10T09:42:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:36.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:36.236264+0000) 2022-09-10T09:42:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:36.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:36.719211+0000) 2022-09-10T09:42:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:36 smithi104 ceph-mon[97044]: pgmap v10028: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:36 smithi110 ceph-mon[97633]: pgmap v10028: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:37.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:42:37.361+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:42:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:37.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:37.236438+0000) 2022-09-10T09:42:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:37.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:37.719437+0000) 2022-09-10T09:42:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:38.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:38.236689+0000) 2022-09-10T09:42:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:38.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:38.719688+0000) 2022-09-10T09:42:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:38 smithi104 ceph-mon[97044]: pgmap v10029: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:38 smithi110 ceph-mon[97633]: pgmap v10029: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:39.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:39.236898+0000) 2022-09-10T09:42:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:39.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:39.719949+0000) 2022-09-10T09:42:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:40.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:40.237176+0000) 2022-09-10T09:42:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:40.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:40.720239+0000) 2022-09-10T09:42:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:40 smithi104 ceph-mon[97044]: pgmap v10030: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:40 smithi110 ceph-mon[97633]: pgmap v10030: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:41.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:41.237371+0000) 2022-09-10T09:42:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:41.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:41.720411+0000) 2022-09-10T09:42:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:42.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:42.237597+0000) 2022-09-10T09:42:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:42.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:42.720630+0000) 2022-09-10T09:42:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:42 smithi104 ceph-mon[97044]: pgmap v10031: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:42 smithi110 ceph-mon[97633]: pgmap v10031: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:43.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:43.237761+0000) 2022-09-10T09:42:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:43.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:43.720790+0000) 2022-09-10T09:42:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:44.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:44.238008+0000) 2022-09-10T09:42:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:44.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:44.721021+0000) 2022-09-10T09:42:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:44 smithi104 ceph-mon[97044]: pgmap v10032: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:44 smithi110 ceph-mon[97633]: pgmap v10032: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:45.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:45.238258+0000) 2022-09-10T09:42:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:45.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:45.721154+0000) 2022-09-10T09:42:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:42:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:42:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:42:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:42:46.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:42:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:46.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:46.238474+0000) 2022-09-10T09:42:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:46.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:46.721314+0000) 2022-09-10T09:42:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:47 smithi104 ceph-mon[97044]: pgmap v10033: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:47 smithi110 ceph-mon[97633]: pgmap v10033: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:47.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:47.238679+0000) 2022-09-10T09:42:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:47.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:47.721535+0000) 2022-09-10T09:42:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:48.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:48.238937+0000) 2022-09-10T09:42:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:48.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:48.721741+0000) 2022-09-10T09:42:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:49 smithi104 ceph-mon[97044]: pgmap v10034: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:49 smithi110 ceph-mon[97633]: pgmap v10034: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:49.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:49.239158+0000) 2022-09-10T09:42:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:49.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:49.721956+0000) 2022-09-10T09:42:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:50.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:50.239383+0000) 2022-09-10T09:42:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:50.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:50.722196+0000) 2022-09-10T09:42:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:51 smithi104 ceph-mon[97044]: pgmap v10035: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:51 smithi110 ceph-mon[97633]: pgmap v10035: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:51.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:51.239627+0000) 2022-09-10T09:42:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:51.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:51.722444+0000) 2022-09-10T09:42:52.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:42:52.377+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:42:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:52.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:52.239789+0000) 2022-09-10T09:42:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:52.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:52.722722+0000) 2022-09-10T09:42:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:53 smithi104 ceph-mon[97044]: pgmap v10036: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:53 smithi110 ceph-mon[97633]: pgmap v10036: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:53.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:53.239993+0000) 2022-09-10T09:42:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:53.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:53.722972+0000) 2022-09-10T09:42:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:54.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:54.240219+0000) 2022-09-10T09:42:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:54.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:54.723225+0000) 2022-09-10T09:42:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:55 smithi104 ceph-mon[97044]: pgmap v10037: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:55 smithi110 ceph-mon[97633]: pgmap v10037: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:55.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:55.240411+0000) 2022-09-10T09:42:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:55.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:55.723406+0000) 2022-09-10T09:42:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:56.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:56.240623+0000) 2022-09-10T09:42:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:56.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:56.723574+0000) 2022-09-10T09:42:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:57 smithi104 ceph-mon[97044]: pgmap v10038: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:57 smithi110 ceph-mon[97633]: pgmap v10038: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:42:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:57.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:57.240810+0000) 2022-09-10T09:42:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:57.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:57.723746+0000) 2022-09-10T09:42:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:58.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:58.240999+0000) 2022-09-10T09:42:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:58.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:58.723993+0000) 2022-09-10T09:42:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:42:59 smithi104 ceph-mon[97044]: pgmap v10039: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:42:59 smithi110 ceph-mon[97633]: pgmap v10039: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:42:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:42:59.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:59.241191+0000) 2022-09-10T09:43:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:42:59.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:42:59.724169+0000) 2022-09-10T09:43:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:00.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:00.241390+0000) 2022-09-10T09:43:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:00.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:00.724365+0000) 2022-09-10T09:43:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:01 smithi104 ceph-mon[97044]: pgmap v10040: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:01 smithi110 ceph-mon[97633]: pgmap v10040: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:01.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:01.241610+0000) 2022-09-10T09:43:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:01.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:01.724605+0000) 2022-09-10T09:43:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:02.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:02.241767+0000) 2022-09-10T09:43:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:02.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:02.724854+0000) 2022-09-10T09:43:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:03 smithi104 ceph-mon[97044]: pgmap v10041: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:03 smithi110 ceph-mon[97633]: pgmap v10041: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:03.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:03.242000+0000) 2022-09-10T09:43:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:03.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:03.725159+0000) 2022-09-10T09:43:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:04.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:04.242206+0000) 2022-09-10T09:43:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:04.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:04.725296+0000) 2022-09-10T09:43:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:05 smithi104 ceph-mon[97044]: pgmap v10042: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:05 smithi110 ceph-mon[97633]: pgmap v10042: 129 pgs: 129 active+clean; 8.8 MiB data, 972 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:05.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:05.242458+0000) 2022-09-10T09:43:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:05.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:05.725494+0000) 2022-09-10T09:43:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:06.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:06.242687+0000) 2022-09-10T09:43:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:06.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:06.725731+0000) 2022-09-10T09:43:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:07 smithi104 ceph-mon[97044]: pgmap v10043: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:43:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:07 smithi110 ceph-mon[97633]: pgmap v10043: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:43:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:07.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:07.242927+0000) 2022-09-10T09:43:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:43:07.390+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:43:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:07.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:07.725900+0000) 2022-09-10T09:43:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:08.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:08.243180+0000) 2022-09-10T09:43:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:08.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:08.726057+0000) 2022-09-10T09:43:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:09 smithi104 ceph-mon[97044]: pgmap v10044: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:43:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:09 smithi104 ceph-mon[97044]: osdmap e708: 12 total, 12 up, 12 in 2022-09-10T09:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:09 smithi110 ceph-mon[97633]: pgmap v10044: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:09 smithi110 ceph-mon[97633]: osdmap e708: 12 total, 12 up, 12 in 2022-09-10T09:43:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:09.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:09.243368+0000) 2022-09-10T09:43:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:09.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:09.726277+0000) 2022-09-10T09:43:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:10.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:10.243530+0000) 2022-09-10T09:43:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:10.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:10.726530+0000) 2022-09-10T09:43:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:11 smithi104 ceph-mon[97044]: pgmap v10046: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:43:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:11 smithi110 ceph-mon[97633]: pgmap v10046: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:43:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:11.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:11.243746+0000) 2022-09-10T09:43:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:11.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:11.726741+0000) 2022-09-10T09:43:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:12.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:12.243994+0000) 2022-09-10T09:43:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:12.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:12.726965+0000) 2022-09-10T09:43:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:12 smithi104 ceph-mon[97044]: pgmap v10047: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:43:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:12 smithi104 ceph-mon[97044]: osdmap e709: 12 total, 12 up, 12 in 2022-09-10T09:43:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:12 smithi110 ceph-mon[97633]: pgmap v10047: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:43:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:12 smithi110 ceph-mon[97633]: osdmap e709: 12 total, 12 up, 12 in 2022-09-10T09:43:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:13.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:13.244248+0000) 2022-09-10T09:43:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:13.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:13.727198+0000) 2022-09-10T09:43:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:14.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:14.244468+0000) 2022-09-10T09:43:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:14.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:14.727362+0000) 2022-09-10T09:43:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:14 smithi104 ceph-mon[97044]: pgmap v10049: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:43:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:14 smithi110 ceph-mon[97633]: pgmap v10049: 129 pgs: 129 active+clean; 8.8 MiB data, 973 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:43:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:15.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:15.244672+0000) 2022-09-10T09:43:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:15.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:15.727583+0000) 2022-09-10T09:43:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:16.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:16.244816+0000) 2022-09-10T09:43:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:16.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:16.727796+0000) 2022-09-10T09:43:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:16 smithi104 ceph-mon[97044]: pgmap v10050: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:43:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:16 smithi110 ceph-mon[97633]: pgmap v10050: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:43:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:17.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:17.245028+0000) 2022-09-10T09:43:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:17.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:17.728006+0000) 2022-09-10T09:43:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:18.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:18.245208+0000) 2022-09-10T09:43:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:18.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:18.728200+0000) 2022-09-10T09:43:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:18 smithi104 ceph-mon[97044]: pgmap v10051: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:43:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:18 smithi110 ceph-mon[97633]: pgmap v10051: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:43:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:19.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:19.245418+0000) 2022-09-10T09:43:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:19.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:19.728382+0000) 2022-09-10T09:43:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:20.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:20.245589+0000) 2022-09-10T09:43:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:20.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:20.728592+0000) 2022-09-10T09:43:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:20 smithi104 ceph-mon[97044]: pgmap v10052: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:43:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:20 smithi110 ceph-mon[97633]: pgmap v10052: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:43:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:21.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:21.245816+0000) 2022-09-10T09:43:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:21.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:21.728818+0000) 2022-09-10T09:43:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:22.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:22.245966+0000) 2022-09-10T09:43:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:43:22.403+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:43:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:22.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:22.729019+0000) 2022-09-10T09:43:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:22 smithi104 ceph-mon[97044]: pgmap v10053: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:43:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:22 smithi110 ceph-mon[97633]: pgmap v10053: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:43:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:23.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:23.246141+0000) 2022-09-10T09:43:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:23.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:23.729252+0000) 2022-09-10T09:43:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:24.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:24.246364+0000) 2022-09-10T09:43:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:24.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:24.729421+0000) 2022-09-10T09:43:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:24 smithi104 ceph-mon[97044]: pgmap v10054: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:43:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:24 smithi110 ceph-mon[97633]: pgmap v10054: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:43:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:25.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:25.246565+0000) 2022-09-10T09:43:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:25.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:25.729577+0000) 2022-09-10T09:43:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:26.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:26.246764+0000) 2022-09-10T09:43:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:26.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:26.729835+0000) 2022-09-10T09:43:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:26 smithi104 ceph-mon[97044]: pgmap v10055: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:43:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:26 smithi110 ceph-mon[97633]: pgmap v10055: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:43:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:27.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:27.246925+0000) 2022-09-10T09:43:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:27.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:27.729997+0000) 2022-09-10T09:43:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:28.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:28.247140+0000) 2022-09-10T09:43:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:28.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:28.730166+0000) 2022-09-10T09:43:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:28 smithi104 ceph-mon[97044]: pgmap v10056: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:28 smithi110 ceph-mon[97633]: pgmap v10056: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:29.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:29.247344+0000) 2022-09-10T09:43:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:29.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:29.730377+0000) 2022-09-10T09:43:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:30.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:30.247591+0000) 2022-09-10T09:43:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:30.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:30.730632+0000) 2022-09-10T09:43:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:30 smithi104 ceph-mon[97044]: pgmap v10057: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:30 smithi110 ceph-mon[97633]: pgmap v10057: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:31.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:31.247804+0000) 2022-09-10T09:43:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:31.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:31.730873+0000) 2022-09-10T09:43:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:32.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:32.248046+0000) 2022-09-10T09:43:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:32.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:32.731121+0000) 2022-09-10T09:43:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:32 smithi104 ceph-mon[97044]: pgmap v10058: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:32 smithi110 ceph-mon[97633]: pgmap v10058: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:33.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:33.248270+0000) 2022-09-10T09:43:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:33.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:33.731373+0000) 2022-09-10T09:43:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:34.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:34.248490+0000) 2022-09-10T09:43:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:34.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:34.731523+0000) 2022-09-10T09:43:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:34 smithi104 ceph-mon[97044]: pgmap v10059: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:34 smithi110 ceph-mon[97633]: pgmap v10059: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:35.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:35.248682+0000) 2022-09-10T09:43:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:35.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:35.731700+0000) 2022-09-10T09:43:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:36.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:36.248930+0000) 2022-09-10T09:43:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:36.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:36.731941+0000) 2022-09-10T09:43:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:37 smithi104 ceph-mon[97044]: pgmap v10060: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:37 smithi110 ceph-mon[97633]: pgmap v10060: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:37.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:37.249204+0000) 2022-09-10T09:43:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:43:37.418+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:43:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:37.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:37.732146+0000) 2022-09-10T09:43:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:38.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:38.249370+0000) 2022-09-10T09:43:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:38.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:38.732373+0000) 2022-09-10T09:43:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:39 smithi104 ceph-mon[97044]: pgmap v10061: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:39 smithi110 ceph-mon[97633]: pgmap v10061: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:39.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:39.249573+0000) 2022-09-10T09:43:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:39.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:39.732597+0000) 2022-09-10T09:43:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:40.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:40.249812+0000) 2022-09-10T09:43:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:40.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:40.732798+0000) 2022-09-10T09:43:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:41 smithi104 ceph-mon[97044]: pgmap v10062: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:41 smithi110 ceph-mon[97633]: pgmap v10062: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:41.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:41.249951+0000) 2022-09-10T09:43:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:41.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:41.733028+0000) 2022-09-10T09:43:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:42.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:42.250235+0000) 2022-09-10T09:43:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:42.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:42.733295+0000) 2022-09-10T09:43:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:43 smithi104 ceph-mon[97044]: pgmap v10063: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:43 smithi110 ceph-mon[97633]: pgmap v10063: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:43.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:43.250458+0000) 2022-09-10T09:43:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:43.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:43.733481+0000) 2022-09-10T09:43:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:44.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:44.250601+0000) 2022-09-10T09:43:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:44.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:44.733647+0000) 2022-09-10T09:43:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:45 smithi104 ceph-mon[97044]: pgmap v10064: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:43:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:43:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:43:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:43:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:45 smithi110 ceph-mon[97633]: pgmap v10064: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:43:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:43:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:43:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:43:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:45.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:45.250839+0000) 2022-09-10T09:43:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:45.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:45.733821+0000) 2022-09-10T09:43:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:46.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:46.251045+0000) 2022-09-10T09:43:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:46.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:46.734008+0000) 2022-09-10T09:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:47 smithi104 ceph-mon[97044]: pgmap v10065: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:47 smithi110 ceph-mon[97633]: pgmap v10065: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:47.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:47.251340+0000) 2022-09-10T09:43:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:47.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:47.734209+0000) 2022-09-10T09:43:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:48.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:48.251619+0000) 2022-09-10T09:43:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:48.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:48.734446+0000) 2022-09-10T09:43:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:49 smithi104 ceph-mon[97044]: pgmap v10066: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:49 smithi110 ceph-mon[97633]: pgmap v10066: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:49.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:49.251774+0000) 2022-09-10T09:43:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:49.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:49.734707+0000) 2022-09-10T09:43:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:50.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:50.251989+0000) 2022-09-10T09:43:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:50.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:50.734884+0000) 2022-09-10T09:43:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:51 smithi104 ceph-mon[97044]: pgmap v10067: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:51 smithi110 ceph-mon[97633]: pgmap v10067: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:51.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:51.252184+0000) 2022-09-10T09:43:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:51.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:51.735159+0000) 2022-09-10T09:43:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:52.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:52.252395+0000) 2022-09-10T09:43:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:43:52.428+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:43:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:52.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:52.735397+0000) 2022-09-10T09:43:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:53 smithi104 ceph-mon[97044]: pgmap v10068: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:53 smithi110 ceph-mon[97633]: pgmap v10068: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:53.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:53.252626+0000) 2022-09-10T09:43:54.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:53.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:53.735603+0000) 2022-09-10T09:43:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:54.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:54.252768+0000) 2022-09-10T09:43:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:54.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:54.735782+0000) 2022-09-10T09:43:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:55 smithi104 ceph-mon[97044]: pgmap v10069: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:55 smithi110 ceph-mon[97633]: pgmap v10069: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:55.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:55.252988+0000) 2022-09-10T09:43:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:55.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:55.736023+0000) 2022-09-10T09:43:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:56.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:56.253253+0000) 2022-09-10T09:43:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:56.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:56.736244+0000) 2022-09-10T09:43:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:57 smithi104 ceph-mon[97044]: pgmap v10070: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:57 smithi110 ceph-mon[97633]: pgmap v10070: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:43:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:57.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:57.253467+0000) 2022-09-10T09:43:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:57.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:57.736482+0000) 2022-09-10T09:43:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:58.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:58.253746+0000) 2022-09-10T09:43:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:58.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:58.736641+0000) 2022-09-10T09:43:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:43:59 smithi104 ceph-mon[97044]: pgmap v10071: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:43:59 smithi110 ceph-mon[97633]: pgmap v10071: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:43:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:43:59.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:59.253945+0000) 2022-09-10T09:44:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:43:59.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:43:59.736854+0000) 2022-09-10T09:44:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:00.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:00.254185+0000) 2022-09-10T09:44:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:00.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:00.737092+0000) 2022-09-10T09:44:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:01 smithi104 ceph-mon[97044]: pgmap v10072: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:01 smithi110 ceph-mon[97633]: pgmap v10072: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:01.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:01.254414+0000) 2022-09-10T09:44:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:01.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:01.737332+0000) 2022-09-10T09:44:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:02.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:02.254584+0000) 2022-09-10T09:44:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:02.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:02.737569+0000) 2022-09-10T09:44:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:03 smithi104 ceph-mon[97044]: pgmap v10073: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:03 smithi110 ceph-mon[97633]: pgmap v10073: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:03.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:03.254766+0000) 2022-09-10T09:44:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:03.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:03.737718+0000) 2022-09-10T09:44:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:04.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:04.255044+0000) 2022-09-10T09:44:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:04.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:04.737875+0000) 2022-09-10T09:44:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:05 smithi104 ceph-mon[97044]: pgmap v10074: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:05 smithi110 ceph-mon[97633]: pgmap v10074: 129 pgs: 129 active+clean; 8.8 MiB data, 974 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:05.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:05.255220+0000) 2022-09-10T09:44:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:05.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:05.738091+0000) 2022-09-10T09:44:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:06.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:06.255395+0000) 2022-09-10T09:44:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:06.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:06.738361+0000) 2022-09-10T09:44:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:07 smithi104 ceph-mon[97044]: pgmap v10075: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:44:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:07 smithi110 ceph-mon[97633]: pgmap v10075: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:44:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:07.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:07.255575+0000) 2022-09-10T09:44:07.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:44:07.442+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:44:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:07.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:07.738617+0000) 2022-09-10T09:44:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:08.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:08.255851+0000) 2022-09-10T09:44:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:08.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:08.738870+0000) 2022-09-10T09:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:09 smithi110 ceph-mon[97633]: pgmap v10076: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:09 smithi110 ceph-mon[97633]: osdmap e710: 12 total, 12 up, 12 in 2022-09-10T09:44:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:09 smithi104 ceph-mon[97044]: pgmap v10076: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:44:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:09 smithi104 ceph-mon[97044]: osdmap e710: 12 total, 12 up, 12 in 2022-09-10T09:44:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:09.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:09.256067+0000) 2022-09-10T09:44:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:09.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:09.739142+0000) 2022-09-10T09:44:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:10.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:10.256323+0000) 2022-09-10T09:44:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:10.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:10.739351+0000) 2022-09-10T09:44:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:11 smithi110 ceph-mon[97633]: pgmap v10078: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:44:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:11 smithi104 ceph-mon[97044]: pgmap v10078: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:44:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:11.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:11.256486+0000) 2022-09-10T09:44:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:11.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:11.739613+0000) 2022-09-10T09:44:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:12.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:12.256671+0000) 2022-09-10T09:44:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:12.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:12.739865+0000) 2022-09-10T09:44:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:12 smithi104 ceph-mon[97044]: pgmap v10079: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:44:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:12 smithi104 ceph-mon[97044]: osdmap e711: 12 total, 12 up, 12 in 2022-09-10T09:44:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:12 smithi110 ceph-mon[97633]: pgmap v10079: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:44:13.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:12 smithi110 ceph-mon[97633]: osdmap e711: 12 total, 12 up, 12 in 2022-09-10T09:44:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:13.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:13.256933+0000) 2022-09-10T09:44:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:13.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:13.740079+0000) 2022-09-10T09:44:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:14.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:14.257195+0000) 2022-09-10T09:44:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:14.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:14.740285+0000) 2022-09-10T09:44:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:14 smithi104 ceph-mon[97044]: pgmap v10081: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:44:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:14 smithi110 ceph-mon[97633]: pgmap v10081: 129 pgs: 129 active+clean; 8.8 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:44:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:15.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:15.257400+0000) 2022-09-10T09:44:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:15.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:15.740468+0000) 2022-09-10T09:44:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:16.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:16.257643+0000) 2022-09-10T09:44:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:16.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:16.740736+0000) 2022-09-10T09:44:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:16 smithi104 ceph-mon[97044]: pgmap v10082: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:44:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:16 smithi110 ceph-mon[97633]: pgmap v10082: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:44:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:17.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:17.257906+0000) 2022-09-10T09:44:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:17.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:17.740979+0000) 2022-09-10T09:44:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:18.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:18.258154+0000) 2022-09-10T09:44:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:18.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:18.741207+0000) 2022-09-10T09:44:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:18 smithi104 ceph-mon[97044]: pgmap v10083: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:44:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:18 smithi110 ceph-mon[97633]: pgmap v10083: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:44:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:19.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:19.258348+0000) 2022-09-10T09:44:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:19.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:19.741406+0000) 2022-09-10T09:44:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:20.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:20.258561+0000) 2022-09-10T09:44:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:20.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:20.741635+0000) 2022-09-10T09:44:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:20 smithi104 ceph-mon[97044]: pgmap v10084: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:44:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:20 smithi110 ceph-mon[97633]: pgmap v10084: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:44:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:21.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:21.258726+0000) 2022-09-10T09:44:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:21.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:21.741922+0000) 2022-09-10T09:44:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:22.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:22.258966+0000) 2022-09-10T09:44:22.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:44:22.447+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:44:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:22.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:22.742188+0000) 2022-09-10T09:44:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:22 smithi104 ceph-mon[97044]: pgmap v10085: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:44:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:22 smithi110 ceph-mon[97633]: pgmap v10085: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:44:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:23.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:23.259210+0000) 2022-09-10T09:44:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:23.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:23.742404+0000) 2022-09-10T09:44:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:24.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:24.259418+0000) 2022-09-10T09:44:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:24.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:24.742601+0000) 2022-09-10T09:44:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:24 smithi104 ceph-mon[97044]: pgmap v10086: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:44:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:24 smithi110 ceph-mon[97633]: pgmap v10086: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:44:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:25.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:25.259625+0000) 2022-09-10T09:44:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:25.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:25.742812+0000) 2022-09-10T09:44:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:26.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:26.259863+0000) 2022-09-10T09:44:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:26.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:26.743013+0000) 2022-09-10T09:44:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:27 smithi104 ceph-mon[97044]: pgmap v10087: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:44:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:27 smithi110 ceph-mon[97633]: pgmap v10087: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:44:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:27.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:27.260081+0000) 2022-09-10T09:44:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:27.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:27.743243+0000) 2022-09-10T09:44:28.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:28.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:28.260291+0000) 2022-09-10T09:44:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:28.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:28.743463+0000) 2022-09-10T09:44:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:29 smithi104 ceph-mon[97044]: pgmap v10088: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:29 smithi110 ceph-mon[97633]: pgmap v10088: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:29.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:29.260471+0000) 2022-09-10T09:44:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:29.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:29.743720+0000) 2022-09-10T09:44:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:30.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:30.260725+0000) 2022-09-10T09:44:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:30.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:30.743913+0000) 2022-09-10T09:44:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:31 smithi104 ceph-mon[97044]: pgmap v10089: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:31 smithi110 ceph-mon[97633]: pgmap v10089: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:31.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:31.260862+0000) 2022-09-10T09:44:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:31.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:31.744188+0000) 2022-09-10T09:44:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:32.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:32.261124+0000) 2022-09-10T09:44:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:32.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:32.744404+0000) 2022-09-10T09:44:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:33 smithi104 ceph-mon[97044]: pgmap v10090: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:33 smithi110 ceph-mon[97633]: pgmap v10090: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:33.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:33.261343+0000) 2022-09-10T09:44:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:33.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:33.744566+0000) 2022-09-10T09:44:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:34.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:34.261559+0000) 2022-09-10T09:44:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:34.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:34.744792+0000) 2022-09-10T09:44:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:35 smithi104 ceph-mon[97044]: pgmap v10091: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:35 smithi110 ceph-mon[97633]: pgmap v10091: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:35.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:35.261772+0000) 2022-09-10T09:44:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:35.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:35.744954+0000) 2022-09-10T09:44:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:36.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:36.261977+0000) 2022-09-10T09:44:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:36.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:36.745192+0000) 2022-09-10T09:44:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:37 smithi104 ceph-mon[97044]: pgmap v10092: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:37 smithi110 ceph-mon[97633]: pgmap v10092: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:37.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:37.262276+0000) 2022-09-10T09:44:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:44:37.459+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:44:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:37.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:37.745393+0000) 2022-09-10T09:44:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:38.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:38.262481+0000) 2022-09-10T09:44:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:38.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:38.745644+0000) 2022-09-10T09:44:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:39 smithi104 ceph-mon[97044]: pgmap v10093: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:39 smithi110 ceph-mon[97633]: pgmap v10093: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:39.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:39.262653+0000) 2022-09-10T09:44:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:39.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:39.745905+0000) 2022-09-10T09:44:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:40.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:40.262902+0000) 2022-09-10T09:44:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:40.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:40.746163+0000) 2022-09-10T09:44:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:41 smithi104 ceph-mon[97044]: pgmap v10094: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:41 smithi110 ceph-mon[97633]: pgmap v10094: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:41.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:41.263127+0000) 2022-09-10T09:44:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:41.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:41.746354+0000) 2022-09-10T09:44:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:42.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:42.263289+0000) 2022-09-10T09:44:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:42.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:42.746511+0000) 2022-09-10T09:44:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:43 smithi104 ceph-mon[97044]: pgmap v10095: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:43 smithi110 ceph-mon[97633]: pgmap v10095: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:43.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:43.263503+0000) 2022-09-10T09:44:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:43.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:43.746699+0000) 2022-09-10T09:44:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:44.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:44.263684+0000) 2022-09-10T09:44:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:44.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:44.746972+0000) 2022-09-10T09:44:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:45 smithi104 ceph-mon[97044]: pgmap v10096: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:44:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:44:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:44:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:44:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:45 smithi110 ceph-mon[97633]: pgmap v10096: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:44:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:44:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:44:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:44:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:45.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:45.263861+0000) 2022-09-10T09:44:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:45.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:45.747230+0000) 2022-09-10T09:44:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:46.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:46.264089+0000) 2022-09-10T09:44:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:46.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:46.747483+0000) 2022-09-10T09:44:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:47 smithi104 ceph-mon[97044]: pgmap v10097: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:47 smithi110 ceph-mon[97633]: pgmap v10097: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:47.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:47.264333+0000) 2022-09-10T09:44:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:47.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:47.747660+0000) 2022-09-10T09:44:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:48.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:48.264501+0000) 2022-09-10T09:44:49.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:48.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:48.747835+0000) 2022-09-10T09:44:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:49 smithi104 ceph-mon[97044]: pgmap v10098: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:49 smithi110 ceph-mon[97633]: pgmap v10098: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:44:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:49.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:49.264671+0000) 2022-09-10T09:44:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:49.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:49.748062+0000) 2022-09-10T09:44:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:50.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:50.264900+0000) 2022-09-10T09:44:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:50.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:50.748273+0000) 2022-09-10T09:44:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:51 smithi104 ceph-mon[97044]: pgmap v10099: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:51 smithi110 ceph-mon[97633]: pgmap v10099: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:51.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:51.265175+0000) 2022-09-10T09:44:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:51.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:51.748467+0000) 2022-09-10T09:44:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:52.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:52.265316+0000) 2022-09-10T09:44:52.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:44:52.460+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:44:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:52.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:52.748738+0000) 2022-09-10T09:44:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:53 smithi104 ceph-mon[97044]: pgmap v10100: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:53 smithi110 ceph-mon[97633]: pgmap v10100: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:53.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:53.265516+0000) 2022-09-10T09:44:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:53.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:53.748879+0000) 2022-09-10T09:44:54.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:54.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:54.265769+0000) 2022-09-10T09:44:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:54.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:54.749110+0000) 2022-09-10T09:44:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:55 smithi104 ceph-mon[97044]: pgmap v10101: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:55 smithi110 ceph-mon[97633]: pgmap v10101: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:55.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:55.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:55.266009+0000) 2022-09-10T09:44:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:55.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:55.749270+0000) 2022-09-10T09:44:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:56.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:56.266226+0000) 2022-09-10T09:44:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:56.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:56.749456+0000) 2022-09-10T09:44:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:57 smithi110 ceph-mon[97633]: pgmap v10102: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:57 smithi104 ceph-mon[97044]: pgmap v10102: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:44:57.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:57.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:57.266451+0000) 2022-09-10T09:44:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:57.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:57.749658+0000) 2022-09-10T09:44:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:58.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:58.266711+0000) 2022-09-10T09:44:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:58.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:58.749925+0000) 2022-09-10T09:44:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:44:59 smithi110 ceph-mon[97633]: pgmap v10103: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:44:59 smithi104 ceph-mon[97044]: pgmap v10103: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:44:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:44:59.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:59.266907+0000) 2022-09-10T09:45:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:44:59.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:44:59.750158+0000) 2022-09-10T09:45:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:00.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:00.267147+0000) 2022-09-10T09:45:01.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:00.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:00.750376+0000) 2022-09-10T09:45:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:01 smithi110 ceph-mon[97633]: pgmap v10104: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:01.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:01.267305+0000) 2022-09-10T09:45:01.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:01 smithi104 ceph-mon[97044]: pgmap v10104: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:01.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:01.750566+0000) 2022-09-10T09:45:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:02.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:02.267532+0000) 2022-09-10T09:45:03.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:02.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:02.750726+0000) 2022-09-10T09:45:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:03 smithi110 ceph-mon[97633]: pgmap v10105: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:03 smithi104 ceph-mon[97044]: pgmap v10105: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:03.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:03.267790+0000) 2022-09-10T09:45:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:03.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:03.750952+0000) 2022-09-10T09:45:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:04.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:04.268048+0000) 2022-09-10T09:45:05.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:04.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:04.751209+0000) 2022-09-10T09:45:05.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:05 smithi110 ceph-mon[97633]: pgmap v10106: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:05 smithi104 ceph-mon[97044]: pgmap v10106: 129 pgs: 129 active+clean; 8.8 MiB data, 977 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:05.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:05.268274+0000) 2022-09-10T09:45:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:05.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:05.751407+0000) 2022-09-10T09:45:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:06.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:06.268408+0000) 2022-09-10T09:45:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:06.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:06.751678+0000) 2022-09-10T09:45:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:07 smithi110 ceph-mon[97633]: pgmap v10107: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:45:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:07 smithi104 ceph-mon[97044]: pgmap v10107: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:45:07.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:07.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:07.268631+0000) 2022-09-10T09:45:07.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:45:07.465+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:45:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:07.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:07.751933+0000) 2022-09-10T09:45:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:08.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:08.268804+0000) 2022-09-10T09:45:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:08.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:08.752191+0000) 2022-09-10T09:45:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:09 smithi110 ceph-mon[97633]: pgmap v10108: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:45:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:09 smithi110 ceph-mon[97633]: osdmap e712: 12 total, 12 up, 12 in 2022-09-10T09:45:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:09 smithi104 ceph-mon[97044]: pgmap v10108: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:45:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:09 smithi104 ceph-mon[97044]: osdmap e712: 12 total, 12 up, 12 in 2022-09-10T09:45:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:09.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:09.269063+0000) 2022-09-10T09:45:10.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:09.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:09.752418+0000) 2022-09-10T09:45:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:10.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:10.269336+0000) 2022-09-10T09:45:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:10.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:10.752564+0000) 2022-09-10T09:45:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:11 smithi110 ceph-mon[97633]: pgmap v10110: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:45:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:11 smithi104 ceph-mon[97044]: pgmap v10110: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:45:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:11.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:11.269550+0000) 2022-09-10T09:45:12.064 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:11.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:11.752797+0000) 2022-09-10T09:45:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:12.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:12.269780+0000) 2022-09-10T09:45:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:12.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:12.753089+0000) 2022-09-10T09:45:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:12 smithi104 ceph-mon[97044]: pgmap v10111: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:45:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:12 smithi104 ceph-mon[97044]: osdmap e713: 12 total, 12 up, 12 in 2022-09-10T09:45:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:12 smithi110 ceph-mon[97633]: pgmap v10111: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:45:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:12 smithi110 ceph-mon[97633]: osdmap e713: 12 total, 12 up, 12 in 2022-09-10T09:45:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:13.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:13.269941+0000) 2022-09-10T09:45:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:13.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:13.753284+0000) 2022-09-10T09:45:14.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:14.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:14.270165+0000) 2022-09-10T09:45:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:14.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:14.753529+0000) 2022-09-10T09:45:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:15 smithi104 ceph-mon[97044]: pgmap v10113: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:15 smithi110 ceph-mon[97633]: pgmap v10113: 129 pgs: 129 active+clean; 8.8 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:45:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:15.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:15.270362+0000) 2022-09-10T09:45:16.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:15.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:15.753770+0000) 2022-09-10T09:45:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:16.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:16.270497+0000) 2022-09-10T09:45:17.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:16.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:16.753935+0000) 2022-09-10T09:45:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:17 smithi104 ceph-mon[97044]: pgmap v10114: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:45:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:17 smithi110 ceph-mon[97633]: pgmap v10114: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:45:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:17.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:17.270691+0000) 2022-09-10T09:45:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:17.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:17.754060+0000) 2022-09-10T09:45:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:18.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:18.270907+0000) 2022-09-10T09:45:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:18.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:18.754336+0000) 2022-09-10T09:45:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:19 smithi104 ceph-mon[97044]: pgmap v10115: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:45:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:19 smithi110 ceph-mon[97633]: pgmap v10115: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:45:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:19.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:19.271180+0000) 2022-09-10T09:45:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:19.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:19.754555+0000) 2022-09-10T09:45:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:20.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:20.271393+0000) 2022-09-10T09:45:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:20.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:20.754719+0000) 2022-09-10T09:45:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:21 smithi104 ceph-mon[97044]: pgmap v10116: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:45:21.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:21 smithi110 ceph-mon[97633]: pgmap v10116: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:45:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:21.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:21.271561+0000) 2022-09-10T09:45:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:21.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:21.754950+0000) 2022-09-10T09:45:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:45:22.480+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:45:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:22.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:22.271814+0000) 2022-09-10T09:45:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:22.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:22.755216+0000) 2022-09-10T09:45:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:23 smithi104 ceph-mon[97044]: pgmap v10117: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:45:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:23 smithi110 ceph-mon[97633]: pgmap v10117: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:45:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:23.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:23.272027+0000) 2022-09-10T09:45:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:23.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:23.755362+0000) 2022-09-10T09:45:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:24.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:24.272266+0000) 2022-09-10T09:45:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:24.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:24.755594+0000) 2022-09-10T09:45:25.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:25 smithi104 ceph-mon[97044]: pgmap v10118: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:45:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:25 smithi110 ceph-mon[97633]: pgmap v10118: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:45:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:25.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:25.272513+0000) 2022-09-10T09:45:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:25.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:25.755787+0000) 2022-09-10T09:45:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:26.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:26.272679+0000) 2022-09-10T09:45:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:26.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:26.756016+0000) 2022-09-10T09:45:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:27 smithi104 ceph-mon[97044]: pgmap v10119: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:45:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:27 smithi110 ceph-mon[97633]: pgmap v10119: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:45:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:27.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:27.272889+0000) 2022-09-10T09:45:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:27.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:27.756209+0000) 2022-09-10T09:45:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:28.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:28.273121+0000) 2022-09-10T09:45:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:28.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:28.756363+0000) 2022-09-10T09:45:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:29 smithi104 ceph-mon[97044]: pgmap v10120: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:29 smithi110 ceph-mon[97633]: pgmap v10120: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:29.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:29.273393+0000) 2022-09-10T09:45:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:29.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:29.756542+0000) 2022-09-10T09:45:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:30.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:30.273539+0000) 2022-09-10T09:45:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:30.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:30.756782+0000) 2022-09-10T09:45:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:31 smithi104 ceph-mon[97044]: pgmap v10121: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:31 smithi110 ceph-mon[97633]: pgmap v10121: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:31.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:31.273700+0000) 2022-09-10T09:45:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:31.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:31.756987+0000) 2022-09-10T09:45:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:32.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:32.273901+0000) 2022-09-10T09:45:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:32.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:32.757175+0000) 2022-09-10T09:45:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:33 smithi104 ceph-mon[97044]: pgmap v10122: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:33.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:33 smithi110 ceph-mon[97633]: pgmap v10122: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:33.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:33.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:33.274088+0000) 2022-09-10T09:45:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:33.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:33.757344+0000) 2022-09-10T09:45:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:34.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:34.274296+0000) 2022-09-10T09:45:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:34.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:34.757518+0000) 2022-09-10T09:45:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:35 smithi104 ceph-mon[97044]: pgmap v10123: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:35 smithi110 ceph-mon[97633]: pgmap v10123: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:35.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:35.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:35.274445+0000) 2022-09-10T09:45:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:35.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:35.757690+0000) 2022-09-10T09:45:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:36.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:36.274668+0000) 2022-09-10T09:45:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:36.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:36.757923+0000) 2022-09-10T09:45:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:37 smithi104 ceph-mon[97044]: pgmap v10124: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:37 smithi110 ceph-mon[97633]: pgmap v10124: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:37.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:37.274942+0000) 2022-09-10T09:45:37.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:45:37.496+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:45:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:37.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:37.758196+0000) 2022-09-10T09:45:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:38.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:38.275165+0000) 2022-09-10T09:45:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:38.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:38.758405+0000) 2022-09-10T09:45:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:39 smithi104 ceph-mon[97044]: pgmap v10125: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:39 smithi110 ceph-mon[97633]: pgmap v10125: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:39.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:39.275296+0000) 2022-09-10T09:45:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:39.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:39.758564+0000) 2022-09-10T09:45:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:40.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:40.275475+0000) 2022-09-10T09:45:41.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:40.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:40.758759+0000) 2022-09-10T09:45:41.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:41 smithi104 ceph-mon[97044]: pgmap v10126: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:41 smithi110 ceph-mon[97633]: pgmap v10126: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:41.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:41.275652+0000) 2022-09-10T09:45:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:41.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:41.759006+0000) 2022-09-10T09:45:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:42.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:42.275905+0000) 2022-09-10T09:45:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:42.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:42.759288+0000) 2022-09-10T09:45:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:43 smithi110 ceph-mon[97633]: pgmap v10127: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:43 smithi104 ceph-mon[97044]: pgmap v10127: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:43.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:43.276091+0000) 2022-09-10T09:45:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:43.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:43.759521+0000) 2022-09-10T09:45:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:44.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:44.276288+0000) 2022-09-10T09:45:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:44.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:44.759765+0000) 2022-09-10T09:45:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:45 smithi110 ceph-mon[97633]: pgmap v10128: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:45:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:45:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:45:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:45:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:45 smithi104 ceph-mon[97044]: pgmap v10128: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:45:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:45:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:45:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:45:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:45.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:45.276487+0000) 2022-09-10T09:45:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:45.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:45.759936+0000) 2022-09-10T09:45:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:46.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:46.276645+0000) 2022-09-10T09:45:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:46.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:46.760148+0000) 2022-09-10T09:45:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:47 smithi110 ceph-mon[97633]: pgmap v10129: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:47 smithi104 ceph-mon[97044]: pgmap v10129: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:47.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:47.276886+0000) 2022-09-10T09:45:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:47.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:47.760399+0000) 2022-09-10T09:45:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:48.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:48.277125+0000) 2022-09-10T09:45:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:48.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:48.760649+0000) 2022-09-10T09:45:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:49 smithi110 ceph-mon[97633]: pgmap v10130: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:49 smithi104 ceph-mon[97044]: pgmap v10130: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:49.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:49.277283+0000) 2022-09-10T09:45:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:49.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:49.760848+0000) 2022-09-10T09:45:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:50.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:50.277479+0000) 2022-09-10T09:45:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:50.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:50.761135+0000) 2022-09-10T09:45:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:51 smithi110 ceph-mon[97633]: pgmap v10131: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:51 smithi104 ceph-mon[97044]: pgmap v10131: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:51.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:51.277642+0000) 2022-09-10T09:45:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:51.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:51.761341+0000) 2022-09-10T09:45:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:52.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:52.277894+0000) 2022-09-10T09:45:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:45:52.507+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:45:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:52.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:52.761638+0000) 2022-09-10T09:45:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:53 smithi110 ceph-mon[97633]: pgmap v10132: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:53 smithi104 ceph-mon[97044]: pgmap v10132: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:53.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:53.278161+0000) 2022-09-10T09:45:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:53.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:53.761820+0000) 2022-09-10T09:45:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:54.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:54.278392+0000) 2022-09-10T09:45:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:54.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:54.762045+0000) 2022-09-10T09:45:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:55 smithi110 ceph-mon[97633]: pgmap v10133: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:55 smithi104 ceph-mon[97044]: pgmap v10133: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:55.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:55.278570+0000) 2022-09-10T09:45:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:55.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:55.762231+0000) 2022-09-10T09:45:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:56.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:56.278763+0000) 2022-09-10T09:45:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:56.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:56.762410+0000) 2022-09-10T09:45:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:57 smithi110 ceph-mon[97633]: pgmap v10134: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:57 smithi104 ceph-mon[97044]: pgmap v10134: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:45:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:57.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:57.278945+0000) 2022-09-10T09:45:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:57.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:57.762605+0000) 2022-09-10T09:45:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:58.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:58.279137+0000) 2022-09-10T09:45:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:58.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:58.762837+0000) 2022-09-10T09:45:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:45:59 smithi110 ceph-mon[97633]: pgmap v10135: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:45:59 smithi104 ceph-mon[97044]: pgmap v10135: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:45:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:45:59.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:59.279352+0000) 2022-09-10T09:46:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:45:59.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:45:59.763121+0000) 2022-09-10T09:46:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:00.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:00.279485+0000) 2022-09-10T09:46:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:00.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:00.763329+0000) 2022-09-10T09:46:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:01 smithi110 ceph-mon[97633]: pgmap v10136: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:01 smithi104 ceph-mon[97044]: pgmap v10136: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:01.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:01.279668+0000) 2022-09-10T09:46:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:01.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:01.763583+0000) 2022-09-10T09:46:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:02.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:02.279876+0000) 2022-09-10T09:46:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:02.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:02.763910+0000) 2022-09-10T09:46:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:03 smithi110 ceph-mon[97633]: pgmap v10137: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:03 smithi104 ceph-mon[97044]: pgmap v10137: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:03.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:03.280085+0000) 2022-09-10T09:46:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:03.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:03.764201+0000) 2022-09-10T09:46:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:04.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:04.280335+0000) 2022-09-10T09:46:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:04.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:04.764442+0000) 2022-09-10T09:46:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:05 smithi110 ceph-mon[97633]: pgmap v10138: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:05 smithi104 ceph-mon[97044]: pgmap v10138: 129 pgs: 129 active+clean; 8.8 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:05.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:05.280537+0000) 2022-09-10T09:46:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:05.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:05.764689+0000) 2022-09-10T09:46:06.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:06.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:06.280707+0000) 2022-09-10T09:46:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:06.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:06.764868+0000) 2022-09-10T09:46:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:07 smithi110 ceph-mon[97633]: pgmap v10139: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:46:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:07 smithi104 ceph-mon[97044]: pgmap v10139: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:46:07.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:46:07.523+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:46:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:07.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:07.280918+0000) 2022-09-10T09:46:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:07.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:07.765036+0000) 2022-09-10T09:46:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:08.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:08.281153+0000) 2022-09-10T09:46:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:08.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:08.765264+0000) 2022-09-10T09:46:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:09 smithi110 ceph-mon[97633]: pgmap v10140: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:46:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:09 smithi110 ceph-mon[97633]: osdmap e714: 12 total, 12 up, 12 in 2022-09-10T09:46:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:09 smithi104 ceph-mon[97044]: pgmap v10140: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:46:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:09 smithi104 ceph-mon[97044]: osdmap e714: 12 total, 12 up, 12 in 2022-09-10T09:46:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:09.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:09.281315+0000) 2022-09-10T09:46:10.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:09.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:09.765423+0000) 2022-09-10T09:46:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:10.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:10.281566+0000) 2022-09-10T09:46:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:10.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:10.765638+0000) 2022-09-10T09:46:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:11 smithi110 ceph-mon[97633]: pgmap v10142: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:46:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:11 smithi104 ceph-mon[97044]: pgmap v10142: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:46:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:11.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:11.281732+0000) 2022-09-10T09:46:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:11.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:11.765893+0000) 2022-09-10T09:46:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:12.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:12.282004+0000) 2022-09-10T09:46:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:12.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:12.766135+0000) 2022-09-10T09:46:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:13 smithi104 ceph-mon[97044]: pgmap v10143: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:46:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:13 smithi104 ceph-mon[97044]: osdmap e715: 12 total, 12 up, 12 in 2022-09-10T09:46:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:13 smithi110 ceph-mon[97633]: pgmap v10143: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:46:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:13 smithi110 ceph-mon[97633]: osdmap e715: 12 total, 12 up, 12 in 2022-09-10T09:46:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:13.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:13.282212+0000) 2022-09-10T09:46:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:13.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:13.766352+0000) 2022-09-10T09:46:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:14.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:14.282406+0000) 2022-09-10T09:46:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:14.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:14.766601+0000) 2022-09-10T09:46:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:15 smithi104 ceph-mon[97044]: pgmap v10145: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:46:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:15 smithi110 ceph-mon[97633]: pgmap v10145: 129 pgs: 129 active+clean; 8.8 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:46:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:15.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:15.282656+0000) 2022-09-10T09:46:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:15.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:15.766842+0000) 2022-09-10T09:46:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:16.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:16.282894+0000) 2022-09-10T09:46:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:16.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:16.767013+0000) 2022-09-10T09:46:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:17 smithi104 ceph-mon[97044]: pgmap v10146: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:46:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:17 smithi110 ceph-mon[97633]: pgmap v10146: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:46:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:17.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:17.283147+0000) 2022-09-10T09:46:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:17.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:17.767207+0000) 2022-09-10T09:46:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:18.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:18.283351+0000) 2022-09-10T09:46:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:18.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:18.767418+0000) 2022-09-10T09:46:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:19 smithi104 ceph-mon[97044]: pgmap v10147: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:46:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:19 smithi110 ceph-mon[97633]: pgmap v10147: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:46:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:19.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:19.283545+0000) 2022-09-10T09:46:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:19.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:19.767639+0000) 2022-09-10T09:46:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:20.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:20.283774+0000) 2022-09-10T09:46:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:20.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:20.767903+0000) 2022-09-10T09:46:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:21 smithi104 ceph-mon[97044]: pgmap v10148: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:46:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:21 smithi110 ceph-mon[97633]: pgmap v10148: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:46:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:21.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:21.284039+0000) 2022-09-10T09:46:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:21.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:21.768195+0000) 2022-09-10T09:46:22.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:46:22.525+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:46:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:22.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:22.284315+0000) 2022-09-10T09:46:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:22.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:22.768430+0000) 2022-09-10T09:46:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:23 smithi104 ceph-mon[97044]: pgmap v10149: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:46:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:23 smithi110 ceph-mon[97633]: pgmap v10149: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:46:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:23.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:23.284537+0000) 2022-09-10T09:46:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:23.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:23.768645+0000) 2022-09-10T09:46:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:24.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:24.284721+0000) 2022-09-10T09:46:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:24.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:24.768874+0000) 2022-09-10T09:46:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:25 smithi104 ceph-mon[97044]: pgmap v10150: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:46:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:25 smithi110 ceph-mon[97633]: pgmap v10150: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:46:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:25.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:25.284947+0000) 2022-09-10T09:46:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:25.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:25.769175+0000) 2022-09-10T09:46:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:26.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:26.285181+0000) 2022-09-10T09:46:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:26.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:26.769358+0000) 2022-09-10T09:46:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:27 smithi104 ceph-mon[97044]: pgmap v10151: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:46:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:27 smithi110 ceph-mon[97633]: pgmap v10151: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:46:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:27.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:27.285339+0000) 2022-09-10T09:46:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:27.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:27.769534+0000) 2022-09-10T09:46:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:28.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:28.285569+0000) 2022-09-10T09:46:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:28.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:28.769801+0000) 2022-09-10T09:46:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:29 smithi104 ceph-mon[97044]: pgmap v10152: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:29 smithi110 ceph-mon[97633]: pgmap v10152: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:29.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:29.285820+0000) 2022-09-10T09:46:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:29.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:29.770001+0000) 2022-09-10T09:46:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:30.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:30.286065+0000) 2022-09-10T09:46:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:30.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:30.770193+0000) 2022-09-10T09:46:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:31 smithi104 ceph-mon[97044]: pgmap v10153: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:31 smithi110 ceph-mon[97633]: pgmap v10153: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:31.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:31.286266+0000) 2022-09-10T09:46:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:31.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:31.770432+0000) 2022-09-10T09:46:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:32.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:32.286492+0000) 2022-09-10T09:46:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:32.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:32.770653+0000) 2022-09-10T09:46:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:33 smithi110 ceph-mon[97633]: pgmap v10154: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:33 smithi104 ceph-mon[97044]: pgmap v10154: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:33.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:33.286696+0000) 2022-09-10T09:46:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:33.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:33.770885+0000) 2022-09-10T09:46:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:34.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:34.286895+0000) 2022-09-10T09:46:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:34.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:34.771123+0000) 2022-09-10T09:46:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:35 smithi110 ceph-mon[97633]: pgmap v10155: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:35 smithi104 ceph-mon[97044]: pgmap v10155: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:35.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:35.287163+0000) 2022-09-10T09:46:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:35.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:35.771346+0000) 2022-09-10T09:46:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:36.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:36.287355+0000) 2022-09-10T09:46:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:36.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:36.771589+0000) 2022-09-10T09:46:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:37 smithi110 ceph-mon[97633]: pgmap v10156: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:37.543 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:37.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:37.287622+0000) 2022-09-10T09:46:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:37 smithi104 ceph-mon[97044]: pgmap v10156: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:37.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:37.771832+0000) 2022-09-10T09:46:37.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:46:37.541+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:46:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:38.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:38.287900+0000) 2022-09-10T09:46:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:38.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:38.772046+0000) 2022-09-10T09:46:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:39 smithi110 ceph-mon[97633]: pgmap v10157: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:39 smithi104 ceph-mon[97044]: pgmap v10157: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:39.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:39.288222+0000) 2022-09-10T09:46:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:39.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:39.772287+0000) 2022-09-10T09:46:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:40.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:40.288446+0000) 2022-09-10T09:46:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:40.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:40.772537+0000) 2022-09-10T09:46:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:41 smithi110 ceph-mon[97633]: pgmap v10158: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:41 smithi104 ceph-mon[97044]: pgmap v10158: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:41.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:41.288704+0000) 2022-09-10T09:46:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:41.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:41.772767+0000) 2022-09-10T09:46:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:42.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:42.288990+0000) 2022-09-10T09:46:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:42.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:42.773024+0000) 2022-09-10T09:46:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:43 smithi110 ceph-mon[97633]: pgmap v10159: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:43 smithi104 ceph-mon[97044]: pgmap v10159: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:43.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:43.289260+0000) 2022-09-10T09:46:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:43.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:43.773188+0000) 2022-09-10T09:46:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:44.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:44.289456+0000) 2022-09-10T09:46:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:44.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:44.773397+0000) 2022-09-10T09:46:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:45 smithi110 ceph-mon[97633]: pgmap v10160: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:46:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:46:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:46:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:46:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:45 smithi104 ceph-mon[97044]: pgmap v10160: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:46:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:46:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:46:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:46:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:45.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:45.289697+0000) 2022-09-10T09:46:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:45.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:45.773623+0000) 2022-09-10T09:46:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:46.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:46.289978+0000) 2022-09-10T09:46:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:46.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:46.773831+0000) 2022-09-10T09:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:47 smithi110 ceph-mon[97633]: pgmap v10161: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:47 smithi104 ceph-mon[97044]: pgmap v10161: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:47.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:47.290203+0000) 2022-09-10T09:46:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:47.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:47.774111+0000) 2022-09-10T09:46:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:48.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:48.290434+0000) 2022-09-10T09:46:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:48.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:48.774353+0000) 2022-09-10T09:46:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:49 smithi110 ceph-mon[97633]: pgmap v10162: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:49 smithi104 ceph-mon[97044]: pgmap v10162: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:49.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:49.290621+0000) 2022-09-10T09:46:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:49.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:49.774600+0000) 2022-09-10T09:46:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:50.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:50.290837+0000) 2022-09-10T09:46:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:50.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:50.774845+0000) 2022-09-10T09:46:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:51 smithi110 ceph-mon[97633]: pgmap v10163: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:51 smithi104 ceph-mon[97044]: pgmap v10163: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:51.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:51.291062+0000) 2022-09-10T09:46:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:51.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:51.775033+0000) 2022-09-10T09:46:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:52.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:52.291320+0000) 2022-09-10T09:46:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:46:52.556+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:46:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:52.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:52.775245+0000) 2022-09-10T09:46:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:53 smithi110 ceph-mon[97633]: pgmap v10164: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:53 smithi104 ceph-mon[97044]: pgmap v10164: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:53.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:53.291555+0000) 2022-09-10T09:46:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:53.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:53.775468+0000) 2022-09-10T09:46:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:54.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:54.291725+0000) 2022-09-10T09:46:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:54.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:54.775646+0000) 2022-09-10T09:46:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:55 smithi110 ceph-mon[97633]: pgmap v10165: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:55 smithi104 ceph-mon[97044]: pgmap v10165: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:55.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:55.291881+0000) 2022-09-10T09:46:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:55.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:55.775801+0000) 2022-09-10T09:46:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:56.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:56.292087+0000) 2022-09-10T09:46:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:56.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:56.776019+0000) 2022-09-10T09:46:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:57 smithi110 ceph-mon[97633]: pgmap v10166: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:57 smithi104 ceph-mon[97044]: pgmap v10166: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:46:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:57.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:57.292347+0000) 2022-09-10T09:46:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:57.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:57.776269+0000) 2022-09-10T09:46:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:58.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:58.292541+0000) 2022-09-10T09:46:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:58.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:58.776510+0000) 2022-09-10T09:46:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:46:59 smithi110 ceph-mon[97633]: pgmap v10167: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:46:59 smithi104 ceph-mon[97044]: pgmap v10167: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:46:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:46:59.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:59.292737+0000) 2022-09-10T09:47:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:46:59.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:46:59.776719+0000) 2022-09-10T09:47:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:00.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:00.292986+0000) 2022-09-10T09:47:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:00.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:00.776943+0000) 2022-09-10T09:47:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:01 smithi110 ceph-mon[97633]: pgmap v10168: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:01 smithi104 ceph-mon[97044]: pgmap v10168: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:01.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:01.293231+0000) 2022-09-10T09:47:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:01.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:01.777177+0000) 2022-09-10T09:47:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:02.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:02.293490+0000) 2022-09-10T09:47:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:02.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:02.777418+0000) 2022-09-10T09:47:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:03 smithi110 ceph-mon[97633]: pgmap v10169: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:03 smithi104 ceph-mon[97044]: pgmap v10169: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:03.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:03.293710+0000) 2022-09-10T09:47:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:03.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:03.777642+0000) 2022-09-10T09:47:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:04.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:04.293891+0000) 2022-09-10T09:47:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:04.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:04.777855+0000) 2022-09-10T09:47:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:05 smithi110 ceph-mon[97633]: pgmap v10170: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:05 smithi104 ceph-mon[97044]: pgmap v10170: 129 pgs: 129 active+clean; 8.8 MiB data, 985 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:05.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:05.294135+0000) 2022-09-10T09:47:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:05.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:05.778129+0000) 2022-09-10T09:47:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:06.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:06.294311+0000) 2022-09-10T09:47:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:06.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:06.778301+0000) 2022-09-10T09:47:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:07 smithi110 ceph-mon[97633]: pgmap v10171: 129 pgs: 129 active+clean; 8.8 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:47:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:07 smithi104 ceph-mon[97044]: pgmap v10171: 129 pgs: 129 active+clean; 8.8 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:47:07.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:07.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:07.294518+0000) 2022-09-10T09:47:08.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:47:07.572+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:47:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:07.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:07.778521+0000) 2022-09-10T09:47:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:08.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:08.294818+0000) 2022-09-10T09:47:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:08.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:08.778691+0000) 2022-09-10T09:47:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:09 smithi110 ceph-mon[97633]: pgmap v10172: 129 pgs: 129 active+clean; 8.8 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:47:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:09 smithi110 ceph-mon[97633]: osdmap e716: 12 total, 12 up, 12 in 2022-09-10T09:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:09 smithi104 ceph-mon[97044]: pgmap v10172: 129 pgs: 129 active+clean; 8.8 MiB data, 986 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:09 smithi104 ceph-mon[97044]: osdmap e716: 12 total, 12 up, 12 in 2022-09-10T09:47:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:09.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:09.294994+0000) 2022-09-10T09:47:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:09.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:09.778857+0000) 2022-09-10T09:47:10.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:10.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:10.295219+0000) 2022-09-10T09:47:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:10.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:10.779028+0000) 2022-09-10T09:47:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:11 smithi110 ceph-mon[97633]: pgmap v10174: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:47:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:11 smithi104 ceph-mon[97044]: pgmap v10174: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:47:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:11.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:11.295482+0000) 2022-09-10T09:47:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:11.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:11.779252+0000) 2022-09-10T09:47:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:12.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:12.295646+0000) 2022-09-10T09:47:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:12.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:12.779407+0000) 2022-09-10T09:47:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:13 smithi104 ceph-mon[97044]: pgmap v10175: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:47:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:13 smithi104 ceph-mon[97044]: osdmap e717: 12 total, 12 up, 12 in 2022-09-10T09:47:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:13 smithi110 ceph-mon[97633]: pgmap v10175: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:47:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:13 smithi110 ceph-mon[97633]: osdmap e717: 12 total, 12 up, 12 in 2022-09-10T09:47:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:13.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:13.295885+0000) 2022-09-10T09:47:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:13.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:13.779582+0000) 2022-09-10T09:47:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:14.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:14.296117+0000) 2022-09-10T09:47:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:14.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:14.779814+0000) 2022-09-10T09:47:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:15 smithi104 ceph-mon[97044]: pgmap v10177: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:15 smithi110 ceph-mon[97633]: pgmap v10177: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:47:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:15.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:15.296265+0000) 2022-09-10T09:47:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:15.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:15.780054+0000) 2022-09-10T09:47:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:16.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:16.296502+0000) 2022-09-10T09:47:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:16.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:16.780266+0000) 2022-09-10T09:47:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:17 smithi104 ceph-mon[97044]: pgmap v10178: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:47:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:17 smithi110 ceph-mon[97633]: pgmap v10178: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:47:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:17.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:17.296677+0000) 2022-09-10T09:47:18.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:17.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:17.780476+0000) 2022-09-10T09:47:18.567 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:18.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:18.296820+0000) 2022-09-10T09:47:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:18.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:18.780634+0000) 2022-09-10T09:47:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:19 smithi104 ceph-mon[97044]: pgmap v10179: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:47:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:19 smithi110 ceph-mon[97633]: pgmap v10179: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:47:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:19.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:19.297052+0000) 2022-09-10T09:47:20.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:19.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:19.780816+0000) 2022-09-10T09:47:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:20.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:20.297249+0000) 2022-09-10T09:47:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:20.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:20.781059+0000) 2022-09-10T09:47:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:21 smithi104 ceph-mon[97044]: pgmap v10180: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:47:21.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:21 smithi110 ceph-mon[97633]: pgmap v10180: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:47:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:21.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:21.297432+0000) 2022-09-10T09:47:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:21.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:21.781333+0000) 2022-09-10T09:47:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:22.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:22.297705+0000) 2022-09-10T09:47:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:22.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:22.781590+0000) 2022-09-10T09:47:23.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:47:22.583+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:47:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:23 smithi110 ceph-mon[97633]: pgmap v10181: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:47:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:23 smithi104 ceph-mon[97044]: pgmap v10181: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:47:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:23.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:23.297911+0000) 2022-09-10T09:47:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:23.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:23.781736+0000) 2022-09-10T09:47:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:24.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:24.298167+0000) 2022-09-10T09:47:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:24.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:24.781907+0000) 2022-09-10T09:47:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:25 smithi110 ceph-mon[97633]: pgmap v10182: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:47:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:25 smithi104 ceph-mon[97044]: pgmap v10182: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:47:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:25.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:25.298413+0000) 2022-09-10T09:47:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:25.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:25.782117+0000) 2022-09-10T09:47:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:26.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:26.298662+0000) 2022-09-10T09:47:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:26.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:26.782274+0000) 2022-09-10T09:47:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:27 smithi110 ceph-mon[97633]: pgmap v10183: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:47:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:27 smithi104 ceph-mon[97044]: pgmap v10183: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:47:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:27.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:27.298889+0000) 2022-09-10T09:47:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:27.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:27.782514+0000) 2022-09-10T09:47:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:28.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:28.299178+0000) 2022-09-10T09:47:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:28.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:28.782755+0000) 2022-09-10T09:47:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:29 smithi110 ceph-mon[97633]: pgmap v10184: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:29 smithi104 ceph-mon[97044]: pgmap v10184: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:29.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:29.299344+0000) 2022-09-10T09:47:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:29.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:29.783027+0000) 2022-09-10T09:47:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:30.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:30.299558+0000) 2022-09-10T09:47:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:30.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:30.783208+0000) 2022-09-10T09:47:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:31 smithi110 ceph-mon[97633]: pgmap v10185: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:31 smithi104 ceph-mon[97044]: pgmap v10185: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:31.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:31.299786+0000) 2022-09-10T09:47:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:31.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:31.783446+0000) 2022-09-10T09:47:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:32.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:32.300016+0000) 2022-09-10T09:47:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:32.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:32.783646+0000) 2022-09-10T09:47:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:33 smithi110 ceph-mon[97633]: pgmap v10186: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:33 smithi104 ceph-mon[97044]: pgmap v10186: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:33.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:33.300242+0000) 2022-09-10T09:47:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:33.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:33.783855+0000) 2022-09-10T09:47:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:34.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:34.300444+0000) 2022-09-10T09:47:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:34.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:34.784079+0000) 2022-09-10T09:47:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:35 smithi110 ceph-mon[97633]: pgmap v10187: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:35 smithi104 ceph-mon[97044]: pgmap v10187: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:35.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:35.300653+0000) 2022-09-10T09:47:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:35.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:35.784336+0000) 2022-09-10T09:47:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:36.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:36.300831+0000) 2022-09-10T09:47:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:36.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:36.784531+0000) 2022-09-10T09:47:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:37 smithi110 ceph-mon[97633]: pgmap v10188: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:37 smithi104 ceph-mon[97044]: pgmap v10188: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:37.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:37.301012+0000) 2022-09-10T09:47:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:37.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:37.784734+0000) 2022-09-10T09:47:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:47:37.594+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:47:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:38.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:38.301223+0000) 2022-09-10T09:47:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:38.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:38.784942+0000) 2022-09-10T09:47:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:39 smithi110 ceph-mon[97633]: pgmap v10189: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:39 smithi104 ceph-mon[97044]: pgmap v10189: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:39.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:39.301436+0000) 2022-09-10T09:47:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:39.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:39.785208+0000) 2022-09-10T09:47:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:40.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:40.301719+0000) 2022-09-10T09:47:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:40.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:40.785425+0000) 2022-09-10T09:47:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:41 smithi110 ceph-mon[97633]: pgmap v10190: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:41 smithi104 ceph-mon[97044]: pgmap v10190: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:41.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:41.301893+0000) 2022-09-10T09:47:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:41.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:41.785563+0000) 2022-09-10T09:47:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:42.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:42.302123+0000) 2022-09-10T09:47:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:42.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:42.785814+0000) 2022-09-10T09:47:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:43 smithi110 ceph-mon[97633]: pgmap v10191: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:43 smithi104 ceph-mon[97044]: pgmap v10191: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:43.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:43.302296+0000) 2022-09-10T09:47:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:43.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:43.786051+0000) 2022-09-10T09:47:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:44.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:44.302508+0000) 2022-09-10T09:47:45.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:44.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:44.786262+0000) 2022-09-10T09:47:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:45 smithi110 ceph-mon[97633]: pgmap v10192: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:47:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:47:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:47:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:47:45.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:45 smithi104 ceph-mon[97044]: pgmap v10192: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:45.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:47:45.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:47:45.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:47:45.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:47:45.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:45.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:45.302726+0000) 2022-09-10T09:47:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:45.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:45.786473+0000) 2022-09-10T09:47:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:46.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:46.302908+0000) 2022-09-10T09:47:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:46.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:46.786691+0000) 2022-09-10T09:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:47 smithi110 ceph-mon[97633]: pgmap v10193: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:47 smithi104 ceph-mon[97044]: pgmap v10193: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:47.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:47.303081+0000) 2022-09-10T09:47:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:47.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:47.786853+0000) 2022-09-10T09:47:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:48.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:48.303346+0000) 2022-09-10T09:47:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:48.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:48.787027+0000) 2022-09-10T09:47:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:49 smithi110 ceph-mon[97633]: pgmap v10194: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:49 smithi104 ceph-mon[97044]: pgmap v10194: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:49.788 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:49.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:49.303583+0000) 2022-09-10T09:47:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:49.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:49.787256+0000) 2022-09-10T09:47:50.788 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:50.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:50.303814+0000) 2022-09-10T09:47:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:50.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:50.787472+0000) 2022-09-10T09:47:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:51 smithi110 ceph-mon[97633]: pgmap v10195: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:51 smithi104 ceph-mon[97044]: pgmap v10195: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:51.798 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:51.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:51.304050+0000) 2022-09-10T09:47:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:51.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:51.787695+0000) 2022-09-10T09:47:52.610 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:52.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:52.304337+0000) 2022-09-10T09:47:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:47:52.608+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:47:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:52.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:52.787932+0000) 2022-09-10T09:47:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:53 smithi110 ceph-mon[97633]: pgmap v10196: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:53 smithi104 ceph-mon[97044]: pgmap v10196: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:53.789 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:53.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:53.304540+0000) 2022-09-10T09:47:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:53.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:53.788226+0000) 2022-09-10T09:47:54.789 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:54.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:54.304711+0000) 2022-09-10T09:47:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:54.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:54.788487+0000) 2022-09-10T09:47:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:55 smithi110 ceph-mon[97633]: pgmap v10197: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:55 smithi104 ceph-mon[97044]: pgmap v10197: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:55.789 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:55.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:55.304891+0000) 2022-09-10T09:47:56.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:55.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:55.788663+0000) 2022-09-10T09:47:56.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:56.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:56.305154+0000) 2022-09-10T09:47:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:56.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:56.788821+0000) 2022-09-10T09:47:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:57 smithi110 ceph-mon[97633]: pgmap v10198: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:57 smithi104 ceph-mon[97044]: pgmap v10198: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:47:57.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:57.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:57.305406+0000) 2022-09-10T09:47:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:57.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:57.789039+0000) 2022-09-10T09:47:58.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:58.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:58.305603+0000) 2022-09-10T09:47:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:58.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:58.789238+0000) 2022-09-10T09:47:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:47:59 smithi110 ceph-mon[97633]: pgmap v10199: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:47:59 smithi104 ceph-mon[97044]: pgmap v10199: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:47:59.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:47:59.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:59.305824+0000) 2022-09-10T09:48:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:47:59.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:47:59.789448+0000) 2022-09-10T09:48:00.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:00.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:00.306069+0000) 2022-09-10T09:48:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:00.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:00.789641+0000) 2022-09-10T09:48:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:01 smithi110 ceph-mon[97633]: pgmap v10200: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:01 smithi104 ceph-mon[97044]: pgmap v10200: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:01.790 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:01.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:01.306367+0000) 2022-09-10T09:48:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:01.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:01.789883+0000) 2022-09-10T09:48:02.791 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:02.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:02.306603+0000) 2022-09-10T09:48:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:02.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:02.790185+0000) 2022-09-10T09:48:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:03 smithi110 ceph-mon[97633]: pgmap v10201: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:03 smithi104 ceph-mon[97044]: pgmap v10201: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:03.791 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:03.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:03.306829+0000) 2022-09-10T09:48:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:03.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:03.790329+0000) 2022-09-10T09:48:04.791 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:04.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:04.307087+0000) 2022-09-10T09:48:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:04.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:04.790533+0000) 2022-09-10T09:48:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:05 smithi110 ceph-mon[97633]: pgmap v10202: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:05 smithi104 ceph-mon[97044]: pgmap v10202: 129 pgs: 129 active+clean; 8.8 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:05.792 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:05.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:05.307314+0000) 2022-09-10T09:48:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:05.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:05.790792+0000) 2022-09-10T09:48:06.792 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:06.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:06.307525+0000) 2022-09-10T09:48:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:06.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:06.790991+0000) 2022-09-10T09:48:07.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:07 smithi110 ceph-mon[97633]: pgmap v10203: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:48:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:07 smithi104 ceph-mon[97044]: pgmap v10203: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:48:07.625 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:07.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:07.307722+0000) 2022-09-10T09:48:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:07.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:07.791234+0000) 2022-09-10T09:48:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:48:07.623+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:48:08.792 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:08.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:08.307901+0000) 2022-09-10T09:48:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:08.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:08.791469+0000) 2022-09-10T09:48:09.384 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:09.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:09.308034+0000) 2022-09-10T09:48:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:09 smithi110 ceph-mon[97633]: pgmap v10204: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:48:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:09 smithi110 ceph-mon[97633]: osdmap e718: 12 total, 12 up, 12 in 2022-09-10T09:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:09 smithi104 ceph-mon[97044]: pgmap v10204: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:48:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:09 smithi104 ceph-mon[97044]: osdmap e718: 12 total, 12 up, 12 in 2022-09-10T09:48:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:09.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:09.791672+0000) 2022-09-10T09:48:10.774 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:10.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:10.308187+0000) 2022-09-10T09:48:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:10.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:10.791857+0000) 2022-09-10T09:48:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:11 smithi110 ceph-mon[97633]: pgmap v10206: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:48:11.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:11 smithi104 ceph-mon[97044]: pgmap v10206: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:48:11.614 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:11.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:11.308319+0000) 2022-09-10T09:48:11.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:11.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:11.792011+0000) 2022-09-10T09:48:12.793 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:12.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:12.308470+0000) 2022-09-10T09:48:13.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:12.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:12.792214+0000) 2022-09-10T09:48:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:13 smithi110 ceph-mon[97633]: pgmap v10207: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:48:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:13 smithi110 ceph-mon[97633]: osdmap e719: 12 total, 12 up, 12 in 2022-09-10T09:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:13 smithi104 ceph-mon[97044]: pgmap v10207: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:13 smithi104 ceph-mon[97044]: osdmap e719: 12 total, 12 up, 12 in 2022-09-10T09:48:13.793 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:13.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:13.308648+0000) 2022-09-10T09:48:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:13.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:13.792502+0000) 2022-09-10T09:48:14.793 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:14.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:14.308835+0000) 2022-09-10T09:48:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:14.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:14.792750+0000) 2022-09-10T09:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:15 smithi110 ceph-mon[97633]: pgmap v10209: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:15 smithi104 ceph-mon[97044]: pgmap v10209: 129 pgs: 129 active+clean; 8.8 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:48:15.794 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:15.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:15.309068+0000) 2022-09-10T09:48:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:15.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:15.792960+0000) 2022-09-10T09:48:16.794 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:16.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:16.309328+0000) 2022-09-10T09:48:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:16.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:16.793200+0000) 2022-09-10T09:48:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:17 smithi110 ceph-mon[97633]: pgmap v10210: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:17 smithi104 ceph-mon[97044]: pgmap v10210: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:48:17.795 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:17.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:17.309589+0000) 2022-09-10T09:48:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:17.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:17.793484+0000) 2022-09-10T09:48:18.794 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:18.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:18.309824+0000) 2022-09-10T09:48:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:18.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:18.793728+0000) 2022-09-10T09:48:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:19 smithi110 ceph-mon[97633]: pgmap v10211: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:48:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:19 smithi104 ceph-mon[97044]: pgmap v10211: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:48:19.795 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:19.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:19.309993+0000) 2022-09-10T09:48:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:19.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:19.793951+0000) 2022-09-10T09:48:20.795 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:20.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:20.310192+0000) 2022-09-10T09:48:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:20.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:20.794185+0000) 2022-09-10T09:48:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:21 smithi110 ceph-mon[97633]: pgmap v10212: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:48:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:21 smithi104 ceph-mon[97044]: pgmap v10212: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:48:21.795 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:21.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:21.310436+0000) 2022-09-10T09:48:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:21.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:21.794444+0000) 2022-09-10T09:48:22.640 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:22.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:22.310673+0000) 2022-09-10T09:48:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:22.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:22.794704+0000) 2022-09-10T09:48:23.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:48:22.639+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:48:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:23 smithi110 ceph-mon[97633]: pgmap v10213: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:48:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:23 smithi104 ceph-mon[97044]: pgmap v10213: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:48:23.796 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:23.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:23.310890+0000) 2022-09-10T09:48:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:23.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:23.794940+0000) 2022-09-10T09:48:24.795 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:24.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:24.311137+0000) 2022-09-10T09:48:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:24.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:24.795142+0000) 2022-09-10T09:48:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:25 smithi110 ceph-mon[97633]: pgmap v10214: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:48:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:25 smithi104 ceph-mon[97044]: pgmap v10214: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:48:25.796 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:25.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:25.311377+0000) 2022-09-10T09:48:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:25.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:25.795355+0000) 2022-09-10T09:48:26.796 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:26.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:26.311625+0000) 2022-09-10T09:48:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:26.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:26.795497+0000) 2022-09-10T09:48:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:27 smithi110 ceph-mon[97633]: pgmap v10215: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:48:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:27 smithi104 ceph-mon[97044]: pgmap v10215: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:48:27.796 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:27.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:27.311782+0000) 2022-09-10T09:48:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:27.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:27.795718+0000) 2022-09-10T09:48:28.796 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:28.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:28.312013+0000) 2022-09-10T09:48:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:28.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:28.795972+0000) 2022-09-10T09:48:29.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:29 smithi110 ceph-mon[97633]: pgmap v10216: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:29 smithi104 ceph-mon[97044]: pgmap v10216: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:29.797 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:29.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:29.312209+0000) 2022-09-10T09:48:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:29.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:29.796217+0000) 2022-09-10T09:48:30.797 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:30.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:30.312434+0000) 2022-09-10T09:48:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:30.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:30.796420+0000) 2022-09-10T09:48:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:31 smithi110 ceph-mon[97633]: pgmap v10217: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:31 smithi104 ceph-mon[97044]: pgmap v10217: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:31.798 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:31.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:31.312581+0000) 2022-09-10T09:48:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:31.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:31.796642+0000) 2022-09-10T09:48:32.797 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:32.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:32.312797+0000) 2022-09-10T09:48:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:32.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:32.796853+0000) 2022-09-10T09:48:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:33 smithi110 ceph-mon[97633]: pgmap v10218: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:33 smithi104 ceph-mon[97044]: pgmap v10218: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:33.798 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:33.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:33.313048+0000) 2022-09-10T09:48:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:33.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:33.797082+0000) 2022-09-10T09:48:34.798 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:34.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:34.313298+0000) 2022-09-10T09:48:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:34.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:34.797272+0000) 2022-09-10T09:48:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:35 smithi110 ceph-mon[97633]: pgmap v10219: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:35 smithi104 ceph-mon[97044]: pgmap v10219: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:35.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:35.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:35.313545+0000) 2022-09-10T09:48:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:35.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:35.797498+0000) 2022-09-10T09:48:36.799 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:36.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:36.313819+0000) 2022-09-10T09:48:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:36.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:36.797748+0000) 2022-09-10T09:48:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:37 smithi110 ceph-mon[97633]: pgmap v10220: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:37 smithi104 ceph-mon[97044]: pgmap v10220: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:37.656 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:37.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:37.314066+0000) 2022-09-10T09:48:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:37.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:37.797987+0000) 2022-09-10T09:48:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:48:37.655+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:48:38.799 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:38.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:38.314308+0000) 2022-09-10T09:48:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:38.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:38.798267+0000) 2022-09-10T09:48:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:39 smithi110 ceph-mon[97633]: pgmap v10221: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:39 smithi104 ceph-mon[97044]: pgmap v10221: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:39.799 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:39.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:39.314521+0000) 2022-09-10T09:48:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:39.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:39.798507+0000) 2022-09-10T09:48:40.799 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:40.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:40.314765+0000) 2022-09-10T09:48:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:40.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:40.798673+0000) 2022-09-10T09:48:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:41 smithi110 ceph-mon[97633]: pgmap v10222: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:41 smithi104 ceph-mon[97044]: pgmap v10222: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:41.799 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:41.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:41.315013+0000) 2022-09-10T09:48:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:41.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:41.798935+0000) 2022-09-10T09:48:42.800 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:42.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:42.315215+0000) 2022-09-10T09:48:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:42.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:42.799153+0000) 2022-09-10T09:48:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:43 smithi110 ceph-mon[97633]: pgmap v10223: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:43 smithi104 ceph-mon[97044]: pgmap v10223: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:43.800 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:43.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:43.315437+0000) 2022-09-10T09:48:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:43.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:43.799324+0000) 2022-09-10T09:48:44.800 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:44.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:44.315620+0000) 2022-09-10T09:48:45.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:44.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:44.799546+0000) 2022-09-10T09:48:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:45 smithi110 ceph-mon[97633]: pgmap v10224: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:48:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:48:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:48:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:48:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:45 smithi104 ceph-mon[97044]: pgmap v10224: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:48:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:48:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:48:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:48:45.800 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:45.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:45.315793+0000) 2022-09-10T09:48:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:45.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:45.799824+0000) 2022-09-10T09:48:46.800 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:46.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:46.316001+0000) 2022-09-10T09:48:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:46.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:46.800012+0000) 2022-09-10T09:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:47 smithi110 ceph-mon[97633]: pgmap v10225: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:47 smithi104 ceph-mon[97044]: pgmap v10225: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:47.801 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:47.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:47.316148+0000) 2022-09-10T09:48:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:47.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:47.800191+0000) 2022-09-10T09:48:48.801 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:48.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:48.316379+0000) 2022-09-10T09:48:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:48.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:48.800396+0000) 2022-09-10T09:48:49.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:49 smithi110 ceph-mon[97633]: pgmap v10226: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:49.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:49 smithi104 ceph-mon[97044]: pgmap v10226: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:49.801 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:49.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:49.316554+0000) 2022-09-10T09:48:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:49.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:49.800591+0000) 2022-09-10T09:48:50.801 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:50.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:50.316780+0000) 2022-09-10T09:48:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:50.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:50.800884+0000) 2022-09-10T09:48:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:51 smithi110 ceph-mon[97633]: pgmap v10227: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:51 smithi104 ceph-mon[97044]: pgmap v10227: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:51.801 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:51.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:51.317012+0000) 2022-09-10T09:48:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:51.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:51.801057+0000) 2022-09-10T09:48:52.672 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:52.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:52.317210+0000) 2022-09-10T09:48:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:52.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:52.801309+0000) 2022-09-10T09:48:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:48:52.670+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:48:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:53 smithi110 ceph-mon[97633]: pgmap v10228: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:53 smithi104 ceph-mon[97044]: pgmap v10228: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:53.802 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:53.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:53.317432+0000) 2022-09-10T09:48:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:53.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:53.801524+0000) 2022-09-10T09:48:54.802 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:54.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:54.317662+0000) 2022-09-10T09:48:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:54.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:54.801728+0000) 2022-09-10T09:48:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:55 smithi110 ceph-mon[97633]: pgmap v10229: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:55 smithi104 ceph-mon[97044]: pgmap v10229: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:55.802 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:55.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:55.317886+0000) 2022-09-10T09:48:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:55.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:55.801927+0000) 2022-09-10T09:48:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:56.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:56.318162+0000) 2022-09-10T09:48:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:56.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:56.802204+0000) 2022-09-10T09:48:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:57 smithi110 ceph-mon[97633]: pgmap v10230: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:57 smithi104 ceph-mon[97044]: pgmap v10230: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:48:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:57.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:57.318426+0000) 2022-09-10T09:48:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:57.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:57.802466+0000) 2022-09-10T09:48:58.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:58.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:58.318681+0000) 2022-09-10T09:48:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:58.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:58.802666+0000) 2022-09-10T09:48:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:48:59 smithi110 ceph-mon[97633]: pgmap v10231: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:48:59 smithi104 ceph-mon[97044]: pgmap v10231: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:48:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:48:59.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:59.318940+0000) 2022-09-10T09:49:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:48:59.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:48:59.802837+0000) 2022-09-10T09:49:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:00.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:00.319210+0000) 2022-09-10T09:49:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:00.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:00.803126+0000) 2022-09-10T09:49:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:01 smithi110 ceph-mon[97633]: pgmap v10232: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:01 smithi104 ceph-mon[97044]: pgmap v10232: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:01.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:01.319341+0000) 2022-09-10T09:49:02.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:01.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:01.803396+0000) 2022-09-10T09:49:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:02.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:02.319517+0000) 2022-09-10T09:49:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:02.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:02.803677+0000) 2022-09-10T09:49:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:03 smithi110 ceph-mon[97633]: pgmap v10233: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:03.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:03 smithi104 ceph-mon[97044]: pgmap v10233: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:03.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:03.319767+0000) 2022-09-10T09:49:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:03.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:03.803918+0000) 2022-09-10T09:49:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:04.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:04.320001+0000) 2022-09-10T09:49:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:04.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:04.804040+0000) 2022-09-10T09:49:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:05 smithi110 ceph-mon[97633]: pgmap v10234: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:05 smithi104 ceph-mon[97044]: pgmap v10234: 129 pgs: 129 active+clean; 8.8 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:05.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:05.320270+0000) 2022-09-10T09:49:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:05.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:05.804168+0000) 2022-09-10T09:49:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:06.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:06.320444+0000) 2022-09-10T09:49:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:06.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:06.804318+0000) 2022-09-10T09:49:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:07 smithi110 ceph-mon[97633]: pgmap v10235: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:49:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:07 smithi104 ceph-mon[97044]: pgmap v10235: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:49:07.685 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:07.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:07.320622+0000) 2022-09-10T09:49:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:07.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:07.804561+0000) 2022-09-10T09:49:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:49:07.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:49:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:08.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:08.320798+0000) 2022-09-10T09:49:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:08.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:08.804782+0000) 2022-09-10T09:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:09 smithi110 ceph-mon[97633]: pgmap v10236: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:09 smithi110 ceph-mon[97633]: osdmap e720: 12 total, 12 up, 12 in 2022-09-10T09:49:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:09 smithi104 ceph-mon[97044]: pgmap v10236: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:49:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:09 smithi104 ceph-mon[97044]: osdmap e720: 12 total, 12 up, 12 in 2022-09-10T09:49:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:09.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:09.321002+0000) 2022-09-10T09:49:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:09.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:09.804977+0000) 2022-09-10T09:49:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:10.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:10.321235+0000) 2022-09-10T09:49:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:10.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:10.805218+0000) 2022-09-10T09:49:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:11 smithi110 ceph-mon[97633]: pgmap v10238: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:49:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:11 smithi104 ceph-mon[97044]: pgmap v10238: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:49:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:11.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:11.321434+0000) 2022-09-10T09:49:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:11.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:11.805409+0000) 2022-09-10T09:49:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:12.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:12.321657+0000) 2022-09-10T09:49:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:12.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:12.805599+0000) 2022-09-10T09:49:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:13 smithi110 ceph-mon[97633]: pgmap v10239: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:49:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:13 smithi110 ceph-mon[97633]: osdmap e721: 12 total, 12 up, 12 in 2022-09-10T09:49:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:13 smithi104 ceph-mon[97044]: pgmap v10239: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:49:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:13 smithi104 ceph-mon[97044]: osdmap e721: 12 total, 12 up, 12 in 2022-09-10T09:49:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:13.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:13.321936+0000) 2022-09-10T09:49:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:13.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:13.805760+0000) 2022-09-10T09:49:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:14.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:14.322221+0000) 2022-09-10T09:49:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:14.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:14.805980+0000) 2022-09-10T09:49:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:15 smithi110 ceph-mon[97633]: pgmap v10241: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:49:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:15 smithi104 ceph-mon[97044]: pgmap v10241: 129 pgs: 129 active+clean; 8.8 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:49:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:15.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:15.322426+0000) 2022-09-10T09:49:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:15.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:15.806215+0000) 2022-09-10T09:49:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:16.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:16.322611+0000) 2022-09-10T09:49:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:16.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:16.806394+0000) 2022-09-10T09:49:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:17 smithi110 ceph-mon[97633]: pgmap v10242: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:49:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:17 smithi104 ceph-mon[97044]: pgmap v10242: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:49:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:17.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:17.322897+0000) 2022-09-10T09:49:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:17.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:17.806560+0000) 2022-09-10T09:49:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:18.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:18.323111+0000) 2022-09-10T09:49:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:18.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:18.806720+0000) 2022-09-10T09:49:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:19 smithi110 ceph-mon[97633]: pgmap v10243: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:49:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:19 smithi104 ceph-mon[97044]: pgmap v10243: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:49:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:19.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:19.323339+0000) 2022-09-10T09:49:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:19.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:19.806953+0000) 2022-09-10T09:49:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:20.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:20.323568+0000) 2022-09-10T09:49:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:20.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:20.807167+0000) 2022-09-10T09:49:21.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:21 smithi110 ceph-mon[97633]: pgmap v10244: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:49:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:21 smithi104 ceph-mon[97044]: pgmap v10244: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:49:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:21.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:21.323801+0000) 2022-09-10T09:49:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:21.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:21.807337+0000) 2022-09-10T09:49:22.688 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:22.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:22.323989+0000) 2022-09-10T09:49:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:22.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:22.807554+0000) 2022-09-10T09:49:23.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:49:22.686+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:49:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:23 smithi110 ceph-mon[97633]: pgmap v10245: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:49:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:23 smithi104 ceph-mon[97044]: pgmap v10245: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:49:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:23.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:23.324205+0000) 2022-09-10T09:49:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:23.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:23.807713+0000) 2022-09-10T09:49:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:24.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:24.324367+0000) 2022-09-10T09:49:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:24.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:24.807900+0000) 2022-09-10T09:49:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:25 smithi110 ceph-mon[97633]: pgmap v10246: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:49:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:25 smithi104 ceph-mon[97044]: pgmap v10246: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:49:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:25.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:25.324507+0000) 2022-09-10T09:49:26.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:25.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:25.808136+0000) 2022-09-10T09:49:26.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:26.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:26.324753+0000) 2022-09-10T09:49:27.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:26.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:26.808308+0000) 2022-09-10T09:49:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:27 smithi110 ceph-mon[97633]: pgmap v10247: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:49:27.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:27 smithi104 ceph-mon[97044]: pgmap v10247: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:49:27.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:27.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:27.324949+0000) 2022-09-10T09:49:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:27.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:27.808498+0000) 2022-09-10T09:49:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:28.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:28.325139+0000) 2022-09-10T09:49:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:28.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:28.808666+0000) 2022-09-10T09:49:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:29 smithi110 ceph-mon[97633]: pgmap v10248: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:29 smithi104 ceph-mon[97044]: pgmap v10248: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:29.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:29.325387+0000) 2022-09-10T09:49:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:29.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:29.808906+0000) 2022-09-10T09:49:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:30.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:30.325559+0000) 2022-09-10T09:49:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:30.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:30.809136+0000) 2022-09-10T09:49:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:31 smithi110 ceph-mon[97633]: pgmap v10249: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:31 smithi104 ceph-mon[97044]: pgmap v10249: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:31.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:31.325739+0000) 2022-09-10T09:49:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:31.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:31.809292+0000) 2022-09-10T09:49:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:32.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:32.325993+0000) 2022-09-10T09:49:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:32.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:32.809473+0000) 2022-09-10T09:49:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:33 smithi110 ceph-mon[97633]: pgmap v10250: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:33 smithi104 ceph-mon[97044]: pgmap v10250: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:33.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:33.326263+0000) 2022-09-10T09:49:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:33.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:33.809630+0000) 2022-09-10T09:49:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:34.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:34.326442+0000) 2022-09-10T09:49:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:34.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:34.809827+0000) 2022-09-10T09:49:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:35 smithi110 ceph-mon[97633]: pgmap v10251: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:35 smithi104 ceph-mon[97044]: pgmap v10251: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:35.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:35.326653+0000) 2022-09-10T09:49:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:35.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:35.809988+0000) 2022-09-10T09:49:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:36.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:36.326923+0000) 2022-09-10T09:49:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:36.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:36.810173+0000) 2022-09-10T09:49:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:37 smithi110 ceph-mon[97633]: pgmap v10252: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:37 smithi104 ceph-mon[97044]: pgmap v10252: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:37.692 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:37.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:37.327192+0000) 2022-09-10T09:49:38.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:49:37.690+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:49:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:37.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:37.810298+0000) 2022-09-10T09:49:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:38.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:38.327401+0000) 2022-09-10T09:49:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:38.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:38.810477+0000) 2022-09-10T09:49:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:39 smithi110 ceph-mon[97633]: pgmap v10253: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:39 smithi104 ceph-mon[97044]: pgmap v10253: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:39.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:39.327591+0000) 2022-09-10T09:49:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:39.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:39.810610+0000) 2022-09-10T09:49:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:40.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:40.327881+0000) 2022-09-10T09:49:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:40.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:40.810834+0000) 2022-09-10T09:49:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:41 smithi110 ceph-mon[97633]: pgmap v10254: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:41 smithi104 ceph-mon[97044]: pgmap v10254: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:41.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:41.328167+0000) 2022-09-10T09:49:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:41.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:41.811051+0000) 2022-09-10T09:49:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:42.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:42.328395+0000) 2022-09-10T09:49:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:42.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:42.811243+0000) 2022-09-10T09:49:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:43 smithi110 ceph-mon[97633]: pgmap v10255: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:43 smithi104 ceph-mon[97044]: pgmap v10255: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:43.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:43.328582+0000) 2022-09-10T09:49:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:43.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:43.811414+0000) 2022-09-10T09:49:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:44.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:44.328778+0000) 2022-09-10T09:49:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:44.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:44.811588+0000) 2022-09-10T09:49:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:45 smithi110 ceph-mon[97633]: pgmap v10256: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:49:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:49:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:49:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:49:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:45 smithi104 ceph-mon[97044]: pgmap v10256: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:49:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:49:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:49:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:49:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:45.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:45.329033+0000) 2022-09-10T09:49:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:45.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:45.811793+0000) 2022-09-10T09:49:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:46.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:46.329299+0000) 2022-09-10T09:49:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:46.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:46.812013+0000) 2022-09-10T09:49:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:47 smithi110 ceph-mon[97633]: pgmap v10257: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:47 smithi104 ceph-mon[97044]: pgmap v10257: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:47.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:47.329488+0000) 2022-09-10T09:49:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:47.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:47.812174+0000) 2022-09-10T09:49:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:48.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:48.329666+0000) 2022-09-10T09:49:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:48.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:48.812284+0000) 2022-09-10T09:49:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:49 smithi110 ceph-mon[97633]: pgmap v10258: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:49 smithi104 ceph-mon[97044]: pgmap v10258: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:49.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:49.329880+0000) 2022-09-10T09:49:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:49.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:49.812410+0000) 2022-09-10T09:49:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:50.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:50.330182+0000) 2022-09-10T09:49:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:50.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:50.812583+0000) 2022-09-10T09:49:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:51 smithi110 ceph-mon[97633]: pgmap v10259: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:51 smithi104 ceph-mon[97044]: pgmap v10259: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:51.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:51.330418+0000) 2022-09-10T09:49:52.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:51.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:51.812783+0000) 2022-09-10T09:49:52.697 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:52.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:52.330658+0000) 2022-09-10T09:49:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:52.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:52.812986+0000) 2022-09-10T09:49:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:49:52.694+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:49:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:53 smithi110 ceph-mon[97633]: pgmap v10260: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:53 smithi104 ceph-mon[97044]: pgmap v10260: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:53.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:53.330915+0000) 2022-09-10T09:49:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:53.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:53.813225+0000) 2022-09-10T09:49:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:54.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:54.331153+0000) 2022-09-10T09:49:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:54.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:54.813437+0000) 2022-09-10T09:49:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:55 smithi110 ceph-mon[97633]: pgmap v10261: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:55 smithi104 ceph-mon[97044]: pgmap v10261: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:55.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:55.331333+0000) 2022-09-10T09:49:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:55.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:55.813604+0000) 2022-09-10T09:49:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:56.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:56.331495+0000) 2022-09-10T09:49:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:56.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:56.813832+0000) 2022-09-10T09:49:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:57 smithi110 ceph-mon[97633]: pgmap v10262: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:57 smithi104 ceph-mon[97044]: pgmap v10262: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:49:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:57.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:57.331686+0000) 2022-09-10T09:49:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:57.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:57.814047+0000) 2022-09-10T09:49:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:58.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:58.331978+0000) 2022-09-10T09:49:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:58.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:58.814196+0000) 2022-09-10T09:49:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:49:59 smithi110 ceph-mon[97633]: pgmap v10263: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:49:59 smithi104 ceph-mon[97044]: pgmap v10263: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:49:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:49:59.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:59.332222+0000) 2022-09-10T09:50:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:49:59.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:49:59.814435+0000) 2022-09-10T09:50:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:50:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T09:50:00.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:00.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:00.332474+0000) 2022-09-10T09:50:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:00.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:00.814577+0000) 2022-09-10T09:50:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:01 smithi110 ceph-mon[97633]: pgmap v10264: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:01.467 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:01 smithi104 ceph-mon[97044]: pgmap v10264: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:01.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:01.332735+0000) 2022-09-10T09:50:02.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:01.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:01.814807+0000) 2022-09-10T09:50:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:02.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:02.332969+0000) 2022-09-10T09:50:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:02.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:02.814988+0000) 2022-09-10T09:50:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:03 smithi110 ceph-mon[97633]: pgmap v10265: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:03 smithi104 ceph-mon[97044]: pgmap v10265: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:03.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:03.333209+0000) 2022-09-10T09:50:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:03.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:03.815185+0000) 2022-09-10T09:50:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:04.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:04.333454+0000) 2022-09-10T09:50:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:04.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:04.815366+0000) 2022-09-10T09:50:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:05 smithi110 ceph-mon[97633]: pgmap v10266: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:05 smithi104 ceph-mon[97044]: pgmap v10266: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:05.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:05.333692+0000) 2022-09-10T09:50:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:05.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:05.815502+0000) 2022-09-10T09:50:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:06.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:06.333904+0000) 2022-09-10T09:50:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:06.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:06.815758+0000) 2022-09-10T09:50:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:07 smithi110 ceph-mon[97633]: pgmap v10267: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:50:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:07 smithi104 ceph-mon[97044]: pgmap v10267: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:50:07.711 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:07.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:07.334156+0000) 2022-09-10T09:50:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:07.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:07.816026+0000) 2022-09-10T09:50:08.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:50:07.710+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:50:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:08.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:08.334393+0000) 2022-09-10T09:50:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:08.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:08.816183+0000) 2022-09-10T09:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:09 smithi110 ceph-mon[97633]: pgmap v10268: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:09 smithi110 ceph-mon[97633]: osdmap e722: 12 total, 12 up, 12 in 2022-09-10T09:50:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:09 smithi104 ceph-mon[97044]: pgmap v10268: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:50:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:09 smithi104 ceph-mon[97044]: osdmap e722: 12 total, 12 up, 12 in 2022-09-10T09:50:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:09.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:09.334558+0000) 2022-09-10T09:50:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:09.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:09.816363+0000) 2022-09-10T09:50:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:10.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:10.334770+0000) 2022-09-10T09:50:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:10.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:10.816594+0000) 2022-09-10T09:50:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:11 smithi104 ceph-mon[97044]: pgmap v10270: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:50:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:11 smithi110 ceph-mon[97633]: pgmap v10270: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:50:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:11.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:11.334979+0000) 2022-09-10T09:50:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:11.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:11.816826+0000) 2022-09-10T09:50:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:12.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:12.335212+0000) 2022-09-10T09:50:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:12.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:12.817007+0000) 2022-09-10T09:50:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:13 smithi110 ceph-mon[97633]: pgmap v10271: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:50:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:13 smithi110 ceph-mon[97633]: osdmap e723: 12 total, 12 up, 12 in 2022-09-10T09:50:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:13 smithi104 ceph-mon[97044]: pgmap v10271: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T09:50:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:13 smithi104 ceph-mon[97044]: osdmap e723: 12 total, 12 up, 12 in 2022-09-10T09:50:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:13.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:13.335377+0000) 2022-09-10T09:50:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:13.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:13.817275+0000) 2022-09-10T09:50:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:14.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:14.335620+0000) 2022-09-10T09:50:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:14.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:14.817502+0000) 2022-09-10T09:50:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:15 smithi110 ceph-mon[97633]: pgmap v10273: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:50:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:15 smithi104 ceph-mon[97044]: pgmap v10273: 129 pgs: 129 active+clean; 8.8 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:50:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:15.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:15.335851+0000) 2022-09-10T09:50:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:15.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:15.817701+0000) 2022-09-10T09:50:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:16.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:16.336049+0000) 2022-09-10T09:50:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:16.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:16.817931+0000) 2022-09-10T09:50:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:17 smithi110 ceph-mon[97633]: pgmap v10274: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:50:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:17 smithi104 ceph-mon[97044]: pgmap v10274: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:50:17.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:17.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:17.336261+0000) 2022-09-10T09:50:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:17.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:17.818104+0000) 2022-09-10T09:50:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:18.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:18.336491+0000) 2022-09-10T09:50:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:18.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:18.818296+0000) 2022-09-10T09:50:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:19 smithi110 ceph-mon[97633]: pgmap v10275: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:50:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:19 smithi104 ceph-mon[97044]: pgmap v10275: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:50:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:19.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:19.336733+0000) 2022-09-10T09:50:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:19.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:19.818447+0000) 2022-09-10T09:50:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:20.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:20.337010+0000) 2022-09-10T09:50:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:20.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:20.818701+0000) 2022-09-10T09:50:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:21 smithi110 ceph-mon[97633]: pgmap v10276: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:50:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:21 smithi104 ceph-mon[97044]: pgmap v10276: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:50:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:21.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:21.337191+0000) 2022-09-10T09:50:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:21.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:21.818966+0000) 2022-09-10T09:50:22.717 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:22.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:22.337411+0000) 2022-09-10T09:50:23.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:50:22.716+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:50:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:22.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:22.819250+0000) 2022-09-10T09:50:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:23 smithi110 ceph-mon[97633]: pgmap v10277: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:50:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:23 smithi104 ceph-mon[97044]: pgmap v10277: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:50:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:23.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:23.337646+0000) 2022-09-10T09:50:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:23.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:23.819497+0000) 2022-09-10T09:50:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:24.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:24.337837+0000) 2022-09-10T09:50:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:24.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:24.819737+0000) 2022-09-10T09:50:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:25 smithi110 ceph-mon[97633]: pgmap v10278: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:50:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:25 smithi104 ceph-mon[97044]: pgmap v10278: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:50:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:25.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:25.338070+0000) 2022-09-10T09:50:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:25.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:25.820009+0000) 2022-09-10T09:50:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:26.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:26.338274+0000) 2022-09-10T09:50:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:26.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:26.820234+0000) 2022-09-10T09:50:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:27 smithi110 ceph-mon[97633]: pgmap v10279: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:50:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:27 smithi104 ceph-mon[97044]: pgmap v10279: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:50:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:27.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:27.338516+0000) 2022-09-10T09:50:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:27.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:27.820454+0000) 2022-09-10T09:50:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:28.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:28.338735+0000) 2022-09-10T09:50:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:28.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:28.820728+0000) 2022-09-10T09:50:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:29 smithi110 ceph-mon[97633]: pgmap v10280: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:29 smithi104 ceph-mon[97044]: pgmap v10280: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:29.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:29.338911+0000) 2022-09-10T09:50:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:29.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:29.820999+0000) 2022-09-10T09:50:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:30.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:30.339147+0000) 2022-09-10T09:50:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:30.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:30.821204+0000) 2022-09-10T09:50:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:31 smithi110 ceph-mon[97633]: pgmap v10281: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:31 smithi104 ceph-mon[97044]: pgmap v10281: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:31.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:31.339395+0000) 2022-09-10T09:50:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:31.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:31.821365+0000) 2022-09-10T09:50:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:32.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:32.339638+0000) 2022-09-10T09:50:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:32.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:32.821491+0000) 2022-09-10T09:50:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:33 smithi110 ceph-mon[97633]: pgmap v10282: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:33 smithi104 ceph-mon[97044]: pgmap v10282: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:33.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:33.339846+0000) 2022-09-10T09:50:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:33.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:33.821691+0000) 2022-09-10T09:50:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:34.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:34.340025+0000) 2022-09-10T09:50:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:34.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:34.821884+0000) 2022-09-10T09:50:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:35 smithi110 ceph-mon[97633]: pgmap v10283: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:35 smithi104 ceph-mon[97044]: pgmap v10283: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:35.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:35.340223+0000) 2022-09-10T09:50:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:35.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:35.822146+0000) 2022-09-10T09:50:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:36.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:36.340482+0000) 2022-09-10T09:50:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:36.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:36.822382+0000) 2022-09-10T09:50:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:37 smithi110 ceph-mon[97633]: pgmap v10284: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:37 smithi104 ceph-mon[97044]: pgmap v10284: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:37.725 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:37.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:37.340641+0000) 2022-09-10T09:50:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:37.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:37.822629+0000) 2022-09-10T09:50:38.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:50:37.717+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:50:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:38.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:38.340902+0000) 2022-09-10T09:50:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:38.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:38.822791+0000) 2022-09-10T09:50:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:39 smithi110 ceph-mon[97633]: pgmap v10285: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:39 smithi104 ceph-mon[97044]: pgmap v10285: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:39.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:39.341072+0000) 2022-09-10T09:50:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:39.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:39.822984+0000) 2022-09-10T09:50:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:40.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:40.341275+0000) 2022-09-10T09:50:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:40.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:40.823141+0000) 2022-09-10T09:50:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:41 smithi110 ceph-mon[97633]: pgmap v10286: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:41 smithi104 ceph-mon[97044]: pgmap v10286: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:41.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:41.341490+0000) 2022-09-10T09:50:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:41.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:41.823331+0000) 2022-09-10T09:50:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:42.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:42.341675+0000) 2022-09-10T09:50:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:42.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:42.823550+0000) 2022-09-10T09:50:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:43 smithi110 ceph-mon[97633]: pgmap v10287: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:43 smithi104 ceph-mon[97044]: pgmap v10287: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:43.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:43.341864+0000) 2022-09-10T09:50:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:43.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:43.823750+0000) 2022-09-10T09:50:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:44.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:44.342130+0000) 2022-09-10T09:50:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:44.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:44.823992+0000) 2022-09-10T09:50:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:45 smithi110 ceph-mon[97633]: pgmap v10288: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:50:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:50:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:50:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:50:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:45 smithi104 ceph-mon[97044]: pgmap v10288: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:50:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:50:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:50:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:50:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:45.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:45.342359+0000) 2022-09-10T09:50:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:45.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:45.824254+0000) 2022-09-10T09:50:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:46.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:46.342600+0000) 2022-09-10T09:50:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:46.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:46.824512+0000) 2022-09-10T09:50:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:47 smithi110 ceph-mon[97633]: pgmap v10289: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:47 smithi104 ceph-mon[97044]: pgmap v10289: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:47.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:47.342816+0000) 2022-09-10T09:50:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:47.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:47.824762+0000) 2022-09-10T09:50:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:48.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:48.343091+0000) 2022-09-10T09:50:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:48.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:48.824938+0000) 2022-09-10T09:50:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:49 smithi110 ceph-mon[97633]: pgmap v10290: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:49 smithi104 ceph-mon[97044]: pgmap v10290: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:49.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:49.343351+0000) 2022-09-10T09:50:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:49.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:49.825213+0000) 2022-09-10T09:50:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:50.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:50.343503+0000) 2022-09-10T09:50:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:50.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:50.825437+0000) 2022-09-10T09:50:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:51 smithi110 ceph-mon[97633]: pgmap v10291: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:51 smithi104 ceph-mon[97044]: pgmap v10291: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:51.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:51.343737+0000) 2022-09-10T09:50:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:51.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:51.825591+0000) 2022-09-10T09:50:52.735 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:52.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:52.343994+0000) 2022-09-10T09:50:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:50:52.733+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:50:53.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:52.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:52.825724+0000) 2022-09-10T09:50:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:53 smithi110 ceph-mon[97633]: pgmap v10292: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:53 smithi104 ceph-mon[97044]: pgmap v10292: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:53.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:53.344193+0000) 2022-09-10T09:50:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:53.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:53.825850+0000) 2022-09-10T09:50:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:54.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:54.344409+0000) 2022-09-10T09:50:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:54.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:54.826068+0000) 2022-09-10T09:50:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:55 smithi110 ceph-mon[97633]: pgmap v10293: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:55 smithi104 ceph-mon[97044]: pgmap v10293: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:55.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:55.344647+0000) 2022-09-10T09:50:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:55.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:55.826337+0000) 2022-09-10T09:50:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:56.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:56.344909+0000) 2022-09-10T09:50:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:56.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:56.826575+0000) 2022-09-10T09:50:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:57 smithi110 ceph-mon[97633]: pgmap v10294: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:57 smithi104 ceph-mon[97044]: pgmap v10294: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:50:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:57.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:57.345141+0000) 2022-09-10T09:50:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:57.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:57.826791+0000) 2022-09-10T09:50:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:58.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:58.345299+0000) 2022-09-10T09:50:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:58.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:58.826935+0000) 2022-09-10T09:50:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:50:59 smithi110 ceph-mon[97633]: pgmap v10295: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:50:59 smithi104 ceph-mon[97044]: pgmap v10295: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:50:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:50:59.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:59.345462+0000) 2022-09-10T09:51:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:50:59.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:50:59.827215+0000) 2022-09-10T09:51:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:00.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:00.345686+0000) 2022-09-10T09:51:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:00.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:00.827437+0000) 2022-09-10T09:51:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:01 smithi110 ceph-mon[97633]: pgmap v10296: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:01 smithi104 ceph-mon[97044]: pgmap v10296: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:01.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:01.345910+0000) 2022-09-10T09:51:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:01.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:01.827668+0000) 2022-09-10T09:51:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:02.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:02.346145+0000) 2022-09-10T09:51:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:02.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:02.827888+0000) 2022-09-10T09:51:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:03 smithi110 ceph-mon[97633]: pgmap v10297: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:03 smithi104 ceph-mon[97044]: pgmap v10297: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:03.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:03.346342+0000) 2022-09-10T09:51:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:03.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:03.828158+0000) 2022-09-10T09:51:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:04.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:04.346576+0000) 2022-09-10T09:51:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:04.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:04.828398+0000) 2022-09-10T09:51:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:05 smithi104 ceph-mon[97044]: pgmap v10298: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:05 smithi110 ceph-mon[97633]: pgmap v10298: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:05.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:05.346770+0000) 2022-09-10T09:51:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:05.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:05.828606+0000) 2022-09-10T09:51:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:06.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:06.347005+0000) 2022-09-10T09:51:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:06.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:06.828786+0000) 2022-09-10T09:51:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:07 smithi104 ceph-mon[97044]: pgmap v10299: 129 pgs: 129 active+clean; 8.8 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:51:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:07 smithi110 ceph-mon[97633]: pgmap v10299: 129 pgs: 129 active+clean; 8.8 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:51:07.751 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:07.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:07.347178+0000) 2022-09-10T09:51:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:07.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:07.828912+0000) 2022-09-10T09:51:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:51:07.749+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:51:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:08.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:08.347415+0000) 2022-09-10T09:51:09.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:08.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:08.829166+0000) 2022-09-10T09:51:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:09 smithi104 ceph-mon[97044]: pgmap v10300: 129 pgs: 129 active+clean; 8.8 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:51:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:09 smithi104 ceph-mon[97044]: osdmap e724: 12 total, 12 up, 12 in 2022-09-10T09:51:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:09 smithi110 ceph-mon[97633]: pgmap v10300: 129 pgs: 129 active+clean; 8.8 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:51:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:09 smithi110 ceph-mon[97633]: osdmap e724: 12 total, 12 up, 12 in 2022-09-10T09:51:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:09.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:09.347645+0000) 2022-09-10T09:51:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:09.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:09.829335+0000) 2022-09-10T09:51:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:10.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:10.347890+0000) 2022-09-10T09:51:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:10.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:10.829509+0000) 2022-09-10T09:51:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:11 smithi104 ceph-mon[97044]: pgmap v10302: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:51:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:11 smithi110 ceph-mon[97633]: pgmap v10302: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:51:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:11.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:11.348177+0000) 2022-09-10T09:51:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:11.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:11.829716+0000) 2022-09-10T09:51:12.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:12.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:12.348436+0000) 2022-09-10T09:51:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:12.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:12.829955+0000) 2022-09-10T09:51:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:13 smithi110 ceph-mon[97633]: pgmap v10303: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:51:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:13 smithi110 ceph-mon[97633]: osdmap e725: 12 total, 12 up, 12 in 2022-09-10T09:51:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:13 smithi104 ceph-mon[97044]: pgmap v10303: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:51:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:13 smithi104 ceph-mon[97044]: osdmap e725: 12 total, 12 up, 12 in 2022-09-10T09:51:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:13.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:13.348634+0000) 2022-09-10T09:51:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:13.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:13.830216+0000) 2022-09-10T09:51:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:14.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:14.348857+0000) 2022-09-10T09:51:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:14.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:14.830467+0000) 2022-09-10T09:51:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:15 smithi110 ceph-mon[97633]: pgmap v10305: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:51:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:15 smithi104 ceph-mon[97044]: pgmap v10305: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:51:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:15.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:15.349041+0000) 2022-09-10T09:51:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:15.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:15.830644+0000) 2022-09-10T09:51:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:16.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:16.349263+0000) 2022-09-10T09:51:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:16.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:16.830904+0000) 2022-09-10T09:51:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:17 smithi110 ceph-mon[97633]: pgmap v10306: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:51:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:17 smithi104 ceph-mon[97044]: pgmap v10306: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:51:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:17.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:17.349501+0000) 2022-09-10T09:51:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:17.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:17.831166+0000) 2022-09-10T09:51:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:18.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:18.349730+0000) 2022-09-10T09:51:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:18.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:18.831350+0000) 2022-09-10T09:51:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:19 smithi110 ceph-mon[97633]: pgmap v10307: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:51:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:19 smithi104 ceph-mon[97044]: pgmap v10307: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:51:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:19.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:19.349952+0000) 2022-09-10T09:51:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:19.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:19.831604+0000) 2022-09-10T09:51:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:20.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:20.350156+0000) 2022-09-10T09:51:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:20.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:20.831801+0000) 2022-09-10T09:51:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:21 smithi110 ceph-mon[97633]: pgmap v10308: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:51:21.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:21 smithi104 ceph-mon[97044]: pgmap v10308: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:51:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:21.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:21.350393+0000) 2022-09-10T09:51:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:21.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:21.832068+0000) 2022-09-10T09:51:22.759 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:22.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:22.350580+0000) 2022-09-10T09:51:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:22.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:22.832369+0000) 2022-09-10T09:51:23.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:51:22.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:51:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:23 smithi110 ceph-mon[97633]: pgmap v10309: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:51:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:23 smithi104 ceph-mon[97044]: pgmap v10309: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:51:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:23.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:23.350815+0000) 2022-09-10T09:51:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:23.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:23.832520+0000) 2022-09-10T09:51:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:24.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:24.351077+0000) 2022-09-10T09:51:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:24.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:24.832706+0000) 2022-09-10T09:51:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:25 smithi110 ceph-mon[97633]: pgmap v10310: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:51:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:25 smithi104 ceph-mon[97044]: pgmap v10310: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:51:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:25.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:25.351332+0000) 2022-09-10T09:51:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:25.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:25.832944+0000) 2022-09-10T09:51:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:26.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:26.351580+0000) 2022-09-10T09:51:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:26.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:26.833194+0000) 2022-09-10T09:51:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:27 smithi110 ceph-mon[97633]: pgmap v10311: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:51:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:27 smithi104 ceph-mon[97044]: pgmap v10311: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:51:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:27.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:27.351807+0000) 2022-09-10T09:51:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:27.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:27.833354+0000) 2022-09-10T09:51:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:28.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:28.352029+0000) 2022-09-10T09:51:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:28.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:28.833553+0000) 2022-09-10T09:51:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:29 smithi110 ceph-mon[97633]: pgmap v10312: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:29 smithi104 ceph-mon[97044]: pgmap v10312: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:29.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:29.352272+0000) 2022-09-10T09:51:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:29.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:29.833855+0000) 2022-09-10T09:51:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:30.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:30.352518+0000) 2022-09-10T09:51:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:30.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:30.834117+0000) 2022-09-10T09:51:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:31 smithi110 ceph-mon[97633]: pgmap v10313: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:31 smithi104 ceph-mon[97044]: pgmap v10313: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:31.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:31.352751+0000) 2022-09-10T09:51:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:31.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:31.834284+0000) 2022-09-10T09:51:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:32.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:32.352991+0000) 2022-09-10T09:51:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:32.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:32.834469+0000) 2022-09-10T09:51:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:33 smithi110 ceph-mon[97633]: pgmap v10314: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:33 smithi104 ceph-mon[97044]: pgmap v10314: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:33.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:33.353197+0000) 2022-09-10T09:51:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:33.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:33.834705+0000) 2022-09-10T09:51:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:34.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:34.353398+0000) 2022-09-10T09:51:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:34.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:34.834958+0000) 2022-09-10T09:51:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:35 smithi110 ceph-mon[97633]: pgmap v10315: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:35 smithi104 ceph-mon[97044]: pgmap v10315: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:35.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:35.353656+0000) 2022-09-10T09:51:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:35.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:35.835135+0000) 2022-09-10T09:51:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:36.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:36.353876+0000) 2022-09-10T09:51:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:36.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:36.835356+0000) 2022-09-10T09:51:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:37 smithi110 ceph-mon[97633]: pgmap v10316: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:37 smithi104 ceph-mon[97044]: pgmap v10316: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:37.769 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:37.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:37.354083+0000) 2022-09-10T09:51:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:37.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:37.835526+0000) 2022-09-10T09:51:38.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:51:37.767+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:51:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:38.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:38.354281+0000) 2022-09-10T09:51:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:38.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:38.835730+0000) 2022-09-10T09:51:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:39 smithi110 ceph-mon[97633]: pgmap v10317: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:39 smithi104 ceph-mon[97044]: pgmap v10317: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:39.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:39.354518+0000) 2022-09-10T09:51:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:39.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:39.835893+0000) 2022-09-10T09:51:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:40.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:40.354721+0000) 2022-09-10T09:51:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:40.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:40.836067+0000) 2022-09-10T09:51:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:41 smithi110 ceph-mon[97633]: pgmap v10318: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:41 smithi104 ceph-mon[97044]: pgmap v10318: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:41.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:41.354937+0000) 2022-09-10T09:51:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:41.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:41.836235+0000) 2022-09-10T09:51:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:42.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:42.355076+0000) 2022-09-10T09:51:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:42.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:42.836430+0000) 2022-09-10T09:51:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:43 smithi110 ceph-mon[97633]: pgmap v10319: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:43 smithi104 ceph-mon[97044]: pgmap v10319: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:43.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:43.355317+0000) 2022-09-10T09:51:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:43.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:43.836662+0000) 2022-09-10T09:51:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:44.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:44.355504+0000) 2022-09-10T09:51:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:44.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:44.836848+0000) 2022-09-10T09:51:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:45 smithi110 ceph-mon[97633]: pgmap v10320: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:51:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:51:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:51:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:51:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:45 smithi104 ceph-mon[97044]: pgmap v10320: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:51:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:51:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:51:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:51:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:45.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:45.355775+0000) 2022-09-10T09:51:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:45.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:45.837082+0000) 2022-09-10T09:51:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:46.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:46.356036+0000) 2022-09-10T09:51:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:46.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:46.837281+0000) 2022-09-10T09:51:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:47 smithi110 ceph-mon[97633]: pgmap v10321: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:47 smithi104 ceph-mon[97044]: pgmap v10321: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:47.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:47.356239+0000) 2022-09-10T09:51:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:47.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:47.837508+0000) 2022-09-10T09:51:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:48.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:48.356424+0000) 2022-09-10T09:51:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:48.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:48.837651+0000) 2022-09-10T09:51:49.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:49 smithi110 ceph-mon[97633]: pgmap v10322: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:49 smithi104 ceph-mon[97044]: pgmap v10322: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:49.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:49.356598+0000) 2022-09-10T09:51:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:49.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:49.837787+0000) 2022-09-10T09:51:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:50.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:50.356822+0000) 2022-09-10T09:51:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:50.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:50.838063+0000) 2022-09-10T09:51:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:51 smithi104 ceph-mon[97044]: pgmap v10323: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:51 smithi110 ceph-mon[97633]: pgmap v10323: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:51.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:51.357017+0000) 2022-09-10T09:51:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:51.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:51.838334+0000) 2022-09-10T09:51:52.783 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:52.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:52.357251+0000) 2022-09-10T09:51:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:51:52.781+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:51:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:52.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:52.838518+0000) 2022-09-10T09:51:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:53 smithi104 ceph-mon[97044]: pgmap v10324: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:53 smithi110 ceph-mon[97633]: pgmap v10324: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:53.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:53.357454+0000) 2022-09-10T09:51:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:53.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:53.838790+0000) 2022-09-10T09:51:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:54.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:54.357725+0000) 2022-09-10T09:51:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:54.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:54.839070+0000) 2022-09-10T09:51:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:55 smithi104 ceph-mon[97044]: pgmap v10325: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:55 smithi110 ceph-mon[97633]: pgmap v10325: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:55.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:55.357967+0000) 2022-09-10T09:51:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:55.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:55.839350+0000) 2022-09-10T09:51:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:56.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:56.358208+0000) 2022-09-10T09:51:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:56.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:56.839536+0000) 2022-09-10T09:51:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:57 smithi104 ceph-mon[97044]: pgmap v10326: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:57 smithi110 ceph-mon[97633]: pgmap v10326: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:51:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:57.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:57.358475+0000) 2022-09-10T09:51:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:57.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:57.839810+0000) 2022-09-10T09:51:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:58.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:58.358688+0000) 2022-09-10T09:51:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:58.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:58.839965+0000) 2022-09-10T09:51:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:51:59 smithi104 ceph-mon[97044]: pgmap v10327: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:51:59 smithi110 ceph-mon[97633]: pgmap v10327: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:51:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:51:59.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:59.358954+0000) 2022-09-10T09:52:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:51:59.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:51:59.840176+0000) 2022-09-10T09:52:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:00.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:00.359166+0000) 2022-09-10T09:52:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:00.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:00.840416+0000) 2022-09-10T09:52:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:01 smithi104 ceph-mon[97044]: pgmap v10328: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:01 smithi110 ceph-mon[97633]: pgmap v10328: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:01.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:01.359361+0000) 2022-09-10T09:52:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:01.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:01.840643+0000) 2022-09-10T09:52:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:02.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:02.359541+0000) 2022-09-10T09:52:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:02.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:02.840912+0000) 2022-09-10T09:52:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:03 smithi104 ceph-mon[97044]: pgmap v10329: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:03 smithi110 ceph-mon[97633]: pgmap v10329: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:03.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:03.359739+0000) 2022-09-10T09:52:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:03.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:03.841163+0000) 2022-09-10T09:52:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:04.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:04.359998+0000) 2022-09-10T09:52:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:04.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:04.841334+0000) 2022-09-10T09:52:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:05 smithi104 ceph-mon[97044]: pgmap v10330: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:05 smithi110 ceph-mon[97633]: pgmap v10330: 129 pgs: 129 active+clean; 8.8 MiB data, 1004 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:05.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:05.360247+0000) 2022-09-10T09:52:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:05.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:05.841596+0000) 2022-09-10T09:52:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:06.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:06.360466+0000) 2022-09-10T09:52:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:06.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:06.841810+0000) 2022-09-10T09:52:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:07 smithi104 ceph-mon[97044]: pgmap v10331: 129 pgs: 129 active+clean; 8.8 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:52:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:07 smithi110 ceph-mon[97633]: pgmap v10331: 129 pgs: 129 active+clean; 8.8 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:52:07.791 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:07.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:07.360705+0000) 2022-09-10T09:52:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:07.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:07.842121+0000) 2022-09-10T09:52:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:52:07.789+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:52:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:08.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:08.360874+0000) 2022-09-10T09:52:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:08.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:08.842283+0000) 2022-09-10T09:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:09 smithi104 ceph-mon[97044]: pgmap v10332: 129 pgs: 129 active+clean; 8.8 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:52:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:09 smithi104 ceph-mon[97044]: osdmap e726: 12 total, 12 up, 12 in 2022-09-10T09:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:09 smithi110 ceph-mon[97633]: pgmap v10332: 129 pgs: 129 active+clean; 8.8 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:09 smithi110 ceph-mon[97633]: osdmap e726: 12 total, 12 up, 12 in 2022-09-10T09:52:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:09.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:09.361055+0000) 2022-09-10T09:52:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:09.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:09.842485+0000) 2022-09-10T09:52:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:10.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:10.361307+0000) 2022-09-10T09:52:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:10.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:10.842686+0000) 2022-09-10T09:52:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:11 smithi104 ceph-mon[97044]: pgmap v10334: 129 pgs: 129 active+clean; 8.8 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:52:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:11 smithi110 ceph-mon[97633]: pgmap v10334: 129 pgs: 129 active+clean; 8.8 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:52:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:11.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:11.361492+0000) 2022-09-10T09:52:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:11.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:11.842915+0000) 2022-09-10T09:52:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:12.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:12.361752+0000) 2022-09-10T09:52:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:12.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:12.843083+0000) 2022-09-10T09:52:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:13 smithi110 ceph-mon[97633]: pgmap v10335: 129 pgs: 129 active+clean; 8.8 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:52:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:13 smithi110 ceph-mon[97633]: osdmap e727: 12 total, 12 up, 12 in 2022-09-10T09:52:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:13 smithi104 ceph-mon[97044]: pgmap v10335: 129 pgs: 129 active+clean; 8.8 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:52:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:13 smithi104 ceph-mon[97044]: osdmap e727: 12 total, 12 up, 12 in 2022-09-10T09:52:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:13.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:13.361953+0000) 2022-09-10T09:52:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:13.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:13.843381+0000) 2022-09-10T09:52:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:14.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:14.362181+0000) 2022-09-10T09:52:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:14.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:14.843631+0000) 2022-09-10T09:52:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:15 smithi110 ceph-mon[97633]: pgmap v10337: 129 pgs: 129 active+clean; 8.8 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:15 smithi104 ceph-mon[97044]: pgmap v10337: 129 pgs: 129 active+clean; 8.8 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:52:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:15.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:15.362345+0000) 2022-09-10T09:52:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:15.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:15.843885+0000) 2022-09-10T09:52:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:16.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:16.362584+0000) 2022-09-10T09:52:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:16.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:16.844150+0000) 2022-09-10T09:52:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:17 smithi110 ceph-mon[97633]: pgmap v10338: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:52:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:17 smithi104 ceph-mon[97044]: pgmap v10338: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:52:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:17.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:17.362819+0000) 2022-09-10T09:52:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:17.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:17.844374+0000) 2022-09-10T09:52:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:18.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:18.363061+0000) 2022-09-10T09:52:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:18.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:18.844663+0000) 2022-09-10T09:52:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:19 smithi110 ceph-mon[97633]: pgmap v10339: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:52:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:19 smithi104 ceph-mon[97044]: pgmap v10339: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:52:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:19.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:19.363254+0000) 2022-09-10T09:52:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:19.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:19.844889+0000) 2022-09-10T09:52:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:20.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:20.363453+0000) 2022-09-10T09:52:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:20.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:20.845071+0000) 2022-09-10T09:52:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:21 smithi110 ceph-mon[97633]: pgmap v10340: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:52:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:21 smithi104 ceph-mon[97044]: pgmap v10340: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:52:21.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:21.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:21.363596+0000) 2022-09-10T09:52:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:21.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:21.845245+0000) 2022-09-10T09:52:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:22.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:22.363840+0000) 2022-09-10T09:52:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:22.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:22.845460+0000) 2022-09-10T09:52:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:52:22.804+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:52:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:23 smithi110 ceph-mon[97633]: pgmap v10341: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:52:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:23 smithi104 ceph-mon[97044]: pgmap v10341: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:52:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:23.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:23.364071+0000) 2022-09-10T09:52:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:23.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:23.845723+0000) 2022-09-10T09:52:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:24.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:24.364245+0000) 2022-09-10T09:52:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:24.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:24.845945+0000) 2022-09-10T09:52:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:25 smithi110 ceph-mon[97633]: pgmap v10342: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:52:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:25 smithi104 ceph-mon[97044]: pgmap v10342: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:52:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:25.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:25.364409+0000) 2022-09-10T09:52:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:25.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:25.846223+0000) 2022-09-10T09:52:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:26.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:26.364653+0000) 2022-09-10T09:52:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:26.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:26.846421+0000) 2022-09-10T09:52:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:27 smithi110 ceph-mon[97633]: pgmap v10343: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:52:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:27 smithi104 ceph-mon[97044]: pgmap v10343: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:52:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:27.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:27.364900+0000) 2022-09-10T09:52:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:27.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:27.846648+0000) 2022-09-10T09:52:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:28.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:28.365178+0000) 2022-09-10T09:52:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:28.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:28.846902+0000) 2022-09-10T09:52:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:29 smithi110 ceph-mon[97633]: pgmap v10344: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:29 smithi104 ceph-mon[97044]: pgmap v10344: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:29.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:29.365324+0000) 2022-09-10T09:52:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:29.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:29.847155+0000) 2022-09-10T09:52:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:30.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:30.365571+0000) 2022-09-10T09:52:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:30.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:30.847368+0000) 2022-09-10T09:52:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:31 smithi110 ceph-mon[97633]: pgmap v10345: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:31 smithi104 ceph-mon[97044]: pgmap v10345: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:31.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:31.365783+0000) 2022-09-10T09:52:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:31.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:31.847573+0000) 2022-09-10T09:52:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:32.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:32.366042+0000) 2022-09-10T09:52:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:32.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:32.847789+0000) 2022-09-10T09:52:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:33 smithi110 ceph-mon[97633]: pgmap v10346: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:33 smithi104 ceph-mon[97044]: pgmap v10346: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:33.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:33.366284+0000) 2022-09-10T09:52:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:33.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:33.847957+0000) 2022-09-10T09:52:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:34.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:34.366490+0000) 2022-09-10T09:52:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:34.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:34.848213+0000) 2022-09-10T09:52:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:35 smithi110 ceph-mon[97633]: pgmap v10347: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:35 smithi104 ceph-mon[97044]: pgmap v10347: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:35.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:35.366677+0000) 2022-09-10T09:52:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:35.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:35.848385+0000) 2022-09-10T09:52:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:36.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:36.366886+0000) 2022-09-10T09:52:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:36.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:36.848615+0000) 2022-09-10T09:52:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:37 smithi110 ceph-mon[97633]: pgmap v10348: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:37 smithi104 ceph-mon[97044]: pgmap v10348: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:37.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:37.367159+0000) 2022-09-10T09:52:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:37.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:37.848867+0000) 2022-09-10T09:52:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:52:37.820+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:52:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:38.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:38.367376+0000) 2022-09-10T09:52:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:38.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:38.849091+0000) 2022-09-10T09:52:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:39 smithi110 ceph-mon[97633]: pgmap v10349: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:39 smithi104 ceph-mon[97044]: pgmap v10349: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:39.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:39.367606+0000) 2022-09-10T09:52:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:39.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:39.849258+0000) 2022-09-10T09:52:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:40.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:40.367807+0000) 2022-09-10T09:52:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:40.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:40.849477+0000) 2022-09-10T09:52:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:41 smithi110 ceph-mon[97633]: pgmap v10350: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:41 smithi104 ceph-mon[97044]: pgmap v10350: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:41.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:41.368056+0000) 2022-09-10T09:52:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:41.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:41.849757+0000) 2022-09-10T09:52:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:42.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:42.368269+0000) 2022-09-10T09:52:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:42.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:42.849947+0000) 2022-09-10T09:52:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:43 smithi104 ceph-mon[97044]: pgmap v10351: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:43 smithi110 ceph-mon[97633]: pgmap v10351: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:43.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:43.368449+0000) 2022-09-10T09:52:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:43.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:43.850219+0000) 2022-09-10T09:52:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:44.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:44.368649+0000) 2022-09-10T09:52:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:44.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:44.850493+0000) 2022-09-10T09:52:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:45 smithi104 ceph-mon[97044]: pgmap v10352: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:52:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:52:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:52:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:52:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:45 smithi110 ceph-mon[97633]: pgmap v10352: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:52:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:52:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:52:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:52:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:45.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:45.368866+0000) 2022-09-10T09:52:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:45.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:45.850779+0000) 2022-09-10T09:52:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:46.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:46.369088+0000) 2022-09-10T09:52:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:46.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:46.851040+0000) 2022-09-10T09:52:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:47 smithi104 ceph-mon[97044]: pgmap v10353: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:47 smithi110 ceph-mon[97633]: pgmap v10353: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:47.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:47.369360+0000) 2022-09-10T09:52:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:47.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:47.851224+0000) 2022-09-10T09:52:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:48.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:48.369610+0000) 2022-09-10T09:52:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:48.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:48.851492+0000) 2022-09-10T09:52:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:49 smithi104 ceph-mon[97044]: pgmap v10354: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:49 smithi110 ceph-mon[97633]: pgmap v10354: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:49.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:49.369833+0000) 2022-09-10T09:52:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:49.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:49.851690+0000) 2022-09-10T09:52:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:50.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:50.370078+0000) 2022-09-10T09:52:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:50.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:50.851855+0000) 2022-09-10T09:52:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:51 smithi104 ceph-mon[97044]: pgmap v10355: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:51 smithi110 ceph-mon[97633]: pgmap v10355: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:51.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:51.370321+0000) 2022-09-10T09:52:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:51.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:51.852121+0000) 2022-09-10T09:52:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:52.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:52.370515+0000) 2022-09-10T09:52:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:52.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:52.852382+0000) 2022-09-10T09:52:53.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:52:52.836+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:52:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:53 smithi104 ceph-mon[97044]: pgmap v10356: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:53 smithi110 ceph-mon[97633]: pgmap v10356: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:53.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:53.370757+0000) 2022-09-10T09:52:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:53.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:53.852641+0000) 2022-09-10T09:52:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:54.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:54.370993+0000) 2022-09-10T09:52:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:54.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:54.852858+0000) 2022-09-10T09:52:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:55 smithi104 ceph-mon[97044]: pgmap v10357: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:55 smithi110 ceph-mon[97633]: pgmap v10357: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:55.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:55.371163+0000) 2022-09-10T09:52:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:55.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:55.853162+0000) 2022-09-10T09:52:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:56.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:56.371523+0000) 2022-09-10T09:52:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:56.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:56.853411+0000) 2022-09-10T09:52:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:57 smithi104 ceph-mon[97044]: pgmap v10358: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:57 smithi110 ceph-mon[97633]: pgmap v10358: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:52:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:57.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:57.371757+0000) 2022-09-10T09:52:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:57.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:57.853674+0000) 2022-09-10T09:52:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:58.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:58.372004+0000) 2022-09-10T09:52:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:58.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:58.853969+0000) 2022-09-10T09:52:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:52:59 smithi104 ceph-mon[97044]: pgmap v10359: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:52:59 smithi110 ceph-mon[97633]: pgmap v10359: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:52:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:52:59.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:59.372144+0000) 2022-09-10T09:53:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:52:59.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:52:59.854163+0000) 2022-09-10T09:53:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:00.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:00.372335+0000) 2022-09-10T09:53:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:00.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:00.854385+0000) 2022-09-10T09:53:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:01 smithi104 ceph-mon[97044]: pgmap v10360: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:01 smithi110 ceph-mon[97633]: pgmap v10360: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:01.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:01.372593+0000) 2022-09-10T09:53:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:01.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:01.854529+0000) 2022-09-10T09:53:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:02.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:02.372850+0000) 2022-09-10T09:53:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:02.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:02.854711+0000) 2022-09-10T09:53:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:03 smithi104 ceph-mon[97044]: pgmap v10361: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:03 smithi110 ceph-mon[97633]: pgmap v10361: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:03.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:03.373128+0000) 2022-09-10T09:53:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:03.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:03.854916+0000) 2022-09-10T09:53:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:04.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:04.373301+0000) 2022-09-10T09:53:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:04.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:04.855109+0000) 2022-09-10T09:53:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:05 smithi104 ceph-mon[97044]: pgmap v10362: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:05 smithi110 ceph-mon[97633]: pgmap v10362: 129 pgs: 129 active+clean; 8.8 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:05.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:05.373470+0000) 2022-09-10T09:53:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:05.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:05.855258+0000) 2022-09-10T09:53:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:06.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:06.373664+0000) 2022-09-10T09:53:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:06.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:06.855462+0000) 2022-09-10T09:53:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:07 smithi104 ceph-mon[97044]: pgmap v10363: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:53:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:07 smithi110 ceph-mon[97633]: pgmap v10363: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:53:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:07.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:07.373848+0000) 2022-09-10T09:53:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:07.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:07.855652+0000) 2022-09-10T09:53:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:53:07.840+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:53:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:08.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:08.374009+0000) 2022-09-10T09:53:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:08.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:08.855839+0000) 2022-09-10T09:53:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:09 smithi104 ceph-mon[97044]: pgmap v10364: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:53:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:09 smithi104 ceph-mon[97044]: osdmap e728: 12 total, 12 up, 12 in 2022-09-10T09:53:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:09 smithi110 ceph-mon[97633]: pgmap v10364: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:53:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:09 smithi110 ceph-mon[97633]: osdmap e728: 12 total, 12 up, 12 in 2022-09-10T09:53:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:09.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:09.374181+0000) 2022-09-10T09:53:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:09.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:09.856069+0000) 2022-09-10T09:53:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:10.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:10.374368+0000) 2022-09-10T09:53:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:10.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:10.856373+0000) 2022-09-10T09:53:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:11 smithi104 ceph-mon[97044]: pgmap v10366: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:53:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:11 smithi110 ceph-mon[97633]: pgmap v10366: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:53:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:11.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:11.374552+0000) 2022-09-10T09:53:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:11.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:11.856626+0000) 2022-09-10T09:53:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:12.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:12.374752+0000) 2022-09-10T09:53:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:12.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:12.856814+0000) 2022-09-10T09:53:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:13 smithi110 ceph-mon[97633]: pgmap v10367: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:53:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:13 smithi110 ceph-mon[97633]: osdmap e729: 12 total, 12 up, 12 in 2022-09-10T09:53:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:13 smithi104 ceph-mon[97044]: pgmap v10367: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:53:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:13 smithi104 ceph-mon[97044]: osdmap e729: 12 total, 12 up, 12 in 2022-09-10T09:53:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:13.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:13.374926+0000) 2022-09-10T09:53:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:13.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:13.857025+0000) 2022-09-10T09:53:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:14.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:14.375142+0000) 2022-09-10T09:53:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:14.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:14.857224+0000) 2022-09-10T09:53:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:15 smithi110 ceph-mon[97633]: pgmap v10369: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:53:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:15 smithi104 ceph-mon[97044]: pgmap v10369: 129 pgs: 129 active+clean; 8.8 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:53:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:15.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:15.375291+0000) 2022-09-10T09:53:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:15.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:15.857487+0000) 2022-09-10T09:53:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:16.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:16.375497+0000) 2022-09-10T09:53:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:16.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:16.857725+0000) 2022-09-10T09:53:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:17 smithi110 ceph-mon[97633]: pgmap v10370: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:53:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:17 smithi104 ceph-mon[97044]: pgmap v10370: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:53:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:17.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:17.375685+0000) 2022-09-10T09:53:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:17.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:17.858057+0000) 2022-09-10T09:53:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:18.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:18.375854+0000) 2022-09-10T09:53:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:18.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:18.858162+0000) 2022-09-10T09:53:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:19 smithi110 ceph-mon[97633]: pgmap v10371: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:53:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:19 smithi104 ceph-mon[97044]: pgmap v10371: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:53:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:19.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:19.376039+0000) 2022-09-10T09:53:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:19.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:19.858375+0000) 2022-09-10T09:53:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:20.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:20.376310+0000) 2022-09-10T09:53:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:20.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:20.858572+0000) 2022-09-10T09:53:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:21 smithi110 ceph-mon[97633]: pgmap v10372: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:53:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:21 smithi104 ceph-mon[97044]: pgmap v10372: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:53:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:21.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:21.376545+0000) 2022-09-10T09:53:22.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:21.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:21.858829+0000) 2022-09-10T09:53:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:22.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:22.376763+0000) 2022-09-10T09:53:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:22.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:22.859030+0000) 2022-09-10T09:53:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:53:22.856+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:53:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:23 smithi110 ceph-mon[97633]: pgmap v10373: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:53:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:23 smithi104 ceph-mon[97044]: pgmap v10373: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:53:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:23.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:23.376971+0000) 2022-09-10T09:53:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:23.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:23.859226+0000) 2022-09-10T09:53:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:24.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:24.377164+0000) 2022-09-10T09:53:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:24.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:24.859452+0000) 2022-09-10T09:53:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:25 smithi110 ceph-mon[97633]: pgmap v10374: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:53:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:25 smithi104 ceph-mon[97044]: pgmap v10374: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:53:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:25.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:25.377378+0000) 2022-09-10T09:53:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:25.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:25.859735+0000) 2022-09-10T09:53:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:26.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:26.377543+0000) 2022-09-10T09:53:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:26.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:26.860004+0000) 2022-09-10T09:53:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:27 smithi110 ceph-mon[97633]: pgmap v10375: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:53:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:27 smithi104 ceph-mon[97044]: pgmap v10375: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:53:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:27.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:27.377805+0000) 2022-09-10T09:53:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:27.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:27.860251+0000) 2022-09-10T09:53:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:28.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:28.378030+0000) 2022-09-10T09:53:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:28.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:28.860476+0000) 2022-09-10T09:53:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:29 smithi110 ceph-mon[97633]: pgmap v10376: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:53:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:29 smithi104 ceph-mon[97044]: pgmap v10376: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:53:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:29.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:29.378263+0000) 2022-09-10T09:53:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:29.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:29.860598+0000) 2022-09-10T09:53:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:30.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:30.378490+0000) 2022-09-10T09:53:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:30.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:30.860834+0000) 2022-09-10T09:53:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:31 smithi110 ceph-mon[97633]: pgmap v10377: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:31 smithi104 ceph-mon[97044]: pgmap v10377: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:31.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:31.378665+0000) 2022-09-10T09:53:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:31.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:31.861080+0000) 2022-09-10T09:53:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:32.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:32.378900+0000) 2022-09-10T09:53:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:32.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:32.861354+0000) 2022-09-10T09:53:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:33 smithi110 ceph-mon[97633]: pgmap v10378: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:53:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:33 smithi104 ceph-mon[97044]: pgmap v10378: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:53:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:33.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:33.379078+0000) 2022-09-10T09:53:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:33.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:33.861610+0000) 2022-09-10T09:53:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:34.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:34.379271+0000) 2022-09-10T09:53:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:34.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:34.861819+0000) 2022-09-10T09:53:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:35 smithi104 ceph-mon[97044]: pgmap v10379: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:53:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:35 smithi110 ceph-mon[97633]: pgmap v10379: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:53:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:35.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:35.379455+0000) 2022-09-10T09:53:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:35.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:35.862084+0000) 2022-09-10T09:53:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:36.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:36.379731+0000) 2022-09-10T09:53:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:36.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:36.862301+0000) 2022-09-10T09:53:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:37 smithi104 ceph-mon[97044]: pgmap v10380: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:37 smithi110 ceph-mon[97633]: pgmap v10380: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:37.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:37.379989+0000) 2022-09-10T09:53:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:37.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:37.862558+0000) 2022-09-10T09:53:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:53:37.858+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:53:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:38.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:38.380225+0000) 2022-09-10T09:53:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:38.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:38.862771+0000) 2022-09-10T09:53:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:39 smithi104 ceph-mon[97044]: pgmap v10381: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:39 smithi110 ceph-mon[97633]: pgmap v10381: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:39.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:39.380388+0000) 2022-09-10T09:53:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:39.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:39.862917+0000) 2022-09-10T09:53:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:40.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:40.380569+0000) 2022-09-10T09:53:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:40.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:40.863076+0000) 2022-09-10T09:53:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:41 smithi104 ceph-mon[97044]: pgmap v10382: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:41 smithi110 ceph-mon[97633]: pgmap v10382: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:41.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:41.380796+0000) 2022-09-10T09:53:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:41.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:41.863325+0000) 2022-09-10T09:53:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:42.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:42.381044+0000) 2022-09-10T09:53:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:42.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:42.863583+0000) 2022-09-10T09:53:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:43 smithi104 ceph-mon[97044]: pgmap v10383: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:43 smithi110 ceph-mon[97633]: pgmap v10383: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:43.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:43.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:43.381252+0000) 2022-09-10T09:53:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:43.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:43.863780+0000) 2022-09-10T09:53:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:44.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:44.381437+0000) 2022-09-10T09:53:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:44.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:44.863979+0000) 2022-09-10T09:53:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:45 smithi104 ceph-mon[97044]: pgmap v10384: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:53:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:53:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:53:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:53:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:45 smithi110 ceph-mon[97633]: pgmap v10384: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:53:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:53:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:53:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:53:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:45.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:45.381680+0000) 2022-09-10T09:53:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:45.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:45.864249+0000) 2022-09-10T09:53:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:46.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:46.381911+0000) 2022-09-10T09:53:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:46.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:46.864474+0000) 2022-09-10T09:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:47 smithi104 ceph-mon[97044]: pgmap v10385: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:47 smithi110 ceph-mon[97633]: pgmap v10385: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:47.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:47.382131+0000) 2022-09-10T09:53:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:47.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:47.864631+0000) 2022-09-10T09:53:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:48.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:48.382298+0000) 2022-09-10T09:53:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:48.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:48.864787+0000) 2022-09-10T09:53:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:49 smithi104 ceph-mon[97044]: pgmap v10386: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:49 smithi110 ceph-mon[97633]: pgmap v10386: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:49.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:49.382480+0000) 2022-09-10T09:53:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:49.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:49.864978+0000) 2022-09-10T09:53:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:50.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:50.382614+0000) 2022-09-10T09:53:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:50.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:50.865199+0000) 2022-09-10T09:53:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:51 smithi104 ceph-mon[97044]: pgmap v10387: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:51 smithi110 ceph-mon[97633]: pgmap v10387: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:51.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:51.382804+0000) 2022-09-10T09:53:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:51.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:51.865459+0000) 2022-09-10T09:53:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:52.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:52.383019+0000) 2022-09-10T09:53:53.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:53:52.859+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:53:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:52.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:52.865620+0000) 2022-09-10T09:53:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:53 smithi104 ceph-mon[97044]: pgmap v10388: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:53 smithi110 ceph-mon[97633]: pgmap v10388: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:53.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:53.383274+0000) 2022-09-10T09:53:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:53.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:53.865864+0000) 2022-09-10T09:53:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:54.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:54.383450+0000) 2022-09-10T09:53:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:54.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:54.866067+0000) 2022-09-10T09:53:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:55 smithi104 ceph-mon[97044]: pgmap v10389: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:55 smithi110 ceph-mon[97633]: pgmap v10389: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:55.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:55.383634+0000) 2022-09-10T09:53:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:55.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:55.866284+0000) 2022-09-10T09:53:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:56.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:56.383860+0000) 2022-09-10T09:53:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:56.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:56.866453+0000) 2022-09-10T09:53:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:57 smithi104 ceph-mon[97044]: pgmap v10390: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:57 smithi110 ceph-mon[97633]: pgmap v10390: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:53:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:57.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:57.384173+0000) 2022-09-10T09:53:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:57.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:57.866688+0000) 2022-09-10T09:53:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:58.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:58.384380+0000) 2022-09-10T09:53:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:58.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:58.866924+0000) 2022-09-10T09:53:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:53:59 smithi104 ceph-mon[97044]: pgmap v10391: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:53:59 smithi110 ceph-mon[97633]: pgmap v10391: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:53:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:53:59.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:59.384605+0000) 2022-09-10T09:54:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:53:59.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:53:59.867111+0000) 2022-09-10T09:54:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:00.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:00.384772+0000) 2022-09-10T09:54:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:00.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:00.867327+0000) 2022-09-10T09:54:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:01 smithi104 ceph-mon[97044]: pgmap v10392: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:01 smithi110 ceph-mon[97633]: pgmap v10392: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:01.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:01.384950+0000) 2022-09-10T09:54:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:01.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:01.867545+0000) 2022-09-10T09:54:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:02.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:02.385184+0000) 2022-09-10T09:54:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:02.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:02.867729+0000) 2022-09-10T09:54:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:03 smithi104 ceph-mon[97044]: pgmap v10393: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:03 smithi110 ceph-mon[97633]: pgmap v10393: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:03.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:03.385382+0000) 2022-09-10T09:54:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:03.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:03.867894+0000) 2022-09-10T09:54:04.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:04.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:04.385551+0000) 2022-09-10T09:54:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:04.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:04.868073+0000) 2022-09-10T09:54:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:05 smithi104 ceph-mon[97044]: pgmap v10394: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:05 smithi110 ceph-mon[97633]: pgmap v10394: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:05.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:05.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:05.385777+0000) 2022-09-10T09:54:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:05.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:05.868339+0000) 2022-09-10T09:54:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:06.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:06.386021+0000) 2022-09-10T09:54:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:06.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:06.868504+0000) 2022-09-10T09:54:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:07 smithi104 ceph-mon[97044]: pgmap v10395: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:54:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:07 smithi110 ceph-mon[97633]: pgmap v10395: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:54:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:07.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:07.386209+0000) 2022-09-10T09:54:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:07.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:07.868692+0000) 2022-09-10T09:54:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:54:07.862+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:54:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:08.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:08.386491+0000) 2022-09-10T09:54:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:08.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:08.868857+0000) 2022-09-10T09:54:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:09 smithi104 ceph-mon[97044]: pgmap v10396: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:54:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:09 smithi104 ceph-mon[97044]: osdmap e730: 12 total, 12 up, 12 in 2022-09-10T09:54:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:09 smithi110 ceph-mon[97633]: pgmap v10396: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:54:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:09 smithi110 ceph-mon[97633]: osdmap e730: 12 total, 12 up, 12 in 2022-09-10T09:54:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:09.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:09.386670+0000) 2022-09-10T09:54:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:09.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:09.869053+0000) 2022-09-10T09:54:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:10.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:10.386911+0000) 2022-09-10T09:54:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:10.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:10.869293+0000) 2022-09-10T09:54:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:11 smithi104 ceph-mon[97044]: pgmap v10398: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:54:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:11 smithi110 ceph-mon[97633]: pgmap v10398: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:54:11.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:11.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:11.387149+0000) 2022-09-10T09:54:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:11.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:11.869477+0000) 2022-09-10T09:54:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:12.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:12.387385+0000) 2022-09-10T09:54:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:12.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:12.869593+0000) 2022-09-10T09:54:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:13 smithi110 ceph-mon[97633]: pgmap v10399: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:54:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:13 smithi110 ceph-mon[97633]: osdmap e731: 12 total, 12 up, 12 in 2022-09-10T09:54:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:13 smithi104 ceph-mon[97044]: pgmap v10399: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:54:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:13 smithi104 ceph-mon[97044]: osdmap e731: 12 total, 12 up, 12 in 2022-09-10T09:54:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:13.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:13.387613+0000) 2022-09-10T09:54:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:13.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:13.869780+0000) 2022-09-10T09:54:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:14.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:14.387829+0000) 2022-09-10T09:54:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:14.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:14.869942+0000) 2022-09-10T09:54:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:15 smithi110 ceph-mon[97633]: pgmap v10401: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:54:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:15 smithi104 ceph-mon[97044]: pgmap v10401: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:54:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:15.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:15.388051+0000) 2022-09-10T09:54:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:15.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:15.870187+0000) 2022-09-10T09:54:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:16.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:16.388253+0000) 2022-09-10T09:54:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:16.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:16.870349+0000) 2022-09-10T09:54:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:17 smithi110 ceph-mon[97633]: pgmap v10402: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:54:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:17 smithi104 ceph-mon[97044]: pgmap v10402: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:54:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:17.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:17.388417+0000) 2022-09-10T09:54:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:17.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:17.870536+0000) 2022-09-10T09:54:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:18.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:18.388664+0000) 2022-09-10T09:54:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:18.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:18.870716+0000) 2022-09-10T09:54:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:19 smithi110 ceph-mon[97633]: pgmap v10403: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:54:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:19 smithi104 ceph-mon[97044]: pgmap v10403: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:54:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:19.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:19.388851+0000) 2022-09-10T09:54:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:19.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:19.870907+0000) 2022-09-10T09:54:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:20.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:20.389070+0000) 2022-09-10T09:54:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:20.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:20.871205+0000) 2022-09-10T09:54:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:21 smithi104 ceph-mon[97044]: pgmap v10404: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:54:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:21 smithi110 ceph-mon[97633]: pgmap v10404: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:54:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:21.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:21.389339+0000) 2022-09-10T09:54:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:21.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:21.871424+0000) 2022-09-10T09:54:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:22.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:22.389596+0000) 2022-09-10T09:54:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:22.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:22.871600+0000) 2022-09-10T09:54:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:54:22.877+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:54:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:23 smithi104 ceph-mon[97044]: pgmap v10405: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:54:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:23 smithi110 ceph-mon[97633]: pgmap v10405: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:54:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:23.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:23.389829+0000) 2022-09-10T09:54:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:23.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:23.871751+0000) 2022-09-10T09:54:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:24.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:24.390121+0000) 2022-09-10T09:54:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:24.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:24.871881+0000) 2022-09-10T09:54:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:25 smithi104 ceph-mon[97044]: pgmap v10406: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:54:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:25 smithi110 ceph-mon[97633]: pgmap v10406: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:54:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:25.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:25.390305+0000) 2022-09-10T09:54:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:25.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:25.872166+0000) 2022-09-10T09:54:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:26.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:26.390432+0000) 2022-09-10T09:54:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:26.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:26.872438+0000) 2022-09-10T09:54:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:27 smithi104 ceph-mon[97044]: pgmap v10407: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:54:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:27 smithi110 ceph-mon[97633]: pgmap v10407: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:54:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:27.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:27.390674+0000) 2022-09-10T09:54:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:27.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:27.872615+0000) 2022-09-10T09:54:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:28.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:28.390947+0000) 2022-09-10T09:54:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:28.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:28.872775+0000) 2022-09-10T09:54:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:29 smithi104 ceph-mon[97044]: pgmap v10408: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:29 smithi110 ceph-mon[97633]: pgmap v10408: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:29.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:29.391191+0000) 2022-09-10T09:54:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:29.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:29.873016+0000) 2022-09-10T09:54:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:30.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:30.391366+0000) 2022-09-10T09:54:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:30.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:30.873222+0000) 2022-09-10T09:54:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:31 smithi104 ceph-mon[97044]: pgmap v10409: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:31 smithi110 ceph-mon[97633]: pgmap v10409: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:31.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:31.391563+0000) 2022-09-10T09:54:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:31.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:31.873471+0000) 2022-09-10T09:54:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:32.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:32.391740+0000) 2022-09-10T09:54:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:32.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:32.873660+0000) 2022-09-10T09:54:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:33 smithi104 ceph-mon[97044]: pgmap v10410: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:33 smithi110 ceph-mon[97633]: pgmap v10410: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:33.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:33.391936+0000) 2022-09-10T09:54:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:33.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:33.873901+0000) 2022-09-10T09:54:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:34.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:34.392194+0000) 2022-09-10T09:54:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:34.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:34.874215+0000) 2022-09-10T09:54:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:35 smithi104 ceph-mon[97044]: pgmap v10411: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:35 smithi110 ceph-mon[97633]: pgmap v10411: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:35.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:35.392407+0000) 2022-09-10T09:54:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:35.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:35.874432+0000) 2022-09-10T09:54:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:36.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:36.392659+0000) 2022-09-10T09:54:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:36.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:36.874609+0000) 2022-09-10T09:54:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:37 smithi104 ceph-mon[97044]: pgmap v10412: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:37 smithi110 ceph-mon[97633]: pgmap v10412: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:37.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:37.392925+0000) 2022-09-10T09:54:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:37.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:37.874798+0000) 2022-09-10T09:54:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:54:37.893+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:54:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:38.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:38.393175+0000) 2022-09-10T09:54:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:38.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:38.875045+0000) 2022-09-10T09:54:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:39 smithi104 ceph-mon[97044]: pgmap v10413: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:39 smithi110 ceph-mon[97633]: pgmap v10413: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:39.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:39.393341+0000) 2022-09-10T09:54:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:39.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:39.875301+0000) 2022-09-10T09:54:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:40.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:40.393603+0000) 2022-09-10T09:54:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:40.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:40.875548+0000) 2022-09-10T09:54:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:41 smithi104 ceph-mon[97044]: pgmap v10414: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:41 smithi110 ceph-mon[97633]: pgmap v10414: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:41.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:41.393810+0000) 2022-09-10T09:54:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:41.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:41.875767+0000) 2022-09-10T09:54:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:42.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:42.394026+0000) 2022-09-10T09:54:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:42.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:42.876038+0000) 2022-09-10T09:54:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:43 smithi104 ceph-mon[97044]: pgmap v10415: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:43 smithi110 ceph-mon[97633]: pgmap v10415: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:43.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:43.394224+0000) 2022-09-10T09:54:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:43.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:43.876259+0000) 2022-09-10T09:54:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:44.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:44.394428+0000) 2022-09-10T09:54:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:44.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:44.876439+0000) 2022-09-10T09:54:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:45 smithi104 ceph-mon[97044]: pgmap v10416: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:54:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:54:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:54:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:54:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:45 smithi110 ceph-mon[97633]: pgmap v10416: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:54:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:54:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:54:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:54:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:45.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:45.394629+0000) 2022-09-10T09:54:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:45.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:45.876663+0000) 2022-09-10T09:54:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:46.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:46.394783+0000) 2022-09-10T09:54:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:46.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:46.876803+0000) 2022-09-10T09:54:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:47 smithi104 ceph-mon[97044]: pgmap v10417: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:47 smithi110 ceph-mon[97633]: pgmap v10417: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:47.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:47.394967+0000) 2022-09-10T09:54:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:47.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:47.877007+0000) 2022-09-10T09:54:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:48.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:48.395163+0000) 2022-09-10T09:54:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:48.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:48.877220+0000) 2022-09-10T09:54:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:49 smithi104 ceph-mon[97044]: pgmap v10418: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:49 smithi110 ceph-mon[97633]: pgmap v10418: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:49.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:49.395372+0000) 2022-09-10T09:54:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:49.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:49.877500+0000) 2022-09-10T09:54:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:50.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:50.395623+0000) 2022-09-10T09:54:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:50.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:50.877753+0000) 2022-09-10T09:54:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:51 smithi104 ceph-mon[97044]: pgmap v10419: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:51 smithi110 ceph-mon[97633]: pgmap v10419: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:51.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:51.395889+0000) 2022-09-10T09:54:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:51.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:51.877993+0000) 2022-09-10T09:54:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:52.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:52.396170+0000) 2022-09-10T09:54:53.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:54:52.904+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:54:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:52.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:52.878215+0000) 2022-09-10T09:54:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:53 smithi104 ceph-mon[97044]: pgmap v10420: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:53 smithi110 ceph-mon[97633]: pgmap v10420: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:53.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:53.396363+0000) 2022-09-10T09:54:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:53.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:53.878448+0000) 2022-09-10T09:54:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:54.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:54.396567+0000) 2022-09-10T09:54:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:54.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:54.878583+0000) 2022-09-10T09:54:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:55 smithi104 ceph-mon[97044]: pgmap v10421: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:55 smithi110 ceph-mon[97633]: pgmap v10421: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:55.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:55.396811+0000) 2022-09-10T09:54:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:55.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:55.878796+0000) 2022-09-10T09:54:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:56.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:56.397071+0000) 2022-09-10T09:54:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:56.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:56.878976+0000) 2022-09-10T09:54:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:57 smithi104 ceph-mon[97044]: pgmap v10422: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:57 smithi110 ceph-mon[97633]: pgmap v10422: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:54:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:57.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:57.397373+0000) 2022-09-10T09:54:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:57.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:57.879195+0000) 2022-09-10T09:54:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:58.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:58.397564+0000) 2022-09-10T09:54:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:58.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:58.879439+0000) 2022-09-10T09:54:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:54:59 smithi104 ceph-mon[97044]: pgmap v10423: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:54:59 smithi110 ceph-mon[97633]: pgmap v10423: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:54:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:54:59.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:59.397703+0000) 2022-09-10T09:55:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:54:59.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:54:59.879667+0000) 2022-09-10T09:55:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:00.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:00.397917+0000) 2022-09-10T09:55:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:00.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:00.879864+0000) 2022-09-10T09:55:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:01 smithi104 ceph-mon[97044]: pgmap v10424: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:01 smithi110 ceph-mon[97633]: pgmap v10424: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:01.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:01.398155+0000) 2022-09-10T09:55:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:01.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:01.880127+0000) 2022-09-10T09:55:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:02.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:02.398381+0000) 2022-09-10T09:55:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:02.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:02.880398+0000) 2022-09-10T09:55:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:03 smithi104 ceph-mon[97044]: pgmap v10425: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:03 smithi110 ceph-mon[97633]: pgmap v10425: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:03.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:03.398593+0000) 2022-09-10T09:55:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:03.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:03.880648+0000) 2022-09-10T09:55:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:04.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:04.398867+0000) 2022-09-10T09:55:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:04.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:04.880854+0000) 2022-09-10T09:55:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:05 smithi104 ceph-mon[97044]: pgmap v10426: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:05 smithi110 ceph-mon[97633]: pgmap v10426: 129 pgs: 129 active+clean; 8.8 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:05.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:05.399134+0000) 2022-09-10T09:55:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:05.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:05.881086+0000) 2022-09-10T09:55:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:06.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:06.399354+0000) 2022-09-10T09:55:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:06.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:06.881280+0000) 2022-09-10T09:55:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:07 smithi104 ceph-mon[97044]: pgmap v10427: 129 pgs: 129 active+clean; 8.8 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:55:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:07 smithi110 ceph-mon[97633]: pgmap v10427: 129 pgs: 129 active+clean; 8.8 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:55:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:07.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:07.399582+0000) 2022-09-10T09:55:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:55:07.914+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:55:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:07.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:07.881476+0000) 2022-09-10T09:55:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:08.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:08.399869+0000) 2022-09-10T09:55:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:08.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:08.881669+0000) 2022-09-10T09:55:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:09 smithi104 ceph-mon[97044]: pgmap v10428: 129 pgs: 129 active+clean; 8.8 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:55:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:09 smithi104 ceph-mon[97044]: osdmap e732: 12 total, 12 up, 12 in 2022-09-10T09:55:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:09 smithi110 ceph-mon[97633]: pgmap v10428: 129 pgs: 129 active+clean; 8.8 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:55:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:09 smithi110 ceph-mon[97633]: osdmap e732: 12 total, 12 up, 12 in 2022-09-10T09:55:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:09.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:09.400090+0000) 2022-09-10T09:55:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:09.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:09.881865+0000) 2022-09-10T09:55:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:10.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:10.400348+0000) 2022-09-10T09:55:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:10.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:10.882117+0000) 2022-09-10T09:55:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:11 smithi104 ceph-mon[97044]: pgmap v10430: 129 pgs: 129 active+clean; 8.8 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:55:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:11 smithi110 ceph-mon[97633]: pgmap v10430: 129 pgs: 129 active+clean; 8.8 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:55:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:11.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:11.400560+0000) 2022-09-10T09:55:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:11.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:11.882344+0000) 2022-09-10T09:55:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:12.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:12.400815+0000) 2022-09-10T09:55:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:12.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:12.882503+0000) 2022-09-10T09:55:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:13 smithi104 ceph-mon[97044]: pgmap v10431: 129 pgs: 129 active+clean; 8.8 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:55:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:13 smithi104 ceph-mon[97044]: osdmap e733: 12 total, 12 up, 12 in 2022-09-10T09:55:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:13 smithi110 ceph-mon[97633]: pgmap v10431: 129 pgs: 129 active+clean; 8.8 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:55:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:13 smithi110 ceph-mon[97633]: osdmap e733: 12 total, 12 up, 12 in 2022-09-10T09:55:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:13.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:13.401062+0000) 2022-09-10T09:55:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:13.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:13.882724+0000) 2022-09-10T09:55:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:14.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:14.401247+0000) 2022-09-10T09:55:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:14.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:14.882989+0000) 2022-09-10T09:55:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:15 smithi104 ceph-mon[97044]: pgmap v10433: 129 pgs: 129 active+clean; 8.8 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:55:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:15 smithi110 ceph-mon[97633]: pgmap v10433: 129 pgs: 129 active+clean; 8.8 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:55:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:15.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:15.401423+0000) 2022-09-10T09:55:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:15.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:15.883166+0000) 2022-09-10T09:55:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:16.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:16.401696+0000) 2022-09-10T09:55:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:16.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:16.883381+0000) 2022-09-10T09:55:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:17 smithi104 ceph-mon[97044]: pgmap v10434: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:55:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:17 smithi110 ceph-mon[97633]: pgmap v10434: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:55:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:17.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:17.401880+0000) 2022-09-10T09:55:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:17.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:17.883589+0000) 2022-09-10T09:55:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:18.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:18.402035+0000) 2022-09-10T09:55:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:18.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:18.883809+0000) 2022-09-10T09:55:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:19 smithi104 ceph-mon[97044]: pgmap v10435: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:55:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:19 smithi110 ceph-mon[97633]: pgmap v10435: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:55:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:19.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:19.402265+0000) 2022-09-10T09:55:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:19.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:19.884059+0000) 2022-09-10T09:55:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:20.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:20.402516+0000) 2022-09-10T09:55:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:20.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:20.884308+0000) 2022-09-10T09:55:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:21 smithi104 ceph-mon[97044]: pgmap v10436: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:55:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:21 smithi110 ceph-mon[97633]: pgmap v10436: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:55:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:21.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:21.402699+0000) 2022-09-10T09:55:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:21.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:21.884577+0000) 2022-09-10T09:55:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:22.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:22.402954+0000) 2022-09-10T09:55:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:22.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:22.884808+0000) 2022-09-10T09:55:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:55:22.923+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:55:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:23 smithi104 ceph-mon[97044]: pgmap v10437: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:55:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:23 smithi110 ceph-mon[97633]: pgmap v10437: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:55:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:23.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:23.403139+0000) 2022-09-10T09:55:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:23.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:23.885033+0000) 2022-09-10T09:55:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:24.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:24.403300+0000) 2022-09-10T09:55:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:24.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:24.885211+0000) 2022-09-10T09:55:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:25 smithi104 ceph-mon[97044]: pgmap v10438: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:55:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:25 smithi110 ceph-mon[97633]: pgmap v10438: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:55:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:25.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:25.403486+0000) 2022-09-10T09:55:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:25.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:25.885349+0000) 2022-09-10T09:55:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:26.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:26.403653+0000) 2022-09-10T09:55:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:26.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:26.885543+0000) 2022-09-10T09:55:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:27 smithi104 ceph-mon[97044]: pgmap v10439: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:55:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:27 smithi110 ceph-mon[97633]: pgmap v10439: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:55:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:27.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:27.403847+0000) 2022-09-10T09:55:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:27.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:27.885799+0000) 2022-09-10T09:55:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:28.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:28.404121+0000) 2022-09-10T09:55:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:28.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:28.886014+0000) 2022-09-10T09:55:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:29 smithi104 ceph-mon[97044]: pgmap v10440: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:29 smithi110 ceph-mon[97633]: pgmap v10440: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:29.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:29.404310+0000) 2022-09-10T09:55:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:29.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:29.886292+0000) 2022-09-10T09:55:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:30.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:30.404472+0000) 2022-09-10T09:55:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:30.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:30.886531+0000) 2022-09-10T09:55:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:31 smithi104 ceph-mon[97044]: pgmap v10441: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:31 smithi110 ceph-mon[97633]: pgmap v10441: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:31.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:31.404622+0000) 2022-09-10T09:55:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:31.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:31.886682+0000) 2022-09-10T09:55:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:32.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:32.404918+0000) 2022-09-10T09:55:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:32.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:32.886838+0000) 2022-09-10T09:55:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:33 smithi104 ceph-mon[97044]: pgmap v10442: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:55:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:33 smithi110 ceph-mon[97633]: pgmap v10442: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:55:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:33.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:33.405170+0000) 2022-09-10T09:55:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:33.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:33.887085+0000) 2022-09-10T09:55:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:34.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:34.405409+0000) 2022-09-10T09:55:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:34.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:34.887322+0000) 2022-09-10T09:55:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:35 smithi104 ceph-mon[97044]: pgmap v10443: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:55:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:35 smithi110 ceph-mon[97633]: pgmap v10443: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:55:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:35.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:35.405582+0000) 2022-09-10T09:55:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:35.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:35.887528+0000) 2022-09-10T09:55:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:36.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:36.405776+0000) 2022-09-10T09:55:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:36.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:36.887739+0000) 2022-09-10T09:55:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:37 smithi104 ceph-mon[97044]: pgmap v10444: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:37 smithi110 ceph-mon[97633]: pgmap v10444: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:37.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:37.405981+0000) 2022-09-10T09:55:38.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:55:37.938+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:55:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:37.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:37.887969+0000) 2022-09-10T09:55:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:38.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:38.406168+0000) 2022-09-10T09:55:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:38.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:38.888218+0000) 2022-09-10T09:55:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:39 smithi104 ceph-mon[97044]: pgmap v10445: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:55:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:39 smithi110 ceph-mon[97633]: pgmap v10445: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:55:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:39.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:39.406366+0000) 2022-09-10T09:55:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:39.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:39.888425+0000) 2022-09-10T09:55:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:40.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:40.406557+0000) 2022-09-10T09:55:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:40.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:40.888602+0000) 2022-09-10T09:55:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:41 smithi104 ceph-mon[97044]: pgmap v10446: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:41 smithi110 ceph-mon[97633]: pgmap v10446: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:41.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:41.406765+0000) 2022-09-10T09:55:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:41.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:41.888778+0000) 2022-09-10T09:55:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:42.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:42.407032+0000) 2022-09-10T09:55:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:42.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:42.889038+0000) 2022-09-10T09:55:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:43 smithi104 ceph-mon[97044]: pgmap v10447: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:43 smithi110 ceph-mon[97633]: pgmap v10447: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:43.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:43.407190+0000) 2022-09-10T09:55:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:43.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:43.889260+0000) 2022-09-10T09:55:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:44.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:44.407450+0000) 2022-09-10T09:55:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:44.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:44.889449+0000) 2022-09-10T09:55:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:45 smithi104 ceph-mon[97044]: pgmap v10448: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:55:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:55:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:55:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:55:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:45 smithi110 ceph-mon[97633]: pgmap v10448: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:55:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:55:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:55:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:55:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:45.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:45.407683+0000) 2022-09-10T09:55:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:45.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:45.889721+0000) 2022-09-10T09:55:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:46.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:46.407926+0000) 2022-09-10T09:55:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:46.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:46.889988+0000) 2022-09-10T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:47 smithi104 ceph-mon[97044]: pgmap v10449: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:47 smithi110 ceph-mon[97633]: pgmap v10449: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:47.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:47.408175+0000) 2022-09-10T09:55:48.323 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:47.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:47.890262+0000) 2022-09-10T09:55:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:48.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:48.408414+0000) 2022-09-10T09:55:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:48.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:48.890514+0000) 2022-09-10T09:55:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:49 smithi104 ceph-mon[97044]: pgmap v10450: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:49 smithi110 ceph-mon[97633]: pgmap v10450: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:49.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:49.408655+0000) 2022-09-10T09:55:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:49.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:49.890824+0000) 2022-09-10T09:55:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:50.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:50.408898+0000) 2022-09-10T09:55:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:50.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:50.890987+0000) 2022-09-10T09:55:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:51 smithi104 ceph-mon[97044]: pgmap v10451: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:51 smithi110 ceph-mon[97633]: pgmap v10451: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:51.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:51.409063+0000) 2022-09-10T09:55:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:51.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:51.891216+0000) 2022-09-10T09:55:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:52.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:52.409311+0000) 2022-09-10T09:55:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:52.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:52.891409+0000) 2022-09-10T09:55:53.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:55:52.950+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:55:53.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:53 smithi104 ceph-mon[97044]: pgmap v10452: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:53 smithi110 ceph-mon[97633]: pgmap v10452: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:53.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:53.409459+0000) 2022-09-10T09:55:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:53.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:53.891625+0000) 2022-09-10T09:55:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:54.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:54.409728+0000) 2022-09-10T09:55:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:54.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:54.891886+0000) 2022-09-10T09:55:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:55 smithi104 ceph-mon[97044]: pgmap v10453: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:55 smithi110 ceph-mon[97633]: pgmap v10453: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:55.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:55.409945+0000) 2022-09-10T09:55:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:55.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:55.892053+0000) 2022-09-10T09:55:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:56.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:56.410245+0000) 2022-09-10T09:55:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:56.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:56.892266+0000) 2022-09-10T09:55:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:57 smithi104 ceph-mon[97044]: pgmap v10454: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:57 smithi110 ceph-mon[97633]: pgmap v10454: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:55:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:57.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:57.410512+0000) 2022-09-10T09:55:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:57.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:57.892478+0000) 2022-09-10T09:55:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:58.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:58.410775+0000) 2022-09-10T09:55:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:58.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:58.892725+0000) 2022-09-10T09:55:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:55:59 smithi104 ceph-mon[97044]: pgmap v10455: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:55:59 smithi110 ceph-mon[97633]: pgmap v10455: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:55:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:55:59.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:59.410983+0000) 2022-09-10T09:56:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:55:59.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:55:59.892989+0000) 2022-09-10T09:56:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:00.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:00.411201+0000) 2022-09-10T09:56:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:00.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:00.893196+0000) 2022-09-10T09:56:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:01 smithi104 ceph-mon[97044]: pgmap v10456: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:01 smithi110 ceph-mon[97633]: pgmap v10456: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:01.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:01.411442+0000) 2022-09-10T09:56:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:01.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:01.893422+0000) 2022-09-10T09:56:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:02.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:02.411682+0000) 2022-09-10T09:56:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:02.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:02.893621+0000) 2022-09-10T09:56:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:03 smithi104 ceph-mon[97044]: pgmap v10457: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:03 smithi110 ceph-mon[97633]: pgmap v10457: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:03.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:03.411923+0000) 2022-09-10T09:56:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:03.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:03.893875+0000) 2022-09-10T09:56:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:04.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:04.412188+0000) 2022-09-10T09:56:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:04.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:04.894074+0000) 2022-09-10T09:56:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:05 smithi104 ceph-mon[97044]: pgmap v10458: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:05 smithi110 ceph-mon[97633]: pgmap v10458: 129 pgs: 129 active+clean; 8.8 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:05.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:05.412364+0000) 2022-09-10T09:56:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:05.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:05.894309+0000) 2022-09-10T09:56:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:06.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:06.412649+0000) 2022-09-10T09:56:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:06.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:06.894590+0000) 2022-09-10T09:56:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:07 smithi104 ceph-mon[97044]: pgmap v10459: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:56:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:07 smithi110 ceph-mon[97633]: pgmap v10459: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:56:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:07.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:07.412838+0000) 2022-09-10T09:56:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:56:07.951+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:56:08.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:07.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:07.894782+0000) 2022-09-10T09:56:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:08.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:08.413062+0000) 2022-09-10T09:56:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:08.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:08.895027+0000) 2022-09-10T09:56:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:09 smithi104 ceph-mon[97044]: pgmap v10460: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T09:56:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:09 smithi104 ceph-mon[97044]: osdmap e734: 12 total, 12 up, 12 in 2022-09-10T09:56:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:09 smithi110 ceph-mon[97633]: pgmap v10460: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T09:56:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:09 smithi110 ceph-mon[97633]: osdmap e734: 12 total, 12 up, 12 in 2022-09-10T09:56:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:09.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:09.413280+0000) 2022-09-10T09:56:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:09.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:09.895257+0000) 2022-09-10T09:56:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:10.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:10.413506+0000) 2022-09-10T09:56:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:10.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:10.895479+0000) 2022-09-10T09:56:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:11 smithi104 ceph-mon[97044]: pgmap v10462: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:56:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:11 smithi110 ceph-mon[97633]: pgmap v10462: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:56:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:11.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:11.413710+0000) 2022-09-10T09:56:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:11.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:11.895673+0000) 2022-09-10T09:56:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:12.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:12.414017+0000) 2022-09-10T09:56:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:12.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:12.895882+0000) 2022-09-10T09:56:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:13 smithi104 ceph-mon[97044]: pgmap v10463: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:56:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:13 smithi104 ceph-mon[97044]: osdmap e735: 12 total, 12 up, 12 in 2022-09-10T09:56:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:13 smithi110 ceph-mon[97633]: pgmap v10463: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:56:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:13 smithi110 ceph-mon[97633]: osdmap e735: 12 total, 12 up, 12 in 2022-09-10T09:56:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:13.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:13.414212+0000) 2022-09-10T09:56:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:13.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:13.896091+0000) 2022-09-10T09:56:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:14.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:14.414382+0000) 2022-09-10T09:56:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:14.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:14.896348+0000) 2022-09-10T09:56:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:15 smithi104 ceph-mon[97044]: pgmap v10465: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:15 smithi110 ceph-mon[97633]: pgmap v10465: 129 pgs: 129 active+clean; 8.8 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:56:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:15.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:15.414581+0000) 2022-09-10T09:56:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:15.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:15.896581+0000) 2022-09-10T09:56:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:16.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:16.414795+0000) 2022-09-10T09:56:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:16.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:16.896809+0000) 2022-09-10T09:56:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:17 smithi104 ceph-mon[97044]: pgmap v10466: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:56:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:17 smithi110 ceph-mon[97633]: pgmap v10466: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:56:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:17.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:17.415066+0000) 2022-09-10T09:56:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:17.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:17.897075+0000) 2022-09-10T09:56:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:18.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:18.415304+0000) 2022-09-10T09:56:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:18.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:18.897272+0000) 2022-09-10T09:56:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:19 smithi104 ceph-mon[97044]: pgmap v10467: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T09:56:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:19 smithi110 ceph-mon[97633]: pgmap v10467: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T09:56:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:19.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:19.415494+0000) 2022-09-10T09:56:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:19.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:19.897471+0000) 2022-09-10T09:56:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:20.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:20.415741+0000) 2022-09-10T09:56:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:20.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:20.897680+0000) 2022-09-10T09:56:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:21 smithi104 ceph-mon[97044]: pgmap v10468: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:56:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:21 smithi110 ceph-mon[97633]: pgmap v10468: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:56:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:21.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:21.415990+0000) 2022-09-10T09:56:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:21.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:21.897899+0000) 2022-09-10T09:56:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:22.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:22.416273+0000) 2022-09-10T09:56:23.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:56:22.955+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:56:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:22.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:22.898192+0000) 2022-09-10T09:56:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:23 smithi104 ceph-mon[97044]: pgmap v10469: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:56:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:23 smithi110 ceph-mon[97633]: pgmap v10469: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:56:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:23.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:23.416473+0000) 2022-09-10T09:56:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:23.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:23.898399+0000) 2022-09-10T09:56:24.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:24.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:24.416607+0000) 2022-09-10T09:56:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:24.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:24.898651+0000) 2022-09-10T09:56:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:25 smithi104 ceph-mon[97044]: pgmap v10470: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:56:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:25 smithi110 ceph-mon[97633]: pgmap v10470: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:56:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:25.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:25.416836+0000) 2022-09-10T09:56:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:25.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:25.898872+0000) 2022-09-10T09:56:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:26.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:26.417060+0000) 2022-09-10T09:56:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:26.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:26.899062+0000) 2022-09-10T09:56:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:27 smithi104 ceph-mon[97044]: pgmap v10471: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:56:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:27 smithi110 ceph-mon[97633]: pgmap v10471: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:56:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:27.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:27.417255+0000) 2022-09-10T09:56:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:27.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:27.899327+0000) 2022-09-10T09:56:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:28.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:28.417501+0000) 2022-09-10T09:56:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:28.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:28.899542+0000) 2022-09-10T09:56:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:29 smithi104 ceph-mon[97044]: pgmap v10472: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:29.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:29 smithi110 ceph-mon[97633]: pgmap v10472: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:29.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:29.417726+0000) 2022-09-10T09:56:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:29.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:29.899706+0000) 2022-09-10T09:56:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:30.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:30.417963+0000) 2022-09-10T09:56:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:30.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:30.899904+0000) 2022-09-10T09:56:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:31 smithi104 ceph-mon[97044]: pgmap v10473: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:31 smithi110 ceph-mon[97633]: pgmap v10473: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:31.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:31.418166+0000) 2022-09-10T09:56:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:31.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:31.900141+0000) 2022-09-10T09:56:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:32.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:32.418399+0000) 2022-09-10T09:56:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:32.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:32.900397+0000) 2022-09-10T09:56:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:33 smithi104 ceph-mon[97044]: pgmap v10474: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:33 smithi110 ceph-mon[97633]: pgmap v10474: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:33.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:33.418626+0000) 2022-09-10T09:56:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:33.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:33.900624+0000) 2022-09-10T09:56:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:34.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:34.418809+0000) 2022-09-10T09:56:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:34.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:34.900884+0000) 2022-09-10T09:56:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:35 smithi104 ceph-mon[97044]: pgmap v10475: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:35 smithi110 ceph-mon[97633]: pgmap v10475: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:35.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:35.419008+0000) 2022-09-10T09:56:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:35.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:35.901163+0000) 2022-09-10T09:56:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:36.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:36.419171+0000) 2022-09-10T09:56:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:36.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:36.901390+0000) 2022-09-10T09:56:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:37 smithi104 ceph-mon[97044]: pgmap v10476: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:37 smithi110 ceph-mon[97633]: pgmap v10476: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:37.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:37.419428+0000) 2022-09-10T09:56:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:37.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:37.901675+0000) 2022-09-10T09:56:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:56:37.960+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:56:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:38.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:38.419645+0000) 2022-09-10T09:56:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:38.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:38.901914+0000) 2022-09-10T09:56:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:39 smithi104 ceph-mon[97044]: pgmap v10477: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:39 smithi110 ceph-mon[97633]: pgmap v10477: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:39.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:39.419834+0000) 2022-09-10T09:56:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:39.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:39.902207+0000) 2022-09-10T09:56:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:40.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:40.420091+0000) 2022-09-10T09:56:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:40.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:40.902448+0000) 2022-09-10T09:56:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:41 smithi104 ceph-mon[97044]: pgmap v10478: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:41 smithi110 ceph-mon[97633]: pgmap v10478: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:41.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:41.420384+0000) 2022-09-10T09:56:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:41.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:41.902673+0000) 2022-09-10T09:56:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:42.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:42.420602+0000) 2022-09-10T09:56:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:42.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:42.902963+0000) 2022-09-10T09:56:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:43 smithi104 ceph-mon[97044]: pgmap v10479: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:43 smithi110 ceph-mon[97633]: pgmap v10479: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:43.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:43.420839+0000) 2022-09-10T09:56:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:43.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:43.903224+0000) 2022-09-10T09:56:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:44.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:44.421145+0000) 2022-09-10T09:56:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:44.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:44.903433+0000) 2022-09-10T09:56:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:45 smithi104 ceph-mon[97044]: pgmap v10480: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:56:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:56:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:56:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:56:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:45 smithi110 ceph-mon[97633]: pgmap v10480: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:56:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:56:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:56:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:56:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:45.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:45.421317+0000) 2022-09-10T09:56:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:45.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:45.903614+0000) 2022-09-10T09:56:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:46.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:46.421480+0000) 2022-09-10T09:56:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:46.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:46.903834+0000) 2022-09-10T09:56:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:47 smithi104 ceph-mon[97044]: pgmap v10481: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:47 smithi110 ceph-mon[97633]: pgmap v10481: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:47.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:47.421666+0000) 2022-09-10T09:56:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:47.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:47.904043+0000) 2022-09-10T09:56:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:48.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:48.421866+0000) 2022-09-10T09:56:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:48.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:48.904238+0000) 2022-09-10T09:56:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:49 smithi104 ceph-mon[97044]: pgmap v10482: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:49 smithi110 ceph-mon[97633]: pgmap v10482: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:49.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:49.422049+0000) 2022-09-10T09:56:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:49.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:49.904459+0000) 2022-09-10T09:56:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:50.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:50.422252+0000) 2022-09-10T09:56:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:50.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:50.904636+0000) 2022-09-10T09:56:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:51 smithi104 ceph-mon[97044]: pgmap v10483: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:51 smithi110 ceph-mon[97633]: pgmap v10483: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:51.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:51.422506+0000) 2022-09-10T09:56:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:51.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:51.904856+0000) 2022-09-10T09:56:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:52.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:52.422710+0000) 2022-09-10T09:56:53.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:56:52.972+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:56:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:52.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:52.905089+0000) 2022-09-10T09:56:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:53 smithi104 ceph-mon[97044]: pgmap v10484: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:53 smithi110 ceph-mon[97633]: pgmap v10484: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:53.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:53.422958+0000) 2022-09-10T09:56:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:53.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:53.905345+0000) 2022-09-10T09:56:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:54.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:54.423163+0000) 2022-09-10T09:56:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:54.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:54.905546+0000) 2022-09-10T09:56:55.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:55 smithi104 ceph-mon[97044]: pgmap v10485: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:55 smithi110 ceph-mon[97633]: pgmap v10485: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:55.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:55.423330+0000) 2022-09-10T09:56:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:55.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:55.905772+0000) 2022-09-10T09:56:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:56.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:56.423578+0000) 2022-09-10T09:56:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:56.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:56.905944+0000) 2022-09-10T09:56:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:57 smithi104 ceph-mon[97044]: pgmap v10486: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:57 smithi110 ceph-mon[97633]: pgmap v10486: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:56:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:57.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:57.423864+0000) 2022-09-10T09:56:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:57.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:57.906187+0000) 2022-09-10T09:56:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:58.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:58.424078+0000) 2022-09-10T09:56:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:58.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:58.906325+0000) 2022-09-10T09:56:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:56:59 smithi104 ceph-mon[97044]: pgmap v10487: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:56:59 smithi110 ceph-mon[97633]: pgmap v10487: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:56:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:56:59.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:59.424298+0000) 2022-09-10T09:57:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:56:59.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:56:59.906507+0000) 2022-09-10T09:57:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:00.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:00.424445+0000) 2022-09-10T09:57:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:00.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:00.906642+0000) 2022-09-10T09:57:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:01 smithi104 ceph-mon[97044]: pgmap v10488: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:01 smithi110 ceph-mon[97633]: pgmap v10488: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:01.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:01.424653+0000) 2022-09-10T09:57:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:01.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:01.906813+0000) 2022-09-10T09:57:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:02.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:02.424891+0000) 2022-09-10T09:57:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:02.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:02.907057+0000) 2022-09-10T09:57:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:03 smithi104 ceph-mon[97044]: pgmap v10489: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:03 smithi110 ceph-mon[97633]: pgmap v10489: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:03.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:03.425138+0000) 2022-09-10T09:57:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:03.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:03.907310+0000) 2022-09-10T09:57:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:04.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:04.425339+0000) 2022-09-10T09:57:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:04.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:04.907447+0000) 2022-09-10T09:57:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:05 smithi104 ceph-mon[97044]: pgmap v10490: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:05 smithi110 ceph-mon[97633]: pgmap v10490: 129 pgs: 129 active+clean; 8.8 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:05.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:05.425608+0000) 2022-09-10T09:57:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:05.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:05.907612+0000) 2022-09-10T09:57:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:06.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:06.425803+0000) 2022-09-10T09:57:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:06.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:06.907781+0000) 2022-09-10T09:57:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:07 smithi104 ceph-mon[97044]: pgmap v10491: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:57:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:07 smithi110 ceph-mon[97633]: pgmap v10491: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:57:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:07.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:07.426000+0000) 2022-09-10T09:57:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:57:07.982+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:57:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:07.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:07.907993+0000) 2022-09-10T09:57:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:08.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:08.426210+0000) 2022-09-10T09:57:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:08.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:08.908176+0000) 2022-09-10T09:57:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:09 smithi104 ceph-mon[97044]: pgmap v10492: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:57:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:09 smithi104 ceph-mon[97044]: osdmap e736: 12 total, 12 up, 12 in 2022-09-10T09:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:09 smithi110 ceph-mon[97633]: pgmap v10492: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:09 smithi110 ceph-mon[97633]: osdmap e736: 12 total, 12 up, 12 in 2022-09-10T09:57:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:09.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:09.426441+0000) 2022-09-10T09:57:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:09.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:09.908334+0000) 2022-09-10T09:57:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:10.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:10.426641+0000) 2022-09-10T09:57:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:10.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:10.908544+0000) 2022-09-10T09:57:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:11 smithi104 ceph-mon[97044]: pgmap v10494: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:57:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:11 smithi110 ceph-mon[97633]: pgmap v10494: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:57:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:11.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:11.426852+0000) 2022-09-10T09:57:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:11.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:11.908763+0000) 2022-09-10T09:57:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:12.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:12.427072+0000) 2022-09-10T09:57:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:12.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:12.909007+0000) 2022-09-10T09:57:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:13 smithi104 ceph-mon[97044]: pgmap v10495: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:57:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:13 smithi104 ceph-mon[97044]: osdmap e737: 12 total, 12 up, 12 in 2022-09-10T09:57:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:13 smithi110 ceph-mon[97633]: pgmap v10495: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:57:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:13 smithi110 ceph-mon[97633]: osdmap e737: 12 total, 12 up, 12 in 2022-09-10T09:57:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:13.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:13.427322+0000) 2022-09-10T09:57:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:13.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:13.909235+0000) 2022-09-10T09:57:14.431 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:14.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:14.427489+0000) 2022-09-10T09:57:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:14.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:14.909487+0000) 2022-09-10T09:57:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:15 smithi104 ceph-mon[97044]: pgmap v10497: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:57:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:15 smithi110 ceph-mon[97633]: pgmap v10497: 129 pgs: 129 active+clean; 8.8 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:57:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:15.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:15.427682+0000) 2022-09-10T09:57:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:15.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:15.909652+0000) 2022-09-10T09:57:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:16.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:16.427928+0000) 2022-09-10T09:57:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:16.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:16.909867+0000) 2022-09-10T09:57:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:17 smithi104 ceph-mon[97044]: pgmap v10498: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:57:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:17 smithi110 ceph-mon[97633]: pgmap v10498: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:57:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:17.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:17.428191+0000) 2022-09-10T09:57:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:17.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:17.910146+0000) 2022-09-10T09:57:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:18.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:18.428390+0000) 2022-09-10T09:57:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:18.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:18.910291+0000) 2022-09-10T09:57:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:19 smithi104 ceph-mon[97044]: pgmap v10499: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:57:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:19 smithi110 ceph-mon[97633]: pgmap v10499: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:57:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:19.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:19.428599+0000) 2022-09-10T09:57:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:19.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:19.910510+0000) 2022-09-10T09:57:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:20.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:20.428796+0000) 2022-09-10T09:57:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:20.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:20.910707+0000) 2022-09-10T09:57:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:21 smithi104 ceph-mon[97044]: pgmap v10500: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:57:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:21 smithi110 ceph-mon[97633]: pgmap v10500: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:57:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:21.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:21.429002+0000) 2022-09-10T09:57:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:21.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:21.910968+0000) 2022-09-10T09:57:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:22.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:22.429160+0000) 2022-09-10T09:57:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:22.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:22.911198+0000) 2022-09-10T09:57:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:57:22.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:57:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:23 smithi104 ceph-mon[97044]: pgmap v10501: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:57:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:23 smithi110 ceph-mon[97633]: pgmap v10501: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:57:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:23.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:23.429348+0000) 2022-09-10T09:57:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:23.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:23.911379+0000) 2022-09-10T09:57:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:24.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:24.429561+0000) 2022-09-10T09:57:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:24.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:24.911540+0000) 2022-09-10T09:57:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:25 smithi104 ceph-mon[97044]: pgmap v10502: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:57:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:25 smithi110 ceph-mon[97633]: pgmap v10502: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:57:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:25.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:25.429787+0000) 2022-09-10T09:57:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:25.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:25.911699+0000) 2022-09-10T09:57:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:26.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:26.430013+0000) 2022-09-10T09:57:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:26.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:26.911949+0000) 2022-09-10T09:57:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:27 smithi104 ceph-mon[97044]: pgmap v10503: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:57:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:27 smithi110 ceph-mon[97633]: pgmap v10503: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:57:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:27.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:27.430221+0000) 2022-09-10T09:57:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:27.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:27.912198+0000) 2022-09-10T09:57:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:28.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:28.430400+0000) 2022-09-10T09:57:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:28.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:28.912400+0000) 2022-09-10T09:57:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:29 smithi104 ceph-mon[97044]: pgmap v10504: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:29 smithi110 ceph-mon[97633]: pgmap v10504: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:29.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:29.430621+0000) 2022-09-10T09:57:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:29.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:29.912640+0000) 2022-09-10T09:57:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:30.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:30.430820+0000) 2022-09-10T09:57:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:30.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:30.912878+0000) 2022-09-10T09:57:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:31 smithi104 ceph-mon[97044]: pgmap v10505: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:31 smithi110 ceph-mon[97633]: pgmap v10505: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:31.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:31.431027+0000) 2022-09-10T09:57:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:31.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:31.913139+0000) 2022-09-10T09:57:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:32.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:32.431284+0000) 2022-09-10T09:57:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:32.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:32.913389+0000) 2022-09-10T09:57:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:33 smithi104 ceph-mon[97044]: pgmap v10506: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:33 smithi110 ceph-mon[97633]: pgmap v10506: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:33.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:33.431488+0000) 2022-09-10T09:57:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:33.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:33.913656+0000) 2022-09-10T09:57:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:34.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:34.431658+0000) 2022-09-10T09:57:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:34.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:34.913824+0000) 2022-09-10T09:57:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:35 smithi104 ceph-mon[97044]: pgmap v10507: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:35 smithi110 ceph-mon[97633]: pgmap v10507: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:35.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:35.431858+0000) 2022-09-10T09:57:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:35.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:35.914028+0000) 2022-09-10T09:57:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:36.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:36.432168+0000) 2022-09-10T09:57:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:36.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:36.914229+0000) 2022-09-10T09:57:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:37 smithi104 ceph-mon[97044]: pgmap v10508: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:37 smithi110 ceph-mon[97633]: pgmap v10508: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:37.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:37.432473+0000) 2022-09-10T09:57:38.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:57:37.995+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:57:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:37.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:37.914405+0000) 2022-09-10T09:57:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:38.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:38.432692+0000) 2022-09-10T09:57:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:38.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:38.914593+0000) 2022-09-10T09:57:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:39 smithi104 ceph-mon[97044]: pgmap v10509: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:39 smithi110 ceph-mon[97633]: pgmap v10509: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:39.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:39.432941+0000) 2022-09-10T09:57:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:39.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:39.914773+0000) 2022-09-10T09:57:40.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:40.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:40.433208+0000) 2022-09-10T09:57:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:40.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:40.914934+0000) 2022-09-10T09:57:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:41 smithi104 ceph-mon[97044]: pgmap v10510: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:41 smithi110 ceph-mon[97633]: pgmap v10510: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:41.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:41.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:41.433444+0000) 2022-09-10T09:57:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:41.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:41.915105+0000) 2022-09-10T09:57:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:42.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:42.433600+0000) 2022-09-10T09:57:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:42.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:42.915294+0000) 2022-09-10T09:57:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:43 smithi104 ceph-mon[97044]: pgmap v10511: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:43 smithi110 ceph-mon[97633]: pgmap v10511: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:43.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:43.433798+0000) 2022-09-10T09:57:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:43.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:43.915496+0000) 2022-09-10T09:57:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:44.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:44.434064+0000) 2022-09-10T09:57:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:44.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:44.915633+0000) 2022-09-10T09:57:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:45 smithi104 ceph-mon[97044]: pgmap v10512: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:57:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:57:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:57:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:57:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:45 smithi110 ceph-mon[97633]: pgmap v10512: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:57:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:57:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:57:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:57:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:45.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:45.434271+0000) 2022-09-10T09:57:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:45.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:45.915886+0000) 2022-09-10T09:57:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:46.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:46.434530+0000) 2022-09-10T09:57:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:46.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:46.916138+0000) 2022-09-10T09:57:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:47 smithi104 ceph-mon[97044]: pgmap v10513: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:47 smithi110 ceph-mon[97633]: pgmap v10513: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:47.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:47.434762+0000) 2022-09-10T09:57:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:47.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:47.916369+0000) 2022-09-10T09:57:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:48.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:48.435014+0000) 2022-09-10T09:57:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:48.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:48.916523+0000) 2022-09-10T09:57:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:49 smithi104 ceph-mon[97044]: pgmap v10514: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:49 smithi110 ceph-mon[97633]: pgmap v10514: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:49.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:49.435239+0000) 2022-09-10T09:57:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:49.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:49.916740+0000) 2022-09-10T09:57:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:50.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:50.435507+0000) 2022-09-10T09:57:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:50.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:50.916929+0000) 2022-09-10T09:57:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:51 smithi104 ceph-mon[97044]: pgmap v10515: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:51 smithi110 ceph-mon[97633]: pgmap v10515: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:51.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:51.435734+0000) 2022-09-10T09:57:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:51.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:51.917204+0000) 2022-09-10T09:57:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:52.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:52.436016+0000) 2022-09-10T09:57:53.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:57:53.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:57:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:52.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:52.917369+0000) 2022-09-10T09:57:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:53 smithi104 ceph-mon[97044]: pgmap v10516: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:53 smithi110 ceph-mon[97633]: pgmap v10516: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:53.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:53.436190+0000) 2022-09-10T09:57:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:53.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:53.917600+0000) 2022-09-10T09:57:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:54.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:54.436360+0000) 2022-09-10T09:57:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:54.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:54.917844+0000) 2022-09-10T09:57:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:55 smithi104 ceph-mon[97044]: pgmap v10517: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:55 smithi110 ceph-mon[97633]: pgmap v10517: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:55.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:55.436535+0000) 2022-09-10T09:57:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:55.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:55.918073+0000) 2022-09-10T09:57:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:56.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:56.436762+0000) 2022-09-10T09:57:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:56.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:56.918275+0000) 2022-09-10T09:57:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:57 smithi104 ceph-mon[97044]: pgmap v10518: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:57 smithi110 ceph-mon[97633]: pgmap v10518: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:57:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:57.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:57.436954+0000) 2022-09-10T09:57:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:57.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:57.918510+0000) 2022-09-10T09:57:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:58.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:58.437195+0000) 2022-09-10T09:57:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:58.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:58.918706+0000) 2022-09-10T09:57:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:57:59 smithi104 ceph-mon[97044]: pgmap v10519: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:57:59 smithi110 ceph-mon[97633]: pgmap v10519: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:57:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:57:59.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:59.437434+0000) 2022-09-10T09:58:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:57:59.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:57:59.918933+0000) 2022-09-10T09:58:00.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:00.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:00.437668+0000) 2022-09-10T09:58:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:00.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:00.919076+0000) 2022-09-10T09:58:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:01 smithi104 ceph-mon[97044]: pgmap v10520: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:01 smithi110 ceph-mon[97633]: pgmap v10520: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:01.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:01.437848+0000) 2022-09-10T09:58:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:01.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:01.919305+0000) 2022-09-10T09:58:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:02.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:02.438034+0000) 2022-09-10T09:58:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:02.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:02.919548+0000) 2022-09-10T09:58:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:03 smithi104 ceph-mon[97044]: pgmap v10521: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:03 smithi110 ceph-mon[97633]: pgmap v10521: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:03.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:03.438242+0000) 2022-09-10T09:58:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:03.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:03.919728+0000) 2022-09-10T09:58:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:04.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:04.438483+0000) 2022-09-10T09:58:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:04.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:04.919946+0000) 2022-09-10T09:58:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:05 smithi104 ceph-mon[97044]: pgmap v10522: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:05 smithi110 ceph-mon[97633]: pgmap v10522: 129 pgs: 129 active+clean; 8.8 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:05.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:05.438723+0000) 2022-09-10T09:58:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:05.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:05.920217+0000) 2022-09-10T09:58:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:06.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:06.438905+0000) 2022-09-10T09:58:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:06.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:06.920493+0000) 2022-09-10T09:58:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:07 smithi104 ceph-mon[97044]: pgmap v10523: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:58:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:07 smithi110 ceph-mon[97633]: pgmap v10523: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:58:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:07.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:07.439171+0000) 2022-09-10T09:58:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:07.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:07.920680+0000) 2022-09-10T09:58:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:58:08.021+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:58:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:08.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:08.439348+0000) 2022-09-10T09:58:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:08.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:08.920879+0000) 2022-09-10T09:58:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:09 smithi104 ceph-mon[97044]: pgmap v10524: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:58:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:09 smithi104 ceph-mon[97044]: osdmap e738: 12 total, 12 up, 12 in 2022-09-10T09:58:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:09 smithi110 ceph-mon[97633]: pgmap v10524: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:58:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:09 smithi110 ceph-mon[97633]: osdmap e738: 12 total, 12 up, 12 in 2022-09-10T09:58:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:09.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:09.439579+0000) 2022-09-10T09:58:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:09.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:09.921126+0000) 2022-09-10T09:58:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:10.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:10.439813+0000) 2022-09-10T09:58:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:10.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:10.921339+0000) 2022-09-10T09:58:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:11 smithi104 ceph-mon[97044]: pgmap v10526: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:58:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:11 smithi110 ceph-mon[97633]: pgmap v10526: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:58:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:11.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:11.439984+0000) 2022-09-10T09:58:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:11.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:11.921514+0000) 2022-09-10T09:58:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:12.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:12.440222+0000) 2022-09-10T09:58:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:12.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:12.921647+0000) 2022-09-10T09:58:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:13 smithi104 ceph-mon[97044]: pgmap v10527: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:58:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:13 smithi104 ceph-mon[97044]: osdmap e739: 12 total, 12 up, 12 in 2022-09-10T09:58:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:13 smithi110 ceph-mon[97633]: pgmap v10527: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:58:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:13 smithi110 ceph-mon[97633]: osdmap e739: 12 total, 12 up, 12 in 2022-09-10T09:58:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:13.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:13.440444+0000) 2022-09-10T09:58:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:13.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:13.921880+0000) 2022-09-10T09:58:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:14.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:14.440687+0000) 2022-09-10T09:58:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:14.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:14.922054+0000) 2022-09-10T09:58:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:15 smithi104 ceph-mon[97044]: pgmap v10529: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:58:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:15 smithi110 ceph-mon[97633]: pgmap v10529: 129 pgs: 129 active+clean; 8.8 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:58:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:15.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:15.440880+0000) 2022-09-10T09:58:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:15.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:15.922256+0000) 2022-09-10T09:58:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:16.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:16.441122+0000) 2022-09-10T09:58:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:16.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:16.922429+0000) 2022-09-10T09:58:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:17 smithi104 ceph-mon[97044]: pgmap v10530: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:58:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:17 smithi110 ceph-mon[97633]: pgmap v10530: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:58:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:17.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:17.441361+0000) 2022-09-10T09:58:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:17.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:17.922654+0000) 2022-09-10T09:58:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:18.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:18.441605+0000) 2022-09-10T09:58:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:18.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:18.922875+0000) 2022-09-10T09:58:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:19 smithi104 ceph-mon[97044]: pgmap v10531: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T09:58:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:19 smithi110 ceph-mon[97633]: pgmap v10531: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T09:58:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:19.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:19.441757+0000) 2022-09-10T09:58:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:19.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:19.923078+0000) 2022-09-10T09:58:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:20.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:20.442047+0000) 2022-09-10T09:58:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:20.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:20.923262+0000) 2022-09-10T09:58:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:21 smithi104 ceph-mon[97044]: pgmap v10532: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:58:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:21 smithi110 ceph-mon[97633]: pgmap v10532: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:58:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:21.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:21.442224+0000) 2022-09-10T09:58:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:21.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:21.923470+0000) 2022-09-10T09:58:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:22.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:22.442464+0000) 2022-09-10T09:58:23.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:58:23.033+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:58:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:22.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:22.923709+0000) 2022-09-10T09:58:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:23 smithi104 ceph-mon[97044]: pgmap v10533: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:58:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:23 smithi110 ceph-mon[97633]: pgmap v10533: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:58:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:23.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:23.442647+0000) 2022-09-10T09:58:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:23.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:23.923936+0000) 2022-09-10T09:58:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:24.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:24.442823+0000) 2022-09-10T09:58:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:24.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:24.924178+0000) 2022-09-10T09:58:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:25 smithi104 ceph-mon[97044]: pgmap v10534: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:58:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:25 smithi110 ceph-mon[97633]: pgmap v10534: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:58:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:25.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:25.443054+0000) 2022-09-10T09:58:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:25.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:25.924359+0000) 2022-09-10T09:58:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:26.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:26.443160+0000) 2022-09-10T09:58:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:26.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:26.924630+0000) 2022-09-10T09:58:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:27 smithi104 ceph-mon[97044]: pgmap v10535: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:58:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:27 smithi110 ceph-mon[97633]: pgmap v10535: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:58:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:27.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:27.443322+0000) 2022-09-10T09:58:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:27.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:27.924847+0000) 2022-09-10T09:58:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:28.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:28.443533+0000) 2022-09-10T09:58:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:28.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:28.925046+0000) 2022-09-10T09:58:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:29 smithi104 ceph-mon[97044]: pgmap v10536: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:29 smithi110 ceph-mon[97633]: pgmap v10536: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:29.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:29.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:29.443787+0000) 2022-09-10T09:58:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:29.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:29.925311+0000) 2022-09-10T09:58:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:30.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:30.443970+0000) 2022-09-10T09:58:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:30.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:30.925465+0000) 2022-09-10T09:58:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:31 smithi104 ceph-mon[97044]: pgmap v10537: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:31 smithi110 ceph-mon[97633]: pgmap v10537: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:31.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:31.444212+0000) 2022-09-10T09:58:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:31.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:31.925696+0000) 2022-09-10T09:58:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:32.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:32.444438+0000) 2022-09-10T09:58:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:32.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:32.925890+0000) 2022-09-10T09:58:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:33 smithi104 ceph-mon[97044]: pgmap v10538: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:33 smithi110 ceph-mon[97633]: pgmap v10538: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:33.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:33.444696+0000) 2022-09-10T09:58:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:33.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:33.926090+0000) 2022-09-10T09:58:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:34.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:34.444897+0000) 2022-09-10T09:58:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:34.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:34.926341+0000) 2022-09-10T09:58:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:35 smithi104 ceph-mon[97044]: pgmap v10539: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:35 smithi110 ceph-mon[97633]: pgmap v10539: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:35.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:35.445158+0000) 2022-09-10T09:58:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:35.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:35.926552+0000) 2022-09-10T09:58:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:36.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:36.445421+0000) 2022-09-10T09:58:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:36.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:36.926767+0000) 2022-09-10T09:58:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:37 smithi104 ceph-mon[97044]: pgmap v10540: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:37 smithi110 ceph-mon[97633]: pgmap v10540: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:37.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:37.445675+0000) 2022-09-10T09:58:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:37.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:37.926918+0000) 2022-09-10T09:58:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:58:38.048+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:58:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:38.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:38.445903+0000) 2022-09-10T09:58:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:38.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:38.927187+0000) 2022-09-10T09:58:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:39 smithi104 ceph-mon[97044]: pgmap v10541: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:39 smithi110 ceph-mon[97633]: pgmap v10541: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:39.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:39.446139+0000) 2022-09-10T09:58:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:39.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:39.927421+0000) 2022-09-10T09:58:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:40.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:40.446331+0000) 2022-09-10T09:58:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:40.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:40.927586+0000) 2022-09-10T09:58:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:41 smithi104 ceph-mon[97044]: pgmap v10542: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:41 smithi110 ceph-mon[97633]: pgmap v10542: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:41.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:41.446562+0000) 2022-09-10T09:58:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:41.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:41.927865+0000) 2022-09-10T09:58:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:42.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:42.446778+0000) 2022-09-10T09:58:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:42.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:42.928087+0000) 2022-09-10T09:58:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:43 smithi104 ceph-mon[97044]: pgmap v10543: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:43 smithi110 ceph-mon[97633]: pgmap v10543: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:43.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:43.447014+0000) 2022-09-10T09:58:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:43.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:43.928340+0000) 2022-09-10T09:58:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:44.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:44.447286+0000) 2022-09-10T09:58:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:44.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:44.928545+0000) 2022-09-10T09:58:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:45 smithi104 ceph-mon[97044]: pgmap v10544: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:58:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:58:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:58:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:58:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:45 smithi110 ceph-mon[97633]: pgmap v10544: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:58:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:58:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:58:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:58:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:45.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:45.447451+0000) 2022-09-10T09:58:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:45.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:45.928775+0000) 2022-09-10T09:58:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:46.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:46.447643+0000) 2022-09-10T09:58:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:46.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:46.929012+0000) 2022-09-10T09:58:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:47 smithi104 ceph-mon[97044]: pgmap v10545: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:47 smithi110 ceph-mon[97633]: pgmap v10545: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:47.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:47.447824+0000) 2022-09-10T09:58:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:47.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:47.929206+0000) 2022-09-10T09:58:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:48.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:48.448068+0000) 2022-09-10T09:58:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:48.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:48.929441+0000) 2022-09-10T09:58:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:49 smithi104 ceph-mon[97044]: pgmap v10546: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:58:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:49 smithi110 ceph-mon[97633]: pgmap v10546: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:58:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:49.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:49.448290+0000) 2022-09-10T09:58:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:49.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:49.929695+0000) 2022-09-10T09:58:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:50.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:50.448428+0000) 2022-09-10T09:58:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:50.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:50.929955+0000) 2022-09-10T09:58:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:51 smithi104 ceph-mon[97044]: pgmap v10547: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:51 smithi110 ceph-mon[97633]: pgmap v10547: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:51.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:51.448659+0000) 2022-09-10T09:58:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:51.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:51.930202+0000) 2022-09-10T09:58:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:52.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:52.448854+0000) 2022-09-10T09:58:53.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:58:53.057+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:58:53.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:52.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:52.930420+0000) 2022-09-10T09:58:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:53 smithi104 ceph-mon[97044]: pgmap v10548: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:58:53.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:53 smithi110 ceph-mon[97633]: pgmap v10548: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:58:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:53.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:53.449134+0000) 2022-09-10T09:58:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:53.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:53.930635+0000) 2022-09-10T09:58:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:54.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:54.449367+0000) 2022-09-10T09:58:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:54.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:54.930866+0000) 2022-09-10T09:58:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:55 smithi104 ceph-mon[97044]: pgmap v10549: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:58:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:55 smithi110 ceph-mon[97633]: pgmap v10549: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T09:58:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:55.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:55.449610+0000) 2022-09-10T09:58:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:55.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:55.931088+0000) 2022-09-10T09:58:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:56.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:56.449793+0000) 2022-09-10T09:58:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:56.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:56.931329+0000) 2022-09-10T09:58:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:57 smithi104 ceph-mon[97044]: pgmap v10550: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:57 smithi110 ceph-mon[97633]: pgmap v10550: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:58:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:57.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:57.449966+0000) 2022-09-10T09:58:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:57.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:57.931601+0000) 2022-09-10T09:58:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:58.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:58.450132+0000) 2022-09-10T09:58:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:58.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:58.931730+0000) 2022-09-10T09:58:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:58:59 smithi104 ceph-mon[97044]: pgmap v10551: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:58:59 smithi110 ceph-mon[97633]: pgmap v10551: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:58:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:58:59.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:59.450359+0000) 2022-09-10T09:59:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:58:59.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:58:59.931961+0000) 2022-09-10T09:59:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:00.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:00.450535+0000) 2022-09-10T09:59:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:00.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:00.932203+0000) 2022-09-10T09:59:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:01 smithi104 ceph-mon[97044]: pgmap v10552: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:01 smithi110 ceph-mon[97633]: pgmap v10552: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:01.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:01.450709+0000) 2022-09-10T09:59:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:01.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:01.932461+0000) 2022-09-10T09:59:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:02.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:02.450894+0000) 2022-09-10T09:59:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:02.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:02.932721+0000) 2022-09-10T09:59:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:03 smithi104 ceph-mon[97044]: pgmap v10553: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:03 smithi110 ceph-mon[97633]: pgmap v10553: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:03.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:03.451073+0000) 2022-09-10T09:59:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:03.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:03.932953+0000) 2022-09-10T09:59:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:04.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:04.451309+0000) 2022-09-10T09:59:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:04.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:04.933192+0000) 2022-09-10T09:59:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:05 smithi110 ceph-mon[97633]: pgmap v10554: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:05 smithi104 ceph-mon[97044]: pgmap v10554: 129 pgs: 129 active+clean; 8.8 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:05.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:05.451567+0000) 2022-09-10T09:59:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:05.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:05.933369+0000) 2022-09-10T09:59:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:06.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:06.451774+0000) 2022-09-10T09:59:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:06.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:06.933596+0000) 2022-09-10T09:59:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:07 smithi110 ceph-mon[97633]: pgmap v10555: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:59:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:07 smithi104 ceph-mon[97044]: pgmap v10555: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T09:59:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:07.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:07.452017+0000) 2022-09-10T09:59:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:07.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:07.933861+0000) 2022-09-10T09:59:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:59:08.071+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:59:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:08.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:08.452346+0000) 2022-09-10T09:59:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:08.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:08.934146+0000) 2022-09-10T09:59:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:09 smithi110 ceph-mon[97633]: pgmap v10556: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:59:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:09 smithi110 ceph-mon[97633]: osdmap e740: 12 total, 12 up, 12 in 2022-09-10T09:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:09 smithi104 ceph-mon[97044]: pgmap v10556: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T09:59:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:09 smithi104 ceph-mon[97044]: osdmap e740: 12 total, 12 up, 12 in 2022-09-10T09:59:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:09.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:09.452585+0000) 2022-09-10T09:59:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:09.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:09.934396+0000) 2022-09-10T09:59:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:10.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:10.452815+0000) 2022-09-10T09:59:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:10.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:10.934600+0000) 2022-09-10T09:59:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:11 smithi110 ceph-mon[97633]: pgmap v10558: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:59:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:11 smithi104 ceph-mon[97044]: pgmap v10558: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:59:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:11.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:11.453067+0000) 2022-09-10T09:59:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:11.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:11.934738+0000) 2022-09-10T09:59:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:12.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:12.453285+0000) 2022-09-10T09:59:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:12.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:12.934979+0000) 2022-09-10T09:59:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:13 smithi104 ceph-mon[97044]: pgmap v10559: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:59:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:13 smithi104 ceph-mon[97044]: osdmap e741: 12 total, 12 up, 12 in 2022-09-10T09:59:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:13 smithi110 ceph-mon[97633]: pgmap v10559: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T09:59:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:13 smithi110 ceph-mon[97633]: osdmap e741: 12 total, 12 up, 12 in 2022-09-10T09:59:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:13.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:13.453527+0000) 2022-09-10T09:59:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:13.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:13.935230+0000) 2022-09-10T09:59:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:14.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:14.453694+0000) 2022-09-10T09:59:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:14.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:14.935490+0000) 2022-09-10T09:59:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:15 smithi104 ceph-mon[97044]: pgmap v10561: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:59:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:15 smithi110 ceph-mon[97633]: pgmap v10561: 129 pgs: 129 active+clean; 8.8 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T09:59:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:15.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:15.453895+0000) 2022-09-10T09:59:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:15.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:15.935710+0000) 2022-09-10T09:59:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:16.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:16.454194+0000) 2022-09-10T09:59:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:16.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:16.935968+0000) 2022-09-10T09:59:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:17 smithi104 ceph-mon[97044]: pgmap v10562: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:59:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:17 smithi110 ceph-mon[97633]: pgmap v10562: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T09:59:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:17.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:17.454459+0000) 2022-09-10T09:59:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:17.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:17.936224+0000) 2022-09-10T09:59:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:18.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:18.454657+0000) 2022-09-10T09:59:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:18.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:18.936428+0000) 2022-09-10T09:59:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:19 smithi104 ceph-mon[97044]: pgmap v10563: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:59:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:19 smithi110 ceph-mon[97633]: pgmap v10563: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T09:59:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:19.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:19.454901+0000) 2022-09-10T09:59:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:19.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:19.936669+0000) 2022-09-10T09:59:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:20.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:20.455141+0000) 2022-09-10T09:59:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:20.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:20.936924+0000) 2022-09-10T09:59:21.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:21 smithi104 ceph-mon[97044]: pgmap v10564: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:59:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:21 smithi110 ceph-mon[97633]: pgmap v10564: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:59:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:21.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:21.455397+0000) 2022-09-10T09:59:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:21.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:21.937168+0000) 2022-09-10T09:59:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:22.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:22.455569+0000) 2022-09-10T09:59:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:22.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:22.937404+0000) 2022-09-10T09:59:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:59:23.087+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:59:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:23 smithi104 ceph-mon[97044]: pgmap v10565: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:59:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:23 smithi110 ceph-mon[97633]: pgmap v10565: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T09:59:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:23.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:23.455750+0000) 2022-09-10T09:59:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:23.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:23.937599+0000) 2022-09-10T09:59:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:24.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:24.455966+0000) 2022-09-10T09:59:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:24.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:24.937841+0000) 2022-09-10T09:59:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:25 smithi104 ceph-mon[97044]: pgmap v10566: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:59:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:25 smithi110 ceph-mon[97633]: pgmap v10566: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T09:59:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:25.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:25.456203+0000) 2022-09-10T09:59:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:25.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:25.938033+0000) 2022-09-10T09:59:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:26.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:26.456366+0000) 2022-09-10T09:59:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:26.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:26.938236+0000) 2022-09-10T09:59:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:27 smithi104 ceph-mon[97044]: pgmap v10567: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:59:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:27 smithi110 ceph-mon[97633]: pgmap v10567: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T09:59:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:27.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:27.456612+0000) 2022-09-10T09:59:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:27.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:27.938430+0000) 2022-09-10T09:59:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:28.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:28.456928+0000) 2022-09-10T09:59:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:28.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:28.938632+0000) 2022-09-10T09:59:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:29 smithi104 ceph-mon[97044]: pgmap v10568: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:29 smithi110 ceph-mon[97633]: pgmap v10568: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:29.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:29.457198+0000) 2022-09-10T09:59:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:29.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:29.938896+0000) 2022-09-10T09:59:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:30.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:30.457357+0000) 2022-09-10T09:59:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:30.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:30.939170+0000) 2022-09-10T09:59:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:31 smithi104 ceph-mon[97044]: pgmap v10569: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:31 smithi110 ceph-mon[97633]: pgmap v10569: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:31.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:31.457595+0000) 2022-09-10T09:59:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:31.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:31.939343+0000) 2022-09-10T09:59:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:32.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:32.457786+0000) 2022-09-10T09:59:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:32.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:32.939553+0000) 2022-09-10T09:59:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:33 smithi104 ceph-mon[97044]: pgmap v10570: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:33 smithi110 ceph-mon[97633]: pgmap v10570: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:33.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:33.458033+0000) 2022-09-10T09:59:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:33.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:33.939800+0000) 2022-09-10T09:59:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:34.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:34.458290+0000) 2022-09-10T09:59:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:34.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:34.940058+0000) 2022-09-10T09:59:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:35 smithi104 ceph-mon[97044]: pgmap v10571: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:35 smithi110 ceph-mon[97633]: pgmap v10571: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:35.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:35.458497+0000) 2022-09-10T09:59:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:35.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:35.940242+0000) 2022-09-10T09:59:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:36.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:36.458814+0000) 2022-09-10T09:59:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:36.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:36.940448+0000) 2022-09-10T09:59:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:37 smithi104 ceph-mon[97044]: pgmap v10572: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:37 smithi110 ceph-mon[97633]: pgmap v10572: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:37.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:37.459130+0000) 2022-09-10T09:59:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:37.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:37.940643+0000) 2022-09-10T09:59:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:59:38.103+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:59:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:38.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:38.459362+0000) 2022-09-10T09:59:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:38.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:38.940818+0000) 2022-09-10T09:59:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:39 smithi104 ceph-mon[97044]: pgmap v10573: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:39 smithi110 ceph-mon[97633]: pgmap v10573: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:39.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:39.459559+0000) 2022-09-10T09:59:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:39.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:39.941032+0000) 2022-09-10T09:59:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:40.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:40.459733+0000) 2022-09-10T09:59:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:40.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:40.941262+0000) 2022-09-10T09:59:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:41 smithi104 ceph-mon[97044]: pgmap v10574: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:41 smithi110 ceph-mon[97633]: pgmap v10574: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:41.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:41.459967+0000) 2022-09-10T09:59:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:41.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:41.941492+0000) 2022-09-10T09:59:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:42.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:42.460169+0000) 2022-09-10T09:59:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:42.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:42.941770+0000) 2022-09-10T09:59:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:43 smithi104 ceph-mon[97044]: pgmap v10575: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:43 smithi110 ceph-mon[97633]: pgmap v10575: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:43.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:43.460300+0000) 2022-09-10T09:59:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:43.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:43.941988+0000) 2022-09-10T09:59:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:44.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:44.460476+0000) 2022-09-10T09:59:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:44.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:44.942196+0000) 2022-09-10T09:59:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:45 smithi104 ceph-mon[97044]: pgmap v10576: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:59:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:59:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:59:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:59:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:45 smithi110 ceph-mon[97633]: pgmap v10576: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:59:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T09:59:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:59:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T09:59:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:45.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:45.460719+0000) 2022-09-10T09:59:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:45.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:45.942440+0000) 2022-09-10T09:59:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:46.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:46.460932+0000) 2022-09-10T09:59:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:46.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:46.942675+0000) 2022-09-10T09:59:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:47 smithi104 ceph-mon[97044]: pgmap v10577: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:47 smithi110 ceph-mon[97633]: pgmap v10577: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:47.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:47.461204+0000) 2022-09-10T09:59:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:47.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:47.942914+0000) 2022-09-10T09:59:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:48.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:48.461457+0000) 2022-09-10T09:59:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:48.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:48.943186+0000) 2022-09-10T09:59:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:49 smithi104 ceph-mon[97044]: pgmap v10578: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:49 smithi110 ceph-mon[97633]: pgmap v10578: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:49.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:49.461594+0000) 2022-09-10T09:59:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:49.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:49.943392+0000) 2022-09-10T09:59:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:50.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:50.461819+0000) 2022-09-10T09:59:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:50.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:50.943614+0000) 2022-09-10T09:59:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:51 smithi104 ceph-mon[97044]: pgmap v10579: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:51 smithi110 ceph-mon[97633]: pgmap v10579: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:51.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:51.462046+0000) 2022-09-10T09:59:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:51.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:51.943780+0000) 2022-09-10T09:59:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:52.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:52.462259+0000) 2022-09-10T09:59:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:52.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:52.944033+0000) 2022-09-10T09:59:53.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 09:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T09:59:53.109+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T09:59:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:53 smithi104 ceph-mon[97044]: pgmap v10580: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:53 smithi110 ceph-mon[97633]: pgmap v10580: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:53.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:53.462495+0000) 2022-09-10T09:59:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:53.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:53.944209+0000) 2022-09-10T09:59:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:54.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:54.462730+0000) 2022-09-10T09:59:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:54.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:54.944430+0000) 2022-09-10T09:59:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:55 smithi104 ceph-mon[97044]: pgmap v10581: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:55 smithi110 ceph-mon[97633]: pgmap v10581: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:55.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:55.462934+0000) 2022-09-10T09:59:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:55.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:55.944683+0000) 2022-09-10T09:59:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:56.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:56.463164+0000) 2022-09-10T09:59:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:56.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:56.944934+0000) 2022-09-10T09:59:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:57 smithi110 ceph-mon[97633]: pgmap v10582: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:57 smithi104 ceph-mon[97044]: pgmap v10582: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T09:59:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:57.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:57.463389+0000) 2022-09-10T09:59:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:57.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:57.945171+0000) 2022-09-10T09:59:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:58.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:58.463657+0000) 2022-09-10T09:59:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:58.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:58.945355+0000) 2022-09-10T09:59:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 09:59:59 smithi110 ceph-mon[97633]: pgmap v10583: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 09:59:59 smithi104 ceph-mon[97044]: pgmap v10583: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T09:59:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 09:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T09:59:59.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:59.463879+0000) 2022-09-10T10:00:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 09:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T09:59:59.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T08:59:59.945539+0000) 2022-09-10T10:00:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:00:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:00:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:00.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:00.464127+0000) 2022-09-10T10:00:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:00.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:00.945706+0000) 2022-09-10T10:00:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:01 smithi110 ceph-mon[97633]: pgmap v10584: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:01 smithi104 ceph-mon[97044]: pgmap v10584: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:01.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:01.464285+0000) 2022-09-10T10:00:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:01.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:01.945921+0000) 2022-09-10T10:00:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:02.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:02.464469+0000) 2022-09-10T10:00:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:02.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:02.946210+0000) 2022-09-10T10:00:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:03 smithi110 ceph-mon[97633]: pgmap v10585: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:03.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:03 smithi104 ceph-mon[97044]: pgmap v10585: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:03.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:03.464655+0000) 2022-09-10T10:00:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:03.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:03.946446+0000) 2022-09-10T10:00:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:04.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:04.464830+0000) 2022-09-10T10:00:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:04.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:04.946657+0000) 2022-09-10T10:00:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:05 smithi110 ceph-mon[97633]: pgmap v10586: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:05 smithi104 ceph-mon[97044]: pgmap v10586: 129 pgs: 129 active+clean; 8.8 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:05.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:05.465076+0000) 2022-09-10T10:00:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:05.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:05.946834+0000) 2022-09-10T10:00:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:06.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:06.465330+0000) 2022-09-10T10:00:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:06.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:06.947067+0000) 2022-09-10T10:00:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:07 smithi110 ceph-mon[97633]: pgmap v10587: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:00:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:07 smithi104 ceph-mon[97044]: pgmap v10587: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:00:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:07.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:07.465545+0000) 2022-09-10T10:00:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:00:08.112+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:00:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:07.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:07.947231+0000) 2022-09-10T10:00:08.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:08.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:08.465788+0000) 2022-09-10T10:00:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:08.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:08.947357+0000) 2022-09-10T10:00:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:09 smithi110 ceph-mon[97633]: pgmap v10588: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:00:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:09 smithi110 ceph-mon[97633]: osdmap e742: 12 total, 12 up, 12 in 2022-09-10T10:00:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:09 smithi104 ceph-mon[97044]: pgmap v10588: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:00:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:09 smithi104 ceph-mon[97044]: osdmap e742: 12 total, 12 up, 12 in 2022-09-10T10:00:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:09.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:09.466013+0000) 2022-09-10T10:00:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:09.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:09.947537+0000) 2022-09-10T10:00:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:10.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:10.466225+0000) 2022-09-10T10:00:11.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:10.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:10.947823+0000) 2022-09-10T10:00:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:11 smithi110 ceph-mon[97633]: pgmap v10590: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:00:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:11 smithi104 ceph-mon[97044]: pgmap v10590: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:00:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:11.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:11.466393+0000) 2022-09-10T10:00:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:11.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:11.948021+0000) 2022-09-10T10:00:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:12.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:12.466589+0000) 2022-09-10T10:00:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:12.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:12.948231+0000) 2022-09-10T10:00:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:13 smithi104 ceph-mon[97044]: pgmap v10591: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:00:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:13 smithi104 ceph-mon[97044]: osdmap e743: 12 total, 12 up, 12 in 2022-09-10T10:00:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:13 smithi110 ceph-mon[97633]: pgmap v10591: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:00:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:13 smithi110 ceph-mon[97633]: osdmap e743: 12 total, 12 up, 12 in 2022-09-10T10:00:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:13.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:13.466805+0000) 2022-09-10T10:00:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:13.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:13.948448+0000) 2022-09-10T10:00:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:14.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:14.467062+0000) 2022-09-10T10:00:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:14.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:14.948576+0000) 2022-09-10T10:00:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:15 smithi104 ceph-mon[97044]: pgmap v10593: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:00:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:15 smithi110 ceph-mon[97633]: pgmap v10593: 129 pgs: 129 active+clean; 8.9 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:00:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:15.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:15.467277+0000) 2022-09-10T10:00:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:15.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:15.948730+0000) 2022-09-10T10:00:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:16.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:16.467483+0000) 2022-09-10T10:00:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:16.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:16.948867+0000) 2022-09-10T10:00:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:17 smithi104 ceph-mon[97044]: pgmap v10594: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:00:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:17 smithi110 ceph-mon[97633]: pgmap v10594: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:00:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:17.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:17.467659+0000) 2022-09-10T10:00:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:17.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:17.949129+0000) 2022-09-10T10:00:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:18.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:18.467904+0000) 2022-09-10T10:00:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:18.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:18.949317+0000) 2022-09-10T10:00:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:19 smithi104 ceph-mon[97044]: pgmap v10595: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:00:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:19 smithi110 ceph-mon[97633]: pgmap v10595: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:00:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:19.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:19.468199+0000) 2022-09-10T10:00:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:19.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:19.949490+0000) 2022-09-10T10:00:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:20.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:20.468416+0000) 2022-09-10T10:00:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:20.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:20.949736+0000) 2022-09-10T10:00:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:21 smithi104 ceph-mon[97044]: pgmap v10596: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:00:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:21 smithi110 ceph-mon[97633]: pgmap v10596: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:00:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:21.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:21.468556+0000) 2022-09-10T10:00:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:21.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:21.949989+0000) 2022-09-10T10:00:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:22.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:22.468721+0000) 2022-09-10T10:00:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:22.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:22.950214+0000) 2022-09-10T10:00:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:00:23.120+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:00:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:23 smithi104 ceph-mon[97044]: pgmap v10597: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:00:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:23 smithi110 ceph-mon[97633]: pgmap v10597: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:00:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:23.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:23.469021+0000) 2022-09-10T10:00:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:23.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:23.950437+0000) 2022-09-10T10:00:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:24.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:24.469227+0000) 2022-09-10T10:00:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:24.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:24.950646+0000) 2022-09-10T10:00:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:25 smithi104 ceph-mon[97044]: pgmap v10598: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:00:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:25 smithi110 ceph-mon[97633]: pgmap v10598: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:00:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:25.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:25.469436+0000) 2022-09-10T10:00:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:25.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:25.950905+0000) 2022-09-10T10:00:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:26.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:26.469666+0000) 2022-09-10T10:00:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:26.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:26.951154+0000) 2022-09-10T10:00:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:27 smithi104 ceph-mon[97044]: pgmap v10599: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:00:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:27 smithi110 ceph-mon[97633]: pgmap v10599: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:00:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:27.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:27.469914+0000) 2022-09-10T10:00:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:27.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:27.951340+0000) 2022-09-10T10:00:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:28.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:28.470205+0000) 2022-09-10T10:00:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:28.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:28.951474+0000) 2022-09-10T10:00:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:29 smithi104 ceph-mon[97044]: pgmap v10600: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:29 smithi110 ceph-mon[97633]: pgmap v10600: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:29.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:29.470456+0000) 2022-09-10T10:00:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:29.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:29.951686+0000) 2022-09-10T10:00:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:30.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:30.470727+0000) 2022-09-10T10:00:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:30.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:30.951937+0000) 2022-09-10T10:00:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:31 smithi104 ceph-mon[97044]: pgmap v10601: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:31 smithi110 ceph-mon[97633]: pgmap v10601: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:31.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:31.470965+0000) 2022-09-10T10:00:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:31.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:31.952226+0000) 2022-09-10T10:00:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:32.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:32.471239+0000) 2022-09-10T10:00:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:32.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:32.952450+0000) 2022-09-10T10:00:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:33 smithi104 ceph-mon[97044]: pgmap v10602: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:33 smithi110 ceph-mon[97633]: pgmap v10602: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:33.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:33.471423+0000) 2022-09-10T10:00:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:33.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:33.952658+0000) 2022-09-10T10:00:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:34.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:34.471713+0000) 2022-09-10T10:00:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:34.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:34.952924+0000) 2022-09-10T10:00:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:35 smithi104 ceph-mon[97044]: pgmap v10603: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:35 smithi110 ceph-mon[97633]: pgmap v10603: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:35.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:35.471894+0000) 2022-09-10T10:00:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:35.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:35.953203+0000) 2022-09-10T10:00:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:36.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:36.472073+0000) 2022-09-10T10:00:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:36.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:36.953367+0000) 2022-09-10T10:00:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:37 smithi104 ceph-mon[97044]: pgmap v10604: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:37 smithi110 ceph-mon[97633]: pgmap v10604: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:37.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:37.472338+0000) 2022-09-10T10:00:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:37.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:37.953589+0000) 2022-09-10T10:00:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:00:38.135+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:00:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:38.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:38.472560+0000) 2022-09-10T10:00:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:38.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:38.953860+0000) 2022-09-10T10:00:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:39 smithi104 ceph-mon[97044]: pgmap v10605: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:39 smithi110 ceph-mon[97633]: pgmap v10605: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:39.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:39.472745+0000) 2022-09-10T10:00:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:39.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:39.954122+0000) 2022-09-10T10:00:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:40.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:40.472919+0000) 2022-09-10T10:00:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:40.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:40.954310+0000) 2022-09-10T10:00:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:41 smithi104 ceph-mon[97044]: pgmap v10606: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:41 smithi110 ceph-mon[97633]: pgmap v10606: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:41.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:41.473183+0000) 2022-09-10T10:00:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:41.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:41.954527+0000) 2022-09-10T10:00:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:42.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:42.473367+0000) 2022-09-10T10:00:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:42.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:42.954799+0000) 2022-09-10T10:00:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:43 smithi104 ceph-mon[97044]: pgmap v10607: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:43 smithi110 ceph-mon[97633]: pgmap v10607: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:43.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:43.473603+0000) 2022-09-10T10:00:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:43.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:43.954994+0000) 2022-09-10T10:00:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:44.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:44.473901+0000) 2022-09-10T10:00:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:44.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:44.955164+0000) 2022-09-10T10:00:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:45 smithi104 ceph-mon[97044]: pgmap v10608: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:00:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:00:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:00:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:00:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:45 smithi110 ceph-mon[97633]: pgmap v10608: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:00:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:00:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:00:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:00:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:45.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:45.474141+0000) 2022-09-10T10:00:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:45.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:45.955380+0000) 2022-09-10T10:00:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:46.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:46.474379+0000) 2022-09-10T10:00:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:46.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:46.955573+0000) 2022-09-10T10:00:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:47 smithi104 ceph-mon[97044]: pgmap v10609: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:47 smithi110 ceph-mon[97633]: pgmap v10609: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:47.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:47.474542+0000) 2022-09-10T10:00:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:47.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:47.955775+0000) 2022-09-10T10:00:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:48.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:48.474710+0000) 2022-09-10T10:00:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:48.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:48.955938+0000) 2022-09-10T10:00:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:49 smithi110 ceph-mon[97633]: pgmap v10610: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:49 smithi104 ceph-mon[97044]: pgmap v10610: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:49.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:49.474938+0000) 2022-09-10T10:00:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:49.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:49.956182+0000) 2022-09-10T10:00:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:50.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:50.475221+0000) 2022-09-10T10:00:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:50.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:50.956367+0000) 2022-09-10T10:00:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:51 smithi110 ceph-mon[97633]: pgmap v10611: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:51 smithi104 ceph-mon[97044]: pgmap v10611: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:51.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:51.475406+0000) 2022-09-10T10:00:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:51.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:51.956587+0000) 2022-09-10T10:00:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:52.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:52.475616+0000) 2022-09-10T10:00:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:52.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:52.956840+0000) 2022-09-10T10:00:53.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:00:53.151+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:00:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:53 smithi110 ceph-mon[97633]: pgmap v10612: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:53 smithi104 ceph-mon[97044]: pgmap v10612: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:53.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:53.475817+0000) 2022-09-10T10:00:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:53.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:53.957145+0000) 2022-09-10T10:00:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:54.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:54.476067+0000) 2022-09-10T10:00:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:54.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:54.957391+0000) 2022-09-10T10:00:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:55 smithi110 ceph-mon[97633]: pgmap v10613: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:55 smithi104 ceph-mon[97044]: pgmap v10613: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:55.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:55.476268+0000) 2022-09-10T10:00:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:55.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:55.957571+0000) 2022-09-10T10:00:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:56.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:56.476515+0000) 2022-09-10T10:00:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:56.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:56.957711+0000) 2022-09-10T10:00:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:57 smithi110 ceph-mon[97633]: pgmap v10614: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:57 smithi104 ceph-mon[97044]: pgmap v10614: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:00:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:57.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:57.476712+0000) 2022-09-10T10:00:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:57.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:57.957896+0000) 2022-09-10T10:00:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:58.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:58.476845+0000) 2022-09-10T10:00:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:58.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:58.958172+0000) 2022-09-10T10:00:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:00:59 smithi110 ceph-mon[97633]: pgmap v10615: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:00:59 smithi104 ceph-mon[97044]: pgmap v10615: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:00:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:00:59.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:59.477026+0000) 2022-09-10T10:01:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:00:59.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:00:59.958416+0000) 2022-09-10T10:01:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:00.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:00.477154+0000) 2022-09-10T10:01:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:00.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:00.958567+0000) 2022-09-10T10:01:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:01 smithi110 ceph-mon[97633]: pgmap v10616: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:01.751 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:01 smithi104 ceph-mon[97044]: pgmap v10616: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:01.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:01.477379+0000) 2022-09-10T10:01:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:01.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:01.958767+0000) 2022-09-10T10:01:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:02.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:02.477523+0000) 2022-09-10T10:01:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:02.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:02.959033+0000) 2022-09-10T10:01:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:03 smithi110 ceph-mon[97633]: pgmap v10617: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:03 smithi104 ceph-mon[97044]: pgmap v10617: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:03.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:03.477659+0000) 2022-09-10T10:01:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:03.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:03.959211+0000) 2022-09-10T10:01:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:04.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:04.477888+0000) 2022-09-10T10:01:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:04.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:04.959334+0000) 2022-09-10T10:01:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:05 smithi110 ceph-mon[97633]: pgmap v10618: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:05 smithi104 ceph-mon[97044]: pgmap v10618: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:05.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:05.478175+0000) 2022-09-10T10:01:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:05.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:05.959554+0000) 2022-09-10T10:01:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:06.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:06.478412+0000) 2022-09-10T10:01:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:06.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:06.959801+0000) 2022-09-10T10:01:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:07 smithi110 ceph-mon[97633]: pgmap v10619: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:01:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:07 smithi104 ceph-mon[97044]: pgmap v10619: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:01:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:07.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:07.478597+0000) 2022-09-10T10:01:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:01:08.163+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:01:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:07.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:07.960054+0000) 2022-09-10T10:01:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:08.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:08.478780+0000) 2022-09-10T10:01:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:08.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:08.960224+0000) 2022-09-10T10:01:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:09 smithi110 ceph-mon[97633]: pgmap v10620: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:01:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:09 smithi110 ceph-mon[97633]: osdmap e744: 12 total, 12 up, 12 in 2022-09-10T10:01:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:09 smithi104 ceph-mon[97044]: pgmap v10620: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:01:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:09 smithi104 ceph-mon[97044]: osdmap e744: 12 total, 12 up, 12 in 2022-09-10T10:01:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:09.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:09.479034+0000) 2022-09-10T10:01:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:09.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:09.960479+0000) 2022-09-10T10:01:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:10.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:10.479203+0000) 2022-09-10T10:01:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:10.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:10.960673+0000) 2022-09-10T10:01:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:11 smithi110 ceph-mon[97633]: pgmap v10622: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:01:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:11 smithi104 ceph-mon[97044]: pgmap v10622: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:01:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:11.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:11.479415+0000) 2022-09-10T10:01:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:11.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:11.960841+0000) 2022-09-10T10:01:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:12.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:12.479623+0000) 2022-09-10T10:01:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:12.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:12.961040+0000) 2022-09-10T10:01:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:13 smithi104 ceph-mon[97044]: pgmap v10623: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:01:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:13 smithi104 ceph-mon[97044]: osdmap e745: 12 total, 12 up, 12 in 2022-09-10T10:01:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:13 smithi110 ceph-mon[97633]: pgmap v10623: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:01:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:13 smithi110 ceph-mon[97633]: osdmap e745: 12 total, 12 up, 12 in 2022-09-10T10:01:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:13.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:13.479856+0000) 2022-09-10T10:01:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:13.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:13.961243+0000) 2022-09-10T10:01:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:14.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:14.480132+0000) 2022-09-10T10:01:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:14.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:14.961441+0000) 2022-09-10T10:01:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:15 smithi104 ceph-mon[97044]: pgmap v10625: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:01:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:15 smithi110 ceph-mon[97633]: pgmap v10625: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:01:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:15.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:15.480374+0000) 2022-09-10T10:01:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:15.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:15.961663+0000) 2022-09-10T10:01:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:16.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:16.480662+0000) 2022-09-10T10:01:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:16.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:16.961902+0000) 2022-09-10T10:01:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:17 smithi104 ceph-mon[97044]: pgmap v10626: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:01:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:17 smithi110 ceph-mon[97633]: pgmap v10626: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:01:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:17.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:17.480842+0000) 2022-09-10T10:01:18.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:17.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:17.962073+0000) 2022-09-10T10:01:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:18.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:18.481136+0000) 2022-09-10T10:01:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:18.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:18.962284+0000) 2022-09-10T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:19 smithi104 ceph-mon[97044]: pgmap v10627: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:01:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:19 smithi110 ceph-mon[97633]: pgmap v10627: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:01:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:19.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:19.481266+0000) 2022-09-10T10:01:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:19.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:19.962535+0000) 2022-09-10T10:01:20.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:20.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:20.481417+0000) 2022-09-10T10:01:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:20.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:20.962761+0000) 2022-09-10T10:01:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:21 smithi104 ceph-mon[97044]: pgmap v10628: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:01:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:21 smithi110 ceph-mon[97633]: pgmap v10628: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:01:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:21.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:21.481656+0000) 2022-09-10T10:01:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:21.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:21.962980+0000) 2022-09-10T10:01:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:22.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:22.481874+0000) 2022-09-10T10:01:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:22.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:22.963169+0000) 2022-09-10T10:01:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:01:23.169+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:01:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:23 smithi104 ceph-mon[97044]: pgmap v10629: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:01:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:23 smithi110 ceph-mon[97633]: pgmap v10629: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:01:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:23.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:23.482129+0000) 2022-09-10T10:01:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:23.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:23.963375+0000) 2022-09-10T10:01:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:24.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:24.482359+0000) 2022-09-10T10:01:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:24.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:24.963634+0000) 2022-09-10T10:01:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:25 smithi104 ceph-mon[97044]: pgmap v10630: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:01:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:25 smithi110 ceph-mon[97633]: pgmap v10630: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:01:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:25.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:25.482605+0000) 2022-09-10T10:01:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:25.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:25.963830+0000) 2022-09-10T10:01:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:26.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:26.482799+0000) 2022-09-10T10:01:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:26.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:26.964122+0000) 2022-09-10T10:01:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:27 smithi104 ceph-mon[97044]: pgmap v10631: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:01:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:27 smithi110 ceph-mon[97633]: pgmap v10631: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:01:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:27.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:27.483009+0000) 2022-09-10T10:01:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:27.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:27.964326+0000) 2022-09-10T10:01:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:28.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:28.483285+0000) 2022-09-10T10:01:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:28.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:28.964476+0000) 2022-09-10T10:01:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:29 smithi104 ceph-mon[97044]: pgmap v10632: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:29 smithi110 ceph-mon[97633]: pgmap v10632: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:29.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:29.483537+0000) 2022-09-10T10:01:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:29.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:29.964690+0000) 2022-09-10T10:01:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:30.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:30.483784+0000) 2022-09-10T10:01:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:30.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:30.964955+0000) 2022-09-10T10:01:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:31 smithi104 ceph-mon[97044]: pgmap v10633: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:31 smithi110 ceph-mon[97633]: pgmap v10633: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:31.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:31.483997+0000) 2022-09-10T10:01:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:31.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:31.965171+0000) 2022-09-10T10:01:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:32.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:32.484243+0000) 2022-09-10T10:01:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:32.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:32.965337+0000) 2022-09-10T10:01:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:33 smithi104 ceph-mon[97044]: pgmap v10634: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:33 smithi110 ceph-mon[97633]: pgmap v10634: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:33.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:33.484442+0000) 2022-09-10T10:01:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:33.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:33.965526+0000) 2022-09-10T10:01:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:34.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:34.484654+0000) 2022-09-10T10:01:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:34.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:34.965687+0000) 2022-09-10T10:01:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:35 smithi104 ceph-mon[97044]: pgmap v10635: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:35 smithi110 ceph-mon[97633]: pgmap v10635: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:35.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:35.484893+0000) 2022-09-10T10:01:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:35.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:35.965900+0000) 2022-09-10T10:01:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:36.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:36.485149+0000) 2022-09-10T10:01:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:36.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:36.966168+0000) 2022-09-10T10:01:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:37 smithi104 ceph-mon[97044]: pgmap v10636: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:37 smithi110 ceph-mon[97633]: pgmap v10636: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:37.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:37.485423+0000) 2022-09-10T10:01:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:37.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:37.966388+0000) 2022-09-10T10:01:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:01:38.183+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:01:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:38.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:38.485691+0000) 2022-09-10T10:01:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:38.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:38.966629+0000) 2022-09-10T10:01:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:39 smithi110 ceph-mon[97633]: pgmap v10637: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:39 smithi104 ceph-mon[97044]: pgmap v10637: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:39.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:39.485940+0000) 2022-09-10T10:01:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:39.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:39.966869+0000) 2022-09-10T10:01:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:40.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:40.486164+0000) 2022-09-10T10:01:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:40.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:40.967055+0000) 2022-09-10T10:01:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:41 smithi110 ceph-mon[97633]: pgmap v10638: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:41 smithi104 ceph-mon[97044]: pgmap v10638: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:41.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:41.486332+0000) 2022-09-10T10:01:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:41.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:41.967291+0000) 2022-09-10T10:01:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:42.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:42.486511+0000) 2022-09-10T10:01:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:42.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:42.967544+0000) 2022-09-10T10:01:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:43 smithi110 ceph-mon[97633]: pgmap v10639: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:43 smithi104 ceph-mon[97044]: pgmap v10639: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:43.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:43.486734+0000) 2022-09-10T10:01:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:43.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:43.967738+0000) 2022-09-10T10:01:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:44.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:44.486968+0000) 2022-09-10T10:01:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:44.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:44.967965+0000) 2022-09-10T10:01:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:45 smithi110 ceph-mon[97633]: pgmap v10640: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:01:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:01:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:01:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:01:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:45 smithi104 ceph-mon[97044]: pgmap v10640: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:45.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:01:45.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:01:45.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:01:45.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:01:45.826 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:45.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:45.487171+0000) 2022-09-10T10:01:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:45.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:45.968204+0000) 2022-09-10T10:01:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:46.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:46.487385+0000) 2022-09-10T10:01:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:46.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:46.968380+0000) 2022-09-10T10:01:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:47 smithi110 ceph-mon[97633]: pgmap v10641: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:47 smithi104 ceph-mon[97044]: pgmap v10641: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:47.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:47.487712+0000) 2022-09-10T10:01:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:47.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:47.968555+0000) 2022-09-10T10:01:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:48.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:48.487954+0000) 2022-09-10T10:01:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:48.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:48.968736+0000) 2022-09-10T10:01:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:49 smithi110 ceph-mon[97633]: pgmap v10642: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:49 smithi104 ceph-mon[97044]: pgmap v10642: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:01:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:49.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:49.488190+0000) 2022-09-10T10:01:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:49.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:49.968984+0000) 2022-09-10T10:01:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:50.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:50.488452+0000) 2022-09-10T10:01:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:50.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:50.969208+0000) 2022-09-10T10:01:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:51 smithi110 ceph-mon[97633]: pgmap v10643: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:51.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:51 smithi104 ceph-mon[97044]: pgmap v10643: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:51.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:51.488639+0000) 2022-09-10T10:01:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:51.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:51.969504+0000) 2022-09-10T10:01:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:52.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:52.488841+0000) 2022-09-10T10:01:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:52.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:52.969706+0000) 2022-09-10T10:01:53.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:01:53.192+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:01:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:53 smithi110 ceph-mon[97633]: pgmap v10644: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:01:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:53 smithi104 ceph-mon[97044]: pgmap v10644: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:01:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:53.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:53.489076+0000) 2022-09-10T10:01:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:53.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:53.969933+0000) 2022-09-10T10:01:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:54.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:54.489315+0000) 2022-09-10T10:01:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:54.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:54.970178+0000) 2022-09-10T10:01:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:55 smithi110 ceph-mon[97633]: pgmap v10645: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:01:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:55 smithi104 ceph-mon[97044]: pgmap v10645: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:01:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:55.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:55.489479+0000) 2022-09-10T10:01:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:55.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:55.970404+0000) 2022-09-10T10:01:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:56.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:56.489671+0000) 2022-09-10T10:01:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:56.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:56.970624+0000) 2022-09-10T10:01:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:57 smithi110 ceph-mon[97633]: pgmap v10646: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:57 smithi104 ceph-mon[97044]: pgmap v10646: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:01:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:57.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:57.489903+0000) 2022-09-10T10:01:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:57.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:57.970831+0000) 2022-09-10T10:01:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:58.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:58.490172+0000) 2022-09-10T10:01:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:58.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:58.970980+0000) 2022-09-10T10:01:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:01:59 smithi110 ceph-mon[97633]: pgmap v10647: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:01:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:01:59 smithi104 ceph-mon[97044]: pgmap v10647: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:01:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:01:59.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:59.490318+0000) 2022-09-10T10:02:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:01:59.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:01:59.971164+0000) 2022-09-10T10:02:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:00.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:00.490479+0000) 2022-09-10T10:02:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:00.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:00.971303+0000) 2022-09-10T10:02:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:01 smithi110 ceph-mon[97633]: pgmap v10648: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:01.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:01 smithi104 ceph-mon[97044]: pgmap v10648: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:01.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:01.490653+0000) 2022-09-10T10:02:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:01.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:01.971568+0000) 2022-09-10T10:02:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:02.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:02.490851+0000) 2022-09-10T10:02:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:02.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:02.971826+0000) 2022-09-10T10:02:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:03 smithi110 ceph-mon[97633]: pgmap v10649: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:03 smithi104 ceph-mon[97044]: pgmap v10649: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:03.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:03.491088+0000) 2022-09-10T10:02:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:03.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:03.972066+0000) 2022-09-10T10:02:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:04.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:04.491361+0000) 2022-09-10T10:02:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:04.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:04.972269+0000) 2022-09-10T10:02:05.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:05 smithi110 ceph-mon[97633]: pgmap v10650: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:05 smithi104 ceph-mon[97044]: pgmap v10650: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:05.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:05.491618+0000) 2022-09-10T10:02:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:05.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:05.972499+0000) 2022-09-10T10:02:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:06.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:06.491767+0000) 2022-09-10T10:02:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:06.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:06.972757+0000) 2022-09-10T10:02:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:07 smithi110 ceph-mon[97633]: pgmap v10651: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:02:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:07 smithi104 ceph-mon[97044]: pgmap v10651: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:02:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:07.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:07.492004+0000) 2022-09-10T10:02:08.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:02:08.208+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:02:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:07.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:07.972895+0000) 2022-09-10T10:02:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:08.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:08.492227+0000) 2022-09-10T10:02:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:08.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:08.973080+0000) 2022-09-10T10:02:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:09 smithi110 ceph-mon[97633]: pgmap v10652: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:02:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:09 smithi110 ceph-mon[97633]: osdmap e746: 12 total, 12 up, 12 in 2022-09-10T10:02:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:09 smithi104 ceph-mon[97044]: pgmap v10652: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:02:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:09 smithi104 ceph-mon[97044]: osdmap e746: 12 total, 12 up, 12 in 2022-09-10T10:02:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:09.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:09.492426+0000) 2022-09-10T10:02:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:09.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:09.973322+0000) 2022-09-10T10:02:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:10.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:10.492610+0000) 2022-09-10T10:02:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:10.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:10.973544+0000) 2022-09-10T10:02:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:11 smithi110 ceph-mon[97633]: pgmap v10654: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:02:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:11 smithi104 ceph-mon[97044]: pgmap v10654: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:02:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:11.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:11.492859+0000) 2022-09-10T10:02:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:11.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:11.973763+0000) 2022-09-10T10:02:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:12.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:12.493134+0000) 2022-09-10T10:02:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:12.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:12.973957+0000) 2022-09-10T10:02:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:13 smithi104 ceph-mon[97044]: pgmap v10655: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:02:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:13 smithi104 ceph-mon[97044]: osdmap e747: 12 total, 12 up, 12 in 2022-09-10T10:02:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:13 smithi110 ceph-mon[97633]: pgmap v10655: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:02:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:13 smithi110 ceph-mon[97633]: osdmap e747: 12 total, 12 up, 12 in 2022-09-10T10:02:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:13.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:13.493336+0000) 2022-09-10T10:02:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:13.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:13.974209+0000) 2022-09-10T10:02:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:14.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:14.493519+0000) 2022-09-10T10:02:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:14.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:14.974457+0000) 2022-09-10T10:02:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:15 smithi104 ceph-mon[97044]: pgmap v10657: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:02:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:15 smithi110 ceph-mon[97633]: pgmap v10657: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:02:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:15.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:15.493714+0000) 2022-09-10T10:02:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:15.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:15.974681+0000) 2022-09-10T10:02:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:16.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:16.493919+0000) 2022-09-10T10:02:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:16.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:16.974835+0000) 2022-09-10T10:02:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:17 smithi104 ceph-mon[97044]: pgmap v10658: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:02:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:17 smithi110 ceph-mon[97633]: pgmap v10658: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:02:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:17.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:17.494116+0000) 2022-09-10T10:02:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:17.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:17.974999+0000) 2022-09-10T10:02:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:18.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:18.494326+0000) 2022-09-10T10:02:19.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:18.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:18.975220+0000) 2022-09-10T10:02:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:19 smithi104 ceph-mon[97044]: pgmap v10659: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:02:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:19 smithi110 ceph-mon[97633]: pgmap v10659: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:02:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:19.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:19.494577+0000) 2022-09-10T10:02:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:19.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:19.975442+0000) 2022-09-10T10:02:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:20.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:20.494758+0000) 2022-09-10T10:02:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:20.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:20.975669+0000) 2022-09-10T10:02:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:21 smithi104 ceph-mon[97044]: pgmap v10660: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:02:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:21 smithi110 ceph-mon[97633]: pgmap v10660: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:02:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:21.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:21.494975+0000) 2022-09-10T10:02:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:21.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:21.975940+0000) 2022-09-10T10:02:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:22.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:22.495181+0000) 2022-09-10T10:02:23.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:02:23.223+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:02:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:22.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:22.976160+0000) 2022-09-10T10:02:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:23 smithi104 ceph-mon[97044]: pgmap v10661: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:02:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:23 smithi110 ceph-mon[97633]: pgmap v10661: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:02:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:23.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:23.495377+0000) 2022-09-10T10:02:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:23.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:23.976333+0000) 2022-09-10T10:02:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:24.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:24.495625+0000) 2022-09-10T10:02:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:24.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:24.976573+0000) 2022-09-10T10:02:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:25 smithi110 ceph-mon[97633]: pgmap v10662: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:02:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:25 smithi104 ceph-mon[97044]: pgmap v10662: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:02:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:25.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:25.495812+0000) 2022-09-10T10:02:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:25.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:25.976817+0000) 2022-09-10T10:02:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:26.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:26.496009+0000) 2022-09-10T10:02:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:26.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:26.977009+0000) 2022-09-10T10:02:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:27 smithi110 ceph-mon[97633]: pgmap v10663: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:02:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:27 smithi104 ceph-mon[97044]: pgmap v10663: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:02:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:27.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:27.496257+0000) 2022-09-10T10:02:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:27.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:27.977255+0000) 2022-09-10T10:02:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:28.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:28.496505+0000) 2022-09-10T10:02:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:28.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:28.977392+0000) 2022-09-10T10:02:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:29 smithi110 ceph-mon[97633]: pgmap v10664: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:29 smithi104 ceph-mon[97044]: pgmap v10664: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:29.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:29.496729+0000) 2022-09-10T10:02:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:29.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:29.977564+0000) 2022-09-10T10:02:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:30.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:30.496947+0000) 2022-09-10T10:02:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:30.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:30.977699+0000) 2022-09-10T10:02:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:31 smithi110 ceph-mon[97633]: pgmap v10665: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:31 smithi104 ceph-mon[97044]: pgmap v10665: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:31.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:31.497182+0000) 2022-09-10T10:02:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:31.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:31.977898+0000) 2022-09-10T10:02:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:32.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:32.497360+0000) 2022-09-10T10:02:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:32.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:32.978060+0000) 2022-09-10T10:02:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:33 smithi110 ceph-mon[97633]: pgmap v10666: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:33 smithi104 ceph-mon[97044]: pgmap v10666: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:33.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:33.497638+0000) 2022-09-10T10:02:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:33.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:33.978297+0000) 2022-09-10T10:02:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:34.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:34.497813+0000) 2022-09-10T10:02:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:34.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:34.978558+0000) 2022-09-10T10:02:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:35 smithi110 ceph-mon[97633]: pgmap v10667: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:35 smithi104 ceph-mon[97044]: pgmap v10667: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:35.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:35.498048+0000) 2022-09-10T10:02:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:35.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:35.978717+0000) 2022-09-10T10:02:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:36.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:36.498233+0000) 2022-09-10T10:02:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:36.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:36.978899+0000) 2022-09-10T10:02:37.674 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:37 smithi110 ceph-mon[97633]: pgmap v10668: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:37 smithi104 ceph-mon[97044]: pgmap v10668: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:37.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:37.498485+0000) 2022-09-10T10:02:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:37.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:37.979172+0000) 2022-09-10T10:02:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:02:38.225+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:02:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:38.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:38.498704+0000) 2022-09-10T10:02:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:38.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:38.979347+0000) 2022-09-10T10:02:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:39 smithi110 ceph-mon[97633]: pgmap v10669: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:39 smithi104 ceph-mon[97044]: pgmap v10669: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:39.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:39.498904+0000) 2022-09-10T10:02:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:39.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:39.979531+0000) 2022-09-10T10:02:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:40.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:40.499089+0000) 2022-09-10T10:02:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:40.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:40.979755+0000) 2022-09-10T10:02:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:41 smithi110 ceph-mon[97633]: pgmap v10670: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:41 smithi104 ceph-mon[97044]: pgmap v10670: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:41.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:41.499353+0000) 2022-09-10T10:02:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:41.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:41.980005+0000) 2022-09-10T10:02:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:42.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:42.499622+0000) 2022-09-10T10:02:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:42.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:42.980219+0000) 2022-09-10T10:02:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:43 smithi110 ceph-mon[97633]: pgmap v10671: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:43 smithi104 ceph-mon[97044]: pgmap v10671: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:43.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:43.499860+0000) 2022-09-10T10:02:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:43.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:43.980453+0000) 2022-09-10T10:02:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:44.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:44.500042+0000) 2022-09-10T10:02:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:44.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:44.980621+0000) 2022-09-10T10:02:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:45 smithi110 ceph-mon[97633]: pgmap v10672: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:02:45.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:02:45.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:02:45.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:02:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:45 smithi104 ceph-mon[97044]: pgmap v10672: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:02:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:02:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:02:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:02:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:45.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:45.500307+0000) 2022-09-10T10:02:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:45.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:45.980853+0000) 2022-09-10T10:02:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:46.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:46.500551+0000) 2022-09-10T10:02:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:46.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:46.981081+0000) 2022-09-10T10:02:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:47 smithi110 ceph-mon[97633]: pgmap v10673: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:47 smithi104 ceph-mon[97044]: pgmap v10673: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:47.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:47.500772+0000) 2022-09-10T10:02:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:47.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:47.981300+0000) 2022-09-10T10:02:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:48.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:48.501025+0000) 2022-09-10T10:02:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:48.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:48.981449+0000) 2022-09-10T10:02:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:49 smithi110 ceph-mon[97633]: pgmap v10674: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:49 smithi104 ceph-mon[97044]: pgmap v10674: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:49.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:49.501191+0000) 2022-09-10T10:02:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:49.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:49.981629+0000) 2022-09-10T10:02:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:50.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:50.501418+0000) 2022-09-10T10:02:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:50.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:50.981838+0000) 2022-09-10T10:02:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:51 smithi110 ceph-mon[97633]: pgmap v10675: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:51 smithi104 ceph-mon[97044]: pgmap v10675: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:51.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:51.501606+0000) 2022-09-10T10:02:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:51.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:51.982005+0000) 2022-09-10T10:02:52.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:52.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:52.501814+0000) 2022-09-10T10:02:53.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:52.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:52.982151+0000) 2022-09-10T10:02:53.503 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:02:53.241+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:02:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:53 smithi110 ceph-mon[97633]: pgmap v10676: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:53 smithi104 ceph-mon[97044]: pgmap v10676: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:53.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:53.501963+0000) 2022-09-10T10:02:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:53.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:53.982368+0000) 2022-09-10T10:02:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:54.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:54.502163+0000) 2022-09-10T10:02:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:54.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:54.982552+0000) 2022-09-10T10:02:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:55 smithi110 ceph-mon[97633]: pgmap v10677: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:55 smithi104 ceph-mon[97044]: pgmap v10677: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:55.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:55.502311+0000) 2022-09-10T10:02:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:55.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:55.982762+0000) 2022-09-10T10:02:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:56.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:56.502470+0000) 2022-09-10T10:02:57.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:56.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:56.982998+0000) 2022-09-10T10:02:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:57 smithi110 ceph-mon[97633]: pgmap v10678: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:57 smithi104 ceph-mon[97044]: pgmap v10678: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:02:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:57.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:57.502668+0000) 2022-09-10T10:02:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:57.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:57.983222+0000) 2022-09-10T10:02:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:58.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:58.502863+0000) 2022-09-10T10:02:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:58.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:58.983482+0000) 2022-09-10T10:02:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:02:59 smithi110 ceph-mon[97633]: pgmap v10679: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:02:59 smithi104 ceph-mon[97044]: pgmap v10679: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:02:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:02:59.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:59.503037+0000) 2022-09-10T10:03:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:02:59.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:02:59.983689+0000) 2022-09-10T10:03:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:00.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:00.503260+0000) 2022-09-10T10:03:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:00.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:00.983847+0000) 2022-09-10T10:03:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:01 smithi110 ceph-mon[97633]: pgmap v10680: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:01 smithi104 ceph-mon[97044]: pgmap v10680: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:01.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:01.503432+0000) 2022-09-10T10:03:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:01.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:01.984013+0000) 2022-09-10T10:03:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:02.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:02.503679+0000) 2022-09-10T10:03:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:02.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:02.984241+0000) 2022-09-10T10:03:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:03 smithi110 ceph-mon[97633]: pgmap v10681: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:03 smithi104 ceph-mon[97044]: pgmap v10681: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:03.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:03.503911+0000) 2022-09-10T10:03:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:03.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:03.984452+0000) 2022-09-10T10:03:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:04.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:04.504167+0000) 2022-09-10T10:03:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:04.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:04.984658+0000) 2022-09-10T10:03:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:05 smithi110 ceph-mon[97633]: pgmap v10682: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:05 smithi104 ceph-mon[97044]: pgmap v10682: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:05.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:05.504327+0000) 2022-09-10T10:03:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:05.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:05.984887+0000) 2022-09-10T10:03:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:06.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:06.504554+0000) 2022-09-10T10:03:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:06.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:06.985090+0000) 2022-09-10T10:03:07.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:07 smithi110 ceph-mon[97633]: pgmap v10683: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:03:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:07 smithi104 ceph-mon[97044]: pgmap v10683: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:03:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:07.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:07.504794+0000) 2022-09-10T10:03:08.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:07.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:07.985362+0000) 2022-09-10T10:03:08.506 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:03:08.246+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:03:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:08.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:08.505048+0000) 2022-09-10T10:03:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:08.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:08.985558+0000) 2022-09-10T10:03:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:09 smithi110 ceph-mon[97633]: pgmap v10684: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:03:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:09 smithi110 ceph-mon[97633]: osdmap e748: 12 total, 12 up, 12 in 2022-09-10T10:03:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:09 smithi104 ceph-mon[97044]: pgmap v10684: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:03:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:09 smithi104 ceph-mon[97044]: osdmap e748: 12 total, 12 up, 12 in 2022-09-10T10:03:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:09.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:09.505257+0000) 2022-09-10T10:03:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:09.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:09.985757+0000) 2022-09-10T10:03:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:10.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:10.505453+0000) 2022-09-10T10:03:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:10.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:10.985907+0000) 2022-09-10T10:03:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:11 smithi110 ceph-mon[97633]: pgmap v10686: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:03:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:11 smithi104 ceph-mon[97044]: pgmap v10686: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:03:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:11.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:11.505644+0000) 2022-09-10T10:03:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:11.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:11.986160+0000) 2022-09-10T10:03:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:12.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:12.505819+0000) 2022-09-10T10:03:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:12.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:12.986356+0000) 2022-09-10T10:03:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:13 smithi104 ceph-mon[97044]: pgmap v10687: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:03:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:13 smithi104 ceph-mon[97044]: osdmap e749: 12 total, 12 up, 12 in 2022-09-10T10:03:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:13 smithi110 ceph-mon[97633]: pgmap v10687: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:03:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:13 smithi110 ceph-mon[97633]: osdmap e749: 12 total, 12 up, 12 in 2022-09-10T10:03:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:13.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:13.505964+0000) 2022-09-10T10:03:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:13.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:13.986571+0000) 2022-09-10T10:03:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:14.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:14.506177+0000) 2022-09-10T10:03:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:14.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:14.986844+0000) 2022-09-10T10:03:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:15 smithi110 ceph-mon[97633]: pgmap v10689: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:03:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:15 smithi104 ceph-mon[97044]: pgmap v10689: 129 pgs: 129 active+clean; 8.9 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:03:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:15.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:15.506375+0000) 2022-09-10T10:03:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:15.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:15.987044+0000) 2022-09-10T10:03:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:16.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:16.506601+0000) 2022-09-10T10:03:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:16.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:16.987272+0000) 2022-09-10T10:03:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:17 smithi110 ceph-mon[97633]: pgmap v10690: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:03:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:17 smithi104 ceph-mon[97044]: pgmap v10690: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:03:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:17.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:17.506860+0000) 2022-09-10T10:03:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:17.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:17.987486+0000) 2022-09-10T10:03:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:18.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:18.507089+0000) 2022-09-10T10:03:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:18.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:18.987715+0000) 2022-09-10T10:03:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:19 smithi110 ceph-mon[97633]: pgmap v10691: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:03:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:19 smithi104 ceph-mon[97044]: pgmap v10691: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:03:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:19.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:19.507303+0000) 2022-09-10T10:03:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:19.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:19.987965+0000) 2022-09-10T10:03:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:20.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:20.507567+0000) 2022-09-10T10:03:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:20.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:20.988208+0000) 2022-09-10T10:03:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:21 smithi110 ceph-mon[97633]: pgmap v10692: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:03:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:21 smithi104 ceph-mon[97044]: pgmap v10692: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:03:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:21.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:21.507770+0000) 2022-09-10T10:03:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:21.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:21.988495+0000) 2022-09-10T10:03:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:22.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:22.507972+0000) 2022-09-10T10:03:23.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:22.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:22.988732+0000) 2022-09-10T10:03:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:23.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:23.508209+0000) 2022-09-10T10:03:23.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:03:23.259+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:03:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:23 smithi110 ceph-mon[97633]: pgmap v10693: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:03:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:23 smithi104 ceph-mon[97044]: pgmap v10693: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:03:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:23.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:23.988949+0000) 2022-09-10T10:03:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:24.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:24.508426+0000) 2022-09-10T10:03:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:24.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:24.989175+0000) 2022-09-10T10:03:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:25 smithi110 ceph-mon[97633]: pgmap v10694: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:03:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:25 smithi104 ceph-mon[97044]: pgmap v10694: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:03:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:25.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:25.508646+0000) 2022-09-10T10:03:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:25.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:25.989339+0000) 2022-09-10T10:03:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:26.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:26.508897+0000) 2022-09-10T10:03:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:26.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:26.989559+0000) 2022-09-10T10:03:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:27 smithi110 ceph-mon[97633]: pgmap v10695: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:03:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:27 smithi104 ceph-mon[97044]: pgmap v10695: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:03:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:27.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:27.509147+0000) 2022-09-10T10:03:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:27.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:27.989782+0000) 2022-09-10T10:03:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:28.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:28.509380+0000) 2022-09-10T10:03:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:28.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:28.989990+0000) 2022-09-10T10:03:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:29 smithi110 ceph-mon[97633]: pgmap v10696: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:29 smithi104 ceph-mon[97044]: pgmap v10696: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:29.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:29.509589+0000) 2022-09-10T10:03:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:29.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:29.990163+0000) 2022-09-10T10:03:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:30.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:30.509766+0000) 2022-09-10T10:03:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:30.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:30.990390+0000) 2022-09-10T10:03:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:31 smithi110 ceph-mon[97633]: pgmap v10697: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:31 smithi104 ceph-mon[97044]: pgmap v10697: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:31.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:31.510006+0000) 2022-09-10T10:03:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:31.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:31.990564+0000) 2022-09-10T10:03:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:32.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:32.510223+0000) 2022-09-10T10:03:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:32.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:32.990819+0000) 2022-09-10T10:03:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:33 smithi110 ceph-mon[97633]: pgmap v10698: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:33 smithi104 ceph-mon[97044]: pgmap v10698: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:33.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:33.510450+0000) 2022-09-10T10:03:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:33.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:33.990996+0000) 2022-09-10T10:03:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:34.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:34.510690+0000) 2022-09-10T10:03:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:34.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:34.991257+0000) 2022-09-10T10:03:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:35 smithi110 ceph-mon[97633]: pgmap v10699: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:35 smithi104 ceph-mon[97044]: pgmap v10699: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:35.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:35.510847+0000) 2022-09-10T10:03:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:35.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:35.991510+0000) 2022-09-10T10:03:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:36.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:36.511133+0000) 2022-09-10T10:03:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:36.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:36.991744+0000) 2022-09-10T10:03:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:37 smithi110 ceph-mon[97633]: pgmap v10700: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:37 smithi104 ceph-mon[97044]: pgmap v10700: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:37.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:37.511386+0000) 2022-09-10T10:03:38.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:37.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:37.992003+0000) 2022-09-10T10:03:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:38.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:38.511619+0000) 2022-09-10T10:03:38.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:03:38.275+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:03:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:38.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:38.992244+0000) 2022-09-10T10:03:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:39 smithi110 ceph-mon[97633]: pgmap v10701: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:39 smithi104 ceph-mon[97044]: pgmap v10701: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:39.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:39.511795+0000) 2022-09-10T10:03:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:39.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:39.992516+0000) 2022-09-10T10:03:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:40.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:40.512036+0000) 2022-09-10T10:03:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:40.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:40.992762+0000) 2022-09-10T10:03:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:41 smithi110 ceph-mon[97633]: pgmap v10702: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:41 smithi104 ceph-mon[97044]: pgmap v10702: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:41.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:41.512240+0000) 2022-09-10T10:03:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:41.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:41.993007+0000) 2022-09-10T10:03:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:42.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:42.512475+0000) 2022-09-10T10:03:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:42.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:42.993229+0000) 2022-09-10T10:03:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:43 smithi110 ceph-mon[97633]: pgmap v10703: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:43 smithi104 ceph-mon[97044]: pgmap v10703: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:43.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:43.512659+0000) 2022-09-10T10:03:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:43.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:43.993482+0000) 2022-09-10T10:03:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:44.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:44.512936+0000) 2022-09-10T10:03:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:44.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:44.993700+0000) 2022-09-10T10:03:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:45 smithi110 ceph-mon[97633]: pgmap v10704: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:03:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:03:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:03:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:03:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:45 smithi104 ceph-mon[97044]: pgmap v10704: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:03:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:03:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:03:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:03:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:45.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:45.513205+0000) 2022-09-10T10:03:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:45.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:45.993929+0000) 2022-09-10T10:03:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:46.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:46.513398+0000) 2022-09-10T10:03:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:46.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:46.994189+0000) 2022-09-10T10:03:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:47 smithi110 ceph-mon[97633]: pgmap v10705: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:47 smithi104 ceph-mon[97044]: pgmap v10705: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:47.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:47.513640+0000) 2022-09-10T10:03:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:47.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:47.994482+0000) 2022-09-10T10:03:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:48.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:48.513864+0000) 2022-09-10T10:03:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:48.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:48.994756+0000) 2022-09-10T10:03:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:49 smithi110 ceph-mon[97633]: pgmap v10706: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:49 smithi104 ceph-mon[97044]: pgmap v10706: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:49.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:49.514018+0000) 2022-09-10T10:03:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:49.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:49.994972+0000) 2022-09-10T10:03:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:50.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:50.514250+0000) 2022-09-10T10:03:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:50.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:50.995158+0000) 2022-09-10T10:03:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:51 smithi110 ceph-mon[97633]: pgmap v10707: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:51 smithi104 ceph-mon[97044]: pgmap v10707: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:51.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:51.514463+0000) 2022-09-10T10:03:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:51.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:51.995330+0000) 2022-09-10T10:03:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:52.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:52.514695+0000) 2022-09-10T10:03:53.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:52.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:52.995506+0000) 2022-09-10T10:03:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:53.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:53.514912+0000) 2022-09-10T10:03:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:03:53.289+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:03:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:53 smithi110 ceph-mon[97633]: pgmap v10708: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:53 smithi104 ceph-mon[97044]: pgmap v10708: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:53.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:53.995787+0000) 2022-09-10T10:03:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:54.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:54.515179+0000) 2022-09-10T10:03:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:54.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:54.996021+0000) 2022-09-10T10:03:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:55 smithi110 ceph-mon[97633]: pgmap v10709: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:55 smithi104 ceph-mon[97044]: pgmap v10709: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:55.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:55.515311+0000) 2022-09-10T10:03:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:55.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:55.996264+0000) 2022-09-10T10:03:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:56.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:56.515560+0000) 2022-09-10T10:03:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:56.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:56.996418+0000) 2022-09-10T10:03:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:57 smithi110 ceph-mon[97633]: pgmap v10710: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:57 smithi104 ceph-mon[97044]: pgmap v10710: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:03:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:57.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:57.515767+0000) 2022-09-10T10:03:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:57.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:57.996570+0000) 2022-09-10T10:03:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:58.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:58.515991+0000) 2022-09-10T10:03:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:58.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:58.996775+0000) 2022-09-10T10:03:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:03:59 smithi110 ceph-mon[97633]: pgmap v10711: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:03:59 smithi104 ceph-mon[97044]: pgmap v10711: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:03:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:03:59.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:59.516225+0000) 2022-09-10T10:04:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:03:59.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:03:59.996939+0000) 2022-09-10T10:04:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:00.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:00.516456+0000) 2022-09-10T10:04:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:00.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:00.997149+0000) 2022-09-10T10:04:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:01 smithi110 ceph-mon[97633]: pgmap v10712: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:01 smithi104 ceph-mon[97044]: pgmap v10712: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:01.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:01.516639+0000) 2022-09-10T10:04:02.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:01.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:01.997349+0000) 2022-09-10T10:04:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:02.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:02.516883+0000) 2022-09-10T10:04:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:02.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:02.997640+0000) 2022-09-10T10:04:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:03 smithi110 ceph-mon[97633]: pgmap v10713: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:03 smithi104 ceph-mon[97044]: pgmap v10713: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:03.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:03.517131+0000) 2022-09-10T10:04:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:03.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:03.997811+0000) 2022-09-10T10:04:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:04.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:04.517304+0000) 2022-09-10T10:04:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:04.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:04.998074+0000) 2022-09-10T10:04:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:05 smithi110 ceph-mon[97633]: pgmap v10714: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:05 smithi104 ceph-mon[97044]: pgmap v10714: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:05.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:05.517462+0000) 2022-09-10T10:04:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:05.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:05.998289+0000) 2022-09-10T10:04:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:06.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:06.517695+0000) 2022-09-10T10:04:07.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:06.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:06.998431+0000) 2022-09-10T10:04:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:07 smithi110 ceph-mon[97633]: pgmap v10715: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:04:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:07 smithi104 ceph-mon[97044]: pgmap v10715: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:04:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:07.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:07.517905+0000) 2022-09-10T10:04:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:07.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:07.998658+0000) 2022-09-10T10:04:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:08.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:08.518152+0000) 2022-09-10T10:04:08.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:04:08.304+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:04:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:08.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:08.998901+0000) 2022-09-10T10:04:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:09 smithi110 ceph-mon[97633]: pgmap v10716: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:04:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:09 smithi110 ceph-mon[97633]: osdmap e750: 12 total, 12 up, 12 in 2022-09-10T10:04:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:09 smithi104 ceph-mon[97044]: pgmap v10716: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:04:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:09 smithi104 ceph-mon[97044]: osdmap e750: 12 total, 12 up, 12 in 2022-09-10T10:04:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:09.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:09.518332+0000) 2022-09-10T10:04:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:09.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:09.999128+0000) 2022-09-10T10:04:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:10.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:10.518552+0000) 2022-09-10T10:04:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:10.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:10.999351+0000) 2022-09-10T10:04:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:11 smithi110 ceph-mon[97633]: pgmap v10718: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:04:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:11 smithi104 ceph-mon[97044]: pgmap v10718: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:04:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:11.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:11.518794+0000) 2022-09-10T10:04:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:11.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:11.999598+0000) 2022-09-10T10:04:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:12.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:12.519021+0000) 2022-09-10T10:04:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:12.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:12.999836+0000) 2022-09-10T10:04:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:13 smithi110 ceph-mon[97633]: pgmap v10719: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:04:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:13 smithi110 ceph-mon[97633]: osdmap e751: 12 total, 12 up, 12 in 2022-09-10T10:04:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:13 smithi104 ceph-mon[97044]: pgmap v10719: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:04:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:13 smithi104 ceph-mon[97044]: osdmap e751: 12 total, 12 up, 12 in 2022-09-10T10:04:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:13.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:13.519260+0000) 2022-09-10T10:04:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:13.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:14.000119+0000) 2022-09-10T10:04:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:14.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:14.519444+0000) 2022-09-10T10:04:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:14.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:15.000357+0000) 2022-09-10T10:04:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:15 smithi110 ceph-mon[97633]: pgmap v10721: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:04:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:15 smithi104 ceph-mon[97044]: pgmap v10721: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:04:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:15.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:15.519653+0000) 2022-09-10T10:04:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:15.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:16.000584+0000) 2022-09-10T10:04:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:16.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:16.519808+0000) 2022-09-10T10:04:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:16.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:17.000771+0000) 2022-09-10T10:04:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:17 smithi110 ceph-mon[97633]: pgmap v10722: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:04:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:17 smithi104 ceph-mon[97044]: pgmap v10722: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:04:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:17.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:17.520008+0000) 2022-09-10T10:04:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:17.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:18.001028+0000) 2022-09-10T10:04:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:18.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:18.520231+0000) 2022-09-10T10:04:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:19.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:19.001282+0000) 2022-09-10T10:04:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:19 smithi110 ceph-mon[97633]: pgmap v10723: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:04:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:19 smithi104 ceph-mon[97044]: pgmap v10723: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:04:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:19.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:19.520459+0000) 2022-09-10T10:04:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:20.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:20.001473+0000) 2022-09-10T10:04:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:20.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:20.520599+0000) 2022-09-10T10:04:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:21.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:21.001659+0000) 2022-09-10T10:04:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:21 smithi110 ceph-mon[97633]: pgmap v10724: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:04:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:21 smithi104 ceph-mon[97044]: pgmap v10724: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:04:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:21.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:21.520777+0000) 2022-09-10T10:04:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:22.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:22.001886+0000) 2022-09-10T10:04:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:22.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:22.521038+0000) 2022-09-10T10:04:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:23.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:23.002078+0000) 2022-09-10T10:04:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:23 smithi110 ceph-mon[97633]: pgmap v10725: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:04:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:23 smithi104 ceph-mon[97044]: pgmap v10725: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:04:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:23.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:23.521216+0000) 2022-09-10T10:04:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:04:23.320+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:04:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:24.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:24.002312+0000) 2022-09-10T10:04:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:24.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:24.521431+0000) 2022-09-10T10:04:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:25.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:25.002511+0000) 2022-09-10T10:04:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:25 smithi110 ceph-mon[97633]: pgmap v10726: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:04:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:25 smithi104 ceph-mon[97044]: pgmap v10726: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:04:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:25.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:25.521629+0000) 2022-09-10T10:04:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:26.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:26.002743+0000) 2022-09-10T10:04:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:26.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:26.521906+0000) 2022-09-10T10:04:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:27.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:27.002936+0000) 2022-09-10T10:04:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:27 smithi110 ceph-mon[97633]: pgmap v10727: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:04:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:27 smithi104 ceph-mon[97044]: pgmap v10727: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:04:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:27.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:27.522151+0000) 2022-09-10T10:04:28.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:28.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:28.003071+0000) 2022-09-10T10:04:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:28.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:28.522395+0000) 2022-09-10T10:04:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:29.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:29.003249+0000) 2022-09-10T10:04:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:29 smithi110 ceph-mon[97633]: pgmap v10728: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:29.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:29 smithi104 ceph-mon[97044]: pgmap v10728: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:29.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:29.522644+0000) 2022-09-10T10:04:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:30.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:30.003422+0000) 2022-09-10T10:04:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:30.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:30.522895+0000) 2022-09-10T10:04:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:31.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:31.003684+0000) 2022-09-10T10:04:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:31 smithi110 ceph-mon[97633]: pgmap v10729: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:31 smithi104 ceph-mon[97044]: pgmap v10729: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:31.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:31.523082+0000) 2022-09-10T10:04:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:32.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:32.003941+0000) 2022-09-10T10:04:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:32.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:32.523345+0000) 2022-09-10T10:04:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:33.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:33.004238+0000) 2022-09-10T10:04:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:33 smithi110 ceph-mon[97633]: pgmap v10730: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:33 smithi104 ceph-mon[97044]: pgmap v10730: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:33.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:33.523545+0000) 2022-09-10T10:04:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:34.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:34.004378+0000) 2022-09-10T10:04:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:34.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:34.523724+0000) 2022-09-10T10:04:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:35.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:35.004547+0000) 2022-09-10T10:04:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:35 smithi110 ceph-mon[97633]: pgmap v10731: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:35 smithi104 ceph-mon[97044]: pgmap v10731: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:35.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:35.523957+0000) 2022-09-10T10:04:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:36.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:36.004776+0000) 2022-09-10T10:04:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:36.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:36.524147+0000) 2022-09-10T10:04:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:37.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:37.004928+0000) 2022-09-10T10:04:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:37 smithi110 ceph-mon[97633]: pgmap v10732: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:37 smithi104 ceph-mon[97044]: pgmap v10732: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:37.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:37.524314+0000) 2022-09-10T10:04:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:38.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:38.005159+0000) 2022-09-10T10:04:38.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:04:38.324+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:04:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:38.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:38.524562+0000) 2022-09-10T10:04:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:39.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:39.005347+0000) 2022-09-10T10:04:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:39 smithi110 ceph-mon[97633]: pgmap v10733: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:39 smithi104 ceph-mon[97044]: pgmap v10733: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:39.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:39.524786+0000) 2022-09-10T10:04:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:40.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:40.005504+0000) 2022-09-10T10:04:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:40.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:40.525038+0000) 2022-09-10T10:04:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:41.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:41.005808+0000) 2022-09-10T10:04:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:41 smithi110 ceph-mon[97633]: pgmap v10734: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:41 smithi104 ceph-mon[97044]: pgmap v10734: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:41.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:41.525186+0000) 2022-09-10T10:04:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:42.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:42.005963+0000) 2022-09-10T10:04:42.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:42.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:42.525396+0000) 2022-09-10T10:04:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:43.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:43.006176+0000) 2022-09-10T10:04:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:43 smithi110 ceph-mon[97633]: pgmap v10735: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:43 smithi104 ceph-mon[97044]: pgmap v10735: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:43.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:43.525529+0000) 2022-09-10T10:04:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:44.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:44.006397+0000) 2022-09-10T10:04:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:44.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:44.525738+0000) 2022-09-10T10:04:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:45.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:45.006668+0000) 2022-09-10T10:04:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:45 smithi110 ceph-mon[97633]: pgmap v10736: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:04:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:04:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:04:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:04:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:45 smithi104 ceph-mon[97044]: pgmap v10736: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:04:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:04:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:04:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:04:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:45.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:45.525939+0000) 2022-09-10T10:04:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:46.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:46.006893+0000) 2022-09-10T10:04:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:46.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:46.526214+0000) 2022-09-10T10:04:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:47.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:47.007168+0000) 2022-09-10T10:04:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:47 smithi110 ceph-mon[97633]: pgmap v10737: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:47 smithi104 ceph-mon[97044]: pgmap v10737: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:47.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:47.526393+0000) 2022-09-10T10:04:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:48.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:48.007396+0000) 2022-09-10T10:04:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:48.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:48.526573+0000) 2022-09-10T10:04:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:49.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:49.007602+0000) 2022-09-10T10:04:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:49 smithi110 ceph-mon[97633]: pgmap v10738: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:49 smithi104 ceph-mon[97044]: pgmap v10738: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:49.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:49.526830+0000) 2022-09-10T10:04:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:50.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:50.007861+0000) 2022-09-10T10:04:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:50.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:50.527082+0000) 2022-09-10T10:04:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:51.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:51.008141+0000) 2022-09-10T10:04:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:51 smithi110 ceph-mon[97633]: pgmap v10739: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:51 smithi104 ceph-mon[97044]: pgmap v10739: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:51.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:51.527233+0000) 2022-09-10T10:04:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:52.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:52.008370+0000) 2022-09-10T10:04:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:52.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:52.527424+0000) 2022-09-10T10:04:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:53.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:53.008624+0000) 2022-09-10T10:04:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:53 smithi110 ceph-mon[97633]: pgmap v10740: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:53 smithi104 ceph-mon[97044]: pgmap v10740: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:53.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:04:53.339+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:04:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:53.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:53.527614+0000) 2022-09-10T10:04:54.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:54.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:54.008853+0000) 2022-09-10T10:04:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:54.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:54.527860+0000) 2022-09-10T10:04:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:55.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:55.009107+0000) 2022-09-10T10:04:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:55 smithi110 ceph-mon[97633]: pgmap v10741: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:55 smithi104 ceph-mon[97044]: pgmap v10741: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:55.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:55.528114+0000) 2022-09-10T10:04:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:56.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:56.009360+0000) 2022-09-10T10:04:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:56.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:56.528324+0000) 2022-09-10T10:04:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:57.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:57.009516+0000) 2022-09-10T10:04:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:57 smithi110 ceph-mon[97633]: pgmap v10742: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:57 smithi104 ceph-mon[97044]: pgmap v10742: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:04:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:57.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:57.528482+0000) 2022-09-10T10:04:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:58.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:58.009742+0000) 2022-09-10T10:04:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:58.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:58.528675+0000) 2022-09-10T10:04:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:04:59.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:59.009991+0000) 2022-09-10T10:04:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:04:59 smithi110 ceph-mon[97633]: pgmap v10743: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:04:59 smithi104 ceph-mon[97044]: pgmap v10743: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:04:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:04:59.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:04:59.528876+0000) 2022-09-10T10:05:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:00.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:00.010162+0000) 2022-09-10T10:05:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:00.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:00.529151+0000) 2022-09-10T10:05:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:01.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:01.010300+0000) 2022-09-10T10:05:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:01 smithi110 ceph-mon[97633]: pgmap v10744: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:01 smithi104 ceph-mon[97044]: pgmap v10744: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:01.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:01.529270+0000) 2022-09-10T10:05:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:02.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:02.010526+0000) 2022-09-10T10:05:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:02.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:02.529446+0000) 2022-09-10T10:05:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:03.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:03.010672+0000) 2022-09-10T10:05:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:03 smithi110 ceph-mon[97633]: pgmap v10745: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:03 smithi104 ceph-mon[97044]: pgmap v10745: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:03.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:03.529617+0000) 2022-09-10T10:05:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:04.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:04.010898+0000) 2022-09-10T10:05:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:04.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:04.529865+0000) 2022-09-10T10:05:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:05.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:05.011161+0000) 2022-09-10T10:05:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:05 smithi110 ceph-mon[97633]: pgmap v10746: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:05 smithi104 ceph-mon[97044]: pgmap v10746: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:05.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:05.530062+0000) 2022-09-10T10:05:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:06.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:06.011348+0000) 2022-09-10T10:05:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:06.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:06.530248+0000) 2022-09-10T10:05:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:07.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:07.011551+0000) 2022-09-10T10:05:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:07 smithi110 ceph-mon[97633]: pgmap v10747: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:05:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:07 smithi104 ceph-mon[97044]: pgmap v10747: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:05:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:07.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:07.530434+0000) 2022-09-10T10:05:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:08.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:08.011804+0000) 2022-09-10T10:05:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:05:08.342+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:05:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:08.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:08.530593+0000) 2022-09-10T10:05:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:09.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:09.012059+0000) 2022-09-10T10:05:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:09 smithi110 ceph-mon[97633]: pgmap v10748: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:05:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:09 smithi110 ceph-mon[97633]: osdmap e752: 12 total, 12 up, 12 in 2022-09-10T10:05:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:09 smithi104 ceph-mon[97044]: pgmap v10748: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:05:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:09 smithi104 ceph-mon[97044]: osdmap e752: 12 total, 12 up, 12 in 2022-09-10T10:05:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:09.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:09.530787+0000) 2022-09-10T10:05:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:10.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:10.012326+0000) 2022-09-10T10:05:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:10.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:10.530933+0000) 2022-09-10T10:05:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:11.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:11.012554+0000) 2022-09-10T10:05:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:11 smithi110 ceph-mon[97633]: pgmap v10750: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:05:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:11 smithi104 ceph-mon[97044]: pgmap v10750: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:05:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:11.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:11.531124+0000) 2022-09-10T10:05:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:12.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:12.012820+0000) 2022-09-10T10:05:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:12.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:12.531270+0000) 2022-09-10T10:05:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:13.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:13.013003+0000) 2022-09-10T10:05:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:13 smithi110 ceph-mon[97633]: pgmap v10751: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:05:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:13 smithi110 ceph-mon[97633]: osdmap e753: 12 total, 12 up, 12 in 2022-09-10T10:05:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:13 smithi104 ceph-mon[97044]: pgmap v10751: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:05:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:13 smithi104 ceph-mon[97044]: osdmap e753: 12 total, 12 up, 12 in 2022-09-10T10:05:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:13.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:13.531424+0000) 2022-09-10T10:05:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:14.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:14.013229+0000) 2022-09-10T10:05:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:14.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:14.531645+0000) 2022-09-10T10:05:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:15.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:15.013484+0000) 2022-09-10T10:05:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:15 smithi110 ceph-mon[97633]: pgmap v10753: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:05:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:15 smithi104 ceph-mon[97044]: pgmap v10753: 129 pgs: 129 active+clean; 8.9 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:05:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:15.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:15.531904+0000) 2022-09-10T10:05:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:16.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:16.013653+0000) 2022-09-10T10:05:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:16.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:16.532163+0000) 2022-09-10T10:05:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:17.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:17.013860+0000) 2022-09-10T10:05:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:17 smithi110 ceph-mon[97633]: pgmap v10754: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:05:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:17 smithi104 ceph-mon[97044]: pgmap v10754: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:05:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:17.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:17.532400+0000) 2022-09-10T10:05:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:18.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:18.014089+0000) 2022-09-10T10:05:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:18.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:18.532645+0000) 2022-09-10T10:05:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:19.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:19.014363+0000) 2022-09-10T10:05:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:19 smithi110 ceph-mon[97633]: pgmap v10755: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:05:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:19 smithi104 ceph-mon[97044]: pgmap v10755: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:05:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:19.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:19.532838+0000) 2022-09-10T10:05:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:20.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:20.014603+0000) 2022-09-10T10:05:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:20.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:20.533034+0000) 2022-09-10T10:05:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:21.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:21.014837+0000) 2022-09-10T10:05:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:21 smithi110 ceph-mon[97633]: pgmap v10756: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:05:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:21 smithi104 ceph-mon[97044]: pgmap v10756: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:05:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:21.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:21.533218+0000) 2022-09-10T10:05:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:22.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:22.015067+0000) 2022-09-10T10:05:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:22.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:22.533437+0000) 2022-09-10T10:05:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:23.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:23.015270+0000) 2022-09-10T10:05:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:23 smithi110 ceph-mon[97633]: pgmap v10757: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:05:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:23 smithi104 ceph-mon[97044]: pgmap v10757: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:05:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:23.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:23.533620+0000) 2022-09-10T10:05:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:05:23.358+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:05:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:24.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:24.015493+0000) 2022-09-10T10:05:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:24.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:24.533839+0000) 2022-09-10T10:05:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:25.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:25.015686+0000) 2022-09-10T10:05:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:25 smithi110 ceph-mon[97633]: pgmap v10758: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:05:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:25 smithi104 ceph-mon[97044]: pgmap v10758: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:05:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:25.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:25.534005+0000) 2022-09-10T10:05:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:26.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:26.015878+0000) 2022-09-10T10:05:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:26.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:26.534276+0000) 2022-09-10T10:05:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:27.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:27.016138+0000) 2022-09-10T10:05:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:27 smithi110 ceph-mon[97633]: pgmap v10759: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:05:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:27 smithi104 ceph-mon[97044]: pgmap v10759: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:05:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:27.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:27.534515+0000) 2022-09-10T10:05:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:28.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:28.016336+0000) 2022-09-10T10:05:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:28.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:28.534709+0000) 2022-09-10T10:05:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:29.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:29.016561+0000) 2022-09-10T10:05:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:29 smithi110 ceph-mon[97633]: pgmap v10760: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:29 smithi104 ceph-mon[97044]: pgmap v10760: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:29.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:29.534870+0000) 2022-09-10T10:05:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:30.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:30.016805+0000) 2022-09-10T10:05:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:30.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:30.535089+0000) 2022-09-10T10:05:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:31.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:31.017089+0000) 2022-09-10T10:05:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:31 smithi110 ceph-mon[97633]: pgmap v10761: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:31 smithi104 ceph-mon[97044]: pgmap v10761: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:31.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:31.535306+0000) 2022-09-10T10:05:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:32.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:32.017321+0000) 2022-09-10T10:05:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:32.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:32.535528+0000) 2022-09-10T10:05:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:33.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:33.017539+0000) 2022-09-10T10:05:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:33 smithi110 ceph-mon[97633]: pgmap v10762: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:33 smithi104 ceph-mon[97044]: pgmap v10762: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:33.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:33.535785+0000) 2022-09-10T10:05:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:34.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:34.017801+0000) 2022-09-10T10:05:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:34.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:34.536037+0000) 2022-09-10T10:05:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:35.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:35.018026+0000) 2022-09-10T10:05:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:35 smithi110 ceph-mon[97633]: pgmap v10763: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:35 smithi104 ceph-mon[97044]: pgmap v10763: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:35.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:35.536322+0000) 2022-09-10T10:05:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:36.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:36.018299+0000) 2022-09-10T10:05:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:36.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:36.536542+0000) 2022-09-10T10:05:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:37.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:37.018514+0000) 2022-09-10T10:05:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:37 smithi110 ceph-mon[97633]: pgmap v10764: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:37 smithi104 ceph-mon[97044]: pgmap v10764: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:37.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:37.536698+0000) 2022-09-10T10:05:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:38.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:38.018761+0000) 2022-09-10T10:05:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:38.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:38.536901+0000) 2022-09-10T10:05:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:05:38.370+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:05:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:39.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:39.018962+0000) 2022-09-10T10:05:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:39 smithi110 ceph-mon[97633]: pgmap v10765: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:39 smithi104 ceph-mon[97044]: pgmap v10765: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:39.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:39.537185+0000) 2022-09-10T10:05:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:40.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:40.019195+0000) 2022-09-10T10:05:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:40.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:40.537441+0000) 2022-09-10T10:05:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:41.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:41.019355+0000) 2022-09-10T10:05:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:41 smithi110 ceph-mon[97633]: pgmap v10766: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:41 smithi104 ceph-mon[97044]: pgmap v10766: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:41.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:41.537619+0000) 2022-09-10T10:05:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:42.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:42.019576+0000) 2022-09-10T10:05:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:42.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:42.537840+0000) 2022-09-10T10:05:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:43.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:43.019798+0000) 2022-09-10T10:05:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:43 smithi110 ceph-mon[97633]: pgmap v10767: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:43 smithi104 ceph-mon[97044]: pgmap v10767: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:43.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:43.538087+0000) 2022-09-10T10:05:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:44.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:44.020038+0000) 2022-09-10T10:05:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:44.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:44.538345+0000) 2022-09-10T10:05:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:45.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:45.020201+0000) 2022-09-10T10:05:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:45 smithi110 ceph-mon[97633]: pgmap v10768: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:05:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:05:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:05:45.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:05:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:45 smithi104 ceph-mon[97044]: pgmap v10768: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:05:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:05:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:05:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:05:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:45.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:45.538608+0000) 2022-09-10T10:05:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:46.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:46.020438+0000) 2022-09-10T10:05:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:46.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:46.538852+0000) 2022-09-10T10:05:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:47.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:47.020617+0000) 2022-09-10T10:05:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:47 smithi110 ceph-mon[97633]: pgmap v10769: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:47 smithi104 ceph-mon[97044]: pgmap v10769: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:47.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:47.539055+0000) 2022-09-10T10:05:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:48.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:48.020858+0000) 2022-09-10T10:05:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:48.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:48.539313+0000) 2022-09-10T10:05:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:49.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:49.021154+0000) 2022-09-10T10:05:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:49 smithi110 ceph-mon[97633]: pgmap v10770: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:49 smithi104 ceph-mon[97044]: pgmap v10770: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:05:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:49.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:49.539547+0000) 2022-09-10T10:05:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:50.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:50.021320+0000) 2022-09-10T10:05:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:50.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:50.539721+0000) 2022-09-10T10:05:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:51.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:51.021560+0000) 2022-09-10T10:05:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:51 smithi110 ceph-mon[97633]: pgmap v10771: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:51 smithi104 ceph-mon[97044]: pgmap v10771: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:51.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:51.539970+0000) 2022-09-10T10:05:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:52.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:52.021826+0000) 2022-09-10T10:05:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:52.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:52.540192+0000) 2022-09-10T10:05:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:53.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:53.022081+0000) 2022-09-10T10:05:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:53 smithi110 ceph-mon[97633]: pgmap v10772: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:05:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:53 smithi104 ceph-mon[97044]: pgmap v10772: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:05:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:53.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:53.540360+0000) 2022-09-10T10:05:53.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:05:53.386+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:05:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:54.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:54.022301+0000) 2022-09-10T10:05:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:54.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:54.540562+0000) 2022-09-10T10:05:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:55.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:55.022549+0000) 2022-09-10T10:05:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:55 smithi110 ceph-mon[97633]: pgmap v10773: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:05:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:55 smithi104 ceph-mon[97044]: pgmap v10773: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:05:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:55.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:55.540741+0000) 2022-09-10T10:05:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:56.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:56.022775+0000) 2022-09-10T10:05:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:56.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:56.540929+0000) 2022-09-10T10:05:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:57.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:57.023009+0000) 2022-09-10T10:05:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:57 smithi110 ceph-mon[97633]: pgmap v10774: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:57 smithi104 ceph-mon[97044]: pgmap v10774: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:05:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:57.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:57.541141+0000) 2022-09-10T10:05:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:58.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:58.023323+0000) 2022-09-10T10:05:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:58.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:58.541310+0000) 2022-09-10T10:05:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:05:59.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:59.023495+0000) 2022-09-10T10:05:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:05:59 smithi110 ceph-mon[97633]: pgmap v10775: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:05:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:05:59 smithi104 ceph-mon[97044]: pgmap v10775: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:05:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:05:59.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:05:59.541507+0000) 2022-09-10T10:06:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:00.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:00.023745+0000) 2022-09-10T10:06:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:00.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:00.541686+0000) 2022-09-10T10:06:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:01.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:01.023932+0000) 2022-09-10T10:06:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:01 smithi110 ceph-mon[97633]: pgmap v10776: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:01 smithi104 ceph-mon[97044]: pgmap v10776: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:01.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:01.541889+0000) 2022-09-10T10:06:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:02.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:02.024091+0000) 2022-09-10T10:06:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:02.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:02.542082+0000) 2022-09-10T10:06:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:03.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:03.024331+0000) 2022-09-10T10:06:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:03 smithi110 ceph-mon[97633]: pgmap v10777: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:03 smithi104 ceph-mon[97044]: pgmap v10777: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:03.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:03.542277+0000) 2022-09-10T10:06:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:04.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:04.024564+0000) 2022-09-10T10:06:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:04.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:04.542561+0000) 2022-09-10T10:06:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:05.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:05.024791+0000) 2022-09-10T10:06:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:05 smithi104 ceph-mon[97044]: pgmap v10778: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:05.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:05.542757+0000) 2022-09-10T10:06:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:05 smithi110 ceph-mon[97633]: pgmap v10778: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:06.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:06.024990+0000) 2022-09-10T10:06:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:06.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:06.542993+0000) 2022-09-10T10:06:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:07.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:07.025176+0000) 2022-09-10T10:06:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:07 smithi104 ceph-mon[97044]: pgmap v10779: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:06:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:07.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:07.543187+0000) 2022-09-10T10:06:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:07 smithi110 ceph-mon[97633]: pgmap v10779: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:06:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:08.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:08.025363+0000) 2022-09-10T10:06:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:06:08.388+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:06:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:08.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:08.543371+0000) 2022-09-10T10:06:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:09.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:09.025517+0000) 2022-09-10T10:06:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:09 smithi104 ceph-mon[97044]: pgmap v10780: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:06:09.857 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:09 smithi104 ceph-mon[97044]: osdmap e754: 12 total, 12 up, 12 in 2022-09-10T10:06:09.859 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:09.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:09.543628+0000) 2022-09-10T10:06:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:09 smithi110 ceph-mon[97633]: pgmap v10780: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:06:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:09 smithi110 ceph-mon[97633]: osdmap e754: 12 total, 12 up, 12 in 2022-09-10T10:06:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:10.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:10.025753+0000) 2022-09-10T10:06:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:10.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:10.543829+0000) 2022-09-10T10:06:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:11.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:11.025931+0000) 2022-09-10T10:06:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:11 smithi104 ceph-mon[97044]: pgmap v10782: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:06:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:11.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:11.544049+0000) 2022-09-10T10:06:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:11 smithi110 ceph-mon[97633]: pgmap v10782: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:06:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:12.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:12.026179+0000) 2022-09-10T10:06:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:12.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:12.544244+0000) 2022-09-10T10:06:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:13.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:13.026320+0000) 2022-09-10T10:06:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:13 smithi110 ceph-mon[97633]: pgmap v10783: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:06:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:13 smithi110 ceph-mon[97633]: osdmap e755: 12 total, 12 up, 12 in 2022-09-10T10:06:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:13 smithi104 ceph-mon[97044]: pgmap v10783: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:06:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:13 smithi104 ceph-mon[97044]: osdmap e755: 12 total, 12 up, 12 in 2022-09-10T10:06:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:13.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:13.544460+0000) 2022-09-10T10:06:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:14.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:14.026531+0000) 2022-09-10T10:06:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:14.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:14.544700+0000) 2022-09-10T10:06:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:15.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:15.026748+0000) 2022-09-10T10:06:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:15 smithi110 ceph-mon[97633]: pgmap v10785: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:06:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:15 smithi104 ceph-mon[97044]: pgmap v10785: 129 pgs: 129 active+clean; 8.9 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:06:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:15.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:15.544945+0000) 2022-09-10T10:06:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:16.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:16.026923+0000) 2022-09-10T10:06:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:16.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:16.545210+0000) 2022-09-10T10:06:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:17.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:17.027183+0000) 2022-09-10T10:06:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:17 smithi110 ceph-mon[97633]: pgmap v10786: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:06:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:17 smithi104 ceph-mon[97044]: pgmap v10786: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:06:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:17.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:17.545501+0000) 2022-09-10T10:06:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:18.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:18.027408+0000) 2022-09-10T10:06:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:18.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:18.545739+0000) 2022-09-10T10:06:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:19.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:19.027592+0000) 2022-09-10T10:06:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:19 smithi110 ceph-mon[97633]: pgmap v10787: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:06:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:19 smithi104 ceph-mon[97044]: pgmap v10787: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:06:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:19.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:19.545943+0000) 2022-09-10T10:06:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:20.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:20.027821+0000) 2022-09-10T10:06:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:20.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:20.546202+0000) 2022-09-10T10:06:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:21.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:21.028016+0000) 2022-09-10T10:06:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:21 smithi110 ceph-mon[97633]: pgmap v10788: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:06:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:21 smithi104 ceph-mon[97044]: pgmap v10788: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:06:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:21.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:21.546372+0000) 2022-09-10T10:06:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:22.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:22.028234+0000) 2022-09-10T10:06:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:22.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:22.546617+0000) 2022-09-10T10:06:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:23.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:23.028368+0000) 2022-09-10T10:06:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:23 smithi110 ceph-mon[97633]: pgmap v10789: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:06:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:23 smithi104 ceph-mon[97044]: pgmap v10789: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:06:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:23.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:23.546840+0000) 2022-09-10T10:06:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:06:23.398+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:06:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:24.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:24.028544+0000) 2022-09-10T10:06:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:24.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:24.547043+0000) 2022-09-10T10:06:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:25.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:25.028788+0000) 2022-09-10T10:06:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:25 smithi110 ceph-mon[97633]: pgmap v10790: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:06:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:25 smithi104 ceph-mon[97044]: pgmap v10790: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:06:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:25.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:25.547159+0000) 2022-09-10T10:06:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:26.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:26.029048+0000) 2022-09-10T10:06:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:26.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:26.547333+0000) 2022-09-10T10:06:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:27.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:27.029242+0000) 2022-09-10T10:06:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:27 smithi110 ceph-mon[97633]: pgmap v10791: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:06:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:27 smithi104 ceph-mon[97044]: pgmap v10791: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:06:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:27.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:27.547493+0000) 2022-09-10T10:06:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:28.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:28.029424+0000) 2022-09-10T10:06:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:28.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:28.547714+0000) 2022-09-10T10:06:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:29.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:29.029581+0000) 2022-09-10T10:06:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:29 smithi110 ceph-mon[97633]: pgmap v10792: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:29 smithi104 ceph-mon[97044]: pgmap v10792: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:29.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:29.547938+0000) 2022-09-10T10:06:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:30.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:30.029823+0000) 2022-09-10T10:06:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:30.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:30.548235+0000) 2022-09-10T10:06:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:31.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:31.030054+0000) 2022-09-10T10:06:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:31 smithi110 ceph-mon[97633]: pgmap v10793: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:31 smithi104 ceph-mon[97044]: pgmap v10793: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:31.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:31.548458+0000) 2022-09-10T10:06:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:32.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:32.030308+0000) 2022-09-10T10:06:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:32.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:32.548720+0000) 2022-09-10T10:06:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:33.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:33.030526+0000) 2022-09-10T10:06:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:33 smithi110 ceph-mon[97633]: pgmap v10794: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:33 smithi104 ceph-mon[97044]: pgmap v10794: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:33.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:33.548945+0000) 2022-09-10T10:06:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:34.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:34.030743+0000) 2022-09-10T10:06:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:34.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:34.549139+0000) 2022-09-10T10:06:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:35.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:35.030962+0000) 2022-09-10T10:06:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:35 smithi110 ceph-mon[97633]: pgmap v10795: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:35 smithi104 ceph-mon[97044]: pgmap v10795: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:35.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:35.549328+0000) 2022-09-10T10:06:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:36.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:36.031205+0000) 2022-09-10T10:06:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:36.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:36.549559+0000) 2022-09-10T10:06:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:37.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:37.031415+0000) 2022-09-10T10:06:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:37 smithi110 ceph-mon[97633]: pgmap v10796: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:37 smithi104 ceph-mon[97044]: pgmap v10796: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:37.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:37.549744+0000) 2022-09-10T10:06:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:38.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:38.031603+0000) 2022-09-10T10:06:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:38.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:38.549891+0000) 2022-09-10T10:06:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:06:38.411+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:06:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:39.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:39.031842+0000) 2022-09-10T10:06:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:39 smithi110 ceph-mon[97633]: pgmap v10797: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:39 smithi104 ceph-mon[97044]: pgmap v10797: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:39.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:39.550085+0000) 2022-09-10T10:06:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:40.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:40.032021+0000) 2022-09-10T10:06:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:40.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:40.550334+0000) 2022-09-10T10:06:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:41.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:41.032263+0000) 2022-09-10T10:06:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:41 smithi110 ceph-mon[97633]: pgmap v10798: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:41 smithi104 ceph-mon[97044]: pgmap v10798: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:41.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:41.550585+0000) 2022-09-10T10:06:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:42.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:42.032459+0000) 2022-09-10T10:06:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:42.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:42.550809+0000) 2022-09-10T10:06:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:43.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:43.032695+0000) 2022-09-10T10:06:43.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:43 smithi110 ceph-mon[97633]: pgmap v10799: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:43 smithi104 ceph-mon[97044]: pgmap v10799: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:43.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:43.550980+0000) 2022-09-10T10:06:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:44.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:44.032923+0000) 2022-09-10T10:06:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:44.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:44.551222+0000) 2022-09-10T10:06:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:45.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:45.033181+0000) 2022-09-10T10:06:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:45 smithi110 ceph-mon[97633]: pgmap v10800: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:06:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:06:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:06:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:06:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:45 smithi104 ceph-mon[97044]: pgmap v10800: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:06:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:06:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:06:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:06:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:45.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:45.551503+0000) 2022-09-10T10:06:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:46.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:46.033390+0000) 2022-09-10T10:06:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:46.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:46.551727+0000) 2022-09-10T10:06:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:47.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:47.033584+0000) 2022-09-10T10:06:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:47 smithi110 ceph-mon[97633]: pgmap v10801: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:47 smithi104 ceph-mon[97044]: pgmap v10801: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:06:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:47.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:47.551961+0000) 2022-09-10T10:06:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:48.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:48.033775+0000) 2022-09-10T10:06:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:48.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:48.552249+0000) 2022-09-10T10:06:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:49.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:49.033954+0000) 2022-09-10T10:06:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:49 smithi110 ceph-mon[97633]: pgmap v10802: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:49.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:49 smithi104 ceph-mon[97044]: pgmap v10802: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:06:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:49.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:49.552439+0000) 2022-09-10T10:06:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:50.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:50.034165+0000) 2022-09-10T10:06:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:50.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:50.552640+0000) 2022-09-10T10:06:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:51.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:51.034335+0000) 2022-09-10T10:06:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:51 smithi110 ceph-mon[97633]: pgmap v10803: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T10:06:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:51 smithi104 ceph-mon[97044]: pgmap v10803: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T10:06:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:51.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:51.552870+0000) 2022-09-10T10:06:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:52.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:52.034506+0000) 2022-09-10T10:06:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:52.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:52.553085+0000) 2022-09-10T10:06:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:53.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:53.034629+0000) 2022-09-10T10:06:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:53 smithi104 ceph-mon[97044]: pgmap v10804: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T10:06:53.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:06:53.427+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:06:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:53.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:53.553400+0000) 2022-09-10T10:06:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:53 smithi110 ceph-mon[97633]: pgmap v10804: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T10:06:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:54.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:54.034804+0000) 2022-09-10T10:06:55.036 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:54.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:54.553740+0000) 2022-09-10T10:06:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:55.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:55.034975+0000) 2022-09-10T10:06:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:55 smithi104 ceph-mon[97044]: pgmap v10805: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T10:06:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:55 smithi110 ceph-mon[97633]: pgmap v10805: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T10:06:56.036 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:55.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:55.553950+0000) 2022-09-10T10:06:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:56.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:56.035171+0000) 2022-09-10T10:06:57.036 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:56.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:56.554205+0000) 2022-09-10T10:06:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:57.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:57.035333+0000) 2022-09-10T10:06:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:57 smithi104 ceph-mon[97044]: pgmap v10806: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T10:06:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:57 smithi110 ceph-mon[97633]: pgmap v10806: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T10:06:58.036 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:57.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:57.554397+0000) 2022-09-10T10:06:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:58.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:58.035551+0000) 2022-09-10T10:06:59.036 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:58.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:58.554607+0000) 2022-09-10T10:06:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:06:59.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:59.035789+0000) 2022-09-10T10:06:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:06:59 smithi104 ceph-mon[97044]: pgmap v10807: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T10:06:59.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:06:59 smithi110 ceph-mon[97633]: pgmap v10807: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T10:07:00.038 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:06:59.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:06:59.554820+0000) 2022-09-10T10:07:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:00.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:00.036034+0000) 2022-09-10T10:07:01.037 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:00.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:00.555082+0000) 2022-09-10T10:07:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:01.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:01.036182+0000) 2022-09-10T10:07:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:01 smithi104 ceph-mon[97044]: pgmap v10808: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T10:07:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:01 smithi110 ceph-mon[97633]: pgmap v10808: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T10:07:02.037 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:01.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:01.555277+0000) 2022-09-10T10:07:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:02.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:02.036419+0000) 2022-09-10T10:07:03.037 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:02.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:02.555455+0000) 2022-09-10T10:07:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:03.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:03.036599+0000) 2022-09-10T10:07:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:03 smithi104 ceph-mon[97044]: pgmap v10809: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:03.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:03 smithi110 ceph-mon[97633]: pgmap v10809: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:04.037 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:03.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:03.555695+0000) 2022-09-10T10:07:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:04.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:04.036810+0000) 2022-09-10T10:07:05.038 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:04.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:04.555897+0000) 2022-09-10T10:07:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:05.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:05.036975+0000) 2022-09-10T10:07:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:05 smithi104 ceph-mon[97044]: pgmap v10810: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:05 smithi110 ceph-mon[97633]: pgmap v10810: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:06.038 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:05.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:05.556170+0000) 2022-09-10T10:07:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:06.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:06.037194+0000) 2022-09-10T10:07:07.038 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:06.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:06.556327+0000) 2022-09-10T10:07:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:07.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:07.037363+0000) 2022-09-10T10:07:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:07 smithi104 ceph-mon[97044]: pgmap v10811: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:07:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:07 smithi110 ceph-mon[97633]: pgmap v10811: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:07:08.038 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:07.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:07.556539+0000) 2022-09-10T10:07:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:08.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:08.037570+0000) 2022-09-10T10:07:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:08.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:08.556705+0000) 2022-09-10T10:07:08.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:07:08.442+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:07:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:09.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:09.037732+0000) 2022-09-10T10:07:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:09 smithi104 ceph-mon[97044]: pgmap v10812: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:07:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:09 smithi104 ceph-mon[97044]: osdmap e756: 12 total, 12 up, 12 in 2022-09-10T10:07:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:09 smithi110 ceph-mon[97633]: pgmap v10812: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:07:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:09 smithi110 ceph-mon[97633]: osdmap e756: 12 total, 12 up, 12 in 2022-09-10T10:07:10.038 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:09.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:09.556878+0000) 2022-09-10T10:07:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:10.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:10.037928+0000) 2022-09-10T10:07:11.039 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:10.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:10.557169+0000) 2022-09-10T10:07:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:11.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:11.038200+0000) 2022-09-10T10:07:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:11 smithi104 ceph-mon[97044]: pgmap v10814: 129 pgs: 129 active+clean; 8.9 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:07:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:11 smithi110 ceph-mon[97633]: pgmap v10814: 129 pgs: 129 active+clean; 8.9 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:07:12.043 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:11.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:11.557349+0000) 2022-09-10T10:07:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:12.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:12.038427+0000) 2022-09-10T10:07:13.039 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:12.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:12.557602+0000) 2022-09-10T10:07:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:13.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:13.038648+0000) 2022-09-10T10:07:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:13 smithi110 ceph-mon[97633]: pgmap v10815: 129 pgs: 129 active+clean; 8.9 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:07:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:13 smithi110 ceph-mon[97633]: osdmap e757: 12 total, 12 up, 12 in 2022-09-10T10:07:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:13 smithi104 ceph-mon[97044]: pgmap v10815: 129 pgs: 129 active+clean; 8.9 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:07:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:13 smithi104 ceph-mon[97044]: osdmap e757: 12 total, 12 up, 12 in 2022-09-10T10:07:14.039 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:13.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:13.557831+0000) 2022-09-10T10:07:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:14.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:14.038816+0000) 2022-09-10T10:07:15.039 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:14.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:14.558029+0000) 2022-09-10T10:07:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:15.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:15.039012+0000) 2022-09-10T10:07:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:15 smithi110 ceph-mon[97633]: pgmap v10817: 129 pgs: 129 active+clean; 8.9 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:07:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:15 smithi104 ceph-mon[97044]: pgmap v10817: 129 pgs: 129 active+clean; 8.9 MiB data, 997 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:07:16.040 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:15.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:15.558199+0000) 2022-09-10T10:07:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:16.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:16.039232+0000) 2022-09-10T10:07:17.040 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:16.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:16.558431+0000) 2022-09-10T10:07:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:17.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:17.039501+0000) 2022-09-10T10:07:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:17 smithi110 ceph-mon[97633]: pgmap v10818: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:07:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:17 smithi104 ceph-mon[97044]: pgmap v10818: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:07:18.040 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:17.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:17.558665+0000) 2022-09-10T10:07:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:18.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:18.039738+0000) 2022-09-10T10:07:19.041 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:18.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:18.558919+0000) 2022-09-10T10:07:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:19.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:19.039946+0000) 2022-09-10T10:07:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:19 smithi110 ceph-mon[97633]: pgmap v10819: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:07:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:19 smithi104 ceph-mon[97044]: pgmap v10819: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:07:20.041 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:19.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:19.559167+0000) 2022-09-10T10:07:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:20.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:20.040077+0000) 2022-09-10T10:07:21.041 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:20.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:20.559328+0000) 2022-09-10T10:07:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:21.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:21.040329+0000) 2022-09-10T10:07:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:21 smithi110 ceph-mon[97633]: pgmap v10820: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:07:21.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:21 smithi104 ceph-mon[97044]: pgmap v10820: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:07:22.041 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:21.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:21.559565+0000) 2022-09-10T10:07:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:22.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:22.040617+0000) 2022-09-10T10:07:23.042 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:22.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:22.559750+0000) 2022-09-10T10:07:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:23.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:23.040790+0000) 2022-09-10T10:07:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:23 smithi110 ceph-mon[97633]: pgmap v10821: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:07:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:23 smithi104 ceph-mon[97044]: pgmap v10821: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:07:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:23.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:23.559949+0000) 2022-09-10T10:07:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:07:23.447+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:07:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:24.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:24.041031+0000) 2022-09-10T10:07:25.042 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:24.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:24.560187+0000) 2022-09-10T10:07:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:25.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:25.041205+0000) 2022-09-10T10:07:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:25 smithi110 ceph-mon[97633]: pgmap v10822: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:07:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:25 smithi104 ceph-mon[97044]: pgmap v10822: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:07:26.042 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:25.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:25.560365+0000) 2022-09-10T10:07:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:26.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:26.041444+0000) 2022-09-10T10:07:27.042 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:26.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:26.560641+0000) 2022-09-10T10:07:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:27.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:27.041608+0000) 2022-09-10T10:07:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:27 smithi110 ceph-mon[97633]: pgmap v10823: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:07:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:27 smithi104 ceph-mon[97044]: pgmap v10823: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:07:28.042 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:27.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:27.560818+0000) 2022-09-10T10:07:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:28.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:28.041805+0000) 2022-09-10T10:07:29.043 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:28.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:28.560999+0000) 2022-09-10T10:07:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:29.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:29.042028+0000) 2022-09-10T10:07:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:29 smithi110 ceph-mon[97633]: pgmap v10824: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:29 smithi104 ceph-mon[97044]: pgmap v10824: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:30.043 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:29.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:29.561241+0000) 2022-09-10T10:07:30.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:30.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:30.042278+0000) 2022-09-10T10:07:31.043 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:30.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:30.561397+0000) 2022-09-10T10:07:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:31.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:31.042540+0000) 2022-09-10T10:07:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:31 smithi110 ceph-mon[97633]: pgmap v10825: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:31 smithi104 ceph-mon[97044]: pgmap v10825: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:32.043 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:31.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:31.561589+0000) 2022-09-10T10:07:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:32.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:32.042801+0000) 2022-09-10T10:07:33.044 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:32.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:32.561764+0000) 2022-09-10T10:07:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:33.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:33.043055+0000) 2022-09-10T10:07:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:33 smithi110 ceph-mon[97633]: pgmap v10826: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:33 smithi104 ceph-mon[97044]: pgmap v10826: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:34.044 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:33.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:33.561889+0000) 2022-09-10T10:07:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:34.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:34.043342+0000) 2022-09-10T10:07:35.044 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:34.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:34.562153+0000) 2022-09-10T10:07:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:35.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:35.043527+0000) 2022-09-10T10:07:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:35 smithi110 ceph-mon[97633]: pgmap v10827: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:35 smithi104 ceph-mon[97044]: pgmap v10827: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:36.044 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:35.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:35.562292+0000) 2022-09-10T10:07:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:36.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:36.043775+0000) 2022-09-10T10:07:37.045 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:36.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:36.562542+0000) 2022-09-10T10:07:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:37.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:37.043899+0000) 2022-09-10T10:07:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:37 smithi110 ceph-mon[97633]: pgmap v10828: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:37 smithi104 ceph-mon[97044]: pgmap v10828: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:38.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:37.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:37.562789+0000) 2022-09-10T10:07:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:38.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:38.044140+0000) 2022-09-10T10:07:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:38.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:38.562941+0000) 2022-09-10T10:07:38.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:07:38.462+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:07:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:39.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:39.044327+0000) 2022-09-10T10:07:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:39 smithi110 ceph-mon[97633]: pgmap v10829: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:39.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:39 smithi104 ceph-mon[97044]: pgmap v10829: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:40.045 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:39.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:39.563171+0000) 2022-09-10T10:07:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:40.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:40.044497+0000) 2022-09-10T10:07:41.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:40.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:40.563374+0000) 2022-09-10T10:07:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:41.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:41.044729+0000) 2022-09-10T10:07:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:41 smithi110 ceph-mon[97633]: pgmap v10830: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:41 smithi104 ceph-mon[97044]: pgmap v10830: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:42.045 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:41.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:41.563605+0000) 2022-09-10T10:07:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:42.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:42.044917+0000) 2022-09-10T10:07:43.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:42.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:42.563791+0000) 2022-09-10T10:07:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:43.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:43.045172+0000) 2022-09-10T10:07:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:43 smithi110 ceph-mon[97633]: pgmap v10831: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:43.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:43 smithi104 ceph-mon[97044]: pgmap v10831: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:44.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:43.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:43.563974+0000) 2022-09-10T10:07:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:44.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:44.045389+0000) 2022-09-10T10:07:45.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:44.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:44.564178+0000) 2022-09-10T10:07:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:45.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:45.045593+0000) 2022-09-10T10:07:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:45 smithi104 ceph-mon[97044]: pgmap v10832: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:07:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:07:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:07:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:07:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:45 smithi110 ceph-mon[97633]: pgmap v10832: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:07:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:07:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:07:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:07:46.046 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:45.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:45.564341+0000) 2022-09-10T10:07:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:46.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:46.045836+0000) 2022-09-10T10:07:47.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:46.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:46.564539+0000) 2022-09-10T10:07:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:47.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:47.046043+0000) 2022-09-10T10:07:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:47 smithi104 ceph-mon[97044]: pgmap v10833: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:47 smithi110 ceph-mon[97633]: pgmap v10833: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:48.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:47.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:47.564674+0000) 2022-09-10T10:07:48.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:48.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:48.046197+0000) 2022-09-10T10:07:49.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:48.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:48.564919+0000) 2022-09-10T10:07:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:49.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:49.046398+0000) 2022-09-10T10:07:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:49 smithi104 ceph-mon[97044]: pgmap v10834: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:49 smithi110 ceph-mon[97633]: pgmap v10834: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:50.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:49.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:49.565172+0000) 2022-09-10T10:07:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:50.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:50.046600+0000) 2022-09-10T10:07:51.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:50.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:50.565373+0000) 2022-09-10T10:07:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:51.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:51.046835+0000) 2022-09-10T10:07:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:51 smithi104 ceph-mon[97044]: pgmap v10835: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:51 smithi110 ceph-mon[97633]: pgmap v10835: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:52.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:51.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:51.565555+0000) 2022-09-10T10:07:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:52.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:52.047010+0000) 2022-09-10T10:07:53.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:52.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:52.565694+0000) 2022-09-10T10:07:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:53.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:53.047219+0000) 2022-09-10T10:07:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:53 smithi104 ceph-mon[97044]: pgmap v10836: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:53.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:07:53.475+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:07:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:53.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:53.565906+0000) 2022-09-10T10:07:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:53 smithi110 ceph-mon[97633]: pgmap v10836: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:54.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:54.047443+0000) 2022-09-10T10:07:55.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:54.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:54.566165+0000) 2022-09-10T10:07:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:55.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:55.047639+0000) 2022-09-10T10:07:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:55 smithi104 ceph-mon[97044]: pgmap v10837: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:55 smithi110 ceph-mon[97633]: pgmap v10837: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:56.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:55.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:55.566380+0000) 2022-09-10T10:07:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:56.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:56.047860+0000) 2022-09-10T10:07:57.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:56.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:56.566654+0000) 2022-09-10T10:07:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:57.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:57.048081+0000) 2022-09-10T10:07:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:57 smithi104 ceph-mon[97044]: pgmap v10838: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:57 smithi110 ceph-mon[97633]: pgmap v10838: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:07:58.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:57.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:57.566910+0000) 2022-09-10T10:07:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:58.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:58.048245+0000) 2022-09-10T10:07:59.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:58.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:58.567171+0000) 2022-09-10T10:07:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:07:59.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:59.048493+0000) 2022-09-10T10:07:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:07:59 smithi104 ceph-mon[97044]: pgmap v10839: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:07:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:07:59 smithi110 ceph-mon[97633]: pgmap v10839: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:00.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:07:59.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:07:59.567381+0000) 2022-09-10T10:08:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:00.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:00.048736+0000) 2022-09-10T10:08:01.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:00.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:00.567540+0000) 2022-09-10T10:08:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:01.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:01.048938+0000) 2022-09-10T10:08:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:01 smithi104 ceph-mon[97044]: pgmap v10840: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:01 smithi110 ceph-mon[97633]: pgmap v10840: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:02.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:01.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:01.567806+0000) 2022-09-10T10:08:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:02.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:02.049130+0000) 2022-09-10T10:08:03.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:02.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:02.568049+0000) 2022-09-10T10:08:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:03.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:03.049419+0000) 2022-09-10T10:08:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:03 smithi104 ceph-mon[97044]: pgmap v10841: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:03 smithi110 ceph-mon[97633]: pgmap v10841: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:03.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:03.568329+0000) 2022-09-10T10:08:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:04.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:04.049637+0000) 2022-09-10T10:08:05.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:04.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:04.568543+0000) 2022-09-10T10:08:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:05.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:05.049828+0000) 2022-09-10T10:08:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:05 smithi104 ceph-mon[97044]: pgmap v10842: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:05 smithi110 ceph-mon[97633]: pgmap v10842: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:06.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:05.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:05.568805+0000) 2022-09-10T10:08:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:06.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:06.050139+0000) 2022-09-10T10:08:07.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:06.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:06.569048+0000) 2022-09-10T10:08:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:07.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:07.050393+0000) 2022-09-10T10:08:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:07 smithi104 ceph-mon[97044]: pgmap v10843: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:08:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:07 smithi110 ceph-mon[97633]: pgmap v10843: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:08:08.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:07.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:07.569321+0000) 2022-09-10T10:08:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:08.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:08.050652+0000) 2022-09-10T10:08:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:08.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:08.569477+0000) 2022-09-10T10:08:08.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:08:08.489+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:08:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:09.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:09.050911+0000) 2022-09-10T10:08:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:09 smithi104 ceph-mon[97044]: pgmap v10844: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:08:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:09 smithi104 ceph-mon[97044]: osdmap e758: 12 total, 12 up, 12 in 2022-09-10T10:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:09 smithi110 ceph-mon[97633]: pgmap v10844: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:09 smithi110 ceph-mon[97633]: osdmap e758: 12 total, 12 up, 12 in 2022-09-10T10:08:10.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:09.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:09.569718+0000) 2022-09-10T10:08:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:10.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:10.051125+0000) 2022-09-10T10:08:11.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:10.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:10.569887+0000) 2022-09-10T10:08:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:11.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:11.051356+0000) 2022-09-10T10:08:11.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:11 smithi104 ceph-mon[97044]: pgmap v10846: 129 pgs: 129 active+clean; 8.9 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:08:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:11 smithi110 ceph-mon[97633]: pgmap v10846: 129 pgs: 129 active+clean; 8.9 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:08:12.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:11.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:11.570174+0000) 2022-09-10T10:08:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:12.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:12.051641+0000) 2022-09-10T10:08:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:12.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:12.570398+0000) 2022-09-10T10:08:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:13.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:13.051913+0000) 2022-09-10T10:08:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:13 smithi110 ceph-mon[97633]: pgmap v10847: 129 pgs: 129 active+clean; 8.9 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:08:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:13 smithi110 ceph-mon[97633]: osdmap e759: 12 total, 12 up, 12 in 2022-09-10T10:08:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:13 smithi104 ceph-mon[97044]: pgmap v10847: 129 pgs: 129 active+clean; 8.9 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:08:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:13 smithi104 ceph-mon[97044]: osdmap e759: 12 total, 12 up, 12 in 2022-09-10T10:08:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:13.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:13.570617+0000) 2022-09-10T10:08:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:14.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:14.052162+0000) 2022-09-10T10:08:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:14.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:14.570877+0000) 2022-09-10T10:08:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:15.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:15.052303+0000) 2022-09-10T10:08:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:15 smithi110 ceph-mon[97633]: pgmap v10849: 129 pgs: 129 active+clean; 8.9 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:08:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:15 smithi104 ceph-mon[97044]: pgmap v10849: 129 pgs: 129 active+clean; 8.9 MiB data, 987 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:08:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:15.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:15.571053+0000) 2022-09-10T10:08:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:16.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:16.052519+0000) 2022-09-10T10:08:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:16.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:16.571260+0000) 2022-09-10T10:08:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:17.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:17.052673+0000) 2022-09-10T10:08:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:17 smithi110 ceph-mon[97633]: pgmap v10850: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:08:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:17 smithi104 ceph-mon[97044]: pgmap v10850: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:08:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:17.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:17.571445+0000) 2022-09-10T10:08:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:18.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:18.052888+0000) 2022-09-10T10:08:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:18.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:18.571673+0000) 2022-09-10T10:08:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:19.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:19.053084+0000) 2022-09-10T10:08:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:19 smithi110 ceph-mon[97633]: pgmap v10851: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:08:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:19 smithi104 ceph-mon[97044]: pgmap v10851: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:08:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:19.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:19.571894+0000) 2022-09-10T10:08:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:20.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:20.053328+0000) 2022-09-10T10:08:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:20.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:20.572086+0000) 2022-09-10T10:08:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:21.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:21.053497+0000) 2022-09-10T10:08:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:21 smithi110 ceph-mon[97633]: pgmap v10852: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:08:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:21 smithi104 ceph-mon[97044]: pgmap v10852: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:08:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:21.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:21.572313+0000) 2022-09-10T10:08:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:22.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:22.053758+0000) 2022-09-10T10:08:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:22.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:22.572556+0000) 2022-09-10T10:08:23.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:23.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:23.054052+0000) 2022-09-10T10:08:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:23 smithi110 ceph-mon[97633]: pgmap v10853: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:08:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:23 smithi104 ceph-mon[97044]: pgmap v10853: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:08:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:23.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:23.572818+0000) 2022-09-10T10:08:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:08:23.498+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:08:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:24.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:24.054288+0000) 2022-09-10T10:08:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:24.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:24.573044+0000) 2022-09-10T10:08:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:25.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:25.054437+0000) 2022-09-10T10:08:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:25 smithi110 ceph-mon[97633]: pgmap v10854: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:08:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:25 smithi104 ceph-mon[97044]: pgmap v10854: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:08:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:25.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:25.573224+0000) 2022-09-10T10:08:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:26.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:26.054639+0000) 2022-09-10T10:08:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:26.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:26.573413+0000) 2022-09-10T10:08:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:27.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:27.054790+0000) 2022-09-10T10:08:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:27 smithi110 ceph-mon[97633]: pgmap v10855: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:08:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:27 smithi104 ceph-mon[97044]: pgmap v10855: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:08:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:27.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:27.573586+0000) 2022-09-10T10:08:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:28.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:28.054980+0000) 2022-09-10T10:08:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:28.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:28.573826+0000) 2022-09-10T10:08:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:29.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:29.055116+0000) 2022-09-10T10:08:29.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:29 smithi104 ceph-mon[97044]: pgmap v10856: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:29 smithi110 ceph-mon[97633]: pgmap v10856: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:29.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:29.574089+0000) 2022-09-10T10:08:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:30.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:30.055244+0000) 2022-09-10T10:08:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:30.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:30.574269+0000) 2022-09-10T10:08:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:31.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:31.055411+0000) 2022-09-10T10:08:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:31 smithi104 ceph-mon[97044]: pgmap v10857: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:31 smithi110 ceph-mon[97633]: pgmap v10857: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:31.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:31.574435+0000) 2022-09-10T10:08:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:32.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:32.055584+0000) 2022-09-10T10:08:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:32.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:32.574643+0000) 2022-09-10T10:08:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:33.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:33.055767+0000) 2022-09-10T10:08:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:33 smithi104 ceph-mon[97044]: pgmap v10858: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:33 smithi110 ceph-mon[97633]: pgmap v10858: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:33.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:33.574848+0000) 2022-09-10T10:08:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:34.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:34.055939+0000) 2022-09-10T10:08:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:34.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:34.575010+0000) 2022-09-10T10:08:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:35.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:35.056070+0000) 2022-09-10T10:08:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:35 smithi104 ceph-mon[97044]: pgmap v10859: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:35 smithi110 ceph-mon[97633]: pgmap v10859: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:35.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:35.575226+0000) 2022-09-10T10:08:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:36.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:36.056269+0000) 2022-09-10T10:08:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:36.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:36.575487+0000) 2022-09-10T10:08:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:37.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:37.056433+0000) 2022-09-10T10:08:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:37 smithi104 ceph-mon[97044]: pgmap v10860: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:37 smithi110 ceph-mon[97633]: pgmap v10860: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:37.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:37.575740+0000) 2022-09-10T10:08:38.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:38.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:38.056607+0000) 2022-09-10T10:08:38.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:08:38.513+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:08:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:38.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:38.575977+0000) 2022-09-10T10:08:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:39.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:39.056866+0000) 2022-09-10T10:08:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:39 smithi104 ceph-mon[97044]: pgmap v10861: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:39 smithi110 ceph-mon[97633]: pgmap v10861: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:39.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:39.576201+0000) 2022-09-10T10:08:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:40.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:40.057123+0000) 2022-09-10T10:08:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:40.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:40.576433+0000) 2022-09-10T10:08:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:41.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:41.057268+0000) 2022-09-10T10:08:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:41 smithi104 ceph-mon[97044]: pgmap v10862: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:41 smithi110 ceph-mon[97633]: pgmap v10862: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:41.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:41.576665+0000) 2022-09-10T10:08:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:42.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:42.057402+0000) 2022-09-10T10:08:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:42.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:42.576884+0000) 2022-09-10T10:08:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:43.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:43.057578+0000) 2022-09-10T10:08:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:43 smithi104 ceph-mon[97044]: pgmap v10863: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:43 smithi110 ceph-mon[97633]: pgmap v10863: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:43.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:43.577158+0000) 2022-09-10T10:08:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:44.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:44.057794+0000) 2022-09-10T10:08:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:44.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:44.577423+0000) 2022-09-10T10:08:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:45.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:45.057990+0000) 2022-09-10T10:08:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:45 smithi104 ceph-mon[97044]: pgmap v10864: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:08:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:08:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:08:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:08:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:45 smithi110 ceph-mon[97633]: pgmap v10864: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:08:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:08:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:08:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:08:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:45.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:45.577610+0000) 2022-09-10T10:08:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:46.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:46.058206+0000) 2022-09-10T10:08:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:46.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:46.577868+0000) 2022-09-10T10:08:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:47.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:47.058395+0000) 2022-09-10T10:08:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:47 smithi104 ceph-mon[97044]: pgmap v10865: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:47 smithi110 ceph-mon[97633]: pgmap v10865: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:47.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:47.578087+0000) 2022-09-10T10:08:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:48.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:48.058583+0000) 2022-09-10T10:08:49.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:48.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:48.578288+0000) 2022-09-10T10:08:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:49.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:49.058798+0000) 2022-09-10T10:08:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:49 smithi104 ceph-mon[97044]: pgmap v10866: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:49 smithi110 ceph-mon[97633]: pgmap v10866: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:49.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:49.578519+0000) 2022-09-10T10:08:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:50.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:50.058989+0000) 2022-09-10T10:08:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:50.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:50.578722+0000) 2022-09-10T10:08:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:51.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:51.059187+0000) 2022-09-10T10:08:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:51 smithi104 ceph-mon[97044]: pgmap v10867: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:51 smithi110 ceph-mon[97633]: pgmap v10867: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:51.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:51.578989+0000) 2022-09-10T10:08:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:52.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:52.059318+0000) 2022-09-10T10:08:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:52.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:52.579192+0000) 2022-09-10T10:08:53.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:53.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:53.059482+0000) 2022-09-10T10:08:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:53 smithi104 ceph-mon[97044]: pgmap v10868: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:53.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:08:53.516+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:08:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:53.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:53.579389+0000) 2022-09-10T10:08:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:53 smithi110 ceph-mon[97633]: pgmap v10868: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:54.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:54.059716+0000) 2022-09-10T10:08:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:54.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:54.579619+0000) 2022-09-10T10:08:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:55.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:55.059950+0000) 2022-09-10T10:08:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:55 smithi104 ceph-mon[97044]: pgmap v10869: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:55 smithi110 ceph-mon[97633]: pgmap v10869: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:55.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:55.579842+0000) 2022-09-10T10:08:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:56.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:56.060203+0000) 2022-09-10T10:08:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:56.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:56.580001+0000) 2022-09-10T10:08:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:57.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:57.060389+0000) 2022-09-10T10:08:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:57 smithi104 ceph-mon[97044]: pgmap v10870: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:57 smithi110 ceph-mon[97633]: pgmap v10870: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:08:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:57.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:57.580234+0000) 2022-09-10T10:08:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:58.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:58.060564+0000) 2022-09-10T10:08:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:58.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:58.580503+0000) 2022-09-10T10:08:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:08:59.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:59.060840+0000) 2022-09-10T10:08:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:08:59 smithi104 ceph-mon[97044]: pgmap v10871: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:08:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:08:59 smithi110 ceph-mon[97633]: pgmap v10871: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:08:59.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:08:59.580682+0000) 2022-09-10T10:09:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:00.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:00.061025+0000) 2022-09-10T10:09:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:00.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:00.580816+0000) 2022-09-10T10:09:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:01.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:01.061181+0000) 2022-09-10T10:09:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:01 smithi104 ceph-mon[97044]: pgmap v10872: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:01 smithi110 ceph-mon[97633]: pgmap v10872: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:01.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:01.581041+0000) 2022-09-10T10:09:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:02.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:02.061350+0000) 2022-09-10T10:09:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:02.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:02.581264+0000) 2022-09-10T10:09:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:03.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:03.061561+0000) 2022-09-10T10:09:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:03 smithi104 ceph-mon[97044]: pgmap v10873: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:03 smithi110 ceph-mon[97633]: pgmap v10873: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:03.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:03.581495+0000) 2022-09-10T10:09:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:04.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:04.061818+0000) 2022-09-10T10:09:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:04.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:04.581738+0000) 2022-09-10T10:09:05.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:05.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:05.062078+0000) 2022-09-10T10:09:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:05 smithi104 ceph-mon[97044]: pgmap v10874: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:05 smithi110 ceph-mon[97633]: pgmap v10874: 129 pgs: 129 active+clean; 8.9 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:05.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:05.581967+0000) 2022-09-10T10:09:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:06.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:06.062285+0000) 2022-09-10T10:09:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:06.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:06.582167+0000) 2022-09-10T10:09:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:07.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:07.062531+0000) 2022-09-10T10:09:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:07 smithi104 ceph-mon[97044]: pgmap v10875: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:09:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:07 smithi110 ceph-mon[97633]: pgmap v10875: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:09:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:07.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:07.582407+0000) 2022-09-10T10:09:08.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:08.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:08.062717+0000) 2022-09-10T10:09:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:08.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:08.582619+0000) 2022-09-10T10:09:08.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:09:08.531+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:09:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:09.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:09.062972+0000) 2022-09-10T10:09:09.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:09 smithi104 ceph-mon[97044]: pgmap v10876: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:09:09.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:09 smithi104 ceph-mon[97044]: osdmap e760: 12 total, 12 up, 12 in 2022-09-10T10:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:09 smithi110 ceph-mon[97633]: pgmap v10876: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:09:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:09 smithi110 ceph-mon[97633]: osdmap e760: 12 total, 12 up, 12 in 2022-09-10T10:09:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:09.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:09.582778+0000) 2022-09-10T10:09:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:10.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:10.063156+0000) 2022-09-10T10:09:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:10.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:10.582970+0000) 2022-09-10T10:09:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:11.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:11.063319+0000) 2022-09-10T10:09:11.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:11 smithi104 ceph-mon[97044]: pgmap v10878: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:09:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:11 smithi110 ceph-mon[97633]: pgmap v10878: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:09:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:11.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:11.583129+0000) 2022-09-10T10:09:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:12.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:12.063551+0000) 2022-09-10T10:09:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:12.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:12.583379+0000) 2022-09-10T10:09:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:13.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:13.063842+0000) 2022-09-10T10:09:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:13 smithi110 ceph-mon[97633]: pgmap v10879: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:09:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:13 smithi110 ceph-mon[97633]: osdmap e761: 12 total, 12 up, 12 in 2022-09-10T10:09:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:13 smithi104 ceph-mon[97044]: pgmap v10879: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:09:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:13 smithi104 ceph-mon[97044]: osdmap e761: 12 total, 12 up, 12 in 2022-09-10T10:09:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:13.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:13.583590+0000) 2022-09-10T10:09:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:14.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:14.064042+0000) 2022-09-10T10:09:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:14.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:14.583805+0000) 2022-09-10T10:09:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:15.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:15.064265+0000) 2022-09-10T10:09:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:15 smithi110 ceph-mon[97633]: pgmap v10881: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:09:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:15 smithi104 ceph-mon[97044]: pgmap v10881: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:09:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:15.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:15.584000+0000) 2022-09-10T10:09:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:16.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:16.064484+0000) 2022-09-10T10:09:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:16.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:16.584212+0000) 2022-09-10T10:09:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:17.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:17.064650+0000) 2022-09-10T10:09:17.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:17 smithi110 ceph-mon[97633]: pgmap v10882: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:09:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:17 smithi104 ceph-mon[97044]: pgmap v10882: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:09:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:17.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:17.584414+0000) 2022-09-10T10:09:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:18.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:18.064823+0000) 2022-09-10T10:09:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:18.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:18.584595+0000) 2022-09-10T10:09:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:19.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:19.065000+0000) 2022-09-10T10:09:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:19 smithi110 ceph-mon[97633]: pgmap v10883: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:09:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:19 smithi104 ceph-mon[97044]: pgmap v10883: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:09:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:19.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:19.584767+0000) 2022-09-10T10:09:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:20.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:20.065250+0000) 2022-09-10T10:09:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:20.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:20.585009+0000) 2022-09-10T10:09:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:21.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:21.065429+0000) 2022-09-10T10:09:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:21 smithi110 ceph-mon[97633]: pgmap v10884: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:09:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:21 smithi104 ceph-mon[97044]: pgmap v10884: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:09:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:21.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:21.585219+0000) 2022-09-10T10:09:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:22.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:22.065610+0000) 2022-09-10T10:09:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:22.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:22.585437+0000) 2022-09-10T10:09:23.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:23.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:23.065789+0000) 2022-09-10T10:09:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:23 smithi104 ceph-mon[97044]: pgmap v10885: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:09:23.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:09:23.545+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:09:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:23.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:23.585675+0000) 2022-09-10T10:09:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:23 smithi110 ceph-mon[97633]: pgmap v10885: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:09:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:24.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:24.065988+0000) 2022-09-10T10:09:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:24.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:24.585934+0000) 2022-09-10T10:09:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:25.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:25.066238+0000) 2022-09-10T10:09:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:25 smithi104 ceph-mon[97044]: pgmap v10886: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:09:25.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:25 smithi110 ceph-mon[97633]: pgmap v10886: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T10:09:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:25.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:25.586174+0000) 2022-09-10T10:09:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:26.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:26.066408+0000) 2022-09-10T10:09:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:26.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:26.586327+0000) 2022-09-10T10:09:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:27.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:27.066582+0000) 2022-09-10T10:09:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:27 smithi104 ceph-mon[97044]: pgmap v10887: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:09:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:27 smithi110 ceph-mon[97633]: pgmap v10887: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:09:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:27.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:27.586544+0000) 2022-09-10T10:09:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:28.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:28.066785+0000) 2022-09-10T10:09:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:28.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:28.586835+0000) 2022-09-10T10:09:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:29.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:29.066977+0000) 2022-09-10T10:09:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:29 smithi104 ceph-mon[97044]: pgmap v10888: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:29 smithi110 ceph-mon[97633]: pgmap v10888: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:29.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:29.587036+0000) 2022-09-10T10:09:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:30.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:30.067256+0000) 2022-09-10T10:09:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:30.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:30.587248+0000) 2022-09-10T10:09:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:31.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:31.067455+0000) 2022-09-10T10:09:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:31 smithi104 ceph-mon[97044]: pgmap v10889: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:31 smithi110 ceph-mon[97633]: pgmap v10889: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:31.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:31.587479+0000) 2022-09-10T10:09:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:32.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:32.067701+0000) 2022-09-10T10:09:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:32.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:32.587610+0000) 2022-09-10T10:09:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:33.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:33.067899+0000) 2022-09-10T10:09:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:33 smithi104 ceph-mon[97044]: pgmap v10890: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:33 smithi110 ceph-mon[97633]: pgmap v10890: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:33.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:33.587860+0000) 2022-09-10T10:09:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:34.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:34.068146+0000) 2022-09-10T10:09:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:34.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:34.588165+0000) 2022-09-10T10:09:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:35.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:35.068312+0000) 2022-09-10T10:09:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:35 smithi104 ceph-mon[97044]: pgmap v10891: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:35 smithi110 ceph-mon[97633]: pgmap v10891: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:35.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:35.588369+0000) 2022-09-10T10:09:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:36.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:36.068521+0000) 2022-09-10T10:09:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:36.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:36.588586+0000) 2022-09-10T10:09:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:37.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:37.068693+0000) 2022-09-10T10:09:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:37 smithi104 ceph-mon[97044]: pgmap v10892: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:37 smithi110 ceph-mon[97633]: pgmap v10892: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:37.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:37.588801+0000) 2022-09-10T10:09:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:38.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:38.068872+0000) 2022-09-10T10:09:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:38.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:38.588966+0000) 2022-09-10T10:09:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:09:38.560+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:09:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:39.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:39.069133+0000) 2022-09-10T10:09:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:39 smithi104 ceph-mon[97044]: pgmap v10893: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:39 smithi110 ceph-mon[97633]: pgmap v10893: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:39.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:39.589238+0000) 2022-09-10T10:09:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:40.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:40.069330+0000) 2022-09-10T10:09:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:40.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:40.589451+0000) 2022-09-10T10:09:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:41.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:41.069489+0000) 2022-09-10T10:09:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:41 smithi104 ceph-mon[97044]: pgmap v10894: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:41 smithi110 ceph-mon[97633]: pgmap v10894: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:41.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:41.589704+0000) 2022-09-10T10:09:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:42.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:42.069682+0000) 2022-09-10T10:09:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:42.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:42.589900+0000) 2022-09-10T10:09:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:43.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:43.069938+0000) 2022-09-10T10:09:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:43 smithi104 ceph-mon[97044]: pgmap v10895: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:43 smithi110 ceph-mon[97633]: pgmap v10895: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:43.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:43.590166+0000) 2022-09-10T10:09:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:44.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:44.070186+0000) 2022-09-10T10:09:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:44.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:44.590391+0000) 2022-09-10T10:09:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:45.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:45.070388+0000) 2022-09-10T10:09:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:45 smithi104 ceph-mon[97044]: pgmap v10896: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:09:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:45 smithi110 ceph-mon[97633]: pgmap v10896: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:09:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:09:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:09:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:09:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:45.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:45.590643+0000) 2022-09-10T10:09:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:46.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:46.070675+0000) 2022-09-10T10:09:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:46.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:46.590832+0000) 2022-09-10T10:09:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:47.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:47.070835+0000) 2022-09-10T10:09:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:47 smithi104 ceph-mon[97044]: pgmap v10897: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:47 smithi110 ceph-mon[97633]: pgmap v10897: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:47.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:47.590976+0000) 2022-09-10T10:09:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:48.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:48.070965+0000) 2022-09-10T10:09:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:48.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:48.591191+0000) 2022-09-10T10:09:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:49.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:49.071186+0000) 2022-09-10T10:09:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:49 smithi104 ceph-mon[97044]: pgmap v10898: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:09:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:49 smithi110 ceph-mon[97633]: pgmap v10898: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:09:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:49.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:49.591398+0000) 2022-09-10T10:09:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:50.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:50.071423+0000) 2022-09-10T10:09:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:50.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:50.591591+0000) 2022-09-10T10:09:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:51.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:51.071695+0000) 2022-09-10T10:09:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:51 smithi104 ceph-mon[97044]: pgmap v10899: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:51 smithi110 ceph-mon[97633]: pgmap v10899: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:51.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:51.591783+0000) 2022-09-10T10:09:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:52.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:52.071927+0000) 2022-09-10T10:09:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:52.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:52.592008+0000) 2022-09-10T10:09:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:53.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:53.072163+0000) 2022-09-10T10:09:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:53 smithi104 ceph-mon[97044]: pgmap v10900: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:09:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:53 smithi110 ceph-mon[97633]: pgmap v10900: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:09:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:53.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:53.592240+0000) 2022-09-10T10:09:54.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:09:53.570+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:09:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:54.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:54.072406+0000) 2022-09-10T10:09:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:54.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:54.592385+0000) 2022-09-10T10:09:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:55.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:55.072678+0000) 2022-09-10T10:09:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:55 smithi104 ceph-mon[97044]: pgmap v10901: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:09:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:55 smithi110 ceph-mon[97633]: pgmap v10901: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:09:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:55.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:55.592598+0000) 2022-09-10T10:09:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:56.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:56.072912+0000) 2022-09-10T10:09:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:56.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:56.592805+0000) 2022-09-10T10:09:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:57.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:57.073170+0000) 2022-09-10T10:09:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:57 smithi104 ceph-mon[97044]: pgmap v10902: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:57 smithi110 ceph-mon[97633]: pgmap v10902: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:09:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:57.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:57.593067+0000) 2022-09-10T10:09:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:58.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:58.073347+0000) 2022-09-10T10:09:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:58.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:58.593279+0000) 2022-09-10T10:09:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:09:59.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:59.073492+0000) 2022-09-10T10:09:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:09:59 smithi104 ceph-mon[97044]: pgmap v10903: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:09:59.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:09:59 smithi110 ceph-mon[97633]: pgmap v10903: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:09:59.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:09:59.593420+0000) 2022-09-10T10:10:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:00.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:00.073694+0000) 2022-09-10T10:10:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:10:00.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:10:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:00.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:00.593561+0000) 2022-09-10T10:10:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:01.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:01.073879+0000) 2022-09-10T10:10:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:01 smithi104 ceph-mon[97044]: pgmap v10904: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:01 smithi110 ceph-mon[97633]: pgmap v10904: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:01.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:01.593804+0000) 2022-09-10T10:10:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:02.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:02.074145+0000) 2022-09-10T10:10:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:02.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:02.594002+0000) 2022-09-10T10:10:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:03.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:03.074418+0000) 2022-09-10T10:10:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:03 smithi104 ceph-mon[97044]: pgmap v10905: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:03 smithi110 ceph-mon[97633]: pgmap v10905: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:03.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:03.594198+0000) 2022-09-10T10:10:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:04.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:04.074618+0000) 2022-09-10T10:10:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:04.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:04.594448+0000) 2022-09-10T10:10:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:05.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:05.074859+0000) 2022-09-10T10:10:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:05 smithi104 ceph-mon[97044]: pgmap v10906: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:05 smithi110 ceph-mon[97633]: pgmap v10906: 129 pgs: 129 active+clean; 8.9 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:05.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:05.594657+0000) 2022-09-10T10:10:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:06.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:06.075168+0000) 2022-09-10T10:10:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:06.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:06.594888+0000) 2022-09-10T10:10:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:07.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:07.075363+0000) 2022-09-10T10:10:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:07 smithi104 ceph-mon[97044]: pgmap v10907: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:10:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:07 smithi110 ceph-mon[97633]: pgmap v10907: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:10:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:07.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:07.595156+0000) 2022-09-10T10:10:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:08.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:08.075539+0000) 2022-09-10T10:10:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:08.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:08.595317+0000) 2022-09-10T10:10:09.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:10:08.585+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:10:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:09.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:09.075723+0000) 2022-09-10T10:10:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:09 smithi104 ceph-mon[97044]: pgmap v10908: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:10:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:09 smithi104 ceph-mon[97044]: osdmap e762: 12 total, 12 up, 12 in 2022-09-10T10:10:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:09 smithi110 ceph-mon[97633]: pgmap v10908: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:10:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:09 smithi110 ceph-mon[97633]: osdmap e762: 12 total, 12 up, 12 in 2022-09-10T10:10:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:09.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:09.595510+0000) 2022-09-10T10:10:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:10.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:10.075967+0000) 2022-09-10T10:10:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:10.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:10.595693+0000) 2022-09-10T10:10:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:11.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:11.076223+0000) 2022-09-10T10:10:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:11 smithi104 ceph-mon[97044]: pgmap v10910: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:10:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:11 smithi110 ceph-mon[97633]: pgmap v10910: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:10:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:11.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:11.595887+0000) 2022-09-10T10:10:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:12.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:12.076451+0000) 2022-09-10T10:10:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:12.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:12.596151+0000) 2022-09-10T10:10:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:13.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:13.076680+0000) 2022-09-10T10:10:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:13 smithi104 ceph-mon[97044]: pgmap v10911: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:10:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:13 smithi104 ceph-mon[97044]: osdmap e763: 12 total, 12 up, 12 in 2022-09-10T10:10:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:13 smithi110 ceph-mon[97633]: pgmap v10911: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:10:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:13 smithi110 ceph-mon[97633]: osdmap e763: 12 total, 12 up, 12 in 2022-09-10T10:10:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:13.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:13.596385+0000) 2022-09-10T10:10:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:14.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:14.076925+0000) 2022-09-10T10:10:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:14.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:14.596583+0000) 2022-09-10T10:10:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:15.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:15.077086+0000) 2022-09-10T10:10:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:15 smithi104 ceph-mon[97044]: pgmap v10913: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:10:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:15 smithi110 ceph-mon[97633]: pgmap v10913: 129 pgs: 129 active+clean; 8.9 MiB data, 978 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:10:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:15.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:15.596718+0000) 2022-09-10T10:10:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:16.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:16.077256+0000) 2022-09-10T10:10:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:16.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:16.596950+0000) 2022-09-10T10:10:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:17.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:17.077441+0000) 2022-09-10T10:10:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:17 smithi104 ceph-mon[97044]: pgmap v10914: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:10:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:17 smithi110 ceph-mon[97633]: pgmap v10914: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:10:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:17.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:17.597218+0000) 2022-09-10T10:10:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:18.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:18.077618+0000) 2022-09-10T10:10:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:18.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:18.597454+0000) 2022-09-10T10:10:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:19.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:19.077852+0000) 2022-09-10T10:10:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:19 smithi104 ceph-mon[97044]: pgmap v10915: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:10:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:19 smithi110 ceph-mon[97633]: pgmap v10915: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T10:10:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:19.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:19.597665+0000) 2022-09-10T10:10:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:20.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:20.078047+0000) 2022-09-10T10:10:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:20.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:20.597898+0000) 2022-09-10T10:10:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:21.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:21.078253+0000) 2022-09-10T10:10:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:21 smithi104 ceph-mon[97044]: pgmap v10916: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:10:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:21 smithi110 ceph-mon[97633]: pgmap v10916: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:10:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:21.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:21.598052+0000) 2022-09-10T10:10:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:22.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:22.078502+0000) 2022-09-10T10:10:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:22.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:22.598288+0000) 2022-09-10T10:10:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:23.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:23.078685+0000) 2022-09-10T10:10:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:23 smithi104 ceph-mon[97044]: pgmap v10917: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:10:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:23 smithi110 ceph-mon[97633]: pgmap v10917: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:10:24.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:10:23.600+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:10:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:23.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:23.598489+0000) 2022-09-10T10:10:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:24.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:24.078906+0000) 2022-09-10T10:10:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:24.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:24.598712+0000) 2022-09-10T10:10:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:25.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:25.079156+0000) 2022-09-10T10:10:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:25 smithi104 ceph-mon[97044]: pgmap v10918: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T10:10:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:25 smithi110 ceph-mon[97633]: pgmap v10918: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T10:10:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:25.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:25.598947+0000) 2022-09-10T10:10:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:26.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:26.079364+0000) 2022-09-10T10:10:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:26.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:26.599152+0000) 2022-09-10T10:10:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:27.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:27.079585+0000) 2022-09-10T10:10:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:27 smithi104 ceph-mon[97044]: pgmap v10919: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:10:27.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:27 smithi110 ceph-mon[97633]: pgmap v10919: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:10:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:27.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:27.599389+0000) 2022-09-10T10:10:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:28.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:28.079755+0000) 2022-09-10T10:10:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:28.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:28.599624+0000) 2022-09-10T10:10:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:29.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:29.079958+0000) 2022-09-10T10:10:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:29 smithi104 ceph-mon[97044]: pgmap v10920: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:29 smithi110 ceph-mon[97633]: pgmap v10920: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:29.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:29.599805+0000) 2022-09-10T10:10:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:30.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:30.080170+0000) 2022-09-10T10:10:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:30.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:30.600015+0000) 2022-09-10T10:10:31.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:31.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:31.080339+0000) 2022-09-10T10:10:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:31 smithi104 ceph-mon[97044]: pgmap v10921: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:31 smithi110 ceph-mon[97633]: pgmap v10921: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:31.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:31.600255+0000) 2022-09-10T10:10:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:32.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:32.080528+0000) 2022-09-10T10:10:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:32.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:32.600487+0000) 2022-09-10T10:10:33.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:33.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:33.080664+0000) 2022-09-10T10:10:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:33 smithi104 ceph-mon[97044]: pgmap v10922: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:33 smithi110 ceph-mon[97633]: pgmap v10922: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:33.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:33.600741+0000) 2022-09-10T10:10:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:34.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:34.080915+0000) 2022-09-10T10:10:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:34.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:34.600975+0000) 2022-09-10T10:10:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:35.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:35.081164+0000) 2022-09-10T10:10:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:35 smithi104 ceph-mon[97044]: pgmap v10923: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:35 smithi110 ceph-mon[97633]: pgmap v10923: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:35.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:35.601188+0000) 2022-09-10T10:10:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:36.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:36.081423+0000) 2022-09-10T10:10:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:36.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:36.601417+0000) 2022-09-10T10:10:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:37.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:37.081600+0000) 2022-09-10T10:10:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:37 smithi104 ceph-mon[97044]: pgmap v10924: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:37 smithi110 ceph-mon[97633]: pgmap v10924: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:37.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:37.601584+0000) 2022-09-10T10:10:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:38.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:38.081795+0000) 2022-09-10T10:10:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:10:38.616+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:10:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:38.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:38.601793+0000) 2022-09-10T10:10:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:39.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:39.082028+0000) 2022-09-10T10:10:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:39 smithi104 ceph-mon[97044]: pgmap v10925: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:39 smithi110 ceph-mon[97633]: pgmap v10925: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:39.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:39.602018+0000) 2022-09-10T10:10:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:40.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:40.082205+0000) 2022-09-10T10:10:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:40.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:40.602202+0000) 2022-09-10T10:10:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:41.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:41.082429+0000) 2022-09-10T10:10:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:41 smithi104 ceph-mon[97044]: pgmap v10926: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:41 smithi110 ceph-mon[97633]: pgmap v10926: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:41.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:41.602367+0000) 2022-09-10T10:10:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:42.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:42.082639+0000) 2022-09-10T10:10:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:42.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:42.602608+0000) 2022-09-10T10:10:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:43.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:43.082900+0000) 2022-09-10T10:10:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:43 smithi104 ceph-mon[97044]: pgmap v10927: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:43.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:43 smithi110 ceph-mon[97633]: pgmap v10927: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:43.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:43.602807+0000) 2022-09-10T10:10:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:44.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:44.083175+0000) 2022-09-10T10:10:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:44.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:44.603065+0000) 2022-09-10T10:10:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:45.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:45.083387+0000) 2022-09-10T10:10:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:45 smithi104 ceph-mon[97044]: pgmap v10928: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:10:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:10:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:10:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:10:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:45 smithi110 ceph-mon[97633]: pgmap v10928: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:10:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:10:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:10:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:10:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:45.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:45.603297+0000) 2022-09-10T10:10:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:46.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:46.083592+0000) 2022-09-10T10:10:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:46.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:46.603442+0000) 2022-09-10T10:10:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:47.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:47.083846+0000) 2022-09-10T10:10:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:47 smithi104 ceph-mon[97044]: pgmap v10929: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:47 smithi110 ceph-mon[97633]: pgmap v10929: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:47.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:47.603669+0000) 2022-09-10T10:10:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:48.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:48.084059+0000) 2022-09-10T10:10:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:48.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:48.603902+0000) 2022-09-10T10:10:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:49.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:49.084315+0000) 2022-09-10T10:10:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:49 smithi104 ceph-mon[97044]: pgmap v10930: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:49 smithi110 ceph-mon[97633]: pgmap v10930: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:10:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:49.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:49.604166+0000) 2022-09-10T10:10:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:50.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:50.084496+0000) 2022-09-10T10:10:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:50.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:50.604357+0000) 2022-09-10T10:10:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:51.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:51.084728+0000) 2022-09-10T10:10:51.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:51 smithi104 ceph-mon[97044]: pgmap v10931: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:51 smithi110 ceph-mon[97633]: pgmap v10931: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:51.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:51.604505+0000) 2022-09-10T10:10:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:52.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:52.084886+0000) 2022-09-10T10:10:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:52.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:52.604785+0000) 2022-09-10T10:10:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:53.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:53.085156+0000) 2022-09-10T10:10:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:53 smithi104 ceph-mon[97044]: pgmap v10932: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:10:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:53 smithi110 ceph-mon[97633]: pgmap v10932: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:10:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:53.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:53.605042+0000) 2022-09-10T10:10:54.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:10:53.626+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:10:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:54.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:54.085348+0000) 2022-09-10T10:10:55.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:54.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:54.605270+0000) 2022-09-10T10:10:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:55.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:55.085514+0000) 2022-09-10T10:10:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:55 smithi104 ceph-mon[97044]: pgmap v10933: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:10:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:55 smithi110 ceph-mon[97633]: pgmap v10933: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:10:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:55.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:55.605454+0000) 2022-09-10T10:10:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:56.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:56.085685+0000) 2022-09-10T10:10:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:56.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:56.605666+0000) 2022-09-10T10:10:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:57.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:57.085903+0000) 2022-09-10T10:10:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:57 smithi104 ceph-mon[97044]: pgmap v10934: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:57 smithi110 ceph-mon[97633]: pgmap v10934: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:10:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:57.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:57.605885+0000) 2022-09-10T10:10:58.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:58.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:58.086151+0000) 2022-09-10T10:10:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:58.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:58.606147+0000) 2022-09-10T10:10:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:10:59.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:59.086354+0000) 2022-09-10T10:10:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:10:59 smithi104 ceph-mon[97044]: pgmap v10935: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:10:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:10:59 smithi110 ceph-mon[97633]: pgmap v10935: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:11:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:10:59.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:10:59.606316+0000) 2022-09-10T10:11:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:00.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:00.086603+0000) 2022-09-10T10:11:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:00.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:00.606502+0000) 2022-09-10T10:11:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:01.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:01.086818+0000) 2022-09-10T10:11:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:01 smithi104 ceph-mon[97044]: pgmap v10936: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:01 smithi110 ceph-mon[97633]: pgmap v10936: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:01.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:01.606746+0000) 2022-09-10T10:11:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:02.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:02.087006+0000) 2022-09-10T10:11:03.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:02.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:02.606950+0000) 2022-09-10T10:11:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:03.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:03.087248+0000) 2022-09-10T10:11:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:03 smithi104 ceph-mon[97044]: pgmap v10937: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:03 smithi110 ceph-mon[97633]: pgmap v10937: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:03.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:03.607182+0000) 2022-09-10T10:11:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:04.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:04.087458+0000) 2022-09-10T10:11:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:04.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:04.607428+0000) 2022-09-10T10:11:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:05.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:05.087588+0000) 2022-09-10T10:11:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:05 smithi104 ceph-mon[97044]: pgmap v10938: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:05 smithi110 ceph-mon[97633]: pgmap v10938: 129 pgs: 129 active+clean; 8.9 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:05.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:05.607680+0000) 2022-09-10T10:11:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:06.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:06.087853+0000) 2022-09-10T10:11:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:06.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:06.607877+0000) 2022-09-10T10:11:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:07.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:07.088085+0000) 2022-09-10T10:11:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:07 smithi104 ceph-mon[97044]: pgmap v10939: 129 pgs: 129 active+clean; 8.9 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:11:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:07 smithi110 ceph-mon[97633]: pgmap v10939: 129 pgs: 129 active+clean; 8.9 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:11:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:07.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:07.608069+0000) 2022-09-10T10:11:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:08.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:08.088259+0000) 2022-09-10T10:11:09.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:11:08.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:11:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:08.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:08.608284+0000) 2022-09-10T10:11:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:09.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:09.088466+0000) 2022-09-10T10:11:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:09 smithi104 ceph-mon[97044]: pgmap v10940: 129 pgs: 129 active+clean; 8.9 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:11:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:09 smithi104 ceph-mon[97044]: osdmap e764: 12 total, 12 up, 12 in 2022-09-10T10:11:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:09 smithi110 ceph-mon[97633]: pgmap v10940: 129 pgs: 129 active+clean; 8.9 MiB data, 984 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:11:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:09 smithi110 ceph-mon[97633]: osdmap e764: 12 total, 12 up, 12 in 2022-09-10T10:11:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:09.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:09.608538+0000) 2022-09-10T10:11:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:10.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:10.088629+0000) 2022-09-10T10:11:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:10.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:10.608762+0000) 2022-09-10T10:11:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:11.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:11.088839+0000) 2022-09-10T10:11:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:11 smithi104 ceph-mon[97044]: pgmap v10942: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:11:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:11 smithi110 ceph-mon[97633]: pgmap v10942: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:11:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:11.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:11.608997+0000) 2022-09-10T10:11:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:12.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:12.089004+0000) 2022-09-10T10:11:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:12.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:12.609211+0000) 2022-09-10T10:11:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:13.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:13.089234+0000) 2022-09-10T10:11:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:13 smithi104 ceph-mon[97044]: pgmap v10943: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:11:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:13 smithi104 ceph-mon[97044]: osdmap e765: 12 total, 12 up, 12 in 2022-09-10T10:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:13 smithi110 ceph-mon[97633]: pgmap v10943: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:11:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:13 smithi110 ceph-mon[97633]: osdmap e765: 12 total, 12 up, 12 in 2022-09-10T10:11:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:13.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:13.609417+0000) 2022-09-10T10:11:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:14.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:14.089414+0000) 2022-09-10T10:11:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:14.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:14.609651+0000) 2022-09-10T10:11:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:15.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:15.089667+0000) 2022-09-10T10:11:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:15 smithi104 ceph-mon[97044]: pgmap v10945: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:15 smithi110 ceph-mon[97633]: pgmap v10945: 129 pgs: 129 active+clean; 8.9 MiB data, 988 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:11:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:15.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:15.609874+0000) 2022-09-10T10:11:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:16.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:16.089919+0000) 2022-09-10T10:11:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:16.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:16.610150+0000) 2022-09-10T10:11:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:17.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:17.090158+0000) 2022-09-10T10:11:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:17 smithi104 ceph-mon[97044]: pgmap v10946: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:11:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:17 smithi110 ceph-mon[97633]: pgmap v10946: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:11:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:17.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:17.610387+0000) 2022-09-10T10:11:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:18.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:18.090319+0000) 2022-09-10T10:11:19.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:18.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:18.610597+0000) 2022-09-10T10:11:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:19.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:19.090557+0000) 2022-09-10T10:11:19.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:19 smithi104 ceph-mon[97044]: pgmap v10947: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:11:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:19 smithi110 ceph-mon[97633]: pgmap v10947: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:11:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:19.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:19.610824+0000) 2022-09-10T10:11:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:20.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:20.090804+0000) 2022-09-10T10:11:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:20.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:20.611033+0000) 2022-09-10T10:11:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:21.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:21.090996+0000) 2022-09-10T10:11:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:21 smithi104 ceph-mon[97044]: pgmap v10948: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:11:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:21 smithi110 ceph-mon[97633]: pgmap v10948: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:11:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:21.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:21.611267+0000) 2022-09-10T10:11:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:22.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:22.091199+0000) 2022-09-10T10:11:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:22.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:22.611515+0000) 2022-09-10T10:11:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:23.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:23.091465+0000) 2022-09-10T10:11:23.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:23 smithi104 ceph-mon[97044]: pgmap v10949: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:11:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:23 smithi110 ceph-mon[97633]: pgmap v10949: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:11:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:23.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:23.611719+0000) 2022-09-10T10:11:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:11:23.653+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:11:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:24.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:24.091675+0000) 2022-09-10T10:11:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:24.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:24.611953+0000) 2022-09-10T10:11:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:25.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:25.091912+0000) 2022-09-10T10:11:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:25 smithi104 ceph-mon[97044]: pgmap v10950: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T10:11:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:25 smithi110 ceph-mon[97633]: pgmap v10950: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T10:11:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:25.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:25.612220+0000) 2022-09-10T10:11:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:26.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:26.092130+0000) 2022-09-10T10:11:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:26.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:26.612450+0000) 2022-09-10T10:11:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:27.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:27.092316+0000) 2022-09-10T10:11:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:27 smithi104 ceph-mon[97044]: pgmap v10951: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:11:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:27 smithi110 ceph-mon[97633]: pgmap v10951: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:11:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:27.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:27.612702+0000) 2022-09-10T10:11:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:28.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:28.092467+0000) 2022-09-10T10:11:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:28.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:28.612962+0000) 2022-09-10T10:11:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:29.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:29.092692+0000) 2022-09-10T10:11:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:29 smithi104 ceph-mon[97044]: pgmap v10952: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:29 smithi110 ceph-mon[97633]: pgmap v10952: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:29.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:29.613231+0000) 2022-09-10T10:11:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:30.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:30.092910+0000) 2022-09-10T10:11:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:30.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:30.613457+0000) 2022-09-10T10:11:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:31.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:31.093146+0000) 2022-09-10T10:11:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:31 smithi104 ceph-mon[97044]: pgmap v10953: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:31 smithi110 ceph-mon[97633]: pgmap v10953: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:31.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:31.613674+0000) 2022-09-10T10:11:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:32.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:32.093376+0000) 2022-09-10T10:11:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:32.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:32.613831+0000) 2022-09-10T10:11:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:33.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:33.093644+0000) 2022-09-10T10:11:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:33 smithi104 ceph-mon[97044]: pgmap v10954: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:33 smithi110 ceph-mon[97633]: pgmap v10954: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:33.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:33.614019+0000) 2022-09-10T10:11:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:34.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:34.093813+0000) 2022-09-10T10:11:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:34.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:34.614228+0000) 2022-09-10T10:11:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:35.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:35.094003+0000) 2022-09-10T10:11:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:35 smithi104 ceph-mon[97044]: pgmap v10955: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:11:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:35 smithi110 ceph-mon[97633]: pgmap v10955: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:11:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:35.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:35.614379+0000) 2022-09-10T10:11:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:36.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:36.094154+0000) 2022-09-10T10:11:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:36.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:36.614533+0000) 2022-09-10T10:11:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:37.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:37.094366+0000) 2022-09-10T10:11:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:37 smithi104 ceph-mon[97044]: pgmap v10956: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:37 smithi110 ceph-mon[97633]: pgmap v10956: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:37.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:37.614683+0000) 2022-09-10T10:11:38.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:38.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:38.094634+0000) 2022-09-10T10:11:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:38.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:38.614908+0000) 2022-09-10T10:11:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:11:38.654+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:11:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:39.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:39.094812+0000) 2022-09-10T10:11:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:39 smithi104 ceph-mon[97044]: pgmap v10957: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:39 smithi110 ceph-mon[97633]: pgmap v10957: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:39.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:39.615123+0000) 2022-09-10T10:11:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:40.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:40.095006+0000) 2022-09-10T10:11:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:40.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:40.615298+0000) 2022-09-10T10:11:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:41.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:41.095186+0000) 2022-09-10T10:11:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:41 smithi104 ceph-mon[97044]: pgmap v10958: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:41 smithi110 ceph-mon[97633]: pgmap v10958: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:41.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:41.615578+0000) 2022-09-10T10:11:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:42.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:42.095358+0000) 2022-09-10T10:11:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:42.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:42.615820+0000) 2022-09-10T10:11:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:43.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:43.095541+0000) 2022-09-10T10:11:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:43 smithi104 ceph-mon[97044]: pgmap v10959: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:43 smithi110 ceph-mon[97633]: pgmap v10959: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:43.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:43.616050+0000) 2022-09-10T10:11:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:44.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:44.095717+0000) 2022-09-10T10:11:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:44.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:44.616279+0000) 2022-09-10T10:11:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:45.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:45.095944+0000) 2022-09-10T10:11:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:45 smithi104 ceph-mon[97044]: pgmap v10960: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:11:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:11:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:11:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:11:45.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:11:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:45 smithi110 ceph-mon[97633]: pgmap v10960: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:11:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:11:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:11:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:11:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:11:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:45.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:45.616524+0000) 2022-09-10T10:11:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:46.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:46.096168+0000) 2022-09-10T10:11:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:46.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:46.616772+0000) 2022-09-10T10:11:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:47.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:47.096381+0000) 2022-09-10T10:11:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:47 smithi104 ceph-mon[97044]: pgmap v10961: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:11:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:47 smithi110 ceph-mon[97633]: pgmap v10961: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:11:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:47.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:47.617039+0000) 2022-09-10T10:11:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:48.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:48.096557+0000) 2022-09-10T10:11:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:48.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:48.617256+0000) 2022-09-10T10:11:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:49.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:49.096803+0000) 2022-09-10T10:11:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:49 smithi104 ceph-mon[97044]: pgmap v10962: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:49 smithi110 ceph-mon[97633]: pgmap v10962: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:49.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:49.617509+0000) 2022-09-10T10:11:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:50.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:50.096976+0000) 2022-09-10T10:11:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:50.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:50.617724+0000) 2022-09-10T10:11:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:51.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:51.097207+0000) 2022-09-10T10:11:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:51 smithi104 ceph-mon[97044]: pgmap v10963: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:51 smithi110 ceph-mon[97633]: pgmap v10963: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:11:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:51.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:51.617937+0000) 2022-09-10T10:11:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:52.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:52.097483+0000) 2022-09-10T10:11:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:52.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:52.618179+0000) 2022-09-10T10:11:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:53.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:53.097688+0000) 2022-09-10T10:11:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:53 smithi104 ceph-mon[97044]: pgmap v10964: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:53 smithi110 ceph-mon[97633]: pgmap v10964: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:53.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:53.618397+0000) 2022-09-10T10:11:54.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:11:53.664+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:11:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:54.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:54.097945+0000) 2022-09-10T10:11:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:54.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:54.618637+0000) 2022-09-10T10:11:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:55.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:55.098118+0000) 2022-09-10T10:11:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:55 smithi104 ceph-mon[97044]: pgmap v10965: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:11:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:55 smithi110 ceph-mon[97633]: pgmap v10965: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:11:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:55.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:55.618811+0000) 2022-09-10T10:11:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:56.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:56.098296+0000) 2022-09-10T10:11:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:56.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:56.619060+0000) 2022-09-10T10:11:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:57.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:57.098514+0000) 2022-09-10T10:11:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:57 smithi104 ceph-mon[97044]: pgmap v10966: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:11:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:57 smithi110 ceph-mon[97633]: pgmap v10966: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:11:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:57.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:57.619294+0000) 2022-09-10T10:11:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:58.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:58.098727+0000) 2022-09-10T10:11:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:58.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:58.619505+0000) 2022-09-10T10:11:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:11:59.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:59.098945+0000) 2022-09-10T10:11:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:11:59 smithi104 ceph-mon[97044]: pgmap v10967: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:11:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:11:59 smithi110 ceph-mon[97633]: pgmap v10967: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:11:59.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:11:59.619746+0000) 2022-09-10T10:12:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:00.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:00.099188+0000) 2022-09-10T10:12:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:00.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:00.619943+0000) 2022-09-10T10:12:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:01.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:01.099368+0000) 2022-09-10T10:12:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:01 smithi104 ceph-mon[97044]: pgmap v10968: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:01 smithi110 ceph-mon[97633]: pgmap v10968: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:01.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:01.620208+0000) 2022-09-10T10:12:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:02.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:02.099551+0000) 2022-09-10T10:12:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:02.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:02.620442+0000) 2022-09-10T10:12:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:03.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:03.099725+0000) 2022-09-10T10:12:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:03 smithi104 ceph-mon[97044]: pgmap v10969: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:03.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:03 smithi110 ceph-mon[97633]: pgmap v10969: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:03.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:03.620624+0000) 2022-09-10T10:12:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:04.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:04.099844+0000) 2022-09-10T10:12:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:04.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:04.620793+0000) 2022-09-10T10:12:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:05.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:05.099979+0000) 2022-09-10T10:12:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:05 smithi104 ceph-mon[97044]: pgmap v10970: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:12:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:05 smithi110 ceph-mon[97633]: pgmap v10970: 129 pgs: 129 active+clean; 8.9 MiB data, 989 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:12:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:05.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:05.621033+0000) 2022-09-10T10:12:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:06.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:06.100168+0000) 2022-09-10T10:12:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:06.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:06.621281+0000) 2022-09-10T10:12:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:07.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:07.100349+0000) 2022-09-10T10:12:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:07 smithi104 ceph-mon[97044]: pgmap v10971: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T10:12:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:07 smithi110 ceph-mon[97633]: pgmap v10971: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T10:12:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:07.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:07.621518+0000) 2022-09-10T10:12:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:08.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:08.100532+0000) 2022-09-10T10:12:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:08.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:08.623738+0000) 2022-09-10T10:12:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:12:08.679+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:12:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:09.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:09.100692+0000) 2022-09-10T10:12:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:09 smithi104 ceph-mon[97044]: pgmap v10972: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:12:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:09 smithi104 ceph-mon[97044]: osdmap e766: 12 total, 12 up, 12 in 2022-09-10T10:12:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:09 smithi110 ceph-mon[97633]: pgmap v10972: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:12:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:09 smithi110 ceph-mon[97633]: osdmap e766: 12 total, 12 up, 12 in 2022-09-10T10:12:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:09.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:09.623958+0000) 2022-09-10T10:12:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:10.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:10.100889+0000) 2022-09-10T10:12:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:10.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:10.624213+0000) 2022-09-10T10:12:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:11.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:11.101050+0000) 2022-09-10T10:12:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:11 smithi104 ceph-mon[97044]: pgmap v10974: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:12:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:11 smithi110 ceph-mon[97633]: pgmap v10974: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:12:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:11.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:11.624400+0000) 2022-09-10T10:12:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:12.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:12.101230+0000) 2022-09-10T10:12:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:12.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:12.624616+0000) 2022-09-10T10:12:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:13.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:13.101435+0000) 2022-09-10T10:12:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:13 smithi104 ceph-mon[97044]: pgmap v10975: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:12:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:13 smithi104 ceph-mon[97044]: osdmap e767: 12 total, 12 up, 12 in 2022-09-10T10:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:13 smithi110 ceph-mon[97633]: pgmap v10975: 129 pgs: 129 active+clean; 8.9 MiB data, 990 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:12:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:13 smithi110 ceph-mon[97633]: osdmap e767: 12 total, 12 up, 12 in 2022-09-10T10:12:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:13.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:13.624879+0000) 2022-09-10T10:12:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:14.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:14.101680+0000) 2022-09-10T10:12:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:14.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:14.625161+0000) 2022-09-10T10:12:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:15.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:15.101888+0000) 2022-09-10T10:12:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:15 smithi104 ceph-mon[97044]: pgmap v10977: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-10T10:12:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:15 smithi110 ceph-mon[97633]: pgmap v10977: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-10T10:12:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:15.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:15.625295+0000) 2022-09-10T10:12:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:16.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:16.102062+0000) 2022-09-10T10:12:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:16.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:16.625477+0000) 2022-09-10T10:12:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:17.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:17.102188+0000) 2022-09-10T10:12:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:17 smithi104 ceph-mon[97044]: pgmap v10978: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:12:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:17 smithi110 ceph-mon[97633]: pgmap v10978: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:12:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:17.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:17.625719+0000) 2022-09-10T10:12:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:18.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:18.102425+0000) 2022-09-10T10:12:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:18.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:18.625936+0000) 2022-09-10T10:12:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:19.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:19.102632+0000) 2022-09-10T10:12:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:19 smithi104 ceph-mon[97044]: pgmap v10979: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:12:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:19 smithi110 ceph-mon[97633]: pgmap v10979: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:12:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:19.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:19.626156+0000) 2022-09-10T10:12:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:20.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:20.102845+0000) 2022-09-10T10:12:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:20.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:20.626334+0000) 2022-09-10T10:12:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:21.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:21.102978+0000) 2022-09-10T10:12:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:21 smithi104 ceph-mon[97044]: pgmap v10980: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:12:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:21 smithi110 ceph-mon[97633]: pgmap v10980: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:12:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:21.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:21.626566+0000) 2022-09-10T10:12:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:22.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:22.103179+0000) 2022-09-10T10:12:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:22.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:22.626795+0000) 2022-09-10T10:12:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:23.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:23.103391+0000) 2022-09-10T10:12:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:23 smithi104 ceph-mon[97044]: pgmap v10981: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:12:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:23 smithi110 ceph-mon[97633]: pgmap v10981: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:12:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:23.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:23.627038+0000) 2022-09-10T10:12:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:12:23.694+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:12:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:24.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:24.103632+0000) 2022-09-10T10:12:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:24.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:24.627330+0000) 2022-09-10T10:12:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:25.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:25.103839+0000) 2022-09-10T10:12:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:25 smithi104 ceph-mon[97044]: pgmap v10982: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T10:12:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:25 smithi110 ceph-mon[97633]: pgmap v10982: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T10:12:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:25.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:25.627474+0000) 2022-09-10T10:12:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:26.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:26.104050+0000) 2022-09-10T10:12:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:26.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:26.627726+0000) 2022-09-10T10:12:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:27.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:27.104301+0000) 2022-09-10T10:12:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:27 smithi104 ceph-mon[97044]: pgmap v10983: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:12:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:27 smithi110 ceph-mon[97633]: pgmap v10983: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:12:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:27.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:27.627983+0000) 2022-09-10T10:12:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:28.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:28.104424+0000) 2022-09-10T10:12:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:28.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:28.628205+0000) 2022-09-10T10:12:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:29.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:29.104632+0000) 2022-09-10T10:12:29.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:29 smithi104 ceph-mon[97044]: pgmap v10984: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:29 smithi110 ceph-mon[97633]: pgmap v10984: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:29.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:29.628391+0000) 2022-09-10T10:12:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:30.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:30.104804+0000) 2022-09-10T10:12:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:30.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:30.628588+0000) 2022-09-10T10:12:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:31.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:31.105053+0000) 2022-09-10T10:12:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:31 smithi104 ceph-mon[97044]: pgmap v10985: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:31 smithi110 ceph-mon[97633]: pgmap v10985: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:31.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:31.628820+0000) 2022-09-10T10:12:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:32.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:32.105346+0000) 2022-09-10T10:12:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:32.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:32.629063+0000) 2022-09-10T10:12:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:33.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:33.105492+0000) 2022-09-10T10:12:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:33 smithi104 ceph-mon[97044]: pgmap v10986: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:33 smithi110 ceph-mon[97633]: pgmap v10986: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:33.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:33.629337+0000) 2022-09-10T10:12:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:34.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:34.105720+0000) 2022-09-10T10:12:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:34.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:34.629539+0000) 2022-09-10T10:12:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:35.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:35.105948+0000) 2022-09-10T10:12:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:35 smithi104 ceph-mon[97044]: pgmap v10987: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:12:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:35 smithi110 ceph-mon[97633]: pgmap v10987: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:12:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:35.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:35.629787+0000) 2022-09-10T10:12:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:36.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:36.106197+0000) 2022-09-10T10:12:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:36.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:36.629998+0000) 2022-09-10T10:12:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:37.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:37.106351+0000) 2022-09-10T10:12:37.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:37 smithi104 ceph-mon[97044]: pgmap v10988: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:12:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:37 smithi110 ceph-mon[97633]: pgmap v10988: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:12:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:37.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:37.630202+0000) 2022-09-10T10:12:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:38.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:38.106583+0000) 2022-09-10T10:12:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:38.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:38.630415+0000) 2022-09-10T10:12:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:12:38.706+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:12:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:39.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:39.106833+0000) 2022-09-10T10:12:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:39 smithi104 ceph-mon[97044]: pgmap v10989: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:39 smithi110 ceph-mon[97633]: pgmap v10989: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:39.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:39.630664+0000) 2022-09-10T10:12:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:40.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:40.107034+0000) 2022-09-10T10:12:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:40.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:40.630921+0000) 2022-09-10T10:12:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:41.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:41.107249+0000) 2022-09-10T10:12:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:41 smithi104 ceph-mon[97044]: pgmap v10990: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:41 smithi110 ceph-mon[97633]: pgmap v10990: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:41.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:41.631127+0000) 2022-09-10T10:12:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:42.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:42.107386+0000) 2022-09-10T10:12:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:42.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:42.631289+0000) 2022-09-10T10:12:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:43.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:43.107643+0000) 2022-09-10T10:12:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:43 smithi104 ceph-mon[97044]: pgmap v10991: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:43.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:43 smithi110 ceph-mon[97633]: pgmap v10991: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:43.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:43.631482+0000) 2022-09-10T10:12:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:44.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:44.107835+0000) 2022-09-10T10:12:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:44.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:44.631733+0000) 2022-09-10T10:12:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:45.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:45.108053+0000) 2022-09-10T10:12:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:45 smithi104 ceph-mon[97044]: pgmap v10992: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T10:12:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:12:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:12:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:12:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:12:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:45 smithi110 ceph-mon[97633]: pgmap v10992: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T10:12:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:12:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:12:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:12:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:12:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:45.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:45.631899+0000) 2022-09-10T10:12:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:46.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:46.108250+0000) 2022-09-10T10:12:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:46.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:46.632054+0000) 2022-09-10T10:12:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:47.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:47.108465+0000) 2022-09-10T10:12:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:47 smithi104 ceph-mon[97044]: pgmap v10993: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:12:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:47 smithi110 ceph-mon[97633]: pgmap v10993: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:12:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:47.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:47.632325+0000) 2022-09-10T10:12:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:48.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:48.108721+0000) 2022-09-10T10:12:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:48.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:48.632578+0000) 2022-09-10T10:12:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:49.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:49.108991+0000) 2022-09-10T10:12:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:49 smithi104 ceph-mon[97044]: pgmap v10994: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:49 smithi110 ceph-mon[97633]: pgmap v10994: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:49.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:49.632795+0000) 2022-09-10T10:12:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:50.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:50.109186+0000) 2022-09-10T10:12:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:50.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:50.633049+0000) 2022-09-10T10:12:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:51.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:51.109402+0000) 2022-09-10T10:12:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:51 smithi104 ceph-mon[97044]: pgmap v10995: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:51.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:51 smithi110 ceph-mon[97633]: pgmap v10995: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:12:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:51.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:51.633237+0000) 2022-09-10T10:12:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:52.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:52.109627+0000) 2022-09-10T10:12:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:52.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:52.633530+0000) 2022-09-10T10:12:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:53.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:53.109869+0000) 2022-09-10T10:12:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:53 smithi104 ceph-mon[97044]: pgmap v10996: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:53 smithi110 ceph-mon[97633]: pgmap v10996: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:54.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:12:53.722+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:12:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:53.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:53.633759+0000) 2022-09-10T10:12:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:54.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:54.110158+0000) 2022-09-10T10:12:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:54.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:54.633984+0000) 2022-09-10T10:12:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:55.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:55.110413+0000) 2022-09-10T10:12:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:55 smithi104 ceph-mon[97044]: pgmap v10997: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T10:12:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:55 smithi110 ceph-mon[97633]: pgmap v10997: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T10:12:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:55.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:55.634142+0000) 2022-09-10T10:12:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:56.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:56.110642+0000) 2022-09-10T10:12:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:56.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:56.634366+0000) 2022-09-10T10:12:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:57.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:57.110884+0000) 2022-09-10T10:12:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:57 smithi104 ceph-mon[97044]: pgmap v10998: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T10:12:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:57 smithi110 ceph-mon[97633]: pgmap v10998: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T10:12:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:57.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:57.634591+0000) 2022-09-10T10:12:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:58.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:58.111046+0000) 2022-09-10T10:12:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:58.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:58.634822+0000) 2022-09-10T10:12:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:12:59.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:59.111262+0000) 2022-09-10T10:12:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:12:59 smithi104 ceph-mon[97044]: pgmap v10999: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:12:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:12:59 smithi110 ceph-mon[97633]: pgmap v10999: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:12:59.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:12:59.635023+0000) 2022-09-10T10:13:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:00.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:00.111509+0000) 2022-09-10T10:13:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:00.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:00.635240+0000) 2022-09-10T10:13:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:01.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:01.111671+0000) 2022-09-10T10:13:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:01 smithi104 ceph-mon[97044]: pgmap v11000: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:01 smithi110 ceph-mon[97633]: pgmap v11000: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:01.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:01.635513+0000) 2022-09-10T10:13:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:02.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:02.111875+0000) 2022-09-10T10:13:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:02.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:02.635712+0000) 2022-09-10T10:13:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:03.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:03.112076+0000) 2022-09-10T10:13:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:03 smithi104 ceph-mon[97044]: pgmap v11001: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:03 smithi110 ceph-mon[97633]: pgmap v11001: 129 pgs: 129 active+clean; 8.9 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:03.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:03.635951+0000) 2022-09-10T10:13:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:04.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:04.112264+0000) 2022-09-10T10:13:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:04.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:04.636213+0000) 2022-09-10T10:13:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:05.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:05.112453+0000) 2022-09-10T10:13:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:05 smithi104 ceph-mon[97044]: pgmap v11002: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T10:13:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:05 smithi110 ceph-mon[97633]: pgmap v11002: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T10:13:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:05.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:05.636395+0000) 2022-09-10T10:13:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:06.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:06.112616+0000) 2022-09-10T10:13:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:06.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:06.636661+0000) 2022-09-10T10:13:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:07.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:07.112869+0000) 2022-09-10T10:13:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:07 smithi104 ceph-mon[97044]: pgmap v11003: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T10:13:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:07 smithi110 ceph-mon[97633]: pgmap v11003: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T10:13:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:07.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:07.636896+0000) 2022-09-10T10:13:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:08.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:08.113170+0000) 2022-09-10T10:13:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:08.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:08.637135+0000) 2022-09-10T10:13:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:13:08.731+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:13:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:09.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:09.113304+0000) 2022-09-10T10:13:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:09 smithi104 ceph-mon[97044]: pgmap v11004: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:13:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:09 smithi104 ceph-mon[97044]: osdmap e768: 12 total, 12 up, 12 in 2022-09-10T10:13:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:09 smithi110 ceph-mon[97633]: pgmap v11004: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:13:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:09 smithi110 ceph-mon[97633]: osdmap e768: 12 total, 12 up, 12 in 2022-09-10T10:13:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:09.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:09.637264+0000) 2022-09-10T10:13:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:10.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:10.113519+0000) 2022-09-10T10:13:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:10.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:10.637413+0000) 2022-09-10T10:13:11.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:11.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:11.113695+0000) 2022-09-10T10:13:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:11 smithi104 ceph-mon[97044]: pgmap v11006: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:13:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:11 smithi110 ceph-mon[97633]: pgmap v11006: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:13:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:11.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:11.637588+0000) 2022-09-10T10:13:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:12.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:12.113927+0000) 2022-09-10T10:13:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:12.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:12.637838+0000) 2022-09-10T10:13:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:13.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:13.114209+0000) 2022-09-10T10:13:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:13 smithi104 ceph-mon[97044]: pgmap v11007: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:13:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:13 smithi104 ceph-mon[97044]: osdmap e769: 12 total, 12 up, 12 in 2022-09-10T10:13:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:13 smithi110 ceph-mon[97633]: pgmap v11007: 129 pgs: 129 active+clean; 8.9 MiB data, 992 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:13:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:13 smithi110 ceph-mon[97633]: osdmap e769: 12 total, 12 up, 12 in 2022-09-10T10:13:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:13.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:13.638038+0000) 2022-09-10T10:13:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:14.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:14.114394+0000) 2022-09-10T10:13:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:14.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:14.638226+0000) 2022-09-10T10:13:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:15.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:15.114611+0000) 2022-09-10T10:13:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:15 smithi104 ceph-mon[97044]: pgmap v11009: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 9 op/s 2022-09-10T10:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:15 smithi110 ceph-mon[97633]: pgmap v11009: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 9 op/s 2022-09-10T10:13:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:15.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:15.638443+0000) 2022-09-10T10:13:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:16.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:16.114853+0000) 2022-09-10T10:13:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:16.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:16.638678+0000) 2022-09-10T10:13:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:17.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:17.115064+0000) 2022-09-10T10:13:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:17 smithi104 ceph-mon[97044]: pgmap v11010: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:13:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:17 smithi110 ceph-mon[97633]: pgmap v11010: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:13:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:17.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:17.638932+0000) 2022-09-10T10:13:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:18.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:18.115210+0000) 2022-09-10T10:13:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:18.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:18.639191+0000) 2022-09-10T10:13:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:19.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:19.115444+0000) 2022-09-10T10:13:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:19 smithi104 ceph-mon[97044]: pgmap v11011: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:13:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:19 smithi110 ceph-mon[97633]: pgmap v11011: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:13:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:19.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:19.639351+0000) 2022-09-10T10:13:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:20.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:20.115674+0000) 2022-09-10T10:13:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:20.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:20.639583+0000) 2022-09-10T10:13:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:21.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:21.115889+0000) 2022-09-10T10:13:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:21 smithi104 ceph-mon[97044]: pgmap v11012: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:13:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:21 smithi110 ceph-mon[97633]: pgmap v11012: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:13:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:21.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:21.639810+0000) 2022-09-10T10:13:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:22.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:22.116125+0000) 2022-09-10T10:13:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:22.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:22.640072+0000) 2022-09-10T10:13:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:23.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:23.116383+0000) 2022-09-10T10:13:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:23 smithi104 ceph-mon[97044]: pgmap v11013: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:13:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:23 smithi110 ceph-mon[97633]: pgmap v11013: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:13:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:23.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:23.640311+0000) 2022-09-10T10:13:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:13:23.746+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:13:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:24.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:24.116527+0000) 2022-09-10T10:13:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:24.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:24.640562+0000) 2022-09-10T10:13:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:25.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:25.116750+0000) 2022-09-10T10:13:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:25 smithi104 ceph-mon[97044]: pgmap v11014: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T10:13:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:25 smithi110 ceph-mon[97633]: pgmap v11014: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T10:13:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:25.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:25.640809+0000) 2022-09-10T10:13:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:26.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:26.116991+0000) 2022-09-10T10:13:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:26.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:26.641007+0000) 2022-09-10T10:13:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:27.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:27.117228+0000) 2022-09-10T10:13:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:27 smithi104 ceph-mon[97044]: pgmap v11015: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T10:13:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:27 smithi110 ceph-mon[97633]: pgmap v11015: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T10:13:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:27.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:27.641263+0000) 2022-09-10T10:13:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:28.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:28.117403+0000) 2022-09-10T10:13:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:28.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:28.641440+0000) 2022-09-10T10:13:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:29.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:29.117666+0000) 2022-09-10T10:13:29.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:29 smithi104 ceph-mon[97044]: pgmap v11016: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:29 smithi110 ceph-mon[97633]: pgmap v11016: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:29.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:29.641654+0000) 2022-09-10T10:13:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:30.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:30.117864+0000) 2022-09-10T10:13:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:30.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:30.641878+0000) 2022-09-10T10:13:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:31.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:31.118091+0000) 2022-09-10T10:13:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:31 smithi104 ceph-mon[97044]: pgmap v11017: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:31 smithi110 ceph-mon[97633]: pgmap v11017: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:31.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:31.642159+0000) 2022-09-10T10:13:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:32.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:32.118307+0000) 2022-09-10T10:13:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:32.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:32.642341+0000) 2022-09-10T10:13:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:33.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:33.118487+0000) 2022-09-10T10:13:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:33 smithi104 ceph-mon[97044]: pgmap v11018: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:33 smithi110 ceph-mon[97633]: pgmap v11018: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:33.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:33.642526+0000) 2022-09-10T10:13:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:34.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:34.118735+0000) 2022-09-10T10:13:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:34.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:34.642707+0000) 2022-09-10T10:13:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:35.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:35.118939+0000) 2022-09-10T10:13:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:35 smithi104 ceph-mon[97044]: pgmap v11019: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T10:13:35.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:35 smithi110 ceph-mon[97633]: pgmap v11019: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T10:13:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:35.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:35.642856+0000) 2022-09-10T10:13:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:36.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:36.119168+0000) 2022-09-10T10:13:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:36.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:36.643017+0000) 2022-09-10T10:13:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:37.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:37.119388+0000) 2022-09-10T10:13:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:37 smithi104 ceph-mon[97044]: pgmap v11020: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T10:13:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:37 smithi110 ceph-mon[97633]: pgmap v11020: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T10:13:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:37.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:37.643212+0000) 2022-09-10T10:13:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:38.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:38.119659+0000) 2022-09-10T10:13:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:38.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:38.643398+0000) 2022-09-10T10:13:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:13:38.762+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:13:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:39.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:39.119863+0000) 2022-09-10T10:13:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:39 smithi104 ceph-mon[97044]: pgmap v11021: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:39 smithi110 ceph-mon[97633]: pgmap v11021: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:39.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:39.643640+0000) 2022-09-10T10:13:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:40.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:40.120043+0000) 2022-09-10T10:13:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:40.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:40.643918+0000) 2022-09-10T10:13:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:41.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:41.120231+0000) 2022-09-10T10:13:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:41 smithi104 ceph-mon[97044]: pgmap v11022: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:41 smithi110 ceph-mon[97633]: pgmap v11022: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:41.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:41.644156+0000) 2022-09-10T10:13:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:42.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:42.120440+0000) 2022-09-10T10:13:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:42.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:42.644370+0000) 2022-09-10T10:13:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:43.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:43.120703+0000) 2022-09-10T10:13:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:43 smithi104 ceph-mon[97044]: pgmap v11023: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:43 smithi110 ceph-mon[97633]: pgmap v11023: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:43.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:43.644553+0000) 2022-09-10T10:13:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:44.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:44.120927+0000) 2022-09-10T10:13:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:44.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:44.644785+0000) 2022-09-10T10:13:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:45.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:45.121166+0000) 2022-09-10T10:13:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:45 smithi104 ceph-mon[97044]: pgmap v11024: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:13:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:13:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:13:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:13:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:45 smithi110 ceph-mon[97633]: pgmap v11024: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:13:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:13:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:13:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:13:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:45.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:45.645019+0000) 2022-09-10T10:13:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:46.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:46.121383+0000) 2022-09-10T10:13:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:46.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:46.645222+0000) 2022-09-10T10:13:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:47.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:47.121595+0000) 2022-09-10T10:13:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:47 smithi104 ceph-mon[97044]: pgmap v11025: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T10:13:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:47 smithi110 ceph-mon[97633]: pgmap v11025: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T10:13:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:47.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:47.645443+0000) 2022-09-10T10:13:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:48.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:48.121861+0000) 2022-09-10T10:13:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:48.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:48.645671+0000) 2022-09-10T10:13:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:49.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:49.122047+0000) 2022-09-10T10:13:49.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:49 smithi104 ceph-mon[97044]: pgmap v11026: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:13:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:49 smithi110 ceph-mon[97633]: pgmap v11026: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:13:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:49.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:49.645919+0000) 2022-09-10T10:13:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:50.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:50.122228+0000) 2022-09-10T10:13:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:50.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:50.646173+0000) 2022-09-10T10:13:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:51.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:51.122416+0000) 2022-09-10T10:13:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:51 smithi104 ceph-mon[97044]: pgmap v11027: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:51 smithi110 ceph-mon[97633]: pgmap v11027: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:51.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:51.646327+0000) 2022-09-10T10:13:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:52.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:52.122652+0000) 2022-09-10T10:13:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:52.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:52.646489+0000) 2022-09-10T10:13:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:53.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:53.122896+0000) 2022-09-10T10:13:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:53 smithi104 ceph-mon[97044]: pgmap v11028: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:13:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:53 smithi110 ceph-mon[97633]: pgmap v11028: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:13:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:53.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:53.646748+0000) 2022-09-10T10:13:54.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:13:53.778+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:13:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:54.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:54.123169+0000) 2022-09-10T10:13:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:54.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:54.646907+0000) 2022-09-10T10:13:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:55.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:55.123399+0000) 2022-09-10T10:13:55.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:55 smithi104 ceph-mon[97044]: pgmap v11029: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:55 smithi110 ceph-mon[97633]: pgmap v11029: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:13:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:55.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:55.647068+0000) 2022-09-10T10:13:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:56.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:56.123592+0000) 2022-09-10T10:13:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:56.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:56.647315+0000) 2022-09-10T10:13:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:57.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:57.123838+0000) 2022-09-10T10:13:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:57 smithi104 ceph-mon[97044]: pgmap v11030: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:57 smithi110 ceph-mon[97633]: pgmap v11030: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:57.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:57.647532+0000) 2022-09-10T10:13:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:58.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:58.124130+0000) 2022-09-10T10:13:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:58.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:58.647759+0000) 2022-09-10T10:13:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:13:59.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:59.124396+0000) 2022-09-10T10:13:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:13:59 smithi104 ceph-mon[97044]: pgmap v11031: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:13:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:13:59 smithi110 ceph-mon[97633]: pgmap v11031: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:13:59.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:13:59.648020+0000) 2022-09-10T10:14:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:00.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:00.124533+0000) 2022-09-10T10:14:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:00.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:00.648263+0000) 2022-09-10T10:14:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:01.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:01.124692+0000) 2022-09-10T10:14:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:01 smithi104 ceph-mon[97044]: pgmap v11032: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:01 smithi110 ceph-mon[97633]: pgmap v11032: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:01.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:01.648457+0000) 2022-09-10T10:14:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:02.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:02.124925+0000) 2022-09-10T10:14:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:02.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:02.648644+0000) 2022-09-10T10:14:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:03.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:03.125189+0000) 2022-09-10T10:14:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:03 smithi104 ceph-mon[97044]: pgmap v11033: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:03 smithi110 ceph-mon[97633]: pgmap v11033: 129 pgs: 129 active+clean; 8.9 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:03.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:03.648877+0000) 2022-09-10T10:14:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:04.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:04.125473+0000) 2022-09-10T10:14:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:04.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:04.649157+0000) 2022-09-10T10:14:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:05.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:05.125640+0000) 2022-09-10T10:14:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:05 smithi104 ceph-mon[97044]: pgmap v11034: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:14:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:05 smithi110 ceph-mon[97633]: pgmap v11034: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:14:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:05.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:05.649361+0000) 2022-09-10T10:14:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:06.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:06.125783+0000) 2022-09-10T10:14:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:06.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:06.649575+0000) 2022-09-10T10:14:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:07.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:07.126003+0000) 2022-09-10T10:14:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:07 smithi110 ceph-mon[97633]: pgmap v11035: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:14:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:07 smithi104 ceph-mon[97044]: pgmap v11035: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:14:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:07.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:07.649837+0000) 2022-09-10T10:14:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:08.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:08.126204+0000) 2022-09-10T10:14:09.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:08.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:08.649981+0000) 2022-09-10T10:14:09.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:14:08.782+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:14:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:09.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:09.126334+0000) 2022-09-10T10:14:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:09 smithi104 ceph-mon[97044]: pgmap v11036: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:14:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:09 smithi104 ceph-mon[97044]: osdmap e770: 12 total, 12 up, 12 in 2022-09-10T10:14:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:09 smithi110 ceph-mon[97633]: pgmap v11036: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:14:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:09 smithi110 ceph-mon[97633]: osdmap e770: 12 total, 12 up, 12 in 2022-09-10T10:14:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:09.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:09.650224+0000) 2022-09-10T10:14:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:10.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:10.126512+0000) 2022-09-10T10:14:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:10.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:10.650404+0000) 2022-09-10T10:14:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:11.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:11.126780+0000) 2022-09-10T10:14:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:11 smithi104 ceph-mon[97044]: pgmap v11038: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:14:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:11 smithi110 ceph-mon[97633]: pgmap v11038: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:14:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:11.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:11.650562+0000) 2022-09-10T10:14:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:12.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:12.127021+0000) 2022-09-10T10:14:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:12.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:12.650782+0000) 2022-09-10T10:14:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:13.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:13.127244+0000) 2022-09-10T10:14:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:13 smithi104 ceph-mon[97044]: pgmap v11039: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:14:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:13 smithi104 ceph-mon[97044]: osdmap e771: 12 total, 12 up, 12 in 2022-09-10T10:14:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:13 smithi110 ceph-mon[97633]: pgmap v11039: 129 pgs: 129 active+clean; 8.9 MiB data, 995 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:14:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:13 smithi110 ceph-mon[97633]: osdmap e771: 12 total, 12 up, 12 in 2022-09-10T10:14:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:13.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:13.651029+0000) 2022-09-10T10:14:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:14.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:14.127504+0000) 2022-09-10T10:14:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:14.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:14.651207+0000) 2022-09-10T10:14:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:15.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:15.127735+0000) 2022-09-10T10:14:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:15 smithi104 ceph-mon[97044]: pgmap v11041: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:14:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:15 smithi110 ceph-mon[97633]: pgmap v11041: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:14:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:15.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:15.651378+0000) 2022-09-10T10:14:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:16.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:16.127993+0000) 2022-09-10T10:14:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:16.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:16.651662+0000) 2022-09-10T10:14:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:17.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:17.128242+0000) 2022-09-10T10:14:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:17 smithi104 ceph-mon[97044]: pgmap v11042: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:14:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:17 smithi110 ceph-mon[97633]: pgmap v11042: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:14:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:17.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:17.651915+0000) 2022-09-10T10:14:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:18.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:18.128446+0000) 2022-09-10T10:14:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:18.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:18.652144+0000) 2022-09-10T10:14:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:19.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:19.128648+0000) 2022-09-10T10:14:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:19 smithi104 ceph-mon[97044]: pgmap v11043: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:14:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:19 smithi110 ceph-mon[97633]: pgmap v11043: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T10:14:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:19.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:19.652296+0000) 2022-09-10T10:14:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:20.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:20.128831+0000) 2022-09-10T10:14:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:20.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:20.652447+0000) 2022-09-10T10:14:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:21.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:21.129000+0000) 2022-09-10T10:14:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:21 smithi104 ceph-mon[97044]: pgmap v11044: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:14:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:21 smithi110 ceph-mon[97633]: pgmap v11044: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:14:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:21.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:21.652603+0000) 2022-09-10T10:14:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:22.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:22.129202+0000) 2022-09-10T10:14:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:22.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:22.652770+0000) 2022-09-10T10:14:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:23.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:23.129394+0000) 2022-09-10T10:14:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:23 smithi104 ceph-mon[97044]: pgmap v11045: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:14:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:23 smithi110 ceph-mon[97633]: pgmap v11045: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:14:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:23.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:23.652960+0000) 2022-09-10T10:14:24.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:14:23.798+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:14:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:24.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:24.129660+0000) 2022-09-10T10:14:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:24.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:24.653199+0000) 2022-09-10T10:14:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:25.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:25.129907+0000) 2022-09-10T10:14:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:25 smithi104 ceph-mon[97044]: pgmap v11046: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:14:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:25 smithi110 ceph-mon[97633]: pgmap v11046: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:14:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:25.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:25.653381+0000) 2022-09-10T10:14:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:26.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:26.130030+0000) 2022-09-10T10:14:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:26.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:26.653558+0000) 2022-09-10T10:14:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:27.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:27.130246+0000) 2022-09-10T10:14:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:27 smithi104 ceph-mon[97044]: pgmap v11047: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:27 smithi110 ceph-mon[97633]: pgmap v11047: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:27.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:27.653742+0000) 2022-09-10T10:14:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:28.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:28.130404+0000) 2022-09-10T10:14:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:28.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:28.653924+0000) 2022-09-10T10:14:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:29.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:29.130658+0000) 2022-09-10T10:14:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:29 smithi104 ceph-mon[97044]: pgmap v11048: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:29 smithi110 ceph-mon[97633]: pgmap v11048: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:29.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:29.654168+0000) 2022-09-10T10:14:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:30.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:30.130931+0000) 2022-09-10T10:14:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:30.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:30.654349+0000) 2022-09-10T10:14:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:31.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:31.131215+0000) 2022-09-10T10:14:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:31 smithi104 ceph-mon[97044]: pgmap v11049: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:31 smithi110 ceph-mon[97633]: pgmap v11049: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:31.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:31.654572+0000) 2022-09-10T10:14:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:32.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:32.131415+0000) 2022-09-10T10:14:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:32.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:32.654836+0000) 2022-09-10T10:14:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:33.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:33.131655+0000) 2022-09-10T10:14:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:33 smithi104 ceph-mon[97044]: pgmap v11050: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:33 smithi110 ceph-mon[97633]: pgmap v11050: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:33.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:33.655048+0000) 2022-09-10T10:14:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:34.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:34.131930+0000) 2022-09-10T10:14:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:34.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:34.655311+0000) 2022-09-10T10:14:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:35.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:35.132143+0000) 2022-09-10T10:14:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:35 smithi104 ceph-mon[97044]: pgmap v11051: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:35 smithi110 ceph-mon[97633]: pgmap v11051: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:35.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:35.655500+0000) 2022-09-10T10:14:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:36.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:36.132367+0000) 2022-09-10T10:14:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:36.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:36.655661+0000) 2022-09-10T10:14:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:37.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:37.132572+0000) 2022-09-10T10:14:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:37 smithi104 ceph-mon[97044]: pgmap v11052: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:37 smithi110 ceph-mon[97633]: pgmap v11052: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:37.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:37.655954+0000) 2022-09-10T10:14:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:38.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:38.132800+0000) 2022-09-10T10:14:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:38.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:38.656197+0000) 2022-09-10T10:14:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:14:38.809+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:14:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:39.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:39.132977+0000) 2022-09-10T10:14:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:39 smithi104 ceph-mon[97044]: pgmap v11053: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:39 smithi110 ceph-mon[97633]: pgmap v11053: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:39.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:39.656386+0000) 2022-09-10T10:14:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:40.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:40.133227+0000) 2022-09-10T10:14:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:40.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:40.656619+0000) 2022-09-10T10:14:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:41.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:41.133441+0000) 2022-09-10T10:14:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:41 smithi104 ceph-mon[97044]: pgmap v11054: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:41 smithi110 ceph-mon[97633]: pgmap v11054: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:41.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:41.656866+0000) 2022-09-10T10:14:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:42.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:42.133612+0000) 2022-09-10T10:14:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:42.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:42.657151+0000) 2022-09-10T10:14:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:43.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:43.133845+0000) 2022-09-10T10:14:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:43 smithi104 ceph-mon[97044]: pgmap v11055: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:43 smithi110 ceph-mon[97633]: pgmap v11055: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:43.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:43.657349+0000) 2022-09-10T10:14:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:44.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:44.134043+0000) 2022-09-10T10:14:45.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:44.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:44.657534+0000) 2022-09-10T10:14:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:45.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:45.134197+0000) 2022-09-10T10:14:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:45 smithi104 ceph-mon[97044]: pgmap v11056: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:14:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:14:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:14:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:14:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:45 smithi110 ceph-mon[97633]: pgmap v11056: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:14:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:14:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:14:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:14:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:45.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:45.657705+0000) 2022-09-10T10:14:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:46.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:46.134335+0000) 2022-09-10T10:14:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:46.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:46.657949+0000) 2022-09-10T10:14:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:47.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:47.134565+0000) 2022-09-10T10:14:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:47 smithi104 ceph-mon[97044]: pgmap v11057: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:47 smithi110 ceph-mon[97633]: pgmap v11057: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:47.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:47.658168+0000) 2022-09-10T10:14:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:48.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:48.134843+0000) 2022-09-10T10:14:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:48.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:48.658413+0000) 2022-09-10T10:14:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:49.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:49.135083+0000) 2022-09-10T10:14:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:49 smithi104 ceph-mon[97044]: pgmap v11058: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:49 smithi110 ceph-mon[97633]: pgmap v11058: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:49.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:49.658636+0000) 2022-09-10T10:14:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:50.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:50.135329+0000) 2022-09-10T10:14:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:50.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:50.658902+0000) 2022-09-10T10:14:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:51.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:51.135544+0000) 2022-09-10T10:14:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:51 smithi104 ceph-mon[97044]: pgmap v11059: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:51 smithi110 ceph-mon[97633]: pgmap v11059: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:51.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:51.659090+0000) 2022-09-10T10:14:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:52.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:52.135672+0000) 2022-09-10T10:14:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:52.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:52.659309+0000) 2022-09-10T10:14:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:53.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:53.135913+0000) 2022-09-10T10:14:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:53 smithi104 ceph-mon[97044]: pgmap v11060: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:53 smithi110 ceph-mon[97633]: pgmap v11060: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:53.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:53.659543+0000) 2022-09-10T10:14:54.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:14:53.812+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:14:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:54.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:54.136215+0000) 2022-09-10T10:14:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:54.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:54.659762+0000) 2022-09-10T10:14:55.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:55.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:55.136464+0000) 2022-09-10T10:14:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:55 smithi104 ceph-mon[97044]: pgmap v11061: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:55 smithi110 ceph-mon[97633]: pgmap v11061: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:14:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:55.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:55.659966+0000) 2022-09-10T10:14:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:56.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:56.136618+0000) 2022-09-10T10:14:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:56.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:56.660200+0000) 2022-09-10T10:14:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:57.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:57.136809+0000) 2022-09-10T10:14:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:57 smithi110 ceph-mon[97633]: pgmap v11062: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:57 smithi104 ceph-mon[97044]: pgmap v11062: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:14:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:57.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:57.660438+0000) 2022-09-10T10:14:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:58.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:58.137030+0000) 2022-09-10T10:14:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:58.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:58.660640+0000) 2022-09-10T10:14:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:14:59.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:59.137281+0000) 2022-09-10T10:14:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:14:59 smithi110 ceph-mon[97633]: pgmap v11063: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:14:59 smithi104 ceph-mon[97044]: pgmap v11063: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:14:59.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:14:59.660801+0000) 2022-09-10T10:15:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:00.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:00.137513+0000) 2022-09-10T10:15:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:00.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:00.660992+0000) 2022-09-10T10:15:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:01.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:01.137679+0000) 2022-09-10T10:15:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:01 smithi110 ceph-mon[97633]: pgmap v11064: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:01 smithi104 ceph-mon[97044]: pgmap v11064: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:01.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:01.661191+0000) 2022-09-10T10:15:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:02.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:02.137897+0000) 2022-09-10T10:15:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:02.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:02.661400+0000) 2022-09-10T10:15:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:03.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:03.138217+0000) 2022-09-10T10:15:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:03 smithi110 ceph-mon[97633]: pgmap v11065: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:03 smithi104 ceph-mon[97044]: pgmap v11065: 129 pgs: 129 active+clean; 8.9 MiB data, 996 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:03.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:03.661612+0000) 2022-09-10T10:15:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:04.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:04.138462+0000) 2022-09-10T10:15:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:04.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:04.661865+0000) 2022-09-10T10:15:05.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:05.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:05.138683+0000) 2022-09-10T10:15:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:05 smithi110 ceph-mon[97633]: pgmap v11066: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:15:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:05 smithi104 ceph-mon[97044]: pgmap v11066: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:15:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:05.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:05.662075+0000) 2022-09-10T10:15:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:06.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:06.138941+0000) 2022-09-10T10:15:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:06.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:06.662296+0000) 2022-09-10T10:15:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:07.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:07.139185+0000) 2022-09-10T10:15:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:07 smithi110 ceph-mon[97633]: pgmap v11067: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:15:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:07 smithi104 ceph-mon[97044]: pgmap v11067: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:15:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:07.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:07.662564+0000) 2022-09-10T10:15:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:08.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:08.139374+0000) 2022-09-10T10:15:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:08 smithi110 ceph-mon[97633]: osdmap e772: 12 total, 12 up, 12 in 2022-09-10T10:15:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:08 smithi104 ceph-mon[97044]: osdmap e772: 12 total, 12 up, 12 in 2022-09-10T10:15:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:08.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:08.662760+0000) 2022-09-10T10:15:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:15:08.821+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:15:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:09.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:09.139587+0000) 2022-09-10T10:15:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:09 smithi110 ceph-mon[97633]: pgmap v11069: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:15:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:09 smithi104 ceph-mon[97044]: pgmap v11069: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:15:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:09.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:09.662981+0000) 2022-09-10T10:15:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:10.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:10.139810+0000) 2022-09-10T10:15:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:10.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:10.663243+0000) 2022-09-10T10:15:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:11.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:11.140040+0000) 2022-09-10T10:15:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:11 smithi110 ceph-mon[97633]: pgmap v11070: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:15:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:11 smithi104 ceph-mon[97044]: pgmap v11070: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:15:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:11.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:11.663493+0000) 2022-09-10T10:15:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:12.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:12.140268+0000) 2022-09-10T10:15:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:12.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:12.663738+0000) 2022-09-10T10:15:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:13.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:13.140480+0000) 2022-09-10T10:15:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:13 smithi104 ceph-mon[97044]: pgmap v11071: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:15:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:13 smithi104 ceph-mon[97044]: osdmap e773: 12 total, 12 up, 12 in 2022-09-10T10:15:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:13 smithi110 ceph-mon[97633]: pgmap v11071: 129 pgs: 129 active+clean; 8.9 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:15:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:13 smithi110 ceph-mon[97633]: osdmap e773: 12 total, 12 up, 12 in 2022-09-10T10:15:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:13.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:13.664007+0000) 2022-09-10T10:15:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:14.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:14.140650+0000) 2022-09-10T10:15:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:14.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:14.664239+0000) 2022-09-10T10:15:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:15.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:15.140899+0000) 2022-09-10T10:15:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:15 smithi104 ceph-mon[97044]: pgmap v11073: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:15:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:15 smithi110 ceph-mon[97633]: pgmap v11073: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:15:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:15.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:15.664493+0000) 2022-09-10T10:15:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:16.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:16.141155+0000) 2022-09-10T10:15:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:16.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:16.664744+0000) 2022-09-10T10:15:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:17.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:17.141336+0000) 2022-09-10T10:15:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:17 smithi104 ceph-mon[97044]: pgmap v11074: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:15:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:17 smithi110 ceph-mon[97633]: pgmap v11074: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:15:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:17.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:17.664963+0000) 2022-09-10T10:15:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:18.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:18.141567+0000) 2022-09-10T10:15:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:18.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:18.665194+0000) 2022-09-10T10:15:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:19.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:19.141808+0000) 2022-09-10T10:15:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:19 smithi104 ceph-mon[97044]: pgmap v11075: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:15:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:19 smithi110 ceph-mon[97633]: pgmap v11075: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:15:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:19.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:19.665440+0000) 2022-09-10T10:15:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:20.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:20.142029+0000) 2022-09-10T10:15:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:20.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:20.665614+0000) 2022-09-10T10:15:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:21.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:21.142228+0000) 2022-09-10T10:15:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:21 smithi104 ceph-mon[97044]: pgmap v11076: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:15:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:21 smithi110 ceph-mon[97633]: pgmap v11076: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:15:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:21.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:21.665834+0000) 2022-09-10T10:15:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:22.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:22.142454+0000) 2022-09-10T10:15:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:22.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:22.666081+0000) 2022-09-10T10:15:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:23.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:23.142722+0000) 2022-09-10T10:15:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:23 smithi104 ceph-mon[97044]: pgmap v11077: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:15:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:23 smithi110 ceph-mon[97633]: pgmap v11077: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:15:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:23.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:23.666294+0000) 2022-09-10T10:15:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:15:23.833+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:15:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:24.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:24.142987+0000) 2022-09-10T10:15:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:24.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:24.666521+0000) 2022-09-10T10:15:25.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:25.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:25.143223+0000) 2022-09-10T10:15:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:25 smithi104 ceph-mon[97044]: pgmap v11078: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:15:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:25 smithi110 ceph-mon[97633]: pgmap v11078: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:15:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:25.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:25.666735+0000) 2022-09-10T10:15:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:26.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:26.143383+0000) 2022-09-10T10:15:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:26.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:26.666944+0000) 2022-09-10T10:15:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:27.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:27.143573+0000) 2022-09-10T10:15:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:27 smithi104 ceph-mon[97044]: pgmap v11079: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:27 smithi110 ceph-mon[97633]: pgmap v11079: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:27.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:27.667143+0000) 2022-09-10T10:15:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:28.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:28.143836+0000) 2022-09-10T10:15:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:28.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:28.667326+0000) 2022-09-10T10:15:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:29.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:29.144051+0000) 2022-09-10T10:15:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:29 smithi104 ceph-mon[97044]: pgmap v11080: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:29 smithi110 ceph-mon[97633]: pgmap v11080: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:29.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:29.667507+0000) 2022-09-10T10:15:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:30.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:30.144287+0000) 2022-09-10T10:15:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:30.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:30.667757+0000) 2022-09-10T10:15:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:31.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:31.144476+0000) 2022-09-10T10:15:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:31 smithi104 ceph-mon[97044]: pgmap v11081: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:31 smithi110 ceph-mon[97633]: pgmap v11081: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:32.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:31.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:31.667985+0000) 2022-09-10T10:15:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:32.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:32.144729+0000) 2022-09-10T10:15:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:32.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:32.668230+0000) 2022-09-10T10:15:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:33.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:33.144883+0000) 2022-09-10T10:15:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:33 smithi104 ceph-mon[97044]: pgmap v11082: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:33 smithi110 ceph-mon[97633]: pgmap v11082: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:33.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:33.668403+0000) 2022-09-10T10:15:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:34.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:34.145087+0000) 2022-09-10T10:15:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:34.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:34.668572+0000) 2022-09-10T10:15:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:35.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:35.145280+0000) 2022-09-10T10:15:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:35 smithi104 ceph-mon[97044]: pgmap v11083: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:35 smithi110 ceph-mon[97633]: pgmap v11083: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:36.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:35.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:35.668775+0000) 2022-09-10T10:15:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:36.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:36.145458+0000) 2022-09-10T10:15:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:36.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:36.669023+0000) 2022-09-10T10:15:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:37.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:37.145730+0000) 2022-09-10T10:15:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:37 smithi104 ceph-mon[97044]: pgmap v11084: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:37 smithi110 ceph-mon[97633]: pgmap v11084: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:37.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:37.669335+0000) 2022-09-10T10:15:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:38.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:38.145984+0000) 2022-09-10T10:15:39.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:15:38.849+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:15:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:38.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:38.669500+0000) 2022-09-10T10:15:39.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:39.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:39.146194+0000) 2022-09-10T10:15:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:39 smithi104 ceph-mon[97044]: pgmap v11085: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:39 smithi110 ceph-mon[97633]: pgmap v11085: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:39.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:39.669710+0000) 2022-09-10T10:15:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:40.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:40.146351+0000) 2022-09-10T10:15:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:40.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:40.669974+0000) 2022-09-10T10:15:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:41.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:41.146498+0000) 2022-09-10T10:15:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:41 smithi104 ceph-mon[97044]: pgmap v11086: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:41 smithi110 ceph-mon[97633]: pgmap v11086: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:41.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:41.670207+0000) 2022-09-10T10:15:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:42.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:42.146729+0000) 2022-09-10T10:15:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:42.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:42.670435+0000) 2022-09-10T10:15:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:43.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:43.146937+0000) 2022-09-10T10:15:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:43 smithi104 ceph-mon[97044]: pgmap v11087: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:43 smithi110 ceph-mon[97633]: pgmap v11087: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:43.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:43.670637+0000) 2022-09-10T10:15:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:44.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:44.147196+0000) 2022-09-10T10:15:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:44.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:44.670828+0000) 2022-09-10T10:15:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:45.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:45.147325+0000) 2022-09-10T10:15:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:45 smithi104 ceph-mon[97044]: pgmap v11088: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:15:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:15:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:15:45.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:15:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:45 smithi110 ceph-mon[97633]: pgmap v11088: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:15:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:45.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:45.671027+0000) 2022-09-10T10:15:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:46.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:46.147460+0000) 2022-09-10T10:15:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:46.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:46.671261+0000) 2022-09-10T10:15:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:47.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:47.147684+0000) 2022-09-10T10:15:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:47 smithi104 ceph-mon[97044]: pgmap v11089: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:47 smithi110 ceph-mon[97633]: pgmap v11089: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:47.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:47.671473+0000) 2022-09-10T10:15:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:48.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:48.147939+0000) 2022-09-10T10:15:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:48.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:48.671701+0000) 2022-09-10T10:15:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:49.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:49.148183+0000) 2022-09-10T10:15:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:49 smithi110 ceph-mon[97633]: pgmap v11090: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:49 smithi104 ceph-mon[97044]: pgmap v11090: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:49.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:49.671940+0000) 2022-09-10T10:15:50.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:50.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:50.148396+0000) 2022-09-10T10:15:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:50.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:50.672177+0000) 2022-09-10T10:15:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:51.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:51.148667+0000) 2022-09-10T10:15:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:51 smithi110 ceph-mon[97633]: pgmap v11091: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:51 smithi104 ceph-mon[97044]: pgmap v11091: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:51.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:51.672302+0000) 2022-09-10T10:15:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:52.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:52.148931+0000) 2022-09-10T10:15:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:52.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:52.672508+0000) 2022-09-10T10:15:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:53.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:53.149189+0000) 2022-09-10T10:15:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:53 smithi110 ceph-mon[97633]: pgmap v11092: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:53 smithi104 ceph-mon[97044]: pgmap v11092: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:53.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:53.672769+0000) 2022-09-10T10:15:54.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:15:53.860+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:15:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:54.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:54.149360+0000) 2022-09-10T10:15:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:54.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:54.672953+0000) 2022-09-10T10:15:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:55.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:55.149545+0000) 2022-09-10T10:15:55.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:55 smithi110 ceph-mon[97633]: pgmap v11093: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:56.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:55 smithi104 ceph-mon[97044]: pgmap v11093: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:15:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:55.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:55.673198+0000) 2022-09-10T10:15:56.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:56.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:56.149742+0000) 2022-09-10T10:15:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:56.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:56.673432+0000) 2022-09-10T10:15:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:57.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:57.149968+0000) 2022-09-10T10:15:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:57 smithi110 ceph-mon[97633]: pgmap v11094: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:57 smithi104 ceph-mon[97044]: pgmap v11094: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:15:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:57.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:57.673665+0000) 2022-09-10T10:15:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:58.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:58.150247+0000) 2022-09-10T10:15:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:58.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:58.673934+0000) 2022-09-10T10:15:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:15:59.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:59.150469+0000) 2022-09-10T10:15:59.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:15:59 smithi110 ceph-mon[97633]: pgmap v11095: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:15:59 smithi104 ceph-mon[97044]: pgmap v11095: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:15:59.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:15:59.674163+0000) 2022-09-10T10:16:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:00.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:00.150689+0000) 2022-09-10T10:16:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:00.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:00.674338+0000) 2022-09-10T10:16:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:01.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:01.150887+0000) 2022-09-10T10:16:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:01 smithi110 ceph-mon[97633]: pgmap v11096: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:01 smithi104 ceph-mon[97044]: pgmap v11096: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:01.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:01.674567+0000) 2022-09-10T10:16:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:02.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:02.151127+0000) 2022-09-10T10:16:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:02.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:02.674834+0000) 2022-09-10T10:16:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:03.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:03.151266+0000) 2022-09-10T10:16:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:03 smithi110 ceph-mon[97633]: pgmap v11097: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:03 smithi104 ceph-mon[97044]: pgmap v11097: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:03.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:03.675072+0000) 2022-09-10T10:16:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:04.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:04.151471+0000) 2022-09-10T10:16:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:04.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:04.675277+0000) 2022-09-10T10:16:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:05.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:05.151652+0000) 2022-09-10T10:16:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:05 smithi110 ceph-mon[97633]: pgmap v11098: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:16:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:05 smithi104 ceph-mon[97044]: pgmap v11098: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:16:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:05.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:05.675522+0000) 2022-09-10T10:16:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:06.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:06.151830+0000) 2022-09-10T10:16:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:06.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:06.675758+0000) 2022-09-10T10:16:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:07.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:07.152000+0000) 2022-09-10T10:16:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:07 smithi110 ceph-mon[97633]: pgmap v11099: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:16:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:07 smithi104 ceph-mon[97044]: pgmap v11099: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:16:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:07.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:07.675947+0000) 2022-09-10T10:16:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:08.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:08.152256+0000) 2022-09-10T10:16:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:08 smithi110 ceph-mon[97633]: osdmap e774: 12 total, 12 up, 12 in 2022-09-10T10:16:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:08 smithi104 ceph-mon[97044]: osdmap e774: 12 total, 12 up, 12 in 2022-09-10T10:16:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:08.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:08.676134+0000) 2022-09-10T10:16:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:16:08.863+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:16:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:09.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:09.152416+0000) 2022-09-10T10:16:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:09 smithi110 ceph-mon[97633]: pgmap v11101: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:16:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:09 smithi104 ceph-mon[97044]: pgmap v11101: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:16:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:09.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:09.676299+0000) 2022-09-10T10:16:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:10.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:10.152640+0000) 2022-09-10T10:16:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:10.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:10.676475+0000) 2022-09-10T10:16:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:11.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:11.152919+0000) 2022-09-10T10:16:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:11 smithi110 ceph-mon[97633]: pgmap v11102: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:16:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:11 smithi104 ceph-mon[97044]: pgmap v11102: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:16:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:11.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:11.676678+0000) 2022-09-10T10:16:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:12.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:12.153168+0000) 2022-09-10T10:16:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:12.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:12.676885+0000) 2022-09-10T10:16:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:13.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:13.153424+0000) 2022-09-10T10:16:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:13 smithi104 ceph-mon[97044]: pgmap v11103: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:16:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:13 smithi104 ceph-mon[97044]: osdmap e775: 12 total, 12 up, 12 in 2022-09-10T10:16:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:13 smithi110 ceph-mon[97633]: pgmap v11103: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:16:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:13 smithi110 ceph-mon[97633]: osdmap e775: 12 total, 12 up, 12 in 2022-09-10T10:16:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:13.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:13.677135+0000) 2022-09-10T10:16:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:14.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:14.153613+0000) 2022-09-10T10:16:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:14.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:14.677366+0000) 2022-09-10T10:16:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:15.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:15.153862+0000) 2022-09-10T10:16:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:15 smithi104 ceph-mon[97044]: pgmap v11105: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:16:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:15 smithi110 ceph-mon[97633]: pgmap v11105: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:16:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:15.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:15.677512+0000) 2022-09-10T10:16:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:16.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:16.154086+0000) 2022-09-10T10:16:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:16.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:16.677730+0000) 2022-09-10T10:16:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:17.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:17.154336+0000) 2022-09-10T10:16:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:17 smithi104 ceph-mon[97044]: pgmap v11106: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:16:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:17 smithi110 ceph-mon[97633]: pgmap v11106: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:16:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:17.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:17.677974+0000) 2022-09-10T10:16:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:18.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:18.154502+0000) 2022-09-10T10:16:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:18.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:18.678206+0000) 2022-09-10T10:16:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:19.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:19.154663+0000) 2022-09-10T10:16:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:19 smithi104 ceph-mon[97044]: pgmap v11107: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:16:19.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:19 smithi110 ceph-mon[97633]: pgmap v11107: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:16:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:19.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:19.678394+0000) 2022-09-10T10:16:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:20.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:20.154917+0000) 2022-09-10T10:16:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:20.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:20.678629+0000) 2022-09-10T10:16:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:21.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:21.155163+0000) 2022-09-10T10:16:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:21 smithi104 ceph-mon[97044]: pgmap v11108: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:16:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:21 smithi110 ceph-mon[97633]: pgmap v11108: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:16:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:21.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:21.678801+0000) 2022-09-10T10:16:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:22.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:22.155348+0000) 2022-09-10T10:16:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:22.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:22.679019+0000) 2022-09-10T10:16:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:23.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:23.155599+0000) 2022-09-10T10:16:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:23 smithi104 ceph-mon[97044]: pgmap v11109: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:16:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:23 smithi110 ceph-mon[97633]: pgmap v11109: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:16:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:23.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:23.679235+0000) 2022-09-10T10:16:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:16:23.879+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:16:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:24.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:24.155799+0000) 2022-09-10T10:16:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:24.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:24.679451+0000) 2022-09-10T10:16:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:25.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:25.156041+0000) 2022-09-10T10:16:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:25 smithi104 ceph-mon[97044]: pgmap v11110: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:16:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:25 smithi110 ceph-mon[97633]: pgmap v11110: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:16:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:25.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:25.679657+0000) 2022-09-10T10:16:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:26.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:26.156253+0000) 2022-09-10T10:16:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:26.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:26.679860+0000) 2022-09-10T10:16:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:27.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:27.156455+0000) 2022-09-10T10:16:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:27 smithi104 ceph-mon[97044]: pgmap v11111: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:27 smithi110 ceph-mon[97633]: pgmap v11111: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:27.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:27.680084+0000) 2022-09-10T10:16:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:28.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:28.156684+0000) 2022-09-10T10:16:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:28.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:28.680340+0000) 2022-09-10T10:16:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:29.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:29.156820+0000) 2022-09-10T10:16:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:29 smithi104 ceph-mon[97044]: pgmap v11112: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:29 smithi110 ceph-mon[97633]: pgmap v11112: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:29.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:29.680595+0000) 2022-09-10T10:16:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:30.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:30.157031+0000) 2022-09-10T10:16:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:30.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:30.680850+0000) 2022-09-10T10:16:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:31.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:31.157340+0000) 2022-09-10T10:16:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:31 smithi104 ceph-mon[97044]: pgmap v11113: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:31 smithi110 ceph-mon[97633]: pgmap v11113: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:31.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:31.681086+0000) 2022-09-10T10:16:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:32.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:32.157517+0000) 2022-09-10T10:16:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:32.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:32.681350+0000) 2022-09-10T10:16:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:33.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:33.157752+0000) 2022-09-10T10:16:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:33 smithi104 ceph-mon[97044]: pgmap v11114: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:33 smithi110 ceph-mon[97633]: pgmap v11114: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:33.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:33.681570+0000) 2022-09-10T10:16:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:34.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:34.157982+0000) 2022-09-10T10:16:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:34.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:34.681772+0000) 2022-09-10T10:16:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:35.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:35.158222+0000) 2022-09-10T10:16:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:35 smithi104 ceph-mon[97044]: pgmap v11115: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:35 smithi110 ceph-mon[97633]: pgmap v11115: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:35.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:35.682016+0000) 2022-09-10T10:16:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:36.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:36.158409+0000) 2022-09-10T10:16:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:36.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:36.682264+0000) 2022-09-10T10:16:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:37.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:37.158628+0000) 2022-09-10T10:16:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:37 smithi110 ceph-mon[97633]: pgmap v11116: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:37 smithi104 ceph-mon[97044]: pgmap v11116: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:37.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:37.682518+0000) 2022-09-10T10:16:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:38.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:38.158876+0000) 2022-09-10T10:16:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:38.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:38.682751+0000) 2022-09-10T10:16:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:16:38.894+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:16:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:39.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:39.159154+0000) 2022-09-10T10:16:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:39 smithi110 ceph-mon[97633]: pgmap v11117: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:39 smithi104 ceph-mon[97044]: pgmap v11117: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:39.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:39.682993+0000) 2022-09-10T10:16:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:40.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:40.159281+0000) 2022-09-10T10:16:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:40.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:40.683224+0000) 2022-09-10T10:16:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:41.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:41.159485+0000) 2022-09-10T10:16:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:41 smithi110 ceph-mon[97633]: pgmap v11118: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:41 smithi104 ceph-mon[97044]: pgmap v11118: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:41.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:41.683401+0000) 2022-09-10T10:16:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:42.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:42.159712+0000) 2022-09-10T10:16:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:42.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:42.683632+0000) 2022-09-10T10:16:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:43.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:43.159953+0000) 2022-09-10T10:16:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:43 smithi110 ceph-mon[97633]: pgmap v11119: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:43 smithi104 ceph-mon[97044]: pgmap v11119: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:43.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:43.683867+0000) 2022-09-10T10:16:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:44.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:44.160199+0000) 2022-09-10T10:16:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:44.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:44.684167+0000) 2022-09-10T10:16:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:45.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:45.160504+0000) 2022-09-10T10:16:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:45 smithi110 ceph-mon[97633]: pgmap v11120: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:16:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:45 smithi104 ceph-mon[97044]: pgmap v11120: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:16:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:16:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:45.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:45.684349+0000) 2022-09-10T10:16:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:46.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:46.160666+0000) 2022-09-10T10:16:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:46.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:46.684605+0000) 2022-09-10T10:16:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:47.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:47.160796+0000) 2022-09-10T10:16:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:47 smithi110 ceph-mon[97633]: pgmap v11121: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:47 smithi104 ceph-mon[97044]: pgmap v11121: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:47.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:47.684828+0000) 2022-09-10T10:16:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:48.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:48.160996+0000) 2022-09-10T10:16:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:48.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:48.685121+0000) 2022-09-10T10:16:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:49.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:49.161166+0000) 2022-09-10T10:16:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:49 smithi110 ceph-mon[97633]: pgmap v11122: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:49 smithi104 ceph-mon[97044]: pgmap v11122: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:49.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:49.685318+0000) 2022-09-10T10:16:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:50.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:50.161342+0000) 2022-09-10T10:16:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:50.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:50.685574+0000) 2022-09-10T10:16:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:51.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:51.161508+0000) 2022-09-10T10:16:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:51 smithi110 ceph-mon[97633]: pgmap v11123: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:51 smithi104 ceph-mon[97044]: pgmap v11123: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:51.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:51.685776+0000) 2022-09-10T10:16:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:52.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:52.161678+0000) 2022-09-10T10:16:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:52.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:52.685960+0000) 2022-09-10T10:16:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:53.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:53.161946+0000) 2022-09-10T10:16:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:53 smithi110 ceph-mon[97633]: pgmap v11124: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:53 smithi104 ceph-mon[97044]: pgmap v11124: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:54.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:16:53.907+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:16:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:53.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:53.686234+0000) 2022-09-10T10:16:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:54.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:54.162165+0000) 2022-09-10T10:16:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:54.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:54.686515+0000) 2022-09-10T10:16:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:55.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:55.162353+0000) 2022-09-10T10:16:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:55 smithi110 ceph-mon[97633]: pgmap v11125: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:55 smithi104 ceph-mon[97044]: pgmap v11125: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:16:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:55.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:55.686739+0000) 2022-09-10T10:16:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:56.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:56.162579+0000) 2022-09-10T10:16:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:56.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:56.686988+0000) 2022-09-10T10:16:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:57.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:57.162788+0000) 2022-09-10T10:16:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:57 smithi110 ceph-mon[97633]: pgmap v11126: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:57 smithi104 ceph-mon[97044]: pgmap v11126: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:16:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:57.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:57.687228+0000) 2022-09-10T10:16:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:58.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:58.163010+0000) 2022-09-10T10:16:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:58.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:58.687457+0000) 2022-09-10T10:16:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:16:59.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:59.163262+0000) 2022-09-10T10:16:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:16:59 smithi110 ceph-mon[97633]: pgmap v11127: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:00.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:16:59 smithi104 ceph-mon[97044]: pgmap v11127: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:16:59.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:16:59.687682+0000) 2022-09-10T10:17:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:00.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:00.163495+0000) 2022-09-10T10:17:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:00.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:00.687850+0000) 2022-09-10T10:17:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:01.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:01.163671+0000) 2022-09-10T10:17:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:01 smithi110 ceph-mon[97633]: pgmap v11128: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:01 smithi104 ceph-mon[97044]: pgmap v11128: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:01.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:01.688071+0000) 2022-09-10T10:17:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:02.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:02.163852+0000) 2022-09-10T10:17:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:02.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:02.688269+0000) 2022-09-10T10:17:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:03.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:03.164087+0000) 2022-09-10T10:17:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:03 smithi110 ceph-mon[97633]: pgmap v11129: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:03 smithi104 ceph-mon[97044]: pgmap v11129: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:03.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:03.688488+0000) 2022-09-10T10:17:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:04.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:04.164361+0000) 2022-09-10T10:17:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:04.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:04.688700+0000) 2022-09-10T10:17:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:05.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:05.164603+0000) 2022-09-10T10:17:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:05 smithi110 ceph-mon[97633]: pgmap v11130: 129 pgs: 129 active+clean; 8.9 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:17:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:05 smithi104 ceph-mon[97044]: pgmap v11130: 129 pgs: 129 active+clean; 8.9 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:17:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:05.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:05.688927+0000) 2022-09-10T10:17:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:06.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:06.164858+0000) 2022-09-10T10:17:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:06.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:06.689186+0000) 2022-09-10T10:17:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:07.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:07.165049+0000) 2022-09-10T10:17:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:07 smithi110 ceph-mon[97633]: pgmap v11131: 129 pgs: 129 active+clean; 8.9 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:17:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:07 smithi104 ceph-mon[97044]: pgmap v11131: 129 pgs: 129 active+clean; 8.9 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:17:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:07.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:07.689433+0000) 2022-09-10T10:17:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:08.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:08.165274+0000) 2022-09-10T10:17:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:08 smithi110 ceph-mon[97633]: osdmap e776: 12 total, 12 up, 12 in 2022-09-10T10:17:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:08 smithi104 ceph-mon[97044]: osdmap e776: 12 total, 12 up, 12 in 2022-09-10T10:17:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:08.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:08.689691+0000) 2022-09-10T10:17:09.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:17:08.919+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:17:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:09.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:09.165512+0000) 2022-09-10T10:17:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:09 smithi110 ceph-mon[97633]: pgmap v11133: 129 pgs: 129 active+clean; 8.9 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:17:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:09 smithi104 ceph-mon[97044]: pgmap v11133: 129 pgs: 129 active+clean; 8.9 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:17:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:09.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:09.689939+0000) 2022-09-10T10:17:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:10.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:10.165784+0000) 2022-09-10T10:17:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:10.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:10.690156+0000) 2022-09-10T10:17:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:11.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:11.166008+0000) 2022-09-10T10:17:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:11 smithi110 ceph-mon[97633]: pgmap v11134: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:17:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:11 smithi104 ceph-mon[97044]: pgmap v11134: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:17:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:11.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:11.690375+0000) 2022-09-10T10:17:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:12.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:12.166255+0000) 2022-09-10T10:17:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:12.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:12.690619+0000) 2022-09-10T10:17:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:13.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:13.166539+0000) 2022-09-10T10:17:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:13 smithi104 ceph-mon[97044]: pgmap v11135: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:17:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:13 smithi104 ceph-mon[97044]: osdmap e777: 12 total, 12 up, 12 in 2022-09-10T10:17:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:13 smithi110 ceph-mon[97633]: pgmap v11135: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:17:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:13 smithi110 ceph-mon[97633]: osdmap e777: 12 total, 12 up, 12 in 2022-09-10T10:17:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:13.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:13.690771+0000) 2022-09-10T10:17:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:14.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:14.166782+0000) 2022-09-10T10:17:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:14.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:14.691028+0000) 2022-09-10T10:17:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:15.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:15.167034+0000) 2022-09-10T10:17:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:15 smithi104 ceph-mon[97044]: pgmap v11137: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:17:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:15 smithi110 ceph-mon[97633]: pgmap v11137: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:17:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:15.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:15.691253+0000) 2022-09-10T10:17:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:16.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:16.167250+0000) 2022-09-10T10:17:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:16.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:16.691409+0000) 2022-09-10T10:17:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:17.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:17.167487+0000) 2022-09-10T10:17:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:17 smithi104 ceph-mon[97044]: pgmap v11138: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:17:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:17 smithi110 ceph-mon[97633]: pgmap v11138: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:17:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:17.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:17.691577+0000) 2022-09-10T10:17:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:18.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:18.167685+0000) 2022-09-10T10:17:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:18.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:18.691787+0000) 2022-09-10T10:17:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:19.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:19.167886+0000) 2022-09-10T10:17:19.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:19 smithi104 ceph-mon[97044]: pgmap v11139: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:17:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:19 smithi110 ceph-mon[97633]: pgmap v11139: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:17:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:19.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:19.691966+0000) 2022-09-10T10:17:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:20.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:20.168135+0000) 2022-09-10T10:17:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:20.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:20.692175+0000) 2022-09-10T10:17:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:21.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:21.168465+0000) 2022-09-10T10:17:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:21 smithi104 ceph-mon[97044]: pgmap v11140: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:17:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:21 smithi110 ceph-mon[97633]: pgmap v11140: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:17:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:21.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:21.692381+0000) 2022-09-10T10:17:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:22.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:22.168666+0000) 2022-09-10T10:17:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:22.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:22.692583+0000) 2022-09-10T10:17:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:23.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:23.168938+0000) 2022-09-10T10:17:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:23 smithi104 ceph-mon[97044]: pgmap v11141: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:17:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:23 smithi110 ceph-mon[97633]: pgmap v11141: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:17:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:23.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:23.692803+0000) 2022-09-10T10:17:24.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:17:23.935+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:17:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:24.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:24.169141+0000) 2022-09-10T10:17:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:24.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:24.692997+0000) 2022-09-10T10:17:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:25.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:25.169323+0000) 2022-09-10T10:17:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:25 smithi104 ceph-mon[97044]: pgmap v11142: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:17:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:25 smithi110 ceph-mon[97633]: pgmap v11142: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:17:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:25.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:25.693189+0000) 2022-09-10T10:17:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:26.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:26.169531+0000) 2022-09-10T10:17:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:26.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:26.693432+0000) 2022-09-10T10:17:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:27.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:27.169689+0000) 2022-09-10T10:17:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:27 smithi110 ceph-mon[97633]: pgmap v11143: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:27 smithi104 ceph-mon[97044]: pgmap v11143: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:27.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:27.693698+0000) 2022-09-10T10:17:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:28.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:28.169970+0000) 2022-09-10T10:17:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:28.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:28.693968+0000) 2022-09-10T10:17:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:29.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:29.170148+0000) 2022-09-10T10:17:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:29 smithi110 ceph-mon[97633]: pgmap v11144: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:29 smithi104 ceph-mon[97044]: pgmap v11144: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:29.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:29.694221+0000) 2022-09-10T10:17:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:30.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:30.170382+0000) 2022-09-10T10:17:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:30.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:30.694431+0000) 2022-09-10T10:17:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:31.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:31.170610+0000) 2022-09-10T10:17:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:31 smithi110 ceph-mon[97633]: pgmap v11145: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:31 smithi104 ceph-mon[97044]: pgmap v11145: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:31.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:31.694652+0000) 2022-09-10T10:17:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:32.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:32.170855+0000) 2022-09-10T10:17:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:32.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:32.694900+0000) 2022-09-10T10:17:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:33.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:33.171009+0000) 2022-09-10T10:17:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:33 smithi110 ceph-mon[97633]: pgmap v11146: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:33 smithi104 ceph-mon[97044]: pgmap v11146: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:33.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:33.695161+0000) 2022-09-10T10:17:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:34.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:34.171214+0000) 2022-09-10T10:17:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:34.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:34.695324+0000) 2022-09-10T10:17:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:35.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:35.171391+0000) 2022-09-10T10:17:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:35 smithi110 ceph-mon[97633]: pgmap v11147: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:35 smithi104 ceph-mon[97044]: pgmap v11147: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:35.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:35.695543+0000) 2022-09-10T10:17:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:36.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:36.171560+0000) 2022-09-10T10:17:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:36.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:36.695769+0000) 2022-09-10T10:17:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:37.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:37.171800+0000) 2022-09-10T10:17:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:37 smithi110 ceph-mon[97633]: pgmap v11148: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:37 smithi104 ceph-mon[97044]: pgmap v11148: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:37.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:37.696013+0000) 2022-09-10T10:17:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:38.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:38.172052+0000) 2022-09-10T10:17:38.948 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:38.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:38.696248+0000) 2022-09-10T10:17:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:39.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:39.172338+0000) 2022-09-10T10:17:39.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:17:38.945+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:17:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:39 smithi110 ceph-mon[97633]: pgmap v11149: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:39 smithi104 ceph-mon[97044]: pgmap v11149: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:39.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:39.696460+0000) 2022-09-10T10:17:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:40.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:40.172555+0000) 2022-09-10T10:17:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:40.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:40.696724+0000) 2022-09-10T10:17:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:41.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:41.172759+0000) 2022-09-10T10:17:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:41 smithi110 ceph-mon[97633]: pgmap v11150: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:41 smithi104 ceph-mon[97044]: pgmap v11150: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:41.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:41.696925+0000) 2022-09-10T10:17:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:42.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:42.173004+0000) 2022-09-10T10:17:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:42.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:42.697139+0000) 2022-09-10T10:17:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:43.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:43.173300+0000) 2022-09-10T10:17:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:43 smithi110 ceph-mon[97633]: pgmap v11151: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:43 smithi104 ceph-mon[97044]: pgmap v11151: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:43.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:43.697303+0000) 2022-09-10T10:17:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:44.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:44.173452+0000) 2022-09-10T10:17:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:44.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:44.697560+0000) 2022-09-10T10:17:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:45.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:45.173618+0000) 2022-09-10T10:17:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:45 smithi110 ceph-mon[97633]: pgmap v11152: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:17:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:17:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:17:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:17:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:45 smithi104 ceph-mon[97044]: pgmap v11152: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:17:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:17:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:45.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:45.697736+0000) 2022-09-10T10:17:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:46.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:46.173767+0000) 2022-09-10T10:17:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:46.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:46.697963+0000) 2022-09-10T10:17:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:47.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:47.173983+0000) 2022-09-10T10:17:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:47 smithi110 ceph-mon[97633]: pgmap v11153: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:47 smithi104 ceph-mon[97044]: pgmap v11153: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:47.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:47.698192+0000) 2022-09-10T10:17:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:48.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:48.174231+0000) 2022-09-10T10:17:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:48.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:48.698381+0000) 2022-09-10T10:17:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:49.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:49.174441+0000) 2022-09-10T10:17:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:49 smithi110 ceph-mon[97633]: pgmap v11154: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:49 smithi104 ceph-mon[97044]: pgmap v11154: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:49.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:49.698548+0000) 2022-09-10T10:17:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:50.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:50.174680+0000) 2022-09-10T10:17:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:50.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:50.698767+0000) 2022-09-10T10:17:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:51.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:51.174894+0000) 2022-09-10T10:17:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:51 smithi110 ceph-mon[97633]: pgmap v11155: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:51 smithi104 ceph-mon[97044]: pgmap v11155: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:51.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:51.698966+0000) 2022-09-10T10:17:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:52.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:52.175143+0000) 2022-09-10T10:17:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:52.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:52.699224+0000) 2022-09-10T10:17:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:53.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:53.175385+0000) 2022-09-10T10:17:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:53 smithi110 ceph-mon[97633]: pgmap v11156: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:53.963 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:53.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:53.699464+0000) 2022-09-10T10:17:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:53 smithi104 ceph-mon[97044]: pgmap v11156: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:54.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:54.175533+0000) 2022-09-10T10:17:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:17:53.961+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:17:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:54.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:54.699717+0000) 2022-09-10T10:17:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:55.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:55.175729+0000) 2022-09-10T10:17:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:55 smithi110 ceph-mon[97633]: pgmap v11157: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:55 smithi104 ceph-mon[97044]: pgmap v11157: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:17:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:55.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:55.699934+0000) 2022-09-10T10:17:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:56.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:56.175929+0000) 2022-09-10T10:17:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:56.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:56.700124+0000) 2022-09-10T10:17:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:57.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:57.176185+0000) 2022-09-10T10:17:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:57 smithi110 ceph-mon[97633]: pgmap v11158: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:57 smithi104 ceph-mon[97044]: pgmap v11158: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:17:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:57.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:57.700329+0000) 2022-09-10T10:17:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:58.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:58.176413+0000) 2022-09-10T10:17:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:58.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:58.700514+0000) 2022-09-10T10:17:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:17:59.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:59.176662+0000) 2022-09-10T10:17:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:17:59 smithi110 ceph-mon[97633]: pgmap v11159: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:17:59 smithi104 ceph-mon[97044]: pgmap v11159: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:17:59.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:17:59.700745+0000) 2022-09-10T10:18:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:00.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:00.176916+0000) 2022-09-10T10:18:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:00.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:00.700938+0000) 2022-09-10T10:18:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:01.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:01.177085+0000) 2022-09-10T10:18:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:01 smithi110 ceph-mon[97633]: pgmap v11160: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:01 smithi104 ceph-mon[97044]: pgmap v11160: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:01.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:01.701164+0000) 2022-09-10T10:18:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:02.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:02.177373+0000) 2022-09-10T10:18:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:02.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:02.701390+0000) 2022-09-10T10:18:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:03.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:03.177593+0000) 2022-09-10T10:18:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:03 smithi110 ceph-mon[97633]: pgmap v11161: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:03 smithi104 ceph-mon[97044]: pgmap v11161: 129 pgs: 129 active+clean; 8.9 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:03.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:03.701562+0000) 2022-09-10T10:18:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:04.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:04.177793+0000) 2022-09-10T10:18:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:04.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:04.701758+0000) 2022-09-10T10:18:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:05.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:05.177943+0000) 2022-09-10T10:18:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:05 smithi110 ceph-mon[97633]: pgmap v11162: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:18:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:05 smithi104 ceph-mon[97044]: pgmap v11162: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:18:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:05.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:05.702010+0000) 2022-09-10T10:18:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:06.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:06.178176+0000) 2022-09-10T10:18:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:06.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:06.702282+0000) 2022-09-10T10:18:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:07.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:07.178351+0000) 2022-09-10T10:18:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:07 smithi110 ceph-mon[97633]: pgmap v11163: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:18:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:07 smithi104 ceph-mon[97044]: pgmap v11163: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:18:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:07.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:07.702514+0000) 2022-09-10T10:18:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:08.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:08.178558+0000) 2022-09-10T10:18:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:08 smithi110 ceph-mon[97633]: osdmap e778: 12 total, 12 up, 12 in 2022-09-10T10:18:08.973 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:08.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:08.702753+0000) 2022-09-10T10:18:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:08 smithi104 ceph-mon[97044]: osdmap e778: 12 total, 12 up, 12 in 2022-09-10T10:18:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:09.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:09.178790+0000) 2022-09-10T10:18:09.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:18:08.972+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:18:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:09 smithi110 ceph-mon[97633]: pgmap v11165: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:18:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:09 smithi104 ceph-mon[97044]: pgmap v11165: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:18:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:09.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:09.703007+0000) 2022-09-10T10:18:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:10.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:10.179010+0000) 2022-09-10T10:18:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:10.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:10.703260+0000) 2022-09-10T10:18:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:11.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:11.179181+0000) 2022-09-10T10:18:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:11 smithi110 ceph-mon[97633]: pgmap v11166: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:18:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:11 smithi104 ceph-mon[97044]: pgmap v11166: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:18:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:11.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:11.703517+0000) 2022-09-10T10:18:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:12.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:12.179409+0000) 2022-09-10T10:18:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:12.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:12.703790+0000) 2022-09-10T10:18:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:13.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:13.179660+0000) 2022-09-10T10:18:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:13 smithi104 ceph-mon[97044]: pgmap v11167: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:18:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:13 smithi104 ceph-mon[97044]: osdmap e779: 12 total, 12 up, 12 in 2022-09-10T10:18:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:13 smithi110 ceph-mon[97633]: pgmap v11167: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:18:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:13 smithi110 ceph-mon[97633]: osdmap e779: 12 total, 12 up, 12 in 2022-09-10T10:18:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:13.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:13.703986+0000) 2022-09-10T10:18:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:14.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:14.179873+0000) 2022-09-10T10:18:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:14.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:14.704194+0000) 2022-09-10T10:18:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:15.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:15.180059+0000) 2022-09-10T10:18:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:15 smithi110 ceph-mon[97633]: pgmap v11169: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:18:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:15 smithi104 ceph-mon[97044]: pgmap v11169: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:18:16.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:15.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:15.704402+0000) 2022-09-10T10:18:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:16.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:16.180216+0000) 2022-09-10T10:18:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:16.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:16.704585+0000) 2022-09-10T10:18:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:17.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:17.180381+0000) 2022-09-10T10:18:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:17 smithi110 ceph-mon[97633]: pgmap v11170: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:18:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:17 smithi104 ceph-mon[97044]: pgmap v11170: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:18:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:17.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:17.704801+0000) 2022-09-10T10:18:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:18.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:18.180611+0000) 2022-09-10T10:18:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:18.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:18.705073+0000) 2022-09-10T10:18:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:19.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:19.180811+0000) 2022-09-10T10:18:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:19 smithi110 ceph-mon[97633]: pgmap v11171: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:18:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:19 smithi104 ceph-mon[97044]: pgmap v11171: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:18:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:19.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:19.705306+0000) 2022-09-10T10:18:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:20.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:20.181041+0000) 2022-09-10T10:18:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:20.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:20.705516+0000) 2022-09-10T10:18:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:21.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:21.181324+0000) 2022-09-10T10:18:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:21 smithi110 ceph-mon[97633]: pgmap v11172: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:18:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:21 smithi104 ceph-mon[97044]: pgmap v11172: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:18:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:21.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:21.705763+0000) 2022-09-10T10:18:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:22.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:22.181568+0000) 2022-09-10T10:18:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:22.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:22.705992+0000) 2022-09-10T10:18:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:23.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:23.181841+0000) 2022-09-10T10:18:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:23 smithi110 ceph-mon[97633]: pgmap v11173: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:18:23.983 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:23.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:23.706213+0000) 2022-09-10T10:18:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:23 smithi104 ceph-mon[97044]: pgmap v11173: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:18:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:24.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:24.182065+0000) 2022-09-10T10:18:24.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:18:23.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:18:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:24.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:24.706419+0000) 2022-09-10T10:18:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:25.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:25.182245+0000) 2022-09-10T10:18:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:25 smithi110 ceph-mon[97633]: pgmap v11174: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:18:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:25 smithi104 ceph-mon[97044]: pgmap v11174: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:18:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:25.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:25.706648+0000) 2022-09-10T10:18:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:26.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:26.182537+0000) 2022-09-10T10:18:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:26.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:26.706788+0000) 2022-09-10T10:18:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:27.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:27.182807+0000) 2022-09-10T10:18:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:27 smithi110 ceph-mon[97633]: pgmap v11175: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:27 smithi104 ceph-mon[97044]: pgmap v11175: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:27.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:27.706968+0000) 2022-09-10T10:18:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:28.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:28.183066+0000) 2022-09-10T10:18:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:28.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:28.707214+0000) 2022-09-10T10:18:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:29.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:29.183308+0000) 2022-09-10T10:18:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:29 smithi110 ceph-mon[97633]: pgmap v11176: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:29 smithi104 ceph-mon[97044]: pgmap v11176: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:29.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:29.707455+0000) 2022-09-10T10:18:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:30.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:30.183524+0000) 2022-09-10T10:18:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:30.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:30.707683+0000) 2022-09-10T10:18:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:31.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:31.183791+0000) 2022-09-10T10:18:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:31 smithi110 ceph-mon[97633]: pgmap v11177: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:31 smithi104 ceph-mon[97044]: pgmap v11177: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:31.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:31.707902+0000) 2022-09-10T10:18:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:32.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:32.184049+0000) 2022-09-10T10:18:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:32.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:32.708125+0000) 2022-09-10T10:18:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:33.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:33.184253+0000) 2022-09-10T10:18:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:33 smithi110 ceph-mon[97633]: pgmap v11178: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:33 smithi104 ceph-mon[97044]: pgmap v11178: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:33.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:33.708371+0000) 2022-09-10T10:18:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:34.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:34.184429+0000) 2022-09-10T10:18:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:34.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:34.708603+0000) 2022-09-10T10:18:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:35.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:35.184647+0000) 2022-09-10T10:18:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:35 smithi110 ceph-mon[97633]: pgmap v11179: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:35 smithi104 ceph-mon[97044]: pgmap v11179: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:35.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:35.708805+0000) 2022-09-10T10:18:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:36.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:36.184818+0000) 2022-09-10T10:18:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:36.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:36.709031+0000) 2022-09-10T10:18:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:37.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:37.185016+0000) 2022-09-10T10:18:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:37 smithi110 ceph-mon[97633]: pgmap v11180: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:37 smithi104 ceph-mon[97044]: pgmap v11180: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:37.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:37.709264+0000) 2022-09-10T10:18:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:38.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:38.185222+0000) 2022-09-10T10:18:38.997 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:38.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:38.709493+0000) 2022-09-10T10:18:39.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:18:38.995+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:18:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:39.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:39.185470+0000) 2022-09-10T10:18:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:39 smithi110 ceph-mon[97633]: pgmap v11181: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:39 smithi104 ceph-mon[97044]: pgmap v11181: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:18:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:39.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:39.709701+0000) 2022-09-10T10:18:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:40.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:40.185677+0000) 2022-09-10T10:18:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:40.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:40.709916+0000) 2022-09-10T10:18:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:41.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:41.185937+0000) 2022-09-10T10:18:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:41 smithi110 ceph-mon[97633]: pgmap v11182: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:41 smithi104 ceph-mon[97044]: pgmap v11182: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:41.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:41.710176+0000) 2022-09-10T10:18:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:42.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:42.186179+0000) 2022-09-10T10:18:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:42.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:42.710431+0000) 2022-09-10T10:18:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:43.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:43.186387+0000) 2022-09-10T10:18:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:43 smithi110 ceph-mon[97633]: pgmap v11183: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:43 smithi104 ceph-mon[97044]: pgmap v11183: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:43.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:43.710616+0000) 2022-09-10T10:18:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:44.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:44.186567+0000) 2022-09-10T10:18:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:44.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:44.710848+0000) 2022-09-10T10:18:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:45.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:45.186791+0000) 2022-09-10T10:18:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:45 smithi110 ceph-mon[97633]: pgmap v11184: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:18:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:18:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:18:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:18:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:45 smithi104 ceph-mon[97044]: pgmap v11184: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:18:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:18:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:18:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:18:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:45.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:45.711083+0000) 2022-09-10T10:18:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:46.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:46.187052+0000) 2022-09-10T10:18:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:46.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:46.711262+0000) 2022-09-10T10:18:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:47.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:47.187262+0000) 2022-09-10T10:18:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:47 smithi110 ceph-mon[97633]: pgmap v11185: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:47 smithi104 ceph-mon[97044]: pgmap v11185: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:47.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:47.711512+0000) 2022-09-10T10:18:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:48.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:48.187401+0000) 2022-09-10T10:18:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:48.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:48.711717+0000) 2022-09-10T10:18:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:49.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:49.187527+0000) 2022-09-10T10:18:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:49 smithi110 ceph-mon[97633]: pgmap v11186: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:49 smithi104 ceph-mon[97044]: pgmap v11186: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:49.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:49.711933+0000) 2022-09-10T10:18:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:50.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:50.187665+0000) 2022-09-10T10:18:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:50.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:50.712141+0000) 2022-09-10T10:18:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:51.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:51.187882+0000) 2022-09-10T10:18:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:51 smithi110 ceph-mon[97633]: pgmap v11187: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:51 smithi104 ceph-mon[97044]: pgmap v11187: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:51.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:51.712286+0000) 2022-09-10T10:18:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:52.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:52.188128+0000) 2022-09-10T10:18:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:52.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:52.712430+0000) 2022-09-10T10:18:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:53.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:53.188382+0000) 2022-09-10T10:18:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:53 smithi110 ceph-mon[97633]: pgmap v11188: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:54.000 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:53.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:53.712594+0000) 2022-09-10T10:18:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:53 smithi104 ceph-mon[97044]: pgmap v11188: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:54.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:18:53.998+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:18:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:54.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:54.188595+0000) 2022-09-10T10:18:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:54.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:54.712791+0000) 2022-09-10T10:18:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:55.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:55.188852+0000) 2022-09-10T10:18:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:55 smithi110 ceph-mon[97633]: pgmap v11189: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:55 smithi104 ceph-mon[97044]: pgmap v11189: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:18:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:55.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:55.712936+0000) 2022-09-10T10:18:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:56.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:56.188998+0000) 2022-09-10T10:18:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:56.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:56.713202+0000) 2022-09-10T10:18:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:57.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:57.189238+0000) 2022-09-10T10:18:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:57 smithi110 ceph-mon[97633]: pgmap v11190: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:57 smithi104 ceph-mon[97044]: pgmap v11190: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:18:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:57.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:57.713436+0000) 2022-09-10T10:18:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:58.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:58.189382+0000) 2022-09-10T10:18:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:58.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:58.713595+0000) 2022-09-10T10:18:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:18:59.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:59.189546+0000) 2022-09-10T10:18:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:18:59 smithi110 ceph-mon[97633]: pgmap v11191: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:18:59 smithi104 ceph-mon[97044]: pgmap v11191: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:18:59.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:18:59.713817+0000) 2022-09-10T10:19:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:00.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:00.189775+0000) 2022-09-10T10:19:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:00.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:00.713994+0000) 2022-09-10T10:19:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:01.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:01.189973+0000) 2022-09-10T10:19:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:01 smithi110 ceph-mon[97633]: pgmap v11192: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:01 smithi104 ceph-mon[97044]: pgmap v11192: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:01.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:01.714226+0000) 2022-09-10T10:19:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:02.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:02.190140+0000) 2022-09-10T10:19:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:02.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:02.714469+0000) 2022-09-10T10:19:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:03.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:03.190359+0000) 2022-09-10T10:19:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:03 smithi110 ceph-mon[97633]: pgmap v11193: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:03 smithi104 ceph-mon[97044]: pgmap v11193: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:03.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:03.714608+0000) 2022-09-10T10:19:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:04.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:04.190591+0000) 2022-09-10T10:19:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:04.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:04.714789+0000) 2022-09-10T10:19:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:05.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:05.190873+0000) 2022-09-10T10:19:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:05 smithi110 ceph-mon[97633]: pgmap v11194: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:19:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:05 smithi104 ceph-mon[97044]: pgmap v11194: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:19:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:05.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:05.715022+0000) 2022-09-10T10:19:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:06.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:06.191143+0000) 2022-09-10T10:19:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:06.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:06.715270+0000) 2022-09-10T10:19:07.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:07.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:07.191351+0000) 2022-09-10T10:19:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:07 smithi110 ceph-mon[97633]: pgmap v11195: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:19:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:07 smithi104 ceph-mon[97044]: pgmap v11195: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:19:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:07.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:07.715551+0000) 2022-09-10T10:19:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:08.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:08.191532+0000) 2022-09-10T10:19:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:08 smithi110 ceph-mon[97633]: osdmap e780: 12 total, 12 up, 12 in 2022-09-10T10:19:09.002 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:08.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:08.715791+0000) 2022-09-10T10:19:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:08 smithi104 ceph-mon[97044]: osdmap e780: 12 total, 12 up, 12 in 2022-09-10T10:19:09.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:19:09.000+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:19:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:09.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:09.191766+0000) 2022-09-10T10:19:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:09 smithi110 ceph-mon[97633]: pgmap v11197: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:19:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:09 smithi104 ceph-mon[97044]: pgmap v11197: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:19:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:09.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:09.715976+0000) 2022-09-10T10:19:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:10.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:10.191978+0000) 2022-09-10T10:19:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:10.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:10.716253+0000) 2022-09-10T10:19:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:11.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:11.192211+0000) 2022-09-10T10:19:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:11 smithi110 ceph-mon[97633]: pgmap v11198: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:19:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:11 smithi104 ceph-mon[97044]: pgmap v11198: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:19:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:11.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:11.716421+0000) 2022-09-10T10:19:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:12.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:12.192462+0000) 2022-09-10T10:19:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:12.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:12.716617+0000) 2022-09-10T10:19:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:13.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:13.192722+0000) 2022-09-10T10:19:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:13 smithi110 ceph-mon[97633]: pgmap v11199: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:19:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:13 smithi110 ceph-mon[97633]: osdmap e781: 12 total, 12 up, 12 in 2022-09-10T10:19:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:13 smithi104 ceph-mon[97044]: pgmap v11199: 129 pgs: 129 active+clean; 8.9 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:19:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:13 smithi104 ceph-mon[97044]: osdmap e781: 12 total, 12 up, 12 in 2022-09-10T10:19:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:13.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:13.716843+0000) 2022-09-10T10:19:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:14.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:14.192930+0000) 2022-09-10T10:19:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:14.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:14.717065+0000) 2022-09-10T10:19:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:15.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:15.193174+0000) 2022-09-10T10:19:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:15 smithi110 ceph-mon[97633]: pgmap v11201: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:19:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:15 smithi104 ceph-mon[97044]: pgmap v11201: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:19:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:15.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:15.717348+0000) 2022-09-10T10:19:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:16.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:16.193407+0000) 2022-09-10T10:19:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:16.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:16.717544+0000) 2022-09-10T10:19:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:17.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:17.193623+0000) 2022-09-10T10:19:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:17 smithi110 ceph-mon[97633]: pgmap v11202: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:19:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:17 smithi104 ceph-mon[97044]: pgmap v11202: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T10:19:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:17.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:17.717783+0000) 2022-09-10T10:19:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:18.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:18.193845+0000) 2022-09-10T10:19:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:18.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:18.717984+0000) 2022-09-10T10:19:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:19.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:19.194078+0000) 2022-09-10T10:19:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:19 smithi110 ceph-mon[97633]: pgmap v11203: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:19:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:19 smithi104 ceph-mon[97044]: pgmap v11203: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:19:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:19.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:19.718182+0000) 2022-09-10T10:19:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:20.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:20.194325+0000) 2022-09-10T10:19:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:20.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:20.718372+0000) 2022-09-10T10:19:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:21.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:21.194562+0000) 2022-09-10T10:19:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:21 smithi110 ceph-mon[97633]: pgmap v11204: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:19:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:21 smithi104 ceph-mon[97044]: pgmap v11204: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:19:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:21.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:21.718579+0000) 2022-09-10T10:19:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:22.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:22.194704+0000) 2022-09-10T10:19:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:22.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:22.718834+0000) 2022-09-10T10:19:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:23.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:23.194953+0000) 2022-09-10T10:19:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:23 smithi110 ceph-mon[97633]: pgmap v11205: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:19:24.014 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:23.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:23.719079+0000) 2022-09-10T10:19:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:23 smithi104 ceph-mon[97044]: pgmap v11205: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:19:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:24.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:24.195171+0000) 2022-09-10T10:19:24.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:19:24.012+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:19:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:24.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:24.719320+0000) 2022-09-10T10:19:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:25.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:25.195352+0000) 2022-09-10T10:19:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:25 smithi110 ceph-mon[97633]: pgmap v11206: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:19:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:25 smithi104 ceph-mon[97044]: pgmap v11206: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:19:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:25.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:25.719524+0000) 2022-09-10T10:19:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:26.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:26.195578+0000) 2022-09-10T10:19:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:26.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:26.719725+0000) 2022-09-10T10:19:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:27.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:27.195827+0000) 2022-09-10T10:19:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:27 smithi110 ceph-mon[97633]: pgmap v11207: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:27 smithi104 ceph-mon[97044]: pgmap v11207: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:27.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:27.719916+0000) 2022-09-10T10:19:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:28.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:28.196078+0000) 2022-09-10T10:19:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:28.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:28.720214+0000) 2022-09-10T10:19:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:29.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:29.196269+0000) 2022-09-10T10:19:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:29 smithi110 ceph-mon[97633]: pgmap v11208: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:29 smithi104 ceph-mon[97044]: pgmap v11208: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:29.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:29.720439+0000) 2022-09-10T10:19:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:30.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:30.196470+0000) 2022-09-10T10:19:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:30.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:30.720638+0000) 2022-09-10T10:19:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:31.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:31.196675+0000) 2022-09-10T10:19:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:31 smithi110 ceph-mon[97633]: pgmap v11209: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:31 smithi104 ceph-mon[97044]: pgmap v11209: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:31.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:31.720859+0000) 2022-09-10T10:19:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:32.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:32.196916+0000) 2022-09-10T10:19:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:32.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:32.721078+0000) 2022-09-10T10:19:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:33.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:33.197151+0000) 2022-09-10T10:19:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:33 smithi110 ceph-mon[97633]: pgmap v11210: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:33 smithi104 ceph-mon[97044]: pgmap v11210: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:33.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:33.721313+0000) 2022-09-10T10:19:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:34.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:34.197321+0000) 2022-09-10T10:19:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:34.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:34.721529+0000) 2022-09-10T10:19:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:35.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:35.197496+0000) 2022-09-10T10:19:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:35 smithi110 ceph-mon[97633]: pgmap v11211: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:35 smithi104 ceph-mon[97044]: pgmap v11211: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:35.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:35.721771+0000) 2022-09-10T10:19:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:36.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:36.197744+0000) 2022-09-10T10:19:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:36.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:36.722055+0000) 2022-09-10T10:19:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:37.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:37.197962+0000) 2022-09-10T10:19:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:37 smithi110 ceph-mon[97633]: pgmap v11212: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:37 smithi104 ceph-mon[97044]: pgmap v11212: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:37.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:37.722339+0000) 2022-09-10T10:19:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:38.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:38.198138+0000) 2022-09-10T10:19:39.029 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:38.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:38.722562+0000) 2022-09-10T10:19:39.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:19:39.028+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:19:39.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:39.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:39.198348+0000) 2022-09-10T10:19:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:39 smithi110 ceph-mon[97633]: pgmap v11213: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:39 smithi104 ceph-mon[97044]: pgmap v11213: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:39.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:39.722796+0000) 2022-09-10T10:19:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:40.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:40.198568+0000) 2022-09-10T10:19:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:40.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:40.723046+0000) 2022-09-10T10:19:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:41.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:41.198779+0000) 2022-09-10T10:19:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:41 smithi110 ceph-mon[97633]: pgmap v11214: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:41 smithi104 ceph-mon[97044]: pgmap v11214: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:41.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:41.723248+0000) 2022-09-10T10:19:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:42.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:42.199022+0000) 2022-09-10T10:19:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:42.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:42.723491+0000) 2022-09-10T10:19:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:43.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:43.199274+0000) 2022-09-10T10:19:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:43 smithi110 ceph-mon[97633]: pgmap v11215: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:43 smithi104 ceph-mon[97044]: pgmap v11215: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:43.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:43.723696+0000) 2022-09-10T10:19:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:44.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:44.199468+0000) 2022-09-10T10:19:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:44.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:44.723928+0000) 2022-09-10T10:19:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:45.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:45.199652+0000) 2022-09-10T10:19:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:45 smithi110 ceph-mon[97633]: pgmap v11216: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:19:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:19:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:19:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:19:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:45 smithi104 ceph-mon[97044]: pgmap v11216: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:19:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:19:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:19:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:19:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:45.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:45.724194+0000) 2022-09-10T10:19:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:46.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:46.199818+0000) 2022-09-10T10:19:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:46.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:46.724421+0000) 2022-09-10T10:19:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:47.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:47.200040+0000) 2022-09-10T10:19:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:47 smithi110 ceph-mon[97633]: pgmap v11217: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:47 smithi104 ceph-mon[97044]: pgmap v11217: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:47.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:47.724609+0000) 2022-09-10T10:19:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:48.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:48.200267+0000) 2022-09-10T10:19:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:48.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:48.724855+0000) 2022-09-10T10:19:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:49.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:49.200503+0000) 2022-09-10T10:19:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:49 smithi110 ceph-mon[97633]: pgmap v11218: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:49 smithi104 ceph-mon[97044]: pgmap v11218: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:19:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:49.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:49.725129+0000) 2022-09-10T10:19:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:50.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:50.200752+0000) 2022-09-10T10:19:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:50.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:50.725294+0000) 2022-09-10T10:19:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:51.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:51.200988+0000) 2022-09-10T10:19:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:51 smithi110 ceph-mon[97633]: pgmap v11219: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:51 smithi104 ceph-mon[97044]: pgmap v11219: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:51.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:51.725542+0000) 2022-09-10T10:19:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:52.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:52.201201+0000) 2022-09-10T10:19:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:52.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:52.725782+0000) 2022-09-10T10:19:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:53.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:53.201432+0000) 2022-09-10T10:19:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:53 smithi110 ceph-mon[97633]: pgmap v11220: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:54.039 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:53.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:53.725980+0000) 2022-09-10T10:19:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:53 smithi104 ceph-mon[97044]: pgmap v11220: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:19:54.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:19:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:54.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:54.201657+0000) 2022-09-10T10:19:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:54.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:54.726207+0000) 2022-09-10T10:19:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:55.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:55.201929+0000) 2022-09-10T10:19:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:55 smithi110 ceph-mon[97633]: pgmap v11221: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:55 smithi104 ceph-mon[97044]: pgmap v11221: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:19:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:55.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:55.726427+0000) 2022-09-10T10:19:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:56.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:56.202129+0000) 2022-09-10T10:19:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:56.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:56.726564+0000) 2022-09-10T10:19:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:57.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:57.202348+0000) 2022-09-10T10:19:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:57 smithi110 ceph-mon[97633]: pgmap v11222: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:57 smithi104 ceph-mon[97044]: pgmap v11222: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:19:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:57.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:57.726785+0000) 2022-09-10T10:19:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:58.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:58.202518+0000) 2022-09-10T10:19:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:58.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:58.727033+0000) 2022-09-10T10:19:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:19:59.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:59.202753+0000) 2022-09-10T10:19:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:19:59 smithi110 ceph-mon[97633]: pgmap v11223: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:20:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:19:59 smithi104 ceph-mon[97044]: pgmap v11223: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:20:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:19:59.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:19:59.727266+0000) 2022-09-10T10:20:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:00.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:00.202910+0000) 2022-09-10T10:20:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:20:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:20:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:00.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:00.727458+0000) 2022-09-10T10:20:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:01.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:01.203134+0000) 2022-09-10T10:20:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:01 smithi104 ceph-mon[97044]: pgmap v11224: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:01 smithi110 ceph-mon[97633]: pgmap v11224: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:01.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:01.727673+0000) 2022-09-10T10:20:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:02.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:02.203360+0000) 2022-09-10T10:20:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:02.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:02.727877+0000) 2022-09-10T10:20:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:03.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:03.203642+0000) 2022-09-10T10:20:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:03 smithi110 ceph-mon[97633]: pgmap v11225: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:03 smithi104 ceph-mon[97044]: pgmap v11225: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:03.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:03.728023+0000) 2022-09-10T10:20:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:04.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:04.203849+0000) 2022-09-10T10:20:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:04.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:04.728213+0000) 2022-09-10T10:20:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:05.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:05.204072+0000) 2022-09-10T10:20:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:05 smithi110 ceph-mon[97633]: pgmap v11226: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:20:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:05 smithi104 ceph-mon[97044]: pgmap v11226: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:20:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:05.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:05.728429+0000) 2022-09-10T10:20:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:06.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:06.204313+0000) 2022-09-10T10:20:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:06.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:06.728613+0000) 2022-09-10T10:20:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:07.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:07.204612+0000) 2022-09-10T10:20:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:07 smithi110 ceph-mon[97633]: pgmap v11227: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:20:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:07 smithi104 ceph-mon[97044]: pgmap v11227: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:20:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:07.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:07.728854+0000) 2022-09-10T10:20:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:08.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:08.204883+0000) 2022-09-10T10:20:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:08 smithi110 ceph-mon[97633]: osdmap e782: 12 total, 12 up, 12 in 2022-09-10T10:20:09.044 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:08.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:08.729037+0000) 2022-09-10T10:20:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:08 smithi104 ceph-mon[97044]: osdmap e782: 12 total, 12 up, 12 in 2022-09-10T10:20:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:09.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:09.205086+0000) 2022-09-10T10:20:09.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:20:09.042+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:20:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:09 smithi110 ceph-mon[97633]: pgmap v11229: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:20:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:09 smithi104 ceph-mon[97044]: pgmap v11229: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:20:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:09.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:09.729285+0000) 2022-09-10T10:20:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:10.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:10.205293+0000) 2022-09-10T10:20:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:10.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:10.729497+0000) 2022-09-10T10:20:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:11.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:11.205543+0000) 2022-09-10T10:20:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:11 smithi104 ceph-mon[97044]: pgmap v11230: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:20:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:11.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:11.729719+0000) 2022-09-10T10:20:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:11 smithi110 ceph-mon[97633]: pgmap v11230: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:20:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:12.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:12.205777+0000) 2022-09-10T10:20:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:12.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:12.729884+0000) 2022-09-10T10:20:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:13.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:13.206064+0000) 2022-09-10T10:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:13 smithi110 ceph-mon[97633]: pgmap v11231: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:20:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:13 smithi110 ceph-mon[97633]: osdmap e783: 12 total, 12 up, 12 in 2022-09-10T10:20:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:13 smithi104 ceph-mon[97044]: pgmap v11231: 129 pgs: 129 active+clean; 8.9 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:20:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:13 smithi104 ceph-mon[97044]: osdmap e783: 12 total, 12 up, 12 in 2022-09-10T10:20:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:13.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:13.730062+0000) 2022-09-10T10:20:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:14.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:14.206372+0000) 2022-09-10T10:20:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:14.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:14.730282+0000) 2022-09-10T10:20:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:15.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:15.206585+0000) 2022-09-10T10:20:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:15 smithi110 ceph-mon[97633]: pgmap v11233: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:20:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:15 smithi104 ceph-mon[97044]: pgmap v11233: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:20:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:15.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:15.730501+0000) 2022-09-10T10:20:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:16.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:16.206788+0000) 2022-09-10T10:20:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:16.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:16.730703+0000) 2022-09-10T10:20:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:17.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:17.207009+0000) 2022-09-10T10:20:17.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:17 smithi110 ceph-mon[97633]: pgmap v11234: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:20:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:17 smithi104 ceph-mon[97044]: pgmap v11234: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:20:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:17.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:17.730861+0000) 2022-09-10T10:20:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:18.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:18.207283+0000) 2022-09-10T10:20:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:18.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:18.731131+0000) 2022-09-10T10:20:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:19.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:19.207549+0000) 2022-09-10T10:20:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:19 smithi110 ceph-mon[97633]: pgmap v11235: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:20:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:19 smithi104 ceph-mon[97044]: pgmap v11235: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:20:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:19.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:19.731316+0000) 2022-09-10T10:20:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:20.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:20.207795+0000) 2022-09-10T10:20:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:20.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:20.731546+0000) 2022-09-10T10:20:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:21.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:21.208013+0000) 2022-09-10T10:20:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:21 smithi110 ceph-mon[97633]: pgmap v11236: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:20:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:21 smithi104 ceph-mon[97044]: pgmap v11236: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:20:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:21.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:21.731778+0000) 2022-09-10T10:20:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:22.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:22.208184+0000) 2022-09-10T10:20:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:22.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:22.732034+0000) 2022-09-10T10:20:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:23.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:23.208436+0000) 2022-09-10T10:20:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:23 smithi110 ceph-mon[97633]: pgmap v11237: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:20:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:23 smithi104 ceph-mon[97044]: pgmap v11237: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:20:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:23.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:23.732292+0000) 2022-09-10T10:20:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:24.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:24.208693+0000) 2022-09-10T10:20:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:20:24.057+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:20:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:24.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:24.732435+0000) 2022-09-10T10:20:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:25.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:25.208990+0000) 2022-09-10T10:20:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:25 smithi110 ceph-mon[97633]: pgmap v11238: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:20:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:25 smithi104 ceph-mon[97044]: pgmap v11238: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:20:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:25.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:25.732681+0000) 2022-09-10T10:20:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:26.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:26.209233+0000) 2022-09-10T10:20:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:26.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:26.732900+0000) 2022-09-10T10:20:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:27.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:27.209458+0000) 2022-09-10T10:20:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:27 smithi110 ceph-mon[97633]: pgmap v11239: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:27 smithi104 ceph-mon[97044]: pgmap v11239: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:27.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:27.733162+0000) 2022-09-10T10:20:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:28.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:28.209622+0000) 2022-09-10T10:20:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:28.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:28.733326+0000) 2022-09-10T10:20:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:29.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:29.209810+0000) 2022-09-10T10:20:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:29 smithi110 ceph-mon[97633]: pgmap v11240: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:29 smithi104 ceph-mon[97044]: pgmap v11240: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:29.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:29.733588+0000) 2022-09-10T10:20:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:30.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:30.209981+0000) 2022-09-10T10:20:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:30.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:30.733800+0000) 2022-09-10T10:20:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:31.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:31.210190+0000) 2022-09-10T10:20:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:31 smithi110 ceph-mon[97633]: pgmap v11241: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:31 smithi104 ceph-mon[97044]: pgmap v11241: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:31.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:31.734037+0000) 2022-09-10T10:20:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:32.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:32.210413+0000) 2022-09-10T10:20:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:32.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:32.734269+0000) 2022-09-10T10:20:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:33.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:33.210651+0000) 2022-09-10T10:20:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:33 smithi110 ceph-mon[97633]: pgmap v11242: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:33 smithi104 ceph-mon[97044]: pgmap v11242: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:33.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:33.734522+0000) 2022-09-10T10:20:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:34.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:34.210849+0000) 2022-09-10T10:20:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:34.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:34.734724+0000) 2022-09-10T10:20:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:35.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:35.211146+0000) 2022-09-10T10:20:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:35 smithi110 ceph-mon[97633]: pgmap v11243: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:35 smithi104 ceph-mon[97044]: pgmap v11243: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:35.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:35.734974+0000) 2022-09-10T10:20:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:36.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:36.211383+0000) 2022-09-10T10:20:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:36.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:36.735174+0000) 2022-09-10T10:20:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:37.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:37.211605+0000) 2022-09-10T10:20:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:37 smithi110 ceph-mon[97633]: pgmap v11244: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:37 smithi104 ceph-mon[97044]: pgmap v11244: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:37.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:37.735436+0000) 2022-09-10T10:20:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:38.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:38.211798+0000) 2022-09-10T10:20:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:38.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:38.735681+0000) 2022-09-10T10:20:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:20:39.071+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:20:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:39.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:39.212008+0000) 2022-09-10T10:20:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:39 smithi110 ceph-mon[97633]: pgmap v11245: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:39 smithi104 ceph-mon[97044]: pgmap v11245: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:39.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:39.735924+0000) 2022-09-10T10:20:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:40.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:40.212244+0000) 2022-09-10T10:20:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:40.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:40.736208+0000) 2022-09-10T10:20:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:41.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:41.212442+0000) 2022-09-10T10:20:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:41 smithi110 ceph-mon[97633]: pgmap v11246: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:41 smithi104 ceph-mon[97044]: pgmap v11246: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:41.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:41.736440+0000) 2022-09-10T10:20:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:42.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:42.212665+0000) 2022-09-10T10:20:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:42.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:42.736679+0000) 2022-09-10T10:20:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:43.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:43.212883+0000) 2022-09-10T10:20:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:43 smithi110 ceph-mon[97633]: pgmap v11247: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:43 smithi104 ceph-mon[97044]: pgmap v11247: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:43.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:43.736878+0000) 2022-09-10T10:20:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:44.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:44.213155+0000) 2022-09-10T10:20:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:44.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:44.737167+0000) 2022-09-10T10:20:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:45.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:45.213333+0000) 2022-09-10T10:20:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:45 smithi110 ceph-mon[97633]: pgmap v11248: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:20:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:20:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:20:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:20:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:45 smithi104 ceph-mon[97044]: pgmap v11248: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:20:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:20:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:20:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:20:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:45.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:45.737404+0000) 2022-09-10T10:20:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:46.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:46.213546+0000) 2022-09-10T10:20:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:46.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:46.737571+0000) 2022-09-10T10:20:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:47.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:47.213796+0000) 2022-09-10T10:20:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:47 smithi110 ceph-mon[97633]: pgmap v11249: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:47 smithi104 ceph-mon[97044]: pgmap v11249: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:47.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:47.737795+0000) 2022-09-10T10:20:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:48.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:48.214019+0000) 2022-09-10T10:20:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:48.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:48.738022+0000) 2022-09-10T10:20:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:49.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:49.214247+0000) 2022-09-10T10:20:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:49 smithi110 ceph-mon[97633]: pgmap v11250: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:49 smithi104 ceph-mon[97044]: pgmap v11250: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:49.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:49.738265+0000) 2022-09-10T10:20:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:50.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:50.214514+0000) 2022-09-10T10:20:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:50.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:50.738507+0000) 2022-09-10T10:20:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:51.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:51.214753+0000) 2022-09-10T10:20:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:51 smithi110 ceph-mon[97633]: pgmap v11251: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:51 smithi104 ceph-mon[97044]: pgmap v11251: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:51.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:51.738757+0000) 2022-09-10T10:20:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:52.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:52.215001+0000) 2022-09-10T10:20:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:52.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:52.738946+0000) 2022-09-10T10:20:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:53.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:53.215218+0000) 2022-09-10T10:20:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:53 smithi110 ceph-mon[97633]: pgmap v11252: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:53 smithi104 ceph-mon[97044]: pgmap v11252: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:53.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:53.739189+0000) 2022-09-10T10:20:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:54.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:54.215435+0000) 2022-09-10T10:20:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:20:54.080+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:20:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:54.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:54.739380+0000) 2022-09-10T10:20:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:55.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:55.215694+0000) 2022-09-10T10:20:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:55 smithi110 ceph-mon[97633]: pgmap v11253: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:55 smithi104 ceph-mon[97044]: pgmap v11253: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:20:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:55.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:55.739584+0000) 2022-09-10T10:20:56.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:56.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:56.215850+0000) 2022-09-10T10:20:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:56.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:56.739837+0000) 2022-09-10T10:20:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:57.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:57.215974+0000) 2022-09-10T10:20:57.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:57 smithi110 ceph-mon[97633]: pgmap v11254: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:57 smithi104 ceph-mon[97044]: pgmap v11254: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:20:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:57.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:57.740064+0000) 2022-09-10T10:20:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:58.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:58.216199+0000) 2022-09-10T10:20:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:58.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:58.740327+0000) 2022-09-10T10:20:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:20:59.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:59.216433+0000) 2022-09-10T10:20:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:20:59 smithi110 ceph-mon[97633]: pgmap v11255: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:20:59 smithi104 ceph-mon[97044]: pgmap v11255: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:20:59.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:20:59.740513+0000) 2022-09-10T10:21:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:00.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:00.216577+0000) 2022-09-10T10:21:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:00.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:00.740732+0000) 2022-09-10T10:21:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:01.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:01.216803+0000) 2022-09-10T10:21:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:01 smithi110 ceph-mon[97633]: pgmap v11256: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:01 smithi104 ceph-mon[97044]: pgmap v11256: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:01.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:01.740976+0000) 2022-09-10T10:21:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:02.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:02.217041+0000) 2022-09-10T10:21:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:02.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:02.741201+0000) 2022-09-10T10:21:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:03.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:03.217256+0000) 2022-09-10T10:21:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:03 smithi110 ceph-mon[97633]: pgmap v11257: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:03 smithi104 ceph-mon[97044]: pgmap v11257: 129 pgs: 129 active+clean; 8.9 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:03.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:03.741470+0000) 2022-09-10T10:21:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:04.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:04.217490+0000) 2022-09-10T10:21:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:04.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:04.741706+0000) 2022-09-10T10:21:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:05.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:05.217728+0000) 2022-09-10T10:21:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:05 smithi110 ceph-mon[97633]: pgmap v11258: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:21:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:05 smithi104 ceph-mon[97044]: pgmap v11258: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:21:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:05.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:05.741920+0000) 2022-09-10T10:21:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:06.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:06.217939+0000) 2022-09-10T10:21:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:06.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:06.742045+0000) 2022-09-10T10:21:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:07.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:07.218221+0000) 2022-09-10T10:21:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:07 smithi110 ceph-mon[97633]: pgmap v11259: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:21:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:07 smithi104 ceph-mon[97044]: pgmap v11259: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:21:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:07.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:07.742242+0000) 2022-09-10T10:21:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:08.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:08.218468+0000) 2022-09-10T10:21:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:08 smithi104 ceph-mon[97044]: osdmap e784: 12 total, 12 up, 12 in 2022-09-10T10:21:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:08.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:08.742477+0000) 2022-09-10T10:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:08 smithi110 ceph-mon[97633]: osdmap e784: 12 total, 12 up, 12 in 2022-09-10T10:21:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:09.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:09.218658+0000) 2022-09-10T10:21:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:21:09.095+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:21:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:09 smithi104 ceph-mon[97044]: pgmap v11261: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:21:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:09.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:09.742628+0000) 2022-09-10T10:21:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:09 smithi110 ceph-mon[97633]: pgmap v11261: 129 pgs: 129 active+clean; 8.9 MiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:21:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:10.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:10.218839+0000) 2022-09-10T10:21:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:10.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:10.742773+0000) 2022-09-10T10:21:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:11.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:11.219065+0000) 2022-09-10T10:21:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:11 smithi104 ceph-mon[97044]: pgmap v11262: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:21:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:11.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:11.742994+0000) 2022-09-10T10:21:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:11 smithi110 ceph-mon[97633]: pgmap v11262: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:21:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:12.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:12.219244+0000) 2022-09-10T10:21:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:12.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:12.743229+0000) 2022-09-10T10:21:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:13.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:13.219469+0000) 2022-09-10T10:21:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:13 smithi110 ceph-mon[97633]: pgmap v11263: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:21:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:13 smithi110 ceph-mon[97633]: osdmap e785: 12 total, 12 up, 12 in 2022-09-10T10:21:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:13 smithi104 ceph-mon[97044]: pgmap v11263: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:21:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:13 smithi104 ceph-mon[97044]: osdmap e785: 12 total, 12 up, 12 in 2022-09-10T10:21:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:13.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:13.743474+0000) 2022-09-10T10:21:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:14.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:14.219718+0000) 2022-09-10T10:21:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:14.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:14.743726+0000) 2022-09-10T10:21:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:15.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:15.219964+0000) 2022-09-10T10:21:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:15 smithi110 ceph-mon[97633]: pgmap v11265: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:21:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:15 smithi104 ceph-mon[97044]: pgmap v11265: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:21:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:15.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:15.743848+0000) 2022-09-10T10:21:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:16.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:16.220207+0000) 2022-09-10T10:21:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:16.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:16.744125+0000) 2022-09-10T10:21:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:17.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:17.220448+0000) 2022-09-10T10:21:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:17 smithi110 ceph-mon[97633]: pgmap v11266: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:21:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:17 smithi104 ceph-mon[97044]: pgmap v11266: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:21:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:17.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:17.744313+0000) 2022-09-10T10:21:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:18.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:18.220700+0000) 2022-09-10T10:21:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:18.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:18.744507+0000) 2022-09-10T10:21:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:19.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:19.220910+0000) 2022-09-10T10:21:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:19 smithi110 ceph-mon[97633]: pgmap v11267: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:21:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:19 smithi104 ceph-mon[97044]: pgmap v11267: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:21:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:19.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:19.744740+0000) 2022-09-10T10:21:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:20.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:20.221153+0000) 2022-09-10T10:21:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:20.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:20.744985+0000) 2022-09-10T10:21:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:21.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:21.221393+0000) 2022-09-10T10:21:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:21 smithi110 ceph-mon[97633]: pgmap v11268: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:21:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:21 smithi104 ceph-mon[97044]: pgmap v11268: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:21:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:21.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:21.745211+0000) 2022-09-10T10:21:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:22.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:22.221653+0000) 2022-09-10T10:21:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:22.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:22.745402+0000) 2022-09-10T10:21:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:23.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:23.221879+0000) 2022-09-10T10:21:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:23 smithi110 ceph-mon[97633]: pgmap v11269: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:21:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:23 smithi104 ceph-mon[97044]: pgmap v11269: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:21:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:23.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:23.745576+0000) 2022-09-10T10:21:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:24.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:24.222133+0000) 2022-09-10T10:21:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:21:24.107+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:21:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:24.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:24.745751+0000) 2022-09-10T10:21:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:25.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:25.222355+0000) 2022-09-10T10:21:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:25 smithi110 ceph-mon[97633]: pgmap v11270: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:21:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:25 smithi104 ceph-mon[97044]: pgmap v11270: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:21:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:25.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:25.745935+0000) 2022-09-10T10:21:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:26.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:26.222570+0000) 2022-09-10T10:21:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:26.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:26.746200+0000) 2022-09-10T10:21:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:27.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:27.222776+0000) 2022-09-10T10:21:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:27 smithi110 ceph-mon[97633]: pgmap v11271: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:27 smithi104 ceph-mon[97044]: pgmap v11271: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:27.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:27.746381+0000) 2022-09-10T10:21:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:28.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:28.223008+0000) 2022-09-10T10:21:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:28.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:28.746640+0000) 2022-09-10T10:21:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:29.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:29.223239+0000) 2022-09-10T10:21:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:29 smithi110 ceph-mon[97633]: pgmap v11272: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:29 smithi104 ceph-mon[97044]: pgmap v11272: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:29.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:29.746839+0000) 2022-09-10T10:21:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:30.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:30.223400+0000) 2022-09-10T10:21:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:30.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:30.747076+0000) 2022-09-10T10:21:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:31.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:31.223602+0000) 2022-09-10T10:21:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:31 smithi110 ceph-mon[97633]: pgmap v11273: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:31 smithi104 ceph-mon[97044]: pgmap v11273: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:31.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:31.747382+0000) 2022-09-10T10:21:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:32.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:32.223856+0000) 2022-09-10T10:21:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:32.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:32.747635+0000) 2022-09-10T10:21:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:33.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:33.224224+0000) 2022-09-10T10:21:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:33 smithi110 ceph-mon[97633]: pgmap v11274: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:33 smithi104 ceph-mon[97044]: pgmap v11274: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:33.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:33.747873+0000) 2022-09-10T10:21:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:34.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:34.224427+0000) 2022-09-10T10:21:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:34.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:34.748062+0000) 2022-09-10T10:21:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:35.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:35.224675+0000) 2022-09-10T10:21:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:35 smithi110 ceph-mon[97633]: pgmap v11275: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:35 smithi104 ceph-mon[97044]: pgmap v11275: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:35.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:35.748258+0000) 2022-09-10T10:21:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:36.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:36.224957+0000) 2022-09-10T10:21:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:36.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:36.748417+0000) 2022-09-10T10:21:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:37.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:37.225230+0000) 2022-09-10T10:21:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:37 smithi110 ceph-mon[97633]: pgmap v11276: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:37 smithi104 ceph-mon[97044]: pgmap v11276: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:37.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:37.748637+0000) 2022-09-10T10:21:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:38.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:38.225497+0000) 2022-09-10T10:21:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:38.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:38.748890+0000) 2022-09-10T10:21:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:39.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:39.225694+0000) 2022-09-10T10:21:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:21:39.123+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:21:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:39 smithi110 ceph-mon[97633]: pgmap v11277: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:39 smithi104 ceph-mon[97044]: pgmap v11277: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:39.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:39.749176+0000) 2022-09-10T10:21:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:40.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:40.225930+0000) 2022-09-10T10:21:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:40.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:40.749398+0000) 2022-09-10T10:21:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:41.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:41.226146+0000) 2022-09-10T10:21:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:41 smithi110 ceph-mon[97633]: pgmap v11278: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:41 smithi104 ceph-mon[97044]: pgmap v11278: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:41.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:41.749644+0000) 2022-09-10T10:21:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:42.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:42.226391+0000) 2022-09-10T10:21:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:42.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:42.749848+0000) 2022-09-10T10:21:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:43.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:43.226663+0000) 2022-09-10T10:21:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:43 smithi110 ceph-mon[97633]: pgmap v11279: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:43 smithi104 ceph-mon[97044]: pgmap v11279: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:43.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:43.750072+0000) 2022-09-10T10:21:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:44.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:44.226873+0000) 2022-09-10T10:21:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:44.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:44.750331+0000) 2022-09-10T10:21:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:45.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:45.227155+0000) 2022-09-10T10:21:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:45 smithi110 ceph-mon[97633]: pgmap v11280: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:21:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:21:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:21:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:21:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:45 smithi104 ceph-mon[97044]: pgmap v11280: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:21:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:45.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:45.750588+0000) 2022-09-10T10:21:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:46.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:46.227363+0000) 2022-09-10T10:21:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:46.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:46.750757+0000) 2022-09-10T10:21:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:47.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:47.227617+0000) 2022-09-10T10:21:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:47 smithi110 ceph-mon[97633]: pgmap v11281: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:47 smithi104 ceph-mon[97044]: pgmap v11281: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:47.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:47.750981+0000) 2022-09-10T10:21:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:48.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:48.227872+0000) 2022-09-10T10:21:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:48.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:48.751182+0000) 2022-09-10T10:21:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:49.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:49.228154+0000) 2022-09-10T10:21:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:49 smithi110 ceph-mon[97633]: pgmap v11282: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:49 smithi104 ceph-mon[97044]: pgmap v11282: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:49.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:49.751427+0000) 2022-09-10T10:21:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:50.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:50.228363+0000) 2022-09-10T10:21:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:50.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:50.751654+0000) 2022-09-10T10:21:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:51.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:51.228505+0000) 2022-09-10T10:21:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:51 smithi110 ceph-mon[97633]: pgmap v11283: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:51 smithi104 ceph-mon[97044]: pgmap v11283: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:51.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:51.751919+0000) 2022-09-10T10:21:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:52.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:52.228740+0000) 2022-09-10T10:21:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:52.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:52.752163+0000) 2022-09-10T10:21:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:53.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:53.229008+0000) 2022-09-10T10:21:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:53 smithi110 ceph-mon[97633]: pgmap v11284: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:53 smithi104 ceph-mon[97044]: pgmap v11284: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:53.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:53.752327+0000) 2022-09-10T10:21:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:54.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:54.229232+0000) 2022-09-10T10:21:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:21:54.135+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:21:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:54.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:54.752492+0000) 2022-09-10T10:21:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:55.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:55.229484+0000) 2022-09-10T10:21:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:55 smithi104 ceph-mon[97044]: pgmap v11285: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:55.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:55.752630+0000) 2022-09-10T10:21:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:55 smithi110 ceph-mon[97633]: pgmap v11285: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:21:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:56.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:56.229745+0000) 2022-09-10T10:21:57.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:56.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:56.752879+0000) 2022-09-10T10:21:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:57.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:57.229993+0000) 2022-09-10T10:21:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:57 smithi104 ceph-mon[97044]: pgmap v11286: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:57.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:57.753089+0000) 2022-09-10T10:21:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:57 smithi110 ceph-mon[97633]: pgmap v11286: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:21:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:58.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:58.230263+0000) 2022-09-10T10:21:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:58.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:58.753347+0000) 2022-09-10T10:21:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:21:59.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:59.230470+0000) 2022-09-10T10:22:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:21:59 smithi104 ceph-mon[97044]: pgmap v11287: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:21:59.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:21:59.753558+0000) 2022-09-10T10:22:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:21:59 smithi110 ceph-mon[97633]: pgmap v11287: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:00.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:00.230717+0000) 2022-09-10T10:22:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:00.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:00.753690+0000) 2022-09-10T10:22:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:01.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:01.230960+0000) 2022-09-10T10:22:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:01 smithi104 ceph-mon[97044]: pgmap v11288: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:01.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:01.753877+0000) 2022-09-10T10:22:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:01 smithi110 ceph-mon[97633]: pgmap v11288: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:02.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:02.231183+0000) 2022-09-10T10:22:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:02.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:02.754153+0000) 2022-09-10T10:22:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:03.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:03.231437+0000) 2022-09-10T10:22:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:03 smithi104 ceph-mon[97044]: pgmap v11289: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:03.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:03.754380+0000) 2022-09-10T10:22:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:03 smithi110 ceph-mon[97633]: pgmap v11289: 129 pgs: 129 active+clean; 8.9 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:04.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:04.231661+0000) 2022-09-10T10:22:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:04.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:04.754593+0000) 2022-09-10T10:22:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:05.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:05.231901+0000) 2022-09-10T10:22:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:05 smithi104 ceph-mon[97044]: pgmap v11290: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:22:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:05.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:05.754824+0000) 2022-09-10T10:22:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:05 smithi110 ceph-mon[97633]: pgmap v11290: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:22:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:06.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:06.232089+0000) 2022-09-10T10:22:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:06.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:06.755047+0000) 2022-09-10T10:22:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:07.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:07.232314+0000) 2022-09-10T10:22:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:07 smithi104 ceph-mon[97044]: pgmap v11291: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:22:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:07.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:07.755236+0000) 2022-09-10T10:22:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:07 smithi110 ceph-mon[97633]: pgmap v11291: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:22:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:08.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:08.232618+0000) 2022-09-10T10:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:08 smithi104 ceph-mon[97044]: osdmap e786: 12 total, 12 up, 12 in 2022-09-10T10:22:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:08.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:08.755449+0000) 2022-09-10T10:22:09.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:08 smithi110 ceph-mon[97633]: osdmap e786: 12 total, 12 up, 12 in 2022-09-10T10:22:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:09.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:09.232813+0000) 2022-09-10T10:22:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:22:09.141+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:22:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:09 smithi104 ceph-mon[97044]: pgmap v11293: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:22:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:09.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:09.755691+0000) 2022-09-10T10:22:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:09 smithi110 ceph-mon[97633]: pgmap v11293: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:22:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:10.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:10.232989+0000) 2022-09-10T10:22:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:10.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:10.755886+0000) 2022-09-10T10:22:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:11.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:11.233242+0000) 2022-09-10T10:22:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:11 smithi104 ceph-mon[97044]: pgmap v11294: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:22:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:11.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:11.756138+0000) 2022-09-10T10:22:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:11 smithi110 ceph-mon[97633]: pgmap v11294: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:22:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:12.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:12.233501+0000) 2022-09-10T10:22:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:12.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:12.756277+0000) 2022-09-10T10:22:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:13.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:13.233778+0000) 2022-09-10T10:22:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:13 smithi110 ceph-mon[97633]: pgmap v11295: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:22:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:13 smithi110 ceph-mon[97633]: osdmap e787: 12 total, 12 up, 12 in 2022-09-10T10:22:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:13 smithi104 ceph-mon[97044]: pgmap v11295: 129 pgs: 129 active+clean; 8.9 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:22:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:13 smithi104 ceph-mon[97044]: osdmap e787: 12 total, 12 up, 12 in 2022-09-10T10:22:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:13.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:13.756485+0000) 2022-09-10T10:22:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:14.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:14.233997+0000) 2022-09-10T10:22:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:14.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:14.756744+0000) 2022-09-10T10:22:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:15.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:15.234233+0000) 2022-09-10T10:22:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:15 smithi110 ceph-mon[97633]: pgmap v11297: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:22:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:15 smithi104 ceph-mon[97044]: pgmap v11297: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:22:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:15.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:15.756963+0000) 2022-09-10T10:22:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:16.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:16.234371+0000) 2022-09-10T10:22:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:16.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:16.757228+0000) 2022-09-10T10:22:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:17.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:17.234577+0000) 2022-09-10T10:22:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:17 smithi110 ceph-mon[97633]: pgmap v11298: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:22:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:17 smithi104 ceph-mon[97044]: pgmap v11298: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:22:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:17.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:17.757470+0000) 2022-09-10T10:22:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:18.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:18.234802+0000) 2022-09-10T10:22:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:18.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:18.757731+0000) 2022-09-10T10:22:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:19.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:19.235049+0000) 2022-09-10T10:22:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:19 smithi110 ceph-mon[97633]: pgmap v11299: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:22:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:19 smithi104 ceph-mon[97044]: pgmap v11299: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:22:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:19.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:19.757868+0000) 2022-09-10T10:22:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:20.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:20.235343+0000) 2022-09-10T10:22:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:20.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:20.758053+0000) 2022-09-10T10:22:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:21.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:21.235563+0000) 2022-09-10T10:22:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:21 smithi110 ceph-mon[97633]: pgmap v11300: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:22:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:21 smithi104 ceph-mon[97044]: pgmap v11300: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:22:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:21.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:21.758276+0000) 2022-09-10T10:22:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:22.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:22.235864+0000) 2022-09-10T10:22:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:22.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:22.758496+0000) 2022-09-10T10:22:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:23.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:23.236178+0000) 2022-09-10T10:22:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:23 smithi110 ceph-mon[97633]: pgmap v11301: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:22:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:23 smithi104 ceph-mon[97044]: pgmap v11301: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:22:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:23.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:23.758734+0000) 2022-09-10T10:22:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:24.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:24.236426+0000) 2022-09-10T10:22:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:22:24.152+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:22:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:24.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:24.759008+0000) 2022-09-10T10:22:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:25.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:25.236685+0000) 2022-09-10T10:22:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:25 smithi110 ceph-mon[97633]: pgmap v11302: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:22:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:25 smithi104 ceph-mon[97044]: pgmap v11302: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:22:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:25.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:25.759201+0000) 2022-09-10T10:22:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:26.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:26.236932+0000) 2022-09-10T10:22:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:26.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:26.759437+0000) 2022-09-10T10:22:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:27.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:27.237186+0000) 2022-09-10T10:22:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:27 smithi110 ceph-mon[97633]: pgmap v11303: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:27 smithi104 ceph-mon[97044]: pgmap v11303: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:27.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:27.759580+0000) 2022-09-10T10:22:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:28.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:28.237371+0000) 2022-09-10T10:22:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:28.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:28.759809+0000) 2022-09-10T10:22:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:29.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:29.237595+0000) 2022-09-10T10:22:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:29 smithi110 ceph-mon[97633]: pgmap v11304: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:29 smithi104 ceph-mon[97044]: pgmap v11304: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:29.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:29.760043+0000) 2022-09-10T10:22:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:30.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:30.237822+0000) 2022-09-10T10:22:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:30.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:30.760323+0000) 2022-09-10T10:22:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:31.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:31.238051+0000) 2022-09-10T10:22:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:31 smithi110 ceph-mon[97633]: pgmap v11305: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:31 smithi104 ceph-mon[97044]: pgmap v11305: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:31.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:31.760579+0000) 2022-09-10T10:22:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:32.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:32.238330+0000) 2022-09-10T10:22:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:32.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:32.760795+0000) 2022-09-10T10:22:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:33.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:33.238585+0000) 2022-09-10T10:22:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:33 smithi110 ceph-mon[97633]: pgmap v11306: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:33 smithi104 ceph-mon[97044]: pgmap v11306: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:33.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:33.761030+0000) 2022-09-10T10:22:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:34.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:34.238767+0000) 2022-09-10T10:22:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:34.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:34.761197+0000) 2022-09-10T10:22:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:35.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:35.238947+0000) 2022-09-10T10:22:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:35 smithi110 ceph-mon[97633]: pgmap v11307: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:35 smithi104 ceph-mon[97044]: pgmap v11307: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:35.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:35.761397+0000) 2022-09-10T10:22:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:36.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:36.239129+0000) 2022-09-10T10:22:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:36.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:36.761637+0000) 2022-09-10T10:22:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:37.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:37.239337+0000) 2022-09-10T10:22:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:37 smithi110 ceph-mon[97633]: pgmap v11308: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:37 smithi104 ceph-mon[97044]: pgmap v11308: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:37.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:37.761828+0000) 2022-09-10T10:22:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:38.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:38.239554+0000) 2022-09-10T10:22:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:38.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:38.762016+0000) 2022-09-10T10:22:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:22:39.153+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:22:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:39.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:39.239801+0000) 2022-09-10T10:22:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:39 smithi110 ceph-mon[97633]: pgmap v11309: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:39 smithi104 ceph-mon[97044]: pgmap v11309: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:39.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:39.762246+0000) 2022-09-10T10:22:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:40.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:40.240059+0000) 2022-09-10T10:22:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:40.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:40.762493+0000) 2022-09-10T10:22:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:41.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:41.240334+0000) 2022-09-10T10:22:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:41 smithi110 ceph-mon[97633]: pgmap v11310: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:41 smithi104 ceph-mon[97044]: pgmap v11310: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:41.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:41.762643+0000) 2022-09-10T10:22:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:42.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:42.240580+0000) 2022-09-10T10:22:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:42.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:42.762823+0000) 2022-09-10T10:22:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:43.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:43.240840+0000) 2022-09-10T10:22:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:43 smithi104 ceph-mon[97044]: pgmap v11311: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:43.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:43.763012+0000) 2022-09-10T10:22:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:43 smithi110 ceph-mon[97633]: pgmap v11311: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:44.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:44.241133+0000) 2022-09-10T10:22:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:44.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:44.763205+0000) 2022-09-10T10:22:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:45.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:45.241375+0000) 2022-09-10T10:22:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:45 smithi104 ceph-mon[97044]: pgmap v11312: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:22:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:22:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:22:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:22:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:45.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:45.763422+0000) 2022-09-10T10:22:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:45 smithi110 ceph-mon[97633]: pgmap v11312: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:22:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:22:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:22:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:22:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:46.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:46.241594+0000) 2022-09-10T10:22:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:46.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:46.763691+0000) 2022-09-10T10:22:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:47.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:47.241789+0000) 2022-09-10T10:22:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:47 smithi104 ceph-mon[97044]: pgmap v11313: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:47.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:47.763923+0000) 2022-09-10T10:22:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:47 smithi110 ceph-mon[97633]: pgmap v11313: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:48.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:48.242001+0000) 2022-09-10T10:22:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:48.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:48.764181+0000) 2022-09-10T10:22:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:49.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:49.242247+0000) 2022-09-10T10:22:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:49 smithi104 ceph-mon[97044]: pgmap v11314: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:49.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:49.764344+0000) 2022-09-10T10:22:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:49 smithi110 ceph-mon[97633]: pgmap v11314: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:50.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:50.242448+0000) 2022-09-10T10:22:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:50.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:50.764596+0000) 2022-09-10T10:22:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:51.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:51.242594+0000) 2022-09-10T10:22:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:51 smithi104 ceph-mon[97044]: pgmap v11315: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:51.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:51.764822+0000) 2022-09-10T10:22:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:51 smithi110 ceph-mon[97633]: pgmap v11315: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:52.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:52.242835+0000) 2022-09-10T10:22:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:52.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:52.765069+0000) 2022-09-10T10:22:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:53.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:53.243075+0000) 2022-09-10T10:22:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:53 smithi104 ceph-mon[97044]: pgmap v11316: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:53.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:53.765347+0000) 2022-09-10T10:22:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:53 smithi110 ceph-mon[97633]: pgmap v11316: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:22:54.159+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:22:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:54.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:54.243289+0000) 2022-09-10T10:22:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:54.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:54.765575+0000) 2022-09-10T10:22:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:55.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:55.243508+0000) 2022-09-10T10:22:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:55 smithi104 ceph-mon[97044]: pgmap v11317: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:55.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:55.765818+0000) 2022-09-10T10:22:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:55 smithi110 ceph-mon[97633]: pgmap v11317: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:22:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:56.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:56.243762+0000) 2022-09-10T10:22:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:56.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:56.766084+0000) 2022-09-10T10:22:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:57.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:57.243958+0000) 2022-09-10T10:22:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:57 smithi104 ceph-mon[97044]: pgmap v11318: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:57.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:57.766373+0000) 2022-09-10T10:22:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:57 smithi110 ceph-mon[97633]: pgmap v11318: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:22:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:58.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:58.244204+0000) 2022-09-10T10:22:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:58.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:58.766596+0000) 2022-09-10T10:22:59.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:22:59.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:59.244445+0000) 2022-09-10T10:23:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:22:59 smithi104 ceph-mon[97044]: pgmap v11319: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:22:59.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:22:59.766822+0000) 2022-09-10T10:23:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:22:59 smithi110 ceph-mon[97633]: pgmap v11319: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:00.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:00.244632+0000) 2022-09-10T10:23:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:00.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:00.767003+0000) 2022-09-10T10:23:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:01.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:01.244845+0000) 2022-09-10T10:23:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:01 smithi104 ceph-mon[97044]: pgmap v11320: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:01.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:01.767236+0000) 2022-09-10T10:23:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:01 smithi110 ceph-mon[97633]: pgmap v11320: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:02.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:02.245069+0000) 2022-09-10T10:23:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:02.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:02.767466+0000) 2022-09-10T10:23:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:03.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:03.245256+0000) 2022-09-10T10:23:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:03 smithi104 ceph-mon[97044]: pgmap v11321: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:03.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:03.767695+0000) 2022-09-10T10:23:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:03 smithi110 ceph-mon[97633]: pgmap v11321: 129 pgs: 129 active+clean; 8.9 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:04.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:04.245475+0000) 2022-09-10T10:23:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:04.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:04.767921+0000) 2022-09-10T10:23:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:05.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:05.245708+0000) 2022-09-10T10:23:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:05 smithi104 ceph-mon[97044]: pgmap v11322: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:23:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:05.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:05.768086+0000) 2022-09-10T10:23:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:05 smithi110 ceph-mon[97633]: pgmap v11322: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:23:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:06.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:06.245934+0000) 2022-09-10T10:23:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:06.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:06.768329+0000) 2022-09-10T10:23:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:07.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:07.246177+0000) 2022-09-10T10:23:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:07 smithi104 ceph-mon[97044]: pgmap v11323: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:23:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:07.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:07.768557+0000) 2022-09-10T10:23:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:07 smithi110 ceph-mon[97633]: pgmap v11323: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:23:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:08.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:08.246337+0000) 2022-09-10T10:23:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:08 smithi104 ceph-mon[97044]: osdmap e788: 12 total, 12 up, 12 in 2022-09-10T10:23:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:08.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:08.768803+0000) 2022-09-10T10:23:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:08 smithi110 ceph-mon[97633]: osdmap e788: 12 total, 12 up, 12 in 2022-09-10T10:23:09.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:23:09.175+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:23:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:09.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:09.246491+0000) 2022-09-10T10:23:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:09 smithi104 ceph-mon[97044]: pgmap v11325: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:23:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:09.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:09.769037+0000) 2022-09-10T10:23:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:09 smithi110 ceph-mon[97633]: pgmap v11325: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:23:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:10.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:10.246675+0000) 2022-09-10T10:23:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:10.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:10.769263+0000) 2022-09-10T10:23:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:11.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:11.246836+0000) 2022-09-10T10:23:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:11 smithi104 ceph-mon[97044]: pgmap v11326: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:23:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:11.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:11.769514+0000) 2022-09-10T10:23:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:11 smithi110 ceph-mon[97633]: pgmap v11326: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:23:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:12.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:12.247034+0000) 2022-09-10T10:23:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:12.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:12.769666+0000) 2022-09-10T10:23:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:13.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:13.247292+0000) 2022-09-10T10:23:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:13 smithi110 ceph-mon[97633]: pgmap v11327: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:23:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:13 smithi110 ceph-mon[97633]: osdmap e789: 12 total, 12 up, 12 in 2022-09-10T10:23:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:13 smithi104 ceph-mon[97044]: pgmap v11327: 129 pgs: 129 active+clean; 8.9 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:23:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:13 smithi104 ceph-mon[97044]: osdmap e789: 12 total, 12 up, 12 in 2022-09-10T10:23:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:13.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:13.769851+0000) 2022-09-10T10:23:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:14.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:14.247490+0000) 2022-09-10T10:23:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:14.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:14.770122+0000) 2022-09-10T10:23:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:15.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:15.247724+0000) 2022-09-10T10:23:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:15 smithi110 ceph-mon[97633]: pgmap v11329: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:23:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:15 smithi104 ceph-mon[97044]: pgmap v11329: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:23:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:15.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:15.770369+0000) 2022-09-10T10:23:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:16.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:16.247925+0000) 2022-09-10T10:23:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:16.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:16.770625+0000) 2022-09-10T10:23:17.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:17.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:17.248131+0000) 2022-09-10T10:23:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:17 smithi110 ceph-mon[97633]: pgmap v11330: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:23:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:17 smithi104 ceph-mon[97044]: pgmap v11330: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:23:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:17.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:17.770847+0000) 2022-09-10T10:23:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:18.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:18.248334+0000) 2022-09-10T10:23:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:18.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:18.771084+0000) 2022-09-10T10:23:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:19.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:19.248548+0000) 2022-09-10T10:23:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:19 smithi110 ceph-mon[97633]: pgmap v11331: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:23:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:19 smithi104 ceph-mon[97044]: pgmap v11331: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:23:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:19.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:19.771253+0000) 2022-09-10T10:23:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:20.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:20.248767+0000) 2022-09-10T10:23:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:20.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:20.771398+0000) 2022-09-10T10:23:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:21.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:21.249022+0000) 2022-09-10T10:23:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:21 smithi110 ceph-mon[97633]: pgmap v11332: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:23:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:21 smithi104 ceph-mon[97044]: pgmap v11332: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:23:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:21.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:21.771591+0000) 2022-09-10T10:23:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:22.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:22.249176+0000) 2022-09-10T10:23:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:22.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:22.771792+0000) 2022-09-10T10:23:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:23.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:23.249409+0000) 2022-09-10T10:23:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:23 smithi110 ceph-mon[97633]: pgmap v11333: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:23:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:23 smithi104 ceph-mon[97044]: pgmap v11333: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:23:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:23.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:23.772064+0000) 2022-09-10T10:23:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:24.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:24.249610+0000) 2022-09-10T10:23:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:23:24.190+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:23:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:24.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:24.772339+0000) 2022-09-10T10:23:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:25.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:25.249814+0000) 2022-09-10T10:23:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:25 smithi110 ceph-mon[97633]: pgmap v11334: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:23:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:25 smithi104 ceph-mon[97044]: pgmap v11334: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:23:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:25.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:25.772481+0000) 2022-09-10T10:23:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:26.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:26.250065+0000) 2022-09-10T10:23:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:26.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:26.772646+0000) 2022-09-10T10:23:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:27.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:27.250278+0000) 2022-09-10T10:23:27.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:27 smithi110 ceph-mon[97633]: pgmap v11335: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:23:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:27 smithi104 ceph-mon[97044]: pgmap v11335: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:23:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:27.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:27.772852+0000) 2022-09-10T10:23:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:28.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:28.250526+0000) 2022-09-10T10:23:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:28.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:28.773087+0000) 2022-09-10T10:23:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:29.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:29.250781+0000) 2022-09-10T10:23:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:29 smithi110 ceph-mon[97633]: pgmap v11336: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:23:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:29 smithi104 ceph-mon[97044]: pgmap v11336: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:23:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:29.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:29.773278+0000) 2022-09-10T10:23:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:30.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:30.250963+0000) 2022-09-10T10:23:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:30.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:30.773486+0000) 2022-09-10T10:23:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:31.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:31.251225+0000) 2022-09-10T10:23:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:31 smithi110 ceph-mon[97633]: pgmap v11337: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:31 smithi104 ceph-mon[97044]: pgmap v11337: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:31.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:31.773663+0000) 2022-09-10T10:23:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:32.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:32.251449+0000) 2022-09-10T10:23:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:32.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:32.773929+0000) 2022-09-10T10:23:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:33.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:33.251652+0000) 2022-09-10T10:23:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:33 smithi104 ceph-mon[97044]: pgmap v11338: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:33.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:33.774166+0000) 2022-09-10T10:23:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:33 smithi110 ceph-mon[97633]: pgmap v11338: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:34.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:34.251834+0000) 2022-09-10T10:23:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:34.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:34.774381+0000) 2022-09-10T10:23:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:35.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:35.252045+0000) 2022-09-10T10:23:36.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:35 smithi104 ceph-mon[97044]: pgmap v11339: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:35.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:35.774610+0000) 2022-09-10T10:23:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:35 smithi110 ceph-mon[97633]: pgmap v11339: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:36.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:36.252259+0000) 2022-09-10T10:23:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:36.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:36.774838+0000) 2022-09-10T10:23:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:37.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:37.252508+0000) 2022-09-10T10:23:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:37 smithi104 ceph-mon[97044]: pgmap v11340: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:37.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:37.775064+0000) 2022-09-10T10:23:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:37 smithi110 ceph-mon[97633]: pgmap v11340: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:38.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:38.252724+0000) 2022-09-10T10:23:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:38.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:38.775329+0000) 2022-09-10T10:23:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:23:39.196+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:23:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:39.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:39.252910+0000) 2022-09-10T10:23:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:39 smithi104 ceph-mon[97044]: pgmap v11341: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:39.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:39.775555+0000) 2022-09-10T10:23:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:39 smithi110 ceph-mon[97633]: pgmap v11341: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:40.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:40.253155+0000) 2022-09-10T10:23:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:40.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:40.775814+0000) 2022-09-10T10:23:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:41.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:41.253289+0000) 2022-09-10T10:23:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:41 smithi104 ceph-mon[97044]: pgmap v11342: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:41.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:41.776032+0000) 2022-09-10T10:23:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:41 smithi110 ceph-mon[97633]: pgmap v11342: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:42.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:42.253463+0000) 2022-09-10T10:23:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:42.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:42.776279+0000) 2022-09-10T10:23:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:43.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:43.253676+0000) 2022-09-10T10:23:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:43 smithi104 ceph-mon[97044]: pgmap v11343: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:43.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:43.776498+0000) 2022-09-10T10:23:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:43 smithi110 ceph-mon[97633]: pgmap v11343: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:44.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:44.253903+0000) 2022-09-10T10:23:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:44.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:44.776754+0000) 2022-09-10T10:23:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:45.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:45.254136+0000) 2022-09-10T10:23:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:45 smithi104 ceph-mon[97044]: pgmap v11344: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:23:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:23:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:23:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:23:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:45.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:45.776901+0000) 2022-09-10T10:23:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:45 smithi110 ceph-mon[97633]: pgmap v11344: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:23:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:23:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:23:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:23:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:46.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:46.254339+0000) 2022-09-10T10:23:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:46.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:46.777152+0000) 2022-09-10T10:23:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:47.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:47.254516+0000) 2022-09-10T10:23:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:47 smithi104 ceph-mon[97044]: pgmap v11345: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:47.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:47.777332+0000) 2022-09-10T10:23:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:47 smithi110 ceph-mon[97633]: pgmap v11345: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:48.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:48.254691+0000) 2022-09-10T10:23:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:48.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:48.777566+0000) 2022-09-10T10:23:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:49.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:49.254866+0000) 2022-09-10T10:23:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:49 smithi104 ceph-mon[97044]: pgmap v11346: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:49.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:49.777744+0000) 2022-09-10T10:23:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:49 smithi110 ceph-mon[97633]: pgmap v11346: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:50.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:50.255072+0000) 2022-09-10T10:23:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:50.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:50.777946+0000) 2022-09-10T10:23:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:51.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:51.255293+0000) 2022-09-10T10:23:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:51 smithi104 ceph-mon[97044]: pgmap v11347: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:51.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:51.778192+0000) 2022-09-10T10:23:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:51 smithi110 ceph-mon[97633]: pgmap v11347: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:52.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:52.255522+0000) 2022-09-10T10:23:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:52.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:52.778388+0000) 2022-09-10T10:23:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:53.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:53.255802+0000) 2022-09-10T10:23:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:53 smithi104 ceph-mon[97044]: pgmap v11348: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:53.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:53.778623+0000) 2022-09-10T10:23:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:53 smithi110 ceph-mon[97633]: pgmap v11348: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:23:54.206+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:23:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:54.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:54.256028+0000) 2022-09-10T10:23:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:54.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:54.778850+0000) 2022-09-10T10:23:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:55.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:55.256198+0000) 2022-09-10T10:23:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:55 smithi104 ceph-mon[97044]: pgmap v11349: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:55.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:55.779120+0000) 2022-09-10T10:23:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:55 smithi110 ceph-mon[97633]: pgmap v11349: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:23:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:56.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:56.256411+0000) 2022-09-10T10:23:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:56.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:56.779341+0000) 2022-09-10T10:23:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:57.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:57.256543+0000) 2022-09-10T10:23:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:57 smithi104 ceph-mon[97044]: pgmap v11350: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:57.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:57.779589+0000) 2022-09-10T10:23:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:57 smithi110 ceph-mon[97633]: pgmap v11350: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:23:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:58.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:58.256790+0000) 2022-09-10T10:23:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:58.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:58.779839+0000) 2022-09-10T10:23:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:23:59.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:59.256973+0000) 2022-09-10T10:24:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:23:59 smithi104 ceph-mon[97044]: pgmap v11351: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:23:59.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:23:59.780019+0000) 2022-09-10T10:24:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:23:59 smithi110 ceph-mon[97633]: pgmap v11351: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:00.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:00.257146+0000) 2022-09-10T10:24:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:00.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:00.780146+0000) 2022-09-10T10:24:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:01.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:01.257295+0000) 2022-09-10T10:24:02.056 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:01 smithi104 ceph-mon[97044]: pgmap v11352: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:02.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:01.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:01.780351+0000) 2022-09-10T10:24:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:01 smithi110 ceph-mon[97633]: pgmap v11352: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:02.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:02.257521+0000) 2022-09-10T10:24:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:02.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:02.780532+0000) 2022-09-10T10:24:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:03.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:03.257767+0000) 2022-09-10T10:24:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:03 smithi104 ceph-mon[97044]: pgmap v11353: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:03.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:03.780791+0000) 2022-09-10T10:24:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:03 smithi110 ceph-mon[97633]: pgmap v11353: 129 pgs: 129 active+clean; 8.9 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:04.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:04.258002+0000) 2022-09-10T10:24:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:04.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:04.780989+0000) 2022-09-10T10:24:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:05.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:05.258230+0000) 2022-09-10T10:24:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:05 smithi104 ceph-mon[97044]: pgmap v11354: 129 pgs: 129 active+clean; 8.9 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:24:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:05.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:05.781239+0000) 2022-09-10T10:24:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:05 smithi110 ceph-mon[97633]: pgmap v11354: 129 pgs: 129 active+clean; 8.9 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:24:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:06.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:06.258450+0000) 2022-09-10T10:24:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:06.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:06.781425+0000) 2022-09-10T10:24:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:07.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:07.258647+0000) 2022-09-10T10:24:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:07 smithi104 ceph-mon[97044]: pgmap v11355: 129 pgs: 129 active+clean; 8.9 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:24:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:07.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:07.781601+0000) 2022-09-10T10:24:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:07 smithi110 ceph-mon[97633]: pgmap v11355: 129 pgs: 129 active+clean; 8.9 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:24:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:08.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:08.258874+0000) 2022-09-10T10:24:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:08 smithi104 ceph-mon[97044]: osdmap e790: 12 total, 12 up, 12 in 2022-09-10T10:24:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:08.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:08.781836+0000) 2022-09-10T10:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:08 smithi110 ceph-mon[97633]: osdmap e790: 12 total, 12 up, 12 in 2022-09-10T10:24:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:09.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:09.259042+0000) 2022-09-10T10:24:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:24:09.214+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:24:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:09 smithi104 ceph-mon[97044]: pgmap v11357: 129 pgs: 129 active+clean; 8.9 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:24:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:09.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:09.782124+0000) 2022-09-10T10:24:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:09 smithi110 ceph-mon[97633]: pgmap v11357: 129 pgs: 129 active+clean; 8.9 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:24:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:10.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:10.259270+0000) 2022-09-10T10:24:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:10.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:10.782352+0000) 2022-09-10T10:24:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:11.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:11.259533+0000) 2022-09-10T10:24:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:11 smithi104 ceph-mon[97044]: pgmap v11358: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:24:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:11.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:11.782580+0000) 2022-09-10T10:24:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:11 smithi110 ceph-mon[97633]: pgmap v11358: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:24:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:12.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:12.259676+0000) 2022-09-10T10:24:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:12.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:12.782837+0000) 2022-09-10T10:24:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:13.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:13.259905+0000) 2022-09-10T10:24:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:13 smithi110 ceph-mon[97633]: pgmap v11359: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:24:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:13 smithi110 ceph-mon[97633]: osdmap e791: 12 total, 12 up, 12 in 2022-09-10T10:24:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:13 smithi104 ceph-mon[97044]: pgmap v11359: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:24:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:13 smithi104 ceph-mon[97044]: osdmap e791: 12 total, 12 up, 12 in 2022-09-10T10:24:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:13.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:13.783084+0000) 2022-09-10T10:24:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:14.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:14.260079+0000) 2022-09-10T10:24:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:14.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:14.783330+0000) 2022-09-10T10:24:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:15.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:15.260303+0000) 2022-09-10T10:24:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:15 smithi110 ceph-mon[97633]: pgmap v11361: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:24:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:15 smithi104 ceph-mon[97044]: pgmap v11361: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:24:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:15.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:15.783510+0000) 2022-09-10T10:24:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:16.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:16.260492+0000) 2022-09-10T10:24:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:16.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:16.783683+0000) 2022-09-10T10:24:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:17.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:17.260740+0000) 2022-09-10T10:24:17.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:17 smithi110 ceph-mon[97633]: pgmap v11362: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:24:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:17 smithi104 ceph-mon[97044]: pgmap v11362: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:24:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:17.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:17.783927+0000) 2022-09-10T10:24:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:18.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:18.260966+0000) 2022-09-10T10:24:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:18.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:18.784135+0000) 2022-09-10T10:24:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:19.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:19.261186+0000) 2022-09-10T10:24:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:19 smithi110 ceph-mon[97633]: pgmap v11363: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:24:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:19 smithi104 ceph-mon[97044]: pgmap v11363: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:24:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:19.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:19.784331+0000) 2022-09-10T10:24:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:20.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:20.261387+0000) 2022-09-10T10:24:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:20.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:20.784574+0000) 2022-09-10T10:24:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:21.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:21.261544+0000) 2022-09-10T10:24:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:21 smithi110 ceph-mon[97633]: pgmap v11364: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:24:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:21 smithi104 ceph-mon[97044]: pgmap v11364: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:24:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:21.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:21.784788+0000) 2022-09-10T10:24:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:22.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:22.261777+0000) 2022-09-10T10:24:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:22.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:22.784962+0000) 2022-09-10T10:24:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:23.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:23.262007+0000) 2022-09-10T10:24:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:23 smithi104 ceph-mon[97044]: pgmap v11365: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:24:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:23.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:23.785181+0000) 2022-09-10T10:24:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:23 smithi110 ceph-mon[97633]: pgmap v11365: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:24:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:24.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:24.262197+0000) 2022-09-10T10:24:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:24:24.222+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:24:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:24.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:24.785345+0000) 2022-09-10T10:24:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:25.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:25.262418+0000) 2022-09-10T10:24:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:25 smithi104 ceph-mon[97044]: pgmap v11366: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:24:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:25.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:25.785515+0000) 2022-09-10T10:24:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:25 smithi110 ceph-mon[97633]: pgmap v11366: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:24:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:26.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:26.262650+0000) 2022-09-10T10:24:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:26.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:26.785698+0000) 2022-09-10T10:24:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:27.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:27.262924+0000) 2022-09-10T10:24:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:27 smithi104 ceph-mon[97044]: pgmap v11367: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:27.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:27.785862+0000) 2022-09-10T10:24:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:27 smithi110 ceph-mon[97633]: pgmap v11367: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:28.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:28.263139+0000) 2022-09-10T10:24:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:28.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:28.786059+0000) 2022-09-10T10:24:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:29.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:29.263303+0000) 2022-09-10T10:24:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:29 smithi104 ceph-mon[97044]: pgmap v11368: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:29.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:29.786290+0000) 2022-09-10T10:24:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:29 smithi110 ceph-mon[97633]: pgmap v11368: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:30.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:30.263526+0000) 2022-09-10T10:24:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:30.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:30.786536+0000) 2022-09-10T10:24:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:31.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:31.263702+0000) 2022-09-10T10:24:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:31 smithi104 ceph-mon[97044]: pgmap v11369: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:31.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:31.786721+0000) 2022-09-10T10:24:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:31 smithi110 ceph-mon[97633]: pgmap v11369: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:32.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:32.263956+0000) 2022-09-10T10:24:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:32.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:32.786957+0000) 2022-09-10T10:24:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:33.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:33.264255+0000) 2022-09-10T10:24:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:33 smithi104 ceph-mon[97044]: pgmap v11370: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:33.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:33.787204+0000) 2022-09-10T10:24:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:33 smithi110 ceph-mon[97633]: pgmap v11370: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:34.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:34.264479+0000) 2022-09-10T10:24:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:34.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:34.787445+0000) 2022-09-10T10:24:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:35.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:35.264716+0000) 2022-09-10T10:24:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:35 smithi104 ceph-mon[97044]: pgmap v11371: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:35.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:35.787717+0000) 2022-09-10T10:24:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:35 smithi110 ceph-mon[97633]: pgmap v11371: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:36.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:36.264914+0000) 2022-09-10T10:24:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:36.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:36.787920+0000) 2022-09-10T10:24:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:37.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:37.265128+0000) 2022-09-10T10:24:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:37 smithi104 ceph-mon[97044]: pgmap v11372: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:37.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:37.788191+0000) 2022-09-10T10:24:38.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:37 smithi110 ceph-mon[97633]: pgmap v11372: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:38.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:38.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:38.265320+0000) 2022-09-10T10:24:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:38.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:38.788351+0000) 2022-09-10T10:24:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:39.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:39.265485+0000) 2022-09-10T10:24:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:24:39.238+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:24:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:39 smithi104 ceph-mon[97044]: pgmap v11373: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:39.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:39.788529+0000) 2022-09-10T10:24:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:39 smithi110 ceph-mon[97633]: pgmap v11373: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:40.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:40.265664+0000) 2022-09-10T10:24:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:40.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:40.788739+0000) 2022-09-10T10:24:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:41.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:41.265899+0000) 2022-09-10T10:24:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:41 smithi104 ceph-mon[97044]: pgmap v11374: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:41.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:41.788954+0000) 2022-09-10T10:24:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:41 smithi110 ceph-mon[97633]: pgmap v11374: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:42.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:42.266156+0000) 2022-09-10T10:24:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:42.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:42.789196+0000) 2022-09-10T10:24:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:43.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:43.266416+0000) 2022-09-10T10:24:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:43 smithi104 ceph-mon[97044]: pgmap v11375: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:43.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:43.789447+0000) 2022-09-10T10:24:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:43 smithi110 ceph-mon[97633]: pgmap v11375: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:44.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:44.266686+0000) 2022-09-10T10:24:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:44.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:44.789672+0000) 2022-09-10T10:24:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:45.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:45.266863+0000) 2022-09-10T10:24:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:45 smithi104 ceph-mon[97044]: pgmap v11376: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:24:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:45.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:45.789897+0000) 2022-09-10T10:24:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:45 smithi110 ceph-mon[97633]: pgmap v11376: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:24:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:24:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:24:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:24:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:46.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:46.267072+0000) 2022-09-10T10:24:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:46.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:46.790053+0000) 2022-09-10T10:24:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:47.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:47.267323+0000) 2022-09-10T10:24:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:47 smithi104 ceph-mon[97044]: pgmap v11377: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:47.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:47.790190+0000) 2022-09-10T10:24:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:47 smithi110 ceph-mon[97633]: pgmap v11377: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:48.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:48.267532+0000) 2022-09-10T10:24:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:48.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:48.790434+0000) 2022-09-10T10:24:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:49.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:49.267740+0000) 2022-09-10T10:24:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:49 smithi104 ceph-mon[97044]: pgmap v11378: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:49.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:49.790670+0000) 2022-09-10T10:24:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:49 smithi110 ceph-mon[97633]: pgmap v11378: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:50.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:50.267924+0000) 2022-09-10T10:24:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:50.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:50.790871+0000) 2022-09-10T10:24:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:51.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:51.268170+0000) 2022-09-10T10:24:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:51 smithi104 ceph-mon[97044]: pgmap v11379: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:51.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:51.791124+0000) 2022-09-10T10:24:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:51 smithi110 ceph-mon[97633]: pgmap v11379: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:52.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:52.268330+0000) 2022-09-10T10:24:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:52.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:52.791258+0000) 2022-09-10T10:24:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:53.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:53.268527+0000) 2022-09-10T10:24:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:53 smithi104 ceph-mon[97044]: pgmap v11380: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:53.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:53.791487+0000) 2022-09-10T10:24:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:53 smithi110 ceph-mon[97633]: pgmap v11380: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:24:54.251+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:24:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:54.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:54.268717+0000) 2022-09-10T10:24:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:54.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:54.791707+0000) 2022-09-10T10:24:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:55.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:55.268930+0000) 2022-09-10T10:24:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:55 smithi104 ceph-mon[97044]: pgmap v11381: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:55.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:55.791933+0000) 2022-09-10T10:24:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:55 smithi110 ceph-mon[97633]: pgmap v11381: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:24:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:56.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:56.269184+0000) 2022-09-10T10:24:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:56.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:56.792168+0000) 2022-09-10T10:24:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:57.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:57.269429+0000) 2022-09-10T10:24:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:57 smithi104 ceph-mon[97044]: pgmap v11382: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:57.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:57.792409+0000) 2022-09-10T10:24:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:57 smithi110 ceph-mon[97633]: pgmap v11382: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:24:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:58.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:58.269624+0000) 2022-09-10T10:24:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:58.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:58.792671+0000) 2022-09-10T10:24:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:24:59.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:59.269838+0000) 2022-09-10T10:25:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:24:59 smithi104 ceph-mon[97044]: pgmap v11383: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:24:59.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:24:59.792879+0000) 2022-09-10T10:25:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:24:59 smithi110 ceph-mon[97633]: pgmap v11383: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:00.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:00.270026+0000) 2022-09-10T10:25:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:00.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:00.793028+0000) 2022-09-10T10:25:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:01.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:01.270233+0000) 2022-09-10T10:25:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:01 smithi104 ceph-mon[97044]: pgmap v11384: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:01.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:01.793252+0000) 2022-09-10T10:25:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:01 smithi110 ceph-mon[97633]: pgmap v11384: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:02.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:02.270485+0000) 2022-09-10T10:25:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:02.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:02.793498+0000) 2022-09-10T10:25:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:03.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:03.270635+0000) 2022-09-10T10:25:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:03 smithi104 ceph-mon[97044]: pgmap v11385: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:04.724 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:03 smithi110 ceph-mon[97633]: pgmap v11385: 129 pgs: 129 active+clean; 8.9 MiB data, 1006 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:04.725 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:03.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:03.793683+0000) 2022-09-10T10:25:04.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:04.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:04.270878+0000) 2022-09-10T10:25:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:04.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:04.793868+0000) 2022-09-10T10:25:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:05.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:05.271147+0000) 2022-09-10T10:25:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:05 smithi104 ceph-mon[97044]: pgmap v11386: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:25:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:05.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:05.794186+0000) 2022-09-10T10:25:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:05 smithi110 ceph-mon[97633]: pgmap v11386: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:25:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:06.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:06.271393+0000) 2022-09-10T10:25:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:06.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:06.794413+0000) 2022-09-10T10:25:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:07.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:07.271646+0000) 2022-09-10T10:25:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:07 smithi104 ceph-mon[97044]: pgmap v11387: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:25:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:07.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:07.794613+0000) 2022-09-10T10:25:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:07 smithi110 ceph-mon[97633]: pgmap v11387: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:25:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:08.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:08.271897+0000) 2022-09-10T10:25:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:08 smithi104 ceph-mon[97044]: osdmap e792: 12 total, 12 up, 12 in 2022-09-10T10:25:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:08.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:08.794819+0000) 2022-09-10T10:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:08 smithi110 ceph-mon[97633]: osdmap e792: 12 total, 12 up, 12 in 2022-09-10T10:25:09.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:25:09.260+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:25:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:09.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:09.272168+0000) 2022-09-10T10:25:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:09 smithi104 ceph-mon[97044]: pgmap v11389: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:25:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:09.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:09.795062+0000) 2022-09-10T10:25:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:09 smithi110 ceph-mon[97633]: pgmap v11389: 129 pgs: 129 active+clean; 8.9 MiB data, 1007 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:25:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:10.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:10.272388+0000) 2022-09-10T10:25:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:10.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:10.795393+0000) 2022-09-10T10:25:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:11.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:11.272548+0000) 2022-09-10T10:25:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:11 smithi104 ceph-mon[97044]: pgmap v11390: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:25:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:11.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:11.795637+0000) 2022-09-10T10:25:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:11 smithi110 ceph-mon[97633]: pgmap v11390: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:25:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:12.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:12.272793+0000) 2022-09-10T10:25:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:12.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:12.795911+0000) 2022-09-10T10:25:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:13.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:13.273031+0000) 2022-09-10T10:25:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:13 smithi104 ceph-mon[97044]: pgmap v11391: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:25:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:13 smithi104 ceph-mon[97044]: osdmap e793: 12 total, 12 up, 12 in 2022-09-10T10:25:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:13.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:13.796211+0000) 2022-09-10T10:25:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:13 smithi110 ceph-mon[97633]: pgmap v11391: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:25:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:13 smithi110 ceph-mon[97633]: osdmap e793: 12 total, 12 up, 12 in 2022-09-10T10:25:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:14.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:14.273275+0000) 2022-09-10T10:25:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:14.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:14.796475+0000) 2022-09-10T10:25:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:15.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:15.273479+0000) 2022-09-10T10:25:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:15 smithi104 ceph-mon[97044]: pgmap v11393: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:25:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:15.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:15.796748+0000) 2022-09-10T10:25:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:15 smithi110 ceph-mon[97633]: pgmap v11393: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:25:16.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:16.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:16.273710+0000) 2022-09-10T10:25:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:16.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:16.797029+0000) 2022-09-10T10:25:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:17.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:17.273952+0000) 2022-09-10T10:25:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:17 smithi104 ceph-mon[97044]: pgmap v11394: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:25:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:17.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:17.797240+0000) 2022-09-10T10:25:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:17 smithi110 ceph-mon[97633]: pgmap v11394: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:25:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:18.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:18.274205+0000) 2022-09-10T10:25:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:18.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:18.797485+0000) 2022-09-10T10:25:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:19.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:19.274390+0000) 2022-09-10T10:25:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:19 smithi104 ceph-mon[97044]: pgmap v11395: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:25:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:19.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:19.797778+0000) 2022-09-10T10:25:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:19 smithi110 ceph-mon[97633]: pgmap v11395: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:25:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:20.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:20.274556+0000) 2022-09-10T10:25:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:20.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:20.798001+0000) 2022-09-10T10:25:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:21.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:21.274788+0000) 2022-09-10T10:25:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:21 smithi104 ceph-mon[97044]: pgmap v11396: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:25:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:21.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:21.798239+0000) 2022-09-10T10:25:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:21 smithi110 ceph-mon[97633]: pgmap v11396: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:25:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:22.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:22.275014+0000) 2022-09-10T10:25:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:22.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:22.798405+0000) 2022-09-10T10:25:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:23.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:23.275250+0000) 2022-09-10T10:25:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:23 smithi104 ceph-mon[97044]: pgmap v11397: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:25:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:23.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:23.798611+0000) 2022-09-10T10:25:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:23 smithi110 ceph-mon[97633]: pgmap v11397: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:25:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:24.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:24.275471+0000) 2022-09-10T10:25:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:25:24.275+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:25:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:24.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:24.798879+0000) 2022-09-10T10:25:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:25.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:25.275610+0000) 2022-09-10T10:25:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:25 smithi104 ceph-mon[97044]: pgmap v11398: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:25:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:25.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:25.799086+0000) 2022-09-10T10:25:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:25 smithi110 ceph-mon[97633]: pgmap v11398: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:25:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:26.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:26.275862+0000) 2022-09-10T10:25:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:26.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:26.799329+0000) 2022-09-10T10:25:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:27.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:27.276022+0000) 2022-09-10T10:25:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:27 smithi104 ceph-mon[97044]: pgmap v11399: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:27.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:27.799572+0000) 2022-09-10T10:25:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:27 smithi110 ceph-mon[97633]: pgmap v11399: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:28.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:28.276215+0000) 2022-09-10T10:25:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:28.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:28.799869+0000) 2022-09-10T10:25:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:29.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:29.276434+0000) 2022-09-10T10:25:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:29 smithi104 ceph-mon[97044]: pgmap v11400: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:29.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:29.800199+0000) 2022-09-10T10:25:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:29 smithi110 ceph-mon[97633]: pgmap v11400: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:30.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:30.276616+0000) 2022-09-10T10:25:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:30.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:30.800413+0000) 2022-09-10T10:25:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:31.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:31.276820+0000) 2022-09-10T10:25:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:31 smithi104 ceph-mon[97044]: pgmap v11401: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:31.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:31.800661+0000) 2022-09-10T10:25:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:31 smithi110 ceph-mon[97633]: pgmap v11401: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:32.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:32.277031+0000) 2022-09-10T10:25:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:32.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:32.800865+0000) 2022-09-10T10:25:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:33.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:33.277265+0000) 2022-09-10T10:25:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:33 smithi104 ceph-mon[97044]: pgmap v11402: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:33.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:33.801074+0000) 2022-09-10T10:25:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:33 smithi110 ceph-mon[97633]: pgmap v11402: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:34.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:34.277496+0000) 2022-09-10T10:25:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:34.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:34.801351+0000) 2022-09-10T10:25:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:35.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:35.277764+0000) 2022-09-10T10:25:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:35 smithi104 ceph-mon[97044]: pgmap v11403: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:35.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:35.801488+0000) 2022-09-10T10:25:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:35 smithi110 ceph-mon[97633]: pgmap v11403: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:36.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:36.278009+0000) 2022-09-10T10:25:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:36.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:36.801711+0000) 2022-09-10T10:25:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:37.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:37.278197+0000) 2022-09-10T10:25:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:37 smithi104 ceph-mon[97044]: pgmap v11404: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:37.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:37.801901+0000) 2022-09-10T10:25:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:37 smithi110 ceph-mon[97633]: pgmap v11404: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:38.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:38.278443+0000) 2022-09-10T10:25:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:38.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:38.802140+0000) 2022-09-10T10:25:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:39.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:39.278695+0000) 2022-09-10T10:25:39.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:25:39.290+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:25:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:39 smithi104 ceph-mon[97044]: pgmap v11405: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:39.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:39.802419+0000) 2022-09-10T10:25:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:39 smithi110 ceph-mon[97633]: pgmap v11405: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:40.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:40.278915+0000) 2022-09-10T10:25:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:40.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:40.802585+0000) 2022-09-10T10:25:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:41.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:41.279177+0000) 2022-09-10T10:25:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:41 smithi104 ceph-mon[97044]: pgmap v11406: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:41.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:41.802845+0000) 2022-09-10T10:25:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:41 smithi110 ceph-mon[97633]: pgmap v11406: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:42.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:42.279408+0000) 2022-09-10T10:25:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:42.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:42.803165+0000) 2022-09-10T10:25:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:43.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:43.279600+0000) 2022-09-10T10:25:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:43 smithi104 ceph-mon[97044]: pgmap v11407: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:43 smithi110 ceph-mon[97633]: pgmap v11407: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:44.280 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:43.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:43.803401+0000) 2022-09-10T10:25:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:44.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:44.279854+0000) 2022-09-10T10:25:45.281 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:44.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:44.803687+0000) 2022-09-10T10:25:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:45.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:45.280131+0000) 2022-09-10T10:25:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:45 smithi104 ceph-mon[97044]: pgmap v11408: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:25:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:25:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:25:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:25:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:45 smithi110 ceph-mon[97633]: pgmap v11408: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:25:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:25:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:25:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:25:46.281 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:45.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:45.803943+0000) 2022-09-10T10:25:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:46.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:46.280385+0000) 2022-09-10T10:25:47.281 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:46.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:46.804197+0000) 2022-09-10T10:25:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:47.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:47.280618+0000) 2022-09-10T10:25:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:47 smithi104 ceph-mon[97044]: pgmap v11409: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:47 smithi110 ceph-mon[97633]: pgmap v11409: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:48.281 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:47.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:47.804399+0000) 2022-09-10T10:25:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:48.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:48.280852+0000) 2022-09-10T10:25:49.282 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:48.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:48.804669+0000) 2022-09-10T10:25:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:49.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:49.281041+0000) 2022-09-10T10:25:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:49 smithi104 ceph-mon[97044]: pgmap v11410: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:49 smithi110 ceph-mon[97633]: pgmap v11410: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:50.282 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:49.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:49.804834+0000) 2022-09-10T10:25:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:50.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:50.281268+0000) 2022-09-10T10:25:51.282 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:50.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:50.804966+0000) 2022-09-10T10:25:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:51.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:51.281526+0000) 2022-09-10T10:25:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:51 smithi104 ceph-mon[97044]: pgmap v11411: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:51 smithi110 ceph-mon[97633]: pgmap v11411: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:52.282 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:51.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:51.805150+0000) 2022-09-10T10:25:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:52.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:52.281732+0000) 2022-09-10T10:25:53.283 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:52.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:52.805360+0000) 2022-09-10T10:25:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:53.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:53.282013+0000) 2022-09-10T10:25:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:53 smithi104 ceph-mon[97044]: pgmap v11412: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:53 smithi110 ceph-mon[97633]: pgmap v11412: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:54.283 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:53.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:53.805587+0000) 2022-09-10T10:25:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:54.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:54.282253+0000) 2022-09-10T10:25:54.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:25:54.306+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:25:55.283 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:54.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:54.805794+0000) 2022-09-10T10:25:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:55.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:55.282453+0000) 2022-09-10T10:25:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:55 smithi104 ceph-mon[97044]: pgmap v11413: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:55 smithi110 ceph-mon[97633]: pgmap v11413: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:25:56.283 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:55.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:55.806004+0000) 2022-09-10T10:25:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:56.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:56.282652+0000) 2022-09-10T10:25:57.283 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:56.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:56.806268+0000) 2022-09-10T10:25:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:57.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:57.282898+0000) 2022-09-10T10:25:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:57 smithi104 ceph-mon[97044]: pgmap v11414: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:57 smithi110 ceph-mon[97633]: pgmap v11414: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:25:58.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:57.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:57.806562+0000) 2022-09-10T10:25:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:58.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:58.283173+0000) 2022-09-10T10:25:59.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:58.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:58.806741+0000) 2022-09-10T10:25:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:25:59.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:59.283412+0000) 2022-09-10T10:26:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:25:59 smithi104 ceph-mon[97044]: pgmap v11415: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:25:59 smithi110 ceph-mon[97633]: pgmap v11415: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:00.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:25:59.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:25:59.806950+0000) 2022-09-10T10:26:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:00.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:00.283635+0000) 2022-09-10T10:26:01.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:00.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:00.807184+0000) 2022-09-10T10:26:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:01.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:01.283820+0000) 2022-09-10T10:26:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:01 smithi104 ceph-mon[97044]: pgmap v11416: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:01 smithi110 ceph-mon[97633]: pgmap v11416: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:02.285 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:01.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:01.807398+0000) 2022-09-10T10:26:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:02.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:02.284001+0000) 2022-09-10T10:26:03.285 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:02.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:02.807588+0000) 2022-09-10T10:26:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:03.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:03.284244+0000) 2022-09-10T10:26:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:03 smithi104 ceph-mon[97044]: pgmap v11417: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:03 smithi110 ceph-mon[97633]: pgmap v11417: 129 pgs: 129 active+clean; 8.9 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:04.285 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:03.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:03.807805+0000) 2022-09-10T10:26:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:04.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:04.284494+0000) 2022-09-10T10:26:05.285 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:04.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:04.808080+0000) 2022-09-10T10:26:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:05.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:05.284732+0000) 2022-09-10T10:26:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:05 smithi104 ceph-mon[97044]: pgmap v11418: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:26:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:05 smithi110 ceph-mon[97633]: pgmap v11418: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:26:06.286 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:05.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:05.808342+0000) 2022-09-10T10:26:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:06.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:06.284983+0000) 2022-09-10T10:26:07.286 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:06.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:06.808567+0000) 2022-09-10T10:26:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:07.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:07.285228+0000) 2022-09-10T10:26:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:07 smithi104 ceph-mon[97044]: pgmap v11419: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:26:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:07 smithi110 ceph-mon[97633]: pgmap v11419: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:26:08.286 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:07.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:07.808866+0000) 2022-09-10T10:26:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:08.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:08.285459+0000) 2022-09-10T10:26:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:08 smithi104 ceph-mon[97044]: osdmap e794: 12 total, 12 up, 12 in 2022-09-10T10:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:08 smithi110 ceph-mon[97633]: osdmap e794: 12 total, 12 up, 12 in 2022-09-10T10:26:09.286 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:08.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:08.809141+0000) 2022-09-10T10:26:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:09.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:09.285695+0000) 2022-09-10T10:26:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:26:09.316+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:26:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:09 smithi104 ceph-mon[97044]: pgmap v11421: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:26:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:09 smithi110 ceph-mon[97633]: pgmap v11421: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:26:10.286 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:09.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:09.809356+0000) 2022-09-10T10:26:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:10.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:10.285921+0000) 2022-09-10T10:26:11.287 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:10.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:10.809561+0000) 2022-09-10T10:26:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:11.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:11.286195+0000) 2022-09-10T10:26:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:11 smithi104 ceph-mon[97044]: pgmap v11422: 129 pgs: 129 active+clean; 9.0 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:26:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:11 smithi110 ceph-mon[97633]: pgmap v11422: 129 pgs: 129 active+clean; 9.0 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:26:12.287 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:11.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:11.809767+0000) 2022-09-10T10:26:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:12.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:12.286441+0000) 2022-09-10T10:26:13.287 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:12.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:12.810025+0000) 2022-09-10T10:26:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:13.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:13.286741+0000) 2022-09-10T10:26:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:13 smithi104 ceph-mon[97044]: pgmap v11423: 129 pgs: 129 active+clean; 9.0 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:26:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:13 smithi104 ceph-mon[97044]: osdmap e795: 12 total, 12 up, 12 in 2022-09-10T10:26:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:13 smithi110 ceph-mon[97633]: pgmap v11423: 129 pgs: 129 active+clean; 9.0 MiB data, 998 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:26:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:13 smithi110 ceph-mon[97633]: osdmap e795: 12 total, 12 up, 12 in 2022-09-10T10:26:14.287 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:13.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:13.810233+0000) 2022-09-10T10:26:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:14.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:14.286948+0000) 2022-09-10T10:26:15.288 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:14.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:14.810413+0000) 2022-09-10T10:26:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:15.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:15.287203+0000) 2022-09-10T10:26:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:15 smithi104 ceph-mon[97044]: pgmap v11425: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:26:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:15 smithi110 ceph-mon[97633]: pgmap v11425: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:26:16.288 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:15.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:15.810650+0000) 2022-09-10T10:26:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:16.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:16.287346+0000) 2022-09-10T10:26:17.288 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:16.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:16.810890+0000) 2022-09-10T10:26:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:17.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:17.287599+0000) 2022-09-10T10:26:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:17 smithi104 ceph-mon[97044]: pgmap v11426: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:26:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:17 smithi110 ceph-mon[97633]: pgmap v11426: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:26:18.288 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:17.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:17.811043+0000) 2022-09-10T10:26:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:18.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:18.287843+0000) 2022-09-10T10:26:19.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:18.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:18.811255+0000) 2022-09-10T10:26:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:19.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:19.288132+0000) 2022-09-10T10:26:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:19 smithi104 ceph-mon[97044]: pgmap v11427: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:26:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:19 smithi110 ceph-mon[97633]: pgmap v11427: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:26:20.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:19.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:19.811504+0000) 2022-09-10T10:26:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:20.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:20.288332+0000) 2022-09-10T10:26:21.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:20.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:20.811662+0000) 2022-09-10T10:26:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:21.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:21.288582+0000) 2022-09-10T10:26:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:21 smithi104 ceph-mon[97044]: pgmap v11428: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:26:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:21 smithi110 ceph-mon[97633]: pgmap v11428: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:26:22.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:21.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:21.811894+0000) 2022-09-10T10:26:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:22.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:22.288821+0000) 2022-09-10T10:26:23.290 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:22.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:22.812203+0000) 2022-09-10T10:26:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:23.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:23.289081+0000) 2022-09-10T10:26:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:23 smithi104 ceph-mon[97044]: pgmap v11429: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:26:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:23 smithi110 ceph-mon[97633]: pgmap v11429: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:26:24.290 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:23.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:23.812398+0000) 2022-09-10T10:26:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:24.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:24.289355+0000) 2022-09-10T10:26:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:26:24.332+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:26:25.290 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:24.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:24.812605+0000) 2022-09-10T10:26:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:25.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:25.289562+0000) 2022-09-10T10:26:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:25 smithi104 ceph-mon[97044]: pgmap v11430: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:26:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:25 smithi110 ceph-mon[97633]: pgmap v11430: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:26:26.290 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:25.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:25.812822+0000) 2022-09-10T10:26:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:26.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:26.289807+0000) 2022-09-10T10:26:27.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:26.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:26.812999+0000) 2022-09-10T10:26:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:27.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:27.289983+0000) 2022-09-10T10:26:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:27 smithi104 ceph-mon[97044]: pgmap v11431: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:27 smithi110 ceph-mon[97633]: pgmap v11431: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:28.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:27.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:27.813203+0000) 2022-09-10T10:26:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:28.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:28.290165+0000) 2022-09-10T10:26:29.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:28.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:28.813336+0000) 2022-09-10T10:26:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:29.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:29.290331+0000) 2022-09-10T10:26:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:29 smithi104 ceph-mon[97044]: pgmap v11432: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:29 smithi110 ceph-mon[97633]: pgmap v11432: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:30.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:29.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:29.813566+0000) 2022-09-10T10:26:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:30.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:30.290563+0000) 2022-09-10T10:26:31.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:30.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:30.813761+0000) 2022-09-10T10:26:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:31.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:31.290745+0000) 2022-09-10T10:26:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:31 smithi104 ceph-mon[97044]: pgmap v11433: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:31 smithi110 ceph-mon[97633]: pgmap v11433: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:32.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:31.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:31.813938+0000) 2022-09-10T10:26:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:32.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:32.290886+0000) 2022-09-10T10:26:33.292 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:32.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:32.814082+0000) 2022-09-10T10:26:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:33.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:33.291157+0000) 2022-09-10T10:26:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:33 smithi104 ceph-mon[97044]: pgmap v11434: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:33 smithi110 ceph-mon[97633]: pgmap v11434: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:34.292 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:33.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:33.814354+0000) 2022-09-10T10:26:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:34.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:34.291341+0000) 2022-09-10T10:26:35.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:34.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:34.814575+0000) 2022-09-10T10:26:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:35.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:35.291634+0000) 2022-09-10T10:26:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:35 smithi104 ceph-mon[97044]: pgmap v11435: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:35 smithi110 ceph-mon[97633]: pgmap v11435: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:36.292 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:35.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:35.814807+0000) 2022-09-10T10:26:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:36.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:36.291819+0000) 2022-09-10T10:26:37.293 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:36.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:36.815035+0000) 2022-09-10T10:26:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:37.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:37.292038+0000) 2022-09-10T10:26:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:37 smithi104 ceph-mon[97044]: pgmap v11436: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:37 smithi110 ceph-mon[97633]: pgmap v11436: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:38.293 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:37.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:37.815236+0000) 2022-09-10T10:26:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:38.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:38.292326+0000) 2022-09-10T10:26:39.293 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:38.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:38.815449+0000) 2022-09-10T10:26:39.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:26:39.339+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:26:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:39.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:39.292579+0000) 2022-09-10T10:26:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:39 smithi104 ceph-mon[97044]: pgmap v11437: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:39 smithi110 ceph-mon[97633]: pgmap v11437: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:40.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:39.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:39.815665+0000) 2022-09-10T10:26:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:40.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:40.292836+0000) 2022-09-10T10:26:41.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:40.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:40.815883+0000) 2022-09-10T10:26:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:41.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:41.293061+0000) 2022-09-10T10:26:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:41 smithi104 ceph-mon[97044]: pgmap v11438: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:41 smithi110 ceph-mon[97633]: pgmap v11438: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:42.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:41.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:41.816126+0000) 2022-09-10T10:26:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:42.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:42.293299+0000) 2022-09-10T10:26:43.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:42.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:42.816328+0000) 2022-09-10T10:26:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:43.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:43.293526+0000) 2022-09-10T10:26:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:43 smithi104 ceph-mon[97044]: pgmap v11439: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:43 smithi110 ceph-mon[97633]: pgmap v11439: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:44.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:43.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:43.816547+0000) 2022-09-10T10:26:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:44.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:44.293733+0000) 2022-09-10T10:26:45.294 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:44.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:44.816760+0000) 2022-09-10T10:26:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:45.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:45.293872+0000) 2022-09-10T10:26:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:45 smithi104 ceph-mon[97044]: pgmap v11440: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:26:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:26:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:26:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:26:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:45 smithi110 ceph-mon[97633]: pgmap v11440: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:26:46.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:45.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:45.816990+0000) 2022-09-10T10:26:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:46.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:46.294058+0000) 2022-09-10T10:26:47.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:46.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:46.817206+0000) 2022-09-10T10:26:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:47.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:47.294341+0000) 2022-09-10T10:26:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:47 smithi104 ceph-mon[97044]: pgmap v11441: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:47 smithi110 ceph-mon[97633]: pgmap v11441: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:48.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:47.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:47.817401+0000) 2022-09-10T10:26:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:48.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:48.294570+0000) 2022-09-10T10:26:49.295 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:48.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:48.817634+0000) 2022-09-10T10:26:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:49.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:49.294820+0000) 2022-09-10T10:26:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:49 smithi104 ceph-mon[97044]: pgmap v11442: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:49 smithi110 ceph-mon[97633]: pgmap v11442: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:50.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:49.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:49.817832+0000) 2022-09-10T10:26:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:50.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:50.295064+0000) 2022-09-10T10:26:51.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:50.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:50.817965+0000) 2022-09-10T10:26:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:51.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:51.295340+0000) 2022-09-10T10:26:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:51 smithi104 ceph-mon[97044]: pgmap v11443: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:51 smithi110 ceph-mon[97633]: pgmap v11443: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:52.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:51.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:51.818208+0000) 2022-09-10T10:26:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:52.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:52.295503+0000) 2022-09-10T10:26:53.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:52.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:52.818412+0000) 2022-09-10T10:26:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:53.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:53.295730+0000) 2022-09-10T10:26:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:53 smithi104 ceph-mon[97044]: pgmap v11444: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:53 smithi110 ceph-mon[97633]: pgmap v11444: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:54.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:53.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:53.818610+0000) 2022-09-10T10:26:54.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:26:54.355+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:26:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:54.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:54.295961+0000) 2022-09-10T10:26:55.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:54.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:54.818892+0000) 2022-09-10T10:26:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:55.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:55.296209+0000) 2022-09-10T10:26:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:55 smithi104 ceph-mon[97044]: pgmap v11445: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:55 smithi110 ceph-mon[97633]: pgmap v11445: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:26:56.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:55.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:55.819126+0000) 2022-09-10T10:26:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:56.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:56.296440+0000) 2022-09-10T10:26:57.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:56.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:56.819303+0000) 2022-09-10T10:26:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:57.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:57.296687+0000) 2022-09-10T10:26:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:57 smithi104 ceph-mon[97044]: pgmap v11446: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:57 smithi110 ceph-mon[97633]: pgmap v11446: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:26:58.297 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:57.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:57.819524+0000) 2022-09-10T10:26:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:58.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:58.296934+0000) 2022-09-10T10:26:59.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:58.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:58.819751+0000) 2022-09-10T10:26:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:26:59.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:59.297215+0000) 2022-09-10T10:27:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:26:59 smithi104 ceph-mon[97044]: pgmap v11447: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:26:59 smithi110 ceph-mon[97633]: pgmap v11447: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:00.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:26:59.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:26:59.820013+0000) 2022-09-10T10:27:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:00.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:00.297471+0000) 2022-09-10T10:27:01.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:00.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:00.820253+0000) 2022-09-10T10:27:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:01.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:01.297716+0000) 2022-09-10T10:27:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:01 smithi104 ceph-mon[97044]: pgmap v11448: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:01 smithi110 ceph-mon[97633]: pgmap v11448: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:02.298 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:01.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:01.820507+0000) 2022-09-10T10:27:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:02.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:02.297937+0000) 2022-09-10T10:27:03.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:02.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:02.820729+0000) 2022-09-10T10:27:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:03.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:03.298171+0000) 2022-09-10T10:27:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:03 smithi104 ceph-mon[97044]: pgmap v11449: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:03 smithi110 ceph-mon[97633]: pgmap v11449: 129 pgs: 129 active+clean; 9.0 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:04.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:03.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:03.820988+0000) 2022-09-10T10:27:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:04.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:04.298355+0000) 2022-09-10T10:27:05.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:04.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:04.821171+0000) 2022-09-10T10:27:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:05.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:05.298564+0000) 2022-09-10T10:27:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:05 smithi104 ceph-mon[97044]: pgmap v11450: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:27:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:05 smithi110 ceph-mon[97633]: pgmap v11450: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:27:06.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:05.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:05.821414+0000) 2022-09-10T10:27:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:06.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:06.298782+0000) 2022-09-10T10:27:07.299 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:06.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:06.821668+0000) 2022-09-10T10:27:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:07.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:07.298982+0000) 2022-09-10T10:27:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:07 smithi104 ceph-mon[97044]: pgmap v11451: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:27:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:07 smithi110 ceph-mon[97633]: pgmap v11451: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:27:08.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:07.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:07.821853+0000) 2022-09-10T10:27:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:08.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:08.299258+0000) 2022-09-10T10:27:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:08 smithi104 ceph-mon[97044]: osdmap e796: 12 total, 12 up, 12 in 2022-09-10T10:27:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:08 smithi110 ceph-mon[97633]: osdmap e796: 12 total, 12 up, 12 in 2022-09-10T10:27:09.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:08.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:08.822065+0000) 2022-09-10T10:27:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:09.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:09.299455+0000) 2022-09-10T10:27:09.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:27:09.366+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:27:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:09 smithi104 ceph-mon[97044]: pgmap v11453: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:27:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:09 smithi110 ceph-mon[97633]: pgmap v11453: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:27:10.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:09.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:09.822303+0000) 2022-09-10T10:27:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:10.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:10.299664+0000) 2022-09-10T10:27:11.300 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:10.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:10.822448+0000) 2022-09-10T10:27:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:11.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:11.299886+0000) 2022-09-10T10:27:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:11 smithi104 ceph-mon[97044]: pgmap v11454: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:27:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:11 smithi110 ceph-mon[97633]: pgmap v11454: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:27:12.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:11.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:11.822671+0000) 2022-09-10T10:27:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:12.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:12.300171+0000) 2022-09-10T10:27:13.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:12.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:12.822837+0000) 2022-09-10T10:27:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:13.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:13.300381+0000) 2022-09-10T10:27:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:13 smithi104 ceph-mon[97044]: pgmap v11455: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:27:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:13 smithi104 ceph-mon[97044]: osdmap e797: 12 total, 12 up, 12 in 2022-09-10T10:27:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:13 smithi110 ceph-mon[97633]: pgmap v11455: 129 pgs: 129 active+clean; 9.0 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:27:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:13 smithi110 ceph-mon[97633]: osdmap e797: 12 total, 12 up, 12 in 2022-09-10T10:27:14.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:13.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:13.823002+0000) 2022-09-10T10:27:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:14.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:14.300543+0000) 2022-09-10T10:27:15.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:14.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:14.823202+0000) 2022-09-10T10:27:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:15.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:15.300779+0000) 2022-09-10T10:27:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:15 smithi104 ceph-mon[97044]: pgmap v11457: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:27:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:15 smithi110 ceph-mon[97633]: pgmap v11457: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:27:16.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:15.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:15.823455+0000) 2022-09-10T10:27:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:16.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:16.300975+0000) 2022-09-10T10:27:17.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:16.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:16.823692+0000) 2022-09-10T10:27:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:17.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:17.301189+0000) 2022-09-10T10:27:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:17 smithi104 ceph-mon[97044]: pgmap v11458: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:27:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:17 smithi110 ceph-mon[97633]: pgmap v11458: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:27:18.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:17.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:17.823957+0000) 2022-09-10T10:27:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:18.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:18.301433+0000) 2022-09-10T10:27:19.302 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:18.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:18.824199+0000) 2022-09-10T10:27:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:19.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:19.301650+0000) 2022-09-10T10:27:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:19 smithi104 ceph-mon[97044]: pgmap v11459: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:27:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:19 smithi110 ceph-mon[97633]: pgmap v11459: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:27:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:19.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:19.824409+0000) 2022-09-10T10:27:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:20.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:20.301882+0000) 2022-09-10T10:27:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:20.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:20.824607+0000) 2022-09-10T10:27:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:21.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:21.302150+0000) 2022-09-10T10:27:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:21 smithi104 ceph-mon[97044]: pgmap v11460: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:27:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:21 smithi110 ceph-mon[97633]: pgmap v11460: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:27:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:21.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:21.824825+0000) 2022-09-10T10:27:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:22.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:22.302359+0000) 2022-09-10T10:27:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:22.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:22.825047+0000) 2022-09-10T10:27:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:23.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:23.302576+0000) 2022-09-10T10:27:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:23 smithi104 ceph-mon[97044]: pgmap v11461: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:27:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:23 smithi110 ceph-mon[97633]: pgmap v11461: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:27:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:23.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:23.825206+0000) 2022-09-10T10:27:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:24.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:24.302780+0000) 2022-09-10T10:27:24.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:27:24.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:27:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:24.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:24.825447+0000) 2022-09-10T10:27:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:25.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:25.303002+0000) 2022-09-10T10:27:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:25 smithi104 ceph-mon[97044]: pgmap v11462: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:27:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:25 smithi110 ceph-mon[97633]: pgmap v11462: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:27:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:25.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:25.825677+0000) 2022-09-10T10:27:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:26.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:26.303216+0000) 2022-09-10T10:27:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:26.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:26.825949+0000) 2022-09-10T10:27:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:27.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:27.303414+0000) 2022-09-10T10:27:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:27 smithi104 ceph-mon[97044]: pgmap v11463: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:27 smithi110 ceph-mon[97633]: pgmap v11463: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:27.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:27.826182+0000) 2022-09-10T10:27:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:28.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:28.303601+0000) 2022-09-10T10:27:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:28.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:28.826410+0000) 2022-09-10T10:27:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:29.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:29.303932+0000) 2022-09-10T10:27:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:29 smithi104 ceph-mon[97044]: pgmap v11464: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:29 smithi110 ceph-mon[97633]: pgmap v11464: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:29.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:29.826654+0000) 2022-09-10T10:27:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:30.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:30.304064+0000) 2022-09-10T10:27:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:30.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:30.826881+0000) 2022-09-10T10:27:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:31.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:31.304162+0000) 2022-09-10T10:27:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:31 smithi104 ceph-mon[97044]: pgmap v11465: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:31 smithi110 ceph-mon[97633]: pgmap v11465: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:31.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:31.827120+0000) 2022-09-10T10:27:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:32.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:32.304285+0000) 2022-09-10T10:27:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:32.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:32.827287+0000) 2022-09-10T10:27:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:33.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:33.304401+0000) 2022-09-10T10:27:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:33 smithi104 ceph-mon[97044]: pgmap v11466: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:33 smithi110 ceph-mon[97633]: pgmap v11466: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:33.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:33.827496+0000) 2022-09-10T10:27:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:34.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:34.304527+0000) 2022-09-10T10:27:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:34.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:34.827756+0000) 2022-09-10T10:27:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:35.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:35.304686+0000) 2022-09-10T10:27:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:35 smithi104 ceph-mon[97044]: pgmap v11467: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:35 smithi110 ceph-mon[97633]: pgmap v11467: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:35.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:35.827971+0000) 2022-09-10T10:27:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:36.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:36.304932+0000) 2022-09-10T10:27:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:36.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:36.828200+0000) 2022-09-10T10:27:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:37.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:37.305163+0000) 2022-09-10T10:27:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:37 smithi104 ceph-mon[97044]: pgmap v11468: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:37 smithi110 ceph-mon[97633]: pgmap v11468: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:37.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:37.828445+0000) 2022-09-10T10:27:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:38.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:38.305338+0000) 2022-09-10T10:27:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:38.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:38.828708+0000) 2022-09-10T10:27:39.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:39.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:39.305522+0000) 2022-09-10T10:27:39.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:27:39.396+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:27:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:39 smithi104 ceph-mon[97044]: pgmap v11469: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:39 smithi110 ceph-mon[97633]: pgmap v11469: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:39.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:39.828924+0000) 2022-09-10T10:27:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:40.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:40.305713+0000) 2022-09-10T10:27:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:40.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:40.829177+0000) 2022-09-10T10:27:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:41.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:41.305925+0000) 2022-09-10T10:27:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:41 smithi104 ceph-mon[97044]: pgmap v11470: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:41 smithi110 ceph-mon[97633]: pgmap v11470: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:41.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:41.829407+0000) 2022-09-10T10:27:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:42.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:42.306165+0000) 2022-09-10T10:27:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:42.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:42.829663+0000) 2022-09-10T10:27:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:43.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:43.306322+0000) 2022-09-10T10:27:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:43 smithi104 ceph-mon[97044]: pgmap v11471: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:43 smithi110 ceph-mon[97633]: pgmap v11471: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:43.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:43.829865+0000) 2022-09-10T10:27:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:44.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:44.306500+0000) 2022-09-10T10:27:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:44.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:44.830076+0000) 2022-09-10T10:27:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:45.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:45.306686+0000) 2022-09-10T10:27:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:45 smithi104 ceph-mon[97044]: pgmap v11472: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:27:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:45 smithi110 ceph-mon[97633]: pgmap v11472: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:27:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:27:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:27:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:27:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:45.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:45.830355+0000) 2022-09-10T10:27:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:46.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:46.306869+0000) 2022-09-10T10:27:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:46.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:46.830607+0000) 2022-09-10T10:27:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:47.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:47.307043+0000) 2022-09-10T10:27:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:47 smithi104 ceph-mon[97044]: pgmap v11473: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:47 smithi110 ceph-mon[97633]: pgmap v11473: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:47.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:47.830847+0000) 2022-09-10T10:27:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:48.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:48.307158+0000) 2022-09-10T10:27:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:48.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:48.831139+0000) 2022-09-10T10:27:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:49.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:49.307330+0000) 2022-09-10T10:27:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:49 smithi104 ceph-mon[97044]: pgmap v11474: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:49 smithi110 ceph-mon[97633]: pgmap v11474: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:49.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:49.831409+0000) 2022-09-10T10:27:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:50.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:50.307529+0000) 2022-09-10T10:27:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:50.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:50.831591+0000) 2022-09-10T10:27:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:51.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:51.307673+0000) 2022-09-10T10:27:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:51 smithi104 ceph-mon[97044]: pgmap v11475: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:51 smithi110 ceph-mon[97633]: pgmap v11475: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:51.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:51.831859+0000) 2022-09-10T10:27:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:52.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:52.307895+0000) 2022-09-10T10:27:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:52.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:52.832131+0000) 2022-09-10T10:27:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:53.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:53.308165+0000) 2022-09-10T10:27:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:53 smithi104 ceph-mon[97044]: pgmap v11476: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:53 smithi110 ceph-mon[97633]: pgmap v11476: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:53.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:53.832382+0000) 2022-09-10T10:27:54.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:27:54.411+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:27:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:54.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:54.308342+0000) 2022-09-10T10:27:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:54.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:54.832545+0000) 2022-09-10T10:27:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:55.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:55.308487+0000) 2022-09-10T10:27:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:55 smithi104 ceph-mon[97044]: pgmap v11477: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:55 smithi110 ceph-mon[97633]: pgmap v11477: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:27:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:55.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:55.832751+0000) 2022-09-10T10:27:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:56.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:56.308652+0000) 2022-09-10T10:27:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:56.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:56.832989+0000) 2022-09-10T10:27:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:57.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:57.308791+0000) 2022-09-10T10:27:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:57 smithi104 ceph-mon[97044]: pgmap v11478: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:57 smithi110 ceph-mon[97633]: pgmap v11478: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:27:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:57.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:57.833265+0000) 2022-09-10T10:27:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:58.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:58.308997+0000) 2022-09-10T10:27:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:58.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:58.833463+0000) 2022-09-10T10:27:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:27:59.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:59.309223+0000) 2022-09-10T10:28:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:27:59 smithi104 ceph-mon[97044]: pgmap v11479: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:27:59 smithi110 ceph-mon[97633]: pgmap v11479: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:27:59.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:27:59.833707+0000) 2022-09-10T10:28:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:00.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:00.309357+0000) 2022-09-10T10:28:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:00.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:00.833970+0000) 2022-09-10T10:28:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:01.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:01.309545+0000) 2022-09-10T10:28:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:01 smithi104 ceph-mon[97044]: pgmap v11480: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:01 smithi110 ceph-mon[97633]: pgmap v11480: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:01.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:01.834197+0000) 2022-09-10T10:28:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:02.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:02.309703+0000) 2022-09-10T10:28:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:02.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:02.834410+0000) 2022-09-10T10:28:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:03.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:03.309876+0000) 2022-09-10T10:28:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:03 smithi104 ceph-mon[97044]: pgmap v11481: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:03 smithi110 ceph-mon[97633]: pgmap v11481: 129 pgs: 129 active+clean; 9.0 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:03.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:03.834639+0000) 2022-09-10T10:28:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:04.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:04.310110+0000) 2022-09-10T10:28:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:04.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:04.834804+0000) 2022-09-10T10:28:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:05.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:05.310261+0000) 2022-09-10T10:28:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:05 smithi104 ceph-mon[97044]: pgmap v11482: 129 pgs: 129 active+clean; 9.0 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:28:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:05 smithi110 ceph-mon[97633]: pgmap v11482: 129 pgs: 129 active+clean; 9.0 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:28:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:05.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:05.834965+0000) 2022-09-10T10:28:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:06.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:06.310423+0000) 2022-09-10T10:28:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:06.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:06.835154+0000) 2022-09-10T10:28:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:07.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:07.310566+0000) 2022-09-10T10:28:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:07 smithi104 ceph-mon[97044]: pgmap v11483: 129 pgs: 129 active+clean; 9.0 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:28:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:07 smithi110 ceph-mon[97633]: pgmap v11483: 129 pgs: 129 active+clean; 9.0 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:28:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:07.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:07.835351+0000) 2022-09-10T10:28:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:08.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:08.310791+0000) 2022-09-10T10:28:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:08 smithi104 ceph-mon[97044]: osdmap e798: 12 total, 12 up, 12 in 2022-09-10T10:28:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:08 smithi110 ceph-mon[97633]: osdmap e798: 12 total, 12 up, 12 in 2022-09-10T10:28:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:08.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:08.835539+0000) 2022-09-10T10:28:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:09.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:09.311018+0000) 2022-09-10T10:28:09.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:28:09.424+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:28:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:09 smithi104 ceph-mon[97044]: pgmap v11485: 129 pgs: 129 active+clean; 9.0 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:28:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:09 smithi110 ceph-mon[97633]: pgmap v11485: 129 pgs: 129 active+clean; 9.0 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:28:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:09.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:09.835754+0000) 2022-09-10T10:28:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:10.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:10.311201+0000) 2022-09-10T10:28:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:10.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:10.836002+0000) 2022-09-10T10:28:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:11.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:11.311410+0000) 2022-09-10T10:28:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:11 smithi104 ceph-mon[97044]: pgmap v11486: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:28:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:11 smithi110 ceph-mon[97633]: pgmap v11486: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:28:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:11.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:11.836263+0000) 2022-09-10T10:28:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:12.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:12.311539+0000) 2022-09-10T10:28:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:12.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:12.836457+0000) 2022-09-10T10:28:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:13.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:13.311688+0000) 2022-09-10T10:28:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:13 smithi104 ceph-mon[97044]: pgmap v11487: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:28:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:13 smithi104 ceph-mon[97044]: osdmap e799: 12 total, 12 up, 12 in 2022-09-10T10:28:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:13 smithi110 ceph-mon[97633]: pgmap v11487: 129 pgs: 129 active+clean; 9.0 MiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:28:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:13 smithi110 ceph-mon[97633]: osdmap e799: 12 total, 12 up, 12 in 2022-09-10T10:28:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:13.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:13.836661+0000) 2022-09-10T10:28:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:14.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:14.311847+0000) 2022-09-10T10:28:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:14.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:14.836843+0000) 2022-09-10T10:28:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:15.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:15.312002+0000) 2022-09-10T10:28:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:15 smithi104 ceph-mon[97044]: pgmap v11489: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:28:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:15 smithi110 ceph-mon[97633]: pgmap v11489: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:28:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:15.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:15.837006+0000) 2022-09-10T10:28:16.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:16.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:16.312153+0000) 2022-09-10T10:28:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:16.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:16.837204+0000) 2022-09-10T10:28:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:17.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:17.312318+0000) 2022-09-10T10:28:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:17 smithi104 ceph-mon[97044]: pgmap v11490: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:28:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:17 smithi110 ceph-mon[97633]: pgmap v11490: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:28:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:17.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:17.837419+0000) 2022-09-10T10:28:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:18.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:18.312486+0000) 2022-09-10T10:28:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:18.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:18.837603+0000) 2022-09-10T10:28:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:19.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:19.312634+0000) 2022-09-10T10:28:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:19 smithi104 ceph-mon[97044]: pgmap v11491: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:28:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:19 smithi110 ceph-mon[97633]: pgmap v11491: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:28:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:19.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:19.837761+0000) 2022-09-10T10:28:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:20.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:20.312832+0000) 2022-09-10T10:28:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:20.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:20.837994+0000) 2022-09-10T10:28:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:21.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:21.312980+0000) 2022-09-10T10:28:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:21 smithi104 ceph-mon[97044]: pgmap v11492: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:28:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:21 smithi110 ceph-mon[97633]: pgmap v11492: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:28:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:21.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:21.838271+0000) 2022-09-10T10:28:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:22.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:22.313177+0000) 2022-09-10T10:28:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:22.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:22.838464+0000) 2022-09-10T10:28:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:23.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:23.313382+0000) 2022-09-10T10:28:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:23 smithi104 ceph-mon[97044]: pgmap v11493: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:28:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:23 smithi110 ceph-mon[97633]: pgmap v11493: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:28:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:23.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:23.838678+0000) 2022-09-10T10:28:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:24.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:24.313611+0000) 2022-09-10T10:28:24.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:28:24.438+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:28:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:24.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:24.838956+0000) 2022-09-10T10:28:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:25.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:25.313788+0000) 2022-09-10T10:28:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:25 smithi104 ceph-mon[97044]: pgmap v11494: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:28:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:25 smithi110 ceph-mon[97633]: pgmap v11494: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:28:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:25.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:25.839210+0000) 2022-09-10T10:28:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:26.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:26.313986+0000) 2022-09-10T10:28:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:26.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:26.839432+0000) 2022-09-10T10:28:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:27.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:27.314222+0000) 2022-09-10T10:28:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:27 smithi104 ceph-mon[97044]: pgmap v11495: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:27 smithi110 ceph-mon[97633]: pgmap v11495: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:27.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:27.839631+0000) 2022-09-10T10:28:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:28.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:28.314466+0000) 2022-09-10T10:28:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:28.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:28.839816+0000) 2022-09-10T10:28:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:29.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:29.314606+0000) 2022-09-10T10:28:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:29 smithi104 ceph-mon[97044]: pgmap v11496: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:29 smithi110 ceph-mon[97633]: pgmap v11496: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:29.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:29.840027+0000) 2022-09-10T10:28:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:30.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:30.314767+0000) 2022-09-10T10:28:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:30.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:30.840241+0000) 2022-09-10T10:28:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:31.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:31.314962+0000) 2022-09-10T10:28:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:31 smithi104 ceph-mon[97044]: pgmap v11497: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:31 smithi110 ceph-mon[97633]: pgmap v11497: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:31.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:31.840501+0000) 2022-09-10T10:28:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:32.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:32.315192+0000) 2022-09-10T10:28:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:32.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:32.840733+0000) 2022-09-10T10:28:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:33.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:33.315400+0000) 2022-09-10T10:28:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:33 smithi104 ceph-mon[97044]: pgmap v11498: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:33 smithi110 ceph-mon[97633]: pgmap v11498: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:33.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:33.840960+0000) 2022-09-10T10:28:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:34.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:34.315586+0000) 2022-09-10T10:28:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:34.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:34.841205+0000) 2022-09-10T10:28:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:35.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:35.315817+0000) 2022-09-10T10:28:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:35 smithi104 ceph-mon[97044]: pgmap v11499: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:35 smithi110 ceph-mon[97633]: pgmap v11499: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:35.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:35.841402+0000) 2022-09-10T10:28:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:36.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:36.315969+0000) 2022-09-10T10:28:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:36.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:36.841659+0000) 2022-09-10T10:28:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:37.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:37.316180+0000) 2022-09-10T10:28:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:37 smithi104 ceph-mon[97044]: pgmap v11500: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:37 smithi110 ceph-mon[97633]: pgmap v11500: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:37.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:37.841808+0000) 2022-09-10T10:28:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:38.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:38.316393+0000) 2022-09-10T10:28:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:38.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:38.841979+0000) 2022-09-10T10:28:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:39.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:39.316614+0000) 2022-09-10T10:28:39.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:28:39.453+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:28:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:39 smithi104 ceph-mon[97044]: pgmap v11501: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:39 smithi110 ceph-mon[97633]: pgmap v11501: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:39.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:39.842197+0000) 2022-09-10T10:28:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:40.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:40.316843+0000) 2022-09-10T10:28:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:40.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:40.842424+0000) 2022-09-10T10:28:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:41.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:41.317045+0000) 2022-09-10T10:28:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:41 smithi104 ceph-mon[97044]: pgmap v11502: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:41 smithi110 ceph-mon[97633]: pgmap v11502: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:28:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:41.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:41.842652+0000) 2022-09-10T10:28:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:42.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:42.317200+0000) 2022-09-10T10:28:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:42.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:42.842884+0000) 2022-09-10T10:28:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:43.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:43.317390+0000) 2022-09-10T10:28:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:43 smithi104 ceph-mon[97044]: pgmap v11503: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:43 smithi110 ceph-mon[97633]: pgmap v11503: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:43.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:43.843151+0000) 2022-09-10T10:28:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:44.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:44.317569+0000) 2022-09-10T10:28:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:44.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:44.843276+0000) 2022-09-10T10:28:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:45.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:45.317786+0000) 2022-09-10T10:28:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:45 smithi104 ceph-mon[97044]: pgmap v11504: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T10:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:45 smithi110 ceph-mon[97633]: pgmap v11504: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T10:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:28:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:28:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:45.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:45.843459+0000) 2022-09-10T10:28:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:46.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:46.318035+0000) 2022-09-10T10:28:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:46.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:46.843634+0000) 2022-09-10T10:28:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:47.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:47.318176+0000) 2022-09-10T10:28:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:47 smithi104 ceph-mon[97044]: pgmap v11505: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T10:28:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:47 smithi110 ceph-mon[97633]: pgmap v11505: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T10:28:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:47.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:47.843817+0000) 2022-09-10T10:28:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:48.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:48.318346+0000) 2022-09-10T10:28:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:48.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:48.844058+0000) 2022-09-10T10:28:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:49.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:49.318554+0000) 2022-09-10T10:28:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:49 smithi104 ceph-mon[97044]: pgmap v11506: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T10:28:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:49 smithi110 ceph-mon[97633]: pgmap v11506: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T10:28:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:49.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:49.844304+0000) 2022-09-10T10:28:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:50.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:50.318803+0000) 2022-09-10T10:28:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:50.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:50.844527+0000) 2022-09-10T10:28:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:51.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:51.319043+0000) 2022-09-10T10:28:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:51 smithi104 ceph-mon[97044]: pgmap v11507: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T10:28:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:51 smithi110 ceph-mon[97633]: pgmap v11507: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T10:28:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:51.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:51.844761+0000) 2022-09-10T10:28:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:52.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:52.319230+0000) 2022-09-10T10:28:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:52.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:52.844974+0000) 2022-09-10T10:28:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:53.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:53.319348+0000) 2022-09-10T10:28:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:53 smithi104 ceph-mon[97044]: pgmap v11508: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T10:28:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:53 smithi110 ceph-mon[97633]: pgmap v11508: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T10:28:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:53.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:53.845247+0000) 2022-09-10T10:28:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:54.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:54.319547+0000) 2022-09-10T10:28:54.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:28:54.463+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:28:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:54.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:54.845393+0000) 2022-09-10T10:28:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:55.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:55.319706+0000) 2022-09-10T10:28:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:55 smithi104 ceph-mon[97044]: pgmap v11509: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T10:28:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:55 smithi110 ceph-mon[97633]: pgmap v11509: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T10:28:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:55.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:55.845608+0000) 2022-09-10T10:28:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:56.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:56.319861+0000) 2022-09-10T10:28:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:56.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:56.845860+0000) 2022-09-10T10:28:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:57.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:57.320054+0000) 2022-09-10T10:28:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:57 smithi104 ceph-mon[97044]: pgmap v11510: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:57 smithi110 ceph-mon[97633]: pgmap v11510: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:28:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:57.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:57.846091+0000) 2022-09-10T10:28:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:58.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:58.320280+0000) 2022-09-10T10:28:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:58.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:58.846380+0000) 2022-09-10T10:28:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:28:59.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:59.320490+0000) 2022-09-10T10:29:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:28:59 smithi104 ceph-mon[97044]: pgmap v11511: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:28:59 smithi110 ceph-mon[97633]: pgmap v11511: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:28:59.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:28:59.846634+0000) 2022-09-10T10:29:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:00.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:00.320749+0000) 2022-09-10T10:29:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:00.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:00.846823+0000) 2022-09-10T10:29:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:01.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:01.320924+0000) 2022-09-10T10:29:02.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:01 smithi104 ceph-mon[97044]: pgmap v11512: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:01 smithi110 ceph-mon[97633]: pgmap v11512: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:01.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:01.847049+0000) 2022-09-10T10:29:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:02.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:02.321170+0000) 2022-09-10T10:29:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:02.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:02.847354+0000) 2022-09-10T10:29:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:03.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:03.321342+0000) 2022-09-10T10:29:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:03 smithi104 ceph-mon[97044]: pgmap v11513: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:03 smithi110 ceph-mon[97633]: pgmap v11513: 129 pgs: 129 active+clean; 9.0 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:03.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:03.847572+0000) 2022-09-10T10:29:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:04.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:04.321530+0000) 2022-09-10T10:29:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:04.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:04.847818+0000) 2022-09-10T10:29:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:05.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:05.321772+0000) 2022-09-10T10:29:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:05 smithi104 ceph-mon[97044]: pgmap v11514: 129 pgs: 129 active+clean; 9.0 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:29:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:05 smithi110 ceph-mon[97633]: pgmap v11514: 129 pgs: 129 active+clean; 9.0 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:29:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:05.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:05.848070+0000) 2022-09-10T10:29:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:06.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:06.321953+0000) 2022-09-10T10:29:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:06.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:06.848345+0000) 2022-09-10T10:29:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:07.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:07.322144+0000) 2022-09-10T10:29:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:07 smithi104 ceph-mon[97044]: pgmap v11515: 129 pgs: 129 active+clean; 9.0 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:29:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:07 smithi110 ceph-mon[97633]: pgmap v11515: 129 pgs: 129 active+clean; 9.0 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:29:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:07.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:07.848642+0000) 2022-09-10T10:29:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:08.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:08.322363+0000) 2022-09-10T10:29:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:08 smithi110 ceph-mon[97633]: osdmap e800: 12 total, 12 up, 12 in 2022-09-10T10:29:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:08 smithi104 ceph-mon[97044]: osdmap e800: 12 total, 12 up, 12 in 2022-09-10T10:29:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:08.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:08.848796+0000) 2022-09-10T10:29:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:09.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:09.322522+0000) 2022-09-10T10:29:09.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:29:09.467+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:29:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:09 smithi110 ceph-mon[97633]: pgmap v11517: 129 pgs: 129 active+clean; 9.0 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:29:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:09 smithi104 ceph-mon[97044]: pgmap v11517: 129 pgs: 129 active+clean; 9.0 MiB data, 1012 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:29:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:09.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:09.848987+0000) 2022-09-10T10:29:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:10.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:10.322774+0000) 2022-09-10T10:29:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:10.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:10.849205+0000) 2022-09-10T10:29:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:11.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:11.323042+0000) 2022-09-10T10:29:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:11 smithi110 ceph-mon[97633]: pgmap v11518: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:29:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:11 smithi104 ceph-mon[97044]: pgmap v11518: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:29:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:11.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:11.849415+0000) 2022-09-10T10:29:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:12.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:12.323281+0000) 2022-09-10T10:29:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:12.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:12.849650+0000) 2022-09-10T10:29:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:13.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:13.323500+0000) 2022-09-10T10:29:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:13 smithi104 ceph-mon[97044]: pgmap v11519: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:29:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:13 smithi104 ceph-mon[97044]: osdmap e801: 12 total, 12 up, 12 in 2022-09-10T10:29:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:13 smithi110 ceph-mon[97633]: pgmap v11519: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:29:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:13 smithi110 ceph-mon[97633]: osdmap e801: 12 total, 12 up, 12 in 2022-09-10T10:29:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:13.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:13.849844+0000) 2022-09-10T10:29:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:14.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:14.323754+0000) 2022-09-10T10:29:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:14.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:14.849991+0000) 2022-09-10T10:29:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:15.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:15.324000+0000) 2022-09-10T10:29:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:15 smithi104 ceph-mon[97044]: pgmap v11521: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:29:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:15 smithi110 ceph-mon[97633]: pgmap v11521: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:29:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:15.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:15.850163+0000) 2022-09-10T10:29:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:16.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:16.324195+0000) 2022-09-10T10:29:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:16.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:16.850362+0000) 2022-09-10T10:29:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:17.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:17.324367+0000) 2022-09-10T10:29:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:17 smithi104 ceph-mon[97044]: pgmap v11522: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:29:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:17 smithi110 ceph-mon[97633]: pgmap v11522: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:29:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:17.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:17.850567+0000) 2022-09-10T10:29:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:18.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:18.324637+0000) 2022-09-10T10:29:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:18.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:18.850814+0000) 2022-09-10T10:29:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:19.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:19.324862+0000) 2022-09-10T10:29:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:19 smithi104 ceph-mon[97044]: pgmap v11523: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:29:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:19 smithi110 ceph-mon[97633]: pgmap v11523: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:29:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:19.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:19.851123+0000) 2022-09-10T10:29:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:20.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:20.325087+0000) 2022-09-10T10:29:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:20.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:20.851287+0000) 2022-09-10T10:29:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:21.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:21.325287+0000) 2022-09-10T10:29:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:21 smithi104 ceph-mon[97044]: pgmap v11524: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:29:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:21 smithi110 ceph-mon[97633]: pgmap v11524: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:29:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:21.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:21.851526+0000) 2022-09-10T10:29:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:22.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:22.325534+0000) 2022-09-10T10:29:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:22.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:22.851771+0000) 2022-09-10T10:29:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:23.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:23.325779+0000) 2022-09-10T10:29:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:23 smithi104 ceph-mon[97044]: pgmap v11525: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:29:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:23 smithi110 ceph-mon[97633]: pgmap v11525: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:29:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:23.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:23.851989+0000) 2022-09-10T10:29:24.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:29:24.475+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:29:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:24.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:24.325924+0000) 2022-09-10T10:29:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:24.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:24.852217+0000) 2022-09-10T10:29:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:25.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:25.326166+0000) 2022-09-10T10:29:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:25 smithi104 ceph-mon[97044]: pgmap v11526: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:29:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:25 smithi110 ceph-mon[97633]: pgmap v11526: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:29:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:25.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:25.852443+0000) 2022-09-10T10:29:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:26.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:26.326390+0000) 2022-09-10T10:29:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:26.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:26.852586+0000) 2022-09-10T10:29:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:27.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:27.326618+0000) 2022-09-10T10:29:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:27 smithi104 ceph-mon[97044]: pgmap v11527: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:27 smithi110 ceph-mon[97633]: pgmap v11527: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:27.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:27.852798+0000) 2022-09-10T10:29:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:28.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:28.326841+0000) 2022-09-10T10:29:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:28.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:28.853007+0000) 2022-09-10T10:29:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:29.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:29.327123+0000) 2022-09-10T10:29:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:29 smithi104 ceph-mon[97044]: pgmap v11528: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:29 smithi110 ceph-mon[97633]: pgmap v11528: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:29.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:29.853244+0000) 2022-09-10T10:29:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:30.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:30.327280+0000) 2022-09-10T10:29:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:30.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:30.853461+0000) 2022-09-10T10:29:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:31.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:31.327521+0000) 2022-09-10T10:29:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:31 smithi104 ceph-mon[97044]: pgmap v11529: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:31 smithi110 ceph-mon[97633]: pgmap v11529: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:31.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:31.853621+0000) 2022-09-10T10:29:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:32.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:32.327719+0000) 2022-09-10T10:29:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:32.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:32.853853+0000) 2022-09-10T10:29:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:33.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:33.327974+0000) 2022-09-10T10:29:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:33 smithi104 ceph-mon[97044]: pgmap v11530: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:33 smithi110 ceph-mon[97633]: pgmap v11530: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:33.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:33.854081+0000) 2022-09-10T10:29:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:34.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:34.328225+0000) 2022-09-10T10:29:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:34.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:34.854329+0000) 2022-09-10T10:29:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:35.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:35.328439+0000) 2022-09-10T10:29:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:35 smithi104 ceph-mon[97044]: pgmap v11531: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:35 smithi110 ceph-mon[97633]: pgmap v11531: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:35.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:35.854577+0000) 2022-09-10T10:29:36.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:36.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:36.328644+0000) 2022-09-10T10:29:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:36.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:36.854828+0000) 2022-09-10T10:29:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:37.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:37.328844+0000) 2022-09-10T10:29:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:37 smithi104 ceph-mon[97044]: pgmap v11532: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:37 smithi110 ceph-mon[97633]: pgmap v11532: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:37.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:37.855053+0000) 2022-09-10T10:29:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:38.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:38.328986+0000) 2022-09-10T10:29:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:38.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:38.855224+0000) 2022-09-10T10:29:39.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:29:39.491+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:29:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:39.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:39.329231+0000) 2022-09-10T10:29:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:39 smithi104 ceph-mon[97044]: pgmap v11533: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:39 smithi110 ceph-mon[97633]: pgmap v11533: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:39.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:39.855466+0000) 2022-09-10T10:29:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:40.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:40.329484+0000) 2022-09-10T10:29:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:40.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:40.855729+0000) 2022-09-10T10:29:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:41.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:41.329713+0000) 2022-09-10T10:29:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:41 smithi104 ceph-mon[97044]: pgmap v11534: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:41 smithi110 ceph-mon[97633]: pgmap v11534: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:41.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:41.855967+0000) 2022-09-10T10:29:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:42.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:42.329852+0000) 2022-09-10T10:29:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:42.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:42.856126+0000) 2022-09-10T10:29:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:43.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:43.330004+0000) 2022-09-10T10:29:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:43 smithi104 ceph-mon[97044]: pgmap v11535: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:43 smithi110 ceph-mon[97633]: pgmap v11535: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:43.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:43.856332+0000) 2022-09-10T10:29:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:44.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:44.330267+0000) 2022-09-10T10:29:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:44.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:44.856599+0000) 2022-09-10T10:29:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:45.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:45.330409+0000) 2022-09-10T10:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:45 smithi104 ceph-mon[97044]: pgmap v11536: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:29:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:29:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:29:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:45 smithi110 ceph-mon[97633]: pgmap v11536: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:29:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:45.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:45.856854+0000) 2022-09-10T10:29:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:46.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:46.330608+0000) 2022-09-10T10:29:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:46.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:46.857032+0000) 2022-09-10T10:29:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:47.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:47.330800+0000) 2022-09-10T10:29:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:47 smithi104 ceph-mon[97044]: pgmap v11537: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:47 smithi110 ceph-mon[97633]: pgmap v11537: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:47.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:47.857273+0000) 2022-09-10T10:29:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:48.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:48.330969+0000) 2022-09-10T10:29:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:48.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:48.857506+0000) 2022-09-10T10:29:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:49.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:49.331194+0000) 2022-09-10T10:29:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:49 smithi104 ceph-mon[97044]: pgmap v11538: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:49 smithi110 ceph-mon[97633]: pgmap v11538: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:49.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:49.857736+0000) 2022-09-10T10:29:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:50.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:50.331396+0000) 2022-09-10T10:29:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:50.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:50.857941+0000) 2022-09-10T10:29:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:51.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:51.331619+0000) 2022-09-10T10:29:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:51 smithi104 ceph-mon[97044]: pgmap v11539: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:51 smithi110 ceph-mon[97633]: pgmap v11539: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:51.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:51.858202+0000) 2022-09-10T10:29:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:52.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:52.331770+0000) 2022-09-10T10:29:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:52.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:52.858412+0000) 2022-09-10T10:29:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:53.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:53.332030+0000) 2022-09-10T10:29:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:53 smithi104 ceph-mon[97044]: pgmap v11540: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:53 smithi110 ceph-mon[97633]: pgmap v11540: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:53.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:53.858624+0000) 2022-09-10T10:29:54.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:29:54.507+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:29:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:54.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:54.332303+0000) 2022-09-10T10:29:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:54.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:54.858904+0000) 2022-09-10T10:29:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:55.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:55.332534+0000) 2022-09-10T10:29:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:55 smithi104 ceph-mon[97044]: pgmap v11541: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:55 smithi110 ceph-mon[97633]: pgmap v11541: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:29:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:55.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:55.859171+0000) 2022-09-10T10:29:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:56.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:56.332725+0000) 2022-09-10T10:29:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:56.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:56.859329+0000) 2022-09-10T10:29:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:57.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:57.332947+0000) 2022-09-10T10:29:58.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:57 smithi104 ceph-mon[97044]: pgmap v11542: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:57 smithi110 ceph-mon[97633]: pgmap v11542: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:29:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:57.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:57.859596+0000) 2022-09-10T10:29:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:58.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:58.333184+0000) 2022-09-10T10:29:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:58.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:58.859857+0000) 2022-09-10T10:29:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:29:59.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:59.333413+0000) 2022-09-10T10:30:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:29:59 smithi104 ceph-mon[97044]: pgmap v11543: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:29:59 smithi110 ceph-mon[97633]: pgmap v11543: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:29:59.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:29:59.860086+0000) 2022-09-10T10:30:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:00.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:00.333582+0000) 2022-09-10T10:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:30:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:30:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:00.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:00.860338+0000) 2022-09-10T10:30:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:01.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:01.333787+0000) 2022-09-10T10:30:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:01 smithi110 ceph-mon[97633]: pgmap v11544: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:01 smithi104 ceph-mon[97044]: pgmap v11544: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:01.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:01.860567+0000) 2022-09-10T10:30:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:02.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:02.334044+0000) 2022-09-10T10:30:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:02.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:02.860736+0000) 2022-09-10T10:30:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:03.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:03.334349+0000) 2022-09-10T10:30:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:03 smithi110 ceph-mon[97633]: pgmap v11545: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:03 smithi104 ceph-mon[97044]: pgmap v11545: 129 pgs: 129 active+clean; 9.0 MiB data, 1013 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:03.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:03.860896+0000) 2022-09-10T10:30:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:04.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:04.334534+0000) 2022-09-10T10:30:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:04.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:04.861134+0000) 2022-09-10T10:30:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:05.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:05.334655+0000) 2022-09-10T10:30:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:05 smithi110 ceph-mon[97633]: pgmap v11546: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:30:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:05 smithi104 ceph-mon[97044]: pgmap v11546: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:30:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:05.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:05.861340+0000) 2022-09-10T10:30:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:06.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:06.334889+0000) 2022-09-10T10:30:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:06.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:06.861529+0000) 2022-09-10T10:30:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:07.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:07.335204+0000) 2022-09-10T10:30:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:07 smithi110 ceph-mon[97633]: pgmap v11547: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:30:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:07 smithi104 ceph-mon[97044]: pgmap v11547: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:30:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:07.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:07.861798+0000) 2022-09-10T10:30:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:08.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:08.335356+0000) 2022-09-10T10:30:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:08 smithi110 ceph-mon[97633]: osdmap e802: 12 total, 12 up, 12 in 2022-09-10T10:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:08 smithi104 ceph-mon[97044]: osdmap e802: 12 total, 12 up, 12 in 2022-09-10T10:30:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:08.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:08.862041+0000) 2022-09-10T10:30:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:30:09.522+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:30:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:09.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:09.335495+0000) 2022-09-10T10:30:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:09 smithi110 ceph-mon[97633]: pgmap v11549: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:30:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:09 smithi104 ceph-mon[97044]: pgmap v11549: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:30:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:09.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:09.862278+0000) 2022-09-10T10:30:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:10.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:10.335691+0000) 2022-09-10T10:30:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:10.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:10.862512+0000) 2022-09-10T10:30:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:11.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:11.335910+0000) 2022-09-10T10:30:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:11 smithi110 ceph-mon[97633]: pgmap v11550: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:30:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:11 smithi104 ceph-mon[97044]: pgmap v11550: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:30:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:11.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:11.862718+0000) 2022-09-10T10:30:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:12.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:12.336160+0000) 2022-09-10T10:30:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:12.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:12.862953+0000) 2022-09-10T10:30:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:13.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:13.336342+0000) 2022-09-10T10:30:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:13 smithi104 ceph-mon[97044]: pgmap v11551: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:30:14.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:13 smithi104 ceph-mon[97044]: osdmap e803: 12 total, 12 up, 12 in 2022-09-10T10:30:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:13 smithi110 ceph-mon[97633]: pgmap v11551: 129 pgs: 129 active+clean; 9.0 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:30:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:13 smithi110 ceph-mon[97633]: osdmap e803: 12 total, 12 up, 12 in 2022-09-10T10:30:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:13.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:13.863196+0000) 2022-09-10T10:30:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:14.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:14.336525+0000) 2022-09-10T10:30:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:14.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:14.863397+0000) 2022-09-10T10:30:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:15.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:15.336706+0000) 2022-09-10T10:30:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:15 smithi104 ceph-mon[97044]: pgmap v11553: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:30:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:15 smithi110 ceph-mon[97633]: pgmap v11553: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:30:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:15.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:15.863556+0000) 2022-09-10T10:30:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:16.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:16.336860+0000) 2022-09-10T10:30:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:16.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:16.863794+0000) 2022-09-10T10:30:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:17.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:17.337064+0000) 2022-09-10T10:30:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:17 smithi104 ceph-mon[97044]: pgmap v11554: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:30:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:17 smithi110 ceph-mon[97633]: pgmap v11554: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:30:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:17.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:17.864071+0000) 2022-09-10T10:30:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:18.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:18.337340+0000) 2022-09-10T10:30:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:18.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:18.864272+0000) 2022-09-10T10:30:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:19.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:19.337536+0000) 2022-09-10T10:30:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:19 smithi104 ceph-mon[97044]: pgmap v11555: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:30:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:19 smithi110 ceph-mon[97633]: pgmap v11555: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:30:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:19.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:19.864523+0000) 2022-09-10T10:30:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:20.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:20.337787+0000) 2022-09-10T10:30:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:20.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:20.864759+0000) 2022-09-10T10:30:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:21.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:21.337973+0000) 2022-09-10T10:30:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:21 smithi104 ceph-mon[97044]: pgmap v11556: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:30:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:21 smithi110 ceph-mon[97633]: pgmap v11556: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:30:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:21.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:21.865017+0000) 2022-09-10T10:30:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:22.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:22.338200+0000) 2022-09-10T10:30:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:22.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:22.865246+0000) 2022-09-10T10:30:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:23.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:23.338461+0000) 2022-09-10T10:30:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:23 smithi104 ceph-mon[97044]: pgmap v11557: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:30:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:23 smithi110 ceph-mon[97633]: pgmap v11557: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:30:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:23.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:23.865496+0000) 2022-09-10T10:30:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:24.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:24.338686+0000) 2022-09-10T10:30:24.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:30:24.534+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:30:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:24.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:24.865768+0000) 2022-09-10T10:30:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:25.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:25.338992+0000) 2022-09-10T10:30:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:25 smithi104 ceph-mon[97044]: pgmap v11558: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:30:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:25 smithi110 ceph-mon[97633]: pgmap v11558: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:30:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:25.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:25.866014+0000) 2022-09-10T10:30:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:26.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:26.339193+0000) 2022-09-10T10:30:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:26.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:26.866256+0000) 2022-09-10T10:30:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:27.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:27.339413+0000) 2022-09-10T10:30:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:27 smithi104 ceph-mon[97044]: pgmap v11559: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:27 smithi110 ceph-mon[97633]: pgmap v11559: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:27.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:27.866457+0000) 2022-09-10T10:30:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:28.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:28.339587+0000) 2022-09-10T10:30:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:28.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:28.866675+0000) 2022-09-10T10:30:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:29.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:29.339811+0000) 2022-09-10T10:30:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:29 smithi104 ceph-mon[97044]: pgmap v11560: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:29 smithi110 ceph-mon[97633]: pgmap v11560: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:29.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:29.866853+0000) 2022-09-10T10:30:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:30.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:30.339986+0000) 2022-09-10T10:30:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:30.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:30.867084+0000) 2022-09-10T10:30:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:31.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:31.340145+0000) 2022-09-10T10:30:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:31 smithi104 ceph-mon[97044]: pgmap v11561: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:31 smithi110 ceph-mon[97633]: pgmap v11561: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:31.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:31.867349+0000) 2022-09-10T10:30:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:32.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:32.340391+0000) 2022-09-10T10:30:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:32.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:32.867600+0000) 2022-09-10T10:30:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:33.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:33.340566+0000) 2022-09-10T10:30:34.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:33 smithi104 ceph-mon[97044]: pgmap v11562: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:33 smithi110 ceph-mon[97633]: pgmap v11562: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:33.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:33.867847+0000) 2022-09-10T10:30:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:34.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:34.340718+0000) 2022-09-10T10:30:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:34.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:34.868045+0000) 2022-09-10T10:30:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:35.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:35.340923+0000) 2022-09-10T10:30:36.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:35 smithi104 ceph-mon[97044]: pgmap v11563: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:35 smithi110 ceph-mon[97633]: pgmap v11563: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:35.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:35.868261+0000) 2022-09-10T10:30:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:36.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:36.341135+0000) 2022-09-10T10:30:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:36.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:36.868552+0000) 2022-09-10T10:30:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:37.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:37.341295+0000) 2022-09-10T10:30:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:37 smithi104 ceph-mon[97044]: pgmap v11564: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:37 smithi110 ceph-mon[97633]: pgmap v11564: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:37.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:37.868756+0000) 2022-09-10T10:30:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:38.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:38.341453+0000) 2022-09-10T10:30:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:38.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:38.869016+0000) 2022-09-10T10:30:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:39.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:39.341695+0000) 2022-09-10T10:30:39.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:30:39.549+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:30:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:39 smithi104 ceph-mon[97044]: pgmap v11565: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:39 smithi110 ceph-mon[97633]: pgmap v11565: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:39.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:39.869249+0000) 2022-09-10T10:30:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:40.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:40.341918+0000) 2022-09-10T10:30:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:40.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:40.869455+0000) 2022-09-10T10:30:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:41.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:41.342170+0000) 2022-09-10T10:30:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:41 smithi104 ceph-mon[97044]: pgmap v11566: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:41 smithi110 ceph-mon[97633]: pgmap v11566: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:41.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:41.869680+0000) 2022-09-10T10:30:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:42.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:42.342414+0000) 2022-09-10T10:30:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:42.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:42.869861+0000) 2022-09-10T10:30:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:43.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:43.342679+0000) 2022-09-10T10:30:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:43 smithi104 ceph-mon[97044]: pgmap v11567: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:43 smithi110 ceph-mon[97633]: pgmap v11567: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:43.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:43.870043+0000) 2022-09-10T10:30:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:44.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:44.342855+0000) 2022-09-10T10:30:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:44.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:44.870275+0000) 2022-09-10T10:30:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:45.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:45.343061+0000) 2022-09-10T10:30:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:45 smithi104 ceph-mon[97044]: pgmap v11568: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:45 smithi110 ceph-mon[97633]: pgmap v11568: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:30:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:45.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:45.870510+0000) 2022-09-10T10:30:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:46.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:46.343316+0000) 2022-09-10T10:30:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:46.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:46.870796+0000) 2022-09-10T10:30:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:47.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:47.343556+0000) 2022-09-10T10:30:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:47 smithi110 ceph-mon[97633]: pgmap v11569: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:47 smithi104 ceph-mon[97044]: pgmap v11569: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:47.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:47.871056+0000) 2022-09-10T10:30:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:48.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:48.343824+0000) 2022-09-10T10:30:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:48.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:48.871180+0000) 2022-09-10T10:30:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:49.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:49.344060+0000) 2022-09-10T10:30:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:49 smithi110 ceph-mon[97633]: pgmap v11570: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:49 smithi104 ceph-mon[97044]: pgmap v11570: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:49.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:49.871372+0000) 2022-09-10T10:30:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:50.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:50.344227+0000) 2022-09-10T10:30:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:50.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:50.871622+0000) 2022-09-10T10:30:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:51.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:51.344430+0000) 2022-09-10T10:30:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:51 smithi110 ceph-mon[97633]: pgmap v11571: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:51 smithi104 ceph-mon[97044]: pgmap v11571: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:51.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:51.871865+0000) 2022-09-10T10:30:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:52.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:52.344610+0000) 2022-09-10T10:30:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:52.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:52.872064+0000) 2022-09-10T10:30:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:53.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:53.344869+0000) 2022-09-10T10:30:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:53 smithi110 ceph-mon[97633]: pgmap v11572: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:53 smithi104 ceph-mon[97044]: pgmap v11572: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:53.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:53.872274+0000) 2022-09-10T10:30:54.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:54.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:54.345155+0000) 2022-09-10T10:30:54.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:30:54.563+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:30:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:54.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:54.872456+0000) 2022-09-10T10:30:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:55.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:55.345314+0000) 2022-09-10T10:30:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:55 smithi110 ceph-mon[97633]: pgmap v11573: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:56.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:55 smithi104 ceph-mon[97044]: pgmap v11573: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:30:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:55.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:55.872619+0000) 2022-09-10T10:30:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:56.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:56.345521+0000) 2022-09-10T10:30:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:56.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:56.872870+0000) 2022-09-10T10:30:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:57.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:57.345804+0000) 2022-09-10T10:30:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:57 smithi110 ceph-mon[97633]: pgmap v11574: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:57 smithi104 ceph-mon[97044]: pgmap v11574: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:30:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:57.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:57.873056+0000) 2022-09-10T10:30:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:58.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:58.346006+0000) 2022-09-10T10:30:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:58.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:58.873200+0000) 2022-09-10T10:30:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:30:59.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:59.346237+0000) 2022-09-10T10:31:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:30:59 smithi110 ceph-mon[97633]: pgmap v11575: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:30:59 smithi104 ceph-mon[97044]: pgmap v11575: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:30:59.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:30:59.873406+0000) 2022-09-10T10:31:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:00.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:00.346516+0000) 2022-09-10T10:31:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:00.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:00.873521+0000) 2022-09-10T10:31:01.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:01.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:01.346765+0000) 2022-09-10T10:31:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:01 smithi110 ceph-mon[97633]: pgmap v11576: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:01 smithi104 ceph-mon[97044]: pgmap v11576: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:01.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:01.873735+0000) 2022-09-10T10:31:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:02.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:02.347005+0000) 2022-09-10T10:31:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:02.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:02.873976+0000) 2022-09-10T10:31:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:03.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:03.347242+0000) 2022-09-10T10:31:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:03 smithi110 ceph-mon[97633]: pgmap v11577: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:03 smithi104 ceph-mon[97044]: pgmap v11577: 129 pgs: 129 active+clean; 9.0 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:03.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:03.874238+0000) 2022-09-10T10:31:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:04.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:04.347449+0000) 2022-09-10T10:31:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:04.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:04.874459+0000) 2022-09-10T10:31:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:05.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:05.347715+0000) 2022-09-10T10:31:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:05 smithi110 ceph-mon[97633]: pgmap v11578: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:31:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:05 smithi104 ceph-mon[97044]: pgmap v11578: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:31:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:05.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:05.874684+0000) 2022-09-10T10:31:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:06.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:06.347968+0000) 2022-09-10T10:31:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:06.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:06.874928+0000) 2022-09-10T10:31:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:07.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:07.348205+0000) 2022-09-10T10:31:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:07 smithi110 ceph-mon[97633]: pgmap v11579: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:31:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:07 smithi104 ceph-mon[97044]: pgmap v11579: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:31:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:07.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:07.875175+0000) 2022-09-10T10:31:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:08.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:08.348427+0000) 2022-09-10T10:31:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:08 smithi110 ceph-mon[97633]: osdmap e804: 12 total, 12 up, 12 in 2022-09-10T10:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:08 smithi104 ceph-mon[97044]: osdmap e804: 12 total, 12 up, 12 in 2022-09-10T10:31:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:08.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:08.875399+0000) 2022-09-10T10:31:09.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:31:09.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:31:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:09.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:09.348562+0000) 2022-09-10T10:31:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:09 smithi110 ceph-mon[97633]: pgmap v11581: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:31:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:09 smithi104 ceph-mon[97044]: pgmap v11581: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:31:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:09.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:09.875637+0000) 2022-09-10T10:31:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:10.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:10.348722+0000) 2022-09-10T10:31:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:10.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:10.875865+0000) 2022-09-10T10:31:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:11.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:11.348894+0000) 2022-09-10T10:31:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:11 smithi110 ceph-mon[97633]: pgmap v11582: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:31:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:11 smithi104 ceph-mon[97044]: pgmap v11582: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:31:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:11.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:11.876083+0000) 2022-09-10T10:31:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:12.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:12.349126+0000) 2022-09-10T10:31:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:12.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:12.876347+0000) 2022-09-10T10:31:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:13.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:13.349302+0000) 2022-09-10T10:31:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:13 smithi104 ceph-mon[97044]: pgmap v11583: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:31:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:13 smithi104 ceph-mon[97044]: osdmap e805: 12 total, 12 up, 12 in 2022-09-10T10:31:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:13 smithi110 ceph-mon[97633]: pgmap v11583: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:31:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:13 smithi110 ceph-mon[97633]: osdmap e805: 12 total, 12 up, 12 in 2022-09-10T10:31:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:13.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:13.876602+0000) 2022-09-10T10:31:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:14.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:14.349527+0000) 2022-09-10T10:31:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:14.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:14.876826+0000) 2022-09-10T10:31:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:15.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:15.349704+0000) 2022-09-10T10:31:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:15 smithi104 ceph-mon[97044]: pgmap v11585: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:31:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:15 smithi110 ceph-mon[97633]: pgmap v11585: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:31:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:15.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:15.877057+0000) 2022-09-10T10:31:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:16.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:16.349927+0000) 2022-09-10T10:31:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:16.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:16.877341+0000) 2022-09-10T10:31:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:17.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:17.350203+0000) 2022-09-10T10:31:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:17 smithi104 ceph-mon[97044]: pgmap v11586: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:31:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:17 smithi110 ceph-mon[97633]: pgmap v11586: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:31:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:17.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:17.877616+0000) 2022-09-10T10:31:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:18.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:18.350422+0000) 2022-09-10T10:31:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:18.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:18.877815+0000) 2022-09-10T10:31:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:19.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:19.350625+0000) 2022-09-10T10:31:20.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:19 smithi104 ceph-mon[97044]: pgmap v11587: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:31:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:19 smithi110 ceph-mon[97633]: pgmap v11587: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:31:20.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:19.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:19.878012+0000) 2022-09-10T10:31:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:20.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:20.350861+0000) 2022-09-10T10:31:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:20.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:20.878253+0000) 2022-09-10T10:31:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:21.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:21.351158+0000) 2022-09-10T10:31:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:21 smithi104 ceph-mon[97044]: pgmap v11588: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:31:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:21 smithi110 ceph-mon[97633]: pgmap v11588: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:31:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:21.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:21.878478+0000) 2022-09-10T10:31:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:22.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:22.351363+0000) 2022-09-10T10:31:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:22.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:22.878748+0000) 2022-09-10T10:31:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:23.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:23.351551+0000) 2022-09-10T10:31:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:23 smithi104 ceph-mon[97044]: pgmap v11589: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:31:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:23 smithi110 ceph-mon[97633]: pgmap v11589: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:31:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:23.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:23.878933+0000) 2022-09-10T10:31:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:24.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:24.351728+0000) 2022-09-10T10:31:24.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:31:24.590+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:31:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:24.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:24.879206+0000) 2022-09-10T10:31:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:25.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:25.351961+0000) 2022-09-10T10:31:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:25 smithi104 ceph-mon[97044]: pgmap v11590: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:31:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:25 smithi110 ceph-mon[97633]: pgmap v11590: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:31:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:25.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:25.879413+0000) 2022-09-10T10:31:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:26.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:26.352193+0000) 2022-09-10T10:31:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:26.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:26.879643+0000) 2022-09-10T10:31:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:27.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:27.352486+0000) 2022-09-10T10:31:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:27 smithi104 ceph-mon[97044]: pgmap v11591: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:27 smithi110 ceph-mon[97633]: pgmap v11591: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:27.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:27.879881+0000) 2022-09-10T10:31:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:28.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:28.352724+0000) 2022-09-10T10:31:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:28.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:28.880155+0000) 2022-09-10T10:31:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:29.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:29.352986+0000) 2022-09-10T10:31:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:29 smithi104 ceph-mon[97044]: pgmap v11592: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:29 smithi110 ceph-mon[97633]: pgmap v11592: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:29.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:29.880394+0000) 2022-09-10T10:31:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:30.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:30.353175+0000) 2022-09-10T10:31:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:30.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:30.880621+0000) 2022-09-10T10:31:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:31.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:31.353381+0000) 2022-09-10T10:31:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:31 smithi104 ceph-mon[97044]: pgmap v11593: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:31 smithi110 ceph-mon[97633]: pgmap v11593: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:31.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:31.880842+0000) 2022-09-10T10:31:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:32.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:32.353549+0000) 2022-09-10T10:31:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:32.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:32.881076+0000) 2022-09-10T10:31:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:33.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:33.353798+0000) 2022-09-10T10:31:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:33 smithi104 ceph-mon[97044]: pgmap v11594: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:33 smithi110 ceph-mon[97633]: pgmap v11594: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:33.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:33.881267+0000) 2022-09-10T10:31:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:34.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:34.353997+0000) 2022-09-10T10:31:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:34.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:34.881515+0000) 2022-09-10T10:31:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:35.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:35.354174+0000) 2022-09-10T10:31:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:35 smithi110 ceph-mon[97633]: pgmap v11595: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:35 smithi104 ceph-mon[97044]: pgmap v11595: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:35.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:35.881700+0000) 2022-09-10T10:31:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:36.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:36.354328+0000) 2022-09-10T10:31:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:36.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:36.881909+0000) 2022-09-10T10:31:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:37.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:37.354493+0000) 2022-09-10T10:31:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:37 smithi110 ceph-mon[97633]: pgmap v11596: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:37 smithi104 ceph-mon[97044]: pgmap v11596: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:37.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:37.882145+0000) 2022-09-10T10:31:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:38.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:38.354733+0000) 2022-09-10T10:31:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:38.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:38.882320+0000) 2022-09-10T10:31:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:39.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:39.354930+0000) 2022-09-10T10:31:39.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:31:39.600+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:31:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:39 smithi110 ceph-mon[97633]: pgmap v11597: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:39 smithi104 ceph-mon[97044]: pgmap v11597: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:39.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:39.882568+0000) 2022-09-10T10:31:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:40.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:40.355135+0000) 2022-09-10T10:31:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:40.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:40.882823+0000) 2022-09-10T10:31:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:41.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:41.355354+0000) 2022-09-10T10:31:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:41 smithi110 ceph-mon[97633]: pgmap v11598: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:41 smithi104 ceph-mon[97044]: pgmap v11598: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:41.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:41.883050+0000) 2022-09-10T10:31:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:42.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:42.355569+0000) 2022-09-10T10:31:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:42.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:42.883214+0000) 2022-09-10T10:31:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:43.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:43.355840+0000) 2022-09-10T10:31:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:43 smithi110 ceph-mon[97633]: pgmap v11599: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:43 smithi104 ceph-mon[97044]: pgmap v11599: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:43.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:43.883397+0000) 2022-09-10T10:31:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:44.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:44.356124+0000) 2022-09-10T10:31:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:44.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:44.883625+0000) 2022-09-10T10:31:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:45.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:45.356275+0000) 2022-09-10T10:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:45 smithi110 ceph-mon[97633]: pgmap v11600: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:31:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:45 smithi104 ceph-mon[97044]: pgmap v11600: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:31:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:31:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:31:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:31:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:45.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:45.883830+0000) 2022-09-10T10:31:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:46.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:46.356477+0000) 2022-09-10T10:31:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:46.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:46.883995+0000) 2022-09-10T10:31:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:47.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:47.356717+0000) 2022-09-10T10:31:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:47 smithi110 ceph-mon[97633]: pgmap v11601: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:47 smithi104 ceph-mon[97044]: pgmap v11601: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:47.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:47.884226+0000) 2022-09-10T10:31:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:48.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:48.356939+0000) 2022-09-10T10:31:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:48.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:48.884465+0000) 2022-09-10T10:31:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:49.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:49.357200+0000) 2022-09-10T10:31:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:49 smithi110 ceph-mon[97633]: pgmap v11602: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:49 smithi104 ceph-mon[97044]: pgmap v11602: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:49.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:49.884655+0000) 2022-09-10T10:31:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:50.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:50.357389+0000) 2022-09-10T10:31:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:50.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:50.884887+0000) 2022-09-10T10:31:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:51.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:51.357586+0000) 2022-09-10T10:31:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:51 smithi110 ceph-mon[97633]: pgmap v11603: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:51 smithi104 ceph-mon[97044]: pgmap v11603: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:51.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:51.885133+0000) 2022-09-10T10:31:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:52.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:52.357786+0000) 2022-09-10T10:31:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:52.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:52.885318+0000) 2022-09-10T10:31:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:53.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:53.358028+0000) 2022-09-10T10:31:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:53 smithi110 ceph-mon[97633]: pgmap v11604: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:53 smithi104 ceph-mon[97044]: pgmap v11604: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:53.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:53.885564+0000) 2022-09-10T10:31:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:54.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:54.358210+0000) 2022-09-10T10:31:54.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:31:54.601+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:31:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:54.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:54.885735+0000) 2022-09-10T10:31:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:55.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:55.358378+0000) 2022-09-10T10:31:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:55 smithi110 ceph-mon[97633]: pgmap v11605: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:55 smithi104 ceph-mon[97044]: pgmap v11605: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:31:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:55.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:55.885963+0000) 2022-09-10T10:31:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:56.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:56.358604+0000) 2022-09-10T10:31:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:56.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:56.886189+0000) 2022-09-10T10:31:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:57.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:57.358846+0000) 2022-09-10T10:31:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:57 smithi110 ceph-mon[97633]: pgmap v11606: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:57 smithi104 ceph-mon[97044]: pgmap v11606: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:31:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:57.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:57.886483+0000) 2022-09-10T10:31:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:58.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:58.359077+0000) 2022-09-10T10:31:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:58.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:58.886738+0000) 2022-09-10T10:31:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:31:59.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:59.359260+0000) 2022-09-10T10:32:00.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:31:59 smithi110 ceph-mon[97633]: pgmap v11607: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:31:59 smithi104 ceph-mon[97044]: pgmap v11607: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:31:59.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:31:59.886964+0000) 2022-09-10T10:32:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:00.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:00.359491+0000) 2022-09-10T10:32:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:00.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:00.887116+0000) 2022-09-10T10:32:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:01.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:01.359728+0000) 2022-09-10T10:32:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:01 smithi110 ceph-mon[97633]: pgmap v11608: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:01 smithi104 ceph-mon[97044]: pgmap v11608: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:01.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:01.887280+0000) 2022-09-10T10:32:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:02.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:02.359953+0000) 2022-09-10T10:32:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:02.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:02.887462+0000) 2022-09-10T10:32:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:03.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:03.360240+0000) 2022-09-10T10:32:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:03 smithi110 ceph-mon[97633]: pgmap v11609: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:03 smithi104 ceph-mon[97044]: pgmap v11609: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:03.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:03.887700+0000) 2022-09-10T10:32:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:04.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:04.360379+0000) 2022-09-10T10:32:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:04.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:04.887869+0000) 2022-09-10T10:32:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:05.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:05.360586+0000) 2022-09-10T10:32:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:05 smithi110 ceph-mon[97633]: pgmap v11610: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:32:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:05 smithi104 ceph-mon[97044]: pgmap v11610: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:32:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:05.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:05.888116+0000) 2022-09-10T10:32:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:06.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:06.360815+0000) 2022-09-10T10:32:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:06.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:06.888342+0000) 2022-09-10T10:32:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:07.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:07.361007+0000) 2022-09-10T10:32:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:07 smithi110 ceph-mon[97633]: pgmap v11611: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:32:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:07 smithi104 ceph-mon[97044]: pgmap v11611: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:32:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:07.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:07.888565+0000) 2022-09-10T10:32:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:08.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:08.361242+0000) 2022-09-10T10:32:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:08 smithi110 ceph-mon[97633]: osdmap e806: 12 total, 12 up, 12 in 2022-09-10T10:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:08 smithi104 ceph-mon[97044]: osdmap e806: 12 total, 12 up, 12 in 2022-09-10T10:32:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:08.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:08.888764+0000) 2022-09-10T10:32:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:09.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:09.361454+0000) 2022-09-10T10:32:09.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:32:09.610+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:32:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:09 smithi110 ceph-mon[97633]: pgmap v11613: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:32:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:09 smithi104 ceph-mon[97044]: pgmap v11613: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:32:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:09.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:09.888940+0000) 2022-09-10T10:32:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:10.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:10.361617+0000) 2022-09-10T10:32:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:10.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:10.889158+0000) 2022-09-10T10:32:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:11.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:11.361826+0000) 2022-09-10T10:32:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:11 smithi110 ceph-mon[97633]: pgmap v11614: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:32:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:11 smithi104 ceph-mon[97044]: pgmap v11614: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:32:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:11.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:11.889360+0000) 2022-09-10T10:32:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:12.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:12.362038+0000) 2022-09-10T10:32:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:12.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:12.889627+0000) 2022-09-10T10:32:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:13.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:13.362213+0000) 2022-09-10T10:32:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:13 smithi104 ceph-mon[97044]: pgmap v11615: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:32:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:13 smithi104 ceph-mon[97044]: osdmap e807: 12 total, 12 up, 12 in 2022-09-10T10:32:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:13 smithi110 ceph-mon[97633]: pgmap v11615: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:32:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:13 smithi110 ceph-mon[97633]: osdmap e807: 12 total, 12 up, 12 in 2022-09-10T10:32:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:13.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:13.889854+0000) 2022-09-10T10:32:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:14.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:14.362441+0000) 2022-09-10T10:32:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:14.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:14.890122+0000) 2022-09-10T10:32:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:15.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:15.362702+0000) 2022-09-10T10:32:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:15 smithi104 ceph-mon[97044]: pgmap v11617: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:32:16.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:15 smithi110 ceph-mon[97633]: pgmap v11617: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:32:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:15.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:15.890381+0000) 2022-09-10T10:32:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:16.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:16.362959+0000) 2022-09-10T10:32:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:16.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:16.890580+0000) 2022-09-10T10:32:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:17.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:17.363225+0000) 2022-09-10T10:32:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:17 smithi104 ceph-mon[97044]: pgmap v11618: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:32:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:17 smithi110 ceph-mon[97633]: pgmap v11618: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:32:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:17.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:17.890805+0000) 2022-09-10T10:32:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:18.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:18.363444+0000) 2022-09-10T10:32:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:18.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:18.891003+0000) 2022-09-10T10:32:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:19.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:19.363571+0000) 2022-09-10T10:32:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:19 smithi104 ceph-mon[97044]: pgmap v11619: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:32:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:19 smithi110 ceph-mon[97633]: pgmap v11619: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:32:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:19.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:19.891247+0000) 2022-09-10T10:32:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:20.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:20.363738+0000) 2022-09-10T10:32:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:20.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:20.891492+0000) 2022-09-10T10:32:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:21.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:21.363999+0000) 2022-09-10T10:32:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:21 smithi104 ceph-mon[97044]: pgmap v11620: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:32:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:21 smithi110 ceph-mon[97633]: pgmap v11620: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:32:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:21.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:21.891735+0000) 2022-09-10T10:32:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:22.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:22.364218+0000) 2022-09-10T10:32:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:22.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:22.891984+0000) 2022-09-10T10:32:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:23.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:23.364388+0000) 2022-09-10T10:32:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:23 smithi104 ceph-mon[97044]: pgmap v11621: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:32:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:23 smithi110 ceph-mon[97633]: pgmap v11621: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:32:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:23.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:23.892171+0000) 2022-09-10T10:32:24.629 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:24.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:24.364529+0000) 2022-09-10T10:32:24.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:32:24.626+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:32:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:24.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:24.892358+0000) 2022-09-10T10:32:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:25.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:25.364650+0000) 2022-09-10T10:32:26.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:25 smithi104 ceph-mon[97044]: pgmap v11622: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:32:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:25 smithi110 ceph-mon[97633]: pgmap v11622: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:32:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:25.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:25.892580+0000) 2022-09-10T10:32:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:26.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:26.364899+0000) 2022-09-10T10:32:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:26.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:26.892775+0000) 2022-09-10T10:32:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:27.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:27.365117+0000) 2022-09-10T10:32:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:27 smithi110 ceph-mon[97633]: pgmap v11623: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:27 smithi104 ceph-mon[97044]: pgmap v11623: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:27.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:27.893025+0000) 2022-09-10T10:32:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:28.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:28.365347+0000) 2022-09-10T10:32:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:28.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:28.893245+0000) 2022-09-10T10:32:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:29.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:29.365526+0000) 2022-09-10T10:32:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:29 smithi110 ceph-mon[97633]: pgmap v11624: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:29 smithi104 ceph-mon[97044]: pgmap v11624: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:30.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:29.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:29.893494+0000) 2022-09-10T10:32:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:30.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:30.365701+0000) 2022-09-10T10:32:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:30.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:30.893765+0000) 2022-09-10T10:32:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:31.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:31.365878+0000) 2022-09-10T10:32:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:31 smithi110 ceph-mon[97633]: pgmap v11625: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:31 smithi104 ceph-mon[97044]: pgmap v11625: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:31.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:31.894003+0000) 2022-09-10T10:32:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:32.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:32.366174+0000) 2022-09-10T10:32:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:32.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:32.894178+0000) 2022-09-10T10:32:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:33.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:33.366398+0000) 2022-09-10T10:32:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:33 smithi110 ceph-mon[97633]: pgmap v11626: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:33 smithi104 ceph-mon[97044]: pgmap v11626: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:33.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:33.894306+0000) 2022-09-10T10:32:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:34.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:34.366537+0000) 2022-09-10T10:32:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:34.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:34.894477+0000) 2022-09-10T10:32:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:35.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:35.366769+0000) 2022-09-10T10:32:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:35 smithi110 ceph-mon[97633]: pgmap v11627: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:35 smithi104 ceph-mon[97044]: pgmap v11627: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:35.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:35.894723+0000) 2022-09-10T10:32:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:36.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:36.366997+0000) 2022-09-10T10:32:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:36.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:36.894926+0000) 2022-09-10T10:32:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:37.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:37.367192+0000) 2022-09-10T10:32:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:37 smithi110 ceph-mon[97633]: pgmap v11628: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:37 smithi104 ceph-mon[97044]: pgmap v11628: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:37.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:37.895200+0000) 2022-09-10T10:32:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:38.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:38.367357+0000) 2022-09-10T10:32:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:38.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:38.895447+0000) 2022-09-10T10:32:39.643 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:39.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:39.367491+0000) 2022-09-10T10:32:39.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:32:39.641+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:32:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:39 smithi110 ceph-mon[97633]: pgmap v11629: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:39 smithi104 ceph-mon[97044]: pgmap v11629: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:39.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:39.895641+0000) 2022-09-10T10:32:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:40.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:40.367708+0000) 2022-09-10T10:32:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:40.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:40.895802+0000) 2022-09-10T10:32:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:41.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:41.367907+0000) 2022-09-10T10:32:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:41 smithi110 ceph-mon[97633]: pgmap v11630: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:41 smithi104 ceph-mon[97044]: pgmap v11630: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:41.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:41.896033+0000) 2022-09-10T10:32:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:42.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:42.368126+0000) 2022-09-10T10:32:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:42.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:42.896259+0000) 2022-09-10T10:32:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:43.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:43.368340+0000) 2022-09-10T10:32:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:43 smithi110 ceph-mon[97633]: pgmap v11631: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:43 smithi104 ceph-mon[97044]: pgmap v11631: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:43.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:43.896508+0000) 2022-09-10T10:32:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:44.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:44.368610+0000) 2022-09-10T10:32:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:44.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:44.896733+0000) 2022-09-10T10:32:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:45.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:45.368856+0000) 2022-09-10T10:32:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:45 smithi110 ceph-mon[97633]: pgmap v11632: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:32:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:45 smithi104 ceph-mon[97044]: pgmap v11632: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:32:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:32:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:32:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:32:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:45.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:45.896953+0000) 2022-09-10T10:32:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:46.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:46.369060+0000) 2022-09-10T10:32:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:46.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:46.897191+0000) 2022-09-10T10:32:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:47.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:47.369260+0000) 2022-09-10T10:32:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:47 smithi110 ceph-mon[97633]: pgmap v11633: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:32:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:47 smithi104 ceph-mon[97044]: pgmap v11633: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:32:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:47.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:47.897373+0000) 2022-09-10T10:32:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:48.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:48.369468+0000) 2022-09-10T10:32:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:48.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:48.897645+0000) 2022-09-10T10:32:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:49.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:49.369696+0000) 2022-09-10T10:32:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:49 smithi110 ceph-mon[97633]: pgmap v11634: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:32:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:49 smithi104 ceph-mon[97044]: pgmap v11634: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:32:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:49.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:49.897896+0000) 2022-09-10T10:32:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:50.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:50.369945+0000) 2022-09-10T10:32:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:50.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:50.898146+0000) 2022-09-10T10:32:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:51.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:51.370203+0000) 2022-09-10T10:32:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:51 smithi110 ceph-mon[97633]: pgmap v11635: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:51 smithi104 ceph-mon[97044]: pgmap v11635: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:51.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:51.898344+0000) 2022-09-10T10:32:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:52.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:52.370456+0000) 2022-09-10T10:32:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:52.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:52.898531+0000) 2022-09-10T10:32:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:53.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:53.370613+0000) 2022-09-10T10:32:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:53 smithi110 ceph-mon[97633]: pgmap v11636: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:32:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:53 smithi104 ceph-mon[97044]: pgmap v11636: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:32:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:53.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:53.898773+0000) 2022-09-10T10:32:54.647 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:54.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:54.370857+0000) 2022-09-10T10:32:54.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:32:54.645+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:32:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:54.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:54.899009+0000) 2022-09-10T10:32:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:55.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:55.371062+0000) 2022-09-10T10:32:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:55 smithi110 ceph-mon[97633]: pgmap v11637: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:55 smithi104 ceph-mon[97044]: pgmap v11637: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:32:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:55.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:55.899239+0000) 2022-09-10T10:32:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:56.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:56.371309+0000) 2022-09-10T10:32:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:56.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:56.899457+0000) 2022-09-10T10:32:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:57.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:57.371431+0000) 2022-09-10T10:32:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:57 smithi110 ceph-mon[97633]: pgmap v11638: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:57 smithi104 ceph-mon[97044]: pgmap v11638: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:32:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:57.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:57.899695+0000) 2022-09-10T10:32:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:58.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:58.371630+0000) 2022-09-10T10:32:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:58.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:58.899906+0000) 2022-09-10T10:32:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:32:59.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:59.371802+0000) 2022-09-10T10:33:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:32:59 smithi110 ceph-mon[97633]: pgmap v11639: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:32:59 smithi104 ceph-mon[97044]: pgmap v11639: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:32:59.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:32:59.900159+0000) 2022-09-10T10:33:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:00.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:00.371930+0000) 2022-09-10T10:33:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:00.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:00.900414+0000) 2022-09-10T10:33:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:01.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:01.372170+0000) 2022-09-10T10:33:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:01 smithi110 ceph-mon[97633]: pgmap v11640: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:01 smithi104 ceph-mon[97044]: pgmap v11640: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:01.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:01.900646+0000) 2022-09-10T10:33:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:02.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:02.372376+0000) 2022-09-10T10:33:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:02.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:02.900830+0000) 2022-09-10T10:33:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:03.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:03.372650+0000) 2022-09-10T10:33:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:03 smithi110 ceph-mon[97633]: pgmap v11641: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:03 smithi104 ceph-mon[97044]: pgmap v11641: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:03.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:03.901051+0000) 2022-09-10T10:33:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:04.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:04.372916+0000) 2022-09-10T10:33:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:04.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:04.901279+0000) 2022-09-10T10:33:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:05.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:05.373080+0000) 2022-09-10T10:33:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:05 smithi110 ceph-mon[97633]: pgmap v11642: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:33:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:05 smithi104 ceph-mon[97044]: pgmap v11642: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:33:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:05.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:05.901466+0000) 2022-09-10T10:33:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:06.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:06.373339+0000) 2022-09-10T10:33:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:06.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:06.901689+0000) 2022-09-10T10:33:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:07.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:07.373524+0000) 2022-09-10T10:33:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:07 smithi110 ceph-mon[97633]: pgmap v11643: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:33:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:07 smithi104 ceph-mon[97044]: pgmap v11643: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:33:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:07.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:07.901932+0000) 2022-09-10T10:33:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:08.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:08.373708+0000) 2022-09-10T10:33:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:08 smithi110 ceph-mon[97633]: osdmap e808: 12 total, 12 up, 12 in 2022-09-10T10:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:08 smithi104 ceph-mon[97044]: osdmap e808: 12 total, 12 up, 12 in 2022-09-10T10:33:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:08.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:08.902142+0000) 2022-09-10T10:33:09.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:09.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:09.373944+0000) 2022-09-10T10:33:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:09.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:09.902306+0000) 2022-09-10T10:33:10.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:33:09.660+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:33:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:09 smithi110 ceph-mon[97633]: pgmap v11645: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:33:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:09 smithi104 ceph-mon[97044]: pgmap v11645: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:33:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:10.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:10.374195+0000) 2022-09-10T10:33:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:10.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:10.902572+0000) 2022-09-10T10:33:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:11.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:11.374421+0000) 2022-09-10T10:33:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:11 smithi110 ceph-mon[97633]: pgmap v11646: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:33:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:11 smithi104 ceph-mon[97044]: pgmap v11646: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:33:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:11.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:11.902810+0000) 2022-09-10T10:33:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:12.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:12.374673+0000) 2022-09-10T10:33:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:12.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:12.903017+0000) 2022-09-10T10:33:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:13.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:13.374912+0000) 2022-09-10T10:33:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:13 smithi104 ceph-mon[97044]: pgmap v11647: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:33:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:13 smithi104 ceph-mon[97044]: osdmap e809: 12 total, 12 up, 12 in 2022-09-10T10:33:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:13 smithi110 ceph-mon[97633]: pgmap v11647: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:33:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:13 smithi110 ceph-mon[97633]: osdmap e809: 12 total, 12 up, 12 in 2022-09-10T10:33:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:13.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:13.903264+0000) 2022-09-10T10:33:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:14.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:14.375179+0000) 2022-09-10T10:33:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:14.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:14.903420+0000) 2022-09-10T10:33:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:15.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:15.375441+0000) 2022-09-10T10:33:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:15 smithi104 ceph-mon[97044]: pgmap v11649: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:33:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:15 smithi110 ceph-mon[97633]: pgmap v11649: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:33:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:15.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:15.903645+0000) 2022-09-10T10:33:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:16.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:16.375682+0000) 2022-09-10T10:33:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:16.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:16.903871+0000) 2022-09-10T10:33:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:17.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:17.375880+0000) 2022-09-10T10:33:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:17 smithi110 ceph-mon[97633]: pgmap v11650: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:33:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:17 smithi104 ceph-mon[97044]: pgmap v11650: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:33:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:17.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:17.904076+0000) 2022-09-10T10:33:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:18.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:18.376145+0000) 2022-09-10T10:33:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:18.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:18.904311+0000) 2022-09-10T10:33:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:19.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:19.376316+0000) 2022-09-10T10:33:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:19 smithi110 ceph-mon[97633]: pgmap v11651: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:33:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:19 smithi104 ceph-mon[97044]: pgmap v11651: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:33:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:19.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:19.904523+0000) 2022-09-10T10:33:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:20.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:20.376541+0000) 2022-09-10T10:33:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:20.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:20.904738+0000) 2022-09-10T10:33:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:21.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:21.376730+0000) 2022-09-10T10:33:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:21 smithi110 ceph-mon[97633]: pgmap v11652: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:33:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:21 smithi104 ceph-mon[97044]: pgmap v11652: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:33:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:21.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:21.904988+0000) 2022-09-10T10:33:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:22.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:22.376926+0000) 2022-09-10T10:33:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:22.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:22.905217+0000) 2022-09-10T10:33:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:23.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:23.377197+0000) 2022-09-10T10:33:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:23 smithi110 ceph-mon[97633]: pgmap v11653: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:33:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:23 smithi104 ceph-mon[97044]: pgmap v11653: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:33:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:23.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:23.905430+0000) 2022-09-10T10:33:24.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:24.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:24.377415+0000) 2022-09-10T10:33:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:33:24.676+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:33:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:24.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:24.905705+0000) 2022-09-10T10:33:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:25.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:25.377583+0000) 2022-09-10T10:33:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:25 smithi110 ceph-mon[97633]: pgmap v11654: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:33:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:25 smithi104 ceph-mon[97044]: pgmap v11654: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:33:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:25.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:25.905939+0000) 2022-09-10T10:33:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:26.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:26.377837+0000) 2022-09-10T10:33:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:26.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:26.906189+0000) 2022-09-10T10:33:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:27.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:27.378115+0000) 2022-09-10T10:33:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:27 smithi110 ceph-mon[97633]: pgmap v11655: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:27 smithi104 ceph-mon[97044]: pgmap v11655: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:27.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:27.906424+0000) 2022-09-10T10:33:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:28.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:28.378335+0000) 2022-09-10T10:33:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:28.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:28.906673+0000) 2022-09-10T10:33:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:29.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:29.378517+0000) 2022-09-10T10:33:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:29 smithi110 ceph-mon[97633]: pgmap v11656: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:29 smithi104 ceph-mon[97044]: pgmap v11656: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:29.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:29.906932+0000) 2022-09-10T10:33:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:30.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:30.378749+0000) 2022-09-10T10:33:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:30.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:30.907204+0000) 2022-09-10T10:33:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:31.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:31.378930+0000) 2022-09-10T10:33:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:31 smithi110 ceph-mon[97633]: pgmap v11657: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:31 smithi104 ceph-mon[97044]: pgmap v11657: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:31.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:31.907371+0000) 2022-09-10T10:33:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:32.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:32.379170+0000) 2022-09-10T10:33:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:32.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:32.907567+0000) 2022-09-10T10:33:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:33.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:33.379402+0000) 2022-09-10T10:33:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:33 smithi110 ceph-mon[97633]: pgmap v11658: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:33 smithi104 ceph-mon[97044]: pgmap v11658: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:33.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:33.907799+0000) 2022-09-10T10:33:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:34.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:34.379581+0000) 2022-09-10T10:33:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:34.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:34.908060+0000) 2022-09-10T10:33:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:35.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:35.379805+0000) 2022-09-10T10:33:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:35 smithi110 ceph-mon[97633]: pgmap v11659: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:35 smithi104 ceph-mon[97044]: pgmap v11659: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:35.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:35.908275+0000) 2022-09-10T10:33:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:36.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:36.380001+0000) 2022-09-10T10:33:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:36.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:36.908461+0000) 2022-09-10T10:33:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:37.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:37.380217+0000) 2022-09-10T10:33:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:37 smithi110 ceph-mon[97633]: pgmap v11660: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:37 smithi104 ceph-mon[97044]: pgmap v11660: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:37.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:37.908713+0000) 2022-09-10T10:33:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:38.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:38.380373+0000) 2022-09-10T10:33:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:38.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:38.908951+0000) 2022-09-10T10:33:39.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:39.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:39.380529+0000) 2022-09-10T10:33:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:33:39.690+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:33:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:39.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:39.909201+0000) 2022-09-10T10:33:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:39 smithi110 ceph-mon[97633]: pgmap v11661: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:39 smithi104 ceph-mon[97044]: pgmap v11661: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:40.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:40.380686+0000) 2022-09-10T10:33:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:40.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:40.909439+0000) 2022-09-10T10:33:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:41.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:41.380852+0000) 2022-09-10T10:33:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:41 smithi110 ceph-mon[97633]: pgmap v11662: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:41 smithi104 ceph-mon[97044]: pgmap v11662: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:41.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:41.909632+0000) 2022-09-10T10:33:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:42.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:42.381091+0000) 2022-09-10T10:33:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:42.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:42.909876+0000) 2022-09-10T10:33:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:43.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:43.381334+0000) 2022-09-10T10:33:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:43 smithi110 ceph-mon[97633]: pgmap v11663: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:43 smithi104 ceph-mon[97044]: pgmap v11663: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:43.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:43.910136+0000) 2022-09-10T10:33:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:44.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:44.381525+0000) 2022-09-10T10:33:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:44.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:44.910372+0000) 2022-09-10T10:33:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:45.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:45.381785+0000) 2022-09-10T10:33:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:45 smithi110 ceph-mon[97633]: pgmap v11664: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:33:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:45 smithi104 ceph-mon[97044]: pgmap v11664: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:33:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:33:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:33:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:33:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:45.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:45.910603+0000) 2022-09-10T10:33:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:46.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:46.382000+0000) 2022-09-10T10:33:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:46.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:46.910809+0000) 2022-09-10T10:33:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:47.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:47.382255+0000) 2022-09-10T10:33:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:47 smithi110 ceph-mon[97633]: pgmap v11665: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:47 smithi104 ceph-mon[97044]: pgmap v11665: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:47.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:47.911088+0000) 2022-09-10T10:33:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:48.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:48.382505+0000) 2022-09-10T10:33:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:48.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:48.911339+0000) 2022-09-10T10:33:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:49.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:49.382751+0000) 2022-09-10T10:33:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:49 smithi110 ceph-mon[97633]: pgmap v11666: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:49 smithi104 ceph-mon[97044]: pgmap v11666: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:49.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:49.911539+0000) 2022-09-10T10:33:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:50.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:50.382972+0000) 2022-09-10T10:33:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:50.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:50.911723+0000) 2022-09-10T10:33:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:51.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:51.383233+0000) 2022-09-10T10:33:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:51 smithi110 ceph-mon[97633]: pgmap v11667: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:51 smithi104 ceph-mon[97044]: pgmap v11667: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:52.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:51.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:51.911948+0000) 2022-09-10T10:33:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:52.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:52.383500+0000) 2022-09-10T10:33:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:52.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:52.912161+0000) 2022-09-10T10:33:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:53.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:53.383795+0000) 2022-09-10T10:33:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:53 smithi110 ceph-mon[97633]: pgmap v11668: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:53 smithi104 ceph-mon[97044]: pgmap v11668: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:53.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:53.912319+0000) 2022-09-10T10:33:54.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:54.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:54.384062+0000) 2022-09-10T10:33:55.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:33:54.705+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:33:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:54.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:54.912501+0000) 2022-09-10T10:33:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:55.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:55.384281+0000) 2022-09-10T10:33:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:55 smithi110 ceph-mon[97633]: pgmap v11669: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:55 smithi104 ceph-mon[97044]: pgmap v11669: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:33:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:55.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:55.912723+0000) 2022-09-10T10:33:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:56.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:56.384509+0000) 2022-09-10T10:33:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:56.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:56.912990+0000) 2022-09-10T10:33:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:57.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:57.384696+0000) 2022-09-10T10:33:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:57 smithi110 ceph-mon[97633]: pgmap v11670: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:57 smithi104 ceph-mon[97044]: pgmap v11670: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:33:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:57.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:57.913245+0000) 2022-09-10T10:33:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:58.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:58.384905+0000) 2022-09-10T10:33:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:58.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:58.913444+0000) 2022-09-10T10:33:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:33:59.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:59.385130+0000) 2022-09-10T10:34:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:33:59 smithi110 ceph-mon[97633]: pgmap v11671: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:33:59 smithi104 ceph-mon[97044]: pgmap v11671: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:33:59.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:33:59.913673+0000) 2022-09-10T10:34:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:00.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:00.385377+0000) 2022-09-10T10:34:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:00.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:00.913899+0000) 2022-09-10T10:34:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:01.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:01.385638+0000) 2022-09-10T10:34:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:01 smithi110 ceph-mon[97633]: pgmap v11672: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:01 smithi104 ceph-mon[97044]: pgmap v11672: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:01.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:01.914167+0000) 2022-09-10T10:34:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:02.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:02.385897+0000) 2022-09-10T10:34:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:02.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:02.914394+0000) 2022-09-10T10:34:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:03.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:03.386133+0000) 2022-09-10T10:34:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:03 smithi110 ceph-mon[97633]: pgmap v11673: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:03 smithi104 ceph-mon[97044]: pgmap v11673: 129 pgs: 129 active+clean; 9.0 MiB data, 1022 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:03.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:03.914597+0000) 2022-09-10T10:34:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:04.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:04.386354+0000) 2022-09-10T10:34:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:04.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:04.914827+0000) 2022-09-10T10:34:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:05.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:05.386521+0000) 2022-09-10T10:34:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:05 smithi110 ceph-mon[97633]: pgmap v11674: 129 pgs: 129 active+clean; 9.0 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:34:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:05 smithi104 ceph-mon[97044]: pgmap v11674: 129 pgs: 129 active+clean; 9.0 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:34:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:05.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:05.915077+0000) 2022-09-10T10:34:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:06.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:06.386682+0000) 2022-09-10T10:34:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:06.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:06.915374+0000) 2022-09-10T10:34:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:07.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:07.386910+0000) 2022-09-10T10:34:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:07 smithi110 ceph-mon[97633]: pgmap v11675: 129 pgs: 129 active+clean; 9.0 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:34:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:07 smithi104 ceph-mon[97044]: pgmap v11675: 129 pgs: 129 active+clean; 9.0 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:34:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:07.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:07.915672+0000) 2022-09-10T10:34:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:08.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:08.387180+0000) 2022-09-10T10:34:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:08 smithi110 ceph-mon[97633]: osdmap e810: 12 total, 12 up, 12 in 2022-09-10T10:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:08 smithi104 ceph-mon[97044]: osdmap e810: 12 total, 12 up, 12 in 2022-09-10T10:34:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:08.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:08.915892+0000) 2022-09-10T10:34:09.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:09.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:09.387411+0000) 2022-09-10T10:34:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:09.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:09.916068+0000) 2022-09-10T10:34:10.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:34:09.718+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:34:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:09 smithi110 ceph-mon[97633]: pgmap v11677: 129 pgs: 129 active+clean; 9.0 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:34:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:09 smithi104 ceph-mon[97044]: pgmap v11677: 129 pgs: 129 active+clean; 9.0 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:34:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:10.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:10.387555+0000) 2022-09-10T10:34:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:10.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:10.916251+0000) 2022-09-10T10:34:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:11.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:11.387787+0000) 2022-09-10T10:34:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:11 smithi110 ceph-mon[97633]: pgmap v11678: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:34:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:11 smithi104 ceph-mon[97044]: pgmap v11678: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:34:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:11.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:11.916453+0000) 2022-09-10T10:34:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:12.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:12.387965+0000) 2022-09-10T10:34:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:12.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:12.916711+0000) 2022-09-10T10:34:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:13.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:13.388201+0000) 2022-09-10T10:34:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:13 smithi110 ceph-mon[97633]: pgmap v11679: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:34:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:13 smithi110 ceph-mon[97633]: osdmap e811: 12 total, 12 up, 12 in 2022-09-10T10:34:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:13 smithi104 ceph-mon[97044]: pgmap v11679: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:34:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:13 smithi104 ceph-mon[97044]: osdmap e811: 12 total, 12 up, 12 in 2022-09-10T10:34:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:13.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:13.916938+0000) 2022-09-10T10:34:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:14.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:14.388448+0000) 2022-09-10T10:34:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:14.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:14.917197+0000) 2022-09-10T10:34:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:15.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:15.388652+0000) 2022-09-10T10:34:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:15 smithi110 ceph-mon[97633]: pgmap v11681: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:34:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:15 smithi104 ceph-mon[97044]: pgmap v11681: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:34:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:15.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:15.917439+0000) 2022-09-10T10:34:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:16.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:16.388878+0000) 2022-09-10T10:34:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:16.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:16.917652+0000) 2022-09-10T10:34:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:17.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:17.389162+0000) 2022-09-10T10:34:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:17 smithi110 ceph-mon[97633]: pgmap v11682: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:34:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:17 smithi104 ceph-mon[97044]: pgmap v11682: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:34:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:17.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:17.917929+0000) 2022-09-10T10:34:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:18.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:18.389301+0000) 2022-09-10T10:34:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:18.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:18.918186+0000) 2022-09-10T10:34:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:19.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:19.389546+0000) 2022-09-10T10:34:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:19 smithi110 ceph-mon[97633]: pgmap v11683: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:34:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:19 smithi104 ceph-mon[97044]: pgmap v11683: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:34:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:19.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:19.918412+0000) 2022-09-10T10:34:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:20.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:20.389740+0000) 2022-09-10T10:34:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:20.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:20.918675+0000) 2022-09-10T10:34:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:21.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:21.389911+0000) 2022-09-10T10:34:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:21 smithi110 ceph-mon[97633]: pgmap v11684: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:34:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:21 smithi104 ceph-mon[97044]: pgmap v11684: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:34:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:21.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:21.918926+0000) 2022-09-10T10:34:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:22.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:22.390123+0000) 2022-09-10T10:34:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:22.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:22.919208+0000) 2022-09-10T10:34:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:23.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:23.390332+0000) 2022-09-10T10:34:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:23 smithi110 ceph-mon[97633]: pgmap v11685: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:34:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:23 smithi104 ceph-mon[97044]: pgmap v11685: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:34:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:23.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:23.919417+0000) 2022-09-10T10:34:24.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:24.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:24.390522+0000) 2022-09-10T10:34:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:34:24.732+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:34:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:24.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:24.919598+0000) 2022-09-10T10:34:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:25.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:25.390781+0000) 2022-09-10T10:34:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:25 smithi110 ceph-mon[97633]: pgmap v11686: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:34:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:25 smithi104 ceph-mon[97044]: pgmap v11686: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:34:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:25.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:25.919836+0000) 2022-09-10T10:34:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:26.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:26.390977+0000) 2022-09-10T10:34:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:26.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:26.920070+0000) 2022-09-10T10:34:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:27.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:27.391134+0000) 2022-09-10T10:34:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:27 smithi110 ceph-mon[97633]: pgmap v11687: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:27 smithi104 ceph-mon[97044]: pgmap v11687: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:27.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:27.920330+0000) 2022-09-10T10:34:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:28.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:28.391342+0000) 2022-09-10T10:34:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:28.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:28.920595+0000) 2022-09-10T10:34:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:29.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:29.391575+0000) 2022-09-10T10:34:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:29 smithi110 ceph-mon[97633]: pgmap v11688: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:29 smithi104 ceph-mon[97044]: pgmap v11688: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:29.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:29.920806+0000) 2022-09-10T10:34:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:30.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:30.391822+0000) 2022-09-10T10:34:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:30.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:30.921042+0000) 2022-09-10T10:34:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:31.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:31.392068+0000) 2022-09-10T10:34:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:31 smithi110 ceph-mon[97633]: pgmap v11689: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:31 smithi104 ceph-mon[97044]: pgmap v11689: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:31.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:31.921262+0000) 2022-09-10T10:34:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:32.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:32.392327+0000) 2022-09-10T10:34:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:32.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:32.921513+0000) 2022-09-10T10:34:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:33.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:33.392522+0000) 2022-09-10T10:34:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:33 smithi110 ceph-mon[97633]: pgmap v11690: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:33 smithi104 ceph-mon[97044]: pgmap v11690: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:33.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:33.921698+0000) 2022-09-10T10:34:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:34.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:34.392782+0000) 2022-09-10T10:34:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:34.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:34.921894+0000) 2022-09-10T10:34:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:35.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:35.392940+0000) 2022-09-10T10:34:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:35 smithi110 ceph-mon[97633]: pgmap v11691: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:35 smithi104 ceph-mon[97044]: pgmap v11691: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:35.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:35.922150+0000) 2022-09-10T10:34:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:36.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:36.393188+0000) 2022-09-10T10:34:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:36.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:36.922295+0000) 2022-09-10T10:34:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:37.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:37.393419+0000) 2022-09-10T10:34:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:37 smithi110 ceph-mon[97633]: pgmap v11692: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:37 smithi104 ceph-mon[97044]: pgmap v11692: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:37.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:37.922537+0000) 2022-09-10T10:34:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:38.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:38.393646+0000) 2022-09-10T10:34:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:38.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:38.922703+0000) 2022-09-10T10:34:39.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:39.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:39.393866+0000) 2022-09-10T10:34:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:39.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:39.922879+0000) 2022-09-10T10:34:40.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:34:39.748+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:34:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:39 smithi110 ceph-mon[97633]: pgmap v11693: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:39 smithi104 ceph-mon[97044]: pgmap v11693: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:40.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:40.394142+0000) 2022-09-10T10:34:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:40.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:40.923166+0000) 2022-09-10T10:34:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:41.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:41.394322+0000) 2022-09-10T10:34:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:41 smithi110 ceph-mon[97633]: pgmap v11694: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:41 smithi104 ceph-mon[97044]: pgmap v11694: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:41.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:41.923390+0000) 2022-09-10T10:34:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:42.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:42.394547+0000) 2022-09-10T10:34:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:42.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:42.923573+0000) 2022-09-10T10:34:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:43.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:43.394813+0000) 2022-09-10T10:34:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:43 smithi110 ceph-mon[97633]: pgmap v11695: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:43 smithi104 ceph-mon[97044]: pgmap v11695: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:43.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:43.923794+0000) 2022-09-10T10:34:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:44.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:44.395063+0000) 2022-09-10T10:34:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:44.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:44.923974+0000) 2022-09-10T10:34:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:45.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:45.395320+0000) 2022-09-10T10:34:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:45 smithi110 ceph-mon[97633]: pgmap v11696: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:34:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:34:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:34:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:45 smithi104 ceph-mon[97044]: pgmap v11696: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:34:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:45.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:45.924238+0000) 2022-09-10T10:34:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:46.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:46.395504+0000) 2022-09-10T10:34:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:46.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:46.924470+0000) 2022-09-10T10:34:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:47.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:47.395708+0000) 2022-09-10T10:34:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:47 smithi110 ceph-mon[97633]: pgmap v11697: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:47 smithi104 ceph-mon[97044]: pgmap v11697: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:47.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:47.924733+0000) 2022-09-10T10:34:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:48.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:48.395940+0000) 2022-09-10T10:34:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:48.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:48.925016+0000) 2022-09-10T10:34:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:49.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:49.396173+0000) 2022-09-10T10:34:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:49 smithi110 ceph-mon[97633]: pgmap v11698: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:49 smithi104 ceph-mon[97044]: pgmap v11698: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:49.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:49.925239+0000) 2022-09-10T10:34:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:50.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:50.396378+0000) 2022-09-10T10:34:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:50.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:50.925434+0000) 2022-09-10T10:34:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:51.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:51.396620+0000) 2022-09-10T10:34:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:51 smithi110 ceph-mon[97633]: pgmap v11699: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:51 smithi104 ceph-mon[97044]: pgmap v11699: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:51.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:51.925651+0000) 2022-09-10T10:34:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:52.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:52.396874+0000) 2022-09-10T10:34:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:52.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:52.925826+0000) 2022-09-10T10:34:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:53.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:53.397143+0000) 2022-09-10T10:34:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:53 smithi110 ceph-mon[97633]: pgmap v11700: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:53 smithi104 ceph-mon[97044]: pgmap v11700: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:53.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:53.926038+0000) 2022-09-10T10:34:54.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:54.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:54.397364+0000) 2022-09-10T10:34:55.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:34:54.763+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:34:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:54.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:54.926239+0000) 2022-09-10T10:34:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:55.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:55.397604+0000) 2022-09-10T10:34:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:55 smithi110 ceph-mon[97633]: pgmap v11701: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:55 smithi104 ceph-mon[97044]: pgmap v11701: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:34:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:55.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:55.926408+0000) 2022-09-10T10:34:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:56.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:56.397778+0000) 2022-09-10T10:34:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:56.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:56.926663+0000) 2022-09-10T10:34:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:57.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:57.397984+0000) 2022-09-10T10:34:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:57 smithi110 ceph-mon[97633]: pgmap v11702: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:57 smithi104 ceph-mon[97044]: pgmap v11702: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:34:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:57.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:57.926924+0000) 2022-09-10T10:34:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:58.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:58.398216+0000) 2022-09-10T10:34:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:58.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:58.927170+0000) 2022-09-10T10:34:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:34:59.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:59.398465+0000) 2022-09-10T10:35:00.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:34:59 smithi110 ceph-mon[97633]: pgmap v11703: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:34:59 smithi104 ceph-mon[97044]: pgmap v11703: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:34:59.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:34:59.927342+0000) 2022-09-10T10:35:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:00.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:00.398625+0000) 2022-09-10T10:35:01.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:00.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:00.927489+0000) 2022-09-10T10:35:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:01.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:01.398781+0000) 2022-09-10T10:35:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:01 smithi110 ceph-mon[97633]: pgmap v11704: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:01 smithi104 ceph-mon[97044]: pgmap v11704: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:01.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:01.927718+0000) 2022-09-10T10:35:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:02.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:02.398978+0000) 2022-09-10T10:35:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:02.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:02.927975+0000) 2022-09-10T10:35:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:03.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:03.399174+0000) 2022-09-10T10:35:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:03 smithi110 ceph-mon[97633]: pgmap v11705: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:03 smithi104 ceph-mon[97044]: pgmap v11705: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:03.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:03.928206+0000) 2022-09-10T10:35:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:04.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:04.399391+0000) 2022-09-10T10:35:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:04.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:04.928411+0000) 2022-09-10T10:35:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:05.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:05.399736+0000) 2022-09-10T10:35:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:05 smithi110 ceph-mon[97633]: pgmap v11706: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:35:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:05 smithi104 ceph-mon[97044]: pgmap v11706: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:35:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:05.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:05.928642+0000) 2022-09-10T10:35:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:06.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:06.399966+0000) 2022-09-10T10:35:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:06.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:06.928843+0000) 2022-09-10T10:35:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:07.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:07.400193+0000) 2022-09-10T10:35:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:07 smithi110 ceph-mon[97633]: pgmap v11707: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:35:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:07 smithi104 ceph-mon[97044]: pgmap v11707: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:35:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:07.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:07.929070+0000) 2022-09-10T10:35:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:08.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:08.400451+0000) 2022-09-10T10:35:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:08 smithi110 ceph-mon[97633]: osdmap e812: 12 total, 12 up, 12 in 2022-09-10T10:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:08 smithi104 ceph-mon[97044]: osdmap e812: 12 total, 12 up, 12 in 2022-09-10T10:35:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:08.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:08.929311+0000) 2022-09-10T10:35:09.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:09.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:09.400728+0000) 2022-09-10T10:35:10.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:35:09.776+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:35:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:09.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:09.929551+0000) 2022-09-10T10:35:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:09 smithi110 ceph-mon[97633]: pgmap v11709: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:35:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:09 smithi104 ceph-mon[97044]: pgmap v11709: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:35:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:10.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:10.400921+0000) 2022-09-10T10:35:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:10.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:10.929800+0000) 2022-09-10T10:35:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:11.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:11.401123+0000) 2022-09-10T10:35:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:11 smithi104 ceph-mon[97044]: pgmap v11710: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:35:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:11.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:11.930015+0000) 2022-09-10T10:35:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:11 smithi110 ceph-mon[97633]: pgmap v11710: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:35:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:12.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:12.401293+0000) 2022-09-10T10:35:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:12.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:12.930253+0000) 2022-09-10T10:35:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:13.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:13.401550+0000) 2022-09-10T10:35:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:13 smithi110 ceph-mon[97633]: pgmap v11711: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:35:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:13 smithi110 ceph-mon[97633]: osdmap e813: 12 total, 12 up, 12 in 2022-09-10T10:35:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:13 smithi104 ceph-mon[97044]: pgmap v11711: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:35:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:13 smithi104 ceph-mon[97044]: osdmap e813: 12 total, 12 up, 12 in 2022-09-10T10:35:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:13.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:13.930437+0000) 2022-09-10T10:35:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:14.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:14.401697+0000) 2022-09-10T10:35:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:14.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:14.930616+0000) 2022-09-10T10:35:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:15.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:15.401869+0000) 2022-09-10T10:35:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:15 smithi110 ceph-mon[97633]: pgmap v11713: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:35:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:15 smithi104 ceph-mon[97044]: pgmap v11713: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:35:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:15.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:15.930847+0000) 2022-09-10T10:35:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:16.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:16.402059+0000) 2022-09-10T10:35:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:16.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:16.931091+0000) 2022-09-10T10:35:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:17.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:17.402291+0000) 2022-09-10T10:35:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:17 smithi110 ceph-mon[97633]: pgmap v11714: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:35:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:17 smithi104 ceph-mon[97044]: pgmap v11714: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:35:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:17.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:17.931312+0000) 2022-09-10T10:35:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:18.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:18.402511+0000) 2022-09-10T10:35:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:18.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:18.931512+0000) 2022-09-10T10:35:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:19.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:19.402675+0000) 2022-09-10T10:35:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:19 smithi110 ceph-mon[97633]: pgmap v11715: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:35:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:19 smithi104 ceph-mon[97044]: pgmap v11715: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:35:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:19.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:19.931705+0000) 2022-09-10T10:35:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:20.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:20.402923+0000) 2022-09-10T10:35:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:20.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:20.931930+0000) 2022-09-10T10:35:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:21.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:21.403197+0000) 2022-09-10T10:35:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:21 smithi110 ceph-mon[97633]: pgmap v11716: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:35:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:21 smithi104 ceph-mon[97044]: pgmap v11716: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:35:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:21.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:21.932183+0000) 2022-09-10T10:35:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:22.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:22.403355+0000) 2022-09-10T10:35:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:22.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:22.932397+0000) 2022-09-10T10:35:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:23.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:23.403547+0000) 2022-09-10T10:35:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:23 smithi110 ceph-mon[97633]: pgmap v11717: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:35:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:23 smithi104 ceph-mon[97044]: pgmap v11717: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:35:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:23.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:23.932623+0000) 2022-09-10T10:35:24.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:24.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:24.403764+0000) 2022-09-10T10:35:25.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:35:24.782+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:35:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:24.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:24.932833+0000) 2022-09-10T10:35:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:25.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:25.403954+0000) 2022-09-10T10:35:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:25 smithi110 ceph-mon[97633]: pgmap v11718: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:35:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:25 smithi104 ceph-mon[97044]: pgmap v11718: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:35:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:25.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:25.933058+0000) 2022-09-10T10:35:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:26.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:26.404111+0000) 2022-09-10T10:35:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:26.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:26.933265+0000) 2022-09-10T10:35:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:27.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:27.404317+0000) 2022-09-10T10:35:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:27 smithi110 ceph-mon[97633]: pgmap v11719: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:27 smithi104 ceph-mon[97044]: pgmap v11719: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:27.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:27.933548+0000) 2022-09-10T10:35:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:28.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:28.404538+0000) 2022-09-10T10:35:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:28.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:28.933770+0000) 2022-09-10T10:35:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:29.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:29.404756+0000) 2022-09-10T10:35:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:29 smithi110 ceph-mon[97633]: pgmap v11720: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:29 smithi104 ceph-mon[97044]: pgmap v11720: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:29.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:29.933962+0000) 2022-09-10T10:35:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:30.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:30.404960+0000) 2022-09-10T10:35:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:30.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:30.934199+0000) 2022-09-10T10:35:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:31.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:31.405206+0000) 2022-09-10T10:35:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:31 smithi110 ceph-mon[97633]: pgmap v11721: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:31 smithi104 ceph-mon[97044]: pgmap v11721: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:31.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:31.934385+0000) 2022-09-10T10:35:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:32.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:32.405498+0000) 2022-09-10T10:35:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:32.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:32.934579+0000) 2022-09-10T10:35:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:33.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:33.405668+0000) 2022-09-10T10:35:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:33 smithi110 ceph-mon[97633]: pgmap v11722: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:33 smithi104 ceph-mon[97044]: pgmap v11722: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:33.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:33.934815+0000) 2022-09-10T10:35:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:34.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:34.405825+0000) 2022-09-10T10:35:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:34.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:34.934968+0000) 2022-09-10T10:35:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:35.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:35.406077+0000) 2022-09-10T10:35:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:35 smithi110 ceph-mon[97633]: pgmap v11723: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:35 smithi104 ceph-mon[97044]: pgmap v11723: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:35.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:35.935176+0000) 2022-09-10T10:35:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:36.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:36.406321+0000) 2022-09-10T10:35:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:36.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:36.935374+0000) 2022-09-10T10:35:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:37.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:37.406526+0000) 2022-09-10T10:35:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:37 smithi110 ceph-mon[97633]: pgmap v11724: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:37 smithi104 ceph-mon[97044]: pgmap v11724: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:37.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:37.935527+0000) 2022-09-10T10:35:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:38.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:38.406809+0000) 2022-09-10T10:35:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:38.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:38.935726+0000) 2022-09-10T10:35:39.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:39.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:39.407066+0000) 2022-09-10T10:35:40.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:35:39.795+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:35:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:39.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:39.935983+0000) 2022-09-10T10:35:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:39 smithi110 ceph-mon[97633]: pgmap v11725: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:39 smithi104 ceph-mon[97044]: pgmap v11725: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:40.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:40.407370+0000) 2022-09-10T10:35:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:40.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:40.936155+0000) 2022-09-10T10:35:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:41.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:41.407597+0000) 2022-09-10T10:35:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:41 smithi110 ceph-mon[97633]: pgmap v11726: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:41 smithi104 ceph-mon[97044]: pgmap v11726: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:41.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:41.936382+0000) 2022-09-10T10:35:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:42.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:42.407840+0000) 2022-09-10T10:35:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:42.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:42.936617+0000) 2022-09-10T10:35:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:43.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:43.408125+0000) 2022-09-10T10:35:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:43 smithi110 ceph-mon[97633]: pgmap v11727: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:43 smithi104 ceph-mon[97044]: pgmap v11727: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:43.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:43.936839+0000) 2022-09-10T10:35:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:44.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:44.408316+0000) 2022-09-10T10:35:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:44.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:44.937064+0000) 2022-09-10T10:35:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:45.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:45.408517+0000) 2022-09-10T10:35:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:45 smithi110 ceph-mon[97633]: pgmap v11728: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:35:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:35:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:45 smithi104 ceph-mon[97044]: pgmap v11728: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:35:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:45.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:45.937324+0000) 2022-09-10T10:35:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:46.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:46.408719+0000) 2022-09-10T10:35:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:46.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:46.937539+0000) 2022-09-10T10:35:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:47.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:47.408915+0000) 2022-09-10T10:35:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:47 smithi110 ceph-mon[97633]: pgmap v11729: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:47 smithi104 ceph-mon[97044]: pgmap v11729: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:47.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:47.937817+0000) 2022-09-10T10:35:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:48.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:48.409180+0000) 2022-09-10T10:35:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:48.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:48.938046+0000) 2022-09-10T10:35:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:49.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:49.409436+0000) 2022-09-10T10:35:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:49 smithi110 ceph-mon[97633]: pgmap v11730: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:49 smithi104 ceph-mon[97044]: pgmap v11730: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:49.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:49.938183+0000) 2022-09-10T10:35:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:50.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:50.409718+0000) 2022-09-10T10:35:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:50.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:50.938414+0000) 2022-09-10T10:35:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:51.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:51.409915+0000) 2022-09-10T10:35:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:51 smithi110 ceph-mon[97633]: pgmap v11731: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:51 smithi104 ceph-mon[97044]: pgmap v11731: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:51.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:51.938614+0000) 2022-09-10T10:35:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:52.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:52.410193+0000) 2022-09-10T10:35:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:52.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:52.938866+0000) 2022-09-10T10:35:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:53.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:53.410451+0000) 2022-09-10T10:35:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:53 smithi110 ceph-mon[97633]: pgmap v11732: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:53 smithi104 ceph-mon[97044]: pgmap v11732: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:53.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:53.939175+0000) 2022-09-10T10:35:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:54.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:54.410707+0000) 2022-09-10T10:35:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:54.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:54.939449+0000) 2022-09-10T10:35:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:35:54.810+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:35:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:55.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:55.410968+0000) 2022-09-10T10:35:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:55 smithi110 ceph-mon[97633]: pgmap v11733: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:55 smithi104 ceph-mon[97044]: pgmap v11733: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:35:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:55.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:55.939681+0000) 2022-09-10T10:35:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:56.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:56.411231+0000) 2022-09-10T10:35:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:56.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:56.939873+0000) 2022-09-10T10:35:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:57.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:57.411485+0000) 2022-09-10T10:35:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:57 smithi110 ceph-mon[97633]: pgmap v11734: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:57 smithi104 ceph-mon[97044]: pgmap v11734: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:35:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:57.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:57.940142+0000) 2022-09-10T10:35:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:58.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:58.411677+0000) 2022-09-10T10:35:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:58.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:58.940296+0000) 2022-09-10T10:35:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:35:59.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:59.411965+0000) 2022-09-10T10:36:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:35:59 smithi110 ceph-mon[97633]: pgmap v11735: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:35:59 smithi104 ceph-mon[97044]: pgmap v11735: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:35:59.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:35:59.940420+0000) 2022-09-10T10:36:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:00.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:00.412179+0000) 2022-09-10T10:36:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:00.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:00.940587+0000) 2022-09-10T10:36:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:01.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:01.412348+0000) 2022-09-10T10:36:02.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:01 smithi110 ceph-mon[97633]: pgmap v11736: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:01 smithi104 ceph-mon[97044]: pgmap v11736: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:01.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:01.940820+0000) 2022-09-10T10:36:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:02.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:02.412597+0000) 2022-09-10T10:36:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:02.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:02.941066+0000) 2022-09-10T10:36:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:03.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:03.412888+0000) 2022-09-10T10:36:04.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:03 smithi110 ceph-mon[97633]: pgmap v11737: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:03 smithi104 ceph-mon[97044]: pgmap v11737: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:03.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:03.941208+0000) 2022-09-10T10:36:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:04.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:04.413205+0000) 2022-09-10T10:36:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:04.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:04.941417+0000) 2022-09-10T10:36:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:05.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:05.413462+0000) 2022-09-10T10:36:06.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:05 smithi110 ceph-mon[97633]: pgmap v11738: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:36:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:05 smithi104 ceph-mon[97044]: pgmap v11738: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:36:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:05.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:05.941653+0000) 2022-09-10T10:36:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:06.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:06.413643+0000) 2022-09-10T10:36:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:06.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:06.941904+0000) 2022-09-10T10:36:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:07.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:07.413805+0000) 2022-09-10T10:36:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:07 smithi104 ceph-mon[97044]: pgmap v11739: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:36:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:07.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:07.942182+0000) 2022-09-10T10:36:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:07 smithi110 ceph-mon[97633]: pgmap v11739: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:36:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:08.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:08.414003+0000) 2022-09-10T10:36:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:08 smithi104 ceph-mon[97044]: osdmap e814: 12 total, 12 up, 12 in 2022-09-10T10:36:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:08.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:08.942358+0000) 2022-09-10T10:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:08 smithi110 ceph-mon[97633]: osdmap e814: 12 total, 12 up, 12 in 2022-09-10T10:36:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:09.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:09.414265+0000) 2022-09-10T10:36:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:09 smithi104 ceph-mon[97044]: pgmap v11741: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:36:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:36:09.817+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:36:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:09.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:09.942557+0000) 2022-09-10T10:36:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:09 smithi110 ceph-mon[97633]: pgmap v11741: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:36:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:10.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:10.414455+0000) 2022-09-10T10:36:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:10.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:10.942787+0000) 2022-09-10T10:36:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:11.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:11.414612+0000) 2022-09-10T10:36:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:11 smithi104 ceph-mon[97044]: pgmap v11742: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:36:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:11.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:11.943038+0000) 2022-09-10T10:36:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:11 smithi110 ceph-mon[97633]: pgmap v11742: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:36:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:12.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:12.414759+0000) 2022-09-10T10:36:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:12.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:12.943248+0000) 2022-09-10T10:36:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:13.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:13.414995+0000) 2022-09-10T10:36:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:13 smithi110 ceph-mon[97633]: pgmap v11743: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:36:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:13 smithi110 ceph-mon[97633]: osdmap e815: 12 total, 12 up, 12 in 2022-09-10T10:36:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:13 smithi104 ceph-mon[97044]: pgmap v11743: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:36:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:13 smithi104 ceph-mon[97044]: osdmap e815: 12 total, 12 up, 12 in 2022-09-10T10:36:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:13.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:13.943437+0000) 2022-09-10T10:36:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:14.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:14.415228+0000) 2022-09-10T10:36:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:14.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:14.943628+0000) 2022-09-10T10:36:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:15.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:15.415494+0000) 2022-09-10T10:36:16.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:15 smithi110 ceph-mon[97633]: pgmap v11745: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:36:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:15 smithi104 ceph-mon[97044]: pgmap v11745: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:36:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:15.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:15.943844+0000) 2022-09-10T10:36:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:16.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:16.415706+0000) 2022-09-10T10:36:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:16.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:16.944152+0000) 2022-09-10T10:36:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:17.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:17.415982+0000) 2022-09-10T10:36:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:17 smithi110 ceph-mon[97633]: pgmap v11746: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:36:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:17 smithi104 ceph-mon[97044]: pgmap v11746: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:36:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:17.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:17.944325+0000) 2022-09-10T10:36:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:18.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:18.416249+0000) 2022-09-10T10:36:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:18.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:18.944562+0000) 2022-09-10T10:36:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:19.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:19.416458+0000) 2022-09-10T10:36:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:19 smithi110 ceph-mon[97633]: pgmap v11747: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:36:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:19 smithi104 ceph-mon[97044]: pgmap v11747: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:36:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:19.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:19.944717+0000) 2022-09-10T10:36:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:20.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:20.416650+0000) 2022-09-10T10:36:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:20.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:20.944984+0000) 2022-09-10T10:36:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:21.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:21.416831+0000) 2022-09-10T10:36:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:21 smithi110 ceph-mon[97633]: pgmap v11748: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:36:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:21 smithi104 ceph-mon[97044]: pgmap v11748: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:36:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:21.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:21.945240+0000) 2022-09-10T10:36:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:22.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:22.417058+0000) 2022-09-10T10:36:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:22.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:22.945401+0000) 2022-09-10T10:36:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:23.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:23.417297+0000) 2022-09-10T10:36:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:23 smithi110 ceph-mon[97633]: pgmap v11749: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:36:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:23 smithi104 ceph-mon[97044]: pgmap v11749: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:36:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:23.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:23.945619+0000) 2022-09-10T10:36:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:24.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:24.417553+0000) 2022-09-10T10:36:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:36:24.824+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:36:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:24.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:24.945874+0000) 2022-09-10T10:36:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:25.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:25.417768+0000) 2022-09-10T10:36:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:25 smithi110 ceph-mon[97633]: pgmap v11750: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:36:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:25 smithi104 ceph-mon[97044]: pgmap v11750: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:36:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:25.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:25.946069+0000) 2022-09-10T10:36:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:26.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:26.417928+0000) 2022-09-10T10:36:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:26.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:26.946272+0000) 2022-09-10T10:36:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:27.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:27.418174+0000) 2022-09-10T10:36:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:27 smithi110 ceph-mon[97633]: pgmap v11751: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:36:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:27 smithi104 ceph-mon[97044]: pgmap v11751: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:36:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:27.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:27.946473+0000) 2022-09-10T10:36:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:28.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:28.418442+0000) 2022-09-10T10:36:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:28.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:28.946677+0000) 2022-09-10T10:36:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:29.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:29.418703+0000) 2022-09-10T10:36:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:29 smithi110 ceph-mon[97633]: pgmap v11752: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:36:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:29 smithi104 ceph-mon[97044]: pgmap v11752: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:36:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:29.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:29.946946+0000) 2022-09-10T10:36:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:30.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:30.418964+0000) 2022-09-10T10:36:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:30.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:30.947222+0000) 2022-09-10T10:36:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:31.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:31.419221+0000) 2022-09-10T10:36:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:31 smithi110 ceph-mon[97633]: pgmap v11753: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:31 smithi104 ceph-mon[97044]: pgmap v11753: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:31.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:31.947465+0000) 2022-09-10T10:36:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:32.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:32.419476+0000) 2022-09-10T10:36:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:32.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:32.947642+0000) 2022-09-10T10:36:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:33.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:33.419699+0000) 2022-09-10T10:36:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:33 smithi110 ceph-mon[97633]: pgmap v11754: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:33 smithi104 ceph-mon[97044]: pgmap v11754: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:34.313 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:33.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:33.947885+0000) 2022-09-10T10:36:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:34.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:34.419970+0000) 2022-09-10T10:36:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:34.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:34.948165+0000) 2022-09-10T10:36:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:35.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:35.420249+0000) 2022-09-10T10:36:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:35 smithi110 ceph-mon[97633]: pgmap v11755: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:35 smithi104 ceph-mon[97044]: pgmap v11755: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:35.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:35.948386+0000) 2022-09-10T10:36:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:36.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:36.420510+0000) 2022-09-10T10:36:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:36.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:36.948620+0000) 2022-09-10T10:36:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:37.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:37.420741+0000) 2022-09-10T10:36:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:37 smithi110 ceph-mon[97633]: pgmap v11756: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:37 smithi104 ceph-mon[97044]: pgmap v11756: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:37.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:37.948849+0000) 2022-09-10T10:36:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:38.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:38.420998+0000) 2022-09-10T10:36:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:38.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:38.949060+0000) 2022-09-10T10:36:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:39.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:39.421257+0000) 2022-09-10T10:36:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:39 smithi110 ceph-mon[97633]: pgmap v11757: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:40.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:39 smithi104 ceph-mon[97044]: pgmap v11757: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:39.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:39.949254+0000) 2022-09-10T10:36:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:36:39.829+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:36:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:40.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:40.421451+0000) 2022-09-10T10:36:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:40.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:40.949524+0000) 2022-09-10T10:36:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:41.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:41.421643+0000) 2022-09-10T10:36:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:41 smithi110 ceph-mon[97633]: pgmap v11758: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:41 smithi104 ceph-mon[97044]: pgmap v11758: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:41.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:41.949795+0000) 2022-09-10T10:36:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:42.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:42.421868+0000) 2022-09-10T10:36:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:42.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:42.950051+0000) 2022-09-10T10:36:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:43.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:43.422155+0000) 2022-09-10T10:36:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:43 smithi110 ceph-mon[97633]: pgmap v11759: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:43 smithi104 ceph-mon[97044]: pgmap v11759: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:43.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:43.950239+0000) 2022-09-10T10:36:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:44.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:44.422316+0000) 2022-09-10T10:36:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:44.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:44.950485+0000) 2022-09-10T10:36:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:45.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:45.422579+0000) 2022-09-10T10:36:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:45 smithi110 ceph-mon[97633]: pgmap v11760: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:36:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:45 smithi104 ceph-mon[97044]: pgmap v11760: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:36:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:36:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:45.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:45.950738+0000) 2022-09-10T10:36:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:46.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:46.422828+0000) 2022-09-10T10:36:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:46.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:46.950957+0000) 2022-09-10T10:36:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:47.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:47.423078+0000) 2022-09-10T10:36:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:47 smithi110 ceph-mon[97633]: pgmap v11761: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:47 smithi104 ceph-mon[97044]: pgmap v11761: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:47.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:47.951138+0000) 2022-09-10T10:36:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:48.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:48.423362+0000) 2022-09-10T10:36:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:48.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:48.951412+0000) 2022-09-10T10:36:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:49.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:49.423593+0000) 2022-09-10T10:36:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:49 smithi110 ceph-mon[97633]: pgmap v11762: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:49 smithi104 ceph-mon[97044]: pgmap v11762: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:49.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:49.951640+0000) 2022-09-10T10:36:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:50.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:50.423871+0000) 2022-09-10T10:36:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:50.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:50.951897+0000) 2022-09-10T10:36:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:51.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:51.424049+0000) 2022-09-10T10:36:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:51 smithi110 ceph-mon[97633]: pgmap v11763: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:51 smithi104 ceph-mon[97044]: pgmap v11763: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:51.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:51.952139+0000) 2022-09-10T10:36:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:52.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:52.424241+0000) 2022-09-10T10:36:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:52.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:52.952357+0000) 2022-09-10T10:36:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:53.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:53.424489+0000) 2022-09-10T10:36:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:53 smithi110 ceph-mon[97633]: pgmap v11764: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:53 smithi104 ceph-mon[97044]: pgmap v11764: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:53.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:53.952588+0000) 2022-09-10T10:36:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:54.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:54.424707+0000) 2022-09-10T10:36:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:54.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:54.952811+0000) 2022-09-10T10:36:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:36:54.835+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:36:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:55.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:55.424936+0000) 2022-09-10T10:36:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:55 smithi110 ceph-mon[97633]: pgmap v11765: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:55 smithi104 ceph-mon[97044]: pgmap v11765: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:36:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:55.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:55.953060+0000) 2022-09-10T10:36:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:56.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:56.425217+0000) 2022-09-10T10:36:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:56.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:56.953348+0000) 2022-09-10T10:36:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:57.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:57.425425+0000) 2022-09-10T10:36:58.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:57 smithi110 ceph-mon[97633]: pgmap v11766: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:57 smithi104 ceph-mon[97044]: pgmap v11766: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:36:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:57.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:57.953587+0000) 2022-09-10T10:36:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:58.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:58.425649+0000) 2022-09-10T10:36:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:58.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:58.953773+0000) 2022-09-10T10:36:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:36:59.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:59.425845+0000) 2022-09-10T10:37:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:36:59 smithi104 ceph-mon[97044]: pgmap v11767: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:36:59.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:36:59.954013+0000) 2022-09-10T10:37:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:36:59 smithi110 ceph-mon[97633]: pgmap v11767: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:00.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:00.426030+0000) 2022-09-10T10:37:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:00.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:00.954266+0000) 2022-09-10T10:37:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:01.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:01.426186+0000) 2022-09-10T10:37:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:01 smithi104 ceph-mon[97044]: pgmap v11768: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:01.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:01.954519+0000) 2022-09-10T10:37:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:01 smithi110 ceph-mon[97633]: pgmap v11768: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:02.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:02.426417+0000) 2022-09-10T10:37:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:02.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:02.954799+0000) 2022-09-10T10:37:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:03.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:03.426658+0000) 2022-09-10T10:37:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:03 smithi104 ceph-mon[97044]: pgmap v11769: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:03.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:03.955020+0000) 2022-09-10T10:37:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:03 smithi110 ceph-mon[97633]: pgmap v11769: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:04.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:04.426876+0000) 2022-09-10T10:37:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:04.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:04.955218+0000) 2022-09-10T10:37:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:05.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:05.427120+0000) 2022-09-10T10:37:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:05 smithi104 ceph-mon[97044]: pgmap v11770: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:37:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:05.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:05.955421+0000) 2022-09-10T10:37:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:05 smithi110 ceph-mon[97633]: pgmap v11770: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:37:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:06.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:06.427335+0000) 2022-09-10T10:37:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:06.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:06.955621+0000) 2022-09-10T10:37:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:07.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:07.427564+0000) 2022-09-10T10:37:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:07 smithi104 ceph-mon[97044]: pgmap v11771: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:37:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:07.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:07.955795+0000) 2022-09-10T10:37:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:07 smithi110 ceph-mon[97633]: pgmap v11771: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:37:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:08.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:08.427828+0000) 2022-09-10T10:37:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:08 smithi104 ceph-mon[97044]: osdmap e816: 12 total, 12 up, 12 in 2022-09-10T10:37:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:08.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:08.956025+0000) 2022-09-10T10:37:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:08 smithi110 ceph-mon[97633]: osdmap e816: 12 total, 12 up, 12 in 2022-09-10T10:37:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:09.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:09.428057+0000) 2022-09-10T10:37:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:09 smithi104 ceph-mon[97044]: pgmap v11773: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:37:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:37:09.851+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:37:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:09.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:09.956213+0000) 2022-09-10T10:37:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:09 smithi110 ceph-mon[97633]: pgmap v11773: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:37:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:10.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:10.428212+0000) 2022-09-10T10:37:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:10.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:10.956392+0000) 2022-09-10T10:37:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:11.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:11.428424+0000) 2022-09-10T10:37:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:11 smithi104 ceph-mon[97044]: pgmap v11774: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:37:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:11.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:11.956626+0000) 2022-09-10T10:37:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:11 smithi110 ceph-mon[97633]: pgmap v11774: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:37:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:12.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:12.428636+0000) 2022-09-10T10:37:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:12.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:12.956863+0000) 2022-09-10T10:37:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:13.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:13.428944+0000) 2022-09-10T10:37:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:13 smithi110 ceph-mon[97633]: pgmap v11775: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:37:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:13 smithi110 ceph-mon[97633]: osdmap e817: 12 total, 12 up, 12 in 2022-09-10T10:37:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:13 smithi104 ceph-mon[97044]: pgmap v11775: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:37:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:13 smithi104 ceph-mon[97044]: osdmap e817: 12 total, 12 up, 12 in 2022-09-10T10:37:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:13.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:13.957091+0000) 2022-09-10T10:37:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:14.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:14.429203+0000) 2022-09-10T10:37:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:14.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:14.957360+0000) 2022-09-10T10:37:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:15.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:15.429401+0000) 2022-09-10T10:37:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:15 smithi110 ceph-mon[97633]: pgmap v11777: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:37:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:15 smithi104 ceph-mon[97044]: pgmap v11777: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:37:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:15.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:15.957570+0000) 2022-09-10T10:37:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:16.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:16.429638+0000) 2022-09-10T10:37:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:16.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:16.957822+0000) 2022-09-10T10:37:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:17.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:17.429895+0000) 2022-09-10T10:37:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:17 smithi110 ceph-mon[97633]: pgmap v11778: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:37:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:17 smithi104 ceph-mon[97044]: pgmap v11778: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:37:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:17.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:17.958078+0000) 2022-09-10T10:37:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:18.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:18.430184+0000) 2022-09-10T10:37:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:18.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:18.958357+0000) 2022-09-10T10:37:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:19.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:19.430444+0000) 2022-09-10T10:37:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:19 smithi110 ceph-mon[97633]: pgmap v11779: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:37:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:19 smithi104 ceph-mon[97044]: pgmap v11779: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:37:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:19.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:19.958527+0000) 2022-09-10T10:37:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:20.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:20.430602+0000) 2022-09-10T10:37:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:20.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:20.958748+0000) 2022-09-10T10:37:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:21.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:21.430842+0000) 2022-09-10T10:37:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:21 smithi110 ceph-mon[97633]: pgmap v11780: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:37:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:21 smithi104 ceph-mon[97044]: pgmap v11780: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:37:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:21.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:21.959033+0000) 2022-09-10T10:37:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:22.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:22.431042+0000) 2022-09-10T10:37:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:22.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:22.959291+0000) 2022-09-10T10:37:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:23.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:23.431212+0000) 2022-09-10T10:37:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:23 smithi110 ceph-mon[97633]: pgmap v11781: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:37:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:23 smithi104 ceph-mon[97044]: pgmap v11781: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:37:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:23.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:23.959412+0000) 2022-09-10T10:37:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:24.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:24.431425+0000) 2022-09-10T10:37:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:24.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:24.959590+0000) 2022-09-10T10:37:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:37:24.867+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:37:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:25.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:25.431617+0000) 2022-09-10T10:37:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:25 smithi110 ceph-mon[97633]: pgmap v11782: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:37:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:25 smithi104 ceph-mon[97044]: pgmap v11782: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:37:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:25.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:25.959770+0000) 2022-09-10T10:37:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:26.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:26.431870+0000) 2022-09-10T10:37:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:26.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:26.959943+0000) 2022-09-10T10:37:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:27.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:27.432076+0000) 2022-09-10T10:37:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:27 smithi110 ceph-mon[97633]: pgmap v11783: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:27 smithi104 ceph-mon[97044]: pgmap v11783: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:27.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:27.960109+0000) 2022-09-10T10:37:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:28.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:28.432331+0000) 2022-09-10T10:37:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:28.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:28.960245+0000) 2022-09-10T10:37:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:29.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:29.432559+0000) 2022-09-10T10:37:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:29 smithi110 ceph-mon[97633]: pgmap v11784: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:29 smithi104 ceph-mon[97044]: pgmap v11784: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:29.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:29.960417+0000) 2022-09-10T10:37:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:30.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:30.432789+0000) 2022-09-10T10:37:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:30.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:30.960540+0000) 2022-09-10T10:37:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:31.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:31.433027+0000) 2022-09-10T10:37:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:31 smithi110 ceph-mon[97633]: pgmap v11785: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:31 smithi104 ceph-mon[97044]: pgmap v11785: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:31.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:31.960663+0000) 2022-09-10T10:37:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:32.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:32.433278+0000) 2022-09-10T10:37:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:32.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:32.960780+0000) 2022-09-10T10:37:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:33.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:33.433513+0000) 2022-09-10T10:37:34.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:33 smithi110 ceph-mon[97633]: pgmap v11786: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:33 smithi104 ceph-mon[97044]: pgmap v11786: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:33.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:33.960901+0000) 2022-09-10T10:37:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:34.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:34.433680+0000) 2022-09-10T10:37:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:34.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:34.961124+0000) 2022-09-10T10:37:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:35.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:35.433877+0000) 2022-09-10T10:37:36.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:35 smithi110 ceph-mon[97633]: pgmap v11787: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:35 smithi104 ceph-mon[97044]: pgmap v11787: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:35.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:35.961280+0000) 2022-09-10T10:37:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:36.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:36.434137+0000) 2022-09-10T10:37:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:36.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:36.961412+0000) 2022-09-10T10:37:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:37.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:37.434365+0000) 2022-09-10T10:37:38.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:37 smithi110 ceph-mon[97633]: pgmap v11788: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:37 smithi104 ceph-mon[97044]: pgmap v11788: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:37.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:37.961542+0000) 2022-09-10T10:37:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:38.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:38.434617+0000) 2022-09-10T10:37:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:38.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:38.961722+0000) 2022-09-10T10:37:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:39.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:39.434754+0000) 2022-09-10T10:37:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:39 smithi110 ceph-mon[97633]: pgmap v11789: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:39 smithi104 ceph-mon[97044]: pgmap v11789: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:39.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:39.961897+0000) 2022-09-10T10:37:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:37:39.877+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:37:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:40.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:40.434983+0000) 2022-09-10T10:37:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:40.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:40.962019+0000) 2022-09-10T10:37:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:41.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:41.435224+0000) 2022-09-10T10:37:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:41 smithi110 ceph-mon[97633]: pgmap v11790: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:41 smithi104 ceph-mon[97044]: pgmap v11790: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:41.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:41.962203+0000) 2022-09-10T10:37:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:42.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:42.435366+0000) 2022-09-10T10:37:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:42.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:42.962344+0000) 2022-09-10T10:37:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:43.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:43.435597+0000) 2022-09-10T10:37:44.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:43 smithi110 ceph-mon[97633]: pgmap v11791: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:43 smithi104 ceph-mon[97044]: pgmap v11791: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:43.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:43.962459+0000) 2022-09-10T10:37:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:44.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:44.435795+0000) 2022-09-10T10:37:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:44.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:44.962581+0000) 2022-09-10T10:37:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:45.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:45.435973+0000) 2022-09-10T10:37:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:45 smithi110 ceph-mon[97633]: pgmap v11792: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:37:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:37:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:37:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:37:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:45 smithi104 ceph-mon[97044]: pgmap v11792: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:37:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:45.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:45.962704+0000) 2022-09-10T10:37:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:46.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:46.436211+0000) 2022-09-10T10:37:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:46.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:46.962826+0000) 2022-09-10T10:37:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:47.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:47.436396+0000) 2022-09-10T10:37:48.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:47 smithi110 ceph-mon[97633]: pgmap v11793: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:47 smithi104 ceph-mon[97044]: pgmap v11793: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:47.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:47.962948+0000) 2022-09-10T10:37:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:48.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:48.436607+0000) 2022-09-10T10:37:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:48.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:48.963105+0000) 2022-09-10T10:37:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:49.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:49.436765+0000) 2022-09-10T10:37:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:49 smithi104 ceph-mon[97044]: pgmap v11794: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:49.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:49.963229+0000) 2022-09-10T10:37:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:49 smithi110 ceph-mon[97633]: pgmap v11794: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:50.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:50.437024+0000) 2022-09-10T10:37:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:50.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:50.963349+0000) 2022-09-10T10:37:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:51.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:51.437173+0000) 2022-09-10T10:37:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:51 smithi104 ceph-mon[97044]: pgmap v11795: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:51.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:51.963498+0000) 2022-09-10T10:37:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:51 smithi110 ceph-mon[97633]: pgmap v11795: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:52.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:52.437335+0000) 2022-09-10T10:37:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:52.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:52.963621+0000) 2022-09-10T10:37:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:53.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:53.437593+0000) 2022-09-10T10:37:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:53 smithi104 ceph-mon[97044]: pgmap v11796: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:53.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:53.963796+0000) 2022-09-10T10:37:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:53 smithi110 ceph-mon[97633]: pgmap v11796: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:54.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:54.437769+0000) 2022-09-10T10:37:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:54.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:54.963937+0000) 2022-09-10T10:37:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:37:54.892+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:37:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:55.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:55.438013+0000) 2022-09-10T10:37:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:55 smithi104 ceph-mon[97044]: pgmap v11797: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:55.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:55.964063+0000) 2022-09-10T10:37:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:55 smithi110 ceph-mon[97633]: pgmap v11797: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:37:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:56.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:56.438259+0000) 2022-09-10T10:37:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:56.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:56.964220+0000) 2022-09-10T10:37:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:57.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:57.438498+0000) 2022-09-10T10:37:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:57 smithi104 ceph-mon[97044]: pgmap v11798: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:57.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:57.964411+0000) 2022-09-10T10:37:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:57 smithi110 ceph-mon[97633]: pgmap v11798: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:37:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:58.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:58.438754+0000) 2022-09-10T10:37:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:58.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:58.964535+0000) 2022-09-10T10:37:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:37:59.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:59.438982+0000) 2022-09-10T10:38:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:37:59 smithi104 ceph-mon[97044]: pgmap v11799: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:37:59.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:37:59.964657+0000) 2022-09-10T10:38:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:37:59 smithi110 ceph-mon[97633]: pgmap v11799: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:00.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:00.439190+0000) 2022-09-10T10:38:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:00.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:00.964830+0000) 2022-09-10T10:38:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:01.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:01.439398+0000) 2022-09-10T10:38:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:01 smithi104 ceph-mon[97044]: pgmap v11800: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:01.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:01.965026+0000) 2022-09-10T10:38:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:01 smithi110 ceph-mon[97633]: pgmap v11800: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:02.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:02.439626+0000) 2022-09-10T10:38:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:02.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:02.965165+0000) 2022-09-10T10:38:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:03.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:03.439881+0000) 2022-09-10T10:38:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:03 smithi104 ceph-mon[97044]: pgmap v11801: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:38:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:03.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:03.965359+0000) 2022-09-10T10:38:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:03 smithi110 ceph-mon[97633]: pgmap v11801: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:38:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:04.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:04.440169+0000) 2022-09-10T10:38:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:04.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:04.965493+0000) 2022-09-10T10:38:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:05.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:05.440414+0000) 2022-09-10T10:38:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:05 smithi104 ceph-mon[97044]: pgmap v11802: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:38:06.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:05.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:05.965627+0000) 2022-09-10T10:38:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:05 smithi110 ceph-mon[97633]: pgmap v11802: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:38:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:06.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:06.440667+0000) 2022-09-10T10:38:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:06.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:06.965774+0000) 2022-09-10T10:38:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:07.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:07.440882+0000) 2022-09-10T10:38:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:07 smithi104 ceph-mon[97044]: pgmap v11803: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:38:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:07.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:07.965947+0000) 2022-09-10T10:38:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:07 smithi110 ceph-mon[97633]: pgmap v11803: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:38:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:08.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:08.441134+0000) 2022-09-10T10:38:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:08 smithi104 ceph-mon[97044]: osdmap e818: 12 total, 12 up, 12 in 2022-09-10T10:38:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:08.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:08.966069+0000) 2022-09-10T10:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:08 smithi110 ceph-mon[97633]: osdmap e818: 12 total, 12 up, 12 in 2022-09-10T10:38:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:09.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:09.441333+0000) 2022-09-10T10:38:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:09 smithi104 ceph-mon[97044]: pgmap v11805: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:38:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:38:09.904+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:38:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:09.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:09.966172+0000) 2022-09-10T10:38:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:09 smithi110 ceph-mon[97633]: pgmap v11805: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:38:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:10.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:10.441555+0000) 2022-09-10T10:38:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:10.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:10.966296+0000) 2022-09-10T10:38:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:11.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:11.441807+0000) 2022-09-10T10:38:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:11 smithi104 ceph-mon[97044]: pgmap v11806: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:38:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:11.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:11.966466+0000) 2022-09-10T10:38:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:11 smithi110 ceph-mon[97633]: pgmap v11806: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:38:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:12.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:12.442058+0000) 2022-09-10T10:38:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:12.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:12.966633+0000) 2022-09-10T10:38:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:13.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:13.442350+0000) 2022-09-10T10:38:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:13 smithi110 ceph-mon[97633]: pgmap v11807: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:38:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:13 smithi110 ceph-mon[97633]: osdmap e819: 12 total, 12 up, 12 in 2022-09-10T10:38:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:13 smithi104 ceph-mon[97044]: pgmap v11807: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:38:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:13 smithi104 ceph-mon[97044]: osdmap e819: 12 total, 12 up, 12 in 2022-09-10T10:38:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:13.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:13.966800+0000) 2022-09-10T10:38:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:14.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:14.442541+0000) 2022-09-10T10:38:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:14.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:14.966932+0000) 2022-09-10T10:38:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:15.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:15.442700+0000) 2022-09-10T10:38:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:15 smithi110 ceph-mon[97633]: pgmap v11809: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:38:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:15 smithi104 ceph-mon[97044]: pgmap v11809: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:38:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:15.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:15.967073+0000) 2022-09-10T10:38:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:16.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:16.442943+0000) 2022-09-10T10:38:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:16.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:16.967191+0000) 2022-09-10T10:38:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:17.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:17.443177+0000) 2022-09-10T10:38:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:17 smithi110 ceph-mon[97633]: pgmap v11810: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:38:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:17 smithi104 ceph-mon[97044]: pgmap v11810: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:38:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:17.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:17.967340+0000) 2022-09-10T10:38:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:18.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:18.443378+0000) 2022-09-10T10:38:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:18.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:18.967489+0000) 2022-09-10T10:38:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:19.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:19.443632+0000) 2022-09-10T10:38:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:19 smithi110 ceph-mon[97633]: pgmap v11811: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:38:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:19 smithi104 ceph-mon[97044]: pgmap v11811: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:38:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:19.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:19.967612+0000) 2022-09-10T10:38:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:20.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:20.443857+0000) 2022-09-10T10:38:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:20.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:20.967727+0000) 2022-09-10T10:38:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:21.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:21.444061+0000) 2022-09-10T10:38:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:21 smithi110 ceph-mon[97633]: pgmap v11812: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:38:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:21 smithi104 ceph-mon[97044]: pgmap v11812: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:38:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:21.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:21.967877+0000) 2022-09-10T10:38:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:22.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:22.444340+0000) 2022-09-10T10:38:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:22.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:22.968050+0000) 2022-09-10T10:38:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:23.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:23.444562+0000) 2022-09-10T10:38:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:23 smithi110 ceph-mon[97633]: pgmap v11813: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:38:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:23 smithi104 ceph-mon[97044]: pgmap v11813: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:38:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:23.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:23.968208+0000) 2022-09-10T10:38:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:24.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:24.444817+0000) 2022-09-10T10:38:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:24.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:24.968398+0000) 2022-09-10T10:38:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:38:24.912+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:38:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:25.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:25.445051+0000) 2022-09-10T10:38:26.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:25 smithi110 ceph-mon[97633]: pgmap v11814: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:38:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:25 smithi104 ceph-mon[97044]: pgmap v11814: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:38:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:25.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:25.968534+0000) 2022-09-10T10:38:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:26.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:26.445275+0000) 2022-09-10T10:38:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:26.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:26.968670+0000) 2022-09-10T10:38:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:27.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:27.445508+0000) 2022-09-10T10:38:28.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:27 smithi110 ceph-mon[97633]: pgmap v11815: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:27 smithi104 ceph-mon[97044]: pgmap v11815: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:27.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:27.968848+0000) 2022-09-10T10:38:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:28.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:28.445651+0000) 2022-09-10T10:38:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:28.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:28.969121+0000) 2022-09-10T10:38:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:29.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:29.445809+0000) 2022-09-10T10:38:30.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:29 smithi110 ceph-mon[97633]: pgmap v11816: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:29 smithi104 ceph-mon[97044]: pgmap v11816: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:29.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:29.969280+0000) 2022-09-10T10:38:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:30.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:30.446015+0000) 2022-09-10T10:38:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:30.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:30.969405+0000) 2022-09-10T10:38:31.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:31.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:31.446226+0000) 2022-09-10T10:38:32.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:31 smithi110 ceph-mon[97633]: pgmap v11817: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:31 smithi104 ceph-mon[97044]: pgmap v11817: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:31.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:31.969534+0000) 2022-09-10T10:38:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:32.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:32.446415+0000) 2022-09-10T10:38:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:32.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:32.969663+0000) 2022-09-10T10:38:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:33.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:33.446565+0000) 2022-09-10T10:38:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:33 smithi104 ceph-mon[97044]: pgmap v11818: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:33.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:33.969837+0000) 2022-09-10T10:38:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:33 smithi110 ceph-mon[97633]: pgmap v11818: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:34.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:34.446807+0000) 2022-09-10T10:38:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:34.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:34.970039+0000) 2022-09-10T10:38:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:35.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:35.447088+0000) 2022-09-10T10:38:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:35 smithi104 ceph-mon[97044]: pgmap v11819: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:35.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:35.970151+0000) 2022-09-10T10:38:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:35 smithi110 ceph-mon[97633]: pgmap v11819: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:36.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:36.447340+0000) 2022-09-10T10:38:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:36.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:36.970314+0000) 2022-09-10T10:38:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:37.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:37.447505+0000) 2022-09-10T10:38:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:37 smithi104 ceph-mon[97044]: pgmap v11820: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:37.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:37.970462+0000) 2022-09-10T10:38:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:37 smithi110 ceph-mon[97633]: pgmap v11820: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:38.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:38.447710+0000) 2022-09-10T10:38:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:38.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:38.970606+0000) 2022-09-10T10:38:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:39.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:39.447921+0000) 2022-09-10T10:38:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:39 smithi104 ceph-mon[97044]: pgmap v11821: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:39.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:39.970767+0000) 2022-09-10T10:38:40.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:38:39.923+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:38:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:39 smithi110 ceph-mon[97633]: pgmap v11821: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:40.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:40.448170+0000) 2022-09-10T10:38:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:40.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:40.970901+0000) 2022-09-10T10:38:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:41.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:41.448362+0000) 2022-09-10T10:38:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:41 smithi104 ceph-mon[97044]: pgmap v11822: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:41.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:41.971037+0000) 2022-09-10T10:38:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:41 smithi110 ceph-mon[97633]: pgmap v11822: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:42.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:42.448565+0000) 2022-09-10T10:38:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:42.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:42.971169+0000) 2022-09-10T10:38:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:43.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:43.448824+0000) 2022-09-10T10:38:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:43 smithi104 ceph-mon[97044]: pgmap v11823: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:44.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:43.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:43.971347+0000) 2022-09-10T10:38:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:43 smithi110 ceph-mon[97633]: pgmap v11823: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:44.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:44.449065+0000) 2022-09-10T10:38:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:44.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:44.971503+0000) 2022-09-10T10:38:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:45.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:45.449278+0000) 2022-09-10T10:38:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:45 smithi104 ceph-mon[97044]: pgmap v11824: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:38:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:45.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:45.971632+0000) 2022-09-10T10:38:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:45 smithi110 ceph-mon[97633]: pgmap v11824: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:38:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:38:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:38:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:38:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:46.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:46.449482+0000) 2022-09-10T10:38:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:46.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:46.971802+0000) 2022-09-10T10:38:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:47.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:47.449738+0000) 2022-09-10T10:38:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:47 smithi104 ceph-mon[97044]: pgmap v11825: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:47.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:47.971981+0000) 2022-09-10T10:38:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:47 smithi110 ceph-mon[97633]: pgmap v11825: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:48.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:48.449930+0000) 2022-09-10T10:38:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:48.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:48.972135+0000) 2022-09-10T10:38:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:49.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:49.450119+0000) 2022-09-10T10:38:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:49 smithi104 ceph-mon[97044]: pgmap v11826: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:49.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:49.972290+0000) 2022-09-10T10:38:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:49 smithi110 ceph-mon[97633]: pgmap v11826: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:50.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:50.450272+0000) 2022-09-10T10:38:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:50.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:50.972407+0000) 2022-09-10T10:38:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:51.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:51.450430+0000) 2022-09-10T10:38:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:51 smithi104 ceph-mon[97044]: pgmap v11827: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:51.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:51.972530+0000) 2022-09-10T10:38:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:51 smithi110 ceph-mon[97633]: pgmap v11827: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:52.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:52.450629+0000) 2022-09-10T10:38:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:52.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:52.972649+0000) 2022-09-10T10:38:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:53.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:53.450906+0000) 2022-09-10T10:38:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:53 smithi104 ceph-mon[97044]: pgmap v11828: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:53.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:53.972859+0000) 2022-09-10T10:38:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:53 smithi110 ceph-mon[97633]: pgmap v11828: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:54.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:54.451170+0000) 2022-09-10T10:38:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:54.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:54.972994+0000) 2022-09-10T10:38:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:38:54.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:38:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:55.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:55.451343+0000) 2022-09-10T10:38:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:55 smithi104 ceph-mon[97044]: pgmap v11829: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:55.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:55.973136+0000) 2022-09-10T10:38:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:55 smithi110 ceph-mon[97633]: pgmap v11829: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:38:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:56.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:56.451509+0000) 2022-09-10T10:38:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:56.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:56.973271+0000) 2022-09-10T10:38:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:57.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:57.451645+0000) 2022-09-10T10:38:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:57 smithi104 ceph-mon[97044]: pgmap v11830: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:57.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:57.973405+0000) 2022-09-10T10:38:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:57 smithi110 ceph-mon[97633]: pgmap v11830: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:38:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:58.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:58.451840+0000) 2022-09-10T10:38:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:58.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:58.973525+0000) 2022-09-10T10:38:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:38:59.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:59.452089+0000) 2022-09-10T10:39:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:38:59 smithi104 ceph-mon[97044]: pgmap v11831: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:38:59.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:38:59.973705+0000) 2022-09-10T10:39:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:38:59 smithi110 ceph-mon[97633]: pgmap v11831: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:00.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:00.452344+0000) 2022-09-10T10:39:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:00.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:00.973831+0000) 2022-09-10T10:39:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:01.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:01.452501+0000) 2022-09-10T10:39:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:01 smithi104 ceph-mon[97044]: pgmap v11832: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:01.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:01.974012+0000) 2022-09-10T10:39:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:01 smithi110 ceph-mon[97633]: pgmap v11832: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:02.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:02.452723+0000) 2022-09-10T10:39:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:02.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:02.974173+0000) 2022-09-10T10:39:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:03.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:03.453021+0000) 2022-09-10T10:39:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:03 smithi104 ceph-mon[97044]: pgmap v11833: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:03.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:03.974307+0000) 2022-09-10T10:39:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:03 smithi110 ceph-mon[97633]: pgmap v11833: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:04.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:04.453244+0000) 2022-09-10T10:39:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:04.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:04.974447+0000) 2022-09-10T10:39:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:05.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:05.453438+0000) 2022-09-10T10:39:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:05 smithi104 ceph-mon[97044]: pgmap v11834: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:39:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:05.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:05.974574+0000) 2022-09-10T10:39:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:05 smithi110 ceph-mon[97633]: pgmap v11834: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:39:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:06.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:06.453683+0000) 2022-09-10T10:39:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:06.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:06.974766+0000) 2022-09-10T10:39:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:07.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:07.453956+0000) 2022-09-10T10:39:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:07 smithi104 ceph-mon[97044]: pgmap v11835: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:39:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:07.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:07.974983+0000) 2022-09-10T10:39:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:07 smithi110 ceph-mon[97633]: pgmap v11835: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:39:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:08.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:08.454167+0000) 2022-09-10T10:39:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:08 smithi104 ceph-mon[97044]: osdmap e820: 12 total, 12 up, 12 in 2022-09-10T10:39:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:08.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:08.975125+0000) 2022-09-10T10:39:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:08 smithi110 ceph-mon[97633]: osdmap e820: 12 total, 12 up, 12 in 2022-09-10T10:39:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:09.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:09.454413+0000) 2022-09-10T10:39:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:09 smithi104 ceph-mon[97044]: pgmap v11837: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:39:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:09.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:09.975250+0000) 2022-09-10T10:39:10.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:39:09.955+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:39:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:09 smithi110 ceph-mon[97633]: pgmap v11837: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:39:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:10.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:10.454673+0000) 2022-09-10T10:39:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:10.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:10.975371+0000) 2022-09-10T10:39:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:11.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:11.454876+0000) 2022-09-10T10:39:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:11 smithi104 ceph-mon[97044]: pgmap v11838: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:39:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:11.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:11.975499+0000) 2022-09-10T10:39:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:11 smithi110 ceph-mon[97633]: pgmap v11838: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:39:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:12.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:12.455136+0000) 2022-09-10T10:39:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:12.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:12.975634+0000) 2022-09-10T10:39:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:13.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:13.455382+0000) 2022-09-10T10:39:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:13 smithi110 ceph-mon[97633]: pgmap v11839: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:39:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:13 smithi110 ceph-mon[97633]: osdmap e821: 12 total, 12 up, 12 in 2022-09-10T10:39:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:13 smithi104 ceph-mon[97044]: pgmap v11839: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:39:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:13 smithi104 ceph-mon[97044]: osdmap e821: 12 total, 12 up, 12 in 2022-09-10T10:39:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:13.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:13.975758+0000) 2022-09-10T10:39:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:14.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:14.455565+0000) 2022-09-10T10:39:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:14.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:14.975883+0000) 2022-09-10T10:39:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:15.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:15.455786+0000) 2022-09-10T10:39:16.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:15 smithi110 ceph-mon[97633]: pgmap v11841: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:39:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:15 smithi104 ceph-mon[97044]: pgmap v11841: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:39:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:15.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:15.975998+0000) 2022-09-10T10:39:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:16.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:16.456012+0000) 2022-09-10T10:39:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:16.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:16.976158+0000) 2022-09-10T10:39:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:17.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:17.456245+0000) 2022-09-10T10:39:18.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:17 smithi110 ceph-mon[97633]: pgmap v11842: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:39:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:17 smithi104 ceph-mon[97044]: pgmap v11842: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:39:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:17.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:17.976300+0000) 2022-09-10T10:39:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:18.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:18.456456+0000) 2022-09-10T10:39:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:18.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:18.976421+0000) 2022-09-10T10:39:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:19.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:19.456689+0000) 2022-09-10T10:39:20.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:19 smithi110 ceph-mon[97633]: pgmap v11843: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:39:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:19 smithi104 ceph-mon[97044]: pgmap v11843: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:39:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:19.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:19.976548+0000) 2022-09-10T10:39:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:20.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:20.456940+0000) 2022-09-10T10:39:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:20.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:20.976675+0000) 2022-09-10T10:39:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:21.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:21.457133+0000) 2022-09-10T10:39:22.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:21 smithi110 ceph-mon[97633]: pgmap v11844: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:39:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:21 smithi104 ceph-mon[97044]: pgmap v11844: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:39:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:21.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:21.976851+0000) 2022-09-10T10:39:22.812 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:22.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:22.457380+0000) 2022-09-10T10:39:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:22.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:22.976999+0000) 2022-09-10T10:39:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:23.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:23.457600+0000) 2022-09-10T10:39:24.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:23 smithi110 ceph-mon[97633]: pgmap v11845: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:39:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:23 smithi104 ceph-mon[97044]: pgmap v11845: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:39:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:23.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:23.977137+0000) 2022-09-10T10:39:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:24.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:24.457883+0000) 2022-09-10T10:39:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:39:24.970+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:39:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:24.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:24.977266+0000) 2022-09-10T10:39:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:25.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:25.458085+0000) 2022-09-10T10:39:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:25 smithi104 ceph-mon[97044]: pgmap v11846: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:39:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:25.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:25.977384+0000) 2022-09-10T10:39:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:25 smithi110 ceph-mon[97633]: pgmap v11846: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:39:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:26.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:26.458295+0000) 2022-09-10T10:39:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:26.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:26.977528+0000) 2022-09-10T10:39:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:27.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:27.458515+0000) 2022-09-10T10:39:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:27 smithi104 ceph-mon[97044]: pgmap v11847: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:39:28.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:27.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:27.977657+0000) 2022-09-10T10:39:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:27 smithi110 ceph-mon[97633]: pgmap v11847: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:39:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:28.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:28.458745+0000) 2022-09-10T10:39:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:28.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:28.977856+0000) 2022-09-10T10:39:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:29.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:29.458916+0000) 2022-09-10T10:39:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:29 smithi104 ceph-mon[97044]: pgmap v11848: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:39:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:29.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:29.977986+0000) 2022-09-10T10:39:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:29 smithi110 ceph-mon[97633]: pgmap v11848: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:39:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:30.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:30.459181+0000) 2022-09-10T10:39:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:30.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:30.978151+0000) 2022-09-10T10:39:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:31.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:31.459391+0000) 2022-09-10T10:39:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:31 smithi104 ceph-mon[97044]: pgmap v11849: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:31.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:31.978309+0000) 2022-09-10T10:39:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:31 smithi110 ceph-mon[97633]: pgmap v11849: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:32.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:32.459639+0000) 2022-09-10T10:39:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:32.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:32.978489+0000) 2022-09-10T10:39:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:33.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:33.459868+0000) 2022-09-10T10:39:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:33 smithi104 ceph-mon[97044]: pgmap v11850: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:33.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:33.978653+0000) 2022-09-10T10:39:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:33 smithi110 ceph-mon[97633]: pgmap v11850: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:34.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:34.460092+0000) 2022-09-10T10:39:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:34.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:34.978781+0000) 2022-09-10T10:39:35.812 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:35.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:35.460299+0000) 2022-09-10T10:39:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:35 smithi104 ceph-mon[97044]: pgmap v11851: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:35.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:35.978898+0000) 2022-09-10T10:39:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:35 smithi110 ceph-mon[97633]: pgmap v11851: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:36.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:36.460510+0000) 2022-09-10T10:39:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:36.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:36.979109+0000) 2022-09-10T10:39:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:37.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:37.460692+0000) 2022-09-10T10:39:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:37 smithi104 ceph-mon[97044]: pgmap v11852: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:37.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:37.979244+0000) 2022-09-10T10:39:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:37 smithi110 ceph-mon[97633]: pgmap v11852: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:38.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:38.460928+0000) 2022-09-10T10:39:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:38.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:38.979370+0000) 2022-09-10T10:39:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:39.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:39.461125+0000) 2022-09-10T10:39:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:39 smithi104 ceph-mon[97044]: pgmap v11853: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:39.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:39.979581+0000) 2022-09-10T10:39:40.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:39:39.978+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:39:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:39 smithi110 ceph-mon[97633]: pgmap v11853: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:40.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:40.461329+0000) 2022-09-10T10:39:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:40.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:40.979705+0000) 2022-09-10T10:39:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:41.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:41.461502+0000) 2022-09-10T10:39:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:41 smithi104 ceph-mon[97044]: pgmap v11854: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:41.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:41.979831+0000) 2022-09-10T10:39:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:41 smithi110 ceph-mon[97633]: pgmap v11854: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:42.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:42.461700+0000) 2022-09-10T10:39:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:42.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:42.979961+0000) 2022-09-10T10:39:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:43.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:43.461933+0000) 2022-09-10T10:39:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:43 smithi104 ceph-mon[97044]: pgmap v11855: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:43.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:43.980083+0000) 2022-09-10T10:39:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:43 smithi110 ceph-mon[97633]: pgmap v11855: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:44.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:44.462193+0000) 2022-09-10T10:39:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:44.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:44.980300+0000) 2022-09-10T10:39:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:45.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:45.462406+0000) 2022-09-10T10:39:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:45 smithi104 ceph-mon[97044]: pgmap v11856: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:39:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:39:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:39:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:39:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:45.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:45.980450+0000) 2022-09-10T10:39:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:45 smithi110 ceph-mon[97633]: pgmap v11856: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:39:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:39:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:39:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:39:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:46.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:46.462568+0000) 2022-09-10T10:39:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:46.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:46.980617+0000) 2022-09-10T10:39:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:47.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:47.462730+0000) 2022-09-10T10:39:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:47 smithi104 ceph-mon[97044]: pgmap v11857: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:47.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:47.980753+0000) 2022-09-10T10:39:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:47 smithi110 ceph-mon[97633]: pgmap v11857: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:48.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:48.462925+0000) 2022-09-10T10:39:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:48.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:48.980882+0000) 2022-09-10T10:39:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:49.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:49.463185+0000) 2022-09-10T10:39:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:49 smithi104 ceph-mon[97044]: pgmap v11858: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:49.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:49.981015+0000) 2022-09-10T10:39:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:49 smithi110 ceph-mon[97633]: pgmap v11858: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:50.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:50.463324+0000) 2022-09-10T10:39:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:50.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:50.981149+0000) 2022-09-10T10:39:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:51.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:51.463480+0000) 2022-09-10T10:39:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:51 smithi104 ceph-mon[97044]: pgmap v11859: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:51.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:51.981282+0000) 2022-09-10T10:39:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:51 smithi110 ceph-mon[97633]: pgmap v11859: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:52.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:52.463654+0000) 2022-09-10T10:39:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:52.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:52.981414+0000) 2022-09-10T10:39:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:53.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:53.463833+0000) 2022-09-10T10:39:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:53 smithi104 ceph-mon[97044]: pgmap v11860: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:53.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:53.981547+0000) 2022-09-10T10:39:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:53 smithi110 ceph-mon[97633]: pgmap v11860: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:54.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:54.464071+0000) 2022-09-10T10:39:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:54.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:54.981692+0000) 2022-09-10T10:39:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:39:54.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:39:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:55.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:55.464343+0000) 2022-09-10T10:39:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:55 smithi104 ceph-mon[97044]: pgmap v11861: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:55.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:55.981878+0000) 2022-09-10T10:39:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:55 smithi110 ceph-mon[97633]: pgmap v11861: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:39:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:56.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:56.464569+0000) 2022-09-10T10:39:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:56.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:56.982015+0000) 2022-09-10T10:39:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:57.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:57.464828+0000) 2022-09-10T10:39:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:57 smithi104 ceph-mon[97044]: pgmap v11862: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:57.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:57.982219+0000) 2022-09-10T10:39:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:57 smithi110 ceph-mon[97633]: pgmap v11862: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:39:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:58.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:58.465023+0000) 2022-09-10T10:39:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:58.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:58.982359+0000) 2022-09-10T10:39:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:39:59.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:59.465262+0000) 2022-09-10T10:40:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:39:59 smithi104 ceph-mon[97044]: pgmap v11863: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:39:59.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:39:59.982485+0000) 2022-09-10T10:40:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:39:59 smithi110 ceph-mon[97633]: pgmap v11863: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:00.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:00.465539+0000) 2022-09-10T10:40:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:40:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:00.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:00.982605+0000) 2022-09-10T10:40:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:40:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:01.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:01.465789+0000) 2022-09-10T10:40:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:01 smithi104 ceph-mon[97044]: pgmap v11864: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:01.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:01.982773+0000) 2022-09-10T10:40:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:01 smithi110 ceph-mon[97633]: pgmap v11864: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:02.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:02.465994+0000) 2022-09-10T10:40:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:02.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:02.982915+0000) 2022-09-10T10:40:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:03.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:03.466206+0000) 2022-09-10T10:40:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:03 smithi104 ceph-mon[97044]: pgmap v11865: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:03.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:03.983034+0000) 2022-09-10T10:40:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:03 smithi110 ceph-mon[97633]: pgmap v11865: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:04.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:04.466453+0000) 2022-09-10T10:40:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:04.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:04.983164+0000) 2022-09-10T10:40:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:05.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:05.466673+0000) 2022-09-10T10:40:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:05 smithi104 ceph-mon[97044]: pgmap v11866: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:40:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:05.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:05.983377+0000) 2022-09-10T10:40:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:05 smithi110 ceph-mon[97633]: pgmap v11866: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:40:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:06.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:06.466901+0000) 2022-09-10T10:40:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:06.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:06.983501+0000) 2022-09-10T10:40:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:07.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:07.467119+0000) 2022-09-10T10:40:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:07 smithi104 ceph-mon[97044]: pgmap v11867: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:40:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:07.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:07.983656+0000) 2022-09-10T10:40:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:07 smithi110 ceph-mon[97633]: pgmap v11867: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:40:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:08.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:08.467367+0000) 2022-09-10T10:40:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:08 smithi104 ceph-mon[97044]: osdmap e822: 12 total, 12 up, 12 in 2022-09-10T10:40:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:08.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:08.983844+0000) 2022-09-10T10:40:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:08 smithi110 ceph-mon[97633]: osdmap e822: 12 total, 12 up, 12 in 2022-09-10T10:40:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:09.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:09.467598+0000) 2022-09-10T10:40:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:09 smithi104 ceph-mon[97044]: pgmap v11869: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:40:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:40:09.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:40:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:09.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:09.984003+0000) 2022-09-10T10:40:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:09 smithi110 ceph-mon[97633]: pgmap v11869: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:40:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:10.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:10.467848+0000) 2022-09-10T10:40:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:10 smithi104 ceph-mon[97044]: pgmap v11870: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:40:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:10.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:10.984163+0000) 2022-09-10T10:40:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:10 smithi110 ceph-mon[97633]: pgmap v11870: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:40:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:11.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:11.468140+0000) 2022-09-10T10:40:12.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:11.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:11.984329+0000) 2022-09-10T10:40:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:12.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:12.468406+0000) 2022-09-10T10:40:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:12.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:12.984457+0000) 2022-09-10T10:40:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:13.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:13.468683+0000) 2022-09-10T10:40:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:13 smithi104 ceph-mon[97044]: pgmap v11871: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:40:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:13 smithi104 ceph-mon[97044]: osdmap e823: 12 total, 12 up, 12 in 2022-09-10T10:40:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:13.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:13.984581+0000) 2022-09-10T10:40:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:13 smithi110 ceph-mon[97633]: pgmap v11871: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:40:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:13 smithi110 ceph-mon[97633]: osdmap e823: 12 total, 12 up, 12 in 2022-09-10T10:40:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:14.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:14.468893+0000) 2022-09-10T10:40:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:14.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:14.984736+0000) 2022-09-10T10:40:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:15.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:15.469160+0000) 2022-09-10T10:40:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:15 smithi104 ceph-mon[97044]: pgmap v11873: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:40:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:15.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:15.984900+0000) 2022-09-10T10:40:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:15 smithi110 ceph-mon[97633]: pgmap v11873: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:40:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:16.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:16.469357+0000) 2022-09-10T10:40:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:16.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:16.985064+0000) 2022-09-10T10:40:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:17.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:17.469599+0000) 2022-09-10T10:40:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:17 smithi104 ceph-mon[97044]: pgmap v11874: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:40:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:17.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:17.985168+0000) 2022-09-10T10:40:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:17 smithi110 ceph-mon[97633]: pgmap v11874: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:40:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:18.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:18.469876+0000) 2022-09-10T10:40:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:18.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:18.985330+0000) 2022-09-10T10:40:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:19.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:19.470124+0000) 2022-09-10T10:40:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:19 smithi104 ceph-mon[97044]: pgmap v11875: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:40:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:19.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:19.985556+0000) 2022-09-10T10:40:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:19 smithi110 ceph-mon[97633]: pgmap v11875: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:40:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:20.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:20.470315+0000) 2022-09-10T10:40:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:20.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:20.985679+0000) 2022-09-10T10:40:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:21.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:21.470515+0000) 2022-09-10T10:40:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:21 smithi104 ceph-mon[97044]: pgmap v11876: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:40:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:21.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:21.985832+0000) 2022-09-10T10:40:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:21 smithi110 ceph-mon[97633]: pgmap v11876: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:40:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:22.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:22.470721+0000) 2022-09-10T10:40:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:22.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:22.985966+0000) 2022-09-10T10:40:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:23.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:23.470947+0000) 2022-09-10T10:40:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:23 smithi104 ceph-mon[97044]: pgmap v11877: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:40:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:23.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:23.986092+0000) 2022-09-10T10:40:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:23 smithi110 ceph-mon[97633]: pgmap v11877: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:40:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:24.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:24.471174+0000) 2022-09-10T10:40:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:24.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:24.986284+0000) 2022-09-10T10:40:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:40:25.011+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:40:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:25.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:25.471441+0000) 2022-09-10T10:40:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:25 smithi104 ceph-mon[97044]: pgmap v11878: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:40:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:25.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:25.986419+0000) 2022-09-10T10:40:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:25 smithi110 ceph-mon[97633]: pgmap v11878: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:40:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:26.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:26.471667+0000) 2022-09-10T10:40:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:26.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:26.986542+0000) 2022-09-10T10:40:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:27.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:27.471865+0000) 2022-09-10T10:40:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:27 smithi104 ceph-mon[97044]: pgmap v11879: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:27.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:27.986675+0000) 2022-09-10T10:40:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:27 smithi110 ceph-mon[97633]: pgmap v11879: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:28.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:28.472077+0000) 2022-09-10T10:40:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:28.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:28.986807+0000) 2022-09-10T10:40:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:29.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:29.472257+0000) 2022-09-10T10:40:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:29 smithi104 ceph-mon[97044]: pgmap v11880: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:29.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:29.987027+0000) 2022-09-10T10:40:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:29 smithi110 ceph-mon[97633]: pgmap v11880: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:30.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:30.472494+0000) 2022-09-10T10:40:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:30.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:30.987167+0000) 2022-09-10T10:40:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:31.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:31.472778+0000) 2022-09-10T10:40:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:31 smithi104 ceph-mon[97044]: pgmap v11881: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:31.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:31.987295+0000) 2022-09-10T10:40:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:31 smithi110 ceph-mon[97633]: pgmap v11881: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:32.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:32.473051+0000) 2022-09-10T10:40:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:32.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:32.987419+0000) 2022-09-10T10:40:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:33.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:33.473358+0000) 2022-09-10T10:40:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:33 smithi104 ceph-mon[97044]: pgmap v11882: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:34.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:33.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:33.987576+0000) 2022-09-10T10:40:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:33 smithi110 ceph-mon[97633]: pgmap v11882: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:34.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:34.473595+0000) 2022-09-10T10:40:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:34.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:34.987712+0000) 2022-09-10T10:40:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:35.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:35.473814+0000) 2022-09-10T10:40:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:35 smithi104 ceph-mon[97044]: pgmap v11883: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:35.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:35.987876+0000) 2022-09-10T10:40:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:35 smithi110 ceph-mon[97633]: pgmap v11883: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:36.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:36.474082+0000) 2022-09-10T10:40:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:36.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:36.988023+0000) 2022-09-10T10:40:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:37.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:37.474252+0000) 2022-09-10T10:40:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:37 smithi104 ceph-mon[97044]: pgmap v11884: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:37.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:37.988156+0000) 2022-09-10T10:40:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:37 smithi110 ceph-mon[97633]: pgmap v11884: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:38.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:38.474486+0000) 2022-09-10T10:40:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:38.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:38.988281+0000) 2022-09-10T10:40:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:39.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:39.474729+0000) 2022-09-10T10:40:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:39 smithi104 ceph-mon[97044]: pgmap v11885: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:39.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:39.988485+0000) 2022-09-10T10:40:40.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:40:40.017+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:40:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:39 smithi110 ceph-mon[97633]: pgmap v11885: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:40.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:40.474925+0000) 2022-09-10T10:40:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:40.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:40.988608+0000) 2022-09-10T10:40:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:41.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:41.475220+0000) 2022-09-10T10:40:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:41 smithi104 ceph-mon[97044]: pgmap v11886: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:41.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:41.988775+0000) 2022-09-10T10:40:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:41 smithi110 ceph-mon[97633]: pgmap v11886: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:42.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:42.475384+0000) 2022-09-10T10:40:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:42.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:42.988918+0000) 2022-09-10T10:40:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:43.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:43.475646+0000) 2022-09-10T10:40:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:43 smithi104 ceph-mon[97044]: pgmap v11887: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:43.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:43.989038+0000) 2022-09-10T10:40:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:43 smithi110 ceph-mon[97633]: pgmap v11887: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:44.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:44.475934+0000) 2022-09-10T10:40:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:44.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:44.989209+0000) 2022-09-10T10:40:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:45.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:45.476136+0000) 2022-09-10T10:40:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:45 smithi104 ceph-mon[97044]: pgmap v11888: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:40:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:45.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:45.989333+0000) 2022-09-10T10:40:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:45 smithi110 ceph-mon[97633]: pgmap v11888: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:40:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:46.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:46.476314+0000) 2022-09-10T10:40:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:46.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:46.989464+0000) 2022-09-10T10:40:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:47.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:47.476446+0000) 2022-09-10T10:40:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:47 smithi104 ceph-mon[97044]: pgmap v11889: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:47.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:47.989590+0000) 2022-09-10T10:40:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:47 smithi110 ceph-mon[97633]: pgmap v11889: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:48.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:48.476574+0000) 2022-09-10T10:40:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:48.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:48.989717+0000) 2022-09-10T10:40:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:49.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:49.476801+0000) 2022-09-10T10:40:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:49 smithi104 ceph-mon[97044]: pgmap v11890: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:49.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:49.989896+0000) 2022-09-10T10:40:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:49 smithi110 ceph-mon[97633]: pgmap v11890: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:50.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:50.476994+0000) 2022-09-10T10:40:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:50.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:50.990047+0000) 2022-09-10T10:40:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:51.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:51.477237+0000) 2022-09-10T10:40:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:51 smithi104 ceph-mon[97044]: pgmap v11891: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:51.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:51.990197+0000) 2022-09-10T10:40:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:51 smithi110 ceph-mon[97633]: pgmap v11891: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:52.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:52.477504+0000) 2022-09-10T10:40:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:52.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:52.990412+0000) 2022-09-10T10:40:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:53.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:53.477663+0000) 2022-09-10T10:40:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:53 smithi104 ceph-mon[97044]: pgmap v11892: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:53.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:53.990572+0000) 2022-09-10T10:40:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:53 smithi110 ceph-mon[97633]: pgmap v11892: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:54.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:54.477843+0000) 2022-09-10T10:40:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:40:55.033+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:40:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:54.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:54.990747+0000) 2022-09-10T10:40:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:55.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:55.478126+0000) 2022-09-10T10:40:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:55 smithi104 ceph-mon[97044]: pgmap v11893: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:55.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:55.990896+0000) 2022-09-10T10:40:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:55 smithi110 ceph-mon[97633]: pgmap v11893: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:40:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:56.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:56.478363+0000) 2022-09-10T10:40:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:56.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:56.991043+0000) 2022-09-10T10:40:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:57.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:57.478591+0000) 2022-09-10T10:40:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:57 smithi104 ceph-mon[97044]: pgmap v11894: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:57.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:57.991171+0000) 2022-09-10T10:40:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:57 smithi110 ceph-mon[97633]: pgmap v11894: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:40:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:58.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:58.478872+0000) 2022-09-10T10:40:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:58.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:58.991306+0000) 2022-09-10T10:40:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:40:59.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:59.479052+0000) 2022-09-10T10:41:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:40:59 smithi104 ceph-mon[97044]: pgmap v11895: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:40:59.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:40:59.991444+0000) 2022-09-10T10:41:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:40:59 smithi110 ceph-mon[97633]: pgmap v11895: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:00.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:00.479253+0000) 2022-09-10T10:41:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:00.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:00.991572+0000) 2022-09-10T10:41:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:01.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:01.479466+0000) 2022-09-10T10:41:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:01 smithi104 ceph-mon[97044]: pgmap v11896: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:01.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:01.991706+0000) 2022-09-10T10:41:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:01 smithi110 ceph-mon[97633]: pgmap v11896: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:02.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:02.479719+0000) 2022-09-10T10:41:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:02.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:02.991841+0000) 2022-09-10T10:41:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:03.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:03.479991+0000) 2022-09-10T10:41:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:03 smithi104 ceph-mon[97044]: pgmap v11897: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:03.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:03.991973+0000) 2022-09-10T10:41:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:03 smithi110 ceph-mon[97633]: pgmap v11897: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:04.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:04.480185+0000) 2022-09-10T10:41:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:04.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:04.992121+0000) 2022-09-10T10:41:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:05.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:05.480378+0000) 2022-09-10T10:41:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:05 smithi104 ceph-mon[97044]: pgmap v11898: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:41:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:05.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:05.992257+0000) 2022-09-10T10:41:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:05 smithi110 ceph-mon[97633]: pgmap v11898: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:41:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:06.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:06.480585+0000) 2022-09-10T10:41:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:06.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:06.992395+0000) 2022-09-10T10:41:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:07.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:07.480809+0000) 2022-09-10T10:41:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:07 smithi104 ceph-mon[97044]: pgmap v11899: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:41:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:07.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:07.992532+0000) 2022-09-10T10:41:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:07 smithi110 ceph-mon[97633]: pgmap v11899: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:41:08.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:08.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:08.481049+0000) 2022-09-10T10:41:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:08 smithi104 ceph-mon[97044]: osdmap e824: 12 total, 12 up, 12 in 2022-09-10T10:41:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:08.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:08.992700+0000) 2022-09-10T10:41:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:08 smithi110 ceph-mon[97633]: osdmap e824: 12 total, 12 up, 12 in 2022-09-10T10:41:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:09.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:09.481275+0000) 2022-09-10T10:41:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:09 smithi104 ceph-mon[97044]: pgmap v11901: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:41:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:09.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:09.992908+0000) 2022-09-10T10:41:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:41:10.048+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:41:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:09 smithi110 ceph-mon[97633]: pgmap v11901: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:41:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:10.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:10.481526+0000) 2022-09-10T10:41:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:11 smithi104 ceph-mon[97044]: pgmap v11902: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:41:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:10.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:10.993064+0000) 2022-09-10T10:41:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:11 smithi110 ceph-mon[97633]: pgmap v11902: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:41:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:11.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:11.481790+0000) 2022-09-10T10:41:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:11.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:11.993167+0000) 2022-09-10T10:41:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:12.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:12.482051+0000) 2022-09-10T10:41:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:12.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:12.993304+0000) 2022-09-10T10:41:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:13.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:13.482357+0000) 2022-09-10T10:41:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:13 smithi104 ceph-mon[97044]: pgmap v11903: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:41:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:13 smithi104 ceph-mon[97044]: osdmap e825: 12 total, 12 up, 12 in 2022-09-10T10:41:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:13.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:13.993482+0000) 2022-09-10T10:41:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:13 smithi110 ceph-mon[97633]: pgmap v11903: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:41:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:13 smithi110 ceph-mon[97633]: osdmap e825: 12 total, 12 up, 12 in 2022-09-10T10:41:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:14.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:14.482620+0000) 2022-09-10T10:41:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:14.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:14.993640+0000) 2022-09-10T10:41:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:15.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:15.482833+0000) 2022-09-10T10:41:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:15 smithi104 ceph-mon[97044]: pgmap v11905: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:41:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:15.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:15.993791+0000) 2022-09-10T10:41:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:15 smithi110 ceph-mon[97633]: pgmap v11905: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:41:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:16.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:16.483017+0000) 2022-09-10T10:41:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:16.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:16.993932+0000) 2022-09-10T10:41:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:17.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:17.483209+0000) 2022-09-10T10:41:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:17 smithi104 ceph-mon[97044]: pgmap v11906: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:41:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:17.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:17.994079+0000) 2022-09-10T10:41:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:17 smithi110 ceph-mon[97633]: pgmap v11906: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:41:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:18.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:18.483443+0000) 2022-09-10T10:41:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:18.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:18.994227+0000) 2022-09-10T10:41:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:19.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:19.483690+0000) 2022-09-10T10:41:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:19 smithi104 ceph-mon[97044]: pgmap v11907: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:41:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:19.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:19.994468+0000) 2022-09-10T10:41:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:19 smithi110 ceph-mon[97633]: pgmap v11907: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:41:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:20.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:20.483931+0000) 2022-09-10T10:41:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:20.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:20.994602+0000) 2022-09-10T10:41:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:21.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:21.484149+0000) 2022-09-10T10:41:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:21 smithi104 ceph-mon[97044]: pgmap v11908: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:41:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:21.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:21.994772+0000) 2022-09-10T10:41:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:21 smithi110 ceph-mon[97633]: pgmap v11908: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:41:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:22.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:22.484373+0000) 2022-09-10T10:41:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:22.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:22.994910+0000) 2022-09-10T10:41:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:23.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:23.484541+0000) 2022-09-10T10:41:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:23 smithi104 ceph-mon[97044]: pgmap v11909: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:41:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:23.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:23.995043+0000) 2022-09-10T10:41:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:23 smithi110 ceph-mon[97633]: pgmap v11909: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T10:41:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:24.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:24.484819+0000) 2022-09-10T10:41:25.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:41:25.064+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:41:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:24.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:24.995197+0000) 2022-09-10T10:41:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:25.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:25.485052+0000) 2022-09-10T10:41:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:25 smithi104 ceph-mon[97044]: pgmap v11910: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:41:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:25.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:25.995320+0000) 2022-09-10T10:41:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:25 smithi110 ceph-mon[97633]: pgmap v11910: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:41:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:26.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:26.485333+0000) 2022-09-10T10:41:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:26.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:26.995461+0000) 2022-09-10T10:41:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:27.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:27.485630+0000) 2022-09-10T10:41:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:27 smithi104 ceph-mon[97044]: pgmap v11911: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:27.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:27.995603+0000) 2022-09-10T10:41:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:27 smithi110 ceph-mon[97633]: pgmap v11911: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:28.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:28.485830+0000) 2022-09-10T10:41:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:28.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:28.995765+0000) 2022-09-10T10:41:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:29.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:29.485965+0000) 2022-09-10T10:41:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:29 smithi104 ceph-mon[97044]: pgmap v11912: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:29.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:29.995903+0000) 2022-09-10T10:41:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:29 smithi110 ceph-mon[97633]: pgmap v11912: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:30.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:30.486142+0000) 2022-09-10T10:41:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:30.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:30.996037+0000) 2022-09-10T10:41:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:31.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:31.486281+0000) 2022-09-10T10:41:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:31 smithi104 ceph-mon[97044]: pgmap v11913: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:31.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:31.996163+0000) 2022-09-10T10:41:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:31 smithi110 ceph-mon[97633]: pgmap v11913: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:32.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:32.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:32.486526+0000) 2022-09-10T10:41:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:32.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:32.996301+0000) 2022-09-10T10:41:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:33.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:33.486681+0000) 2022-09-10T10:41:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:33 smithi104 ceph-mon[97044]: pgmap v11914: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:33.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:33.996423+0000) 2022-09-10T10:41:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:33 smithi110 ceph-mon[97633]: pgmap v11914: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:34.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:34.486856+0000) 2022-09-10T10:41:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:34.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:34.996555+0000) 2022-09-10T10:41:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:35.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:35.487128+0000) 2022-09-10T10:41:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:35 smithi104 ceph-mon[97044]: pgmap v11915: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:35.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:35.996670+0000) 2022-09-10T10:41:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:35 smithi110 ceph-mon[97633]: pgmap v11915: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:36.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:36.487279+0000) 2022-09-10T10:41:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:36.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:36.996796+0000) 2022-09-10T10:41:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:37.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:37.487442+0000) 2022-09-10T10:41:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:37 smithi104 ceph-mon[97044]: pgmap v11916: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:37.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:37.996929+0000) 2022-09-10T10:41:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:37 smithi110 ceph-mon[97633]: pgmap v11916: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:38.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:38.487659+0000) 2022-09-10T10:41:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:38.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:38.997055+0000) 2022-09-10T10:41:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:39.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:39.487839+0000) 2022-09-10T10:41:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:39 smithi104 ceph-mon[97044]: pgmap v11917: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:41:40.080+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:41:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:39.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:39.997287+0000) 2022-09-10T10:41:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:39 smithi110 ceph-mon[97633]: pgmap v11917: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:40.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:40.488085+0000) 2022-09-10T10:41:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:40.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:40.997434+0000) 2022-09-10T10:41:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:41.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:41.488419+0000) 2022-09-10T10:41:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:41 smithi104 ceph-mon[97044]: pgmap v11918: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:41.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:41.997627+0000) 2022-09-10T10:41:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:41 smithi110 ceph-mon[97633]: pgmap v11918: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:42.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:42.488617+0000) 2022-09-10T10:41:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:42.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:42.997795+0000) 2022-09-10T10:41:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:43.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:43.488813+0000) 2022-09-10T10:41:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:43 smithi104 ceph-mon[97044]: pgmap v11919: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:43.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:43.997982+0000) 2022-09-10T10:41:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:43 smithi110 ceph-mon[97633]: pgmap v11919: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:44.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:44.489137+0000) 2022-09-10T10:41:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:44.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:44.998215+0000) 2022-09-10T10:41:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:45.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:45.489357+0000) 2022-09-10T10:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:45 smithi104 ceph-mon[97044]: pgmap v11920: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:41:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:41:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:41:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:41:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:45.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:45.998356+0000) 2022-09-10T10:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:45 smithi110 ceph-mon[97633]: pgmap v11920: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:41:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:41:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:41:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:41:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:46.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:46.489615+0000) 2022-09-10T10:41:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:46.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:46.998496+0000) 2022-09-10T10:41:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:47.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:47.489804+0000) 2022-09-10T10:41:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:47 smithi104 ceph-mon[97044]: pgmap v11921: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:47.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:47.998660+0000) 2022-09-10T10:41:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:47 smithi110 ceph-mon[97633]: pgmap v11921: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:48.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:48.490034+0000) 2022-09-10T10:41:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:48.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:48.998809+0000) 2022-09-10T10:41:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:49.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:49.490219+0000) 2022-09-10T10:41:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:49 smithi104 ceph-mon[97044]: pgmap v11922: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:49.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:49.998945+0000) 2022-09-10T10:41:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:49 smithi110 ceph-mon[97633]: pgmap v11922: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:50.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:50.490364+0000) 2022-09-10T10:41:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:50.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:50.999075+0000) 2022-09-10T10:41:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:51.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:51.490592+0000) 2022-09-10T10:41:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:51 smithi104 ceph-mon[97044]: pgmap v11923: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:51.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:51.999228+0000) 2022-09-10T10:41:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:51 smithi110 ceph-mon[97633]: pgmap v11923: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:52.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:52.490781+0000) 2022-09-10T10:41:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:52.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:52.999381+0000) 2022-09-10T10:41:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:53.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:53.491030+0000) 2022-09-10T10:41:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:53 smithi104 ceph-mon[97044]: pgmap v11924: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:53.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:53.999557+0000) 2022-09-10T10:41:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:53 smithi110 ceph-mon[97633]: pgmap v11924: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:54.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:54.491162+0000) 2022-09-10T10:41:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:54.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:54.999693+0000) 2022-09-10T10:41:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:41:55.095+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:41:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:55.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:55.491377+0000) 2022-09-10T10:41:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:55 smithi104 ceph-mon[97044]: pgmap v11925: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:55.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:55.999821+0000) 2022-09-10T10:41:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:55 smithi110 ceph-mon[97633]: pgmap v11925: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:41:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:56.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:56.491628+0000) 2022-09-10T10:41:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:56.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:56.999961+0000) 2022-09-10T10:41:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:57.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:57.491831+0000) 2022-09-10T10:41:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:57 smithi104 ceph-mon[97044]: pgmap v11926: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:57.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:58.000125+0000) 2022-09-10T10:41:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:57 smithi110 ceph-mon[97633]: pgmap v11926: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:41:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:58.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:58.492032+0000) 2022-09-10T10:41:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:58.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:59.000263+0000) 2022-09-10T10:41:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:41:59.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:41:59.492258+0000) 2022-09-10T10:42:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:41:59 smithi104 ceph-mon[97044]: pgmap v11927: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:41:59.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:00.000402+0000) 2022-09-10T10:42:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:41:59 smithi110 ceph-mon[97633]: pgmap v11927: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:00.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:00.492421+0000) 2022-09-10T10:42:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:00.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:01.000529+0000) 2022-09-10T10:42:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:01.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:01.492646+0000) 2022-09-10T10:42:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:01 smithi104 ceph-mon[97044]: pgmap v11928: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:01.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:02.000715+0000) 2022-09-10T10:42:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:01 smithi110 ceph-mon[97633]: pgmap v11928: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:02.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:02.492896+0000) 2022-09-10T10:42:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:02.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:03.000904+0000) 2022-09-10T10:42:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:03.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:03.493184+0000) 2022-09-10T10:42:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:03 smithi104 ceph-mon[97044]: pgmap v11929: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:03.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:04.001040+0000) 2022-09-10T10:42:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:03 smithi110 ceph-mon[97633]: pgmap v11929: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:04.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:04.493436+0000) 2022-09-10T10:42:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:05.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:05.001214+0000) 2022-09-10T10:42:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:05.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:05.493702+0000) 2022-09-10T10:42:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:05 smithi104 ceph-mon[97044]: pgmap v11930: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:42:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:06.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:06.001348+0000) 2022-09-10T10:42:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:05 smithi110 ceph-mon[97633]: pgmap v11930: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:42:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:06.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:06.493890+0000) 2022-09-10T10:42:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:07.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:07.001486+0000) 2022-09-10T10:42:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:07.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:07.494176+0000) 2022-09-10T10:42:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:07 smithi104 ceph-mon[97044]: pgmap v11931: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:42:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:08.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:08.001638+0000) 2022-09-10T10:42:08.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:07 smithi110 ceph-mon[97633]: pgmap v11931: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:42:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:08.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:08.494413+0000) 2022-09-10T10:42:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:08 smithi104 ceph-mon[97044]: osdmap e826: 12 total, 12 up, 12 in 2022-09-10T10:42:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:09.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:09.001772+0000) 2022-09-10T10:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:08 smithi110 ceph-mon[97633]: osdmap e826: 12 total, 12 up, 12 in 2022-09-10T10:42:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:09.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:09.494601+0000) 2022-09-10T10:42:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:09 smithi104 ceph-mon[97044]: pgmap v11933: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:42:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:10.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:10.001907+0000) 2022-09-10T10:42:10.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:42:10.111+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:42:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:09 smithi110 ceph-mon[97633]: pgmap v11933: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:42:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:10.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:10.494829+0000) 2022-09-10T10:42:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:11.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:11.002042+0000) 2022-09-10T10:42:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:11.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:11.495053+0000) 2022-09-10T10:42:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:11 smithi104 ceph-mon[97044]: pgmap v11934: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:42:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:12.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:12.002170+0000) 2022-09-10T10:42:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:11 smithi110 ceph-mon[97633]: pgmap v11934: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:42:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:12.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:12.495229+0000) 2022-09-10T10:42:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:13.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:13.002326+0000) 2022-09-10T10:42:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:13.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:13.495436+0000) 2022-09-10T10:42:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:13 smithi104 ceph-mon[97044]: pgmap v11935: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:42:14.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:13 smithi104 ceph-mon[97044]: osdmap e827: 12 total, 12 up, 12 in 2022-09-10T10:42:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:14.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:14.002525+0000) 2022-09-10T10:42:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:13 smithi110 ceph-mon[97633]: pgmap v11935: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:42:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:13 smithi110 ceph-mon[97633]: osdmap e827: 12 total, 12 up, 12 in 2022-09-10T10:42:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:14.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:14.495681+0000) 2022-09-10T10:42:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:15.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:15.002688+0000) 2022-09-10T10:42:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:15.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:15.495881+0000) 2022-09-10T10:42:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:15 smithi104 ceph-mon[97044]: pgmap v11937: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:42:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:16.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:16.002851+0000) 2022-09-10T10:42:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:15 smithi110 ceph-mon[97633]: pgmap v11937: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:42:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:16.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:16.496057+0000) 2022-09-10T10:42:17.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:17.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:17.002994+0000) 2022-09-10T10:42:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:17.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:17.496282+0000) 2022-09-10T10:42:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:17 smithi104 ceph-mon[97044]: pgmap v11938: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 227 B/s wr, 11 op/s 2022-09-10T10:42:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:18.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:18.003173+0000) 2022-09-10T10:42:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:17 smithi110 ceph-mon[97633]: pgmap v11938: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 227 B/s wr, 11 op/s 2022-09-10T10:42:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:18.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:18.496481+0000) 2022-09-10T10:42:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:19.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:19.003295+0000) 2022-09-10T10:42:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:19.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:19.496689+0000) 2022-09-10T10:42:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:19 smithi104 ceph-mon[97044]: pgmap v11939: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:42:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:20.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:20.003412+0000) 2022-09-10T10:42:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:19 smithi110 ceph-mon[97633]: pgmap v11939: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:42:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:20.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:20.496917+0000) 2022-09-10T10:42:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:21.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:21.003523+0000) 2022-09-10T10:42:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:21.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:21.497172+0000) 2022-09-10T10:42:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:21 smithi104 ceph-mon[97044]: pgmap v11940: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:42:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:22.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:22.003682+0000) 2022-09-10T10:42:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:21 smithi110 ceph-mon[97633]: pgmap v11940: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:42:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:22.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:22.497430+0000) 2022-09-10T10:42:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:23.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:23.003811+0000) 2022-09-10T10:42:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:23.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:23.497637+0000) 2022-09-10T10:42:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:23 smithi104 ceph-mon[97044]: pgmap v11941: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:42:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:24.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:24.003950+0000) 2022-09-10T10:42:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:23 smithi110 ceph-mon[97633]: pgmap v11941: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:42:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:24.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:24.497822+0000) 2022-09-10T10:42:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:25.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:25.004073+0000) 2022-09-10T10:42:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:42:25.120+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:42:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:25.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:25.498015+0000) 2022-09-10T10:42:26.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:25 smithi104 ceph-mon[97044]: pgmap v11942: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:42:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:26.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:26.004223+0000) 2022-09-10T10:42:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:25 smithi110 ceph-mon[97633]: pgmap v11942: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:42:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:26.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:26.498212+0000) 2022-09-10T10:42:27.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:27.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:27.004348+0000) 2022-09-10T10:42:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:27.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:27.498391+0000) 2022-09-10T10:42:28.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:27 smithi104 ceph-mon[97044]: pgmap v11943: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:28.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:28.004475+0000) 2022-09-10T10:42:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:27 smithi110 ceph-mon[97633]: pgmap v11943: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:28.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:28.498633+0000) 2022-09-10T10:42:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:29.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:29.004627+0000) 2022-09-10T10:42:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:29.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:29.498802+0000) 2022-09-10T10:42:30.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:29 smithi104 ceph-mon[97044]: pgmap v11944: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:30.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:30.004750+0000) 2022-09-10T10:42:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:29 smithi110 ceph-mon[97633]: pgmap v11944: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:30.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:30.499057+0000) 2022-09-10T10:42:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:31.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:31.004872+0000) 2022-09-10T10:42:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:31.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:31.499336+0000) 2022-09-10T10:42:32.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:31 smithi104 ceph-mon[97044]: pgmap v11945: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:32.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:32.004999+0000) 2022-09-10T10:42:32.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:31 smithi110 ceph-mon[97633]: pgmap v11945: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:32.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:32.499501+0000) 2022-09-10T10:42:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:33.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:33.005136+0000) 2022-09-10T10:42:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:33.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:33.499715+0000) 2022-09-10T10:42:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:33 smithi104 ceph-mon[97044]: pgmap v11946: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:34.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:34.005259+0000) 2022-09-10T10:42:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:33 smithi110 ceph-mon[97633]: pgmap v11946: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:34.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:34.499936+0000) 2022-09-10T10:42:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:35.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:35.005384+0000) 2022-09-10T10:42:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:35.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:35.500232+0000) 2022-09-10T10:42:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:35 smithi104 ceph-mon[97044]: pgmap v11947: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:36.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:36.005499+0000) 2022-09-10T10:42:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:35 smithi110 ceph-mon[97633]: pgmap v11947: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:36.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:36.500431+0000) 2022-09-10T10:42:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:37.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:37.005634+0000) 2022-09-10T10:42:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:37.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:37.500635+0000) 2022-09-10T10:42:38.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:37 smithi104 ceph-mon[97044]: pgmap v11948: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:38.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:38.005761+0000) 2022-09-10T10:42:38.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:37 smithi110 ceph-mon[97633]: pgmap v11948: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:38.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:38.500855+0000) 2022-09-10T10:42:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:39.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:39.005886+0000) 2022-09-10T10:42:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:39.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:39.501061+0000) 2022-09-10T10:42:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:39 smithi104 ceph-mon[97044]: pgmap v11949: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:42:40.124+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:42:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:40.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:40.006034+0000) 2022-09-10T10:42:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:39 smithi110 ceph-mon[97633]: pgmap v11949: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:40.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:40.501299+0000) 2022-09-10T10:42:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:41.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:41.006163+0000) 2022-09-10T10:42:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:41.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:41.501547+0000) 2022-09-10T10:42:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:41 smithi104 ceph-mon[97044]: pgmap v11950: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:42.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:42.006331+0000) 2022-09-10T10:42:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:41 smithi110 ceph-mon[97633]: pgmap v11950: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:42.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:42.501758+0000) 2022-09-10T10:42:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:43.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:43.006464+0000) 2022-09-10T10:42:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:43.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:43.502009+0000) 2022-09-10T10:42:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:43 smithi104 ceph-mon[97044]: pgmap v11951: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:44.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:44.006594+0000) 2022-09-10T10:42:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:43 smithi110 ceph-mon[97633]: pgmap v11951: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:44.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:44.502272+0000) 2022-09-10T10:42:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:45.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:45.006732+0000) 2022-09-10T10:42:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:45.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:45.502497+0000) 2022-09-10T10:42:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:45 smithi104 ceph-mon[97044]: pgmap v11952: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:42:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:42:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:46.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:46.006902+0000) 2022-09-10T10:42:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:45 smithi110 ceph-mon[97633]: pgmap v11952: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:42:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:42:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:42:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:42:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:46.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:46.502706+0000) 2022-09-10T10:42:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:47.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:47.007046+0000) 2022-09-10T10:42:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:47.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:47.502885+0000) 2022-09-10T10:42:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:47 smithi104 ceph-mon[97044]: pgmap v11953: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:48.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:48.007163+0000) 2022-09-10T10:42:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:47 smithi110 ceph-mon[97633]: pgmap v11953: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:48.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:48.503081+0000) 2022-09-10T10:42:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:49.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:49.007298+0000) 2022-09-10T10:42:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:49.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:49.503236+0000) 2022-09-10T10:42:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:49 smithi104 ceph-mon[97044]: pgmap v11954: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:50.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:50.007430+0000) 2022-09-10T10:42:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:49 smithi110 ceph-mon[97633]: pgmap v11954: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:50.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:50.503456+0000) 2022-09-10T10:42:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:51.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:51.007612+0000) 2022-09-10T10:42:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:51.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:51.503703+0000) 2022-09-10T10:42:52.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:51 smithi104 ceph-mon[97044]: pgmap v11955: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:52.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:52.007782+0000) 2022-09-10T10:42:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:51 smithi110 ceph-mon[97633]: pgmap v11955: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:52.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:52.503929+0000) 2022-09-10T10:42:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:53.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:53.007998+0000) 2022-09-10T10:42:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:53.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:53.504178+0000) 2022-09-10T10:42:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:53 smithi104 ceph-mon[97044]: pgmap v11956: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:54.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:54.008166+0000) 2022-09-10T10:42:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:53 smithi110 ceph-mon[97633]: pgmap v11956: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:54.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:54.504359+0000) 2022-09-10T10:42:55.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:42:55.128+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:42:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:55.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:55.008296+0000) 2022-09-10T10:42:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:55.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:55.504531+0000) 2022-09-10T10:42:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:55 smithi104 ceph-mon[97044]: pgmap v11957: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:56.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:56.008425+0000) 2022-09-10T10:42:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:55 smithi110 ceph-mon[97633]: pgmap v11957: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:42:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:56.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:56.504655+0000) 2022-09-10T10:42:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:57.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:57.008620+0000) 2022-09-10T10:42:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:57.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:57.504883+0000) 2022-09-10T10:42:58.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:57 smithi104 ceph-mon[97044]: pgmap v11958: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:58.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:58.008838+0000) 2022-09-10T10:42:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:57 smithi110 ceph-mon[97633]: pgmap v11958: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:42:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:58.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:58.505053+0000) 2022-09-10T10:42:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:42:59.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:59.009014+0000) 2022-09-10T10:42:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:42:59.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:42:59.505260+0000) 2022-09-10T10:43:00.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:42:59 smithi104 ceph-mon[97044]: pgmap v11959: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:00.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:00.009188+0000) 2022-09-10T10:43:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:42:59 smithi110 ceph-mon[97633]: pgmap v11959: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:00.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:00.505421+0000) 2022-09-10T10:43:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:01.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:01.009309+0000) 2022-09-10T10:43:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:01.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:01.505633+0000) 2022-09-10T10:43:02.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:01 smithi104 ceph-mon[97044]: pgmap v11960: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:02.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:02.009474+0000) 2022-09-10T10:43:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:01 smithi110 ceph-mon[97633]: pgmap v11960: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:02.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:02.505891+0000) 2022-09-10T10:43:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:03.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:03.009658+0000) 2022-09-10T10:43:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:03.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:03.506184+0000) 2022-09-10T10:43:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:03 smithi104 ceph-mon[97044]: pgmap v11961: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:04.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:04.009832+0000) 2022-09-10T10:43:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:03 smithi110 ceph-mon[97633]: pgmap v11961: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:04.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:04.506392+0000) 2022-09-10T10:43:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:05.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:05.009964+0000) 2022-09-10T10:43:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:05.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:05.506655+0000) 2022-09-10T10:43:06.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:06 smithi104 ceph-mon[97044]: pgmap v11962: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:43:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:06.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:06.010082+0000) 2022-09-10T10:43:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:06 smithi110 ceph-mon[97633]: pgmap v11962: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:43:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:06.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:06.506868+0000) 2022-09-10T10:43:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:07.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:07.010295+0000) 2022-09-10T10:43:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:07 smithi110 ceph-mon[97633]: pgmap v11963: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:43:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:07 smithi104 ceph-mon[97044]: pgmap v11963: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:43:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:07.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:07.507090+0000) 2022-09-10T10:43:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:08.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:08.010472+0000) 2022-09-10T10:43:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:08.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:08.507344+0000) 2022-09-10T10:43:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:09.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:09.010600+0000) 2022-09-10T10:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:09 smithi110 ceph-mon[97633]: osdmap e828: 12 total, 12 up, 12 in 2022-09-10T10:43:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:09 smithi110 ceph-mon[97633]: pgmap v11965: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:43:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:09 smithi104 ceph-mon[97044]: osdmap e828: 12 total, 12 up, 12 in 2022-09-10T10:43:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:09 smithi104 ceph-mon[97044]: pgmap v11965: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:43:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:09.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:09.507547+0000) 2022-09-10T10:43:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:10.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:10.010735+0000) 2022-09-10T10:43:10.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:43:10.144+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:43:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:10.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:10.507722+0000) 2022-09-10T10:43:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:11.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:11.010861+0000) 2022-09-10T10:43:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:11.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:11.507897+0000) 2022-09-10T10:43:12.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:12 smithi104 ceph-mon[97044]: pgmap v11966: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:43:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:12.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:12.011043+0000) 2022-09-10T10:43:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:12 smithi110 ceph-mon[97633]: pgmap v11966: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:43:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:12.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:12.508181+0000) 2022-09-10T10:43:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:13.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:13.011225+0000) 2022-09-10T10:43:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:13.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:13.508367+0000) 2022-09-10T10:43:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:13 smithi104 ceph-mon[97044]: osdmap e829: 12 total, 12 up, 12 in 2022-09-10T10:43:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:13 smithi104 ceph-mon[97044]: pgmap v11968: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:43:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:14.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:14.011355+0000) 2022-09-10T10:43:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:13 smithi110 ceph-mon[97633]: osdmap e829: 12 total, 12 up, 12 in 2022-09-10T10:43:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:13 smithi110 ceph-mon[97633]: pgmap v11968: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:43:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:14.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:14.508587+0000) 2022-09-10T10:43:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:15.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:15.011542+0000) 2022-09-10T10:43:15.813 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:15.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:15.508857+0000) 2022-09-10T10:43:16.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:15 smithi104 ceph-mon[97044]: pgmap v11969: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:43:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:16.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:16.011691+0000) 2022-09-10T10:43:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:15 smithi110 ceph-mon[97633]: pgmap v11969: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:43:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:16.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:16.509131+0000) 2022-09-10T10:43:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:17.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:17.011878+0000) 2022-09-10T10:43:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:17.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:17.509338+0000) 2022-09-10T10:43:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:17 smithi104 ceph-mon[97044]: pgmap v11970: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:43:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:18.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:18.012070+0000) 2022-09-10T10:43:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:17 smithi110 ceph-mon[97633]: pgmap v11970: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:43:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:18.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:18.509610+0000) 2022-09-10T10:43:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:19.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:19.012180+0000) 2022-09-10T10:43:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:19.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:19.509850+0000) 2022-09-10T10:43:20.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:19 smithi104 ceph-mon[97044]: pgmap v11971: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:43:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:20.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:20.012328+0000) 2022-09-10T10:43:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:19 smithi110 ceph-mon[97633]: pgmap v11971: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:43:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:20.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:20.510052+0000) 2022-09-10T10:43:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:21.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:21.012459+0000) 2022-09-10T10:43:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:21.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:21.510332+0000) 2022-09-10T10:43:22.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:21 smithi104 ceph-mon[97044]: pgmap v11972: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:43:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:22.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:22.012636+0000) 2022-09-10T10:43:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:21 smithi110 ceph-mon[97633]: pgmap v11972: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:43:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:22.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:22.510575+0000) 2022-09-10T10:43:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:23.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:23.012795+0000) 2022-09-10T10:43:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:23.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:23.510767+0000) 2022-09-10T10:43:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:23 smithi104 ceph-mon[97044]: pgmap v11973: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:43:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:24.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:24.012924+0000) 2022-09-10T10:43:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:23 smithi110 ceph-mon[97633]: pgmap v11973: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:43:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:24.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:24.511006+0000) 2022-09-10T10:43:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:25.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:25.013056+0000) 2022-09-10T10:43:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:43:25.160+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:43:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:25 smithi110 ceph-mon[97633]: pgmap v11974: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:43:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:25 smithi104 ceph-mon[97044]: pgmap v11974: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:43:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:25.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:25.511191+0000) 2022-09-10T10:43:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:26.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:26.013275+0000) 2022-09-10T10:43:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:26.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:26.511437+0000) 2022-09-10T10:43:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:27.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:27.013413+0000) 2022-09-10T10:43:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:27 smithi110 ceph-mon[97633]: pgmap v11975: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:27 smithi104 ceph-mon[97044]: pgmap v11975: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:27.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:27.511701+0000) 2022-09-10T10:43:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:28.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:28.013587+0000) 2022-09-10T10:43:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:28.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:28.511915+0000) 2022-09-10T10:43:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:29.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:29.013734+0000) 2022-09-10T10:43:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:29 smithi110 ceph-mon[97633]: pgmap v11976: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:29 smithi104 ceph-mon[97044]: pgmap v11976: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:29.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:29.512160+0000) 2022-09-10T10:43:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:30.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:30.013886+0000) 2022-09-10T10:43:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:30.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:30.512357+0000) 2022-09-10T10:43:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:31.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:31.014023+0000) 2022-09-10T10:43:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:31 smithi110 ceph-mon[97633]: pgmap v11977: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:31 smithi104 ceph-mon[97044]: pgmap v11977: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:31.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:31.512533+0000) 2022-09-10T10:43:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:32.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:32.014167+0000) 2022-09-10T10:43:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:32.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:32.512676+0000) 2022-09-10T10:43:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:33.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:33.014329+0000) 2022-09-10T10:43:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:33.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:33.512915+0000) 2022-09-10T10:43:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:33 smithi104 ceph-mon[97044]: pgmap v11978: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:34.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:34.014460+0000) 2022-09-10T10:43:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:33 smithi110 ceph-mon[97633]: pgmap v11978: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:34.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:34.513140+0000) 2022-09-10T10:43:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:35.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:35.014596+0000) 2022-09-10T10:43:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:35 smithi110 ceph-mon[97633]: pgmap v11979: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:35 smithi104 ceph-mon[97044]: pgmap v11979: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:35.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:35.513342+0000) 2022-09-10T10:43:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:36.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:36.014720+0000) 2022-09-10T10:43:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:36.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:36.513547+0000) 2022-09-10T10:43:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:37.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:37.014858+0000) 2022-09-10T10:43:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:37 smithi110 ceph-mon[97633]: pgmap v11980: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:37 smithi104 ceph-mon[97044]: pgmap v11980: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:37.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:37.513704+0000) 2022-09-10T10:43:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:38.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:38.015018+0000) 2022-09-10T10:43:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:38.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:38.513945+0000) 2022-09-10T10:43:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:39.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:39.015180+0000) 2022-09-10T10:43:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:39.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:39.514154+0000) 2022-09-10T10:43:40.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:39 smithi104 ceph-mon[97044]: pgmap v11981: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:43:40.173+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:43:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:40.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:40.015316+0000) 2022-09-10T10:43:40.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:39 smithi110 ceph-mon[97633]: pgmap v11981: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:40.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:40.514395+0000) 2022-09-10T10:43:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:41.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:41.015473+0000) 2022-09-10T10:43:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:41.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:41.514616+0000) 2022-09-10T10:43:42.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:41 smithi104 ceph-mon[97044]: pgmap v11982: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:42.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:42.015642+0000) 2022-09-10T10:43:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:41 smithi110 ceph-mon[97633]: pgmap v11982: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:42.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:42.514822+0000) 2022-09-10T10:43:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:43.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:43.015779+0000) 2022-09-10T10:43:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:43.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:43.514991+0000) 2022-09-10T10:43:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:43 smithi104 ceph-mon[97044]: pgmap v11983: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:44.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:44.015938+0000) 2022-09-10T10:43:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:43 smithi110 ceph-mon[97633]: pgmap v11983: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:44.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:44.515169+0000) 2022-09-10T10:43:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:45.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:45.016079+0000) 2022-09-10T10:43:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:45.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:45.515314+0000) 2022-09-10T10:43:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:45 smithi104 ceph-mon[97044]: pgmap v11984: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:43:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:43:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:43:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:43:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:46.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:46.016230+0000) 2022-09-10T10:43:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:45 smithi110 ceph-mon[97633]: pgmap v11984: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:43:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:43:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:43:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:43:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:46.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:46.515539+0000) 2022-09-10T10:43:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:47.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:47.016369+0000) 2022-09-10T10:43:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:47.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:47.515808+0000) 2022-09-10T10:43:48.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:47 smithi104 ceph-mon[97044]: pgmap v11985: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:48.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:48.016513+0000) 2022-09-10T10:43:48.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:47 smithi110 ceph-mon[97633]: pgmap v11985: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:48.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:48.516082+0000) 2022-09-10T10:43:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:49.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:49.016644+0000) 2022-09-10T10:43:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:49.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:49.516352+0000) 2022-09-10T10:43:50.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:49 smithi104 ceph-mon[97044]: pgmap v11986: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:50.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:50.016781+0000) 2022-09-10T10:43:50.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:49 smithi110 ceph-mon[97633]: pgmap v11986: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:50.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:50.516598+0000) 2022-09-10T10:43:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:51.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:51.016907+0000) 2022-09-10T10:43:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:51.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:51.516854+0000) 2022-09-10T10:43:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:51 smithi104 ceph-mon[97044]: pgmap v11987: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:52.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:52.017064+0000) 2022-09-10T10:43:52.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:51 smithi110 ceph-mon[97633]: pgmap v11987: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:52.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:52.517130+0000) 2022-09-10T10:43:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:53.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:53.017212+0000) 2022-09-10T10:43:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:53.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:53.517280+0000) 2022-09-10T10:43:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:53 smithi104 ceph-mon[97044]: pgmap v11988: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:54.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:54.017342+0000) 2022-09-10T10:43:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:53 smithi110 ceph-mon[97633]: pgmap v11988: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:54.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:54.517482+0000) 2022-09-10T10:43:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:55.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:55.017471+0000) 2022-09-10T10:43:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:43:55.180+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:43:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:55 smithi110 ceph-mon[97633]: pgmap v11989: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:55 smithi104 ceph-mon[97044]: pgmap v11989: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:43:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:55.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:55.517663+0000) 2022-09-10T10:43:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:56.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:56.017599+0000) 2022-09-10T10:43:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:56.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:56.517870+0000) 2022-09-10T10:43:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:57.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:57.017728+0000) 2022-09-10T10:43:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:57 smithi110 ceph-mon[97633]: pgmap v11990: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:57 smithi104 ceph-mon[97044]: pgmap v11990: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:57.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:57.518061+0000) 2022-09-10T10:43:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:58.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:58.017878+0000) 2022-09-10T10:43:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:58.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:58.518267+0000) 2022-09-10T10:43:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:43:59.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:59.018021+0000) 2022-09-10T10:43:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:43:59 smithi110 ceph-mon[97633]: pgmap v11991: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:43:59 smithi104 ceph-mon[97044]: pgmap v11991: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:43:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:43:59.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:43:59.518428+0000) 2022-09-10T10:44:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:00.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:00.018185+0000) 2022-09-10T10:44:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:00.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:00.518659+0000) 2022-09-10T10:44:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:01.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:01.018309+0000) 2022-09-10T10:44:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:01 smithi110 ceph-mon[97633]: pgmap v11992: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:01 smithi104 ceph-mon[97044]: pgmap v11992: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:01.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:01.518906+0000) 2022-09-10T10:44:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:02.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:02.018480+0000) 2022-09-10T10:44:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:02.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:02.519170+0000) 2022-09-10T10:44:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:03.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:03.018621+0000) 2022-09-10T10:44:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:03.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:03.519410+0000) 2022-09-10T10:44:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:03 smithi104 ceph-mon[97044]: pgmap v11993: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:04.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:04.018733+0000) 2022-09-10T10:44:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:03 smithi110 ceph-mon[97633]: pgmap v11993: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:04.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:04.519654+0000) 2022-09-10T10:44:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:05.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:05.018859+0000) 2022-09-10T10:44:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:05 smithi110 ceph-mon[97633]: pgmap v11994: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:44:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:05 smithi104 ceph-mon[97044]: pgmap v11994: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:44:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:05.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:05.519902+0000) 2022-09-10T10:44:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:06.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:06.018971+0000) 2022-09-10T10:44:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:06.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:06.520152+0000) 2022-09-10T10:44:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:07.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:07.019074+0000) 2022-09-10T10:44:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:07 smithi110 ceph-mon[97633]: pgmap v11995: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:44:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:07 smithi104 ceph-mon[97044]: pgmap v11995: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:44:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:07.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:07.520354+0000) 2022-09-10T10:44:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:08.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:08.019241+0000) 2022-09-10T10:44:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:08.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:08.520609+0000) 2022-09-10T10:44:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:09.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:09.019363+0000) 2022-09-10T10:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:09 smithi110 ceph-mon[97633]: osdmap e830: 12 total, 12 up, 12 in 2022-09-10T10:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:09 smithi110 ceph-mon[97633]: pgmap v11997: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:44:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:09 smithi104 ceph-mon[97044]: osdmap e830: 12 total, 12 up, 12 in 2022-09-10T10:44:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:09 smithi104 ceph-mon[97044]: pgmap v11997: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:44:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:09.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:09.520850+0000) 2022-09-10T10:44:10.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:10.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:10.019585+0000) 2022-09-10T10:44:10.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:44:10.194+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:44:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:10.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:10.521082+0000) 2022-09-10T10:44:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:11.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:11.019717+0000) 2022-09-10T10:44:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:11 smithi110 ceph-mon[97633]: pgmap v11998: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:44:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:11 smithi104 ceph-mon[97044]: pgmap v11998: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:44:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:11.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:11.521296+0000) 2022-09-10T10:44:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:12.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:12.019874+0000) 2022-09-10T10:44:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:12.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:12.521487+0000) 2022-09-10T10:44:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:13.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:13.020039+0000) 2022-09-10T10:44:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:13.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:13.521682+0000) 2022-09-10T10:44:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:13 smithi104 ceph-mon[97044]: osdmap e831: 12 total, 12 up, 12 in 2022-09-10T10:44:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:13 smithi104 ceph-mon[97044]: pgmap v12000: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:44:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:14.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:14.020157+0000) 2022-09-10T10:44:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:13 smithi110 ceph-mon[97633]: osdmap e831: 12 total, 12 up, 12 in 2022-09-10T10:44:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:13 smithi110 ceph-mon[97633]: pgmap v12000: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:44:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:14.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:14.521933+0000) 2022-09-10T10:44:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:15.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:15.020343+0000) 2022-09-10T10:44:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:15 smithi110 ceph-mon[97633]: pgmap v12001: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:44:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:15 smithi104 ceph-mon[97044]: pgmap v12001: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:44:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:15.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:15.522187+0000) 2022-09-10T10:44:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:16.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:16.020471+0000) 2022-09-10T10:44:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:16.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:16.522425+0000) 2022-09-10T10:44:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:17.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:17.020598+0000) 2022-09-10T10:44:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:17 smithi110 ceph-mon[97633]: pgmap v12002: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:44:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:17 smithi104 ceph-mon[97044]: pgmap v12002: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T10:44:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:17.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:17.522658+0000) 2022-09-10T10:44:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:18.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:18.020745+0000) 2022-09-10T10:44:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:18.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:18.522903+0000) 2022-09-10T10:44:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:19.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:19.020912+0000) 2022-09-10T10:44:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:19 smithi110 ceph-mon[97633]: pgmap v12003: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:44:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:19 smithi104 ceph-mon[97044]: pgmap v12003: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:44:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:19.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:19.523153+0000) 2022-09-10T10:44:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:20.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:20.021047+0000) 2022-09-10T10:44:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:20.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:20.523317+0000) 2022-09-10T10:44:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:21.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:21.021151+0000) 2022-09-10T10:44:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:21 smithi110 ceph-mon[97633]: pgmap v12004: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:44:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:21 smithi104 ceph-mon[97044]: pgmap v12004: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:44:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:21.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:21.523508+0000) 2022-09-10T10:44:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:22.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:22.021274+0000) 2022-09-10T10:44:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:22.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:22.523712+0000) 2022-09-10T10:44:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:23.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:23.021400+0000) 2022-09-10T10:44:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:23.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:23.523939+0000) 2022-09-10T10:44:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:23 smithi104 ceph-mon[97044]: pgmap v12005: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:44:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:24.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:24.021518+0000) 2022-09-10T10:44:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:23 smithi110 ceph-mon[97633]: pgmap v12005: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:44:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:24.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:24.524188+0000) 2022-09-10T10:44:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:25.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:25.021641+0000) 2022-09-10T10:44:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:44:25.200+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:44:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:25 smithi110 ceph-mon[97633]: pgmap v12006: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:44:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:25 smithi104 ceph-mon[97044]: pgmap v12006: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:44:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:25.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:25.524421+0000) 2022-09-10T10:44:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:26.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:26.021771+0000) 2022-09-10T10:44:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:26.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:26.524597+0000) 2022-09-10T10:44:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:27.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:27.021904+0000) 2022-09-10T10:44:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:27 smithi110 ceph-mon[97633]: pgmap v12007: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:44:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:27 smithi104 ceph-mon[97044]: pgmap v12007: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:44:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:27.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:27.524855+0000) 2022-09-10T10:44:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:28.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:28.022033+0000) 2022-09-10T10:44:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:28.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:28.525123+0000) 2022-09-10T10:44:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:29.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:29.022282+0000) 2022-09-10T10:44:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:29 smithi110 ceph-mon[97633]: pgmap v12008: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:44:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:29 smithi104 ceph-mon[97044]: pgmap v12008: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:44:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:29.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:29.525338+0000) 2022-09-10T10:44:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:30.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:30.022441+0000) 2022-09-10T10:44:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:30.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:30.525559+0000) 2022-09-10T10:44:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:31.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:31.022567+0000) 2022-09-10T10:44:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:31 smithi110 ceph-mon[97633]: pgmap v12009: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:31 smithi104 ceph-mon[97044]: pgmap v12009: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:31.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:31.525781+0000) 2022-09-10T10:44:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:32.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:32.022687+0000) 2022-09-10T10:44:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:32.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:32.526049+0000) 2022-09-10T10:44:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:33.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:33.022878+0000) 2022-09-10T10:44:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:33.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:33.526292+0000) 2022-09-10T10:44:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:33 smithi104 ceph-mon[97044]: pgmap v12010: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:34.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:34.023019+0000) 2022-09-10T10:44:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:33 smithi110 ceph-mon[97633]: pgmap v12010: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:34.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:34.526447+0000) 2022-09-10T10:44:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:35.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:35.023167+0000) 2022-09-10T10:44:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:35 smithi110 ceph-mon[97633]: pgmap v12011: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:35 smithi104 ceph-mon[97044]: pgmap v12011: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:35.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:35.526658+0000) 2022-09-10T10:44:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:36.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:36.023321+0000) 2022-09-10T10:44:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:36.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:36.526874+0000) 2022-09-10T10:44:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:37.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:37.023464+0000) 2022-09-10T10:44:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:37 smithi110 ceph-mon[97633]: pgmap v12012: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:37 smithi104 ceph-mon[97044]: pgmap v12012: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:37.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:37.527110+0000) 2022-09-10T10:44:38.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:38.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:38.023668+0000) 2022-09-10T10:44:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:38.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:38.527349+0000) 2022-09-10T10:44:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:39.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:39.023803+0000) 2022-09-10T10:44:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:39 smithi110 ceph-mon[97633]: pgmap v12013: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:39 smithi104 ceph-mon[97044]: pgmap v12013: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:39.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:39.527529+0000) 2022-09-10T10:44:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:40.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:40.023938+0000) 2022-09-10T10:44:40.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:44:40.215+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:44:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:40.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:40.527774+0000) 2022-09-10T10:44:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:41.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:41.024146+0000) 2022-09-10T10:44:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:41 smithi110 ceph-mon[97633]: pgmap v12014: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:41 smithi104 ceph-mon[97044]: pgmap v12014: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:41.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:41.527996+0000) 2022-09-10T10:44:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:42.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:42.024286+0000) 2022-09-10T10:44:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:42.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:42.528270+0000) 2022-09-10T10:44:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:43.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:43.024412+0000) 2022-09-10T10:44:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:43.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:43.528514+0000) 2022-09-10T10:44:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:43 smithi104 ceph-mon[97044]: pgmap v12015: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:44.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:44.024574+0000) 2022-09-10T10:44:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:43 smithi110 ceph-mon[97633]: pgmap v12015: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:44.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:44.528656+0000) 2022-09-10T10:44:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:45.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:45.024713+0000) 2022-09-10T10:44:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:45 smithi110 ceph-mon[97633]: pgmap v12016: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:45 smithi104 ceph-mon[97044]: pgmap v12016: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:45.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:45.528863+0000) 2022-09-10T10:44:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:46.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:46.024840+0000) 2022-09-10T10:44:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:44:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:44:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:44:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:44:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:44:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:44:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:44:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:44:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:46.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:46.529178+0000) 2022-09-10T10:44:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:47.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:47.025007+0000) 2022-09-10T10:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:47 smithi110 ceph-mon[97633]: pgmap v12017: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:47 smithi104 ceph-mon[97044]: pgmap v12017: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:47.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:47.529419+0000) 2022-09-10T10:44:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:48.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:48.025164+0000) 2022-09-10T10:44:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:48.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:48.529604+0000) 2022-09-10T10:44:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:49.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:49.025289+0000) 2022-09-10T10:44:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:49 smithi110 ceph-mon[97633]: pgmap v12018: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:49 smithi104 ceph-mon[97044]: pgmap v12018: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:49.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:49.529843+0000) 2022-09-10T10:44:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:50.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:50.025430+0000) 2022-09-10T10:44:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:50.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:50.530074+0000) 2022-09-10T10:44:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:51.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:51.025547+0000) 2022-09-10T10:44:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:51 smithi110 ceph-mon[97633]: pgmap v12019: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:51 smithi104 ceph-mon[97044]: pgmap v12019: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:51.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:51.530179+0000) 2022-09-10T10:44:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:52.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:52.025659+0000) 2022-09-10T10:44:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:52.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:52.530362+0000) 2022-09-10T10:44:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:53.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:53.025784+0000) 2022-09-10T10:44:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:53.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:53.530630+0000) 2022-09-10T10:44:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:53 smithi104 ceph-mon[97044]: pgmap v12020: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:54.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:54.025940+0000) 2022-09-10T10:44:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:53 smithi110 ceph-mon[97633]: pgmap v12020: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:54.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:54.530890+0000) 2022-09-10T10:44:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:55.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:55.026069+0000) 2022-09-10T10:44:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:44:55.221+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:44:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:55 smithi110 ceph-mon[97633]: pgmap v12021: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:55 smithi104 ceph-mon[97044]: pgmap v12021: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:44:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:55.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:55.531089+0000) 2022-09-10T10:44:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:56.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:56.026167+0000) 2022-09-10T10:44:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:56.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:56.531272+0000) 2022-09-10T10:44:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:57.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:57.026319+0000) 2022-09-10T10:44:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:57 smithi110 ceph-mon[97633]: pgmap v12022: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:57 smithi104 ceph-mon[97044]: pgmap v12022: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:57.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:57.531411+0000) 2022-09-10T10:44:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:58.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:58.026491+0000) 2022-09-10T10:44:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:58.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:58.531603+0000) 2022-09-10T10:44:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:44:59.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:59.026652+0000) 2022-09-10T10:44:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:44:59 smithi110 ceph-mon[97633]: pgmap v12023: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:44:59 smithi104 ceph-mon[97044]: pgmap v12023: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:44:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:44:59.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:44:59.531846+0000) 2022-09-10T10:45:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:00.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:00.026820+0000) 2022-09-10T10:45:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:00.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:00.532070+0000) 2022-09-10T10:45:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:01.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:01.026932+0000) 2022-09-10T10:45:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:01 smithi110 ceph-mon[97633]: pgmap v12024: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:01 smithi104 ceph-mon[97044]: pgmap v12024: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:01.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:01.532311+0000) 2022-09-10T10:45:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:02.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:02.027059+0000) 2022-09-10T10:45:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:02.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:02.532564+0000) 2022-09-10T10:45:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:03.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:03.027177+0000) 2022-09-10T10:45:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:03.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:03.532825+0000) 2022-09-10T10:45:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:03 smithi104 ceph-mon[97044]: pgmap v12025: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:04.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:04.027340+0000) 2022-09-10T10:45:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:03 smithi110 ceph-mon[97633]: pgmap v12025: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:04.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:04.533030+0000) 2022-09-10T10:45:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:05.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:05.027497+0000) 2022-09-10T10:45:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:05 smithi110 ceph-mon[97633]: pgmap v12026: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:45:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:05 smithi104 ceph-mon[97044]: pgmap v12026: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:45:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:05.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:05.533248+0000) 2022-09-10T10:45:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:06.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:06.027673+0000) 2022-09-10T10:45:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:06.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:06.533452+0000) 2022-09-10T10:45:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:07.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:07.027807+0000) 2022-09-10T10:45:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:07 smithi110 ceph-mon[97633]: pgmap v12027: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:45:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:07 smithi104 ceph-mon[97044]: pgmap v12027: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:45:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:07.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:07.533656+0000) 2022-09-10T10:45:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:08.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:08.027988+0000) 2022-09-10T10:45:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:08.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:08.533869+0000) 2022-09-10T10:45:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:09.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:09.028132+0000) 2022-09-10T10:45:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:09 smithi110 ceph-mon[97633]: osdmap e832: 12 total, 12 up, 12 in 2022-09-10T10:45:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:09 smithi110 ceph-mon[97633]: pgmap v12029: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:09 smithi104 ceph-mon[97044]: osdmap e832: 12 total, 12 up, 12 in 2022-09-10T10:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:09 smithi104 ceph-mon[97044]: pgmap v12029: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:45:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:09.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:09.534136+0000) 2022-09-10T10:45:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:10.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:10.028269+0000) 2022-09-10T10:45:10.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:45:10.224+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:45:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:10.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:10.534362+0000) 2022-09-10T10:45:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:11.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:11.028409+0000) 2022-09-10T10:45:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:11 smithi110 ceph-mon[97633]: pgmap v12030: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:45:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:11 smithi104 ceph-mon[97044]: pgmap v12030: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:45:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:11.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:11.534616+0000) 2022-09-10T10:45:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:12.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:12.028550+0000) 2022-09-10T10:45:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:12.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:12.534802+0000) 2022-09-10T10:45:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:13.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:13.028685+0000) 2022-09-10T10:45:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:13.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:13.535010+0000) 2022-09-10T10:45:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:13 smithi104 ceph-mon[97044]: osdmap e833: 12 total, 12 up, 12 in 2022-09-10T10:45:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:13 smithi104 ceph-mon[97044]: pgmap v12032: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:45:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:14.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:14.028871+0000) 2022-09-10T10:45:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:13 smithi110 ceph-mon[97633]: osdmap e833: 12 total, 12 up, 12 in 2022-09-10T10:45:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:13 smithi110 ceph-mon[97633]: pgmap v12032: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:45:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:14.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:14.535238+0000) 2022-09-10T10:45:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:15.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:15.029032+0000) 2022-09-10T10:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:15 smithi110 ceph-mon[97633]: pgmap v12033: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:45:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:15 smithi104 ceph-mon[97044]: pgmap v12033: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:45:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:15.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:15.535479+0000) 2022-09-10T10:45:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:16.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:16.029203+0000) 2022-09-10T10:45:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:16.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:16.535719+0000) 2022-09-10T10:45:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:17.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:17.029334+0000) 2022-09-10T10:45:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:17 smithi110 ceph-mon[97633]: pgmap v12034: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:45:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:17 smithi104 ceph-mon[97044]: pgmap v12034: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:45:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:17.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:17.535961+0000) 2022-09-10T10:45:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:18.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:18.029485+0000) 2022-09-10T10:45:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:18.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:18.536203+0000) 2022-09-10T10:45:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:19.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:19.029670+0000) 2022-09-10T10:45:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:19 smithi110 ceph-mon[97633]: pgmap v12035: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:45:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:19 smithi104 ceph-mon[97044]: pgmap v12035: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:45:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:19.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:19.536409+0000) 2022-09-10T10:45:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:20.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:20.029846+0000) 2022-09-10T10:45:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:20.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:20.536642+0000) 2022-09-10T10:45:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:21.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:21.029988+0000) 2022-09-10T10:45:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:21 smithi110 ceph-mon[97633]: pgmap v12036: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:45:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:21 smithi104 ceph-mon[97044]: pgmap v12036: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:45:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:21.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:21.536876+0000) 2022-09-10T10:45:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:22.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:22.030164+0000) 2022-09-10T10:45:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:22.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:22.537178+0000) 2022-09-10T10:45:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:23.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:23.030321+0000) 2022-09-10T10:45:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:23.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:23.537448+0000) 2022-09-10T10:45:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:23 smithi104 ceph-mon[97044]: pgmap v12037: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:45:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:24.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:24.030446+0000) 2022-09-10T10:45:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:23 smithi110 ceph-mon[97633]: pgmap v12037: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:45:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:24.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:24.537665+0000) 2022-09-10T10:45:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:25.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:25.030580+0000) 2022-09-10T10:45:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:45:25.240+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:45:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:25 smithi110 ceph-mon[97633]: pgmap v12038: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:45:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:25 smithi104 ceph-mon[97044]: pgmap v12038: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:45:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:25.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:25.537887+0000) 2022-09-10T10:45:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:26.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:26.030789+0000) 2022-09-10T10:45:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:26.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:26.538174+0000) 2022-09-10T10:45:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:27.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:27.030935+0000) 2022-09-10T10:45:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:27 smithi110 ceph-mon[97633]: pgmap v12039: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:27 smithi104 ceph-mon[97044]: pgmap v12039: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:27.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:27.538398+0000) 2022-09-10T10:45:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:28.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:28.031219+0000) 2022-09-10T10:45:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:28.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:28.538589+0000) 2022-09-10T10:45:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:29.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:29.031395+0000) 2022-09-10T10:45:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:29 smithi110 ceph-mon[97633]: pgmap v12040: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:29 smithi104 ceph-mon[97044]: pgmap v12040: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:29.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:29.538833+0000) 2022-09-10T10:45:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:30.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:30.031541+0000) 2022-09-10T10:45:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:30.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:30.539013+0000) 2022-09-10T10:45:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:31.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:31.031709+0000) 2022-09-10T10:45:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:31 smithi110 ceph-mon[97633]: pgmap v12041: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:31 smithi104 ceph-mon[97044]: pgmap v12041: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:31.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:31.539244+0000) 2022-09-10T10:45:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:32.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:32.031873+0000) 2022-09-10T10:45:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:32.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:32.539464+0000) 2022-09-10T10:45:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:33.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:33.032051+0000) 2022-09-10T10:45:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:33.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:33.539721+0000) 2022-09-10T10:45:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:33 smithi104 ceph-mon[97044]: pgmap v12042: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:34.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:34.032265+0000) 2022-09-10T10:45:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:33 smithi110 ceph-mon[97633]: pgmap v12042: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:34.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:34.539941+0000) 2022-09-10T10:45:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:35.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:35.032429+0000) 2022-09-10T10:45:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:35 smithi110 ceph-mon[97633]: pgmap v12043: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:35 smithi104 ceph-mon[97044]: pgmap v12043: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:35.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:35.540175+0000) 2022-09-10T10:45:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:36.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:36.032554+0000) 2022-09-10T10:45:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:36.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:36.540375+0000) 2022-09-10T10:45:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:37.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:37.032737+0000) 2022-09-10T10:45:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:37 smithi110 ceph-mon[97633]: pgmap v12044: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:37 smithi104 ceph-mon[97044]: pgmap v12044: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:37.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:37.540643+0000) 2022-09-10T10:45:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:38.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:38.033068+0000) 2022-09-10T10:45:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:38.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:38.540836+0000) 2022-09-10T10:45:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:39.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:39.033292+0000) 2022-09-10T10:45:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:39 smithi110 ceph-mon[97633]: pgmap v12045: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:39 smithi104 ceph-mon[97044]: pgmap v12045: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:39.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:39.541055+0000) 2022-09-10T10:45:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:40.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:40.033429+0000) 2022-09-10T10:45:40.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:45:40.250+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:45:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:40.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:40.541305+0000) 2022-09-10T10:45:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:41.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:41.033628+0000) 2022-09-10T10:45:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:41 smithi110 ceph-mon[97633]: pgmap v12046: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:41 smithi104 ceph-mon[97044]: pgmap v12046: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:41.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:41.541461+0000) 2022-09-10T10:45:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:42.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:42.033824+0000) 2022-09-10T10:45:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:42.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:42.541640+0000) 2022-09-10T10:45:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:43.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:43.034021+0000) 2022-09-10T10:45:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:43.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:43.541902+0000) 2022-09-10T10:45:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:44 smithi104 ceph-mon[97044]: pgmap v12047: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:44.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:44.034212+0000) 2022-09-10T10:45:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:44 smithi110 ceph-mon[97633]: pgmap v12047: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:44.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:44.542186+0000) 2022-09-10T10:45:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:45.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:45.034382+0000) 2022-09-10T10:45:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:45 smithi110 ceph-mon[97633]: pgmap v12048: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:45 smithi104 ceph-mon[97044]: pgmap v12048: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:45.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:45.542392+0000) 2022-09-10T10:45:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:46.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:46.034608+0000) 2022-09-10T10:45:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:45:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:45:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:45:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:45:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:45:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:45:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:45:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:45:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:46.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:46.542559+0000) 2022-09-10T10:45:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:47.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:47.034761+0000) 2022-09-10T10:45:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:47 smithi110 ceph-mon[97633]: pgmap v12049: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:47 smithi104 ceph-mon[97044]: pgmap v12049: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:47.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:47.542769+0000) 2022-09-10T10:45:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:48.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:48.034988+0000) 2022-09-10T10:45:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:48.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:48.543016+0000) 2022-09-10T10:45:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:49.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:49.035213+0000) 2022-09-10T10:45:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:49 smithi110 ceph-mon[97633]: pgmap v12050: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:49 smithi104 ceph-mon[97044]: pgmap v12050: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:49.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:49.543257+0000) 2022-09-10T10:45:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:50.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:50.035429+0000) 2022-09-10T10:45:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:50.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:50.543495+0000) 2022-09-10T10:45:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:51.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:51.035579+0000) 2022-09-10T10:45:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:51 smithi110 ceph-mon[97633]: pgmap v12051: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:51 smithi104 ceph-mon[97044]: pgmap v12051: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:51.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:51.543737+0000) 2022-09-10T10:45:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:52.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:52.035806+0000) 2022-09-10T10:45:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:52.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:52.543957+0000) 2022-09-10T10:45:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:53.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:53.036048+0000) 2022-09-10T10:45:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:53.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:53.544168+0000) 2022-09-10T10:45:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:54 smithi104 ceph-mon[97044]: pgmap v12052: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:54.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:54.036284+0000) 2022-09-10T10:45:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:54 smithi110 ceph-mon[97633]: pgmap v12052: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:54.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:54.544448+0000) 2022-09-10T10:45:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:55.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:55.036477+0000) 2022-09-10T10:45:55.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:45:55.264+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:45:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:55 smithi110 ceph-mon[97633]: pgmap v12053: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:55 smithi104 ceph-mon[97044]: pgmap v12053: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:45:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:55.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:55.544655+0000) 2022-09-10T10:45:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:56.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:56.036665+0000) 2022-09-10T10:45:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:56.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:56.544851+0000) 2022-09-10T10:45:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:57.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:57.036913+0000) 2022-09-10T10:45:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:57 smithi110 ceph-mon[97633]: pgmap v12054: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:57 smithi104 ceph-mon[97044]: pgmap v12054: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:57.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:57.545136+0000) 2022-09-10T10:45:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:58.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:58.037126+0000) 2022-09-10T10:45:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:58.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:58.545301+0000) 2022-09-10T10:45:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:45:59.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:59.037340+0000) 2022-09-10T10:45:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:45:59 smithi110 ceph-mon[97633]: pgmap v12055: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:45:59 smithi104 ceph-mon[97044]: pgmap v12055: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:45:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:45:59.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:45:59.545524+0000) 2022-09-10T10:46:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:00.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:00.037509+0000) 2022-09-10T10:46:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:00.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:00.545741+0000) 2022-09-10T10:46:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:01.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:01.037624+0000) 2022-09-10T10:46:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:01 smithi110 ceph-mon[97633]: pgmap v12056: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:01 smithi104 ceph-mon[97044]: pgmap v12056: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:01.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:01.545961+0000) 2022-09-10T10:46:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:02.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:02.037831+0000) 2022-09-10T10:46:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:02.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:02.546192+0000) 2022-09-10T10:46:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:03.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:03.038036+0000) 2022-09-10T10:46:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:03.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:03.546376+0000) 2022-09-10T10:46:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:04 smithi104 ceph-mon[97044]: pgmap v12057: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:04.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:04.038253+0000) 2022-09-10T10:46:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:04 smithi110 ceph-mon[97633]: pgmap v12057: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:04.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:04.546603+0000) 2022-09-10T10:46:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:05.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:05.038470+0000) 2022-09-10T10:46:05.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:05 smithi110 ceph-mon[97633]: pgmap v12058: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:46:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:05 smithi104 ceph-mon[97044]: pgmap v12058: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:46:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:05.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:05.546854+0000) 2022-09-10T10:46:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:06.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:06.038684+0000) 2022-09-10T10:46:06.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:06.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:06.547043+0000) 2022-09-10T10:46:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:07.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:07.038878+0000) 2022-09-10T10:46:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:07 smithi110 ceph-mon[97633]: pgmap v12059: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:46:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:07 smithi104 ceph-mon[97044]: pgmap v12059: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:46:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:07.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:07.547265+0000) 2022-09-10T10:46:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:08.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:08.039151+0000) 2022-09-10T10:46:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:08.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:08.547493+0000) 2022-09-10T10:46:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:09.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:09.039380+0000) 2022-09-10T10:46:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:09 smithi110 ceph-mon[97633]: osdmap e834: 12 total, 12 up, 12 in 2022-09-10T10:46:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:09 smithi110 ceph-mon[97633]: pgmap v12061: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:46:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:09 smithi104 ceph-mon[97044]: osdmap e834: 12 total, 12 up, 12 in 2022-09-10T10:46:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:09 smithi104 ceph-mon[97044]: pgmap v12061: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:46:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:09.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:09.547764+0000) 2022-09-10T10:46:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:10.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:10.039606+0000) 2022-09-10T10:46:10.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:46:10.275+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:46:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:10.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:10.547996+0000) 2022-09-10T10:46:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:11.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:11.039780+0000) 2022-09-10T10:46:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:11 smithi110 ceph-mon[97633]: pgmap v12062: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:46:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:11 smithi104 ceph-mon[97044]: pgmap v12062: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T10:46:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:11.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:11.548145+0000) 2022-09-10T10:46:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:12.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:12.039946+0000) 2022-09-10T10:46:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:12.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:12.548352+0000) 2022-09-10T10:46:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:13.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:13.040082+0000) 2022-09-10T10:46:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:13.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:13.548531+0000) 2022-09-10T10:46:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:14 smithi104 ceph-mon[97044]: osdmap e835: 12 total, 12 up, 12 in 2022-09-10T10:46:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:14 smithi104 ceph-mon[97044]: pgmap v12064: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:46:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:14.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:14.040270+0000) 2022-09-10T10:46:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:14 smithi110 ceph-mon[97633]: osdmap e835: 12 total, 12 up, 12 in 2022-09-10T10:46:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:14 smithi110 ceph-mon[97633]: pgmap v12064: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:46:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:14.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:14.548710+0000) 2022-09-10T10:46:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:15.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:15.040450+0000) 2022-09-10T10:46:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:15 smithi110 ceph-mon[97633]: pgmap v12065: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:46:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:15 smithi104 ceph-mon[97044]: pgmap v12065: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:46:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:15.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:15.548921+0000) 2022-09-10T10:46:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:16.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:16.040716+0000) 2022-09-10T10:46:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:16.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:16.549189+0000) 2022-09-10T10:46:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:17.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:17.040966+0000) 2022-09-10T10:46:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:17 smithi110 ceph-mon[97633]: pgmap v12066: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:46:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:17 smithi104 ceph-mon[97044]: pgmap v12066: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:46:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:17.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:17.549461+0000) 2022-09-10T10:46:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:18.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:18.041172+0000) 2022-09-10T10:46:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:18.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:18.549738+0000) 2022-09-10T10:46:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:19.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:19.041372+0000) 2022-09-10T10:46:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:19 smithi110 ceph-mon[97633]: pgmap v12067: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:46:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:19 smithi104 ceph-mon[97044]: pgmap v12067: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:46:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:19.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:19.549950+0000) 2022-09-10T10:46:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:20.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:20.041578+0000) 2022-09-10T10:46:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:20.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:20.550184+0000) 2022-09-10T10:46:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:21.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:21.041741+0000) 2022-09-10T10:46:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:21 smithi110 ceph-mon[97633]: pgmap v12068: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:46:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:21 smithi104 ceph-mon[97044]: pgmap v12068: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:46:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:21.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:21.550316+0000) 2022-09-10T10:46:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:22.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:22.041906+0000) 2022-09-10T10:46:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:22.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:22.550540+0000) 2022-09-10T10:46:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:23.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:23.042064+0000) 2022-09-10T10:46:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:23.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:23.550749+0000) 2022-09-10T10:46:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:24 smithi104 ceph-mon[97044]: pgmap v12069: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:46:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:24.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:24.042250+0000) 2022-09-10T10:46:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:24 smithi110 ceph-mon[97633]: pgmap v12069: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:46:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:24.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:24.550994+0000) 2022-09-10T10:46:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:25.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:25.042430+0000) 2022-09-10T10:46:25.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:46:25.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:46:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:25 smithi110 ceph-mon[97633]: pgmap v12070: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:46:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:25 smithi104 ceph-mon[97044]: pgmap v12070: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:46:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:25.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:25.551245+0000) 2022-09-10T10:46:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:26.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:26.042626+0000) 2022-09-10T10:46:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:26.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:26.551499+0000) 2022-09-10T10:46:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:27.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:27.042823+0000) 2022-09-10T10:46:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:27 smithi110 ceph-mon[97633]: pgmap v12071: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:27 smithi104 ceph-mon[97044]: pgmap v12071: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:27.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:27.551725+0000) 2022-09-10T10:46:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:28.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:28.043026+0000) 2022-09-10T10:46:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:28.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:28.551940+0000) 2022-09-10T10:46:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:29.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:29.043173+0000) 2022-09-10T10:46:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:29 smithi110 ceph-mon[97633]: pgmap v12072: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:29 smithi104 ceph-mon[97044]: pgmap v12072: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:29.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:29.552214+0000) 2022-09-10T10:46:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:30.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:30.043404+0000) 2022-09-10T10:46:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:30.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:30.552475+0000) 2022-09-10T10:46:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:31.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:31.043678+0000) 2022-09-10T10:46:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:31 smithi110 ceph-mon[97633]: pgmap v12073: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:31 smithi104 ceph-mon[97044]: pgmap v12073: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:31.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:31.552658+0000) 2022-09-10T10:46:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:32.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:32.043896+0000) 2022-09-10T10:46:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:32.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:32.552850+0000) 2022-09-10T10:46:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:33.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:33.044157+0000) 2022-09-10T10:46:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:33.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:33.553044+0000) 2022-09-10T10:46:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:34 smithi104 ceph-mon[97044]: pgmap v12074: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:34.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:34.044326+0000) 2022-09-10T10:46:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:34 smithi110 ceph-mon[97633]: pgmap v12074: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:34.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:34.553155+0000) 2022-09-10T10:46:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:35.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:35.044499+0000) 2022-09-10T10:46:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:35 smithi110 ceph-mon[97633]: pgmap v12075: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:35 smithi104 ceph-mon[97044]: pgmap v12075: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:35.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:35.553316+0000) 2022-09-10T10:46:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:36.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:36.044710+0000) 2022-09-10T10:46:37.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:36.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:36.553480+0000) 2022-09-10T10:46:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:37.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:37.044949+0000) 2022-09-10T10:46:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:37 smithi110 ceph-mon[97633]: pgmap v12076: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:37 smithi104 ceph-mon[97044]: pgmap v12076: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:38.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:37.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:37.553654+0000) 2022-09-10T10:46:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:38.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:38.045241+0000) 2022-09-10T10:46:39.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:38.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:38.553931+0000) 2022-09-10T10:46:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:39.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:39.045493+0000) 2022-09-10T10:46:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:39 smithi110 ceph-mon[97633]: pgmap v12077: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:39 smithi104 ceph-mon[97044]: pgmap v12077: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:40.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:39.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:39.554134+0000) 2022-09-10T10:46:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:40.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:40.045711+0000) 2022-09-10T10:46:40.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:46:40.290+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:46:41.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:40.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:40.554314+0000) 2022-09-10T10:46:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:41.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:41.045911+0000) 2022-09-10T10:46:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:41 smithi110 ceph-mon[97633]: pgmap v12078: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:41 smithi104 ceph-mon[97044]: pgmap v12078: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:42.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:41.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:41.554547+0000) 2022-09-10T10:46:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:42.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:42.046159+0000) 2022-09-10T10:46:43.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:42.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:42.554806+0000) 2022-09-10T10:46:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:43.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:43.046388+0000) 2022-09-10T10:46:44.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:43.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:43.555084+0000) 2022-09-10T10:46:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:44 smithi104 ceph-mon[97044]: pgmap v12079: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:44.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:44.046633+0000) 2022-09-10T10:46:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:44 smithi110 ceph-mon[97633]: pgmap v12079: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:45.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:44.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:44.555327+0000) 2022-09-10T10:46:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:45.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:45.046830+0000) 2022-09-10T10:46:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:45 smithi110 ceph-mon[97633]: pgmap v12080: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:45 smithi104 ceph-mon[97044]: pgmap v12080: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:46.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:45.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:45.555595+0000) 2022-09-10T10:46:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:46.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:46.047030+0000) 2022-09-10T10:46:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:46:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:46:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:46:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:46:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:46:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:46:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:46:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:46:47.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:46.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:46.555821+0000) 2022-09-10T10:46:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:47.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:47.047250+0000) 2022-09-10T10:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:47 smithi110 ceph-mon[97633]: pgmap v12081: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:47 smithi104 ceph-mon[97044]: pgmap v12081: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:48.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:47.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:47.555992+0000) 2022-09-10T10:46:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:48.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:48.047459+0000) 2022-09-10T10:46:49.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:48.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:48.556212+0000) 2022-09-10T10:46:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:49.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:49.047620+0000) 2022-09-10T10:46:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:49 smithi110 ceph-mon[97633]: pgmap v12082: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:49 smithi104 ceph-mon[97044]: pgmap v12082: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:50.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:49.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:49.556394+0000) 2022-09-10T10:46:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:50.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:50.047798+0000) 2022-09-10T10:46:51.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:50.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:50.556582+0000) 2022-09-10T10:46:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:51.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:51.047952+0000) 2022-09-10T10:46:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:51 smithi110 ceph-mon[97633]: pgmap v12083: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:51 smithi104 ceph-mon[97044]: pgmap v12083: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:52.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:51.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:51.556736+0000) 2022-09-10T10:46:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:52.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:52.048189+0000) 2022-09-10T10:46:53.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:52.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:52.556947+0000) 2022-09-10T10:46:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:53.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:53.048409+0000) 2022-09-10T10:46:54.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:53.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:53.557196+0000) 2022-09-10T10:46:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:54 smithi104 ceph-mon[97044]: pgmap v12084: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:54.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:54.048573+0000) 2022-09-10T10:46:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:54 smithi110 ceph-mon[97633]: pgmap v12084: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:55.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:54.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:54.557434+0000) 2022-09-10T10:46:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:55.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:55.048786+0000) 2022-09-10T10:46:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:55 smithi110 ceph-mon[97633]: pgmap v12085: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:55 smithi104 ceph-mon[97044]: pgmap v12085: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:46:55.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:46:55.305+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:46:56.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:55.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:55.557682+0000) 2022-09-10T10:46:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:56.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:56.049019+0000) 2022-09-10T10:46:57.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:56.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:56.557925+0000) 2022-09-10T10:46:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:57.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:57.049210+0000) 2022-09-10T10:46:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:57 smithi110 ceph-mon[97633]: pgmap v12086: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:57 smithi104 ceph-mon[97044]: pgmap v12086: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:58.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:57.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:57.558143+0000) 2022-09-10T10:46:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:58.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:58.049483+0000) 2022-09-10T10:46:59.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:58.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:58.558354+0000) 2022-09-10T10:46:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:46:59.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:59.049709+0000) 2022-09-10T10:46:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:46:59 smithi110 ceph-mon[97633]: pgmap v12087: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:46:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:46:59 smithi104 ceph-mon[97044]: pgmap v12087: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:00.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:46:59.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:46:59.558531+0000) 2022-09-10T10:47:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:00.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:00.049923+0000) 2022-09-10T10:47:01.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:00.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:00.558690+0000) 2022-09-10T10:47:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:01.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:01.050195+0000) 2022-09-10T10:47:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:01 smithi110 ceph-mon[97633]: pgmap v12088: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:01 smithi104 ceph-mon[97044]: pgmap v12088: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:02.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:01.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:01.558887+0000) 2022-09-10T10:47:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:02.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:02.050401+0000) 2022-09-10T10:47:03.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:02.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:02.559119+0000) 2022-09-10T10:47:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:03.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:03.050678+0000) 2022-09-10T10:47:04.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:03.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:03.559340+0000) 2022-09-10T10:47:04.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:04 smithi104 ceph-mon[97044]: pgmap v12089: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:04.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:04.050882+0000) 2022-09-10T10:47:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:04 smithi110 ceph-mon[97633]: pgmap v12089: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:05.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:04.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:04.559598+0000) 2022-09-10T10:47:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:05.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:05.051139+0000) 2022-09-10T10:47:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:05 smithi110 ceph-mon[97633]: pgmap v12090: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:47:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:05 smithi104 ceph-mon[97044]: pgmap v12090: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:47:06.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:05.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:05.559820+0000) 2022-09-10T10:47:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:06.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:06.051259+0000) 2022-09-10T10:47:07.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:06.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:06.559979+0000) 2022-09-10T10:47:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:07.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:07.051451+0000) 2022-09-10T10:47:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:07 smithi110 ceph-mon[97633]: pgmap v12091: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:47:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:07 smithi104 ceph-mon[97044]: pgmap v12091: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:47:08.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:07.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:07.560157+0000) 2022-09-10T10:47:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:08.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:08.051761+0000) 2022-09-10T10:47:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:08.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:08.560377+0000) 2022-09-10T10:47:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:09.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:09.052001+0000) 2022-09-10T10:47:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:09 smithi110 ceph-mon[97633]: osdmap e836: 12 total, 12 up, 12 in 2022-09-10T10:47:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:09 smithi110 ceph-mon[97633]: pgmap v12093: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:09 smithi104 ceph-mon[97044]: osdmap e836: 12 total, 12 up, 12 in 2022-09-10T10:47:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:09 smithi104 ceph-mon[97044]: pgmap v12093: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:47:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:09.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:09.560508+0000) 2022-09-10T10:47:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:10.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:10.052214+0000) 2022-09-10T10:47:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:10.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:10.560684+0000) 2022-09-10T10:47:10.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:47:10.321+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:47:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:11.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:11.052390+0000) 2022-09-10T10:47:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:11 smithi110 ceph-mon[97633]: pgmap v12094: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:47:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:11 smithi104 ceph-mon[97044]: pgmap v12094: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:47:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:11.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:11.560895+0000) 2022-09-10T10:47:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:12.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:12.052541+0000) 2022-09-10T10:47:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:12.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:12.561073+0000) 2022-09-10T10:47:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:13.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:13.052681+0000) 2022-09-10T10:47:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:13.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:13.561235+0000) 2022-09-10T10:47:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:14 smithi104 ceph-mon[97044]: osdmap e837: 12 total, 12 up, 12 in 2022-09-10T10:47:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:14 smithi104 ceph-mon[97044]: pgmap v12096: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:47:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:14.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:14.052822+0000) 2022-09-10T10:47:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:14 smithi110 ceph-mon[97633]: osdmap e837: 12 total, 12 up, 12 in 2022-09-10T10:47:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:14 smithi110 ceph-mon[97633]: pgmap v12096: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:47:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:14.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:14.561398+0000) 2022-09-10T10:47:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:15.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:15.052975+0000) 2022-09-10T10:47:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:15 smithi110 ceph-mon[97633]: pgmap v12097: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:47:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:15 smithi104 ceph-mon[97044]: pgmap v12097: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:47:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:15.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:15.561557+0000) 2022-09-10T10:47:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:16.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:16.053208+0000) 2022-09-10T10:47:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:16.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:16.561695+0000) 2022-09-10T10:47:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:17.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:17.053342+0000) 2022-09-10T10:47:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:17 smithi110 ceph-mon[97633]: pgmap v12098: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:47:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:17 smithi104 ceph-mon[97044]: pgmap v12098: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:47:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:17.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:17.561852+0000) 2022-09-10T10:47:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:18.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:18.053539+0000) 2022-09-10T10:47:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:18.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:18.562066+0000) 2022-09-10T10:47:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:19 smithi110 ceph-mon[97633]: pgmap v12099: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:47:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:19 smithi104 ceph-mon[97044]: pgmap v12099: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:47:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:19.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:19.053812+0000) 2022-09-10T10:47:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:19.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:19.562256+0000) 2022-09-10T10:47:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:20.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:20.054009+0000) 2022-09-10T10:47:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:20.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:20.562438+0000) 2022-09-10T10:47:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:21 smithi110 ceph-mon[97633]: pgmap v12100: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:47:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:21 smithi104 ceph-mon[97044]: pgmap v12100: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:47:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:21.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:21.054168+0000) 2022-09-10T10:47:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:21.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:21.562596+0000) 2022-09-10T10:47:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:22.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:22.054306+0000) 2022-09-10T10:47:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:22.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:22.562792+0000) 2022-09-10T10:47:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:23.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:23.054503+0000) 2022-09-10T10:47:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:23.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:23.563050+0000) 2022-09-10T10:47:24.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:24 smithi104 ceph-mon[97044]: pgmap v12101: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:47:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:24 smithi110 ceph-mon[97633]: pgmap v12101: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:47:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:24.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:24.054696+0000) 2022-09-10T10:47:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:24.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:24.563212+0000) 2022-09-10T10:47:25.333 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:25.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:25.054874+0000) 2022-09-10T10:47:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:25 smithi110 ceph-mon[97633]: pgmap v12102: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:47:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:25 smithi104 ceph-mon[97044]: pgmap v12102: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:47:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:25.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:25.563424+0000) 2022-09-10T10:47:25.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:47:25.331+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:47:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:26.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:26.055129+0000) 2022-09-10T10:47:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:26.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:26.563659+0000) 2022-09-10T10:47:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:27 smithi110 ceph-mon[97633]: pgmap v12103: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:27.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:27 smithi104 ceph-mon[97044]: pgmap v12103: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:27.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:27.055297+0000) 2022-09-10T10:47:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:27.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:27.563871+0000) 2022-09-10T10:47:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:28.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:28.055482+0000) 2022-09-10T10:47:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:28.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:28.564076+0000) 2022-09-10T10:47:29.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:29 smithi110 ceph-mon[97633]: pgmap v12104: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:29.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:29 smithi104 ceph-mon[97044]: pgmap v12104: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:29.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:29.055677+0000) 2022-09-10T10:47:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:29.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:29.564335+0000) 2022-09-10T10:47:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:30.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:30.055875+0000) 2022-09-10T10:47:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:30.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:30.564502+0000) 2022-09-10T10:47:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:31 smithi110 ceph-mon[97633]: pgmap v12105: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:31 smithi104 ceph-mon[97044]: pgmap v12105: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:31.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:31.056102+0000) 2022-09-10T10:47:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:31.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:31.564667+0000) 2022-09-10T10:47:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:32.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:32.056287+0000) 2022-09-10T10:47:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:32.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:32.564889+0000) 2022-09-10T10:47:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:33.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:33.056453+0000) 2022-09-10T10:47:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:33.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:33.565062+0000) 2022-09-10T10:47:34.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:34 smithi104 ceph-mon[97044]: pgmap v12106: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:34 smithi110 ceph-mon[97633]: pgmap v12106: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:34.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:34.056586+0000) 2022-09-10T10:47:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:34.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:34.565251+0000) 2022-09-10T10:47:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:35 smithi110 ceph-mon[97633]: pgmap v12107: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:35 smithi104 ceph-mon[97044]: pgmap v12107: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:35.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:35.056770+0000) 2022-09-10T10:47:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:35.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:35.565431+0000) 2022-09-10T10:47:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:36.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:36.056972+0000) 2022-09-10T10:47:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:36.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:36.565603+0000) 2022-09-10T10:47:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:37 smithi110 ceph-mon[97633]: pgmap v12108: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:37 smithi104 ceph-mon[97044]: pgmap v12108: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:37.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:37.057181+0000) 2022-09-10T10:47:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:37.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:37.565786+0000) 2022-09-10T10:47:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:38.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:38.057367+0000) 2022-09-10T10:47:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:38.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:38.565965+0000) 2022-09-10T10:47:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:39 smithi110 ceph-mon[97633]: pgmap v12109: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:39 smithi104 ceph-mon[97044]: pgmap v12109: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:39.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:39.057560+0000) 2022-09-10T10:47:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:39.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:39.566210+0000) 2022-09-10T10:47:40.342 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:40.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:40.057718+0000) 2022-09-10T10:47:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:40.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:40.566406+0000) 2022-09-10T10:47:40.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:47:40.341+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:47:41.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:41 smithi110 ceph-mon[97633]: pgmap v12110: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:41.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:41 smithi104 ceph-mon[97044]: pgmap v12110: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:41.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:41.057886+0000) 2022-09-10T10:47:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:41.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:41.566566+0000) 2022-09-10T10:47:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:42.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:42.058051+0000) 2022-09-10T10:47:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:42.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:42.566841+0000) 2022-09-10T10:47:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:43.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:43.058230+0000) 2022-09-10T10:47:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:43.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:43.567082+0000) 2022-09-10T10:47:44.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:44 smithi104 ceph-mon[97044]: pgmap v12111: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:44 smithi110 ceph-mon[97633]: pgmap v12111: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:44.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:44.058444+0000) 2022-09-10T10:47:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:44.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:44.567310+0000) 2022-09-10T10:47:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:45 smithi110 ceph-mon[97633]: pgmap v12112: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:45 smithi104 ceph-mon[97044]: pgmap v12112: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:45.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:45.058608+0000) 2022-09-10T10:47:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:45.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:45.567545+0000) 2022-09-10T10:47:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:47:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:47:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:47:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:47:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:47:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:47:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:47:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:47:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:46.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:46.058837+0000) 2022-09-10T10:47:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:46.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:46.567786+0000) 2022-09-10T10:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:47 smithi110 ceph-mon[97633]: pgmap v12113: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:47 smithi104 ceph-mon[97044]: pgmap v12113: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:47.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:47.059020+0000) 2022-09-10T10:47:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:47.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:47.568036+0000) 2022-09-10T10:47:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:48.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:48.059265+0000) 2022-09-10T10:47:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:48.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:48.568253+0000) 2022-09-10T10:47:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:49 smithi110 ceph-mon[97633]: pgmap v12114: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:49 smithi104 ceph-mon[97044]: pgmap v12114: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:49.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:49.059483+0000) 2022-09-10T10:47:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:49.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:49.568498+0000) 2022-09-10T10:47:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:50.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:50.059648+0000) 2022-09-10T10:47:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:50.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:50.568723+0000) 2022-09-10T10:47:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:51 smithi110 ceph-mon[97633]: pgmap v12115: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:51 smithi104 ceph-mon[97044]: pgmap v12115: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:51.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:51.059835+0000) 2022-09-10T10:47:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:51.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:51.568960+0000) 2022-09-10T10:47:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:52.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:52.060083+0000) 2022-09-10T10:47:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:52.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:52.569181+0000) 2022-09-10T10:47:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:53.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:53.060339+0000) 2022-09-10T10:47:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:53.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:53.569372+0000) 2022-09-10T10:47:54.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:54 smithi104 ceph-mon[97044]: pgmap v12116: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:54 smithi110 ceph-mon[97633]: pgmap v12116: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:54.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:54.060509+0000) 2022-09-10T10:47:55.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:54.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:54.569624+0000) 2022-09-10T10:47:55.351 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:55.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:55.060708+0000) 2022-09-10T10:47:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:55 smithi110 ceph-mon[97633]: pgmap v12117: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:55 smithi104 ceph-mon[97044]: pgmap v12117: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:47:55.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:47:55.349+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:47:55.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:55.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:55.569878+0000) 2022-09-10T10:47:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:56.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:56.060972+0000) 2022-09-10T10:47:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:56.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:56.570063+0000) 2022-09-10T10:47:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:57 smithi110 ceph-mon[97633]: pgmap v12118: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:57 smithi104 ceph-mon[97044]: pgmap v12118: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:57.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:57.061227+0000) 2022-09-10T10:47:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:57.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:57.570278+0000) 2022-09-10T10:47:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:58.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:58.061449+0000) 2022-09-10T10:47:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:58.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:58.570455+0000) 2022-09-10T10:47:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:47:59 smithi110 ceph-mon[97633]: pgmap v12119: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:47:59 smithi104 ceph-mon[97044]: pgmap v12119: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:47:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:47:59.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:59.061642+0000) 2022-09-10T10:48:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:47:59.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:47:59.570640+0000) 2022-09-10T10:48:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:00.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:00.061896+0000) 2022-09-10T10:48:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:00.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:00.570824+0000) 2022-09-10T10:48:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:01 smithi110 ceph-mon[97633]: pgmap v12120: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:01 smithi104 ceph-mon[97044]: pgmap v12120: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:01.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:01.062174+0000) 2022-09-10T10:48:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:01.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:01.571075+0000) 2022-09-10T10:48:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:02.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:02.062410+0000) 2022-09-10T10:48:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:02.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:02.571317+0000) 2022-09-10T10:48:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:03.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:03.062652+0000) 2022-09-10T10:48:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:03.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:03.571511+0000) 2022-09-10T10:48:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:04 smithi104 ceph-mon[97044]: pgmap v12121: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:48:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:04 smithi110 ceph-mon[97633]: pgmap v12121: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:48:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:04.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:04.062901+0000) 2022-09-10T10:48:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:04.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:04.571751+0000) 2022-09-10T10:48:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:05 smithi110 ceph-mon[97633]: pgmap v12122: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:48:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:05 smithi104 ceph-mon[97044]: pgmap v12122: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:48:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:05.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:05.063222+0000) 2022-09-10T10:48:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:05.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:05.571965+0000) 2022-09-10T10:48:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:06.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:06.063389+0000) 2022-09-10T10:48:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:06.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:06.572160+0000) 2022-09-10T10:48:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:07 smithi110 ceph-mon[97633]: pgmap v12123: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T10:48:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:07 smithi104 ceph-mon[97044]: pgmap v12123: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T10:48:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:07.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:07.063549+0000) 2022-09-10T10:48:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:07.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:07.572314+0000) 2022-09-10T10:48:08.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:08.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:08.063737+0000) 2022-09-10T10:48:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:08.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:08.572543+0000) 2022-09-10T10:48:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:09 smithi110 ceph-mon[97633]: osdmap e838: 12 total, 12 up, 12 in 2022-09-10T10:48:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:09 smithi110 ceph-mon[97633]: pgmap v12125: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:09 smithi104 ceph-mon[97044]: osdmap e838: 12 total, 12 up, 12 in 2022-09-10T10:48:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:09 smithi104 ceph-mon[97044]: pgmap v12125: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:48:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:09.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:09.063983+0000) 2022-09-10T10:48:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:09.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:09.572773+0000) 2022-09-10T10:48:10.367 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:10.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:10.064263+0000) 2022-09-10T10:48:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:10.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:10.573019+0000) 2022-09-10T10:48:10.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:48:10.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:48:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:11 smithi110 ceph-mon[97633]: pgmap v12126: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:48:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:11 smithi104 ceph-mon[97044]: pgmap v12126: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:48:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:11.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:11.064444+0000) 2022-09-10T10:48:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:11.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:11.573254+0000) 2022-09-10T10:48:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:12.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:12.064653+0000) 2022-09-10T10:48:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:12.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:12.573510+0000) 2022-09-10T10:48:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:13.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:13.064852+0000) 2022-09-10T10:48:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:13.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:13.573798+0000) 2022-09-10T10:48:14.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:14 smithi104 ceph-mon[97044]: osdmap e839: 12 total, 12 up, 12 in 2022-09-10T10:48:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:14 smithi110 ceph-mon[97633]: osdmap e839: 12 total, 12 up, 12 in 2022-09-10T10:48:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:14.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:14.065030+0000) 2022-09-10T10:48:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:14.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:14.574019+0000) 2022-09-10T10:48:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:15 smithi104 ceph-mon[97044]: pgmap v12128: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:15 smithi110 ceph-mon[97633]: pgmap v12128: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:48:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:15.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:15.065311+0000) 2022-09-10T10:48:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:15.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:15.574267+0000) 2022-09-10T10:48:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:16.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:16.065531+0000) 2022-09-10T10:48:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:16.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:16.574435+0000) 2022-09-10T10:48:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:17 smithi110 ceph-mon[97633]: pgmap v12129: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:17 smithi104 ceph-mon[97044]: pgmap v12129: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:48:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:17.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:17.065773+0000) 2022-09-10T10:48:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:17.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:17.574647+0000) 2022-09-10T10:48:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:18.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:18.066042+0000) 2022-09-10T10:48:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:18.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:18.574894+0000) 2022-09-10T10:48:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:19 smithi110 ceph-mon[97633]: pgmap v12130: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:48:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:19 smithi104 ceph-mon[97044]: pgmap v12130: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:48:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:19.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:19.066269+0000) 2022-09-10T10:48:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:19.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:19.575140+0000) 2022-09-10T10:48:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:20.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:20.066496+0000) 2022-09-10T10:48:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:20.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:20.575311+0000) 2022-09-10T10:48:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:21 smithi110 ceph-mon[97633]: pgmap v12131: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:48:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:21 smithi104 ceph-mon[97044]: pgmap v12131: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:48:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:21.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:21.066733+0000) 2022-09-10T10:48:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:21.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:21.575545+0000) 2022-09-10T10:48:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:22.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:22.066870+0000) 2022-09-10T10:48:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:22.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:22.575811+0000) 2022-09-10T10:48:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:23 smithi110 ceph-mon[97633]: pgmap v12132: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:48:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:23 smithi104 ceph-mon[97044]: pgmap v12132: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:48:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:23.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:23.067047+0000) 2022-09-10T10:48:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:23.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:23.576056+0000) 2022-09-10T10:48:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:24 smithi110 ceph-mon[97633]: pgmap v12133: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:48:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:24 smithi104 ceph-mon[97044]: pgmap v12133: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:48:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:24.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:24.067297+0000) 2022-09-10T10:48:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:24.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:24.576295+0000) 2022-09-10T10:48:25.383 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:25.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:25.067595+0000) 2022-09-10T10:48:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:25.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:25.576501+0000) 2022-09-10T10:48:25.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:48:25.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:48:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:26 smithi110 ceph-mon[97633]: pgmap v12134: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:48:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:26 smithi104 ceph-mon[97044]: pgmap v12134: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:48:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:26.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:26.067763+0000) 2022-09-10T10:48:27.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:26.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:26.576748+0000) 2022-09-10T10:48:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:27.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:27.067937+0000) 2022-09-10T10:48:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:27.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:27.576958+0000) 2022-09-10T10:48:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:28.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:28.068201+0000) 2022-09-10T10:48:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:28.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:28.577191+0000) 2022-09-10T10:48:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:29 smithi104 ceph-mon[97044]: pgmap v12135: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:29 smithi110 ceph-mon[97633]: pgmap v12135: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:29.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:29.068426+0000) 2022-09-10T10:48:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:29.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:29.577415+0000) 2022-09-10T10:48:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:30.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:30.068693+0000) 2022-09-10T10:48:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:30.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:30.577603+0000) 2022-09-10T10:48:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:31 smithi110 ceph-mon[97633]: pgmap v12136: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:31 smithi104 ceph-mon[97044]: pgmap v12136: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:31.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:31.068900+0000) 2022-09-10T10:48:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:31.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:31.577823+0000) 2022-09-10T10:48:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:32.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:32.069110+0000) 2022-09-10T10:48:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:32.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:32.578063+0000) 2022-09-10T10:48:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:33 smithi110 ceph-mon[97633]: pgmap v12137: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:33 smithi104 ceph-mon[97044]: pgmap v12137: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:33.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:33.069407+0000) 2022-09-10T10:48:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:33.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:33.578313+0000) 2022-09-10T10:48:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:34.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:34.069655+0000) 2022-09-10T10:48:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:34.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:34.578486+0000) 2022-09-10T10:48:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:35 smithi110 ceph-mon[97633]: pgmap v12138: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:35 smithi104 ceph-mon[97044]: pgmap v12138: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:35.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:35.069866+0000) 2022-09-10T10:48:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:35.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:35.578760+0000) 2022-09-10T10:48:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:36.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:36.070089+0000) 2022-09-10T10:48:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:36.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:36.578992+0000) 2022-09-10T10:48:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:37 smithi110 ceph-mon[97633]: pgmap v12139: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:37 smithi104 ceph-mon[97044]: pgmap v12139: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:37.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:37.070282+0000) 2022-09-10T10:48:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:37.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:37.579216+0000) 2022-09-10T10:48:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:38.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:38.070534+0000) 2022-09-10T10:48:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:38.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:38.579474+0000) 2022-09-10T10:48:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:39 smithi110 ceph-mon[97633]: pgmap v12140: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:39 smithi104 ceph-mon[97044]: pgmap v12140: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:39.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:39.070798+0000) 2022-09-10T10:48:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:39.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:39.579730+0000) 2022-09-10T10:48:40.384 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:40.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:40.071125+0000) 2022-09-10T10:48:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:48:40.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:48:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:40.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:40.579987+0000) 2022-09-10T10:48:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:41 smithi110 ceph-mon[97633]: pgmap v12141: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:41 smithi104 ceph-mon[97044]: pgmap v12141: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:41.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:41.071281+0000) 2022-09-10T10:48:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:41.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:41.580209+0000) 2022-09-10T10:48:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:42 smithi110 ceph-mon[97633]: pgmap v12142: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:42 smithi104 ceph-mon[97044]: pgmap v12142: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:42.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:42.071526+0000) 2022-09-10T10:48:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:42.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:42.580403+0000) 2022-09-10T10:48:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:43.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:43.071747+0000) 2022-09-10T10:48:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:43.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:43.580634+0000) 2022-09-10T10:48:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:44.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:44.072017+0000) 2022-09-10T10:48:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:44.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:44.580875+0000) 2022-09-10T10:48:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:45 smithi110 ceph-mon[97633]: pgmap v12143: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:45 smithi104 ceph-mon[97044]: pgmap v12143: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:45.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:45.072214+0000) 2022-09-10T10:48:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:45.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:45.581126+0000) 2022-09-10T10:48:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:48:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:48:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:48:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:48:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:48:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:48:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:48:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:48:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:46.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:46.072485+0000) 2022-09-10T10:48:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:46.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:46.581359+0000) 2022-09-10T10:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:47 smithi110 ceph-mon[97633]: pgmap v12144: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:47 smithi104 ceph-mon[97044]: pgmap v12144: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:47.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:47.072784+0000) 2022-09-10T10:48:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:47.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:47.581598+0000) 2022-09-10T10:48:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:48.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:48.073046+0000) 2022-09-10T10:48:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:48.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:48.581873+0000) 2022-09-10T10:48:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:49 smithi110 ceph-mon[97633]: pgmap v12145: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:49 smithi104 ceph-mon[97044]: pgmap v12145: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:49.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:49.073183+0000) 2022-09-10T10:48:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:49.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:49.582149+0000) 2022-09-10T10:48:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:50.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:50.073328+0000) 2022-09-10T10:48:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:50.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:50.582396+0000) 2022-09-10T10:48:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:51 smithi110 ceph-mon[97633]: pgmap v12146: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:51.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:51 smithi104 ceph-mon[97044]: pgmap v12146: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:51.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:51.073555+0000) 2022-09-10T10:48:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:51.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:51.582642+0000) 2022-09-10T10:48:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:52 smithi110 ceph-mon[97633]: pgmap v12147: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:52 smithi104 ceph-mon[97044]: pgmap v12147: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:52.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:52.073800+0000) 2022-09-10T10:48:53.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:52.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:52.582862+0000) 2022-09-10T10:48:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:53.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:53.074064+0000) 2022-09-10T10:48:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:53.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:53.583067+0000) 2022-09-10T10:48:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:54.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:54.074264+0000) 2022-09-10T10:48:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:54.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:54.583302+0000) 2022-09-10T10:48:55.391 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:55.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:55.074421+0000) 2022-09-10T10:48:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:55 smithi110 ceph-mon[97633]: pgmap v12148: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:55 smithi104 ceph-mon[97044]: pgmap v12148: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:55.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:55.583533+0000) 2022-09-10T10:48:55.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:48:55.389+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:48:56.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:56.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:56.074657+0000) 2022-09-10T10:48:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:56.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:56.583808+0000) 2022-09-10T10:48:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:57 smithi110 ceph-mon[97633]: pgmap v12149: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:57 smithi104 ceph-mon[97044]: pgmap v12149: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:48:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:57.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:57.074857+0000) 2022-09-10T10:48:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:57.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:57.584022+0000) 2022-09-10T10:48:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:58.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:58.075054+0000) 2022-09-10T10:48:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:58.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:58.584278+0000) 2022-09-10T10:48:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:48:59 smithi110 ceph-mon[97633]: pgmap v12150: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:48:59 smithi104 ceph-mon[97044]: pgmap v12150: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:48:59.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:48:59.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:59.075266+0000) 2022-09-10T10:49:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:48:59.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:48:59.584498+0000) 2022-09-10T10:49:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:00.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:00.075475+0000) 2022-09-10T10:49:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:00.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:00.584674+0000) 2022-09-10T10:49:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:01 smithi110 ceph-mon[97633]: pgmap v12151: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:01 smithi104 ceph-mon[97044]: pgmap v12151: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:01.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:01.075677+0000) 2022-09-10T10:49:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:01.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:01.584876+0000) 2022-09-10T10:49:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:02 smithi110 ceph-mon[97633]: pgmap v12152: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:02 smithi104 ceph-mon[97044]: pgmap v12152: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:02.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:02.075908+0000) 2022-09-10T10:49:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:02.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:02.585132+0000) 2022-09-10T10:49:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:03.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:03.076180+0000) 2022-09-10T10:49:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:03.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:03.585375+0000) 2022-09-10T10:49:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:04.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:04.076461+0000) 2022-09-10T10:49:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:04.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:04.585567+0000) 2022-09-10T10:49:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:05 smithi110 ceph-mon[97633]: pgmap v12153: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:05 smithi104 ceph-mon[97044]: pgmap v12153: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:05.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:05.076649+0000) 2022-09-10T10:49:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:05.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:05.585811+0000) 2022-09-10T10:49:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:06.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:06.076853+0000) 2022-09-10T10:49:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:06.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:06.586064+0000) 2022-09-10T10:49:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:07 smithi110 ceph-mon[97633]: pgmap v12154: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:49:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:07 smithi104 ceph-mon[97044]: pgmap v12154: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:49:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:07.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:07.077132+0000) 2022-09-10T10:49:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:07.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:07.586235+0000) 2022-09-10T10:49:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:08.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:08.077413+0000) 2022-09-10T10:49:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:08.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:08.586419+0000) 2022-09-10T10:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:09 smithi110 ceph-mon[97633]: pgmap v12155: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:09 smithi110 ceph-mon[97633]: osdmap e840: 12 total, 12 up, 12 in 2022-09-10T10:49:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:09 smithi104 ceph-mon[97044]: pgmap v12155: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:49:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:09 smithi104 ceph-mon[97044]: osdmap e840: 12 total, 12 up, 12 in 2022-09-10T10:49:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:09.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:09.077608+0000) 2022-09-10T10:49:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:09.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:09.586645+0000) 2022-09-10T10:49:10.406 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:10.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:10.077835+0000) 2022-09-10T10:49:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:10 smithi110 ceph-mon[97633]: pgmap v12157: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:49:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:10 smithi104 ceph-mon[97044]: pgmap v12157: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:49:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:49:10.404+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:49:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:10.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:10.586890+0000) 2022-09-10T10:49:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:11.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:11.078086+0000) 2022-09-10T10:49:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:11.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:11.587183+0000) 2022-09-10T10:49:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:12 smithi110 ceph-mon[97633]: pgmap v12158: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:49:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:12 smithi104 ceph-mon[97044]: pgmap v12158: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:49:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:12.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:12.078288+0000) 2022-09-10T10:49:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:12.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:12.587380+0000) 2022-09-10T10:49:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:13.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:13.078445+0000) 2022-09-10T10:49:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:13.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:13.587625+0000) 2022-09-10T10:49:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:14 smithi110 ceph-mon[97633]: osdmap e841: 12 total, 12 up, 12 in 2022-09-10T10:49:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:14 smithi104 ceph-mon[97044]: osdmap e841: 12 total, 12 up, 12 in 2022-09-10T10:49:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:14.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:14.078637+0000) 2022-09-10T10:49:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:14.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:14.587838+0000) 2022-09-10T10:49:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:15 smithi110 ceph-mon[97633]: pgmap v12160: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:49:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:15 smithi104 ceph-mon[97044]: pgmap v12160: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:49:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:15.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:15.078862+0000) 2022-09-10T10:49:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:15.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:15.587984+0000) 2022-09-10T10:49:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:16.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:16.079086+0000) 2022-09-10T10:49:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:16.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:16.588252+0000) 2022-09-10T10:49:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:17 smithi110 ceph-mon[97633]: pgmap v12161: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:49:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:17 smithi104 ceph-mon[97044]: pgmap v12161: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:49:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:17.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:17.079399+0000) 2022-09-10T10:49:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:17.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:17.588407+0000) 2022-09-10T10:49:18.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:18.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:18.079663+0000) 2022-09-10T10:49:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:18.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:18.588651+0000) 2022-09-10T10:49:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:19 smithi110 ceph-mon[97633]: pgmap v12162: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 227 B/s wr, 11 op/s 2022-09-10T10:49:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:19 smithi104 ceph-mon[97044]: pgmap v12162: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 227 B/s wr, 11 op/s 2022-09-10T10:49:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:19.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:19.079856+0000) 2022-09-10T10:49:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:19.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:19.588918+0000) 2022-09-10T10:49:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:20 smithi110 ceph-mon[97633]: pgmap v12163: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:49:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:20 smithi104 ceph-mon[97044]: pgmap v12163: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:49:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:20.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:20.080035+0000) 2022-09-10T10:49:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:20.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:20.589208+0000) 2022-09-10T10:49:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:21.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:21.080275+0000) 2022-09-10T10:49:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:21.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:21.589396+0000) 2022-09-10T10:49:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:22 smithi110 ceph-mon[97633]: pgmap v12164: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:49:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:22 smithi104 ceph-mon[97044]: pgmap v12164: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:49:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:22.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:22.080447+0000) 2022-09-10T10:49:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:22.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:22.589631+0000) 2022-09-10T10:49:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:23.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:23.080708+0000) 2022-09-10T10:49:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:23.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:23.589862+0000) 2022-09-10T10:49:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:24.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:24.080975+0000) 2022-09-10T10:49:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:24.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:24.590146+0000) 2022-09-10T10:49:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:25 smithi110 ceph-mon[97633]: pgmap v12165: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:49:25.421 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:25.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:25.081177+0000) 2022-09-10T10:49:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:25 smithi104 ceph-mon[97044]: pgmap v12165: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:49:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:49:25.420+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:49:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:25.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:25.590392+0000) 2022-09-10T10:49:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:26.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:26.081407+0000) 2022-09-10T10:49:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:26.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:26.590618+0000) 2022-09-10T10:49:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:27 smithi110 ceph-mon[97633]: pgmap v12166: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:49:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:27 smithi104 ceph-mon[97044]: pgmap v12166: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:49:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:27.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:27.081677+0000) 2022-09-10T10:49:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:27.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:27.590819+0000) 2022-09-10T10:49:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:28.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:28.081941+0000) 2022-09-10T10:49:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:28.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:28.591035+0000) 2022-09-10T10:49:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:29 smithi110 ceph-mon[97633]: pgmap v12167: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:29 smithi104 ceph-mon[97044]: pgmap v12167: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:29.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:29.082108+0000) 2022-09-10T10:49:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:29.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:29.591224+0000) 2022-09-10T10:49:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:30.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:30.082307+0000) 2022-09-10T10:49:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:30.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:30.591404+0000) 2022-09-10T10:49:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:31 smithi110 ceph-mon[97633]: pgmap v12168: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:31 smithi104 ceph-mon[97044]: pgmap v12168: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:31.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:31.082486+0000) 2022-09-10T10:49:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:31.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:31.591566+0000) 2022-09-10T10:49:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:32.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:32.082740+0000) 2022-09-10T10:49:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:32.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:32.591805+0000) 2022-09-10T10:49:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:33 smithi110 ceph-mon[97633]: pgmap v12169: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:33 smithi104 ceph-mon[97044]: pgmap v12169: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:33.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:33.082929+0000) 2022-09-10T10:49:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:33.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:33.592067+0000) 2022-09-10T10:49:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:34 smithi110 ceph-mon[97633]: pgmap v12170: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:34 smithi104 ceph-mon[97044]: pgmap v12170: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:34.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:34.083140+0000) 2022-09-10T10:49:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:34.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:34.592276+0000) 2022-09-10T10:49:35.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:35.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:35.083344+0000) 2022-09-10T10:49:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:35.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:35.592478+0000) 2022-09-10T10:49:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:36 smithi110 ceph-mon[97633]: pgmap v12171: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:36 smithi104 ceph-mon[97044]: pgmap v12171: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:36.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:36.083574+0000) 2022-09-10T10:49:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:36.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:36.592748+0000) 2022-09-10T10:49:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:37.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:37.083824+0000) 2022-09-10T10:49:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:37.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:37.592985+0000) 2022-09-10T10:49:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:38.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:38.084134+0000) 2022-09-10T10:49:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:38.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:38.593231+0000) 2022-09-10T10:49:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:39 smithi110 ceph-mon[97633]: pgmap v12172: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:39 smithi104 ceph-mon[97044]: pgmap v12172: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:39.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:39.084274+0000) 2022-09-10T10:49:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:39.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:39.593402+0000) 2022-09-10T10:49:40.437 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:40.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:40.084457+0000) 2022-09-10T10:49:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:49:40.435+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:49:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:40.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:40.593580+0000) 2022-09-10T10:49:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:41 smithi110 ceph-mon[97633]: pgmap v12173: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:41 smithi104 ceph-mon[97044]: pgmap v12173: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:41.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:41.084677+0000) 2022-09-10T10:49:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:41.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:41.593823+0000) 2022-09-10T10:49:42.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:42 smithi110 ceph-mon[97633]: pgmap v12174: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:42 smithi104 ceph-mon[97044]: pgmap v12174: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:42.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:42.084961+0000) 2022-09-10T10:49:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:42.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:42.594088+0000) 2022-09-10T10:49:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:43.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:43.085180+0000) 2022-09-10T10:49:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:43.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:43.594330+0000) 2022-09-10T10:49:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:44.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:44.085353+0000) 2022-09-10T10:49:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:44.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:44.594574+0000) 2022-09-10T10:49:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:45 smithi110 ceph-mon[97633]: pgmap v12175: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:45 smithi104 ceph-mon[97044]: pgmap v12175: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:45.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:45.085559+0000) 2022-09-10T10:49:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:45.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:45.594844+0000) 2022-09-10T10:49:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:49:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:49:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:49:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:49:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:49:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:46.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:46.085727+0000) 2022-09-10T10:49:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:46.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:46.595040+0000) 2022-09-10T10:49:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:47 smithi110 ceph-mon[97633]: pgmap v12176: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:47 smithi104 ceph-mon[97044]: pgmap v12176: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:47.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:47.085912+0000) 2022-09-10T10:49:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:47.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:47.595298+0000) 2022-09-10T10:49:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:48.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:48.086185+0000) 2022-09-10T10:49:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:48.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:48.595502+0000) 2022-09-10T10:49:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:49 smithi110 ceph-mon[97633]: pgmap v12177: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:49 smithi104 ceph-mon[97044]: pgmap v12177: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:49.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:49.086306+0000) 2022-09-10T10:49:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:49.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:49.595700+0000) 2022-09-10T10:49:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:50.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:50.086452+0000) 2022-09-10T10:49:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:50.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:50.595891+0000) 2022-09-10T10:49:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:51 smithi110 ceph-mon[97633]: pgmap v12178: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:51 smithi104 ceph-mon[97044]: pgmap v12178: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:51.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:51.086619+0000) 2022-09-10T10:49:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:51.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:51.596166+0000) 2022-09-10T10:49:52.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:52 smithi110 ceph-mon[97633]: pgmap v12179: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:52 smithi104 ceph-mon[97044]: pgmap v12179: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:52.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:52.086829+0000) 2022-09-10T10:49:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:52.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:52.596337+0000) 2022-09-10T10:49:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:53.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:53.087084+0000) 2022-09-10T10:49:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:53.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:53.596511+0000) 2022-09-10T10:49:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:54.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:54.087408+0000) 2022-09-10T10:49:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:54.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:54.596772+0000) 2022-09-10T10:49:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:55 smithi110 ceph-mon[97633]: pgmap v12180: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:55.443 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:55.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:55.087650+0000) 2022-09-10T10:49:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:55 smithi104 ceph-mon[97044]: pgmap v12180: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:55.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:55.597000+0000) 2022-09-10T10:49:55.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:49:55.441+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:49:56.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:56 smithi110 ceph-mon[97633]: pgmap v12181: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:56 smithi104 ceph-mon[97044]: pgmap v12181: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:49:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:56.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:56.087886+0000) 2022-09-10T10:49:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:56.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:56.597244+0000) 2022-09-10T10:49:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:57.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:57.088136+0000) 2022-09-10T10:49:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:57.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:57.597447+0000) 2022-09-10T10:49:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:58.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:58.088394+0000) 2022-09-10T10:49:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:58.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:58.597690+0000) 2022-09-10T10:49:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:49:59 smithi110 ceph-mon[97633]: pgmap v12182: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:49:59 smithi104 ceph-mon[97044]: pgmap v12182: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:49:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:49:59.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:59.088643+0000) 2022-09-10T10:50:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:49:59.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:49:59.597965+0000) 2022-09-10T10:50:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:50:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T10:50:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:00.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:00.088837+0000) 2022-09-10T10:50:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:00.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:00.598146+0000) 2022-09-10T10:50:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:01 smithi110 ceph-mon[97633]: pgmap v12183: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:01.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:01 smithi104 ceph-mon[97044]: pgmap v12183: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:01.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:01.089025+0000) 2022-09-10T10:50:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:01.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:01.598380+0000) 2022-09-10T10:50:02.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:02 smithi110 ceph-mon[97633]: pgmap v12184: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:02 smithi104 ceph-mon[97044]: pgmap v12184: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:02.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:02.089287+0000) 2022-09-10T10:50:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:02.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:02.598657+0000) 2022-09-10T10:50:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:03.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:03.089501+0000) 2022-09-10T10:50:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:03.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:03.598948+0000) 2022-09-10T10:50:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:04.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:04.089628+0000) 2022-09-10T10:50:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:04.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:04.599162+0000) 2022-09-10T10:50:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:05 smithi110 ceph-mon[97633]: pgmap v12185: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:05 smithi104 ceph-mon[97044]: pgmap v12185: 129 pgs: 129 active+clean; 9.0 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:05.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:05.089790+0000) 2022-09-10T10:50:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:05.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:05.599306+0000) 2022-09-10T10:50:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:06 smithi110 ceph-mon[97633]: pgmap v12186: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:50:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:06 smithi104 ceph-mon[97044]: pgmap v12186: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:50:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:06.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:06.089947+0000) 2022-09-10T10:50:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:06.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:06.599466+0000) 2022-09-10T10:50:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:07.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:07.090083+0000) 2022-09-10T10:50:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:07.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:07.599733+0000) 2022-09-10T10:50:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:08.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:08.090305+0000) 2022-09-10T10:50:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:08.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:08.599880+0000) 2022-09-10T10:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:09 smithi110 ceph-mon[97633]: pgmap v12187: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T10:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:09 smithi110 ceph-mon[97633]: osdmap e842: 12 total, 12 up, 12 in 2022-09-10T10:50:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:09 smithi104 ceph-mon[97044]: pgmap v12187: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T10:50:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:09 smithi104 ceph-mon[97044]: osdmap e842: 12 total, 12 up, 12 in 2022-09-10T10:50:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:09.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:09.090524+0000) 2022-09-10T10:50:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:09.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:09.600122+0000) 2022-09-10T10:50:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:10 smithi110 ceph-mon[97633]: pgmap v12189: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:50:10.458 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:10.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:10.090757+0000) 2022-09-10T10:50:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:10 smithi104 ceph-mon[97044]: pgmap v12189: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:50:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:50:10.457+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:50:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:10.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:10.600344+0000) 2022-09-10T10:50:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:11.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:11.090960+0000) 2022-09-10T10:50:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:11.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:11.600570+0000) 2022-09-10T10:50:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:12 smithi110 ceph-mon[97633]: pgmap v12190: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:50:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:12 smithi104 ceph-mon[97044]: pgmap v12190: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:50:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:12.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:12.091202+0000) 2022-09-10T10:50:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:12.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:12.600748+0000) 2022-09-10T10:50:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:13.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:13.091395+0000) 2022-09-10T10:50:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:13.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:13.600985+0000) 2022-09-10T10:50:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:14 smithi110 ceph-mon[97633]: osdmap e843: 12 total, 12 up, 12 in 2022-09-10T10:50:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:14 smithi104 ceph-mon[97044]: osdmap e843: 12 total, 12 up, 12 in 2022-09-10T10:50:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:14.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:14.091530+0000) 2022-09-10T10:50:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:14.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:14.601235+0000) 2022-09-10T10:50:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:15 smithi110 ceph-mon[97633]: pgmap v12192: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:50:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:15 smithi104 ceph-mon[97044]: pgmap v12192: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:50:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:15.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:15.091766+0000) 2022-09-10T10:50:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:15.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:15.601449+0000) 2022-09-10T10:50:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:16 smithi110 ceph-mon[97633]: pgmap v12193: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:50:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:16 smithi104 ceph-mon[97044]: pgmap v12193: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:50:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:16.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:16.092004+0000) 2022-09-10T10:50:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:16.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:16.601699+0000) 2022-09-10T10:50:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:17.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:17.092210+0000) 2022-09-10T10:50:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:17.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:17.601947+0000) 2022-09-10T10:50:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:18 smithi110 ceph-mon[97633]: pgmap v12194: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:50:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:18 smithi104 ceph-mon[97044]: pgmap v12194: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:50:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:18.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:18.092441+0000) 2022-09-10T10:50:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:18.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:18.602161+0000) 2022-09-10T10:50:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:19.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:19.092660+0000) 2022-09-10T10:50:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:19.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:19.602426+0000) 2022-09-10T10:50:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:20 smithi110 ceph-mon[97633]: pgmap v12195: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:50:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:20 smithi104 ceph-mon[97044]: pgmap v12195: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:50:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:20.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:20.092815+0000) 2022-09-10T10:50:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:20.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:20.602613+0000) 2022-09-10T10:50:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:21.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:21.093033+0000) 2022-09-10T10:50:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:21.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:21.602851+0000) 2022-09-10T10:50:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:22 smithi110 ceph-mon[97633]: pgmap v12196: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:50:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:22 smithi104 ceph-mon[97044]: pgmap v12196: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:50:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:22.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:22.093275+0000) 2022-09-10T10:50:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:22.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:22.603154+0000) 2022-09-10T10:50:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:23.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:23.093515+0000) 2022-09-10T10:50:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:23.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:23.603358+0000) 2022-09-10T10:50:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:24 smithi110 ceph-mon[97633]: pgmap v12197: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:50:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:24 smithi104 ceph-mon[97044]: pgmap v12197: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:50:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:24.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:24.093763+0000) 2022-09-10T10:50:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:24.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:24.603579+0000) 2022-09-10T10:50:25.471 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:25.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:25.093967+0000) 2022-09-10T10:50:25.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:50:25.469+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:50:25.899 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:25.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:25.603801+0000) 2022-09-10T10:50:26.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:26 smithi110 ceph-mon[97633]: pgmap v12198: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:50:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:26 smithi104 ceph-mon[97044]: pgmap v12198: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:50:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:26.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:26.094140+0000) 2022-09-10T10:50:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:26.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:26.603949+0000) 2022-09-10T10:50:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:27.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:27.094306+0000) 2022-09-10T10:50:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:27.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:27.604193+0000) 2022-09-10T10:50:28.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:28 smithi110 ceph-mon[97633]: pgmap v12199: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:28 smithi104 ceph-mon[97044]: pgmap v12199: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:28.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:28.094477+0000) 2022-09-10T10:50:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:28.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:28.604397+0000) 2022-09-10T10:50:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:29.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:29.094684+0000) 2022-09-10T10:50:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:29.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:29.604562+0000) 2022-09-10T10:50:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:30 smithi104 ceph-mon[97044]: pgmap v12200: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:30.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:30.094916+0000) 2022-09-10T10:50:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:30 smithi110 ceph-mon[97633]: pgmap v12200: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:30.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:30.604742+0000) 2022-09-10T10:50:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:31.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:31.095150+0000) 2022-09-10T10:50:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:31.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:31.604971+0000) 2022-09-10T10:50:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:32 smithi104 ceph-mon[97044]: pgmap v12201: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:32.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:32.095310+0000) 2022-09-10T10:50:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:32 smithi110 ceph-mon[97633]: pgmap v12201: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:32.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:32.605227+0000) 2022-09-10T10:50:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:33.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:33.095547+0000) 2022-09-10T10:50:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:33.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:33.605458+0000) 2022-09-10T10:50:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:34 smithi110 ceph-mon[97633]: pgmap v12202: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:34.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:34 smithi104 ceph-mon[97044]: pgmap v12202: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:34.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:34.095790+0000) 2022-09-10T10:50:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:34.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:34.605682+0000) 2022-09-10T10:50:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:35.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:35.096000+0000) 2022-09-10T10:50:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:35.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:35.605937+0000) 2022-09-10T10:50:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:36 smithi110 ceph-mon[97633]: pgmap v12203: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:36 smithi104 ceph-mon[97044]: pgmap v12203: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:36.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:36.096207+0000) 2022-09-10T10:50:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:36.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:36.606172+0000) 2022-09-10T10:50:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:37.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:37.096468+0000) 2022-09-10T10:50:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:37.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:37.606307+0000) 2022-09-10T10:50:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:38 smithi104 ceph-mon[97044]: pgmap v12204: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:38.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:38.096619+0000) 2022-09-10T10:50:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:38 smithi110 ceph-mon[97633]: pgmap v12204: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:38.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:38.606570+0000) 2022-09-10T10:50:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:39.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:39.096860+0000) 2022-09-10T10:50:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:39.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:39.606821+0000) 2022-09-10T10:50:40.475 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:40.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:40.097065+0000) 2022-09-10T10:50:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:40 smithi104 ceph-mon[97044]: pgmap v12205: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:40 smithi110 ceph-mon[97633]: pgmap v12205: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:40.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:40.607051+0000) 2022-09-10T10:50:40.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:50:40.473+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:50:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:41.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:41.097254+0000) 2022-09-10T10:50:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:41.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:41.607237+0000) 2022-09-10T10:50:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:42 smithi104 ceph-mon[97044]: pgmap v12206: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:42.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:42.097476+0000) 2022-09-10T10:50:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:42 smithi110 ceph-mon[97633]: pgmap v12206: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:42.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:42.607405+0000) 2022-09-10T10:50:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:43.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:43.097697+0000) 2022-09-10T10:50:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:43.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:43.607642+0000) 2022-09-10T10:50:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:44 smithi110 ceph-mon[97633]: pgmap v12207: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:44 smithi104 ceph-mon[97044]: pgmap v12207: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:44.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:44.097855+0000) 2022-09-10T10:50:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:44.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:44.607902+0000) 2022-09-10T10:50:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:45.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:45.098088+0000) 2022-09-10T10:50:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:45.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:45.608152+0000) 2022-09-10T10:50:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:46 smithi104 ceph-mon[97044]: pgmap v12208: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:50:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:46.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:46.098292+0000) 2022-09-10T10:50:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:46 smithi110 ceph-mon[97633]: pgmap v12208: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:50:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:50:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:50:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:50:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:46.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:46.608293+0000) 2022-09-10T10:50:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:47.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:47.098488+0000) 2022-09-10T10:50:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:47.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:47.608527+0000) 2022-09-10T10:50:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:48 smithi104 ceph-mon[97044]: pgmap v12209: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:48.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:48.098740+0000) 2022-09-10T10:50:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:48 smithi110 ceph-mon[97633]: pgmap v12209: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:48.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:48.608670+0000) 2022-09-10T10:50:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:49.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:49.098945+0000) 2022-09-10T10:50:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:49.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:49.608854+0000) 2022-09-10T10:50:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:50 smithi104 ceph-mon[97044]: pgmap v12210: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:50.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:50.099193+0000) 2022-09-10T10:50:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:50 smithi110 ceph-mon[97633]: pgmap v12210: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:50.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:50.609062+0000) 2022-09-10T10:50:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:51.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:51.099348+0000) 2022-09-10T10:50:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:51.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:51.609256+0000) 2022-09-10T10:50:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:52 smithi104 ceph-mon[97044]: pgmap v12211: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:52.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:52.099476+0000) 2022-09-10T10:50:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:52 smithi110 ceph-mon[97633]: pgmap v12211: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:52.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:52.609450+0000) 2022-09-10T10:50:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:53.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:53.099676+0000) 2022-09-10T10:50:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:53.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:53.609702+0000) 2022-09-10T10:50:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:54 smithi110 ceph-mon[97633]: pgmap v12212: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:54 smithi104 ceph-mon[97044]: pgmap v12212: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:54.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:54.099934+0000) 2022-09-10T10:50:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:54.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:54.609953+0000) 2022-09-10T10:50:55.484 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:55.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:55.100149+0000) 2022-09-10T10:50:55.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:50:55.482+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:50:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:55.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:55.610191+0000) 2022-09-10T10:50:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:56 smithi104 ceph-mon[97044]: pgmap v12213: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:56.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:56.100361+0000) 2022-09-10T10:50:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:56 smithi110 ceph-mon[97633]: pgmap v12213: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:50:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:56.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:56.610426+0000) 2022-09-10T10:50:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:57.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:57.100581+0000) 2022-09-10T10:50:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:57.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:57.610641+0000) 2022-09-10T10:50:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:50:58 smithi104 ceph-mon[97044]: pgmap v12214: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:58.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:58.100915+0000) 2022-09-10T10:50:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:50:58 smithi110 ceph-mon[97633]: pgmap v12214: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:50:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:58.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:58.610887+0000) 2022-09-10T10:50:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:50:59.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:59.101225+0000) 2022-09-10T10:51:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:50:59.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:50:59.611078+0000) 2022-09-10T10:51:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:00 smithi104 ceph-mon[97044]: pgmap v12215: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:00.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:00.101390+0000) 2022-09-10T10:51:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:00 smithi110 ceph-mon[97633]: pgmap v12215: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:00.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:00.611331+0000) 2022-09-10T10:51:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:01.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:01.101543+0000) 2022-09-10T10:51:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:01.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:01.611580+0000) 2022-09-10T10:51:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:02 smithi104 ceph-mon[97044]: pgmap v12216: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:02.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:02.101786+0000) 2022-09-10T10:51:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:02 smithi110 ceph-mon[97633]: pgmap v12216: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:02.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:02.611823+0000) 2022-09-10T10:51:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:03.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:03.101989+0000) 2022-09-10T10:51:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:03.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:03.612086+0000) 2022-09-10T10:51:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:04 smithi110 ceph-mon[97633]: pgmap v12217: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:04 smithi104 ceph-mon[97044]: pgmap v12217: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:04.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:04.102256+0000) 2022-09-10T10:51:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:04.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:04.612279+0000) 2022-09-10T10:51:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:05.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:05.102397+0000) 2022-09-10T10:51:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:05.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:05.612487+0000) 2022-09-10T10:51:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:06 smithi104 ceph-mon[97044]: pgmap v12218: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:51:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:06.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:06.102576+0000) 2022-09-10T10:51:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:06 smithi110 ceph-mon[97633]: pgmap v12218: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:51:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:06.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:06.612729+0000) 2022-09-10T10:51:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:07.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:07.102759+0000) 2022-09-10T10:51:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:07.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:07.612908+0000) 2022-09-10T10:51:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:08 smithi104 ceph-mon[97044]: pgmap v12219: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:51:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:08.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:08.102983+0000) 2022-09-10T10:51:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:08 smithi110 ceph-mon[97633]: pgmap v12219: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:51:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:08.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:08.613127+0000) 2022-09-10T10:51:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:09 smithi104 ceph-mon[97044]: osdmap e844: 12 total, 12 up, 12 in 2022-09-10T10:51:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:09.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:09.103232+0000) 2022-09-10T10:51:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:09 smithi110 ceph-mon[97633]: osdmap e844: 12 total, 12 up, 12 in 2022-09-10T10:51:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:09.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:09.613370+0000) 2022-09-10T10:51:10.503 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:10.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:10.103468+0000) 2022-09-10T10:51:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:10 smithi104 ceph-mon[97044]: pgmap v12221: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:51:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:10 smithi110 ceph-mon[97633]: pgmap v12221: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:51:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:10.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:10.613602+0000) 2022-09-10T10:51:10.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:51:10.496+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:51:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:11.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:11.103633+0000) 2022-09-10T10:51:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:11.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:11.613793+0000) 2022-09-10T10:51:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:12 smithi104 ceph-mon[97044]: pgmap v12222: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:51:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:12.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:12.103826+0000) 2022-09-10T10:51:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:12 smithi110 ceph-mon[97633]: pgmap v12222: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:51:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:12.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:12.614008+0000) 2022-09-10T10:51:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:13.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:13.104007+0000) 2022-09-10T10:51:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:13.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:13.614247+0000) 2022-09-10T10:51:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:14 smithi110 ceph-mon[97633]: osdmap e845: 12 total, 12 up, 12 in 2022-09-10T10:51:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:14 smithi110 ceph-mon[97633]: pgmap v12224: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:51:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:14 smithi104 ceph-mon[97044]: osdmap e845: 12 total, 12 up, 12 in 2022-09-10T10:51:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:14 smithi104 ceph-mon[97044]: pgmap v12224: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:51:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:14.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:14.104251+0000) 2022-09-10T10:51:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:14.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:14.614479+0000) 2022-09-10T10:51:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:15.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:15.104438+0000) 2022-09-10T10:51:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:15.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:15.614737+0000) 2022-09-10T10:51:16.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:16 smithi104 ceph-mon[97044]: pgmap v12225: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:51:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:16.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:16.104666+0000) 2022-09-10T10:51:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:16 smithi110 ceph-mon[97633]: pgmap v12225: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:51:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:16.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:16.614934+0000) 2022-09-10T10:51:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:17.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:17.104926+0000) 2022-09-10T10:51:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:17.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:17.615193+0000) 2022-09-10T10:51:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:18 smithi104 ceph-mon[97044]: pgmap v12226: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:51:18.567 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:18.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:18.105168+0000) 2022-09-10T10:51:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:18 smithi110 ceph-mon[97633]: pgmap v12226: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:51:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:18.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:18.615373+0000) 2022-09-10T10:51:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:19.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:19.105429+0000) 2022-09-10T10:51:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:19.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:19.615554+0000) 2022-09-10T10:51:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:20 smithi104 ceph-mon[97044]: pgmap v12227: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:51:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:20.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:20.105632+0000) 2022-09-10T10:51:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:20 smithi110 ceph-mon[97633]: pgmap v12227: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:51:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:20.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:20.615740+0000) 2022-09-10T10:51:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:21.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:21.105894+0000) 2022-09-10T10:51:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:21.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:21.615990+0000) 2022-09-10T10:51:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:22 smithi104 ceph-mon[97044]: pgmap v12228: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:51:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:22.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:22.106146+0000) 2022-09-10T10:51:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:22 smithi110 ceph-mon[97633]: pgmap v12228: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:51:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:22.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:22.616222+0000) 2022-09-10T10:51:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:23.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:23.106312+0000) 2022-09-10T10:51:24.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:23.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:23.616398+0000) 2022-09-10T10:51:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:24 smithi110 ceph-mon[97633]: pgmap v12229: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:51:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:24 smithi104 ceph-mon[97044]: pgmap v12229: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:51:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:24.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:24.106483+0000) 2022-09-10T10:51:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:24.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:24.616659+0000) 2022-09-10T10:51:25.499 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:25.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:25.106723+0000) 2022-09-10T10:51:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:25.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:25.616863+0000) 2022-09-10T10:51:25.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:51:25.497+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:51:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:26 smithi104 ceph-mon[97044]: pgmap v12230: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:51:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:26.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:26.106876+0000) 2022-09-10T10:51:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:26 smithi110 ceph-mon[97633]: pgmap v12230: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:51:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:26.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:26.617138+0000) 2022-09-10T10:51:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:27.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:27.107048+0000) 2022-09-10T10:51:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:27.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:27.617375+0000) 2022-09-10T10:51:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:28 smithi104 ceph-mon[97044]: pgmap v12231: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:28.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:28.107243+0000) 2022-09-10T10:51:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:28 smithi110 ceph-mon[97633]: pgmap v12231: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:28.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:28.617629+0000) 2022-09-10T10:51:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:29.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:29.107397+0000) 2022-09-10T10:51:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:29.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:29.617836+0000) 2022-09-10T10:51:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:30 smithi104 ceph-mon[97044]: pgmap v12232: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:30.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:30.107556+0000) 2022-09-10T10:51:30.666 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:30 smithi110 ceph-mon[97633]: pgmap v12232: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:30.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:30.618066+0000) 2022-09-10T10:51:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:31.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:31.107769+0000) 2022-09-10T10:51:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:31.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:31.618315+0000) 2022-09-10T10:51:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:32 smithi104 ceph-mon[97044]: pgmap v12233: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:32.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:32.107948+0000) 2022-09-10T10:51:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:32 smithi110 ceph-mon[97633]: pgmap v12233: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:32.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:32.618526+0000) 2022-09-10T10:51:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:33.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:33.108153+0000) 2022-09-10T10:51:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:33.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:33.618810+0000) 2022-09-10T10:51:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:34 smithi110 ceph-mon[97633]: pgmap v12234: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:34 smithi104 ceph-mon[97044]: pgmap v12234: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:34.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:34.108352+0000) 2022-09-10T10:51:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:34.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:34.619008+0000) 2022-09-10T10:51:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:35.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:35.108497+0000) 2022-09-10T10:51:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:35.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:35.619240+0000) 2022-09-10T10:51:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:36 smithi104 ceph-mon[97044]: pgmap v12235: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:36.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:36.108645+0000) 2022-09-10T10:51:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:36 smithi110 ceph-mon[97633]: pgmap v12235: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:36.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:36.619484+0000) 2022-09-10T10:51:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:37.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:37.108771+0000) 2022-09-10T10:51:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:37.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:37.619729+0000) 2022-09-10T10:51:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:38 smithi104 ceph-mon[97044]: pgmap v12236: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:38.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:38.108989+0000) 2022-09-10T10:51:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:38 smithi110 ceph-mon[97633]: pgmap v12236: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:38.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:38.619932+0000) 2022-09-10T10:51:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:39.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:39.109187+0000) 2022-09-10T10:51:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:39.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:39.620168+0000) 2022-09-10T10:51:40.508 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:40.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:40.109416+0000) 2022-09-10T10:51:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:40 smithi104 ceph-mon[97044]: pgmap v12237: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:40 smithi110 ceph-mon[97633]: pgmap v12237: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:51:40.506+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:51:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:40.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:40.620413+0000) 2022-09-10T10:51:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:41.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:41.109651+0000) 2022-09-10T10:51:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:41.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:41.620632+0000) 2022-09-10T10:51:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:42 smithi104 ceph-mon[97044]: pgmap v12238: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:42.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:42.109850+0000) 2022-09-10T10:51:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:42 smithi110 ceph-mon[97633]: pgmap v12238: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:42.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:42.620856+0000) 2022-09-10T10:51:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:43.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:43.110090+0000) 2022-09-10T10:51:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:43.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:43.621063+0000) 2022-09-10T10:51:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:44 smithi110 ceph-mon[97633]: pgmap v12239: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:44 smithi104 ceph-mon[97044]: pgmap v12239: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:44.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:44.110297+0000) 2022-09-10T10:51:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:44.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:44.621299+0000) 2022-09-10T10:51:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:45.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:45.110583+0000) 2022-09-10T10:51:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:45.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:45.621548+0000) 2022-09-10T10:51:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:46 smithi104 ceph-mon[97044]: pgmap v12240: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:51:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:51:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:51:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:51:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:46.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:46.110811+0000) 2022-09-10T10:51:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:46 smithi110 ceph-mon[97633]: pgmap v12240: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:51:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:51:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:51:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:51:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:46.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:46.621734+0000) 2022-09-10T10:51:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:47.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:47.110974+0000) 2022-09-10T10:51:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:47.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:47.621981+0000) 2022-09-10T10:51:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:48 smithi104 ceph-mon[97044]: pgmap v12241: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:48.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:48.111208+0000) 2022-09-10T10:51:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:48 smithi110 ceph-mon[97633]: pgmap v12241: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:48.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:48.622214+0000) 2022-09-10T10:51:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:49.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:49.111436+0000) 2022-09-10T10:51:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:49.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:49.622458+0000) 2022-09-10T10:51:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:50 smithi104 ceph-mon[97044]: pgmap v12242: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:50.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:50.111616+0000) 2022-09-10T10:51:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:50 smithi110 ceph-mon[97633]: pgmap v12242: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:50.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:50.622705+0000) 2022-09-10T10:51:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:51.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:51.111825+0000) 2022-09-10T10:51:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:51.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:51.622966+0000) 2022-09-10T10:51:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:52 smithi104 ceph-mon[97044]: pgmap v12243: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:52.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:52.112077+0000) 2022-09-10T10:51:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:52 smithi110 ceph-mon[97633]: pgmap v12243: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:52.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:52.623220+0000) 2022-09-10T10:51:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:53.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:53.112365+0000) 2022-09-10T10:51:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:53.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:53.623497+0000) 2022-09-10T10:51:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:54 smithi110 ceph-mon[97633]: pgmap v12244: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:54 smithi104 ceph-mon[97044]: pgmap v12244: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:54.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:54.112579+0000) 2022-09-10T10:51:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:54.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:54.623743+0000) 2022-09-10T10:51:55.515 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:55.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:55.112827+0000) 2022-09-10T10:51:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:55.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:55.623977+0000) 2022-09-10T10:51:55.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:51:55.513+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:51:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:56 smithi104 ceph-mon[97044]: pgmap v12245: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:56.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:56.113022+0000) 2022-09-10T10:51:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:56 smithi110 ceph-mon[97633]: pgmap v12245: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:51:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:56.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:56.624157+0000) 2022-09-10T10:51:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:57.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:57.113302+0000) 2022-09-10T10:51:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:57.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:57.624361+0000) 2022-09-10T10:51:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:51:58 smithi104 ceph-mon[97044]: pgmap v12246: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:58.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:58.113513+0000) 2022-09-10T10:51:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:51:58 smithi110 ceph-mon[97633]: pgmap v12246: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:51:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:58.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:58.624603+0000) 2022-09-10T10:51:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:51:59.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:59.113709+0000) 2022-09-10T10:52:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:51:59.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:51:59.624827+0000) 2022-09-10T10:52:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:00 smithi104 ceph-mon[97044]: pgmap v12247: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:00.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:00.113897+0000) 2022-09-10T10:52:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:00 smithi110 ceph-mon[97633]: pgmap v12247: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:00.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:00.625053+0000) 2022-09-10T10:52:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:01.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:01.114047+0000) 2022-09-10T10:52:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:01.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:01.625337+0000) 2022-09-10T10:52:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:02 smithi104 ceph-mon[97044]: pgmap v12248: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:02.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:02.114237+0000) 2022-09-10T10:52:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:02 smithi110 ceph-mon[97633]: pgmap v12248: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:02.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:02.625569+0000) 2022-09-10T10:52:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:03.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:03.114504+0000) 2022-09-10T10:52:04.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:03.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:03.625814+0000) 2022-09-10T10:52:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:04 smithi110 ceph-mon[97633]: pgmap v12249: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:04 smithi104 ceph-mon[97044]: pgmap v12249: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:04.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:04.114748+0000) 2022-09-10T10:52:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:04.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:04.626069+0000) 2022-09-10T10:52:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:05.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:05.115009+0000) 2022-09-10T10:52:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:05.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:05.626292+0000) 2022-09-10T10:52:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:06 smithi104 ceph-mon[97044]: pgmap v12250: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:52:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:06.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:06.115246+0000) 2022-09-10T10:52:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:06 smithi110 ceph-mon[97633]: pgmap v12250: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:52:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:06.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:06.626465+0000) 2022-09-10T10:52:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:07.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:07.115486+0000) 2022-09-10T10:52:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:07.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:07.626675+0000) 2022-09-10T10:52:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:08 smithi104 ceph-mon[97044]: pgmap v12251: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:52:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:08.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:08.115757+0000) 2022-09-10T10:52:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:08 smithi110 ceph-mon[97633]: pgmap v12251: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:52:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:08.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:08.626915+0000) 2022-09-10T10:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:09 smithi104 ceph-mon[97044]: osdmap e846: 12 total, 12 up, 12 in 2022-09-10T10:52:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:09.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:09.115964+0000) 2022-09-10T10:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:09 smithi110 ceph-mon[97633]: osdmap e846: 12 total, 12 up, 12 in 2022-09-10T10:52:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:09.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:09.627090+0000) 2022-09-10T10:52:10.520 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:10.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:10.116201+0000) 2022-09-10T10:52:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:10 smithi104 ceph-mon[97044]: pgmap v12253: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:52:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:10 smithi110 ceph-mon[97633]: pgmap v12253: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:52:10.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:52:10.518+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:52:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:10.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:10.627318+0000) 2022-09-10T10:52:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:11.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:11.116378+0000) 2022-09-10T10:52:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:11.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:11.627503+0000) 2022-09-10T10:52:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:12 smithi104 ceph-mon[97044]: pgmap v12254: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:52:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:12.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:12.116601+0000) 2022-09-10T10:52:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:12 smithi110 ceph-mon[97633]: pgmap v12254: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:52:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:12.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:12.627753+0000) 2022-09-10T10:52:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:13.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:13.116824+0000) 2022-09-10T10:52:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:13.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:13.627996+0000) 2022-09-10T10:52:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:14 smithi110 ceph-mon[97633]: osdmap e847: 12 total, 12 up, 12 in 2022-09-10T10:52:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:14 smithi110 ceph-mon[97633]: pgmap v12256: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:52:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:14 smithi104 ceph-mon[97044]: osdmap e847: 12 total, 12 up, 12 in 2022-09-10T10:52:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:14 smithi104 ceph-mon[97044]: pgmap v12256: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:52:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:14.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:14.117048+0000) 2022-09-10T10:52:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:14.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:14.628231+0000) 2022-09-10T10:52:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:15.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:15.117161+0000) 2022-09-10T10:52:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:15.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:15.628428+0000) 2022-09-10T10:52:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:16 smithi104 ceph-mon[97044]: pgmap v12257: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:52:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:16.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:16.117378+0000) 2022-09-10T10:52:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:16 smithi110 ceph-mon[97633]: pgmap v12257: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:52:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:16.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:16.628680+0000) 2022-09-10T10:52:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:17.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:17.117573+0000) 2022-09-10T10:52:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:17.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:17.628922+0000) 2022-09-10T10:52:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:18 smithi104 ceph-mon[97044]: pgmap v12258: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:52:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:18.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:18.117803+0000) 2022-09-10T10:52:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:18 smithi110 ceph-mon[97633]: pgmap v12258: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:52:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:18.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:18.629171+0000) 2022-09-10T10:52:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:19.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:19.118054+0000) 2022-09-10T10:52:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:19.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:19.629313+0000) 2022-09-10T10:52:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:20 smithi104 ceph-mon[97044]: pgmap v12259: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:52:20.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:20.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:20.118283+0000) 2022-09-10T10:52:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:20 smithi110 ceph-mon[97633]: pgmap v12259: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:52:21.064 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:20.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:20.629483+0000) 2022-09-10T10:52:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:21.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:21.118524+0000) 2022-09-10T10:52:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:21.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:21.629673+0000) 2022-09-10T10:52:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:22 smithi104 ceph-mon[97044]: pgmap v12260: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:52:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:22.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:22.118688+0000) 2022-09-10T10:52:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:22 smithi110 ceph-mon[97633]: pgmap v12260: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:52:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:22.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:22.629817+0000) 2022-09-10T10:52:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:23.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:23.118923+0000) 2022-09-10T10:52:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:23.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:23.630016+0000) 2022-09-10T10:52:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:24 smithi110 ceph-mon[97633]: pgmap v12261: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:52:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:24 smithi104 ceph-mon[97044]: pgmap v12261: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:52:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:24.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:24.119076+0000) 2022-09-10T10:52:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:24.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:24.630242+0000) 2022-09-10T10:52:25.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:25.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:25.119262+0000) 2022-09-10T10:52:25.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:52:25.525+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:52:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:25.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:25.630516+0000) 2022-09-10T10:52:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:26 smithi104 ceph-mon[97044]: pgmap v12262: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:52:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:26.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:26.119476+0000) 2022-09-10T10:52:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:26 smithi110 ceph-mon[97633]: pgmap v12262: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:52:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:26.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:26.630742+0000) 2022-09-10T10:52:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:27.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:27.119719+0000) 2022-09-10T10:52:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:27.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:27.631001+0000) 2022-09-10T10:52:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:28 smithi104 ceph-mon[97044]: pgmap v12263: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:28.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:28.119901+0000) 2022-09-10T10:52:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:28 smithi110 ceph-mon[97633]: pgmap v12263: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:28.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:28.631271+0000) 2022-09-10T10:52:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:29.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:29.120176+0000) 2022-09-10T10:52:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:29.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:29.631556+0000) 2022-09-10T10:52:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:30 smithi104 ceph-mon[97044]: pgmap v12264: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:30.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:30.120359+0000) 2022-09-10T10:52:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:30 smithi110 ceph-mon[97633]: pgmap v12264: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:30.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:30.631781+0000) 2022-09-10T10:52:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:31.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:31.120574+0000) 2022-09-10T10:52:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:31.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:31.632034+0000) 2022-09-10T10:52:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:32 smithi104 ceph-mon[97044]: pgmap v12265: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:32.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:32.120818+0000) 2022-09-10T10:52:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:32 smithi110 ceph-mon[97633]: pgmap v12265: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:32.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:32.632280+0000) 2022-09-10T10:52:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:33.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:33.121060+0000) 2022-09-10T10:52:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:33.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:33.632503+0000) 2022-09-10T10:52:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:34 smithi110 ceph-mon[97633]: pgmap v12266: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:34 smithi104 ceph-mon[97044]: pgmap v12266: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:34.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:34.121188+0000) 2022-09-10T10:52:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:34.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:34.632745+0000) 2022-09-10T10:52:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:35.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:35.121414+0000) 2022-09-10T10:52:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:35.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:35.632992+0000) 2022-09-10T10:52:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:36 smithi104 ceph-mon[97044]: pgmap v12267: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:36.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:36.121553+0000) 2022-09-10T10:52:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:36 smithi110 ceph-mon[97633]: pgmap v12267: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:36.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:36.633202+0000) 2022-09-10T10:52:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:37.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:37.121786+0000) 2022-09-10T10:52:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:37.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:37.633414+0000) 2022-09-10T10:52:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:38 smithi104 ceph-mon[97044]: pgmap v12268: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:38.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:38.121996+0000) 2022-09-10T10:52:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:38 smithi110 ceph-mon[97633]: pgmap v12268: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:38.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:38.633660+0000) 2022-09-10T10:52:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:39.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:39.122158+0000) 2022-09-10T10:52:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:39.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:39.633830+0000) 2022-09-10T10:52:40.542 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:40.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:40.122363+0000) 2022-09-10T10:52:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:40 smithi104 ceph-mon[97044]: pgmap v12269: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:40 smithi110 ceph-mon[97633]: pgmap v12269: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:40.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:40.634038+0000) 2022-09-10T10:52:40.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:52:40.541+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:52:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:41.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:41.122585+0000) 2022-09-10T10:52:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:41.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:41.634304+0000) 2022-09-10T10:52:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:42 smithi104 ceph-mon[97044]: pgmap v12270: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:42.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:42.122817+0000) 2022-09-10T10:52:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:42 smithi110 ceph-mon[97633]: pgmap v12270: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:42.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:42.634556+0000) 2022-09-10T10:52:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:43.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:43.123015+0000) 2022-09-10T10:52:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:43.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:43.634834+0000) 2022-09-10T10:52:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:44 smithi110 ceph-mon[97633]: pgmap v12271: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:44 smithi104 ceph-mon[97044]: pgmap v12271: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:44.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:44.123240+0000) 2022-09-10T10:52:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:44.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:44.635026+0000) 2022-09-10T10:52:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:45.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:45.123485+0000) 2022-09-10T10:52:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:45.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:45.635224+0000) 2022-09-10T10:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:46 smithi104 ceph-mon[97044]: pgmap v12272: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:52:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:52:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:52:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:52:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:46.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:46.123664+0000) 2022-09-10T10:52:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:46 smithi110 ceph-mon[97633]: pgmap v12272: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:52:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:52:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:52:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:52:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:46.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:46.635475+0000) 2022-09-10T10:52:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:47.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:47.123888+0000) 2022-09-10T10:52:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:47.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:47.635717+0000) 2022-09-10T10:52:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:48 smithi104 ceph-mon[97044]: pgmap v12273: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:48.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:48.124192+0000) 2022-09-10T10:52:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:48 smithi110 ceph-mon[97633]: pgmap v12273: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:48.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:48.635952+0000) 2022-09-10T10:52:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:49.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:49.124384+0000) 2022-09-10T10:52:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:49.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:49.636209+0000) 2022-09-10T10:52:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:50 smithi104 ceph-mon[97044]: pgmap v12274: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:50.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:50.124618+0000) 2022-09-10T10:52:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:50 smithi110 ceph-mon[97633]: pgmap v12274: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:50.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:50.636442+0000) 2022-09-10T10:52:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:51.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:51.124784+0000) 2022-09-10T10:52:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:51.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:51.636608+0000) 2022-09-10T10:52:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:52 smithi104 ceph-mon[97044]: pgmap v12275: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:52.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:52.124933+0000) 2022-09-10T10:52:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:52 smithi110 ceph-mon[97633]: pgmap v12275: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:52.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:52.636822+0000) 2022-09-10T10:52:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:53.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:53.125202+0000) 2022-09-10T10:52:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:53.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:53.637070+0000) 2022-09-10T10:52:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:54 smithi110 ceph-mon[97633]: pgmap v12276: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:54 smithi104 ceph-mon[97044]: pgmap v12276: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:54.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:54.125466+0000) 2022-09-10T10:52:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:54.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:54.637345+0000) 2022-09-10T10:52:55.543 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:55.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:55.125627+0000) 2022-09-10T10:52:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:55.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:55.637592+0000) 2022-09-10T10:52:55.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:52:55.541+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:52:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:56 smithi104 ceph-mon[97044]: pgmap v12277: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:56.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:56.125830+0000) 2022-09-10T10:52:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:56 smithi110 ceph-mon[97633]: pgmap v12277: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:52:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:56.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:56.637826+0000) 2022-09-10T10:52:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:57.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:57.126064+0000) 2022-09-10T10:52:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:57.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:57.638028+0000) 2022-09-10T10:52:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:52:58 smithi104 ceph-mon[97044]: pgmap v12278: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:58.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:58.126322+0000) 2022-09-10T10:52:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:52:58 smithi110 ceph-mon[97633]: pgmap v12278: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:52:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:58.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:58.638255+0000) 2022-09-10T10:52:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:52:59.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:59.126580+0000) 2022-09-10T10:53:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:52:59.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:52:59.638484+0000) 2022-09-10T10:53:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:00 smithi104 ceph-mon[97044]: pgmap v12279: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:00.578 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:00.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:00.126820+0000) 2022-09-10T10:53:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:00 smithi110 ceph-mon[97633]: pgmap v12279: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:00.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:00.638706+0000) 2022-09-10T10:53:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:01.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:01.126980+0000) 2022-09-10T10:53:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:01.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:01.638898+0000) 2022-09-10T10:53:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:02 smithi104 ceph-mon[97044]: pgmap v12280: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:02.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:02.127214+0000) 2022-09-10T10:53:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:02 smithi110 ceph-mon[97633]: pgmap v12280: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:02.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:02.639153+0000) 2022-09-10T10:53:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:03.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:03.127476+0000) 2022-09-10T10:53:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:03.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:03.639391+0000) 2022-09-10T10:53:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:04 smithi110 ceph-mon[97633]: pgmap v12281: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:04 smithi104 ceph-mon[97044]: pgmap v12281: 129 pgs: 129 active+clean; 9.0 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:04.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:04.127738+0000) 2022-09-10T10:53:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:04.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:04.639538+0000) 2022-09-10T10:53:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:05.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:05.127962+0000) 2022-09-10T10:53:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:05.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:05.639725+0000) 2022-09-10T10:53:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:06 smithi104 ceph-mon[97044]: pgmap v12282: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:53:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:06.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:06.128244+0000) 2022-09-10T10:53:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:06 smithi110 ceph-mon[97633]: pgmap v12282: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:53:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:06.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:06.639972+0000) 2022-09-10T10:53:07.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:07.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:07.128466+0000) 2022-09-10T10:53:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:07.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:07.640239+0000) 2022-09-10T10:53:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:08 smithi104 ceph-mon[97044]: pgmap v12283: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:53:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:08.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:08.128765+0000) 2022-09-10T10:53:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:08 smithi110 ceph-mon[97633]: pgmap v12283: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:53:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:08.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:08.640474+0000) 2022-09-10T10:53:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:09 smithi104 ceph-mon[97044]: osdmap e848: 12 total, 12 up, 12 in 2022-09-10T10:53:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:09.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:09.129053+0000) 2022-09-10T10:53:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:09 smithi110 ceph-mon[97633]: osdmap e848: 12 total, 12 up, 12 in 2022-09-10T10:53:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:09.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:09.640676+0000) 2022-09-10T10:53:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:10 smithi104 ceph-mon[97044]: pgmap v12285: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:53:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:10.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:10.129315+0000) 2022-09-10T10:53:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:10 smithi110 ceph-mon[97633]: pgmap v12285: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:53:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:10.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:10.640919+0000) 2022-09-10T10:53:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:53:10.552+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:53:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:11.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:11.129582+0000) 2022-09-10T10:53:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:11.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:11.641119+0000) 2022-09-10T10:53:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:12 smithi104 ceph-mon[97044]: pgmap v12286: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:53:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:12.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:12.129780+0000) 2022-09-10T10:53:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:12 smithi110 ceph-mon[97633]: pgmap v12286: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:53:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:12.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:12.641376+0000) 2022-09-10T10:53:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:13.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:13.129995+0000) 2022-09-10T10:53:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:13.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:13.641638+0000) 2022-09-10T10:53:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:14 smithi110 ceph-mon[97633]: osdmap e849: 12 total, 12 up, 12 in 2022-09-10T10:53:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:14 smithi110 ceph-mon[97633]: pgmap v12288: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:53:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:14 smithi104 ceph-mon[97044]: osdmap e849: 12 total, 12 up, 12 in 2022-09-10T10:53:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:14 smithi104 ceph-mon[97044]: pgmap v12288: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:53:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:14.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:14.130250+0000) 2022-09-10T10:53:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:14.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:14.641887+0000) 2022-09-10T10:53:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:15.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:15.130518+0000) 2022-09-10T10:53:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:15.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:15.642054+0000) 2022-09-10T10:53:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:16 smithi104 ceph-mon[97044]: pgmap v12289: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:53:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:16.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:16.130737+0000) 2022-09-10T10:53:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:16 smithi110 ceph-mon[97633]: pgmap v12289: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:53:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:16.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:16.642265+0000) 2022-09-10T10:53:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:17.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:17.130924+0000) 2022-09-10T10:53:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:17.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:17.642461+0000) 2022-09-10T10:53:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:18 smithi104 ceph-mon[97044]: pgmap v12290: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:53:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:18.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:18.131128+0000) 2022-09-10T10:53:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:18 smithi110 ceph-mon[97633]: pgmap v12290: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:53:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:18.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:18.642599+0000) 2022-09-10T10:53:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:19.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:19.131345+0000) 2022-09-10T10:53:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:19.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:19.642773+0000) 2022-09-10T10:53:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:20 smithi104 ceph-mon[97044]: pgmap v12291: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:53:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:20.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:20.131605+0000) 2022-09-10T10:53:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:20 smithi110 ceph-mon[97633]: pgmap v12291: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:53:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:20.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:20.643016+0000) 2022-09-10T10:53:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:21.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:21.131857+0000) 2022-09-10T10:53:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:21.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:21.643237+0000) 2022-09-10T10:53:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:22 smithi104 ceph-mon[97044]: pgmap v12292: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:53:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:22.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:22.132084+0000) 2022-09-10T10:53:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:22 smithi110 ceph-mon[97633]: pgmap v12292: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:53:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:22.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:22.643418+0000) 2022-09-10T10:53:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:23.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:23.132324+0000) 2022-09-10T10:53:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:23.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:23.643624+0000) 2022-09-10T10:53:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:24 smithi110 ceph-mon[97633]: pgmap v12293: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:53:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:24 smithi104 ceph-mon[97044]: pgmap v12293: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:53:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:24.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:24.132522+0000) 2022-09-10T10:53:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:24.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:24.643887+0000) 2022-09-10T10:53:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:25.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:25.132690+0000) 2022-09-10T10:53:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:53:25.568+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:53:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:25.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:25.644158+0000) 2022-09-10T10:53:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:26 smithi104 ceph-mon[97044]: pgmap v12294: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:53:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:26.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:26.132830+0000) 2022-09-10T10:53:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:26 smithi110 ceph-mon[97633]: pgmap v12294: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:53:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:26.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:26.644370+0000) 2022-09-10T10:53:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:27.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:27.133079+0000) 2022-09-10T10:53:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:27.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:27.644596+0000) 2022-09-10T10:53:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:28 smithi104 ceph-mon[97044]: pgmap v12295: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:53:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:28.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:28.133252+0000) 2022-09-10T10:53:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:28 smithi110 ceph-mon[97633]: pgmap v12295: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:53:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:28.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:28.644824+0000) 2022-09-10T10:53:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:29.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:29.133401+0000) 2022-09-10T10:53:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:29.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:29.645136+0000) 2022-09-10T10:53:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:30 smithi104 ceph-mon[97044]: pgmap v12296: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:53:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:30.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:30.133613+0000) 2022-09-10T10:53:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:30 smithi110 ceph-mon[97633]: pgmap v12296: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:53:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:30.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:30.645367+0000) 2022-09-10T10:53:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:31.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:31.133866+0000) 2022-09-10T10:53:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:31.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:31.645588+0000) 2022-09-10T10:53:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:32 smithi104 ceph-mon[97044]: pgmap v12297: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:32.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:32.134091+0000) 2022-09-10T10:53:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:32 smithi110 ceph-mon[97633]: pgmap v12297: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:32.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:32.645743+0000) 2022-09-10T10:53:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:33.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:33.134343+0000) 2022-09-10T10:53:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:33.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:33.645932+0000) 2022-09-10T10:53:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:34 smithi110 ceph-mon[97633]: pgmap v12298: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:34 smithi104 ceph-mon[97044]: pgmap v12298: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:34.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:34.134551+0000) 2022-09-10T10:53:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:34.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:34.646206+0000) 2022-09-10T10:53:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:35.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:35.134712+0000) 2022-09-10T10:53:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:35.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:35.646412+0000) 2022-09-10T10:53:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:36 smithi104 ceph-mon[97044]: pgmap v12299: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:36.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:36.134965+0000) 2022-09-10T10:53:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:36 smithi110 ceph-mon[97633]: pgmap v12299: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:36.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:36.646624+0000) 2022-09-10T10:53:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:37.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:37.135194+0000) 2022-09-10T10:53:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:37.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:37.646877+0000) 2022-09-10T10:53:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:38 smithi104 ceph-mon[97044]: pgmap v12300: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:38.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:38.135342+0000) 2022-09-10T10:53:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:38 smithi110 ceph-mon[97633]: pgmap v12300: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:38.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:38.647153+0000) 2022-09-10T10:53:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:39.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:39.135573+0000) 2022-09-10T10:53:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:39.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:39.647399+0000) 2022-09-10T10:53:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:40 smithi104 ceph-mon[97044]: pgmap v12301: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:40.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:40.135825+0000) 2022-09-10T10:53:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:40 smithi110 ceph-mon[97633]: pgmap v12301: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:40.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:40.647626+0000) 2022-09-10T10:53:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:53:40.574+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:53:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:41.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:41.136079+0000) 2022-09-10T10:53:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:41.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:41.647865+0000) 2022-09-10T10:53:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:42 smithi104 ceph-mon[97044]: pgmap v12302: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:42.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:42.136301+0000) 2022-09-10T10:53:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:42 smithi110 ceph-mon[97633]: pgmap v12302: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:42.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:42.648010+0000) 2022-09-10T10:53:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:43.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:43.136479+0000) 2022-09-10T10:53:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:43.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:43.648244+0000) 2022-09-10T10:53:44.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:44 smithi110 ceph-mon[97633]: pgmap v12303: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:44 smithi104 ceph-mon[97044]: pgmap v12303: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:44.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:44.136688+0000) 2022-09-10T10:53:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:44.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:44.648467+0000) 2022-09-10T10:53:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:45.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:45.136944+0000) 2022-09-10T10:53:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:45.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:45.648642+0000) 2022-09-10T10:53:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:46 smithi104 ceph-mon[97044]: pgmap v12304: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:53:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:53:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:53:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:53:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:46.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:46.137219+0000) 2022-09-10T10:53:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:46 smithi110 ceph-mon[97633]: pgmap v12304: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:53:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:53:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:53:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:53:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:46.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:46.648869+0000) 2022-09-10T10:53:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:47.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:47.137416+0000) 2022-09-10T10:53:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:47.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:47.649054+0000) 2022-09-10T10:53:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:48 smithi104 ceph-mon[97044]: pgmap v12305: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:48.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:48.137662+0000) 2022-09-10T10:53:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:48 smithi110 ceph-mon[97633]: pgmap v12305: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:48.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:48.649331+0000) 2022-09-10T10:53:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:49.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:49.137935+0000) 2022-09-10T10:53:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:49.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:49.649550+0000) 2022-09-10T10:53:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:50 smithi104 ceph-mon[97044]: pgmap v12306: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:50.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:50.138186+0000) 2022-09-10T10:53:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:50 smithi110 ceph-mon[97633]: pgmap v12306: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:50.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:50.649777+0000) 2022-09-10T10:53:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:51.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:51.138394+0000) 2022-09-10T10:53:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:51.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:51.650013+0000) 2022-09-10T10:53:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:52 smithi104 ceph-mon[97044]: pgmap v12307: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:52.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:52.138625+0000) 2022-09-10T10:53:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:52 smithi110 ceph-mon[97633]: pgmap v12307: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:52.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:52.650240+0000) 2022-09-10T10:53:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:53.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:53.138883+0000) 2022-09-10T10:53:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:53.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:53.650449+0000) 2022-09-10T10:53:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:54 smithi110 ceph-mon[97633]: pgmap v12308: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:54 smithi104 ceph-mon[97044]: pgmap v12308: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:54.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:54.139165+0000) 2022-09-10T10:53:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:54.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:54.650648+0000) 2022-09-10T10:53:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:55.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:55.139408+0000) 2022-09-10T10:53:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:55.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:55.650902+0000) 2022-09-10T10:53:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:53:55.583+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:53:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:56 smithi104 ceph-mon[97044]: pgmap v12309: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:56.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:56.139623+0000) 2022-09-10T10:53:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:56 smithi110 ceph-mon[97633]: pgmap v12309: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:53:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:56.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:56.651134+0000) 2022-09-10T10:53:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:57.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:57.139793+0000) 2022-09-10T10:53:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:57.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:57.651349+0000) 2022-09-10T10:53:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:53:58 smithi104 ceph-mon[97044]: pgmap v12310: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:58.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:58.140016+0000) 2022-09-10T10:53:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:53:58 smithi110 ceph-mon[97633]: pgmap v12310: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:53:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:58.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:58.651601+0000) 2022-09-10T10:53:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:53:59.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:59.140257+0000) 2022-09-10T10:54:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:53:59.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:53:59.651830+0000) 2022-09-10T10:54:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:00 smithi104 ceph-mon[97044]: pgmap v12311: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:00.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:00.140395+0000) 2022-09-10T10:54:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:00 smithi110 ceph-mon[97633]: pgmap v12311: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:00.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:00.652007+0000) 2022-09-10T10:54:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:01.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:01.140603+0000) 2022-09-10T10:54:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:01.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:01.652195+0000) 2022-09-10T10:54:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:02 smithi104 ceph-mon[97044]: pgmap v12312: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:02.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:02.140783+0000) 2022-09-10T10:54:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:02 smithi110 ceph-mon[97633]: pgmap v12312: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:02.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:02.652439+0000) 2022-09-10T10:54:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:03.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:03.141000+0000) 2022-09-10T10:54:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:03.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:03.652586+0000) 2022-09-10T10:54:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:04 smithi110 ceph-mon[97633]: pgmap v12313: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:04 smithi104 ceph-mon[97044]: pgmap v12313: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:04.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:04.141205+0000) 2022-09-10T10:54:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:04.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:04.652756+0000) 2022-09-10T10:54:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:05.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:05.141377+0000) 2022-09-10T10:54:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:05.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:05.652991+0000) 2022-09-10T10:54:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:06 smithi104 ceph-mon[97044]: pgmap v12314: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:54:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:06.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:06.141583+0000) 2022-09-10T10:54:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:06 smithi110 ceph-mon[97633]: pgmap v12314: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:54:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:06.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:06.653229+0000) 2022-09-10T10:54:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:07.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:07.141804+0000) 2022-09-10T10:54:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:07.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:07.653406+0000) 2022-09-10T10:54:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:08 smithi104 ceph-mon[97044]: pgmap v12315: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:54:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:08.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:08.142027+0000) 2022-09-10T10:54:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:08 smithi110 ceph-mon[97633]: pgmap v12315: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:54:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:08.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:08.653677+0000) 2022-09-10T10:54:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:09 smithi104 ceph-mon[97044]: osdmap e850: 12 total, 12 up, 12 in 2022-09-10T10:54:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:09.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:09.142191+0000) 2022-09-10T10:54:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:09 smithi110 ceph-mon[97633]: osdmap e850: 12 total, 12 up, 12 in 2022-09-10T10:54:10.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:09.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:09.653904+0000) 2022-09-10T10:54:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:10 smithi104 ceph-mon[97044]: pgmap v12317: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:54:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:10.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:10.142410+0000) 2022-09-10T10:54:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:10 smithi110 ceph-mon[97633]: pgmap v12317: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:54:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:10.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:10.654055+0000) 2022-09-10T10:54:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:54:10.599+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:54:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:11.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:11.142590+0000) 2022-09-10T10:54:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:11.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:11.654240+0000) 2022-09-10T10:54:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:12 smithi104 ceph-mon[97044]: pgmap v12318: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:54:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:12.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:12.142827+0000) 2022-09-10T10:54:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:12 smithi110 ceph-mon[97633]: pgmap v12318: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:54:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:12.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:12.654502+0000) 2022-09-10T10:54:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:13.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:13.143109+0000) 2022-09-10T10:54:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:13.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:13.654785+0000) 2022-09-10T10:54:14.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:14 smithi110 ceph-mon[97633]: osdmap e851: 12 total, 12 up, 12 in 2022-09-10T10:54:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:14 smithi110 ceph-mon[97633]: pgmap v12320: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:54:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:14 smithi104 ceph-mon[97044]: osdmap e851: 12 total, 12 up, 12 in 2022-09-10T10:54:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:14 smithi104 ceph-mon[97044]: pgmap v12320: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:54:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:14.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:14.143210+0000) 2022-09-10T10:54:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:14.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:14.654986+0000) 2022-09-10T10:54:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:15.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:15.143339+0000) 2022-09-10T10:54:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:15.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:15.655226+0000) 2022-09-10T10:54:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:16 smithi104 ceph-mon[97044]: pgmap v12321: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:54:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:16.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:16.143502+0000) 2022-09-10T10:54:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:16 smithi110 ceph-mon[97633]: pgmap v12321: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:54:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:16.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:16.655421+0000) 2022-09-10T10:54:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:17.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:17.143703+0000) 2022-09-10T10:54:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:17.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:17.655619+0000) 2022-09-10T10:54:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:18 smithi104 ceph-mon[97044]: pgmap v12322: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:54:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:18.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:18.143884+0000) 2022-09-10T10:54:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:18 smithi110 ceph-mon[97633]: pgmap v12322: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:54:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:18.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:18.655865+0000) 2022-09-10T10:54:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:19.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:19.144089+0000) 2022-09-10T10:54:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:19.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:19.656174+0000) 2022-09-10T10:54:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:20 smithi104 ceph-mon[97044]: pgmap v12323: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:54:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:20.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:20.144358+0000) 2022-09-10T10:54:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:20 smithi110 ceph-mon[97633]: pgmap v12323: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:54:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:20.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:20.656342+0000) 2022-09-10T10:54:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:21.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:21.144594+0000) 2022-09-10T10:54:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:21.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:21.656556+0000) 2022-09-10T10:54:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:22 smithi104 ceph-mon[97044]: pgmap v12324: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:54:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:22.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:22.144713+0000) 2022-09-10T10:54:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:22 smithi110 ceph-mon[97633]: pgmap v12324: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:54:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:22.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:22.656798+0000) 2022-09-10T10:54:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:23.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:23.144896+0000) 2022-09-10T10:54:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:23.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:23.657039+0000) 2022-09-10T10:54:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:24 smithi110 ceph-mon[97633]: pgmap v12325: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:54:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:24 smithi104 ceph-mon[97044]: pgmap v12325: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:54:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:24.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:24.145144+0000) 2022-09-10T10:54:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:24.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:24.657262+0000) 2022-09-10T10:54:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:25.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:25.145306+0000) 2022-09-10T10:54:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:25.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:25.657492+0000) 2022-09-10T10:54:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:54:25.615+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:54:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:26 smithi104 ceph-mon[97044]: pgmap v12326: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:54:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:26.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:26.145500+0000) 2022-09-10T10:54:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:26 smithi110 ceph-mon[97633]: pgmap v12326: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:54:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:26.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:26.657754+0000) 2022-09-10T10:54:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:27.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:27.145705+0000) 2022-09-10T10:54:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:27.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:27.657944+0000) 2022-09-10T10:54:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:28 smithi104 ceph-mon[97044]: pgmap v12327: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:28.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:28.145953+0000) 2022-09-10T10:54:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:28 smithi110 ceph-mon[97633]: pgmap v12327: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:28.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:28.658123+0000) 2022-09-10T10:54:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:29.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:29.146206+0000) 2022-09-10T10:54:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:29.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:29.658307+0000) 2022-09-10T10:54:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:30 smithi104 ceph-mon[97044]: pgmap v12328: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:30.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:30.146401+0000) 2022-09-10T10:54:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:30 smithi110 ceph-mon[97633]: pgmap v12328: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:30.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:30.658542+0000) 2022-09-10T10:54:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:31.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:31.146583+0000) 2022-09-10T10:54:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:31.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:31.658712+0000) 2022-09-10T10:54:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:32 smithi104 ceph-mon[97044]: pgmap v12329: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:32.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:32.146726+0000) 2022-09-10T10:54:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:32 smithi110 ceph-mon[97633]: pgmap v12329: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:32.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:32.658960+0000) 2022-09-10T10:54:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:33.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:33.146927+0000) 2022-09-10T10:54:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:33.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:33.659214+0000) 2022-09-10T10:54:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:34 smithi110 ceph-mon[97633]: pgmap v12330: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:34 smithi104 ceph-mon[97044]: pgmap v12330: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:34.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:34.147163+0000) 2022-09-10T10:54:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:34.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:34.659487+0000) 2022-09-10T10:54:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:35.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:35.147307+0000) 2022-09-10T10:54:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:35.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:35.659697+0000) 2022-09-10T10:54:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:36 smithi104 ceph-mon[97044]: pgmap v12331: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:36.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:36.147486+0000) 2022-09-10T10:54:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:36 smithi110 ceph-mon[97633]: pgmap v12331: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:36.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:36.659916+0000) 2022-09-10T10:54:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:37.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:37.147722+0000) 2022-09-10T10:54:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:37.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:37.660195+0000) 2022-09-10T10:54:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:38 smithi104 ceph-mon[97044]: pgmap v12332: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:38.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:38.147996+0000) 2022-09-10T10:54:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:38 smithi110 ceph-mon[97633]: pgmap v12332: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:38.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:38.660365+0000) 2022-09-10T10:54:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:39.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:39.148216+0000) 2022-09-10T10:54:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:39.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:39.660610+0000) 2022-09-10T10:54:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:40 smithi104 ceph-mon[97044]: pgmap v12333: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:40.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:40.148446+0000) 2022-09-10T10:54:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:40 smithi110 ceph-mon[97633]: pgmap v12333: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:40.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:40.660799+0000) 2022-09-10T10:54:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:54:40.623+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:54:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:41.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:41.148708+0000) 2022-09-10T10:54:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:41.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:41.661033+0000) 2022-09-10T10:54:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:42 smithi104 ceph-mon[97044]: pgmap v12334: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:42.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:42.148899+0000) 2022-09-10T10:54:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:42 smithi110 ceph-mon[97633]: pgmap v12334: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:42.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:42.661288+0000) 2022-09-10T10:54:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:43.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:43.149060+0000) 2022-09-10T10:54:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:43.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:43.661482+0000) 2022-09-10T10:54:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:44 smithi104 ceph-mon[97044]: pgmap v12335: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:44.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:44.149299+0000) 2022-09-10T10:54:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:44 smithi110 ceph-mon[97633]: pgmap v12335: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:44.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:44.661693+0000) 2022-09-10T10:54:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:45.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:45.149529+0000) 2022-09-10T10:54:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:45.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:45.661885+0000) 2022-09-10T10:54:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:46 smithi104 ceph-mon[97044]: pgmap v12336: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:54:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:54:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:54:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:54:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:46.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:46.149773+0000) 2022-09-10T10:54:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:46 smithi110 ceph-mon[97633]: pgmap v12336: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:54:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:54:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:54:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:54:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:46.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:46.662085+0000) 2022-09-10T10:54:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:47.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:47.150037+0000) 2022-09-10T10:54:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:47.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:47.662360+0000) 2022-09-10T10:54:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:48 smithi104 ceph-mon[97044]: pgmap v12337: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:48.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:48.150289+0000) 2022-09-10T10:54:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:48 smithi110 ceph-mon[97633]: pgmap v12337: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:48.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:48.662557+0000) 2022-09-10T10:54:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:49.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:49.150456+0000) 2022-09-10T10:54:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:49.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:49.662741+0000) 2022-09-10T10:54:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:50 smithi104 ceph-mon[97044]: pgmap v12338: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:50.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:50.150703+0000) 2022-09-10T10:54:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:50 smithi110 ceph-mon[97633]: pgmap v12338: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:50.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:50.662902+0000) 2022-09-10T10:54:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:51.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:51.150936+0000) 2022-09-10T10:54:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:51.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:51.663129+0000) 2022-09-10T10:54:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:52 smithi104 ceph-mon[97044]: pgmap v12339: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:52.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:52.151144+0000) 2022-09-10T10:54:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:52 smithi110 ceph-mon[97633]: pgmap v12339: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:52.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:52.663315+0000) 2022-09-10T10:54:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:53.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:53.151424+0000) 2022-09-10T10:54:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:53.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:53.663551+0000) 2022-09-10T10:54:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:54 smithi110 ceph-mon[97633]: pgmap v12340: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:54 smithi104 ceph-mon[97044]: pgmap v12340: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:54.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:54.151605+0000) 2022-09-10T10:54:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:54.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:54.663798+0000) 2022-09-10T10:54:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:55.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:55.151838+0000) 2022-09-10T10:54:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:54:55.638+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:54:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:55.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:55.664044+0000) 2022-09-10T10:54:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:56 smithi104 ceph-mon[97044]: pgmap v12341: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:56.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:56.152071+0000) 2022-09-10T10:54:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:56 smithi110 ceph-mon[97633]: pgmap v12341: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:54:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:56.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:56.664327+0000) 2022-09-10T10:54:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:57.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:57.152276+0000) 2022-09-10T10:54:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:57.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:57.664578+0000) 2022-09-10T10:54:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:54:58 smithi104 ceph-mon[97044]: pgmap v12342: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:58.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:58.152493+0000) 2022-09-10T10:54:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:54:58 smithi110 ceph-mon[97633]: pgmap v12342: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:54:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:58.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:58.664774+0000) 2022-09-10T10:54:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:54:59.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:59.152710+0000) 2022-09-10T10:55:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:54:59.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:54:59.665006+0000) 2022-09-10T10:55:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:00 smithi104 ceph-mon[97044]: pgmap v12343: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:00.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:00.152913+0000) 2022-09-10T10:55:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:00 smithi110 ceph-mon[97633]: pgmap v12343: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:00.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:00.665168+0000) 2022-09-10T10:55:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:01.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:01.153111+0000) 2022-09-10T10:55:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:01.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:01.665395+0000) 2022-09-10T10:55:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:02 smithi104 ceph-mon[97044]: pgmap v12344: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:02.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:02.153294+0000) 2022-09-10T10:55:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:02 smithi110 ceph-mon[97633]: pgmap v12344: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:02.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:02.665571+0000) 2022-09-10T10:55:03.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:03.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:03.153472+0000) 2022-09-10T10:55:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:03.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:03.665849+0000) 2022-09-10T10:55:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:04 smithi104 ceph-mon[97044]: pgmap v12345: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:04.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:04.153631+0000) 2022-09-10T10:55:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:04 smithi110 ceph-mon[97633]: pgmap v12345: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:04.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:04.666122+0000) 2022-09-10T10:55:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:05.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:05.153877+0000) 2022-09-10T10:55:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:05.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:05.666369+0000) 2022-09-10T10:55:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:06 smithi104 ceph-mon[97044]: pgmap v12346: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:55:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:06.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:06.154054+0000) 2022-09-10T10:55:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:06 smithi110 ceph-mon[97633]: pgmap v12346: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:55:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:06.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:06.666519+0000) 2022-09-10T10:55:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:07.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:07.154275+0000) 2022-09-10T10:55:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:07.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:07.666701+0000) 2022-09-10T10:55:08.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:08 smithi104 ceph-mon[97044]: pgmap v12347: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:55:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:08.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:08.154525+0000) 2022-09-10T10:55:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:08 smithi110 ceph-mon[97633]: pgmap v12347: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:55:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:08.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:08.666953+0000) 2022-09-10T10:55:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:09 smithi104 ceph-mon[97044]: osdmap e852: 12 total, 12 up, 12 in 2022-09-10T10:55:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:09.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:09.154774+0000) 2022-09-10T10:55:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:09 smithi110 ceph-mon[97633]: osdmap e852: 12 total, 12 up, 12 in 2022-09-10T10:55:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:09.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:09.667212+0000) 2022-09-10T10:55:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:10 smithi104 ceph-mon[97044]: pgmap v12349: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:55:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:10.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:10.155020+0000) 2022-09-10T10:55:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:10 smithi110 ceph-mon[97633]: pgmap v12349: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:55:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:10.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:10.667421+0000) 2022-09-10T10:55:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:55:10.654+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:55:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:11.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:11.155225+0000) 2022-09-10T10:55:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:11.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:11.667589+0000) 2022-09-10T10:55:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:12 smithi104 ceph-mon[97044]: pgmap v12350: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:55:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:12.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:12.155450+0000) 2022-09-10T10:55:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:12 smithi110 ceph-mon[97633]: pgmap v12350: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:55:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:12.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:12.667808+0000) 2022-09-10T10:55:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:13.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:13.155708+0000) 2022-09-10T10:55:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:13.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:13.668004+0000) 2022-09-10T10:55:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:14 smithi104 ceph-mon[97044]: osdmap e853: 12 total, 12 up, 12 in 2022-09-10T10:55:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:14 smithi104 ceph-mon[97044]: pgmap v12352: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:55:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:14.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:14.155875+0000) 2022-09-10T10:55:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:14 smithi110 ceph-mon[97633]: osdmap e853: 12 total, 12 up, 12 in 2022-09-10T10:55:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:14 smithi110 ceph-mon[97633]: pgmap v12352: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:55:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:14.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:14.668252+0000) 2022-09-10T10:55:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:15.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:15.156092+0000) 2022-09-10T10:55:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:15.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:15.668440+0000) 2022-09-10T10:55:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:16 smithi104 ceph-mon[97044]: pgmap v12353: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:55:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:16.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:16.156252+0000) 2022-09-10T10:55:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:16 smithi110 ceph-mon[97633]: pgmap v12353: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:55:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:16.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:16.668636+0000) 2022-09-10T10:55:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:17.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:17.156475+0000) 2022-09-10T10:55:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:17.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:17.668793+0000) 2022-09-10T10:55:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:18 smithi104 ceph-mon[97044]: pgmap v12354: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:55:18.634 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:18.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:18.156697+0000) 2022-09-10T10:55:18.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:18 smithi110 ceph-mon[97633]: pgmap v12354: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:55:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:18.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:18.669018+0000) 2022-09-10T10:55:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:19.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:19.156871+0000) 2022-09-10T10:55:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:19.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:19.669241+0000) 2022-09-10T10:55:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:20 smithi104 ceph-mon[97044]: pgmap v12355: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:55:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:20.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:20.157076+0000) 2022-09-10T10:55:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:20 smithi110 ceph-mon[97633]: pgmap v12355: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:55:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:20.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:20.669494+0000) 2022-09-10T10:55:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:21.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:21.157298+0000) 2022-09-10T10:55:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:21.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:21.669686+0000) 2022-09-10T10:55:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:22 smithi104 ceph-mon[97044]: pgmap v12356: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:55:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:22.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:22.157515+0000) 2022-09-10T10:55:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:22 smithi110 ceph-mon[97633]: pgmap v12356: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:55:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:22.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:22.669896+0000) 2022-09-10T10:55:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:23.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:23.157655+0000) 2022-09-10T10:55:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:23.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:23.670116+0000) 2022-09-10T10:55:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:24 smithi104 ceph-mon[97044]: pgmap v12357: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:55:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:24.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:24.157886+0000) 2022-09-10T10:55:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:24 smithi110 ceph-mon[97633]: pgmap v12357: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:55:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:24.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:24.670366+0000) 2022-09-10T10:55:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:25.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:25.158123+0000) 2022-09-10T10:55:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:25.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:25.670531+0000) 2022-09-10T10:55:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:55:25.668+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:55:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:26 smithi104 ceph-mon[97044]: pgmap v12358: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:55:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:26.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:26.158381+0000) 2022-09-10T10:55:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:26 smithi110 ceph-mon[97633]: pgmap v12358: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:55:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:26.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:26.670762+0000) 2022-09-10T10:55:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:27.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:27.158601+0000) 2022-09-10T10:55:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:27.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:27.670959+0000) 2022-09-10T10:55:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:28 smithi104 ceph-mon[97044]: pgmap v12359: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:28.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:28.158891+0000) 2022-09-10T10:55:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:28 smithi110 ceph-mon[97633]: pgmap v12359: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:28.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:28.671198+0000) 2022-09-10T10:55:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:29.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:29.159115+0000) 2022-09-10T10:55:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:29.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:29.671400+0000) 2022-09-10T10:55:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:30 smithi104 ceph-mon[97044]: pgmap v12360: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:30.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:30.159350+0000) 2022-09-10T10:55:30.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:30 smithi110 ceph-mon[97633]: pgmap v12360: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:30.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:30.671649+0000) 2022-09-10T10:55:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:31.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:31.159535+0000) 2022-09-10T10:55:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:31.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:31.671847+0000) 2022-09-10T10:55:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:32 smithi104 ceph-mon[97044]: pgmap v12361: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:32.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:32.159696+0000) 2022-09-10T10:55:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:32 smithi110 ceph-mon[97633]: pgmap v12361: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:32.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:32.672058+0000) 2022-09-10T10:55:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:33.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:33.159987+0000) 2022-09-10T10:55:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:33.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:33.672295+0000) 2022-09-10T10:55:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:34 smithi104 ceph-mon[97044]: pgmap v12362: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:34.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:34.160250+0000) 2022-09-10T10:55:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:34 smithi110 ceph-mon[97633]: pgmap v12362: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:34.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:34.672480+0000) 2022-09-10T10:55:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:35.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:35.160452+0000) 2022-09-10T10:55:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:35.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:35.672710+0000) 2022-09-10T10:55:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:36 smithi104 ceph-mon[97044]: pgmap v12363: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:36.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:36.160692+0000) 2022-09-10T10:55:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:36 smithi110 ceph-mon[97633]: pgmap v12363: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:36.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:36.672929+0000) 2022-09-10T10:55:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:37.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:37.160923+0000) 2022-09-10T10:55:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:37.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:37.673217+0000) 2022-09-10T10:55:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:38 smithi104 ceph-mon[97044]: pgmap v12364: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:38.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:38.161167+0000) 2022-09-10T10:55:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:38 smithi110 ceph-mon[97633]: pgmap v12364: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:38.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:38.673441+0000) 2022-09-10T10:55:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:39.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:39.161292+0000) 2022-09-10T10:55:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:39.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:39.673663+0000) 2022-09-10T10:55:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:40 smithi104 ceph-mon[97044]: pgmap v12365: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:40.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:40.161475+0000) 2022-09-10T10:55:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:40 smithi110 ceph-mon[97633]: pgmap v12365: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:40.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:40.673924+0000) 2022-09-10T10:55:41.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:55:40.677+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:55:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:41.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:41.161705+0000) 2022-09-10T10:55:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:41.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:41.674195+0000) 2022-09-10T10:55:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:42 smithi104 ceph-mon[97044]: pgmap v12366: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:42.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:42.161886+0000) 2022-09-10T10:55:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:42 smithi110 ceph-mon[97633]: pgmap v12366: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:42.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:42.674373+0000) 2022-09-10T10:55:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:43.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:43.162140+0000) 2022-09-10T10:55:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:43.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:43.674608+0000) 2022-09-10T10:55:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:44 smithi104 ceph-mon[97044]: pgmap v12367: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:44.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:44.162365+0000) 2022-09-10T10:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:44 smithi110 ceph-mon[97633]: pgmap v12367: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:44.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:44.674863+0000) 2022-09-10T10:55:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:45.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:45.162543+0000) 2022-09-10T10:55:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:45.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:45.675087+0000) 2022-09-10T10:55:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:46 smithi104 ceph-mon[97044]: pgmap v12368: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:55:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:46.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:46.162758+0000) 2022-09-10T10:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:46 smithi110 ceph-mon[97633]: pgmap v12368: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:55:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:55:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:55:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:55:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:46.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:46.675308+0000) 2022-09-10T10:55:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:47.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:47.163012+0000) 2022-09-10T10:55:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:47.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:47.675533+0000) 2022-09-10T10:55:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:48 smithi104 ceph-mon[97044]: pgmap v12369: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:48.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:48.163255+0000) 2022-09-10T10:55:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:48 smithi110 ceph-mon[97633]: pgmap v12369: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:48.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:48.675670+0000) 2022-09-10T10:55:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:49.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:49.163473+0000) 2022-09-10T10:55:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:49.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:49.675899+0000) 2022-09-10T10:55:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:50 smithi104 ceph-mon[97044]: pgmap v12370: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:50.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:50.163647+0000) 2022-09-10T10:55:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:50 smithi110 ceph-mon[97633]: pgmap v12370: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:50.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:50.676112+0000) 2022-09-10T10:55:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:51.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:51.163841+0000) 2022-09-10T10:55:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:51.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:51.676362+0000) 2022-09-10T10:55:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:52 smithi104 ceph-mon[97044]: pgmap v12371: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:52.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:52.164046+0000) 2022-09-10T10:55:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:52 smithi110 ceph-mon[97633]: pgmap v12371: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:52.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:52.676601+0000) 2022-09-10T10:55:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:53.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:53.164238+0000) 2022-09-10T10:55:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:53.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:53.676867+0000) 2022-09-10T10:55:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:54 smithi104 ceph-mon[97044]: pgmap v12372: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:54.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:54.164461+0000) 2022-09-10T10:55:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:54 smithi110 ceph-mon[97633]: pgmap v12372: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:54.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:54.677142+0000) 2022-09-10T10:55:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:55.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:55.164646+0000) 2022-09-10T10:55:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:55.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:55.677381+0000) 2022-09-10T10:55:56.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:55:55.681+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:55:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:56 smithi104 ceph-mon[97044]: pgmap v12373: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:56.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:56.164889+0000) 2022-09-10T10:55:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:56 smithi110 ceph-mon[97633]: pgmap v12373: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:55:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:56.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:56.677598+0000) 2022-09-10T10:55:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:57.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:57.165173+0000) 2022-09-10T10:55:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:57.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:57.677845+0000) 2022-09-10T10:55:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:55:58 smithi104 ceph-mon[97044]: pgmap v12374: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:58.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:58.165452+0000) 2022-09-10T10:55:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:55:58 smithi110 ceph-mon[97633]: pgmap v12374: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:55:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:58.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:58.678071+0000) 2022-09-10T10:55:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:55:59.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:59.165754+0000) 2022-09-10T10:56:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:55:59.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:55:59.678338+0000) 2022-09-10T10:56:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:00 smithi104 ceph-mon[97044]: pgmap v12375: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:00.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:00.166024+0000) 2022-09-10T10:56:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:00 smithi110 ceph-mon[97633]: pgmap v12375: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:00.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:00.678561+0000) 2022-09-10T10:56:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:01.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:01.166254+0000) 2022-09-10T10:56:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:01.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:01.678791+0000) 2022-09-10T10:56:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:02 smithi104 ceph-mon[97044]: pgmap v12376: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:02.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:02.166490+0000) 2022-09-10T10:56:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:02 smithi110 ceph-mon[97633]: pgmap v12376: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:02.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:02.678988+0000) 2022-09-10T10:56:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:03.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:03.166666+0000) 2022-09-10T10:56:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:03.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:03.679134+0000) 2022-09-10T10:56:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:04 smithi104 ceph-mon[97044]: pgmap v12377: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:04.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:04.166848+0000) 2022-09-10T10:56:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:04 smithi110 ceph-mon[97633]: pgmap v12377: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:04.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:04.679364+0000) 2022-09-10T10:56:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:05.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:05.167081+0000) 2022-09-10T10:56:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:05.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:05.679588+0000) 2022-09-10T10:56:06.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:06 smithi104 ceph-mon[97044]: pgmap v12378: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:56:06.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:06 smithi110 ceph-mon[97633]: pgmap v12378: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:56:06.660 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:06.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:06.167294+0000) 2022-09-10T10:56:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:06.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:06.679813+0000) 2022-09-10T10:56:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:07.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:07.167465+0000) 2022-09-10T10:56:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:07.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:07.680028+0000) 2022-09-10T10:56:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:08 smithi104 ceph-mon[97044]: pgmap v12379: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:56:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:08.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:08.167699+0000) 2022-09-10T10:56:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:08 smithi110 ceph-mon[97633]: pgmap v12379: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:56:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:08.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:08.680264+0000) 2022-09-10T10:56:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:09 smithi104 ceph-mon[97044]: osdmap e854: 12 total, 12 up, 12 in 2022-09-10T10:56:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:09.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:09.167947+0000) 2022-09-10T10:56:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:09 smithi110 ceph-mon[97633]: osdmap e854: 12 total, 12 up, 12 in 2022-09-10T10:56:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:09.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:09.680424+0000) 2022-09-10T10:56:10.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:10 smithi104 ceph-mon[97044]: pgmap v12381: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:56:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:10.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:10.168219+0000) 2022-09-10T10:56:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:10 smithi110 ceph-mon[97633]: pgmap v12381: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:56:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:10.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:10.680653+0000) 2022-09-10T10:56:11.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:56:10.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:56:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:11.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:11.168387+0000) 2022-09-10T10:56:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:11.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:11.680878+0000) 2022-09-10T10:56:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:12 smithi104 ceph-mon[97044]: pgmap v12382: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:56:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:12.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:12.168596+0000) 2022-09-10T10:56:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:12 smithi110 ceph-mon[97633]: pgmap v12382: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:56:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:12.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:12.681135+0000) 2022-09-10T10:56:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:13.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:13.168863+0000) 2022-09-10T10:56:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:13.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:13.681393+0000) 2022-09-10T10:56:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:14 smithi104 ceph-mon[97044]: osdmap e855: 12 total, 12 up, 12 in 2022-09-10T10:56:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:14 smithi104 ceph-mon[97044]: pgmap v12384: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:56:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:14.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:14.169132+0000) 2022-09-10T10:56:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:14 smithi110 ceph-mon[97633]: osdmap e855: 12 total, 12 up, 12 in 2022-09-10T10:56:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:14 smithi110 ceph-mon[97633]: pgmap v12384: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:56:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:14.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:14.681618+0000) 2022-09-10T10:56:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:15.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:15.169361+0000) 2022-09-10T10:56:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:15.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:15.681814+0000) 2022-09-10T10:56:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:16 smithi104 ceph-mon[97044]: pgmap v12385: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:56:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:16.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:16.169602+0000) 2022-09-10T10:56:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:16 smithi110 ceph-mon[97633]: pgmap v12385: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:56:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:16.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:16.682010+0000) 2022-09-10T10:56:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:17.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:17.169824+0000) 2022-09-10T10:56:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:17.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:17.682284+0000) 2022-09-10T10:56:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:18 smithi104 ceph-mon[97044]: pgmap v12386: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:56:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:18.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:18.170007+0000) 2022-09-10T10:56:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:18 smithi110 ceph-mon[97633]: pgmap v12386: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:56:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:18.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:18.682520+0000) 2022-09-10T10:56:19.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:19.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:19.170196+0000) 2022-09-10T10:56:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:19.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:19.682760+0000) 2022-09-10T10:56:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:20 smithi104 ceph-mon[97044]: pgmap v12387: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:56:20.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:20.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:20.170377+0000) 2022-09-10T10:56:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:20 smithi110 ceph-mon[97633]: pgmap v12387: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:56:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:20.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:20.682978+0000) 2022-09-10T10:56:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:21.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:21.170605+0000) 2022-09-10T10:56:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:21.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:21.683188+0000) 2022-09-10T10:56:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:22 smithi104 ceph-mon[97044]: pgmap v12388: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:56:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:22.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:22.170860+0000) 2022-09-10T10:56:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:22 smithi110 ceph-mon[97633]: pgmap v12388: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:56:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:22.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:22.683444+0000) 2022-09-10T10:56:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:23.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:23.171067+0000) 2022-09-10T10:56:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:23.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:23.683704+0000) 2022-09-10T10:56:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:24.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:24.171298+0000) 2022-09-10T10:56:24.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:24 smithi104 ceph-mon[97044]: pgmap v12389: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:56:24.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:24 smithi110 ceph-mon[97633]: pgmap v12389: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:56:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:24.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:24.683974+0000) 2022-09-10T10:56:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:25.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:25.171552+0000) 2022-09-10T10:56:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:25.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:25.684191+0000) 2022-09-10T10:56:26.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:56:25.700+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:56:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:26 smithi104 ceph-mon[97044]: pgmap v12390: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:56:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:26.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:26.171760+0000) 2022-09-10T10:56:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:26 smithi110 ceph-mon[97633]: pgmap v12390: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:56:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:26.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:26.684433+0000) 2022-09-10T10:56:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:27.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:27.172000+0000) 2022-09-10T10:56:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:27.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:27.684687+0000) 2022-09-10T10:56:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:28 smithi104 ceph-mon[97044]: pgmap v12391: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:28.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:28.172248+0000) 2022-09-10T10:56:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:28 smithi110 ceph-mon[97633]: pgmap v12391: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:28.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:28.684906+0000) 2022-09-10T10:56:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:29.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:29.172478+0000) 2022-09-10T10:56:30.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:29.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:29.685197+0000) 2022-09-10T10:56:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:30 smithi104 ceph-mon[97044]: pgmap v12392: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:30.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:30.172654+0000) 2022-09-10T10:56:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:30 smithi110 ceph-mon[97633]: pgmap v12392: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:30.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:30.685412+0000) 2022-09-10T10:56:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:31.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:31.172886+0000) 2022-09-10T10:56:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:31.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:31.685625+0000) 2022-09-10T10:56:32.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:32 smithi104 ceph-mon[97044]: pgmap v12393: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:32.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:32.173157+0000) 2022-09-10T10:56:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:32 smithi110 ceph-mon[97633]: pgmap v12393: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:32.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:32.685877+0000) 2022-09-10T10:56:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:33.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:33.173391+0000) 2022-09-10T10:56:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:33.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:33.686181+0000) 2022-09-10T10:56:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:34 smithi104 ceph-mon[97044]: pgmap v12394: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:34.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:34.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:34.173612+0000) 2022-09-10T10:56:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:34 smithi110 ceph-mon[97633]: pgmap v12394: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:34.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:34.686357+0000) 2022-09-10T10:56:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:35.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:35.173805+0000) 2022-09-10T10:56:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:35.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:35.686562+0000) 2022-09-10T10:56:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:36 smithi104 ceph-mon[97044]: pgmap v12395: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:36.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:36.174028+0000) 2022-09-10T10:56:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:36 smithi110 ceph-mon[97633]: pgmap v12395: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:36.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:36.686747+0000) 2022-09-10T10:56:37.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:37.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:37.174254+0000) 2022-09-10T10:56:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:37.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:37.687000+0000) 2022-09-10T10:56:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:38 smithi104 ceph-mon[97044]: pgmap v12396: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:38.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:38.174479+0000) 2022-09-10T10:56:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:38 smithi110 ceph-mon[97633]: pgmap v12396: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:38.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:38.687209+0000) 2022-09-10T10:56:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:39.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:39.174654+0000) 2022-09-10T10:56:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:39.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:39.687430+0000) 2022-09-10T10:56:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:40 smithi104 ceph-mon[97044]: pgmap v12397: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:40.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:40.174903+0000) 2022-09-10T10:56:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:40 smithi110 ceph-mon[97633]: pgmap v12397: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:40.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:40.687662+0000) 2022-09-10T10:56:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:56:40.716+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:56:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:41.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:41.175083+0000) 2022-09-10T10:56:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:41.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:41.687804+0000) 2022-09-10T10:56:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:42 smithi104 ceph-mon[97044]: pgmap v12398: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:42.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:42.175332+0000) 2022-09-10T10:56:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:42 smithi110 ceph-mon[97633]: pgmap v12398: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:42.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:42.688074+0000) 2022-09-10T10:56:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:43.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:43.175559+0000) 2022-09-10T10:56:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:43.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:43.688257+0000) 2022-09-10T10:56:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:44 smithi104 ceph-mon[97044]: pgmap v12399: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:44.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:44.175803+0000) 2022-09-10T10:56:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:44 smithi110 ceph-mon[97633]: pgmap v12399: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:44.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:44.688433+0000) 2022-09-10T10:56:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:45.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:45.176077+0000) 2022-09-10T10:56:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:45.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:45.688713+0000) 2022-09-10T10:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:46 smithi104 ceph-mon[97044]: pgmap v12400: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:56:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:46.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:46.176294+0000) 2022-09-10T10:56:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:46 smithi110 ceph-mon[97633]: pgmap v12400: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:56:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:56:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:56:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:56:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:46.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:46.688954+0000) 2022-09-10T10:56:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:47.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:47.176550+0000) 2022-09-10T10:56:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:47.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:47.689189+0000) 2022-09-10T10:56:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:48 smithi104 ceph-mon[97044]: pgmap v12401: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:48.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:48.176789+0000) 2022-09-10T10:56:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:48 smithi110 ceph-mon[97633]: pgmap v12401: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:48.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:48.689389+0000) 2022-09-10T10:56:49.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:49.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:49.177055+0000) 2022-09-10T10:56:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:49.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:49.689598+0000) 2022-09-10T10:56:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:50 smithi104 ceph-mon[97044]: pgmap v12402: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:50.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:50.177206+0000) 2022-09-10T10:56:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:50 smithi110 ceph-mon[97633]: pgmap v12402: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:51.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:50.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:50.689849+0000) 2022-09-10T10:56:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:51.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:51.177410+0000) 2022-09-10T10:56:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:51.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:51.690131+0000) 2022-09-10T10:56:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:52 smithi104 ceph-mon[97044]: pgmap v12403: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:52.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:52.177593+0000) 2022-09-10T10:56:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:52 smithi110 ceph-mon[97633]: pgmap v12403: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:52.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:52.690337+0000) 2022-09-10T10:56:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:53.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:53.177719+0000) 2022-09-10T10:56:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:53.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:53.690576+0000) 2022-09-10T10:56:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:54 smithi104 ceph-mon[97044]: pgmap v12404: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:54.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:54.177943+0000) 2022-09-10T10:56:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:54 smithi110 ceph-mon[97633]: pgmap v12404: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:54.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:54.690788+0000) 2022-09-10T10:56:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:55.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:55.178201+0000) 2022-09-10T10:56:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:55.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:55.690954+0000) 2022-09-10T10:56:56.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:56:55.731+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:56:56.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:56 smithi104 ceph-mon[97044]: pgmap v12405: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:56.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:56.178397+0000) 2022-09-10T10:56:56.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:56 smithi110 ceph-mon[97633]: pgmap v12405: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:56:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:56.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:56.691212+0000) 2022-09-10T10:56:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:57.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:57.178626+0000) 2022-09-10T10:56:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:57.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:57.691417+0000) 2022-09-10T10:56:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:56:58 smithi104 ceph-mon[97044]: pgmap v12406: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:58.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:58.178830+0000) 2022-09-10T10:56:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:56:58 smithi110 ceph-mon[97633]: pgmap v12406: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:56:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:58.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:58.691601+0000) 2022-09-10T10:56:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:56:59.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:59.179024+0000) 2022-09-10T10:57:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:56:59.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:56:59.691841+0000) 2022-09-10T10:57:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:00 smithi104 ceph-mon[97044]: pgmap v12407: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:00.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:00.179251+0000) 2022-09-10T10:57:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:00 smithi110 ceph-mon[97633]: pgmap v12407: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:00.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:00.691993+0000) 2022-09-10T10:57:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:01.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:01.179485+0000) 2022-09-10T10:57:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:01.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:01.692204+0000) 2022-09-10T10:57:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:02 smithi104 ceph-mon[97044]: pgmap v12408: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:02.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:02.179764+0000) 2022-09-10T10:57:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:02 smithi110 ceph-mon[97633]: pgmap v12408: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:02.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:02.692411+0000) 2022-09-10T10:57:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:03.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:03.180041+0000) 2022-09-10T10:57:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:03.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:03.692665+0000) 2022-09-10T10:57:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:04 smithi104 ceph-mon[97044]: pgmap v12409: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:04.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:04.180237+0000) 2022-09-10T10:57:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:04 smithi110 ceph-mon[97633]: pgmap v12409: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:04.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:04.692949+0000) 2022-09-10T10:57:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:05.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:05.180404+0000) 2022-09-10T10:57:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:05.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:05.693201+0000) 2022-09-10T10:57:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:06 smithi104 ceph-mon[97044]: pgmap v12410: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:57:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:06.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:06.180639+0000) 2022-09-10T10:57:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:06 smithi110 ceph-mon[97633]: pgmap v12410: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:57:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:06.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:06.693419+0000) 2022-09-10T10:57:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:07.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:07.180894+0000) 2022-09-10T10:57:08.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:07.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:07.693654+0000) 2022-09-10T10:57:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:08 smithi104 ceph-mon[97044]: pgmap v12411: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:57:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:08.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:08.181166+0000) 2022-09-10T10:57:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:08 smithi110 ceph-mon[97633]: pgmap v12411: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:57:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:08.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:08.693867+0000) 2022-09-10T10:57:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:09 smithi104 ceph-mon[97044]: osdmap e856: 12 total, 12 up, 12 in 2022-09-10T10:57:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:09.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:09.181353+0000) 2022-09-10T10:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:09 smithi110 ceph-mon[97633]: osdmap e856: 12 total, 12 up, 12 in 2022-09-10T10:57:10.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:09.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:09.694068+0000) 2022-09-10T10:57:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:10.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:10.181621+0000) 2022-09-10T10:57:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:10 smithi110 ceph-mon[97633]: pgmap v12413: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:57:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:10 smithi104 ceph-mon[97044]: pgmap v12413: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:57:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:57:10.746+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:57:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:10.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:10.694321+0000) 2022-09-10T10:57:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:11.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:11.181908+0000) 2022-09-10T10:57:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:11.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:11.694571+0000) 2022-09-10T10:57:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:12.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:12.182197+0000) 2022-09-10T10:57:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:12 smithi110 ceph-mon[97633]: pgmap v12414: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:57:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:12 smithi104 ceph-mon[97044]: pgmap v12414: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:57:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:12.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:12.694808+0000) 2022-09-10T10:57:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:13.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:13.182475+0000) 2022-09-10T10:57:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:13.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:13.695067+0000) 2022-09-10T10:57:14.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:14 smithi104 ceph-mon[97044]: osdmap e857: 12 total, 12 up, 12 in 2022-09-10T10:57:14.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:14 smithi104 ceph-mon[97044]: pgmap v12416: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:57:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:14.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:14.182740+0000) 2022-09-10T10:57:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:14 smithi110 ceph-mon[97633]: osdmap e857: 12 total, 12 up, 12 in 2022-09-10T10:57:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:14 smithi110 ceph-mon[97633]: pgmap v12416: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:57:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:14.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:14.695275+0000) 2022-09-10T10:57:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:15.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:15.183010+0000) 2022-09-10T10:57:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:15.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:15.695487+0000) 2022-09-10T10:57:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:16 smithi104 ceph-mon[97044]: pgmap v12417: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:57:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:16.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:16.183266+0000) 2022-09-10T10:57:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:16 smithi110 ceph-mon[97633]: pgmap v12417: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:57:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:16.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:16.695662+0000) 2022-09-10T10:57:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:17.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:17.183454+0000) 2022-09-10T10:57:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:17.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:17.695895+0000) 2022-09-10T10:57:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:18 smithi104 ceph-mon[97044]: pgmap v12418: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:57:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:18.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:18.183681+0000) 2022-09-10T10:57:18.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:18 smithi110 ceph-mon[97633]: pgmap v12418: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T10:57:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:18.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:18.696157+0000) 2022-09-10T10:57:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:19.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:19.183930+0000) 2022-09-10T10:57:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:19.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:19.696397+0000) 2022-09-10T10:57:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:20 smithi104 ceph-mon[97044]: pgmap v12419: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:57:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:20.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:20.184149+0000) 2022-09-10T10:57:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:20 smithi110 ceph-mon[97633]: pgmap v12419: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:57:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:20.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:20.696630+0000) 2022-09-10T10:57:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:21.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:21.184288+0000) 2022-09-10T10:57:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:21.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:21.696805+0000) 2022-09-10T10:57:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:22 smithi104 ceph-mon[97044]: pgmap v12420: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:57:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:22.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:22.184472+0000) 2022-09-10T10:57:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:22 smithi110 ceph-mon[97633]: pgmap v12420: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:57:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:22.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:22.697065+0000) 2022-09-10T10:57:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:23.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:23.184710+0000) 2022-09-10T10:57:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:23.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:23.697309+0000) 2022-09-10T10:57:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:24 smithi104 ceph-mon[97044]: pgmap v12421: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T10:57:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:24.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:24.184915+0000) 2022-09-10T10:57:24.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:24 smithi110 ceph-mon[97633]: pgmap v12421: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T10:57:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:24.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:24.697504+0000) 2022-09-10T10:57:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:25.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:25.185171+0000) 2022-09-10T10:57:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:25.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:25.697765+0000) 2022-09-10T10:57:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:57:25.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:57:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:26 smithi104 ceph-mon[97044]: pgmap v12422: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:57:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:26.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:26.185398+0000) 2022-09-10T10:57:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:26 smithi110 ceph-mon[97633]: pgmap v12422: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:57:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:26.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:26.698006+0000) 2022-09-10T10:57:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:27.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:27.185630+0000) 2022-09-10T10:57:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:27.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:27.698213+0000) 2022-09-10T10:57:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:28 smithi104 ceph-mon[97044]: pgmap v12423: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:28.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:28.185840+0000) 2022-09-10T10:57:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:28 smithi110 ceph-mon[97633]: pgmap v12423: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:28.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:28.698423+0000) 2022-09-10T10:57:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:29.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:29.186045+0000) 2022-09-10T10:57:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:29.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:29.698566+0000) 2022-09-10T10:57:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:30 smithi104 ceph-mon[97044]: pgmap v12424: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:30.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:30.186230+0000) 2022-09-10T10:57:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:30 smithi110 ceph-mon[97633]: pgmap v12424: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:30.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:30.698742+0000) 2022-09-10T10:57:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:31.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:31.186477+0000) 2022-09-10T10:57:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:31.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:31.698978+0000) 2022-09-10T10:57:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:32.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:32.186671+0000) 2022-09-10T10:57:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:32 smithi110 ceph-mon[97633]: pgmap v12425: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:32 smithi104 ceph-mon[97044]: pgmap v12425: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:32.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:32.699165+0000) 2022-09-10T10:57:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:33.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:33.186895+0000) 2022-09-10T10:57:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:33.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:33.699408+0000) 2022-09-10T10:57:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:34 smithi104 ceph-mon[97044]: pgmap v12426: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:34.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:34.187080+0000) 2022-09-10T10:57:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:34 smithi110 ceph-mon[97633]: pgmap v12426: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:34.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:34.699607+0000) 2022-09-10T10:57:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:35.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:35.187313+0000) 2022-09-10T10:57:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:35.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:35.699818+0000) 2022-09-10T10:57:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:36 smithi104 ceph-mon[97044]: pgmap v12427: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:36.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:36.187502+0000) 2022-09-10T10:57:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:36 smithi110 ceph-mon[97633]: pgmap v12427: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:36.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:36.700031+0000) 2022-09-10T10:57:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:37.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:37.187746+0000) 2022-09-10T10:57:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:37.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:37.700251+0000) 2022-09-10T10:57:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:38 smithi104 ceph-mon[97044]: pgmap v12428: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:38.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:38.188023+0000) 2022-09-10T10:57:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:38 smithi110 ceph-mon[97633]: pgmap v12428: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:38.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:38.700480+0000) 2022-09-10T10:57:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:39.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:39.188159+0000) 2022-09-10T10:57:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:39.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:39.700656+0000) 2022-09-10T10:57:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:40.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:40.188326+0000) 2022-09-10T10:57:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:40 smithi110 ceph-mon[97633]: pgmap v12429: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:40 smithi104 ceph-mon[97044]: pgmap v12429: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:40.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:40.700908+0000) 2022-09-10T10:57:41.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:57:40.762+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:57:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:41.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:41.188511+0000) 2022-09-10T10:57:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:41.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:41.701119+0000) 2022-09-10T10:57:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:42.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:42.188755+0000) 2022-09-10T10:57:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:42 smithi110 ceph-mon[97633]: pgmap v12430: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:42 smithi104 ceph-mon[97044]: pgmap v12430: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:42.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:42.701343+0000) 2022-09-10T10:57:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:43.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:43.189046+0000) 2022-09-10T10:57:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:43.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:43.701616+0000) 2022-09-10T10:57:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:44 smithi104 ceph-mon[97044]: pgmap v12431: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:44.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:44.189255+0000) 2022-09-10T10:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:44 smithi110 ceph-mon[97633]: pgmap v12431: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:44.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:44.701869+0000) 2022-09-10T10:57:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:45.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:45.189480+0000) 2022-09-10T10:57:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:45.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:45.702148+0000) 2022-09-10T10:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:46 smithi104 ceph-mon[97044]: pgmap v12432: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:57:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:57:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:57:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:57:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:46.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:46.189712+0000) 2022-09-10T10:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:46 smithi110 ceph-mon[97633]: pgmap v12432: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:57:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:57:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:57:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:57:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:46.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:46.702374+0000) 2022-09-10T10:57:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:47.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:47.189925+0000) 2022-09-10T10:57:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:47.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:47.702514+0000) 2022-09-10T10:57:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:48.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:48.190184+0000) 2022-09-10T10:57:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:48 smithi110 ceph-mon[97633]: pgmap v12433: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:48 smithi104 ceph-mon[97044]: pgmap v12433: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:48.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:48.702740+0000) 2022-09-10T10:57:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:49.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:49.190351+0000) 2022-09-10T10:57:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:49.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:49.702911+0000) 2022-09-10T10:57:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:50.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:50.190493+0000) 2022-09-10T10:57:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:50 smithi110 ceph-mon[97633]: pgmap v12434: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:50 smithi104 ceph-mon[97044]: pgmap v12434: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:50.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:50.703155+0000) 2022-09-10T10:57:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:51.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:51.190635+0000) 2022-09-10T10:57:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:51.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:51.703284+0000) 2022-09-10T10:57:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:52.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:52.190866+0000) 2022-09-10T10:57:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:52 smithi110 ceph-mon[97633]: pgmap v12435: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:52 smithi104 ceph-mon[97044]: pgmap v12435: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:52.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:52.703482+0000) 2022-09-10T10:57:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:53.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:53.191056+0000) 2022-09-10T10:57:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:53.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:53.703671+0000) 2022-09-10T10:57:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:54 smithi104 ceph-mon[97044]: pgmap v12436: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:54.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:54.191151+0000) 2022-09-10T10:57:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:54 smithi110 ceph-mon[97633]: pgmap v12436: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:54.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:54.703906+0000) 2022-09-10T10:57:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:55.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:55.191339+0000) 2022-09-10T10:57:56.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:57:55.767+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:57:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:55.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:55.704164+0000) 2022-09-10T10:57:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:56.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:56.191531+0000) 2022-09-10T10:57:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:56 smithi110 ceph-mon[97633]: pgmap v12437: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:56 smithi104 ceph-mon[97044]: pgmap v12437: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:57:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:56.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:56.704344+0000) 2022-09-10T10:57:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:57.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:57.191771+0000) 2022-09-10T10:57:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:57.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:57.704574+0000) 2022-09-10T10:57:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:58.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:58.191934+0000) 2022-09-10T10:57:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:57:58 smithi110 ceph-mon[97633]: pgmap v12438: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:57:58 smithi104 ceph-mon[97044]: pgmap v12438: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:57:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:58.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:58.704717+0000) 2022-09-10T10:57:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:57:59.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:59.192146+0000) 2022-09-10T10:58:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:57:59.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:57:59.704967+0000) 2022-09-10T10:58:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:00.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:00.192385+0000) 2022-09-10T10:58:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:00 smithi110 ceph-mon[97633]: pgmap v12439: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:00 smithi104 ceph-mon[97044]: pgmap v12439: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:00.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:00.705189+0000) 2022-09-10T10:58:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:01.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:01.192555+0000) 2022-09-10T10:58:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:01.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:01.705423+0000) 2022-09-10T10:58:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:02.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:02.192752+0000) 2022-09-10T10:58:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:02 smithi110 ceph-mon[97633]: pgmap v12440: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:02 smithi104 ceph-mon[97044]: pgmap v12440: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:02.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:02.705586+0000) 2022-09-10T10:58:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:03.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:03.192997+0000) 2022-09-10T10:58:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:03.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:03.705868+0000) 2022-09-10T10:58:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:04 smithi104 ceph-mon[97044]: pgmap v12441: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:04.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:04.193271+0000) 2022-09-10T10:58:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:04 smithi110 ceph-mon[97633]: pgmap v12441: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:04.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:04.706151+0000) 2022-09-10T10:58:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:05.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:05.193490+0000) 2022-09-10T10:58:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:05.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:05.706378+0000) 2022-09-10T10:58:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:06.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:06.193692+0000) 2022-09-10T10:58:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:06 smithi110 ceph-mon[97633]: pgmap v12442: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:58:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:06 smithi104 ceph-mon[97044]: pgmap v12442: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:58:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:06.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:06.706540+0000) 2022-09-10T10:58:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:07.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:07.193918+0000) 2022-09-10T10:58:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:07.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:07.706821+0000) 2022-09-10T10:58:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:08.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:08.194184+0000) 2022-09-10T10:58:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:08 smithi110 ceph-mon[97633]: pgmap v12443: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:58:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:08 smithi104 ceph-mon[97044]: pgmap v12443: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:58:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:08.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:08.707006+0000) 2022-09-10T10:58:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:09.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:09.194362+0000) 2022-09-10T10:58:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:09 smithi110 ceph-mon[97633]: osdmap e858: 12 total, 12 up, 12 in 2022-09-10T10:58:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:09 smithi104 ceph-mon[97044]: osdmap e858: 12 total, 12 up, 12 in 2022-09-10T10:58:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:09.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:09.707209+0000) 2022-09-10T10:58:10.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:10.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:10.194586+0000) 2022-09-10T10:58:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:10 smithi110 ceph-mon[97633]: pgmap v12445: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:58:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:10 smithi104 ceph-mon[97044]: pgmap v12445: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:58:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:58:10.782+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:58:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:10.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:10.707391+0000) 2022-09-10T10:58:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:11.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:11.194767+0000) 2022-09-10T10:58:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:11.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:11.707637+0000) 2022-09-10T10:58:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:12.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:12.194998+0000) 2022-09-10T10:58:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:12 smithi110 ceph-mon[97633]: pgmap v12446: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:58:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:12 smithi104 ceph-mon[97044]: pgmap v12446: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:58:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:12.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:12.707887+0000) 2022-09-10T10:58:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:13.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:13.195223+0000) 2022-09-10T10:58:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:13.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:13.708205+0000) 2022-09-10T10:58:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:14 smithi104 ceph-mon[97044]: osdmap e859: 12 total, 12 up, 12 in 2022-09-10T10:58:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:14 smithi104 ceph-mon[97044]: pgmap v12448: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:58:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:14.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:14.195459+0000) 2022-09-10T10:58:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:14 smithi110 ceph-mon[97633]: osdmap e859: 12 total, 12 up, 12 in 2022-09-10T10:58:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:14 smithi110 ceph-mon[97633]: pgmap v12448: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:58:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:14.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:14.708416+0000) 2022-09-10T10:58:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:15.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:15.195620+0000) 2022-09-10T10:58:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:15.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:15.708621+0000) 2022-09-10T10:58:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:16.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:16.195849+0000) 2022-09-10T10:58:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:16 smithi110 ceph-mon[97633]: pgmap v12449: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:58:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:16 smithi104 ceph-mon[97044]: pgmap v12449: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:58:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:16.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:16.708872+0000) 2022-09-10T10:58:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:17.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:17.196070+0000) 2022-09-10T10:58:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:17.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:17.709068+0000) 2022-09-10T10:58:18.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:18.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:18.196314+0000) 2022-09-10T10:58:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:18 smithi110 ceph-mon[97633]: pgmap v12450: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:58:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:18 smithi104 ceph-mon[97044]: pgmap v12450: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:58:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:18.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:18.709334+0000) 2022-09-10T10:58:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:19.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:19.196574+0000) 2022-09-10T10:58:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:19.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:19.709601+0000) 2022-09-10T10:58:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:20.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:20.196771+0000) 2022-09-10T10:58:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:20 smithi110 ceph-mon[97633]: pgmap v12451: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:58:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:20 smithi104 ceph-mon[97044]: pgmap v12451: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:58:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:20.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:20.709832+0000) 2022-09-10T10:58:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:21.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:21.197003+0000) 2022-09-10T10:58:22.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:21.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:21.710073+0000) 2022-09-10T10:58:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:22.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:22.197253+0000) 2022-09-10T10:58:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:22 smithi110 ceph-mon[97633]: pgmap v12452: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:58:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:22 smithi104 ceph-mon[97044]: pgmap v12452: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:58:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:22.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:22.710307+0000) 2022-09-10T10:58:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:23.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:23.197482+0000) 2022-09-10T10:58:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:23.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:23.710510+0000) 2022-09-10T10:58:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:24 smithi104 ceph-mon[97044]: pgmap v12453: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T10:58:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:24.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:24.197686+0000) 2022-09-10T10:58:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:24 smithi110 ceph-mon[97633]: pgmap v12453: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T10:58:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:24.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:24.710705+0000) 2022-09-10T10:58:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:25.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:25.197834+0000) 2022-09-10T10:58:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:25.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:25.710933+0000) 2022-09-10T10:58:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:58:25.798+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:58:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:26.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:26.198054+0000) 2022-09-10T10:58:26.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:26 smithi110 ceph-mon[97633]: pgmap v12454: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:58:26.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:26 smithi104 ceph-mon[97044]: pgmap v12454: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:58:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:26.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:26.711149+0000) 2022-09-10T10:58:27.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:27.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:27.198279+0000) 2022-09-10T10:58:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:27.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:27.711379+0000) 2022-09-10T10:58:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:28.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:28.198513+0000) 2022-09-10T10:58:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:28 smithi110 ceph-mon[97633]: pgmap v12455: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:28 smithi104 ceph-mon[97044]: pgmap v12455: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:28.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:28.711576+0000) 2022-09-10T10:58:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:29.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:29.198680+0000) 2022-09-10T10:58:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:29.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:29.711836+0000) 2022-09-10T10:58:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:30.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:30.198826+0000) 2022-09-10T10:58:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:30 smithi110 ceph-mon[97633]: pgmap v12456: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:30 smithi104 ceph-mon[97044]: pgmap v12456: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:30.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:30.712065+0000) 2022-09-10T10:58:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:31.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:31.199040+0000) 2022-09-10T10:58:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:31.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:31.712355+0000) 2022-09-10T10:58:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:32.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:32.199286+0000) 2022-09-10T10:58:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:32 smithi110 ceph-mon[97633]: pgmap v12457: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:32.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:32 smithi104 ceph-mon[97044]: pgmap v12457: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:32.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:32.712551+0000) 2022-09-10T10:58:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:33.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:33.199468+0000) 2022-09-10T10:58:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:33.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:33.712802+0000) 2022-09-10T10:58:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:34 smithi104 ceph-mon[97044]: pgmap v12458: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:34.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:34.199653+0000) 2022-09-10T10:58:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:34 smithi110 ceph-mon[97633]: pgmap v12458: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:34.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:34.712999+0000) 2022-09-10T10:58:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:35.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:35.199888+0000) 2022-09-10T10:58:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:35.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:35.713214+0000) 2022-09-10T10:58:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:36.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:36.200140+0000) 2022-09-10T10:58:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:36 smithi110 ceph-mon[97633]: pgmap v12459: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:36 smithi104 ceph-mon[97044]: pgmap v12459: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:36.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:36.713461+0000) 2022-09-10T10:58:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:37.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:37.200267+0000) 2022-09-10T10:58:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:37.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:37.713723+0000) 2022-09-10T10:58:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:38.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:38.200398+0000) 2022-09-10T10:58:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:38 smithi110 ceph-mon[97633]: pgmap v12460: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:38 smithi104 ceph-mon[97044]: pgmap v12460: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:38.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:38.713975+0000) 2022-09-10T10:58:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:39.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:39.200541+0000) 2022-09-10T10:58:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:39.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:39.714248+0000) 2022-09-10T10:58:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:40.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:40.200777+0000) 2022-09-10T10:58:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:40 smithi110 ceph-mon[97633]: pgmap v12461: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:40 smithi104 ceph-mon[97044]: pgmap v12461: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:40.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:40.714501+0000) 2022-09-10T10:58:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:58:40.803+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:58:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:41.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:41.200979+0000) 2022-09-10T10:58:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:41.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:41.714780+0000) 2022-09-10T10:58:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:42.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:42.201189+0000) 2022-09-10T10:58:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:42 smithi110 ceph-mon[97633]: pgmap v12462: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:42 smithi104 ceph-mon[97044]: pgmap v12462: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:42.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:42.715049+0000) 2022-09-10T10:58:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:43.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:43.201419+0000) 2022-09-10T10:58:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:43.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:43.715348+0000) 2022-09-10T10:58:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:44 smithi104 ceph-mon[97044]: pgmap v12463: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:44.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:44.201656+0000) 2022-09-10T10:58:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:44 smithi110 ceph-mon[97633]: pgmap v12463: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:44.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:44.715621+0000) 2022-09-10T10:58:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:45.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:45.201844+0000) 2022-09-10T10:58:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:45.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:45.715876+0000) 2022-09-10T10:58:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:46.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:46.202003+0000) 2022-09-10T10:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:46 smithi110 ceph-mon[97633]: pgmap v12464: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:58:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:58:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:58:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:58:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:46 smithi104 ceph-mon[97044]: pgmap v12464: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:58:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:58:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:58:46.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:58:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:46.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:46.716174+0000) 2022-09-10T10:58:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:47.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:47.202235+0000) 2022-09-10T10:58:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:47.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:47.716353+0000) 2022-09-10T10:58:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:48.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:48.202479+0000) 2022-09-10T10:58:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:48 smithi110 ceph-mon[97633]: pgmap v12465: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:48 smithi104 ceph-mon[97044]: pgmap v12465: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:48.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:48.716622+0000) 2022-09-10T10:58:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:49.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:49.202700+0000) 2022-09-10T10:58:50.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:49.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:49.716837+0000) 2022-09-10T10:58:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:50.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:50.202939+0000) 2022-09-10T10:58:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:50 smithi110 ceph-mon[97633]: pgmap v12466: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:58:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:50 smithi104 ceph-mon[97044]: pgmap v12466: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:58:51.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:50.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:50.717051+0000) 2022-09-10T10:58:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:51.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:51.203207+0000) 2022-09-10T10:58:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:51.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:51.717366+0000) 2022-09-10T10:58:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:52.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:52.203368+0000) 2022-09-10T10:58:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:52 smithi110 ceph-mon[97633]: pgmap v12467: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:52 smithi104 ceph-mon[97044]: pgmap v12467: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:52.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:52.717609+0000) 2022-09-10T10:58:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:53.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:53.203623+0000) 2022-09-10T10:58:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:53.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:53.717821+0000) 2022-09-10T10:58:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:54 smithi104 ceph-mon[97044]: pgmap v12468: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:54.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:54.203892+0000) 2022-09-10T10:58:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:54 smithi110 ceph-mon[97633]: pgmap v12468: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:54.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:54.718075+0000) 2022-09-10T10:58:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:55.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:55.204127+0000) 2022-09-10T10:58:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:55.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:55.718300+0000) 2022-09-10T10:58:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:58:55.816+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:58:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:56.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:56.204264+0000) 2022-09-10T10:58:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:56 smithi110 ceph-mon[97633]: pgmap v12469: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:56 smithi104 ceph-mon[97044]: pgmap v12469: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:58:57.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:56.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:56.718470+0000) 2022-09-10T10:58:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:57.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:57.204453+0000) 2022-09-10T10:58:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:57.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:57.718699+0000) 2022-09-10T10:58:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:58.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:58.204627+0000) 2022-09-10T10:58:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:58:58 smithi110 ceph-mon[97633]: pgmap v12470: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:58:58 smithi104 ceph-mon[97044]: pgmap v12470: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:58:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:58.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:58.718878+0000) 2022-09-10T10:58:59.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:58:59.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:59.204792+0000) 2022-09-10T10:59:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:58:59.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:58:59.719147+0000) 2022-09-10T10:59:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:00.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:00.204969+0000) 2022-09-10T10:59:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:00 smithi110 ceph-mon[97633]: pgmap v12471: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:59:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:00 smithi104 ceph-mon[97044]: pgmap v12471: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T10:59:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:00.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:00.719345+0000) 2022-09-10T10:59:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:01.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:01.205165+0000) 2022-09-10T10:59:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:01.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:01.719574+0000) 2022-09-10T10:59:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:02.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:02.205413+0000) 2022-09-10T10:59:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:02 smithi110 ceph-mon[97633]: pgmap v12472: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:59:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:02 smithi104 ceph-mon[97044]: pgmap v12472: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T10:59:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:02.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:02.719793+0000) 2022-09-10T10:59:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:03.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:03.205590+0000) 2022-09-10T10:59:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:03.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:03.719978+0000) 2022-09-10T10:59:04.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:04 smithi104 ceph-mon[97044]: pgmap v12473: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:04.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:04.205790+0000) 2022-09-10T10:59:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:04 smithi110 ceph-mon[97633]: pgmap v12473: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:04.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:04.720157+0000) 2022-09-10T10:59:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:05.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:05.206054+0000) 2022-09-10T10:59:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:05.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:05.720370+0000) 2022-09-10T10:59:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:06.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:06.206276+0000) 2022-09-10T10:59:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:06 smithi110 ceph-mon[97633]: pgmap v12474: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:59:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:06 smithi104 ceph-mon[97044]: pgmap v12474: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T10:59:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:06.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:06.720613+0000) 2022-09-10T10:59:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:07.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:07.206515+0000) 2022-09-10T10:59:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:07.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:07.720813+0000) 2022-09-10T10:59:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:08.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:08.206749+0000) 2022-09-10T10:59:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:08 smithi110 ceph-mon[97633]: pgmap v12475: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:59:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:08 smithi104 ceph-mon[97044]: pgmap v12475: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T10:59:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:08.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:08.721015+0000) 2022-09-10T10:59:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:09.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:09.206966+0000) 2022-09-10T10:59:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:09 smithi110 ceph-mon[97633]: osdmap e860: 12 total, 12 up, 12 in 2022-09-10T10:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:09 smithi104 ceph-mon[97044]: osdmap e860: 12 total, 12 up, 12 in 2022-09-10T10:59:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:09.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:09.721226+0000) 2022-09-10T10:59:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:10.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:10.207166+0000) 2022-09-10T10:59:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:10 smithi110 ceph-mon[97633]: pgmap v12477: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:59:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:10 smithi104 ceph-mon[97044]: pgmap v12477: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:59:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:10.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:10.721442+0000) 2022-09-10T10:59:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:59:10.823+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:59:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:11.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:11.207364+0000) 2022-09-10T10:59:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:11.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:11.721603+0000) 2022-09-10T10:59:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:12.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:12.207624+0000) 2022-09-10T10:59:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:12 smithi110 ceph-mon[97633]: pgmap v12478: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:59:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:12 smithi104 ceph-mon[97044]: pgmap v12478: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T10:59:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:12.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:12.721843+0000) 2022-09-10T10:59:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:13.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:13.207876+0000) 2022-09-10T10:59:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:13.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:13.722028+0000) 2022-09-10T10:59:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:14 smithi104 ceph-mon[97044]: osdmap e861: 12 total, 12 up, 12 in 2022-09-10T10:59:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:14 smithi104 ceph-mon[97044]: pgmap v12480: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:59:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:14.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:14.208038+0000) 2022-09-10T10:59:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:14 smithi110 ceph-mon[97633]: osdmap e861: 12 total, 12 up, 12 in 2022-09-10T10:59:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:14 smithi110 ceph-mon[97633]: pgmap v12480: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T10:59:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:14.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:14.722260+0000) 2022-09-10T10:59:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:15.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:15.208288+0000) 2022-09-10T10:59:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:15.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:15.722511+0000) 2022-09-10T10:59:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:16.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:16.208464+0000) 2022-09-10T10:59:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:16 smithi110 ceph-mon[97633]: pgmap v12481: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:59:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:16 smithi104 ceph-mon[97044]: pgmap v12481: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T10:59:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:16.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:16.722669+0000) 2022-09-10T10:59:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:17.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:17.208672+0000) 2022-09-10T10:59:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:17.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:17.722894+0000) 2022-09-10T10:59:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:18.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:18.208915+0000) 2022-09-10T10:59:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:18 smithi110 ceph-mon[97633]: pgmap v12482: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:59:18.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:18 smithi104 ceph-mon[97044]: pgmap v12482: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T10:59:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:18.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:18.723171+0000) 2022-09-10T10:59:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:19.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:19.209110+0000) 2022-09-10T10:59:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:19.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:19.723320+0000) 2022-09-10T10:59:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:20.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:20.209283+0000) 2022-09-10T10:59:20.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:20 smithi110 ceph-mon[97633]: pgmap v12483: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:59:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:20 smithi104 ceph-mon[97044]: pgmap v12483: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T10:59:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:20.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:20.723542+0000) 2022-09-10T10:59:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:21.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:21.209455+0000) 2022-09-10T10:59:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:21.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:21.723721+0000) 2022-09-10T10:59:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:22.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:22.209640+0000) 2022-09-10T10:59:22.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:22 smithi110 ceph-mon[97633]: pgmap v12484: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:59:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:22 smithi104 ceph-mon[97044]: pgmap v12484: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T10:59:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:22.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:22.723960+0000) 2022-09-10T10:59:23.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:23.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:23.209862+0000) 2022-09-10T10:59:24.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:23.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:23.724221+0000) 2022-09-10T10:59:24.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:24 smithi104 ceph-mon[97044]: pgmap v12485: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T10:59:24.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:24.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:24.210081+0000) 2022-09-10T10:59:24.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:24 smithi110 ceph-mon[97633]: pgmap v12485: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T10:59:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:24.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:24.724379+0000) 2022-09-10T10:59:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:25.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:25.210300+0000) 2022-09-10T10:59:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:59:25.833+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:59:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:25.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:25.724636+0000) 2022-09-10T10:59:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:26.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:26.210503+0000) 2022-09-10T10:59:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:26 smithi110 ceph-mon[97633]: pgmap v12486: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:59:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:26 smithi104 ceph-mon[97044]: pgmap v12486: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T10:59:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:26.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:26.724847+0000) 2022-09-10T10:59:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:27.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:27.210740+0000) 2022-09-10T10:59:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:27.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:27.725084+0000) 2022-09-10T10:59:28.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:28.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:28.210977+0000) 2022-09-10T10:59:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:28 smithi110 ceph-mon[97633]: pgmap v12487: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:28.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:28 smithi104 ceph-mon[97044]: pgmap v12487: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:28.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:28.725370+0000) 2022-09-10T10:59:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:29.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:29.211206+0000) 2022-09-10T10:59:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:29.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:29.725514+0000) 2022-09-10T10:59:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:30.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:30.211419+0000) 2022-09-10T10:59:30.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:30 smithi110 ceph-mon[97633]: pgmap v12488: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T10:59:30.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:30 smithi104 ceph-mon[97044]: pgmap v12488: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T10:59:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:30.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:30.725708+0000) 2022-09-10T10:59:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:31.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:31.211634+0000) 2022-09-10T10:59:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:31.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:31.725948+0000) 2022-09-10T10:59:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:32.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:32.211867+0000) 2022-09-10T10:59:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:32 smithi110 ceph-mon[97633]: pgmap v12489: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T10:59:32.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:32 smithi104 ceph-mon[97044]: pgmap v12489: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T10:59:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:32.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:32.726162+0000) 2022-09-10T10:59:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:33.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:33.212149+0000) 2022-09-10T10:59:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:33.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:33.726411+0000) 2022-09-10T10:59:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:34 smithi104 ceph-mon[97044]: pgmap v12490: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:34.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:34.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:34.212363+0000) 2022-09-10T10:59:34.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:34 smithi110 ceph-mon[97633]: pgmap v12490: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:35.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:34.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:34.726630+0000) 2022-09-10T10:59:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:35.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:35.212538+0000) 2022-09-10T10:59:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:35.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:35.726897+0000) 2022-09-10T10:59:36.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:36.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:36.212696+0000) 2022-09-10T10:59:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:36 smithi110 ceph-mon[97633]: pgmap v12491: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:59:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:36 smithi104 ceph-mon[97044]: pgmap v12491: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:59:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:36.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:36.727114+0000) 2022-09-10T10:59:37.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:37.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:37.212941+0000) 2022-09-10T10:59:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:37.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:37.727303+0000) 2022-09-10T10:59:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:38.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:38.213138+0000) 2022-09-10T10:59:38.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:38 smithi110 ceph-mon[97633]: pgmap v12492: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:59:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:38 smithi104 ceph-mon[97044]: pgmap v12492: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:59:39.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:38.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:38.727523+0000) 2022-09-10T10:59:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:39.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:39.213396+0000) 2022-09-10T10:59:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:39.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:39.727749+0000) 2022-09-10T10:59:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:40.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:40.213606+0000) 2022-09-10T10:59:40.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:40 smithi110 ceph-mon[97633]: pgmap v12493: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T10:59:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:40 smithi104 ceph-mon[97044]: pgmap v12493: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T10:59:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:40.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:40.728006+0000) 2022-09-10T10:59:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:59:40.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:59:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:41.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:41.213769+0000) 2022-09-10T10:59:42.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:41.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:41.728274+0000) 2022-09-10T10:59:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:42.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:42.214010+0000) 2022-09-10T10:59:42.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:42 smithi110 ceph-mon[97633]: pgmap v12494: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T10:59:42.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:42 smithi104 ceph-mon[97044]: pgmap v12494: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T10:59:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:42.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:42.728516+0000) 2022-09-10T10:59:43.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:43.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:43.214205+0000) 2022-09-10T10:59:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:43.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:43.728736+0000) 2022-09-10T10:59:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:44 smithi104 ceph-mon[97044]: pgmap v12495: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:59:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:44.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:44.214447+0000) 2022-09-10T10:59:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:44 smithi110 ceph-mon[97633]: pgmap v12495: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T10:59:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:44.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:44.728978+0000) 2022-09-10T10:59:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:45.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:45.214659+0000) 2022-09-10T10:59:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:45.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:45.729165+0000) 2022-09-10T10:59:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:46.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:46.214866+0000) 2022-09-10T10:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:46 smithi110 ceph-mon[97633]: pgmap v12496: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:59:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:59:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:59:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:59:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:59:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:46 smithi104 ceph-mon[97044]: pgmap v12496: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:59:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:59:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T10:59:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:59:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T10:59:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:46.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:46.729388+0000) 2022-09-10T10:59:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:47.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:47.215124+0000) 2022-09-10T10:59:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:47.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:47.729615+0000) 2022-09-10T10:59:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:48.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:48.215345+0000) 2022-09-10T10:59:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:48 smithi110 ceph-mon[97633]: pgmap v12497: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:48 smithi104 ceph-mon[97044]: pgmap v12497: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:48.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:48.729847+0000) 2022-09-10T10:59:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:49.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:49.215482+0000) 2022-09-10T10:59:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:49.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:49.730116+0000) 2022-09-10T10:59:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:50.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:50.215735+0000) 2022-09-10T10:59:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:50 smithi110 ceph-mon[97633]: pgmap v12498: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T10:59:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:50 smithi104 ceph-mon[97044]: pgmap v12498: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T10:59:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:50.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:50.730310+0000) 2022-09-10T10:59:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:51.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:51.215984+0000) 2022-09-10T10:59:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:51.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:51.730518+0000) 2022-09-10T10:59:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:52.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:52.216202+0000) 2022-09-10T10:59:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:52 smithi110 ceph-mon[97633]: pgmap v12499: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T10:59:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:52 smithi104 ceph-mon[97044]: pgmap v12499: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T10:59:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:52.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:52.730761+0000) 2022-09-10T10:59:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:53.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:53.216396+0000) 2022-09-10T10:59:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:53.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:53.731037+0000) 2022-09-10T10:59:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:54 smithi104 ceph-mon[97044]: pgmap v12500: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:54.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:54.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:54.216656+0000) 2022-09-10T10:59:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:54 smithi110 ceph-mon[97633]: pgmap v12500: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:54.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:54.731300+0000) 2022-09-10T10:59:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:55.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:55.216840+0000) 2022-09-10T10:59:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 10:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T10:59:55.853+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T10:59:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:55.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:55.731556+0000) 2022-09-10T10:59:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:56.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:56.217064+0000) 2022-09-10T10:59:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:56 smithi110 ceph-mon[97633]: pgmap v12501: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:59:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:56 smithi104 ceph-mon[97044]: pgmap v12501: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T10:59:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:56.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:56.731762+0000) 2022-09-10T10:59:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:57.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:57.217310+0000) 2022-09-10T10:59:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:57.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:57.731981+0000) 2022-09-10T10:59:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:58.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:58.217571+0000) 2022-09-10T10:59:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 10:59:58 smithi110 ceph-mon[97633]: pgmap v12502: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 10:59:58 smithi104 ceph-mon[97044]: pgmap v12502: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T10:59:59.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:58.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:58.732188+0000) 2022-09-10T10:59:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 10:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T10:59:59.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:59.217826+0000) 2022-09-10T11:00:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 10:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T10:59:59.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T09:59:59.732338+0000) 2022-09-10T11:00:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:00.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:00.218000+0000) 2022-09-10T11:00:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:00 smithi110 ceph-mon[97633]: pgmap v12503: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:00:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:00:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:00 smithi104 ceph-mon[97044]: pgmap v12503: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:00:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:00:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:00.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:00.732455+0000) 2022-09-10T11:00:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:01.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:01.218219+0000) 2022-09-10T11:00:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:01.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:01.732632+0000) 2022-09-10T11:00:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:02.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:02.218487+0000) 2022-09-10T11:00:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:02 smithi110 ceph-mon[97633]: pgmap v12504: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:00:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:02 smithi104 ceph-mon[97044]: pgmap v12504: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:00:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:02.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:02.732859+0000) 2022-09-10T11:00:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:03.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:03.218715+0000) 2022-09-10T11:00:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:03.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:03.733024+0000) 2022-09-10T11:00:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:04 smithi104 ceph-mon[97044]: pgmap v12505: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:04.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:04.218932+0000) 2022-09-10T11:00:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:04 smithi110 ceph-mon[97633]: pgmap v12505: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:04.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:04.733260+0000) 2022-09-10T11:00:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:05.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:05.219216+0000) 2022-09-10T11:00:06.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:05.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:05.733447+0000) 2022-09-10T11:00:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:06.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:06.219446+0000) 2022-09-10T11:00:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:06 smithi110 ceph-mon[97633]: pgmap v12506: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:00:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:06 smithi104 ceph-mon[97044]: pgmap v12506: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:00:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:06.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:06.733601+0000) 2022-09-10T11:00:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:07.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:07.219633+0000) 2022-09-10T11:00:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:07.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:07.733827+0000) 2022-09-10T11:00:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:08.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:08.219896+0000) 2022-09-10T11:00:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:08 smithi110 ceph-mon[97633]: pgmap v12507: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:00:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:08 smithi104 ceph-mon[97044]: pgmap v12507: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:00:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:08.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:08.734072+0000) 2022-09-10T11:00:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:09.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:09.220174+0000) 2022-09-10T11:00:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:09 smithi110 ceph-mon[97633]: osdmap e862: 12 total, 12 up, 12 in 2022-09-10T11:00:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:09 smithi104 ceph-mon[97044]: osdmap e862: 12 total, 12 up, 12 in 2022-09-10T11:00:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:09.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:09.734270+0000) 2022-09-10T11:00:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:10.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:10.220345+0000) 2022-09-10T11:00:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:10 smithi110 ceph-mon[97633]: pgmap v12509: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:00:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:10 smithi104 ceph-mon[97044]: pgmap v12509: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:00:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:00:10.855+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:00:11.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:10.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:10.734493+0000) 2022-09-10T11:00:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:11.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:11.220565+0000) 2022-09-10T11:00:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:11.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:11.734740+0000) 2022-09-10T11:00:12.567 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:12.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:12.220775+0000) 2022-09-10T11:00:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:12 smithi110 ceph-mon[97633]: pgmap v12510: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:00:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:12 smithi104 ceph-mon[97044]: pgmap v12510: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:00:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:12.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:12.734985+0000) 2022-09-10T11:00:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:13.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:13.221036+0000) 2022-09-10T11:00:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:13.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:13.735242+0000) 2022-09-10T11:00:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:14 smithi104 ceph-mon[97044]: osdmap e863: 12 total, 12 up, 12 in 2022-09-10T11:00:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:14 smithi104 ceph-mon[97044]: pgmap v12512: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:00:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:14.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:14.221156+0000) 2022-09-10T11:00:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:14 smithi110 ceph-mon[97633]: osdmap e863: 12 total, 12 up, 12 in 2022-09-10T11:00:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:14 smithi110 ceph-mon[97633]: pgmap v12512: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:00:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:14.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:14.735433+0000) 2022-09-10T11:00:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:15.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:15.221319+0000) 2022-09-10T11:00:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:15.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:15.735711+0000) 2022-09-10T11:00:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:16.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:16.221523+0000) 2022-09-10T11:00:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:16 smithi110 ceph-mon[97633]: pgmap v12513: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:00:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:16 smithi104 ceph-mon[97044]: pgmap v12513: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:00:17.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:16.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:16.735968+0000) 2022-09-10T11:00:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:17.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:17.221720+0000) 2022-09-10T11:00:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:17.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:17.736239+0000) 2022-09-10T11:00:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:18.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:18.221975+0000) 2022-09-10T11:00:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:18 smithi110 ceph-mon[97633]: pgmap v12514: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T11:00:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:18 smithi104 ceph-mon[97044]: pgmap v12514: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T11:00:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:18.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:18.736489+0000) 2022-09-10T11:00:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:19.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:19.222216+0000) 2022-09-10T11:00:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:19.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:19.736679+0000) 2022-09-10T11:00:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:20.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:20.222433+0000) 2022-09-10T11:00:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:20 smithi110 ceph-mon[97633]: pgmap v12515: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:00:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:20 smithi104 ceph-mon[97044]: pgmap v12515: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:00:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:20.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:20.736870+0000) 2022-09-10T11:00:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:21.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:21.222600+0000) 2022-09-10T11:00:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:21.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:21.737085+0000) 2022-09-10T11:00:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:22.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:22.222778+0000) 2022-09-10T11:00:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:22 smithi110 ceph-mon[97633]: pgmap v12516: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:00:22.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:22 smithi104 ceph-mon[97044]: pgmap v12516: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:00:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:22.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:22.737362+0000) 2022-09-10T11:00:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:23.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:23.223000+0000) 2022-09-10T11:00:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:23.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:23.737551+0000) 2022-09-10T11:00:24.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:24 smithi104 ceph-mon[97044]: pgmap v12517: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:00:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:24.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:24.223220+0000) 2022-09-10T11:00:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:24 smithi110 ceph-mon[97633]: pgmap v12517: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:00:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:24.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:24.737774+0000) 2022-09-10T11:00:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:25.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:25.223402+0000) 2022-09-10T11:00:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:00:25.869+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:00:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:25.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:25.738003+0000) 2022-09-10T11:00:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:26.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:26.223581+0000) 2022-09-10T11:00:26.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:26 smithi110 ceph-mon[97633]: pgmap v12518: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:00:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:26 smithi104 ceph-mon[97044]: pgmap v12518: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:00:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:26.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:26.738263+0000) 2022-09-10T11:00:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:27.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:27.223728+0000) 2022-09-10T11:00:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:27.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:27.738473+0000) 2022-09-10T11:00:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:28.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:28.223950+0000) 2022-09-10T11:00:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:28 smithi110 ceph-mon[97633]: pgmap v12519: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:00:28.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:28 smithi104 ceph-mon[97044]: pgmap v12519: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:00:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:28.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:28.738712+0000) 2022-09-10T11:00:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:29.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:29.224155+0000) 2022-09-10T11:00:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:29.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:29.738977+0000) 2022-09-10T11:00:30.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:30.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:30.224367+0000) 2022-09-10T11:00:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:30 smithi110 ceph-mon[97633]: pgmap v12520: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T11:00:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:30 smithi104 ceph-mon[97044]: pgmap v12520: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T11:00:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:30.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:30.739174+0000) 2022-09-10T11:00:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:31.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:31.224582+0000) 2022-09-10T11:00:32.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:31.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:31.739408+0000) 2022-09-10T11:00:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:32.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:32.224800+0000) 2022-09-10T11:00:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:32 smithi110 ceph-mon[97633]: pgmap v12521: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T11:00:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:32 smithi104 ceph-mon[97044]: pgmap v12521: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T11:00:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:32.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:32.739610+0000) 2022-09-10T11:00:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:33.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:33.225028+0000) 2022-09-10T11:00:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:33.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:33.739821+0000) 2022-09-10T11:00:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:34 smithi104 ceph-mon[97044]: pgmap v12522: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:34.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:34.225295+0000) 2022-09-10T11:00:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:34 smithi110 ceph-mon[97633]: pgmap v12522: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:34.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:34.740020+0000) 2022-09-10T11:00:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:35.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:35.225541+0000) 2022-09-10T11:00:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:35.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:35.740287+0000) 2022-09-10T11:00:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:36.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:36.225759+0000) 2022-09-10T11:00:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:36 smithi110 ceph-mon[97633]: pgmap v12523: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:00:36.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:36 smithi104 ceph-mon[97044]: pgmap v12523: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:00:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:36.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:36.740459+0000) 2022-09-10T11:00:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:37.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:37.225913+0000) 2022-09-10T11:00:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:37.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:37.740652+0000) 2022-09-10T11:00:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:38.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:38.226052+0000) 2022-09-10T11:00:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:38 smithi110 ceph-mon[97633]: pgmap v12524: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:38 smithi104 ceph-mon[97044]: pgmap v12524: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:39.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:38.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:38.740844+0000) 2022-09-10T11:00:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:39.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:39.226202+0000) 2022-09-10T11:00:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:39.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:39.741077+0000) 2022-09-10T11:00:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:40.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:40.226430+0000) 2022-09-10T11:00:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:40 smithi110 ceph-mon[97633]: pgmap v12525: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:00:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:40 smithi104 ceph-mon[97044]: pgmap v12525: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:00:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:00:40.875+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:00:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:40.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:40.741311+0000) 2022-09-10T11:00:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:41.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:41.226606+0000) 2022-09-10T11:00:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:41.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:41.741482+0000) 2022-09-10T11:00:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:42.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:42.226765+0000) 2022-09-10T11:00:42.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:42 smithi110 ceph-mon[97633]: pgmap v12526: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:00:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:42 smithi104 ceph-mon[97044]: pgmap v12526: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:00:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:42.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:42.741730+0000) 2022-09-10T11:00:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:43.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:43.226897+0000) 2022-09-10T11:00:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:43.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:43.742012+0000) 2022-09-10T11:00:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:44 smithi104 ceph-mon[97044]: pgmap v12527: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:44.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:44.227134+0000) 2022-09-10T11:00:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:44 smithi110 ceph-mon[97633]: pgmap v12527: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:45.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:44.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:44.742174+0000) 2022-09-10T11:00:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:45.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:45.227343+0000) 2022-09-10T11:00:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:45.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:45.742388+0000) 2022-09-10T11:00:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:46.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:46.227546+0000) 2022-09-10T11:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:46 smithi110 ceph-mon[97633]: pgmap v12528: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:00:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:00:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:00:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:00:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:46 smithi104 ceph-mon[97044]: pgmap v12528: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:00:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:00:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:00:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:00:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:00:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:46.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:46.742616+0000) 2022-09-10T11:00:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:47.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:47.227789+0000) 2022-09-10T11:00:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:47.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:47.742794+0000) 2022-09-10T11:00:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:48.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:48.227967+0000) 2022-09-10T11:00:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:48 smithi110 ceph-mon[97633]: pgmap v12529: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:48 smithi104 ceph-mon[97044]: pgmap v12529: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:48.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:48.742953+0000) 2022-09-10T11:00:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:49.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:49.228215+0000) 2022-09-10T11:00:50.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:49.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:49.743203+0000) 2022-09-10T11:00:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:50.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:50.228355+0000) 2022-09-10T11:00:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:50 smithi110 ceph-mon[97633]: pgmap v12530: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:00:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:50 smithi104 ceph-mon[97044]: pgmap v12530: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:00:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:50.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:50.743421+0000) 2022-09-10T11:00:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:51.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:51.228598+0000) 2022-09-10T11:00:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:51.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:51.743678+0000) 2022-09-10T11:00:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:52.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:52.228838+0000) 2022-09-10T11:00:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:52 smithi110 ceph-mon[97633]: pgmap v12531: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:00:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:52 smithi104 ceph-mon[97044]: pgmap v12531: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:00:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:52.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:52.743918+0000) 2022-09-10T11:00:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:53.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:53.229036+0000) 2022-09-10T11:00:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:53.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:53.744125+0000) 2022-09-10T11:00:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:54 smithi104 ceph-mon[97044]: pgmap v12532: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:54.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:54.229311+0000) 2022-09-10T11:00:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:54 smithi110 ceph-mon[97633]: pgmap v12532: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:54.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:54.744327+0000) 2022-09-10T11:00:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:55.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:55.229546+0000) 2022-09-10T11:00:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:55.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:55.744558+0000) 2022-09-10T11:00:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:00:55.877+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:00:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:56.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:56.229725+0000) 2022-09-10T11:00:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:56 smithi110 ceph-mon[97633]: pgmap v12533: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:00:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:56 smithi104 ceph-mon[97044]: pgmap v12533: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:00:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:56.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:56.744731+0000) 2022-09-10T11:00:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:57.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:57.229951+0000) 2022-09-10T11:00:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:57.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:57.744932+0000) 2022-09-10T11:00:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:58.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:58.230211+0000) 2022-09-10T11:00:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:00:58 smithi110 ceph-mon[97633]: pgmap v12534: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:00:58 smithi104 ceph-mon[97044]: pgmap v12534: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:00:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:58.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:58.745199+0000) 2022-09-10T11:00:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:00:59.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:59.230451+0000) 2022-09-10T11:01:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:00:59.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:00:59.745438+0000) 2022-09-10T11:01:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:00.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:00.230618+0000) 2022-09-10T11:01:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:00 smithi110 ceph-mon[97633]: pgmap v12535: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:00 smithi104 ceph-mon[97044]: pgmap v12535: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:01.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:00.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:00.745643+0000) 2022-09-10T11:01:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:01.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:01.230834+0000) 2022-09-10T11:01:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:01.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:01.745801+0000) 2022-09-10T11:01:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:02.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:02.231057+0000) 2022-09-10T11:01:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:02 smithi110 ceph-mon[97633]: pgmap v12536: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:02 smithi104 ceph-mon[97044]: pgmap v12536: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:03.062 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:02.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:02.745960+0000) 2022-09-10T11:01:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:03.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:03.231307+0000) 2022-09-10T11:01:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:03.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:03.746229+0000) 2022-09-10T11:01:04.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:04 smithi104 ceph-mon[97044]: pgmap v12537: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:04.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:04.231535+0000) 2022-09-10T11:01:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:04 smithi110 ceph-mon[97633]: pgmap v12537: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:04.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:04.746427+0000) 2022-09-10T11:01:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:05.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:05.231790+0000) 2022-09-10T11:01:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:05.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:05.746635+0000) 2022-09-10T11:01:06.564 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:06.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:06.231969+0000) 2022-09-10T11:01:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:06 smithi110 ceph-mon[97633]: pgmap v12538: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:01:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:06 smithi104 ceph-mon[97044]: pgmap v12538: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:01:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:06.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:06.746876+0000) 2022-09-10T11:01:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:07.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:07.232176+0000) 2022-09-10T11:01:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:07.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:07.747154+0000) 2022-09-10T11:01:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:08.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:08.232404+0000) 2022-09-10T11:01:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:08 smithi110 ceph-mon[97633]: pgmap v12539: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:01:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:08 smithi104 ceph-mon[97044]: pgmap v12539: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:01:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:08.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:08.747421+0000) 2022-09-10T11:01:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:09.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:09.232598+0000) 2022-09-10T11:01:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:09 smithi110 ceph-mon[97633]: osdmap e864: 12 total, 12 up, 12 in 2022-09-10T11:01:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:09 smithi104 ceph-mon[97044]: osdmap e864: 12 total, 12 up, 12 in 2022-09-10T11:01:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:09.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:09.747660+0000) 2022-09-10T11:01:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:10.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:10.232810+0000) 2022-09-10T11:01:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:10 smithi110 ceph-mon[97633]: pgmap v12541: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:01:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:10 smithi104 ceph-mon[97044]: pgmap v12541: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:01:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:10.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:10.747897+0000) 2022-09-10T11:01:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:01:10.884+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:01:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:11.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:11.232994+0000) 2022-09-10T11:01:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:11.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:11.748042+0000) 2022-09-10T11:01:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:12.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:12.233166+0000) 2022-09-10T11:01:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:12 smithi110 ceph-mon[97633]: pgmap v12542: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:01:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:12 smithi104 ceph-mon[97044]: pgmap v12542: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:01:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:12.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:12.748203+0000) 2022-09-10T11:01:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:13.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:13.233341+0000) 2022-09-10T11:01:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:13.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:13.748361+0000) 2022-09-10T11:01:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:14 smithi104 ceph-mon[97044]: osdmap e865: 12 total, 12 up, 12 in 2022-09-10T11:01:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:14 smithi104 ceph-mon[97044]: pgmap v12544: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:01:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:14.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:14.233472+0000) 2022-09-10T11:01:14.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:14 smithi110 ceph-mon[97633]: osdmap e865: 12 total, 12 up, 12 in 2022-09-10T11:01:14.668 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:14 smithi110 ceph-mon[97633]: pgmap v12544: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:01:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:14.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:14.748526+0000) 2022-09-10T11:01:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:15.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:15.233670+0000) 2022-09-10T11:01:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:15.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:15.748782+0000) 2022-09-10T11:01:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:16.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:16.233872+0000) 2022-09-10T11:01:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:16 smithi110 ceph-mon[97633]: pgmap v12545: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:01:16.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:16 smithi104 ceph-mon[97044]: pgmap v12545: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:01:17.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:16.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:16.749000+0000) 2022-09-10T11:01:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:17.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:17.234061+0000) 2022-09-10T11:01:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:17.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:17.749248+0000) 2022-09-10T11:01:18.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:18.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:18.234260+0000) 2022-09-10T11:01:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:18 smithi110 ceph-mon[97633]: pgmap v12546: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:01:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:18 smithi104 ceph-mon[97044]: pgmap v12546: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:01:19.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:18.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:18.749490+0000) 2022-09-10T11:01:19.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:19.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:19.234442+0000) 2022-09-10T11:01:20.065 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:19.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:19.749698+0000) 2022-09-10T11:01:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:20.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:20.234677+0000) 2022-09-10T11:01:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:20 smithi110 ceph-mon[97633]: pgmap v12547: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:01:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:20 smithi104 ceph-mon[97044]: pgmap v12547: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:01:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:20.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:20.749910+0000) 2022-09-10T11:01:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:21.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:21.234880+0000) 2022-09-10T11:01:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:21.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:21.750182+0000) 2022-09-10T11:01:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:22.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:22.235063+0000) 2022-09-10T11:01:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:22 smithi110 ceph-mon[97633]: pgmap v12548: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T11:01:22.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:22 smithi104 ceph-mon[97044]: pgmap v12548: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T11:01:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:22.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:22.750323+0000) 2022-09-10T11:01:23.564 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:23.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:23.235217+0000) 2022-09-10T11:01:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:23.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:23.750509+0000) 2022-09-10T11:01:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:24.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:24.235397+0000) 2022-09-10T11:01:24.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:24 smithi104 ceph-mon[97044]: pgmap v12549: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T11:01:24.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:24 smithi110 ceph-mon[97633]: pgmap v12549: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T11:01:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:24.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:24.750679+0000) 2022-09-10T11:01:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:25.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:25.235640+0000) 2022-09-10T11:01:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:25.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:25.750932+0000) 2022-09-10T11:01:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:01:25.894+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:01:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:26.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:26.235834+0000) 2022-09-10T11:01:26.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:26 smithi110 ceph-mon[97633]: pgmap v12550: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:01:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:26 smithi104 ceph-mon[97044]: pgmap v12550: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:01:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:26.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:26.751159+0000) 2022-09-10T11:01:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:27.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:27.236023+0000) 2022-09-10T11:01:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:27.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:27.751351+0000) 2022-09-10T11:01:28.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:28.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:28.236155+0000) 2022-09-10T11:01:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:28 smithi110 ceph-mon[97633]: pgmap v12551: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:28.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:28 smithi104 ceph-mon[97044]: pgmap v12551: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:28.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:28.751524+0000) 2022-09-10T11:01:29.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:29.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:29.236314+0000) 2022-09-10T11:01:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:29.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:29.751755+0000) 2022-09-10T11:01:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:30.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:30.236495+0000) 2022-09-10T11:01:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:30 smithi110 ceph-mon[97633]: pgmap v12552: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:30 smithi104 ceph-mon[97044]: pgmap v12552: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:30.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:30.752010+0000) 2022-09-10T11:01:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:31.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:31.236730+0000) 2022-09-10T11:01:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:31.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:31.752154+0000) 2022-09-10T11:01:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:32.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:32.236900+0000) 2022-09-10T11:01:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:32 smithi110 ceph-mon[97633]: pgmap v12553: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:32 smithi104 ceph-mon[97044]: pgmap v12553: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:32.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:32.752341+0000) 2022-09-10T11:01:33.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:33.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:33.237140+0000) 2022-09-10T11:01:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:33.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:33.752523+0000) 2022-09-10T11:01:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:34 smithi104 ceph-mon[97044]: pgmap v12554: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:34.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:34.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:34.237303+0000) 2022-09-10T11:01:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:34 smithi110 ceph-mon[97633]: pgmap v12554: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:35.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:34.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:34.752695+0000) 2022-09-10T11:01:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:35.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:35.237531+0000) 2022-09-10T11:01:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:35.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:35.752923+0000) 2022-09-10T11:01:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:36.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:36.237768+0000) 2022-09-10T11:01:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:36 smithi110 ceph-mon[97633]: pgmap v12555: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:01:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:36 smithi104 ceph-mon[97044]: pgmap v12555: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:01:37.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:36.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:36.753165+0000) 2022-09-10T11:01:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:37.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:37.237955+0000) 2022-09-10T11:01:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:37.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:37.753392+0000) 2022-09-10T11:01:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:38.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:38.238218+0000) 2022-09-10T11:01:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:38 smithi110 ceph-mon[97633]: pgmap v12556: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:38 smithi104 ceph-mon[97044]: pgmap v12556: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:38.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:38.753656+0000) 2022-09-10T11:01:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:39.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:39.238443+0000) 2022-09-10T11:01:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:39.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:39.753874+0000) 2022-09-10T11:01:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:40.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:40.238652+0000) 2022-09-10T11:01:40.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:40 smithi110 ceph-mon[97633]: pgmap v12557: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:40.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:40 smithi104 ceph-mon[97044]: pgmap v12557: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:40.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:40.754142+0000) 2022-09-10T11:01:41.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:01:40.910+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:01:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:41.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:41.238837+0000) 2022-09-10T11:01:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:41.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:41.754323+0000) 2022-09-10T11:01:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:42.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:42.239076+0000) 2022-09-10T11:01:42.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:42 smithi110 ceph-mon[97633]: pgmap v12558: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:42 smithi104 ceph-mon[97044]: pgmap v12558: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:42.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:42.754525+0000) 2022-09-10T11:01:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:43.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:43.239317+0000) 2022-09-10T11:01:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:43.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:43.754752+0000) 2022-09-10T11:01:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:44 smithi104 ceph-mon[97044]: pgmap v12559: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:44.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:44.239497+0000) 2022-09-10T11:01:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:44 smithi110 ceph-mon[97633]: pgmap v12559: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:44.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:44.754979+0000) 2022-09-10T11:01:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:45.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:45.239683+0000) 2022-09-10T11:01:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:45.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:45.755229+0000) 2022-09-10T11:01:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:46.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:46.239930+0000) 2022-09-10T11:01:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:46 smithi110 ceph-mon[97633]: pgmap v12560: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:01:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:01:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:01:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:01:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:01:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:46 smithi104 ceph-mon[97044]: pgmap v12560: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:01:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:01:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:01:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:01:46.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:01:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:46.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:46.755452+0000) 2022-09-10T11:01:47.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:47.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:47.240171+0000) 2022-09-10T11:01:48.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:47.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:47.755687+0000) 2022-09-10T11:01:48.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:48.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:48.240417+0000) 2022-09-10T11:01:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:48 smithi110 ceph-mon[97633]: pgmap v12561: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:48.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:48 smithi104 ceph-mon[97044]: pgmap v12561: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:49.062 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:48.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:48.755956+0000) 2022-09-10T11:01:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:49.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:49.240663+0000) 2022-09-10T11:01:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:49.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:49.756210+0000) 2022-09-10T11:01:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:50.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:50.240934+0000) 2022-09-10T11:01:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:50 smithi110 ceph-mon[97633]: pgmap v12562: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:50.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:50 smithi104 ceph-mon[97044]: pgmap v12562: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:01:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:50.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:50.756422+0000) 2022-09-10T11:01:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:51.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:51.241197+0000) 2022-09-10T11:01:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:51.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:51.756583+0000) 2022-09-10T11:01:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:52.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:52.241405+0000) 2022-09-10T11:01:52.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:52 smithi110 ceph-mon[97633]: pgmap v12563: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:52 smithi104 ceph-mon[97044]: pgmap v12563: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:01:53.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:52.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:52.756797+0000) 2022-09-10T11:01:53.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:53.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:53.241573+0000) 2022-09-10T11:01:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:53.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:53.757040+0000) 2022-09-10T11:01:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:54 smithi104 ceph-mon[97044]: pgmap v12564: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:54.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:54.241771+0000) 2022-09-10T11:01:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:54 smithi110 ceph-mon[97633]: pgmap v12564: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:54.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:54.757288+0000) 2022-09-10T11:01:55.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:55.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:55.241920+0000) 2022-09-10T11:01:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:55.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:55.757469+0000) 2022-09-10T11:01:56.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:01:55.918+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:01:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:56.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:56.242153+0000) 2022-09-10T11:01:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:56 smithi110 ceph-mon[97633]: pgmap v12565: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:01:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:56 smithi104 ceph-mon[97044]: pgmap v12565: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:01:57.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:56.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:56.757714+0000) 2022-09-10T11:01:57.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:57.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:57.242355+0000) 2022-09-10T11:01:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:57.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:57.757952+0000) 2022-09-10T11:01:58.566 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:58.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:58.242583+0000) 2022-09-10T11:01:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:01:58 smithi110 ceph-mon[97633]: pgmap v12566: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:58.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:01:58 smithi104 ceph-mon[97044]: pgmap v12566: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:01:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:58.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:58.758205+0000) 2022-09-10T11:01:59.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:01:59.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:59.242822+0000) 2022-09-10T11:02:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:01:59.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:01:59.758346+0000) 2022-09-10T11:02:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:00.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:00.242993+0000) 2022-09-10T11:02:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:00 smithi110 ceph-mon[97633]: pgmap v12567: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:02:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:00 smithi104 ceph-mon[97044]: pgmap v12567: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:02:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:00.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:00.758525+0000) 2022-09-10T11:02:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:01.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:01.243178+0000) 2022-09-10T11:02:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:01.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:01.758700+0000) 2022-09-10T11:02:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:02.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:02.243408+0000) 2022-09-10T11:02:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:02 smithi110 ceph-mon[97633]: pgmap v12568: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:02:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:02 smithi104 ceph-mon[97044]: pgmap v12568: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:02:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:02.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:02.758942+0000) 2022-09-10T11:02:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:03.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:03.243602+0000) 2022-09-10T11:02:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:03.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:03.759163+0000) 2022-09-10T11:02:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:04 smithi104 ceph-mon[97044]: pgmap v12569: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:04.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:04.243757+0000) 2022-09-10T11:02:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:04 smithi110 ceph-mon[97633]: pgmap v12569: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:04.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:04.759410+0000) 2022-09-10T11:02:05.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:05.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:05.244011+0000) 2022-09-10T11:02:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:05.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:05.759627+0000) 2022-09-10T11:02:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:06.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:06.244210+0000) 2022-09-10T11:02:06.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:06 smithi110 ceph-mon[97633]: pgmap v12570: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:02:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:06 smithi104 ceph-mon[97044]: pgmap v12570: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:02:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:06.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:06.759795+0000) 2022-09-10T11:02:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:07.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:07.244382+0000) 2022-09-10T11:02:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:07.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:07.760039+0000) 2022-09-10T11:02:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:08.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:08.244590+0000) 2022-09-10T11:02:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:08 smithi110 ceph-mon[97633]: pgmap v12571: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:02:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:08 smithi104 ceph-mon[97044]: pgmap v12571: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:02:09.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:08.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:08.760280+0000) 2022-09-10T11:02:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:09.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:09.244756+0000) 2022-09-10T11:02:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:09 smithi110 ceph-mon[97633]: osdmap e866: 12 total, 12 up, 12 in 2022-09-10T11:02:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:09 smithi104 ceph-mon[97044]: osdmap e866: 12 total, 12 up, 12 in 2022-09-10T11:02:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:09.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:09.760522+0000) 2022-09-10T11:02:10.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:10.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:10.244953+0000) 2022-09-10T11:02:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:10 smithi110 ceph-mon[97633]: pgmap v12573: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:02:10.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:10 smithi104 ceph-mon[97044]: pgmap v12573: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:02:11.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:02:10.927+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:02:11.064 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:10.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:10.760797+0000) 2022-09-10T11:02:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:11.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:11.245212+0000) 2022-09-10T11:02:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:11.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:11.761018+0000) 2022-09-10T11:02:12.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:12.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:12.245408+0000) 2022-09-10T11:02:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:12 smithi110 ceph-mon[97633]: pgmap v12574: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:02:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:12 smithi104 ceph-mon[97044]: pgmap v12574: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:02:13.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:12.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:12.761214+0000) 2022-09-10T11:02:13.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:13.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:13.245642+0000) 2022-09-10T11:02:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:13.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:13.761447+0000) 2022-09-10T11:02:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:14 smithi104 ceph-mon[97044]: osdmap e867: 12 total, 12 up, 12 in 2022-09-10T11:02:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:14 smithi104 ceph-mon[97044]: pgmap v12576: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:02:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:14.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:14.245870+0000) 2022-09-10T11:02:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:14 smithi110 ceph-mon[97633]: osdmap e867: 12 total, 12 up, 12 in 2022-09-10T11:02:14.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:14 smithi110 ceph-mon[97633]: pgmap v12576: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:02:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:14.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:14.761669+0000) 2022-09-10T11:02:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:15.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:15.246127+0000) 2022-09-10T11:02:16.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:15.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:15.761860+0000) 2022-09-10T11:02:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:16.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:16.246347+0000) 2022-09-10T11:02:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:16 smithi110 ceph-mon[97633]: pgmap v12577: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:02:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:16 smithi104 ceph-mon[97044]: pgmap v12577: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:02:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:16.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:16.762160+0000) 2022-09-10T11:02:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:17.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:17.246526+0000) 2022-09-10T11:02:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:17.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:17.762356+0000) 2022-09-10T11:02:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:18.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:18.246777+0000) 2022-09-10T11:02:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:18 smithi110 ceph-mon[97633]: pgmap v12578: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:02:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:18 smithi104 ceph-mon[97044]: pgmap v12578: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:02:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:18.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:18.762582+0000) 2022-09-10T11:02:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:19.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:19.246985+0000) 2022-09-10T11:02:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:19.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:19.762837+0000) 2022-09-10T11:02:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:20.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:20.247187+0000) 2022-09-10T11:02:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:20 smithi110 ceph-mon[97633]: pgmap v12579: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:02:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:20 smithi104 ceph-mon[97044]: pgmap v12579: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:02:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:20.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:20.763070+0000) 2022-09-10T11:02:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:21.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:21.247428+0000) 2022-09-10T11:02:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:21.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:21.763346+0000) 2022-09-10T11:02:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:22.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:22.247642+0000) 2022-09-10T11:02:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:22 smithi110 ceph-mon[97633]: pgmap v12580: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:02:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:22 smithi104 ceph-mon[97044]: pgmap v12580: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:02:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:22.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:22.763513+0000) 2022-09-10T11:02:23.563 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:23.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:23.247862+0000) 2022-09-10T11:02:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:23.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:23.763743+0000) 2022-09-10T11:02:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:24 smithi104 ceph-mon[97044]: pgmap v12581: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:02:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:24.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:24.248066+0000) 2022-09-10T11:02:24.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:24 smithi110 ceph-mon[97633]: pgmap v12581: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:02:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:24.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:24.763948+0000) 2022-09-10T11:02:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:25.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:25.248270+0000) 2022-09-10T11:02:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:25.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:25.764160+0000) 2022-09-10T11:02:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:02:25.940+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:02:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:26.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:26.248487+0000) 2022-09-10T11:02:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:26 smithi110 ceph-mon[97633]: pgmap v12582: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:02:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:26 smithi104 ceph-mon[97044]: pgmap v12582: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:02:27.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:26.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:26.764336+0000) 2022-09-10T11:02:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:27.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:27.248756+0000) 2022-09-10T11:02:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:27.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:27.764511+0000) 2022-09-10T11:02:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:28.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:28.249041+0000) 2022-09-10T11:02:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:28 smithi110 ceph-mon[97633]: pgmap v12583: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:02:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:28 smithi104 ceph-mon[97044]: pgmap v12583: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:02:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:28.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:28.764646+0000) 2022-09-10T11:02:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:29.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:29.249279+0000) 2022-09-10T11:02:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:29.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:29.764824+0000) 2022-09-10T11:02:30.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:30.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:30.249469+0000) 2022-09-10T11:02:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:30 smithi110 ceph-mon[97633]: pgmap v12584: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T11:02:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:30 smithi104 ceph-mon[97044]: pgmap v12584: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T11:02:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:30.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:30.765073+0000) 2022-09-10T11:02:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:31.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:31.249721+0000) 2022-09-10T11:02:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:31.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:31.765264+0000) 2022-09-10T11:02:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:32.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:32.249936+0000) 2022-09-10T11:02:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:32 smithi110 ceph-mon[97633]: pgmap v12585: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T11:02:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:32 smithi104 ceph-mon[97044]: pgmap v12585: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T11:02:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:32.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:32.765428+0000) 2022-09-10T11:02:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:33.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:33.250216+0000) 2022-09-10T11:02:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:33.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:33.765688+0000) 2022-09-10T11:02:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:34 smithi104 ceph-mon[97044]: pgmap v12586: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:34.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:34.250477+0000) 2022-09-10T11:02:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:34 smithi110 ceph-mon[97633]: pgmap v12586: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:34.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:34.765922+0000) 2022-09-10T11:02:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:35.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:35.250740+0000) 2022-09-10T11:02:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:35.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:35.766210+0000) 2022-09-10T11:02:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:36.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:36.250988+0000) 2022-09-10T11:02:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:36 smithi110 ceph-mon[97633]: pgmap v12587: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:02:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:36 smithi104 ceph-mon[97044]: pgmap v12587: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:02:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:36.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:36.766435+0000) 2022-09-10T11:02:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:37.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:37.251206+0000) 2022-09-10T11:02:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:37.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:37.766683+0000) 2022-09-10T11:02:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:38.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:38.251445+0000) 2022-09-10T11:02:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:38 smithi110 ceph-mon[97633]: pgmap v12588: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:02:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:38 smithi104 ceph-mon[97044]: pgmap v12588: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:02:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:38.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:38.766961+0000) 2022-09-10T11:02:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:39.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:39.251636+0000) 2022-09-10T11:02:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:39.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:39.767194+0000) 2022-09-10T11:02:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:40.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:40.251863+0000) 2022-09-10T11:02:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:40 smithi110 ceph-mon[97633]: pgmap v12589: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T11:02:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:40 smithi104 ceph-mon[97044]: pgmap v12589: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2022-09-10T11:02:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:40.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:40.767422+0000) 2022-09-10T11:02:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:02:40.943+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:02:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:41.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:41.252067+0000) 2022-09-10T11:02:42.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:41.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:41.767645+0000) 2022-09-10T11:02:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:42.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:42.252314+0000) 2022-09-10T11:02:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:42 smithi110 ceph-mon[97633]: pgmap v12590: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T11:02:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:42 smithi104 ceph-mon[97044]: pgmap v12590: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T11:02:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:42.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:42.767846+0000) 2022-09-10T11:02:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:43.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:43.252549+0000) 2022-09-10T11:02:44.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:43.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:43.768046+0000) 2022-09-10T11:02:44.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:44 smithi104 ceph-mon[97044]: pgmap v12591: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:02:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:44.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:44.252819+0000) 2022-09-10T11:02:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:44 smithi110 ceph-mon[97633]: pgmap v12591: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:02:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:44.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:44.768254+0000) 2022-09-10T11:02:45.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:45.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:45.253051+0000) 2022-09-10T11:02:46.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:45.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:45.768523+0000) 2022-09-10T11:02:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:46.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:46.253270+0000) 2022-09-10T11:02:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:46 smithi110 ceph-mon[97633]: pgmap v12592: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:02:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:02:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:02:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:02:46.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:02:46.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:46 smithi104 ceph-mon[97044]: pgmap v12592: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:02:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:02:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:02:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:02:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:02:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:46.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:46.768791+0000) 2022-09-10T11:02:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:47.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:47.253502+0000) 2022-09-10T11:02:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:47.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:47.768983+0000) 2022-09-10T11:02:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:48.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:48.253652+0000) 2022-09-10T11:02:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:48 smithi110 ceph-mon[97633]: pgmap v12593: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:48 smithi104 ceph-mon[97044]: pgmap v12593: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:48.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:48.769179+0000) 2022-09-10T11:02:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:49.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:49.253879+0000) 2022-09-10T11:02:50.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:49.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:49.769371+0000) 2022-09-10T11:02:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:50.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:50.254133+0000) 2022-09-10T11:02:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:50 smithi110 ceph-mon[97633]: pgmap v12594: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:02:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:50 smithi104 ceph-mon[97044]: pgmap v12594: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:02:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:50.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:50.769538+0000) 2022-09-10T11:02:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:51.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:51.254370+0000) 2022-09-10T11:02:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:51.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:51.769727+0000) 2022-09-10T11:02:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:52.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:52.254609+0000) 2022-09-10T11:02:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:52 smithi110 ceph-mon[97633]: pgmap v12595: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:02:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:52 smithi104 ceph-mon[97044]: pgmap v12595: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:02:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:52.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:52.769899+0000) 2022-09-10T11:02:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:53.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:53.254851+0000) 2022-09-10T11:02:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:53.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:53.770197+0000) 2022-09-10T11:02:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:54 smithi104 ceph-mon[97044]: pgmap v12596: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:54.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:54.255120+0000) 2022-09-10T11:02:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:54 smithi110 ceph-mon[97633]: pgmap v12596: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:54.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:54.770391+0000) 2022-09-10T11:02:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:55.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:55.255346+0000) 2022-09-10T11:02:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:55.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:55.770642+0000) 2022-09-10T11:02:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:02:55.949+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:02:56.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:56.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:56.255543+0000) 2022-09-10T11:02:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:56 smithi110 ceph-mon[97633]: pgmap v12597: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:02:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:56 smithi104 ceph-mon[97044]: pgmap v12597: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:02:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:56.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:56.770884+0000) 2022-09-10T11:02:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:57.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:57.255778+0000) 2022-09-10T11:02:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:57.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:57.771178+0000) 2022-09-10T11:02:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:58.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:58.255936+0000) 2022-09-10T11:02:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:02:58 smithi110 ceph-mon[97633]: pgmap v12598: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:02:58 smithi104 ceph-mon[97044]: pgmap v12598: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:02:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:58.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:58.771363+0000) 2022-09-10T11:02:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:02:59.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:59.256186+0000) 2022-09-10T11:03:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:02:59.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:02:59.771618+0000) 2022-09-10T11:03:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:00.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:00.256436+0000) 2022-09-10T11:03:00.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:00 smithi110 ceph-mon[97633]: pgmap v12599: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:00 smithi104 ceph-mon[97044]: pgmap v12599: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:00.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:00.771889+0000) 2022-09-10T11:03:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:01.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:01.256686+0000) 2022-09-10T11:03:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:01.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:01.772174+0000) 2022-09-10T11:03:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:02.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:02.256868+0000) 2022-09-10T11:03:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:02 smithi110 ceph-mon[97633]: pgmap v12600: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:02 smithi104 ceph-mon[97044]: pgmap v12600: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:02.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:02.772358+0000) 2022-09-10T11:03:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:03.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:03.257126+0000) 2022-09-10T11:03:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:03.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:03.772553+0000) 2022-09-10T11:03:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:04 smithi104 ceph-mon[97044]: pgmap v12601: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:04.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:04.257266+0000) 2022-09-10T11:03:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:04 smithi110 ceph-mon[97633]: pgmap v12601: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:05.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:04.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:04.772764+0000) 2022-09-10T11:03:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:05.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:05.257496+0000) 2022-09-10T11:03:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:05.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:05.772980+0000) 2022-09-10T11:03:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:06.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:06.257730+0000) 2022-09-10T11:03:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:06 smithi110 ceph-mon[97633]: pgmap v12602: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:03:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:06 smithi104 ceph-mon[97044]: pgmap v12602: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:03:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:06.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:06.773179+0000) 2022-09-10T11:03:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:07.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:07.257920+0000) 2022-09-10T11:03:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:07.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:07.773445+0000) 2022-09-10T11:03:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:08.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:08.258139+0000) 2022-09-10T11:03:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:08 smithi104 ceph-mon[97044]: pgmap v12603: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:03:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:08 smithi110 ceph-mon[97633]: pgmap v12603: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:03:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:08.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:08.773649+0000) 2022-09-10T11:03:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:09.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:09.258267+0000) 2022-09-10T11:03:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:09 smithi104 ceph-mon[97044]: osdmap e868: 12 total, 12 up, 12 in 2022-09-10T11:03:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:09 smithi110 ceph-mon[97633]: osdmap e868: 12 total, 12 up, 12 in 2022-09-10T11:03:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:09.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:09.773911+0000) 2022-09-10T11:03:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:10.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:10.258512+0000) 2022-09-10T11:03:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:10 smithi104 ceph-mon[97044]: pgmap v12605: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:03:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:10 smithi110 ceph-mon[97633]: pgmap v12605: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:03:11.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:03:10.950+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:03:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:10.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:10.774204+0000) 2022-09-10T11:03:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:11.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:11.258714+0000) 2022-09-10T11:03:12.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:11.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:11.774457+0000) 2022-09-10T11:03:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:12.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:12.258930+0000) 2022-09-10T11:03:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:12 smithi104 ceph-mon[97044]: pgmap v12606: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:03:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:12 smithi110 ceph-mon[97633]: pgmap v12606: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:03:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:12.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:12.774680+0000) 2022-09-10T11:03:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:13.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:13.259130+0000) 2022-09-10T11:03:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:13.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:13.774881+0000) 2022-09-10T11:03:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:14 smithi104 ceph-mon[97044]: osdmap e869: 12 total, 12 up, 12 in 2022-09-10T11:03:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:14 smithi104 ceph-mon[97044]: pgmap v12608: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:03:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:14.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:14.259348+0000) 2022-09-10T11:03:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:14 smithi110 ceph-mon[97633]: osdmap e869: 12 total, 12 up, 12 in 2022-09-10T11:03:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:14 smithi110 ceph-mon[97633]: pgmap v12608: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:03:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:14.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:14.775128+0000) 2022-09-10T11:03:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:15.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:15.259521+0000) 2022-09-10T11:03:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:15.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:15.775363+0000) 2022-09-10T11:03:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:16.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:16.259765+0000) 2022-09-10T11:03:16.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:16 smithi110 ceph-mon[97633]: pgmap v12609: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:03:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:16 smithi104 ceph-mon[97044]: pgmap v12609: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:03:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:16.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:16.775612+0000) 2022-09-10T11:03:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:17.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:17.259988+0000) 2022-09-10T11:03:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:17.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:17.775900+0000) 2022-09-10T11:03:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:18.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:18.260234+0000) 2022-09-10T11:03:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:18 smithi110 ceph-mon[97633]: pgmap v12610: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:03:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:18 smithi104 ceph-mon[97044]: pgmap v12610: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:03:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:18.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:18.776142+0000) 2022-09-10T11:03:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:19.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:19.260420+0000) 2022-09-10T11:03:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:19.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:19.776325+0000) 2022-09-10T11:03:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:20.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:20.260657+0000) 2022-09-10T11:03:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:20 smithi104 ceph-mon[97044]: pgmap v12611: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:03:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:20 smithi110 ceph-mon[97633]: pgmap v12611: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:03:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:20.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:20.776559+0000) 2022-09-10T11:03:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:21.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:21.260904+0000) 2022-09-10T11:03:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:21.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:21.776800+0000) 2022-09-10T11:03:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:22.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:22.261085+0000) 2022-09-10T11:03:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:22 smithi104 ceph-mon[97044]: pgmap v12612: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:03:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:22 smithi110 ceph-mon[97633]: pgmap v12612: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:03:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:22.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:22.777065+0000) 2022-09-10T11:03:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:23.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:23.261299+0000) 2022-09-10T11:03:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:23.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:23.777391+0000) 2022-09-10T11:03:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:24.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:24.261519+0000) 2022-09-10T11:03:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:24 smithi110 ceph-mon[97633]: pgmap v12613: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:03:24.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:24 smithi104 ceph-mon[97044]: pgmap v12613: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:03:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:24.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:24.777639+0000) 2022-09-10T11:03:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:25.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:25.261705+0000) 2022-09-10T11:03:26.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:03:25.964+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:03:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:25.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:25.777828+0000) 2022-09-10T11:03:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:26.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:26.261980+0000) 2022-09-10T11:03:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:26 smithi104 ceph-mon[97044]: pgmap v12614: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:03:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:26 smithi110 ceph-mon[97633]: pgmap v12614: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:03:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:26.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:26.778013+0000) 2022-09-10T11:03:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:27.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:27.262211+0000) 2022-09-10T11:03:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:27.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:27.778210+0000) 2022-09-10T11:03:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:28.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:28.262480+0000) 2022-09-10T11:03:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:28 smithi104 ceph-mon[97044]: pgmap v12615: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:28 smithi110 ceph-mon[97633]: pgmap v12615: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:28.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:28.778405+0000) 2022-09-10T11:03:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:29.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:29.262722+0000) 2022-09-10T11:03:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:29.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:29.778600+0000) 2022-09-10T11:03:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:30.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:30.262940+0000) 2022-09-10T11:03:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:30 smithi104 ceph-mon[97044]: pgmap v12616: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:30.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:30 smithi110 ceph-mon[97633]: pgmap v12616: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:30.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:30.778808+0000) 2022-09-10T11:03:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:31.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:31.263135+0000) 2022-09-10T11:03:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:31.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:31.778973+0000) 2022-09-10T11:03:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:32.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:32.263363+0000) 2022-09-10T11:03:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:32 smithi104 ceph-mon[97044]: pgmap v12617: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:32 smithi110 ceph-mon[97633]: pgmap v12617: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:32.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:32.779211+0000) 2022-09-10T11:03:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:33.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:33.263549+0000) 2022-09-10T11:03:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:33.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:33.779371+0000) 2022-09-10T11:03:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:34.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:34.263756+0000) 2022-09-10T11:03:34.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:34 smithi110 ceph-mon[97633]: pgmap v12618: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:34 smithi104 ceph-mon[97044]: pgmap v12618: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:34.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:34.779572+0000) 2022-09-10T11:03:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:35.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:35.263982+0000) 2022-09-10T11:03:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:35.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:35.779770+0000) 2022-09-10T11:03:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:36.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:36.264166+0000) 2022-09-10T11:03:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:36 smithi104 ceph-mon[97044]: pgmap v12619: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:03:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:36 smithi110 ceph-mon[97633]: pgmap v12619: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:03:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:36.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:36.779986+0000) 2022-09-10T11:03:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:37.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:37.264385+0000) 2022-09-10T11:03:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:37.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:37.780213+0000) 2022-09-10T11:03:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:38.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:38.264605+0000) 2022-09-10T11:03:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:38 smithi104 ceph-mon[97044]: pgmap v12620: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:38 smithi110 ceph-mon[97633]: pgmap v12620: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:38.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:38.780450+0000) 2022-09-10T11:03:39.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:39.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:39.264765+0000) 2022-09-10T11:03:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:39.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:39.780639+0000) 2022-09-10T11:03:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:40.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:40.264938+0000) 2022-09-10T11:03:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:40 smithi104 ceph-mon[97044]: pgmap v12621: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:40 smithi110 ceph-mon[97633]: pgmap v12621: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:40.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:40.780847+0000) 2022-09-10T11:03:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:03:40.967+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:03:41.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:41.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:41.265134+0000) 2022-09-10T11:03:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:41.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:41.781091+0000) 2022-09-10T11:03:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:42.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:42.265323+0000) 2022-09-10T11:03:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:42 smithi104 ceph-mon[97044]: pgmap v12622: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:42 smithi110 ceph-mon[97633]: pgmap v12622: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:42.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:42.781379+0000) 2022-09-10T11:03:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:43.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:43.265469+0000) 2022-09-10T11:03:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:43.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:43.781614+0000) 2022-09-10T11:03:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:44.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:44.265696+0000) 2022-09-10T11:03:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:44 smithi110 ceph-mon[97633]: pgmap v12623: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:44 smithi104 ceph-mon[97044]: pgmap v12623: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:44.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:44.781824+0000) 2022-09-10T11:03:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:45.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:45.265939+0000) 2022-09-10T11:03:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:45.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:45.782021+0000) 2022-09-10T11:03:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:46.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:46.266192+0000) 2022-09-10T11:03:46.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:46 smithi104 ceph-mon[97044]: pgmap v12624: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:03:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:03:46.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:03:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:03:46.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:03:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:46 smithi110 ceph-mon[97633]: pgmap v12624: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:03:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:03:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:03:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:03:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:03:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:46.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:46.782241+0000) 2022-09-10T11:03:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:47.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:47.266416+0000) 2022-09-10T11:03:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:47.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:47.782475+0000) 2022-09-10T11:03:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:48.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:48.266654+0000) 2022-09-10T11:03:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:48 smithi104 ceph-mon[97044]: pgmap v12625: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:48.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:48 smithi110 ceph-mon[97633]: pgmap v12625: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:48.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:48.782627+0000) 2022-09-10T11:03:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:49.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:49.266839+0000) 2022-09-10T11:03:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:49.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:49.782821+0000) 2022-09-10T11:03:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:50.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:50.267091+0000) 2022-09-10T11:03:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:50 smithi104 ceph-mon[97044]: pgmap v12626: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:50 smithi110 ceph-mon[97633]: pgmap v12626: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:03:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:50.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:50.783081+0000) 2022-09-10T11:03:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:51.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:51.267360+0000) 2022-09-10T11:03:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:51.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:51.783314+0000) 2022-09-10T11:03:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:52.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:52.267532+0000) 2022-09-10T11:03:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:52 smithi104 ceph-mon[97044]: pgmap v12627: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:52 smithi110 ceph-mon[97633]: pgmap v12627: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:03:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:52.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:52.783527+0000) 2022-09-10T11:03:53.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:53.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:53.267772+0000) 2022-09-10T11:03:54.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:53.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:53.783743+0000) 2022-09-10T11:03:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:54.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:54.267936+0000) 2022-09-10T11:03:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:54 smithi110 ceph-mon[97633]: pgmap v12628: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:54 smithi104 ceph-mon[97044]: pgmap v12628: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:54.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:54.783990+0000) 2022-09-10T11:03:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:55.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:55.268151+0000) 2022-09-10T11:03:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:03:55.983+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:03:56.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:55.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:55.784144+0000) 2022-09-10T11:03:56.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:56.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:56.268355+0000) 2022-09-10T11:03:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:56 smithi104 ceph-mon[97044]: pgmap v12629: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:03:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:56 smithi110 ceph-mon[97633]: pgmap v12629: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:03:57.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:56.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:56.784229+0000) 2022-09-10T11:03:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:57.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:57.268536+0000) 2022-09-10T11:03:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:57.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:57.784465+0000) 2022-09-10T11:03:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:58.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:58.268708+0000) 2022-09-10T11:03:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:03:58 smithi104 ceph-mon[97044]: pgmap v12630: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:03:58 smithi110 ceph-mon[97633]: pgmap v12630: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:03:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:58.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:58.784695+0000) 2022-09-10T11:03:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:03:59.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:59.268943+0000) 2022-09-10T11:04:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:03:59.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:03:59.784959+0000) 2022-09-10T11:04:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:00.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:00.269142+0000) 2022-09-10T11:04:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:00 smithi104 ceph-mon[97044]: pgmap v12631: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:00 smithi110 ceph-mon[97633]: pgmap v12631: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:00.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:00.785123+0000) 2022-09-10T11:04:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:01.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:01.269337+0000) 2022-09-10T11:04:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:01.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:01.785290+0000) 2022-09-10T11:04:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:02.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:02.269543+0000) 2022-09-10T11:04:02.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:02 smithi104 ceph-mon[97044]: pgmap v12632: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:02 smithi110 ceph-mon[97633]: pgmap v12632: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:02.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:02.785506+0000) 2022-09-10T11:04:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:03.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:03.269733+0000) 2022-09-10T11:04:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:03.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:03.785743+0000) 2022-09-10T11:04:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:04.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:04.269936+0000) 2022-09-10T11:04:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:04 smithi110 ceph-mon[97633]: pgmap v12633: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:04 smithi104 ceph-mon[97044]: pgmap v12633: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:04.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:04.786003+0000) 2022-09-10T11:04:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:05.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:05.270196+0000) 2022-09-10T11:04:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:05.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:05.786224+0000) 2022-09-10T11:04:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:06.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:06.270399+0000) 2022-09-10T11:04:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:06 smithi104 ceph-mon[97044]: pgmap v12634: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:04:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:06 smithi110 ceph-mon[97633]: pgmap v12634: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:04:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:06.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:06.786479+0000) 2022-09-10T11:04:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:07.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:07.270587+0000) 2022-09-10T11:04:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:07.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:07.786710+0000) 2022-09-10T11:04:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:08.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:08.270803+0000) 2022-09-10T11:04:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:08 smithi104 ceph-mon[97044]: pgmap v12635: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:04:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:08 smithi110 ceph-mon[97633]: pgmap v12635: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:04:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:08.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:08.786973+0000) 2022-09-10T11:04:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:09.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:09.270945+0000) 2022-09-10T11:04:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:09 smithi104 ceph-mon[97044]: osdmap e870: 12 total, 12 up, 12 in 2022-09-10T11:04:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:09 smithi110 ceph-mon[97633]: osdmap e870: 12 total, 12 up, 12 in 2022-09-10T11:04:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:09.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:09.787215+0000) 2022-09-10T11:04:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:10.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:10.271209+0000) 2022-09-10T11:04:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:10 smithi104 ceph-mon[97044]: pgmap v12637: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:04:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:10 smithi110 ceph-mon[97633]: pgmap v12637: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:04:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:10.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:10.787476+0000) 2022-09-10T11:04:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:04:10.995+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:04:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:11.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:11.271390+0000) 2022-09-10T11:04:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:11.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:11.787716+0000) 2022-09-10T11:04:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:12.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:12.271532+0000) 2022-09-10T11:04:12.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:12 smithi104 ceph-mon[97044]: pgmap v12638: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:04:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:12 smithi110 ceph-mon[97633]: pgmap v12638: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:04:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:12.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:12.787952+0000) 2022-09-10T11:04:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:13.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:13.271681+0000) 2022-09-10T11:04:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:13.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:13.788212+0000) 2022-09-10T11:04:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:14.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:14.271848+0000) 2022-09-10T11:04:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:14 smithi110 ceph-mon[97633]: osdmap e871: 12 total, 12 up, 12 in 2022-09-10T11:04:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:14 smithi110 ceph-mon[97633]: pgmap v12640: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:04:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:14 smithi104 ceph-mon[97044]: osdmap e871: 12 total, 12 up, 12 in 2022-09-10T11:04:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:14 smithi104 ceph-mon[97044]: pgmap v12640: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:04:15.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:14.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:14.788432+0000) 2022-09-10T11:04:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:15.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:15.272030+0000) 2022-09-10T11:04:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:15.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:15.788631+0000) 2022-09-10T11:04:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:16.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:16.272245+0000) 2022-09-10T11:04:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:16 smithi104 ceph-mon[97044]: pgmap v12641: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:04:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:16 smithi110 ceph-mon[97633]: pgmap v12641: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:04:17.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:16.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:16.788755+0000) 2022-09-10T11:04:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:17.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:17.272472+0000) 2022-09-10T11:04:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:17.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:17.788973+0000) 2022-09-10T11:04:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:18.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:18.272708+0000) 2022-09-10T11:04:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:18 smithi104 ceph-mon[97044]: pgmap v12642: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:04:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:18 smithi110 ceph-mon[97633]: pgmap v12642: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:04:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:18.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:18.789208+0000) 2022-09-10T11:04:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:19.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:19.272928+0000) 2022-09-10T11:04:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:19.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:19.789414+0000) 2022-09-10T11:04:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:20.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:20.273068+0000) 2022-09-10T11:04:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:20 smithi104 ceph-mon[97044]: pgmap v12643: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:04:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:20 smithi110 ceph-mon[97633]: pgmap v12643: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:04:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:20.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:20.789634+0000) 2022-09-10T11:04:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:21.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:21.273319+0000) 2022-09-10T11:04:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:21.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:21.789852+0000) 2022-09-10T11:04:22.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:22.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:22.273461+0000) 2022-09-10T11:04:22.787 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:22 smithi104 ceph-mon[97044]: pgmap v12644: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:04:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:22 smithi110 ceph-mon[97633]: pgmap v12644: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:04:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:22.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:22.790136+0000) 2022-09-10T11:04:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:23.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:23.273702+0000) 2022-09-10T11:04:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:23.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:23.790338+0000) 2022-09-10T11:04:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:24.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:24.273906+0000) 2022-09-10T11:04:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:24 smithi110 ceph-mon[97633]: pgmap v12645: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:04:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:24 smithi104 ceph-mon[97044]: pgmap v12645: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:04:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:24.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:24.790565+0000) 2022-09-10T11:04:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:25.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:25.274124+0000) 2022-09-10T11:04:26.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:04:26.010+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:04:26.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:25.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:25.790803+0000) 2022-09-10T11:04:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:26.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:26.274322+0000) 2022-09-10T11:04:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:26 smithi104 ceph-mon[97044]: pgmap v12646: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:04:26.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:26 smithi110 ceph-mon[97633]: pgmap v12646: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:04:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:26.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:26.791031+0000) 2022-09-10T11:04:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:27.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:27.274497+0000) 2022-09-10T11:04:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:27.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:27.791224+0000) 2022-09-10T11:04:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:28.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:28.274755+0000) 2022-09-10T11:04:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:28 smithi104 ceph-mon[97044]: pgmap v12647: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:28 smithi110 ceph-mon[97633]: pgmap v12647: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:28.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:28.791407+0000) 2022-09-10T11:04:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:29.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:29.274952+0000) 2022-09-10T11:04:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:29.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:29.791577+0000) 2022-09-10T11:04:30.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:30.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:30.275190+0000) 2022-09-10T11:04:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:30 smithi104 ceph-mon[97044]: pgmap v12648: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:30 smithi110 ceph-mon[97633]: pgmap v12648: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:30.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:30.791787+0000) 2022-09-10T11:04:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:31.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:31.275394+0000) 2022-09-10T11:04:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:31.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:31.792056+0000) 2022-09-10T11:04:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:32.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:32.275611+0000) 2022-09-10T11:04:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:32 smithi104 ceph-mon[97044]: pgmap v12649: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:32 smithi110 ceph-mon[97633]: pgmap v12649: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:32.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:32.792358+0000) 2022-09-10T11:04:33.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:33.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:33.275832+0000) 2022-09-10T11:04:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:33.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:33.792657+0000) 2022-09-10T11:04:34.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:34.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:34.276070+0000) 2022-09-10T11:04:34.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:34 smithi110 ceph-mon[97633]: pgmap v12650: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:34 smithi104 ceph-mon[97044]: pgmap v12650: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:34.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:34.792841+0000) 2022-09-10T11:04:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:35.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:35.276384+0000) 2022-09-10T11:04:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:35.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:35.793144+0000) 2022-09-10T11:04:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:36.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:36.276616+0000) 2022-09-10T11:04:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:36 smithi104 ceph-mon[97044]: pgmap v12651: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:04:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:36 smithi110 ceph-mon[97633]: pgmap v12651: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:04:37.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:36.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:36.793272+0000) 2022-09-10T11:04:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:37.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:37.276790+0000) 2022-09-10T11:04:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:37.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:37.793478+0000) 2022-09-10T11:04:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:38.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:38.277055+0000) 2022-09-10T11:04:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:38 smithi104 ceph-mon[97044]: pgmap v12652: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:38 smithi110 ceph-mon[97633]: pgmap v12652: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:38.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:38.793643+0000) 2022-09-10T11:04:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:39.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:39.277330+0000) 2022-09-10T11:04:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:39.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:39.793857+0000) 2022-09-10T11:04:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:40.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:40.277526+0000) 2022-09-10T11:04:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:40 smithi104 ceph-mon[97044]: pgmap v12653: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:40 smithi110 ceph-mon[97633]: pgmap v12653: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:40.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:40.794143+0000) 2022-09-10T11:04:41.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:04:41.024+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:04:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:41.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:41.277777+0000) 2022-09-10T11:04:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:41.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:41.794370+0000) 2022-09-10T11:04:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:42.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:42.277986+0000) 2022-09-10T11:04:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:42 smithi104 ceph-mon[97044]: pgmap v12654: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:42 smithi110 ceph-mon[97633]: pgmap v12654: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:42.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:42.794634+0000) 2022-09-10T11:04:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:43.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:43.278186+0000) 2022-09-10T11:04:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:43.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:43.794875+0000) 2022-09-10T11:04:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:44.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:44.278377+0000) 2022-09-10T11:04:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:44 smithi110 ceph-mon[97633]: pgmap v12655: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:44.663 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:44 smithi104 ceph-mon[97044]: pgmap v12655: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:44.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:44.795146+0000) 2022-09-10T11:04:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:45.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:45.278619+0000) 2022-09-10T11:04:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:45.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:45.795358+0000) 2022-09-10T11:04:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:46.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:46.278816+0000) 2022-09-10T11:04:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:46 smithi104 ceph-mon[97044]: pgmap v12656: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:04:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:04:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:04:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:04:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:04:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:46 smithi110 ceph-mon[97633]: pgmap v12656: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:04:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:04:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:04:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:04:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:04:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:46.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:46.795582+0000) 2022-09-10T11:04:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:47.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:47.278993+0000) 2022-09-10T11:04:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:47.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:47.795842+0000) 2022-09-10T11:04:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:48.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:48.279251+0000) 2022-09-10T11:04:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:48 smithi104 ceph-mon[97044]: pgmap v12657: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:48 smithi110 ceph-mon[97633]: pgmap v12657: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:48.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:48.796013+0000) 2022-09-10T11:04:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:49.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:49.279446+0000) 2022-09-10T11:04:50.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:49.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:49.796261+0000) 2022-09-10T11:04:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:50.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:50.279701+0000) 2022-09-10T11:04:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:50 smithi104 ceph-mon[97044]: pgmap v12658: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:50 smithi110 ceph-mon[97633]: pgmap v12658: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:04:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:50.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:50.796440+0000) 2022-09-10T11:04:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:51.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:51.279910+0000) 2022-09-10T11:04:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:51.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:51.796638+0000) 2022-09-10T11:04:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:52.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:52.280163+0000) 2022-09-10T11:04:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:52 smithi104 ceph-mon[97044]: pgmap v12659: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:52 smithi110 ceph-mon[97633]: pgmap v12659: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:04:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:52.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:52.796829+0000) 2022-09-10T11:04:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:53.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:53.280379+0000) 2022-09-10T11:04:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:53.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:53.797068+0000) 2022-09-10T11:04:54.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:54.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:54.280591+0000) 2022-09-10T11:04:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:54 smithi110 ceph-mon[97633]: pgmap v12660: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:54 smithi104 ceph-mon[97044]: pgmap v12660: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:54.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:54.797270+0000) 2022-09-10T11:04:55.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:55.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:55.280840+0000) 2022-09-10T11:04:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:55.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:55.797536+0000) 2022-09-10T11:04:56.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:04:56.033+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:04:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:56.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:56.281079+0000) 2022-09-10T11:04:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:56 smithi104 ceph-mon[97044]: pgmap v12661: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:04:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:56 smithi110 ceph-mon[97633]: pgmap v12661: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:04:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:56.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:56.797766+0000) 2022-09-10T11:04:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:57.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:57.281285+0000) 2022-09-10T11:04:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:57.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:57.798013+0000) 2022-09-10T11:04:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:58.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:58.281575+0000) 2022-09-10T11:04:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:04:58 smithi104 ceph-mon[97044]: pgmap v12662: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:04:58 smithi110 ceph-mon[97633]: pgmap v12662: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:04:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:58.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:58.798187+0000) 2022-09-10T11:04:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:04:59.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:59.281842+0000) 2022-09-10T11:05:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:04:59.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:04:59.798383+0000) 2022-09-10T11:05:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:00.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:00.282079+0000) 2022-09-10T11:05:00.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:00 smithi104 ceph-mon[97044]: pgmap v12663: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:00 smithi110 ceph-mon[97633]: pgmap v12663: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:00.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:00.798535+0000) 2022-09-10T11:05:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:01.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:01.282354+0000) 2022-09-10T11:05:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:01.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:01.798758+0000) 2022-09-10T11:05:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:02.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:02.282604+0000) 2022-09-10T11:05:02.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:02 smithi104 ceph-mon[97044]: pgmap v12664: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:02 smithi110 ceph-mon[97633]: pgmap v12664: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:02.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:02.798988+0000) 2022-09-10T11:05:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:03.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:03.282880+0000) 2022-09-10T11:05:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:03.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:03.799215+0000) 2022-09-10T11:05:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:04.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:04.283068+0000) 2022-09-10T11:05:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:04 smithi110 ceph-mon[97633]: pgmap v12665: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:04 smithi104 ceph-mon[97044]: pgmap v12665: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:04.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:04.799425+0000) 2022-09-10T11:05:05.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:05.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:05.283245+0000) 2022-09-10T11:05:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:05.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:05.799656+0000) 2022-09-10T11:05:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:06.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:06.283485+0000) 2022-09-10T11:05:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:06 smithi104 ceph-mon[97044]: pgmap v12666: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:05:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:06 smithi110 ceph-mon[97633]: pgmap v12666: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:05:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:06.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:06.799832+0000) 2022-09-10T11:05:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:07.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:07.283746+0000) 2022-09-10T11:05:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:07.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:07.800056+0000) 2022-09-10T11:05:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:08.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:08.284061+0000) 2022-09-10T11:05:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:08 smithi104 ceph-mon[97044]: pgmap v12667: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:05:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:08 smithi110 ceph-mon[97633]: pgmap v12667: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:05:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:08.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:08.800339+0000) 2022-09-10T11:05:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:09.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:09.284289+0000) 2022-09-10T11:05:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:09 smithi104 ceph-mon[97044]: osdmap e872: 12 total, 12 up, 12 in 2022-09-10T11:05:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:09 smithi110 ceph-mon[97633]: osdmap e872: 12 total, 12 up, 12 in 2022-09-10T11:05:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:09.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:09.800527+0000) 2022-09-10T11:05:10.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:10.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:10.284502+0000) 2022-09-10T11:05:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:10 smithi104 ceph-mon[97044]: pgmap v12669: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:05:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:10 smithi110 ceph-mon[97633]: pgmap v12669: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:05:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:10.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:10.800667+0000) 2022-09-10T11:05:11.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:05:11.034+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:05:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:11.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:11.284677+0000) 2022-09-10T11:05:12.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:11.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:11.800860+0000) 2022-09-10T11:05:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:12.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:12.284900+0000) 2022-09-10T11:05:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:12 smithi104 ceph-mon[97044]: pgmap v12670: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:05:12.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:12 smithi110 ceph-mon[97633]: pgmap v12670: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:05:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:12.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:12.801149+0000) 2022-09-10T11:05:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:13.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:13.285176+0000) 2022-09-10T11:05:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:13.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:13.801312+0000) 2022-09-10T11:05:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:14.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:14.285389+0000) 2022-09-10T11:05:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:14 smithi110 ceph-mon[97633]: osdmap e873: 12 total, 12 up, 12 in 2022-09-10T11:05:14.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:14 smithi110 ceph-mon[97633]: pgmap v12672: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:05:14.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:14 smithi104 ceph-mon[97044]: osdmap e873: 12 total, 12 up, 12 in 2022-09-10T11:05:14.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:14 smithi104 ceph-mon[97044]: pgmap v12672: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:05:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:14.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:14.801559+0000) 2022-09-10T11:05:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:15.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:15.285617+0000) 2022-09-10T11:05:16.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:15.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:15.801819+0000) 2022-09-10T11:05:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:16.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:16.285876+0000) 2022-09-10T11:05:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:16 smithi104 ceph-mon[97044]: pgmap v12673: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:05:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:16 smithi110 ceph-mon[97633]: pgmap v12673: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:05:17.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:16.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:16.802004+0000) 2022-09-10T11:05:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:17.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:17.286131+0000) 2022-09-10T11:05:18.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:17.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:17.802209+0000) 2022-09-10T11:05:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:18.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:18.286398+0000) 2022-09-10T11:05:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:18 smithi104 ceph-mon[97044]: pgmap v12674: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:05:18.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:18 smithi110 ceph-mon[97633]: pgmap v12674: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:05:19.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:18.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:18.802395+0000) 2022-09-10T11:05:19.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:19.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:19.286654+0000) 2022-09-10T11:05:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:19.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:19.802581+0000) 2022-09-10T11:05:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:20.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:20.286846+0000) 2022-09-10T11:05:20.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:20 smithi104 ceph-mon[97044]: pgmap v12675: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:05:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:20 smithi110 ceph-mon[97633]: pgmap v12675: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:05:21.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:20.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:20.802801+0000) 2022-09-10T11:05:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:21.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:21.287026+0000) 2022-09-10T11:05:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:21.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:21.803025+0000) 2022-09-10T11:05:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:22.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:22.287209+0000) 2022-09-10T11:05:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:22 smithi104 ceph-mon[97044]: pgmap v12676: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:05:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:22 smithi110 ceph-mon[97633]: pgmap v12676: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:05:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:22.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:22.803204+0000) 2022-09-10T11:05:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:23.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:23.287349+0000) 2022-09-10T11:05:24.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:23.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:23.803383+0000) 2022-09-10T11:05:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:24.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:24.287603+0000) 2022-09-10T11:05:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:24 smithi110 ceph-mon[97633]: pgmap v12677: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:05:24.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:24 smithi104 ceph-mon[97044]: pgmap v12677: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:05:25.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:24.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:24.803583+0000) 2022-09-10T11:05:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:25.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:25.287792+0000) 2022-09-10T11:05:26.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:25.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:25.803713+0000) 2022-09-10T11:05:26.291 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:05:26.040+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:05:26.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:26.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:26.287976+0000) 2022-09-10T11:05:26.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:26 smithi104 ceph-mon[97044]: pgmap v12678: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:05:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:26 smithi110 ceph-mon[97633]: pgmap v12678: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:05:27.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:26.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:26.803968+0000) 2022-09-10T11:05:27.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:27.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:27.288225+0000) 2022-09-10T11:05:28.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:27.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:27.804182+0000) 2022-09-10T11:05:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:28.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:28.288463+0000) 2022-09-10T11:05:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:28 smithi104 ceph-mon[97044]: pgmap v12679: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:28.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:28 smithi110 ceph-mon[97633]: pgmap v12679: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:29.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:28.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:28.804344+0000) 2022-09-10T11:05:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:29.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:29.288695+0000) 2022-09-10T11:05:30.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:29.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:29.804594+0000) 2022-09-10T11:05:30.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:30.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:30.288943+0000) 2022-09-10T11:05:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:30 smithi104 ceph-mon[97044]: pgmap v12680: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:30.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:30 smithi110 ceph-mon[97633]: pgmap v12680: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:31.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:30.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:30.804844+0000) 2022-09-10T11:05:31.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:31.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:31.289206+0000) 2022-09-10T11:05:32.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:31.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:31.805169+0000) 2022-09-10T11:05:32.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:32.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:32.289387+0000) 2022-09-10T11:05:32.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:32 smithi104 ceph-mon[97044]: pgmap v12681: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:32.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:32 smithi110 ceph-mon[97633]: pgmap v12681: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:33.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:32.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:32.805371+0000) 2022-09-10T11:05:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:33.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:33.289572+0000) 2022-09-10T11:05:34.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:33.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:33.805596+0000) 2022-09-10T11:05:34.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:34.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:34.289809+0000) 2022-09-10T11:05:34.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:34 smithi110 ceph-mon[97633]: pgmap v12682: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:34 smithi104 ceph-mon[97044]: pgmap v12682: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:35.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:34.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:34.805815+0000) 2022-09-10T11:05:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:35.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:35.289946+0000) 2022-09-10T11:05:36.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:35.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:35.806058+0000) 2022-09-10T11:05:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:36.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:36.290208+0000) 2022-09-10T11:05:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:36 smithi104 ceph-mon[97044]: pgmap v12683: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:05:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:36 smithi110 ceph-mon[97633]: pgmap v12683: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:05:37.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:36.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:36.806261+0000) 2022-09-10T11:05:37.573 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:37.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:37.290395+0000) 2022-09-10T11:05:38.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:37.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:37.806438+0000) 2022-09-10T11:05:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:38.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:38.290616+0000) 2022-09-10T11:05:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:38 smithi104 ceph-mon[97044]: pgmap v12684: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:38 smithi110 ceph-mon[97633]: pgmap v12684: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:39.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:38.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:38.806694+0000) 2022-09-10T11:05:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:39.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:39.290793+0000) 2022-09-10T11:05:40.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:39.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:39.806923+0000) 2022-09-10T11:05:40.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:40.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:40.290995+0000) 2022-09-10T11:05:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:40 smithi104 ceph-mon[97044]: pgmap v12685: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:40 smithi110 ceph-mon[97633]: pgmap v12685: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:41.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:40.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:40.807210+0000) 2022-09-10T11:05:41.300 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:05:41.047+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:05:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:41.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:41.291206+0000) 2022-09-10T11:05:42.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:41.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:41.807436+0000) 2022-09-10T11:05:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:42.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:42.291403+0000) 2022-09-10T11:05:42.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:42 smithi104 ceph-mon[97044]: pgmap v12686: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:42 smithi110 ceph-mon[97633]: pgmap v12686: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:43.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:42.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:42.807611+0000) 2022-09-10T11:05:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:43.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:43.291668+0000) 2022-09-10T11:05:44.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:43.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:43.807875+0000) 2022-09-10T11:05:44.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:44.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:44.291856+0000) 2022-09-10T11:05:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:44 smithi110 ceph-mon[97633]: pgmap v12687: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:44 smithi104 ceph-mon[97044]: pgmap v12687: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:45.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:44.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:44.808162+0000) 2022-09-10T11:05:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:45.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:45.292086+0000) 2022-09-10T11:05:46.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:45.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:45.808372+0000) 2022-09-10T11:05:46.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:46.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:46.292348+0000) 2022-09-10T11:05:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:46 smithi104 ceph-mon[97044]: pgmap v12688: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:05:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:05:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:05:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:05:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:05:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:46 smithi110 ceph-mon[97633]: pgmap v12688: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:05:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:05:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:05:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:05:46.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:05:47.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:46.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:46.808608+0000) 2022-09-10T11:05:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:47.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:47.292504+0000) 2022-09-10T11:05:48.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:47.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:47.808832+0000) 2022-09-10T11:05:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:48.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:48.292694+0000) 2022-09-10T11:05:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:48 smithi104 ceph-mon[97044]: pgmap v12689: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:48 smithi110 ceph-mon[97633]: pgmap v12689: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:49.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:48.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:48.809089+0000) 2022-09-10T11:05:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:49.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:49.292887+0000) 2022-09-10T11:05:50.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:49.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:49.809360+0000) 2022-09-10T11:05:50.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:50.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:50.293171+0000) 2022-09-10T11:05:50.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:50 smithi104 ceph-mon[97044]: pgmap v12690: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:50.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:50 smithi110 ceph-mon[97633]: pgmap v12690: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:05:51.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:50.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:50.809600+0000) 2022-09-10T11:05:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:51.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:51.293350+0000) 2022-09-10T11:05:52.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:51.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:51.809856+0000) 2022-09-10T11:05:52.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:52.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:52.293495+0000) 2022-09-10T11:05:52.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:52 smithi104 ceph-mon[97044]: pgmap v12691: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:52 smithi110 ceph-mon[97633]: pgmap v12691: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:05:53.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:52.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:52.810043+0000) 2022-09-10T11:05:53.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:53.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:53.293734+0000) 2022-09-10T11:05:54.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:53.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:53.810208+0000) 2022-09-10T11:05:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:54.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:54.293946+0000) 2022-09-10T11:05:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:54 smithi110 ceph-mon[97633]: pgmap v12692: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:54.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:54 smithi104 ceph-mon[97044]: pgmap v12692: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:55.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:54.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:54.810414+0000) 2022-09-10T11:05:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:55.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:55.294163+0000) 2022-09-10T11:05:56.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:55.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:55.810637+0000) 2022-09-10T11:05:56.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:05:56.056+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:05:56.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:56.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:56.294330+0000) 2022-09-10T11:05:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:56 smithi104 ceph-mon[97044]: pgmap v12693: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:05:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:56 smithi110 ceph-mon[97633]: pgmap v12693: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:05:57.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:56.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:56.810904+0000) 2022-09-10T11:05:57.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:57.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:57.294500+0000) 2022-09-10T11:05:58.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:57.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:57.811178+0000) 2022-09-10T11:05:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:58.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:58.294694+0000) 2022-09-10T11:05:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:05:58 smithi104 ceph-mon[97044]: pgmap v12694: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:05:58 smithi110 ceph-mon[97633]: pgmap v12694: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:05:59.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:58.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:58.811342+0000) 2022-09-10T11:05:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:05:59.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:59.294946+0000) 2022-09-10T11:06:00.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:05:59.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:05:59.811576+0000) 2022-09-10T11:06:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:00.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:00.295212+0000) 2022-09-10T11:06:00.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:00 smithi104 ceph-mon[97044]: pgmap v12695: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:00 smithi110 ceph-mon[97633]: pgmap v12695: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:01.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:00.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:00.811810+0000) 2022-09-10T11:06:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:01.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:01.295432+0000) 2022-09-10T11:06:02.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:01.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:01.811993+0000) 2022-09-10T11:06:02.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:02.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:02.295638+0000) 2022-09-10T11:06:02.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:02 smithi104 ceph-mon[97044]: pgmap v12696: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:02.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:02 smithi110 ceph-mon[97633]: pgmap v12696: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:03.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:02.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:02.812205+0000) 2022-09-10T11:06:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:03.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:03.295879+0000) 2022-09-10T11:06:04.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:03.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:03.812448+0000) 2022-09-10T11:06:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:04.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:04.296136+0000) 2022-09-10T11:06:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:04 smithi110 ceph-mon[97633]: pgmap v12697: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:04.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:04 smithi104 ceph-mon[97044]: pgmap v12697: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:05.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:04.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:04.812688+0000) 2022-09-10T11:06:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:05.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:05.296367+0000) 2022-09-10T11:06:06.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:05.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:05.812870+0000) 2022-09-10T11:06:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:06.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:06.296620+0000) 2022-09-10T11:06:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:06 smithi104 ceph-mon[97044]: pgmap v12698: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:06:06.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:06 smithi110 ceph-mon[97633]: pgmap v12698: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:06:07.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:06.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:06.813061+0000) 2022-09-10T11:06:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:07.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:07.296850+0000) 2022-09-10T11:06:08.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:07.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:07.813242+0000) 2022-09-10T11:06:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:08.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:08.297042+0000) 2022-09-10T11:06:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:08 smithi104 ceph-mon[97044]: pgmap v12699: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:06:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:08 smithi110 ceph-mon[97633]: pgmap v12699: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:06:09.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:08.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:08.813421+0000) 2022-09-10T11:06:09.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:09.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:09.297260+0000) 2022-09-10T11:06:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:09 smithi104 ceph-mon[97044]: osdmap e874: 12 total, 12 up, 12 in 2022-09-10T11:06:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:09 smithi110 ceph-mon[97633]: osdmap e874: 12 total, 12 up, 12 in 2022-09-10T11:06:10.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:09.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:09.813646+0000) 2022-09-10T11:06:10.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:10.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:10.297500+0000) 2022-09-10T11:06:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:10 smithi104 ceph-mon[97044]: pgmap v12701: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:06:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:10 smithi110 ceph-mon[97633]: pgmap v12701: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:06:11.074 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:10.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:10.813875+0000) 2022-09-10T11:06:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:11.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:11.297769+0000) 2022-09-10T11:06:11.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:06:11.072+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:06:12.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:11.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:11.814134+0000) 2022-09-10T11:06:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:12.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:12.298010+0000) 2022-09-10T11:06:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:12 smithi104 ceph-mon[97044]: pgmap v12702: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:06:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:12 smithi110 ceph-mon[97633]: pgmap v12702: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:06:13.314 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:12.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:12.814320+0000) 2022-09-10T11:06:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:13.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:13.298241+0000) 2022-09-10T11:06:14.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:13.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:13.814542+0000) 2022-09-10T11:06:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:14.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:14.298395+0000) 2022-09-10T11:06:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:14 smithi110 ceph-mon[97633]: osdmap e875: 12 total, 12 up, 12 in 2022-09-10T11:06:14.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:14 smithi110 ceph-mon[97633]: pgmap v12704: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:06:14.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:14 smithi104 ceph-mon[97044]: osdmap e875: 12 total, 12 up, 12 in 2022-09-10T11:06:14.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:14 smithi104 ceph-mon[97044]: pgmap v12704: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:06:15.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:14.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:14.814733+0000) 2022-09-10T11:06:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:15.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:15.298636+0000) 2022-09-10T11:06:16.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:15.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:15.815022+0000) 2022-09-10T11:06:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:16.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:16.298872+0000) 2022-09-10T11:06:16.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:16 smithi104 ceph-mon[97044]: pgmap v12705: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:06:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:16 smithi110 ceph-mon[97633]: pgmap v12705: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:06:17.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:16.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:16.815252+0000) 2022-09-10T11:06:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:17.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:17.299039+0000) 2022-09-10T11:06:18.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:17.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:17.815474+0000) 2022-09-10T11:06:18.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:18.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:18.299275+0000) 2022-09-10T11:06:18.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:18 smithi104 ceph-mon[97044]: pgmap v12706: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T11:06:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:18 smithi110 ceph-mon[97633]: pgmap v12706: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T11:06:19.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:18.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:18.815721+0000) 2022-09-10T11:06:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:19.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:19.299432+0000) 2022-09-10T11:06:20.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:19.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:19.815951+0000) 2022-09-10T11:06:20.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:20.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:20.299586+0000) 2022-09-10T11:06:20.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:20 smithi104 ceph-mon[97044]: pgmap v12707: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:06:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:20 smithi110 ceph-mon[97633]: pgmap v12707: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:06:21.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:20.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:20.816182+0000) 2022-09-10T11:06:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:21.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:21.299760+0000) 2022-09-10T11:06:22.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:21.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:21.816327+0000) 2022-09-10T11:06:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:22.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:22.300019+0000) 2022-09-10T11:06:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:22 smithi104 ceph-mon[97044]: pgmap v12708: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:06:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:22 smithi110 ceph-mon[97633]: pgmap v12708: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:06:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:22.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:22.816490+0000) 2022-09-10T11:06:23.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:23.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:23.300270+0000) 2022-09-10T11:06:24.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:23.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:23.816765+0000) 2022-09-10T11:06:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:24.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:24.300468+0000) 2022-09-10T11:06:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:24 smithi110 ceph-mon[97633]: pgmap v12709: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:06:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:24 smithi104 ceph-mon[97044]: pgmap v12709: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:06:25.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:24.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:24.817026+0000) 2022-09-10T11:06:25.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:25.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:25.300701+0000) 2022-09-10T11:06:26.088 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:25.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:25.817235+0000) 2022-09-10T11:06:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:26.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:26.300937+0000) 2022-09-10T11:06:26.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:06:26.086+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:06:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:26 smithi104 ceph-mon[97044]: pgmap v12710: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:06:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:26 smithi110 ceph-mon[97633]: pgmap v12710: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:06:27.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:26.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:26.817507+0000) 2022-09-10T11:06:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:27.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:27.301211+0000) 2022-09-10T11:06:28.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:27.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:27.817736+0000) 2022-09-10T11:06:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:28.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:28.301430+0000) 2022-09-10T11:06:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:28 smithi104 ceph-mon[97044]: pgmap v12711: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:28 smithi110 ceph-mon[97633]: pgmap v12711: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:28.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:28.817973+0000) 2022-09-10T11:06:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:29.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:29.301649+0000) 2022-09-10T11:06:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:29.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:29.818186+0000) 2022-09-10T11:06:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:30.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:30.301876+0000) 2022-09-10T11:06:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:30 smithi104 ceph-mon[97044]: pgmap v12712: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:30 smithi110 ceph-mon[97633]: pgmap v12712: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:30.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:30.818351+0000) 2022-09-10T11:06:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:31.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:31.302158+0000) 2022-09-10T11:06:32.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:31.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:31.818622+0000) 2022-09-10T11:06:32.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:32.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:32.302362+0000) 2022-09-10T11:06:32.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:32 smithi104 ceph-mon[97044]: pgmap v12713: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:32 smithi110 ceph-mon[97633]: pgmap v12713: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:33.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:32.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:32.818821+0000) 2022-09-10T11:06:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:33.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:33.302644+0000) 2022-09-10T11:06:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:33.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:33.819080+0000) 2022-09-10T11:06:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:34.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:34.302862+0000) 2022-09-10T11:06:34.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:34 smithi110 ceph-mon[97633]: pgmap v12714: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:34.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:34 smithi104 ceph-mon[97044]: pgmap v12714: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:34.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:34.819319+0000) 2022-09-10T11:06:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:35.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:35.303058+0000) 2022-09-10T11:06:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:35.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:35.819520+0000) 2022-09-10T11:06:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:36.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:36.303254+0000) 2022-09-10T11:06:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:36 smithi104 ceph-mon[97044]: pgmap v12715: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:06:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:36 smithi110 ceph-mon[97633]: pgmap v12715: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:06:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:36.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:36.819750+0000) 2022-09-10T11:06:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:37.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:37.303406+0000) 2022-09-10T11:06:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:37.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:37.819983+0000) 2022-09-10T11:06:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:38 smithi104 ceph-mon[97044]: pgmap v12716: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:38.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:38.303573+0000) 2022-09-10T11:06:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:38 smithi110 ceph-mon[97633]: pgmap v12716: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:38.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:38.820158+0000) 2022-09-10T11:06:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:39.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:39.303771+0000) 2022-09-10T11:06:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:39.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:39.820401+0000) 2022-09-10T11:06:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:40 smithi104 ceph-mon[97044]: pgmap v12717: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:40.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:40.303973+0000) 2022-09-10T11:06:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:40 smithi110 ceph-mon[97633]: pgmap v12717: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:41.104 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:40.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:40.820640+0000) 2022-09-10T11:06:41.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:06:41.102+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:06:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:41.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:41.304197+0000) 2022-09-10T11:06:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:41.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:41.820861+0000) 2022-09-10T11:06:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:42 smithi104 ceph-mon[97044]: pgmap v12718: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:42.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:42.304346+0000) 2022-09-10T11:06:42.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:42 smithi110 ceph-mon[97633]: pgmap v12718: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:42.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:42.821085+0000) 2022-09-10T11:06:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:43.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:43.304494+0000) 2022-09-10T11:06:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:43.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:43.821348+0000) 2022-09-10T11:06:44.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:44 smithi110 ceph-mon[97633]: pgmap v12719: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:44 smithi104 ceph-mon[97044]: pgmap v12719: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:44.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:44.304638+0000) 2022-09-10T11:06:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:44.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:44.821552+0000) 2022-09-10T11:06:45.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:45.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:45.304830+0000) 2022-09-10T11:06:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:45.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:45.821749+0000) 2022-09-10T11:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:46 smithi104 ceph-mon[97044]: pgmap v12720: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:06:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:06:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:06:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:06:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:06:46.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:46.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:46.305029+0000) 2022-09-10T11:06:46.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:46 smithi110 ceph-mon[97633]: pgmap v12720: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:06:46.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:06:46.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:06:46.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:06:46.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:06:47.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:46.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:46.821947+0000) 2022-09-10T11:06:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:47.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:47.305211+0000) 2022-09-10T11:06:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:47.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:47.822167+0000) 2022-09-10T11:06:48.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:48.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:48.305393+0000) 2022-09-10T11:06:48.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:48 smithi104 ceph-mon[97044]: pgmap v12721: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:48.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:48 smithi110 ceph-mon[97633]: pgmap v12721: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:48.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:48.822292+0000) 2022-09-10T11:06:49.825 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:49.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:49.305545+0000) 2022-09-10T11:06:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:49.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:49.822459+0000) 2022-09-10T11:06:50.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:50 smithi104 ceph-mon[97044]: pgmap v12722: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:50.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:50.305721+0000) 2022-09-10T11:06:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:50 smithi110 ceph-mon[97633]: pgmap v12722: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:06:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:50.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:50.822625+0000) 2022-09-10T11:06:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:51.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:51.305965+0000) 2022-09-10T11:06:52.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:51.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:51.822842+0000) 2022-09-10T11:06:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:52 smithi104 ceph-mon[97044]: pgmap v12723: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:52.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:52.306185+0000) 2022-09-10T11:06:52.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:52 smithi110 ceph-mon[97633]: pgmap v12723: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:06:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:52.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:52.823078+0000) 2022-09-10T11:06:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:53.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:53.306375+0000) 2022-09-10T11:06:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:53.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:53.823319+0000) 2022-09-10T11:06:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:54 smithi110 ceph-mon[97633]: pgmap v12724: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:54.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:54 smithi104 ceph-mon[97044]: pgmap v12724: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:54.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:54.306498+0000) 2022-09-10T11:06:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:54.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:54.823538+0000) 2022-09-10T11:06:55.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:55.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:55.306648+0000) 2022-09-10T11:06:56.107 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:55.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:55.823798+0000) 2022-09-10T11:06:56.563 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:56.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:56.306826+0000) 2022-09-10T11:06:56.564 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:06:56.103+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:06:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:56 smithi104 ceph-mon[97044]: pgmap v12725: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:06:56.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:56 smithi110 ceph-mon[97633]: pgmap v12725: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:06:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:56.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:56.823978+0000) 2022-09-10T11:06:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:57.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:57.307046+0000) 2022-09-10T11:06:58.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:57.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:57.824152+0000) 2022-09-10T11:06:58.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:06:58 smithi104 ceph-mon[97044]: pgmap v12726: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:58.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:58.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:58.307252+0000) 2022-09-10T11:06:58.920 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:06:58 smithi110 ceph-mon[97633]: pgmap v12726: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:06:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:58.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:58.824352+0000) 2022-09-10T11:06:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:06:59.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:59.307355+0000) 2022-09-10T11:07:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:06:59.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:06:59.824581+0000) 2022-09-10T11:07:00.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:00 smithi104 ceph-mon[97044]: pgmap v12727: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:00.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:00.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:00.307506+0000) 2022-09-10T11:07:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:00 smithi110 ceph-mon[97633]: pgmap v12727: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:00.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:00.824817+0000) 2022-09-10T11:07:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:01.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:01.307708+0000) 2022-09-10T11:07:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:01.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:01.825041+0000) 2022-09-10T11:07:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:02 smithi104 ceph-mon[97044]: pgmap v12728: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:02.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:02.307870+0000) 2022-09-10T11:07:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:02 smithi110 ceph-mon[97633]: pgmap v12728: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:02.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:02.825247+0000) 2022-09-10T11:07:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:03.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:03.308152+0000) 2022-09-10T11:07:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:03.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:03.825455+0000) 2022-09-10T11:07:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:04 smithi110 ceph-mon[97633]: pgmap v12729: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:04 smithi104 ceph-mon[97044]: pgmap v12729: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:04.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:04.308363+0000) 2022-09-10T11:07:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:04.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:04.825708+0000) 2022-09-10T11:07:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:05.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:05.308593+0000) 2022-09-10T11:07:06.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:05.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:05.825955+0000) 2022-09-10T11:07:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:06 smithi104 ceph-mon[97044]: pgmap v12730: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:07:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:06.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:06.308813+0000) 2022-09-10T11:07:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:06 smithi110 ceph-mon[97633]: pgmap v12730: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:07:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:06.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:06.826181+0000) 2022-09-10T11:07:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:07.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:07.309054+0000) 2022-09-10T11:07:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:07.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:07.826389+0000) 2022-09-10T11:07:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:08 smithi104 ceph-mon[97044]: pgmap v12731: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:07:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:08.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:08.309252+0000) 2022-09-10T11:07:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:08 smithi110 ceph-mon[97633]: pgmap v12731: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:07:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:08.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:08.826606+0000) 2022-09-10T11:07:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:09 smithi104 ceph-mon[97044]: osdmap e876: 12 total, 12 up, 12 in 2022-09-10T11:07:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:09.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:09.309439+0000) 2022-09-10T11:07:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:09 smithi110 ceph-mon[97633]: osdmap e876: 12 total, 12 up, 12 in 2022-09-10T11:07:10.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:09.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:09.826828+0000) 2022-09-10T11:07:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:10 smithi104 ceph-mon[97044]: pgmap v12733: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:07:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:10.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:10.309662+0000) 2022-09-10T11:07:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:10 smithi110 ceph-mon[97633]: pgmap v12733: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:07:11.129 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:10.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:10.826979+0000) 2022-09-10T11:07:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:11.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:11.309927+0000) 2022-09-10T11:07:11.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:07:11.118+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:07:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:11.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:11.827208+0000) 2022-09-10T11:07:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:12 smithi104 ceph-mon[97044]: pgmap v12734: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:07:12.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:12.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:12.310184+0000) 2022-09-10T11:07:12.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:12 smithi110 ceph-mon[97633]: pgmap v12734: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:07:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:12.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:12.827442+0000) 2022-09-10T11:07:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:13.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:13.310388+0000) 2022-09-10T11:07:14.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:13.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:13.827639+0000) 2022-09-10T11:07:14.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:14 smithi110 ceph-mon[97633]: osdmap e877: 12 total, 12 up, 12 in 2022-09-10T11:07:14.671 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:14 smithi110 ceph-mon[97633]: pgmap v12736: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:07:14.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:14 smithi104 ceph-mon[97044]: osdmap e877: 12 total, 12 up, 12 in 2022-09-10T11:07:14.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:14 smithi104 ceph-mon[97044]: pgmap v12736: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:07:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:14.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:14.310649+0000) 2022-09-10T11:07:15.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:14.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:14.827904+0000) 2022-09-10T11:07:15.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:15.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:15.310906+0000) 2022-09-10T11:07:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:15.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:15.828152+0000) 2022-09-10T11:07:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:16 smithi104 ceph-mon[97044]: pgmap v12737: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:07:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:16.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:16.311169+0000) 2022-09-10T11:07:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:16 smithi110 ceph-mon[97633]: pgmap v12737: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:07:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:16.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:16.828294+0000) 2022-09-10T11:07:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:17.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:17.311309+0000) 2022-09-10T11:07:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:17.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:17.828558+0000) 2022-09-10T11:07:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:18 smithi104 ceph-mon[97044]: pgmap v12738: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T11:07:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:18.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:18.311448+0000) 2022-09-10T11:07:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:18 smithi110 ceph-mon[97633]: pgmap v12738: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T11:07:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:18.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:18.828790+0000) 2022-09-10T11:07:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:19.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:19.311716+0000) 2022-09-10T11:07:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:19.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:19.829022+0000) 2022-09-10T11:07:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:20 smithi104 ceph-mon[97044]: pgmap v12739: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:07:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:20.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:20.311943+0000) 2022-09-10T11:07:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:20 smithi110 ceph-mon[97633]: pgmap v12739: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:07:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:20.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:20.829247+0000) 2022-09-10T11:07:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:21.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:21.312231+0000) 2022-09-10T11:07:22.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:21.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:21.829517+0000) 2022-09-10T11:07:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:22 smithi104 ceph-mon[97044]: pgmap v12740: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:07:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:22.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:22.312457+0000) 2022-09-10T11:07:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:22 smithi110 ceph-mon[97633]: pgmap v12740: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:07:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:22.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:22.829749+0000) 2022-09-10T11:07:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:23.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:23.312632+0000) 2022-09-10T11:07:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:23.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:23.830023+0000) 2022-09-10T11:07:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:24 smithi110 ceph-mon[97633]: pgmap v12741: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:07:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:24 smithi104 ceph-mon[97044]: pgmap v12741: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:07:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:24.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:24.312877+0000) 2022-09-10T11:07:25.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:24.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:24.830277+0000) 2022-09-10T11:07:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:25.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:25.313085+0000) 2022-09-10T11:07:26.136 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:25.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:25.830500+0000) 2022-09-10T11:07:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:26.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:26.313346+0000) 2022-09-10T11:07:26.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:07:26.131+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:07:26.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:26 smithi104 ceph-mon[97044]: pgmap v12742: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:07:26.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:26 smithi110 ceph-mon[97633]: pgmap v12742: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:07:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:26.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:26.830640+0000) 2022-09-10T11:07:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:27.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:27.313563+0000) 2022-09-10T11:07:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:27.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:27.830821+0000) 2022-09-10T11:07:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:28 smithi104 ceph-mon[97044]: pgmap v12743: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:28.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:28.313795+0000) 2022-09-10T11:07:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:28 smithi110 ceph-mon[97633]: pgmap v12743: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:28.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:28.831065+0000) 2022-09-10T11:07:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:29.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:29.314049+0000) 2022-09-10T11:07:30.316 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:29.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:29.831334+0000) 2022-09-10T11:07:30.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:30 smithi104 ceph-mon[97044]: pgmap v12744: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:30.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:30.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:30.314274+0000) 2022-09-10T11:07:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:30 smithi110 ceph-mon[97633]: pgmap v12744: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:31.319 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:30.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:30.831459+0000) 2022-09-10T11:07:31.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:31.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:31.314466+0000) 2022-09-10T11:07:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:31.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:31.831663+0000) 2022-09-10T11:07:32.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:32 smithi104 ceph-mon[97044]: pgmap v12745: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:32.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:32.314611+0000) 2022-09-10T11:07:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:32 smithi110 ceph-mon[97633]: pgmap v12745: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:32.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:32.831838+0000) 2022-09-10T11:07:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:33.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:33.314793+0000) 2022-09-10T11:07:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:33.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:33.832055+0000) 2022-09-10T11:07:34.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:34 smithi110 ceph-mon[97633]: pgmap v12746: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:34 smithi104 ceph-mon[97044]: pgmap v12746: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:34.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:34.315006+0000) 2022-09-10T11:07:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:34.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:34.832336+0000) 2022-09-10T11:07:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:35.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:35.315227+0000) 2022-09-10T11:07:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:35.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:35.832510+0000) 2022-09-10T11:07:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:36 smithi104 ceph-mon[97044]: pgmap v12747: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:07:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:36.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:36.315390+0000) 2022-09-10T11:07:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:36 smithi110 ceph-mon[97633]: pgmap v12747: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:07:37.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:36.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:36.832715+0000) 2022-09-10T11:07:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:37.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:37.315622+0000) 2022-09-10T11:07:38.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:37.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:37.832977+0000) 2022-09-10T11:07:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:38 smithi104 ceph-mon[97044]: pgmap v12748: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:38.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:38.315879+0000) 2022-09-10T11:07:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:38 smithi110 ceph-mon[97633]: pgmap v12748: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:38.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:38.833160+0000) 2022-09-10T11:07:39.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:39.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:39.316127+0000) 2022-09-10T11:07:40.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:39.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:39.833436+0000) 2022-09-10T11:07:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:40 smithi104 ceph-mon[97044]: pgmap v12749: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:40.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:40.316334+0000) 2022-09-10T11:07:40.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:40 smithi110 ceph-mon[97633]: pgmap v12749: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:41.148 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:40.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:40.833654+0000) 2022-09-10T11:07:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:41.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:41.316501+0000) 2022-09-10T11:07:41.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:07:41.147+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:07:42.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:41.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:41.833884+0000) 2022-09-10T11:07:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:42 smithi104 ceph-mon[97044]: pgmap v12750: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:42.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:42.316735+0000) 2022-09-10T11:07:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:42 smithi110 ceph-mon[97633]: pgmap v12750: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:42.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:42.834133+0000) 2022-09-10T11:07:43.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:43.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:43.316897+0000) 2022-09-10T11:07:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:43.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:43.834321+0000) 2022-09-10T11:07:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:44 smithi110 ceph-mon[97633]: pgmap v12751: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:44 smithi104 ceph-mon[97044]: pgmap v12751: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:44.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:44.317206+0000) 2022-09-10T11:07:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:44.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:44.834549+0000) 2022-09-10T11:07:45.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:07:45.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:07:45.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:07:45.805 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:07:45.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:45.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:45.317435+0000) 2022-09-10T11:07:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:07:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:07:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:07:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:07:46.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:45.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:45.834796+0000) 2022-09-10T11:07:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:46 smithi104 ceph-mon[97044]: pgmap v12752: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:07:46.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:46.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:46.317645+0000) 2022-09-10T11:07:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:46 smithi110 ceph-mon[97633]: pgmap v12752: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:07:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:46.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:46.835019+0000) 2022-09-10T11:07:47.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:47.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:47.317888+0000) 2022-09-10T11:07:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:47.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:47.835270+0000) 2022-09-10T11:07:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:48 smithi104 ceph-mon[97044]: pgmap v12753: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:48.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:48.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:48.318160+0000) 2022-09-10T11:07:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:48 smithi110 ceph-mon[97633]: pgmap v12753: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:48.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:48.835506+0000) 2022-09-10T11:07:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:49.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:49.318339+0000) 2022-09-10T11:07:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:49.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:49.835746+0000) 2022-09-10T11:07:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:50 smithi104 ceph-mon[97044]: pgmap v12754: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:50.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:50.318499+0000) 2022-09-10T11:07:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:50 smithi110 ceph-mon[97633]: pgmap v12754: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:07:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:50.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:50.835987+0000) 2022-09-10T11:07:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:51.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:51.318714+0000) 2022-09-10T11:07:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:51.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:51.836240+0000) 2022-09-10T11:07:52.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:52 smithi104 ceph-mon[97044]: pgmap v12755: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:52.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:52.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:52.318932+0000) 2022-09-10T11:07:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:52 smithi110 ceph-mon[97633]: pgmap v12755: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:07:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:52.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:52.836472+0000) 2022-09-10T11:07:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:53.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:53.319207+0000) 2022-09-10T11:07:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:53.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:53.836638+0000) 2022-09-10T11:07:54.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:54 smithi110 ceph-mon[97633]: pgmap v12756: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:54.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:54 smithi104 ceph-mon[97044]: pgmap v12756: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:54.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:54.319429+0000) 2022-09-10T11:07:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:54.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:54.836841+0000) 2022-09-10T11:07:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:55.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:55.319647+0000) 2022-09-10T11:07:56.154 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:55.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:55.837087+0000) 2022-09-10T11:07:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:56.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:56.319882+0000) 2022-09-10T11:07:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:07:56.152+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:07:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:56 smithi104 ceph-mon[97044]: pgmap v12757: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:07:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:56 smithi110 ceph-mon[97633]: pgmap v12757: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:07:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:56.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:56.837342+0000) 2022-09-10T11:07:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:57.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:57.320110+0000) 2022-09-10T11:07:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:57.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:57.837590+0000) 2022-09-10T11:07:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:07:58 smithi104 ceph-mon[97044]: pgmap v12758: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:58.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:58.320323+0000) 2022-09-10T11:07:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:07:58 smithi110 ceph-mon[97633]: pgmap v12758: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:07:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:58.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:58.837828+0000) 2022-09-10T11:07:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:07:59.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:59.320557+0000) 2022-09-10T11:08:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:07:59.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:07:59.838080+0000) 2022-09-10T11:08:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:00 smithi104 ceph-mon[97044]: pgmap v12759: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:00.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:00.320697+0000) 2022-09-10T11:08:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:00 smithi110 ceph-mon[97633]: pgmap v12759: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:01.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:00.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:00.838302+0000) 2022-09-10T11:08:01.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:01.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:01.320829+0000) 2022-09-10T11:08:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:01.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:01.838550+0000) 2022-09-10T11:08:02.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:02 smithi104 ceph-mon[97044]: pgmap v12760: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:08:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:02.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:02.321061+0000) 2022-09-10T11:08:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:02 smithi110 ceph-mon[97633]: pgmap v12760: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:08:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:02.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:02.838801+0000) 2022-09-10T11:08:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:03.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:03.321320+0000) 2022-09-10T11:08:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:03.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:03.838956+0000) 2022-09-10T11:08:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:04 smithi110 ceph-mon[97633]: pgmap v12761: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:04 smithi104 ceph-mon[97044]: pgmap v12761: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:04.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:04.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:04.321494+0000) 2022-09-10T11:08:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:04.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:04.839209+0000) 2022-09-10T11:08:05.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:05.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:05.321697+0000) 2022-09-10T11:08:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:05.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:05.839474+0000) 2022-09-10T11:08:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:06 smithi104 ceph-mon[97044]: pgmap v12762: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:08:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:06.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:06.321921+0000) 2022-09-10T11:08:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:06 smithi110 ceph-mon[97633]: pgmap v12762: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:08:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:06.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:06.839739+0000) 2022-09-10T11:08:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:07.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:07.322163+0000) 2022-09-10T11:08:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:07.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:07.839966+0000) 2022-09-10T11:08:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:08 smithi104 ceph-mon[97044]: pgmap v12763: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:08:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:08.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:08.322368+0000) 2022-09-10T11:08:08.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:08 smithi110 ceph-mon[97633]: pgmap v12763: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:08:09.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:08.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:08.840207+0000) 2022-09-10T11:08:09.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:09 smithi104 ceph-mon[97044]: osdmap e878: 12 total, 12 up, 12 in 2022-09-10T11:08:09.819 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:09.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:09.322551+0000) 2022-09-10T11:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:09 smithi110 ceph-mon[97633]: osdmap e878: 12 total, 12 up, 12 in 2022-09-10T11:08:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:09.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:09.840439+0000) 2022-09-10T11:08:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:10 smithi104 ceph-mon[97044]: pgmap v12765: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:08:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:10.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:10.322728+0000) 2022-09-10T11:08:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:10 smithi110 ceph-mon[97633]: pgmap v12765: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:08:11.156 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:10.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:10.840693+0000) 2022-09-10T11:08:11.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:11.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:11.322995+0000) 2022-09-10T11:08:11.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:08:11.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:08:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:11.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:11.840937+0000) 2022-09-10T11:08:12.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:12 smithi104 ceph-mon[97044]: pgmap v12766: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:08:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:12.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:12.323149+0000) 2022-09-10T11:08:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:12 smithi110 ceph-mon[97633]: pgmap v12766: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:08:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:12.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:12.841202+0000) 2022-09-10T11:08:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:13.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:13.323341+0000) 2022-09-10T11:08:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:13.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:13.841472+0000) 2022-09-10T11:08:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:14 smithi110 ceph-mon[97633]: osdmap e879: 12 total, 12 up, 12 in 2022-09-10T11:08:14.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:14 smithi110 ceph-mon[97633]: pgmap v12768: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:08:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:14 smithi104 ceph-mon[97044]: osdmap e879: 12 total, 12 up, 12 in 2022-09-10T11:08:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:14 smithi104 ceph-mon[97044]: pgmap v12768: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:08:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:14.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:14.323558+0000) 2022-09-10T11:08:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:14.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:14.841719+0000) 2022-09-10T11:08:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:15.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:15.323722+0000) 2022-09-10T11:08:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:15.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:15.841919+0000) 2022-09-10T11:08:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:16 smithi104 ceph-mon[97044]: pgmap v12769: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:08:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:16.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:16.323990+0000) 2022-09-10T11:08:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:16 smithi110 ceph-mon[97633]: pgmap v12769: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:08:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:16.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:16.842193+0000) 2022-09-10T11:08:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:17.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:17.324200+0000) 2022-09-10T11:08:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:17.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:17.842423+0000) 2022-09-10T11:08:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:18 smithi104 ceph-mon[97044]: pgmap v12770: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:08:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:18.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:18.324421+0000) 2022-09-10T11:08:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:18 smithi110 ceph-mon[97633]: pgmap v12770: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:08:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:18.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:18.842667+0000) 2022-09-10T11:08:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:19.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:19.324548+0000) 2022-09-10T11:08:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:19.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:19.842913+0000) 2022-09-10T11:08:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:20 smithi104 ceph-mon[97044]: pgmap v12771: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:08:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:20.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:20.324754+0000) 2022-09-10T11:08:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:20 smithi110 ceph-mon[97633]: pgmap v12771: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:08:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:20.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:20.843080+0000) 2022-09-10T11:08:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:21.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:21.324964+0000) 2022-09-10T11:08:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:21.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:21.843305+0000) 2022-09-10T11:08:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:22 smithi104 ceph-mon[97044]: pgmap v12772: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:08:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:22.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:22.325205+0000) 2022-09-10T11:08:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:22 smithi110 ceph-mon[97633]: pgmap v12772: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:08:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:22.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:22.843558+0000) 2022-09-10T11:08:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:23.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:23.325395+0000) 2022-09-10T11:08:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:23.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:23.843776+0000) 2022-09-10T11:08:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:24 smithi110 ceph-mon[97633]: pgmap v12773: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:08:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:24 smithi104 ceph-mon[97044]: pgmap v12773: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:08:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:24.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:24.325574+0000) 2022-09-10T11:08:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:24.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:24.843992+0000) 2022-09-10T11:08:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:25.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:25.325813+0000) 2022-09-10T11:08:26.173 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:25.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:25.844253+0000) 2022-09-10T11:08:26.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:08:26.170+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:08:26.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:26.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:26.326018+0000) 2022-09-10T11:08:26.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:26 smithi104 ceph-mon[97044]: pgmap v12774: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:08:26.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:26 smithi110 ceph-mon[97633]: pgmap v12774: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:08:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:26.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:26.844467+0000) 2022-09-10T11:08:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:27.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:27.326270+0000) 2022-09-10T11:08:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:27.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:27.844624+0000) 2022-09-10T11:08:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:28 smithi104 ceph-mon[97044]: pgmap v12775: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:28.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:28.326483+0000) 2022-09-10T11:08:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:28 smithi110 ceph-mon[97633]: pgmap v12775: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:29.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:28.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:28.844836+0000) 2022-09-10T11:08:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:29.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:29.326758+0000) 2022-09-10T11:08:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:29.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:29.845123+0000) 2022-09-10T11:08:30.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:30.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:30.326981+0000) 2022-09-10T11:08:30.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:30 smithi104 ceph-mon[97044]: pgmap v12776: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:30 smithi110 ceph-mon[97633]: pgmap v12776: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:30.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:30.845318+0000) 2022-09-10T11:08:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:31.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:31.327222+0000) 2022-09-10T11:08:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:31.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:31.845488+0000) 2022-09-10T11:08:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:32 smithi104 ceph-mon[97044]: pgmap v12777: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:32.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:32.327444+0000) 2022-09-10T11:08:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:32 smithi110 ceph-mon[97633]: pgmap v12777: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:32.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:32.845689+0000) 2022-09-10T11:08:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:33.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:33.327727+0000) 2022-09-10T11:08:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:33.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:33.845947+0000) 2022-09-10T11:08:34.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:34 smithi110 ceph-mon[97633]: pgmap v12778: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:34.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:34 smithi104 ceph-mon[97044]: pgmap v12778: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:34.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:34.327973+0000) 2022-09-10T11:08:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:34.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:34.846180+0000) 2022-09-10T11:08:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:35.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:35.328258+0000) 2022-09-10T11:08:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:35.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:35.846338+0000) 2022-09-10T11:08:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:36 smithi104 ceph-mon[97044]: pgmap v12779: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:36.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:36.328459+0000) 2022-09-10T11:08:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:36 smithi110 ceph-mon[97633]: pgmap v12779: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:36.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:36.846512+0000) 2022-09-10T11:08:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:37.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:37.328655+0000) 2022-09-10T11:08:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:37.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:37.846717+0000) 2022-09-10T11:08:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:38 smithi104 ceph-mon[97044]: pgmap v12780: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:38.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:38.328897+0000) 2022-09-10T11:08:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:38 smithi110 ceph-mon[97633]: pgmap v12780: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:38.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:38.847002+0000) 2022-09-10T11:08:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:39.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:39.329149+0000) 2022-09-10T11:08:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:39.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:39.847247+0000) 2022-09-10T11:08:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:40 smithi104 ceph-mon[97044]: pgmap v12781: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:40.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:40.329363+0000) 2022-09-10T11:08:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:40 smithi110 ceph-mon[97633]: pgmap v12781: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:41.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:40.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:40.847478+0000) 2022-09-10T11:08:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:41.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:41.329547+0000) 2022-09-10T11:08:41.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:08:41.186+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:08:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:41.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:41.847628+0000) 2022-09-10T11:08:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:42 smithi104 ceph-mon[97044]: pgmap v12782: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:42.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:42.329782+0000) 2022-09-10T11:08:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:42 smithi110 ceph-mon[97633]: pgmap v12782: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:42.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:42.847849+0000) 2022-09-10T11:08:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:43.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:43.330011+0000) 2022-09-10T11:08:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:43.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:43.848147+0000) 2022-09-10T11:08:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:44 smithi110 ceph-mon[97633]: pgmap v12783: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:44 smithi104 ceph-mon[97044]: pgmap v12783: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:44.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:44.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:44.330257+0000) 2022-09-10T11:08:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:44.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:44.848350+0000) 2022-09-10T11:08:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:08:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:08:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:08:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:08:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:45.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:45.330470+0000) 2022-09-10T11:08:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:08:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:08:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:08:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:08:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:45.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:45.848528+0000) 2022-09-10T11:08:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:46 smithi104 ceph-mon[97044]: pgmap v12784: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:46.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:46.330704+0000) 2022-09-10T11:08:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:46 smithi110 ceph-mon[97633]: pgmap v12784: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:47.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:46.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:46.848731+0000) 2022-09-10T11:08:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:47.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:47.330956+0000) 2022-09-10T11:08:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:47.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:47.848973+0000) 2022-09-10T11:08:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:48 smithi104 ceph-mon[97044]: pgmap v12785: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:48.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:48.331236+0000) 2022-09-10T11:08:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:48 smithi110 ceph-mon[97633]: pgmap v12785: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:48.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:48.849214+0000) 2022-09-10T11:08:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:49.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:49.331489+0000) 2022-09-10T11:08:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:49.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:49.849364+0000) 2022-09-10T11:08:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:50 smithi104 ceph-mon[97044]: pgmap v12786: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:50.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:50.331758+0000) 2022-09-10T11:08:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:50 smithi110 ceph-mon[97633]: pgmap v12786: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:50.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:50.849579+0000) 2022-09-10T11:08:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:51.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:51.331922+0000) 2022-09-10T11:08:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:51.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:51.849835+0000) 2022-09-10T11:08:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:52 smithi104 ceph-mon[97044]: pgmap v12787: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:52.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:52.332187+0000) 2022-09-10T11:08:52.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:52 smithi110 ceph-mon[97633]: pgmap v12787: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:52.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:52.850075+0000) 2022-09-10T11:08:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:53.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:53.332398+0000) 2022-09-10T11:08:54.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:53.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:53.850360+0000) 2022-09-10T11:08:54.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:54 smithi110 ceph-mon[97633]: pgmap v12788: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:54.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:54 smithi104 ceph-mon[97044]: pgmap v12788: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:54.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:54.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:54.332680+0000) 2022-09-10T11:08:55.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:54.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:54.850593+0000) 2022-09-10T11:08:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:55.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:55.332947+0000) 2022-09-10T11:08:56.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:55.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:55.850810+0000) 2022-09-10T11:08:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:08:56.201+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:08:56.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:56.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:56.333225+0000) 2022-09-10T11:08:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:56 smithi104 ceph-mon[97044]: pgmap v12789: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:56 smithi110 ceph-mon[97633]: pgmap v12789: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:08:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:56.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:56.851039+0000) 2022-09-10T11:08:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:57.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:57.333494+0000) 2022-09-10T11:08:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:57.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:57.851276+0000) 2022-09-10T11:08:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:08:58 smithi104 ceph-mon[97044]: pgmap v12790: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:58.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:58.333770+0000) 2022-09-10T11:08:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:08:58 smithi110 ceph-mon[97633]: pgmap v12790: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:08:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:58.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:58.851487+0000) 2022-09-10T11:08:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:08:59.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:59.334043+0000) 2022-09-10T11:09:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:08:59.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:08:59.851725+0000) 2022-09-10T11:09:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:00 smithi104 ceph-mon[97044]: pgmap v12791: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:00.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:00.334284+0000) 2022-09-10T11:09:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:00 smithi110 ceph-mon[97633]: pgmap v12791: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:00.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:00.851939+0000) 2022-09-10T11:09:01.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:01.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:01.334520+0000) 2022-09-10T11:09:02.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:01.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:01.852206+0000) 2022-09-10T11:09:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:02 smithi104 ceph-mon[97044]: pgmap v12792: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:02.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:02.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:02.334758+0000) 2022-09-10T11:09:02.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:02 smithi110 ceph-mon[97633]: pgmap v12792: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:02.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:02.852389+0000) 2022-09-10T11:09:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:03.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:03.334960+0000) 2022-09-10T11:09:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:03.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:03.852567+0000) 2022-09-10T11:09:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:04 smithi110 ceph-mon[97633]: pgmap v12793: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:04.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:04 smithi104 ceph-mon[97044]: pgmap v12793: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:04.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:04.335221+0000) 2022-09-10T11:09:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:04.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:04.852814+0000) 2022-09-10T11:09:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:05.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:05.335520+0000) 2022-09-10T11:09:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:05.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:05.853060+0000) 2022-09-10T11:09:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:06 smithi104 ceph-mon[97044]: pgmap v12794: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:09:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:06.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:06.335683+0000) 2022-09-10T11:09:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:06 smithi110 ceph-mon[97633]: pgmap v12794: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:09:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:06.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:06.853337+0000) 2022-09-10T11:09:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:07.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:07.335931+0000) 2022-09-10T11:09:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:07.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:07.853498+0000) 2022-09-10T11:09:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:08 smithi104 ceph-mon[97044]: pgmap v12795: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:09:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:08.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:08.336194+0000) 2022-09-10T11:09:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:08 smithi110 ceph-mon[97633]: pgmap v12795: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:09:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:08.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:08.853732+0000) 2022-09-10T11:09:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:09 smithi104 ceph-mon[97044]: osdmap e880: 12 total, 12 up, 12 in 2022-09-10T11:09:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:09.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:09.336442+0000) 2022-09-10T11:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:09 smithi110 ceph-mon[97633]: osdmap e880: 12 total, 12 up, 12 in 2022-09-10T11:09:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:09.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:09.853929+0000) 2022-09-10T11:09:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:10 smithi104 ceph-mon[97044]: pgmap v12797: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:09:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:10.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:10.336644+0000) 2022-09-10T11:09:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:10 smithi110 ceph-mon[97633]: pgmap v12797: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:09:11.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:10.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:10.854110+0000) 2022-09-10T11:09:11.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:09:11.217+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:09:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:11.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:11.336859+0000) 2022-09-10T11:09:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:11.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:11.854326+0000) 2022-09-10T11:09:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:12 smithi104 ceph-mon[97044]: pgmap v12798: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:09:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:12.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:12.337038+0000) 2022-09-10T11:09:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:12 smithi110 ceph-mon[97633]: pgmap v12798: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:09:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:12.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:12.854572+0000) 2022-09-10T11:09:13.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:13.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:13.337230+0000) 2022-09-10T11:09:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:13.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:13.854840+0000) 2022-09-10T11:09:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:14 smithi110 ceph-mon[97633]: osdmap e881: 12 total, 12 up, 12 in 2022-09-10T11:09:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:14 smithi110 ceph-mon[97633]: pgmap v12800: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:09:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:14 smithi104 ceph-mon[97044]: osdmap e881: 12 total, 12 up, 12 in 2022-09-10T11:09:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:14 smithi104 ceph-mon[97044]: pgmap v12800: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:09:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:14.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:14.337477+0000) 2022-09-10T11:09:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:14.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:14.855061+0000) 2022-09-10T11:09:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:15.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:15.337655+0000) 2022-09-10T11:09:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:15.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:15.855318+0000) 2022-09-10T11:09:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:16 smithi104 ceph-mon[97044]: pgmap v12801: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:09:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:16.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:16.337894+0000) 2022-09-10T11:09:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:16 smithi110 ceph-mon[97633]: pgmap v12801: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:09:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:16.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:16.855588+0000) 2022-09-10T11:09:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:17.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:17.338159+0000) 2022-09-10T11:09:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:17.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:17.855854+0000) 2022-09-10T11:09:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:18 smithi104 ceph-mon[97044]: pgmap v12802: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:09:18.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:18.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:18.338392+0000) 2022-09-10T11:09:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:18 smithi110 ceph-mon[97633]: pgmap v12802: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:09:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:18.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:18.856032+0000) 2022-09-10T11:09:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:19.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:19.338587+0000) 2022-09-10T11:09:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:19.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:19.856260+0000) 2022-09-10T11:09:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:20 smithi104 ceph-mon[97044]: pgmap v12803: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:09:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:20.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:20.338748+0000) 2022-09-10T11:09:20.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:20 smithi110 ceph-mon[97633]: pgmap v12803: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:09:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:20.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:20.856471+0000) 2022-09-10T11:09:21.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:21.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:21.338921+0000) 2022-09-10T11:09:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:21.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:21.856680+0000) 2022-09-10T11:09:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:22 smithi104 ceph-mon[97044]: pgmap v12804: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:09:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:22.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:22.339130+0000) 2022-09-10T11:09:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:22 smithi110 ceph-mon[97633]: pgmap v12804: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:09:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:22.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:22.856902+0000) 2022-09-10T11:09:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:23.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:23.339359+0000) 2022-09-10T11:09:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:23.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:23.857175+0000) 2022-09-10T11:09:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:24 smithi110 ceph-mon[97633]: pgmap v12805: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:09:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:24 smithi104 ceph-mon[97044]: pgmap v12805: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:09:24.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:24.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:24.339534+0000) 2022-09-10T11:09:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:24.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:24.857407+0000) 2022-09-10T11:09:25.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:25.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:25.339723+0000) 2022-09-10T11:09:26.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:25.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:25.857590+0000) 2022-09-10T11:09:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:09:26.234+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:09:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:26.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:26.339952+0000) 2022-09-10T11:09:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:26 smithi104 ceph-mon[97044]: pgmap v12806: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:09:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:26 smithi110 ceph-mon[97633]: pgmap v12806: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:09:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:26.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:26.857842+0000) 2022-09-10T11:09:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:27.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:27.340178+0000) 2022-09-10T11:09:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:27.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:27.858122+0000) 2022-09-10T11:09:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:28 smithi104 ceph-mon[97044]: pgmap v12807: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:28.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:28.340395+0000) 2022-09-10T11:09:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:28 smithi110 ceph-mon[97633]: pgmap v12807: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:28.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:28.858353+0000) 2022-09-10T11:09:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:29.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:29.340585+0000) 2022-09-10T11:09:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:29.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:29.858520+0000) 2022-09-10T11:09:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:30 smithi104 ceph-mon[97044]: pgmap v12808: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:30.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:30.340839+0000) 2022-09-10T11:09:30.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:30 smithi110 ceph-mon[97633]: pgmap v12808: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:30.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:30.858772+0000) 2022-09-10T11:09:31.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:31.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:31.341060+0000) 2022-09-10T11:09:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:31.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:31.859034+0000) 2022-09-10T11:09:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:32 smithi104 ceph-mon[97044]: pgmap v12809: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:32.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:32.341247+0000) 2022-09-10T11:09:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:32 smithi110 ceph-mon[97633]: pgmap v12809: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:33.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:32.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:32.859241+0000) 2022-09-10T11:09:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:33.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:33.341457+0000) 2022-09-10T11:09:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:33.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:33.859480+0000) 2022-09-10T11:09:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:34 smithi110 ceph-mon[97633]: pgmap v12810: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:34 smithi104 ceph-mon[97044]: pgmap v12810: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:34.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:34.341716+0000) 2022-09-10T11:09:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:34.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:34.859646+0000) 2022-09-10T11:09:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:35.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:35.341913+0000) 2022-09-10T11:09:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:35.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:35.859876+0000) 2022-09-10T11:09:36.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:36 smithi104 ceph-mon[97044]: pgmap v12811: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:36.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:36.342165+0000) 2022-09-10T11:09:36.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:36 smithi110 ceph-mon[97633]: pgmap v12811: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:36.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:36.860171+0000) 2022-09-10T11:09:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:37.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:37.342376+0000) 2022-09-10T11:09:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:37.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:37.860352+0000) 2022-09-10T11:09:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:38 smithi104 ceph-mon[97044]: pgmap v12812: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:38.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:38.342581+0000) 2022-09-10T11:09:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:38 smithi110 ceph-mon[97633]: pgmap v12812: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:38.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:38.860563+0000) 2022-09-10T11:09:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:39.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:39.342841+0000) 2022-09-10T11:09:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:39.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:39.860807+0000) 2022-09-10T11:09:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:40 smithi104 ceph-mon[97044]: pgmap v12813: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T11:09:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:40.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:40.343064+0000) 2022-09-10T11:09:40.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:40 smithi110 ceph-mon[97633]: pgmap v12813: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T11:09:41.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:40.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:40.861016+0000) 2022-09-10T11:09:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:41.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:41.343263+0000) 2022-09-10T11:09:41.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:09:41.248+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:09:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:41.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:41.861193+0000) 2022-09-10T11:09:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:42 smithi104 ceph-mon[97044]: pgmap v12814: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:42.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:42.343484+0000) 2022-09-10T11:09:42.921 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:42 smithi110 ceph-mon[97633]: pgmap v12814: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:42.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:42.861366+0000) 2022-09-10T11:09:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:43.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:43.343717+0000) 2022-09-10T11:09:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:43.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:43.861593+0000) 2022-09-10T11:09:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:44 smithi110 ceph-mon[97633]: pgmap v12815: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:44 smithi104 ceph-mon[97044]: pgmap v12815: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:44.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:44.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:44.343931+0000) 2022-09-10T11:09:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:44.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:44.861770+0000) 2022-09-10T11:09:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:09:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:09:45.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:45.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:45.344201+0000) 2022-09-10T11:09:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:09:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:09:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:09:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:09:46.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:45.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:45.861948+0000) 2022-09-10T11:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:46 smithi104 ceph-mon[97044]: pgmap v12816: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:46.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:46.344429+0000) 2022-09-10T11:09:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:46 smithi110 ceph-mon[97633]: pgmap v12816: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:46.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:46.862194+0000) 2022-09-10T11:09:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:47.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:47.344575+0000) 2022-09-10T11:09:48.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:47.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:47.862420+0000) 2022-09-10T11:09:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:48 smithi104 ceph-mon[97044]: pgmap v12817: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:48.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:48.344737+0000) 2022-09-10T11:09:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:48 smithi110 ceph-mon[97633]: pgmap v12817: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:48.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:48.862686+0000) 2022-09-10T11:09:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:49.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:49.344963+0000) 2022-09-10T11:09:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:49.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:49.862826+0000) 2022-09-10T11:09:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:50.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:50.345160+0000) 2022-09-10T11:09:50.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:50 smithi110 ceph-mon[97633]: pgmap v12818: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:51.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:50 smithi104 ceph-mon[97044]: pgmap v12818: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:50.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:50.863122+0000) 2022-09-10T11:09:51.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:51.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:51.345393+0000) 2022-09-10T11:09:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:51.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:51.863367+0000) 2022-09-10T11:09:52.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:52.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:52.345653+0000) 2022-09-10T11:09:52.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:52 smithi110 ceph-mon[97633]: pgmap v12819: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:52 smithi104 ceph-mon[97044]: pgmap v12819: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:52.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:52.863593+0000) 2022-09-10T11:09:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:53.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:53.345886+0000) 2022-09-10T11:09:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:53.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:53.863855+0000) 2022-09-10T11:09:54.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:54 smithi110 ceph-mon[97633]: pgmap v12820: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:54 smithi104 ceph-mon[97044]: pgmap v12820: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:54.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:54.346147+0000) 2022-09-10T11:09:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:54.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:54.864062+0000) 2022-09-10T11:09:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:55.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:55.346304+0000) 2022-09-10T11:09:56.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:55.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:55.864320+0000) 2022-09-10T11:09:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:56.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:56.346551+0000) 2022-09-10T11:09:56.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:09:56.252+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:09:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:56 smithi104 ceph-mon[97044]: pgmap v12821: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:56 smithi110 ceph-mon[97633]: pgmap v12821: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:09:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:56.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:56.864587+0000) 2022-09-10T11:09:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:57.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:57.346826+0000) 2022-09-10T11:09:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:57.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:57.864770+0000) 2022-09-10T11:09:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:09:58 smithi104 ceph-mon[97044]: pgmap v12822: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:58.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:58.347011+0000) 2022-09-10T11:09:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:09:58 smithi110 ceph-mon[97633]: pgmap v12822: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:09:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:58.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:58.865015+0000) 2022-09-10T11:09:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:09:59.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:59.347201+0000) 2022-09-10T11:10:00.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:09:59.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:09:59.865197+0000) 2022-09-10T11:10:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:00 smithi104 ceph-mon[97044]: pgmap v12823: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:10:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:00.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:00.347445+0000) 2022-09-10T11:10:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:00 smithi110 ceph-mon[97633]: pgmap v12823: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:10:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:00.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:00.865378+0000) 2022-09-10T11:10:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:01.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:01.347666+0000) 2022-09-10T11:10:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:01.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:01.865584+0000) 2022-09-10T11:10:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:02 smithi104 ceph-mon[97044]: pgmap v12824: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:02.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:02.347913+0000) 2022-09-10T11:10:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:02 smithi110 ceph-mon[97633]: pgmap v12824: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:02.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:02.865790+0000) 2022-09-10T11:10:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:03.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:03.348158+0000) 2022-09-10T11:10:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:03.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:03.866000+0000) 2022-09-10T11:10:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:04 smithi110 ceph-mon[97633]: pgmap v12825: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:04 smithi104 ceph-mon[97044]: pgmap v12825: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:04.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:04.348309+0000) 2022-09-10T11:10:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:04.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:04.866262+0000) 2022-09-10T11:10:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:05.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:05.348519+0000) 2022-09-10T11:10:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:05.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:05.866468+0000) 2022-09-10T11:10:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:06 smithi104 ceph-mon[97044]: pgmap v12826: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:10:06.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:06.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:06.348719+0000) 2022-09-10T11:10:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:06 smithi110 ceph-mon[97633]: pgmap v12826: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:10:07.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:06.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:06.866698+0000) 2022-09-10T11:10:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:07.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:07.348974+0000) 2022-09-10T11:10:08.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:07.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:07.866961+0000) 2022-09-10T11:10:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:08 smithi104 ceph-mon[97044]: pgmap v12827: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:10:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:08.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:08.349222+0000) 2022-09-10T11:10:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:08 smithi110 ceph-mon[97633]: pgmap v12827: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:10:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:08.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:08.867193+0000) 2022-09-10T11:10:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:09.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:09.349424+0000) 2022-09-10T11:10:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:09 smithi110 ceph-mon[97633]: osdmap e882: 12 total, 12 up, 12 in 2022-09-10T11:10:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:09 smithi104 ceph-mon[97044]: osdmap e882: 12 total, 12 up, 12 in 2022-09-10T11:10:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:09.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:09.867386+0000) 2022-09-10T11:10:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:10.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:10.349611+0000) 2022-09-10T11:10:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:10 smithi110 ceph-mon[97633]: pgmap v12829: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:10:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:10 smithi104 ceph-mon[97044]: pgmap v12829: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:10:11.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:10.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:10.867628+0000) 2022-09-10T11:10:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:11.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:11.349835+0000) 2022-09-10T11:10:11.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:10:11.268+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:10:12.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:11.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:11.867790+0000) 2022-09-10T11:10:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:12.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:12.350068+0000) 2022-09-10T11:10:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:12 smithi110 ceph-mon[97633]: pgmap v12830: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:10:13.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:12 smithi104 ceph-mon[97044]: pgmap v12830: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:10:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:12.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:12.868034+0000) 2022-09-10T11:10:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:13.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:13.350216+0000) 2022-09-10T11:10:14.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:13.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:13.868231+0000) 2022-09-10T11:10:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:14 smithi110 ceph-mon[97633]: pgmap v12831: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:10:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:14 smithi104 ceph-mon[97044]: pgmap v12831: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:10:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:14.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:14.350416+0000) 2022-09-10T11:10:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:14.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:14.868393+0000) 2022-09-10T11:10:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:15.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:15.350672+0000) 2022-09-10T11:10:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:15.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:15.868567+0000) 2022-09-10T11:10:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:16 smithi104 ceph-mon[97044]: pgmap v12832: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:10:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:16.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:16.350921+0000) 2022-09-10T11:10:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:16 smithi110 ceph-mon[97633]: pgmap v12832: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:10:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:16.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:16.868796+0000) 2022-09-10T11:10:17.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:17.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:17.351203+0000) 2022-09-10T11:10:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:17.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:17.868994+0000) 2022-09-10T11:10:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:18 smithi104 ceph-mon[97044]: pgmap v12833: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:10:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:18 smithi104 ceph-mon[97044]: osdmap e883: 12 total, 12 up, 12 in 2022-09-10T11:10:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:18.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:18.351393+0000) 2022-09-10T11:10:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:18 smithi110 ceph-mon[97633]: pgmap v12833: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T11:10:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:18 smithi110 ceph-mon[97633]: osdmap e883: 12 total, 12 up, 12 in 2022-09-10T11:10:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:18.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:18.869207+0000) 2022-09-10T11:10:19.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:19.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:19.351618+0000) 2022-09-10T11:10:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:19.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:19.869395+0000) 2022-09-10T11:10:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:20.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:20.351910+0000) 2022-09-10T11:10:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:20 smithi110 ceph-mon[97633]: pgmap v12835: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:10:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:20 smithi104 ceph-mon[97044]: pgmap v12835: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:10:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:20.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:20.869624+0000) 2022-09-10T11:10:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:21.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:21.352176+0000) 2022-09-10T11:10:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:21.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:21.869873+0000) 2022-09-10T11:10:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:22.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:22.352416+0000) 2022-09-10T11:10:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:22 smithi110 ceph-mon[97633]: pgmap v12836: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:10:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:22 smithi104 ceph-mon[97044]: pgmap v12836: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:10:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:22.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:22.870158+0000) 2022-09-10T11:10:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:23.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:23.352663+0000) 2022-09-10T11:10:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:23.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:23.870359+0000) 2022-09-10T11:10:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:24 smithi104 ceph-mon[97044]: pgmap v12837: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:10:24.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:24.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:24.352914+0000) 2022-09-10T11:10:24.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:24 smithi110 ceph-mon[97633]: pgmap v12837: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:10:25.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:24.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:24.870564+0000) 2022-09-10T11:10:25.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:25.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:25.353125+0000) 2022-09-10T11:10:26.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:25.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:25.870789+0000) 2022-09-10T11:10:26.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:10:26.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:10:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:26.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:26.353311+0000) 2022-09-10T11:10:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:26 smithi104 ceph-mon[97044]: pgmap v12838: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T11:10:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:26 smithi110 ceph-mon[97633]: pgmap v12838: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T11:10:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:26.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:26.871022+0000) 2022-09-10T11:10:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:27.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:27.353558+0000) 2022-09-10T11:10:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:27.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:27.871198+0000) 2022-09-10T11:10:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:28 smithi104 ceph-mon[97044]: pgmap v12839: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T11:10:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:28.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:28.353747+0000) 2022-09-10T11:10:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:28 smithi110 ceph-mon[97633]: pgmap v12839: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T11:10:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:28.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:28.871437+0000) 2022-09-10T11:10:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:29.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:29.353943+0000) 2022-09-10T11:10:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:29.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:29.871639+0000) 2022-09-10T11:10:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:30.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:30.354192+0000) 2022-09-10T11:10:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:30 smithi110 ceph-mon[97633]: pgmap v12840: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 13 op/s 2022-09-10T11:10:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:30 smithi104 ceph-mon[97044]: pgmap v12840: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 13 op/s 2022-09-10T11:10:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:30.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:30.871904+0000) 2022-09-10T11:10:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:31.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:31.354390+0000) 2022-09-10T11:10:32.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:31.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:31.872186+0000) 2022-09-10T11:10:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:32.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:32.354635+0000) 2022-09-10T11:10:32.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:32 smithi110 ceph-mon[97633]: pgmap v12841: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:32 smithi104 ceph-mon[97044]: pgmap v12841: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:32.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:32.872429+0000) 2022-09-10T11:10:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:33.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:33.354811+0000) 2022-09-10T11:10:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:33.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:33.872682+0000) 2022-09-10T11:10:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:34 smithi104 ceph-mon[97044]: pgmap v12842: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:34.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:34.355049+0000) 2022-09-10T11:10:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:34 smithi110 ceph-mon[97633]: pgmap v12842: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:34.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:34.872962+0000) 2022-09-10T11:10:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:35.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:35.355184+0000) 2022-09-10T11:10:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:35.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:35.873152+0000) 2022-09-10T11:10:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:36.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:36.355343+0000) 2022-09-10T11:10:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:36 smithi110 ceph-mon[97633]: pgmap v12843: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:36 smithi104 ceph-mon[97044]: pgmap v12843: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:36.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:36.873369+0000) 2022-09-10T11:10:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:37.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:37.355498+0000) 2022-09-10T11:10:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:37.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:37.873572+0000) 2022-09-10T11:10:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:38.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:38.355640+0000) 2022-09-10T11:10:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:38 smithi110 ceph-mon[97633]: pgmap v12844: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:38 smithi104 ceph-mon[97044]: pgmap v12844: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:39.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:38.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:38.873761+0000) 2022-09-10T11:10:39.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:39.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:39.355872+0000) 2022-09-10T11:10:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:39.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:39.873976+0000) 2022-09-10T11:10:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:40.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:40.356086+0000) 2022-09-10T11:10:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:40 smithi110 ceph-mon[97633]: pgmap v12845: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:40 smithi104 ceph-mon[97044]: pgmap v12845: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:41.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:40.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:40.874257+0000) 2022-09-10T11:10:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:41.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:41.356255+0000) 2022-09-10T11:10:41.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:10:41.298+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:10:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:41.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:41.874468+0000) 2022-09-10T11:10:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:42.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:42.356483+0000) 2022-09-10T11:10:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:42 smithi110 ceph-mon[97633]: pgmap v12846: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:10:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:42 smithi104 ceph-mon[97044]: pgmap v12846: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:10:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:42.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:42.874674+0000) 2022-09-10T11:10:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:43.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:43.356732+0000) 2022-09-10T11:10:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:43.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:43.874935+0000) 2022-09-10T11:10:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:44 smithi104 ceph-mon[97044]: pgmap v12847: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:10:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:44.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:44.356978+0000) 2022-09-10T11:10:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:44 smithi110 ceph-mon[97633]: pgmap v12847: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:10:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:44.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:44.875205+0000) 2022-09-10T11:10:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:10:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:10:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:10:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:10:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:45.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:45.357214+0000) 2022-09-10T11:10:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:10:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:10:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:10:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:10:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:45.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:45.875467+0000) 2022-09-10T11:10:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:46.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:46.357364+0000) 2022-09-10T11:10:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:46 smithi110 ceph-mon[97633]: pgmap v12848: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:46 smithi104 ceph-mon[97044]: pgmap v12848: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:46.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:46.875650+0000) 2022-09-10T11:10:47.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:47.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:47.357586+0000) 2022-09-10T11:10:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:47.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:47.875872+0000) 2022-09-10T11:10:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:48.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:48.357837+0000) 2022-09-10T11:10:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:48 smithi110 ceph-mon[97633]: pgmap v12849: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:10:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:48 smithi104 ceph-mon[97044]: pgmap v12849: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:10:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:48.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:48.876069+0000) 2022-09-10T11:10:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:49.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:49.358004+0000) 2022-09-10T11:10:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:49.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:49.876275+0000) 2022-09-10T11:10:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:50.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:50.358236+0000) 2022-09-10T11:10:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:50 smithi110 ceph-mon[97633]: pgmap v12850: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:50 smithi104 ceph-mon[97044]: pgmap v12850: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:50.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:50.876458+0000) 2022-09-10T11:10:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:51.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:51.358477+0000) 2022-09-10T11:10:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:51.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:51.876684+0000) 2022-09-10T11:10:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:52.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:52.358720+0000) 2022-09-10T11:10:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:52 smithi110 ceph-mon[97633]: pgmap v12851: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:52 smithi104 ceph-mon[97044]: pgmap v12851: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:53.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:52.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:52.876950+0000) 2022-09-10T11:10:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:53.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:53.358934+0000) 2022-09-10T11:10:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:53.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:53.877229+0000) 2022-09-10T11:10:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:54 smithi104 ceph-mon[97044]: pgmap v12852: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:54.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:54.359151+0000) 2022-09-10T11:10:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:54 smithi110 ceph-mon[97633]: pgmap v12852: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:54.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:54.877498+0000) 2022-09-10T11:10:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:55.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:55.359378+0000) 2022-09-10T11:10:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:55.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:55.877768+0000) 2022-09-10T11:10:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:56.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:56.359606+0000) 2022-09-10T11:10:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:10:56.311+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:10:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:56 smithi110 ceph-mon[97633]: pgmap v12853: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:56 smithi104 ceph-mon[97044]: pgmap v12853: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:10:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:56.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:56.878020+0000) 2022-09-10T11:10:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:57.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:57.359869+0000) 2022-09-10T11:10:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:57.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:57.878234+0000) 2022-09-10T11:10:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:58.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:58.360082+0000) 2022-09-10T11:10:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:10:58 smithi110 ceph-mon[97633]: pgmap v12854: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:10:58 smithi104 ceph-mon[97044]: pgmap v12854: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:10:59.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:58.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:58.878392+0000) 2022-09-10T11:10:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:10:59.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:59.360242+0000) 2022-09-10T11:11:00.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:10:59.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:10:59.878607+0000) 2022-09-10T11:11:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:00.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:00.360447+0000) 2022-09-10T11:11:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:00 smithi110 ceph-mon[97633]: pgmap v12855: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:00 smithi104 ceph-mon[97044]: pgmap v12855: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:01.314 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:00.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:00.878757+0000) 2022-09-10T11:11:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:01.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:01.360660+0000) 2022-09-10T11:11:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:01.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:01.878921+0000) 2022-09-10T11:11:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:02.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:02.360796+0000) 2022-09-10T11:11:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:02 smithi110 ceph-mon[97633]: pgmap v12856: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:03.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:02 smithi104 ceph-mon[97044]: pgmap v12856: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:02.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:02.879089+0000) 2022-09-10T11:11:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:03.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:03.360986+0000) 2022-09-10T11:11:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:03.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:03.879325+0000) 2022-09-10T11:11:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:04 smithi104 ceph-mon[97044]: pgmap v12857: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:04.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:04.361187+0000) 2022-09-10T11:11:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:04 smithi110 ceph-mon[97633]: pgmap v12857: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:04.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:04.879575+0000) 2022-09-10T11:11:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:05.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:05.361391+0000) 2022-09-10T11:11:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:05.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:05.879745+0000) 2022-09-10T11:11:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:06.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:06.361632+0000) 2022-09-10T11:11:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:06 smithi110 ceph-mon[97633]: pgmap v12858: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:11:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:06 smithi104 ceph-mon[97044]: pgmap v12858: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:11:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:06.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:06.880000+0000) 2022-09-10T11:11:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:07.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:07.361831+0000) 2022-09-10T11:11:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:07.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:07.880217+0000) 2022-09-10T11:11:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:08.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:08.361975+0000) 2022-09-10T11:11:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:08 smithi110 ceph-mon[97633]: pgmap v12859: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:11:09.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:08 smithi104 ceph-mon[97044]: pgmap v12859: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:11:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:08.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:08.880408+0000) 2022-09-10T11:11:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:09.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:09.362238+0000) 2022-09-10T11:11:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:09 smithi110 ceph-mon[97633]: osdmap e884: 12 total, 12 up, 12 in 2022-09-10T11:11:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:09 smithi104 ceph-mon[97044]: osdmap e884: 12 total, 12 up, 12 in 2022-09-10T11:11:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:09.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:09.880649+0000) 2022-09-10T11:11:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:10.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:10.362444+0000) 2022-09-10T11:11:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:10 smithi110 ceph-mon[97633]: pgmap v12861: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:11:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:10 smithi104 ceph-mon[97044]: pgmap v12861: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:11:11.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:10.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:10.880896+0000) 2022-09-10T11:11:11.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:11:11.326+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:11:11.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:11.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:11.362642+0000) 2022-09-10T11:11:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:11.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:11.881175+0000) 2022-09-10T11:11:12.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:12.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:12.362873+0000) 2022-09-10T11:11:12.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:12 smithi110 ceph-mon[97633]: pgmap v12862: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:11:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:12 smithi104 ceph-mon[97044]: pgmap v12862: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:11:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:12.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:12.881397+0000) 2022-09-10T11:11:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:13.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:13.363059+0000) 2022-09-10T11:11:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:13.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:13.881582+0000) 2022-09-10T11:11:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:14 smithi104 ceph-mon[97044]: osdmap e885: 12 total, 12 up, 12 in 2022-09-10T11:11:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:14 smithi104 ceph-mon[97044]: pgmap v12864: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:11:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:14.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:14.363311+0000) 2022-09-10T11:11:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:14 smithi110 ceph-mon[97633]: osdmap e885: 12 total, 12 up, 12 in 2022-09-10T11:11:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:14 smithi110 ceph-mon[97633]: pgmap v12864: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:11:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:14.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:14.881761+0000) 2022-09-10T11:11:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:15.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:15.363517+0000) 2022-09-10T11:11:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:15.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:15.881956+0000) 2022-09-10T11:11:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:16.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:16.363714+0000) 2022-09-10T11:11:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:16 smithi110 ceph-mon[97633]: pgmap v12865: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:11:17.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:16 smithi104 ceph-mon[97044]: pgmap v12865: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:11:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:16.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:16.882202+0000) 2022-09-10T11:11:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:17.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:17.363961+0000) 2022-09-10T11:11:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:17.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:17.882375+0000) 2022-09-10T11:11:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:18.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:18.364180+0000) 2022-09-10T11:11:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:18 smithi110 ceph-mon[97633]: pgmap v12866: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:11:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:18 smithi104 ceph-mon[97044]: pgmap v12866: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T11:11:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:18.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:18.882550+0000) 2022-09-10T11:11:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:19.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:19.364365+0000) 2022-09-10T11:11:20.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:19.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:19.882747+0000) 2022-09-10T11:11:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:20.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:20.364555+0000) 2022-09-10T11:11:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:20 smithi110 ceph-mon[97633]: pgmap v12867: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:11:21.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:20 smithi104 ceph-mon[97044]: pgmap v12867: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:11:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:20.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:20.882952+0000) 2022-09-10T11:11:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:21.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:21.364752+0000) 2022-09-10T11:11:22.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:21.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:21.883234+0000) 2022-09-10T11:11:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:22.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:22.365015+0000) 2022-09-10T11:11:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:22 smithi110 ceph-mon[97633]: pgmap v12868: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:11:23.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:22 smithi104 ceph-mon[97044]: pgmap v12868: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:11:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:22.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:22.883466+0000) 2022-09-10T11:11:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:23.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:23.365277+0000) 2022-09-10T11:11:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:23.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:23.883676+0000) 2022-09-10T11:11:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:24 smithi104 ceph-mon[97044]: pgmap v12869: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:11:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:24.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:24.365429+0000) 2022-09-10T11:11:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:24 smithi110 ceph-mon[97633]: pgmap v12869: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:11:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:24.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:24.883879+0000) 2022-09-10T11:11:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:25.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:25.365688+0000) 2022-09-10T11:11:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:25.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:25.884156+0000) 2022-09-10T11:11:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:11:26.342+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:11:26.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:26.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:26.365952+0000) 2022-09-10T11:11:26.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:26 smithi110 ceph-mon[97633]: pgmap v12870: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:11:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:26 smithi104 ceph-mon[97044]: pgmap v12870: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:11:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:26.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:26.884344+0000) 2022-09-10T11:11:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:27.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:27.366169+0000) 2022-09-10T11:11:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:27.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:27.884538+0000) 2022-09-10T11:11:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:28.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:28.366411+0000) 2022-09-10T11:11:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:28 smithi110 ceph-mon[97633]: pgmap v12871: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:28 smithi104 ceph-mon[97044]: pgmap v12871: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:28.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:28.884780+0000) 2022-09-10T11:11:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:29.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:29.366669+0000) 2022-09-10T11:11:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:29.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:29.884985+0000) 2022-09-10T11:11:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:30.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:30.366916+0000) 2022-09-10T11:11:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:30 smithi110 ceph-mon[97633]: pgmap v12872: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:30 smithi104 ceph-mon[97044]: pgmap v12872: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:30.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:30.885179+0000) 2022-09-10T11:11:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:31.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:31.367068+0000) 2022-09-10T11:11:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:31.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:31.885395+0000) 2022-09-10T11:11:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:32.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:32.367241+0000) 2022-09-10T11:11:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:32 smithi110 ceph-mon[97633]: pgmap v12873: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:32 smithi104 ceph-mon[97044]: pgmap v12873: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:32.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:32.885595+0000) 2022-09-10T11:11:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:33.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:33.367379+0000) 2022-09-10T11:11:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:33.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:33.885832+0000) 2022-09-10T11:11:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:34 smithi104 ceph-mon[97044]: pgmap v12874: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:34.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:34.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:34.367515+0000) 2022-09-10T11:11:34.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:34 smithi110 ceph-mon[97633]: pgmap v12874: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:34.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:34.886023+0000) 2022-09-10T11:11:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:35.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:35.367704+0000) 2022-09-10T11:11:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:35.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:35.886199+0000) 2022-09-10T11:11:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:36.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:36.367937+0000) 2022-09-10T11:11:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:36 smithi110 ceph-mon[97633]: pgmap v12875: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:36 smithi104 ceph-mon[97044]: pgmap v12875: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:36.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:36.886475+0000) 2022-09-10T11:11:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:37.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:37.368202+0000) 2022-09-10T11:11:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:37.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:37.886684+0000) 2022-09-10T11:11:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:38.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:38.368460+0000) 2022-09-10T11:11:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:38 smithi110 ceph-mon[97633]: pgmap v12876: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:11:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:38 smithi104 ceph-mon[97044]: pgmap v12876: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:11:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:38.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:38.886951+0000) 2022-09-10T11:11:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:39.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:39.368603+0000) 2022-09-10T11:11:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:39.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:39.887090+0000) 2022-09-10T11:11:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:40.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:40.368785+0000) 2022-09-10T11:11:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:40 smithi110 ceph-mon[97633]: pgmap v12877: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:40 smithi104 ceph-mon[97044]: pgmap v12877: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:40.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:40.887298+0000) 2022-09-10T11:11:41.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:41.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:41.368996+0000) 2022-09-10T11:11:41.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:11:41.357+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:11:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:41.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:41.887520+0000) 2022-09-10T11:11:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:42.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:42.369242+0000) 2022-09-10T11:11:42.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:42 smithi110 ceph-mon[97633]: pgmap v12878: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:11:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:42 smithi104 ceph-mon[97044]: pgmap v12878: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:11:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:42.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:42.887751+0000) 2022-09-10T11:11:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:43.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:43.369475+0000) 2022-09-10T11:11:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:43.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:43.887900+0000) 2022-09-10T11:11:44.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:44.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:44.369623+0000) 2022-09-10T11:11:44.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:44 smithi104 ceph-mon[97044]: pgmap v12879: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:11:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:44 smithi110 ceph-mon[97633]: pgmap v12879: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:11:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:44.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:44.888188+0000) 2022-09-10T11:11:45.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:45.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:45.369862+0000) 2022-09-10T11:11:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:11:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:11:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:11:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:11:46.056 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:11:46.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:11:46.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:11:46.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:11:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:45.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:45.888382+0000) 2022-09-10T11:11:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:46.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:46.370129+0000) 2022-09-10T11:11:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:46 smithi110 ceph-mon[97633]: pgmap v12880: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:46 smithi104 ceph-mon[97044]: pgmap v12880: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:46.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:46.888591+0000) 2022-09-10T11:11:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:47.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:47.370309+0000) 2022-09-10T11:11:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:47.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:47.888844+0000) 2022-09-10T11:11:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:48.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:48.370465+0000) 2022-09-10T11:11:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:48 smithi110 ceph-mon[97633]: pgmap v12881: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:48 smithi104 ceph-mon[97044]: pgmap v12881: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:48.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:48.889081+0000) 2022-09-10T11:11:49.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:49.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:49.370673+0000) 2022-09-10T11:11:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:49.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:49.889363+0000) 2022-09-10T11:11:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:50.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:50.370906+0000) 2022-09-10T11:11:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:50 smithi110 ceph-mon[97633]: pgmap v12882: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:50 smithi104 ceph-mon[97044]: pgmap v12882: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:50.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:50.889602+0000) 2022-09-10T11:11:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:51.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:51.371178+0000) 2022-09-10T11:11:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:51.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:51.889869+0000) 2022-09-10T11:11:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:52.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:52.371441+0000) 2022-09-10T11:11:52.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:52 smithi110 ceph-mon[97633]: pgmap v12883: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:52 smithi104 ceph-mon[97044]: pgmap v12883: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:53.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:52.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:52.890209+0000) 2022-09-10T11:11:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:53.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:53.371612+0000) 2022-09-10T11:11:54.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:53.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:53.890477+0000) 2022-09-10T11:11:54.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:54 smithi104 ceph-mon[97044]: pgmap v12884: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:54.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:54.371801+0000) 2022-09-10T11:11:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:54 smithi110 ceph-mon[97633]: pgmap v12884: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:54.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:54.890745+0000) 2022-09-10T11:11:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:55.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:55.372032+0000) 2022-09-10T11:11:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:55.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:55.890933+0000) 2022-09-10T11:11:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:11:56.363+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:11:56.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:56.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:56.372260+0000) 2022-09-10T11:11:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:56 smithi110 ceph-mon[97633]: pgmap v12885: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:56 smithi104 ceph-mon[97044]: pgmap v12885: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:11:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:56.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:56.891177+0000) 2022-09-10T11:11:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:57.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:57.372493+0000) 2022-09-10T11:11:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:57.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:57.891336+0000) 2022-09-10T11:11:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:58.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:58.372685+0000) 2022-09-10T11:11:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:11:58 smithi110 ceph-mon[97633]: pgmap v12886: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:11:58 smithi104 ceph-mon[97044]: pgmap v12886: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:11:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:58.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:58.891525+0000) 2022-09-10T11:11:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:11:59.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:59.372920+0000) 2022-09-10T11:12:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:11:59.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:11:59.891754+0000) 2022-09-10T11:12:00.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:00.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:00.373192+0000) 2022-09-10T11:12:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:00 smithi110 ceph-mon[97633]: pgmap v12887: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:00 smithi104 ceph-mon[97044]: pgmap v12887: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:00.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:00.891925+0000) 2022-09-10T11:12:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:01.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:01.373361+0000) 2022-09-10T11:12:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:01.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:01.892101+0000) 2022-09-10T11:12:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:02.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:02.373550+0000) 2022-09-10T11:12:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:02 smithi110 ceph-mon[97633]: pgmap v12888: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:03.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:02 smithi104 ceph-mon[97044]: pgmap v12888: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:02.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:02.892316+0000) 2022-09-10T11:12:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:03.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:03.373782+0000) 2022-09-10T11:12:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:03.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:03.892558+0000) 2022-09-10T11:12:04.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:04 smithi104 ceph-mon[97044]: pgmap v12889: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:04.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:04.373981+0000) 2022-09-10T11:12:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:04 smithi110 ceph-mon[97633]: pgmap v12889: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:04.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:04.892777+0000) 2022-09-10T11:12:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:05.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:05.374179+0000) 2022-09-10T11:12:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:05.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:05.893021+0000) 2022-09-10T11:12:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:06.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:06.374392+0000) 2022-09-10T11:12:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:06 smithi110 ceph-mon[97633]: pgmap v12890: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:12:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:06 smithi104 ceph-mon[97044]: pgmap v12890: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:12:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:06.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:06.893220+0000) 2022-09-10T11:12:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:07.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:07.374644+0000) 2022-09-10T11:12:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:07.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:07.893400+0000) 2022-09-10T11:12:08.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:08.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:08.374855+0000) 2022-09-10T11:12:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:08 smithi110 ceph-mon[97633]: pgmap v12891: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:12:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:08 smithi110 ceph-mon[97633]: osdmap e886: 12 total, 12 up, 12 in 2022-09-10T11:12:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:08 smithi104 ceph-mon[97044]: pgmap v12891: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:12:09.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:08 smithi104 ceph-mon[97044]: osdmap e886: 12 total, 12 up, 12 in 2022-09-10T11:12:09.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:08.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:08.893623+0000) 2022-09-10T11:12:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:09.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:09.375040+0000) 2022-09-10T11:12:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:09.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:09.893858+0000) 2022-09-10T11:12:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:10.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:10.375238+0000) 2022-09-10T11:12:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:10 smithi110 ceph-mon[97633]: pgmap v12893: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:12:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:10 smithi104 ceph-mon[97044]: pgmap v12893: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:12:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:10.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:10.894074+0000) 2022-09-10T11:12:11.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:11.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:11.375407+0000) 2022-09-10T11:12:11.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:12:11.375+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:12:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:11.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:11.894250+0000) 2022-09-10T11:12:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:12.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:12.375688+0000) 2022-09-10T11:12:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:12 smithi110 ceph-mon[97633]: pgmap v12894: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:12:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:12 smithi104 ceph-mon[97044]: pgmap v12894: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:12:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:12.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:12.894524+0000) 2022-09-10T11:12:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:13.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:13.375933+0000) 2022-09-10T11:12:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:13.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:13.894746+0000) 2022-09-10T11:12:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:14 smithi104 ceph-mon[97044]: osdmap e887: 12 total, 12 up, 12 in 2022-09-10T11:12:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:14 smithi104 ceph-mon[97044]: pgmap v12896: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:12:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:14.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:14.376181+0000) 2022-09-10T11:12:14.915 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:14 smithi110 ceph-mon[97633]: osdmap e887: 12 total, 12 up, 12 in 2022-09-10T11:12:14.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:14 smithi110 ceph-mon[97633]: pgmap v12896: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:12:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:14.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:14.895007+0000) 2022-09-10T11:12:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:15.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:15.376342+0000) 2022-09-10T11:12:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:15.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:15.895253+0000) 2022-09-10T11:12:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:16.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:16.376532+0000) 2022-09-10T11:12:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:16 smithi110 ceph-mon[97633]: pgmap v12897: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:12:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:16 smithi104 ceph-mon[97044]: pgmap v12897: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:12:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:16.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:16.895444+0000) 2022-09-10T11:12:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:17.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:17.376771+0000) 2022-09-10T11:12:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:17.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:17.895683+0000) 2022-09-10T11:12:18.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:18.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:18.376983+0000) 2022-09-10T11:12:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:18 smithi110 ceph-mon[97633]: pgmap v12898: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:12:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:18 smithi104 ceph-mon[97044]: pgmap v12898: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:12:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:18.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:18.895918+0000) 2022-09-10T11:12:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:19.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:19.377229+0000) 2022-09-10T11:12:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:19.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:19.896175+0000) 2022-09-10T11:12:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:20.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:20.377392+0000) 2022-09-10T11:12:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:20 smithi110 ceph-mon[97633]: pgmap v12899: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:12:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:20 smithi104 ceph-mon[97044]: pgmap v12899: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:12:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:20.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:20.896419+0000) 2022-09-10T11:12:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:21.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:21.377647+0000) 2022-09-10T11:12:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:21.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:21.896570+0000) 2022-09-10T11:12:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:22.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:22.377853+0000) 2022-09-10T11:12:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:22 smithi110 ceph-mon[97633]: pgmap v12900: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:12:23.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:22 smithi104 ceph-mon[97044]: pgmap v12900: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:12:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:22.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:22.896789+0000) 2022-09-10T11:12:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:23.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:23.378066+0000) 2022-09-10T11:12:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:23.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:23.897058+0000) 2022-09-10T11:12:24.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:24 smithi104 ceph-mon[97044]: pgmap v12901: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:12:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:24.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:24.378295+0000) 2022-09-10T11:12:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:24 smithi110 ceph-mon[97633]: pgmap v12901: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:12:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:24.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:24.897313+0000) 2022-09-10T11:12:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:25.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:25.378517+0000) 2022-09-10T11:12:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:25.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:25.897535+0000) 2022-09-10T11:12:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:26.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:26.378693+0000) 2022-09-10T11:12:26.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:12:26.390+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:12:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:26 smithi110 ceph-mon[97633]: pgmap v12902: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:12:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:26 smithi104 ceph-mon[97044]: pgmap v12902: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:12:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:26.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:26.897695+0000) 2022-09-10T11:12:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:27.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:27.378936+0000) 2022-09-10T11:12:28.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:27.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:27.897994+0000) 2022-09-10T11:12:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:28.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:28.379154+0000) 2022-09-10T11:12:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:28 smithi110 ceph-mon[97633]: pgmap v12903: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:29.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:28 smithi104 ceph-mon[97044]: pgmap v12903: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:28.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:28.898235+0000) 2022-09-10T11:12:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:29.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:29.379364+0000) 2022-09-10T11:12:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:29.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:29.898449+0000) 2022-09-10T11:12:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:30.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:30.379596+0000) 2022-09-10T11:12:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:30 smithi110 ceph-mon[97633]: pgmap v12904: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:31.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:30 smithi104 ceph-mon[97044]: pgmap v12904: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:30.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:30.898646+0000) 2022-09-10T11:12:31.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:31.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:31.379850+0000) 2022-09-10T11:12:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:31.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:31.898902+0000) 2022-09-10T11:12:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:32.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:32.380031+0000) 2022-09-10T11:12:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:32 smithi110 ceph-mon[97633]: pgmap v12905: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:32 smithi104 ceph-mon[97044]: pgmap v12905: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:32.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:32.899160+0000) 2022-09-10T11:12:33.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:33.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:33.380260+0000) 2022-09-10T11:12:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:33.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:33.899421+0000) 2022-09-10T11:12:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:34 smithi104 ceph-mon[97044]: pgmap v12906: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:34.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:34.380457+0000) 2022-09-10T11:12:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:34 smithi110 ceph-mon[97633]: pgmap v12906: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:34.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:34.899585+0000) 2022-09-10T11:12:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:35.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:35.380687+0000) 2022-09-10T11:12:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:35.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:35.899824+0000) 2022-09-10T11:12:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:36.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:36.380857+0000) 2022-09-10T11:12:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:36 smithi110 ceph-mon[97633]: pgmap v12907: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:36 smithi104 ceph-mon[97044]: pgmap v12907: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:37.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:36.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:36.900123+0000) 2022-09-10T11:12:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:37.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:37.381128+0000) 2022-09-10T11:12:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:37.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:37.900319+0000) 2022-09-10T11:12:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:38.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:38.381345+0000) 2022-09-10T11:12:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:38 smithi110 ceph-mon[97633]: pgmap v12908: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:38 smithi104 ceph-mon[97044]: pgmap v12908: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:38.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:38.900499+0000) 2022-09-10T11:12:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:39.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:39.381532+0000) 2022-09-10T11:12:40.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:39.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:39.900736+0000) 2022-09-10T11:12:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:40.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:40.381750+0000) 2022-09-10T11:12:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:40 smithi110 ceph-mon[97633]: pgmap v12909: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:40 smithi104 ceph-mon[97044]: pgmap v12909: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:40.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:40.900975+0000) 2022-09-10T11:12:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:41.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:41.381998+0000) 2022-09-10T11:12:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:12:41.400+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:12:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:41.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:41.901214+0000) 2022-09-10T11:12:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:42.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:42.382194+0000) 2022-09-10T11:12:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:42 smithi110 ceph-mon[97633]: pgmap v12910: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:42 smithi104 ceph-mon[97044]: pgmap v12910: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:42.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:42.901479+0000) 2022-09-10T11:12:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:43.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:43.382320+0000) 2022-09-10T11:12:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:43.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:43.901706+0000) 2022-09-10T11:12:44.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:44 smithi104 ceph-mon[97044]: pgmap v12911: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:44.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:44.382449+0000) 2022-09-10T11:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:44 smithi110 ceph-mon[97633]: pgmap v12911: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:44.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:44.901922+0000) 2022-09-10T11:12:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:45.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:45.382619+0000) 2022-09-10T11:12:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:12:45.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:12:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:12:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:12:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:12:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:12:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:12:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:12:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:45.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:45.902146+0000) 2022-09-10T11:12:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:46.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:46.382766+0000) 2022-09-10T11:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:46 smithi110 ceph-mon[97633]: pgmap v12912: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:46 smithi104 ceph-mon[97044]: pgmap v12912: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:47.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:46.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:46.902356+0000) 2022-09-10T11:12:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:47.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:47.382895+0000) 2022-09-10T11:12:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:47.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:47.902581+0000) 2022-09-10T11:12:48.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:48.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:48.383188+0000) 2022-09-10T11:12:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:48 smithi110 ceph-mon[97633]: pgmap v12913: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:49.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:48 smithi104 ceph-mon[97044]: pgmap v12913: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:48.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:48.902809+0000) 2022-09-10T11:12:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:49.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:49.383381+0000) 2022-09-10T11:12:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:49.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:49.903082+0000) 2022-09-10T11:12:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:50.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:50.383557+0000) 2022-09-10T11:12:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:50 smithi110 ceph-mon[97633]: pgmap v12914: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:50 smithi104 ceph-mon[97044]: pgmap v12914: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:51.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:50.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:50.903282+0000) 2022-09-10T11:12:51.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:51.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:51.383767+0000) 2022-09-10T11:12:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:51.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:51.903504+0000) 2022-09-10T11:12:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:52.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:52.383976+0000) 2022-09-10T11:12:52.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:52 smithi110 ceph-mon[97633]: pgmap v12915: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:52 smithi104 ceph-mon[97044]: pgmap v12915: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:52.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:52.903772+0000) 2022-09-10T11:12:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:53.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:53.384268+0000) 2022-09-10T11:12:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:53.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:53.904005+0000) 2022-09-10T11:12:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:54 smithi104 ceph-mon[97044]: pgmap v12916: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:54.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:54.384442+0000) 2022-09-10T11:12:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:54 smithi110 ceph-mon[97633]: pgmap v12916: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:54.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:54.904205+0000) 2022-09-10T11:12:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:55.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:55.384708+0000) 2022-09-10T11:12:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:55.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:55.904435+0000) 2022-09-10T11:12:56.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:56.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:56.384978+0000) 2022-09-10T11:12:56.813 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:12:56.416+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:12:56.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:56 smithi110 ceph-mon[97633]: pgmap v12917: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:56 smithi104 ceph-mon[97044]: pgmap v12917: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:12:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:56.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:56.904633+0000) 2022-09-10T11:12:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:57.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:57.385224+0000) 2022-09-10T11:12:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:57.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:57.904809+0000) 2022-09-10T11:12:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:58.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:58.385386+0000) 2022-09-10T11:12:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:12:58 smithi110 ceph-mon[97633]: pgmap v12918: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:12:58 smithi104 ceph-mon[97044]: pgmap v12918: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:12:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:58.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:58.905015+0000) 2022-09-10T11:12:59.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:12:59.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:59.385519+0000) 2022-09-10T11:13:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:12:59.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:12:59.905217+0000) 2022-09-10T11:13:00.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:00.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:00.385692+0000) 2022-09-10T11:13:00.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:00 smithi110 ceph-mon[97633]: pgmap v12919: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:00 smithi104 ceph-mon[97044]: pgmap v12919: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:01.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:00.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:00.905357+0000) 2022-09-10T11:13:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:01.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:01.385903+0000) 2022-09-10T11:13:02.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:01.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:01.905524+0000) 2022-09-10T11:13:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:02.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:02.386086+0000) 2022-09-10T11:13:02.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:02 smithi110 ceph-mon[97633]: pgmap v12920: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:03.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:02 smithi104 ceph-mon[97044]: pgmap v12920: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:02.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:02.905751+0000) 2022-09-10T11:13:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:03.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:03.386393+0000) 2022-09-10T11:13:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:03.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:03.905988+0000) 2022-09-10T11:13:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:04 smithi104 ceph-mon[97044]: pgmap v12921: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:04.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:04.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:04.386615+0000) 2022-09-10T11:13:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:04 smithi110 ceph-mon[97633]: pgmap v12921: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:05.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:04.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:04.906231+0000) 2022-09-10T11:13:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:05.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:05.386845+0000) 2022-09-10T11:13:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:05.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:05.906465+0000) 2022-09-10T11:13:06.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:06.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:06.386999+0000) 2022-09-10T11:13:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:06 smithi110 ceph-mon[97633]: pgmap v12922: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:13:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:06 smithi104 ceph-mon[97044]: pgmap v12922: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:13:07.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:06.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:06.906682+0000) 2022-09-10T11:13:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:07.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:07.387183+0000) 2022-09-10T11:13:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:07.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:07.906909+0000) 2022-09-10T11:13:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:08.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:08.387455+0000) 2022-09-10T11:13:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:08 smithi110 ceph-mon[97633]: pgmap v12923: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:13:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:08 smithi110 ceph-mon[97633]: osdmap e888: 12 total, 12 up, 12 in 2022-09-10T11:13:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:08 smithi104 ceph-mon[97044]: pgmap v12923: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:13:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:08 smithi104 ceph-mon[97044]: osdmap e888: 12 total, 12 up, 12 in 2022-09-10T11:13:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:08.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:08.907199+0000) 2022-09-10T11:13:09.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:09.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:09.387648+0000) 2022-09-10T11:13:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:09.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:09.907445+0000) 2022-09-10T11:13:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:10.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:10.387847+0000) 2022-09-10T11:13:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:10 smithi110 ceph-mon[97633]: pgmap v12925: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:13:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:10 smithi104 ceph-mon[97044]: pgmap v12925: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:13:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:10.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:10.907680+0000) 2022-09-10T11:13:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:11.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:11.387988+0000) 2022-09-10T11:13:11.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:13:11.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:13:12.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:11.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:11.907900+0000) 2022-09-10T11:13:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:12.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:12.388259+0000) 2022-09-10T11:13:12.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:12 smithi110 ceph-mon[97633]: pgmap v12926: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:13:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:12 smithi104 ceph-mon[97044]: pgmap v12926: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:13:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:12.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:12.908194+0000) 2022-09-10T11:13:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:13.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:13.388487+0000) 2022-09-10T11:13:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:13.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:13.908342+0000) 2022-09-10T11:13:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:14 smithi104 ceph-mon[97044]: osdmap e889: 12 total, 12 up, 12 in 2022-09-10T11:13:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:14 smithi104 ceph-mon[97044]: pgmap v12928: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:13:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:14.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:14.388638+0000) 2022-09-10T11:13:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:14 smithi110 ceph-mon[97633]: osdmap e889: 12 total, 12 up, 12 in 2022-09-10T11:13:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:14 smithi110 ceph-mon[97633]: pgmap v12928: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:13:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:14.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:14.908558+0000) 2022-09-10T11:13:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:15.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:15.388894+0000) 2022-09-10T11:13:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:15.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:15.908798+0000) 2022-09-10T11:13:16.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:16.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:16.389126+0000) 2022-09-10T11:13:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:16 smithi110 ceph-mon[97633]: pgmap v12929: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:13:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:16 smithi104 ceph-mon[97044]: pgmap v12929: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:13:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:16.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:16.909044+0000) 2022-09-10T11:13:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:17.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:17.389310+0000) 2022-09-10T11:13:18.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:17.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:17.909306+0000) 2022-09-10T11:13:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:18.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:18.389512+0000) 2022-09-10T11:13:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:18 smithi110 ceph-mon[97633]: pgmap v12930: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:13:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:18 smithi104 ceph-mon[97044]: pgmap v12930: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:13:19.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:18.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:18.909463+0000) 2022-09-10T11:13:19.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:19.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:19.389647+0000) 2022-09-10T11:13:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:19.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:19.909677+0000) 2022-09-10T11:13:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:20.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:20.389833+0000) 2022-09-10T11:13:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:20 smithi110 ceph-mon[97633]: pgmap v12931: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:13:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:20 smithi104 ceph-mon[97044]: pgmap v12931: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:13:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:20.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:20.909953+0000) 2022-09-10T11:13:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:21.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:21.389969+0000) 2022-09-10T11:13:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:21.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:21.910204+0000) 2022-09-10T11:13:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:22.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:22.390219+0000) 2022-09-10T11:13:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:22 smithi110 ceph-mon[97633]: pgmap v12932: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:13:23.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:22 smithi104 ceph-mon[97044]: pgmap v12932: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:13:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:22.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:22.910410+0000) 2022-09-10T11:13:23.819 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:23.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:23.390497+0000) 2022-09-10T11:13:24.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:23.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:23.910639+0000) 2022-09-10T11:13:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:24 smithi104 ceph-mon[97044]: pgmap v12933: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:13:24.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:24.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:24.390683+0000) 2022-09-10T11:13:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:24 smithi110 ceph-mon[97633]: pgmap v12933: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:13:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:24.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:24.910863+0000) 2022-09-10T11:13:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:25.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:25.390921+0000) 2022-09-10T11:13:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:25.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:25.911051+0000) 2022-09-10T11:13:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:26.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:26.391216+0000) 2022-09-10T11:13:26.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:13:26.443+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:13:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:26 smithi110 ceph-mon[97633]: pgmap v12934: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:13:27.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:26 smithi104 ceph-mon[97044]: pgmap v12934: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:13:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:26.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:26.911273+0000) 2022-09-10T11:13:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:27.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:27.391444+0000) 2022-09-10T11:13:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:27.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:27.911470+0000) 2022-09-10T11:13:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:28.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:28.391627+0000) 2022-09-10T11:13:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:28 smithi110 ceph-mon[97633]: pgmap v12935: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:28 smithi104 ceph-mon[97044]: pgmap v12935: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:28.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:28.911658+0000) 2022-09-10T11:13:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:29.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:29.391837+0000) 2022-09-10T11:13:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:29.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:29.911884+0000) 2022-09-10T11:13:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:30.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:30.392128+0000) 2022-09-10T11:13:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:30 smithi110 ceph-mon[97633]: pgmap v12936: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:30 smithi104 ceph-mon[97044]: pgmap v12936: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:30.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:30.912144+0000) 2022-09-10T11:13:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:31.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:31.392290+0000) 2022-09-10T11:13:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:31.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:31.912388+0000) 2022-09-10T11:13:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:32.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:32.392483+0000) 2022-09-10T11:13:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:32 smithi110 ceph-mon[97633]: pgmap v12937: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:32 smithi104 ceph-mon[97044]: pgmap v12937: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:32.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:32.912613+0000) 2022-09-10T11:13:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:33.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:33.392681+0000) 2022-09-10T11:13:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:33.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:33.912830+0000) 2022-09-10T11:13:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:34 smithi104 ceph-mon[97044]: pgmap v12938: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:34.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:34.392854+0000) 2022-09-10T11:13:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:34 smithi110 ceph-mon[97633]: pgmap v12938: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:34.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:34.912995+0000) 2022-09-10T11:13:35.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:35.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:35.393054+0000) 2022-09-10T11:13:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:35.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:35.913229+0000) 2022-09-10T11:13:36.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:36.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:36.393294+0000) 2022-09-10T11:13:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:36 smithi110 ceph-mon[97633]: pgmap v12939: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:36 smithi104 ceph-mon[97044]: pgmap v12939: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:36.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:36.913444+0000) 2022-09-10T11:13:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:37.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:37.393539+0000) 2022-09-10T11:13:38.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:37.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:37.913648+0000) 2022-09-10T11:13:38.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:38.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:38.393760+0000) 2022-09-10T11:13:38.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:38 smithi110 ceph-mon[97633]: pgmap v12940: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:38 smithi104 ceph-mon[97044]: pgmap v12940: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:38.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:38.913857+0000) 2022-09-10T11:13:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:39.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:39.393937+0000) 2022-09-10T11:13:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:39.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:39.914082+0000) 2022-09-10T11:13:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:40.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:40.394183+0000) 2022-09-10T11:13:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:40 smithi110 ceph-mon[97633]: pgmap v12941: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:40 smithi104 ceph-mon[97044]: pgmap v12941: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:40.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:40.914316+0000) 2022-09-10T11:13:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:41.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:41.394435+0000) 2022-09-10T11:13:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:13:41.448+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:13:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:41.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:41.914504+0000) 2022-09-10T11:13:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:42.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:42.394699+0000) 2022-09-10T11:13:42.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:42 smithi110 ceph-mon[97633]: pgmap v12942: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:42 smithi104 ceph-mon[97044]: pgmap v12942: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:42.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:42.914739+0000) 2022-09-10T11:13:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:43.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:43.394882+0000) 2022-09-10T11:13:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:43.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:43.914951+0000) 2022-09-10T11:13:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:44 smithi104 ceph-mon[97044]: pgmap v12943: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:44.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:44.395119+0000) 2022-09-10T11:13:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:44 smithi110 ceph-mon[97633]: pgmap v12943: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:44.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:44.915190+0000) 2022-09-10T11:13:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:45.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:45.395309+0000) 2022-09-10T11:13:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:13:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:13:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:13:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:13:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:13:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:13:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:13:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:13:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:45.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:45.915357+0000) 2022-09-10T11:13:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:46.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:46.395482+0000) 2022-09-10T11:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:46 smithi110 ceph-mon[97633]: pgmap v12944: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:46 smithi104 ceph-mon[97044]: pgmap v12944: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:47.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:46.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:46.915562+0000) 2022-09-10T11:13:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:47.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:47.395678+0000) 2022-09-10T11:13:48.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:47.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:47.915834+0000) 2022-09-10T11:13:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:48.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:48.395925+0000) 2022-09-10T11:13:48.918 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:48 smithi110 ceph-mon[97633]: pgmap v12945: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:13:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:48 smithi104 ceph-mon[97044]: pgmap v12945: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:13:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:48.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:48.916045+0000) 2022-09-10T11:13:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:49.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:49.396193+0000) 2022-09-10T11:13:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:49.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:49.916273+0000) 2022-09-10T11:13:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:50.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:50.396381+0000) 2022-09-10T11:13:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:50 smithi110 ceph-mon[97633]: pgmap v12946: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:50 smithi104 ceph-mon[97044]: pgmap v12946: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:50.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:50.916500+0000) 2022-09-10T11:13:51.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:51.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:51.396608+0000) 2022-09-10T11:13:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:51.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:51.916687+0000) 2022-09-10T11:13:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:52.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:52.396816+0000) 2022-09-10T11:13:52.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:52 smithi110 ceph-mon[97633]: pgmap v12947: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:13:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:52 smithi104 ceph-mon[97044]: pgmap v12947: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:13:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:52.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:52.916919+0000) 2022-09-10T11:13:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:53.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:53.397059+0000) 2022-09-10T11:13:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:53.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:53.917195+0000) 2022-09-10T11:13:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:54 smithi104 ceph-mon[97044]: pgmap v12948: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:13:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:54.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:54.397263+0000) 2022-09-10T11:13:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:54 smithi110 ceph-mon[97633]: pgmap v12948: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:13:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:54.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:54.917414+0000) 2022-09-10T11:13:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:55.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:55.397450+0000) 2022-09-10T11:13:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:55.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:55.917659+0000) 2022-09-10T11:13:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:56.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:56.397623+0000) 2022-09-10T11:13:56.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:13:56.462+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:13:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:56 smithi110 ceph-mon[97633]: pgmap v12949: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:56 smithi104 ceph-mon[97044]: pgmap v12949: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:13:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:56.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:56.917842+0000) 2022-09-10T11:13:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:57.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:57.397862+0000) 2022-09-10T11:13:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:57.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:57.918047+0000) 2022-09-10T11:13:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:58.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:58.398185+0000) 2022-09-10T11:13:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:13:58 smithi110 ceph-mon[97633]: pgmap v12950: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:13:58 smithi104 ceph-mon[97044]: pgmap v12950: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:13:59.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:58.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:58.918239+0000) 2022-09-10T11:13:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:13:59.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:59.398380+0000) 2022-09-10T11:14:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:13:59.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:13:59.918409+0000) 2022-09-10T11:14:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:00.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:00.398595+0000) 2022-09-10T11:14:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:00 smithi110 ceph-mon[97633]: pgmap v12951: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:00 smithi104 ceph-mon[97044]: pgmap v12951: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:00.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:00.918563+0000) 2022-09-10T11:14:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:01.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:01.398838+0000) 2022-09-10T11:14:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:01.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:01.918687+0000) 2022-09-10T11:14:02.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:02.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:02.399091+0000) 2022-09-10T11:14:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:02 smithi110 ceph-mon[97633]: pgmap v12952: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:02 smithi104 ceph-mon[97044]: pgmap v12952: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:02.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:02.918910+0000) 2022-09-10T11:14:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:03.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:03.399346+0000) 2022-09-10T11:14:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:03.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:03.919219+0000) 2022-09-10T11:14:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:04 smithi104 ceph-mon[97044]: pgmap v12953: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:04.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:04.399484+0000) 2022-09-10T11:14:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:04 smithi110 ceph-mon[97633]: pgmap v12953: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:04.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:04.919466+0000) 2022-09-10T11:14:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:05.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:05.399697+0000) 2022-09-10T11:14:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:05.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:05.919664+0000) 2022-09-10T11:14:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:06.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:06.399946+0000) 2022-09-10T11:14:06.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:06 smithi110 ceph-mon[97633]: pgmap v12954: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:14:07.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:06 smithi104 ceph-mon[97044]: pgmap v12954: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:14:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:06.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:06.919967+0000) 2022-09-10T11:14:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:07.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:07.400167+0000) 2022-09-10T11:14:08.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:07.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:07.920244+0000) 2022-09-10T11:14:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:08.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:08.400325+0000) 2022-09-10T11:14:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:08 smithi110 ceph-mon[97633]: pgmap v12955: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:14:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:08 smithi110 ceph-mon[97633]: osdmap e890: 12 total, 12 up, 12 in 2022-09-10T11:14:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:08 smithi104 ceph-mon[97044]: pgmap v12955: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:14:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:08 smithi104 ceph-mon[97044]: osdmap e890: 12 total, 12 up, 12 in 2022-09-10T11:14:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:08.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:08.920442+0000) 2022-09-10T11:14:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:09.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:09.400536+0000) 2022-09-10T11:14:10.317 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:09.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:09.920709+0000) 2022-09-10T11:14:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:10.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:10.400785+0000) 2022-09-10T11:14:10.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:10 smithi110 ceph-mon[97633]: pgmap v12957: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:14:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:10 smithi104 ceph-mon[97044]: pgmap v12957: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:14:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:10.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:10.920877+0000) 2022-09-10T11:14:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:11.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:11.401012+0000) 2022-09-10T11:14:11.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:14:11.465+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:14:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:11.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:11.921083+0000) 2022-09-10T11:14:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:12.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:12.401240+0000) 2022-09-10T11:14:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:12 smithi110 ceph-mon[97633]: pgmap v12958: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:14:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:12 smithi104 ceph-mon[97044]: pgmap v12958: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:14:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:12.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:12.921311+0000) 2022-09-10T11:14:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:13.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:13.401380+0000) 2022-09-10T11:14:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:13.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:13.921564+0000) 2022-09-10T11:14:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:14 smithi104 ceph-mon[97044]: osdmap e891: 12 total, 12 up, 12 in 2022-09-10T11:14:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:14 smithi104 ceph-mon[97044]: pgmap v12960: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:14:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:14.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:14.401607+0000) 2022-09-10T11:14:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:14 smithi110 ceph-mon[97633]: osdmap e891: 12 total, 12 up, 12 in 2022-09-10T11:14:14.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:14 smithi110 ceph-mon[97633]: pgmap v12960: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:14:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:14.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:14.921768+0000) 2022-09-10T11:14:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:15.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:15.401753+0000) 2022-09-10T11:14:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:15.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:15.921985+0000) 2022-09-10T11:14:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:16.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:16.402015+0000) 2022-09-10T11:14:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:16 smithi110 ceph-mon[97633]: pgmap v12961: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:14:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:16 smithi104 ceph-mon[97044]: pgmap v12961: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:14:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:16.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:16.922185+0000) 2022-09-10T11:14:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:17.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:17.402189+0000) 2022-09-10T11:14:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:17.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:17.922386+0000) 2022-09-10T11:14:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:18.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:18.402438+0000) 2022-09-10T11:14:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:18 smithi110 ceph-mon[97633]: pgmap v12962: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:14:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:18 smithi104 ceph-mon[97044]: pgmap v12962: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:14:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:18.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:18.922637+0000) 2022-09-10T11:14:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:19.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:19.402662+0000) 2022-09-10T11:14:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:19.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:19.922835+0000) 2022-09-10T11:14:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:20.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:20.402900+0000) 2022-09-10T11:14:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:20 smithi110 ceph-mon[97633]: pgmap v12963: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:14:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:20 smithi104 ceph-mon[97044]: pgmap v12963: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:14:21.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:20.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:20.923057+0000) 2022-09-10T11:14:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:21.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:21.403193+0000) 2022-09-10T11:14:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:21.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:21.923336+0000) 2022-09-10T11:14:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:22.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:22.403398+0000) 2022-09-10T11:14:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:22 smithi110 ceph-mon[97633]: pgmap v12964: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:14:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:22 smithi104 ceph-mon[97044]: pgmap v12964: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:14:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:22.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:22.923503+0000) 2022-09-10T11:14:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:23.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:23.403628+0000) 2022-09-10T11:14:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:23.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:23.923775+0000) 2022-09-10T11:14:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:24 smithi104 ceph-mon[97044]: pgmap v12965: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:14:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:24.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:24.403858+0000) 2022-09-10T11:14:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:24 smithi110 ceph-mon[97633]: pgmap v12965: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:14:25.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:24.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:24.923933+0000) 2022-09-10T11:14:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:25.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:25.404065+0000) 2022-09-10T11:14:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:25.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:25.924184+0000) 2022-09-10T11:14:26.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:14:26.481+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:14:26.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:26.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:26.404269+0000) 2022-09-10T11:14:26.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:26 smithi110 ceph-mon[97633]: pgmap v12966: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:14:27.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:26 smithi104 ceph-mon[97044]: pgmap v12966: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:14:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:26.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:26.924428+0000) 2022-09-10T11:14:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:27.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:27.404505+0000) 2022-09-10T11:14:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:27.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:27.924685+0000) 2022-09-10T11:14:28.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:28.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:28.404743+0000) 2022-09-10T11:14:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:28 smithi110 ceph-mon[97633]: pgmap v12967: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:28 smithi104 ceph-mon[97044]: pgmap v12967: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:28.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:28.924931+0000) 2022-09-10T11:14:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:29.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:29.404912+0000) 2022-09-10T11:14:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:29.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:29.925135+0000) 2022-09-10T11:14:30.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:30.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:30.405092+0000) 2022-09-10T11:14:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:30 smithi110 ceph-mon[97633]: pgmap v12968: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:30 smithi104 ceph-mon[97044]: pgmap v12968: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:30.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:30.925306+0000) 2022-09-10T11:14:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:31.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:31.405349+0000) 2022-09-10T11:14:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:31.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:31.925543+0000) 2022-09-10T11:14:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:32.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:32.405559+0000) 2022-09-10T11:14:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:32 smithi110 ceph-mon[97633]: pgmap v12969: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:32 smithi104 ceph-mon[97044]: pgmap v12969: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:32.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:32.925767+0000) 2022-09-10T11:14:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:33.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:33.405793+0000) 2022-09-10T11:14:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:33.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:33.925966+0000) 2022-09-10T11:14:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:34 smithi104 ceph-mon[97044]: pgmap v12970: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:34.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:34.406054+0000) 2022-09-10T11:14:34.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:34 smithi110 ceph-mon[97633]: pgmap v12970: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:35.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:34.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:34.926204+0000) 2022-09-10T11:14:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:35.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:35.406228+0000) 2022-09-10T11:14:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:35.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:35.926440+0000) 2022-09-10T11:14:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:36.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:36.406422+0000) 2022-09-10T11:14:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:36 smithi110 ceph-mon[97633]: pgmap v12971: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:36 smithi104 ceph-mon[97044]: pgmap v12971: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:36.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:36.926630+0000) 2022-09-10T11:14:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:37.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:37.406619+0000) 2022-09-10T11:14:38.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:37.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:37.926817+0000) 2022-09-10T11:14:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:38.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:38.406854+0000) 2022-09-10T11:14:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:38 smithi110 ceph-mon[97633]: pgmap v12972: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:38 smithi104 ceph-mon[97044]: pgmap v12972: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:38.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:38.926983+0000) 2022-09-10T11:14:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:39.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:39.407085+0000) 2022-09-10T11:14:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:39.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:39.927186+0000) 2022-09-10T11:14:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:40.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:40.407359+0000) 2022-09-10T11:14:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:40 smithi110 ceph-mon[97633]: pgmap v12973: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:40 smithi104 ceph-mon[97044]: pgmap v12973: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:40.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:40.927383+0000) 2022-09-10T11:14:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:41.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:41.407536+0000) 2022-09-10T11:14:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:14:41.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:14:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:41.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:41.927601+0000) 2022-09-10T11:14:42.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:42.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:42.407761+0000) 2022-09-10T11:14:42.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:42 smithi110 ceph-mon[97633]: pgmap v12974: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:42 smithi104 ceph-mon[97044]: pgmap v12974: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:43.316 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:42.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:42.927863+0000) 2022-09-10T11:14:43.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:43.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:43.407945+0000) 2022-09-10T11:14:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:43.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:43.928141+0000) 2022-09-10T11:14:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:44 smithi104 ceph-mon[97044]: pgmap v12975: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:44.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:44.408139+0000) 2022-09-10T11:14:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:44 smithi110 ceph-mon[97633]: pgmap v12975: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:44.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:44.928364+0000) 2022-09-10T11:14:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:45.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:45.408349+0000) 2022-09-10T11:14:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:14:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:14:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:14:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:14:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:14:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:14:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:14:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:14:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:45.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:45.928611+0000) 2022-09-10T11:14:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:46.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:46.408535+0000) 2022-09-10T11:14:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:46 smithi110 ceph-mon[97633]: pgmap v12976: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:46 smithi104 ceph-mon[97044]: pgmap v12976: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:46.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:46.928852+0000) 2022-09-10T11:14:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:47.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:47.408774+0000) 2022-09-10T11:14:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:47.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:47.929158+0000) 2022-09-10T11:14:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:48.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:48.409003+0000) 2022-09-10T11:14:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:48 smithi110 ceph-mon[97633]: pgmap v12977: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:48 smithi104 ceph-mon[97044]: pgmap v12977: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:48.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:48.929335+0000) 2022-09-10T11:14:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:49.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:49.409206+0000) 2022-09-10T11:14:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:49.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:49.929555+0000) 2022-09-10T11:14:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:50.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:50.409371+0000) 2022-09-10T11:14:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:50 smithi110 ceph-mon[97633]: pgmap v12978: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:50 smithi104 ceph-mon[97044]: pgmap v12978: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:50.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:50.929724+0000) 2022-09-10T11:14:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:51.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:51.409532+0000) 2022-09-10T11:14:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:51.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:51.929955+0000) 2022-09-10T11:14:52.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:52.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:52.409709+0000) 2022-09-10T11:14:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:52 smithi110 ceph-mon[97633]: pgmap v12979: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:52 smithi104 ceph-mon[97044]: pgmap v12979: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:52.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:52.930200+0000) 2022-09-10T11:14:53.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:53.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:53.409944+0000) 2022-09-10T11:14:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:53.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:53.930364+0000) 2022-09-10T11:14:54.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:54 smithi104 ceph-mon[97044]: pgmap v12980: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:54.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:54.410206+0000) 2022-09-10T11:14:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:54 smithi110 ceph-mon[97633]: pgmap v12980: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:54.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:54.930585+0000) 2022-09-10T11:14:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:55.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:55.410424+0000) 2022-09-10T11:14:56.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:55.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:55.930781+0000) 2022-09-10T11:14:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:56.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:56.410600+0000) 2022-09-10T11:14:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:14:56.493+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:14:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:56 smithi110 ceph-mon[97633]: pgmap v12981: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:56 smithi104 ceph-mon[97044]: pgmap v12981: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:14:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:56.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:56.930974+0000) 2022-09-10T11:14:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:57.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:57.410820+0000) 2022-09-10T11:14:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:57.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:57.931195+0000) 2022-09-10T11:14:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:58.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:58.411070+0000) 2022-09-10T11:14:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:14:58 smithi110 ceph-mon[97633]: pgmap v12982: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:14:58 smithi104 ceph-mon[97044]: pgmap v12982: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:14:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:58.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:58.931424+0000) 2022-09-10T11:14:59.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:14:59.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:59.411277+0000) 2022-09-10T11:15:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:14:59.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:14:59.931627+0000) 2022-09-10T11:15:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:00.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:00.411458+0000) 2022-09-10T11:15:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:00 smithi110 ceph-mon[97633]: pgmap v12983: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:00 smithi104 ceph-mon[97044]: pgmap v12983: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:00.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:00.931836+0000) 2022-09-10T11:15:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:01.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:01.411624+0000) 2022-09-10T11:15:02.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:01.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:01.932076+0000) 2022-09-10T11:15:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:02.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:02.411868+0000) 2022-09-10T11:15:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:02 smithi110 ceph-mon[97633]: pgmap v12984: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:02 smithi104 ceph-mon[97044]: pgmap v12984: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:03.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:02.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:02.932356+0000) 2022-09-10T11:15:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:03.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:03.412172+0000) 2022-09-10T11:15:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:03.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:03.932581+0000) 2022-09-10T11:15:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:04 smithi104 ceph-mon[97044]: pgmap v12985: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:04.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:04.412390+0000) 2022-09-10T11:15:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:04 smithi110 ceph-mon[97633]: pgmap v12985: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:04.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:04.932829+0000) 2022-09-10T11:15:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:05.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:05.412616+0000) 2022-09-10T11:15:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:05.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:05.933129+0000) 2022-09-10T11:15:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:06.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:06.412958+0000) 2022-09-10T11:15:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:06 smithi110 ceph-mon[97633]: pgmap v12986: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:15:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:06 smithi104 ceph-mon[97044]: pgmap v12986: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:15:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:06.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:06.933318+0000) 2022-09-10T11:15:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:07.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:07.413191+0000) 2022-09-10T11:15:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:07.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:07.933510+0000) 2022-09-10T11:15:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:08.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:08.413398+0000) 2022-09-10T11:15:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:08 smithi110 ceph-mon[97633]: pgmap v12987: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:15:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:08 smithi110 ceph-mon[97633]: osdmap e892: 12 total, 12 up, 12 in 2022-09-10T11:15:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:08 smithi104 ceph-mon[97044]: pgmap v12987: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:15:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:08 smithi104 ceph-mon[97044]: osdmap e892: 12 total, 12 up, 12 in 2022-09-10T11:15:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:08.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:08.933806+0000) 2022-09-10T11:15:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:09.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:09.413597+0000) 2022-09-10T11:15:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:09.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:09.934043+0000) 2022-09-10T11:15:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:10.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:10.413820+0000) 2022-09-10T11:15:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:10 smithi110 ceph-mon[97633]: pgmap v12989: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:15:11.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:10 smithi104 ceph-mon[97044]: pgmap v12989: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:15:11.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:10.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:10.934303+0000) 2022-09-10T11:15:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:11.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:11.414026+0000) 2022-09-10T11:15:11.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:15:11.509+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:15:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:11.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:11.934574+0000) 2022-09-10T11:15:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:12.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:12.414252+0000) 2022-09-10T11:15:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:12 smithi110 ceph-mon[97633]: pgmap v12990: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:15:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:12 smithi104 ceph-mon[97044]: pgmap v12990: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:15:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:12.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:12.934841+0000) 2022-09-10T11:15:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:13.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:13.414415+0000) 2022-09-10T11:15:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:13.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:13.935170+0000) 2022-09-10T11:15:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:14 smithi104 ceph-mon[97044]: osdmap e893: 12 total, 12 up, 12 in 2022-09-10T11:15:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:14 smithi104 ceph-mon[97044]: pgmap v12992: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:15:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:14.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:14.414646+0000) 2022-09-10T11:15:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:14 smithi110 ceph-mon[97633]: osdmap e893: 12 total, 12 up, 12 in 2022-09-10T11:15:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:14 smithi110 ceph-mon[97633]: pgmap v12992: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:15:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:14.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:14.935448+0000) 2022-09-10T11:15:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:15.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:15.414889+0000) 2022-09-10T11:15:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:15.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:15.935698+0000) 2022-09-10T11:15:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:16.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:16.415089+0000) 2022-09-10T11:15:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:16 smithi110 ceph-mon[97633]: pgmap v12993: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:15:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:16 smithi104 ceph-mon[97044]: pgmap v12993: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:15:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:16.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:16.935889+0000) 2022-09-10T11:15:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:17.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:17.415283+0000) 2022-09-10T11:15:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:17.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:17.936137+0000) 2022-09-10T11:15:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:18.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:18.415462+0000) 2022-09-10T11:15:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:18 smithi110 ceph-mon[97633]: pgmap v12994: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:15:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:18 smithi104 ceph-mon[97044]: pgmap v12994: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:15:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:18.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:18.936342+0000) 2022-09-10T11:15:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:19.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:19.415712+0000) 2022-09-10T11:15:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:19.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:19.936561+0000) 2022-09-10T11:15:20.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:20.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:20.415901+0000) 2022-09-10T11:15:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:20 smithi110 ceph-mon[97633]: pgmap v12995: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:15:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:20 smithi104 ceph-mon[97044]: pgmap v12995: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:15:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:20.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:20.936762+0000) 2022-09-10T11:15:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:21.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:21.416140+0000) 2022-09-10T11:15:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:21.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:21.936966+0000) 2022-09-10T11:15:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:22.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:22.416258+0000) 2022-09-10T11:15:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:22 smithi110 ceph-mon[97633]: pgmap v12996: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:15:23.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:22 smithi104 ceph-mon[97044]: pgmap v12996: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:15:23.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:22.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:22.937112+0000) 2022-09-10T11:15:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:23.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:23.416476+0000) 2022-09-10T11:15:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:23.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:23.937331+0000) 2022-09-10T11:15:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:24 smithi104 ceph-mon[97044]: pgmap v12997: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:15:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:24.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:24.416733+0000) 2022-09-10T11:15:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:24 smithi110 ceph-mon[97633]: pgmap v12997: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:15:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:24.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:24.937557+0000) 2022-09-10T11:15:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:25.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:25.416986+0000) 2022-09-10T11:15:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:25.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:25.937809+0000) 2022-09-10T11:15:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:26.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:26.417212+0000) 2022-09-10T11:15:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:15:26.524+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:15:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:26 smithi110 ceph-mon[97633]: pgmap v12998: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:15:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:26 smithi104 ceph-mon[97044]: pgmap v12998: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:15:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:26.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:26.938005+0000) 2022-09-10T11:15:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:27.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:27.417442+0000) 2022-09-10T11:15:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:27.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:27.938218+0000) 2022-09-10T11:15:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:28.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:28.417673+0000) 2022-09-10T11:15:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:28 smithi110 ceph-mon[97633]: pgmap v12999: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:28 smithi104 ceph-mon[97044]: pgmap v12999: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:28.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:28.938496+0000) 2022-09-10T11:15:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:29.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:29.417907+0000) 2022-09-10T11:15:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:29.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:29.938766+0000) 2022-09-10T11:15:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:30.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:30.418129+0000) 2022-09-10T11:15:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:30 smithi110 ceph-mon[97633]: pgmap v13000: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:30 smithi104 ceph-mon[97044]: pgmap v13000: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:30.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:30.939009+0000) 2022-09-10T11:15:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:31.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:31.418280+0000) 2022-09-10T11:15:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:31.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:31.939207+0000) 2022-09-10T11:15:32.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:32.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:32.418519+0000) 2022-09-10T11:15:32.915 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:32 smithi110 ceph-mon[97633]: pgmap v13001: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:32 smithi104 ceph-mon[97044]: pgmap v13001: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:32.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:32.939404+0000) 2022-09-10T11:15:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:33.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:33.418728+0000) 2022-09-10T11:15:34.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:33.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:33.939698+0000) 2022-09-10T11:15:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:34 smithi104 ceph-mon[97044]: pgmap v13002: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:34.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:34.418938+0000) 2022-09-10T11:15:34.916 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:34 smithi110 ceph-mon[97633]: pgmap v13002: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:35.314 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:34.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:34.939942+0000) 2022-09-10T11:15:35.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:35.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:35.419199+0000) 2022-09-10T11:15:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:35.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:35.940218+0000) 2022-09-10T11:15:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:36.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:36.419421+0000) 2022-09-10T11:15:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:36 smithi110 ceph-mon[97633]: pgmap v13003: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:36 smithi104 ceph-mon[97044]: pgmap v13003: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:37.316 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:36.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:36.940375+0000) 2022-09-10T11:15:37.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:37.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:37.419598+0000) 2022-09-10T11:15:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:37.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:37.940581+0000) 2022-09-10T11:15:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:38.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:38.419816+0000) 2022-09-10T11:15:38.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:38 smithi110 ceph-mon[97633]: pgmap v13004: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:39.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:38 smithi104 ceph-mon[97044]: pgmap v13004: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:39.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:38.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:38.940761+0000) 2022-09-10T11:15:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:39.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:39.420023+0000) 2022-09-10T11:15:40.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:39.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:39.940923+0000) 2022-09-10T11:15:40.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:40.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:40.420197+0000) 2022-09-10T11:15:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:40 smithi110 ceph-mon[97633]: pgmap v13005: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:41.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:40 smithi104 ceph-mon[97044]: pgmap v13005: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:40.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:40.941165+0000) 2022-09-10T11:15:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:41.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:41.420330+0000) 2022-09-10T11:15:41.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:15:41.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:15:42.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:41.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:41.941406+0000) 2022-09-10T11:15:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:42.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:42.420539+0000) 2022-09-10T11:15:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:42 smithi110 ceph-mon[97633]: pgmap v13006: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:42 smithi104 ceph-mon[97044]: pgmap v13006: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:43.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:42.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:42.941600+0000) 2022-09-10T11:15:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:43.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:43.420829+0000) 2022-09-10T11:15:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:43.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:43.941873+0000) 2022-09-10T11:15:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:44 smithi104 ceph-mon[97044]: pgmap v13007: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:44.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:44.421064+0000) 2022-09-10T11:15:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:44 smithi110 ceph-mon[97633]: pgmap v13007: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:44.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:44.942052+0000) 2022-09-10T11:15:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:45.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:45.421264+0000) 2022-09-10T11:15:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:15:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:15:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:15:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:15:46.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:15:46.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:15:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:45.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:45.942270+0000) 2022-09-10T11:15:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:46.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:46.421507+0000) 2022-09-10T11:15:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:46 smithi110 ceph-mon[97633]: pgmap v13008: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:46 smithi104 ceph-mon[97044]: pgmap v13008: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:46.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:46.942462+0000) 2022-09-10T11:15:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:47.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:47.421714+0000) 2022-09-10T11:15:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:47.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:47.942724+0000) 2022-09-10T11:15:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:48.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:48.421961+0000) 2022-09-10T11:15:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:48 smithi110 ceph-mon[97633]: pgmap v13009: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:48 smithi104 ceph-mon[97044]: pgmap v13009: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:48.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:48.942984+0000) 2022-09-10T11:15:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:49.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:49.422214+0000) 2022-09-10T11:15:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:49.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:49.943230+0000) 2022-09-10T11:15:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:50.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:50.422337+0000) 2022-09-10T11:15:50.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:50 smithi110 ceph-mon[97633]: pgmap v13010: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:50 smithi104 ceph-mon[97044]: pgmap v13010: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:50.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:50.943392+0000) 2022-09-10T11:15:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:51.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:51.422574+0000) 2022-09-10T11:15:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:51.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:51.943621+0000) 2022-09-10T11:15:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:52.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:52.422804+0000) 2022-09-10T11:15:52.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:52 smithi110 ceph-mon[97633]: pgmap v13011: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:52 smithi104 ceph-mon[97044]: pgmap v13011: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:53.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:52.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:52.943892+0000) 2022-09-10T11:15:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:53.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:53.422972+0000) 2022-09-10T11:15:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:53.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:53.944167+0000) 2022-09-10T11:15:54.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:54 smithi104 ceph-mon[97044]: pgmap v13012: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:54.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:54.423201+0000) 2022-09-10T11:15:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:54 smithi110 ceph-mon[97633]: pgmap v13012: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:55.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:54.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:54.944382+0000) 2022-09-10T11:15:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:55.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:55.423362+0000) 2022-09-10T11:15:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:55.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:55.944610+0000) 2022-09-10T11:15:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:56.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:56.423577+0000) 2022-09-10T11:15:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:15:56.547+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:15:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:56 smithi110 ceph-mon[97633]: pgmap v13013: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:57.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:56 smithi104 ceph-mon[97044]: pgmap v13013: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:15:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:56.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:56.944875+0000) 2022-09-10T11:15:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:57.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:57.423822+0000) 2022-09-10T11:15:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:57.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:57.945186+0000) 2022-09-10T11:15:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:58.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:58.424128+0000) 2022-09-10T11:15:58.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:15:58 smithi110 ceph-mon[97633]: pgmap v13014: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:15:58 smithi104 ceph-mon[97044]: pgmap v13014: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:15:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:58.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:58.945446+0000) 2022-09-10T11:15:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:15:59.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:59.424375+0000) 2022-09-10T11:16:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:15:59.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:15:59.945643+0000) 2022-09-10T11:16:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:00.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:00.424581+0000) 2022-09-10T11:16:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:00 smithi110 ceph-mon[97633]: pgmap v13015: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:00 smithi104 ceph-mon[97044]: pgmap v13015: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:01.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:00.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:00.945774+0000) 2022-09-10T11:16:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:01.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:01.424800+0000) 2022-09-10T11:16:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:01.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:01.946009+0000) 2022-09-10T11:16:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:02.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:02.425007+0000) 2022-09-10T11:16:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:02 smithi110 ceph-mon[97633]: pgmap v13016: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:02 smithi104 ceph-mon[97044]: pgmap v13016: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:02.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:02.946300+0000) 2022-09-10T11:16:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:03.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:03.425210+0000) 2022-09-10T11:16:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:03.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:03.946551+0000) 2022-09-10T11:16:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:04 smithi104 ceph-mon[97044]: pgmap v13017: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:04.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:04.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:04.425474+0000) 2022-09-10T11:16:04.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:04 smithi110 ceph-mon[97633]: pgmap v13017: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:04.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:04.946716+0000) 2022-09-10T11:16:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:05.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:05.425662+0000) 2022-09-10T11:16:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:05.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:05.946926+0000) 2022-09-10T11:16:06.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:06.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:06.425882+0000) 2022-09-10T11:16:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:06 smithi110 ceph-mon[97633]: pgmap v13018: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:16:07.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:06 smithi104 ceph-mon[97044]: pgmap v13018: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:16:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:06.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:06.947184+0000) 2022-09-10T11:16:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:07.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:07.426192+0000) 2022-09-10T11:16:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:07.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:07.947438+0000) 2022-09-10T11:16:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:08.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:08.426401+0000) 2022-09-10T11:16:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:08 smithi104 ceph-mon[97044]: pgmap v13019: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:16:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:08 smithi104 ceph-mon[97044]: osdmap e894: 12 total, 12 up, 12 in 2022-09-10T11:16:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:08 smithi110 ceph-mon[97633]: pgmap v13019: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:16:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:08 smithi110 ceph-mon[97633]: osdmap e894: 12 total, 12 up, 12 in 2022-09-10T11:16:09.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:08.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:08.947622+0000) 2022-09-10T11:16:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:09.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:09.426638+0000) 2022-09-10T11:16:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:09.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:09.947825+0000) 2022-09-10T11:16:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:10.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:10.426907+0000) 2022-09-10T11:16:11.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:10 smithi104 ceph-mon[97044]: pgmap v13021: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:16:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:10 smithi110 ceph-mon[97633]: pgmap v13021: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:16:11.314 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:10.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:10.948077+0000) 2022-09-10T11:16:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:11.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:11.427193+0000) 2022-09-10T11:16:11.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:16:11.553+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:16:12.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:11.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:11.948374+0000) 2022-09-10T11:16:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:12.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:12.427414+0000) 2022-09-10T11:16:13.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:12 smithi104 ceph-mon[97044]: pgmap v13022: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:16:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:12 smithi110 ceph-mon[97633]: pgmap v13022: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:16:13.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:12.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:12.948551+0000) 2022-09-10T11:16:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:13.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:13.427657+0000) 2022-09-10T11:16:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:13.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:13.948748+0000) 2022-09-10T11:16:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:14 smithi104 ceph-mon[97044]: osdmap e895: 12 total, 12 up, 12 in 2022-09-10T11:16:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:14 smithi104 ceph-mon[97044]: pgmap v13024: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:16:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:14.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:14.427857+0000) 2022-09-10T11:16:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:14 smithi110 ceph-mon[97633]: osdmap e895: 12 total, 12 up, 12 in 2022-09-10T11:16:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:14 smithi110 ceph-mon[97633]: pgmap v13024: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:16:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:14.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:14.948988+0000) 2022-09-10T11:16:15.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:15.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:15.428091+0000) 2022-09-10T11:16:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:15.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:15.949255+0000) 2022-09-10T11:16:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:16.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:16.428340+0000) 2022-09-10T11:16:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:16 smithi110 ceph-mon[97633]: pgmap v13025: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:16:17.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:16 smithi104 ceph-mon[97044]: pgmap v13025: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:16:17.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:16.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:16.949434+0000) 2022-09-10T11:16:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:17.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:17.428564+0000) 2022-09-10T11:16:18.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:17.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:17.949663+0000) 2022-09-10T11:16:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:18.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:18.428809+0000) 2022-09-10T11:16:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:18 smithi110 ceph-mon[97633]: pgmap v13026: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:16:19.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:18 smithi104 ceph-mon[97044]: pgmap v13026: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:16:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:18.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:18.949897+0000) 2022-09-10T11:16:19.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:19.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:19.429059+0000) 2022-09-10T11:16:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:19.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:19.950197+0000) 2022-09-10T11:16:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:20.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:20.429309+0000) 2022-09-10T11:16:21.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:20 smithi104 ceph-mon[97044]: pgmap v13027: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:16:21.171 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:20 smithi110 ceph-mon[97633]: pgmap v13027: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:16:21.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:20.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:20.950401+0000) 2022-09-10T11:16:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:21.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:21.429475+0000) 2022-09-10T11:16:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:21.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:21.950607+0000) 2022-09-10T11:16:22.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:22.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:22.429673+0000) 2022-09-10T11:16:23.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:22 smithi104 ceph-mon[97044]: pgmap v13028: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:16:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:22 smithi110 ceph-mon[97633]: pgmap v13028: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:16:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:22.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:22.950841+0000) 2022-09-10T11:16:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:23.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:23.429898+0000) 2022-09-10T11:16:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:23.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:23.951119+0000) 2022-09-10T11:16:24.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:24 smithi104 ceph-mon[97044]: pgmap v13029: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:16:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:24.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:24.430175+0000) 2022-09-10T11:16:24.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:24 smithi110 ceph-mon[97633]: pgmap v13029: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:16:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:24.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:24.951352+0000) 2022-09-10T11:16:25.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:25.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:25.430408+0000) 2022-09-10T11:16:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:25.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:25.951545+0000) 2022-09-10T11:16:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:16:26.567+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:16:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:26.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:26.430603+0000) 2022-09-10T11:16:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:26 smithi104 ceph-mon[97044]: pgmap v13030: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:16:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:26 smithi110 ceph-mon[97633]: pgmap v13030: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:16:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:26.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:26.951758+0000) 2022-09-10T11:16:27.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:27.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:27.430763+0000) 2022-09-10T11:16:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:27.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:27.951971+0000) 2022-09-10T11:16:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:28.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:28.430968+0000) 2022-09-10T11:16:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:28 smithi104 ceph-mon[97044]: pgmap v13031: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:28 smithi110 ceph-mon[97633]: pgmap v13031: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:28.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:28.952203+0000) 2022-09-10T11:16:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:29.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:29.431202+0000) 2022-09-10T11:16:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:29.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:29.952425+0000) 2022-09-10T11:16:30.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:30.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:30.431449+0000) 2022-09-10T11:16:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:30 smithi104 ceph-mon[97044]: pgmap v13032: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:30 smithi110 ceph-mon[97633]: pgmap v13032: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:30.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:30.952657+0000) 2022-09-10T11:16:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:31.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:31.431646+0000) 2022-09-10T11:16:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:31.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:31.952901+0000) 2022-09-10T11:16:32.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:32.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:32.431865+0000) 2022-09-10T11:16:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:32 smithi104 ceph-mon[97044]: pgmap v13033: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:33.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:32 smithi110 ceph-mon[97633]: pgmap v13033: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:32.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:32.953147+0000) 2022-09-10T11:16:33.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:33.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:33.432068+0000) 2022-09-10T11:16:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:33.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:33.953338+0000) 2022-09-10T11:16:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:34 smithi104 ceph-mon[97044]: pgmap v13034: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:34.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:34.432295+0000) 2022-09-10T11:16:34.920 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:34 smithi110 ceph-mon[97633]: pgmap v13034: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:34.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:34.953576+0000) 2022-09-10T11:16:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:35.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:35.432542+0000) 2022-09-10T11:16:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:35.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:35.953805+0000) 2022-09-10T11:16:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:36.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:36.432767+0000) 2022-09-10T11:16:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:36 smithi104 ceph-mon[97044]: pgmap v13035: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:36 smithi110 ceph-mon[97633]: pgmap v13035: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:36.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:36.953987+0000) 2022-09-10T11:16:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:37.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:37.432933+0000) 2022-09-10T11:16:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:37.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:37.954244+0000) 2022-09-10T11:16:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:38.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:38.433198+0000) 2022-09-10T11:16:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:38 smithi104 ceph-mon[97044]: pgmap v13036: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:38 smithi110 ceph-mon[97633]: pgmap v13036: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:39.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:38.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:38.954480+0000) 2022-09-10T11:16:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:39.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:39.433454+0000) 2022-09-10T11:16:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:39.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:39.954697+0000) 2022-09-10T11:16:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:40.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:40.433687+0000) 2022-09-10T11:16:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:40 smithi104 ceph-mon[97044]: pgmap v13037: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:40 smithi110 ceph-mon[97633]: pgmap v13037: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:40.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:40.954932+0000) 2022-09-10T11:16:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:41.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:41.433944+0000) 2022-09-10T11:16:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:16:41.580+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:16:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:41.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:41.955174+0000) 2022-09-10T11:16:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:42.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:42.434183+0000) 2022-09-10T11:16:43.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:42 smithi104 ceph-mon[97044]: pgmap v13038: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:42 smithi110 ceph-mon[97633]: pgmap v13038: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:42.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:42.955419+0000) 2022-09-10T11:16:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:43.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:43.434354+0000) 2022-09-10T11:16:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:43.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:43.955615+0000) 2022-09-10T11:16:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:44 smithi104 ceph-mon[97044]: pgmap v13039: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:44.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:44.434527+0000) 2022-09-10T11:16:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:44 smithi110 ceph-mon[97633]: pgmap v13039: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:44.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:44.955805+0000) 2022-09-10T11:16:45.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:45.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:45.434749+0000) 2022-09-10T11:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:16:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:16:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:16:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:16:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:16:46.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:16:46.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:16:46.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:16:46.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:45.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:45.956067+0000) 2022-09-10T11:16:46.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:46.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:46.434911+0000) 2022-09-10T11:16:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:46 smithi110 ceph-mon[97633]: pgmap v13040: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:46 smithi104 ceph-mon[97044]: pgmap v13040: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:46.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:46.956311+0000) 2022-09-10T11:16:47.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:47.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:47.435085+0000) 2022-09-10T11:16:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:47.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:47.956513+0000) 2022-09-10T11:16:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:48.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:48.435297+0000) 2022-09-10T11:16:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:48 smithi104 ceph-mon[97044]: pgmap v13041: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:48 smithi110 ceph-mon[97633]: pgmap v13041: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:49.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:48.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:48.956728+0000) 2022-09-10T11:16:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:49.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:49.435480+0000) 2022-09-10T11:16:50.315 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:49.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:49.956949+0000) 2022-09-10T11:16:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:50.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:50.435681+0000) 2022-09-10T11:16:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:50 smithi104 ceph-mon[97044]: pgmap v13042: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:51.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:50 smithi110 ceph-mon[97633]: pgmap v13042: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:50.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:50.957205+0000) 2022-09-10T11:16:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:51.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:51.435930+0000) 2022-09-10T11:16:52.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:51.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:51.957446+0000) 2022-09-10T11:16:52.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:52.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:52.436202+0000) 2022-09-10T11:16:53.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:52 smithi104 ceph-mon[97044]: pgmap v13043: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:52 smithi110 ceph-mon[97633]: pgmap v13043: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:52.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:52.957662+0000) 2022-09-10T11:16:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:53.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:53.436381+0000) 2022-09-10T11:16:54.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:53.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:53.957877+0000) 2022-09-10T11:16:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:54 smithi104 ceph-mon[97044]: pgmap v13044: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:54.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:54.436620+0000) 2022-09-10T11:16:54.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:54 smithi110 ceph-mon[97633]: pgmap v13044: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:55.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:54.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:54.958074+0000) 2022-09-10T11:16:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:55.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:55.436853+0000) 2022-09-10T11:16:56.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:55.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:55.958286+0000) 2022-09-10T11:16:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:56.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:56.437046+0000) 2022-09-10T11:16:56.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:16:56.595+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:16:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:56 smithi104 ceph-mon[97044]: pgmap v13045: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:56 smithi110 ceph-mon[97633]: pgmap v13045: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:16:57.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:56.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:56.958487+0000) 2022-09-10T11:16:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:57.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:57.437306+0000) 2022-09-10T11:16:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:57.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:57.958651+0000) 2022-09-10T11:16:58.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:58.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:58.437585+0000) 2022-09-10T11:16:59.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:16:58 smithi104 ceph-mon[97044]: pgmap v13046: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:59.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:16:58 smithi110 ceph-mon[97633]: pgmap v13046: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:16:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:58.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:58.958904+0000) 2022-09-10T11:16:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:16:59.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:59.437814+0000) 2022-09-10T11:17:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:16:59.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:16:59.959179+0000) 2022-09-10T11:17:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:00.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:00.438036+0000) 2022-09-10T11:17:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:00 smithi104 ceph-mon[97044]: pgmap v13047: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:00 smithi110 ceph-mon[97633]: pgmap v13047: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:01.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:00.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:00.959445+0000) 2022-09-10T11:17:01.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:01.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:01.438202+0000) 2022-09-10T11:17:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:01.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:01.959659+0000) 2022-09-10T11:17:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:02.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:02.438368+0000) 2022-09-10T11:17:03.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:02 smithi104 ceph-mon[97044]: pgmap v13048: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:02 smithi110 ceph-mon[97633]: pgmap v13048: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:02.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:02.959895+0000) 2022-09-10T11:17:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:03.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:03.438502+0000) 2022-09-10T11:17:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:03.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:03.960187+0000) 2022-09-10T11:17:04.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:04 smithi104 ceph-mon[97044]: pgmap v13049: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:04.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:04.438687+0000) 2022-09-10T11:17:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:04 smithi110 ceph-mon[97633]: pgmap v13049: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:04.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:04.960421+0000) 2022-09-10T11:17:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:05.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:05.438963+0000) 2022-09-10T11:17:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:05.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:05.960694+0000) 2022-09-10T11:17:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:06.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:06.439203+0000) 2022-09-10T11:17:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:06 smithi104 ceph-mon[97044]: pgmap v13050: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:17:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:06 smithi110 ceph-mon[97633]: pgmap v13050: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:17:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:06.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:06.960944+0000) 2022-09-10T11:17:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:07.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:07.439366+0000) 2022-09-10T11:17:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:07.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:07.961205+0000) 2022-09-10T11:17:08.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:08.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:08.439619+0000) 2022-09-10T11:17:09.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:08 smithi104 ceph-mon[97044]: pgmap v13051: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:17:09.055 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:08 smithi104 ceph-mon[97044]: osdmap e896: 12 total, 12 up, 12 in 2022-09-10T11:17:09.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:08 smithi110 ceph-mon[97633]: pgmap v13051: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:17:09.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:08 smithi110 ceph-mon[97633]: osdmap e896: 12 total, 12 up, 12 in 2022-09-10T11:17:09.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:08.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:08.961421+0000) 2022-09-10T11:17:09.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:09.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:09.439818+0000) 2022-09-10T11:17:10.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:09.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:09.961663+0000) 2022-09-10T11:17:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:10.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:10.440036+0000) 2022-09-10T11:17:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:10 smithi104 ceph-mon[97044]: pgmap v13053: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:17:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:10 smithi110 ceph-mon[97633]: pgmap v13053: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:17:11.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:10.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:10.961867+0000) 2022-09-10T11:17:11.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:17:11.611+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:17:11.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:11.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:11.440195+0000) 2022-09-10T11:17:12.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:11.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:11.962045+0000) 2022-09-10T11:17:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:12.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:12.440350+0000) 2022-09-10T11:17:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:12 smithi104 ceph-mon[97044]: pgmap v13054: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:17:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:12 smithi110 ceph-mon[97633]: pgmap v13054: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:17:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:12.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:12.962257+0000) 2022-09-10T11:17:13.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:13.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:13.440569+0000) 2022-09-10T11:17:14.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:13.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:13.962527+0000) 2022-09-10T11:17:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:14 smithi104 ceph-mon[97044]: osdmap e897: 12 total, 12 up, 12 in 2022-09-10T11:17:14.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:14 smithi104 ceph-mon[97044]: pgmap v13056: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:17:14.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:14.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:14.440801+0000) 2022-09-10T11:17:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:14 smithi110 ceph-mon[97633]: osdmap e897: 12 total, 12 up, 12 in 2022-09-10T11:17:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:14 smithi110 ceph-mon[97633]: pgmap v13056: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:17:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:14.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:14.962752+0000) 2022-09-10T11:17:15.817 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:15.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:15.441047+0000) 2022-09-10T11:17:16.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:15.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:15.962940+0000) 2022-09-10T11:17:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:16.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:16.441207+0000) 2022-09-10T11:17:17.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:16 smithi104 ceph-mon[97044]: pgmap v13057: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:17:17.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:16 smithi110 ceph-mon[97633]: pgmap v13057: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:17:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:16.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:16.963251+0000) 2022-09-10T11:17:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:17.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:17.441412+0000) 2022-09-10T11:17:18.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:17.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:17.963535+0000) 2022-09-10T11:17:18.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:18.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:18.441602+0000) 2022-09-10T11:17:19.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:18 smithi104 ceph-mon[97044]: pgmap v13058: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:17:19.177 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:18 smithi110 ceph-mon[97633]: pgmap v13058: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:17:19.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:18.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:18.963703+0000) 2022-09-10T11:17:19.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:19.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:19.441827+0000) 2022-09-10T11:17:20.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:19.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:19.963955+0000) 2022-09-10T11:17:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:20.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:20.442052+0000) 2022-09-10T11:17:21.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:20 smithi104 ceph-mon[97044]: pgmap v13059: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:17:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:20 smithi110 ceph-mon[97633]: pgmap v13059: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:17:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:20.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:20.964207+0000) 2022-09-10T11:17:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:21.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:21.442239+0000) 2022-09-10T11:17:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:21.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:21.964400+0000) 2022-09-10T11:17:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:22.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:22.442415+0000) 2022-09-10T11:17:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:22 smithi104 ceph-mon[97044]: pgmap v13060: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:17:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:22 smithi110 ceph-mon[97633]: pgmap v13060: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:17:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:22.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:22.964665+0000) 2022-09-10T11:17:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:23.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:23.442661+0000) 2022-09-10T11:17:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:23.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:23.964918+0000) 2022-09-10T11:17:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:24 smithi104 ceph-mon[97044]: pgmap v13061: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:17:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:24.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:24.442895+0000) 2022-09-10T11:17:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:24 smithi110 ceph-mon[97633]: pgmap v13061: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:17:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:24.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:24.965197+0000) 2022-09-10T11:17:25.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:25.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:25.443076+0000) 2022-09-10T11:17:26.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:25.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:25.965471+0000) 2022-09-10T11:17:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:26.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:26.443291+0000) 2022-09-10T11:17:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:17:26.624+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:17:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:26 smithi104 ceph-mon[97044]: pgmap v13062: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:17:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:26 smithi110 ceph-mon[97633]: pgmap v13062: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:17:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:26.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:26.965686+0000) 2022-09-10T11:17:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:27.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:27.443524+0000) 2022-09-10T11:17:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:27.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:27.965924+0000) 2022-09-10T11:17:28.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:28.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:28.443752+0000) 2022-09-10T11:17:29.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:28 smithi104 ceph-mon[97044]: pgmap v13063: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:29.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:28 smithi110 ceph-mon[97633]: pgmap v13063: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:28.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:28.966194+0000) 2022-09-10T11:17:29.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:29.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:29.443984+0000) 2022-09-10T11:17:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:29.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:29.966443+0000) 2022-09-10T11:17:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:30.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:30.444212+0000) 2022-09-10T11:17:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:30 smithi104 ceph-mon[97044]: pgmap v13064: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:30 smithi110 ceph-mon[97633]: pgmap v13064: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:30.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:30.966699+0000) 2022-09-10T11:17:31.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:31.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:31.444435+0000) 2022-09-10T11:17:32.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:31.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:31.966958+0000) 2022-09-10T11:17:32.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:32.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:32.444617+0000) 2022-09-10T11:17:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:32 smithi104 ceph-mon[97044]: pgmap v13065: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:33.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:32 smithi110 ceph-mon[97633]: pgmap v13065: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:32.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:32.967232+0000) 2022-09-10T11:17:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:33.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:33.444853+0000) 2022-09-10T11:17:34.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:33.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:33.967417+0000) 2022-09-10T11:17:34.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:34 smithi104 ceph-mon[97044]: pgmap v13066: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:34.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:34.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:34.445114+0000) 2022-09-10T11:17:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:34 smithi110 ceph-mon[97633]: pgmap v13066: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:34.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:34.967609+0000) 2022-09-10T11:17:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:35.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:35.445280+0000) 2022-09-10T11:17:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:35.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:35.967886+0000) 2022-09-10T11:17:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:36.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:36.445456+0000) 2022-09-10T11:17:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:36 smithi104 ceph-mon[97044]: pgmap v13067: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:37.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:36 smithi110 ceph-mon[97633]: pgmap v13067: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:36.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:36.968178+0000) 2022-09-10T11:17:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:37.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:37.445632+0000) 2022-09-10T11:17:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:37.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:37.968422+0000) 2022-09-10T11:17:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:38.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:38.445873+0000) 2022-09-10T11:17:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:38 smithi104 ceph-mon[97044]: pgmap v13068: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:38 smithi110 ceph-mon[97633]: pgmap v13068: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:38.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:38.968667+0000) 2022-09-10T11:17:39.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:39.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:39.446161+0000) 2022-09-10T11:17:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:39.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:39.968826+0000) 2022-09-10T11:17:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:40.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:40.446453+0000) 2022-09-10T11:17:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:40 smithi104 ceph-mon[97044]: pgmap v13069: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:40 smithi110 ceph-mon[97633]: pgmap v13069: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:41.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:40.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:40.969042+0000) 2022-09-10T11:17:41.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:41.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:41.446718+0000) 2022-09-10T11:17:41.846 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:17:41.629+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:17:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:41.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:41.969271+0000) 2022-09-10T11:17:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:42.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:42.446927+0000) 2022-09-10T11:17:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:42 smithi104 ceph-mon[97044]: pgmap v13070: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:42 smithi110 ceph-mon[97633]: pgmap v13070: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:42.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:42.969505+0000) 2022-09-10T11:17:43.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:43.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:43.447189+0000) 2022-09-10T11:17:44.315 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:43.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:43.969743+0000) 2022-09-10T11:17:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:44 smithi104 ceph-mon[97044]: pgmap v13071: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:44.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:44.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:44.447448+0000) 2022-09-10T11:17:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:44 smithi110 ceph-mon[97633]: pgmap v13071: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:44.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:44.969984+0000) 2022-09-10T11:17:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:17:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:17:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:17:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:17:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:45.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:45.447708+0000) 2022-09-10T11:17:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:17:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:17:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:17:45.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:17:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:45.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:45.970157+0000) 2022-09-10T11:17:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:46 smithi104 ceph-mon[97044]: pgmap v13072: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:46.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:46.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:46.447970+0000) 2022-09-10T11:17:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:46 smithi110 ceph-mon[97633]: pgmap v13072: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:46.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:46.970402+0000) 2022-09-10T11:17:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:47.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:47.448176+0000) 2022-09-10T11:17:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:47.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:47.970620+0000) 2022-09-10T11:17:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:48.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:48.448365+0000) 2022-09-10T11:17:49.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:48 smithi104 ceph-mon[97044]: pgmap v13073: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:48 smithi110 ceph-mon[97633]: pgmap v13073: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:48.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:48.970762+0000) 2022-09-10T11:17:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:49.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:49.448504+0000) 2022-09-10T11:17:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:49.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:49.970960+0000) 2022-09-10T11:17:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:50.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:50.448701+0000) 2022-09-10T11:17:51.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:50 smithi104 ceph-mon[97044]: pgmap v13074: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:50 smithi110 ceph-mon[97633]: pgmap v13074: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:51.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:50.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:50.971173+0000) 2022-09-10T11:17:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:51.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:51.448844+0000) 2022-09-10T11:17:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:51.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:51.971384+0000) 2022-09-10T11:17:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:52.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:52.448985+0000) 2022-09-10T11:17:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:52 smithi104 ceph-mon[97044]: pgmap v13075: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:52 smithi110 ceph-mon[97633]: pgmap v13075: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:52.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:52.971629+0000) 2022-09-10T11:17:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:53.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:53.449146+0000) 2022-09-10T11:17:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:53.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:53.971893+0000) 2022-09-10T11:17:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:54 smithi104 ceph-mon[97044]: pgmap v13076: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:54.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:54.449388+0000) 2022-09-10T11:17:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:54 smithi110 ceph-mon[97633]: pgmap v13076: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:55.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:54.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:54.972082+0000) 2022-09-10T11:17:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:55.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:55.449655+0000) 2022-09-10T11:17:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:55.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:55.972336+0000) 2022-09-10T11:17:56.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:56.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:56.449872+0000) 2022-09-10T11:17:56.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:17:56.640+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:17:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:56 smithi104 ceph-mon[97044]: pgmap v13077: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:56 smithi110 ceph-mon[97633]: pgmap v13077: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:17:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:56.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:56.972529+0000) 2022-09-10T11:17:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:57.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:57.450073+0000) 2022-09-10T11:17:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:57.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:57.972714+0000) 2022-09-10T11:17:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:58.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:58.450299+0000) 2022-09-10T11:17:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:17:58 smithi104 ceph-mon[97044]: pgmap v13078: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:17:58 smithi110 ceph-mon[97633]: pgmap v13078: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:17:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:58.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:58.972935+0000) 2022-09-10T11:17:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:17:59.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:59.450491+0000) 2022-09-10T11:18:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:17:59.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:17:59.973163+0000) 2022-09-10T11:18:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:00.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:00.450662+0000) 2022-09-10T11:18:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:00 smithi104 ceph-mon[97044]: pgmap v13079: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:00 smithi110 ceph-mon[97633]: pgmap v13079: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:00.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:00.973296+0000) 2022-09-10T11:18:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:01.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:01.450807+0000) 2022-09-10T11:18:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:01.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:01.973501+0000) 2022-09-10T11:18:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:02.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:02.451045+0000) 2022-09-10T11:18:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:02 smithi104 ceph-mon[97044]: pgmap v13080: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:02 smithi110 ceph-mon[97633]: pgmap v13080: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:02.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:02.973752+0000) 2022-09-10T11:18:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:03.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:03.451273+0000) 2022-09-10T11:18:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:03.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:03.974005+0000) 2022-09-10T11:18:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:04 smithi104 ceph-mon[97044]: pgmap v13081: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:04.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:04.451539+0000) 2022-09-10T11:18:04.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:04 smithi110 ceph-mon[97633]: pgmap v13081: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:04.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:04.974227+0000) 2022-09-10T11:18:05.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:05.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:05.451824+0000) 2022-09-10T11:18:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:05.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:05.974486+0000) 2022-09-10T11:18:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:06.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:06.452066+0000) 2022-09-10T11:18:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:06 smithi104 ceph-mon[97044]: pgmap v13082: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:18:07.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:06 smithi110 ceph-mon[97633]: pgmap v13082: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:18:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:06.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:06.974644+0000) 2022-09-10T11:18:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:07.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:07.452285+0000) 2022-09-10T11:18:08.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:07.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:07.974848+0000) 2022-09-10T11:18:08.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:08.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:08.452534+0000) 2022-09-10T11:18:09.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:08 smithi104 ceph-mon[97044]: pgmap v13083: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:18:09.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:08 smithi104 ceph-mon[97044]: osdmap e898: 12 total, 12 up, 12 in 2022-09-10T11:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:08 smithi110 ceph-mon[97633]: pgmap v13083: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:18:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:08 smithi110 ceph-mon[97633]: osdmap e898: 12 total, 12 up, 12 in 2022-09-10T11:18:09.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:08.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:08.975030+0000) 2022-09-10T11:18:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:09.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:09.452797+0000) 2022-09-10T11:18:10.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:09.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:09.975259+0000) 2022-09-10T11:18:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:10.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:10.453001+0000) 2022-09-10T11:18:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:10 smithi104 ceph-mon[97044]: pgmap v13085: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:18:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:10 smithi110 ceph-mon[97633]: pgmap v13085: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:18:11.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:10.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:10.975516+0000) 2022-09-10T11:18:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:11.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:11.453196+0000) 2022-09-10T11:18:11.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:18:11.650+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:18:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:11.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:11.975761+0000) 2022-09-10T11:18:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:12.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:12.453426+0000) 2022-09-10T11:18:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:12 smithi104 ceph-mon[97044]: pgmap v13086: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:18:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:12 smithi110 ceph-mon[97633]: pgmap v13086: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:18:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:12.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:12.975951+0000) 2022-09-10T11:18:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:13.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:13.453692+0000) 2022-09-10T11:18:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:13.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:13.976225+0000) 2022-09-10T11:18:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:14 smithi104 ceph-mon[97044]: osdmap e899: 12 total, 12 up, 12 in 2022-09-10T11:18:14.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:14 smithi104 ceph-mon[97044]: pgmap v13088: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:18:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:14.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:14.453830+0000) 2022-09-10T11:18:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:14 smithi110 ceph-mon[97633]: osdmap e899: 12 total, 12 up, 12 in 2022-09-10T11:18:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:14 smithi110 ceph-mon[97633]: pgmap v13088: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:18:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:14.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:14.976391+0000) 2022-09-10T11:18:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:15.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:15.453978+0000) 2022-09-10T11:18:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:15.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:15.976574+0000) 2022-09-10T11:18:16.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:16.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:16.454226+0000) 2022-09-10T11:18:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:16 smithi104 ceph-mon[97044]: pgmap v13089: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:18:17.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:16 smithi110 ceph-mon[97633]: pgmap v13089: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:18:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:16.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:16.976781+0000) 2022-09-10T11:18:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:17.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:17.454522+0000) 2022-09-10T11:18:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:17.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:17.977005+0000) 2022-09-10T11:18:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:18.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:18.454811+0000) 2022-09-10T11:18:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:18 smithi104 ceph-mon[97044]: pgmap v13090: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:18:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:18 smithi110 ceph-mon[97633]: pgmap v13090: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:18:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:18.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:18.977240+0000) 2022-09-10T11:18:19.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:19.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:19.455085+0000) 2022-09-10T11:18:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:19.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:19.977462+0000) 2022-09-10T11:18:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:20.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:20.455260+0000) 2022-09-10T11:18:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:20 smithi104 ceph-mon[97044]: pgmap v13091: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:18:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:20 smithi110 ceph-mon[97633]: pgmap v13091: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:18:21.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:20.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:20.977689+0000) 2022-09-10T11:18:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:21.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:21.455496+0000) 2022-09-10T11:18:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:21.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:21.977913+0000) 2022-09-10T11:18:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:22.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:22.455729+0000) 2022-09-10T11:18:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:22 smithi104 ceph-mon[97044]: pgmap v13092: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:18:23.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:22 smithi110 ceph-mon[97633]: pgmap v13092: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:18:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:22.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:22.978196+0000) 2022-09-10T11:18:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:23.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:23.455972+0000) 2022-09-10T11:18:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:23.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:23.978409+0000) 2022-09-10T11:18:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:24 smithi104 ceph-mon[97044]: pgmap v13093: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:18:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:24.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:24.456163+0000) 2022-09-10T11:18:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:24 smithi110 ceph-mon[97633]: pgmap v13093: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:18:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:24.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:24.978615+0000) 2022-09-10T11:18:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:25.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:25.456423+0000) 2022-09-10T11:18:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:25.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:25.978825+0000) 2022-09-10T11:18:26.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:18:26.665+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:18:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:26.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:26.456637+0000) 2022-09-10T11:18:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:26 smithi104 ceph-mon[97044]: pgmap v13094: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:18:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:26 smithi110 ceph-mon[97633]: pgmap v13094: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:18:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:26.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:26.979030+0000) 2022-09-10T11:18:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:27.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:27.456873+0000) 2022-09-10T11:18:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:27.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:27.979260+0000) 2022-09-10T11:18:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:28.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:28.457183+0000) 2022-09-10T11:18:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:28 smithi104 ceph-mon[97044]: pgmap v13095: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:28 smithi110 ceph-mon[97633]: pgmap v13095: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:28.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:28.979511+0000) 2022-09-10T11:18:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:29.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:29.457442+0000) 2022-09-10T11:18:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:29.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:29.979729+0000) 2022-09-10T11:18:30.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:30.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:30.457700+0000) 2022-09-10T11:18:31.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:30 smithi104 ceph-mon[97044]: pgmap v13096: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:31.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:30 smithi110 ceph-mon[97633]: pgmap v13096: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:30.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:30.979954+0000) 2022-09-10T11:18:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:31.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:31.457985+0000) 2022-09-10T11:18:32.318 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:31.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:31.980182+0000) 2022-09-10T11:18:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:32.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:32.458210+0000) 2022-09-10T11:18:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:32 smithi104 ceph-mon[97044]: pgmap v13097: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:32 smithi110 ceph-mon[97633]: pgmap v13097: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:33.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:32.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:32.980353+0000) 2022-09-10T11:18:33.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:33.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:33.458410+0000) 2022-09-10T11:18:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:33.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:33.980527+0000) 2022-09-10T11:18:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:34 smithi104 ceph-mon[97044]: pgmap v13098: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:34.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:34.458669+0000) 2022-09-10T11:18:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:34 smithi110 ceph-mon[97633]: pgmap v13098: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:34.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:34.980761+0000) 2022-09-10T11:18:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:35.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:35.458898+0000) 2022-09-10T11:18:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:35.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:35.980897+0000) 2022-09-10T11:18:36.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:36 smithi104 ceph-mon[97044]: pgmap v13099: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:36.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:36.459166+0000) 2022-09-10T11:18:37.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:36 smithi110 ceph-mon[97633]: pgmap v13099: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:36.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:36.981132+0000) 2022-09-10T11:18:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:37.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:37.459379+0000) 2022-09-10T11:18:38.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:37.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:37.981367+0000) 2022-09-10T11:18:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:38.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:38.459599+0000) 2022-09-10T11:18:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:38 smithi104 ceph-mon[97044]: pgmap v13100: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:38 smithi110 ceph-mon[97633]: pgmap v13100: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:38.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:38.981596+0000) 2022-09-10T11:18:39.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:39.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:39.459811+0000) 2022-09-10T11:18:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:39.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:39.981861+0000) 2022-09-10T11:18:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:40.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:40.460056+0000) 2022-09-10T11:18:41.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:40 smithi104 ceph-mon[97044]: pgmap v13101: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:40 smithi110 ceph-mon[97633]: pgmap v13101: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:40.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:40.982060+0000) 2022-09-10T11:18:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:41.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:41.460346+0000) 2022-09-10T11:18:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:18:41.674+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:18:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:41.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:41.982300+0000) 2022-09-10T11:18:42.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:42.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:42.460575+0000) 2022-09-10T11:18:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:42 smithi104 ceph-mon[97044]: pgmap v13102: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:42 smithi110 ceph-mon[97633]: pgmap v13102: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:42.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:42.982503+0000) 2022-09-10T11:18:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:43.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:43.460777+0000) 2022-09-10T11:18:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:43.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:43.982722+0000) 2022-09-10T11:18:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:44.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:44.461025+0000) 2022-09-10T11:18:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:44 smithi110 ceph-mon[97633]: pgmap v13103: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:44 smithi104 ceph-mon[97044]: pgmap v13103: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:44.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:44.982957+0000) 2022-09-10T11:18:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:45.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:45.461237+0000) 2022-09-10T11:18:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:18:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:18:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:18:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:18:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:18:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:18:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:18:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:18:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:45.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:45.983145+0000) 2022-09-10T11:18:46.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:46.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:46.461398+0000) 2022-09-10T11:18:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:46 smithi110 ceph-mon[97633]: pgmap v13104: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:46 smithi104 ceph-mon[97044]: pgmap v13104: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:46.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:46.983286+0000) 2022-09-10T11:18:47.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:47.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:47.461559+0000) 2022-09-10T11:18:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:47.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:47.983507+0000) 2022-09-10T11:18:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:48.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:48.461817+0000) 2022-09-10T11:18:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:48 smithi104 ceph-mon[97044]: pgmap v13105: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:49.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:48 smithi110 ceph-mon[97633]: pgmap v13105: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:48.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:48.983700+0000) 2022-09-10T11:18:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:49.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:49.462064+0000) 2022-09-10T11:18:50.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:49.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:49.983903+0000) 2022-09-10T11:18:50.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:50.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:50.462262+0000) 2022-09-10T11:18:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:50 smithi104 ceph-mon[97044]: pgmap v13106: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:50 smithi110 ceph-mon[97633]: pgmap v13106: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:50.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:50.984158+0000) 2022-09-10T11:18:51.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:51.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:51.462424+0000) 2022-09-10T11:18:52.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:51.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:51.984329+0000) 2022-09-10T11:18:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:52.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:52.462632+0000) 2022-09-10T11:18:53.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:52 smithi104 ceph-mon[97044]: pgmap v13107: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:53.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:52 smithi110 ceph-mon[97633]: pgmap v13107: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:53.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:52.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:52.984453+0000) 2022-09-10T11:18:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:53.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:53.462830+0000) 2022-09-10T11:18:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:53.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:53.984713+0000) 2022-09-10T11:18:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:54.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:54.463080+0000) 2022-09-10T11:18:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:54 smithi110 ceph-mon[97633]: pgmap v13108: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:54 smithi104 ceph-mon[97044]: pgmap v13108: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:18:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:54.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:54.984914+0000) 2022-09-10T11:18:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:55.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:55.463322+0000) 2022-09-10T11:18:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:55.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:55.985185+0000) 2022-09-10T11:18:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:56.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:56.463553+0000) 2022-09-10T11:18:56.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:18:56.683+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:18:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:56 smithi104 ceph-mon[97044]: pgmap v13109: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:57.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:56 smithi110 ceph-mon[97633]: pgmap v13109: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:18:57.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:56.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:56.985407+0000) 2022-09-10T11:18:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:57.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:57.463755+0000) 2022-09-10T11:18:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:57.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:57.985633+0000) 2022-09-10T11:18:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:58.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:58.463996+0000) 2022-09-10T11:18:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:18:58 smithi104 ceph-mon[97044]: pgmap v13110: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:18:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:18:58 smithi110 ceph-mon[97633]: pgmap v13110: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:18:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:58.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:58.985793+0000) 2022-09-10T11:18:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:18:59.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:59.464245+0000) 2022-09-10T11:19:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:18:59.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:18:59.985999+0000) 2022-09-10T11:19:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:00.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:00.464456+0000) 2022-09-10T11:19:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:00 smithi104 ceph-mon[97044]: pgmap v13111: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:00 smithi110 ceph-mon[97633]: pgmap v13111: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:00.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:00.986146+0000) 2022-09-10T11:19:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:01.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:01.464653+0000) 2022-09-10T11:19:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:01.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:01.986363+0000) 2022-09-10T11:19:02.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:02.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:02.464877+0000) 2022-09-10T11:19:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:02 smithi104 ceph-mon[97044]: pgmap v13112: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:02 smithi110 ceph-mon[97633]: pgmap v13112: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:02.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:02.986584+0000) 2022-09-10T11:19:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:03.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:03.465143+0000) 2022-09-10T11:19:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:03.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:03.986775+0000) 2022-09-10T11:19:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:04.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:04.465349+0000) 2022-09-10T11:19:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:04 smithi110 ceph-mon[97633]: pgmap v13113: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:04 smithi104 ceph-mon[97044]: pgmap v13113: 129 pgs: 129 active+clean; 9.1 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:04.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:04.986998+0000) 2022-09-10T11:19:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:05.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:05.465605+0000) 2022-09-10T11:19:06.315 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:05.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:05.987244+0000) 2022-09-10T11:19:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:06.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:06.465841+0000) 2022-09-10T11:19:07.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:06 smithi104 ceph-mon[97044]: pgmap v13114: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:19:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:06 smithi110 ceph-mon[97633]: pgmap v13114: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:19:07.317 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:06.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:06.987495+0000) 2022-09-10T11:19:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:07.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:07.466080+0000) 2022-09-10T11:19:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:07.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:07.987743+0000) 2022-09-10T11:19:08.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:08.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:08.466388+0000) 2022-09-10T11:19:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:08 smithi104 ceph-mon[97044]: pgmap v13115: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:19:09.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:08 smithi104 ceph-mon[97044]: osdmap e900: 12 total, 12 up, 12 in 2022-09-10T11:19:09.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:08 smithi110 ceph-mon[97633]: pgmap v13115: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:19:09.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:08 smithi110 ceph-mon[97633]: osdmap e900: 12 total, 12 up, 12 in 2022-09-10T11:19:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:08.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:08.987975+0000) 2022-09-10T11:19:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:09.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:09.466570+0000) 2022-09-10T11:19:10.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:09.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:09.988181+0000) 2022-09-10T11:19:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:10.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:10.466783+0000) 2022-09-10T11:19:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:10 smithi104 ceph-mon[97044]: pgmap v13117: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:19:11.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:10 smithi110 ceph-mon[97633]: pgmap v13117: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:19:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:10.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:10.988373+0000) 2022-09-10T11:19:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:11.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:11.466964+0000) 2022-09-10T11:19:11.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:19:11.696+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:19:12.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:11.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:11.988555+0000) 2022-09-10T11:19:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:12.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:12.467174+0000) 2022-09-10T11:19:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:12 smithi104 ceph-mon[97044]: pgmap v13118: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:19:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:12 smithi110 ceph-mon[97633]: pgmap v13118: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:19:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:12.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:12.988821+0000) 2022-09-10T11:19:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:13.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:13.467418+0000) 2022-09-10T11:19:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:13.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:13.989128+0000) 2022-09-10T11:19:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:14.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:14.467605+0000) 2022-09-10T11:19:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:14 smithi110 ceph-mon[97633]: osdmap e901: 12 total, 12 up, 12 in 2022-09-10T11:19:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:14 smithi110 ceph-mon[97633]: pgmap v13120: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:19:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:14 smithi104 ceph-mon[97044]: osdmap e901: 12 total, 12 up, 12 in 2022-09-10T11:19:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:14 smithi104 ceph-mon[97044]: pgmap v13120: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:19:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:14.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:14.989334+0000) 2022-09-10T11:19:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:15.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:15.467770+0000) 2022-09-10T11:19:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:15.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:15.989533+0000) 2022-09-10T11:19:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:16.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:16.468013+0000) 2022-09-10T11:19:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:16 smithi104 ceph-mon[97044]: pgmap v13121: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:19:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:16 smithi110 ceph-mon[97633]: pgmap v13121: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:19:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:16.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:16.989722+0000) 2022-09-10T11:19:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:17.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:17.468269+0000) 2022-09-10T11:19:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:17.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:17.989927+0000) 2022-09-10T11:19:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:18.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:18.468487+0000) 2022-09-10T11:19:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:18 smithi104 ceph-mon[97044]: pgmap v13122: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:19:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:18 smithi110 ceph-mon[97633]: pgmap v13122: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:19:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:18.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:18.990207+0000) 2022-09-10T11:19:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:19.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:19.468675+0000) 2022-09-10T11:19:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:19.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:19.990343+0000) 2022-09-10T11:19:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:20.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:20.468871+0000) 2022-09-10T11:19:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:20 smithi104 ceph-mon[97044]: pgmap v13123: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:19:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:20 smithi110 ceph-mon[97633]: pgmap v13123: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:19:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:20.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:20.990507+0000) 2022-09-10T11:19:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:21.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:21.469139+0000) 2022-09-10T11:19:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:21.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:21.990735+0000) 2022-09-10T11:19:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:22.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:22.469387+0000) 2022-09-10T11:19:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:22 smithi104 ceph-mon[97044]: pgmap v13124: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:19:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:22 smithi110 ceph-mon[97633]: pgmap v13124: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:19:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:22.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:22.990959+0000) 2022-09-10T11:19:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:23.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:23.469615+0000) 2022-09-10T11:19:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:23.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:23.991270+0000) 2022-09-10T11:19:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:24.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:24.469886+0000) 2022-09-10T11:19:24.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:24 smithi110 ceph-mon[97633]: pgmap v13125: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T11:19:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:24 smithi104 ceph-mon[97044]: pgmap v13125: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T11:19:25.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:24.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:24.991414+0000) 2022-09-10T11:19:25.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:25.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:25.470169+0000) 2022-09-10T11:19:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:25.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:25.991611+0000) 2022-09-10T11:19:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:26.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:26.470370+0000) 2022-09-10T11:19:26.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:19:26.698+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:19:27.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:26 smithi104 ceph-mon[97044]: pgmap v13126: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:19:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:26 smithi110 ceph-mon[97633]: pgmap v13126: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:19:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:26.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:26.991813+0000) 2022-09-10T11:19:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:27.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:27.470546+0000) 2022-09-10T11:19:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:27.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:27.992059+0000) 2022-09-10T11:19:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:28.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:28.470805+0000) 2022-09-10T11:19:29.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:28 smithi104 ceph-mon[97044]: pgmap v13127: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:29.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:28 smithi110 ceph-mon[97633]: pgmap v13127: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:28.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:28.992355+0000) 2022-09-10T11:19:29.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:29.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:29.471072+0000) 2022-09-10T11:19:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:29.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:29.992616+0000) 2022-09-10T11:19:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:30.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:30.471284+0000) 2022-09-10T11:19:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:30 smithi104 ceph-mon[97044]: pgmap v13128: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:30 smithi110 ceph-mon[97633]: pgmap v13128: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:30.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:30.992858+0000) 2022-09-10T11:19:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:31.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:31.471503+0000) 2022-09-10T11:19:32.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:31.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:31.993064+0000) 2022-09-10T11:19:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:32.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:32.471760+0000) 2022-09-10T11:19:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:32 smithi104 ceph-mon[97044]: pgmap v13129: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:32 smithi110 ceph-mon[97633]: pgmap v13129: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:32.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:32.993305+0000) 2022-09-10T11:19:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:33.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:33.472052+0000) 2022-09-10T11:19:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:33.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:33.993549+0000) 2022-09-10T11:19:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:34.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:34.472301+0000) 2022-09-10T11:19:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:34 smithi110 ceph-mon[97633]: pgmap v13130: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:34 smithi104 ceph-mon[97044]: pgmap v13130: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:34.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:34.993796+0000) 2022-09-10T11:19:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:35.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:35.472481+0000) 2022-09-10T11:19:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:35.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:35.993986+0000) 2022-09-10T11:19:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:36.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:36.472666+0000) 2022-09-10T11:19:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:36 smithi104 ceph-mon[97044]: pgmap v13131: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:36 smithi110 ceph-mon[97633]: pgmap v13131: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:36.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:36.994250+0000) 2022-09-10T11:19:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:37.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:37.472904+0000) 2022-09-10T11:19:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:37.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:37.994411+0000) 2022-09-10T11:19:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:38.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:38.473165+0000) 2022-09-10T11:19:39.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:38 smithi104 ceph-mon[97044]: pgmap v13132: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:39.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:38 smithi110 ceph-mon[97633]: pgmap v13132: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:38.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:38.994661+0000) 2022-09-10T11:19:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:39.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:39.473391+0000) 2022-09-10T11:19:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:39.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:39.994908+0000) 2022-09-10T11:19:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:40.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:40.473585+0000) 2022-09-10T11:19:41.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:40 smithi104 ceph-mon[97044]: pgmap v13133: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:40 smithi110 ceph-mon[97633]: pgmap v13133: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:40.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:40.995191+0000) 2022-09-10T11:19:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:41.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:41.473843+0000) 2022-09-10T11:19:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:19:41.710+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:19:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:41.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:41.995357+0000) 2022-09-10T11:19:42.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:42.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:42.474137+0000) 2022-09-10T11:19:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:42 smithi104 ceph-mon[97044]: pgmap v13134: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:43.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:42 smithi110 ceph-mon[97633]: pgmap v13134: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:43.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:42.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:42.995600+0000) 2022-09-10T11:19:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:43.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:43.474310+0000) 2022-09-10T11:19:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:43.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:43.995898+0000) 2022-09-10T11:19:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:44.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:44.474537+0000) 2022-09-10T11:19:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:44 smithi110 ceph-mon[97633]: pgmap v13135: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:44 smithi104 ceph-mon[97044]: pgmap v13135: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:44.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:44.996198+0000) 2022-09-10T11:19:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:45.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:45.474801+0000) 2022-09-10T11:19:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:19:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:19:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:19:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:19:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:19:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:19:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:19:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:19:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:45.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:45.996422+0000) 2022-09-10T11:19:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:46.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:46.475032+0000) 2022-09-10T11:19:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:46 smithi110 ceph-mon[97633]: pgmap v13136: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:46 smithi104 ceph-mon[97044]: pgmap v13136: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:46.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:46.996613+0000) 2022-09-10T11:19:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:47.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:47.475263+0000) 2022-09-10T11:19:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:47.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:47.996827+0000) 2022-09-10T11:19:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:48.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:48.475505+0000) 2022-09-10T11:19:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:48 smithi104 ceph-mon[97044]: pgmap v13137: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:48 smithi110 ceph-mon[97633]: pgmap v13137: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:48.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:48.997045+0000) 2022-09-10T11:19:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:49.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:49.475741+0000) 2022-09-10T11:19:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:49.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:49.997199+0000) 2022-09-10T11:19:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:50.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:50.475949+0000) 2022-09-10T11:19:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:50 smithi104 ceph-mon[97044]: pgmap v13138: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:50 smithi110 ceph-mon[97633]: pgmap v13138: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:50.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:50.997365+0000) 2022-09-10T11:19:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:51.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:51.476202+0000) 2022-09-10T11:19:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:51.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:51.997557+0000) 2022-09-10T11:19:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:52.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:52.476424+0000) 2022-09-10T11:19:53.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:52 smithi104 ceph-mon[97044]: pgmap v13139: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:52 smithi110 ceph-mon[97633]: pgmap v13139: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:52.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:52.997724+0000) 2022-09-10T11:19:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:53.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:53.476632+0000) 2022-09-10T11:19:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:53.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:53.997971+0000) 2022-09-10T11:19:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:54.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:54.476885+0000) 2022-09-10T11:19:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:54 smithi110 ceph-mon[97633]: pgmap v13140: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:54 smithi104 ceph-mon[97044]: pgmap v13140: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:19:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:54.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:54.998173+0000) 2022-09-10T11:19:55.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:55.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:55.477139+0000) 2022-09-10T11:19:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:55.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:55.998450+0000) 2022-09-10T11:19:56.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:56.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:56.477413+0000) 2022-09-10T11:19:56.813 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:19:56.711+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:19:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:56 smithi104 ceph-mon[97044]: pgmap v13141: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:56 smithi110 ceph-mon[97633]: pgmap v13141: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:19:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:56.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:56.998648+0000) 2022-09-10T11:19:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:57.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:57.477612+0000) 2022-09-10T11:19:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:57.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:57.998868+0000) 2022-09-10T11:19:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:58.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:58.477838+0000) 2022-09-10T11:19:59.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:19:58 smithi104 ceph-mon[97044]: pgmap v13142: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:59.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:19:58 smithi110 ceph-mon[97633]: pgmap v13142: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:19:59.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:58.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:58.999062+0000) 2022-09-10T11:19:59.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:19:59.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:59.478074+0000) 2022-09-10T11:20:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:19:59.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:19:59.999232+0000) 2022-09-10T11:20:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:00.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:00.478195+0000) 2022-09-10T11:20:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:00 smithi104 ceph-mon[97044]: pgmap v13143: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:20:01.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:00 smithi110 ceph-mon[97633]: pgmap v13143: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:01.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:20:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:00.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:00.999428+0000) 2022-09-10T11:20:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:01.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:01.478430+0000) 2022-09-10T11:20:02.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:01.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:01.999640+0000) 2022-09-10T11:20:02.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:02.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:02.478660+0000) 2022-09-10T11:20:03.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:02 smithi104 ceph-mon[97044]: pgmap v13144: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:02 smithi110 ceph-mon[97633]: pgmap v13144: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:02.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:02.999841+0000) 2022-09-10T11:20:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:03.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:03.478801+0000) 2022-09-10T11:20:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:03.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:04.000022+0000) 2022-09-10T11:20:04.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:04.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:04.479033+0000) 2022-09-10T11:20:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:04 smithi110 ceph-mon[97633]: pgmap v13145: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:05.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:04 smithi104 ceph-mon[97044]: pgmap v13145: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:04.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:05.000271+0000) 2022-09-10T11:20:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:05.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:05.479238+0000) 2022-09-10T11:20:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:05.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:06.000495+0000) 2022-09-10T11:20:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:06.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:06.479429+0000) 2022-09-10T11:20:07.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:06 smithi104 ceph-mon[97044]: pgmap v13146: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:20:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:06 smithi110 ceph-mon[97633]: pgmap v13146: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:20:07.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:06.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:07.000738+0000) 2022-09-10T11:20:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:07.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:07.479561+0000) 2022-09-10T11:20:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:07.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:08.000929+0000) 2022-09-10T11:20:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:08.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:08.479797+0000) 2022-09-10T11:20:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:08 smithi104 ceph-mon[97044]: pgmap v13147: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:20:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:08 smithi104 ceph-mon[97044]: osdmap e902: 12 total, 12 up, 12 in 2022-09-10T11:20:09.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:08 smithi110 ceph-mon[97633]: pgmap v13147: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:20:09.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:08 smithi110 ceph-mon[97633]: osdmap e902: 12 total, 12 up, 12 in 2022-09-10T11:20:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:09.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:09.001221+0000) 2022-09-10T11:20:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:09.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:09.480012+0000) 2022-09-10T11:20:10.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:10.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:10.001429+0000) 2022-09-10T11:20:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:10.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:10.480222+0000) 2022-09-10T11:20:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:10 smithi104 ceph-mon[97044]: pgmap v13149: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:20:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:10 smithi110 ceph-mon[97633]: pgmap v13149: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:20:11.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:11.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:11.001691+0000) 2022-09-10T11:20:11.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:20:11.713+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:20:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:11.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:11.480446+0000) 2022-09-10T11:20:12.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:12.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:12.001863+0000) 2022-09-10T11:20:12.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:12.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:12.480687+0000) 2022-09-10T11:20:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:12 smithi104 ceph-mon[97044]: pgmap v13150: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:20:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:12 smithi110 ceph-mon[97633]: pgmap v13150: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:20:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:13.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:13.002055+0000) 2022-09-10T11:20:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:13.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:13.480932+0000) 2022-09-10T11:20:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:14.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:14.002263+0000) 2022-09-10T11:20:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:14.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:14.481201+0000) 2022-09-10T11:20:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:14 smithi110 ceph-mon[97633]: osdmap e903: 12 total, 12 up, 12 in 2022-09-10T11:20:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:14 smithi110 ceph-mon[97633]: pgmap v13152: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:20:15.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:14 smithi104 ceph-mon[97044]: osdmap e903: 12 total, 12 up, 12 in 2022-09-10T11:20:15.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:14 smithi104 ceph-mon[97044]: pgmap v13152: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:20:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:15.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:15.002412+0000) 2022-09-10T11:20:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:15.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:15.481391+0000) 2022-09-10T11:20:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:16.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:16.002541+0000) 2022-09-10T11:20:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:16.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:16.481640+0000) 2022-09-10T11:20:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:16 smithi104 ceph-mon[97044]: pgmap v13153: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:20:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:16 smithi110 ceph-mon[97633]: pgmap v13153: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:20:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:17.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:17.002676+0000) 2022-09-10T11:20:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:17.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:17.481904+0000) 2022-09-10T11:20:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:18.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:18.002862+0000) 2022-09-10T11:20:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:18.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:18.482147+0000) 2022-09-10T11:20:19.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:18 smithi104 ceph-mon[97044]: pgmap v13154: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:20:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:18 smithi110 ceph-mon[97633]: pgmap v13154: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:20:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:19.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:19.003077+0000) 2022-09-10T11:20:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:19.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:19.482306+0000) 2022-09-10T11:20:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:20.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:20.003324+0000) 2022-09-10T11:20:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:20.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:20.482489+0000) 2022-09-10T11:20:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:20 smithi104 ceph-mon[97044]: pgmap v13155: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:20:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:20 smithi110 ceph-mon[97633]: pgmap v13155: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:20:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:21.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:21.003577+0000) 2022-09-10T11:20:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:21.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:21.482672+0000) 2022-09-10T11:20:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:22.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:22.003827+0000) 2022-09-10T11:20:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:22.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:22.482892+0000) 2022-09-10T11:20:23.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:22 smithi104 ceph-mon[97044]: pgmap v13156: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:20:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:22 smithi110 ceph-mon[97633]: pgmap v13156: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:20:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:23.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:23.004046+0000) 2022-09-10T11:20:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:23.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:23.483183+0000) 2022-09-10T11:20:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:24.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:24.004253+0000) 2022-09-10T11:20:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:24.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:24.483432+0000) 2022-09-10T11:20:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:24 smithi110 ceph-mon[97633]: pgmap v13157: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:20:25.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:24 smithi104 ceph-mon[97044]: pgmap v13157: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:20:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:25.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:25.004449+0000) 2022-09-10T11:20:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:25.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:25.483660+0000) 2022-09-10T11:20:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:26.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:26.004699+0000) 2022-09-10T11:20:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:26.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:26.483939+0000) 2022-09-10T11:20:26.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:20:26.723+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:20:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:26 smithi104 ceph-mon[97044]: pgmap v13158: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:20:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:26 smithi110 ceph-mon[97633]: pgmap v13158: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:20:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:27.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:27.004929+0000) 2022-09-10T11:20:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:27.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:27.484139+0000) 2022-09-10T11:20:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:28.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:28.005167+0000) 2022-09-10T11:20:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:28.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:28.484356+0000) 2022-09-10T11:20:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:28 smithi104 ceph-mon[97044]: pgmap v13159: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:28 smithi110 ceph-mon[97633]: pgmap v13159: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:29.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:29.005412+0000) 2022-09-10T11:20:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:29.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:29.484540+0000) 2022-09-10T11:20:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:30.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:30.005640+0000) 2022-09-10T11:20:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:30.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:30.484764+0000) 2022-09-10T11:20:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:30 smithi104 ceph-mon[97044]: pgmap v13160: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:30 smithi110 ceph-mon[97633]: pgmap v13160: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:31.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:31.005819+0000) 2022-09-10T11:20:31.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:31.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:31.485034+0000) 2022-09-10T11:20:32.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:32.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:32.006044+0000) 2022-09-10T11:20:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:32.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:32.485276+0000) 2022-09-10T11:20:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:32 smithi104 ceph-mon[97044]: pgmap v13161: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:32 smithi110 ceph-mon[97633]: pgmap v13161: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:33.320 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:33.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:33.006269+0000) 2022-09-10T11:20:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:33.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:33.485504+0000) 2022-09-10T11:20:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:34.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:34.006553+0000) 2022-09-10T11:20:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:34.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:34.485705+0000) 2022-09-10T11:20:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:34 smithi110 ceph-mon[97633]: pgmap v13162: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:34 smithi104 ceph-mon[97044]: pgmap v13162: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:35.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:35.006782+0000) 2022-09-10T11:20:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:35.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:35.485910+0000) 2022-09-10T11:20:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:36.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:36.007009+0000) 2022-09-10T11:20:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:36.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:36.486070+0000) 2022-09-10T11:20:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:36 smithi104 ceph-mon[97044]: pgmap v13163: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:36 smithi110 ceph-mon[97633]: pgmap v13163: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:37.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:37.007270+0000) 2022-09-10T11:20:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:37.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:37.486297+0000) 2022-09-10T11:20:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:38.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:38.007504+0000) 2022-09-10T11:20:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:38.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:38.486529+0000) 2022-09-10T11:20:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:38 smithi104 ceph-mon[97044]: pgmap v13164: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:38 smithi110 ceph-mon[97633]: pgmap v13164: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:39.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:39.007742+0000) 2022-09-10T11:20:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:39.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:39.486710+0000) 2022-09-10T11:20:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:40.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:40.007940+0000) 2022-09-10T11:20:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:40.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:40.486996+0000) 2022-09-10T11:20:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:40 smithi104 ceph-mon[97044]: pgmap v13165: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:40 smithi110 ceph-mon[97633]: pgmap v13165: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:41.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:41.008199+0000) 2022-09-10T11:20:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:41.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:41.487270+0000) 2022-09-10T11:20:41.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:20:41.727+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:20:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:42.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:42.008459+0000) 2022-09-10T11:20:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:42.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:42.487499+0000) 2022-09-10T11:20:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:42 smithi104 ceph-mon[97044]: pgmap v13166: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:42 smithi110 ceph-mon[97633]: pgmap v13166: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:43.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:43.008694+0000) 2022-09-10T11:20:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:43.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:43.487643+0000) 2022-09-10T11:20:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:44.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:44.008963+0000) 2022-09-10T11:20:44.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:44.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:44.487823+0000) 2022-09-10T11:20:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:44 smithi110 ceph-mon[97633]: pgmap v13167: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:44 smithi104 ceph-mon[97044]: pgmap v13167: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:45.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:45.009201+0000) 2022-09-10T11:20:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:45.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:45.488020+0000) 2022-09-10T11:20:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:20:45.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:20:45.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:20:45.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:20:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:20:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:20:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:20:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:20:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:46.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:46.009421+0000) 2022-09-10T11:20:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:46.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:46.488271+0000) 2022-09-10T11:20:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:46 smithi110 ceph-mon[97633]: pgmap v13168: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:46 smithi104 ceph-mon[97044]: pgmap v13168: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:47.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:47.009589+0000) 2022-09-10T11:20:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:47.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:47.488476+0000) 2022-09-10T11:20:48.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:48.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:48.009836+0000) 2022-09-10T11:20:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:48.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:48.488694+0000) 2022-09-10T11:20:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:48 smithi104 ceph-mon[97044]: pgmap v13169: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:49.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:48 smithi110 ceph-mon[97633]: pgmap v13169: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:49.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:49.010149+0000) 2022-09-10T11:20:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:49.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:49.488908+0000) 2022-09-10T11:20:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:50.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:50.010375+0000) 2022-09-10T11:20:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:50.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:50.489192+0000) 2022-09-10T11:20:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:50 smithi104 ceph-mon[97044]: pgmap v13170: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:50 smithi110 ceph-mon[97633]: pgmap v13170: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:51.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:51.010622+0000) 2022-09-10T11:20:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:51.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:51.489430+0000) 2022-09-10T11:20:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:52.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:52.010876+0000) 2022-09-10T11:20:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:52.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:52.489704+0000) 2022-09-10T11:20:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:52 smithi104 ceph-mon[97044]: pgmap v13171: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:52 smithi110 ceph-mon[97633]: pgmap v13171: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:53.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:53.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:53.011077+0000) 2022-09-10T11:20:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:53.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:53.489941+0000) 2022-09-10T11:20:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:54.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:54.011332+0000) 2022-09-10T11:20:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:54.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:54.490174+0000) 2022-09-10T11:20:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:54 smithi110 ceph-mon[97633]: pgmap v13172: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:54 smithi104 ceph-mon[97044]: pgmap v13172: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:55.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:55.011599+0000) 2022-09-10T11:20:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:55.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:55.490404+0000) 2022-09-10T11:20:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:56.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:56.011819+0000) 2022-09-10T11:20:56.745 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:56.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:56.490577+0000) 2022-09-10T11:20:57.013 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:20:56.743+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:20:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:56 smithi104 ceph-mon[97044]: pgmap v13173: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:56 smithi110 ceph-mon[97633]: pgmap v13173: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:20:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:57.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:57.012001+0000) 2022-09-10T11:20:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:57.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:57.490729+0000) 2022-09-10T11:20:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:58.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:58.012242+0000) 2022-09-10T11:20:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:58.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:58.490980+0000) 2022-09-10T11:20:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:20:58 smithi104 ceph-mon[97044]: pgmap v13174: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:20:58 smithi110 ceph-mon[97633]: pgmap v13174: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:20:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:20:59.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:59.012512+0000) 2022-09-10T11:20:59.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:20:59.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:20:59.491190+0000) 2022-09-10T11:21:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:00.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:00.012712+0000) 2022-09-10T11:21:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:00.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:00.491341+0000) 2022-09-10T11:21:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:00 smithi104 ceph-mon[97044]: pgmap v13175: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:00 smithi110 ceph-mon[97633]: pgmap v13175: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:01.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:01.012969+0000) 2022-09-10T11:21:01.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:01.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:01.491565+0000) 2022-09-10T11:21:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:02.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:02.013174+0000) 2022-09-10T11:21:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:02.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:02.491770+0000) 2022-09-10T11:21:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:02 smithi104 ceph-mon[97044]: pgmap v13176: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:02 smithi110 ceph-mon[97633]: pgmap v13176: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:03.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:03.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:03.013402+0000) 2022-09-10T11:21:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:03.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:03.491914+0000) 2022-09-10T11:21:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:04.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:04.013638+0000) 2022-09-10T11:21:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:04.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:04.492117+0000) 2022-09-10T11:21:04.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:04 smithi110 ceph-mon[97633]: pgmap v13177: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:05.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:04 smithi104 ceph-mon[97044]: pgmap v13177: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:05.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:05.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:05.013829+0000) 2022-09-10T11:21:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:05.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:05.492319+0000) 2022-09-10T11:21:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:06.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:06.014034+0000) 2022-09-10T11:21:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:06.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:06.492499+0000) 2022-09-10T11:21:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:06 smithi104 ceph-mon[97044]: pgmap v13178: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:21:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:06 smithi110 ceph-mon[97633]: pgmap v13178: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:21:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:07.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:07.014221+0000) 2022-09-10T11:21:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:07.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:07.492750+0000) 2022-09-10T11:21:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:08.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:08.014352+0000) 2022-09-10T11:21:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:08.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:08.493013+0000) 2022-09-10T11:21:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:08 smithi104 ceph-mon[97044]: pgmap v13179: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:21:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:08 smithi104 ceph-mon[97044]: osdmap e904: 12 total, 12 up, 12 in 2022-09-10T11:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:08 smithi110 ceph-mon[97633]: pgmap v13179: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:21:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:08 smithi110 ceph-mon[97633]: osdmap e904: 12 total, 12 up, 12 in 2022-09-10T11:21:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:09.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:09.014577+0000) 2022-09-10T11:21:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:09.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:09.493215+0000) 2022-09-10T11:21:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:10.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:10.014776+0000) 2022-09-10T11:21:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:10.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:10.493424+0000) 2022-09-10T11:21:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:10 smithi104 ceph-mon[97044]: pgmap v13181: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:21:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:10 smithi110 ceph-mon[97633]: pgmap v13181: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:21:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:11.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:11.015003+0000) 2022-09-10T11:21:11.757 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:11.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:11.493660+0000) 2022-09-10T11:21:12.016 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:21:11.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:21:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:12.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:12.015240+0000) 2022-09-10T11:21:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:12.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:12.493934+0000) 2022-09-10T11:21:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:12 smithi104 ceph-mon[97044]: pgmap v13182: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:21:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:12 smithi110 ceph-mon[97633]: pgmap v13182: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:21:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:13.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:13.015466+0000) 2022-09-10T11:21:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:13.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:13.494131+0000) 2022-09-10T11:21:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:14.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:14.015672+0000) 2022-09-10T11:21:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:14.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:14.494362+0000) 2022-09-10T11:21:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:14 smithi110 ceph-mon[97633]: osdmap e905: 12 total, 12 up, 12 in 2022-09-10T11:21:14.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:14 smithi110 ceph-mon[97633]: pgmap v13184: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:21:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:14 smithi104 ceph-mon[97044]: osdmap e905: 12 total, 12 up, 12 in 2022-09-10T11:21:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:14 smithi104 ceph-mon[97044]: pgmap v13184: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:21:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:15.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:15.015885+0000) 2022-09-10T11:21:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:15.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:15.494490+0000) 2022-09-10T11:21:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:16.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:16.016141+0000) 2022-09-10T11:21:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:16.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:16.494733+0000) 2022-09-10T11:21:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:16 smithi104 ceph-mon[97044]: pgmap v13185: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:21:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:16 smithi110 ceph-mon[97633]: pgmap v13185: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:21:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:17.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:17.016344+0000) 2022-09-10T11:21:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:17.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:17.494935+0000) 2022-09-10T11:21:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:18.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:18.016515+0000) 2022-09-10T11:21:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:18.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:18.495199+0000) 2022-09-10T11:21:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:18 smithi104 ceph-mon[97044]: pgmap v13186: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:21:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:18 smithi110 ceph-mon[97633]: pgmap v13186: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:21:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:19.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:19.016785+0000) 2022-09-10T11:21:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:19.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:19.495365+0000) 2022-09-10T11:21:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:20.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:20.016948+0000) 2022-09-10T11:21:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:20.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:20.495624+0000) 2022-09-10T11:21:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:20 smithi104 ceph-mon[97044]: pgmap v13187: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:21:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:20 smithi110 ceph-mon[97633]: pgmap v13187: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:21:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:21.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:21.017196+0000) 2022-09-10T11:21:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:21.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:21.495892+0000) 2022-09-10T11:21:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:22.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:22.017438+0000) 2022-09-10T11:21:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:22.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:22.496151+0000) 2022-09-10T11:21:23.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:22 smithi104 ceph-mon[97044]: pgmap v13188: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:21:23.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:22 smithi110 ceph-mon[97633]: pgmap v13188: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:21:23.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:23.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:23.017653+0000) 2022-09-10T11:21:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:23.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:23.496396+0000) 2022-09-10T11:21:24.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:24.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:24.017961+0000) 2022-09-10T11:21:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:24.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:24.496636+0000) 2022-09-10T11:21:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:24 smithi110 ceph-mon[97633]: pgmap v13189: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:21:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:24 smithi104 ceph-mon[97044]: pgmap v13189: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:21:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:25.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:25.018202+0000) 2022-09-10T11:21:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:25.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:25.496804+0000) 2022-09-10T11:21:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:26.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:26.018389+0000) 2022-09-10T11:21:26.770 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:26.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:26.497006+0000) 2022-09-10T11:21:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:26 smithi104 ceph-mon[97044]: pgmap v13190: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:21:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:27.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:27.018569+0000) 2022-09-10T11:21:27.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:21:26.768+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:21:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:26 smithi110 ceph-mon[97633]: pgmap v13190: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:21:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:27.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:27.497257+0000) 2022-09-10T11:21:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:28.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:28.018819+0000) 2022-09-10T11:21:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:28.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:28.497439+0000) 2022-09-10T11:21:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:28 smithi104 ceph-mon[97044]: pgmap v13191: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:28 smithi110 ceph-mon[97633]: pgmap v13191: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:29.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:29.019063+0000) 2022-09-10T11:21:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:29.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:29.497679+0000) 2022-09-10T11:21:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:30.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:30.019282+0000) 2022-09-10T11:21:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:30.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:30.497877+0000) 2022-09-10T11:21:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:30 smithi104 ceph-mon[97044]: pgmap v13192: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:30 smithi110 ceph-mon[97633]: pgmap v13192: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:31.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:31.019450+0000) 2022-09-10T11:21:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:31.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:31.498164+0000) 2022-09-10T11:21:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:32.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:32.019682+0000) 2022-09-10T11:21:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:32.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:32.498416+0000) 2022-09-10T11:21:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:32 smithi104 ceph-mon[97044]: pgmap v13193: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:32 smithi110 ceph-mon[97633]: pgmap v13193: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:33.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:33.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:33.019855+0000) 2022-09-10T11:21:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:33.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:33.498686+0000) 2022-09-10T11:21:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:34.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:34.020080+0000) 2022-09-10T11:21:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:34.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:34.498924+0000) 2022-09-10T11:21:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:34 smithi110 ceph-mon[97633]: pgmap v13194: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:34 smithi104 ceph-mon[97044]: pgmap v13194: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:35.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:35.020343+0000) 2022-09-10T11:21:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:35.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:35.499182+0000) 2022-09-10T11:21:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:36.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:36.020604+0000) 2022-09-10T11:21:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:36.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:36.499372+0000) 2022-09-10T11:21:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:36 smithi104 ceph-mon[97044]: pgmap v13195: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:36 smithi110 ceph-mon[97633]: pgmap v13195: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:37.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:37.020848+0000) 2022-09-10T11:21:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:37.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:37.499594+0000) 2022-09-10T11:21:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:38.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:38.021014+0000) 2022-09-10T11:21:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:38.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:38.499828+0000) 2022-09-10T11:21:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:38 smithi104 ceph-mon[97044]: pgmap v13196: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:39.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:38 smithi110 ceph-mon[97633]: pgmap v13196: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:39.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:39.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:39.021153+0000) 2022-09-10T11:21:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:39.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:39.500076+0000) 2022-09-10T11:21:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:40.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:40.021389+0000) 2022-09-10T11:21:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:40.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:40.500277+0000) 2022-09-10T11:21:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:40 smithi104 ceph-mon[97044]: pgmap v13197: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:40 smithi110 ceph-mon[97633]: pgmap v13197: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:41.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:41.021584+0000) 2022-09-10T11:21:41.777 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:41.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:41.500449+0000) 2022-09-10T11:21:42.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:42.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:42.021826+0000) 2022-09-10T11:21:42.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:21:41.774+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:21:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:42.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:42.500658+0000) 2022-09-10T11:21:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:42 smithi104 ceph-mon[97044]: pgmap v13198: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:42 smithi110 ceph-mon[97633]: pgmap v13198: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:43.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:43.022054+0000) 2022-09-10T11:21:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:43.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:43.500889+0000) 2022-09-10T11:21:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:44.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:44.022347+0000) 2022-09-10T11:21:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:44.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:44.501140+0000) 2022-09-10T11:21:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:44 smithi110 ceph-mon[97633]: pgmap v13199: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:44 smithi104 ceph-mon[97044]: pgmap v13199: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:45.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:45.022586+0000) 2022-09-10T11:21:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:45.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:45.501390+0000) 2022-09-10T11:21:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:21:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:21:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:21:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:21:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:21:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:21:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:46.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:46.022838+0000) 2022-09-10T11:21:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:46.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:46.501555+0000) 2022-09-10T11:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:46 smithi110 ceph-mon[97633]: pgmap v13200: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:46 smithi104 ceph-mon[97044]: pgmap v13200: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:47.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:47.023082+0000) 2022-09-10T11:21:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:47.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:47.501771+0000) 2022-09-10T11:21:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:48.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:48.023287+0000) 2022-09-10T11:21:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:48.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:48.501981+0000) 2022-09-10T11:21:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:48 smithi104 ceph-mon[97044]: pgmap v13201: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:49.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:48 smithi110 ceph-mon[97633]: pgmap v13201: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:49.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:49.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:49.023565+0000) 2022-09-10T11:21:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:49.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:49.502198+0000) 2022-09-10T11:21:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:50.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:50.023729+0000) 2022-09-10T11:21:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:50.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:50.502491+0000) 2022-09-10T11:21:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:50 smithi104 ceph-mon[97044]: pgmap v13202: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:50 smithi110 ceph-mon[97633]: pgmap v13202: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:51.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:51.023929+0000) 2022-09-10T11:21:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:51.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:51.502700+0000) 2022-09-10T11:21:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:52.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:52.024068+0000) 2022-09-10T11:21:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:52.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:52.502925+0000) 2022-09-10T11:21:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:52 smithi104 ceph-mon[97044]: pgmap v13203: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:52 smithi110 ceph-mon[97633]: pgmap v13203: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:53.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:53.024252+0000) 2022-09-10T11:21:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:53.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:53.503152+0000) 2022-09-10T11:21:54.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:54.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:54.024575+0000) 2022-09-10T11:21:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:54.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:54.503397+0000) 2022-09-10T11:21:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:54 smithi110 ceph-mon[97633]: pgmap v13204: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:54 smithi104 ceph-mon[97044]: pgmap v13204: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:55.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:55.024835+0000) 2022-09-10T11:21:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:55.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:55.503536+0000) 2022-09-10T11:21:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:56.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:56.025033+0000) 2022-09-10T11:21:56.780 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:56.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:56.503728+0000) 2022-09-10T11:21:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:56 smithi104 ceph-mon[97044]: pgmap v13205: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:57.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:57.025245+0000) 2022-09-10T11:21:57.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:21:56.778+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:21:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:56 smithi110 ceph-mon[97633]: pgmap v13205: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:21:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:57.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:57.503971+0000) 2022-09-10T11:21:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:58.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:58.025465+0000) 2022-09-10T11:21:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:58.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:58.504203+0000) 2022-09-10T11:21:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:21:58 smithi104 ceph-mon[97044]: pgmap v13206: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:21:58 smithi110 ceph-mon[97633]: pgmap v13206: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:21:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:21:59.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:59.025623+0000) 2022-09-10T11:21:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:21:59.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:21:59.504387+0000) 2022-09-10T11:22:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:00.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:00.025817+0000) 2022-09-10T11:22:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:00.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:00.504553+0000) 2022-09-10T11:22:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:00 smithi104 ceph-mon[97044]: pgmap v13207: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:00 smithi110 ceph-mon[97633]: pgmap v13207: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:01.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:01.026058+0000) 2022-09-10T11:22:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:01.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:01.504768+0000) 2022-09-10T11:22:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:02.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:02.026241+0000) 2022-09-10T11:22:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:02.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:02.505000+0000) 2022-09-10T11:22:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:02 smithi104 ceph-mon[97044]: pgmap v13208: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:02 smithi110 ceph-mon[97633]: pgmap v13208: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:03.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:03.026481+0000) 2022-09-10T11:22:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:03.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:03.505185+0000) 2022-09-10T11:22:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:04.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:04.026758+0000) 2022-09-10T11:22:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:04.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:04.505439+0000) 2022-09-10T11:22:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:04 smithi110 ceph-mon[97633]: pgmap v13209: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:04 smithi104 ceph-mon[97044]: pgmap v13209: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:05.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:05.027022+0000) 2022-09-10T11:22:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:05.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:05.505665+0000) 2022-09-10T11:22:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:06.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:06.027191+0000) 2022-09-10T11:22:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:06.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:06.505869+0000) 2022-09-10T11:22:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:06 smithi104 ceph-mon[97044]: pgmap v13210: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:22:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:06 smithi110 ceph-mon[97633]: pgmap v13210: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:22:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:07.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:07.027454+0000) 2022-09-10T11:22:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:07.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:07.506050+0000) 2022-09-10T11:22:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:08.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:08.027711+0000) 2022-09-10T11:22:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:08.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:08.506247+0000) 2022-09-10T11:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:08 smithi104 ceph-mon[97044]: pgmap v13211: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:08 smithi104 ceph-mon[97044]: osdmap e906: 12 total, 12 up, 12 in 2022-09-10T11:22:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:08 smithi110 ceph-mon[97633]: pgmap v13211: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:22:09.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:08 smithi110 ceph-mon[97633]: osdmap e906: 12 total, 12 up, 12 in 2022-09-10T11:22:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:09.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:09.027900+0000) 2022-09-10T11:22:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:09.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:09.506471+0000) 2022-09-10T11:22:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:10.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:10.028082+0000) 2022-09-10T11:22:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:10.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:10.506714+0000) 2022-09-10T11:22:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:10 smithi104 ceph-mon[97044]: pgmap v13213: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:22:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:10 smithi110 ceph-mon[97633]: pgmap v13213: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:22:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:11.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:11.028288+0000) 2022-09-10T11:22:11.796 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:11.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:11.506950+0000) 2022-09-10T11:22:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:12.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:12.028499+0000) 2022-09-10T11:22:12.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:22:11.794+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:22:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:12.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:12.507239+0000) 2022-09-10T11:22:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:12 smithi104 ceph-mon[97044]: pgmap v13214: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:22:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:12 smithi110 ceph-mon[97633]: pgmap v13214: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:22:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:13.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:13.028780+0000) 2022-09-10T11:22:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:13.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:13.507385+0000) 2022-09-10T11:22:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:14.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:14.028935+0000) 2022-09-10T11:22:14.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:14.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:14.507585+0000) 2022-09-10T11:22:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:14 smithi110 ceph-mon[97633]: osdmap e907: 12 total, 12 up, 12 in 2022-09-10T11:22:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:14 smithi110 ceph-mon[97633]: pgmap v13216: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:22:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:14 smithi104 ceph-mon[97044]: osdmap e907: 12 total, 12 up, 12 in 2022-09-10T11:22:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:14 smithi104 ceph-mon[97044]: pgmap v13216: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:22:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:15.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:15.029083+0000) 2022-09-10T11:22:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:15.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:15.507727+0000) 2022-09-10T11:22:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:16.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:16.029331+0000) 2022-09-10T11:22:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:16.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:16.507980+0000) 2022-09-10T11:22:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:16 smithi104 ceph-mon[97044]: pgmap v13217: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:22:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:16 smithi110 ceph-mon[97633]: pgmap v13217: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:22:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:17.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:17.029509+0000) 2022-09-10T11:22:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:17.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:17.508186+0000) 2022-09-10T11:22:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:18.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:18.029772+0000) 2022-09-10T11:22:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:18.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:18.508393+0000) 2022-09-10T11:22:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:18 smithi104 ceph-mon[97044]: pgmap v13218: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:22:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:18 smithi110 ceph-mon[97633]: pgmap v13218: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:22:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:19.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:19.029962+0000) 2022-09-10T11:22:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:19.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:19.508605+0000) 2022-09-10T11:22:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:20.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:20.030207+0000) 2022-09-10T11:22:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:20.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:20.508829+0000) 2022-09-10T11:22:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:20 smithi104 ceph-mon[97044]: pgmap v13219: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:22:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:20 smithi110 ceph-mon[97633]: pgmap v13219: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:22:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:21.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:21.030431+0000) 2022-09-10T11:22:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:21.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:21.509077+0000) 2022-09-10T11:22:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:22.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:22.030595+0000) 2022-09-10T11:22:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:22.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:22.509374+0000) 2022-09-10T11:22:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:22 smithi104 ceph-mon[97044]: pgmap v13220: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:22:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:22 smithi110 ceph-mon[97633]: pgmap v13220: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:22:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:23.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:23.030818+0000) 2022-09-10T11:22:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:23.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:23.509583+0000) 2022-09-10T11:22:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:24.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:24.031040+0000) 2022-09-10T11:22:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:24.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:24.509791+0000) 2022-09-10T11:22:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:24 smithi110 ceph-mon[97633]: pgmap v13221: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:22:25.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:24 smithi104 ceph-mon[97044]: pgmap v13221: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:22:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:25.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:25.031239+0000) 2022-09-10T11:22:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:25.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:25.510035+0000) 2022-09-10T11:22:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:26.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:26.031441+0000) 2022-09-10T11:22:26.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:26.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:26.510251+0000) 2022-09-10T11:22:27.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:26 smithi104 ceph-mon[97044]: pgmap v13222: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:22:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:26 smithi110 ceph-mon[97633]: pgmap v13222: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:22:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:27.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:27.031618+0000) 2022-09-10T11:22:27.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:22:26.809+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:22:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:27.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:27.510450+0000) 2022-09-10T11:22:28.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:28.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:28.031800+0000) 2022-09-10T11:22:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:28.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:28.510660+0000) 2022-09-10T11:22:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:28 smithi104 ceph-mon[97044]: pgmap v13223: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:29.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:28 smithi110 ceph-mon[97633]: pgmap v13223: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:29.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:29.032018+0000) 2022-09-10T11:22:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:29.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:29.510891+0000) 2022-09-10T11:22:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:30.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:30.032192+0000) 2022-09-10T11:22:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:30.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:30.511078+0000) 2022-09-10T11:22:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:30 smithi104 ceph-mon[97044]: pgmap v13224: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:30 smithi110 ceph-mon[97633]: pgmap v13224: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:31.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:31.032460+0000) 2022-09-10T11:22:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:31.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:31.511266+0000) 2022-09-10T11:22:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:32.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:32.032637+0000) 2022-09-10T11:22:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:32.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:32.511452+0000) 2022-09-10T11:22:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:32 smithi104 ceph-mon[97044]: pgmap v13225: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:32 smithi110 ceph-mon[97633]: pgmap v13225: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:33.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:33.032901+0000) 2022-09-10T11:22:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:33.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:33.511638+0000) 2022-09-10T11:22:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:34.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:34.033199+0000) 2022-09-10T11:22:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:34.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:34.511866+0000) 2022-09-10T11:22:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:34 smithi110 ceph-mon[97633]: pgmap v13226: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:34 smithi104 ceph-mon[97044]: pgmap v13226: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:35.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:35.033420+0000) 2022-09-10T11:22:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:35.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:35.512082+0000) 2022-09-10T11:22:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:36.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:36.033628+0000) 2022-09-10T11:22:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:36.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:36.512381+0000) 2022-09-10T11:22:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:36 smithi104 ceph-mon[97044]: pgmap v13227: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:36 smithi110 ceph-mon[97633]: pgmap v13227: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:37.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:37.033897+0000) 2022-09-10T11:22:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:37.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:37.512638+0000) 2022-09-10T11:22:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:38.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:38.034084+0000) 2022-09-10T11:22:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:38.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:38.512875+0000) 2022-09-10T11:22:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:38 smithi104 ceph-mon[97044]: pgmap v13228: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:38 smithi110 ceph-mon[97633]: pgmap v13228: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:39.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:39.034320+0000) 2022-09-10T11:22:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:39.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:39.513060+0000) 2022-09-10T11:22:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:40.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:40.034463+0000) 2022-09-10T11:22:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:40.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:40.513274+0000) 2022-09-10T11:22:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:40 smithi104 ceph-mon[97044]: pgmap v13229: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:40 smithi110 ceph-mon[97633]: pgmap v13229: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:41.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:41.034668+0000) 2022-09-10T11:22:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:41.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:41.513525+0000) 2022-09-10T11:22:42.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:22:41.825+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:22:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:42.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:42.034892+0000) 2022-09-10T11:22:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:42.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:42.513756+0000) 2022-09-10T11:22:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:42 smithi104 ceph-mon[97044]: pgmap v13230: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:42 smithi110 ceph-mon[97633]: pgmap v13230: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:43.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:43.035066+0000) 2022-09-10T11:22:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:43.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:43.513933+0000) 2022-09-10T11:22:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:44.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:44.035276+0000) 2022-09-10T11:22:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:44.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:44.514186+0000) 2022-09-10T11:22:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:44 smithi110 ceph-mon[97633]: pgmap v13231: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:44 smithi104 ceph-mon[97044]: pgmap v13231: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:45.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:45.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:45.035523+0000) 2022-09-10T11:22:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:45.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:45.514405+0000) 2022-09-10T11:22:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:22:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:22:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:22:45.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:22:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:22:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:22:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:22:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:22:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:46.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:46.035773+0000) 2022-09-10T11:22:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:46.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:46.514665+0000) 2022-09-10T11:22:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:46 smithi110 ceph-mon[97633]: pgmap v13232: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:46 smithi104 ceph-mon[97044]: pgmap v13232: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:47.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:47.036040+0000) 2022-09-10T11:22:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:47.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:47.514882+0000) 2022-09-10T11:22:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:48.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:48.036305+0000) 2022-09-10T11:22:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:48.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:48.515131+0000) 2022-09-10T11:22:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:48 smithi104 ceph-mon[97044]: pgmap v13233: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:48 smithi110 ceph-mon[97633]: pgmap v13233: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:49.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:49.036502+0000) 2022-09-10T11:22:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:49.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:49.515357+0000) 2022-09-10T11:22:50.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:50.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:50.036674+0000) 2022-09-10T11:22:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:50.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:50.515585+0000) 2022-09-10T11:22:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:50 smithi104 ceph-mon[97044]: pgmap v13234: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:50 smithi110 ceph-mon[97633]: pgmap v13234: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:51.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:51.036895+0000) 2022-09-10T11:22:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:51.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:51.515794+0000) 2022-09-10T11:22:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:52.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:52.037145+0000) 2022-09-10T11:22:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:52.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:52.516024+0000) 2022-09-10T11:22:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:52 smithi104 ceph-mon[97044]: pgmap v13235: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:52 smithi110 ceph-mon[97633]: pgmap v13235: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:53.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:53.037275+0000) 2022-09-10T11:22:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:53.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:53.516255+0000) 2022-09-10T11:22:54.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:54.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:54.037526+0000) 2022-09-10T11:22:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:54.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:54.516420+0000) 2022-09-10T11:22:54.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:54 smithi110 ceph-mon[97633]: pgmap v13236: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:54 smithi104 ceph-mon[97044]: pgmap v13236: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:55.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:55.037788+0000) 2022-09-10T11:22:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:55.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:55.516661+0000) 2022-09-10T11:22:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:56.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:56.038009+0000) 2022-09-10T11:22:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:56.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:56.516916+0000) 2022-09-10T11:22:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:56 smithi104 ceph-mon[97044]: pgmap v13237: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:57.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:56 smithi110 ceph-mon[97633]: pgmap v13237: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:22:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:22:56.828+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:22:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:57.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:57.038228+0000) 2022-09-10T11:22:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:57.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:57.517129+0000) 2022-09-10T11:22:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:58.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:58.038416+0000) 2022-09-10T11:22:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:58.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:58.517369+0000) 2022-09-10T11:22:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:22:58 smithi104 ceph-mon[97044]: pgmap v13238: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:22:58 smithi110 ceph-mon[97633]: pgmap v13238: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:22:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:22:59.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:59.038617+0000) 2022-09-10T11:22:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:22:59.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:22:59.517603+0000) 2022-09-10T11:23:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:00.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:00.038816+0000) 2022-09-10T11:23:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:00.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:00.517848+0000) 2022-09-10T11:23:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:00 smithi104 ceph-mon[97044]: pgmap v13239: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:00 smithi110 ceph-mon[97633]: pgmap v13239: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:01.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:01.039039+0000) 2022-09-10T11:23:01.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:01.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:01.518120+0000) 2022-09-10T11:23:02.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:02.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:02.039235+0000) 2022-09-10T11:23:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:02.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:02.518284+0000) 2022-09-10T11:23:03.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:02 smithi104 ceph-mon[97044]: pgmap v13240: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:02 smithi110 ceph-mon[97633]: pgmap v13240: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:03.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:03.039488+0000) 2022-09-10T11:23:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:03.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:03.518542+0000) 2022-09-10T11:23:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:04.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:04.039756+0000) 2022-09-10T11:23:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:04.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:04.518741+0000) 2022-09-10T11:23:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:04 smithi110 ceph-mon[97633]: pgmap v13241: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:04 smithi104 ceph-mon[97044]: pgmap v13241: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:05.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:05.039975+0000) 2022-09-10T11:23:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:05.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:05.518979+0000) 2022-09-10T11:23:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:06.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:06.040228+0000) 2022-09-10T11:23:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:06.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:06.519186+0000) 2022-09-10T11:23:07.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:06 smithi104 ceph-mon[97044]: pgmap v13242: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:23:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:06 smithi110 ceph-mon[97633]: pgmap v13242: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:23:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:07.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:07.040457+0000) 2022-09-10T11:23:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:07.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:07.519383+0000) 2022-09-10T11:23:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:08.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:08.040647+0000) 2022-09-10T11:23:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:08.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:08.519633+0000) 2022-09-10T11:23:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:08 smithi110 ceph-mon[97633]: pgmap v13243: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:23:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:08 smithi110 ceph-mon[97633]: osdmap e908: 12 total, 12 up, 12 in 2022-09-10T11:23:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:08 smithi104 ceph-mon[97044]: pgmap v13243: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:23:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:08 smithi104 ceph-mon[97044]: osdmap e908: 12 total, 12 up, 12 in 2022-09-10T11:23:09.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:09.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:09.040883+0000) 2022-09-10T11:23:09.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:09.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:09.519878+0000) 2022-09-10T11:23:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:10.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:10.041086+0000) 2022-09-10T11:23:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:10.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:10.520175+0000) 2022-09-10T11:23:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:10 smithi110 ceph-mon[97633]: pgmap v13245: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:23:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:10 smithi104 ceph-mon[97044]: pgmap v13245: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:23:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:11.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:11.041293+0000) 2022-09-10T11:23:11.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:11.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:11.520349+0000) 2022-09-10T11:23:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:12.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:12.041464+0000) 2022-09-10T11:23:12.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:23:11.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:23:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:12.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:12.520585+0000) 2022-09-10T11:23:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:12 smithi110 ceph-mon[97633]: pgmap v13246: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:23:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:12 smithi104 ceph-mon[97044]: pgmap v13246: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:23:13.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:13.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:13.041680+0000) 2022-09-10T11:23:13.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:13.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:13.520802+0000) 2022-09-10T11:23:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:14.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:14.041946+0000) 2022-09-10T11:23:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:14.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:14.521070+0000) 2022-09-10T11:23:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:14 smithi110 ceph-mon[97633]: osdmap e909: 12 total, 12 up, 12 in 2022-09-10T11:23:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:14 smithi110 ceph-mon[97633]: pgmap v13248: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:23:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:14 smithi104 ceph-mon[97044]: osdmap e909: 12 total, 12 up, 12 in 2022-09-10T11:23:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:14 smithi104 ceph-mon[97044]: pgmap v13248: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:23:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:15.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:15.042201+0000) 2022-09-10T11:23:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:15.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:15.521313+0000) 2022-09-10T11:23:16.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:16.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:16.042400+0000) 2022-09-10T11:23:16.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:16.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:16.521575+0000) 2022-09-10T11:23:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:16 smithi104 ceph-mon[97044]: pgmap v13249: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:23:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:16 smithi110 ceph-mon[97633]: pgmap v13249: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:23:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:17.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:17.042606+0000) 2022-09-10T11:23:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:17.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:17.521781+0000) 2022-09-10T11:23:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:18.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:18.042817+0000) 2022-09-10T11:23:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:18.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:18.522038+0000) 2022-09-10T11:23:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:18 smithi104 ceph-mon[97044]: pgmap v13250: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:23:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:18 smithi110 ceph-mon[97633]: pgmap v13250: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:23:19.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:19.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:19.043065+0000) 2022-09-10T11:23:19.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:19.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:19.522240+0000) 2022-09-10T11:23:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:20.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:20.043318+0000) 2022-09-10T11:23:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:20.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:20.522484+0000) 2022-09-10T11:23:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:20 smithi110 ceph-mon[97633]: pgmap v13251: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:23:21.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:20 smithi104 ceph-mon[97044]: pgmap v13251: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:23:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:21.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:21.043517+0000) 2022-09-10T11:23:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:21.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:21.522665+0000) 2022-09-10T11:23:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:22.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:22.043718+0000) 2022-09-10T11:23:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:22.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:22.522844+0000) 2022-09-10T11:23:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:22 smithi110 ceph-mon[97633]: pgmap v13252: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:23:23.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:22 smithi104 ceph-mon[97044]: pgmap v13252: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:23:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:23.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:23.043890+0000) 2022-09-10T11:23:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:23.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:23.523063+0000) 2022-09-10T11:23:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:24.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:24.044168+0000) 2022-09-10T11:23:24.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:24.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:24.523399+0000) 2022-09-10T11:23:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:24 smithi110 ceph-mon[97633]: pgmap v13253: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:23:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:24 smithi104 ceph-mon[97044]: pgmap v13253: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:23:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:25.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:25.044404+0000) 2022-09-10T11:23:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:25.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:25.523594+0000) 2022-09-10T11:23:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:26.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:26.044632+0000) 2022-09-10T11:23:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:26.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:26.523792+0000) 2022-09-10T11:23:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:26 smithi104 ceph-mon[97044]: pgmap v13254: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:23:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:26 smithi110 ceph-mon[97633]: pgmap v13254: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:23:27.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:23:26.855+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:23:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:27.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:27.044887+0000) 2022-09-10T11:23:27.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:27.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:27.524066+0000) 2022-09-10T11:23:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:28.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:28.045087+0000) 2022-09-10T11:23:28.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:28.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:28.524326+0000) 2022-09-10T11:23:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:28 smithi110 ceph-mon[97633]: pgmap v13255: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:28 smithi104 ceph-mon[97044]: pgmap v13255: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:29.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:29.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:29.045265+0000) 2022-09-10T11:23:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:29.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:29.524611+0000) 2022-09-10T11:23:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:30.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:30.045496+0000) 2022-09-10T11:23:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:30.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:30.524839+0000) 2022-09-10T11:23:31.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:30 smithi110 ceph-mon[97633]: pgmap v13256: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:31.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:30 smithi104 ceph-mon[97044]: pgmap v13256: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:31.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:31.045708+0000) 2022-09-10T11:23:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:31.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:31.525063+0000) 2022-09-10T11:23:32.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:32.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:32.045985+0000) 2022-09-10T11:23:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:32.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:32.525290+0000) 2022-09-10T11:23:33.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:32 smithi110 ceph-mon[97633]: pgmap v13257: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:33.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:32 smithi104 ceph-mon[97044]: pgmap v13257: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:33.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:33.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:33.046186+0000) 2022-09-10T11:23:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:33.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:33.525455+0000) 2022-09-10T11:23:34.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:34.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:34.046399+0000) 2022-09-10T11:23:34.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:34.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:34.525681+0000) 2022-09-10T11:23:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:34 smithi110 ceph-mon[97633]: pgmap v13258: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:34 smithi104 ceph-mon[97044]: pgmap v13258: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:35.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:35.046635+0000) 2022-09-10T11:23:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:35.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:35.525915+0000) 2022-09-10T11:23:36.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:36.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:36.046862+0000) 2022-09-10T11:23:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:36.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:36.526201+0000) 2022-09-10T11:23:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:36 smithi110 ceph-mon[97633]: pgmap v13259: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:36 smithi104 ceph-mon[97044]: pgmap v13259: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:37.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:37.047151+0000) 2022-09-10T11:23:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:37.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:37.526436+0000) 2022-09-10T11:23:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:38.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:38.047319+0000) 2022-09-10T11:23:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:38.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:38.526616+0000) 2022-09-10T11:23:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:38 smithi110 ceph-mon[97633]: pgmap v13260: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:39.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:38 smithi104 ceph-mon[97044]: pgmap v13260: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:39.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:39.047571+0000) 2022-09-10T11:23:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:39.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:39.526780+0000) 2022-09-10T11:23:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:40.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:40.047828+0000) 2022-09-10T11:23:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:40.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:40.526961+0000) 2022-09-10T11:23:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:40 smithi110 ceph-mon[97633]: pgmap v13261: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:40 smithi104 ceph-mon[97044]: pgmap v13261: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:41.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:41.048078+0000) 2022-09-10T11:23:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:41.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:41.527174+0000) 2022-09-10T11:23:42.320 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:42.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:42.048312+0000) 2022-09-10T11:23:42.324 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:23:41.870+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:23:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:42.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:42.527395+0000) 2022-09-10T11:23:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:42 smithi110 ceph-mon[97633]: pgmap v13262: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:42 smithi104 ceph-mon[97044]: pgmap v13262: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:43.318 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:43.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:43.048508+0000) 2022-09-10T11:23:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:43.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:43.527617+0000) 2022-09-10T11:23:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:44.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:44.048712+0000) 2022-09-10T11:23:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:44.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:44.527874+0000) 2022-09-10T11:23:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:44 smithi110 ceph-mon[97633]: pgmap v13263: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:44 smithi104 ceph-mon[97044]: pgmap v13263: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:45.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:45.048886+0000) 2022-09-10T11:23:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:45.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:45.528144+0000) 2022-09-10T11:23:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:23:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:23:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:23:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:23:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:23:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:23:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:23:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:23:46.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:46.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:46.049138+0000) 2022-09-10T11:23:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:46.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:46.528391+0000) 2022-09-10T11:23:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:46 smithi110 ceph-mon[97633]: pgmap v13264: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:46 smithi104 ceph-mon[97044]: pgmap v13264: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:47.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:47.049348+0000) 2022-09-10T11:23:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:47.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:47.528599+0000) 2022-09-10T11:23:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:48.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:48.049488+0000) 2022-09-10T11:23:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:48.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:48.528815+0000) 2022-09-10T11:23:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:48 smithi110 ceph-mon[97633]: pgmap v13265: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:48 smithi104 ceph-mon[97044]: pgmap v13265: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:49.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:49.049707+0000) 2022-09-10T11:23:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:49.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:49.529053+0000) 2022-09-10T11:23:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:50.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:50.049954+0000) 2022-09-10T11:23:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:50.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:50.529334+0000) 2022-09-10T11:23:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:50 smithi110 ceph-mon[97633]: pgmap v13266: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:50 smithi104 ceph-mon[97044]: pgmap v13266: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:51.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:51.050209+0000) 2022-09-10T11:23:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:51.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:51.529526+0000) 2022-09-10T11:23:52.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:52.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:52.050453+0000) 2022-09-10T11:23:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:52.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:52.529703+0000) 2022-09-10T11:23:53.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:52 smithi110 ceph-mon[97633]: pgmap v13267: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:52 smithi104 ceph-mon[97044]: pgmap v13267: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:53.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:53.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:53.050669+0000) 2022-09-10T11:23:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:53.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:53.529894+0000) 2022-09-10T11:23:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:54.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:54.050847+0000) 2022-09-10T11:23:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:54.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:54.530088+0000) 2022-09-10T11:23:54.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:54 smithi110 ceph-mon[97633]: pgmap v13268: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:54 smithi104 ceph-mon[97044]: pgmap v13268: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:55.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:55.051071+0000) 2022-09-10T11:23:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:55.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:55.530292+0000) 2022-09-10T11:23:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:56.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:56.051219+0000) 2022-09-10T11:23:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:56.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:56.530508+0000) 2022-09-10T11:23:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:56 smithi110 ceph-mon[97633]: pgmap v13269: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:56 smithi104 ceph-mon[97044]: pgmap v13269: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:23:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:57.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:57.051433+0000) 2022-09-10T11:23:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:23:56.886+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:23:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:57.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:57.530770+0000) 2022-09-10T11:23:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:58.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:58.051678+0000) 2022-09-10T11:23:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:58.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:58.531011+0000) 2022-09-10T11:23:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:23:58 smithi110 ceph-mon[97633]: pgmap v13270: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:23:58 smithi104 ceph-mon[97044]: pgmap v13270: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:23:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:23:59.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:59.051864+0000) 2022-09-10T11:23:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:23:59.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:23:59.531192+0000) 2022-09-10T11:24:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:00.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:00.052051+0000) 2022-09-10T11:24:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:00.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:00.531431+0000) 2022-09-10T11:24:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:00 smithi110 ceph-mon[97633]: pgmap v13271: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:00 smithi104 ceph-mon[97044]: pgmap v13271: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:01.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:01.052291+0000) 2022-09-10T11:24:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:01.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:01.531711+0000) 2022-09-10T11:24:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:02.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:02.052448+0000) 2022-09-10T11:24:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:02.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:02.531916+0000) 2022-09-10T11:24:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:02 smithi110 ceph-mon[97633]: pgmap v13272: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:02 smithi104 ceph-mon[97044]: pgmap v13272: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:03.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:03.052640+0000) 2022-09-10T11:24:03.824 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:03.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:03.532189+0000) 2022-09-10T11:24:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:04.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:04.052822+0000) 2022-09-10T11:24:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:04.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:04.532447+0000) 2022-09-10T11:24:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:04 smithi110 ceph-mon[97633]: pgmap v13273: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:05.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:04 smithi104 ceph-mon[97044]: pgmap v13273: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:05.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:05.053080+0000) 2022-09-10T11:24:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:05.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:05.532606+0000) 2022-09-10T11:24:06.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:06.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:06.053360+0000) 2022-09-10T11:24:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:06.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:06.532828+0000) 2022-09-10T11:24:07.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:06 smithi110 ceph-mon[97633]: pgmap v13274: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:24:07.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:06 smithi104 ceph-mon[97044]: pgmap v13274: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:24:07.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:07.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:07.053585+0000) 2022-09-10T11:24:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:07.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:07.532981+0000) 2022-09-10T11:24:08.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:08.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:08.053794+0000) 2022-09-10T11:24:08.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:08.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:08.533158+0000) 2022-09-10T11:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:08 smithi110 ceph-mon[97633]: pgmap v13275: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:24:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:08 smithi110 ceph-mon[97633]: osdmap e910: 12 total, 12 up, 12 in 2022-09-10T11:24:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:08 smithi104 ceph-mon[97044]: pgmap v13275: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:24:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:08 smithi104 ceph-mon[97044]: osdmap e910: 12 total, 12 up, 12 in 2022-09-10T11:24:09.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:09.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:09.054021+0000) 2022-09-10T11:24:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:09.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:09.533345+0000) 2022-09-10T11:24:10.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:10.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:10.054283+0000) 2022-09-10T11:24:10.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:10.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:10.533572+0000) 2022-09-10T11:24:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:10 smithi110 ceph-mon[97633]: pgmap v13277: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:24:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:10 smithi104 ceph-mon[97044]: pgmap v13277: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:24:11.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:11.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:11.054475+0000) 2022-09-10T11:24:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:11.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:11.533791+0000) 2022-09-10T11:24:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:12.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:12.054652+0000) 2022-09-10T11:24:12.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:24:11.893+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:24:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:12.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:12.534012+0000) 2022-09-10T11:24:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:12 smithi110 ceph-mon[97633]: pgmap v13278: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:24:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:12 smithi104 ceph-mon[97044]: pgmap v13278: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:24:13.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:13.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:13.054877+0000) 2022-09-10T11:24:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:13.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:13.534250+0000) 2022-09-10T11:24:14.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:14.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:14.055198+0000) 2022-09-10T11:24:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:14.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:14.534468+0000) 2022-09-10T11:24:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:14 smithi110 ceph-mon[97633]: osdmap e911: 12 total, 12 up, 12 in 2022-09-10T11:24:14.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:14 smithi110 ceph-mon[97633]: pgmap v13280: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:24:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:14 smithi104 ceph-mon[97044]: osdmap e911: 12 total, 12 up, 12 in 2022-09-10T11:24:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:14 smithi104 ceph-mon[97044]: pgmap v13280: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:24:15.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:15.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:15.055415+0000) 2022-09-10T11:24:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:15.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:15.534617+0000) 2022-09-10T11:24:16.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:16.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:16.055635+0000) 2022-09-10T11:24:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:16.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:16.534837+0000) 2022-09-10T11:24:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:16 smithi110 ceph-mon[97633]: pgmap v13281: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:24:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:16 smithi104 ceph-mon[97044]: pgmap v13281: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:24:17.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:17.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:17.055881+0000) 2022-09-10T11:24:17.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:17.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:17.535113+0000) 2022-09-10T11:24:18.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:18.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:18.056140+0000) 2022-09-10T11:24:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:18.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:18.535251+0000) 2022-09-10T11:24:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:18 smithi110 ceph-mon[97633]: pgmap v13282: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:24:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:18 smithi104 ceph-mon[97044]: pgmap v13282: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:24:19.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:19.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:19.056377+0000) 2022-09-10T11:24:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:19.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:19.535439+0000) 2022-09-10T11:24:20.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:20.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:20.056650+0000) 2022-09-10T11:24:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:20.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:20.535698+0000) 2022-09-10T11:24:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:20 smithi110 ceph-mon[97633]: pgmap v13283: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:24:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:20 smithi104 ceph-mon[97044]: pgmap v13283: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:24:21.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:21.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:21.056883+0000) 2022-09-10T11:24:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:21.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:21.535957+0000) 2022-09-10T11:24:22.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:22.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:22.057179+0000) 2022-09-10T11:24:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:22.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:22.536201+0000) 2022-09-10T11:24:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:22 smithi110 ceph-mon[97633]: pgmap v13284: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:24:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:22 smithi104 ceph-mon[97044]: pgmap v13284: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:24:23.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:23.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:23.057375+0000) 2022-09-10T11:24:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:23.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:23.536412+0000) 2022-09-10T11:24:24.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:24.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:24.057669+0000) 2022-09-10T11:24:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:24.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:24.536599+0000) 2022-09-10T11:24:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:24 smithi110 ceph-mon[97633]: pgmap v13285: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T11:24:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:24 smithi104 ceph-mon[97044]: pgmap v13285: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 101 B/s wr, 9 op/s 2022-09-10T11:24:25.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:25.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:25.057947+0000) 2022-09-10T11:24:25.819 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:25.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:25.536800+0000) 2022-09-10T11:24:26.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:26.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:26.058206+0000) 2022-09-10T11:24:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:26.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:26.537020+0000) 2022-09-10T11:24:27.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:26 smithi110 ceph-mon[97633]: pgmap v13286: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:24:27.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:26 smithi104 ceph-mon[97044]: pgmap v13286: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:24:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:27.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:27.058450+0000) 2022-09-10T11:24:27.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:24:26.909+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:24:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:27.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:27.537183+0000) 2022-09-10T11:24:28.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:28.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:28.058703+0000) 2022-09-10T11:24:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:28.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:28.537415+0000) 2022-09-10T11:24:29.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:28 smithi110 ceph-mon[97633]: pgmap v13287: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:24:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:28 smithi104 ceph-mon[97044]: pgmap v13287: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:24:29.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:29.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:29.058918+0000) 2022-09-10T11:24:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:29.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:29.537625+0000) 2022-09-10T11:24:30.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:30.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:30.059146+0000) 2022-09-10T11:24:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:30.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:30.537801+0000) 2022-09-10T11:24:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:30 smithi110 ceph-mon[97633]: pgmap v13288: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:30 smithi104 ceph-mon[97044]: pgmap v13288: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:31.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:31.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:31.059356+0000) 2022-09-10T11:24:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:31.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:31.538030+0000) 2022-09-10T11:24:32.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:32.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:32.059547+0000) 2022-09-10T11:24:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:32.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:32.538219+0000) 2022-09-10T11:24:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:32 smithi110 ceph-mon[97633]: pgmap v13289: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:32 smithi104 ceph-mon[97044]: pgmap v13289: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:33.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:33.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:33.059763+0000) 2022-09-10T11:24:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:33.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:33.538392+0000) 2022-09-10T11:24:34.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:34.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:34.059914+0000) 2022-09-10T11:24:34.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:34.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:34.538563+0000) 2022-09-10T11:24:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:34 smithi110 ceph-mon[97633]: pgmap v13290: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:34 smithi104 ceph-mon[97044]: pgmap v13290: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:35.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:35.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:35.060148+0000) 2022-09-10T11:24:35.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:35.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:35.538742+0000) 2022-09-10T11:24:36.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:36.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:36.060310+0000) 2022-09-10T11:24:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:36.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:36.538960+0000) 2022-09-10T11:24:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:36 smithi110 ceph-mon[97633]: pgmap v13291: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:36 smithi104 ceph-mon[97044]: pgmap v13291: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:37.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:37.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:37.060507+0000) 2022-09-10T11:24:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:37.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:37.539188+0000) 2022-09-10T11:24:38.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:38.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:38.060675+0000) 2022-09-10T11:24:38.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:38.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:38.539342+0000) 2022-09-10T11:24:39.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:38 smithi110 ceph-mon[97633]: pgmap v13292: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:38 smithi104 ceph-mon[97044]: pgmap v13292: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:39.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:39.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:39.060930+0000) 2022-09-10T11:24:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:39.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:39.539574+0000) 2022-09-10T11:24:40.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:40.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:40.061210+0000) 2022-09-10T11:24:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:40.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:40.539846+0000) 2022-09-10T11:24:41.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:40 smithi110 ceph-mon[97633]: pgmap v13293: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:40 smithi104 ceph-mon[97044]: pgmap v13293: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:41.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:41.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:41.061434+0000) 2022-09-10T11:24:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:41.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:41.540027+0000) 2022-09-10T11:24:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:42.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:42.061701+0000) 2022-09-10T11:24:42.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:24:41.919+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:24:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:42.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:42.540268+0000) 2022-09-10T11:24:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:42 smithi110 ceph-mon[97633]: pgmap v13294: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:42 smithi104 ceph-mon[97044]: pgmap v13294: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:43.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:43.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:43.061882+0000) 2022-09-10T11:24:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:43.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:43.540477+0000) 2022-09-10T11:24:44.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:44.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:44.062149+0000) 2022-09-10T11:24:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:44.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:44.540665+0000) 2022-09-10T11:24:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:44 smithi110 ceph-mon[97633]: pgmap v13295: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:44 smithi104 ceph-mon[97044]: pgmap v13295: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:45.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:45.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:45.062345+0000) 2022-09-10T11:24:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:45.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:45.540871+0000) 2022-09-10T11:24:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:24:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:24:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:24:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:24:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:24:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:24:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:24:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:24:46.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:46.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:46.062528+0000) 2022-09-10T11:24:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:46.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:46.541059+0000) 2022-09-10T11:24:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:46 smithi110 ceph-mon[97633]: pgmap v13296: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:46 smithi104 ceph-mon[97044]: pgmap v13296: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:47.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:47.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:47.062796+0000) 2022-09-10T11:24:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:47.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:47.541215+0000) 2022-09-10T11:24:48.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:48.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:48.063023+0000) 2022-09-10T11:24:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:48.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:48.541397+0000) 2022-09-10T11:24:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:48 smithi110 ceph-mon[97633]: pgmap v13297: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:48 smithi104 ceph-mon[97044]: pgmap v13297: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:49.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:49.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:49.063210+0000) 2022-09-10T11:24:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:49.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:49.541602+0000) 2022-09-10T11:24:50.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:50.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:50.063440+0000) 2022-09-10T11:24:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:50.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:50.541831+0000) 2022-09-10T11:24:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:50 smithi110 ceph-mon[97633]: pgmap v13298: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:50 smithi104 ceph-mon[97044]: pgmap v13298: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:51.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:51.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:51.063662+0000) 2022-09-10T11:24:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:51.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:51.542041+0000) 2022-09-10T11:24:52.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:52.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:52.063863+0000) 2022-09-10T11:24:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:52.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:52.542279+0000) 2022-09-10T11:24:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:52 smithi110 ceph-mon[97633]: pgmap v13299: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:52 smithi104 ceph-mon[97044]: pgmap v13299: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:53.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:53.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:53.064042+0000) 2022-09-10T11:24:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:53.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:53.542476+0000) 2022-09-10T11:24:54.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:54.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:54.064316+0000) 2022-09-10T11:24:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:54.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:54.542745+0000) 2022-09-10T11:24:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:54 smithi110 ceph-mon[97633]: pgmap v13300: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:55.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:54 smithi104 ceph-mon[97044]: pgmap v13300: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:55.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:55.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:55.064529+0000) 2022-09-10T11:24:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:55.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:55.542928+0000) 2022-09-10T11:24:56.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:56.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:56.064702+0000) 2022-09-10T11:24:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:56.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:56.543190+0000) 2022-09-10T11:24:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:56 smithi110 ceph-mon[97633]: pgmap v13301: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:56 smithi104 ceph-mon[97044]: pgmap v13301: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:24:57.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:24:56.925+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:24:57.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:57.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:57.064946+0000) 2022-09-10T11:24:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:57.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:57.543396+0000) 2022-09-10T11:24:58.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:58.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:58.065146+0000) 2022-09-10T11:24:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:58.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:58.543640+0000) 2022-09-10T11:24:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:24:58 smithi110 ceph-mon[97633]: pgmap v13302: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:24:58 smithi104 ceph-mon[97044]: pgmap v13302: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:24:59.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:24:59.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:59.065298+0000) 2022-09-10T11:24:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:24:59.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:24:59.543854+0000) 2022-09-10T11:25:00.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:00.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:00.065441+0000) 2022-09-10T11:25:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:00.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:00.544147+0000) 2022-09-10T11:25:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:00 smithi110 ceph-mon[97633]: pgmap v13303: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:00 smithi104 ceph-mon[97044]: pgmap v13303: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:01.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:01.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:01.065579+0000) 2022-09-10T11:25:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:01.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:01.544358+0000) 2022-09-10T11:25:02.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:02.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:02.065792+0000) 2022-09-10T11:25:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:02.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:02.544573+0000) 2022-09-10T11:25:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:02 smithi110 ceph-mon[97633]: pgmap v13304: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:02 smithi104 ceph-mon[97044]: pgmap v13304: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:03.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:03.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:03.066042+0000) 2022-09-10T11:25:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:03.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:03.544821+0000) 2022-09-10T11:25:04.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:04.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:04.066192+0000) 2022-09-10T11:25:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:04.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:04.545054+0000) 2022-09-10T11:25:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:04 smithi110 ceph-mon[97633]: pgmap v13305: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:04 smithi104 ceph-mon[97044]: pgmap v13305: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:05.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:05.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:05.066366+0000) 2022-09-10T11:25:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:05.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:05.545279+0000) 2022-09-10T11:25:06.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:06.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:06.066585+0000) 2022-09-10T11:25:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:06.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:06.545443+0000) 2022-09-10T11:25:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:06 smithi110 ceph-mon[97633]: pgmap v13306: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:25:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:06 smithi104 ceph-mon[97044]: pgmap v13306: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:25:07.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:07.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:07.066803+0000) 2022-09-10T11:25:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:07.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:07.545580+0000) 2022-09-10T11:25:08.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:08.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:08.067043+0000) 2022-09-10T11:25:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:08.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:08.545801+0000) 2022-09-10T11:25:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:08 smithi110 ceph-mon[97633]: pgmap v13307: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:25:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:08 smithi110 ceph-mon[97633]: osdmap e912: 12 total, 12 up, 12 in 2022-09-10T11:25:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:08 smithi104 ceph-mon[97044]: pgmap v13307: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:25:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:08 smithi104 ceph-mon[97044]: osdmap e912: 12 total, 12 up, 12 in 2022-09-10T11:25:09.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:09.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:09.067260+0000) 2022-09-10T11:25:09.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:09.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:09.546013+0000) 2022-09-10T11:25:10.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:10.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:10.067453+0000) 2022-09-10T11:25:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:10.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:10.546263+0000) 2022-09-10T11:25:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:10 smithi110 ceph-mon[97633]: pgmap v13309: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:25:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:10 smithi104 ceph-mon[97044]: pgmap v13309: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:25:11.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:11.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:11.067641+0000) 2022-09-10T11:25:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:11.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:11.546451+0000) 2022-09-10T11:25:12.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:25:11.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:25:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:12.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:12.067837+0000) 2022-09-10T11:25:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:12.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:12.546657+0000) 2022-09-10T11:25:13.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:12 smithi110 ceph-mon[97633]: pgmap v13310: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:25:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:12 smithi104 ceph-mon[97044]: pgmap v13310: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:25:13.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:13.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:13.068147+0000) 2022-09-10T11:25:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:13.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:13.546845+0000) 2022-09-10T11:25:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:14.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:14.068423+0000) 2022-09-10T11:25:14.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:14.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:14.547021+0000) 2022-09-10T11:25:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:14 smithi110 ceph-mon[97633]: osdmap e913: 12 total, 12 up, 12 in 2022-09-10T11:25:14.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:14 smithi110 ceph-mon[97633]: pgmap v13312: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:25:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:14 smithi104 ceph-mon[97044]: osdmap e913: 12 total, 12 up, 12 in 2022-09-10T11:25:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:14 smithi104 ceph-mon[97044]: pgmap v13312: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:25:15.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:15.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:15.068656+0000) 2022-09-10T11:25:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:15.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:15.547235+0000) 2022-09-10T11:25:16.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:16.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:16.068834+0000) 2022-09-10T11:25:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:16.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:16.547470+0000) 2022-09-10T11:25:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:16 smithi110 ceph-mon[97633]: pgmap v13313: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:25:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:16 smithi104 ceph-mon[97044]: pgmap v13313: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:25:17.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:17.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:17.069006+0000) 2022-09-10T11:25:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:17.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:17.547732+0000) 2022-09-10T11:25:18.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:18.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:18.069215+0000) 2022-09-10T11:25:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:18.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:18.548031+0000) 2022-09-10T11:25:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:18 smithi110 ceph-mon[97633]: pgmap v13314: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:25:19.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:18 smithi104 ceph-mon[97044]: pgmap v13314: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:25:19.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:19.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:19.069368+0000) 2022-09-10T11:25:19.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:19.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:19.548271+0000) 2022-09-10T11:25:20.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:20.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:20.069586+0000) 2022-09-10T11:25:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:20.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:20.548496+0000) 2022-09-10T11:25:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:20 smithi110 ceph-mon[97633]: pgmap v13315: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:25:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:20 smithi104 ceph-mon[97044]: pgmap v13315: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:25:21.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:21.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:21.069829+0000) 2022-09-10T11:25:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:21.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:21.548792+0000) 2022-09-10T11:25:22.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:22.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:22.070010+0000) 2022-09-10T11:25:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:22.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:22.549022+0000) 2022-09-10T11:25:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:22 smithi110 ceph-mon[97633]: pgmap v13316: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:25:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:22 smithi104 ceph-mon[97044]: pgmap v13316: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:25:23.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:23.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:23.070250+0000) 2022-09-10T11:25:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:23.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:23.549231+0000) 2022-09-10T11:25:24.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:24.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:24.070516+0000) 2022-09-10T11:25:24.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:24.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:24.549456+0000) 2022-09-10T11:25:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:24 smithi104 ceph-mon[97044]: pgmap v13317: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:25:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:24 smithi110 ceph-mon[97633]: pgmap v13317: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:25:25.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:25.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:25.070764+0000) 2022-09-10T11:25:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:25.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:25.549638+0000) 2022-09-10T11:25:26.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:26.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:26.070975+0000) 2022-09-10T11:25:26.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:26.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:26.549836+0000) 2022-09-10T11:25:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:26 smithi110 ceph-mon[97633]: pgmap v13318: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:25:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:26 smithi104 ceph-mon[97044]: pgmap v13318: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:25:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:27.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:27.071159+0000) 2022-09-10T11:25:27.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:25:26.955+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:25:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:27.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:27.550061+0000) 2022-09-10T11:25:28.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:28.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:28.071381+0000) 2022-09-10T11:25:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:28.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:28.550246+0000) 2022-09-10T11:25:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:28 smithi110 ceph-mon[97633]: pgmap v13319: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:28 smithi104 ceph-mon[97044]: pgmap v13319: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:29.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:29.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:29.071561+0000) 2022-09-10T11:25:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:29.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:29.550432+0000) 2022-09-10T11:25:30.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:30.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:30.071761+0000) 2022-09-10T11:25:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:30.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:30.550596+0000) 2022-09-10T11:25:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:30 smithi110 ceph-mon[97633]: pgmap v13320: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:31.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:30 smithi104 ceph-mon[97044]: pgmap v13320: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:31.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:31.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:31.071961+0000) 2022-09-10T11:25:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:31.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:31.550768+0000) 2022-09-10T11:25:32.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:32.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:32.072179+0000) 2022-09-10T11:25:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:32.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:32.550999+0000) 2022-09-10T11:25:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:32 smithi110 ceph-mon[97633]: pgmap v13321: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:32 smithi104 ceph-mon[97044]: pgmap v13321: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:33.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:33.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:33.072312+0000) 2022-09-10T11:25:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:33.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:33.551207+0000) 2022-09-10T11:25:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:34.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:34.072538+0000) 2022-09-10T11:25:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:34.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:34.551333+0000) 2022-09-10T11:25:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:34 smithi104 ceph-mon[97044]: pgmap v13322: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:34 smithi110 ceph-mon[97633]: pgmap v13322: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:35.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:35.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:35.072764+0000) 2022-09-10T11:25:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:35.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:35.551498+0000) 2022-09-10T11:25:36.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:36.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:36.072973+0000) 2022-09-10T11:25:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:36.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:36.551671+0000) 2022-09-10T11:25:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:36 smithi110 ceph-mon[97633]: pgmap v13323: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:36 smithi104 ceph-mon[97044]: pgmap v13323: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:37.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:37.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:37.073165+0000) 2022-09-10T11:25:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:37.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:37.551906+0000) 2022-09-10T11:25:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:38.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:38.073388+0000) 2022-09-10T11:25:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:38.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:38.552031+0000) 2022-09-10T11:25:39.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:38 smithi110 ceph-mon[97633]: pgmap v13324: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:39.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:38 smithi104 ceph-mon[97044]: pgmap v13324: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:39.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:39.073540+0000) 2022-09-10T11:25:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:39.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:39.552164+0000) 2022-09-10T11:25:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:40.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:40.073782+0000) 2022-09-10T11:25:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:40.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:40.552308+0000) 2022-09-10T11:25:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:40 smithi110 ceph-mon[97633]: pgmap v13325: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:40 smithi104 ceph-mon[97044]: pgmap v13325: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:41.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:41.074009+0000) 2022-09-10T11:25:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:41.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:41.552503+0000) 2022-09-10T11:25:42.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:25:41.968+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:25:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:42.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:42.074301+0000) 2022-09-10T11:25:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:42.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:42.552717+0000) 2022-09-10T11:25:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:42 smithi110 ceph-mon[97633]: pgmap v13326: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:42 smithi104 ceph-mon[97044]: pgmap v13326: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:43.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:43.074533+0000) 2022-09-10T11:25:43.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:43.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:43.552890+0000) 2022-09-10T11:25:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:44.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:44.074734+0000) 2022-09-10T11:25:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:44.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:44.553090+0000) 2022-09-10T11:25:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:44 smithi104 ceph-mon[97044]: pgmap v13327: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:44 smithi110 ceph-mon[97633]: pgmap v13327: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:45.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:45.074984+0000) 2022-09-10T11:25:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:25:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:25:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:25:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:25:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:45.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:45.553368+0000) 2022-09-10T11:25:46.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:25:46.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:25:46.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:25:46.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:25:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:46.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:46.075188+0000) 2022-09-10T11:25:47.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:46 smithi104 ceph-mon[97044]: pgmap v13328: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:46.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:46.553585+0000) 2022-09-10T11:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:46 smithi110 ceph-mon[97633]: pgmap v13328: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:47.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:47.075349+0000) 2022-09-10T11:25:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:47.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:47.553889+0000) 2022-09-10T11:25:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:48.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:48.075564+0000) 2022-09-10T11:25:49.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:48.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:48.554033+0000) 2022-09-10T11:25:49.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:48 smithi110 ceph-mon[97633]: pgmap v13329: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:49.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:48 smithi104 ceph-mon[97044]: pgmap v13329: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:49.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:49.075752+0000) 2022-09-10T11:25:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:49.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:49.554166+0000) 2022-09-10T11:25:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:50.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:50.076009+0000) 2022-09-10T11:25:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:50.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:50.554293+0000) 2022-09-10T11:25:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:50 smithi110 ceph-mon[97633]: pgmap v13330: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:50 smithi104 ceph-mon[97044]: pgmap v13330: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:51.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:51.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:51.076214+0000) 2022-09-10T11:25:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:51.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:51.554417+0000) 2022-09-10T11:25:52.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:52.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:52.076464+0000) 2022-09-10T11:25:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:52.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:52.554598+0000) 2022-09-10T11:25:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:52 smithi110 ceph-mon[97633]: pgmap v13331: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:52 smithi104 ceph-mon[97044]: pgmap v13331: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:53.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:53.076699+0000) 2022-09-10T11:25:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:53.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:53.554781+0000) 2022-09-10T11:25:54.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:54.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:54.076934+0000) 2022-09-10T11:25:55.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:54 smithi104 ceph-mon[97044]: pgmap v13332: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:54.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:54.554968+0000) 2022-09-10T11:25:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:54 smithi110 ceph-mon[97633]: pgmap v13332: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:55.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:55.077190+0000) 2022-09-10T11:25:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:55.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:55.555150+0000) 2022-09-10T11:25:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:56.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:56.077451+0000) 2022-09-10T11:25:56.978 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:56.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:56.555289+0000) 2022-09-10T11:25:57.173 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:56 smithi110 ceph-mon[97633]: pgmap v13333: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:57.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:56 smithi104 ceph-mon[97044]: pgmap v13333: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:25:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:57.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:57.077669+0000) 2022-09-10T11:25:57.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:25:56.973+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:25:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:57.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:57.555450+0000) 2022-09-10T11:25:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:58.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:58.077918+0000) 2022-09-10T11:25:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:58.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:58.555629+0000) 2022-09-10T11:25:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:25:58 smithi110 ceph-mon[97633]: pgmap v13334: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:25:58 smithi104 ceph-mon[97044]: pgmap v13334: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:25:59.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:25:59.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:59.078187+0000) 2022-09-10T11:26:00.072 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:25:59.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:25:59.555781+0000) 2022-09-10T11:26:00.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:00.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:00.078388+0000) 2022-09-10T11:26:01.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:00.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:00.555929+0000) 2022-09-10T11:26:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:00 smithi110 ceph-mon[97633]: pgmap v13335: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:00 smithi104 ceph-mon[97044]: pgmap v13335: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:01.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:01.078618+0000) 2022-09-10T11:26:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:01.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:01.556075+0000) 2022-09-10T11:26:02.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:02.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:02.078923+0000) 2022-09-10T11:26:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:02.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:02.556203+0000) 2022-09-10T11:26:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:02 smithi110 ceph-mon[97633]: pgmap v13336: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:03.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:02 smithi104 ceph-mon[97044]: pgmap v13336: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:03.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:03.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:03.079212+0000) 2022-09-10T11:26:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:03.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:03.556377+0000) 2022-09-10T11:26:04.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:04.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:04.079472+0000) 2022-09-10T11:26:05.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:04 smithi104 ceph-mon[97044]: pgmap v13337: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:05.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:04.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:04.556555+0000) 2022-09-10T11:26:05.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:04 smithi110 ceph-mon[97633]: pgmap v13337: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:05.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:05.079727+0000) 2022-09-10T11:26:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:05.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:05.556734+0000) 2022-09-10T11:26:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:06.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:06.079950+0000) 2022-09-10T11:26:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:06.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:06.556908+0000) 2022-09-10T11:26:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:06 smithi110 ceph-mon[97633]: pgmap v13338: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:26:07.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:06 smithi104 ceph-mon[97044]: pgmap v13338: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:26:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:07.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:07.080089+0000) 2022-09-10T11:26:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:07.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:07.557052+0000) 2022-09-10T11:26:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:08.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:08.080300+0000) 2022-09-10T11:26:09.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:08.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:08.557213+0000) 2022-09-10T11:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:08 smithi110 ceph-mon[97633]: pgmap v13339: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:26:09.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:08 smithi110 ceph-mon[97633]: osdmap e914: 12 total, 12 up, 12 in 2022-09-10T11:26:09.305 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:08 smithi104 ceph-mon[97044]: pgmap v13339: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:26:09.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:08 smithi104 ceph-mon[97044]: osdmap e914: 12 total, 12 up, 12 in 2022-09-10T11:26:09.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:09.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:09.080524+0000) 2022-09-10T11:26:10.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:09.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:09.557379+0000) 2022-09-10T11:26:10.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:10.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:10.080800+0000) 2022-09-10T11:26:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:10.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:10.557546+0000) 2022-09-10T11:26:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:10 smithi110 ceph-mon[97633]: pgmap v13341: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:26:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:10 smithi104 ceph-mon[97044]: pgmap v13341: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:26:11.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:11.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:11.081017+0000) 2022-09-10T11:26:11.984 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:11.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:11.557698+0000) 2022-09-10T11:26:12.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:26:11.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:26:12.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:12.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:12.081233+0000) 2022-09-10T11:26:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:12.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:12.557970+0000) 2022-09-10T11:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:12 smithi110 ceph-mon[97633]: pgmap v13342: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:26:13.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:12 smithi104 ceph-mon[97044]: pgmap v13342: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:26:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:13.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:13.081439+0000) 2022-09-10T11:26:14.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:13.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:13.558131+0000) 2022-09-10T11:26:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:14.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:14.081698+0000) 2022-09-10T11:26:15.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:14 smithi104 ceph-mon[97044]: osdmap e915: 12 total, 12 up, 12 in 2022-09-10T11:26:15.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:14 smithi104 ceph-mon[97044]: pgmap v13344: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:26:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:14.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:14.558379+0000) 2022-09-10T11:26:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:14 smithi110 ceph-mon[97633]: osdmap e915: 12 total, 12 up, 12 in 2022-09-10T11:26:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:14 smithi110 ceph-mon[97633]: pgmap v13344: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:26:15.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:15.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:15.081889+0000) 2022-09-10T11:26:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:15.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:15.558579+0000) 2022-09-10T11:26:16.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:16.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:16.082140+0000) 2022-09-10T11:26:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:16.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:16.558788+0000) 2022-09-10T11:26:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:16 smithi110 ceph-mon[97633]: pgmap v13345: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:26:17.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:16 smithi104 ceph-mon[97044]: pgmap v13345: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:26:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:17.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:17.082392+0000) 2022-09-10T11:26:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:17.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:17.558993+0000) 2022-09-10T11:26:18.569 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:18.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:18.082682+0000) 2022-09-10T11:26:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:18.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:18.559199+0000) 2022-09-10T11:26:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:18 smithi110 ceph-mon[97633]: pgmap v13346: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:26:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:18 smithi104 ceph-mon[97044]: pgmap v13346: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:26:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:19.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:19.082897+0000) 2022-09-10T11:26:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:19.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:19.559326+0000) 2022-09-10T11:26:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:20.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:20.083174+0000) 2022-09-10T11:26:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:20.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:20.559511+0000) 2022-09-10T11:26:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:20 smithi110 ceph-mon[97633]: pgmap v13347: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:26:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:20 smithi104 ceph-mon[97044]: pgmap v13347: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:26:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:21.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:21.083376+0000) 2022-09-10T11:26:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:21.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:21.559724+0000) 2022-09-10T11:26:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:22.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:22.083608+0000) 2022-09-10T11:26:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:22.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:22.559937+0000) 2022-09-10T11:26:23.173 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:22 smithi110 ceph-mon[97633]: pgmap v13348: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:26:23.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:22 smithi104 ceph-mon[97044]: pgmap v13348: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:26:23.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:23.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:23.083757+0000) 2022-09-10T11:26:24.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:23.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:23.560194+0000) 2022-09-10T11:26:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:24.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:24.083958+0000) 2022-09-10T11:26:25.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:24 smithi104 ceph-mon[97044]: pgmap v13349: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:26:25.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:24.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:24.560359+0000) 2022-09-10T11:26:25.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:24 smithi110 ceph-mon[97633]: pgmap v13349: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:26:25.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:25.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:25.084169+0000) 2022-09-10T11:26:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:25.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:25.560544+0000) 2022-09-10T11:26:26.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:26.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:26.084379+0000) 2022-09-10T11:26:27.006 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:26.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:26.560775+0000) 2022-09-10T11:26:27.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:26 smithi110 ceph-mon[97633]: pgmap v13350: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:26:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:26 smithi104 ceph-mon[97044]: pgmap v13350: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:26:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:27.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:27.084577+0000) 2022-09-10T11:26:27.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:26:26.995+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:26:28.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:27.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:27.561015+0000) 2022-09-10T11:26:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:28.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:28.084793+0000) 2022-09-10T11:26:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:28.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:28.561342+0000) 2022-09-10T11:26:29.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:28 smithi110 ceph-mon[97633]: pgmap v13351: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:29.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:28 smithi104 ceph-mon[97044]: pgmap v13351: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:29.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:29.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:29.085013+0000) 2022-09-10T11:26:30.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:29.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:29.561502+0000) 2022-09-10T11:26:30.569 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:30.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:30.085209+0000) 2022-09-10T11:26:31.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:30.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:30.561762+0000) 2022-09-10T11:26:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:30 smithi110 ceph-mon[97633]: pgmap v13352: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:31.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:30 smithi104 ceph-mon[97044]: pgmap v13352: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:31.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:31.085344+0000) 2022-09-10T11:26:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:31.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:31.561965+0000) 2022-09-10T11:26:32.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:32.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:32.085496+0000) 2022-09-10T11:26:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:32.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:32.562209+0000) 2022-09-10T11:26:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:32 smithi110 ceph-mon[97633]: pgmap v13353: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:33.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:32 smithi104 ceph-mon[97044]: pgmap v13353: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:33.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:33.085717+0000) 2022-09-10T11:26:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:33.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:33.562493+0000) 2022-09-10T11:26:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:34.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:34.085943+0000) 2022-09-10T11:26:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:34 smithi104 ceph-mon[97044]: pgmap v13354: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:34.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:34.562650+0000) 2022-09-10T11:26:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:34 smithi110 ceph-mon[97633]: pgmap v13354: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:35.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:35.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:35.086189+0000) 2022-09-10T11:26:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:35.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:35.562822+0000) 2022-09-10T11:26:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:36.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:36.086459+0000) 2022-09-10T11:26:37.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:36.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:36.563033+0000) 2022-09-10T11:26:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:36 smithi110 ceph-mon[97633]: pgmap v13355: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:36 smithi104 ceph-mon[97044]: pgmap v13355: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:37.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:37.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:37.086657+0000) 2022-09-10T11:26:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:37.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:37.563207+0000) 2022-09-10T11:26:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:38.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:38.086822+0000) 2022-09-10T11:26:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:38.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:38.563397+0000) 2022-09-10T11:26:39.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:38 smithi110 ceph-mon[97633]: pgmap v13356: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:39.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:38 smithi104 ceph-mon[97044]: pgmap v13356: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:39.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:39.087070+0000) 2022-09-10T11:26:40.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:39.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:39.563632+0000) 2022-09-10T11:26:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:40.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:40.087345+0000) 2022-09-10T11:26:41.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:40.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:40.563874+0000) 2022-09-10T11:26:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:40 smithi110 ceph-mon[97633]: pgmap v13357: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:40 smithi104 ceph-mon[97044]: pgmap v13357: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:41.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:41.087510+0000) 2022-09-10T11:26:42.002 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:41.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:41.564126+0000) 2022-09-10T11:26:42.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:26:41.999+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:26:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:42.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:42.087791+0000) 2022-09-10T11:26:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:42.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:42.564290+0000) 2022-09-10T11:26:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:42 smithi110 ceph-mon[97633]: pgmap v13358: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:43.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:42 smithi104 ceph-mon[97044]: pgmap v13358: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:43.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:43.087994+0000) 2022-09-10T11:26:44.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:43.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:43.564460+0000) 2022-09-10T11:26:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:44.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:44.088247+0000) 2022-09-10T11:26:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:44 smithi104 ceph-mon[97044]: pgmap v13359: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:44.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:44.564682+0000) 2022-09-10T11:26:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:44 smithi110 ceph-mon[97633]: pgmap v13359: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:45.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:45.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:45.088426+0000) 2022-09-10T11:26:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:26:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:26:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:26:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:26:46.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:45.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:45.564919+0000) 2022-09-10T11:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:26:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:26:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:26:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:46.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:46.088618+0000) 2022-09-10T11:26:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:46 smithi104 ceph-mon[97044]: pgmap v13360: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:47.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:46.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:46.565187+0000) 2022-09-10T11:26:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:46 smithi110 ceph-mon[97633]: pgmap v13360: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:47.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:47.088854+0000) 2022-09-10T11:26:48.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:47.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:47.565367+0000) 2022-09-10T11:26:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:48.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:48.089048+0000) 2022-09-10T11:26:49.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:48.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:48.565628+0000) 2022-09-10T11:26:49.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:48 smithi110 ceph-mon[97633]: pgmap v13361: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:48 smithi104 ceph-mon[97044]: pgmap v13361: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:49.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:49.089237+0000) 2022-09-10T11:26:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:49.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:49.565821+0000) 2022-09-10T11:26:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:50.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:50.089439+0000) 2022-09-10T11:26:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:50.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:50.566085+0000) 2022-09-10T11:26:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:50 smithi110 ceph-mon[97633]: pgmap v13362: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:50 smithi104 ceph-mon[97044]: pgmap v13362: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:51.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:51.089660+0000) 2022-09-10T11:26:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:51.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:51.566295+0000) 2022-09-10T11:26:52.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:52.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:52.089880+0000) 2022-09-10T11:26:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:52.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:52.566453+0000) 2022-09-10T11:26:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:52 smithi110 ceph-mon[97633]: pgmap v13363: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:53.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:52 smithi104 ceph-mon[97044]: pgmap v13363: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:53.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:53.090085+0000) 2022-09-10T11:26:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:53.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:53.566638+0000) 2022-09-10T11:26:54.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:54.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:54.090373+0000) 2022-09-10T11:26:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:54 smithi104 ceph-mon[97044]: pgmap v13364: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:54.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:54.566881+0000) 2022-09-10T11:26:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:54 smithi110 ceph-mon[97633]: pgmap v13364: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:55.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:55.090629+0000) 2022-09-10T11:26:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:55.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:55.567169+0000) 2022-09-10T11:26:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:56.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:56.090905+0000) 2022-09-10T11:26:57.015 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:56.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:56.567407+0000) 2022-09-10T11:26:57.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:56 smithi110 ceph-mon[97633]: pgmap v13365: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:56 smithi104 ceph-mon[97044]: pgmap v13365: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:26:57.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:26:57.013+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:26:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:57.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:57.091145+0000) 2022-09-10T11:26:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:57.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:57.567637+0000) 2022-09-10T11:26:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:58.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:58.091394+0000) 2022-09-10T11:26:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:58.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:58.567773+0000) 2022-09-10T11:26:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:26:58 smithi110 ceph-mon[97633]: pgmap v13366: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:26:58 smithi104 ceph-mon[97044]: pgmap v13366: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:26:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:26:59.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:59.091586+0000) 2022-09-10T11:27:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:26:59.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:26:59.567955+0000) 2022-09-10T11:27:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:00.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:00.091798+0000) 2022-09-10T11:27:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:00.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:00.568183+0000) 2022-09-10T11:27:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:00 smithi110 ceph-mon[97633]: pgmap v13367: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:01.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:00 smithi104 ceph-mon[97044]: pgmap v13367: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:01.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:01.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:01.092013+0000) 2022-09-10T11:27:02.066 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:01.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:01.568337+0000) 2022-09-10T11:27:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:02.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:02.092270+0000) 2022-09-10T11:27:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:02.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:02.568548+0000) 2022-09-10T11:27:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:02 smithi110 ceph-mon[97633]: pgmap v13368: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:02 smithi104 ceph-mon[97044]: pgmap v13368: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:03.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:03.092524+0000) 2022-09-10T11:27:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:03.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:03.568790+0000) 2022-09-10T11:27:04.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:04.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:04.092756+0000) 2022-09-10T11:27:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:04 smithi104 ceph-mon[97044]: pgmap v13369: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:04.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:04.568997+0000) 2022-09-10T11:27:05.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:04 smithi110 ceph-mon[97633]: pgmap v13369: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:05.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:05.092965+0000) 2022-09-10T11:27:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:05.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:05.569246+0000) 2022-09-10T11:27:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:06.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:06.093142+0000) 2022-09-10T11:27:07.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:06.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:06.569457+0000) 2022-09-10T11:27:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:06 smithi110 ceph-mon[97633]: pgmap v13370: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:27:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:06 smithi104 ceph-mon[97044]: pgmap v13370: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:27:07.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:07.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:07.093375+0000) 2022-09-10T11:27:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:07.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:07.569659+0000) 2022-09-10T11:27:08.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:08.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:08.093551+0000) 2022-09-10T11:27:09.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:08.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:08.569886+0000) 2022-09-10T11:27:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:08 smithi110 ceph-mon[97633]: pgmap v13371: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:27:09.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:08 smithi110 ceph-mon[97633]: osdmap e916: 12 total, 12 up, 12 in 2022-09-10T11:27:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:08 smithi104 ceph-mon[97044]: pgmap v13371: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:27:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:08 smithi104 ceph-mon[97044]: osdmap e916: 12 total, 12 up, 12 in 2022-09-10T11:27:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:09.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:09.093766+0000) 2022-09-10T11:27:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:09.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:09.570125+0000) 2022-09-10T11:27:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:10.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:10.093996+0000) 2022-09-10T11:27:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:10.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:10.570348+0000) 2022-09-10T11:27:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:10 smithi110 ceph-mon[97633]: pgmap v13373: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:27:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:10 smithi104 ceph-mon[97044]: pgmap v13373: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:27:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:11.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:11.094228+0000) 2022-09-10T11:27:12.031 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:11.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:11.570506+0000) 2022-09-10T11:27:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:12.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:12.094424+0000) 2022-09-10T11:27:12.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:27:12.027+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:27:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:12.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:12.570722+0000) 2022-09-10T11:27:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:12 smithi110 ceph-mon[97633]: pgmap v13374: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:27:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:12 smithi104 ceph-mon[97044]: pgmap v13374: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:27:13.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:13.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:13.094673+0000) 2022-09-10T11:27:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:13.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:13.570930+0000) 2022-09-10T11:27:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:14.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:14.094977+0000) 2022-09-10T11:27:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:14 smithi104 ceph-mon[97044]: osdmap e917: 12 total, 12 up, 12 in 2022-09-10T11:27:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:14 smithi104 ceph-mon[97044]: pgmap v13376: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:27:15.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:14.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:14.571125+0000) 2022-09-10T11:27:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:14 smithi110 ceph-mon[97633]: osdmap e917: 12 total, 12 up, 12 in 2022-09-10T11:27:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:14 smithi110 ceph-mon[97633]: pgmap v13376: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:27:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:15.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:15.095187+0000) 2022-09-10T11:27:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:15.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:15.571267+0000) 2022-09-10T11:27:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:16.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:16.095406+0000) 2022-09-10T11:27:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:16.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:16.571483+0000) 2022-09-10T11:27:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:16 smithi110 ceph-mon[97633]: pgmap v13377: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:27:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:16 smithi104 ceph-mon[97044]: pgmap v13377: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:27:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:17.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:17.095591+0000) 2022-09-10T11:27:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:17.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:17.571746+0000) 2022-09-10T11:27:18.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:18.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:18.095840+0000) 2022-09-10T11:27:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:18.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:18.571960+0000) 2022-09-10T11:27:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:18 smithi110 ceph-mon[97633]: pgmap v13378: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:27:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:18 smithi104 ceph-mon[97044]: pgmap v13378: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:27:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:19.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:19.096120+0000) 2022-09-10T11:27:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:19.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:19.572199+0000) 2022-09-10T11:27:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:20.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:20.096271+0000) 2022-09-10T11:27:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:20.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:20.572461+0000) 2022-09-10T11:27:21.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:20 smithi110 ceph-mon[97633]: pgmap v13379: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:27:21.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:20 smithi104 ceph-mon[97044]: pgmap v13379: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:27:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:21.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:21.096463+0000) 2022-09-10T11:27:22.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:21.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:21.572608+0000) 2022-09-10T11:27:22.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:22.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:22.096690+0000) 2022-09-10T11:27:23.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:22.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:22.572870+0000) 2022-09-10T11:27:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:22 smithi110 ceph-mon[97633]: pgmap v13380: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:27:23.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:22 smithi104 ceph-mon[97044]: pgmap v13380: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:27:23.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:23.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:23.096935+0000) 2022-09-10T11:27:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:23.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:23.573080+0000) 2022-09-10T11:27:24.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:24.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:24.097223+0000) 2022-09-10T11:27:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:24 smithi104 ceph-mon[97044]: pgmap v13381: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:27:25.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:24.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:24.573290+0000) 2022-09-10T11:27:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:24 smithi110 ceph-mon[97633]: pgmap v13381: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:27:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:25.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:25.097459+0000) 2022-09-10T11:27:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:25.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:25.573514+0000) 2022-09-10T11:27:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:26.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:26.097709+0000) 2022-09-10T11:27:27.047 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:26.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:26.573760+0000) 2022-09-10T11:27:27.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:26 smithi110 ceph-mon[97633]: pgmap v13382: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:27:27.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:26 smithi104 ceph-mon[97044]: pgmap v13382: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:27:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:27.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:27.097980+0000) 2022-09-10T11:27:27.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:27:27.043+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:27:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:27.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:27.574036+0000) 2022-09-10T11:27:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:28.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:28.098204+0000) 2022-09-10T11:27:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:28.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:28.574263+0000) 2022-09-10T11:27:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:28 smithi110 ceph-mon[97633]: pgmap v13383: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:29.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:28 smithi104 ceph-mon[97044]: pgmap v13383: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:29.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:29.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:29.098429+0000) 2022-09-10T11:27:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:29.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:29.574461+0000) 2022-09-10T11:27:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:30.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:30.098651+0000) 2022-09-10T11:27:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:30.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:30.574661+0000) 2022-09-10T11:27:31.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:30 smithi110 ceph-mon[97633]: pgmap v13384: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:31.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:30 smithi104 ceph-mon[97044]: pgmap v13384: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:31.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:31.098903+0000) 2022-09-10T11:27:32.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:31.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:31.574818+0000) 2022-09-10T11:27:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:32.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:32.099196+0000) 2022-09-10T11:27:33.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:32.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:32.575052+0000) 2022-09-10T11:27:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:32 smithi110 ceph-mon[97633]: pgmap v13385: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:33.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:32 smithi104 ceph-mon[97044]: pgmap v13385: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:33.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:33.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:33.099405+0000) 2022-09-10T11:27:34.077 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:33.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:33.575328+0000) 2022-09-10T11:27:34.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:34.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:34.099649+0000) 2022-09-10T11:27:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:34 smithi104 ceph-mon[97044]: pgmap v13386: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:34.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:34.575490+0000) 2022-09-10T11:27:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:34 smithi110 ceph-mon[97633]: pgmap v13386: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:35.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:35.099906+0000) 2022-09-10T11:27:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:35.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:35.575685+0000) 2022-09-10T11:27:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:36.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:36.100188+0000) 2022-09-10T11:27:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:36.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:36.575952+0000) 2022-09-10T11:27:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:36 smithi110 ceph-mon[97633]: pgmap v13387: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:36 smithi104 ceph-mon[97044]: pgmap v13387: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:37.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:37.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:37.100432+0000) 2022-09-10T11:27:38.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:37.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:37.576178+0000) 2022-09-10T11:27:38.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:38.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:38.100631+0000) 2022-09-10T11:27:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:38.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:38.576381+0000) 2022-09-10T11:27:39.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:38 smithi110 ceph-mon[97633]: pgmap v13388: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:39.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:38 smithi104 ceph-mon[97044]: pgmap v13388: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:39.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:39.100838+0000) 2022-09-10T11:27:40.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:39.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:39.576645+0000) 2022-09-10T11:27:40.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:40.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:40.101114+0000) 2022-09-10T11:27:41.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:40.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:40.576841+0000) 2022-09-10T11:27:41.168 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:40 smithi110 ceph-mon[97633]: pgmap v13389: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:41.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:40 smithi104 ceph-mon[97044]: pgmap v13389: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:41.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:41.101349+0000) 2022-09-10T11:27:42.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:41.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:41.577053+0000) 2022-09-10T11:27:42.321 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:42.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:42.101570+0000) 2022-09-10T11:27:42.323 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:27:42.047+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:27:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:42.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:42.577225+0000) 2022-09-10T11:27:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:42 smithi110 ceph-mon[97633]: pgmap v13390: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:42 smithi104 ceph-mon[97044]: pgmap v13390: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:43.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:43.101818+0000) 2022-09-10T11:27:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:43.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:43.577446+0000) 2022-09-10T11:27:44.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:44.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:44.102060+0000) 2022-09-10T11:27:45.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:44 smithi104 ceph-mon[97044]: pgmap v13391: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:45.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:44.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:44.577672+0000) 2022-09-10T11:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:44 smithi110 ceph-mon[97633]: pgmap v13391: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:45.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:45.102302+0000) 2022-09-10T11:27:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:27:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:27:46.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:27:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:27:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:45.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:45.577918+0000) 2022-09-10T11:27:46.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:27:46.168 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:27:46.169 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:27:46.173 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:27:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:46.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:46.102552+0000) 2022-09-10T11:27:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:46 smithi104 ceph-mon[97044]: pgmap v13392: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:46.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:46.578211+0000) 2022-09-10T11:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:46 smithi110 ceph-mon[97633]: pgmap v13392: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:47.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:47.102796+0000) 2022-09-10T11:27:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:47.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:47.578410+0000) 2022-09-10T11:27:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:48.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:48.102995+0000) 2022-09-10T11:27:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:48.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:48.578608+0000) 2022-09-10T11:27:49.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:48 smithi110 ceph-mon[97633]: pgmap v13393: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:49.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:48 smithi104 ceph-mon[97044]: pgmap v13393: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:49.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:49.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:49.103236+0000) 2022-09-10T11:27:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:49.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:49.578869+0000) 2022-09-10T11:27:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:50.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:50.103467+0000) 2022-09-10T11:27:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:50.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:50.579175+0000) 2022-09-10T11:27:51.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:50 smithi110 ceph-mon[97633]: pgmap v13394: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:50 smithi104 ceph-mon[97044]: pgmap v13394: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:51.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:51.103691+0000) 2022-09-10T11:27:52.069 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:51.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:51.579355+0000) 2022-09-10T11:27:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:52.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:52.103962+0000) 2022-09-10T11:27:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:52.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:52.579611+0000) 2022-09-10T11:27:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:52 smithi110 ceph-mon[97633]: pgmap v13395: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:52 smithi104 ceph-mon[97044]: pgmap v13395: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:53.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:53.104239+0000) 2022-09-10T11:27:54.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:53.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:53.579828+0000) 2022-09-10T11:27:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:54.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:54.104462+0000) 2022-09-10T11:27:55.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:54 smithi104 ceph-mon[97044]: pgmap v13396: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:55.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:54.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:54.580051+0000) 2022-09-10T11:27:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:54 smithi110 ceph-mon[97633]: pgmap v13396: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:27:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:55.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:55.104738+0000) 2022-09-10T11:27:56.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:55.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:55.580225+0000) 2022-09-10T11:27:56.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:56.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:56.105040+0000) 2022-09-10T11:27:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:56.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:56.580453+0000) 2022-09-10T11:27:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:56 smithi110 ceph-mon[97633]: pgmap v13397: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:56 smithi104 ceph-mon[97044]: pgmap v13397: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:27:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:57.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:57.105241+0000) 2022-09-10T11:27:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:27:57.063+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:27:58.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:57.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:57.580704+0000) 2022-09-10T11:27:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:58.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:58.105442+0000) 2022-09-10T11:27:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:58.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:58.580967+0000) 2022-09-10T11:27:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:27:58 smithi110 ceph-mon[97633]: pgmap v13398: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:27:59.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:27:58 smithi104 ceph-mon[97044]: pgmap v13398: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:27:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:27:59.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:59.105680+0000) 2022-09-10T11:28:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:27:59.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:27:59.581217+0000) 2022-09-10T11:28:00.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:00.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:00.105896+0000) 2022-09-10T11:28:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:00.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:00.581430+0000) 2022-09-10T11:28:01.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:00 smithi110 ceph-mon[97633]: pgmap v13399: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:01.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:00 smithi104 ceph-mon[97044]: pgmap v13399: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:01.564 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:01.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:01.106091+0000) 2022-09-10T11:28:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:01.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:01.581680+0000) 2022-09-10T11:28:02.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:02.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:02.106348+0000) 2022-09-10T11:28:03.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:02.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:02.581924+0000) 2022-09-10T11:28:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:02 smithi110 ceph-mon[97633]: pgmap v13400: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:28:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:02 smithi104 ceph-mon[97044]: pgmap v13400: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:28:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:03.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:03.106513+0000) 2022-09-10T11:28:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:03.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:03.582075+0000) 2022-09-10T11:28:04.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:04.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:04.106674+0000) 2022-09-10T11:28:05.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:04 smithi104 ceph-mon[97044]: pgmap v13401: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:28:05.078 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:04.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:04.582282+0000) 2022-09-10T11:28:05.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:04 smithi110 ceph-mon[97633]: pgmap v13401: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:28:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:05.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:05.106823+0000) 2022-09-10T11:28:06.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:05.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:05.582443+0000) 2022-09-10T11:28:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:06.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:06.106998+0000) 2022-09-10T11:28:07.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:06.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:06.582641+0000) 2022-09-10T11:28:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:06 smithi110 ceph-mon[97633]: pgmap v13402: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:28:07.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:06 smithi104 ceph-mon[97044]: pgmap v13402: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:28:07.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:07.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:07.107248+0000) 2022-09-10T11:28:08.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:07.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:07.582830+0000) 2022-09-10T11:28:08.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:08.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:08.107453+0000) 2022-09-10T11:28:09.067 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:08.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:08.583040+0000) 2022-09-10T11:28:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:08 smithi110 ceph-mon[97633]: pgmap v13403: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:28:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:08 smithi110 ceph-mon[97633]: osdmap e918: 12 total, 12 up, 12 in 2022-09-10T11:28:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:08 smithi104 ceph-mon[97044]: pgmap v13403: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:28:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:08 smithi104 ceph-mon[97044]: osdmap e918: 12 total, 12 up, 12 in 2022-09-10T11:28:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:09.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:09.107672+0000) 2022-09-10T11:28:10.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:09.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:09.583283+0000) 2022-09-10T11:28:10.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:10.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:10.107948+0000) 2022-09-10T11:28:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:10.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:10.583511+0000) 2022-09-10T11:28:11.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:10 smithi110 ceph-mon[97633]: pgmap v13405: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:28:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:10 smithi104 ceph-mon[97044]: pgmap v13405: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:28:11.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:11.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:11.108195+0000) 2022-09-10T11:28:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:11.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:11.583723+0000) 2022-09-10T11:28:12.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:12.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:12.108424+0000) 2022-09-10T11:28:12.559 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:28:12.078+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:28:13.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:12.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:12.583966+0000) 2022-09-10T11:28:13.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:12 smithi110 ceph-mon[97633]: pgmap v13406: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:28:13.305 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:12 smithi104 ceph-mon[97044]: pgmap v13406: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:28:13.564 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:13.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:13.108621+0000) 2022-09-10T11:28:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:13.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:13.584161+0000) 2022-09-10T11:28:14.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:14.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:14.108871+0000) 2022-09-10T11:28:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:14 smithi104 ceph-mon[97044]: osdmap e919: 12 total, 12 up, 12 in 2022-09-10T11:28:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:14 smithi104 ceph-mon[97044]: pgmap v13408: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:28:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:14.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:14.584388+0000) 2022-09-10T11:28:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:14 smithi110 ceph-mon[97633]: osdmap e919: 12 total, 12 up, 12 in 2022-09-10T11:28:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:14 smithi110 ceph-mon[97633]: pgmap v13408: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:28:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:15.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:15.109097+0000) 2022-09-10T11:28:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:15.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:15.584592+0000) 2022-09-10T11:28:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:16.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:16.109413+0000) 2022-09-10T11:28:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:16.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:16.584859+0000) 2022-09-10T11:28:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:16 smithi110 ceph-mon[97633]: pgmap v13409: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:28:17.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:16 smithi104 ceph-mon[97044]: pgmap v13409: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:28:17.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:17.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:17.109552+0000) 2022-09-10T11:28:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:17.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:17.585137+0000) 2022-09-10T11:28:18.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:18.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:18.109751+0000) 2022-09-10T11:28:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:18.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:18.585369+0000) 2022-09-10T11:28:19.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:18 smithi110 ceph-mon[97633]: pgmap v13410: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:28:19.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:18 smithi104 ceph-mon[97044]: pgmap v13410: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:28:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:19.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:19.109909+0000) 2022-09-10T11:28:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:19.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:19.585573+0000) 2022-09-10T11:28:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:20.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:20.110172+0000) 2022-09-10T11:28:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:20.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:20.585845+0000) 2022-09-10T11:28:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:20 smithi110 ceph-mon[97633]: pgmap v13411: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:28:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:20 smithi104 ceph-mon[97044]: pgmap v13411: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:28:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:21.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:21.110323+0000) 2022-09-10T11:28:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:21.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:21.586092+0000) 2022-09-10T11:28:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:22.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:22.110588+0000) 2022-09-10T11:28:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:22.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:22.586239+0000) 2022-09-10T11:28:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:22 smithi110 ceph-mon[97633]: pgmap v13412: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:28:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:22 smithi104 ceph-mon[97044]: pgmap v13412: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:28:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:23.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:23.110813+0000) 2022-09-10T11:28:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:23.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:23.586492+0000) 2022-09-10T11:28:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:24.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:24.111067+0000) 2022-09-10T11:28:25.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:24 smithi104 ceph-mon[97044]: pgmap v13413: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:28:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:24.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:24.586748+0000) 2022-09-10T11:28:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:24 smithi110 ceph-mon[97633]: pgmap v13413: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:28:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:25.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:25.111248+0000) 2022-09-10T11:28:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:25.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:25.586946+0000) 2022-09-10T11:28:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:26.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:26.111477+0000) 2022-09-10T11:28:27.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:26.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:26.587230+0000) 2022-09-10T11:28:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:26 smithi110 ceph-mon[97633]: pgmap v13414: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:28:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:26 smithi104 ceph-mon[97044]: pgmap v13414: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:28:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:27.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:27.111627+0000) 2022-09-10T11:28:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:28:27.088+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:28:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:27.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:27.587482+0000) 2022-09-10T11:28:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:28.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:28.111844+0000) 2022-09-10T11:28:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:28.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:28.587705+0000) 2022-09-10T11:28:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:28 smithi110 ceph-mon[97633]: pgmap v13415: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:28 smithi104 ceph-mon[97044]: pgmap v13415: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:29.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:29.112079+0000) 2022-09-10T11:28:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:29.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:29.587976+0000) 2022-09-10T11:28:30.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:30.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:30.112387+0000) 2022-09-10T11:28:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:30.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:30.588230+0000) 2022-09-10T11:28:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:30 smithi110 ceph-mon[97633]: pgmap v13416: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:30 smithi104 ceph-mon[97044]: pgmap v13416: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:31.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:31.112650+0000) 2022-09-10T11:28:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:31.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:31.588474+0000) 2022-09-10T11:28:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:32.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:32.112867+0000) 2022-09-10T11:28:33.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:32.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:32.588724+0000) 2022-09-10T11:28:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:32 smithi110 ceph-mon[97633]: pgmap v13417: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:32 smithi104 ceph-mon[97044]: pgmap v13417: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:33.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:33.113073+0000) 2022-09-10T11:28:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:33.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:33.588980+0000) 2022-09-10T11:28:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:34.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:34.113303+0000) 2022-09-10T11:28:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:34 smithi104 ceph-mon[97044]: pgmap v13418: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:34.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:34.589232+0000) 2022-09-10T11:28:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:34 smithi110 ceph-mon[97633]: pgmap v13418: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:35.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:35.113536+0000) 2022-09-10T11:28:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:35.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:35.589413+0000) 2022-09-10T11:28:36.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:36.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:36.113731+0000) 2022-09-10T11:28:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:36.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:36.589551+0000) 2022-09-10T11:28:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:36 smithi110 ceph-mon[97633]: pgmap v13419: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:36 smithi104 ceph-mon[97044]: pgmap v13419: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:37.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:37.113944+0000) 2022-09-10T11:28:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:37.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:37.589744+0000) 2022-09-10T11:28:38.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:38.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:38.114183+0000) 2022-09-10T11:28:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:38.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:38.589953+0000) 2022-09-10T11:28:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:38 smithi110 ceph-mon[97633]: pgmap v13420: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:38 smithi104 ceph-mon[97044]: pgmap v13420: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:39.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:39.114472+0000) 2022-09-10T11:28:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:39.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:39.590177+0000) 2022-09-10T11:28:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:40.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:40.114721+0000) 2022-09-10T11:28:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:40.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:40.590377+0000) 2022-09-10T11:28:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:40 smithi110 ceph-mon[97633]: pgmap v13421: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:40 smithi104 ceph-mon[97044]: pgmap v13421: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:41.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:41.114972+0000) 2022-09-10T11:28:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:41.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:41.590640+0000) 2022-09-10T11:28:42.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:28:42.104+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:28:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:42.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:42.115208+0000) 2022-09-10T11:28:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:42.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:42.590926+0000) 2022-09-10T11:28:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:42 smithi110 ceph-mon[97633]: pgmap v13422: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:42 smithi104 ceph-mon[97044]: pgmap v13422: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:43.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:43.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:43.115385+0000) 2022-09-10T11:28:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:43.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:43.591197+0000) 2022-09-10T11:28:44.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:44.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:44.115604+0000) 2022-09-10T11:28:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:44 smithi104 ceph-mon[97044]: pgmap v13423: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:44.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:44.591406+0000) 2022-09-10T11:28:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:44 smithi110 ceph-mon[97633]: pgmap v13423: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:45.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:45.115850+0000) 2022-09-10T11:28:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:28:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:28:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:45.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:45.591593+0000) 2022-09-10T11:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:28:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:28:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:28:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:46.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:46.116161+0000) 2022-09-10T11:28:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:46 smithi104 ceph-mon[97044]: pgmap v13424: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:46.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:46.591869+0000) 2022-09-10T11:28:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:46 smithi110 ceph-mon[97633]: pgmap v13424: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:47.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:47.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:47.116349+0000) 2022-09-10T11:28:48.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:47.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:47.592045+0000) 2022-09-10T11:28:48.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:48.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:48.116526+0000) 2022-09-10T11:28:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:48.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:48.592271+0000) 2022-09-10T11:28:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:48 smithi110 ceph-mon[97633]: pgmap v13425: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:49.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:48 smithi104 ceph-mon[97044]: pgmap v13425: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:49.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:49.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:49.116726+0000) 2022-09-10T11:28:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:49.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:49.592570+0000) 2022-09-10T11:28:50.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:50.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:50.116913+0000) 2022-09-10T11:28:51.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:50.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:50.592795+0000) 2022-09-10T11:28:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:50 smithi110 ceph-mon[97633]: pgmap v13426: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:50 smithi104 ceph-mon[97044]: pgmap v13426: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:51.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:51.117201+0000) 2022-09-10T11:28:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:51.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:51.593053+0000) 2022-09-10T11:28:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:52.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:52.117394+0000) 2022-09-10T11:28:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:52.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:52.593277+0000) 2022-09-10T11:28:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:52 smithi110 ceph-mon[97633]: pgmap v13427: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:52 smithi104 ceph-mon[97044]: pgmap v13427: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:53.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:53.117522+0000) 2022-09-10T11:28:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:53.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:53.593497+0000) 2022-09-10T11:28:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:54.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:54.117674+0000) 2022-09-10T11:28:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:54 smithi104 ceph-mon[97044]: pgmap v13428: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:55.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:54.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:54.593681+0000) 2022-09-10T11:28:55.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:54 smithi110 ceph-mon[97633]: pgmap v13428: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:55.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:55.117890+0000) 2022-09-10T11:28:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:55.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:55.593921+0000) 2022-09-10T11:28:56.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:56.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:56.118086+0000) 2022-09-10T11:28:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:56.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:56.594205+0000) 2022-09-10T11:28:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:56 smithi110 ceph-mon[97633]: pgmap v13429: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:56 smithi104 ceph-mon[97044]: pgmap v13429: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:28:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:57.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:57.118246+0000) 2022-09-10T11:28:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:28:57.108+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:28:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:57.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:57.594428+0000) 2022-09-10T11:28:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:58.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:58.118440+0000) 2022-09-10T11:28:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:58.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:58.594605+0000) 2022-09-10T11:28:59.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:28:58 smithi110 ceph-mon[97633]: pgmap v13430: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:28:58 smithi104 ceph-mon[97044]: pgmap v13430: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:28:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:28:59.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:59.118683+0000) 2022-09-10T11:29:00.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:28:59.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:28:59.594789+0000) 2022-09-10T11:29:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:00.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:00.118914+0000) 2022-09-10T11:29:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:00.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:00.594908+0000) 2022-09-10T11:29:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:00 smithi110 ceph-mon[97633]: pgmap v13431: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:00 smithi104 ceph-mon[97044]: pgmap v13431: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:01.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:01.119031+0000) 2022-09-10T11:29:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:01.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:01.595144+0000) 2022-09-10T11:29:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:02.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:02.119207+0000) 2022-09-10T11:29:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:02.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:02.595298+0000) 2022-09-10T11:29:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:02 smithi104 ceph-mon[97044]: pgmap v13432: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:02 smithi110 ceph-mon[97633]: pgmap v13432: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:03.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:03.119434+0000) 2022-09-10T11:29:04.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:03.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:03.595511+0000) 2022-09-10T11:29:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:04.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:04.119610+0000) 2022-09-10T11:29:05.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:04 smithi104 ceph-mon[97044]: pgmap v13433: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:04.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:04.595657+0000) 2022-09-10T11:29:05.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:04 smithi110 ceph-mon[97633]: pgmap v13433: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:05.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:05.119756+0000) 2022-09-10T11:29:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:05.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:05.595817+0000) 2022-09-10T11:29:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:06.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:06.119954+0000) 2022-09-10T11:29:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:06.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:06.596043+0000) 2022-09-10T11:29:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:06 smithi110 ceph-mon[97633]: pgmap v13434: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:29:07.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:06 smithi104 ceph-mon[97044]: pgmap v13434: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:29:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:07.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:07.120175+0000) 2022-09-10T11:29:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:07.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:07.596170+0000) 2022-09-10T11:29:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:08.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:08.120316+0000) 2022-09-10T11:29:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:08.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:08.596427+0000) 2022-09-10T11:29:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:08 smithi104 ceph-mon[97044]: pgmap v13435: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:29:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:08 smithi104 ceph-mon[97044]: osdmap e920: 12 total, 12 up, 12 in 2022-09-10T11:29:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:08 smithi110 ceph-mon[97633]: pgmap v13435: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:29:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:08 smithi110 ceph-mon[97633]: osdmap e920: 12 total, 12 up, 12 in 2022-09-10T11:29:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:09.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:09.120511+0000) 2022-09-10T11:29:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:09.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:09.596679+0000) 2022-09-10T11:29:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:10.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:10.120652+0000) 2022-09-10T11:29:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:10.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:10.596895+0000) 2022-09-10T11:29:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:10 smithi104 ceph-mon[97044]: pgmap v13437: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:29:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:10 smithi110 ceph-mon[97633]: pgmap v13437: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:29:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:11.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:11.120826+0000) 2022-09-10T11:29:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:11.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:11.597052+0000) 2022-09-10T11:29:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:29:12.111+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:29:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:12.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:12.121014+0000) 2022-09-10T11:29:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:12.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:12.597169+0000) 2022-09-10T11:29:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:12 smithi104 ceph-mon[97044]: pgmap v13438: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:29:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:12 smithi110 ceph-mon[97633]: pgmap v13438: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:29:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:13.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:13.121248+0000) 2022-09-10T11:29:14.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:13.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:13.597364+0000) 2022-09-10T11:29:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:14.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:14.121570+0000) 2022-09-10T11:29:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:14 smithi104 ceph-mon[97044]: osdmap e921: 12 total, 12 up, 12 in 2022-09-10T11:29:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:14 smithi104 ceph-mon[97044]: pgmap v13440: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:29:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:14.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:14.597542+0000) 2022-09-10T11:29:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:14 smithi110 ceph-mon[97633]: osdmap e921: 12 total, 12 up, 12 in 2022-09-10T11:29:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:14 smithi110 ceph-mon[97633]: pgmap v13440: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:29:15.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:15.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:15.121751+0000) 2022-09-10T11:29:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:15.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:15.597665+0000) 2022-09-10T11:29:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:16.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:16.121935+0000) 2022-09-10T11:29:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:16.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:16.597851+0000) 2022-09-10T11:29:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:16 smithi104 ceph-mon[97044]: pgmap v13441: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:29:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:16 smithi110 ceph-mon[97633]: pgmap v13441: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:29:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:17.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:17.122172+0000) 2022-09-10T11:29:18.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:17.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:17.597984+0000) 2022-09-10T11:29:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:18.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:18.122382+0000) 2022-09-10T11:29:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:18.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:18.598139+0000) 2022-09-10T11:29:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:18 smithi104 ceph-mon[97044]: pgmap v13442: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:29:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:18 smithi110 ceph-mon[97633]: pgmap v13442: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:29:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:19.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:19.122576+0000) 2022-09-10T11:29:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:19.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:19.598322+0000) 2022-09-10T11:29:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:20.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:20.122735+0000) 2022-09-10T11:29:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:20.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:20.598517+0000) 2022-09-10T11:29:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:20 smithi104 ceph-mon[97044]: pgmap v13443: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:29:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:20 smithi110 ceph-mon[97633]: pgmap v13443: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:29:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:21.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:21.122964+0000) 2022-09-10T11:29:22.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:21.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:21.598727+0000) 2022-09-10T11:29:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:22.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:22.123184+0000) 2022-09-10T11:29:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:22.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:22.598904+0000) 2022-09-10T11:29:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:22 smithi104 ceph-mon[97044]: pgmap v13444: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:29:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:22 smithi110 ceph-mon[97633]: pgmap v13444: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:29:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:23.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:23.123421+0000) 2022-09-10T11:29:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:23.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:23.599144+0000) 2022-09-10T11:29:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:24.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:24.123611+0000) 2022-09-10T11:29:25.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:24 smithi104 ceph-mon[97044]: pgmap v13445: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:29:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:24.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:24.599414+0000) 2022-09-10T11:29:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:24 smithi110 ceph-mon[97633]: pgmap v13445: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 101 B/s wr, 10 op/s 2022-09-10T11:29:25.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:25.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:25.123861+0000) 2022-09-10T11:29:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:25.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:25.599605+0000) 2022-09-10T11:29:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:26.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:26.124171+0000) 2022-09-10T11:29:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:26.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:26.599797+0000) 2022-09-10T11:29:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:26 smithi104 ceph-mon[97044]: pgmap v13446: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:29:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:26 smithi110 ceph-mon[97633]: pgmap v13446: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:29:27.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:29:27.112+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:29:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:27.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:27.124332+0000) 2022-09-10T11:29:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:27.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:27.600058+0000) 2022-09-10T11:29:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:28.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:28.124559+0000) 2022-09-10T11:29:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:28.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:28.600381+0000) 2022-09-10T11:29:29.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:28 smithi104 ceph-mon[97044]: pgmap v13447: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:28 smithi110 ceph-mon[97633]: pgmap v13447: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:29.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:29.124710+0000) 2022-09-10T11:29:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:29.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:29.600550+0000) 2022-09-10T11:29:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:30.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:30.124929+0000) 2022-09-10T11:29:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:30.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:30.600746+0000) 2022-09-10T11:29:31.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:30 smithi104 ceph-mon[97044]: pgmap v13448: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:31.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:30 smithi110 ceph-mon[97633]: pgmap v13448: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:31.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:31.125200+0000) 2022-09-10T11:29:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:31.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:31.600986+0000) 2022-09-10T11:29:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:32.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:32.125447+0000) 2022-09-10T11:29:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:32.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:32.601178+0000) 2022-09-10T11:29:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:32 smithi104 ceph-mon[97044]: pgmap v13449: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:32 smithi110 ceph-mon[97633]: pgmap v13449: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:33.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:33.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:33.125678+0000) 2022-09-10T11:29:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:33.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:33.601438+0000) 2022-09-10T11:29:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:34.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:34.125829+0000) 2022-09-10T11:29:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:34 smithi104 ceph-mon[97044]: pgmap v13450: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:34.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:34.601683+0000) 2022-09-10T11:29:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:34 smithi110 ceph-mon[97633]: pgmap v13450: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:35.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:35.126055+0000) 2022-09-10T11:29:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:35.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:35.601938+0000) 2022-09-10T11:29:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:36.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:36.126316+0000) 2022-09-10T11:29:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:36.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:36.602235+0000) 2022-09-10T11:29:37.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:36 smithi104 ceph-mon[97044]: pgmap v13451: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:36 smithi110 ceph-mon[97633]: pgmap v13451: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:37.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:37.126463+0000) 2022-09-10T11:29:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:37.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:37.602457+0000) 2022-09-10T11:29:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:38.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:38.126639+0000) 2022-09-10T11:29:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:38.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:38.602644+0000) 2022-09-10T11:29:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:38 smithi104 ceph-mon[97044]: pgmap v13452: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:39.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:38 smithi110 ceph-mon[97633]: pgmap v13452: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:39.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:39.126840+0000) 2022-09-10T11:29:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:39.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:39.602861+0000) 2022-09-10T11:29:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:40.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:40.127156+0000) 2022-09-10T11:29:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:40.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:40.603134+0000) 2022-09-10T11:29:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:40 smithi104 ceph-mon[97044]: pgmap v13453: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:40 smithi110 ceph-mon[97633]: pgmap v13453: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:41.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:41.127355+0000) 2022-09-10T11:29:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:41.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:41.603412+0000) 2022-09-10T11:29:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:42.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:42.127579+0000) 2022-09-10T11:29:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:29:42.127+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:29:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:42.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:42.603632+0000) 2022-09-10T11:29:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:42 smithi104 ceph-mon[97044]: pgmap v13454: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:42 smithi110 ceph-mon[97633]: pgmap v13454: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:43.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:43.127830+0000) 2022-09-10T11:29:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:43.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:43.603787+0000) 2022-09-10T11:29:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:44.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:44.128026+0000) 2022-09-10T11:29:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:44 smithi104 ceph-mon[97044]: pgmap v13455: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:44.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:44.603974+0000) 2022-09-10T11:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:44 smithi110 ceph-mon[97633]: pgmap v13455: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:45.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:45.128230+0000) 2022-09-10T11:29:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:29:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:45.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:45.604136+0000) 2022-09-10T11:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:29:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:29:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:46.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:46.128483+0000) 2022-09-10T11:29:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:46 smithi104 ceph-mon[97044]: pgmap v13456: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:46.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:46.604345+0000) 2022-09-10T11:29:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:46 smithi110 ceph-mon[97633]: pgmap v13456: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:47.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:47.128696+0000) 2022-09-10T11:29:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:47.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:47.604612+0000) 2022-09-10T11:29:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:48.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:48.128918+0000) 2022-09-10T11:29:49.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:48.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:48.604890+0000) 2022-09-10T11:29:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:48 smithi104 ceph-mon[97044]: pgmap v13457: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:48 smithi110 ceph-mon[97633]: pgmap v13457: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:49.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:49.129129+0000) 2022-09-10T11:29:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:49.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:49.605188+0000) 2022-09-10T11:29:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:50.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:50.129331+0000) 2022-09-10T11:29:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:50.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:50.605438+0000) 2022-09-10T11:29:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:50 smithi104 ceph-mon[97044]: pgmap v13458: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:50 smithi110 ceph-mon[97633]: pgmap v13458: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:51.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:51.129555+0000) 2022-09-10T11:29:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:51.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:51.605688+0000) 2022-09-10T11:29:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:52.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:52.129725+0000) 2022-09-10T11:29:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:52.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:52.605969+0000) 2022-09-10T11:29:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:52 smithi104 ceph-mon[97044]: pgmap v13459: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:52 smithi110 ceph-mon[97633]: pgmap v13459: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:53.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:53.129865+0000) 2022-09-10T11:29:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:53.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:53.606141+0000) 2022-09-10T11:29:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:54.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:54.130043+0000) 2022-09-10T11:29:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:54 smithi104 ceph-mon[97044]: pgmap v13460: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:54.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:54.606344+0000) 2022-09-10T11:29:55.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:54 smithi110 ceph-mon[97633]: pgmap v13460: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:55.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:55.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:55.130273+0000) 2022-09-10T11:29:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:55.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:55.606553+0000) 2022-09-10T11:29:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:56.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:56.130511+0000) 2022-09-10T11:29:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:56.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:56.606771+0000) 2022-09-10T11:29:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:56 smithi104 ceph-mon[97044]: pgmap v13461: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:56 smithi110 ceph-mon[97633]: pgmap v13461: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:29:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:29:57.129+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:29:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:57.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:57.130683+0000) 2022-09-10T11:29:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:57.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:57.606935+0000) 2022-09-10T11:29:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:58.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:58.130901+0000) 2022-09-10T11:29:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:58.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:58.607214+0000) 2022-09-10T11:29:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:29:58 smithi104 ceph-mon[97044]: pgmap v13462: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:29:58 smithi110 ceph-mon[97633]: pgmap v13462: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:29:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:29:59.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:59.131171+0000) 2022-09-10T11:30:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:29:59.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:29:59.607397+0000) 2022-09-10T11:30:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:00.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:00.131383+0000) 2022-09-10T11:30:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:00.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:00.607569+0000) 2022-09-10T11:30:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:00 smithi104 ceph-mon[97044]: pgmap v13463: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:30:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:00 smithi110 ceph-mon[97633]: pgmap v13463: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:30:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:01.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:01.131533+0000) 2022-09-10T11:30:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:01.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:01.607802+0000) 2022-09-10T11:30:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:02.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:02.131758+0000) 2022-09-10T11:30:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:02.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:02.608080+0000) 2022-09-10T11:30:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:02 smithi104 ceph-mon[97044]: pgmap v13464: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:02 smithi110 ceph-mon[97633]: pgmap v13464: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:03.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:03.132021+0000) 2022-09-10T11:30:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:03.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:03.608260+0000) 2022-09-10T11:30:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:04.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:04.132283+0000) 2022-09-10T11:30:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:04 smithi104 ceph-mon[97044]: pgmap v13465: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:04.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:04.608463+0000) 2022-09-10T11:30:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:04 smithi110 ceph-mon[97633]: pgmap v13465: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:05.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:05.132522+0000) 2022-09-10T11:30:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:05.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:05.608762+0000) 2022-09-10T11:30:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:06.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:06.132711+0000) 2022-09-10T11:30:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:06.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:06.608988+0000) 2022-09-10T11:30:07.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:06 smithi104 ceph-mon[97044]: pgmap v13466: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:30:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:06 smithi110 ceph-mon[97633]: pgmap v13466: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:30:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:07.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:07.132845+0000) 2022-09-10T11:30:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:07.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:07.609220+0000) 2022-09-10T11:30:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:08.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:08.133048+0000) 2022-09-10T11:30:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:08.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:08.609476+0000) 2022-09-10T11:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:08 smithi104 ceph-mon[97044]: pgmap v13467: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:08 smithi104 ceph-mon[97044]: osdmap e922: 12 total, 12 up, 12 in 2022-09-10T11:30:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:08 smithi110 ceph-mon[97633]: pgmap v13467: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:30:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:08 smithi110 ceph-mon[97633]: osdmap e922: 12 total, 12 up, 12 in 2022-09-10T11:30:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:09.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:09.133165+0000) 2022-09-10T11:30:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:09.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:09.609685+0000) 2022-09-10T11:30:10.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:10.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:10.133331+0000) 2022-09-10T11:30:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:10.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:10.609844+0000) 2022-09-10T11:30:11.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:10 smithi104 ceph-mon[97044]: pgmap v13469: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:30:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:10 smithi110 ceph-mon[97633]: pgmap v13469: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:30:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:11.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:11.133564+0000) 2022-09-10T11:30:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:11.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:11.610067+0000) 2022-09-10T11:30:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:30:12.144+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:30:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:12.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:12.133777+0000) 2022-09-10T11:30:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:12.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:12.610306+0000) 2022-09-10T11:30:13.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:12 smithi104 ceph-mon[97044]: pgmap v13470: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:30:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:12 smithi110 ceph-mon[97633]: pgmap v13470: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:30:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:13.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:13.134028+0000) 2022-09-10T11:30:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:13.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:13.610447+0000) 2022-09-10T11:30:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:14.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:14.134333+0000) 2022-09-10T11:30:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:14 smithi104 ceph-mon[97044]: osdmap e923: 12 total, 12 up, 12 in 2022-09-10T11:30:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:14 smithi104 ceph-mon[97044]: pgmap v13472: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:30:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:14.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:14.610610+0000) 2022-09-10T11:30:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:14 smithi110 ceph-mon[97633]: osdmap e923: 12 total, 12 up, 12 in 2022-09-10T11:30:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:14 smithi110 ceph-mon[97633]: pgmap v13472: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:30:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:15.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:15.134528+0000) 2022-09-10T11:30:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:15.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:15.610782+0000) 2022-09-10T11:30:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:16.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:16.134783+0000) 2022-09-10T11:30:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:16.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:16.611018+0000) 2022-09-10T11:30:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:16 smithi104 ceph-mon[97044]: pgmap v13473: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:30:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:16 smithi110 ceph-mon[97633]: pgmap v13473: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:30:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:17.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:17.135003+0000) 2022-09-10T11:30:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:17.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:17.611282+0000) 2022-09-10T11:30:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:18.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:18.135287+0000) 2022-09-10T11:30:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:18.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:18.611478+0000) 2022-09-10T11:30:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:18 smithi104 ceph-mon[97044]: pgmap v13474: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:30:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:18 smithi110 ceph-mon[97633]: pgmap v13474: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:30:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:19.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:19.135539+0000) 2022-09-10T11:30:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:19.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:19.611724+0000) 2022-09-10T11:30:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:20.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:20.135732+0000) 2022-09-10T11:30:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:20.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:20.611972+0000) 2022-09-10T11:30:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:20 smithi104 ceph-mon[97044]: pgmap v13475: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:30:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:20 smithi110 ceph-mon[97633]: pgmap v13475: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:30:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:21.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:21.135938+0000) 2022-09-10T11:30:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:21.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:21.612168+0000) 2022-09-10T11:30:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:22.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:22.136218+0000) 2022-09-10T11:30:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:22.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:22.612392+0000) 2022-09-10T11:30:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:22 smithi104 ceph-mon[97044]: pgmap v13476: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:30:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:22 smithi110 ceph-mon[97633]: pgmap v13476: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:30:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:23.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:23.136485+0000) 2022-09-10T11:30:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:23.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:23.612599+0000) 2022-09-10T11:30:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:24.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:24.136666+0000) 2022-09-10T11:30:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:24 smithi104 ceph-mon[97044]: pgmap v13477: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:30:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:24.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:24.612830+0000) 2022-09-10T11:30:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:24 smithi110 ceph-mon[97633]: pgmap v13477: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:30:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:25.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:25.136781+0000) 2022-09-10T11:30:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:25.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:25.613057+0000) 2022-09-10T11:30:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:26.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:26.137007+0000) 2022-09-10T11:30:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:26.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:26.613338+0000) 2022-09-10T11:30:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:26 smithi104 ceph-mon[97044]: pgmap v13478: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:30:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:26 smithi110 ceph-mon[97633]: pgmap v13478: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:30:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:27.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:27.137186+0000) 2022-09-10T11:30:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:30:27.154+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:30:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:27.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:27.613520+0000) 2022-09-10T11:30:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:28.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:28.137408+0000) 2022-09-10T11:30:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:28.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:28.613696+0000) 2022-09-10T11:30:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:28 smithi104 ceph-mon[97044]: pgmap v13479: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:28 smithi110 ceph-mon[97633]: pgmap v13479: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:29.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:29.137587+0000) 2022-09-10T11:30:30.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:29.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:29.613948+0000) 2022-09-10T11:30:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:30.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:30.137758+0000) 2022-09-10T11:30:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:30.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:30.614205+0000) 2022-09-10T11:30:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:30 smithi104 ceph-mon[97044]: pgmap v13480: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:30 smithi110 ceph-mon[97633]: pgmap v13480: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:31.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:31.137980+0000) 2022-09-10T11:30:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:31.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:31.614419+0000) 2022-09-10T11:30:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:32.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:32.138194+0000) 2022-09-10T11:30:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:32.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:32.614646+0000) 2022-09-10T11:30:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:32 smithi104 ceph-mon[97044]: pgmap v13481: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:32 smithi110 ceph-mon[97633]: pgmap v13481: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:33.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:33.138377+0000) 2022-09-10T11:30:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:33.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:33.614876+0000) 2022-09-10T11:30:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:34.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:34.138639+0000) 2022-09-10T11:30:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:34 smithi104 ceph-mon[97044]: pgmap v13482: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:34.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:34.615161+0000) 2022-09-10T11:30:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:34 smithi110 ceph-mon[97633]: pgmap v13482: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:35.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:35.138834+0000) 2022-09-10T11:30:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:35.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:35.615291+0000) 2022-09-10T11:30:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:36.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:36.139075+0000) 2022-09-10T11:30:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:36.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:36.615521+0000) 2022-09-10T11:30:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:36 smithi104 ceph-mon[97044]: pgmap v13483: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:36 smithi110 ceph-mon[97633]: pgmap v13483: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:37.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:37.139274+0000) 2022-09-10T11:30:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:37.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:37.615771+0000) 2022-09-10T11:30:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:38.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:38.139465+0000) 2022-09-10T11:30:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:38.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:38.616029+0000) 2022-09-10T11:30:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:38 smithi104 ceph-mon[97044]: pgmap v13484: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:38 smithi110 ceph-mon[97633]: pgmap v13484: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:39.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:39.139729+0000) 2022-09-10T11:30:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:39.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:39.616256+0000) 2022-09-10T11:30:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:40.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:40.139954+0000) 2022-09-10T11:30:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:40.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:40.616463+0000) 2022-09-10T11:30:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:40 smithi104 ceph-mon[97044]: pgmap v13485: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:40 smithi110 ceph-mon[97633]: pgmap v13485: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:41.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:41.140194+0000) 2022-09-10T11:30:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:41.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:41.616606+0000) 2022-09-10T11:30:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:42.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:42.140439+0000) 2022-09-10T11:30:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:30:42.160+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:30:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:42.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:42.616789+0000) 2022-09-10T11:30:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:42 smithi104 ceph-mon[97044]: pgmap v13486: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:42 smithi110 ceph-mon[97633]: pgmap v13486: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:43.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:43.140689+0000) 2022-09-10T11:30:44.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:43.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:43.616982+0000) 2022-09-10T11:30:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:44.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:44.140915+0000) 2022-09-10T11:30:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:44 smithi104 ceph-mon[97044]: pgmap v13487: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:45.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:44.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:44.617151+0000) 2022-09-10T11:30:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:44 smithi110 ceph-mon[97633]: pgmap v13487: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:45.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:45.141160+0000) 2022-09-10T11:30:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:30:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:45.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:45.617358+0000) 2022-09-10T11:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:30:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:30:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:46.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:46.141293+0000) 2022-09-10T11:30:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:46 smithi104 ceph-mon[97044]: pgmap v13488: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:46.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:46.617493+0000) 2022-09-10T11:30:47.171 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:46 smithi110 ceph-mon[97633]: pgmap v13488: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:47.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:47.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:47.141461+0000) 2022-09-10T11:30:48.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:47.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:47.617687+0000) 2022-09-10T11:30:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:48.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:48.141707+0000) 2022-09-10T11:30:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:48.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:48.617941+0000) 2022-09-10T11:30:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:48 smithi104 ceph-mon[97044]: pgmap v13489: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:48 smithi110 ceph-mon[97633]: pgmap v13489: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:49.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:49.141884+0000) 2022-09-10T11:30:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:49.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:49.618211+0000) 2022-09-10T11:30:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:50.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:50.142186+0000) 2022-09-10T11:30:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:50.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:50.618377+0000) 2022-09-10T11:30:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:50 smithi104 ceph-mon[97044]: pgmap v13490: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:50 smithi110 ceph-mon[97633]: pgmap v13490: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:51.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:51.142429+0000) 2022-09-10T11:30:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:51.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:51.618562+0000) 2022-09-10T11:30:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:52.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:52.142633+0000) 2022-09-10T11:30:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:52.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:52.618703+0000) 2022-09-10T11:30:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:52 smithi104 ceph-mon[97044]: pgmap v13491: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:52 smithi110 ceph-mon[97633]: pgmap v13491: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:53.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:53.142819+0000) 2022-09-10T11:30:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:53.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:53.618850+0000) 2022-09-10T11:30:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:54.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:54.143065+0000) 2022-09-10T11:30:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:54 smithi104 ceph-mon[97044]: pgmap v13492: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:54.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:54.619077+0000) 2022-09-10T11:30:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:54 smithi110 ceph-mon[97633]: pgmap v13492: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:55.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:55.143326+0000) 2022-09-10T11:30:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:55.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:55.619271+0000) 2022-09-10T11:30:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:56.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:56.143568+0000) 2022-09-10T11:30:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:56.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:56.619447+0000) 2022-09-10T11:30:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:56 smithi104 ceph-mon[97044]: pgmap v13493: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:56 smithi110 ceph-mon[97633]: pgmap v13493: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:30:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:57.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:57.143818+0000) 2022-09-10T11:30:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:30:57.176+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:30:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:57.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:57.619690+0000) 2022-09-10T11:30:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:58.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:58.144072+0000) 2022-09-10T11:30:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:58.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:58.619966+0000) 2022-09-10T11:30:59.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:30:58 smithi104 ceph-mon[97044]: pgmap v13494: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:59.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:30:58 smithi110 ceph-mon[97633]: pgmap v13494: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:30:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:30:59.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:59.144218+0000) 2022-09-10T11:31:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:30:59.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:30:59.620216+0000) 2022-09-10T11:31:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:00.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:00.144367+0000) 2022-09-10T11:31:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:00.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:00.620404+0000) 2022-09-10T11:31:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:00 smithi104 ceph-mon[97044]: pgmap v13495: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:00 smithi110 ceph-mon[97633]: pgmap v13495: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:01.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:01.144515+0000) 2022-09-10T11:31:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:01.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:01.620658+0000) 2022-09-10T11:31:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:02.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:02.144730+0000) 2022-09-10T11:31:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:02.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:02.620933+0000) 2022-09-10T11:31:03.305 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:02 smithi104 ceph-mon[97044]: pgmap v13496: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:02 smithi110 ceph-mon[97633]: pgmap v13496: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:03.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:03.145013+0000) 2022-09-10T11:31:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:03.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:03.621165+0000) 2022-09-10T11:31:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:04.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:04.145211+0000) 2022-09-10T11:31:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:04 smithi104 ceph-mon[97044]: pgmap v13497: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:04.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:04.621346+0000) 2022-09-10T11:31:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:04 smithi110 ceph-mon[97633]: pgmap v13497: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:05.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:05.145454+0000) 2022-09-10T11:31:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:05.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:05.621595+0000) 2022-09-10T11:31:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:06.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:06.145756+0000) 2022-09-10T11:31:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:06.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:06.621839+0000) 2022-09-10T11:31:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:06 smithi104 ceph-mon[97044]: pgmap v13498: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:31:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:06 smithi110 ceph-mon[97633]: pgmap v13498: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:31:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:07.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:07.145994+0000) 2022-09-10T11:31:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:07.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:07.622072+0000) 2022-09-10T11:31:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:08.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:08.146274+0000) 2022-09-10T11:31:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:08.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:08.622299+0000) 2022-09-10T11:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:08 smithi104 ceph-mon[97044]: pgmap v13499: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:31:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:08 smithi104 ceph-mon[97044]: osdmap e924: 12 total, 12 up, 12 in 2022-09-10T11:31:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:08 smithi110 ceph-mon[97633]: pgmap v13499: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:31:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:08 smithi110 ceph-mon[97633]: osdmap e924: 12 total, 12 up, 12 in 2022-09-10T11:31:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:09.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:09.146439+0000) 2022-09-10T11:31:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:09.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:09.622530+0000) 2022-09-10T11:31:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:10.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:10.146720+0000) 2022-09-10T11:31:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:10.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:10.622749+0000) 2022-09-10T11:31:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:10 smithi104 ceph-mon[97044]: pgmap v13501: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:31:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:10 smithi110 ceph-mon[97633]: pgmap v13501: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:31:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:11.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:11.146919+0000) 2022-09-10T11:31:12.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:11.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:11.622903+0000) 2022-09-10T11:31:12.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:31:12.192+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:31:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:12.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:12.147178+0000) 2022-09-10T11:31:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:12.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:12.623155+0000) 2022-09-10T11:31:13.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:12 smithi104 ceph-mon[97044]: pgmap v13502: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:31:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:12 smithi110 ceph-mon[97633]: pgmap v13502: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:31:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:13.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:13.147320+0000) 2022-09-10T11:31:14.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:13.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:13.623398+0000) 2022-09-10T11:31:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:14.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:14.147466+0000) 2022-09-10T11:31:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:14 smithi104 ceph-mon[97044]: osdmap e925: 12 total, 12 up, 12 in 2022-09-10T11:31:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:14 smithi104 ceph-mon[97044]: pgmap v13504: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:31:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:14.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:14.623622+0000) 2022-09-10T11:31:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:14 smithi110 ceph-mon[97633]: osdmap e925: 12 total, 12 up, 12 in 2022-09-10T11:31:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:14 smithi110 ceph-mon[97633]: pgmap v13504: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:31:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:15.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:15.147726+0000) 2022-09-10T11:31:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:15.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:15.623801+0000) 2022-09-10T11:31:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:16.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:16.147958+0000) 2022-09-10T11:31:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:16.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:16.624037+0000) 2022-09-10T11:31:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:16 smithi104 ceph-mon[97044]: pgmap v13505: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:31:17.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:16 smithi110 ceph-mon[97633]: pgmap v13505: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:31:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:17.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:17.148190+0000) 2022-09-10T11:31:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:17.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:17.624197+0000) 2022-09-10T11:31:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:18.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:18.148437+0000) 2022-09-10T11:31:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:18.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:18.624436+0000) 2022-09-10T11:31:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:18 smithi104 ceph-mon[97044]: pgmap v13506: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:31:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:18 smithi110 ceph-mon[97633]: pgmap v13506: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:31:19.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:19.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:19.148641+0000) 2022-09-10T11:31:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:19.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:19.624545+0000) 2022-09-10T11:31:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:20.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:20.148848+0000) 2022-09-10T11:31:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:20.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:20.624729+0000) 2022-09-10T11:31:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:20 smithi104 ceph-mon[97044]: pgmap v13507: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:31:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:20 smithi110 ceph-mon[97633]: pgmap v13507: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:31:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:21.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:21.148978+0000) 2022-09-10T11:31:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:21.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:21.624910+0000) 2022-09-10T11:31:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:22.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:22.149198+0000) 2022-09-10T11:31:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:22.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:22.625314+0000) 2022-09-10T11:31:23.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:22 smithi104 ceph-mon[97044]: pgmap v13508: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:31:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:22 smithi110 ceph-mon[97633]: pgmap v13508: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:31:23.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:23.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:23.149339+0000) 2022-09-10T11:31:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:23.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:23.625495+0000) 2022-09-10T11:31:24.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:24.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:24.149560+0000) 2022-09-10T11:31:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:24 smithi104 ceph-mon[97044]: pgmap v13509: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:31:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:24.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:24.625739+0000) 2022-09-10T11:31:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:24 smithi110 ceph-mon[97633]: pgmap v13509: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:31:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:25.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:25.149790+0000) 2022-09-10T11:31:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:25.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:25.625948+0000) 2022-09-10T11:31:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:26.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:26.150047+0000) 2022-09-10T11:31:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:26.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:26.626159+0000) 2022-09-10T11:31:27.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:26 smithi104 ceph-mon[97044]: pgmap v13510: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:31:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:26 smithi110 ceph-mon[97633]: pgmap v13510: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:31:27.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:31:27.206+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:31:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:27.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:27.150270+0000) 2022-09-10T11:31:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:27.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:27.626398+0000) 2022-09-10T11:31:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:28.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:28.150541+0000) 2022-09-10T11:31:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:28.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:28.626613+0000) 2022-09-10T11:31:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:28 smithi104 ceph-mon[97044]: pgmap v13511: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:28 smithi110 ceph-mon[97633]: pgmap v13511: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:29.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:29.150785+0000) 2022-09-10T11:31:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:29.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:29.626855+0000) 2022-09-10T11:31:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:30.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:30.150980+0000) 2022-09-10T11:31:31.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:30.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:30.627132+0000) 2022-09-10T11:31:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:30 smithi104 ceph-mon[97044]: pgmap v13512: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:30 smithi110 ceph-mon[97633]: pgmap v13512: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:31.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:31.151175+0000) 2022-09-10T11:31:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:31.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:31.627313+0000) 2022-09-10T11:31:32.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:32.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:32.151444+0000) 2022-09-10T11:31:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:32.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:32.627487+0000) 2022-09-10T11:31:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:32 smithi104 ceph-mon[97044]: pgmap v13513: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:32 smithi110 ceph-mon[97633]: pgmap v13513: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:33.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:33.151618+0000) 2022-09-10T11:31:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:33.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:33.627695+0000) 2022-09-10T11:31:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:34.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:34.151852+0000) 2022-09-10T11:31:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:34 smithi104 ceph-mon[97044]: pgmap v13514: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:34.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:34.627915+0000) 2022-09-10T11:31:35.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:34 smithi110 ceph-mon[97633]: pgmap v13514: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:35.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:35.152053+0000) 2022-09-10T11:31:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:35.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:35.628167+0000) 2022-09-10T11:31:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:36.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:36.152301+0000) 2022-09-10T11:31:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:36.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:36.628336+0000) 2022-09-10T11:31:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:36 smithi104 ceph-mon[97044]: pgmap v13515: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:36 smithi110 ceph-mon[97633]: pgmap v13515: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:37.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:37.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:37.152515+0000) 2022-09-10T11:31:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:37.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:37.628466+0000) 2022-09-10T11:31:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:38.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:38.152748+0000) 2022-09-10T11:31:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:38.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:38.628615+0000) 2022-09-10T11:31:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:38 smithi104 ceph-mon[97044]: pgmap v13516: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:38 smithi110 ceph-mon[97633]: pgmap v13516: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:39.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:39.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:39.152993+0000) 2022-09-10T11:31:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:39.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:39.628833+0000) 2022-09-10T11:31:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:40.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:40.153201+0000) 2022-09-10T11:31:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:40.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:40.629041+0000) 2022-09-10T11:31:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:40 smithi104 ceph-mon[97044]: pgmap v13517: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:41.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:40 smithi110 ceph-mon[97633]: pgmap v13517: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:41.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:41.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:41.153402+0000) 2022-09-10T11:31:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:41.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:41.629250+0000) 2022-09-10T11:31:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:42.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:42.153640+0000) 2022-09-10T11:31:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:31:42.209+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:31:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:42.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:42.629458+0000) 2022-09-10T11:31:43.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:42 smithi104 ceph-mon[97044]: pgmap v13518: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:42 smithi110 ceph-mon[97633]: pgmap v13518: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:43.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:43.153819+0000) 2022-09-10T11:31:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:43.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:43.629622+0000) 2022-09-10T11:31:44.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:44.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:44.154000+0000) 2022-09-10T11:31:45.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:44 smithi104 ceph-mon[97044]: pgmap v13519: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:44.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:44.629815+0000) 2022-09-10T11:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:44 smithi110 ceph-mon[97633]: pgmap v13519: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:45.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:45.154218+0000) 2022-09-10T11:31:46.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:31:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:31:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:31:46.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:31:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:45.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:45.629995+0000) 2022-09-10T11:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:31:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:31:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:46.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:46.154409+0000) 2022-09-10T11:31:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:46 smithi104 ceph-mon[97044]: pgmap v13520: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:46.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:46.630185+0000) 2022-09-10T11:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:46 smithi110 ceph-mon[97633]: pgmap v13520: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:47.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:47.154625+0000) 2022-09-10T11:31:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:47.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:47.630384+0000) 2022-09-10T11:31:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:48.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:48.154859+0000) 2022-09-10T11:31:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:48.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:48.630613+0000) 2022-09-10T11:31:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:48 smithi104 ceph-mon[97044]: pgmap v13521: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:48 smithi110 ceph-mon[97633]: pgmap v13521: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:49.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:49.155130+0000) 2022-09-10T11:31:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:49.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:49.630839+0000) 2022-09-10T11:31:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:50.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:50.155372+0000) 2022-09-10T11:31:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:50.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:50.631120+0000) 2022-09-10T11:31:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:50 smithi104 ceph-mon[97044]: pgmap v13522: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:50 smithi110 ceph-mon[97633]: pgmap v13522: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:51.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:51.155610+0000) 2022-09-10T11:31:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:51.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:51.631361+0000) 2022-09-10T11:31:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:52.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:52.155844+0000) 2022-09-10T11:31:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:52.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:52.631530+0000) 2022-09-10T11:31:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:52 smithi104 ceph-mon[97044]: pgmap v13523: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:52 smithi110 ceph-mon[97633]: pgmap v13523: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:53.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:53.156055+0000) 2022-09-10T11:31:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:53.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:53.631734+0000) 2022-09-10T11:31:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:54.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:54.156201+0000) 2022-09-10T11:31:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:54 smithi104 ceph-mon[97044]: pgmap v13524: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:54.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:54.632015+0000) 2022-09-10T11:31:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:54 smithi110 ceph-mon[97633]: pgmap v13524: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:55.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:55.156428+0000) 2022-09-10T11:31:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:55.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:55.632237+0000) 2022-09-10T11:31:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:56.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:56.156633+0000) 2022-09-10T11:31:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:56.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:56.632454+0000) 2022-09-10T11:31:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:56 smithi104 ceph-mon[97044]: pgmap v13525: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:56 smithi110 ceph-mon[97633]: pgmap v13525: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:31:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:57.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:57.156842+0000) 2022-09-10T11:31:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:31:57.217+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:31:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:57.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:57.632651+0000) 2022-09-10T11:31:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:58.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:58.157014+0000) 2022-09-10T11:31:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:58.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:58.632843+0000) 2022-09-10T11:31:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:31:58 smithi104 ceph-mon[97044]: pgmap v13526: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:31:58 smithi110 ceph-mon[97633]: pgmap v13526: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:31:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:31:59.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:59.157195+0000) 2022-09-10T11:32:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:31:59.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:31:59.633028+0000) 2022-09-10T11:32:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:00.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:00.157433+0000) 2022-09-10T11:32:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:00.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:00.633175+0000) 2022-09-10T11:32:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:00 smithi104 ceph-mon[97044]: pgmap v13527: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:00 smithi110 ceph-mon[97633]: pgmap v13527: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:01.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:01.157588+0000) 2022-09-10T11:32:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:01.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:01.633375+0000) 2022-09-10T11:32:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:02.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:02.157784+0000) 2022-09-10T11:32:03.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:02.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:02.633652+0000) 2022-09-10T11:32:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:02 smithi104 ceph-mon[97044]: pgmap v13528: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:02 smithi110 ceph-mon[97633]: pgmap v13528: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:03.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:03.158006+0000) 2022-09-10T11:32:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:03.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:03.633834+0000) 2022-09-10T11:32:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:04.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:04.158253+0000) 2022-09-10T11:32:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:04 smithi104 ceph-mon[97044]: pgmap v13529: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:04.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:04.634068+0000) 2022-09-10T11:32:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:04 smithi110 ceph-mon[97633]: pgmap v13529: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:05.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:05.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:05.158375+0000) 2022-09-10T11:32:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:05.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:05.634164+0000) 2022-09-10T11:32:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:06.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:06.158608+0000) 2022-09-10T11:32:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:06.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:06.634367+0000) 2022-09-10T11:32:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:06 smithi104 ceph-mon[97044]: pgmap v13530: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:32:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:06 smithi110 ceph-mon[97633]: pgmap v13530: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:32:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:07.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:07.158857+0000) 2022-09-10T11:32:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:07.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:07.634602+0000) 2022-09-10T11:32:08.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:08.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:08.159119+0000) 2022-09-10T11:32:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:08.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:08.634820+0000) 2022-09-10T11:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:08 smithi104 ceph-mon[97044]: pgmap v13531: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:32:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:08 smithi104 ceph-mon[97044]: osdmap e926: 12 total, 12 up, 12 in 2022-09-10T11:32:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:08 smithi110 ceph-mon[97633]: pgmap v13531: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:32:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:08 smithi110 ceph-mon[97633]: osdmap e926: 12 total, 12 up, 12 in 2022-09-10T11:32:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:09.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:09.159326+0000) 2022-09-10T11:32:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:09.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:09.635080+0000) 2022-09-10T11:32:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:10.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:10.159513+0000) 2022-09-10T11:32:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:10.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:10.635301+0000) 2022-09-10T11:32:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:10 smithi104 ceph-mon[97044]: pgmap v13533: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:32:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:10 smithi110 ceph-mon[97633]: pgmap v13533: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:32:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:11.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:11.159682+0000) 2022-09-10T11:32:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:11.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:11.635499+0000) 2022-09-10T11:32:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:12.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:12.159880+0000) 2022-09-10T11:32:12.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:32:12.231+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:32:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:12.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:12.635698+0000) 2022-09-10T11:32:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:12 smithi104 ceph-mon[97044]: pgmap v13534: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:32:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:12 smithi110 ceph-mon[97633]: pgmap v13534: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:32:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:13.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:13.160138+0000) 2022-09-10T11:32:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:13.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:13.635904+0000) 2022-09-10T11:32:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:14.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:14.160340+0000) 2022-09-10T11:32:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:14 smithi104 ceph-mon[97044]: osdmap e927: 12 total, 12 up, 12 in 2022-09-10T11:32:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:14 smithi104 ceph-mon[97044]: pgmap v13536: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:32:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:14.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:14.636164+0000) 2022-09-10T11:32:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:14 smithi110 ceph-mon[97633]: osdmap e927: 12 total, 12 up, 12 in 2022-09-10T11:32:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:14 smithi110 ceph-mon[97633]: pgmap v13536: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:32:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:15.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:15.160602+0000) 2022-09-10T11:32:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:15.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:15.636366+0000) 2022-09-10T11:32:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:16.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:16.160866+0000) 2022-09-10T11:32:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:16.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:16.636542+0000) 2022-09-10T11:32:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:16 smithi104 ceph-mon[97044]: pgmap v13537: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:32:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:16 smithi110 ceph-mon[97633]: pgmap v13537: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:32:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:17.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:17.161039+0000) 2022-09-10T11:32:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:17.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:17.636763+0000) 2022-09-10T11:32:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:18.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:18.161231+0000) 2022-09-10T11:32:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:18.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:18.637030+0000) 2022-09-10T11:32:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:18 smithi104 ceph-mon[97044]: pgmap v13538: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:32:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:18 smithi110 ceph-mon[97633]: pgmap v13538: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:32:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:19.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:19.161454+0000) 2022-09-10T11:32:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:19.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:19.637259+0000) 2022-09-10T11:32:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:20.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:20.161623+0000) 2022-09-10T11:32:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:20.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:20.637484+0000) 2022-09-10T11:32:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:20 smithi104 ceph-mon[97044]: pgmap v13539: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:32:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:20 smithi110 ceph-mon[97633]: pgmap v13539: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:32:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:21.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:21.161822+0000) 2022-09-10T11:32:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:21.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:21.637664+0000) 2022-09-10T11:32:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:22.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:22.162017+0000) 2022-09-10T11:32:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:22.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:22.637886+0000) 2022-09-10T11:32:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:22 smithi104 ceph-mon[97044]: pgmap v13540: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:32:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:22 smithi110 ceph-mon[97633]: pgmap v13540: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:32:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:23.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:23.162211+0000) 2022-09-10T11:32:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:23.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:23.638073+0000) 2022-09-10T11:32:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:24.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:24.162459+0000) 2022-09-10T11:32:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:24 smithi104 ceph-mon[97044]: pgmap v13541: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:32:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:24.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:24.638315+0000) 2022-09-10T11:32:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:24 smithi110 ceph-mon[97633]: pgmap v13541: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:32:25.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:25.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:25.162716+0000) 2022-09-10T11:32:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:25.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:25.638545+0000) 2022-09-10T11:32:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:26.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:26.162966+0000) 2022-09-10T11:32:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:26.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:26.638779+0000) 2022-09-10T11:32:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:26 smithi104 ceph-mon[97044]: pgmap v13542: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:32:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:26 smithi110 ceph-mon[97633]: pgmap v13542: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:32:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:27.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:27.163210+0000) 2022-09-10T11:32:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:32:27.232+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:32:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:27.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:27.638942+0000) 2022-09-10T11:32:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:28.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:28.163400+0000) 2022-09-10T11:32:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:28.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:28.639222+0000) 2022-09-10T11:32:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:28 smithi104 ceph-mon[97044]: pgmap v13543: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:28 smithi110 ceph-mon[97633]: pgmap v13543: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:29.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:29.163606+0000) 2022-09-10T11:32:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:29.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:29.639469+0000) 2022-09-10T11:32:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:30.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:30.163783+0000) 2022-09-10T11:32:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:30.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:30.639671+0000) 2022-09-10T11:32:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:30 smithi104 ceph-mon[97044]: pgmap v13544: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:30 smithi110 ceph-mon[97633]: pgmap v13544: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:31.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:31.164000+0000) 2022-09-10T11:32:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:31.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:31.639888+0000) 2022-09-10T11:32:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:32.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:32.164215+0000) 2022-09-10T11:32:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:32.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:32.640213+0000) 2022-09-10T11:32:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:32 smithi104 ceph-mon[97044]: pgmap v13545: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:32 smithi110 ceph-mon[97633]: pgmap v13545: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:33.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:33.164460+0000) 2022-09-10T11:32:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:33.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:33.640360+0000) 2022-09-10T11:32:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:34.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:34.164753+0000) 2022-09-10T11:32:35.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:34 smithi104 ceph-mon[97044]: pgmap v13546: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:34.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:34.640549+0000) 2022-09-10T11:32:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:34 smithi110 ceph-mon[97633]: pgmap v13546: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:35.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:35.164948+0000) 2022-09-10T11:32:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:35.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:35.640785+0000) 2022-09-10T11:32:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:36.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:36.165161+0000) 2022-09-10T11:32:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:36.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:36.641043+0000) 2022-09-10T11:32:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:36 smithi104 ceph-mon[97044]: pgmap v13547: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:36 smithi110 ceph-mon[97633]: pgmap v13547: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:37.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:37.165277+0000) 2022-09-10T11:32:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:37.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:37.641284+0000) 2022-09-10T11:32:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:38.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:38.165504+0000) 2022-09-10T11:32:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:38.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:38.641509+0000) 2022-09-10T11:32:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:38 smithi104 ceph-mon[97044]: pgmap v13548: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:38 smithi110 ceph-mon[97633]: pgmap v13548: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:39.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:39.165677+0000) 2022-09-10T11:32:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:39.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:39.641721+0000) 2022-09-10T11:32:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:40.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:40.165824+0000) 2022-09-10T11:32:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:40.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:40.641947+0000) 2022-09-10T11:32:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:40 smithi104 ceph-mon[97044]: pgmap v13549: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:40 smithi110 ceph-mon[97633]: pgmap v13549: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:41.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:41.166052+0000) 2022-09-10T11:32:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:41.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:41.642226+0000) 2022-09-10T11:32:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:42.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:42.166281+0000) 2022-09-10T11:32:42.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:32:42.246+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:32:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:42.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:42.642484+0000) 2022-09-10T11:32:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:42 smithi104 ceph-mon[97044]: pgmap v13550: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:42 smithi110 ceph-mon[97633]: pgmap v13550: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:43.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:43.166497+0000) 2022-09-10T11:32:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:43.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:43.642757+0000) 2022-09-10T11:32:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:44.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:44.166770+0000) 2022-09-10T11:32:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:44 smithi104 ceph-mon[97044]: pgmap v13551: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:44.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:44.643000+0000) 2022-09-10T11:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:44 smithi110 ceph-mon[97633]: pgmap v13551: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:45.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:45.166927+0000) 2022-09-10T11:32:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:32:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:32:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:32:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:32:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:45.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:45.643240+0000) 2022-09-10T11:32:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:32:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:32:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:46.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:46.167149+0000) 2022-09-10T11:32:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:46 smithi104 ceph-mon[97044]: pgmap v13552: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:46.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:46.643486+0000) 2022-09-10T11:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:46 smithi110 ceph-mon[97633]: pgmap v13552: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:47.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:47.167419+0000) 2022-09-10T11:32:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:47.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:47.643733+0000) 2022-09-10T11:32:48.564 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:48.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:48.167653+0000) 2022-09-10T11:32:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:48.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:48.643947+0000) 2022-09-10T11:32:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:48 smithi104 ceph-mon[97044]: pgmap v13553: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:48 smithi110 ceph-mon[97633]: pgmap v13553: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:49.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:49.167806+0000) 2022-09-10T11:32:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:49.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:49.644171+0000) 2022-09-10T11:32:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:50.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:50.168055+0000) 2022-09-10T11:32:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:50.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:50.644360+0000) 2022-09-10T11:32:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:50 smithi104 ceph-mon[97044]: pgmap v13554: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:50 smithi110 ceph-mon[97633]: pgmap v13554: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:51.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:51.168247+0000) 2022-09-10T11:32:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:51.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:51.644547+0000) 2022-09-10T11:32:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:52.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:52.168424+0000) 2022-09-10T11:32:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:52.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:52.644801+0000) 2022-09-10T11:32:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:52 smithi104 ceph-mon[97044]: pgmap v13555: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:52 smithi110 ceph-mon[97633]: pgmap v13555: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:53.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:53.168627+0000) 2022-09-10T11:32:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:53.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:53.645054+0000) 2022-09-10T11:32:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:54.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:54.168911+0000) 2022-09-10T11:32:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:54 smithi104 ceph-mon[97044]: pgmap v13556: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:54.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:54.645276+0000) 2022-09-10T11:32:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:54 smithi110 ceph-mon[97633]: pgmap v13556: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:55.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:55.169082+0000) 2022-09-10T11:32:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:55.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:55.645407+0000) 2022-09-10T11:32:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:56.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:56.169296+0000) 2022-09-10T11:32:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:56.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:56.645663+0000) 2022-09-10T11:32:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:56 smithi104 ceph-mon[97044]: pgmap v13557: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:57.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:56 smithi110 ceph-mon[97633]: pgmap v13557: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:32:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:32:57.260+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:32:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:57.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:57.169568+0000) 2022-09-10T11:32:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:57.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:57.645913+0000) 2022-09-10T11:32:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:58.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:58.169743+0000) 2022-09-10T11:32:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:58.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:58.646198+0000) 2022-09-10T11:32:59.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:32:58 smithi104 ceph-mon[97044]: pgmap v13558: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:32:58 smithi110 ceph-mon[97633]: pgmap v13558: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:32:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:32:59.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:59.169891+0000) 2022-09-10T11:33:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:32:59.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:32:59.646352+0000) 2022-09-10T11:33:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:00.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:00.170021+0000) 2022-09-10T11:33:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:00.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:00.646615+0000) 2022-09-10T11:33:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:00 smithi104 ceph-mon[97044]: pgmap v13559: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:00 smithi110 ceph-mon[97633]: pgmap v13559: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:01.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:01.170244+0000) 2022-09-10T11:33:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:01.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:01.646823+0000) 2022-09-10T11:33:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:02.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:02.170546+0000) 2022-09-10T11:33:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:02.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:02.647026+0000) 2022-09-10T11:33:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:02 smithi104 ceph-mon[97044]: pgmap v13560: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:02 smithi110 ceph-mon[97633]: pgmap v13560: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:03.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:03.170783+0000) 2022-09-10T11:33:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:03.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:03.647218+0000) 2022-09-10T11:33:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:04.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:04.171006+0000) 2022-09-10T11:33:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:04 smithi104 ceph-mon[97044]: pgmap v13561: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:04.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:04.647445+0000) 2022-09-10T11:33:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:04 smithi110 ceph-mon[97633]: pgmap v13561: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:05.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:05.171195+0000) 2022-09-10T11:33:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:05.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:05.647632+0000) 2022-09-10T11:33:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:06.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:06.171364+0000) 2022-09-10T11:33:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:06.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:06.647856+0000) 2022-09-10T11:33:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:06 smithi104 ceph-mon[97044]: pgmap v13562: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:33:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:06 smithi110 ceph-mon[97633]: pgmap v13562: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:33:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:07.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:07.171542+0000) 2022-09-10T11:33:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:07.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:07.648079+0000) 2022-09-10T11:33:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:08.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:08.171787+0000) 2022-09-10T11:33:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:08.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:08.648363+0000) 2022-09-10T11:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:08 smithi104 ceph-mon[97044]: pgmap v13563: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:33:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:08 smithi104 ceph-mon[97044]: osdmap e928: 12 total, 12 up, 12 in 2022-09-10T11:33:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:08 smithi110 ceph-mon[97633]: pgmap v13563: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:33:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:08 smithi110 ceph-mon[97633]: osdmap e928: 12 total, 12 up, 12 in 2022-09-10T11:33:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:09.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:09.172020+0000) 2022-09-10T11:33:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:09.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:09.648614+0000) 2022-09-10T11:33:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:10.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:10.172366+0000) 2022-09-10T11:33:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:10.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:10.648859+0000) 2022-09-10T11:33:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:10 smithi104 ceph-mon[97044]: pgmap v13565: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:33:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:10 smithi110 ceph-mon[97633]: pgmap v13565: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:33:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:11.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:11.172575+0000) 2022-09-10T11:33:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:11.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:11.649155+0000) 2022-09-10T11:33:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:12.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:12.172708+0000) 2022-09-10T11:33:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:33:12.273+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:33:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:12.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:12.649348+0000) 2022-09-10T11:33:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:12 smithi104 ceph-mon[97044]: pgmap v13566: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:33:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:12 smithi110 ceph-mon[97633]: pgmap v13566: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:33:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:13.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:13.172922+0000) 2022-09-10T11:33:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:13.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:13.649585+0000) 2022-09-10T11:33:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:14.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:14.173163+0000) 2022-09-10T11:33:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:14 smithi104 ceph-mon[97044]: osdmap e929: 12 total, 12 up, 12 in 2022-09-10T11:33:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:14 smithi104 ceph-mon[97044]: pgmap v13568: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:33:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:14.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:14.649824+0000) 2022-09-10T11:33:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:14 smithi110 ceph-mon[97633]: osdmap e929: 12 total, 12 up, 12 in 2022-09-10T11:33:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:14 smithi110 ceph-mon[97633]: pgmap v13568: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:33:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:15.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:15.173295+0000) 2022-09-10T11:33:16.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:15.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:15.650030+0000) 2022-09-10T11:33:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:16.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:16.173519+0000) 2022-09-10T11:33:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:16.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:16.650241+0000) 2022-09-10T11:33:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:16 smithi104 ceph-mon[97044]: pgmap v13569: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:33:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:16 smithi110 ceph-mon[97633]: pgmap v13569: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:33:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:17.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:17.173723+0000) 2022-09-10T11:33:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:17.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:17.650433+0000) 2022-09-10T11:33:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:18.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:18.173939+0000) 2022-09-10T11:33:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:18.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:18.650675+0000) 2022-09-10T11:33:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:18 smithi104 ceph-mon[97044]: pgmap v13570: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:33:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:18 smithi110 ceph-mon[97633]: pgmap v13570: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:33:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:19.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:19.174111+0000) 2022-09-10T11:33:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:19.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:19.650887+0000) 2022-09-10T11:33:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:20.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:20.174300+0000) 2022-09-10T11:33:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:20.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:20.651185+0000) 2022-09-10T11:33:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:20 smithi104 ceph-mon[97044]: pgmap v13571: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:33:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:20 smithi110 ceph-mon[97633]: pgmap v13571: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:33:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:21.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:21.174465+0000) 2022-09-10T11:33:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:21.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:21.651391+0000) 2022-09-10T11:33:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:22.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:22.174750+0000) 2022-09-10T11:33:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:22.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:22.651569+0000) 2022-09-10T11:33:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:22 smithi104 ceph-mon[97044]: pgmap v13572: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:33:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:22 smithi110 ceph-mon[97633]: pgmap v13572: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:33:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:23.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:23.174994+0000) 2022-09-10T11:33:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:23.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:23.651813+0000) 2022-09-10T11:33:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:24.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:24.175265+0000) 2022-09-10T11:33:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:24 smithi104 ceph-mon[97044]: pgmap v13573: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:33:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:24.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:24.652015+0000) 2022-09-10T11:33:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:24 smithi110 ceph-mon[97633]: pgmap v13573: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:33:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:25.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:25.175484+0000) 2022-09-10T11:33:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:25.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:25.652266+0000) 2022-09-10T11:33:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:26.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:26.175725+0000) 2022-09-10T11:33:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:26.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:26.652441+0000) 2022-09-10T11:33:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:26 smithi104 ceph-mon[97044]: pgmap v13574: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:33:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:26 smithi110 ceph-mon[97633]: pgmap v13574: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:33:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:27.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:27.175891+0000) 2022-09-10T11:33:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:33:27.288+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:33:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:27.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:27.652595+0000) 2022-09-10T11:33:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:28.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:28.176133+0000) 2022-09-10T11:33:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:28.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:28.652761+0000) 2022-09-10T11:33:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:28 smithi104 ceph-mon[97044]: pgmap v13575: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:28 smithi110 ceph-mon[97633]: pgmap v13575: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:29.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:29.176370+0000) 2022-09-10T11:33:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:29.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:29.652992+0000) 2022-09-10T11:33:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:30.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:30.176651+0000) 2022-09-10T11:33:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:30.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:30.653182+0000) 2022-09-10T11:33:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:30 smithi104 ceph-mon[97044]: pgmap v13576: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:30 smithi110 ceph-mon[97633]: pgmap v13576: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:31.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:31.176899+0000) 2022-09-10T11:33:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:31.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:31.653417+0000) 2022-09-10T11:33:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:32.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:32.177174+0000) 2022-09-10T11:33:33.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:32.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:32.653653+0000) 2022-09-10T11:33:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:32 smithi104 ceph-mon[97044]: pgmap v13577: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:32 smithi110 ceph-mon[97633]: pgmap v13577: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:33.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:33.177312+0000) 2022-09-10T11:33:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:33.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:33.653928+0000) 2022-09-10T11:33:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:34.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:34.177554+0000) 2022-09-10T11:33:35.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:34 smithi104 ceph-mon[97044]: pgmap v13578: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:34.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:34.654137+0000) 2022-09-10T11:33:35.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:34 smithi110 ceph-mon[97633]: pgmap v13578: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:35.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:35.177808+0000) 2022-09-10T11:33:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:35.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:35.654312+0000) 2022-09-10T11:33:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:36.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:36.178002+0000) 2022-09-10T11:33:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:36.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:36.654491+0000) 2022-09-10T11:33:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:36 smithi104 ceph-mon[97044]: pgmap v13579: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:36 smithi110 ceph-mon[97633]: pgmap v13579: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:37.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:37.178246+0000) 2022-09-10T11:33:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:37.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:37.654735+0000) 2022-09-10T11:33:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:38.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:38.178460+0000) 2022-09-10T11:33:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:38.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:38.654978+0000) 2022-09-10T11:33:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:38 smithi104 ceph-mon[97044]: pgmap v13580: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:38 smithi110 ceph-mon[97633]: pgmap v13580: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:33:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:39.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:39.178620+0000) 2022-09-10T11:33:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:39.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:39.655189+0000) 2022-09-10T11:33:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:40.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:40.178884+0000) 2022-09-10T11:33:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:40.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:40.655318+0000) 2022-09-10T11:33:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:40 smithi104 ceph-mon[97044]: pgmap v13581: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:40 smithi110 ceph-mon[97633]: pgmap v13581: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:41.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:41.179086+0000) 2022-09-10T11:33:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:41.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:41.655549+0000) 2022-09-10T11:33:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:42.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:42.179340+0000) 2022-09-10T11:33:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:33:42.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:33:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:42.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:42.655768+0000) 2022-09-10T11:33:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:43 smithi104 ceph-mon[97044]: pgmap v13582: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:43 smithi110 ceph-mon[97633]: pgmap v13582: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:43.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:43.179590+0000) 2022-09-10T11:33:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:43.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:43.655954+0000) 2022-09-10T11:33:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:44.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:44.179777+0000) 2022-09-10T11:33:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:44 smithi104 ceph-mon[97044]: pgmap v13583: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:44.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:44.656140+0000) 2022-09-10T11:33:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:44 smithi110 ceph-mon[97633]: pgmap v13583: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:45.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:45.179966+0000) 2022-09-10T11:33:46.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:33:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:33:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:33:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:33:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:45.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:45.656364+0000) 2022-09-10T11:33:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:33:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:46.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:46.180208+0000) 2022-09-10T11:33:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:46.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:46.656601+0000) 2022-09-10T11:33:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:46 smithi110 ceph-mon[97633]: pgmap v13584: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:46 smithi104 ceph-mon[97044]: pgmap v13584: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:47.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:47.180435+0000) 2022-09-10T11:33:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:47.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:47.656767+0000) 2022-09-10T11:33:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:48.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:48.180703+0000) 2022-09-10T11:33:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:48.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:48.656946+0000) 2022-09-10T11:33:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:48 smithi104 ceph-mon[97044]: pgmap v13585: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:48 smithi110 ceph-mon[97633]: pgmap v13585: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:49.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:49.180971+0000) 2022-09-10T11:33:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:49.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:49.657219+0000) 2022-09-10T11:33:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:50.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:50.181161+0000) 2022-09-10T11:33:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:50.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:50.657393+0000) 2022-09-10T11:33:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:51 smithi104 ceph-mon[97044]: pgmap v13586: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T11:33:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:51 smithi110 ceph-mon[97633]: pgmap v13586: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T11:33:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:51.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:51.181381+0000) 2022-09-10T11:33:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:51.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:51.657633+0000) 2022-09-10T11:33:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:52.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:52.181601+0000) 2022-09-10T11:33:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:52.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:52.657836+0000) 2022-09-10T11:33:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:53 smithi104 ceph-mon[97044]: pgmap v13587: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:53 smithi110 ceph-mon[97633]: pgmap v13587: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:53.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:53.181781+0000) 2022-09-10T11:33:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:53.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:53.658081+0000) 2022-09-10T11:33:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:54.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:54.182035+0000) 2022-09-10T11:33:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:54.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:54.658256+0000) 2022-09-10T11:33:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:54 smithi110 ceph-mon[97633]: pgmap v13588: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:54 smithi104 ceph-mon[97044]: pgmap v13588: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:55.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:55.182259+0000) 2022-09-10T11:33:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:55.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:55.658503+0000) 2022-09-10T11:33:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:56.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:56.182503+0000) 2022-09-10T11:33:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:56.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:56.658709+0000) 2022-09-10T11:33:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:56 smithi104 ceph-mon[97044]: pgmap v13589: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:56 smithi110 ceph-mon[97633]: pgmap v13589: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:33:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:57.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:57.182730+0000) 2022-09-10T11:33:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:33:57.315+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:33:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:57.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:57.658935+0000) 2022-09-10T11:33:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:58.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:58.182958+0000) 2022-09-10T11:33:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:58.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:58.659219+0000) 2022-09-10T11:33:59.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:33:59 smithi104 ceph-mon[97044]: pgmap v13590: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:33:59 smithi110 ceph-mon[97633]: pgmap v13590: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:33:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:33:59.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:59.183215+0000) 2022-09-10T11:34:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:33:59.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:33:59.659461+0000) 2022-09-10T11:34:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:00.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:00.183440+0000) 2022-09-10T11:34:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:00.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:00.659713+0000) 2022-09-10T11:34:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:01 smithi104 ceph-mon[97044]: pgmap v13591: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:01 smithi110 ceph-mon[97633]: pgmap v13591: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:01.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:01.183596+0000) 2022-09-10T11:34:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:01.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:01.659940+0000) 2022-09-10T11:34:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:02.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:02.183829+0000) 2022-09-10T11:34:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:02.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:02.660154+0000) 2022-09-10T11:34:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:03 smithi104 ceph-mon[97044]: pgmap v13592: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:03 smithi110 ceph-mon[97633]: pgmap v13592: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:03.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:03.184063+0000) 2022-09-10T11:34:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:03.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:03.660301+0000) 2022-09-10T11:34:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:04.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:04.184234+0000) 2022-09-10T11:34:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:04.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:04.660519+0000) 2022-09-10T11:34:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:04 smithi110 ceph-mon[97633]: pgmap v13593: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:04 smithi104 ceph-mon[97044]: pgmap v13593: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:05.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:05.184413+0000) 2022-09-10T11:34:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:05.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:05.660656+0000) 2022-09-10T11:34:06.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:06.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:06.184633+0000) 2022-09-10T11:34:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:06.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:06.660893+0000) 2022-09-10T11:34:07.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:07 smithi104 ceph-mon[97044]: pgmap v13594: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:34:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:07 smithi110 ceph-mon[97633]: pgmap v13594: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:34:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:07.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:07.184850+0000) 2022-09-10T11:34:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:07.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:07.661167+0000) 2022-09-10T11:34:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:08.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:08.185052+0000) 2022-09-10T11:34:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:08.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:08.661360+0000) 2022-09-10T11:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:09 smithi104 ceph-mon[97044]: pgmap v13595: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:34:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:09 smithi104 ceph-mon[97044]: osdmap e930: 12 total, 12 up, 12 in 2022-09-10T11:34:09.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:09 smithi110 ceph-mon[97633]: pgmap v13595: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:34:09.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:09 smithi110 ceph-mon[97633]: osdmap e930: 12 total, 12 up, 12 in 2022-09-10T11:34:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:09.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:09.185192+0000) 2022-09-10T11:34:10.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:09.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:09.661574+0000) 2022-09-10T11:34:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:10.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:10.185408+0000) 2022-09-10T11:34:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:10.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:10.661782+0000) 2022-09-10T11:34:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:11 smithi104 ceph-mon[97044]: pgmap v13597: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:34:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:11 smithi110 ceph-mon[97633]: pgmap v13597: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:34:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:11.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:11.185689+0000) 2022-09-10T11:34:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:11.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:11.662012+0000) 2022-09-10T11:34:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:34:12.321+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:34:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:12.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:12.185878+0000) 2022-09-10T11:34:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:12.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:12.662263+0000) 2022-09-10T11:34:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:13 smithi104 ceph-mon[97044]: pgmap v13598: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:34:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:13 smithi110 ceph-mon[97633]: pgmap v13598: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:34:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:13.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:13.186173+0000) 2022-09-10T11:34:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:13.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:13.662474+0000) 2022-09-10T11:34:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:14.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:14.186365+0000) 2022-09-10T11:34:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:14.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:14.662684+0000) 2022-09-10T11:34:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:14 smithi110 ceph-mon[97633]: osdmap e931: 12 total, 12 up, 12 in 2022-09-10T11:34:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:14 smithi110 ceph-mon[97633]: pgmap v13600: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:14 smithi104 ceph-mon[97044]: osdmap e931: 12 total, 12 up, 12 in 2022-09-10T11:34:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:14 smithi104 ceph-mon[97044]: pgmap v13600: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:34:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:15.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:15.186610+0000) 2022-09-10T11:34:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:15.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:15.662899+0000) 2022-09-10T11:34:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:16.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:16.186808+0000) 2022-09-10T11:34:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:16.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:16.663180+0000) 2022-09-10T11:34:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:17 smithi104 ceph-mon[97044]: pgmap v13601: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:34:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:17 smithi110 ceph-mon[97633]: pgmap v13601: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:34:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:17.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:17.187029+0000) 2022-09-10T11:34:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:17.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:17.663415+0000) 2022-09-10T11:34:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:18.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:18.187234+0000) 2022-09-10T11:34:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:18.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:18.663603+0000) 2022-09-10T11:34:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:19 smithi104 ceph-mon[97044]: pgmap v13602: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:34:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:19 smithi110 ceph-mon[97633]: pgmap v13602: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:34:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:19.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:19.187444+0000) 2022-09-10T11:34:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:19.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:19.663816+0000) 2022-09-10T11:34:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:20.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:20.187707+0000) 2022-09-10T11:34:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:20.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:20.664121+0000) 2022-09-10T11:34:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:21 smithi104 ceph-mon[97044]: pgmap v13603: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:34:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:21 smithi110 ceph-mon[97633]: pgmap v13603: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:34:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:21.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:21.187938+0000) 2022-09-10T11:34:22.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:21.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:21.664297+0000) 2022-09-10T11:34:22.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:22.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:22.188246+0000) 2022-09-10T11:34:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:22.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:22.664545+0000) 2022-09-10T11:34:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:23 smithi104 ceph-mon[97044]: pgmap v13604: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:34:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:23 smithi110 ceph-mon[97633]: pgmap v13604: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:34:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:23.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:23.188484+0000) 2022-09-10T11:34:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:23.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:23.664803+0000) 2022-09-10T11:34:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:24.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:24.188761+0000) 2022-09-10T11:34:25.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:24.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:24.665026+0000) 2022-09-10T11:34:25.166 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:24 smithi110 ceph-mon[97633]: pgmap v13605: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:34:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:24 smithi104 ceph-mon[97044]: pgmap v13605: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:34:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:25.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:25.189027+0000) 2022-09-10T11:34:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:25.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:25.665236+0000) 2022-09-10T11:34:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:26.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:26.189277+0000) 2022-09-10T11:34:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:26.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:26.665428+0000) 2022-09-10T11:34:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:27 smithi104 ceph-mon[97044]: pgmap v13606: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:34:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:27 smithi110 ceph-mon[97633]: pgmap v13606: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:34:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:27.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:27.189515+0000) 2022-09-10T11:34:27.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:34:27.327+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:34:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:27.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:27.665620+0000) 2022-09-10T11:34:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:28.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:28.189725+0000) 2022-09-10T11:34:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:28.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:28.665834+0000) 2022-09-10T11:34:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:29 smithi104 ceph-mon[97044]: pgmap v13607: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:29 smithi110 ceph-mon[97633]: pgmap v13607: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:29.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:29.189908+0000) 2022-09-10T11:34:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:29.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:29.666020+0000) 2022-09-10T11:34:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:30.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:30.190077+0000) 2022-09-10T11:34:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:30.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:30.666214+0000) 2022-09-10T11:34:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:31 smithi104 ceph-mon[97044]: pgmap v13608: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:31 smithi110 ceph-mon[97633]: pgmap v13608: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:31.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:31.190372+0000) 2022-09-10T11:34:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:31.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:31.666427+0000) 2022-09-10T11:34:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:32.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:32.190608+0000) 2022-09-10T11:34:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:32.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:32.666615+0000) 2022-09-10T11:34:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:33 smithi104 ceph-mon[97044]: pgmap v13609: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:33 smithi110 ceph-mon[97633]: pgmap v13609: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:33.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:33.190768+0000) 2022-09-10T11:34:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:33.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:33.666825+0000) 2022-09-10T11:34:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:34.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:34.190989+0000) 2022-09-10T11:34:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:34.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:34.667087+0000) 2022-09-10T11:34:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:34 smithi110 ceph-mon[97633]: pgmap v13610: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:34 smithi104 ceph-mon[97044]: pgmap v13610: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:35.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:35.191233+0000) 2022-09-10T11:34:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:35.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:35.667336+0000) 2022-09-10T11:34:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:36.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:36.191481+0000) 2022-09-10T11:34:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:36.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:36.667610+0000) 2022-09-10T11:34:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:37 smithi104 ceph-mon[97044]: pgmap v13611: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:37 smithi110 ceph-mon[97633]: pgmap v13611: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:37.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:37.191781+0000) 2022-09-10T11:34:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:37.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:37.667835+0000) 2022-09-10T11:34:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:38.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:38.192001+0000) 2022-09-10T11:34:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:38.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:38.668054+0000) 2022-09-10T11:34:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:39 smithi104 ceph-mon[97044]: pgmap v13612: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:39 smithi110 ceph-mon[97633]: pgmap v13612: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:39.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:39.192225+0000) 2022-09-10T11:34:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:39.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:39.668289+0000) 2022-09-10T11:34:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:40.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:40.192396+0000) 2022-09-10T11:34:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:40.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:40.668531+0000) 2022-09-10T11:34:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:41 smithi104 ceph-mon[97044]: pgmap v13613: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:41 smithi110 ceph-mon[97633]: pgmap v13613: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:41.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:41.192602+0000) 2022-09-10T11:34:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:41.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:41.668728+0000) 2022-09-10T11:34:42.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:34:42.338+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:34:42.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:42.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:42.192829+0000) 2022-09-10T11:34:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:42.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:42.668992+0000) 2022-09-10T11:34:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:43 smithi104 ceph-mon[97044]: pgmap v13614: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:43 smithi110 ceph-mon[97633]: pgmap v13614: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:43.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:43.193060+0000) 2022-09-10T11:34:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:43.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:43.669214+0000) 2022-09-10T11:34:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:44.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:44.193320+0000) 2022-09-10T11:34:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:44.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:44.669425+0000) 2022-09-10T11:34:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:44 smithi110 ceph-mon[97633]: pgmap v13615: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:44 smithi104 ceph-mon[97044]: pgmap v13615: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:45.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:45.193507+0000) 2022-09-10T11:34:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:45.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:45.669633+0000) 2022-09-10T11:34:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:34:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:34:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:34:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:34:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:46.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:46.193730+0000) 2022-09-10T11:34:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:46.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:46.669827+0000) 2022-09-10T11:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:46 smithi110 ceph-mon[97633]: pgmap v13616: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:46 smithi104 ceph-mon[97044]: pgmap v13616: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:47.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:47.193977+0000) 2022-09-10T11:34:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:47.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:47.670046+0000) 2022-09-10T11:34:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:48.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:48.194221+0000) 2022-09-10T11:34:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:48.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:48.670347+0000) 2022-09-10T11:34:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:49 smithi104 ceph-mon[97044]: pgmap v13617: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:49 smithi110 ceph-mon[97633]: pgmap v13617: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:49.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:49.194354+0000) 2022-09-10T11:34:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:49.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:49.670587+0000) 2022-09-10T11:34:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:50.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:50.194588+0000) 2022-09-10T11:34:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:50.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:50.670770+0000) 2022-09-10T11:34:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:51 smithi104 ceph-mon[97044]: pgmap v13618: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:51 smithi110 ceph-mon[97633]: pgmap v13618: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:51.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:51.194840+0000) 2022-09-10T11:34:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:51.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:51.670970+0000) 2022-09-10T11:34:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:52.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:52.195035+0000) 2022-09-10T11:34:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:52.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:52.671188+0000) 2022-09-10T11:34:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:53 smithi104 ceph-mon[97044]: pgmap v13619: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:53 smithi110 ceph-mon[97633]: pgmap v13619: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:53.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:53.195192+0000) 2022-09-10T11:34:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:53.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:53.671349+0000) 2022-09-10T11:34:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:54.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:54.195386+0000) 2022-09-10T11:34:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:54.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:54.671566+0000) 2022-09-10T11:34:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:54 smithi110 ceph-mon[97633]: pgmap v13620: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:54 smithi104 ceph-mon[97044]: pgmap v13620: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:55.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:55.195618+0000) 2022-09-10T11:34:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:55.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:55.671821+0000) 2022-09-10T11:34:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:56.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:56.195832+0000) 2022-09-10T11:34:57.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:56.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:56.672082+0000) 2022-09-10T11:34:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:57 smithi104 ceph-mon[97044]: pgmap v13621: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:57 smithi110 ceph-mon[97633]: pgmap v13621: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:34:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:57.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:57.196040+0000) 2022-09-10T11:34:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:34:57.353+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:34:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:57.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:57.672346+0000) 2022-09-10T11:34:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:58.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:58.196286+0000) 2022-09-10T11:34:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:58.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:58.672531+0000) 2022-09-10T11:34:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:34:59 smithi104 ceph-mon[97044]: pgmap v13622: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:34:59 smithi110 ceph-mon[97633]: pgmap v13622: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:34:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:34:59.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:59.196492+0000) 2022-09-10T11:35:00.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:34:59.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:34:59.672704+0000) 2022-09-10T11:35:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:00.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:00.196706+0000) 2022-09-10T11:35:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:00.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:00.672951+0000) 2022-09-10T11:35:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:01 smithi104 ceph-mon[97044]: pgmap v13623: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:01 smithi110 ceph-mon[97633]: pgmap v13623: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:01.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:01.196897+0000) 2022-09-10T11:35:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:01.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:01.673151+0000) 2022-09-10T11:35:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:02.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:02.197164+0000) 2022-09-10T11:35:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:02.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:02.673322+0000) 2022-09-10T11:35:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:03 smithi104 ceph-mon[97044]: pgmap v13624: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:03 smithi110 ceph-mon[97633]: pgmap v13624: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:03.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:03.197415+0000) 2022-09-10T11:35:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:03.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:03.673535+0000) 2022-09-10T11:35:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:04.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:04.197669+0000) 2022-09-10T11:35:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:04.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:04.673741+0000) 2022-09-10T11:35:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:04 smithi110 ceph-mon[97633]: pgmap v13625: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:04 smithi104 ceph-mon[97044]: pgmap v13625: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:05.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:05.197939+0000) 2022-09-10T11:35:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:05.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:05.673971+0000) 2022-09-10T11:35:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:06.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:06.198200+0000) 2022-09-10T11:35:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:06.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:06.674215+0000) 2022-09-10T11:35:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:07 smithi104 ceph-mon[97044]: pgmap v13626: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:35:07.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:07 smithi110 ceph-mon[97633]: pgmap v13626: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:35:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:07.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:07.198443+0000) 2022-09-10T11:35:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:07.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:07.674427+0000) 2022-09-10T11:35:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:08.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:08.198649+0000) 2022-09-10T11:35:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:08.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:08.674675+0000) 2022-09-10T11:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:09 smithi104 ceph-mon[97044]: pgmap v13627: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:35:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:09 smithi104 ceph-mon[97044]: osdmap e932: 12 total, 12 up, 12 in 2022-09-10T11:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:09 smithi110 ceph-mon[97633]: pgmap v13627: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:35:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:09 smithi110 ceph-mon[97633]: osdmap e932: 12 total, 12 up, 12 in 2022-09-10T11:35:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:09.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:09.198872+0000) 2022-09-10T11:35:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:09.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:09.674856+0000) 2022-09-10T11:35:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:10.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:10.199196+0000) 2022-09-10T11:35:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:10.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:10.675129+0000) 2022-09-10T11:35:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:11 smithi104 ceph-mon[97044]: pgmap v13629: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:35:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:11 smithi110 ceph-mon[97633]: pgmap v13629: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:35:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:11.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:11.199371+0000) 2022-09-10T11:35:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:11.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:11.675306+0000) 2022-09-10T11:35:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:12.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:12.199530+0000) 2022-09-10T11:35:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:35:12.366+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:35:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:12.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:12.675487+0000) 2022-09-10T11:35:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:13 smithi104 ceph-mon[97044]: pgmap v13630: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:35:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:13 smithi110 ceph-mon[97633]: pgmap v13630: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:35:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:13.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:13.199742+0000) 2022-09-10T11:35:14.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:13.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:13.675737+0000) 2022-09-10T11:35:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:14.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:14.200067+0000) 2022-09-10T11:35:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:14.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:14.676001+0000) 2022-09-10T11:35:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:14 smithi110 ceph-mon[97633]: osdmap e933: 12 total, 12 up, 12 in 2022-09-10T11:35:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:14 smithi110 ceph-mon[97633]: pgmap v13632: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:35:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:14 smithi104 ceph-mon[97044]: osdmap e933: 12 total, 12 up, 12 in 2022-09-10T11:35:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:14 smithi104 ceph-mon[97044]: pgmap v13632: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:35:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:15.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:15.200351+0000) 2022-09-10T11:35:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:15.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:15.676214+0000) 2022-09-10T11:35:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:16.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:16.200573+0000) 2022-09-10T11:35:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:16.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:16.676470+0000) 2022-09-10T11:35:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:17 smithi104 ceph-mon[97044]: pgmap v13633: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:35:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:17 smithi110 ceph-mon[97633]: pgmap v13633: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:35:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:17.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:17.200778+0000) 2022-09-10T11:35:18.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:17.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:17.676707+0000) 2022-09-10T11:35:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:18.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:18.200994+0000) 2022-09-10T11:35:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:18.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:18.676945+0000) 2022-09-10T11:35:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:19 smithi104 ceph-mon[97044]: pgmap v13634: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:35:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:19 smithi110 ceph-mon[97633]: pgmap v13634: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:35:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:19.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:19.201238+0000) 2022-09-10T11:35:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:19.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:19.677201+0000) 2022-09-10T11:35:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:20.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:20.201546+0000) 2022-09-10T11:35:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:20.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:20.677445+0000) 2022-09-10T11:35:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:21 smithi104 ceph-mon[97044]: pgmap v13635: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:35:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:21 smithi110 ceph-mon[97633]: pgmap v13635: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:35:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:21.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:21.201734+0000) 2022-09-10T11:35:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:21.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:21.677676+0000) 2022-09-10T11:35:22.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:22.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:22.201955+0000) 2022-09-10T11:35:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:22.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:22.677918+0000) 2022-09-10T11:35:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:23 smithi104 ceph-mon[97044]: pgmap v13636: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:35:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:23 smithi110 ceph-mon[97633]: pgmap v13636: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:35:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:23.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:23.202234+0000) 2022-09-10T11:35:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:23.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:23.678174+0000) 2022-09-10T11:35:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:24.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:24.202447+0000) 2022-09-10T11:35:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:24.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:24.678426+0000) 2022-09-10T11:35:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:24 smithi110 ceph-mon[97633]: pgmap v13637: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:35:25.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:24 smithi104 ceph-mon[97044]: pgmap v13637: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:35:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:25.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:25.202663+0000) 2022-09-10T11:35:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:25.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:25.678686+0000) 2022-09-10T11:35:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:26.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:26.202912+0000) 2022-09-10T11:35:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:26.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:26.678938+0000) 2022-09-10T11:35:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:27 smithi104 ceph-mon[97044]: pgmap v13638: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:35:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:27 smithi110 ceph-mon[97633]: pgmap v13638: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:35:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:27.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:27.203172+0000) 2022-09-10T11:35:27.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:35:27.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:35:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:27.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:27.679149+0000) 2022-09-10T11:35:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:28.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:28.203362+0000) 2022-09-10T11:35:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:28.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:28.679337+0000) 2022-09-10T11:35:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:29 smithi104 ceph-mon[97044]: pgmap v13639: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:29 smithi110 ceph-mon[97633]: pgmap v13639: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:29.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:29.203522+0000) 2022-09-10T11:35:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:29.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:29.679536+0000) 2022-09-10T11:35:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:30.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:30.203649+0000) 2022-09-10T11:35:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:30.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:30.679779+0000) 2022-09-10T11:35:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:31 smithi104 ceph-mon[97044]: pgmap v13640: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:31 smithi110 ceph-mon[97633]: pgmap v13640: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:31.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:31.203866+0000) 2022-09-10T11:35:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:31.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:31.680004+0000) 2022-09-10T11:35:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:32.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:32.204133+0000) 2022-09-10T11:35:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:32.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:32.680244+0000) 2022-09-10T11:35:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:33 smithi104 ceph-mon[97044]: pgmap v13641: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:33 smithi110 ceph-mon[97633]: pgmap v13641: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:33.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:33.204295+0000) 2022-09-10T11:35:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:33.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:33.680451+0000) 2022-09-10T11:35:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:34.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:34.204472+0000) 2022-09-10T11:35:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:34.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:34.680687+0000) 2022-09-10T11:35:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:34 smithi110 ceph-mon[97633]: pgmap v13642: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:34 smithi104 ceph-mon[97044]: pgmap v13642: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:35.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:35.204725+0000) 2022-09-10T11:35:36.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:35.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:35.680859+0000) 2022-09-10T11:35:36.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:36.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:36.204974+0000) 2022-09-10T11:35:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:36.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:36.681069+0000) 2022-09-10T11:35:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:37 smithi104 ceph-mon[97044]: pgmap v13643: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:37 smithi110 ceph-mon[97633]: pgmap v13643: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:37.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:37.205210+0000) 2022-09-10T11:35:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:37.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:37.681305+0000) 2022-09-10T11:35:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:38.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:38.205424+0000) 2022-09-10T11:35:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:38.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:38.681538+0000) 2022-09-10T11:35:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:39 smithi104 ceph-mon[97044]: pgmap v13644: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:39 smithi110 ceph-mon[97633]: pgmap v13644: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:39.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:39.205612+0000) 2022-09-10T11:35:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:39.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:39.681719+0000) 2022-09-10T11:35:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:40.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:40.205795+0000) 2022-09-10T11:35:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:40.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:40.681932+0000) 2022-09-10T11:35:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:41 smithi104 ceph-mon[97044]: pgmap v13645: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:41.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:41 smithi110 ceph-mon[97633]: pgmap v13645: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:41.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:41.206014+0000) 2022-09-10T11:35:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:41.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:41.682157+0000) 2022-09-10T11:35:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:42.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:42.206254+0000) 2022-09-10T11:35:42.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:35:42.397+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:35:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:42.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:42.682289+0000) 2022-09-10T11:35:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:43 smithi104 ceph-mon[97044]: pgmap v13646: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:43 smithi110 ceph-mon[97633]: pgmap v13646: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:43.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:43.206468+0000) 2022-09-10T11:35:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:43.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:43.682503+0000) 2022-09-10T11:35:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:44.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:44.206695+0000) 2022-09-10T11:35:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:44.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:44.682777+0000) 2022-09-10T11:35:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:44 smithi110 ceph-mon[97633]: pgmap v13647: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:44 smithi104 ceph-mon[97044]: pgmap v13647: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:45.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:45.206927+0000) 2022-09-10T11:35:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:45.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:45.682954+0000) 2022-09-10T11:35:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:35:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:35:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:35:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:35:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:35:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:46.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:46.207179+0000) 2022-09-10T11:35:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:46.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:46.683195+0000) 2022-09-10T11:35:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:46 smithi110 ceph-mon[97633]: pgmap v13648: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:46 smithi104 ceph-mon[97044]: pgmap v13648: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:47.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:47.207392+0000) 2022-09-10T11:35:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:47.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:47.683452+0000) 2022-09-10T11:35:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:48.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:48.207586+0000) 2022-09-10T11:35:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:48.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:48.683761+0000) 2022-09-10T11:35:49.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:49 smithi104 ceph-mon[97044]: pgmap v13649: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:49 smithi110 ceph-mon[97633]: pgmap v13649: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:49.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:49.207799+0000) 2022-09-10T11:35:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:49.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:49.683995+0000) 2022-09-10T11:35:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:50.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:50.208018+0000) 2022-09-10T11:35:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:50.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:50.684250+0000) 2022-09-10T11:35:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:51 smithi104 ceph-mon[97044]: pgmap v13650: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:51 smithi110 ceph-mon[97633]: pgmap v13650: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:51.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:51.208247+0000) 2022-09-10T11:35:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:51.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:51.684401+0000) 2022-09-10T11:35:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:52.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:52.208417+0000) 2022-09-10T11:35:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:52.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:52.684578+0000) 2022-09-10T11:35:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:53 smithi104 ceph-mon[97044]: pgmap v13651: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:53 smithi110 ceph-mon[97633]: pgmap v13651: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:53.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:53.208592+0000) 2022-09-10T11:35:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:53.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:53.684841+0000) 2022-09-10T11:35:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:54.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:54.208861+0000) 2022-09-10T11:35:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:54.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:54.685083+0000) 2022-09-10T11:35:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:54 smithi110 ceph-mon[97633]: pgmap v13652: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:54 smithi104 ceph-mon[97044]: pgmap v13652: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:55.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:55.209131+0000) 2022-09-10T11:35:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:55.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:55.685281+0000) 2022-09-10T11:35:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:56.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:56.209278+0000) 2022-09-10T11:35:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:56.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:56.685451+0000) 2022-09-10T11:35:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:57 smithi104 ceph-mon[97044]: pgmap v13653: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:57 smithi110 ceph-mon[97633]: pgmap v13653: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:35:57.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:35:57.413+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:35:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:57.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:57.209392+0000) 2022-09-10T11:35:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:57.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:57.685643+0000) 2022-09-10T11:35:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:58.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:58.209594+0000) 2022-09-10T11:35:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:58.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:58.685967+0000) 2022-09-10T11:35:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:35:59 smithi104 ceph-mon[97044]: pgmap v13654: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:35:59 smithi110 ceph-mon[97633]: pgmap v13654: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:35:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:35:59.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:59.209818+0000) 2022-09-10T11:36:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:35:59.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:35:59.686160+0000) 2022-09-10T11:36:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:00.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:00.210073+0000) 2022-09-10T11:36:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:00.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:00.686296+0000) 2022-09-10T11:36:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:01 smithi104 ceph-mon[97044]: pgmap v13655: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:01 smithi110 ceph-mon[97633]: pgmap v13655: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:01.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:01.210314+0000) 2022-09-10T11:36:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:01.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:01.686497+0000) 2022-09-10T11:36:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:02.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:02.210545+0000) 2022-09-10T11:36:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:02.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:02.686753+0000) 2022-09-10T11:36:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:03 smithi110 ceph-mon[97633]: pgmap v13656: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:03 smithi104 ceph-mon[97044]: pgmap v13656: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:03.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:03.210688+0000) 2022-09-10T11:36:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:03.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:03.686972+0000) 2022-09-10T11:36:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:04.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:04.210938+0000) 2022-09-10T11:36:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:04.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:04.687198+0000) 2022-09-10T11:36:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:04 smithi110 ceph-mon[97633]: pgmap v13657: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:04 smithi104 ceph-mon[97044]: pgmap v13657: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:05.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:05.211159+0000) 2022-09-10T11:36:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:05.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:05.687397+0000) 2022-09-10T11:36:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:06.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:06.211307+0000) 2022-09-10T11:36:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:06.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:06.687627+0000) 2022-09-10T11:36:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:07 smithi104 ceph-mon[97044]: pgmap v13658: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:36:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:07 smithi110 ceph-mon[97633]: pgmap v13658: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:36:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:07.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:07.211475+0000) 2022-09-10T11:36:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:07.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:07.687805+0000) 2022-09-10T11:36:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:08.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:08.211660+0000) 2022-09-10T11:36:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:08.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:08.687974+0000) 2022-09-10T11:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:09 smithi110 ceph-mon[97633]: pgmap v13659: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:36:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:09 smithi110 ceph-mon[97633]: osdmap e934: 12 total, 12 up, 12 in 2022-09-10T11:36:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:09 smithi104 ceph-mon[97044]: pgmap v13659: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:36:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:09 smithi104 ceph-mon[97044]: osdmap e934: 12 total, 12 up, 12 in 2022-09-10T11:36:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:09.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:09.211845+0000) 2022-09-10T11:36:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:09.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:09.688202+0000) 2022-09-10T11:36:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:10.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:10.212007+0000) 2022-09-10T11:36:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:10.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:10.688382+0000) 2022-09-10T11:36:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:11 smithi110 ceph-mon[97633]: pgmap v13661: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:36:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:11 smithi104 ceph-mon[97044]: pgmap v13661: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:36:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:11.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:11.212209+0000) 2022-09-10T11:36:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:11.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:11.688582+0000) 2022-09-10T11:36:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:12.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:12.212373+0000) 2022-09-10T11:36:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:36:12.417+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:36:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:12.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:12.688822+0000) 2022-09-10T11:36:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:13 smithi110 ceph-mon[97633]: pgmap v13662: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:36:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:13 smithi104 ceph-mon[97044]: pgmap v13662: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:36:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:13.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:13.212563+0000) 2022-09-10T11:36:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:13.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:13.689075+0000) 2022-09-10T11:36:14.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:14.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:14.212786+0000) 2022-09-10T11:36:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:14.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:14.689310+0000) 2022-09-10T11:36:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:14 smithi110 ceph-mon[97633]: osdmap e935: 12 total, 12 up, 12 in 2022-09-10T11:36:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:14 smithi110 ceph-mon[97633]: pgmap v13664: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:36:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:14 smithi104 ceph-mon[97044]: osdmap e935: 12 total, 12 up, 12 in 2022-09-10T11:36:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:14 smithi104 ceph-mon[97044]: pgmap v13664: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:36:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:15.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:15.213012+0000) 2022-09-10T11:36:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:15.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:15.689493+0000) 2022-09-10T11:36:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:16.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:16.213278+0000) 2022-09-10T11:36:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:16.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:16.689669+0000) 2022-09-10T11:36:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:17 smithi110 ceph-mon[97633]: pgmap v13665: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:36:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:17 smithi104 ceph-mon[97044]: pgmap v13665: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:36:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:17.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:17.213420+0000) 2022-09-10T11:36:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:17.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:17.689900+0000) 2022-09-10T11:36:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:18.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:18.213619+0000) 2022-09-10T11:36:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:18.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:18.690196+0000) 2022-09-10T11:36:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:19 smithi110 ceph-mon[97633]: pgmap v13666: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:36:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:19 smithi104 ceph-mon[97044]: pgmap v13666: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:36:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:19.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:19.213907+0000) 2022-09-10T11:36:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:19.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:19.690390+0000) 2022-09-10T11:36:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:20.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:20.214124+0000) 2022-09-10T11:36:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:20.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:20.690564+0000) 2022-09-10T11:36:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:21 smithi110 ceph-mon[97633]: pgmap v13667: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:36:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:21 smithi104 ceph-mon[97044]: pgmap v13667: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:36:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:21.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:21.214332+0000) 2022-09-10T11:36:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:21.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:21.690742+0000) 2022-09-10T11:36:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:22.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:22.214581+0000) 2022-09-10T11:36:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:22.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:22.690993+0000) 2022-09-10T11:36:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:23 smithi110 ceph-mon[97633]: pgmap v13668: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:36:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:23 smithi104 ceph-mon[97044]: pgmap v13668: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:36:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:23.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:23.214832+0000) 2022-09-10T11:36:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:23.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:23.691231+0000) 2022-09-10T11:36:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:24.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:24.215069+0000) 2022-09-10T11:36:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:24.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:24.691467+0000) 2022-09-10T11:36:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:24 smithi110 ceph-mon[97633]: pgmap v13669: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:36:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:24 smithi104 ceph-mon[97044]: pgmap v13669: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:36:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:25.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:25.215339+0000) 2022-09-10T11:36:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:25.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:25.691706+0000) 2022-09-10T11:36:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:26.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:26.215529+0000) 2022-09-10T11:36:27.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:26.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:26.691949+0000) 2022-09-10T11:36:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:27 smithi110 ceph-mon[97633]: pgmap v13670: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:36:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:27 smithi104 ceph-mon[97044]: pgmap v13670: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:36:27.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:36:27.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:36:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:27.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:27.215674+0000) 2022-09-10T11:36:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:27.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:27.692178+0000) 2022-09-10T11:36:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:28.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:28.215837+0000) 2022-09-10T11:36:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:28.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:28.692424+0000) 2022-09-10T11:36:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:29 smithi110 ceph-mon[97633]: pgmap v13671: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:29 smithi104 ceph-mon[97044]: pgmap v13671: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:29.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:29.215977+0000) 2022-09-10T11:36:30.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:29.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:29.692614+0000) 2022-09-10T11:36:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:30.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:30.216181+0000) 2022-09-10T11:36:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:30.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:30.692826+0000) 2022-09-10T11:36:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:31 smithi110 ceph-mon[97633]: pgmap v13672: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:31 smithi104 ceph-mon[97044]: pgmap v13672: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:31.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:31.216422+0000) 2022-09-10T11:36:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:31.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:31.693064+0000) 2022-09-10T11:36:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:32.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:32.216619+0000) 2022-09-10T11:36:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:32.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:32.693319+0000) 2022-09-10T11:36:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:33 smithi110 ceph-mon[97633]: pgmap v13673: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:33 smithi104 ceph-mon[97044]: pgmap v13673: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:33.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:33.216749+0000) 2022-09-10T11:36:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:33.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:33.693593+0000) 2022-09-10T11:36:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:34.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:34.216984+0000) 2022-09-10T11:36:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:34.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:34.693841+0000) 2022-09-10T11:36:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:34 smithi110 ceph-mon[97633]: pgmap v13674: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:35.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:34 smithi104 ceph-mon[97044]: pgmap v13674: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:35.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:35.217240+0000) 2022-09-10T11:36:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:35.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:35.694018+0000) 2022-09-10T11:36:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:36.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:36.217413+0000) 2022-09-10T11:36:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:36.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:36.694235+0000) 2022-09-10T11:36:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:37 smithi110 ceph-mon[97633]: pgmap v13675: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:37 smithi104 ceph-mon[97044]: pgmap v13675: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:37.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:37.217626+0000) 2022-09-10T11:36:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:37.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:37.694478+0000) 2022-09-10T11:36:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:38.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:38.217823+0000) 2022-09-10T11:36:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:38.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:38.694719+0000) 2022-09-10T11:36:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:39 smithi110 ceph-mon[97633]: pgmap v13676: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:39 smithi104 ceph-mon[97044]: pgmap v13676: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:39.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:39.218006+0000) 2022-09-10T11:36:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:39.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:39.694915+0000) 2022-09-10T11:36:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:40.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:40.218165+0000) 2022-09-10T11:36:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:40.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:40.695182+0000) 2022-09-10T11:36:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:41 smithi110 ceph-mon[97633]: pgmap v13677: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:41 smithi104 ceph-mon[97044]: pgmap v13677: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:41.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:41.218328+0000) 2022-09-10T11:36:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:41.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:41.695339+0000) 2022-09-10T11:36:42.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:36:42.445+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:36:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:42.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:42.218586+0000) 2022-09-10T11:36:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:42.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:42.695554+0000) 2022-09-10T11:36:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:43 smithi110 ceph-mon[97633]: pgmap v13678: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:43 smithi104 ceph-mon[97044]: pgmap v13678: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:43.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:43.218747+0000) 2022-09-10T11:36:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:43.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:43.695795+0000) 2022-09-10T11:36:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:44.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:44.218912+0000) 2022-09-10T11:36:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:44.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:44.696046+0000) 2022-09-10T11:36:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:44 smithi110 ceph-mon[97633]: pgmap v13679: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:44 smithi104 ceph-mon[97044]: pgmap v13679: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:45.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:45.219151+0000) 2022-09-10T11:36:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:45.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:45.696160+0000) 2022-09-10T11:36:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:36:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:36:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:36:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:36:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:36:46.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:36:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:46.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:46.219379+0000) 2022-09-10T11:36:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:46.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:46.696366+0000) 2022-09-10T11:36:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:46 smithi110 ceph-mon[97633]: pgmap v13680: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:46 smithi104 ceph-mon[97044]: pgmap v13680: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:47.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:47.219631+0000) 2022-09-10T11:36:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:47.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:47.696516+0000) 2022-09-10T11:36:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:48.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:48.219791+0000) 2022-09-10T11:36:49.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:48.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:48.696748+0000) 2022-09-10T11:36:49.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:49 smithi110 ceph-mon[97633]: pgmap v13681: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:49 smithi104 ceph-mon[97044]: pgmap v13681: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:49.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:49.219946+0000) 2022-09-10T11:36:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:49.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:49.696976+0000) 2022-09-10T11:36:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:50.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:50.220207+0000) 2022-09-10T11:36:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:50.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:50.697204+0000) 2022-09-10T11:36:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:51 smithi110 ceph-mon[97633]: pgmap v13682: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:51 smithi104 ceph-mon[97044]: pgmap v13682: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:51.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:51.220416+0000) 2022-09-10T11:36:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:51.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:51.697411+0000) 2022-09-10T11:36:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:52.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:52.220640+0000) 2022-09-10T11:36:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:52.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:52.697590+0000) 2022-09-10T11:36:53.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:53 smithi110 ceph-mon[97633]: pgmap v13683: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:53 smithi104 ceph-mon[97044]: pgmap v13683: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:53.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:53.220845+0000) 2022-09-10T11:36:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:53.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:53.697841+0000) 2022-09-10T11:36:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:54.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:54.221136+0000) 2022-09-10T11:36:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:54.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:54.698009+0000) 2022-09-10T11:36:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:54 smithi110 ceph-mon[97633]: pgmap v13684: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:54 smithi104 ceph-mon[97044]: pgmap v13684: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:55.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:55.221356+0000) 2022-09-10T11:36:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:55.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:55.698228+0000) 2022-09-10T11:36:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:56.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:56.221585+0000) 2022-09-10T11:36:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:56.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:56.698411+0000) 2022-09-10T11:36:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:57 smithi110 ceph-mon[97633]: pgmap v13685: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:57 smithi104 ceph-mon[97044]: pgmap v13685: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:36:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:57.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:57.221825+0000) 2022-09-10T11:36:57.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:36:57.461+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:36:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:57.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:57.698670+0000) 2022-09-10T11:36:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:58.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:58.222039+0000) 2022-09-10T11:36:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:58.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:58.698948+0000) 2022-09-10T11:36:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:36:59 smithi110 ceph-mon[97633]: pgmap v13686: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:36:59 smithi104 ceph-mon[97044]: pgmap v13686: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:36:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:36:59.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:59.222320+0000) 2022-09-10T11:37:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:36:59.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:36:59.699218+0000) 2022-09-10T11:37:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:00.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:00.222481+0000) 2022-09-10T11:37:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:00.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:00.699472+0000) 2022-09-10T11:37:01.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:01 smithi110 ceph-mon[97633]: pgmap v13687: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:01 smithi104 ceph-mon[97044]: pgmap v13687: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:01.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:01.222698+0000) 2022-09-10T11:37:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:01.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:01.699667+0000) 2022-09-10T11:37:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:02.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:02.222909+0000) 2022-09-10T11:37:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:02.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:02.699862+0000) 2022-09-10T11:37:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:03 smithi110 ceph-mon[97633]: pgmap v13688: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:03 smithi104 ceph-mon[97044]: pgmap v13688: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:03.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:03.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:03.223185+0000) 2022-09-10T11:37:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:03.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:03.700090+0000) 2022-09-10T11:37:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:04.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:04.223387+0000) 2022-09-10T11:37:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:04.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:04.700396+0000) 2022-09-10T11:37:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:04 smithi110 ceph-mon[97633]: pgmap v13689: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:04 smithi104 ceph-mon[97044]: pgmap v13689: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:05.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:05.223583+0000) 2022-09-10T11:37:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:05.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:05.700621+0000) 2022-09-10T11:37:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:06.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:06.223811+0000) 2022-09-10T11:37:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:06.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:06.700865+0000) 2022-09-10T11:37:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:07 smithi110 ceph-mon[97633]: pgmap v13690: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:37:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:07 smithi104 ceph-mon[97044]: pgmap v13690: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:37:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:07.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:07.224048+0000) 2022-09-10T11:37:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:07.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:07.701087+0000) 2022-09-10T11:37:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:08.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:08.224356+0000) 2022-09-10T11:37:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:08.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:08.701394+0000) 2022-09-10T11:37:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:09 smithi110 ceph-mon[97633]: pgmap v13691: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:37:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:09 smithi110 ceph-mon[97633]: osdmap e936: 12 total, 12 up, 12 in 2022-09-10T11:37:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:09 smithi104 ceph-mon[97044]: pgmap v13691: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:37:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:09 smithi104 ceph-mon[97044]: osdmap e936: 12 total, 12 up, 12 in 2022-09-10T11:37:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:09.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:09.224533+0000) 2022-09-10T11:37:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:09.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:09.701616+0000) 2022-09-10T11:37:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:10.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:10.224752+0000) 2022-09-10T11:37:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:10.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:10.701845+0000) 2022-09-10T11:37:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:11 smithi110 ceph-mon[97633]: pgmap v13693: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:37:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:11 smithi104 ceph-mon[97044]: pgmap v13693: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:37:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:11.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:11.224990+0000) 2022-09-10T11:37:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:11.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:11.702048+0000) 2022-09-10T11:37:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:12.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:12.225229+0000) 2022-09-10T11:37:12.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:37:12.466+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:37:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:12.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:12.702319+0000) 2022-09-10T11:37:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:13 smithi110 ceph-mon[97633]: pgmap v13694: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:37:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:13 smithi104 ceph-mon[97044]: pgmap v13694: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:37:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:13.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:13.225479+0000) 2022-09-10T11:37:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:13.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:13.702474+0000) 2022-09-10T11:37:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:14.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:14.225648+0000) 2022-09-10T11:37:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:14.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:14.702720+0000) 2022-09-10T11:37:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:14 smithi110 ceph-mon[97633]: osdmap e937: 12 total, 12 up, 12 in 2022-09-10T11:37:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:14 smithi110 ceph-mon[97633]: pgmap v13696: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:37:15.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:14 smithi104 ceph-mon[97044]: osdmap e937: 12 total, 12 up, 12 in 2022-09-10T11:37:15.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:14 smithi104 ceph-mon[97044]: pgmap v13696: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:37:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:15.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:15.225893+0000) 2022-09-10T11:37:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:15.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:15.702917+0000) 2022-09-10T11:37:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:16.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:16.226082+0000) 2022-09-10T11:37:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:16.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:16.703160+0000) 2022-09-10T11:37:17.428 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:17 smithi110 ceph-mon[97633]: pgmap v13697: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:37:17.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:17 smithi104 ceph-mon[97044]: pgmap v13697: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:37:17.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:17.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:17.226238+0000) 2022-09-10T11:37:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:17.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:17.703369+0000) 2022-09-10T11:37:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:18.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:18.226436+0000) 2022-09-10T11:37:19.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:18.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:18.703598+0000) 2022-09-10T11:37:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:19 smithi110 ceph-mon[97633]: pgmap v13698: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:37:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:19 smithi104 ceph-mon[97044]: pgmap v13698: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:37:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:19.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:19.226607+0000) 2022-09-10T11:37:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:19.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:19.703841+0000) 2022-09-10T11:37:20.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:20.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:20.226748+0000) 2022-09-10T11:37:21.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:20.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:20.704130+0000) 2022-09-10T11:37:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:21 smithi110 ceph-mon[97633]: pgmap v13699: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T11:37:21.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:21 smithi104 ceph-mon[97044]: pgmap v13699: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T11:37:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:21.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:21.226952+0000) 2022-09-10T11:37:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:21.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:21.704290+0000) 2022-09-10T11:37:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:22.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:22.227215+0000) 2022-09-10T11:37:23.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:22.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:22.704506+0000) 2022-09-10T11:37:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:23 smithi110 ceph-mon[97633]: pgmap v13700: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T11:37:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:23 smithi104 ceph-mon[97044]: pgmap v13700: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T11:37:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:23.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:23.227405+0000) 2022-09-10T11:37:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:23.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:23.704802+0000) 2022-09-10T11:37:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:24.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:24.227634+0000) 2022-09-10T11:37:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:24.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:24.705003+0000) 2022-09-10T11:37:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:24 smithi110 ceph-mon[97633]: pgmap v13701: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:37:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:24 smithi104 ceph-mon[97044]: pgmap v13701: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:37:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:25.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:25.227845+0000) 2022-09-10T11:37:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:25.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:25.705251+0000) 2022-09-10T11:37:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:26.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:26.228074+0000) 2022-09-10T11:37:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:26.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:26.705496+0000) 2022-09-10T11:37:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:27 smithi110 ceph-mon[97633]: pgmap v13702: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:37:27.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:27.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:27.228307+0000) 2022-09-10T11:37:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:27 smithi104 ceph-mon[97044]: pgmap v13702: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:37:27.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:37:27.478+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:37:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:27.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:27.705735+0000) 2022-09-10T11:37:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:28.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:28.228537+0000) 2022-09-10T11:37:29.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:28.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:28.706015+0000) 2022-09-10T11:37:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:29 smithi110 ceph-mon[97633]: pgmap v13703: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:29.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:29 smithi104 ceph-mon[97044]: pgmap v13703: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:29.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:29.228773+0000) 2022-09-10T11:37:30.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:29.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:29.706233+0000) 2022-09-10T11:37:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:30.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:30.229020+0000) 2022-09-10T11:37:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:30.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:30.706431+0000) 2022-09-10T11:37:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:31 smithi110 ceph-mon[97633]: pgmap v13704: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:31 smithi104 ceph-mon[97044]: pgmap v13704: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:31.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:31.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:31.229277+0000) 2022-09-10T11:37:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:31.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:31.706667+0000) 2022-09-10T11:37:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:32.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:32.229489+0000) 2022-09-10T11:37:33.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:32.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:32.706914+0000) 2022-09-10T11:37:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:33 smithi110 ceph-mon[97633]: pgmap v13705: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:33.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:33 smithi104 ceph-mon[97044]: pgmap v13705: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:33.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:33.229667+0000) 2022-09-10T11:37:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:33.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:33.707176+0000) 2022-09-10T11:37:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:34.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:34.229908+0000) 2022-09-10T11:37:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:34.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:34.707341+0000) 2022-09-10T11:37:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:34 smithi110 ceph-mon[97633]: pgmap v13706: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:34 smithi104 ceph-mon[97044]: pgmap v13706: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:35.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:35.230138+0000) 2022-09-10T11:37:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:35.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:35.707555+0000) 2022-09-10T11:37:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:36.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:36.230340+0000) 2022-09-10T11:37:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:36.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:36.707740+0000) 2022-09-10T11:37:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:37 smithi110 ceph-mon[97633]: pgmap v13707: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:37 smithi104 ceph-mon[97044]: pgmap v13707: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:37.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:37.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:37.230531+0000) 2022-09-10T11:37:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:37.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:37.707990+0000) 2022-09-10T11:37:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:38.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:38.230706+0000) 2022-09-10T11:37:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:38.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:38.708203+0000) 2022-09-10T11:37:39.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:39 smithi110 ceph-mon[97633]: pgmap v13708: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:39 smithi104 ceph-mon[97044]: pgmap v13708: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:39.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:39.230937+0000) 2022-09-10T11:37:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:39.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:39.708372+0000) 2022-09-10T11:37:40.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:40.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:40.231201+0000) 2022-09-10T11:37:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:40.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:40.708622+0000) 2022-09-10T11:37:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:41 smithi110 ceph-mon[97633]: pgmap v13709: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:41 smithi104 ceph-mon[97044]: pgmap v13709: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:41.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:41.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:41.231456+0000) 2022-09-10T11:37:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:41.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:41.708875+0000) 2022-09-10T11:37:42.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:42.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:42.231717+0000) 2022-09-10T11:37:42.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:42.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:42.709176+0000) 2022-09-10T11:37:42.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:37:42.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:37:43.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:43 smithi110 ceph-mon[97633]: pgmap v13710: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:43 smithi104 ceph-mon[97044]: pgmap v13710: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:43.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:43.231946+0000) 2022-09-10T11:37:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:43.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:43.709344+0000) 2022-09-10T11:37:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:44.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:44.232205+0000) 2022-09-10T11:37:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:44.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:44.709544+0000) 2022-09-10T11:37:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:44 smithi110 ceph-mon[97633]: pgmap v13711: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:45.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:44 smithi104 ceph-mon[97044]: pgmap v13711: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:45.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:45.232361+0000) 2022-09-10T11:37:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:45.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:45.709690+0000) 2022-09-10T11:37:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:37:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:37:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:37:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:37:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:37:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:37:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:37:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:46.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:46.232582+0000) 2022-09-10T11:37:47.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:46.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:46.709937+0000) 2022-09-10T11:37:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:46 smithi110 ceph-mon[97633]: pgmap v13712: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:46 smithi104 ceph-mon[97044]: pgmap v13712: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:47.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:47.232813+0000) 2022-09-10T11:37:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:47.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:47.710181+0000) 2022-09-10T11:37:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:48.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:48.232936+0000) 2022-09-10T11:37:49.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:48.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:48.710406+0000) 2022-09-10T11:37:49.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:49 smithi110 ceph-mon[97633]: pgmap v13713: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:49 smithi104 ceph-mon[97044]: pgmap v13713: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:49.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:49.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:49.233083+0000) 2022-09-10T11:37:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:49.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:49.710638+0000) 2022-09-10T11:37:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:50.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:50.233321+0000) 2022-09-10T11:37:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:50.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:50.710842+0000) 2022-09-10T11:37:51.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:51 smithi110 ceph-mon[97633]: pgmap v13714: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:51 smithi104 ceph-mon[97044]: pgmap v13714: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:51.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:51.233545+0000) 2022-09-10T11:37:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:51.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:51.711048+0000) 2022-09-10T11:37:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:52.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:52.233808+0000) 2022-09-10T11:37:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:52.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:52.711232+0000) 2022-09-10T11:37:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:53 smithi110 ceph-mon[97633]: pgmap v13715: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:53 smithi104 ceph-mon[97044]: pgmap v13715: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:53.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:53.233953+0000) 2022-09-10T11:37:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:53.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:53.711457+0000) 2022-09-10T11:37:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:54.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:54.234157+0000) 2022-09-10T11:37:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:54.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:54.711655+0000) 2022-09-10T11:37:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:54 smithi110 ceph-mon[97633]: pgmap v13716: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:55.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:54 smithi104 ceph-mon[97044]: pgmap v13716: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:55.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:55.234419+0000) 2022-09-10T11:37:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:55.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:55.711805+0000) 2022-09-10T11:37:56.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:56.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:56.234700+0000) 2022-09-10T11:37:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:56.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:56.712035+0000) 2022-09-10T11:37:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:57 smithi110 ceph-mon[97633]: pgmap v13717: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:57.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:57.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:57.234890+0000) 2022-09-10T11:37:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:57 smithi104 ceph-mon[97044]: pgmap v13717: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:37:57.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:37:57.488+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:37:57.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:57.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:57.712227+0000) 2022-09-10T11:37:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:58.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:58.235116+0000) 2022-09-10T11:37:59.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:58.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:58.712470+0000) 2022-09-10T11:37:59.418 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:37:59 smithi110 ceph-mon[97633]: pgmap v13718: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:37:59 smithi104 ceph-mon[97044]: pgmap v13718: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:37:59.565 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:37:59.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:59.235400+0000) 2022-09-10T11:38:00.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:37:59.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:37:59.712698+0000) 2022-09-10T11:38:00.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:00.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:00.235604+0000) 2022-09-10T11:38:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:00.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:00.712970+0000) 2022-09-10T11:38:01.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:01 smithi110 ceph-mon[97633]: pgmap v13719: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:01.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:01 smithi104 ceph-mon[97044]: pgmap v13719: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:01.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:01.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:01.235776+0000) 2022-09-10T11:38:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:01.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:01.713206+0000) 2022-09-10T11:38:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:02.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:02.236013+0000) 2022-09-10T11:38:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:02.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:02.713438+0000) 2022-09-10T11:38:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:03 smithi110 ceph-mon[97633]: pgmap v13720: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:03.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:03.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:03.236284+0000) 2022-09-10T11:38:03.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:03 smithi104 ceph-mon[97044]: pgmap v13720: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:04.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:03.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:03.713699+0000) 2022-09-10T11:38:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:04.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:04.236506+0000) 2022-09-10T11:38:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:04.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:04.713945+0000) 2022-09-10T11:38:05.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:04 smithi110 ceph-mon[97633]: pgmap v13721: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:04 smithi104 ceph-mon[97044]: pgmap v13721: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:05.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:05.236692+0000) 2022-09-10T11:38:06.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:05.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:05.714232+0000) 2022-09-10T11:38:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:06.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:06.236919+0000) 2022-09-10T11:38:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:06.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:06.714462+0000) 2022-09-10T11:38:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:07 smithi110 ceph-mon[97633]: pgmap v13722: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:38:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:07 smithi104 ceph-mon[97044]: pgmap v13722: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:38:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:07.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:07.237200+0000) 2022-09-10T11:38:08.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:07.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:07.714600+0000) 2022-09-10T11:38:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:08.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:08.237478+0000) 2022-09-10T11:38:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:08.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:08.714869+0000) 2022-09-10T11:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:09 smithi110 ceph-mon[97633]: pgmap v13723: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:38:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:09 smithi110 ceph-mon[97633]: osdmap e938: 12 total, 12 up, 12 in 2022-09-10T11:38:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:09 smithi104 ceph-mon[97044]: pgmap v13723: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:38:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:09 smithi104 ceph-mon[97044]: osdmap e938: 12 total, 12 up, 12 in 2022-09-10T11:38:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:09.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:09.237758+0000) 2022-09-10T11:38:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:09.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:09.715072+0000) 2022-09-10T11:38:10.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:10.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:10.238026+0000) 2022-09-10T11:38:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:10.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:10.715357+0000) 2022-09-10T11:38:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:11 smithi110 ceph-mon[97633]: pgmap v13725: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:38:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:11 smithi104 ceph-mon[97044]: pgmap v13725: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:38:11.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:11.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:11.238249+0000) 2022-09-10T11:38:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:11.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:11.715554+0000) 2022-09-10T11:38:12.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:12.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:12.238492+0000) 2022-09-10T11:38:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:12.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:12.715719+0000) 2022-09-10T11:38:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:38:12.499+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:38:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:13 smithi110 ceph-mon[97633]: pgmap v13726: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:38:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:13 smithi104 ceph-mon[97044]: pgmap v13726: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:38:13.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:13.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:13.238745+0000) 2022-09-10T11:38:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:13.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:13.715886+0000) 2022-09-10T11:38:14.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:14.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:14.238917+0000) 2022-09-10T11:38:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:14.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:14.716194+0000) 2022-09-10T11:38:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:14 smithi110 ceph-mon[97633]: osdmap e939: 12 total, 12 up, 12 in 2022-09-10T11:38:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:14 smithi110 ceph-mon[97633]: pgmap v13728: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:38:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:14 smithi104 ceph-mon[97044]: osdmap e939: 12 total, 12 up, 12 in 2022-09-10T11:38:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:14 smithi104 ceph-mon[97044]: pgmap v13728: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:38:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:15.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:15.239180+0000) 2022-09-10T11:38:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:15.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:15.716355+0000) 2022-09-10T11:38:16.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:16.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:16.239379+0000) 2022-09-10T11:38:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:16.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:16.716564+0000) 2022-09-10T11:38:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:17 smithi110 ceph-mon[97633]: pgmap v13729: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:38:17.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:17 smithi104 ceph-mon[97044]: pgmap v13729: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:38:17.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:17.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:17.239573+0000) 2022-09-10T11:38:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:17.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:17.716749+0000) 2022-09-10T11:38:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:18.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:18.239854+0000) 2022-09-10T11:38:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:18.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:18.717029+0000) 2022-09-10T11:38:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:19 smithi110 ceph-mon[97633]: pgmap v13730: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:38:19.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:19 smithi104 ceph-mon[97044]: pgmap v13730: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:38:19.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:19.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:19.240159+0000) 2022-09-10T11:38:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:19.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:19.717267+0000) 2022-09-10T11:38:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:20.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:20.240392+0000) 2022-09-10T11:38:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:20.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:20.717525+0000) 2022-09-10T11:38:21.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:21 smithi110 ceph-mon[97633]: pgmap v13731: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:38:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:21 smithi104 ceph-mon[97044]: pgmap v13731: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:38:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:21.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:21.240515+0000) 2022-09-10T11:38:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:21.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:21.717793+0000) 2022-09-10T11:38:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:22.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:22.240755+0000) 2022-09-10T11:38:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:22.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:22.718023+0000) 2022-09-10T11:38:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:23 smithi110 ceph-mon[97633]: pgmap v13732: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:38:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:23 smithi104 ceph-mon[97044]: pgmap v13732: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:38:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:23.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:23.241020+0000) 2022-09-10T11:38:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:23.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:23.718259+0000) 2022-09-10T11:38:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:24.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:24.241225+0000) 2022-09-10T11:38:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:24.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:24.718516+0000) 2022-09-10T11:38:25.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:24 smithi110 ceph-mon[97633]: pgmap v13733: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:38:25.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:24 smithi104 ceph-mon[97044]: pgmap v13733: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:38:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:25.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:25.241389+0000) 2022-09-10T11:38:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:25.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:25.718662+0000) 2022-09-10T11:38:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:26.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:26.241609+0000) 2022-09-10T11:38:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:26.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:26.718884+0000) 2022-09-10T11:38:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:27 smithi110 ceph-mon[97633]: pgmap v13734: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:38:27.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:27.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:27.241844+0000) 2022-09-10T11:38:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:27 smithi104 ceph-mon[97044]: pgmap v13734: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:38:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:27.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:27.719092+0000) 2022-09-10T11:38:27.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:38:27.505+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:38:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:28.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:28.242125+0000) 2022-09-10T11:38:29.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:28.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:28.719288+0000) 2022-09-10T11:38:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:29 smithi110 ceph-mon[97633]: pgmap v13735: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:29 smithi104 ceph-mon[97044]: pgmap v13735: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:29.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:29.242368+0000) 2022-09-10T11:38:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:29.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:29.719496+0000) 2022-09-10T11:38:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:30.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:30.242606+0000) 2022-09-10T11:38:31.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:30.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:30.719697+0000) 2022-09-10T11:38:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:31 smithi110 ceph-mon[97633]: pgmap v13736: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:31 smithi104 ceph-mon[97044]: pgmap v13736: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:31.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:31.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:31.242847+0000) 2022-09-10T11:38:32.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:31.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:31.719857+0000) 2022-09-10T11:38:32.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:32.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:32.243138+0000) 2022-09-10T11:38:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:32.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:32.720114+0000) 2022-09-10T11:38:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:33 smithi110 ceph-mon[97633]: pgmap v13737: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:33 smithi104 ceph-mon[97044]: pgmap v13737: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:33.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:33.243356+0000) 2022-09-10T11:38:34.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:33.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:33.720315+0000) 2022-09-10T11:38:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:34.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:34.243561+0000) 2022-09-10T11:38:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:34.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:34.720556+0000) 2022-09-10T11:38:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:34 smithi110 ceph-mon[97633]: pgmap v13738: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:34 smithi104 ceph-mon[97044]: pgmap v13738: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:35.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:35.243768+0000) 2022-09-10T11:38:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:35.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:35.720770+0000) 2022-09-10T11:38:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:36.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:36.243905+0000) 2022-09-10T11:38:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:36.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:36.721045+0000) 2022-09-10T11:38:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:37 smithi110 ceph-mon[97633]: pgmap v13739: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:37 smithi104 ceph-mon[97044]: pgmap v13739: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:37.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:37.244069+0000) 2022-09-10T11:38:38.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:37.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:37.721329+0000) 2022-09-10T11:38:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:38.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:38.244284+0000) 2022-09-10T11:38:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:38.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:38.721585+0000) 2022-09-10T11:38:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:39 smithi110 ceph-mon[97633]: pgmap v13740: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:39 smithi104 ceph-mon[97044]: pgmap v13740: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:39.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:39.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:39.244508+0000) 2022-09-10T11:38:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:39.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:39.721794+0000) 2022-09-10T11:38:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:40.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:40.244723+0000) 2022-09-10T11:38:41.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:40.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:40.721973+0000) 2022-09-10T11:38:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:41 smithi110 ceph-mon[97633]: pgmap v13741: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:41 smithi104 ceph-mon[97044]: pgmap v13741: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:41.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:41.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:41.244900+0000) 2022-09-10T11:38:42.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:41.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:41.722211+0000) 2022-09-10T11:38:42.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:42.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:42.245136+0000) 2022-09-10T11:38:42.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:38:42.520+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:38:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:42.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:42.722402+0000) 2022-09-10T11:38:43.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:43 smithi110 ceph-mon[97633]: pgmap v13742: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:43 smithi104 ceph-mon[97044]: pgmap v13742: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:43.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:43.245401+0000) 2022-09-10T11:38:44.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:43.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:43.722596+0000) 2022-09-10T11:38:44.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:44.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:44.245606+0000) 2022-09-10T11:38:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:44.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:44.722760+0000) 2022-09-10T11:38:45.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:44 smithi110 ceph-mon[97633]: pgmap v13743: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:44 smithi104 ceph-mon[97044]: pgmap v13743: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:45.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:45.245833+0000) 2022-09-10T11:38:46.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:45.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:45.722901+0000) 2022-09-10T11:38:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:38:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:38:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:38:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:38:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:38:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:38:46.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:38:46.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:38:46.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:46.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:46.246060+0000) 2022-09-10T11:38:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:46.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:46.723181+0000) 2022-09-10T11:38:47.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:46 smithi110 ceph-mon[97633]: pgmap v13744: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:46 smithi104 ceph-mon[97044]: pgmap v13744: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:47.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:47.246301+0000) 2022-09-10T11:38:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:47.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:47.723298+0000) 2022-09-10T11:38:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:48.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:48.246507+0000) 2022-09-10T11:38:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:48.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:48.723515+0000) 2022-09-10T11:38:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:49 smithi110 ceph-mon[97633]: pgmap v13745: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:49 smithi104 ceph-mon[97044]: pgmap v13745: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:49.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:49.246668+0000) 2022-09-10T11:38:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:49.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:49.723716+0000) 2022-09-10T11:38:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:50.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:50.246915+0000) 2022-09-10T11:38:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:50.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:50.723998+0000) 2022-09-10T11:38:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:51 smithi110 ceph-mon[97633]: pgmap v13746: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:51 smithi104 ceph-mon[97044]: pgmap v13746: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:51.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:51.247139+0000) 2022-09-10T11:38:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:51.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:51.724156+0000) 2022-09-10T11:38:52.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:52.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:52.247399+0000) 2022-09-10T11:38:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:52.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:52.724326+0000) 2022-09-10T11:38:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:53 smithi110 ceph-mon[97633]: pgmap v13747: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:53 smithi104 ceph-mon[97044]: pgmap v13747: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:53.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:53.247552+0000) 2022-09-10T11:38:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:53.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:53.724476+0000) 2022-09-10T11:38:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:54.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:54.247816+0000) 2022-09-10T11:38:55.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:54.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:54.724662+0000) 2022-09-10T11:38:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:54 smithi110 ceph-mon[97633]: pgmap v13748: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:55.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:54 smithi104 ceph-mon[97044]: pgmap v13748: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:55.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:55.248080+0000) 2022-09-10T11:38:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:55.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:55.724835+0000) 2022-09-10T11:38:56.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:56.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:56.248335+0000) 2022-09-10T11:38:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:56.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:56.725084+0000) 2022-09-10T11:38:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:57 smithi110 ceph-mon[97633]: pgmap v13749: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:57.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:57.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:57.248588+0000) 2022-09-10T11:38:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:57 smithi104 ceph-mon[97044]: pgmap v13749: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:38:57.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:38:57.528+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:38:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:57.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:57.725334+0000) 2022-09-10T11:38:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:58.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:58.248820+0000) 2022-09-10T11:38:59.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:58.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:58.725617+0000) 2022-09-10T11:38:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:38:59 smithi110 ceph-mon[97633]: pgmap v13750: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:38:59 smithi104 ceph-mon[97044]: pgmap v13750: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:38:59.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:38:59.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:59.249046+0000) 2022-09-10T11:39:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:38:59.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:38:59.725865+0000) 2022-09-10T11:39:00.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:00.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:00.249160+0000) 2022-09-10T11:39:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:00.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:00.726134+0000) 2022-09-10T11:39:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:01 smithi110 ceph-mon[97633]: pgmap v13751: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:01 smithi104 ceph-mon[97044]: pgmap v13751: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:01.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:01.249319+0000) 2022-09-10T11:39:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:01.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:01.726341+0000) 2022-09-10T11:39:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:02.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:02.249561+0000) 2022-09-10T11:39:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:02.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:02.726589+0000) 2022-09-10T11:39:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:03 smithi110 ceph-mon[97633]: pgmap v13752: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:03 smithi104 ceph-mon[97044]: pgmap v13752: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:03.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:03.249828+0000) 2022-09-10T11:39:04.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:03.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:03.726827+0000) 2022-09-10T11:39:04.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:04.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:04.250070+0000) 2022-09-10T11:39:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:04.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:04.727026+0000) 2022-09-10T11:39:05.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:04 smithi110 ceph-mon[97633]: pgmap v13753: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:04 smithi104 ceph-mon[97044]: pgmap v13753: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:05.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:05.250303+0000) 2022-09-10T11:39:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:05.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:05.727232+0000) 2022-09-10T11:39:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:06.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:06.250558+0000) 2022-09-10T11:39:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:06.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:06.727414+0000) 2022-09-10T11:39:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:07 smithi110 ceph-mon[97633]: pgmap v13754: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:39:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:07 smithi104 ceph-mon[97044]: pgmap v13754: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:39:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:07.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:07.250783+0000) 2022-09-10T11:39:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:07.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:07.727682+0000) 2022-09-10T11:39:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:08.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:08.250980+0000) 2022-09-10T11:39:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:08.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:08.727931+0000) 2022-09-10T11:39:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:09 smithi110 ceph-mon[97633]: pgmap v13755: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:39:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:09 smithi110 ceph-mon[97633]: osdmap e940: 12 total, 12 up, 12 in 2022-09-10T11:39:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:09 smithi104 ceph-mon[97044]: pgmap v13755: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:39:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:09 smithi104 ceph-mon[97044]: osdmap e940: 12 total, 12 up, 12 in 2022-09-10T11:39:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:09.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:09.251144+0000) 2022-09-10T11:39:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:09.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:09.728194+0000) 2022-09-10T11:39:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:10.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:10.251406+0000) 2022-09-10T11:39:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:10.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:10.728407+0000) 2022-09-10T11:39:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:11 smithi110 ceph-mon[97633]: pgmap v13757: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:39:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:11 smithi104 ceph-mon[97044]: pgmap v13757: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:39:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:11.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:11.251660+0000) 2022-09-10T11:39:12.065 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:11.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:11.728585+0000) 2022-09-10T11:39:12.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:12.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:12.251922+0000) 2022-09-10T11:39:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:12.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:12.728825+0000) 2022-09-10T11:39:12.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:39:12.533+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:39:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:13 smithi110 ceph-mon[97633]: pgmap v13758: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:39:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:13 smithi104 ceph-mon[97044]: pgmap v13758: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:39:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:13.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:13.252074+0000) 2022-09-10T11:39:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:13.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:13.729057+0000) 2022-09-10T11:39:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:14.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:14.252307+0000) 2022-09-10T11:39:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:14.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:14.729300+0000) 2022-09-10T11:39:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:14 smithi110 ceph-mon[97633]: osdmap e941: 12 total, 12 up, 12 in 2022-09-10T11:39:15.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:14 smithi110 ceph-mon[97633]: pgmap v13760: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:39:15.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:14 smithi104 ceph-mon[97044]: osdmap e941: 12 total, 12 up, 12 in 2022-09-10T11:39:15.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:14 smithi104 ceph-mon[97044]: pgmap v13760: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:39:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:15.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:15.252537+0000) 2022-09-10T11:39:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:15.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:15.729545+0000) 2022-09-10T11:39:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:16.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:16.252732+0000) 2022-09-10T11:39:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:16.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:16.729847+0000) 2022-09-10T11:39:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:17 smithi110 ceph-mon[97633]: pgmap v13761: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:39:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:17 smithi104 ceph-mon[97044]: pgmap v13761: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:39:17.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:17.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:17.252948+0000) 2022-09-10T11:39:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:17.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:17.730130+0000) 2022-09-10T11:39:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:18.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:18.253224+0000) 2022-09-10T11:39:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:18.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:18.730385+0000) 2022-09-10T11:39:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:19 smithi110 ceph-mon[97633]: pgmap v13762: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:39:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:19 smithi104 ceph-mon[97044]: pgmap v13762: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:39:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:19.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:19.253399+0000) 2022-09-10T11:39:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:19.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:19.730593+0000) 2022-09-10T11:39:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:20.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:20.253576+0000) 2022-09-10T11:39:21.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:20.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:20.730818+0000) 2022-09-10T11:39:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:21 smithi110 ceph-mon[97633]: pgmap v13763: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:39:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:21 smithi104 ceph-mon[97044]: pgmap v13763: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:39:21.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:21.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:21.253760+0000) 2022-09-10T11:39:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:21.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:21.731090+0000) 2022-09-10T11:39:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:22.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:22.253898+0000) 2022-09-10T11:39:23.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:22.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:22.731409+0000) 2022-09-10T11:39:23.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:23 smithi110 ceph-mon[97633]: pgmap v13764: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:39:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:23 smithi104 ceph-mon[97044]: pgmap v13764: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:39:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:23.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:23.254142+0000) 2022-09-10T11:39:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:23.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:23.731588+0000) 2022-09-10T11:39:24.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:24.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:24.254369+0000) 2022-09-10T11:39:25.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:24.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:24.731798+0000) 2022-09-10T11:39:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:24 smithi110 ceph-mon[97633]: pgmap v13765: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:39:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:24 smithi104 ceph-mon[97044]: pgmap v13765: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:39:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:25.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:25.254616+0000) 2022-09-10T11:39:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:25.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:25.732026+0000) 2022-09-10T11:39:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:26.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:26.254914+0000) 2022-09-10T11:39:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:26.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:26.732244+0000) 2022-09-10T11:39:27.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:27 smithi110 ceph-mon[97633]: pgmap v13766: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:39:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:27 smithi104 ceph-mon[97044]: pgmap v13766: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:39:27.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:27.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:27.255140+0000) 2022-09-10T11:39:27.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:27.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:27.732462+0000) 2022-09-10T11:39:27.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:39:27.549+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:39:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:28.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:28.255329+0000) 2022-09-10T11:39:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:28.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:28.732644+0000) 2022-09-10T11:39:29.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:29 smithi110 ceph-mon[97633]: pgmap v13767: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:29 smithi104 ceph-mon[97044]: pgmap v13767: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:29.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:29.255538+0000) 2022-09-10T11:39:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:29.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:29.732863+0000) 2022-09-10T11:39:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:30.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:30.255764+0000) 2022-09-10T11:39:31.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:30.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:30.733141+0000) 2022-09-10T11:39:31.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:31 smithi110 ceph-mon[97633]: pgmap v13768: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:31 smithi104 ceph-mon[97044]: pgmap v13768: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:31.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:31.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:31.255937+0000) 2022-09-10T11:39:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:31.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:31.733372+0000) 2022-09-10T11:39:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:32.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:32.256205+0000) 2022-09-10T11:39:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:32.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:32.733647+0000) 2022-09-10T11:39:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:33 smithi110 ceph-mon[97633]: pgmap v13769: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:33 smithi104 ceph-mon[97044]: pgmap v13769: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:33.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:33.256412+0000) 2022-09-10T11:39:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:33.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:33.733817+0000) 2022-09-10T11:39:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:34.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:34.256658+0000) 2022-09-10T11:39:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:34.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:34.734012+0000) 2022-09-10T11:39:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:34 smithi110 ceph-mon[97633]: pgmap v13770: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:34 smithi104 ceph-mon[97044]: pgmap v13770: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:35.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:35.256883+0000) 2022-09-10T11:39:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:35.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:35.734228+0000) 2022-09-10T11:39:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:36.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:36.257153+0000) 2022-09-10T11:39:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:36.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:36.734488+0000) 2022-09-10T11:39:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:37 smithi110 ceph-mon[97633]: pgmap v13771: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:37 smithi104 ceph-mon[97044]: pgmap v13771: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:37.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:37.257345+0000) 2022-09-10T11:39:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:37.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:37.734669+0000) 2022-09-10T11:39:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:38.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:38.257486+0000) 2022-09-10T11:39:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:38.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:38.734897+0000) 2022-09-10T11:39:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:39 smithi110 ceph-mon[97633]: pgmap v13772: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:39 smithi104 ceph-mon[97044]: pgmap v13772: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:39.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:39.257664+0000) 2022-09-10T11:39:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:39.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:39.735175+0000) 2022-09-10T11:39:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:40.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:40.257896+0000) 2022-09-10T11:39:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:40.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:40.735365+0000) 2022-09-10T11:39:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:41 smithi110 ceph-mon[97633]: pgmap v13773: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:41 smithi104 ceph-mon[97044]: pgmap v13773: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:41.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:41.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:41.258153+0000) 2022-09-10T11:39:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:41.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:41.735579+0000) 2022-09-10T11:39:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:42.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:42.258312+0000) 2022-09-10T11:39:43.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:39:42.564+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:39:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:42.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:42.735875+0000) 2022-09-10T11:39:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:43 smithi110 ceph-mon[97633]: pgmap v13774: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:43 smithi104 ceph-mon[97044]: pgmap v13774: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:43.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:43.258446+0000) 2022-09-10T11:39:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:43.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:43.736124+0000) 2022-09-10T11:39:44.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:44.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:44.258617+0000) 2022-09-10T11:39:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:44.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:44.736375+0000) 2022-09-10T11:39:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:44 smithi110 ceph-mon[97633]: pgmap v13775: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:44 smithi104 ceph-mon[97044]: pgmap v13775: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:45.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:45.258819+0000) 2022-09-10T11:39:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:45.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:45.736607+0000) 2022-09-10T11:39:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:39:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:39:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:39:46.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:39:46.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:39:46.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:39:46.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:39:46.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:39:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:46.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:46.259038+0000) 2022-09-10T11:39:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:46.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:46.736803+0000) 2022-09-10T11:39:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:46 smithi110 ceph-mon[97633]: pgmap v13776: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:46 smithi104 ceph-mon[97044]: pgmap v13776: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:47.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:47.259263+0000) 2022-09-10T11:39:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:47.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:47.737035+0000) 2022-09-10T11:39:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:48.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:48.259515+0000) 2022-09-10T11:39:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:48.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:48.737273+0000) 2022-09-10T11:39:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:49 smithi110 ceph-mon[97633]: pgmap v13777: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:49 smithi104 ceph-mon[97044]: pgmap v13777: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:39:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:49.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:49.259699+0000) 2022-09-10T11:39:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:49.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:49.737556+0000) 2022-09-10T11:39:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:50.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:50.259914+0000) 2022-09-10T11:39:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:50.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:50.737756+0000) 2022-09-10T11:39:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:51 smithi110 ceph-mon[97633]: pgmap v13778: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:51 smithi104 ceph-mon[97044]: pgmap v13778: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:51.655 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:51.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:51.260185+0000) 2022-09-10T11:39:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:51.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:51.737970+0000) 2022-09-10T11:39:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:52.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:52.260380+0000) 2022-09-10T11:39:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:52.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:52.738249+0000) 2022-09-10T11:39:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:53 smithi110 ceph-mon[97633]: pgmap v13779: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:39:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:53 smithi104 ceph-mon[97044]: pgmap v13779: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:39:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:53.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:53.260614+0000) 2022-09-10T11:39:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:53.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:53.738492+0000) 2022-09-10T11:39:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:54.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:54.260949+0000) 2022-09-10T11:39:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:54.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:54.738635+0000) 2022-09-10T11:39:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:54 smithi110 ceph-mon[97633]: pgmap v13780: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:39:55.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:54 smithi104 ceph-mon[97044]: pgmap v13780: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:39:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:55.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:55.261221+0000) 2022-09-10T11:39:56.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:55.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:55.738853+0000) 2022-09-10T11:39:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:56.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:56.261448+0000) 2022-09-10T11:39:57.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:56.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:56.739133+0000) 2022-09-10T11:39:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:57 smithi110 ceph-mon[97633]: pgmap v13781: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:57 smithi104 ceph-mon[97044]: pgmap v13781: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:39:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:57.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:57.261684+0000) 2022-09-10T11:39:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:57.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:57.739343+0000) 2022-09-10T11:39:58.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:39:57.566+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:39:58.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:58.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:58.261911+0000) 2022-09-10T11:39:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:58.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:58.739500+0000) 2022-09-10T11:39:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:39:59 smithi110 ceph-mon[97633]: pgmap v13782: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:39:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:39:59 smithi104 ceph-mon[97044]: pgmap v13782: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:39:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:39:59.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:59.262124+0000) 2022-09-10T11:40:00.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:39:59.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:39:59.739686+0000) 2022-09-10T11:40:00.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:40:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:40:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:00.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:00.262251+0000) 2022-09-10T11:40:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:00.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:00.739871+0000) 2022-09-10T11:40:01.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:01 smithi110 ceph-mon[97633]: pgmap v13783: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:01.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:01 smithi104 ceph-mon[97044]: pgmap v13783: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:01.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:01.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:01.262459+0000) 2022-09-10T11:40:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:01.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:01.740132+0000) 2022-09-10T11:40:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:02.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:02.262707+0000) 2022-09-10T11:40:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:02.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:02.740385+0000) 2022-09-10T11:40:03.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:03 smithi110 ceph-mon[97633]: pgmap v13784: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:03.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:03 smithi104 ceph-mon[97044]: pgmap v13784: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:03.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:03.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:03.262938+0000) 2022-09-10T11:40:04.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:03.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:03.740601+0000) 2022-09-10T11:40:04.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:04.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:04.263239+0000) 2022-09-10T11:40:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:04.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:04.740868+0000) 2022-09-10T11:40:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:04 smithi110 ceph-mon[97633]: pgmap v13785: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:04 smithi104 ceph-mon[97044]: pgmap v13785: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:05.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:05.263421+0000) 2022-09-10T11:40:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:05.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:05.741080+0000) 2022-09-10T11:40:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:06.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:06.263652+0000) 2022-09-10T11:40:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:06.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:06.741274+0000) 2022-09-10T11:40:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:07 smithi110 ceph-mon[97633]: pgmap v13786: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:40:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:07 smithi104 ceph-mon[97044]: pgmap v13786: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:40:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:07.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:07.263837+0000) 2022-09-10T11:40:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:07.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:07.741485+0000) 2022-09-10T11:40:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:08.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:08.264013+0000) 2022-09-10T11:40:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:08.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:08.741723+0000) 2022-09-10T11:40:09.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:09 smithi110 ceph-mon[97633]: pgmap v13787: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:40:09.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:09 smithi110 ceph-mon[97633]: osdmap e942: 12 total, 12 up, 12 in 2022-09-10T11:40:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:09 smithi104 ceph-mon[97044]: pgmap v13787: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:40:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:09 smithi104 ceph-mon[97044]: osdmap e942: 12 total, 12 up, 12 in 2022-09-10T11:40:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:09.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:09.264189+0000) 2022-09-10T11:40:10.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:09.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:09.741915+0000) 2022-09-10T11:40:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:10.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:10.264414+0000) 2022-09-10T11:40:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:10.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:10.742146+0000) 2022-09-10T11:40:11.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:11 smithi110 ceph-mon[97633]: pgmap v13789: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:40:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:11 smithi104 ceph-mon[97044]: pgmap v13789: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:40:11.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:11.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:11.264591+0000) 2022-09-10T11:40:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:11.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:11.742345+0000) 2022-09-10T11:40:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:12.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:12.264803+0000) 2022-09-10T11:40:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:12.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:12.742545+0000) 2022-09-10T11:40:13.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:40:12.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:40:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:13 smithi110 ceph-mon[97633]: pgmap v13790: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:40:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:13 smithi104 ceph-mon[97044]: pgmap v13790: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:40:13.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:13.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:13.265010+0000) 2022-09-10T11:40:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:13.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:13.742754+0000) 2022-09-10T11:40:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:14.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:14.265301+0000) 2022-09-10T11:40:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:14.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:14.743028+0000) 2022-09-10T11:40:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:14 smithi110 ceph-mon[97633]: osdmap e943: 12 total, 12 up, 12 in 2022-09-10T11:40:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:14 smithi110 ceph-mon[97633]: pgmap v13792: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:40:15.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:14 smithi104 ceph-mon[97044]: osdmap e943: 12 total, 12 up, 12 in 2022-09-10T11:40:15.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:14 smithi104 ceph-mon[97044]: pgmap v13792: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:40:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:15.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:15.265550+0000) 2022-09-10T11:40:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:15.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:15.743220+0000) 2022-09-10T11:40:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:16.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:16.265777+0000) 2022-09-10T11:40:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:16.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:16.743380+0000) 2022-09-10T11:40:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:17 smithi110 ceph-mon[97633]: pgmap v13793: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:40:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:17 smithi104 ceph-mon[97044]: pgmap v13793: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:40:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:17.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:17.266004+0000) 2022-09-10T11:40:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:17.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:17.743574+0000) 2022-09-10T11:40:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:18.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:18.266203+0000) 2022-09-10T11:40:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:18.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:18.743798+0000) 2022-09-10T11:40:19.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:19 smithi110 ceph-mon[97633]: pgmap v13794: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:40:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:19 smithi104 ceph-mon[97044]: pgmap v13794: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:40:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:19.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:19.266409+0000) 2022-09-10T11:40:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:19.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:19.744032+0000) 2022-09-10T11:40:20.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:20.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:20.266612+0000) 2022-09-10T11:40:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:20.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:20.744243+0000) 2022-09-10T11:40:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:21 smithi110 ceph-mon[97633]: pgmap v13795: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:40:21.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:21 smithi104 ceph-mon[97044]: pgmap v13795: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:40:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:21.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:21.266829+0000) 2022-09-10T11:40:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:21.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:21.744508+0000) 2022-09-10T11:40:22.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:22.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:22.267019+0000) 2022-09-10T11:40:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:22.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:22.744798+0000) 2022-09-10T11:40:23.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:23 smithi110 ceph-mon[97633]: pgmap v13796: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:40:23.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:23 smithi104 ceph-mon[97044]: pgmap v13796: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:40:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:23.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:23.267193+0000) 2022-09-10T11:40:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:23.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:23.744981+0000) 2022-09-10T11:40:24.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:24.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:24.267439+0000) 2022-09-10T11:40:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:24.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:24.745210+0000) 2022-09-10T11:40:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:24 smithi110 ceph-mon[97633]: pgmap v13797: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:40:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:24 smithi104 ceph-mon[97044]: pgmap v13797: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:40:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:25.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:25.267690+0000) 2022-09-10T11:40:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:25.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:25.745432+0000) 2022-09-10T11:40:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:26.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:26.267889+0000) 2022-09-10T11:40:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:26.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:26.745619+0000) 2022-09-10T11:40:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:27 smithi110 ceph-mon[97633]: pgmap v13798: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:40:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:27 smithi104 ceph-mon[97044]: pgmap v13798: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:40:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:27.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:27.268161+0000) 2022-09-10T11:40:28.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:27.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:27.745777+0000) 2022-09-10T11:40:28.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:40:27.584+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:40:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:28.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:28.268373+0000) 2022-09-10T11:40:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:28.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:28.746037+0000) 2022-09-10T11:40:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:29 smithi110 ceph-mon[97633]: pgmap v13799: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:29.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:29 smithi104 ceph-mon[97044]: pgmap v13799: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:29.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:29.268595+0000) 2022-09-10T11:40:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:29.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:29.746254+0000) 2022-09-10T11:40:30.585 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:30.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:30.268766+0000) 2022-09-10T11:40:31.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:30.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:30.746459+0000) 2022-09-10T11:40:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:31 smithi110 ceph-mon[97633]: pgmap v13800: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:31 smithi104 ceph-mon[97044]: pgmap v13800: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:31.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:31.268993+0000) 2022-09-10T11:40:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:31.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:31.746686+0000) 2022-09-10T11:40:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:32.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:32.269176+0000) 2022-09-10T11:40:33.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:32.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:32.746890+0000) 2022-09-10T11:40:33.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:33 smithi110 ceph-mon[97633]: pgmap v13801: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:33.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:33 smithi104 ceph-mon[97044]: pgmap v13801: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:33.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:33.269364+0000) 2022-09-10T11:40:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:33.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:33.747157+0000) 2022-09-10T11:40:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:34.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:34.269601+0000) 2022-09-10T11:40:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:34.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:34.747387+0000) 2022-09-10T11:40:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:34 smithi110 ceph-mon[97633]: pgmap v13802: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:35.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:34 smithi104 ceph-mon[97044]: pgmap v13802: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:35.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:35.269826+0000) 2022-09-10T11:40:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:35.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:35.747637+0000) 2022-09-10T11:40:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:36.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:36.270057+0000) 2022-09-10T11:40:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:36.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:36.747886+0000) 2022-09-10T11:40:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:37 smithi110 ceph-mon[97633]: pgmap v13803: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:37.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:37 smithi104 ceph-mon[97044]: pgmap v13803: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:37.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:37.270219+0000) 2022-09-10T11:40:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:37.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:37.748204+0000) 2022-09-10T11:40:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:38.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:38.270467+0000) 2022-09-10T11:40:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:38.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:38.748363+0000) 2022-09-10T11:40:39.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:39 smithi110 ceph-mon[97633]: pgmap v13804: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:39 smithi104 ceph-mon[97044]: pgmap v13804: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:39.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:39.270633+0000) 2022-09-10T11:40:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:39.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:39.748481+0000) 2022-09-10T11:40:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:40.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:40.270823+0000) 2022-09-10T11:40:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:40.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:40.748659+0000) 2022-09-10T11:40:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:41 smithi110 ceph-mon[97633]: pgmap v13805: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:41.576 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:41.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:41.271024+0000) 2022-09-10T11:40:41.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:41 smithi104 ceph-mon[97044]: pgmap v13805: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:42.065 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:41.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:41.748875+0000) 2022-09-10T11:40:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:42.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:42.271214+0000) 2022-09-10T11:40:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:40:42.600+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:40:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:42.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:42.749185+0000) 2022-09-10T11:40:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:43 smithi110 ceph-mon[97633]: pgmap v13806: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:43.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:43 smithi104 ceph-mon[97044]: pgmap v13806: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:43.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:43.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:43.271354+0000) 2022-09-10T11:40:44.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:43.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:43.749440+0000) 2022-09-10T11:40:44.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:44.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:44.271574+0000) 2022-09-10T11:40:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:44.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:44.749676+0000) 2022-09-10T11:40:45.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:44 smithi104 ceph-mon[97044]: pgmap v13807: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:45.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:44 smithi110 ceph-mon[97633]: pgmap v13807: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:45.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:45.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:45.271800+0000) 2022-09-10T11:40:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:45.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:45.749871+0000) 2022-09-10T11:40:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:40:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:40:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:40:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:40:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:40:46.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:46.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:46.272067+0000) 2022-09-10T11:40:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:46.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:46.750055+0000) 2022-09-10T11:40:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:46 smithi104 ceph-mon[97044]: pgmap v13808: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:46 smithi110 ceph-mon[97633]: pgmap v13808: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:47.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:47.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:47.272328+0000) 2022-09-10T11:40:48.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:47.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:47.750338+0000) 2022-09-10T11:40:48.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:48.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:48.272566+0000) 2022-09-10T11:40:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:48.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:48.750474+0000) 2022-09-10T11:40:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:49 smithi110 ceph-mon[97633]: pgmap v13809: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:49 smithi104 ceph-mon[97044]: pgmap v13809: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:49.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:49.272768+0000) 2022-09-10T11:40:50.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:49.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:49.750628+0000) 2022-09-10T11:40:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:50.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:50.272981+0000) 2022-09-10T11:40:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:50.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:50.750850+0000) 2022-09-10T11:40:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:51 smithi110 ceph-mon[97633]: pgmap v13810: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:51.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:51.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:51.273138+0000) 2022-09-10T11:40:51.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:51 smithi104 ceph-mon[97044]: pgmap v13810: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:51.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:51.751111+0000) 2022-09-10T11:40:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:52.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:52.273390+0000) 2022-09-10T11:40:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:52.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:52.751365+0000) 2022-09-10T11:40:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:53 smithi110 ceph-mon[97633]: pgmap v13811: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:53 smithi104 ceph-mon[97044]: pgmap v13811: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:53.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:53.273593+0000) 2022-09-10T11:40:54.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:53.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:53.751601+0000) 2022-09-10T11:40:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:54.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:54.273834+0000) 2022-09-10T11:40:55.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:54.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:54.751891+0000) 2022-09-10T11:40:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:54 smithi104 ceph-mon[97044]: pgmap v13812: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:55.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:54 smithi110 ceph-mon[97633]: pgmap v13812: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:55.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:55.274134+0000) 2022-09-10T11:40:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:55.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:55.752174+0000) 2022-09-10T11:40:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:56.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:56.274341+0000) 2022-09-10T11:40:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:56.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:56.752433+0000) 2022-09-10T11:40:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:57 smithi110 ceph-mon[97633]: pgmap v13813: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:57.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:57.274527+0000) 2022-09-10T11:40:57.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:57 smithi104 ceph-mon[97044]: pgmap v13813: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:40:58.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:57.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:57.752708+0000) 2022-09-10T11:40:58.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:40:57.610+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:40:58.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:58.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:58.274728+0000) 2022-09-10T11:40:59.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:58.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:58.752972+0000) 2022-09-10T11:40:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:40:59 smithi110 ceph-mon[97633]: pgmap v13814: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:40:59 smithi104 ceph-mon[97044]: pgmap v13814: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:40:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:40:59.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:59.274914+0000) 2022-09-10T11:41:00.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:40:59.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:40:59.753201+0000) 2022-09-10T11:41:00.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:00.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:00.275069+0000) 2022-09-10T11:41:01.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:00.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:00.753354+0000) 2022-09-10T11:41:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:01 smithi110 ceph-mon[97633]: pgmap v13815: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:01.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:01.275219+0000) 2022-09-10T11:41:01.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:01 smithi104 ceph-mon[97044]: pgmap v13815: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:01.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:01.753577+0000) 2022-09-10T11:41:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:02.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:02.275476+0000) 2022-09-10T11:41:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:02.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:02.753768+0000) 2022-09-10T11:41:03.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:03 smithi110 ceph-mon[97633]: pgmap v13816: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:03.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:03 smithi104 ceph-mon[97044]: pgmap v13816: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:03.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:03.275749+0000) 2022-09-10T11:41:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:03.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:03.754018+0000) 2022-09-10T11:41:04.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:04.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:04.276017+0000) 2022-09-10T11:41:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:04.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:04.754350+0000) 2022-09-10T11:41:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:04 smithi104 ceph-mon[97044]: pgmap v13817: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:05.432 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:04 smithi110 ceph-mon[97633]: pgmap v13817: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:05.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:05.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:05.276169+0000) 2022-09-10T11:41:06.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:05.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:05.754596+0000) 2022-09-10T11:41:06.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:06.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:06.276393+0000) 2022-09-10T11:41:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:06.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:06.754784+0000) 2022-09-10T11:41:07.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:07 smithi110 ceph-mon[97633]: pgmap v13818: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:41:07.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:07 smithi104 ceph-mon[97044]: pgmap v13818: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:41:07.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:07.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:07.276654+0000) 2022-09-10T11:41:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:07.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:07.755035+0000) 2022-09-10T11:41:08.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:08.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:08.276913+0000) 2022-09-10T11:41:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:08.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:08.755358+0000) 2022-09-10T11:41:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:09 smithi110 ceph-mon[97633]: pgmap v13819: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:41:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:09 smithi110 ceph-mon[97633]: osdmap e944: 12 total, 12 up, 12 in 2022-09-10T11:41:09.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:09 smithi104 ceph-mon[97044]: pgmap v13819: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:41:09.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:09 smithi104 ceph-mon[97044]: osdmap e944: 12 total, 12 up, 12 in 2022-09-10T11:41:09.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:09.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:09.277177+0000) 2022-09-10T11:41:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:09.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:09.755608+0000) 2022-09-10T11:41:10.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:10.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:10.277404+0000) 2022-09-10T11:41:11.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:10.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:10.755821+0000) 2022-09-10T11:41:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:11 smithi110 ceph-mon[97633]: pgmap v13821: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:41:11.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:11 smithi104 ceph-mon[97044]: pgmap v13821: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:41:11.568 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:11.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:11.277613+0000) 2022-09-10T11:41:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:11.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:11.756045+0000) 2022-09-10T11:41:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:12.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:12.277842+0000) 2022-09-10T11:41:13.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:12.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:12.756307+0000) 2022-09-10T11:41:13.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:41:12.622+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:41:13.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:13 smithi110 ceph-mon[97633]: pgmap v13822: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:41:13.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:13 smithi104 ceph-mon[97044]: pgmap v13822: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:41:13.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:13.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:13.278133+0000) 2022-09-10T11:41:14.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:13.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:13.756524+0000) 2022-09-10T11:41:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:14.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:14.278380+0000) 2022-09-10T11:41:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:14.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:14.756782+0000) 2022-09-10T11:41:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:14 smithi104 ceph-mon[97044]: osdmap e945: 12 total, 12 up, 12 in 2022-09-10T11:41:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:14 smithi104 ceph-mon[97044]: pgmap v13824: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:41:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:14 smithi110 ceph-mon[97633]: osdmap e945: 12 total, 12 up, 12 in 2022-09-10T11:41:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:14 smithi110 ceph-mon[97633]: pgmap v13824: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:41:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:15.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:15.278567+0000) 2022-09-10T11:41:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:15.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:15.756953+0000) 2022-09-10T11:41:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:16.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:16.278826+0000) 2022-09-10T11:41:17.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:16.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:16.757156+0000) 2022-09-10T11:41:17.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:17 smithi110 ceph-mon[97633]: pgmap v13825: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:41:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:17 smithi104 ceph-mon[97044]: pgmap v13825: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:41:17.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:17.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:17.279007+0000) 2022-09-10T11:41:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:17.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:17.757377+0000) 2022-09-10T11:41:18.565 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:18.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:18.279172+0000) 2022-09-10T11:41:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:18.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:18.757606+0000) 2022-09-10T11:41:19.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:19 smithi110 ceph-mon[97633]: pgmap v13826: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:41:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:19 smithi104 ceph-mon[97044]: pgmap v13826: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:41:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:19.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:19.279353+0000) 2022-09-10T11:41:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:19.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:19.757853+0000) 2022-09-10T11:41:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:20.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:20.279607+0000) 2022-09-10T11:41:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:20.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:20.758138+0000) 2022-09-10T11:41:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:21 smithi110 ceph-mon[97633]: pgmap v13827: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:41:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:21 smithi104 ceph-mon[97044]: pgmap v13827: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:41:21.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:21.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:21.279856+0000) 2022-09-10T11:41:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:21.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:21.758367+0000) 2022-09-10T11:41:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:22.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:22.280021+0000) 2022-09-10T11:41:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:22.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:22.758561+0000) 2022-09-10T11:41:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:23 smithi110 ceph-mon[97633]: pgmap v13828: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:41:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:23 smithi104 ceph-mon[97044]: pgmap v13828: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:41:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:23.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:23.280288+0000) 2022-09-10T11:41:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:23.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:23.758798+0000) 2022-09-10T11:41:24.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:24.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:24.280555+0000) 2022-09-10T11:41:25.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:24.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:24.759034+0000) 2022-09-10T11:41:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:24 smithi104 ceph-mon[97044]: pgmap v13829: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:41:25.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:24 smithi110 ceph-mon[97633]: pgmap v13829: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:41:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:25.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:25.280678+0000) 2022-09-10T11:41:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:25.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:25.759250+0000) 2022-09-10T11:41:26.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:26.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:26.280937+0000) 2022-09-10T11:41:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:26.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:26.759471+0000) 2022-09-10T11:41:27.426 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:27 smithi110 ceph-mon[97633]: pgmap v13830: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:41:27.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:27.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:27.281164+0000) 2022-09-10T11:41:27.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:27 smithi104 ceph-mon[97044]: pgmap v13830: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:41:28.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:27.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:27.759704+0000) 2022-09-10T11:41:28.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:41:27.627+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:41:28.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:28.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:28.281338+0000) 2022-09-10T11:41:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:28.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:28.760003+0000) 2022-09-10T11:41:29.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:29 smithi110 ceph-mon[97633]: pgmap v13831: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:29 smithi104 ceph-mon[97044]: pgmap v13831: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:29.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:29.281567+0000) 2022-09-10T11:41:30.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:29.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:29.760262+0000) 2022-09-10T11:41:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:30.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:30.281724+0000) 2022-09-10T11:41:31.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:30.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:30.760498+0000) 2022-09-10T11:41:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:31 smithi110 ceph-mon[97633]: pgmap v13832: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:31.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:31 smithi104 ceph-mon[97044]: pgmap v13832: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:31.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:31.281981+0000) 2022-09-10T11:41:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:31.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:31.760670+0000) 2022-09-10T11:41:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:32.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:32.282253+0000) 2022-09-10T11:41:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:32.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:32.760887+0000) 2022-09-10T11:41:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:33 smithi110 ceph-mon[97633]: pgmap v13833: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:33 smithi104 ceph-mon[97044]: pgmap v13833: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:33.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:33.282419+0000) 2022-09-10T11:41:34.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:33.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:33.761178+0000) 2022-09-10T11:41:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:34.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:34.282603+0000) 2022-09-10T11:41:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:34.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:34.761415+0000) 2022-09-10T11:41:35.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:34 smithi104 ceph-mon[97044]: pgmap v13834: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:35.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:34 smithi110 ceph-mon[97633]: pgmap v13834: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:35.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:35.282801+0000) 2022-09-10T11:41:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:35.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:35.761610+0000) 2022-09-10T11:41:36.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:36.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:36.283049+0000) 2022-09-10T11:41:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:36.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:36.761891+0000) 2022-09-10T11:41:37.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:37 smithi110 ceph-mon[97633]: pgmap v13835: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:37.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:37 smithi104 ceph-mon[97044]: pgmap v13835: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:37.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:37.283276+0000) 2022-09-10T11:41:38.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:37.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:37.762152+0000) 2022-09-10T11:41:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:38.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:38.283533+0000) 2022-09-10T11:41:39.066 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:38.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:38.762353+0000) 2022-09-10T11:41:39.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:39 smithi110 ceph-mon[97633]: pgmap v13836: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:39 smithi104 ceph-mon[97044]: pgmap v13836: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:39.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:39.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:39.283775+0000) 2022-09-10T11:41:40.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:39.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:39.762511+0000) 2022-09-10T11:41:40.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:40.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:40.283983+0000) 2022-09-10T11:41:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:40.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:40.762796+0000) 2022-09-10T11:41:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:41 smithi110 ceph-mon[97633]: pgmap v13837: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:41.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:41 smithi104 ceph-mon[97044]: pgmap v13837: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:41.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:41.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:41.284221+0000) 2022-09-10T11:41:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:41.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:41.763035+0000) 2022-09-10T11:41:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:42.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:42.284435+0000) 2022-09-10T11:41:43.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:42.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:42.763253+0000) 2022-09-10T11:41:43.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:41:42.629+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:41:43.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:43 smithi104 ceph-mon[97044]: pgmap v13838: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:43.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:43.284641+0000) 2022-09-10T11:41:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:43 smithi110 ceph-mon[97633]: pgmap v13838: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:44.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:43.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:43.763419+0000) 2022-09-10T11:41:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:44.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:44.284824+0000) 2022-09-10T11:41:45.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:44.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:44.763679+0000) 2022-09-10T11:41:45.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:44 smithi104 ceph-mon[97044]: pgmap v13839: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:44 smithi110 ceph-mon[97633]: pgmap v13839: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:45.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:45.285042+0000) 2022-09-10T11:41:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:45.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:45.763910+0000) 2022-09-10T11:41:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:41:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:41:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:41:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:46.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:46.285272+0000) 2022-09-10T11:41:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:46.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:46.764210+0000) 2022-09-10T11:41:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:46 smithi104 ceph-mon[97044]: pgmap v13840: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:46 smithi110 ceph-mon[97633]: pgmap v13840: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:47.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:47.285528+0000) 2022-09-10T11:41:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:47.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:47.764379+0000) 2022-09-10T11:41:48.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:48.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:48.285753+0000) 2022-09-10T11:41:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:48.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:48.764546+0000) 2022-09-10T11:41:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:49 smithi110 ceph-mon[97633]: pgmap v13841: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:49.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:49 smithi104 ceph-mon[97044]: pgmap v13841: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:49.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:49.285980+0000) 2022-09-10T11:41:50.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:49.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:49.764676+0000) 2022-09-10T11:41:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:50.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:50.286206+0000) 2022-09-10T11:41:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:50.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:50.764901+0000) 2022-09-10T11:41:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:51 smithi110 ceph-mon[97633]: pgmap v13842: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:51 smithi104 ceph-mon[97044]: pgmap v13842: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:51.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:51.286370+0000) 2022-09-10T11:41:52.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:51.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:51.765182+0000) 2022-09-10T11:41:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:52.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:52.286584+0000) 2022-09-10T11:41:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:52.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:52.765411+0000) 2022-09-10T11:41:53.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:53 smithi104 ceph-mon[97044]: pgmap v13843: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:53.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:53.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:53.286778+0000) 2022-09-10T11:41:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:53 smithi110 ceph-mon[97633]: pgmap v13843: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:54.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:53.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:53.765582+0000) 2022-09-10T11:41:54.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:54.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:54.287003+0000) 2022-09-10T11:41:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:54.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:54.765813+0000) 2022-09-10T11:41:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:54 smithi104 ceph-mon[97044]: pgmap v13844: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:55.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:54 smithi110 ceph-mon[97633]: pgmap v13844: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:55.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:55.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:55.287245+0000) 2022-09-10T11:41:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:55.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:55.765988+0000) 2022-09-10T11:41:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:56.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:56.287407+0000) 2022-09-10T11:41:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:56.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:56.766194+0000) 2022-09-10T11:41:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:57 smithi110 ceph-mon[97633]: pgmap v13845: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:57.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:57.287613+0000) 2022-09-10T11:41:57.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:57 smithi104 ceph-mon[97044]: pgmap v13845: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:41:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:57.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:57.766427+0000) 2022-09-10T11:41:58.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:41:57.644+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:41:58.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:58.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:58.287821+0000) 2022-09-10T11:41:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:58.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:58.766580+0000) 2022-09-10T11:41:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:41:59 smithi104 ceph-mon[97044]: pgmap v13846: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:41:59.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:41:59.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:59.288065+0000) 2022-09-10T11:41:59.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:41:59 smithi110 ceph-mon[97633]: pgmap v13846: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:00.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:41:59.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:41:59.766827+0000) 2022-09-10T11:42:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:00.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:00.288342+0000) 2022-09-10T11:42:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:00.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:00.767021+0000) 2022-09-10T11:42:01.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:01 smithi104 ceph-mon[97044]: pgmap v13847: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:01.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:01.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:01.288574+0000) 2022-09-10T11:42:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:01 smithi110 ceph-mon[97633]: pgmap v13847: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:01.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:01.767261+0000) 2022-09-10T11:42:02.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:02.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:02.288752+0000) 2022-09-10T11:42:03.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:02.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:02.767484+0000) 2022-09-10T11:42:03.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:03.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:03.289003+0000) 2022-09-10T11:42:03.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:03 smithi104 ceph-mon[97044]: pgmap v13848: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:03.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:03 smithi110 ceph-mon[97633]: pgmap v13848: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:03.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:03.767704+0000) 2022-09-10T11:42:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:04.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:04.289276+0000) 2022-09-10T11:42:05.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:04.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:04.767963+0000) 2022-09-10T11:42:05.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:04 smithi104 ceph-mon[97044]: pgmap v13849: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:05.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:04 smithi110 ceph-mon[97633]: pgmap v13849: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:05.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:05.289406+0000) 2022-09-10T11:42:06.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:05.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:05.768222+0000) 2022-09-10T11:42:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:06.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:06.289617+0000) 2022-09-10T11:42:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:06.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:06.768399+0000) 2022-09-10T11:42:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:07 smithi104 ceph-mon[97044]: pgmap v13850: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:42:07.570 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:07.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:07.289818+0000) 2022-09-10T11:42:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:07 smithi110 ceph-mon[97633]: pgmap v13850: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:42:08.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:07.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:07.768576+0000) 2022-09-10T11:42:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:08.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:08.290059+0000) 2022-09-10T11:42:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:08.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:08.768805+0000) 2022-09-10T11:42:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:09 smithi104 ceph-mon[97044]: pgmap v13851: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:42:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:09 smithi104 ceph-mon[97044]: osdmap e946: 12 total, 12 up, 12 in 2022-09-10T11:42:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:09.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:09.290316+0000) 2022-09-10T11:42:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:09 smithi110 ceph-mon[97633]: pgmap v13851: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:42:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:09 smithi110 ceph-mon[97633]: osdmap e946: 12 total, 12 up, 12 in 2022-09-10T11:42:10.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:09.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:09.769006+0000) 2022-09-10T11:42:10.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:10.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:10.290581+0000) 2022-09-10T11:42:11.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:10.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:10.769202+0000) 2022-09-10T11:42:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:11 smithi104 ceph-mon[97044]: pgmap v13853: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:42:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:11.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:11.290824+0000) 2022-09-10T11:42:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:11 smithi110 ceph-mon[97633]: pgmap v13853: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:42:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:11.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:11.769423+0000) 2022-09-10T11:42:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:12.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:12.291080+0000) 2022-09-10T11:42:13.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:42:12.657+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:42:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:12.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:12.769665+0000) 2022-09-10T11:42:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:13 smithi104 ceph-mon[97044]: pgmap v13854: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:42:13.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:13.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:13.291271+0000) 2022-09-10T11:42:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:13 smithi110 ceph-mon[97633]: pgmap v13854: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:42:14.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:13.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:13.769926+0000) 2022-09-10T11:42:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:14.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:14.291523+0000) 2022-09-10T11:42:15.067 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:14.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:14.770131+0000) 2022-09-10T11:42:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:14 smithi104 ceph-mon[97044]: osdmap e947: 12 total, 12 up, 12 in 2022-09-10T11:42:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:14 smithi104 ceph-mon[97044]: pgmap v13856: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:42:15.423 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:14 smithi110 ceph-mon[97633]: osdmap e947: 12 total, 12 up, 12 in 2022-09-10T11:42:15.423 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:14 smithi110 ceph-mon[97633]: pgmap v13856: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:42:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:15.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:15.291771+0000) 2022-09-10T11:42:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:15.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:15.770360+0000) 2022-09-10T11:42:16.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:16.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:16.292017+0000) 2022-09-10T11:42:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:16.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:16.770589+0000) 2022-09-10T11:42:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:17 smithi104 ceph-mon[97044]: pgmap v13857: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:42:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:17.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:17.292287+0000) 2022-09-10T11:42:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:17 smithi110 ceph-mon[97633]: pgmap v13857: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:42:18.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:17.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:17.770776+0000) 2022-09-10T11:42:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:18.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:18.292527+0000) 2022-09-10T11:42:19.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:18.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:18.770993+0000) 2022-09-10T11:42:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:19 smithi104 ceph-mon[97044]: pgmap v13858: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:42:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:19.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:19.292747+0000) 2022-09-10T11:42:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:19 smithi110 ceph-mon[97633]: pgmap v13858: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:42:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:19.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:19.771233+0000) 2022-09-10T11:42:20.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:20.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:20.292982+0000) 2022-09-10T11:42:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:20.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:20.771455+0000) 2022-09-10T11:42:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:21 smithi104 ceph-mon[97044]: pgmap v13859: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:42:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:21.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:21.293183+0000) 2022-09-10T11:42:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:21 smithi110 ceph-mon[97633]: pgmap v13859: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:42:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:21.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:21.771657+0000) 2022-09-10T11:42:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:22.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:22.293408+0000) 2022-09-10T11:42:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:22.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:22.771942+0000) 2022-09-10T11:42:23.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:23 smithi104 ceph-mon[97044]: pgmap v13860: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:42:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:23.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:23.293606+0000) 2022-09-10T11:42:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:23 smithi110 ceph-mon[97633]: pgmap v13860: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:42:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:23.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:23.772155+0000) 2022-09-10T11:42:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:24.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:24.293830+0000) 2022-09-10T11:42:25.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:24.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:24.772337+0000) 2022-09-10T11:42:25.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:24 smithi104 ceph-mon[97044]: pgmap v13861: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:42:25.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:24 smithi110 ceph-mon[97633]: pgmap v13861: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:42:25.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:25.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:25.294082+0000) 2022-09-10T11:42:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:25.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:25.772545+0000) 2022-09-10T11:42:26.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:26.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:26.294324+0000) 2022-09-10T11:42:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:26.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:26.772746+0000) 2022-09-10T11:42:27.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:27 smithi104 ceph-mon[97044]: pgmap v13862: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:42:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:27.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:27.294527+0000) 2022-09-10T11:42:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:27 smithi110 ceph-mon[97633]: pgmap v13862: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:42:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:27.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:27.772977+0000) 2022-09-10T11:42:28.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:42:27.673+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:42:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:28.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:28.294731+0000) 2022-09-10T11:42:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:28.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:28.773174+0000) 2022-09-10T11:42:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:29 smithi104 ceph-mon[97044]: pgmap v13863: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:29.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:29.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:29.294940+0000) 2022-09-10T11:42:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:29 smithi110 ceph-mon[97633]: pgmap v13863: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:29.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:29.773393+0000) 2022-09-10T11:42:30.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:30.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:30.295210+0000) 2022-09-10T11:42:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:30.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:30.773560+0000) 2022-09-10T11:42:31.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:31 smithi104 ceph-mon[97044]: pgmap v13864: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:31.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:31.295409+0000) 2022-09-10T11:42:31.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:31 smithi110 ceph-mon[97633]: pgmap v13864: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:31.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:31.773809+0000) 2022-09-10T11:42:32.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:32.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:32.295656+0000) 2022-09-10T11:42:33.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:32.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:32.774006+0000) 2022-09-10T11:42:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:33 smithi104 ceph-mon[97044]: pgmap v13865: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:33.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:33.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:33.295861+0000) 2022-09-10T11:42:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:33 smithi110 ceph-mon[97633]: pgmap v13865: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:33.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:33.774252+0000) 2022-09-10T11:42:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:34.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:34.296089+0000) 2022-09-10T11:42:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:34.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:34.774478+0000) 2022-09-10T11:42:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:34 smithi104 ceph-mon[97044]: pgmap v13866: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:34 smithi110 ceph-mon[97633]: pgmap v13866: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:35.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:35.296302+0000) 2022-09-10T11:42:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:35.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:35.774613+0000) 2022-09-10T11:42:36.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:36.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:36.296477+0000) 2022-09-10T11:42:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:36.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:36.774826+0000) 2022-09-10T11:42:37.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:37 smithi104 ceph-mon[97044]: pgmap v13867: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:37.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:37.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:37.296687+0000) 2022-09-10T11:42:37.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:37 smithi110 ceph-mon[97633]: pgmap v13867: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:38.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:37.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:37.775054+0000) 2022-09-10T11:42:38.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:38.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:38.296919+0000) 2022-09-10T11:42:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:38.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:38.775359+0000) 2022-09-10T11:42:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:39 smithi104 ceph-mon[97044]: pgmap v13868: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:39.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:39.297183+0000) 2022-09-10T11:42:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:39 smithi110 ceph-mon[97633]: pgmap v13868: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:39.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:39.775516+0000) 2022-09-10T11:42:40.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:40.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:40.297422+0000) 2022-09-10T11:42:41.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:40.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:40.775716+0000) 2022-09-10T11:42:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:41 smithi104 ceph-mon[97044]: pgmap v13869: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:41.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:41.297573+0000) 2022-09-10T11:42:41.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:41 smithi110 ceph-mon[97633]: pgmap v13869: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:41.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:41.775900+0000) 2022-09-10T11:42:42.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:42.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:42.297757+0000) 2022-09-10T11:42:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:42.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:42.776163+0000) 2022-09-10T11:42:43.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:42:42.686+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:42:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:43 smithi104 ceph-mon[97044]: pgmap v13870: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:43.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:43.298001+0000) 2022-09-10T11:42:43.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:43 smithi110 ceph-mon[97633]: pgmap v13870: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:43.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:43.776385+0000) 2022-09-10T11:42:44.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:44.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:44.298283+0000) 2022-09-10T11:42:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:44.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:44.776578+0000) 2022-09-10T11:42:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:44 smithi104 ceph-mon[97044]: pgmap v13871: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:45.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:44 smithi110 ceph-mon[97633]: pgmap v13871: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:45.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:45.298498+0000) 2022-09-10T11:42:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:45.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:45.776836+0000) 2022-09-10T11:42:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:42:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:42:46.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:42:46.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:42:46.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:42:46.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:42:46.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:46.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:46.298712+0000) 2022-09-10T11:42:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:46.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:46.777054+0000) 2022-09-10T11:42:47.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:46 smithi104 ceph-mon[97044]: pgmap v13872: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:47.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:46 smithi110 ceph-mon[97633]: pgmap v13872: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:47.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:47.298994+0000) 2022-09-10T11:42:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:47.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:47.777268+0000) 2022-09-10T11:42:48.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:48.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:48.299270+0000) 2022-09-10T11:42:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:48.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:48.777523+0000) 2022-09-10T11:42:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:49 smithi104 ceph-mon[97044]: pgmap v13873: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:49.571 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:49.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:49.299469+0000) 2022-09-10T11:42:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:49 smithi110 ceph-mon[97633]: pgmap v13873: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:49.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:49.777753+0000) 2022-09-10T11:42:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:50.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:50.299671+0000) 2022-09-10T11:42:51.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:50.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:50.777961+0000) 2022-09-10T11:42:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:51 smithi104 ceph-mon[97044]: pgmap v13874: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:51.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:51.299874+0000) 2022-09-10T11:42:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:51 smithi110 ceph-mon[97633]: pgmap v13874: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:51.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:51.778172+0000) 2022-09-10T11:42:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:52.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:52.300059+0000) 2022-09-10T11:42:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:52.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:52.778414+0000) 2022-09-10T11:42:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:53 smithi104 ceph-mon[97044]: pgmap v13875: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:53.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:53.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:53.300256+0000) 2022-09-10T11:42:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:53 smithi110 ceph-mon[97633]: pgmap v13875: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:53.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:53.778596+0000) 2022-09-10T11:42:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:54.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:54.300492+0000) 2022-09-10T11:42:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:54.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:54.778847+0000) 2022-09-10T11:42:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:54 smithi104 ceph-mon[97044]: pgmap v13876: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:54 smithi110 ceph-mon[97633]: pgmap v13876: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:55.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:55.300741+0000) 2022-09-10T11:42:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:55.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:55.779127+0000) 2022-09-10T11:42:56.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:56.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:56.300962+0000) 2022-09-10T11:42:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:56.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:56.779404+0000) 2022-09-10T11:42:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:57 smithi104 ceph-mon[97044]: pgmap v13877: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:57.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:57.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:57.301206+0000) 2022-09-10T11:42:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:57 smithi110 ceph-mon[97633]: pgmap v13877: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:42:58.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:57.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:57.779665+0000) 2022-09-10T11:42:58.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:42:57.692+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:42:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:58.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:58.301453+0000) 2022-09-10T11:42:59.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:58.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:58.779933+0000) 2022-09-10T11:42:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:42:59 smithi104 ceph-mon[97044]: pgmap v13878: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:42:59.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:42:59.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:59.301624+0000) 2022-09-10T11:42:59.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:42:59 smithi110 ceph-mon[97633]: pgmap v13878: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:00.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:42:59.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:42:59.780131+0000) 2022-09-10T11:43:00.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:00.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:00.301846+0000) 2022-09-10T11:43:01.065 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:00.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:00.780250+0000) 2022-09-10T11:43:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:01 smithi104 ceph-mon[97044]: pgmap v13879: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:01.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:01.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:01.302016+0000) 2022-09-10T11:43:01.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:01 smithi110 ceph-mon[97633]: pgmap v13879: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:01.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:01.780489+0000) 2022-09-10T11:43:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:02.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:02.302238+0000) 2022-09-10T11:43:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:02.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:02.780680+0000) 2022-09-10T11:43:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:03 smithi104 ceph-mon[97044]: pgmap v13880: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:03.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:03.302444+0000) 2022-09-10T11:43:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:03 smithi110 ceph-mon[97633]: pgmap v13880: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:03.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:03.780913+0000) 2022-09-10T11:43:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:04.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:04.302721+0000) 2022-09-10T11:43:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:04.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:04.781177+0000) 2022-09-10T11:43:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:04 smithi104 ceph-mon[97044]: pgmap v13881: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:04 smithi110 ceph-mon[97633]: pgmap v13881: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:05.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:05.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:05.302981+0000) 2022-09-10T11:43:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:05.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:05.781412+0000) 2022-09-10T11:43:06.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:06.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:06.303220+0000) 2022-09-10T11:43:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:06.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:06.781653+0000) 2022-09-10T11:43:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:07 smithi104 ceph-mon[97044]: pgmap v13882: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:43:07.714 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:07 smithi110 ceph-mon[97633]: pgmap v13882: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:43:07.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:07.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:07.303442+0000) 2022-09-10T11:43:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:07.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:07.781902+0000) 2022-09-10T11:43:08.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:08.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:08.303596+0000) 2022-09-10T11:43:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:08.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:08.782183+0000) 2022-09-10T11:43:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:09 smithi104 ceph-mon[97044]: pgmap v13883: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:43:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:09 smithi104 ceph-mon[97044]: osdmap e948: 12 total, 12 up, 12 in 2022-09-10T11:43:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:09 smithi110 ceph-mon[97633]: pgmap v13883: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:43:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:09 smithi110 ceph-mon[97633]: osdmap e948: 12 total, 12 up, 12 in 2022-09-10T11:43:09.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:09.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:09.303744+0000) 2022-09-10T11:43:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:09.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:09.782430+0000) 2022-09-10T11:43:10.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:10.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:10.303950+0000) 2022-09-10T11:43:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:10.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:10.782680+0000) 2022-09-10T11:43:11.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:11 smithi104 ceph-mon[97044]: pgmap v13885: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:43:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:11 smithi110 ceph-mon[97633]: pgmap v13885: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:43:11.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:11.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:11.304161+0000) 2022-09-10T11:43:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:11.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:11.782920+0000) 2022-09-10T11:43:12.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:12.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:12.304403+0000) 2022-09-10T11:43:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:12.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:12.783129+0000) 2022-09-10T11:43:13.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:43:12.698+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:43:13.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:13 smithi104 ceph-mon[97044]: pgmap v13886: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:43:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:13 smithi110 ceph-mon[97633]: pgmap v13886: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:43:13.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:13.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:13.304550+0000) 2022-09-10T11:43:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:13.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:13.783299+0000) 2022-09-10T11:43:14.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:14.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:14.304778+0000) 2022-09-10T11:43:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:14.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:14.783431+0000) 2022-09-10T11:43:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:14 smithi104 ceph-mon[97044]: osdmap e949: 12 total, 12 up, 12 in 2022-09-10T11:43:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:14 smithi104 ceph-mon[97044]: pgmap v13888: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:43:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:14 smithi110 ceph-mon[97633]: osdmap e949: 12 total, 12 up, 12 in 2022-09-10T11:43:15.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:14 smithi110 ceph-mon[97633]: pgmap v13888: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:43:15.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:15.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:15.305024+0000) 2022-09-10T11:43:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:15.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:15.783547+0000) 2022-09-10T11:43:16.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:16.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:16.305253+0000) 2022-09-10T11:43:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:16.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:16.783770+0000) 2022-09-10T11:43:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:17 smithi104 ceph-mon[97044]: pgmap v13889: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:43:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:17 smithi110 ceph-mon[97633]: pgmap v13889: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:43:17.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:17.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:17.305519+0000) 2022-09-10T11:43:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:17.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:17.784004+0000) 2022-09-10T11:43:18.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:18.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:18.305793+0000) 2022-09-10T11:43:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:18.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:18.784190+0000) 2022-09-10T11:43:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:19 smithi104 ceph-mon[97044]: pgmap v13890: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:43:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:19 smithi110 ceph-mon[97633]: pgmap v13890: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:43:19.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:19.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:19.306021+0000) 2022-09-10T11:43:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:19.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:19.784382+0000) 2022-09-10T11:43:20.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:20.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:20.306274+0000) 2022-09-10T11:43:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:20.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:20.784601+0000) 2022-09-10T11:43:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:21 smithi104 ceph-mon[97044]: pgmap v13891: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:43:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:21 smithi110 ceph-mon[97633]: pgmap v13891: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:43:21.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:21.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:21.306444+0000) 2022-09-10T11:43:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:21.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:21.784757+0000) 2022-09-10T11:43:22.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:22.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:22.306727+0000) 2022-09-10T11:43:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:22.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:22.784971+0000) 2022-09-10T11:43:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:23 smithi104 ceph-mon[97044]: pgmap v13892: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:43:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:23 smithi110 ceph-mon[97633]: pgmap v13892: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:43:23.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:23.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:23.306949+0000) 2022-09-10T11:43:24.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:23.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:23.785232+0000) 2022-09-10T11:43:24.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:24.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:24.307203+0000) 2022-09-10T11:43:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:24.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:24.785493+0000) 2022-09-10T11:43:25.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:24 smithi104 ceph-mon[97044]: pgmap v13893: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:43:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:24 smithi110 ceph-mon[97633]: pgmap v13893: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:43:25.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:25.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:25.307465+0000) 2022-09-10T11:43:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:25.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:25.785730+0000) 2022-09-10T11:43:26.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:26.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:26.307688+0000) 2022-09-10T11:43:27.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:26.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:26.785933+0000) 2022-09-10T11:43:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:27 smithi104 ceph-mon[97044]: pgmap v13894: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:43:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:27 smithi110 ceph-mon[97633]: pgmap v13894: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:43:27.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:27.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:27.307931+0000) 2022-09-10T11:43:28.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:43:27.703+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:43:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:27.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:27.786187+0000) 2022-09-10T11:43:28.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:28.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:28.308170+0000) 2022-09-10T11:43:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:28.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:28.786451+0000) 2022-09-10T11:43:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:29 smithi104 ceph-mon[97044]: pgmap v13895: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:29 smithi110 ceph-mon[97633]: pgmap v13895: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:29.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:29.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:29.308320+0000) 2022-09-10T11:43:30.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:29.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:29.786679+0000) 2022-09-10T11:43:30.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:30.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:30.308496+0000) 2022-09-10T11:43:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:30.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:30.786866+0000) 2022-09-10T11:43:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:31 smithi104 ceph-mon[97044]: pgmap v13896: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:31 smithi110 ceph-mon[97633]: pgmap v13896: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:31.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:31.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:31.308761+0000) 2022-09-10T11:43:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:31.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:31.787055+0000) 2022-09-10T11:43:32.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:32.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:32.309012+0000) 2022-09-10T11:43:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:32.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:32.787245+0000) 2022-09-10T11:43:33.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:33 smithi104 ceph-mon[97044]: pgmap v13897: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:33 smithi110 ceph-mon[97633]: pgmap v13897: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:33.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:33.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:33.309202+0000) 2022-09-10T11:43:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:33.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:33.787454+0000) 2022-09-10T11:43:34.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:34.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:34.309394+0000) 2022-09-10T11:43:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:34.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:34.787711+0000) 2022-09-10T11:43:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:34 smithi104 ceph-mon[97044]: pgmap v13898: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:35.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:34 smithi110 ceph-mon[97633]: pgmap v13898: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:35.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:35.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:35.309575+0000) 2022-09-10T11:43:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:35.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:35.787962+0000) 2022-09-10T11:43:36.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:36.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:36.309848+0000) 2022-09-10T11:43:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:36.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:36.788262+0000) 2022-09-10T11:43:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:37 smithi104 ceph-mon[97044]: pgmap v13899: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:37 smithi110 ceph-mon[97633]: pgmap v13899: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:37.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:37.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:37.310120+0000) 2022-09-10T11:43:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:37.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:37.788456+0000) 2022-09-10T11:43:38.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:38.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:38.310351+0000) 2022-09-10T11:43:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:38.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:38.788644+0000) 2022-09-10T11:43:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:39 smithi104 ceph-mon[97044]: pgmap v13900: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:39 smithi110 ceph-mon[97633]: pgmap v13900: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:39.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:39.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:39.310579+0000) 2022-09-10T11:43:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:39.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:39.788868+0000) 2022-09-10T11:43:40.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:40.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:40.310837+0000) 2022-09-10T11:43:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:40.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:40.789092+0000) 2022-09-10T11:43:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:41 smithi104 ceph-mon[97044]: pgmap v13901: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:41 smithi110 ceph-mon[97633]: pgmap v13901: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:41.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:41.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:41.311059+0000) 2022-09-10T11:43:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:41.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:41.789372+0000) 2022-09-10T11:43:42.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:42.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:42.311224+0000) 2022-09-10T11:43:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:42.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:42.789626+0000) 2022-09-10T11:43:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:43:42.719+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:43:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:43 smithi104 ceph-mon[97044]: pgmap v13902: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:43.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:43 smithi110 ceph-mon[97633]: pgmap v13902: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:43.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:43.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:43.311436+0000) 2022-09-10T11:43:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:43.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:43.789844+0000) 2022-09-10T11:43:44.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:44.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:44.311667+0000) 2022-09-10T11:43:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:44.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:44.790059+0000) 2022-09-10T11:43:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:44 smithi104 ceph-mon[97044]: pgmap v13903: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:44 smithi110 ceph-mon[97633]: pgmap v13903: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:45.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:45.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:45.311886+0000) 2022-09-10T11:43:46.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:45.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:45.790277+0000) 2022-09-10T11:43:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:43:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:43:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:43:46.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:43:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:43:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:43:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:43:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:43:46.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:46.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:46.312076+0000) 2022-09-10T11:43:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:46.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:46.790470+0000) 2022-09-10T11:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:46 smithi104 ceph-mon[97044]: pgmap v13904: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:47.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:46 smithi110 ceph-mon[97633]: pgmap v13904: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:47.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:47.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:47.312278+0000) 2022-09-10T11:43:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:47.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:47.790681+0000) 2022-09-10T11:43:48.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:48.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:48.312519+0000) 2022-09-10T11:43:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:48.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:48.790904+0000) 2022-09-10T11:43:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:49 smithi104 ceph-mon[97044]: pgmap v13905: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:49 smithi110 ceph-mon[97633]: pgmap v13905: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:49.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:49.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:49.312705+0000) 2022-09-10T11:43:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:49.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:49.791179+0000) 2022-09-10T11:43:50.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:50.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:50.312933+0000) 2022-09-10T11:43:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:50.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:50.791398+0000) 2022-09-10T11:43:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:51 smithi104 ceph-mon[97044]: pgmap v13906: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:51 smithi110 ceph-mon[97633]: pgmap v13906: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:51.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:51.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:51.313208+0000) 2022-09-10T11:43:52.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:51.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:51.791654+0000) 2022-09-10T11:43:52.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:52.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:52.313378+0000) 2022-09-10T11:43:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:52.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:52.791915+0000) 2022-09-10T11:43:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:53 smithi104 ceph-mon[97044]: pgmap v13907: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:53 smithi110 ceph-mon[97633]: pgmap v13907: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:53.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:53.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:53.313567+0000) 2022-09-10T11:43:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:53.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:53.792084+0000) 2022-09-10T11:43:54.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:54.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:54.313752+0000) 2022-09-10T11:43:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:54.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:54.792340+0000) 2022-09-10T11:43:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:54 smithi104 ceph-mon[97044]: pgmap v13908: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:54 smithi110 ceph-mon[97633]: pgmap v13908: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:55.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:55.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:55.313936+0000) 2022-09-10T11:43:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:55.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:55.792478+0000) 2022-09-10T11:43:56.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:56.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:56.314188+0000) 2022-09-10T11:43:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:56.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:56.792711+0000) 2022-09-10T11:43:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:57 smithi104 ceph-mon[97044]: pgmap v13909: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:57 smithi110 ceph-mon[97633]: pgmap v13909: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:43:57.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:57.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:57.314387+0000) 2022-09-10T11:43:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:57.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:57.792884+0000) 2022-09-10T11:43:58.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:43:57.731+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:43:58.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:58.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:58.314588+0000) 2022-09-10T11:43:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:58.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:58.793059+0000) 2022-09-10T11:43:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:43:59 smithi104 ceph-mon[97044]: pgmap v13910: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:43:59 smithi110 ceph-mon[97633]: pgmap v13910: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:43:59.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:43:59.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:59.314756+0000) 2022-09-10T11:44:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:43:59.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:43:59.793318+0000) 2022-09-10T11:44:00.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:00.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:00.314979+0000) 2022-09-10T11:44:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:00.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:00.793481+0000) 2022-09-10T11:44:01.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:01 smithi104 ceph-mon[97044]: pgmap v13911: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:01 smithi110 ceph-mon[97633]: pgmap v13911: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:01.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:01.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:01.315183+0000) 2022-09-10T11:44:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:01.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:01.793716+0000) 2022-09-10T11:44:02.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:02.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:02.315425+0000) 2022-09-10T11:44:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:02.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:02.793961+0000) 2022-09-10T11:44:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:03 smithi104 ceph-mon[97044]: pgmap v13912: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:03 smithi110 ceph-mon[97633]: pgmap v13912: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:03.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:03.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:03.315677+0000) 2022-09-10T11:44:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:03.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:03.794218+0000) 2022-09-10T11:44:04.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:04.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:04.315851+0000) 2022-09-10T11:44:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:04.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:04.794417+0000) 2022-09-10T11:44:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:04 smithi104 ceph-mon[97044]: pgmap v13913: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:04 smithi110 ceph-mon[97633]: pgmap v13913: 129 pgs: 129 active+clean; 9.2 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:05.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:05.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:05.316037+0000) 2022-09-10T11:44:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:05.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:05.794602+0000) 2022-09-10T11:44:06.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:06.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:06.316196+0000) 2022-09-10T11:44:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:06.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:06.794843+0000) 2022-09-10T11:44:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:07 smithi104 ceph-mon[97044]: pgmap v13914: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:44:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:07 smithi110 ceph-mon[97633]: pgmap v13914: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:44:07.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:07.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:07.316381+0000) 2022-09-10T11:44:08.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:07.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:07.795136+0000) 2022-09-10T11:44:08.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:08.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:08.316604+0000) 2022-09-10T11:44:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:08.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:08.795401+0000) 2022-09-10T11:44:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:09 smithi104 ceph-mon[97044]: pgmap v13915: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:44:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:09 smithi104 ceph-mon[97044]: osdmap e950: 12 total, 12 up, 12 in 2022-09-10T11:44:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:09 smithi110 ceph-mon[97633]: pgmap v13915: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:44:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:09 smithi110 ceph-mon[97633]: osdmap e950: 12 total, 12 up, 12 in 2022-09-10T11:44:09.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:09.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:09.316828+0000) 2022-09-10T11:44:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:09.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:09.795646+0000) 2022-09-10T11:44:10.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:10.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:10.317024+0000) 2022-09-10T11:44:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:10.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:10.795857+0000) 2022-09-10T11:44:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:11 smithi104 ceph-mon[97044]: pgmap v13917: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:44:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:11 smithi110 ceph-mon[97633]: pgmap v13917: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:44:11.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:11.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:11.317248+0000) 2022-09-10T11:44:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:11.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:11.796048+0000) 2022-09-10T11:44:12.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:12.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:12.317498+0000) 2022-09-10T11:44:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:12.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:12.796273+0000) 2022-09-10T11:44:13.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:44:12.739+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:44:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:13 smithi104 ceph-mon[97044]: pgmap v13918: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:44:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:13 smithi110 ceph-mon[97633]: pgmap v13918: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:44:13.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:13.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:13.317735+0000) 2022-09-10T11:44:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:13.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:13.796524+0000) 2022-09-10T11:44:14.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:14.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:14.317955+0000) 2022-09-10T11:44:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:14.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:14.796743+0000) 2022-09-10T11:44:15.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:14 smithi104 ceph-mon[97044]: osdmap e951: 12 total, 12 up, 12 in 2022-09-10T11:44:15.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:14 smithi104 ceph-mon[97044]: pgmap v13920: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:44:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:14 smithi110 ceph-mon[97633]: osdmap e951: 12 total, 12 up, 12 in 2022-09-10T11:44:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:14 smithi110 ceph-mon[97633]: pgmap v13920: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:44:15.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:15.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:15.318173+0000) 2022-09-10T11:44:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:15.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:15.796866+0000) 2022-09-10T11:44:16.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:16.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:16.318334+0000) 2022-09-10T11:44:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:16.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:16.797033+0000) 2022-09-10T11:44:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:17 smithi104 ceph-mon[97044]: pgmap v13921: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:44:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:17 smithi110 ceph-mon[97633]: pgmap v13921: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:44:17.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:17.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:17.318576+0000) 2022-09-10T11:44:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:17.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:17.797323+0000) 2022-09-10T11:44:18.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:18.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:18.318868+0000) 2022-09-10T11:44:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:18.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:18.797532+0000) 2022-09-10T11:44:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:19 smithi104 ceph-mon[97044]: pgmap v13922: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:44:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:19 smithi110 ceph-mon[97633]: pgmap v13922: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:44:19.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:19.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:19.319082+0000) 2022-09-10T11:44:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:19.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:19.797738+0000) 2022-09-10T11:44:20.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:20.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:20.319358+0000) 2022-09-10T11:44:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:20.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:20.798001+0000) 2022-09-10T11:44:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:21 smithi104 ceph-mon[97044]: pgmap v13923: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:44:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:21 smithi110 ceph-mon[97633]: pgmap v13923: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:44:21.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:21.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:21.319579+0000) 2022-09-10T11:44:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:21.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:21.798235+0000) 2022-09-10T11:44:22.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:22.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:22.319827+0000) 2022-09-10T11:44:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:22.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:22.798482+0000) 2022-09-10T11:44:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:23 smithi104 ceph-mon[97044]: pgmap v13924: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:44:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:23 smithi110 ceph-mon[97633]: pgmap v13924: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:44:23.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:23.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:23.320027+0000) 2022-09-10T11:44:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:23.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:23.798674+0000) 2022-09-10T11:44:24.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:24.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:24.320227+0000) 2022-09-10T11:44:25.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:24.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:24.798859+0000) 2022-09-10T11:44:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:24 smithi104 ceph-mon[97044]: pgmap v13925: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:44:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:24 smithi110 ceph-mon[97633]: pgmap v13925: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:44:25.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:25.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:25.320407+0000) 2022-09-10T11:44:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:25.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:25.799163+0000) 2022-09-10T11:44:26.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:26.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:26.320585+0000) 2022-09-10T11:44:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:26.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:26.799408+0000) 2022-09-10T11:44:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:27 smithi104 ceph-mon[97044]: pgmap v13926: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:44:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:27 smithi110 ceph-mon[97633]: pgmap v13926: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:44:27.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:27.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:27.320742+0000) 2022-09-10T11:44:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:27.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:27.799563+0000) 2022-09-10T11:44:28.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:44:27.749+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:44:28.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:28.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:28.320908+0000) 2022-09-10T11:44:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:28.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:28.799796+0000) 2022-09-10T11:44:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:29 smithi104 ceph-mon[97044]: pgmap v13927: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:29.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:29 smithi110 ceph-mon[97633]: pgmap v13927: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:29.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:29.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:29.321154+0000) 2022-09-10T11:44:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:29.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:29.800027+0000) 2022-09-10T11:44:30.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:30.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:30.321342+0000) 2022-09-10T11:44:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:30.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:30.800236+0000) 2022-09-10T11:44:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:31 smithi104 ceph-mon[97044]: pgmap v13928: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:31 smithi110 ceph-mon[97633]: pgmap v13928: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:31.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:31.321594+0000) 2022-09-10T11:44:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:31.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:31.800439+0000) 2022-09-10T11:44:32.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:32.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:32.321784+0000) 2022-09-10T11:44:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:32.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:32.800644+0000) 2022-09-10T11:44:33.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:33 smithi104 ceph-mon[97044]: pgmap v13929: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:33.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:33 smithi110 ceph-mon[97633]: pgmap v13929: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:33.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:33.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:33.321978+0000) 2022-09-10T11:44:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:33.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:33.800879+0000) 2022-09-10T11:44:34.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:34.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:34.322212+0000) 2022-09-10T11:44:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:34.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:34.801145+0000) 2022-09-10T11:44:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:34 smithi104 ceph-mon[97044]: pgmap v13930: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:34 smithi110 ceph-mon[97633]: pgmap v13930: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:35.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:35.322460+0000) 2022-09-10T11:44:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:35.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:35.801351+0000) 2022-09-10T11:44:36.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:36.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:36.322656+0000) 2022-09-10T11:44:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:36.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:36.801538+0000) 2022-09-10T11:44:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:37 smithi104 ceph-mon[97044]: pgmap v13931: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:37 smithi110 ceph-mon[97633]: pgmap v13931: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:37.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:37.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:37.322860+0000) 2022-09-10T11:44:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:37.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:37.801753+0000) 2022-09-10T11:44:38.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:38.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:38.323166+0000) 2022-09-10T11:44:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:38.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:38.802027+0000) 2022-09-10T11:44:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:39 smithi104 ceph-mon[97044]: pgmap v13932: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:39 smithi110 ceph-mon[97633]: pgmap v13932: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:39.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:39.323366+0000) 2022-09-10T11:44:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:39.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:39.802330+0000) 2022-09-10T11:44:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:40.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:40.323565+0000) 2022-09-10T11:44:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:40.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:40.802612+0000) 2022-09-10T11:44:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:41 smithi104 ceph-mon[97044]: pgmap v13933: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:41.667 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:41 smithi110 ceph-mon[97633]: pgmap v13933: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:41.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:41.323803+0000) 2022-09-10T11:44:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:41.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:41.802894+0000) 2022-09-10T11:44:42.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:42.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:42.324088+0000) 2022-09-10T11:44:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:42.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:42.803084+0000) 2022-09-10T11:44:43.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:44:42.763+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:44:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:43 smithi104 ceph-mon[97044]: pgmap v13934: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:43 smithi110 ceph-mon[97633]: pgmap v13934: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:43.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:43.324311+0000) 2022-09-10T11:44:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:43.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:43.803330+0000) 2022-09-10T11:44:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:44.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:44.324581+0000) 2022-09-10T11:44:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:44 smithi104 ceph-mon[97044]: pgmap v13935: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:44.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:44.803622+0000) 2022-09-10T11:44:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:44 smithi110 ceph-mon[97633]: pgmap v13935: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:45.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:45.324794+0000) 2022-09-10T11:44:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:44:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:44:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:44:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:44:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:45.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:45.803850+0000) 2022-09-10T11:44:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:44:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:44:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:44:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:44:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:46.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:46.325123+0000) 2022-09-10T11:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:46 smithi104 ceph-mon[97044]: pgmap v13936: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:47.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:46.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:46.803979+0000) 2022-09-10T11:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:46 smithi110 ceph-mon[97633]: pgmap v13936: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:47.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:47.325308+0000) 2022-09-10T11:44:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:47.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:47.804160+0000) 2022-09-10T11:44:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:48.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:48.325567+0000) 2022-09-10T11:44:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:48.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:48.804324+0000) 2022-09-10T11:44:49.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:49 smithi104 ceph-mon[97044]: pgmap v13937: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:44:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:49 smithi110 ceph-mon[97633]: pgmap v13937: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:44:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:49.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:49.325845+0000) 2022-09-10T11:44:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:49.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:49.804495+0000) 2022-09-10T11:44:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:50.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:50.326121+0000) 2022-09-10T11:44:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:50.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:50.804689+0000) 2022-09-10T11:44:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:51 smithi104 ceph-mon[97044]: pgmap v13938: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:51 smithi110 ceph-mon[97633]: pgmap v13938: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:51.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:51.326378+0000) 2022-09-10T11:44:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:51.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:51.804843+0000) 2022-09-10T11:44:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:52.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:52.326588+0000) 2022-09-10T11:44:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:52.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:52.805080+0000) 2022-09-10T11:44:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:53 smithi104 ceph-mon[97044]: pgmap v13939: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:44:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:53 smithi110 ceph-mon[97633]: pgmap v13939: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:44:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:53.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:53.326750+0000) 2022-09-10T11:44:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:53.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:53.805242+0000) 2022-09-10T11:44:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:54.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:54.326962+0000) 2022-09-10T11:44:55.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:54 smithi104 ceph-mon[97044]: pgmap v13940: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:44:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:54.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:54.805447+0000) 2022-09-10T11:44:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:54 smithi110 ceph-mon[97633]: pgmap v13940: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:44:55.819 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:55.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:55.327159+0000) 2022-09-10T11:44:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:55.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:55.805618+0000) 2022-09-10T11:44:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:56.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:56.327424+0000) 2022-09-10T11:44:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:56.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:56.805854+0000) 2022-09-10T11:44:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:57 smithi104 ceph-mon[97044]: pgmap v13941: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:57 smithi110 ceph-mon[97633]: pgmap v13941: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:44:57.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:57.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:57.327651+0000) 2022-09-10T11:44:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:57.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:57.806145+0000) 2022-09-10T11:44:58.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:44:57.775+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:44:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:58.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:58.327832+0000) 2022-09-10T11:44:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:58.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:58.806377+0000) 2022-09-10T11:44:59.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:44:59 smithi104 ceph-mon[97044]: pgmap v13942: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:59.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:44:59 smithi110 ceph-mon[97633]: pgmap v13942: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:44:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:44:59.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:59.328040+0000) 2022-09-10T11:45:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:44:59.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:44:59.806569+0000) 2022-09-10T11:45:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:00.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:00.328243+0000) 2022-09-10T11:45:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:00.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:00.806802+0000) 2022-09-10T11:45:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:01 smithi104 ceph-mon[97044]: pgmap v13943: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:01 smithi110 ceph-mon[97633]: pgmap v13943: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:01.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:01.328390+0000) 2022-09-10T11:45:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:01.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:01.807023+0000) 2022-09-10T11:45:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:02.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:02.328604+0000) 2022-09-10T11:45:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:02.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:02.807196+0000) 2022-09-10T11:45:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:03 smithi104 ceph-mon[97044]: pgmap v13944: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:03 smithi110 ceph-mon[97633]: pgmap v13944: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:03.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:03.328854+0000) 2022-09-10T11:45:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:03.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:03.807371+0000) 2022-09-10T11:45:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:04.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:04.329204+0000) 2022-09-10T11:45:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:04 smithi104 ceph-mon[97044]: pgmap v13945: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:04.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:04.807573+0000) 2022-09-10T11:45:05.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:04 smithi110 ceph-mon[97633]: pgmap v13945: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:05.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:05.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:05.329364+0000) 2022-09-10T11:45:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:05.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:05.807805+0000) 2022-09-10T11:45:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:06.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:06.329616+0000) 2022-09-10T11:45:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:06.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:06.808037+0000) 2022-09-10T11:45:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:07 smithi104 ceph-mon[97044]: pgmap v13946: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:45:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:07 smithi110 ceph-mon[97633]: pgmap v13946: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:45:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:07.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:07.329809+0000) 2022-09-10T11:45:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:07.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:07.808256+0000) 2022-09-10T11:45:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:08.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:08.330041+0000) 2022-09-10T11:45:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:08.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:08.808469+0000) 2022-09-10T11:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:09 smithi104 ceph-mon[97044]: pgmap v13947: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:45:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:09 smithi104 ceph-mon[97044]: osdmap e952: 12 total, 12 up, 12 in 2022-09-10T11:45:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:09 smithi110 ceph-mon[97633]: pgmap v13947: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:45:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:09 smithi110 ceph-mon[97633]: osdmap e952: 12 total, 12 up, 12 in 2022-09-10T11:45:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:09.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:09.330224+0000) 2022-09-10T11:45:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:09.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:09.808662+0000) 2022-09-10T11:45:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:10.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:10.330431+0000) 2022-09-10T11:45:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:10.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:10.808849+0000) 2022-09-10T11:45:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:11 smithi104 ceph-mon[97044]: pgmap v13949: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:45:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:11 smithi110 ceph-mon[97633]: pgmap v13949: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:45:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:11.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:11.330649+0000) 2022-09-10T11:45:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:11.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:11.809058+0000) 2022-09-10T11:45:12.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:12.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:12.330874+0000) 2022-09-10T11:45:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:12.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:12.809242+0000) 2022-09-10T11:45:13.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:45:12.780+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:45:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:13 smithi104 ceph-mon[97044]: pgmap v13950: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:45:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:13 smithi110 ceph-mon[97633]: pgmap v13950: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:45:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:13.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:13.331141+0000) 2022-09-10T11:45:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:13.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:13.809387+0000) 2022-09-10T11:45:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:14.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:14.331361+0000) 2022-09-10T11:45:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:14 smithi104 ceph-mon[97044]: osdmap e953: 12 total, 12 up, 12 in 2022-09-10T11:45:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:14 smithi104 ceph-mon[97044]: pgmap v13952: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:45:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:14.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:14.809571+0000) 2022-09-10T11:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:14 smithi110 ceph-mon[97633]: osdmap e953: 12 total, 12 up, 12 in 2022-09-10T11:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:14 smithi110 ceph-mon[97633]: pgmap v13952: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:45:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:15.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:15.331567+0000) 2022-09-10T11:45:16.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:15.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:15.809803+0000) 2022-09-10T11:45:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:16.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:16.331767+0000) 2022-09-10T11:45:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:16.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:16.810046+0000) 2022-09-10T11:45:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:17 smithi104 ceph-mon[97044]: pgmap v13953: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:45:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:17 smithi110 ceph-mon[97633]: pgmap v13953: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:45:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:17.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:17.332028+0000) 2022-09-10T11:45:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:17.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:17.810254+0000) 2022-09-10T11:45:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:18.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:18.332274+0000) 2022-09-10T11:45:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:18.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:18.810390+0000) 2022-09-10T11:45:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:19 smithi104 ceph-mon[97044]: pgmap v13954: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:45:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:19 smithi110 ceph-mon[97633]: pgmap v13954: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:45:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:19.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:19.332511+0000) 2022-09-10T11:45:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:19.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:19.810536+0000) 2022-09-10T11:45:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:20.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:20.332694+0000) 2022-09-10T11:45:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:20.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:20.810723+0000) 2022-09-10T11:45:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:21 smithi104 ceph-mon[97044]: pgmap v13955: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:45:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:21 smithi110 ceph-mon[97633]: pgmap v13955: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:45:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:21.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:21.332914+0000) 2022-09-10T11:45:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:21.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:21.810948+0000) 2022-09-10T11:45:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:22.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:22.333137+0000) 2022-09-10T11:45:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:22.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:22.811134+0000) 2022-09-10T11:45:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:23 smithi104 ceph-mon[97044]: pgmap v13956: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:45:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:23 smithi110 ceph-mon[97633]: pgmap v13956: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:45:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:23.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:23.333360+0000) 2022-09-10T11:45:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:23.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:23.811348+0000) 2022-09-10T11:45:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:24.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:24.333547+0000) 2022-09-10T11:45:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:24 smithi104 ceph-mon[97044]: pgmap v13957: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:45:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:24.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:24.811563+0000) 2022-09-10T11:45:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:24 smithi110 ceph-mon[97633]: pgmap v13957: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:45:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:25.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:25.333759+0000) 2022-09-10T11:45:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:25.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:25.811782+0000) 2022-09-10T11:45:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:26.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:26.333995+0000) 2022-09-10T11:45:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:26.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:26.812012+0000) 2022-09-10T11:45:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:27 smithi104 ceph-mon[97044]: pgmap v13958: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:45:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:27 smithi110 ceph-mon[97633]: pgmap v13958: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:45:27.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:27.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:27.334223+0000) 2022-09-10T11:45:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:27.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:27.812236+0000) 2022-09-10T11:45:28.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:45:27.791+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:45:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:28.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:28.334397+0000) 2022-09-10T11:45:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:28.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:28.812435+0000) 2022-09-10T11:45:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:29 smithi104 ceph-mon[97044]: pgmap v13959: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:29 smithi110 ceph-mon[97633]: pgmap v13959: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:29.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:29.334574+0000) 2022-09-10T11:45:30.315 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:29.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:29.812607+0000) 2022-09-10T11:45:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:30.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:30.334821+0000) 2022-09-10T11:45:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:30.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:30.812844+0000) 2022-09-10T11:45:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:31 smithi104 ceph-mon[97044]: pgmap v13960: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:31 smithi110 ceph-mon[97633]: pgmap v13960: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:31.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:31.335028+0000) 2022-09-10T11:45:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:31.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:31.813046+0000) 2022-09-10T11:45:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:32.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:32.335228+0000) 2022-09-10T11:45:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:32.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:32.813257+0000) 2022-09-10T11:45:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:33 smithi104 ceph-mon[97044]: pgmap v13961: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:33.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:33 smithi110 ceph-mon[97633]: pgmap v13961: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:33.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:33.335385+0000) 2022-09-10T11:45:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:33.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:33.813385+0000) 2022-09-10T11:45:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:34.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:34.335621+0000) 2022-09-10T11:45:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:34 smithi104 ceph-mon[97044]: pgmap v13962: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:34.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:34.813511+0000) 2022-09-10T11:45:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:34 smithi110 ceph-mon[97633]: pgmap v13962: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:35.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:35.335828+0000) 2022-09-10T11:45:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:35.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:35.813685+0000) 2022-09-10T11:45:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:36.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:36.336043+0000) 2022-09-10T11:45:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:36.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:36.813931+0000) 2022-09-10T11:45:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:37 smithi104 ceph-mon[97044]: pgmap v13963: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:37 smithi110 ceph-mon[97633]: pgmap v13963: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:37.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:37.336208+0000) 2022-09-10T11:45:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:37.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:37.814145+0000) 2022-09-10T11:45:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:38.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:38.336427+0000) 2022-09-10T11:45:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:38.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:38.814341+0000) 2022-09-10T11:45:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:39 smithi104 ceph-mon[97044]: pgmap v13964: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:39 smithi110 ceph-mon[97633]: pgmap v13964: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:39.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:39.336695+0000) 2022-09-10T11:45:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:39.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:39.814556+0000) 2022-09-10T11:45:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:40.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:40.336955+0000) 2022-09-10T11:45:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:40.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:40.814800+0000) 2022-09-10T11:45:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:41 smithi104 ceph-mon[97044]: pgmap v13965: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:41 smithi110 ceph-mon[97633]: pgmap v13965: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:41.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:41.337191+0000) 2022-09-10T11:45:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:41.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:41.815047+0000) 2022-09-10T11:45:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:42.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:42.337494+0000) 2022-09-10T11:45:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:42.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:42.815261+0000) 2022-09-10T11:45:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:45:42.806+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:45:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:43 smithi104 ceph-mon[97044]: pgmap v13966: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:43 smithi110 ceph-mon[97633]: pgmap v13966: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:43.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:43.337776+0000) 2022-09-10T11:45:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:43.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:43.815476+0000) 2022-09-10T11:45:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:44.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:44.337968+0000) 2022-09-10T11:45:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:44 smithi104 ceph-mon[97044]: pgmap v13967: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:44.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:44.815681+0000) 2022-09-10T11:45:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:44 smithi110 ceph-mon[97633]: pgmap v13967: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:45.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:45.338226+0000) 2022-09-10T11:45:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:45:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:45:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:45:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:45 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:45:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:45.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:45.815900+0000) 2022-09-10T11:45:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:45:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:45:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:45:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:45 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:45:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:46.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:46.338458+0000) 2022-09-10T11:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:47 smithi104 ceph-mon[97044]: pgmap v13968: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:46.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:46.816044+0000) 2022-09-10T11:45:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:47 smithi110 ceph-mon[97633]: pgmap v13968: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:47.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:47.338692+0000) 2022-09-10T11:45:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:47.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:47.816239+0000) 2022-09-10T11:45:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:48.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:48.338996+0000) 2022-09-10T11:45:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:48.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:48.816422+0000) 2022-09-10T11:45:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:49 smithi104 ceph-mon[97044]: pgmap v13969: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:45:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:49 smithi110 ceph-mon[97633]: pgmap v13969: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:45:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:49.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:49.339234+0000) 2022-09-10T11:45:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:49.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:49.816599+0000) 2022-09-10T11:45:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:50.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:50.339511+0000) 2022-09-10T11:45:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:50.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:50.816845+0000) 2022-09-10T11:45:51.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:51 smithi104 ceph-mon[97044]: pgmap v13970: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:51 smithi110 ceph-mon[97633]: pgmap v13970: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:51.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:51.339688+0000) 2022-09-10T11:45:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:51.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:51.817130+0000) 2022-09-10T11:45:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:52.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:52.339917+0000) 2022-09-10T11:45:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:52.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:52.817301+0000) 2022-09-10T11:45:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:53 smithi104 ceph-mon[97044]: pgmap v13971: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:45:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:53 smithi110 ceph-mon[97633]: pgmap v13971: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:45:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:53.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:53.340070+0000) 2022-09-10T11:45:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:53.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:53.817465+0000) 2022-09-10T11:45:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:54.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:54.340246+0000) 2022-09-10T11:45:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:54 smithi104 ceph-mon[97044]: pgmap v13972: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:45:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:54.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:54.817674+0000) 2022-09-10T11:45:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:54 smithi110 ceph-mon[97633]: pgmap v13972: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:45:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:55.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:55.340442+0000) 2022-09-10T11:45:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:55.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:55.817909+0000) 2022-09-10T11:45:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:56.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:56.340640+0000) 2022-09-10T11:45:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:56.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:56.818200+0000) 2022-09-10T11:45:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:57 smithi104 ceph-mon[97044]: pgmap v13973: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:57 smithi110 ceph-mon[97633]: pgmap v13973: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:45:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:57.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:57.340857+0000) 2022-09-10T11:45:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:45:57.821+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:45:58.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:57.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:57.818420+0000) 2022-09-10T11:45:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:58.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:58.341070+0000) 2022-09-10T11:45:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:58.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:58.818634+0000) 2022-09-10T11:45:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:45:59 smithi104 ceph-mon[97044]: pgmap v13974: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:45:59 smithi110 ceph-mon[97633]: pgmap v13974: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:45:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:45:59.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:59.341322+0000) 2022-09-10T11:46:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:45:59.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:45:59.818780+0000) 2022-09-10T11:46:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:00.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:00.341453+0000) 2022-09-10T11:46:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:00.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:00.818997+0000) 2022-09-10T11:46:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:01 smithi104 ceph-mon[97044]: pgmap v13975: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:01 smithi110 ceph-mon[97633]: pgmap v13975: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:01.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:01.341590+0000) 2022-09-10T11:46:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:01.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:01.819223+0000) 2022-09-10T11:46:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:02.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:02.341830+0000) 2022-09-10T11:46:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:02.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:02.819434+0000) 2022-09-10T11:46:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:03 smithi104 ceph-mon[97044]: pgmap v13976: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:03 smithi110 ceph-mon[97633]: pgmap v13976: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:03.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:03.342056+0000) 2022-09-10T11:46:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:03.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:03.819607+0000) 2022-09-10T11:46:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:04.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:04.342427+0000) 2022-09-10T11:46:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:04 smithi104 ceph-mon[97044]: pgmap v13977: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:05.432 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:04 smithi110 ceph-mon[97633]: pgmap v13977: 129 pgs: 129 active+clean; 9.2 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:05.433 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:04.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:04.819767+0000) 2022-09-10T11:46:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:05.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:05.342576+0000) 2022-09-10T11:46:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:05.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:05.820003+0000) 2022-09-10T11:46:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:06.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:06.342743+0000) 2022-09-10T11:46:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:06.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:06.820212+0000) 2022-09-10T11:46:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:07 smithi104 ceph-mon[97044]: pgmap v13978: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:46:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:07 smithi110 ceph-mon[97633]: pgmap v13978: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:46:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:07.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:07.342970+0000) 2022-09-10T11:46:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:07.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:07.820355+0000) 2022-09-10T11:46:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:08.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:08.343170+0000) 2022-09-10T11:46:09.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:08.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:08.820542+0000) 2022-09-10T11:46:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:09 smithi104 ceph-mon[97044]: pgmap v13979: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:46:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:09 smithi104 ceph-mon[97044]: osdmap e954: 12 total, 12 up, 12 in 2022-09-10T11:46:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:09 smithi110 ceph-mon[97633]: pgmap v13979: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:46:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:09 smithi110 ceph-mon[97633]: osdmap e954: 12 total, 12 up, 12 in 2022-09-10T11:46:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:09.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:09.343412+0000) 2022-09-10T11:46:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:09.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:09.820802+0000) 2022-09-10T11:46:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:10.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:10.343605+0000) 2022-09-10T11:46:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:10.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:10.821005+0000) 2022-09-10T11:46:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:11 smithi104 ceph-mon[97044]: pgmap v13981: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:46:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:11 smithi110 ceph-mon[97633]: pgmap v13981: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:46:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:11.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:11.343861+0000) 2022-09-10T11:46:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:11.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:11.821250+0000) 2022-09-10T11:46:12.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:12.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:12.344111+0000) 2022-09-10T11:46:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:12.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:12.821422+0000) 2022-09-10T11:46:13.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:46:12.837+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:46:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:13 smithi104 ceph-mon[97044]: pgmap v13982: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:46:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:13 smithi110 ceph-mon[97633]: pgmap v13982: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:46:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:13.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:13.344335+0000) 2022-09-10T11:46:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:13.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:13.821565+0000) 2022-09-10T11:46:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:14.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:14.344559+0000) 2022-09-10T11:46:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:14 smithi104 ceph-mon[97044]: osdmap e955: 12 total, 12 up, 12 in 2022-09-10T11:46:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:14 smithi104 ceph-mon[97044]: pgmap v13984: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:46:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:14.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:14.821727+0000) 2022-09-10T11:46:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:14 smithi110 ceph-mon[97633]: osdmap e955: 12 total, 12 up, 12 in 2022-09-10T11:46:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:14 smithi110 ceph-mon[97633]: pgmap v13984: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:46:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:15.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:15.344806+0000) 2022-09-10T11:46:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:15.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:15.821871+0000) 2022-09-10T11:46:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:16.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:16.345035+0000) 2022-09-10T11:46:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:16.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:16.822164+0000) 2022-09-10T11:46:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:17 smithi104 ceph-mon[97044]: pgmap v13985: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:46:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:17 smithi110 ceph-mon[97633]: pgmap v13985: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:46:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:17.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:17.345217+0000) 2022-09-10T11:46:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:17.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:17.822384+0000) 2022-09-10T11:46:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:18.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:18.345396+0000) 2022-09-10T11:46:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:18.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:18.822612+0000) 2022-09-10T11:46:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:19 smithi104 ceph-mon[97044]: pgmap v13986: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:46:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:19 smithi110 ceph-mon[97633]: pgmap v13986: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:46:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:19.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:19.345539+0000) 2022-09-10T11:46:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:19.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:19.822804+0000) 2022-09-10T11:46:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:20.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:20.345768+0000) 2022-09-10T11:46:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:20.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:20.822982+0000) 2022-09-10T11:46:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:21 smithi104 ceph-mon[97044]: pgmap v13987: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:46:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:21 smithi110 ceph-mon[97633]: pgmap v13987: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:46:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:21.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:21.345984+0000) 2022-09-10T11:46:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:21.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:21.823198+0000) 2022-09-10T11:46:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:22.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:22.346134+0000) 2022-09-10T11:46:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:22.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:22.823335+0000) 2022-09-10T11:46:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:23 smithi104 ceph-mon[97044]: pgmap v13988: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:46:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:23 smithi110 ceph-mon[97633]: pgmap v13988: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:46:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:23.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:23.346361+0000) 2022-09-10T11:46:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:23.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:23.823579+0000) 2022-09-10T11:46:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:24.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:24.346566+0000) 2022-09-10T11:46:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:25 smithi104 ceph-mon[97044]: pgmap v13989: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:46:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:24.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:24.823803+0000) 2022-09-10T11:46:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:25 smithi110 ceph-mon[97633]: pgmap v13989: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:46:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:25.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:25.346775+0000) 2022-09-10T11:46:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:25.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:25.823990+0000) 2022-09-10T11:46:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:26.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:26.347007+0000) 2022-09-10T11:46:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:26.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:26.824222+0000) 2022-09-10T11:46:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:27 smithi104 ceph-mon[97044]: pgmap v13990: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:46:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:27 smithi110 ceph-mon[97633]: pgmap v13990: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:46:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:27.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:27.347225+0000) 2022-09-10T11:46:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:46:27.852+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:46:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:27.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:27.824419+0000) 2022-09-10T11:46:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:28.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:28.347485+0000) 2022-09-10T11:46:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:28.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:28.824631+0000) 2022-09-10T11:46:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:29 smithi104 ceph-mon[97044]: pgmap v13991: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:29 smithi110 ceph-mon[97633]: pgmap v13991: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:29.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:29.347673+0000) 2022-09-10T11:46:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:29.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:29.824845+0000) 2022-09-10T11:46:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:30.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:30.347801+0000) 2022-09-10T11:46:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:30.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:30.825073+0000) 2022-09-10T11:46:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:31 smithi104 ceph-mon[97044]: pgmap v13992: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:31 smithi110 ceph-mon[97633]: pgmap v13992: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:31.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:31.348051+0000) 2022-09-10T11:46:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:31.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:31.825308+0000) 2022-09-10T11:46:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:32.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:32.348151+0000) 2022-09-10T11:46:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:32.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:32.825530+0000) 2022-09-10T11:46:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:33 smithi104 ceph-mon[97044]: pgmap v13993: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:33 smithi110 ceph-mon[97633]: pgmap v13993: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:33.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:33.348376+0000) 2022-09-10T11:46:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:33.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:33.825784+0000) 2022-09-10T11:46:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:34.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:34.348571+0000) 2022-09-10T11:46:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:35 smithi104 ceph-mon[97044]: pgmap v13994: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:34.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:34.826045+0000) 2022-09-10T11:46:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:35 smithi110 ceph-mon[97633]: pgmap v13994: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:35.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:35.348794+0000) 2022-09-10T11:46:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:35.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:35.826250+0000) 2022-09-10T11:46:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:36.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:36.349045+0000) 2022-09-10T11:46:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:36.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:36.826435+0000) 2022-09-10T11:46:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:37 smithi104 ceph-mon[97044]: pgmap v13995: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:37 smithi110 ceph-mon[97633]: pgmap v13995: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:37.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:37.349182+0000) 2022-09-10T11:46:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:37.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:37.826688+0000) 2022-09-10T11:46:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:38.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:38.349405+0000) 2022-09-10T11:46:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:38.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:38.826934+0000) 2022-09-10T11:46:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:39 smithi104 ceph-mon[97044]: pgmap v13996: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:39 smithi110 ceph-mon[97633]: pgmap v13996: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:39.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:39.349583+0000) 2022-09-10T11:46:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:39.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:39.827112+0000) 2022-09-10T11:46:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:40.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:40.349784+0000) 2022-09-10T11:46:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:40.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:40.827316+0000) 2022-09-10T11:46:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:41 smithi104 ceph-mon[97044]: pgmap v13997: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T11:46:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:41 smithi110 ceph-mon[97633]: pgmap v13997: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T11:46:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:41.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:41.349956+0000) 2022-09-10T11:46:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:41.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:41.827570+0000) 2022-09-10T11:46:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:42.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:42.350172+0000) 2022-09-10T11:46:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:42.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:42.827788+0000) 2022-09-10T11:46:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:46:42.861+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:46:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:43 smithi104 ceph-mon[97044]: pgmap v13998: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:43 smithi110 ceph-mon[97633]: pgmap v13998: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:43.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:43.350330+0000) 2022-09-10T11:46:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:43.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:43.828045+0000) 2022-09-10T11:46:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:44.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:44.350543+0000) 2022-09-10T11:46:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:45 smithi104 ceph-mon[97044]: pgmap v13999: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:44.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:44.828194+0000) 2022-09-10T11:46:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:45 smithi110 ceph-mon[97633]: pgmap v13999: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:45.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:45.350699+0000) 2022-09-10T11:46:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:46:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:46:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:46:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:46:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:45.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:45.828397+0000) 2022-09-10T11:46:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:46:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:46:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:46:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:46:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:46.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:46.350920+0000) 2022-09-10T11:46:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:47 smithi104 ceph-mon[97044]: pgmap v14000: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:46.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:46.828632+0000) 2022-09-10T11:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:47 smithi110 ceph-mon[97633]: pgmap v14000: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:47.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:47.351207+0000) 2022-09-10T11:46:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:47.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:47.828883+0000) 2022-09-10T11:46:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:48.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:48.351429+0000) 2022-09-10T11:46:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:48.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:48.829190+0000) 2022-09-10T11:46:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:49 smithi104 ceph-mon[97044]: pgmap v14001: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:49 smithi110 ceph-mon[97633]: pgmap v14001: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:46:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:49.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:49.351621+0000) 2022-09-10T11:46:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:49.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:49.829398+0000) 2022-09-10T11:46:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:50.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:50.351856+0000) 2022-09-10T11:46:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:50.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:50.829543+0000) 2022-09-10T11:46:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:51 smithi104 ceph-mon[97044]: pgmap v14002: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:51 smithi110 ceph-mon[97633]: pgmap v14002: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:51.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:51.352016+0000) 2022-09-10T11:46:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:51.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:51.829796+0000) 2022-09-10T11:46:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:52.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:52.352235+0000) 2022-09-10T11:46:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:52.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:52.830057+0000) 2022-09-10T11:46:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:53 smithi104 ceph-mon[97044]: pgmap v14003: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:53 smithi110 ceph-mon[97633]: pgmap v14003: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:53.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:53.352439+0000) 2022-09-10T11:46:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:53.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:53.830285+0000) 2022-09-10T11:46:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:54.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:54.352677+0000) 2022-09-10T11:46:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:55 smithi104 ceph-mon[97044]: pgmap v14004: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:54.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:54.830519+0000) 2022-09-10T11:46:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:55 smithi110 ceph-mon[97633]: pgmap v14004: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:55.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:55.352883+0000) 2022-09-10T11:46:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:55.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:55.830775+0000) 2022-09-10T11:46:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:56.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:56.353136+0000) 2022-09-10T11:46:57.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:56.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:56.830953+0000) 2022-09-10T11:46:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:57 smithi104 ceph-mon[97044]: pgmap v14005: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:57.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:57 smithi110 ceph-mon[97633]: pgmap v14005: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:46:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:57.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:57.353314+0000) 2022-09-10T11:46:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:57.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:57.831156+0000) 2022-09-10T11:46:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:46:57.869+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:46:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:58.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:58.353568+0000) 2022-09-10T11:46:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:58.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:58.831377+0000) 2022-09-10T11:46:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:46:59 smithi104 ceph-mon[97044]: pgmap v14006: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:46:59 smithi110 ceph-mon[97633]: pgmap v14006: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:46:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:46:59.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:59.353807+0000) 2022-09-10T11:47:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:46:59.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:46:59.831595+0000) 2022-09-10T11:47:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:00.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:00.353969+0000) 2022-09-10T11:47:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:00.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:00.831871+0000) 2022-09-10T11:47:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:01 smithi104 ceph-mon[97044]: pgmap v14007: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:01 smithi110 ceph-mon[97633]: pgmap v14007: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:01.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:01.354190+0000) 2022-09-10T11:47:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:01.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:01.832177+0000) 2022-09-10T11:47:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:02.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:02.354328+0000) 2022-09-10T11:47:03.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:02.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:02.832464+0000) 2022-09-10T11:47:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:03 smithi104 ceph-mon[97044]: pgmap v14008: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:03.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:03 smithi110 ceph-mon[97633]: pgmap v14008: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:03.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:03.354561+0000) 2022-09-10T11:47:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:03.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:03.832725+0000) 2022-09-10T11:47:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:04.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:04.354814+0000) 2022-09-10T11:47:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:05 smithi104 ceph-mon[97044]: pgmap v14009: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:04.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:04.832934+0000) 2022-09-10T11:47:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:05 smithi110 ceph-mon[97633]: pgmap v14009: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:05.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:05.354986+0000) 2022-09-10T11:47:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:05.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:05.833171+0000) 2022-09-10T11:47:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:06.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:06.355217+0000) 2022-09-10T11:47:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:06.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:06.833360+0000) 2022-09-10T11:47:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:07 smithi104 ceph-mon[97044]: pgmap v14010: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:47:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:07 smithi110 ceph-mon[97633]: pgmap v14010: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:47:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:07.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:07.355457+0000) 2022-09-10T11:47:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:07.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:07.833529+0000) 2022-09-10T11:47:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:08.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:08.355606+0000) 2022-09-10T11:47:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:08.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:08.833705+0000) 2022-09-10T11:47:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:09 smithi104 ceph-mon[97044]: pgmap v14011: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:47:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:09 smithi104 ceph-mon[97044]: osdmap e956: 12 total, 12 up, 12 in 2022-09-10T11:47:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:09 smithi110 ceph-mon[97633]: pgmap v14011: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:47:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:09 smithi110 ceph-mon[97633]: osdmap e956: 12 total, 12 up, 12 in 2022-09-10T11:47:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:09.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:09.355857+0000) 2022-09-10T11:47:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:09.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:09.833949+0000) 2022-09-10T11:47:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:10.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:10.356079+0000) 2022-09-10T11:47:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:10.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:10.834199+0000) 2022-09-10T11:47:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:11 smithi104 ceph-mon[97044]: pgmap v14013: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:47:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:11 smithi110 ceph-mon[97633]: pgmap v14013: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:47:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:11.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:11.356297+0000) 2022-09-10T11:47:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:11.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:11.834448+0000) 2022-09-10T11:47:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:12.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:12.356517+0000) 2022-09-10T11:47:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:12.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:12.834714+0000) 2022-09-10T11:47:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:47:12.885+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:47:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:13 smithi104 ceph-mon[97044]: pgmap v14014: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:47:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:13 smithi110 ceph-mon[97633]: pgmap v14014: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:47:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:13.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:13.356745+0000) 2022-09-10T11:47:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:13.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:13.834897+0000) 2022-09-10T11:47:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:14.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:14.356974+0000) 2022-09-10T11:47:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:15 smithi104 ceph-mon[97044]: osdmap e957: 12 total, 12 up, 12 in 2022-09-10T11:47:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:15 smithi104 ceph-mon[97044]: pgmap v14016: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:47:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:14.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:14.835176+0000) 2022-09-10T11:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:15 smithi110 ceph-mon[97633]: osdmap e957: 12 total, 12 up, 12 in 2022-09-10T11:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:15 smithi110 ceph-mon[97633]: pgmap v14016: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:47:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:15.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:15.357175+0000) 2022-09-10T11:47:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:15.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:15.835335+0000) 2022-09-10T11:47:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:16.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:16.357447+0000) 2022-09-10T11:47:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:16.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:16.835524+0000) 2022-09-10T11:47:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:17 smithi104 ceph-mon[97044]: pgmap v14017: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:47:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:17 smithi110 ceph-mon[97633]: pgmap v14017: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:47:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:17.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:17.357637+0000) 2022-09-10T11:47:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:17.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:17.835731+0000) 2022-09-10T11:47:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:18.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:18.357865+0000) 2022-09-10T11:47:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:18.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:18.835989+0000) 2022-09-10T11:47:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:19 smithi104 ceph-mon[97044]: pgmap v14018: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:47:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:19 smithi110 ceph-mon[97633]: pgmap v14018: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:47:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:19.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:19.358175+0000) 2022-09-10T11:47:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:19.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:19.836186+0000) 2022-09-10T11:47:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:20.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:20.358357+0000) 2022-09-10T11:47:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:20.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:20.836413+0000) 2022-09-10T11:47:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:21 smithi104 ceph-mon[97044]: pgmap v14019: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:47:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:21 smithi110 ceph-mon[97633]: pgmap v14019: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:47:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:21.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:21.358579+0000) 2022-09-10T11:47:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:21.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:21.836588+0000) 2022-09-10T11:47:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:22.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:22.358885+0000) 2022-09-10T11:47:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:22.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:22.836796+0000) 2022-09-10T11:47:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:23 smithi104 ceph-mon[97044]: pgmap v14020: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:47:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:23 smithi110 ceph-mon[97633]: pgmap v14020: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:47:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:23.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:23.359055+0000) 2022-09-10T11:47:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:23.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:23.837018+0000) 2022-09-10T11:47:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:24.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:24.359240+0000) 2022-09-10T11:47:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:25 smithi104 ceph-mon[97044]: pgmap v14021: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:47:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:24.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:24.837251+0000) 2022-09-10T11:47:25.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:25 smithi110 ceph-mon[97633]: pgmap v14021: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:47:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:25.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:25.359472+0000) 2022-09-10T11:47:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:25.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:25.837471+0000) 2022-09-10T11:47:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:26.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:26.359721+0000) 2022-09-10T11:47:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:26.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:26.837726+0000) 2022-09-10T11:47:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:27 smithi104 ceph-mon[97044]: pgmap v14022: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:47:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:27 smithi110 ceph-mon[97633]: pgmap v14022: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:47:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:27.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:27.359944+0000) 2022-09-10T11:47:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:27.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:27.838014+0000) 2022-09-10T11:47:28.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:47:27.901+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:47:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:28.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:28.360205+0000) 2022-09-10T11:47:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:28.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:28.838252+0000) 2022-09-10T11:47:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:29 smithi104 ceph-mon[97044]: pgmap v14023: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:29 smithi110 ceph-mon[97633]: pgmap v14023: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:29.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:29.360443+0000) 2022-09-10T11:47:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:29.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:29.838497+0000) 2022-09-10T11:47:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:30.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:30.360708+0000) 2022-09-10T11:47:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:30.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:30.838723+0000) 2022-09-10T11:47:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:31 smithi104 ceph-mon[97044]: pgmap v14024: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:31 smithi110 ceph-mon[97633]: pgmap v14024: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:31.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:31.360968+0000) 2022-09-10T11:47:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:31.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:31.838986+0000) 2022-09-10T11:47:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:32.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:32.361231+0000) 2022-09-10T11:47:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:32.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:32.839181+0000) 2022-09-10T11:47:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:33 smithi104 ceph-mon[97044]: pgmap v14025: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:33 smithi110 ceph-mon[97633]: pgmap v14025: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:33.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:33.361453+0000) 2022-09-10T11:47:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:33.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:33.839326+0000) 2022-09-10T11:47:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:34.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:34.361712+0000) 2022-09-10T11:47:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:35 smithi104 ceph-mon[97044]: pgmap v14026: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:34.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:34.839520+0000) 2022-09-10T11:47:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:35 smithi110 ceph-mon[97633]: pgmap v14026: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:35.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:35.361995+0000) 2022-09-10T11:47:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:35.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:35.839730+0000) 2022-09-10T11:47:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:36.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:36.362147+0000) 2022-09-10T11:47:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:36.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:36.839958+0000) 2022-09-10T11:47:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:37 smithi104 ceph-mon[97044]: pgmap v14027: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:37 smithi110 ceph-mon[97633]: pgmap v14027: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:37.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:37.362365+0000) 2022-09-10T11:47:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:37.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:37.840135+0000) 2022-09-10T11:47:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:38.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:38.362556+0000) 2022-09-10T11:47:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:38.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:38.840278+0000) 2022-09-10T11:47:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:39 smithi104 ceph-mon[97044]: pgmap v14028: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:39 smithi110 ceph-mon[97633]: pgmap v14028: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:39.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:39.362762+0000) 2022-09-10T11:47:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:39.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:39.840434+0000) 2022-09-10T11:47:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:40.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:40.363025+0000) 2022-09-10T11:47:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:40.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:40.840663+0000) 2022-09-10T11:47:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:41 smithi104 ceph-mon[97044]: pgmap v14029: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:41 smithi110 ceph-mon[97633]: pgmap v14029: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:41.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:41.363255+0000) 2022-09-10T11:47:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:41.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:41.840920+0000) 2022-09-10T11:47:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:42.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:42.363544+0000) 2022-09-10T11:47:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:42.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:42.841221+0000) 2022-09-10T11:47:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:47:42.908+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:47:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:43 smithi104 ceph-mon[97044]: pgmap v14030: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:43 smithi110 ceph-mon[97633]: pgmap v14030: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:43.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:43.363781+0000) 2022-09-10T11:47:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:43.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:43.841429+0000) 2022-09-10T11:47:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:44.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:44.363970+0000) 2022-09-10T11:47:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:45 smithi104 ceph-mon[97044]: pgmap v14031: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:44.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:44.841670+0000) 2022-09-10T11:47:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:45 smithi110 ceph-mon[97633]: pgmap v14031: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:45.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:45.364196+0000) 2022-09-10T11:47:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:47:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:47:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:47:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:47:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:45.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:45.841902+0000) 2022-09-10T11:47:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:47:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:47:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:47:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:47:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:46.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:46.364404+0000) 2022-09-10T11:47:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:47 smithi104 ceph-mon[97044]: pgmap v14032: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:47.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:46.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:46.842066+0000) 2022-09-10T11:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:47 smithi110 ceph-mon[97633]: pgmap v14032: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:47.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:47.364641+0000) 2022-09-10T11:47:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:47.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:47.842252+0000) 2022-09-10T11:47:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:48.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:48.364896+0000) 2022-09-10T11:47:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:48.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:48.842479+0000) 2022-09-10T11:47:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:49 smithi104 ceph-mon[97044]: pgmap v14033: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:49 smithi110 ceph-mon[97633]: pgmap v14033: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:49.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:49.365048+0000) 2022-09-10T11:47:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:49.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:49.842716+0000) 2022-09-10T11:47:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:50.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:50.365222+0000) 2022-09-10T11:47:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:50.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:50.842983+0000) 2022-09-10T11:47:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:51 smithi104 ceph-mon[97044]: pgmap v14034: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:51 smithi110 ceph-mon[97633]: pgmap v14034: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:51.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:51.365403+0000) 2022-09-10T11:47:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:51.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:51.843142+0000) 2022-09-10T11:47:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:52.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:52.365652+0000) 2022-09-10T11:47:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:52.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:52.843327+0000) 2022-09-10T11:47:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:53 smithi104 ceph-mon[97044]: pgmap v14035: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:53 smithi110 ceph-mon[97633]: pgmap v14035: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:53.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:53.365876+0000) 2022-09-10T11:47:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:53.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:53.843551+0000) 2022-09-10T11:47:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:54.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:54.366138+0000) 2022-09-10T11:47:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:55 smithi104 ceph-mon[97044]: pgmap v14036: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:54.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:54.843741+0000) 2022-09-10T11:47:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:55 smithi110 ceph-mon[97633]: pgmap v14036: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:55.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:55.366345+0000) 2022-09-10T11:47:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:55.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:55.843891+0000) 2022-09-10T11:47:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:56.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:56.366544+0000) 2022-09-10T11:47:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:56.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:56.844124+0000) 2022-09-10T11:47:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:57 smithi104 ceph-mon[97044]: pgmap v14037: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:57 smithi110 ceph-mon[97633]: pgmap v14037: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:47:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:57.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:57.366782+0000) 2022-09-10T11:47:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:57.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:57.844256+0000) 2022-09-10T11:47:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:47:57.912+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:47:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:58.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:58.367011+0000) 2022-09-10T11:47:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:58.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:58.844485+0000) 2022-09-10T11:47:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:47:59 smithi104 ceph-mon[97044]: pgmap v14038: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:47:59 smithi110 ceph-mon[97633]: pgmap v14038: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:47:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:47:59.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:59.367298+0000) 2022-09-10T11:48:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:47:59.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:47:59.844607+0000) 2022-09-10T11:48:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:00.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:00.367501+0000) 2022-09-10T11:48:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:00.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:00.844880+0000) 2022-09-10T11:48:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:01 smithi104 ceph-mon[97044]: pgmap v14039: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:01 smithi110 ceph-mon[97633]: pgmap v14039: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:01.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:01.367772+0000) 2022-09-10T11:48:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:01.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:01.845174+0000) 2022-09-10T11:48:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:02.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:02.367981+0000) 2022-09-10T11:48:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:02.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:02.845464+0000) 2022-09-10T11:48:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:03 smithi104 ceph-mon[97044]: pgmap v14040: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:03 smithi110 ceph-mon[97633]: pgmap v14040: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:03.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:03.368251+0000) 2022-09-10T11:48:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:03.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:03.845736+0000) 2022-09-10T11:48:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:04.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:04.368465+0000) 2022-09-10T11:48:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:05 smithi104 ceph-mon[97044]: pgmap v14041: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:04.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:04.845994+0000) 2022-09-10T11:48:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:05 smithi110 ceph-mon[97633]: pgmap v14041: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:05.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:05.368689+0000) 2022-09-10T11:48:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:05.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:05.846199+0000) 2022-09-10T11:48:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:06.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:06.368831+0000) 2022-09-10T11:48:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:06.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:06.846390+0000) 2022-09-10T11:48:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:07 smithi104 ceph-mon[97044]: pgmap v14042: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:48:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:07 smithi110 ceph-mon[97633]: pgmap v14042: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:48:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:07.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:07.369076+0000) 2022-09-10T11:48:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:07.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:07.846646+0000) 2022-09-10T11:48:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:08.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:08.369342+0000) 2022-09-10T11:48:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:08.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:08.846901+0000) 2022-09-10T11:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:09 smithi104 ceph-mon[97044]: pgmap v14043: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:09 smithi104 ceph-mon[97044]: osdmap e958: 12 total, 12 up, 12 in 2022-09-10T11:48:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:09 smithi110 ceph-mon[97633]: pgmap v14043: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:48:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:09 smithi110 ceph-mon[97633]: osdmap e958: 12 total, 12 up, 12 in 2022-09-10T11:48:09.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:09.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:09.369546+0000) 2022-09-10T11:48:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:09.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:09.847080+0000) 2022-09-10T11:48:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:10.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:10.369737+0000) 2022-09-10T11:48:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:10.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:10.847224+0000) 2022-09-10T11:48:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:11 smithi104 ceph-mon[97044]: pgmap v14045: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:48:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:11 smithi110 ceph-mon[97633]: pgmap v14045: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:48:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:11.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:11.369866+0000) 2022-09-10T11:48:12.099 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:11.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:11.847350+0000) 2022-09-10T11:48:12.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:12.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:12.370027+0000) 2022-09-10T11:48:12.930 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:12.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:12.847606+0000) 2022-09-10T11:48:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:48:12.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:13 smithi104 ceph-mon[97044]: pgmap v14046: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:48:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:13 smithi110 ceph-mon[97633]: pgmap v14046: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:48:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:13.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:13.370228+0000) 2022-09-10T11:48:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:13.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:13.847889+0000) 2022-09-10T11:48:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:14.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:14.370512+0000) 2022-09-10T11:48:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:15 smithi104 ceph-mon[97044]: osdmap e959: 12 total, 12 up, 12 in 2022-09-10T11:48:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:15 smithi104 ceph-mon[97044]: pgmap v14048: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:48:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:14.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:14.848117+0000) 2022-09-10T11:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:15 smithi110 ceph-mon[97633]: osdmap e959: 12 total, 12 up, 12 in 2022-09-10T11:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:15 smithi110 ceph-mon[97633]: pgmap v14048: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:48:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:15.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:15.370691+0000) 2022-09-10T11:48:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:15.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:15.848290+0000) 2022-09-10T11:48:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:16.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:16.370945+0000) 2022-09-10T11:48:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:16.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:16.848525+0000) 2022-09-10T11:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:17 smithi104 ceph-mon[97044]: pgmap v14049: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:48:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:17 smithi110 ceph-mon[97633]: pgmap v14049: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:48:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:17.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:17.371210+0000) 2022-09-10T11:48:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:17.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:17.848739+0000) 2022-09-10T11:48:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:18.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:18.371433+0000) 2022-09-10T11:48:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:18.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:18.848996+0000) 2022-09-10T11:48:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:19 smithi104 ceph-mon[97044]: pgmap v14050: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:48:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:19 smithi110 ceph-mon[97633]: pgmap v14050: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:48:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:19.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:19.371645+0000) 2022-09-10T11:48:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:19.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:19.849201+0000) 2022-09-10T11:48:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:20.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:20.371812+0000) 2022-09-10T11:48:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:20.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:20.849434+0000) 2022-09-10T11:48:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:21 smithi104 ceph-mon[97044]: pgmap v14051: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:48:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:21 smithi110 ceph-mon[97633]: pgmap v14051: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:48:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:21.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:21.371936+0000) 2022-09-10T11:48:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:21.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:21.849646+0000) 2022-09-10T11:48:22.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:22.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:22.372176+0000) 2022-09-10T11:48:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:22.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:22.849868+0000) 2022-09-10T11:48:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:23 smithi104 ceph-mon[97044]: pgmap v14052: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:48:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:23 smithi110 ceph-mon[97633]: pgmap v14052: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:48:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:23.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:23.372408+0000) 2022-09-10T11:48:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:23.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:23.850074+0000) 2022-09-10T11:48:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:24.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:24.372691+0000) 2022-09-10T11:48:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:25 smithi104 ceph-mon[97044]: pgmap v14053: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:48:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:24.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:24.850313+0000) 2022-09-10T11:48:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:25 smithi110 ceph-mon[97633]: pgmap v14053: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:48:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:25.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:25.372916+0000) 2022-09-10T11:48:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:25.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:25.850536+0000) 2022-09-10T11:48:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:26.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:26.373053+0000) 2022-09-10T11:48:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:26.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:26.850819+0000) 2022-09-10T11:48:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:27 smithi104 ceph-mon[97044]: pgmap v14054: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:48:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:27 smithi110 ceph-mon[97633]: pgmap v14054: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:48:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:27.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:27.373324+0000) 2022-09-10T11:48:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:27.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:27.851039+0000) 2022-09-10T11:48:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:48:27.944+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:48:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:28.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:28.373568+0000) 2022-09-10T11:48:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:28.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:28.851336+0000) 2022-09-10T11:48:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:29 smithi104 ceph-mon[97044]: pgmap v14055: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:29 smithi110 ceph-mon[97633]: pgmap v14055: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:29.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:29.373763+0000) 2022-09-10T11:48:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:29.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:29.851554+0000) 2022-09-10T11:48:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:30.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:30.374023+0000) 2022-09-10T11:48:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:30.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:30.851770+0000) 2022-09-10T11:48:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:31 smithi104 ceph-mon[97044]: pgmap v14056: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:31 smithi110 ceph-mon[97633]: pgmap v14056: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:31.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:31.374247+0000) 2022-09-10T11:48:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:31.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:31.852000+0000) 2022-09-10T11:48:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:32.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:32.374474+0000) 2022-09-10T11:48:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:32.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:32.852228+0000) 2022-09-10T11:48:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:33 smithi104 ceph-mon[97044]: pgmap v14057: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:33 smithi110 ceph-mon[97633]: pgmap v14057: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:33.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:33.374726+0000) 2022-09-10T11:48:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:33.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:33.852477+0000) 2022-09-10T11:48:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:34.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:34.374972+0000) 2022-09-10T11:48:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:35 smithi104 ceph-mon[97044]: pgmap v14058: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:34.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:34.852751+0000) 2022-09-10T11:48:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:35 smithi110 ceph-mon[97633]: pgmap v14058: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:35.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:35.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:35.375192+0000) 2022-09-10T11:48:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:35.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:35.852923+0000) 2022-09-10T11:48:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:36.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:36.375414+0000) 2022-09-10T11:48:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:36.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:36.853199+0000) 2022-09-10T11:48:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:37 smithi104 ceph-mon[97044]: pgmap v14059: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:37 smithi110 ceph-mon[97633]: pgmap v14059: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:37.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:37.375624+0000) 2022-09-10T11:48:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:37.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:37.853434+0000) 2022-09-10T11:48:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:38.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:38.375804+0000) 2022-09-10T11:48:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:38.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:38.853620+0000) 2022-09-10T11:48:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:39 smithi104 ceph-mon[97044]: pgmap v14060: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:39 smithi110 ceph-mon[97633]: pgmap v14060: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:39.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:39.376034+0000) 2022-09-10T11:48:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:39.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:39.853858+0000) 2022-09-10T11:48:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:40.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:40.376269+0000) 2022-09-10T11:48:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:40.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:40.854088+0000) 2022-09-10T11:48:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:41 smithi104 ceph-mon[97044]: pgmap v14061: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:41.630 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:41 smithi110 ceph-mon[97633]: pgmap v14061: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:41.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:41.376430+0000) 2022-09-10T11:48:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:41.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:41.854363+0000) 2022-09-10T11:48:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:42.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:42.376648+0000) 2022-09-10T11:48:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:42.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:42.854537+0000) 2022-09-10T11:48:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:48:42.957+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:48:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:43 smithi104 ceph-mon[97044]: pgmap v14062: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:43 smithi110 ceph-mon[97633]: pgmap v14062: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:43.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:43.376895+0000) 2022-09-10T11:48:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:43.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:43.854744+0000) 2022-09-10T11:48:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:44.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:44.377165+0000) 2022-09-10T11:48:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:45 smithi104 ceph-mon[97044]: pgmap v14063: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:44.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:44.855006+0000) 2022-09-10T11:48:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:45 smithi110 ceph-mon[97633]: pgmap v14063: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:45.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:45.377374+0000) 2022-09-10T11:48:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:48:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:48:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:48:46.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:48:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:48:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:48:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:48:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:48:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:45.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:45.855199+0000) 2022-09-10T11:48:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:46.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:46.377638+0000) 2022-09-10T11:48:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:46.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:46.855371+0000) 2022-09-10T11:48:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:47 smithi110 ceph-mon[97633]: pgmap v14064: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:47 smithi104 ceph-mon[97044]: pgmap v14064: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:47.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:47.377854+0000) 2022-09-10T11:48:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:47.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:47.855553+0000) 2022-09-10T11:48:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:48.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:48.378087+0000) 2022-09-10T11:48:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:48.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:48.855707+0000) 2022-09-10T11:48:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:49 smithi104 ceph-mon[97044]: pgmap v14065: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:49 smithi110 ceph-mon[97633]: pgmap v14065: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:49.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:49.378296+0000) 2022-09-10T11:48:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:49.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:49.855969+0000) 2022-09-10T11:48:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:50.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:50.378517+0000) 2022-09-10T11:48:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:50.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:50.856208+0000) 2022-09-10T11:48:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:51 smithi104 ceph-mon[97044]: pgmap v14066: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:51 smithi110 ceph-mon[97633]: pgmap v14066: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:51.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:51.378718+0000) 2022-09-10T11:48:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:51.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:51.856369+0000) 2022-09-10T11:48:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:52.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:52.378950+0000) 2022-09-10T11:48:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:52.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:52.856522+0000) 2022-09-10T11:48:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:53 smithi104 ceph-mon[97044]: pgmap v14067: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:53 smithi110 ceph-mon[97633]: pgmap v14067: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:53.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:53.379168+0000) 2022-09-10T11:48:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:53.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:53.856763+0000) 2022-09-10T11:48:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:54.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:54.379389+0000) 2022-09-10T11:48:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:55 smithi104 ceph-mon[97044]: pgmap v14068: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:54.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:54.856980+0000) 2022-09-10T11:48:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:55 smithi110 ceph-mon[97633]: pgmap v14068: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:55.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:55.379605+0000) 2022-09-10T11:48:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:55.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:55.857194+0000) 2022-09-10T11:48:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:56.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:56.379862+0000) 2022-09-10T11:48:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:56.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:56.857449+0000) 2022-09-10T11:48:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:57 smithi104 ceph-mon[97044]: pgmap v14069: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:57 smithi110 ceph-mon[97633]: pgmap v14069: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:48:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:57.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:57.380157+0000) 2022-09-10T11:48:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:57.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:57.857666+0000) 2022-09-10T11:48:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:48:57.968+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:48:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:58.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:58.380381+0000) 2022-09-10T11:48:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:58.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:58.857892+0000) 2022-09-10T11:48:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:48:59 smithi104 ceph-mon[97044]: pgmap v14070: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:48:59 smithi110 ceph-mon[97633]: pgmap v14070: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:48:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:48:59.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:59.380575+0000) 2022-09-10T11:49:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:48:59.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:48:59.858151+0000) 2022-09-10T11:49:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:00.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:00.380794+0000) 2022-09-10T11:49:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:00.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:00.858322+0000) 2022-09-10T11:49:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:01 smithi110 ceph-mon[97633]: pgmap v14071: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:01 smithi104 ceph-mon[97044]: pgmap v14071: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:01.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:01.380913+0000) 2022-09-10T11:49:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:01.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:01.858502+0000) 2022-09-10T11:49:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:02.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:02.381125+0000) 2022-09-10T11:49:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:02.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:02.858735+0000) 2022-09-10T11:49:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:03 smithi110 ceph-mon[97633]: pgmap v14072: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:03 smithi104 ceph-mon[97044]: pgmap v14072: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:03.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:03.381427+0000) 2022-09-10T11:49:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:03.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:03.858958+0000) 2022-09-10T11:49:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:04.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:04.381669+0000) 2022-09-10T11:49:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:04.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:04.859150+0000) 2022-09-10T11:49:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:05 smithi110 ceph-mon[97633]: pgmap v14073: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:05 smithi104 ceph-mon[97044]: pgmap v14073: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:05.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:05.381812+0000) 2022-09-10T11:49:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:05.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:05.859385+0000) 2022-09-10T11:49:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:06.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:06.382023+0000) 2022-09-10T11:49:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:06.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:06.859594+0000) 2022-09-10T11:49:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:07 smithi104 ceph-mon[97044]: pgmap v14074: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:49:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:07 smithi110 ceph-mon[97633]: pgmap v14074: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:49:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:07.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:07.382261+0000) 2022-09-10T11:49:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:07.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:07.859728+0000) 2022-09-10T11:49:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:08.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:08.382481+0000) 2022-09-10T11:49:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:08.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:08.859867+0000) 2022-09-10T11:49:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:09 smithi110 ceph-mon[97633]: pgmap v14075: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:49:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:09 smithi110 ceph-mon[97633]: osdmap e960: 12 total, 12 up, 12 in 2022-09-10T11:49:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:09 smithi104 ceph-mon[97044]: pgmap v14075: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:49:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:09 smithi104 ceph-mon[97044]: osdmap e960: 12 total, 12 up, 12 in 2022-09-10T11:49:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:09.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:09.382692+0000) 2022-09-10T11:49:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:09.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:09.860154+0000) 2022-09-10T11:49:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:10.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:10.382975+0000) 2022-09-10T11:49:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:10.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:10.860338+0000) 2022-09-10T11:49:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:11 smithi110 ceph-mon[97633]: pgmap v14077: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:49:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:11 smithi104 ceph-mon[97044]: pgmap v14077: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:49:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:11.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:11.383256+0000) 2022-09-10T11:49:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:11.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:11.860522+0000) 2022-09-10T11:49:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:12.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:12.383486+0000) 2022-09-10T11:49:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:49:12.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:49:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:12.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:12.860770+0000) 2022-09-10T11:49:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:13 smithi110 ceph-mon[97633]: pgmap v14078: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:49:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:13 smithi104 ceph-mon[97044]: pgmap v14078: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T11:49:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:13.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:13.383722+0000) 2022-09-10T11:49:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:13.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:13.861045+0000) 2022-09-10T11:49:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:14.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:14.383978+0000) 2022-09-10T11:49:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:15 smithi104 ceph-mon[97044]: osdmap e961: 12 total, 12 up, 12 in 2022-09-10T11:49:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:15 smithi104 ceph-mon[97044]: pgmap v14080: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:49:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:14.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:14.861309+0000) 2022-09-10T11:49:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:15 smithi110 ceph-mon[97633]: osdmap e961: 12 total, 12 up, 12 in 2022-09-10T11:49:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:15 smithi110 ceph-mon[97633]: pgmap v14080: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:49:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:15.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:15.384198+0000) 2022-09-10T11:49:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:15.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:15.861533+0000) 2022-09-10T11:49:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:16.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:16.384437+0000) 2022-09-10T11:49:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:16.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:16.861752+0000) 2022-09-10T11:49:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:17 smithi110 ceph-mon[97633]: pgmap v14081: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:49:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:17 smithi104 ceph-mon[97044]: pgmap v14081: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:49:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:17.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:17.384623+0000) 2022-09-10T11:49:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:17.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:17.861939+0000) 2022-09-10T11:49:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:18.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:18.384826+0000) 2022-09-10T11:49:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:18.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:18.862164+0000) 2022-09-10T11:49:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:19 smithi110 ceph-mon[97633]: pgmap v14082: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:49:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:19 smithi104 ceph-mon[97044]: pgmap v14082: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:49:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:19.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:19.385048+0000) 2022-09-10T11:49:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:19.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:19.862389+0000) 2022-09-10T11:49:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:20.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:20.385269+0000) 2022-09-10T11:49:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:20.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:20.862531+0000) 2022-09-10T11:49:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:21 smithi110 ceph-mon[97633]: pgmap v14083: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:49:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:21 smithi104 ceph-mon[97044]: pgmap v14083: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:49:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:21.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:21.385459+0000) 2022-09-10T11:49:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:21.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:21.862787+0000) 2022-09-10T11:49:22.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:22.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:22.385684+0000) 2022-09-10T11:49:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:22.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:22.863004+0000) 2022-09-10T11:49:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:23 smithi110 ceph-mon[97633]: pgmap v14084: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:49:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:23 smithi104 ceph-mon[97044]: pgmap v14084: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:49:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:23.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:23.385936+0000) 2022-09-10T11:49:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:23.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:23.863212+0000) 2022-09-10T11:49:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:24.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:24.386205+0000) 2022-09-10T11:49:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:24.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:24.863444+0000) 2022-09-10T11:49:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:25 smithi110 ceph-mon[97633]: pgmap v14085: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:49:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:25 smithi104 ceph-mon[97044]: pgmap v14085: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:49:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:25.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:25.386381+0000) 2022-09-10T11:49:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:25.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:25.863587+0000) 2022-09-10T11:49:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:26.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:26.386576+0000) 2022-09-10T11:49:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:26.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:26.863762+0000) 2022-09-10T11:49:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:27 smithi110 ceph-mon[97633]: pgmap v14086: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:49:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:27 smithi104 ceph-mon[97044]: pgmap v14086: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:49:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:27.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:27.386806+0000) 2022-09-10T11:49:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:27.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:27.863914+0000) 2022-09-10T11:49:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:49:27.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:49:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:28.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:28.387022+0000) 2022-09-10T11:49:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:28.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:28.864162+0000) 2022-09-10T11:49:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:29 smithi110 ceph-mon[97633]: pgmap v14087: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:29 smithi104 ceph-mon[97044]: pgmap v14087: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:29.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:29.387285+0000) 2022-09-10T11:49:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:29.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:29.864353+0000) 2022-09-10T11:49:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:30.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:30.387504+0000) 2022-09-10T11:49:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:30.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:30.864523+0000) 2022-09-10T11:49:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:31 smithi110 ceph-mon[97633]: pgmap v14088: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:31 smithi104 ceph-mon[97044]: pgmap v14088: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:31.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:31.387758+0000) 2022-09-10T11:49:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:31.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:31.864736+0000) 2022-09-10T11:49:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:32.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:32.388019+0000) 2022-09-10T11:49:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:32.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:32.864988+0000) 2022-09-10T11:49:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:33 smithi110 ceph-mon[97633]: pgmap v14089: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:33 smithi104 ceph-mon[97044]: pgmap v14089: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:33.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:33.388241+0000) 2022-09-10T11:49:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:33.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:33.865207+0000) 2022-09-10T11:49:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:34.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:34.388473+0000) 2022-09-10T11:49:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:34.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:34.865422+0000) 2022-09-10T11:49:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:35 smithi110 ceph-mon[97633]: pgmap v14090: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:35 smithi104 ceph-mon[97044]: pgmap v14090: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:35.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:35.388660+0000) 2022-09-10T11:49:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:35.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:35.865628+0000) 2022-09-10T11:49:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:36.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:36.388855+0000) 2022-09-10T11:49:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:36.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:36.865913+0000) 2022-09-10T11:49:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:37 smithi110 ceph-mon[97633]: pgmap v14091: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:37 smithi104 ceph-mon[97044]: pgmap v14091: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:37.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:37.389079+0000) 2022-09-10T11:49:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:37.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:37.866092+0000) 2022-09-10T11:49:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:38.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:38.389319+0000) 2022-09-10T11:49:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:38.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:38.866304+0000) 2022-09-10T11:49:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:39 smithi110 ceph-mon[97633]: pgmap v14092: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:39 smithi104 ceph-mon[97044]: pgmap v14092: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:39.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:39.389507+0000) 2022-09-10T11:49:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:39.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:39.866494+0000) 2022-09-10T11:49:40.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:40.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:40.389710+0000) 2022-09-10T11:49:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:40.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:40.866732+0000) 2022-09-10T11:49:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:41 smithi110 ceph-mon[97633]: pgmap v14093: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:41 smithi104 ceph-mon[97044]: pgmap v14093: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:41.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:41.389949+0000) 2022-09-10T11:49:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:41.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:41.866913+0000) 2022-09-10T11:49:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:42.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:42.390186+0000) 2022-09-10T11:49:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:49:42.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:49:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:42.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:42.867123+0000) 2022-09-10T11:49:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:43 smithi110 ceph-mon[97633]: pgmap v14094: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:49:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:43 smithi104 ceph-mon[97044]: pgmap v14094: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:49:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:43.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:43.390426+0000) 2022-09-10T11:49:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:43.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:43.867349+0000) 2022-09-10T11:49:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:44.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:44.390675+0000) 2022-09-10T11:49:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:44.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:44.867573+0000) 2022-09-10T11:49:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:45 smithi110 ceph-mon[97633]: pgmap v14095: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:49:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:45 smithi104 ceph-mon[97044]: pgmap v14095: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:49:45.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:45.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:45.390930+0000) 2022-09-10T11:49:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:45.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:45.867803+0000) 2022-09-10T11:49:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:49:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:49:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:49:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:49:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:49:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:46.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:46.391176+0000) 2022-09-10T11:49:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:46.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:46.868054+0000) 2022-09-10T11:49:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:47 smithi110 ceph-mon[97633]: pgmap v14096: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:47 smithi104 ceph-mon[97044]: pgmap v14096: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:47.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:47.391344+0000) 2022-09-10T11:49:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:47.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:47.868284+0000) 2022-09-10T11:49:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:48.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:48.391532+0000) 2022-09-10T11:49:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:48.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:48.868446+0000) 2022-09-10T11:49:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:49 smithi110 ceph-mon[97633]: pgmap v14097: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:49:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:49 smithi104 ceph-mon[97044]: pgmap v14097: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:49:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:49.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:49.391709+0000) 2022-09-10T11:49:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:49.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:49.868592+0000) 2022-09-10T11:49:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:50.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:50.391931+0000) 2022-09-10T11:49:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:50.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:50.868828+0000) 2022-09-10T11:49:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:51 smithi110 ceph-mon[97633]: pgmap v14098: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:51 smithi104 ceph-mon[97044]: pgmap v14098: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:51.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:51.392211+0000) 2022-09-10T11:49:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:51.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:51.869052+0000) 2022-09-10T11:49:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:52.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:52.392435+0000) 2022-09-10T11:49:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:52.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:52.869239+0000) 2022-09-10T11:49:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:53 smithi110 ceph-mon[97633]: pgmap v14099: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:53.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:53 smithi104 ceph-mon[97044]: pgmap v14099: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:53.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:53.392640+0000) 2022-09-10T11:49:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:53.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:53.869445+0000) 2022-09-10T11:49:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:54.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:54.392838+0000) 2022-09-10T11:49:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:54.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:54.869669+0000) 2022-09-10T11:49:55.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:55 smithi110 ceph-mon[97633]: pgmap v14100: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:55 smithi104 ceph-mon[97044]: pgmap v14100: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:55.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:55.393067+0000) 2022-09-10T11:49:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:55.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:55.869839+0000) 2022-09-10T11:49:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:56.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:56.393340+0000) 2022-09-10T11:49:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:56.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:56.870043+0000) 2022-09-10T11:49:57.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:57 smithi110 ceph-mon[97633]: pgmap v14101: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:57 smithi104 ceph-mon[97044]: pgmap v14101: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:49:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:57.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:57.393578+0000) 2022-09-10T11:49:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:57.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:57.870260+0000) 2022-09-10T11:49:58.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:49:58.011+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:49:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:58.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:58.393826+0000) 2022-09-10T11:49:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:58.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:58.870487+0000) 2022-09-10T11:49:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:49:59 smithi110 ceph-mon[97633]: pgmap v14102: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:49:59 smithi104 ceph-mon[97044]: pgmap v14102: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:49:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:49:59.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:59.394063+0000) 2022-09-10T11:50:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:49:59.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:49:59.870712+0000) 2022-09-10T11:50:00.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:50:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T11:50:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:00.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:00.394253+0000) 2022-09-10T11:50:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:00.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:00.870839+0000) 2022-09-10T11:50:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:01 smithi110 ceph-mon[97633]: pgmap v14103: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:01 smithi104 ceph-mon[97044]: pgmap v14103: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:01.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:01.394413+0000) 2022-09-10T11:50:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:01.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:01.871003+0000) 2022-09-10T11:50:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:02.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:02.394649+0000) 2022-09-10T11:50:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:02.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:02.871212+0000) 2022-09-10T11:50:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:03 smithi110 ceph-mon[97633]: pgmap v14104: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:03 smithi104 ceph-mon[97044]: pgmap v14104: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:03.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:03.394871+0000) 2022-09-10T11:50:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:03.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:03.871373+0000) 2022-09-10T11:50:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:04.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:04.395109+0000) 2022-09-10T11:50:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:04.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:04.871545+0000) 2022-09-10T11:50:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:05 smithi110 ceph-mon[97633]: pgmap v14105: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:05 smithi104 ceph-mon[97044]: pgmap v14105: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:05.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:05.395379+0000) 2022-09-10T11:50:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:05.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:05.871748+0000) 2022-09-10T11:50:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:06.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:06.395646+0000) 2022-09-10T11:50:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:06.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:06.871965+0000) 2022-09-10T11:50:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:07 smithi110 ceph-mon[97633]: pgmap v14106: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:50:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:07 smithi104 ceph-mon[97044]: pgmap v14106: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:50:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:07.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:07.395786+0000) 2022-09-10T11:50:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:07.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:07.872202+0000) 2022-09-10T11:50:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:08.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:08.395984+0000) 2022-09-10T11:50:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:08.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:08.872394+0000) 2022-09-10T11:50:09.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:09 smithi110 ceph-mon[97633]: pgmap v14107: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:50:09.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:09 smithi110 ceph-mon[97633]: osdmap e962: 12 total, 12 up, 12 in 2022-09-10T11:50:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:09 smithi104 ceph-mon[97044]: pgmap v14107: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:50:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:09 smithi104 ceph-mon[97044]: osdmap e962: 12 total, 12 up, 12 in 2022-09-10T11:50:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:09.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:09.396199+0000) 2022-09-10T11:50:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:09.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:09.872570+0000) 2022-09-10T11:50:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:10.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:10.396379+0000) 2022-09-10T11:50:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:10.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:10.872804+0000) 2022-09-10T11:50:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:11 smithi110 ceph-mon[97633]: pgmap v14109: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:50:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:11 smithi104 ceph-mon[97044]: pgmap v14109: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:50:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:11.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:11.396620+0000) 2022-09-10T11:50:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:11.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:11.873014+0000) 2022-09-10T11:50:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:12.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:12.396928+0000) 2022-09-10T11:50:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:12.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:12.873224+0000) 2022-09-10T11:50:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:50:13.024+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:50:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:13 smithi110 ceph-mon[97633]: pgmap v14110: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:50:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:13 smithi104 ceph-mon[97044]: pgmap v14110: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:50:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:13.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:13.397155+0000) 2022-09-10T11:50:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:13.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:13.873449+0000) 2022-09-10T11:50:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:14.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:14.397425+0000) 2022-09-10T11:50:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:14.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:14.873634+0000) 2022-09-10T11:50:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:15 smithi110 ceph-mon[97633]: osdmap e963: 12 total, 12 up, 12 in 2022-09-10T11:50:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:15 smithi110 ceph-mon[97633]: pgmap v14112: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:50:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:15 smithi104 ceph-mon[97044]: osdmap e963: 12 total, 12 up, 12 in 2022-09-10T11:50:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:15 smithi104 ceph-mon[97044]: pgmap v14112: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:50:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:15.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:15.397695+0000) 2022-09-10T11:50:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:15.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:15.873853+0000) 2022-09-10T11:50:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:16.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:16.397930+0000) 2022-09-10T11:50:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:16.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:16.874156+0000) 2022-09-10T11:50:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:17 smithi110 ceph-mon[97633]: pgmap v14113: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:50:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:17 smithi104 ceph-mon[97044]: pgmap v14113: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:50:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:17.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:17.398206+0000) 2022-09-10T11:50:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:17.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:17.874341+0000) 2022-09-10T11:50:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:18.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:18.398365+0000) 2022-09-10T11:50:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:18.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:18.874529+0000) 2022-09-10T11:50:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:19 smithi110 ceph-mon[97633]: pgmap v14114: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:50:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:19 smithi104 ceph-mon[97044]: pgmap v14114: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:50:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:19.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:19.398561+0000) 2022-09-10T11:50:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:19.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:19.874727+0000) 2022-09-10T11:50:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:20.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:20.398732+0000) 2022-09-10T11:50:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:20.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:20.874925+0000) 2022-09-10T11:50:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:21 smithi110 ceph-mon[97633]: pgmap v14115: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:50:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:21 smithi104 ceph-mon[97044]: pgmap v14115: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:50:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:21.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:21.398982+0000) 2022-09-10T11:50:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:21.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:21.875188+0000) 2022-09-10T11:50:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:22.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:22.399166+0000) 2022-09-10T11:50:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:22.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:22.875412+0000) 2022-09-10T11:50:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:23 smithi110 ceph-mon[97633]: pgmap v14116: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:50:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:23 smithi104 ceph-mon[97044]: pgmap v14116: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:50:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:23.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:23.399406+0000) 2022-09-10T11:50:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:23.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:23.875591+0000) 2022-09-10T11:50:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:24.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:24.399576+0000) 2022-09-10T11:50:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:24.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:24.875773+0000) 2022-09-10T11:50:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:25 smithi110 ceph-mon[97633]: pgmap v14117: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:50:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:25 smithi104 ceph-mon[97044]: pgmap v14117: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:50:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:25.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:25.399820+0000) 2022-09-10T11:50:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:25.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:25.875993+0000) 2022-09-10T11:50:26.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:26.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:26.400079+0000) 2022-09-10T11:50:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:26.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:26.876243+0000) 2022-09-10T11:50:27.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:27 smithi110 ceph-mon[97633]: pgmap v14118: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:50:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:27 smithi104 ceph-mon[97044]: pgmap v14118: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:50:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:27.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:27.400381+0000) 2022-09-10T11:50:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:27.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:27.876452+0000) 2022-09-10T11:50:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:50:28.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:50:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:28.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:28.400645+0000) 2022-09-10T11:50:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:28.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:28.876667+0000) 2022-09-10T11:50:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:29 smithi110 ceph-mon[97633]: pgmap v14119: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:29 smithi104 ceph-mon[97044]: pgmap v14119: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:29.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:29.400909+0000) 2022-09-10T11:50:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:29.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:29.876870+0000) 2022-09-10T11:50:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:30.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:30.401212+0000) 2022-09-10T11:50:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:30.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:30.877134+0000) 2022-09-10T11:50:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:31 smithi110 ceph-mon[97633]: pgmap v14120: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:31 smithi104 ceph-mon[97044]: pgmap v14120: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:31.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:31.401476+0000) 2022-09-10T11:50:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:31.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:31.877353+0000) 2022-09-10T11:50:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:32.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:32.401723+0000) 2022-09-10T11:50:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:32.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:32.877583+0000) 2022-09-10T11:50:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:33 smithi110 ceph-mon[97633]: pgmap v14121: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:33 smithi104 ceph-mon[97044]: pgmap v14121: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:33.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:33.401889+0000) 2022-09-10T11:50:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:33.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:33.877828+0000) 2022-09-10T11:50:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:34.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:34.402158+0000) 2022-09-10T11:50:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:34.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:34.878055+0000) 2022-09-10T11:50:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:35 smithi110 ceph-mon[97633]: pgmap v14122: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:35 smithi104 ceph-mon[97044]: pgmap v14122: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:35.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:35.402318+0000) 2022-09-10T11:50:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:35.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:35.878260+0000) 2022-09-10T11:50:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:36.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:36.402539+0000) 2022-09-10T11:50:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:36.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:36.878447+0000) 2022-09-10T11:50:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:37 smithi110 ceph-mon[97633]: pgmap v14123: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:37 smithi104 ceph-mon[97044]: pgmap v14123: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:37.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:37.402780+0000) 2022-09-10T11:50:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:37.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:37.878630+0000) 2022-09-10T11:50:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:38.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:38.403055+0000) 2022-09-10T11:50:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:38.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:38.878868+0000) 2022-09-10T11:50:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:39 smithi110 ceph-mon[97633]: pgmap v14124: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:39 smithi104 ceph-mon[97044]: pgmap v14124: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:39.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:39.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:39.403256+0000) 2022-09-10T11:50:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:39.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:39.879078+0000) 2022-09-10T11:50:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:40.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:40.403391+0000) 2022-09-10T11:50:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:40.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:40.879369+0000) 2022-09-10T11:50:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:41 smithi110 ceph-mon[97633]: pgmap v14125: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:41 smithi104 ceph-mon[97044]: pgmap v14125: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:41.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:41.403583+0000) 2022-09-10T11:50:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:41.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:41.879616+0000) 2022-09-10T11:50:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:42.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:42.403760+0000) 2022-09-10T11:50:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:42.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:42.879867+0000) 2022-09-10T11:50:43.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:50:43.052+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:50:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:43 smithi110 ceph-mon[97633]: pgmap v14126: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:43 smithi104 ceph-mon[97044]: pgmap v14126: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:43.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:43.403916+0000) 2022-09-10T11:50:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:43.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:43.880127+0000) 2022-09-10T11:50:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:44.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:44.404084+0000) 2022-09-10T11:50:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:44.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:44.880402+0000) 2022-09-10T11:50:45.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:45 smithi110 ceph-mon[97633]: pgmap v14127: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:45 smithi104 ceph-mon[97044]: pgmap v14127: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:45.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:45.404348+0000) 2022-09-10T11:50:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:45.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:45.880648+0000) 2022-09-10T11:50:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:50:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:50:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:50:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:50:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:50:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:50:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:50:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:46.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:46.404568+0000) 2022-09-10T11:50:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:46.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:46.880911+0000) 2022-09-10T11:50:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:47 smithi110 ceph-mon[97633]: pgmap v14128: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:47 smithi104 ceph-mon[97044]: pgmap v14128: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:47.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:47.404826+0000) 2022-09-10T11:50:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:47.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:47.881135+0000) 2022-09-10T11:50:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:48.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:48.405143+0000) 2022-09-10T11:50:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:48.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:48.881285+0000) 2022-09-10T11:50:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:49 smithi110 ceph-mon[97633]: pgmap v14129: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:49 smithi104 ceph-mon[97044]: pgmap v14129: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:49.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:49.405332+0000) 2022-09-10T11:50:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:49.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:49.881433+0000) 2022-09-10T11:50:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:50.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:50.405509+0000) 2022-09-10T11:50:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:50.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:50.881665+0000) 2022-09-10T11:50:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:51 smithi110 ceph-mon[97633]: pgmap v14130: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:51 smithi104 ceph-mon[97044]: pgmap v14130: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:51.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:51.405711+0000) 2022-09-10T11:50:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:51.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:51.881885+0000) 2022-09-10T11:50:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:52.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:52.405962+0000) 2022-09-10T11:50:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:52.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:52.882167+0000) 2022-09-10T11:50:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:53 smithi110 ceph-mon[97633]: pgmap v14131: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:53 smithi104 ceph-mon[97044]: pgmap v14131: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:53.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:53.406195+0000) 2022-09-10T11:50:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:53.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:53.882397+0000) 2022-09-10T11:50:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:54.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:54.406426+0000) 2022-09-10T11:50:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:54.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:54.882602+0000) 2022-09-10T11:50:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:55 smithi110 ceph-mon[97633]: pgmap v14132: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:55 smithi104 ceph-mon[97044]: pgmap v14132: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:55.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:55.406676+0000) 2022-09-10T11:50:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:55.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:55.882837+0000) 2022-09-10T11:50:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:56.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:56.406884+0000) 2022-09-10T11:50:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:56.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:56.883074+0000) 2022-09-10T11:50:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:57 smithi110 ceph-mon[97633]: pgmap v14133: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:57 smithi104 ceph-mon[97044]: pgmap v14133: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:50:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:57.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:57.407166+0000) 2022-09-10T11:50:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:50:58.067+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:50:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:57.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:57.883234+0000) 2022-09-10T11:50:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:58.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:58.407388+0000) 2022-09-10T11:50:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:58.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:58.883382+0000) 2022-09-10T11:50:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:50:59 smithi110 ceph-mon[97633]: pgmap v14134: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:59.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:50:59 smithi104 ceph-mon[97044]: pgmap v14134: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:50:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:50:59.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:59.407613+0000) 2022-09-10T11:51:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:50:59.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:50:59.883646+0000) 2022-09-10T11:51:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:00.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:00.407869+0000) 2022-09-10T11:51:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:00.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:00.883839+0000) 2022-09-10T11:51:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:01 smithi110 ceph-mon[97633]: pgmap v14135: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:01 smithi104 ceph-mon[97044]: pgmap v14135: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:01.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:01.408087+0000) 2022-09-10T11:51:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:01.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:01.884079+0000) 2022-09-10T11:51:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:02.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:02.408332+0000) 2022-09-10T11:51:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:02.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:02.884332+0000) 2022-09-10T11:51:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:03 smithi110 ceph-mon[97633]: pgmap v14136: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:03 smithi104 ceph-mon[97044]: pgmap v14136: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:03.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:03.408536+0000) 2022-09-10T11:51:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:03.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:03.884554+0000) 2022-09-10T11:51:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:04.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:04.408795+0000) 2022-09-10T11:51:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:04.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:04.884814+0000) 2022-09-10T11:51:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:05 smithi110 ceph-mon[97633]: pgmap v14137: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:05 smithi104 ceph-mon[97044]: pgmap v14137: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:05.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:05.409031+0000) 2022-09-10T11:51:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:05.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:05.885081+0000) 2022-09-10T11:51:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:06.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:06.409271+0000) 2022-09-10T11:51:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:06.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:06.885348+0000) 2022-09-10T11:51:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:07 smithi110 ceph-mon[97633]: pgmap v14138: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:51:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:07 smithi104 ceph-mon[97044]: pgmap v14138: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:51:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:07.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:07.409504+0000) 2022-09-10T11:51:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:07.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:07.885573+0000) 2022-09-10T11:51:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:08.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:08.409703+0000) 2022-09-10T11:51:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:08.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:08.885792+0000) 2022-09-10T11:51:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:09 smithi110 ceph-mon[97633]: pgmap v14139: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:51:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:09 smithi110 ceph-mon[97633]: osdmap e964: 12 total, 12 up, 12 in 2022-09-10T11:51:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:09 smithi104 ceph-mon[97044]: pgmap v14139: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:51:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:09 smithi104 ceph-mon[97044]: osdmap e964: 12 total, 12 up, 12 in 2022-09-10T11:51:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:09.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:09.409886+0000) 2022-09-10T11:51:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:09.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:09.886040+0000) 2022-09-10T11:51:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:10.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:10.410044+0000) 2022-09-10T11:51:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:10.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:10.886258+0000) 2022-09-10T11:51:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:11 smithi110 ceph-mon[97633]: pgmap v14141: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:51:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:11 smithi104 ceph-mon[97044]: pgmap v14141: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:51:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:11.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:11.410223+0000) 2022-09-10T11:51:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:11.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:11.886500+0000) 2022-09-10T11:51:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:12.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:12.410364+0000) 2022-09-10T11:51:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:51:13.077+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:51:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:12.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:12.886700+0000) 2022-09-10T11:51:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:13 smithi110 ceph-mon[97633]: pgmap v14142: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:51:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:13 smithi104 ceph-mon[97044]: pgmap v14142: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:51:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:13.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:13.410482+0000) 2022-09-10T11:51:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:13.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:13.886949+0000) 2022-09-10T11:51:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:14.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:14.410706+0000) 2022-09-10T11:51:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:14.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:14.887233+0000) 2022-09-10T11:51:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:15 smithi110 ceph-mon[97633]: osdmap e965: 12 total, 12 up, 12 in 2022-09-10T11:51:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:15 smithi110 ceph-mon[97633]: pgmap v14144: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:51:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:15 smithi104 ceph-mon[97044]: osdmap e965: 12 total, 12 up, 12 in 2022-09-10T11:51:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:15 smithi104 ceph-mon[97044]: pgmap v14144: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:51:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:15.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:15.410951+0000) 2022-09-10T11:51:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:15.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:15.887482+0000) 2022-09-10T11:51:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:16.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:16.411164+0000) 2022-09-10T11:51:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:16.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:16.887702+0000) 2022-09-10T11:51:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:17 smithi110 ceph-mon[97633]: pgmap v14145: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:51:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:17 smithi104 ceph-mon[97044]: pgmap v14145: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:51:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:17.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:17.411334+0000) 2022-09-10T11:51:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:17.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:17.887909+0000) 2022-09-10T11:51:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:18.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:18.411494+0000) 2022-09-10T11:51:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:18.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:18.888190+0000) 2022-09-10T11:51:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:19 smithi110 ceph-mon[97633]: pgmap v14146: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:51:19.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:19 smithi104 ceph-mon[97044]: pgmap v14146: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:51:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:19.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:19.411677+0000) 2022-09-10T11:51:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:19.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:19.888442+0000) 2022-09-10T11:51:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:20.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:20.411921+0000) 2022-09-10T11:51:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:20.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:20.888665+0000) 2022-09-10T11:51:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:21 smithi110 ceph-mon[97633]: pgmap v14147: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:51:21.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:21.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:21.412178+0000) 2022-09-10T11:51:21.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:21 smithi104 ceph-mon[97044]: pgmap v14147: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:51:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:21.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:21.888878+0000) 2022-09-10T11:51:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:22.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:22.412405+0000) 2022-09-10T11:51:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:22.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:22.889016+0000) 2022-09-10T11:51:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:23 smithi110 ceph-mon[97633]: pgmap v14148: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:51:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:23 smithi104 ceph-mon[97044]: pgmap v14148: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:51:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:23.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:23.412568+0000) 2022-09-10T11:51:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:23.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:23.889237+0000) 2022-09-10T11:51:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:24.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:24.412770+0000) 2022-09-10T11:51:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:24.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:24.889434+0000) 2022-09-10T11:51:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:25 smithi110 ceph-mon[97633]: pgmap v14149: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:51:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:25 smithi104 ceph-mon[97044]: pgmap v14149: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:51:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:25.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:25.412934+0000) 2022-09-10T11:51:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:25.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:25.889591+0000) 2022-09-10T11:51:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:26.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:26.413136+0000) 2022-09-10T11:51:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:26.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:26.889883+0000) 2022-09-10T11:51:27.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:27 smithi110 ceph-mon[97633]: pgmap v14150: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:51:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:27 smithi104 ceph-mon[97044]: pgmap v14150: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:51:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:27.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:27.413331+0000) 2022-09-10T11:51:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:51:28.092+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:51:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:27.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:27.890137+0000) 2022-09-10T11:51:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:28.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:28.413535+0000) 2022-09-10T11:51:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:28.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:28.890380+0000) 2022-09-10T11:51:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:29 smithi110 ceph-mon[97633]: pgmap v14151: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:29 smithi104 ceph-mon[97044]: pgmap v14151: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:29.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:29.413777+0000) 2022-09-10T11:51:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:29.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:29.890642+0000) 2022-09-10T11:51:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:30.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:30.414008+0000) 2022-09-10T11:51:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:30.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:30.890904+0000) 2022-09-10T11:51:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:31 smithi110 ceph-mon[97633]: pgmap v14152: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:31 smithi104 ceph-mon[97044]: pgmap v14152: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:31.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:31.414266+0000) 2022-09-10T11:51:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:31.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:31.891168+0000) 2022-09-10T11:51:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:32.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:32.414498+0000) 2022-09-10T11:51:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:32.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:32.891437+0000) 2022-09-10T11:51:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:33 smithi110 ceph-mon[97633]: pgmap v14153: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:33 smithi104 ceph-mon[97044]: pgmap v14153: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:33.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:33.414752+0000) 2022-09-10T11:51:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:33.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:33.891632+0000) 2022-09-10T11:51:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:34.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:34.414981+0000) 2022-09-10T11:51:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:34.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:34.891902+0000) 2022-09-10T11:51:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:35 smithi110 ceph-mon[97633]: pgmap v14154: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:35 smithi104 ceph-mon[97044]: pgmap v14154: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:35.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:35.415169+0000) 2022-09-10T11:51:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:35.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:35.892071+0000) 2022-09-10T11:51:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:36.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:36.415399+0000) 2022-09-10T11:51:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:36.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:36.892260+0000) 2022-09-10T11:51:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:37 smithi110 ceph-mon[97633]: pgmap v14155: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:37 smithi104 ceph-mon[97044]: pgmap v14155: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:37.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:37.415609+0000) 2022-09-10T11:51:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:37.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:37.892512+0000) 2022-09-10T11:51:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:38.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:38.415827+0000) 2022-09-10T11:51:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:38.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:38.892759+0000) 2022-09-10T11:51:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:39 smithi110 ceph-mon[97633]: pgmap v14156: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:39 smithi104 ceph-mon[97044]: pgmap v14156: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:39.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:39.416134+0000) 2022-09-10T11:51:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:39.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:39.892931+0000) 2022-09-10T11:51:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:40.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:40.416387+0000) 2022-09-10T11:51:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:40.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:40.893201+0000) 2022-09-10T11:51:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:41 smithi110 ceph-mon[97633]: pgmap v14157: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:41 smithi104 ceph-mon[97044]: pgmap v14157: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:41.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:41.416643+0000) 2022-09-10T11:51:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:41.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:41.893423+0000) 2022-09-10T11:51:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:42.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:42.416843+0000) 2022-09-10T11:51:43.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:51:43.103+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:51:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:42.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:42.893641+0000) 2022-09-10T11:51:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:43 smithi110 ceph-mon[97633]: pgmap v14158: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:43 smithi104 ceph-mon[97044]: pgmap v14158: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:43.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:43.417086+0000) 2022-09-10T11:51:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:43.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:43.893881+0000) 2022-09-10T11:51:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:44.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:44.417369+0000) 2022-09-10T11:51:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:44.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:44.894161+0000) 2022-09-10T11:51:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:45 smithi110 ceph-mon[97633]: pgmap v14159: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:45 smithi104 ceph-mon[97044]: pgmap v14159: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:45.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:45.417620+0000) 2022-09-10T11:51:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:45.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:45.894397+0000) 2022-09-10T11:51:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:51:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:51:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:51:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:51:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:51:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:51:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:51:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:51:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:46.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:46.417903+0000) 2022-09-10T11:51:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:46.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:46.894599+0000) 2022-09-10T11:51:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:47 smithi110 ceph-mon[97633]: pgmap v14160: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:47 smithi104 ceph-mon[97044]: pgmap v14160: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:47.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:47.418218+0000) 2022-09-10T11:51:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:47.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:47.894830+0000) 2022-09-10T11:51:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:48.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:48.418497+0000) 2022-09-10T11:51:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:48.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:48.895033+0000) 2022-09-10T11:51:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:49 smithi110 ceph-mon[97633]: pgmap v14161: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:49 smithi104 ceph-mon[97044]: pgmap v14161: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:49.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:49.418654+0000) 2022-09-10T11:51:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:49.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:49.895234+0000) 2022-09-10T11:51:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:50.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:50.418879+0000) 2022-09-10T11:51:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:50.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:50.895468+0000) 2022-09-10T11:51:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:51 smithi110 ceph-mon[97633]: pgmap v14162: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:51 smithi104 ceph-mon[97044]: pgmap v14162: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:51.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:51.419044+0000) 2022-09-10T11:51:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:51.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:51.895716+0000) 2022-09-10T11:51:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:52.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:52.419236+0000) 2022-09-10T11:51:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:52.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:52.895991+0000) 2022-09-10T11:51:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:53 smithi110 ceph-mon[97633]: pgmap v14163: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:53 smithi104 ceph-mon[97044]: pgmap v14163: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:53.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:53.419466+0000) 2022-09-10T11:51:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:53.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:53.896217+0000) 2022-09-10T11:51:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:54.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:54.419759+0000) 2022-09-10T11:51:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:54.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:54.896394+0000) 2022-09-10T11:51:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:55 smithi110 ceph-mon[97633]: pgmap v14164: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:55 smithi104 ceph-mon[97044]: pgmap v14164: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:55.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:55.420020+0000) 2022-09-10T11:51:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:55.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:55.896511+0000) 2022-09-10T11:51:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:56.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:56.420289+0000) 2022-09-10T11:51:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:56.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:56.896737+0000) 2022-09-10T11:51:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:57 smithi110 ceph-mon[97633]: pgmap v14165: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:57 smithi104 ceph-mon[97044]: pgmap v14165: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:51:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:57.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:57.420547+0000) 2022-09-10T11:51:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:57.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:57.896958+0000) 2022-09-10T11:51:58.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:51:58.118+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:51:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:58.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:58.420789+0000) 2022-09-10T11:51:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:58.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:58.897175+0000) 2022-09-10T11:51:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:51:59 smithi110 ceph-mon[97633]: pgmap v14166: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:51:59 smithi104 ceph-mon[97044]: pgmap v14166: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:51:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:51:59.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:59.420945+0000) 2022-09-10T11:52:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:51:59.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:51:59.897345+0000) 2022-09-10T11:52:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:00.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:00.421219+0000) 2022-09-10T11:52:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:00.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:00.897571+0000) 2022-09-10T11:52:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:01 smithi110 ceph-mon[97633]: pgmap v14167: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:01 smithi104 ceph-mon[97044]: pgmap v14167: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:01.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:01.421424+0000) 2022-09-10T11:52:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:01.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:01.897800+0000) 2022-09-10T11:52:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:02.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:02.421655+0000) 2022-09-10T11:52:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:02.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:02.898015+0000) 2022-09-10T11:52:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:03 smithi110 ceph-mon[97633]: pgmap v14168: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:03 smithi104 ceph-mon[97044]: pgmap v14168: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:03.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:03.421884+0000) 2022-09-10T11:52:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:03.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:03.898260+0000) 2022-09-10T11:52:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:04.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:04.422185+0000) 2022-09-10T11:52:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:04.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:04.898431+0000) 2022-09-10T11:52:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:05 smithi110 ceph-mon[97633]: pgmap v14169: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:05 smithi104 ceph-mon[97044]: pgmap v14169: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:05.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:05.422338+0000) 2022-09-10T11:52:06.315 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:05.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:05.898599+0000) 2022-09-10T11:52:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:06.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:06.422576+0000) 2022-09-10T11:52:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:06.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:06.898843+0000) 2022-09-10T11:52:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:07 smithi110 ceph-mon[97633]: pgmap v14170: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:52:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:07 smithi104 ceph-mon[97044]: pgmap v14170: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:52:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:07.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:07.422822+0000) 2022-09-10T11:52:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:07.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:07.899084+0000) 2022-09-10T11:52:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:08.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:08.423123+0000) 2022-09-10T11:52:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:08.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:08.899302+0000) 2022-09-10T11:52:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:09 smithi110 ceph-mon[97633]: pgmap v14171: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:52:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:09 smithi110 ceph-mon[97633]: osdmap e966: 12 total, 12 up, 12 in 2022-09-10T11:52:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:09 smithi104 ceph-mon[97044]: pgmap v14171: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:52:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:09 smithi104 ceph-mon[97044]: osdmap e966: 12 total, 12 up, 12 in 2022-09-10T11:52:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:09.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:09.423418+0000) 2022-09-10T11:52:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:09.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:09.899492+0000) 2022-09-10T11:52:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:10.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:10.423678+0000) 2022-09-10T11:52:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:10.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:10.899634+0000) 2022-09-10T11:52:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:11 smithi110 ceph-mon[97633]: pgmap v14173: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:52:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:11 smithi104 ceph-mon[97044]: pgmap v14173: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:52:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:11.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:11.423932+0000) 2022-09-10T11:52:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:11.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:11.899751+0000) 2022-09-10T11:52:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:12.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:12.424231+0000) 2022-09-10T11:52:13.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:52:13.131+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:52:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:12.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:12.899987+0000) 2022-09-10T11:52:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:13 smithi110 ceph-mon[97633]: pgmap v14174: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:52:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:13 smithi104 ceph-mon[97044]: pgmap v14174: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:52:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:13.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:13.424457+0000) 2022-09-10T11:52:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:13.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:13.900206+0000) 2022-09-10T11:52:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:14.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:14.424729+0000) 2022-09-10T11:52:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:14.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:14.900433+0000) 2022-09-10T11:52:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:15 smithi110 ceph-mon[97633]: osdmap e967: 12 total, 12 up, 12 in 2022-09-10T11:52:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:15 smithi110 ceph-mon[97633]: pgmap v14176: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:15 smithi104 ceph-mon[97044]: osdmap e967: 12 total, 12 up, 12 in 2022-09-10T11:52:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:15 smithi104 ceph-mon[97044]: pgmap v14176: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:52:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:15.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:15.424958+0000) 2022-09-10T11:52:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:15.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:15.900664+0000) 2022-09-10T11:52:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:16.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:16.425215+0000) 2022-09-10T11:52:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:16.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:16.900901+0000) 2022-09-10T11:52:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:17 smithi110 ceph-mon[97633]: pgmap v14177: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:52:17.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:17 smithi104 ceph-mon[97044]: pgmap v14177: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:52:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:17.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:17.425451+0000) 2022-09-10T11:52:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:17.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:17.901180+0000) 2022-09-10T11:52:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:18.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:18.425678+0000) 2022-09-10T11:52:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:18.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:18.901410+0000) 2022-09-10T11:52:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:19 smithi110 ceph-mon[97633]: pgmap v14178: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:52:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:19 smithi104 ceph-mon[97044]: pgmap v14178: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:52:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:19.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:19.425953+0000) 2022-09-10T11:52:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:19.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:19.901607+0000) 2022-09-10T11:52:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:20.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:20.426199+0000) 2022-09-10T11:52:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:20.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:20.901785+0000) 2022-09-10T11:52:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:21 smithi110 ceph-mon[97633]: pgmap v14179: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:52:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:21 smithi104 ceph-mon[97044]: pgmap v14179: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:52:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:21.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:21.426403+0000) 2022-09-10T11:52:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:21.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:21.901962+0000) 2022-09-10T11:52:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:22.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:22.426659+0000) 2022-09-10T11:52:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:22.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:22.902205+0000) 2022-09-10T11:52:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:23 smithi110 ceph-mon[97633]: pgmap v14180: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:52:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:23 smithi104 ceph-mon[97044]: pgmap v14180: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:52:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:23.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:23.426906+0000) 2022-09-10T11:52:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:23.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:23.902402+0000) 2022-09-10T11:52:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:24.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:24.427206+0000) 2022-09-10T11:52:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:24.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:24.902623+0000) 2022-09-10T11:52:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:25 smithi110 ceph-mon[97633]: pgmap v14181: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:52:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:25 smithi104 ceph-mon[97044]: pgmap v14181: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:52:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:25.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:25.427418+0000) 2022-09-10T11:52:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:25.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:25.902853+0000) 2022-09-10T11:52:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:26.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:26.427621+0000) 2022-09-10T11:52:27.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:26.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:26.903159+0000) 2022-09-10T11:52:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:27 smithi110 ceph-mon[97633]: pgmap v14182: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:52:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:27 smithi104 ceph-mon[97044]: pgmap v14182: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:52:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:27.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:27.427847+0000) 2022-09-10T11:52:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:27.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:27.903429+0000) 2022-09-10T11:52:28.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:52:28.147+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:52:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:28.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:28.428140+0000) 2022-09-10T11:52:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:28.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:28.903687+0000) 2022-09-10T11:52:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:29 smithi110 ceph-mon[97633]: pgmap v14183: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:29 smithi104 ceph-mon[97044]: pgmap v14183: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:29.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:29.428346+0000) 2022-09-10T11:52:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:29.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:29.904015+0000) 2022-09-10T11:52:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:30.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:30.428537+0000) 2022-09-10T11:52:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:30.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:30.904186+0000) 2022-09-10T11:52:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:31 smithi110 ceph-mon[97633]: pgmap v14184: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:31 smithi104 ceph-mon[97044]: pgmap v14184: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:31.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:31.428764+0000) 2022-09-10T11:52:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:31.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:31.904361+0000) 2022-09-10T11:52:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:32.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:32.429000+0000) 2022-09-10T11:52:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:32.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:32.904554+0000) 2022-09-10T11:52:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:33 smithi110 ceph-mon[97633]: pgmap v14185: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:33 smithi104 ceph-mon[97044]: pgmap v14185: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:33.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:33.429196+0000) 2022-09-10T11:52:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:33.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:33.904735+0000) 2022-09-10T11:52:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:34.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:34.429417+0000) 2022-09-10T11:52:35.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:34.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:34.904990+0000) 2022-09-10T11:52:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:35 smithi110 ceph-mon[97633]: pgmap v14186: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:35 smithi104 ceph-mon[97044]: pgmap v14186: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:35.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:35.429652+0000) 2022-09-10T11:52:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:35.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:35.905207+0000) 2022-09-10T11:52:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:36.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:36.429903+0000) 2022-09-10T11:52:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:36.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:36.905408+0000) 2022-09-10T11:52:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:37 smithi110 ceph-mon[97633]: pgmap v14187: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:37 smithi104 ceph-mon[97044]: pgmap v14187: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:37.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:37.430180+0000) 2022-09-10T11:52:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:37.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:37.905742+0000) 2022-09-10T11:52:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:38.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:38.430442+0000) 2022-09-10T11:52:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:38.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:38.906008+0000) 2022-09-10T11:52:39.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:39 smithi110 ceph-mon[97633]: pgmap v14188: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:39 smithi104 ceph-mon[97044]: pgmap v14188: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:39.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:39.430662+0000) 2022-09-10T11:52:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:39.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:39.906262+0000) 2022-09-10T11:52:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:40.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:40.430903+0000) 2022-09-10T11:52:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:40.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:40.906503+0000) 2022-09-10T11:52:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:41 smithi110 ceph-mon[97633]: pgmap v14189: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:41 smithi104 ceph-mon[97044]: pgmap v14189: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:41.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:41.431141+0000) 2022-09-10T11:52:42.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:41.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:41.906649+0000) 2022-09-10T11:52:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:42.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:42.431286+0000) 2022-09-10T11:52:43.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:42.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:42.906817+0000) 2022-09-10T11:52:43.433 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:52:43.156+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:52:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:43 smithi110 ceph-mon[97633]: pgmap v14190: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:43 smithi104 ceph-mon[97044]: pgmap v14190: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:43.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:43.431477+0000) 2022-09-10T11:52:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:43.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:43.907022+0000) 2022-09-10T11:52:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:44.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:44.431743+0000) 2022-09-10T11:52:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:44.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:44.907166+0000) 2022-09-10T11:52:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:45 smithi110 ceph-mon[97633]: pgmap v14191: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:45 smithi104 ceph-mon[97044]: pgmap v14191: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:45.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:45.431916+0000) 2022-09-10T11:52:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:45.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:45.907338+0000) 2022-09-10T11:52:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:52:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:52:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:52:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:52:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:52:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:46.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:46.432152+0000) 2022-09-10T11:52:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:46.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:46.907508+0000) 2022-09-10T11:52:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:47 smithi110 ceph-mon[97633]: pgmap v14192: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:47 smithi104 ceph-mon[97044]: pgmap v14192: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:47.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:47.432410+0000) 2022-09-10T11:52:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:47.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:47.907683+0000) 2022-09-10T11:52:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:48.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:48.432596+0000) 2022-09-10T11:52:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:48.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:48.907844+0000) 2022-09-10T11:52:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:49 smithi110 ceph-mon[97633]: pgmap v14193: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:49 smithi104 ceph-mon[97044]: pgmap v14193: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:49.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:49.432779+0000) 2022-09-10T11:52:50.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:49.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:49.907984+0000) 2022-09-10T11:52:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:50.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:50.432995+0000) 2022-09-10T11:52:51.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:50.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:50.908151+0000) 2022-09-10T11:52:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:51 smithi110 ceph-mon[97633]: pgmap v14194: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:51 smithi104 ceph-mon[97044]: pgmap v14194: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:51.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:51.433235+0000) 2022-09-10T11:52:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:51.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:51.908369+0000) 2022-09-10T11:52:52.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:52.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:52.433473+0000) 2022-09-10T11:52:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:52.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:52.908542+0000) 2022-09-10T11:52:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:53 smithi110 ceph-mon[97633]: pgmap v14195: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:53 smithi104 ceph-mon[97044]: pgmap v14195: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:53.817 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:53.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:53.433750+0000) 2022-09-10T11:52:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:53.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:53.908803+0000) 2022-09-10T11:52:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:54.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:54.433974+0000) 2022-09-10T11:52:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:54.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:54.909033+0000) 2022-09-10T11:52:55.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:55 smithi110 ceph-mon[97633]: pgmap v14196: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:55 smithi104 ceph-mon[97044]: pgmap v14196: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:55.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:55.434246+0000) 2022-09-10T11:52:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:55.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:55.909258+0000) 2022-09-10T11:52:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:56.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:56.434437+0000) 2022-09-10T11:52:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:56.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:56.909464+0000) 2022-09-10T11:52:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:57 smithi110 ceph-mon[97633]: pgmap v14197: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:57 smithi104 ceph-mon[97044]: pgmap v14197: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:52:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:57.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:57.434665+0000) 2022-09-10T11:52:58.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:57.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:57.909698+0000) 2022-09-10T11:52:58.435 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:52:58.159+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:52:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:58.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:58.434836+0000) 2022-09-10T11:52:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:58.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:58.909854+0000) 2022-09-10T11:52:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:52:59 smithi110 ceph-mon[97633]: pgmap v14198: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:52:59 smithi104 ceph-mon[97044]: pgmap v14198: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:52:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:52:59.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:59.435084+0000) 2022-09-10T11:53:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:52:59.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:52:59.910048+0000) 2022-09-10T11:53:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:00.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:00.435265+0000) 2022-09-10T11:53:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:00.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:00.910245+0000) 2022-09-10T11:53:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:01 smithi110 ceph-mon[97633]: pgmap v14199: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:01 smithi104 ceph-mon[97044]: pgmap v14199: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:01.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:01.435490+0000) 2022-09-10T11:53:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:01.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:01.910486+0000) 2022-09-10T11:53:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:02.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:02.435720+0000) 2022-09-10T11:53:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:02.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:02.910739+0000) 2022-09-10T11:53:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:03 smithi110 ceph-mon[97633]: pgmap v14200: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:03 smithi104 ceph-mon[97044]: pgmap v14200: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:03.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:03.435949+0000) 2022-09-10T11:53:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:03.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:03.910982+0000) 2022-09-10T11:53:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:04.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:04.436193+0000) 2022-09-10T11:53:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:04.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:04.911229+0000) 2022-09-10T11:53:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:05 smithi110 ceph-mon[97633]: pgmap v14201: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:05 smithi104 ceph-mon[97044]: pgmap v14201: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:05.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:05.436386+0000) 2022-09-10T11:53:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:05.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:05.911416+0000) 2022-09-10T11:53:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:06.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:06.436679+0000) 2022-09-10T11:53:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:06.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:06.911624+0000) 2022-09-10T11:53:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:07 smithi110 ceph-mon[97633]: pgmap v14202: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:53:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:07 smithi104 ceph-mon[97044]: pgmap v14202: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:53:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:07.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:07.436932+0000) 2022-09-10T11:53:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:07.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:07.911828+0000) 2022-09-10T11:53:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:08.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:08.437219+0000) 2022-09-10T11:53:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:08.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:08.912141+0000) 2022-09-10T11:53:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:09 smithi110 ceph-mon[97633]: pgmap v14203: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:53:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:09 smithi110 ceph-mon[97633]: osdmap e968: 12 total, 12 up, 12 in 2022-09-10T11:53:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:09 smithi104 ceph-mon[97044]: pgmap v14203: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:53:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:09 smithi104 ceph-mon[97044]: osdmap e968: 12 total, 12 up, 12 in 2022-09-10T11:53:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:09.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:09.437408+0000) 2022-09-10T11:53:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:09.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:09.912382+0000) 2022-09-10T11:53:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:10.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:10.437686+0000) 2022-09-10T11:53:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:10.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:10.912655+0000) 2022-09-10T11:53:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:11 smithi110 ceph-mon[97633]: pgmap v14205: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:53:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:11 smithi104 ceph-mon[97044]: pgmap v14205: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:53:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:11.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:11.437951+0000) 2022-09-10T11:53:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:11.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:11.912909+0000) 2022-09-10T11:53:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:12.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:12.438162+0000) 2022-09-10T11:53:13.168 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:12.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:12.913174+0000) 2022-09-10T11:53:13.439 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:53:13.167+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:53:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:13 smithi110 ceph-mon[97633]: pgmap v14206: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:53:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:13 smithi104 ceph-mon[97044]: pgmap v14206: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:53:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:13.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:13.438418+0000) 2022-09-10T11:53:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:13.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:13.913374+0000) 2022-09-10T11:53:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:14.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:14.438655+0000) 2022-09-10T11:53:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:14.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:14.913629+0000) 2022-09-10T11:53:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:15 smithi110 ceph-mon[97633]: osdmap e969: 12 total, 12 up, 12 in 2022-09-10T11:53:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:15 smithi110 ceph-mon[97633]: pgmap v14208: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:53:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:15 smithi104 ceph-mon[97044]: osdmap e969: 12 total, 12 up, 12 in 2022-09-10T11:53:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:15 smithi104 ceph-mon[97044]: pgmap v14208: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T11:53:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:15.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:15.438864+0000) 2022-09-10T11:53:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:15.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:15.913893+0000) 2022-09-10T11:53:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:16.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:16.439135+0000) 2022-09-10T11:53:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:16.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:16.914156+0000) 2022-09-10T11:53:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:17 smithi110 ceph-mon[97633]: pgmap v14209: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:53:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:17 smithi104 ceph-mon[97044]: pgmap v14209: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:53:17.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:17.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:17.439297+0000) 2022-09-10T11:53:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:17.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:17.914351+0000) 2022-09-10T11:53:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:18.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:18.439520+0000) 2022-09-10T11:53:19.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:18.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:18.914576+0000) 2022-09-10T11:53:19.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:19 smithi110 ceph-mon[97633]: pgmap v14210: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:53:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:19 smithi104 ceph-mon[97044]: pgmap v14210: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:53:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:19.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:19.439688+0000) 2022-09-10T11:53:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:19.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:19.914826+0000) 2022-09-10T11:53:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:20.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:20.439912+0000) 2022-09-10T11:53:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:20.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:20.915047+0000) 2022-09-10T11:53:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:21 smithi110 ceph-mon[97633]: pgmap v14211: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:53:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:21 smithi104 ceph-mon[97044]: pgmap v14211: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:53:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:21.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:21.440187+0000) 2022-09-10T11:53:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:21.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:21.915221+0000) 2022-09-10T11:53:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:22.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:22.440377+0000) 2022-09-10T11:53:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:22.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:22.915452+0000) 2022-09-10T11:53:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:23 smithi110 ceph-mon[97633]: pgmap v14212: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:53:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:23 smithi104 ceph-mon[97044]: pgmap v14212: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:53:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:23.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:23.440630+0000) 2022-09-10T11:53:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:23.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:23.915578+0000) 2022-09-10T11:53:24.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:24.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:24.440883+0000) 2022-09-10T11:53:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:24.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:24.915810+0000) 2022-09-10T11:53:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:25 smithi110 ceph-mon[97633]: pgmap v14213: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:53:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:25 smithi104 ceph-mon[97044]: pgmap v14213: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 100 B/s wr, 9 op/s 2022-09-10T11:53:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:25.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:25.441058+0000) 2022-09-10T11:53:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:25.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:25.916047+0000) 2022-09-10T11:53:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:26.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:26.441238+0000) 2022-09-10T11:53:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:26.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:26.916330+0000) 2022-09-10T11:53:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:27 smithi110 ceph-mon[97633]: pgmap v14214: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:53:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:27 smithi104 ceph-mon[97044]: pgmap v14214: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T11:53:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:27.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:27.441431+0000) 2022-09-10T11:53:28.184 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:27.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:27.916542+0000) 2022-09-10T11:53:28.442 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:53:28.183+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:53:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:28.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:28.441571+0000) 2022-09-10T11:53:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:28.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:28.916708+0000) 2022-09-10T11:53:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:29 smithi110 ceph-mon[97633]: pgmap v14215: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:29 smithi104 ceph-mon[97044]: pgmap v14215: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:29.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:29.441787+0000) 2022-09-10T11:53:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:29.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:29.916973+0000) 2022-09-10T11:53:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:30.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:30.442031+0000) 2022-09-10T11:53:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:30.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:30.917204+0000) 2022-09-10T11:53:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:31 smithi110 ceph-mon[97633]: pgmap v14216: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:31 smithi104 ceph-mon[97044]: pgmap v14216: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:31.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:31.442315+0000) 2022-09-10T11:53:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:31.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:31.917427+0000) 2022-09-10T11:53:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:32.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:32.442511+0000) 2022-09-10T11:53:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:32.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:32.917679+0000) 2022-09-10T11:53:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:33 smithi110 ceph-mon[97633]: pgmap v14217: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:33.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:33 smithi104 ceph-mon[97044]: pgmap v14217: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:33.813 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:33.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:33.442741+0000) 2022-09-10T11:53:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:33.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:33.917930+0000) 2022-09-10T11:53:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:34.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:34.442982+0000) 2022-09-10T11:53:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:34.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:34.918196+0000) 2022-09-10T11:53:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:35 smithi110 ceph-mon[97633]: pgmap v14218: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:35 smithi104 ceph-mon[97044]: pgmap v14218: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:35.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:35.443195+0000) 2022-09-10T11:53:36.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:35.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:35.918391+0000) 2022-09-10T11:53:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:36.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:36.443346+0000) 2022-09-10T11:53:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:36.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:36.918661+0000) 2022-09-10T11:53:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:37 smithi110 ceph-mon[97633]: pgmap v14219: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:37 smithi104 ceph-mon[97044]: pgmap v14219: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:37.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:37.443610+0000) 2022-09-10T11:53:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:37.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:37.918868+0000) 2022-09-10T11:53:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:38.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:38.443911+0000) 2022-09-10T11:53:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:38.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:38.919170+0000) 2022-09-10T11:53:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:39 smithi110 ceph-mon[97633]: pgmap v14220: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:39 smithi104 ceph-mon[97044]: pgmap v14220: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:39.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:39.444183+0000) 2022-09-10T11:53:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:39.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:39.919359+0000) 2022-09-10T11:53:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:40.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:40.444360+0000) 2022-09-10T11:53:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:40.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:40.919574+0000) 2022-09-10T11:53:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:41 smithi110 ceph-mon[97633]: pgmap v14221: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:41 smithi104 ceph-mon[97044]: pgmap v14221: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:41.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:41.444533+0000) 2022-09-10T11:53:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:41.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:41.919839+0000) 2022-09-10T11:53:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:42.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:42.444725+0000) 2022-09-10T11:53:43.197 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:42.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:42.920036+0000) 2022-09-10T11:53:43.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:53:43.195+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:53:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:43.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:43.444966+0000) 2022-09-10T11:53:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:43 smithi110 ceph-mon[97633]: pgmap v14222: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:43 smithi104 ceph-mon[97044]: pgmap v14222: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:43.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:43.920224+0000) 2022-09-10T11:53:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:44.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:44.445225+0000) 2022-09-10T11:53:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:44.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:44.920440+0000) 2022-09-10T11:53:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:45 smithi110 ceph-mon[97633]: pgmap v14223: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:45 smithi104 ceph-mon[97044]: pgmap v14223: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:45.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:45.445394+0000) 2022-09-10T11:53:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:45.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:45.920687+0000) 2022-09-10T11:53:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:53:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:53:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:53:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:53:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:53:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:53:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:53:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:53:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:46.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:46.445628+0000) 2022-09-10T11:53:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:46.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:46.920889+0000) 2022-09-10T11:53:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:47 smithi110 ceph-mon[97633]: pgmap v14224: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:47 smithi104 ceph-mon[97044]: pgmap v14224: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:47.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:47.445893+0000) 2022-09-10T11:53:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:47.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:47.921165+0000) 2022-09-10T11:53:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:48.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:48.446191+0000) 2022-09-10T11:53:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:48.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:48.921423+0000) 2022-09-10T11:53:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:49 smithi110 ceph-mon[97633]: pgmap v14225: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:49.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:49 smithi104 ceph-mon[97044]: pgmap v14225: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:49.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:49.446447+0000) 2022-09-10T11:53:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:49.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:49.921654+0000) 2022-09-10T11:53:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:50.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:50.446652+0000) 2022-09-10T11:53:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:50.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:50.921884+0000) 2022-09-10T11:53:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:51 smithi110 ceph-mon[97633]: pgmap v14226: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:51 smithi104 ceph-mon[97044]: pgmap v14226: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:51.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:51.446869+0000) 2022-09-10T11:53:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:51.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:51.922133+0000) 2022-09-10T11:53:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:52.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:52.447176+0000) 2022-09-10T11:53:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:52.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:52.922368+0000) 2022-09-10T11:53:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:53 smithi110 ceph-mon[97633]: pgmap v14227: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:53 smithi104 ceph-mon[97044]: pgmap v14227: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:53.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:53.447422+0000) 2022-09-10T11:53:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:53.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:53.922612+0000) 2022-09-10T11:53:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:54.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:54.447641+0000) 2022-09-10T11:53:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:54.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:54.922814+0000) 2022-09-10T11:53:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:55 smithi110 ceph-mon[97633]: pgmap v14228: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:53:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:55 smithi104 ceph-mon[97044]: pgmap v14228: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:53:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:55.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:55.447882+0000) 2022-09-10T11:53:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:55.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:55.922962+0000) 2022-09-10T11:53:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:56.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:56.448142+0000) 2022-09-10T11:53:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:56.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:56.923196+0000) 2022-09-10T11:53:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:57 smithi110 ceph-mon[97633]: pgmap v14229: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:57 smithi104 ceph-mon[97044]: pgmap v14229: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:53:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:57.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:57.448292+0000) 2022-09-10T11:53:58.216 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:57.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:57.923422+0000) 2022-09-10T11:53:58.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:53:58.209+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:53:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:58.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:58.448510+0000) 2022-09-10T11:53:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:58.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:58.923633+0000) 2022-09-10T11:53:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:53:59 smithi110 ceph-mon[97633]: pgmap v14230: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:53:59 smithi104 ceph-mon[97044]: pgmap v14230: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:53:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:53:59.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:59.448774+0000) 2022-09-10T11:54:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:53:59.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:53:59.923803+0000) 2022-09-10T11:54:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:00.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:00.448945+0000) 2022-09-10T11:54:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:00.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:00.923940+0000) 2022-09-10T11:54:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:01 smithi110 ceph-mon[97633]: pgmap v14231: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:01 smithi104 ceph-mon[97044]: pgmap v14231: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:01.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:01.449224+0000) 2022-09-10T11:54:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:01.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:01.924136+0000) 2022-09-10T11:54:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:02.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:02.449443+0000) 2022-09-10T11:54:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:02.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:02.924332+0000) 2022-09-10T11:54:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:03 smithi110 ceph-mon[97633]: pgmap v14232: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:03 smithi104 ceph-mon[97044]: pgmap v14232: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:03.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:03.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:03.449636+0000) 2022-09-10T11:54:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:03.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:03.924460+0000) 2022-09-10T11:54:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:04.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:04.449834+0000) 2022-09-10T11:54:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:04.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:04.924653+0000) 2022-09-10T11:54:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:05 smithi110 ceph-mon[97633]: pgmap v14233: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T11:54:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:05 smithi104 ceph-mon[97044]: pgmap v14233: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 21 op/s 2022-09-10T11:54:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:05.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:05.450077+0000) 2022-09-10T11:54:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:05.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:05.924899+0000) 2022-09-10T11:54:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:06.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:06.450407+0000) 2022-09-10T11:54:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:06.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:06.925189+0000) 2022-09-10T11:54:07.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:07 smithi110 ceph-mon[97633]: pgmap v14234: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T11:54:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:07 smithi104 ceph-mon[97044]: pgmap v14234: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T11:54:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:07.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:07.450629+0000) 2022-09-10T11:54:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:07.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:07.925382+0000) 2022-09-10T11:54:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:08.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:08.450829+0000) 2022-09-10T11:54:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:08.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:08.925591+0000) 2022-09-10T11:54:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:09 smithi110 ceph-mon[97633]: pgmap v14235: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:54:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:09 smithi110 ceph-mon[97633]: osdmap e970: 12 total, 12 up, 12 in 2022-09-10T11:54:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:09 smithi104 ceph-mon[97044]: pgmap v14235: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:54:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:09 smithi104 ceph-mon[97044]: osdmap e970: 12 total, 12 up, 12 in 2022-09-10T11:54:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:09.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:09.451071+0000) 2022-09-10T11:54:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:09.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:09.925785+0000) 2022-09-10T11:54:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:10.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:10.451269+0000) 2022-09-10T11:54:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:10.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:10.926016+0000) 2022-09-10T11:54:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:11 smithi110 ceph-mon[97633]: pgmap v14237: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:54:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:11 smithi104 ceph-mon[97044]: pgmap v14237: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:54:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:11.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:11.451454+0000) 2022-09-10T11:54:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:11.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:11.926298+0000) 2022-09-10T11:54:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:12.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:12.451683+0000) 2022-09-10T11:54:13.221 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:12.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:12.926542+0000) 2022-09-10T11:54:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:13.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:13.451936+0000) 2022-09-10T11:54:13.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:54:13.219+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:54:13.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:13 smithi110 ceph-mon[97633]: pgmap v14238: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:54:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:13 smithi104 ceph-mon[97044]: pgmap v14238: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:54:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:13.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:13.926789+0000) 2022-09-10T11:54:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:14.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:14.452231+0000) 2022-09-10T11:54:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:14.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:14.927039+0000) 2022-09-10T11:54:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:15 smithi110 ceph-mon[97633]: osdmap e971: 12 total, 12 up, 12 in 2022-09-10T11:54:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:15 smithi110 ceph-mon[97633]: pgmap v14240: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-10T11:54:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:15 smithi104 ceph-mon[97044]: osdmap e971: 12 total, 12 up, 12 in 2022-09-10T11:54:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:15 smithi104 ceph-mon[97044]: pgmap v14240: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2022-09-10T11:54:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:15.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:15.452489+0000) 2022-09-10T11:54:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:15.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:15.927223+0000) 2022-09-10T11:54:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:16.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:16.452701+0000) 2022-09-10T11:54:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:16.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:16.927439+0000) 2022-09-10T11:54:17.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:17 smithi110 ceph-mon[97633]: pgmap v14241: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:54:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:17 smithi104 ceph-mon[97044]: pgmap v14241: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:54:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:17.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:17.452921+0000) 2022-09-10T11:54:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:17.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:17.927682+0000) 2022-09-10T11:54:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:18.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:18.453187+0000) 2022-09-10T11:54:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:18.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:18.927990+0000) 2022-09-10T11:54:19.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:19 smithi110 ceph-mon[97633]: pgmap v14242: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:54:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:19 smithi104 ceph-mon[97044]: pgmap v14242: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:54:19.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:19.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:19.453386+0000) 2022-09-10T11:54:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:19.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:19.928177+0000) 2022-09-10T11:54:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:20.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:20.453557+0000) 2022-09-10T11:54:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:20.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:20.928356+0000) 2022-09-10T11:54:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:21 smithi110 ceph-mon[97633]: pgmap v14243: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:54:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:21 smithi104 ceph-mon[97044]: pgmap v14243: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:54:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:21.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:21.453744+0000) 2022-09-10T11:54:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:21.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:21.928584+0000) 2022-09-10T11:54:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:22.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:22.454009+0000) 2022-09-10T11:54:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:22.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:22.928745+0000) 2022-09-10T11:54:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:23 smithi104 ceph-mon[97044]: pgmap v14244: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:54:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:23.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:23.454274+0000) 2022-09-10T11:54:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:23 smithi110 ceph-mon[97633]: pgmap v14244: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:54:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:23.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:23.928894+0000) 2022-09-10T11:54:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:24.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:24.454582+0000) 2022-09-10T11:54:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:24.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:24.929188+0000) 2022-09-10T11:54:25.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:25 smithi110 ceph-mon[97633]: pgmap v14245: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:54:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:25 smithi104 ceph-mon[97044]: pgmap v14245: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 100 B/s wr, 10 op/s 2022-09-10T11:54:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:25.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:25.454862+0000) 2022-09-10T11:54:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:25.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:25.929403+0000) 2022-09-10T11:54:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:26.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:26.455089+0000) 2022-09-10T11:54:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:26.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:26.929611+0000) 2022-09-10T11:54:27.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:27 smithi110 ceph-mon[97633]: pgmap v14246: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:27 smithi104 ceph-mon[97044]: pgmap v14246: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:27.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:27.455385+0000) 2022-09-10T11:54:28.224 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:27.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:27.929837+0000) 2022-09-10T11:54:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:28.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:28.455652+0000) 2022-09-10T11:54:28.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:54:28.222+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:54:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:28.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:28.930039+0000) 2022-09-10T11:54:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:29 smithi110 ceph-mon[97633]: pgmap v14247: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:29 smithi104 ceph-mon[97044]: pgmap v14247: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:29.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:29.455908+0000) 2022-09-10T11:54:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:29.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:29.930284+0000) 2022-09-10T11:54:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:30.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:30.456127+0000) 2022-09-10T11:54:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:30.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:30.930469+0000) 2022-09-10T11:54:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:31 smithi110 ceph-mon[97633]: pgmap v14248: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:31 smithi104 ceph-mon[97044]: pgmap v14248: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:31.813 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:31.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:31.456262+0000) 2022-09-10T11:54:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:31.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:31.930697+0000) 2022-09-10T11:54:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:32.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:32.456434+0000) 2022-09-10T11:54:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:32.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:32.930924+0000) 2022-09-10T11:54:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:33 smithi110 ceph-mon[97633]: pgmap v14249: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:33 smithi104 ceph-mon[97044]: pgmap v14249: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:33.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:33.456630+0000) 2022-09-10T11:54:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:33.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:33.931149+0000) 2022-09-10T11:54:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:34.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:34.456872+0000) 2022-09-10T11:54:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:34.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:34.931363+0000) 2022-09-10T11:54:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:35 smithi110 ceph-mon[97633]: pgmap v14250: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:54:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:35 smithi104 ceph-mon[97044]: pgmap v14250: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:54:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:35.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:35.457070+0000) 2022-09-10T11:54:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:35.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:35.931496+0000) 2022-09-10T11:54:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:36.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:36.457306+0000) 2022-09-10T11:54:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:36.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:36.931739+0000) 2022-09-10T11:54:37.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:37 smithi110 ceph-mon[97633]: pgmap v14251: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:37 smithi104 ceph-mon[97044]: pgmap v14251: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:37.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:37.457531+0000) 2022-09-10T11:54:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:37.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:37.931959+0000) 2022-09-10T11:54:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:38.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:38.457741+0000) 2022-09-10T11:54:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:38.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:38.932257+0000) 2022-09-10T11:54:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:39 smithi110 ceph-mon[97633]: pgmap v14252: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:39 smithi104 ceph-mon[97044]: pgmap v14252: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:39.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:39.457973+0000) 2022-09-10T11:54:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:39.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:39.932532+0000) 2022-09-10T11:54:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:40.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:40.458222+0000) 2022-09-10T11:54:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:40.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:40.932776+0000) 2022-09-10T11:54:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:41 smithi110 ceph-mon[97633]: pgmap v14253: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:41 smithi104 ceph-mon[97044]: pgmap v14253: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:41.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:41.458479+0000) 2022-09-10T11:54:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:41.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:41.932991+0000) 2022-09-10T11:54:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:42.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:42.458641+0000) 2022-09-10T11:54:43.240 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:42.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:42.933226+0000) 2022-09-10T11:54:43.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:54:43.238+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:54:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:43.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:43.458866+0000) 2022-09-10T11:54:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:43 smithi104 ceph-mon[97044]: pgmap v14254: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:43 smithi110 ceph-mon[97633]: pgmap v14254: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:43.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:43.933457+0000) 2022-09-10T11:54:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:44.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:44.459156+0000) 2022-09-10T11:54:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:44.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:44.933695+0000) 2022-09-10T11:54:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:45 smithi110 ceph-mon[97633]: pgmap v14255: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:54:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:45 smithi104 ceph-mon[97044]: pgmap v14255: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:54:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:45.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:45.459364+0000) 2022-09-10T11:54:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:45.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:45.933939+0000) 2022-09-10T11:54:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:54:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:54:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:54:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:54:46.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:54:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:46.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:46.459648+0000) 2022-09-10T11:54:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:46.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:46.934179+0000) 2022-09-10T11:54:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:47 smithi110 ceph-mon[97633]: pgmap v14256: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:47 smithi104 ceph-mon[97044]: pgmap v14256: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:47.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:47.459838+0000) 2022-09-10T11:54:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:47.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:47.934410+0000) 2022-09-10T11:54:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:48.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:48.460011+0000) 2022-09-10T11:54:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:48.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:48.934679+0000) 2022-09-10T11:54:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:49 smithi110 ceph-mon[97633]: pgmap v14257: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:49.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:49 smithi104 ceph-mon[97044]: pgmap v14257: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:49.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:49.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:49.460226+0000) 2022-09-10T11:54:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:49.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:49.934926+0000) 2022-09-10T11:54:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:50.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:50.460430+0000) 2022-09-10T11:54:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:50.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:50.935205+0000) 2022-09-10T11:54:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:51 smithi110 ceph-mon[97633]: pgmap v14258: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:51 smithi104 ceph-mon[97044]: pgmap v14258: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:54:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:51.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:51.460595+0000) 2022-09-10T11:54:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:51.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:51.935411+0000) 2022-09-10T11:54:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:52.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:52.460822+0000) 2022-09-10T11:54:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:52.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:52.935576+0000) 2022-09-10T11:54:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:53 smithi104 ceph-mon[97044]: pgmap v14259: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:53.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:53.461019+0000) 2022-09-10T11:54:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:53 smithi110 ceph-mon[97633]: pgmap v14259: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:53.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:53.935779+0000) 2022-09-10T11:54:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:54.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:54.461248+0000) 2022-09-10T11:54:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:54.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:54.935975+0000) 2022-09-10T11:54:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:55 smithi110 ceph-mon[97633]: pgmap v14260: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:54:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:55 smithi104 ceph-mon[97044]: pgmap v14260: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T11:54:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:55.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:55.461410+0000) 2022-09-10T11:54:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:55.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:55.936210+0000) 2022-09-10T11:54:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:56.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:56.461631+0000) 2022-09-10T11:54:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:56.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:56.936498+0000) 2022-09-10T11:54:57.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:57 smithi110 ceph-mon[97633]: pgmap v14261: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:57 smithi104 ceph-mon[97044]: pgmap v14261: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T11:54:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:57.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:57.461811+0000) 2022-09-10T11:54:58.247 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:57.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:57.936761+0000) 2022-09-10T11:54:58.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:54:58.246+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:54:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:58.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:58.462050+0000) 2022-09-10T11:54:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:58.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:58.936973+0000) 2022-09-10T11:54:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:54:59 smithi104 ceph-mon[97044]: pgmap v14262: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:54:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:54:59.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:59.462329+0000) 2022-09-10T11:54:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:54:59 smithi110 ceph-mon[97633]: pgmap v14262: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:54:59.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:54:59.937163+0000) 2022-09-10T11:55:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:00.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:00.462571+0000) 2022-09-10T11:55:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:00.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:00.937286+0000) 2022-09-10T11:55:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:01 smithi104 ceph-mon[97044]: pgmap v14263: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:01.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:01.462741+0000) 2022-09-10T11:55:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:01 smithi110 ceph-mon[97633]: pgmap v14263: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:01.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:01.937519+0000) 2022-09-10T11:55:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:02.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:02.462934+0000) 2022-09-10T11:55:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:02.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:02.937741+0000) 2022-09-10T11:55:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:03 smithi104 ceph-mon[97044]: pgmap v14264: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:03.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:03.463133+0000) 2022-09-10T11:55:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:03 smithi110 ceph-mon[97633]: pgmap v14264: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:03.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:03.937978+0000) 2022-09-10T11:55:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:04.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:04.463317+0000) 2022-09-10T11:55:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:04.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:04.938252+0000) 2022-09-10T11:55:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:05 smithi110 ceph-mon[97633]: pgmap v14265: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T11:55:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:05 smithi104 ceph-mon[97044]: pgmap v14265: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T11:55:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:05.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:05.463548+0000) 2022-09-10T11:55:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:05.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:05.938512+0000) 2022-09-10T11:55:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:06.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:06.463721+0000) 2022-09-10T11:55:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:06.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:06.938759+0000) 2022-09-10T11:55:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:07 smithi104 ceph-mon[97044]: pgmap v14266: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T11:55:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:07.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:07.463916+0000) 2022-09-10T11:55:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:07 smithi110 ceph-mon[97633]: pgmap v14266: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T11:55:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:07.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:07.938979+0000) 2022-09-10T11:55:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:08.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:08.464145+0000) 2022-09-10T11:55:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:08.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:08.939220+0000) 2022-09-10T11:55:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:09 smithi104 ceph-mon[97044]: pgmap v14267: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:55:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:09 smithi104 ceph-mon[97044]: osdmap e972: 12 total, 12 up, 12 in 2022-09-10T11:55:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:09.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:09.464310+0000) 2022-09-10T11:55:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:09 smithi110 ceph-mon[97633]: pgmap v14267: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:55:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:09 smithi110 ceph-mon[97633]: osdmap e972: 12 total, 12 up, 12 in 2022-09-10T11:55:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:09.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:09.939464+0000) 2022-09-10T11:55:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:10.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:10.464559+0000) 2022-09-10T11:55:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:10.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:10.939714+0000) 2022-09-10T11:55:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:11 smithi104 ceph-mon[97044]: pgmap v14269: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:55:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:11.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:11.464796+0000) 2022-09-10T11:55:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:11 smithi110 ceph-mon[97633]: pgmap v14269: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:55:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:11.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:11.939948+0000) 2022-09-10T11:55:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:12.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:12.464964+0000) 2022-09-10T11:55:13.262 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:12.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:12.940179+0000) 2022-09-10T11:55:13.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:55:13.261+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:55:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:13.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:13.465229+0000) 2022-09-10T11:55:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:13 smithi104 ceph-mon[97044]: pgmap v14270: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:55:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:13 smithi110 ceph-mon[97633]: pgmap v14270: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:55:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:13.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:13.940383+0000) 2022-09-10T11:55:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:14.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:14.465408+0000) 2022-09-10T11:55:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:14.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:14.940591+0000) 2022-09-10T11:55:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:15 smithi110 ceph-mon[97633]: osdmap e973: 12 total, 12 up, 12 in 2022-09-10T11:55:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:15 smithi110 ceph-mon[97633]: pgmap v14272: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:55:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:15 smithi104 ceph-mon[97044]: osdmap e973: 12 total, 12 up, 12 in 2022-09-10T11:55:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:15 smithi104 ceph-mon[97044]: pgmap v14272: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:55:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:15.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:15.465615+0000) 2022-09-10T11:55:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:15.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:15.940789+0000) 2022-09-10T11:55:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:16.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:16.465855+0000) 2022-09-10T11:55:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:16.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:16.941042+0000) 2022-09-10T11:55:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:17 smithi104 ceph-mon[97044]: pgmap v14273: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:55:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:17.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:17.466137+0000) 2022-09-10T11:55:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:17 smithi110 ceph-mon[97633]: pgmap v14273: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:55:18.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:17.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:17.941236+0000) 2022-09-10T11:55:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:18.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:18.466352+0000) 2022-09-10T11:55:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:18.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:18.941460+0000) 2022-09-10T11:55:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:19 smithi104 ceph-mon[97044]: pgmap v14274: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:55:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:19.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:19.466522+0000) 2022-09-10T11:55:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:19 smithi110 ceph-mon[97633]: pgmap v14274: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:55:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:19.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:19.941715+0000) 2022-09-10T11:55:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:20.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:20.466668+0000) 2022-09-10T11:55:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:20.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:20.941922+0000) 2022-09-10T11:55:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:21 smithi104 ceph-mon[97044]: pgmap v14275: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:55:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:21.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:21.466945+0000) 2022-09-10T11:55:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:21 smithi110 ceph-mon[97633]: pgmap v14275: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:55:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:21.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:21.942128+0000) 2022-09-10T11:55:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:22.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:22.467230+0000) 2022-09-10T11:55:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:22.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:22.942318+0000) 2022-09-10T11:55:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:23 smithi104 ceph-mon[97044]: pgmap v14276: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:55:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:23.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:23.467391+0000) 2022-09-10T11:55:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:23 smithi110 ceph-mon[97633]: pgmap v14276: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:55:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:23.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:23.942473+0000) 2022-09-10T11:55:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:24.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:24.467626+0000) 2022-09-10T11:55:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:24.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:24.942748+0000) 2022-09-10T11:55:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:25 smithi110 ceph-mon[97633]: pgmap v14277: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:55:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:25 smithi104 ceph-mon[97044]: pgmap v14277: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:55:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:25.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:25.467831+0000) 2022-09-10T11:55:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:25.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:25.942967+0000) 2022-09-10T11:55:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:26.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:26.468079+0000) 2022-09-10T11:55:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:26.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:26.943177+0000) 2022-09-10T11:55:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:27 smithi104 ceph-mon[97044]: pgmap v14278: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:27.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:27.468391+0000) 2022-09-10T11:55:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:27 smithi110 ceph-mon[97633]: pgmap v14278: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:28.277 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:27.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:27.943378+0000) 2022-09-10T11:55:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:28.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:28.468609+0000) 2022-09-10T11:55:28.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:55:28.275+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:55:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:28.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:28.943645+0000) 2022-09-10T11:55:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:29 smithi104 ceph-mon[97044]: pgmap v14279: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:29.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:29.468756+0000) 2022-09-10T11:55:29.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:29 smithi110 ceph-mon[97633]: pgmap v14279: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:29.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:29.943901+0000) 2022-09-10T11:55:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:30.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:30.468977+0000) 2022-09-10T11:55:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:30.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:30.944173+0000) 2022-09-10T11:55:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:31 smithi104 ceph-mon[97044]: pgmap v14280: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:31.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:31.469185+0000) 2022-09-10T11:55:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:31 smithi110 ceph-mon[97633]: pgmap v14280: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:31.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:31.944331+0000) 2022-09-10T11:55:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:32.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:32.469429+0000) 2022-09-10T11:55:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:32.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:32.944576+0000) 2022-09-10T11:55:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:33 smithi104 ceph-mon[97044]: pgmap v14281: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:33.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:33.469673+0000) 2022-09-10T11:55:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:33 smithi110 ceph-mon[97633]: pgmap v14281: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:33.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:33.944742+0000) 2022-09-10T11:55:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:34.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:34.469841+0000) 2022-09-10T11:55:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:34.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:34.944925+0000) 2022-09-10T11:55:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:35 smithi110 ceph-mon[97633]: pgmap v14282: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:55:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:35 smithi104 ceph-mon[97044]: pgmap v14282: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:55:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:35.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:35.470053+0000) 2022-09-10T11:55:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:35.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:35.945168+0000) 2022-09-10T11:55:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:36.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:36.470189+0000) 2022-09-10T11:55:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:36.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:36.945415+0000) 2022-09-10T11:55:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:37 smithi104 ceph-mon[97044]: pgmap v14283: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:37.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:37.470424+0000) 2022-09-10T11:55:37.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:37 smithi110 ceph-mon[97633]: pgmap v14283: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:37.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:37.945641+0000) 2022-09-10T11:55:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:38.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:38.470721+0000) 2022-09-10T11:55:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:38.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:38.945860+0000) 2022-09-10T11:55:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:39 smithi104 ceph-mon[97044]: pgmap v14284: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:39.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:39.470986+0000) 2022-09-10T11:55:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:39 smithi110 ceph-mon[97633]: pgmap v14284: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:39.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:39.946129+0000) 2022-09-10T11:55:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:40.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:40.471243+0000) 2022-09-10T11:55:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:40.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:40.946382+0000) 2022-09-10T11:55:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:41 smithi104 ceph-mon[97044]: pgmap v14285: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:41.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:41.471466+0000) 2022-09-10T11:55:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:41 smithi110 ceph-mon[97633]: pgmap v14285: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:41.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:41.946632+0000) 2022-09-10T11:55:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:42.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:42.471761+0000) 2022-09-10T11:55:43.285 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:42.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:42.946838+0000) 2022-09-10T11:55:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:43.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:43.471973+0000) 2022-09-10T11:55:43.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:55:43.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:55:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:43 smithi104 ceph-mon[97044]: pgmap v14286: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:43 smithi110 ceph-mon[97633]: pgmap v14286: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:43.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:43.947003+0000) 2022-09-10T11:55:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:44.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:44.472219+0000) 2022-09-10T11:55:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:44.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:44.947195+0000) 2022-09-10T11:55:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:45 smithi104 ceph-mon[97044]: pgmap v14287: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:55:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:45 smithi110 ceph-mon[97633]: pgmap v14287: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:55:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:45.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:45.472395+0000) 2022-09-10T11:55:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:45.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:45.947437+0000) 2022-09-10T11:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:55:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:55:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:55:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:46.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:46.472616+0000) 2022-09-10T11:55:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:46.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:46.947654+0000) 2022-09-10T11:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:47 smithi104 ceph-mon[97044]: pgmap v14288: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:47 smithi110 ceph-mon[97633]: pgmap v14288: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:47.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:47.472885+0000) 2022-09-10T11:55:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:47.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:47.947849+0000) 2022-09-10T11:55:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:48.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:48.473163+0000) 2022-09-10T11:55:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:48.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:48.948092+0000) 2022-09-10T11:55:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:49 smithi104 ceph-mon[97044]: pgmap v14289: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:49.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:49.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:49.473347+0000) 2022-09-10T11:55:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:49 smithi110 ceph-mon[97633]: pgmap v14289: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:49.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:49.948344+0000) 2022-09-10T11:55:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:50.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:50.473593+0000) 2022-09-10T11:55:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:50.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:50.948507+0000) 2022-09-10T11:55:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:51 smithi104 ceph-mon[97044]: pgmap v14290: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:51.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:51.473873+0000) 2022-09-10T11:55:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:51 smithi110 ceph-mon[97633]: pgmap v14290: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:55:52.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:51.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:51.948723+0000) 2022-09-10T11:55:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:52.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:52.474069+0000) 2022-09-10T11:55:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:52.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:52.948969+0000) 2022-09-10T11:55:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:53 smithi104 ceph-mon[97044]: pgmap v14291: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:53.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:53.474308+0000) 2022-09-10T11:55:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:53 smithi110 ceph-mon[97633]: pgmap v14291: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:53.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:53.949191+0000) 2022-09-10T11:55:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:54.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:54.474622+0000) 2022-09-10T11:55:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:54.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:54.949393+0000) 2022-09-10T11:55:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:55 smithi104 ceph-mon[97044]: pgmap v14292: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:55:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:55 smithi110 ceph-mon[97633]: pgmap v14292: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:55:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:55.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:55.474887+0000) 2022-09-10T11:55:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:55.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:55.949615+0000) 2022-09-10T11:55:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:56.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:56.475061+0000) 2022-09-10T11:55:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:56.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:56.949844+0000) 2022-09-10T11:55:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:57 smithi104 ceph-mon[97044]: pgmap v14293: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:57.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:57.475152+0000) 2022-09-10T11:55:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:57 smithi110 ceph-mon[97633]: pgmap v14293: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:55:58.296 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:57.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:57.950128+0000) 2022-09-10T11:55:58.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:55:58.293+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:55:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:58.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:58.475406+0000) 2022-09-10T11:55:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:58.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:58.950358+0000) 2022-09-10T11:55:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:55:59 smithi104 ceph-mon[97044]: pgmap v14294: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:55:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:55:59.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:59.475669+0000) 2022-09-10T11:55:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:55:59 smithi110 ceph-mon[97633]: pgmap v14294: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:55:59.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:55:59.950546+0000) 2022-09-10T11:56:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:00.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:00.475886+0000) 2022-09-10T11:56:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:00.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:00.950679+0000) 2022-09-10T11:56:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:01 smithi104 ceph-mon[97044]: pgmap v14295: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:56:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:01.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:01.476051+0000) 2022-09-10T11:56:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:01 smithi110 ceph-mon[97633]: pgmap v14295: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:56:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:01.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:01.950867+0000) 2022-09-10T11:56:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:02.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:02.476248+0000) 2022-09-10T11:56:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:02.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:02.951069+0000) 2022-09-10T11:56:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:03 smithi104 ceph-mon[97044]: pgmap v14296: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:03.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:03.476432+0000) 2022-09-10T11:56:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:03 smithi110 ceph-mon[97633]: pgmap v14296: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:03.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:03.951304+0000) 2022-09-10T11:56:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:04.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:04.476672+0000) 2022-09-10T11:56:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:04.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:04.951565+0000) 2022-09-10T11:56:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:05 smithi104 ceph-mon[97044]: pgmap v14297: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T11:56:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:05 smithi110 ceph-mon[97633]: pgmap v14297: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T11:56:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:05.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:05.476918+0000) 2022-09-10T11:56:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:05.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:05.951802+0000) 2022-09-10T11:56:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:06.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:06.477175+0000) 2022-09-10T11:56:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:06.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:06.952047+0000) 2022-09-10T11:56:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:07 smithi104 ceph-mon[97044]: pgmap v14298: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T11:56:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:07.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:07.477412+0000) 2022-09-10T11:56:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:07 smithi110 ceph-mon[97633]: pgmap v14298: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T11:56:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:07.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:07.952293+0000) 2022-09-10T11:56:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:08.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:08.477576+0000) 2022-09-10T11:56:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:08.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:08.952533+0000) 2022-09-10T11:56:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:09 smithi104 ceph-mon[97044]: pgmap v14299: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:56:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:09 smithi104 ceph-mon[97044]: osdmap e974: 12 total, 12 up, 12 in 2022-09-10T11:56:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:09.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:09.477783+0000) 2022-09-10T11:56:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:09 smithi110 ceph-mon[97633]: pgmap v14299: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:56:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:09 smithi110 ceph-mon[97633]: osdmap e974: 12 total, 12 up, 12 in 2022-09-10T11:56:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:09.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:09.952774+0000) 2022-09-10T11:56:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:10.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:10.477985+0000) 2022-09-10T11:56:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:10.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:10.952956+0000) 2022-09-10T11:56:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:11 smithi104 ceph-mon[97044]: pgmap v14301: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:56:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:11.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:11.478202+0000) 2022-09-10T11:56:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:11 smithi110 ceph-mon[97633]: pgmap v14301: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:56:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:11.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:11.953207+0000) 2022-09-10T11:56:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:12.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:12.478386+0000) 2022-09-10T11:56:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:12.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:12.953425+0000) 2022-09-10T11:56:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:13 smithi104 ceph-mon[97044]: pgmap v14302: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:56:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:13.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:13.478555+0000) 2022-09-10T11:56:13.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:56:13.308+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:56:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:13 smithi110 ceph-mon[97633]: pgmap v14302: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:56:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:13.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:13.953628+0000) 2022-09-10T11:56:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:14.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:14.478727+0000) 2022-09-10T11:56:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:14.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:14.953865+0000) 2022-09-10T11:56:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:15 smithi104 ceph-mon[97044]: osdmap e975: 12 total, 12 up, 12 in 2022-09-10T11:56:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:15 smithi104 ceph-mon[97044]: pgmap v14304: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:56:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:15 smithi110 ceph-mon[97633]: osdmap e975: 12 total, 12 up, 12 in 2022-09-10T11:56:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:15 smithi110 ceph-mon[97633]: pgmap v14304: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:56:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:15.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:15.479002+0000) 2022-09-10T11:56:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:15.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:15.954029+0000) 2022-09-10T11:56:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:16.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:16.479222+0000) 2022-09-10T11:56:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:16.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:16.954263+0000) 2022-09-10T11:56:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:17 smithi104 ceph-mon[97044]: pgmap v14305: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:56:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:17.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:17.479467+0000) 2022-09-10T11:56:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:17 smithi110 ceph-mon[97633]: pgmap v14305: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:56:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:17.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:17.954441+0000) 2022-09-10T11:56:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:18.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:18.479738+0000) 2022-09-10T11:56:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:18.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:18.954664+0000) 2022-09-10T11:56:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:19 smithi104 ceph-mon[97044]: pgmap v14306: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:56:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:19.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:19.479914+0000) 2022-09-10T11:56:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:19 smithi110 ceph-mon[97633]: pgmap v14306: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:56:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:19.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:19.954913+0000) 2022-09-10T11:56:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:20.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:20.480091+0000) 2022-09-10T11:56:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:20.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:20.955169+0000) 2022-09-10T11:56:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:21 smithi104 ceph-mon[97044]: pgmap v14307: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:56:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:21.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:21.480390+0000) 2022-09-10T11:56:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:21 smithi110 ceph-mon[97633]: pgmap v14307: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:56:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:21.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:21.955400+0000) 2022-09-10T11:56:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:22.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:22.480676+0000) 2022-09-10T11:56:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:22.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:22.955565+0000) 2022-09-10T11:56:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:23 smithi104 ceph-mon[97044]: pgmap v14308: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:56:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:23.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:23.480985+0000) 2022-09-10T11:56:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:23 smithi110 ceph-mon[97633]: pgmap v14308: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:56:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:23.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:23.955730+0000) 2022-09-10T11:56:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:24.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:24.481259+0000) 2022-09-10T11:56:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:24.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:24.955935+0000) 2022-09-10T11:56:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:25 smithi104 ceph-mon[97044]: pgmap v14309: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:56:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:25 smithi110 ceph-mon[97633]: pgmap v14309: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:56:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:25.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:25.481486+0000) 2022-09-10T11:56:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:25.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:25.956191+0000) 2022-09-10T11:56:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:26.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:26.481750+0000) 2022-09-10T11:56:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:26.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:26.956428+0000) 2022-09-10T11:56:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:27 smithi104 ceph-mon[97044]: pgmap v14310: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:56:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:27.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:27.482005+0000) 2022-09-10T11:56:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:27 smithi110 ceph-mon[97633]: pgmap v14310: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:56:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:27.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:27.956592+0000) 2022-09-10T11:56:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:56:28.317+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:56:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:28.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:28.482226+0000) 2022-09-10T11:56:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:28.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:28.956860+0000) 2022-09-10T11:56:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:29 smithi104 ceph-mon[97044]: pgmap v14311: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:29.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:29.482451+0000) 2022-09-10T11:56:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:29 smithi110 ceph-mon[97633]: pgmap v14311: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:29.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:29.957024+0000) 2022-09-10T11:56:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:30.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:30.482729+0000) 2022-09-10T11:56:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:30.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:30.957256+0000) 2022-09-10T11:56:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:31 smithi104 ceph-mon[97044]: pgmap v14312: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:56:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:31.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:31.482938+0000) 2022-09-10T11:56:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:31 smithi110 ceph-mon[97633]: pgmap v14312: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:56:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:31.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:31.957497+0000) 2022-09-10T11:56:32.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:32.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:32.483194+0000) 2022-09-10T11:56:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:32.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:32.957722+0000) 2022-09-10T11:56:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:33 smithi104 ceph-mon[97044]: pgmap v14313: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:33.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:33.483385+0000) 2022-09-10T11:56:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:33 smithi110 ceph-mon[97633]: pgmap v14313: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:33.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:33.957948+0000) 2022-09-10T11:56:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:34.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:34.483593+0000) 2022-09-10T11:56:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:34.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:34.958177+0000) 2022-09-10T11:56:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:35 smithi104 ceph-mon[97044]: pgmap v14314: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:56:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:35 smithi110 ceph-mon[97633]: pgmap v14314: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:56:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:35.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:35.483738+0000) 2022-09-10T11:56:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:35.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:35.958381+0000) 2022-09-10T11:56:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:36.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:36.483980+0000) 2022-09-10T11:56:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:36.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:36.958603+0000) 2022-09-10T11:56:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:37 smithi104 ceph-mon[97044]: pgmap v14315: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:56:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:37.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:37.484246+0000) 2022-09-10T11:56:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:37 smithi110 ceph-mon[97633]: pgmap v14315: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:56:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:37.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:37.958828+0000) 2022-09-10T11:56:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:38.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:38.484464+0000) 2022-09-10T11:56:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:38.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:38.959063+0000) 2022-09-10T11:56:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:39 smithi104 ceph-mon[97044]: pgmap v14316: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:39.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:39.484640+0000) 2022-09-10T11:56:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:39 smithi110 ceph-mon[97633]: pgmap v14316: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:39.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:39.959316+0000) 2022-09-10T11:56:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:40.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:40.484808+0000) 2022-09-10T11:56:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:40.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:40.959571+0000) 2022-09-10T11:56:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:41 smithi104 ceph-mon[97044]: pgmap v14317: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:56:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:41.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:41.485043+0000) 2022-09-10T11:56:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:41 smithi110 ceph-mon[97633]: pgmap v14317: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:56:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:41.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:41.959801+0000) 2022-09-10T11:56:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:42.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:42.485249+0000) 2022-09-10T11:56:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:42.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:42.960044+0000) 2022-09-10T11:56:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:43 smithi104 ceph-mon[97044]: pgmap v14318: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:43.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:43.485448+0000) 2022-09-10T11:56:43.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:56:43.327+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:56:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:43 smithi110 ceph-mon[97633]: pgmap v14318: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:56:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:43.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:43.960302+0000) 2022-09-10T11:56:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:44.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:44.485694+0000) 2022-09-10T11:56:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:44.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:44.960486+0000) 2022-09-10T11:56:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:45 smithi104 ceph-mon[97044]: pgmap v14319: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T11:56:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:45 smithi110 ceph-mon[97633]: pgmap v14319: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2022-09-10T11:56:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:45.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:45.485936+0000) 2022-09-10T11:56:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:45.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:45.960693+0000) 2022-09-10T11:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:56:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:56:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:56:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:56:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:56:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:46.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:46.486211+0000) 2022-09-10T11:56:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:46.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:46.960900+0000) 2022-09-10T11:56:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:47 smithi104 ceph-mon[97044]: pgmap v14320: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s 2022-09-10T11:56:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:47 smithi110 ceph-mon[97633]: pgmap v14320: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s 2022-09-10T11:56:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:47.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:47.486478+0000) 2022-09-10T11:56:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:47.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:47.961085+0000) 2022-09-10T11:56:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:48.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:48.486725+0000) 2022-09-10T11:56:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:48.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:48.961300+0000) 2022-09-10T11:56:49.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:49 smithi104 ceph-mon[97044]: pgmap v14321: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T11:56:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:49.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:49.486971+0000) 2022-09-10T11:56:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:49 smithi110 ceph-mon[97633]: pgmap v14321: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T11:56:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:49.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:49.961461+0000) 2022-09-10T11:56:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:50.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:50.487202+0000) 2022-09-10T11:56:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:50.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:50.961729+0000) 2022-09-10T11:56:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:51 smithi104 ceph-mon[97044]: pgmap v14322: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T11:56:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:51.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:51.487394+0000) 2022-09-10T11:56:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:51 smithi110 ceph-mon[97633]: pgmap v14322: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T11:56:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:51.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:51.961895+0000) 2022-09-10T11:56:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:52.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:52.487555+0000) 2022-09-10T11:56:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:52.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:52.962162+0000) 2022-09-10T11:56:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:53 smithi104 ceph-mon[97044]: pgmap v14323: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2022-09-10T11:56:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:53.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:53.487713+0000) 2022-09-10T11:56:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:53 smithi110 ceph-mon[97633]: pgmap v14323: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2022-09-10T11:56:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:53.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:53.962370+0000) 2022-09-10T11:56:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:54.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:54.487969+0000) 2022-09-10T11:56:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:54.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:54.962606+0000) 2022-09-10T11:56:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:55 smithi104 ceph-mon[97044]: pgmap v14324: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T11:56:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:55 smithi110 ceph-mon[97633]: pgmap v14324: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2022-09-10T11:56:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:55.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:55.488176+0000) 2022-09-10T11:56:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:55.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:55.962819+0000) 2022-09-10T11:56:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:56.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:56.488447+0000) 2022-09-10T11:56:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:56.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:56.963048+0000) 2022-09-10T11:56:57.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:57 smithi104 ceph-mon[97044]: pgmap v14325: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s 2022-09-10T11:56:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:57.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:57.488581+0000) 2022-09-10T11:56:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:57 smithi110 ceph-mon[97633]: pgmap v14325: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s 2022-09-10T11:56:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:57.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:57.963296+0000) 2022-09-10T11:56:58.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:58.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:58.488790+0000) 2022-09-10T11:56:58.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:56:58.342+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:56:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:58.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:58.963521+0000) 2022-09-10T11:56:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:56:59 smithi104 ceph-mon[97044]: pgmap v14326: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T11:56:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:56:59.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:59.488924+0000) 2022-09-10T11:56:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:56:59 smithi110 ceph-mon[97633]: pgmap v14326: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T11:57:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:56:59.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:56:59.963712+0000) 2022-09-10T11:57:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:00.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:00.489147+0000) 2022-09-10T11:57:01.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:00.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:00.963830+0000) 2022-09-10T11:57:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:01 smithi104 ceph-mon[97044]: pgmap v14327: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T11:57:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:01.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:01.489393+0000) 2022-09-10T11:57:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:01 smithi110 ceph-mon[97633]: pgmap v14327: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T11:57:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:01.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:01.964061+0000) 2022-09-10T11:57:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:02.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:02.489658+0000) 2022-09-10T11:57:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:02.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:02.964189+0000) 2022-09-10T11:57:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:03 smithi104 ceph-mon[97044]: pgmap v14328: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:57:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:03.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:03.489838+0000) 2022-09-10T11:57:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:03 smithi110 ceph-mon[97633]: pgmap v14328: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T11:57:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:03.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:03.964426+0000) 2022-09-10T11:57:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:04.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:04.490004+0000) 2022-09-10T11:57:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:04.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:04.964692+0000) 2022-09-10T11:57:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:05 smithi104 ceph-mon[97044]: pgmap v14329: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2022-09-10T11:57:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:05 smithi110 ceph-mon[97633]: pgmap v14329: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2022-09-10T11:57:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:05.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:05.490221+0000) 2022-09-10T11:57:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:05.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:05.964848+0000) 2022-09-10T11:57:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:06.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:06.490423+0000) 2022-09-10T11:57:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:06.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:06.965070+0000) 2022-09-10T11:57:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:07 smithi104 ceph-mon[97044]: pgmap v14330: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T11:57:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:07.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:07.490591+0000) 2022-09-10T11:57:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:07 smithi110 ceph-mon[97633]: pgmap v14330: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T11:57:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:07.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:07.965363+0000) 2022-09-10T11:57:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:08.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:08.490841+0000) 2022-09-10T11:57:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:08.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:08.965573+0000) 2022-09-10T11:57:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:09 smithi104 ceph-mon[97044]: pgmap v14331: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:57:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:09 smithi104 ceph-mon[97044]: osdmap e976: 12 total, 12 up, 12 in 2022-09-10T11:57:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:09.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:09.491051+0000) 2022-09-10T11:57:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:09 smithi110 ceph-mon[97633]: pgmap v14331: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:57:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:09 smithi110 ceph-mon[97633]: osdmap e976: 12 total, 12 up, 12 in 2022-09-10T11:57:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:09.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:09.965737+0000) 2022-09-10T11:57:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:10.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:10.491267+0000) 2022-09-10T11:57:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:10.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:10.965995+0000) 2022-09-10T11:57:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:11 smithi104 ceph-mon[97044]: pgmap v14333: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:57:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:11.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:11.491475+0000) 2022-09-10T11:57:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:11 smithi110 ceph-mon[97633]: pgmap v14333: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:57:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:11.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:11.966225+0000) 2022-09-10T11:57:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:12.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:12.491658+0000) 2022-09-10T11:57:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:12.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:12.966437+0000) 2022-09-10T11:57:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:13 smithi104 ceph-mon[97044]: pgmap v14334: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:57:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:13.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:13.491911+0000) 2022-09-10T11:57:13.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:57:13.353+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:57:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:13 smithi110 ceph-mon[97633]: pgmap v14334: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:57:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:13.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:13.966712+0000) 2022-09-10T11:57:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:14.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:14.492157+0000) 2022-09-10T11:57:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:14.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:14.966945+0000) 2022-09-10T11:57:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:15 smithi104 ceph-mon[97044]: osdmap e977: 12 total, 12 up, 12 in 2022-09-10T11:57:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:15 smithi104 ceph-mon[97044]: pgmap v14336: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:57:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:15 smithi110 ceph-mon[97633]: osdmap e977: 12 total, 12 up, 12 in 2022-09-10T11:57:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:15 smithi110 ceph-mon[97633]: pgmap v14336: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:57:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:15.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:15.492373+0000) 2022-09-10T11:57:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:15.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:15.967188+0000) 2022-09-10T11:57:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:16.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:16.492594+0000) 2022-09-10T11:57:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:16.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:16.967416+0000) 2022-09-10T11:57:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:17 smithi104 ceph-mon[97044]: pgmap v14337: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:57:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:17.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:17.492805+0000) 2022-09-10T11:57:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:17 smithi110 ceph-mon[97633]: pgmap v14337: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:57:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:17.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:17.967671+0000) 2022-09-10T11:57:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:18.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:18.493053+0000) 2022-09-10T11:57:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:18.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:18.967871+0000) 2022-09-10T11:57:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:19 smithi104 ceph-mon[97044]: pgmap v14338: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:57:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:19.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:19.493241+0000) 2022-09-10T11:57:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:19 smithi110 ceph-mon[97633]: pgmap v14338: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T11:57:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:19.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:19.968156+0000) 2022-09-10T11:57:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:20.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:20.493433+0000) 2022-09-10T11:57:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:20.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:20.968382+0000) 2022-09-10T11:57:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:21 smithi104 ceph-mon[97044]: pgmap v14339: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:57:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:21.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:21.493592+0000) 2022-09-10T11:57:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:21 smithi110 ceph-mon[97633]: pgmap v14339: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:57:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:21.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:21.968543+0000) 2022-09-10T11:57:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:22.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:22.493825+0000) 2022-09-10T11:57:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:22.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:22.968779+0000) 2022-09-10T11:57:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:23 smithi104 ceph-mon[97044]: pgmap v14340: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:57:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:23.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:23.494057+0000) 2022-09-10T11:57:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:23 smithi110 ceph-mon[97633]: pgmap v14340: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:57:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:23.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:23.969023+0000) 2022-09-10T11:57:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:24.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:24.494219+0000) 2022-09-10T11:57:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:24.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:24.969269+0000) 2022-09-10T11:57:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:25 smithi104 ceph-mon[97044]: pgmap v14341: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:57:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:25 smithi110 ceph-mon[97633]: pgmap v14341: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:57:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:25.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:25.494390+0000) 2022-09-10T11:57:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:25.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:25.969489+0000) 2022-09-10T11:57:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:26.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:26.494637+0000) 2022-09-10T11:57:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:26.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:26.969705+0000) 2022-09-10T11:57:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:27 smithi104 ceph-mon[97044]: pgmap v14342: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:27.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:27.494874+0000) 2022-09-10T11:57:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:27 smithi110 ceph-mon[97633]: pgmap v14342: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:27.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:27.969983+0000) 2022-09-10T11:57:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:57:28.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:57:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:28.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:28.495170+0000) 2022-09-10T11:57:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:28.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:28.970219+0000) 2022-09-10T11:57:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:29 smithi104 ceph-mon[97044]: pgmap v14343: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:29.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:29.495377+0000) 2022-09-10T11:57:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:29 smithi110 ceph-mon[97633]: pgmap v14343: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:29.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:29.970469+0000) 2022-09-10T11:57:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:30.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:30.495635+0000) 2022-09-10T11:57:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:30.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:30.970676+0000) 2022-09-10T11:57:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:31 smithi104 ceph-mon[97044]: pgmap v14344: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:57:31.812 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:31.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:31.495880+0000) 2022-09-10T11:57:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:31 smithi110 ceph-mon[97633]: pgmap v14344: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:57:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:31.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:31.970901+0000) 2022-09-10T11:57:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:32.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:32.496056+0000) 2022-09-10T11:57:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:32.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:32.971113+0000) 2022-09-10T11:57:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:33 smithi104 ceph-mon[97044]: pgmap v14345: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:33.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:33.496203+0000) 2022-09-10T11:57:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:33 smithi110 ceph-mon[97633]: pgmap v14345: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:33.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:33.971360+0000) 2022-09-10T11:57:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:34.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:34.496375+0000) 2022-09-10T11:57:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:34.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:34.971579+0000) 2022-09-10T11:57:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:35 smithi104 ceph-mon[97044]: pgmap v14346: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:57:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:35 smithi110 ceph-mon[97633]: pgmap v14346: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:57:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:35.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:35.496551+0000) 2022-09-10T11:57:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:35.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:35.971777+0000) 2022-09-10T11:57:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:36.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:36.496750+0000) 2022-09-10T11:57:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:36.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:36.971975+0000) 2022-09-10T11:57:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:37 smithi104 ceph-mon[97044]: pgmap v14347: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:37.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:37.496948+0000) 2022-09-10T11:57:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:37 smithi110 ceph-mon[97633]: pgmap v14347: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:37.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:37.972197+0000) 2022-09-10T11:57:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:38.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:38.497202+0000) 2022-09-10T11:57:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:38.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:38.972372+0000) 2022-09-10T11:57:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:39 smithi104 ceph-mon[97044]: pgmap v14348: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:39.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:39.497427+0000) 2022-09-10T11:57:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:39 smithi110 ceph-mon[97633]: pgmap v14348: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:39.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:39.972564+0000) 2022-09-10T11:57:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:40.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:40.497692+0000) 2022-09-10T11:57:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:40.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:40.972698+0000) 2022-09-10T11:57:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:41 smithi104 ceph-mon[97044]: pgmap v14349: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:57:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:41.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:41.497959+0000) 2022-09-10T11:57:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:41 smithi110 ceph-mon[97633]: pgmap v14349: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:57:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:41.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:41.972926+0000) 2022-09-10T11:57:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:42.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:42.498169+0000) 2022-09-10T11:57:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:42.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:42.973132+0000) 2022-09-10T11:57:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:43 smithi104 ceph-mon[97044]: pgmap v14350: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:43.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:43.498342+0000) 2022-09-10T11:57:43.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:57:43.371+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:57:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:43 smithi110 ceph-mon[97633]: pgmap v14350: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:43.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:43.973287+0000) 2022-09-10T11:57:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:44.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:44.498486+0000) 2022-09-10T11:57:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:44.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:44.973483+0000) 2022-09-10T11:57:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:45 smithi104 ceph-mon[97044]: pgmap v14351: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:57:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:45 smithi110 ceph-mon[97633]: pgmap v14351: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:57:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:45.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:45.498683+0000) 2022-09-10T11:57:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:45.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:45.973689+0000) 2022-09-10T11:57:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:57:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:57:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:46.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:46.498928+0000) 2022-09-10T11:57:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:46.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:46.973882+0000) 2022-09-10T11:57:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:47 smithi104 ceph-mon[97044]: pgmap v14352: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:47 smithi110 ceph-mon[97633]: pgmap v14352: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:47.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:47.499228+0000) 2022-09-10T11:57:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:47.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:47.974124+0000) 2022-09-10T11:57:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:48.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:48.499414+0000) 2022-09-10T11:57:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:48.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:48.974338+0000) 2022-09-10T11:57:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:49 smithi104 ceph-mon[97044]: pgmap v14353: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:49.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:49.499679+0000) 2022-09-10T11:57:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:49 smithi110 ceph-mon[97633]: pgmap v14353: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:49.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:49.974590+0000) 2022-09-10T11:57:50.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:50.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:50.499946+0000) 2022-09-10T11:57:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:50.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:50.974768+0000) 2022-09-10T11:57:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:51 smithi104 ceph-mon[97044]: pgmap v14354: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:57:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:51.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:51.500150+0000) 2022-09-10T11:57:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:51 smithi110 ceph-mon[97633]: pgmap v14354: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:57:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:51.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:51.974983+0000) 2022-09-10T11:57:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:52.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:52.500323+0000) 2022-09-10T11:57:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:52.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:52.975221+0000) 2022-09-10T11:57:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:53 smithi104 ceph-mon[97044]: pgmap v14355: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:53.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:53.500502+0000) 2022-09-10T11:57:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:53 smithi110 ceph-mon[97633]: pgmap v14355: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:53.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:53.975398+0000) 2022-09-10T11:57:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:54.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:54.500753+0000) 2022-09-10T11:57:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:54.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:54.975641+0000) 2022-09-10T11:57:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:55 smithi104 ceph-mon[97044]: pgmap v14356: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:57:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:55 smithi110 ceph-mon[97633]: pgmap v14356: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:57:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:55.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:55.501014+0000) 2022-09-10T11:57:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:55.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:55.975886+0000) 2022-09-10T11:57:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:56.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:56.501193+0000) 2022-09-10T11:57:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:56.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:56.976138+0000) 2022-09-10T11:57:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:57 smithi104 ceph-mon[97044]: pgmap v14357: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:57.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:57.501450+0000) 2022-09-10T11:57:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:57 smithi110 ceph-mon[97633]: pgmap v14357: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:57:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:57.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:57.976333+0000) 2022-09-10T11:57:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:57:58.386+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:57:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:58.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:58.501728+0000) 2022-09-10T11:57:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:58.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:58.976511+0000) 2022-09-10T11:57:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:57:59 smithi104 ceph-mon[97044]: pgmap v14358: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:57:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:57:59.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:59.501988+0000) 2022-09-10T11:57:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:57:59 smithi110 ceph-mon[97633]: pgmap v14358: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:57:59.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:57:59.976711+0000) 2022-09-10T11:58:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:00.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:00.502211+0000) 2022-09-10T11:58:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:00.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:00.976936+0000) 2022-09-10T11:58:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:01 smithi104 ceph-mon[97044]: pgmap v14359: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:01.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:01.502409+0000) 2022-09-10T11:58:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:01 smithi110 ceph-mon[97633]: pgmap v14359: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:01.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:01.977187+0000) 2022-09-10T11:58:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:02.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:02.502607+0000) 2022-09-10T11:58:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:02.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:02.977367+0000) 2022-09-10T11:58:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:03 smithi104 ceph-mon[97044]: pgmap v14360: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:03.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:03.502868+0000) 2022-09-10T11:58:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:03 smithi110 ceph-mon[97633]: pgmap v14360: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:03.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:03.977583+0000) 2022-09-10T11:58:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:04.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:04.503175+0000) 2022-09-10T11:58:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:04.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:04.977762+0000) 2022-09-10T11:58:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:05 smithi104 ceph-mon[97044]: pgmap v14361: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T11:58:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:05 smithi110 ceph-mon[97633]: pgmap v14361: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T11:58:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:05.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:05.503391+0000) 2022-09-10T11:58:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:05.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:05.977981+0000) 2022-09-10T11:58:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:06.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:06.503679+0000) 2022-09-10T11:58:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:06.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:06.978160+0000) 2022-09-10T11:58:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:07 smithi104 ceph-mon[97044]: pgmap v14362: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T11:58:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:07.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:07.504009+0000) 2022-09-10T11:58:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:07 smithi110 ceph-mon[97633]: pgmap v14362: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T11:58:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:07.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:07.978339+0000) 2022-09-10T11:58:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:08.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:08.504245+0000) 2022-09-10T11:58:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:08.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:08.978596+0000) 2022-09-10T11:58:09.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:09 smithi104 ceph-mon[97044]: pgmap v14363: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:58:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:09 smithi104 ceph-mon[97044]: osdmap e978: 12 total, 12 up, 12 in 2022-09-10T11:58:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:09.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:09.504428+0000) 2022-09-10T11:58:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:09 smithi110 ceph-mon[97633]: pgmap v14363: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:58:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:09 smithi110 ceph-mon[97633]: osdmap e978: 12 total, 12 up, 12 in 2022-09-10T11:58:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:09.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:09.978844+0000) 2022-09-10T11:58:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:10.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:10.504710+0000) 2022-09-10T11:58:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:10.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:10.979128+0000) 2022-09-10T11:58:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:11 smithi104 ceph-mon[97044]: pgmap v14365: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:58:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:11.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:11.504936+0000) 2022-09-10T11:58:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:11 smithi110 ceph-mon[97633]: pgmap v14365: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:58:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:11.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:11.979337+0000) 2022-09-10T11:58:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:12.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:12.505216+0000) 2022-09-10T11:58:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:12.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:12.979570+0000) 2022-09-10T11:58:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:13 smithi104 ceph-mon[97044]: pgmap v14366: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:58:13.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:58:13.401+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:58:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:13.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:13.505431+0000) 2022-09-10T11:58:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:13 smithi110 ceph-mon[97633]: pgmap v14366: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:58:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:13.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:13.979828+0000) 2022-09-10T11:58:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:14.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:14.505632+0000) 2022-09-10T11:58:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:14.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:14.980059+0000) 2022-09-10T11:58:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:15 smithi104 ceph-mon[97044]: osdmap e979: 12 total, 12 up, 12 in 2022-09-10T11:58:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:15 smithi104 ceph-mon[97044]: pgmap v14368: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:58:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:15 smithi110 ceph-mon[97633]: osdmap e979: 12 total, 12 up, 12 in 2022-09-10T11:58:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:15 smithi110 ceph-mon[97633]: pgmap v14368: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 9 op/s 2022-09-10T11:58:15.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:15.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:15.505810+0000) 2022-09-10T11:58:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:15.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:15.980359+0000) 2022-09-10T11:58:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:16.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:16.506055+0000) 2022-09-10T11:58:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:16.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:16.980526+0000) 2022-09-10T11:58:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:17 smithi104 ceph-mon[97044]: pgmap v14369: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:58:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:17.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:17.506233+0000) 2022-09-10T11:58:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:17 smithi110 ceph-mon[97633]: pgmap v14369: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:58:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:17.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:17.980756+0000) 2022-09-10T11:58:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:18.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:18.506480+0000) 2022-09-10T11:58:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:18.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:18.981018+0000) 2022-09-10T11:58:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:19 smithi104 ceph-mon[97044]: pgmap v14370: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:58:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:19.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:19.506747+0000) 2022-09-10T11:58:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:19 smithi110 ceph-mon[97633]: pgmap v14370: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:58:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:19.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:19.981189+0000) 2022-09-10T11:58:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:20.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:20.506932+0000) 2022-09-10T11:58:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:20.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:20.981488+0000) 2022-09-10T11:58:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:21 smithi104 ceph-mon[97044]: pgmap v14371: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:58:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:21.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:21.507167+0000) 2022-09-10T11:58:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:21 smithi110 ceph-mon[97633]: pgmap v14371: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:58:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:21.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:21.981730+0000) 2022-09-10T11:58:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:22.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:22.507392+0000) 2022-09-10T11:58:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:22.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:22.981999+0000) 2022-09-10T11:58:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:23 smithi104 ceph-mon[97044]: pgmap v14372: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:58:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:23.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:23.507621+0000) 2022-09-10T11:58:23.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:23 smithi110 ceph-mon[97633]: pgmap v14372: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:58:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:23.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:23.982262+0000) 2022-09-10T11:58:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:24.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:24.507819+0000) 2022-09-10T11:58:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:24.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:24.982514+0000) 2022-09-10T11:58:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:25 smithi104 ceph-mon[97044]: pgmap v14373: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:58:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:25 smithi110 ceph-mon[97633]: pgmap v14373: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 100 B/s wr, 12 op/s 2022-09-10T11:58:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:25.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:25.508048+0000) 2022-09-10T11:58:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:25.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:25.982677+0000) 2022-09-10T11:58:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:26.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:26.508271+0000) 2022-09-10T11:58:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:26.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:26.982848+0000) 2022-09-10T11:58:27.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:27 smithi104 ceph-mon[97044]: pgmap v14374: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:58:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:27.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:27.508453+0000) 2022-09-10T11:58:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:27 smithi110 ceph-mon[97633]: pgmap v14374: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:58:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:27.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:27.983034+0000) 2022-09-10T11:58:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:28.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:28.508705+0000) 2022-09-10T11:58:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:58:28.417+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:58:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:28.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:28.983267+0000) 2022-09-10T11:58:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:29 smithi104 ceph-mon[97044]: pgmap v14375: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:29.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:29.508945+0000) 2022-09-10T11:58:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:29 smithi110 ceph-mon[97633]: pgmap v14375: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:29.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:29.983462+0000) 2022-09-10T11:58:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:30.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:30.509160+0000) 2022-09-10T11:58:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:30.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:30.983694+0000) 2022-09-10T11:58:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:31 smithi104 ceph-mon[97044]: pgmap v14376: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:31.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:31.509330+0000) 2022-09-10T11:58:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:31 smithi110 ceph-mon[97633]: pgmap v14376: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:31.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:31.983914+0000) 2022-09-10T11:58:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:32.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:32.509525+0000) 2022-09-10T11:58:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:32.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:32.984165+0000) 2022-09-10T11:58:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:33 smithi104 ceph-mon[97044]: pgmap v14377: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:33.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:33.509757+0000) 2022-09-10T11:58:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:33 smithi110 ceph-mon[97633]: pgmap v14377: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:33.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:33.984392+0000) 2022-09-10T11:58:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:34.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:34.510000+0000) 2022-09-10T11:58:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:34.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:34.984647+0000) 2022-09-10T11:58:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:35 smithi104 ceph-mon[97044]: pgmap v14378: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:58:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:35 smithi110 ceph-mon[97633]: pgmap v14378: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2022-09-10T11:58:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:35.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:35.510273+0000) 2022-09-10T11:58:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:35.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:35.984903+0000) 2022-09-10T11:58:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:36.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:36.510521+0000) 2022-09-10T11:58:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:36.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:36.985171+0000) 2022-09-10T11:58:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:37 smithi104 ceph-mon[97044]: pgmap v14379: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:58:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:37.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:37.510692+0000) 2022-09-10T11:58:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:37 smithi110 ceph-mon[97633]: pgmap v14379: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:58:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:37.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:37.985330+0000) 2022-09-10T11:58:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:38.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:38.510903+0000) 2022-09-10T11:58:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:38.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:38.985527+0000) 2022-09-10T11:58:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:39 smithi104 ceph-mon[97044]: pgmap v14380: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:39.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:39.511141+0000) 2022-09-10T11:58:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:39 smithi110 ceph-mon[97633]: pgmap v14380: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:39.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:39.985737+0000) 2022-09-10T11:58:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:40.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:40.511388+0000) 2022-09-10T11:58:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:40.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:40.986012+0000) 2022-09-10T11:58:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:41 smithi104 ceph-mon[97044]: pgmap v14381: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:41.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:41.511620+0000) 2022-09-10T11:58:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:41 smithi110 ceph-mon[97633]: pgmap v14381: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:41.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:41.986202+0000) 2022-09-10T11:58:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:42.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:42.511873+0000) 2022-09-10T11:58:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:42.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:42.986424+0000) 2022-09-10T11:58:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:43 smithi104 ceph-mon[97044]: pgmap v14382: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:43.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:43.512130+0000) 2022-09-10T11:58:43.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:58:43.432+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:58:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:43 smithi110 ceph-mon[97633]: pgmap v14382: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:43.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:43.986559+0000) 2022-09-10T11:58:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:44.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:44.512357+0000) 2022-09-10T11:58:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:44.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:44.986743+0000) 2022-09-10T11:58:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:45 smithi104 ceph-mon[97044]: pgmap v14383: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:45 smithi110 ceph-mon[97633]: pgmap v14383: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:45.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:45.512567+0000) 2022-09-10T11:58:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:45.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:45.986981+0000) 2022-09-10T11:58:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:58:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:58:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:58:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:58:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:58:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:46.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:46.512814+0000) 2022-09-10T11:58:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:46.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:46.987230+0000) 2022-09-10T11:58:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:47 smithi104 ceph-mon[97044]: pgmap v14384: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:58:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:47 smithi110 ceph-mon[97633]: pgmap v14384: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2022-09-10T11:58:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:47.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:47.513034+0000) 2022-09-10T11:58:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:47.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:47.987476+0000) 2022-09-10T11:58:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:48.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:48.513263+0000) 2022-09-10T11:58:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:48.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:48.987674+0000) 2022-09-10T11:58:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:49 smithi104 ceph-mon[97044]: pgmap v14385: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:49.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:49.513488+0000) 2022-09-10T11:58:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:49 smithi110 ceph-mon[97633]: pgmap v14385: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:49.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:49.987848+0000) 2022-09-10T11:58:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:50.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:50.513634+0000) 2022-09-10T11:58:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:50.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:50.988079+0000) 2022-09-10T11:58:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:51 smithi104 ceph-mon[97044]: pgmap v14386: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:51.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:51.513864+0000) 2022-09-10T11:58:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:51 smithi110 ceph-mon[97633]: pgmap v14386: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:51.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:51.988308+0000) 2022-09-10T11:58:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:52.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:52.514145+0000) 2022-09-10T11:58:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:52.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:52.988537+0000) 2022-09-10T11:58:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:53 smithi104 ceph-mon[97044]: pgmap v14387: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:53.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:53.514285+0000) 2022-09-10T11:58:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:53 smithi110 ceph-mon[97633]: pgmap v14387: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:53.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:53.988783+0000) 2022-09-10T11:58:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:54.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:54.514465+0000) 2022-09-10T11:58:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:54.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:54.989026+0000) 2022-09-10T11:58:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:55 smithi104 ceph-mon[97044]: pgmap v14388: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:55 smithi110 ceph-mon[97633]: pgmap v14388: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:58:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:55.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:55.514732+0000) 2022-09-10T11:58:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:55.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:55.989318+0000) 2022-09-10T11:58:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:56.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:56.514928+0000) 2022-09-10T11:58:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:56.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:56.989528+0000) 2022-09-10T11:58:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:57 smithi104 ceph-mon[97044]: pgmap v14389: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:57.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:57.515083+0000) 2022-09-10T11:58:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:57 smithi110 ceph-mon[97633]: pgmap v14389: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:57.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:57.989751+0000) 2022-09-10T11:58:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:58.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:58.515278+0000) 2022-09-10T11:58:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:58:58.441+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:58:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:58.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:58.989957+0000) 2022-09-10T11:58:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:58:59 smithi104 ceph-mon[97044]: pgmap v14390: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:58:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:58:59.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:59.515531+0000) 2022-09-10T11:58:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:58:59 smithi110 ceph-mon[97633]: pgmap v14390: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:58:59.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:58:59.990196+0000) 2022-09-10T11:59:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:00.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:00.515770+0000) 2022-09-10T11:59:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:00.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:00.990407+0000) 2022-09-10T11:59:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:01 smithi104 ceph-mon[97044]: pgmap v14391: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:01.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:01.516014+0000) 2022-09-10T11:59:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:01 smithi110 ceph-mon[97633]: pgmap v14391: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:01.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:01.990579+0000) 2022-09-10T11:59:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:02.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:02.516226+0000) 2022-09-10T11:59:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:02.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:02.990756+0000) 2022-09-10T11:59:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:03 smithi104 ceph-mon[97044]: pgmap v14392: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:03.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:03.516405+0000) 2022-09-10T11:59:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:03 smithi110 ceph-mon[97633]: pgmap v14392: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:03.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:03.991006+0000) 2022-09-10T11:59:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:04.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:04.516682+0000) 2022-09-10T11:59:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:04.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:04.991248+0000) 2022-09-10T11:59:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:05 smithi104 ceph-mon[97044]: pgmap v14393: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:59:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:05 smithi110 ceph-mon[97633]: pgmap v14393: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T11:59:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:05.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:05.516902+0000) 2022-09-10T11:59:06.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:05.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:05.991480+0000) 2022-09-10T11:59:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:06.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:06.517168+0000) 2022-09-10T11:59:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:06.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:06.991686+0000) 2022-09-10T11:59:07.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:07 smithi104 ceph-mon[97044]: pgmap v14394: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:59:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:07.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:07.517328+0000) 2022-09-10T11:59:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:07 smithi110 ceph-mon[97633]: pgmap v14394: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:59:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:07.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:07.991942+0000) 2022-09-10T11:59:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:08.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:08.517487+0000) 2022-09-10T11:59:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:08.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:08.992203+0000) 2022-09-10T11:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:09 smithi104 ceph-mon[97044]: pgmap v14395: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:09 smithi104 ceph-mon[97044]: osdmap e980: 12 total, 12 up, 12 in 2022-09-10T11:59:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:09.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:09.517678+0000) 2022-09-10T11:59:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:09 smithi110 ceph-mon[97633]: pgmap v14395: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T11:59:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:09 smithi110 ceph-mon[97633]: osdmap e980: 12 total, 12 up, 12 in 2022-09-10T11:59:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:09.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:09.992437+0000) 2022-09-10T11:59:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:10.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:10.517864+0000) 2022-09-10T11:59:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:10.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:10.992696+0000) 2022-09-10T11:59:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:11 smithi104 ceph-mon[97044]: pgmap v14397: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:59:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:11.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:11.518079+0000) 2022-09-10T11:59:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:11 smithi110 ceph-mon[97633]: pgmap v14397: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:59:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:11.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:11.992931+0000) 2022-09-10T11:59:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:12.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:12.518323+0000) 2022-09-10T11:59:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:12.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:12.993205+0000) 2022-09-10T11:59:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:13 smithi104 ceph-mon[97044]: pgmap v14398: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:59:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:13.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:13.518566+0000) 2022-09-10T11:59:13.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:59:13.456+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:59:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:13 smithi110 ceph-mon[97633]: pgmap v14398: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T11:59:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:13.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:13.993367+0000) 2022-09-10T11:59:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:14.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:14.518784+0000) 2022-09-10T11:59:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:14.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:14.993640+0000) 2022-09-10T11:59:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:15 smithi104 ceph-mon[97044]: osdmap e981: 12 total, 12 up, 12 in 2022-09-10T11:59:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:15 smithi104 ceph-mon[97044]: pgmap v14400: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:59:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:15 smithi110 ceph-mon[97633]: osdmap e981: 12 total, 12 up, 12 in 2022-09-10T11:59:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:15 smithi110 ceph-mon[97633]: pgmap v14400: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:59:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:15.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:15.519050+0000) 2022-09-10T11:59:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:15.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:15.993900+0000) 2022-09-10T11:59:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:16.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:16.519286+0000) 2022-09-10T11:59:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:16.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:16.994124+0000) 2022-09-10T11:59:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:17 smithi104 ceph-mon[97044]: pgmap v14401: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:59:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:17.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:17.519494+0000) 2022-09-10T11:59:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:17 smithi110 ceph-mon[97633]: pgmap v14401: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T11:59:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:17.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:17.994311+0000) 2022-09-10T11:59:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:18.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:18.519699+0000) 2022-09-10T11:59:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:18.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:18.994485+0000) 2022-09-10T11:59:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:19 smithi104 ceph-mon[97044]: pgmap v14402: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:59:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:19.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:19.519929+0000) 2022-09-10T11:59:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:19 smithi110 ceph-mon[97633]: pgmap v14402: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T11:59:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:19.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:19.994712+0000) 2022-09-10T11:59:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:20.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:20.520177+0000) 2022-09-10T11:59:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:20.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:20.994973+0000) 2022-09-10T11:59:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:21 smithi104 ceph-mon[97044]: pgmap v14403: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:59:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:21.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:21.520388+0000) 2022-09-10T11:59:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:21 smithi110 ceph-mon[97633]: pgmap v14403: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:59:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:21.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:21.995225+0000) 2022-09-10T11:59:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:22.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:22.520642+0000) 2022-09-10T11:59:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:22.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:22.995458+0000) 2022-09-10T11:59:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:23 smithi104 ceph-mon[97044]: pgmap v14404: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:59:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:23.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:23.520865+0000) 2022-09-10T11:59:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:23 smithi110 ceph-mon[97633]: pgmap v14404: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T11:59:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:23.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:23.995709+0000) 2022-09-10T11:59:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:24.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:24.521137+0000) 2022-09-10T11:59:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:24.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:24.995964+0000) 2022-09-10T11:59:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:25 smithi104 ceph-mon[97044]: pgmap v14405: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T11:59:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:25 smithi110 ceph-mon[97633]: pgmap v14405: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T11:59:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:25.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:25.521397+0000) 2022-09-10T11:59:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:25.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:25.996214+0000) 2022-09-10T11:59:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:26.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:26.521620+0000) 2022-09-10T11:59:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:26.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:26.996407+0000) 2022-09-10T11:59:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:27 smithi104 ceph-mon[97044]: pgmap v14406: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:27.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:27.521855+0000) 2022-09-10T11:59:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:27 smithi110 ceph-mon[97633]: pgmap v14406: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:27.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:27.996577+0000) 2022-09-10T11:59:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:59:28.468+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:59:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:28.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:28.522085+0000) 2022-09-10T11:59:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:28.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:28.996835+0000) 2022-09-10T11:59:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:29 smithi104 ceph-mon[97044]: pgmap v14407: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:29.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:29.522361+0000) 2022-09-10T11:59:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:29 smithi110 ceph-mon[97633]: pgmap v14407: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:29.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:29.997122+0000) 2022-09-10T11:59:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:30.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:30.522587+0000) 2022-09-10T11:59:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:30.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:30.997336+0000) 2022-09-10T11:59:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:31 smithi104 ceph-mon[97044]: pgmap v14408: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:31.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:31.522834+0000) 2022-09-10T11:59:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:31 smithi110 ceph-mon[97633]: pgmap v14408: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:31.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:31.997570+0000) 2022-09-10T11:59:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:32.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:32.523084+0000) 2022-09-10T11:59:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:32.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:32.997705+0000) 2022-09-10T11:59:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:33 smithi104 ceph-mon[97044]: pgmap v14409: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:33.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:33.523329+0000) 2022-09-10T11:59:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:33 smithi110 ceph-mon[97633]: pgmap v14409: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:33.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:33.997905+0000) 2022-09-10T11:59:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:34.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:34.523491+0000) 2022-09-10T11:59:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:34.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:34.998146+0000) 2022-09-10T11:59:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:35 smithi104 ceph-mon[97044]: pgmap v14410: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:35 smithi110 ceph-mon[97633]: pgmap v14410: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:35.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:35.523703+0000) 2022-09-10T11:59:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:35.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:35.998370+0000) 2022-09-10T11:59:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:36.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:36.523935+0000) 2022-09-10T11:59:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:36.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:36.998558+0000) 2022-09-10T11:59:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:37 smithi104 ceph-mon[97044]: pgmap v14411: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:37.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:37.524213+0000) 2022-09-10T11:59:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:37 smithi110 ceph-mon[97633]: pgmap v14411: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:37.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:37.998815+0000) 2022-09-10T11:59:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:38.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:38.524336+0000) 2022-09-10T11:59:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:38.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:38.998992+0000) 2022-09-10T11:59:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:39 smithi104 ceph-mon[97044]: pgmap v14412: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:39.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:39.524486+0000) 2022-09-10T11:59:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:39 smithi110 ceph-mon[97633]: pgmap v14412: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:39.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:39.999187+0000) 2022-09-10T11:59:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:40.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:40.524687+0000) 2022-09-10T11:59:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:40.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:40.999330+0000) 2022-09-10T11:59:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:41 smithi104 ceph-mon[97044]: pgmap v14413: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:41.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:41.524886+0000) 2022-09-10T11:59:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:41 smithi110 ceph-mon[97633]: pgmap v14413: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:41.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:41.999514+0000) 2022-09-10T11:59:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:42.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:42.525080+0000) 2022-09-10T11:59:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:42.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:42.999806+0000) 2022-09-10T11:59:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:43 smithi104 ceph-mon[97044]: pgmap v14414: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:43.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:43.525271+0000) 2022-09-10T11:59:43.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:59:43.476+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:59:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:43 smithi110 ceph-mon[97633]: pgmap v14414: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:43.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:44.000041+0000) 2022-09-10T11:59:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:44.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:44.525509+0000) 2022-09-10T11:59:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:44.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:45.000203+0000) 2022-09-10T11:59:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:45 smithi104 ceph-mon[97044]: pgmap v14415: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:45 smithi110 ceph-mon[97633]: pgmap v14415: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:45.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:45.525739+0000) 2022-09-10T11:59:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:45.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:46.000436+0000) 2022-09-10T11:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:59:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T11:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T11:59:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:46.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:46.525987+0000) 2022-09-10T11:59:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:46.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:47.000663+0000) 2022-09-10T11:59:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:47 smithi104 ceph-mon[97044]: pgmap v14416: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:47 smithi110 ceph-mon[97633]: pgmap v14416: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:47.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:47.526196+0000) 2022-09-10T11:59:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:47.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:48.000906+0000) 2022-09-10T11:59:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:48.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:48.526320+0000) 2022-09-10T11:59:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:49.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:49.001187+0000) 2022-09-10T11:59:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:49 smithi104 ceph-mon[97044]: pgmap v14417: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:49.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:49.526517+0000) 2022-09-10T11:59:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:49 smithi110 ceph-mon[97633]: pgmap v14417: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:50.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:50.001405+0000) 2022-09-10T11:59:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:50.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:50.526732+0000) 2022-09-10T11:59:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:51.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:51.001652+0000) 2022-09-10T11:59:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:51 smithi104 ceph-mon[97044]: pgmap v14418: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:51.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:51.526980+0000) 2022-09-10T11:59:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:51 smithi110 ceph-mon[97633]: pgmap v14418: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:52.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:52.001877+0000) 2022-09-10T11:59:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:52.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:52.527200+0000) 2022-09-10T11:59:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:53.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:53.002091+0000) 2022-09-10T11:59:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:53 smithi104 ceph-mon[97044]: pgmap v14419: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:53.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:53.527373+0000) 2022-09-10T11:59:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:53 smithi110 ceph-mon[97633]: pgmap v14419: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:54.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:54.002333+0000) 2022-09-10T11:59:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:54.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:54.527535+0000) 2022-09-10T11:59:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:55.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:55.002569+0000) 2022-09-10T11:59:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:55 smithi104 ceph-mon[97044]: pgmap v14420: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:55 smithi110 ceph-mon[97633]: pgmap v14420: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T11:59:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:55.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:55.527712+0000) 2022-09-10T11:59:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:56.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:56.002718+0000) 2022-09-10T11:59:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:56.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:56.527937+0000) 2022-09-10T11:59:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:57.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:57.002946+0000) 2022-09-10T11:59:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:57 smithi104 ceph-mon[97044]: pgmap v14421: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:57.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:57.528181+0000) 2022-09-10T11:59:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:57 smithi110 ceph-mon[97633]: pgmap v14421: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:58.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:58.003211+0000) 2022-09-10T11:59:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:58.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:58.528431+0000) 2022-09-10T11:59:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 11:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T11:59:58.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T11:59:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 11:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T11:59:59.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:59.003418+0000) 2022-09-10T11:59:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 11:59:59 smithi104 ceph-mon[97044]: pgmap v14422: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T11:59:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 11:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T11:59:59.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T10:59:59.528700+0000) 2022-09-10T11:59:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 11:59:59 smithi110 ceph-mon[97633]: pgmap v14422: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:00.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:00.003620+0000) 2022-09-10T12:00:00.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:00:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:00.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:00.528845+0000) 2022-09-10T12:00:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:00:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:01.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:01.003838+0000) 2022-09-10T12:00:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:01 smithi104 ceph-mon[97044]: pgmap v14423: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:01.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:01.529035+0000) 2022-09-10T12:00:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:01 smithi110 ceph-mon[97633]: pgmap v14423: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:02.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:02.004066+0000) 2022-09-10T12:00:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:02.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:02.529255+0000) 2022-09-10T12:00:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:03.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:03.004368+0000) 2022-09-10T12:00:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:03 smithi104 ceph-mon[97044]: pgmap v14424: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:03.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:03.529465+0000) 2022-09-10T12:00:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:03 smithi110 ceph-mon[97633]: pgmap v14424: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:04.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:04.004579+0000) 2022-09-10T12:00:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:04.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:04.529720+0000) 2022-09-10T12:00:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:05.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:05.004803+0000) 2022-09-10T12:00:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:05 smithi104 ceph-mon[97044]: pgmap v14425: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:00:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:05 smithi110 ceph-mon[97633]: pgmap v14425: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:00:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:05.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:05.529961+0000) 2022-09-10T12:00:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:06.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:06.005049+0000) 2022-09-10T12:00:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:06.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:06.530170+0000) 2022-09-10T12:00:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:07.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:07.005345+0000) 2022-09-10T12:00:07.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:07 smithi104 ceph-mon[97044]: pgmap v14426: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:00:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:07.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:07.530399+0000) 2022-09-10T12:00:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:07 smithi110 ceph-mon[97633]: pgmap v14426: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:00:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:08.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:08.005610+0000) 2022-09-10T12:00:08.812 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:08.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:08.530553+0000) 2022-09-10T12:00:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:09.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:09.005780+0000) 2022-09-10T12:00:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:09 smithi104 ceph-mon[97044]: pgmap v14427: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:00:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:09 smithi104 ceph-mon[97044]: osdmap e982: 12 total, 12 up, 12 in 2022-09-10T12:00:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:09.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:09.530758+0000) 2022-09-10T12:00:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:09 smithi110 ceph-mon[97633]: pgmap v14427: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:00:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:09 smithi110 ceph-mon[97633]: osdmap e982: 12 total, 12 up, 12 in 2022-09-10T12:00:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:10.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:10.006003+0000) 2022-09-10T12:00:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:10.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:10.530987+0000) 2022-09-10T12:00:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:11.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:11.006159+0000) 2022-09-10T12:00:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:11 smithi104 ceph-mon[97044]: pgmap v14429: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:00:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:11.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:11.531232+0000) 2022-09-10T12:00:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:11 smithi110 ceph-mon[97633]: pgmap v14429: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:00:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:12.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:12.006339+0000) 2022-09-10T12:00:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:12.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:12.531481+0000) 2022-09-10T12:00:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:13.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:13.006547+0000) 2022-09-10T12:00:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:13 smithi104 ceph-mon[97044]: pgmap v14430: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:00:13.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:00:13.498+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:00:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:13.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:13.531667+0000) 2022-09-10T12:00:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:13 smithi110 ceph-mon[97633]: pgmap v14430: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:00:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:14.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:14.006789+0000) 2022-09-10T12:00:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:14.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:14.531923+0000) 2022-09-10T12:00:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:15.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:15.007013+0000) 2022-09-10T12:00:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:15 smithi104 ceph-mon[97044]: osdmap e983: 12 total, 12 up, 12 in 2022-09-10T12:00:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:15 smithi104 ceph-mon[97044]: pgmap v14432: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:00:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:15 smithi110 ceph-mon[97633]: osdmap e983: 12 total, 12 up, 12 in 2022-09-10T12:00:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:15 smithi110 ceph-mon[97633]: pgmap v14432: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:00:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:15.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:15.532191+0000) 2022-09-10T12:00:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:16.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:16.007224+0000) 2022-09-10T12:00:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:16.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:16.532380+0000) 2022-09-10T12:00:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:17.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:17.007445+0000) 2022-09-10T12:00:17.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:17 smithi104 ceph-mon[97044]: pgmap v14433: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:00:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:17.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:17.532570+0000) 2022-09-10T12:00:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:17 smithi110 ceph-mon[97633]: pgmap v14433: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:00:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:18.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:18.007691+0000) 2022-09-10T12:00:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:18.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:18.532790+0000) 2022-09-10T12:00:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:19.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:19.007902+0000) 2022-09-10T12:00:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:19 smithi104 ceph-mon[97044]: pgmap v14434: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:00:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:19.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:19.533051+0000) 2022-09-10T12:00:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:19 smithi110 ceph-mon[97633]: pgmap v14434: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:00:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:20.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:20.008121+0000) 2022-09-10T12:00:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:20.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:20.533280+0000) 2022-09-10T12:00:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:21.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:21.008360+0000) 2022-09-10T12:00:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:21 smithi104 ceph-mon[97044]: pgmap v14435: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:00:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:21.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:21.533478+0000) 2022-09-10T12:00:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:21 smithi110 ceph-mon[97633]: pgmap v14435: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:00:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:22.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:22.008620+0000) 2022-09-10T12:00:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:22.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:22.533646+0000) 2022-09-10T12:00:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:23.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:23.008857+0000) 2022-09-10T12:00:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:23 smithi104 ceph-mon[97044]: pgmap v14436: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:00:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:23.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:23.533870+0000) 2022-09-10T12:00:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:23 smithi110 ceph-mon[97633]: pgmap v14436: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:00:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:24.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:24.009148+0000) 2022-09-10T12:00:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:24.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:24.534189+0000) 2022-09-10T12:00:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:25.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:25.009360+0000) 2022-09-10T12:00:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:25 smithi104 ceph-mon[97044]: pgmap v14437: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:00:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:25 smithi110 ceph-mon[97633]: pgmap v14437: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:00:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:25.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:25.534388+0000) 2022-09-10T12:00:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:26.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:26.009609+0000) 2022-09-10T12:00:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:26.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:26.534591+0000) 2022-09-10T12:00:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:27.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:27.009822+0000) 2022-09-10T12:00:27.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:27 smithi104 ceph-mon[97044]: pgmap v14438: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:27.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:27.534824+0000) 2022-09-10T12:00:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:27 smithi110 ceph-mon[97633]: pgmap v14438: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:28.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:28.010001+0000) 2022-09-10T12:00:28.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:00:28.513+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:00:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:28.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:28.535086+0000) 2022-09-10T12:00:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:29.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:29.010229+0000) 2022-09-10T12:00:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:29 smithi104 ceph-mon[97044]: pgmap v14439: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:29.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:29.535334+0000) 2022-09-10T12:00:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:29 smithi110 ceph-mon[97633]: pgmap v14439: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:30.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:30.010479+0000) 2022-09-10T12:00:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:30.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:30.535596+0000) 2022-09-10T12:00:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:31.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:31.010754+0000) 2022-09-10T12:00:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:31 smithi104 ceph-mon[97044]: pgmap v14440: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:31.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:31.535776+0000) 2022-09-10T12:00:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:31 smithi110 ceph-mon[97633]: pgmap v14440: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:32.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:32.010986+0000) 2022-09-10T12:00:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:32.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:32.536003+0000) 2022-09-10T12:00:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:33.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:33.011245+0000) 2022-09-10T12:00:33.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:33 smithi104 ceph-mon[97044]: pgmap v14441: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:33.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:33.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:33.536252+0000) 2022-09-10T12:00:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:33 smithi110 ceph-mon[97633]: pgmap v14441: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:34.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:34.011489+0000) 2022-09-10T12:00:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:34.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:34.536447+0000) 2022-09-10T12:00:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:35.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:35.011651+0000) 2022-09-10T12:00:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:35 smithi104 ceph-mon[97044]: pgmap v14442: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:35 smithi110 ceph-mon[97633]: pgmap v14442: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:35.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:35.536679+0000) 2022-09-10T12:00:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:36.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:36.011853+0000) 2022-09-10T12:00:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:36.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:36.536911+0000) 2022-09-10T12:00:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:37.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:37.012127+0000) 2022-09-10T12:00:37.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:37 smithi104 ceph-mon[97044]: pgmap v14443: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:37.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:37.537138+0000) 2022-09-10T12:00:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:37 smithi110 ceph-mon[97633]: pgmap v14443: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:38.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:38.012318+0000) 2022-09-10T12:00:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:38.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:38.537321+0000) 2022-09-10T12:00:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:39.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:39.012549+0000) 2022-09-10T12:00:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:39 smithi104 ceph-mon[97044]: pgmap v14444: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:39.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:39.537560+0000) 2022-09-10T12:00:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:39 smithi110 ceph-mon[97633]: pgmap v14444: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:40.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:40.012783+0000) 2022-09-10T12:00:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:40.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:40.537706+0000) 2022-09-10T12:00:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:41.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:41.013054+0000) 2022-09-10T12:00:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:41 smithi104 ceph-mon[97044]: pgmap v14445: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:41.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:41.537938+0000) 2022-09-10T12:00:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:41 smithi110 ceph-mon[97633]: pgmap v14445: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:42.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:42.013225+0000) 2022-09-10T12:00:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:42.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:42.538202+0000) 2022-09-10T12:00:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:43.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:43.013437+0000) 2022-09-10T12:00:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:43 smithi104 ceph-mon[97044]: pgmap v14446: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:43.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:43.538377+0000) 2022-09-10T12:00:43.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:00:43.519+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:00:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:43 smithi110 ceph-mon[97633]: pgmap v14446: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:44.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:44.013679+0000) 2022-09-10T12:00:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:44.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:44.538641+0000) 2022-09-10T12:00:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:45.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:45.013915+0000) 2022-09-10T12:00:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:45 smithi104 ceph-mon[97044]: pgmap v14447: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:45 smithi110 ceph-mon[97633]: pgmap v14447: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:45.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:45.538807+0000) 2022-09-10T12:00:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:46.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:46.014156+0000) 2022-09-10T12:00:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:00:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:00:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:00:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:00:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:00:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:46.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:46.539051+0000) 2022-09-10T12:00:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:47.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:47.014367+0000) 2022-09-10T12:00:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:47 smithi104 ceph-mon[97044]: pgmap v14448: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:47 smithi110 ceph-mon[97633]: pgmap v14448: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:47.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:47.539276+0000) 2022-09-10T12:00:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:48.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:48.014606+0000) 2022-09-10T12:00:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:48.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:48.539552+0000) 2022-09-10T12:00:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:49.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:49.014891+0000) 2022-09-10T12:00:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:49 smithi104 ceph-mon[97044]: pgmap v14449: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:49.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:49.539781+0000) 2022-09-10T12:00:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:49 smithi110 ceph-mon[97633]: pgmap v14449: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:50.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:50.015174+0000) 2022-09-10T12:00:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:50.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:50.540013+0000) 2022-09-10T12:00:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:51.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:51.015387+0000) 2022-09-10T12:00:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:51 smithi104 ceph-mon[97044]: pgmap v14450: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:51.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:51.540269+0000) 2022-09-10T12:00:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:51 smithi110 ceph-mon[97633]: pgmap v14450: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:52.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:52.015565+0000) 2022-09-10T12:00:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:52.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:52.540497+0000) 2022-09-10T12:00:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:53.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:53.015799+0000) 2022-09-10T12:00:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:53 smithi104 ceph-mon[97044]: pgmap v14451: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:53.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:53.540752+0000) 2022-09-10T12:00:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:53 smithi110 ceph-mon[97633]: pgmap v14451: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:54.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:54.016068+0000) 2022-09-10T12:00:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:54.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:54.540990+0000) 2022-09-10T12:00:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:55.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:55.016286+0000) 2022-09-10T12:00:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:55 smithi104 ceph-mon[97044]: pgmap v14452: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:55 smithi110 ceph-mon[97633]: pgmap v14452: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:00:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:55.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:55.541251+0000) 2022-09-10T12:00:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:56.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:56.016509+0000) 2022-09-10T12:00:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:56.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:56.541447+0000) 2022-09-10T12:00:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:57.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:57.016709+0000) 2022-09-10T12:00:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:57 smithi104 ceph-mon[97044]: pgmap v14453: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:57.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:57.541665+0000) 2022-09-10T12:00:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:57 smithi110 ceph-mon[97633]: pgmap v14453: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:58.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:58.016936+0000) 2022-09-10T12:00:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:58.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:58.541929+0000) 2022-09-10T12:00:58.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:00:58.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:00:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:00:59.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:59.017130+0000) 2022-09-10T12:00:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:00:59 smithi104 ceph-mon[97044]: pgmap v14454: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:00:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:00:59.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:00:59.542111+0000) 2022-09-10T12:00:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:00:59 smithi110 ceph-mon[97633]: pgmap v14454: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:00.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:00.017325+0000) 2022-09-10T12:01:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:00.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:00.542250+0000) 2022-09-10T12:01:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:01.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:01.017552+0000) 2022-09-10T12:01:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:01 smithi104 ceph-mon[97044]: pgmap v14455: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:01.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:01.542398+0000) 2022-09-10T12:01:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:01 smithi110 ceph-mon[97633]: pgmap v14455: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:02.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:02.017741+0000) 2022-09-10T12:01:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:02.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:02.542536+0000) 2022-09-10T12:01:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:03.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:03.017918+0000) 2022-09-10T12:01:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:03 smithi104 ceph-mon[97044]: pgmap v14456: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:03.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:03.542738+0000) 2022-09-10T12:01:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:03 smithi110 ceph-mon[97633]: pgmap v14456: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:04.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:04.018124+0000) 2022-09-10T12:01:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:04.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:04.542965+0000) 2022-09-10T12:01:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:05.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:05.018270+0000) 2022-09-10T12:01:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:05 smithi104 ceph-mon[97044]: pgmap v14457: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:01:05.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:05 smithi110 ceph-mon[97633]: pgmap v14457: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:01:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:05.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:05.543203+0000) 2022-09-10T12:01:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:06.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:06.018487+0000) 2022-09-10T12:01:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:06.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:06.543428+0000) 2022-09-10T12:01:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:07.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:07.018669+0000) 2022-09-10T12:01:07.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:07 smithi104 ceph-mon[97044]: pgmap v14458: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:01:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:07.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:07.543600+0000) 2022-09-10T12:01:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:07 smithi110 ceph-mon[97633]: pgmap v14458: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:01:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:08.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:08.018914+0000) 2022-09-10T12:01:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:08.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:08.543870+0000) 2022-09-10T12:01:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:09.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:09.019112+0000) 2022-09-10T12:01:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:09 smithi104 ceph-mon[97044]: pgmap v14459: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:01:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:09 smithi104 ceph-mon[97044]: osdmap e984: 12 total, 12 up, 12 in 2022-09-10T12:01:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:09.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:09.544126+0000) 2022-09-10T12:01:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:09 smithi110 ceph-mon[97633]: pgmap v14459: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:01:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:09 smithi110 ceph-mon[97633]: osdmap e984: 12 total, 12 up, 12 in 2022-09-10T12:01:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:10.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:10.019358+0000) 2022-09-10T12:01:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:10.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:10.544292+0000) 2022-09-10T12:01:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:11.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:11.019612+0000) 2022-09-10T12:01:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:11 smithi104 ceph-mon[97044]: pgmap v14461: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:01:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:11.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:11.544508+0000) 2022-09-10T12:01:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:11 smithi110 ceph-mon[97633]: pgmap v14461: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:01:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:12.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:12.019754+0000) 2022-09-10T12:01:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:12.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:12.544738+0000) 2022-09-10T12:01:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:13.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:13.019989+0000) 2022-09-10T12:01:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:13 smithi104 ceph-mon[97044]: pgmap v14462: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:01:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:13.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:13.544903+0000) 2022-09-10T12:01:13.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:01:13.539+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:01:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:13 smithi110 ceph-mon[97633]: pgmap v14462: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:01:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:14.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:14.020252+0000) 2022-09-10T12:01:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:14.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:14.545166+0000) 2022-09-10T12:01:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:15.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:15.020436+0000) 2022-09-10T12:01:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:15 smithi104 ceph-mon[97044]: osdmap e985: 12 total, 12 up, 12 in 2022-09-10T12:01:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:15 smithi104 ceph-mon[97044]: pgmap v14464: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:01:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:15 smithi110 ceph-mon[97633]: osdmap e985: 12 total, 12 up, 12 in 2022-09-10T12:01:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:15 smithi110 ceph-mon[97633]: pgmap v14464: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:01:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:15.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:15.545418+0000) 2022-09-10T12:01:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:16.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:16.020606+0000) 2022-09-10T12:01:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:16.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:16.545579+0000) 2022-09-10T12:01:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:17.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:17.020816+0000) 2022-09-10T12:01:17.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:17 smithi104 ceph-mon[97044]: pgmap v14465: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:01:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:17.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:17.545810+0000) 2022-09-10T12:01:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:17 smithi110 ceph-mon[97633]: pgmap v14465: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:01:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:18.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:18.021045+0000) 2022-09-10T12:01:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:18.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:18.546032+0000) 2022-09-10T12:01:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:19.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:19.021191+0000) 2022-09-10T12:01:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:19 smithi104 ceph-mon[97044]: pgmap v14466: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:01:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:19.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:19.546244+0000) 2022-09-10T12:01:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:19 smithi110 ceph-mon[97633]: pgmap v14466: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:01:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:20.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:20.021351+0000) 2022-09-10T12:01:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:20.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:20.546464+0000) 2022-09-10T12:01:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:21.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:21.021539+0000) 2022-09-10T12:01:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:21 smithi104 ceph-mon[97044]: pgmap v14467: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:01:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:21.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:21.546668+0000) 2022-09-10T12:01:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:21 smithi110 ceph-mon[97633]: pgmap v14467: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:01:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:22.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:22.021721+0000) 2022-09-10T12:01:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:22.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:22.546907+0000) 2022-09-10T12:01:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:23.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:23.021864+0000) 2022-09-10T12:01:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:23 smithi104 ceph-mon[97044]: pgmap v14468: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:01:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:23.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:23.547133+0000) 2022-09-10T12:01:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:23 smithi110 ceph-mon[97633]: pgmap v14468: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:01:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:24.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:24.022133+0000) 2022-09-10T12:01:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:24.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:24.547277+0000) 2022-09-10T12:01:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:25.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:25.022315+0000) 2022-09-10T12:01:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:25 smithi104 ceph-mon[97044]: pgmap v14469: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:01:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:25 smithi110 ceph-mon[97633]: pgmap v14469: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:01:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:25.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:25.547493+0000) 2022-09-10T12:01:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:26.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:26.022526+0000) 2022-09-10T12:01:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:26.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:26.547752+0000) 2022-09-10T12:01:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:27.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:27.022703+0000) 2022-09-10T12:01:27.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:27 smithi104 ceph-mon[97044]: pgmap v14470: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:27.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:27.547949+0000) 2022-09-10T12:01:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:27 smithi110 ceph-mon[97633]: pgmap v14470: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:28.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:28.022906+0000) 2022-09-10T12:01:28.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:01:28.543+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:01:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:28.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:28.548194+0000) 2022-09-10T12:01:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:29.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:29.023174+0000) 2022-09-10T12:01:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:29 smithi104 ceph-mon[97044]: pgmap v14471: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:29.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:29.548432+0000) 2022-09-10T12:01:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:29 smithi110 ceph-mon[97633]: pgmap v14471: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:30.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:30.023334+0000) 2022-09-10T12:01:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:30.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:30.548625+0000) 2022-09-10T12:01:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:31.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:31.023482+0000) 2022-09-10T12:01:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:31 smithi104 ceph-mon[97044]: pgmap v14472: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:31.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:31.548838+0000) 2022-09-10T12:01:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:31 smithi110 ceph-mon[97633]: pgmap v14472: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:32.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:32.023648+0000) 2022-09-10T12:01:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:32.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:32.549073+0000) 2022-09-10T12:01:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:33.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:33.023839+0000) 2022-09-10T12:01:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:33 smithi104 ceph-mon[97044]: pgmap v14473: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:33.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:33.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:33.549347+0000) 2022-09-10T12:01:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:33 smithi110 ceph-mon[97633]: pgmap v14473: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:34.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:34.024081+0000) 2022-09-10T12:01:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:34.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:34.549586+0000) 2022-09-10T12:01:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:35.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:35.024350+0000) 2022-09-10T12:01:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:35 smithi104 ceph-mon[97044]: pgmap v14474: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:35 smithi110 ceph-mon[97633]: pgmap v14474: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:35.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:35.549770+0000) 2022-09-10T12:01:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:36.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:36.024600+0000) 2022-09-10T12:01:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:36.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:36.549960+0000) 2022-09-10T12:01:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:37.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:37.024746+0000) 2022-09-10T12:01:37.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:37 smithi104 ceph-mon[97044]: pgmap v14475: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:37.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:37.550162+0000) 2022-09-10T12:01:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:37 smithi110 ceph-mon[97633]: pgmap v14475: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:38.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:38.024975+0000) 2022-09-10T12:01:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:38.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:38.550358+0000) 2022-09-10T12:01:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:39.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:39.025214+0000) 2022-09-10T12:01:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:39 smithi104 ceph-mon[97044]: pgmap v14476: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:39.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:39.550580+0000) 2022-09-10T12:01:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:39 smithi110 ceph-mon[97633]: pgmap v14476: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:40.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:40.025383+0000) 2022-09-10T12:01:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:40.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:40.550748+0000) 2022-09-10T12:01:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:41.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:41.025575+0000) 2022-09-10T12:01:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:41 smithi104 ceph-mon[97044]: pgmap v14477: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:41.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:41.551003+0000) 2022-09-10T12:01:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:41 smithi110 ceph-mon[97633]: pgmap v14477: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:42.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:42.025756+0000) 2022-09-10T12:01:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:42.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:42.551268+0000) 2022-09-10T12:01:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:43.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:43.025988+0000) 2022-09-10T12:01:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:43 smithi104 ceph-mon[97044]: pgmap v14478: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:43.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:43.551420+0000) 2022-09-10T12:01:43.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:01:43.558+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:01:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:43 smithi110 ceph-mon[97633]: pgmap v14478: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:44.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:44.026216+0000) 2022-09-10T12:01:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:44.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:44.551635+0000) 2022-09-10T12:01:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:45.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:45.026387+0000) 2022-09-10T12:01:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:45 smithi104 ceph-mon[97044]: pgmap v14479: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:45 smithi110 ceph-mon[97633]: pgmap v14479: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:45.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:45.551859+0000) 2022-09-10T12:01:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:46.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:46.026568+0000) 2022-09-10T12:01:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:01:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:01:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:01:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:01:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:01:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:46.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:46.552111+0000) 2022-09-10T12:01:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:47.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:47.026815+0000) 2022-09-10T12:01:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:47 smithi104 ceph-mon[97044]: pgmap v14480: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:47 smithi110 ceph-mon[97633]: pgmap v14480: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:47.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:47.552336+0000) 2022-09-10T12:01:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:48.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:48.027031+0000) 2022-09-10T12:01:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:48.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:48.552558+0000) 2022-09-10T12:01:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:49.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:49.027328+0000) 2022-09-10T12:01:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:49 smithi104 ceph-mon[97044]: pgmap v14481: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:49.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:49.552748+0000) 2022-09-10T12:01:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:49 smithi110 ceph-mon[97633]: pgmap v14481: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:50.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:50.027540+0000) 2022-09-10T12:01:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:50.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:50.552967+0000) 2022-09-10T12:01:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:51.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:51.027736+0000) 2022-09-10T12:01:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:51 smithi104 ceph-mon[97044]: pgmap v14482: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:51.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:51.553226+0000) 2022-09-10T12:01:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:51 smithi110 ceph-mon[97633]: pgmap v14482: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:52.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:52.027962+0000) 2022-09-10T12:01:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:52.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:52.553390+0000) 2022-09-10T12:01:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:53.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:53.028205+0000) 2022-09-10T12:01:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:53 smithi110 ceph-mon[97633]: pgmap v14483: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:54.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:53.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:53.553591+0000) 2022-09-10T12:01:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:53 smithi104 ceph-mon[97044]: pgmap v14483: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:54.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:54.028393+0000) 2022-09-10T12:01:55.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:54.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:54.553820+0000) 2022-09-10T12:01:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:55.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:55.028550+0000) 2022-09-10T12:01:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:55 smithi104 ceph-mon[97044]: pgmap v14484: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:55 smithi110 ceph-mon[97633]: pgmap v14484: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:01:56.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:55.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:55.554060+0000) 2022-09-10T12:01:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:56.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:56.028734+0000) 2022-09-10T12:01:57.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:56.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:56.554340+0000) 2022-09-10T12:01:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:57.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:57.028973+0000) 2022-09-10T12:01:57.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:57 smithi104 ceph-mon[97044]: pgmap v14485: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:57 smithi110 ceph-mon[97633]: pgmap v14485: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:01:58.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:57.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:57.554571+0000) 2022-09-10T12:01:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:58.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:58.029263+0000) 2022-09-10T12:01:59.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:58.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:58.554814+0000) 2022-09-10T12:01:59.030 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:01:58.575+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:01:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:01:59.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:59.029455+0000) 2022-09-10T12:01:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:01:59 smithi110 ceph-mon[97633]: pgmap v14486: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:00.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:01:59.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:01:59.555053+0000) 2022-09-10T12:02:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:01:59 smithi104 ceph-mon[97044]: pgmap v14486: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:00.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:00.029667+0000) 2022-09-10T12:02:01.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:00.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:00.555312+0000) 2022-09-10T12:02:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:01.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:01.029944+0000) 2022-09-10T12:02:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:01 smithi110 ceph-mon[97633]: pgmap v14487: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:02.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:01.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:01.555477+0000) 2022-09-10T12:02:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:01 smithi104 ceph-mon[97044]: pgmap v14487: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:02.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:02.030212+0000) 2022-09-10T12:02:03.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:02.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:02.555711+0000) 2022-09-10T12:02:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:03.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:03.030423+0000) 2022-09-10T12:02:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:03 smithi110 ceph-mon[97633]: pgmap v14488: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:04.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:03.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:03.555917+0000) 2022-09-10T12:02:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:03 smithi104 ceph-mon[97044]: pgmap v14488: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:04.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:04.030631+0000) 2022-09-10T12:02:05.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:04.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:04.556091+0000) 2022-09-10T12:02:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:05.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:05.030844+0000) 2022-09-10T12:02:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:05 smithi104 ceph-mon[97044]: pgmap v14489: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:02:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:05 smithi110 ceph-mon[97633]: pgmap v14489: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:02:06.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:05.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:05.556329+0000) 2022-09-10T12:02:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:06.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:06.031073+0000) 2022-09-10T12:02:07.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:06.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:06.556556+0000) 2022-09-10T12:02:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:07.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:07.031379+0000) 2022-09-10T12:02:07.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:07 smithi110 ceph-mon[97633]: pgmap v14490: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:02:08.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:07.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:07.556718+0000) 2022-09-10T12:02:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:07 smithi104 ceph-mon[97044]: pgmap v14490: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:02:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:08.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:08.031650+0000) 2022-09-10T12:02:09.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:08.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:08.556899+0000) 2022-09-10T12:02:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:09.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:09.031899+0000) 2022-09-10T12:02:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:09 smithi110 ceph-mon[97633]: pgmap v14491: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:02:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:09 smithi110 ceph-mon[97633]: osdmap e986: 12 total, 12 up, 12 in 2022-09-10T12:02:10.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:09.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:09.557161+0000) 2022-09-10T12:02:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:09 smithi104 ceph-mon[97044]: pgmap v14491: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:02:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:09 smithi104 ceph-mon[97044]: osdmap e986: 12 total, 12 up, 12 in 2022-09-10T12:02:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:10.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:10.032176+0000) 2022-09-10T12:02:11.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:10.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:10.557370+0000) 2022-09-10T12:02:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:11.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:11.032434+0000) 2022-09-10T12:02:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:11 smithi110 ceph-mon[97633]: pgmap v14493: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:02:12.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:11.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:11.557618+0000) 2022-09-10T12:02:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:11 smithi104 ceph-mon[97044]: pgmap v14493: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:02:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:12.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:12.032675+0000) 2022-09-10T12:02:13.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:12.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:12.557750+0000) 2022-09-10T12:02:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:13.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:13.032946+0000) 2022-09-10T12:02:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:13 smithi110 ceph-mon[97633]: pgmap v14494: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:02:14.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:13.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:13.557979+0000) 2022-09-10T12:02:14.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:02:13.589+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:02:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:13 smithi104 ceph-mon[97044]: pgmap v14494: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:02:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:14.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:14.033169+0000) 2022-09-10T12:02:15.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:14.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:14.558278+0000) 2022-09-10T12:02:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:15.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:15.033449+0000) 2022-09-10T12:02:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:15 smithi104 ceph-mon[97044]: osdmap e987: 12 total, 12 up, 12 in 2022-09-10T12:02:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:15 smithi104 ceph-mon[97044]: pgmap v14496: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:02:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:15 smithi110 ceph-mon[97633]: osdmap e987: 12 total, 12 up, 12 in 2022-09-10T12:02:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:15 smithi110 ceph-mon[97633]: pgmap v14496: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:02:16.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:15.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:15.558466+0000) 2022-09-10T12:02:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:16.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:16.033699+0000) 2022-09-10T12:02:17.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:16.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:16.558678+0000) 2022-09-10T12:02:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:17.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:17.033960+0000) 2022-09-10T12:02:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:17 smithi110 ceph-mon[97633]: pgmap v14497: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:02:18.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:17.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:17.558875+0000) 2022-09-10T12:02:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:17 smithi104 ceph-mon[97044]: pgmap v14497: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:02:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:18.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:18.034213+0000) 2022-09-10T12:02:19.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:18.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:18.559055+0000) 2022-09-10T12:02:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:19.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:19.034494+0000) 2022-09-10T12:02:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:19 smithi110 ceph-mon[97633]: pgmap v14498: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:02:20.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:19.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:19.559301+0000) 2022-09-10T12:02:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:19 smithi104 ceph-mon[97044]: pgmap v14498: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:02:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:20.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:20.034657+0000) 2022-09-10T12:02:21.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:20.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:20.559542+0000) 2022-09-10T12:02:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:21.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:21.034827+0000) 2022-09-10T12:02:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:21 smithi110 ceph-mon[97633]: pgmap v14499: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:02:22.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:21.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:21.559818+0000) 2022-09-10T12:02:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:21 smithi104 ceph-mon[97044]: pgmap v14499: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:02:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:22.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:22.035057+0000) 2022-09-10T12:02:23.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:22.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:22.560025+0000) 2022-09-10T12:02:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:23.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:23.035333+0000) 2022-09-10T12:02:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:23 smithi110 ceph-mon[97633]: pgmap v14500: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:02:24.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:23.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:23.560272+0000) 2022-09-10T12:02:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:23 smithi104 ceph-mon[97044]: pgmap v14500: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:02:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:24.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:24.035572+0000) 2022-09-10T12:02:25.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:24.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:24.560553+0000) 2022-09-10T12:02:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:25.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:25.035844+0000) 2022-09-10T12:02:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:25 smithi104 ceph-mon[97044]: pgmap v14501: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:02:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:25 smithi110 ceph-mon[97633]: pgmap v14501: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:02:26.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:25.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:25.560815+0000) 2022-09-10T12:02:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:26.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:26.036036+0000) 2022-09-10T12:02:27.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:26.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:26.561089+0000) 2022-09-10T12:02:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:27.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:27.036251+0000) 2022-09-10T12:02:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:27 smithi110 ceph-mon[97633]: pgmap v14502: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:28.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:27.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:27.561366+0000) 2022-09-10T12:02:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:27 smithi104 ceph-mon[97044]: pgmap v14502: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:28.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:28.036377+0000) 2022-09-10T12:02:29.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:28.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:28.561594+0000) 2022-09-10T12:02:29.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:02:28.605+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:02:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:29.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:29.036575+0000) 2022-09-10T12:02:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:29 smithi110 ceph-mon[97633]: pgmap v14503: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:30.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:29.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:29.561786+0000) 2022-09-10T12:02:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:29 smithi104 ceph-mon[97044]: pgmap v14503: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:30.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:30.036806+0000) 2022-09-10T12:02:31.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:30.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:30.562024+0000) 2022-09-10T12:02:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:31.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:31.037041+0000) 2022-09-10T12:02:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:31 smithi110 ceph-mon[97633]: pgmap v14504: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:32.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:31.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:31.562187+0000) 2022-09-10T12:02:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:31 smithi104 ceph-mon[97044]: pgmap v14504: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:32.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:32.037254+0000) 2022-09-10T12:02:33.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:32.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:32.562346+0000) 2022-09-10T12:02:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:33.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:33.037482+0000) 2022-09-10T12:02:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:33 smithi110 ceph-mon[97633]: pgmap v14505: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:34.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:33.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:33.562556+0000) 2022-09-10T12:02:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:33 smithi104 ceph-mon[97044]: pgmap v14505: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:34.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:34.037708+0000) 2022-09-10T12:02:35.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:34.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:34.562756+0000) 2022-09-10T12:02:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:35.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:35.037914+0000) 2022-09-10T12:02:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:35 smithi104 ceph-mon[97044]: pgmap v14506: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:35 smithi110 ceph-mon[97633]: pgmap v14506: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:36.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:35.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:35.562995+0000) 2022-09-10T12:02:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:36.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:36.038069+0000) 2022-09-10T12:02:37.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:36.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:36.563216+0000) 2022-09-10T12:02:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:37.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:37.038349+0000) 2022-09-10T12:02:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:37 smithi110 ceph-mon[97633]: pgmap v14507: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:38.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:37.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:37.563433+0000) 2022-09-10T12:02:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:37 smithi104 ceph-mon[97044]: pgmap v14507: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:38.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:38.038530+0000) 2022-09-10T12:02:39.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:38.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:38.563647+0000) 2022-09-10T12:02:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:39.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:39.038695+0000) 2022-09-10T12:02:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:39 smithi110 ceph-mon[97633]: pgmap v14508: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:40.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:39.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:39.563855+0000) 2022-09-10T12:02:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:39 smithi104 ceph-mon[97044]: pgmap v14508: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:40.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:40.038828+0000) 2022-09-10T12:02:41.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:40.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:40.564075+0000) 2022-09-10T12:02:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:41.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:41.039015+0000) 2022-09-10T12:02:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:41 smithi110 ceph-mon[97633]: pgmap v14509: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:42.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:41.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:41.564254+0000) 2022-09-10T12:02:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:41 smithi104 ceph-mon[97044]: pgmap v14509: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:42.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:42.039235+0000) 2022-09-10T12:02:43.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:42.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:42.564483+0000) 2022-09-10T12:02:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:43.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:43.039468+0000) 2022-09-10T12:02:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:43 smithi110 ceph-mon[97633]: pgmap v14510: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:44.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:43.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:43.564774+0000) 2022-09-10T12:02:44.041 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:02:43.621+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:02:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:43 smithi104 ceph-mon[97044]: pgmap v14510: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:44.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:44.039738+0000) 2022-09-10T12:02:45.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:44.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:44.564956+0000) 2022-09-10T12:02:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:45.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:45.039987+0000) 2022-09-10T12:02:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:45 smithi104 ceph-mon[97044]: pgmap v14511: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:45 smithi110 ceph-mon[97633]: pgmap v14511: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:46.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:45.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:45.565182+0000) 2022-09-10T12:02:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:46.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:46.040243+0000) 2022-09-10T12:02:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:02:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:02:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:02:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:02:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:02:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:02:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:02:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:02:47.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:46.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:46.565410+0000) 2022-09-10T12:02:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:47.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:47.040413+0000) 2022-09-10T12:02:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:47 smithi104 ceph-mon[97044]: pgmap v14512: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:47 smithi110 ceph-mon[97633]: pgmap v14512: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:48.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:47.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:47.565661+0000) 2022-09-10T12:02:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:48.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:48.040608+0000) 2022-09-10T12:02:49.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:48.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:48.565913+0000) 2022-09-10T12:02:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:49.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:49.040817+0000) 2022-09-10T12:02:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:49 smithi110 ceph-mon[97633]: pgmap v14513: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:50.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:49.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:49.566137+0000) 2022-09-10T12:02:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:49 smithi104 ceph-mon[97044]: pgmap v14513: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:50.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:50.041010+0000) 2022-09-10T12:02:51.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:50.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:50.566276+0000) 2022-09-10T12:02:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:51.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:51.041258+0000) 2022-09-10T12:02:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:51 smithi110 ceph-mon[97633]: pgmap v14514: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:52.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:51.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:51.566441+0000) 2022-09-10T12:02:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:51 smithi104 ceph-mon[97044]: pgmap v14514: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:52.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:52.041471+0000) 2022-09-10T12:02:53.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:52.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:52.566609+0000) 2022-09-10T12:02:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:53.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:53.041680+0000) 2022-09-10T12:02:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:53 smithi110 ceph-mon[97633]: pgmap v14515: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:54.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:53.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:53.566822+0000) 2022-09-10T12:02:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:53 smithi104 ceph-mon[97044]: pgmap v14515: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:54.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:54.041913+0000) 2022-09-10T12:02:55.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:54.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:54.567042+0000) 2022-09-10T12:02:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:55.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:55.042136+0000) 2022-09-10T12:02:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:55 smithi104 ceph-mon[97044]: pgmap v14516: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:55 smithi110 ceph-mon[97633]: pgmap v14516: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:02:56.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:55.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:55.567210+0000) 2022-09-10T12:02:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:56.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:56.042302+0000) 2022-09-10T12:02:57.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:56.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:56.567450+0000) 2022-09-10T12:02:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:57.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:57.042547+0000) 2022-09-10T12:02:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:57 smithi110 ceph-mon[97633]: pgmap v14517: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:58.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:57.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:57.567689+0000) 2022-09-10T12:02:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:57 smithi104 ceph-mon[97044]: pgmap v14517: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:02:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:58.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:58.042811+0000) 2022-09-10T12:02:59.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:58.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:58.567947+0000) 2022-09-10T12:02:59.043 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:02:58.637+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:02:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:02:59.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:59.043049+0000) 2022-09-10T12:02:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:02:59 smithi110 ceph-mon[97633]: pgmap v14518: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:00.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:02:59.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:02:59.568193+0000) 2022-09-10T12:03:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:02:59 smithi104 ceph-mon[97044]: pgmap v14518: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:00.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:00.043258+0000) 2022-09-10T12:03:01.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:00.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:00.568364+0000) 2022-09-10T12:03:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:01.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:01.043419+0000) 2022-09-10T12:03:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:01 smithi110 ceph-mon[97633]: pgmap v14519: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:02.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:01.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:01.568572+0000) 2022-09-10T12:03:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:01 smithi104 ceph-mon[97044]: pgmap v14519: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:02.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:02.043582+0000) 2022-09-10T12:03:03.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:02.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:02.568786+0000) 2022-09-10T12:03:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:03.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:03.043806+0000) 2022-09-10T12:03:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:03 smithi110 ceph-mon[97633]: pgmap v14520: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:04.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:03.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:03.569009+0000) 2022-09-10T12:03:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:03 smithi104 ceph-mon[97044]: pgmap v14520: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:04.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:04.043986+0000) 2022-09-10T12:03:05.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:04.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:04.569268+0000) 2022-09-10T12:03:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:05.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:05.044222+0000) 2022-09-10T12:03:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:05 smithi104 ceph-mon[97044]: pgmap v14521: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:03:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:05 smithi110 ceph-mon[97633]: pgmap v14521: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:03:06.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:05.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:05.569456+0000) 2022-09-10T12:03:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:06.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:06.044445+0000) 2022-09-10T12:03:07.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:06.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:06.569686+0000) 2022-09-10T12:03:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:07.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:07.044648+0000) 2022-09-10T12:03:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:07 smithi110 ceph-mon[97633]: pgmap v14522: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:03:08.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:07.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:07.569940+0000) 2022-09-10T12:03:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:07 smithi104 ceph-mon[97044]: pgmap v14522: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:03:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:08.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:08.044856+0000) 2022-09-10T12:03:09.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:08.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:08.570179+0000) 2022-09-10T12:03:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:09.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:09.045133+0000) 2022-09-10T12:03:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:09 smithi110 ceph-mon[97633]: pgmap v14523: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:03:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:09 smithi110 ceph-mon[97633]: osdmap e988: 12 total, 12 up, 12 in 2022-09-10T12:03:10.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:09.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:09.570353+0000) 2022-09-10T12:03:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:09 smithi104 ceph-mon[97044]: pgmap v14523: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:03:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:09 smithi104 ceph-mon[97044]: osdmap e988: 12 total, 12 up, 12 in 2022-09-10T12:03:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:10.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:10.045356+0000) 2022-09-10T12:03:11.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:10.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:10.570564+0000) 2022-09-10T12:03:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:11.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:11.045583+0000) 2022-09-10T12:03:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:11 smithi110 ceph-mon[97633]: pgmap v14525: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:03:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:11 smithi104 ceph-mon[97044]: pgmap v14525: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:03:12.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:11.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:11.570790+0000) 2022-09-10T12:03:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:12.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:12.045838+0000) 2022-09-10T12:03:13.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:12.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:12.571004+0000) 2022-09-10T12:03:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:13.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:13.046046+0000) 2022-09-10T12:03:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:13 smithi110 ceph-mon[97633]: pgmap v14526: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:03:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:13 smithi104 ceph-mon[97044]: pgmap v14526: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:03:14.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:13.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:13.571229+0000) 2022-09-10T12:03:14.047 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:03:13.641+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:03:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:14.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:14.046233+0000) 2022-09-10T12:03:15.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:14.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:14.571469+0000) 2022-09-10T12:03:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:15.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:15.046454+0000) 2022-09-10T12:03:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:15 smithi104 ceph-mon[97044]: osdmap e989: 12 total, 12 up, 12 in 2022-09-10T12:03:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:15 smithi104 ceph-mon[97044]: pgmap v14528: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:03:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:15 smithi110 ceph-mon[97633]: osdmap e989: 12 total, 12 up, 12 in 2022-09-10T12:03:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:15 smithi110 ceph-mon[97633]: pgmap v14528: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:03:16.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:15.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:15.571665+0000) 2022-09-10T12:03:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:16.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:16.046703+0000) 2022-09-10T12:03:17.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:16.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:16.571802+0000) 2022-09-10T12:03:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:17.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:17.046956+0000) 2022-09-10T12:03:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:17 smithi110 ceph-mon[97633]: pgmap v14529: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:03:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:17 smithi104 ceph-mon[97044]: pgmap v14529: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:03:18.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:17.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:17.572009+0000) 2022-09-10T12:03:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:18.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:18.047195+0000) 2022-09-10T12:03:19.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:18.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:18.572250+0000) 2022-09-10T12:03:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:19.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:19.047405+0000) 2022-09-10T12:03:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:19 smithi110 ceph-mon[97633]: pgmap v14530: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:03:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:19 smithi104 ceph-mon[97044]: pgmap v14530: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:03:20.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:19.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:19.572497+0000) 2022-09-10T12:03:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:20.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:20.047580+0000) 2022-09-10T12:03:21.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:20.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:20.572740+0000) 2022-09-10T12:03:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:21.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:21.047898+0000) 2022-09-10T12:03:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:21 smithi110 ceph-mon[97633]: pgmap v14531: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:03:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:21 smithi104 ceph-mon[97044]: pgmap v14531: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:03:22.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:21.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:21.573019+0000) 2022-09-10T12:03:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:22.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:22.048174+0000) 2022-09-10T12:03:23.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:22.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:22.573207+0000) 2022-09-10T12:03:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:23.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:23.048381+0000) 2022-09-10T12:03:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:23 smithi110 ceph-mon[97633]: pgmap v14532: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:03:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:23 smithi104 ceph-mon[97044]: pgmap v14532: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:03:24.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:23.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:23.573441+0000) 2022-09-10T12:03:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:24.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:24.048619+0000) 2022-09-10T12:03:25.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:24.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:24.573695+0000) 2022-09-10T12:03:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:25.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:25.048897+0000) 2022-09-10T12:03:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:25 smithi104 ceph-mon[97044]: pgmap v14533: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:03:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:25 smithi110 ceph-mon[97633]: pgmap v14533: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 100 B/s wr, 14 op/s 2022-09-10T12:03:26.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:25.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:25.573884+0000) 2022-09-10T12:03:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:26.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:26.049208+0000) 2022-09-10T12:03:27.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:26.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:26.574150+0000) 2022-09-10T12:03:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:27.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:27.049429+0000) 2022-09-10T12:03:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:27 smithi110 ceph-mon[97633]: pgmap v14534: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:27 smithi104 ceph-mon[97044]: pgmap v14534: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:28.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:27.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:27.574378+0000) 2022-09-10T12:03:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:28.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:28.049611+0000) 2022-09-10T12:03:29.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:28.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:28.574536+0000) 2022-09-10T12:03:29.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:03:28.648+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:03:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:29.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:29.049836+0000) 2022-09-10T12:03:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:29 smithi110 ceph-mon[97633]: pgmap v14535: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:29 smithi104 ceph-mon[97044]: pgmap v14535: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:30.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:29.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:29.574759+0000) 2022-09-10T12:03:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:30.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:30.050064+0000) 2022-09-10T12:03:31.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:30.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:30.574962+0000) 2022-09-10T12:03:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:31.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:31.050268+0000) 2022-09-10T12:03:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:31 smithi110 ceph-mon[97633]: pgmap v14536: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:31 smithi104 ceph-mon[97044]: pgmap v14536: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:32.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:31.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:31.575173+0000) 2022-09-10T12:03:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:32.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:32.050528+0000) 2022-09-10T12:03:33.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:32.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:32.575371+0000) 2022-09-10T12:03:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:33.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:33.050747+0000) 2022-09-10T12:03:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:33 smithi110 ceph-mon[97633]: pgmap v14537: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:33 smithi104 ceph-mon[97044]: pgmap v14537: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:34.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:33.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:33.575564+0000) 2022-09-10T12:03:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:34.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:34.050901+0000) 2022-09-10T12:03:35.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:34.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:34.575753+0000) 2022-09-10T12:03:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:35.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:35.051165+0000) 2022-09-10T12:03:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:35 smithi104 ceph-mon[97044]: pgmap v14538: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:35 smithi110 ceph-mon[97633]: pgmap v14538: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:36.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:35.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:35.575979+0000) 2022-09-10T12:03:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:36.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:36.051349+0000) 2022-09-10T12:03:37.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:36.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:36.576236+0000) 2022-09-10T12:03:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:37.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:37.051567+0000) 2022-09-10T12:03:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:37 smithi110 ceph-mon[97633]: pgmap v14539: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:37 smithi104 ceph-mon[97044]: pgmap v14539: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:38.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:37.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:37.576388+0000) 2022-09-10T12:03:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:38.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:38.051796+0000) 2022-09-10T12:03:39.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:38.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:38.576566+0000) 2022-09-10T12:03:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:39.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:39.052008+0000) 2022-09-10T12:03:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:39 smithi110 ceph-mon[97633]: pgmap v14540: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:39 smithi104 ceph-mon[97044]: pgmap v14540: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:40.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:39.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:39.576778+0000) 2022-09-10T12:03:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:40.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:40.052191+0000) 2022-09-10T12:03:41.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:40.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:40.576901+0000) 2022-09-10T12:03:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:41.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:41.052369+0000) 2022-09-10T12:03:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:41 smithi110 ceph-mon[97633]: pgmap v14541: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:41 smithi104 ceph-mon[97044]: pgmap v14541: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:42.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:41.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:41.577125+0000) 2022-09-10T12:03:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:42.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:42.052548+0000) 2022-09-10T12:03:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:42.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:42.577379+0000) 2022-09-10T12:03:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:43.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:43.052790+0000) 2022-09-10T12:03:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:43 smithi110 ceph-mon[97633]: pgmap v14542: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:43 smithi104 ceph-mon[97044]: pgmap v14542: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:43.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:43.577588+0000) 2022-09-10T12:03:44.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:03:43.664+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:03:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:44.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:44.053033+0000) 2022-09-10T12:03:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:44.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:44.577833+0000) 2022-09-10T12:03:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:45.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:45.053238+0000) 2022-09-10T12:03:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:45 smithi104 ceph-mon[97044]: pgmap v14543: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:45 smithi110 ceph-mon[97633]: pgmap v14543: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:45.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:45.578027+0000) 2022-09-10T12:03:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:03:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:03:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:03:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:03:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:46.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:46.053472+0000) 2022-09-10T12:03:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:03:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:03:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:03:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:03:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:46.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:46.578198+0000) 2022-09-10T12:03:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:47.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:47.053708+0000) 2022-09-10T12:03:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:47 smithi110 ceph-mon[97633]: pgmap v14544: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:47 smithi104 ceph-mon[97044]: pgmap v14544: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:47.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:47.578427+0000) 2022-09-10T12:03:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:48.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:48.053927+0000) 2022-09-10T12:03:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:48.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:48.578661+0000) 2022-09-10T12:03:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:49.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:49.054073+0000) 2022-09-10T12:03:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:49 smithi110 ceph-mon[97633]: pgmap v14545: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:49 smithi104 ceph-mon[97044]: pgmap v14545: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:49.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:49.578909+0000) 2022-09-10T12:03:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:50.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:50.054181+0000) 2022-09-10T12:03:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:50.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:50.579068+0000) 2022-09-10T12:03:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:51.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:51.054316+0000) 2022-09-10T12:03:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:51 smithi110 ceph-mon[97633]: pgmap v14546: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:51 smithi104 ceph-mon[97044]: pgmap v14546: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:51.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:51.579272+0000) 2022-09-10T12:03:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:52.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:52.054466+0000) 2022-09-10T12:03:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:52.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:52.579476+0000) 2022-09-10T12:03:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:53.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:53.054626+0000) 2022-09-10T12:03:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:53 smithi110 ceph-mon[97633]: pgmap v14547: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:53 smithi104 ceph-mon[97044]: pgmap v14547: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:53.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:53.579723+0000) 2022-09-10T12:03:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:54.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:54.054834+0000) 2022-09-10T12:03:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:54.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:54.579908+0000) 2022-09-10T12:03:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:55 smithi104 ceph-mon[97044]: pgmap v14548: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:55.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:55.054999+0000) 2022-09-10T12:03:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:55 smithi110 ceph-mon[97633]: pgmap v14548: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:03:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:55.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:55.580149+0000) 2022-09-10T12:03:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:56.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:56.055207+0000) 2022-09-10T12:03:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:56.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:56.580397+0000) 2022-09-10T12:03:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:57.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:57.055345+0000) 2022-09-10T12:03:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:57 smithi110 ceph-mon[97633]: pgmap v14549: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:57 smithi104 ceph-mon[97044]: pgmap v14549: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:03:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:57.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:57.580607+0000) 2022-09-10T12:03:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:58.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:58.055545+0000) 2022-09-10T12:03:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:58.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:58.580790+0000) 2022-09-10T12:03:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:03:58.678+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:03:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:03:59.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:59.055813+0000) 2022-09-10T12:03:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:03:59 smithi110 ceph-mon[97633]: pgmap v14550: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:03:59 smithi104 ceph-mon[97044]: pgmap v14550: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:03:59.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:03:59.580995+0000) 2022-09-10T12:04:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:00.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:00.056023+0000) 2022-09-10T12:04:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:00.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:00.581209+0000) 2022-09-10T12:04:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:01.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:01.056175+0000) 2022-09-10T12:04:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:01 smithi110 ceph-mon[97633]: pgmap v14551: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:01 smithi104 ceph-mon[97044]: pgmap v14551: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:01.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:01.581408+0000) 2022-09-10T12:04:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:02.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:02.056360+0000) 2022-09-10T12:04:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:02.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:02.581620+0000) 2022-09-10T12:04:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:03.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:03.056553+0000) 2022-09-10T12:04:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:03 smithi110 ceph-mon[97633]: pgmap v14552: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:03 smithi104 ceph-mon[97044]: pgmap v14552: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:03.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:03.581870+0000) 2022-09-10T12:04:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:04.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:04.056738+0000) 2022-09-10T12:04:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:04.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:04.582077+0000) 2022-09-10T12:04:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:05 smithi104 ceph-mon[97044]: pgmap v14553: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:04:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:05.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:05.056902+0000) 2022-09-10T12:04:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:05 smithi110 ceph-mon[97633]: pgmap v14553: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:04:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:05.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:05.582314+0000) 2022-09-10T12:04:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:06.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:06.057072+0000) 2022-09-10T12:04:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:06.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:06.582558+0000) 2022-09-10T12:04:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:07.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:07.057239+0000) 2022-09-10T12:04:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:07 smithi110 ceph-mon[97633]: pgmap v14554: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:04:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:07 smithi104 ceph-mon[97044]: pgmap v14554: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:04:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:07.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:07.582807+0000) 2022-09-10T12:04:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:08.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:08.057432+0000) 2022-09-10T12:04:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:08.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:08.583036+0000) 2022-09-10T12:04:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:09.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:09.057666+0000) 2022-09-10T12:04:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:09 smithi110 ceph-mon[97633]: pgmap v14555: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:04:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:09 smithi110 ceph-mon[97633]: osdmap e990: 12 total, 12 up, 12 in 2022-09-10T12:04:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:09 smithi104 ceph-mon[97044]: pgmap v14555: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:04:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:09 smithi104 ceph-mon[97044]: osdmap e990: 12 total, 12 up, 12 in 2022-09-10T12:04:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:09.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:09.583212+0000) 2022-09-10T12:04:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:10.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:10.057840+0000) 2022-09-10T12:04:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:10.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:10.583443+0000) 2022-09-10T12:04:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:11.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:11.058036+0000) 2022-09-10T12:04:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:11 smithi110 ceph-mon[97633]: pgmap v14557: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:04:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:11 smithi104 ceph-mon[97044]: pgmap v14557: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:04:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:11.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:11.583716+0000) 2022-09-10T12:04:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:12.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:12.058215+0000) 2022-09-10T12:04:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:12.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:12.583995+0000) 2022-09-10T12:04:13.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:13.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:13.058349+0000) 2022-09-10T12:04:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:13 smithi110 ceph-mon[97633]: pgmap v14558: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:04:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:13 smithi104 ceph-mon[97044]: pgmap v14558: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:04:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:13.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:13.584213+0000) 2022-09-10T12:04:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:04:13.693+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:04:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:14.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:14.058487+0000) 2022-09-10T12:04:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:14.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:14.584452+0000) 2022-09-10T12:04:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:15 smithi104 ceph-mon[97044]: osdmap e991: 12 total, 12 up, 12 in 2022-09-10T12:04:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:15 smithi104 ceph-mon[97044]: pgmap v14560: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:04:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:15.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:15.058694+0000) 2022-09-10T12:04:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:15 smithi110 ceph-mon[97633]: osdmap e991: 12 total, 12 up, 12 in 2022-09-10T12:04:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:15 smithi110 ceph-mon[97633]: pgmap v14560: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:04:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:15.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:15.584676+0000) 2022-09-10T12:04:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:16.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:16.058891+0000) 2022-09-10T12:04:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:16.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:16.584864+0000) 2022-09-10T12:04:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:17.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:17.059140+0000) 2022-09-10T12:04:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:17 smithi110 ceph-mon[97633]: pgmap v14561: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:04:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:17 smithi104 ceph-mon[97044]: pgmap v14561: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:04:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:17.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:17.585139+0000) 2022-09-10T12:04:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:18.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:18.059318+0000) 2022-09-10T12:04:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:18.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:18.585322+0000) 2022-09-10T12:04:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:19.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:19.059475+0000) 2022-09-10T12:04:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:19 smithi110 ceph-mon[97633]: pgmap v14562: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:04:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:19 smithi104 ceph-mon[97044]: pgmap v14562: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:04:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:19.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:19.585551+0000) 2022-09-10T12:04:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:20.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:20.059629+0000) 2022-09-10T12:04:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:20.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:20.585749+0000) 2022-09-10T12:04:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:21.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:21.059853+0000) 2022-09-10T12:04:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:21 smithi110 ceph-mon[97633]: pgmap v14563: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:04:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:21 smithi104 ceph-mon[97044]: pgmap v14563: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:04:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:21.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:21.585971+0000) 2022-09-10T12:04:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:22.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:22.060054+0000) 2022-09-10T12:04:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:22.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:22.586213+0000) 2022-09-10T12:04:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:23.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:23.060188+0000) 2022-09-10T12:04:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:23 smithi110 ceph-mon[97633]: pgmap v14564: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:04:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:23 smithi104 ceph-mon[97044]: pgmap v14564: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:04:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:23.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:23.586458+0000) 2022-09-10T12:04:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:24.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:24.060352+0000) 2022-09-10T12:04:25.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:24.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:24.586680+0000) 2022-09-10T12:04:25.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:25 smithi104 ceph-mon[97044]: pgmap v14565: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T12:04:25.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:25.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:25.060532+0000) 2022-09-10T12:04:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:25 smithi110 ceph-mon[97633]: pgmap v14565: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T12:04:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:25.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:25.586894+0000) 2022-09-10T12:04:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:26.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:26.060670+0000) 2022-09-10T12:04:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:26.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:26.587171+0000) 2022-09-10T12:04:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:27.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:27.060903+0000) 2022-09-10T12:04:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:27 smithi110 ceph-mon[97633]: pgmap v14566: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:27 smithi104 ceph-mon[97044]: pgmap v14566: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:27.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:27.587421+0000) 2022-09-10T12:04:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:28.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:28.061132+0000) 2022-09-10T12:04:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:28.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:28.587628+0000) 2022-09-10T12:04:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:04:28.708+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:04:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:29.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:29.061291+0000) 2022-09-10T12:04:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:29 smithi110 ceph-mon[97633]: pgmap v14567: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:29 smithi104 ceph-mon[97044]: pgmap v14567: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:29.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:29.587893+0000) 2022-09-10T12:04:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:30.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:30.061462+0000) 2022-09-10T12:04:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:30.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:30.588168+0000) 2022-09-10T12:04:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:31.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:31.061650+0000) 2022-09-10T12:04:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:31 smithi110 ceph-mon[97633]: pgmap v14568: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:31 smithi104 ceph-mon[97044]: pgmap v14568: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:31.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:31.588362+0000) 2022-09-10T12:04:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:32.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:32.061864+0000) 2022-09-10T12:04:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:32.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:32.588591+0000) 2022-09-10T12:04:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:33.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:33.062134+0000) 2022-09-10T12:04:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:33 smithi110 ceph-mon[97633]: pgmap v14569: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:33 smithi104 ceph-mon[97044]: pgmap v14569: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:33.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:33.588809+0000) 2022-09-10T12:04:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:34.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:34.062320+0000) 2022-09-10T12:04:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:34.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:34.589054+0000) 2022-09-10T12:04:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:35.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:35.062510+0000) 2022-09-10T12:04:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:35 smithi110 ceph-mon[97633]: pgmap v14570: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:35 smithi104 ceph-mon[97044]: pgmap v14570: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:35.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:35.589292+0000) 2022-09-10T12:04:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:36.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:36.062696+0000) 2022-09-10T12:04:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:36.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:36.589557+0000) 2022-09-10T12:04:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:37.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:37.062933+0000) 2022-09-10T12:04:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:37 smithi110 ceph-mon[97633]: pgmap v14571: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:37 smithi104 ceph-mon[97044]: pgmap v14571: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:37.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:37.589813+0000) 2022-09-10T12:04:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:38.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:38.063132+0000) 2022-09-10T12:04:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:38.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:38.590143+0000) 2022-09-10T12:04:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:39.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:39.063325+0000) 2022-09-10T12:04:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:39 smithi110 ceph-mon[97633]: pgmap v14572: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:39 smithi104 ceph-mon[97044]: pgmap v14572: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:39.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:39.590344+0000) 2022-09-10T12:04:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:40.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:40.063501+0000) 2022-09-10T12:04:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:40.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:40.590487+0000) 2022-09-10T12:04:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:41.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:41.063733+0000) 2022-09-10T12:04:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:41 smithi110 ceph-mon[97633]: pgmap v14573: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:42.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:41 smithi104 ceph-mon[97044]: pgmap v14573: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:41.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:41.590675+0000) 2022-09-10T12:04:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:42.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:42.063975+0000) 2022-09-10T12:04:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:42.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:42.590901+0000) 2022-09-10T12:04:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:43.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:43.064147+0000) 2022-09-10T12:04:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:43 smithi110 ceph-mon[97633]: pgmap v14574: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:43 smithi104 ceph-mon[97044]: pgmap v14574: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:43.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:43.591130+0000) 2022-09-10T12:04:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:04:43.720+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:04:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:44.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:44.064284+0000) 2022-09-10T12:04:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:44.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:44.591375+0000) 2022-09-10T12:04:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:45.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:45.064497+0000) 2022-09-10T12:04:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:45 smithi110 ceph-mon[97633]: pgmap v14575: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:45 smithi104 ceph-mon[97044]: pgmap v14575: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:45.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:45.591637+0000) 2022-09-10T12:04:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:46.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:46.064737+0000) 2022-09-10T12:04:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:04:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:04:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:04:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:04:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:04:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:04:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:04:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:04:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:46.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:46.591885+0000) 2022-09-10T12:04:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:47.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:47.065003+0000) 2022-09-10T12:04:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:47 smithi110 ceph-mon[97633]: pgmap v14576: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:47 smithi104 ceph-mon[97044]: pgmap v14576: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:47.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:47.592067+0000) 2022-09-10T12:04:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:48.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:48.065223+0000) 2022-09-10T12:04:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:48.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:48.592292+0000) 2022-09-10T12:04:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:49.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:49.065359+0000) 2022-09-10T12:04:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:49 smithi110 ceph-mon[97633]: pgmap v14577: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:49 smithi104 ceph-mon[97044]: pgmap v14577: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:49.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:49.592559+0000) 2022-09-10T12:04:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:50.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:50.065630+0000) 2022-09-10T12:04:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:50.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:50.592800+0000) 2022-09-10T12:04:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:51.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:51.065861+0000) 2022-09-10T12:04:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:51 smithi110 ceph-mon[97633]: pgmap v14578: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:51 smithi104 ceph-mon[97044]: pgmap v14578: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:51.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:51.593048+0000) 2022-09-10T12:04:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:52.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:52.066072+0000) 2022-09-10T12:04:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:52.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:52.593281+0000) 2022-09-10T12:04:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:53.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:53.066338+0000) 2022-09-10T12:04:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:53 smithi110 ceph-mon[97633]: pgmap v14579: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:53 smithi104 ceph-mon[97044]: pgmap v14579: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:53.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:53.593524+0000) 2022-09-10T12:04:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:54.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:54.066569+0000) 2022-09-10T12:04:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:54.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:54.593786+0000) 2022-09-10T12:04:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:55.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:55.066810+0000) 2022-09-10T12:04:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:55 smithi110 ceph-mon[97633]: pgmap v14580: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:55 smithi104 ceph-mon[97044]: pgmap v14580: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:04:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:55.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:55.594063+0000) 2022-09-10T12:04:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:56.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:56.067044+0000) 2022-09-10T12:04:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:56.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:56.594291+0000) 2022-09-10T12:04:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:57.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:57.067241+0000) 2022-09-10T12:04:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:57 smithi110 ceph-mon[97633]: pgmap v14581: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:57 smithi104 ceph-mon[97044]: pgmap v14581: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:04:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:57.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:57.594501+0000) 2022-09-10T12:04:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:58.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:58.067475+0000) 2022-09-10T12:04:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:58.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:58.594712+0000) 2022-09-10T12:04:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:04:58.736+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:04:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:04:59.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:59.067683+0000) 2022-09-10T12:04:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:04:59 smithi110 ceph-mon[97633]: pgmap v14582: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:04:59 smithi104 ceph-mon[97044]: pgmap v14582: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:04:59.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:04:59.594962+0000) 2022-09-10T12:05:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:00.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:00.067907+0000) 2022-09-10T12:05:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:00.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:00.595221+0000) 2022-09-10T12:05:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:01.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:01.068138+0000) 2022-09-10T12:05:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:01 smithi110 ceph-mon[97633]: pgmap v14583: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:01 smithi104 ceph-mon[97044]: pgmap v14583: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:01.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:01.595464+0000) 2022-09-10T12:05:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:02.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:02.068324+0000) 2022-09-10T12:05:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:02.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:02.595661+0000) 2022-09-10T12:05:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:03.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:03.068519+0000) 2022-09-10T12:05:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:03 smithi110 ceph-mon[97633]: pgmap v14584: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:03 smithi104 ceph-mon[97044]: pgmap v14584: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:03.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:03.595852+0000) 2022-09-10T12:05:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:04.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:04.068757+0000) 2022-09-10T12:05:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:04.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:04.596146+0000) 2022-09-10T12:05:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:05.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:05.068961+0000) 2022-09-10T12:05:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:05 smithi110 ceph-mon[97633]: pgmap v14585: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:05:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:05 smithi104 ceph-mon[97044]: pgmap v14585: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:05:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:05.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:05.596376+0000) 2022-09-10T12:05:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:06.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:06.069194+0000) 2022-09-10T12:05:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:06.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:06.596616+0000) 2022-09-10T12:05:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:07.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:07.069410+0000) 2022-09-10T12:05:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:07 smithi110 ceph-mon[97633]: pgmap v14586: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:05:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:07 smithi104 ceph-mon[97044]: pgmap v14586: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:05:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:07.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:07.596875+0000) 2022-09-10T12:05:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:08.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:08.069580+0000) 2022-09-10T12:05:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:08.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:08.597089+0000) 2022-09-10T12:05:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:09.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:09.069814+0000) 2022-09-10T12:05:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:09 smithi110 ceph-mon[97633]: pgmap v14587: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:05:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:09 smithi110 ceph-mon[97633]: osdmap e992: 12 total, 12 up, 12 in 2022-09-10T12:05:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:09 smithi104 ceph-mon[97044]: pgmap v14587: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:05:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:09 smithi104 ceph-mon[97044]: osdmap e992: 12 total, 12 up, 12 in 2022-09-10T12:05:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:09.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:09.597294+0000) 2022-09-10T12:05:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:10.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:10.070043+0000) 2022-09-10T12:05:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:10.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:10.597466+0000) 2022-09-10T12:05:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:11.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:11.070216+0000) 2022-09-10T12:05:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:11 smithi110 ceph-mon[97633]: pgmap v14589: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:05:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:11 smithi104 ceph-mon[97044]: pgmap v14589: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:05:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:11.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:11.597664+0000) 2022-09-10T12:05:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:12.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:12.070427+0000) 2022-09-10T12:05:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:12.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:12.597911+0000) 2022-09-10T12:05:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:13.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:13.070595+0000) 2022-09-10T12:05:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:13 smithi110 ceph-mon[97633]: pgmap v14590: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:05:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:13 smithi104 ceph-mon[97044]: pgmap v14590: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:05:14.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:05:13.751+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:05:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:13.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:13.598089+0000) 2022-09-10T12:05:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:14.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:14.070798+0000) 2022-09-10T12:05:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:14.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:14.598361+0000) 2022-09-10T12:05:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:15.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:15.070944+0000) 2022-09-10T12:05:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:15 smithi110 ceph-mon[97633]: osdmap e993: 12 total, 12 up, 12 in 2022-09-10T12:05:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:15 smithi110 ceph-mon[97633]: pgmap v14592: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:05:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:15 smithi104 ceph-mon[97044]: osdmap e993: 12 total, 12 up, 12 in 2022-09-10T12:05:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:15 smithi104 ceph-mon[97044]: pgmap v14592: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:05:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:15.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:15.598607+0000) 2022-09-10T12:05:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:16.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:16.071147+0000) 2022-09-10T12:05:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:16.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:16.598816+0000) 2022-09-10T12:05:17.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:17.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:17.071305+0000) 2022-09-10T12:05:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:17 smithi110 ceph-mon[97633]: pgmap v14593: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:05:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:17 smithi104 ceph-mon[97044]: pgmap v14593: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:05:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:17.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:17.599034+0000) 2022-09-10T12:05:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:18.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:18.071492+0000) 2022-09-10T12:05:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:18.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:18.599279+0000) 2022-09-10T12:05:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:19.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:19.071678+0000) 2022-09-10T12:05:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:19 smithi110 ceph-mon[97633]: pgmap v14594: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T12:05:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:19 smithi104 ceph-mon[97044]: pgmap v14594: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T12:05:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:19.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:19.599455+0000) 2022-09-10T12:05:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:20.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:20.071854+0000) 2022-09-10T12:05:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:20.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:20.599715+0000) 2022-09-10T12:05:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:21.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:21.072076+0000) 2022-09-10T12:05:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:21 smithi110 ceph-mon[97633]: pgmap v14595: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:05:22.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:21 smithi104 ceph-mon[97044]: pgmap v14595: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:05:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:21.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:21.599941+0000) 2022-09-10T12:05:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:22.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:22.072284+0000) 2022-09-10T12:05:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:22.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:22.600200+0000) 2022-09-10T12:05:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:23.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:23.072491+0000) 2022-09-10T12:05:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:23 smithi110 ceph-mon[97633]: pgmap v14596: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:05:24.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:23 smithi104 ceph-mon[97044]: pgmap v14596: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:05:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:23.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:23.600479+0000) 2022-09-10T12:05:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:24.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:24.072685+0000) 2022-09-10T12:05:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:24.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:24.600689+0000) 2022-09-10T12:05:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:25.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:25.072926+0000) 2022-09-10T12:05:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:25 smithi110 ceph-mon[97633]: pgmap v14597: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 13 op/s 2022-09-10T12:05:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:25 smithi104 ceph-mon[97044]: pgmap v14597: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 13 op/s 2022-09-10T12:05:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:25.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:25.600827+0000) 2022-09-10T12:05:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:26.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:26.073207+0000) 2022-09-10T12:05:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:26.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:26.601046+0000) 2022-09-10T12:05:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:27.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:27.073384+0000) 2022-09-10T12:05:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:27 smithi110 ceph-mon[97633]: pgmap v14598: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:28.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:27 smithi104 ceph-mon[97044]: pgmap v14598: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:27.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:27.601280+0000) 2022-09-10T12:05:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:28.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:28.073574+0000) 2022-09-10T12:05:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:28.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:28.601486+0000) 2022-09-10T12:05:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:05:28.756+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:05:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:29.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:29.073820+0000) 2022-09-10T12:05:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:29 smithi110 ceph-mon[97633]: pgmap v14599: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:30.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:29 smithi104 ceph-mon[97044]: pgmap v14599: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:29.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:29.601717+0000) 2022-09-10T12:05:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:30.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:30.074062+0000) 2022-09-10T12:05:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:30.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:30.601919+0000) 2022-09-10T12:05:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:31.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:31.074274+0000) 2022-09-10T12:05:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:31 smithi110 ceph-mon[97633]: pgmap v14600: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:31 smithi104 ceph-mon[97044]: pgmap v14600: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:31.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:31.602167+0000) 2022-09-10T12:05:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:32.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:32.074516+0000) 2022-09-10T12:05:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:32.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:32.602338+0000) 2022-09-10T12:05:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:33.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:33.074757+0000) 2022-09-10T12:05:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:33 smithi110 ceph-mon[97633]: pgmap v14601: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:33 smithi104 ceph-mon[97044]: pgmap v14601: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:33.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:33.602533+0000) 2022-09-10T12:05:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:34.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:34.074982+0000) 2022-09-10T12:05:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:34.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:34.602715+0000) 2022-09-10T12:05:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:35.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:35.075220+0000) 2022-09-10T12:05:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:35 smithi110 ceph-mon[97633]: pgmap v14602: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:35 smithi104 ceph-mon[97044]: pgmap v14602: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:35.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:35.602915+0000) 2022-09-10T12:05:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:36.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:36.075431+0000) 2022-09-10T12:05:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:36.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:36.603196+0000) 2022-09-10T12:05:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:37.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:37.075692+0000) 2022-09-10T12:05:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:37 smithi110 ceph-mon[97633]: pgmap v14603: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:37 smithi104 ceph-mon[97044]: pgmap v14603: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:37.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:37.603469+0000) 2022-09-10T12:05:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:38.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:38.075852+0000) 2022-09-10T12:05:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:38.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:38.603629+0000) 2022-09-10T12:05:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:39.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:39.076118+0000) 2022-09-10T12:05:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:39 smithi110 ceph-mon[97633]: pgmap v14604: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:39 smithi104 ceph-mon[97044]: pgmap v14604: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:39.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:39.603847+0000) 2022-09-10T12:05:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:40.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:40.076346+0000) 2022-09-10T12:05:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:40.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:40.604041+0000) 2022-09-10T12:05:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:41.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:41.076608+0000) 2022-09-10T12:05:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:41 smithi110 ceph-mon[97633]: pgmap v14605: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:41 smithi104 ceph-mon[97044]: pgmap v14605: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:42.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:41.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:41.604282+0000) 2022-09-10T12:05:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:42.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:42.076869+0000) 2022-09-10T12:05:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:42.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:42.604447+0000) 2022-09-10T12:05:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:43.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:43.077088+0000) 2022-09-10T12:05:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:43 smithi110 ceph-mon[97633]: pgmap v14606: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:43 smithi104 ceph-mon[97044]: pgmap v14606: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:05:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:43.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:43.604673+0000) 2022-09-10T12:05:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:05:43.761+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:05:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:44.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:44.077316+0000) 2022-09-10T12:05:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:44.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:44.604960+0000) 2022-09-10T12:05:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:45.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:45.077558+0000) 2022-09-10T12:05:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:45 smithi110 ceph-mon[97633]: pgmap v14607: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:45.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:45 smithi104 ceph-mon[97044]: pgmap v14607: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:45.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:45.605213+0000) 2022-09-10T12:05:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:46.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:46.077784+0000) 2022-09-10T12:05:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:05:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:05:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:05:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:05:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:05:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:05:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:05:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:05:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:46.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:46.605409+0000) 2022-09-10T12:05:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:47.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:47.078011+0000) 2022-09-10T12:05:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:47 smithi110 ceph-mon[97633]: pgmap v14608: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:47 smithi104 ceph-mon[97044]: pgmap v14608: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:47.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:47.605649+0000) 2022-09-10T12:05:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:48.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:48.078225+0000) 2022-09-10T12:05:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:48.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:48.605927+0000) 2022-09-10T12:05:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:49.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:49.078414+0000) 2022-09-10T12:05:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:49 smithi110 ceph-mon[97633]: pgmap v14609: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:49 smithi104 ceph-mon[97044]: pgmap v14609: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:49.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:49.606171+0000) 2022-09-10T12:05:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:50.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:50.078626+0000) 2022-09-10T12:05:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:50.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:50.606358+0000) 2022-09-10T12:05:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:51.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:51.078854+0000) 2022-09-10T12:05:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:51 smithi110 ceph-mon[97633]: pgmap v14610: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:51 smithi104 ceph-mon[97044]: pgmap v14610: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:05:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:51.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:51.606581+0000) 2022-09-10T12:05:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:52.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:52.079067+0000) 2022-09-10T12:05:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:52.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:52.606844+0000) 2022-09-10T12:05:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:53.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:53.079277+0000) 2022-09-10T12:05:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:53 smithi110 ceph-mon[97633]: pgmap v14611: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:53 smithi104 ceph-mon[97044]: pgmap v14611: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:53.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:53.607057+0000) 2022-09-10T12:05:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:54.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:54.079454+0000) 2022-09-10T12:05:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:54.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:54.607280+0000) 2022-09-10T12:05:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:55.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:55.079685+0000) 2022-09-10T12:05:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:55 smithi110 ceph-mon[97633]: pgmap v14612: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T12:05:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:55 smithi104 ceph-mon[97044]: pgmap v14612: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T12:05:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:55.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:55.607451+0000) 2022-09-10T12:05:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:56.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:56.079897+0000) 2022-09-10T12:05:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:56.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:56.607672+0000) 2022-09-10T12:05:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:57.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:57.080177+0000) 2022-09-10T12:05:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:57 smithi110 ceph-mon[97633]: pgmap v14613: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:57 smithi104 ceph-mon[97044]: pgmap v14613: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:05:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:57.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:57.607906+0000) 2022-09-10T12:05:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:58.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:58.080423+0000) 2022-09-10T12:05:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:58.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:58.608204+0000) 2022-09-10T12:05:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:05:58.773+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:05:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:05:59.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:59.080679+0000) 2022-09-10T12:05:59.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:05:59 smithi110 ceph-mon[97633]: pgmap v14614: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:06:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:05:59 smithi104 ceph-mon[97044]: pgmap v14614: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:06:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:05:59.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:05:59.608373+0000) 2022-09-10T12:06:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:00.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:00.080894+0000) 2022-09-10T12:06:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:00.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:00.608582+0000) 2022-09-10T12:06:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:01.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:01.081151+0000) 2022-09-10T12:06:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:01 smithi110 ceph-mon[97633]: pgmap v14615: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:06:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:01 smithi104 ceph-mon[97044]: pgmap v14615: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:06:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:01.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:01.608803+0000) 2022-09-10T12:06:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:02.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:02.081365+0000) 2022-09-10T12:06:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:02.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:02.609068+0000) 2022-09-10T12:06:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:03.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:03.081589+0000) 2022-09-10T12:06:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:03 smithi110 ceph-mon[97633]: pgmap v14616: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:06:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:03 smithi104 ceph-mon[97044]: pgmap v14616: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:06:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:03.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:03.609306+0000) 2022-09-10T12:06:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:04.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:04.081766+0000) 2022-09-10T12:06:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:04.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:04.609564+0000) 2022-09-10T12:06:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:05.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:05.081981+0000) 2022-09-10T12:06:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:05 smithi110 ceph-mon[97633]: pgmap v14617: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:06:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:05 smithi104 ceph-mon[97044]: pgmap v14617: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:06:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:05.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:05.609716+0000) 2022-09-10T12:06:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:06.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:06.082228+0000) 2022-09-10T12:06:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:06.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:06.609943+0000) 2022-09-10T12:06:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:07.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:07.082413+0000) 2022-09-10T12:06:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:07 smithi110 ceph-mon[97633]: pgmap v14618: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:06:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:07 smithi104 ceph-mon[97044]: pgmap v14618: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:06:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:07.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:07.610169+0000) 2022-09-10T12:06:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:08.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:08.082665+0000) 2022-09-10T12:06:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:08.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:08.610330+0000) 2022-09-10T12:06:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:09.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:09.082939+0000) 2022-09-10T12:06:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:09 smithi110 ceph-mon[97633]: pgmap v14619: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:06:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:09 smithi110 ceph-mon[97633]: osdmap e994: 12 total, 12 up, 12 in 2022-09-10T12:06:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:09 smithi104 ceph-mon[97044]: pgmap v14619: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:06:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:09 smithi104 ceph-mon[97044]: osdmap e994: 12 total, 12 up, 12 in 2022-09-10T12:06:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:09.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:09.610494+0000) 2022-09-10T12:06:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:10.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:10.083196+0000) 2022-09-10T12:06:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:10.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:10.610710+0000) 2022-09-10T12:06:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:11.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:11.083402+0000) 2022-09-10T12:06:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:11 smithi110 ceph-mon[97633]: pgmap v14621: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 29 op/s 2022-09-10T12:06:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:11 smithi104 ceph-mon[97044]: pgmap v14621: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 29 op/s 2022-09-10T12:06:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:11.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:11.610882+0000) 2022-09-10T12:06:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:12.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:12.083606+0000) 2022-09-10T12:06:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:12.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:12.611075+0000) 2022-09-10T12:06:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:13.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:13.083836+0000) 2022-09-10T12:06:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:13 smithi110 ceph-mon[97633]: pgmap v14622: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 29 op/s 2022-09-10T12:06:14.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:13 smithi104 ceph-mon[97044]: pgmap v14622: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 29 op/s 2022-09-10T12:06:14.082 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:13.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:13.611340+0000) 2022-09-10T12:06:14.082 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:06:13.789+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:06:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:14.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:14.084051+0000) 2022-09-10T12:06:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:14.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:14.611613+0000) 2022-09-10T12:06:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:15.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:15.084258+0000) 2022-09-10T12:06:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:15 smithi110 ceph-mon[97633]: osdmap e995: 12 total, 12 up, 12 in 2022-09-10T12:06:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:15 smithi110 ceph-mon[97633]: pgmap v14624: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 14 op/s 2022-09-10T12:06:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:15 smithi104 ceph-mon[97044]: osdmap e995: 12 total, 12 up, 12 in 2022-09-10T12:06:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:15 smithi104 ceph-mon[97044]: pgmap v14624: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 14 op/s 2022-09-10T12:06:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:15.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:15.611868+0000) 2022-09-10T12:06:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:16.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:16.084503+0000) 2022-09-10T12:06:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:16.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:16.612065+0000) 2022-09-10T12:06:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:17.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:17.084683+0000) 2022-09-10T12:06:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:17 smithi110 ceph-mon[97633]: pgmap v14625: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 14 op/s 2022-09-10T12:06:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:17 smithi104 ceph-mon[97044]: pgmap v14625: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 14 op/s 2022-09-10T12:06:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:17.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:17.612337+0000) 2022-09-10T12:06:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:18.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:18.084911+0000) 2022-09-10T12:06:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:18.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:18.612496+0000) 2022-09-10T12:06:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:19.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:19.085175+0000) 2022-09-10T12:06:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:19 smithi110 ceph-mon[97633]: pgmap v14626: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 11 op/s 2022-09-10T12:06:20.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:19 smithi104 ceph-mon[97044]: pgmap v14626: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 205 B/s wr, 11 op/s 2022-09-10T12:06:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:19.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:19.612671+0000) 2022-09-10T12:06:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:20.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:20.085357+0000) 2022-09-10T12:06:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:20.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:20.612937+0000) 2022-09-10T12:06:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:21.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:21.085582+0000) 2022-09-10T12:06:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:21 smithi110 ceph-mon[97633]: pgmap v14627: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 11 op/s 2022-09-10T12:06:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:21 smithi104 ceph-mon[97044]: pgmap v14627: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 11 op/s 2022-09-10T12:06:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:21.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:21.613178+0000) 2022-09-10T12:06:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:22.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:22.085791+0000) 2022-09-10T12:06:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:22.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:22.613409+0000) 2022-09-10T12:06:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:23.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:23.085989+0000) 2022-09-10T12:06:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:23 smithi110 ceph-mon[97633]: pgmap v14628: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 11 op/s 2022-09-10T12:06:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:23 smithi104 ceph-mon[97044]: pgmap v14628: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 11 op/s 2022-09-10T12:06:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:23.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:23.613595+0000) 2022-09-10T12:06:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:24.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:24.086224+0000) 2022-09-10T12:06:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:24.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:24.613855+0000) 2022-09-10T12:06:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:25.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:25.086454+0000) 2022-09-10T12:06:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:25 smithi110 ceph-mon[97633]: pgmap v14629: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T12:06:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:25 smithi104 ceph-mon[97044]: pgmap v14629: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 99 B/s wr, 14 op/s 2022-09-10T12:06:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:25.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:25.614082+0000) 2022-09-10T12:06:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:26.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:26.086683+0000) 2022-09-10T12:06:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:26.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:26.614339+0000) 2022-09-10T12:06:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:27.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:27.086915+0000) 2022-09-10T12:06:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:27 smithi110 ceph-mon[97633]: pgmap v14630: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:27 smithi104 ceph-mon[97044]: pgmap v14630: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:27.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:27.614582+0000) 2022-09-10T12:06:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:28.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:28.087172+0000) 2022-09-10T12:06:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:28.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:28.614776+0000) 2022-09-10T12:06:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:06:28.801+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:06:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:29.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:29.087432+0000) 2022-09-10T12:06:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:29 smithi110 ceph-mon[97633]: pgmap v14631: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:29 smithi104 ceph-mon[97044]: pgmap v14631: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:29.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:29.614952+0000) 2022-09-10T12:06:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:30.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:30.087665+0000) 2022-09-10T12:06:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:30.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:30.615162+0000) 2022-09-10T12:06:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:31.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:31.087887+0000) 2022-09-10T12:06:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:31 smithi110 ceph-mon[97633]: pgmap v14632: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:31 smithi104 ceph-mon[97044]: pgmap v14632: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:31.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:31.615417+0000) 2022-09-10T12:06:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:32.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:32.088103+0000) 2022-09-10T12:06:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:32.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:32.615557+0000) 2022-09-10T12:06:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:33.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:33.088334+0000) 2022-09-10T12:06:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:33 smithi110 ceph-mon[97633]: pgmap v14633: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:33 smithi104 ceph-mon[97044]: pgmap v14633: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:33.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:33.615785+0000) 2022-09-10T12:06:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:34.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:34.088589+0000) 2022-09-10T12:06:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:34.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:34.616025+0000) 2022-09-10T12:06:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:35.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:35.088770+0000) 2022-09-10T12:06:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:35 smithi110 ceph-mon[97633]: pgmap v14634: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:35 smithi104 ceph-mon[97044]: pgmap v14634: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:35.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:35.616236+0000) 2022-09-10T12:06:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:36.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:36.088985+0000) 2022-09-10T12:06:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:36.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:36.616412+0000) 2022-09-10T12:06:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:37.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:37.089255+0000) 2022-09-10T12:06:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:37 smithi110 ceph-mon[97633]: pgmap v14635: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:37 smithi104 ceph-mon[97044]: pgmap v14635: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:37.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:37.616553+0000) 2022-09-10T12:06:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:38.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:38.089553+0000) 2022-09-10T12:06:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:38.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:38.616791+0000) 2022-09-10T12:06:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:39.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:39.089766+0000) 2022-09-10T12:06:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:39 smithi110 ceph-mon[97633]: pgmap v14636: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:39 smithi104 ceph-mon[97044]: pgmap v14636: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:39.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:39.617015+0000) 2022-09-10T12:06:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:40.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:40.089979+0000) 2022-09-10T12:06:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:40.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:40.617218+0000) 2022-09-10T12:06:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:41.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:41.090199+0000) 2022-09-10T12:06:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:41 smithi110 ceph-mon[97633]: pgmap v14637: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:41 smithi104 ceph-mon[97044]: pgmap v14637: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:41.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:41.617504+0000) 2022-09-10T12:06:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:42.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:42.090414+0000) 2022-09-10T12:06:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:42.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:42.617769+0000) 2022-09-10T12:06:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:43.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:43.090600+0000) 2022-09-10T12:06:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:43 smithi110 ceph-mon[97633]: pgmap v14638: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:43 smithi104 ceph-mon[97044]: pgmap v14638: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:43.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:43.617918+0000) 2022-09-10T12:06:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:06:43.816+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:06:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:44.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:44.090832+0000) 2022-09-10T12:06:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:44.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:44.618085+0000) 2022-09-10T12:06:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:45.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:45.091000+0000) 2022-09-10T12:06:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:45 smithi110 ceph-mon[97633]: pgmap v14639: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:45 smithi104 ceph-mon[97044]: pgmap v14639: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:45.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:45.618251+0000) 2022-09-10T12:06:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:46.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:46.091232+0000) 2022-09-10T12:06:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:06:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:06:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:06:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:06:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:06:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:46.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:46.618413+0000) 2022-09-10T12:06:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:47.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:47.091467+0000) 2022-09-10T12:06:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:47 smithi110 ceph-mon[97633]: pgmap v14640: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:47 smithi104 ceph-mon[97044]: pgmap v14640: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:47.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:47.618650+0000) 2022-09-10T12:06:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:48.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:48.091680+0000) 2022-09-10T12:06:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:48.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:48.618892+0000) 2022-09-10T12:06:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:49.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:49.091895+0000) 2022-09-10T12:06:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:49 smithi110 ceph-mon[97633]: pgmap v14641: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:49 smithi104 ceph-mon[97044]: pgmap v14641: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:49.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:49.619170+0000) 2022-09-10T12:06:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:50.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:50.092158+0000) 2022-09-10T12:06:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:50.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:50.619310+0000) 2022-09-10T12:06:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:51.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:51.092325+0000) 2022-09-10T12:06:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:51 smithi110 ceph-mon[97633]: pgmap v14642: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:51 smithi104 ceph-mon[97044]: pgmap v14642: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:51.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:51.619529+0000) 2022-09-10T12:06:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:52.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:52.092523+0000) 2022-09-10T12:06:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:52.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:52.619765+0000) 2022-09-10T12:06:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:53.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:53.092808+0000) 2022-09-10T12:06:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:53 smithi110 ceph-mon[97633]: pgmap v14643: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:53 smithi104 ceph-mon[97044]: pgmap v14643: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:53.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:53.619942+0000) 2022-09-10T12:06:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:54.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:54.093010+0000) 2022-09-10T12:06:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:54.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:54.620163+0000) 2022-09-10T12:06:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:55.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:55.093221+0000) 2022-09-10T12:06:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:55 smithi110 ceph-mon[97633]: pgmap v14644: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:55 smithi104 ceph-mon[97044]: pgmap v14644: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:06:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:55.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:55.620304+0000) 2022-09-10T12:06:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:56.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:56.093423+0000) 2022-09-10T12:06:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:56.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:56.620485+0000) 2022-09-10T12:06:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:57.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:57.093643+0000) 2022-09-10T12:06:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:57 smithi110 ceph-mon[97633]: pgmap v14645: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:57 smithi104 ceph-mon[97044]: pgmap v14645: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:06:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:57.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:57.620748+0000) 2022-09-10T12:06:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:58.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:58.093825+0000) 2022-09-10T12:06:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:58.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:58.620925+0000) 2022-09-10T12:06:59.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:06:58.832+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:06:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:06:59.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:59.094017+0000) 2022-09-10T12:06:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:06:59 smithi110 ceph-mon[97633]: pgmap v14646: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:07:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:06:59 smithi104 ceph-mon[97044]: pgmap v14646: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:07:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:06:59.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:06:59.621195+0000) 2022-09-10T12:07:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:00.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:00.094248+0000) 2022-09-10T12:07:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:00.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:00.621339+0000) 2022-09-10T12:07:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:01.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:01.094477+0000) 2022-09-10T12:07:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:01 smithi110 ceph-mon[97633]: pgmap v14647: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:07:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:01 smithi104 ceph-mon[97044]: pgmap v14647: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 13 op/s 2022-09-10T12:07:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:01.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:01.621556+0000) 2022-09-10T12:07:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:02.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:02.094720+0000) 2022-09-10T12:07:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:02.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:02.621825+0000) 2022-09-10T12:07:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:03.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:03.094940+0000) 2022-09-10T12:07:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:03 smithi110 ceph-mon[97633]: pgmap v14648: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:07:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:03 smithi104 ceph-mon[97044]: pgmap v14648: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T12:07:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:03.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:03.622021+0000) 2022-09-10T12:07:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:04.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:04.095190+0000) 2022-09-10T12:07:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:04.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:04.622274+0000) 2022-09-10T12:07:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:05.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:05.095439+0000) 2022-09-10T12:07:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:05 smithi110 ceph-mon[97633]: pgmap v14649: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 28 op/s 2022-09-10T12:07:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:05 smithi104 ceph-mon[97044]: pgmap v14649: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 28 op/s 2022-09-10T12:07:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:05.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:05.622455+0000) 2022-09-10T12:07:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:06.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:06.095689+0000) 2022-09-10T12:07:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:06.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:06.622636+0000) 2022-09-10T12:07:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:07.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:07.095880+0000) 2022-09-10T12:07:07.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:07 smithi110 ceph-mon[97633]: pgmap v14650: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:07:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:07 smithi104 ceph-mon[97044]: pgmap v14650: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:07:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:07.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:07.622871+0000) 2022-09-10T12:07:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:08.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:08.096159+0000) 2022-09-10T12:07:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:08.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:08.623161+0000) 2022-09-10T12:07:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:09.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:09.096351+0000) 2022-09-10T12:07:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:09 smithi110 ceph-mon[97633]: pgmap v14651: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:07:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:09 smithi110 ceph-mon[97633]: osdmap e996: 12 total, 12 up, 12 in 2022-09-10T12:07:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:09 smithi104 ceph-mon[97044]: pgmap v14651: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:07:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:09 smithi104 ceph-mon[97044]: osdmap e996: 12 total, 12 up, 12 in 2022-09-10T12:07:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:09.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:09.623389+0000) 2022-09-10T12:07:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:10.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:10.096603+0000) 2022-09-10T12:07:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:10.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:10.623599+0000) 2022-09-10T12:07:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:11.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:11.096851+0000) 2022-09-10T12:07:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:11 smithi110 ceph-mon[97633]: pgmap v14653: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:07:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:11 smithi104 ceph-mon[97044]: pgmap v14653: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:07:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:11.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:11.623842+0000) 2022-09-10T12:07:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:12.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:12.097045+0000) 2022-09-10T12:07:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:12.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:12.624010+0000) 2022-09-10T12:07:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:13.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:13.097271+0000) 2022-09-10T12:07:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:13 smithi110 ceph-mon[97633]: pgmap v14654: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:07:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:13 smithi104 ceph-mon[97044]: pgmap v14654: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:07:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:13.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:13.624234+0000) 2022-09-10T12:07:14.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:07:13.846+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:07:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:14.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:14.097451+0000) 2022-09-10T12:07:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:14.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:14.624428+0000) 2022-09-10T12:07:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:15.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:15.097640+0000) 2022-09-10T12:07:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:15 smithi110 ceph-mon[97633]: osdmap e997: 12 total, 12 up, 12 in 2022-09-10T12:07:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:15 smithi110 ceph-mon[97633]: pgmap v14656: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:07:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:15 smithi104 ceph-mon[97044]: osdmap e997: 12 total, 12 up, 12 in 2022-09-10T12:07:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:15 smithi104 ceph-mon[97044]: pgmap v14656: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:07:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:15.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:15.624648+0000) 2022-09-10T12:07:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:16.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:16.097870+0000) 2022-09-10T12:07:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:16.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:16.624879+0000) 2022-09-10T12:07:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:17.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:17.098053+0000) 2022-09-10T12:07:17.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:17 smithi110 ceph-mon[97633]: pgmap v14657: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:07:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:17 smithi104 ceph-mon[97044]: pgmap v14657: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:07:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:17.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:17.625066+0000) 2022-09-10T12:07:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:18.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:18.098209+0000) 2022-09-10T12:07:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:18.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:18.625334+0000) 2022-09-10T12:07:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:19.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:19.098418+0000) 2022-09-10T12:07:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:19 smithi110 ceph-mon[97633]: pgmap v14658: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:07:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:19 smithi104 ceph-mon[97044]: pgmap v14658: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T12:07:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:19.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:19.625581+0000) 2022-09-10T12:07:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:20.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:20.098647+0000) 2022-09-10T12:07:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:20.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:20.625822+0000) 2022-09-10T12:07:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:21.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:21.098849+0000) 2022-09-10T12:07:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:21 smithi110 ceph-mon[97633]: pgmap v14659: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:07:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:21 smithi104 ceph-mon[97044]: pgmap v14659: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:07:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:21.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:21.626066+0000) 2022-09-10T12:07:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:22.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:22.099027+0000) 2022-09-10T12:07:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:22.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:22.626282+0000) 2022-09-10T12:07:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:23.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:23.099236+0000) 2022-09-10T12:07:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:23 smithi110 ceph-mon[97633]: pgmap v14660: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:07:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:23 smithi104 ceph-mon[97044]: pgmap v14660: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:07:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:23.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:23.626462+0000) 2022-09-10T12:07:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:24.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:24.099411+0000) 2022-09-10T12:07:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:24.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:24.626735+0000) 2022-09-10T12:07:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:25.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:25.099644+0000) 2022-09-10T12:07:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:25 smithi110 ceph-mon[97633]: pgmap v14661: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:07:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:25 smithi104 ceph-mon[97044]: pgmap v14661: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:07:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:25.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:25.626980+0000) 2022-09-10T12:07:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:26.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:26.099892+0000) 2022-09-10T12:07:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:26.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:26.627215+0000) 2022-09-10T12:07:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:27.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:27.100158+0000) 2022-09-10T12:07:27.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:27 smithi110 ceph-mon[97633]: pgmap v14662: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:27 smithi104 ceph-mon[97044]: pgmap v14662: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:27.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:27.627394+0000) 2022-09-10T12:07:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:28.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:28.100279+0000) 2022-09-10T12:07:29.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:07:28.862+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:07:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:28.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:28.627606+0000) 2022-09-10T12:07:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:29.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:29.100464+0000) 2022-09-10T12:07:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:29 smithi110 ceph-mon[97633]: pgmap v14663: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:29 smithi104 ceph-mon[97044]: pgmap v14663: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:29.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:29.627877+0000) 2022-09-10T12:07:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:30.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:30.100720+0000) 2022-09-10T12:07:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:30.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:30.628145+0000) 2022-09-10T12:07:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:31.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:31.100935+0000) 2022-09-10T12:07:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:31 smithi104 ceph-mon[97044]: pgmap v14664: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:31.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:31.628348+0000) 2022-09-10T12:07:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:31 smithi110 ceph-mon[97633]: pgmap v14664: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:32.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:32.101116+0000) 2022-09-10T12:07:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:32.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:32.628610+0000) 2022-09-10T12:07:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:33.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:33.101276+0000) 2022-09-10T12:07:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:33 smithi104 ceph-mon[97044]: pgmap v14665: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:33.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:33.628792+0000) 2022-09-10T12:07:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:33 smithi110 ceph-mon[97633]: pgmap v14665: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:34.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:34.101507+0000) 2022-09-10T12:07:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:34.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:34.629001+0000) 2022-09-10T12:07:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:35.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:35.101738+0000) 2022-09-10T12:07:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:35 smithi110 ceph-mon[97633]: pgmap v14666: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:35 smithi104 ceph-mon[97044]: pgmap v14666: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:35.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:35.629232+0000) 2022-09-10T12:07:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:36.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:36.101980+0000) 2022-09-10T12:07:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:36.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:36.629460+0000) 2022-09-10T12:07:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:37.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:37.102245+0000) 2022-09-10T12:07:37.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:37 smithi110 ceph-mon[97633]: pgmap v14667: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:37 smithi104 ceph-mon[97044]: pgmap v14667: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:37.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:37.629703+0000) 2022-09-10T12:07:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:38.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:38.102497+0000) 2022-09-10T12:07:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:38.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:38.629934+0000) 2022-09-10T12:07:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:39.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:39.102683+0000) 2022-09-10T12:07:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:39 smithi110 ceph-mon[97633]: pgmap v14668: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:39 smithi104 ceph-mon[97044]: pgmap v14668: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:39.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:39.630177+0000) 2022-09-10T12:07:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:40.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:40.102932+0000) 2022-09-10T12:07:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:40.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:40.630392+0000) 2022-09-10T12:07:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:41.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:41.103199+0000) 2022-09-10T12:07:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:41 smithi110 ceph-mon[97633]: pgmap v14669: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:41 smithi104 ceph-mon[97044]: pgmap v14669: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:41.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:41.630531+0000) 2022-09-10T12:07:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:42.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:42.103427+0000) 2022-09-10T12:07:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:42.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:42.630766+0000) 2022-09-10T12:07:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:43.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:43.103665+0000) 2022-09-10T12:07:44.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:43 smithi104 ceph-mon[97044]: pgmap v14670: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:43.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:43.630938+0000) 2022-09-10T12:07:44.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:07:43.877+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:07:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:43 smithi110 ceph-mon[97633]: pgmap v14670: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:44.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:44.103844+0000) 2022-09-10T12:07:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:44.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:44.631138+0000) 2022-09-10T12:07:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:45.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:45.104081+0000) 2022-09-10T12:07:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:45 smithi110 ceph-mon[97633]: pgmap v14671: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:45 smithi104 ceph-mon[97044]: pgmap v14671: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:45.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:45.631385+0000) 2022-09-10T12:07:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:46.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:46.104366+0000) 2022-09-10T12:07:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:07:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:07:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:46.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:46.631648+0000) 2022-09-10T12:07:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:47.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:47.104637+0000) 2022-09-10T12:07:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:47 smithi110 ceph-mon[97633]: pgmap v14672: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:47 smithi104 ceph-mon[97044]: pgmap v14672: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:47.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:47.631887+0000) 2022-09-10T12:07:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:48.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:48.104868+0000) 2022-09-10T12:07:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:48.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:48.632152+0000) 2022-09-10T12:07:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:49.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:49.105141+0000) 2022-09-10T12:07:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:49 smithi110 ceph-mon[97633]: pgmap v14673: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:49 smithi104 ceph-mon[97044]: pgmap v14673: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:49.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:49.632348+0000) 2022-09-10T12:07:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:50.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:50.105404+0000) 2022-09-10T12:07:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:50.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:50.632627+0000) 2022-09-10T12:07:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:51.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:51.105547+0000) 2022-09-10T12:07:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:51 smithi104 ceph-mon[97044]: pgmap v14674: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:51.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:51.632884+0000) 2022-09-10T12:07:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:51 smithi110 ceph-mon[97633]: pgmap v14674: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:52.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:52.105760+0000) 2022-09-10T12:07:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:52.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:52.633041+0000) 2022-09-10T12:07:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:53.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:53.106006+0000) 2022-09-10T12:07:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:53 smithi104 ceph-mon[97044]: pgmap v14675: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:53.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:53.633215+0000) 2022-09-10T12:07:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:53 smithi110 ceph-mon[97633]: pgmap v14675: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:54.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:54.106251+0000) 2022-09-10T12:07:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:54.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:54.633437+0000) 2022-09-10T12:07:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:55.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:55.106403+0000) 2022-09-10T12:07:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:55 smithi110 ceph-mon[97633]: pgmap v14676: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:55 smithi104 ceph-mon[97044]: pgmap v14676: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:07:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:55.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:55.633592+0000) 2022-09-10T12:07:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:56.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:56.106597+0000) 2022-09-10T12:07:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:56.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:56.633837+0000) 2022-09-10T12:07:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:57.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:57.106854+0000) 2022-09-10T12:07:57.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:57 smithi110 ceph-mon[97633]: pgmap v14677: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:57 smithi104 ceph-mon[97044]: pgmap v14677: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:07:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:57.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:57.634087+0000) 2022-09-10T12:07:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:58.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:58.107034+0000) 2022-09-10T12:07:58.891 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:58.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:58.634268+0000) 2022-09-10T12:07:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:07:59.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:59.107274+0000) 2022-09-10T12:07:59.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:07:58.891+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:08:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:07:59 smithi104 ceph-mon[97044]: pgmap v14678: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:07:59.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:07:59.634494+0000) 2022-09-10T12:08:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:07:59 smithi110 ceph-mon[97633]: pgmap v14678: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:00.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:00.107475+0000) 2022-09-10T12:08:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:00.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:00.634678+0000) 2022-09-10T12:08:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:01.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:01.107648+0000) 2022-09-10T12:08:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:01 smithi104 ceph-mon[97044]: pgmap v14679: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:01.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:01.634888+0000) 2022-09-10T12:08:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:01 smithi110 ceph-mon[97633]: pgmap v14679: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:02.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:02.107883+0000) 2022-09-10T12:08:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:02.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:02.635103+0000) 2022-09-10T12:08:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:03.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:03.108085+0000) 2022-09-10T12:08:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:03 smithi104 ceph-mon[97044]: pgmap v14680: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:08:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:03.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:03.635322+0000) 2022-09-10T12:08:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:03 smithi110 ceph-mon[97633]: pgmap v14680: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:08:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:04.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:04.108246+0000) 2022-09-10T12:08:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:04.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:04.635587+0000) 2022-09-10T12:08:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:05.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:05.108501+0000) 2022-09-10T12:08:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:05 smithi110 ceph-mon[97633]: pgmap v14681: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:08:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:05 smithi104 ceph-mon[97044]: pgmap v14681: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:08:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:05.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:05.635823+0000) 2022-09-10T12:08:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:06.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:06.108677+0000) 2022-09-10T12:08:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:06.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:06.636147+0000) 2022-09-10T12:08:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:07.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:07.108915+0000) 2022-09-10T12:08:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:07 smithi104 ceph-mon[97044]: pgmap v14682: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T12:08:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:07.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:07.636384+0000) 2022-09-10T12:08:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:07 smithi110 ceph-mon[97633]: pgmap v14682: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T12:08:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:08.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:08.109191+0000) 2022-09-10T12:08:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:08 smithi104 ceph-mon[97044]: osdmap e998: 12 total, 12 up, 12 in 2022-09-10T12:08:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:08.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:08.636567+0000) 2022-09-10T12:08:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:08 smithi110 ceph-mon[97633]: osdmap e998: 12 total, 12 up, 12 in 2022-09-10T12:08:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:09.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:09.109427+0000) 2022-09-10T12:08:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:09 smithi104 ceph-mon[97044]: pgmap v14684: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:08:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:09.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:09.636734+0000) 2022-09-10T12:08:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:09 smithi110 ceph-mon[97633]: pgmap v14684: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:08:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:10.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:10.109648+0000) 2022-09-10T12:08:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:10.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:10.636954+0000) 2022-09-10T12:08:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:11.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:11.109906+0000) 2022-09-10T12:08:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:11 smithi104 ceph-mon[97044]: pgmap v14685: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:08:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:11.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:11.637208+0000) 2022-09-10T12:08:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:11 smithi110 ceph-mon[97633]: pgmap v14685: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:08:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:12.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:12.110183+0000) 2022-09-10T12:08:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:12.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:12.637453+0000) 2022-09-10T12:08:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:13.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:13.110394+0000) 2022-09-10T12:08:13.896 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:13.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:13.637655+0000) 2022-09-10T12:08:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:13 smithi104 ceph-mon[97044]: pgmap v14686: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:08:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:13 smithi110 ceph-mon[97633]: pgmap v14686: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:08:14.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:08:13.894+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:08:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:14.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:14.110551+0000) 2022-09-10T12:08:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:14.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:14.637933+0000) 2022-09-10T12:08:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:15.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:15.110823+0000) 2022-09-10T12:08:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:15 smithi110 ceph-mon[97633]: osdmap e999: 12 total, 12 up, 12 in 2022-09-10T12:08:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:15 smithi110 ceph-mon[97633]: pgmap v14688: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:08:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:15 smithi104 ceph-mon[97044]: osdmap e999: 12 total, 12 up, 12 in 2022-09-10T12:08:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:15 smithi104 ceph-mon[97044]: pgmap v14688: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:08:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:15.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:15.638187+0000) 2022-09-10T12:08:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:16.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:16.111006+0000) 2022-09-10T12:08:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:16.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:16.638402+0000) 2022-09-10T12:08:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:17.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:17.111243+0000) 2022-09-10T12:08:18.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:17 smithi104 ceph-mon[97044]: pgmap v14689: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:08:18.098 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:17.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:17.638634+0000) 2022-09-10T12:08:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:17 smithi110 ceph-mon[97633]: pgmap v14689: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:08:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:18.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:18.111507+0000) 2022-09-10T12:08:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:18.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:18.638813+0000) 2022-09-10T12:08:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:19.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:19.111801+0000) 2022-09-10T12:08:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:19 smithi104 ceph-mon[97044]: pgmap v14690: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:08:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:19.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:19.639090+0000) 2022-09-10T12:08:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:19 smithi110 ceph-mon[97633]: pgmap v14690: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:08:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:20.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:20.111971+0000) 2022-09-10T12:08:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:20.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:20.639327+0000) 2022-09-10T12:08:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:21.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:21.112204+0000) 2022-09-10T12:08:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:21 smithi104 ceph-mon[97044]: pgmap v14691: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:08:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:21.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:21.639515+0000) 2022-09-10T12:08:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:21 smithi110 ceph-mon[97633]: pgmap v14691: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:08:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:22.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:22.112380+0000) 2022-09-10T12:08:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:22.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:22.639780+0000) 2022-09-10T12:08:23.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:23.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:23.112565+0000) 2022-09-10T12:08:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:23 smithi104 ceph-mon[97044]: pgmap v14692: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:08:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:23.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:23.640039+0000) 2022-09-10T12:08:24.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:23 smithi110 ceph-mon[97633]: pgmap v14692: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:08:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:24.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:24.112759+0000) 2022-09-10T12:08:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:24.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:24.640253+0000) 2022-09-10T12:08:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:25.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:25.112978+0000) 2022-09-10T12:08:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:25 smithi110 ceph-mon[97633]: pgmap v14693: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:08:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:25 smithi104 ceph-mon[97044]: pgmap v14693: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:08:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:25.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:25.640469+0000) 2022-09-10T12:08:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:26.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:26.113170+0000) 2022-09-10T12:08:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:26.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:26.640704+0000) 2022-09-10T12:08:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:27.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:27.113434+0000) 2022-09-10T12:08:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:27 smithi104 ceph-mon[97044]: pgmap v14694: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:27.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:27.640972+0000) 2022-09-10T12:08:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:27 smithi110 ceph-mon[97633]: pgmap v14694: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:28.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:28.113648+0000) 2022-09-10T12:08:28.910 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:28.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:28.641167+0000) 2022-09-10T12:08:29.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:08:28.909+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:08:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:29.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:29.113850+0000) 2022-09-10T12:08:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:29 smithi104 ceph-mon[97044]: pgmap v14695: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:29.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:29.641385+0000) 2022-09-10T12:08:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:29 smithi110 ceph-mon[97633]: pgmap v14695: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:30.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:30.114053+0000) 2022-09-10T12:08:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:30.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:30.641584+0000) 2022-09-10T12:08:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:31.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:31.114294+0000) 2022-09-10T12:08:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:31 smithi104 ceph-mon[97044]: pgmap v14696: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:31.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:31.641800+0000) 2022-09-10T12:08:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:31 smithi110 ceph-mon[97633]: pgmap v14696: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:32.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:32.114560+0000) 2022-09-10T12:08:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:32.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:32.642003+0000) 2022-09-10T12:08:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:33.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:33.114725+0000) 2022-09-10T12:08:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:33 smithi104 ceph-mon[97044]: pgmap v14697: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:33.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:33.642223+0000) 2022-09-10T12:08:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:33 smithi110 ceph-mon[97633]: pgmap v14697: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:34.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:34.114995+0000) 2022-09-10T12:08:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:34.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:34.642410+0000) 2022-09-10T12:08:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:35.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:35.115226+0000) 2022-09-10T12:08:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:35 smithi110 ceph-mon[97633]: pgmap v14698: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:35 smithi104 ceph-mon[97044]: pgmap v14698: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:35.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:35.642625+0000) 2022-09-10T12:08:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:36.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:36.115446+0000) 2022-09-10T12:08:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:36.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:36.642857+0000) 2022-09-10T12:08:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:37.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:37.115578+0000) 2022-09-10T12:08:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:37 smithi104 ceph-mon[97044]: pgmap v14699: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:37.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:37.643075+0000) 2022-09-10T12:08:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:37 smithi110 ceph-mon[97633]: pgmap v14699: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:38.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:38.115771+0000) 2022-09-10T12:08:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:38.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:38.643312+0000) 2022-09-10T12:08:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:39.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:39.115940+0000) 2022-09-10T12:08:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:39 smithi104 ceph-mon[97044]: pgmap v14700: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:39.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:39.643496+0000) 2022-09-10T12:08:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:39 smithi110 ceph-mon[97633]: pgmap v14700: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:40.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:40.116187+0000) 2022-09-10T12:08:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:40.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:40.643646+0000) 2022-09-10T12:08:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:41.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:41.116460+0000) 2022-09-10T12:08:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:41 smithi104 ceph-mon[97044]: pgmap v14701: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:41.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:41.643889+0000) 2022-09-10T12:08:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:41 smithi110 ceph-mon[97633]: pgmap v14701: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:42.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:42.116695+0000) 2022-09-10T12:08:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:42.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:42.644115+0000) 2022-09-10T12:08:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:43.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:43.116908+0000) 2022-09-10T12:08:43.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:43.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:43.644315+0000) 2022-09-10T12:08:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:43 smithi104 ceph-mon[97044]: pgmap v14702: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:43 smithi110 ceph-mon[97633]: pgmap v14702: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:44.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:08:43.925+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:08:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:44.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:44.117200+0000) 2022-09-10T12:08:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:44.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:44.644599+0000) 2022-09-10T12:08:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:45.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:45.117394+0000) 2022-09-10T12:08:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:45 smithi110 ceph-mon[97633]: pgmap v14703: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:45 smithi104 ceph-mon[97044]: pgmap v14703: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:45.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:45.644780+0000) 2022-09-10T12:08:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:46.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:46.117588+0000) 2022-09-10T12:08:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:08:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:08:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:08:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:08:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:08:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:46.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:46.645031+0000) 2022-09-10T12:08:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:47.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:47.117804+0000) 2022-09-10T12:08:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:47 smithi110 ceph-mon[97633]: pgmap v14704: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:47 smithi104 ceph-mon[97044]: pgmap v14704: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:47.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:47.645262+0000) 2022-09-10T12:08:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:48.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:48.117990+0000) 2022-09-10T12:08:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:48.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:48.645492+0000) 2022-09-10T12:08:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:49.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:49.118203+0000) 2022-09-10T12:08:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:49 smithi104 ceph-mon[97044]: pgmap v14705: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:49.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:49.645711+0000) 2022-09-10T12:08:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:49 smithi110 ceph-mon[97633]: pgmap v14705: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:50.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:50.118431+0000) 2022-09-10T12:08:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:50.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:50.645961+0000) 2022-09-10T12:08:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:51.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:51.118654+0000) 2022-09-10T12:08:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:51 smithi104 ceph-mon[97044]: pgmap v14706: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:51.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:51.646143+0000) 2022-09-10T12:08:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:51 smithi110 ceph-mon[97633]: pgmap v14706: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:52.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:52.118884+0000) 2022-09-10T12:08:53.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:52.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:52.646309+0000) 2022-09-10T12:08:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:53.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:53.119157+0000) 2022-09-10T12:08:54.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:53 smithi104 ceph-mon[97044]: pgmap v14707: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:53.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:53.646519+0000) 2022-09-10T12:08:54.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:53 smithi110 ceph-mon[97633]: pgmap v14707: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:54.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:54.119347+0000) 2022-09-10T12:08:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:54.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:54.646756+0000) 2022-09-10T12:08:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:55.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:55.119519+0000) 2022-09-10T12:08:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:55 smithi110 ceph-mon[97633]: pgmap v14708: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:55 smithi104 ceph-mon[97044]: pgmap v14708: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:08:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:55.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:55.646986+0000) 2022-09-10T12:08:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:56.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:56.119782+0000) 2022-09-10T12:08:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:56.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:56.647233+0000) 2022-09-10T12:08:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:57.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:57.120009+0000) 2022-09-10T12:08:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:57 smithi104 ceph-mon[97044]: pgmap v14709: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:57.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:57.647452+0000) 2022-09-10T12:08:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:57 smithi110 ceph-mon[97633]: pgmap v14709: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:08:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:58.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:58.120265+0000) 2022-09-10T12:08:58.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:58.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:58.647652+0000) 2022-09-10T12:08:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:08:59.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:59.120439+0000) 2022-09-10T12:08:59.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:08:58.932+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:09:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:08:59 smithi104 ceph-mon[97044]: pgmap v14710: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:08:59.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:08:59.647914+0000) 2022-09-10T12:09:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:08:59 smithi110 ceph-mon[97633]: pgmap v14710: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:00.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:00.120602+0000) 2022-09-10T12:09:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:00.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:00.648192+0000) 2022-09-10T12:09:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:01.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:01.120768+0000) 2022-09-10T12:09:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:01 smithi104 ceph-mon[97044]: pgmap v14711: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:01.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:01.648311+0000) 2022-09-10T12:09:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:01 smithi110 ceph-mon[97633]: pgmap v14711: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:02.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:02.120989+0000) 2022-09-10T12:09:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:02.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:02.648467+0000) 2022-09-10T12:09:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:03.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:03.121200+0000) 2022-09-10T12:09:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:03 smithi104 ceph-mon[97044]: pgmap v14712: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:03.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:03.648593+0000) 2022-09-10T12:09:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:03 smithi110 ceph-mon[97633]: pgmap v14712: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:04.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:04.121405+0000) 2022-09-10T12:09:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:04.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:04.648812+0000) 2022-09-10T12:09:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:05.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:05.121625+0000) 2022-09-10T12:09:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:05 smithi110 ceph-mon[97633]: pgmap v14713: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:09:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:05 smithi104 ceph-mon[97044]: pgmap v14713: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:09:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:05.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:05.649050+0000) 2022-09-10T12:09:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:06.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:06.121840+0000) 2022-09-10T12:09:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:06.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:06.649281+0000) 2022-09-10T12:09:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:07.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:07.122077+0000) 2022-09-10T12:09:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:07 smithi104 ceph-mon[97044]: pgmap v14714: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:09:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:07.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:07.649442+0000) 2022-09-10T12:09:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:07 smithi110 ceph-mon[97633]: pgmap v14714: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:09:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:08.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:08.122393+0000) 2022-09-10T12:09:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:08 smithi104 ceph-mon[97044]: osdmap e1000: 12 total, 12 up, 12 in 2022-09-10T12:09:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:08.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:08.649588+0000) 2022-09-10T12:09:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:08 smithi110 ceph-mon[97633]: osdmap e1000: 12 total, 12 up, 12 in 2022-09-10T12:09:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:09.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:09.122636+0000) 2022-09-10T12:09:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:09 smithi104 ceph-mon[97044]: pgmap v14716: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:09:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:09.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:09.649767+0000) 2022-09-10T12:09:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:09 smithi110 ceph-mon[97633]: pgmap v14716: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:09:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:10.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:10.122857+0000) 2022-09-10T12:09:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:10.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:10.649992+0000) 2022-09-10T12:09:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:11.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:11.123151+0000) 2022-09-10T12:09:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:11 smithi104 ceph-mon[97044]: pgmap v14717: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:09:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:11.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:11.650229+0000) 2022-09-10T12:09:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:11 smithi110 ceph-mon[97633]: pgmap v14717: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:09:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:12.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:12.123390+0000) 2022-09-10T12:09:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:12.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:12.650408+0000) 2022-09-10T12:09:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:13.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:13.123646+0000) 2022-09-10T12:09:13.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:13.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:13.650530+0000) 2022-09-10T12:09:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:13 smithi104 ceph-mon[97044]: pgmap v14718: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:09:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:13 smithi110 ceph-mon[97633]: pgmap v14718: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:09:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:14.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:14.123888+0000) 2022-09-10T12:09:14.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:09:13.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:09:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:14.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:14.650661+0000) 2022-09-10T12:09:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:15.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:15.124126+0000) 2022-09-10T12:09:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:15 smithi110 ceph-mon[97633]: osdmap e1001: 12 total, 12 up, 12 in 2022-09-10T12:09:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:15 smithi110 ceph-mon[97633]: pgmap v14720: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:09:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:15 smithi104 ceph-mon[97044]: osdmap e1001: 12 total, 12 up, 12 in 2022-09-10T12:09:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:15 smithi104 ceph-mon[97044]: pgmap v14720: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:09:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:15.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:15.650895+0000) 2022-09-10T12:09:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:16.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:16.124333+0000) 2022-09-10T12:09:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:16.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:16.651129+0000) 2022-09-10T12:09:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:17.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:17.124547+0000) 2022-09-10T12:09:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:17 smithi104 ceph-mon[97044]: pgmap v14721: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:09:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:17.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:17.651317+0000) 2022-09-10T12:09:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:17 smithi110 ceph-mon[97633]: pgmap v14721: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:09:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:18.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:18.124762+0000) 2022-09-10T12:09:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:18.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:18.651531+0000) 2022-09-10T12:09:19.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:19.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:19.125012+0000) 2022-09-10T12:09:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:19 smithi104 ceph-mon[97044]: pgmap v14722: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:09:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:19.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:19.651768+0000) 2022-09-10T12:09:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:19 smithi110 ceph-mon[97633]: pgmap v14722: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:09:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:20.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:20.125190+0000) 2022-09-10T12:09:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:20.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:20.651966+0000) 2022-09-10T12:09:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:21.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:21.125398+0000) 2022-09-10T12:09:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:21 smithi104 ceph-mon[97044]: pgmap v14723: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:09:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:21.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:21.652179+0000) 2022-09-10T12:09:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:21 smithi110 ceph-mon[97633]: pgmap v14723: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:09:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:22.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:22.125588+0000) 2022-09-10T12:09:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:22.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:22.652353+0000) 2022-09-10T12:09:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:23.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:23.125819+0000) 2022-09-10T12:09:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:23 smithi104 ceph-mon[97044]: pgmap v14724: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:09:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:23.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:23.652551+0000) 2022-09-10T12:09:24.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:23 smithi110 ceph-mon[97633]: pgmap v14724: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:09:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:24.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:24.126032+0000) 2022-09-10T12:09:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:24.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:24.652782+0000) 2022-09-10T12:09:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:25.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:25.126270+0000) 2022-09-10T12:09:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:25 smithi110 ceph-mon[97633]: pgmap v14725: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:09:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:25 smithi104 ceph-mon[97044]: pgmap v14725: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:09:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:25.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:25.653040+0000) 2022-09-10T12:09:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:26.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:26.126565+0000) 2022-09-10T12:09:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:26.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:26.653312+0000) 2022-09-10T12:09:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:27.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:27.126791+0000) 2022-09-10T12:09:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:27 smithi104 ceph-mon[97044]: pgmap v14726: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:27.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:27.653511+0000) 2022-09-10T12:09:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:27 smithi110 ceph-mon[97633]: pgmap v14726: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:28.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:28.127027+0000) 2022-09-10T12:09:28.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:28.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:28.653734+0000) 2022-09-10T12:09:29.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:09:28.948+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:09:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:29.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:29.127228+0000) 2022-09-10T12:09:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:29 smithi104 ceph-mon[97044]: pgmap v14727: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:29.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:29.653993+0000) 2022-09-10T12:09:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:29 smithi110 ceph-mon[97633]: pgmap v14727: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:30.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:30.127413+0000) 2022-09-10T12:09:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:30.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:30.654245+0000) 2022-09-10T12:09:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:31.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:31.127652+0000) 2022-09-10T12:09:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:31 smithi104 ceph-mon[97044]: pgmap v14728: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:31.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:31.654441+0000) 2022-09-10T12:09:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:31 smithi110 ceph-mon[97633]: pgmap v14728: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:32.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:32.127850+0000) 2022-09-10T12:09:33.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:32.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:32.654572+0000) 2022-09-10T12:09:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:33.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:33.128084+0000) 2022-09-10T12:09:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:33 smithi104 ceph-mon[97044]: pgmap v14729: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:33.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:33.654734+0000) 2022-09-10T12:09:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:33 smithi110 ceph-mon[97633]: pgmap v14729: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:34.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:34.128399+0000) 2022-09-10T12:09:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:34.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:34.654945+0000) 2022-09-10T12:09:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:35.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:35.128691+0000) 2022-09-10T12:09:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:35 smithi110 ceph-mon[97633]: pgmap v14730: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:35 smithi104 ceph-mon[97044]: pgmap v14730: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:35.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:35.655216+0000) 2022-09-10T12:09:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:36.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:36.128951+0000) 2022-09-10T12:09:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:36.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:36.655459+0000) 2022-09-10T12:09:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:37.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:37.129232+0000) 2022-09-10T12:09:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:37 smithi104 ceph-mon[97044]: pgmap v14731: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:37.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:37.655624+0000) 2022-09-10T12:09:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:37 smithi110 ceph-mon[97633]: pgmap v14731: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:38.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:38.129436+0000) 2022-09-10T12:09:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:38.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:38.655863+0000) 2022-09-10T12:09:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:39.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:39.129664+0000) 2022-09-10T12:09:40.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:39 smithi104 ceph-mon[97044]: pgmap v14732: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:39.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:39.656055+0000) 2022-09-10T12:09:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:39 smithi110 ceph-mon[97633]: pgmap v14732: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:40.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:40.129882+0000) 2022-09-10T12:09:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:40.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:40.656271+0000) 2022-09-10T12:09:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:41.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:41.130052+0000) 2022-09-10T12:09:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:41 smithi104 ceph-mon[97044]: pgmap v14733: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:41.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:41.656490+0000) 2022-09-10T12:09:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:41 smithi110 ceph-mon[97633]: pgmap v14733: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:42.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:42.130279+0000) 2022-09-10T12:09:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:42.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:42.656732+0000) 2022-09-10T12:09:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:43.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:43.130516+0000) 2022-09-10T12:09:43.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:43.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:43.656979+0000) 2022-09-10T12:09:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:43 smithi104 ceph-mon[97044]: pgmap v14734: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:43 smithi110 ceph-mon[97633]: pgmap v14734: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:44.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:44.130752+0000) 2022-09-10T12:09:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:09:43.962+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:09:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:44.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:44.657235+0000) 2022-09-10T12:09:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:45.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:45.131021+0000) 2022-09-10T12:09:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:45 smithi110 ceph-mon[97633]: pgmap v14735: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:45 smithi104 ceph-mon[97044]: pgmap v14735: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:45.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:45.657438+0000) 2022-09-10T12:09:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:46.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:46.131290+0000) 2022-09-10T12:09:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:09:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:09:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:09:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:09:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:09:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:46.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:46.657649+0000) 2022-09-10T12:09:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:47.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:47.131552+0000) 2022-09-10T12:09:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:47 smithi110 ceph-mon[97633]: pgmap v14736: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:47 smithi104 ceph-mon[97044]: pgmap v14736: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:47.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:47.657783+0000) 2022-09-10T12:09:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:48.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:48.131830+0000) 2022-09-10T12:09:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:48.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:48.657916+0000) 2022-09-10T12:09:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:49.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:49.132030+0000) 2022-09-10T12:09:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:49 smithi104 ceph-mon[97044]: pgmap v14737: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:49.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:49.658136+0000) 2022-09-10T12:09:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:49 smithi110 ceph-mon[97633]: pgmap v14737: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:50.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:50.132217+0000) 2022-09-10T12:09:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:50.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:50.658345+0000) 2022-09-10T12:09:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:51.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:51.132372+0000) 2022-09-10T12:09:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:51 smithi104 ceph-mon[97044]: pgmap v14738: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:51.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:51.658555+0000) 2022-09-10T12:09:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:51 smithi110 ceph-mon[97633]: pgmap v14738: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:52.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:52.132552+0000) 2022-09-10T12:09:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:52.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:52.658732+0000) 2022-09-10T12:09:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:53.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:53.132790+0000) 2022-09-10T12:09:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:53 smithi104 ceph-mon[97044]: pgmap v14739: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:53.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:53.658937+0000) 2022-09-10T12:09:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:53 smithi110 ceph-mon[97633]: pgmap v14739: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:54.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:54.133045+0000) 2022-09-10T12:09:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:54.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:54.659117+0000) 2022-09-10T12:09:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:55.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:55.133191+0000) 2022-09-10T12:09:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:55 smithi110 ceph-mon[97633]: pgmap v14740: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:55 smithi104 ceph-mon[97044]: pgmap v14740: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:09:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:55.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:55.659259+0000) 2022-09-10T12:09:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:56.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:56.133347+0000) 2022-09-10T12:09:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:56.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:56.659480+0000) 2022-09-10T12:09:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:57.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:57.133548+0000) 2022-09-10T12:09:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:57 smithi104 ceph-mon[97044]: pgmap v14741: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:57.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:57.659620+0000) 2022-09-10T12:09:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:57 smithi110 ceph-mon[97633]: pgmap v14741: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:09:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:58.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:58.133729+0000) 2022-09-10T12:09:58.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:58.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:58.659824+0000) 2022-09-10T12:09:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:09:59.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:59.133987+0000) 2022-09-10T12:09:59.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:09:58.975+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:10:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:09:59 smithi104 ceph-mon[97044]: pgmap v14742: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:09:59.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:09:59.660071+0000) 2022-09-10T12:10:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:09:59 smithi110 ceph-mon[97633]: pgmap v14742: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:00.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:00.134249+0000) 2022-09-10T12:10:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:10:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:00.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:00.660317+0000) 2022-09-10T12:10:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:10:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:01.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:01.134451+0000) 2022-09-10T12:10:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:01 smithi104 ceph-mon[97044]: pgmap v14743: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:01.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:01.660472+0000) 2022-09-10T12:10:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:01 smithi110 ceph-mon[97633]: pgmap v14743: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:02.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:02.134668+0000) 2022-09-10T12:10:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:02.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:02.660644+0000) 2022-09-10T12:10:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:03.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:03.134898+0000) 2022-09-10T12:10:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:03 smithi104 ceph-mon[97044]: pgmap v14744: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:03.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:03.660850+0000) 2022-09-10T12:10:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:03 smithi110 ceph-mon[97633]: pgmap v14744: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:04.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:04.135168+0000) 2022-09-10T12:10:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:04.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:04.661120+0000) 2022-09-10T12:10:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:05.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:05.135331+0000) 2022-09-10T12:10:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:05 smithi110 ceph-mon[97633]: pgmap v14745: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:10:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:05 smithi104 ceph-mon[97044]: pgmap v14745: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:10:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:05.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:05.661326+0000) 2022-09-10T12:10:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:06.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:06.135545+0000) 2022-09-10T12:10:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:06.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:06.661477+0000) 2022-09-10T12:10:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:07.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:07.135732+0000) 2022-09-10T12:10:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:07 smithi104 ceph-mon[97044]: pgmap v14746: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:10:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:07.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:07.661657+0000) 2022-09-10T12:10:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:07 smithi110 ceph-mon[97633]: pgmap v14746: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:10:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:08.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:08.135968+0000) 2022-09-10T12:10:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:08 smithi104 ceph-mon[97044]: osdmap e1002: 12 total, 12 up, 12 in 2022-09-10T12:10:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:08.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:08.661881+0000) 2022-09-10T12:10:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:08 smithi110 ceph-mon[97633]: osdmap e1002: 12 total, 12 up, 12 in 2022-09-10T12:10:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:09.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:09.136202+0000) 2022-09-10T12:10:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:09 smithi104 ceph-mon[97044]: pgmap v14748: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:10:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:09.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:09.662069+0000) 2022-09-10T12:10:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:09 smithi110 ceph-mon[97633]: pgmap v14748: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:10:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:10.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:10.136411+0000) 2022-09-10T12:10:11.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:10.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:10.662340+0000) 2022-09-10T12:10:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:11.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:11.136586+0000) 2022-09-10T12:10:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:11 smithi104 ceph-mon[97044]: pgmap v14749: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:10:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:11.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:11.662493+0000) 2022-09-10T12:10:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:11 smithi110 ceph-mon[97633]: pgmap v14749: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:10:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:12.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:12.136847+0000) 2022-09-10T12:10:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:12.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:12.662702+0000) 2022-09-10T12:10:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:13.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:13.137055+0000) 2022-09-10T12:10:13.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:13.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:13.662933+0000) 2022-09-10T12:10:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:13 smithi104 ceph-mon[97044]: pgmap v14750: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:10:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:13 smithi110 ceph-mon[97633]: pgmap v14750: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:10:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:14.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:14.137277+0000) 2022-09-10T12:10:14.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:10:13.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:10:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:14.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:14.663264+0000) 2022-09-10T12:10:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:15.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:15.137461+0000) 2022-09-10T12:10:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:15 smithi110 ceph-mon[97633]: osdmap e1003: 12 total, 12 up, 12 in 2022-09-10T12:10:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:15 smithi110 ceph-mon[97633]: pgmap v14752: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:10:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:15 smithi104 ceph-mon[97044]: osdmap e1003: 12 total, 12 up, 12 in 2022-09-10T12:10:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:15 smithi104 ceph-mon[97044]: pgmap v14752: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:10:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:15.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:15.663476+0000) 2022-09-10T12:10:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:16.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:16.137708+0000) 2022-09-10T12:10:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:16.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:16.663701+0000) 2022-09-10T12:10:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:17.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:17.137947+0000) 2022-09-10T12:10:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:17 smithi104 ceph-mon[97044]: pgmap v14753: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:10:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:17.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:17.663973+0000) 2022-09-10T12:10:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:17 smithi110 ceph-mon[97633]: pgmap v14753: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:10:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:18.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:18.138168+0000) 2022-09-10T12:10:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:18.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:18.664168+0000) 2022-09-10T12:10:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:19.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:19.138332+0000) 2022-09-10T12:10:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:19 smithi104 ceph-mon[97044]: pgmap v14754: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:10:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:19.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:19.664439+0000) 2022-09-10T12:10:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:19 smithi110 ceph-mon[97633]: pgmap v14754: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:10:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:20.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:20.138527+0000) 2022-09-10T12:10:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:20.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:20.664664+0000) 2022-09-10T12:10:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:21.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:21.138693+0000) 2022-09-10T12:10:22.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:21 smithi104 ceph-mon[97044]: pgmap v14755: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:10:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:21.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:21.664922+0000) 2022-09-10T12:10:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:21 smithi110 ceph-mon[97633]: pgmap v14755: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:10:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:22.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:22.138945+0000) 2022-09-10T12:10:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:22.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:22.665170+0000) 2022-09-10T12:10:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:23.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:23.139203+0000) 2022-09-10T12:10:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:23 smithi104 ceph-mon[97044]: pgmap v14756: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:10:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:23.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:23.665370+0000) 2022-09-10T12:10:24.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:23 smithi110 ceph-mon[97633]: pgmap v14756: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:10:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:24.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:24.139333+0000) 2022-09-10T12:10:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:24.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:24.665618+0000) 2022-09-10T12:10:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:25.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:25.139530+0000) 2022-09-10T12:10:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:25 smithi110 ceph-mon[97633]: pgmap v14757: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:10:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:25 smithi104 ceph-mon[97044]: pgmap v14757: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:10:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:25.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:25.665851+0000) 2022-09-10T12:10:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:26.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:26.139716+0000) 2022-09-10T12:10:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:26.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:26.666125+0000) 2022-09-10T12:10:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:27.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:27.139917+0000) 2022-09-10T12:10:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:27 smithi104 ceph-mon[97044]: pgmap v14758: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:27.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:27.666397+0000) 2022-09-10T12:10:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:27 smithi110 ceph-mon[97633]: pgmap v14758: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:28.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:28.140192+0000) 2022-09-10T12:10:29.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:28.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:28.666615+0000) 2022-09-10T12:10:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:29.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:29.140383+0000) 2022-09-10T12:10:29.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:10:29.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:10:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:29 smithi104 ceph-mon[97044]: pgmap v14759: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:29.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:29.666873+0000) 2022-09-10T12:10:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:29 smithi110 ceph-mon[97633]: pgmap v14759: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:30.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:30.140559+0000) 2022-09-10T12:10:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:30.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:30.667157+0000) 2022-09-10T12:10:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:31.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:31.140782+0000) 2022-09-10T12:10:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:31 smithi104 ceph-mon[97044]: pgmap v14760: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:31.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:31.667319+0000) 2022-09-10T12:10:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:31 smithi110 ceph-mon[97633]: pgmap v14760: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:32.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:32.141018+0000) 2022-09-10T12:10:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:32.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:32.667541+0000) 2022-09-10T12:10:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:33.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:33.141290+0000) 2022-09-10T12:10:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:33 smithi104 ceph-mon[97044]: pgmap v14761: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:33.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:33.667787+0000) 2022-09-10T12:10:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:33 smithi110 ceph-mon[97633]: pgmap v14761: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:34.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:34.141508+0000) 2022-09-10T12:10:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:34.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:34.667946+0000) 2022-09-10T12:10:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:35.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:35.141770+0000) 2022-09-10T12:10:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:35 smithi110 ceph-mon[97633]: pgmap v14762: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:35 smithi104 ceph-mon[97044]: pgmap v14762: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:35.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:35.668150+0000) 2022-09-10T12:10:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:36.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:36.141991+0000) 2022-09-10T12:10:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:36.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:36.668317+0000) 2022-09-10T12:10:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:37.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:37.142186+0000) 2022-09-10T12:10:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:37 smithi104 ceph-mon[97044]: pgmap v14763: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:37.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:37.668511+0000) 2022-09-10T12:10:38.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:37 smithi110 ceph-mon[97633]: pgmap v14763: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:38.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:38.142416+0000) 2022-09-10T12:10:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:38.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:38.668706+0000) 2022-09-10T12:10:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:39.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:39.142595+0000) 2022-09-10T12:10:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:39 smithi104 ceph-mon[97044]: pgmap v14764: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:39.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:39.668912+0000) 2022-09-10T12:10:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:39 smithi110 ceph-mon[97633]: pgmap v14764: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:40.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:40.142821+0000) 2022-09-10T12:10:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:40.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:40.669166+0000) 2022-09-10T12:10:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:41.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:41.143012+0000) 2022-09-10T12:10:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:41 smithi104 ceph-mon[97044]: pgmap v14765: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:41.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:41.669384+0000) 2022-09-10T12:10:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:41 smithi110 ceph-mon[97633]: pgmap v14765: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:42.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:42.143235+0000) 2022-09-10T12:10:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:42.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:42.669565+0000) 2022-09-10T12:10:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:43.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:43.143452+0000) 2022-09-10T12:10:44.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:43.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:43.669783+0000) 2022-09-10T12:10:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:43 smithi104 ceph-mon[97044]: pgmap v14766: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:10:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:43 smithi110 ceph-mon[97633]: pgmap v14766: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:10:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:44.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:44.143649+0000) 2022-09-10T12:10:44.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:10:44.019+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:10:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:44.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:44.670025+0000) 2022-09-10T12:10:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:45.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:45.143833+0000) 2022-09-10T12:10:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:45 smithi104 ceph-mon[97044]: pgmap v14767: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:45 smithi110 ceph-mon[97633]: pgmap v14767: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:45.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:45.670272+0000) 2022-09-10T12:10:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:46.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:46.144064+0000) 2022-09-10T12:10:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:10:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:10:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:10:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:10:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:10:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:46.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:46.670472+0000) 2022-09-10T12:10:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:47.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:47.144285+0000) 2022-09-10T12:10:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:47 smithi104 ceph-mon[97044]: pgmap v14768: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:10:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:47 smithi110 ceph-mon[97633]: pgmap v14768: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:10:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:47.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:47.670753+0000) 2022-09-10T12:10:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:48.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:48.144461+0000) 2022-09-10T12:10:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:48.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:48.670980+0000) 2022-09-10T12:10:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:49.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:49.144604+0000) 2022-09-10T12:10:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:49 smithi104 ceph-mon[97044]: pgmap v14769: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:10:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:49.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:49.671244+0000) 2022-09-10T12:10:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:49 smithi110 ceph-mon[97633]: pgmap v14769: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:10:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:50.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:50.144846+0000) 2022-09-10T12:10:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:50.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:50.671479+0000) 2022-09-10T12:10:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:51.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:51.145145+0000) 2022-09-10T12:10:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:51 smithi104 ceph-mon[97044]: pgmap v14770: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:51.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:51.671739+0000) 2022-09-10T12:10:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:51 smithi110 ceph-mon[97633]: pgmap v14770: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:52.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:52.145354+0000) 2022-09-10T12:10:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:52.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:52.672006+0000) 2022-09-10T12:10:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:53.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:53.145629+0000) 2022-09-10T12:10:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:53 smithi104 ceph-mon[97044]: pgmap v14771: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:53.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:53.672220+0000) 2022-09-10T12:10:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:53 smithi110 ceph-mon[97633]: pgmap v14771: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:54.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:54.145820+0000) 2022-09-10T12:10:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:54.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:54.672379+0000) 2022-09-10T12:10:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:55.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:55.145973+0000) 2022-09-10T12:10:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:55 smithi110 ceph-mon[97633]: pgmap v14772: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:55 smithi104 ceph-mon[97044]: pgmap v14772: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:10:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:55.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:55.672549+0000) 2022-09-10T12:10:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:56.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:56.146236+0000) 2022-09-10T12:10:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:56.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:56.672756+0000) 2022-09-10T12:10:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:57.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:57.146462+0000) 2022-09-10T12:10:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:57 smithi104 ceph-mon[97044]: pgmap v14773: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:57.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:57.672982+0000) 2022-09-10T12:10:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:57 smithi110 ceph-mon[97633]: pgmap v14773: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:10:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:58.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:58.146713+0000) 2022-09-10T12:10:59.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:58.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:58.673224+0000) 2022-09-10T12:10:59.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:10:59.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:10:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:10:59.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:59.146941+0000) 2022-09-10T12:11:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:10:59 smithi104 ceph-mon[97044]: pgmap v14774: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:10:59.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:10:59.673490+0000) 2022-09-10T12:11:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:10:59 smithi110 ceph-mon[97633]: pgmap v14774: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:00.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:00.147077+0000) 2022-09-10T12:11:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:00.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:00.673670+0000) 2022-09-10T12:11:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:01.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:01.147349+0000) 2022-09-10T12:11:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:01 smithi104 ceph-mon[97044]: pgmap v14775: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:01.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:01.673889+0000) 2022-09-10T12:11:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:01 smithi110 ceph-mon[97633]: pgmap v14775: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:02.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:02.147547+0000) 2022-09-10T12:11:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:02.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:02.674191+0000) 2022-09-10T12:11:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:03.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:03.147777+0000) 2022-09-10T12:11:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:03 smithi104 ceph-mon[97044]: pgmap v14776: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:03.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:03.674385+0000) 2022-09-10T12:11:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:03 smithi110 ceph-mon[97633]: pgmap v14776: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:04.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:04.147958+0000) 2022-09-10T12:11:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:04.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:04.674552+0000) 2022-09-10T12:11:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:05.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:05.148156+0000) 2022-09-10T12:11:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:05 smithi104 ceph-mon[97044]: pgmap v14777: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:11:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:05 smithi110 ceph-mon[97633]: pgmap v14777: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:11:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:05.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:05.674750+0000) 2022-09-10T12:11:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:06.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:06.148372+0000) 2022-09-10T12:11:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:06.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:06.674973+0000) 2022-09-10T12:11:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:07.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:07.148631+0000) 2022-09-10T12:11:08.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:07 smithi104 ceph-mon[97044]: pgmap v14778: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:11:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:07.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:07.675244+0000) 2022-09-10T12:11:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:07 smithi110 ceph-mon[97633]: pgmap v14778: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:11:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:08.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:08.148788+0000) 2022-09-10T12:11:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:08 smithi104 ceph-mon[97044]: osdmap e1004: 12 total, 12 up, 12 in 2022-09-10T12:11:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:08.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:08.675524+0000) 2022-09-10T12:11:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:08 smithi110 ceph-mon[97633]: osdmap e1004: 12 total, 12 up, 12 in 2022-09-10T12:11:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:09.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:09.149001+0000) 2022-09-10T12:11:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:09 smithi104 ceph-mon[97044]: pgmap v14780: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:11:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:09.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:09.675772+0000) 2022-09-10T12:11:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:09 smithi110 ceph-mon[97633]: pgmap v14780: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:11:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:10.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:10.149247+0000) 2022-09-10T12:11:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:10.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:10.676004+0000) 2022-09-10T12:11:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:11.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:11.149470+0000) 2022-09-10T12:11:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:11 smithi104 ceph-mon[97044]: pgmap v14781: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:11:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:11.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:11.676243+0000) 2022-09-10T12:11:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:11 smithi110 ceph-mon[97633]: pgmap v14781: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:11:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:12.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:12.149655+0000) 2022-09-10T12:11:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:12.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:12.676463+0000) 2022-09-10T12:11:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:13.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:13.149823+0000) 2022-09-10T12:11:14.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:13.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:13.676715+0000) 2022-09-10T12:11:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:13 smithi104 ceph-mon[97044]: pgmap v14782: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:11:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:13 smithi110 ceph-mon[97633]: pgmap v14782: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:11:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:14.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:14.150016+0000) 2022-09-10T12:11:14.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:11:14.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:11:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:14.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:14.676946+0000) 2022-09-10T12:11:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:15.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:15.150221+0000) 2022-09-10T12:11:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:15 smithi104 ceph-mon[97044]: osdmap e1005: 12 total, 12 up, 12 in 2022-09-10T12:11:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:15 smithi104 ceph-mon[97044]: pgmap v14784: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:11:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:15 smithi110 ceph-mon[97633]: osdmap e1005: 12 total, 12 up, 12 in 2022-09-10T12:11:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:15 smithi110 ceph-mon[97633]: pgmap v14784: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:11:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:15.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:15.677167+0000) 2022-09-10T12:11:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:16.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:16.150453+0000) 2022-09-10T12:11:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:16.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:16.677316+0000) 2022-09-10T12:11:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:17.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:17.150604+0000) 2022-09-10T12:11:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:17 smithi104 ceph-mon[97044]: pgmap v14785: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:11:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:17.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:17.677504+0000) 2022-09-10T12:11:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:17 smithi110 ceph-mon[97633]: pgmap v14785: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:11:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:18.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:18.150838+0000) 2022-09-10T12:11:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:18.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:18.677682+0000) 2022-09-10T12:11:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:19.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:19.151123+0000) 2022-09-10T12:11:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:19 smithi104 ceph-mon[97044]: pgmap v14786: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:11:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:19.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:19.677930+0000) 2022-09-10T12:11:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:19 smithi110 ceph-mon[97633]: pgmap v14786: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:11:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:20.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:20.151385+0000) 2022-09-10T12:11:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:20.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:20.678200+0000) 2022-09-10T12:11:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:21.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:21.151604+0000) 2022-09-10T12:11:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:21 smithi104 ceph-mon[97044]: pgmap v14787: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:11:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:21.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:21.678380+0000) 2022-09-10T12:11:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:21 smithi110 ceph-mon[97633]: pgmap v14787: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:11:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:22.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:22.151832+0000) 2022-09-10T12:11:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:22.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:22.678550+0000) 2022-09-10T12:11:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:23.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:23.152028+0000) 2022-09-10T12:11:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:23 smithi104 ceph-mon[97044]: pgmap v14788: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:11:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:23.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:23.678786+0000) 2022-09-10T12:11:24.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:23 smithi110 ceph-mon[97633]: pgmap v14788: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:11:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:24.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:24.152256+0000) 2022-09-10T12:11:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:24.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:24.679071+0000) 2022-09-10T12:11:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:25.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:25.152510+0000) 2022-09-10T12:11:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:25 smithi104 ceph-mon[97044]: pgmap v14789: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:11:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:25 smithi110 ceph-mon[97633]: pgmap v14789: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:11:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:25.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:25.679325+0000) 2022-09-10T12:11:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:26.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:26.152672+0000) 2022-09-10T12:11:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:26.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:26.679523+0000) 2022-09-10T12:11:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:27.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:27.152917+0000) 2022-09-10T12:11:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:27 smithi104 ceph-mon[97044]: pgmap v14790: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:27.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:27.679711+0000) 2022-09-10T12:11:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:27 smithi110 ceph-mon[97633]: pgmap v14790: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:28.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:28.153149+0000) 2022-09-10T12:11:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:28.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:28.679955+0000) 2022-09-10T12:11:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:29.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:29.153359+0000) 2022-09-10T12:11:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:11:29.052+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:11:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:29 smithi104 ceph-mon[97044]: pgmap v14791: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:29.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:29.680166+0000) 2022-09-10T12:11:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:29 smithi110 ceph-mon[97633]: pgmap v14791: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:30.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:30.153561+0000) 2022-09-10T12:11:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:30.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:30.680370+0000) 2022-09-10T12:11:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:31.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:31.153696+0000) 2022-09-10T12:11:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:31 smithi104 ceph-mon[97044]: pgmap v14792: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:31.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:31.680575+0000) 2022-09-10T12:11:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:31 smithi110 ceph-mon[97633]: pgmap v14792: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:32.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:32.153888+0000) 2022-09-10T12:11:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:32.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:32.680752+0000) 2022-09-10T12:11:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:33.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:33.154083+0000) 2022-09-10T12:11:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:33 smithi104 ceph-mon[97044]: pgmap v14793: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:33.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:33.680929+0000) 2022-09-10T12:11:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:33 smithi110 ceph-mon[97633]: pgmap v14793: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:34.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:34.154293+0000) 2022-09-10T12:11:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:34.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:34.681180+0000) 2022-09-10T12:11:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:35.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:35.154461+0000) 2022-09-10T12:11:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:35 smithi104 ceph-mon[97044]: pgmap v14794: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:35 smithi110 ceph-mon[97633]: pgmap v14794: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:35.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:35.681314+0000) 2022-09-10T12:11:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:36.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:36.154649+0000) 2022-09-10T12:11:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:36.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:36.681550+0000) 2022-09-10T12:11:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:37.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:37.154926+0000) 2022-09-10T12:11:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:37 smithi104 ceph-mon[97044]: pgmap v14795: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:37.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:37.681756+0000) 2022-09-10T12:11:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:37 smithi110 ceph-mon[97633]: pgmap v14795: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:38.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:38.155155+0000) 2022-09-10T12:11:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:38.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:38.682016+0000) 2022-09-10T12:11:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:39.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:39.155340+0000) 2022-09-10T12:11:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:39 smithi104 ceph-mon[97044]: pgmap v14796: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:39.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:39.682270+0000) 2022-09-10T12:11:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:39 smithi110 ceph-mon[97633]: pgmap v14796: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:40.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:40.155576+0000) 2022-09-10T12:11:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:40.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:40.682502+0000) 2022-09-10T12:11:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:41.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:41.155787+0000) 2022-09-10T12:11:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:41 smithi104 ceph-mon[97044]: pgmap v14797: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:41.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:41.682693+0000) 2022-09-10T12:11:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:41 smithi110 ceph-mon[97633]: pgmap v14797: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:42.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:42.156010+0000) 2022-09-10T12:11:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:42.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:42.682914+0000) 2022-09-10T12:11:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:43.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:43.156311+0000) 2022-09-10T12:11:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:43 smithi104 ceph-mon[97044]: pgmap v14798: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:43.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:43.683206+0000) 2022-09-10T12:11:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:43 smithi110 ceph-mon[97633]: pgmap v14798: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:44.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:11:44.068+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:11:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:44.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:44.156487+0000) 2022-09-10T12:11:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:44.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:44.683379+0000) 2022-09-10T12:11:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:45.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:45.156725+0000) 2022-09-10T12:11:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:45 smithi104 ceph-mon[97044]: pgmap v14799: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:45 smithi110 ceph-mon[97633]: pgmap v14799: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:45.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:45.683507+0000) 2022-09-10T12:11:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:46.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:46.156990+0000) 2022-09-10T12:11:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:11:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:11:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:11:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:11:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:11:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:11:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:11:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:11:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:46.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:46.683688+0000) 2022-09-10T12:11:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:47.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:47.157197+0000) 2022-09-10T12:11:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:47 smithi104 ceph-mon[97044]: pgmap v14800: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:47 smithi110 ceph-mon[97633]: pgmap v14800: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:47.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:47.683890+0000) 2022-09-10T12:11:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:48.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:48.157357+0000) 2022-09-10T12:11:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:48.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:48.684180+0000) 2022-09-10T12:11:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:49.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:49.157496+0000) 2022-09-10T12:11:50.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:49 smithi104 ceph-mon[97044]: pgmap v14801: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:49.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:49.684361+0000) 2022-09-10T12:11:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:49 smithi110 ceph-mon[97633]: pgmap v14801: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:50.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:50.157713+0000) 2022-09-10T12:11:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:50.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:50.684556+0000) 2022-09-10T12:11:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:51.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:51.157918+0000) 2022-09-10T12:11:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:51 smithi104 ceph-mon[97044]: pgmap v14802: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:51.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:51.684785+0000) 2022-09-10T12:11:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:51 smithi110 ceph-mon[97633]: pgmap v14802: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:52.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:52.158170+0000) 2022-09-10T12:11:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:52.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:52.685003+0000) 2022-09-10T12:11:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:53.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:53.158373+0000) 2022-09-10T12:11:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:53 smithi104 ceph-mon[97044]: pgmap v14803: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:53.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:53.685214+0000) 2022-09-10T12:11:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:53 smithi110 ceph-mon[97633]: pgmap v14803: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:54.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:54.158599+0000) 2022-09-10T12:11:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:54.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:54.685504+0000) 2022-09-10T12:11:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:55.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:55.158844+0000) 2022-09-10T12:11:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:55 smithi104 ceph-mon[97044]: pgmap v14804: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:55 smithi110 ceph-mon[97633]: pgmap v14804: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:11:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:55.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:55.685724+0000) 2022-09-10T12:11:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:56.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:56.159088+0000) 2022-09-10T12:11:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:56.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:56.685945+0000) 2022-09-10T12:11:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:57.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:57.159347+0000) 2022-09-10T12:11:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:57 smithi104 ceph-mon[97044]: pgmap v14805: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:57.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:57.686205+0000) 2022-09-10T12:11:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:57 smithi110 ceph-mon[97633]: pgmap v14805: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:11:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:58.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:58.159540+0000) 2022-09-10T12:11:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:58.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:58.686473+0000) 2022-09-10T12:11:59.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:11:59.081+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:11:59.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:11:59.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:59.159772+0000) 2022-09-10T12:12:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:11:59 smithi104 ceph-mon[97044]: pgmap v14806: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:11:59.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:11:59.686701+0000) 2022-09-10T12:12:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:11:59 smithi110 ceph-mon[97633]: pgmap v14806: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:00.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:00.159989+0000) 2022-09-10T12:12:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:00.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:00.686861+0000) 2022-09-10T12:12:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:01.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:01.160127+0000) 2022-09-10T12:12:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:01 smithi104 ceph-mon[97044]: pgmap v14807: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:01.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:01.687016+0000) 2022-09-10T12:12:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:01 smithi110 ceph-mon[97633]: pgmap v14807: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:02.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:02.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:02.160347+0000) 2022-09-10T12:12:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:02.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:02.687240+0000) 2022-09-10T12:12:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:03.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:03.160592+0000) 2022-09-10T12:12:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:03 smithi104 ceph-mon[97044]: pgmap v14808: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:03.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:03.687384+0000) 2022-09-10T12:12:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:03 smithi110 ceph-mon[97633]: pgmap v14808: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:04.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:04.160856+0000) 2022-09-10T12:12:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:04.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:04.687578+0000) 2022-09-10T12:12:05.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:05.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:05.161130+0000) 2022-09-10T12:12:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:05 smithi104 ceph-mon[97044]: pgmap v14809: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:12:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:05 smithi110 ceph-mon[97633]: pgmap v14809: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:12:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:05.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:05.687736+0000) 2022-09-10T12:12:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:06.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:06.161299+0000) 2022-09-10T12:12:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:06.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:06.687913+0000) 2022-09-10T12:12:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:07.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:07.161497+0000) 2022-09-10T12:12:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:07 smithi104 ceph-mon[97044]: pgmap v14810: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:12:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:07.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:07.688184+0000) 2022-09-10T12:12:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:07 smithi110 ceph-mon[97633]: pgmap v14810: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:12:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:08.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:08.161661+0000) 2022-09-10T12:12:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:08 smithi104 ceph-mon[97044]: osdmap e1006: 12 total, 12 up, 12 in 2022-09-10T12:12:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:08.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:08.688433+0000) 2022-09-10T12:12:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:08 smithi110 ceph-mon[97633]: osdmap e1006: 12 total, 12 up, 12 in 2022-09-10T12:12:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:09.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:09.161931+0000) 2022-09-10T12:12:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:09 smithi104 ceph-mon[97044]: pgmap v14812: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:12:10.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:09.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:09.688677+0000) 2022-09-10T12:12:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:09 smithi110 ceph-mon[97633]: pgmap v14812: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:12:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:10.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:10.162148+0000) 2022-09-10T12:12:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:10.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:10.688918+0000) 2022-09-10T12:12:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:11.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:11.162393+0000) 2022-09-10T12:12:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:11 smithi104 ceph-mon[97044]: pgmap v14813: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:12:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:11.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:11.689143+0000) 2022-09-10T12:12:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:11 smithi110 ceph-mon[97633]: pgmap v14813: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:12:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:12.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:12.162557+0000) 2022-09-10T12:12:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:12.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:12.689327+0000) 2022-09-10T12:12:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:13.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:13.162802+0000) 2022-09-10T12:12:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:13 smithi104 ceph-mon[97044]: pgmap v14814: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:12:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:13.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:13.689573+0000) 2022-09-10T12:12:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:13 smithi110 ceph-mon[97633]: pgmap v14814: 129 pgs: 129 active+clean; 9.3 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:12:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:14.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:14.163027+0000) 2022-09-10T12:12:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:12:14.097+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:12:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:14.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:14.689797+0000) 2022-09-10T12:12:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:15.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:15.163179+0000) 2022-09-10T12:12:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:15 smithi104 ceph-mon[97044]: osdmap e1007: 12 total, 12 up, 12 in 2022-09-10T12:12:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:15 smithi104 ceph-mon[97044]: pgmap v14816: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:12:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:15 smithi110 ceph-mon[97633]: osdmap e1007: 12 total, 12 up, 12 in 2022-09-10T12:12:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:15 smithi110 ceph-mon[97633]: pgmap v14816: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:12:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:15.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:15.690053+0000) 2022-09-10T12:12:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:16.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:16.163410+0000) 2022-09-10T12:12:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:16.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:16.690286+0000) 2022-09-10T12:12:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:17.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:17.163593+0000) 2022-09-10T12:12:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:17 smithi104 ceph-mon[97044]: pgmap v14817: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:12:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:17.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:17.690485+0000) 2022-09-10T12:12:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:17 smithi110 ceph-mon[97633]: pgmap v14817: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:12:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:18.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:18.163779+0000) 2022-09-10T12:12:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:18.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:18.690770+0000) 2022-09-10T12:12:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:19.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:19.163997+0000) 2022-09-10T12:12:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:19 smithi104 ceph-mon[97044]: pgmap v14818: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:12:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:19.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:19.690958+0000) 2022-09-10T12:12:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:19 smithi110 ceph-mon[97633]: pgmap v14818: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:12:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:20.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:20.164196+0000) 2022-09-10T12:12:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:20.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:20.691203+0000) 2022-09-10T12:12:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:21.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:21.164438+0000) 2022-09-10T12:12:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:21 smithi104 ceph-mon[97044]: pgmap v14819: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:12:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:21.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:21.691417+0000) 2022-09-10T12:12:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:21 smithi110 ceph-mon[97633]: pgmap v14819: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:12:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:22.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:22.164661+0000) 2022-09-10T12:12:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:22.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:22.691547+0000) 2022-09-10T12:12:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:23.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:23.164924+0000) 2022-09-10T12:12:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:23 smithi104 ceph-mon[97044]: pgmap v14820: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:12:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:23.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:23.691766+0000) 2022-09-10T12:12:24.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:23 smithi110 ceph-mon[97633]: pgmap v14820: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:12:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:24.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:24.165079+0000) 2022-09-10T12:12:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:24.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:24.691999+0000) 2022-09-10T12:12:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:25.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:25.165240+0000) 2022-09-10T12:12:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:25 smithi104 ceph-mon[97044]: pgmap v14821: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:12:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:25 smithi110 ceph-mon[97633]: pgmap v14821: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:12:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:25.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:25.692170+0000) 2022-09-10T12:12:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:26.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:26.165443+0000) 2022-09-10T12:12:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:26.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:26.692391+0000) 2022-09-10T12:12:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:27.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:27.165679+0000) 2022-09-10T12:12:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:27 smithi104 ceph-mon[97044]: pgmap v14822: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:27.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:27.692536+0000) 2022-09-10T12:12:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:27 smithi110 ceph-mon[97633]: pgmap v14822: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:28.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:28.165827+0000) 2022-09-10T12:12:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:28.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:28.692774+0000) 2022-09-10T12:12:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:29.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:29.166073+0000) 2022-09-10T12:12:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:12:29.107+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:12:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:29 smithi104 ceph-mon[97044]: pgmap v14823: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:29.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:29.692929+0000) 2022-09-10T12:12:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:29 smithi110 ceph-mon[97633]: pgmap v14823: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:30.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:30.166279+0000) 2022-09-10T12:12:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:30.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:30.693160+0000) 2022-09-10T12:12:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:31.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:31.166445+0000) 2022-09-10T12:12:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:31 smithi104 ceph-mon[97044]: pgmap v14824: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:31.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:31.693288+0000) 2022-09-10T12:12:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:31 smithi110 ceph-mon[97633]: pgmap v14824: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:32.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:32.166655+0000) 2022-09-10T12:12:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:32.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:32.693441+0000) 2022-09-10T12:12:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:33.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:33.166855+0000) 2022-09-10T12:12:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:33 smithi104 ceph-mon[97044]: pgmap v14825: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:33.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:33.693645+0000) 2022-09-10T12:12:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:33 smithi110 ceph-mon[97633]: pgmap v14825: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:34.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:34.167161+0000) 2022-09-10T12:12:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:34.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:34.693850+0000) 2022-09-10T12:12:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:35.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:35.167393+0000) 2022-09-10T12:12:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:35 smithi104 ceph-mon[97044]: pgmap v14826: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:35 smithi110 ceph-mon[97633]: pgmap v14826: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:35.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:35.694064+0000) 2022-09-10T12:12:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:36.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:36.167547+0000) 2022-09-10T12:12:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:36.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:36.694257+0000) 2022-09-10T12:12:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:37.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:37.167769+0000) 2022-09-10T12:12:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:37 smithi104 ceph-mon[97044]: pgmap v14827: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:37.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:37.694440+0000) 2022-09-10T12:12:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:37 smithi110 ceph-mon[97633]: pgmap v14827: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:38.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:38.168003+0000) 2022-09-10T12:12:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:38.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:38.694698+0000) 2022-09-10T12:12:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:39.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:39.168237+0000) 2022-09-10T12:12:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:39 smithi104 ceph-mon[97044]: pgmap v14828: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:39.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:39.694956+0000) 2022-09-10T12:12:40.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:39 smithi110 ceph-mon[97633]: pgmap v14828: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:40.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:40.168439+0000) 2022-09-10T12:12:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:40.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:40.695101+0000) 2022-09-10T12:12:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:41.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:41.168662+0000) 2022-09-10T12:12:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:41 smithi104 ceph-mon[97044]: pgmap v14829: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:41.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:41.695355+0000) 2022-09-10T12:12:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:41 smithi110 ceph-mon[97633]: pgmap v14829: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:42.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:42.168921+0000) 2022-09-10T12:12:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:42.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:42.695496+0000) 2022-09-10T12:12:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:43.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:43.169188+0000) 2022-09-10T12:12:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:43 smithi104 ceph-mon[97044]: pgmap v14830: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:43.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:43.695710+0000) 2022-09-10T12:12:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:43 smithi110 ceph-mon[97633]: pgmap v14830: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:44.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:12:44.123+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:12:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:44.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:44.169382+0000) 2022-09-10T12:12:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:44.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:44.695955+0000) 2022-09-10T12:12:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:45.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:45.169614+0000) 2022-09-10T12:12:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:45 smithi104 ceph-mon[97044]: pgmap v14831: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:45 smithi110 ceph-mon[97633]: pgmap v14831: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:45.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:45.696220+0000) 2022-09-10T12:12:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:46.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:46.169849+0000) 2022-09-10T12:12:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:12:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:12:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:12:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:46.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:46.696375+0000) 2022-09-10T12:12:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:47.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:47.170090+0000) 2022-09-10T12:12:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:47 smithi104 ceph-mon[97044]: pgmap v14832: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:47 smithi110 ceph-mon[97633]: pgmap v14832: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:47.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:47.696545+0000) 2022-09-10T12:12:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:48.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:48.170316+0000) 2022-09-10T12:12:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:48.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:48.696808+0000) 2022-09-10T12:12:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:49.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:49.170517+0000) 2022-09-10T12:12:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:49 smithi104 ceph-mon[97044]: pgmap v14833: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:49.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:49.697051+0000) 2022-09-10T12:12:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:49 smithi110 ceph-mon[97633]: pgmap v14833: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:50.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:50.170674+0000) 2022-09-10T12:12:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:50.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:50.697329+0000) 2022-09-10T12:12:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:51.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:51.170855+0000) 2022-09-10T12:12:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:51 smithi104 ceph-mon[97044]: pgmap v14834: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:51.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:51.697570+0000) 2022-09-10T12:12:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:51 smithi110 ceph-mon[97633]: pgmap v14834: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:52.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:52.171128+0000) 2022-09-10T12:12:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:52.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:52.697765+0000) 2022-09-10T12:12:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:53.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:53.171385+0000) 2022-09-10T12:12:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:53 smithi104 ceph-mon[97044]: pgmap v14835: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:53.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:53.698009+0000) 2022-09-10T12:12:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:53 smithi110 ceph-mon[97633]: pgmap v14835: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:54.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:54.171630+0000) 2022-09-10T12:12:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:54.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:54.698270+0000) 2022-09-10T12:12:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:55.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:55.171864+0000) 2022-09-10T12:12:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:55 smithi104 ceph-mon[97044]: pgmap v14836: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:55 smithi110 ceph-mon[97633]: pgmap v14836: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:12:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:55.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:55.698521+0000) 2022-09-10T12:12:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:56.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:56.172162+0000) 2022-09-10T12:12:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:56.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:56.698727+0000) 2022-09-10T12:12:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:57.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:57.172388+0000) 2022-09-10T12:12:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:57 smithi104 ceph-mon[97044]: pgmap v14837: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:57.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:57.698970+0000) 2022-09-10T12:12:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:57 smithi110 ceph-mon[97633]: pgmap v14837: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:12:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:58.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:58.172525+0000) 2022-09-10T12:12:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:58.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:58.699180+0000) 2022-09-10T12:12:59.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:12:59.132+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:12:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:12:59.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:59.172768+0000) 2022-09-10T12:13:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:12:59 smithi104 ceph-mon[97044]: pgmap v14838: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:12:59.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:12:59.699383+0000) 2022-09-10T12:13:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:12:59 smithi110 ceph-mon[97633]: pgmap v14838: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:00.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:00.172959+0000) 2022-09-10T12:13:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:00.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:00.699507+0000) 2022-09-10T12:13:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:01.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:01.173190+0000) 2022-09-10T12:13:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:01 smithi104 ceph-mon[97044]: pgmap v14839: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:01.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:01.699731+0000) 2022-09-10T12:13:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:01 smithi110 ceph-mon[97633]: pgmap v14839: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:02.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:02.173432+0000) 2022-09-10T12:13:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:02.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:02.699954+0000) 2022-09-10T12:13:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:03.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:03.173603+0000) 2022-09-10T12:13:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:03 smithi104 ceph-mon[97044]: pgmap v14840: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:03.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:03.700209+0000) 2022-09-10T12:13:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:03 smithi110 ceph-mon[97633]: pgmap v14840: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:04.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:04.173737+0000) 2022-09-10T12:13:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:04.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:04.700390+0000) 2022-09-10T12:13:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:05.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:05.174007+0000) 2022-09-10T12:13:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:05 smithi104 ceph-mon[97044]: pgmap v14841: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:13:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:05 smithi110 ceph-mon[97633]: pgmap v14841: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:13:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:05.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:05.700602+0000) 2022-09-10T12:13:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:06.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:06.174276+0000) 2022-09-10T12:13:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:06.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:06.700819+0000) 2022-09-10T12:13:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:07.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:07.174447+0000) 2022-09-10T12:13:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:07 smithi104 ceph-mon[97044]: pgmap v14842: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:13:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:07.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:07.701012+0000) 2022-09-10T12:13:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:07 smithi110 ceph-mon[97633]: pgmap v14842: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:13:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:08.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:08.174621+0000) 2022-09-10T12:13:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:08 smithi104 ceph-mon[97044]: osdmap e1008: 12 total, 12 up, 12 in 2022-09-10T12:13:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:08.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:08.701171+0000) 2022-09-10T12:13:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:08 smithi110 ceph-mon[97633]: osdmap e1008: 12 total, 12 up, 12 in 2022-09-10T12:13:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:09.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:09.174835+0000) 2022-09-10T12:13:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:09 smithi104 ceph-mon[97044]: pgmap v14844: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:13:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:09.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:09.701320+0000) 2022-09-10T12:13:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:09 smithi110 ceph-mon[97633]: pgmap v14844: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:13:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:10.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:10.175019+0000) 2022-09-10T12:13:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:10.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:10.701508+0000) 2022-09-10T12:13:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:11.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:11.175189+0000) 2022-09-10T12:13:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:11 smithi104 ceph-mon[97044]: pgmap v14845: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:13:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:11.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:11.701670+0000) 2022-09-10T12:13:12.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:11 smithi110 ceph-mon[97633]: pgmap v14845: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:13:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:12.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:12.175384+0000) 2022-09-10T12:13:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:12.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:12.701920+0000) 2022-09-10T12:13:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:13.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:13.175593+0000) 2022-09-10T12:13:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:13 smithi104 ceph-mon[97044]: pgmap v14846: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:13:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:13.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:13.702198+0000) 2022-09-10T12:13:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:13 smithi110 ceph-mon[97633]: pgmap v14846: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:13:14.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:13:14.143+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:13:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:14.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:14.175851+0000) 2022-09-10T12:13:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:14.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:14.702393+0000) 2022-09-10T12:13:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:15.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:15.176149+0000) 2022-09-10T12:13:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:15 smithi104 ceph-mon[97044]: osdmap e1009: 12 total, 12 up, 12 in 2022-09-10T12:13:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:15 smithi104 ceph-mon[97044]: pgmap v14848: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:13:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:15 smithi110 ceph-mon[97633]: osdmap e1009: 12 total, 12 up, 12 in 2022-09-10T12:13:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:15 smithi110 ceph-mon[97633]: pgmap v14848: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:13:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:15.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:15.702564+0000) 2022-09-10T12:13:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:16.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:16.176386+0000) 2022-09-10T12:13:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:16.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:16.702704+0000) 2022-09-10T12:13:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:17.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:17.176603+0000) 2022-09-10T12:13:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:17 smithi104 ceph-mon[97044]: pgmap v14849: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:13:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:17.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:17.702915+0000) 2022-09-10T12:13:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:17 smithi110 ceph-mon[97633]: pgmap v14849: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:13:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:18.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:18.176833+0000) 2022-09-10T12:13:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:18.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:18.703082+0000) 2022-09-10T12:13:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:19.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:19.176988+0000) 2022-09-10T12:13:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:19 smithi104 ceph-mon[97044]: pgmap v14850: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:13:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:19.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:19.703346+0000) 2022-09-10T12:13:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:19 smithi110 ceph-mon[97633]: pgmap v14850: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:13:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:20.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:20.177196+0000) 2022-09-10T12:13:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:20.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:20.703570+0000) 2022-09-10T12:13:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:21.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:21.177462+0000) 2022-09-10T12:13:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:21 smithi104 ceph-mon[97044]: pgmap v14851: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:13:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:21.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:21.703861+0000) 2022-09-10T12:13:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:21 smithi110 ceph-mon[97633]: pgmap v14851: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:13:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:22.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:22.177696+0000) 2022-09-10T12:13:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:22.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:22.704151+0000) 2022-09-10T12:13:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:23.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:23.177949+0000) 2022-09-10T12:13:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:23 smithi104 ceph-mon[97044]: pgmap v14852: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:13:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:23.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:23.704270+0000) 2022-09-10T12:13:24.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:23 smithi110 ceph-mon[97633]: pgmap v14852: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:13:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:24.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:24.178191+0000) 2022-09-10T12:13:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:24.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:24.704455+0000) 2022-09-10T12:13:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:25.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:25.178437+0000) 2022-09-10T12:13:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:25 smithi104 ceph-mon[97044]: pgmap v14853: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:13:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:25 smithi110 ceph-mon[97633]: pgmap v14853: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 99 B/s wr, 11 op/s 2022-09-10T12:13:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:25.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:25.704637+0000) 2022-09-10T12:13:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:26.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:26.178605+0000) 2022-09-10T12:13:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:26.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:26.704866+0000) 2022-09-10T12:13:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:27.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:27.178822+0000) 2022-09-10T12:13:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:27 smithi104 ceph-mon[97044]: pgmap v14854: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:27.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:27.705007+0000) 2022-09-10T12:13:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:27 smithi110 ceph-mon[97633]: pgmap v14854: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:28.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:28.179078+0000) 2022-09-10T12:13:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:28.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:28.705209+0000) 2022-09-10T12:13:29.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:13:29.146+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:13:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:29.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:29.179319+0000) 2022-09-10T12:13:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:29 smithi104 ceph-mon[97044]: pgmap v14855: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:29.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:29.705436+0000) 2022-09-10T12:13:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:29 smithi110 ceph-mon[97633]: pgmap v14855: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:30.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:30.179550+0000) 2022-09-10T12:13:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:30.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:30.705658+0000) 2022-09-10T12:13:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:31.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:31.179785+0000) 2022-09-10T12:13:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:31 smithi104 ceph-mon[97044]: pgmap v14856: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:31.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:31.705854+0000) 2022-09-10T12:13:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:31 smithi110 ceph-mon[97633]: pgmap v14856: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:32.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:32.179994+0000) 2022-09-10T12:13:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:32.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:32.706082+0000) 2022-09-10T12:13:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:33.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:33.180272+0000) 2022-09-10T12:13:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:33 smithi104 ceph-mon[97044]: pgmap v14857: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:33.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:33.706355+0000) 2022-09-10T12:13:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:33 smithi110 ceph-mon[97633]: pgmap v14857: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:34.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:34.180438+0000) 2022-09-10T12:13:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:34.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:34.706553+0000) 2022-09-10T12:13:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:35.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:35.180599+0000) 2022-09-10T12:13:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:35 smithi104 ceph-mon[97044]: pgmap v14858: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:35 smithi110 ceph-mon[97633]: pgmap v14858: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:35.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:35.706812+0000) 2022-09-10T12:13:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:36.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:36.180826+0000) 2022-09-10T12:13:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:36.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:36.707039+0000) 2022-09-10T12:13:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:37.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:37.181032+0000) 2022-09-10T12:13:38.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:37 smithi104 ceph-mon[97044]: pgmap v14859: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:37.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:37.707261+0000) 2022-09-10T12:13:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:37 smithi110 ceph-mon[97633]: pgmap v14859: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:38.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:38.181237+0000) 2022-09-10T12:13:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:38.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:38.707471+0000) 2022-09-10T12:13:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:39.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:39.181482+0000) 2022-09-10T12:13:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:39 smithi104 ceph-mon[97044]: pgmap v14860: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:39.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:39.707710+0000) 2022-09-10T12:13:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:39 smithi110 ceph-mon[97633]: pgmap v14860: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:40.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:40.181748+0000) 2022-09-10T12:13:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:40.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:40.707943+0000) 2022-09-10T12:13:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:41.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:41.181979+0000) 2022-09-10T12:13:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:41 smithi104 ceph-mon[97044]: pgmap v14861: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:41.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:41.708211+0000) 2022-09-10T12:13:42.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:41 smithi110 ceph-mon[97633]: pgmap v14861: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:42.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:42.182188+0000) 2022-09-10T12:13:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:42.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:42.708409+0000) 2022-09-10T12:13:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:43.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:43.182484+0000) 2022-09-10T12:13:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:43 smithi104 ceph-mon[97044]: pgmap v14862: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:43.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:43.708589+0000) 2022-09-10T12:13:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:43 smithi110 ceph-mon[97633]: pgmap v14862: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:44.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:44.182724+0000) 2022-09-10T12:13:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:13:44.153+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:13:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:44.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:44.708774+0000) 2022-09-10T12:13:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:45.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:45.182996+0000) 2022-09-10T12:13:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:45 smithi104 ceph-mon[97044]: pgmap v14863: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:45 smithi110 ceph-mon[97633]: pgmap v14863: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:45.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:45.708954+0000) 2022-09-10T12:13:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:46.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:46.183224+0000) 2022-09-10T12:13:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:13:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:13:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:13:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:13:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:13:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:13:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:46.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:46.709225+0000) 2022-09-10T12:13:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:47.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:47.183500+0000) 2022-09-10T12:13:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:47 smithi104 ceph-mon[97044]: pgmap v14864: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:13:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:47 smithi110 ceph-mon[97633]: pgmap v14864: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:13:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:47.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:47.709435+0000) 2022-09-10T12:13:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:48.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:48.183667+0000) 2022-09-10T12:13:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:48.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:48.709630+0000) 2022-09-10T12:13:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:49.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:49.183919+0000) 2022-09-10T12:13:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:49 smithi104 ceph-mon[97044]: pgmap v14865: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:13:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:49.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:49.709854+0000) 2022-09-10T12:13:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:49 smithi110 ceph-mon[97633]: pgmap v14865: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:13:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:50.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:50.184210+0000) 2022-09-10T12:13:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:50.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:50.710132+0000) 2022-09-10T12:13:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:51.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:51.184386+0000) 2022-09-10T12:13:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:51 smithi104 ceph-mon[97044]: pgmap v14866: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:51.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:51.710377+0000) 2022-09-10T12:13:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:51 smithi110 ceph-mon[97633]: pgmap v14866: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:52.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:52.184605+0000) 2022-09-10T12:13:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:52.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:52.710508+0000) 2022-09-10T12:13:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:53.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:53.184815+0000) 2022-09-10T12:13:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:53 smithi104 ceph-mon[97044]: pgmap v14867: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:13:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:53.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:53.710702+0000) 2022-09-10T12:13:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:53 smithi110 ceph-mon[97633]: pgmap v14867: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:13:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:54.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:54.184930+0000) 2022-09-10T12:13:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:54.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:54.710958+0000) 2022-09-10T12:13:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:55.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:55.185188+0000) 2022-09-10T12:13:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:55 smithi104 ceph-mon[97044]: pgmap v14868: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:55 smithi110 ceph-mon[97633]: pgmap v14868: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:13:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:55.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:55.711179+0000) 2022-09-10T12:13:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:56.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:56.185412+0000) 2022-09-10T12:13:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:56.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:56.711303+0000) 2022-09-10T12:13:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:57.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:57.185652+0000) 2022-09-10T12:13:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:57 smithi104 ceph-mon[97044]: pgmap v14869: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:57.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:57.711512+0000) 2022-09-10T12:13:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:57 smithi110 ceph-mon[97633]: pgmap v14869: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:13:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:58.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:58.185929+0000) 2022-09-10T12:13:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:58.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:58.711745+0000) 2022-09-10T12:13:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:13:59.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:59.186210+0000) 2022-09-10T12:13:59.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:13:59.169+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:14:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:13:59 smithi104 ceph-mon[97044]: pgmap v14870: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:13:59.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:13:59.711970+0000) 2022-09-10T12:14:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:13:59 smithi110 ceph-mon[97633]: pgmap v14870: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:00.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:00.186443+0000) 2022-09-10T12:14:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:00.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:00.712137+0000) 2022-09-10T12:14:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:01.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:01.186647+0000) 2022-09-10T12:14:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:01 smithi104 ceph-mon[97044]: pgmap v14871: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:01.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:01.712314+0000) 2022-09-10T12:14:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:01 smithi110 ceph-mon[97633]: pgmap v14871: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:02.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:02.186856+0000) 2022-09-10T12:14:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:02.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:02.712556+0000) 2022-09-10T12:14:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:03.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:03.187047+0000) 2022-09-10T12:14:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:03 smithi104 ceph-mon[97044]: pgmap v14872: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:03.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:03.712718+0000) 2022-09-10T12:14:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:03 smithi110 ceph-mon[97633]: pgmap v14872: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:04.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:04.187236+0000) 2022-09-10T12:14:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:04.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:04.712942+0000) 2022-09-10T12:14:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:05.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:05.187517+0000) 2022-09-10T12:14:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:05 smithi104 ceph-mon[97044]: pgmap v14873: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:14:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:05 smithi110 ceph-mon[97633]: pgmap v14873: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:14:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:05.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:05.713187+0000) 2022-09-10T12:14:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:06.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:06.187714+0000) 2022-09-10T12:14:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:06.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:06.713371+0000) 2022-09-10T12:14:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:07.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:07.187957+0000) 2022-09-10T12:14:08.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:07 smithi104 ceph-mon[97044]: pgmap v14874: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:14:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:07.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:07.713569+0000) 2022-09-10T12:14:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:07 smithi110 ceph-mon[97633]: pgmap v14874: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:14:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:08.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:08.188214+0000) 2022-09-10T12:14:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:08 smithi104 ceph-mon[97044]: osdmap e1010: 12 total, 12 up, 12 in 2022-09-10T12:14:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:08.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:08.713769+0000) 2022-09-10T12:14:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:08 smithi110 ceph-mon[97633]: osdmap e1010: 12 total, 12 up, 12 in 2022-09-10T12:14:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:09.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:09.188449+0000) 2022-09-10T12:14:10.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:09 smithi104 ceph-mon[97044]: pgmap v14876: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:14:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:09.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:09.713987+0000) 2022-09-10T12:14:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:09 smithi110 ceph-mon[97633]: pgmap v14876: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:14:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:10.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:10.188715+0000) 2022-09-10T12:14:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:10.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:10.714222+0000) 2022-09-10T12:14:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:11.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:11.188982+0000) 2022-09-10T12:14:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:11 smithi104 ceph-mon[97044]: pgmap v14877: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:14:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:11.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:11.714418+0000) 2022-09-10T12:14:12.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:11 smithi110 ceph-mon[97633]: pgmap v14877: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:14:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:12.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:12.189192+0000) 2022-09-10T12:14:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:12.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:12.714682+0000) 2022-09-10T12:14:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:13.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:13.189420+0000) 2022-09-10T12:14:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:13 smithi104 ceph-mon[97044]: pgmap v14878: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:14:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:13.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:13.714887+0000) 2022-09-10T12:14:14.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:13 smithi110 ceph-mon[97633]: pgmap v14878: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:14:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:14.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:14.189606+0000) 2022-09-10T12:14:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:14:14.183+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:14:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:14.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:14.715070+0000) 2022-09-10T12:14:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:15.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:15.189782+0000) 2022-09-10T12:14:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:15 smithi104 ceph-mon[97044]: osdmap e1011: 12 total, 12 up, 12 in 2022-09-10T12:14:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:15 smithi104 ceph-mon[97044]: pgmap v14880: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:14:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:15 smithi110 ceph-mon[97633]: osdmap e1011: 12 total, 12 up, 12 in 2022-09-10T12:14:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:15 smithi110 ceph-mon[97633]: pgmap v14880: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:14:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:15.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:15.715312+0000) 2022-09-10T12:14:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:16.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:16.190026+0000) 2022-09-10T12:14:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:16.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:16.715593+0000) 2022-09-10T12:14:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:17.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:17.190212+0000) 2022-09-10T12:14:18.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:17 smithi104 ceph-mon[97044]: pgmap v14881: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:14:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:17.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:17.715790+0000) 2022-09-10T12:14:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:17 smithi110 ceph-mon[97633]: pgmap v14881: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:14:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:18.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:18.190432+0000) 2022-09-10T12:14:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:18.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:18.716087+0000) 2022-09-10T12:14:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:19.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:19.190669+0000) 2022-09-10T12:14:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:19 smithi104 ceph-mon[97044]: pgmap v14882: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:14:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:19.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:19.716402+0000) 2022-09-10T12:14:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:19 smithi110 ceph-mon[97633]: pgmap v14882: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:14:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:20.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:20.190898+0000) 2022-09-10T12:14:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:20.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:20.716669+0000) 2022-09-10T12:14:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:21.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:21.191193+0000) 2022-09-10T12:14:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:21 smithi104 ceph-mon[97044]: pgmap v14883: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:14:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:21.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:21.716910+0000) 2022-09-10T12:14:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:21 smithi110 ceph-mon[97633]: pgmap v14883: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:14:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:22.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:22.191433+0000) 2022-09-10T12:14:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:22.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:22.717166+0000) 2022-09-10T12:14:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:23.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:23.191689+0000) 2022-09-10T12:14:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:23 smithi104 ceph-mon[97044]: pgmap v14884: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:14:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:23.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:23.717385+0000) 2022-09-10T12:14:24.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:23 smithi110 ceph-mon[97633]: pgmap v14884: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:14:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:24.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:24.191925+0000) 2022-09-10T12:14:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:24.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:24.717579+0000) 2022-09-10T12:14:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:25.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:25.192137+0000) 2022-09-10T12:14:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:25 smithi104 ceph-mon[97044]: pgmap v14885: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:14:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:25 smithi110 ceph-mon[97633]: pgmap v14885: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:14:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:25.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:25.717810+0000) 2022-09-10T12:14:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:26.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:26.192396+0000) 2022-09-10T12:14:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:26.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:26.718013+0000) 2022-09-10T12:14:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:27.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:27.192616+0000) 2022-09-10T12:14:28.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:27 smithi104 ceph-mon[97044]: pgmap v14886: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:27.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:27.718202+0000) 2022-09-10T12:14:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:27 smithi110 ceph-mon[97633]: pgmap v14886: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:28.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:28.192839+0000) 2022-09-10T12:14:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:28.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:28.718444+0000) 2022-09-10T12:14:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:29.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:29.193133+0000) 2022-09-10T12:14:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:14:29.198+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:14:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:29 smithi104 ceph-mon[97044]: pgmap v14887: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:29.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:29.718599+0000) 2022-09-10T12:14:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:29 smithi110 ceph-mon[97633]: pgmap v14887: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:30.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:30.193325+0000) 2022-09-10T12:14:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:30.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:30.718794+0000) 2022-09-10T12:14:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:31.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:31.193529+0000) 2022-09-10T12:14:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:31 smithi104 ceph-mon[97044]: pgmap v14888: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:31.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:31.719068+0000) 2022-09-10T12:14:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:31 smithi110 ceph-mon[97633]: pgmap v14888: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:32.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:32.193720+0000) 2022-09-10T12:14:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:32.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:32.719295+0000) 2022-09-10T12:14:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:33.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:33.193955+0000) 2022-09-10T12:14:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:33 smithi104 ceph-mon[97044]: pgmap v14889: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:33.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:33.719521+0000) 2022-09-10T12:14:34.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:33 smithi110 ceph-mon[97633]: pgmap v14889: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:34.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:34.194171+0000) 2022-09-10T12:14:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:34.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:34.719699+0000) 2022-09-10T12:14:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:35.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:35.194402+0000) 2022-09-10T12:14:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:35 smithi104 ceph-mon[97044]: pgmap v14890: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:35 smithi110 ceph-mon[97633]: pgmap v14890: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:35.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:35.719894+0000) 2022-09-10T12:14:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:36.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:36.194618+0000) 2022-09-10T12:14:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:36.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:36.720162+0000) 2022-09-10T12:14:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:37.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:37.194777+0000) 2022-09-10T12:14:38.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:37 smithi104 ceph-mon[97044]: pgmap v14891: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:37.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:37.720344+0000) 2022-09-10T12:14:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:37 smithi110 ceph-mon[97633]: pgmap v14891: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:38.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:38.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:38.194996+0000) 2022-09-10T12:14:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:38.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:38.720569+0000) 2022-09-10T12:14:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:39.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:39.195183+0000) 2022-09-10T12:14:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:39 smithi104 ceph-mon[97044]: pgmap v14892: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:39.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:39.720775+0000) 2022-09-10T12:14:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:39 smithi110 ceph-mon[97633]: pgmap v14892: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:40.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:40.195369+0000) 2022-09-10T12:14:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:40.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:40.721058+0000) 2022-09-10T12:14:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:41.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:41.195591+0000) 2022-09-10T12:14:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:41 smithi104 ceph-mon[97044]: pgmap v14893: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:41.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:41.721275+0000) 2022-09-10T12:14:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:41 smithi110 ceph-mon[97633]: pgmap v14893: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:42.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:42.195766+0000) 2022-09-10T12:14:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:42.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:42.721456+0000) 2022-09-10T12:14:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:43.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:43.196030+0000) 2022-09-10T12:14:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:43 smithi104 ceph-mon[97044]: pgmap v14894: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:43.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:43.721654+0000) 2022-09-10T12:14:44.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:43 smithi110 ceph-mon[97633]: pgmap v14894: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:44.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:44.196274+0000) 2022-09-10T12:14:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:14:44.211+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:14:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:44.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:44.721916+0000) 2022-09-10T12:14:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:45.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:45.196525+0000) 2022-09-10T12:14:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:45 smithi104 ceph-mon[97044]: pgmap v14895: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:45 smithi110 ceph-mon[97633]: pgmap v14895: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:45.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:45.722092+0000) 2022-09-10T12:14:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:46.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:46.196754+0000) 2022-09-10T12:14:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:14:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:14:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:14:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:14:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:14:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:14:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:14:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:14:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:46.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:46.722303+0000) 2022-09-10T12:14:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:47.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:47.196953+0000) 2022-09-10T12:14:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:47 smithi104 ceph-mon[97044]: pgmap v14896: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:47 smithi110 ceph-mon[97633]: pgmap v14896: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:47.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:47.722485+0000) 2022-09-10T12:14:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:48.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:48.197185+0000) 2022-09-10T12:14:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:48.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:48.722745+0000) 2022-09-10T12:14:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:49.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:49.197379+0000) 2022-09-10T12:14:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:49 smithi104 ceph-mon[97044]: pgmap v14897: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:49.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:49.723004+0000) 2022-09-10T12:14:50.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:49 smithi110 ceph-mon[97633]: pgmap v14897: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:50.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:50.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:50.197608+0000) 2022-09-10T12:14:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:50.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:50.723198+0000) 2022-09-10T12:14:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:51.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:51.197843+0000) 2022-09-10T12:14:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:51 smithi104 ceph-mon[97044]: pgmap v14898: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:51.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:51.723373+0000) 2022-09-10T12:14:52.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:51 smithi110 ceph-mon[97633]: pgmap v14898: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:52.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:52.198069+0000) 2022-09-10T12:14:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:52.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:52.723522+0000) 2022-09-10T12:14:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:53.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:53.198333+0000) 2022-09-10T12:14:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:53 smithi104 ceph-mon[97044]: pgmap v14899: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:53.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:53.723765+0000) 2022-09-10T12:14:54.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:53 smithi110 ceph-mon[97633]: pgmap v14899: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:54.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:54.198562+0000) 2022-09-10T12:14:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:54.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:54.723970+0000) 2022-09-10T12:14:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:55.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:55.198731+0000) 2022-09-10T12:14:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:55 smithi104 ceph-mon[97044]: pgmap v14900: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:55 smithi110 ceph-mon[97633]: pgmap v14900: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:14:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:55.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:55.724219+0000) 2022-09-10T12:14:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:56.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:56.198916+0000) 2022-09-10T12:14:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:56.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:56.724438+0000) 2022-09-10T12:14:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:57.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:57.199126+0000) 2022-09-10T12:14:58.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:57 smithi104 ceph-mon[97044]: pgmap v14901: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:57.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:57.724685+0000) 2022-09-10T12:14:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:57 smithi110 ceph-mon[97633]: pgmap v14901: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:14:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:58.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:58.199310+0000) 2022-09-10T12:14:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:58.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:58.724918+0000) 2022-09-10T12:14:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:14:59.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:59.199559+0000) 2022-09-10T12:14:59.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:14:59.223+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:15:00.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:14:59 smithi104 ceph-mon[97044]: pgmap v14902: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:14:59.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:14:59.725219+0000) 2022-09-10T12:15:00.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:14:59 smithi110 ceph-mon[97633]: pgmap v14902: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:00.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:00.199785+0000) 2022-09-10T12:15:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:00.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:00.725387+0000) 2022-09-10T12:15:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:01.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:01.199950+0000) 2022-09-10T12:15:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:01 smithi104 ceph-mon[97044]: pgmap v14903: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:01.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:01.725629+0000) 2022-09-10T12:15:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:01 smithi110 ceph-mon[97633]: pgmap v14903: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:02.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:02.200160+0000) 2022-09-10T12:15:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:02.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:02.725863+0000) 2022-09-10T12:15:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:03.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:03.200334+0000) 2022-09-10T12:15:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:03.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:03.726161+0000) 2022-09-10T12:15:04.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:03 smithi110 ceph-mon[97633]: pgmap v14904: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:04.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:03 smithi104 ceph-mon[97044]: pgmap v14904: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:04.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:04.200562+0000) 2022-09-10T12:15:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:04.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:04.726439+0000) 2022-09-10T12:15:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:05.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:05.200813+0000) 2022-09-10T12:15:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:05 smithi104 ceph-mon[97044]: pgmap v14905: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:15:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:05 smithi110 ceph-mon[97633]: pgmap v14905: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:15:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:05.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:05.726653+0000) 2022-09-10T12:15:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:06.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:06.201019+0000) 2022-09-10T12:15:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:06.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:06.726889+0000) 2022-09-10T12:15:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:07.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:07.201263+0000) 2022-09-10T12:15:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:07.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:07.727084+0000) 2022-09-10T12:15:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:07 smithi110 ceph-mon[97633]: pgmap v14906: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:15:08.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:07 smithi104 ceph-mon[97044]: pgmap v14906: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:15:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:08.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:08.201462+0000) 2022-09-10T12:15:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:08.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:08.727266+0000) 2022-09-10T12:15:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:08 smithi110 ceph-mon[97633]: osdmap e1012: 12 total, 12 up, 12 in 2022-09-10T12:15:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:08 smithi104 ceph-mon[97044]: osdmap e1012: 12 total, 12 up, 12 in 2022-09-10T12:15:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:09.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:09.201701+0000) 2022-09-10T12:15:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:09.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:09.727450+0000) 2022-09-10T12:15:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:09 smithi110 ceph-mon[97633]: pgmap v14908: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:15:10.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:09 smithi104 ceph-mon[97044]: pgmap v14908: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:15:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:10.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:10.201921+0000) 2022-09-10T12:15:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:10.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:10.727628+0000) 2022-09-10T12:15:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:10 smithi110 ceph-mon[97633]: pgmap v14909: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:15:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:10 smithi104 ceph-mon[97044]: pgmap v14909: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:15:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:11.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:11.202162+0000) 2022-09-10T12:15:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:11.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:11.727854+0000) 2022-09-10T12:15:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:12.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:12.202373+0000) 2022-09-10T12:15:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:12.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:12.728054+0000) 2022-09-10T12:15:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:13.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:13.202571+0000) 2022-09-10T12:15:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:13.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:13.728258+0000) 2022-09-10T12:15:14.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:13 smithi110 ceph-mon[97633]: pgmap v14910: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:15:14.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:13 smithi104 ceph-mon[97044]: pgmap v14910: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:15:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:15:14.228+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:15:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:14.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:14.202795+0000) 2022-09-10T12:15:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:14.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:14.728506+0000) 2022-09-10T12:15:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:15.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:15.203039+0000) 2022-09-10T12:15:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:15 smithi104 ceph-mon[97044]: osdmap e1013: 12 total, 12 up, 12 in 2022-09-10T12:15:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:15 smithi104 ceph-mon[97044]: pgmap v14912: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:15:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:15 smithi110 ceph-mon[97633]: osdmap e1013: 12 total, 12 up, 12 in 2022-09-10T12:15:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:15 smithi110 ceph-mon[97633]: pgmap v14912: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:15:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:15.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:15.728724+0000) 2022-09-10T12:15:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:16.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:16.203242+0000) 2022-09-10T12:15:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:16.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:16.728940+0000) 2022-09-10T12:15:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:17.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:17.203465+0000) 2022-09-10T12:15:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:17.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:17.729154+0000) 2022-09-10T12:15:18.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:17 smithi110 ceph-mon[97633]: pgmap v14913: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:15:18.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:17 smithi104 ceph-mon[97044]: pgmap v14913: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:15:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:18.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:18.203645+0000) 2022-09-10T12:15:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:18.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:18.729385+0000) 2022-09-10T12:15:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:19.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:19.203903+0000) 2022-09-10T12:15:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:19.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:19.729621+0000) 2022-09-10T12:15:20.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:19 smithi110 ceph-mon[97633]: pgmap v14914: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:15:20.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:19 smithi104 ceph-mon[97044]: pgmap v14914: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:15:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:20.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:20.204092+0000) 2022-09-10T12:15:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:20.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:20.729878+0000) 2022-09-10T12:15:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:21.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:21.204304+0000) 2022-09-10T12:15:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:21.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:21.730062+0000) 2022-09-10T12:15:22.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:21 smithi110 ceph-mon[97633]: pgmap v14915: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:15:22.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:21 smithi104 ceph-mon[97044]: pgmap v14915: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:15:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:22.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:22.204430+0000) 2022-09-10T12:15:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:22.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:22.730311+0000) 2022-09-10T12:15:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:22 smithi110 ceph-mon[97633]: pgmap v14916: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:15:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:22 smithi104 ceph-mon[97044]: pgmap v14916: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:15:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:23.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:23.204647+0000) 2022-09-10T12:15:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:23.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:23.730516+0000) 2022-09-10T12:15:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:24.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:24.204864+0000) 2022-09-10T12:15:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:24.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:24.730777+0000) 2022-09-10T12:15:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:25.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:25.205127+0000) 2022-09-10T12:15:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:25 smithi104 ceph-mon[97044]: pgmap v14917: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:15:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:25 smithi110 ceph-mon[97633]: pgmap v14917: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:15:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:25.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:25.731019+0000) 2022-09-10T12:15:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:26.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:26.205359+0000) 2022-09-10T12:15:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:26.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:26.731284+0000) 2022-09-10T12:15:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:27.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:27.205556+0000) 2022-09-10T12:15:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:27.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:27.731539+0000) 2022-09-10T12:15:28.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:27 smithi110 ceph-mon[97633]: pgmap v14918: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T12:15:28.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:27 smithi104 ceph-mon[97044]: pgmap v14918: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T12:15:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:28.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:28.205787+0000) 2022-09-10T12:15:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:28.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:28.731719+0000) 2022-09-10T12:15:29.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:15:29.236+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:15:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:29.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:29.206033+0000) 2022-09-10T12:15:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:29.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:29.731915+0000) 2022-09-10T12:15:30.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:29 smithi110 ceph-mon[97633]: pgmap v14919: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T12:15:30.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:29 smithi104 ceph-mon[97044]: pgmap v14919: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T12:15:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:30.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:30.206165+0000) 2022-09-10T12:15:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:30.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:30.732129+0000) 2022-09-10T12:15:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:31.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:31.206408+0000) 2022-09-10T12:15:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:31.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:31.732346+0000) 2022-09-10T12:15:32.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:31 smithi110 ceph-mon[97633]: pgmap v14920: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T12:15:32.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:31 smithi104 ceph-mon[97044]: pgmap v14920: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T12:15:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:32.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:32.206644+0000) 2022-09-10T12:15:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:32.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:32.732556+0000) 2022-09-10T12:15:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:32 smithi110 ceph-mon[97633]: pgmap v14921: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:15:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:32 smithi104 ceph-mon[97044]: pgmap v14921: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:15:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:33.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:33.206893+0000) 2022-09-10T12:15:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:33.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:33.732819+0000) 2022-09-10T12:15:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:34.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:34.207148+0000) 2022-09-10T12:15:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:34.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:34.733067+0000) 2022-09-10T12:15:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:35.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:35.207376+0000) 2022-09-10T12:15:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:35 smithi104 ceph-mon[97044]: pgmap v14922: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:35 smithi110 ceph-mon[97633]: pgmap v14922: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:35.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:35.733266+0000) 2022-09-10T12:15:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:36.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:36.207546+0000) 2022-09-10T12:15:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:36.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:36.733498+0000) 2022-09-10T12:15:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:37.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:37.207833+0000) 2022-09-10T12:15:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:37.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:37.733727+0000) 2022-09-10T12:15:38.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:37 smithi110 ceph-mon[97633]: pgmap v14923: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:38.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:37 smithi104 ceph-mon[97044]: pgmap v14923: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:38.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:38.208025+0000) 2022-09-10T12:15:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:38.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:38.733926+0000) 2022-09-10T12:15:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:39.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:39.208223+0000) 2022-09-10T12:15:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:39.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:39.734219+0000) 2022-09-10T12:15:40.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:39 smithi110 ceph-mon[97633]: pgmap v14924: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:40.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:39 smithi104 ceph-mon[97044]: pgmap v14924: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:40.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:40.208390+0000) 2022-09-10T12:15:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:40.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:40.734491+0000) 2022-09-10T12:15:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:41.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:41.208593+0000) 2022-09-10T12:15:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:41.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:41.734769+0000) 2022-09-10T12:15:42.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:41 smithi110 ceph-mon[97633]: pgmap v14925: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:42.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:41 smithi104 ceph-mon[97044]: pgmap v14925: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:42.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:42.208717+0000) 2022-09-10T12:15:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:42.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:42.734918+0000) 2022-09-10T12:15:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:42 smithi110 ceph-mon[97633]: pgmap v14926: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:42 smithi104 ceph-mon[97044]: pgmap v14926: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:43.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:43.208891+0000) 2022-09-10T12:15:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:43.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:43.735118+0000) 2022-09-10T12:15:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:44.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:44.209088+0000) 2022-09-10T12:15:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:15:44.251+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:15:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:44.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:44.735353+0000) 2022-09-10T12:15:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:45.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:45.209354+0000) 2022-09-10T12:15:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:45 smithi104 ceph-mon[97044]: pgmap v14927: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:45 smithi110 ceph-mon[97633]: pgmap v14927: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:45.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:45.735531+0000) 2022-09-10T12:15:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:46.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:46.209597+0000) 2022-09-10T12:15:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:15:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:15:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:15:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:15:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:15:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:46.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:46.735752+0000) 2022-09-10T12:15:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:47.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:47.209818+0000) 2022-09-10T12:15:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:47 smithi104 ceph-mon[97044]: pgmap v14928: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:47 smithi110 ceph-mon[97633]: pgmap v14928: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:47.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:47.736004+0000) 2022-09-10T12:15:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:48.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:48.210027+0000) 2022-09-10T12:15:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:48.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:48.736320+0000) 2022-09-10T12:15:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:49.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:49.210264+0000) 2022-09-10T12:15:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:49.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:49.736565+0000) 2022-09-10T12:15:50.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:49 smithi110 ceph-mon[97633]: pgmap v14929: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:50.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:49 smithi104 ceph-mon[97044]: pgmap v14929: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:50.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:50.210515+0000) 2022-09-10T12:15:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:50.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:50.736797+0000) 2022-09-10T12:15:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:51.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:51.210746+0000) 2022-09-10T12:15:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:51.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:51.736984+0000) 2022-09-10T12:15:52.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:51 smithi110 ceph-mon[97633]: pgmap v14930: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:52.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:51 smithi104 ceph-mon[97044]: pgmap v14930: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:52.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:52.210914+0000) 2022-09-10T12:15:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:52.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:52.737211+0000) 2022-09-10T12:15:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:52 smithi110 ceph-mon[97633]: pgmap v14931: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:52 smithi104 ceph-mon[97044]: pgmap v14931: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:53.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:53.211150+0000) 2022-09-10T12:15:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:53.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:53.737484+0000) 2022-09-10T12:15:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:54.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:54.211329+0000) 2022-09-10T12:15:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:54.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:54.737766+0000) 2022-09-10T12:15:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:55.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:55.211527+0000) 2022-09-10T12:15:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:55 smithi104 ceph-mon[97044]: pgmap v14932: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:55 smithi110 ceph-mon[97633]: pgmap v14932: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:15:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:55.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:55.737982+0000) 2022-09-10T12:15:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:56.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:56.211715+0000) 2022-09-10T12:15:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:56.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:56.738176+0000) 2022-09-10T12:15:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:57.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:57.211946+0000) 2022-09-10T12:15:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:57.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:57.738402+0000) 2022-09-10T12:15:58.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:57 smithi110 ceph-mon[97633]: pgmap v14933: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:58.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:57 smithi104 ceph-mon[97044]: pgmap v14933: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:58.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:58.212208+0000) 2022-09-10T12:15:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:58.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:58.738667+0000) 2022-09-10T12:15:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:15:58 smithi110 ceph-mon[97633]: pgmap v14934: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:15:58 smithi104 ceph-mon[97044]: pgmap v14934: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:15:59.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:15:59.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:15:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:15:59.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:59.212368+0000) 2022-09-10T12:16:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:15:59.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:15:59.738911+0000) 2022-09-10T12:16:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:00.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:00.212540+0000) 2022-09-10T12:16:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:00.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:00.739138+0000) 2022-09-10T12:16:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:01.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:01.212719+0000) 2022-09-10T12:16:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:01.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:01.739348+0000) 2022-09-10T12:16:02.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:01 smithi110 ceph-mon[97633]: pgmap v14935: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:02.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:01 smithi104 ceph-mon[97044]: pgmap v14935: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:02.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:02.212946+0000) 2022-09-10T12:16:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:02.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:02.739604+0000) 2022-09-10T12:16:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:02 smithi110 ceph-mon[97633]: pgmap v14936: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:02 smithi104 ceph-mon[97044]: pgmap v14936: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:03.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:03.213123+0000) 2022-09-10T12:16:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:03.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:03.739883+0000) 2022-09-10T12:16:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:04.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:04.213370+0000) 2022-09-10T12:16:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:04.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:04.740160+0000) 2022-09-10T12:16:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:05.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:05.213546+0000) 2022-09-10T12:16:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:05 smithi104 ceph-mon[97044]: pgmap v14937: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:16:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:05 smithi110 ceph-mon[97633]: pgmap v14937: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:16:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:05.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:05.740396+0000) 2022-09-10T12:16:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:06.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:06.213765+0000) 2022-09-10T12:16:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:06.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:06.740623+0000) 2022-09-10T12:16:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:07.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:07.213996+0000) 2022-09-10T12:16:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:07.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:07.740883+0000) 2022-09-10T12:16:08.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:07 smithi110 ceph-mon[97633]: pgmap v14938: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:16:08.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:07 smithi104 ceph-mon[97044]: pgmap v14938: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:16:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:08.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:08.214219+0000) 2022-09-10T12:16:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:08.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:08.741079+0000) 2022-09-10T12:16:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:08 smithi110 ceph-mon[97633]: osdmap e1014: 12 total, 12 up, 12 in 2022-09-10T12:16:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:08 smithi110 ceph-mon[97633]: pgmap v14940: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:16:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:08 smithi104 ceph-mon[97044]: osdmap e1014: 12 total, 12 up, 12 in 2022-09-10T12:16:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:08 smithi104 ceph-mon[97044]: pgmap v14940: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:16:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:09.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:09.214430+0000) 2022-09-10T12:16:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:09.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:09.741320+0000) 2022-09-10T12:16:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:10.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:10.214596+0000) 2022-09-10T12:16:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:10.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:10.741506+0000) 2022-09-10T12:16:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:10 smithi110 ceph-mon[97633]: pgmap v14941: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:16:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:10 smithi104 ceph-mon[97044]: pgmap v14941: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:16:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:11.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:11.214777+0000) 2022-09-10T12:16:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:11.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:11.741757+0000) 2022-09-10T12:16:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:12.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:12.214908+0000) 2022-09-10T12:16:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:12.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:12.742017+0000) 2022-09-10T12:16:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:12 smithi110 ceph-mon[97633]: pgmap v14942: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:16:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:12 smithi104 ceph-mon[97044]: pgmap v14942: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:16:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:13.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:13.215148+0000) 2022-09-10T12:16:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:13.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:13.742284+0000) 2022-09-10T12:16:14.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:16:14.269+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:16:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:14.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:14.215307+0000) 2022-09-10T12:16:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:14.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:14.742514+0000) 2022-09-10T12:16:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:15.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:15.215488+0000) 2022-09-10T12:16:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:15 smithi104 ceph-mon[97044]: osdmap e1015: 12 total, 12 up, 12 in 2022-09-10T12:16:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:15 smithi104 ceph-mon[97044]: pgmap v14944: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:16:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:15 smithi110 ceph-mon[97633]: osdmap e1015: 12 total, 12 up, 12 in 2022-09-10T12:16:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:15 smithi110 ceph-mon[97633]: pgmap v14944: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:16:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:15.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:15.742790+0000) 2022-09-10T12:16:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:16.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:16.215670+0000) 2022-09-10T12:16:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:16.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:16.742944+0000) 2022-09-10T12:16:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:16 smithi110 ceph-mon[97633]: pgmap v14945: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:16:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:16 smithi104 ceph-mon[97044]: pgmap v14945: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:16:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:17.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:17.215909+0000) 2022-09-10T12:16:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:17.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:17.743207+0000) 2022-09-10T12:16:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:18.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:18.216136+0000) 2022-09-10T12:16:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:18.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:18.743420+0000) 2022-09-10T12:16:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:18 smithi110 ceph-mon[97633]: pgmap v14946: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:16:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:18 smithi104 ceph-mon[97044]: pgmap v14946: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:16:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:19.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:19.216294+0000) 2022-09-10T12:16:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:19.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:19.743620+0000) 2022-09-10T12:16:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:20.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:20.216494+0000) 2022-09-10T12:16:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:20.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:20.743800+0000) 2022-09-10T12:16:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:20 smithi110 ceph-mon[97633]: pgmap v14947: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:16:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:20 smithi104 ceph-mon[97044]: pgmap v14947: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:16:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:21.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:21.216739+0000) 2022-09-10T12:16:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:21.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:21.743962+0000) 2022-09-10T12:16:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:22.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:22.217000+0000) 2022-09-10T12:16:23.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:22.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:22.744183+0000) 2022-09-10T12:16:23.255 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:22 smithi110 ceph-mon[97633]: pgmap v14948: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:16:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:22 smithi104 ceph-mon[97044]: pgmap v14948: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:16:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:23.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:23.217231+0000) 2022-09-10T12:16:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:23.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:23.744387+0000) 2022-09-10T12:16:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:24.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:24.217498+0000) 2022-09-10T12:16:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:24.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:24.744576+0000) 2022-09-10T12:16:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:25.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:25.217682+0000) 2022-09-10T12:16:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:25 smithi104 ceph-mon[97044]: pgmap v14949: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:16:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:25 smithi110 ceph-mon[97633]: pgmap v14949: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:16:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:25.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:25.744725+0000) 2022-09-10T12:16:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:26.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:26.217858+0000) 2022-09-10T12:16:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:26.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:26.744978+0000) 2022-09-10T12:16:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:26 smithi110 ceph-mon[97633]: pgmap v14950: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:26 smithi104 ceph-mon[97044]: pgmap v14950: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:27.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:27.218054+0000) 2022-09-10T12:16:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:27.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:27.745256+0000) 2022-09-10T12:16:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:28.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:28.218142+0000) 2022-09-10T12:16:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:28.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:28.745497+0000) 2022-09-10T12:16:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:28 smithi110 ceph-mon[97633]: pgmap v14951: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:28 smithi104 ceph-mon[97044]: pgmap v14951: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:29.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:16:29.285+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:16:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:29.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:29.218373+0000) 2022-09-10T12:16:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:29.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:29.745755+0000) 2022-09-10T12:16:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:30.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:30.218552+0000) 2022-09-10T12:16:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:30.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:30.745966+0000) 2022-09-10T12:16:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:30 smithi110 ceph-mon[97633]: pgmap v14952: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:30 smithi104 ceph-mon[97044]: pgmap v14952: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:31.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:31.218737+0000) 2022-09-10T12:16:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:31.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:31.746227+0000) 2022-09-10T12:16:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:32.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:32.218974+0000) 2022-09-10T12:16:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:32.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:32.746377+0000) 2022-09-10T12:16:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:32 smithi110 ceph-mon[97633]: pgmap v14953: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:32 smithi104 ceph-mon[97044]: pgmap v14953: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:33.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:33.219217+0000) 2022-09-10T12:16:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:33.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:33.746570+0000) 2022-09-10T12:16:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:34.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:34.219460+0000) 2022-09-10T12:16:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:34.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:34.746801+0000) 2022-09-10T12:16:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:35.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:35.219683+0000) 2022-09-10T12:16:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:35 smithi104 ceph-mon[97044]: pgmap v14954: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:35 smithi110 ceph-mon[97633]: pgmap v14954: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:35.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:35.747030+0000) 2022-09-10T12:16:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:36.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:36.219874+0000) 2022-09-10T12:16:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:36.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:36.747305+0000) 2022-09-10T12:16:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:36 smithi110 ceph-mon[97633]: pgmap v14955: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:36 smithi104 ceph-mon[97044]: pgmap v14955: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:37.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:37.220079+0000) 2022-09-10T12:16:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:37.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:37.747458+0000) 2022-09-10T12:16:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:38.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:38.220320+0000) 2022-09-10T12:16:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:38.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:38.747686+0000) 2022-09-10T12:16:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:38 smithi110 ceph-mon[97633]: pgmap v14956: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:38 smithi104 ceph-mon[97044]: pgmap v14956: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:39.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:39.220529+0000) 2022-09-10T12:16:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:39.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:39.747953+0000) 2022-09-10T12:16:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:40.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:40.220676+0000) 2022-09-10T12:16:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:40.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:40.748129+0000) 2022-09-10T12:16:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:40 smithi110 ceph-mon[97633]: pgmap v14957: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:40 smithi104 ceph-mon[97044]: pgmap v14957: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:41.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:41.220902+0000) 2022-09-10T12:16:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:41.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:41.748307+0000) 2022-09-10T12:16:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:42.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:42.221198+0000) 2022-09-10T12:16:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:42.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:42.748512+0000) 2022-09-10T12:16:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:42 smithi110 ceph-mon[97633]: pgmap v14958: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:43.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:42 smithi104 ceph-mon[97044]: pgmap v14958: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:43.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:43.221442+0000) 2022-09-10T12:16:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:43.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:43.748734+0000) 2022-09-10T12:16:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:44.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:44.221658+0000) 2022-09-10T12:16:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:16:44.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:16:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:44.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:44.749012+0000) 2022-09-10T12:16:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:45.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:45.221876+0000) 2022-09-10T12:16:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:45 smithi104 ceph-mon[97044]: pgmap v14959: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:45 smithi110 ceph-mon[97633]: pgmap v14959: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:45.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:45.749257+0000) 2022-09-10T12:16:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:46.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:46.222036+0000) 2022-09-10T12:16:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:16:46.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:16:46.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:16:46.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:16:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:16:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:16:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:16:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:16:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:46.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:46.749441+0000) 2022-09-10T12:16:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:47.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:47.222205+0000) 2022-09-10T12:16:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:47 smithi104 ceph-mon[97044]: pgmap v14960: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:16:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:47 smithi110 ceph-mon[97633]: pgmap v14960: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:16:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:47.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:47.749637+0000) 2022-09-10T12:16:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:48.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:48.222358+0000) 2022-09-10T12:16:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:48.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:48.749822+0000) 2022-09-10T12:16:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:48 smithi110 ceph-mon[97633]: pgmap v14961: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:16:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:48 smithi104 ceph-mon[97044]: pgmap v14961: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:16:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:49.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:49.222556+0000) 2022-09-10T12:16:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:49.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:49.750036+0000) 2022-09-10T12:16:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:50.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:50.222802+0000) 2022-09-10T12:16:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:50.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:50.750255+0000) 2022-09-10T12:16:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:50 smithi110 ceph-mon[97633]: pgmap v14962: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:50 smithi104 ceph-mon[97044]: pgmap v14962: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:51.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:51.223047+0000) 2022-09-10T12:16:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:51.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:51.750548+0000) 2022-09-10T12:16:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:52.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:52.223275+0000) 2022-09-10T12:16:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:52.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:52.750789+0000) 2022-09-10T12:16:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:52 smithi110 ceph-mon[97633]: pgmap v14963: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:16:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:52 smithi104 ceph-mon[97044]: pgmap v14963: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:16:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:53.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:53.223505+0000) 2022-09-10T12:16:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:53.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:53.751011+0000) 2022-09-10T12:16:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:54.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:54.223752+0000) 2022-09-10T12:16:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:54.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:54.751172+0000) 2022-09-10T12:16:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:55.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:55.223962+0000) 2022-09-10T12:16:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:55 smithi104 ceph-mon[97044]: pgmap v14964: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:55 smithi110 ceph-mon[97633]: pgmap v14964: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:16:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:55.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:55.751414+0000) 2022-09-10T12:16:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:56.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:56.224187+0000) 2022-09-10T12:16:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:56.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:56.751600+0000) 2022-09-10T12:16:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:56 smithi110 ceph-mon[97633]: pgmap v14965: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:56 smithi104 ceph-mon[97044]: pgmap v14965: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:57.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:57.224383+0000) 2022-09-10T12:16:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:57.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:57.751806+0000) 2022-09-10T12:16:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:58.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:58.224573+0000) 2022-09-10T12:16:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:58.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:58.752021+0000) 2022-09-10T12:16:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:16:58 smithi110 ceph-mon[97633]: pgmap v14966: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:16:58 smithi104 ceph-mon[97044]: pgmap v14966: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:16:59.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:16:59.308+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:16:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:16:59.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:59.224838+0000) 2022-09-10T12:17:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:16:59.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:16:59.752165+0000) 2022-09-10T12:17:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:00.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:00.225003+0000) 2022-09-10T12:17:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:00.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:00.752376+0000) 2022-09-10T12:17:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:00 smithi110 ceph-mon[97633]: pgmap v14967: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:00 smithi104 ceph-mon[97044]: pgmap v14967: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:01.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:01.225179+0000) 2022-09-10T12:17:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:01.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:01.752606+0000) 2022-09-10T12:17:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:02.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:02.225345+0000) 2022-09-10T12:17:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:02.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:02.752845+0000) 2022-09-10T12:17:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:02 smithi110 ceph-mon[97633]: pgmap v14968: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:02 smithi104 ceph-mon[97044]: pgmap v14968: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:03.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:03.225620+0000) 2022-09-10T12:17:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:03.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:03.753018+0000) 2022-09-10T12:17:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:04.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:04.225796+0000) 2022-09-10T12:17:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:04.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:04.753224+0000) 2022-09-10T12:17:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:05.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:05.226000+0000) 2022-09-10T12:17:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:05 smithi104 ceph-mon[97044]: pgmap v14969: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:17:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:05 smithi110 ceph-mon[97633]: pgmap v14969: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:17:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:05.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:05.753470+0000) 2022-09-10T12:17:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:06.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:06.226180+0000) 2022-09-10T12:17:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:06.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:06.753716+0000) 2022-09-10T12:17:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:06 smithi110 ceph-mon[97633]: pgmap v14970: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:17:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:06 smithi104 ceph-mon[97044]: pgmap v14970: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:17:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:07.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:07.226409+0000) 2022-09-10T12:17:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:07.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:07.753910+0000) 2022-09-10T12:17:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:08.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:08.226628+0000) 2022-09-10T12:17:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:08.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:08.754058+0000) 2022-09-10T12:17:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:08 smithi110 ceph-mon[97633]: osdmap e1016: 12 total, 12 up, 12 in 2022-09-10T12:17:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:08 smithi110 ceph-mon[97633]: pgmap v14972: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:17:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:08 smithi104 ceph-mon[97044]: osdmap e1016: 12 total, 12 up, 12 in 2022-09-10T12:17:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:08 smithi104 ceph-mon[97044]: pgmap v14972: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:17:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:09.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:09.226845+0000) 2022-09-10T12:17:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:09.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:09.754341+0000) 2022-09-10T12:17:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:10.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:10.227030+0000) 2022-09-10T12:17:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:10.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:10.754593+0000) 2022-09-10T12:17:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:10 smithi110 ceph-mon[97633]: pgmap v14973: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:17:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:10 smithi104 ceph-mon[97044]: pgmap v14973: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:17:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:11.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:11.227277+0000) 2022-09-10T12:17:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:11.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:11.754782+0000) 2022-09-10T12:17:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:12.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:12.227547+0000) 2022-09-10T12:17:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:12.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:12.755023+0000) 2022-09-10T12:17:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:12 smithi110 ceph-mon[97633]: pgmap v14974: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:17:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:12 smithi104 ceph-mon[97044]: pgmap v14974: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:17:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:13.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:13.227797+0000) 2022-09-10T12:17:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:13.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:13.755229+0000) 2022-09-10T12:17:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:14.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:14.227997+0000) 2022-09-10T12:17:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:17:14.322+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:17:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:14.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:14.755435+0000) 2022-09-10T12:17:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:15.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:15.228219+0000) 2022-09-10T12:17:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:15 smithi104 ceph-mon[97044]: osdmap e1017: 12 total, 12 up, 12 in 2022-09-10T12:17:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:15 smithi104 ceph-mon[97044]: pgmap v14976: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:17:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:15 smithi110 ceph-mon[97633]: osdmap e1017: 12 total, 12 up, 12 in 2022-09-10T12:17:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:15 smithi110 ceph-mon[97633]: pgmap v14976: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:17:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:15.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:15.755627+0000) 2022-09-10T12:17:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:16.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:16.228454+0000) 2022-09-10T12:17:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:16.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:16.755887+0000) 2022-09-10T12:17:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:16 smithi110 ceph-mon[97633]: pgmap v14977: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:17:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:16 smithi104 ceph-mon[97044]: pgmap v14977: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:17:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:17.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:17.228651+0000) 2022-09-10T12:17:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:17.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:17.756155+0000) 2022-09-10T12:17:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:18.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:18.228880+0000) 2022-09-10T12:17:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:18.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:18.756332+0000) 2022-09-10T12:17:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:18 smithi110 ceph-mon[97633]: pgmap v14978: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:17:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:18 smithi104 ceph-mon[97044]: pgmap v14978: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:17:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:19.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:19.229185+0000) 2022-09-10T12:17:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:19.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:19.756533+0000) 2022-09-10T12:17:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:20.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:20.229414+0000) 2022-09-10T12:17:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:20.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:20.756814+0000) 2022-09-10T12:17:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:20 smithi110 ceph-mon[97633]: pgmap v14979: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:17:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:20 smithi104 ceph-mon[97044]: pgmap v14979: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:17:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:21.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:21.229623+0000) 2022-09-10T12:17:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:21.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:21.756995+0000) 2022-09-10T12:17:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:22.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:22.229845+0000) 2022-09-10T12:17:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:22.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:22.757226+0000) 2022-09-10T12:17:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:22 smithi110 ceph-mon[97633]: pgmap v14980: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:17:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:22 smithi104 ceph-mon[97044]: pgmap v14980: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:17:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:23.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:23.230168+0000) 2022-09-10T12:17:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:23.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:23.757394+0000) 2022-09-10T12:17:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:24.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:24.230383+0000) 2022-09-10T12:17:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:24.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:24.757641+0000) 2022-09-10T12:17:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:25.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:25.230567+0000) 2022-09-10T12:17:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:25 smithi104 ceph-mon[97044]: pgmap v14981: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:17:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:25 smithi110 ceph-mon[97633]: pgmap v14981: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:17:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:25.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:25.757831+0000) 2022-09-10T12:17:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:26.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:26.230888+0000) 2022-09-10T12:17:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:26.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:26.757996+0000) 2022-09-10T12:17:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:26 smithi110 ceph-mon[97633]: pgmap v14982: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:26 smithi104 ceph-mon[97044]: pgmap v14982: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:27.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:27.231052+0000) 2022-09-10T12:17:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:27.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:27.758127+0000) 2022-09-10T12:17:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:28.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:28.231229+0000) 2022-09-10T12:17:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:28.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:28.758307+0000) 2022-09-10T12:17:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:28 smithi110 ceph-mon[97633]: pgmap v14983: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:28 smithi104 ceph-mon[97044]: pgmap v14983: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:29.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:17:29.333+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:17:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:29.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:29.231496+0000) 2022-09-10T12:17:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:29.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:29.758469+0000) 2022-09-10T12:17:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:30.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:30.231731+0000) 2022-09-10T12:17:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:30.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:30.758643+0000) 2022-09-10T12:17:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:30 smithi110 ceph-mon[97633]: pgmap v14984: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:30 smithi104 ceph-mon[97044]: pgmap v14984: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:31.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:31.231976+0000) 2022-09-10T12:17:32.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:31.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:31.758843+0000) 2022-09-10T12:17:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:32.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:32.232168+0000) 2022-09-10T12:17:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:32.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:32.759049+0000) 2022-09-10T12:17:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:32 smithi110 ceph-mon[97633]: pgmap v14985: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:32 smithi104 ceph-mon[97044]: pgmap v14985: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:33.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:33.232429+0000) 2022-09-10T12:17:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:33.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:33.759330+0000) 2022-09-10T12:17:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:34.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:34.232580+0000) 2022-09-10T12:17:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:34.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:34.759555+0000) 2022-09-10T12:17:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:35.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:35.232820+0000) 2022-09-10T12:17:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:35 smithi104 ceph-mon[97044]: pgmap v14986: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:35 smithi110 ceph-mon[97633]: pgmap v14986: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:35.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:35.759744+0000) 2022-09-10T12:17:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:36.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:36.233044+0000) 2022-09-10T12:17:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:36.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:36.759984+0000) 2022-09-10T12:17:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:36 smithi110 ceph-mon[97633]: pgmap v14987: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:36 smithi104 ceph-mon[97044]: pgmap v14987: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:37.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:37.233277+0000) 2022-09-10T12:17:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:37.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:37.760119+0000) 2022-09-10T12:17:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:38.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:38.233498+0000) 2022-09-10T12:17:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:38.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:38.760275+0000) 2022-09-10T12:17:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:38 smithi110 ceph-mon[97633]: pgmap v14988: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:38 smithi104 ceph-mon[97044]: pgmap v14988: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:39.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:39.233700+0000) 2022-09-10T12:17:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:39.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:39.760458+0000) 2022-09-10T12:17:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:40.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:40.233929+0000) 2022-09-10T12:17:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:40.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:40.760695+0000) 2022-09-10T12:17:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:40 smithi110 ceph-mon[97633]: pgmap v14989: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:40 smithi104 ceph-mon[97044]: pgmap v14989: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:41.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:41.234146+0000) 2022-09-10T12:17:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:41.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:41.760863+0000) 2022-09-10T12:17:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:42.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:42.234293+0000) 2022-09-10T12:17:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:42.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:42.761112+0000) 2022-09-10T12:17:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:42 smithi110 ceph-mon[97633]: pgmap v14990: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:42 smithi104 ceph-mon[97044]: pgmap v14990: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:43.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:43.234539+0000) 2022-09-10T12:17:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:43.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:43.761320+0000) 2022-09-10T12:17:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:44.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:44.234788+0000) 2022-09-10T12:17:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:17:44.346+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:17:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:44.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:44.761514+0000) 2022-09-10T12:17:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:45.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:45.235048+0000) 2022-09-10T12:17:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:45 smithi104 ceph-mon[97044]: pgmap v14991: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:45 smithi110 ceph-mon[97633]: pgmap v14991: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:45.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:45.761728+0000) 2022-09-10T12:17:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:46.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:46.235392+0000) 2022-09-10T12:17:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:17:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:17:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:17:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:17:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:17:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:46.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:46.761991+0000) 2022-09-10T12:17:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:47.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:47.235587+0000) 2022-09-10T12:17:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:47 smithi104 ceph-mon[97044]: pgmap v14992: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:47 smithi110 ceph-mon[97633]: pgmap v14992: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:47.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:47.762191+0000) 2022-09-10T12:17:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:48.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:48.235816+0000) 2022-09-10T12:17:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:48.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:48.762359+0000) 2022-09-10T12:17:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:48 smithi110 ceph-mon[97633]: pgmap v14993: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:48 smithi104 ceph-mon[97044]: pgmap v14993: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:49.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:49.236057+0000) 2022-09-10T12:17:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:49.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:49.762526+0000) 2022-09-10T12:17:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:50.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:50.236274+0000) 2022-09-10T12:17:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:50.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:50.762781+0000) 2022-09-10T12:17:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:50 smithi110 ceph-mon[97633]: pgmap v14994: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:50 smithi104 ceph-mon[97044]: pgmap v14994: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:51.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:51.236448+0000) 2022-09-10T12:17:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:51.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:51.762981+0000) 2022-09-10T12:17:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:52.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:52.236584+0000) 2022-09-10T12:17:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:52.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:52.763232+0000) 2022-09-10T12:17:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:52 smithi110 ceph-mon[97633]: pgmap v14995: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:52 smithi104 ceph-mon[97044]: pgmap v14995: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:53.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:53.236805+0000) 2022-09-10T12:17:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:53.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:53.763482+0000) 2022-09-10T12:17:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:54.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:54.236983+0000) 2022-09-10T12:17:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:54.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:54.763749+0000) 2022-09-10T12:17:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:55.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:55.237217+0000) 2022-09-10T12:17:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:55 smithi104 ceph-mon[97044]: pgmap v14996: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:55.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:55 smithi110 ceph-mon[97633]: pgmap v14996: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:17:56.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:55.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:55.763934+0000) 2022-09-10T12:17:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:56.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:56.237487+0000) 2022-09-10T12:17:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:56.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:56.764185+0000) 2022-09-10T12:17:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:56 smithi110 ceph-mon[97633]: pgmap v14997: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:56 smithi104 ceph-mon[97044]: pgmap v14997: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:57.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:57.237724+0000) 2022-09-10T12:17:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:57.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:57.764403+0000) 2022-09-10T12:17:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:58.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:58.237885+0000) 2022-09-10T12:17:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:58.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:58.764623+0000) 2022-09-10T12:17:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:17:58 smithi110 ceph-mon[97633]: pgmap v14998: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:17:58 smithi104 ceph-mon[97044]: pgmap v14998: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:17:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:17:59.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:59.238127+0000) 2022-09-10T12:17:59.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:17:59.355+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:18:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:17:59.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:17:59.764818+0000) 2022-09-10T12:18:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:00.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:00.238278+0000) 2022-09-10T12:18:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:00.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:00.764970+0000) 2022-09-10T12:18:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:00 smithi110 ceph-mon[97633]: pgmap v14999: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:00 smithi104 ceph-mon[97044]: pgmap v14999: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:01.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:01.238495+0000) 2022-09-10T12:18:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:01.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:01.765225+0000) 2022-09-10T12:18:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:02.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:02.238663+0000) 2022-09-10T12:18:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:02.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:02.765501+0000) 2022-09-10T12:18:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:02 smithi110 ceph-mon[97633]: pgmap v15000: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:02 smithi104 ceph-mon[97044]: pgmap v15000: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:03.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:03.238900+0000) 2022-09-10T12:18:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:03.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:03.765666+0000) 2022-09-10T12:18:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:04.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:04.239122+0000) 2022-09-10T12:18:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:04.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:04.765871+0000) 2022-09-10T12:18:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:05.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:05.239321+0000) 2022-09-10T12:18:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:05 smithi104 ceph-mon[97044]: pgmap v15001: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:18:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:05 smithi110 ceph-mon[97633]: pgmap v15001: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:18:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:05.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:05.766145+0000) 2022-09-10T12:18:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:06.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:06.239513+0000) 2022-09-10T12:18:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:06.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:06.766399+0000) 2022-09-10T12:18:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:06 smithi110 ceph-mon[97633]: pgmap v15002: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:18:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:06 smithi104 ceph-mon[97044]: pgmap v15002: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:18:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:07.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:07.239740+0000) 2022-09-10T12:18:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:07.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:07.766621+0000) 2022-09-10T12:18:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:08.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:08.239950+0000) 2022-09-10T12:18:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:08.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:08.766818+0000) 2022-09-10T12:18:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:08 smithi110 ceph-mon[97633]: osdmap e1018: 12 total, 12 up, 12 in 2022-09-10T12:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:08 smithi110 ceph-mon[97633]: pgmap v15004: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:18:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:08 smithi104 ceph-mon[97044]: osdmap e1018: 12 total, 12 up, 12 in 2022-09-10T12:18:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:08 smithi104 ceph-mon[97044]: pgmap v15004: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:18:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:09.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:09.240146+0000) 2022-09-10T12:18:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:09.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:09.767028+0000) 2022-09-10T12:18:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:10.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:10.240366+0000) 2022-09-10T12:18:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:10.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:10.767234+0000) 2022-09-10T12:18:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:10 smithi110 ceph-mon[97633]: pgmap v15005: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:18:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:10 smithi104 ceph-mon[97044]: pgmap v15005: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:18:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:11.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:11.240643+0000) 2022-09-10T12:18:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:11.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:11.767498+0000) 2022-09-10T12:18:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:12.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:12.240870+0000) 2022-09-10T12:18:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:12.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:12.767712+0000) 2022-09-10T12:18:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:12 smithi110 ceph-mon[97633]: pgmap v15006: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:18:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:12 smithi104 ceph-mon[97044]: pgmap v15006: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:18:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:13.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:13.241133+0000) 2022-09-10T12:18:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:13.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:13.767904+0000) 2022-09-10T12:18:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:14.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:14.241364+0000) 2022-09-10T12:18:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:18:14.364+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:18:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:14.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:14.768072+0000) 2022-09-10T12:18:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:15.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:15.241618+0000) 2022-09-10T12:18:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:15 smithi104 ceph-mon[97044]: osdmap e1019: 12 total, 12 up, 12 in 2022-09-10T12:18:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:15 smithi104 ceph-mon[97044]: pgmap v15008: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:18:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:15 smithi110 ceph-mon[97633]: osdmap e1019: 12 total, 12 up, 12 in 2022-09-10T12:18:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:15 smithi110 ceph-mon[97633]: pgmap v15008: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:18:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:15.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:15.768274+0000) 2022-09-10T12:18:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:16.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:16.241863+0000) 2022-09-10T12:18:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:16.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:16.768420+0000) 2022-09-10T12:18:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:16 smithi110 ceph-mon[97633]: pgmap v15009: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:18:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:16 smithi104 ceph-mon[97044]: pgmap v15009: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:18:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:17.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:17.242124+0000) 2022-09-10T12:18:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:17.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:17.768619+0000) 2022-09-10T12:18:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:18.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:18.242362+0000) 2022-09-10T12:18:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:18.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:18.768837+0000) 2022-09-10T12:18:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:18 smithi110 ceph-mon[97633]: pgmap v15010: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:18:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:18 smithi104 ceph-mon[97044]: pgmap v15010: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:18:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:19.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:19.242603+0000) 2022-09-10T12:18:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:19.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:19.768978+0000) 2022-09-10T12:18:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:20.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:20.242789+0000) 2022-09-10T12:18:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:20.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:20.769132+0000) 2022-09-10T12:18:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:20 smithi110 ceph-mon[97633]: pgmap v15011: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:18:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:20 smithi104 ceph-mon[97044]: pgmap v15011: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:18:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:21.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:21.242999+0000) 2022-09-10T12:18:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:21.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:21.769295+0000) 2022-09-10T12:18:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:22.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:22.243175+0000) 2022-09-10T12:18:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:22.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:22.769456+0000) 2022-09-10T12:18:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:22 smithi110 ceph-mon[97633]: pgmap v15012: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:18:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:22 smithi104 ceph-mon[97044]: pgmap v15012: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:18:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:23.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:23.243383+0000) 2022-09-10T12:18:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:23.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:23.769620+0000) 2022-09-10T12:18:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:24.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:24.243659+0000) 2022-09-10T12:18:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:24.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:24.769784+0000) 2022-09-10T12:18:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:25.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:25.243887+0000) 2022-09-10T12:18:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:25 smithi104 ceph-mon[97044]: pgmap v15013: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:18:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:25 smithi110 ceph-mon[97633]: pgmap v15013: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:18:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:25.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:25.769896+0000) 2022-09-10T12:18:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:26.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:26.244166+0000) 2022-09-10T12:18:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:26.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:26.770184+0000) 2022-09-10T12:18:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:26 smithi110 ceph-mon[97633]: pgmap v15014: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:26 smithi104 ceph-mon[97044]: pgmap v15014: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:27.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:27.244387+0000) 2022-09-10T12:18:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:27.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:27.770338+0000) 2022-09-10T12:18:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:28.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:28.244623+0000) 2022-09-10T12:18:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:28.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:28.770509+0000) 2022-09-10T12:18:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:28 smithi110 ceph-mon[97633]: pgmap v15015: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:28 smithi104 ceph-mon[97044]: pgmap v15015: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:29.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:18:29.369+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:18:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:29.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:29.244911+0000) 2022-09-10T12:18:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:29.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:29.770679+0000) 2022-09-10T12:18:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:30.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:30.245183+0000) 2022-09-10T12:18:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:30.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:30.770853+0000) 2022-09-10T12:18:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:30 smithi110 ceph-mon[97633]: pgmap v15016: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:30 smithi104 ceph-mon[97044]: pgmap v15016: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:31.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:31.245423+0000) 2022-09-10T12:18:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:31.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:31.771131+0000) 2022-09-10T12:18:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:32.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:32.245674+0000) 2022-09-10T12:18:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:32.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:32.771271+0000) 2022-09-10T12:18:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:32 smithi110 ceph-mon[97633]: pgmap v15017: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:32 smithi104 ceph-mon[97044]: pgmap v15017: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:33.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:33.245901+0000) 2022-09-10T12:18:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:33.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:33.771496+0000) 2022-09-10T12:18:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:34.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:34.246073+0000) 2022-09-10T12:18:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:34.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:34.771719+0000) 2022-09-10T12:18:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:35.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:35.246293+0000) 2022-09-10T12:18:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:35 smithi104 ceph-mon[97044]: pgmap v15018: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:35 smithi110 ceph-mon[97633]: pgmap v15018: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:35.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:35.771931+0000) 2022-09-10T12:18:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:36.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:36.246456+0000) 2022-09-10T12:18:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:36.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:36.772180+0000) 2022-09-10T12:18:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:36 smithi110 ceph-mon[97633]: pgmap v15019: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:36 smithi104 ceph-mon[97044]: pgmap v15019: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:37.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:37.246687+0000) 2022-09-10T12:18:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:37.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:37.772375+0000) 2022-09-10T12:18:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:38.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:38.246838+0000) 2022-09-10T12:18:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:38.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:38.772558+0000) 2022-09-10T12:18:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:38 smithi110 ceph-mon[97633]: pgmap v15020: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:38 smithi104 ceph-mon[97044]: pgmap v15020: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:39.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:39.247019+0000) 2022-09-10T12:18:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:39.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:39.772789+0000) 2022-09-10T12:18:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:40.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:40.247255+0000) 2022-09-10T12:18:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:40.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:40.772957+0000) 2022-09-10T12:18:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:40 smithi110 ceph-mon[97633]: pgmap v15021: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:40 smithi104 ceph-mon[97044]: pgmap v15021: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:41.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:41.247504+0000) 2022-09-10T12:18:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:41.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:41.773212+0000) 2022-09-10T12:18:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:42.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:42.247750+0000) 2022-09-10T12:18:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:42.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:42.773382+0000) 2022-09-10T12:18:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:42 smithi110 ceph-mon[97633]: pgmap v15022: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:42 smithi104 ceph-mon[97044]: pgmap v15022: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:43.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:43.248000+0000) 2022-09-10T12:18:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:43.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:43.773598+0000) 2022-09-10T12:18:44.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:18:44.376+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:18:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:44.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:44.248218+0000) 2022-09-10T12:18:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:44.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:44.773852+0000) 2022-09-10T12:18:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:45.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:45.248496+0000) 2022-09-10T12:18:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:45 smithi104 ceph-mon[97044]: pgmap v15023: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:45 smithi110 ceph-mon[97633]: pgmap v15023: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:45.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:45.773983+0000) 2022-09-10T12:18:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:46.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:46.248738+0000) 2022-09-10T12:18:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:18:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:18:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:18:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:18:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:18:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:18:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:18:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:18:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:46.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:46.774212+0000) 2022-09-10T12:18:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:47.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:47.248997+0000) 2022-09-10T12:18:47.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:47 smithi104 ceph-mon[97044]: pgmap v15024: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:47 smithi110 ceph-mon[97633]: pgmap v15024: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:47.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:47.774422+0000) 2022-09-10T12:18:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:48.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:48.249226+0000) 2022-09-10T12:18:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:48.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:48.774620+0000) 2022-09-10T12:18:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:48 smithi110 ceph-mon[97633]: pgmap v15025: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:48 smithi104 ceph-mon[97044]: pgmap v15025: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:49.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:49.249550+0000) 2022-09-10T12:18:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:49.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:49.774868+0000) 2022-09-10T12:18:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:50.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:50.249723+0000) 2022-09-10T12:18:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:50.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:50.775168+0000) 2022-09-10T12:18:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:50 smithi110 ceph-mon[97633]: pgmap v15026: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:50 smithi104 ceph-mon[97044]: pgmap v15026: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:51.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:51.249930+0000) 2022-09-10T12:18:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:51.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:51.775416+0000) 2022-09-10T12:18:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:52.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:52.250153+0000) 2022-09-10T12:18:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:52.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:52.775657+0000) 2022-09-10T12:18:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:52 smithi110 ceph-mon[97633]: pgmap v15027: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:52 smithi104 ceph-mon[97044]: pgmap v15027: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:53.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:53.250392+0000) 2022-09-10T12:18:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:53.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:53.775888+0000) 2022-09-10T12:18:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:54.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:54.250532+0000) 2022-09-10T12:18:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:54.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:54.776156+0000) 2022-09-10T12:18:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:55.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:55.250718+0000) 2022-09-10T12:18:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:55 smithi104 ceph-mon[97044]: pgmap v15028: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:55 smithi110 ceph-mon[97633]: pgmap v15028: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:18:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:55.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:55.776348+0000) 2022-09-10T12:18:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:56.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:56.250920+0000) 2022-09-10T12:18:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:56.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:56.776493+0000) 2022-09-10T12:18:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:56 smithi110 ceph-mon[97633]: pgmap v15029: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:56 smithi104 ceph-mon[97044]: pgmap v15029: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:57.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:57.251091+0000) 2022-09-10T12:18:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:57.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:57.776703+0000) 2022-09-10T12:18:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:58.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:58.251263+0000) 2022-09-10T12:18:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:58.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:58.776954+0000) 2022-09-10T12:18:59.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:18:58 smithi110 ceph-mon[97633]: pgmap v15030: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:18:58 smithi104 ceph-mon[97044]: pgmap v15030: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:18:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:18:59.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:59.251412+0000) 2022-09-10T12:18:59.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:18:59.391+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:19:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:18:59.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:18:59.777203+0000) 2022-09-10T12:19:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:00.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:00.251561+0000) 2022-09-10T12:19:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:00.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:00.777416+0000) 2022-09-10T12:19:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:00 smithi110 ceph-mon[97633]: pgmap v15031: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:00 smithi104 ceph-mon[97044]: pgmap v15031: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:01.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:01.251719+0000) 2022-09-10T12:19:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:01.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:01.777627+0000) 2022-09-10T12:19:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:02.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:02.251920+0000) 2022-09-10T12:19:03.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:02.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:02.777864+0000) 2022-09-10T12:19:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:02 smithi110 ceph-mon[97633]: pgmap v15032: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:02 smithi104 ceph-mon[97044]: pgmap v15032: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:03.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:03.252162+0000) 2022-09-10T12:19:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:03.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:03.778059+0000) 2022-09-10T12:19:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:04.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:04.252356+0000) 2022-09-10T12:19:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:04.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:04.778282+0000) 2022-09-10T12:19:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:05.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:05.252500+0000) 2022-09-10T12:19:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:05 smithi104 ceph-mon[97044]: pgmap v15033: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:19:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:05 smithi110 ceph-mon[97633]: pgmap v15033: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:19:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:05.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:05.778490+0000) 2022-09-10T12:19:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:06.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:06.252725+0000) 2022-09-10T12:19:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:06.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:06.778743+0000) 2022-09-10T12:19:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:06 smithi110 ceph-mon[97633]: pgmap v15034: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:19:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:06 smithi104 ceph-mon[97044]: pgmap v15034: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:19:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:07.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:07.252960+0000) 2022-09-10T12:19:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:07.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:07.779013+0000) 2022-09-10T12:19:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:08.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:08.253182+0000) 2022-09-10T12:19:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:08.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:08.779265+0000) 2022-09-10T12:19:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:08 smithi110 ceph-mon[97633]: osdmap e1020: 12 total, 12 up, 12 in 2022-09-10T12:19:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:08 smithi110 ceph-mon[97633]: pgmap v15036: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:19:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:08 smithi104 ceph-mon[97044]: osdmap e1020: 12 total, 12 up, 12 in 2022-09-10T12:19:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:08 smithi104 ceph-mon[97044]: pgmap v15036: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:19:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:09.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:09.253389+0000) 2022-09-10T12:19:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:09.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:09.779510+0000) 2022-09-10T12:19:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:10.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:10.253667+0000) 2022-09-10T12:19:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:10.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:10.779664+0000) 2022-09-10T12:19:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:10 smithi110 ceph-mon[97633]: pgmap v15037: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:19:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:10 smithi104 ceph-mon[97044]: pgmap v15037: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:19:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:11.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:11.253902+0000) 2022-09-10T12:19:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:11.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:11.779866+0000) 2022-09-10T12:19:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:12.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:12.254127+0000) 2022-09-10T12:19:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:12.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:12.780139+0000) 2022-09-10T12:19:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:12 smithi110 ceph-mon[97633]: pgmap v15038: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:19:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:12 smithi104 ceph-mon[97044]: pgmap v15038: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:19:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:13.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:13.254294+0000) 2022-09-10T12:19:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:13.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:13.780337+0000) 2022-09-10T12:19:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:14.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:14.254467+0000) 2022-09-10T12:19:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:19:14.400+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:19:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:14.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:14.780475+0000) 2022-09-10T12:19:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:15.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:15.254658+0000) 2022-09-10T12:19:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:15 smithi104 ceph-mon[97044]: osdmap e1021: 12 total, 12 up, 12 in 2022-09-10T12:19:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:15 smithi104 ceph-mon[97044]: pgmap v15040: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:19:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:15 smithi110 ceph-mon[97633]: osdmap e1021: 12 total, 12 up, 12 in 2022-09-10T12:19:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:15 smithi110 ceph-mon[97633]: pgmap v15040: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:19:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:15.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:15.780646+0000) 2022-09-10T12:19:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:16.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:16.254882+0000) 2022-09-10T12:19:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:16.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:16.780827+0000) 2022-09-10T12:19:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:16 smithi110 ceph-mon[97633]: pgmap v15041: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:19:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:16 smithi104 ceph-mon[97044]: pgmap v15041: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:19:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:17.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:17.255155+0000) 2022-09-10T12:19:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:17.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:17.781057+0000) 2022-09-10T12:19:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:18.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:18.255373+0000) 2022-09-10T12:19:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:18.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:18.781291+0000) 2022-09-10T12:19:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:18 smithi110 ceph-mon[97633]: pgmap v15042: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:19:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:18 smithi104 ceph-mon[97044]: pgmap v15042: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:19:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:19.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:19.255560+0000) 2022-09-10T12:19:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:19.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:19.781432+0000) 2022-09-10T12:19:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:20.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:20.255778+0000) 2022-09-10T12:19:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:20.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:20.781570+0000) 2022-09-10T12:19:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:20 smithi110 ceph-mon[97633]: pgmap v15043: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:19:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:20 smithi104 ceph-mon[97044]: pgmap v15043: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:19:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:21.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:21.255960+0000) 2022-09-10T12:19:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:21.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:21.781802+0000) 2022-09-10T12:19:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:22.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:22.256194+0000) 2022-09-10T12:19:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:22.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:22.782051+0000) 2022-09-10T12:19:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:22 smithi110 ceph-mon[97633]: pgmap v15044: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:19:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:22 smithi104 ceph-mon[97044]: pgmap v15044: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:19:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:23.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:23.256361+0000) 2022-09-10T12:19:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:23.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:23.782306+0000) 2022-09-10T12:19:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:24.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:24.256582+0000) 2022-09-10T12:19:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:24.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:24.782480+0000) 2022-09-10T12:19:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:25.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:25.256819+0000) 2022-09-10T12:19:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:25 smithi110 ceph-mon[97633]: pgmap v15045: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:19:26.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:25 smithi104 ceph-mon[97044]: pgmap v15045: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:19:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:25.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:25.782611+0000) 2022-09-10T12:19:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:26.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:26.257009+0000) 2022-09-10T12:19:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:26.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:26.782837+0000) 2022-09-10T12:19:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:26 smithi110 ceph-mon[97633]: pgmap v15046: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:26 smithi104 ceph-mon[97044]: pgmap v15046: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:27.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:27.257202+0000) 2022-09-10T12:19:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:27.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:27.783082+0000) 2022-09-10T12:19:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:28.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:28.257468+0000) 2022-09-10T12:19:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:28.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:28.783352+0000) 2022-09-10T12:19:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:28 smithi110 ceph-mon[97633]: pgmap v15047: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:28 smithi104 ceph-mon[97044]: pgmap v15047: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:29.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:29.257653+0000) 2022-09-10T12:19:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:19:29.414+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:19:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:29.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:29.783623+0000) 2022-09-10T12:19:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:30.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:30.257862+0000) 2022-09-10T12:19:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:30.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:30.783832+0000) 2022-09-10T12:19:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:30 smithi110 ceph-mon[97633]: pgmap v15048: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:30 smithi104 ceph-mon[97044]: pgmap v15048: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:31.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:31.258068+0000) 2022-09-10T12:19:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:31.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:31.784053+0000) 2022-09-10T12:19:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:32.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:32.258355+0000) 2022-09-10T12:19:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:32.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:32.784203+0000) 2022-09-10T12:19:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:32 smithi110 ceph-mon[97633]: pgmap v15049: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:32 smithi104 ceph-mon[97044]: pgmap v15049: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:33.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:33.258494+0000) 2022-09-10T12:19:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:33.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:33.784420+0000) 2022-09-10T12:19:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:34.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:34.258706+0000) 2022-09-10T12:19:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:34.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:34.784688+0000) 2022-09-10T12:19:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:35.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:35.258901+0000) 2022-09-10T12:19:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:35 smithi110 ceph-mon[97633]: pgmap v15050: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:36.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:35 smithi104 ceph-mon[97044]: pgmap v15050: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:35.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:35.784934+0000) 2022-09-10T12:19:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:36.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:36.259084+0000) 2022-09-10T12:19:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:36.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:36.785111+0000) 2022-09-10T12:19:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:36 smithi110 ceph-mon[97633]: pgmap v15051: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:36 smithi104 ceph-mon[97044]: pgmap v15051: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:37.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:37.259327+0000) 2022-09-10T12:19:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:37.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:37.785328+0000) 2022-09-10T12:19:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:38.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:38.259604+0000) 2022-09-10T12:19:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:38.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:38.785569+0000) 2022-09-10T12:19:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:38 smithi110 ceph-mon[97633]: pgmap v15052: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:38 smithi104 ceph-mon[97044]: pgmap v15052: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:39.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:39.259841+0000) 2022-09-10T12:19:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:39.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:39.785832+0000) 2022-09-10T12:19:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:40.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:40.260062+0000) 2022-09-10T12:19:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:40.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:40.786124+0000) 2022-09-10T12:19:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:40 smithi110 ceph-mon[97633]: pgmap v15053: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:40 smithi104 ceph-mon[97044]: pgmap v15053: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:41.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:41.260343+0000) 2022-09-10T12:19:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:41.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:41.786413+0000) 2022-09-10T12:19:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:42.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:42.260540+0000) 2022-09-10T12:19:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:42.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:42.786632+0000) 2022-09-10T12:19:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:42 smithi110 ceph-mon[97633]: pgmap v15054: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:42 smithi104 ceph-mon[97044]: pgmap v15054: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:43.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:43.260771+0000) 2022-09-10T12:19:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:43.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:43.786874+0000) 2022-09-10T12:19:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:44.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:44.261038+0000) 2022-09-10T12:19:44.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:19:44.430+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:19:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:44.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:44.787187+0000) 2022-09-10T12:19:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:45.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:45.261216+0000) 2022-09-10T12:19:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:45 smithi110 ceph-mon[97633]: pgmap v15055: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:46.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:45 smithi104 ceph-mon[97044]: pgmap v15055: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:45.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:45.787368+0000) 2022-09-10T12:19:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:46.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:46.261427+0000) 2022-09-10T12:19:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:19:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:19:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:19:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:19:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:19:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:19:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:19:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:19:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:46.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:46.787630+0000) 2022-09-10T12:19:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:47.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:47.261638+0000) 2022-09-10T12:19:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:47 smithi110 ceph-mon[97633]: pgmap v15056: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:48.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:47 smithi104 ceph-mon[97044]: pgmap v15056: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:47.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:47.787861+0000) 2022-09-10T12:19:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:48.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:48.261851+0000) 2022-09-10T12:19:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:48.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:48.788091+0000) 2022-09-10T12:19:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:48 smithi110 ceph-mon[97633]: pgmap v15057: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:48 smithi104 ceph-mon[97044]: pgmap v15057: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:49.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:49.262118+0000) 2022-09-10T12:19:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:49.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:49.788373+0000) 2022-09-10T12:19:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:50.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:50.262262+0000) 2022-09-10T12:19:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:50.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:50.788668+0000) 2022-09-10T12:19:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:50 smithi110 ceph-mon[97633]: pgmap v15058: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:50 smithi104 ceph-mon[97044]: pgmap v15058: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:51.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:51.262455+0000) 2022-09-10T12:19:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:51.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:51.788927+0000) 2022-09-10T12:19:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:52.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:52.262659+0000) 2022-09-10T12:19:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:52.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:52.789226+0000) 2022-09-10T12:19:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:52 smithi110 ceph-mon[97633]: pgmap v15059: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:52 smithi104 ceph-mon[97044]: pgmap v15059: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:53.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:53.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:53.262873+0000) 2022-09-10T12:19:54.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:53.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:53.789452+0000) 2022-09-10T12:19:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:54.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:54.263077+0000) 2022-09-10T12:19:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:54.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:54.789598+0000) 2022-09-10T12:19:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:55.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:55.263377+0000) 2022-09-10T12:19:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:55 smithi110 ceph-mon[97633]: pgmap v15060: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:56.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:55 smithi104 ceph-mon[97044]: pgmap v15060: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:19:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:55.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:55.789821+0000) 2022-09-10T12:19:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:56.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:56.263559+0000) 2022-09-10T12:19:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:56.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:56.790152+0000) 2022-09-10T12:19:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:56 smithi110 ceph-mon[97633]: pgmap v15061: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:56 smithi104 ceph-mon[97044]: pgmap v15061: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:57.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:57.263772+0000) 2022-09-10T12:19:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:57.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:57.790347+0000) 2022-09-10T12:19:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:58.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:58.264016+0000) 2022-09-10T12:19:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:58.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:58.790581+0000) 2022-09-10T12:19:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:19:58 smithi110 ceph-mon[97633]: pgmap v15062: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:19:58 smithi104 ceph-mon[97044]: pgmap v15062: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:19:59.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:19:59.445+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:19:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:19:59.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:59.264283+0000) 2022-09-10T12:20:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:19:59.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:19:59.790828+0000) 2022-09-10T12:20:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:00.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:00.264508+0000) 2022-09-10T12:20:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:20:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:20:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:00.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:00.790987+0000) 2022-09-10T12:20:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:01.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:01.264733+0000) 2022-09-10T12:20:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:01 smithi104 ceph-mon[97044]: pgmap v15063: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:01 smithi110 ceph-mon[97633]: pgmap v15063: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:02.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:01.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:01.791238+0000) 2022-09-10T12:20:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:02.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:02.264986+0000) 2022-09-10T12:20:03.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:02.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:02.791495+0000) 2022-09-10T12:20:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:02 smithi110 ceph-mon[97633]: pgmap v15064: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:02 smithi104 ceph-mon[97044]: pgmap v15064: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:03.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:03.265222+0000) 2022-09-10T12:20:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:03.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:03.791702+0000) 2022-09-10T12:20:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:04.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:04.265431+0000) 2022-09-10T12:20:05.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:04.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:04.792009+0000) 2022-09-10T12:20:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:05.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:05.265658+0000) 2022-09-10T12:20:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:05 smithi110 ceph-mon[97633]: pgmap v15065: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:20:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:05 smithi104 ceph-mon[97044]: pgmap v15065: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:20:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:05.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:05.792275+0000) 2022-09-10T12:20:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:06.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:06.265923+0000) 2022-09-10T12:20:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:06.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:06.792436+0000) 2022-09-10T12:20:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:06 smithi110 ceph-mon[97633]: pgmap v15066: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:20:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:06 smithi104 ceph-mon[97044]: pgmap v15066: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:20:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:07.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:07.266148+0000) 2022-09-10T12:20:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:07.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:07.792604+0000) 2022-09-10T12:20:08.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:08.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:08.266400+0000) 2022-09-10T12:20:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:08.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:08.792857+0000) 2022-09-10T12:20:09.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:08 smithi110 ceph-mon[97633]: osdmap e1022: 12 total, 12 up, 12 in 2022-09-10T12:20:09.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:08 smithi110 ceph-mon[97633]: pgmap v15068: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:20:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:08 smithi104 ceph-mon[97044]: osdmap e1022: 12 total, 12 up, 12 in 2022-09-10T12:20:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:08 smithi104 ceph-mon[97044]: pgmap v15068: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:20:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:09.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:09.266669+0000) 2022-09-10T12:20:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:09.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:09.793136+0000) 2022-09-10T12:20:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:10.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:10.266927+0000) 2022-09-10T12:20:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:10.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:10.793359+0000) 2022-09-10T12:20:11.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:10 smithi110 ceph-mon[97633]: pgmap v15069: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:20:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:10 smithi104 ceph-mon[97044]: pgmap v15069: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:20:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:11.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:11.267214+0000) 2022-09-10T12:20:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:11.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:11.793546+0000) 2022-09-10T12:20:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:12.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:12.267459+0000) 2022-09-10T12:20:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:12.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:12.793787+0000) 2022-09-10T12:20:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:12 smithi110 ceph-mon[97633]: pgmap v15070: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:20:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:12 smithi104 ceph-mon[97044]: pgmap v15070: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:20:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:13.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:13.267723+0000) 2022-09-10T12:20:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:13.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:13.794072+0000) 2022-09-10T12:20:14.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:20:14.459+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:20:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:14.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:14.267917+0000) 2022-09-10T12:20:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:14.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:14.794402+0000) 2022-09-10T12:20:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:15.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:15.268175+0000) 2022-09-10T12:20:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:15 smithi110 ceph-mon[97633]: osdmap e1023: 12 total, 12 up, 12 in 2022-09-10T12:20:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:15 smithi110 ceph-mon[97633]: pgmap v15072: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:20:16.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:15 smithi104 ceph-mon[97044]: osdmap e1023: 12 total, 12 up, 12 in 2022-09-10T12:20:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:15 smithi104 ceph-mon[97044]: pgmap v15072: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:20:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:15.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:15.794668+0000) 2022-09-10T12:20:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:16.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:16.268439+0000) 2022-09-10T12:20:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:16.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:16.794845+0000) 2022-09-10T12:20:17.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:16 smithi110 ceph-mon[97633]: pgmap v15073: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:20:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:16 smithi104 ceph-mon[97044]: pgmap v15073: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:20:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:17.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:17.268690+0000) 2022-09-10T12:20:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:17.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:17.795153+0000) 2022-09-10T12:20:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:18.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:18.268950+0000) 2022-09-10T12:20:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:18.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:18.795382+0000) 2022-09-10T12:20:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:18 smithi110 ceph-mon[97633]: pgmap v15074: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:20:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:18 smithi104 ceph-mon[97044]: pgmap v15074: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:20:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:19.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:19.269213+0000) 2022-09-10T12:20:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:19.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:19.795556+0000) 2022-09-10T12:20:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:20.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:20.269450+0000) 2022-09-10T12:20:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:20.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:20.795810+0000) 2022-09-10T12:20:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:20 smithi110 ceph-mon[97633]: pgmap v15075: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:20:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:20 smithi104 ceph-mon[97044]: pgmap v15075: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:20:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:21.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:21.269636+0000) 2022-09-10T12:20:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:21.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:21.796032+0000) 2022-09-10T12:20:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:22.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:22.269878+0000) 2022-09-10T12:20:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:22.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:22.796280+0000) 2022-09-10T12:20:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:22 smithi110 ceph-mon[97633]: pgmap v15076: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:20:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:22 smithi104 ceph-mon[97044]: pgmap v15076: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:20:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:23.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:23.270162+0000) 2022-09-10T12:20:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:23.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:23.796447+0000) 2022-09-10T12:20:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:24.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:24.270372+0000) 2022-09-10T12:20:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:24.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:24.796579+0000) 2022-09-10T12:20:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:25.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:25.270628+0000) 2022-09-10T12:20:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:25 smithi110 ceph-mon[97633]: pgmap v15077: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:20:26.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:25 smithi104 ceph-mon[97044]: pgmap v15077: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:20:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:25.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:25.796767+0000) 2022-09-10T12:20:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:26.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:26.270827+0000) 2022-09-10T12:20:27.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:26.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:26.796966+0000) 2022-09-10T12:20:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:26 smithi110 ceph-mon[97633]: pgmap v15078: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:26 smithi104 ceph-mon[97044]: pgmap v15078: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:27.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:27.270997+0000) 2022-09-10T12:20:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:27.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:27.797177+0000) 2022-09-10T12:20:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:28.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:28.271210+0000) 2022-09-10T12:20:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:28.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:28.797407+0000) 2022-09-10T12:20:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:28 smithi110 ceph-mon[97633]: pgmap v15079: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:28 smithi104 ceph-mon[97044]: pgmap v15079: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:20:29.475+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:20:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:29.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:29.271548+0000) 2022-09-10T12:20:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:29.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:29.797663+0000) 2022-09-10T12:20:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:30.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:30.271728+0000) 2022-09-10T12:20:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:30.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:30.797888+0000) 2022-09-10T12:20:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:30 smithi110 ceph-mon[97633]: pgmap v15080: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:31.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:30 smithi104 ceph-mon[97044]: pgmap v15080: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:31.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:31.271944+0000) 2022-09-10T12:20:32.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:31.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:31.798137+0000) 2022-09-10T12:20:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:32.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:32.272109+0000) 2022-09-10T12:20:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:32.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:32.798316+0000) 2022-09-10T12:20:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:32 smithi110 ceph-mon[97633]: pgmap v15081: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:32 smithi104 ceph-mon[97044]: pgmap v15081: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:33.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:33.272247+0000) 2022-09-10T12:20:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:33.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:33.798504+0000) 2022-09-10T12:20:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:34.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:34.272425+0000) 2022-09-10T12:20:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:34.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:34.798740+0000) 2022-09-10T12:20:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:35.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:35.272618+0000) 2022-09-10T12:20:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:35 smithi110 ceph-mon[97633]: pgmap v15082: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:36.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:35 smithi104 ceph-mon[97044]: pgmap v15082: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:35.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:35.799020+0000) 2022-09-10T12:20:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:36.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:36.272779+0000) 2022-09-10T12:20:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:36.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:36.799290+0000) 2022-09-10T12:20:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:36 smithi110 ceph-mon[97633]: pgmap v15083: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:36 smithi104 ceph-mon[97044]: pgmap v15083: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:37.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:37.272938+0000) 2022-09-10T12:20:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:37.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:37.799437+0000) 2022-09-10T12:20:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:38.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:38.273191+0000) 2022-09-10T12:20:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:38.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:38.799647+0000) 2022-09-10T12:20:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:38 smithi110 ceph-mon[97633]: pgmap v15084: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:38 smithi104 ceph-mon[97044]: pgmap v15084: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:39.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:39.273446+0000) 2022-09-10T12:20:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:39.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:39.799843+0000) 2022-09-10T12:20:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:40.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:40.273621+0000) 2022-09-10T12:20:41.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:40.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:40.800056+0000) 2022-09-10T12:20:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:40 smithi110 ceph-mon[97633]: pgmap v15085: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:40 smithi104 ceph-mon[97044]: pgmap v15085: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:41.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:41.273801+0000) 2022-09-10T12:20:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:41.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:41.800296+0000) 2022-09-10T12:20:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:42.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:42.274045+0000) 2022-09-10T12:20:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:42.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:42.800547+0000) 2022-09-10T12:20:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:42 smithi110 ceph-mon[97633]: pgmap v15086: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:42 smithi104 ceph-mon[97044]: pgmap v15086: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:43.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:43.274216+0000) 2022-09-10T12:20:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:43.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:43.800726+0000) 2022-09-10T12:20:44.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:20:44.491+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:20:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:44.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:44.274427+0000) 2022-09-10T12:20:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:44.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:44.800991+0000) 2022-09-10T12:20:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:45.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:45.274598+0000) 2022-09-10T12:20:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:45 smithi110 ceph-mon[97633]: pgmap v15087: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:46.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:45 smithi104 ceph-mon[97044]: pgmap v15087: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:45.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:45.801256+0000) 2022-09-10T12:20:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:46.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:46.274802+0000) 2022-09-10T12:20:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:20:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:20:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:20:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:20:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:20:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:20:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:20:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:46.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:46.801485+0000) 2022-09-10T12:20:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:47.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:47.275004+0000) 2022-09-10T12:20:47.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:47 smithi110 ceph-mon[97633]: pgmap v15088: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:48.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:47 smithi104 ceph-mon[97044]: pgmap v15088: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:47.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:47.801685+0000) 2022-09-10T12:20:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:48.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:48.275205+0000) 2022-09-10T12:20:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:48.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:48.801921+0000) 2022-09-10T12:20:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:48 smithi110 ceph-mon[97633]: pgmap v15089: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:48 smithi104 ceph-mon[97044]: pgmap v15089: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:49.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:49.275422+0000) 2022-09-10T12:20:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:49.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:49.802187+0000) 2022-09-10T12:20:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:50.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:50.275600+0000) 2022-09-10T12:20:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:50.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:50.802445+0000) 2022-09-10T12:20:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:50 smithi110 ceph-mon[97633]: pgmap v15090: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:50 smithi104 ceph-mon[97044]: pgmap v15090: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:51.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:51.275755+0000) 2022-09-10T12:20:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:51.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:51.802617+0000) 2022-09-10T12:20:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:52.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:52.275946+0000) 2022-09-10T12:20:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:52.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:52.802861+0000) 2022-09-10T12:20:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:52 smithi110 ceph-mon[97633]: pgmap v15091: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:52 smithi104 ceph-mon[97044]: pgmap v15091: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:53.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:53.276145+0000) 2022-09-10T12:20:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:53.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:53.803045+0000) 2022-09-10T12:20:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:54.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:54.276382+0000) 2022-09-10T12:20:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:54.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:54.803298+0000) 2022-09-10T12:20:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:55.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:55.276537+0000) 2022-09-10T12:20:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:55 smithi110 ceph-mon[97633]: pgmap v15092: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:56.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:55 smithi104 ceph-mon[97044]: pgmap v15092: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:20:56.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:55.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:55.803483+0000) 2022-09-10T12:20:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:56.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:56.276778+0000) 2022-09-10T12:20:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:56 smithi110 ceph-mon[97633]: pgmap v15093: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:57.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:56.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:56.803677+0000) 2022-09-10T12:20:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:56 smithi104 ceph-mon[97044]: pgmap v15093: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:57.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:57.277017+0000) 2022-09-10T12:20:58.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:57.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:57.803887+0000) 2022-09-10T12:20:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:58.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:58.277232+0000) 2022-09-10T12:20:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:20:58 smithi110 ceph-mon[97633]: pgmap v15094: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:59.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:58.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:58.804070+0000) 2022-09-10T12:20:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:20:58 smithi104 ceph-mon[97044]: pgmap v15094: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:20:59.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:20:59.505+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:20:59.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:20:59.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:59.277471+0000) 2022-09-10T12:21:00.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:20:59.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:20:59.804333+0000) 2022-09-10T12:21:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:00.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:00.277721+0000) 2022-09-10T12:21:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:00 smithi110 ceph-mon[97633]: pgmap v15095: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:01.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:00.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:00.804477+0000) 2022-09-10T12:21:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:00 smithi104 ceph-mon[97044]: pgmap v15095: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:01.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:01.277991+0000) 2022-09-10T12:21:02.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:01.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:01.804624+0000) 2022-09-10T12:21:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:02.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:02.278255+0000) 2022-09-10T12:21:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:02 smithi110 ceph-mon[97633]: pgmap v15096: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:03.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:02.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:02.804781+0000) 2022-09-10T12:21:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:02 smithi104 ceph-mon[97044]: pgmap v15096: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:03.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:03.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:03.278528+0000) 2022-09-10T12:21:04.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:03.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:03.805008+0000) 2022-09-10T12:21:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:04.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:04.278721+0000) 2022-09-10T12:21:05.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:04.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:04.805253+0000) 2022-09-10T12:21:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:05.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:05.278950+0000) 2022-09-10T12:21:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:05 smithi110 ceph-mon[97633]: pgmap v15097: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:21:06.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:05 smithi104 ceph-mon[97044]: pgmap v15097: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:21:06.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:05.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:05.805448+0000) 2022-09-10T12:21:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:06.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:06.279207+0000) 2022-09-10T12:21:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:06 smithi110 ceph-mon[97633]: pgmap v15098: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:21:07.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:06.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:06.805656+0000) 2022-09-10T12:21:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:06 smithi104 ceph-mon[97044]: pgmap v15098: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:21:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:07.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:07.279426+0000) 2022-09-10T12:21:08.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:07.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:07.805800+0000) 2022-09-10T12:21:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:08.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:08.279681+0000) 2022-09-10T12:21:09.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:08.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:08.806029+0000) 2022-09-10T12:21:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:08 smithi104 ceph-mon[97044]: osdmap e1024: 12 total, 12 up, 12 in 2022-09-10T12:21:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:08 smithi104 ceph-mon[97044]: pgmap v15100: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:21:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:08 smithi110 ceph-mon[97633]: osdmap e1024: 12 total, 12 up, 12 in 2022-09-10T12:21:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:08 smithi110 ceph-mon[97633]: pgmap v15100: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:21:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:09.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:09.279916+0000) 2022-09-10T12:21:10.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:09.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:09.806224+0000) 2022-09-10T12:21:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:10.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:10.280125+0000) 2022-09-10T12:21:11.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:10.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:10.806441+0000) 2022-09-10T12:21:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:10 smithi104 ceph-mon[97044]: pgmap v15101: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:21:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:10 smithi110 ceph-mon[97633]: pgmap v15101: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:21:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:11.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:11.280332+0000) 2022-09-10T12:21:12.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:11.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:11.806701+0000) 2022-09-10T12:21:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:12.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:12.280575+0000) 2022-09-10T12:21:13.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:12.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:12.806923+0000) 2022-09-10T12:21:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:12 smithi104 ceph-mon[97044]: pgmap v15102: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:21:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:12 smithi110 ceph-mon[97633]: pgmap v15102: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:21:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:13.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:13.280773+0000) 2022-09-10T12:21:14.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:13.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:13.807151+0000) 2022-09-10T12:21:14.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:21:14.514+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:21:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:14.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:14.281016+0000) 2022-09-10T12:21:15.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:14.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:14.807395+0000) 2022-09-10T12:21:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:15.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:15.281250+0000) 2022-09-10T12:21:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:15 smithi110 ceph-mon[97633]: osdmap e1025: 12 total, 12 up, 12 in 2022-09-10T12:21:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:15 smithi110 ceph-mon[97633]: pgmap v15104: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:21:16.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:15 smithi104 ceph-mon[97044]: osdmap e1025: 12 total, 12 up, 12 in 2022-09-10T12:21:16.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:15 smithi104 ceph-mon[97044]: pgmap v15104: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:21:16.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:15.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:15.807642+0000) 2022-09-10T12:21:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:16.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:16.281468+0000) 2022-09-10T12:21:17.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:16.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:16.807890+0000) 2022-09-10T12:21:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:16 smithi104 ceph-mon[97044]: pgmap v15105: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:21:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:16 smithi110 ceph-mon[97633]: pgmap v15105: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 229 B/s wr, 11 op/s 2022-09-10T12:21:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:17.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:17.281725+0000) 2022-09-10T12:21:18.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:17.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:17.808146+0000) 2022-09-10T12:21:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:18.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:18.282021+0000) 2022-09-10T12:21:19.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:18.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:18.808305+0000) 2022-09-10T12:21:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:18 smithi104 ceph-mon[97044]: pgmap v15106: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:21:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:18 smithi110 ceph-mon[97633]: pgmap v15106: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:21:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:19.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:19.282235+0000) 2022-09-10T12:21:20.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:19.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:19.808511+0000) 2022-09-10T12:21:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:20.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:20.282392+0000) 2022-09-10T12:21:21.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:20.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:20.808761+0000) 2022-09-10T12:21:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:20 smithi104 ceph-mon[97044]: pgmap v15107: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:21:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:20 smithi110 ceph-mon[97633]: pgmap v15107: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:21:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:21.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:21.282581+0000) 2022-09-10T12:21:22.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:21.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:21.809017+0000) 2022-09-10T12:21:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:22.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:22.282850+0000) 2022-09-10T12:21:23.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:22.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:22.809272+0000) 2022-09-10T12:21:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:22 smithi104 ceph-mon[97044]: pgmap v15108: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:21:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:22 smithi110 ceph-mon[97633]: pgmap v15108: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:21:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:23.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:23.283087+0000) 2022-09-10T12:21:24.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:23.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:23.809427+0000) 2022-09-10T12:21:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:24.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:24.283364+0000) 2022-09-10T12:21:25.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:24.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:24.809678+0000) 2022-09-10T12:21:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:25.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:25.283579+0000) 2022-09-10T12:21:26.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:25.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:25.809899+0000) 2022-09-10T12:21:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:26.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:26.283800+0000) 2022-09-10T12:21:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:26 smithi110 ceph-mon[97633]: pgmap v15109: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:21:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:26 smithi104 ceph-mon[97044]: pgmap v15109: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:21:27.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:26.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:26.810070+0000) 2022-09-10T12:21:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:27.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:27.284023+0000) 2022-09-10T12:21:28.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:27.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:27.810233+0000) 2022-09-10T12:21:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:28.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:28.284257+0000) 2022-09-10T12:21:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:28 smithi110 ceph-mon[97633]: pgmap v15110: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:28 smithi104 ceph-mon[97044]: pgmap v15110: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:29.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:28.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:28.810462+0000) 2022-09-10T12:21:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:29.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:29.284488+0000) 2022-09-10T12:21:29.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:21:29.530+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:21:30.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:29.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:29.810600+0000) 2022-09-10T12:21:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:30.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:30.284647+0000) 2022-09-10T12:21:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:30 smithi110 ceph-mon[97633]: pgmap v15111: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:30 smithi104 ceph-mon[97044]: pgmap v15111: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:31.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:30.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:30.810738+0000) 2022-09-10T12:21:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:31.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:31.284857+0000) 2022-09-10T12:21:32.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:31.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:31.810888+0000) 2022-09-10T12:21:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:32.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:32.285045+0000) 2022-09-10T12:21:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:32 smithi110 ceph-mon[97633]: pgmap v15112: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:32 smithi104 ceph-mon[97044]: pgmap v15112: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:33.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:32.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:32.811158+0000) 2022-09-10T12:21:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:33.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:33.285230+0000) 2022-09-10T12:21:34.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:33.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:33.811391+0000) 2022-09-10T12:21:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:34.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:34.285460+0000) 2022-09-10T12:21:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:34 smithi110 ceph-mon[97633]: pgmap v15113: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:34 smithi104 ceph-mon[97044]: pgmap v15113: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:35.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:34.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:34.811586+0000) 2022-09-10T12:21:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:35.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:35.285692+0000) 2022-09-10T12:21:36.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:35.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:35.811833+0000) 2022-09-10T12:21:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:36.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:36.285846+0000) 2022-09-10T12:21:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:36 smithi110 ceph-mon[97633]: pgmap v15114: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:36 smithi104 ceph-mon[97044]: pgmap v15114: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:37.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:36.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:36.812070+0000) 2022-09-10T12:21:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:37.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:37.286078+0000) 2022-09-10T12:21:38.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:37.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:37.812300+0000) 2022-09-10T12:21:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:38.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:38.286320+0000) 2022-09-10T12:21:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:38 smithi110 ceph-mon[97633]: pgmap v15115: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:38 smithi104 ceph-mon[97044]: pgmap v15115: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:39.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:38.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:38.812477+0000) 2022-09-10T12:21:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:39.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:39.286549+0000) 2022-09-10T12:21:40.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:39.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:39.812651+0000) 2022-09-10T12:21:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:40.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:40.286762+0000) 2022-09-10T12:21:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:40 smithi110 ceph-mon[97633]: pgmap v15116: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:40 smithi104 ceph-mon[97044]: pgmap v15116: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:41.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:40.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:40.812870+0000) 2022-09-10T12:21:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:41.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:41.286968+0000) 2022-09-10T12:21:42.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:41.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:41.813032+0000) 2022-09-10T12:21:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:42.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:42.287195+0000) 2022-09-10T12:21:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:42 smithi110 ceph-mon[97633]: pgmap v15117: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:42 smithi104 ceph-mon[97044]: pgmap v15117: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:43.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:42.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:42.813246+0000) 2022-09-10T12:21:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:43.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:43.287372+0000) 2022-09-10T12:21:44.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:43.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:43.813456+0000) 2022-09-10T12:21:44.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:44.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:44.287581+0000) 2022-09-10T12:21:44.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:21:44.545+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:21:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:44 smithi110 ceph-mon[97633]: pgmap v15118: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:44 smithi104 ceph-mon[97044]: pgmap v15118: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:45.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:44.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:44.813733+0000) 2022-09-10T12:21:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:45.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:45.287770+0000) 2022-09-10T12:21:46.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:45.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:45.813937+0000) 2022-09-10T12:21:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:46.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:46.287943+0000) 2022-09-10T12:21:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:46 smithi110 ceph-mon[97633]: pgmap v15119: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:21:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:46 smithi104 ceph-mon[97044]: pgmap v15119: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:21:47.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:46.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:46.814214+0000) 2022-09-10T12:21:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:47.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:47.288193+0000) 2022-09-10T12:21:48.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:47.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:47.814470+0000) 2022-09-10T12:21:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:48.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:48.288371+0000) 2022-09-10T12:21:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:48 smithi110 ceph-mon[97633]: pgmap v15120: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:48 smithi104 ceph-mon[97044]: pgmap v15120: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:49.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:48.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:48.814635+0000) 2022-09-10T12:21:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:49.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:49.288624+0000) 2022-09-10T12:21:50.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:49.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:49.814853+0000) 2022-09-10T12:21:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:50.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:50.288844+0000) 2022-09-10T12:21:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:50 smithi110 ceph-mon[97633]: pgmap v15121: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:50 smithi104 ceph-mon[97044]: pgmap v15121: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:51.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:50.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:50.815132+0000) 2022-09-10T12:21:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:51.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:51.289065+0000) 2022-09-10T12:21:52.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:51.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:51.815347+0000) 2022-09-10T12:21:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:52.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:52.289261+0000) 2022-09-10T12:21:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:52 smithi110 ceph-mon[97633]: pgmap v15122: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:52 smithi104 ceph-mon[97044]: pgmap v15122: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:53.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:52.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:52.815530+0000) 2022-09-10T12:21:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:53.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:53.289491+0000) 2022-09-10T12:21:54.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:53.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:53.815728+0000) 2022-09-10T12:21:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:54.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:54.289657+0000) 2022-09-10T12:21:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:54 smithi110 ceph-mon[97633]: pgmap v15123: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:54 smithi104 ceph-mon[97044]: pgmap v15123: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:55.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:54.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:54.815925+0000) 2022-09-10T12:21:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:55.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:55.289876+0000) 2022-09-10T12:21:56.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:55.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:55.816188+0000) 2022-09-10T12:21:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:56.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:56.290130+0000) 2022-09-10T12:21:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:56 smithi110 ceph-mon[97633]: pgmap v15124: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:56 smithi104 ceph-mon[97044]: pgmap v15124: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:21:57.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:56.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:56.816395+0000) 2022-09-10T12:21:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:57.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:57.290377+0000) 2022-09-10T12:21:58.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:57.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:57.816611+0000) 2022-09-10T12:21:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:58.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:58.290593+0000) 2022-09-10T12:21:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:21:58 smithi110 ceph-mon[97633]: pgmap v15125: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:21:58 smithi104 ceph-mon[97044]: pgmap v15125: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:21:59.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:58.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:58.816840+0000) 2022-09-10T12:21:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:21:59.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:59.290832+0000) 2022-09-10T12:21:59.816 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:21:59.561+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:22:00.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:21:59.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:21:59.816962+0000) 2022-09-10T12:22:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:00.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:00.291078+0000) 2022-09-10T12:22:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:00 smithi110 ceph-mon[97633]: pgmap v15126: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:00 smithi104 ceph-mon[97044]: pgmap v15126: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:01.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:00.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:00.817127+0000) 2022-09-10T12:22:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:01.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:01.291308+0000) 2022-09-10T12:22:02.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:01.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:01.817341+0000) 2022-09-10T12:22:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:02.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:02.291528+0000) 2022-09-10T12:22:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:02 smithi110 ceph-mon[97633]: pgmap v15127: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:02 smithi104 ceph-mon[97044]: pgmap v15127: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:03.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:02.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:02.817538+0000) 2022-09-10T12:22:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:03.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:03.291755+0000) 2022-09-10T12:22:04.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:03.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:03.817758+0000) 2022-09-10T12:22:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:04.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:04.291995+0000) 2022-09-10T12:22:04.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:04 smithi110 ceph-mon[97633]: pgmap v15128: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:04 smithi104 ceph-mon[97044]: pgmap v15128: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:05.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:04.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:04.817994+0000) 2022-09-10T12:22:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:05.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:05.292200+0000) 2022-09-10T12:22:06.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:05.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:05.818244+0000) 2022-09-10T12:22:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:06.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:06.292415+0000) 2022-09-10T12:22:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:06 smithi110 ceph-mon[97633]: pgmap v15129: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:22:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:06 smithi104 ceph-mon[97044]: pgmap v15129: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:22:07.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:06.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:06.818497+0000) 2022-09-10T12:22:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:07.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:07.292577+0000) 2022-09-10T12:22:08.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:07.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:07.818716+0000) 2022-09-10T12:22:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:08.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:08.292774+0000) 2022-09-10T12:22:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:08 smithi110 ceph-mon[97633]: pgmap v15130: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:22:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:08 smithi104 ceph-mon[97044]: pgmap v15130: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:22:09.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:08.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:08.818934+0000) 2022-09-10T12:22:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:09.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:09.293030+0000) 2022-09-10T12:22:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:09 smithi110 ceph-mon[97633]: osdmap e1026: 12 total, 12 up, 12 in 2022-09-10T12:22:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:09 smithi104 ceph-mon[97044]: osdmap e1026: 12 total, 12 up, 12 in 2022-09-10T12:22:10.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:09.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:09.819174+0000) 2022-09-10T12:22:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:10.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:10.293264+0000) 2022-09-10T12:22:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:10 smithi110 ceph-mon[97633]: pgmap v15132: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:22:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:10 smithi104 ceph-mon[97044]: pgmap v15132: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:22:11.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:10.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:10.819302+0000) 2022-09-10T12:22:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:11.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:11.293462+0000) 2022-09-10T12:22:12.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:11.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:11.819526+0000) 2022-09-10T12:22:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:12.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:12.293658+0000) 2022-09-10T12:22:12.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:12 smithi110 ceph-mon[97633]: pgmap v15133: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:22:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:12 smithi104 ceph-mon[97044]: pgmap v15133: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:22:13.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:12.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:12.819707+0000) 2022-09-10T12:22:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:13.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:13.293924+0000) 2022-09-10T12:22:14.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:13.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:13.819875+0000) 2022-09-10T12:22:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:14.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:14.294197+0000) 2022-09-10T12:22:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:14 smithi110 ceph-mon[97633]: pgmap v15134: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:22:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:14 smithi104 ceph-mon[97044]: pgmap v15134: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:22:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:14.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:14.820055+0000) 2022-09-10T12:22:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:22:14.573+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:22:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:15.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:15.294420+0000) 2022-09-10T12:22:16.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:15.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:15.820199+0000) 2022-09-10T12:22:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:16.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:16.294643+0000) 2022-09-10T12:22:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:16 smithi104 ceph-mon[97044]: pgmap v15135: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:22:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:16 smithi110 ceph-mon[97633]: pgmap v15135: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:22:17.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:16.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:16.820414+0000) 2022-09-10T12:22:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:17.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:17.294866+0000) 2022-09-10T12:22:18.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:17.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:17.820557+0000) 2022-09-10T12:22:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:18.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:18.295144+0000) 2022-09-10T12:22:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:18 smithi104 ceph-mon[97044]: pgmap v15136: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:22:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:18 smithi110 ceph-mon[97633]: pgmap v15136: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:22:19.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:18.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:18.820748+0000) 2022-09-10T12:22:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:19.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:19.295336+0000) 2022-09-10T12:22:20.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:19.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:19.820941+0000) 2022-09-10T12:22:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:20.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:20.295597+0000) 2022-09-10T12:22:20.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:20 smithi110 ceph-mon[97633]: pgmap v15137: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 200 B/s wr, 10 op/s 2022-09-10T12:22:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:20 smithi110 ceph-mon[97633]: osdmap e1027: 12 total, 12 up, 12 in 2022-09-10T12:22:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:20 smithi104 ceph-mon[97044]: pgmap v15137: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 200 B/s wr, 10 op/s 2022-09-10T12:22:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:20 smithi104 ceph-mon[97044]: osdmap e1027: 12 total, 12 up, 12 in 2022-09-10T12:22:21.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:20.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:20.821165+0000) 2022-09-10T12:22:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:21.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:21.295787+0000) 2022-09-10T12:22:22.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:21.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:21.821351+0000) 2022-09-10T12:22:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:22.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:22.295964+0000) 2022-09-10T12:22:22.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:22 smithi110 ceph-mon[97633]: pgmap v15139: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:22:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:22 smithi104 ceph-mon[97044]: pgmap v15139: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:22:23.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:22.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:22.821588+0000) 2022-09-10T12:22:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:23.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:23.296155+0000) 2022-09-10T12:22:24.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:23.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:23.821809+0000) 2022-09-10T12:22:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:24.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:24.296362+0000) 2022-09-10T12:22:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:24 smithi110 ceph-mon[97633]: pgmap v15140: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:22:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:24 smithi104 ceph-mon[97044]: pgmap v15140: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:22:25.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:24.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:24.822087+0000) 2022-09-10T12:22:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:25.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:25.296545+0000) 2022-09-10T12:22:26.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:25.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:25.822319+0000) 2022-09-10T12:22:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:26.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:26.296777+0000) 2022-09-10T12:22:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:26 smithi110 ceph-mon[97633]: pgmap v15141: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:22:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:26 smithi104 ceph-mon[97044]: pgmap v15141: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:22:27.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:26.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:26.822519+0000) 2022-09-10T12:22:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:27.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:27.296937+0000) 2022-09-10T12:22:28.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:27.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:27.822746+0000) 2022-09-10T12:22:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:28.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:28.297148+0000) 2022-09-10T12:22:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:28 smithi110 ceph-mon[97633]: pgmap v15142: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:22:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:28 smithi104 ceph-mon[97044]: pgmap v15142: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:22:29.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:28.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:28.822895+0000) 2022-09-10T12:22:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:29.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:29.297352+0000) 2022-09-10T12:22:30.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:22:29.589+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:22:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:29.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:29.823168+0000) 2022-09-10T12:22:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:30.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:30.297548+0000) 2022-09-10T12:22:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:30 smithi110 ceph-mon[97633]: pgmap v15143: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:22:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:30 smithi104 ceph-mon[97044]: pgmap v15143: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:22:31.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:30.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:30.823399+0000) 2022-09-10T12:22:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:31.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:31.297794+0000) 2022-09-10T12:22:32.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:31.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:31.823635+0000) 2022-09-10T12:22:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:32.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:32.298054+0000) 2022-09-10T12:22:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:32 smithi110 ceph-mon[97633]: pgmap v15144: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-10T12:22:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:32 smithi104 ceph-mon[97044]: pgmap v15144: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-10T12:22:33.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:32.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:32.823870+0000) 2022-09-10T12:22:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:33.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:33.298276+0000) 2022-09-10T12:22:34.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:33.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:33.824015+0000) 2022-09-10T12:22:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:34.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:34.298496+0000) 2022-09-10T12:22:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:34 smithi110 ceph-mon[97633]: pgmap v15145: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:34 smithi104 ceph-mon[97044]: pgmap v15145: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:35.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:34.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:34.824237+0000) 2022-09-10T12:22:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:35.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:35.298750+0000) 2022-09-10T12:22:36.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:35.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:35.824401+0000) 2022-09-10T12:22:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:36.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:36.298986+0000) 2022-09-10T12:22:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:36 smithi110 ceph-mon[97633]: pgmap v15146: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:36 smithi104 ceph-mon[97044]: pgmap v15146: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:37.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:36.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:36.824527+0000) 2022-09-10T12:22:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:37.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:37.299209+0000) 2022-09-10T12:22:38.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:37.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:37.824724+0000) 2022-09-10T12:22:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:38.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:38.299343+0000) 2022-09-10T12:22:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:38 smithi110 ceph-mon[97633]: pgmap v15147: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:38 smithi104 ceph-mon[97044]: pgmap v15147: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:39.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:38.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:38.824931+0000) 2022-09-10T12:22:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:39.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:39.299572+0000) 2022-09-10T12:22:40.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:39.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:39.825196+0000) 2022-09-10T12:22:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:40.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:40.299736+0000) 2022-09-10T12:22:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:40 smithi110 ceph-mon[97633]: pgmap v15148: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:40 smithi104 ceph-mon[97044]: pgmap v15148: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:41.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:40.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:40.825471+0000) 2022-09-10T12:22:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:41.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:41.299971+0000) 2022-09-10T12:22:42.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:41.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:41.825721+0000) 2022-09-10T12:22:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:42.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:42.300216+0000) 2022-09-10T12:22:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:42 smithi110 ceph-mon[97633]: pgmap v15149: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:42 smithi104 ceph-mon[97044]: pgmap v15149: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:43.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:42.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:42.825951+0000) 2022-09-10T12:22:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:43.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:43.300443+0000) 2022-09-10T12:22:44.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:43.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:43.826172+0000) 2022-09-10T12:22:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:44.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:44.300576+0000) 2022-09-10T12:22:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:44 smithi110 ceph-mon[97633]: pgmap v15150: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:44 smithi104 ceph-mon[97044]: pgmap v15150: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:45.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:22:44.605+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:22:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:44.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:44.826391+0000) 2022-09-10T12:22:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:45.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:45.300813+0000) 2022-09-10T12:22:46.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:45.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:45.826561+0000) 2022-09-10T12:22:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:46.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:46.301074+0000) 2022-09-10T12:22:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:46 smithi110 ceph-mon[97633]: pgmap v15151: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:22:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:46 smithi104 ceph-mon[97044]: pgmap v15151: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:22:47.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:46.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:46.826807+0000) 2022-09-10T12:22:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:47.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:47.301272+0000) 2022-09-10T12:22:48.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:47.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:47.827043+0000) 2022-09-10T12:22:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:48.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:48.301523+0000) 2022-09-10T12:22:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:48 smithi110 ceph-mon[97633]: pgmap v15152: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:48 smithi104 ceph-mon[97044]: pgmap v15152: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:49.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:48.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:48.827275+0000) 2022-09-10T12:22:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:49.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:49.301682+0000) 2022-09-10T12:22:50.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:49.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:49.827502+0000) 2022-09-10T12:22:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:50.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:50.301899+0000) 2022-09-10T12:22:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:50 smithi110 ceph-mon[97633]: pgmap v15153: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:50 smithi104 ceph-mon[97044]: pgmap v15153: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:51.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:50.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:50.827717+0000) 2022-09-10T12:22:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:51.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:51.302171+0000) 2022-09-10T12:22:52.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:51.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:51.827957+0000) 2022-09-10T12:22:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:52.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:52.302367+0000) 2022-09-10T12:22:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:52 smithi110 ceph-mon[97633]: pgmap v15154: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:53.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:52 smithi104 ceph-mon[97044]: pgmap v15154: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:52.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:52.828212+0000) 2022-09-10T12:22:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:53.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:53.302556+0000) 2022-09-10T12:22:54.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:53.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:53.828440+0000) 2022-09-10T12:22:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:54.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:54.302800+0000) 2022-09-10T12:22:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:54 smithi110 ceph-mon[97633]: pgmap v15155: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:54 smithi104 ceph-mon[97044]: pgmap v15155: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:54.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:54.828737+0000) 2022-09-10T12:22:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:55.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:55.303065+0000) 2022-09-10T12:22:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:55.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:55.828906+0000) 2022-09-10T12:22:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:56.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:56.303263+0000) 2022-09-10T12:22:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:56 smithi110 ceph-mon[97633]: pgmap v15156: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:56 smithi104 ceph-mon[97044]: pgmap v15156: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:22:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:56.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:56.829130+0000) 2022-09-10T12:22:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:57.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:57.303482+0000) 2022-09-10T12:22:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:57.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:57.829364+0000) 2022-09-10T12:22:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:58.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:58.303696+0000) 2022-09-10T12:22:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:22:58 smithi110 ceph-mon[97633]: pgmap v15157: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:22:58 smithi104 ceph-mon[97044]: pgmap v15157: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:22:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:58.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:58.829584+0000) 2022-09-10T12:22:59.620 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:22:59.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:59.303960+0000) 2022-09-10T12:23:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:22:59.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:22:59.829791+0000) 2022-09-10T12:23:00.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:22:59.619+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:23:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:00.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:00.304182+0000) 2022-09-10T12:23:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:00 smithi110 ceph-mon[97633]: pgmap v15158: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:00 smithi104 ceph-mon[97044]: pgmap v15158: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:00.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:00.829990+0000) 2022-09-10T12:23:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:01.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:01.304319+0000) 2022-09-10T12:23:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:01.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:01.830185+0000) 2022-09-10T12:23:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:02.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:02.304464+0000) 2022-09-10T12:23:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:02 smithi110 ceph-mon[97633]: pgmap v15159: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:02 smithi104 ceph-mon[97044]: pgmap v15159: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:02.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:02.830313+0000) 2022-09-10T12:23:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:03.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:03.304629+0000) 2022-09-10T12:23:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:03.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:03.830541+0000) 2022-09-10T12:23:04.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:04.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:04.304836+0000) 2022-09-10T12:23:04.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:04 smithi110 ceph-mon[97633]: pgmap v15160: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:04 smithi104 ceph-mon[97044]: pgmap v15160: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:04.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:04.830748+0000) 2022-09-10T12:23:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:05.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:05.304965+0000) 2022-09-10T12:23:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:05.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:05.831027+0000) 2022-09-10T12:23:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:06.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:06.305167+0000) 2022-09-10T12:23:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:06 smithi104 ceph-mon[97044]: pgmap v15161: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:23:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:06 smithi110 ceph-mon[97633]: pgmap v15161: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:23:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:06.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:06.831264+0000) 2022-09-10T12:23:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:07.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:07.305297+0000) 2022-09-10T12:23:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:07.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:07.831498+0000) 2022-09-10T12:23:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:08.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:08.305463+0000) 2022-09-10T12:23:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:08 smithi104 ceph-mon[97044]: pgmap v15162: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:23:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:08 smithi110 ceph-mon[97633]: pgmap v15162: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:23:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:08.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:08.831653+0000) 2022-09-10T12:23:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:09.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:09.305675+0000) 2022-09-10T12:23:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:09 smithi104 ceph-mon[97044]: osdmap e1028: 12 total, 12 up, 12 in 2022-09-10T12:23:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:09 smithi110 ceph-mon[97633]: osdmap e1028: 12 total, 12 up, 12 in 2022-09-10T12:23:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:09.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:09.831812+0000) 2022-09-10T12:23:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:10.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:10.305861+0000) 2022-09-10T12:23:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:10 smithi104 ceph-mon[97044]: pgmap v15164: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:23:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:10 smithi110 ceph-mon[97633]: pgmap v15164: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:23:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:10.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:10.832005+0000) 2022-09-10T12:23:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:11.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:11.306062+0000) 2022-09-10T12:23:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:11.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:11.832230+0000) 2022-09-10T12:23:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:12.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:12.306247+0000) 2022-09-10T12:23:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:12 smithi104 ceph-mon[97044]: pgmap v15165: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:23:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:12 smithi110 ceph-mon[97633]: pgmap v15165: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:23:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:12.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:12.832411+0000) 2022-09-10T12:23:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:13.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:13.306395+0000) 2022-09-10T12:23:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:13.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:13.832574+0000) 2022-09-10T12:23:14.633 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:14.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:14.306574+0000) 2022-09-10T12:23:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:14 smithi104 ceph-mon[97044]: pgmap v15166: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:23:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:14 smithi104 ceph-mon[97044]: osdmap e1029: 12 total, 12 up, 12 in 2022-09-10T12:23:15.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:23:14.633+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:23:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:14.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:14.832784+0000) 2022-09-10T12:23:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:14 smithi110 ceph-mon[97633]: pgmap v15166: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:23:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:14 smithi110 ceph-mon[97633]: osdmap e1029: 12 total, 12 up, 12 in 2022-09-10T12:23:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:15.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:15.306729+0000) 2022-09-10T12:23:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:15.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:15.832983+0000) 2022-09-10T12:23:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:16.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:16.306926+0000) 2022-09-10T12:23:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:16 smithi104 ceph-mon[97044]: pgmap v15168: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:23:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:16 smithi110 ceph-mon[97633]: pgmap v15168: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:23:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:16.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:16.833178+0000) 2022-09-10T12:23:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:17.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:17.307164+0000) 2022-09-10T12:23:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:17.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:17.833337+0000) 2022-09-10T12:23:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:18.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:18.307340+0000) 2022-09-10T12:23:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:18 smithi104 ceph-mon[97044]: pgmap v15169: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T12:23:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:18 smithi110 ceph-mon[97633]: pgmap v15169: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T12:23:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:18.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:18.833595+0000) 2022-09-10T12:23:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:19.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:19.307539+0000) 2022-09-10T12:23:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:19.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:19.833820+0000) 2022-09-10T12:23:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:20.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:20.307756+0000) 2022-09-10T12:23:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:20 smithi104 ceph-mon[97044]: pgmap v15170: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:23:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:20 smithi110 ceph-mon[97633]: pgmap v15170: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:23:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:20.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:20.834013+0000) 2022-09-10T12:23:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:21.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:21.307919+0000) 2022-09-10T12:23:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:21.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:21.834240+0000) 2022-09-10T12:23:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:22.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:22.308141+0000) 2022-09-10T12:23:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:22 smithi104 ceph-mon[97044]: pgmap v15171: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:23:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:22 smithi110 ceph-mon[97633]: pgmap v15171: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:23:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:22.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:22.834426+0000) 2022-09-10T12:23:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:23.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:23.308315+0000) 2022-09-10T12:23:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:23.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:23.834615+0000) 2022-09-10T12:23:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:24.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:24.308495+0000) 2022-09-10T12:23:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:24 smithi104 ceph-mon[97044]: pgmap v15172: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:23:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:24 smithi110 ceph-mon[97633]: pgmap v15172: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:23:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:24.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:24.834845+0000) 2022-09-10T12:23:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:25.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:25.308677+0000) 2022-09-10T12:23:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:25.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:25.835059+0000) 2022-09-10T12:23:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:26.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:26.308855+0000) 2022-09-10T12:23:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:26 smithi104 ceph-mon[97044]: pgmap v15173: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:23:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:26 smithi110 ceph-mon[97633]: pgmap v15173: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:23:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:26.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:26.835301+0000) 2022-09-10T12:23:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:27.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:27.309036+0000) 2022-09-10T12:23:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:27.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:27.835591+0000) 2022-09-10T12:23:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:28.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:28.309257+0000) 2022-09-10T12:23:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:28 smithi104 ceph-mon[97044]: pgmap v15174: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:28 smithi110 ceph-mon[97633]: pgmap v15174: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:28.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:28.835785+0000) 2022-09-10T12:23:29.641 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:29.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:29.309421+0000) 2022-09-10T12:23:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:29.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:29.835991+0000) 2022-09-10T12:23:30.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:23:29.640+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:23:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:30.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:30.309684+0000) 2022-09-10T12:23:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:30 smithi104 ceph-mon[97044]: pgmap v15175: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:30 smithi110 ceph-mon[97633]: pgmap v15175: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:30.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:30.836259+0000) 2022-09-10T12:23:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:31.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:31.309828+0000) 2022-09-10T12:23:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:31.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:31.836510+0000) 2022-09-10T12:23:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:32.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:32.310034+0000) 2022-09-10T12:23:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:32 smithi104 ceph-mon[97044]: pgmap v15176: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:32 smithi110 ceph-mon[97633]: pgmap v15176: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:32.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:32.836717+0000) 2022-09-10T12:23:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:33.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:33.310190+0000) 2022-09-10T12:23:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:33.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:33.836936+0000) 2022-09-10T12:23:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:34.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:34.310333+0000) 2022-09-10T12:23:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:34 smithi104 ceph-mon[97044]: pgmap v15177: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:34 smithi110 ceph-mon[97633]: pgmap v15177: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:34.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:34.837207+0000) 2022-09-10T12:23:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:35.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:35.310511+0000) 2022-09-10T12:23:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:35.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:35.837490+0000) 2022-09-10T12:23:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:36.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:36.310674+0000) 2022-09-10T12:23:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:36 smithi104 ceph-mon[97044]: pgmap v15178: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:36 smithi110 ceph-mon[97633]: pgmap v15178: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:36.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:36.837687+0000) 2022-09-10T12:23:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:37.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:37.310890+0000) 2022-09-10T12:23:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:37.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:37.837870+0000) 2022-09-10T12:23:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:38.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:38.311127+0000) 2022-09-10T12:23:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:38 smithi104 ceph-mon[97044]: pgmap v15179: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:38 smithi110 ceph-mon[97633]: pgmap v15179: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:38.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:38.838022+0000) 2022-09-10T12:23:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:39.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:39.311349+0000) 2022-09-10T12:23:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:39.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:39.838256+0000) 2022-09-10T12:23:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:40.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:40.311518+0000) 2022-09-10T12:23:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:40 smithi104 ceph-mon[97044]: pgmap v15180: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:40 smithi110 ceph-mon[97633]: pgmap v15180: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:40.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:40.838490+0000) 2022-09-10T12:23:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:41.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:41.311690+0000) 2022-09-10T12:23:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:41.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:41.838678+0000) 2022-09-10T12:23:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:42.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:42.311888+0000) 2022-09-10T12:23:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:42 smithi104 ceph-mon[97044]: pgmap v15181: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:42 smithi110 ceph-mon[97633]: pgmap v15181: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:42.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:42.838939+0000) 2022-09-10T12:23:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:43.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:43.312167+0000) 2022-09-10T12:23:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:43.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:43.839227+0000) 2022-09-10T12:23:44.656 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:44.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:44.312363+0000) 2022-09-10T12:23:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:44 smithi104 ceph-mon[97044]: pgmap v15182: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:44.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:44.839477+0000) 2022-09-10T12:23:45.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:23:44.655+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:23:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:44 smithi110 ceph-mon[97633]: pgmap v15182: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:45.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:45.312552+0000) 2022-09-10T12:23:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:45.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:45.839627+0000) 2022-09-10T12:23:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:46.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:46.312778+0000) 2022-09-10T12:23:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:46 smithi104 ceph-mon[97044]: pgmap v15183: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:23:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:46 smithi110 ceph-mon[97633]: pgmap v15183: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:23:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:23:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:23:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:23:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:46.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:46.839767+0000) 2022-09-10T12:23:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:47.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:47.312979+0000) 2022-09-10T12:23:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:47.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:47.840050+0000) 2022-09-10T12:23:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:48.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:48.313210+0000) 2022-09-10T12:23:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:48 smithi104 ceph-mon[97044]: pgmap v15184: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:48 smithi110 ceph-mon[97633]: pgmap v15184: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:48.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:48.840283+0000) 2022-09-10T12:23:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:49.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:49.313382+0000) 2022-09-10T12:23:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:49.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:49.840522+0000) 2022-09-10T12:23:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:50.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:50.313553+0000) 2022-09-10T12:23:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:50 smithi104 ceph-mon[97044]: pgmap v15185: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:50 smithi110 ceph-mon[97633]: pgmap v15185: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:50.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:50.840764+0000) 2022-09-10T12:23:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:51.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:51.313718+0000) 2022-09-10T12:23:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:51.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:51.841045+0000) 2022-09-10T12:23:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:52.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:52.313920+0000) 2022-09-10T12:23:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:52 smithi104 ceph-mon[97044]: pgmap v15186: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:52 smithi110 ceph-mon[97633]: pgmap v15186: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:52.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:52.841234+0000) 2022-09-10T12:23:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:53.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:53.314176+0000) 2022-09-10T12:23:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:53.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:53.841443+0000) 2022-09-10T12:23:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:54.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:54.314318+0000) 2022-09-10T12:23:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:54 smithi104 ceph-mon[97044]: pgmap v15187: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:54 smithi110 ceph-mon[97633]: pgmap v15187: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:54.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:54.841651+0000) 2022-09-10T12:23:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:55.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:55.314555+0000) 2022-09-10T12:23:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:55.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:55.841825+0000) 2022-09-10T12:23:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:56.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:56.314790+0000) 2022-09-10T12:23:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:56 smithi104 ceph-mon[97044]: pgmap v15188: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:56 smithi110 ceph-mon[97633]: pgmap v15188: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:23:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:56.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:56.842064+0000) 2022-09-10T12:23:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:57.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:57.314962+0000) 2022-09-10T12:23:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:57.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:57.842358+0000) 2022-09-10T12:23:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:58.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:58.315156+0000) 2022-09-10T12:23:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:23:58 smithi104 ceph-mon[97044]: pgmap v15189: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:23:58 smithi110 ceph-mon[97633]: pgmap v15189: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:23:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:58.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:58.842540+0000) 2022-09-10T12:23:59.669 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:23:59.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:59.315335+0000) 2022-09-10T12:24:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:23:59.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:23:59.842727+0000) 2022-09-10T12:24:00.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:23:59.669+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:24:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:00.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:00.315474+0000) 2022-09-10T12:24:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:00 smithi104 ceph-mon[97044]: pgmap v15190: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:00 smithi110 ceph-mon[97633]: pgmap v15190: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:00.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:00.842980+0000) 2022-09-10T12:24:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:01.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:01.315686+0000) 2022-09-10T12:24:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:01.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:01.843171+0000) 2022-09-10T12:24:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:02.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:02.315958+0000) 2022-09-10T12:24:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:02 smithi104 ceph-mon[97044]: pgmap v15191: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:02 smithi110 ceph-mon[97633]: pgmap v15191: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:02.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:02.843366+0000) 2022-09-10T12:24:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:03.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:03.316116+0000) 2022-09-10T12:24:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:03.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:03.843603+0000) 2022-09-10T12:24:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:04.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:04.316281+0000) 2022-09-10T12:24:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:04 smithi104 ceph-mon[97044]: pgmap v15192: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:04 smithi110 ceph-mon[97633]: pgmap v15192: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:04.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:04.843830+0000) 2022-09-10T12:24:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:05.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:05.316448+0000) 2022-09-10T12:24:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:05.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:05.844080+0000) 2022-09-10T12:24:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:06.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:06.316680+0000) 2022-09-10T12:24:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:06 smithi104 ceph-mon[97044]: pgmap v15193: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:24:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:06 smithi110 ceph-mon[97633]: pgmap v15193: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:24:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:06.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:06.844258+0000) 2022-09-10T12:24:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:07.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:07.316884+0000) 2022-09-10T12:24:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:07.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:07.844838+0000) 2022-09-10T12:24:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:08.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:08.317167+0000) 2022-09-10T12:24:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:08 smithi104 ceph-mon[97044]: pgmap v15194: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:24:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:08 smithi104 ceph-mon[97044]: osdmap e1030: 12 total, 12 up, 12 in 2022-09-10T12:24:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:08 smithi110 ceph-mon[97633]: pgmap v15194: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:24:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:08 smithi110 ceph-mon[97633]: osdmap e1030: 12 total, 12 up, 12 in 2022-09-10T12:24:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:08.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:08.845050+0000) 2022-09-10T12:24:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:09.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:09.317379+0000) 2022-09-10T12:24:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:09.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:09.845243+0000) 2022-09-10T12:24:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:10.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:10.317606+0000) 2022-09-10T12:24:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:10 smithi104 ceph-mon[97044]: pgmap v15196: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:24:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:10 smithi110 ceph-mon[97633]: pgmap v15196: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:24:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:10.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:10.845352+0000) 2022-09-10T12:24:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:11.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:11.317842+0000) 2022-09-10T12:24:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:11.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:11.845543+0000) 2022-09-10T12:24:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:12.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:12.318069+0000) 2022-09-10T12:24:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:12 smithi104 ceph-mon[97044]: pgmap v15197: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:24:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:12 smithi110 ceph-mon[97633]: pgmap v15197: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:24:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:12.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:12.845746+0000) 2022-09-10T12:24:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:13.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:13.318335+0000) 2022-09-10T12:24:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:13.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:13.845987+0000) 2022-09-10T12:24:14.683 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:14.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:14.318505+0000) 2022-09-10T12:24:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:14 smithi104 ceph-mon[97044]: pgmap v15198: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:24:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:14 smithi104 ceph-mon[97044]: osdmap e1031: 12 total, 12 up, 12 in 2022-09-10T12:24:15.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:24:14.682+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:24:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:14.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:14.846229+0000) 2022-09-10T12:24:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:14 smithi110 ceph-mon[97633]: pgmap v15198: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:24:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:14 smithi110 ceph-mon[97633]: osdmap e1031: 12 total, 12 up, 12 in 2022-09-10T12:24:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:15.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:15.318771+0000) 2022-09-10T12:24:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:15.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:15.846456+0000) 2022-09-10T12:24:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:16.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:16.318979+0000) 2022-09-10T12:24:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:16 smithi104 ceph-mon[97044]: pgmap v15200: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:24:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:16 smithi110 ceph-mon[97633]: pgmap v15200: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:24:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:16.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:16.846722+0000) 2022-09-10T12:24:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:17.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:17.319207+0000) 2022-09-10T12:24:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:17.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:17.846927+0000) 2022-09-10T12:24:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:18.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:18.319426+0000) 2022-09-10T12:24:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:18 smithi104 ceph-mon[97044]: pgmap v15201: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T12:24:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:18 smithi110 ceph-mon[97633]: pgmap v15201: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T12:24:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:18.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:18.847123+0000) 2022-09-10T12:24:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:19.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:19.319606+0000) 2022-09-10T12:24:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:19.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:19.847376+0000) 2022-09-10T12:24:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:20.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:20.319851+0000) 2022-09-10T12:24:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:20 smithi104 ceph-mon[97044]: pgmap v15202: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:24:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:20 smithi110 ceph-mon[97633]: pgmap v15202: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:24:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:20.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:20.847591+0000) 2022-09-10T12:24:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:21.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:21.320030+0000) 2022-09-10T12:24:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:21.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:21.847847+0000) 2022-09-10T12:24:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:22.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:22.320274+0000) 2022-09-10T12:24:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:22 smithi104 ceph-mon[97044]: pgmap v15203: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:24:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:22 smithi110 ceph-mon[97633]: pgmap v15203: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:24:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:22.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:22.848123+0000) 2022-09-10T12:24:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:23.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:23.320503+0000) 2022-09-10T12:24:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:23.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:23.848331+0000) 2022-09-10T12:24:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:24.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:24.320648+0000) 2022-09-10T12:24:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:24 smithi104 ceph-mon[97044]: pgmap v15204: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:24:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:24 smithi110 ceph-mon[97633]: pgmap v15204: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:24:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:24.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:24.848514+0000) 2022-09-10T12:24:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:25.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:25.320812+0000) 2022-09-10T12:24:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:25.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:25.848681+0000) 2022-09-10T12:24:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:26.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:26.320984+0000) 2022-09-10T12:24:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:26 smithi104 ceph-mon[97044]: pgmap v15205: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:24:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:26 smithi110 ceph-mon[97633]: pgmap v15205: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:24:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:26.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:26.848793+0000) 2022-09-10T12:24:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:27.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:27.321203+0000) 2022-09-10T12:24:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:27.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:27.848938+0000) 2022-09-10T12:24:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:28.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:28.321424+0000) 2022-09-10T12:24:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:28 smithi104 ceph-mon[97044]: pgmap v15206: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:28 smithi110 ceph-mon[97633]: pgmap v15206: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:28.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:28.849134+0000) 2022-09-10T12:24:29.692 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:29.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:29.321592+0000) 2022-09-10T12:24:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:29.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:29.849314+0000) 2022-09-10T12:24:30.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:24:29.691+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:24:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:30.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:30.321852+0000) 2022-09-10T12:24:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:30 smithi110 ceph-mon[97633]: pgmap v15207: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:30 smithi104 ceph-mon[97044]: pgmap v15207: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:30.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:30.849508+0000) 2022-09-10T12:24:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:31.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:31.322129+0000) 2022-09-10T12:24:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:31.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:31.849684+0000) 2022-09-10T12:24:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:32.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:32.322320+0000) 2022-09-10T12:24:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:32 smithi110 ceph-mon[97633]: pgmap v15208: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:32 smithi104 ceph-mon[97044]: pgmap v15208: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:32.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:32.849860+0000) 2022-09-10T12:24:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:33.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:33.322550+0000) 2022-09-10T12:24:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:33.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:33.850041+0000) 2022-09-10T12:24:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:34.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:34.322782+0000) 2022-09-10T12:24:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:34 smithi110 ceph-mon[97633]: pgmap v15209: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:34 smithi104 ceph-mon[97044]: pgmap v15209: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:34.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:34.850322+0000) 2022-09-10T12:24:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:35.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:35.323037+0000) 2022-09-10T12:24:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:35.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:35.850559+0000) 2022-09-10T12:24:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:36.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:36.323263+0000) 2022-09-10T12:24:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:36 smithi110 ceph-mon[97633]: pgmap v15210: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:36 smithi104 ceph-mon[97044]: pgmap v15210: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:36.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:36.850733+0000) 2022-09-10T12:24:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:37.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:37.323482+0000) 2022-09-10T12:24:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:37.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:37.850989+0000) 2022-09-10T12:24:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:38.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:38.323674+0000) 2022-09-10T12:24:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:38 smithi110 ceph-mon[97633]: pgmap v15211: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:38 smithi104 ceph-mon[97044]: pgmap v15211: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:38.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:38.851186+0000) 2022-09-10T12:24:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:39.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:39.323908+0000) 2022-09-10T12:24:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:39.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:39.851392+0000) 2022-09-10T12:24:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:40.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:40.324178+0000) 2022-09-10T12:24:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:40 smithi110 ceph-mon[97633]: pgmap v15212: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:40 smithi104 ceph-mon[97044]: pgmap v15212: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:40.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:40.851642+0000) 2022-09-10T12:24:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:41.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:41.324392+0000) 2022-09-10T12:24:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:41.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:41.851897+0000) 2022-09-10T12:24:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:42.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:42.324612+0000) 2022-09-10T12:24:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:42 smithi110 ceph-mon[97633]: pgmap v15213: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:42 smithi104 ceph-mon[97044]: pgmap v15213: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:42.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:42.852076+0000) 2022-09-10T12:24:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:43.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:43.324856+0000) 2022-09-10T12:24:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:43.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:43.852250+0000) 2022-09-10T12:24:44.699 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:44.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:44.325021+0000) 2022-09-10T12:24:45.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:24:44.699+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:24:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:44.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:44.852439+0000) 2022-09-10T12:24:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:44 smithi110 ceph-mon[97633]: pgmap v15214: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:44 smithi104 ceph-mon[97044]: pgmap v15214: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:45.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:45.325251+0000) 2022-09-10T12:24:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:45.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:45.852720+0000) 2022-09-10T12:24:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:46.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:46.325478+0000) 2022-09-10T12:24:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:46 smithi110 ceph-mon[97633]: pgmap v15215: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:24:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:24:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:24:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:24:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:46 smithi104 ceph-mon[97044]: pgmap v15215: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:24:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:24:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:24:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:24:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:46.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:46.852880+0000) 2022-09-10T12:24:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:47.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:47.325621+0000) 2022-09-10T12:24:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:47.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:47.853163+0000) 2022-09-10T12:24:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:48.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:48.325799+0000) 2022-09-10T12:24:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:48 smithi110 ceph-mon[97633]: pgmap v15216: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:48 smithi104 ceph-mon[97044]: pgmap v15216: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:48.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:48.853377+0000) 2022-09-10T12:24:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:49.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:49.326028+0000) 2022-09-10T12:24:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:49.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:49.853639+0000) 2022-09-10T12:24:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:50.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:50.326255+0000) 2022-09-10T12:24:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:50 smithi110 ceph-mon[97633]: pgmap v15217: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:50 smithi104 ceph-mon[97044]: pgmap v15217: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:50.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:50.853952+0000) 2022-09-10T12:24:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:51.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:51.326506+0000) 2022-09-10T12:24:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:51.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:51.854160+0000) 2022-09-10T12:24:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:52.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:52.326708+0000) 2022-09-10T12:24:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:52 smithi110 ceph-mon[97633]: pgmap v15218: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:52 smithi104 ceph-mon[97044]: pgmap v15218: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:52.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:52.854408+0000) 2022-09-10T12:24:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:53.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:53.326968+0000) 2022-09-10T12:24:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:53.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:53.854636+0000) 2022-09-10T12:24:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:54.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:54.327214+0000) 2022-09-10T12:24:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:54 smithi110 ceph-mon[97633]: pgmap v15219: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:54 smithi104 ceph-mon[97044]: pgmap v15219: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:54.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:54.854882+0000) 2022-09-10T12:24:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:55.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:55.327425+0000) 2022-09-10T12:24:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:55.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:55.855109+0000) 2022-09-10T12:24:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:56.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:56.327576+0000) 2022-09-10T12:24:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:56 smithi110 ceph-mon[97633]: pgmap v15220: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:56 smithi104 ceph-mon[97044]: pgmap v15220: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:24:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:56.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:56.855259+0000) 2022-09-10T12:24:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:57.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:57.327789+0000) 2022-09-10T12:24:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:57.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:57.855463+0000) 2022-09-10T12:24:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:58.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:58.328024+0000) 2022-09-10T12:24:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:24:58 smithi110 ceph-mon[97633]: pgmap v15221: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:24:58 smithi104 ceph-mon[97044]: pgmap v15221: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:24:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:58.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:58.855597+0000) 2022-09-10T12:24:59.711 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:24:59.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:59.328255+0000) 2022-09-10T12:25:00.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:24:59.711+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:25:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:24:59.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:24:59.855824+0000) 2022-09-10T12:25:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:00.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:00.328408+0000) 2022-09-10T12:25:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:00 smithi110 ceph-mon[97633]: pgmap v15222: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:00 smithi104 ceph-mon[97044]: pgmap v15222: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:00.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:00.856024+0000) 2022-09-10T12:25:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:01.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:01.328662+0000) 2022-09-10T12:25:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:01.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:01.856211+0000) 2022-09-10T12:25:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:02.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:02.328844+0000) 2022-09-10T12:25:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:02 smithi110 ceph-mon[97633]: pgmap v15223: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:02 smithi104 ceph-mon[97044]: pgmap v15223: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:02.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:02.856374+0000) 2022-09-10T12:25:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:03.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:03.329066+0000) 2022-09-10T12:25:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:03.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:03.856567+0000) 2022-09-10T12:25:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:04.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:04.329226+0000) 2022-09-10T12:25:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:04 smithi110 ceph-mon[97633]: pgmap v15224: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:04 smithi104 ceph-mon[97044]: pgmap v15224: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:04.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:04.856757+0000) 2022-09-10T12:25:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:05.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:05.329399+0000) 2022-09-10T12:25:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:05.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:05.856966+0000) 2022-09-10T12:25:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:06.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:06.329588+0000) 2022-09-10T12:25:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:06 smithi110 ceph-mon[97633]: pgmap v15225: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:25:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:06 smithi104 ceph-mon[97044]: pgmap v15225: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:25:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:06.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:06.857193+0000) 2022-09-10T12:25:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:07.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:07.329829+0000) 2022-09-10T12:25:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:07.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:07.857377+0000) 2022-09-10T12:25:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:08.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:08.330146+0000) 2022-09-10T12:25:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:08 smithi110 ceph-mon[97633]: pgmap v15226: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:08 smithi110 ceph-mon[97633]: osdmap e1032: 12 total, 12 up, 12 in 2022-09-10T12:25:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:08 smithi104 ceph-mon[97044]: pgmap v15226: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:25:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:08 smithi104 ceph-mon[97044]: osdmap e1032: 12 total, 12 up, 12 in 2022-09-10T12:25:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:08.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:08.857610+0000) 2022-09-10T12:25:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:09.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:09.330365+0000) 2022-09-10T12:25:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:09.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:09.857838+0000) 2022-09-10T12:25:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:10.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:10.330508+0000) 2022-09-10T12:25:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:10 smithi110 ceph-mon[97633]: pgmap v15228: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:25:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:10 smithi104 ceph-mon[97044]: pgmap v15228: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:25:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:10.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:10.858079+0000) 2022-09-10T12:25:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:11.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:11.330659+0000) 2022-09-10T12:25:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:11.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:11.858324+0000) 2022-09-10T12:25:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:12.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:12.330859+0000) 2022-09-10T12:25:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:12 smithi110 ceph-mon[97633]: pgmap v15229: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:25:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:12 smithi104 ceph-mon[97044]: pgmap v15229: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:25:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:12.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:12.858551+0000) 2022-09-10T12:25:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:13.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:13.330997+0000) 2022-09-10T12:25:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:13.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:13.858795+0000) 2022-09-10T12:25:14.718 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:14.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:14.331142+0000) 2022-09-10T12:25:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:14.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:14.859027+0000) 2022-09-10T12:25:15.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:25:14.717+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:25:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:14 smithi110 ceph-mon[97633]: pgmap v15230: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:25:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:14 smithi110 ceph-mon[97633]: osdmap e1033: 12 total, 12 up, 12 in 2022-09-10T12:25:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:14 smithi104 ceph-mon[97044]: pgmap v15230: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:25:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:14 smithi104 ceph-mon[97044]: osdmap e1033: 12 total, 12 up, 12 in 2022-09-10T12:25:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:15.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:15.331267+0000) 2022-09-10T12:25:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:15.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:15.859250+0000) 2022-09-10T12:25:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:16.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:16.331387+0000) 2022-09-10T12:25:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:16 smithi110 ceph-mon[97633]: pgmap v15232: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:25:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:16 smithi104 ceph-mon[97044]: pgmap v15232: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:25:17.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:16.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:16.859449+0000) 2022-09-10T12:25:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:17.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:17.331558+0000) 2022-09-10T12:25:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:17.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:17.859641+0000) 2022-09-10T12:25:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:18.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:18.331685+0000) 2022-09-10T12:25:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:18 smithi110 ceph-mon[97633]: pgmap v15233: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 225 B/s wr, 11 op/s 2022-09-10T12:25:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:18 smithi104 ceph-mon[97044]: pgmap v15233: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 225 B/s wr, 11 op/s 2022-09-10T12:25:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:18.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:18.859827+0000) 2022-09-10T12:25:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:19.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:19.331816+0000) 2022-09-10T12:25:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:19.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:19.860051+0000) 2022-09-10T12:25:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:20.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:20.331977+0000) 2022-09-10T12:25:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:20 smithi110 ceph-mon[97633]: pgmap v15234: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:25:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:20 smithi104 ceph-mon[97044]: pgmap v15234: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:25:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:20.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:20.860275+0000) 2022-09-10T12:25:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:21.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:21.332130+0000) 2022-09-10T12:25:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:21.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:21.860435+0000) 2022-09-10T12:25:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:22.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:22.332253+0000) 2022-09-10T12:25:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:22 smithi110 ceph-mon[97633]: pgmap v15235: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:25:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:22 smithi104 ceph-mon[97044]: pgmap v15235: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:25:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:22.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:22.860623+0000) 2022-09-10T12:25:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:23.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:23.332378+0000) 2022-09-10T12:25:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:23.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:23.860835+0000) 2022-09-10T12:25:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:24.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:24.332544+0000) 2022-09-10T12:25:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:24 smithi110 ceph-mon[97633]: pgmap v15236: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:25:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:24 smithi104 ceph-mon[97044]: pgmap v15236: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:25:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:24.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:24.861126+0000) 2022-09-10T12:25:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:25.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:25.332709+0000) 2022-09-10T12:25:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:25.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:25.861302+0000) 2022-09-10T12:25:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:26.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:26.332900+0000) 2022-09-10T12:25:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:26 smithi110 ceph-mon[97633]: pgmap v15237: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:25:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:26 smithi104 ceph-mon[97044]: pgmap v15237: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:25:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:26.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:26.861541+0000) 2022-09-10T12:25:27.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:27.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:27.333040+0000) 2022-09-10T12:25:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:27.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:27.861740+0000) 2022-09-10T12:25:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:28.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:28.333232+0000) 2022-09-10T12:25:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:28 smithi110 ceph-mon[97633]: pgmap v15238: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:25:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:28 smithi104 ceph-mon[97044]: pgmap v15238: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:25:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:28.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:28.861938+0000) 2022-09-10T12:25:29.727 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:29.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:29.333398+0000) 2022-09-10T12:25:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:29.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:29.862212+0000) 2022-09-10T12:25:30.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:25:29.726+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:25:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:30.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:30.333577+0000) 2022-09-10T12:25:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:30 smithi110 ceph-mon[97633]: pgmap v15239: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:25:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:30 smithi104 ceph-mon[97044]: pgmap v15239: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:25:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:30.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:30.862418+0000) 2022-09-10T12:25:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:31.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:31.333723+0000) 2022-09-10T12:25:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:31.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:31.862594+0000) 2022-09-10T12:25:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:32.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:32.333847+0000) 2022-09-10T12:25:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:32 smithi110 ceph-mon[97633]: pgmap v15240: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:32 smithi104 ceph-mon[97044]: pgmap v15240: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:32.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:32.862773+0000) 2022-09-10T12:25:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:33.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:33.334005+0000) 2022-09-10T12:25:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:33.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:33.862945+0000) 2022-09-10T12:25:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:34.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:34.334183+0000) 2022-09-10T12:25:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:34 smithi110 ceph-mon[97633]: pgmap v15241: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:25:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:34 smithi104 ceph-mon[97044]: pgmap v15241: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:25:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:34.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:34.863187+0000) 2022-09-10T12:25:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:35.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:35.334423+0000) 2022-09-10T12:25:36.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:35 smithi110 ceph-mon[97633]: pgmap v15242: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:36.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:35 smithi104 ceph-mon[97044]: pgmap v15242: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:35.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:35.863323+0000) 2022-09-10T12:25:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:36.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:36.334650+0000) 2022-09-10T12:25:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:36.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:36.863541+0000) 2022-09-10T12:25:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:37.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:37.334803+0000) 2022-09-10T12:25:38.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:37 smithi104 ceph-mon[97044]: pgmap v15243: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:37.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:37.863787+0000) 2022-09-10T12:25:38.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:37 smithi110 ceph-mon[97633]: pgmap v15243: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:38.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:38.335037+0000) 2022-09-10T12:25:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:38.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:38.864036+0000) 2022-09-10T12:25:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:39.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:39.335313+0000) 2022-09-10T12:25:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:39.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:39.864244+0000) 2022-09-10T12:25:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:40.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:40.335531+0000) 2022-09-10T12:25:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:40 smithi110 ceph-mon[97633]: pgmap v15244: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:40 smithi104 ceph-mon[97044]: pgmap v15244: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:40.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:40.864449+0000) 2022-09-10T12:25:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:41.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:41.335754+0000) 2022-09-10T12:25:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:41.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:41.864642+0000) 2022-09-10T12:25:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:42.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:42.335988+0000) 2022-09-10T12:25:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:42 smithi110 ceph-mon[97633]: pgmap v15245: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:42 smithi104 ceph-mon[97044]: pgmap v15245: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:43.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:42.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:42.864894+0000) 2022-09-10T12:25:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:43.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:43.336177+0000) 2022-09-10T12:25:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:43.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:43.865187+0000) 2022-09-10T12:25:44.738 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:44.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:44.336355+0000) 2022-09-10T12:25:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:44 smithi110 ceph-mon[97633]: pgmap v15246: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:44 smithi104 ceph-mon[97044]: pgmap v15246: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:45.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:25:44.738+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:25:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:44.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:44.865409+0000) 2022-09-10T12:25:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:45.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:45.336568+0000) 2022-09-10T12:25:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:45.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:45.865697+0000) 2022-09-10T12:25:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:46.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:46.336777+0000) 2022-09-10T12:25:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:46 smithi104 ceph-mon[97044]: pgmap v15247: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:25:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:25:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:25:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:25:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:46 smithi110 ceph-mon[97633]: pgmap v15247: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:25:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:25:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:46.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:46.865938+0000) 2022-09-10T12:25:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:47.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:47.337027+0000) 2022-09-10T12:25:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:47.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:47.866142+0000) 2022-09-10T12:25:48.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:48.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:48.337273+0000) 2022-09-10T12:25:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:48 smithi104 ceph-mon[97044]: pgmap v15248: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:48 smithi110 ceph-mon[97633]: pgmap v15248: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:48.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:48.866345+0000) 2022-09-10T12:25:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:49.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:49.337537+0000) 2022-09-10T12:25:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:49.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:49.866554+0000) 2022-09-10T12:25:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:50.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:50.337733+0000) 2022-09-10T12:25:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:50 smithi104 ceph-mon[97044]: pgmap v15249: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:50 smithi110 ceph-mon[97633]: pgmap v15249: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:50.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:50.866754+0000) 2022-09-10T12:25:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:51.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:51.337980+0000) 2022-09-10T12:25:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:51.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:51.866928+0000) 2022-09-10T12:25:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:52.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:52.338165+0000) 2022-09-10T12:25:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:52 smithi104 ceph-mon[97044]: pgmap v15250: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:52 smithi110 ceph-mon[97633]: pgmap v15250: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:52.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:52.867092+0000) 2022-09-10T12:25:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:53.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:53.338340+0000) 2022-09-10T12:25:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:53.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:53.867315+0000) 2022-09-10T12:25:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:54.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:54.338575+0000) 2022-09-10T12:25:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:54 smithi104 ceph-mon[97044]: pgmap v15251: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:54 smithi110 ceph-mon[97633]: pgmap v15251: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:54.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:54.867536+0000) 2022-09-10T12:25:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:55.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:55.338820+0000) 2022-09-10T12:25:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:55.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:55.867767+0000) 2022-09-10T12:25:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:56.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:56.339001+0000) 2022-09-10T12:25:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:56 smithi104 ceph-mon[97044]: pgmap v15252: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:56 smithi110 ceph-mon[97633]: pgmap v15252: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:25:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:56.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:56.868019+0000) 2022-09-10T12:25:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:57.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:57.339232+0000) 2022-09-10T12:25:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:57.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:57.868233+0000) 2022-09-10T12:25:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:58.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:58.339455+0000) 2022-09-10T12:25:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:25:58 smithi104 ceph-mon[97044]: pgmap v15253: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:25:58 smithi110 ceph-mon[97633]: pgmap v15253: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:25:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:58.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:58.868419+0000) 2022-09-10T12:25:59.754 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:25:59.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:59.339717+0000) 2022-09-10T12:26:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:25:59.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:25:59.868607+0000) 2022-09-10T12:26:00.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:25:59.753+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:26:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:00.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:00.339922+0000) 2022-09-10T12:26:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:00 smithi104 ceph-mon[97044]: pgmap v15254: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:00 smithi110 ceph-mon[97633]: pgmap v15254: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:00.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:00.868825+0000) 2022-09-10T12:26:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:01.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:01.340151+0000) 2022-09-10T12:26:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:01.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:01.869007+0000) 2022-09-10T12:26:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:02.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:02.340331+0000) 2022-09-10T12:26:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:02 smithi104 ceph-mon[97044]: pgmap v15255: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:02 smithi110 ceph-mon[97633]: pgmap v15255: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:02.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:02.869192+0000) 2022-09-10T12:26:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:03.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:03.340586+0000) 2022-09-10T12:26:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:03.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:03.869354+0000) 2022-09-10T12:26:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:04.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:04.340703+0000) 2022-09-10T12:26:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:04 smithi104 ceph-mon[97044]: pgmap v15256: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:04 smithi110 ceph-mon[97633]: pgmap v15256: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:04.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:04.869579+0000) 2022-09-10T12:26:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:05.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:05.340931+0000) 2022-09-10T12:26:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:05.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:05.869839+0000) 2022-09-10T12:26:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:06.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:06.341198+0000) 2022-09-10T12:26:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:06 smithi104 ceph-mon[97044]: pgmap v15257: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:26:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:06 smithi110 ceph-mon[97633]: pgmap v15257: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:26:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:06.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:06.870085+0000) 2022-09-10T12:26:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:07.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:07.341366+0000) 2022-09-10T12:26:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:07.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:07.870293+0000) 2022-09-10T12:26:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:08.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:08.341551+0000) 2022-09-10T12:26:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:08 smithi104 ceph-mon[97044]: pgmap v15258: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:26:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:08 smithi104 ceph-mon[97044]: osdmap e1034: 12 total, 12 up, 12 in 2022-09-10T12:26:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:08 smithi110 ceph-mon[97633]: pgmap v15258: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:26:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:08 smithi110 ceph-mon[97633]: osdmap e1034: 12 total, 12 up, 12 in 2022-09-10T12:26:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:08.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:08.870499+0000) 2022-09-10T12:26:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:09.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:09.341706+0000) 2022-09-10T12:26:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:09.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:09.870640+0000) 2022-09-10T12:26:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:10.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:10.341918+0000) 2022-09-10T12:26:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:10 smithi104 ceph-mon[97044]: pgmap v15260: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:26:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:10 smithi110 ceph-mon[97633]: pgmap v15260: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:26:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:10.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:10.870855+0000) 2022-09-10T12:26:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:11.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:11.342129+0000) 2022-09-10T12:26:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:11.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:11.871151+0000) 2022-09-10T12:26:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:12.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:12.342309+0000) 2022-09-10T12:26:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:12 smithi104 ceph-mon[97044]: pgmap v15261: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:26:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:12 smithi110 ceph-mon[97633]: pgmap v15261: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:26:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:12.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:12.871311+0000) 2022-09-10T12:26:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:13.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:13.342547+0000) 2022-09-10T12:26:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:13.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:13.871495+0000) 2022-09-10T12:26:14.758 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:14.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:14.342778+0000) 2022-09-10T12:26:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:14 smithi104 ceph-mon[97044]: pgmap v15262: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:26:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:14 smithi104 ceph-mon[97044]: osdmap e1035: 12 total, 12 up, 12 in 2022-09-10T12:26:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:14.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:14.871758+0000) 2022-09-10T12:26:15.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:26:14.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:26:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:14 smithi110 ceph-mon[97633]: pgmap v15262: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:26:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:14 smithi110 ceph-mon[97633]: osdmap e1035: 12 total, 12 up, 12 in 2022-09-10T12:26:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:15.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:15.342924+0000) 2022-09-10T12:26:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:15.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:15.872009+0000) 2022-09-10T12:26:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:16.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:16.343084+0000) 2022-09-10T12:26:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:16 smithi104 ceph-mon[97044]: pgmap v15264: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:26:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:16 smithi110 ceph-mon[97633]: pgmap v15264: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:26:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:16.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:16.872211+0000) 2022-09-10T12:26:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:17.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:17.343297+0000) 2022-09-10T12:26:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:17.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:17.872475+0000) 2022-09-10T12:26:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:18.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:18.343539+0000) 2022-09-10T12:26:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:18 smithi104 ceph-mon[97044]: pgmap v15265: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:26:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:18 smithi110 ceph-mon[97633]: pgmap v15265: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:26:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:18.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:18.872719+0000) 2022-09-10T12:26:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:19.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:19.343753+0000) 2022-09-10T12:26:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:19.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:19.872999+0000) 2022-09-10T12:26:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:20.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:20.343994+0000) 2022-09-10T12:26:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:20 smithi104 ceph-mon[97044]: pgmap v15266: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:26:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:20 smithi110 ceph-mon[97633]: pgmap v15266: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:26:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:20.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:20.873248+0000) 2022-09-10T12:26:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:21.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:21.344277+0000) 2022-09-10T12:26:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:21.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:21.873416+0000) 2022-09-10T12:26:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:22.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:22.344507+0000) 2022-09-10T12:26:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:22 smithi104 ceph-mon[97044]: pgmap v15267: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:26:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:22 smithi110 ceph-mon[97633]: pgmap v15267: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:26:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:22.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:22.873649+0000) 2022-09-10T12:26:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:23.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:23.344692+0000) 2022-09-10T12:26:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:23.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:23.873822+0000) 2022-09-10T12:26:24.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:24.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:24.344902+0000) 2022-09-10T12:26:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:24 smithi104 ceph-mon[97044]: pgmap v15268: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:26:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:24 smithi110 ceph-mon[97633]: pgmap v15268: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:26:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:24.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:24.874055+0000) 2022-09-10T12:26:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:25.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:25.345183+0000) 2022-09-10T12:26:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:25.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:25.874333+0000) 2022-09-10T12:26:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:26.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:26.345405+0000) 2022-09-10T12:26:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:26 smithi104 ceph-mon[97044]: pgmap v15269: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:26:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:26 smithi110 ceph-mon[97633]: pgmap v15269: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:26:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:26.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:26.874586+0000) 2022-09-10T12:26:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:27.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:27.345584+0000) 2022-09-10T12:26:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:27.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:27.874830+0000) 2022-09-10T12:26:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:28.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:28.345806+0000) 2022-09-10T12:26:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:28 smithi104 ceph-mon[97044]: pgmap v15270: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:28 smithi110 ceph-mon[97633]: pgmap v15270: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:28.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:28.875076+0000) 2022-09-10T12:26:29.760 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:29.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:29.346087+0000) 2022-09-10T12:26:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:29.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:29.875282+0000) 2022-09-10T12:26:30.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:26:29.759+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:26:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:30.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:30.346300+0000) 2022-09-10T12:26:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:30 smithi104 ceph-mon[97044]: pgmap v15271: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:30 smithi110 ceph-mon[97633]: pgmap v15271: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:30.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:30.875421+0000) 2022-09-10T12:26:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:31.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:31.346479+0000) 2022-09-10T12:26:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:31.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:31.875640+0000) 2022-09-10T12:26:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:32.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:32.346701+0000) 2022-09-10T12:26:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:32 smithi104 ceph-mon[97044]: pgmap v15272: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:32 smithi110 ceph-mon[97633]: pgmap v15272: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:32.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:32.875884+0000) 2022-09-10T12:26:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:33.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:33.346897+0000) 2022-09-10T12:26:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:33.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:33.876160+0000) 2022-09-10T12:26:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:34.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:34.347053+0000) 2022-09-10T12:26:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:34 smithi104 ceph-mon[97044]: pgmap v15273: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:34 smithi110 ceph-mon[97633]: pgmap v15273: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:34.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:34.876410+0000) 2022-09-10T12:26:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:35.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:35.347366+0000) 2022-09-10T12:26:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:35.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:35.876609+0000) 2022-09-10T12:26:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:36.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:36.347551+0000) 2022-09-10T12:26:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:36 smithi104 ceph-mon[97044]: pgmap v15274: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:36 smithi110 ceph-mon[97633]: pgmap v15274: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:36.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:36.876824+0000) 2022-09-10T12:26:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:37.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:37.347730+0000) 2022-09-10T12:26:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:37.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:37.877074+0000) 2022-09-10T12:26:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:38.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:38.347913+0000) 2022-09-10T12:26:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:38 smithi104 ceph-mon[97044]: pgmap v15275: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:38 smithi110 ceph-mon[97633]: pgmap v15275: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:38.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:38.877346+0000) 2022-09-10T12:26:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:39.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:39.348174+0000) 2022-09-10T12:26:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:39.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:39.877604+0000) 2022-09-10T12:26:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:40.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:40.348367+0000) 2022-09-10T12:26:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:40 smithi104 ceph-mon[97044]: pgmap v15276: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:40 smithi110 ceph-mon[97633]: pgmap v15276: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:40.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:40.877821+0000) 2022-09-10T12:26:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:41.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:41.348624+0000) 2022-09-10T12:26:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:41.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:41.877999+0000) 2022-09-10T12:26:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:42.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:42.348781+0000) 2022-09-10T12:26:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:42 smithi104 ceph-mon[97044]: pgmap v15277: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:42 smithi110 ceph-mon[97633]: pgmap v15277: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:42.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:42.878186+0000) 2022-09-10T12:26:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:43.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:43.349005+0000) 2022-09-10T12:26:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:43.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:43.878359+0000) 2022-09-10T12:26:44.776 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:44.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:44.349215+0000) 2022-09-10T12:26:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:44 smithi104 ceph-mon[97044]: pgmap v15278: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:45.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:44.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:44.878594+0000) 2022-09-10T12:26:45.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:26:44.775+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:26:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:44 smithi110 ceph-mon[97633]: pgmap v15278: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:45.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:45.349480+0000) 2022-09-10T12:26:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:45.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:45.878736+0000) 2022-09-10T12:26:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:46.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:46.349746+0000) 2022-09-10T12:26:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:46 smithi104 ceph-mon[97044]: pgmap v15279: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:26:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:26:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:26:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:26:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:46 smithi110 ceph-mon[97633]: pgmap v15279: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:26:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:26:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:26:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:26:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:46.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:46.878939+0000) 2022-09-10T12:26:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:47.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:47.349954+0000) 2022-09-10T12:26:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:47.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:47.879170+0000) 2022-09-10T12:26:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:48.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:48.350191+0000) 2022-09-10T12:26:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:48 smithi104 ceph-mon[97044]: pgmap v15280: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:48 smithi110 ceph-mon[97633]: pgmap v15280: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:48.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:48.879351+0000) 2022-09-10T12:26:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:49.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:49.350497+0000) 2022-09-10T12:26:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:49.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:49.879529+0000) 2022-09-10T12:26:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:50.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:50.350706+0000) 2022-09-10T12:26:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:50 smithi104 ceph-mon[97044]: pgmap v15281: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:50 smithi110 ceph-mon[97633]: pgmap v15281: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:50.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:50.879746+0000) 2022-09-10T12:26:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:51.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:51.350863+0000) 2022-09-10T12:26:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:51.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:51.879885+0000) 2022-09-10T12:26:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:52.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:52.351072+0000) 2022-09-10T12:26:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:52 smithi104 ceph-mon[97044]: pgmap v15282: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:52 smithi110 ceph-mon[97633]: pgmap v15282: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:52.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:52.880044+0000) 2022-09-10T12:26:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:53.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:53.351276+0000) 2022-09-10T12:26:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:53.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:53.880315+0000) 2022-09-10T12:26:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:54.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:54.351532+0000) 2022-09-10T12:26:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:54 smithi104 ceph-mon[97044]: pgmap v15283: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:54 smithi110 ceph-mon[97633]: pgmap v15283: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:54.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:54.880554+0000) 2022-09-10T12:26:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:55.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:55.351784+0000) 2022-09-10T12:26:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:55.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:55.880757+0000) 2022-09-10T12:26:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:56.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:56.351950+0000) 2022-09-10T12:26:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:56 smithi104 ceph-mon[97044]: pgmap v15284: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:56 smithi110 ceph-mon[97633]: pgmap v15284: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:26:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:56.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:56.880943+0000) 2022-09-10T12:26:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:57.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:57.352231+0000) 2022-09-10T12:26:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:57.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:57.881134+0000) 2022-09-10T12:26:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:58.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:58.352499+0000) 2022-09-10T12:26:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:26:58 smithi104 ceph-mon[97044]: pgmap v15285: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:26:58 smithi110 ceph-mon[97633]: pgmap v15285: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:26:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:58.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:58.881352+0000) 2022-09-10T12:26:59.792 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:26:59.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:59.352710+0000) 2022-09-10T12:27:00.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:26:59.791+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:27:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:26:59.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:26:59.881528+0000) 2022-09-10T12:27:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:00.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:00.352872+0000) 2022-09-10T12:27:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:00 smithi104 ceph-mon[97044]: pgmap v15286: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:00 smithi110 ceph-mon[97633]: pgmap v15286: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:00.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:00.881774+0000) 2022-09-10T12:27:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:01.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:01.353127+0000) 2022-09-10T12:27:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:01.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:01.881988+0000) 2022-09-10T12:27:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:02.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:02.353356+0000) 2022-09-10T12:27:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:02 smithi104 ceph-mon[97044]: pgmap v15287: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:02 smithi110 ceph-mon[97633]: pgmap v15287: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:02.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:02.882238+0000) 2022-09-10T12:27:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:03.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:03.353591+0000) 2022-09-10T12:27:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:03.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:03.882436+0000) 2022-09-10T12:27:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:04.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:04.353774+0000) 2022-09-10T12:27:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:04 smithi104 ceph-mon[97044]: pgmap v15288: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:04 smithi110 ceph-mon[97633]: pgmap v15288: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:04.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:04.882669+0000) 2022-09-10T12:27:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:05.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:05.353954+0000) 2022-09-10T12:27:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:05.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:05.882895+0000) 2022-09-10T12:27:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:06.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:06.354198+0000) 2022-09-10T12:27:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:06 smithi104 ceph-mon[97044]: pgmap v15289: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:27:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:06 smithi110 ceph-mon[97633]: pgmap v15289: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:27:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:06.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:06.883070+0000) 2022-09-10T12:27:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:07.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:07.354455+0000) 2022-09-10T12:27:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:07.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:07.883274+0000) 2022-09-10T12:27:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:08.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:08.354683+0000) 2022-09-10T12:27:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:08 smithi104 ceph-mon[97044]: pgmap v15290: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:27:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:08 smithi104 ceph-mon[97044]: osdmap e1036: 12 total, 12 up, 12 in 2022-09-10T12:27:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:08 smithi110 ceph-mon[97633]: pgmap v15290: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:27:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:08 smithi110 ceph-mon[97633]: osdmap e1036: 12 total, 12 up, 12 in 2022-09-10T12:27:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:08.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:08.883453+0000) 2022-09-10T12:27:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:09.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:09.354871+0000) 2022-09-10T12:27:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:09.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:09.883668+0000) 2022-09-10T12:27:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:10.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:10.355032+0000) 2022-09-10T12:27:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:10 smithi104 ceph-mon[97044]: pgmap v15292: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:27:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:10 smithi110 ceph-mon[97633]: pgmap v15292: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:27:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:10.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:10.883918+0000) 2022-09-10T12:27:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:11.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:11.355292+0000) 2022-09-10T12:27:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:11.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:11.884187+0000) 2022-09-10T12:27:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:12.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:12.355501+0000) 2022-09-10T12:27:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:12 smithi104 ceph-mon[97044]: pgmap v15293: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:27:13.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:12 smithi110 ceph-mon[97633]: pgmap v15293: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:27:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:12.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:12.884389+0000) 2022-09-10T12:27:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:13.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:13.355652+0000) 2022-09-10T12:27:14.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:13.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:13.884574+0000) 2022-09-10T12:27:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:14.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:14.355814+0000) 2022-09-10T12:27:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:14 smithi104 ceph-mon[97044]: pgmap v15294: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:27:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:14 smithi104 ceph-mon[97044]: osdmap e1037: 12 total, 12 up, 12 in 2022-09-10T12:27:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:14 smithi110 ceph-mon[97633]: pgmap v15294: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:27:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:14 smithi110 ceph-mon[97633]: osdmap e1037: 12 total, 12 up, 12 in 2022-09-10T12:27:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:14.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:14.884778+0000) 2022-09-10T12:27:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:27:14.806+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:27:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:15.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:15.356018+0000) 2022-09-10T12:27:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:15.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:15.884951+0000) 2022-09-10T12:27:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:16.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:16.356248+0000) 2022-09-10T12:27:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:16 smithi104 ceph-mon[97044]: pgmap v15296: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:27:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:16 smithi110 ceph-mon[97633]: pgmap v15296: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:27:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:16.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:16.885176+0000) 2022-09-10T12:27:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:17.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:17.356456+0000) 2022-09-10T12:27:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:17.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:17.885430+0000) 2022-09-10T12:27:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:18.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:18.356686+0000) 2022-09-10T12:27:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:18 smithi110 ceph-mon[97633]: pgmap v15297: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T12:27:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:18 smithi104 ceph-mon[97044]: pgmap v15297: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T12:27:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:18.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:18.885595+0000) 2022-09-10T12:27:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:19.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:19.356874+0000) 2022-09-10T12:27:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:19.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:19.885777+0000) 2022-09-10T12:27:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:20.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:20.357146+0000) 2022-09-10T12:27:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:20 smithi110 ceph-mon[97633]: pgmap v15298: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:27:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:20 smithi104 ceph-mon[97044]: pgmap v15298: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:27:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:20.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:20.886013+0000) 2022-09-10T12:27:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:21.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:21.357327+0000) 2022-09-10T12:27:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:21.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:21.886262+0000) 2022-09-10T12:27:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:22.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:22.357515+0000) 2022-09-10T12:27:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:22 smithi110 ceph-mon[97633]: pgmap v15299: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:27:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:22 smithi104 ceph-mon[97044]: pgmap v15299: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:27:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:22.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:22.886511+0000) 2022-09-10T12:27:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:23.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:23.357684+0000) 2022-09-10T12:27:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:23.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:23.886712+0000) 2022-09-10T12:27:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:24.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:24.357887+0000) 2022-09-10T12:27:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:24 smithi110 ceph-mon[97633]: pgmap v15300: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:27:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:24 smithi104 ceph-mon[97044]: pgmap v15300: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:27:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:24.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:24.886887+0000) 2022-09-10T12:27:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:25.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:25.358118+0000) 2022-09-10T12:27:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:25.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:25.887079+0000) 2022-09-10T12:27:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:26.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:26.358321+0000) 2022-09-10T12:27:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:26 smithi110 ceph-mon[97633]: pgmap v15301: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:27:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:26 smithi104 ceph-mon[97044]: pgmap v15301: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:27:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:26.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:26.887234+0000) 2022-09-10T12:27:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:27.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:27.358562+0000) 2022-09-10T12:27:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:27.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:27.887377+0000) 2022-09-10T12:27:28.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:28.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:28.358717+0000) 2022-09-10T12:27:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:28 smithi110 ceph-mon[97633]: pgmap v15302: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:28 smithi104 ceph-mon[97044]: pgmap v15302: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:28.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:28.887611+0000) 2022-09-10T12:27:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:29.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:29.359014+0000) 2022-09-10T12:27:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:29.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:29.887838+0000) 2022-09-10T12:27:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:27:29.823+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:27:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:30.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:30.359204+0000) 2022-09-10T12:27:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:30 smithi110 ceph-mon[97633]: pgmap v15303: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:30 smithi104 ceph-mon[97044]: pgmap v15303: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:30.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:30.888035+0000) 2022-09-10T12:27:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:31.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:31.359434+0000) 2022-09-10T12:27:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:31.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:31.888316+0000) 2022-09-10T12:27:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:32.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:32.359591+0000) 2022-09-10T12:27:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:32 smithi110 ceph-mon[97633]: pgmap v15304: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:32 smithi104 ceph-mon[97044]: pgmap v15304: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:32.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:32.888570+0000) 2022-09-10T12:27:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:33.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:33.359802+0000) 2022-09-10T12:27:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:33.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:33.888737+0000) 2022-09-10T12:27:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:34.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:34.360052+0000) 2022-09-10T12:27:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:34 smithi110 ceph-mon[97633]: pgmap v15305: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:34 smithi104 ceph-mon[97044]: pgmap v15305: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:34.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:34.888992+0000) 2022-09-10T12:27:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:35.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:35.360284+0000) 2022-09-10T12:27:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:35.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:35.889243+0000) 2022-09-10T12:27:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:36.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:36.360450+0000) 2022-09-10T12:27:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:36 smithi110 ceph-mon[97633]: pgmap v15306: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:36 smithi104 ceph-mon[97044]: pgmap v15306: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:36.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:36.889449+0000) 2022-09-10T12:27:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:37.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:37.360687+0000) 2022-09-10T12:27:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:37.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:37.889653+0000) 2022-09-10T12:27:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:38.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:38.360930+0000) 2022-09-10T12:27:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:38 smithi110 ceph-mon[97633]: pgmap v15307: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:38 smithi104 ceph-mon[97044]: pgmap v15307: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:38.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:38.889922+0000) 2022-09-10T12:27:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:39.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:39.361190+0000) 2022-09-10T12:27:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:39.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:39.890211+0000) 2022-09-10T12:27:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:40.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:40.361366+0000) 2022-09-10T12:27:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:40 smithi110 ceph-mon[97633]: pgmap v15308: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:40 smithi104 ceph-mon[97044]: pgmap v15308: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:40.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:40.890369+0000) 2022-09-10T12:27:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:41.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:41.361545+0000) 2022-09-10T12:27:42.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:41.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:41.890599+0000) 2022-09-10T12:27:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:42.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:42.361769+0000) 2022-09-10T12:27:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:42 smithi110 ceph-mon[97633]: pgmap v15309: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:42 smithi104 ceph-mon[97044]: pgmap v15309: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:42.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:42.890790+0000) 2022-09-10T12:27:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:43.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:43.362017+0000) 2022-09-10T12:27:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:43.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:43.890967+0000) 2022-09-10T12:27:44.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:44.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:44.362276+0000) 2022-09-10T12:27:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:44 smithi110 ceph-mon[97633]: pgmap v15310: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:44 smithi104 ceph-mon[97044]: pgmap v15310: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:44.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:44.891199+0000) 2022-09-10T12:27:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:27:44.834+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:27:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:45.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:45.362488+0000) 2022-09-10T12:27:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:45.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:45.891398+0000) 2022-09-10T12:27:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:46.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:46.362719+0000) 2022-09-10T12:27:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:46 smithi110 ceph-mon[97633]: pgmap v15311: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:27:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:27:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:46 smithi104 ceph-mon[97044]: pgmap v15311: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:27:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:27:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:27:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:27:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:46.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:46.891627+0000) 2022-09-10T12:27:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:47.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:47.362937+0000) 2022-09-10T12:27:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:47.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:47.891897+0000) 2022-09-10T12:27:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:48.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:48.363194+0000) 2022-09-10T12:27:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:48 smithi110 ceph-mon[97633]: pgmap v15312: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:48 smithi104 ceph-mon[97044]: pgmap v15312: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:48.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:48.892173+0000) 2022-09-10T12:27:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:49.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:49.363391+0000) 2022-09-10T12:27:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:49.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:49.892346+0000) 2022-09-10T12:27:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:50.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:50.363535+0000) 2022-09-10T12:27:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:50 smithi110 ceph-mon[97633]: pgmap v15313: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:50 smithi104 ceph-mon[97044]: pgmap v15313: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:50.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:50.892567+0000) 2022-09-10T12:27:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:51.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:51.363724+0000) 2022-09-10T12:27:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:51.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:51.892836+0000) 2022-09-10T12:27:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:52.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:52.363948+0000) 2022-09-10T12:27:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:52 smithi110 ceph-mon[97633]: pgmap v15314: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:52 smithi104 ceph-mon[97044]: pgmap v15314: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:52.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:52.893072+0000) 2022-09-10T12:27:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:53.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:53.364202+0000) 2022-09-10T12:27:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:53.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:53.893284+0000) 2022-09-10T12:27:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:54.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:54.364417+0000) 2022-09-10T12:27:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:54 smithi110 ceph-mon[97633]: pgmap v15315: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:54 smithi104 ceph-mon[97044]: pgmap v15315: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:54.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:54.893529+0000) 2022-09-10T12:27:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:55.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:55.364675+0000) 2022-09-10T12:27:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:55.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:55.893782+0000) 2022-09-10T12:27:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:56.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:56.364914+0000) 2022-09-10T12:27:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:56 smithi110 ceph-mon[97633]: pgmap v15316: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:56 smithi104 ceph-mon[97044]: pgmap v15316: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:27:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:56.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:56.894060+0000) 2022-09-10T12:27:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:57.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:57.365163+0000) 2022-09-10T12:27:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:57.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:57.894303+0000) 2022-09-10T12:27:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:58.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:58.365328+0000) 2022-09-10T12:27:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:27:58 smithi110 ceph-mon[97633]: pgmap v15317: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:27:58 smithi104 ceph-mon[97044]: pgmap v15317: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:27:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:58.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:58.894474+0000) 2022-09-10T12:27:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:27:59.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:59.365512+0000) 2022-09-10T12:28:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:27:59.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:27:59.894687+0000) 2022-09-10T12:28:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:27:59.835+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:28:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:00.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:00.365692+0000) 2022-09-10T12:28:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:00 smithi110 ceph-mon[97633]: pgmap v15318: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:00 smithi104 ceph-mon[97044]: pgmap v15318: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:00.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:00.894821+0000) 2022-09-10T12:28:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:01.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:01.365857+0000) 2022-09-10T12:28:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:01.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:01.895021+0000) 2022-09-10T12:28:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:02.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:02.366080+0000) 2022-09-10T12:28:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:02 smithi110 ceph-mon[97633]: pgmap v15319: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:02 smithi104 ceph-mon[97044]: pgmap v15319: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:02.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:02.895232+0000) 2022-09-10T12:28:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:03.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:03.366361+0000) 2022-09-10T12:28:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:03.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:03.895388+0000) 2022-09-10T12:28:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:04.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:04.366582+0000) 2022-09-10T12:28:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:04 smithi110 ceph-mon[97633]: pgmap v15320: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:04 smithi104 ceph-mon[97044]: pgmap v15320: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:04.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:04.895613+0000) 2022-09-10T12:28:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:05.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:05.366851+0000) 2022-09-10T12:28:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:05.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:05.895853+0000) 2022-09-10T12:28:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:06.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:06.367032+0000) 2022-09-10T12:28:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:06 smithi110 ceph-mon[97633]: pgmap v15321: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:28:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:06 smithi104 ceph-mon[97044]: pgmap v15321: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:28:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:06.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:06.896078+0000) 2022-09-10T12:28:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:07.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:07.367278+0000) 2022-09-10T12:28:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:07.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:07.896269+0000) 2022-09-10T12:28:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:08.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:08.367516+0000) 2022-09-10T12:28:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:08 smithi110 ceph-mon[97633]: pgmap v15322: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T12:28:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:08 smithi110 ceph-mon[97633]: osdmap e1038: 12 total, 12 up, 12 in 2022-09-10T12:28:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:08 smithi104 ceph-mon[97044]: pgmap v15322: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T12:28:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:08 smithi104 ceph-mon[97044]: osdmap e1038: 12 total, 12 up, 12 in 2022-09-10T12:28:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:08.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:08.896464+0000) 2022-09-10T12:28:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:09.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:09.367746+0000) 2022-09-10T12:28:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:09.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:09.896705+0000) 2022-09-10T12:28:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:10.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:10.367930+0000) 2022-09-10T12:28:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:10 smithi110 ceph-mon[97633]: pgmap v15324: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:28:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:10 smithi104 ceph-mon[97044]: pgmap v15324: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:28:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:10.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:10.896856+0000) 2022-09-10T12:28:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:11.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:11.368174+0000) 2022-09-10T12:28:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:11.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:11.897068+0000) 2022-09-10T12:28:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:12.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:12.368382+0000) 2022-09-10T12:28:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:12 smithi110 ceph-mon[97633]: pgmap v15325: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:28:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:12 smithi104 ceph-mon[97044]: pgmap v15325: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:28:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:12.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:12.897161+0000) 2022-09-10T12:28:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:13.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:13.368590+0000) 2022-09-10T12:28:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:13.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:13.897346+0000) 2022-09-10T12:28:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:14.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:14.368742+0000) 2022-09-10T12:28:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:14 smithi110 ceph-mon[97633]: pgmap v15326: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:28:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:14 smithi110 ceph-mon[97633]: osdmap e1039: 12 total, 12 up, 12 in 2022-09-10T12:28:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:14 smithi104 ceph-mon[97044]: pgmap v15326: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:28:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:14 smithi104 ceph-mon[97044]: osdmap e1039: 12 total, 12 up, 12 in 2022-09-10T12:28:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:14.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:14.897538+0000) 2022-09-10T12:28:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:28:14.841+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:28:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:15.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:15.368926+0000) 2022-09-10T12:28:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:15.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:15.897756+0000) 2022-09-10T12:28:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:16.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:16.369156+0000) 2022-09-10T12:28:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:16 smithi110 ceph-mon[97633]: pgmap v15328: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:28:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:16 smithi104 ceph-mon[97044]: pgmap v15328: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:28:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:16.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:16.898002+0000) 2022-09-10T12:28:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:17.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:17.369378+0000) 2022-09-10T12:28:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:17.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:17.898234+0000) 2022-09-10T12:28:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:18.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:18.369564+0000) 2022-09-10T12:28:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:18 smithi110 ceph-mon[97633]: pgmap v15329: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 224 B/s wr, 11 op/s 2022-09-10T12:28:19.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:18 smithi104 ceph-mon[97044]: pgmap v15329: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 224 B/s wr, 11 op/s 2022-09-10T12:28:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:18.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:18.898443+0000) 2022-09-10T12:28:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:19.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:19.369715+0000) 2022-09-10T12:28:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:19.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:19.898639+0000) 2022-09-10T12:28:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:20.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:20.369949+0000) 2022-09-10T12:28:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:20 smithi110 ceph-mon[97633]: pgmap v15330: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:28:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:20 smithi104 ceph-mon[97044]: pgmap v15330: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:28:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:20.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:20.898841+0000) 2022-09-10T12:28:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:21.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:21.370166+0000) 2022-09-10T12:28:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:21.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:21.899013+0000) 2022-09-10T12:28:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:22.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:22.370417+0000) 2022-09-10T12:28:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:22 smithi110 ceph-mon[97633]: pgmap v15331: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:28:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:22 smithi104 ceph-mon[97044]: pgmap v15331: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:28:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:22.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:22.899289+0000) 2022-09-10T12:28:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:23.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:23.370619+0000) 2022-09-10T12:28:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:23.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:23.899529+0000) 2022-09-10T12:28:24.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:24.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:24.370800+0000) 2022-09-10T12:28:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:24 smithi110 ceph-mon[97633]: pgmap v15332: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:28:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:24 smithi104 ceph-mon[97044]: pgmap v15332: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:28:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:24.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:24.899750+0000) 2022-09-10T12:28:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:25.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:25.371030+0000) 2022-09-10T12:28:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:25.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:25.900008+0000) 2022-09-10T12:28:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:26.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:26.371262+0000) 2022-09-10T12:28:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:26 smithi110 ceph-mon[97633]: pgmap v15333: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:28:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:26 smithi104 ceph-mon[97044]: pgmap v15333: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:28:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:26.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:26.900282+0000) 2022-09-10T12:28:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:27.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:27.371453+0000) 2022-09-10T12:28:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:27.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:27.900471+0000) 2022-09-10T12:28:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:28.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:28.371680+0000) 2022-09-10T12:28:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:28 smithi110 ceph-mon[97633]: pgmap v15334: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:28 smithi104 ceph-mon[97044]: pgmap v15334: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:28.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:28.900632+0000) 2022-09-10T12:28:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:29.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:29.371949+0000) 2022-09-10T12:28:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:29.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:29.900765+0000) 2022-09-10T12:28:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:28:29.852+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:28:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:30.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:30.372207+0000) 2022-09-10T12:28:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:30 smithi104 ceph-mon[97044]: pgmap v15335: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:30.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:30.900988+0000) 2022-09-10T12:28:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:30 smithi110 ceph-mon[97633]: pgmap v15335: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:31.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:31.372446+0000) 2022-09-10T12:28:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:31.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:31.901182+0000) 2022-09-10T12:28:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:32.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:32.372620+0000) 2022-09-10T12:28:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:32 smithi104 ceph-mon[97044]: pgmap v15336: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:32.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:32.901386+0000) 2022-09-10T12:28:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:32 smithi110 ceph-mon[97633]: pgmap v15336: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:33.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:33.372890+0000) 2022-09-10T12:28:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:33.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:33.901570+0000) 2022-09-10T12:28:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:34.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:34.373076+0000) 2022-09-10T12:28:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:34 smithi104 ceph-mon[97044]: pgmap v15337: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:34.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:34.901848+0000) 2022-09-10T12:28:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:34 smithi110 ceph-mon[97633]: pgmap v15337: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:35.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:35.373332+0000) 2022-09-10T12:28:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:35.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:35.902084+0000) 2022-09-10T12:28:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:36.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:36.373600+0000) 2022-09-10T12:28:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:36 smithi104 ceph-mon[97044]: pgmap v15338: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:36.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:36.902386+0000) 2022-09-10T12:28:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:36 smithi110 ceph-mon[97633]: pgmap v15338: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:37.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:37.373793+0000) 2022-09-10T12:28:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:37.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:37.902585+0000) 2022-09-10T12:28:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:38.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:38.374002+0000) 2022-09-10T12:28:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:38 smithi104 ceph-mon[97044]: pgmap v15339: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:38.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:38.902799+0000) 2022-09-10T12:28:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:38 smithi110 ceph-mon[97633]: pgmap v15339: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:39.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:39.374259+0000) 2022-09-10T12:28:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:39.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:39.903066+0000) 2022-09-10T12:28:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:40.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:40.374479+0000) 2022-09-10T12:28:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:40 smithi104 ceph-mon[97044]: pgmap v15340: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:40 smithi110 ceph-mon[97633]: pgmap v15340: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:40.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:40.903214+0000) 2022-09-10T12:28:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:41.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:41.374639+0000) 2022-09-10T12:28:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:41.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:41.903436+0000) 2022-09-10T12:28:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:42.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:42.374815+0000) 2022-09-10T12:28:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:42 smithi104 ceph-mon[97044]: pgmap v15341: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:42 smithi110 ceph-mon[97633]: pgmap v15341: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:42.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:42.903607+0000) 2022-09-10T12:28:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:43.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:43.375034+0000) 2022-09-10T12:28:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:43.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:43.903844+0000) 2022-09-10T12:28:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:44.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:44.375223+0000) 2022-09-10T12:28:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:44 smithi104 ceph-mon[97044]: pgmap v15342: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:44 smithi110 ceph-mon[97633]: pgmap v15342: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:45.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:28:44.859+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:28:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:44.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:44.904129+0000) 2022-09-10T12:28:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:45.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:45.375391+0000) 2022-09-10T12:28:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:45.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:45.904352+0000) 2022-09-10T12:28:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:46.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:46.375655+0000) 2022-09-10T12:28:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:46 smithi104 ceph-mon[97044]: pgmap v15343: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:28:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:28:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:28:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:28:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:46 smithi110 ceph-mon[97633]: pgmap v15343: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:28:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:28:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:28:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:28:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:46.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:46.904547+0000) 2022-09-10T12:28:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:47.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:47.375884+0000) 2022-09-10T12:28:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:47.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:47.904762+0000) 2022-09-10T12:28:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:48.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:48.376133+0000) 2022-09-10T12:28:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:48 smithi104 ceph-mon[97044]: pgmap v15344: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:48 smithi110 ceph-mon[97633]: pgmap v15344: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:48.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:48.904988+0000) 2022-09-10T12:28:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:49.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:49.376327+0000) 2022-09-10T12:28:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:49.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:49.905241+0000) 2022-09-10T12:28:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:50.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:50.376547+0000) 2022-09-10T12:28:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:50 smithi104 ceph-mon[97044]: pgmap v15345: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:50 smithi110 ceph-mon[97633]: pgmap v15345: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:50.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:50.905450+0000) 2022-09-10T12:28:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:51.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:51.376743+0000) 2022-09-10T12:28:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:51.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:51.905659+0000) 2022-09-10T12:28:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:52.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:52.376953+0000) 2022-09-10T12:28:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:52 smithi104 ceph-mon[97044]: pgmap v15346: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:52 smithi110 ceph-mon[97633]: pgmap v15346: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:52.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:52.905888+0000) 2022-09-10T12:28:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:53.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:53.377213+0000) 2022-09-10T12:28:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:53.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:53.906154+0000) 2022-09-10T12:28:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:54.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:54.377400+0000) 2022-09-10T12:28:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:54 smithi104 ceph-mon[97044]: pgmap v15347: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:54 smithi110 ceph-mon[97633]: pgmap v15347: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:54.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:54.906306+0000) 2022-09-10T12:28:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:55.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:55.377644+0000) 2022-09-10T12:28:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:55.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:55.906485+0000) 2022-09-10T12:28:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:56.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:56.377896+0000) 2022-09-10T12:28:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:56 smithi104 ceph-mon[97044]: pgmap v15348: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:56 smithi110 ceph-mon[97633]: pgmap v15348: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:28:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:56.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:56.906666+0000) 2022-09-10T12:28:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:57.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:57.378154+0000) 2022-09-10T12:28:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:57.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:57.906921+0000) 2022-09-10T12:28:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:58.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:58.378399+0000) 2022-09-10T12:28:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:28:58 smithi104 ceph-mon[97044]: pgmap v15349: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:28:58 smithi110 ceph-mon[97633]: pgmap v15349: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:28:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:58.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:58.907128+0000) 2022-09-10T12:28:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:28:59.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:59.378585+0000) 2022-09-10T12:29:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:28:59.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:28:59.907361+0000) 2022-09-10T12:29:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:28:59.863+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:29:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:00.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:00.378775+0000) 2022-09-10T12:29:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:00 smithi104 ceph-mon[97044]: pgmap v15350: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:00 smithi110 ceph-mon[97633]: pgmap v15350: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:00.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:00.907605+0000) 2022-09-10T12:29:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:01.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:01.378984+0000) 2022-09-10T12:29:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:01.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:01.907866+0000) 2022-09-10T12:29:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:02.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:02.379267+0000) 2022-09-10T12:29:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:02 smithi104 ceph-mon[97044]: pgmap v15351: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:02 smithi110 ceph-mon[97633]: pgmap v15351: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:02.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:02.908067+0000) 2022-09-10T12:29:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:03.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:03.379466+0000) 2022-09-10T12:29:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:03.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:03.908261+0000) 2022-09-10T12:29:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:04.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:04.379652+0000) 2022-09-10T12:29:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:04 smithi104 ceph-mon[97044]: pgmap v15352: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:04 smithi110 ceph-mon[97633]: pgmap v15352: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:04.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:04.908415+0000) 2022-09-10T12:29:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:05.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:05.379830+0000) 2022-09-10T12:29:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:05.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:05.908644+0000) 2022-09-10T12:29:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:06.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:06.380020+0000) 2022-09-10T12:29:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:06 smithi104 ceph-mon[97044]: pgmap v15353: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:29:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:06 smithi110 ceph-mon[97633]: pgmap v15353: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:29:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:06.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:06.908890+0000) 2022-09-10T12:29:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:07.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:07.380271+0000) 2022-09-10T12:29:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:07.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:07.909089+0000) 2022-09-10T12:29:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:08.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:08.380524+0000) 2022-09-10T12:29:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:08 smithi104 ceph-mon[97044]: pgmap v15354: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:29:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:08 smithi104 ceph-mon[97044]: osdmap e1040: 12 total, 12 up, 12 in 2022-09-10T12:29:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:08 smithi110 ceph-mon[97633]: pgmap v15354: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:29:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:08 smithi110 ceph-mon[97633]: osdmap e1040: 12 total, 12 up, 12 in 2022-09-10T12:29:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:08.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:08.909303+0000) 2022-09-10T12:29:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:09.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:09.380779+0000) 2022-09-10T12:29:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:09.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:09.909464+0000) 2022-09-10T12:29:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:10.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:10.380917+0000) 2022-09-10T12:29:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:10 smithi104 ceph-mon[97044]: pgmap v15356: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:29:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:10 smithi110 ceph-mon[97633]: pgmap v15356: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:29:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:10.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:10.909698+0000) 2022-09-10T12:29:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:11.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:11.381191+0000) 2022-09-10T12:29:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:11.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:11.909921+0000) 2022-09-10T12:29:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:12.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:12.381326+0000) 2022-09-10T12:29:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:12 smithi104 ceph-mon[97044]: pgmap v15357: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:29:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:12 smithi110 ceph-mon[97633]: pgmap v15357: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:29:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:12.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:12.910121+0000) 2022-09-10T12:29:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:13.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:13.381507+0000) 2022-09-10T12:29:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:13.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:13.910301+0000) 2022-09-10T12:29:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:14.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:14.381709+0000) 2022-09-10T12:29:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:14 smithi104 ceph-mon[97044]: pgmap v15358: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:29:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:14 smithi104 ceph-mon[97044]: osdmap e1041: 12 total, 12 up, 12 in 2022-09-10T12:29:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:14 smithi110 ceph-mon[97633]: pgmap v15358: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:29:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:14 smithi110 ceph-mon[97633]: osdmap e1041: 12 total, 12 up, 12 in 2022-09-10T12:29:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:14.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:14.910517+0000) 2022-09-10T12:29:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:29:14.875+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:29:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:15.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:15.381962+0000) 2022-09-10T12:29:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:15.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:15.910784+0000) 2022-09-10T12:29:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:16.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:16.382199+0000) 2022-09-10T12:29:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:16 smithi104 ceph-mon[97044]: pgmap v15360: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:29:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:16 smithi110 ceph-mon[97633]: pgmap v15360: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:29:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:16.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:16.911005+0000) 2022-09-10T12:29:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:17.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:17.382388+0000) 2022-09-10T12:29:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:17.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:17.911266+0000) 2022-09-10T12:29:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:18.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:18.382558+0000) 2022-09-10T12:29:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:18 smithi104 ceph-mon[97044]: pgmap v15361: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:29:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:18 smithi110 ceph-mon[97633]: pgmap v15361: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:29:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:18.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:18.911517+0000) 2022-09-10T12:29:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:19.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:19.382803+0000) 2022-09-10T12:29:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:19.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:19.911764+0000) 2022-09-10T12:29:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:20.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:20.383051+0000) 2022-09-10T12:29:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:20 smithi104 ceph-mon[97044]: pgmap v15362: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:29:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:20 smithi110 ceph-mon[97633]: pgmap v15362: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:29:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:20.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:20.911994+0000) 2022-09-10T12:29:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:21.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:21.383316+0000) 2022-09-10T12:29:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:21.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:21.912186+0000) 2022-09-10T12:29:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:22.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:22.383439+0000) 2022-09-10T12:29:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:22 smithi104 ceph-mon[97044]: pgmap v15363: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:29:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:22 smithi110 ceph-mon[97633]: pgmap v15363: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:29:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:22.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:22.912458+0000) 2022-09-10T12:29:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:23.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:23.383636+0000) 2022-09-10T12:29:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:23.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:23.912708+0000) 2022-09-10T12:29:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:24.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:24.383834+0000) 2022-09-10T12:29:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:24 smithi104 ceph-mon[97044]: pgmap v15364: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:29:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:24 smithi110 ceph-mon[97633]: pgmap v15364: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:29:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:24.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:24.912909+0000) 2022-09-10T12:29:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:25.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:25.384074+0000) 2022-09-10T12:29:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:25.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:25.913184+0000) 2022-09-10T12:29:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:26.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:26.384286+0000) 2022-09-10T12:29:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:26 smithi104 ceph-mon[97044]: pgmap v15365: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:29:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:26 smithi110 ceph-mon[97633]: pgmap v15365: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:29:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:26.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:26.913404+0000) 2022-09-10T12:29:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:27.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:27.384516+0000) 2022-09-10T12:29:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:27.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:27.913621+0000) 2022-09-10T12:29:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:28.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:28.384709+0000) 2022-09-10T12:29:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:28 smithi104 ceph-mon[97044]: pgmap v15366: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:28 smithi110 ceph-mon[97633]: pgmap v15366: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:29.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:28.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:28.913782+0000) 2022-09-10T12:29:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:29.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:29.384898+0000) 2022-09-10T12:29:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:29.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:29.913928+0000) 2022-09-10T12:29:30.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:29:29.881+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:29:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:30.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:30.385175+0000) 2022-09-10T12:29:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:30 smithi104 ceph-mon[97044]: pgmap v15367: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:30 smithi110 ceph-mon[97633]: pgmap v15367: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:30.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:30.914180+0000) 2022-09-10T12:29:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:31.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:31.385321+0000) 2022-09-10T12:29:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:31.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:31.914372+0000) 2022-09-10T12:29:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:32.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:32.385536+0000) 2022-09-10T12:29:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:32 smithi104 ceph-mon[97044]: pgmap v15368: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:32 smithi110 ceph-mon[97633]: pgmap v15368: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:32.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:32.914540+0000) 2022-09-10T12:29:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:33.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:33.385806+0000) 2022-09-10T12:29:34.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:33.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:33.914741+0000) 2022-09-10T12:29:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:34.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:34.386083+0000) 2022-09-10T12:29:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:34 smithi104 ceph-mon[97044]: pgmap v15369: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:34 smithi110 ceph-mon[97633]: pgmap v15369: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:34.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:34.914951+0000) 2022-09-10T12:29:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:35.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:35.386354+0000) 2022-09-10T12:29:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:35.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:35.915127+0000) 2022-09-10T12:29:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:36.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:36.386582+0000) 2022-09-10T12:29:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:36 smithi104 ceph-mon[97044]: pgmap v15370: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:36 smithi110 ceph-mon[97633]: pgmap v15370: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:36.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:36.915314+0000) 2022-09-10T12:29:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:37.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:37.386781+0000) 2022-09-10T12:29:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:37.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:37.915504+0000) 2022-09-10T12:29:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:38.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:38.387009+0000) 2022-09-10T12:29:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:38 smithi104 ceph-mon[97044]: pgmap v15371: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:38 smithi110 ceph-mon[97633]: pgmap v15371: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:38.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:38.915694+0000) 2022-09-10T12:29:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:39.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:39.387249+0000) 2022-09-10T12:29:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:39.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:39.915878+0000) 2022-09-10T12:29:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:40.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:40.387429+0000) 2022-09-10T12:29:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:40 smithi104 ceph-mon[97044]: pgmap v15372: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:40 smithi110 ceph-mon[97633]: pgmap v15372: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:40.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:40.916056+0000) 2022-09-10T12:29:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:41.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:41.387649+0000) 2022-09-10T12:29:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:41.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:41.916227+0000) 2022-09-10T12:29:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:42.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:42.387872+0000) 2022-09-10T12:29:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:42 smithi104 ceph-mon[97044]: pgmap v15373: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:42 smithi110 ceph-mon[97633]: pgmap v15373: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:42.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:42.916415+0000) 2022-09-10T12:29:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:43.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:43.388195+0000) 2022-09-10T12:29:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:43.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:43.916554+0000) 2022-09-10T12:29:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:44.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:44.388386+0000) 2022-09-10T12:29:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:44 smithi104 ceph-mon[97044]: pgmap v15374: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:44 smithi110 ceph-mon[97633]: pgmap v15374: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:45.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:29:44.885+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:29:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:44.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:44.916806+0000) 2022-09-10T12:29:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:45.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:45.388629+0000) 2022-09-10T12:29:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:45.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:45.917006+0000) 2022-09-10T12:29:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:46.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:46.388829+0000) 2022-09-10T12:29:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:46 smithi110 ceph-mon[97633]: pgmap v15375: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:29:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:29:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:29:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:29:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:46 smithi104 ceph-mon[97044]: pgmap v15375: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:29:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:29:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:29:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:29:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:46.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:46.917262+0000) 2022-09-10T12:29:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:47.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:47.388969+0000) 2022-09-10T12:29:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:47.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:47.917464+0000) 2022-09-10T12:29:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:48.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:48.389159+0000) 2022-09-10T12:29:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:48 smithi110 ceph-mon[97633]: pgmap v15376: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:48 smithi104 ceph-mon[97044]: pgmap v15376: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:48.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:48.917664+0000) 2022-09-10T12:29:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:49.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:49.389310+0000) 2022-09-10T12:29:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:49.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:49.917864+0000) 2022-09-10T12:29:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:50.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:50.389469+0000) 2022-09-10T12:29:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:50 smithi110 ceph-mon[97633]: pgmap v15377: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:50 smithi104 ceph-mon[97044]: pgmap v15377: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:50.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:50.918126+0000) 2022-09-10T12:29:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:51.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:51.389700+0000) 2022-09-10T12:29:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:51.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:51.918366+0000) 2022-09-10T12:29:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:52.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:52.389904+0000) 2022-09-10T12:29:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:52 smithi110 ceph-mon[97633]: pgmap v15378: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:52 smithi104 ceph-mon[97044]: pgmap v15378: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:52.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:52.918577+0000) 2022-09-10T12:29:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:53.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:53.390087+0000) 2022-09-10T12:29:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:53.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:53.918750+0000) 2022-09-10T12:29:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:54.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:54.390344+0000) 2022-09-10T12:29:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:54 smithi110 ceph-mon[97633]: pgmap v15379: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:54 smithi104 ceph-mon[97044]: pgmap v15379: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:54.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:54.919006+0000) 2022-09-10T12:29:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:55.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:55.390528+0000) 2022-09-10T12:29:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:55.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:55.919192+0000) 2022-09-10T12:29:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:56.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:56.390784+0000) 2022-09-10T12:29:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:56 smithi110 ceph-mon[97633]: pgmap v15380: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:56 smithi104 ceph-mon[97044]: pgmap v15380: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:29:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:56.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:56.919366+0000) 2022-09-10T12:29:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:57.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:57.391022+0000) 2022-09-10T12:29:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:57.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:57.919501+0000) 2022-09-10T12:29:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:58.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:58.391281+0000) 2022-09-10T12:29:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:29:58 smithi110 ceph-mon[97633]: pgmap v15381: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:29:58 smithi104 ceph-mon[97044]: pgmap v15381: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:29:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:58.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:58.919718+0000) 2022-09-10T12:29:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:29:59.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:59.391519+0000) 2022-09-10T12:30:00.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:29:59.902+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:30:00.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:29:59.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:29:59.919979+0000) 2022-09-10T12:30:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:00.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:00.391738+0000) 2022-09-10T12:30:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:00 smithi110 ceph-mon[97633]: pgmap v15382: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:30:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:00 smithi104 ceph-mon[97044]: pgmap v15382: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:30:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:00.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:00.920225+0000) 2022-09-10T12:30:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:01.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:01.391973+0000) 2022-09-10T12:30:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:01.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:01.920440+0000) 2022-09-10T12:30:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:02.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:02.392153+0000) 2022-09-10T12:30:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:02 smithi110 ceph-mon[97633]: pgmap v15383: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:02 smithi104 ceph-mon[97044]: pgmap v15383: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:02.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:02.920681+0000) 2022-09-10T12:30:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:03.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:03.392347+0000) 2022-09-10T12:30:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:03.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:03.920926+0000) 2022-09-10T12:30:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:04.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:04.392563+0000) 2022-09-10T12:30:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:04 smithi110 ceph-mon[97633]: pgmap v15384: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:30:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:04 smithi104 ceph-mon[97044]: pgmap v15384: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:30:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:04.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:04.921187+0000) 2022-09-10T12:30:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:05.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:05.392746+0000) 2022-09-10T12:30:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:05.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:05.921389+0000) 2022-09-10T12:30:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:06.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:06.392999+0000) 2022-09-10T12:30:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:06 smithi110 ceph-mon[97633]: pgmap v15385: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:30:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:06 smithi104 ceph-mon[97044]: pgmap v15385: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:30:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:06.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:06.921639+0000) 2022-09-10T12:30:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:07.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:07.393238+0000) 2022-09-10T12:30:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:07.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:07.921882+0000) 2022-09-10T12:30:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:08.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:08.393462+0000) 2022-09-10T12:30:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:08 smithi110 ceph-mon[97633]: pgmap v15386: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:30:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:08 smithi110 ceph-mon[97633]: osdmap e1042: 12 total, 12 up, 12 in 2022-09-10T12:30:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:08 smithi104 ceph-mon[97044]: pgmap v15386: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:30:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:08 smithi104 ceph-mon[97044]: osdmap e1042: 12 total, 12 up, 12 in 2022-09-10T12:30:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:08.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:08.922178+0000) 2022-09-10T12:30:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:09.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:09.393656+0000) 2022-09-10T12:30:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:09.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:09.922385+0000) 2022-09-10T12:30:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:10.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:10.393885+0000) 2022-09-10T12:30:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:10 smithi110 ceph-mon[97633]: pgmap v15388: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:30:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:10 smithi104 ceph-mon[97044]: pgmap v15388: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:30:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:10.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:10.922523+0000) 2022-09-10T12:30:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:11.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:11.394166+0000) 2022-09-10T12:30:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:11.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:11.922716+0000) 2022-09-10T12:30:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:12.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:12.394377+0000) 2022-09-10T12:30:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:12 smithi110 ceph-mon[97633]: pgmap v15389: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:30:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:12 smithi104 ceph-mon[97044]: pgmap v15389: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:30:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:12.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:12.922949+0000) 2022-09-10T12:30:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:13.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:13.394565+0000) 2022-09-10T12:30:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:13.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:13.923174+0000) 2022-09-10T12:30:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:14.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:14.394797+0000) 2022-09-10T12:30:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:14 smithi110 ceph-mon[97633]: pgmap v15390: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:30:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:14 smithi110 ceph-mon[97633]: osdmap e1043: 12 total, 12 up, 12 in 2022-09-10T12:30:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:14 smithi104 ceph-mon[97044]: pgmap v15390: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:30:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:14 smithi104 ceph-mon[97044]: osdmap e1043: 12 total, 12 up, 12 in 2022-09-10T12:30:15.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:30:14.916+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:30:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:14.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:14.923326+0000) 2022-09-10T12:30:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:15.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:15.395015+0000) 2022-09-10T12:30:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:15.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:15.923557+0000) 2022-09-10T12:30:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:16.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:16.395218+0000) 2022-09-10T12:30:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:16 smithi110 ceph-mon[97633]: pgmap v15392: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:30:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:16 smithi104 ceph-mon[97044]: pgmap v15392: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:30:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:16.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:16.923759+0000) 2022-09-10T12:30:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:17.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:17.395469+0000) 2022-09-10T12:30:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:17.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:17.924027+0000) 2022-09-10T12:30:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:18.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:18.395686+0000) 2022-09-10T12:30:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:18 smithi110 ceph-mon[97633]: pgmap v15393: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 222 B/s wr, 11 op/s 2022-09-10T12:30:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:18 smithi104 ceph-mon[97044]: pgmap v15393: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 222 B/s wr, 11 op/s 2022-09-10T12:30:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:18.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:18.924285+0000) 2022-09-10T12:30:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:19.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:19.395954+0000) 2022-09-10T12:30:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:19.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:19.924448+0000) 2022-09-10T12:30:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:20.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:20.396193+0000) 2022-09-10T12:30:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:20 smithi110 ceph-mon[97633]: pgmap v15394: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:30:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:20 smithi104 ceph-mon[97044]: pgmap v15394: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:30:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:20.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:20.924636+0000) 2022-09-10T12:30:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:21.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:21.396436+0000) 2022-09-10T12:30:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:21.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:21.924853+0000) 2022-09-10T12:30:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:22.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:22.396621+0000) 2022-09-10T12:30:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:22 smithi110 ceph-mon[97633]: pgmap v15395: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:30:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:22 smithi104 ceph-mon[97044]: pgmap v15395: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:30:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:22.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:22.925007+0000) 2022-09-10T12:30:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:23.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:23.396832+0000) 2022-09-10T12:30:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:23.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:23.925229+0000) 2022-09-10T12:30:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:24.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:24.397147+0000) 2022-09-10T12:30:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:24 smithi110 ceph-mon[97633]: pgmap v15396: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:30:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:24 smithi104 ceph-mon[97044]: pgmap v15396: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:30:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:24.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:24.925498+0000) 2022-09-10T12:30:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:25.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:25.397376+0000) 2022-09-10T12:30:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:25.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:25.925690+0000) 2022-09-10T12:30:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:26.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:26.397603+0000) 2022-09-10T12:30:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:26 smithi110 ceph-mon[97633]: pgmap v15397: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:30:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:26 smithi104 ceph-mon[97044]: pgmap v15397: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:30:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:26.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:26.925869+0000) 2022-09-10T12:30:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:27.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:27.397791+0000) 2022-09-10T12:30:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:27.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:27.926123+0000) 2022-09-10T12:30:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:28.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:28.398007+0000) 2022-09-10T12:30:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:28 smithi110 ceph-mon[97633]: pgmap v15398: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:28 smithi104 ceph-mon[97044]: pgmap v15398: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:28.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:28.926277+0000) 2022-09-10T12:30:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:29.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:29.398274+0000) 2022-09-10T12:30:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:29.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:29.926401+0000) 2022-09-10T12:30:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:30:29.932+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:30:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:30.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:30.398528+0000) 2022-09-10T12:30:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:30 smithi110 ceph-mon[97633]: pgmap v15399: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:30 smithi104 ceph-mon[97044]: pgmap v15399: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:30.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:30.926616+0000) 2022-09-10T12:30:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:31.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:31.398761+0000) 2022-09-10T12:30:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:31.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:31.926849+0000) 2022-09-10T12:30:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:32.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:32.399032+0000) 2022-09-10T12:30:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:32 smithi110 ceph-mon[97633]: pgmap v15400: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:32 smithi104 ceph-mon[97044]: pgmap v15400: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:32.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:32.927039+0000) 2022-09-10T12:30:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:33.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:33.399284+0000) 2022-09-10T12:30:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:33.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:33.927204+0000) 2022-09-10T12:30:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:34.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:34.399542+0000) 2022-09-10T12:30:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:34 smithi110 ceph-mon[97633]: pgmap v15401: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:34 smithi104 ceph-mon[97044]: pgmap v15401: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:34.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:34.927423+0000) 2022-09-10T12:30:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:35.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:35.399808+0000) 2022-09-10T12:30:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:35.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:35.927649+0000) 2022-09-10T12:30:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:36.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:36.399999+0000) 2022-09-10T12:30:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:36 smithi110 ceph-mon[97633]: pgmap v15402: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:36 smithi104 ceph-mon[97044]: pgmap v15402: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:36.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:36.927925+0000) 2022-09-10T12:30:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:37.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:37.400220+0000) 2022-09-10T12:30:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:37.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:37.928142+0000) 2022-09-10T12:30:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:38.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:38.400470+0000) 2022-09-10T12:30:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:38 smithi110 ceph-mon[97633]: pgmap v15403: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:38 smithi104 ceph-mon[97044]: pgmap v15403: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:38.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:38.928356+0000) 2022-09-10T12:30:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:39.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:39.400689+0000) 2022-09-10T12:30:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:39.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:39.928513+0000) 2022-09-10T12:30:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:40.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:40.400885+0000) 2022-09-10T12:30:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:40 smithi110 ceph-mon[97633]: pgmap v15404: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:40 smithi104 ceph-mon[97044]: pgmap v15404: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:40.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:40.928641+0000) 2022-09-10T12:30:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:41.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:41.401047+0000) 2022-09-10T12:30:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:41.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:41.928870+0000) 2022-09-10T12:30:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:42.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:42.401223+0000) 2022-09-10T12:30:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:42 smithi110 ceph-mon[97633]: pgmap v15405: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:42 smithi104 ceph-mon[97044]: pgmap v15405: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:42.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:42.929119+0000) 2022-09-10T12:30:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:43.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:43.401424+0000) 2022-09-10T12:30:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:43.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:43.929319+0000) 2022-09-10T12:30:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:44.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:44.401586+0000) 2022-09-10T12:30:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:44 smithi110 ceph-mon[97633]: pgmap v15406: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:44 smithi104 ceph-mon[97044]: pgmap v15406: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:44.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:44.929556+0000) 2022-09-10T12:30:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:30:44.945+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:30:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:45.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:45.401834+0000) 2022-09-10T12:30:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:45.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:45.929756+0000) 2022-09-10T12:30:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:46.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:46.402067+0000) 2022-09-10T12:30:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:46 smithi110 ceph-mon[97633]: pgmap v15407: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:30:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:46 smithi104 ceph-mon[97044]: pgmap v15407: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:30:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:30:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:30:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:30:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:46.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:46.929902+0000) 2022-09-10T12:30:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:47.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:47.402349+0000) 2022-09-10T12:30:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:47.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:47.930176+0000) 2022-09-10T12:30:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:48.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:48.402580+0000) 2022-09-10T12:30:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:48 smithi110 ceph-mon[97633]: pgmap v15408: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:48 smithi104 ceph-mon[97044]: pgmap v15408: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:48.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:48.930389+0000) 2022-09-10T12:30:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:49.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:49.402810+0000) 2022-09-10T12:30:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:49.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:49.930608+0000) 2022-09-10T12:30:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:50.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:50.403067+0000) 2022-09-10T12:30:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:50 smithi110 ceph-mon[97633]: pgmap v15409: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:50 smithi104 ceph-mon[97044]: pgmap v15409: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:50.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:50.930826+0000) 2022-09-10T12:30:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:51.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:51.403345+0000) 2022-09-10T12:30:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:51.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:51.931058+0000) 2022-09-10T12:30:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:52.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:52.403569+0000) 2022-09-10T12:30:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:52 smithi110 ceph-mon[97633]: pgmap v15410: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:52 smithi104 ceph-mon[97044]: pgmap v15410: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:52.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:52.931288+0000) 2022-09-10T12:30:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:53.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:53.403853+0000) 2022-09-10T12:30:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:53.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:53.931483+0000) 2022-09-10T12:30:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:54.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:54.404088+0000) 2022-09-10T12:30:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:54 smithi110 ceph-mon[97633]: pgmap v15411: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:54 smithi104 ceph-mon[97044]: pgmap v15411: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:54.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:54.931718+0000) 2022-09-10T12:30:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:55.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:55.404265+0000) 2022-09-10T12:30:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:55.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:55.931954+0000) 2022-09-10T12:30:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:56.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:56.404487+0000) 2022-09-10T12:30:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:56 smithi110 ceph-mon[97633]: pgmap v15412: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:56 smithi104 ceph-mon[97044]: pgmap v15412: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:30:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:56.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:56.932180+0000) 2022-09-10T12:30:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:57.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:57.404714+0000) 2022-09-10T12:30:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:57.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:57.932389+0000) 2022-09-10T12:30:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:58.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:58.404927+0000) 2022-09-10T12:30:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:30:58 smithi110 ceph-mon[97633]: pgmap v15413: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:30:58 smithi104 ceph-mon[97044]: pgmap v15413: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:30:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:58.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:58.932671+0000) 2022-09-10T12:30:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:30:59.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:59.405128+0000) 2022-09-10T12:31:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:30:59.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:30:59.932866+0000) 2022-09-10T12:31:00.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:30:59.950+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:31:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:00.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:00.405249+0000) 2022-09-10T12:31:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:00 smithi104 ceph-mon[97044]: pgmap v15414: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:00.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:00.933044+0000) 2022-09-10T12:31:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:00 smithi110 ceph-mon[97633]: pgmap v15414: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:01.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:01.405407+0000) 2022-09-10T12:31:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:01.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:01.933232+0000) 2022-09-10T12:31:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:02.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:02.405673+0000) 2022-09-10T12:31:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:02 smithi104 ceph-mon[97044]: pgmap v15415: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:02.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:02.933381+0000) 2022-09-10T12:31:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:02 smithi110 ceph-mon[97633]: pgmap v15415: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:03.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:03.405852+0000) 2022-09-10T12:31:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:03.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:03.933585+0000) 2022-09-10T12:31:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:04.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:04.406027+0000) 2022-09-10T12:31:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:04 smithi104 ceph-mon[97044]: pgmap v15416: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:04.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:04.933797+0000) 2022-09-10T12:31:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:04 smithi110 ceph-mon[97633]: pgmap v15416: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:05.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:05.406233+0000) 2022-09-10T12:31:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:05.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:05.934019+0000) 2022-09-10T12:31:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:06.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:06.406481+0000) 2022-09-10T12:31:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:06 smithi104 ceph-mon[97044]: pgmap v15417: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:31:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:06.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:06.934194+0000) 2022-09-10T12:31:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:06 smithi110 ceph-mon[97633]: pgmap v15417: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:31:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:07.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:07.406732+0000) 2022-09-10T12:31:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:07.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:07.934332+0000) 2022-09-10T12:31:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:08.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:08.406957+0000) 2022-09-10T12:31:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:08 smithi104 ceph-mon[97044]: pgmap v15418: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:31:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:08 smithi104 ceph-mon[97044]: osdmap e1044: 12 total, 12 up, 12 in 2022-09-10T12:31:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:08.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:08.934540+0000) 2022-09-10T12:31:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:08 smithi110 ceph-mon[97633]: pgmap v15418: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:31:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:08 smithi110 ceph-mon[97633]: osdmap e1044: 12 total, 12 up, 12 in 2022-09-10T12:31:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:09.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:09.407238+0000) 2022-09-10T12:31:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:09.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:09.934724+0000) 2022-09-10T12:31:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:10.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:10.407495+0000) 2022-09-10T12:31:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:10 smithi104 ceph-mon[97044]: pgmap v15420: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:31:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:10.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:10.934929+0000) 2022-09-10T12:31:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:10 smithi110 ceph-mon[97633]: pgmap v15420: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:31:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:11.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:11.407709+0000) 2022-09-10T12:31:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:11.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:11.935128+0000) 2022-09-10T12:31:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:12.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:12.407921+0000) 2022-09-10T12:31:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:12 smithi104 ceph-mon[97044]: pgmap v15421: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:31:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:12.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:12.935296+0000) 2022-09-10T12:31:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:12 smithi110 ceph-mon[97633]: pgmap v15421: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:31:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:13.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:13.408173+0000) 2022-09-10T12:31:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:13.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:13.935525+0000) 2022-09-10T12:31:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:14.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:14.408359+0000) 2022-09-10T12:31:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:14 smithi104 ceph-mon[97044]: pgmap v15422: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:31:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:14 smithi104 ceph-mon[97044]: osdmap e1045: 12 total, 12 up, 12 in 2022-09-10T12:31:15.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:31:14.954+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:31:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:14.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:14.935727+0000) 2022-09-10T12:31:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:14 smithi110 ceph-mon[97633]: pgmap v15422: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:31:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:14 smithi110 ceph-mon[97633]: osdmap e1045: 12 total, 12 up, 12 in 2022-09-10T12:31:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:15.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:15.408501+0000) 2022-09-10T12:31:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:15.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:15.935899+0000) 2022-09-10T12:31:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:16.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:16.408725+0000) 2022-09-10T12:31:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:16 smithi104 ceph-mon[97044]: pgmap v15424: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:31:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:16.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:16.936057+0000) 2022-09-10T12:31:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:16 smithi110 ceph-mon[97633]: pgmap v15424: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:31:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:17.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:17.408946+0000) 2022-09-10T12:31:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:17.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:17.936339+0000) 2022-09-10T12:31:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:18.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:18.409160+0000) 2022-09-10T12:31:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:18 smithi104 ceph-mon[97044]: pgmap v15425: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 224 B/s wr, 11 op/s 2022-09-10T12:31:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:18.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:18.936599+0000) 2022-09-10T12:31:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:18 smithi110 ceph-mon[97633]: pgmap v15425: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 224 B/s wr, 11 op/s 2022-09-10T12:31:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:19.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:19.409369+0000) 2022-09-10T12:31:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:19.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:19.936845+0000) 2022-09-10T12:31:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:20.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:20.409599+0000) 2022-09-10T12:31:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:20 smithi104 ceph-mon[97044]: pgmap v15426: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:31:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:20.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:20.937153+0000) 2022-09-10T12:31:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:20 smithi110 ceph-mon[97633]: pgmap v15426: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:31:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:21.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:21.409834+0000) 2022-09-10T12:31:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:21.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:21.937336+0000) 2022-09-10T12:31:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:22.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:22.409999+0000) 2022-09-10T12:31:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:22 smithi104 ceph-mon[97044]: pgmap v15427: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:31:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:22.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:22.937544+0000) 2022-09-10T12:31:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:22 smithi110 ceph-mon[97633]: pgmap v15427: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:31:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:23.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:23.410236+0000) 2022-09-10T12:31:24.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:23 smithi104 ceph-mon[97044]: pgmap v15428: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:31:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:23.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:23.937782+0000) 2022-09-10T12:31:24.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:23 smithi110 ceph-mon[97633]: pgmap v15428: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:31:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:24.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:24.410381+0000) 2022-09-10T12:31:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:24.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:24.938016+0000) 2022-09-10T12:31:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:25.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:25.410605+0000) 2022-09-10T12:31:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:25.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:25.938262+0000) 2022-09-10T12:31:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:26.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:26.410848+0000) 2022-09-10T12:31:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:26 smithi104 ceph-mon[97044]: pgmap v15429: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:31:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:26 smithi110 ceph-mon[97633]: pgmap v15429: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:31:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:26.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:26.938472+0000) 2022-09-10T12:31:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:27.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:27.411063+0000) 2022-09-10T12:31:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:27.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:27.938667+0000) 2022-09-10T12:31:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:28.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:28.411256+0000) 2022-09-10T12:31:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:28 smithi104 ceph-mon[97044]: pgmap v15430: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:28 smithi110 ceph-mon[97633]: pgmap v15430: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:28.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:28.938892+0000) 2022-09-10T12:31:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:29.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:29.411510+0000) 2022-09-10T12:31:30.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:31:29.970+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:31:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:29.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:29.939074+0000) 2022-09-10T12:31:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:30.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:30.411765+0000) 2022-09-10T12:31:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:30 smithi104 ceph-mon[97044]: pgmap v15431: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:30 smithi110 ceph-mon[97633]: pgmap v15431: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:30.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:30.939380+0000) 2022-09-10T12:31:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:31.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:31.412015+0000) 2022-09-10T12:31:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:31.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:31.939606+0000) 2022-09-10T12:31:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:32.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:32.412235+0000) 2022-09-10T12:31:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:32 smithi104 ceph-mon[97044]: pgmap v15432: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:32 smithi110 ceph-mon[97633]: pgmap v15432: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:32.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:32.939865+0000) 2022-09-10T12:31:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:33.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:33.412475+0000) 2022-09-10T12:31:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:33.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:33.940012+0000) 2022-09-10T12:31:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:34.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:34.412665+0000) 2022-09-10T12:31:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:34 smithi104 ceph-mon[97044]: pgmap v15433: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:34 smithi110 ceph-mon[97633]: pgmap v15433: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:34.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:34.940199+0000) 2022-09-10T12:31:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:35.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:35.412908+0000) 2022-09-10T12:31:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:35.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:35.940406+0000) 2022-09-10T12:31:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:36.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:36.413198+0000) 2022-09-10T12:31:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:36 smithi104 ceph-mon[97044]: pgmap v15434: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:36 smithi110 ceph-mon[97633]: pgmap v15434: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:36.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:36.940674+0000) 2022-09-10T12:31:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:37.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:37.413430+0000) 2022-09-10T12:31:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:37.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:37.940814+0000) 2022-09-10T12:31:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:38.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:38.413653+0000) 2022-09-10T12:31:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:38 smithi104 ceph-mon[97044]: pgmap v15435: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:38 smithi110 ceph-mon[97633]: pgmap v15435: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:38.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:38.941050+0000) 2022-09-10T12:31:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:39.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:39.413843+0000) 2022-09-10T12:31:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:39.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:39.941327+0000) 2022-09-10T12:31:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:40.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:40.414089+0000) 2022-09-10T12:31:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:40 smithi104 ceph-mon[97044]: pgmap v15436: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:40 smithi110 ceph-mon[97633]: pgmap v15436: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:40.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:40.941546+0000) 2022-09-10T12:31:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:41.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:41.414329+0000) 2022-09-10T12:31:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:41.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:41.941783+0000) 2022-09-10T12:31:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:42.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:42.414528+0000) 2022-09-10T12:31:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:42 smithi104 ceph-mon[97044]: pgmap v15437: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:42 smithi110 ceph-mon[97633]: pgmap v15437: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:42.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:42.942016+0000) 2022-09-10T12:31:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:43.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:43.414797+0000) 2022-09-10T12:31:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:43.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:43.942254+0000) 2022-09-10T12:31:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:44.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:44.415024+0000) 2022-09-10T12:31:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:44 smithi104 ceph-mon[97044]: pgmap v15438: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:44 smithi110 ceph-mon[97633]: pgmap v15438: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:44.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:44.942536+0000) 2022-09-10T12:31:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:31:44.974+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:31:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:45.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:45.415273+0000) 2022-09-10T12:31:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:45.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:45.942786+0000) 2022-09-10T12:31:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:46.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:46.415506+0000) 2022-09-10T12:31:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:46 smithi104 ceph-mon[97044]: pgmap v15439: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:31:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:31:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:31:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:31:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:46 smithi110 ceph-mon[97633]: pgmap v15439: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:31:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:46.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:46.942950+0000) 2022-09-10T12:31:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:47.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:47.415701+0000) 2022-09-10T12:31:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:47.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:47.943084+0000) 2022-09-10T12:31:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:48.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:48.415893+0000) 2022-09-10T12:31:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:48 smithi104 ceph-mon[97044]: pgmap v15440: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:49.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:48 smithi110 ceph-mon[97633]: pgmap v15440: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:48.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:48.943226+0000) 2022-09-10T12:31:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:49.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:49.416182+0000) 2022-09-10T12:31:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:49.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:49.943455+0000) 2022-09-10T12:31:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:50.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:50.416395+0000) 2022-09-10T12:31:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:50 smithi104 ceph-mon[97044]: pgmap v15441: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:50 smithi110 ceph-mon[97633]: pgmap v15441: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:50.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:50.943586+0000) 2022-09-10T12:31:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:51.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:51.416624+0000) 2022-09-10T12:31:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:51.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:51.943818+0000) 2022-09-10T12:31:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:52.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:52.416795+0000) 2022-09-10T12:31:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:52 smithi104 ceph-mon[97044]: pgmap v15442: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:52 smithi110 ceph-mon[97633]: pgmap v15442: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:52.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:52.944011+0000) 2022-09-10T12:31:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:53.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:53.417019+0000) 2022-09-10T12:31:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:53.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:53.944244+0000) 2022-09-10T12:31:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:54.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:54.417233+0000) 2022-09-10T12:31:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:54 smithi104 ceph-mon[97044]: pgmap v15443: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:54 smithi110 ceph-mon[97633]: pgmap v15443: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:54.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:54.944506+0000) 2022-09-10T12:31:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:55.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:55.417375+0000) 2022-09-10T12:31:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:55.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:55.944664+0000) 2022-09-10T12:31:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:56.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:56.417618+0000) 2022-09-10T12:31:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:56 smithi104 ceph-mon[97044]: pgmap v15444: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:56 smithi110 ceph-mon[97633]: pgmap v15444: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:31:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:56.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:56.944855+0000) 2022-09-10T12:31:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:57.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:57.417835+0000) 2022-09-10T12:31:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:57.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:57.945131+0000) 2022-09-10T12:31:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:58.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:58.418126+0000) 2022-09-10T12:31:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:31:58 smithi104 ceph-mon[97044]: pgmap v15445: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:31:58 smithi110 ceph-mon[97633]: pgmap v15445: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:31:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:58.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:58.945395+0000) 2022-09-10T12:31:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:31:59.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:59.418384+0000) 2022-09-10T12:32:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:31:59.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:31:59.945657+0000) 2022-09-10T12:32:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:31:59.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:32:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:00.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:00.418528+0000) 2022-09-10T12:32:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:00 smithi110 ceph-mon[97633]: pgmap v15446: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:00 smithi104 ceph-mon[97044]: pgmap v15446: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:00.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:00.945825+0000) 2022-09-10T12:32:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:01.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:01.418680+0000) 2022-09-10T12:32:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:01.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:01.946020+0000) 2022-09-10T12:32:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:02.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:02.418902+0000) 2022-09-10T12:32:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:02 smithi110 ceph-mon[97633]: pgmap v15447: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:02 smithi104 ceph-mon[97044]: pgmap v15447: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:02.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:02.946230+0000) 2022-09-10T12:32:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:03.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:03.419180+0000) 2022-09-10T12:32:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:03.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:03.946364+0000) 2022-09-10T12:32:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:04.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:04.419314+0000) 2022-09-10T12:32:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:04 smithi110 ceph-mon[97633]: pgmap v15448: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:04 smithi104 ceph-mon[97044]: pgmap v15448: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:04.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:04.946492+0000) 2022-09-10T12:32:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:05.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:05.419518+0000) 2022-09-10T12:32:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:05.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:05.946627+0000) 2022-09-10T12:32:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:06.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:06.419733+0000) 2022-09-10T12:32:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:06 smithi110 ceph-mon[97633]: pgmap v15449: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:32:07.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:06 smithi104 ceph-mon[97044]: pgmap v15449: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:32:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:06.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:06.946826+0000) 2022-09-10T12:32:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:07.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:07.419957+0000) 2022-09-10T12:32:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:07.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:07.947031+0000) 2022-09-10T12:32:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:08.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:08.420213+0000) 2022-09-10T12:32:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:08 smithi110 ceph-mon[97633]: pgmap v15450: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:32:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:08 smithi110 ceph-mon[97633]: osdmap e1046: 12 total, 12 up, 12 in 2022-09-10T12:32:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:08 smithi104 ceph-mon[97044]: pgmap v15450: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:32:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:08 smithi104 ceph-mon[97044]: osdmap e1046: 12 total, 12 up, 12 in 2022-09-10T12:32:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:08.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:08.947171+0000) 2022-09-10T12:32:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:09.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:09.420464+0000) 2022-09-10T12:32:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:09.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:09.947330+0000) 2022-09-10T12:32:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:10.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:10.420641+0000) 2022-09-10T12:32:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:10 smithi110 ceph-mon[97633]: pgmap v15452: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:32:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:10 smithi104 ceph-mon[97044]: pgmap v15452: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:32:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:10.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:10.947464+0000) 2022-09-10T12:32:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:11.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:11.420836+0000) 2022-09-10T12:32:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:11.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:11.947597+0000) 2022-09-10T12:32:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:12.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:12.421091+0000) 2022-09-10T12:32:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:12 smithi110 ceph-mon[97633]: pgmap v15453: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:32:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:12 smithi104 ceph-mon[97044]: pgmap v15453: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:32:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:12.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:12.947740+0000) 2022-09-10T12:32:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:13.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:13.421389+0000) 2022-09-10T12:32:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:13.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:13.947895+0000) 2022-09-10T12:32:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:14.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:14.421540+0000) 2022-09-10T12:32:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:14 smithi110 ceph-mon[97633]: pgmap v15454: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:32:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:14 smithi110 ceph-mon[97633]: osdmap e1047: 12 total, 12 up, 12 in 2022-09-10T12:32:15.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:14 smithi104 ceph-mon[97044]: pgmap v15454: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:32:15.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:14 smithi104 ceph-mon[97044]: osdmap e1047: 12 total, 12 up, 12 in 2022-09-10T12:32:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:14.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:14.948035+0000) 2022-09-10T12:32:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:32:15.000+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:32:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:15.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:15.421760+0000) 2022-09-10T12:32:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:15.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:15.948167+0000) 2022-09-10T12:32:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:16.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:16.421957+0000) 2022-09-10T12:32:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:16 smithi110 ceph-mon[97633]: pgmap v15456: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:32:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:16 smithi104 ceph-mon[97044]: pgmap v15456: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:32:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:16.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:16.948301+0000) 2022-09-10T12:32:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:17.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:17.422159+0000) 2022-09-10T12:32:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:17.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:17.948438+0000) 2022-09-10T12:32:18.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:18.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:18.422374+0000) 2022-09-10T12:32:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:18 smithi110 ceph-mon[97633]: pgmap v15457: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:32:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:18 smithi104 ceph-mon[97044]: pgmap v15457: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:32:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:18.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:18.948609+0000) 2022-09-10T12:32:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:19.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:19.422577+0000) 2022-09-10T12:32:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:19.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:19.948757+0000) 2022-09-10T12:32:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:20.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:20.422816+0000) 2022-09-10T12:32:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:20 smithi110 ceph-mon[97633]: pgmap v15458: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:32:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:20 smithi104 ceph-mon[97044]: pgmap v15458: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:32:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:20.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:20.948885+0000) 2022-09-10T12:32:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:21.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:21.423019+0000) 2022-09-10T12:32:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:21.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:21.949077+0000) 2022-09-10T12:32:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:22.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:22.423201+0000) 2022-09-10T12:32:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:22 smithi110 ceph-mon[97633]: pgmap v15459: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:32:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:22 smithi104 ceph-mon[97044]: pgmap v15459: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:32:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:22.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:22.949217+0000) 2022-09-10T12:32:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:23.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:23.423379+0000) 2022-09-10T12:32:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:23.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:23.949349+0000) 2022-09-10T12:32:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:24.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:24.423566+0000) 2022-09-10T12:32:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:24 smithi110 ceph-mon[97633]: pgmap v15460: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:32:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:24 smithi104 ceph-mon[97044]: pgmap v15460: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:32:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:24.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:24.949486+0000) 2022-09-10T12:32:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:25.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:25.423856+0000) 2022-09-10T12:32:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:25.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:25.949637+0000) 2022-09-10T12:32:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:26.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:26.424071+0000) 2022-09-10T12:32:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:26 smithi110 ceph-mon[97633]: pgmap v15461: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:32:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:26 smithi104 ceph-mon[97044]: pgmap v15461: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:32:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:26.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:26.949819+0000) 2022-09-10T12:32:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:27.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:27.424368+0000) 2022-09-10T12:32:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:27.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:27.949950+0000) 2022-09-10T12:32:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:28.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:28.424594+0000) 2022-09-10T12:32:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:28 smithi110 ceph-mon[97633]: pgmap v15462: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:28 smithi104 ceph-mon[97044]: pgmap v15462: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:28.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:28.950077+0000) 2022-09-10T12:32:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:29.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:29.424848+0000) 2022-09-10T12:32:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:29.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:29.950238+0000) 2022-09-10T12:32:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:32:30.015+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:32:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:30.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:30.425030+0000) 2022-09-10T12:32:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:30 smithi110 ceph-mon[97633]: pgmap v15463: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:30 smithi104 ceph-mon[97044]: pgmap v15463: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:30.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:30.950374+0000) 2022-09-10T12:32:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:31.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:31.425236+0000) 2022-09-10T12:32:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:31.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:31.950518+0000) 2022-09-10T12:32:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:32.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:32.425457+0000) 2022-09-10T12:32:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:32 smithi110 ceph-mon[97633]: pgmap v15464: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:32 smithi104 ceph-mon[97044]: pgmap v15464: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:32.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:32.950691+0000) 2022-09-10T12:32:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:33.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:33.425628+0000) 2022-09-10T12:32:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:33.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:33.950862+0000) 2022-09-10T12:32:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:34.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:34.425839+0000) 2022-09-10T12:32:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:34 smithi110 ceph-mon[97633]: pgmap v15465: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:34 smithi104 ceph-mon[97044]: pgmap v15465: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:34.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:34.950986+0000) 2022-09-10T12:32:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:35.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:35.426059+0000) 2022-09-10T12:32:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:35.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:35.951132+0000) 2022-09-10T12:32:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:36.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:36.426282+0000) 2022-09-10T12:32:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:36 smithi110 ceph-mon[97633]: pgmap v15466: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:36 smithi104 ceph-mon[97044]: pgmap v15466: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:36.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:36.951326+0000) 2022-09-10T12:32:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:37.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:37.426543+0000) 2022-09-10T12:32:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:37.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:37.951448+0000) 2022-09-10T12:32:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:38.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:38.426796+0000) 2022-09-10T12:32:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:38 smithi110 ceph-mon[97633]: pgmap v15467: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:38 smithi104 ceph-mon[97044]: pgmap v15467: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:38.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:38.951573+0000) 2022-09-10T12:32:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:39.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:39.427037+0000) 2022-09-10T12:32:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:39.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:39.951778+0000) 2022-09-10T12:32:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:40.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:40.427222+0000) 2022-09-10T12:32:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:40 smithi110 ceph-mon[97633]: pgmap v15468: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:40 smithi104 ceph-mon[97044]: pgmap v15468: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:40.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:40.951919+0000) 2022-09-10T12:32:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:41.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:41.427422+0000) 2022-09-10T12:32:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:41.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:41.952060+0000) 2022-09-10T12:32:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:42.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:42.427619+0000) 2022-09-10T12:32:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:42 smithi110 ceph-mon[97633]: pgmap v15469: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:42 smithi104 ceph-mon[97044]: pgmap v15469: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:42.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:42.952157+0000) 2022-09-10T12:32:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:43.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:43.427813+0000) 2022-09-10T12:32:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:43.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:43.952285+0000) 2022-09-10T12:32:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:44.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:44.428005+0000) 2022-09-10T12:32:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:44 smithi110 ceph-mon[97633]: pgmap v15470: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:32:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:44 smithi104 ceph-mon[97044]: pgmap v15470: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:32:45.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:32:45.022+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:32:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:44.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:44.952420+0000) 2022-09-10T12:32:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:45.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:45.428181+0000) 2022-09-10T12:32:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:45.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:45.952556+0000) 2022-09-10T12:32:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:46.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:46.428343+0000) 2022-09-10T12:32:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:46 smithi110 ceph-mon[97633]: pgmap v15471: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:32:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:46 smithi104 ceph-mon[97044]: pgmap v15471: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:32:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:32:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:32:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:32:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:46.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:46.952745+0000) 2022-09-10T12:32:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:47.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:47.428530+0000) 2022-09-10T12:32:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:47.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:47.952883+0000) 2022-09-10T12:32:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:48.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:48.428754+0000) 2022-09-10T12:32:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:48 smithi110 ceph-mon[97633]: pgmap v15472: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:32:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:48 smithi104 ceph-mon[97044]: pgmap v15472: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:32:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:48.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:48.953008+0000) 2022-09-10T12:32:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:49.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:49.428949+0000) 2022-09-10T12:32:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:49.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:49.953141+0000) 2022-09-10T12:32:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:50.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:50.429126+0000) 2022-09-10T12:32:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:50 smithi110 ceph-mon[97633]: pgmap v15473: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:32:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:50 smithi104 ceph-mon[97044]: pgmap v15473: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:32:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:50.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:50.953299+0000) 2022-09-10T12:32:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:51.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:51.429319+0000) 2022-09-10T12:32:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:51.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:51.953466+0000) 2022-09-10T12:32:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:52.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:52.429459+0000) 2022-09-10T12:32:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:52 smithi110 ceph-mon[97633]: pgmap v15474: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:52 smithi104 ceph-mon[97044]: pgmap v15474: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:52.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:52.953614+0000) 2022-09-10T12:32:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:53.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:53.429600+0000) 2022-09-10T12:32:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:53.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:53.953757+0000) 2022-09-10T12:32:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:54.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:54.429730+0000) 2022-09-10T12:32:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:54 smithi110 ceph-mon[97633]: pgmap v15475: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:54 smithi104 ceph-mon[97044]: pgmap v15475: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:54.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:54.953900+0000) 2022-09-10T12:32:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:55.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:55.429940+0000) 2022-09-10T12:32:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:55.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:55.954021+0000) 2022-09-10T12:32:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:56.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:56.430132+0000) 2022-09-10T12:32:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:56 smithi110 ceph-mon[97633]: pgmap v15476: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:56 smithi104 ceph-mon[97044]: pgmap v15476: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:32:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:56.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:56.954162+0000) 2022-09-10T12:32:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:57.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:57.430301+0000) 2022-09-10T12:32:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:57.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:57.954290+0000) 2022-09-10T12:32:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:58.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:58.430486+0000) 2022-09-10T12:32:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:32:58 smithi110 ceph-mon[97633]: pgmap v15477: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:32:58 smithi104 ceph-mon[97044]: pgmap v15477: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:32:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:58.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:58.954414+0000) 2022-09-10T12:32:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:32:59.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:59.430671+0000) 2022-09-10T12:33:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:32:59.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:32:59.954537+0000) 2022-09-10T12:33:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:33:00.038+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:33:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:00.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:00.430813+0000) 2022-09-10T12:33:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:00 smithi110 ceph-mon[97633]: pgmap v15478: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:00 smithi104 ceph-mon[97044]: pgmap v15478: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:00.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:00.954651+0000) 2022-09-10T12:33:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:01.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:01.430954+0000) 2022-09-10T12:33:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:01.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:01.954774+0000) 2022-09-10T12:33:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:02.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:02.431163+0000) 2022-09-10T12:33:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:02 smithi110 ceph-mon[97633]: pgmap v15479: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:02 smithi104 ceph-mon[97044]: pgmap v15479: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:02.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:02.954916+0000) 2022-09-10T12:33:03.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:03.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:03.431322+0000) 2022-09-10T12:33:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:03.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:03.955059+0000) 2022-09-10T12:33:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:04.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:04.431477+0000) 2022-09-10T12:33:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:04 smithi110 ceph-mon[97633]: pgmap v15480: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:05.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:04 smithi104 ceph-mon[97044]: pgmap v15480: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:04.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:04.955161+0000) 2022-09-10T12:33:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:05.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:05.431667+0000) 2022-09-10T12:33:06.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:05.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:05.955285+0000) 2022-09-10T12:33:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:06.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:06.431796+0000) 2022-09-10T12:33:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:06 smithi110 ceph-mon[97633]: pgmap v15481: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:33:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:06 smithi104 ceph-mon[97044]: pgmap v15481: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:33:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:06.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:06.955412+0000) 2022-09-10T12:33:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:07.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:07.431924+0000) 2022-09-10T12:33:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:07.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:07.955543+0000) 2022-09-10T12:33:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:08.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:08.432089+0000) 2022-09-10T12:33:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:08 smithi110 ceph-mon[97633]: pgmap v15482: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:33:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:08 smithi110 ceph-mon[97633]: osdmap e1048: 12 total, 12 up, 12 in 2022-09-10T12:33:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:08 smithi104 ceph-mon[97044]: pgmap v15482: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:33:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:08 smithi104 ceph-mon[97044]: osdmap e1048: 12 total, 12 up, 12 in 2022-09-10T12:33:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:08.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:08.955668+0000) 2022-09-10T12:33:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:09.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:09.432266+0000) 2022-09-10T12:33:10.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:09.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:09.955790+0000) 2022-09-10T12:33:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:10.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:10.432446+0000) 2022-09-10T12:33:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:10 smithi104 ceph-mon[97044]: pgmap v15484: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:33:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:10.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:10.955917+0000) 2022-09-10T12:33:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:10 smithi110 ceph-mon[97633]: pgmap v15484: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:33:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:11.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:11.432579+0000) 2022-09-10T12:33:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:11.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:11.956052+0000) 2022-09-10T12:33:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:12.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:12.432716+0000) 2022-09-10T12:33:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:12 smithi104 ceph-mon[97044]: pgmap v15485: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:33:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:12.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:12.956159+0000) 2022-09-10T12:33:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:12 smithi110 ceph-mon[97633]: pgmap v15485: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:33:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:13.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:13.432913+0000) 2022-09-10T12:33:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:13.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:13.956302+0000) 2022-09-10T12:33:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:14.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:14.433113+0000) 2022-09-10T12:33:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:14 smithi104 ceph-mon[97044]: pgmap v15486: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:33:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:14 smithi104 ceph-mon[97044]: osdmap e1049: 12 total, 12 up, 12 in 2022-09-10T12:33:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:14.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:14.956450+0000) 2022-09-10T12:33:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:33:15.054+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:33:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:14 smithi110 ceph-mon[97633]: pgmap v15486: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:33:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:14 smithi110 ceph-mon[97633]: osdmap e1049: 12 total, 12 up, 12 in 2022-09-10T12:33:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:15.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:15.433303+0000) 2022-09-10T12:33:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:15.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:15.956593+0000) 2022-09-10T12:33:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:16.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:16.433460+0000) 2022-09-10T12:33:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:16 smithi104 ceph-mon[97044]: pgmap v15488: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:33:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:16.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:16.956719+0000) 2022-09-10T12:33:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:16 smithi110 ceph-mon[97633]: pgmap v15488: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:33:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:17.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:17.433598+0000) 2022-09-10T12:33:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:17.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:17.956858+0000) 2022-09-10T12:33:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:18.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:18.433770+0000) 2022-09-10T12:33:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:18 smithi104 ceph-mon[97044]: pgmap v15489: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T12:33:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:18.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:18.956978+0000) 2022-09-10T12:33:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:18 smithi110 ceph-mon[97633]: pgmap v15489: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 223 B/s wr, 11 op/s 2022-09-10T12:33:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:19.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:19.433995+0000) 2022-09-10T12:33:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:19.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:19.957131+0000) 2022-09-10T12:33:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:20.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:20.434128+0000) 2022-09-10T12:33:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:20 smithi104 ceph-mon[97044]: pgmap v15490: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:33:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:20.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:20.957250+0000) 2022-09-10T12:33:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:20 smithi110 ceph-mon[97633]: pgmap v15490: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:33:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:21.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:21.434255+0000) 2022-09-10T12:33:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:21.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:21.957375+0000) 2022-09-10T12:33:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:22.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:22.434439+0000) 2022-09-10T12:33:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:22 smithi104 ceph-mon[97044]: pgmap v15491: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:33:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:22.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:22.957535+0000) 2022-09-10T12:33:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:22 smithi110 ceph-mon[97633]: pgmap v15491: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:33:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:23.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:23.434622+0000) 2022-09-10T12:33:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:23.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:23.957715+0000) 2022-09-10T12:33:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:24.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:24.434767+0000) 2022-09-10T12:33:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:24 smithi104 ceph-mon[97044]: pgmap v15492: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:33:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:24.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:24.957874+0000) 2022-09-10T12:33:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:24 smithi110 ceph-mon[97633]: pgmap v15492: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:33:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:25.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:25.434920+0000) 2022-09-10T12:33:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:25.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:25.957984+0000) 2022-09-10T12:33:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:26.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:26.435171+0000) 2022-09-10T12:33:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:26 smithi104 ceph-mon[97044]: pgmap v15493: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:33:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:26.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:26.958196+0000) 2022-09-10T12:33:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:26 smithi110 ceph-mon[97633]: pgmap v15493: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:33:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:27.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:27.435334+0000) 2022-09-10T12:33:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:27.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:27.958384+0000) 2022-09-10T12:33:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:28.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:28.435563+0000) 2022-09-10T12:33:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:28 smithi104 ceph-mon[97044]: pgmap v15494: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:28.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:28.958571+0000) 2022-09-10T12:33:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:28 smithi110 ceph-mon[97633]: pgmap v15494: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:29.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:29.435742+0000) 2022-09-10T12:33:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:29.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:29.958743+0000) 2022-09-10T12:33:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:33:30.057+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:33:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:30.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:30.435949+0000) 2022-09-10T12:33:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:30 smithi104 ceph-mon[97044]: pgmap v15495: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:30.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:30.958998+0000) 2022-09-10T12:33:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:30 smithi110 ceph-mon[97633]: pgmap v15495: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:31.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:31.436146+0000) 2022-09-10T12:33:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:31.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:31.959210+0000) 2022-09-10T12:33:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:32.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:32.436276+0000) 2022-09-10T12:33:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:32 smithi104 ceph-mon[97044]: pgmap v15496: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:32.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:32.959349+0000) 2022-09-10T12:33:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:32 smithi110 ceph-mon[97633]: pgmap v15496: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:33.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:33.436414+0000) 2022-09-10T12:33:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:33.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:33.959515+0000) 2022-09-10T12:33:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:34.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:34.436543+0000) 2022-09-10T12:33:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:34 smithi104 ceph-mon[97044]: pgmap v15497: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:34.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:34.959706+0000) 2022-09-10T12:33:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:34 smithi110 ceph-mon[97633]: pgmap v15497: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:35.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:35.436704+0000) 2022-09-10T12:33:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:35.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:35.959879+0000) 2022-09-10T12:33:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:36.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:36.436846+0000) 2022-09-10T12:33:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:36 smithi104 ceph-mon[97044]: pgmap v15498: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:36.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:36.960160+0000) 2022-09-10T12:33:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:36 smithi110 ceph-mon[97633]: pgmap v15498: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:37.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:37.437017+0000) 2022-09-10T12:33:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:37.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:37.960283+0000) 2022-09-10T12:33:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:38.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:38.437187+0000) 2022-09-10T12:33:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:38 smithi104 ceph-mon[97044]: pgmap v15499: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:38.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:38.960495+0000) 2022-09-10T12:33:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:38 smithi110 ceph-mon[97633]: pgmap v15499: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:39.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:39.437395+0000) 2022-09-10T12:33:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:39.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:39.960618+0000) 2022-09-10T12:33:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:40.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:40.437603+0000) 2022-09-10T12:33:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:40 smithi104 ceph-mon[97044]: pgmap v15500: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:40.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:40.960771+0000) 2022-09-10T12:33:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:40 smithi110 ceph-mon[97633]: pgmap v15500: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:41.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:41.437757+0000) 2022-09-10T12:33:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:41.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:41.960920+0000) 2022-09-10T12:33:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:42.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:42.437922+0000) 2022-09-10T12:33:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:42 smithi104 ceph-mon[97044]: pgmap v15501: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:42.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:42.961156+0000) 2022-09-10T12:33:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:42 smithi110 ceph-mon[97633]: pgmap v15501: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:43.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:43.438092+0000) 2022-09-10T12:33:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:43.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:43.961339+0000) 2022-09-10T12:33:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:44.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:44.438264+0000) 2022-09-10T12:33:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:44 smithi104 ceph-mon[97044]: pgmap v15502: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:44.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:44.961603+0000) 2022-09-10T12:33:45.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:33:45.071+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:33:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:44 smithi110 ceph-mon[97633]: pgmap v15502: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:45.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:45.438397+0000) 2022-09-10T12:33:46.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:45.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:45.961779+0000) 2022-09-10T12:33:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:46.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:46.438608+0000) 2022-09-10T12:33:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:46 smithi104 ceph-mon[97044]: pgmap v15503: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:33:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:33:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:33:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:33:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:46.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:46.961945+0000) 2022-09-10T12:33:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:46 smithi110 ceph-mon[97633]: pgmap v15503: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:33:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:33:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:33:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:33:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:47.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:47.438845+0000) 2022-09-10T12:33:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:47.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:47.962157+0000) 2022-09-10T12:33:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:48.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:48.439002+0000) 2022-09-10T12:33:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:48 smithi104 ceph-mon[97044]: pgmap v15504: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:48.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:48.962341+0000) 2022-09-10T12:33:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:48 smithi110 ceph-mon[97633]: pgmap v15504: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:49.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:49.439160+0000) 2022-09-10T12:33:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:49.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:49.962518+0000) 2022-09-10T12:33:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:50.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:50.439295+0000) 2022-09-10T12:33:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:50 smithi104 ceph-mon[97044]: pgmap v15505: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:50 smithi110 ceph-mon[97633]: pgmap v15505: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:50.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:50.962693+0000) 2022-09-10T12:33:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:51.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:51.439454+0000) 2022-09-10T12:33:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:51.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:51.962861+0000) 2022-09-10T12:33:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:52.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:52.439586+0000) 2022-09-10T12:33:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:52 smithi104 ceph-mon[97044]: pgmap v15506: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:52 smithi110 ceph-mon[97633]: pgmap v15506: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:52.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:52.963090+0000) 2022-09-10T12:33:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:53.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:53.439745+0000) 2022-09-10T12:33:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:53.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:53.963355+0000) 2022-09-10T12:33:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:54.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:54.439953+0000) 2022-09-10T12:33:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:54 smithi104 ceph-mon[97044]: pgmap v15507: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:54 smithi110 ceph-mon[97633]: pgmap v15507: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:54.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:54.963551+0000) 2022-09-10T12:33:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:55.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:55.440203+0000) 2022-09-10T12:33:56.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:55.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:55.963796+0000) 2022-09-10T12:33:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:56.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:56.440394+0000) 2022-09-10T12:33:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:56 smithi110 ceph-mon[97633]: pgmap v15508: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:56 smithi104 ceph-mon[97044]: pgmap v15508: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:33:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:56.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:56.964041+0000) 2022-09-10T12:33:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:57.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:57.440606+0000) 2022-09-10T12:33:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:57.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:57.964281+0000) 2022-09-10T12:33:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:58.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:58.440840+0000) 2022-09-10T12:33:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:33:58 smithi110 ceph-mon[97633]: pgmap v15509: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:33:58 smithi104 ceph-mon[97044]: pgmap v15509: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:33:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:58.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:58.964536+0000) 2022-09-10T12:33:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:33:59.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:59.441029+0000) 2022-09-10T12:34:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:33:59.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:33:59.964719+0000) 2022-09-10T12:34:00.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:34:00.075+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:34:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:00.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:00.441186+0000) 2022-09-10T12:34:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:00 smithi110 ceph-mon[97633]: pgmap v15510: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:00 smithi104 ceph-mon[97044]: pgmap v15510: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:00.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:00.964865+0000) 2022-09-10T12:34:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:01.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:01.441361+0000) 2022-09-10T12:34:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:01.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:01.965075+0000) 2022-09-10T12:34:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:02.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:02.441587+0000) 2022-09-10T12:34:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:02 smithi110 ceph-mon[97633]: pgmap v15511: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:02 smithi104 ceph-mon[97044]: pgmap v15511: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:02.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:02.965327+0000) 2022-09-10T12:34:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:03.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:03.441815+0000) 2022-09-10T12:34:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:03.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:03.965529+0000) 2022-09-10T12:34:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:04.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:04.441976+0000) 2022-09-10T12:34:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:04 smithi110 ceph-mon[97633]: pgmap v15512: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:04 smithi104 ceph-mon[97044]: pgmap v15512: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:04.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:04.965758+0000) 2022-09-10T12:34:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:05.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:05.442131+0000) 2022-09-10T12:34:06.320 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:05.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:05.965927+0000) 2022-09-10T12:34:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:06.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:06.442266+0000) 2022-09-10T12:34:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:06 smithi110 ceph-mon[97633]: pgmap v15513: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:34:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:06 smithi104 ceph-mon[97044]: pgmap v15513: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:34:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:06.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:06.966228+0000) 2022-09-10T12:34:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:07.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:07.442466+0000) 2022-09-10T12:34:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:07.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:07.966464+0000) 2022-09-10T12:34:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:08.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:08.442661+0000) 2022-09-10T12:34:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:08 smithi110 ceph-mon[97633]: pgmap v15514: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:34:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:08 smithi110 ceph-mon[97633]: osdmap e1050: 12 total, 12 up, 12 in 2022-09-10T12:34:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:08 smithi104 ceph-mon[97044]: pgmap v15514: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:34:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:08 smithi104 ceph-mon[97044]: osdmap e1050: 12 total, 12 up, 12 in 2022-09-10T12:34:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:08.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:08.966640+0000) 2022-09-10T12:34:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:09.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:09.442817+0000) 2022-09-10T12:34:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:09.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:09.966897+0000) 2022-09-10T12:34:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:10.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:10.443010+0000) 2022-09-10T12:34:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:10 smithi110 ceph-mon[97633]: pgmap v15516: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:34:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:10 smithi104 ceph-mon[97044]: pgmap v15516: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:34:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:10.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:10.967150+0000) 2022-09-10T12:34:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:11.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:11.443174+0000) 2022-09-10T12:34:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:11.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:11.967358+0000) 2022-09-10T12:34:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:12.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:12.443307+0000) 2022-09-10T12:34:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:12 smithi110 ceph-mon[97633]: pgmap v15517: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:34:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:12 smithi104 ceph-mon[97044]: pgmap v15517: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:34:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:12.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:12.967610+0000) 2022-09-10T12:34:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:13.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:13.443484+0000) 2022-09-10T12:34:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:13.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:13.967824+0000) 2022-09-10T12:34:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:14.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:14.443626+0000) 2022-09-10T12:34:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:14 smithi110 ceph-mon[97633]: pgmap v15518: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:34:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:14 smithi110 ceph-mon[97633]: osdmap e1051: 12 total, 12 up, 12 in 2022-09-10T12:34:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:14 smithi104 ceph-mon[97044]: pgmap v15518: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:34:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:14 smithi104 ceph-mon[97044]: osdmap e1051: 12 total, 12 up, 12 in 2022-09-10T12:34:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:14.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:14.968024+0000) 2022-09-10T12:34:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:34:15.092+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:34:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:15.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:15.443768+0000) 2022-09-10T12:34:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:15.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:15.968225+0000) 2022-09-10T12:34:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:16.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:16.443912+0000) 2022-09-10T12:34:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:16 smithi110 ceph-mon[97633]: pgmap v15520: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:34:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:16 smithi104 ceph-mon[97044]: pgmap v15520: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:34:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:16.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:16.968407+0000) 2022-09-10T12:34:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:17.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:17.444085+0000) 2022-09-10T12:34:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:17.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:17.968653+0000) 2022-09-10T12:34:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:18.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:18.444280+0000) 2022-09-10T12:34:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:18 smithi110 ceph-mon[97633]: pgmap v15521: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:34:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:18 smithi104 ceph-mon[97044]: pgmap v15521: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:34:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:18.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:18.968839+0000) 2022-09-10T12:34:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:19.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:19.444488+0000) 2022-09-10T12:34:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:19.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:19.969017+0000) 2022-09-10T12:34:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:20.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:20.444717+0000) 2022-09-10T12:34:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:20 smithi110 ceph-mon[97633]: pgmap v15522: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:34:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:20 smithi104 ceph-mon[97044]: pgmap v15522: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:34:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:20.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:20.969199+0000) 2022-09-10T12:34:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:21.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:21.444923+0000) 2022-09-10T12:34:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:21.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:21.969431+0000) 2022-09-10T12:34:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:22.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:22.445116+0000) 2022-09-10T12:34:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:22 smithi110 ceph-mon[97633]: pgmap v15523: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:34:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:22 smithi104 ceph-mon[97044]: pgmap v15523: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:34:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:22.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:22.969618+0000) 2022-09-10T12:34:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:23.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:23.445255+0000) 2022-09-10T12:34:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:23.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:23.969821+0000) 2022-09-10T12:34:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:24.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:24.445494+0000) 2022-09-10T12:34:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:24 smithi110 ceph-mon[97633]: pgmap v15524: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:34:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:24 smithi104 ceph-mon[97044]: pgmap v15524: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:34:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:24.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:24.970033+0000) 2022-09-10T12:34:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:25.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:25.445666+0000) 2022-09-10T12:34:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:25.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:25.970258+0000) 2022-09-10T12:34:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:26.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:26.445821+0000) 2022-09-10T12:34:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:26 smithi110 ceph-mon[97633]: pgmap v15525: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:34:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:26 smithi104 ceph-mon[97044]: pgmap v15525: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T12:34:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:26.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:26.970510+0000) 2022-09-10T12:34:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:27.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:27.446057+0000) 2022-09-10T12:34:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:27.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:27.970716+0000) 2022-09-10T12:34:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:28.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:28.446181+0000) 2022-09-10T12:34:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:28 smithi110 ceph-mon[97633]: pgmap v15526: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:28 smithi104 ceph-mon[97044]: pgmap v15526: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:28.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:28.970942+0000) 2022-09-10T12:34:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:29.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:29.446387+0000) 2022-09-10T12:34:30.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:34:30.105+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:34:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:29.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:29.971157+0000) 2022-09-10T12:34:30.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:30.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:30.446635+0000) 2022-09-10T12:34:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:30 smithi110 ceph-mon[97633]: pgmap v15527: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:30 smithi104 ceph-mon[97044]: pgmap v15527: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:30.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:30.971387+0000) 2022-09-10T12:34:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:31.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:31.446804+0000) 2022-09-10T12:34:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:31.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:31.971660+0000) 2022-09-10T12:34:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:32.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:32.446972+0000) 2022-09-10T12:34:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:32 smithi110 ceph-mon[97633]: pgmap v15528: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:32 smithi104 ceph-mon[97044]: pgmap v15528: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:32.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:32.971922+0000) 2022-09-10T12:34:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:33.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:33.447207+0000) 2022-09-10T12:34:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:33.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:33.972087+0000) 2022-09-10T12:34:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:34.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:34.447355+0000) 2022-09-10T12:34:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:34 smithi110 ceph-mon[97633]: pgmap v15529: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:34 smithi104 ceph-mon[97044]: pgmap v15529: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:34.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:34.972370+0000) 2022-09-10T12:34:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:35.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:35.447517+0000) 2022-09-10T12:34:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:35.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:35.972550+0000) 2022-09-10T12:34:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:36.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:36.447746+0000) 2022-09-10T12:34:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:36 smithi110 ceph-mon[97633]: pgmap v15530: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:36 smithi104 ceph-mon[97044]: pgmap v15530: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:36.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:36.972805+0000) 2022-09-10T12:34:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:37.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:37.447918+0000) 2022-09-10T12:34:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:37.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:37.973028+0000) 2022-09-10T12:34:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:38.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:38.448193+0000) 2022-09-10T12:34:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:38 smithi110 ceph-mon[97633]: pgmap v15531: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:38 smithi104 ceph-mon[97044]: pgmap v15531: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:38.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:38.973220+0000) 2022-09-10T12:34:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:39.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:39.448363+0000) 2022-09-10T12:34:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:39.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:39.973449+0000) 2022-09-10T12:34:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:40.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:40.448518+0000) 2022-09-10T12:34:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:40 smithi110 ceph-mon[97633]: pgmap v15532: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:40 smithi104 ceph-mon[97044]: pgmap v15532: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:40.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:40.973665+0000) 2022-09-10T12:34:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:41.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:41.448723+0000) 2022-09-10T12:34:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:41.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:41.973865+0000) 2022-09-10T12:34:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:42.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:42.448938+0000) 2022-09-10T12:34:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:42 smithi110 ceph-mon[97633]: pgmap v15533: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:42 smithi104 ceph-mon[97044]: pgmap v15533: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:42.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:42.974134+0000) 2022-09-10T12:34:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:43.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:43.449116+0000) 2022-09-10T12:34:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:43.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:43.974313+0000) 2022-09-10T12:34:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:44.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:44.449356+0000) 2022-09-10T12:34:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:44 smithi110 ceph-mon[97633]: pgmap v15534: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:44 smithi104 ceph-mon[97044]: pgmap v15534: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:45.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:34:45.120+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:34:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:44.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:44.974505+0000) 2022-09-10T12:34:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:45.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:45.449580+0000) 2022-09-10T12:34:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:45.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:45.974689+0000) 2022-09-10T12:34:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:46.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:46.449733+0000) 2022-09-10T12:34:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:46 smithi110 ceph-mon[97633]: pgmap v15535: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:34:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:46 smithi104 ceph-mon[97044]: pgmap v15535: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:34:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:34:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:34:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:34:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:46.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:46.974925+0000) 2022-09-10T12:34:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:47.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:47.449908+0000) 2022-09-10T12:34:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:47.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:47.975181+0000) 2022-09-10T12:34:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:48.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:48.450112+0000) 2022-09-10T12:34:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:48 smithi110 ceph-mon[97633]: pgmap v15536: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:48 smithi104 ceph-mon[97044]: pgmap v15536: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:48.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:48.975335+0000) 2022-09-10T12:34:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:49.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:49.450314+0000) 2022-09-10T12:34:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:49.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:49.975544+0000) 2022-09-10T12:34:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:50.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:50.450470+0000) 2022-09-10T12:34:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:50 smithi110 ceph-mon[97633]: pgmap v15537: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:50 smithi104 ceph-mon[97044]: pgmap v15537: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:50.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:50.975788+0000) 2022-09-10T12:34:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:51.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:51.450609+0000) 2022-09-10T12:34:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:51.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:51.975996+0000) 2022-09-10T12:34:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:52.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:52.450753+0000) 2022-09-10T12:34:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:52 smithi110 ceph-mon[97633]: pgmap v15538: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:53.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:52 smithi104 ceph-mon[97044]: pgmap v15538: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:52.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:52.976250+0000) 2022-09-10T12:34:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:53.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:53.450892+0000) 2022-09-10T12:34:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:53.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:53.976498+0000) 2022-09-10T12:34:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:54.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:54.451043+0000) 2022-09-10T12:34:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:54 smithi110 ceph-mon[97633]: pgmap v15539: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:54 smithi104 ceph-mon[97044]: pgmap v15539: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:54.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:54.976648+0000) 2022-09-10T12:34:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:55.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:55.451198+0000) 2022-09-10T12:34:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:55.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:55.976889+0000) 2022-09-10T12:34:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:56.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:56.451353+0000) 2022-09-10T12:34:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:56 smithi110 ceph-mon[97633]: pgmap v15540: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:56 smithi104 ceph-mon[97044]: pgmap v15540: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:34:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:56.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:56.977137+0000) 2022-09-10T12:34:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:57.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:57.451500+0000) 2022-09-10T12:34:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:57.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:57.977285+0000) 2022-09-10T12:34:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:58.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:58.451646+0000) 2022-09-10T12:34:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:34:58 smithi110 ceph-mon[97633]: pgmap v15541: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:34:58 smithi104 ceph-mon[97044]: pgmap v15541: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:34:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:58.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:58.977466+0000) 2022-09-10T12:34:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:34:59.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:59.451858+0000) 2022-09-10T12:35:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:34:59.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:34:59.977652+0000) 2022-09-10T12:35:00.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:35:00.136+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:35:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:00.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:00.452004+0000) 2022-09-10T12:35:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:00 smithi110 ceph-mon[97633]: pgmap v15542: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:00 smithi104 ceph-mon[97044]: pgmap v15542: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:00.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:00.977787+0000) 2022-09-10T12:35:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:01.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:01.452217+0000) 2022-09-10T12:35:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:01.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:01.977952+0000) 2022-09-10T12:35:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:02.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:02.452404+0000) 2022-09-10T12:35:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:02 smithi110 ceph-mon[97633]: pgmap v15543: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:02 smithi104 ceph-mon[97044]: pgmap v15543: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:02.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:02.978226+0000) 2022-09-10T12:35:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:03.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:03.452635+0000) 2022-09-10T12:35:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:03.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:03.978409+0000) 2022-09-10T12:35:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:04.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:04.452847+0000) 2022-09-10T12:35:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:04 smithi110 ceph-mon[97633]: pgmap v15544: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:04 smithi104 ceph-mon[97044]: pgmap v15544: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:04.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:04.978607+0000) 2022-09-10T12:35:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:05.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:05.452994+0000) 2022-09-10T12:35:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:05.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:05.978828+0000) 2022-09-10T12:35:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:06.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:06.453210+0000) 2022-09-10T12:35:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:06 smithi110 ceph-mon[97633]: pgmap v15545: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:35:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:06 smithi104 ceph-mon[97044]: pgmap v15545: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:35:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:06.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:06.979017+0000) 2022-09-10T12:35:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:07.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:07.453466+0000) 2022-09-10T12:35:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:07.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:07.979191+0000) 2022-09-10T12:35:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:08.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:08.453603+0000) 2022-09-10T12:35:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:08 smithi104 ceph-mon[97044]: pgmap v15546: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:08 smithi104 ceph-mon[97044]: osdmap e1052: 12 total, 12 up, 12 in 2022-09-10T12:35:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:08.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:08.979443+0000) 2022-09-10T12:35:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:08 smithi110 ceph-mon[97633]: pgmap v15546: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:08 smithi110 ceph-mon[97633]: osdmap e1052: 12 total, 12 up, 12 in 2022-09-10T12:35:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:09.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:09.453745+0000) 2022-09-10T12:35:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:09.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:09.979646+0000) 2022-09-10T12:35:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:10.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:10.453904+0000) 2022-09-10T12:35:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:10 smithi104 ceph-mon[97044]: pgmap v15548: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:35:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:10.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:10.979853+0000) 2022-09-10T12:35:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:10 smithi110 ceph-mon[97633]: pgmap v15548: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:35:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:11.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:11.454075+0000) 2022-09-10T12:35:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:11.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:11.980079+0000) 2022-09-10T12:35:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:12.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:12.454306+0000) 2022-09-10T12:35:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:12 smithi104 ceph-mon[97044]: pgmap v15549: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:35:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:12.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:12.980335+0000) 2022-09-10T12:35:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:12 smithi110 ceph-mon[97633]: pgmap v15549: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:35:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:13.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:13.454480+0000) 2022-09-10T12:35:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:13.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:13.980545+0000) 2022-09-10T12:35:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:14.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:14.454654+0000) 2022-09-10T12:35:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:14 smithi104 ceph-mon[97044]: pgmap v15550: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:35:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:14 smithi104 ceph-mon[97044]: osdmap e1053: 12 total, 12 up, 12 in 2022-09-10T12:35:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:14.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:14.980788+0000) 2022-09-10T12:35:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:35:15.151+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:35:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:14 smithi110 ceph-mon[97633]: pgmap v15550: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:35:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:14 smithi110 ceph-mon[97633]: osdmap e1053: 12 total, 12 up, 12 in 2022-09-10T12:35:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:15.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:15.454791+0000) 2022-09-10T12:35:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:15.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:15.981028+0000) 2022-09-10T12:35:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:16.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:16.454937+0000) 2022-09-10T12:35:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:16 smithi104 ceph-mon[97044]: pgmap v15552: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:35:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:16.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:16.981262+0000) 2022-09-10T12:35:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:16 smithi110 ceph-mon[97633]: pgmap v15552: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:35:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:17.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:17.455190+0000) 2022-09-10T12:35:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:17.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:17.981511+0000) 2022-09-10T12:35:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:18.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:18.455398+0000) 2022-09-10T12:35:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:18 smithi104 ceph-mon[97044]: pgmap v15553: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 11 op/s 2022-09-10T12:35:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:18.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:18.981757+0000) 2022-09-10T12:35:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:18 smithi110 ceph-mon[97633]: pgmap v15553: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 11 op/s 2022-09-10T12:35:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:19.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:19.455576+0000) 2022-09-10T12:35:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:19.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:19.981964+0000) 2022-09-10T12:35:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:20.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:20.455801+0000) 2022-09-10T12:35:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:20 smithi104 ceph-mon[97044]: pgmap v15554: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:35:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:20.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:20.982195+0000) 2022-09-10T12:35:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:20 smithi110 ceph-mon[97633]: pgmap v15554: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:35:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:21.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:21.455999+0000) 2022-09-10T12:35:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:21.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:21.982367+0000) 2022-09-10T12:35:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:22.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:22.456236+0000) 2022-09-10T12:35:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:22 smithi104 ceph-mon[97044]: pgmap v15555: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:35:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:22.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:22.982526+0000) 2022-09-10T12:35:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:22 smithi110 ceph-mon[97633]: pgmap v15555: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:35:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:23.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:23.456395+0000) 2022-09-10T12:35:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:23.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:23.982754+0000) 2022-09-10T12:35:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:24.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:24.456520+0000) 2022-09-10T12:35:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:24 smithi104 ceph-mon[97044]: pgmap v15556: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:35:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:24.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:24.982912+0000) 2022-09-10T12:35:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:24 smithi110 ceph-mon[97633]: pgmap v15556: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:35:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:25.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:25.456667+0000) 2022-09-10T12:35:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:25.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:25.983168+0000) 2022-09-10T12:35:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:26.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:26.456817+0000) 2022-09-10T12:35:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:26 smithi104 ceph-mon[97044]: pgmap v15557: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:35:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:26.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:26.983402+0000) 2022-09-10T12:35:27.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:26 smithi110 ceph-mon[97633]: pgmap v15557: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:35:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:27.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:27.457029+0000) 2022-09-10T12:35:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:27.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:27.983599+0000) 2022-09-10T12:35:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:28.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:28.457198+0000) 2022-09-10T12:35:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:28 smithi104 ceph-mon[97044]: pgmap v15558: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:28.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:28.983845+0000) 2022-09-10T12:35:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:28 smithi110 ceph-mon[97633]: pgmap v15558: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:29.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:29.457347+0000) 2022-09-10T12:35:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:29.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:29.984013+0000) 2022-09-10T12:35:30.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:35:30.167+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:35:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:30.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:30.457491+0000) 2022-09-10T12:35:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:30 smithi104 ceph-mon[97044]: pgmap v15559: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:30.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:30.984242+0000) 2022-09-10T12:35:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:30 smithi110 ceph-mon[97633]: pgmap v15559: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:31.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:31.457630+0000) 2022-09-10T12:35:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:31.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:31.984419+0000) 2022-09-10T12:35:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:32.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:32.457817+0000) 2022-09-10T12:35:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:32 smithi104 ceph-mon[97044]: pgmap v15560: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:32.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:32.984603+0000) 2022-09-10T12:35:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:32 smithi110 ceph-mon[97633]: pgmap v15560: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:33.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:33.457958+0000) 2022-09-10T12:35:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:33.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:33.984947+0000) 2022-09-10T12:35:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:34.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:34.458175+0000) 2022-09-10T12:35:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:34 smithi104 ceph-mon[97044]: pgmap v15561: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:34.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:34.985197+0000) 2022-09-10T12:35:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:34 smithi110 ceph-mon[97633]: pgmap v15561: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:35.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:35.458362+0000) 2022-09-10T12:35:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:35.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:35.985403+0000) 2022-09-10T12:35:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:36.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:36.458522+0000) 2022-09-10T12:35:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:36 smithi104 ceph-mon[97044]: pgmap v15562: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:36.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:36.985575+0000) 2022-09-10T12:35:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:36 smithi110 ceph-mon[97633]: pgmap v15562: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:37.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:37.458702+0000) 2022-09-10T12:35:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:37.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:37.985725+0000) 2022-09-10T12:35:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:38.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:38.458909+0000) 2022-09-10T12:35:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:38 smithi104 ceph-mon[97044]: pgmap v15563: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:38.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:38.985914+0000) 2022-09-10T12:35:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:38 smithi110 ceph-mon[97633]: pgmap v15563: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:39.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:39.459091+0000) 2022-09-10T12:35:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:39.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:39.986147+0000) 2022-09-10T12:35:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:40.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:40.459232+0000) 2022-09-10T12:35:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:40 smithi104 ceph-mon[97044]: pgmap v15564: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:40.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:40.986381+0000) 2022-09-10T12:35:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:40 smithi110 ceph-mon[97633]: pgmap v15564: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:41.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:41.459380+0000) 2022-09-10T12:35:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:41.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:41.986574+0000) 2022-09-10T12:35:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:42.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:42.459571+0000) 2022-09-10T12:35:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:42 smithi104 ceph-mon[97044]: pgmap v15565: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:42.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:42.986797+0000) 2022-09-10T12:35:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:42 smithi110 ceph-mon[97633]: pgmap v15565: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:43.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:43.459726+0000) 2022-09-10T12:35:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:43.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:43.987036+0000) 2022-09-10T12:35:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:44.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:44.459894+0000) 2022-09-10T12:35:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:44 smithi104 ceph-mon[97044]: pgmap v15566: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:45.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:35:45.179+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:35:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:44.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:44.987335+0000) 2022-09-10T12:35:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:44 smithi110 ceph-mon[97633]: pgmap v15566: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:45.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:45.460055+0000) 2022-09-10T12:35:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:45.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:45.987504+0000) 2022-09-10T12:35:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:46.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:46.460264+0000) 2022-09-10T12:35:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:46 smithi104 ceph-mon[97044]: pgmap v15567: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:35:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:35:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:35:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:35:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:46.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:46.987701+0000) 2022-09-10T12:35:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:46 smithi110 ceph-mon[97633]: pgmap v15567: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:35:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:35:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:35:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:35:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:47.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:47.460521+0000) 2022-09-10T12:35:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:47.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:47.987871+0000) 2022-09-10T12:35:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:48.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:48.460657+0000) 2022-09-10T12:35:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:48 smithi104 ceph-mon[97044]: pgmap v15568: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:48.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:48.988170+0000) 2022-09-10T12:35:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:48 smithi110 ceph-mon[97633]: pgmap v15568: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:49.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:49.460811+0000) 2022-09-10T12:35:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:49.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:49.988386+0000) 2022-09-10T12:35:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:50.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:50.461013+0000) 2022-09-10T12:35:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:50 smithi104 ceph-mon[97044]: pgmap v15569: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:50.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:50.988531+0000) 2022-09-10T12:35:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:50 smithi110 ceph-mon[97633]: pgmap v15569: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:51.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:51.461169+0000) 2022-09-10T12:35:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:51.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:51.988707+0000) 2022-09-10T12:35:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:52.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:52.461330+0000) 2022-09-10T12:35:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:52 smithi104 ceph-mon[97044]: pgmap v15570: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:52.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:52.988873+0000) 2022-09-10T12:35:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:52 smithi110 ceph-mon[97633]: pgmap v15570: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:53.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:53.461543+0000) 2022-09-10T12:35:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:53.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:53.989157+0000) 2022-09-10T12:35:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:54.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:54.461688+0000) 2022-09-10T12:35:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:54 smithi104 ceph-mon[97044]: pgmap v15571: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:54.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:54.989389+0000) 2022-09-10T12:35:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:54 smithi110 ceph-mon[97633]: pgmap v15571: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:55.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:55.461855+0000) 2022-09-10T12:35:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:55.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:55.989521+0000) 2022-09-10T12:35:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:56.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:56.462016+0000) 2022-09-10T12:35:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:56 smithi104 ceph-mon[97044]: pgmap v15572: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:56.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:56.989746+0000) 2022-09-10T12:35:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:56 smithi110 ceph-mon[97633]: pgmap v15572: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:35:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:57.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:57.462173+0000) 2022-09-10T12:35:58.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:35:57 smithi104 ceph-mon[97044]: pgmap v15573: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:57.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:57.989901+0000) 2022-09-10T12:35:58.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:35:57 smithi110 ceph-mon[97633]: pgmap v15573: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:35:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:58.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:58.462339+0000) 2022-09-10T12:35:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:58.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:58.990075+0000) 2022-09-10T12:35:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:35:59.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:59.462476+0000) 2022-09-10T12:36:00.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:36:00.193+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:36:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:35:59.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:35:59.990242+0000) 2022-09-10T12:36:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:00.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:00.462609+0000) 2022-09-10T12:36:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:00 smithi110 ceph-mon[97633]: pgmap v15574: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:00 smithi104 ceph-mon[97044]: pgmap v15574: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:00.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:00.990425+0000) 2022-09-10T12:36:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:01.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:01.462844+0000) 2022-09-10T12:36:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:01.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:01.990606+0000) 2022-09-10T12:36:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:02.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:02.463062+0000) 2022-09-10T12:36:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:02 smithi110 ceph-mon[97633]: pgmap v15575: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:02 smithi104 ceph-mon[97044]: pgmap v15575: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:02.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:02.990789+0000) 2022-09-10T12:36:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:03.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:03.463251+0000) 2022-09-10T12:36:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:03.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:03.990980+0000) 2022-09-10T12:36:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:04.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:04.463399+0000) 2022-09-10T12:36:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:04 smithi110 ceph-mon[97633]: pgmap v15576: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:04 smithi104 ceph-mon[97044]: pgmap v15576: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:04.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:04.991189+0000) 2022-09-10T12:36:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:05.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:05.463618+0000) 2022-09-10T12:36:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:05.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:05.991391+0000) 2022-09-10T12:36:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:06.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:06.463776+0000) 2022-09-10T12:36:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:06 smithi110 ceph-mon[97633]: pgmap v15577: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:36:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:06 smithi104 ceph-mon[97044]: pgmap v15577: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:36:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:06.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:06.991555+0000) 2022-09-10T12:36:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:07.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:07.463949+0000) 2022-09-10T12:36:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:07.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:07.991687+0000) 2022-09-10T12:36:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:08.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:08.464147+0000) 2022-09-10T12:36:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:08 smithi110 ceph-mon[97633]: pgmap v15578: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:36:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:08 smithi110 ceph-mon[97633]: osdmap e1054: 12 total, 12 up, 12 in 2022-09-10T12:36:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:08 smithi104 ceph-mon[97044]: pgmap v15578: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:36:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:08 smithi104 ceph-mon[97044]: osdmap e1054: 12 total, 12 up, 12 in 2022-09-10T12:36:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:08.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:08.991885+0000) 2022-09-10T12:36:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:09.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:09.464293+0000) 2022-09-10T12:36:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:09.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:09.992130+0000) 2022-09-10T12:36:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:10.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:10.464426+0000) 2022-09-10T12:36:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:10 smithi110 ceph-mon[97633]: pgmap v15580: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:36:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:10 smithi104 ceph-mon[97044]: pgmap v15580: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:36:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:10.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:10.992338+0000) 2022-09-10T12:36:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:11.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:11.464580+0000) 2022-09-10T12:36:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:11.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:11.992550+0000) 2022-09-10T12:36:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:12.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:12.464719+0000) 2022-09-10T12:36:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:12 smithi110 ceph-mon[97633]: pgmap v15581: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:36:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:12 smithi104 ceph-mon[97044]: pgmap v15581: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:36:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:12.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:12.992794+0000) 2022-09-10T12:36:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:13.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:13.464865+0000) 2022-09-10T12:36:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:13.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:13.993007+0000) 2022-09-10T12:36:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:14.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:14.465145+0000) 2022-09-10T12:36:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:14 smithi110 ceph-mon[97633]: pgmap v15582: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:36:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:14 smithi110 ceph-mon[97633]: osdmap e1055: 12 total, 12 up, 12 in 2022-09-10T12:36:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:14 smithi104 ceph-mon[97044]: pgmap v15582: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:36:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:14 smithi104 ceph-mon[97044]: osdmap e1055: 12 total, 12 up, 12 in 2022-09-10T12:36:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:14.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:14.993234+0000) 2022-09-10T12:36:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:36:15.208+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:36:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:15.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:15.465311+0000) 2022-09-10T12:36:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:15.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:15.993416+0000) 2022-09-10T12:36:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:16.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:16.465518+0000) 2022-09-10T12:36:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:16 smithi110 ceph-mon[97633]: pgmap v15584: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:36:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:16 smithi104 ceph-mon[97044]: pgmap v15584: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:36:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:16.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:16.993642+0000) 2022-09-10T12:36:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:17.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:17.465761+0000) 2022-09-10T12:36:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:17.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:17.993867+0000) 2022-09-10T12:36:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:18.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:18.466004+0000) 2022-09-10T12:36:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:18 smithi110 ceph-mon[97633]: pgmap v15585: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:36:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:18 smithi104 ceph-mon[97044]: pgmap v15585: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:36:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:18.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:18.994069+0000) 2022-09-10T12:36:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:19.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:19.466180+0000) 2022-09-10T12:36:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:19.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:19.994339+0000) 2022-09-10T12:36:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:20.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:20.466356+0000) 2022-09-10T12:36:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:20 smithi110 ceph-mon[97633]: pgmap v15586: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:36:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:20 smithi104 ceph-mon[97044]: pgmap v15586: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:36:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:20.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:20.994555+0000) 2022-09-10T12:36:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:21.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:21.466479+0000) 2022-09-10T12:36:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:21.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:21.994759+0000) 2022-09-10T12:36:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:22.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:22.466656+0000) 2022-09-10T12:36:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:22 smithi110 ceph-mon[97633]: pgmap v15587: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:36:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:22 smithi104 ceph-mon[97044]: pgmap v15587: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:36:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:22.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:22.995009+0000) 2022-09-10T12:36:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:23.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:23.466849+0000) 2022-09-10T12:36:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:23.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:23.995244+0000) 2022-09-10T12:36:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:24.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:24.467055+0000) 2022-09-10T12:36:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:24 smithi110 ceph-mon[97633]: pgmap v15588: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:36:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:24 smithi104 ceph-mon[97044]: pgmap v15588: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:36:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:24.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:24.995484+0000) 2022-09-10T12:36:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:25.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:25.467320+0000) 2022-09-10T12:36:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:25.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:25.995648+0000) 2022-09-10T12:36:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:26.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:26.467486+0000) 2022-09-10T12:36:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:26 smithi110 ceph-mon[97633]: pgmap v15589: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:36:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:26 smithi104 ceph-mon[97044]: pgmap v15589: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:36:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:26.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:26.995826+0000) 2022-09-10T12:36:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:27.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:27.467663+0000) 2022-09-10T12:36:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:27.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:27.996070+0000) 2022-09-10T12:36:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:28.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:28.467840+0000) 2022-09-10T12:36:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:28 smithi110 ceph-mon[97633]: pgmap v15590: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:28 smithi104 ceph-mon[97044]: pgmap v15590: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:28.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:28.996349+0000) 2022-09-10T12:36:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:29.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:29.468002+0000) 2022-09-10T12:36:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:29.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:29.996579+0000) 2022-09-10T12:36:30.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:36:30.224+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:36:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:30.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:30.468239+0000) 2022-09-10T12:36:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:30 smithi110 ceph-mon[97633]: pgmap v15591: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:30 smithi104 ceph-mon[97044]: pgmap v15591: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:30.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:30.996855+0000) 2022-09-10T12:36:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:31.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:31.468453+0000) 2022-09-10T12:36:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:31.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:31.997137+0000) 2022-09-10T12:36:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:32.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:32.468616+0000) 2022-09-10T12:36:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:32 smithi110 ceph-mon[97633]: pgmap v15592: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:32 smithi104 ceph-mon[97044]: pgmap v15592: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:32.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:32.997351+0000) 2022-09-10T12:36:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:33.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:33.468752+0000) 2022-09-10T12:36:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:33.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:33.997537+0000) 2022-09-10T12:36:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:34.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:34.468891+0000) 2022-09-10T12:36:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:34 smithi110 ceph-mon[97633]: pgmap v15593: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:34 smithi104 ceph-mon[97044]: pgmap v15593: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:34.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:34.997775+0000) 2022-09-10T12:36:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:35.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:35.469030+0000) 2022-09-10T12:36:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:35.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:35.997952+0000) 2022-09-10T12:36:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:36.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:36.469212+0000) 2022-09-10T12:36:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:36 smithi110 ceph-mon[97633]: pgmap v15594: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:36 smithi104 ceph-mon[97044]: pgmap v15594: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:36.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:36.998125+0000) 2022-09-10T12:36:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:37.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:37.469378+0000) 2022-09-10T12:36:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:37.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:37.998335+0000) 2022-09-10T12:36:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:38.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:38.469526+0000) 2022-09-10T12:36:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:38 smithi110 ceph-mon[97633]: pgmap v15595: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:38 smithi104 ceph-mon[97044]: pgmap v15595: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:38.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:38.998472+0000) 2022-09-10T12:36:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:39.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:39.469739+0000) 2022-09-10T12:36:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:39.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:39.998643+0000) 2022-09-10T12:36:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:40.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:40.469989+0000) 2022-09-10T12:36:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:40 smithi110 ceph-mon[97633]: pgmap v15596: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:40 smithi104 ceph-mon[97044]: pgmap v15596: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:40.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:40.998858+0000) 2022-09-10T12:36:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:41.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:41.470174+0000) 2022-09-10T12:36:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:41.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:41.999138+0000) 2022-09-10T12:36:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:42.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:42.470380+0000) 2022-09-10T12:36:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:42 smithi110 ceph-mon[97633]: pgmap v15597: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:42 smithi104 ceph-mon[97044]: pgmap v15597: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:42.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:42.999279+0000) 2022-09-10T12:36:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:43.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:43.470606+0000) 2022-09-10T12:36:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:43.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:43.999484+0000) 2022-09-10T12:36:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:44.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:44.470794+0000) 2022-09-10T12:36:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:44 smithi110 ceph-mon[97633]: pgmap v15598: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:44 smithi104 ceph-mon[97044]: pgmap v15598: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:44.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:44.999700+0000) 2022-09-10T12:36:45.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:36:45.241+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:36:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:45.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:45.470945+0000) 2022-09-10T12:36:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:45.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:45.999902+0000) 2022-09-10T12:36:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:46.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:46.471195+0000) 2022-09-10T12:36:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:46 smithi110 ceph-mon[97633]: pgmap v15599: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:36:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:46 smithi104 ceph-mon[97044]: pgmap v15599: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:36:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:36:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:36:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:47.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:47.000153+0000) 2022-09-10T12:36:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:47.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:47.471384+0000) 2022-09-10T12:36:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:47.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:48.000418+0000) 2022-09-10T12:36:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:48.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:48.471573+0000) 2022-09-10T12:36:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:48 smithi110 ceph-mon[97633]: pgmap v15600: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:48 smithi104 ceph-mon[97044]: pgmap v15600: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:48.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:49.000631+0000) 2022-09-10T12:36:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:49.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:49.471791+0000) 2022-09-10T12:36:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:49.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:50.000830+0000) 2022-09-10T12:36:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:50.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:50.472013+0000) 2022-09-10T12:36:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:50 smithi110 ceph-mon[97633]: pgmap v15601: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:50 smithi104 ceph-mon[97044]: pgmap v15601: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:50.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:51.001038+0000) 2022-09-10T12:36:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:51.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:51.472211+0000) 2022-09-10T12:36:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:52.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:52.001306+0000) 2022-09-10T12:36:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:52.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:52.472370+0000) 2022-09-10T12:36:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:52 smithi110 ceph-mon[97633]: pgmap v15602: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:52 smithi104 ceph-mon[97044]: pgmap v15602: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:53.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:53.001490+0000) 2022-09-10T12:36:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:53.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:53.472527+0000) 2022-09-10T12:36:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:54.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:54.001653+0000) 2022-09-10T12:36:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:54.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:54.472738+0000) 2022-09-10T12:36:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:54 smithi110 ceph-mon[97633]: pgmap v15603: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:54 smithi104 ceph-mon[97044]: pgmap v15603: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:55.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:55.001836+0000) 2022-09-10T12:36:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:55.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:55.472972+0000) 2022-09-10T12:36:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:56.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:56.002130+0000) 2022-09-10T12:36:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:56.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:56.473237+0000) 2022-09-10T12:36:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:56 smithi104 ceph-mon[97044]: pgmap v15604: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:57.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:57.002402+0000) 2022-09-10T12:36:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:56 smithi110 ceph-mon[97633]: pgmap v15604: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:36:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:57.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:57.473476+0000) 2022-09-10T12:36:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:58.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:58.002587+0000) 2022-09-10T12:36:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:58.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:58.473669+0000) 2022-09-10T12:36:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:36:58 smithi104 ceph-mon[97044]: pgmap v15605: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:36:59.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:59.002801+0000) 2022-09-10T12:36:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:36:58 smithi110 ceph-mon[97633]: pgmap v15605: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:36:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:36:59.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:36:59.473816+0000) 2022-09-10T12:37:00.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:00.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:00.003041+0000) 2022-09-10T12:37:00.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:37:00.255+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:37:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:00.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:00.474017+0000) 2022-09-10T12:37:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:00 smithi104 ceph-mon[97044]: pgmap v15606: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:01.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:01.003291+0000) 2022-09-10T12:37:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:00 smithi110 ceph-mon[97633]: pgmap v15606: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:01.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:01.474177+0000) 2022-09-10T12:37:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:02.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:02.003529+0000) 2022-09-10T12:37:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:02.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:02.474341+0000) 2022-09-10T12:37:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:02 smithi104 ceph-mon[97044]: pgmap v15607: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:03.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:03.003771+0000) 2022-09-10T12:37:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:02 smithi110 ceph-mon[97633]: pgmap v15607: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:03.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:03.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:03.474522+0000) 2022-09-10T12:37:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:04.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:04.003963+0000) 2022-09-10T12:37:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:04.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:04.474718+0000) 2022-09-10T12:37:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:04 smithi104 ceph-mon[97044]: pgmap v15608: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:05.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:05.004194+0000) 2022-09-10T12:37:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:04 smithi110 ceph-mon[97633]: pgmap v15608: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:05.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:05.474933+0000) 2022-09-10T12:37:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:06.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:06.004425+0000) 2022-09-10T12:37:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:06.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:06.475148+0000) 2022-09-10T12:37:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:06 smithi104 ceph-mon[97044]: pgmap v15609: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:37:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:07.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:07.004670+0000) 2022-09-10T12:37:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:06 smithi110 ceph-mon[97633]: pgmap v15609: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:37:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:07.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:07.475327+0000) 2022-09-10T12:37:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:08.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:08.004941+0000) 2022-09-10T12:37:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:08.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:08.475553+0000) 2022-09-10T12:37:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:08 smithi104 ceph-mon[97044]: pgmap v15610: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:37:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:08 smithi104 ceph-mon[97044]: osdmap e1056: 12 total, 12 up, 12 in 2022-09-10T12:37:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:09.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:09.005166+0000) 2022-09-10T12:37:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:08 smithi110 ceph-mon[97633]: pgmap v15610: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:37:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:08 smithi110 ceph-mon[97633]: osdmap e1056: 12 total, 12 up, 12 in 2022-09-10T12:37:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:09.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:09.475732+0000) 2022-09-10T12:37:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:10.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:10.005331+0000) 2022-09-10T12:37:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:10.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:10.475935+0000) 2022-09-10T12:37:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:10 smithi104 ceph-mon[97044]: pgmap v15612: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:37:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:11.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:11.005513+0000) 2022-09-10T12:37:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:10 smithi110 ceph-mon[97633]: pgmap v15612: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:37:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:11.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:11.476120+0000) 2022-09-10T12:37:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:12.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:12.005710+0000) 2022-09-10T12:37:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:12.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:12.476256+0000) 2022-09-10T12:37:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:12 smithi104 ceph-mon[97044]: pgmap v15613: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:37:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:13.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:13.005939+0000) 2022-09-10T12:37:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:12 smithi110 ceph-mon[97633]: pgmap v15613: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:37:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:13.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:13.476469+0000) 2022-09-10T12:37:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:14.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:14.006190+0000) 2022-09-10T12:37:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:14.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:14.476622+0000) 2022-09-10T12:37:15.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:15.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:15.006460+0000) 2022-09-10T12:37:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:14 smithi104 ceph-mon[97044]: pgmap v15614: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:37:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:14 smithi104 ceph-mon[97044]: osdmap e1057: 12 total, 12 up, 12 in 2022-09-10T12:37:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:14 smithi110 ceph-mon[97633]: pgmap v15614: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:37:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:14 smithi110 ceph-mon[97633]: osdmap e1057: 12 total, 12 up, 12 in 2022-09-10T12:37:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:15.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:15.476837+0000) 2022-09-10T12:37:15.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:37:15.260+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:37:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:16.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:16.006638+0000) 2022-09-10T12:37:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:16.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:16.476974+0000) 2022-09-10T12:37:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:16 smithi104 ceph-mon[97044]: pgmap v15616: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:37:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:17.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:17.006878+0000) 2022-09-10T12:37:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:16 smithi110 ceph-mon[97633]: pgmap v15616: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:37:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:17.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:17.477215+0000) 2022-09-10T12:37:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:18.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:18.007033+0000) 2022-09-10T12:37:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:18.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:18.477346+0000) 2022-09-10T12:37:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:18 smithi104 ceph-mon[97044]: pgmap v15617: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:37:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:19.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:19.007271+0000) 2022-09-10T12:37:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:18 smithi110 ceph-mon[97633]: pgmap v15617: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:37:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:19.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:19.477534+0000) 2022-09-10T12:37:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:20.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:20.007543+0000) 2022-09-10T12:37:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:20.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:20.477730+0000) 2022-09-10T12:37:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:20 smithi104 ceph-mon[97044]: pgmap v15618: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:37:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:21.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:21.007744+0000) 2022-09-10T12:37:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:20 smithi110 ceph-mon[97633]: pgmap v15618: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:37:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:21.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:21.477931+0000) 2022-09-10T12:37:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:22.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:22.007995+0000) 2022-09-10T12:37:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:22.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:22.478169+0000) 2022-09-10T12:37:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:22 smithi104 ceph-mon[97044]: pgmap v15619: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:37:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:23.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:23.008179+0000) 2022-09-10T12:37:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:22 smithi110 ceph-mon[97633]: pgmap v15619: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:37:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:23.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:23.478319+0000) 2022-09-10T12:37:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:24.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:24.008335+0000) 2022-09-10T12:37:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:24.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:24.478450+0000) 2022-09-10T12:37:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:24 smithi104 ceph-mon[97044]: pgmap v15620: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:37:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:25.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:25.008478+0000) 2022-09-10T12:37:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:24 smithi110 ceph-mon[97633]: pgmap v15620: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:37:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:25.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:25.478610+0000) 2022-09-10T12:37:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:26.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:26.008670+0000) 2022-09-10T12:37:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:26.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:26.478761+0000) 2022-09-10T12:37:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:26 smithi104 ceph-mon[97044]: pgmap v15621: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:37:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:27.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:27.008901+0000) 2022-09-10T12:37:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:26 smithi110 ceph-mon[97633]: pgmap v15621: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:37:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:27.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:27.478910+0000) 2022-09-10T12:37:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:28.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:28.009155+0000) 2022-09-10T12:37:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:28.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:28.479088+0000) 2022-09-10T12:37:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:28 smithi104 ceph-mon[97044]: pgmap v15622: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:37:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:29.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:29.009312+0000) 2022-09-10T12:37:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:28 smithi110 ceph-mon[97633]: pgmap v15622: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:37:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:29.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:29.479312+0000) 2022-09-10T12:37:30.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:30.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:30.009468+0000) 2022-09-10T12:37:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:30.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:30.479438+0000) 2022-09-10T12:37:30.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:37:30.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:37:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:30 smithi104 ceph-mon[97044]: pgmap v15623: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:37:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:31.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:31.009588+0000) 2022-09-10T12:37:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:30 smithi110 ceph-mon[97633]: pgmap v15623: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:37:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:31.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:31.479592+0000) 2022-09-10T12:37:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:32.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:32.009841+0000) 2022-09-10T12:37:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:32.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:32.479749+0000) 2022-09-10T12:37:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:32 smithi104 ceph-mon[97044]: pgmap v15624: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:33.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:33.010000+0000) 2022-09-10T12:37:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:32 smithi110 ceph-mon[97633]: pgmap v15624: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:33.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:33.479917+0000) 2022-09-10T12:37:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:34.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:34.010193+0000) 2022-09-10T12:37:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:34.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:34.480092+0000) 2022-09-10T12:37:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:34 smithi104 ceph-mon[97044]: pgmap v15625: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:37:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:35.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:35.010382+0000) 2022-09-10T12:37:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:34 smithi110 ceph-mon[97633]: pgmap v15625: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:37:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:35.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:35.480236+0000) 2022-09-10T12:37:36.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:36.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:36.010611+0000) 2022-09-10T12:37:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:36.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:36.480397+0000) 2022-09-10T12:37:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:36 smithi104 ceph-mon[97044]: pgmap v15626: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:37.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:37.010887+0000) 2022-09-10T12:37:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:36 smithi110 ceph-mon[97633]: pgmap v15626: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:37.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:37.480645+0000) 2022-09-10T12:37:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:38.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:38.011172+0000) 2022-09-10T12:37:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:38.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:38.480841+0000) 2022-09-10T12:37:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:38 smithi104 ceph-mon[97044]: pgmap v15627: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:39.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:39.011417+0000) 2022-09-10T12:37:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:38 smithi110 ceph-mon[97633]: pgmap v15627: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:39.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:39.481029+0000) 2022-09-10T12:37:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:40.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:40.011689+0000) 2022-09-10T12:37:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:40.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:40.481212+0000) 2022-09-10T12:37:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:40 smithi104 ceph-mon[97044]: pgmap v15628: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:40 smithi110 ceph-mon[97633]: pgmap v15628: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:41.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:41.011944+0000) 2022-09-10T12:37:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:41.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:41.481351+0000) 2022-09-10T12:37:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:42.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:42.012129+0000) 2022-09-10T12:37:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:42.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:42.481571+0000) 2022-09-10T12:37:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:42 smithi104 ceph-mon[97044]: pgmap v15629: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:43.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:43.012315+0000) 2022-09-10T12:37:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:42 smithi110 ceph-mon[97633]: pgmap v15629: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:43.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:43.481807+0000) 2022-09-10T12:37:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:44.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:44.012516+0000) 2022-09-10T12:37:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:44.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:44.482013+0000) 2022-09-10T12:37:45.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:45.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:45.012770+0000) 2022-09-10T12:37:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:44 smithi104 ceph-mon[97044]: pgmap v15630: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:44 smithi110 ceph-mon[97633]: pgmap v15630: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:45.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:45.482164+0000) 2022-09-10T12:37:45.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:37:45.282+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:37:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:46.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:46.012969+0000) 2022-09-10T12:37:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:46.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:46.482340+0000) 2022-09-10T12:37:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:46 smithi104 ceph-mon[97044]: pgmap v15631: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:37:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:37:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:37:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:37:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:47.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:47.013223+0000) 2022-09-10T12:37:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:46 smithi110 ceph-mon[97633]: pgmap v15631: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:37:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:37:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:37:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:37:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:47.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:47.482576+0000) 2022-09-10T12:37:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:48.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:48.013404+0000) 2022-09-10T12:37:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:48.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:48.482767+0000) 2022-09-10T12:37:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:48 smithi104 ceph-mon[97044]: pgmap v15632: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:49.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:49.013584+0000) 2022-09-10T12:37:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:48 smithi110 ceph-mon[97633]: pgmap v15632: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:49.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:49.482978+0000) 2022-09-10T12:37:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:50.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:50.013766+0000) 2022-09-10T12:37:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:50.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:50.483123+0000) 2022-09-10T12:37:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:50 smithi104 ceph-mon[97044]: pgmap v15633: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:51.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:51.014028+0000) 2022-09-10T12:37:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:50 smithi110 ceph-mon[97633]: pgmap v15633: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:51.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:51.483287+0000) 2022-09-10T12:37:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:52.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:52.014154+0000) 2022-09-10T12:37:52.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:52 smithi110 ceph-mon[97633]: pgmap v15634: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:52.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:52 smithi104 ceph-mon[97044]: pgmap v15634: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:52.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:52.483526+0000) 2022-09-10T12:37:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:53.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:53.014313+0000) 2022-09-10T12:37:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:53.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:53.483672+0000) 2022-09-10T12:37:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:54.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:54.014569+0000) 2022-09-10T12:37:54.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:54 smithi104 ceph-mon[97044]: pgmap v15635: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:54.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:54 smithi110 ceph-mon[97633]: pgmap v15635: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:54.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:54.483831+0000) 2022-09-10T12:37:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:55.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:55.014832+0000) 2022-09-10T12:37:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:55.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:55.484014+0000) 2022-09-10T12:37:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:56.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:56.015127+0000) 2022-09-10T12:37:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:56.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:56.484176+0000) 2022-09-10T12:37:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:56 smithi110 ceph-mon[97633]: pgmap v15636: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:56 smithi104 ceph-mon[97044]: pgmap v15636: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:37:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:57.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:57.015267+0000) 2022-09-10T12:37:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:57.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:57.484318+0000) 2022-09-10T12:37:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:58.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:58.015496+0000) 2022-09-10T12:37:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:58.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:58.484461+0000) 2022-09-10T12:37:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:37:58 smithi110 ceph-mon[97633]: pgmap v15637: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:37:58 smithi104 ceph-mon[97044]: pgmap v15637: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:37:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:37:59.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:59.015716+0000) 2022-09-10T12:37:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:37:59.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:37:59.484639+0000) 2022-09-10T12:38:00.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:00.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:00.015949+0000) 2022-09-10T12:38:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:00.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:00.484767+0000) 2022-09-10T12:38:00.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:38:00.291+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:38:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:00 smithi110 ceph-mon[97633]: pgmap v15638: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:00 smithi104 ceph-mon[97044]: pgmap v15638: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:01.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:01.016154+0000) 2022-09-10T12:38:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:01.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:01.484909+0000) 2022-09-10T12:38:02.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:02.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:02.016362+0000) 2022-09-10T12:38:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:02.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:02.485070+0000) 2022-09-10T12:38:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:02 smithi110 ceph-mon[97633]: pgmap v15639: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:02 smithi104 ceph-mon[97044]: pgmap v15639: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:03.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:03.016611+0000) 2022-09-10T12:38:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:03.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:03.485250+0000) 2022-09-10T12:38:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:04.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:04.016889+0000) 2022-09-10T12:38:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:04.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:04.485390+0000) 2022-09-10T12:38:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:04 smithi110 ceph-mon[97633]: pgmap v15640: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:04 smithi104 ceph-mon[97044]: pgmap v15640: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:05.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:05.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:05.017187+0000) 2022-09-10T12:38:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:05.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:05.485532+0000) 2022-09-10T12:38:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:06.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:06.017405+0000) 2022-09-10T12:38:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:06.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:06.485695+0000) 2022-09-10T12:38:07.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:06 smithi110 ceph-mon[97633]: pgmap v15641: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:38:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:06 smithi104 ceph-mon[97044]: pgmap v15641: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:38:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:07.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:07.017655+0000) 2022-09-10T12:38:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:07.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:07.485862+0000) 2022-09-10T12:38:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:08.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:08.017893+0000) 2022-09-10T12:38:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:08.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:08.486019+0000) 2022-09-10T12:38:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:08 smithi110 ceph-mon[97633]: pgmap v15642: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:38:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:08 smithi110 ceph-mon[97633]: osdmap e1058: 12 total, 12 up, 12 in 2022-09-10T12:38:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:08 smithi104 ceph-mon[97044]: pgmap v15642: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:38:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:08 smithi104 ceph-mon[97044]: osdmap e1058: 12 total, 12 up, 12 in 2022-09-10T12:38:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:09.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:09.018174+0000) 2022-09-10T12:38:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:09.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:09.486222+0000) 2022-09-10T12:38:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:10.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:10.018449+0000) 2022-09-10T12:38:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:10.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:10.486354+0000) 2022-09-10T12:38:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:10 smithi110 ceph-mon[97633]: pgmap v15644: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:38:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:10 smithi104 ceph-mon[97044]: pgmap v15644: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:38:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:11.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:11.018639+0000) 2022-09-10T12:38:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:11.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:11.486487+0000) 2022-09-10T12:38:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:12.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:12.018867+0000) 2022-09-10T12:38:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:12.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:12.486656+0000) 2022-09-10T12:38:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:12 smithi110 ceph-mon[97633]: pgmap v15645: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:38:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:12 smithi104 ceph-mon[97044]: pgmap v15645: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:38:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:13.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:13.019155+0000) 2022-09-10T12:38:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:13.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:13.486828+0000) 2022-09-10T12:38:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:14.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:14.019349+0000) 2022-09-10T12:38:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:14.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:14.487047+0000) 2022-09-10T12:38:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:14 smithi110 ceph-mon[97633]: pgmap v15646: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:38:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:14 smithi110 ceph-mon[97633]: osdmap e1059: 12 total, 12 up, 12 in 2022-09-10T12:38:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:14 smithi104 ceph-mon[97044]: pgmap v15646: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:38:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:14 smithi104 ceph-mon[97044]: osdmap e1059: 12 total, 12 up, 12 in 2022-09-10T12:38:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:15.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:15.019615+0000) 2022-09-10T12:38:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:15.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:15.487217+0000) 2022-09-10T12:38:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:38:15.306+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:38:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:16.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:16.019846+0000) 2022-09-10T12:38:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:16.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:16.487326+0000) 2022-09-10T12:38:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:16 smithi110 ceph-mon[97633]: pgmap v15648: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:38:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:16 smithi104 ceph-mon[97044]: pgmap v15648: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:38:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:17.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:17.020062+0000) 2022-09-10T12:38:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:17.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:17.487523+0000) 2022-09-10T12:38:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:18.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:18.020364+0000) 2022-09-10T12:38:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:18.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:18.487724+0000) 2022-09-10T12:38:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:18 smithi110 ceph-mon[97633]: pgmap v15649: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:38:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:18 smithi104 ceph-mon[97044]: pgmap v15649: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:38:19.314 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:19.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:19.020629+0000) 2022-09-10T12:38:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:19.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:19.487926+0000) 2022-09-10T12:38:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:20.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:20.020822+0000) 2022-09-10T12:38:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:20.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:20.488065+0000) 2022-09-10T12:38:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:20 smithi110 ceph-mon[97633]: pgmap v15650: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:38:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:20 smithi104 ceph-mon[97044]: pgmap v15650: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:38:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:21.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:21.021034+0000) 2022-09-10T12:38:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:21.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:21.488170+0000) 2022-09-10T12:38:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:22.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:22.021173+0000) 2022-09-10T12:38:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:22.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:22.488334+0000) 2022-09-10T12:38:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:22 smithi110 ceph-mon[97633]: pgmap v15651: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:38:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:22 smithi104 ceph-mon[97044]: pgmap v15651: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:38:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:23.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:23.021408+0000) 2022-09-10T12:38:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:23.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:23.488500+0000) 2022-09-10T12:38:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:24.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:24.021546+0000) 2022-09-10T12:38:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:24.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:24.488668+0000) 2022-09-10T12:38:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:24 smithi110 ceph-mon[97633]: pgmap v15652: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:38:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:24 smithi104 ceph-mon[97044]: pgmap v15652: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:38:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:25.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:25.021707+0000) 2022-09-10T12:38:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:25.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:25.488794+0000) 2022-09-10T12:38:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:26.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:26.021842+0000) 2022-09-10T12:38:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:26.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:26.488962+0000) 2022-09-10T12:38:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:26 smithi110 ceph-mon[97633]: pgmap v15653: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:38:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:26 smithi104 ceph-mon[97044]: pgmap v15653: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:38:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:27.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:27.021994+0000) 2022-09-10T12:38:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:27.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:27.489182+0000) 2022-09-10T12:38:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:28.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:28.022191+0000) 2022-09-10T12:38:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:28.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:28.489401+0000) 2022-09-10T12:38:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:28 smithi110 ceph-mon[97633]: pgmap v15654: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:28 smithi104 ceph-mon[97044]: pgmap v15654: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:29.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:29.022366+0000) 2022-09-10T12:38:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:29.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:29.489592+0000) 2022-09-10T12:38:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:30.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:30.022578+0000) 2022-09-10T12:38:30.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:30.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:30.489754+0000) 2022-09-10T12:38:30.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:38:30.321+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:38:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:30 smithi110 ceph-mon[97633]: pgmap v15655: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:30 smithi104 ceph-mon[97044]: pgmap v15655: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:31.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:31.022755+0000) 2022-09-10T12:38:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:31.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:31.489913+0000) 2022-09-10T12:38:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:32.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:32.022980+0000) 2022-09-10T12:38:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:32.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:32.490085+0000) 2022-09-10T12:38:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:32 smithi110 ceph-mon[97633]: pgmap v15656: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:32 smithi104 ceph-mon[97044]: pgmap v15656: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:33.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:33.023203+0000) 2022-09-10T12:38:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:33.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:33.490288+0000) 2022-09-10T12:38:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:34.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:34.023419+0000) 2022-09-10T12:38:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:34.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:34.490415+0000) 2022-09-10T12:38:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:34 smithi104 ceph-mon[97044]: pgmap v15657: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:35.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:35.023650+0000) 2022-09-10T12:38:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:34 smithi110 ceph-mon[97633]: pgmap v15657: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:35.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:35.490601+0000) 2022-09-10T12:38:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:36.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:36.023821+0000) 2022-09-10T12:38:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:36.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:36.490719+0000) 2022-09-10T12:38:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:36 smithi104 ceph-mon[97044]: pgmap v15658: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:37.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:37.024051+0000) 2022-09-10T12:38:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:36 smithi110 ceph-mon[97633]: pgmap v15658: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:37.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:37.490841+0000) 2022-09-10T12:38:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:38.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:38.024234+0000) 2022-09-10T12:38:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:38.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:38.490990+0000) 2022-09-10T12:38:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:38 smithi104 ceph-mon[97044]: pgmap v15659: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:39.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:39.024464+0000) 2022-09-10T12:38:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:38 smithi110 ceph-mon[97633]: pgmap v15659: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:39.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:39.491165+0000) 2022-09-10T12:38:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:40.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:40.024630+0000) 2022-09-10T12:38:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:40.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:40.491296+0000) 2022-09-10T12:38:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:40 smithi104 ceph-mon[97044]: pgmap v15660: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:41.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:41.024806+0000) 2022-09-10T12:38:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:40 smithi110 ceph-mon[97633]: pgmap v15660: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:41.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:41.491426+0000) 2022-09-10T12:38:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:42.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:42.024954+0000) 2022-09-10T12:38:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:42.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:42.491564+0000) 2022-09-10T12:38:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:42 smithi104 ceph-mon[97044]: pgmap v15661: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:43.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:43.025195+0000) 2022-09-10T12:38:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:42 smithi110 ceph-mon[97633]: pgmap v15661: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:43.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:43.491702+0000) 2022-09-10T12:38:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:44.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:44.025428+0000) 2022-09-10T12:38:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:44.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:44.491842+0000) 2022-09-10T12:38:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:44 smithi104 ceph-mon[97044]: pgmap v15662: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:45.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:45.025693+0000) 2022-09-10T12:38:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:44 smithi110 ceph-mon[97633]: pgmap v15662: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:45.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:45.492038+0000) 2022-09-10T12:38:45.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:38:45.337+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:38:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:46.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:46.025945+0000) 2022-09-10T12:38:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:46.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:46.492238+0000) 2022-09-10T12:38:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:46 smithi104 ceph-mon[97044]: pgmap v15663: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:38:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:47.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:47.026151+0000) 2022-09-10T12:38:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:46 smithi110 ceph-mon[97633]: pgmap v15663: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:38:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:38:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:38:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:38:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:47.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:47.492433+0000) 2022-09-10T12:38:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:48.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:48.026394+0000) 2022-09-10T12:38:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:48.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:48.492594+0000) 2022-09-10T12:38:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:48 smithi104 ceph-mon[97044]: pgmap v15664: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:49.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:49.026585+0000) 2022-09-10T12:38:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:48 smithi110 ceph-mon[97633]: pgmap v15664: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:49.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:49.492768+0000) 2022-09-10T12:38:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:50.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:50.026804+0000) 2022-09-10T12:38:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:50.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:50.492953+0000) 2022-09-10T12:38:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:50 smithi104 ceph-mon[97044]: pgmap v15665: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:51.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:51.027045+0000) 2022-09-10T12:38:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:50 smithi110 ceph-mon[97633]: pgmap v15665: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:51.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:51.493142+0000) 2022-09-10T12:38:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:52.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:52.027264+0000) 2022-09-10T12:38:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:52.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:52.493276+0000) 2022-09-10T12:38:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:52 smithi104 ceph-mon[97044]: pgmap v15666: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:53.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:53.027533+0000) 2022-09-10T12:38:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:52 smithi110 ceph-mon[97633]: pgmap v15666: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:53.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:53.493513+0000) 2022-09-10T12:38:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:54.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:54.027773+0000) 2022-09-10T12:38:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:54.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:54.493701+0000) 2022-09-10T12:38:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:54 smithi104 ceph-mon[97044]: pgmap v15667: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:55.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:55.028148+0000) 2022-09-10T12:38:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:54 smithi110 ceph-mon[97633]: pgmap v15667: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:55.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:55.493875+0000) 2022-09-10T12:38:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:56.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:56.028348+0000) 2022-09-10T12:38:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:56.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:56.494018+0000) 2022-09-10T12:38:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:56 smithi104 ceph-mon[97044]: pgmap v15668: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:57.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:57.028563+0000) 2022-09-10T12:38:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:56 smithi110 ceph-mon[97633]: pgmap v15668: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:38:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:57.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:57.494188+0000) 2022-09-10T12:38:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:58.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:58.028756+0000) 2022-09-10T12:38:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:58.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:58.494330+0000) 2022-09-10T12:38:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:38:58 smithi104 ceph-mon[97044]: pgmap v15669: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:38:59.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:59.028986+0000) 2022-09-10T12:38:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:38:58 smithi110 ceph-mon[97633]: pgmap v15669: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:38:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:38:59.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:38:59.494519+0000) 2022-09-10T12:39:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:00.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:00.029151+0000) 2022-09-10T12:39:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:00.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:00.494685+0000) 2022-09-10T12:39:00.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:39:00.339+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:39:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:00 smithi104 ceph-mon[97044]: pgmap v15670: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:01.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:01.029307+0000) 2022-09-10T12:39:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:00 smithi110 ceph-mon[97633]: pgmap v15670: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:01.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:01.494809+0000) 2022-09-10T12:39:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:02.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:02.029470+0000) 2022-09-10T12:39:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:02.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:02.494947+0000) 2022-09-10T12:39:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:02 smithi104 ceph-mon[97044]: pgmap v15671: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:03.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:03.029706+0000) 2022-09-10T12:39:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:02 smithi110 ceph-mon[97633]: pgmap v15671: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:03.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:03.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:03.495113+0000) 2022-09-10T12:39:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:04.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:04.029964+0000) 2022-09-10T12:39:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:04.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:04.495266+0000) 2022-09-10T12:39:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:04 smithi104 ceph-mon[97044]: pgmap v15672: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:05.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:05.030235+0000) 2022-09-10T12:39:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:04 smithi110 ceph-mon[97633]: pgmap v15672: 129 pgs: 129 active+clean; 9.4 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:05.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:05.495469+0000) 2022-09-10T12:39:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:06.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:06.030488+0000) 2022-09-10T12:39:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:06.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:06.495689+0000) 2022-09-10T12:39:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:06 smithi104 ceph-mon[97044]: pgmap v15673: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:39:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:07.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:07.030739+0000) 2022-09-10T12:39:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:06 smithi110 ceph-mon[97633]: pgmap v15673: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:39:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:07.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:07.495840+0000) 2022-09-10T12:39:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:08.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:08.030939+0000) 2022-09-10T12:39:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:08.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:08.495993+0000) 2022-09-10T12:39:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:08 smithi104 ceph-mon[97044]: pgmap v15674: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:39:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:08 smithi104 ceph-mon[97044]: osdmap e1060: 12 total, 12 up, 12 in 2022-09-10T12:39:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:09.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:09.031214+0000) 2022-09-10T12:39:09.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:08 smithi110 ceph-mon[97633]: pgmap v15674: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:39:09.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:08 smithi110 ceph-mon[97633]: osdmap e1060: 12 total, 12 up, 12 in 2022-09-10T12:39:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:09.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:09.496169+0000) 2022-09-10T12:39:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:10.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:10.031453+0000) 2022-09-10T12:39:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:10.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:10.496359+0000) 2022-09-10T12:39:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:10 smithi104 ceph-mon[97044]: pgmap v15676: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:39:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:11.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:11.031661+0000) 2022-09-10T12:39:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:10 smithi110 ceph-mon[97633]: pgmap v15676: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:39:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:11.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:11.496597+0000) 2022-09-10T12:39:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:12.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:12.031883+0000) 2022-09-10T12:39:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:12.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:12.496752+0000) 2022-09-10T12:39:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:12 smithi104 ceph-mon[97044]: pgmap v15677: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:39:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:13.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:13.032140+0000) 2022-09-10T12:39:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:12 smithi110 ceph-mon[97633]: pgmap v15677: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:39:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:13.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:13.496946+0000) 2022-09-10T12:39:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:14.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:14.032369+0000) 2022-09-10T12:39:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:14.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:14.497108+0000) 2022-09-10T12:39:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:14 smithi104 ceph-mon[97044]: pgmap v15678: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:39:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:14 smithi104 ceph-mon[97044]: osdmap e1061: 12 total, 12 up, 12 in 2022-09-10T12:39:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:15.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:15.032537+0000) 2022-09-10T12:39:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:14 smithi110 ceph-mon[97633]: pgmap v15678: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:39:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:14 smithi110 ceph-mon[97633]: osdmap e1061: 12 total, 12 up, 12 in 2022-09-10T12:39:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:15.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:15.497245+0000) 2022-09-10T12:39:15.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:39:15.346+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:39:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:16.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:16.032732+0000) 2022-09-10T12:39:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:16.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:16.497371+0000) 2022-09-10T12:39:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:16 smithi104 ceph-mon[97044]: pgmap v15680: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:39:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:17.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:17.032932+0000) 2022-09-10T12:39:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:16 smithi110 ceph-mon[97633]: pgmap v15680: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:39:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:17.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:17.497560+0000) 2022-09-10T12:39:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:18.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:18.033198+0000) 2022-09-10T12:39:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:18.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:18.497709+0000) 2022-09-10T12:39:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:18 smithi104 ceph-mon[97044]: pgmap v15681: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:39:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:19.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:19.033400+0000) 2022-09-10T12:39:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:18 smithi110 ceph-mon[97633]: pgmap v15681: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:39:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:19.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:19.497865+0000) 2022-09-10T12:39:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:20.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:20.033653+0000) 2022-09-10T12:39:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:20.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:20.498050+0000) 2022-09-10T12:39:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:20 smithi104 ceph-mon[97044]: pgmap v15682: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:39:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:21.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:21.033897+0000) 2022-09-10T12:39:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:20 smithi110 ceph-mon[97633]: pgmap v15682: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:39:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:21.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:21.498249+0000) 2022-09-10T12:39:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:22.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:22.034175+0000) 2022-09-10T12:39:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:22.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:22.498466+0000) 2022-09-10T12:39:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:22 smithi104 ceph-mon[97044]: pgmap v15683: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:39:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:23.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:23.034364+0000) 2022-09-10T12:39:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:22 smithi110 ceph-mon[97633]: pgmap v15683: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:39:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:23.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:23.498673+0000) 2022-09-10T12:39:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:24.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:24.034539+0000) 2022-09-10T12:39:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:24.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:24.498844+0000) 2022-09-10T12:39:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:24 smithi104 ceph-mon[97044]: pgmap v15684: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:39:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:25.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:25.034730+0000) 2022-09-10T12:39:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:24 smithi110 ceph-mon[97633]: pgmap v15684: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:39:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:25.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:25.499041+0000) 2022-09-10T12:39:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:26.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:26.034934+0000) 2022-09-10T12:39:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:26.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:26.499293+0000) 2022-09-10T12:39:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:26 smithi104 ceph-mon[97044]: pgmap v15685: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:39:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:27.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:27.035191+0000) 2022-09-10T12:39:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:26 smithi110 ceph-mon[97633]: pgmap v15685: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:39:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:27.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:27.499480+0000) 2022-09-10T12:39:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:28.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:28.035418+0000) 2022-09-10T12:39:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:28.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:28.499630+0000) 2022-09-10T12:39:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:28 smithi104 ceph-mon[97044]: pgmap v15686: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:29.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:29.035603+0000) 2022-09-10T12:39:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:28 smithi110 ceph-mon[97633]: pgmap v15686: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:29.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:29.499833+0000) 2022-09-10T12:39:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:30.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:30.035769+0000) 2022-09-10T12:39:30.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:39:30.353+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:39:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:30.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:30.499999+0000) 2022-09-10T12:39:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:30 smithi104 ceph-mon[97044]: pgmap v15687: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:31.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:31.036029+0000) 2022-09-10T12:39:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:30 smithi110 ceph-mon[97633]: pgmap v15687: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:31.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:31.500212+0000) 2022-09-10T12:39:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:32.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:32.036286+0000) 2022-09-10T12:39:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:32.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:32.500391+0000) 2022-09-10T12:39:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:32 smithi104 ceph-mon[97044]: pgmap v15688: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:33.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:33.036564+0000) 2022-09-10T12:39:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:32 smithi110 ceph-mon[97633]: pgmap v15688: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:33.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:33.500551+0000) 2022-09-10T12:39:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:34.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:34.036847+0000) 2022-09-10T12:39:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:34.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:34.500681+0000) 2022-09-10T12:39:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:35 smithi104 ceph-mon[97044]: pgmap v15689: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:35.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:35.037057+0000) 2022-09-10T12:39:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:34 smithi110 ceph-mon[97633]: pgmap v15689: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:35.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:35.500868+0000) 2022-09-10T12:39:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:36.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:36.037223+0000) 2022-09-10T12:39:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:36.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:36.501007+0000) 2022-09-10T12:39:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:37 smithi104 ceph-mon[97044]: pgmap v15690: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:37.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:37.037435+0000) 2022-09-10T12:39:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:37 smithi110 ceph-mon[97633]: pgmap v15690: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:37.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:37.501197+0000) 2022-09-10T12:39:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:38.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:38.037689+0000) 2022-09-10T12:39:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:38.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:38.501372+0000) 2022-09-10T12:39:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:39 smithi104 ceph-mon[97044]: pgmap v15691: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:39.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:39.037912+0000) 2022-09-10T12:39:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:39 smithi110 ceph-mon[97633]: pgmap v15691: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:39.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:39.501578+0000) 2022-09-10T12:39:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:40.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:40.038216+0000) 2022-09-10T12:39:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:40.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:40.501802+0000) 2022-09-10T12:39:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:41 smithi104 ceph-mon[97044]: pgmap v15692: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:41.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:41.038424+0000) 2022-09-10T12:39:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:41 smithi110 ceph-mon[97633]: pgmap v15692: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:41.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:41.501948+0000) 2022-09-10T12:39:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:42.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:42.038610+0000) 2022-09-10T12:39:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:42.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:42.502196+0000) 2022-09-10T12:39:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:43 smithi104 ceph-mon[97044]: pgmap v15693: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:43.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:43.038843+0000) 2022-09-10T12:39:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:43 smithi110 ceph-mon[97633]: pgmap v15693: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:43.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:43.502379+0000) 2022-09-10T12:39:44.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:44 smithi104 ceph-mon[97044]: pgmap v15694: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:39:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:44.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:44.039130+0000) 2022-09-10T12:39:44.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:44 smithi110 ceph-mon[97633]: pgmap v15694: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:39:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:44.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:44.502564+0000) 2022-09-10T12:39:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:45.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:45.039437+0000) 2022-09-10T12:39:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:45.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:45.502779+0000) 2022-09-10T12:39:45.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:39:45.355+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:39:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:46.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:46.039701+0000) 2022-09-10T12:39:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:46.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:46.502956+0000) 2022-09-10T12:39:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:46 smithi110 ceph-mon[97633]: pgmap v15695: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:39:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:39:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:39:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:39:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:46 smithi104 ceph-mon[97044]: pgmap v15695: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:39:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:39:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:39:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:39:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:47.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:47.039906+0000) 2022-09-10T12:39:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:47.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:47.503194+0000) 2022-09-10T12:39:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:48.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:48.040164+0000) 2022-09-10T12:39:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:48.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:48.503403+0000) 2022-09-10T12:39:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:48 smithi110 ceph-mon[97633]: pgmap v15696: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:39:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:48 smithi104 ceph-mon[97044]: pgmap v15696: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:39:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:49.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:49.040351+0000) 2022-09-10T12:39:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:49.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:49.503607+0000) 2022-09-10T12:39:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:50.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:50.040581+0000) 2022-09-10T12:39:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:50.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:50.503836+0000) 2022-09-10T12:39:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:50 smithi110 ceph-mon[97633]: pgmap v15697: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:39:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:50 smithi104 ceph-mon[97044]: pgmap v15697: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:39:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:51.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:51.040823+0000) 2022-09-10T12:39:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:51.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:51.504076+0000) 2022-09-10T12:39:52.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:52.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:52.041056+0000) 2022-09-10T12:39:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:52.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:52.504327+0000) 2022-09-10T12:39:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:52 smithi110 ceph-mon[97633]: pgmap v15698: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:52 smithi104 ceph-mon[97044]: pgmap v15698: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:53.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:53.041249+0000) 2022-09-10T12:39:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:53.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:53.504497+0000) 2022-09-10T12:39:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:54.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:54.041486+0000) 2022-09-10T12:39:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:54.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:54.504729+0000) 2022-09-10T12:39:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:54 smithi110 ceph-mon[97633]: pgmap v15699: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:54 smithi104 ceph-mon[97044]: pgmap v15699: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:55.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:55.041722+0000) 2022-09-10T12:39:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:55.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:55.504962+0000) 2022-09-10T12:39:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:56.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:56.041912+0000) 2022-09-10T12:39:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:56.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:56.505220+0000) 2022-09-10T12:39:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:56 smithi110 ceph-mon[97633]: pgmap v15700: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:56 smithi104 ceph-mon[97044]: pgmap v15700: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:39:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:57.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:57.042067+0000) 2022-09-10T12:39:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:57.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:57.505373+0000) 2022-09-10T12:39:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:58.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:58.042261+0000) 2022-09-10T12:39:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:58.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:58.505591+0000) 2022-09-10T12:39:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:39:58 smithi110 ceph-mon[97633]: pgmap v15701: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:39:58 smithi104 ceph-mon[97044]: pgmap v15701: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:39:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:39:59.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:59.042491+0000) 2022-09-10T12:39:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:39:59.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:39:59.505817+0000) 2022-09-10T12:40:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:00.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:00.042726+0000) 2022-09-10T12:40:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:00.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:00.506018+0000) 2022-09-10T12:40:00.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:40:00.371+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:40:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:00 smithi110 ceph-mon[97633]: pgmap v15702: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:40:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:00 smithi104 ceph-mon[97044]: pgmap v15702: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:40:01.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:01.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:01.042953+0000) 2022-09-10T12:40:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:01.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:01.506242+0000) 2022-09-10T12:40:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:02.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:02.043155+0000) 2022-09-10T12:40:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:02.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:02.506376+0000) 2022-09-10T12:40:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:02 smithi104 ceph-mon[97044]: pgmap v15703: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:03.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:03.043298+0000) 2022-09-10T12:40:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:02 smithi110 ceph-mon[97633]: pgmap v15703: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:03.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:03.506554+0000) 2022-09-10T12:40:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:04.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:04.043479+0000) 2022-09-10T12:40:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:04.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:04.506766+0000) 2022-09-10T12:40:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:04 smithi104 ceph-mon[97044]: pgmap v15704: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:05.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:05.043728+0000) 2022-09-10T12:40:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:04 smithi110 ceph-mon[97633]: pgmap v15704: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:05.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:05.506959+0000) 2022-09-10T12:40:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:06.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:06.043908+0000) 2022-09-10T12:40:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:06.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:06.507163+0000) 2022-09-10T12:40:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:06 smithi104 ceph-mon[97044]: pgmap v15705: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:40:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:07.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:07.044199+0000) 2022-09-10T12:40:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:06 smithi110 ceph-mon[97633]: pgmap v15705: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:40:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:07.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:07.507302+0000) 2022-09-10T12:40:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:08.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:08.044371+0000) 2022-09-10T12:40:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:08.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:08.507490+0000) 2022-09-10T12:40:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:08 smithi104 ceph-mon[97044]: pgmap v15706: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:40:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:08 smithi104 ceph-mon[97044]: osdmap e1062: 12 total, 12 up, 12 in 2022-09-10T12:40:09.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:09.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:09.044584+0000) 2022-09-10T12:40:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:08 smithi110 ceph-mon[97633]: pgmap v15706: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:40:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:08 smithi110 ceph-mon[97633]: osdmap e1062: 12 total, 12 up, 12 in 2022-09-10T12:40:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:09.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:09.507709+0000) 2022-09-10T12:40:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:10.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:10.044802+0000) 2022-09-10T12:40:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:10.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:10.507931+0000) 2022-09-10T12:40:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:10 smithi104 ceph-mon[97044]: pgmap v15708: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:40:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:11.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:11.045025+0000) 2022-09-10T12:40:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:10 smithi110 ceph-mon[97633]: pgmap v15708: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:40:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:11.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:11.508083+0000) 2022-09-10T12:40:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:12.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:12.045257+0000) 2022-09-10T12:40:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:12.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:12.508236+0000) 2022-09-10T12:40:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:12 smithi104 ceph-mon[97044]: pgmap v15709: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:40:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:13.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:13.045441+0000) 2022-09-10T12:40:13.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:12 smithi110 ceph-mon[97633]: pgmap v15709: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:40:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:13.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:13.508400+0000) 2022-09-10T12:40:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:14.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:14.045685+0000) 2022-09-10T12:40:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:14.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:14.508533+0000) 2022-09-10T12:40:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:14 smithi104 ceph-mon[97044]: pgmap v15710: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:40:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:14 smithi104 ceph-mon[97044]: osdmap e1063: 12 total, 12 up, 12 in 2022-09-10T12:40:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:15.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:15.045951+0000) 2022-09-10T12:40:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:14 smithi110 ceph-mon[97633]: pgmap v15710: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:40:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:14 smithi110 ceph-mon[97633]: osdmap e1063: 12 total, 12 up, 12 in 2022-09-10T12:40:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:15.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:15.508732+0000) 2022-09-10T12:40:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:40:15.381+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:40:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:16.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:16.046159+0000) 2022-09-10T12:40:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:16.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:16.508888+0000) 2022-09-10T12:40:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:16 smithi104 ceph-mon[97044]: pgmap v15712: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:40:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:17.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:17.046376+0000) 2022-09-10T12:40:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:16 smithi110 ceph-mon[97633]: pgmap v15712: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:40:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:17.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:17.509074+0000) 2022-09-10T12:40:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:18.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:18.046605+0000) 2022-09-10T12:40:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:18.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:18.509230+0000) 2022-09-10T12:40:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:18 smithi104 ceph-mon[97044]: pgmap v15713: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:40:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:19.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:19.046850+0000) 2022-09-10T12:40:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:18 smithi110 ceph-mon[97633]: pgmap v15713: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:40:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:19.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:19.509411+0000) 2022-09-10T12:40:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:20.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:20.047127+0000) 2022-09-10T12:40:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:20.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:20.509580+0000) 2022-09-10T12:40:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:20 smithi104 ceph-mon[97044]: pgmap v15714: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:40:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:21.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:21.047378+0000) 2022-09-10T12:40:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:20 smithi110 ceph-mon[97633]: pgmap v15714: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:40:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:21.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:21.509764+0000) 2022-09-10T12:40:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:22.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:22.047595+0000) 2022-09-10T12:40:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:22.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:22.509908+0000) 2022-09-10T12:40:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:22 smithi104 ceph-mon[97044]: pgmap v15715: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:40:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:23.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:23.047830+0000) 2022-09-10T12:40:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:22 smithi110 ceph-mon[97633]: pgmap v15715: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:40:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:23.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:23.510042+0000) 2022-09-10T12:40:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:24.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:24.048122+0000) 2022-09-10T12:40:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:24.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:24.510214+0000) 2022-09-10T12:40:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:24 smithi104 ceph-mon[97044]: pgmap v15716: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:40:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:25.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:25.048296+0000) 2022-09-10T12:40:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:24 smithi110 ceph-mon[97633]: pgmap v15716: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:40:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:25.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:25.510480+0000) 2022-09-10T12:40:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:26.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:26.048480+0000) 2022-09-10T12:40:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:26.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:26.510700+0000) 2022-09-10T12:40:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:26 smithi104 ceph-mon[97044]: pgmap v15717: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:40:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:27.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:27.048649+0000) 2022-09-10T12:40:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:26 smithi110 ceph-mon[97633]: pgmap v15717: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:40:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:27.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:27.510922+0000) 2022-09-10T12:40:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:28.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:28.048834+0000) 2022-09-10T12:40:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:28.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:28.511173+0000) 2022-09-10T12:40:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:28 smithi104 ceph-mon[97044]: pgmap v15718: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:40:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:29.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:29.049027+0000) 2022-09-10T12:40:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:28 smithi110 ceph-mon[97633]: pgmap v15718: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:40:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:29.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:29.511417+0000) 2022-09-10T12:40:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:30.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:30.049260+0000) 2022-09-10T12:40:30.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:40:30.396+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:40:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:30.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:30.511646+0000) 2022-09-10T12:40:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:30 smithi104 ceph-mon[97044]: pgmap v15719: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:40:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:31.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:31.049452+0000) 2022-09-10T12:40:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:30 smithi110 ceph-mon[97633]: pgmap v15719: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:40:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:31.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:31.511835+0000) 2022-09-10T12:40:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:32.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:32.049634+0000) 2022-09-10T12:40:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:32.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:32.512045+0000) 2022-09-10T12:40:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:32 smithi104 ceph-mon[97044]: pgmap v15720: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:33.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:33.049884+0000) 2022-09-10T12:40:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:32 smithi110 ceph-mon[97633]: pgmap v15720: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:33.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:33.512272+0000) 2022-09-10T12:40:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:34.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:34.050148+0000) 2022-09-10T12:40:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:34.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:34.512488+0000) 2022-09-10T12:40:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:34 smithi104 ceph-mon[97044]: pgmap v15721: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:35.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:35.050337+0000) 2022-09-10T12:40:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:34 smithi110 ceph-mon[97633]: pgmap v15721: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:35.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:35.512632+0000) 2022-09-10T12:40:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:36.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:36.050567+0000) 2022-09-10T12:40:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:36.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:36.512808+0000) 2022-09-10T12:40:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:36 smithi104 ceph-mon[97044]: pgmap v15722: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:37.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:37.050801+0000) 2022-09-10T12:40:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:36 smithi110 ceph-mon[97633]: pgmap v15722: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:37.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:37.512971+0000) 2022-09-10T12:40:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:38.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:38.051066+0000) 2022-09-10T12:40:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:38.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:38.513244+0000) 2022-09-10T12:40:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:38 smithi104 ceph-mon[97044]: pgmap v15723: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:39.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:39.051347+0000) 2022-09-10T12:40:39.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:38 smithi110 ceph-mon[97633]: pgmap v15723: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:39.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:39.513407+0000) 2022-09-10T12:40:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:40.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:40.051557+0000) 2022-09-10T12:40:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:40.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:40.513607+0000) 2022-09-10T12:40:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:40 smithi104 ceph-mon[97044]: pgmap v15724: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:41.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:41.051814+0000) 2022-09-10T12:40:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:40 smithi110 ceph-mon[97633]: pgmap v15724: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:41.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:41.513824+0000) 2022-09-10T12:40:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:42.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:42.052023+0000) 2022-09-10T12:40:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:42.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:42.514028+0000) 2022-09-10T12:40:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:42 smithi110 ceph-mon[97633]: pgmap v15725: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:42 smithi104 ceph-mon[97044]: pgmap v15725: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:43.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:43.052247+0000) 2022-09-10T12:40:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:43.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:43.514179+0000) 2022-09-10T12:40:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:44.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:44.052413+0000) 2022-09-10T12:40:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:44.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:44.514324+0000) 2022-09-10T12:40:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:44 smithi104 ceph-mon[97044]: pgmap v15726: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:45.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:45.052605+0000) 2022-09-10T12:40:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:44 smithi110 ceph-mon[97633]: pgmap v15726: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:45.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:45.514521+0000) 2022-09-10T12:40:45.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:40:45.412+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:40:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:46.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:46.052830+0000) 2022-09-10T12:40:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:46.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:46.514686+0000) 2022-09-10T12:40:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:46 smithi104 ceph-mon[97044]: pgmap v15727: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:40:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:47.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:47.053077+0000) 2022-09-10T12:40:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:46 smithi110 ceph-mon[97633]: pgmap v15727: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:40:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:40:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:40:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:40:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:47.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:47.514825+0000) 2022-09-10T12:40:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:48.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:48.053361+0000) 2022-09-10T12:40:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:48.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:48.515014+0000) 2022-09-10T12:40:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:48 smithi104 ceph-mon[97044]: pgmap v15728: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:48 smithi110 ceph-mon[97633]: pgmap v15728: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:49.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:49.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:49.053520+0000) 2022-09-10T12:40:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:49.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:49.515176+0000) 2022-09-10T12:40:50.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:50.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:50.053680+0000) 2022-09-10T12:40:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:50.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:50.515320+0000) 2022-09-10T12:40:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:50 smithi104 ceph-mon[97044]: pgmap v15729: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:50 smithi110 ceph-mon[97633]: pgmap v15729: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:51.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:51.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:51.053837+0000) 2022-09-10T12:40:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:51.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:51.515465+0000) 2022-09-10T12:40:52.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:52.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:52.053970+0000) 2022-09-10T12:40:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:52.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:52.515595+0000) 2022-09-10T12:40:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:52 smithi104 ceph-mon[97044]: pgmap v15730: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:52 smithi110 ceph-mon[97633]: pgmap v15730: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:53.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:53.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:53.054196+0000) 2022-09-10T12:40:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:53.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:53.515795+0000) 2022-09-10T12:40:54.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:54.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:54.054393+0000) 2022-09-10T12:40:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:54.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:54.515987+0000) 2022-09-10T12:40:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:54 smithi104 ceph-mon[97044]: pgmap v15731: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:54 smithi110 ceph-mon[97633]: pgmap v15731: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:55.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:55.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:55.054590+0000) 2022-09-10T12:40:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:55.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:55.516216+0000) 2022-09-10T12:40:56.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:56.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:56.054823+0000) 2022-09-10T12:40:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:56.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:56.516355+0000) 2022-09-10T12:40:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:56 smithi104 ceph-mon[97044]: pgmap v15732: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:56 smithi110 ceph-mon[97633]: pgmap v15732: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:40:57.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:57.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:57.055072+0000) 2022-09-10T12:40:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:57.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:57.516590+0000) 2022-09-10T12:40:58.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:58.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:58.055319+0000) 2022-09-10T12:40:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:58.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:58.516761+0000) 2022-09-10T12:40:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:40:58 smithi104 ceph-mon[97044]: pgmap v15733: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:40:58 smithi110 ceph-mon[97633]: pgmap v15733: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:40:59.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:40:59.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:59.055591+0000) 2022-09-10T12:40:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:40:59.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:40:59.516941+0000) 2022-09-10T12:41:00.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:00.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:00.055843+0000) 2022-09-10T12:41:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:00.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:00.517089+0000) 2022-09-10T12:41:00.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:41:00.428+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:41:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:00 smithi104 ceph-mon[97044]: pgmap v15734: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:00 smithi110 ceph-mon[97633]: pgmap v15734: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:01.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:01.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:01.055991+0000) 2022-09-10T12:41:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:01.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:01.517255+0000) 2022-09-10T12:41:02.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:02.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:02.056214+0000) 2022-09-10T12:41:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:02.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:02.517427+0000) 2022-09-10T12:41:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:02 smithi104 ceph-mon[97044]: pgmap v15735: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:02 smithi110 ceph-mon[97633]: pgmap v15735: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:03.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:03.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:03.056476+0000) 2022-09-10T12:41:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:03.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:03.517558+0000) 2022-09-10T12:41:04.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:04.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:04.056706+0000) 2022-09-10T12:41:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:04.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:04.517691+0000) 2022-09-10T12:41:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:04 smithi104 ceph-mon[97044]: pgmap v15736: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:04 smithi110 ceph-mon[97633]: pgmap v15736: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:05.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:05.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:05.056880+0000) 2022-09-10T12:41:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:05.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:05.517852+0000) 2022-09-10T12:41:06.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:06.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:06.057148+0000) 2022-09-10T12:41:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:06.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:06.518028+0000) 2022-09-10T12:41:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:07 smithi104 ceph-mon[97044]: pgmap v15737: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:41:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:06 smithi110 ceph-mon[97633]: pgmap v15737: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:41:07.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:07.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:07.057369+0000) 2022-09-10T12:41:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:07.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:07.518195+0000) 2022-09-10T12:41:08.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:08.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:08.057582+0000) 2022-09-10T12:41:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:08.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:08.518348+0000) 2022-09-10T12:41:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:09 smithi104 ceph-mon[97044]: pgmap v15738: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:41:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:09 smithi104 ceph-mon[97044]: osdmap e1064: 12 total, 12 up, 12 in 2022-09-10T12:41:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:09 smithi110 ceph-mon[97633]: pgmap v15738: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:41:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:09 smithi110 ceph-mon[97633]: osdmap e1064: 12 total, 12 up, 12 in 2022-09-10T12:41:09.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:09.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:09.057781+0000) 2022-09-10T12:41:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:09.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:09.518548+0000) 2022-09-10T12:41:10.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:10.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:10.058032+0000) 2022-09-10T12:41:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:10.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:10.518733+0000) 2022-09-10T12:41:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:11 smithi104 ceph-mon[97044]: pgmap v15740: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:41:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:11 smithi110 ceph-mon[97633]: pgmap v15740: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:41:11.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:11.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:11.058282+0000) 2022-09-10T12:41:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:11.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:11.518915+0000) 2022-09-10T12:41:12.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:12.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:12.058530+0000) 2022-09-10T12:41:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:12.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:12.519112+0000) 2022-09-10T12:41:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:13 smithi104 ceph-mon[97044]: pgmap v15741: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:41:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:13 smithi110 ceph-mon[97633]: pgmap v15741: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:41:13.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:13.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:13.058755+0000) 2022-09-10T12:41:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:13.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:13.519296+0000) 2022-09-10T12:41:14.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:14.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:14.058966+0000) 2022-09-10T12:41:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:14.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:14.519523+0000) 2022-09-10T12:41:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:15 smithi104 ceph-mon[97044]: pgmap v15742: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:41:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:15 smithi104 ceph-mon[97044]: osdmap e1065: 12 total, 12 up, 12 in 2022-09-10T12:41:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:15 smithi110 ceph-mon[97633]: pgmap v15742: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:41:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:15 smithi110 ceph-mon[97633]: osdmap e1065: 12 total, 12 up, 12 in 2022-09-10T12:41:15.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:15.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:15.059148+0000) 2022-09-10T12:41:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:15.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:15.519777+0000) 2022-09-10T12:41:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:41:15.443+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:41:16.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:16.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:16.059360+0000) 2022-09-10T12:41:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:16.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:16.519998+0000) 2022-09-10T12:41:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:17 smithi104 ceph-mon[97044]: pgmap v15744: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:41:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:17 smithi110 ceph-mon[97633]: pgmap v15744: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:41:17.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:17.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:17.059551+0000) 2022-09-10T12:41:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:17.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:17.520225+0000) 2022-09-10T12:41:18.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:18.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:18.059748+0000) 2022-09-10T12:41:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:18.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:18.520361+0000) 2022-09-10T12:41:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:19 smithi104 ceph-mon[97044]: pgmap v15745: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 11 op/s 2022-09-10T12:41:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:19 smithi110 ceph-mon[97633]: pgmap v15745: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 222 B/s wr, 11 op/s 2022-09-10T12:41:19.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:19.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:19.059972+0000) 2022-09-10T12:41:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:19.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:19.520581+0000) 2022-09-10T12:41:20.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:20.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:20.060173+0000) 2022-09-10T12:41:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:20.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:20.520801+0000) 2022-09-10T12:41:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:21 smithi104 ceph-mon[97044]: pgmap v15746: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:41:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:21 smithi110 ceph-mon[97633]: pgmap v15746: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:41:21.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:21.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:21.060332+0000) 2022-09-10T12:41:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:21.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:21.520941+0000) 2022-09-10T12:41:22.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:22.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:22.060484+0000) 2022-09-10T12:41:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:22.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:22.521198+0000) 2022-09-10T12:41:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:23 smithi104 ceph-mon[97044]: pgmap v15747: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:41:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:23 smithi110 ceph-mon[97633]: pgmap v15747: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:41:23.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:23.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:23.060681+0000) 2022-09-10T12:41:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:23.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:23.521441+0000) 2022-09-10T12:41:24.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:24 smithi110 ceph-mon[97633]: pgmap v15748: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:41:24.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:24.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:24.060930+0000) 2022-09-10T12:41:24.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:24 smithi104 ceph-mon[97044]: pgmap v15748: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:41:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:24.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:24.521629+0000) 2022-09-10T12:41:25.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:25.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:25.061198+0000) 2022-09-10T12:41:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:25.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:25.521801+0000) 2022-09-10T12:41:26.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:26.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:26.061402+0000) 2022-09-10T12:41:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:26.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:26.521978+0000) 2022-09-10T12:41:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:26 smithi104 ceph-mon[97044]: pgmap v15749: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:41:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:26 smithi110 ceph-mon[97633]: pgmap v15749: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:41:27.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:27.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:27.061626+0000) 2022-09-10T12:41:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:27.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:27.522116+0000) 2022-09-10T12:41:28.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:28.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:28.061923+0000) 2022-09-10T12:41:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:28.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:28.522251+0000) 2022-09-10T12:41:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:28 smithi104 ceph-mon[97044]: pgmap v15750: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:41:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:28 smithi110 ceph-mon[97633]: pgmap v15750: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:41:29.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:29.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:29.062109+0000) 2022-09-10T12:41:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:29.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:29.522391+0000) 2022-09-10T12:41:30.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:30.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:30.062380+0000) 2022-09-10T12:41:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:30.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:30.522551+0000) 2022-09-10T12:41:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:41:30.455+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:41:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:30 smithi104 ceph-mon[97044]: pgmap v15751: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:41:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:30 smithi110 ceph-mon[97633]: pgmap v15751: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:41:31.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:31.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:31.062635+0000) 2022-09-10T12:41:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:31.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:31.522682+0000) 2022-09-10T12:41:32.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:32.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:32.062874+0000) 2022-09-10T12:41:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:32.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:32.522837+0000) 2022-09-10T12:41:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:32 smithi104 ceph-mon[97044]: pgmap v15752: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:32 smithi110 ceph-mon[97633]: pgmap v15752: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:33.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:33.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:33.063129+0000) 2022-09-10T12:41:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:33.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:33.522963+0000) 2022-09-10T12:41:34.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:34.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:34.063266+0000) 2022-09-10T12:41:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:34.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:34.523111+0000) 2022-09-10T12:41:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:34 smithi104 ceph-mon[97044]: pgmap v15753: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:34 smithi110 ceph-mon[97633]: pgmap v15753: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:35.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:35.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:35.063471+0000) 2022-09-10T12:41:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:35.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:35.523260+0000) 2022-09-10T12:41:36.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:36.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:36.063690+0000) 2022-09-10T12:41:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:36.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:36.523420+0000) 2022-09-10T12:41:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:36 smithi104 ceph-mon[97044]: pgmap v15754: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:36 smithi110 ceph-mon[97633]: pgmap v15754: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:37.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:37.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:37.063929+0000) 2022-09-10T12:41:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:37.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:37.523595+0000) 2022-09-10T12:41:38.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:38.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:38.064202+0000) 2022-09-10T12:41:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:38.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:38.523787+0000) 2022-09-10T12:41:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:38 smithi104 ceph-mon[97044]: pgmap v15755: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:38 smithi110 ceph-mon[97633]: pgmap v15755: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:39.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:39.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:39.064453+0000) 2022-09-10T12:41:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:39.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:39.523943+0000) 2022-09-10T12:41:40.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:40.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:40.064697+0000) 2022-09-10T12:41:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:40.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:40.524113+0000) 2022-09-10T12:41:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:40 smithi104 ceph-mon[97044]: pgmap v15756: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:40 smithi110 ceph-mon[97633]: pgmap v15756: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:41.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:41.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:41.064833+0000) 2022-09-10T12:41:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:41.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:41.524323+0000) 2022-09-10T12:41:42.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:42.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:42.064984+0000) 2022-09-10T12:41:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:42.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:42.524560+0000) 2022-09-10T12:41:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:42 smithi104 ceph-mon[97044]: pgmap v15757: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:42 smithi110 ceph-mon[97633]: pgmap v15757: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:43.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:43.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:43.065155+0000) 2022-09-10T12:41:43.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:43.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:43.524775+0000) 2022-09-10T12:41:44.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:44.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:44.065386+0000) 2022-09-10T12:41:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:44.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:44.524927+0000) 2022-09-10T12:41:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:44 smithi104 ceph-mon[97044]: pgmap v15758: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:44 smithi110 ceph-mon[97633]: pgmap v15758: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:45.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:45.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:45.065664+0000) 2022-09-10T12:41:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:45.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:45.525080+0000) 2022-09-10T12:41:45.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:41:45.458+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:41:46.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:46.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:46.065854+0000) 2022-09-10T12:41:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:46.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:46.525246+0000) 2022-09-10T12:41:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:46 smithi104 ceph-mon[97044]: pgmap v15759: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:41:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:46 smithi110 ceph-mon[97633]: pgmap v15759: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:41:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:41:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:41:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:41:47.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:47.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:47.066067+0000) 2022-09-10T12:41:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:47.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:47.525408+0000) 2022-09-10T12:41:48.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:48.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:48.066361+0000) 2022-09-10T12:41:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:48.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:48.525622+0000) 2022-09-10T12:41:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:48 smithi104 ceph-mon[97044]: pgmap v15760: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:48 smithi110 ceph-mon[97633]: pgmap v15760: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:49.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:49.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:49.066529+0000) 2022-09-10T12:41:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:49.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:49.525799+0000) 2022-09-10T12:41:50.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:50.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:50.066789+0000) 2022-09-10T12:41:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:50.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:50.525978+0000) 2022-09-10T12:41:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:50 smithi104 ceph-mon[97044]: pgmap v15761: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:50 smithi110 ceph-mon[97633]: pgmap v15761: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:51.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:51.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:51.066965+0000) 2022-09-10T12:41:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:51.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:51.526162+0000) 2022-09-10T12:41:52.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:52.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:52.067197+0000) 2022-09-10T12:41:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:52.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:52.526370+0000) 2022-09-10T12:41:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:52 smithi104 ceph-mon[97044]: pgmap v15762: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:52 smithi110 ceph-mon[97633]: pgmap v15762: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:53.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:53.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:53.067423+0000) 2022-09-10T12:41:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:53.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:53.526610+0000) 2022-09-10T12:41:54.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:54.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:54.067670+0000) 2022-09-10T12:41:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:54.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:54.526788+0000) 2022-09-10T12:41:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:54 smithi104 ceph-mon[97044]: pgmap v15763: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:54 smithi110 ceph-mon[97633]: pgmap v15763: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:55.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:55.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:55.067924+0000) 2022-09-10T12:41:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:55.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:55.527005+0000) 2022-09-10T12:41:56.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:56.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:56.068132+0000) 2022-09-10T12:41:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:56.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:56.527217+0000) 2022-09-10T12:41:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:56 smithi104 ceph-mon[97044]: pgmap v15764: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:56 smithi110 ceph-mon[97633]: pgmap v15764: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:41:57.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:57.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:57.068309+0000) 2022-09-10T12:41:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:57.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:57.527431+0000) 2022-09-10T12:41:58.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:58.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:58.068531+0000) 2022-09-10T12:41:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:58.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:58.527714+0000) 2022-09-10T12:41:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:41:58 smithi104 ceph-mon[97044]: pgmap v15765: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:41:58 smithi110 ceph-mon[97633]: pgmap v15765: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:41:59.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:41:59.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:59.068777+0000) 2022-09-10T12:41:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:41:59.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:41:59.527939+0000) 2022-09-10T12:42:00.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:00.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:00.069004+0000) 2022-09-10T12:42:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:00.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:00.528115+0000) 2022-09-10T12:42:00.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:42:00.459+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:42:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:00 smithi104 ceph-mon[97044]: pgmap v15766: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:00 smithi110 ceph-mon[97633]: pgmap v15766: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:01.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:01.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:01.069150+0000) 2022-09-10T12:42:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:01.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:01.528238+0000) 2022-09-10T12:42:02.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:02.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:02.069387+0000) 2022-09-10T12:42:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:02.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:02.528423+0000) 2022-09-10T12:42:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:02 smithi104 ceph-mon[97044]: pgmap v15767: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:02 smithi110 ceph-mon[97633]: pgmap v15767: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:03.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:03.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:03.069607+0000) 2022-09-10T12:42:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:03.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:03.528601+0000) 2022-09-10T12:42:04.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:04.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:04.069813+0000) 2022-09-10T12:42:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:04.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:04.528768+0000) 2022-09-10T12:42:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:04 smithi104 ceph-mon[97044]: pgmap v15768: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:04 smithi110 ceph-mon[97633]: pgmap v15768: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:05.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:05.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:05.070039+0000) 2022-09-10T12:42:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:05.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:05.528972+0000) 2022-09-10T12:42:06.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:06.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:06.070282+0000) 2022-09-10T12:42:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:06.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:06.529166+0000) 2022-09-10T12:42:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:06 smithi104 ceph-mon[97044]: pgmap v15769: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:42:07.431 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:06 smithi110 ceph-mon[97633]: pgmap v15769: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:42:07.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:07.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:07.070555+0000) 2022-09-10T12:42:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:07.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:07.529405+0000) 2022-09-10T12:42:08.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:08.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:08.070783+0000) 2022-09-10T12:42:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:08.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:08.529590+0000) 2022-09-10T12:42:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:08 smithi104 ceph-mon[97044]: pgmap v15770: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:42:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:08 smithi104 ceph-mon[97044]: osdmap e1066: 12 total, 12 up, 12 in 2022-09-10T12:42:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:08 smithi110 ceph-mon[97633]: pgmap v15770: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:42:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:08 smithi110 ceph-mon[97633]: osdmap e1066: 12 total, 12 up, 12 in 2022-09-10T12:42:09.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:09.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:09.070982+0000) 2022-09-10T12:42:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:09.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:09.529778+0000) 2022-09-10T12:42:10.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:10.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:10.071231+0000) 2022-09-10T12:42:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:10.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:10.529981+0000) 2022-09-10T12:42:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:10 smithi104 ceph-mon[97044]: pgmap v15772: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:42:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:10 smithi110 ceph-mon[97633]: pgmap v15772: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:42:11.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:11.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:11.071418+0000) 2022-09-10T12:42:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:11.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:11.530105+0000) 2022-09-10T12:42:12.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:12.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:12.071624+0000) 2022-09-10T12:42:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:12.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:12.530230+0000) 2022-09-10T12:42:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:12 smithi104 ceph-mon[97044]: pgmap v15773: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:42:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:12 smithi110 ceph-mon[97633]: pgmap v15773: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:42:13.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:13.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:13.071802+0000) 2022-09-10T12:42:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:13.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:13.530454+0000) 2022-09-10T12:42:14.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:14.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:14.072033+0000) 2022-09-10T12:42:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:14.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:14.530651+0000) 2022-09-10T12:42:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:14 smithi104 ceph-mon[97044]: pgmap v15774: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:42:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:14 smithi104 ceph-mon[97044]: osdmap e1067: 12 total, 12 up, 12 in 2022-09-10T12:42:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:14 smithi110 ceph-mon[97633]: pgmap v15774: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:42:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:14 smithi110 ceph-mon[97633]: osdmap e1067: 12 total, 12 up, 12 in 2022-09-10T12:42:15.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:15.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:15.072272+0000) 2022-09-10T12:42:15.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:42:15.470+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:42:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:15.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:15.530881+0000) 2022-09-10T12:42:16.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:16.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:16.072428+0000) 2022-09-10T12:42:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:16.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:16.531085+0000) 2022-09-10T12:42:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:16 smithi104 ceph-mon[97044]: pgmap v15776: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:42:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:16 smithi110 ceph-mon[97633]: pgmap v15776: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:42:17.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:17.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:17.072591+0000) 2022-09-10T12:42:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:17.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:17.531253+0000) 2022-09-10T12:42:18.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:18.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:18.072770+0000) 2022-09-10T12:42:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:18.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:18.531399+0000) 2022-09-10T12:42:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:18 smithi104 ceph-mon[97044]: pgmap v15777: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:42:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:18 smithi110 ceph-mon[97633]: pgmap v15777: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:42:19.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:19.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:19.073045+0000) 2022-09-10T12:42:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:19.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:19.531603+0000) 2022-09-10T12:42:20.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:20.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:20.073333+0000) 2022-09-10T12:42:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:20.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:20.531822+0000) 2022-09-10T12:42:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:20 smithi104 ceph-mon[97044]: pgmap v15778: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:42:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:20 smithi110 ceph-mon[97633]: pgmap v15778: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:42:21.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:21.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:21.073598+0000) 2022-09-10T12:42:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:21.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:21.532006+0000) 2022-09-10T12:42:22.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:22.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:22.073845+0000) 2022-09-10T12:42:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:22.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:22.532133+0000) 2022-09-10T12:42:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:22 smithi104 ceph-mon[97044]: pgmap v15779: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:42:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:22 smithi110 ceph-mon[97633]: pgmap v15779: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:42:23.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:23.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:23.074144+0000) 2022-09-10T12:42:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:23.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:23.532375+0000) 2022-09-10T12:42:24.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:24.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:24.074302+0000) 2022-09-10T12:42:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:24.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:24.532599+0000) 2022-09-10T12:42:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:24 smithi104 ceph-mon[97044]: pgmap v15780: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:42:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:24 smithi110 ceph-mon[97633]: pgmap v15780: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:42:25.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:25.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:25.074507+0000) 2022-09-10T12:42:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:25.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:25.532761+0000) 2022-09-10T12:42:26.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:26.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:26.074710+0000) 2022-09-10T12:42:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:26.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:26.532904+0000) 2022-09-10T12:42:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:27 smithi104 ceph-mon[97044]: pgmap v15781: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:42:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:27 smithi110 ceph-mon[97633]: pgmap v15781: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T12:42:27.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:27.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:27.074954+0000) 2022-09-10T12:42:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:27.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:27.533080+0000) 2022-09-10T12:42:28.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:28.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:28.075163+0000) 2022-09-10T12:42:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:28.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:28.533320+0000) 2022-09-10T12:42:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:29 smithi104 ceph-mon[97044]: pgmap v15782: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:42:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:29 smithi110 ceph-mon[97633]: pgmap v15782: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:42:29.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:29.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:29.075406+0000) 2022-09-10T12:42:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:29.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:29.533506+0000) 2022-09-10T12:42:30.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:30.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:30.075623+0000) 2022-09-10T12:42:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:30.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:30.533721+0000) 2022-09-10T12:42:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:42:30.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:42:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:31 smithi104 ceph-mon[97044]: pgmap v15783: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:42:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:31 smithi110 ceph-mon[97633]: pgmap v15783: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:42:31.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:31.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:31.075820+0000) 2022-09-10T12:42:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:31.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:31.533956+0000) 2022-09-10T12:42:32.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:32.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:32.076124+0000) 2022-09-10T12:42:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:32.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:32.534131+0000) 2022-09-10T12:42:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:33 smithi104 ceph-mon[97044]: pgmap v15784: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:33 smithi110 ceph-mon[97633]: pgmap v15784: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:33.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:33.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:33.076412+0000) 2022-09-10T12:42:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:33.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:33.534308+0000) 2022-09-10T12:42:34.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:34.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:34.076634+0000) 2022-09-10T12:42:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:34.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:34.534481+0000) 2022-09-10T12:42:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:35 smithi104 ceph-mon[97044]: pgmap v15785: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:42:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:35 smithi110 ceph-mon[97633]: pgmap v15785: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:42:35.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:35.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:35.076835+0000) 2022-09-10T12:42:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:35.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:35.534648+0000) 2022-09-10T12:42:36.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:36.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:36.077065+0000) 2022-09-10T12:42:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:36.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:36.534800+0000) 2022-09-10T12:42:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:37 smithi104 ceph-mon[97044]: pgmap v15786: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:37 smithi110 ceph-mon[97633]: pgmap v15786: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:37.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:37.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:37.077281+0000) 2022-09-10T12:42:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:37.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:37.535005+0000) 2022-09-10T12:42:38.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:38.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:38.077513+0000) 2022-09-10T12:42:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:38.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:38.535184+0000) 2022-09-10T12:42:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:39 smithi104 ceph-mon[97044]: pgmap v15787: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:39 smithi110 ceph-mon[97633]: pgmap v15787: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:39.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:39.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:39.077724+0000) 2022-09-10T12:42:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:39.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:39.535391+0000) 2022-09-10T12:42:40.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:40.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:40.077894+0000) 2022-09-10T12:42:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:40.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:40.535535+0000) 2022-09-10T12:42:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:41 smithi104 ceph-mon[97044]: pgmap v15788: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:41 smithi110 ceph-mon[97633]: pgmap v15788: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:41.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:41.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:41.078090+0000) 2022-09-10T12:42:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:41.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:41.535682+0000) 2022-09-10T12:42:42.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:42.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:42.078298+0000) 2022-09-10T12:42:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:42.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:42.535852+0000) 2022-09-10T12:42:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:43 smithi104 ceph-mon[97044]: pgmap v15789: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:43 smithi110 ceph-mon[97633]: pgmap v15789: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:43.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:43.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:43.078534+0000) 2022-09-10T12:42:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:43.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:43.536043+0000) 2022-09-10T12:42:44.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:44.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:44.078774+0000) 2022-09-10T12:42:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:44.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:44.536164+0000) 2022-09-10T12:42:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:45 smithi104 ceph-mon[97044]: pgmap v15790: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:45 smithi110 ceph-mon[97633]: pgmap v15790: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:45.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:45.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:45.079043+0000) 2022-09-10T12:42:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:45.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:45.536359+0000) 2022-09-10T12:42:45.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:42:45.493+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:42:46.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:42:46.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:42:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:42:46.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:42:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:42:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:42:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:42:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:42:46.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:46.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:46.079329+0000) 2022-09-10T12:42:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:46.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:46.536510+0000) 2022-09-10T12:42:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:47 smithi104 ceph-mon[97044]: pgmap v15791: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:47 smithi110 ceph-mon[97633]: pgmap v15791: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:47.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:47.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:47.079556+0000) 2022-09-10T12:42:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:47.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:47.536632+0000) 2022-09-10T12:42:48.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:48.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:48.079779+0000) 2022-09-10T12:42:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:48.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:48.536781+0000) 2022-09-10T12:42:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:49 smithi104 ceph-mon[97044]: pgmap v15792: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:49 smithi110 ceph-mon[97633]: pgmap v15792: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:49.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:49.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:49.080038+0000) 2022-09-10T12:42:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:49.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:49.536992+0000) 2022-09-10T12:42:50.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:50.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:50.080241+0000) 2022-09-10T12:42:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:50.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:50.537226+0000) 2022-09-10T12:42:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:51 smithi104 ceph-mon[97044]: pgmap v15793: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:51 smithi110 ceph-mon[97633]: pgmap v15793: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:51.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:51.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:51.080430+0000) 2022-09-10T12:42:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:51.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:51.537467+0000) 2022-09-10T12:42:52.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:52 smithi104 ceph-mon[97044]: pgmap v15794: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:52.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:52 smithi110 ceph-mon[97633]: pgmap v15794: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:52.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:52.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:52.080614+0000) 2022-09-10T12:42:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:52.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:52.537678+0000) 2022-09-10T12:42:53.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:53.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:53.080891+0000) 2022-09-10T12:42:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:53.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:53.537869+0000) 2022-09-10T12:42:54.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:54.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:54.081181+0000) 2022-09-10T12:42:54.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:54 smithi110 ceph-mon[97633]: pgmap v15795: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:54.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:54 smithi104 ceph-mon[97044]: pgmap v15795: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:54.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:54.538018+0000) 2022-09-10T12:42:55.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:55.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:55.081454+0000) 2022-09-10T12:42:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:55.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:55.538247+0000) 2022-09-10T12:42:56.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:56.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:56.081721+0000) 2022-09-10T12:42:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:56.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:56.538423+0000) 2022-09-10T12:42:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:56 smithi104 ceph-mon[97044]: pgmap v15796: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:56 smithi110 ceph-mon[97633]: pgmap v15796: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:42:57.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:57.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:57.081956+0000) 2022-09-10T12:42:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:57.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:57.538573+0000) 2022-09-10T12:42:58.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:58.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:58.082155+0000) 2022-09-10T12:42:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:58.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:58.538749+0000) 2022-09-10T12:42:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:42:58 smithi104 ceph-mon[97044]: pgmap v15797: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:42:58 smithi110 ceph-mon[97633]: pgmap v15797: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:42:59.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:42:59.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:59.082441+0000) 2022-09-10T12:42:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:42:59.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:42:59.538929+0000) 2022-09-10T12:43:00.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:00.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:00.082689+0000) 2022-09-10T12:43:00.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:43:00.508+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:43:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:00.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:00.539054+0000) 2022-09-10T12:43:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:00 smithi104 ceph-mon[97044]: pgmap v15798: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:00 smithi110 ceph-mon[97633]: pgmap v15798: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:01.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:01.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:01.082903+0000) 2022-09-10T12:43:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:01.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:01.539224+0000) 2022-09-10T12:43:02.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:02.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:02.083135+0000) 2022-09-10T12:43:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:02.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:02.539444+0000) 2022-09-10T12:43:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:02 smithi104 ceph-mon[97044]: pgmap v15799: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:02 smithi110 ceph-mon[97633]: pgmap v15799: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:03.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:03.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:03.083351+0000) 2022-09-10T12:43:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:03.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:03.539614+0000) 2022-09-10T12:43:04.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:04.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:04.083576+0000) 2022-09-10T12:43:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:04.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:04.539795+0000) 2022-09-10T12:43:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:04 smithi104 ceph-mon[97044]: pgmap v15800: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:04 smithi110 ceph-mon[97633]: pgmap v15800: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:05.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:05.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:05.083786+0000) 2022-09-10T12:43:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:05.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:05.540014+0000) 2022-09-10T12:43:06.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:06.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:06.083971+0000) 2022-09-10T12:43:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:06.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:06.540238+0000) 2022-09-10T12:43:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:06 smithi104 ceph-mon[97044]: pgmap v15801: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:43:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:06 smithi110 ceph-mon[97633]: pgmap v15801: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:43:07.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:07.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:07.084181+0000) 2022-09-10T12:43:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:07.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:07.540409+0000) 2022-09-10T12:43:08.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:08.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:08.084355+0000) 2022-09-10T12:43:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:08.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:08.540659+0000) 2022-09-10T12:43:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:08 smithi104 ceph-mon[97044]: pgmap v15802: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:43:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:08 smithi104 ceph-mon[97044]: osdmap e1068: 12 total, 12 up, 12 in 2022-09-10T12:43:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:08 smithi110 ceph-mon[97633]: pgmap v15802: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:08 smithi110 ceph-mon[97633]: osdmap e1068: 12 total, 12 up, 12 in 2022-09-10T12:43:09.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:09.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:09.084589+0000) 2022-09-10T12:43:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:09.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:09.540856+0000) 2022-09-10T12:43:10.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:10.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:10.084790+0000) 2022-09-10T12:43:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:10.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:10.541038+0000) 2022-09-10T12:43:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:10 smithi104 ceph-mon[97044]: pgmap v15804: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:43:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:10 smithi110 ceph-mon[97633]: pgmap v15804: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:43:11.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:11.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:11.085054+0000) 2022-09-10T12:43:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:11.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:11.541249+0000) 2022-09-10T12:43:12.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:12.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:12.085323+0000) 2022-09-10T12:43:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:12.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:12.541449+0000) 2022-09-10T12:43:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:12 smithi104 ceph-mon[97044]: pgmap v15805: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:43:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:12 smithi110 ceph-mon[97633]: pgmap v15805: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:43:13.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:13.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:13.085511+0000) 2022-09-10T12:43:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:13.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:13.541678+0000) 2022-09-10T12:43:14.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:14.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:14.085715+0000) 2022-09-10T12:43:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:14.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:14.541806+0000) 2022-09-10T12:43:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:14 smithi104 ceph-mon[97044]: pgmap v15806: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:43:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:14 smithi104 ceph-mon[97044]: osdmap e1069: 12 total, 12 up, 12 in 2022-09-10T12:43:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:14 smithi110 ceph-mon[97633]: pgmap v15806: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:43:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:14 smithi110 ceph-mon[97633]: osdmap e1069: 12 total, 12 up, 12 in 2022-09-10T12:43:15.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:15.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:15.085882+0000) 2022-09-10T12:43:15.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:43:15.519+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:43:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:15.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:15.541943+0000) 2022-09-10T12:43:16.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:16.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:16.086048+0000) 2022-09-10T12:43:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:16.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:16.542148+0000) 2022-09-10T12:43:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:16 smithi104 ceph-mon[97044]: pgmap v15808: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:43:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:16 smithi110 ceph-mon[97633]: pgmap v15808: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:43:17.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:17.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:17.086248+0000) 2022-09-10T12:43:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:17.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:17.542378+0000) 2022-09-10T12:43:18.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:18.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:18.086379+0000) 2022-09-10T12:43:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:18.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:18.542554+0000) 2022-09-10T12:43:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:18 smithi104 ceph-mon[97044]: pgmap v15809: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:43:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:18 smithi110 ceph-mon[97633]: pgmap v15809: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:43:19.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:19.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:19.086574+0000) 2022-09-10T12:43:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:19.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:19.542793+0000) 2022-09-10T12:43:20.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:20.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:20.086829+0000) 2022-09-10T12:43:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:20.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:20.542961+0000) 2022-09-10T12:43:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:20 smithi104 ceph-mon[97044]: pgmap v15810: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:43:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:20 smithi110 ceph-mon[97633]: pgmap v15810: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T12:43:21.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:21.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:21.087024+0000) 2022-09-10T12:43:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:21.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:21.543198+0000) 2022-09-10T12:43:22.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:22.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:22.087261+0000) 2022-09-10T12:43:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:22.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:22.543390+0000) 2022-09-10T12:43:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:22 smithi104 ceph-mon[97044]: pgmap v15811: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:43:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:22 smithi110 ceph-mon[97633]: pgmap v15811: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:43:23.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:23.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:23.087484+0000) 2022-09-10T12:43:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:23.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:23.543599+0000) 2022-09-10T12:43:24.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:24.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:24.087704+0000) 2022-09-10T12:43:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:24.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:24.543772+0000) 2022-09-10T12:43:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:24 smithi104 ceph-mon[97044]: pgmap v15812: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:43:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:24 smithi110 ceph-mon[97633]: pgmap v15812: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:43:25.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:25.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:25.087931+0000) 2022-09-10T12:43:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:25.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:25.543950+0000) 2022-09-10T12:43:26.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:26.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:26.088200+0000) 2022-09-10T12:43:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:26.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:26.544142+0000) 2022-09-10T12:43:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:26 smithi104 ceph-mon[97044]: pgmap v15813: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:43:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:26 smithi110 ceph-mon[97633]: pgmap v15813: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:43:27.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:27.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:27.088402+0000) 2022-09-10T12:43:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:27.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:27.544272+0000) 2022-09-10T12:43:28.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:28.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:28.088677+0000) 2022-09-10T12:43:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:28.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:28.544456+0000) 2022-09-10T12:43:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:28 smithi104 ceph-mon[97044]: pgmap v15814: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:28 smithi110 ceph-mon[97633]: pgmap v15814: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:29.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:29.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:29.088923+0000) 2022-09-10T12:43:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:29.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:29.544617+0000) 2022-09-10T12:43:30.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:30.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:30.089179+0000) 2022-09-10T12:43:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:30.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:30.544746+0000) 2022-09-10T12:43:30.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:43:30.533+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:43:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:30 smithi104 ceph-mon[97044]: pgmap v15815: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:30 smithi110 ceph-mon[97633]: pgmap v15815: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:31.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:31.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:31.089408+0000) 2022-09-10T12:43:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:31.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:31.544910+0000) 2022-09-10T12:43:32.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:32.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:32.089659+0000) 2022-09-10T12:43:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:32.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:32.545151+0000) 2022-09-10T12:43:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:32 smithi104 ceph-mon[97044]: pgmap v15816: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:32 smithi110 ceph-mon[97633]: pgmap v15816: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:33.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:33.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:33.089838+0000) 2022-09-10T12:43:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:33.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:33.545334+0000) 2022-09-10T12:43:34.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:34.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:34.090035+0000) 2022-09-10T12:43:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:34.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:34.545510+0000) 2022-09-10T12:43:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:34 smithi104 ceph-mon[97044]: pgmap v15817: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:34 smithi110 ceph-mon[97633]: pgmap v15817: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:35.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:35.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:35.090238+0000) 2022-09-10T12:43:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:35.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:35.545762+0000) 2022-09-10T12:43:36.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:36.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:36.090400+0000) 2022-09-10T12:43:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:36.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:36.545907+0000) 2022-09-10T12:43:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:36 smithi104 ceph-mon[97044]: pgmap v15818: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:36 smithi110 ceph-mon[97633]: pgmap v15818: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:37.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:37.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:37.090633+0000) 2022-09-10T12:43:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:37.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:37.546085+0000) 2022-09-10T12:43:38.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:38.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:38.090836+0000) 2022-09-10T12:43:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:38.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:38.546238+0000) 2022-09-10T12:43:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:38 smithi104 ceph-mon[97044]: pgmap v15819: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:38 smithi110 ceph-mon[97633]: pgmap v15819: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:39.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:39.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:39.091058+0000) 2022-09-10T12:43:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:39.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:39.546421+0000) 2022-09-10T12:43:40.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:40.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:40.091320+0000) 2022-09-10T12:43:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:40.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:40.546625+0000) 2022-09-10T12:43:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:41 smithi104 ceph-mon[97044]: pgmap v15820: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:40 smithi110 ceph-mon[97633]: pgmap v15820: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:41.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:41.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:41.091531+0000) 2022-09-10T12:43:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:41.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:41.546762+0000) 2022-09-10T12:43:42.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:42.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:42.091725+0000) 2022-09-10T12:43:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:42.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:42.546981+0000) 2022-09-10T12:43:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:43 smithi104 ceph-mon[97044]: pgmap v15821: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:43 smithi110 ceph-mon[97633]: pgmap v15821: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:43.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:43.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:43.091949+0000) 2022-09-10T12:43:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:43.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:43.547172+0000) 2022-09-10T12:43:44.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:44.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:44.092201+0000) 2022-09-10T12:43:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:44.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:44.547310+0000) 2022-09-10T12:43:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:45 smithi104 ceph-mon[97044]: pgmap v15822: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:45.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:45 smithi110 ceph-mon[97633]: pgmap v15822: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:45.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:45.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:45.092463+0000) 2022-09-10T12:43:45.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:43:45.547+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:43:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:45.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:45.547482+0000) 2022-09-10T12:43:46.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:46.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:46.092644+0000) 2022-09-10T12:43:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:46.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:46.547649+0000) 2022-09-10T12:43:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:47 smithi104 ceph-mon[97044]: pgmap v15823: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:43:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:47 smithi110 ceph-mon[97633]: pgmap v15823: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:43:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:43:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:43:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:43:47.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:47.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:47.092801+0000) 2022-09-10T12:43:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:47.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:47.547822+0000) 2022-09-10T12:43:48.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:48.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:48.093006+0000) 2022-09-10T12:43:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:48.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:48.547992+0000) 2022-09-10T12:43:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:49 smithi104 ceph-mon[97044]: pgmap v15824: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:49 smithi110 ceph-mon[97633]: pgmap v15824: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:49.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:49.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:49.093264+0000) 2022-09-10T12:43:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:49.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:49.548157+0000) 2022-09-10T12:43:50.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:50.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:50.093451+0000) 2022-09-10T12:43:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:50.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:50.548354+0000) 2022-09-10T12:43:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:51 smithi104 ceph-mon[97044]: pgmap v15825: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T12:43:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:51 smithi110 ceph-mon[97633]: pgmap v15825: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T12:43:51.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:51.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:51.093668+0000) 2022-09-10T12:43:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:51.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:51.548575+0000) 2022-09-10T12:43:52.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:52.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:52.093858+0000) 2022-09-10T12:43:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:52.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:52.548759+0000) 2022-09-10T12:43:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:53 smithi104 ceph-mon[97044]: pgmap v15826: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:53 smithi110 ceph-mon[97633]: pgmap v15826: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:53.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:53.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:53.094087+0000) 2022-09-10T12:43:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:53.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:53.548961+0000) 2022-09-10T12:43:54.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:54.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:54.094347+0000) 2022-09-10T12:43:54.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:54.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:54.549187+0000) 2022-09-10T12:43:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:55 smithi104 ceph-mon[97044]: pgmap v15827: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:55 smithi110 ceph-mon[97633]: pgmap v15827: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:55.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:55.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:55.094603+0000) 2022-09-10T12:43:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:55.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:55.549347+0000) 2022-09-10T12:43:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:56.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:56.094825+0000) 2022-09-10T12:43:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:56.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:56.549521+0000) 2022-09-10T12:43:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:57 smithi104 ceph-mon[97044]: pgmap v15828: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:57 smithi110 ceph-mon[97633]: pgmap v15828: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:43:57.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:57.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:57.095068+0000) 2022-09-10T12:43:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:57.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:57.549647+0000) 2022-09-10T12:43:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:58.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:58.095264+0000) 2022-09-10T12:43:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:58.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:58.549815+0000) 2022-09-10T12:43:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:43:59 smithi104 ceph-mon[97044]: pgmap v15829: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:43:59 smithi110 ceph-mon[97633]: pgmap v15829: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:43:59.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:43:59.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:59.095441+0000) 2022-09-10T12:43:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:43:59.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:43:59.549962+0000) 2022-09-10T12:44:00.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:00.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:00.095620+0000) 2022-09-10T12:44:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:00.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:00.550079+0000) 2022-09-10T12:44:00.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:44:00.548+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:44:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:01 smithi104 ceph-mon[97044]: pgmap v15830: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:01 smithi110 ceph-mon[97633]: pgmap v15830: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:01.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:01.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:01.095769+0000) 2022-09-10T12:44:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:01.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:01.550240+0000) 2022-09-10T12:44:02.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:02.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:02.096000+0000) 2022-09-10T12:44:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:02.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:02.550404+0000) 2022-09-10T12:44:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:03 smithi104 ceph-mon[97044]: pgmap v15831: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T12:44:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:03 smithi110 ceph-mon[97633]: pgmap v15831: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T12:44:03.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:03.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:03.096244+0000) 2022-09-10T12:44:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:03.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:03.550555+0000) 2022-09-10T12:44:04.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:04.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:04.096471+0000) 2022-09-10T12:44:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:04.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:04.550743+0000) 2022-09-10T12:44:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:05 smithi104 ceph-mon[97044]: pgmap v15832: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:05.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:05 smithi110 ceph-mon[97633]: pgmap v15832: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:05.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:05.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:05.096697+0000) 2022-09-10T12:44:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:05.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:05.550947+0000) 2022-09-10T12:44:06.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:06.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:06.096898+0000) 2022-09-10T12:44:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:06.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:06.551120+0000) 2022-09-10T12:44:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:07 smithi104 ceph-mon[97044]: pgmap v15833: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:44:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:07 smithi110 ceph-mon[97633]: pgmap v15833: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:44:07.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:07.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:07.097085+0000) 2022-09-10T12:44:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:07.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:07.551312+0000) 2022-09-10T12:44:08.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:08.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:08.097374+0000) 2022-09-10T12:44:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:08.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:08.551490+0000) 2022-09-10T12:44:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:09 smithi110 ceph-mon[97633]: pgmap v15834: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:09 smithi110 ceph-mon[97633]: osdmap e1070: 12 total, 12 up, 12 in 2022-09-10T12:44:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:09 smithi104 ceph-mon[97044]: pgmap v15834: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:44:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:09 smithi104 ceph-mon[97044]: osdmap e1070: 12 total, 12 up, 12 in 2022-09-10T12:44:09.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:09.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:09.097582+0000) 2022-09-10T12:44:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:09.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:09.551639+0000) 2022-09-10T12:44:10.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:10.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:10.097771+0000) 2022-09-10T12:44:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:10.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:10.551874+0000) 2022-09-10T12:44:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:11 smithi110 ceph-mon[97633]: pgmap v15836: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:44:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:11 smithi104 ceph-mon[97044]: pgmap v15836: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:44:11.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:11.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:11.097934+0000) 2022-09-10T12:44:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:11.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:11.552066+0000) 2022-09-10T12:44:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:12 smithi110 ceph-mon[97633]: pgmap v15837: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:44:12.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:12 smithi104 ceph-mon[97044]: pgmap v15837: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:44:12.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:12.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:12.098179+0000) 2022-09-10T12:44:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:12.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:12.552280+0000) 2022-09-10T12:44:13.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:13.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:13.098393+0000) 2022-09-10T12:44:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:13.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:13.552482+0000) 2022-09-10T12:44:14.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:14.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:14.098635+0000) 2022-09-10T12:44:14.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:14 smithi110 ceph-mon[97633]: pgmap v15838: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:44:14.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:14 smithi104 ceph-mon[97044]: pgmap v15838: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:44:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:14.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:14.552690+0000) 2022-09-10T12:44:15.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:15.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:15.098832+0000) 2022-09-10T12:44:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:15.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:15.552868+0000) 2022-09-10T12:44:15.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:44:15.563+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:44:16.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:15 smithi104 ceph-mon[97044]: osdmap e1071: 12 total, 12 up, 12 in 2022-09-10T12:44:16.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:15 smithi110 ceph-mon[97633]: osdmap e1071: 12 total, 12 up, 12 in 2022-09-10T12:44:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:16.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:16.099028+0000) 2022-09-10T12:44:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:16.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:16.553038+0000) 2022-09-10T12:44:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:16 smithi104 ceph-mon[97044]: pgmap v15840: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:44:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:16 smithi110 ceph-mon[97633]: pgmap v15840: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:44:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:17.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:17.099222+0000) 2022-09-10T12:44:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:17.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:17.553250+0000) 2022-09-10T12:44:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:18.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:18.099461+0000) 2022-09-10T12:44:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:18.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:18.553427+0000) 2022-09-10T12:44:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:18 smithi104 ceph-mon[97044]: pgmap v15841: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:44:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:18 smithi110 ceph-mon[97633]: pgmap v15841: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2022-09-10T12:44:19.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:19.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:19.099675+0000) 2022-09-10T12:44:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:19.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:19.553642+0000) 2022-09-10T12:44:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:20.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:20.099863+0000) 2022-09-10T12:44:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:20.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:20.553916+0000) 2022-09-10T12:44:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:20 smithi104 ceph-mon[97044]: pgmap v15842: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:44:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:20 smithi110 ceph-mon[97633]: pgmap v15842: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:44:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:21.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:21.100031+0000) 2022-09-10T12:44:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:21.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:21.554075+0000) 2022-09-10T12:44:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:22.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:22.100243+0000) 2022-09-10T12:44:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:22.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:22.554217+0000) 2022-09-10T12:44:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:22 smithi104 ceph-mon[97044]: pgmap v15843: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:44:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:22 smithi110 ceph-mon[97633]: pgmap v15843: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:44:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:23.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:23.100438+0000) 2022-09-10T12:44:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:23.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:23.554344+0000) 2022-09-10T12:44:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:24.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:24.100633+0000) 2022-09-10T12:44:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:24.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:24.554471+0000) 2022-09-10T12:44:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:24 smithi104 ceph-mon[97044]: pgmap v15844: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:44:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:24 smithi110 ceph-mon[97633]: pgmap v15844: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:44:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:25.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:25.100839+0000) 2022-09-10T12:44:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:25.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:25.554600+0000) 2022-09-10T12:44:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:26.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:26.101127+0000) 2022-09-10T12:44:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:26.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:26.554761+0000) 2022-09-10T12:44:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:26 smithi104 ceph-mon[97044]: pgmap v15845: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:44:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:26 smithi110 ceph-mon[97633]: pgmap v15845: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:44:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:27.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:27.101298+0000) 2022-09-10T12:44:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:27.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:27.554897+0000) 2022-09-10T12:44:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:28.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:28.101521+0000) 2022-09-10T12:44:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:28.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:28.555026+0000) 2022-09-10T12:44:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:28 smithi104 ceph-mon[97044]: pgmap v15846: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:28 smithi110 ceph-mon[97633]: pgmap v15846: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:29.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:29.101788+0000) 2022-09-10T12:44:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:29.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:29.555207+0000) 2022-09-10T12:44:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:30.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:30.102001+0000) 2022-09-10T12:44:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:30.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:30.555406+0000) 2022-09-10T12:44:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:44:30.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:44:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:30 smithi104 ceph-mon[97044]: pgmap v15847: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:30 smithi110 ceph-mon[97633]: pgmap v15847: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:31.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:31.102224+0000) 2022-09-10T12:44:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:31.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:31.555598+0000) 2022-09-10T12:44:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:32.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:32.102392+0000) 2022-09-10T12:44:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:32.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:32.555779+0000) 2022-09-10T12:44:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:32 smithi104 ceph-mon[97044]: pgmap v15848: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:44:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:32 smithi110 ceph-mon[97633]: pgmap v15848: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:44:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:33.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:33.102621+0000) 2022-09-10T12:44:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:33.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:33.555956+0000) 2022-09-10T12:44:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:34.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:34.102823+0000) 2022-09-10T12:44:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:34.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:34.556091+0000) 2022-09-10T12:44:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:34 smithi104 ceph-mon[97044]: pgmap v15849: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:34 smithi110 ceph-mon[97633]: pgmap v15849: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:35.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:35.103079+0000) 2022-09-10T12:44:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:35.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:35.556304+0000) 2022-09-10T12:44:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:36.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:36.103363+0000) 2022-09-10T12:44:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:36.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:36.556480+0000) 2022-09-10T12:44:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:36 smithi104 ceph-mon[97044]: pgmap v15850: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:44:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:36 smithi110 ceph-mon[97633]: pgmap v15850: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:44:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:37.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:37.103620+0000) 2022-09-10T12:44:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:37.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:37.556677+0000) 2022-09-10T12:44:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:38.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:38.103882+0000) 2022-09-10T12:44:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:38.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:38.556907+0000) 2022-09-10T12:44:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:38 smithi104 ceph-mon[97044]: pgmap v15851: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:38 smithi110 ceph-mon[97633]: pgmap v15851: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:39.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:39.104163+0000) 2022-09-10T12:44:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:39.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:39.557053+0000) 2022-09-10T12:44:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:40.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:40.104392+0000) 2022-09-10T12:44:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:40.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:40.557265+0000) 2022-09-10T12:44:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:40 smithi104 ceph-mon[97044]: pgmap v15852: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:40 smithi110 ceph-mon[97633]: pgmap v15852: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:41.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:41.104606+0000) 2022-09-10T12:44:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:41.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:41.557451+0000) 2022-09-10T12:44:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:42.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:42.104836+0000) 2022-09-10T12:44:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:42.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:42.557618+0000) 2022-09-10T12:44:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:42 smithi104 ceph-mon[97044]: pgmap v15853: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:44:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:42 smithi110 ceph-mon[97633]: pgmap v15853: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:44:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:43.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:43.105036+0000) 2022-09-10T12:44:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:43.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:43.557836+0000) 2022-09-10T12:44:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:44.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:44.105270+0000) 2022-09-10T12:44:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:44.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:44.557994+0000) 2022-09-10T12:44:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:45 smithi104 ceph-mon[97044]: pgmap v15854: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:45 smithi110 ceph-mon[97633]: pgmap v15854: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:45.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:45.105475+0000) 2022-09-10T12:44:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:45.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:45.558200+0000) 2022-09-10T12:44:46.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:44:45.594+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:44:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:46.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:46.105719+0000) 2022-09-10T12:44:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:46.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:46.558358+0000) 2022-09-10T12:44:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:47 smithi104 ceph-mon[97044]: pgmap v15855: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:44:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:44:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:47 smithi110 ceph-mon[97633]: pgmap v15855: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:44:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:44:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:47.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:47.105953+0000) 2022-09-10T12:44:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:47.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:47.558500+0000) 2022-09-10T12:44:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:48.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:48.106217+0000) 2022-09-10T12:44:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:48.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:48.558640+0000) 2022-09-10T12:44:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:49 smithi104 ceph-mon[97044]: pgmap v15856: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:49 smithi110 ceph-mon[97633]: pgmap v15856: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:49.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:49.106474+0000) 2022-09-10T12:44:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:49.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:49.558790+0000) 2022-09-10T12:44:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:50.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:50.106697+0000) 2022-09-10T12:44:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:50.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:50.558995+0000) 2022-09-10T12:44:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:51 smithi104 ceph-mon[97044]: pgmap v15857: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:51 smithi110 ceph-mon[97633]: pgmap v15857: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:44:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:51.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:51.106898+0000) 2022-09-10T12:44:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:51.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:51.559159+0000) 2022-09-10T12:44:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:52.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:52.107065+0000) 2022-09-10T12:44:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:52.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:52.559282+0000) 2022-09-10T12:44:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:53 smithi104 ceph-mon[97044]: pgmap v15858: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:44:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:53 smithi110 ceph-mon[97633]: pgmap v15858: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:44:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:53.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:53.107298+0000) 2022-09-10T12:44:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:53.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:53.559419+0000) 2022-09-10T12:44:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:54.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:54.107548+0000) 2022-09-10T12:44:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:54.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:54.559550+0000) 2022-09-10T12:44:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:55 smithi104 ceph-mon[97044]: pgmap v15859: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:55 smithi110 ceph-mon[97633]: pgmap v15859: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:55.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:55.107751+0000) 2022-09-10T12:44:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:55.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:55.559742+0000) 2022-09-10T12:44:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:56.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:56.107996+0000) 2022-09-10T12:44:56.947 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:56.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:56.559874+0000) 2022-09-10T12:44:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:57 smithi104 ceph-mon[97044]: pgmap v15860: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:44:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:57 smithi110 ceph-mon[97633]: pgmap v15860: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:44:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:57.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:57.108261+0000) 2022-09-10T12:44:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:57.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:57.560000+0000) 2022-09-10T12:44:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:58.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:58.108455+0000) 2022-09-10T12:44:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:58.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:58.560157+0000) 2022-09-10T12:44:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:44:59 smithi104 ceph-mon[97044]: pgmap v15861: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:44:59 smithi110 ceph-mon[97633]: pgmap v15861: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:44:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:44:59.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:59.108691+0000) 2022-09-10T12:45:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:44:59.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:44:59.560347+0000) 2022-09-10T12:45:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:00.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:00.108911+0000) 2022-09-10T12:45:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:00.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:00.560498+0000) 2022-09-10T12:45:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:45:00.609+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:45:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:01 smithi104 ceph-mon[97044]: pgmap v15862: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:01 smithi110 ceph-mon[97633]: pgmap v15862: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:01.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:01.109146+0000) 2022-09-10T12:45:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:01.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:01.560625+0000) 2022-09-10T12:45:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:02.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:02.109357+0000) 2022-09-10T12:45:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:02.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:02.560769+0000) 2022-09-10T12:45:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:03 smithi104 ceph-mon[97044]: pgmap v15863: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:03 smithi110 ceph-mon[97633]: pgmap v15863: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:03.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:03.109608+0000) 2022-09-10T12:45:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:03.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:03.560980+0000) 2022-09-10T12:45:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:04.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:04.109824+0000) 2022-09-10T12:45:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:04.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:04.561240+0000) 2022-09-10T12:45:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:05 smithi104 ceph-mon[97044]: pgmap v15864: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:05 smithi110 ceph-mon[97633]: pgmap v15864: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:05.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:05.110069+0000) 2022-09-10T12:45:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:05.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:05.561476+0000) 2022-09-10T12:45:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:06.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:06.110338+0000) 2022-09-10T12:45:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:06.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:06.561604+0000) 2022-09-10T12:45:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:07 smithi104 ceph-mon[97044]: pgmap v15865: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:45:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:07 smithi110 ceph-mon[97633]: pgmap v15865: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:45:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:07.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:07.110502+0000) 2022-09-10T12:45:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:07.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:07.561754+0000) 2022-09-10T12:45:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:08.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:08.110663+0000) 2022-09-10T12:45:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:08.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:08.561886+0000) 2022-09-10T12:45:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:09 smithi104 ceph-mon[97044]: pgmap v15866: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:45:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:09 smithi104 ceph-mon[97044]: osdmap e1072: 12 total, 12 up, 12 in 2022-09-10T12:45:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:09 smithi110 ceph-mon[97633]: pgmap v15866: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:45:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:09 smithi110 ceph-mon[97633]: osdmap e1072: 12 total, 12 up, 12 in 2022-09-10T12:45:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:09.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:09.110900+0000) 2022-09-10T12:45:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:09.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:09.562073+0000) 2022-09-10T12:45:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:10.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:10.111190+0000) 2022-09-10T12:45:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:10.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:10.562294+0000) 2022-09-10T12:45:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:11 smithi104 ceph-mon[97044]: pgmap v15868: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:45:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:11 smithi110 ceph-mon[97633]: pgmap v15868: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:45:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:11.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:11.111426+0000) 2022-09-10T12:45:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:11.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:11.562429+0000) 2022-09-10T12:45:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:12.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:12.111614+0000) 2022-09-10T12:45:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:12.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:12.562561+0000) 2022-09-10T12:45:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:13 smithi110 ceph-mon[97633]: pgmap v15869: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:45:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:13 smithi104 ceph-mon[97044]: pgmap v15869: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:45:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:13.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:13.111804+0000) 2022-09-10T12:45:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:13.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:13.562731+0000) 2022-09-10T12:45:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:14.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:14.112027+0000) 2022-09-10T12:45:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:14.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:14.562941+0000) 2022-09-10T12:45:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:15 smithi110 ceph-mon[97633]: pgmap v15870: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:45:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:15 smithi110 ceph-mon[97633]: osdmap e1073: 12 total, 12 up, 12 in 2022-09-10T12:45:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:15 smithi104 ceph-mon[97044]: pgmap v15870: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:45:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:15 smithi104 ceph-mon[97044]: osdmap e1073: 12 total, 12 up, 12 in 2022-09-10T12:45:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:15.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:15.112194+0000) 2022-09-10T12:45:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:15.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:15.563082+0000) 2022-09-10T12:45:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:45:15.612+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:45:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:16.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:16.112455+0000) 2022-09-10T12:45:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:16.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:16.563328+0000) 2022-09-10T12:45:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:17 smithi110 ceph-mon[97633]: pgmap v15872: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:45:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:17 smithi104 ceph-mon[97044]: pgmap v15872: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:45:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:17.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:17.112744+0000) 2022-09-10T12:45:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:17.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:17.563479+0000) 2022-09-10T12:45:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:18.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:18.112967+0000) 2022-09-10T12:45:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:18.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:18.563708+0000) 2022-09-10T12:45:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:19 smithi110 ceph-mon[97633]: pgmap v15873: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:45:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:19 smithi104 ceph-mon[97044]: pgmap v15873: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:45:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:19.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:19.113168+0000) 2022-09-10T12:45:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:19.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:19.563859+0000) 2022-09-10T12:45:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:20.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:20.113424+0000) 2022-09-10T12:45:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:20.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:20.563996+0000) 2022-09-10T12:45:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:21 smithi110 ceph-mon[97633]: pgmap v15874: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:45:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:21 smithi104 ceph-mon[97044]: pgmap v15874: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:45:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:21.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:21.113657+0000) 2022-09-10T12:45:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:21.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:21.564220+0000) 2022-09-10T12:45:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:22.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:22.113853+0000) 2022-09-10T12:45:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:22.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:22.564409+0000) 2022-09-10T12:45:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:23 smithi110 ceph-mon[97633]: pgmap v15875: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:45:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:23 smithi104 ceph-mon[97044]: pgmap v15875: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:45:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:23.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:23.114079+0000) 2022-09-10T12:45:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:23.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:23.564563+0000) 2022-09-10T12:45:24.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:24 smithi110 ceph-mon[97633]: pgmap v15876: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:45:24.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:24 smithi104 ceph-mon[97044]: pgmap v15876: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:45:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:24.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:24.114336+0000) 2022-09-10T12:45:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:24.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:24.564778+0000) 2022-09-10T12:45:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:25.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:25.114597+0000) 2022-09-10T12:45:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:25.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:25.564906+0000) 2022-09-10T12:45:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:26.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:26.114838+0000) 2022-09-10T12:45:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:26.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:26.565069+0000) 2022-09-10T12:45:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:26 smithi104 ceph-mon[97044]: pgmap v15877: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:45:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:26 smithi110 ceph-mon[97633]: pgmap v15877: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:45:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:27.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:27.115058+0000) 2022-09-10T12:45:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:27.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:27.565240+0000) 2022-09-10T12:45:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:28.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:28.115345+0000) 2022-09-10T12:45:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:28.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:28.565435+0000) 2022-09-10T12:45:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:28 smithi104 ceph-mon[97044]: pgmap v15878: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:28 smithi110 ceph-mon[97633]: pgmap v15878: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:29.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:29.115605+0000) 2022-09-10T12:45:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:29.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:29.565644+0000) 2022-09-10T12:45:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:30.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:30.115803+0000) 2022-09-10T12:45:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:30.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:30.565810+0000) 2022-09-10T12:45:31.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:45:30.617+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:45:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:30 smithi104 ceph-mon[97044]: pgmap v15879: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:30 smithi110 ceph-mon[97633]: pgmap v15879: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:31.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:31.116018+0000) 2022-09-10T12:45:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:31.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:31.566028+0000) 2022-09-10T12:45:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:32.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:32.116246+0000) 2022-09-10T12:45:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:32.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:32.566206+0000) 2022-09-10T12:45:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:32 smithi104 ceph-mon[97044]: pgmap v15880: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:32 smithi110 ceph-mon[97633]: pgmap v15880: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:33.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:33.116458+0000) 2022-09-10T12:45:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:33.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:33.566389+0000) 2022-09-10T12:45:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:34.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:34.116635+0000) 2022-09-10T12:45:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:34.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:34.566564+0000) 2022-09-10T12:45:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:34 smithi104 ceph-mon[97044]: pgmap v15881: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:34 smithi110 ceph-mon[97633]: pgmap v15881: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:35.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:35.116894+0000) 2022-09-10T12:45:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:35.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:35.566712+0000) 2022-09-10T12:45:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:36.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:36.117086+0000) 2022-09-10T12:45:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:36.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:36.566889+0000) 2022-09-10T12:45:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:36 smithi104 ceph-mon[97044]: pgmap v15882: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:45:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:36 smithi110 ceph-mon[97633]: pgmap v15882: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:45:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:37.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:37.117324+0000) 2022-09-10T12:45:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:37.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:37.567052+0000) 2022-09-10T12:45:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:38.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:38.117549+0000) 2022-09-10T12:45:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:38.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:38.567256+0000) 2022-09-10T12:45:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:38 smithi104 ceph-mon[97044]: pgmap v15883: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:38 smithi110 ceph-mon[97633]: pgmap v15883: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:39.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:39.117774+0000) 2022-09-10T12:45:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:39.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:39.567488+0000) 2022-09-10T12:45:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:40.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:40.118012+0000) 2022-09-10T12:45:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:40.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:40.567664+0000) 2022-09-10T12:45:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:40 smithi104 ceph-mon[97044]: pgmap v15884: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:41.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:40 smithi110 ceph-mon[97633]: pgmap v15884: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:41.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:41.118234+0000) 2022-09-10T12:45:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:41.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:41.567876+0000) 2022-09-10T12:45:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:42.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:42.118424+0000) 2022-09-10T12:45:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:42.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:42.568089+0000) 2022-09-10T12:45:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:42 smithi104 ceph-mon[97044]: pgmap v15885: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:42 smithi110 ceph-mon[97633]: pgmap v15885: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:43.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:43.118644+0000) 2022-09-10T12:45:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:43.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:43.568303+0000) 2022-09-10T12:45:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:44.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:44.118942+0000) 2022-09-10T12:45:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:44.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:44.568464+0000) 2022-09-10T12:45:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:45 smithi104 ceph-mon[97044]: pgmap v15886: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:45 smithi110 ceph-mon[97633]: pgmap v15886: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:45.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:45.119199+0000) 2022-09-10T12:45:46.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:45:45.620+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:45:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:45.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:45.568669+0000) 2022-09-10T12:45:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:46.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:46.119464+0000) 2022-09-10T12:45:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:46.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:46.568885+0000) 2022-09-10T12:45:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:47 smithi104 ceph-mon[97044]: pgmap v15887: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:45:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:45:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:47 smithi110 ceph-mon[97633]: pgmap v15887: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:45:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:45:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:45:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:45:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:45:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:47.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:47.119728+0000) 2022-09-10T12:45:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:47.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:47.569088+0000) 2022-09-10T12:45:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:48.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:48.119989+0000) 2022-09-10T12:45:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:48.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:48.569329+0000) 2022-09-10T12:45:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:49 smithi104 ceph-mon[97044]: pgmap v15888: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:49 smithi110 ceph-mon[97633]: pgmap v15888: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:49.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:49.120262+0000) 2022-09-10T12:45:50.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:49.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:49.569560+0000) 2022-09-10T12:45:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:50.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:50.120487+0000) 2022-09-10T12:45:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:50.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:50.569750+0000) 2022-09-10T12:45:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:51 smithi104 ceph-mon[97044]: pgmap v15889: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:51 smithi110 ceph-mon[97633]: pgmap v15889: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:45:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:51.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:51.120618+0000) 2022-09-10T12:45:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:51.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:51.569954+0000) 2022-09-10T12:45:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:52.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:52.120800+0000) 2022-09-10T12:45:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:52.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:52.570173+0000) 2022-09-10T12:45:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:53 smithi104 ceph-mon[97044]: pgmap v15890: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:53 smithi110 ceph-mon[97633]: pgmap v15890: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:45:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:53.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:53.121040+0000) 2022-09-10T12:45:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:53.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:53.570335+0000) 2022-09-10T12:45:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:54.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:54.121244+0000) 2022-09-10T12:45:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:54.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:54.570536+0000) 2022-09-10T12:45:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:55 smithi104 ceph-mon[97044]: pgmap v15891: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:55 smithi110 ceph-mon[97633]: pgmap v15891: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:55.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:55.121391+0000) 2022-09-10T12:45:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:55.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:55.570712+0000) 2022-09-10T12:45:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:56.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:56.121618+0000) 2022-09-10T12:45:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:56.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:56.570862+0000) 2022-09-10T12:45:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:57 smithi104 ceph-mon[97044]: pgmap v15892: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:45:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:57 smithi110 ceph-mon[97633]: pgmap v15892: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:45:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:57.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:57.121825+0000) 2022-09-10T12:45:57.875 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:57.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:57.571028+0000) 2022-09-10T12:45:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:58.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:58.122083+0000) 2022-09-10T12:45:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:58.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:58.571195+0000) 2022-09-10T12:45:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:45:59 smithi104 ceph-mon[97044]: pgmap v15893: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:45:59 smithi110 ceph-mon[97633]: pgmap v15893: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:45:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:45:59.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:59.122390+0000) 2022-09-10T12:46:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:45:59.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:45:59.571373+0000) 2022-09-10T12:46:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:00.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:00.122604+0000) 2022-09-10T12:46:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:00.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:00.571592+0000) 2022-09-10T12:46:01.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:46:00.637+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:46:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:01 smithi104 ceph-mon[97044]: pgmap v15894: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:46:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:01 smithi110 ceph-mon[97633]: pgmap v15894: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:46:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:01.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:01.122796+0000) 2022-09-10T12:46:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:01.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:01.571794+0000) 2022-09-10T12:46:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:02.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:02.123021+0000) 2022-09-10T12:46:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:02.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:02.572002+0000) 2022-09-10T12:46:03.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:03 smithi104 ceph-mon[97044]: pgmap v15895: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:03 smithi110 ceph-mon[97633]: pgmap v15895: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:03.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:03.123278+0000) 2022-09-10T12:46:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:03.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:03.572222+0000) 2022-09-10T12:46:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:04.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:04.123473+0000) 2022-09-10T12:46:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:04.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:04.572399+0000) 2022-09-10T12:46:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:05 smithi104 ceph-mon[97044]: pgmap v15896: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:05 smithi110 ceph-mon[97633]: pgmap v15896: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:05.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:05.123675+0000) 2022-09-10T12:46:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:05.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:05.572613+0000) 2022-09-10T12:46:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:06.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:06.123925+0000) 2022-09-10T12:46:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:06.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:06.572840+0000) 2022-09-10T12:46:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:07 smithi104 ceph-mon[97044]: pgmap v15897: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:46:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:07 smithi110 ceph-mon[97633]: pgmap v15897: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:46:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:07.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:07.124201+0000) 2022-09-10T12:46:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:07.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:07.573057+0000) 2022-09-10T12:46:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:08.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:08.124389+0000) 2022-09-10T12:46:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:08.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:08.573313+0000) 2022-09-10T12:46:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:09 smithi104 ceph-mon[97044]: pgmap v15898: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:46:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:09 smithi104 ceph-mon[97044]: osdmap e1074: 12 total, 12 up, 12 in 2022-09-10T12:46:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:09 smithi110 ceph-mon[97633]: pgmap v15898: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:46:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:09 smithi110 ceph-mon[97633]: osdmap e1074: 12 total, 12 up, 12 in 2022-09-10T12:46:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:09.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:09.124635+0000) 2022-09-10T12:46:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:09.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:09.573539+0000) 2022-09-10T12:46:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:10.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:10.124813+0000) 2022-09-10T12:46:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:10.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:10.573717+0000) 2022-09-10T12:46:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:11 smithi104 ceph-mon[97044]: pgmap v15900: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:46:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:11 smithi110 ceph-mon[97633]: pgmap v15900: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:46:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:11.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:11.125020+0000) 2022-09-10T12:46:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:11.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:11.573895+0000) 2022-09-10T12:46:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:12.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:12.125266+0000) 2022-09-10T12:46:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:12.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:12.574134+0000) 2022-09-10T12:46:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:13 smithi104 ceph-mon[97044]: pgmap v15901: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:46:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:13 smithi110 ceph-mon[97633]: pgmap v15901: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:46:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:13.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:13.125453+0000) 2022-09-10T12:46:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:13.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:13.574325+0000) 2022-09-10T12:46:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:14.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:14.125637+0000) 2022-09-10T12:46:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:14.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:14.574536+0000) 2022-09-10T12:46:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:15 smithi110 ceph-mon[97633]: pgmap v15902: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:46:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:15 smithi110 ceph-mon[97633]: osdmap e1075: 12 total, 12 up, 12 in 2022-09-10T12:46:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:15 smithi104 ceph-mon[97044]: pgmap v15902: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:46:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:15 smithi104 ceph-mon[97044]: osdmap e1075: 12 total, 12 up, 12 in 2022-09-10T12:46:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:15.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:15.125855+0000) 2022-09-10T12:46:16.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:46:15.651+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:46:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:15.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:15.574745+0000) 2022-09-10T12:46:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:16.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:16.126144+0000) 2022-09-10T12:46:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:16.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:16.574940+0000) 2022-09-10T12:46:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:17 smithi110 ceph-mon[97633]: pgmap v15904: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:46:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:17 smithi104 ceph-mon[97044]: pgmap v15904: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:46:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:17.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:17.126343+0000) 2022-09-10T12:46:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:17.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:17.575189+0000) 2022-09-10T12:46:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:18.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:18.126552+0000) 2022-09-10T12:46:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:18.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:18.575335+0000) 2022-09-10T12:46:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:19 smithi110 ceph-mon[97633]: pgmap v15905: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:46:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:19 smithi104 ceph-mon[97044]: pgmap v15905: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:46:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:19.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:19.126762+0000) 2022-09-10T12:46:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:19.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:19.575545+0000) 2022-09-10T12:46:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:20.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:20.127001+0000) 2022-09-10T12:46:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:20.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:20.575763+0000) 2022-09-10T12:46:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:21 smithi110 ceph-mon[97633]: pgmap v15906: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:46:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:21 smithi104 ceph-mon[97044]: pgmap v15906: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:46:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:21.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:21.127243+0000) 2022-09-10T12:46:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:21.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:21.575923+0000) 2022-09-10T12:46:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:22.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:22.127385+0000) 2022-09-10T12:46:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:22.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:22.576144+0000) 2022-09-10T12:46:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:23 smithi110 ceph-mon[97633]: pgmap v15907: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:46:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:23 smithi104 ceph-mon[97044]: pgmap v15907: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:46:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:23.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:23.127561+0000) 2022-09-10T12:46:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:23.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:23.576301+0000) 2022-09-10T12:46:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:24.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:24.127788+0000) 2022-09-10T12:46:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:24.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:24.576520+0000) 2022-09-10T12:46:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:25 smithi110 ceph-mon[97633]: pgmap v15908: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:46:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:25 smithi104 ceph-mon[97044]: pgmap v15908: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:46:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:25.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:25.128041+0000) 2022-09-10T12:46:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:25.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:25.576721+0000) 2022-09-10T12:46:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:26.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:26.128281+0000) 2022-09-10T12:46:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:26.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:26.576915+0000) 2022-09-10T12:46:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:27 smithi110 ceph-mon[97633]: pgmap v15909: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:46:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:27 smithi104 ceph-mon[97044]: pgmap v15909: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:46:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:27.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:27.128531+0000) 2022-09-10T12:46:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:27.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:27.577151+0000) 2022-09-10T12:46:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:28.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:28.128741+0000) 2022-09-10T12:46:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:28.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:28.577316+0000) 2022-09-10T12:46:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:29 smithi110 ceph-mon[97633]: pgmap v15910: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:29 smithi104 ceph-mon[97044]: pgmap v15910: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:29.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:29.128965+0000) 2022-09-10T12:46:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:29.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:29.577490+0000) 2022-09-10T12:46:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:30.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:30.129223+0000) 2022-09-10T12:46:31.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:46:30.655+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:46:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:30.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:30.577697+0000) 2022-09-10T12:46:31.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:30 smithi104 ceph-mon[97044]: pgmap v15911: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T12:46:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:30 smithi110 ceph-mon[97633]: pgmap v15911: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2022-09-10T12:46:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:31.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:31.129503+0000) 2022-09-10T12:46:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:31.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:31.577911+0000) 2022-09-10T12:46:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:32.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:32.129712+0000) 2022-09-10T12:46:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:32.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:32.578038+0000) 2022-09-10T12:46:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:32 smithi104 ceph-mon[97044]: pgmap v15912: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:32 smithi110 ceph-mon[97633]: pgmap v15912: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:33.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:33.129841+0000) 2022-09-10T12:46:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:33.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:33.578223+0000) 2022-09-10T12:46:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:34.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:34.130111+0000) 2022-09-10T12:46:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:34.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:34.578436+0000) 2022-09-10T12:46:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:34 smithi104 ceph-mon[97044]: pgmap v15913: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:34 smithi110 ceph-mon[97633]: pgmap v15913: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:35.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:35.130342+0000) 2022-09-10T12:46:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:35.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:35.578637+0000) 2022-09-10T12:46:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:36.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:36.130598+0000) 2022-09-10T12:46:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:36.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:36.578829+0000) 2022-09-10T12:46:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:36 smithi104 ceph-mon[97044]: pgmap v15914: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:46:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:36 smithi110 ceph-mon[97633]: pgmap v15914: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:46:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:37.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:37.130837+0000) 2022-09-10T12:46:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:37.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:37.579055+0000) 2022-09-10T12:46:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:38.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:38.131048+0000) 2022-09-10T12:46:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:38.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:38.579219+0000) 2022-09-10T12:46:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:39 smithi104 ceph-mon[97044]: pgmap v15915: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:39 smithi110 ceph-mon[97633]: pgmap v15915: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:39.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:39.131282+0000) 2022-09-10T12:46:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:39.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:39.579410+0000) 2022-09-10T12:46:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:40.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:40.131567+0000) 2022-09-10T12:46:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:40.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:40.579594+0000) 2022-09-10T12:46:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:41 smithi104 ceph-mon[97044]: pgmap v15916: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:46:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:41 smithi110 ceph-mon[97633]: pgmap v15916: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:46:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:41.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:41.131786+0000) 2022-09-10T12:46:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:41.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:41.579765+0000) 2022-09-10T12:46:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:42.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:42.132046+0000) 2022-09-10T12:46:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:42.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:42.579987+0000) 2022-09-10T12:46:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:43 smithi104 ceph-mon[97044]: pgmap v15917: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:43 smithi110 ceph-mon[97633]: pgmap v15917: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:43.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:43.132242+0000) 2022-09-10T12:46:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:43.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:43.580185+0000) 2022-09-10T12:46:44.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:44.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:44.132452+0000) 2022-09-10T12:46:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:44.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:44.580384+0000) 2022-09-10T12:46:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:45 smithi104 ceph-mon[97044]: pgmap v15918: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:45 smithi110 ceph-mon[97633]: pgmap v15918: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:45.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:45.132631+0000) 2022-09-10T12:46:46.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:46:45.671+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:46:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:45.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:45.580579+0000) 2022-09-10T12:46:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:46.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:46.132839+0000) 2022-09-10T12:46:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:46.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:46.580761+0000) 2022-09-10T12:46:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:47 smithi104 ceph-mon[97044]: pgmap v15919: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:46:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:46:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:46:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:46:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:46:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:47 smithi110 ceph-mon[97633]: pgmap v15919: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:46:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:47.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:47.133040+0000) 2022-09-10T12:46:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:47.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:47.580986+0000) 2022-09-10T12:46:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:48.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:48.133270+0000) 2022-09-10T12:46:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:48.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:48.581182+0000) 2022-09-10T12:46:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:49 smithi104 ceph-mon[97044]: pgmap v15920: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:49 smithi110 ceph-mon[97633]: pgmap v15920: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:49.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:49.133508+0000) 2022-09-10T12:46:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:49.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:49.581370+0000) 2022-09-10T12:46:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:50.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:50.133743+0000) 2022-09-10T12:46:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:50.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:50.581516+0000) 2022-09-10T12:46:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:51 smithi104 ceph-mon[97044]: pgmap v15921: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:46:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:51 smithi110 ceph-mon[97633]: pgmap v15921: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:46:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:51.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:51.133947+0000) 2022-09-10T12:46:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:51.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:51.581734+0000) 2022-09-10T12:46:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:52.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:52.134168+0000) 2022-09-10T12:46:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:52.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:52.581907+0000) 2022-09-10T12:46:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:53 smithi104 ceph-mon[97044]: pgmap v15922: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:53 smithi110 ceph-mon[97633]: pgmap v15922: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:46:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:53.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:53.134433+0000) 2022-09-10T12:46:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:53.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:53.582160+0000) 2022-09-10T12:46:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:54.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:54.134615+0000) 2022-09-10T12:46:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:54.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:54.582353+0000) 2022-09-10T12:46:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:55 smithi104 ceph-mon[97044]: pgmap v15923: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:55 smithi110 ceph-mon[97633]: pgmap v15923: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:55.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:55.134918+0000) 2022-09-10T12:46:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:55.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:55.582589+0000) 2022-09-10T12:46:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:56.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:56.135193+0000) 2022-09-10T12:46:57.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:56.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:56.582774+0000) 2022-09-10T12:46:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:57 smithi104 ceph-mon[97044]: pgmap v15924: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:46:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:57 smithi110 ceph-mon[97633]: pgmap v15924: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:46:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:57.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:57.135372+0000) 2022-09-10T12:46:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:57.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:57.582946+0000) 2022-09-10T12:46:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:58.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:58.135532+0000) 2022-09-10T12:46:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:58.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:58.583115+0000) 2022-09-10T12:46:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:46:59 smithi104 ceph-mon[97044]: pgmap v15925: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:46:59 smithi110 ceph-mon[97633]: pgmap v15925: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:46:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:46:59.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:59.135804+0000) 2022-09-10T12:47:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:46:59.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:46:59.583255+0000) 2022-09-10T12:47:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:00.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:00.136019+0000) 2022-09-10T12:47:01.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:47:00.680+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:47:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:00.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:00.583389+0000) 2022-09-10T12:47:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:01 smithi104 ceph-mon[97044]: pgmap v15926: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:47:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:01 smithi110 ceph-mon[97633]: pgmap v15926: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:47:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:01.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:01.136204+0000) 2022-09-10T12:47:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:01.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:01.583567+0000) 2022-09-10T12:47:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:02.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:02.136433+0000) 2022-09-10T12:47:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:02.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:02.583770+0000) 2022-09-10T12:47:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:03 smithi104 ceph-mon[97044]: pgmap v15927: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:47:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:03 smithi110 ceph-mon[97633]: pgmap v15927: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:47:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:03.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:03.136644+0000) 2022-09-10T12:47:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:03.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:03.583915+0000) 2022-09-10T12:47:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:04.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:04.136896+0000) 2022-09-10T12:47:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:04.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:04.584086+0000) 2022-09-10T12:47:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:05 smithi104 ceph-mon[97044]: pgmap v15928: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:05 smithi110 ceph-mon[97633]: pgmap v15928: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:05.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:05.137185+0000) 2022-09-10T12:47:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:05.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:05.584253+0000) 2022-09-10T12:47:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:06.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:06.137430+0000) 2022-09-10T12:47:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:06.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:06.584447+0000) 2022-09-10T12:47:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:07 smithi104 ceph-mon[97044]: pgmap v15929: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:47:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:07 smithi110 ceph-mon[97633]: pgmap v15929: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:47:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:07.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:07.137677+0000) 2022-09-10T12:47:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:07.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:07.584679+0000) 2022-09-10T12:47:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:08.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:08.137882+0000) 2022-09-10T12:47:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:08.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:08.584919+0000) 2022-09-10T12:47:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:09 smithi110 ceph-mon[97633]: pgmap v15930: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:47:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:09 smithi110 ceph-mon[97633]: osdmap e1076: 12 total, 12 up, 12 in 2022-09-10T12:47:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:09 smithi104 ceph-mon[97044]: pgmap v15930: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:47:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:09 smithi104 ceph-mon[97044]: osdmap e1076: 12 total, 12 up, 12 in 2022-09-10T12:47:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:09.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:09.138130+0000) 2022-09-10T12:47:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:09.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:09.585069+0000) 2022-09-10T12:47:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:10.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:10.138359+0000) 2022-09-10T12:47:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:10.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:10.585276+0000) 2022-09-10T12:47:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:11 smithi110 ceph-mon[97633]: pgmap v15932: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:47:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:11 smithi104 ceph-mon[97044]: pgmap v15932: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:47:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:11.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:11.138578+0000) 2022-09-10T12:47:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:11.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:11.585484+0000) 2022-09-10T12:47:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:12.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:12.138764+0000) 2022-09-10T12:47:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:12.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:12.585609+0000) 2022-09-10T12:47:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:13 smithi110 ceph-mon[97633]: pgmap v15933: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:47:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:13 smithi104 ceph-mon[97044]: pgmap v15933: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:47:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:13.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:13.138889+0000) 2022-09-10T12:47:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:13.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:13.585801+0000) 2022-09-10T12:47:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:14.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:14.139058+0000) 2022-09-10T12:47:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:14.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:14.585962+0000) 2022-09-10T12:47:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:15 smithi110 ceph-mon[97633]: pgmap v15934: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:47:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:15 smithi110 ceph-mon[97633]: osdmap e1077: 12 total, 12 up, 12 in 2022-09-10T12:47:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:15 smithi104 ceph-mon[97044]: pgmap v15934: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:47:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:15 smithi104 ceph-mon[97044]: osdmap e1077: 12 total, 12 up, 12 in 2022-09-10T12:47:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:15.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:15.139325+0000) 2022-09-10T12:47:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:15.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:15.586126+0000) 2022-09-10T12:47:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:47:15.696+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:47:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:16.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:16.139554+0000) 2022-09-10T12:47:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:16.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:16.586289+0000) 2022-09-10T12:47:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:17 smithi110 ceph-mon[97633]: pgmap v15936: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:47:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:17 smithi104 ceph-mon[97044]: pgmap v15936: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:47:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:17.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:17.139769+0000) 2022-09-10T12:47:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:17.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:17.586482+0000) 2022-09-10T12:47:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:18.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:18.139943+0000) 2022-09-10T12:47:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:18.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:18.586655+0000) 2022-09-10T12:47:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:19 smithi110 ceph-mon[97633]: pgmap v15937: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:47:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:19 smithi104 ceph-mon[97044]: pgmap v15937: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:47:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:19.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:19.140191+0000) 2022-09-10T12:47:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:19.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:19.586892+0000) 2022-09-10T12:47:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:20.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:20.140378+0000) 2022-09-10T12:47:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:20.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:20.587122+0000) 2022-09-10T12:47:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:21 smithi110 ceph-mon[97633]: pgmap v15938: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:47:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:21 smithi104 ceph-mon[97044]: pgmap v15938: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:47:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:21.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:21.140554+0000) 2022-09-10T12:47:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:21.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:21.587272+0000) 2022-09-10T12:47:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:22.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:22.140750+0000) 2022-09-10T12:47:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:22.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:22.587443+0000) 2022-09-10T12:47:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:23 smithi110 ceph-mon[97633]: pgmap v15939: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:47:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:23 smithi104 ceph-mon[97044]: pgmap v15939: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:47:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:23.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:23.140942+0000) 2022-09-10T12:47:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:23.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:23.587621+0000) 2022-09-10T12:47:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:24.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:24.141187+0000) 2022-09-10T12:47:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:24.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:24.587788+0000) 2022-09-10T12:47:25.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:25 smithi110 ceph-mon[97633]: pgmap v15940: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:47:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:25 smithi104 ceph-mon[97044]: pgmap v15940: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:47:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:25.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:25.141419+0000) 2022-09-10T12:47:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:25.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:25.587969+0000) 2022-09-10T12:47:26.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:26 smithi110 ceph-mon[97633]: pgmap v15941: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:47:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:26 smithi104 ceph-mon[97044]: pgmap v15941: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:47:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:26.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:26.141667+0000) 2022-09-10T12:47:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:26.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:26.588127+0000) 2022-09-10T12:47:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:27.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:27.141861+0000) 2022-09-10T12:47:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:27.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:27.588311+0000) 2022-09-10T12:47:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:28.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:28.142030+0000) 2022-09-10T12:47:28.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:28 smithi104 ceph-mon[97044]: pgmap v15942: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:28 smithi110 ceph-mon[97633]: pgmap v15942: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:29.012 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:28.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:28.588463+0000) 2022-09-10T12:47:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:29.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:29.142198+0000) 2022-09-10T12:47:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:29.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:29.588622+0000) 2022-09-10T12:47:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:30.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:30.142410+0000) 2022-09-10T12:47:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:30.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:30.588811+0000) 2022-09-10T12:47:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:47:30.701+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:47:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:31 smithi104 ceph-mon[97044]: pgmap v15943: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:47:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:31 smithi110 ceph-mon[97633]: pgmap v15943: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:47:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:31.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:31.142576+0000) 2022-09-10T12:47:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:31.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:31.589027+0000) 2022-09-10T12:47:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:32.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:32.142787+0000) 2022-09-10T12:47:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:32.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:32.589164+0000) 2022-09-10T12:47:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:33 smithi104 ceph-mon[97044]: pgmap v15944: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:47:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:33 smithi110 ceph-mon[97633]: pgmap v15944: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:47:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:33.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:33.142973+0000) 2022-09-10T12:47:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:33.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:33.589288+0000) 2022-09-10T12:47:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:34.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:34.143190+0000) 2022-09-10T12:47:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:34.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:34.589461+0000) 2022-09-10T12:47:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:35 smithi104 ceph-mon[97044]: pgmap v15945: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:35 smithi110 ceph-mon[97633]: pgmap v15945: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:35.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:35.143451+0000) 2022-09-10T12:47:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:35.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:35.589631+0000) 2022-09-10T12:47:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:36.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:36.143618+0000) 2022-09-10T12:47:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:36.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:36.589795+0000) 2022-09-10T12:47:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:37 smithi104 ceph-mon[97044]: pgmap v15946: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:47:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:37 smithi110 ceph-mon[97633]: pgmap v15946: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:47:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:37.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:37.143858+0000) 2022-09-10T12:47:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:37.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:37.589953+0000) 2022-09-10T12:47:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:38.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:38.144135+0000) 2022-09-10T12:47:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:38.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:38.590104+0000) 2022-09-10T12:47:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:39 smithi104 ceph-mon[97044]: pgmap v15947: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:39 smithi110 ceph-mon[97633]: pgmap v15947: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:39.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:39.144349+0000) 2022-09-10T12:47:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:39.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:39.590318+0000) 2022-09-10T12:47:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:40.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:40.144605+0000) 2022-09-10T12:47:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:40.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:40.590444+0000) 2022-09-10T12:47:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:41 smithi104 ceph-mon[97044]: pgmap v15948: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:47:41.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:41 smithi110 ceph-mon[97633]: pgmap v15948: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:47:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:41.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:41.144831+0000) 2022-09-10T12:47:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:41.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:41.590578+0000) 2022-09-10T12:47:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:42.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:42.144980+0000) 2022-09-10T12:47:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:42.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:42.590750+0000) 2022-09-10T12:47:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:43 smithi104 ceph-mon[97044]: pgmap v15949: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:47:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:43 smithi110 ceph-mon[97633]: pgmap v15949: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:47:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:43.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:43.145170+0000) 2022-09-10T12:47:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:43.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:43.590953+0000) 2022-09-10T12:47:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:44.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:44.145386+0000) 2022-09-10T12:47:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:44.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:44.591176+0000) 2022-09-10T12:47:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:45 smithi104 ceph-mon[97044]: pgmap v15950: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:45 smithi110 ceph-mon[97633]: pgmap v15950: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:45.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:45.145613+0000) 2022-09-10T12:47:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:45.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:45.591308+0000) 2022-09-10T12:47:46.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:47:45.704+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:47:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:46.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:46.145871+0000) 2022-09-10T12:47:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:46.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:46.591462+0000) 2022-09-10T12:47:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:47 smithi104 ceph-mon[97044]: pgmap v15951: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:47:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:47:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:47:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:47:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:47:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:47 smithi110 ceph-mon[97633]: pgmap v15951: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:47:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:47:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:47:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:47.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:47.146067+0000) 2022-09-10T12:47:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:47.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:47.591651+0000) 2022-09-10T12:47:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:48.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:48.146302+0000) 2022-09-10T12:47:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:48.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:48.591833+0000) 2022-09-10T12:47:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:49 smithi104 ceph-mon[97044]: pgmap v15952: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:49 smithi110 ceph-mon[97633]: pgmap v15952: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:47:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:49.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:49.146521+0000) 2022-09-10T12:47:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:49.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:49.592079+0000) 2022-09-10T12:47:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:50.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:50.146750+0000) 2022-09-10T12:47:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:50.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:50.592244+0000) 2022-09-10T12:47:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:51 smithi104 ceph-mon[97044]: pgmap v15953: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T12:47:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:51 smithi110 ceph-mon[97633]: pgmap v15953: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T12:47:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:51.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:51.146971+0000) 2022-09-10T12:47:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:51.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:51.592387+0000) 2022-09-10T12:47:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:52.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:52.147204+0000) 2022-09-10T12:47:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:52.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:52.592528+0000) 2022-09-10T12:47:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:53 smithi104 ceph-mon[97044]: pgmap v15954: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-09-10T12:47:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:53 smithi110 ceph-mon[97633]: pgmap v15954: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-09-10T12:47:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:53.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:53.147430+0000) 2022-09-10T12:47:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:53.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:53.592675+0000) 2022-09-10T12:47:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:54.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:54.147649+0000) 2022-09-10T12:47:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:54.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:54.592848+0000) 2022-09-10T12:47:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:55 smithi104 ceph-mon[97044]: pgmap v15955: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:47:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:55 smithi110 ceph-mon[97633]: pgmap v15955: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:47:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:55.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:55.147877+0000) 2022-09-10T12:47:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:55.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:55.593056+0000) 2022-09-10T12:47:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:56.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:56.148143+0000) 2022-09-10T12:47:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:56.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:56.593237+0000) 2022-09-10T12:47:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:57 smithi110 ceph-mon[97633]: pgmap v15956: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:47:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:57 smithi104 ceph-mon[97044]: pgmap v15956: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:47:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:57.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:57.148367+0000) 2022-09-10T12:47:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:57.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:57.593373+0000) 2022-09-10T12:47:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:58.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:58.148547+0000) 2022-09-10T12:47:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:58.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:58.593551+0000) 2022-09-10T12:47:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:47:59 smithi110 ceph-mon[97633]: pgmap v15957: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:47:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:47:59 smithi104 ceph-mon[97044]: pgmap v15957: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:47:59.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:47:59.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:59.148761+0000) 2022-09-10T12:48:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:47:59.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:47:59.593807+0000) 2022-09-10T12:48:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:00.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:00.148950+0000) 2022-09-10T12:48:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:00.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:00.593934+0000) 2022-09-10T12:48:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:48:00.720+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:48:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:01 smithi110 ceph-mon[97633]: pgmap v15958: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T12:48:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:01 smithi104 ceph-mon[97044]: pgmap v15958: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T12:48:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:01.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:01.149174+0000) 2022-09-10T12:48:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:01.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:01.594058+0000) 2022-09-10T12:48:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:02.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:02.149297+0000) 2022-09-10T12:48:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:02.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:02.594156+0000) 2022-09-10T12:48:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:03 smithi110 ceph-mon[97633]: pgmap v15959: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:03 smithi104 ceph-mon[97044]: pgmap v15959: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:03.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:03.149513+0000) 2022-09-10T12:48:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:03.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:03.594346+0000) 2022-09-10T12:48:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:04.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:04.149742+0000) 2022-09-10T12:48:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:04.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:04.594502+0000) 2022-09-10T12:48:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:05 smithi110 ceph-mon[97633]: pgmap v15960: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:05 smithi104 ceph-mon[97044]: pgmap v15960: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:05.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:05.149947+0000) 2022-09-10T12:48:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:05.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:05.594634+0000) 2022-09-10T12:48:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:06.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:06.150205+0000) 2022-09-10T12:48:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:06.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:06.594811+0000) 2022-09-10T12:48:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:07 smithi110 ceph-mon[97633]: pgmap v15961: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:48:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:07 smithi104 ceph-mon[97044]: pgmap v15961: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:48:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:07.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:07.150465+0000) 2022-09-10T12:48:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:07.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:07.595043+0000) 2022-09-10T12:48:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:08.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:08.150688+0000) 2022-09-10T12:48:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:08.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:08.595188+0000) 2022-09-10T12:48:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:09 smithi110 ceph-mon[97633]: pgmap v15962: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:48:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:09 smithi110 ceph-mon[97633]: osdmap e1078: 12 total, 12 up, 12 in 2022-09-10T12:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:09 smithi104 ceph-mon[97044]: pgmap v15962: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:48:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:09 smithi104 ceph-mon[97044]: osdmap e1078: 12 total, 12 up, 12 in 2022-09-10T12:48:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:09.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:09.150943+0000) 2022-09-10T12:48:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:09.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:09.595336+0000) 2022-09-10T12:48:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:10.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:10.151197+0000) 2022-09-10T12:48:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:10.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:10.595491+0000) 2022-09-10T12:48:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:11 smithi110 ceph-mon[97633]: pgmap v15964: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:48:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:11 smithi104 ceph-mon[97044]: pgmap v15964: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:48:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:11.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:11.151381+0000) 2022-09-10T12:48:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:11.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:11.595697+0000) 2022-09-10T12:48:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:12.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:12.151622+0000) 2022-09-10T12:48:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:12.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:12.595896+0000) 2022-09-10T12:48:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:13 smithi110 ceph-mon[97633]: pgmap v15965: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:48:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:13 smithi104 ceph-mon[97044]: pgmap v15965: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:48:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:13.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:13.151847+0000) 2022-09-10T12:48:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:13.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:13.596165+0000) 2022-09-10T12:48:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:14.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:14.152014+0000) 2022-09-10T12:48:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:14.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:14.596319+0000) 2022-09-10T12:48:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:15 smithi110 ceph-mon[97633]: pgmap v15966: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:48:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:15 smithi110 ceph-mon[97633]: osdmap e1079: 12 total, 12 up, 12 in 2022-09-10T12:48:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:15 smithi104 ceph-mon[97044]: pgmap v15966: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:48:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:15 smithi104 ceph-mon[97044]: osdmap e1079: 12 total, 12 up, 12 in 2022-09-10T12:48:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:15.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:15.152221+0000) 2022-09-10T12:48:16.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:48:15.727+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:48:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:15.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:15.596469+0000) 2022-09-10T12:48:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:16.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:16.152420+0000) 2022-09-10T12:48:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:16.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:16.596663+0000) 2022-09-10T12:48:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:17 smithi110 ceph-mon[97633]: pgmap v15968: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:48:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:17 smithi104 ceph-mon[97044]: pgmap v15968: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:48:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:17.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:17.152654+0000) 2022-09-10T12:48:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:17.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:17.596822+0000) 2022-09-10T12:48:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:18.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:18.152877+0000) 2022-09-10T12:48:19.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:18.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:18.596958+0000) 2022-09-10T12:48:19.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:19 smithi110 ceph-mon[97633]: pgmap v15969: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:48:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:19 smithi104 ceph-mon[97044]: pgmap v15969: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:48:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:19.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:19.153174+0000) 2022-09-10T12:48:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:19.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:19.597166+0000) 2022-09-10T12:48:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:20.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:20.153391+0000) 2022-09-10T12:48:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:20.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:20.597293+0000) 2022-09-10T12:48:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:21 smithi110 ceph-mon[97633]: pgmap v15970: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:48:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:21 smithi104 ceph-mon[97044]: pgmap v15970: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:48:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:21.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:21.153646+0000) 2022-09-10T12:48:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:21.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:21.597424+0000) 2022-09-10T12:48:22.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:22 smithi110 ceph-mon[97633]: pgmap v15971: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:48:22.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:22 smithi104 ceph-mon[97044]: pgmap v15971: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:48:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:22.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:22.153859+0000) 2022-09-10T12:48:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:22.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:22.597592+0000) 2022-09-10T12:48:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:23.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:23.154069+0000) 2022-09-10T12:48:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:23.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:23.597742+0000) 2022-09-10T12:48:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:24.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:24.154173+0000) 2022-09-10T12:48:24.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:24 smithi104 ceph-mon[97044]: pgmap v15972: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:48:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:24 smithi110 ceph-mon[97633]: pgmap v15972: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:48:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:24.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:24.597875+0000) 2022-09-10T12:48:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:25.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:25.154352+0000) 2022-09-10T12:48:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:25.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:25.598062+0000) 2022-09-10T12:48:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:26.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:26.154544+0000) 2022-09-10T12:48:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:26.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:26.598158+0000) 2022-09-10T12:48:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:27 smithi104 ceph-mon[97044]: pgmap v15973: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:48:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:27 smithi110 ceph-mon[97633]: pgmap v15973: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:48:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:27.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:27.154773+0000) 2022-09-10T12:48:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:27.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:27.598364+0000) 2022-09-10T12:48:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:28.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:28.155005+0000) 2022-09-10T12:48:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:28.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:28.598494+0000) 2022-09-10T12:48:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:29 smithi104 ceph-mon[97044]: pgmap v15974: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:29 smithi110 ceph-mon[97633]: pgmap v15974: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:29.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:29.155258+0000) 2022-09-10T12:48:29.940 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:29.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:29.598719+0000) 2022-09-10T12:48:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:30.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:30.155439+0000) 2022-09-10T12:48:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:30.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:30.598882+0000) 2022-09-10T12:48:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:48:30.742+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:48:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:31 smithi104 ceph-mon[97044]: pgmap v15975: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:48:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:31 smithi110 ceph-mon[97633]: pgmap v15975: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:48:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:31.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:31.155656+0000) 2022-09-10T12:48:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:31.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:31.599023+0000) 2022-09-10T12:48:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:32.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:32.155848+0000) 2022-09-10T12:48:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:32.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:32.599164+0000) 2022-09-10T12:48:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:33 smithi104 ceph-mon[97044]: pgmap v15976: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:33 smithi110 ceph-mon[97633]: pgmap v15976: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:33.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:33.156078+0000) 2022-09-10T12:48:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:33.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:33.599352+0000) 2022-09-10T12:48:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:34.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:34.156352+0000) 2022-09-10T12:48:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:34.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:34.599520+0000) 2022-09-10T12:48:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:35 smithi104 ceph-mon[97044]: pgmap v15977: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:35 smithi110 ceph-mon[97633]: pgmap v15977: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:35.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:35.156636+0000) 2022-09-10T12:48:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:35.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:35.599656+0000) 2022-09-10T12:48:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:36.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:36.156890+0000) 2022-09-10T12:48:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:36.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:36.599792+0000) 2022-09-10T12:48:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:37 smithi104 ceph-mon[97044]: pgmap v15978: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:48:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:37 smithi110 ceph-mon[97633]: pgmap v15978: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:48:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:37.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:37.157070+0000) 2022-09-10T12:48:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:37.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:37.599950+0000) 2022-09-10T12:48:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:38.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:38.157281+0000) 2022-09-10T12:48:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:38.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:38.600086+0000) 2022-09-10T12:48:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:39 smithi104 ceph-mon[97044]: pgmap v15979: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:39 smithi110 ceph-mon[97633]: pgmap v15979: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:39.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:39.157537+0000) 2022-09-10T12:48:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:39.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:39.600265+0000) 2022-09-10T12:48:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:40.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:40.157754+0000) 2022-09-10T12:48:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:40.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:40.600422+0000) 2022-09-10T12:48:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:41 smithi104 ceph-mon[97044]: pgmap v15980: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:48:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:41 smithi110 ceph-mon[97633]: pgmap v15980: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:48:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:41.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:41.157917+0000) 2022-09-10T12:48:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:41.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:41.600596+0000) 2022-09-10T12:48:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:42.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:42.158088+0000) 2022-09-10T12:48:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:42.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:42.600790+0000) 2022-09-10T12:48:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:43 smithi110 ceph-mon[97633]: pgmap v15981: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:43 smithi104 ceph-mon[97044]: pgmap v15981: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:43.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:43.158347+0000) 2022-09-10T12:48:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:43.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:43.600941+0000) 2022-09-10T12:48:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:44.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:44.158612+0000) 2022-09-10T12:48:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:44.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:44.601067+0000) 2022-09-10T12:48:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:45 smithi110 ceph-mon[97633]: pgmap v15982: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:45 smithi104 ceph-mon[97044]: pgmap v15982: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:45.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:45.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:45.158798+0000) 2022-09-10T12:48:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:45.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:45.601217+0000) 2022-09-10T12:48:46.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:48:45.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:48:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:46.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:46.158998+0000) 2022-09-10T12:48:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:46.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:46.601420+0000) 2022-09-10T12:48:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:47 smithi110 ceph-mon[97633]: pgmap v15983: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:48:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:48:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:47 smithi104 ceph-mon[97044]: pgmap v15983: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:48:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:48:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:48:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:47.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:47.159240+0000) 2022-09-10T12:48:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:47.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:47.601634+0000) 2022-09-10T12:48:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:48.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:48.159432+0000) 2022-09-10T12:48:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:48.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:48.601800+0000) 2022-09-10T12:48:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:49 smithi110 ceph-mon[97633]: pgmap v15984: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:49 smithi104 ceph-mon[97044]: pgmap v15984: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:49.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:49.159715+0000) 2022-09-10T12:48:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:49.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:49.601997+0000) 2022-09-10T12:48:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:50.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:50.159945+0000) 2022-09-10T12:48:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:50.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:50.602185+0000) 2022-09-10T12:48:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:51 smithi110 ceph-mon[97633]: pgmap v15985: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:48:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:51 smithi104 ceph-mon[97044]: pgmap v15985: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:48:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:51.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:51.160194+0000) 2022-09-10T12:48:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:51.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:51.602365+0000) 2022-09-10T12:48:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:52.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:52.160390+0000) 2022-09-10T12:48:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:52.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:52.602523+0000) 2022-09-10T12:48:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:53 smithi110 ceph-mon[97633]: pgmap v15986: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:53 smithi104 ceph-mon[97044]: pgmap v15986: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:48:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:53.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:53.160600+0000) 2022-09-10T12:48:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:53.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:53.602700+0000) 2022-09-10T12:48:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:54.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:54.160856+0000) 2022-09-10T12:48:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:54.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:54.602950+0000) 2022-09-10T12:48:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:55 smithi110 ceph-mon[97633]: pgmap v15987: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:55 smithi104 ceph-mon[97044]: pgmap v15987: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:55.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:55.161076+0000) 2022-09-10T12:48:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:55.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:55.603228+0000) 2022-09-10T12:48:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:56.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:56.161315+0000) 2022-09-10T12:48:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:56.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:56.603363+0000) 2022-09-10T12:48:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:57 smithi110 ceph-mon[97633]: pgmap v15988: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:48:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:57 smithi104 ceph-mon[97044]: pgmap v15988: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:48:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:57.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:57.161567+0000) 2022-09-10T12:48:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:57.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:57.603547+0000) 2022-09-10T12:48:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:58.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:58.161801+0000) 2022-09-10T12:48:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:58.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:58.603696+0000) 2022-09-10T12:48:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:48:59 smithi110 ceph-mon[97633]: pgmap v15989: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:48:59 smithi104 ceph-mon[97044]: pgmap v15989: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:48:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:48:59.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:59.162016+0000) 2022-09-10T12:49:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:48:59.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:48:59.603918+0000) 2022-09-10T12:49:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:00.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:00.162169+0000) 2022-09-10T12:49:01.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:49:00.771+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:49:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:00.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:00.604048+0000) 2022-09-10T12:49:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:01 smithi110 ceph-mon[97633]: pgmap v15990: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:01 smithi104 ceph-mon[97044]: pgmap v15990: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:01.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:01.162303+0000) 2022-09-10T12:49:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:01.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:01.604239+0000) 2022-09-10T12:49:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:02.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:02.162520+0000) 2022-09-10T12:49:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:02.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:02.604453+0000) 2022-09-10T12:49:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:03 smithi110 ceph-mon[97633]: pgmap v15991: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:03 smithi104 ceph-mon[97044]: pgmap v15991: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:03.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:03.162778+0000) 2022-09-10T12:49:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:03.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:03.604607+0000) 2022-09-10T12:49:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:04.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:04.163038+0000) 2022-09-10T12:49:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:04.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:04.604839+0000) 2022-09-10T12:49:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:05 smithi110 ceph-mon[97633]: pgmap v15992: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:05 smithi104 ceph-mon[97044]: pgmap v15992: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:05.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:05.163344+0000) 2022-09-10T12:49:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:05.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:05.604980+0000) 2022-09-10T12:49:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:06.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:06.163633+0000) 2022-09-10T12:49:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:06.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:06.605236+0000) 2022-09-10T12:49:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:07 smithi110 ceph-mon[97633]: pgmap v15993: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:49:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:07 smithi104 ceph-mon[97044]: pgmap v15993: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:49:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:07.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:07.163883+0000) 2022-09-10T12:49:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:07.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:07.605436+0000) 2022-09-10T12:49:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:08.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:08.164182+0000) 2022-09-10T12:49:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:08.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:08.605607+0000) 2022-09-10T12:49:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:09 smithi110 ceph-mon[97633]: pgmap v15994: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:49:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:09 smithi110 ceph-mon[97633]: osdmap e1080: 12 total, 12 up, 12 in 2022-09-10T12:49:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:09 smithi104 ceph-mon[97044]: pgmap v15994: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:49:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:09 smithi104 ceph-mon[97044]: osdmap e1080: 12 total, 12 up, 12 in 2022-09-10T12:49:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:09.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:09.164396+0000) 2022-09-10T12:49:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:09.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:09.605789+0000) 2022-09-10T12:49:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:10.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:10.164600+0000) 2022-09-10T12:49:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:10.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:10.606008+0000) 2022-09-10T12:49:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:11 smithi110 ceph-mon[97633]: pgmap v15996: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:49:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:11 smithi104 ceph-mon[97044]: pgmap v15996: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:49:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:11.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:11.164811+0000) 2022-09-10T12:49:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:11.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:11.606184+0000) 2022-09-10T12:49:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:12 smithi110 ceph-mon[97633]: pgmap v15997: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:49:12.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:12 smithi104 ceph-mon[97044]: pgmap v15997: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:49:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:12.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:12.165013+0000) 2022-09-10T12:49:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:12.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:12.606368+0000) 2022-09-10T12:49:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:13.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:13.165189+0000) 2022-09-10T12:49:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:13.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:13.606573+0000) 2022-09-10T12:49:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:14.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:14.165418+0000) 2022-09-10T12:49:14.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:14 smithi104 ceph-mon[97044]: pgmap v15998: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:49:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:14 smithi110 ceph-mon[97633]: pgmap v15998: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:49:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:14.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:14.606714+0000) 2022-09-10T12:49:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:15.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:15.165567+0000) 2022-09-10T12:49:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:15.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:15.606884+0000) 2022-09-10T12:49:16.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:49:15.787+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:49:16.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:16 smithi104 ceph-mon[97044]: osdmap e1081: 12 total, 12 up, 12 in 2022-09-10T12:49:16.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:16 smithi110 ceph-mon[97633]: osdmap e1081: 12 total, 12 up, 12 in 2022-09-10T12:49:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:16.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:16.165798+0000) 2022-09-10T12:49:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:16.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:16.607034+0000) 2022-09-10T12:49:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:17 smithi104 ceph-mon[97044]: pgmap v16000: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:49:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:17 smithi110 ceph-mon[97633]: pgmap v16000: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:49:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:17.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:17.166018+0000) 2022-09-10T12:49:18.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:17.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:17.607249+0000) 2022-09-10T12:49:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:18.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:18.166226+0000) 2022-09-10T12:49:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:18.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:18.607388+0000) 2022-09-10T12:49:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:19 smithi110 ceph-mon[97633]: pgmap v16001: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:49:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:19 smithi104 ceph-mon[97044]: pgmap v16001: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T12:49:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:19.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:19.166449+0000) 2022-09-10T12:49:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:19.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:19.607551+0000) 2022-09-10T12:49:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:20.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:20.166712+0000) 2022-09-10T12:49:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:20.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:20.607679+0000) 2022-09-10T12:49:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:21 smithi110 ceph-mon[97633]: pgmap v16002: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:49:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:21 smithi104 ceph-mon[97044]: pgmap v16002: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:49:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:21.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:21.166930+0000) 2022-09-10T12:49:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:21.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:21.607900+0000) 2022-09-10T12:49:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:22.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:22.167176+0000) 2022-09-10T12:49:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:22.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:22.608089+0000) 2022-09-10T12:49:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:23 smithi110 ceph-mon[97633]: pgmap v16003: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:49:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:23 smithi104 ceph-mon[97044]: pgmap v16003: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:49:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:23.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:23.167377+0000) 2022-09-10T12:49:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:23.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:23.608297+0000) 2022-09-10T12:49:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:24.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:24.167593+0000) 2022-09-10T12:49:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:24.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:24.608441+0000) 2022-09-10T12:49:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:25 smithi110 ceph-mon[97633]: pgmap v16004: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:49:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:25 smithi104 ceph-mon[97044]: pgmap v16004: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:49:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:25.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:25.167854+0000) 2022-09-10T12:49:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:25.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:25.608614+0000) 2022-09-10T12:49:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:26.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:26.168124+0000) 2022-09-10T12:49:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:26.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:26.608812+0000) 2022-09-10T12:49:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:27 smithi110 ceph-mon[97633]: pgmap v16005: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:49:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:27 smithi104 ceph-mon[97044]: pgmap v16005: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:49:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:27.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:27.168315+0000) 2022-09-10T12:49:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:27.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:27.609051+0000) 2022-09-10T12:49:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:28.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:28.168520+0000) 2022-09-10T12:49:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:28.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:28.609250+0000) 2022-09-10T12:49:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:29 smithi110 ceph-mon[97633]: pgmap v16006: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:29 smithi104 ceph-mon[97044]: pgmap v16006: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:29.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:29.168786+0000) 2022-09-10T12:49:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:29.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:29.609469+0000) 2022-09-10T12:49:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:30.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:30.169005+0000) 2022-09-10T12:49:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:30.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:30.609673+0000) 2022-09-10T12:49:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:49:30.788+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:49:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:31 smithi110 ceph-mon[97633]: pgmap v16007: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:31 smithi104 ceph-mon[97044]: pgmap v16007: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:31.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:31.169228+0000) 2022-09-10T12:49:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:31.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:31.609866+0000) 2022-09-10T12:49:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:32.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:32.169465+0000) 2022-09-10T12:49:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:32.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:32.610041+0000) 2022-09-10T12:49:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:33 smithi110 ceph-mon[97633]: pgmap v16008: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:33 smithi104 ceph-mon[97044]: pgmap v16008: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:33.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:33.169643+0000) 2022-09-10T12:49:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:33.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:33.610205+0000) 2022-09-10T12:49:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:34.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:34.169855+0000) 2022-09-10T12:49:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:34.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:34.610372+0000) 2022-09-10T12:49:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:35 smithi110 ceph-mon[97633]: pgmap v16009: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:35 smithi104 ceph-mon[97044]: pgmap v16009: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:35.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:35.170065+0000) 2022-09-10T12:49:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:35.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:35.610599+0000) 2022-09-10T12:49:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:36.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:36.170312+0000) 2022-09-10T12:49:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:36.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:36.610753+0000) 2022-09-10T12:49:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:37 smithi110 ceph-mon[97633]: pgmap v16010: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:49:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:37 smithi104 ceph-mon[97044]: pgmap v16010: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:49:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:37.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:37.170559+0000) 2022-09-10T12:49:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:37.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:37.610902+0000) 2022-09-10T12:49:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:38.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:38.170758+0000) 2022-09-10T12:49:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:38.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:38.611136+0000) 2022-09-10T12:49:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:39 smithi110 ceph-mon[97633]: pgmap v16011: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:39 smithi104 ceph-mon[97044]: pgmap v16011: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:39.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:39.171027+0000) 2022-09-10T12:49:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:39.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:39.611346+0000) 2022-09-10T12:49:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:40.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:40.171256+0000) 2022-09-10T12:49:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:40.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:40.611485+0000) 2022-09-10T12:49:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:41 smithi110 ceph-mon[97633]: pgmap v16012: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:41 smithi104 ceph-mon[97044]: pgmap v16012: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:41.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:41.171493+0000) 2022-09-10T12:49:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:41.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:41.611605+0000) 2022-09-10T12:49:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:42.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:42.171742+0000) 2022-09-10T12:49:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:42.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:42.611839+0000) 2022-09-10T12:49:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:43 smithi110 ceph-mon[97633]: pgmap v16013: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:43 smithi104 ceph-mon[97044]: pgmap v16013: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:43.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:43.171936+0000) 2022-09-10T12:49:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:43.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:43.612080+0000) 2022-09-10T12:49:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:44.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:44.172201+0000) 2022-09-10T12:49:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:44.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:44.612244+0000) 2022-09-10T12:49:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:45 smithi110 ceph-mon[97633]: pgmap v16014: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:45 smithi104 ceph-mon[97044]: pgmap v16014: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:45.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:45.172489+0000) 2022-09-10T12:49:46.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:49:45.801+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:49:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:45.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:45.612405+0000) 2022-09-10T12:49:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:49:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:49:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:49:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:49:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:49:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:49:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:49:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:49:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:46.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:46.172669+0000) 2022-09-10T12:49:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:46.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:46.612664+0000) 2022-09-10T12:49:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:47 smithi110 ceph-mon[97633]: pgmap v16015: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:49:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:47 smithi104 ceph-mon[97044]: pgmap v16015: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:49:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:47.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:47.172900+0000) 2022-09-10T12:49:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:47.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:47.612810+0000) 2022-09-10T12:49:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:48.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:48.173173+0000) 2022-09-10T12:49:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:48.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:48.612982+0000) 2022-09-10T12:49:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:49 smithi110 ceph-mon[97633]: pgmap v16016: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:49 smithi104 ceph-mon[97044]: pgmap v16016: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:49.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:49.173305+0000) 2022-09-10T12:49:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:49.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:49.613149+0000) 2022-09-10T12:49:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:50.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:50.173520+0000) 2022-09-10T12:49:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:50.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:50.613313+0000) 2022-09-10T12:49:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:51 smithi110 ceph-mon[97633]: pgmap v16017: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:51 smithi104 ceph-mon[97044]: pgmap v16017: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:49:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:51.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:51.173720+0000) 2022-09-10T12:49:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:51.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:51.613455+0000) 2022-09-10T12:49:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:52.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:52.173914+0000) 2022-09-10T12:49:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:52.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:52.613656+0000) 2022-09-10T12:49:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:53 smithi110 ceph-mon[97633]: pgmap v16018: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:53 smithi104 ceph-mon[97044]: pgmap v16018: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:49:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:53.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:53.174062+0000) 2022-09-10T12:49:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:53.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:53.613882+0000) 2022-09-10T12:49:54.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:54 smithi110 ceph-mon[97633]: pgmap v16019: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:54.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:54 smithi104 ceph-mon[97044]: pgmap v16019: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:54.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:54.174275+0000) 2022-09-10T12:49:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:54.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:54.614049+0000) 2022-09-10T12:49:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:55.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:55.174513+0000) 2022-09-10T12:49:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:55.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:55.614190+0000) 2022-09-10T12:49:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:56.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:56.174704+0000) 2022-09-10T12:49:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:56.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:56.614376+0000) 2022-09-10T12:49:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:57 smithi110 ceph-mon[97633]: pgmap v16020: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:49:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:57 smithi104 ceph-mon[97044]: pgmap v16020: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:49:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:57.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:57.174941+0000) 2022-09-10T12:49:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:57.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:57.614538+0000) 2022-09-10T12:49:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:58.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:58.175174+0000) 2022-09-10T12:49:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:58.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:58.614808+0000) 2022-09-10T12:49:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:49:59 smithi110 ceph-mon[97633]: pgmap v16021: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:49:59 smithi104 ceph-mon[97044]: pgmap v16021: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:49:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:49:59.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:59.175387+0000) 2022-09-10T12:50:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:49:59.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:49:59.615063+0000) 2022-09-10T12:50:00.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:50:00.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T12:50:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:00.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:00.175702+0000) 2022-09-10T12:50:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:00.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:00.615290+0000) 2022-09-10T12:50:01.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:50:00.817+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:50:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:01 smithi110 ceph-mon[97633]: pgmap v16022: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T12:50:01.466 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:01 smithi104 ceph-mon[97044]: pgmap v16022: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2022-09-10T12:50:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:01.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:01.175857+0000) 2022-09-10T12:50:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:01.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:01.615473+0000) 2022-09-10T12:50:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:02.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:02.176042+0000) 2022-09-10T12:50:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:02.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:02.615609+0000) 2022-09-10T12:50:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:03 smithi110 ceph-mon[97633]: pgmap v16023: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-09-10T12:50:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:03 smithi104 ceph-mon[97044]: pgmap v16023: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-09-10T12:50:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:03.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:03.176275+0000) 2022-09-10T12:50:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:03.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:03.615752+0000) 2022-09-10T12:50:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:04.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:04.176526+0000) 2022-09-10T12:50:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:04.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:04.615893+0000) 2022-09-10T12:50:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:05 smithi110 ceph-mon[97633]: pgmap v16024: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:50:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:05 smithi104 ceph-mon[97044]: pgmap v16024: 129 pgs: 129 active+clean; 9.5 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:50:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:05.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:05.176788+0000) 2022-09-10T12:50:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:05.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:05.616030+0000) 2022-09-10T12:50:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:06.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:06.177014+0000) 2022-09-10T12:50:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:06.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:06.616276+0000) 2022-09-10T12:50:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:07 smithi110 ceph-mon[97633]: pgmap v16025: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:50:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:07 smithi104 ceph-mon[97044]: pgmap v16025: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:50:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:07.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:07.177159+0000) 2022-09-10T12:50:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:07.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:07.616457+0000) 2022-09-10T12:50:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:08.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:08.177334+0000) 2022-09-10T12:50:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:08.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:08.616698+0000) 2022-09-10T12:50:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:09 smithi110 ceph-mon[97633]: pgmap v16026: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:50:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:09 smithi110 ceph-mon[97633]: osdmap e1082: 12 total, 12 up, 12 in 2022-09-10T12:50:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:09 smithi104 ceph-mon[97044]: pgmap v16026: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:50:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:09 smithi104 ceph-mon[97044]: osdmap e1082: 12 total, 12 up, 12 in 2022-09-10T12:50:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:09.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:09.177560+0000) 2022-09-10T12:50:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:09.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:09.616929+0000) 2022-09-10T12:50:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:10.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:10.177806+0000) 2022-09-10T12:50:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:10.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:10.617065+0000) 2022-09-10T12:50:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:11 smithi110 ceph-mon[97633]: pgmap v16028: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:50:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:11 smithi104 ceph-mon[97044]: pgmap v16028: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:50:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:11.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:11.178051+0000) 2022-09-10T12:50:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:11.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:11.617194+0000) 2022-09-10T12:50:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:12.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:12.178298+0000) 2022-09-10T12:50:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:12.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:12.617385+0000) 2022-09-10T12:50:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:13 smithi110 ceph-mon[97633]: pgmap v16029: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:50:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:13 smithi104 ceph-mon[97044]: pgmap v16029: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:50:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:13.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:13.178443+0000) 2022-09-10T12:50:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:13.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:13.617533+0000) 2022-09-10T12:50:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:14.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:14.178703+0000) 2022-09-10T12:50:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:14.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:14.617723+0000) 2022-09-10T12:50:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:15 smithi110 ceph-mon[97633]: pgmap v16030: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:50:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:15 smithi110 ceph-mon[97633]: osdmap e1083: 12 total, 12 up, 12 in 2022-09-10T12:50:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:15 smithi104 ceph-mon[97044]: pgmap v16030: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:50:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:15 smithi104 ceph-mon[97044]: osdmap e1083: 12 total, 12 up, 12 in 2022-09-10T12:50:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:15.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:15.178991+0000) 2022-09-10T12:50:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:15.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:15.617919+0000) 2022-09-10T12:50:16.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:50:15.833+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:50:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:16.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:16.179197+0000) 2022-09-10T12:50:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:16.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:16.618119+0000) 2022-09-10T12:50:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:17 smithi110 ceph-mon[97633]: pgmap v16032: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:50:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:17 smithi104 ceph-mon[97044]: pgmap v16032: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:50:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:17.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:17.179358+0000) 2022-09-10T12:50:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:17.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:17.618308+0000) 2022-09-10T12:50:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:18.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:18.179560+0000) 2022-09-10T12:50:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:18.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:18.618483+0000) 2022-09-10T12:50:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:19 smithi110 ceph-mon[97633]: pgmap v16033: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:50:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:19 smithi104 ceph-mon[97044]: pgmap v16033: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:50:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:19.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:19.179780+0000) 2022-09-10T12:50:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:19.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:19.618636+0000) 2022-09-10T12:50:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:20.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:20.179965+0000) 2022-09-10T12:50:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:20.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:20.618812+0000) 2022-09-10T12:50:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:21 smithi110 ceph-mon[97633]: pgmap v16034: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:50:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:21 smithi104 ceph-mon[97044]: pgmap v16034: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:50:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:21.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:21.180190+0000) 2022-09-10T12:50:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:21.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:21.618986+0000) 2022-09-10T12:50:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:22.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:22.180406+0000) 2022-09-10T12:50:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:22.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:22.619178+0000) 2022-09-10T12:50:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:23 smithi110 ceph-mon[97633]: pgmap v16035: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:50:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:23 smithi104 ceph-mon[97044]: pgmap v16035: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:50:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:23.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:23.180653+0000) 2022-09-10T12:50:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:23.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:23.619426+0000) 2022-09-10T12:50:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:24.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:24.180836+0000) 2022-09-10T12:50:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:24.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:24.619568+0000) 2022-09-10T12:50:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:25 smithi110 ceph-mon[97633]: pgmap v16036: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:50:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:25 smithi104 ceph-mon[97044]: pgmap v16036: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:50:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:25.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:25.181125+0000) 2022-09-10T12:50:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:25.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:25.619706+0000) 2022-09-10T12:50:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:26.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:26.181349+0000) 2022-09-10T12:50:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:26.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:26.619899+0000) 2022-09-10T12:50:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:27 smithi110 ceph-mon[97633]: pgmap v16037: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:50:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:27 smithi104 ceph-mon[97044]: pgmap v16037: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:50:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:27.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:27.181544+0000) 2022-09-10T12:50:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:27.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:27.620060+0000) 2022-09-10T12:50:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:28.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:28.181748+0000) 2022-09-10T12:50:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:28.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:28.620193+0000) 2022-09-10T12:50:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:29 smithi110 ceph-mon[97633]: pgmap v16038: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:29 smithi104 ceph-mon[97044]: pgmap v16038: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:29.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:29.181938+0000) 2022-09-10T12:50:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:29.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:29.620350+0000) 2022-09-10T12:50:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:30.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:30.182221+0000) 2022-09-10T12:50:31.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:50:30.838+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:50:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:30.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:30.620482+0000) 2022-09-10T12:50:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:31 smithi110 ceph-mon[97633]: pgmap v16039: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:50:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:31 smithi104 ceph-mon[97044]: pgmap v16039: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:50:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:31.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:31.182398+0000) 2022-09-10T12:50:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:31.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:31.620629+0000) 2022-09-10T12:50:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:32.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:32.182605+0000) 2022-09-10T12:50:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:32.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:32.620778+0000) 2022-09-10T12:50:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:33 smithi110 ceph-mon[97633]: pgmap v16040: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:50:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:33 smithi104 ceph-mon[97044]: pgmap v16040: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:50:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:33.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:33.182859+0000) 2022-09-10T12:50:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:33.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:33.620930+0000) 2022-09-10T12:50:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:34.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:34.183058+0000) 2022-09-10T12:50:35.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:34.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:34.621073+0000) 2022-09-10T12:50:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:35 smithi110 ceph-mon[97633]: pgmap v16041: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:35 smithi104 ceph-mon[97044]: pgmap v16041: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:35.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:35.183283+0000) 2022-09-10T12:50:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:35.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:35.621180+0000) 2022-09-10T12:50:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:36.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:36.183499+0000) 2022-09-10T12:50:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:36.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:36.621328+0000) 2022-09-10T12:50:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:37 smithi110 ceph-mon[97633]: pgmap v16042: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:50:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:37 smithi104 ceph-mon[97044]: pgmap v16042: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:50:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:37.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:37.183747+0000) 2022-09-10T12:50:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:37.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:37.621517+0000) 2022-09-10T12:50:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:38.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:38.183963+0000) 2022-09-10T12:50:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:38.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:38.621645+0000) 2022-09-10T12:50:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:39 smithi110 ceph-mon[97633]: pgmap v16043: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:39 smithi104 ceph-mon[97044]: pgmap v16043: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:39.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:39.184206+0000) 2022-09-10T12:50:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:39.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:39.621817+0000) 2022-09-10T12:50:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:40.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:40.184375+0000) 2022-09-10T12:50:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:40.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:40.622002+0000) 2022-09-10T12:50:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:41 smithi110 ceph-mon[97633]: pgmap v16044: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:50:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:41 smithi104 ceph-mon[97044]: pgmap v16044: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:50:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:41.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:41.184622+0000) 2022-09-10T12:50:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:41.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:41.622197+0000) 2022-09-10T12:50:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:42.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:42.184815+0000) 2022-09-10T12:50:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:42.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:42.622407+0000) 2022-09-10T12:50:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:43 smithi110 ceph-mon[97633]: pgmap v16045: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:50:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:43 smithi104 ceph-mon[97044]: pgmap v16045: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:50:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:43.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:43.185019+0000) 2022-09-10T12:50:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:43.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:43.622589+0000) 2022-09-10T12:50:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:44.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:44.185161+0000) 2022-09-10T12:50:45.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:44.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:44.622802+0000) 2022-09-10T12:50:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:45 smithi110 ceph-mon[97633]: pgmap v16046: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:45 smithi104 ceph-mon[97044]: pgmap v16046: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:45.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:45.185378+0000) 2022-09-10T12:50:46.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:50:45.854+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:50:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:45.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:45.623035+0000) 2022-09-10T12:50:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:50:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:50:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:50:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:50:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:50:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:50:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:50:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:46.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:46.185586+0000) 2022-09-10T12:50:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:46.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:46.623221+0000) 2022-09-10T12:50:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:47 smithi110 ceph-mon[97633]: pgmap v16047: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:50:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:47 smithi104 ceph-mon[97044]: pgmap v16047: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:50:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:47.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:47.185862+0000) 2022-09-10T12:50:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:47.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:47.623424+0000) 2022-09-10T12:50:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:48.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:48.186084+0000) 2022-09-10T12:50:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:48.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:48.623627+0000) 2022-09-10T12:50:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:49 smithi110 ceph-mon[97633]: pgmap v16048: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:49 smithi104 ceph-mon[97044]: pgmap v16048: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:49.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:49.186317+0000) 2022-09-10T12:50:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:49.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:49.623837+0000) 2022-09-10T12:50:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:50.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:50.186549+0000) 2022-09-10T12:50:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:50.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:50.623965+0000) 2022-09-10T12:50:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:51 smithi110 ceph-mon[97633]: pgmap v16049: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:50:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:51 smithi104 ceph-mon[97044]: pgmap v16049: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2022-09-10T12:50:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:51.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:51.186767+0000) 2022-09-10T12:50:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:51.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:51.624198+0000) 2022-09-10T12:50:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:52.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:52.187015+0000) 2022-09-10T12:50:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:52.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:52.624432+0000) 2022-09-10T12:50:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:53 smithi110 ceph-mon[97633]: pgmap v16050: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:50:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:53 smithi104 ceph-mon[97044]: pgmap v16050: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:50:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:53.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:53.187229+0000) 2022-09-10T12:50:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:53.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:53.624560+0000) 2022-09-10T12:50:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:54.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:54.187484+0000) 2022-09-10T12:50:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:54.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:54.624719+0000) 2022-09-10T12:50:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:55 smithi110 ceph-mon[97633]: pgmap v16051: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:55 smithi104 ceph-mon[97044]: pgmap v16051: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:55.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:55.187740+0000) 2022-09-10T12:50:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:55.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:55.624859+0000) 2022-09-10T12:50:56.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:56 smithi110 ceph-mon[97633]: pgmap v16052: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:50:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:56 smithi104 ceph-mon[97044]: pgmap v16052: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:50:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:56.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:56.187923+0000) 2022-09-10T12:50:57.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:56.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:56.625084+0000) 2022-09-10T12:50:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:57.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:57.188080+0000) 2022-09-10T12:50:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:57.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:57.625293+0000) 2022-09-10T12:50:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:58.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:58.188309+0000) 2022-09-10T12:50:58.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:50:58 smithi104 ceph-mon[97044]: pgmap v16053: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:50:58 smithi110 ceph-mon[97633]: pgmap v16053: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:50:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:58.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:58.625435+0000) 2022-09-10T12:50:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:50:59.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:59.188525+0000) 2022-09-10T12:51:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:50:59.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:50:59.625669+0000) 2022-09-10T12:51:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:00.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:00.188775+0000) 2022-09-10T12:51:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:00.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:00.625802+0000) 2022-09-10T12:51:01.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:51:00.868+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:51:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:01 smithi110 ceph-mon[97633]: pgmap v16054: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:01 smithi104 ceph-mon[97044]: pgmap v16054: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:01.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:01.189024+0000) 2022-09-10T12:51:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:01.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:01.625984+0000) 2022-09-10T12:51:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:02.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:02.189256+0000) 2022-09-10T12:51:03.027 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:02.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:02.626204+0000) 2022-09-10T12:51:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:03 smithi110 ceph-mon[97633]: pgmap v16055: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:51:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:03 smithi104 ceph-mon[97044]: pgmap v16055: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2022-09-10T12:51:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:03.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:03.189532+0000) 2022-09-10T12:51:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:03.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:03.626460+0000) 2022-09-10T12:51:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:04.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:04.189732+0000) 2022-09-10T12:51:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:04.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:04.626682+0000) 2022-09-10T12:51:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:05 smithi110 ceph-mon[97633]: pgmap v16056: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:05 smithi104 ceph-mon[97044]: pgmap v16056: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:05.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:05.189912+0000) 2022-09-10T12:51:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:05.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:05.626846+0000) 2022-09-10T12:51:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:06.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:06.190164+0000) 2022-09-10T12:51:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:06.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:06.626989+0000) 2022-09-10T12:51:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:07 smithi110 ceph-mon[97633]: pgmap v16057: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:51:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:07 smithi104 ceph-mon[97044]: pgmap v16057: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:51:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:07.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:07.190384+0000) 2022-09-10T12:51:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:07.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:07.627208+0000) 2022-09-10T12:51:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:08.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:08.190648+0000) 2022-09-10T12:51:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:08.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:08.627386+0000) 2022-09-10T12:51:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:09 smithi110 ceph-mon[97633]: pgmap v16058: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:51:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:09 smithi110 ceph-mon[97633]: osdmap e1084: 12 total, 12 up, 12 in 2022-09-10T12:51:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:09 smithi104 ceph-mon[97044]: pgmap v16058: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:51:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:09 smithi104 ceph-mon[97044]: osdmap e1084: 12 total, 12 up, 12 in 2022-09-10T12:51:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:09.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:09.190901+0000) 2022-09-10T12:51:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:09.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:09.627551+0000) 2022-09-10T12:51:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:10.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:10.191169+0000) 2022-09-10T12:51:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:10.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:10.627717+0000) 2022-09-10T12:51:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:11 smithi110 ceph-mon[97633]: pgmap v16060: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:51:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:11 smithi104 ceph-mon[97044]: pgmap v16060: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:51:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:11.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:11.191420+0000) 2022-09-10T12:51:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:11.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:11.627895+0000) 2022-09-10T12:51:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:12.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:12.191650+0000) 2022-09-10T12:51:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:12.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:12.628076+0000) 2022-09-10T12:51:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:13 smithi110 ceph-mon[97633]: pgmap v16061: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:51:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:13 smithi104 ceph-mon[97044]: pgmap v16061: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:51:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:13.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:13.191881+0000) 2022-09-10T12:51:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:13.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:13.628168+0000) 2022-09-10T12:51:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:14.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:14.192090+0000) 2022-09-10T12:51:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:14.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:14.628310+0000) 2022-09-10T12:51:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:15 smithi110 ceph-mon[97633]: pgmap v16062: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:51:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:15 smithi110 ceph-mon[97633]: osdmap e1085: 12 total, 12 up, 12 in 2022-09-10T12:51:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:15 smithi104 ceph-mon[97044]: pgmap v16062: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T12:51:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:15 smithi104 ceph-mon[97044]: osdmap e1085: 12 total, 12 up, 12 in 2022-09-10T12:51:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:15.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:15.192310+0000) 2022-09-10T12:51:16.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:51:15.870+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:51:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:15.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:15.628528+0000) 2022-09-10T12:51:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:16.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:16.192561+0000) 2022-09-10T12:51:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:16.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:16.628761+0000) 2022-09-10T12:51:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:17 smithi110 ceph-mon[97633]: pgmap v16064: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:51:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:17 smithi104 ceph-mon[97044]: pgmap v16064: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:51:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:17.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:17.192778+0000) 2022-09-10T12:51:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:17.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:17.629001+0000) 2022-09-10T12:51:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:18.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:18.192945+0000) 2022-09-10T12:51:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:18.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:18.629186+0000) 2022-09-10T12:51:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:19 smithi110 ceph-mon[97633]: pgmap v16065: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:51:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:19 smithi104 ceph-mon[97044]: pgmap v16065: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:51:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:19.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:19.193195+0000) 2022-09-10T12:51:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:19.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:19.629388+0000) 2022-09-10T12:51:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:20.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:20.193406+0000) 2022-09-10T12:51:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:20.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:20.629611+0000) 2022-09-10T12:51:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:21 smithi110 ceph-mon[97633]: pgmap v16066: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:51:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:21 smithi104 ceph-mon[97044]: pgmap v16066: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:51:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:21.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:21.193645+0000) 2022-09-10T12:51:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:21.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:21.629836+0000) 2022-09-10T12:51:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:22.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:22.193892+0000) 2022-09-10T12:51:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:22.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:22.630081+0000) 2022-09-10T12:51:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:23 smithi110 ceph-mon[97633]: pgmap v16067: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:51:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:23 smithi104 ceph-mon[97044]: pgmap v16067: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:51:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:23.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:23.194165+0000) 2022-09-10T12:51:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:23.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:23.630258+0000) 2022-09-10T12:51:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:24.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:24.194364+0000) 2022-09-10T12:51:25.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:24.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:24.630418+0000) 2022-09-10T12:51:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:25 smithi110 ceph-mon[97633]: pgmap v16068: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:51:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:25 smithi104 ceph-mon[97044]: pgmap v16068: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:51:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:25.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:25.194504+0000) 2022-09-10T12:51:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:25.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:25.630613+0000) 2022-09-10T12:51:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:26.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:26.194726+0000) 2022-09-10T12:51:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:26.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:26.630750+0000) 2022-09-10T12:51:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:27 smithi110 ceph-mon[97633]: pgmap v16069: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:51:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:27 smithi104 ceph-mon[97044]: pgmap v16069: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:51:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:27.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:27.194966+0000) 2022-09-10T12:51:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:27.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:27.630896+0000) 2022-09-10T12:51:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:28.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:28.195215+0000) 2022-09-10T12:51:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:28.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:28.631070+0000) 2022-09-10T12:51:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:29 smithi110 ceph-mon[97633]: pgmap v16070: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:29 smithi104 ceph-mon[97044]: pgmap v16070: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:29.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:29.195457+0000) 2022-09-10T12:51:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:29.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:29.631325+0000) 2022-09-10T12:51:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:30.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:30.195733+0000) 2022-09-10T12:51:30.884 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:30.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:30.631481+0000) 2022-09-10T12:51:31.196 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:51:30.883+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:51:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:31 smithi110 ceph-mon[97633]: pgmap v16071: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:31 smithi104 ceph-mon[97044]: pgmap v16071: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:31.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:31.195974+0000) 2022-09-10T12:51:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:31.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:31.631636+0000) 2022-09-10T12:51:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:32.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:32.196238+0000) 2022-09-10T12:51:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:32.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:32.631804+0000) 2022-09-10T12:51:33.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:33 smithi110 ceph-mon[97633]: pgmap v16072: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:33 smithi104 ceph-mon[97044]: pgmap v16072: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:33.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:33.196435+0000) 2022-09-10T12:51:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:33.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:33.631998+0000) 2022-09-10T12:51:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:34.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:34.196666+0000) 2022-09-10T12:51:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:34.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:34.632238+0000) 2022-09-10T12:51:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:35 smithi110 ceph-mon[97633]: pgmap v16073: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:35 smithi104 ceph-mon[97044]: pgmap v16073: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:35.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:35.196882+0000) 2022-09-10T12:51:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:35.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:35.632406+0000) 2022-09-10T12:51:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:36.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:36.197154+0000) 2022-09-10T12:51:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:36.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:36.632606+0000) 2022-09-10T12:51:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:37 smithi110 ceph-mon[97633]: pgmap v16074: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:37 smithi104 ceph-mon[97044]: pgmap v16074: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:37.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:37.197354+0000) 2022-09-10T12:51:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:37.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:37.632767+0000) 2022-09-10T12:51:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:38.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:38.197553+0000) 2022-09-10T12:51:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:38.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:38.632945+0000) 2022-09-10T12:51:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:39 smithi110 ceph-mon[97633]: pgmap v16075: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:39 smithi104 ceph-mon[97044]: pgmap v16075: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:39.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:39.197755+0000) 2022-09-10T12:51:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:39.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:39.633233+0000) 2022-09-10T12:51:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:40.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:40.198028+0000) 2022-09-10T12:51:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:40.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:40.633399+0000) 2022-09-10T12:51:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:41 smithi110 ceph-mon[97633]: pgmap v16076: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:41.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:41 smithi104 ceph-mon[97044]: pgmap v16076: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:41.568 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:41.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:41.198236+0000) 2022-09-10T12:51:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:41.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:41.633536+0000) 2022-09-10T12:51:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:42.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:42.198508+0000) 2022-09-10T12:51:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:42.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:42.633701+0000) 2022-09-10T12:51:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:43 smithi110 ceph-mon[97633]: pgmap v16077: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:43 smithi104 ceph-mon[97044]: pgmap v16077: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:43.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:43.198749+0000) 2022-09-10T12:51:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:43.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:43.633869+0000) 2022-09-10T12:51:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:44.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:44.198951+0000) 2022-09-10T12:51:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:44.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:44.634083+0000) 2022-09-10T12:51:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:45 smithi110 ceph-mon[97633]: pgmap v16078: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:45 smithi104 ceph-mon[97044]: pgmap v16078: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:45.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:45.199166+0000) 2022-09-10T12:51:45.893 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:45.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:45.634241+0000) 2022-09-10T12:51:46.199 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:51:45.892+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:51:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:51:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:51:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:51:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:51:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:51:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:51:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:51:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:51:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:46.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:46.199298+0000) 2022-09-10T12:51:47.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:46.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:46.634398+0000) 2022-09-10T12:51:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:47 smithi110 ceph-mon[97633]: pgmap v16079: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:47 smithi104 ceph-mon[97044]: pgmap v16079: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:47.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:47.199527+0000) 2022-09-10T12:51:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:47.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:47.634620+0000) 2022-09-10T12:51:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:48.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:48.199659+0000) 2022-09-10T12:51:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:48.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:48.634832+0000) 2022-09-10T12:51:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:49 smithi110 ceph-mon[97633]: pgmap v16080: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:49 smithi104 ceph-mon[97044]: pgmap v16080: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:49.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:49.199788+0000) 2022-09-10T12:51:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:49.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:49.635076+0000) 2022-09-10T12:51:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:50.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:50.200039+0000) 2022-09-10T12:51:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:50.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:50.635265+0000) 2022-09-10T12:51:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:51 smithi110 ceph-mon[97633]: pgmap v16081: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:51 smithi104 ceph-mon[97044]: pgmap v16081: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:51.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:51.200253+0000) 2022-09-10T12:51:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:51.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:51.635449+0000) 2022-09-10T12:51:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:52.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:52.200494+0000) 2022-09-10T12:51:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:52.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:52.635674+0000) 2022-09-10T12:51:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:53 smithi110 ceph-mon[97633]: pgmap v16082: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:53 smithi104 ceph-mon[97044]: pgmap v16082: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:53.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:53.200737+0000) 2022-09-10T12:51:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:53.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:53.635819+0000) 2022-09-10T12:51:54.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:54 smithi104 ceph-mon[97044]: pgmap v16083: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:54.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:54.200999+0000) 2022-09-10T12:51:54.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:54 smithi110 ceph-mon[97633]: pgmap v16083: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:54.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:54.635947+0000) 2022-09-10T12:51:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:55.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:55.201224+0000) 2022-09-10T12:51:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:55.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:55.636148+0000) 2022-09-10T12:51:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:56.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:56.201488+0000) 2022-09-10T12:51:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:56.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:56.636346+0000) 2022-09-10T12:51:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:57 smithi110 ceph-mon[97633]: pgmap v16084: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:57 smithi104 ceph-mon[97044]: pgmap v16084: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:51:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:57.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:57.201740+0000) 2022-09-10T12:51:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:57.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:57.636534+0000) 2022-09-10T12:51:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:58.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:58.202003+0000) 2022-09-10T12:51:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:58.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:58.636697+0000) 2022-09-10T12:51:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:51:59 smithi110 ceph-mon[97633]: pgmap v16085: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:51:59 smithi104 ceph-mon[97044]: pgmap v16085: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:51:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:51:59.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:59.202225+0000) 2022-09-10T12:52:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:51:59.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:51:59.636864+0000) 2022-09-10T12:52:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:00.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:00.202476+0000) 2022-09-10T12:52:00.907 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:00.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:00.637005+0000) 2022-09-10T12:52:01.202 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:52:00.906+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:52:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:01 smithi110 ceph-mon[97633]: pgmap v16086: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:01 smithi104 ceph-mon[97044]: pgmap v16086: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:01.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:01.202685+0000) 2022-09-10T12:52:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:01.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:01.637166+0000) 2022-09-10T12:52:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:02.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:02.202918+0000) 2022-09-10T12:52:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:02.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:02.637372+0000) 2022-09-10T12:52:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:03 smithi110 ceph-mon[97633]: pgmap v16087: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:03 smithi104 ceph-mon[97044]: pgmap v16087: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:03.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:03.203200+0000) 2022-09-10T12:52:03.956 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:03.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:03.637554+0000) 2022-09-10T12:52:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:04.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:04.203392+0000) 2022-09-10T12:52:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:04.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:04.637775+0000) 2022-09-10T12:52:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:05 smithi110 ceph-mon[97633]: pgmap v16088: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:05 smithi104 ceph-mon[97044]: pgmap v16088: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:05.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:05.203646+0000) 2022-09-10T12:52:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:05.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:05.638009+0000) 2022-09-10T12:52:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:06.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:06.203894+0000) 2022-09-10T12:52:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:06.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:06.638226+0000) 2022-09-10T12:52:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:07 smithi110 ceph-mon[97633]: pgmap v16089: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:52:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:07 smithi104 ceph-mon[97044]: pgmap v16089: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:52:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:07.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:07.204190+0000) 2022-09-10T12:52:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:07.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:07.638422+0000) 2022-09-10T12:52:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:08.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:08.204350+0000) 2022-09-10T12:52:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:08.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:08.638587+0000) 2022-09-10T12:52:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:09 smithi110 ceph-mon[97633]: pgmap v16090: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:52:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:09 smithi110 ceph-mon[97633]: osdmap e1086: 12 total, 12 up, 12 in 2022-09-10T12:52:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:09 smithi104 ceph-mon[97044]: pgmap v16090: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:09 smithi104 ceph-mon[97044]: osdmap e1086: 12 total, 12 up, 12 in 2022-09-10T12:52:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:09.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:09.204558+0000) 2022-09-10T12:52:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:09.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:09.638796+0000) 2022-09-10T12:52:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:10.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:10.204782+0000) 2022-09-10T12:52:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:10.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:10.638976+0000) 2022-09-10T12:52:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:11 smithi110 ceph-mon[97633]: pgmap v16092: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:52:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:11 smithi104 ceph-mon[97044]: pgmap v16092: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:52:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:11.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:11.205007+0000) 2022-09-10T12:52:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:11.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:11.639199+0000) 2022-09-10T12:52:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:12.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:12.205246+0000) 2022-09-10T12:52:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:12.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:12.639348+0000) 2022-09-10T12:52:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:13 smithi110 ceph-mon[97633]: pgmap v16093: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:52:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:13 smithi104 ceph-mon[97044]: pgmap v16093: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:52:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:13.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:13.205526+0000) 2022-09-10T12:52:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:13.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:13.639588+0000) 2022-09-10T12:52:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:14.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:14.205780+0000) 2022-09-10T12:52:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:14.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:14.639736+0000) 2022-09-10T12:52:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:15 smithi110 ceph-mon[97633]: pgmap v16094: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:52:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:15 smithi110 ceph-mon[97633]: osdmap e1087: 12 total, 12 up, 12 in 2022-09-10T12:52:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:15 smithi104 ceph-mon[97044]: pgmap v16094: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:52:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:15 smithi104 ceph-mon[97044]: osdmap e1087: 12 total, 12 up, 12 in 2022-09-10T12:52:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:15.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:15.206092+0000) 2022-09-10T12:52:15.916 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:15.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:15.639891+0000) 2022-09-10T12:52:16.206 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:52:15.915+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:52:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:16.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:16.206363+0000) 2022-09-10T12:52:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:16.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:16.640080+0000) 2022-09-10T12:52:17.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:17 smithi110 ceph-mon[97633]: pgmap v16096: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:52:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:17 smithi104 ceph-mon[97044]: pgmap v16096: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:52:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:17.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:17.206596+0000) 2022-09-10T12:52:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:17.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:17.640286+0000) 2022-09-10T12:52:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:18.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:18.206807+0000) 2022-09-10T12:52:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:18.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:18.640454+0000) 2022-09-10T12:52:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:19 smithi110 ceph-mon[97633]: pgmap v16097: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:52:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:19 smithi104 ceph-mon[97044]: pgmap v16097: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:52:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:19.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:19.206992+0000) 2022-09-10T12:52:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:19.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:19.640617+0000) 2022-09-10T12:52:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:20.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:20.207258+0000) 2022-09-10T12:52:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:20.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:20.640775+0000) 2022-09-10T12:52:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:21 smithi110 ceph-mon[97633]: pgmap v16098: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:52:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:21 smithi104 ceph-mon[97044]: pgmap v16098: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:52:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:21.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:21.207498+0000) 2022-09-10T12:52:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:21.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:21.640939+0000) 2022-09-10T12:52:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:22.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:22.207737+0000) 2022-09-10T12:52:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:22.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:22.641085+0000) 2022-09-10T12:52:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:23 smithi110 ceph-mon[97633]: pgmap v16099: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:52:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:23 smithi104 ceph-mon[97044]: pgmap v16099: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:52:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:23.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:23.208012+0000) 2022-09-10T12:52:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:23.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:23.641256+0000) 2022-09-10T12:52:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:24.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:24.208273+0000) 2022-09-10T12:52:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:24.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:24.641407+0000) 2022-09-10T12:52:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:25 smithi110 ceph-mon[97633]: pgmap v16100: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:52:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:25 smithi104 ceph-mon[97044]: pgmap v16100: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:52:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:25.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:25.208543+0000) 2022-09-10T12:52:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:25.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:25.641589+0000) 2022-09-10T12:52:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:26.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:26.208800+0000) 2022-09-10T12:52:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:26.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:26.641808+0000) 2022-09-10T12:52:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:27 smithi110 ceph-mon[97633]: pgmap v16101: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:52:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:27 smithi104 ceph-mon[97044]: pgmap v16101: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:52:27.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:27.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:27.209026+0000) 2022-09-10T12:52:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:27.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:27.642039+0000) 2022-09-10T12:52:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:28.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:28.209232+0000) 2022-09-10T12:52:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:28.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:28.642190+0000) 2022-09-10T12:52:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:29 smithi110 ceph-mon[97633]: pgmap v16102: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:29 smithi104 ceph-mon[97044]: pgmap v16102: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:29.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:29.209449+0000) 2022-09-10T12:52:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:29.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:29.642358+0000) 2022-09-10T12:52:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:30.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:30.209678+0000) 2022-09-10T12:52:30.930 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:30.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:30.642536+0000) 2022-09-10T12:52:31.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:52:30.929+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:52:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:31 smithi110 ceph-mon[97633]: pgmap v16103: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:31 smithi104 ceph-mon[97044]: pgmap v16103: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:31.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:31.209899+0000) 2022-09-10T12:52:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:31.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:31.642770+0000) 2022-09-10T12:52:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:32.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:32.210198+0000) 2022-09-10T12:52:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:32.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:32.642975+0000) 2022-09-10T12:52:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:33 smithi110 ceph-mon[97633]: pgmap v16104: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:33 smithi104 ceph-mon[97044]: pgmap v16104: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:33.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:33.210420+0000) 2022-09-10T12:52:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:33.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:33.643130+0000) 2022-09-10T12:52:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:34.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:34.210601+0000) 2022-09-10T12:52:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:34.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:34.643274+0000) 2022-09-10T12:52:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:35 smithi110 ceph-mon[97633]: pgmap v16105: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:35 smithi104 ceph-mon[97044]: pgmap v16105: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:35.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:35.210908+0000) 2022-09-10T12:52:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:35.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:35.643419+0000) 2022-09-10T12:52:36.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:36.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:36.211169+0000) 2022-09-10T12:52:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:36.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:36.643586+0000) 2022-09-10T12:52:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:37 smithi110 ceph-mon[97633]: pgmap v16106: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:37 smithi104 ceph-mon[97044]: pgmap v16106: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:37.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:37.211389+0000) 2022-09-10T12:52:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:37.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:37.643799+0000) 2022-09-10T12:52:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:38.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:38.211579+0000) 2022-09-10T12:52:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:38.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:38.643973+0000) 2022-09-10T12:52:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:39 smithi110 ceph-mon[97633]: pgmap v16107: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:39 smithi104 ceph-mon[97044]: pgmap v16107: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:39.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:39.211737+0000) 2022-09-10T12:52:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:39.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:39.644201+0000) 2022-09-10T12:52:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:40.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:40.211967+0000) 2022-09-10T12:52:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:40.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:40.644380+0000) 2022-09-10T12:52:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:41 smithi110 ceph-mon[97633]: pgmap v16108: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:41 smithi104 ceph-mon[97044]: pgmap v16108: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:41.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:41.212188+0000) 2022-09-10T12:52:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:41.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:41.644550+0000) 2022-09-10T12:52:42.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:42.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:42.212396+0000) 2022-09-10T12:52:43.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:42.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:42.644683+0000) 2022-09-10T12:52:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:43 smithi110 ceph-mon[97633]: pgmap v16109: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:43 smithi104 ceph-mon[97044]: pgmap v16109: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:43.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:43.212661+0000) 2022-09-10T12:52:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:43.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:43.644820+0000) 2022-09-10T12:52:44.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:44 smithi110 ceph-mon[97633]: pgmap v16110: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:44.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:44 smithi104 ceph-mon[97044]: pgmap v16110: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:44.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:44.212862+0000) 2022-09-10T12:52:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:44.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:44.645020+0000) 2022-09-10T12:52:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:45.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:45.213181+0000) 2022-09-10T12:52:45.946 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:45.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:45.645188+0000) 2022-09-10T12:52:46.213 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:52:45.946+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:52:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:52:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:52:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:52:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:52:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:52:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:52:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:46.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:46.213336+0000) 2022-09-10T12:52:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:46.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:46.645434+0000) 2022-09-10T12:52:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:47 smithi110 ceph-mon[97633]: pgmap v16111: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:47 smithi104 ceph-mon[97044]: pgmap v16111: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:47.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:47.213597+0000) 2022-09-10T12:52:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:47.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:47.645565+0000) 2022-09-10T12:52:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:48.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:48.213770+0000) 2022-09-10T12:52:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:48.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:48.645724+0000) 2022-09-10T12:52:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:49 smithi110 ceph-mon[97633]: pgmap v16112: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:49 smithi104 ceph-mon[97044]: pgmap v16112: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:49.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:49.213984+0000) 2022-09-10T12:52:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:49.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:49.645873+0000) 2022-09-10T12:52:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:50.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:50.214206+0000) 2022-09-10T12:52:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:50.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:50.646083+0000) 2022-09-10T12:52:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:51 smithi110 ceph-mon[97633]: pgmap v16113: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:51 smithi104 ceph-mon[97044]: pgmap v16113: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:51.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:51.214402+0000) 2022-09-10T12:52:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:51.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:51.646243+0000) 2022-09-10T12:52:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:52.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:52.214625+0000) 2022-09-10T12:52:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:52.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:52.646423+0000) 2022-09-10T12:52:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:53 smithi110 ceph-mon[97633]: pgmap v16114: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:53 smithi104 ceph-mon[97044]: pgmap v16114: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:53.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:53.214817+0000) 2022-09-10T12:52:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:53.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:53.646628+0000) 2022-09-10T12:52:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:54.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:54.214986+0000) 2022-09-10T12:52:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:54.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:54.646816+0000) 2022-09-10T12:52:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:55 smithi110 ceph-mon[97633]: pgmap v16115: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:55 smithi104 ceph-mon[97044]: pgmap v16115: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:55.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:55.215204+0000) 2022-09-10T12:52:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:55.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:55.646964+0000) 2022-09-10T12:52:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:56.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:56.215394+0000) 2022-09-10T12:52:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:56.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:56.647205+0000) 2022-09-10T12:52:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:57 smithi110 ceph-mon[97633]: pgmap v16116: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:57 smithi104 ceph-mon[97044]: pgmap v16116: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:52:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:57.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:57.215555+0000) 2022-09-10T12:52:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:57.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:57.647445+0000) 2022-09-10T12:52:58.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:58.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:58.215801+0000) 2022-09-10T12:52:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:58.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:58.647596+0000) 2022-09-10T12:52:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:52:59 smithi110 ceph-mon[97633]: pgmap v16117: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:52:59 smithi104 ceph-mon[97044]: pgmap v16117: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:52:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:52:59.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:59.215967+0000) 2022-09-10T12:53:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:52:59.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:52:59.647739+0000) 2022-09-10T12:53:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:00.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:00.216204+0000) 2022-09-10T12:53:00.959 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:00.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:00.647881+0000) 2022-09-10T12:53:01.216 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:53:00.958+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:53:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:01 smithi110 ceph-mon[97633]: pgmap v16118: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:01.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:01 smithi104 ceph-mon[97044]: pgmap v16118: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:01.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:01.216459+0000) 2022-09-10T12:53:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:01.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:01.648050+0000) 2022-09-10T12:53:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:02.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:02.216636+0000) 2022-09-10T12:53:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:02.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:02.648211+0000) 2022-09-10T12:53:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:03 smithi110 ceph-mon[97633]: pgmap v16119: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:03 smithi104 ceph-mon[97044]: pgmap v16119: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:03.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:03.216807+0000) 2022-09-10T12:53:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:03.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:03.648414+0000) 2022-09-10T12:53:04.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:04 smithi110 ceph-mon[97633]: pgmap v16120: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:04.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:04 smithi104 ceph-mon[97044]: pgmap v16120: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:04.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:04.217015+0000) 2022-09-10T12:53:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:04.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:04.648638+0000) 2022-09-10T12:53:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:05.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:05.217239+0000) 2022-09-10T12:53:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:05.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:05.648866+0000) 2022-09-10T12:53:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:06.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:06.217481+0000) 2022-09-10T12:53:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:06.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:06.649026+0000) 2022-09-10T12:53:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:07 smithi110 ceph-mon[97633]: pgmap v16121: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:53:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:07 smithi104 ceph-mon[97044]: pgmap v16121: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:53:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:07.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:07.217740+0000) 2022-09-10T12:53:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:07.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:07.649221+0000) 2022-09-10T12:53:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:08.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:08.217948+0000) 2022-09-10T12:53:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:08.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:08.649458+0000) 2022-09-10T12:53:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:09 smithi110 ceph-mon[97633]: pgmap v16122: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:53:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:09 smithi110 ceph-mon[97633]: osdmap e1088: 12 total, 12 up, 12 in 2022-09-10T12:53:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:09 smithi104 ceph-mon[97044]: pgmap v16122: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:53:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:09 smithi104 ceph-mon[97044]: osdmap e1088: 12 total, 12 up, 12 in 2022-09-10T12:53:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:09.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:09.218192+0000) 2022-09-10T12:53:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:09.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:09.649679+0000) 2022-09-10T12:53:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:10.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:10.218438+0000) 2022-09-10T12:53:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:10.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:10.649889+0000) 2022-09-10T12:53:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:11 smithi110 ceph-mon[97633]: pgmap v16124: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:53:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:11 smithi104 ceph-mon[97044]: pgmap v16124: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:53:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:11.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:11.218686+0000) 2022-09-10T12:53:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:11.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:11.650169+0000) 2022-09-10T12:53:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:12.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:12.218908+0000) 2022-09-10T12:53:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:12.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:12.650373+0000) 2022-09-10T12:53:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:13 smithi110 ceph-mon[97633]: pgmap v16125: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:53:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:13 smithi104 ceph-mon[97044]: pgmap v16125: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:53:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:13.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:13.219142+0000) 2022-09-10T12:53:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:13.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:13.650534+0000) 2022-09-10T12:53:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:14.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:14.219350+0000) 2022-09-10T12:53:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:14.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:14.650733+0000) 2022-09-10T12:53:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:15 smithi110 ceph-mon[97633]: pgmap v16126: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:53:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:15 smithi110 ceph-mon[97633]: osdmap e1089: 12 total, 12 up, 12 in 2022-09-10T12:53:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:15 smithi104 ceph-mon[97044]: pgmap v16126: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:53:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:15 smithi104 ceph-mon[97044]: osdmap e1089: 12 total, 12 up, 12 in 2022-09-10T12:53:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:15.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:15.219571+0000) 2022-09-10T12:53:15.960 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:15.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:15.650915+0000) 2022-09-10T12:53:16.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:53:15.959+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:53:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:16.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:16.219752+0000) 2022-09-10T12:53:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:16.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:16.651083+0000) 2022-09-10T12:53:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:17 smithi110 ceph-mon[97633]: pgmap v16128: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:53:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:17 smithi104 ceph-mon[97044]: pgmap v16128: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:53:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:17.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:17.219937+0000) 2022-09-10T12:53:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:17.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:17.651265+0000) 2022-09-10T12:53:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:18.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:18.220176+0000) 2022-09-10T12:53:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:18.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:18.651464+0000) 2022-09-10T12:53:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:19 smithi110 ceph-mon[97633]: pgmap v16129: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T12:53:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:19 smithi104 ceph-mon[97044]: pgmap v16129: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T12:53:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:19.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:19.220383+0000) 2022-09-10T12:53:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:19.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:19.651693+0000) 2022-09-10T12:53:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:20.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:20.220559+0000) 2022-09-10T12:53:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:20.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:20.651899+0000) 2022-09-10T12:53:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:21 smithi110 ceph-mon[97633]: pgmap v16130: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:53:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:21 smithi104 ceph-mon[97044]: pgmap v16130: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:53:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:21.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:21.220755+0000) 2022-09-10T12:53:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:21.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:21.652125+0000) 2022-09-10T12:53:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:22.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:22.221035+0000) 2022-09-10T12:53:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:22.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:22.652326+0000) 2022-09-10T12:53:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:23 smithi110 ceph-mon[97633]: pgmap v16131: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:53:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:23 smithi104 ceph-mon[97044]: pgmap v16131: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:53:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:23.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:23.221285+0000) 2022-09-10T12:53:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:23.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:23.652515+0000) 2022-09-10T12:53:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:24.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:24.221518+0000) 2022-09-10T12:53:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:24.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:24.652640+0000) 2022-09-10T12:53:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:25 smithi110 ceph-mon[97633]: pgmap v16132: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:53:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:25 smithi104 ceph-mon[97044]: pgmap v16132: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:53:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:25.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:25.221694+0000) 2022-09-10T12:53:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:25.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:25.652763+0000) 2022-09-10T12:53:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:26.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:26.221906+0000) 2022-09-10T12:53:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:26.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:26.652944+0000) 2022-09-10T12:53:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:27 smithi110 ceph-mon[97633]: pgmap v16133: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:53:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:27 smithi104 ceph-mon[97044]: pgmap v16133: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:53:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:27.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:27.222183+0000) 2022-09-10T12:53:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:27.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:27.653178+0000) 2022-09-10T12:53:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:28.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:28.222310+0000) 2022-09-10T12:53:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:28.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:28.653367+0000) 2022-09-10T12:53:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:29 smithi110 ceph-mon[97633]: pgmap v16134: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:29 smithi104 ceph-mon[97044]: pgmap v16134: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:29.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:29.222506+0000) 2022-09-10T12:53:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:29.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:29.653557+0000) 2022-09-10T12:53:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:30.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:30.222687+0000) 2022-09-10T12:53:30.976 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:30.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:30.653719+0000) 2022-09-10T12:53:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:31.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:31.222911+0000) 2022-09-10T12:53:31.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:53:30.975+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:53:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:31 smithi110 ceph-mon[97633]: pgmap v16135: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:31 smithi104 ceph-mon[97044]: pgmap v16135: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:31.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:31.653919+0000) 2022-09-10T12:53:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:32.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:32.223194+0000) 2022-09-10T12:53:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:32.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:32.654076+0000) 2022-09-10T12:53:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:33 smithi110 ceph-mon[97633]: pgmap v16136: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:33 smithi104 ceph-mon[97044]: pgmap v16136: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:33.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:33.223380+0000) 2022-09-10T12:53:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:33.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:33.654190+0000) 2022-09-10T12:53:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:34.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:34.223580+0000) 2022-09-10T12:53:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:34.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:34.654383+0000) 2022-09-10T12:53:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:35 smithi110 ceph-mon[97633]: pgmap v16137: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:35 smithi104 ceph-mon[97044]: pgmap v16137: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:35.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:35.223714+0000) 2022-09-10T12:53:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:35.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:35.654624+0000) 2022-09-10T12:53:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:36.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:36.223924+0000) 2022-09-10T12:53:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:36.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:36.654809+0000) 2022-09-10T12:53:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:37 smithi110 ceph-mon[97633]: pgmap v16138: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:37 smithi104 ceph-mon[97044]: pgmap v16138: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:37.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:37.224089+0000) 2022-09-10T12:53:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:37.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:37.654956+0000) 2022-09-10T12:53:38.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:38 smithi110 ceph-mon[97633]: pgmap v16139: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:38.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:38 smithi104 ceph-mon[97044]: pgmap v16139: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:38.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:38.224342+0000) 2022-09-10T12:53:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:38.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:38.655145+0000) 2022-09-10T12:53:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:39.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:39.224580+0000) 2022-09-10T12:53:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:39.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:39.655418+0000) 2022-09-10T12:53:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:40.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:40.224819+0000) 2022-09-10T12:53:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:40.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:40.655595+0000) 2022-09-10T12:53:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:41 smithi110 ceph-mon[97633]: pgmap v16140: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:41 smithi104 ceph-mon[97044]: pgmap v16140: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:41.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:41.224984+0000) 2022-09-10T12:53:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:41.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:41.655829+0000) 2022-09-10T12:53:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:42.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:42.225199+0000) 2022-09-10T12:53:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:42.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:42.655981+0000) 2022-09-10T12:53:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:43 smithi110 ceph-mon[97633]: pgmap v16141: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:43 smithi104 ceph-mon[97044]: pgmap v16141: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:43.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:43.225361+0000) 2022-09-10T12:53:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:43.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:43.656229+0000) 2022-09-10T12:53:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:44.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:44.225583+0000) 2022-09-10T12:53:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:44.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:44.656396+0000) 2022-09-10T12:53:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:45 smithi110 ceph-mon[97633]: pgmap v16142: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:45 smithi104 ceph-mon[97044]: pgmap v16142: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:45.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:45.225800+0000) 2022-09-10T12:53:45.991 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:45.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:45.656597+0000) 2022-09-10T12:53:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:46.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:46.226029+0000) 2022-09-10T12:53:46.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:53:45.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:53:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:53:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:53:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:53:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:53:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:53:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:53:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:53:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:53:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:46.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:46.656790+0000) 2022-09-10T12:53:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:47 smithi110 ceph-mon[97633]: pgmap v16143: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:47 smithi104 ceph-mon[97044]: pgmap v16143: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:47.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:47.226229+0000) 2022-09-10T12:53:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:47.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:47.657001+0000) 2022-09-10T12:53:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:48.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:48.226456+0000) 2022-09-10T12:53:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:48.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:48.657213+0000) 2022-09-10T12:53:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:49 smithi110 ceph-mon[97633]: pgmap v16144: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:49 smithi104 ceph-mon[97044]: pgmap v16144: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:49.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:49.226705+0000) 2022-09-10T12:53:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:49.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:49.657460+0000) 2022-09-10T12:53:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:50.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:50.226927+0000) 2022-09-10T12:53:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:50.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:50.657629+0000) 2022-09-10T12:53:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:51 smithi110 ceph-mon[97633]: pgmap v16145: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:51 smithi104 ceph-mon[97044]: pgmap v16145: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:51.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:51.227168+0000) 2022-09-10T12:53:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:51.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:51.657770+0000) 2022-09-10T12:53:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:52.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:52.227364+0000) 2022-09-10T12:53:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:52.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:52.658008+0000) 2022-09-10T12:53:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:53 smithi110 ceph-mon[97633]: pgmap v16146: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:53 smithi104 ceph-mon[97044]: pgmap v16146: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:53.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:53.227576+0000) 2022-09-10T12:53:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:53.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:53.658203+0000) 2022-09-10T12:53:54.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:54 smithi110 ceph-mon[97633]: pgmap v16147: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:54.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:54 smithi104 ceph-mon[97044]: pgmap v16147: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:54.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:54.227748+0000) 2022-09-10T12:53:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:54.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:54.658340+0000) 2022-09-10T12:53:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:55.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:55.227959+0000) 2022-09-10T12:53:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:55.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:55.658487+0000) 2022-09-10T12:53:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:56.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:56.228158+0000) 2022-09-10T12:53:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:56.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:56.658676+0000) 2022-09-10T12:53:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:57 smithi110 ceph-mon[97633]: pgmap v16148: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:57 smithi104 ceph-mon[97044]: pgmap v16148: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:53:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:57.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:57.228329+0000) 2022-09-10T12:53:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:57.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:57.658838+0000) 2022-09-10T12:53:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:58.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:58.228569+0000) 2022-09-10T12:53:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:58.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:58.659001+0000) 2022-09-10T12:53:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:53:59 smithi110 ceph-mon[97633]: pgmap v16149: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:53:59 smithi104 ceph-mon[97044]: pgmap v16149: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:53:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:53:59.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:59.228722+0000) 2022-09-10T12:54:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:53:59.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:53:59.659241+0000) 2022-09-10T12:54:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:00.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:00.228941+0000) 2022-09-10T12:54:01.006 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:00.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:00.659427+0000) 2022-09-10T12:54:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:01.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:01.229160+0000) 2022-09-10T12:54:01.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:54:01.005+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:54:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:01 smithi110 ceph-mon[97633]: pgmap v16150: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:01 smithi104 ceph-mon[97044]: pgmap v16150: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:01.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:01.659561+0000) 2022-09-10T12:54:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:02.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:02.229342+0000) 2022-09-10T12:54:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:02.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:02.659785+0000) 2022-09-10T12:54:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:03 smithi110 ceph-mon[97633]: pgmap v16151: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:03 smithi104 ceph-mon[97044]: pgmap v16151: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:03.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:03.229605+0000) 2022-09-10T12:54:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:03.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:03.659928+0000) 2022-09-10T12:54:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:04.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:04.229837+0000) 2022-09-10T12:54:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:04.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:04.660117+0000) 2022-09-10T12:54:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:05 smithi110 ceph-mon[97633]: pgmap v16152: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:05 smithi104 ceph-mon[97044]: pgmap v16152: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:05.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:05.230125+0000) 2022-09-10T12:54:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:05.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:05.660256+0000) 2022-09-10T12:54:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:06.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:06.230356+0000) 2022-09-10T12:54:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:06.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:06.660393+0000) 2022-09-10T12:54:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:07 smithi110 ceph-mon[97633]: pgmap v16153: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:54:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:07 smithi104 ceph-mon[97044]: pgmap v16153: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:54:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:07.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:07.230544+0000) 2022-09-10T12:54:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:07.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:07.660562+0000) 2022-09-10T12:54:08.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:08 smithi110 ceph-mon[97633]: pgmap v16154: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:54:08.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:08 smithi104 ceph-mon[97044]: pgmap v16154: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:54:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:08.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:08.230797+0000) 2022-09-10T12:54:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:08.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:08.660755+0000) 2022-09-10T12:54:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:09 smithi110 ceph-mon[97633]: osdmap e1090: 12 total, 12 up, 12 in 2022-09-10T12:54:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:09 smithi104 ceph-mon[97044]: osdmap e1090: 12 total, 12 up, 12 in 2022-09-10T12:54:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:09.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:09.230966+0000) 2022-09-10T12:54:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:09.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:09.660911+0000) 2022-09-10T12:54:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:10.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:10.231208+0000) 2022-09-10T12:54:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:10.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:10.661053+0000) 2022-09-10T12:54:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:11 smithi110 ceph-mon[97633]: pgmap v16156: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:54:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:11 smithi104 ceph-mon[97044]: pgmap v16156: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:54:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:11.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:11.231382+0000) 2022-09-10T12:54:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:11.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:11.661163+0000) 2022-09-10T12:54:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:12.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:12.231558+0000) 2022-09-10T12:54:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:12.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:12.661337+0000) 2022-09-10T12:54:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:13 smithi110 ceph-mon[97633]: pgmap v16157: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:54:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:13 smithi104 ceph-mon[97044]: pgmap v16157: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:54:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:13.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:13.231785+0000) 2022-09-10T12:54:14.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:13.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:13.661558+0000) 2022-09-10T12:54:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:14.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:14.232050+0000) 2022-09-10T12:54:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:14.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:14.661778+0000) 2022-09-10T12:54:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:15 smithi110 ceph-mon[97633]: pgmap v16158: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:54:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:15 smithi104 ceph-mon[97044]: pgmap v16158: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:54:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:15.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:15.232331+0000) 2022-09-10T12:54:16.021 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:15.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:15.661937+0000) 2022-09-10T12:54:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:16.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:16.232539+0000) 2022-09-10T12:54:16.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:54:16.020+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:54:16.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:16 smithi110 ceph-mon[97633]: osdmap e1091: 12 total, 12 up, 12 in 2022-09-10T12:54:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:16 smithi104 ceph-mon[97044]: osdmap e1091: 12 total, 12 up, 12 in 2022-09-10T12:54:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:16.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:16.662108+0000) 2022-09-10T12:54:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:17 smithi110 ceph-mon[97633]: pgmap v16160: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:54:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:17 smithi104 ceph-mon[97044]: pgmap v16160: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:54:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:17.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:17.232759+0000) 2022-09-10T12:54:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:17.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:17.662236+0000) 2022-09-10T12:54:18.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:18 smithi110 ceph-mon[97633]: pgmap v16161: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:54:18.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:18 smithi104 ceph-mon[97044]: pgmap v16161: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T12:54:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:18.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:18.232908+0000) 2022-09-10T12:54:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:18.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:18.662369+0000) 2022-09-10T12:54:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:19.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:19.233177+0000) 2022-09-10T12:54:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:19.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:19.662549+0000) 2022-09-10T12:54:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:20.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:20.233386+0000) 2022-09-10T12:54:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:20.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:20.662714+0000) 2022-09-10T12:54:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:21 smithi110 ceph-mon[97633]: pgmap v16162: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:54:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:21 smithi104 ceph-mon[97044]: pgmap v16162: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:54:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:21.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:21.233633+0000) 2022-09-10T12:54:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:21.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:21.662845+0000) 2022-09-10T12:54:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:22.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:22.233843+0000) 2022-09-10T12:54:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:22.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:22.663088+0000) 2022-09-10T12:54:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:23 smithi110 ceph-mon[97633]: pgmap v16163: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:54:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:23 smithi104 ceph-mon[97044]: pgmap v16163: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:54:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:23.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:23.234050+0000) 2022-09-10T12:54:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:23.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:23.663243+0000) 2022-09-10T12:54:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:24.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:24.234305+0000) 2022-09-10T12:54:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:24.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:24.663477+0000) 2022-09-10T12:54:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:25 smithi110 ceph-mon[97633]: pgmap v16164: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:54:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:25 smithi104 ceph-mon[97044]: pgmap v16164: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:54:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:25.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:25.234508+0000) 2022-09-10T12:54:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:25.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:25.663671+0000) 2022-09-10T12:54:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:26.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:26.234757+0000) 2022-09-10T12:54:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:26.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:26.663803+0000) 2022-09-10T12:54:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:27 smithi110 ceph-mon[97633]: pgmap v16165: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:54:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:27 smithi104 ceph-mon[97044]: pgmap v16165: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:54:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:27.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:27.234893+0000) 2022-09-10T12:54:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:27.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:27.664028+0000) 2022-09-10T12:54:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:28.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:28.235130+0000) 2022-09-10T12:54:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:28.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:28.664226+0000) 2022-09-10T12:54:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:29 smithi110 ceph-mon[97633]: pgmap v16166: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:29 smithi104 ceph-mon[97044]: pgmap v16166: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:29.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:29.235306+0000) 2022-09-10T12:54:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:29.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:29.664442+0000) 2022-09-10T12:54:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:30.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:30.235545+0000) 2022-09-10T12:54:31.033 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:30.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:30.664576+0000) 2022-09-10T12:54:31.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:54:31.032+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:54:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:31.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:31.235796+0000) 2022-09-10T12:54:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:31 smithi110 ceph-mon[97633]: pgmap v16167: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:31 smithi104 ceph-mon[97044]: pgmap v16167: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:31.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:31.664734+0000) 2022-09-10T12:54:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:32.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:32.236044+0000) 2022-09-10T12:54:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:32.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:32.664895+0000) 2022-09-10T12:54:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:33 smithi110 ceph-mon[97633]: pgmap v16168: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:33 smithi104 ceph-mon[97044]: pgmap v16168: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:33.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:33.236309+0000) 2022-09-10T12:54:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:33.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:33.665027+0000) 2022-09-10T12:54:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:34.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:34.236499+0000) 2022-09-10T12:54:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:34.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:34.665212+0000) 2022-09-10T12:54:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:35 smithi110 ceph-mon[97633]: pgmap v16169: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:35 smithi104 ceph-mon[97044]: pgmap v16169: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:35.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:35.236677+0000) 2022-09-10T12:54:36.019 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:35.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:35.665358+0000) 2022-09-10T12:54:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:36.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:36.236867+0000) 2022-09-10T12:54:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:36.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:36.665487+0000) 2022-09-10T12:54:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:37 smithi110 ceph-mon[97633]: pgmap v16170: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:37 smithi104 ceph-mon[97044]: pgmap v16170: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:37.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:37.237076+0000) 2022-09-10T12:54:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:37.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:37.665630+0000) 2022-09-10T12:54:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:38.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:38.237326+0000) 2022-09-10T12:54:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:38.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:38.665831+0000) 2022-09-10T12:54:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:39 smithi110 ceph-mon[97633]: pgmap v16171: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:39 smithi104 ceph-mon[97044]: pgmap v16171: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:39.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:39.237555+0000) 2022-09-10T12:54:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:39.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:39.666046+0000) 2022-09-10T12:54:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:40.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:40.237808+0000) 2022-09-10T12:54:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:40.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:40.666247+0000) 2022-09-10T12:54:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:41 smithi110 ceph-mon[97633]: pgmap v16172: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:41 smithi104 ceph-mon[97044]: pgmap v16172: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:41.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:41.238003+0000) 2022-09-10T12:54:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:41.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:41.666375+0000) 2022-09-10T12:54:42.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:42 smithi110 ceph-mon[97633]: pgmap v16173: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:42.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:42 smithi104 ceph-mon[97044]: pgmap v16173: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:42.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:42.238264+0000) 2022-09-10T12:54:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:42.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:42.666510+0000) 2022-09-10T12:54:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:43.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:43.238515+0000) 2022-09-10T12:54:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:43.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:43.666715+0000) 2022-09-10T12:54:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:44.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:44.238763+0000) 2022-09-10T12:54:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:44 smithi110 ceph-mon[97633]: pgmap v16174: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:44 smithi104 ceph-mon[97044]: pgmap v16174: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:45.132 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:44.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:44.666905+0000) 2022-09-10T12:54:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:45.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:45.238971+0000) 2022-09-10T12:54:46.045 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:45.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:45.667153+0000) 2022-09-10T12:54:46.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:54:46.044+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:54:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:46.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:46.239249+0000) 2022-09-10T12:54:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:54:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:54:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:54:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:54:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:54:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:54:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:46.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:46.667388+0000) 2022-09-10T12:54:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:47 smithi110 ceph-mon[97633]: pgmap v16175: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:47 smithi104 ceph-mon[97044]: pgmap v16175: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:47.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:47.239434+0000) 2022-09-10T12:54:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:47.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:47.667577+0000) 2022-09-10T12:54:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:48.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:48.239761+0000) 2022-09-10T12:54:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:48.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:48.667773+0000) 2022-09-10T12:54:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:49 smithi110 ceph-mon[97633]: pgmap v16176: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:49 smithi104 ceph-mon[97044]: pgmap v16176: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:49.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:49.239930+0000) 2022-09-10T12:54:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:49.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:49.667929+0000) 2022-09-10T12:54:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:50.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:50.240213+0000) 2022-09-10T12:54:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:50.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:50.668089+0000) 2022-09-10T12:54:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:51 smithi110 ceph-mon[97633]: pgmap v16177: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:51 smithi104 ceph-mon[97044]: pgmap v16177: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:51.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:51.240377+0000) 2022-09-10T12:54:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:51.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:51.668260+0000) 2022-09-10T12:54:52.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:52 smithi104 ceph-mon[97044]: pgmap v16178: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:52.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:52.240599+0000) 2022-09-10T12:54:52.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:52 smithi110 ceph-mon[97633]: pgmap v16178: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:52.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:52.668524+0000) 2022-09-10T12:54:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:53.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:53.240843+0000) 2022-09-10T12:54:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:53.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:53.668677+0000) 2022-09-10T12:54:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:54.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:54.241126+0000) 2022-09-10T12:54:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:54 smithi110 ceph-mon[97633]: pgmap v16179: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:54 smithi104 ceph-mon[97044]: pgmap v16179: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:54.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:54.668848+0000) 2022-09-10T12:54:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:55.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:55.241442+0000) 2022-09-10T12:54:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:55.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:55.669014+0000) 2022-09-10T12:54:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:56.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:56.241635+0000) 2022-09-10T12:54:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:56.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:56.669215+0000) 2022-09-10T12:54:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:57 smithi110 ceph-mon[97633]: pgmap v16180: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:57 smithi104 ceph-mon[97044]: pgmap v16180: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:54:57.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:57.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:57.241846+0000) 2022-09-10T12:54:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:57.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:57.669378+0000) 2022-09-10T12:54:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:58.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:58.242169+0000) 2022-09-10T12:54:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:58.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:58.669506+0000) 2022-09-10T12:54:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:54:59 smithi110 ceph-mon[97633]: pgmap v16181: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:54:59 smithi104 ceph-mon[97044]: pgmap v16181: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:54:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:54:59.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:59.242454+0000) 2022-09-10T12:55:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:54:59.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:54:59.669685+0000) 2022-09-10T12:55:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:00.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:00.242588+0000) 2022-09-10T12:55:01.052 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:00.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:00.669807+0000) 2022-09-10T12:55:01.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:55:01.051+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:55:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:01.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:01.242809+0000) 2022-09-10T12:55:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:01 smithi110 ceph-mon[97633]: pgmap v16182: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:01 smithi104 ceph-mon[97044]: pgmap v16182: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:01.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:01.669996+0000) 2022-09-10T12:55:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:02.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:02.243042+0000) 2022-09-10T12:55:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:02.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:02.670194+0000) 2022-09-10T12:55:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:03 smithi110 ceph-mon[97633]: pgmap v16183: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:03 smithi104 ceph-mon[97044]: pgmap v16183: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:03.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:03.243198+0000) 2022-09-10T12:55:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:03.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:03.670335+0000) 2022-09-10T12:55:04.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:04 smithi104 ceph-mon[97044]: pgmap v16184: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:04.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:04.243391+0000) 2022-09-10T12:55:04.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:04 smithi110 ceph-mon[97633]: pgmap v16184: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:04.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:04.670556+0000) 2022-09-10T12:55:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:05.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:05.243643+0000) 2022-09-10T12:55:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:05.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:05.670762+0000) 2022-09-10T12:55:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:06.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:06.243893+0000) 2022-09-10T12:55:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:06.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:06.670980+0000) 2022-09-10T12:55:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:07 smithi110 ceph-mon[97633]: pgmap v16185: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:55:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:07 smithi104 ceph-mon[97044]: pgmap v16185: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:55:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:07.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:07.244166+0000) 2022-09-10T12:55:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:07.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:07.671179+0000) 2022-09-10T12:55:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:08.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:08.244370+0000) 2022-09-10T12:55:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:08.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:08.671313+0000) 2022-09-10T12:55:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:09 smithi110 ceph-mon[97633]: pgmap v16186: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:55:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:09 smithi110 ceph-mon[97633]: osdmap e1092: 12 total, 12 up, 12 in 2022-09-10T12:55:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:09 smithi104 ceph-mon[97044]: pgmap v16186: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:55:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:09 smithi104 ceph-mon[97044]: osdmap e1092: 12 total, 12 up, 12 in 2022-09-10T12:55:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:09.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:09.244615+0000) 2022-09-10T12:55:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:09.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:09.671447+0000) 2022-09-10T12:55:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:10.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:10.244883+0000) 2022-09-10T12:55:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:10.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:10.671633+0000) 2022-09-10T12:55:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:11 smithi110 ceph-mon[97633]: pgmap v16188: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:55:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:11 smithi104 ceph-mon[97044]: pgmap v16188: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:55:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:11.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:11.245185+0000) 2022-09-10T12:55:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:11.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:11.671793+0000) 2022-09-10T12:55:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:12 smithi110 ceph-mon[97633]: pgmap v16189: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:55:12.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:12 smithi104 ceph-mon[97044]: pgmap v16189: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:55:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:12.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:12.245432+0000) 2022-09-10T12:55:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:12.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:12.671993+0000) 2022-09-10T12:55:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:13.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:13.245694+0000) 2022-09-10T12:55:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:13.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:13.672215+0000) 2022-09-10T12:55:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:14.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:14.245909+0000) 2022-09-10T12:55:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:14 smithi110 ceph-mon[97633]: pgmap v16190: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:55:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:14 smithi104 ceph-mon[97044]: pgmap v16190: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:55:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:14.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:14.672426+0000) 2022-09-10T12:55:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:15.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:15.246210+0000) 2022-09-10T12:55:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:15.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:15.672626+0000) 2022-09-10T12:55:16.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:16 smithi110 ceph-mon[97633]: osdmap e1093: 12 total, 12 up, 12 in 2022-09-10T12:55:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:16 smithi104 ceph-mon[97044]: osdmap e1093: 12 total, 12 up, 12 in 2022-09-10T12:55:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:16.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:16.246428+0000) 2022-09-10T12:55:16.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:55:16.067+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:55:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:16.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:16.672866+0000) 2022-09-10T12:55:17.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:17 smithi110 ceph-mon[97633]: pgmap v16192: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:55:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:17 smithi104 ceph-mon[97044]: pgmap v16192: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:55:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:17.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:17.246661+0000) 2022-09-10T12:55:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:17.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:17.673064+0000) 2022-09-10T12:55:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:18.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:18.246921+0000) 2022-09-10T12:55:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:18.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:18.673265+0000) 2022-09-10T12:55:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:19 smithi110 ceph-mon[97633]: pgmap v16193: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T12:55:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:19 smithi104 ceph-mon[97044]: pgmap v16193: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T12:55:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:19.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:19.247133+0000) 2022-09-10T12:55:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:19.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:19.673468+0000) 2022-09-10T12:55:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:20.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:20.247391+0000) 2022-09-10T12:55:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:20.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:20.673679+0000) 2022-09-10T12:55:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:21 smithi110 ceph-mon[97633]: pgmap v16194: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:55:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:21 smithi104 ceph-mon[97044]: pgmap v16194: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:55:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:21.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:21.247652+0000) 2022-09-10T12:55:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:21.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:21.673826+0000) 2022-09-10T12:55:22.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:22 smithi110 ceph-mon[97633]: pgmap v16195: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:55:22.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:22 smithi104 ceph-mon[97044]: pgmap v16195: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:55:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:22.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:22.247906+0000) 2022-09-10T12:55:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:22.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:22.673983+0000) 2022-09-10T12:55:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:23.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:23.248203+0000) 2022-09-10T12:55:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:23.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:23.674216+0000) 2022-09-10T12:55:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:24.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:24.248399+0000) 2022-09-10T12:55:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:24 smithi110 ceph-mon[97633]: pgmap v16196: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:55:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:24 smithi104 ceph-mon[97044]: pgmap v16196: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:55:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:24.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:24.674422+0000) 2022-09-10T12:55:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:25.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:25.248661+0000) 2022-09-10T12:55:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:25.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:25.674610+0000) 2022-09-10T12:55:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:26.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:26.248944+0000) 2022-09-10T12:55:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:26.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:26.674770+0000) 2022-09-10T12:55:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:27 smithi110 ceph-mon[97633]: pgmap v16197: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:55:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:27 smithi104 ceph-mon[97044]: pgmap v16197: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2022-09-10T12:55:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:27.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:27.249181+0000) 2022-09-10T12:55:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:27.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:27.674938+0000) 2022-09-10T12:55:28.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:28 smithi104 ceph-mon[97044]: pgmap v16198: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:28.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:28.249394+0000) 2022-09-10T12:55:28.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:28 smithi110 ceph-mon[97633]: pgmap v16198: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:28.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:28.675209+0000) 2022-09-10T12:55:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:29.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:29.249656+0000) 2022-09-10T12:55:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:29.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:29.675465+0000) 2022-09-10T12:55:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:30.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:30.249852+0000) 2022-09-10T12:55:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:30.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:30.675625+0000) 2022-09-10T12:55:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:31 smithi110 ceph-mon[97633]: pgmap v16199: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:31 smithi104 ceph-mon[97044]: pgmap v16199: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:31.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:55:31.070+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:55:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:31.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:31.250080+0000) 2022-09-10T12:55:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:31.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:31.675809+0000) 2022-09-10T12:55:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:32.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:32.250344+0000) 2022-09-10T12:55:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:32.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:32.675933+0000) 2022-09-10T12:55:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:33 smithi110 ceph-mon[97633]: pgmap v16200: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:33 smithi104 ceph-mon[97044]: pgmap v16200: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:33.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:33.250545+0000) 2022-09-10T12:55:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:33.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:33.676145+0000) 2022-09-10T12:55:34.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:34 smithi110 ceph-mon[97633]: pgmap v16201: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:34.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:34 smithi104 ceph-mon[97044]: pgmap v16201: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:34.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:34.250714+0000) 2022-09-10T12:55:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:34.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:34.676344+0000) 2022-09-10T12:55:35.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:35.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:35.250981+0000) 2022-09-10T12:55:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:35.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:35.676525+0000) 2022-09-10T12:55:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:36.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:36.251214+0000) 2022-09-10T12:55:36.948 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:36.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:36.676696+0000) 2022-09-10T12:55:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:37 smithi110 ceph-mon[97633]: pgmap v16202: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:37 smithi104 ceph-mon[97044]: pgmap v16202: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:37.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:37.251354+0000) 2022-09-10T12:55:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:37.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:37.676860+0000) 2022-09-10T12:55:38.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:38 smithi104 ceph-mon[97044]: pgmap v16203: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:38.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:38.251577+0000) 2022-09-10T12:55:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:38 smithi110 ceph-mon[97633]: pgmap v16203: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:38.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:38.677064+0000) 2022-09-10T12:55:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:39.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:39.251809+0000) 2022-09-10T12:55:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:39.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:39.677252+0000) 2022-09-10T12:55:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:40.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:40.252070+0000) 2022-09-10T12:55:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:40.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:40.677502+0000) 2022-09-10T12:55:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:41 smithi110 ceph-mon[97633]: pgmap v16204: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:41 smithi104 ceph-mon[97044]: pgmap v16204: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:41.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:41.252296+0000) 2022-09-10T12:55:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:41.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:41.677633+0000) 2022-09-10T12:55:42.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:42 smithi104 ceph-mon[97044]: pgmap v16205: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:42.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:42.252530+0000) 2022-09-10T12:55:42.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:42 smithi110 ceph-mon[97633]: pgmap v16205: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:42.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:42.677771+0000) 2022-09-10T12:55:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:43.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:43.252824+0000) 2022-09-10T12:55:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:43.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:43.677914+0000) 2022-09-10T12:55:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:44.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:44.253086+0000) 2022-09-10T12:55:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:44 smithi110 ceph-mon[97633]: pgmap v16206: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:44 smithi104 ceph-mon[97044]: pgmap v16206: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:44.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:44.678060+0000) 2022-09-10T12:55:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:45.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:45.253277+0000) 2022-09-10T12:55:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:45.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:45.678226+0000) 2022-09-10T12:55:46.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:55:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:55:46.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:55:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:55:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:55:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:55:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:55:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:55:46.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:55:46.076+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:55:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:46.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:46.253522+0000) 2022-09-10T12:55:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:46.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:46.678443+0000) 2022-09-10T12:55:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:47 smithi110 ceph-mon[97633]: pgmap v16207: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:47 smithi104 ceph-mon[97044]: pgmap v16207: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:47.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:47.253776+0000) 2022-09-10T12:55:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:47.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:47.678663+0000) 2022-09-10T12:55:48.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:48 smithi104 ceph-mon[97044]: pgmap v16208: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:48.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:48.254020+0000) 2022-09-10T12:55:48.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:48 smithi110 ceph-mon[97633]: pgmap v16208: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:48.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:48.678877+0000) 2022-09-10T12:55:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:49.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:49.254194+0000) 2022-09-10T12:55:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:49.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:49.679063+0000) 2022-09-10T12:55:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:50.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:50.254432+0000) 2022-09-10T12:55:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:50.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:50.679272+0000) 2022-09-10T12:55:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:51 smithi110 ceph-mon[97633]: pgmap v16209: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:51 smithi104 ceph-mon[97044]: pgmap v16209: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:51.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:51.254655+0000) 2022-09-10T12:55:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:51.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:51.679475+0000) 2022-09-10T12:55:52.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:52 smithi110 ceph-mon[97633]: pgmap v16210: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:52.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:52 smithi104 ceph-mon[97044]: pgmap v16210: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:52.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:52.254892+0000) 2022-09-10T12:55:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:52.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:52.679671+0000) 2022-09-10T12:55:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:53.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:53.255129+0000) 2022-09-10T12:55:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:53.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:53.679861+0000) 2022-09-10T12:55:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:54.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:54.255352+0000) 2022-09-10T12:55:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:54 smithi110 ceph-mon[97633]: pgmap v16211: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:54 smithi104 ceph-mon[97044]: pgmap v16211: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:54.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:54.680001+0000) 2022-09-10T12:55:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:55.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:55.255592+0000) 2022-09-10T12:55:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:55.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:55.680155+0000) 2022-09-10T12:55:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:56 smithi104 ceph-mon[97044]: pgmap v16212: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:56.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:56.255760+0000) 2022-09-10T12:55:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:56 smithi110 ceph-mon[97633]: pgmap v16212: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:55:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:56.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:56.680321+0000) 2022-09-10T12:55:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:57.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:57.255990+0000) 2022-09-10T12:55:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:57.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:57.680535+0000) 2022-09-10T12:55:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:58.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:58.256267+0000) 2022-09-10T12:55:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:55:58 smithi110 ceph-mon[97633]: pgmap v16213: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:55:58 smithi104 ceph-mon[97044]: pgmap v16213: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:55:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:58.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:58.680717+0000) 2022-09-10T12:55:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:55:59.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:59.256485+0000) 2022-09-10T12:56:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:55:59.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:55:59.680929+0000) 2022-09-10T12:56:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:00.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:00.256647+0000) 2022-09-10T12:56:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:00 smithi110 ceph-mon[97633]: pgmap v16214: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:00 smithi104 ceph-mon[97044]: pgmap v16214: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:00.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:00.681056+0000) 2022-09-10T12:56:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:01.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:01.256900+0000) 2022-09-10T12:56:01.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:56:01.086+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:56:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:01.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:01.681196+0000) 2022-09-10T12:56:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:02.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:02.257039+0000) 2022-09-10T12:56:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:02 smithi110 ceph-mon[97633]: pgmap v16215: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:02 smithi104 ceph-mon[97044]: pgmap v16215: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:02.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:02.681376+0000) 2022-09-10T12:56:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:03.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:03.257294+0000) 2022-09-10T12:56:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:03.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:03.681587+0000) 2022-09-10T12:56:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:04.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:04.257456+0000) 2022-09-10T12:56:04.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:04 smithi110 ceph-mon[97633]: pgmap v16216: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:04 smithi104 ceph-mon[97044]: pgmap v16216: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:04.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:04.681758+0000) 2022-09-10T12:56:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:05.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:05.257689+0000) 2022-09-10T12:56:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:05.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:05.681931+0000) 2022-09-10T12:56:06.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:06 smithi110 ceph-mon[97633]: pgmap v16217: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:56:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:06 smithi104 ceph-mon[97044]: pgmap v16217: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:56:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:06.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:06.257858+0000) 2022-09-10T12:56:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:06.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:06.682131+0000) 2022-09-10T12:56:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:07.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:07.258107+0000) 2022-09-10T12:56:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:07.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:07.682333+0000) 2022-09-10T12:56:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:08.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:08.258327+0000) 2022-09-10T12:56:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:08 smithi110 ceph-mon[97633]: pgmap v16218: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:56:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:08 smithi104 ceph-mon[97044]: pgmap v16218: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:56:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:08.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:08.682507+0000) 2022-09-10T12:56:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:09.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:09.258588+0000) 2022-09-10T12:56:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:09 smithi110 ceph-mon[97633]: osdmap e1094: 12 total, 12 up, 12 in 2022-09-10T12:56:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:09 smithi104 ceph-mon[97044]: osdmap e1094: 12 total, 12 up, 12 in 2022-09-10T12:56:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:09.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:09.682672+0000) 2022-09-10T12:56:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:10.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:10.258754+0000) 2022-09-10T12:56:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:10 smithi110 ceph-mon[97633]: pgmap v16220: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:56:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:10 smithi110 ceph-mon[97633]: osdmap e1095: 12 total, 12 up, 12 in 2022-09-10T12:56:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:10 smithi104 ceph-mon[97044]: pgmap v16220: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:56:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:10 smithi104 ceph-mon[97044]: osdmap e1095: 12 total, 12 up, 12 in 2022-09-10T12:56:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:10.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:10.682863+0000) 2022-09-10T12:56:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:11.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:11.259028+0000) 2022-09-10T12:56:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:11.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:11.683079+0000) 2022-09-10T12:56:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:12.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:12.259223+0000) 2022-09-10T12:56:12.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:12 smithi110 ceph-mon[97633]: pgmap v16222: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:56:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:12 smithi104 ceph-mon[97044]: pgmap v16222: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:56:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:12.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:12.683303+0000) 2022-09-10T12:56:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:13.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:13.259469+0000) 2022-09-10T12:56:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:13.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:13.683489+0000) 2022-09-10T12:56:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:14.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:14.259668+0000) 2022-09-10T12:56:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:14 smithi110 ceph-mon[97633]: pgmap v16223: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:56:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:14 smithi104 ceph-mon[97044]: pgmap v16223: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:56:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:14.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:14.683618+0000) 2022-09-10T12:56:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:15.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:15.259920+0000) 2022-09-10T12:56:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:15.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:15.683871+0000) 2022-09-10T12:56:16.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:16 smithi110 ceph-mon[97633]: pgmap v16224: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:56:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:16 smithi104 ceph-mon[97044]: pgmap v16224: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:56:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:16.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:16.260130+0000) 2022-09-10T12:56:16.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:56:16.101+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:56:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:16.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:16.684079+0000) 2022-09-10T12:56:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:17.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:17.260293+0000) 2022-09-10T12:56:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:17.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:17.684248+0000) 2022-09-10T12:56:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:18.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:18.260495+0000) 2022-09-10T12:56:18.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:18 smithi110 ceph-mon[97633]: pgmap v16225: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:56:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:18 smithi104 ceph-mon[97044]: pgmap v16225: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:56:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:18.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:18.684418+0000) 2022-09-10T12:56:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:19.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:19.260761+0000) 2022-09-10T12:56:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:19.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:19.684589+0000) 2022-09-10T12:56:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:20.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:20.260978+0000) 2022-09-10T12:56:20.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:20 smithi110 ceph-mon[97633]: pgmap v16226: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:56:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:20 smithi104 ceph-mon[97044]: pgmap v16226: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:56:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:20.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:20.684724+0000) 2022-09-10T12:56:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:21.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:21.261196+0000) 2022-09-10T12:56:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:21.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:21.684851+0000) 2022-09-10T12:56:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:22.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:22.261368+0000) 2022-09-10T12:56:22.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:22 smithi110 ceph-mon[97633]: pgmap v16227: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-10T12:56:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:22 smithi104 ceph-mon[97044]: pgmap v16227: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-10T12:56:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:22.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:22.685037+0000) 2022-09-10T12:56:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:23.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:23.261576+0000) 2022-09-10T12:56:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:23.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:23.685164+0000) 2022-09-10T12:56:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:24.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:24.261706+0000) 2022-09-10T12:56:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:24 smithi110 ceph-mon[97633]: pgmap v16228: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:56:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:24 smithi104 ceph-mon[97044]: pgmap v16228: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:56:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:24.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:24.685296+0000) 2022-09-10T12:56:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:25.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:25.261965+0000) 2022-09-10T12:56:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:25.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:25.685550+0000) 2022-09-10T12:56:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:26 smithi104 ceph-mon[97044]: pgmap v16229: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:26.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:26.262209+0000) 2022-09-10T12:56:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:26 smithi110 ceph-mon[97633]: pgmap v16229: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:56:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:26.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:26.685746+0000) 2022-09-10T12:56:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:27.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:27.262369+0000) 2022-09-10T12:56:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:27.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:27.685925+0000) 2022-09-10T12:56:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:28.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:28.262538+0000) 2022-09-10T12:56:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:28 smithi110 ceph-mon[97633]: pgmap v16230: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:28 smithi104 ceph-mon[97044]: pgmap v16230: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:28.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:28.686133+0000) 2022-09-10T12:56:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:29.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:29.262727+0000) 2022-09-10T12:56:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:29.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:29.686283+0000) 2022-09-10T12:56:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:30.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:30.262954+0000) 2022-09-10T12:56:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:30 smithi110 ceph-mon[97633]: pgmap v16231: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:30 smithi104 ceph-mon[97044]: pgmap v16231: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:30.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:30.686421+0000) 2022-09-10T12:56:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:31.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:31.263181+0000) 2022-09-10T12:56:31.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:56:31.108+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:56:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:31.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:31.686577+0000) 2022-09-10T12:56:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:32.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:32.263421+0000) 2022-09-10T12:56:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:32 smithi110 ceph-mon[97633]: pgmap v16232: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:32 smithi104 ceph-mon[97044]: pgmap v16232: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:32.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:32.686756+0000) 2022-09-10T12:56:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:33.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:33.263642+0000) 2022-09-10T12:56:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:33.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:33.686895+0000) 2022-09-10T12:56:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:34.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:34.263854+0000) 2022-09-10T12:56:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:34 smithi110 ceph-mon[97633]: pgmap v16233: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:34 smithi104 ceph-mon[97044]: pgmap v16233: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:34.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:34.687057+0000) 2022-09-10T12:56:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:35.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:35.264153+0000) 2022-09-10T12:56:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:35.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:35.687180+0000) 2022-09-10T12:56:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:36 smithi104 ceph-mon[97044]: pgmap v16234: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:36.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:36.264395+0000) 2022-09-10T12:56:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:36 smithi110 ceph-mon[97633]: pgmap v16234: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:36.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:36.687308+0000) 2022-09-10T12:56:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:37.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:37.264588+0000) 2022-09-10T12:56:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:37.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:37.687536+0000) 2022-09-10T12:56:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:38.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:38.264834+0000) 2022-09-10T12:56:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:38 smithi110 ceph-mon[97633]: pgmap v16235: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:38 smithi104 ceph-mon[97044]: pgmap v16235: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:38.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:38.687787+0000) 2022-09-10T12:56:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:39.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:39.265076+0000) 2022-09-10T12:56:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:39.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:39.688021+0000) 2022-09-10T12:56:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:40.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:40.265263+0000) 2022-09-10T12:56:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:40 smithi110 ceph-mon[97633]: pgmap v16236: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:40 smithi104 ceph-mon[97044]: pgmap v16236: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:40.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:40.688168+0000) 2022-09-10T12:56:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:41.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:41.265500+0000) 2022-09-10T12:56:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:41.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:41.688306+0000) 2022-09-10T12:56:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:42.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:42.265703+0000) 2022-09-10T12:56:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:42 smithi110 ceph-mon[97633]: pgmap v16237: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:42 smithi104 ceph-mon[97044]: pgmap v16237: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:42.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:42.688522+0000) 2022-09-10T12:56:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:43.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:43.265986+0000) 2022-09-10T12:56:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:43.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:43.688723+0000) 2022-09-10T12:56:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:44.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:44.266220+0000) 2022-09-10T12:56:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:44 smithi110 ceph-mon[97633]: pgmap v16238: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:44 smithi104 ceph-mon[97044]: pgmap v16238: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:44.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:44.688923+0000) 2022-09-10T12:56:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:45.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:45.266442+0000) 2022-09-10T12:56:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:45.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:45.689138+0000) 2022-09-10T12:56:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:46 smithi104 ceph-mon[97044]: pgmap v16239: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:56:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:46.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:46.266700+0000) 2022-09-10T12:56:46.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:56:46.122+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:56:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:46 smithi110 ceph-mon[97633]: pgmap v16239: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:56:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:56:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:56:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:56:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:46.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:46.689355+0000) 2022-09-10T12:56:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:47.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:47.266939+0000) 2022-09-10T12:56:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:47.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:47.689514+0000) 2022-09-10T12:56:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:48.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:48.267197+0000) 2022-09-10T12:56:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:48 smithi110 ceph-mon[97633]: pgmap v16240: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:48 smithi104 ceph-mon[97044]: pgmap v16240: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:48.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:48.689651+0000) 2022-09-10T12:56:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:49.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:49.267381+0000) 2022-09-10T12:56:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:49.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:49.689864+0000) 2022-09-10T12:56:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:50.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:50.267612+0000) 2022-09-10T12:56:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:50 smithi110 ceph-mon[97633]: pgmap v16241: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:50 smithi104 ceph-mon[97044]: pgmap v16241: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:50.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:50.690016+0000) 2022-09-10T12:56:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:51.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:51.267830+0000) 2022-09-10T12:56:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:51.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:51.690198+0000) 2022-09-10T12:56:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:52.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:52.268051+0000) 2022-09-10T12:56:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:52 smithi110 ceph-mon[97633]: pgmap v16242: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:52 smithi104 ceph-mon[97044]: pgmap v16242: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:52.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:52.690399+0000) 2022-09-10T12:56:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:53.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:53.268336+0000) 2022-09-10T12:56:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:53.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:53.690578+0000) 2022-09-10T12:56:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:54.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:54.268547+0000) 2022-09-10T12:56:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:54 smithi110 ceph-mon[97633]: pgmap v16243: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:54 smithi104 ceph-mon[97044]: pgmap v16243: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:54.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:54.690749+0000) 2022-09-10T12:56:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:55.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:55.268763+0000) 2022-09-10T12:56:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:55.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:55.690970+0000) 2022-09-10T12:56:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:56 smithi104 ceph-mon[97044]: pgmap v16244: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:56.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:56.269062+0000) 2022-09-10T12:56:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:56 smithi110 ceph-mon[97633]: pgmap v16244: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:56:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:56.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:56.691142+0000) 2022-09-10T12:56:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:57.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:57.269357+0000) 2022-09-10T12:56:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:57.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:57.691304+0000) 2022-09-10T12:56:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:58.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:58.269523+0000) 2022-09-10T12:56:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:56:58 smithi110 ceph-mon[97633]: pgmap v16245: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:56:58 smithi104 ceph-mon[97044]: pgmap v16245: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:56:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:58.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:58.691453+0000) 2022-09-10T12:56:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:56:59.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:59.269696+0000) 2022-09-10T12:57:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:56:59.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:56:59.691666+0000) 2022-09-10T12:57:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:00.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:00.269941+0000) 2022-09-10T12:57:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:00 smithi110 ceph-mon[97633]: pgmap v16246: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:00 smithi104 ceph-mon[97044]: pgmap v16246: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:00.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:00.691830+0000) 2022-09-10T12:57:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:01.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:01.270166+0000) 2022-09-10T12:57:01.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:57:01.132+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:57:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:01.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:01.691985+0000) 2022-09-10T12:57:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:02.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:02.270418+0000) 2022-09-10T12:57:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:02 smithi110 ceph-mon[97633]: pgmap v16247: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:02 smithi104 ceph-mon[97044]: pgmap v16247: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:02.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:02.692130+0000) 2022-09-10T12:57:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:03.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:03.270670+0000) 2022-09-10T12:57:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:03.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:03.692300+0000) 2022-09-10T12:57:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:04.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:04.270834+0000) 2022-09-10T12:57:04.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:04 smithi110 ceph-mon[97633]: pgmap v16248: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:04 smithi104 ceph-mon[97044]: pgmap v16248: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:04.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:04.692443+0000) 2022-09-10T12:57:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:05.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:05.271120+0000) 2022-09-10T12:57:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:05.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:05.692677+0000) 2022-09-10T12:57:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:06 smithi104 ceph-mon[97044]: pgmap v16249: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:57:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:06.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:06.271364+0000) 2022-09-10T12:57:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:06 smithi110 ceph-mon[97633]: pgmap v16249: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:57:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:06.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:06.692893+0000) 2022-09-10T12:57:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:07.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:07.271606+0000) 2022-09-10T12:57:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:07.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:07.693105+0000) 2022-09-10T12:57:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:08.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:08.271850+0000) 2022-09-10T12:57:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:08 smithi110 ceph-mon[97633]: pgmap v16250: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:57:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:08 smithi104 ceph-mon[97044]: pgmap v16250: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:57:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:08.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:08.693245+0000) 2022-09-10T12:57:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:09.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:09.272089+0000) 2022-09-10T12:57:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:09 smithi110 ceph-mon[97633]: osdmap e1096: 12 total, 12 up, 12 in 2022-09-10T12:57:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:09 smithi104 ceph-mon[97044]: osdmap e1096: 12 total, 12 up, 12 in 2022-09-10T12:57:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:09.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:09.693429+0000) 2022-09-10T12:57:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:10.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:10.272325+0000) 2022-09-10T12:57:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:10 smithi110 ceph-mon[97633]: pgmap v16252: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:57:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:10 smithi110 ceph-mon[97633]: osdmap e1097: 12 total, 12 up, 12 in 2022-09-10T12:57:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:10 smithi104 ceph-mon[97044]: pgmap v16252: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:57:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:10 smithi104 ceph-mon[97044]: osdmap e1097: 12 total, 12 up, 12 in 2022-09-10T12:57:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:10.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:10.693603+0000) 2022-09-10T12:57:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:11.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:11.272605+0000) 2022-09-10T12:57:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:11.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:11.693781+0000) 2022-09-10T12:57:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:12.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:12.272829+0000) 2022-09-10T12:57:12.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:12 smithi110 ceph-mon[97633]: pgmap v16254: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:57:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:12 smithi104 ceph-mon[97044]: pgmap v16254: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:57:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:12.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:12.693985+0000) 2022-09-10T12:57:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:13.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:13.273080+0000) 2022-09-10T12:57:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:13.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:13.694151+0000) 2022-09-10T12:57:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:14.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:14.273385+0000) 2022-09-10T12:57:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:14 smithi110 ceph-mon[97633]: pgmap v16255: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:57:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:14 smithi104 ceph-mon[97044]: pgmap v16255: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:57:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:14.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:14.694405+0000) 2022-09-10T12:57:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:15.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:15.273662+0000) 2022-09-10T12:57:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:15.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:15.694605+0000) 2022-09-10T12:57:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:16 smithi104 ceph-mon[97044]: pgmap v16256: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:57:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:16.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:16.273885+0000) 2022-09-10T12:57:16.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:57:16.138+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:57:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:16 smithi110 ceph-mon[97633]: pgmap v16256: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:57:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:16.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:16.694772+0000) 2022-09-10T12:57:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:17.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:17.274067+0000) 2022-09-10T12:57:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:17.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:17.695002+0000) 2022-09-10T12:57:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:18.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:18.274360+0000) 2022-09-10T12:57:18.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:18 smithi110 ceph-mon[97633]: pgmap v16257: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:57:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:18 smithi104 ceph-mon[97044]: pgmap v16257: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:57:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:18.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:18.695168+0000) 2022-09-10T12:57:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:19.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:19.274516+0000) 2022-09-10T12:57:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:19.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:19.695333+0000) 2022-09-10T12:57:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:20.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:20.274710+0000) 2022-09-10T12:57:20.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:20 smithi110 ceph-mon[97633]: pgmap v16258: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T12:57:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:20 smithi104 ceph-mon[97044]: pgmap v16258: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T12:57:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:20.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:20.695466+0000) 2022-09-10T12:57:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:21.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:21.274889+0000) 2022-09-10T12:57:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:21.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:21.695641+0000) 2022-09-10T12:57:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:22.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:22.275169+0000) 2022-09-10T12:57:22.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:22 smithi110 ceph-mon[97633]: pgmap v16259: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-10T12:57:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:22 smithi104 ceph-mon[97044]: pgmap v16259: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 90 B/s wr, 8 op/s 2022-09-10T12:57:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:22.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:22.695799+0000) 2022-09-10T12:57:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:23.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:23.275344+0000) 2022-09-10T12:57:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:23.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:23.695943+0000) 2022-09-10T12:57:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:24.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:24.275618+0000) 2022-09-10T12:57:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:24 smithi110 ceph-mon[97633]: pgmap v16260: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:57:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:24 smithi104 ceph-mon[97044]: pgmap v16260: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:57:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:24.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:24.696195+0000) 2022-09-10T12:57:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:25.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:25.275879+0000) 2022-09-10T12:57:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:25.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:25.696327+0000) 2022-09-10T12:57:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:26 smithi104 ceph-mon[97044]: pgmap v16261: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:57:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:26.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:26.276132+0000) 2022-09-10T12:57:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:26 smithi110 ceph-mon[97633]: pgmap v16261: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:57:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:26.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:26.696457+0000) 2022-09-10T12:57:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:27.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:27.276333+0000) 2022-09-10T12:57:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:27.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:27.696583+0000) 2022-09-10T12:57:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:28.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:28.276483+0000) 2022-09-10T12:57:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:28 smithi110 ceph-mon[97633]: pgmap v16262: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:57:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:28 smithi104 ceph-mon[97044]: pgmap v16262: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:57:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:28.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:28.696795+0000) 2022-09-10T12:57:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:29.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:29.276726+0000) 2022-09-10T12:57:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:29.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:29.697071+0000) 2022-09-10T12:57:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:30.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:30.276986+0000) 2022-09-10T12:57:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:30 smithi110 ceph-mon[97633]: pgmap v16263: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:30 smithi104 ceph-mon[97044]: pgmap v16263: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:30.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:30.697255+0000) 2022-09-10T12:57:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:31.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:31.277227+0000) 2022-09-10T12:57:31.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:57:31.153+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:57:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:31.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:31.697476+0000) 2022-09-10T12:57:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:32.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:32.277494+0000) 2022-09-10T12:57:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:32 smithi110 ceph-mon[97633]: pgmap v16264: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:32 smithi104 ceph-mon[97044]: pgmap v16264: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:32.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:32.697644+0000) 2022-09-10T12:57:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:33.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:33.277657+0000) 2022-09-10T12:57:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:33.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:33.697849+0000) 2022-09-10T12:57:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:34.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:34.277919+0000) 2022-09-10T12:57:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:34 smithi110 ceph-mon[97633]: pgmap v16265: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:34 smithi104 ceph-mon[97044]: pgmap v16265: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:34.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:34.698002+0000) 2022-09-10T12:57:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:35.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:35.278216+0000) 2022-09-10T12:57:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:35.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:35.698164+0000) 2022-09-10T12:57:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:36 smithi104 ceph-mon[97044]: pgmap v16266: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:36.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:36.278392+0000) 2022-09-10T12:57:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:36 smithi110 ceph-mon[97633]: pgmap v16266: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:36.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:36.698342+0000) 2022-09-10T12:57:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:37.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:37.278647+0000) 2022-09-10T12:57:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:37.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:37.698517+0000) 2022-09-10T12:57:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:38.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:38.278849+0000) 2022-09-10T12:57:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:38 smithi110 ceph-mon[97633]: pgmap v16267: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:38 smithi104 ceph-mon[97044]: pgmap v16267: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:38.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:38.698662+0000) 2022-09-10T12:57:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:39.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:39.279115+0000) 2022-09-10T12:57:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:39.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:39.698831+0000) 2022-09-10T12:57:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:40.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:40.279318+0000) 2022-09-10T12:57:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:40 smithi110 ceph-mon[97633]: pgmap v16268: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:40 smithi104 ceph-mon[97044]: pgmap v16268: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:40.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:40.699067+0000) 2022-09-10T12:57:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:41.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:41.279519+0000) 2022-09-10T12:57:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:41.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:41.699254+0000) 2022-09-10T12:57:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:42.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:42.279746+0000) 2022-09-10T12:57:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:42 smithi110 ceph-mon[97633]: pgmap v16269: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:42 smithi104 ceph-mon[97044]: pgmap v16269: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:42.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:42.699456+0000) 2022-09-10T12:57:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:43.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:43.279964+0000) 2022-09-10T12:57:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:43.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:43.699693+0000) 2022-09-10T12:57:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:44.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:44.280202+0000) 2022-09-10T12:57:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:44 smithi110 ceph-mon[97633]: pgmap v16270: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:44 smithi104 ceph-mon[97044]: pgmap v16270: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:44.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:44.699907+0000) 2022-09-10T12:57:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:45.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:45.280423+0000) 2022-09-10T12:57:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:45.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:45.700037+0000) 2022-09-10T12:57:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:46 smithi104 ceph-mon[97044]: pgmap v16271: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:57:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:57:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:46.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:46.280567+0000) 2022-09-10T12:57:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:57:46.168+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:57:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:46 smithi110 ceph-mon[97633]: pgmap v16271: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:57:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:57:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:46.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:46.700213+0000) 2022-09-10T12:57:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:47.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:47.280823+0000) 2022-09-10T12:57:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:47.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:47.700436+0000) 2022-09-10T12:57:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:48.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:48.281078+0000) 2022-09-10T12:57:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:48 smithi110 ceph-mon[97633]: pgmap v16272: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:48 smithi104 ceph-mon[97044]: pgmap v16272: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:48.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:48.700645+0000) 2022-09-10T12:57:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:49.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:49.281341+0000) 2022-09-10T12:57:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:49.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:49.700852+0000) 2022-09-10T12:57:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:50.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:50.281588+0000) 2022-09-10T12:57:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:50 smithi110 ceph-mon[97633]: pgmap v16273: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:50 smithi104 ceph-mon[97044]: pgmap v16273: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:50.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:50.701151+0000) 2022-09-10T12:57:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:51.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:51.281750+0000) 2022-09-10T12:57:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:51.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:51.701367+0000) 2022-09-10T12:57:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:52.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:52.281979+0000) 2022-09-10T12:57:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:52 smithi110 ceph-mon[97633]: pgmap v16274: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:52 smithi104 ceph-mon[97044]: pgmap v16274: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:52.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:52.701495+0000) 2022-09-10T12:57:53.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:53.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:53.282174+0000) 2022-09-10T12:57:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:53.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:53.701645+0000) 2022-09-10T12:57:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:54.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:54.282348+0000) 2022-09-10T12:57:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:54 smithi110 ceph-mon[97633]: pgmap v16275: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:54 smithi104 ceph-mon[97044]: pgmap v16275: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:54.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:54.701877+0000) 2022-09-10T12:57:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:55.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:55.282574+0000) 2022-09-10T12:57:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:55.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:55.702046+0000) 2022-09-10T12:57:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:56 smithi104 ceph-mon[97044]: pgmap v16276: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:56.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:56.282822+0000) 2022-09-10T12:57:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:56 smithi110 ceph-mon[97633]: pgmap v16276: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:57:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:56.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:56.702203+0000) 2022-09-10T12:57:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:57.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:57.283022+0000) 2022-09-10T12:57:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:57.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:57.702395+0000) 2022-09-10T12:57:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:58.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:58.283308+0000) 2022-09-10T12:57:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:57:58 smithi110 ceph-mon[97633]: pgmap v16277: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:57:58 smithi104 ceph-mon[97044]: pgmap v16277: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:57:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:58.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:58.702565+0000) 2022-09-10T12:57:59.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:57:59.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:59.283502+0000) 2022-09-10T12:58:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:57:59.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:57:59.702759+0000) 2022-09-10T12:58:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:00.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:00.283760+0000) 2022-09-10T12:58:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:00 smithi110 ceph-mon[97633]: pgmap v16278: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:00 smithi104 ceph-mon[97044]: pgmap v16278: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:00.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:00.702890+0000) 2022-09-10T12:58:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:01.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:01.283956+0000) 2022-09-10T12:58:01.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:58:01.173+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:58:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:01.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:01.703126+0000) 2022-09-10T12:58:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:02.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:02.284185+0000) 2022-09-10T12:58:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:02 smithi110 ceph-mon[97633]: pgmap v16279: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:02 smithi104 ceph-mon[97044]: pgmap v16279: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:02.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:02.703276+0000) 2022-09-10T12:58:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:03.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:03.284398+0000) 2022-09-10T12:58:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:03.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:03.703434+0000) 2022-09-10T12:58:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:04.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:04.284627+0000) 2022-09-10T12:58:04.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:04 smithi110 ceph-mon[97633]: pgmap v16280: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:04 smithi104 ceph-mon[97044]: pgmap v16280: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:04.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:04.703572+0000) 2022-09-10T12:58:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:05.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:05.284902+0000) 2022-09-10T12:58:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:05.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:05.703724+0000) 2022-09-10T12:58:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:06 smithi104 ceph-mon[97044]: pgmap v16281: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:58:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:06.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:06.285192+0000) 2022-09-10T12:58:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:06 smithi110 ceph-mon[97633]: pgmap v16281: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:58:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:06.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:06.703917+0000) 2022-09-10T12:58:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:07.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:07.285453+0000) 2022-09-10T12:58:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:07.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:07.704059+0000) 2022-09-10T12:58:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:08.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:08.285675+0000) 2022-09-10T12:58:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:08 smithi110 ceph-mon[97633]: pgmap v16282: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:58:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:08 smithi104 ceph-mon[97044]: pgmap v16282: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:58:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:08.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:08.704171+0000) 2022-09-10T12:58:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:09.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:09.285897+0000) 2022-09-10T12:58:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:09 smithi110 ceph-mon[97633]: osdmap e1098: 12 total, 12 up, 12 in 2022-09-10T12:58:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:09 smithi104 ceph-mon[97044]: osdmap e1098: 12 total, 12 up, 12 in 2022-09-10T12:58:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:09.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:09.704347+0000) 2022-09-10T12:58:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:10.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:10.286143+0000) 2022-09-10T12:58:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:10 smithi110 ceph-mon[97633]: pgmap v16284: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:58:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:10 smithi110 ceph-mon[97633]: osdmap e1099: 12 total, 12 up, 12 in 2022-09-10T12:58:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:10 smithi104 ceph-mon[97044]: pgmap v16284: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:58:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:10 smithi104 ceph-mon[97044]: osdmap e1099: 12 total, 12 up, 12 in 2022-09-10T12:58:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:10.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:10.704484+0000) 2022-09-10T12:58:11.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:11.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:11.286386+0000) 2022-09-10T12:58:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:11.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:11.704610+0000) 2022-09-10T12:58:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:12.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:12.286691+0000) 2022-09-10T12:58:12.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:12 smithi110 ceph-mon[97633]: pgmap v16286: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:58:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:12 smithi104 ceph-mon[97044]: pgmap v16286: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:58:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:12.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:12.704811+0000) 2022-09-10T12:58:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:13.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:13.286961+0000) 2022-09-10T12:58:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:13.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:13.705037+0000) 2022-09-10T12:58:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:14.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:14.287200+0000) 2022-09-10T12:58:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:14 smithi110 ceph-mon[97633]: pgmap v16287: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:58:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:14 smithi104 ceph-mon[97044]: pgmap v16287: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:58:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:14.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:14.705229+0000) 2022-09-10T12:58:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:15.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:15.287440+0000) 2022-09-10T12:58:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:15.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:15.705415+0000) 2022-09-10T12:58:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:16 smithi104 ceph-mon[97044]: pgmap v16288: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:58:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:16.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:16.287627+0000) 2022-09-10T12:58:16.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:58:16.180+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:58:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:16 smithi110 ceph-mon[97633]: pgmap v16288: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:58:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:16.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:16.705594+0000) 2022-09-10T12:58:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:17.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:17.287835+0000) 2022-09-10T12:58:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:17.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:17.705797+0000) 2022-09-10T12:58:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:18.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:18.288089+0000) 2022-09-10T12:58:18.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:18 smithi110 ceph-mon[97633]: pgmap v16289: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:58:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:18 smithi104 ceph-mon[97044]: pgmap v16289: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:58:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:18.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:18.706029+0000) 2022-09-10T12:58:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:19.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:19.288354+0000) 2022-09-10T12:58:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:19.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:19.706193+0000) 2022-09-10T12:58:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:20.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:20.288581+0000) 2022-09-10T12:58:20.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:20 smithi110 ceph-mon[97633]: pgmap v16290: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:58:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:20 smithi104 ceph-mon[97044]: pgmap v16290: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:58:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:20.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:20.706372+0000) 2022-09-10T12:58:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:21.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:21.288851+0000) 2022-09-10T12:58:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:21.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:21.706539+0000) 2022-09-10T12:58:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:22.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:22.289125+0000) 2022-09-10T12:58:22.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:22 smithi110 ceph-mon[97633]: pgmap v16291: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T12:58:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:22 smithi104 ceph-mon[97044]: pgmap v16291: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T12:58:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:22.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:22.706794+0000) 2022-09-10T12:58:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:23.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:23.289392+0000) 2022-09-10T12:58:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:23.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:23.706972+0000) 2022-09-10T12:58:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:24.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:24.289641+0000) 2022-09-10T12:58:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:24 smithi110 ceph-mon[97633]: pgmap v16292: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:58:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:24 smithi104 ceph-mon[97044]: pgmap v16292: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:58:25.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:24.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:24.707146+0000) 2022-09-10T12:58:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:25.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:25.289888+0000) 2022-09-10T12:58:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:25.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:25.707350+0000) 2022-09-10T12:58:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:26 smithi104 ceph-mon[97044]: pgmap v16293: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:58:26.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:26.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:26.290170+0000) 2022-09-10T12:58:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:26 smithi110 ceph-mon[97633]: pgmap v16293: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:58:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:26.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:26.707573+0000) 2022-09-10T12:58:27.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:27.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:27.290382+0000) 2022-09-10T12:58:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:27.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:27.707802+0000) 2022-09-10T12:58:28.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:28.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:28.290610+0000) 2022-09-10T12:58:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:28 smithi110 ceph-mon[97633]: pgmap v16294: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:28 smithi104 ceph-mon[97044]: pgmap v16294: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:28.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:28.707945+0000) 2022-09-10T12:58:29.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:29.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:29.290844+0000) 2022-09-10T12:58:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:29.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:29.708202+0000) 2022-09-10T12:58:30.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:30.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:30.291040+0000) 2022-09-10T12:58:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:30 smithi110 ceph-mon[97633]: pgmap v16295: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:30 smithi104 ceph-mon[97044]: pgmap v16295: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:30.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:30.708368+0000) 2022-09-10T12:58:31.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:58:31.190+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:58:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:31.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:31.291263+0000) 2022-09-10T12:58:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:31.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:31.708536+0000) 2022-09-10T12:58:32.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:32.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:32.291484+0000) 2022-09-10T12:58:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:32 smithi110 ceph-mon[97633]: pgmap v16296: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:32 smithi104 ceph-mon[97044]: pgmap v16296: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:32.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:32.708751+0000) 2022-09-10T12:58:33.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:33.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:33.291690+0000) 2022-09-10T12:58:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:33.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:33.708992+0000) 2022-09-10T12:58:34.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:34.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:34.291863+0000) 2022-09-10T12:58:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:34 smithi110 ceph-mon[97633]: pgmap v16297: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:34 smithi104 ceph-mon[97044]: pgmap v16297: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:34.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:34.709261+0000) 2022-09-10T12:58:35.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:35.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:35.292151+0000) 2022-09-10T12:58:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:35.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:35.709435+0000) 2022-09-10T12:58:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:36 smithi104 ceph-mon[97044]: pgmap v16298: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:36.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:36.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:36.292332+0000) 2022-09-10T12:58:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:36 smithi110 ceph-mon[97633]: pgmap v16298: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:36.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:36.709636+0000) 2022-09-10T12:58:37.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:37.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:37.292552+0000) 2022-09-10T12:58:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:37.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:37.709775+0000) 2022-09-10T12:58:38.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:38.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:38.292780+0000) 2022-09-10T12:58:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:38 smithi110 ceph-mon[97633]: pgmap v16299: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:38 smithi104 ceph-mon[97044]: pgmap v16299: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:38.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:38.709937+0000) 2022-09-10T12:58:39.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:39.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:39.292976+0000) 2022-09-10T12:58:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:39.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:39.710150+0000) 2022-09-10T12:58:40.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:40.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:40.293195+0000) 2022-09-10T12:58:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:40 smithi110 ceph-mon[97633]: pgmap v16300: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:40 smithi104 ceph-mon[97044]: pgmap v16300: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:40.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:40.710335+0000) 2022-09-10T12:58:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:41.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:41.293462+0000) 2022-09-10T12:58:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:41.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:41.710562+0000) 2022-09-10T12:58:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:42.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:42.293637+0000) 2022-09-10T12:58:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:42 smithi110 ceph-mon[97633]: pgmap v16301: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:42 smithi104 ceph-mon[97044]: pgmap v16301: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:42.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:42.710745+0000) 2022-09-10T12:58:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:43.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:43.293862+0000) 2022-09-10T12:58:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:43.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:43.711008+0000) 2022-09-10T12:58:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:44.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:44.294030+0000) 2022-09-10T12:58:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:44 smithi110 ceph-mon[97633]: pgmap v16302: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:44 smithi104 ceph-mon[97044]: pgmap v16302: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:44.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:44.711181+0000) 2022-09-10T12:58:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:45.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:45.294256+0000) 2022-09-10T12:58:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:45.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:45.711323+0000) 2022-09-10T12:58:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:46 smithi104 ceph-mon[97044]: pgmap v16303: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:58:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:58:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:58:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:58:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:46.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:46.294445+0000) 2022-09-10T12:58:46.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:58:46.202+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:58:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:46 smithi110 ceph-mon[97633]: pgmap v16303: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:58:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:58:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:46.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:46.711455+0000) 2022-09-10T12:58:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:47.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:47.294603+0000) 2022-09-10T12:58:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:47.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:47.711585+0000) 2022-09-10T12:58:48.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:48.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:48.294816+0000) 2022-09-10T12:58:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:48 smithi110 ceph-mon[97633]: pgmap v16304: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:48 smithi104 ceph-mon[97044]: pgmap v16304: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:48.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:48.711730+0000) 2022-09-10T12:58:49.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:49.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:49.295071+0000) 2022-09-10T12:58:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:49.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:49.711908+0000) 2022-09-10T12:58:50.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:50.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:50.295300+0000) 2022-09-10T12:58:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:50 smithi110 ceph-mon[97633]: pgmap v16305: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:50 smithi104 ceph-mon[97044]: pgmap v16305: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:50.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:50.712164+0000) 2022-09-10T12:58:51.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:51.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:51.295560+0000) 2022-09-10T12:58:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:51.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:51.712344+0000) 2022-09-10T12:58:52.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:52.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:52.295755+0000) 2022-09-10T12:58:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:52 smithi110 ceph-mon[97633]: pgmap v16306: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:52 smithi104 ceph-mon[97044]: pgmap v16306: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:52.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:52.712531+0000) 2022-09-10T12:58:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:53.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:53.295937+0000) 2022-09-10T12:58:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:53.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:53.712672+0000) 2022-09-10T12:58:54.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:54.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:54.296177+0000) 2022-09-10T12:58:54.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:54 smithi110 ceph-mon[97633]: pgmap v16307: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:54 smithi104 ceph-mon[97044]: pgmap v16307: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:54.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:54.712841+0000) 2022-09-10T12:58:55.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:55.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:55.296396+0000) 2022-09-10T12:58:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:55.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:55.712978+0000) 2022-09-10T12:58:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:56 smithi104 ceph-mon[97044]: pgmap v16308: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:56.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:56.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:56.296629+0000) 2022-09-10T12:58:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:56 smithi110 ceph-mon[97633]: pgmap v16308: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:58:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:56.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:56.713196+0000) 2022-09-10T12:58:57.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:57.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:57.296862+0000) 2022-09-10T12:58:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:57.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:57.713333+0000) 2022-09-10T12:58:58.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:58.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:58.297143+0000) 2022-09-10T12:58:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:58:58 smithi110 ceph-mon[97633]: pgmap v16309: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:58:58 smithi104 ceph-mon[97044]: pgmap v16309: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:58:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:58.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:58.713517+0000) 2022-09-10T12:58:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:58:59.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:59.297336+0000) 2022-09-10T12:59:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:58:59.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:58:59.713653+0000) 2022-09-10T12:59:00.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:00.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:00.297585+0000) 2022-09-10T12:59:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:00 smithi110 ceph-mon[97633]: pgmap v16310: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:00 smithi104 ceph-mon[97044]: pgmap v16310: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:00.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:00.713789+0000) 2022-09-10T12:59:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:01.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:01.297738+0000) 2022-09-10T12:59:01.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:59:01.215+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:59:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:01.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:01.713911+0000) 2022-09-10T12:59:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:02.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:02.297975+0000) 2022-09-10T12:59:02.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:02 smithi110 ceph-mon[97633]: pgmap v16311: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:02 smithi104 ceph-mon[97044]: pgmap v16311: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:02.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:02.714153+0000) 2022-09-10T12:59:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:03.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:03.298221+0000) 2022-09-10T12:59:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:03.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:03.714382+0000) 2022-09-10T12:59:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:04.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:04.298424+0000) 2022-09-10T12:59:04.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:04 smithi110 ceph-mon[97633]: pgmap v16312: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:04 smithi104 ceph-mon[97044]: pgmap v16312: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:04.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:04.714551+0000) 2022-09-10T12:59:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:05.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:05.298656+0000) 2022-09-10T12:59:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:05.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:05.714714+0000) 2022-09-10T12:59:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:06 smithi104 ceph-mon[97044]: pgmap v16313: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:59:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:06.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:06.298881+0000) 2022-09-10T12:59:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:06 smithi110 ceph-mon[97633]: pgmap v16313: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T12:59:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:06.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:06.714930+0000) 2022-09-10T12:59:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:07.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:07.299170+0000) 2022-09-10T12:59:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:07.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:07.715065+0000) 2022-09-10T12:59:08.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:08.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:08.299353+0000) 2022-09-10T12:59:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:08 smithi110 ceph-mon[97633]: pgmap v16314: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:59:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:08 smithi104 ceph-mon[97044]: pgmap v16314: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T12:59:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:08.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:08.715244+0000) 2022-09-10T12:59:09.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:09.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:09.299552+0000) 2022-09-10T12:59:09.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:09 smithi110 ceph-mon[97633]: osdmap e1100: 12 total, 12 up, 12 in 2022-09-10T12:59:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:09 smithi104 ceph-mon[97044]: osdmap e1100: 12 total, 12 up, 12 in 2022-09-10T12:59:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:09.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:09.715451+0000) 2022-09-10T12:59:10.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:10.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:10.299769+0000) 2022-09-10T12:59:10.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:10 smithi110 ceph-mon[97633]: pgmap v16316: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:59:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:10 smithi110 ceph-mon[97633]: osdmap e1101: 12 total, 12 up, 12 in 2022-09-10T12:59:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:10 smithi104 ceph-mon[97044]: pgmap v16316: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T12:59:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:10 smithi104 ceph-mon[97044]: osdmap e1101: 12 total, 12 up, 12 in 2022-09-10T12:59:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:10.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:10.715617+0000) 2022-09-10T12:59:11.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:11.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:11.300005+0000) 2022-09-10T12:59:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:11.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:11.715830+0000) 2022-09-10T12:59:12.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:12.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:12.300220+0000) 2022-09-10T12:59:12.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:12 smithi110 ceph-mon[97633]: pgmap v16318: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:59:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:12 smithi104 ceph-mon[97044]: pgmap v16318: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T12:59:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:12.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:12.716030+0000) 2022-09-10T12:59:13.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:13.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:13.300452+0000) 2022-09-10T12:59:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:13.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:13.716269+0000) 2022-09-10T12:59:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:14.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:14.300635+0000) 2022-09-10T12:59:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:14 smithi110 ceph-mon[97633]: pgmap v16319: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:59:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:14 smithi104 ceph-mon[97044]: pgmap v16319: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T12:59:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:14.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:14.716481+0000) 2022-09-10T12:59:15.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:15.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:15.300862+0000) 2022-09-10T12:59:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:15.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:15.716614+0000) 2022-09-10T12:59:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:16 smithi104 ceph-mon[97044]: pgmap v16320: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:59:16.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:59:16.219+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:59:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:16.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:16.301073+0000) 2022-09-10T12:59:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:16 smithi110 ceph-mon[97633]: pgmap v16320: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T12:59:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:16.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:16.716752+0000) 2022-09-10T12:59:17.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:17.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:17.301335+0000) 2022-09-10T12:59:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:17.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:17.716886+0000) 2022-09-10T12:59:18.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:18.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:18.301603+0000) 2022-09-10T12:59:18.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:18 smithi110 ceph-mon[97633]: pgmap v16321: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:59:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:18 smithi104 ceph-mon[97044]: pgmap v16321: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T12:59:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:18.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:18.717048+0000) 2022-09-10T12:59:19.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:19.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:19.301815+0000) 2022-09-10T12:59:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:19.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:19.717189+0000) 2022-09-10T12:59:20.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:20.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:20.302039+0000) 2022-09-10T12:59:20.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:20 smithi110 ceph-mon[97633]: pgmap v16322: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:59:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:20 smithi104 ceph-mon[97044]: pgmap v16322: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T12:59:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:20.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:20.717395+0000) 2022-09-10T12:59:21.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:21.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:21.302320+0000) 2022-09-10T12:59:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:21.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:21.717547+0000) 2022-09-10T12:59:22.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:22.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:22.302530+0000) 2022-09-10T12:59:22.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:22 smithi110 ceph-mon[97633]: pgmap v16323: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T12:59:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:22 smithi104 ceph-mon[97044]: pgmap v16323: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T12:59:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:22.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:22.717667+0000) 2022-09-10T12:59:23.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:23.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:23.302755+0000) 2022-09-10T12:59:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:23.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:23.717859+0000) 2022-09-10T12:59:24.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:24.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:24.303009+0000) 2022-09-10T12:59:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:24 smithi110 ceph-mon[97633]: pgmap v16324: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:59:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:24 smithi104 ceph-mon[97044]: pgmap v16324: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T12:59:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:24.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:24.717998+0000) 2022-09-10T12:59:25.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:25.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:25.303245+0000) 2022-09-10T12:59:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:25.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:25.718205+0000) 2022-09-10T12:59:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:26 smithi104 ceph-mon[97044]: pgmap v16325: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:59:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:26 smithi110 ceph-mon[97633]: pgmap v16325: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T12:59:26.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:26.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:26.303495+0000) 2022-09-10T12:59:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:26.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:26.718436+0000) 2022-09-10T12:59:27.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:27.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:27.303677+0000) 2022-09-10T12:59:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:27.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:27.718608+0000) 2022-09-10T12:59:28.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:28.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:28.303890+0000) 2022-09-10T12:59:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:28 smithi110 ceph-mon[97633]: pgmap v16326: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:28 smithi104 ceph-mon[97044]: pgmap v16326: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:28.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:28.718825+0000) 2022-09-10T12:59:29.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:29.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:29.304163+0000) 2022-09-10T12:59:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:29.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:29.719033+0000) 2022-09-10T12:59:30.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:30.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:30.304394+0000) 2022-09-10T12:59:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:30 smithi110 ceph-mon[97633]: pgmap v16327: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:30 smithi104 ceph-mon[97044]: pgmap v16327: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:30.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:30.719248+0000) 2022-09-10T12:59:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:31.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:31.304608+0000) 2022-09-10T12:59:31.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:59:31.226+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:59:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:31.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:31.719463+0000) 2022-09-10T12:59:32.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:32.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:32.304822+0000) 2022-09-10T12:59:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:32 smithi110 ceph-mon[97633]: pgmap v16328: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:32 smithi104 ceph-mon[97044]: pgmap v16328: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:32.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:32.719667+0000) 2022-09-10T12:59:33.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:33.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:33.305071+0000) 2022-09-10T12:59:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:33.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:33.719882+0000) 2022-09-10T12:59:34.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:34.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:34.305277+0000) 2022-09-10T12:59:34.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:34 smithi110 ceph-mon[97633]: pgmap v16329: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:34 smithi104 ceph-mon[97044]: pgmap v16329: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:34.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:34.720088+0000) 2022-09-10T12:59:35.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:35.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:35.305529+0000) 2022-09-10T12:59:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:35.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:35.720285+0000) 2022-09-10T12:59:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:36 smithi104 ceph-mon[97044]: pgmap v16330: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:36 smithi110 ceph-mon[97633]: pgmap v16330: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:36.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:36.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:36.305733+0000) 2022-09-10T12:59:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:36.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:36.720533+0000) 2022-09-10T12:59:37.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:37.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:37.305908+0000) 2022-09-10T12:59:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:37.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:37.720723+0000) 2022-09-10T12:59:38.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:38.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:38.306052+0000) 2022-09-10T12:59:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:38 smithi110 ceph-mon[97633]: pgmap v16331: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:38 smithi104 ceph-mon[97044]: pgmap v16331: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:38.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:38.720946+0000) 2022-09-10T12:59:39.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:39.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:39.306308+0000) 2022-09-10T12:59:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:39.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:39.721232+0000) 2022-09-10T12:59:40.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:40.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:40.306531+0000) 2022-09-10T12:59:40.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:40 smithi110 ceph-mon[97633]: pgmap v16332: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:40 smithi104 ceph-mon[97044]: pgmap v16332: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:40.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:40.721390+0000) 2022-09-10T12:59:41.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:41.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:41.306782+0000) 2022-09-10T12:59:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:41.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:41.721531+0000) 2022-09-10T12:59:42.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:42.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:42.307014+0000) 2022-09-10T12:59:42.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:42 smithi110 ceph-mon[97633]: pgmap v16333: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:42 smithi104 ceph-mon[97044]: pgmap v16333: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:42.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:42.721738+0000) 2022-09-10T12:59:43.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:43.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:43.307291+0000) 2022-09-10T12:59:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:43.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:43.721918+0000) 2022-09-10T12:59:44.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:44.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:44.307500+0000) 2022-09-10T12:59:44.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:44 smithi110 ceph-mon[97633]: pgmap v16334: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:44 smithi104 ceph-mon[97044]: pgmap v16334: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:44.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:44.722089+0000) 2022-09-10T12:59:45.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:45.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:45.307794+0000) 2022-09-10T12:59:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:45.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:45.722269+0000) 2022-09-10T12:59:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:46 smithi104 ceph-mon[97044]: pgmap v16335: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:59:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:59:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:46.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:46.308055+0000) 2022-09-10T12:59:46.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 12:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T12:59:46.231+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T12:59:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:46 smithi110 ceph-mon[97633]: pgmap v16335: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:59:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T12:59:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T12:59:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:46.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:46.722481+0000) 2022-09-10T12:59:47.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:47.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:47.308363+0000) 2022-09-10T12:59:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:47.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:47.722727+0000) 2022-09-10T12:59:48.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:48.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:48.308613+0000) 2022-09-10T12:59:48.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:48 smithi110 ceph-mon[97633]: pgmap v16336: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:59:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:48 smithi104 ceph-mon[97044]: pgmap v16336: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:59:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:48.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:48.722921+0000) 2022-09-10T12:59:49.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:49.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:49.308857+0000) 2022-09-10T12:59:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:49.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:49.723166+0000) 2022-09-10T12:59:50.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:50.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:50.309083+0000) 2022-09-10T12:59:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:50 smithi110 ceph-mon[97633]: pgmap v16337: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:50 smithi104 ceph-mon[97044]: pgmap v16337: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:50.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:50.723354+0000) 2022-09-10T12:59:51.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:51.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:51.309332+0000) 2022-09-10T12:59:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:51.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:51.723575+0000) 2022-09-10T12:59:52.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:52.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:52.309533+0000) 2022-09-10T12:59:52.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:52 smithi110 ceph-mon[97633]: pgmap v16338: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:59:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:52 smithi104 ceph-mon[97044]: pgmap v16338: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:59:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:52.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:52.723730+0000) 2022-09-10T12:59:53.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:53.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:53.309759+0000) 2022-09-10T12:59:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:53.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:53.723863+0000) 2022-09-10T12:59:54.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:54.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:54.309934+0000) 2022-09-10T12:59:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:54 smithi104 ceph-mon[97044]: pgmap v16339: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:59:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:54.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:54.723989+0000) 2022-09-10T12:59:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:54 smithi110 ceph-mon[97633]: pgmap v16339: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T12:59:55.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:55.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:55.310205+0000) 2022-09-10T12:59:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:55.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:55.724238+0000) 2022-09-10T12:59:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:56 smithi104 ceph-mon[97044]: pgmap v16340: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:56 smithi110 ceph-mon[97633]: pgmap v16340: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T12:59:56.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:56.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:56.310374+0000) 2022-09-10T12:59:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:56.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:56.724460+0000) 2022-09-10T12:59:57.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:57.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:57.310600+0000) 2022-09-10T12:59:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:57.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:57.724591+0000) 2022-09-10T12:59:58.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:58.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:58.310772+0000) 2022-09-10T12:59:58.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 12:59:58 smithi110 ceph-mon[97633]: pgmap v16341: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 12:59:58 smithi104 ceph-mon[97044]: pgmap v16341: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T12:59:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:58.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:58.724768+0000) 2022-09-10T12:59:59.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 12:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T12:59:59.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:59.310947+0000) 2022-09-10T13:00:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 12:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T12:59:59.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T11:59:59.724910+0000) 2022-09-10T13:00:00.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:00.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:00.311201+0000) 2022-09-10T13:00:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:00 smithi110 ceph-mon[97633]: pgmap v16342: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:00:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:00 smithi104 ceph-mon[97044]: pgmap v16342: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:00:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:00.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:00.725048+0000) 2022-09-10T13:00:01.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:01.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:01.311373+0000) 2022-09-10T13:00:01.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:00:01.235+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:00:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:01.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:01.725175+0000) 2022-09-10T13:00:02.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:02.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:02.311574+0000) 2022-09-10T13:00:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:02 smithi104 ceph-mon[97044]: pgmap v16343: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:02.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:02.725360+0000) 2022-09-10T13:00:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:02 smithi110 ceph-mon[97633]: pgmap v16343: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:03.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:03.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:03.311853+0000) 2022-09-10T13:00:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:03.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:03.725516+0000) 2022-09-10T13:00:04.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:04.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:04.312082+0000) 2022-09-10T13:00:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:04 smithi104 ceph-mon[97044]: pgmap v16344: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:04.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:04.725704+0000) 2022-09-10T13:00:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:04 smithi110 ceph-mon[97633]: pgmap v16344: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:05.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:05.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:05.312305+0000) 2022-09-10T13:00:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:05.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:05.725875+0000) 2022-09-10T13:00:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:06 smithi104 ceph-mon[97044]: pgmap v16345: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:00:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:06 smithi110 ceph-mon[97633]: pgmap v16345: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:00:06.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:06.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:06.312510+0000) 2022-09-10T13:00:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:06.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:06.726126+0000) 2022-09-10T13:00:07.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:07.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:07.312720+0000) 2022-09-10T13:00:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:07.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:07.726335+0000) 2022-09-10T13:00:08.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:08.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:08.312936+0000) 2022-09-10T13:00:08.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:08 smithi110 ceph-mon[97633]: pgmap v16346: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:00:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:08 smithi104 ceph-mon[97044]: pgmap v16346: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:00:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:08.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:08.726508+0000) 2022-09-10T13:00:09.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:09.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:09.313190+0000) 2022-09-10T13:00:10.053 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:09 smithi104 ceph-mon[97044]: osdmap e1102: 12 total, 12 up, 12 in 2022-09-10T13:00:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:09.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:09.726660+0000) 2022-09-10T13:00:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:09 smithi110 ceph-mon[97633]: osdmap e1102: 12 total, 12 up, 12 in 2022-09-10T13:00:10.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:10.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:10.313376+0000) 2022-09-10T13:00:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:10 smithi104 ceph-mon[97044]: pgmap v16348: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:00:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:10 smithi104 ceph-mon[97044]: osdmap e1103: 12 total, 12 up, 12 in 2022-09-10T13:00:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:10.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:10.726849+0000) 2022-09-10T13:00:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:10 smithi110 ceph-mon[97633]: pgmap v16348: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:00:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:10 smithi110 ceph-mon[97633]: osdmap e1103: 12 total, 12 up, 12 in 2022-09-10T13:00:11.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:11.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:11.313605+0000) 2022-09-10T13:00:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:11.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:11.727044+0000) 2022-09-10T13:00:12.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:12.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:12.313792+0000) 2022-09-10T13:00:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:12 smithi104 ceph-mon[97044]: pgmap v16350: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:00:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:12.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:12.727166+0000) 2022-09-10T13:00:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:12 smithi110 ceph-mon[97633]: pgmap v16350: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:00:13.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:13.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:13.314005+0000) 2022-09-10T13:00:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:13.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:13.727343+0000) 2022-09-10T13:00:14.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:14.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:14.314271+0000) 2022-09-10T13:00:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:14 smithi104 ceph-mon[97044]: pgmap v16351: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:00:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:14.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:14.727529+0000) 2022-09-10T13:00:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:14 smithi110 ceph-mon[97633]: pgmap v16351: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:00:15.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:15.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:15.314507+0000) 2022-09-10T13:00:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:15.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:15.727694+0000) 2022-09-10T13:00:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:16 smithi104 ceph-mon[97044]: pgmap v16352: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:00:16.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:00:16.251+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:00:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:16.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:16.314731+0000) 2022-09-10T13:00:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:16 smithi110 ceph-mon[97633]: pgmap v16352: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:00:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:16.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:16.727911+0000) 2022-09-10T13:00:17.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:17.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:17.314991+0000) 2022-09-10T13:00:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:17.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:17.728092+0000) 2022-09-10T13:00:18.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:18.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:18.315199+0000) 2022-09-10T13:00:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:18 smithi104 ceph-mon[97044]: pgmap v16353: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T13:00:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:18.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:18.728325+0000) 2022-09-10T13:00:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:18 smithi110 ceph-mon[97633]: pgmap v16353: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T13:00:19.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:19.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:19.315449+0000) 2022-09-10T13:00:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:19.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:19.728533+0000) 2022-09-10T13:00:20.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:20.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:20.315618+0000) 2022-09-10T13:00:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:20 smithi104 ceph-mon[97044]: pgmap v16354: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:00:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:20.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:20.728725+0000) 2022-09-10T13:00:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:20 smithi110 ceph-mon[97633]: pgmap v16354: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:00:21.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:21.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:21.315845+0000) 2022-09-10T13:00:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:21.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:21.728928+0000) 2022-09-10T13:00:22.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:22.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:22.316058+0000) 2022-09-10T13:00:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:22 smithi104 ceph-mon[97044]: pgmap v16355: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:00:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:22.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:22.729126+0000) 2022-09-10T13:00:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:22 smithi110 ceph-mon[97633]: pgmap v16355: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:00:23.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:23.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:23.316280+0000) 2022-09-10T13:00:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:23.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:23.729269+0000) 2022-09-10T13:00:24.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:24.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:24.316530+0000) 2022-09-10T13:00:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:24 smithi104 ceph-mon[97044]: pgmap v16356: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:00:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:24.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:24.729492+0000) 2022-09-10T13:00:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:24 smithi110 ceph-mon[97633]: pgmap v16356: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:00:25.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:25.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:25.316802+0000) 2022-09-10T13:00:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:25.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:25.729651+0000) 2022-09-10T13:00:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:26 smithi104 ceph-mon[97044]: pgmap v16357: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:00:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:26 smithi110 ceph-mon[97633]: pgmap v16357: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:00:26.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:26.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:26.317069+0000) 2022-09-10T13:00:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:26.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:26.729860+0000) 2022-09-10T13:00:27.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:27.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:27.317300+0000) 2022-09-10T13:00:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:27.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:27.730087+0000) 2022-09-10T13:00:28.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:28.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:28.317542+0000) 2022-09-10T13:00:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:28 smithi104 ceph-mon[97044]: pgmap v16358: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:00:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:28.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:28.730292+0000) 2022-09-10T13:00:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:28 smithi110 ceph-mon[97633]: pgmap v16358: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:00:29.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:29.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:29.317725+0000) 2022-09-10T13:00:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:29.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:29.730426+0000) 2022-09-10T13:00:30.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:30.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:30.317882+0000) 2022-09-10T13:00:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:30 smithi104 ceph-mon[97044]: pgmap v16359: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:30.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:30.730655+0000) 2022-09-10T13:00:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:30 smithi110 ceph-mon[97633]: pgmap v16359: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:31.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:31.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:31.318170+0000) 2022-09-10T13:00:31.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:00:31.255+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:00:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:31.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:31.730839+0000) 2022-09-10T13:00:32.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:32.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:32.318386+0000) 2022-09-10T13:00:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:32 smithi104 ceph-mon[97044]: pgmap v16360: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:32.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:32.730971+0000) 2022-09-10T13:00:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:32 smithi110 ceph-mon[97633]: pgmap v16360: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:33.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:33.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:33.318570+0000) 2022-09-10T13:00:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:33.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:33.731153+0000) 2022-09-10T13:00:34.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:34.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:34.318734+0000) 2022-09-10T13:00:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:34 smithi104 ceph-mon[97044]: pgmap v16361: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:34.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:34.731282+0000) 2022-09-10T13:00:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:34 smithi110 ceph-mon[97633]: pgmap v16361: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:35.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:35.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:35.318985+0000) 2022-09-10T13:00:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:35.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:35.731433+0000) 2022-09-10T13:00:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:36 smithi104 ceph-mon[97044]: pgmap v16362: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:36 smithi110 ceph-mon[97633]: pgmap v16362: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:36.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:36.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:36.319260+0000) 2022-09-10T13:00:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:36.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:36.731608+0000) 2022-09-10T13:00:37.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:37.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:37.319510+0000) 2022-09-10T13:00:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:37.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:37.731729+0000) 2022-09-10T13:00:38.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:38.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:38.319739+0000) 2022-09-10T13:00:39.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:38 smithi104 ceph-mon[97044]: pgmap v16363: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:38.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:38.731948+0000) 2022-09-10T13:00:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:38 smithi110 ceph-mon[97633]: pgmap v16363: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:39.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:39.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:39.319948+0000) 2022-09-10T13:00:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:39.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:39.732268+0000) 2022-09-10T13:00:40.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:40.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:40.320195+0000) 2022-09-10T13:00:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:40 smithi104 ceph-mon[97044]: pgmap v16364: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:40.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:40.732398+0000) 2022-09-10T13:00:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:40 smithi110 ceph-mon[97633]: pgmap v16364: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:41.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:41.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:41.320473+0000) 2022-09-10T13:00:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:41.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:41.732662+0000) 2022-09-10T13:00:42.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:42.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:42.320651+0000) 2022-09-10T13:00:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:42 smithi104 ceph-mon[97044]: pgmap v16365: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:42.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:42.732788+0000) 2022-09-10T13:00:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:42 smithi110 ceph-mon[97633]: pgmap v16365: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:43.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:43.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:43.320824+0000) 2022-09-10T13:00:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:43.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:43.732911+0000) 2022-09-10T13:00:44.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:44.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:44.321065+0000) 2022-09-10T13:00:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:44 smithi104 ceph-mon[97044]: pgmap v16366: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:44.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:44.733061+0000) 2022-09-10T13:00:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:44 smithi110 ceph-mon[97633]: pgmap v16366: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:45.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:45.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:45.321279+0000) 2022-09-10T13:00:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:45.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:45.733210+0000) 2022-09-10T13:00:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:46 smithi104 ceph-mon[97044]: pgmap v16367: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:00:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:00:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:00:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:00:46.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:00:46.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:00:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:46.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:46.321523+0000) 2022-09-10T13:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:46 smithi110 ceph-mon[97633]: pgmap v16367: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:00:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:00:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:00:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:00:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:46.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:46.733367+0000) 2022-09-10T13:00:47.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:47.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:47.321751+0000) 2022-09-10T13:00:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:47.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:47.733608+0000) 2022-09-10T13:00:48.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:48.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:48.322047+0000) 2022-09-10T13:00:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:48 smithi104 ceph-mon[97044]: pgmap v16368: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:48.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:48.733832+0000) 2022-09-10T13:00:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:48 smithi110 ceph-mon[97633]: pgmap v16368: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:49.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:49.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:49.322236+0000) 2022-09-10T13:00:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:49.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:49.734006+0000) 2022-09-10T13:00:50.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:50.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:50.322477+0000) 2022-09-10T13:00:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:50 smithi104 ceph-mon[97044]: pgmap v16369: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:50.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:50.734163+0000) 2022-09-10T13:00:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:50 smithi110 ceph-mon[97633]: pgmap v16369: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:51.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:51.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:51.322703+0000) 2022-09-10T13:00:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:51.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:51.734301+0000) 2022-09-10T13:00:52.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:52.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:52.322946+0000) 2022-09-10T13:00:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:52 smithi104 ceph-mon[97044]: pgmap v16370: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:52.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:52.734434+0000) 2022-09-10T13:00:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:52 smithi110 ceph-mon[97633]: pgmap v16370: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:53.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:53.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:53.323090+0000) 2022-09-10T13:00:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:53.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:53.734585+0000) 2022-09-10T13:00:54.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:54.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:54.323260+0000) 2022-09-10T13:00:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:54 smithi104 ceph-mon[97044]: pgmap v16371: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:54.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:54.734757+0000) 2022-09-10T13:00:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:54 smithi110 ceph-mon[97633]: pgmap v16371: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:55.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:55.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:55.323492+0000) 2022-09-10T13:00:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:55.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:55.734927+0000) 2022-09-10T13:00:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:56 smithi104 ceph-mon[97044]: pgmap v16372: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:56 smithi110 ceph-mon[97633]: pgmap v16372: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:00:56.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:56.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:56.323752+0000) 2022-09-10T13:00:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:56.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:56.735117+0000) 2022-09-10T13:00:57.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:57.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:57.323906+0000) 2022-09-10T13:00:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:57.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:57.735313+0000) 2022-09-10T13:00:58.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:58.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:58.324174+0000) 2022-09-10T13:00:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:00:58 smithi104 ceph-mon[97044]: pgmap v16373: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:58.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:58.735531+0000) 2022-09-10T13:00:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:00:58 smithi110 ceph-mon[97633]: pgmap v16373: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:00:59.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:00:59.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:59.324386+0000) 2022-09-10T13:01:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:00:59.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:00:59.735756+0000) 2022-09-10T13:01:00.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:00.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:00.324618+0000) 2022-09-10T13:01:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:00 smithi104 ceph-mon[97044]: pgmap v16374: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:00.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:00.735949+0000) 2022-09-10T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:00 smithi110 ceph-mon[97633]: pgmap v16374: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:01.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:01:01.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:01:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:01.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:01.324844+0000) 2022-09-10T13:01:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:01.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:01.736116+0000) 2022-09-10T13:01:02.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:02.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:02.325045+0000) 2022-09-10T13:01:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:02 smithi104 ceph-mon[97044]: pgmap v16375: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:02.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:02.736299+0000) 2022-09-10T13:01:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:02 smithi110 ceph-mon[97633]: pgmap v16375: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:03.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:03.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:03.325286+0000) 2022-09-10T13:01:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:03.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:03.736437+0000) 2022-09-10T13:01:04.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:04.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:04.325501+0000) 2022-09-10T13:01:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:04 smithi104 ceph-mon[97044]: pgmap v16376: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:04.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:04.736591+0000) 2022-09-10T13:01:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:04 smithi110 ceph-mon[97633]: pgmap v16376: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:05.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:05.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:05.325743+0000) 2022-09-10T13:01:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:05.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:05.736791+0000) 2022-09-10T13:01:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:06 smithi104 ceph-mon[97044]: pgmap v16377: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:01:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:06 smithi110 ceph-mon[97633]: pgmap v16377: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:01:06.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:06.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:06.325983+0000) 2022-09-10T13:01:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:06.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:06.737010+0000) 2022-09-10T13:01:07.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:07.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:07.326237+0000) 2022-09-10T13:01:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:07.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:07.737258+0000) 2022-09-10T13:01:08.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:08.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:08.326483+0000) 2022-09-10T13:01:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:08 smithi104 ceph-mon[97044]: pgmap v16378: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:01:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:08.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:08.737425+0000) 2022-09-10T13:01:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:08 smithi110 ceph-mon[97633]: pgmap v16378: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:01:09.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:09.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:09.326691+0000) 2022-09-10T13:01:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:09 smithi104 ceph-mon[97044]: osdmap e1104: 12 total, 12 up, 12 in 2022-09-10T13:01:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:09.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:09.737572+0000) 2022-09-10T13:01:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:09 smithi110 ceph-mon[97633]: osdmap e1104: 12 total, 12 up, 12 in 2022-09-10T13:01:10.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:10.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:10.326824+0000) 2022-09-10T13:01:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:10 smithi104 ceph-mon[97044]: pgmap v16380: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:01:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:10 smithi104 ceph-mon[97044]: osdmap e1105: 12 total, 12 up, 12 in 2022-09-10T13:01:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:10.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:10.737718+0000) 2022-09-10T13:01:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:10 smithi110 ceph-mon[97633]: pgmap v16380: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:01:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:10 smithi110 ceph-mon[97633]: osdmap e1105: 12 total, 12 up, 12 in 2022-09-10T13:01:11.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:11.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:11.327041+0000) 2022-09-10T13:01:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:11.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:11.737864+0000) 2022-09-10T13:01:12.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:12.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:12.327310+0000) 2022-09-10T13:01:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:12 smithi104 ceph-mon[97044]: pgmap v16382: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:01:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:12.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:12.738125+0000) 2022-09-10T13:01:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:12 smithi110 ceph-mon[97633]: pgmap v16382: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:01:13.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:13.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:13.327513+0000) 2022-09-10T13:01:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:13.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:13.738303+0000) 2022-09-10T13:01:14.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:14.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:14.327756+0000) 2022-09-10T13:01:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:14 smithi104 ceph-mon[97044]: pgmap v16383: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:01:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:14.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:14.738475+0000) 2022-09-10T13:01:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:14 smithi110 ceph-mon[97633]: pgmap v16383: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:01:15.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:15.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:15.327990+0000) 2022-09-10T13:01:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:15.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:15.738695+0000) 2022-09-10T13:01:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:16 smithi104 ceph-mon[97044]: pgmap v16384: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:01:16.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:01:16.287+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:01:16.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:16.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:16.328201+0000) 2022-09-10T13:01:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:16 smithi110 ceph-mon[97633]: pgmap v16384: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:01:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:16.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:16.738901+0000) 2022-09-10T13:01:17.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:17.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:17.328437+0000) 2022-09-10T13:01:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:17.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:17.739074+0000) 2022-09-10T13:01:18.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:18.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:18.328662+0000) 2022-09-10T13:01:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:18 smithi104 ceph-mon[97044]: pgmap v16385: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T13:01:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:18.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:18.739236+0000) 2022-09-10T13:01:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:18 smithi110 ceph-mon[97633]: pgmap v16385: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T13:01:19.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:19.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:19.328901+0000) 2022-09-10T13:01:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:19.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:19.739388+0000) 2022-09-10T13:01:20.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:20.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:20.329148+0000) 2022-09-10T13:01:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:20 smithi104 ceph-mon[97044]: pgmap v16386: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:01:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:20.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:20.739516+0000) 2022-09-10T13:01:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:20 smithi110 ceph-mon[97633]: pgmap v16386: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:01:21.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:21.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:21.329363+0000) 2022-09-10T13:01:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:21.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:21.739676+0000) 2022-09-10T13:01:22.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:22.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:22.329519+0000) 2022-09-10T13:01:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:22 smithi104 ceph-mon[97044]: pgmap v16387: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:01:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:22.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:22.739842+0000) 2022-09-10T13:01:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:22 smithi110 ceph-mon[97633]: pgmap v16387: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:01:23.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:23.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:23.329708+0000) 2022-09-10T13:01:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:23.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:23.740030+0000) 2022-09-10T13:01:24.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:24.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:24.329998+0000) 2022-09-10T13:01:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:24 smithi104 ceph-mon[97044]: pgmap v16388: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:01:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:24.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:24.740235+0000) 2022-09-10T13:01:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:24 smithi110 ceph-mon[97633]: pgmap v16388: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:01:25.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:25.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:25.330237+0000) 2022-09-10T13:01:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:25.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:25.740430+0000) 2022-09-10T13:01:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:26 smithi104 ceph-mon[97044]: pgmap v16389: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:01:26.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:26 smithi110 ceph-mon[97633]: pgmap v16389: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:01:26.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:26.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:26.330466+0000) 2022-09-10T13:01:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:26.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:26.740577+0000) 2022-09-10T13:01:27.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:27.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:27.330705+0000) 2022-09-10T13:01:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:27.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:27.740793+0000) 2022-09-10T13:01:28.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:28.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:28.330951+0000) 2022-09-10T13:01:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:28 smithi104 ceph-mon[97044]: pgmap v16390: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:28.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:28.740961+0000) 2022-09-10T13:01:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:28 smithi110 ceph-mon[97633]: pgmap v16390: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:29.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:29.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:29.331180+0000) 2022-09-10T13:01:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:29.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:29.741113+0000) 2022-09-10T13:01:30.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:30.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:30.331396+0000) 2022-09-10T13:01:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:30 smithi104 ceph-mon[97044]: pgmap v16391: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:30.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:30.741242+0000) 2022-09-10T13:01:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:30 smithi110 ceph-mon[97633]: pgmap v16391: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:31.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:01:31.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:01:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:31.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:31.331649+0000) 2022-09-10T13:01:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:31.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:31.741382+0000) 2022-09-10T13:01:32.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:32.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:32.331814+0000) 2022-09-10T13:01:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:32 smithi104 ceph-mon[97044]: pgmap v16392: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:32.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:32.741514+0000) 2022-09-10T13:01:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:32 smithi110 ceph-mon[97633]: pgmap v16392: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:33.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:33.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:33.332039+0000) 2022-09-10T13:01:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:33.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:33.741663+0000) 2022-09-10T13:01:34.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:34.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:34.332293+0000) 2022-09-10T13:01:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:34 smithi104 ceph-mon[97044]: pgmap v16393: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:34.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:34.741878+0000) 2022-09-10T13:01:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:34 smithi110 ceph-mon[97633]: pgmap v16393: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:35.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:35.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:35.332505+0000) 2022-09-10T13:01:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:35.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:35.742039+0000) 2022-09-10T13:01:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:36 smithi104 ceph-mon[97044]: pgmap v16394: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:36 smithi110 ceph-mon[97633]: pgmap v16394: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:36.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:36.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:36.332731+0000) 2022-09-10T13:01:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:36.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:36.742191+0000) 2022-09-10T13:01:37.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:37.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:37.332894+0000) 2022-09-10T13:01:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:37.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:37.742368+0000) 2022-09-10T13:01:38.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:38.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:38.333153+0000) 2022-09-10T13:01:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:38 smithi104 ceph-mon[97044]: pgmap v16395: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:38.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:38.742624+0000) 2022-09-10T13:01:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:38 smithi110 ceph-mon[97633]: pgmap v16395: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:39.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:39.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:39.333381+0000) 2022-09-10T13:01:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:39.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:39.742818+0000) 2022-09-10T13:01:40.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:40.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:40.333578+0000) 2022-09-10T13:01:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:40 smithi104 ceph-mon[97044]: pgmap v16396: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:41.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:40.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:40.743029+0000) 2022-09-10T13:01:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:40 smithi110 ceph-mon[97633]: pgmap v16396: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:41.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:41.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:41.333792+0000) 2022-09-10T13:01:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:41.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:41.743291+0000) 2022-09-10T13:01:42.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:42.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:42.334010+0000) 2022-09-10T13:01:43.028 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:42.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:42.743493+0000) 2022-09-10T13:01:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:42 smithi104 ceph-mon[97044]: pgmap v16397: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:42 smithi110 ceph-mon[97633]: pgmap v16397: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:43.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:43.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:43.334213+0000) 2022-09-10T13:01:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:43.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:43.743667+0000) 2022-09-10T13:01:44.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:44.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:44.334448+0000) 2022-09-10T13:01:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:44 smithi104 ceph-mon[97044]: pgmap v16398: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:44.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:44.743878+0000) 2022-09-10T13:01:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:44 smithi110 ceph-mon[97633]: pgmap v16398: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:45.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:45.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:45.334696+0000) 2022-09-10T13:01:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:45.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:45.744013+0000) 2022-09-10T13:01:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:46 smithi104 ceph-mon[97044]: pgmap v16399: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:01:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:01:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:46 smithi110 ceph-mon[97633]: pgmap v16399: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:01:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:01:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:01:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:01:46.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:46.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:46.334900+0000) 2022-09-10T13:01:46.744 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:01:46.316+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:01:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:46.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:46.744260+0000) 2022-09-10T13:01:47.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:47.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:47.335123+0000) 2022-09-10T13:01:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:47.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:47.744432+0000) 2022-09-10T13:01:48.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:48.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:48.335334+0000) 2022-09-10T13:01:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:48 smithi104 ceph-mon[97044]: pgmap v16400: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:48.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:48.744566+0000) 2022-09-10T13:01:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:48 smithi110 ceph-mon[97633]: pgmap v16400: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:49.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:49.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:49.335564+0000) 2022-09-10T13:01:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:49.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:49.744748+0000) 2022-09-10T13:01:50.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:50.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:50.335791+0000) 2022-09-10T13:01:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:50 smithi104 ceph-mon[97044]: pgmap v16401: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:50.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:50.744929+0000) 2022-09-10T13:01:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:50 smithi110 ceph-mon[97633]: pgmap v16401: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:51.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:51.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:51.335981+0000) 2022-09-10T13:01:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:51.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:51.745075+0000) 2022-09-10T13:01:52.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:52.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:52.336229+0000) 2022-09-10T13:01:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:52 smithi104 ceph-mon[97044]: pgmap v16402: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:52.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:52.745229+0000) 2022-09-10T13:01:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:52 smithi110 ceph-mon[97633]: pgmap v16402: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:53.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:53.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:53.336446+0000) 2022-09-10T13:01:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:53.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:53.745448+0000) 2022-09-10T13:01:54.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:54.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:54.336660+0000) 2022-09-10T13:01:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:54 smithi104 ceph-mon[97044]: pgmap v16403: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:54.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:54.745629+0000) 2022-09-10T13:01:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:54 smithi110 ceph-mon[97633]: pgmap v16403: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:55.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:55.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:55.336903+0000) 2022-09-10T13:01:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:55.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:55.745838+0000) 2022-09-10T13:01:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:56 smithi104 ceph-mon[97044]: pgmap v16404: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:56 smithi110 ceph-mon[97633]: pgmap v16404: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:01:56.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:56.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:56.337133+0000) 2022-09-10T13:01:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:56.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:56.746100+0000) 2022-09-10T13:01:57.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:57.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:57.337323+0000) 2022-09-10T13:01:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:57.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:57.746268+0000) 2022-09-10T13:01:58.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:58.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:58.337512+0000) 2022-09-10T13:01:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:01:58 smithi104 ceph-mon[97044]: pgmap v16405: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:58.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:58.746414+0000) 2022-09-10T13:01:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:01:58 smithi110 ceph-mon[97633]: pgmap v16405: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:01:59.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:01:59.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:59.337796+0000) 2022-09-10T13:02:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:01:59.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:01:59.746566+0000) 2022-09-10T13:02:00.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:00.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:00.337930+0000) 2022-09-10T13:02:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:00 smithi104 ceph-mon[97044]: pgmap v16406: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:00.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:00.746696+0000) 2022-09-10T13:02:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:00 smithi110 ceph-mon[97633]: pgmap v16406: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:01.747 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:02:01.329+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:02:01.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:01.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:01.338154+0000) 2022-09-10T13:02:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:01.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:01.746937+0000) 2022-09-10T13:02:02.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:02.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:02.338359+0000) 2022-09-10T13:02:03.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:02 smithi104 ceph-mon[97044]: pgmap v16407: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:02.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:02.747125+0000) 2022-09-10T13:02:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:02 smithi110 ceph-mon[97633]: pgmap v16407: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:03.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:03.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:03.338634+0000) 2022-09-10T13:02:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:03.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:03.747257+0000) 2022-09-10T13:02:04.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:04.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:04.338907+0000) 2022-09-10T13:02:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:04 smithi104 ceph-mon[97044]: pgmap v16408: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:04.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:04.747418+0000) 2022-09-10T13:02:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:04 smithi110 ceph-mon[97633]: pgmap v16408: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:05.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:05.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:05.339085+0000) 2022-09-10T13:02:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:05.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:05.747556+0000) 2022-09-10T13:02:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:06 smithi104 ceph-mon[97044]: pgmap v16409: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:02:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:06 smithi110 ceph-mon[97633]: pgmap v16409: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:02:06.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:06.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:06.339348+0000) 2022-09-10T13:02:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:06.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:06.747682+0000) 2022-09-10T13:02:07.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:07.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:07.339578+0000) 2022-09-10T13:02:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:07.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:07.747812+0000) 2022-09-10T13:02:08.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:08.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:08.339765+0000) 2022-09-10T13:02:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:08 smithi104 ceph-mon[97044]: pgmap v16410: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:02:09.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:08.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:08.748002+0000) 2022-09-10T13:02:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:08 smithi110 ceph-mon[97633]: pgmap v16410: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:02:09.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:09.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:09.340009+0000) 2022-09-10T13:02:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:09 smithi104 ceph-mon[97044]: osdmap e1106: 12 total, 12 up, 12 in 2022-09-10T13:02:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:09.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:09.748191+0000) 2022-09-10T13:02:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:09 smithi110 ceph-mon[97633]: osdmap e1106: 12 total, 12 up, 12 in 2022-09-10T13:02:10.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:10.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:10.340255+0000) 2022-09-10T13:02:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:10 smithi104 ceph-mon[97044]: pgmap v16412: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:02:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:10 smithi104 ceph-mon[97044]: osdmap e1107: 12 total, 12 up, 12 in 2022-09-10T13:02:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:10.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:10.748347+0000) 2022-09-10T13:02:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:10 smithi110 ceph-mon[97633]: pgmap v16412: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:02:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:10 smithi110 ceph-mon[97633]: osdmap e1107: 12 total, 12 up, 12 in 2022-09-10T13:02:11.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:11.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:11.340462+0000) 2022-09-10T13:02:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:11.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:11.748518+0000) 2022-09-10T13:02:12.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:12.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:12.340679+0000) 2022-09-10T13:02:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:12 smithi104 ceph-mon[97044]: pgmap v16414: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:02:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:12.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:12.748698+0000) 2022-09-10T13:02:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:12 smithi110 ceph-mon[97633]: pgmap v16414: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:02:13.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:13.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:13.340880+0000) 2022-09-10T13:02:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:13.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:13.748845+0000) 2022-09-10T13:02:14.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:14.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:14.341138+0000) 2022-09-10T13:02:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:14 smithi104 ceph-mon[97044]: pgmap v16415: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:02:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:14.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:14.749007+0000) 2022-09-10T13:02:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:14 smithi110 ceph-mon[97633]: pgmap v16415: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:02:15.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:15.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:15.341432+0000) 2022-09-10T13:02:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:15.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:15.749177+0000) 2022-09-10T13:02:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:16 smithi104 ceph-mon[97044]: pgmap v16416: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:02:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:16 smithi110 ceph-mon[97633]: pgmap v16416: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:02:16.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:16.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:16.341627+0000) 2022-09-10T13:02:16.749 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:02:16.343+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:02:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:16.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:16.749327+0000) 2022-09-10T13:02:17.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:17.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:17.341844+0000) 2022-09-10T13:02:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:17.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:17.749509+0000) 2022-09-10T13:02:18.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:18.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:18.342141+0000) 2022-09-10T13:02:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:18 smithi104 ceph-mon[97044]: pgmap v16417: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T13:02:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:18.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:18.749640+0000) 2022-09-10T13:02:19.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:18 smithi110 ceph-mon[97633]: pgmap v16417: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 228 B/s wr, 11 op/s 2022-09-10T13:02:19.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:19.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:19.342413+0000) 2022-09-10T13:02:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:19.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:19.749778+0000) 2022-09-10T13:02:20.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:20.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:20.342625+0000) 2022-09-10T13:02:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:20 smithi104 ceph-mon[97044]: pgmap v16418: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:02:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:20.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:20.749983+0000) 2022-09-10T13:02:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:20 smithi110 ceph-mon[97633]: pgmap v16418: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:02:21.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:21.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:21.342800+0000) 2022-09-10T13:02:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:21.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:21.750194+0000) 2022-09-10T13:02:22.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:22.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:22.342980+0000) 2022-09-10T13:02:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:22 smithi104 ceph-mon[97044]: pgmap v16419: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:02:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:22.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:22.750372+0000) 2022-09-10T13:02:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:22 smithi110 ceph-mon[97633]: pgmap v16419: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:02:23.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:23.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:23.343250+0000) 2022-09-10T13:02:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:23.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:23.750584+0000) 2022-09-10T13:02:24.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:24.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:24.343536+0000) 2022-09-10T13:02:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:24 smithi104 ceph-mon[97044]: pgmap v16420: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:02:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:24.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:24.750835+0000) 2022-09-10T13:02:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:24 smithi110 ceph-mon[97633]: pgmap v16420: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:02:25.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:25.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:25.343848+0000) 2022-09-10T13:02:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:25.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:25.751027+0000) 2022-09-10T13:02:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:26 smithi104 ceph-mon[97044]: pgmap v16421: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:02:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:26 smithi110 ceph-mon[97633]: pgmap v16421: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:02:26.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:26.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:26.344011+0000) 2022-09-10T13:02:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:26.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:26.751249+0000) 2022-09-10T13:02:27.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:27.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:27.344206+0000) 2022-09-10T13:02:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:27.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:27.751375+0000) 2022-09-10T13:02:28.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:28.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:28.344409+0000) 2022-09-10T13:02:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:28 smithi104 ceph-mon[97044]: pgmap v16422: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:28.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:28.751563+0000) 2022-09-10T13:02:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:28 smithi110 ceph-mon[97633]: pgmap v16422: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:29.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:29.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:29.344646+0000) 2022-09-10T13:02:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:29.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:29.751734+0000) 2022-09-10T13:02:30.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:30.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:30.344891+0000) 2022-09-10T13:02:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:30 smithi104 ceph-mon[97044]: pgmap v16423: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:30.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:30.751870+0000) 2022-09-10T13:02:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:30 smithi110 ceph-mon[97633]: pgmap v16423: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:31.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:31.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:31.345182+0000) 2022-09-10T13:02:31.752 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:02:31.358+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:02:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:31.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:31.752006+0000) 2022-09-10T13:02:32.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:32.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:32.345414+0000) 2022-09-10T13:02:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:32 smithi104 ceph-mon[97044]: pgmap v16424: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:32.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:32.752148+0000) 2022-09-10T13:02:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:32 smithi110 ceph-mon[97633]: pgmap v16424: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:33.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:33.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:33.345660+0000) 2022-09-10T13:02:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:33.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:33.752359+0000) 2022-09-10T13:02:34.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:34.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:34.345862+0000) 2022-09-10T13:02:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:34 smithi104 ceph-mon[97044]: pgmap v16425: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:34.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:34.752523+0000) 2022-09-10T13:02:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:34 smithi110 ceph-mon[97633]: pgmap v16425: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:35.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:35.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:35.346063+0000) 2022-09-10T13:02:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:35.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:35.752656+0000) 2022-09-10T13:02:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:36 smithi104 ceph-mon[97044]: pgmap v16426: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:36 smithi110 ceph-mon[97633]: pgmap v16426: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:36.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:36.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:36.346250+0000) 2022-09-10T13:02:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:36.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:36.752886+0000) 2022-09-10T13:02:37.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:37.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:37.346491+0000) 2022-09-10T13:02:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:37.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:37.753027+0000) 2022-09-10T13:02:38.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:38.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:38.346664+0000) 2022-09-10T13:02:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:38 smithi104 ceph-mon[97044]: pgmap v16427: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:38.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:38.753171+0000) 2022-09-10T13:02:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:38 smithi110 ceph-mon[97633]: pgmap v16427: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:39.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:39.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:39.346910+0000) 2022-09-10T13:02:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:39.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:39.753316+0000) 2022-09-10T13:02:40.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:40.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:40.347176+0000) 2022-09-10T13:02:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:40 smithi104 ceph-mon[97044]: pgmap v16428: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:40.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:40.753526+0000) 2022-09-10T13:02:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:40 smithi110 ceph-mon[97633]: pgmap v16428: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:41.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:41.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:41.347385+0000) 2022-09-10T13:02:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:41.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:41.753723+0000) 2022-09-10T13:02:42.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:42.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:42.347588+0000) 2022-09-10T13:02:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:42 smithi104 ceph-mon[97044]: pgmap v16429: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:42.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:42.753927+0000) 2022-09-10T13:02:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:42 smithi110 ceph-mon[97633]: pgmap v16429: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:43.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:43.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:43.347812+0000) 2022-09-10T13:02:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:43.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:43.754055+0000) 2022-09-10T13:02:44.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:44.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:44.348092+0000) 2022-09-10T13:02:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:44 smithi104 ceph-mon[97044]: pgmap v16430: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:44.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:44.754267+0000) 2022-09-10T13:02:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:44 smithi110 ceph-mon[97633]: pgmap v16430: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:45.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:45.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:45.348383+0000) 2022-09-10T13:02:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:45.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:45.754430+0000) 2022-09-10T13:02:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:46 smithi104 ceph-mon[97044]: pgmap v16431: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:46.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:02:46.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:02:46.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:02:46.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:02:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:46 smithi110 ceph-mon[97633]: pgmap v16431: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:02:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:02:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:02:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:02:46.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:46.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:46.348564+0000) 2022-09-10T13:02:46.755 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:02:46.374+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:02:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:46.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:46.754671+0000) 2022-09-10T13:02:47.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:47.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:47.348786+0000) 2022-09-10T13:02:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:47.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:47.754896+0000) 2022-09-10T13:02:48.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:48.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:48.349043+0000) 2022-09-10T13:02:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:48 smithi104 ceph-mon[97044]: pgmap v16432: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:48.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:48.755072+0000) 2022-09-10T13:02:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:48 smithi110 ceph-mon[97633]: pgmap v16432: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:49.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:49.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:49.349284+0000) 2022-09-10T13:02:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:49.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:49.755312+0000) 2022-09-10T13:02:50.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:50.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:50.349515+0000) 2022-09-10T13:02:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:50 smithi104 ceph-mon[97044]: pgmap v16433: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:50.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:50.755469+0000) 2022-09-10T13:02:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:50 smithi110 ceph-mon[97633]: pgmap v16433: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:51.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:51.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:51.349758+0000) 2022-09-10T13:02:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:51.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:51.755637+0000) 2022-09-10T13:02:52.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:52.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:52.349989+0000) 2022-09-10T13:02:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:52 smithi104 ceph-mon[97044]: pgmap v16434: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:52.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:52.755840+0000) 2022-09-10T13:02:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:52 smithi110 ceph-mon[97633]: pgmap v16434: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:53.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:53.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:53.350243+0000) 2022-09-10T13:02:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:53.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:53.755989+0000) 2022-09-10T13:02:54.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:54.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:54.350422+0000) 2022-09-10T13:02:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:54 smithi104 ceph-mon[97044]: pgmap v16435: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:54.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:54.756169+0000) 2022-09-10T13:02:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:54 smithi110 ceph-mon[97633]: pgmap v16435: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:55.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:55.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:55.350639+0000) 2022-09-10T13:02:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:55.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:55.756371+0000) 2022-09-10T13:02:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:56 smithi104 ceph-mon[97044]: pgmap v16436: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:56 smithi110 ceph-mon[97633]: pgmap v16436: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:02:56.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:56.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:56.350835+0000) 2022-09-10T13:02:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:56.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:56.756597+0000) 2022-09-10T13:02:57.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:57.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:57.351161+0000) 2022-09-10T13:02:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:57.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:57.756798+0000) 2022-09-10T13:02:58.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:58.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:58.351336+0000) 2022-09-10T13:02:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:02:58 smithi104 ceph-mon[97044]: pgmap v16437: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:58.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:58.756990+0000) 2022-09-10T13:02:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:02:58 smithi110 ceph-mon[97633]: pgmap v16437: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:02:59.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:02:59.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:59.351495+0000) 2022-09-10T13:03:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:02:59.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:02:59.757231+0000) 2022-09-10T13:03:00.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:00.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:00.351673+0000) 2022-09-10T13:03:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:00 smithi104 ceph-mon[97044]: pgmap v16438: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:00.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:00.757377+0000) 2022-09-10T13:03:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:00 smithi110 ceph-mon[97633]: pgmap v16438: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:01.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:01.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:01.351835+0000) 2022-09-10T13:03:01.758 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:03:01.385+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:03:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:01.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:01.757518+0000) 2022-09-10T13:03:02.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:02.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:02.352033+0000) 2022-09-10T13:03:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:02 smithi104 ceph-mon[97044]: pgmap v16439: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:02.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:02.757667+0000) 2022-09-10T13:03:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:02 smithi110 ceph-mon[97633]: pgmap v16439: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:03.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:03.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:03.352261+0000) 2022-09-10T13:03:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:03.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:03.757826+0000) 2022-09-10T13:03:04.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:04.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:04.352508+0000) 2022-09-10T13:03:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:04 smithi104 ceph-mon[97044]: pgmap v16440: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:04.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:04.758008+0000) 2022-09-10T13:03:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:04 smithi110 ceph-mon[97633]: pgmap v16440: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:05.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:05.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:05.352750+0000) 2022-09-10T13:03:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:05.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:05.758211+0000) 2022-09-10T13:03:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:06 smithi104 ceph-mon[97044]: pgmap v16441: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:03:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:06 smithi110 ceph-mon[97633]: pgmap v16441: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:03:06.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:06.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:06.352908+0000) 2022-09-10T13:03:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:06.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:06.758393+0000) 2022-09-10T13:03:07.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:07.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:07.353065+0000) 2022-09-10T13:03:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:07.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:07.758548+0000) 2022-09-10T13:03:08.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:08.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:08.353304+0000) 2022-09-10T13:03:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:08 smithi104 ceph-mon[97044]: pgmap v16442: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:03:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:08.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:08.758694+0000) 2022-09-10T13:03:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:08 smithi110 ceph-mon[97633]: pgmap v16442: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:03:09.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:09.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:09.353543+0000) 2022-09-10T13:03:10.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:09 smithi104 ceph-mon[97044]: osdmap e1108: 12 total, 12 up, 12 in 2022-09-10T13:03:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:09.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:09.758919+0000) 2022-09-10T13:03:10.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:09 smithi110 ceph-mon[97633]: osdmap e1108: 12 total, 12 up, 12 in 2022-09-10T13:03:10.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:10.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:10.353788+0000) 2022-09-10T13:03:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:10 smithi104 ceph-mon[97044]: pgmap v16444: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:03:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:10.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:10.759206+0000) 2022-09-10T13:03:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:10 smithi110 ceph-mon[97633]: pgmap v16444: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:03:11.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:11.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:11.354042+0000) 2022-09-10T13:03:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:11.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:11.759368+0000) 2022-09-10T13:03:12.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:12.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:12.354241+0000) 2022-09-10T13:03:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:12 smithi104 ceph-mon[97044]: pgmap v16445: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:03:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:12.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:12.759515+0000) 2022-09-10T13:03:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:12 smithi110 ceph-mon[97633]: pgmap v16445: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:03:13.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:13.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:13.354431+0000) 2022-09-10T13:03:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:13.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:13.759660+0000) 2022-09-10T13:03:14.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:14.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:14.354606+0000) 2022-09-10T13:03:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:14 smithi104 ceph-mon[97044]: pgmap v16446: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:03:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:14.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:14.759814+0000) 2022-09-10T13:03:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:14 smithi110 ceph-mon[97633]: pgmap v16446: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:03:15.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:15.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:15.354793+0000) 2022-09-10T13:03:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:15.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:15.759988+0000) 2022-09-10T13:03:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:16 smithi104 ceph-mon[97044]: pgmap v16447: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:03:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:16 smithi110 ceph-mon[97633]: pgmap v16447: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:03:16.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:16.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:16.354998+0000) 2022-09-10T13:03:16.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:03:16.401+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:03:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:16.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:16.760130+0000) 2022-09-10T13:03:17.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:17.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:17.355257+0000) 2022-09-10T13:03:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:17.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:17.760367+0000) 2022-09-10T13:03:18.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:18.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:18.355498+0000) 2022-09-10T13:03:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:18 smithi104 ceph-mon[97044]: pgmap v16448: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:03:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:18.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:18.760574+0000) 2022-09-10T13:03:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:18 smithi110 ceph-mon[97633]: pgmap v16448: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:03:19.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:19.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:19.355697+0000) 2022-09-10T13:03:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:19.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:19.760779+0000) 2022-09-10T13:03:20.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:20.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:20.355858+0000) 2022-09-10T13:03:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:20 smithi104 ceph-mon[97044]: pgmap v16449: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 187 B/s wr, 14 op/s 2022-09-10T13:03:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:20 smithi104 ceph-mon[97044]: osdmap e1109: 12 total, 12 up, 12 in 2022-09-10T13:03:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:20.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:20.760921+0000) 2022-09-10T13:03:21.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:20 smithi110 ceph-mon[97633]: pgmap v16449: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 187 B/s wr, 14 op/s 2022-09-10T13:03:21.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:20 smithi110 ceph-mon[97633]: osdmap e1109: 12 total, 12 up, 12 in 2022-09-10T13:03:21.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:21.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:21.356076+0000) 2022-09-10T13:03:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:21.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:21.761062+0000) 2022-09-10T13:03:22.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:22.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:22.356272+0000) 2022-09-10T13:03:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:22 smithi104 ceph-mon[97044]: pgmap v16451: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:03:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:22.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:22.761225+0000) 2022-09-10T13:03:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:22 smithi110 ceph-mon[97633]: pgmap v16451: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:03:23.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:23.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:23.356475+0000) 2022-09-10T13:03:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:23.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:23.761423+0000) 2022-09-10T13:03:24.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:24.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:24.356704+0000) 2022-09-10T13:03:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:24 smithi104 ceph-mon[97044]: pgmap v16452: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:03:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:24.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:24.761587+0000) 2022-09-10T13:03:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:24 smithi110 ceph-mon[97633]: pgmap v16452: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:03:25.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:25.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:25.356993+0000) 2022-09-10T13:03:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:25.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:25.761727+0000) 2022-09-10T13:03:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:26 smithi104 ceph-mon[97044]: pgmap v16453: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:03:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:26 smithi110 ceph-mon[97633]: pgmap v16453: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:03:26.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:26.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:26.357230+0000) 2022-09-10T13:03:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:26.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:26.761903+0000) 2022-09-10T13:03:27.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:27.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:27.357500+0000) 2022-09-10T13:03:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:27.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:27.762179+0000) 2022-09-10T13:03:28.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:28.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:28.357704+0000) 2022-09-10T13:03:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:28 smithi104 ceph-mon[97044]: pgmap v16454: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:03:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:28.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:28.762389+0000) 2022-09-10T13:03:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:28 smithi110 ceph-mon[97633]: pgmap v16454: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:03:29.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:29.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:29.357858+0000) 2022-09-10T13:03:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:29.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:29.762644+0000) 2022-09-10T13:03:30.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:30.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:30.358130+0000) 2022-09-10T13:03:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:30 smithi104 ceph-mon[97044]: pgmap v16455: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:03:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:30.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:30.762817+0000) 2022-09-10T13:03:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:30 smithi110 ceph-mon[97633]: pgmap v16455: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:03:31.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:31.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:31.358317+0000) 2022-09-10T13:03:31.772 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:03:31.411+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:03:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:31.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:31.763041+0000) 2022-09-10T13:03:32.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:32.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:32.358544+0000) 2022-09-10T13:03:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:32 smithi104 ceph-mon[97044]: pgmap v16456: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-10T13:03:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:32.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:32.763259+0000) 2022-09-10T13:03:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:32 smithi110 ceph-mon[97633]: pgmap v16456: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2022-09-10T13:03:33.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:33.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:33.358765+0000) 2022-09-10T13:03:34.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:33.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:33.763422+0000) 2022-09-10T13:03:34.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:34.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:34.358979+0000) 2022-09-10T13:03:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:34 smithi104 ceph-mon[97044]: pgmap v16457: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:34.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:34.763556+0000) 2022-09-10T13:03:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:34 smithi110 ceph-mon[97633]: pgmap v16457: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:35.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:35.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:35.359197+0000) 2022-09-10T13:03:36.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:35.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:35.763693+0000) 2022-09-10T13:03:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:36 smithi104 ceph-mon[97044]: pgmap v16458: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:36 smithi110 ceph-mon[97633]: pgmap v16458: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:36.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:36.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:36.359436+0000) 2022-09-10T13:03:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:36.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:36.763827+0000) 2022-09-10T13:03:37.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:37.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:37.359583+0000) 2022-09-10T13:03:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:37.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:37.763970+0000) 2022-09-10T13:03:38.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:38.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:38.359763+0000) 2022-09-10T13:03:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:38 smithi104 ceph-mon[97044]: pgmap v16459: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:38.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:38.764120+0000) 2022-09-10T13:03:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:38 smithi110 ceph-mon[97633]: pgmap v16459: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:39.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:39.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:39.359925+0000) 2022-09-10T13:03:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:39.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:39.764253+0000) 2022-09-10T13:03:40.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:40.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:40.360148+0000) 2022-09-10T13:03:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:40 smithi104 ceph-mon[97044]: pgmap v16460: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:40.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:40.764450+0000) 2022-09-10T13:03:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:40 smithi110 ceph-mon[97633]: pgmap v16460: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:41.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:41.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:41.360346+0000) 2022-09-10T13:03:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:41.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:41.764671+0000) 2022-09-10T13:03:42.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:42.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:42.360600+0000) 2022-09-10T13:03:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:42 smithi104 ceph-mon[97044]: pgmap v16461: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:42.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:42.764806+0000) 2022-09-10T13:03:43.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:42 smithi110 ceph-mon[97633]: pgmap v16461: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:43.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:43.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:43.360852+0000) 2022-09-10T13:03:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:43.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:43.764972+0000) 2022-09-10T13:03:44.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:44.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:44.361135+0000) 2022-09-10T13:03:45.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:44 smithi104 ceph-mon[97044]: pgmap v16462: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:44.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:44.765131+0000) 2022-09-10T13:03:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:44 smithi110 ceph-mon[97633]: pgmap v16462: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:45.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:45.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:45.361357+0000) 2022-09-10T13:03:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:45.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:45.765354+0000) 2022-09-10T13:03:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:46 smithi104 ceph-mon[97044]: pgmap v16463: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:03:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:03:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:03:46.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:03:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:46 smithi110 ceph-mon[97633]: pgmap v16463: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:03:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:03:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:03:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:03:46.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:46.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:46.361543+0000) 2022-09-10T13:03:46.766 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:03:46.414+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:03:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:46.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:46.765509+0000) 2022-09-10T13:03:47.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:47.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:47.361822+0000) 2022-09-10T13:03:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:47.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:47.765677+0000) 2022-09-10T13:03:48.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:48.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:48.362056+0000) 2022-09-10T13:03:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:48 smithi104 ceph-mon[97044]: pgmap v16464: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:48.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:48.765904+0000) 2022-09-10T13:03:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:48 smithi110 ceph-mon[97633]: pgmap v16464: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:49.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:49.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:49.362244+0000) 2022-09-10T13:03:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:49.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:49.766187+0000) 2022-09-10T13:03:50.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:50.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:50.362475+0000) 2022-09-10T13:03:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:50 smithi104 ceph-mon[97044]: pgmap v16465: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:50.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:50.766351+0000) 2022-09-10T13:03:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:50 smithi110 ceph-mon[97633]: pgmap v16465: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:51.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:51.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:51.362723+0000) 2022-09-10T13:03:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:51.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:51.766544+0000) 2022-09-10T13:03:52.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:52.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:52.362943+0000) 2022-09-10T13:03:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:52 smithi104 ceph-mon[97044]: pgmap v16466: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:52.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:52.766694+0000) 2022-09-10T13:03:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:52 smithi110 ceph-mon[97633]: pgmap v16466: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:53.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:53.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:53.363129+0000) 2022-09-10T13:03:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:53.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:53.766881+0000) 2022-09-10T13:03:54.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:54.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:54.363387+0000) 2022-09-10T13:03:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:54 smithi104 ceph-mon[97044]: pgmap v16467: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:54.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:54.767022+0000) 2022-09-10T13:03:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:54 smithi110 ceph-mon[97633]: pgmap v16467: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:55.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:55.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:55.363592+0000) 2022-09-10T13:03:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:55.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:55.767170+0000) 2022-09-10T13:03:56.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:56 smithi104 ceph-mon[97044]: pgmap v16468: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:56 smithi110 ceph-mon[97633]: pgmap v16468: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:03:56.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:56.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:56.363857+0000) 2022-09-10T13:03:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:56.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:56.767317+0000) 2022-09-10T13:03:57.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:57.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:57.364140+0000) 2022-09-10T13:03:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:57.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:57.767496+0000) 2022-09-10T13:03:58.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:58.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:58.364358+0000) 2022-09-10T13:03:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:03:58 smithi104 ceph-mon[97044]: pgmap v16469: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:58.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:58.767674+0000) 2022-09-10T13:03:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:03:58 smithi110 ceph-mon[97633]: pgmap v16469: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:03:59.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:03:59.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:59.364546+0000) 2022-09-10T13:04:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:03:59.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:03:59.767927+0000) 2022-09-10T13:04:00.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:00.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:00.364684+0000) 2022-09-10T13:04:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:00 smithi104 ceph-mon[97044]: pgmap v16470: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:00.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:00.768147+0000) 2022-09-10T13:04:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:00 smithi110 ceph-mon[97633]: pgmap v16470: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:01.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:01.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:01.364902+0000) 2022-09-10T13:04:01.769 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:04:01.417+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:04:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:01.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:01.768313+0000) 2022-09-10T13:04:02.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:02.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:02.365187+0000) 2022-09-10T13:04:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:02 smithi104 ceph-mon[97044]: pgmap v16471: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:02.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:02.768453+0000) 2022-09-10T13:04:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:02 smithi110 ceph-mon[97633]: pgmap v16471: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:03.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:03.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:03.365388+0000) 2022-09-10T13:04:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:03.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:03.768613+0000) 2022-09-10T13:04:04.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:04.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:04.365580+0000) 2022-09-10T13:04:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:04 smithi104 ceph-mon[97044]: pgmap v16472: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:04.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:04.768791+0000) 2022-09-10T13:04:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:04 smithi110 ceph-mon[97633]: pgmap v16472: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:05.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:05.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:05.365778+0000) 2022-09-10T13:04:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:05.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:05.768983+0000) 2022-09-10T13:04:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:06 smithi104 ceph-mon[97044]: pgmap v16473: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:04:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:06 smithi110 ceph-mon[97633]: pgmap v16473: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:04:06.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:06.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:06.366007+0000) 2022-09-10T13:04:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:06.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:06.769218+0000) 2022-09-10T13:04:07.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:07.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:07.366302+0000) 2022-09-10T13:04:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:07.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:07.769414+0000) 2022-09-10T13:04:08.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:08.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:08.366580+0000) 2022-09-10T13:04:09.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:08 smithi104 ceph-mon[97044]: pgmap v16474: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:04:09.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:08 smithi104 ceph-mon[97044]: osdmap e1110: 12 total, 12 up, 12 in 2022-09-10T13:04:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:08.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:08.769546+0000) 2022-09-10T13:04:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:08 smithi110 ceph-mon[97633]: pgmap v16474: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:04:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:08 smithi110 ceph-mon[97633]: osdmap e1110: 12 total, 12 up, 12 in 2022-09-10T13:04:09.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:09.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:09.366825+0000) 2022-09-10T13:04:10.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:09.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:09.769752+0000) 2022-09-10T13:04:10.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:10.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:10.367046+0000) 2022-09-10T13:04:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:10 smithi104 ceph-mon[97044]: pgmap v16476: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:04:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:10 smithi104 ceph-mon[97044]: osdmap e1111: 12 total, 12 up, 12 in 2022-09-10T13:04:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:10.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:10.770004+0000) 2022-09-10T13:04:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:10 smithi110 ceph-mon[97633]: pgmap v16476: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:04:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:10 smithi110 ceph-mon[97633]: osdmap e1111: 12 total, 12 up, 12 in 2022-09-10T13:04:11.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:11.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:11.367289+0000) 2022-09-10T13:04:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:11.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:11.770189+0000) 2022-09-10T13:04:12.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:12.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:12.367514+0000) 2022-09-10T13:04:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:12 smithi104 ceph-mon[97044]: pgmap v16478: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:04:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:12.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:12.770423+0000) 2022-09-10T13:04:13.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:12 smithi110 ceph-mon[97633]: pgmap v16478: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:04:13.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:13.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:13.367691+0000) 2022-09-10T13:04:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:13.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:13.770587+0000) 2022-09-10T13:04:14.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:14.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:14.367902+0000) 2022-09-10T13:04:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:14 smithi104 ceph-mon[97044]: pgmap v16479: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:04:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:14.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:14.770750+0000) 2022-09-10T13:04:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:14 smithi110 ceph-mon[97633]: pgmap v16479: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:04:15.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:15.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:15.368120+0000) 2022-09-10T13:04:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:15.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:15.770950+0000) 2022-09-10T13:04:16.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:16 smithi104 ceph-mon[97044]: pgmap v16480: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:04:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:16 smithi110 ceph-mon[97633]: pgmap v16480: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:04:16.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:16.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:16.368344+0000) 2022-09-10T13:04:16.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:04:16.432+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:04:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:16.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:16.771197+0000) 2022-09-10T13:04:17.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:17.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:17.368583+0000) 2022-09-10T13:04:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:17.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:17.771362+0000) 2022-09-10T13:04:18.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:18.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:18.368820+0000) 2022-09-10T13:04:19.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:18 smithi104 ceph-mon[97044]: pgmap v16481: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T13:04:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:18.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:18.771514+0000) 2022-09-10T13:04:19.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:18 smithi110 ceph-mon[97633]: pgmap v16481: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T13:04:19.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:19.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:19.369042+0000) 2022-09-10T13:04:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:19.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:19.771662+0000) 2022-09-10T13:04:20.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:20.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:20.369318+0000) 2022-09-10T13:04:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:20 smithi104 ceph-mon[97044]: pgmap v16482: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:04:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:20.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:20.771810+0000) 2022-09-10T13:04:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:20 smithi110 ceph-mon[97633]: pgmap v16482: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:04:21.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:21.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:21.369488+0000) 2022-09-10T13:04:22.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:21.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:21.771943+0000) 2022-09-10T13:04:22.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:22.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:22.369656+0000) 2022-09-10T13:04:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:22 smithi104 ceph-mon[97044]: pgmap v16483: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:04:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:22.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:22.772075+0000) 2022-09-10T13:04:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:22 smithi110 ceph-mon[97633]: pgmap v16483: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:04:23.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:23.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:23.369883+0000) 2022-09-10T13:04:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:23.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:23.772266+0000) 2022-09-10T13:04:24.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:24.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:24.370090+0000) 2022-09-10T13:04:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:24 smithi104 ceph-mon[97044]: pgmap v16484: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:04:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:24.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:24.772438+0000) 2022-09-10T13:04:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:24 smithi110 ceph-mon[97633]: pgmap v16484: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:04:25.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:25.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:25.370345+0000) 2022-09-10T13:04:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:25.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:25.772575+0000) 2022-09-10T13:04:26.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:26 smithi104 ceph-mon[97044]: pgmap v16485: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:04:26.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:26 smithi110 ceph-mon[97633]: pgmap v16485: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:04:26.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:26.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:26.370549+0000) 2022-09-10T13:04:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:26.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:26.772750+0000) 2022-09-10T13:04:27.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:27.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:27.370790+0000) 2022-09-10T13:04:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:27.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:27.772934+0000) 2022-09-10T13:04:28.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:28.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:28.371051+0000) 2022-09-10T13:04:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:28 smithi104 ceph-mon[97044]: pgmap v16486: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:28.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:28.773128+0000) 2022-09-10T13:04:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:28 smithi110 ceph-mon[97633]: pgmap v16486: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:29.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:29.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:29.371319+0000) 2022-09-10T13:04:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:29.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:29.773324+0000) 2022-09-10T13:04:30.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:30.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:30.371569+0000) 2022-09-10T13:04:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:30 smithi104 ceph-mon[97044]: pgmap v16487: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:30.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:30.773535+0000) 2022-09-10T13:04:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:30 smithi110 ceph-mon[97633]: pgmap v16487: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:31.773 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:04:31.444+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:04:31.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:31.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:31.371819+0000) 2022-09-10T13:04:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:31.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:31.773731+0000) 2022-09-10T13:04:32.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:32.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:32.371986+0000) 2022-09-10T13:04:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:32 smithi104 ceph-mon[97044]: pgmap v16488: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:32.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:32.773866+0000) 2022-09-10T13:04:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:32 smithi110 ceph-mon[97633]: pgmap v16488: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:33.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:33.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:33.372223+0000) 2022-09-10T13:04:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:33.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:33.774077+0000) 2022-09-10T13:04:34.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:34.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:34.372406+0000) 2022-09-10T13:04:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:34 smithi104 ceph-mon[97044]: pgmap v16489: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:34.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:34.774269+0000) 2022-09-10T13:04:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:34 smithi110 ceph-mon[97633]: pgmap v16489: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:35.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:35.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:35.372603+0000) 2022-09-10T13:04:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:35.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:35.774439+0000) 2022-09-10T13:04:36.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:36 smithi104 ceph-mon[97044]: pgmap v16490: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:36 smithi110 ceph-mon[97633]: pgmap v16490: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:36.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:36.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:36.372849+0000) 2022-09-10T13:04:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:36.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:36.774655+0000) 2022-09-10T13:04:37.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:37.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:37.373127+0000) 2022-09-10T13:04:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:37.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:37.774849+0000) 2022-09-10T13:04:38.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:38.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:38.373299+0000) 2022-09-10T13:04:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:38 smithi104 ceph-mon[97044]: pgmap v16491: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:38.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:38.775037+0000) 2022-09-10T13:04:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:38 smithi110 ceph-mon[97633]: pgmap v16491: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:39.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:39.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:39.373520+0000) 2022-09-10T13:04:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:39.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:39.775207+0000) 2022-09-10T13:04:40.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:40.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:40.373781+0000) 2022-09-10T13:04:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:40 smithi104 ceph-mon[97044]: pgmap v16492: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:40.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:40.775404+0000) 2022-09-10T13:04:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:40 smithi110 ceph-mon[97633]: pgmap v16492: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:41.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:41.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:41.373951+0000) 2022-09-10T13:04:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:41.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:41.775591+0000) 2022-09-10T13:04:42.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:42.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:42.374170+0000) 2022-09-10T13:04:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:42 smithi104 ceph-mon[97044]: pgmap v16493: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:42.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:42.775794+0000) 2022-09-10T13:04:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:42 smithi110 ceph-mon[97633]: pgmap v16493: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:43.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:43.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:43.374428+0000) 2022-09-10T13:04:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:43.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:43.775927+0000) 2022-09-10T13:04:44.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:44.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:44.374604+0000) 2022-09-10T13:04:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:44 smithi104 ceph-mon[97044]: pgmap v16494: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:44.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:44.776178+0000) 2022-09-10T13:04:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:44 smithi110 ceph-mon[97633]: pgmap v16494: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:45.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:45.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:45.374844+0000) 2022-09-10T13:04:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:45.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:45.776399+0000) 2022-09-10T13:04:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:46 smithi104 ceph-mon[97044]: pgmap v16495: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:46.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:04:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:04:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:04:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:04:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:46 smithi110 ceph-mon[97633]: pgmap v16495: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:04:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:04:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:04:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:04:46.776 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:04:46.460+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:04:46.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:46.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:46.375154+0000) 2022-09-10T13:04:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:46.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:46.776581+0000) 2022-09-10T13:04:47.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:47.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:47.375351+0000) 2022-09-10T13:04:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:47.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:47.776727+0000) 2022-09-10T13:04:48.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:48.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:48.375602+0000) 2022-09-10T13:04:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:48 smithi104 ceph-mon[97044]: pgmap v16496: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:48.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:48.776866+0000) 2022-09-10T13:04:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:48 smithi110 ceph-mon[97633]: pgmap v16496: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:49.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:49.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:49.375808+0000) 2022-09-10T13:04:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:49.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:49.777071+0000) 2022-09-10T13:04:50.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:50.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:50.376078+0000) 2022-09-10T13:04:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:50 smithi104 ceph-mon[97044]: pgmap v16497: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:50.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:50.777282+0000) 2022-09-10T13:04:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:50 smithi110 ceph-mon[97633]: pgmap v16497: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:51.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:51.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:51.376288+0000) 2022-09-10T13:04:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:51.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:51.777416+0000) 2022-09-10T13:04:52.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:52.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:52.376534+0000) 2022-09-10T13:04:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:52 smithi104 ceph-mon[97044]: pgmap v16498: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:52.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:52.777655+0000) 2022-09-10T13:04:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:52 smithi110 ceph-mon[97633]: pgmap v16498: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:53.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:53.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:53.376728+0000) 2022-09-10T13:04:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:53.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:53.777785+0000) 2022-09-10T13:04:54.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:54.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:54.376955+0000) 2022-09-10T13:04:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:54 smithi104 ceph-mon[97044]: pgmap v16499: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:54.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:54.777926+0000) 2022-09-10T13:04:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:54 smithi110 ceph-mon[97633]: pgmap v16499: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:55.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:55.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:55.377192+0000) 2022-09-10T13:04:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:55.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:55.778115+0000) 2022-09-10T13:04:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:56 smithi110 ceph-mon[97633]: pgmap v16500: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:56.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:56.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:56.377454+0000) 2022-09-10T13:04:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:56 smithi104 ceph-mon[97044]: pgmap v16500: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:04:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:56.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:56.778217+0000) 2022-09-10T13:04:57.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:57.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:57.377679+0000) 2022-09-10T13:04:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:57.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:57.778403+0000) 2022-09-10T13:04:58.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:58.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:58.377930+0000) 2022-09-10T13:04:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:04:58 smithi104 ceph-mon[97044]: pgmap v16501: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:58.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:58.778563+0000) 2022-09-10T13:04:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:04:58 smithi110 ceph-mon[97633]: pgmap v16501: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:04:59.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:04:59.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:59.378209+0000) 2022-09-10T13:05:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:04:59.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:04:59.778747+0000) 2022-09-10T13:05:00.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:00.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:00.378461+0000) 2022-09-10T13:05:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:00 smithi104 ceph-mon[97044]: pgmap v16502: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:00.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:00.778888+0000) 2022-09-10T13:05:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:00 smithi110 ceph-mon[97633]: pgmap v16502: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:01.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:01.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:01.378661+0000) 2022-09-10T13:05:01.779 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:05:01.475+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:05:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:01.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:01.779059+0000) 2022-09-10T13:05:02.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:02.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:02.378906+0000) 2022-09-10T13:05:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:02 smithi104 ceph-mon[97044]: pgmap v16503: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:02.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:02.779285+0000) 2022-09-10T13:05:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:02 smithi110 ceph-mon[97633]: pgmap v16503: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:03.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:03.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:03.379196+0000) 2022-09-10T13:05:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:03.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:03.779424+0000) 2022-09-10T13:05:04.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:04.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:04.379395+0000) 2022-09-10T13:05:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:04 smithi104 ceph-mon[97044]: pgmap v16504: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:04.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:04.779614+0000) 2022-09-10T13:05:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:04 smithi110 ceph-mon[97633]: pgmap v16504: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:05.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:05.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:05.379579+0000) 2022-09-10T13:05:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:05.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:05.779821+0000) 2022-09-10T13:05:06.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:06 smithi104 ceph-mon[97044]: pgmap v16505: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:05:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:06 smithi110 ceph-mon[97633]: pgmap v16505: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:05:06.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:06.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:06.379736+0000) 2022-09-10T13:05:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:06.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:06.779976+0000) 2022-09-10T13:05:07.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:07.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:07.379964+0000) 2022-09-10T13:05:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:07.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:07.780136+0000) 2022-09-10T13:05:08.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:08.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:08.380197+0000) 2022-09-10T13:05:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:08 smithi104 ceph-mon[97044]: pgmap v16506: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:05:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:08 smithi104 ceph-mon[97044]: osdmap e1112: 12 total, 12 up, 12 in 2022-09-10T13:05:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:08.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:08.780292+0000) 2022-09-10T13:05:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:08 smithi110 ceph-mon[97633]: pgmap v16506: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:05:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:08 smithi110 ceph-mon[97633]: osdmap e1112: 12 total, 12 up, 12 in 2022-09-10T13:05:09.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:09.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:09.380385+0000) 2022-09-10T13:05:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:09.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:09.780446+0000) 2022-09-10T13:05:10.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:10.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:10.380619+0000) 2022-09-10T13:05:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:10 smithi104 ceph-mon[97044]: pgmap v16508: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:05:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:10 smithi104 ceph-mon[97044]: osdmap e1113: 12 total, 12 up, 12 in 2022-09-10T13:05:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:10.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:10.780577+0000) 2022-09-10T13:05:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:10 smithi110 ceph-mon[97633]: pgmap v16508: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:05:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:10 smithi110 ceph-mon[97633]: osdmap e1113: 12 total, 12 up, 12 in 2022-09-10T13:05:11.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:11.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:11.380864+0000) 2022-09-10T13:05:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:11.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:11.780703+0000) 2022-09-10T13:05:12.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:12.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:12.381117+0000) 2022-09-10T13:05:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:12 smithi104 ceph-mon[97044]: pgmap v16510: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:05:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:12.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:12.780839+0000) 2022-09-10T13:05:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:12 smithi110 ceph-mon[97633]: pgmap v16510: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:05:13.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:13.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:13.381297+0000) 2022-09-10T13:05:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:13.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:13.780974+0000) 2022-09-10T13:05:14.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:14.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:14.381527+0000) 2022-09-10T13:05:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:14 smithi104 ceph-mon[97044]: pgmap v16511: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:05:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:14.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:14.781192+0000) 2022-09-10T13:05:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:14 smithi110 ceph-mon[97633]: pgmap v16511: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:05:15.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:15.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:15.381675+0000) 2022-09-10T13:05:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:15.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:15.781385+0000) 2022-09-10T13:05:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:16 smithi110 ceph-mon[97633]: pgmap v16512: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:05:16.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:16.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:16.381879+0000) 2022-09-10T13:05:16.782 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:05:16.483+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:05:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:16 smithi104 ceph-mon[97044]: pgmap v16512: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:05:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:16.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:16.781590+0000) 2022-09-10T13:05:17.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:17.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:17.382170+0000) 2022-09-10T13:05:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:17.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:17.781817+0000) 2022-09-10T13:05:18.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:18.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:18.382407+0000) 2022-09-10T13:05:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:18 smithi104 ceph-mon[97044]: pgmap v16513: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T13:05:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:18.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:18.782032+0000) 2022-09-10T13:05:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:18 smithi110 ceph-mon[97633]: pgmap v16513: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T13:05:19.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:19.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:19.382655+0000) 2022-09-10T13:05:20.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:19.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:19.782167+0000) 2022-09-10T13:05:20.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:20.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:20.382899+0000) 2022-09-10T13:05:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:20 smithi104 ceph-mon[97044]: pgmap v16514: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:05:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:20.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:20.782339+0000) 2022-09-10T13:05:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:20 smithi110 ceph-mon[97633]: pgmap v16514: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:05:21.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:21.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:21.383129+0000) 2022-09-10T13:05:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:21.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:21.782509+0000) 2022-09-10T13:05:22.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:22.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:22.383268+0000) 2022-09-10T13:05:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:22 smithi104 ceph-mon[97044]: pgmap v16515: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:05:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:22.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:22.782645+0000) 2022-09-10T13:05:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:22 smithi110 ceph-mon[97633]: pgmap v16515: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:05:23.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:23.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:23.383478+0000) 2022-09-10T13:05:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:23.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:23.782867+0000) 2022-09-10T13:05:24.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:24.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:24.383754+0000) 2022-09-10T13:05:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:24 smithi104 ceph-mon[97044]: pgmap v16516: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:05:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:24.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:24.783129+0000) 2022-09-10T13:05:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:24 smithi110 ceph-mon[97633]: pgmap v16516: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:05:25.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:25.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:25.383986+0000) 2022-09-10T13:05:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:25.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:25.783327+0000) 2022-09-10T13:05:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:26 smithi110 ceph-mon[97633]: pgmap v16517: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:05:26.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:26.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:26.384191+0000) 2022-09-10T13:05:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:26 smithi104 ceph-mon[97044]: pgmap v16517: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:05:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:26.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:26.783542+0000) 2022-09-10T13:05:27.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:27.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:27.384409+0000) 2022-09-10T13:05:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:27.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:27.783702+0000) 2022-09-10T13:05:28.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:28.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:28.384606+0000) 2022-09-10T13:05:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:28 smithi104 ceph-mon[97044]: pgmap v16518: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:05:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:28.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:28.783838+0000) 2022-09-10T13:05:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:28 smithi110 ceph-mon[97633]: pgmap v16518: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:05:29.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:29.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:29.384769+0000) 2022-09-10T13:05:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:29.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:29.784027+0000) 2022-09-10T13:05:30.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:30.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:30.385038+0000) 2022-09-10T13:05:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:30 smithi104 ceph-mon[97044]: pgmap v16519: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:30.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:30.784244+0000) 2022-09-10T13:05:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:30 smithi110 ceph-mon[97633]: pgmap v16519: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:31.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:31.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:31.385284+0000) 2022-09-10T13:05:31.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:05:31.488+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:05:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:31.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:31.784403+0000) 2022-09-10T13:05:32.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:32.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:32.385516+0000) 2022-09-10T13:05:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:32 smithi104 ceph-mon[97044]: pgmap v16520: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:32.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:32.784612+0000) 2022-09-10T13:05:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:32 smithi110 ceph-mon[97633]: pgmap v16520: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:33.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:33.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:33.385713+0000) 2022-09-10T13:05:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:33.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:33.784771+0000) 2022-09-10T13:05:34.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:34.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:34.385970+0000) 2022-09-10T13:05:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:34 smithi104 ceph-mon[97044]: pgmap v16521: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:34.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:34.784906+0000) 2022-09-10T13:05:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:34 smithi110 ceph-mon[97633]: pgmap v16521: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:35.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:35.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:35.386219+0000) 2022-09-10T13:05:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:35.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:35.785045+0000) 2022-09-10T13:05:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:36 smithi110 ceph-mon[97633]: pgmap v16522: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:36.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:36.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:36.386402+0000) 2022-09-10T13:05:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:36 smithi104 ceph-mon[97044]: pgmap v16522: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:36.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:36.785204+0000) 2022-09-10T13:05:37.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:37.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:37.386572+0000) 2022-09-10T13:05:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:37.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:37.785398+0000) 2022-09-10T13:05:38.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:38.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:38.386864+0000) 2022-09-10T13:05:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:38 smithi104 ceph-mon[97044]: pgmap v16523: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:38.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:38.785596+0000) 2022-09-10T13:05:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:38 smithi110 ceph-mon[97633]: pgmap v16523: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:39.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:39.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:39.387044+0000) 2022-09-10T13:05:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:39.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:39.785833+0000) 2022-09-10T13:05:40.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:40.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:40.387296+0000) 2022-09-10T13:05:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:40 smithi104 ceph-mon[97044]: pgmap v16524: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:40.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:40.786050+0000) 2022-09-10T13:05:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:40 smithi110 ceph-mon[97633]: pgmap v16524: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:41.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:41.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:41.387473+0000) 2022-09-10T13:05:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:41.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:41.786234+0000) 2022-09-10T13:05:42.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:42.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:42.387621+0000) 2022-09-10T13:05:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:42 smithi104 ceph-mon[97044]: pgmap v16525: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:42.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:42.786466+0000) 2022-09-10T13:05:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:42 smithi110 ceph-mon[97633]: pgmap v16525: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:43.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:43.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:43.387784+0000) 2022-09-10T13:05:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:43.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:43.786693+0000) 2022-09-10T13:05:44.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:44.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:44.387960+0000) 2022-09-10T13:05:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:44 smithi104 ceph-mon[97044]: pgmap v16526: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:44.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:44.786932+0000) 2022-09-10T13:05:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:44 smithi110 ceph-mon[97633]: pgmap v16526: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:45.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:45.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:45.388176+0000) 2022-09-10T13:05:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:45.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:45.787149+0000) 2022-09-10T13:05:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:46 smithi110 ceph-mon[97633]: pgmap v16527: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:05:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:05:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:05:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:05:46.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:46.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:46.388431+0000) 2022-09-10T13:05:46.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:05:46.503+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:05:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:46 smithi104 ceph-mon[97044]: pgmap v16527: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:05:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:05:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:05:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:05:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:46.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:46.787371+0000) 2022-09-10T13:05:47.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:47.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:47.388596+0000) 2022-09-10T13:05:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:47.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:47.787597+0000) 2022-09-10T13:05:48.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:48.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:48.388842+0000) 2022-09-10T13:05:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:48 smithi104 ceph-mon[97044]: pgmap v16528: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:48.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:48.787861+0000) 2022-09-10T13:05:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:48 smithi110 ceph-mon[97633]: pgmap v16528: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:49.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:49.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:49.389055+0000) 2022-09-10T13:05:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:49.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:49.788067+0000) 2022-09-10T13:05:50.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:50.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:50.389287+0000) 2022-09-10T13:05:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:50 smithi104 ceph-mon[97044]: pgmap v16529: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:50.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:50.788291+0000) 2022-09-10T13:05:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:50 smithi110 ceph-mon[97633]: pgmap v16529: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:51.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:51.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:51.389535+0000) 2022-09-10T13:05:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:51.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:51.788451+0000) 2022-09-10T13:05:52.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:52.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:52.389813+0000) 2022-09-10T13:05:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:52 smithi104 ceph-mon[97044]: pgmap v16530: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:52.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:52.788617+0000) 2022-09-10T13:05:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:52 smithi110 ceph-mon[97633]: pgmap v16530: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:53.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:53.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:53.390038+0000) 2022-09-10T13:05:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:53.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:53.788827+0000) 2022-09-10T13:05:54.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:54.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:54.390288+0000) 2022-09-10T13:05:55.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:54 smithi104 ceph-mon[97044]: pgmap v16531: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:54.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:54.789005+0000) 2022-09-10T13:05:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:54 smithi110 ceph-mon[97633]: pgmap v16531: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:55.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:55.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:55.390503+0000) 2022-09-10T13:05:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:55.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:55.789238+0000) 2022-09-10T13:05:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:56 smithi110 ceph-mon[97633]: pgmap v16532: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:56.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:56.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:56.390763+0000) 2022-09-10T13:05:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:56 smithi104 ceph-mon[97044]: pgmap v16532: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:05:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:56.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:56.789444+0000) 2022-09-10T13:05:57.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:57.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:57.390978+0000) 2022-09-10T13:05:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:57.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:57.789588+0000) 2022-09-10T13:05:58.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:58.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:58.391204+0000) 2022-09-10T13:05:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:05:58 smithi104 ceph-mon[97044]: pgmap v16533: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:58.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:58.789744+0000) 2022-09-10T13:05:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:05:58 smithi110 ceph-mon[97633]: pgmap v16533: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:05:59.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:05:59.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:59.391409+0000) 2022-09-10T13:06:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:05:59.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:05:59.789969+0000) 2022-09-10T13:06:00.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:00.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:00.391611+0000) 2022-09-10T13:06:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:00 smithi104 ceph-mon[97044]: pgmap v16534: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:00.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:00.790180+0000) 2022-09-10T13:06:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:00 smithi110 ceph-mon[97633]: pgmap v16534: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:01.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:01.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:01.391820+0000) 2022-09-10T13:06:01.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:06:01.507+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:06:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:01.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:01.790299+0000) 2022-09-10T13:06:02.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:02.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:02.392077+0000) 2022-09-10T13:06:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:02 smithi104 ceph-mon[97044]: pgmap v16535: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:02.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:02.790421+0000) 2022-09-10T13:06:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:02 smithi110 ceph-mon[97633]: pgmap v16535: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:03.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:03.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:03.392335+0000) 2022-09-10T13:06:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:03.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:03.790631+0000) 2022-09-10T13:06:04.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:04.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:04.392561+0000) 2022-09-10T13:06:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:04 smithi104 ceph-mon[97044]: pgmap v16536: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:04.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:04.790798+0000) 2022-09-10T13:06:05.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:04 smithi110 ceph-mon[97633]: pgmap v16536: 129 pgs: 129 active+clean; 9.5 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:05.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:05.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:05.392785+0000) 2022-09-10T13:06:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:05.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:05.790956+0000) 2022-09-10T13:06:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:06 smithi110 ceph-mon[97633]: pgmap v16537: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:06:06.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:06.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:06.393035+0000) 2022-09-10T13:06:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:06 smithi104 ceph-mon[97044]: pgmap v16537: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:06:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:06.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:06.791177+0000) 2022-09-10T13:06:07.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:07.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:07.393258+0000) 2022-09-10T13:06:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:07.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:07.791402+0000) 2022-09-10T13:06:08.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:08.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:08.393527+0000) 2022-09-10T13:06:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:08 smithi104 ceph-mon[97044]: pgmap v16538: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:06:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:08 smithi104 ceph-mon[97044]: osdmap e1114: 12 total, 12 up, 12 in 2022-09-10T13:06:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:08.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:08.791635+0000) 2022-09-10T13:06:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:08 smithi110 ceph-mon[97633]: pgmap v16538: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:06:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:08 smithi110 ceph-mon[97633]: osdmap e1114: 12 total, 12 up, 12 in 2022-09-10T13:06:09.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:09.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:09.393784+0000) 2022-09-10T13:06:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:09.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:09.791810+0000) 2022-09-10T13:06:10.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:10.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:10.394044+0000) 2022-09-10T13:06:11.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:10 smithi104 ceph-mon[97044]: pgmap v16540: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:06:11.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:10 smithi104 ceph-mon[97044]: osdmap e1115: 12 total, 12 up, 12 in 2022-09-10T13:06:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:10.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:10.792016+0000) 2022-09-10T13:06:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:10 smithi110 ceph-mon[97633]: pgmap v16540: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:06:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:10 smithi110 ceph-mon[97633]: osdmap e1115: 12 total, 12 up, 12 in 2022-09-10T13:06:11.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:11.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:11.394296+0000) 2022-09-10T13:06:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:11.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:11.792234+0000) 2022-09-10T13:06:12.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:12.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:12.394559+0000) 2022-09-10T13:06:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:12 smithi104 ceph-mon[97044]: pgmap v16542: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:06:13.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:12.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:12.792374+0000) 2022-09-10T13:06:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:12 smithi110 ceph-mon[97633]: pgmap v16542: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:06:13.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:13.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:13.394804+0000) 2022-09-10T13:06:14.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:13.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:13.792531+0000) 2022-09-10T13:06:14.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:14.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:14.395041+0000) 2022-09-10T13:06:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:14 smithi104 ceph-mon[97044]: pgmap v16543: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:06:15.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:14.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:14.792737+0000) 2022-09-10T13:06:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:14 smithi110 ceph-mon[97633]: pgmap v16543: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:06:15.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:15.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:15.395253+0000) 2022-09-10T13:06:16.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:15.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:15.792935+0000) 2022-09-10T13:06:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:16 smithi110 ceph-mon[97633]: pgmap v16544: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:06:16.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:16.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:16.395481+0000) 2022-09-10T13:06:16.793 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:06:16.511+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:06:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:16 smithi104 ceph-mon[97044]: pgmap v16544: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:06:17.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:16.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:16.793091+0000) 2022-09-10T13:06:17.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:17.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:17.395728+0000) 2022-09-10T13:06:18.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:17.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:17.793263+0000) 2022-09-10T13:06:18.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:18.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:18.395977+0000) 2022-09-10T13:06:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:18 smithi104 ceph-mon[97044]: pgmap v16545: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T13:06:19.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:18.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:18.793448+0000) 2022-09-10T13:06:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:18 smithi110 ceph-mon[97633]: pgmap v16545: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T13:06:19.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:19.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:19.396271+0000) 2022-09-10T13:06:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:19.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:19.793597+0000) 2022-09-10T13:06:20.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:20.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:20.396430+0000) 2022-09-10T13:06:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:20 smithi104 ceph-mon[97044]: pgmap v16546: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:06:21.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:20.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:20.793730+0000) 2022-09-10T13:06:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:20 smithi110 ceph-mon[97633]: pgmap v16546: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:06:21.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:21.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:21.396613+0000) 2022-09-10T13:06:22.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:21.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:21.793879+0000) 2022-09-10T13:06:22.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:22.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:22.396818+0000) 2022-09-10T13:06:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:22 smithi104 ceph-mon[97044]: pgmap v16547: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:06:23.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:22.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:22.794070+0000) 2022-09-10T13:06:23.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:22 smithi110 ceph-mon[97633]: pgmap v16547: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:06:23.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:23.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:23.397065+0000) 2022-09-10T13:06:24.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:23.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:23.794218+0000) 2022-09-10T13:06:24.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:24.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:24.397271+0000) 2022-09-10T13:06:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:24 smithi104 ceph-mon[97044]: pgmap v16548: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:06:25.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:24.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:24.794357+0000) 2022-09-10T13:06:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:24 smithi110 ceph-mon[97633]: pgmap v16548: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:06:25.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:25.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:25.397465+0000) 2022-09-10T13:06:26.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:25.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:25.794579+0000) 2022-09-10T13:06:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:26 smithi110 ceph-mon[97633]: pgmap v16549: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:06:26.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:26.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:26.397696+0000) 2022-09-10T13:06:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:26 smithi104 ceph-mon[97044]: pgmap v16549: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:06:27.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:26.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:26.794785+0000) 2022-09-10T13:06:27.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:27.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:27.397908+0000) 2022-09-10T13:06:28.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:27.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:27.795017+0000) 2022-09-10T13:06:28.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:28.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:28.398176+0000) 2022-09-10T13:06:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:28 smithi104 ceph-mon[97044]: pgmap v16550: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:29.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:28.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:28.795172+0000) 2022-09-10T13:06:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:28 smithi110 ceph-mon[97633]: pgmap v16550: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:29.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:29.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:29.398404+0000) 2022-09-10T13:06:30.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:29.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:29.795373+0000) 2022-09-10T13:06:30.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:30.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:30.398571+0000) 2022-09-10T13:06:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:30 smithi104 ceph-mon[97044]: pgmap v16551: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:31.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:30.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:30.795594+0000) 2022-09-10T13:06:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:30 smithi110 ceph-mon[97633]: pgmap v16551: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:31.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:31.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:31.398811+0000) 2022-09-10T13:06:31.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:06:31.514+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:06:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:31.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:31.795733+0000) 2022-09-10T13:06:32.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:32.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:32.399048+0000) 2022-09-10T13:06:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:32 smithi104 ceph-mon[97044]: pgmap v16552: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:33.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:32.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:32.795913+0000) 2022-09-10T13:06:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:32 smithi110 ceph-mon[97633]: pgmap v16552: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:33.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:33.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:33.399222+0000) 2022-09-10T13:06:34.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:33.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:33.796165+0000) 2022-09-10T13:06:34.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:34.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:34.399454+0000) 2022-09-10T13:06:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:34 smithi104 ceph-mon[97044]: pgmap v16553: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:35.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:34.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:34.796396+0000) 2022-09-10T13:06:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:34 smithi110 ceph-mon[97633]: pgmap v16553: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:35.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:35.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:35.399726+0000) 2022-09-10T13:06:36.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:35.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:35.796588+0000) 2022-09-10T13:06:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:36 smithi110 ceph-mon[97633]: pgmap v16554: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:36 smithi104 ceph-mon[97044]: pgmap v16554: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:36.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:36.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:36.399971+0000) 2022-09-10T13:06:37.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:36.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:36.796814+0000) 2022-09-10T13:06:37.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:37.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:37.400156+0000) 2022-09-10T13:06:38.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:37.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:37.796984+0000) 2022-09-10T13:06:38.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:38.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:38.400388+0000) 2022-09-10T13:06:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:38 smithi104 ceph-mon[97044]: pgmap v16555: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:39.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:38.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:38.797231+0000) 2022-09-10T13:06:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:38 smithi110 ceph-mon[97633]: pgmap v16555: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:39.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:39.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:39.400669+0000) 2022-09-10T13:06:40.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:39.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:39.797405+0000) 2022-09-10T13:06:40.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:40.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:40.400933+0000) 2022-09-10T13:06:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:40 smithi104 ceph-mon[97044]: pgmap v16556: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:41.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:40.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:40.797544+0000) 2022-09-10T13:06:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:40 smithi110 ceph-mon[97633]: pgmap v16556: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:41.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:41.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:41.401184+0000) 2022-09-10T13:06:42.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:41.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:41.797741+0000) 2022-09-10T13:06:42.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:42.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:42.401321+0000) 2022-09-10T13:06:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:42 smithi104 ceph-mon[97044]: pgmap v16557: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:43.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:42.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:42.797871+0000) 2022-09-10T13:06:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:42 smithi110 ceph-mon[97633]: pgmap v16557: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:43.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:43.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:43.401551+0000) 2022-09-10T13:06:44.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:43.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:43.798053+0000) 2022-09-10T13:06:44.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:44.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:44.401692+0000) 2022-09-10T13:06:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:44 smithi104 ceph-mon[97044]: pgmap v16558: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:45.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:44.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:44.798262+0000) 2022-09-10T13:06:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:44 smithi110 ceph-mon[97633]: pgmap v16558: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:45.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:45.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:45.401860+0000) 2022-09-10T13:06:46.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:45.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:45.798400+0000) 2022-09-10T13:06:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:46 smithi110 ceph-mon[97633]: pgmap v16559: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:06:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:06:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:06:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:06:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:46 smithi104 ceph-mon[97044]: pgmap v16559: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:06:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:06:46.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:46.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:46.402057+0000) 2022-09-10T13:06:46.799 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:06:46.519+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:06:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:46.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:46.798538+0000) 2022-09-10T13:06:47.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:47.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:47.402308+0000) 2022-09-10T13:06:48.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:47.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:47.798726+0000) 2022-09-10T13:06:48.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:48.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:48.402509+0000) 2022-09-10T13:06:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:48 smithi104 ceph-mon[97044]: pgmap v16560: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:49.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:48.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:48.798935+0000) 2022-09-10T13:06:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:48 smithi110 ceph-mon[97633]: pgmap v16560: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:49.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:49.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:49.402729+0000) 2022-09-10T13:06:50.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:49.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:49.799192+0000) 2022-09-10T13:06:50.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:50.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:50.402981+0000) 2022-09-10T13:06:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:50 smithi104 ceph-mon[97044]: pgmap v16561: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:51.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:50.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:50.799339+0000) 2022-09-10T13:06:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:50 smithi110 ceph-mon[97633]: pgmap v16561: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:51.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:51.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:51.403226+0000) 2022-09-10T13:06:52.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:51.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:51.799493+0000) 2022-09-10T13:06:52.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:52.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:52.403484+0000) 2022-09-10T13:06:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:52 smithi104 ceph-mon[97044]: pgmap v16562: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:53.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:52.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:52.799662+0000) 2022-09-10T13:06:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:52 smithi110 ceph-mon[97633]: pgmap v16562: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:53.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:53.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:53.403611+0000) 2022-09-10T13:06:54.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:53.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:53.799828+0000) 2022-09-10T13:06:54.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:54.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:54.403844+0000) 2022-09-10T13:06:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:54 smithi104 ceph-mon[97044]: pgmap v16563: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:55.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:54.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:54.799956+0000) 2022-09-10T13:06:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:54 smithi110 ceph-mon[97633]: pgmap v16563: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:55.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:55.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:55.404070+0000) 2022-09-10T13:06:56.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:55.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:55.800169+0000) 2022-09-10T13:06:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:56 smithi110 ceph-mon[97633]: pgmap v16564: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:56 smithi104 ceph-mon[97044]: pgmap v16564: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:06:56.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:56.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:56.404285+0000) 2022-09-10T13:06:57.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:56.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:56.800347+0000) 2022-09-10T13:06:57.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:57.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:57.404508+0000) 2022-09-10T13:06:58.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:57.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:57.800524+0000) 2022-09-10T13:06:58.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:58.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:58.404735+0000) 2022-09-10T13:06:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:06:58 smithi104 ceph-mon[97044]: pgmap v16565: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:59.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:58.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:58.800703+0000) 2022-09-10T13:06:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:06:58 smithi110 ceph-mon[97633]: pgmap v16565: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:06:59.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:06:59.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:59.404912+0000) 2022-09-10T13:07:00.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:06:59.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:06:59.800893+0000) 2022-09-10T13:07:00.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:00.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:00.405132+0000) 2022-09-10T13:07:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:00 smithi104 ceph-mon[97044]: pgmap v16566: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:01.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:00.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:00.801058+0000) 2022-09-10T13:07:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:00 smithi110 ceph-mon[97633]: pgmap v16566: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:01.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:01.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:01.405311+0000) 2022-09-10T13:07:01.801 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:07:01.528+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:07:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:01.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:01.801230+0000) 2022-09-10T13:07:02.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:02.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:02.405572+0000) 2022-09-10T13:07:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:02 smithi104 ceph-mon[97044]: pgmap v16567: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:03.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:02.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:02.801428+0000) 2022-09-10T13:07:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:02 smithi110 ceph-mon[97633]: pgmap v16567: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:03.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:03.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:03.405747+0000) 2022-09-10T13:07:04.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:03.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:03.801654+0000) 2022-09-10T13:07:04.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:04.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:04.405946+0000) 2022-09-10T13:07:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:04 smithi104 ceph-mon[97044]: pgmap v16568: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:05.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:04.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:04.801842+0000) 2022-09-10T13:07:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:04 smithi110 ceph-mon[97633]: pgmap v16568: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:05.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:05.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:05.406173+0000) 2022-09-10T13:07:06.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:05.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:05.802053+0000) 2022-09-10T13:07:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:06 smithi110 ceph-mon[97633]: pgmap v16569: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:07:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:06 smithi104 ceph-mon[97044]: pgmap v16569: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:07:06.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:06.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:06.406394+0000) 2022-09-10T13:07:07.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:06.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:06.802215+0000) 2022-09-10T13:07:07.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:07.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:07.406648+0000) 2022-09-10T13:07:08.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:07.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:07.802379+0000) 2022-09-10T13:07:08.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:08.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:08.406858+0000) 2022-09-10T13:07:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:08 smithi104 ceph-mon[97044]: pgmap v16570: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:07:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:08 smithi104 ceph-mon[97044]: osdmap e1116: 12 total, 12 up, 12 in 2022-09-10T13:07:09.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:08.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:08.802540+0000) 2022-09-10T13:07:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:08 smithi110 ceph-mon[97633]: pgmap v16570: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:07:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:08 smithi110 ceph-mon[97633]: osdmap e1116: 12 total, 12 up, 12 in 2022-09-10T13:07:09.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:09.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:09.407005+0000) 2022-09-10T13:07:10.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:09.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:09.802674+0000) 2022-09-10T13:07:10.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:10.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:10.407213+0000) 2022-09-10T13:07:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:10 smithi104 ceph-mon[97044]: pgmap v16572: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:07:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:10 smithi104 ceph-mon[97044]: osdmap e1117: 12 total, 12 up, 12 in 2022-09-10T13:07:11.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:10.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:10.802848+0000) 2022-09-10T13:07:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:10 smithi110 ceph-mon[97633]: pgmap v16572: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:07:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:10 smithi110 ceph-mon[97633]: osdmap e1117: 12 total, 12 up, 12 in 2022-09-10T13:07:11.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:11.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:11.407389+0000) 2022-09-10T13:07:12.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:11.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:11.803050+0000) 2022-09-10T13:07:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:12.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:12.407589+0000) 2022-09-10T13:07:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:12 smithi104 ceph-mon[97044]: pgmap v16574: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:07:13.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:12.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:12.803250+0000) 2022-09-10T13:07:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:12 smithi110 ceph-mon[97633]: pgmap v16574: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:07:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:13.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:13.407772+0000) 2022-09-10T13:07:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:13.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:13.803470+0000) 2022-09-10T13:07:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:14.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:14.408026+0000) 2022-09-10T13:07:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:14 smithi104 ceph-mon[97044]: pgmap v16575: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:07:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:14 smithi110 ceph-mon[97633]: pgmap v16575: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:07:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:14.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:14.803681+0000) 2022-09-10T13:07:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:15.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:15.408176+0000) 2022-09-10T13:07:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:15.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:15.803859+0000) 2022-09-10T13:07:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:16 smithi110 ceph-mon[97633]: pgmap v16576: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:07:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:16 smithi104 ceph-mon[97044]: pgmap v16576: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:07:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:16.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:16.408340+0000) 2022-09-10T13:07:16.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:07:16.532+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:07:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:16.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:16.804006+0000) 2022-09-10T13:07:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:17.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:17.408488+0000) 2022-09-10T13:07:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:17.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:17.804152+0000) 2022-09-10T13:07:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:18.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:18.408670+0000) 2022-09-10T13:07:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:18 smithi104 ceph-mon[97044]: pgmap v16577: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T13:07:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:18 smithi110 ceph-mon[97633]: pgmap v16577: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T13:07:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:18.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:18.804278+0000) 2022-09-10T13:07:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:19.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:19.408853+0000) 2022-09-10T13:07:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:19.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:19.804411+0000) 2022-09-10T13:07:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:20.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:20.409121+0000) 2022-09-10T13:07:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:20 smithi104 ceph-mon[97044]: pgmap v16578: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:07:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:20 smithi110 ceph-mon[97633]: pgmap v16578: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:07:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:20.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:20.804536+0000) 2022-09-10T13:07:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:21.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:21.409279+0000) 2022-09-10T13:07:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:21.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:21.804658+0000) 2022-09-10T13:07:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:22.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:22.409432+0000) 2022-09-10T13:07:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:22 smithi110 ceph-mon[97633]: pgmap v16579: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:07:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:22 smithi104 ceph-mon[97044]: pgmap v16579: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:07:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:22.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:22.804811+0000) 2022-09-10T13:07:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:23.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:23.409629+0000) 2022-09-10T13:07:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:23.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:23.804936+0000) 2022-09-10T13:07:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:24.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:24.409880+0000) 2022-09-10T13:07:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:24 smithi110 ceph-mon[97633]: pgmap v16580: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:07:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:24 smithi104 ceph-mon[97044]: pgmap v16580: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:07:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:24.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:24.805101+0000) 2022-09-10T13:07:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:25.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:25.410147+0000) 2022-09-10T13:07:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:25.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:25.805239+0000) 2022-09-10T13:07:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:26 smithi110 ceph-mon[97633]: pgmap v16581: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:07:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:26 smithi104 ceph-mon[97044]: pgmap v16581: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:07:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:26.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:26.410372+0000) 2022-09-10T13:07:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:26.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:26.805376+0000) 2022-09-10T13:07:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:27.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:27.410610+0000) 2022-09-10T13:07:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:27.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:27.805567+0000) 2022-09-10T13:07:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:28.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:28.410826+0000) 2022-09-10T13:07:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:28 smithi104 ceph-mon[97044]: pgmap v16582: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:28 smithi110 ceph-mon[97633]: pgmap v16582: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:28.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:28.805703+0000) 2022-09-10T13:07:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:29.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:29.411056+0000) 2022-09-10T13:07:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:29.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:29.805840+0000) 2022-09-10T13:07:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:30.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:30.411322+0000) 2022-09-10T13:07:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:30 smithi110 ceph-mon[97633]: pgmap v16583: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:30 smithi104 ceph-mon[97044]: pgmap v16583: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:30.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:30.805981+0000) 2022-09-10T13:07:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:31.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:31.411586+0000) 2022-09-10T13:07:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:07:31.534+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:07:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:31.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:31.806188+0000) 2022-09-10T13:07:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:32.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:32.411816+0000) 2022-09-10T13:07:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:32 smithi110 ceph-mon[97633]: pgmap v16584: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:32 smithi104 ceph-mon[97044]: pgmap v16584: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:32.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:32.806331+0000) 2022-09-10T13:07:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:33.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:33.412015+0000) 2022-09-10T13:07:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:33.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:33.806455+0000) 2022-09-10T13:07:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:34.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:34.412271+0000) 2022-09-10T13:07:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:34 smithi110 ceph-mon[97633]: pgmap v16585: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:34 smithi104 ceph-mon[97044]: pgmap v16585: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:34.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:34.806583+0000) 2022-09-10T13:07:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:35.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:35.412502+0000) 2022-09-10T13:07:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:35.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:35.806695+0000) 2022-09-10T13:07:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:36 smithi110 ceph-mon[97633]: pgmap v16586: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:36 smithi104 ceph-mon[97044]: pgmap v16586: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:36.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:36.412763+0000) 2022-09-10T13:07:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:36.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:36.806866+0000) 2022-09-10T13:07:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:37.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:37.412927+0000) 2022-09-10T13:07:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:37.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:37.807012+0000) 2022-09-10T13:07:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:38.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:38.413194+0000) 2022-09-10T13:07:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:38 smithi110 ceph-mon[97633]: pgmap v16587: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:38 smithi104 ceph-mon[97044]: pgmap v16587: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:38.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:38.807141+0000) 2022-09-10T13:07:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:39.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:39.413419+0000) 2022-09-10T13:07:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:39.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:39.807267+0000) 2022-09-10T13:07:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:40.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:40.413593+0000) 2022-09-10T13:07:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:40 smithi110 ceph-mon[97633]: pgmap v16588: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:40 smithi104 ceph-mon[97044]: pgmap v16588: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:40.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:40.807427+0000) 2022-09-10T13:07:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:41.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:41.413829+0000) 2022-09-10T13:07:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:41.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:41.807556+0000) 2022-09-10T13:07:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:42.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:42.414016+0000) 2022-09-10T13:07:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:42 smithi110 ceph-mon[97633]: pgmap v16589: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:42 smithi104 ceph-mon[97044]: pgmap v16589: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:42.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:42.807676+0000) 2022-09-10T13:07:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:43.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:43.414268+0000) 2022-09-10T13:07:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:43.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:43.807850+0000) 2022-09-10T13:07:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:44.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:44.414494+0000) 2022-09-10T13:07:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:44 smithi110 ceph-mon[97633]: pgmap v16590: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:44 smithi104 ceph-mon[97044]: pgmap v16590: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:44.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:44.807990+0000) 2022-09-10T13:07:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:45.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:45.414642+0000) 2022-09-10T13:07:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:45.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:45.808131+0000) 2022-09-10T13:07:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:46 smithi110 ceph-mon[97633]: pgmap v16591: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:07:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:46 smithi104 ceph-mon[97044]: pgmap v16591: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:07:46.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:07:46.538+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:07:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:46.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:46.414852+0000) 2022-09-10T13:07:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:46.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:46.808256+0000) 2022-09-10T13:07:47.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:47.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:47.415083+0000) 2022-09-10T13:07:48.084 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:47.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:47.808406+0000) 2022-09-10T13:07:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:48.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:48.415277+0000) 2022-09-10T13:07:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:48 smithi110 ceph-mon[97633]: pgmap v16592: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:48 smithi104 ceph-mon[97044]: pgmap v16592: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:48.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:48.808541+0000) 2022-09-10T13:07:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:49.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:49.415431+0000) 2022-09-10T13:07:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:49.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:49.808691+0000) 2022-09-10T13:07:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:50.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:50.415623+0000) 2022-09-10T13:07:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:50 smithi110 ceph-mon[97633]: pgmap v16593: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:50 smithi104 ceph-mon[97044]: pgmap v16593: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:50.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:50.808833+0000) 2022-09-10T13:07:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:51.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:51.415864+0000) 2022-09-10T13:07:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:51.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:51.808954+0000) 2022-09-10T13:07:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:52.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:52.416132+0000) 2022-09-10T13:07:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:52 smithi110 ceph-mon[97633]: pgmap v16594: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:52 smithi104 ceph-mon[97044]: pgmap v16594: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:52.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:52.809111+0000) 2022-09-10T13:07:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:53.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:53.416297+0000) 2022-09-10T13:07:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:53.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:53.809278+0000) 2022-09-10T13:07:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:54.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:54.416502+0000) 2022-09-10T13:07:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:54 smithi110 ceph-mon[97633]: pgmap v16595: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:54 smithi104 ceph-mon[97044]: pgmap v16595: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:54.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:54.809433+0000) 2022-09-10T13:07:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:55.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:55.416730+0000) 2022-09-10T13:07:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:55.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:55.809559+0000) 2022-09-10T13:07:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:56 smithi110 ceph-mon[97633]: pgmap v16596: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:56 smithi104 ceph-mon[97044]: pgmap v16596: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:07:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:56.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:56.416910+0000) 2022-09-10T13:07:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:56.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:56.809702+0000) 2022-09-10T13:07:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:57.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:57.417171+0000) 2022-09-10T13:07:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:57.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:57.809882+0000) 2022-09-10T13:07:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:58.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:58.417347+0000) 2022-09-10T13:07:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:07:58 smithi110 ceph-mon[97633]: pgmap v16597: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:07:58 smithi104 ceph-mon[97044]: pgmap v16597: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:07:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:58.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:58.810036+0000) 2022-09-10T13:07:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:07:59.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:59.417544+0000) 2022-09-10T13:08:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:07:59.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:07:59.810186+0000) 2022-09-10T13:08:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:00.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:00.417718+0000) 2022-09-10T13:08:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:00 smithi110 ceph-mon[97633]: pgmap v16598: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:00 smithi104 ceph-mon[97044]: pgmap v16598: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:00.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:00.810368+0000) 2022-09-10T13:08:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:01.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:01.417920+0000) 2022-09-10T13:08:01.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:08:01.554+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:08:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:01.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:01.810591+0000) 2022-09-10T13:08:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:02.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:02.418197+0000) 2022-09-10T13:08:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:02 smithi110 ceph-mon[97633]: pgmap v16599: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:02 smithi104 ceph-mon[97044]: pgmap v16599: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:02.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:02.810810+0000) 2022-09-10T13:08:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:03.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:03.418453+0000) 2022-09-10T13:08:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:03.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:03.810969+0000) 2022-09-10T13:08:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:04.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:04.418707+0000) 2022-09-10T13:08:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:04 smithi110 ceph-mon[97633]: pgmap v16600: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:04 smithi104 ceph-mon[97044]: pgmap v16600: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:04.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:04.811207+0000) 2022-09-10T13:08:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:05.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:05.418879+0000) 2022-09-10T13:08:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:05.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:05.811405+0000) 2022-09-10T13:08:06.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:06 smithi110 ceph-mon[97633]: pgmap v16601: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:08:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:06 smithi104 ceph-mon[97044]: pgmap v16601: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:08:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:06.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:06.419190+0000) 2022-09-10T13:08:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:06.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:06.811597+0000) 2022-09-10T13:08:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:07.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:07.419379+0000) 2022-09-10T13:08:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:07.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:07.811779+0000) 2022-09-10T13:08:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:08.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:08.419599+0000) 2022-09-10T13:08:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:08 smithi104 ceph-mon[97044]: pgmap v16602: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:08:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:08 smithi104 ceph-mon[97044]: osdmap e1118: 12 total, 12 up, 12 in 2022-09-10T13:08:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:08 smithi110 ceph-mon[97633]: pgmap v16602: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:08:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:08 smithi110 ceph-mon[97633]: osdmap e1118: 12 total, 12 up, 12 in 2022-09-10T13:08:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:08.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:08.811956+0000) 2022-09-10T13:08:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:09.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:09.419776+0000) 2022-09-10T13:08:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:09.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:09.812152+0000) 2022-09-10T13:08:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:10.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:10.419962+0000) 2022-09-10T13:08:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:10 smithi104 ceph-mon[97044]: pgmap v16604: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:08:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:10 smithi104 ceph-mon[97044]: osdmap e1119: 12 total, 12 up, 12 in 2022-09-10T13:08:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:10 smithi110 ceph-mon[97633]: pgmap v16604: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:08:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:10 smithi110 ceph-mon[97633]: osdmap e1119: 12 total, 12 up, 12 in 2022-09-10T13:08:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:10.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:10.812294+0000) 2022-09-10T13:08:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:11.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:11.420201+0000) 2022-09-10T13:08:12.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:11.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:11.812524+0000) 2022-09-10T13:08:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:12.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:12.420430+0000) 2022-09-10T13:08:13.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:12 smithi104 ceph-mon[97044]: pgmap v16606: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:08:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:12 smithi110 ceph-mon[97633]: pgmap v16606: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:08:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:12.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:12.812651+0000) 2022-09-10T13:08:13.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:13.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:13.420613+0000) 2022-09-10T13:08:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:13.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:13.812878+0000) 2022-09-10T13:08:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:14.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:14.420775+0000) 2022-09-10T13:08:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:14 smithi104 ceph-mon[97044]: pgmap v16607: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:08:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:14 smithi110 ceph-mon[97633]: pgmap v16607: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:08:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:14.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:14.813167+0000) 2022-09-10T13:08:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:15.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:15.420992+0000) 2022-09-10T13:08:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:15.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:15.813376+0000) 2022-09-10T13:08:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:16 smithi110 ceph-mon[97633]: pgmap v16608: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:08:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:16 smithi104 ceph-mon[97044]: pgmap v16608: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:08:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:16.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:16.421170+0000) 2022-09-10T13:08:16.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:08:16.565+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:08:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:16.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:16.813577+0000) 2022-09-10T13:08:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:17.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:17.421417+0000) 2022-09-10T13:08:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:17.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:17.813826+0000) 2022-09-10T13:08:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:18.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:18.421689+0000) 2022-09-10T13:08:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:18 smithi110 ceph-mon[97633]: pgmap v16609: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T13:08:19.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:18 smithi104 ceph-mon[97044]: pgmap v16609: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T13:08:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:18.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:18.814064+0000) 2022-09-10T13:08:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:19.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:19.421922+0000) 2022-09-10T13:08:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:19.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:19.814271+0000) 2022-09-10T13:08:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:20.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:20.422173+0000) 2022-09-10T13:08:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:20 smithi110 ceph-mon[97633]: pgmap v16610: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:08:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:20 smithi104 ceph-mon[97044]: pgmap v16610: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:08:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:20.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:20.814492+0000) 2022-09-10T13:08:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:21.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:21.422369+0000) 2022-09-10T13:08:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:21.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:21.814711+0000) 2022-09-10T13:08:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:22.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:22.422621+0000) 2022-09-10T13:08:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:22 smithi110 ceph-mon[97633]: pgmap v16611: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:08:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:22 smithi104 ceph-mon[97044]: pgmap v16611: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:08:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:22.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:22.815004+0000) 2022-09-10T13:08:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:23.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:23.422870+0000) 2022-09-10T13:08:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:23.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:23.815209+0000) 2022-09-10T13:08:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:24.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:24.423199+0000) 2022-09-10T13:08:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:24 smithi110 ceph-mon[97633]: pgmap v16612: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:08:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:24 smithi104 ceph-mon[97044]: pgmap v16612: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:08:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:24.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:24.815407+0000) 2022-09-10T13:08:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:25.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:25.423418+0000) 2022-09-10T13:08:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:25.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:25.815573+0000) 2022-09-10T13:08:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:26 smithi110 ceph-mon[97633]: pgmap v16613: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:08:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:26 smithi104 ceph-mon[97044]: pgmap v16613: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:08:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:26.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:26.423673+0000) 2022-09-10T13:08:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:26.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:26.815702+0000) 2022-09-10T13:08:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:27.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:27.423897+0000) 2022-09-10T13:08:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:27.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:27.815856+0000) 2022-09-10T13:08:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:28.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:28.424180+0000) 2022-09-10T13:08:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:28 smithi110 ceph-mon[97633]: pgmap v16614: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:28 smithi104 ceph-mon[97044]: pgmap v16614: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:28.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:28.816089+0000) 2022-09-10T13:08:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:29.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:29.424342+0000) 2022-09-10T13:08:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:29.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:29.816396+0000) 2022-09-10T13:08:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:30.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:30.424607+0000) 2022-09-10T13:08:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:30 smithi110 ceph-mon[97633]: pgmap v16615: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:30 smithi104 ceph-mon[97044]: pgmap v16615: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:30.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:30.816665+0000) 2022-09-10T13:08:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:31.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:31.424829+0000) 2022-09-10T13:08:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:08:31.581+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:08:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:31.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:31.816909+0000) 2022-09-10T13:08:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:32.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:32.425069+0000) 2022-09-10T13:08:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:32 smithi110 ceph-mon[97633]: pgmap v16616: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:32 smithi104 ceph-mon[97044]: pgmap v16616: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:32.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:32.817194+0000) 2022-09-10T13:08:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:33.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:33.425276+0000) 2022-09-10T13:08:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:33.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:33.817367+0000) 2022-09-10T13:08:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:34.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:34.425470+0000) 2022-09-10T13:08:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:34 smithi110 ceph-mon[97633]: pgmap v16617: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:34 smithi104 ceph-mon[97044]: pgmap v16617: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:34.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:34.817507+0000) 2022-09-10T13:08:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:35.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:35.425724+0000) 2022-09-10T13:08:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:35.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:35.817685+0000) 2022-09-10T13:08:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:36 smithi110 ceph-mon[97633]: pgmap v16618: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:36 smithi104 ceph-mon[97044]: pgmap v16618: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:36.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:36.425985+0000) 2022-09-10T13:08:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:36.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:36.817869+0000) 2022-09-10T13:08:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:37.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:37.426244+0000) 2022-09-10T13:08:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:37.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:37.818107+0000) 2022-09-10T13:08:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:38.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:38.426500+0000) 2022-09-10T13:08:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:38 smithi110 ceph-mon[97633]: pgmap v16619: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:38 smithi104 ceph-mon[97044]: pgmap v16619: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:38.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:38.818349+0000) 2022-09-10T13:08:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:39.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:39.426702+0000) 2022-09-10T13:08:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:39.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:39.818594+0000) 2022-09-10T13:08:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:40.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:40.426866+0000) 2022-09-10T13:08:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:40 smithi110 ceph-mon[97633]: pgmap v16620: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:41.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:40 smithi104 ceph-mon[97044]: pgmap v16620: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:40.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:40.818771+0000) 2022-09-10T13:08:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:41.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:41.426988+0000) 2022-09-10T13:08:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:41.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:41.818951+0000) 2022-09-10T13:08:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:42.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:42.427191+0000) 2022-09-10T13:08:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:42 smithi110 ceph-mon[97633]: pgmap v16621: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:42 smithi104 ceph-mon[97044]: pgmap v16621: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:42.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:42.819151+0000) 2022-09-10T13:08:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:43.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:43.427375+0000) 2022-09-10T13:08:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:43.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:43.819321+0000) 2022-09-10T13:08:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:44.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:44.427561+0000) 2022-09-10T13:08:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:44 smithi110 ceph-mon[97633]: pgmap v16622: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:44 smithi104 ceph-mon[97044]: pgmap v16622: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:44.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:44.819496+0000) 2022-09-10T13:08:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:45.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:45.427724+0000) 2022-09-10T13:08:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:45.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:45.819686+0000) 2022-09-10T13:08:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:46 smithi110 ceph-mon[97633]: pgmap v16623: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:08:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:08:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:46 smithi104 ceph-mon[97044]: pgmap v16623: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:08:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:08:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:08:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:08:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:46.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:46.427934+0000) 2022-09-10T13:08:46.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:08:46.596+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:08:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:46.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:46.819865+0000) 2022-09-10T13:08:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:47.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:47.428210+0000) 2022-09-10T13:08:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:47.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:47.820053+0000) 2022-09-10T13:08:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:48.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:48.428410+0000) 2022-09-10T13:08:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:48 smithi110 ceph-mon[97633]: pgmap v16624: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:48 smithi104 ceph-mon[97044]: pgmap v16624: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:48.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:48.820301+0000) 2022-09-10T13:08:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:49.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:49.428656+0000) 2022-09-10T13:08:50.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:49.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:49.820477+0000) 2022-09-10T13:08:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:50.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:50.428911+0000) 2022-09-10T13:08:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:50 smithi110 ceph-mon[97633]: pgmap v16625: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:50 smithi104 ceph-mon[97044]: pgmap v16625: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:50.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:50.820634+0000) 2022-09-10T13:08:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:51.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:51.429067+0000) 2022-09-10T13:08:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:51.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:51.820796+0000) 2022-09-10T13:08:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:52.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:52.429329+0000) 2022-09-10T13:08:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:52 smithi110 ceph-mon[97633]: pgmap v16626: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:52 smithi104 ceph-mon[97044]: pgmap v16626: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:52.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:52.820946+0000) 2022-09-10T13:08:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:53.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:53.429495+0000) 2022-09-10T13:08:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:53.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:53.821137+0000) 2022-09-10T13:08:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:54.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:54.429652+0000) 2022-09-10T13:08:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:54 smithi110 ceph-mon[97633]: pgmap v16627: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:54 smithi104 ceph-mon[97044]: pgmap v16627: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:54.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:54.821385+0000) 2022-09-10T13:08:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:55.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:55.429784+0000) 2022-09-10T13:08:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:55.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:55.821600+0000) 2022-09-10T13:08:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:56 smithi110 ceph-mon[97633]: pgmap v16628: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:56 smithi104 ceph-mon[97044]: pgmap v16628: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:08:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:56.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:56.430016+0000) 2022-09-10T13:08:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:56.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:56.821817+0000) 2022-09-10T13:08:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:57.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:57.430217+0000) 2022-09-10T13:08:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:57.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:57.822032+0000) 2022-09-10T13:08:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:58.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:58.430466+0000) 2022-09-10T13:08:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:08:58 smithi110 ceph-mon[97633]: pgmap v16629: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:08:58 smithi104 ceph-mon[97044]: pgmap v16629: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:08:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:58.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:58.822247+0000) 2022-09-10T13:08:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:08:59.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:59.430722+0000) 2022-09-10T13:09:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:08:59.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:08:59.822527+0000) 2022-09-10T13:09:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:00.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:00.430871+0000) 2022-09-10T13:09:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:00 smithi110 ceph-mon[97633]: pgmap v16630: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:00 smithi104 ceph-mon[97044]: pgmap v16630: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:00.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:00.822686+0000) 2022-09-10T13:09:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:01.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:01.431080+0000) 2022-09-10T13:09:01.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:09:01.601+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:09:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:01.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:01.822925+0000) 2022-09-10T13:09:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:02.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:02.431354+0000) 2022-09-10T13:09:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:02 smithi110 ceph-mon[97633]: pgmap v16631: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:03.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:02 smithi104 ceph-mon[97044]: pgmap v16631: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:02.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:02.823150+0000) 2022-09-10T13:09:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:03.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:03.431630+0000) 2022-09-10T13:09:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:03.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:03.823348+0000) 2022-09-10T13:09:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:04.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:04.431872+0000) 2022-09-10T13:09:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:04 smithi110 ceph-mon[97633]: pgmap v16632: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:04 smithi104 ceph-mon[97044]: pgmap v16632: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:04.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:04.823619+0000) 2022-09-10T13:09:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:05.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:05.432177+0000) 2022-09-10T13:09:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:05.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:05.823809+0000) 2022-09-10T13:09:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:06 smithi110 ceph-mon[97633]: pgmap v16633: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:09:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:06 smithi104 ceph-mon[97044]: pgmap v16633: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:09:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:06.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:06.432312+0000) 2022-09-10T13:09:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:06.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:06.824029+0000) 2022-09-10T13:09:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:07.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:07.432555+0000) 2022-09-10T13:09:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:07.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:07.824247+0000) 2022-09-10T13:09:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:08.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:08.432814+0000) 2022-09-10T13:09:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:08 smithi104 ceph-mon[97044]: pgmap v16634: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:09:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:08 smithi104 ceph-mon[97044]: osdmap e1120: 12 total, 12 up, 12 in 2022-09-10T13:09:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:08 smithi110 ceph-mon[97633]: pgmap v16634: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:09:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:08 smithi110 ceph-mon[97633]: osdmap e1120: 12 total, 12 up, 12 in 2022-09-10T13:09:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:08.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:08.824483+0000) 2022-09-10T13:09:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:09.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:09.433034+0000) 2022-09-10T13:09:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:09.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:09.824694+0000) 2022-09-10T13:09:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:10.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:10.433214+0000) 2022-09-10T13:09:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:10 smithi104 ceph-mon[97044]: pgmap v16636: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:09:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:10 smithi104 ceph-mon[97044]: osdmap e1121: 12 total, 12 up, 12 in 2022-09-10T13:09:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:10 smithi110 ceph-mon[97633]: pgmap v16636: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:09:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:10 smithi110 ceph-mon[97633]: osdmap e1121: 12 total, 12 up, 12 in 2022-09-10T13:09:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:10.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:10.824937+0000) 2022-09-10T13:09:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:11.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:11.433466+0000) 2022-09-10T13:09:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:11.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:11.825120+0000) 2022-09-10T13:09:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:12.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:12.433698+0000) 2022-09-10T13:09:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:12 smithi104 ceph-mon[97044]: pgmap v16638: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:09:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:12 smithi110 ceph-mon[97633]: pgmap v16638: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:09:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:12.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:12.825331+0000) 2022-09-10T13:09:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:13.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:13.433969+0000) 2022-09-10T13:09:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:13.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:13.825551+0000) 2022-09-10T13:09:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:14.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:14.434242+0000) 2022-09-10T13:09:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:14 smithi104 ceph-mon[97044]: pgmap v16639: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:09:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:14 smithi110 ceph-mon[97633]: pgmap v16639: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:09:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:14.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:14.825781+0000) 2022-09-10T13:09:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:15.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:15.434486+0000) 2022-09-10T13:09:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:15.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:15.825962+0000) 2022-09-10T13:09:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:16 smithi110 ceph-mon[97633]: pgmap v16640: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:09:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:16 smithi104 ceph-mon[97044]: pgmap v16640: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:09:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:16.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:16.434783+0000) 2022-09-10T13:09:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:09:16.614+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:09:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:16.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:16.826152+0000) 2022-09-10T13:09:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:17.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:17.435030+0000) 2022-09-10T13:09:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:17.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:17.826312+0000) 2022-09-10T13:09:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:18.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:18.435332+0000) 2022-09-10T13:09:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:18 smithi110 ceph-mon[97633]: pgmap v16641: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T13:09:19.219 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:18.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:18.826543+0000) 2022-09-10T13:09:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:18 smithi104 ceph-mon[97044]: pgmap v16641: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 205 B/s wr, 10 op/s 2022-09-10T13:09:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:19.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:19.435612+0000) 2022-09-10T13:09:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:19.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:19.826827+0000) 2022-09-10T13:09:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:20.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:20.435850+0000) 2022-09-10T13:09:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:20 smithi110 ceph-mon[97633]: pgmap v16642: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:09:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:20 smithi104 ceph-mon[97044]: pgmap v16642: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:09:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:20.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:20.827023+0000) 2022-09-10T13:09:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:21.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:21.436054+0000) 2022-09-10T13:09:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:21.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:21.827216+0000) 2022-09-10T13:09:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:22.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:22.436213+0000) 2022-09-10T13:09:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:22 smithi110 ceph-mon[97633]: pgmap v16643: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:09:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:22 smithi104 ceph-mon[97044]: pgmap v16643: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:09:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:22.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:22.827430+0000) 2022-09-10T13:09:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:23.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:23.436441+0000) 2022-09-10T13:09:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:23.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:23.827618+0000) 2022-09-10T13:09:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:24.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:24.436657+0000) 2022-09-10T13:09:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:24 smithi110 ceph-mon[97633]: pgmap v16644: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:09:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:24 smithi104 ceph-mon[97044]: pgmap v16644: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:09:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:24.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:24.827836+0000) 2022-09-10T13:09:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:25.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:25.436850+0000) 2022-09-10T13:09:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:25.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:25.828067+0000) 2022-09-10T13:09:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:26 smithi110 ceph-mon[97633]: pgmap v16645: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:09:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:26 smithi104 ceph-mon[97044]: pgmap v16645: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:09:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:26.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:26.437062+0000) 2022-09-10T13:09:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:26.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:26.828358+0000) 2022-09-10T13:09:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:27.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:27.437328+0000) 2022-09-10T13:09:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:27.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:27.828580+0000) 2022-09-10T13:09:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:28.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:28.437532+0000) 2022-09-10T13:09:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:28 smithi110 ceph-mon[97633]: pgmap v16646: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:28 smithi104 ceph-mon[97044]: pgmap v16646: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:28.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:28.828839+0000) 2022-09-10T13:09:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:29.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:29.437697+0000) 2022-09-10T13:09:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:29.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:29.829084+0000) 2022-09-10T13:09:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:30.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:30.437876+0000) 2022-09-10T13:09:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:30 smithi110 ceph-mon[97633]: pgmap v16647: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:30 smithi104 ceph-mon[97044]: pgmap v16647: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:30.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:30.829338+0000) 2022-09-10T13:09:31.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:09:31.622+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:09:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:31.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:31.438068+0000) 2022-09-10T13:09:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:31.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:31.829487+0000) 2022-09-10T13:09:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:32.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:32.438236+0000) 2022-09-10T13:09:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:32 smithi110 ceph-mon[97633]: pgmap v16648: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:32 smithi104 ceph-mon[97044]: pgmap v16648: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:32.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:32.829626+0000) 2022-09-10T13:09:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:33.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:33.438391+0000) 2022-09-10T13:09:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:33.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:33.829881+0000) 2022-09-10T13:09:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:34.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:34.438555+0000) 2022-09-10T13:09:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:34 smithi110 ceph-mon[97633]: pgmap v16649: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:34 smithi104 ceph-mon[97044]: pgmap v16649: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:34.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:34.830123+0000) 2022-09-10T13:09:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:35.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:35.438724+0000) 2022-09-10T13:09:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:35.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:35.830293+0000) 2022-09-10T13:09:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:36 smithi110 ceph-mon[97633]: pgmap v16650: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:36 smithi104 ceph-mon[97044]: pgmap v16650: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:36.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:36.438953+0000) 2022-09-10T13:09:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:36.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:36.830495+0000) 2022-09-10T13:09:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:37.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:37.439177+0000) 2022-09-10T13:09:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:37.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:37.830688+0000) 2022-09-10T13:09:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:38.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:38.439411+0000) 2022-09-10T13:09:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:38 smithi110 ceph-mon[97633]: pgmap v16651: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:38 smithi104 ceph-mon[97044]: pgmap v16651: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:38.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:38.830968+0000) 2022-09-10T13:09:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:39.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:39.439536+0000) 2022-09-10T13:09:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:39.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:39.831160+0000) 2022-09-10T13:09:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:40.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:40.439728+0000) 2022-09-10T13:09:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:40 smithi110 ceph-mon[97633]: pgmap v16652: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:40 smithi104 ceph-mon[97044]: pgmap v16652: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:40.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:40.831348+0000) 2022-09-10T13:09:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:41.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:41.439955+0000) 2022-09-10T13:09:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:41.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:41.831507+0000) 2022-09-10T13:09:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:42.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:42.440173+0000) 2022-09-10T13:09:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:42 smithi110 ceph-mon[97633]: pgmap v16653: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:42 smithi104 ceph-mon[97044]: pgmap v16653: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:42.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:42.831747+0000) 2022-09-10T13:09:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:43.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:43.440401+0000) 2022-09-10T13:09:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:43.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:43.831921+0000) 2022-09-10T13:09:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:44.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:44.440589+0000) 2022-09-10T13:09:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:44 smithi110 ceph-mon[97633]: pgmap v16654: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:44 smithi104 ceph-mon[97044]: pgmap v16654: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:44.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:44.832172+0000) 2022-09-10T13:09:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:45.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:45.440808+0000) 2022-09-10T13:09:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:45.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:45.832455+0000) 2022-09-10T13:09:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:46 smithi110 ceph-mon[97633]: pgmap v16655: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:09:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:09:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:09:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:09:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:46 smithi104 ceph-mon[97044]: pgmap v16655: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:09:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:09:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:46.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:46.441001+0000) 2022-09-10T13:09:46.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:09:46.623+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:09:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:46.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:46.832722+0000) 2022-09-10T13:09:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:47.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:47.441213+0000) 2022-09-10T13:09:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:47.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:47.832969+0000) 2022-09-10T13:09:48.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:48.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:48.441353+0000) 2022-09-10T13:09:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:48 smithi110 ceph-mon[97633]: pgmap v16656: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:48 smithi104 ceph-mon[97044]: pgmap v16656: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:48.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:48.833156+0000) 2022-09-10T13:09:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:49.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:49.441513+0000) 2022-09-10T13:09:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:49.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:49.833299+0000) 2022-09-10T13:09:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:50.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:50.441700+0000) 2022-09-10T13:09:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:50 smithi110 ceph-mon[97633]: pgmap v16657: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:50 smithi104 ceph-mon[97044]: pgmap v16657: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:50.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:50.833489+0000) 2022-09-10T13:09:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:51.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:51.441862+0000) 2022-09-10T13:09:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:51.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:51.833664+0000) 2022-09-10T13:09:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:52.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:52.442082+0000) 2022-09-10T13:09:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:52 smithi110 ceph-mon[97633]: pgmap v16658: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:52 smithi104 ceph-mon[97044]: pgmap v16658: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:52.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:52.833880+0000) 2022-09-10T13:09:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:53.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:53.442280+0000) 2022-09-10T13:09:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:53.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:53.834044+0000) 2022-09-10T13:09:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:54.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:54.442459+0000) 2022-09-10T13:09:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:54 smithi110 ceph-mon[97633]: pgmap v16659: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:54 smithi104 ceph-mon[97044]: pgmap v16659: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:54.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:54.834216+0000) 2022-09-10T13:09:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:55.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:55.442722+0000) 2022-09-10T13:09:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:55.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:55.834416+0000) 2022-09-10T13:09:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:56 smithi110 ceph-mon[97633]: pgmap v16660: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:56 smithi104 ceph-mon[97044]: pgmap v16660: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:09:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:56.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:56.442971+0000) 2022-09-10T13:09:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:56.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:56.834572+0000) 2022-09-10T13:09:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:57.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:57.443210+0000) 2022-09-10T13:09:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:57.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:57.834705+0000) 2022-09-10T13:09:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:58.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:58.443462+0000) 2022-09-10T13:09:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:09:58 smithi110 ceph-mon[97633]: pgmap v16661: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:09:58 smithi104 ceph-mon[97044]: pgmap v16661: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:09:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:58.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:58.834904+0000) 2022-09-10T13:09:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:09:59.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:59.443694+0000) 2022-09-10T13:10:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:09:59.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:09:59.835197+0000) 2022-09-10T13:10:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:00.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:00.443910+0000) 2022-09-10T13:10:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:00 smithi110 ceph-mon[97633]: pgmap v16662: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:10:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:00 smithi104 ceph-mon[97044]: pgmap v16662: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:10:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:00.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:00.835445+0000) 2022-09-10T13:10:01.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:10:01.627+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:10:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:01.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:01.444170+0000) 2022-09-10T13:10:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:01.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:01.835633+0000) 2022-09-10T13:10:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:02.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:02.444350+0000) 2022-09-10T13:10:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:02 smithi110 ceph-mon[97633]: pgmap v16663: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:02 smithi104 ceph-mon[97044]: pgmap v16663: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:02.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:02.835807+0000) 2022-09-10T13:10:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:03.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:03.444600+0000) 2022-09-10T13:10:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:03.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:03.835968+0000) 2022-09-10T13:10:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:04.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:04.444861+0000) 2022-09-10T13:10:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:04 smithi110 ceph-mon[97633]: pgmap v16664: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:04 smithi104 ceph-mon[97044]: pgmap v16664: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:04.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:04.836198+0000) 2022-09-10T13:10:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:05.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:05.445066+0000) 2022-09-10T13:10:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:05.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:05.836421+0000) 2022-09-10T13:10:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:06 smithi110 ceph-mon[97633]: pgmap v16665: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:10:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:06 smithi104 ceph-mon[97044]: pgmap v16665: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:10:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:06.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:06.445333+0000) 2022-09-10T13:10:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:06.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:06.836607+0000) 2022-09-10T13:10:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:07.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:07.445523+0000) 2022-09-10T13:10:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:07.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:07.836862+0000) 2022-09-10T13:10:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:08.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:08.445738+0000) 2022-09-10T13:10:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:08 smithi104 ceph-mon[97044]: osdmap e1122: 12 total, 12 up, 12 in 2022-09-10T13:10:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:08 smithi104 ceph-mon[97044]: pgmap v16667: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:10:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:08 smithi110 ceph-mon[97633]: osdmap e1122: 12 total, 12 up, 12 in 2022-09-10T13:10:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:08 smithi110 ceph-mon[97633]: pgmap v16667: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:10:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:08.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:08.837053+0000) 2022-09-10T13:10:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:09.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:09.445979+0000) 2022-09-10T13:10:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:09.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:09.837275+0000) 2022-09-10T13:10:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:10.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:10.446137+0000) 2022-09-10T13:10:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:10 smithi110 ceph-mon[97633]: pgmap v16668: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:10:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:10 smithi110 ceph-mon[97633]: osdmap e1123: 12 total, 12 up, 12 in 2022-09-10T13:10:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:10 smithi104 ceph-mon[97044]: pgmap v16668: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:10:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:10 smithi104 ceph-mon[97044]: osdmap e1123: 12 total, 12 up, 12 in 2022-09-10T13:10:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:10.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:10.837480+0000) 2022-09-10T13:10:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:11.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:11.446351+0000) 2022-09-10T13:10:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:11.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:11.837726+0000) 2022-09-10T13:10:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:12.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:12.446565+0000) 2022-09-10T13:10:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:12 smithi110 ceph-mon[97633]: pgmap v16670: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:10:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:12 smithi104 ceph-mon[97044]: pgmap v16670: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:10:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:12.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:12.837995+0000) 2022-09-10T13:10:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:13.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:13.446761+0000) 2022-09-10T13:10:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:13.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:13.838187+0000) 2022-09-10T13:10:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:14.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:14.447005+0000) 2022-09-10T13:10:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:14 smithi110 ceph-mon[97633]: pgmap v16671: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:10:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:14 smithi104 ceph-mon[97044]: pgmap v16671: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:10:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:14.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:14.838383+0000) 2022-09-10T13:10:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:15.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:15.447196+0000) 2022-09-10T13:10:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:15.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:15.838614+0000) 2022-09-10T13:10:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:16 smithi110 ceph-mon[97633]: pgmap v16672: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:10:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:16 smithi104 ceph-mon[97044]: pgmap v16672: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:10:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:16.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:16.447423+0000) 2022-09-10T13:10:16.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:10:16.637+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:10:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:16.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:16.838895+0000) 2022-09-10T13:10:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:17.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:17.447625+0000) 2022-09-10T13:10:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:17.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:17.839180+0000) 2022-09-10T13:10:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:18.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:18.447806+0000) 2022-09-10T13:10:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:18 smithi110 ceph-mon[97633]: pgmap v16673: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:10:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:18 smithi104 ceph-mon[97044]: pgmap v16673: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:10:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:18.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:18.839385+0000) 2022-09-10T13:10:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:19.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:19.448049+0000) 2022-09-10T13:10:20.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:19.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:19.839620+0000) 2022-09-10T13:10:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:20.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:20.448275+0000) 2022-09-10T13:10:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:20 smithi110 ceph-mon[97633]: pgmap v16674: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:10:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:20 smithi104 ceph-mon[97044]: pgmap v16674: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:10:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:20.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:20.839814+0000) 2022-09-10T13:10:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:21.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:21.448517+0000) 2022-09-10T13:10:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:21.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:21.839981+0000) 2022-09-10T13:10:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:22.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:22.448725+0000) 2022-09-10T13:10:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:22 smithi110 ceph-mon[97633]: pgmap v16675: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:10:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:22 smithi104 ceph-mon[97044]: pgmap v16675: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:10:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:22.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:22.840233+0000) 2022-09-10T13:10:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:23.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:23.448972+0000) 2022-09-10T13:10:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:23.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:23.840383+0000) 2022-09-10T13:10:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:24.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:24.449221+0000) 2022-09-10T13:10:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:24 smithi110 ceph-mon[97633]: pgmap v16676: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:10:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:24 smithi104 ceph-mon[97044]: pgmap v16676: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:10:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:24.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:24.840646+0000) 2022-09-10T13:10:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:25.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:25.449456+0000) 2022-09-10T13:10:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:25.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:25.840894+0000) 2022-09-10T13:10:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:26 smithi110 ceph-mon[97633]: pgmap v16677: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:10:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:26 smithi104 ceph-mon[97044]: pgmap v16677: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:10:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:26.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:26.449710+0000) 2022-09-10T13:10:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:26.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:26.841145+0000) 2022-09-10T13:10:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:27.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:27.449943+0000) 2022-09-10T13:10:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:27.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:27.841324+0000) 2022-09-10T13:10:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:28.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:28.450202+0000) 2022-09-10T13:10:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:28 smithi110 ceph-mon[97633]: pgmap v16678: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:28 smithi104 ceph-mon[97044]: pgmap v16678: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:28.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:28.841593+0000) 2022-09-10T13:10:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:29.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:29.450424+0000) 2022-09-10T13:10:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:29.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:29.841827+0000) 2022-09-10T13:10:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:30.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:30.450610+0000) 2022-09-10T13:10:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:30 smithi110 ceph-mon[97633]: pgmap v16679: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:30 smithi104 ceph-mon[97044]: pgmap v16679: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:30.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:30.842122+0000) 2022-09-10T13:10:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:31.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:31.450802+0000) 2022-09-10T13:10:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:10:31.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:10:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:31.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:31.842322+0000) 2022-09-10T13:10:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:32.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:32.451055+0000) 2022-09-10T13:10:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:32 smithi110 ceph-mon[97633]: pgmap v16680: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:32 smithi104 ceph-mon[97044]: pgmap v16680: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:32.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:32.842602+0000) 2022-09-10T13:10:33.813 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:33.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:33.451192+0000) 2022-09-10T13:10:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:33.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:33.842845+0000) 2022-09-10T13:10:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:34.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:34.451370+0000) 2022-09-10T13:10:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:34 smithi110 ceph-mon[97633]: pgmap v16681: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:34 smithi104 ceph-mon[97044]: pgmap v16681: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:34.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:34.842980+0000) 2022-09-10T13:10:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:35.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:35.451573+0000) 2022-09-10T13:10:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:35.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:35.843222+0000) 2022-09-10T13:10:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:36 smithi110 ceph-mon[97633]: pgmap v16682: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:36 smithi104 ceph-mon[97044]: pgmap v16682: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:36.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:36.451765+0000) 2022-09-10T13:10:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:36.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:36.843454+0000) 2022-09-10T13:10:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:37.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:37.451924+0000) 2022-09-10T13:10:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:37.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:37.843724+0000) 2022-09-10T13:10:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:38.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:38.452130+0000) 2022-09-10T13:10:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:38 smithi110 ceph-mon[97633]: pgmap v16683: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:38 smithi104 ceph-mon[97044]: pgmap v16683: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:38.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:38.843987+0000) 2022-09-10T13:10:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:39.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:39.452389+0000) 2022-09-10T13:10:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:39.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:39.844282+0000) 2022-09-10T13:10:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:40.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:40.452565+0000) 2022-09-10T13:10:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:40 smithi110 ceph-mon[97633]: pgmap v16684: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:40 smithi104 ceph-mon[97044]: pgmap v16684: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:40.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:40.844525+0000) 2022-09-10T13:10:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:41.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:41.452754+0000) 2022-09-10T13:10:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:41.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:41.844744+0000) 2022-09-10T13:10:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:42.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:42.452946+0000) 2022-09-10T13:10:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:42 smithi110 ceph-mon[97633]: pgmap v16685: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:42 smithi104 ceph-mon[97044]: pgmap v16685: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:42.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:42.844902+0000) 2022-09-10T13:10:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:43.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:43.453068+0000) 2022-09-10T13:10:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:43.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:43.845053+0000) 2022-09-10T13:10:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:44.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:44.453288+0000) 2022-09-10T13:10:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:44 smithi110 ceph-mon[97633]: pgmap v16686: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:44 smithi104 ceph-mon[97044]: pgmap v16686: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:44.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:44.845260+0000) 2022-09-10T13:10:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:45.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:45.453550+0000) 2022-09-10T13:10:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:45.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:45.845419+0000) 2022-09-10T13:10:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:46 smithi110 ceph-mon[97633]: pgmap v16687: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:10:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:10:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:10:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:10:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:46 smithi104 ceph-mon[97044]: pgmap v16687: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:10:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:10:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:46.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:46.453772+0000) 2022-09-10T13:10:46.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:10:46.645+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:10:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:46.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:46.845610+0000) 2022-09-10T13:10:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:47.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:47.454018+0000) 2022-09-10T13:10:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:47.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:47.845789+0000) 2022-09-10T13:10:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:48.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:48.454244+0000) 2022-09-10T13:10:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:48 smithi110 ceph-mon[97633]: pgmap v16688: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:48 smithi104 ceph-mon[97044]: pgmap v16688: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:48.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:48.846045+0000) 2022-09-10T13:10:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:49.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:49.454436+0000) 2022-09-10T13:10:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:49.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:49.846331+0000) 2022-09-10T13:10:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:50.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:50.454654+0000) 2022-09-10T13:10:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:50 smithi110 ceph-mon[97633]: pgmap v16689: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:51.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:50 smithi104 ceph-mon[97044]: pgmap v16689: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:50.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:50.846550+0000) 2022-09-10T13:10:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:51.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:51.454899+0000) 2022-09-10T13:10:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:51.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:51.846795+0000) 2022-09-10T13:10:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:52.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:52.455173+0000) 2022-09-10T13:10:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:52 smithi110 ceph-mon[97633]: pgmap v16690: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:52 smithi104 ceph-mon[97044]: pgmap v16690: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:52.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:52.846981+0000) 2022-09-10T13:10:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:53.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:53.455314+0000) 2022-09-10T13:10:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:53.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:53.847193+0000) 2022-09-10T13:10:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:54.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:54.455564+0000) 2022-09-10T13:10:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:54 smithi110 ceph-mon[97633]: pgmap v16691: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:54 smithi104 ceph-mon[97044]: pgmap v16691: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:54.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:54.847399+0000) 2022-09-10T13:10:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:55.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:55.455837+0000) 2022-09-10T13:10:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:55.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:55.847578+0000) 2022-09-10T13:10:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:56 smithi110 ceph-mon[97633]: pgmap v16692: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:56 smithi104 ceph-mon[97044]: pgmap v16692: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:10:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:56.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:56.456065+0000) 2022-09-10T13:10:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:56.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:56.847770+0000) 2022-09-10T13:10:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:57.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:57.456337+0000) 2022-09-10T13:10:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:57.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:57.847991+0000) 2022-09-10T13:10:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:58.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:58.456606+0000) 2022-09-10T13:10:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:10:58 smithi110 ceph-mon[97633]: pgmap v16693: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:10:58 smithi104 ceph-mon[97044]: pgmap v16693: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:10:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:58.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:58.848228+0000) 2022-09-10T13:10:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:10:59.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:59.456878+0000) 2022-09-10T13:11:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:10:59.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:10:59.848354+0000) 2022-09-10T13:11:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:00.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:00.457072+0000) 2022-09-10T13:11:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:00 smithi110 ceph-mon[97633]: pgmap v16694: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:00 smithi104 ceph-mon[97044]: pgmap v16694: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:00.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:00.848484+0000) 2022-09-10T13:11:01.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:11:01.660+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:11:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:01.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:01.457318+0000) 2022-09-10T13:11:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:01.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:01.848654+0000) 2022-09-10T13:11:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:02.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:02.457608+0000) 2022-09-10T13:11:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:02 smithi110 ceph-mon[97633]: pgmap v16695: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:02 smithi104 ceph-mon[97044]: pgmap v16695: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:02.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:02.848917+0000) 2022-09-10T13:11:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:03.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:03.457866+0000) 2022-09-10T13:11:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:03.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:03.849160+0000) 2022-09-10T13:11:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:04.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:04.458139+0000) 2022-09-10T13:11:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:04 smithi110 ceph-mon[97633]: pgmap v16696: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:04 smithi104 ceph-mon[97044]: pgmap v16696: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:04.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:04.849322+0000) 2022-09-10T13:11:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:05.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:05.458315+0000) 2022-09-10T13:11:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:05.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:05.849539+0000) 2022-09-10T13:11:06.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:06 smithi110 ceph-mon[97633]: pgmap v16697: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:11:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:06 smithi104 ceph-mon[97044]: pgmap v16697: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:11:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:06.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:06.458591+0000) 2022-09-10T13:11:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:06.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:06.849797+0000) 2022-09-10T13:11:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:07.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:07.458839+0000) 2022-09-10T13:11:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:07.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:07.850086+0000) 2022-09-10T13:11:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:08.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:08.459069+0000) 2022-09-10T13:11:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:08 smithi104 ceph-mon[97044]: osdmap e1124: 12 total, 12 up, 12 in 2022-09-10T13:11:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:08 smithi104 ceph-mon[97044]: pgmap v16699: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:11:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:08 smithi110 ceph-mon[97633]: osdmap e1124: 12 total, 12 up, 12 in 2022-09-10T13:11:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:08 smithi110 ceph-mon[97633]: pgmap v16699: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:11:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:08.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:08.850352+0000) 2022-09-10T13:11:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:09.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:09.459307+0000) 2022-09-10T13:11:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:09.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:09.850577+0000) 2022-09-10T13:11:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:10.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:10.459470+0000) 2022-09-10T13:11:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:10 smithi110 ceph-mon[97633]: pgmap v16700: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:11:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:10 smithi110 ceph-mon[97633]: osdmap e1125: 12 total, 12 up, 12 in 2022-09-10T13:11:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:10 smithi104 ceph-mon[97044]: pgmap v16700: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:11:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:10 smithi104 ceph-mon[97044]: osdmap e1125: 12 total, 12 up, 12 in 2022-09-10T13:11:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:10.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:10.850798+0000) 2022-09-10T13:11:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:11.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:11.459598+0000) 2022-09-10T13:11:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:11.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:11.851000+0000) 2022-09-10T13:11:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:12.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:12.459816+0000) 2022-09-10T13:11:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:12 smithi110 ceph-mon[97633]: pgmap v16702: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:11:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:12 smithi104 ceph-mon[97044]: pgmap v16702: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:11:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:12.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:12.851224+0000) 2022-09-10T13:11:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:13.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:13.460005+0000) 2022-09-10T13:11:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:13.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:13.851492+0000) 2022-09-10T13:11:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:14.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:14.460225+0000) 2022-09-10T13:11:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:14 smithi110 ceph-mon[97633]: pgmap v16703: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:11:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:14 smithi104 ceph-mon[97044]: pgmap v16703: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:11:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:14.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:14.851744+0000) 2022-09-10T13:11:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:15.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:15.460407+0000) 2022-09-10T13:11:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:15.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:15.851989+0000) 2022-09-10T13:11:16.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:16 smithi110 ceph-mon[97633]: pgmap v16704: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:11:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:16 smithi104 ceph-mon[97044]: pgmap v16704: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:11:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:16.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:16.460572+0000) 2022-09-10T13:11:16.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:11:16.668+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:11:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:16.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:16.852209+0000) 2022-09-10T13:11:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:17.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:17.460741+0000) 2022-09-10T13:11:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:17.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:17.852362+0000) 2022-09-10T13:11:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:18.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:18.460964+0000) 2022-09-10T13:11:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:18 smithi110 ceph-mon[97633]: pgmap v16705: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:11:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:18 smithi104 ceph-mon[97044]: pgmap v16705: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:11:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:18.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:18.852531+0000) 2022-09-10T13:11:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:19.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:19.461249+0000) 2022-09-10T13:11:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:19.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:19.852695+0000) 2022-09-10T13:11:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:20.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:20.461505+0000) 2022-09-10T13:11:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:20 smithi110 ceph-mon[97633]: pgmap v16706: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:11:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:20 smithi104 ceph-mon[97044]: pgmap v16706: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:11:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:20.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:20.852927+0000) 2022-09-10T13:11:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:21.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:21.461687+0000) 2022-09-10T13:11:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:21.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:21.853173+0000) 2022-09-10T13:11:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:22.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:22.461899+0000) 2022-09-10T13:11:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:22 smithi110 ceph-mon[97633]: pgmap v16707: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:11:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:22 smithi104 ceph-mon[97044]: pgmap v16707: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:11:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:22.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:22.853382+0000) 2022-09-10T13:11:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:23.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:23.462185+0000) 2022-09-10T13:11:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:23.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:23.853640+0000) 2022-09-10T13:11:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:24.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:24.462436+0000) 2022-09-10T13:11:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:24 smithi110 ceph-mon[97633]: pgmap v16708: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:11:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:24 smithi104 ceph-mon[97044]: pgmap v16708: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:11:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:24.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:24.853875+0000) 2022-09-10T13:11:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:25.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:25.462693+0000) 2022-09-10T13:11:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:25.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:25.854092+0000) 2022-09-10T13:11:26.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:26 smithi110 ceph-mon[97633]: pgmap v16709: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:11:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:26 smithi104 ceph-mon[97044]: pgmap v16709: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:11:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:26.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:26.462905+0000) 2022-09-10T13:11:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:26.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:26.854349+0000) 2022-09-10T13:11:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:27.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:27.463077+0000) 2022-09-10T13:11:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:27.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:27.854505+0000) 2022-09-10T13:11:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:28.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:28.463265+0000) 2022-09-10T13:11:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:28 smithi110 ceph-mon[97633]: pgmap v16710: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:28 smithi104 ceph-mon[97044]: pgmap v16710: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:28.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:28.854673+0000) 2022-09-10T13:11:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:29.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:29.463401+0000) 2022-09-10T13:11:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:29.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:29.854886+0000) 2022-09-10T13:11:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:30.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:30.463650+0000) 2022-09-10T13:11:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:30 smithi110 ceph-mon[97633]: pgmap v16711: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:11:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:30 smithi104 ceph-mon[97044]: pgmap v16711: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:11:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:30.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:30.855164+0000) 2022-09-10T13:11:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:31.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:31.463871+0000) 2022-09-10T13:11:31.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:11:31.669+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:11:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:31.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:31.855368+0000) 2022-09-10T13:11:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:32.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:32.464088+0000) 2022-09-10T13:11:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:32 smithi110 ceph-mon[97633]: pgmap v16712: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T13:11:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:32 smithi104 ceph-mon[97044]: pgmap v16712: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T13:11:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:32.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:32.855628+0000) 2022-09-10T13:11:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:33.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:33.464308+0000) 2022-09-10T13:11:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:33.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:33.855798+0000) 2022-09-10T13:11:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:34.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:34.464511+0000) 2022-09-10T13:11:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:34 smithi110 ceph-mon[97633]: pgmap v16713: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T13:11:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:34 smithi104 ceph-mon[97044]: pgmap v16713: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2022-09-10T13:11:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:34.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:34.856003+0000) 2022-09-10T13:11:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:35.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:35.464741+0000) 2022-09-10T13:11:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:35.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:35.856252+0000) 2022-09-10T13:11:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:36 smithi110 ceph-mon[97633]: pgmap v16714: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:11:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:36 smithi104 ceph-mon[97044]: pgmap v16714: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:11:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:36.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:36.464922+0000) 2022-09-10T13:11:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:36.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:36.856497+0000) 2022-09-10T13:11:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:37.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:37.465161+0000) 2022-09-10T13:11:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:37.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:37.856754+0000) 2022-09-10T13:11:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:38.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:38.465388+0000) 2022-09-10T13:11:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:38 smithi110 ceph-mon[97633]: pgmap v16715: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:11:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:38 smithi104 ceph-mon[97044]: pgmap v16715: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:11:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:38.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:38.856993+0000) 2022-09-10T13:11:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:39.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:39.465641+0000) 2022-09-10T13:11:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:39.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:39.857233+0000) 2022-09-10T13:11:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:40.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:40.465838+0000) 2022-09-10T13:11:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:40 smithi110 ceph-mon[97633]: pgmap v16716: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:11:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:40 smithi104 ceph-mon[97044]: pgmap v16716: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:11:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:40.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:40.857441+0000) 2022-09-10T13:11:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:41.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:41.466075+0000) 2022-09-10T13:11:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:41.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:41.857685+0000) 2022-09-10T13:11:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:42.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:42.466371+0000) 2022-09-10T13:11:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:42 smithi110 ceph-mon[97633]: pgmap v16717: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:42 smithi104 ceph-mon[97044]: pgmap v16717: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:42.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:42.857943+0000) 2022-09-10T13:11:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:43.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:43.466602+0000) 2022-09-10T13:11:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:43.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:43.858111+0000) 2022-09-10T13:11:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:44.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:44.466786+0000) 2022-09-10T13:11:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:44 smithi110 ceph-mon[97633]: pgmap v16718: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:44 smithi104 ceph-mon[97044]: pgmap v16718: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:44.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:44.858291+0000) 2022-09-10T13:11:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:45.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:45.467038+0000) 2022-09-10T13:11:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:45.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:45.858500+0000) 2022-09-10T13:11:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:46 smithi110 ceph-mon[97633]: pgmap v16719: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:11:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:11:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:11:46.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:11:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:46 smithi104 ceph-mon[97044]: pgmap v16719: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:11:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:11:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:11:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:11:46.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:11:46.685+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:11:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:46.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:46.467236+0000) 2022-09-10T13:11:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:46.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:46.858686+0000) 2022-09-10T13:11:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:47.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:47.467498+0000) 2022-09-10T13:11:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:47.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:47.858882+0000) 2022-09-10T13:11:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:48.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:48.467682+0000) 2022-09-10T13:11:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:48 smithi110 ceph-mon[97633]: pgmap v16720: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:48 smithi104 ceph-mon[97044]: pgmap v16720: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:48.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:48.859142+0000) 2022-09-10T13:11:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:49.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:49.467931+0000) 2022-09-10T13:11:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:49.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:49.859434+0000) 2022-09-10T13:11:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:50.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:50.468207+0000) 2022-09-10T13:11:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:50 smithi110 ceph-mon[97633]: pgmap v16721: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:50 smithi104 ceph-mon[97044]: pgmap v16721: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:50.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:50.859592+0000) 2022-09-10T13:11:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:51.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:51.468404+0000) 2022-09-10T13:11:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:51.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:51.859749+0000) 2022-09-10T13:11:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:52.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:52.468581+0000) 2022-09-10T13:11:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:52 smithi110 ceph-mon[97633]: pgmap v16722: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:52 smithi104 ceph-mon[97044]: pgmap v16722: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:52.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:52.860027+0000) 2022-09-10T13:11:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:53.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:53.468761+0000) 2022-09-10T13:11:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:53.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:53.860213+0000) 2022-09-10T13:11:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:54.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:54.469002+0000) 2022-09-10T13:11:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:54 smithi110 ceph-mon[97633]: pgmap v16723: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:54 smithi104 ceph-mon[97044]: pgmap v16723: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:54.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:54.860481+0000) 2022-09-10T13:11:55.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:55.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:55.469222+0000) 2022-09-10T13:11:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:55.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:55.860737+0000) 2022-09-10T13:11:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:56 smithi104 ceph-mon[97044]: pgmap v16724: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:56.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:56.469486+0000) 2022-09-10T13:11:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:56 smithi110 ceph-mon[97633]: pgmap v16724: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:11:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:56.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:56.860985+0000) 2022-09-10T13:11:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:57.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:57.469719+0000) 2022-09-10T13:11:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:57.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:57.861274+0000) 2022-09-10T13:11:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:58.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:58.469898+0000) 2022-09-10T13:11:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:11:58 smithi110 ceph-mon[97633]: pgmap v16725: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:11:58 smithi104 ceph-mon[97044]: pgmap v16725: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:11:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:58.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:58.861474+0000) 2022-09-10T13:11:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:11:59.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:59.470211+0000) 2022-09-10T13:12:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:11:59.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:11:59.861682+0000) 2022-09-10T13:12:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:00.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:00.470427+0000) 2022-09-10T13:12:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:00 smithi110 ceph-mon[97633]: pgmap v16726: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:00 smithi104 ceph-mon[97044]: pgmap v16726: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:00.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:00.861839+0000) 2022-09-10T13:12:01.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:12:01.698+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:12:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:01.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:01.470630+0000) 2022-09-10T13:12:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:01.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:01.862156+0000) 2022-09-10T13:12:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:02.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:02.470862+0000) 2022-09-10T13:12:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:02 smithi110 ceph-mon[97633]: pgmap v16727: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:02 smithi104 ceph-mon[97044]: pgmap v16727: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:02.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:02.862313+0000) 2022-09-10T13:12:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:03.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:03.471071+0000) 2022-09-10T13:12:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:03.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:03.862540+0000) 2022-09-10T13:12:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:04.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:04.471207+0000) 2022-09-10T13:12:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:04 smithi110 ceph-mon[97633]: pgmap v16728: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:04 smithi104 ceph-mon[97044]: pgmap v16728: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:04.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:04.862803+0000) 2022-09-10T13:12:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:05.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:05.471410+0000) 2022-09-10T13:12:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:05.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:05.862962+0000) 2022-09-10T13:12:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:06 smithi104 ceph-mon[97044]: pgmap v16729: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:12:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:06.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:06.471560+0000) 2022-09-10T13:12:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:06 smithi110 ceph-mon[97633]: pgmap v16729: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:12:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:06.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:06.863218+0000) 2022-09-10T13:12:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:07.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:07.471769+0000) 2022-09-10T13:12:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:07.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:07.863461+0000) 2022-09-10T13:12:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:08.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:08.472022+0000) 2022-09-10T13:12:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:08 smithi104 ceph-mon[97044]: osdmap e1126: 12 total, 12 up, 12 in 2022-09-10T13:12:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:08 smithi104 ceph-mon[97044]: pgmap v16731: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:12:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:08 smithi110 ceph-mon[97633]: osdmap e1126: 12 total, 12 up, 12 in 2022-09-10T13:12:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:08 smithi110 ceph-mon[97633]: pgmap v16731: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:12:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:08.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:08.863637+0000) 2022-09-10T13:12:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:09.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:09.472224+0000) 2022-09-10T13:12:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:09.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:09.863890+0000) 2022-09-10T13:12:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:10.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:10.472491+0000) 2022-09-10T13:12:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:10 smithi110 ceph-mon[97633]: pgmap v16732: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:12:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:10 smithi110 ceph-mon[97633]: osdmap e1127: 12 total, 12 up, 12 in 2022-09-10T13:12:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:10 smithi104 ceph-mon[97044]: pgmap v16732: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:12:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:10 smithi104 ceph-mon[97044]: osdmap e1127: 12 total, 12 up, 12 in 2022-09-10T13:12:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:10.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:10.864207+0000) 2022-09-10T13:12:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:11.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:11.472777+0000) 2022-09-10T13:12:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:11.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:11.864358+0000) 2022-09-10T13:12:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:12.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:12.472945+0000) 2022-09-10T13:12:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:12 smithi110 ceph-mon[97633]: pgmap v16734: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:12:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:12 smithi104 ceph-mon[97044]: pgmap v16734: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:12:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:12.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:12.864533+0000) 2022-09-10T13:12:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:13.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:13.473207+0000) 2022-09-10T13:12:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:13.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:13.864729+0000) 2022-09-10T13:12:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:14.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:14.473467+0000) 2022-09-10T13:12:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:14 smithi110 ceph-mon[97633]: pgmap v16735: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:12:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:14 smithi104 ceph-mon[97044]: pgmap v16735: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:12:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:14.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:14.864909+0000) 2022-09-10T13:12:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:15.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:15.473774+0000) 2022-09-10T13:12:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:15.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:15.865183+0000) 2022-09-10T13:12:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:16 smithi104 ceph-mon[97044]: pgmap v16736: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T13:12:16.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:12:16.708+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:12:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:16.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:16.474034+0000) 2022-09-10T13:12:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:16 smithi110 ceph-mon[97633]: pgmap v16736: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T13:12:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:16.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:16.865379+0000) 2022-09-10T13:12:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:17.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:17.474263+0000) 2022-09-10T13:12:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:17.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:17.865608+0000) 2022-09-10T13:12:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:18.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:18.474526+0000) 2022-09-10T13:12:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:18 smithi110 ceph-mon[97633]: pgmap v16737: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:12:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:18 smithi104 ceph-mon[97044]: pgmap v16737: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:12:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:18.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:18.865809+0000) 2022-09-10T13:12:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:19.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:19.474804+0000) 2022-09-10T13:12:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:19.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:19.865984+0000) 2022-09-10T13:12:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:20.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:20.475002+0000) 2022-09-10T13:12:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:20 smithi110 ceph-mon[97633]: pgmap v16738: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:12:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:20 smithi104 ceph-mon[97044]: pgmap v16738: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:12:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:20.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:20.866228+0000) 2022-09-10T13:12:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:21.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:21.475201+0000) 2022-09-10T13:12:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:21.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:21.866522+0000) 2022-09-10T13:12:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:22.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:22.475406+0000) 2022-09-10T13:12:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:22 smithi110 ceph-mon[97633]: pgmap v16739: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:12:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:22 smithi104 ceph-mon[97044]: pgmap v16739: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:12:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:22.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:22.866744+0000) 2022-09-10T13:12:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:23.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:23.475625+0000) 2022-09-10T13:12:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:23.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:23.866943+0000) 2022-09-10T13:12:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:24.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:24.475802+0000) 2022-09-10T13:12:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:24 smithi110 ceph-mon[97633]: pgmap v16740: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:12:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:24 smithi104 ceph-mon[97044]: pgmap v16740: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:12:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:24.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:24.867192+0000) 2022-09-10T13:12:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:25.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:25.476072+0000) 2022-09-10T13:12:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:25.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:25.867358+0000) 2022-09-10T13:12:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:26 smithi104 ceph-mon[97044]: pgmap v16741: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:12:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:26.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:26.476351+0000) 2022-09-10T13:12:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:26 smithi110 ceph-mon[97633]: pgmap v16741: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:12:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:26.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:26.867549+0000) 2022-09-10T13:12:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:27.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:27.476589+0000) 2022-09-10T13:12:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:27.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:27.867687+0000) 2022-09-10T13:12:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:28.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:28.476793+0000) 2022-09-10T13:12:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:28 smithi110 ceph-mon[97633]: pgmap v16742: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:28 smithi104 ceph-mon[97044]: pgmap v16742: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:28.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:28.867848+0000) 2022-09-10T13:12:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:29.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:29.477029+0000) 2022-09-10T13:12:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:29.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:29.868016+0000) 2022-09-10T13:12:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:30.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:30.477285+0000) 2022-09-10T13:12:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:30 smithi110 ceph-mon[97633]: pgmap v16743: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:30 smithi104 ceph-mon[97044]: pgmap v16743: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:30.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:30.868257+0000) 2022-09-10T13:12:31.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:12:31.720+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:12:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:31.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:31.477535+0000) 2022-09-10T13:12:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:31.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:31.868490+0000) 2022-09-10T13:12:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:32.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:32.477746+0000) 2022-09-10T13:12:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:32 smithi110 ceph-mon[97633]: pgmap v16744: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:32 smithi104 ceph-mon[97044]: pgmap v16744: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:32.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:32.868757+0000) 2022-09-10T13:12:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:33.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:33.477976+0000) 2022-09-10T13:12:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:33.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:33.869022+0000) 2022-09-10T13:12:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:34.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:34.478261+0000) 2022-09-10T13:12:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:34 smithi110 ceph-mon[97633]: pgmap v16745: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:34 smithi104 ceph-mon[97044]: pgmap v16745: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:34.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:34.869269+0000) 2022-09-10T13:12:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:35.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:35.478553+0000) 2022-09-10T13:12:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:35.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:35.869435+0000) 2022-09-10T13:12:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:36 smithi104 ceph-mon[97044]: pgmap v16746: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:36.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:36.478745+0000) 2022-09-10T13:12:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:36 smithi110 ceph-mon[97633]: pgmap v16746: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:36.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:36.869662+0000) 2022-09-10T13:12:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:37.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:37.478867+0000) 2022-09-10T13:12:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:37.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:37.869928+0000) 2022-09-10T13:12:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:38.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:38.479075+0000) 2022-09-10T13:12:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:38 smithi110 ceph-mon[97633]: pgmap v16747: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:38 smithi104 ceph-mon[97044]: pgmap v16747: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:38.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:38.870143+0000) 2022-09-10T13:12:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:39.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:39.479246+0000) 2022-09-10T13:12:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:39.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:39.870364+0000) 2022-09-10T13:12:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:40.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:40.479449+0000) 2022-09-10T13:12:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:40 smithi110 ceph-mon[97633]: pgmap v16748: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:40 smithi104 ceph-mon[97044]: pgmap v16748: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:40.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:40.870597+0000) 2022-09-10T13:12:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:41.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:41.479687+0000) 2022-09-10T13:12:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:41.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:41.870810+0000) 2022-09-10T13:12:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:42.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:42.479884+0000) 2022-09-10T13:12:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:42 smithi110 ceph-mon[97633]: pgmap v16749: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:42 smithi104 ceph-mon[97044]: pgmap v16749: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:42.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:42.871045+0000) 2022-09-10T13:12:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:43.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:43.480075+0000) 2022-09-10T13:12:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:43.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:43.871322+0000) 2022-09-10T13:12:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:44.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:44.480381+0000) 2022-09-10T13:12:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:44 smithi110 ceph-mon[97633]: pgmap v16750: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:44 smithi104 ceph-mon[97044]: pgmap v16750: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:44.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:44.871531+0000) 2022-09-10T13:12:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:45.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:45.480543+0000) 2022-09-10T13:12:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:45.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:45.871754+0000) 2022-09-10T13:12:46.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:46.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:46.480796+0000) 2022-09-10T13:12:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:46 smithi104 ceph-mon[97044]: pgmap v16751: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:12:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:46 smithi110 ceph-mon[97633]: pgmap v16751: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:12:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:12:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:12:47.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:46.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:46.871973+0000) 2022-09-10T13:12:47.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:12:46.734+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:12:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:47.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:47.481064+0000) 2022-09-10T13:12:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:47.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:47.872138+0000) 2022-09-10T13:12:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:48.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:48.481364+0000) 2022-09-10T13:12:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:48 smithi110 ceph-mon[97633]: pgmap v16752: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:48 smithi104 ceph-mon[97044]: pgmap v16752: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:48.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:48.872345+0000) 2022-09-10T13:12:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:49.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:49.481518+0000) 2022-09-10T13:12:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:49.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:49.872517+0000) 2022-09-10T13:12:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:50.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:50.481764+0000) 2022-09-10T13:12:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:50 smithi110 ceph-mon[97633]: pgmap v16753: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:50 smithi104 ceph-mon[97044]: pgmap v16753: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:50.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:50.872731+0000) 2022-09-10T13:12:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:51.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:51.482058+0000) 2022-09-10T13:12:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:51.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:51.872956+0000) 2022-09-10T13:12:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:52.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:52.482285+0000) 2022-09-10T13:12:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:52 smithi110 ceph-mon[97633]: pgmap v16754: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:52 smithi104 ceph-mon[97044]: pgmap v16754: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:52.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:52.873202+0000) 2022-09-10T13:12:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:53.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:53.482412+0000) 2022-09-10T13:12:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:53.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:53.873436+0000) 2022-09-10T13:12:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:54.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:54.482603+0000) 2022-09-10T13:12:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:54 smithi110 ceph-mon[97633]: pgmap v16755: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:54 smithi104 ceph-mon[97044]: pgmap v16755: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:54.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:54.873642+0000) 2022-09-10T13:12:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:55.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:55.482868+0000) 2022-09-10T13:12:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:55.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:55.873877+0000) 2022-09-10T13:12:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:56 smithi104 ceph-mon[97044]: pgmap v16756: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:56.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:56.483012+0000) 2022-09-10T13:12:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:56 smithi110 ceph-mon[97633]: pgmap v16756: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:12:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:56.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:56.874195+0000) 2022-09-10T13:12:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:57.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:57.483246+0000) 2022-09-10T13:12:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:57.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:57.874422+0000) 2022-09-10T13:12:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:58.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:58.483497+0000) 2022-09-10T13:12:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:12:58 smithi110 ceph-mon[97633]: pgmap v16757: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:12:58 smithi104 ceph-mon[97044]: pgmap v16757: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:12:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:58.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:58.874569+0000) 2022-09-10T13:12:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:12:59.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:59.483743+0000) 2022-09-10T13:13:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:12:59.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:12:59.874809+0000) 2022-09-10T13:13:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:00.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:00.483903+0000) 2022-09-10T13:13:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:00 smithi110 ceph-mon[97633]: pgmap v16758: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:00 smithi104 ceph-mon[97044]: pgmap v16758: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:00.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:00.875049+0000) 2022-09-10T13:13:01.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:01.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:01.484124+0000) 2022-09-10T13:13:02.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:01.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:01.875289+0000) 2022-09-10T13:13:02.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:13:01.750+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:13:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:02.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:02.484298+0000) 2022-09-10T13:13:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:02 smithi110 ceph-mon[97633]: pgmap v16759: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:02 smithi104 ceph-mon[97044]: pgmap v16759: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:02.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:02.875472+0000) 2022-09-10T13:13:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:03.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:03.484511+0000) 2022-09-10T13:13:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:03.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:03.875645+0000) 2022-09-10T13:13:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:04.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:04.484755+0000) 2022-09-10T13:13:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:04 smithi104 ceph-mon[97044]: pgmap v16760: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:04.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:04.875860+0000) 2022-09-10T13:13:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:04 smithi110 ceph-mon[97633]: pgmap v16760: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:05.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:05.485022+0000) 2022-09-10T13:13:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:05.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:05.876136+0000) 2022-09-10T13:13:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:06 smithi104 ceph-mon[97044]: pgmap v16761: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:13:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:06.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:06.485210+0000) 2022-09-10T13:13:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:06 smithi110 ceph-mon[97633]: pgmap v16761: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:13:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:06.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:06.876342+0000) 2022-09-10T13:13:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:07.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:07.485433+0000) 2022-09-10T13:13:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:07.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:07.876544+0000) 2022-09-10T13:13:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:08.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:08.485639+0000) 2022-09-10T13:13:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:08 smithi104 ceph-mon[97044]: osdmap e1128: 12 total, 12 up, 12 in 2022-09-10T13:13:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:08 smithi104 ceph-mon[97044]: pgmap v16763: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:13:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:08 smithi110 ceph-mon[97633]: osdmap e1128: 12 total, 12 up, 12 in 2022-09-10T13:13:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:08 smithi110 ceph-mon[97633]: pgmap v16763: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:13:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:08.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:08.876788+0000) 2022-09-10T13:13:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:09.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:09.485846+0000) 2022-09-10T13:13:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:09.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:09.877031+0000) 2022-09-10T13:13:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:10.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:10.486133+0000) 2022-09-10T13:13:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:10 smithi110 ceph-mon[97633]: pgmap v16764: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:13:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:10 smithi110 ceph-mon[97633]: osdmap e1129: 12 total, 12 up, 12 in 2022-09-10T13:13:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:10 smithi104 ceph-mon[97044]: pgmap v16764: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:13:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:10 smithi104 ceph-mon[97044]: osdmap e1129: 12 total, 12 up, 12 in 2022-09-10T13:13:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:10.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:10.877284+0000) 2022-09-10T13:13:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:11.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:11.486349+0000) 2022-09-10T13:13:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:11.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:11.877498+0000) 2022-09-10T13:13:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:12.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:12.486534+0000) 2022-09-10T13:13:13.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:12 smithi110 ceph-mon[97633]: pgmap v16766: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:13:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:12 smithi104 ceph-mon[97044]: pgmap v16766: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:13:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:12.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:12.877742+0000) 2022-09-10T13:13:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:13.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:13.486732+0000) 2022-09-10T13:13:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:13.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:13.877994+0000) 2022-09-10T13:13:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:14.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:14.486956+0000) 2022-09-10T13:13:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:14 smithi104 ceph-mon[97044]: pgmap v16767: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:13:15.346 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:14.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:14.878180+0000) 2022-09-10T13:13:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:14 smithi110 ceph-mon[97633]: pgmap v16767: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:13:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:15.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:15.487176+0000) 2022-09-10T13:13:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:15.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:15.878408+0000) 2022-09-10T13:13:16.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:16.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:16.487389+0000) 2022-09-10T13:13:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:16 smithi104 ceph-mon[97044]: pgmap v16768: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T13:13:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:16 smithi110 ceph-mon[97633]: pgmap v16768: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T13:13:17.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:13:16.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:13:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:16.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:16.878695+0000) 2022-09-10T13:13:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:17.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:17.487573+0000) 2022-09-10T13:13:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:17.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:17.878898+0000) 2022-09-10T13:13:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:18.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:18.487802+0000) 2022-09-10T13:13:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:18 smithi110 ceph-mon[97633]: pgmap v16769: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:13:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:18 smithi104 ceph-mon[97044]: pgmap v16769: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:13:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:18.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:18.879087+0000) 2022-09-10T13:13:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:19.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:19.488018+0000) 2022-09-10T13:13:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:19.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:19.879333+0000) 2022-09-10T13:13:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:20.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:20.488196+0000) 2022-09-10T13:13:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:20 smithi104 ceph-mon[97044]: pgmap v16770: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:13:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:20.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:20.879606+0000) 2022-09-10T13:13:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:20 smithi110 ceph-mon[97633]: pgmap v16770: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:13:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:21.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:21.488412+0000) 2022-09-10T13:13:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:21.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:21.879837+0000) 2022-09-10T13:13:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:22.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:22.488579+0000) 2022-09-10T13:13:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:22 smithi104 ceph-mon[97044]: pgmap v16771: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:13:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:22.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:22.880047+0000) 2022-09-10T13:13:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:22 smithi110 ceph-mon[97633]: pgmap v16771: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:13:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:23.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:23.488827+0000) 2022-09-10T13:13:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:23.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:23.880290+0000) 2022-09-10T13:13:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:24.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:24.489076+0000) 2022-09-10T13:13:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:24 smithi104 ceph-mon[97044]: pgmap v16772: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:13:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:24.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:24.880571+0000) 2022-09-10T13:13:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:24 smithi110 ceph-mon[97633]: pgmap v16772: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:13:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:25.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:25.489303+0000) 2022-09-10T13:13:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:25.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:25.880820+0000) 2022-09-10T13:13:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:26 smithi104 ceph-mon[97044]: pgmap v16773: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:13:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:26.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:26.489497+0000) 2022-09-10T13:13:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:26 smithi110 ceph-mon[97633]: pgmap v16773: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:13:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:26.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:26.881066+0000) 2022-09-10T13:13:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:27.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:27.489735+0000) 2022-09-10T13:13:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:27.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:27.881341+0000) 2022-09-10T13:13:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:28.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:28.489979+0000) 2022-09-10T13:13:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:28 smithi104 ceph-mon[97044]: pgmap v16774: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:28.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:28.881602+0000) 2022-09-10T13:13:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:28 smithi110 ceph-mon[97633]: pgmap v16774: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:29.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:29.490225+0000) 2022-09-10T13:13:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:29.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:29.881802+0000) 2022-09-10T13:13:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:30.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:30.490503+0000) 2022-09-10T13:13:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:30 smithi104 ceph-mon[97044]: pgmap v16775: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:30.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:30.882025+0000) 2022-09-10T13:13:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:30 smithi110 ceph-mon[97633]: pgmap v16775: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:31.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:31.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:31.490752+0000) 2022-09-10T13:13:32.053 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:31.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:31.882263+0000) 2022-09-10T13:13:32.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:13:31.771+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:13:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:32.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:32.490989+0000) 2022-09-10T13:13:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:32 smithi104 ceph-mon[97044]: pgmap v16776: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:32.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:32.882473+0000) 2022-09-10T13:13:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:32 smithi110 ceph-mon[97633]: pgmap v16776: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:33.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:33.491236+0000) 2022-09-10T13:13:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:33.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:33.882676+0000) 2022-09-10T13:13:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:34.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:34.491431+0000) 2022-09-10T13:13:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:34 smithi104 ceph-mon[97044]: pgmap v16777: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:34.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:34.882856+0000) 2022-09-10T13:13:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:34 smithi110 ceph-mon[97633]: pgmap v16777: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:35.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:35.491629+0000) 2022-09-10T13:13:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:35.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:35.883060+0000) 2022-09-10T13:13:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:36 smithi104 ceph-mon[97044]: pgmap v16778: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:36.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:36.491859+0000) 2022-09-10T13:13:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:36 smithi110 ceph-mon[97633]: pgmap v16778: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:36.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:36.883275+0000) 2022-09-10T13:13:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:37.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:37.492143+0000) 2022-09-10T13:13:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:37.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:37.883494+0000) 2022-09-10T13:13:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:38.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:38.492315+0000) 2022-09-10T13:13:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:38 smithi104 ceph-mon[97044]: pgmap v16779: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:38.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:38.883701+0000) 2022-09-10T13:13:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:38 smithi110 ceph-mon[97633]: pgmap v16779: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:39.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:39.492461+0000) 2022-09-10T13:13:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:39.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:39.883911+0000) 2022-09-10T13:13:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:40.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:40.492701+0000) 2022-09-10T13:13:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:40 smithi104 ceph-mon[97044]: pgmap v16780: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:40.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:40.884135+0000) 2022-09-10T13:13:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:40 smithi110 ceph-mon[97633]: pgmap v16780: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:41.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:41.492892+0000) 2022-09-10T13:13:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:41.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:41.884353+0000) 2022-09-10T13:13:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:42.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:42.493147+0000) 2022-09-10T13:13:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:42 smithi104 ceph-mon[97044]: pgmap v16781: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:42.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:42.884550+0000) 2022-09-10T13:13:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:42 smithi110 ceph-mon[97633]: pgmap v16781: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:43.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:43.493377+0000) 2022-09-10T13:13:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:43.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:43.884782+0000) 2022-09-10T13:13:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:44.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:44.493547+0000) 2022-09-10T13:13:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:44 smithi104 ceph-mon[97044]: pgmap v16782: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:44.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:44.885062+0000) 2022-09-10T13:13:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:44 smithi110 ceph-mon[97633]: pgmap v16782: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:45.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:45.493784+0000) 2022-09-10T13:13:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:45.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:45.885359+0000) 2022-09-10T13:13:46.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:46.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:46.494006+0000) 2022-09-10T13:13:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:46 smithi104 ceph-mon[97044]: pgmap v16783: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:13:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:13:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:13:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:13:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:46 smithi110 ceph-mon[97633]: pgmap v16783: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:13:47.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:13:46.774+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:13:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:46.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:46.885535+0000) 2022-09-10T13:13:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:47.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:47.494161+0000) 2022-09-10T13:13:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:47.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:47.885804+0000) 2022-09-10T13:13:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:48.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:48.494371+0000) 2022-09-10T13:13:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:48 smithi104 ceph-mon[97044]: pgmap v16784: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:48.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:48.885974+0000) 2022-09-10T13:13:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:48 smithi110 ceph-mon[97633]: pgmap v16784: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:49.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:49.494610+0000) 2022-09-10T13:13:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:49.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:49.886206+0000) 2022-09-10T13:13:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:50.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:50.494821+0000) 2022-09-10T13:13:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:50 smithi104 ceph-mon[97044]: pgmap v16785: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:50.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:50.886464+0000) 2022-09-10T13:13:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:50 smithi110 ceph-mon[97633]: pgmap v16785: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:51.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:51.495090+0000) 2022-09-10T13:13:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:51.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:51.886658+0000) 2022-09-10T13:13:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:52.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:52.495323+0000) 2022-09-10T13:13:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:52 smithi104 ceph-mon[97044]: pgmap v16786: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:52.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:52.886900+0000) 2022-09-10T13:13:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:52 smithi110 ceph-mon[97633]: pgmap v16786: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:53.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:53.495532+0000) 2022-09-10T13:13:54.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:53.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:53.887179+0000) 2022-09-10T13:13:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:54.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:54.495791+0000) 2022-09-10T13:13:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:54 smithi104 ceph-mon[97044]: pgmap v16787: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:54.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:54.887403+0000) 2022-09-10T13:13:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:54 smithi110 ceph-mon[97633]: pgmap v16787: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:55.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:55.496070+0000) 2022-09-10T13:13:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:55.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:55.887560+0000) 2022-09-10T13:13:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:56 smithi104 ceph-mon[97044]: pgmap v16788: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:56.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:56.496345+0000) 2022-09-10T13:13:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:56 smithi110 ceph-mon[97633]: pgmap v16788: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:13:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:56.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:56.887776+0000) 2022-09-10T13:13:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:57.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:57.496559+0000) 2022-09-10T13:13:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:57.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:57.887959+0000) 2022-09-10T13:13:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:58.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:58.496838+0000) 2022-09-10T13:13:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:13:58 smithi104 ceph-mon[97044]: pgmap v16789: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:58.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:58.888190+0000) 2022-09-10T13:13:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:13:58 smithi110 ceph-mon[97633]: pgmap v16789: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:13:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:13:59.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:59.497035+0000) 2022-09-10T13:14:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:13:59.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:13:59.888387+0000) 2022-09-10T13:14:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:00.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:00.497235+0000) 2022-09-10T13:14:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:00 smithi104 ceph-mon[97044]: pgmap v16790: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:00.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:00.888633+0000) 2022-09-10T13:14:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:00 smithi110 ceph-mon[97633]: pgmap v16790: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:01.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:01.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:01.497444+0000) 2022-09-10T13:14:02.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:14:01.784+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:14:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:01.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:01.888781+0000) 2022-09-10T13:14:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:02.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:02.497710+0000) 2022-09-10T13:14:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:02 smithi104 ceph-mon[97044]: pgmap v16791: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:02.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:02.889027+0000) 2022-09-10T13:14:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:02 smithi110 ceph-mon[97633]: pgmap v16791: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:03.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:03.497933+0000) 2022-09-10T13:14:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:03.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:03.889145+0000) 2022-09-10T13:14:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:04.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:04.498155+0000) 2022-09-10T13:14:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:04 smithi104 ceph-mon[97044]: pgmap v16792: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:04.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:04.889375+0000) 2022-09-10T13:14:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:04 smithi110 ceph-mon[97633]: pgmap v16792: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:05.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:05.498338+0000) 2022-09-10T13:14:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:05.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:05.889568+0000) 2022-09-10T13:14:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:06 smithi104 ceph-mon[97044]: pgmap v16793: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:14:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:06.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:06.498573+0000) 2022-09-10T13:14:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:06 smithi110 ceph-mon[97633]: pgmap v16793: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:14:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:06.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:06.889797+0000) 2022-09-10T13:14:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:07.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:07.498781+0000) 2022-09-10T13:14:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:07.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:07.889975+0000) 2022-09-10T13:14:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:08.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:08.498979+0000) 2022-09-10T13:14:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:08 smithi104 ceph-mon[97044]: osdmap e1130: 12 total, 12 up, 12 in 2022-09-10T13:14:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:08 smithi104 ceph-mon[97044]: pgmap v16795: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:14:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:08 smithi110 ceph-mon[97633]: osdmap e1130: 12 total, 12 up, 12 in 2022-09-10T13:14:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:08 smithi110 ceph-mon[97633]: pgmap v16795: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:14:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:08.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:08.890203+0000) 2022-09-10T13:14:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:09.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:09.499252+0000) 2022-09-10T13:14:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:09.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:09.890394+0000) 2022-09-10T13:14:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:10.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:10.499500+0000) 2022-09-10T13:14:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:10 smithi104 ceph-mon[97044]: pgmap v16796: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:14:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:10 smithi104 ceph-mon[97044]: osdmap e1131: 12 total, 12 up, 12 in 2022-09-10T13:14:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:10.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:10.890642+0000) 2022-09-10T13:14:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:10 smithi110 ceph-mon[97633]: pgmap v16796: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:14:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:10 smithi110 ceph-mon[97633]: osdmap e1131: 12 total, 12 up, 12 in 2022-09-10T13:14:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:11.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:11.499744+0000) 2022-09-10T13:14:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:11.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:11.890828+0000) 2022-09-10T13:14:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:12.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:12.499920+0000) 2022-09-10T13:14:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:12 smithi104 ceph-mon[97044]: pgmap v16798: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:14:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:12.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:12.891053+0000) 2022-09-10T13:14:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:12 smithi110 ceph-mon[97633]: pgmap v16798: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:14:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:13.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:13.500155+0000) 2022-09-10T13:14:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:13.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:13.891281+0000) 2022-09-10T13:14:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:14.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:14.500347+0000) 2022-09-10T13:14:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:14 smithi104 ceph-mon[97044]: pgmap v16799: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:14:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:14.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:14.891509+0000) 2022-09-10T13:14:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:14 smithi110 ceph-mon[97633]: pgmap v16799: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:14:15.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:15.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:15.500535+0000) 2022-09-10T13:14:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:15.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:15.891714+0000) 2022-09-10T13:14:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:16 smithi104 ceph-mon[97044]: pgmap v16800: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T13:14:16.819 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:16.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:16.500755+0000) 2022-09-10T13:14:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:16 smithi110 ceph-mon[97633]: pgmap v16800: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 253 B/s wr, 12 op/s 2022-09-10T13:14:17.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:14:16.800+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:14:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:16.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:16.891902+0000) 2022-09-10T13:14:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:17.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:17.501005+0000) 2022-09-10T13:14:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:17.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:17.892157+0000) 2022-09-10T13:14:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:18.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:18.501212+0000) 2022-09-10T13:14:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:18 smithi104 ceph-mon[97044]: pgmap v16801: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:14:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:18.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:18.892387+0000) 2022-09-10T13:14:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:18 smithi110 ceph-mon[97633]: pgmap v16801: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:14:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:19.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:19.501471+0000) 2022-09-10T13:14:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:19.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:19.892609+0000) 2022-09-10T13:14:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:20.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:20.501639+0000) 2022-09-10T13:14:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:20 smithi104 ceph-mon[97044]: pgmap v16802: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:14:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:20.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:20.892778+0000) 2022-09-10T13:14:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:20 smithi110 ceph-mon[97633]: pgmap v16802: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:14:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:21.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:21.501862+0000) 2022-09-10T13:14:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:21.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:21.892997+0000) 2022-09-10T13:14:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:22.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:22.502133+0000) 2022-09-10T13:14:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:22 smithi104 ceph-mon[97044]: pgmap v16803: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:14:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:22.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:22.893241+0000) 2022-09-10T13:14:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:22 smithi110 ceph-mon[97633]: pgmap v16803: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:14:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:23.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:23.502359+0000) 2022-09-10T13:14:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:23.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:23.893462+0000) 2022-09-10T13:14:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:24.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:24.502621+0000) 2022-09-10T13:14:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:24 smithi104 ceph-mon[97044]: pgmap v16804: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:14:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:24.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:24.893732+0000) 2022-09-10T13:14:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:24 smithi110 ceph-mon[97633]: pgmap v16804: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:14:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:25.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:25.502892+0000) 2022-09-10T13:14:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:25.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:25.893927+0000) 2022-09-10T13:14:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:26 smithi104 ceph-mon[97044]: pgmap v16805: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:14:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:26.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:26.503154+0000) 2022-09-10T13:14:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:26 smithi110 ceph-mon[97633]: pgmap v16805: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:14:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:26.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:26.894201+0000) 2022-09-10T13:14:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:27.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:27.503412+0000) 2022-09-10T13:14:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:27.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:27.894362+0000) 2022-09-10T13:14:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:28.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:28.503611+0000) 2022-09-10T13:14:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:28 smithi104 ceph-mon[97044]: pgmap v16806: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:14:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:28.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:28.894605+0000) 2022-09-10T13:14:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:28 smithi110 ceph-mon[97633]: pgmap v16806: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:14:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:29.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:29.503817+0000) 2022-09-10T13:14:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:29.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:29.894864+0000) 2022-09-10T13:14:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:30.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:30.504130+0000) 2022-09-10T13:14:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:30 smithi104 ceph-mon[97044]: pgmap v16807: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:30.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:30.895067+0000) 2022-09-10T13:14:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:30 smithi110 ceph-mon[97633]: pgmap v16807: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:31.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:31.504376+0000) 2022-09-10T13:14:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:31.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:31.895305+0000) 2022-09-10T13:14:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:14:31.815+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:14:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:32.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:32.504538+0000) 2022-09-10T13:14:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:32 smithi104 ceph-mon[97044]: pgmap v16808: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:14:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:32.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:32.895463+0000) 2022-09-10T13:14:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:32 smithi110 ceph-mon[97633]: pgmap v16808: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:14:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:33.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:33.504704+0000) 2022-09-10T13:14:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:33.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:33.895631+0000) 2022-09-10T13:14:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:34.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:34.504941+0000) 2022-09-10T13:14:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:34 smithi104 ceph-mon[97044]: pgmap v16809: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:14:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:34.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:34.895857+0000) 2022-09-10T13:14:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:34 smithi110 ceph-mon[97633]: pgmap v16809: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:14:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:35.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:35.505196+0000) 2022-09-10T13:14:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:35.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:35.896130+0000) 2022-09-10T13:14:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:36 smithi104 ceph-mon[97044]: pgmap v16810: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:36.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:36.505388+0000) 2022-09-10T13:14:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:36 smithi110 ceph-mon[97633]: pgmap v16810: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:36.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:36.896361+0000) 2022-09-10T13:14:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:37.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:37.505593+0000) 2022-09-10T13:14:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:37.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:37.896532+0000) 2022-09-10T13:14:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:38.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:38.505852+0000) 2022-09-10T13:14:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:38 smithi104 ceph-mon[97044]: pgmap v16811: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:38.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:38.896720+0000) 2022-09-10T13:14:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:38 smithi110 ceph-mon[97633]: pgmap v16811: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:39.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:39.506151+0000) 2022-09-10T13:14:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:39.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:39.896973+0000) 2022-09-10T13:14:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:40.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:40.506410+0000) 2022-09-10T13:14:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:40 smithi104 ceph-mon[97044]: pgmap v16812: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:40.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:40.897126+0000) 2022-09-10T13:14:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:40 smithi110 ceph-mon[97633]: pgmap v16812: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:41.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:41.506667+0000) 2022-09-10T13:14:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:41.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:41.897231+0000) 2022-09-10T13:14:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:42.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:42.506946+0000) 2022-09-10T13:14:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:42 smithi104 ceph-mon[97044]: pgmap v16813: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:42.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:42.897480+0000) 2022-09-10T13:14:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:42 smithi110 ceph-mon[97633]: pgmap v16813: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:43.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:43.507202+0000) 2022-09-10T13:14:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:43.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:43.897746+0000) 2022-09-10T13:14:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:44.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:44.507453+0000) 2022-09-10T13:14:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:44 smithi104 ceph-mon[97044]: pgmap v16814: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:44.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:44.897980+0000) 2022-09-10T13:14:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:44 smithi110 ceph-mon[97633]: pgmap v16814: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:45.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:45.507715+0000) 2022-09-10T13:14:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:45.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:45.898207+0000) 2022-09-10T13:14:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:46 smithi104 ceph-mon[97044]: pgmap v16815: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:14:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:14:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:14:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:14:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:46.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:46.507976+0000) 2022-09-10T13:14:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:46 smithi110 ceph-mon[97633]: pgmap v16815: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:14:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:14:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:14:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:14:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:46.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:46.898396+0000) 2022-09-10T13:14:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:14:46.830+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:14:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:47.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:47.508197+0000) 2022-09-10T13:14:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:47.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:47.898591+0000) 2022-09-10T13:14:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:48.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:48.508356+0000) 2022-09-10T13:14:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:48 smithi104 ceph-mon[97044]: pgmap v16816: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:48.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:48.898777+0000) 2022-09-10T13:14:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:48 smithi110 ceph-mon[97633]: pgmap v16816: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:49.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:49.508574+0000) 2022-09-10T13:14:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:49.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:49.899003+0000) 2022-09-10T13:14:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:50.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:50.508843+0000) 2022-09-10T13:14:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:50 smithi104 ceph-mon[97044]: pgmap v16817: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:50.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:50.899207+0000) 2022-09-10T13:14:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:50 smithi110 ceph-mon[97633]: pgmap v16817: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:51.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:51.509089+0000) 2022-09-10T13:14:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:51.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:51.899403+0000) 2022-09-10T13:14:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:52.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:52.509377+0000) 2022-09-10T13:14:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:52 smithi104 ceph-mon[97044]: pgmap v16818: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:52.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:52.899662+0000) 2022-09-10T13:14:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:52 smithi110 ceph-mon[97633]: pgmap v16818: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:53.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:53.509524+0000) 2022-09-10T13:14:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:53.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:53.899862+0000) 2022-09-10T13:14:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:54.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:54.509740+0000) 2022-09-10T13:14:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:54 smithi104 ceph-mon[97044]: pgmap v16819: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:54.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:54.900070+0000) 2022-09-10T13:14:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:54 smithi110 ceph-mon[97633]: pgmap v16819: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:55.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:55.509976+0000) 2022-09-10T13:14:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:55.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:55.900290+0000) 2022-09-10T13:14:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:56 smithi104 ceph-mon[97044]: pgmap v16820: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:56.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:56.510217+0000) 2022-09-10T13:14:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:56 smithi110 ceph-mon[97633]: pgmap v16820: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:14:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:56.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:56.900470+0000) 2022-09-10T13:14:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:57.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:57.510427+0000) 2022-09-10T13:14:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:57.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:57.900698+0000) 2022-09-10T13:14:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:58.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:58.510613+0000) 2022-09-10T13:14:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:58.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:58.900920+0000) 2022-09-10T13:14:59.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:14:58 smithi104 ceph-mon[97044]: pgmap v16821: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:14:58 smithi110 ceph-mon[97633]: pgmap v16821: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:14:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:14:59.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:59.510818+0000) 2022-09-10T13:15:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:14:59.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:14:59.901063+0000) 2022-09-10T13:15:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:00.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:00.511008+0000) 2022-09-10T13:15:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:00 smithi104 ceph-mon[97044]: pgmap v16822: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:01.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:00.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:00.901288+0000) 2022-09-10T13:15:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:00 smithi110 ceph-mon[97633]: pgmap v16822: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:01.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:01.511238+0000) 2022-09-10T13:15:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:01.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:01.901494+0000) 2022-09-10T13:15:02.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:15:01.835+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:15:02.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:02.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:02.511546+0000) 2022-09-10T13:15:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:02 smithi104 ceph-mon[97044]: pgmap v16823: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:02.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:02.901741+0000) 2022-09-10T13:15:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:02 smithi110 ceph-mon[97633]: pgmap v16823: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:03.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:03.511809+0000) 2022-09-10T13:15:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:03.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:03.901979+0000) 2022-09-10T13:15:04.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:04.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:04.512064+0000) 2022-09-10T13:15:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:04 smithi104 ceph-mon[97044]: pgmap v16824: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:04.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:04.902261+0000) 2022-09-10T13:15:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:04 smithi110 ceph-mon[97633]: pgmap v16824: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:05.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:05.512278+0000) 2022-09-10T13:15:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:05.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:05.902494+0000) 2022-09-10T13:15:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:06 smithi104 ceph-mon[97044]: pgmap v16825: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:15:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:06.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:06.512485+0000) 2022-09-10T13:15:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:06 smithi110 ceph-mon[97633]: pgmap v16825: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:15:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:06.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:06.902684+0000) 2022-09-10T13:15:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:07.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:07.512712+0000) 2022-09-10T13:15:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:07.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:07.902860+0000) 2022-09-10T13:15:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:08.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:08.512950+0000) 2022-09-10T13:15:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:08 smithi104 ceph-mon[97044]: osdmap e1132: 12 total, 12 up, 12 in 2022-09-10T13:15:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:08 smithi104 ceph-mon[97044]: pgmap v16827: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:15:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:08 smithi110 ceph-mon[97633]: osdmap e1132: 12 total, 12 up, 12 in 2022-09-10T13:15:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:08 smithi110 ceph-mon[97633]: pgmap v16827: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:15:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:08.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:08.903073+0000) 2022-09-10T13:15:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:09.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:09.513139+0000) 2022-09-10T13:15:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:09.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:09.903392+0000) 2022-09-10T13:15:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:10.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:10.513336+0000) 2022-09-10T13:15:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:10 smithi104 ceph-mon[97044]: pgmap v16828: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:15:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:10 smithi104 ceph-mon[97044]: osdmap e1133: 12 total, 12 up, 12 in 2022-09-10T13:15:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:10.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:10.903585+0000) 2022-09-10T13:15:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:10 smithi110 ceph-mon[97633]: pgmap v16828: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:15:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:10 smithi110 ceph-mon[97633]: osdmap e1133: 12 total, 12 up, 12 in 2022-09-10T13:15:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:11.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:11.513522+0000) 2022-09-10T13:15:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:11.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:11.903821+0000) 2022-09-10T13:15:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:12.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:12.513765+0000) 2022-09-10T13:15:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:12 smithi104 ceph-mon[97044]: pgmap v16830: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:15:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:12.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:12.904015+0000) 2022-09-10T13:15:13.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:12 smithi110 ceph-mon[97633]: pgmap v16830: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:15:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:13.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:13.514038+0000) 2022-09-10T13:15:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:13.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:13.904252+0000) 2022-09-10T13:15:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:14.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:14.514287+0000) 2022-09-10T13:15:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:14 smithi104 ceph-mon[97044]: pgmap v16831: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:15:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:14.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:14.904500+0000) 2022-09-10T13:15:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:14 smithi110 ceph-mon[97633]: pgmap v16831: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:15:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:15.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:15.514495+0000) 2022-09-10T13:15:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:15.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:15.904693+0000) 2022-09-10T13:15:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:16 smithi104 ceph-mon[97044]: pgmap v16832: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T13:15:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:16.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:16.514726+0000) 2022-09-10T13:15:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:16 smithi110 ceph-mon[97633]: pgmap v16832: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T13:15:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:16.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:16.904932+0000) 2022-09-10T13:15:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:15:16.845+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:15:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:17.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:17.514877+0000) 2022-09-10T13:15:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:17.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:17.905138+0000) 2022-09-10T13:15:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:18.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:18.515157+0000) 2022-09-10T13:15:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:18 smithi104 ceph-mon[97044]: pgmap v16833: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:15:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:18.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:18.905321+0000) 2022-09-10T13:15:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:18 smithi110 ceph-mon[97633]: pgmap v16833: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:15:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:19.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:19.515305+0000) 2022-09-10T13:15:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:19.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:19.905555+0000) 2022-09-10T13:15:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:20.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:20.515509+0000) 2022-09-10T13:15:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:20 smithi104 ceph-mon[97044]: pgmap v16834: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:15:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:20.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:20.905767+0000) 2022-09-10T13:15:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:20 smithi110 ceph-mon[97633]: pgmap v16834: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:15:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:21.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:21.515740+0000) 2022-09-10T13:15:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:21.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:21.905961+0000) 2022-09-10T13:15:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:22.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:22.516011+0000) 2022-09-10T13:15:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:22 smithi104 ceph-mon[97044]: pgmap v16835: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:15:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:22.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:22.906161+0000) 2022-09-10T13:15:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:22 smithi110 ceph-mon[97633]: pgmap v16835: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:15:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:23.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:23.516249+0000) 2022-09-10T13:15:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:23.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:23.906381+0000) 2022-09-10T13:15:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:24.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:24.516514+0000) 2022-09-10T13:15:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:24 smithi104 ceph-mon[97044]: pgmap v16836: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:15:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:24.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:24.906579+0000) 2022-09-10T13:15:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:24 smithi110 ceph-mon[97633]: pgmap v16836: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:15:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:25.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:25.516767+0000) 2022-09-10T13:15:26.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:25.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:25.906793+0000) 2022-09-10T13:15:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:26 smithi104 ceph-mon[97044]: pgmap v16837: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:15:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:26.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:26.516993+0000) 2022-09-10T13:15:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:26 smithi110 ceph-mon[97633]: pgmap v16837: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:15:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:26.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:26.906977+0000) 2022-09-10T13:15:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:27.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:27.517290+0000) 2022-09-10T13:15:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:27.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:27.907173+0000) 2022-09-10T13:15:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:28.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:28.517551+0000) 2022-09-10T13:15:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:28 smithi104 ceph-mon[97044]: pgmap v16838: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:28.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:28.907392+0000) 2022-09-10T13:15:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:28 smithi110 ceph-mon[97633]: pgmap v16838: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:29.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:29.517769+0000) 2022-09-10T13:15:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:29.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:29.907608+0000) 2022-09-10T13:15:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:30.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:30.518027+0000) 2022-09-10T13:15:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:30 smithi104 ceph-mon[97044]: pgmap v16839: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:30.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:30.907787+0000) 2022-09-10T13:15:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:30 smithi110 ceph-mon[97633]: pgmap v16839: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:31.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:31.518209+0000) 2022-09-10T13:15:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:31.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:31.908003+0000) 2022-09-10T13:15:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:15:31.860+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:15:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:32.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:32.518416+0000) 2022-09-10T13:15:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:32 smithi104 ceph-mon[97044]: pgmap v16840: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:32.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:32.908273+0000) 2022-09-10T13:15:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:32 smithi110 ceph-mon[97633]: pgmap v16840: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:33.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:33.518639+0000) 2022-09-10T13:15:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:33.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:33.908485+0000) 2022-09-10T13:15:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:34.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:34.518928+0000) 2022-09-10T13:15:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:34 smithi104 ceph-mon[97044]: pgmap v16841: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:34.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:34.908757+0000) 2022-09-10T13:15:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:34 smithi110 ceph-mon[97633]: pgmap v16841: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:35.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:35.519201+0000) 2022-09-10T13:15:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:35.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:35.908936+0000) 2022-09-10T13:15:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:36 smithi104 ceph-mon[97044]: pgmap v16842: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:36.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:36.519394+0000) 2022-09-10T13:15:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:36 smithi110 ceph-mon[97633]: pgmap v16842: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:36.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:36.909183+0000) 2022-09-10T13:15:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:37.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:37.519580+0000) 2022-09-10T13:15:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:37.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:37.909400+0000) 2022-09-10T13:15:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:38.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:38.519820+0000) 2022-09-10T13:15:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:38 smithi104 ceph-mon[97044]: pgmap v16843: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:38.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:38.909606+0000) 2022-09-10T13:15:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:38 smithi110 ceph-mon[97633]: pgmap v16843: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:39.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:39.520085+0000) 2022-09-10T13:15:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:39.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:39.909828+0000) 2022-09-10T13:15:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:40.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:40.520300+0000) 2022-09-10T13:15:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:40 smithi104 ceph-mon[97044]: pgmap v16844: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:40.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:40.910030+0000) 2022-09-10T13:15:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:40 smithi110 ceph-mon[97633]: pgmap v16844: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:41.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:41.520473+0000) 2022-09-10T13:15:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:41.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:41.910257+0000) 2022-09-10T13:15:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:42.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:42.520687+0000) 2022-09-10T13:15:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:42 smithi104 ceph-mon[97044]: pgmap v16845: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:42.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:42.910496+0000) 2022-09-10T13:15:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:42 smithi110 ceph-mon[97633]: pgmap v16845: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:43.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:43.520907+0000) 2022-09-10T13:15:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:43.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:43.910682+0000) 2022-09-10T13:15:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:44.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:44.521162+0000) 2022-09-10T13:15:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:44 smithi104 ceph-mon[97044]: pgmap v16846: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:44.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:44.910932+0000) 2022-09-10T13:15:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:44 smithi110 ceph-mon[97633]: pgmap v16846: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:45.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:45.521384+0000) 2022-09-10T13:15:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:45.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:45.911158+0000) 2022-09-10T13:15:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:46 smithi104 ceph-mon[97044]: pgmap v16847: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:15:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:15:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:46.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:46.521584+0000) 2022-09-10T13:15:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:46 smithi110 ceph-mon[97633]: pgmap v16847: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:15:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:15:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:15:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:15:47.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:15:46.875+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:15:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:46.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:46.911310+0000) 2022-09-10T13:15:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:47.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:47.521780+0000) 2022-09-10T13:15:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:47.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:47.911500+0000) 2022-09-10T13:15:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:48.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:48.522014+0000) 2022-09-10T13:15:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:48 smithi104 ceph-mon[97044]: pgmap v16848: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:48.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:48.911773+0000) 2022-09-10T13:15:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:48 smithi110 ceph-mon[97633]: pgmap v16848: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:49.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:49.522265+0000) 2022-09-10T13:15:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:49.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:49.911987+0000) 2022-09-10T13:15:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:50.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:50.522500+0000) 2022-09-10T13:15:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:50 smithi104 ceph-mon[97044]: pgmap v16849: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:50.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:50.912209+0000) 2022-09-10T13:15:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:50 smithi110 ceph-mon[97633]: pgmap v16849: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:51.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:51.522712+0000) 2022-09-10T13:15:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:51.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:51.912448+0000) 2022-09-10T13:15:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:52.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:52.522901+0000) 2022-09-10T13:15:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:52 smithi104 ceph-mon[97044]: pgmap v16850: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:52.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:52.912686+0000) 2022-09-10T13:15:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:52 smithi110 ceph-mon[97633]: pgmap v16850: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:53.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:53.523176+0000) 2022-09-10T13:15:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:53.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:53.912904+0000) 2022-09-10T13:15:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:54.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:54.523411+0000) 2022-09-10T13:15:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:54 smithi104 ceph-mon[97044]: pgmap v16851: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:54.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:54.913164+0000) 2022-09-10T13:15:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:54 smithi110 ceph-mon[97633]: pgmap v16851: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:55.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:55.523634+0000) 2022-09-10T13:15:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:55.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:55.913288+0000) 2022-09-10T13:15:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:56 smithi104 ceph-mon[97044]: pgmap v16852: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:56.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:56.523835+0000) 2022-09-10T13:15:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:56 smithi110 ceph-mon[97633]: pgmap v16852: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:15:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:56.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:56.913485+0000) 2022-09-10T13:15:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:57.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:57.524038+0000) 2022-09-10T13:15:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:57.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:57.913735+0000) 2022-09-10T13:15:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:58.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:58.524267+0000) 2022-09-10T13:15:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:15:58 smithi104 ceph-mon[97044]: pgmap v16853: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:58.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:58.913974+0000) 2022-09-10T13:15:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:15:58 smithi110 ceph-mon[97633]: pgmap v16853: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:15:59.812 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:15:59.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:59.524503+0000) 2022-09-10T13:16:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:15:59.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:15:59.914166+0000) 2022-09-10T13:16:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:00.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:00.524667+0000) 2022-09-10T13:16:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:00 smithi104 ceph-mon[97044]: pgmap v16854: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:00.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:00.914344+0000) 2022-09-10T13:16:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:00 smithi110 ceph-mon[97633]: pgmap v16854: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:01.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:01.524855+0000) 2022-09-10T13:16:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:01.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:01.914601+0000) 2022-09-10T13:16:02.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:16:01.880+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:16:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:02.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:02.525128+0000) 2022-09-10T13:16:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:02 smithi104 ceph-mon[97044]: pgmap v16855: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:02.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:02.914780+0000) 2022-09-10T13:16:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:02 smithi110 ceph-mon[97633]: pgmap v16855: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:03.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:03.525354+0000) 2022-09-10T13:16:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:03.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:03.914967+0000) 2022-09-10T13:16:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:04.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:04.525629+0000) 2022-09-10T13:16:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:04 smithi104 ceph-mon[97044]: pgmap v16856: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:04.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:04.915200+0000) 2022-09-10T13:16:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:04 smithi110 ceph-mon[97633]: pgmap v16856: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:05.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:05.525857+0000) 2022-09-10T13:16:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:05.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:05.915395+0000) 2022-09-10T13:16:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:06 smithi104 ceph-mon[97044]: pgmap v16857: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:16:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:06.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:06.526085+0000) 2022-09-10T13:16:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:06 smithi110 ceph-mon[97633]: pgmap v16857: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:16:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:06.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:06.915597+0000) 2022-09-10T13:16:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:07.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:07.526369+0000) 2022-09-10T13:16:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:07.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:07.915809+0000) 2022-09-10T13:16:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:08.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:08.526593+0000) 2022-09-10T13:16:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:08 smithi110 ceph-mon[97633]: osdmap e1134: 12 total, 12 up, 12 in 2022-09-10T13:16:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:08 smithi110 ceph-mon[97633]: pgmap v16859: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:16:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:08 smithi104 ceph-mon[97044]: osdmap e1134: 12 total, 12 up, 12 in 2022-09-10T13:16:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:08 smithi104 ceph-mon[97044]: pgmap v16859: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:16:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:08.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:08.916028+0000) 2022-09-10T13:16:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:09.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:09.526827+0000) 2022-09-10T13:16:10.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:09.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:09.916234+0000) 2022-09-10T13:16:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:10.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:10.527024+0000) 2022-09-10T13:16:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:10 smithi104 ceph-mon[97044]: pgmap v16860: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:16:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:10 smithi104 ceph-mon[97044]: osdmap e1135: 12 total, 12 up, 12 in 2022-09-10T13:16:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:10.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:10.916467+0000) 2022-09-10T13:16:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:10 smithi110 ceph-mon[97633]: pgmap v16860: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:16:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:10 smithi110 ceph-mon[97633]: osdmap e1135: 12 total, 12 up, 12 in 2022-09-10T13:16:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:11.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:11.527250+0000) 2022-09-10T13:16:12.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:11.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:11.916689+0000) 2022-09-10T13:16:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:12.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:12.527483+0000) 2022-09-10T13:16:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:12 smithi104 ceph-mon[97044]: pgmap v16862: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:16:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:12.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:12.916928+0000) 2022-09-10T13:16:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:12 smithi110 ceph-mon[97633]: pgmap v16862: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:16:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:13.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:13.527690+0000) 2022-09-10T13:16:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:13.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:13.917183+0000) 2022-09-10T13:16:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:14.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:14.527903+0000) 2022-09-10T13:16:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:14 smithi104 ceph-mon[97044]: pgmap v16863: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:16:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:14.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:14.917362+0000) 2022-09-10T13:16:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:14 smithi110 ceph-mon[97633]: pgmap v16863: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:16:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:15.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:15.528174+0000) 2022-09-10T13:16:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:15.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:15.917544+0000) 2022-09-10T13:16:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:16 smithi104 ceph-mon[97044]: pgmap v16864: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T13:16:16.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:16.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:16.528361+0000) 2022-09-10T13:16:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:16 smithi110 ceph-mon[97633]: pgmap v16864: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2022-09-10T13:16:17.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:16:16.887+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:16:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:16.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:16.917759+0000) 2022-09-10T13:16:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:17.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:17.528569+0000) 2022-09-10T13:16:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:17.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:17.917928+0000) 2022-09-10T13:16:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:18.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:18.528778+0000) 2022-09-10T13:16:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:18 smithi104 ceph-mon[97044]: pgmap v16865: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:16:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:18.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:18.918190+0000) 2022-09-10T13:16:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:18 smithi110 ceph-mon[97633]: pgmap v16865: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:16:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:19.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:19.528977+0000) 2022-09-10T13:16:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:19.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:19.918435+0000) 2022-09-10T13:16:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:20.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:20.529223+0000) 2022-09-10T13:16:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:20 smithi104 ceph-mon[97044]: pgmap v16866: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:16:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:20.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:20.918752+0000) 2022-09-10T13:16:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:20 smithi110 ceph-mon[97633]: pgmap v16866: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:16:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:21.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:21.529495+0000) 2022-09-10T13:16:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:21.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:21.918981+0000) 2022-09-10T13:16:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:22.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:22.529713+0000) 2022-09-10T13:16:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:22 smithi104 ceph-mon[97044]: pgmap v16867: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:16:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:22.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:22.919252+0000) 2022-09-10T13:16:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:22 smithi110 ceph-mon[97633]: pgmap v16867: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:16:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:23.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:23.529931+0000) 2022-09-10T13:16:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:23.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:23.919473+0000) 2022-09-10T13:16:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:24.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:24.530215+0000) 2022-09-10T13:16:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:24 smithi104 ceph-mon[97044]: pgmap v16868: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:16:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:24.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:24.919687+0000) 2022-09-10T13:16:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:24 smithi110 ceph-mon[97633]: pgmap v16868: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:16:25.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:25.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:25.530390+0000) 2022-09-10T13:16:26.228 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:25.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:25.919898+0000) 2022-09-10T13:16:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:26 smithi104 ceph-mon[97044]: pgmap v16869: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:16:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:26.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:26.530597+0000) 2022-09-10T13:16:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:26 smithi110 ceph-mon[97633]: pgmap v16869: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:16:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:26.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:26.920145+0000) 2022-09-10T13:16:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:27.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:27.530772+0000) 2022-09-10T13:16:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:27.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:27.920348+0000) 2022-09-10T13:16:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:28.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:28.531008+0000) 2022-09-10T13:16:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:28 smithi104 ceph-mon[97044]: pgmap v16870: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:28.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:28.920527+0000) 2022-09-10T13:16:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:28 smithi110 ceph-mon[97633]: pgmap v16870: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:29.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:29.531282+0000) 2022-09-10T13:16:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:29.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:29.920726+0000) 2022-09-10T13:16:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:30.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:30.531535+0000) 2022-09-10T13:16:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:30 smithi104 ceph-mon[97044]: pgmap v16871: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:30.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:30.920958+0000) 2022-09-10T13:16:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:30 smithi110 ceph-mon[97633]: pgmap v16871: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:31.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:31.531811+0000) 2022-09-10T13:16:32.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:16:31.893+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:16:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:31.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:31.921143+0000) 2022-09-10T13:16:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:32.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:32.532043+0000) 2022-09-10T13:16:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:32 smithi104 ceph-mon[97044]: pgmap v16872: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:32.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:32.921381+0000) 2022-09-10T13:16:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:32 smithi110 ceph-mon[97633]: pgmap v16872: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:33.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:33.532312+0000) 2022-09-10T13:16:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:33.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:33.921561+0000) 2022-09-10T13:16:34.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:34.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:34.532530+0000) 2022-09-10T13:16:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:34 smithi104 ceph-mon[97044]: pgmap v16873: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:34.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:34.921818+0000) 2022-09-10T13:16:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:34 smithi110 ceph-mon[97633]: pgmap v16873: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:35.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:35.532793+0000) 2022-09-10T13:16:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:35.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:35.922033+0000) 2022-09-10T13:16:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:36 smithi104 ceph-mon[97044]: pgmap v16874: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:36.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:36.532989+0000) 2022-09-10T13:16:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:36 smithi110 ceph-mon[97633]: pgmap v16874: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:36.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:36.922379+0000) 2022-09-10T13:16:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:37.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:37.533197+0000) 2022-09-10T13:16:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:37.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:37.922565+0000) 2022-09-10T13:16:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:38.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:38.533371+0000) 2022-09-10T13:16:39.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:38 smithi104 ceph-mon[97044]: pgmap v16875: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:38.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:38.922798+0000) 2022-09-10T13:16:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:38 smithi110 ceph-mon[97633]: pgmap v16875: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:39.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:39.533562+0000) 2022-09-10T13:16:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:39.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:39.922987+0000) 2022-09-10T13:16:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:40.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:40.533797+0000) 2022-09-10T13:16:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:40 smithi104 ceph-mon[97044]: pgmap v16876: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:40.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:40.923248+0000) 2022-09-10T13:16:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:40 smithi110 ceph-mon[97633]: pgmap v16876: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:41.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:41.533980+0000) 2022-09-10T13:16:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:41.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:41.923450+0000) 2022-09-10T13:16:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:42.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:42.534183+0000) 2022-09-10T13:16:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:42 smithi104 ceph-mon[97044]: pgmap v16877: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:42.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:42.923669+0000) 2022-09-10T13:16:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:42 smithi110 ceph-mon[97633]: pgmap v16877: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:43.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:43.534421+0000) 2022-09-10T13:16:44.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:43.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:43.923907+0000) 2022-09-10T13:16:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:44.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:44.534645+0000) 2022-09-10T13:16:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:44 smithi104 ceph-mon[97044]: pgmap v16878: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:44.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:44.924160+0000) 2022-09-10T13:16:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:44 smithi110 ceph-mon[97633]: pgmap v16878: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:45.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:45.534906+0000) 2022-09-10T13:16:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:45.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:45.924377+0000) 2022-09-10T13:16:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:46 smithi104 ceph-mon[97044]: pgmap v16879: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:16:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:16:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:16:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:16:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:46.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:46.535110+0000) 2022-09-10T13:16:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:46 smithi110 ceph-mon[97633]: pgmap v16879: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:16:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:16:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:16:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:16:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:46.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:46.924575+0000) 2022-09-10T13:16:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:16:46.908+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:16:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:47.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:47.535328+0000) 2022-09-10T13:16:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:47.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:47.924767+0000) 2022-09-10T13:16:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:48.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:48.535515+0000) 2022-09-10T13:16:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:48 smithi104 ceph-mon[97044]: pgmap v16880: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:48.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:48.924997+0000) 2022-09-10T13:16:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:48 smithi110 ceph-mon[97633]: pgmap v16880: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:49.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:49.535789+0000) 2022-09-10T13:16:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:49.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:49.925226+0000) 2022-09-10T13:16:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:50.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:50.536026+0000) 2022-09-10T13:16:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:50 smithi104 ceph-mon[97044]: pgmap v16881: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:50.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:50.925436+0000) 2022-09-10T13:16:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:50 smithi110 ceph-mon[97633]: pgmap v16881: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:51.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:51.536267+0000) 2022-09-10T13:16:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:51.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:51.925637+0000) 2022-09-10T13:16:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:52.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:52.536528+0000) 2022-09-10T13:16:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:52 smithi104 ceph-mon[97044]: pgmap v16882: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:52.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:52.925841+0000) 2022-09-10T13:16:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:52 smithi110 ceph-mon[97633]: pgmap v16882: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:53.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:53.536682+0000) 2022-09-10T13:16:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:53.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:53.926074+0000) 2022-09-10T13:16:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:54.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:54.536899+0000) 2022-09-10T13:16:55.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:54 smithi104 ceph-mon[97044]: pgmap v16883: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:54.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:54.926352+0000) 2022-09-10T13:16:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:54 smithi110 ceph-mon[97633]: pgmap v16883: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:55.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:55.537064+0000) 2022-09-10T13:16:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:55.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:55.926558+0000) 2022-09-10T13:16:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:56 smithi104 ceph-mon[97044]: pgmap v16884: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:56.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:56.537250+0000) 2022-09-10T13:16:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:56 smithi110 ceph-mon[97633]: pgmap v16884: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:16:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:56.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:56.926808+0000) 2022-09-10T13:16:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:57.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:57.537452+0000) 2022-09-10T13:16:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:57.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:57.927050+0000) 2022-09-10T13:16:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:58.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:58.537621+0000) 2022-09-10T13:16:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:16:58 smithi104 ceph-mon[97044]: pgmap v16885: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:58.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:58.927252+0000) 2022-09-10T13:16:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:16:58 smithi110 ceph-mon[97633]: pgmap v16885: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:16:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:16:59.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:59.537788+0000) 2022-09-10T13:17:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:16:59.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:16:59.927486+0000) 2022-09-10T13:17:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:00.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:00.538063+0000) 2022-09-10T13:17:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:00 smithi104 ceph-mon[97044]: pgmap v16886: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:00.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:00.927703+0000) 2022-09-10T13:17:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:00 smithi110 ceph-mon[97633]: pgmap v16886: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:01.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:01.538347+0000) 2022-09-10T13:17:02.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:17:01.917+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:17:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:01.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:01.927947+0000) 2022-09-10T13:17:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:02.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:02.538643+0000) 2022-09-10T13:17:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:02 smithi104 ceph-mon[97044]: pgmap v16887: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:02.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:02.928171+0000) 2022-09-10T13:17:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:02 smithi110 ceph-mon[97633]: pgmap v16887: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:03.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:03.538875+0000) 2022-09-10T13:17:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:03.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:03.928365+0000) 2022-09-10T13:17:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:04.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:04.539123+0000) 2022-09-10T13:17:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:04 smithi104 ceph-mon[97044]: pgmap v16888: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:04.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:04.928580+0000) 2022-09-10T13:17:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:04 smithi110 ceph-mon[97633]: pgmap v16888: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:05.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:05.539305+0000) 2022-09-10T13:17:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:05.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:05.928744+0000) 2022-09-10T13:17:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:06 smithi104 ceph-mon[97044]: pgmap v16889: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:17:06.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:06.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:06.539540+0000) 2022-09-10T13:17:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:06 smithi110 ceph-mon[97633]: pgmap v16889: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:17:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:06.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:06.928944+0000) 2022-09-10T13:17:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:07.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:07.539739+0000) 2022-09-10T13:17:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:07.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:07.929157+0000) 2022-09-10T13:17:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:08.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:08.539994+0000) 2022-09-10T13:17:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:08 smithi110 ceph-mon[97633]: osdmap e1136: 12 total, 12 up, 12 in 2022-09-10T13:17:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:08 smithi110 ceph-mon[97633]: pgmap v16891: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:17:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:08 smithi104 ceph-mon[97044]: osdmap e1136: 12 total, 12 up, 12 in 2022-09-10T13:17:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:08 smithi104 ceph-mon[97044]: pgmap v16891: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:17:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:08.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:08.929370+0000) 2022-09-10T13:17:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:09.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:09.540244+0000) 2022-09-10T13:17:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:09.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:09.929583+0000) 2022-09-10T13:17:10.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:10.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:10.540469+0000) 2022-09-10T13:17:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:10 smithi104 ceph-mon[97044]: pgmap v16892: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:17:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:10 smithi104 ceph-mon[97044]: osdmap e1137: 12 total, 12 up, 12 in 2022-09-10T13:17:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:10.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:10.929839+0000) 2022-09-10T13:17:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:10 smithi110 ceph-mon[97633]: pgmap v16892: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:17:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:10 smithi110 ceph-mon[97633]: osdmap e1137: 12 total, 12 up, 12 in 2022-09-10T13:17:11.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:11.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:11.540689+0000) 2022-09-10T13:17:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:11.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:11.930140+0000) 2022-09-10T13:17:12.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:12.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:12.540923+0000) 2022-09-10T13:17:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:12 smithi104 ceph-mon[97044]: pgmap v16894: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:17:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:12.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:12.930367+0000) 2022-09-10T13:17:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:12 smithi110 ceph-mon[97633]: pgmap v16894: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:17:13.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:13.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:13.541187+0000) 2022-09-10T13:17:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:13.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:13.930529+0000) 2022-09-10T13:17:14.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:14.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:14.541358+0000) 2022-09-10T13:17:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:14 smithi104 ceph-mon[97044]: pgmap v16895: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:17:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:14.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:14.930772+0000) 2022-09-10T13:17:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:14 smithi110 ceph-mon[97633]: pgmap v16895: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:17:15.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:15.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:15.541604+0000) 2022-09-10T13:17:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:15.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:15.931017+0000) 2022-09-10T13:17:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:16 smithi104 ceph-mon[97044]: pgmap v16896: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T13:17:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:16.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:16.541818+0000) 2022-09-10T13:17:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:16 smithi110 ceph-mon[97633]: pgmap v16896: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T13:17:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:16.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:16.931231+0000) 2022-09-10T13:17:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:17:16.932+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:17:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:17.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:17.542029+0000) 2022-09-10T13:17:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:17.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:17.931411+0000) 2022-09-10T13:17:18.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:18.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:18.542237+0000) 2022-09-10T13:17:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:18 smithi104 ceph-mon[97044]: pgmap v16897: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:17:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:18.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:18.931661+0000) 2022-09-10T13:17:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:18 smithi110 ceph-mon[97633]: pgmap v16897: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:17:19.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:19.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:19.542462+0000) 2022-09-10T13:17:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:19.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:19.931822+0000) 2022-09-10T13:17:20.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:20.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:20.542618+0000) 2022-09-10T13:17:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:20 smithi104 ceph-mon[97044]: pgmap v16898: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:17:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:20.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:20.932052+0000) 2022-09-10T13:17:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:20 smithi110 ceph-mon[97633]: pgmap v16898: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:17:21.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:21.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:21.542798+0000) 2022-09-10T13:17:22.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:21.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:21.932209+0000) 2022-09-10T13:17:22.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:22.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:22.543030+0000) 2022-09-10T13:17:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:22 smithi104 ceph-mon[97044]: pgmap v16899: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:17:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:22.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:22.932438+0000) 2022-09-10T13:17:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:22 smithi110 ceph-mon[97633]: pgmap v16899: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:17:23.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:23.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:23.543295+0000) 2022-09-10T13:17:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:23.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:23.932685+0000) 2022-09-10T13:17:24.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:24.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:24.543563+0000) 2022-09-10T13:17:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:24 smithi104 ceph-mon[97044]: pgmap v16900: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:17:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:24.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:24.932964+0000) 2022-09-10T13:17:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:24 smithi110 ceph-mon[97633]: pgmap v16900: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:17:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:25.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:25.543739+0000) 2022-09-10T13:17:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:25.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:25.933178+0000) 2022-09-10T13:17:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:26 smithi104 ceph-mon[97044]: pgmap v16901: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:17:26.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:26.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:26.543910+0000) 2022-09-10T13:17:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:26 smithi110 ceph-mon[97633]: pgmap v16901: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:17:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:26.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:26.933393+0000) 2022-09-10T13:17:27.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:27.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:27.544174+0000) 2022-09-10T13:17:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:27.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:27.933612+0000) 2022-09-10T13:17:28.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:28.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:28.544385+0000) 2022-09-10T13:17:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:28 smithi104 ceph-mon[97044]: pgmap v16902: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:28.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:28.933859+0000) 2022-09-10T13:17:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:28 smithi110 ceph-mon[97633]: pgmap v16902: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:29.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:29.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:29.544644+0000) 2022-09-10T13:17:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:29.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:29.934067+0000) 2022-09-10T13:17:30.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:30.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:30.544921+0000) 2022-09-10T13:17:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:30 smithi104 ceph-mon[97044]: pgmap v16903: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:30.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:30.934312+0000) 2022-09-10T13:17:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:30 smithi110 ceph-mon[97633]: pgmap v16903: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:31.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:31.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:31.545195+0000) 2022-09-10T13:17:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:31.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:31.934566+0000) 2022-09-10T13:17:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:17:31.948+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:17:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:32.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:32.545392+0000) 2022-09-10T13:17:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:32 smithi104 ceph-mon[97044]: pgmap v16904: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:32.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:32.934837+0000) 2022-09-10T13:17:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:32 smithi110 ceph-mon[97633]: pgmap v16904: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:33.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:33.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:33.545573+0000) 2022-09-10T13:17:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:33.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:33.935047+0000) 2022-09-10T13:17:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:34.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:34.545754+0000) 2022-09-10T13:17:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:34 smithi104 ceph-mon[97044]: pgmap v16905: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:34.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:34.935189+0000) 2022-09-10T13:17:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:34 smithi110 ceph-mon[97633]: pgmap v16905: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:35.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:35.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:35.546004+0000) 2022-09-10T13:17:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:35.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:35.935369+0000) 2022-09-10T13:17:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:36 smithi104 ceph-mon[97044]: pgmap v16906: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:36.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:36.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:36.546253+0000) 2022-09-10T13:17:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:36 smithi110 ceph-mon[97633]: pgmap v16906: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:36.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:36.935569+0000) 2022-09-10T13:17:37.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:37.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:37.546549+0000) 2022-09-10T13:17:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:37.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:37.935761+0000) 2022-09-10T13:17:38.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:38.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:38.546805+0000) 2022-09-10T13:17:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:38 smithi104 ceph-mon[97044]: pgmap v16907: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:38.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:38.935937+0000) 2022-09-10T13:17:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:38 smithi110 ceph-mon[97633]: pgmap v16907: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:39.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:39.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:39.547025+0000) 2022-09-10T13:17:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:39.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:39.936135+0000) 2022-09-10T13:17:40.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:40.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:40.547241+0000) 2022-09-10T13:17:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:40 smithi104 ceph-mon[97044]: pgmap v16908: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:40.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:40.936353+0000) 2022-09-10T13:17:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:40 smithi110 ceph-mon[97633]: pgmap v16908: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:41.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:41.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:41.547466+0000) 2022-09-10T13:17:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:41.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:41.936588+0000) 2022-09-10T13:17:42.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:42.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:42.547694+0000) 2022-09-10T13:17:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:42 smithi104 ceph-mon[97044]: pgmap v16909: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:42.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:42.936787+0000) 2022-09-10T13:17:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:42 smithi110 ceph-mon[97633]: pgmap v16909: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:43.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:43.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:43.547906+0000) 2022-09-10T13:17:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:43.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:43.936969+0000) 2022-09-10T13:17:44.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:44.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:44.548133+0000) 2022-09-10T13:17:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:44 smithi104 ceph-mon[97044]: pgmap v16910: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:44.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:44.937195+0000) 2022-09-10T13:17:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:44 smithi110 ceph-mon[97633]: pgmap v16910: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:45.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:45.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:45.548329+0000) 2022-09-10T13:17:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:45.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:45.937377+0000) 2022-09-10T13:17:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:46 smithi104 ceph-mon[97044]: pgmap v16911: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:17:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:17:46.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:46.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:46.548511+0000) 2022-09-10T13:17:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:46 smithi110 ceph-mon[97633]: pgmap v16911: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:17:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:17:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:17:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:17:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:46.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:46.937594+0000) 2022-09-10T13:17:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:17:46.951+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:17:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:47.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:47.548711+0000) 2022-09-10T13:17:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:47.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:47.937824+0000) 2022-09-10T13:17:48.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:48.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:48.548923+0000) 2022-09-10T13:17:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:48 smithi104 ceph-mon[97044]: pgmap v16912: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:48.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:48.937970+0000) 2022-09-10T13:17:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:48 smithi110 ceph-mon[97633]: pgmap v16912: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:49.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:49.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:49.549137+0000) 2022-09-10T13:17:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:49.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:49.938230+0000) 2022-09-10T13:17:50.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:50.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:50.549315+0000) 2022-09-10T13:17:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:50 smithi104 ceph-mon[97044]: pgmap v16913: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:50.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:50.938435+0000) 2022-09-10T13:17:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:50 smithi110 ceph-mon[97633]: pgmap v16913: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:51.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:51.549508+0000) 2022-09-10T13:17:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:51.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:51.938630+0000) 2022-09-10T13:17:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:52.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:52.549680+0000) 2022-09-10T13:17:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:52 smithi104 ceph-mon[97044]: pgmap v16914: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:52.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:52.938858+0000) 2022-09-10T13:17:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:52 smithi110 ceph-mon[97633]: pgmap v16914: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:53.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:53.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:53.549931+0000) 2022-09-10T13:17:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:53.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:53.939135+0000) 2022-09-10T13:17:54.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:54.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:54.550129+0000) 2022-09-10T13:17:55.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:55 smithi104 ceph-mon[97044]: pgmap v16915: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:54.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:54.939363+0000) 2022-09-10T13:17:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:55 smithi110 ceph-mon[97633]: pgmap v16915: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:17:55.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:55.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:55.550353+0000) 2022-09-10T13:17:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:55.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:55.939525+0000) 2022-09-10T13:17:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:56 smithi104 ceph-mon[97044]: pgmap v16916: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:56.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:56.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:56.550564+0000) 2022-09-10T13:17:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:56 smithi110 ceph-mon[97633]: pgmap v16916: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:17:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:56.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:56.939683+0000) 2022-09-10T13:17:57.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:57.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:57.550731+0000) 2022-09-10T13:17:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:57.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:57.939881+0000) 2022-09-10T13:17:58.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:58.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:58.550895+0000) 2022-09-10T13:17:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:17:58 smithi104 ceph-mon[97044]: pgmap v16917: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:17:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:58.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:58.940149+0000) 2022-09-10T13:17:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:17:58 smithi110 ceph-mon[97633]: pgmap v16917: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:17:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:17:59.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:59.551157+0000) 2022-09-10T13:18:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:17:59.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:17:59.940364+0000) 2022-09-10T13:18:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:00.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:00.551417+0000) 2022-09-10T13:18:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:00 smithi104 ceph-mon[97044]: pgmap v16918: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:00.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:00.940450+0000) 2022-09-10T13:18:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:00 smithi110 ceph-mon[97633]: pgmap v16918: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:01.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:01.551646+0000) 2022-09-10T13:18:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:01.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:01.940609+0000) 2022-09-10T13:18:02.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:18:01.966+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:18:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:02.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:02.551887+0000) 2022-09-10T13:18:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:03 smithi104 ceph-mon[97044]: pgmap v16919: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:18:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:02.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:02.940817+0000) 2022-09-10T13:18:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:03 smithi110 ceph-mon[97633]: pgmap v16919: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:18:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:03.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:03.552091+0000) 2022-09-10T13:18:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:03.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:03.941074+0000) 2022-09-10T13:18:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:04.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:04.552344+0000) 2022-09-10T13:18:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:05 smithi104 ceph-mon[97044]: pgmap v16920: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:18:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:04.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:04.941359+0000) 2022-09-10T13:18:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:05 smithi110 ceph-mon[97633]: pgmap v16920: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:18:05.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:05.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:05.552536+0000) 2022-09-10T13:18:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:05.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:05.941597+0000) 2022-09-10T13:18:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:06 smithi104 ceph-mon[97044]: pgmap v16921: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:18:06.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:06.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:06.552766+0000) 2022-09-10T13:18:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:06 smithi110 ceph-mon[97633]: pgmap v16921: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:18:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:06.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:06.941869+0000) 2022-09-10T13:18:07.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:07.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:07.552938+0000) 2022-09-10T13:18:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:07.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:07.942101+0000) 2022-09-10T13:18:08.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:08.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:08.553116+0000) 2022-09-10T13:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:08 smithi110 ceph-mon[97633]: osdmap e1138: 12 total, 12 up, 12 in 2022-09-10T13:18:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:08 smithi110 ceph-mon[97633]: pgmap v16923: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:18:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:08 smithi104 ceph-mon[97044]: osdmap e1138: 12 total, 12 up, 12 in 2022-09-10T13:18:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:08 smithi104 ceph-mon[97044]: pgmap v16923: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:18:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:08.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:08.942264+0000) 2022-09-10T13:18:09.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:09.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:09.553264+0000) 2022-09-10T13:18:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:09.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:09.942483+0000) 2022-09-10T13:18:10.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:10.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:10.553441+0000) 2022-09-10T13:18:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:10 smithi104 ceph-mon[97044]: pgmap v16924: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:18:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:11 smithi104 ceph-mon[97044]: osdmap e1139: 12 total, 12 up, 12 in 2022-09-10T13:18:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:10.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:10.942722+0000) 2022-09-10T13:18:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:10 smithi110 ceph-mon[97633]: pgmap v16924: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:18:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:10 smithi110 ceph-mon[97633]: osdmap e1139: 12 total, 12 up, 12 in 2022-09-10T13:18:11.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:11.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:11.553661+0000) 2022-09-10T13:18:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:11.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:11.942938+0000) 2022-09-10T13:18:12.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:12.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:12.553802+0000) 2022-09-10T13:18:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:13 smithi104 ceph-mon[97044]: pgmap v16926: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:18:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:12.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:12.943218+0000) 2022-09-10T13:18:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:13 smithi110 ceph-mon[97633]: pgmap v16926: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:18:13.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:13.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:13.553975+0000) 2022-09-10T13:18:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:13.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:13.943455+0000) 2022-09-10T13:18:14.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:14.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:14.554208+0000) 2022-09-10T13:18:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:15 smithi104 ceph-mon[97044]: pgmap v16927: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:18:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:14.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:14.943684+0000) 2022-09-10T13:18:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:15 smithi110 ceph-mon[97633]: pgmap v16927: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:18:15.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:15.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:15.554435+0000) 2022-09-10T13:18:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:15.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:15.943901+0000) 2022-09-10T13:18:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:16 smithi104 ceph-mon[97044]: pgmap v16928: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T13:18:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:16 smithi110 ceph-mon[97633]: pgmap v16928: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T13:18:16.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:16.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:16.554643+0000) 2022-09-10T13:18:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:16.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:16.944208+0000) 2022-09-10T13:18:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:18:16.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:18:17.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:17.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:17.554856+0000) 2022-09-10T13:18:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:17.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:17.944434+0000) 2022-09-10T13:18:18.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:18.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:18.555061+0000) 2022-09-10T13:18:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:19 smithi104 ceph-mon[97044]: pgmap v16929: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:18:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:18.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:18.944611+0000) 2022-09-10T13:18:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:19 smithi110 ceph-mon[97633]: pgmap v16929: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:18:19.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:19.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:19.555301+0000) 2022-09-10T13:18:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:19.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:19.944888+0000) 2022-09-10T13:18:20.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:20.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:20.555537+0000) 2022-09-10T13:18:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:21 smithi104 ceph-mon[97044]: pgmap v16930: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:18:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:20.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:20.945166+0000) 2022-09-10T13:18:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:21 smithi110 ceph-mon[97633]: pgmap v16930: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:18:21.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:21.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:21.555742+0000) 2022-09-10T13:18:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:21.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:21.945375+0000) 2022-09-10T13:18:22.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:22.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:22.555934+0000) 2022-09-10T13:18:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:23 smithi104 ceph-mon[97044]: pgmap v16931: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:18:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:22.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:22.945602+0000) 2022-09-10T13:18:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:23 smithi110 ceph-mon[97633]: pgmap v16931: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:18:23.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:23.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:23.556132+0000) 2022-09-10T13:18:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:23.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:23.945818+0000) 2022-09-10T13:18:24.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:24.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:24.556268+0000) 2022-09-10T13:18:25.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:25 smithi104 ceph-mon[97044]: pgmap v16932: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:18:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:24.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:24.946025+0000) 2022-09-10T13:18:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:25 smithi110 ceph-mon[97633]: pgmap v16932: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:18:25.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:25.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:25.556475+0000) 2022-09-10T13:18:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:25.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:25.946186+0000) 2022-09-10T13:18:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:26 smithi104 ceph-mon[97044]: pgmap v16933: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:18:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:26 smithi110 ceph-mon[97633]: pgmap v16933: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:18:26.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:26.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:26.556705+0000) 2022-09-10T13:18:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:26.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:26.946354+0000) 2022-09-10T13:18:27.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:27.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:27.556868+0000) 2022-09-10T13:18:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:27.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:27.946570+0000) 2022-09-10T13:18:28.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:28.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:28.557073+0000) 2022-09-10T13:18:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:29 smithi104 ceph-mon[97044]: pgmap v16934: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:28.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:28.946826+0000) 2022-09-10T13:18:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:29 smithi110 ceph-mon[97633]: pgmap v16934: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:29.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:29.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:29.557303+0000) 2022-09-10T13:18:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:29.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:29.947084+0000) 2022-09-10T13:18:30.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:30.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:30.557435+0000) 2022-09-10T13:18:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:31 smithi104 ceph-mon[97044]: pgmap v16935: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:30.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:30.947292+0000) 2022-09-10T13:18:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:31 smithi110 ceph-mon[97633]: pgmap v16935: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:31.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:31.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:31.557575+0000) 2022-09-10T13:18:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:18:31.996+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:18:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:31.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:31.947455+0000) 2022-09-10T13:18:32.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:32.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:32.557791+0000) 2022-09-10T13:18:33.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:33 smithi104 ceph-mon[97044]: pgmap v16936: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:32.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:32.947709+0000) 2022-09-10T13:18:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:33 smithi110 ceph-mon[97633]: pgmap v16936: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:33.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:33.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:33.558000+0000) 2022-09-10T13:18:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:33.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:33.947952+0000) 2022-09-10T13:18:34.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:34.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:34.558213+0000) 2022-09-10T13:18:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:35 smithi104 ceph-mon[97044]: pgmap v16937: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:34.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:34.948216+0000) 2022-09-10T13:18:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:35 smithi110 ceph-mon[97633]: pgmap v16937: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:35.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:35.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:35.558394+0000) 2022-09-10T13:18:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:35.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:35.948410+0000) 2022-09-10T13:18:36.765 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:36 smithi104 ceph-mon[97044]: pgmap v16938: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:36 smithi110 ceph-mon[97633]: pgmap v16938: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:36.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:36.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:36.558523+0000) 2022-09-10T13:18:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:36.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:36.948663+0000) 2022-09-10T13:18:37.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:37.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:37.558669+0000) 2022-09-10T13:18:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:37.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:37.948809+0000) 2022-09-10T13:18:38.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:38.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:38.558849+0000) 2022-09-10T13:18:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:39 smithi104 ceph-mon[97044]: pgmap v16939: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:38.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:38.949034+0000) 2022-09-10T13:18:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:39 smithi110 ceph-mon[97633]: pgmap v16939: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:39.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:39.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:39.559000+0000) 2022-09-10T13:18:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:39.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:39.949311+0000) 2022-09-10T13:18:40.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:40.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:40.559192+0000) 2022-09-10T13:18:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:41 smithi104 ceph-mon[97044]: pgmap v16940: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:41.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:40.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:40.949510+0000) 2022-09-10T13:18:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:41 smithi110 ceph-mon[97633]: pgmap v16940: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:41.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:41.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:41.559360+0000) 2022-09-10T13:18:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:41.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:41.949712+0000) 2022-09-10T13:18:42.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:42.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:42.559556+0000) 2022-09-10T13:18:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:43 smithi104 ceph-mon[97044]: pgmap v16941: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:42.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:42.949940+0000) 2022-09-10T13:18:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:43 smithi110 ceph-mon[97633]: pgmap v16941: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:43.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:43.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:43.559765+0000) 2022-09-10T13:18:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:43.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:43.950158+0000) 2022-09-10T13:18:44.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:44.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:44.560020+0000) 2022-09-10T13:18:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:45 smithi104 ceph-mon[97044]: pgmap v16942: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:44.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:44.950343+0000) 2022-09-10T13:18:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:45 smithi110 ceph-mon[97633]: pgmap v16942: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:45.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:45.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:45.560247+0000) 2022-09-10T13:18:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:45.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:45.950507+0000) 2022-09-10T13:18:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:46 smithi104 ceph-mon[97044]: pgmap v16943: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:18:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:18:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:18:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:18:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:46 smithi110 ceph-mon[97633]: pgmap v16943: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:18:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:18:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:18:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:18:46.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:46.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:46.560469+0000) 2022-09-10T13:18:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:46.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:46.950741+0000) 2022-09-10T13:18:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:18:47.012+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:18:47.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:47.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:47.560696+0000) 2022-09-10T13:18:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:47.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:47.950949+0000) 2022-09-10T13:18:48.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:48.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:48.560934+0000) 2022-09-10T13:18:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:49 smithi104 ceph-mon[97044]: pgmap v16944: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:48.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:48.951200+0000) 2022-09-10T13:18:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:49 smithi110 ceph-mon[97633]: pgmap v16944: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:49.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:49.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:49.561194+0000) 2022-09-10T13:18:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:49.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:49.951447+0000) 2022-09-10T13:18:50.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:50.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:50.561411+0000) 2022-09-10T13:18:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:51 smithi104 ceph-mon[97044]: pgmap v16945: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:50.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:50.951581+0000) 2022-09-10T13:18:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:51 smithi110 ceph-mon[97633]: pgmap v16945: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:51.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:51.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:51.561659+0000) 2022-09-10T13:18:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:51.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:51.951795+0000) 2022-09-10T13:18:52.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:52.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:52.561903+0000) 2022-09-10T13:18:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:53 smithi104 ceph-mon[97044]: pgmap v16946: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:52.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:52.952034+0000) 2022-09-10T13:18:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:53 smithi110 ceph-mon[97633]: pgmap v16946: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:53.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:53.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:53.562187+0000) 2022-09-10T13:18:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:53.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:53.952257+0000) 2022-09-10T13:18:54.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:54.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:54.562402+0000) 2022-09-10T13:18:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:55 smithi104 ceph-mon[97044]: pgmap v16947: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:54.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:54.952473+0000) 2022-09-10T13:18:55.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:55 smithi110 ceph-mon[97633]: pgmap v16947: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:55.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:55.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:55.562625+0000) 2022-09-10T13:18:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:55.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:55.952703+0000) 2022-09-10T13:18:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:56 smithi104 ceph-mon[97044]: pgmap v16948: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:56 smithi110 ceph-mon[97633]: pgmap v16948: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:18:56.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:56.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:56.562836+0000) 2022-09-10T13:18:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:56.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:56.952955+0000) 2022-09-10T13:18:57.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:57.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:57.563087+0000) 2022-09-10T13:18:58.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:57.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:57.953187+0000) 2022-09-10T13:18:58.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:58.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:58.563281+0000) 2022-09-10T13:18:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:18:59 smithi104 ceph-mon[97044]: pgmap v16949: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:58.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:58.953427+0000) 2022-09-10T13:18:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:18:59 smithi110 ceph-mon[97633]: pgmap v16949: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:18:59.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:18:59.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:59.563461+0000) 2022-09-10T13:19:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:18:59.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:18:59.953707+0000) 2022-09-10T13:19:00.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:00.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:00.563633+0000) 2022-09-10T13:19:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:01 smithi104 ceph-mon[97044]: pgmap v16950: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:00.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:00.953910+0000) 2022-09-10T13:19:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:01 smithi110 ceph-mon[97633]: pgmap v16950: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:01.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:01.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:01.563810+0000) 2022-09-10T13:19:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:01.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:01.954075+0000) 2022-09-10T13:19:02.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:19:02.018+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:19:02.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:02.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:02.564000+0000) 2022-09-10T13:19:03.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:03 smithi104 ceph-mon[97044]: pgmap v16951: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:02.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:02.954332+0000) 2022-09-10T13:19:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:03 smithi110 ceph-mon[97633]: pgmap v16951: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:03.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:03.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:03.564218+0000) 2022-09-10T13:19:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:03.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:03.954502+0000) 2022-09-10T13:19:04.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:04.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:04.564405+0000) 2022-09-10T13:19:05.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:05 smithi104 ceph-mon[97044]: pgmap v16952: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:04.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:04.954737+0000) 2022-09-10T13:19:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:05 smithi110 ceph-mon[97633]: pgmap v16952: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:05.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:05.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:05.564593+0000) 2022-09-10T13:19:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:05.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:05.954930+0000) 2022-09-10T13:19:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:06 smithi104 ceph-mon[97044]: pgmap v16953: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:19:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:06 smithi110 ceph-mon[97633]: pgmap v16953: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:19:06.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:06.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:06.564764+0000) 2022-09-10T13:19:07.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:06.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:06.955164+0000) 2022-09-10T13:19:07.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:07.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:07.564964+0000) 2022-09-10T13:19:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:07.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:07.955295+0000) 2022-09-10T13:19:08.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:08.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:08.565149+0000) 2022-09-10T13:19:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:08 smithi110 ceph-mon[97633]: osdmap e1140: 12 total, 12 up, 12 in 2022-09-10T13:19:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:08 smithi110 ceph-mon[97633]: pgmap v16955: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:19:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:08 smithi104 ceph-mon[97044]: osdmap e1140: 12 total, 12 up, 12 in 2022-09-10T13:19:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:08 smithi104 ceph-mon[97044]: pgmap v16955: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:19:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:08.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:08.955541+0000) 2022-09-10T13:19:09.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:09.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:09.565299+0000) 2022-09-10T13:19:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:09.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:09.955756+0000) 2022-09-10T13:19:10.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:10.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:10.565458+0000) 2022-09-10T13:19:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:11 smithi104 ceph-mon[97044]: pgmap v16956: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:19:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:11 smithi104 ceph-mon[97044]: osdmap e1141: 12 total, 12 up, 12 in 2022-09-10T13:19:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:10.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:10.956001+0000) 2022-09-10T13:19:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:11 smithi110 ceph-mon[97633]: pgmap v16956: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:19:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:11 smithi110 ceph-mon[97633]: osdmap e1141: 12 total, 12 up, 12 in 2022-09-10T13:19:11.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:11.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:11.565632+0000) 2022-09-10T13:19:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:11.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:11.956214+0000) 2022-09-10T13:19:12.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:12.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:12.565803+0000) 2022-09-10T13:19:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:13 smithi104 ceph-mon[97044]: pgmap v16958: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:19:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:12.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:12.956366+0000) 2022-09-10T13:19:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:13 smithi110 ceph-mon[97633]: pgmap v16958: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:19:13.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:13.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:13.566052+0000) 2022-09-10T13:19:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:13.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:13.956577+0000) 2022-09-10T13:19:14.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:14.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:14.566294+0000) 2022-09-10T13:19:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:15 smithi104 ceph-mon[97044]: pgmap v16959: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:19:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:14.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:14.956808+0000) 2022-09-10T13:19:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:15 smithi110 ceph-mon[97633]: pgmap v16959: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:19:15.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:15.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:15.566446+0000) 2022-09-10T13:19:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:15.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:15.957089+0000) 2022-09-10T13:19:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:16 smithi104 ceph-mon[97044]: pgmap v16960: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T13:19:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:16 smithi110 ceph-mon[97633]: pgmap v16960: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T13:19:16.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:16.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:16.566633+0000) 2022-09-10T13:19:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:16.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:16.957403+0000) 2022-09-10T13:19:17.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:19:17.023+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:19:17.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:17.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:17.566817+0000) 2022-09-10T13:19:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:17.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:17.957566+0000) 2022-09-10T13:19:18.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:18.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:18.567066+0000) 2022-09-10T13:19:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:19 smithi104 ceph-mon[97044]: pgmap v16961: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:19:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:18.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:18.957744+0000) 2022-09-10T13:19:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:19 smithi110 ceph-mon[97633]: pgmap v16961: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:19:19.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:19.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:19.567275+0000) 2022-09-10T13:19:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:19.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:19.957933+0000) 2022-09-10T13:19:20.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:20.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:20.567486+0000) 2022-09-10T13:19:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:21 smithi104 ceph-mon[97044]: pgmap v16962: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:19:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:20.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:20.958193+0000) 2022-09-10T13:19:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:21 smithi110 ceph-mon[97633]: pgmap v16962: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:19:21.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:21.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:21.567690+0000) 2022-09-10T13:19:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:21.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:21.958417+0000) 2022-09-10T13:19:22.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:22.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:22.567918+0000) 2022-09-10T13:19:23.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:23 smithi104 ceph-mon[97044]: pgmap v16963: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:19:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:22.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:22.958630+0000) 2022-09-10T13:19:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:23 smithi110 ceph-mon[97633]: pgmap v16963: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:19:23.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:23.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:23.568199+0000) 2022-09-10T13:19:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:23.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:23.958890+0000) 2022-09-10T13:19:24.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:24.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:24.568470+0000) 2022-09-10T13:19:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:25 smithi104 ceph-mon[97044]: pgmap v16964: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:19:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:24.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:24.959158+0000) 2022-09-10T13:19:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:25 smithi110 ceph-mon[97633]: pgmap v16964: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:19:25.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:25.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:25.568692+0000) 2022-09-10T13:19:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:25.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:25.959348+0000) 2022-09-10T13:19:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:26 smithi104 ceph-mon[97044]: pgmap v16965: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:19:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:26 smithi110 ceph-mon[97633]: pgmap v16965: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:19:26.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:26.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:26.568939+0000) 2022-09-10T13:19:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:26.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:26.959566+0000) 2022-09-10T13:19:27.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:27.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:27.569147+0000) 2022-09-10T13:19:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:27.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:27.959779+0000) 2022-09-10T13:19:28.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:28.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:28.569285+0000) 2022-09-10T13:19:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:29 smithi104 ceph-mon[97044]: pgmap v16966: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:28.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:28.959948+0000) 2022-09-10T13:19:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:29 smithi110 ceph-mon[97633]: pgmap v16966: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:29.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:29.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:29.569530+0000) 2022-09-10T13:19:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:29.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:29.960209+0000) 2022-09-10T13:19:30.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:30.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:30.569752+0000) 2022-09-10T13:19:31.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:31 smithi104 ceph-mon[97044]: pgmap v16967: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:30.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:30.960442+0000) 2022-09-10T13:19:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:31 smithi110 ceph-mon[97633]: pgmap v16967: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:31.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:31.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:31.569967+0000) 2022-09-10T13:19:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:31.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:31.960674+0000) 2022-09-10T13:19:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:19:32.027+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:19:32.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:32.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:32.570218+0000) 2022-09-10T13:19:33.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:33 smithi104 ceph-mon[97044]: pgmap v16968: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:32.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:32.960862+0000) 2022-09-10T13:19:33.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:33 smithi110 ceph-mon[97633]: pgmap v16968: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:33.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:33.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:33.570449+0000) 2022-09-10T13:19:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:33.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:33.961034+0000) 2022-09-10T13:19:34.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:34.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:34.570698+0000) 2022-09-10T13:19:35.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:35 smithi104 ceph-mon[97044]: pgmap v16969: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:34.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:34.961206+0000) 2022-09-10T13:19:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:35 smithi110 ceph-mon[97633]: pgmap v16969: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:35.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:35.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:35.570935+0000) 2022-09-10T13:19:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:35.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:35.961400+0000) 2022-09-10T13:19:36.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:36 smithi104 ceph-mon[97044]: pgmap v16970: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:36.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:36 smithi110 ceph-mon[97633]: pgmap v16970: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:36.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:36.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:36.571195+0000) 2022-09-10T13:19:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:36.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:36.961653+0000) 2022-09-10T13:19:37.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:37.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:37.571428+0000) 2022-09-10T13:19:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:37.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:37.961905+0000) 2022-09-10T13:19:38.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:38.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:38.571598+0000) 2022-09-10T13:19:39.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:39 smithi104 ceph-mon[97044]: pgmap v16971: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:38.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:38.962143+0000) 2022-09-10T13:19:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:39 smithi110 ceph-mon[97633]: pgmap v16971: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:39.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:39.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:39.571819+0000) 2022-09-10T13:19:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:39.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:39.962334+0000) 2022-09-10T13:19:40.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:40.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:40.572045+0000) 2022-09-10T13:19:41.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:41 smithi104 ceph-mon[97044]: pgmap v16972: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:40.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:40.962551+0000) 2022-09-10T13:19:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:41 smithi110 ceph-mon[97633]: pgmap v16972: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:41.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:41.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:41.572283+0000) 2022-09-10T13:19:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:41.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:41.962690+0000) 2022-09-10T13:19:42.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:42.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:42.572443+0000) 2022-09-10T13:19:43.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:43 smithi104 ceph-mon[97044]: pgmap v16973: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:42.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:42.962929+0000) 2022-09-10T13:19:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:43 smithi110 ceph-mon[97633]: pgmap v16973: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:43.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:43.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:43.572672+0000) 2022-09-10T13:19:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:43.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:43.963196+0000) 2022-09-10T13:19:44.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:44.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:44.572946+0000) 2022-09-10T13:19:45.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:45 smithi104 ceph-mon[97044]: pgmap v16974: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:44.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:44.963431+0000) 2022-09-10T13:19:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:45 smithi110 ceph-mon[97633]: pgmap v16974: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:45.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:45.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:45.573176+0000) 2022-09-10T13:19:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:45.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:45.963661+0000) 2022-09-10T13:19:46.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:46 smithi104 ceph-mon[97044]: pgmap v16975: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:19:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:19:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:19:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:19:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:46 smithi110 ceph-mon[97633]: pgmap v16975: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:19:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:19:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:19:46.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:19:46.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:46.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:46.573349+0000) 2022-09-10T13:19:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:46.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:46.963932+0000) 2022-09-10T13:19:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:19:47.030+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:19:47.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:47.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:47.573534+0000) 2022-09-10T13:19:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:47.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:47.964138+0000) 2022-09-10T13:19:48.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:48.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:48.573746+0000) 2022-09-10T13:19:49.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:49 smithi104 ceph-mon[97044]: pgmap v16976: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:48.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:48.964317+0000) 2022-09-10T13:19:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:49 smithi110 ceph-mon[97633]: pgmap v16976: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:49.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:49.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:49.573901+0000) 2022-09-10T13:19:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:49.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:49.964554+0000) 2022-09-10T13:19:50.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:50.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:50.574081+0000) 2022-09-10T13:19:51.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:51 smithi104 ceph-mon[97044]: pgmap v16977: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:50.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:50.964805+0000) 2022-09-10T13:19:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:51 smithi110 ceph-mon[97633]: pgmap v16977: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:51.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:51.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:51.574300+0000) 2022-09-10T13:19:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:51.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:51.965037+0000) 2022-09-10T13:19:52.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:52.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:52.574468+0000) 2022-09-10T13:19:53.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:53 smithi104 ceph-mon[97044]: pgmap v16978: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:52.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:52.965327+0000) 2022-09-10T13:19:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:53 smithi110 ceph-mon[97633]: pgmap v16978: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:53.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:53.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:53.574690+0000) 2022-09-10T13:19:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:53.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:53.965543+0000) 2022-09-10T13:19:54.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:54.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:54.574915+0000) 2022-09-10T13:19:55.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:55 smithi104 ceph-mon[97044]: pgmap v16979: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:54.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:54.965787+0000) 2022-09-10T13:19:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:55 smithi110 ceph-mon[97633]: pgmap v16979: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:55.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:55.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:55.575180+0000) 2022-09-10T13:19:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:55.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:55.966004+0000) 2022-09-10T13:19:56.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:56 smithi104 ceph-mon[97044]: pgmap v16980: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:56 smithi110 ceph-mon[97633]: pgmap v16980: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:19:56.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:56.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:56.575407+0000) 2022-09-10T13:19:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:56.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:56.966244+0000) 2022-09-10T13:19:57.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:57.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:57.575640+0000) 2022-09-10T13:19:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:57.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:57.966397+0000) 2022-09-10T13:19:58.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:58.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:58.575907+0000) 2022-09-10T13:19:59.219 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:58.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:58.966625+0000) 2022-09-10T13:19:59.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:19:59 smithi104 ceph-mon[97044]: pgmap v16981: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:19:59 smithi110 ceph-mon[97633]: pgmap v16981: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:19:59.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:19:59.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:59.576193+0000) 2022-09-10T13:20:00.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:20:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:19:59.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:19:59.966866+0000) 2022-09-10T13:20:00.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:20:00.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:00.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:00.576361+0000) 2022-09-10T13:20:01.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:01 smithi104 ceph-mon[97044]: pgmap v16982: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:00.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:00.967055+0000) 2022-09-10T13:20:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:01 smithi110 ceph-mon[97633]: pgmap v16982: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:01.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:01.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:01.576606+0000) 2022-09-10T13:20:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:01.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:01.967277+0000) 2022-09-10T13:20:02.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:20:02.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:20:02.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:02.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:02.576870+0000) 2022-09-10T13:20:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:03 smithi104 ceph-mon[97044]: pgmap v16983: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:02.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:02.967523+0000) 2022-09-10T13:20:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:03 smithi110 ceph-mon[97633]: pgmap v16983: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:03.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:03.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:03.577091+0000) 2022-09-10T13:20:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:03.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:03.967690+0000) 2022-09-10T13:20:04.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:04.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:04.577274+0000) 2022-09-10T13:20:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:04.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:04.967939+0000) 2022-09-10T13:20:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:05 smithi110 ceph-mon[97633]: pgmap v16984: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:05 smithi104 ceph-mon[97044]: pgmap v16984: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:05.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:05.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:05.577515+0000) 2022-09-10T13:20:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:05.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:05.968176+0000) 2022-09-10T13:20:06.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:06 smithi104 ceph-mon[97044]: pgmap v16985: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:20:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:06 smithi110 ceph-mon[97633]: pgmap v16985: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:20:06.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:06.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:06.577776+0000) 2022-09-10T13:20:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:06.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:06.968419+0000) 2022-09-10T13:20:07.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:07.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:07.578054+0000) 2022-09-10T13:20:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:07.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:07.968618+0000) 2022-09-10T13:20:08.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:08.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:08.578377+0000) 2022-09-10T13:20:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:08 smithi110 ceph-mon[97633]: osdmap e1142: 12 total, 12 up, 12 in 2022-09-10T13:20:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:08 smithi110 ceph-mon[97633]: pgmap v16987: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:20:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:08 smithi104 ceph-mon[97044]: osdmap e1142: 12 total, 12 up, 12 in 2022-09-10T13:20:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:08 smithi104 ceph-mon[97044]: pgmap v16987: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:20:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:08.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:08.968896+0000) 2022-09-10T13:20:09.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:09.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:09.578599+0000) 2022-09-10T13:20:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:09.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:09.969181+0000) 2022-09-10T13:20:10.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:10.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:10.578867+0000) 2022-09-10T13:20:11.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:11 smithi104 ceph-mon[97044]: pgmap v16988: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:20:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:11 smithi104 ceph-mon[97044]: osdmap e1143: 12 total, 12 up, 12 in 2022-09-10T13:20:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:10.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:10.969421+0000) 2022-09-10T13:20:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:11 smithi110 ceph-mon[97633]: pgmap v16988: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:20:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:11 smithi110 ceph-mon[97633]: osdmap e1143: 12 total, 12 up, 12 in 2022-09-10T13:20:11.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:11.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:11.579159+0000) 2022-09-10T13:20:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:11.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:11.969640+0000) 2022-09-10T13:20:12.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:12.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:12.579416+0000) 2022-09-10T13:20:13.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:13 smithi104 ceph-mon[97044]: pgmap v16990: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:20:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:12.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:12.969867+0000) 2022-09-10T13:20:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:13 smithi110 ceph-mon[97633]: pgmap v16990: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:20:13.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:13.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:13.579659+0000) 2022-09-10T13:20:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:13.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:13.970073+0000) 2022-09-10T13:20:14.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:14.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:14.579869+0000) 2022-09-10T13:20:15.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:15 smithi104 ceph-mon[97044]: pgmap v16991: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:20:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:14.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:14.970249+0000) 2022-09-10T13:20:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:15 smithi110 ceph-mon[97633]: pgmap v16991: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:20:15.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:15.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:15.580137+0000) 2022-09-10T13:20:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:15.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:15.970536+0000) 2022-09-10T13:20:16.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:16 smithi104 ceph-mon[97044]: pgmap v16992: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T13:20:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:16 smithi110 ceph-mon[97633]: pgmap v16992: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T13:20:16.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:16.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:16.580364+0000) 2022-09-10T13:20:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:16.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:16.970733+0000) 2022-09-10T13:20:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:20:17.051+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:20:17.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:17.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:17.580608+0000) 2022-09-10T13:20:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:17.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:17.970917+0000) 2022-09-10T13:20:18.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:18.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:18.580884+0000) 2022-09-10T13:20:19.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:19 smithi104 ceph-mon[97044]: pgmap v16993: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:20:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:18.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:18.971168+0000) 2022-09-10T13:20:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:19 smithi110 ceph-mon[97633]: pgmap v16993: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:20:19.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:19.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:19.581172+0000) 2022-09-10T13:20:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:19.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:19.971376+0000) 2022-09-10T13:20:20.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:20.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:20.581364+0000) 2022-09-10T13:20:21.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:21 smithi104 ceph-mon[97044]: pgmap v16994: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:20:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:20.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:20.971604+0000) 2022-09-10T13:20:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:21 smithi110 ceph-mon[97633]: pgmap v16994: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:20:21.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:21.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:21.581549+0000) 2022-09-10T13:20:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:21.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:21.971809+0000) 2022-09-10T13:20:22.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:22.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:22.581739+0000) 2022-09-10T13:20:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:22.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:22.972057+0000) 2022-09-10T13:20:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:23 smithi110 ceph-mon[97633]: pgmap v16995: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:20:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:23 smithi104 ceph-mon[97044]: pgmap v16995: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:20:23.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:23.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:23.581941+0000) 2022-09-10T13:20:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:23.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:23.972274+0000) 2022-09-10T13:20:24.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:24.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:24.582127+0000) 2022-09-10T13:20:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:24.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:24.972536+0000) 2022-09-10T13:20:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:25 smithi110 ceph-mon[97633]: pgmap v16996: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:20:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:25 smithi104 ceph-mon[97044]: pgmap v16996: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:20:25.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:25.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:25.582309+0000) 2022-09-10T13:20:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:25.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:25.972738+0000) 2022-09-10T13:20:26.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:26 smithi104 ceph-mon[97044]: pgmap v16997: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:20:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:26 smithi110 ceph-mon[97633]: pgmap v16997: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:20:26.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:26.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:26.582515+0000) 2022-09-10T13:20:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:26.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:26.972974+0000) 2022-09-10T13:20:27.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:27.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:27.582759+0000) 2022-09-10T13:20:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:27.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:27.973161+0000) 2022-09-10T13:20:28.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:28.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:28.583005+0000) 2022-09-10T13:20:29.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:29 smithi104 ceph-mon[97044]: pgmap v16998: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:28.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:28.973330+0000) 2022-09-10T13:20:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:29 smithi110 ceph-mon[97633]: pgmap v16998: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:29.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:29.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:29.583270+0000) 2022-09-10T13:20:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:29.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:29.973483+0000) 2022-09-10T13:20:30.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:30.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:30.583525+0000) 2022-09-10T13:20:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:30.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:30.973694+0000) 2022-09-10T13:20:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:31 smithi110 ceph-mon[97633]: pgmap v16999: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:31 smithi104 ceph-mon[97044]: pgmap v16999: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:31.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:31.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:31.583777+0000) 2022-09-10T13:20:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:31.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:31.973910+0000) 2022-09-10T13:20:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:20:32.061+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:20:32.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:32.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:32.583975+0000) 2022-09-10T13:20:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:32.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:32.974195+0000) 2022-09-10T13:20:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:33 smithi110 ceph-mon[97633]: pgmap v17000: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:33 smithi104 ceph-mon[97044]: pgmap v17000: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:33.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:33.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:33.584213+0000) 2022-09-10T13:20:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:33.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:33.974382+0000) 2022-09-10T13:20:34.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:34.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:34.584374+0000) 2022-09-10T13:20:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:34.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:34.974654+0000) 2022-09-10T13:20:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:35 smithi110 ceph-mon[97633]: pgmap v17001: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:35 smithi104 ceph-mon[97044]: pgmap v17001: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:35.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:35.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:35.584599+0000) 2022-09-10T13:20:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:35.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:35.974874+0000) 2022-09-10T13:20:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:36 smithi110 ceph-mon[97633]: pgmap v17002: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:36.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:36.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:36.584825+0000) 2022-09-10T13:20:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:36 smithi104 ceph-mon[97044]: pgmap v17002: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:36.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:36.975149+0000) 2022-09-10T13:20:37.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:37.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:37.585068+0000) 2022-09-10T13:20:38.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:37.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:37.975323+0000) 2022-09-10T13:20:38.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:38.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:38.585332+0000) 2022-09-10T13:20:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:38.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:38.975520+0000) 2022-09-10T13:20:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:39 smithi110 ceph-mon[97633]: pgmap v17003: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:39 smithi104 ceph-mon[97044]: pgmap v17003: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:39.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:39.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:39.585582+0000) 2022-09-10T13:20:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:39.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:39.975692+0000) 2022-09-10T13:20:40.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:40.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:40.585810+0000) 2022-09-10T13:20:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:40.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:40.975995+0000) 2022-09-10T13:20:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:41 smithi110 ceph-mon[97633]: pgmap v17004: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:41 smithi104 ceph-mon[97044]: pgmap v17004: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:41.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:41.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:41.585945+0000) 2022-09-10T13:20:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:41.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:41.976231+0000) 2022-09-10T13:20:42.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:42.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:42.586188+0000) 2022-09-10T13:20:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:42.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:42.976433+0000) 2022-09-10T13:20:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:43 smithi110 ceph-mon[97633]: pgmap v17005: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:43 smithi104 ceph-mon[97044]: pgmap v17005: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:43.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:43.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:43.586412+0000) 2022-09-10T13:20:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:43.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:43.976643+0000) 2022-09-10T13:20:44.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:44.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:44.586649+0000) 2022-09-10T13:20:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:44.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:44.976850+0000) 2022-09-10T13:20:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:45 smithi110 ceph-mon[97633]: pgmap v17006: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:45 smithi104 ceph-mon[97044]: pgmap v17006: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:45.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:45.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:45.586897+0000) 2022-09-10T13:20:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:45.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:45.977067+0000) 2022-09-10T13:20:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:46 smithi110 ceph-mon[97633]: pgmap v17007: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:20:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:20:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:20:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:20:46.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:46.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:46.587167+0000) 2022-09-10T13:20:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:46 smithi104 ceph-mon[97044]: pgmap v17007: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:20:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:20:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:46.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:46.977265+0000) 2022-09-10T13:20:47.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:20:47.076+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:20:47.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:47.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:47.587393+0000) 2022-09-10T13:20:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:47.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:47.977505+0000) 2022-09-10T13:20:48.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:48.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:48.587642+0000) 2022-09-10T13:20:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:48.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:48.977753+0000) 2022-09-10T13:20:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:49 smithi110 ceph-mon[97633]: pgmap v17008: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:49 smithi104 ceph-mon[97044]: pgmap v17008: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:49.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:49.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:49.587867+0000) 2022-09-10T13:20:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:49.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:49.977998+0000) 2022-09-10T13:20:50.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:50.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:50.588076+0000) 2022-09-10T13:20:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:50.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:50.978224+0000) 2022-09-10T13:20:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:51 smithi110 ceph-mon[97633]: pgmap v17009: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:51 smithi104 ceph-mon[97044]: pgmap v17009: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:51.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:51.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:51.588334+0000) 2022-09-10T13:20:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:51.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:51.978418+0000) 2022-09-10T13:20:52.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:52.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:52.588603+0000) 2022-09-10T13:20:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:52.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:52.978599+0000) 2022-09-10T13:20:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:53 smithi110 ceph-mon[97633]: pgmap v17010: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:53 smithi104 ceph-mon[97044]: pgmap v17010: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:53.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:53.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:53.588805+0000) 2022-09-10T13:20:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:53.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:53.978784+0000) 2022-09-10T13:20:54.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:54.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:54.588991+0000) 2022-09-10T13:20:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:54.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:54.979028+0000) 2022-09-10T13:20:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:55 smithi110 ceph-mon[97633]: pgmap v17011: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:55 smithi104 ceph-mon[97044]: pgmap v17011: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:55.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:55.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:55.589223+0000) 2022-09-10T13:20:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:55.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:55.979220+0000) 2022-09-10T13:20:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:56 smithi110 ceph-mon[97633]: pgmap v17012: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:56.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:56.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:56.589415+0000) 2022-09-10T13:20:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:56 smithi104 ceph-mon[97044]: pgmap v17012: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:20:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:56.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:56.979476+0000) 2022-09-10T13:20:57.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:57.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:57.589645+0000) 2022-09-10T13:20:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:57.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:57.979727+0000) 2022-09-10T13:20:58.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:58.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:58.589867+0000) 2022-09-10T13:20:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:58.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:58.979950+0000) 2022-09-10T13:20:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:20:59 smithi110 ceph-mon[97633]: pgmap v17013: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:20:59 smithi104 ceph-mon[97044]: pgmap v17013: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:20:59.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:20:59.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:59.590086+0000) 2022-09-10T13:21:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:20:59.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:20:59.980205+0000) 2022-09-10T13:21:00.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:00.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:00.590278+0000) 2022-09-10T13:21:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:00.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:00.980377+0000) 2022-09-10T13:21:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:01 smithi110 ceph-mon[97633]: pgmap v17014: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:01 smithi104 ceph-mon[97044]: pgmap v17014: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:01.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:01.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:01.590427+0000) 2022-09-10T13:21:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:01.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:01.980614+0000) 2022-09-10T13:21:02.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:21:02.091+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:21:02.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:02.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:02.590589+0000) 2022-09-10T13:21:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:02.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:02.980867+0000) 2022-09-10T13:21:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:03 smithi110 ceph-mon[97633]: pgmap v17015: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:03 smithi104 ceph-mon[97044]: pgmap v17015: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:03.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:03.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:03.590791+0000) 2022-09-10T13:21:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:03.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:03.981082+0000) 2022-09-10T13:21:04.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:04.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:04.590977+0000) 2022-09-10T13:21:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:04.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:04.981255+0000) 2022-09-10T13:21:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:05 smithi110 ceph-mon[97633]: pgmap v17016: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:05 smithi104 ceph-mon[97044]: pgmap v17016: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:05.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:05.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:05.591223+0000) 2022-09-10T13:21:06.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:05.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:05.981414+0000) 2022-09-10T13:21:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:06 smithi110 ceph-mon[97633]: pgmap v17017: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:21:06.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:06.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:06.591391+0000) 2022-09-10T13:21:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:06 smithi104 ceph-mon[97044]: pgmap v17017: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:21:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:06.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:06.981613+0000) 2022-09-10T13:21:07.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:07.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:07.591606+0000) 2022-09-10T13:21:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:07.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:07.981832+0000) 2022-09-10T13:21:08.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:08.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:08.591839+0000) 2022-09-10T13:21:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:08 smithi110 ceph-mon[97633]: osdmap e1144: 12 total, 12 up, 12 in 2022-09-10T13:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:08 smithi110 ceph-mon[97633]: pgmap v17019: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:21:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:08 smithi104 ceph-mon[97044]: osdmap e1144: 12 total, 12 up, 12 in 2022-09-10T13:21:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:08 smithi104 ceph-mon[97044]: pgmap v17019: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:21:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:08.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:08.982120+0000) 2022-09-10T13:21:09.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:09.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:09.592049+0000) 2022-09-10T13:21:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:09.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:09.982339+0000) 2022-09-10T13:21:10.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:10.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:10.592440+0000) 2022-09-10T13:21:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:10.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:10.982514+0000) 2022-09-10T13:21:11.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:11 smithi110 ceph-mon[97633]: pgmap v17020: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:21:11.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:11 smithi110 ceph-mon[97633]: osdmap e1145: 12 total, 12 up, 12 in 2022-09-10T13:21:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:11 smithi104 ceph-mon[97044]: pgmap v17020: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:21:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:11 smithi104 ceph-mon[97044]: osdmap e1145: 12 total, 12 up, 12 in 2022-09-10T13:21:11.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:11.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:11.592639+0000) 2022-09-10T13:21:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:11.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:11.982678+0000) 2022-09-10T13:21:12.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:12.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:12.592898+0000) 2022-09-10T13:21:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:12.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:12.982927+0000) 2022-09-10T13:21:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:13 smithi110 ceph-mon[97633]: pgmap v17022: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:21:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:13 smithi104 ceph-mon[97044]: pgmap v17022: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:21:13.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:13.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:13.593140+0000) 2022-09-10T13:21:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:13.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:13.983163+0000) 2022-09-10T13:21:14.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:14.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:14.593364+0000) 2022-09-10T13:21:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:14.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:14.983422+0000) 2022-09-10T13:21:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:15 smithi110 ceph-mon[97633]: pgmap v17023: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:21:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:15 smithi104 ceph-mon[97044]: pgmap v17023: 129 pgs: 129 active+clean; 9.6 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:21:15.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:15.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:15.593601+0000) 2022-09-10T13:21:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:15.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:15.983591+0000) 2022-09-10T13:21:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:16 smithi110 ceph-mon[97633]: pgmap v17024: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T13:21:16.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:16.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:16.593822+0000) 2022-09-10T13:21:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:16 smithi104 ceph-mon[97044]: pgmap v17024: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T13:21:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:16.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:16.983815+0000) 2022-09-10T13:21:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:21:17.107+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:21:17.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:17.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:17.594056+0000) 2022-09-10T13:21:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:17.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:17.984040+0000) 2022-09-10T13:21:18.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:18.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:18.594345+0000) 2022-09-10T13:21:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:18.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:18.984310+0000) 2022-09-10T13:21:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:19 smithi110 ceph-mon[97633]: pgmap v17025: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:21:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:19 smithi104 ceph-mon[97044]: pgmap v17025: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:21:19.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:19.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:19.594523+0000) 2022-09-10T13:21:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:19.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:19.984541+0000) 2022-09-10T13:21:20.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:20.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:20.594745+0000) 2022-09-10T13:21:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:20.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:20.984719+0000) 2022-09-10T13:21:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:21 smithi110 ceph-mon[97633]: pgmap v17026: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:21:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:21 smithi104 ceph-mon[97044]: pgmap v17026: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:21:21.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:21.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:21.594895+0000) 2022-09-10T13:21:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:21.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:21.984956+0000) 2022-09-10T13:21:22.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:22.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:22.595123+0000) 2022-09-10T13:21:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:22.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:22.985229+0000) 2022-09-10T13:21:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:23 smithi110 ceph-mon[97633]: pgmap v17027: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:21:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:23 smithi104 ceph-mon[97044]: pgmap v17027: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:21:23.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:23.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:23.595322+0000) 2022-09-10T13:21:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:23.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:23.985447+0000) 2022-09-10T13:21:24.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:24.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:24.595444+0000) 2022-09-10T13:21:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:24.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:24.985636+0000) 2022-09-10T13:21:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:25 smithi110 ceph-mon[97633]: pgmap v17028: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:21:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:25 smithi104 ceph-mon[97044]: pgmap v17028: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:21:25.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:25.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:25.595670+0000) 2022-09-10T13:21:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:25.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:25.985886+0000) 2022-09-10T13:21:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:26 smithi110 ceph-mon[97633]: pgmap v17029: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:21:26.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:26.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:26.595907+0000) 2022-09-10T13:21:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:26 smithi104 ceph-mon[97044]: pgmap v17029: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:21:27.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:26.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:26.986160+0000) 2022-09-10T13:21:27.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:27.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:27.596146+0000) 2022-09-10T13:21:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:27.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:27.986343+0000) 2022-09-10T13:21:28.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:28.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:28.596299+0000) 2022-09-10T13:21:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:28.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:28.986568+0000) 2022-09-10T13:21:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:29 smithi110 ceph-mon[97633]: pgmap v17030: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:29 smithi104 ceph-mon[97044]: pgmap v17030: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:29.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:29.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:29.596547+0000) 2022-09-10T13:21:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:29.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:29.986808+0000) 2022-09-10T13:21:30.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:30.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:30.596676+0000) 2022-09-10T13:21:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:30.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:30.987040+0000) 2022-09-10T13:21:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:31 smithi110 ceph-mon[97633]: pgmap v17031: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:31 smithi104 ceph-mon[97044]: pgmap v17031: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:31.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:31.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:31.596877+0000) 2022-09-10T13:21:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:31.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:31.987225+0000) 2022-09-10T13:21:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:21:32.118+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:21:32.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:32.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:32.597152+0000) 2022-09-10T13:21:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:32.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:32.987399+0000) 2022-09-10T13:21:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:33 smithi110 ceph-mon[97633]: pgmap v17032: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:33 smithi104 ceph-mon[97044]: pgmap v17032: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:33.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:33.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:33.597361+0000) 2022-09-10T13:21:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:33.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:33.987519+0000) 2022-09-10T13:21:34.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:34.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:34.597574+0000) 2022-09-10T13:21:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:34.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:34.987683+0000) 2022-09-10T13:21:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:35 smithi110 ceph-mon[97633]: pgmap v17033: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:35 smithi104 ceph-mon[97044]: pgmap v17033: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:35.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:35.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:35.597818+0000) 2022-09-10T13:21:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:35.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:35.987869+0000) 2022-09-10T13:21:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:36 smithi110 ceph-mon[97633]: pgmap v17034: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:36.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:36.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:36.598079+0000) 2022-09-10T13:21:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:36 smithi104 ceph-mon[97044]: pgmap v17034: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:36.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:36.988134+0000) 2022-09-10T13:21:37.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:37.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:37.598218+0000) 2022-09-10T13:21:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:37.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:37.988350+0000) 2022-09-10T13:21:38.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:38.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:38.598399+0000) 2022-09-10T13:21:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:38.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:38.988549+0000) 2022-09-10T13:21:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:39 smithi110 ceph-mon[97633]: pgmap v17035: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:39 smithi104 ceph-mon[97044]: pgmap v17035: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:39.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:39.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:39.598595+0000) 2022-09-10T13:21:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:39.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:39.988728+0000) 2022-09-10T13:21:40.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:40.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:40.598844+0000) 2022-09-10T13:21:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:40.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:40.988974+0000) 2022-09-10T13:21:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:41 smithi110 ceph-mon[97633]: pgmap v17036: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:41 smithi104 ceph-mon[97044]: pgmap v17036: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:41.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:41.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:41.599048+0000) 2022-09-10T13:21:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:41.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:41.989205+0000) 2022-09-10T13:21:42.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:42.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:42.599270+0000) 2022-09-10T13:21:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:42.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:42.989405+0000) 2022-09-10T13:21:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:43 smithi110 ceph-mon[97633]: pgmap v17037: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:43 smithi104 ceph-mon[97044]: pgmap v17037: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:43.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:43.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:43.599522+0000) 2022-09-10T13:21:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:43.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:43.989551+0000) 2022-09-10T13:21:44.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:44.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:44.599741+0000) 2022-09-10T13:21:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:44.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:44.989746+0000) 2022-09-10T13:21:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:45 smithi110 ceph-mon[97633]: pgmap v17038: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:45 smithi104 ceph-mon[97044]: pgmap v17038: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:45.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:45.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:45.599966+0000) 2022-09-10T13:21:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:45.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:45.989927+0000) 2022-09-10T13:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:46 smithi110 ceph-mon[97633]: pgmap v17039: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:21:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:21:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:21:46.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:46.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:46.600212+0000) 2022-09-10T13:21:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:46 smithi104 ceph-mon[97044]: pgmap v17039: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:21:47.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:21:47.126+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:21:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:46.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:46.990152+0000) 2022-09-10T13:21:47.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:47.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:47.600390+0000) 2022-09-10T13:21:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:47.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:47.990262+0000) 2022-09-10T13:21:48.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:48.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:48.600553+0000) 2022-09-10T13:21:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:48.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:48.990483+0000) 2022-09-10T13:21:49.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:49 smithi110 ceph-mon[97633]: pgmap v17040: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:49 smithi104 ceph-mon[97044]: pgmap v17040: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:49.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:49.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:49.600767+0000) 2022-09-10T13:21:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:49.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:49.990769+0000) 2022-09-10T13:21:50.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:50.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:50.600993+0000) 2022-09-10T13:21:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:50.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:50.991010+0000) 2022-09-10T13:21:51.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:51 smithi110 ceph-mon[97633]: pgmap v17041: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:51 smithi104 ceph-mon[97044]: pgmap v17041: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:51.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:51.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:51.601222+0000) 2022-09-10T13:21:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:51.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:51.991182+0000) 2022-09-10T13:21:52.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:52.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:52.601445+0000) 2022-09-10T13:21:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:52.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:52.991395+0000) 2022-09-10T13:21:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:53 smithi110 ceph-mon[97633]: pgmap v17042: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:53 smithi104 ceph-mon[97044]: pgmap v17042: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:53.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:53.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:53.601677+0000) 2022-09-10T13:21:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:53.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:53.991642+0000) 2022-09-10T13:21:54.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:54.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:54.601918+0000) 2022-09-10T13:21:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:54.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:54.991901+0000) 2022-09-10T13:21:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:55 smithi110 ceph-mon[97633]: pgmap v17043: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:55 smithi104 ceph-mon[97044]: pgmap v17043: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:55.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:55.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:55.602126+0000) 2022-09-10T13:21:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:55.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:55.992165+0000) 2022-09-10T13:21:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:56 smithi110 ceph-mon[97633]: pgmap v17044: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:56.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:56.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:56.602302+0000) 2022-09-10T13:21:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:56 smithi104 ceph-mon[97044]: pgmap v17044: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:21:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:56.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:56.992354+0000) 2022-09-10T13:21:57.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:57.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:57.602544+0000) 2022-09-10T13:21:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:57.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:57.992571+0000) 2022-09-10T13:21:58.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:58.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:58.602730+0000) 2022-09-10T13:21:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:58.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:58.992817+0000) 2022-09-10T13:21:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:21:59 smithi110 ceph-mon[97633]: pgmap v17045: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:21:59 smithi104 ceph-mon[97044]: pgmap v17045: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:21:59.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:21:59.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:59.602956+0000) 2022-09-10T13:22:00.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:21:59.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:21:59.993036+0000) 2022-09-10T13:22:00.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:00.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:00.603090+0000) 2022-09-10T13:22:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:00.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:00.993301+0000) 2022-09-10T13:22:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:01 smithi110 ceph-mon[97633]: pgmap v17046: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:01.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:01 smithi104 ceph-mon[97044]: pgmap v17046: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:01.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:01.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:01.603313+0000) 2022-09-10T13:22:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:01.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:01.993551+0000) 2022-09-10T13:22:02.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:22:02.141+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:22:02.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:02.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:02.603465+0000) 2022-09-10T13:22:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:02.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:02.993772+0000) 2022-09-10T13:22:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:03 smithi110 ceph-mon[97633]: pgmap v17047: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:03 smithi104 ceph-mon[97044]: pgmap v17047: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:03.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:03.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:03.603592+0000) 2022-09-10T13:22:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:03.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:03.993933+0000) 2022-09-10T13:22:04.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:04.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:04.603822+0000) 2022-09-10T13:22:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:04.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:04.994156+0000) 2022-09-10T13:22:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:05 smithi110 ceph-mon[97633]: pgmap v17048: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:05 smithi104 ceph-mon[97044]: pgmap v17048: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:05.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:05.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:05.604069+0000) 2022-09-10T13:22:06.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:05.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:05.994389+0000) 2022-09-10T13:22:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:06 smithi110 ceph-mon[97633]: pgmap v17049: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:22:06.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:06.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:06.604290+0000) 2022-09-10T13:22:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:06 smithi104 ceph-mon[97044]: pgmap v17049: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:22:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:06.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:06.994556+0000) 2022-09-10T13:22:07.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:07.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:07.604448+0000) 2022-09-10T13:22:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:07.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:07.994809+0000) 2022-09-10T13:22:08.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:08.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:08.604698+0000) 2022-09-10T13:22:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:08 smithi110 ceph-mon[97633]: osdmap e1146: 12 total, 12 up, 12 in 2022-09-10T13:22:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:08 smithi110 ceph-mon[97633]: pgmap v17051: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:22:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:08 smithi104 ceph-mon[97044]: osdmap e1146: 12 total, 12 up, 12 in 2022-09-10T13:22:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:08 smithi104 ceph-mon[97044]: pgmap v17051: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:22:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:08.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:08.995042+0000) 2022-09-10T13:22:09.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:09.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:09.604945+0000) 2022-09-10T13:22:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:09.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:09.995283+0000) 2022-09-10T13:22:10.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:10.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:10.605185+0000) 2022-09-10T13:22:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:10.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:10.995477+0000) 2022-09-10T13:22:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:11 smithi110 ceph-mon[97633]: pgmap v17052: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:22:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:11 smithi110 ceph-mon[97633]: osdmap e1147: 12 total, 12 up, 12 in 2022-09-10T13:22:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:11 smithi104 ceph-mon[97044]: pgmap v17052: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:22:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:11 smithi104 ceph-mon[97044]: osdmap e1147: 12 total, 12 up, 12 in 2022-09-10T13:22:11.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:11.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:11.605427+0000) 2022-09-10T13:22:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:11.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:11.995599+0000) 2022-09-10T13:22:12.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:12.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:12.605650+0000) 2022-09-10T13:22:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:12.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:12.995780+0000) 2022-09-10T13:22:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:13 smithi110 ceph-mon[97633]: pgmap v17054: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:22:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:13 smithi104 ceph-mon[97044]: pgmap v17054: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:22:13.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:13.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:13.605901+0000) 2022-09-10T13:22:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:13.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:13.995958+0000) 2022-09-10T13:22:14.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:14.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:14.606128+0000) 2022-09-10T13:22:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:14.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:14.996223+0000) 2022-09-10T13:22:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:15 smithi110 ceph-mon[97633]: pgmap v17055: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:22:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:15 smithi104 ceph-mon[97044]: pgmap v17055: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:22:16.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:15.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:15.606307+0000) 2022-09-10T13:22:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:15.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:15.996443+0000) 2022-09-10T13:22:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:16 smithi110 ceph-mon[97633]: pgmap v17056: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T13:22:16.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:16.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:16.606488+0000) 2022-09-10T13:22:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:16 smithi104 ceph-mon[97044]: pgmap v17056: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T13:22:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:16.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:16.996653+0000) 2022-09-10T13:22:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:22:17.152+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:22:18.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:17.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:17.606706+0000) 2022-09-10T13:22:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:17.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:17.996873+0000) 2022-09-10T13:22:18.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:18.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:18.606916+0000) 2022-09-10T13:22:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:18.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:18.997128+0000) 2022-09-10T13:22:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:19 smithi110 ceph-mon[97633]: pgmap v17057: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:22:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:19 smithi104 ceph-mon[97044]: pgmap v17057: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:22:19.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:19.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:19.607145+0000) 2022-09-10T13:22:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:19.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:19.997390+0000) 2022-09-10T13:22:20.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:20.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:20.607374+0000) 2022-09-10T13:22:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:20.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:20.997568+0000) 2022-09-10T13:22:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:21 smithi110 ceph-mon[97633]: pgmap v17058: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:22:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:21 smithi104 ceph-mon[97044]: pgmap v17058: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:22:22.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:21.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:21.607608+0000) 2022-09-10T13:22:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:21.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:21.997766+0000) 2022-09-10T13:22:22.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:22.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:22.607853+0000) 2022-09-10T13:22:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:22.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:22.998011+0000) 2022-09-10T13:22:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:23 smithi110 ceph-mon[97633]: pgmap v17059: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:22:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:23 smithi104 ceph-mon[97044]: pgmap v17059: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:22:23.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:23.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:23.608126+0000) 2022-09-10T13:22:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:23.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:23.998227+0000) 2022-09-10T13:22:24.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:24.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:24.608380+0000) 2022-09-10T13:22:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:24.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:24.998462+0000) 2022-09-10T13:22:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:25 smithi110 ceph-mon[97633]: pgmap v17060: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:22:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:25 smithi104 ceph-mon[97044]: pgmap v17060: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:22:25.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:25.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:25.608671+0000) 2022-09-10T13:22:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:25.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:25.998649+0000) 2022-09-10T13:22:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:26 smithi110 ceph-mon[97633]: pgmap v17061: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:22:26.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:26.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:26.608841+0000) 2022-09-10T13:22:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:26 smithi104 ceph-mon[97044]: pgmap v17061: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:22:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:26.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:26.998896+0000) 2022-09-10T13:22:27.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:27.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:27.609024+0000) 2022-09-10T13:22:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:27.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:27.999167+0000) 2022-09-10T13:22:28.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:28.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:28.609196+0000) 2022-09-10T13:22:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:28.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:28.999412+0000) 2022-09-10T13:22:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:29 smithi110 ceph-mon[97633]: pgmap v17062: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:29.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:29 smithi104 ceph-mon[97044]: pgmap v17062: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:29.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:29.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:29.609437+0000) 2022-09-10T13:22:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:29.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:29.999609+0000) 2022-09-10T13:22:30.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:30.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:30.609690+0000) 2022-09-10T13:22:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:30.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:30.999894+0000) 2022-09-10T13:22:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:31 smithi110 ceph-mon[97633]: pgmap v17063: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:31 smithi104 ceph-mon[97044]: pgmap v17063: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:32.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:31.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:31.609916+0000) 2022-09-10T13:22:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:31.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:32.000076+0000) 2022-09-10T13:22:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:22:32.158+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:22:33.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:32.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:32.610187+0000) 2022-09-10T13:22:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:32.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:33.000265+0000) 2022-09-10T13:22:33.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:33 smithi110 ceph-mon[97633]: pgmap v17064: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:33 smithi104 ceph-mon[97044]: pgmap v17064: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:34.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:33.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:33.610446+0000) 2022-09-10T13:22:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:33.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:34.000451+0000) 2022-09-10T13:22:35.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:34.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:34.610667+0000) 2022-09-10T13:22:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:34.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:35.000655+0000) 2022-09-10T13:22:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:35 smithi110 ceph-mon[97633]: pgmap v17065: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:35 smithi104 ceph-mon[97044]: pgmap v17065: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:36.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:35.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:35.610891+0000) 2022-09-10T13:22:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:36.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:36.000837+0000) 2022-09-10T13:22:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:36 smithi110 ceph-mon[97633]: pgmap v17066: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:37.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:36.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:36.611058+0000) 2022-09-10T13:22:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:36 smithi104 ceph-mon[97044]: pgmap v17066: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:37.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:37.001090+0000) 2022-09-10T13:22:38.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:37.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:37.611196+0000) 2022-09-10T13:22:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:38.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:38.001339+0000) 2022-09-10T13:22:39.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:38.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:38.611389+0000) 2022-09-10T13:22:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:39.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:39.001524+0000) 2022-09-10T13:22:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:39 smithi110 ceph-mon[97633]: pgmap v17067: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:39 smithi104 ceph-mon[97044]: pgmap v17067: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:40.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:39.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:39.611599+0000) 2022-09-10T13:22:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:40.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:40.001741+0000) 2022-09-10T13:22:41.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:40.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:40.611887+0000) 2022-09-10T13:22:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:41.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:41.001962+0000) 2022-09-10T13:22:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:41 smithi110 ceph-mon[97633]: pgmap v17068: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:41 smithi104 ceph-mon[97044]: pgmap v17068: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:42.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:41.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:41.612160+0000) 2022-09-10T13:22:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:42.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:42.002196+0000) 2022-09-10T13:22:43.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:42.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:42.612378+0000) 2022-09-10T13:22:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:43.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:43.002380+0000) 2022-09-10T13:22:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:43 smithi110 ceph-mon[97633]: pgmap v17069: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:43 smithi104 ceph-mon[97044]: pgmap v17069: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:44.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:43.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:43.612536+0000) 2022-09-10T13:22:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:44.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:44.002618+0000) 2022-09-10T13:22:45.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:44.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:44.612748+0000) 2022-09-10T13:22:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:45.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:45.002826+0000) 2022-09-10T13:22:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:45 smithi110 ceph-mon[97633]: pgmap v17070: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:45 smithi104 ceph-mon[97044]: pgmap v17070: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:46.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:45.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:45.613021+0000) 2022-09-10T13:22:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:46.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:46.003129+0000) 2022-09-10T13:22:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:46 smithi110 ceph-mon[97633]: pgmap v17071: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:22:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:22:47.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:46.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:46.613152+0000) 2022-09-10T13:22:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:46 smithi104 ceph-mon[97044]: pgmap v17071: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:22:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:22:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:22:47.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:22:47.168+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:22:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:47.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:47.003359+0000) 2022-09-10T13:22:48.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:47.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:47.613319+0000) 2022-09-10T13:22:48.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:48.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:48.003588+0000) 2022-09-10T13:22:49.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:48.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:48.613557+0000) 2022-09-10T13:22:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:49.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:49.003767+0000) 2022-09-10T13:22:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:49 smithi110 ceph-mon[97633]: pgmap v17072: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:49 smithi104 ceph-mon[97044]: pgmap v17072: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:50.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:49.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:49.613802+0000) 2022-09-10T13:22:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:50.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:50.003969+0000) 2022-09-10T13:22:51.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:50.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:50.614051+0000) 2022-09-10T13:22:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:51.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:51.004243+0000) 2022-09-10T13:22:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:51 smithi110 ceph-mon[97633]: pgmap v17073: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:51 smithi104 ceph-mon[97044]: pgmap v17073: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:52.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:51.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:51.614275+0000) 2022-09-10T13:22:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:52.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:52.004422+0000) 2022-09-10T13:22:53.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:52.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:52.614544+0000) 2022-09-10T13:22:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:53.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:53.004648+0000) 2022-09-10T13:22:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:53 smithi110 ceph-mon[97633]: pgmap v17074: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:53 smithi104 ceph-mon[97044]: pgmap v17074: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:54.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:53.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:53.614762+0000) 2022-09-10T13:22:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:54.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:54.004806+0000) 2022-09-10T13:22:55.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:54.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:54.615026+0000) 2022-09-10T13:22:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:55.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:55.005011+0000) 2022-09-10T13:22:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:55 smithi110 ceph-mon[97633]: pgmap v17075: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:55 smithi104 ceph-mon[97044]: pgmap v17075: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:56.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:55.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:55.615242+0000) 2022-09-10T13:22:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:56.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:56.005198+0000) 2022-09-10T13:22:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:56 smithi110 ceph-mon[97633]: pgmap v17076: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:57.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:56.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:56.615484+0000) 2022-09-10T13:22:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:56 smithi104 ceph-mon[97044]: pgmap v17076: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:22:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:57.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:57.005428+0000) 2022-09-10T13:22:58.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:57.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:57.615657+0000) 2022-09-10T13:22:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:58.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:58.005690+0000) 2022-09-10T13:22:59.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:58.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:58.615894+0000) 2022-09-10T13:22:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:22:59.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:59.005938+0000) 2022-09-10T13:22:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:22:59 smithi110 ceph-mon[97633]: pgmap v17077: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:22:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:22:59 smithi104 ceph-mon[97044]: pgmap v17077: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:00.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:22:59.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:22:59.616164+0000) 2022-09-10T13:23:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:00.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:00.006182+0000) 2022-09-10T13:23:01.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:00.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:00.616308+0000) 2022-09-10T13:23:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:01.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:01.006381+0000) 2022-09-10T13:23:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:01 smithi110 ceph-mon[97633]: pgmap v17078: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:01 smithi104 ceph-mon[97044]: pgmap v17078: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:02.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:01.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:01.616505+0000) 2022-09-10T13:23:02.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:23:02.175+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:23:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:02.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:02.006573+0000) 2022-09-10T13:23:03.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:02.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:02.616683+0000) 2022-09-10T13:23:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:03.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:03.006803+0000) 2022-09-10T13:23:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:03 smithi110 ceph-mon[97633]: pgmap v17079: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:03 smithi104 ceph-mon[97044]: pgmap v17079: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:04.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:03.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:03.616942+0000) 2022-09-10T13:23:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:04.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:04.006998+0000) 2022-09-10T13:23:05.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:04.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:04.617210+0000) 2022-09-10T13:23:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:05.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:05.007194+0000) 2022-09-10T13:23:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:05 smithi110 ceph-mon[97633]: pgmap v17080: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:05 smithi104 ceph-mon[97044]: pgmap v17080: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:06.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:05.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:05.617412+0000) 2022-09-10T13:23:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:06.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:06.007337+0000) 2022-09-10T13:23:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:06 smithi110 ceph-mon[97633]: pgmap v17081: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:23:07.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:06.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:06.617642+0000) 2022-09-10T13:23:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:06 smithi104 ceph-mon[97044]: pgmap v17081: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:23:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:07.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:07.007567+0000) 2022-09-10T13:23:08.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:07.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:07.617916+0000) 2022-09-10T13:23:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:08.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:08.007820+0000) 2022-09-10T13:23:09.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:08.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:08.618090+0000) 2022-09-10T13:23:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:08 smithi110 ceph-mon[97633]: osdmap e1148: 12 total, 12 up, 12 in 2022-09-10T13:23:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:08 smithi110 ceph-mon[97633]: pgmap v17083: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:23:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:08 smithi104 ceph-mon[97044]: osdmap e1148: 12 total, 12 up, 12 in 2022-09-10T13:23:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:08 smithi104 ceph-mon[97044]: pgmap v17083: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:23:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:09.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:09.007999+0000) 2022-09-10T13:23:10.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:09.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:09.618286+0000) 2022-09-10T13:23:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:10.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:10.008221+0000) 2022-09-10T13:23:11.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:10.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:10.618502+0000) 2022-09-10T13:23:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:11.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:11.008494+0000) 2022-09-10T13:23:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:11 smithi110 ceph-mon[97633]: pgmap v17084: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:23:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:11 smithi110 ceph-mon[97633]: osdmap e1149: 12 total, 12 up, 12 in 2022-09-10T13:23:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:11 smithi104 ceph-mon[97044]: pgmap v17084: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:23:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:11 smithi104 ceph-mon[97044]: osdmap e1149: 12 total, 12 up, 12 in 2022-09-10T13:23:12.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:11.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:11.618723+0000) 2022-09-10T13:23:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:12.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:12.008702+0000) 2022-09-10T13:23:13.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:12.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:12.618944+0000) 2022-09-10T13:23:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:13.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:13.008952+0000) 2022-09-10T13:23:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:13 smithi110 ceph-mon[97633]: pgmap v17086: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:23:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:13 smithi104 ceph-mon[97044]: pgmap v17086: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:23:14.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:13.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:13.619162+0000) 2022-09-10T13:23:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:14.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:14.009188+0000) 2022-09-10T13:23:15.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:14.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:14.619399+0000) 2022-09-10T13:23:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:15.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:15.009418+0000) 2022-09-10T13:23:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:15 smithi110 ceph-mon[97633]: pgmap v17087: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:23:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:15 smithi104 ceph-mon[97044]: pgmap v17087: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:23:16.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:15.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:15.619656+0000) 2022-09-10T13:23:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:16.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:16.009644+0000) 2022-09-10T13:23:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:16 smithi110 ceph-mon[97633]: pgmap v17088: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T13:23:17.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:16.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:16.619872+0000) 2022-09-10T13:23:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:16 smithi104 ceph-mon[97044]: pgmap v17088: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T13:23:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:17.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:17.009911+0000) 2022-09-10T13:23:17.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:23:17.190+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:23:18.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:17.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:17.620120+0000) 2022-09-10T13:23:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:18.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:18.010179+0000) 2022-09-10T13:23:19.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:18.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:18.620328+0000) 2022-09-10T13:23:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:19.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:19.010405+0000) 2022-09-10T13:23:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:19 smithi110 ceph-mon[97633]: pgmap v17089: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:23:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:19 smithi104 ceph-mon[97044]: pgmap v17089: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:23:20.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:19.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:19.620516+0000) 2022-09-10T13:23:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:20.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:20.010626+0000) 2022-09-10T13:23:21.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:20.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:20.620776+0000) 2022-09-10T13:23:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:21.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:21.010875+0000) 2022-09-10T13:23:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:21 smithi110 ceph-mon[97633]: pgmap v17090: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:23:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:21 smithi104 ceph-mon[97044]: pgmap v17090: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:23:22.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:21.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:21.621024+0000) 2022-09-10T13:23:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:22.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:22.011138+0000) 2022-09-10T13:23:23.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:22.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:22.621223+0000) 2022-09-10T13:23:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:23.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:23.011283+0000) 2022-09-10T13:23:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:23 smithi110 ceph-mon[97633]: pgmap v17091: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:23:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:23 smithi104 ceph-mon[97044]: pgmap v17091: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:23:24.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:23.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:23.621452+0000) 2022-09-10T13:23:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:24.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:24.011498+0000) 2022-09-10T13:23:25.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:24.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:24.621665+0000) 2022-09-10T13:23:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:25.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:25.011757+0000) 2022-09-10T13:23:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:25 smithi110 ceph-mon[97633]: pgmap v17092: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:23:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:25 smithi104 ceph-mon[97044]: pgmap v17092: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:23:26.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:25.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:25.621889+0000) 2022-09-10T13:23:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:26.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:26.012013+0000) 2022-09-10T13:23:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:26 smithi110 ceph-mon[97633]: pgmap v17093: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:23:27.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:26.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:26.622179+0000) 2022-09-10T13:23:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:26 smithi104 ceph-mon[97044]: pgmap v17093: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:23:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:27.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:27.012162+0000) 2022-09-10T13:23:28.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:27.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:27.622392+0000) 2022-09-10T13:23:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:28.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:28.012318+0000) 2022-09-10T13:23:29.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:28.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:28.622554+0000) 2022-09-10T13:23:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:29.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:29.012501+0000) 2022-09-10T13:23:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:29 smithi110 ceph-mon[97633]: pgmap v17094: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:23:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:29 smithi104 ceph-mon[97044]: pgmap v17094: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:23:30.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:29.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:29.622782+0000) 2022-09-10T13:23:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:30.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:30.012687+0000) 2022-09-10T13:23:31.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:30.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:30.623051+0000) 2022-09-10T13:23:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:31.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:31.012893+0000) 2022-09-10T13:23:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:31 smithi110 ceph-mon[97633]: pgmap v17095: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T13:23:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:31 smithi104 ceph-mon[97044]: pgmap v17095: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T13:23:32.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:31.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:31.623285+0000) 2022-09-10T13:23:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:32.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:32.013117+0000) 2022-09-10T13:23:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:23:32.206+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:23:33.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:32.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:32.623531+0000) 2022-09-10T13:23:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:33.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:33.013280+0000) 2022-09-10T13:23:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:33 smithi110 ceph-mon[97633]: pgmap v17096: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T13:23:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:33 smithi104 ceph-mon[97044]: pgmap v17096: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T13:23:34.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:33.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:33.623754+0000) 2022-09-10T13:23:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:34.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:34.013517+0000) 2022-09-10T13:23:35.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:34.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:34.624024+0000) 2022-09-10T13:23:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:35.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:35.013770+0000) 2022-09-10T13:23:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:35 smithi110 ceph-mon[97633]: pgmap v17097: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T13:23:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:35 smithi104 ceph-mon[97044]: pgmap v17097: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2022-09-10T13:23:36.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:35.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:35.624303+0000) 2022-09-10T13:23:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:36.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:36.013979+0000) 2022-09-10T13:23:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:36 smithi110 ceph-mon[97633]: pgmap v17098: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T13:23:37.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:36.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:36.624438+0000) 2022-09-10T13:23:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:36 smithi104 ceph-mon[97044]: pgmap v17098: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2022-09-10T13:23:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:37.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:37.014220+0000) 2022-09-10T13:23:38.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:37.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:37.624684+0000) 2022-09-10T13:23:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:38.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:38.014393+0000) 2022-09-10T13:23:39.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:38.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:38.624939+0000) 2022-09-10T13:23:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:39.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:39.014638+0000) 2022-09-10T13:23:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:39 smithi110 ceph-mon[97633]: pgmap v17099: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:23:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:39 smithi104 ceph-mon[97044]: pgmap v17099: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:23:40.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:39.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:39.625124+0000) 2022-09-10T13:23:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:40.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:40.014879+0000) 2022-09-10T13:23:41.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:40.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:40.625357+0000) 2022-09-10T13:23:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:41.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:41.015171+0000) 2022-09-10T13:23:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:41 smithi110 ceph-mon[97633]: pgmap v17100: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:23:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:41 smithi104 ceph-mon[97044]: pgmap v17100: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:23:42.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:41.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:41.625593+0000) 2022-09-10T13:23:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:42.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:42.015326+0000) 2022-09-10T13:23:43.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:42.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:42.625856+0000) 2022-09-10T13:23:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:43.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:43.015534+0000) 2022-09-10T13:23:43.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:43 smithi110 ceph-mon[97633]: pgmap v17101: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:43 smithi104 ceph-mon[97044]: pgmap v17101: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:44.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:43.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:43.626072+0000) 2022-09-10T13:23:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:44.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:44.015700+0000) 2022-09-10T13:23:45.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:44.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:44.626348+0000) 2022-09-10T13:23:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:45.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:45.015946+0000) 2022-09-10T13:23:45.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:45 smithi110 ceph-mon[97633]: pgmap v17102: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:45 smithi104 ceph-mon[97044]: pgmap v17102: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:46.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:45.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:45.626566+0000) 2022-09-10T13:23:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:46.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:46.016188+0000) 2022-09-10T13:23:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:46 smithi110 ceph-mon[97633]: pgmap v17103: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:23:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:23:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:23:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:23:47.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:46.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:46.626810+0000) 2022-09-10T13:23:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:46 smithi104 ceph-mon[97044]: pgmap v17103: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:23:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:23:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:23:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:23:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:47.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:47.016410+0000) 2022-09-10T13:23:47.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:23:47.219+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:23:48.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:47.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:47.626990+0000) 2022-09-10T13:23:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:48.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:48.016590+0000) 2022-09-10T13:23:49.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:48.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:48.627217+0000) 2022-09-10T13:23:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:49.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:49.016777+0000) 2022-09-10T13:23:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:49 smithi110 ceph-mon[97633]: pgmap v17104: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:49 smithi104 ceph-mon[97044]: pgmap v17104: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:50.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:49.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:49.627415+0000) 2022-09-10T13:23:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:50.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:50.017005+0000) 2022-09-10T13:23:51.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:50.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:50.627572+0000) 2022-09-10T13:23:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:51.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:51.017198+0000) 2022-09-10T13:23:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:51 smithi110 ceph-mon[97633]: pgmap v17105: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:51 smithi104 ceph-mon[97044]: pgmap v17105: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:52.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:51.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:51.627792+0000) 2022-09-10T13:23:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:52.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:52.017327+0000) 2022-09-10T13:23:53.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:52.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:52.628005+0000) 2022-09-10T13:23:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:53.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:53.017515+0000) 2022-09-10T13:23:53.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:53 smithi110 ceph-mon[97633]: pgmap v17106: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:53 smithi104 ceph-mon[97044]: pgmap v17106: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:54.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:53.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:53.628261+0000) 2022-09-10T13:23:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:54.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:54.017740+0000) 2022-09-10T13:23:55.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:54.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:54.628461+0000) 2022-09-10T13:23:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:55.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:55.017923+0000) 2022-09-10T13:23:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:55 smithi110 ceph-mon[97633]: pgmap v17107: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:55 smithi104 ceph-mon[97044]: pgmap v17107: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:56.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:55.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:55.628640+0000) 2022-09-10T13:23:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:56.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:56.018150+0000) 2022-09-10T13:23:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:56 smithi110 ceph-mon[97633]: pgmap v17108: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:57.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:56.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:56.628860+0000) 2022-09-10T13:23:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:56 smithi104 ceph-mon[97044]: pgmap v17108: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:23:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:57.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:57.018367+0000) 2022-09-10T13:23:58.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:57.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:57.629136+0000) 2022-09-10T13:23:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:58.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:58.018612+0000) 2022-09-10T13:23:59.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:58.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:58.629369+0000) 2022-09-10T13:23:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:23:59.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:59.018863+0000) 2022-09-10T13:23:59.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:23:59 smithi110 ceph-mon[97633]: pgmap v17109: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:23:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:23:59 smithi104 ceph-mon[97044]: pgmap v17109: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:00.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:23:59.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:23:59.629613+0000) 2022-09-10T13:24:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:00.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:00.019179+0000) 2022-09-10T13:24:01.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:00.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:00.629828+0000) 2022-09-10T13:24:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:01.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:01.019374+0000) 2022-09-10T13:24:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:01 smithi110 ceph-mon[97633]: pgmap v17110: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:01 smithi104 ceph-mon[97044]: pgmap v17110: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:02.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:01.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:01.630063+0000) 2022-09-10T13:24:02.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:24:02.235+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:24:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:02.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:02.019635+0000) 2022-09-10T13:24:03.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:02.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:02.630386+0000) 2022-09-10T13:24:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:03.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:03.019827+0000) 2022-09-10T13:24:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:03 smithi110 ceph-mon[97633]: pgmap v17111: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:03 smithi104 ceph-mon[97044]: pgmap v17111: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:04.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:03.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:03.630625+0000) 2022-09-10T13:24:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:04.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:04.020029+0000) 2022-09-10T13:24:05.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:04.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:04.630834+0000) 2022-09-10T13:24:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:05.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:05.020234+0000) 2022-09-10T13:24:05.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:05 smithi110 ceph-mon[97633]: pgmap v17112: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:05 smithi104 ceph-mon[97044]: pgmap v17112: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:06.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:05.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:05.631039+0000) 2022-09-10T13:24:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:06.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:06.020466+0000) 2022-09-10T13:24:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:06 smithi110 ceph-mon[97633]: pgmap v17113: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:24:07.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:06.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:06.631298+0000) 2022-09-10T13:24:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:06 smithi104 ceph-mon[97044]: pgmap v17113: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:24:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:07.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:07.020663+0000) 2022-09-10T13:24:08.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:07.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:07.631452+0000) 2022-09-10T13:24:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:08.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:08.020801+0000) 2022-09-10T13:24:09.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:08.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:08.631671+0000) 2022-09-10T13:24:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:08 smithi110 ceph-mon[97633]: osdmap e1150: 12 total, 12 up, 12 in 2022-09-10T13:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:08 smithi110 ceph-mon[97633]: pgmap v17115: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:24:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:08 smithi104 ceph-mon[97044]: osdmap e1150: 12 total, 12 up, 12 in 2022-09-10T13:24:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:08 smithi104 ceph-mon[97044]: pgmap v17115: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:24:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:09.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:09.021014+0000) 2022-09-10T13:24:10.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:09.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:09.631872+0000) 2022-09-10T13:24:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:10.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:10.021296+0000) 2022-09-10T13:24:11.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:10.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:10.632085+0000) 2022-09-10T13:24:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:11.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:11.021511+0000) 2022-09-10T13:24:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:11 smithi110 ceph-mon[97633]: pgmap v17116: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:24:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:11 smithi110 ceph-mon[97633]: osdmap e1151: 12 total, 12 up, 12 in 2022-09-10T13:24:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:11 smithi104 ceph-mon[97044]: pgmap v17116: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:24:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:11 smithi104 ceph-mon[97044]: osdmap e1151: 12 total, 12 up, 12 in 2022-09-10T13:24:12.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:11.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:11.632318+0000) 2022-09-10T13:24:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:12.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:12.021750+0000) 2022-09-10T13:24:13.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:12.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:12.632551+0000) 2022-09-10T13:24:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:13.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:13.021960+0000) 2022-09-10T13:24:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:13 smithi110 ceph-mon[97633]: pgmap v17118: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:24:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:13 smithi104 ceph-mon[97044]: pgmap v17118: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:24:14.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:13.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:13.632767+0000) 2022-09-10T13:24:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:14.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:14.022173+0000) 2022-09-10T13:24:15.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:14.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:14.633005+0000) 2022-09-10T13:24:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:15.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:15.022391+0000) 2022-09-10T13:24:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:15 smithi110 ceph-mon[97633]: pgmap v17119: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:24:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:15 smithi104 ceph-mon[97044]: pgmap v17119: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:24:16.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:15.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:15.633243+0000) 2022-09-10T13:24:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:16.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:16.022654+0000) 2022-09-10T13:24:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:16 smithi110 ceph-mon[97633]: pgmap v17120: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T13:24:17.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:16.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:16.633430+0000) 2022-09-10T13:24:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:16 smithi104 ceph-mon[97044]: pgmap v17120: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T13:24:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:17.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:17.022909+0000) 2022-09-10T13:24:17.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:24:17.248+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:24:18.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:17.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:17.633602+0000) 2022-09-10T13:24:18.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:18.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:18.023073+0000) 2022-09-10T13:24:19.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:18.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:18.633828+0000) 2022-09-10T13:24:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:19.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:19.023323+0000) 2022-09-10T13:24:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:19 smithi110 ceph-mon[97633]: pgmap v17121: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:24:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:19 smithi104 ceph-mon[97044]: pgmap v17121: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:24:20.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:19.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:19.634038+0000) 2022-09-10T13:24:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:20.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:20.023520+0000) 2022-09-10T13:24:21.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:20.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:20.634254+0000) 2022-09-10T13:24:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:21.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:21.023649+0000) 2022-09-10T13:24:21.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:21 smithi110 ceph-mon[97633]: pgmap v17122: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:24:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:21 smithi104 ceph-mon[97044]: pgmap v17122: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:24:22.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:21.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:21.634502+0000) 2022-09-10T13:24:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:22.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:22.023858+0000) 2022-09-10T13:24:23.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:22.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:22.634722+0000) 2022-09-10T13:24:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:23.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:23.024022+0000) 2022-09-10T13:24:23.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:23 smithi110 ceph-mon[97633]: pgmap v17123: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:24:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:23 smithi104 ceph-mon[97044]: pgmap v17123: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:24:24.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:23.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:23.634944+0000) 2022-09-10T13:24:24.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:24.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:24.024230+0000) 2022-09-10T13:24:25.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:24.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:24.635196+0000) 2022-09-10T13:24:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:25.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:25.024390+0000) 2022-09-10T13:24:25.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:25 smithi110 ceph-mon[97633]: pgmap v17124: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:24:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:25 smithi104 ceph-mon[97044]: pgmap v17124: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:24:26.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:25.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:25.635423+0000) 2022-09-10T13:24:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:26.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:26.024579+0000) 2022-09-10T13:24:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:26 smithi110 ceph-mon[97633]: pgmap v17125: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:24:27.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:26.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:26.635672+0000) 2022-09-10T13:24:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:26 smithi104 ceph-mon[97044]: pgmap v17125: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:24:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:27.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:27.024754+0000) 2022-09-10T13:24:28.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:27.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:27.635807+0000) 2022-09-10T13:24:28.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:28.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:28.024953+0000) 2022-09-10T13:24:29.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:28.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:28.635982+0000) 2022-09-10T13:24:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:29.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:29.025144+0000) 2022-09-10T13:24:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:29 smithi110 ceph-mon[97633]: pgmap v17126: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:29 smithi104 ceph-mon[97044]: pgmap v17126: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:30.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:29.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:29.636225+0000) 2022-09-10T13:24:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:30.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:30.025364+0000) 2022-09-10T13:24:31.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:30.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:30.636351+0000) 2022-09-10T13:24:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:31.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:31.025554+0000) 2022-09-10T13:24:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:31 smithi110 ceph-mon[97633]: pgmap v17127: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:31 smithi104 ceph-mon[97044]: pgmap v17127: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:32.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:31.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:31.636572+0000) 2022-09-10T13:24:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:32.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:32.025785+0000) 2022-09-10T13:24:32.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:24:32.262+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:24:33.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:32.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:32.636799+0000) 2022-09-10T13:24:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:33.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:33.026029+0000) 2022-09-10T13:24:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:33 smithi110 ceph-mon[97633]: pgmap v17128: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:33 smithi104 ceph-mon[97044]: pgmap v17128: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:34.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:33.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:33.637048+0000) 2022-09-10T13:24:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:34.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:34.026256+0000) 2022-09-10T13:24:35.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:34.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:34.637237+0000) 2022-09-10T13:24:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:35.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:35.026444+0000) 2022-09-10T13:24:35.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:35 smithi110 ceph-mon[97633]: pgmap v17129: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:35 smithi104 ceph-mon[97044]: pgmap v17129: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:36.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:35.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:35.637385+0000) 2022-09-10T13:24:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:36.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:36.026661+0000) 2022-09-10T13:24:36.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:36 smithi110 ceph-mon[97633]: pgmap v17130: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:37.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:36.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:36.637637+0000) 2022-09-10T13:24:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:36 smithi104 ceph-mon[97044]: pgmap v17130: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:37.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:37.026917+0000) 2022-09-10T13:24:38.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:37.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:37.637879+0000) 2022-09-10T13:24:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:38.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:38.027142+0000) 2022-09-10T13:24:39.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:38.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:38.638139+0000) 2022-09-10T13:24:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:39.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:39.027302+0000) 2022-09-10T13:24:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:39 smithi110 ceph-mon[97633]: pgmap v17131: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:39 smithi104 ceph-mon[97044]: pgmap v17131: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:40.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:39.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:39.638313+0000) 2022-09-10T13:24:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:40.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:40.027461+0000) 2022-09-10T13:24:41.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:40.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:40.638532+0000) 2022-09-10T13:24:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:41.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:41.027689+0000) 2022-09-10T13:24:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:41 smithi110 ceph-mon[97633]: pgmap v17132: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:41 smithi104 ceph-mon[97044]: pgmap v17132: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:42.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:41.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:41.638779+0000) 2022-09-10T13:24:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:42.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:42.027852+0000) 2022-09-10T13:24:43.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:42.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:42.639051+0000) 2022-09-10T13:24:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:43.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:43.028111+0000) 2022-09-10T13:24:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:43 smithi110 ceph-mon[97633]: pgmap v17133: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:43 smithi104 ceph-mon[97044]: pgmap v17133: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:44.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:43.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:43.639254+0000) 2022-09-10T13:24:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:44.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:44.028264+0000) 2022-09-10T13:24:45.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:44.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:44.639449+0000) 2022-09-10T13:24:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:45.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:45.028459+0000) 2022-09-10T13:24:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:45 smithi110 ceph-mon[97633]: pgmap v17134: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:45 smithi104 ceph-mon[97044]: pgmap v17134: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:46.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:45.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:45.639607+0000) 2022-09-10T13:24:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:46.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:46.028686+0000) 2022-09-10T13:24:46.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:46 smithi110 ceph-mon[97633]: pgmap v17135: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:24:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:24:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:24:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:24:47.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:46.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:46.639750+0000) 2022-09-10T13:24:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:46 smithi104 ceph-mon[97044]: pgmap v17135: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:24:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:24:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:24:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:24:47.279 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:47.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:47.028956+0000) 2022-09-10T13:24:47.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:24:47.278+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:24:48.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:47.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:47.639990+0000) 2022-09-10T13:24:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:48.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:48.029191+0000) 2022-09-10T13:24:49.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:48.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:48.640209+0000) 2022-09-10T13:24:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:49.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:49.029425+0000) 2022-09-10T13:24:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:49 smithi110 ceph-mon[97633]: pgmap v17136: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:49 smithi104 ceph-mon[97044]: pgmap v17136: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:50.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:49.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:49.640437+0000) 2022-09-10T13:24:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:50.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:50.029611+0000) 2022-09-10T13:24:51.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:50.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:50.640703+0000) 2022-09-10T13:24:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:51.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:51.029844+0000) 2022-09-10T13:24:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:51 smithi110 ceph-mon[97633]: pgmap v17137: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:51 smithi104 ceph-mon[97044]: pgmap v17137: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:52.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:51.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:51.640945+0000) 2022-09-10T13:24:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:52.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:52.030059+0000) 2022-09-10T13:24:53.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:52.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:52.641169+0000) 2022-09-10T13:24:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:53.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:53.030249+0000) 2022-09-10T13:24:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:53 smithi110 ceph-mon[97633]: pgmap v17138: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:53 smithi104 ceph-mon[97044]: pgmap v17138: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:54.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:53.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:53.641416+0000) 2022-09-10T13:24:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:54.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:54.030436+0000) 2022-09-10T13:24:55.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:54.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:54.641671+0000) 2022-09-10T13:24:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:55.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:55.030621+0000) 2022-09-10T13:24:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:55 smithi110 ceph-mon[97633]: pgmap v17139: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:55 smithi104 ceph-mon[97044]: pgmap v17139: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:56.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:55.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:55.641860+0000) 2022-09-10T13:24:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:56.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:56.030821+0000) 2022-09-10T13:24:56.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:56 smithi110 ceph-mon[97633]: pgmap v17140: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:57.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:56.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:56.642050+0000) 2022-09-10T13:24:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:56 smithi104 ceph-mon[97044]: pgmap v17140: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:24:57.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:57.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:57.031139+0000) 2022-09-10T13:24:58.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:57.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:57.642292+0000) 2022-09-10T13:24:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:58.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:58.031333+0000) 2022-09-10T13:24:59.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:58.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:58.642455+0000) 2022-09-10T13:24:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:24:59.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:59.031508+0000) 2022-09-10T13:24:59.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:24:59 smithi110 ceph-mon[97633]: pgmap v17141: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:24:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:24:59 smithi104 ceph-mon[97044]: pgmap v17141: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:00.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:24:59.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:24:59.642696+0000) 2022-09-10T13:25:00.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:00.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:00.031718+0000) 2022-09-10T13:25:01.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:00.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:00.642924+0000) 2022-09-10T13:25:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:01.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:01.031866+0000) 2022-09-10T13:25:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:01 smithi110 ceph-mon[97633]: pgmap v17142: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:01 smithi104 ceph-mon[97044]: pgmap v17142: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:02.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:01.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:01.643102+0000) 2022-09-10T13:25:02.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:02.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:02.032120+0000) 2022-09-10T13:25:02.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:25:02.289+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:25:03.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:02.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:02.643327+0000) 2022-09-10T13:25:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:03.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:03.032337+0000) 2022-09-10T13:25:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:03 smithi110 ceph-mon[97633]: pgmap v17143: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:03 smithi104 ceph-mon[97044]: pgmap v17143: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:04.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:03.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:03.643529+0000) 2022-09-10T13:25:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:04.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:04.032541+0000) 2022-09-10T13:25:05.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:04.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:04.643723+0000) 2022-09-10T13:25:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:05.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:05.032801+0000) 2022-09-10T13:25:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:05 smithi110 ceph-mon[97633]: pgmap v17144: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:05 smithi104 ceph-mon[97044]: pgmap v17144: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:06.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:05.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:05.643910+0000) 2022-09-10T13:25:06.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:06.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:06.033070+0000) 2022-09-10T13:25:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:06 smithi110 ceph-mon[97633]: pgmap v17145: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:25:07.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:06.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:06.644179+0000) 2022-09-10T13:25:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:06 smithi104 ceph-mon[97044]: pgmap v17145: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:25:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:07.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:07.033248+0000) 2022-09-10T13:25:08.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:07.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:07.644366+0000) 2022-09-10T13:25:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:08.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:08.033385+0000) 2022-09-10T13:25:09.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:08.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:08.644667+0000) 2022-09-10T13:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:08 smithi110 ceph-mon[97633]: osdmap e1152: 12 total, 12 up, 12 in 2022-09-10T13:25:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:08 smithi110 ceph-mon[97633]: pgmap v17147: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:25:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:08 smithi104 ceph-mon[97044]: osdmap e1152: 12 total, 12 up, 12 in 2022-09-10T13:25:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:08 smithi104 ceph-mon[97044]: pgmap v17147: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:25:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:09.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:09.033564+0000) 2022-09-10T13:25:10.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:09.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:09.644926+0000) 2022-09-10T13:25:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:10.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:10.033840+0000) 2022-09-10T13:25:11.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:10.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:10.645058+0000) 2022-09-10T13:25:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:11.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:11.034137+0000) 2022-09-10T13:25:11.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:11 smithi110 ceph-mon[97633]: pgmap v17148: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:25:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:11 smithi110 ceph-mon[97633]: osdmap e1153: 12 total, 12 up, 12 in 2022-09-10T13:25:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:11 smithi104 ceph-mon[97044]: pgmap v17148: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:25:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:11 smithi104 ceph-mon[97044]: osdmap e1153: 12 total, 12 up, 12 in 2022-09-10T13:25:12.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:11.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:11.645270+0000) 2022-09-10T13:25:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:12.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:12.034334+0000) 2022-09-10T13:25:13.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:12.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:12.645505+0000) 2022-09-10T13:25:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:13.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:13.034521+0000) 2022-09-10T13:25:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:13 smithi110 ceph-mon[97633]: pgmap v17150: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:25:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:13 smithi104 ceph-mon[97044]: pgmap v17150: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:25:14.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:13.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:13.645737+0000) 2022-09-10T13:25:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:14.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:14.034734+0000) 2022-09-10T13:25:15.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:14.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:14.645944+0000) 2022-09-10T13:25:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:15.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:15.034875+0000) 2022-09-10T13:25:15.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:15 smithi110 ceph-mon[97633]: pgmap v17151: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:25:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:15 smithi104 ceph-mon[97044]: pgmap v17151: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:25:16.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:15.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:15.646174+0000) 2022-09-10T13:25:16.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:16.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:16.035147+0000) 2022-09-10T13:25:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:16 smithi110 ceph-mon[97633]: pgmap v17152: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T13:25:17.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:16.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:16.646400+0000) 2022-09-10T13:25:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:16 smithi104 ceph-mon[97044]: pgmap v17152: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T13:25:17.301 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:17.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:17.035341+0000) 2022-09-10T13:25:17.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:25:17.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:25:18.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:17.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:17.646635+0000) 2022-09-10T13:25:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:18.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:18.035524+0000) 2022-09-10T13:25:19.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:18.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:18.646883+0000) 2022-09-10T13:25:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:19.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:19.035744+0000) 2022-09-10T13:25:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:19 smithi110 ceph-mon[97633]: pgmap v17153: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:25:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:19 smithi104 ceph-mon[97044]: pgmap v17153: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:25:20.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:19.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:19.647170+0000) 2022-09-10T13:25:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:20.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:20.035964+0000) 2022-09-10T13:25:21.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:20.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:20.647389+0000) 2022-09-10T13:25:21.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:21.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:21.036171+0000) 2022-09-10T13:25:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:21 smithi110 ceph-mon[97633]: pgmap v17154: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:25:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:21 smithi104 ceph-mon[97044]: pgmap v17154: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:25:22.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:21.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:21.647643+0000) 2022-09-10T13:25:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:22.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:22.036344+0000) 2022-09-10T13:25:23.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:22.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:22.647868+0000) 2022-09-10T13:25:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:23.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:23.036504+0000) 2022-09-10T13:25:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:23 smithi110 ceph-mon[97633]: pgmap v17155: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:25:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:23 smithi104 ceph-mon[97044]: pgmap v17155: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:25:24.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:23.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:23.648151+0000) 2022-09-10T13:25:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:24.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:24.036684+0000) 2022-09-10T13:25:25.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:24.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:24.648360+0000) 2022-09-10T13:25:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:25.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:25.036879+0000) 2022-09-10T13:25:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:25 smithi110 ceph-mon[97633]: pgmap v17156: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:25:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:25 smithi104 ceph-mon[97044]: pgmap v17156: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:25:26.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:25.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:25.648580+0000) 2022-09-10T13:25:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:26.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:26.037082+0000) 2022-09-10T13:25:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:26 smithi110 ceph-mon[97633]: pgmap v17157: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:25:27.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:26.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:26.648864+0000) 2022-09-10T13:25:27.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:26 smithi104 ceph-mon[97044]: pgmap v17157: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:25:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:27.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:27.037366+0000) 2022-09-10T13:25:28.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:27.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:27.649071+0000) 2022-09-10T13:25:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:28.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:28.037492+0000) 2022-09-10T13:25:29.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:28.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:28.649260+0000) 2022-09-10T13:25:29.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:29.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:29.037645+0000) 2022-09-10T13:25:29.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:29 smithi110 ceph-mon[97633]: pgmap v17158: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:29 smithi104 ceph-mon[97044]: pgmap v17158: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:30.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:29.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:29.649417+0000) 2022-09-10T13:25:30.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:30.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:30.037843+0000) 2022-09-10T13:25:31.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:30.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:30.649644+0000) 2022-09-10T13:25:31.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:31.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:31.038039+0000) 2022-09-10T13:25:31.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:31 smithi110 ceph-mon[97633]: pgmap v17159: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:31 smithi104 ceph-mon[97044]: pgmap v17159: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:32.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:31.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:31.649892+0000) 2022-09-10T13:25:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:32.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:32.038261+0000) 2022-09-10T13:25:32.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:25:32.316+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:25:33.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:32.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:32.650172+0000) 2022-09-10T13:25:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:33.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:33.038480+0000) 2022-09-10T13:25:33.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:33 smithi110 ceph-mon[97633]: pgmap v17160: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:33.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:33 smithi104 ceph-mon[97044]: pgmap v17160: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:34.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:33.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:33.650326+0000) 2022-09-10T13:25:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:34.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:34.038722+0000) 2022-09-10T13:25:35.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:34.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:34.650519+0000) 2022-09-10T13:25:35.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:35.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:35.038933+0000) 2022-09-10T13:25:35.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:35 smithi110 ceph-mon[97633]: pgmap v17161: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:35 smithi104 ceph-mon[97044]: pgmap v17161: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:36.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:35.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:35.650789+0000) 2022-09-10T13:25:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:36.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:36.039082+0000) 2022-09-10T13:25:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:36 smithi110 ceph-mon[97633]: pgmap v17162: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:37.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:36.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:36.651026+0000) 2022-09-10T13:25:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:36 smithi104 ceph-mon[97044]: pgmap v17162: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:37.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:37.039281+0000) 2022-09-10T13:25:38.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:37.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:37.651267+0000) 2022-09-10T13:25:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:38.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:38.039548+0000) 2022-09-10T13:25:39.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:38.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:38.651491+0000) 2022-09-10T13:25:39.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:39.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:39.039806+0000) 2022-09-10T13:25:39.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:39 smithi110 ceph-mon[97633]: pgmap v17163: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:39 smithi104 ceph-mon[97044]: pgmap v17163: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:40.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:39.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:39.651751+0000) 2022-09-10T13:25:40.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:40.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:40.040074+0000) 2022-09-10T13:25:41.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:40.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:40.652025+0000) 2022-09-10T13:25:41.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:41.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:41.040338+0000) 2022-09-10T13:25:41.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:41 smithi110 ceph-mon[97633]: pgmap v17164: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:41 smithi104 ceph-mon[97044]: pgmap v17164: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:42.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:41.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:41.652266+0000) 2022-09-10T13:25:42.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:42.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:42.040593+0000) 2022-09-10T13:25:43.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:42.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:42.652489+0000) 2022-09-10T13:25:43.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:43.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:43.040846+0000) 2022-09-10T13:25:43.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:43 smithi110 ceph-mon[97633]: pgmap v17165: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:43 smithi104 ceph-mon[97044]: pgmap v17165: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:44.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:43.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:43.652601+0000) 2022-09-10T13:25:44.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:44.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:44.041026+0000) 2022-09-10T13:25:45.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:44.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:44.652850+0000) 2022-09-10T13:25:45.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:45.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:45.041262+0000) 2022-09-10T13:25:45.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:45 smithi110 ceph-mon[97633]: pgmap v17166: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:45 smithi104 ceph-mon[97044]: pgmap v17166: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:46.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:45.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:45.653147+0000) 2022-09-10T13:25:46.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:46.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:46.041528+0000) 2022-09-10T13:25:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:46 smithi110 ceph-mon[97633]: pgmap v17167: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:25:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:25:47.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:46.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:46.653358+0000) 2022-09-10T13:25:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:46 smithi104 ceph-mon[97044]: pgmap v17167: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:25:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:25:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:25:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:25:47.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:47.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:47.041753+0000) 2022-09-10T13:25:47.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:25:47.329+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:25:48.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:47.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:47.653565+0000) 2022-09-10T13:25:48.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:48.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:48.041977+0000) 2022-09-10T13:25:49.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:48.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:48.653805+0000) 2022-09-10T13:25:49.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:49.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:49.042195+0000) 2022-09-10T13:25:49.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:49 smithi110 ceph-mon[97633]: pgmap v17168: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:49 smithi104 ceph-mon[97044]: pgmap v17168: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:50.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:49.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:49.653953+0000) 2022-09-10T13:25:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:50.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:50.042393+0000) 2022-09-10T13:25:51.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:50.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:50.654187+0000) 2022-09-10T13:25:51.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:51.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:51.042609+0000) 2022-09-10T13:25:51.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:51 smithi110 ceph-mon[97633]: pgmap v17169: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:51 smithi104 ceph-mon[97044]: pgmap v17169: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:52.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:51.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:51.654418+0000) 2022-09-10T13:25:52.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:52.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:52.042788+0000) 2022-09-10T13:25:53.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:52.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:52.654652+0000) 2022-09-10T13:25:53.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:53.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:53.043042+0000) 2022-09-10T13:25:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:53 smithi110 ceph-mon[97633]: pgmap v17170: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:53 smithi104 ceph-mon[97044]: pgmap v17170: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:54.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:53.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:53.654888+0000) 2022-09-10T13:25:54.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:54.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:54.043231+0000) 2022-09-10T13:25:55.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:54.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:54.655175+0000) 2022-09-10T13:25:55.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:55.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:55.043495+0000) 2022-09-10T13:25:55.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:55 smithi110 ceph-mon[97633]: pgmap v17171: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:55 smithi104 ceph-mon[97044]: pgmap v17171: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:56.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:55.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:55.655430+0000) 2022-09-10T13:25:56.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:56.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:56.043805+0000) 2022-09-10T13:25:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:56 smithi110 ceph-mon[97633]: pgmap v17172: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:57.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:56.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:56.655649+0000) 2022-09-10T13:25:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:56 smithi104 ceph-mon[97044]: pgmap v17172: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:25:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:57.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:57.043971+0000) 2022-09-10T13:25:58.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:57.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:57.655870+0000) 2022-09-10T13:25:58.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:58.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:58.044159+0000) 2022-09-10T13:25:59.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:58.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:58.656129+0000) 2022-09-10T13:25:59.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:25:59.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:59.044393+0000) 2022-09-10T13:25:59.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:25:59 smithi110 ceph-mon[97633]: pgmap v17173: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:25:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:25:59 smithi104 ceph-mon[97044]: pgmap v17173: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:00.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:25:59.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:25:59.656305+0000) 2022-09-10T13:26:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:00.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:00.044580+0000) 2022-09-10T13:26:01.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:00.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:00.656563+0000) 2022-09-10T13:26:01.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:01.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:01.044790+0000) 2022-09-10T13:26:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:01 smithi110 ceph-mon[97633]: pgmap v17174: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:01 smithi104 ceph-mon[97044]: pgmap v17174: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:02.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:01.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:01.656783+0000) 2022-09-10T13:26:02.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:02.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:02.045050+0000) 2022-09-10T13:26:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:26:02.342+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:26:03.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:02.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:02.657051+0000) 2022-09-10T13:26:03.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:03.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:03.045290+0000) 2022-09-10T13:26:03.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:03 smithi110 ceph-mon[97633]: pgmap v17175: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:03 smithi104 ceph-mon[97044]: pgmap v17175: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:04.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:03.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:03.657273+0000) 2022-09-10T13:26:04.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:04.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:04.045500+0000) 2022-09-10T13:26:05.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:04.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:04.657475+0000) 2022-09-10T13:26:05.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:05.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:05.045710+0000) 2022-09-10T13:26:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:05 smithi104 ceph-mon[97044]: pgmap v17176: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:05 smithi110 ceph-mon[97633]: pgmap v17176: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:06.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:05.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:05.657644+0000) 2022-09-10T13:26:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:06.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:06.045958+0000) 2022-09-10T13:26:06.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:06 smithi110 ceph-mon[97633]: pgmap v17177: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:26:07.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:06.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:06.657822+0000) 2022-09-10T13:26:07.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:06 smithi104 ceph-mon[97044]: pgmap v17177: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:26:07.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:07.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:07.046172+0000) 2022-09-10T13:26:08.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:07.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:07.658052+0000) 2022-09-10T13:26:08.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:08.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:08.046374+0000) 2022-09-10T13:26:09.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:08.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:08.658285+0000) 2022-09-10T13:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:08 smithi110 ceph-mon[97633]: osdmap e1154: 12 total, 12 up, 12 in 2022-09-10T13:26:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:08 smithi110 ceph-mon[97633]: pgmap v17179: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:26:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:08 smithi104 ceph-mon[97044]: osdmap e1154: 12 total, 12 up, 12 in 2022-09-10T13:26:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:08 smithi104 ceph-mon[97044]: pgmap v17179: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:26:09.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:09.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:09.046607+0000) 2022-09-10T13:26:10.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:09.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:09.658464+0000) 2022-09-10T13:26:10.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:10.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:10.046829+0000) 2022-09-10T13:26:11.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:10.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:10.658633+0000) 2022-09-10T13:26:11.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:11.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:11.047086+0000) 2022-09-10T13:26:11.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:11 smithi110 ceph-mon[97633]: pgmap v17180: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:26:11.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:11 smithi110 ceph-mon[97633]: osdmap e1155: 12 total, 12 up, 12 in 2022-09-10T13:26:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:11 smithi104 ceph-mon[97044]: pgmap v17180: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:26:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:11 smithi104 ceph-mon[97044]: osdmap e1155: 12 total, 12 up, 12 in 2022-09-10T13:26:12.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:11.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:11.658864+0000) 2022-09-10T13:26:12.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:12.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:12.047316+0000) 2022-09-10T13:26:13.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:12.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:12.659086+0000) 2022-09-10T13:26:13.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:13.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:13.047542+0000) 2022-09-10T13:26:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:13 smithi110 ceph-mon[97633]: pgmap v17182: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:26:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:13 smithi104 ceph-mon[97044]: pgmap v17182: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:26:14.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:13.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:13.659375+0000) 2022-09-10T13:26:14.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:14.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:14.047800+0000) 2022-09-10T13:26:15.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:14.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:14.659604+0000) 2022-09-10T13:26:15.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:15.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:15.048069+0000) 2022-09-10T13:26:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:15 smithi104 ceph-mon[97044]: pgmap v17183: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:26:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:15 smithi110 ceph-mon[97633]: pgmap v17183: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:26:16.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:15.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:15.659794+0000) 2022-09-10T13:26:16.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:16.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:16.048312+0000) 2022-09-10T13:26:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:16 smithi110 ceph-mon[97633]: pgmap v17184: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T13:26:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:16 smithi104 ceph-mon[97044]: pgmap v17184: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T13:26:17.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:16.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:16.660037+0000) 2022-09-10T13:26:17.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:17.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:17.048574+0000) 2022-09-10T13:26:17.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:26:17.347+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:26:18.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:17.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:17.660252+0000) 2022-09-10T13:26:18.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:18.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:18.048859+0000) 2022-09-10T13:26:19.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:18.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:18.660483+0000) 2022-09-10T13:26:19.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:19.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:19.049108+0000) 2022-09-10T13:26:19.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:19 smithi110 ceph-mon[97633]: pgmap v17185: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:26:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:19 smithi104 ceph-mon[97044]: pgmap v17185: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:26:20.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:19.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:19.660769+0000) 2022-09-10T13:26:20.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:20.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:20.049296+0000) 2022-09-10T13:26:21.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:20.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:20.660925+0000) 2022-09-10T13:26:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:21.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:21.049495+0000) 2022-09-10T13:26:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:21 smithi104 ceph-mon[97044]: pgmap v17186: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:26:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:21 smithi110 ceph-mon[97633]: pgmap v17186: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:26:22.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:21.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:21.661158+0000) 2022-09-10T13:26:22.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:22.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:22.049727+0000) 2022-09-10T13:26:23.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:22.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:22.661343+0000) 2022-09-10T13:26:23.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:23.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:23.049975+0000) 2022-09-10T13:26:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:23 smithi104 ceph-mon[97044]: pgmap v17187: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:26:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:23 smithi110 ceph-mon[97633]: pgmap v17187: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:26:24.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:23.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:23.661528+0000) 2022-09-10T13:26:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:24.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:24.050213+0000) 2022-09-10T13:26:25.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:24.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:24.661768+0000) 2022-09-10T13:26:25.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:25.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:25.050453+0000) 2022-09-10T13:26:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:25 smithi104 ceph-mon[97044]: pgmap v17188: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:26:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:25 smithi110 ceph-mon[97633]: pgmap v17188: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:26:26.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:25.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:25.662003+0000) 2022-09-10T13:26:26.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:26.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:26.050620+0000) 2022-09-10T13:26:26.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:26 smithi110 ceph-mon[97633]: pgmap v17189: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:26:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:26 smithi104 ceph-mon[97044]: pgmap v17189: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:26:27.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:26.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:26.662209+0000) 2022-09-10T13:26:27.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:27.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:27.050757+0000) 2022-09-10T13:26:28.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:27.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:27.662430+0000) 2022-09-10T13:26:28.315 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:28.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:28.051042+0000) 2022-09-10T13:26:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:28.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:28.662693+0000) 2022-09-10T13:26:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:29.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:29.051267+0000) 2022-09-10T13:26:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:29 smithi104 ceph-mon[97044]: pgmap v17190: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:29 smithi110 ceph-mon[97633]: pgmap v17190: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:30.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:29.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:29.662945+0000) 2022-09-10T13:26:30.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:30.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:30.051489+0000) 2022-09-10T13:26:31.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:30.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:30.663209+0000) 2022-09-10T13:26:31.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:31.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:31.051694+0000) 2022-09-10T13:26:31.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:31 smithi104 ceph-mon[97044]: pgmap v17191: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:31 smithi110 ceph-mon[97633]: pgmap v17191: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:32.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:31.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:31.663408+0000) 2022-09-10T13:26:32.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:32.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:32.051916+0000) 2022-09-10T13:26:32.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:26:32.348+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:26:33.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:32.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:32.663618+0000) 2022-09-10T13:26:33.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:33.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:33.052197+0000) 2022-09-10T13:26:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:33 smithi104 ceph-mon[97044]: pgmap v17192: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:33 smithi110 ceph-mon[97633]: pgmap v17192: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:34.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:33.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:33.663855+0000) 2022-09-10T13:26:34.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:34.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:34.052396+0000) 2022-09-10T13:26:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:34.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:34.664133+0000) 2022-09-10T13:26:35.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:35.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:35.052611+0000) 2022-09-10T13:26:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:35 smithi104 ceph-mon[97044]: pgmap v17193: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:35 smithi110 ceph-mon[97633]: pgmap v17193: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:36.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:35.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:35.664399+0000) 2022-09-10T13:26:36.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:36.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:36.052801+0000) 2022-09-10T13:26:36.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:36 smithi110 ceph-mon[97633]: pgmap v17194: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:36 smithi104 ceph-mon[97044]: pgmap v17194: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:37.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:36.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:36.664646+0000) 2022-09-10T13:26:37.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:37.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:37.052988+0000) 2022-09-10T13:26:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:37.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:37.664833+0000) 2022-09-10T13:26:38.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:38.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:38.053201+0000) 2022-09-10T13:26:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:38.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:38.665157+0000) 2022-09-10T13:26:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:39 smithi104 ceph-mon[97044]: pgmap v17195: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:39.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:39.053431+0000) 2022-09-10T13:26:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:39 smithi110 ceph-mon[97633]: pgmap v17195: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:40.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:39.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:39.665389+0000) 2022-09-10T13:26:40.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:40.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:40.053681+0000) 2022-09-10T13:26:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:40.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:40.665587+0000) 2022-09-10T13:26:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:41 smithi104 ceph-mon[97044]: pgmap v17196: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:41.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:41.053870+0000) 2022-09-10T13:26:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:41 smithi110 ceph-mon[97633]: pgmap v17196: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:41.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:41.665847+0000) 2022-09-10T13:26:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:42.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:42.054018+0000) 2022-09-10T13:26:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:42.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:42.666123+0000) 2022-09-10T13:26:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:43 smithi104 ceph-mon[97044]: pgmap v17197: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:43.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:43.054171+0000) 2022-09-10T13:26:43.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:43 smithi110 ceph-mon[97633]: pgmap v17197: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:43.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:43.666335+0000) 2022-09-10T13:26:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:44.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:44.054323+0000) 2022-09-10T13:26:45.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:44.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:44.666591+0000) 2022-09-10T13:26:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:45 smithi104 ceph-mon[97044]: pgmap v17198: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:45.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:45.054460+0000) 2022-09-10T13:26:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:45 smithi110 ceph-mon[97633]: pgmap v17198: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:46.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:45.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:45.666803+0000) 2022-09-10T13:26:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:46.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:46.054624+0000) 2022-09-10T13:26:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:46 smithi110 ceph-mon[97633]: pgmap v17199: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:26:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:26:46.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:26:46.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:26:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:46 smithi104 ceph-mon[97044]: pgmap v17199: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:26:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:26:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:26:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:26:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:46.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:46.667050+0000) 2022-09-10T13:26:47.364 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:47.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:47.054845+0000) 2022-09-10T13:26:47.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:26:47.362+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:26:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:47.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:47.667276+0000) 2022-09-10T13:26:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:48.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:48.055059+0000) 2022-09-10T13:26:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:48.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:48.667425+0000) 2022-09-10T13:26:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:49 smithi104 ceph-mon[97044]: pgmap v17200: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:49.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:49.055270+0000) 2022-09-10T13:26:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:49 smithi110 ceph-mon[97633]: pgmap v17200: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:50.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:49.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:49.667617+0000) 2022-09-10T13:26:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:50.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:50.055495+0000) 2022-09-10T13:26:51.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:50.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:50.667817+0000) 2022-09-10T13:26:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:51 smithi104 ceph-mon[97044]: pgmap v17201: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:51.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:51.055631+0000) 2022-09-10T13:26:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:51 smithi110 ceph-mon[97633]: pgmap v17201: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:51.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:51.668081+0000) 2022-09-10T13:26:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:52.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:52.055812+0000) 2022-09-10T13:26:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:52.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:52.668307+0000) 2022-09-10T13:26:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:53 smithi104 ceph-mon[97044]: pgmap v17202: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:53.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:53.056008+0000) 2022-09-10T13:26:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:53 smithi110 ceph-mon[97633]: pgmap v17202: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:53.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:53.668501+0000) 2022-09-10T13:26:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:54.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:54.056157+0000) 2022-09-10T13:26:54.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:54.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:54.668728+0000) 2022-09-10T13:26:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:55 smithi104 ceph-mon[97044]: pgmap v17203: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:55.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:55.056356+0000) 2022-09-10T13:26:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:55 smithi110 ceph-mon[97633]: pgmap v17203: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:55.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:55.668932+0000) 2022-09-10T13:26:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:56.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:56.056567+0000) 2022-09-10T13:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:56 smithi110 ceph-mon[97633]: pgmap v17204: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:56 smithi104 ceph-mon[97044]: pgmap v17204: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:26:57.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:56.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:56.669205+0000) 2022-09-10T13:26:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:57.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:57.056788+0000) 2022-09-10T13:26:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:57.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:57.669353+0000) 2022-09-10T13:26:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:58.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:58.057042+0000) 2022-09-10T13:26:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:58.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:58.669549+0000) 2022-09-10T13:26:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:26:59 smithi104 ceph-mon[97044]: pgmap v17205: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:26:59.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:26:59.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:59.057240+0000) 2022-09-10T13:26:59.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:26:59 smithi110 ceph-mon[97633]: pgmap v17205: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:26:59.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:26:59.669783+0000) 2022-09-10T13:27:00.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:00.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:00.057391+0000) 2022-09-10T13:27:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:00.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:00.669972+0000) 2022-09-10T13:27:01.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:01 smithi104 ceph-mon[97044]: pgmap v17206: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:01.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:01.057532+0000) 2022-09-10T13:27:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:01 smithi110 ceph-mon[97633]: pgmap v17206: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:01.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:01.670181+0000) 2022-09-10T13:27:02.383 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:02.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:02.057697+0000) 2022-09-10T13:27:02.680 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:27:02.377+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:27:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:02.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:02.670375+0000) 2022-09-10T13:27:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:03 smithi104 ceph-mon[97044]: pgmap v17207: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:03.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:03.057916+0000) 2022-09-10T13:27:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:03 smithi110 ceph-mon[97633]: pgmap v17207: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:03.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:03.670651+0000) 2022-09-10T13:27:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:04.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:04.058164+0000) 2022-09-10T13:27:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:04.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:04.670918+0000) 2022-09-10T13:27:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:05 smithi104 ceph-mon[97044]: pgmap v17208: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:05.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:05.058364+0000) 2022-09-10T13:27:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:05 smithi110 ceph-mon[97633]: pgmap v17208: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:05.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:05.671228+0000) 2022-09-10T13:27:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:06.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:06.058549+0000) 2022-09-10T13:27:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:06 smithi110 ceph-mon[97633]: pgmap v17209: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:27:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:06 smithi104 ceph-mon[97044]: pgmap v17209: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:27:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:06.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:06.671451+0000) 2022-09-10T13:27:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:07.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:07.058726+0000) 2022-09-10T13:27:08.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:07.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:07.671615+0000) 2022-09-10T13:27:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:08.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:08.058942+0000) 2022-09-10T13:27:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:08.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:08.671806+0000) 2022-09-10T13:27:09.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:08 smithi110 ceph-mon[97633]: osdmap e1156: 12 total, 12 up, 12 in 2022-09-10T13:27:09.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:08 smithi110 ceph-mon[97633]: pgmap v17211: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:27:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:08 smithi104 ceph-mon[97044]: osdmap e1156: 12 total, 12 up, 12 in 2022-09-10T13:27:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:08 smithi104 ceph-mon[97044]: pgmap v17211: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:27:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:09.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:09.059188+0000) 2022-09-10T13:27:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:09.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:09.672028+0000) 2022-09-10T13:27:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:10.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:10.059370+0000) 2022-09-10T13:27:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:10.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:10.672298+0000) 2022-09-10T13:27:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:11 smithi104 ceph-mon[97044]: pgmap v17212: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:27:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:11 smithi104 ceph-mon[97044]: osdmap e1157: 12 total, 12 up, 12 in 2022-09-10T13:27:11.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:11.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:11.059513+0000) 2022-09-10T13:27:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:11 smithi110 ceph-mon[97633]: pgmap v17212: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:27:11.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:11 smithi110 ceph-mon[97633]: osdmap e1157: 12 total, 12 up, 12 in 2022-09-10T13:27:12.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:11.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:11.672544+0000) 2022-09-10T13:27:12.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:12.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:12.059690+0000) 2022-09-10T13:27:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:12.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:12.672805+0000) 2022-09-10T13:27:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:13 smithi104 ceph-mon[97044]: pgmap v17214: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:27:13.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:13.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:13.059930+0000) 2022-09-10T13:27:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:13 smithi110 ceph-mon[97633]: pgmap v17214: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:27:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:13.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:13.673043+0000) 2022-09-10T13:27:14.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:14.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:14.060165+0000) 2022-09-10T13:27:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:14.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:14.673314+0000) 2022-09-10T13:27:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:15 smithi104 ceph-mon[97044]: pgmap v17215: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:27:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:15.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:15.060291+0000) 2022-09-10T13:27:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:15 smithi110 ceph-mon[97633]: pgmap v17215: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:27:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:15.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:15.673552+0000) 2022-09-10T13:27:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:16.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:16.060458+0000) 2022-09-10T13:27:16.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:16 smithi110 ceph-mon[97633]: pgmap v17216: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T13:27:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:16 smithi104 ceph-mon[97044]: pgmap v17216: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T13:27:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:16.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:16.673776+0000) 2022-09-10T13:27:17.392 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:17.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:17.060638+0000) 2022-09-10T13:27:17.674 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:27:17.391+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:27:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:17.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:17.674000+0000) 2022-09-10T13:27:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:18.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:18.060854+0000) 2022-09-10T13:27:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:18.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:18.674265+0000) 2022-09-10T13:27:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:19 smithi104 ceph-mon[97044]: pgmap v17217: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:27:19.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:19.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:19.061056+0000) 2022-09-10T13:27:19.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:19 smithi110 ceph-mon[97633]: pgmap v17217: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:27:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:19.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:19.674482+0000) 2022-09-10T13:27:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:20.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:20.061237+0000) 2022-09-10T13:27:21.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:20.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:20.674675+0000) 2022-09-10T13:27:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:21 smithi104 ceph-mon[97044]: pgmap v17218: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:27:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:21.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:21.061459+0000) 2022-09-10T13:27:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:21 smithi110 ceph-mon[97633]: pgmap v17218: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:27:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:21.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:21.674867+0000) 2022-09-10T13:27:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:22.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:22.061647+0000) 2022-09-10T13:27:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:22.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:22.675141+0000) 2022-09-10T13:27:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:23 smithi104 ceph-mon[97044]: pgmap v17219: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:27:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:23.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:23.061847+0000) 2022-09-10T13:27:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:23 smithi110 ceph-mon[97633]: pgmap v17219: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:27:24.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:23.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:23.675353+0000) 2022-09-10T13:27:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:24.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:24.062071+0000) 2022-09-10T13:27:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:24.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:24.675579+0000) 2022-09-10T13:27:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:25 smithi104 ceph-mon[97044]: pgmap v17220: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:27:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:25.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:25.062278+0000) 2022-09-10T13:27:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:25 smithi110 ceph-mon[97633]: pgmap v17220: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:27:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:25.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:25.675836+0000) 2022-09-10T13:27:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:26.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:26.062466+0000) 2022-09-10T13:27:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:26 smithi110 ceph-mon[97633]: pgmap v17221: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:27:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:26 smithi104 ceph-mon[97044]: pgmap v17221: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:27:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:26.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:26.676122+0000) 2022-09-10T13:27:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:27.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:27.062609+0000) 2022-09-10T13:27:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:27.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:27.676340+0000) 2022-09-10T13:27:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:28.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:28.062846+0000) 2022-09-10T13:27:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:28.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:28.676603+0000) 2022-09-10T13:27:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:29 smithi104 ceph-mon[97044]: pgmap v17222: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:29.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:29.063063+0000) 2022-09-10T13:27:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:29 smithi110 ceph-mon[97633]: pgmap v17222: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:29.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:29.676887+0000) 2022-09-10T13:27:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:30.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:30.063267+0000) 2022-09-10T13:27:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:30.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:30.677166+0000) 2022-09-10T13:27:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:31 smithi104 ceph-mon[97044]: pgmap v17223: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:31.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:31.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:31.063396+0000) 2022-09-10T13:27:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:31 smithi110 ceph-mon[97633]: pgmap v17223: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:31.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:31.677333+0000) 2022-09-10T13:27:32.394 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:32.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:32.063535+0000) 2022-09-10T13:27:32.677 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:27:32.392+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:27:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:32.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:32.677601+0000) 2022-09-10T13:27:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:33 smithi104 ceph-mon[97044]: pgmap v17224: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:33.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:33.063708+0000) 2022-09-10T13:27:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:33 smithi110 ceph-mon[97633]: pgmap v17224: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:33.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:33.677779+0000) 2022-09-10T13:27:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:34.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:34.063936+0000) 2022-09-10T13:27:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:34.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:34.678010+0000) 2022-09-10T13:27:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:35 smithi104 ceph-mon[97044]: pgmap v17225: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:35.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:35.064156+0000) 2022-09-10T13:27:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:35 smithi110 ceph-mon[97633]: pgmap v17225: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:35.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:35.678280+0000) 2022-09-10T13:27:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:36.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:36.064319+0000) 2022-09-10T13:27:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:36 smithi104 ceph-mon[97044]: pgmap v17226: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:36.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:36.678505+0000) 2022-09-10T13:27:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:36 smithi110 ceph-mon[97633]: pgmap v17226: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:37.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:37.064536+0000) 2022-09-10T13:27:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:37.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:37.678760+0000) 2022-09-10T13:27:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:38.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:38.064808+0000) 2022-09-10T13:27:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:38.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:38.678990+0000) 2022-09-10T13:27:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:39 smithi104 ceph-mon[97044]: pgmap v17227: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:39.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:39.065043+0000) 2022-09-10T13:27:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:39 smithi110 ceph-mon[97633]: pgmap v17227: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:39.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:39.679227+0000) 2022-09-10T13:27:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:40.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:40.065274+0000) 2022-09-10T13:27:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:40.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:40.679466+0000) 2022-09-10T13:27:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:41 smithi104 ceph-mon[97044]: pgmap v17228: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:41.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:41.065458+0000) 2022-09-10T13:27:41.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:41 smithi110 ceph-mon[97633]: pgmap v17228: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:41.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:41.679653+0000) 2022-09-10T13:27:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:42.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:42.065705+0000) 2022-09-10T13:27:43.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:42.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:42.679896+0000) 2022-09-10T13:27:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:43 smithi104 ceph-mon[97044]: pgmap v17229: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:43.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:43.065924+0000) 2022-09-10T13:27:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:43 smithi110 ceph-mon[97633]: pgmap v17229: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:43.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:43.680150+0000) 2022-09-10T13:27:44.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:44.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:44.066138+0000) 2022-09-10T13:27:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:44.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:44.680338+0000) 2022-09-10T13:27:45.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:45 smithi104 ceph-mon[97044]: pgmap v17230: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:45.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:45.066321+0000) 2022-09-10T13:27:45.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:45 smithi110 ceph-mon[97633]: pgmap v17230: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:45.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:45.680624+0000) 2022-09-10T13:27:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:46.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:46.066509+0000) 2022-09-10T13:27:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:46 smithi104 ceph-mon[97044]: pgmap v17231: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:27:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:27:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:27:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:27:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:46.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:46.680859+0000) 2022-09-10T13:27:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:46 smithi110 ceph-mon[97633]: pgmap v17231: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:27:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:27:47.402 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:47.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:47.066667+0000) 2022-09-10T13:27:47.681 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:27:47.401+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:27:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:47.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:47.681000+0000) 2022-09-10T13:27:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:48.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:48.066835+0000) 2022-09-10T13:27:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:48.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:48.681219+0000) 2022-09-10T13:27:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:49 smithi104 ceph-mon[97044]: pgmap v17232: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:49.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:49.067016+0000) 2022-09-10T13:27:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:49 smithi110 ceph-mon[97633]: pgmap v17232: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:50.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:49.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:49.681429+0000) 2022-09-10T13:27:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:50.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:50.067281+0000) 2022-09-10T13:27:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:50.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:50.681598+0000) 2022-09-10T13:27:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:51 smithi104 ceph-mon[97044]: pgmap v17233: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:51.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:51.067441+0000) 2022-09-10T13:27:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:51 smithi110 ceph-mon[97633]: pgmap v17233: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:51.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:51.681820+0000) 2022-09-10T13:27:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:52.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:52.067626+0000) 2022-09-10T13:27:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:52.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:52.682065+0000) 2022-09-10T13:27:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:53 smithi104 ceph-mon[97044]: pgmap v17234: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:53.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:53.067938+0000) 2022-09-10T13:27:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:53 smithi110 ceph-mon[97633]: pgmap v17234: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:53.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:53.682267+0000) 2022-09-10T13:27:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:54.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:54.068221+0000) 2022-09-10T13:27:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:54.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:54.682450+0000) 2022-09-10T13:27:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:55 smithi104 ceph-mon[97044]: pgmap v17235: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:55.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:55.068418+0000) 2022-09-10T13:27:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:55 smithi110 ceph-mon[97633]: pgmap v17235: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:55.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:55.682710+0000) 2022-09-10T13:27:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:56.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:56.068649+0000) 2022-09-10T13:27:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:56 smithi104 ceph-mon[97044]: pgmap v17236: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:56.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:56.682959+0000) 2022-09-10T13:27:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:56 smithi110 ceph-mon[97633]: pgmap v17236: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:27:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:57.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:57.068886+0000) 2022-09-10T13:27:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:57.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:57.683126+0000) 2022-09-10T13:27:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:58.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:58.069086+0000) 2022-09-10T13:27:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:58.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:58.683321+0000) 2022-09-10T13:27:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:27:59 smithi104 ceph-mon[97044]: pgmap v17237: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:27:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:27:59.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:59.069323+0000) 2022-09-10T13:27:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:27:59 smithi110 ceph-mon[97633]: pgmap v17237: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:27:59.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:27:59.683546+0000) 2022-09-10T13:28:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:00.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:00.069570+0000) 2022-09-10T13:28:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:00.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:00.683751+0000) 2022-09-10T13:28:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:01 smithi104 ceph-mon[97044]: pgmap v17238: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:01.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:01.069834+0000) 2022-09-10T13:28:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:01 smithi110 ceph-mon[97633]: pgmap v17238: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:01.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:01.683980+0000) 2022-09-10T13:28:02.418 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:02.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:02.070010+0000) 2022-09-10T13:28:02.684 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:28:02.417+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:28:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:02.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:02.684194+0000) 2022-09-10T13:28:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:03 smithi104 ceph-mon[97044]: pgmap v17239: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:03.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:03.070264+0000) 2022-09-10T13:28:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:03 smithi110 ceph-mon[97633]: pgmap v17239: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:03.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:03.684446+0000) 2022-09-10T13:28:04.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:04.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:04.070476+0000) 2022-09-10T13:28:04.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:04.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:04.684677+0000) 2022-09-10T13:28:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:05 smithi104 ceph-mon[97044]: pgmap v17240: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:05.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:05.070679+0000) 2022-09-10T13:28:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:05 smithi110 ceph-mon[97633]: pgmap v17240: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:05.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:05.684924+0000) 2022-09-10T13:28:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:06.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:06.070899+0000) 2022-09-10T13:28:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:06 smithi104 ceph-mon[97044]: pgmap v17241: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:28:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:06.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:06.685204+0000) 2022-09-10T13:28:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:06 smithi110 ceph-mon[97633]: pgmap v17241: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:28:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:07.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:07.071192+0000) 2022-09-10T13:28:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:07.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:07.685409+0000) 2022-09-10T13:28:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:08.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:08.071337+0000) 2022-09-10T13:28:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:08.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:08.685637+0000) 2022-09-10T13:28:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:08 smithi110 ceph-mon[97633]: osdmap e1158: 12 total, 12 up, 12 in 2022-09-10T13:28:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:08 smithi110 ceph-mon[97633]: pgmap v17243: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:28:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:08 smithi104 ceph-mon[97044]: osdmap e1158: 12 total, 12 up, 12 in 2022-09-10T13:28:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:08 smithi104 ceph-mon[97044]: pgmap v17243: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:28:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:09.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:09.071481+0000) 2022-09-10T13:28:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:09.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:09.685861+0000) 2022-09-10T13:28:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:10.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:10.071726+0000) 2022-09-10T13:28:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:10.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:10.686092+0000) 2022-09-10T13:28:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:11 smithi104 ceph-mon[97044]: pgmap v17244: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:28:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:11 smithi104 ceph-mon[97044]: osdmap e1159: 12 total, 12 up, 12 in 2022-09-10T13:28:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:11.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:11.072000+0000) 2022-09-10T13:28:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:11 smithi110 ceph-mon[97633]: pgmap v17244: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:28:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:11 smithi110 ceph-mon[97633]: osdmap e1159: 12 total, 12 up, 12 in 2022-09-10T13:28:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:11.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:11.686337+0000) 2022-09-10T13:28:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:12.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:12.072184+0000) 2022-09-10T13:28:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:12.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:12.686541+0000) 2022-09-10T13:28:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:13 smithi104 ceph-mon[97044]: pgmap v17246: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:28:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:13.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:13.072389+0000) 2022-09-10T13:28:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:13 smithi110 ceph-mon[97633]: pgmap v17246: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:28:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:13.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:13.686714+0000) 2022-09-10T13:28:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:14.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:14.072593+0000) 2022-09-10T13:28:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:14.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:14.686879+0000) 2022-09-10T13:28:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:15 smithi104 ceph-mon[97044]: pgmap v17247: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:28:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:15.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:15.072791+0000) 2022-09-10T13:28:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:15 smithi110 ceph-mon[97633]: pgmap v17247: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:28:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:15.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:15.687089+0000) 2022-09-10T13:28:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:16.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:16.073049+0000) 2022-09-10T13:28:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:16 smithi104 ceph-mon[97044]: pgmap v17248: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T13:28:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:16.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:16.687352+0000) 2022-09-10T13:28:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:16 smithi110 ceph-mon[97633]: pgmap v17248: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T13:28:17.433 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:17.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:17.073269+0000) 2022-09-10T13:28:17.687 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:28:17.432+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:28:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:17.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:17.687652+0000) 2022-09-10T13:28:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:18.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:18.073458+0000) 2022-09-10T13:28:19.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:18.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:18.687880+0000) 2022-09-10T13:28:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:19 smithi104 ceph-mon[97044]: pgmap v17249: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:28:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:19.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:19.073619+0000) 2022-09-10T13:28:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:19 smithi110 ceph-mon[97633]: pgmap v17249: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:28:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:19.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:19.688139+0000) 2022-09-10T13:28:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:20.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:20.073819+0000) 2022-09-10T13:28:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:20.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:20.688284+0000) 2022-09-10T13:28:21.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:21 smithi104 ceph-mon[97044]: pgmap v17250: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:28:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:21.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:21.074052+0000) 2022-09-10T13:28:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:21 smithi110 ceph-mon[97633]: pgmap v17250: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:28:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:21.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:21.688475+0000) 2022-09-10T13:28:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:22.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:22.074325+0000) 2022-09-10T13:28:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:22.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:22.688714+0000) 2022-09-10T13:28:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:23 smithi104 ceph-mon[97044]: pgmap v17251: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:28:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:23.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:23.074574+0000) 2022-09-10T13:28:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:23 smithi110 ceph-mon[97633]: pgmap v17251: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:28:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:23.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:23.688988+0000) 2022-09-10T13:28:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:24.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:24.074807+0000) 2022-09-10T13:28:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:24.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:24.689212+0000) 2022-09-10T13:28:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:25 smithi104 ceph-mon[97044]: pgmap v17252: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:28:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:25.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:25.075028+0000) 2022-09-10T13:28:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:25 smithi110 ceph-mon[97633]: pgmap v17252: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:28:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:25.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:25.689401+0000) 2022-09-10T13:28:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:26.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:26.075242+0000) 2022-09-10T13:28:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:26 smithi104 ceph-mon[97044]: pgmap v17253: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:28:27.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:26.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:26.689614+0000) 2022-09-10T13:28:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:26 smithi110 ceph-mon[97633]: pgmap v17253: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:28:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:27.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:27.075400+0000) 2022-09-10T13:28:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:27.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:27.689841+0000) 2022-09-10T13:28:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:28.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:28.075613+0000) 2022-09-10T13:28:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:28.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:28.690034+0000) 2022-09-10T13:28:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:29 smithi104 ceph-mon[97044]: pgmap v17254: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:29.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:29.075857+0000) 2022-09-10T13:28:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:29 smithi110 ceph-mon[97633]: pgmap v17254: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:29.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:29.690268+0000) 2022-09-10T13:28:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:30.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:30.076053+0000) 2022-09-10T13:28:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:30.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:30.690505+0000) 2022-09-10T13:28:31.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:31 smithi104 ceph-mon[97044]: pgmap v17255: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:31.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:31.076286+0000) 2022-09-10T13:28:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:31 smithi110 ceph-mon[97633]: pgmap v17255: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:31.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:31.690782+0000) 2022-09-10T13:28:32.449 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:32.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:32.076502+0000) 2022-09-10T13:28:32.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:28:32.448+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:28:32.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:32.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:32.691040+0000) 2022-09-10T13:28:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:33 smithi104 ceph-mon[97044]: pgmap v17256: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:33.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:33.076747+0000) 2022-09-10T13:28:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:33 smithi110 ceph-mon[97633]: pgmap v17256: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:33.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:33.691287+0000) 2022-09-10T13:28:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:34.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:34.076967+0000) 2022-09-10T13:28:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:34.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:34.691549+0000) 2022-09-10T13:28:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:35 smithi104 ceph-mon[97044]: pgmap v17257: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:35.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:35.077199+0000) 2022-09-10T13:28:35.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:35 smithi110 ceph-mon[97633]: pgmap v17257: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:35.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:35.691803+0000) 2022-09-10T13:28:36.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:36.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:36.077406+0000) 2022-09-10T13:28:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:36 smithi104 ceph-mon[97044]: pgmap v17258: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:36.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:36.692047+0000) 2022-09-10T13:28:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:36 smithi110 ceph-mon[97633]: pgmap v17258: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:37.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:37.077615+0000) 2022-09-10T13:28:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:37.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:37.692162+0000) 2022-09-10T13:28:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:38.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:38.077856+0000) 2022-09-10T13:28:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:38.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:38.692328+0000) 2022-09-10T13:28:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:39 smithi104 ceph-mon[97044]: pgmap v17259: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:39.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:39.078071+0000) 2022-09-10T13:28:39.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:39 smithi110 ceph-mon[97633]: pgmap v17259: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:40.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:39.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:39.692547+0000) 2022-09-10T13:28:40.565 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:40.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:40.078298+0000) 2022-09-10T13:28:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:40.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:40.692763+0000) 2022-09-10T13:28:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:41 smithi104 ceph-mon[97044]: pgmap v17260: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:41.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:41.078471+0000) 2022-09-10T13:28:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:41 smithi110 ceph-mon[97633]: pgmap v17260: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:28:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:41.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:41.692988+0000) 2022-09-10T13:28:42.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:42.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:42.078647+0000) 2022-09-10T13:28:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:42.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:42.693249+0000) 2022-09-10T13:28:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:43 smithi104 ceph-mon[97044]: pgmap v17261: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:43.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:43.078858+0000) 2022-09-10T13:28:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:43 smithi110 ceph-mon[97633]: pgmap v17261: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:43.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:43.693441+0000) 2022-09-10T13:28:44.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:44.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:44.079071+0000) 2022-09-10T13:28:45.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:44.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:44.693578+0000) 2022-09-10T13:28:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:45.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:45.079265+0000) 2022-09-10T13:28:45.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:45 smithi104 ceph-mon[97044]: pgmap v17262: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:45.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:45 smithi110 ceph-mon[97633]: pgmap v17262: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:45.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:45.693712+0000) 2022-09-10T13:28:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:46.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:46.079456+0000) 2022-09-10T13:28:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:46 smithi104 ceph-mon[97044]: pgmap v17263: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:28:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:28:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:28:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:28:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:28:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:46.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:46.693966+0000) 2022-09-10T13:28:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:46 smithi110 ceph-mon[97633]: pgmap v17263: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:28:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:28:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:28:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:28:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:28:47.460 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:47.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:47.079640+0000) 2022-09-10T13:28:47.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:28:47.457+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:28:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:47.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:47.694163+0000) 2022-09-10T13:28:48.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:48.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:48.079810+0000) 2022-09-10T13:28:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:48.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:48.694401+0000) 2022-09-10T13:28:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:49 smithi104 ceph-mon[97044]: pgmap v17264: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:28:49.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:49.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:49.079987+0000) 2022-09-10T13:28:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:49 smithi110 ceph-mon[97633]: pgmap v17264: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:28:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:49.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:49.694623+0000) 2022-09-10T13:28:50.567 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:50.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:50.080203+0000) 2022-09-10T13:28:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:50.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:50.694827+0000) 2022-09-10T13:28:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:51 smithi104 ceph-mon[97044]: pgmap v17265: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:28:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:51.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:51.080434+0000) 2022-09-10T13:28:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:51 smithi110 ceph-mon[97633]: pgmap v17265: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:28:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:51.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:51.694976+0000) 2022-09-10T13:28:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:52.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:52.080664+0000) 2022-09-10T13:28:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:52.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:52.695166+0000) 2022-09-10T13:28:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:53 smithi104 ceph-mon[97044]: pgmap v17266: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:28:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:53.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:53.080809+0000) 2022-09-10T13:28:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:53 smithi110 ceph-mon[97633]: pgmap v17266: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:28:54.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:53.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:53.695345+0000) 2022-09-10T13:28:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:54.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:54.081024+0000) 2022-09-10T13:28:55.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:54.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:54.695529+0000) 2022-09-10T13:28:55.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:55 smithi104 ceph-mon[97044]: pgmap v17267: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:28:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:55.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:55.081288+0000) 2022-09-10T13:28:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:55 smithi110 ceph-mon[97633]: pgmap v17267: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:28:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:55.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:55.695728+0000) 2022-09-10T13:28:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:56.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:56.081462+0000) 2022-09-10T13:28:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:56 smithi104 ceph-mon[97044]: pgmap v17268: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:28:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:56.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:56.695970+0000) 2022-09-10T13:28:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:56 smithi110 ceph-mon[97633]: pgmap v17268: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:28:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:57.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:57.081705+0000) 2022-09-10T13:28:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:57.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:57.696244+0000) 2022-09-10T13:28:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:58.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:58.081982+0000) 2022-09-10T13:28:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:58.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:58.696472+0000) 2022-09-10T13:28:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:28:59 smithi104 ceph-mon[97044]: pgmap v17269: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:28:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:28:59.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:59.082226+0000) 2022-09-10T13:28:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:28:59 smithi110 ceph-mon[97633]: pgmap v17269: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:28:59.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:28:59.696690+0000) 2022-09-10T13:29:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:00.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:00.082406+0000) 2022-09-10T13:29:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:00.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:00.696849+0000) 2022-09-10T13:29:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:01 smithi104 ceph-mon[97044]: pgmap v17270: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:01.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:01.082596+0000) 2022-09-10T13:29:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:01 smithi110 ceph-mon[97633]: pgmap v17270: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:01.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:01.697052+0000) 2022-09-10T13:29:02.460 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:02.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:02.082808+0000) 2022-09-10T13:29:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:02.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:02.697301+0000) 2022-09-10T13:29:02.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:29:02.459+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:29:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:03 smithi104 ceph-mon[97044]: pgmap v17271: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:03.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:03.082998+0000) 2022-09-10T13:29:03.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:03 smithi110 ceph-mon[97633]: pgmap v17271: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:03.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:03.697532+0000) 2022-09-10T13:29:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:04.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:04.083173+0000) 2022-09-10T13:29:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:04.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:04.697782+0000) 2022-09-10T13:29:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:05 smithi104 ceph-mon[97044]: pgmap v17272: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:05.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:05.083362+0000) 2022-09-10T13:29:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:05 smithi110 ceph-mon[97633]: pgmap v17272: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:05.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:05.698005+0000) 2022-09-10T13:29:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:06.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:06.083489+0000) 2022-09-10T13:29:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:06 smithi104 ceph-mon[97044]: pgmap v17273: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:29:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:06.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:06.698215+0000) 2022-09-10T13:29:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:06 smithi110 ceph-mon[97633]: pgmap v17273: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:29:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:07.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:07.083698+0000) 2022-09-10T13:29:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:07.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:07.698492+0000) 2022-09-10T13:29:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:08.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:08.083910+0000) 2022-09-10T13:29:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:08.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:08.698719+0000) 2022-09-10T13:29:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:08 smithi110 ceph-mon[97633]: osdmap e1160: 12 total, 12 up, 12 in 2022-09-10T13:29:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:08 smithi110 ceph-mon[97633]: pgmap v17275: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:29:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:08 smithi104 ceph-mon[97044]: osdmap e1160: 12 total, 12 up, 12 in 2022-09-10T13:29:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:08 smithi104 ceph-mon[97044]: pgmap v17275: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:29:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:09.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:09.084172+0000) 2022-09-10T13:29:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:09.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:09.698832+0000) 2022-09-10T13:29:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:10.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:10.084388+0000) 2022-09-10T13:29:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:10.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:10.700208+0000) 2022-09-10T13:29:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:11 smithi104 ceph-mon[97044]: pgmap v17276: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:29:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:11 smithi104 ceph-mon[97044]: osdmap e1161: 12 total, 12 up, 12 in 2022-09-10T13:29:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:11.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:11.084637+0000) 2022-09-10T13:29:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:11 smithi110 ceph-mon[97633]: pgmap v17276: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:29:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:11 smithi110 ceph-mon[97633]: osdmap e1161: 12 total, 12 up, 12 in 2022-09-10T13:29:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:11.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:11.700406+0000) 2022-09-10T13:29:12.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:12.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:12.084853+0000) 2022-09-10T13:29:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:12.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:12.700545+0000) 2022-09-10T13:29:13.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:13 smithi104 ceph-mon[97044]: pgmap v17278: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:29:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:13.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:13.085035+0000) 2022-09-10T13:29:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:13 smithi110 ceph-mon[97633]: pgmap v17278: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:29:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:13.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:13.700738+0000) 2022-09-10T13:29:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:14.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:14.085154+0000) 2022-09-10T13:29:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:14.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:14.700990+0000) 2022-09-10T13:29:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:15 smithi104 ceph-mon[97044]: pgmap v17279: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:29:15.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:15.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:15.085361+0000) 2022-09-10T13:29:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:15 smithi110 ceph-mon[97633]: pgmap v17279: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:29:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:15.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:15.701194+0000) 2022-09-10T13:29:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:16.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:16.085615+0000) 2022-09-10T13:29:17.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:16 smithi104 ceph-mon[97044]: pgmap v17280: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T13:29:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:16.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:16.701402+0000) 2022-09-10T13:29:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:16 smithi110 ceph-mon[97633]: pgmap v17280: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T13:29:17.474 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:17.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:17.085865+0000) 2022-09-10T13:29:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:17.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:17.701562+0000) 2022-09-10T13:29:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:29:17.471+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:29:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:18.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:18.086147+0000) 2022-09-10T13:29:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:18.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:18.701792+0000) 2022-09-10T13:29:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:19 smithi104 ceph-mon[97044]: pgmap v17281: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:29:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:19.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:19.086369+0000) 2022-09-10T13:29:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:19 smithi110 ceph-mon[97633]: pgmap v17281: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:29:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:19.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:19.702015+0000) 2022-09-10T13:29:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:20.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:20.086545+0000) 2022-09-10T13:29:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:20.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:20.702233+0000) 2022-09-10T13:29:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:21 smithi104 ceph-mon[97044]: pgmap v17282: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:29:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:21.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:21.086744+0000) 2022-09-10T13:29:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:21 smithi110 ceph-mon[97633]: pgmap v17282: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:29:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:21.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:21.702467+0000) 2022-09-10T13:29:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:22.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:22.087010+0000) 2022-09-10T13:29:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:22.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:22.702690+0000) 2022-09-10T13:29:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:23 smithi104 ceph-mon[97044]: pgmap v17283: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:29:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:23.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:23.087245+0000) 2022-09-10T13:29:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:23 smithi110 ceph-mon[97633]: pgmap v17283: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:29:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:23.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:23.702935+0000) 2022-09-10T13:29:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:24.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:24.087454+0000) 2022-09-10T13:29:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:24.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:24.703186+0000) 2022-09-10T13:29:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:25 smithi104 ceph-mon[97044]: pgmap v17284: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:29:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:25.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:25.087712+0000) 2022-09-10T13:29:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:25 smithi110 ceph-mon[97633]: pgmap v17284: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:29:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:25.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:25.703332+0000) 2022-09-10T13:29:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:26.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:26.087883+0000) 2022-09-10T13:29:27.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:26.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:26.703512+0000) 2022-09-10T13:29:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:26 smithi104 ceph-mon[97044]: pgmap v17285: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:29:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:26 smithi110 ceph-mon[97633]: pgmap v17285: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:29:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:27.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:27.088108+0000) 2022-09-10T13:29:28.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:27.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:27.703738+0000) 2022-09-10T13:29:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:28.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:28.088251+0000) 2022-09-10T13:29:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:28.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:28.703939+0000) 2022-09-10T13:29:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:29 smithi104 ceph-mon[97044]: pgmap v17286: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:29.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:29.088410+0000) 2022-09-10T13:29:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:29 smithi110 ceph-mon[97633]: pgmap v17286: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:29.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:29.704139+0000) 2022-09-10T13:29:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:30.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:30.088632+0000) 2022-09-10T13:29:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:30.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:30.704350+0000) 2022-09-10T13:29:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:31 smithi104 ceph-mon[97044]: pgmap v17287: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:31.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:31.088897+0000) 2022-09-10T13:29:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:31 smithi110 ceph-mon[97633]: pgmap v17287: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:31.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:31.704571+0000) 2022-09-10T13:29:32.482 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:32.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:32.089138+0000) 2022-09-10T13:29:32.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:32.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:32.704747+0000) 2022-09-10T13:29:32.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:29:32.481+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:29:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:33 smithi104 ceph-mon[97044]: pgmap v17288: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:33.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:33.089302+0000) 2022-09-10T13:29:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:33 smithi110 ceph-mon[97633]: pgmap v17288: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:33.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:33.704990+0000) 2022-09-10T13:29:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:34.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:34.089512+0000) 2022-09-10T13:29:35.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:34.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:34.705187+0000) 2022-09-10T13:29:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:35 smithi104 ceph-mon[97044]: pgmap v17289: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:35.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:35.089745+0000) 2022-09-10T13:29:35.664 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:35 smithi110 ceph-mon[97633]: pgmap v17289: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:35.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:35.705438+0000) 2022-09-10T13:29:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:36.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:36.089903+0000) 2022-09-10T13:29:37.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:36 smithi104 ceph-mon[97044]: pgmap v17290: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:37.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:36.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:36.705635+0000) 2022-09-10T13:29:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:36 smithi110 ceph-mon[97633]: pgmap v17290: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:37.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:37.090168+0000) 2022-09-10T13:29:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:37.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:37.705902+0000) 2022-09-10T13:29:38.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:38.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:38.090319+0000) 2022-09-10T13:29:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:38.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:38.706173+0000) 2022-09-10T13:29:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:39 smithi104 ceph-mon[97044]: pgmap v17291: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:39.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:39.090549+0000) 2022-09-10T13:29:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:39 smithi110 ceph-mon[97633]: pgmap v17291: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:39.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:39.706416+0000) 2022-09-10T13:29:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:40.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:40.090802+0000) 2022-09-10T13:29:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:40.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:40.706659+0000) 2022-09-10T13:29:41.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:41 smithi104 ceph-mon[97044]: pgmap v17292: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:41.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:41.091028+0000) 2022-09-10T13:29:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:41 smithi110 ceph-mon[97633]: pgmap v17292: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:41.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:41.707001+0000) 2022-09-10T13:29:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:42.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:42.091244+0000) 2022-09-10T13:29:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:42.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:42.707271+0000) 2022-09-10T13:29:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:43 smithi104 ceph-mon[97044]: pgmap v17293: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:43.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:43.091421+0000) 2022-09-10T13:29:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:43 smithi110 ceph-mon[97633]: pgmap v17293: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:43.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:43.707414+0000) 2022-09-10T13:29:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:44.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:44.091618+0000) 2022-09-10T13:29:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:44.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:44.707560+0000) 2022-09-10T13:29:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:45 smithi104 ceph-mon[97044]: pgmap v17294: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:45.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:45.091875+0000) 2022-09-10T13:29:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:45 smithi110 ceph-mon[97633]: pgmap v17294: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:46.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:45.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:45.707752+0000) 2022-09-10T13:29:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:46.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:46.092089+0000) 2022-09-10T13:29:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:46 smithi104 ceph-mon[97044]: pgmap v17295: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:29:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:29:47.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:29:47.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:29:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:46.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:46.708020+0000) 2022-09-10T13:29:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:46 smithi110 ceph-mon[97633]: pgmap v17295: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:29:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:29:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:29:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:29:47.485 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:47.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:47.092374+0000) 2022-09-10T13:29:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:47.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:47.708271+0000) 2022-09-10T13:29:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:29:47.485+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:29:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:48.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:48.092559+0000) 2022-09-10T13:29:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:48.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:48.708530+0000) 2022-09-10T13:29:49.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:49 smithi104 ceph-mon[97044]: pgmap v17296: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:49.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:49.092836+0000) 2022-09-10T13:29:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:49 smithi110 ceph-mon[97633]: pgmap v17296: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:49.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:49.708708+0000) 2022-09-10T13:29:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:50.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:50.093110+0000) 2022-09-10T13:29:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:50.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:50.708900+0000) 2022-09-10T13:29:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:51 smithi104 ceph-mon[97044]: pgmap v17297: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:51.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:51.093243+0000) 2022-09-10T13:29:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:51 smithi110 ceph-mon[97633]: pgmap v17297: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:51.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:51.709083+0000) 2022-09-10T13:29:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:52.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:52.093471+0000) 2022-09-10T13:29:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:52.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:52.709342+0000) 2022-09-10T13:29:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:53 smithi104 ceph-mon[97044]: pgmap v17298: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:53.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:53.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:53.093740+0000) 2022-09-10T13:29:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:53 smithi110 ceph-mon[97633]: pgmap v17298: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:53.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:53.709564+0000) 2022-09-10T13:29:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:54.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:54.094005+0000) 2022-09-10T13:29:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:54.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:54.709783+0000) 2022-09-10T13:29:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:55 smithi104 ceph-mon[97044]: pgmap v17299: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:55.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:55.094258+0000) 2022-09-10T13:29:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:55 smithi110 ceph-mon[97633]: pgmap v17299: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:55.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:55.710046+0000) 2022-09-10T13:29:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:56.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:56.094488+0000) 2022-09-10T13:29:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:56 smithi104 ceph-mon[97044]: pgmap v17300: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:56.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:56.710216+0000) 2022-09-10T13:29:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:56 smithi110 ceph-mon[97633]: pgmap v17300: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:29:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:57.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:57.094723+0000) 2022-09-10T13:29:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:57.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:57.710421+0000) 2022-09-10T13:29:58.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:58.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:58.094946+0000) 2022-09-10T13:29:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:58.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:58.710614+0000) 2022-09-10T13:29:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:29:59 smithi104 ceph-mon[97044]: pgmap v17301: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:29:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:29:59.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:59.095173+0000) 2022-09-10T13:29:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:29:59 smithi110 ceph-mon[97633]: pgmap v17301: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:29:59.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:29:59.710833+0000) 2022-09-10T13:30:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:30:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:00.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:00.095326+0000) 2022-09-10T13:30:00.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:30:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:00.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:00.711136+0000) 2022-09-10T13:30:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:01 smithi104 ceph-mon[97044]: pgmap v17302: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:01.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:01.095491+0000) 2022-09-10T13:30:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:01 smithi110 ceph-mon[97633]: pgmap v17302: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:01.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:01.711305+0000) 2022-09-10T13:30:02.497 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:02.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:02.095726+0000) 2022-09-10T13:30:02.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:30:02.496+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:30:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:02.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:02.711491+0000) 2022-09-10T13:30:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:03 smithi104 ceph-mon[97044]: pgmap v17303: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:03.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:03.095956+0000) 2022-09-10T13:30:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:03 smithi110 ceph-mon[97633]: pgmap v17303: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:03.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:03.711659+0000) 2022-09-10T13:30:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:04.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:04.096178+0000) 2022-09-10T13:30:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:04.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:04.711907+0000) 2022-09-10T13:30:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:05 smithi104 ceph-mon[97044]: pgmap v17304: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:05.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:05.096368+0000) 2022-09-10T13:30:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:05 smithi110 ceph-mon[97633]: pgmap v17304: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:05.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:05.712086+0000) 2022-09-10T13:30:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:06.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:06.096596+0000) 2022-09-10T13:30:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:06 smithi104 ceph-mon[97044]: pgmap v17305: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:30:07.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:06.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:06.712300+0000) 2022-09-10T13:30:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:06 smithi110 ceph-mon[97633]: pgmap v17305: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:30:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:07.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:07.096860+0000) 2022-09-10T13:30:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:07.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:07.712504+0000) 2022-09-10T13:30:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:08.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:08.097079+0000) 2022-09-10T13:30:09.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:08.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:08.712755+0000) 2022-09-10T13:30:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:08 smithi110 ceph-mon[97633]: osdmap e1162: 12 total, 12 up, 12 in 2022-09-10T13:30:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:08 smithi110 ceph-mon[97633]: pgmap v17307: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:08 smithi104 ceph-mon[97044]: osdmap e1162: 12 total, 12 up, 12 in 2022-09-10T13:30:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:08 smithi104 ceph-mon[97044]: pgmap v17307: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:30:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:09.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:09.097303+0000) 2022-09-10T13:30:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:09.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:09.712985+0000) 2022-09-10T13:30:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:10.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:10.097532+0000) 2022-09-10T13:30:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:10.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:10.713247+0000) 2022-09-10T13:30:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:11 smithi104 ceph-mon[97044]: pgmap v17308: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:30:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:11 smithi104 ceph-mon[97044]: osdmap e1163: 12 total, 12 up, 12 in 2022-09-10T13:30:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:11.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:11.097696+0000) 2022-09-10T13:30:11.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:11 smithi110 ceph-mon[97633]: pgmap v17308: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:30:11.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:11 smithi110 ceph-mon[97633]: osdmap e1163: 12 total, 12 up, 12 in 2022-09-10T13:30:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:11.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:11.713466+0000) 2022-09-10T13:30:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:12.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:12.097920+0000) 2022-09-10T13:30:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:12.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:12.713694+0000) 2022-09-10T13:30:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:13 smithi104 ceph-mon[97044]: pgmap v17310: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:30:13.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:13.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:13.098183+0000) 2022-09-10T13:30:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:13 smithi110 ceph-mon[97633]: pgmap v17310: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:30:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:13.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:13.713861+0000) 2022-09-10T13:30:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:14.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:14.098419+0000) 2022-09-10T13:30:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:14.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:14.714073+0000) 2022-09-10T13:30:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:15 smithi104 ceph-mon[97044]: pgmap v17311: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:30:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:15.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:15.098609+0000) 2022-09-10T13:30:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:15 smithi110 ceph-mon[97633]: pgmap v17311: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:30:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:15.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:15.714319+0000) 2022-09-10T13:30:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:16.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:16.098870+0000) 2022-09-10T13:30:17.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:16 smithi104 ceph-mon[97044]: pgmap v17312: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T13:30:17.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:16.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:16.714479+0000) 2022-09-10T13:30:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:16 smithi110 ceph-mon[97633]: pgmap v17312: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T13:30:17.513 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:17.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:17.099141+0000) 2022-09-10T13:30:17.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:17.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:17.714640+0000) 2022-09-10T13:30:17.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:30:17.512+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:30:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:18.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:18.099356+0000) 2022-09-10T13:30:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:18.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:18.714879+0000) 2022-09-10T13:30:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:19 smithi104 ceph-mon[97044]: pgmap v17313: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:30:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:19.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:19.099633+0000) 2022-09-10T13:30:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:19 smithi110 ceph-mon[97633]: pgmap v17313: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:30:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:19.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:19.715165+0000) 2022-09-10T13:30:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:20.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:20.099793+0000) 2022-09-10T13:30:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:20.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:20.715294+0000) 2022-09-10T13:30:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:21 smithi104 ceph-mon[97044]: pgmap v17314: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:30:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:21.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:21.100015+0000) 2022-09-10T13:30:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:21 smithi110 ceph-mon[97633]: pgmap v17314: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:30:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:21.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:21.715497+0000) 2022-09-10T13:30:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:22.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:22.100244+0000) 2022-09-10T13:30:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:22.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:22.715695+0000) 2022-09-10T13:30:23.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:23 smithi104 ceph-mon[97044]: pgmap v17315: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:30:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:23.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:23.100447+0000) 2022-09-10T13:30:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:23 smithi110 ceph-mon[97633]: pgmap v17315: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:30:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:23.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:23.715922+0000) 2022-09-10T13:30:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:24.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:24.100629+0000) 2022-09-10T13:30:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:24.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:24.716127+0000) 2022-09-10T13:30:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:25 smithi104 ceph-mon[97044]: pgmap v17316: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:30:25.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:25.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:25.100886+0000) 2022-09-10T13:30:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:25 smithi110 ceph-mon[97633]: pgmap v17316: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:30:26.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:25.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:25.716312+0000) 2022-09-10T13:30:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:26.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:26.101122+0000) 2022-09-10T13:30:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:26 smithi104 ceph-mon[97044]: pgmap v17317: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:30:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:26.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:26.716504+0000) 2022-09-10T13:30:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:26 smithi110 ceph-mon[97633]: pgmap v17317: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:30:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:27.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:27.101277+0000) 2022-09-10T13:30:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:27.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:27.716720+0000) 2022-09-10T13:30:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:28.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:28.101444+0000) 2022-09-10T13:30:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:28.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:28.716981+0000) 2022-09-10T13:30:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:29 smithi104 ceph-mon[97044]: pgmap v17318: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:29.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:29.101674+0000) 2022-09-10T13:30:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:29 smithi110 ceph-mon[97633]: pgmap v17318: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:29.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:29.717203+0000) 2022-09-10T13:30:30.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:30.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:30.101898+0000) 2022-09-10T13:30:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:30.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:30.717402+0000) 2022-09-10T13:30:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:31 smithi104 ceph-mon[97044]: pgmap v17319: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:31.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:31.102070+0000) 2022-09-10T13:30:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:31 smithi110 ceph-mon[97633]: pgmap v17319: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:31.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:31.717607+0000) 2022-09-10T13:30:32.531 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:32.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:32.102313+0000) 2022-09-10T13:30:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:32.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:32.717824+0000) 2022-09-10T13:30:32.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:30:32.528+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:30:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:33 smithi104 ceph-mon[97044]: pgmap v17320: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:33.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:33.102539+0000) 2022-09-10T13:30:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:33 smithi110 ceph-mon[97633]: pgmap v17320: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:33.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:33.718017+0000) 2022-09-10T13:30:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:34.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:34.102799+0000) 2022-09-10T13:30:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:34.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:34.718249+0000) 2022-09-10T13:30:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:35 smithi104 ceph-mon[97044]: pgmap v17321: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:35.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:35.103039+0000) 2022-09-10T13:30:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:35 smithi110 ceph-mon[97633]: pgmap v17321: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:35.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:35.718434+0000) 2022-09-10T13:30:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:36.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:36.103261+0000) 2022-09-10T13:30:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:36 smithi104 ceph-mon[97044]: pgmap v17322: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:36.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:36.718652+0000) 2022-09-10T13:30:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:36 smithi110 ceph-mon[97633]: pgmap v17322: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:37.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:37.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:37.103492+0000) 2022-09-10T13:30:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:37.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:37.718907+0000) 2022-09-10T13:30:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:38.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:38.103748+0000) 2022-09-10T13:30:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:38.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:38.719202+0000) 2022-09-10T13:30:39.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:39 smithi104 ceph-mon[97044]: pgmap v17323: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:39.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:39.103940+0000) 2022-09-10T13:30:39.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:39 smithi110 ceph-mon[97633]: pgmap v17323: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:39.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:39.719421+0000) 2022-09-10T13:30:40.571 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:40.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:40.104160+0000) 2022-09-10T13:30:41.064 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:40.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:40.719667+0000) 2022-09-10T13:30:41.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:41 smithi104 ceph-mon[97044]: pgmap v17324: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:41.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:41.104326+0000) 2022-09-10T13:30:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:41 smithi110 ceph-mon[97633]: pgmap v17324: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:41.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:41.719867+0000) 2022-09-10T13:30:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:42.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:42.104531+0000) 2022-09-10T13:30:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:42.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:42.720070+0000) 2022-09-10T13:30:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:43 smithi104 ceph-mon[97044]: pgmap v17325: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:43.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:43.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:43.104762+0000) 2022-09-10T13:30:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:43 smithi110 ceph-mon[97633]: pgmap v17325: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:43.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:43.720317+0000) 2022-09-10T13:30:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:44.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:44.105033+0000) 2022-09-10T13:30:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:44.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:44.720571+0000) 2022-09-10T13:30:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:45 smithi104 ceph-mon[97044]: pgmap v17326: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:45.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:45.105282+0000) 2022-09-10T13:30:45.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:45 smithi110 ceph-mon[97633]: pgmap v17326: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:45.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:45.720795+0000) 2022-09-10T13:30:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:46.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:46.105535+0000) 2022-09-10T13:30:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:46 smithi104 ceph-mon[97044]: pgmap v17327: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:30:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:30:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:30:47.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:30:47.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:46.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:46.721026+0000) 2022-09-10T13:30:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:46 smithi110 ceph-mon[97633]: pgmap v17327: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:30:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:30:47.536 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:47.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:47.105739+0000) 2022-09-10T13:30:47.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:47.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:47.721242+0000) 2022-09-10T13:30:47.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:30:47.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:30:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:48.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:48.105969+0000) 2022-09-10T13:30:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:48.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:48.721512+0000) 2022-09-10T13:30:49.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:49 smithi104 ceph-mon[97044]: pgmap v17328: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:49.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:49.106224+0000) 2022-09-10T13:30:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:49 smithi110 ceph-mon[97633]: pgmap v17328: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:49.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:49.721663+0000) 2022-09-10T13:30:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:50.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:50.106378+0000) 2022-09-10T13:30:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:50.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:50.721820+0000) 2022-09-10T13:30:51.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:51 smithi104 ceph-mon[97044]: pgmap v17329: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:51.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:51.106612+0000) 2022-09-10T13:30:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:51 smithi110 ceph-mon[97633]: pgmap v17329: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:52.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:51.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:51.721945+0000) 2022-09-10T13:30:52.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:52.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:52.106806+0000) 2022-09-10T13:30:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:52.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:52.722142+0000) 2022-09-10T13:30:53.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:53 smithi104 ceph-mon[97044]: pgmap v17330: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:53.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:53.107043+0000) 2022-09-10T13:30:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:53 smithi110 ceph-mon[97633]: pgmap v17330: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:53.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:53.722361+0000) 2022-09-10T13:30:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:54.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:54.107301+0000) 2022-09-10T13:30:55.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:54.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:54.722556+0000) 2022-09-10T13:30:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:55 smithi104 ceph-mon[97044]: pgmap v17331: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:55.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:55.107468+0000) 2022-09-10T13:30:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:55 smithi110 ceph-mon[97633]: pgmap v17331: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:55.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:55.722775+0000) 2022-09-10T13:30:56.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:56.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:56.107702+0000) 2022-09-10T13:30:57.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:56 smithi104 ceph-mon[97044]: pgmap v17332: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:56.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:56.723049+0000) 2022-09-10T13:30:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:56 smithi110 ceph-mon[97633]: pgmap v17332: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:30:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:57.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:57.107938+0000) 2022-09-10T13:30:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:57.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:57.723309+0000) 2022-09-10T13:30:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:58.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:58.108181+0000) 2022-09-10T13:30:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:58.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:58.723476+0000) 2022-09-10T13:30:59.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:30:59 smithi104 ceph-mon[97044]: pgmap v17333: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:30:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:30:59.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:59.108305+0000) 2022-09-10T13:30:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:30:59 smithi110 ceph-mon[97633]: pgmap v17333: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:30:59.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:30:59.723723+0000) 2022-09-10T13:31:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:00.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:00.108474+0000) 2022-09-10T13:31:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:00.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:00.723951+0000) 2022-09-10T13:31:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:01 smithi104 ceph-mon[97044]: pgmap v17334: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:01.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:01.108646+0000) 2022-09-10T13:31:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:01 smithi110 ceph-mon[97633]: pgmap v17334: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:02.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:01.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:01.724174+0000) 2022-09-10T13:31:02.552 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:02.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:02.108909+0000) 2022-09-10T13:31:02.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:31:02.551+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:31:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:02.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:02.724399+0000) 2022-09-10T13:31:03.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:03 smithi104 ceph-mon[97044]: pgmap v17335: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:03.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:03.109170+0000) 2022-09-10T13:31:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:03 smithi110 ceph-mon[97633]: pgmap v17335: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:03.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:03.724609+0000) 2022-09-10T13:31:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:04.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:04.109388+0000) 2022-09-10T13:31:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:04.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:04.724794+0000) 2022-09-10T13:31:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:05 smithi104 ceph-mon[97044]: pgmap v17336: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:05.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:05.109644+0000) 2022-09-10T13:31:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:05 smithi110 ceph-mon[97633]: pgmap v17336: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:05.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:05.725012+0000) 2022-09-10T13:31:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:06.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:06.109881+0000) 2022-09-10T13:31:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:06 smithi104 ceph-mon[97044]: pgmap v17337: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:31:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:06.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:06.725224+0000) 2022-09-10T13:31:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:06 smithi110 ceph-mon[97633]: pgmap v17337: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:31:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:07.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:07.110160+0000) 2022-09-10T13:31:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:07.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:07.725485+0000) 2022-09-10T13:31:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:08.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:08.110390+0000) 2022-09-10T13:31:09.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:08.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:08.725723+0000) 2022-09-10T13:31:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:08 smithi110 ceph-mon[97633]: osdmap e1164: 12 total, 12 up, 12 in 2022-09-10T13:31:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:08 smithi110 ceph-mon[97633]: pgmap v17339: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:08 smithi104 ceph-mon[97044]: osdmap e1164: 12 total, 12 up, 12 in 2022-09-10T13:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:08 smithi104 ceph-mon[97044]: pgmap v17339: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:31:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:09.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:09.110616+0000) 2022-09-10T13:31:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:09.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:09.725937+0000) 2022-09-10T13:31:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:10.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:10.110877+0000) 2022-09-10T13:31:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:10.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:10.726138+0000) 2022-09-10T13:31:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:11 smithi104 ceph-mon[97044]: pgmap v17340: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:31:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:11 smithi104 ceph-mon[97044]: osdmap e1165: 12 total, 12 up, 12 in 2022-09-10T13:31:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:11.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:11.111044+0000) 2022-09-10T13:31:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:11 smithi110 ceph-mon[97633]: pgmap v17340: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:31:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:11 smithi110 ceph-mon[97633]: osdmap e1165: 12 total, 12 up, 12 in 2022-09-10T13:31:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:11.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:11.726370+0000) 2022-09-10T13:31:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:12.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:12.111183+0000) 2022-09-10T13:31:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:12.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:12.726578+0000) 2022-09-10T13:31:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:13 smithi104 ceph-mon[97044]: pgmap v17342: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:31:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:13.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:13.111406+0000) 2022-09-10T13:31:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:13 smithi110 ceph-mon[97633]: pgmap v17342: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:31:14.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:13.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:13.726798+0000) 2022-09-10T13:31:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:14.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:14.111585+0000) 2022-09-10T13:31:15.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:14.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:14.727013+0000) 2022-09-10T13:31:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:15 smithi104 ceph-mon[97044]: pgmap v17343: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:31:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:15.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:15.111837+0000) 2022-09-10T13:31:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:15 smithi110 ceph-mon[97633]: pgmap v17343: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:31:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:15.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:15.727208+0000) 2022-09-10T13:31:16.562 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:16.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:16.112070+0000) 2022-09-10T13:31:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:16 smithi104 ceph-mon[97044]: pgmap v17344: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T13:31:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:16.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:16.727445+0000) 2022-09-10T13:31:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:16 smithi110 ceph-mon[97633]: pgmap v17344: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T13:31:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:17.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:17.112301+0000) 2022-09-10T13:31:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:17.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:17.727614+0000) 2022-09-10T13:31:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:31:17.563+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:31:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:18.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:18.112490+0000) 2022-09-10T13:31:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:18.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:18.727828+0000) 2022-09-10T13:31:19.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:19 smithi104 ceph-mon[97044]: pgmap v17345: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:31:19.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:19.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:19.112695+0000) 2022-09-10T13:31:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:19 smithi110 ceph-mon[97633]: pgmap v17345: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:31:20.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:19.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:19.728003+0000) 2022-09-10T13:31:20.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:20.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:20.112886+0000) 2022-09-10T13:31:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:20.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:20.728192+0000) 2022-09-10T13:31:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:21 smithi104 ceph-mon[97044]: pgmap v17346: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T13:31:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:21.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:21.113128+0000) 2022-09-10T13:31:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:21 smithi110 ceph-mon[97633]: pgmap v17346: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T13:31:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:21.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:21.728394+0000) 2022-09-10T13:31:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:22.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:22.113354+0000) 2022-09-10T13:31:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:22.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:22.728571+0000) 2022-09-10T13:31:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:23 smithi104 ceph-mon[97044]: pgmap v17347: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:31:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:23.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:23.113558+0000) 2022-09-10T13:31:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:23 smithi110 ceph-mon[97633]: pgmap v17347: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:31:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:23.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:23.728823+0000) 2022-09-10T13:31:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:24.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:24.113775+0000) 2022-09-10T13:31:25.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:24.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:24.729052+0000) 2022-09-10T13:31:25.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:25 smithi104 ceph-mon[97044]: pgmap v17348: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:31:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:25.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:25.113993+0000) 2022-09-10T13:31:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:25 smithi110 ceph-mon[97633]: pgmap v17348: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:31:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:25.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:25.729351+0000) 2022-09-10T13:31:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:26.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:26.114212+0000) 2022-09-10T13:31:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:26 smithi104 ceph-mon[97044]: pgmap v17349: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:31:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:26.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:26.729567+0000) 2022-09-10T13:31:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:26 smithi110 ceph-mon[97633]: pgmap v17349: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:31:27.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:27.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:27.114421+0000) 2022-09-10T13:31:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:27.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:27.729837+0000) 2022-09-10T13:31:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:28.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:28.114572+0000) 2022-09-10T13:31:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:28.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:28.730054+0000) 2022-09-10T13:31:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:29 smithi104 ceph-mon[97044]: pgmap v17350: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:31:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:29.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:29.114794+0000) 2022-09-10T13:31:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:29 smithi110 ceph-mon[97633]: pgmap v17350: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:31:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:29.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:29.730294+0000) 2022-09-10T13:31:30.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:30.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:30.115059+0000) 2022-09-10T13:31:31.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:30.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:30.730438+0000) 2022-09-10T13:31:31.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:31 smithi104 ceph-mon[97044]: pgmap v17351: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:31.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:31.115315+0000) 2022-09-10T13:31:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:31 smithi110 ceph-mon[97633]: pgmap v17351: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:31.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:31.730652+0000) 2022-09-10T13:31:32.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:32.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:32.115557+0000) 2022-09-10T13:31:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:32.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:32.730863+0000) 2022-09-10T13:31:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:31:32.576+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:31:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:33 smithi104 ceph-mon[97044]: pgmap v17352: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:33.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:33.115728+0000) 2022-09-10T13:31:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:33 smithi110 ceph-mon[97633]: pgmap v17352: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:33.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:33.731069+0000) 2022-09-10T13:31:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:34.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:34.115926+0000) 2022-09-10T13:31:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:34.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:34.731353+0000) 2022-09-10T13:31:35.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:35 smithi104 ceph-mon[97044]: pgmap v17353: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:35.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:35.116192+0000) 2022-09-10T13:31:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:35 smithi110 ceph-mon[97633]: pgmap v17353: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:35.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:35.731626+0000) 2022-09-10T13:31:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:36.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:36.116374+0000) 2022-09-10T13:31:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:36 smithi104 ceph-mon[97044]: pgmap v17354: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:36.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:36.731799+0000) 2022-09-10T13:31:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:36 smithi110 ceph-mon[97633]: pgmap v17354: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:37.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:37.116610+0000) 2022-09-10T13:31:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:37.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:37.732035+0000) 2022-09-10T13:31:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:38.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:38.116861+0000) 2022-09-10T13:31:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:38.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:38.732287+0000) 2022-09-10T13:31:39.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:39 smithi104 ceph-mon[97044]: pgmap v17355: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:39.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:39.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:39.117067+0000) 2022-09-10T13:31:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:39 smithi110 ceph-mon[97633]: pgmap v17355: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:39.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:39.732504+0000) 2022-09-10T13:31:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:40.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:40.117359+0000) 2022-09-10T13:31:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:40.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:40.732727+0000) 2022-09-10T13:31:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:41 smithi104 ceph-mon[97044]: pgmap v17356: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:41.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:41.117604+0000) 2022-09-10T13:31:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:41 smithi110 ceph-mon[97633]: pgmap v17356: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:41.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:41.732934+0000) 2022-09-10T13:31:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:42.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:42.117855+0000) 2022-09-10T13:31:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:42.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:42.733165+0000) 2022-09-10T13:31:43.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:43 smithi104 ceph-mon[97044]: pgmap v17357: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:43.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:43.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:43.118016+0000) 2022-09-10T13:31:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:43 smithi110 ceph-mon[97633]: pgmap v17357: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:43.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:43.733286+0000) 2022-09-10T13:31:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:44.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:44.118214+0000) 2022-09-10T13:31:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:44.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:44.733406+0000) 2022-09-10T13:31:45.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:45 smithi104 ceph-mon[97044]: pgmap v17358: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:45.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:45.118411+0000) 2022-09-10T13:31:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:45 smithi110 ceph-mon[97633]: pgmap v17358: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:45.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:45.733581+0000) 2022-09-10T13:31:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:46.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:46.118615+0000) 2022-09-10T13:31:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:46 smithi104 ceph-mon[97044]: pgmap v17359: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:31:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:31:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:31:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:31:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:46.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:46.733774+0000) 2022-09-10T13:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:46 smithi110 ceph-mon[97633]: pgmap v17359: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:31:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:31:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:31:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:31:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:47.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:47.118841+0000) 2022-09-10T13:31:48.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:31:47.592+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:31:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:47.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:47.734023+0000) 2022-09-10T13:31:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:48.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:48.119047+0000) 2022-09-10T13:31:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:48.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:48.734284+0000) 2022-09-10T13:31:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:49 smithi104 ceph-mon[97044]: pgmap v17360: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:49.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:49.119193+0000) 2022-09-10T13:31:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:49 smithi110 ceph-mon[97633]: pgmap v17360: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:49.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:49.734527+0000) 2022-09-10T13:31:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:50.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:50.119421+0000) 2022-09-10T13:31:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:50.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:50.734799+0000) 2022-09-10T13:31:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:51 smithi104 ceph-mon[97044]: pgmap v17361: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:51.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:51.119650+0000) 2022-09-10T13:31:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:51 smithi110 ceph-mon[97633]: pgmap v17361: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:51.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:51.735021+0000) 2022-09-10T13:31:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:52.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:52.119841+0000) 2022-09-10T13:31:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:52.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:52.735226+0000) 2022-09-10T13:31:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:53 smithi104 ceph-mon[97044]: pgmap v17362: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:53.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:53.120082+0000) 2022-09-10T13:31:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:53 smithi110 ceph-mon[97633]: pgmap v17362: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:54.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:53.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:53.735378+0000) 2022-09-10T13:31:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:54.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:54.120383+0000) 2022-09-10T13:31:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:54.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:54.735586+0000) 2022-09-10T13:31:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:55 smithi104 ceph-mon[97044]: pgmap v17363: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:55.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:55.120662+0000) 2022-09-10T13:31:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:55 smithi110 ceph-mon[97633]: pgmap v17363: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:55.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:55.735835+0000) 2022-09-10T13:31:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:56.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:56.120873+0000) 2022-09-10T13:31:57.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:56 smithi104 ceph-mon[97044]: pgmap v17364: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:56.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:56.736076+0000) 2022-09-10T13:31:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:56 smithi110 ceph-mon[97633]: pgmap v17364: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:31:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:57.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:57.121187+0000) 2022-09-10T13:31:58.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:57.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:57.736348+0000) 2022-09-10T13:31:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:58.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:58.121447+0000) 2022-09-10T13:31:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:58.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:58.736593+0000) 2022-09-10T13:31:59.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:31:59 smithi104 ceph-mon[97044]: pgmap v17365: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:31:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:31:59.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:59.121692+0000) 2022-09-10T13:31:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:31:59 smithi110 ceph-mon[97633]: pgmap v17365: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:31:59.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:31:59.736792+0000) 2022-09-10T13:32:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:00.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:00.121870+0000) 2022-09-10T13:32:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:00.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:00.737008+0000) 2022-09-10T13:32:01.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:01 smithi104 ceph-mon[97044]: pgmap v17366: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:01.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:01.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:01.122089+0000) 2022-09-10T13:32:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:01 smithi110 ceph-mon[97633]: pgmap v17366: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:01.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:01.737237+0000) 2022-09-10T13:32:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:02.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:02.122369+0000) 2022-09-10T13:32:03.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:32:02.600+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:32:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:02.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:02.737424+0000) 2022-09-10T13:32:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:03 smithi104 ceph-mon[97044]: pgmap v17367: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:03.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:03.122530+0000) 2022-09-10T13:32:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:03 smithi110 ceph-mon[97633]: pgmap v17367: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:03.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:03.737670+0000) 2022-09-10T13:32:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:04.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:04.122756+0000) 2022-09-10T13:32:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:04.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:04.737927+0000) 2022-09-10T13:32:05.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:05 smithi104 ceph-mon[97044]: pgmap v17368: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:05.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:05.123007+0000) 2022-09-10T13:32:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:05 smithi110 ceph-mon[97633]: pgmap v17368: 129 pgs: 129 active+clean; 9.6 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:05.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:05.738169+0000) 2022-09-10T13:32:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:06.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:06.123257+0000) 2022-09-10T13:32:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:06 smithi104 ceph-mon[97044]: pgmap v17369: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:32:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:06.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:06.738300+0000) 2022-09-10T13:32:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:06 smithi110 ceph-mon[97633]: pgmap v17369: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:32:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:07.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:07.123482+0000) 2022-09-10T13:32:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:07.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:07.738516+0000) 2022-09-10T13:32:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:08.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:08.123632+0000) 2022-09-10T13:32:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:08.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:08.738780+0000) 2022-09-10T13:32:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:08 smithi110 ceph-mon[97633]: osdmap e1166: 12 total, 12 up, 12 in 2022-09-10T13:32:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:08 smithi110 ceph-mon[97633]: pgmap v17371: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:32:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:08 smithi104 ceph-mon[97044]: osdmap e1166: 12 total, 12 up, 12 in 2022-09-10T13:32:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:08 smithi104 ceph-mon[97044]: pgmap v17371: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:32:09.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:09.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:09.123870+0000) 2022-09-10T13:32:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:09.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:09.739018+0000) 2022-09-10T13:32:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:10.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:10.124075+0000) 2022-09-10T13:32:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:10.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:10.739205+0000) 2022-09-10T13:32:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:11 smithi104 ceph-mon[97044]: pgmap v17372: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:32:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:11 smithi104 ceph-mon[97044]: osdmap e1167: 12 total, 12 up, 12 in 2022-09-10T13:32:11.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:11.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:11.124334+0000) 2022-09-10T13:32:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:11 smithi110 ceph-mon[97633]: pgmap v17372: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:32:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:11 smithi110 ceph-mon[97633]: osdmap e1167: 12 total, 12 up, 12 in 2022-09-10T13:32:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:11.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:11.739422+0000) 2022-09-10T13:32:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:12.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:12.124553+0000) 2022-09-10T13:32:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:12.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:12.739600+0000) 2022-09-10T13:32:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:13 smithi104 ceph-mon[97044]: pgmap v17374: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:32:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:13.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:13.124818+0000) 2022-09-10T13:32:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:13 smithi110 ceph-mon[97633]: pgmap v17374: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:32:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:13.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:13.739838+0000) 2022-09-10T13:32:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:14.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:14.125000+0000) 2022-09-10T13:32:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:14.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:14.740032+0000) 2022-09-10T13:32:15.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:15 smithi104 ceph-mon[97044]: pgmap v17375: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:32:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:15.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:15.125257+0000) 2022-09-10T13:32:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:15 smithi110 ceph-mon[97633]: pgmap v17375: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:32:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:15.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:15.740253+0000) 2022-09-10T13:32:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:16.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:16.125454+0000) 2022-09-10T13:32:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:16 smithi104 ceph-mon[97044]: pgmap v17376: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T13:32:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:16.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:16.740526+0000) 2022-09-10T13:32:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:16 smithi110 ceph-mon[97633]: pgmap v17376: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T13:32:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:17.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:17.125714+0000) 2022-09-10T13:32:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:17.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:17.740746+0000) 2022-09-10T13:32:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:32:17.609+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:32:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:18.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:18.125934+0000) 2022-09-10T13:32:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:18.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:18.740983+0000) 2022-09-10T13:32:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:19 smithi104 ceph-mon[97044]: pgmap v17377: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:32:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:19.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:19.126206+0000) 2022-09-10T13:32:19.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:19 smithi110 ceph-mon[97633]: pgmap v17377: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:32:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:19.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:19.741261+0000) 2022-09-10T13:32:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:20.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:20.126425+0000) 2022-09-10T13:32:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:20.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:20.741513+0000) 2022-09-10T13:32:21.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:21 smithi104 ceph-mon[97044]: pgmap v17378: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:32:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:21.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:21.126631+0000) 2022-09-10T13:32:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:21 smithi110 ceph-mon[97633]: pgmap v17378: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:32:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:21.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:21.741774+0000) 2022-09-10T13:32:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:22.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:22.126858+0000) 2022-09-10T13:32:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:22.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:22.741955+0000) 2022-09-10T13:32:23.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:23 smithi104 ceph-mon[97044]: pgmap v17379: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:32:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:23.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:23.127062+0000) 2022-09-10T13:32:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:23 smithi110 ceph-mon[97633]: pgmap v17379: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:32:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:23.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:23.742174+0000) 2022-09-10T13:32:24.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:24.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:24.127320+0000) 2022-09-10T13:32:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:24.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:24.742419+0000) 2022-09-10T13:32:25.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:25 smithi104 ceph-mon[97044]: pgmap v17380: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:32:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:25.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:25.127553+0000) 2022-09-10T13:32:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:25 smithi110 ceph-mon[97633]: pgmap v17380: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:32:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:25.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:25.742653+0000) 2022-09-10T13:32:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:26.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:26.127748+0000) 2022-09-10T13:32:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:26 smithi104 ceph-mon[97044]: pgmap v17381: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:32:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:26.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:26.742834+0000) 2022-09-10T13:32:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:26 smithi110 ceph-mon[97633]: pgmap v17381: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:32:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:27.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:27.128028+0000) 2022-09-10T13:32:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:27.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:27.743126+0000) 2022-09-10T13:32:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:28.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:28.128266+0000) 2022-09-10T13:32:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:28.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:28.743292+0000) 2022-09-10T13:32:29.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:29 smithi104 ceph-mon[97044]: pgmap v17382: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:29.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:29.128417+0000) 2022-09-10T13:32:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:29 smithi110 ceph-mon[97633]: pgmap v17382: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:29.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:29.743476+0000) 2022-09-10T13:32:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:30.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:30.128655+0000) 2022-09-10T13:32:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:30.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:30.743653+0000) 2022-09-10T13:32:31.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:31 smithi104 ceph-mon[97044]: pgmap v17383: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:31.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:31.128883+0000) 2022-09-10T13:32:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:31 smithi110 ceph-mon[97633]: pgmap v17383: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:31.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:31.743868+0000) 2022-09-10T13:32:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:32.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:32.129121+0000) 2022-09-10T13:32:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:32.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:32.744119+0000) 2022-09-10T13:32:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:32:32.625+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:32:33.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:33 smithi104 ceph-mon[97044]: pgmap v17384: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:33.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:33.129371+0000) 2022-09-10T13:32:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:33 smithi110 ceph-mon[97633]: pgmap v17384: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:33.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:33.744286+0000) 2022-09-10T13:32:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:34.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:34.129624+0000) 2022-09-10T13:32:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:34.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:34.744444+0000) 2022-09-10T13:32:35.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:35 smithi104 ceph-mon[97044]: pgmap v17385: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:35.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:35.129803+0000) 2022-09-10T13:32:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:35 smithi110 ceph-mon[97633]: pgmap v17385: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:36.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:35.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:35.744729+0000) 2022-09-10T13:32:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:36.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:36.130027+0000) 2022-09-10T13:32:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:36 smithi104 ceph-mon[97044]: pgmap v17386: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:32:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:36.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:36.744982+0000) 2022-09-10T13:32:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:36 smithi110 ceph-mon[97633]: pgmap v17386: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:32:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:37.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:37.130276+0000) 2022-09-10T13:32:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:37.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:37.745198+0000) 2022-09-10T13:32:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:38.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:38.130550+0000) 2022-09-10T13:32:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:38.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:38.745407+0000) 2022-09-10T13:32:39.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:39 smithi104 ceph-mon[97044]: pgmap v17387: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:32:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:39.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:39.130798+0000) 2022-09-10T13:32:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:39 smithi110 ceph-mon[97633]: pgmap v17387: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:32:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:39.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:39.745647+0000) 2022-09-10T13:32:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:40.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:40.131048+0000) 2022-09-10T13:32:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:40.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:40.745931+0000) 2022-09-10T13:32:41.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:41 smithi104 ceph-mon[97044]: pgmap v17388: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:32:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:41.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:41.131349+0000) 2022-09-10T13:32:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:41 smithi110 ceph-mon[97633]: pgmap v17388: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:32:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:41.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:41.746205+0000) 2022-09-10T13:32:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:42.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:42.131605+0000) 2022-09-10T13:32:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:42.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:42.746388+0000) 2022-09-10T13:32:43.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:43 smithi104 ceph-mon[97044]: pgmap v17389: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:43.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:43.131851+0000) 2022-09-10T13:32:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:43 smithi110 ceph-mon[97633]: pgmap v17389: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:43.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:43.746533+0000) 2022-09-10T13:32:44.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:44.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:44.132077+0000) 2022-09-10T13:32:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:44.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:44.746740+0000) 2022-09-10T13:32:45.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:45 smithi104 ceph-mon[97044]: pgmap v17390: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:45.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:45.132371+0000) 2022-09-10T13:32:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:45 smithi110 ceph-mon[97633]: pgmap v17390: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:32:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:45.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:45.747000+0000) 2022-09-10T13:32:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:46.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:46.132590+0000) 2022-09-10T13:32:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:46 smithi104 ceph-mon[97044]: pgmap v17391: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:32:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:32:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:32:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:32:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:46.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:46.747218+0000) 2022-09-10T13:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:46 smithi110 ceph-mon[97633]: pgmap v17391: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:32:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:32:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:32:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:32:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:47.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:47.132832+0000) 2022-09-10T13:32:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:47.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:47.747441+0000) 2022-09-10T13:32:48.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:32:47.633+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:32:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:48.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:48.133050+0000) 2022-09-10T13:32:49.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:48.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:48.747659+0000) 2022-09-10T13:32:49.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:49 smithi104 ceph-mon[97044]: pgmap v17392: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:49.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:49.133266+0000) 2022-09-10T13:32:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:49 smithi110 ceph-mon[97633]: pgmap v17392: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:49.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:49.747882+0000) 2022-09-10T13:32:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:50.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:50.133480+0000) 2022-09-10T13:32:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:50.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:50.748189+0000) 2022-09-10T13:32:51.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:51 smithi104 ceph-mon[97044]: pgmap v17393: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:51.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:51.133707+0000) 2022-09-10T13:32:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:51 smithi110 ceph-mon[97633]: pgmap v17393: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:51.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:51.748365+0000) 2022-09-10T13:32:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:52.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:52.133915+0000) 2022-09-10T13:32:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:52.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:52.748490+0000) 2022-09-10T13:32:53.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:53 smithi104 ceph-mon[97044]: pgmap v17394: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:53.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:53.134186+0000) 2022-09-10T13:32:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:53 smithi110 ceph-mon[97633]: pgmap v17394: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:53.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:53.748627+0000) 2022-09-10T13:32:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:54.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:54.134341+0000) 2022-09-10T13:32:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:54.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:54.748805+0000) 2022-09-10T13:32:55.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:55 smithi104 ceph-mon[97044]: pgmap v17395: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:55.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:55.134499+0000) 2022-09-10T13:32:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:55 smithi110 ceph-mon[97633]: pgmap v17395: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:55.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:55.749001+0000) 2022-09-10T13:32:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:56.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:56.134737+0000) 2022-09-10T13:32:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:56 smithi104 ceph-mon[97044]: pgmap v17396: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:56.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:56.749265+0000) 2022-09-10T13:32:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:56 smithi110 ceph-mon[97633]: pgmap v17396: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:32:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:57.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:57.134995+0000) 2022-09-10T13:32:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:57.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:57.749460+0000) 2022-09-10T13:32:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:58.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:58.135238+0000) 2022-09-10T13:32:59.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:58.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:58.749676+0000) 2022-09-10T13:32:59.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:32:59 smithi104 ceph-mon[97044]: pgmap v17397: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:32:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:32:59.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:59.135391+0000) 2022-09-10T13:32:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:32:59 smithi110 ceph-mon[97633]: pgmap v17397: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:00.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:32:59.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:32:59.749932+0000) 2022-09-10T13:33:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:00.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:00.135595+0000) 2022-09-10T13:33:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:00.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:00.750121+0000) 2022-09-10T13:33:01.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:01 smithi104 ceph-mon[97044]: pgmap v17398: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:01.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:01.135786+0000) 2022-09-10T13:33:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:01 smithi110 ceph-mon[97633]: pgmap v17398: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:01.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:01.750316+0000) 2022-09-10T13:33:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:02.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:02.135992+0000) 2022-09-10T13:33:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:02.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:02.750519+0000) 2022-09-10T13:33:03.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:33:02.645+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:33:03.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:03 smithi104 ceph-mon[97044]: pgmap v17399: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:03.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:03.136245+0000) 2022-09-10T13:33:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:03 smithi110 ceph-mon[97633]: pgmap v17399: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:03.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:03.750763+0000) 2022-09-10T13:33:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:04.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:04.136467+0000) 2022-09-10T13:33:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:04.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:04.750918+0000) 2022-09-10T13:33:05.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:05 smithi104 ceph-mon[97044]: pgmap v17400: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:05.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:05.136709+0000) 2022-09-10T13:33:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:05 smithi110 ceph-mon[97633]: pgmap v17400: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:05.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:05.751159+0000) 2022-09-10T13:33:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:06.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:06.136958+0000) 2022-09-10T13:33:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:06 smithi104 ceph-mon[97044]: pgmap v17401: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:33:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:06.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:06.751341+0000) 2022-09-10T13:33:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:06 smithi110 ceph-mon[97633]: pgmap v17401: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:33:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:07.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:07.137215+0000) 2022-09-10T13:33:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:07.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:07.751572+0000) 2022-09-10T13:33:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:08.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:08.137452+0000) 2022-09-10T13:33:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:08.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:08.751803+0000) 2022-09-10T13:33:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:08 smithi110 ceph-mon[97633]: osdmap e1168: 12 total, 12 up, 12 in 2022-09-10T13:33:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:08 smithi110 ceph-mon[97633]: pgmap v17403: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:33:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:08 smithi104 ceph-mon[97044]: osdmap e1168: 12 total, 12 up, 12 in 2022-09-10T13:33:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:08 smithi104 ceph-mon[97044]: pgmap v17403: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:33:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:09.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:09.137589+0000) 2022-09-10T13:33:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:09.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:09.751971+0000) 2022-09-10T13:33:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:10.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:10.137862+0000) 2022-09-10T13:33:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:10.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:10.752150+0000) 2022-09-10T13:33:11.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:11 smithi104 ceph-mon[97044]: pgmap v17404: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:33:11.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:11 smithi104 ceph-mon[97044]: osdmap e1169: 12 total, 12 up, 12 in 2022-09-10T13:33:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:11.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:11.138129+0000) 2022-09-10T13:33:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:11 smithi110 ceph-mon[97633]: pgmap v17404: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:33:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:11 smithi110 ceph-mon[97633]: osdmap e1169: 12 total, 12 up, 12 in 2022-09-10T13:33:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:11.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:11.752286+0000) 2022-09-10T13:33:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:12.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:12.138334+0000) 2022-09-10T13:33:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:12.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:12.752554+0000) 2022-09-10T13:33:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:13 smithi104 ceph-mon[97044]: pgmap v17406: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:33:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:13.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:13.138607+0000) 2022-09-10T13:33:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:13 smithi110 ceph-mon[97633]: pgmap v17406: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:33:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:13.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:13.752794+0000) 2022-09-10T13:33:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:14.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:14.138836+0000) 2022-09-10T13:33:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:14.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:14.753050+0000) 2022-09-10T13:33:15.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:15 smithi104 ceph-mon[97044]: pgmap v17407: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:33:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:15.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:15.139067+0000) 2022-09-10T13:33:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:15 smithi110 ceph-mon[97633]: pgmap v17407: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:33:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:15.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:15.753326+0000) 2022-09-10T13:33:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:16.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:16.139295+0000) 2022-09-10T13:33:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:16 smithi104 ceph-mon[97044]: pgmap v17408: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T13:33:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:16.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:16.753493+0000) 2022-09-10T13:33:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:16 smithi110 ceph-mon[97633]: pgmap v17408: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T13:33:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:17.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:17.139580+0000) 2022-09-10T13:33:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:17.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:17.753686+0000) 2022-09-10T13:33:18.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:33:17.647+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:33:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:18.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:18.139759+0000) 2022-09-10T13:33:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:18.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:18.753857+0000) 2022-09-10T13:33:19.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:19 smithi104 ceph-mon[97044]: pgmap v17409: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:33:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:19.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:19.139947+0000) 2022-09-10T13:33:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:19 smithi110 ceph-mon[97633]: pgmap v17409: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:33:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:19.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:19.754083+0000) 2022-09-10T13:33:20.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:20.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:20.140204+0000) 2022-09-10T13:33:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:20.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:20.754398+0000) 2022-09-10T13:33:21.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:21 smithi104 ceph-mon[97044]: pgmap v17410: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:33:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:21.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:21.140436+0000) 2022-09-10T13:33:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:21 smithi110 ceph-mon[97633]: pgmap v17410: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:33:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:21.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:21.754620+0000) 2022-09-10T13:33:22.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:22.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:22.140662+0000) 2022-09-10T13:33:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:22.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:22.754883+0000) 2022-09-10T13:33:23.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:23.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:23.140910+0000) 2022-09-10T13:33:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:23 smithi110 ceph-mon[97633]: pgmap v17411: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:33:23.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:23 smithi104 ceph-mon[97044]: pgmap v17411: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:33:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:23.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:23.755131+0000) 2022-09-10T13:33:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:24.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:24.141048+0000) 2022-09-10T13:33:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:24.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:24.755315+0000) 2022-09-10T13:33:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:25.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:25.141214+0000) 2022-09-10T13:33:25.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:25 smithi110 ceph-mon[97633]: pgmap v17412: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:33:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:25 smithi104 ceph-mon[97044]: pgmap v17412: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:33:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:25.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:25.755509+0000) 2022-09-10T13:33:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:26.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:26.141420+0000) 2022-09-10T13:33:27.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:26 smithi104 ceph-mon[97044]: pgmap v17413: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:33:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:26.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:26.755731+0000) 2022-09-10T13:33:27.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:26 smithi110 ceph-mon[97633]: pgmap v17413: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:33:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:27.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:27.141669+0000) 2022-09-10T13:33:28.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:27.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:27.755980+0000) 2022-09-10T13:33:28.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:28.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:28.141910+0000) 2022-09-10T13:33:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:28.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:28.756249+0000) 2022-09-10T13:33:29.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:29 smithi104 ceph-mon[97044]: pgmap v17414: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:29.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:29.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:29.142194+0000) 2022-09-10T13:33:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:29 smithi110 ceph-mon[97633]: pgmap v17414: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:30.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:29.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:29.756461+0000) 2022-09-10T13:33:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:30.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:30.142420+0000) 2022-09-10T13:33:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:30.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:30.756736+0000) 2022-09-10T13:33:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:31.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:31.142618+0000) 2022-09-10T13:33:31.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:31 smithi110 ceph-mon[97633]: pgmap v17415: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:31 smithi104 ceph-mon[97044]: pgmap v17415: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:32.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:31.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:31.756898+0000) 2022-09-10T13:33:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:32.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:32.142842+0000) 2022-09-10T13:33:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:32.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:32.757189+0000) 2022-09-10T13:33:33.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:33:32.661+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:33:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:33.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:33.143136+0000) 2022-09-10T13:33:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:33 smithi110 ceph-mon[97633]: pgmap v17416: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:33 smithi104 ceph-mon[97044]: pgmap v17416: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:33.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:33.757406+0000) 2022-09-10T13:33:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:34.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:34.143369+0000) 2022-09-10T13:33:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:34.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:34.757654+0000) 2022-09-10T13:33:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:35.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:35.143546+0000) 2022-09-10T13:33:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:35 smithi110 ceph-mon[97633]: pgmap v17417: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:35 smithi104 ceph-mon[97044]: pgmap v17417: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:35.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:35.757916+0000) 2022-09-10T13:33:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:36.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:36.143782+0000) 2022-09-10T13:33:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:36 smithi104 ceph-mon[97044]: pgmap v17418: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:36.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:36.758155+0000) 2022-09-10T13:33:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:36 smithi110 ceph-mon[97633]: pgmap v17418: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:37.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:37.143990+0000) 2022-09-10T13:33:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:37.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:37.758320+0000) 2022-09-10T13:33:38.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:38.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:38.144243+0000) 2022-09-10T13:33:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:38.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:38.758457+0000) 2022-09-10T13:33:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:39.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:39.144530+0000) 2022-09-10T13:33:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:39 smithi110 ceph-mon[97633]: pgmap v17419: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:39 smithi104 ceph-mon[97044]: pgmap v17419: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:39.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:39.758702+0000) 2022-09-10T13:33:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:40.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:40.144772+0000) 2022-09-10T13:33:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:40.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:40.758929+0000) 2022-09-10T13:33:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:41.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:41.145024+0000) 2022-09-10T13:33:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:41 smithi110 ceph-mon[97633]: pgmap v17420: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:41 smithi104 ceph-mon[97044]: pgmap v17420: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:41.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:41.759184+0000) 2022-09-10T13:33:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:42.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:42.145274+0000) 2022-09-10T13:33:43.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:42.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:42.759404+0000) 2022-09-10T13:33:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:43.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:43.145493+0000) 2022-09-10T13:33:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:43 smithi110 ceph-mon[97633]: pgmap v17421: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:43 smithi104 ceph-mon[97044]: pgmap v17421: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:44.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:43.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:43.759610+0000) 2022-09-10T13:33:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:44.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:44.145732+0000) 2022-09-10T13:33:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:44.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:44.759804+0000) 2022-09-10T13:33:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:45.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:45.145895+0000) 2022-09-10T13:33:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:45 smithi110 ceph-mon[97633]: pgmap v17422: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:45 smithi104 ceph-mon[97044]: pgmap v17422: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:45.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:45.760018+0000) 2022-09-10T13:33:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:46.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:46.146117+0000) 2022-09-10T13:33:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:46 smithi104 ceph-mon[97044]: pgmap v17423: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:33:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:33:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:33:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:33:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:46.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:46.760221+0000) 2022-09-10T13:33:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:46 smithi110 ceph-mon[97633]: pgmap v17423: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:33:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:33:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:33:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:33:47.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:47.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:47.146375+0000) 2022-09-10T13:33:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:47.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:47.760433+0000) 2022-09-10T13:33:48.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:33:47.677+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:33:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:48.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:48.146614+0000) 2022-09-10T13:33:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:48.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:48.760583+0000) 2022-09-10T13:33:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:49.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:49.146851+0000) 2022-09-10T13:33:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:49 smithi110 ceph-mon[97633]: pgmap v17424: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:49 smithi104 ceph-mon[97044]: pgmap v17424: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:49.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:49.760816+0000) 2022-09-10T13:33:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:50.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:50.147089+0000) 2022-09-10T13:33:51.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:50.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:50.761074+0000) 2022-09-10T13:33:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:51.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:51.147364+0000) 2022-09-10T13:33:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:51 smithi110 ceph-mon[97633]: pgmap v17425: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:51 smithi104 ceph-mon[97044]: pgmap v17425: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:51.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:51.761304+0000) 2022-09-10T13:33:52.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:52.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:52.147639+0000) 2022-09-10T13:33:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:52.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:52.761501+0000) 2022-09-10T13:33:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:53.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:53.147863+0000) 2022-09-10T13:33:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:53 smithi110 ceph-mon[97633]: pgmap v17426: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:53 smithi104 ceph-mon[97044]: pgmap v17426: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:53.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:53.761698+0000) 2022-09-10T13:33:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:54.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:54.148150+0000) 2022-09-10T13:33:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:54.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:54.761881+0000) 2022-09-10T13:33:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:55.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:55.148344+0000) 2022-09-10T13:33:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:55 smithi110 ceph-mon[97633]: pgmap v17427: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:55 smithi104 ceph-mon[97044]: pgmap v17427: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:55.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:55.762117+0000) 2022-09-10T13:33:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:56.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:56.148578+0000) 2022-09-10T13:33:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:56 smithi104 ceph-mon[97044]: pgmap v17428: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:56.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:56.762360+0000) 2022-09-10T13:33:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:56 smithi110 ceph-mon[97633]: pgmap v17428: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:33:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:57.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:57.148838+0000) 2022-09-10T13:33:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:57.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:57.762567+0000) 2022-09-10T13:33:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:58.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:58.149147+0000) 2022-09-10T13:33:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:58.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:58.762754+0000) 2022-09-10T13:33:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:33:59.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:59.149361+0000) 2022-09-10T13:33:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:33:59 smithi110 ceph-mon[97633]: pgmap v17429: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:33:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:33:59 smithi104 ceph-mon[97044]: pgmap v17429: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:33:59.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:33:59.762951+0000) 2022-09-10T13:34:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:00.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:00.149561+0000) 2022-09-10T13:34:01.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:00.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:00.763203+0000) 2022-09-10T13:34:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:01.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:01.149751+0000) 2022-09-10T13:34:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:01 smithi110 ceph-mon[97633]: pgmap v17430: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:01 smithi104 ceph-mon[97044]: pgmap v17430: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:01.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:01.763322+0000) 2022-09-10T13:34:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:02.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:02.149932+0000) 2022-09-10T13:34:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:02.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:02.763496+0000) 2022-09-10T13:34:03.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:34:02.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:34:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:03.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:03.150172+0000) 2022-09-10T13:34:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:03 smithi110 ceph-mon[97633]: pgmap v17431: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:03 smithi104 ceph-mon[97044]: pgmap v17431: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:03.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:03.763709+0000) 2022-09-10T13:34:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:04.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:04.150317+0000) 2022-09-10T13:34:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:04.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:04.763872+0000) 2022-09-10T13:34:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:05.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:05.150560+0000) 2022-09-10T13:34:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:05 smithi110 ceph-mon[97633]: pgmap v17432: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:05 smithi104 ceph-mon[97044]: pgmap v17432: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:05.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:05.764027+0000) 2022-09-10T13:34:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:06.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:06.150813+0000) 2022-09-10T13:34:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:06 smithi104 ceph-mon[97044]: pgmap v17433: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:34:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:06.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:06.764205+0000) 2022-09-10T13:34:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:06 smithi110 ceph-mon[97633]: pgmap v17433: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:34:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:07.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:07.151060+0000) 2022-09-10T13:34:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:07.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:07.764395+0000) 2022-09-10T13:34:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:08.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:08.151295+0000) 2022-09-10T13:34:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:08.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:08.764651+0000) 2022-09-10T13:34:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:08 smithi110 ceph-mon[97633]: osdmap e1170: 12 total, 12 up, 12 in 2022-09-10T13:34:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:08 smithi110 ceph-mon[97633]: pgmap v17435: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:34:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:08 smithi104 ceph-mon[97044]: osdmap e1170: 12 total, 12 up, 12 in 2022-09-10T13:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:08 smithi104 ceph-mon[97044]: pgmap v17435: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:34:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:09.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:09.151467+0000) 2022-09-10T13:34:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:09.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:09.764805+0000) 2022-09-10T13:34:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:10.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:10.151735+0000) 2022-09-10T13:34:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:10.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:10.764943+0000) 2022-09-10T13:34:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:11.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:11.152048+0000) 2022-09-10T13:34:11.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:11 smithi110 ceph-mon[97633]: pgmap v17436: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:34:11.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:11 smithi110 ceph-mon[97633]: osdmap e1171: 12 total, 12 up, 12 in 2022-09-10T13:34:11.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:11 smithi104 ceph-mon[97044]: pgmap v17436: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:34:11.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:11 smithi104 ceph-mon[97044]: osdmap e1171: 12 total, 12 up, 12 in 2022-09-10T13:34:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:11.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:11.765206+0000) 2022-09-10T13:34:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:12.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:12.152270+0000) 2022-09-10T13:34:13.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:12.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:12.765358+0000) 2022-09-10T13:34:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:13.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:13.152481+0000) 2022-09-10T13:34:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:13 smithi110 ceph-mon[97633]: pgmap v17438: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:34:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:13 smithi104 ceph-mon[97044]: pgmap v17438: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 35 op/s 2022-09-10T13:34:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:13.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:13.765559+0000) 2022-09-10T13:34:14.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:14.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:14.152737+0000) 2022-09-10T13:34:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:14.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:14.765781+0000) 2022-09-10T13:34:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:15.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:15.152981+0000) 2022-09-10T13:34:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:15 smithi110 ceph-mon[97633]: pgmap v17439: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:34:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:15 smithi104 ceph-mon[97044]: pgmap v17439: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:34:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:15.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:15.766060+0000) 2022-09-10T13:34:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:16.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:16.153203+0000) 2022-09-10T13:34:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:16 smithi104 ceph-mon[97044]: pgmap v17440: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T13:34:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:16.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:16.766336+0000) 2022-09-10T13:34:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:16 smithi110 ceph-mon[97633]: pgmap v17440: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T13:34:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:17.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:17.153456+0000) 2022-09-10T13:34:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:17.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:17.766574+0000) 2022-09-10T13:34:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:34:17.691+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:34:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:18.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:18.153624+0000) 2022-09-10T13:34:19.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:18.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:18.766763+0000) 2022-09-10T13:34:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:19.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:19.153836+0000) 2022-09-10T13:34:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:19 smithi110 ceph-mon[97633]: pgmap v17441: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:34:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:19 smithi104 ceph-mon[97044]: pgmap v17441: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:34:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:19.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:19.766982+0000) 2022-09-10T13:34:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:20.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:20.154055+0000) 2022-09-10T13:34:21.059 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:20.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:20.767161+0000) 2022-09-10T13:34:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:21.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:21.154306+0000) 2022-09-10T13:34:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:21 smithi110 ceph-mon[97633]: pgmap v17442: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:34:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:21 smithi104 ceph-mon[97044]: pgmap v17442: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:34:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:21.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:21.767348+0000) 2022-09-10T13:34:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:22.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:22.154480+0000) 2022-09-10T13:34:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:22.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:22.767517+0000) 2022-09-10T13:34:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:23.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:23.154683+0000) 2022-09-10T13:34:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:23 smithi110 ceph-mon[97633]: pgmap v17443: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:34:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:23 smithi104 ceph-mon[97044]: pgmap v17443: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:34:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:23.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:23.767734+0000) 2022-09-10T13:34:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:24.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:24.154923+0000) 2022-09-10T13:34:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:24.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:24.767980+0000) 2022-09-10T13:34:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:25.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:25.155152+0000) 2022-09-10T13:34:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:25 smithi110 ceph-mon[97633]: pgmap v17444: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:34:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:25 smithi104 ceph-mon[97044]: pgmap v17444: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:34:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:25.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:25.768191+0000) 2022-09-10T13:34:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:26.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:26.155388+0000) 2022-09-10T13:34:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:26 smithi104 ceph-mon[97044]: pgmap v17445: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:34:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:26.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:26.768433+0000) 2022-09-10T13:34:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:26 smithi110 ceph-mon[97633]: pgmap v17445: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:34:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:27.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:27.155603+0000) 2022-09-10T13:34:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:27.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:27.768712+0000) 2022-09-10T13:34:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:28.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:28.155823+0000) 2022-09-10T13:34:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:28.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:28.768931+0000) 2022-09-10T13:34:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:29.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:29.155998+0000) 2022-09-10T13:34:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:29 smithi110 ceph-mon[97633]: pgmap v17446: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:29 smithi104 ceph-mon[97044]: pgmap v17446: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:29.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:29.769161+0000) 2022-09-10T13:34:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:30.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:30.156205+0000) 2022-09-10T13:34:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:30.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:30.769374+0000) 2022-09-10T13:34:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:31.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:31.156404+0000) 2022-09-10T13:34:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:31 smithi110 ceph-mon[97633]: pgmap v17447: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:31 smithi104 ceph-mon[97044]: pgmap v17447: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:31.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:31.769543+0000) 2022-09-10T13:34:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:32.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:32.156659+0000) 2022-09-10T13:34:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:32.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:32.769766+0000) 2022-09-10T13:34:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:34:32.707+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:34:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:33.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:33.156896+0000) 2022-09-10T13:34:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:33 smithi110 ceph-mon[97633]: pgmap v17448: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:33 smithi104 ceph-mon[97044]: pgmap v17448: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:33.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:33.769988+0000) 2022-09-10T13:34:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:34.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:34.157170+0000) 2022-09-10T13:34:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:34.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:34.770122+0000) 2022-09-10T13:34:35.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:35.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:35.157395+0000) 2022-09-10T13:34:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:35 smithi110 ceph-mon[97633]: pgmap v17449: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:35 smithi104 ceph-mon[97044]: pgmap v17449: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:35.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:35.770318+0000) 2022-09-10T13:34:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:36.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:36.157568+0000) 2022-09-10T13:34:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:36 smithi104 ceph-mon[97044]: pgmap v17450: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:36.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:36.770560+0000) 2022-09-10T13:34:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:36 smithi110 ceph-mon[97633]: pgmap v17450: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:37.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:37.157770+0000) 2022-09-10T13:34:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:37.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:37.770807+0000) 2022-09-10T13:34:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:38.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:38.158002+0000) 2022-09-10T13:34:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:38.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:38.771038+0000) 2022-09-10T13:34:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:39.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:39.158224+0000) 2022-09-10T13:34:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:39 smithi110 ceph-mon[97633]: pgmap v17451: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:39 smithi104 ceph-mon[97044]: pgmap v17451: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:39.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:39.771297+0000) 2022-09-10T13:34:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:40.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:40.158479+0000) 2022-09-10T13:34:41.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:40.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:40.771546+0000) 2022-09-10T13:34:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:41.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:41.158676+0000) 2022-09-10T13:34:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:41 smithi110 ceph-mon[97633]: pgmap v17452: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:41 smithi104 ceph-mon[97044]: pgmap v17452: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:41.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:41.771810+0000) 2022-09-10T13:34:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:42.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:42.158913+0000) 2022-09-10T13:34:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:42.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:42.772058+0000) 2022-09-10T13:34:43.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:43.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:43.159175+0000) 2022-09-10T13:34:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:43 smithi110 ceph-mon[97633]: pgmap v17453: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:43 smithi104 ceph-mon[97044]: pgmap v17453: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:43.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:43.772284+0000) 2022-09-10T13:34:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:44.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:44.159349+0000) 2022-09-10T13:34:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:44.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:44.772403+0000) 2022-09-10T13:34:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:45.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:45.159569+0000) 2022-09-10T13:34:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:45 smithi110 ceph-mon[97633]: pgmap v17454: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:45 smithi104 ceph-mon[97044]: pgmap v17454: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:45.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:45.772595+0000) 2022-09-10T13:34:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:46.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:46.159795+0000) 2022-09-10T13:34:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:46 smithi104 ceph-mon[97044]: pgmap v17455: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:34:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:34:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:34:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:34:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:46.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:46.772794+0000) 2022-09-10T13:34:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:46 smithi110 ceph-mon[97633]: pgmap v17455: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:34:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:34:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:34:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:34:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:47.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:47.160046+0000) 2022-09-10T13:34:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:47.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:47.773021+0000) 2022-09-10T13:34:48.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:34:47.711+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:34:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:48.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:48.160266+0000) 2022-09-10T13:34:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:48.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:48.773269+0000) 2022-09-10T13:34:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:49.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:49.160467+0000) 2022-09-10T13:34:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:49 smithi110 ceph-mon[97633]: pgmap v17456: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:49 smithi104 ceph-mon[97044]: pgmap v17456: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:49.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:49.773478+0000) 2022-09-10T13:34:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:50.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:50.160708+0000) 2022-09-10T13:34:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:50.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:50.773744+0000) 2022-09-10T13:34:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:51.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:51.160958+0000) 2022-09-10T13:34:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:51 smithi110 ceph-mon[97633]: pgmap v17457: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:51 smithi104 ceph-mon[97044]: pgmap v17457: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:51.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:51.773972+0000) 2022-09-10T13:34:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:52.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:52.161202+0000) 2022-09-10T13:34:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:52.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:52.774122+0000) 2022-09-10T13:34:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:53.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:53.161435+0000) 2022-09-10T13:34:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:53 smithi110 ceph-mon[97633]: pgmap v17458: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:53 smithi104 ceph-mon[97044]: pgmap v17458: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:53.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:53.774270+0000) 2022-09-10T13:34:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:54.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:54.161660+0000) 2022-09-10T13:34:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:54.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:54.774477+0000) 2022-09-10T13:34:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:55.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:55.161887+0000) 2022-09-10T13:34:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:55 smithi110 ceph-mon[97633]: pgmap v17459: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:55 smithi104 ceph-mon[97044]: pgmap v17459: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:55.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:55.774660+0000) 2022-09-10T13:34:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:56.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:56.162137+0000) 2022-09-10T13:34:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:56 smithi104 ceph-mon[97044]: pgmap v17460: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:56.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:56.774817+0000) 2022-09-10T13:34:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:56 smithi110 ceph-mon[97633]: pgmap v17460: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:34:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:57.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:57.162353+0000) 2022-09-10T13:34:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:57.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:57.775012+0000) 2022-09-10T13:34:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:58.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:58.162500+0000) 2022-09-10T13:34:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:58.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:58.775260+0000) 2022-09-10T13:34:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:34:59.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:59.162658+0000) 2022-09-10T13:34:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:34:59 smithi110 ceph-mon[97633]: pgmap v17461: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:34:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:34:59 smithi104 ceph-mon[97044]: pgmap v17461: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:34:59.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:34:59.775520+0000) 2022-09-10T13:35:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:00.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:00.162824+0000) 2022-09-10T13:35:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:00.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:00.775734+0000) 2022-09-10T13:35:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:01.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:01.162967+0000) 2022-09-10T13:35:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:01 smithi110 ceph-mon[97633]: pgmap v17462: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:01 smithi104 ceph-mon[97044]: pgmap v17462: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:01.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:01.775902+0000) 2022-09-10T13:35:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:02.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:02.163200+0000) 2022-09-10T13:35:03.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:35:02.723+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:35:03.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:02.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:02.776175+0000) 2022-09-10T13:35:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:03.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:03.163414+0000) 2022-09-10T13:35:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:03 smithi110 ceph-mon[97633]: pgmap v17463: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:03 smithi104 ceph-mon[97044]: pgmap v17463: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:03.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:03.776405+0000) 2022-09-10T13:35:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:04.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:04.163647+0000) 2022-09-10T13:35:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:04.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:04.776601+0000) 2022-09-10T13:35:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:05.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:05.163878+0000) 2022-09-10T13:35:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:05 smithi110 ceph-mon[97633]: pgmap v17464: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:05 smithi104 ceph-mon[97044]: pgmap v17464: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:05.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:05.776798+0000) 2022-09-10T13:35:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:06.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:06.164174+0000) 2022-09-10T13:35:07.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:06 smithi104 ceph-mon[97044]: pgmap v17465: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:35:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:06.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:06.776998+0000) 2022-09-10T13:35:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:06 smithi110 ceph-mon[97633]: pgmap v17465: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:35:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:07.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:07.164317+0000) 2022-09-10T13:35:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:07.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:07.777218+0000) 2022-09-10T13:35:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:08.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:08.164532+0000) 2022-09-10T13:35:09.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:08 smithi104 ceph-mon[97044]: osdmap e1172: 12 total, 12 up, 12 in 2022-09-10T13:35:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:08 smithi104 ceph-mon[97044]: pgmap v17467: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:35:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:08.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:08.777488+0000) 2022-09-10T13:35:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:08 smithi110 ceph-mon[97633]: osdmap e1172: 12 total, 12 up, 12 in 2022-09-10T13:35:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:08 smithi110 ceph-mon[97633]: pgmap v17467: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:35:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:09.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:09.164687+0000) 2022-09-10T13:35:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:09.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:09.777723+0000) 2022-09-10T13:35:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:10.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:10.164880+0000) 2022-09-10T13:35:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:10.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:10.777890+0000) 2022-09-10T13:35:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:11.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:11.165088+0000) 2022-09-10T13:35:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:11 smithi110 ceph-mon[97633]: pgmap v17468: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:35:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:11 smithi110 ceph-mon[97633]: osdmap e1173: 12 total, 12 up, 12 in 2022-09-10T13:35:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:11 smithi104 ceph-mon[97044]: pgmap v17468: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:35:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:11 smithi104 ceph-mon[97044]: osdmap e1173: 12 total, 12 up, 12 in 2022-09-10T13:35:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:11.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:11.778072+0000) 2022-09-10T13:35:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:12.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:12.165335+0000) 2022-09-10T13:35:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:12.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:12.778323+0000) 2022-09-10T13:35:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:13.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:13.165583+0000) 2022-09-10T13:35:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:13 smithi110 ceph-mon[97633]: pgmap v17470: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:35:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:13 smithi104 ceph-mon[97044]: pgmap v17470: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:35:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:13.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:13.778543+0000) 2022-09-10T13:35:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:14.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:14.165804+0000) 2022-09-10T13:35:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:14.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:14.778779+0000) 2022-09-10T13:35:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:15.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:15.166035+0000) 2022-09-10T13:35:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:15 smithi110 ceph-mon[97633]: pgmap v17471: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:35:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:15 smithi104 ceph-mon[97044]: pgmap v17471: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:35:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:15.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:15.778974+0000) 2022-09-10T13:35:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:16.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:16.166260+0000) 2022-09-10T13:35:17.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:16 smithi104 ceph-mon[97044]: pgmap v17472: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:35:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:16.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:16.779207+0000) 2022-09-10T13:35:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:16 smithi110 ceph-mon[97633]: pgmap v17472: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:35:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:17.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:17.166500+0000) 2022-09-10T13:35:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:17.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:17.779460+0000) 2022-09-10T13:35:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:35:17.739+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:35:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:18.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:18.166648+0000) 2022-09-10T13:35:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:18.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:18.779683+0000) 2022-09-10T13:35:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:19.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:19.166835+0000) 2022-09-10T13:35:19.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:19 smithi110 ceph-mon[97633]: pgmap v17473: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:35:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:19 smithi104 ceph-mon[97044]: pgmap v17473: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:35:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:19.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:19.779901+0000) 2022-09-10T13:35:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:20.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:20.166994+0000) 2022-09-10T13:35:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:20.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:20.780171+0000) 2022-09-10T13:35:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:21.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:21.167214+0000) 2022-09-10T13:35:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:21 smithi110 ceph-mon[97633]: pgmap v17474: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:35:21.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:21 smithi104 ceph-mon[97044]: pgmap v17474: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:35:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:21.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:21.780359+0000) 2022-09-10T13:35:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:22.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:22.167420+0000) 2022-09-10T13:35:23.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:22.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:22.780533+0000) 2022-09-10T13:35:23.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:23.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:23.167581+0000) 2022-09-10T13:35:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:23 smithi110 ceph-mon[97633]: pgmap v17475: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:35:23.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:23 smithi104 ceph-mon[97044]: pgmap v17475: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:35:24.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:23.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:23.780674+0000) 2022-09-10T13:35:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:24.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:24.167758+0000) 2022-09-10T13:35:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:24.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:24.780831+0000) 2022-09-10T13:35:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:25.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:25.168018+0000) 2022-09-10T13:35:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:25 smithi110 ceph-mon[97633]: pgmap v17476: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:35:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:25 smithi104 ceph-mon[97044]: pgmap v17476: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:35:26.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:25.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:25.781033+0000) 2022-09-10T13:35:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:26.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:26.168225+0000) 2022-09-10T13:35:27.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:26 smithi104 ceph-mon[97044]: pgmap v17477: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:35:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:26.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:26.781182+0000) 2022-09-10T13:35:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:26 smithi110 ceph-mon[97633]: pgmap v17477: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:35:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:27.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:27.168492+0000) 2022-09-10T13:35:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:27.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:27.781382+0000) 2022-09-10T13:35:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:28.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:28.168674+0000) 2022-09-10T13:35:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:28.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:28.781567+0000) 2022-09-10T13:35:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:29.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:29.168879+0000) 2022-09-10T13:35:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:29 smithi110 ceph-mon[97633]: pgmap v17478: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:29 smithi104 ceph-mon[97044]: pgmap v17478: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:29.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:29.781742+0000) 2022-09-10T13:35:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:30.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:30.169131+0000) 2022-09-10T13:35:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:30.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:30.781970+0000) 2022-09-10T13:35:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:31.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:31.169325+0000) 2022-09-10T13:35:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:31 smithi110 ceph-mon[97633]: pgmap v17479: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:31 smithi104 ceph-mon[97044]: pgmap v17479: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:31.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:31.782209+0000) 2022-09-10T13:35:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:32.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:32.169556+0000) 2022-09-10T13:35:33.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:32.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:32.782423+0000) 2022-09-10T13:35:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:35:32.752+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:35:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:33.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:33.169789+0000) 2022-09-10T13:35:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:33 smithi110 ceph-mon[97633]: pgmap v17480: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:33 smithi104 ceph-mon[97044]: pgmap v17480: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:34.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:33.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:33.782545+0000) 2022-09-10T13:35:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:34.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:34.170034+0000) 2022-09-10T13:35:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:34.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:34.782734+0000) 2022-09-10T13:35:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:35.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:35.170285+0000) 2022-09-10T13:35:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:35 smithi110 ceph-mon[97633]: pgmap v17481: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:35 smithi104 ceph-mon[97044]: pgmap v17481: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:35.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:35.782921+0000) 2022-09-10T13:35:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:36.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:36.170485+0000) 2022-09-10T13:35:37.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:36 smithi104 ceph-mon[97044]: pgmap v17482: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:37.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:36.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:36.783154+0000) 2022-09-10T13:35:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:36 smithi110 ceph-mon[97633]: pgmap v17482: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:37.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:37.170740+0000) 2022-09-10T13:35:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:37.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:37.783393+0000) 2022-09-10T13:35:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:38.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:38.170976+0000) 2022-09-10T13:35:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:38.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:38.783603+0000) 2022-09-10T13:35:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:39.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:39.171196+0000) 2022-09-10T13:35:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:39 smithi110 ceph-mon[97633]: pgmap v17483: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:39 smithi104 ceph-mon[97044]: pgmap v17483: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:39.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:39.783808+0000) 2022-09-10T13:35:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:40.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:40.171469+0000) 2022-09-10T13:35:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:40.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:40.784070+0000) 2022-09-10T13:35:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:41.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:41.171729+0000) 2022-09-10T13:35:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:41 smithi110 ceph-mon[97633]: pgmap v17484: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:41 smithi104 ceph-mon[97044]: pgmap v17484: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:41.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:41.784233+0000) 2022-09-10T13:35:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:42.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:42.172012+0000) 2022-09-10T13:35:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:42.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:42.784459+0000) 2022-09-10T13:35:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:43.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:43.172194+0000) 2022-09-10T13:35:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:43 smithi110 ceph-mon[97633]: pgmap v17485: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:43 smithi104 ceph-mon[97044]: pgmap v17485: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:43.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:43.784628+0000) 2022-09-10T13:35:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:44.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:44.172386+0000) 2022-09-10T13:35:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:44.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:44.784855+0000) 2022-09-10T13:35:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:45.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:45.172582+0000) 2022-09-10T13:35:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:45 smithi110 ceph-mon[97633]: pgmap v17486: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:45 smithi104 ceph-mon[97044]: pgmap v17486: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:46.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:45.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:45.785216+0000) 2022-09-10T13:35:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:46.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:46.172809+0000) 2022-09-10T13:35:47.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:46 smithi104 ceph-mon[97044]: pgmap v17487: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:35:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:35:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:35:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:35:47.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:46.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:46.785430+0000) 2022-09-10T13:35:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:46 smithi110 ceph-mon[97633]: pgmap v17487: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:35:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:35:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:35:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:35:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:47.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:47.173060+0000) 2022-09-10T13:35:48.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:47.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:47.785571+0000) 2022-09-10T13:35:48.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:35:47.758+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:35:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:48.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:48.173325+0000) 2022-09-10T13:35:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:48.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:48.785838+0000) 2022-09-10T13:35:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:49.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:49.173564+0000) 2022-09-10T13:35:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:49 smithi110 ceph-mon[97633]: pgmap v17488: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:49 smithi104 ceph-mon[97044]: pgmap v17488: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:49.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:49.786073+0000) 2022-09-10T13:35:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:50.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:50.173756+0000) 2022-09-10T13:35:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:50.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:50.786271+0000) 2022-09-10T13:35:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:51.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:51.173959+0000) 2022-09-10T13:35:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:51 smithi110 ceph-mon[97633]: pgmap v17489: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:51 smithi104 ceph-mon[97044]: pgmap v17489: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:51.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:51.786410+0000) 2022-09-10T13:35:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:52.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:52.174217+0000) 2022-09-10T13:35:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:52.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:52.786593+0000) 2022-09-10T13:35:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:53.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:53.174359+0000) 2022-09-10T13:35:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:53 smithi110 ceph-mon[97633]: pgmap v17490: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:53 smithi104 ceph-mon[97044]: pgmap v17490: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:53.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:53.786807+0000) 2022-09-10T13:35:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:54.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:54.174607+0000) 2022-09-10T13:35:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:54.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:54.786986+0000) 2022-09-10T13:35:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:55.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:55.174841+0000) 2022-09-10T13:35:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:55 smithi110 ceph-mon[97633]: pgmap v17491: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:55 smithi104 ceph-mon[97044]: pgmap v17491: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:56.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:55.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:55.787222+0000) 2022-09-10T13:35:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:56.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:56.175006+0000) 2022-09-10T13:35:57.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:56 smithi104 ceph-mon[97044]: pgmap v17492: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:56.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:56.787401+0000) 2022-09-10T13:35:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:56 smithi110 ceph-mon[97633]: pgmap v17492: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:35:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:57.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:57.175244+0000) 2022-09-10T13:35:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:57.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:57.787534+0000) 2022-09-10T13:35:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:58.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:58.175427+0000) 2022-09-10T13:35:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:58.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:58.787672+0000) 2022-09-10T13:35:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:35:59.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:59.175590+0000) 2022-09-10T13:35:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:35:59 smithi110 ceph-mon[97633]: pgmap v17493: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:35:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:35:59 smithi104 ceph-mon[97044]: pgmap v17493: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:35:59.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:35:59.787901+0000) 2022-09-10T13:36:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:00.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:00.175811+0000) 2022-09-10T13:36:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:00.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:00.788038+0000) 2022-09-10T13:36:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:01.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:01.176005+0000) 2022-09-10T13:36:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:01 smithi110 ceph-mon[97633]: pgmap v17494: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:01 smithi104 ceph-mon[97044]: pgmap v17494: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:01.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:01.788256+0000) 2022-09-10T13:36:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:02.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:02.176213+0000) 2022-09-10T13:36:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:02.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:02.788462+0000) 2022-09-10T13:36:03.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:36:02.774+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:36:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:03.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:03.176422+0000) 2022-09-10T13:36:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:03 smithi110 ceph-mon[97633]: pgmap v17495: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:03 smithi104 ceph-mon[97044]: pgmap v17495: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:04.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:03.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:03.788696+0000) 2022-09-10T13:36:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:04.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:04.176633+0000) 2022-09-10T13:36:05.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:04.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:04.788934+0000) 2022-09-10T13:36:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:05.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:05.176833+0000) 2022-09-10T13:36:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:05 smithi110 ceph-mon[97633]: pgmap v17496: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:05 smithi104 ceph-mon[97044]: pgmap v17496: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:05.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:05.789206+0000) 2022-09-10T13:36:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:06.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:06.177063+0000) 2022-09-10T13:36:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:06.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:06.789447+0000) 2022-09-10T13:36:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:06 smithi110 ceph-mon[97633]: pgmap v17497: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:36:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:06 smithi104 ceph-mon[97044]: pgmap v17497: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:36:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:07.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:07.177249+0000) 2022-09-10T13:36:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:07.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:07.789655+0000) 2022-09-10T13:36:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:08.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:08.177476+0000) 2022-09-10T13:36:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:08.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:08.789897+0000) 2022-09-10T13:36:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:08 smithi110 ceph-mon[97633]: osdmap e1174: 12 total, 12 up, 12 in 2022-09-10T13:36:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:08 smithi110 ceph-mon[97633]: pgmap v17499: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:36:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:08 smithi104 ceph-mon[97044]: osdmap e1174: 12 total, 12 up, 12 in 2022-09-10T13:36:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:08 smithi104 ceph-mon[97044]: pgmap v17499: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:36:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:09.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:09.177740+0000) 2022-09-10T13:36:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:09.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:09.790187+0000) 2022-09-10T13:36:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:10.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:10.177977+0000) 2022-09-10T13:36:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:10.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:10.790347+0000) 2022-09-10T13:36:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:11.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:11.178162+0000) 2022-09-10T13:36:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:11 smithi110 ceph-mon[97633]: pgmap v17500: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:36:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:11 smithi110 ceph-mon[97633]: osdmap e1175: 12 total, 12 up, 12 in 2022-09-10T13:36:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:11 smithi104 ceph-mon[97044]: pgmap v17500: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:36:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:11 smithi104 ceph-mon[97044]: osdmap e1175: 12 total, 12 up, 12 in 2022-09-10T13:36:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:11.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:11.790579+0000) 2022-09-10T13:36:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:12.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:12.178361+0000) 2022-09-10T13:36:13.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:12.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:12.790759+0000) 2022-09-10T13:36:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:13.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:13.178600+0000) 2022-09-10T13:36:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:13 smithi110 ceph-mon[97633]: pgmap v17502: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:36:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:13 smithi104 ceph-mon[97044]: pgmap v17502: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:36:14.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:13.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:13.790923+0000) 2022-09-10T13:36:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:14.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:14.178827+0000) 2022-09-10T13:36:15.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:14.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:14.791185+0000) 2022-09-10T13:36:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:15.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:15.178986+0000) 2022-09-10T13:36:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:15 smithi110 ceph-mon[97633]: pgmap v17503: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:36:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:15 smithi104 ceph-mon[97044]: pgmap v17503: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:36:16.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:15.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:15.791393+0000) 2022-09-10T13:36:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:16.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:16.179223+0000) 2022-09-10T13:36:17.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:16.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:16.791600+0000) 2022-09-10T13:36:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:16 smithi110 ceph-mon[97633]: pgmap v17504: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:36:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:16 smithi104 ceph-mon[97044]: pgmap v17504: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:36:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:17.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:17.179446+0000) 2022-09-10T13:36:18.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:17.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:17.791757+0000) 2022-09-10T13:36:18.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:36:17.779+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:36:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:18.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:18.179646+0000) 2022-09-10T13:36:19.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:18.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:18.791989+0000) 2022-09-10T13:36:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:19.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:19.179871+0000) 2022-09-10T13:36:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:19 smithi110 ceph-mon[97633]: pgmap v17505: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:36:19.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:19 smithi104 ceph-mon[97044]: pgmap v17505: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:36:20.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:19.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:19.792297+0000) 2022-09-10T13:36:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:20.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:20.180191+0000) 2022-09-10T13:36:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:20.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:20.792496+0000) 2022-09-10T13:36:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:21.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:21.180326+0000) 2022-09-10T13:36:21.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:21 smithi110 ceph-mon[97633]: pgmap v17506: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:36:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:21 smithi104 ceph-mon[97044]: pgmap v17506: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:36:22.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:21.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:21.792746+0000) 2022-09-10T13:36:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:22.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:22.180541+0000) 2022-09-10T13:36:23.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:22.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:22.793013+0000) 2022-09-10T13:36:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:23.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:23.180667+0000) 2022-09-10T13:36:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:23 smithi110 ceph-mon[97633]: pgmap v17507: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:36:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:23 smithi104 ceph-mon[97044]: pgmap v17507: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:36:24.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:23.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:23.793239+0000) 2022-09-10T13:36:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:24.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:24.180834+0000) 2022-09-10T13:36:25.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:24.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:24.793417+0000) 2022-09-10T13:36:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:25.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:25.181130+0000) 2022-09-10T13:36:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:25 smithi110 ceph-mon[97633]: pgmap v17508: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:36:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:25 smithi104 ceph-mon[97044]: pgmap v17508: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:36:26.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:25.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:25.793596+0000) 2022-09-10T13:36:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:26.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:26.181307+0000) 2022-09-10T13:36:27.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:26.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:26.793825+0000) 2022-09-10T13:36:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:26 smithi110 ceph-mon[97633]: pgmap v17509: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:36:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:26 smithi104 ceph-mon[97044]: pgmap v17509: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:36:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:27.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:27.181487+0000) 2022-09-10T13:36:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:27.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:27.794004+0000) 2022-09-10T13:36:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:28.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:28.181685+0000) 2022-09-10T13:36:29.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:28.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:28.794226+0000) 2022-09-10T13:36:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:29.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:29.181961+0000) 2022-09-10T13:36:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:29 smithi110 ceph-mon[97633]: pgmap v17510: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:29 smithi104 ceph-mon[97044]: pgmap v17510: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:30.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:29.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:29.794402+0000) 2022-09-10T13:36:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:30.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:30.182211+0000) 2022-09-10T13:36:31.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:30.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:30.794600+0000) 2022-09-10T13:36:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:31.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:31.182474+0000) 2022-09-10T13:36:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:31 smithi110 ceph-mon[97633]: pgmap v17511: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:31 smithi104 ceph-mon[97044]: pgmap v17511: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:32.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:31.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:31.794776+0000) 2022-09-10T13:36:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:32.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:32.182690+0000) 2022-09-10T13:36:33.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:32.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:32.794942+0000) 2022-09-10T13:36:33.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:36:32.793+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:36:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:33.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:33.182953+0000) 2022-09-10T13:36:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:33 smithi110 ceph-mon[97633]: pgmap v17512: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:33 smithi104 ceph-mon[97044]: pgmap v17512: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:34.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:33.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:33.795141+0000) 2022-09-10T13:36:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:34.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:34.183162+0000) 2022-09-10T13:36:35.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:34.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:34.795328+0000) 2022-09-10T13:36:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:35.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:35.183388+0000) 2022-09-10T13:36:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:35 smithi110 ceph-mon[97633]: pgmap v17513: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:35 smithi104 ceph-mon[97044]: pgmap v17513: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:36.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:35.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:35.795538+0000) 2022-09-10T13:36:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:36.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:36.183631+0000) 2022-09-10T13:36:37.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:36.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:36.795674+0000) 2022-09-10T13:36:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:36 smithi110 ceph-mon[97633]: pgmap v17514: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:36 smithi104 ceph-mon[97044]: pgmap v17514: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:37.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:37.183866+0000) 2022-09-10T13:36:38.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:37.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:37.795901+0000) 2022-09-10T13:36:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:38.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:38.184140+0000) 2022-09-10T13:36:39.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:38.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:38.796125+0000) 2022-09-10T13:36:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:39.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:39.184358+0000) 2022-09-10T13:36:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:39 smithi110 ceph-mon[97633]: pgmap v17515: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:39 smithi104 ceph-mon[97044]: pgmap v17515: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:40.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:39.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:39.796373+0000) 2022-09-10T13:36:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:40.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:40.184578+0000) 2022-09-10T13:36:41.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:40.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:40.796596+0000) 2022-09-10T13:36:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:41.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:41.184818+0000) 2022-09-10T13:36:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:41 smithi110 ceph-mon[97633]: pgmap v17516: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:41 smithi104 ceph-mon[97044]: pgmap v17516: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:42.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:41.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:41.796857+0000) 2022-09-10T13:36:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:42.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:42.185064+0000) 2022-09-10T13:36:43.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:42.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:42.797089+0000) 2022-09-10T13:36:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:43.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:43.185302+0000) 2022-09-10T13:36:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:43 smithi110 ceph-mon[97633]: pgmap v17517: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:43 smithi104 ceph-mon[97044]: pgmap v17517: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:44.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:43.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:43.797290+0000) 2022-09-10T13:36:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:44.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:44.185524+0000) 2022-09-10T13:36:45.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:44.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:44.797434+0000) 2022-09-10T13:36:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:45.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:45.185787+0000) 2022-09-10T13:36:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:45 smithi110 ceph-mon[97633]: pgmap v17518: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:45 smithi104 ceph-mon[97044]: pgmap v17518: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:46.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:45.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:45.797600+0000) 2022-09-10T13:36:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:46.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:46.186050+0000) 2022-09-10T13:36:47.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:46.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:46.797839+0000) 2022-09-10T13:36:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:46 smithi110 ceph-mon[97633]: pgmap v17519: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:36:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:36:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:36:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:36:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:46 smithi104 ceph-mon[97044]: pgmap v17519: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:36:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:36:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:36:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:47.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:47.186293+0000) 2022-09-10T13:36:48.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:47.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:47.797969+0000) 2022-09-10T13:36:48.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:36:47.797+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:36:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:48.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:48.186520+0000) 2022-09-10T13:36:49.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:48.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:48.798196+0000) 2022-09-10T13:36:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:49.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:49.186710+0000) 2022-09-10T13:36:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:49 smithi110 ceph-mon[97633]: pgmap v17520: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:49 smithi104 ceph-mon[97044]: pgmap v17520: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:50.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:49.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:49.798435+0000) 2022-09-10T13:36:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:50.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:50.186899+0000) 2022-09-10T13:36:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:50.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:50.798600+0000) 2022-09-10T13:36:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:51.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:51.187156+0000) 2022-09-10T13:36:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:51 smithi110 ceph-mon[97633]: pgmap v17521: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:51 smithi104 ceph-mon[97044]: pgmap v17521: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:52.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:51.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:51.798816+0000) 2022-09-10T13:36:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:52.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:52.187329+0000) 2022-09-10T13:36:53.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:52.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:52.799024+0000) 2022-09-10T13:36:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:53.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:53.187553+0000) 2022-09-10T13:36:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:53 smithi110 ceph-mon[97633]: pgmap v17522: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:53 smithi104 ceph-mon[97044]: pgmap v17522: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:54.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:53.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:53.799147+0000) 2022-09-10T13:36:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:54.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:54.187798+0000) 2022-09-10T13:36:55.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:54.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:54.799374+0000) 2022-09-10T13:36:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:55.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:55.188050+0000) 2022-09-10T13:36:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:55 smithi110 ceph-mon[97633]: pgmap v17523: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:55 smithi104 ceph-mon[97044]: pgmap v17523: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:36:56.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:55.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:55.799624+0000) 2022-09-10T13:36:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:56.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:56.188223+0000) 2022-09-10T13:36:57.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:56.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:56.799847+0000) 2022-09-10T13:36:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:56 smithi110 ceph-mon[97633]: pgmap v17524: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:56 smithi104 ceph-mon[97044]: pgmap v17524: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:36:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:57.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:57.188387+0000) 2022-09-10T13:36:58.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:57.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:57.800133+0000) 2022-09-10T13:36:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:58.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:58.188575+0000) 2022-09-10T13:36:59.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:58.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:58.800258+0000) 2022-09-10T13:36:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:36:59.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:59.188792+0000) 2022-09-10T13:36:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:36:59 smithi110 ceph-mon[97633]: pgmap v17525: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:36:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:36:59 smithi104 ceph-mon[97044]: pgmap v17525: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:00.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:36:59.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:36:59.800444+0000) 2022-09-10T13:37:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:00.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:00.189062+0000) 2022-09-10T13:37:01.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:00.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:00.800676+0000) 2022-09-10T13:37:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:01.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:01.189313+0000) 2022-09-10T13:37:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:01 smithi110 ceph-mon[97633]: pgmap v17526: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:01 smithi104 ceph-mon[97044]: pgmap v17526: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:02.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:01.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:01.800931+0000) 2022-09-10T13:37:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:02.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:02.189542+0000) 2022-09-10T13:37:03.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:02.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:02.801194+0000) 2022-09-10T13:37:03.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:37:02.808+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:37:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:03.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:03.189723+0000) 2022-09-10T13:37:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:03 smithi110 ceph-mon[97633]: pgmap v17527: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:03 smithi104 ceph-mon[97044]: pgmap v17527: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:03.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:03.801354+0000) 2022-09-10T13:37:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:04.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:04.189888+0000) 2022-09-10T13:37:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:04.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:04.801608+0000) 2022-09-10T13:37:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:05.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:05.190148+0000) 2022-09-10T13:37:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:05 smithi110 ceph-mon[97633]: pgmap v17528: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:05 smithi104 ceph-mon[97044]: pgmap v17528: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:06.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:05.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:05.801804+0000) 2022-09-10T13:37:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:06.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:06.190318+0000) 2022-09-10T13:37:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:06.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:06.802016+0000) 2022-09-10T13:37:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:06 smithi110 ceph-mon[97633]: pgmap v17529: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:37:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:06 smithi104 ceph-mon[97044]: pgmap v17529: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:37:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:07.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:07.190585+0000) 2022-09-10T13:37:08.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:07.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:07.802241+0000) 2022-09-10T13:37:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:08.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:08.190721+0000) 2022-09-10T13:37:09.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:08.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:08.802481+0000) 2022-09-10T13:37:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:08 smithi110 ceph-mon[97633]: osdmap e1176: 12 total, 12 up, 12 in 2022-09-10T13:37:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:08 smithi110 ceph-mon[97633]: pgmap v17531: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:37:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:08 smithi104 ceph-mon[97044]: osdmap e1176: 12 total, 12 up, 12 in 2022-09-10T13:37:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:08 smithi104 ceph-mon[97044]: pgmap v17531: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:37:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:09.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:09.190880+0000) 2022-09-10T13:37:10.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:09.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:09.802734+0000) 2022-09-10T13:37:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:10.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:10.191065+0000) 2022-09-10T13:37:11.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:10.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:10.802956+0000) 2022-09-10T13:37:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:11.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:11.191267+0000) 2022-09-10T13:37:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:11 smithi110 ceph-mon[97633]: pgmap v17532: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:37:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:11 smithi110 ceph-mon[97633]: osdmap e1177: 12 total, 12 up, 12 in 2022-09-10T13:37:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:11 smithi104 ceph-mon[97044]: pgmap v17532: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:37:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:11 smithi104 ceph-mon[97044]: osdmap e1177: 12 total, 12 up, 12 in 2022-09-10T13:37:12.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:11.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:11.803225+0000) 2022-09-10T13:37:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:12.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:12.191454+0000) 2022-09-10T13:37:13.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:12.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:12.803507+0000) 2022-09-10T13:37:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:13.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:13.191684+0000) 2022-09-10T13:37:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:13 smithi110 ceph-mon[97633]: pgmap v17534: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:37:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:13 smithi104 ceph-mon[97044]: pgmap v17534: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:37:14.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:13.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:13.803687+0000) 2022-09-10T13:37:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:14.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:14.191880+0000) 2022-09-10T13:37:15.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:14.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:14.803948+0000) 2022-09-10T13:37:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:15.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:15.192123+0000) 2022-09-10T13:37:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:15 smithi110 ceph-mon[97633]: pgmap v17535: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:37:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:15 smithi104 ceph-mon[97044]: pgmap v17535: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:37:16.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:15.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:15.804177+0000) 2022-09-10T13:37:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:16.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:16.192309+0000) 2022-09-10T13:37:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:16 smithi110 ceph-mon[97633]: pgmap v17536: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:37:17.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:16.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:16.804413+0000) 2022-09-10T13:37:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:16 smithi104 ceph-mon[97044]: pgmap v17536: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:37:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:17.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:17.192466+0000) 2022-09-10T13:37:18.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:17.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:17.804556+0000) 2022-09-10T13:37:18.193 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:37:17.812+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:37:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:18.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:18.192702+0000) 2022-09-10T13:37:19.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:18.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:18.804808+0000) 2022-09-10T13:37:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:19.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:19.192891+0000) 2022-09-10T13:37:19.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:19 smithi110 ceph-mon[97633]: pgmap v17537: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:37:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:19 smithi104 ceph-mon[97044]: pgmap v17537: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:37:20.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:19.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:19.805045+0000) 2022-09-10T13:37:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:20.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:20.193060+0000) 2022-09-10T13:37:21.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:20.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:20.805214+0000) 2022-09-10T13:37:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:21.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:21.193274+0000) 2022-09-10T13:37:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:21 smithi110 ceph-mon[97633]: pgmap v17538: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:37:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:21 smithi104 ceph-mon[97044]: pgmap v17538: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:37:22.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:21.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:21.805352+0000) 2022-09-10T13:37:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:22.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:22.193531+0000) 2022-09-10T13:37:23.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:22.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:22.805565+0000) 2022-09-10T13:37:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:23.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:23.193748+0000) 2022-09-10T13:37:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:23 smithi110 ceph-mon[97633]: pgmap v17539: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:37:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:23 smithi104 ceph-mon[97044]: pgmap v17539: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2022-09-10T13:37:24.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:23.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:23.805837+0000) 2022-09-10T13:37:24.559 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:24.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:24.193893+0000) 2022-09-10T13:37:25.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:24.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:24.806080+0000) 2022-09-10T13:37:25.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:25.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:25.194084+0000) 2022-09-10T13:37:25.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:25 smithi110 ceph-mon[97633]: pgmap v17540: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:37:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:25 smithi104 ceph-mon[97044]: pgmap v17540: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2022-09-10T13:37:26.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:25.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:25.806284+0000) 2022-09-10T13:37:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:26.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:26.194358+0000) 2022-09-10T13:37:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:26 smithi110 ceph-mon[97633]: pgmap v17541: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:37:27.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:26.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:26.806410+0000) 2022-09-10T13:37:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:26 smithi104 ceph-mon[97044]: pgmap v17541: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:37:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:27.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:27.194576+0000) 2022-09-10T13:37:28.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:27.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:27.806595+0000) 2022-09-10T13:37:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:28.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:28.194789+0000) 2022-09-10T13:37:29.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:28.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:28.806771+0000) 2022-09-10T13:37:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:29.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:29.194967+0000) 2022-09-10T13:37:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:29 smithi110 ceph-mon[97633]: pgmap v17542: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:29 smithi104 ceph-mon[97044]: pgmap v17542: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:30.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:29.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:29.806936+0000) 2022-09-10T13:37:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:30.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:30.195165+0000) 2022-09-10T13:37:31.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:30.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:30.807203+0000) 2022-09-10T13:37:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:31.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:31.195348+0000) 2022-09-10T13:37:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:31 smithi110 ceph-mon[97633]: pgmap v17543: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:31 smithi104 ceph-mon[97044]: pgmap v17543: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:32.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:31.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:31.807357+0000) 2022-09-10T13:37:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:32.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:32.195552+0000) 2022-09-10T13:37:33.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:32.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:32.807483+0000) 2022-09-10T13:37:33.196 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:37:32.828+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:37:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:33.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:33.195711+0000) 2022-09-10T13:37:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:33 smithi110 ceph-mon[97633]: pgmap v17544: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:33 smithi104 ceph-mon[97044]: pgmap v17544: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:34.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:33.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:33.807641+0000) 2022-09-10T13:37:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:34.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:34.195939+0000) 2022-09-10T13:37:35.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:34.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:34.807837+0000) 2022-09-10T13:37:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:35.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:35.196224+0000) 2022-09-10T13:37:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:35 smithi110 ceph-mon[97633]: pgmap v17545: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:35 smithi104 ceph-mon[97044]: pgmap v17545: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:36.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:35.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:35.808072+0000) 2022-09-10T13:37:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:36.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:36.196396+0000) 2022-09-10T13:37:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:36 smithi110 ceph-mon[97633]: pgmap v17546: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:37.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:36.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:36.808242+0000) 2022-09-10T13:37:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:36 smithi104 ceph-mon[97044]: pgmap v17546: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:37.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:37.196649+0000) 2022-09-10T13:37:38.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:37.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:37.808500+0000) 2022-09-10T13:37:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:38.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:38.196886+0000) 2022-09-10T13:37:39.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:38.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:38.808692+0000) 2022-09-10T13:37:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:39.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:39.197074+0000) 2022-09-10T13:37:39.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:39 smithi110 ceph-mon[97633]: pgmap v17547: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:39.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:39 smithi104 ceph-mon[97044]: pgmap v17547: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:40.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:39.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:39.808896+0000) 2022-09-10T13:37:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:40.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:40.197293+0000) 2022-09-10T13:37:41.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:40.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:40.809169+0000) 2022-09-10T13:37:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:41.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:41.197514+0000) 2022-09-10T13:37:41.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:41 smithi110 ceph-mon[97633]: pgmap v17548: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:41 smithi104 ceph-mon[97044]: pgmap v17548: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:42.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:41.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:41.809383+0000) 2022-09-10T13:37:42.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:42.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:42.197745+0000) 2022-09-10T13:37:43.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:42.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:42.809570+0000) 2022-09-10T13:37:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:43.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:43.197916+0000) 2022-09-10T13:37:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:43 smithi110 ceph-mon[97633]: pgmap v17549: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:43 smithi104 ceph-mon[97044]: pgmap v17549: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:44.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:43.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:43.809820+0000) 2022-09-10T13:37:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:44.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:44.198168+0000) 2022-09-10T13:37:45.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:44.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:44.810068+0000) 2022-09-10T13:37:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:45.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:45.198332+0000) 2022-09-10T13:37:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:45 smithi110 ceph-mon[97633]: pgmap v17550: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:45 smithi104 ceph-mon[97044]: pgmap v17550: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:37:46.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:45.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:45.810329+0000) 2022-09-10T13:37:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:46.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:46.198496+0000) 2022-09-10T13:37:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:46 smithi110 ceph-mon[97633]: pgmap v17551: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:37:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:37:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:37:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:37:47.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:46.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:46.810488+0000) 2022-09-10T13:37:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:46 smithi104 ceph-mon[97044]: pgmap v17551: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:37:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:37:47.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:37:47.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:37:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:47.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:47.198655+0000) 2022-09-10T13:37:48.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:47.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:47.810631+0000) 2022-09-10T13:37:48.199 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:37:47.833+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:37:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:48.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:48.198931+0000) 2022-09-10T13:37:49.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:48.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:48.810806+0000) 2022-09-10T13:37:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:49.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:49.199185+0000) 2022-09-10T13:37:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:49 smithi110 ceph-mon[97633]: pgmap v17552: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:49 smithi104 ceph-mon[97044]: pgmap v17552: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:50.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:49.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:49.811025+0000) 2022-09-10T13:37:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:50.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:50.199397+0000) 2022-09-10T13:37:51.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:50.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:50.811263+0000) 2022-09-10T13:37:51.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:51.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:51.199712+0000) 2022-09-10T13:37:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:51 smithi110 ceph-mon[97633]: pgmap v17553: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:51 smithi104 ceph-mon[97044]: pgmap v17553: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:52.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:51.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:51.811439+0000) 2022-09-10T13:37:52.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:52.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:52.199953+0000) 2022-09-10T13:37:53.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:52.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:52.811657+0000) 2022-09-10T13:37:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:53.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:53.200173+0000) 2022-09-10T13:37:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:53 smithi110 ceph-mon[97633]: pgmap v17554: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:53 smithi104 ceph-mon[97044]: pgmap v17554: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:54.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:53.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:53.811900+0000) 2022-09-10T13:37:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:54.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:54.200400+0000) 2022-09-10T13:37:55.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:54.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:54.812172+0000) 2022-09-10T13:37:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:55.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:55.200653+0000) 2022-09-10T13:37:55.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:55 smithi110 ceph-mon[97633]: pgmap v17555: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:55 smithi104 ceph-mon[97044]: pgmap v17555: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:56.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:55.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:55.812413+0000) 2022-09-10T13:37:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:56.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:56.200922+0000) 2022-09-10T13:37:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:56 smithi110 ceph-mon[97633]: pgmap v17556: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:57.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:56.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:56.812723+0000) 2022-09-10T13:37:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:56 smithi104 ceph-mon[97044]: pgmap v17556: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:37:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:57.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:57.201173+0000) 2022-09-10T13:37:58.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:57.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:57.812959+0000) 2022-09-10T13:37:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:58.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:58.201414+0000) 2022-09-10T13:37:59.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:58.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:58.813176+0000) 2022-09-10T13:37:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:37:59.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:59.201601+0000) 2022-09-10T13:37:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:37:59 smithi110 ceph-mon[97633]: pgmap v17557: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:37:59.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:37:59 smithi104 ceph-mon[97044]: pgmap v17557: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:00.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:37:59.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:37:59.813468+0000) 2022-09-10T13:38:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:00.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:00.201730+0000) 2022-09-10T13:38:01.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:00.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:00.813675+0000) 2022-09-10T13:38:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:01.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:01.201945+0000) 2022-09-10T13:38:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:01 smithi110 ceph-mon[97633]: pgmap v17558: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:01 smithi104 ceph-mon[97044]: pgmap v17558: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:02.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:01.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:01.813889+0000) 2022-09-10T13:38:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:02.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:02.202131+0000) 2022-09-10T13:38:03.203 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:38:02.848+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:38:03.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:02.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:02.814144+0000) 2022-09-10T13:38:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:03.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:03.202375+0000) 2022-09-10T13:38:03.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:03 smithi110 ceph-mon[97633]: pgmap v17559: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:03 smithi104 ceph-mon[97044]: pgmap v17559: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:04.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:03.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:03.814325+0000) 2022-09-10T13:38:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:04.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:04.202581+0000) 2022-09-10T13:38:05.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:04.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:04.814566+0000) 2022-09-10T13:38:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:05.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:05.202823+0000) 2022-09-10T13:38:05.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:05 smithi110 ceph-mon[97633]: pgmap v17560: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:05 smithi104 ceph-mon[97044]: pgmap v17560: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:06.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:05.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:05.814786+0000) 2022-09-10T13:38:06.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:06.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:06.203106+0000) 2022-09-10T13:38:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:06 smithi110 ceph-mon[97633]: pgmap v17561: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:38:07.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:06.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:06.815005+0000) 2022-09-10T13:38:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:06 smithi104 ceph-mon[97044]: pgmap v17561: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:38:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:07.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:07.203344+0000) 2022-09-10T13:38:08.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:07.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:07.815211+0000) 2022-09-10T13:38:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:08.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:08.203508+0000) 2022-09-10T13:38:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:08 smithi110 ceph-mon[97633]: osdmap e1178: 12 total, 12 up, 12 in 2022-09-10T13:38:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:08 smithi110 ceph-mon[97633]: pgmap v17563: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:38:09.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:08.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:08.815442+0000) 2022-09-10T13:38:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:08 smithi104 ceph-mon[97044]: osdmap e1178: 12 total, 12 up, 12 in 2022-09-10T13:38:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:08 smithi104 ceph-mon[97044]: pgmap v17563: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:38:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:09.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:09.203693+0000) 2022-09-10T13:38:10.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:09.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:09.815665+0000) 2022-09-10T13:38:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:10.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:10.203991+0000) 2022-09-10T13:38:11.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:10.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:10.815900+0000) 2022-09-10T13:38:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:11.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:11.204249+0000) 2022-09-10T13:38:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:11 smithi110 ceph-mon[97633]: pgmap v17564: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:38:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:11 smithi110 ceph-mon[97633]: osdmap e1179: 12 total, 12 up, 12 in 2022-09-10T13:38:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:11 smithi104 ceph-mon[97044]: pgmap v17564: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:38:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:11 smithi104 ceph-mon[97044]: osdmap e1179: 12 total, 12 up, 12 in 2022-09-10T13:38:12.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:11.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:11.816170+0000) 2022-09-10T13:38:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:12.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:12.204450+0000) 2022-09-10T13:38:13.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:12.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:12.816382+0000) 2022-09-10T13:38:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:13.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:13.204679+0000) 2022-09-10T13:38:13.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:13 smithi110 ceph-mon[97633]: pgmap v17566: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:38:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:13 smithi104 ceph-mon[97044]: pgmap v17566: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 67 KiB/s wr, 34 op/s 2022-09-10T13:38:14.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:13.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:13.816601+0000) 2022-09-10T13:38:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:14.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:14.204850+0000) 2022-09-10T13:38:15.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:14.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:14.816863+0000) 2022-09-10T13:38:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:15.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:15.205013+0000) 2022-09-10T13:38:15.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:15 smithi110 ceph-mon[97633]: pgmap v17567: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:38:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:15 smithi104 ceph-mon[97044]: pgmap v17567: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:38:16.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:15.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:15.817090+0000) 2022-09-10T13:38:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:16.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:16.205224+0000) 2022-09-10T13:38:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:16 smithi110 ceph-mon[97633]: pgmap v17568: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:38:17.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:16.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:16.817341+0000) 2022-09-10T13:38:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:16 smithi104 ceph-mon[97044]: pgmap v17568: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:38:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:17.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:17.205417+0000) 2022-09-10T13:38:18.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:17.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:17.817613+0000) 2022-09-10T13:38:18.207 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:38:17.853+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:38:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:18.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:18.205582+0000) 2022-09-10T13:38:19.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:18.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:18.817775+0000) 2022-09-10T13:38:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:19.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:19.205755+0000) 2022-09-10T13:38:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:19 smithi110 ceph-mon[97633]: pgmap v17569: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:38:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:19 smithi104 ceph-mon[97044]: pgmap v17569: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:38:20.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:19.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:19.817959+0000) 2022-09-10T13:38:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:20.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:20.205967+0000) 2022-09-10T13:38:21.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:20.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:20.818216+0000) 2022-09-10T13:38:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:21.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:21.206163+0000) 2022-09-10T13:38:21.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:21 smithi110 ceph-mon[97633]: pgmap v17570: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:38:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:21 smithi104 ceph-mon[97044]: pgmap v17570: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:38:22.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:21.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:21.818455+0000) 2022-09-10T13:38:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:22.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:22.206289+0000) 2022-09-10T13:38:23.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:22.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:22.818679+0000) 2022-09-10T13:38:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:23.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:23.206513+0000) 2022-09-10T13:38:23.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:23 smithi110 ceph-mon[97633]: pgmap v17571: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:38:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:23 smithi104 ceph-mon[97044]: pgmap v17571: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:38:24.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:23.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:23.818914+0000) 2022-09-10T13:38:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:24.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:24.206750+0000) 2022-09-10T13:38:25.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:24.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:24.819156+0000) 2022-09-10T13:38:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:25.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:25.206921+0000) 2022-09-10T13:38:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:25 smithi110 ceph-mon[97633]: pgmap v17572: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2022-09-10T13:38:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:25 smithi104 ceph-mon[97044]: pgmap v17572: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 85 B/s wr, 9 op/s 2022-09-10T13:38:26.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:25.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:25.819348+0000) 2022-09-10T13:38:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:26.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:26.207190+0000) 2022-09-10T13:38:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:26 smithi110 ceph-mon[97633]: pgmap v17573: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:38:27.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:26.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:26.819539+0000) 2022-09-10T13:38:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:26 smithi104 ceph-mon[97044]: pgmap v17573: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:38:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:27.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:27.207378+0000) 2022-09-10T13:38:28.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:27.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:27.819784+0000) 2022-09-10T13:38:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:28.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:28.207576+0000) 2022-09-10T13:38:29.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:28.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:28.820052+0000) 2022-09-10T13:38:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:29.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:29.207835+0000) 2022-09-10T13:38:29.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:29 smithi110 ceph-mon[97633]: pgmap v17574: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:29 smithi104 ceph-mon[97044]: pgmap v17574: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:30.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:29.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:29.820316+0000) 2022-09-10T13:38:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:30.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:30.208011+0000) 2022-09-10T13:38:31.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:30.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:30.820505+0000) 2022-09-10T13:38:31.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:31.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:31.208168+0000) 2022-09-10T13:38:31.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:31 smithi110 ceph-mon[97633]: pgmap v17575: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:31 smithi104 ceph-mon[97044]: pgmap v17575: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:32.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:31.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:31.820783+0000) 2022-09-10T13:38:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:32.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:32.208327+0000) 2022-09-10T13:38:33.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:32.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:32.821025+0000) 2022-09-10T13:38:33.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:38:32.869+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:38:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:33.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:33.208527+0000) 2022-09-10T13:38:33.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:33 smithi110 ceph-mon[97633]: pgmap v17576: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:33 smithi104 ceph-mon[97044]: pgmap v17576: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:34.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:33.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:33.821204+0000) 2022-09-10T13:38:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:34.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:34.208755+0000) 2022-09-10T13:38:35.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:34.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:34.821388+0000) 2022-09-10T13:38:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:35.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:35.209006+0000) 2022-09-10T13:38:35.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:35 smithi110 ceph-mon[97633]: pgmap v17577: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:38:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:35 smithi104 ceph-mon[97044]: pgmap v17577: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:38:36.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:35.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:35.821650+0000) 2022-09-10T13:38:36.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:36.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:36.209201+0000) 2022-09-10T13:38:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:36 smithi110 ceph-mon[97633]: pgmap v17578: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-09-10T13:38:37.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:36.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:36.821860+0000) 2022-09-10T13:38:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:36 smithi104 ceph-mon[97044]: pgmap v17578: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-09-10T13:38:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:37.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:37.209355+0000) 2022-09-10T13:38:38.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:37.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:37.822127+0000) 2022-09-10T13:38:38.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:38.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:38.209654+0000) 2022-09-10T13:38:39.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:38.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:38.822354+0000) 2022-09-10T13:38:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:39.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:39.209839+0000) 2022-09-10T13:38:39.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:39 smithi110 ceph-mon[97633]: pgmap v17579: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:39.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:39 smithi104 ceph-mon[97044]: pgmap v17579: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:40.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:39.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:39.822583+0000) 2022-09-10T13:38:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:40.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:40.210078+0000) 2022-09-10T13:38:41.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:40.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:40.822772+0000) 2022-09-10T13:38:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:41.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:41.210359+0000) 2022-09-10T13:38:41.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:41 smithi110 ceph-mon[97633]: pgmap v17580: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:41 smithi104 ceph-mon[97044]: pgmap v17580: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:42.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:41.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:41.822934+0000) 2022-09-10T13:38:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:42.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:42.210541+0000) 2022-09-10T13:38:43.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:42.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:42.823187+0000) 2022-09-10T13:38:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:43.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:43.210718+0000) 2022-09-10T13:38:43.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:43 smithi110 ceph-mon[97633]: pgmap v17581: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:43 smithi104 ceph-mon[97044]: pgmap v17581: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:44.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:43.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:43.823418+0000) 2022-09-10T13:38:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:44.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:44.210920+0000) 2022-09-10T13:38:45.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:44.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:44.823671+0000) 2022-09-10T13:38:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:45.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:45.211132+0000) 2022-09-10T13:38:45.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:45 smithi110 ceph-mon[97633]: pgmap v17582: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:38:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:45 smithi104 ceph-mon[97044]: pgmap v17582: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:38:46.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:45.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:45.823850+0000) 2022-09-10T13:38:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:46.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:46.211308+0000) 2022-09-10T13:38:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:38:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:38:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:38:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:38:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:46 smithi110 ceph-mon[97633]: pgmap v17583: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-09-10T13:38:47.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:46.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:46.824120+0000) 2022-09-10T13:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:38:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:38:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:38:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:38:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:46 smithi104 ceph-mon[97044]: pgmap v17583: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-09-10T13:38:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:47.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:47.211541+0000) 2022-09-10T13:38:48.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:47.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:47.824404+0000) 2022-09-10T13:38:48.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:38:47.884+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:38:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:48.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:48.211693+0000) 2022-09-10T13:38:49.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:48.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:48.824645+0000) 2022-09-10T13:38:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:49.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:49.211868+0000) 2022-09-10T13:38:49.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:49 smithi110 ceph-mon[97633]: pgmap v17584: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:49 smithi104 ceph-mon[97044]: pgmap v17584: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:50.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:49.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:49.824876+0000) 2022-09-10T13:38:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:50.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:50.212149+0000) 2022-09-10T13:38:51.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:50.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:50.825142+0000) 2022-09-10T13:38:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:51.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:51.212305+0000) 2022-09-10T13:38:51.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:51 smithi110 ceph-mon[97633]: pgmap v17585: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:51 smithi104 ceph-mon[97044]: pgmap v17585: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:38:52.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:51.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:51.825356+0000) 2022-09-10T13:38:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:52.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:52.212453+0000) 2022-09-10T13:38:53.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:52.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:52.825608+0000) 2022-09-10T13:38:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:53.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:53.212642+0000) 2022-09-10T13:38:53.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:53 smithi110 ceph-mon[97633]: pgmap v17586: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:53.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:53 smithi104 ceph-mon[97044]: pgmap v17586: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:54.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:53.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:53.825865+0000) 2022-09-10T13:38:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:54.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:54.212796+0000) 2022-09-10T13:38:55.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:54.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:54.826129+0000) 2022-09-10T13:38:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:55.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:55.213000+0000) 2022-09-10T13:38:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:55 smithi104 ceph-mon[97044]: pgmap v17587: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:38:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:55 smithi110 ceph-mon[97633]: pgmap v17587: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:38:56.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:55.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:55.826376+0000) 2022-09-10T13:38:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:56.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:56.213180+0000) 2022-09-10T13:38:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:56 smithi110 ceph-mon[97633]: pgmap v17588: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-09-10T13:38:57.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:56.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:56.826636+0000) 2022-09-10T13:38:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:56 smithi104 ceph-mon[97044]: pgmap v17588: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2022-09-10T13:38:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:57.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:57.213410+0000) 2022-09-10T13:38:58.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:57.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:57.826884+0000) 2022-09-10T13:38:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:58.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:58.213626+0000) 2022-09-10T13:38:59.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:58.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:58.827180+0000) 2022-09-10T13:38:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:38:59.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:59.213831+0000) 2022-09-10T13:38:59.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:38:59 smithi110 ceph-mon[97633]: pgmap v17589: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:38:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:38:59 smithi104 ceph-mon[97044]: pgmap v17589: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:00.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:38:59.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:38:59.827380+0000) 2022-09-10T13:39:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:00.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:00.213981+0000) 2022-09-10T13:39:01.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:00.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:00.827609+0000) 2022-09-10T13:39:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:01.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:01.214205+0000) 2022-09-10T13:39:01.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:01 smithi110 ceph-mon[97633]: pgmap v17590: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:01 smithi104 ceph-mon[97044]: pgmap v17590: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:02.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:01.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:01.827804+0000) 2022-09-10T13:39:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:02.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:02.214470+0000) 2022-09-10T13:39:03.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:02.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:02.828038+0000) 2022-09-10T13:39:03.215 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:39:02.893+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:39:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:03.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:03.214607+0000) 2022-09-10T13:39:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:03 smithi104 ceph-mon[97044]: pgmap v17591: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:03 smithi110 ceph-mon[97633]: pgmap v17591: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:04.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:03.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:03.828255+0000) 2022-09-10T13:39:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:04.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:04.214737+0000) 2022-09-10T13:39:05.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:04.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:04.828487+0000) 2022-09-10T13:39:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:05.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:05.214939+0000) 2022-09-10T13:39:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:05 smithi104 ceph-mon[97044]: pgmap v17592: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 KiB/s wr, 21 op/s 2022-09-10T13:39:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:05 smithi110 ceph-mon[97633]: pgmap v17592: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 KiB/s wr, 21 op/s 2022-09-10T13:39:06.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:05.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:05.828706+0000) 2022-09-10T13:39:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:06.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:06.215195+0000) 2022-09-10T13:39:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:06 smithi110 ceph-mon[97633]: pgmap v17593: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T13:39:07.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:06.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:06.828951+0000) 2022-09-10T13:39:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:06 smithi104 ceph-mon[97044]: pgmap v17593: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T13:39:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:07.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:07.215351+0000) 2022-09-10T13:39:08.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:07.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:07.829184+0000) 2022-09-10T13:39:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:08.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:08.215517+0000) 2022-09-10T13:39:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:08 smithi110 ceph-mon[97633]: osdmap e1180: 12 total, 12 up, 12 in 2022-09-10T13:39:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:08 smithi110 ceph-mon[97633]: pgmap v17595: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:39:09.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:08.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:08.829429+0000) 2022-09-10T13:39:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:08 smithi104 ceph-mon[97044]: osdmap e1180: 12 total, 12 up, 12 in 2022-09-10T13:39:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:08 smithi104 ceph-mon[97044]: pgmap v17595: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:39:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:09.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:09.215724+0000) 2022-09-10T13:39:10.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:09.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:09.829654+0000) 2022-09-10T13:39:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:10.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:10.216007+0000) 2022-09-10T13:39:11.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:10.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:10.829854+0000) 2022-09-10T13:39:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:11.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:11.216232+0000) 2022-09-10T13:39:11.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:11 smithi110 ceph-mon[97633]: pgmap v17596: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:39:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:11 smithi110 ceph-mon[97633]: osdmap e1181: 12 total, 12 up, 12 in 2022-09-10T13:39:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:11 smithi104 ceph-mon[97044]: pgmap v17596: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:39:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:11 smithi104 ceph-mon[97044]: osdmap e1181: 12 total, 12 up, 12 in 2022-09-10T13:39:12.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:11.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:11.830120+0000) 2022-09-10T13:39:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:12.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:12.216464+0000) 2022-09-10T13:39:13.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:12.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:12.830301+0000) 2022-09-10T13:39:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:13.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:13.216673+0000) 2022-09-10T13:39:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:13 smithi104 ceph-mon[97044]: pgmap v17598: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 35 KiB/s wr, 15 op/s 2022-09-10T13:39:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:13 smithi110 ceph-mon[97633]: pgmap v17598: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 35 KiB/s wr, 15 op/s 2022-09-10T13:39:14.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:13.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:13.830524+0000) 2022-09-10T13:39:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:14.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:14.216900+0000) 2022-09-10T13:39:15.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:14.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:14.830753+0000) 2022-09-10T13:39:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:15.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:15.217126+0000) 2022-09-10T13:39:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:15 smithi104 ceph-mon[97044]: pgmap v17599: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-10T13:39:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:15 smithi110 ceph-mon[97633]: pgmap v17599: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 127 B/s wr, 7 op/s 2022-09-10T13:39:16.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:15.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:15.830989+0000) 2022-09-10T13:39:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:16.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:16.217273+0000) 2022-09-10T13:39:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:16 smithi110 ceph-mon[97633]: pgmap v17600: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:39:17.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:16.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:16.831156+0000) 2022-09-10T13:39:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:16 smithi104 ceph-mon[97044]: pgmap v17600: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:39:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:17.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:17.217406+0000) 2022-09-10T13:39:18.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:39:17.900+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:39:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:17.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:17.831282+0000) 2022-09-10T13:39:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:18.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:18.217537+0000) 2022-09-10T13:39:19.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:18.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:18.831389+0000) 2022-09-10T13:39:19.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:19.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:19.217658+0000) 2022-09-10T13:39:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:19 smithi104 ceph-mon[97044]: pgmap v17601: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:39:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:19 smithi110 ceph-mon[97633]: pgmap v17601: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:39:20.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:19.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:19.831519+0000) 2022-09-10T13:39:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:20.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:20.217810+0000) 2022-09-10T13:39:21.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:20.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:20.831653+0000) 2022-09-10T13:39:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:21.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:21.217976+0000) 2022-09-10T13:39:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:21 smithi104 ceph-mon[97044]: pgmap v17602: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:39:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:21 smithi110 ceph-mon[97633]: pgmap v17602: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:39:22.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:21.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:21.831766+0000) 2022-09-10T13:39:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:22.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:22.218141+0000) 2022-09-10T13:39:23.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:22.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:22.831895+0000) 2022-09-10T13:39:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:23.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:23.218292+0000) 2022-09-10T13:39:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:23 smithi104 ceph-mon[97044]: pgmap v17603: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:39:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:23 smithi110 ceph-mon[97633]: pgmap v17603: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:39:24.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:23.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:23.832023+0000) 2022-09-10T13:39:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:24.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:24.218432+0000) 2022-09-10T13:39:25.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:24.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:24.832141+0000) 2022-09-10T13:39:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:25.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:25.218595+0000) 2022-09-10T13:39:25.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:25 smithi104 ceph-mon[97044]: pgmap v17604: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T13:39:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:25 smithi110 ceph-mon[97633]: pgmap v17604: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 10 op/s 2022-09-10T13:39:26.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:25.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:25.832294+0000) 2022-09-10T13:39:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:26.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:26.218734+0000) 2022-09-10T13:39:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:26 smithi110 ceph-mon[97633]: pgmap v17605: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:39:27.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:26.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:26.832427+0000) 2022-09-10T13:39:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:26 smithi104 ceph-mon[97044]: pgmap v17605: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:39:27.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:27.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:27.218900+0000) 2022-09-10T13:39:28.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:27.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:27.832553+0000) 2022-09-10T13:39:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:28.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:28.219060+0000) 2022-09-10T13:39:29.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:28.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:28.832665+0000) 2022-09-10T13:39:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:29.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:29.219228+0000) 2022-09-10T13:39:29.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:29 smithi104 ceph-mon[97044]: pgmap v17606: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:29 smithi110 ceph-mon[97633]: pgmap v17606: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:30.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:29.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:29.832776+0000) 2022-09-10T13:39:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:30.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:30.219447+0000) 2022-09-10T13:39:31.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:30.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:30.832949+0000) 2022-09-10T13:39:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:31.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:31.219737+0000) 2022-09-10T13:39:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:31 smithi104 ceph-mon[97044]: pgmap v17607: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:31 smithi110 ceph-mon[97633]: pgmap v17607: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:32.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:31.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:31.833195+0000) 2022-09-10T13:39:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:32.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:32.219930+0000) 2022-09-10T13:39:33.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:32.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:32.833408+0000) 2022-09-10T13:39:33.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:39:32.916+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:39:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:33.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:33.220203+0000) 2022-09-10T13:39:33.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:33 smithi104 ceph-mon[97044]: pgmap v17608: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:33 smithi110 ceph-mon[97633]: pgmap v17608: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:34.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:33.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:33.833569+0000) 2022-09-10T13:39:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:34.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:34.220378+0000) 2022-09-10T13:39:35.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:34.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:34.833755+0000) 2022-09-10T13:39:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:35.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:35.220538+0000) 2022-09-10T13:39:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:35 smithi104 ceph-mon[97044]: pgmap v17609: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2022-09-10T13:39:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:35 smithi110 ceph-mon[97633]: pgmap v17609: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2022-09-10T13:39:36.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:35.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:35.833973+0000) 2022-09-10T13:39:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:36.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:36.220742+0000) 2022-09-10T13:39:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:36 smithi110 ceph-mon[97633]: pgmap v17610: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:39:37.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:36.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:36.834164+0000) 2022-09-10T13:39:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:36 smithi104 ceph-mon[97044]: pgmap v17610: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:39:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:37.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:37.220979+0000) 2022-09-10T13:39:38.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:37.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:37.834398+0000) 2022-09-10T13:39:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:38.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:38.221223+0000) 2022-09-10T13:39:39.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:38.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:38.834667+0000) 2022-09-10T13:39:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:39.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:39.221458+0000) 2022-09-10T13:39:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:39 smithi104 ceph-mon[97044]: pgmap v17611: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:39 smithi110 ceph-mon[97633]: pgmap v17611: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:40.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:39.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:39.834857+0000) 2022-09-10T13:39:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:40.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:40.221683+0000) 2022-09-10T13:39:41.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:40.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:40.835054+0000) 2022-09-10T13:39:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:41.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:41.221871+0000) 2022-09-10T13:39:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:41 smithi104 ceph-mon[97044]: pgmap v17612: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:41 smithi110 ceph-mon[97633]: pgmap v17612: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:42.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:41.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:41.835214+0000) 2022-09-10T13:39:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:42.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:42.222075+0000) 2022-09-10T13:39:43.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:42.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:42.835452+0000) 2022-09-10T13:39:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:43.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:43.222340+0000) 2022-09-10T13:39:43.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:43 smithi104 ceph-mon[97044]: pgmap v17613: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:43 smithi110 ceph-mon[97633]: pgmap v17613: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:44.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:43.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:43.835732+0000) 2022-09-10T13:39:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:44.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:44.222602+0000) 2022-09-10T13:39:45.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:44.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:44.835961+0000) 2022-09-10T13:39:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:45.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:45.222781+0000) 2022-09-10T13:39:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:45 smithi104 ceph-mon[97044]: pgmap v17614: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2022-09-10T13:39:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:45 smithi110 ceph-mon[97633]: pgmap v17614: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2022-09-10T13:39:46.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:45.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:45.836181+0000) 2022-09-10T13:39:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:46.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:46.223015+0000) 2022-09-10T13:39:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:39:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:39:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:39:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:39:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:46 smithi110 ceph-mon[97633]: pgmap v17615: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:39:47.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:46.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:46.836327+0000) 2022-09-10T13:39:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:39:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:39:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:39:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:39:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:46 smithi104 ceph-mon[97044]: pgmap v17615: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:39:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:47.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:47.223182+0000) 2022-09-10T13:39:48.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:47.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:47.836488+0000) 2022-09-10T13:39:48.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:39:47.932+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:39:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:48.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:48.223371+0000) 2022-09-10T13:39:49.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:48.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:48.836734+0000) 2022-09-10T13:39:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:49.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:49.223609+0000) 2022-09-10T13:39:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:49 smithi104 ceph-mon[97044]: pgmap v17616: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:49 smithi110 ceph-mon[97633]: pgmap v17616: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:50.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:49.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:49.836979+0000) 2022-09-10T13:39:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:50.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:50.223843+0000) 2022-09-10T13:39:51.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:50.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:50.837233+0000) 2022-09-10T13:39:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:51.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:51.224086+0000) 2022-09-10T13:39:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:51 smithi104 ceph-mon[97044]: pgmap v17617: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:51.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:51 smithi110 ceph-mon[97633]: pgmap v17617: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:39:52.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:51.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:51.837466+0000) 2022-09-10T13:39:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:52.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:52.224310+0000) 2022-09-10T13:39:53.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:52.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:52.837620+0000) 2022-09-10T13:39:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:53.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:53.224510+0000) 2022-09-10T13:39:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:53 smithi104 ceph-mon[97044]: pgmap v17618: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:53 smithi110 ceph-mon[97633]: pgmap v17618: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:54.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:53.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:53.837846+0000) 2022-09-10T13:39:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:54.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:54.224765+0000) 2022-09-10T13:39:55.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:54.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:54.838043+0000) 2022-09-10T13:39:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:55.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:55.224950+0000) 2022-09-10T13:39:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:55 smithi104 ceph-mon[97044]: pgmap v17619: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2022-09-10T13:39:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:55 smithi110 ceph-mon[97633]: pgmap v17619: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2022-09-10T13:39:56.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:55.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:55.838266+0000) 2022-09-10T13:39:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:56.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:56.225206+0000) 2022-09-10T13:39:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:56 smithi110 ceph-mon[97633]: pgmap v17620: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:39:57.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:56.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:56.838482+0000) 2022-09-10T13:39:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:56 smithi104 ceph-mon[97044]: pgmap v17620: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:39:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:57.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:57.225362+0000) 2022-09-10T13:39:58.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:57.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:57.838707+0000) 2022-09-10T13:39:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:58.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:58.225558+0000) 2022-09-10T13:39:59.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:58.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:58.838956+0000) 2022-09-10T13:39:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:39:59.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:59.225679+0000) 2022-09-10T13:39:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:39:59 smithi104 ceph-mon[97044]: pgmap v17621: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:39:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:39:59 smithi110 ceph-mon[97633]: pgmap v17621: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:00.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:39:59.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:39:59.839156+0000) 2022-09-10T13:40:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:00.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:00.225803+0000) 2022-09-10T13:40:00.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:40:00.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:40:01.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:00.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:00.839318+0000) 2022-09-10T13:40:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:01.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:01.225950+0000) 2022-09-10T13:40:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:01 smithi104 ceph-mon[97044]: pgmap v17622: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:01 smithi110 ceph-mon[97633]: pgmap v17622: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:02.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:01.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:01.839527+0000) 2022-09-10T13:40:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:02.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:02.226201+0000) 2022-09-10T13:40:03.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:02.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:02.839714+0000) 2022-09-10T13:40:03.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:40:02.947+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:40:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:03.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:03.226344+0000) 2022-09-10T13:40:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:03 smithi104 ceph-mon[97044]: pgmap v17623: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:03.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:03 smithi110 ceph-mon[97633]: pgmap v17623: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:04.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:03.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:03.839966+0000) 2022-09-10T13:40:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:04.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:04.226525+0000) 2022-09-10T13:40:05.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:04.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:04.840195+0000) 2022-09-10T13:40:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:05.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:05.226692+0000) 2022-09-10T13:40:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:05 smithi104 ceph-mon[97044]: pgmap v17624: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T13:40:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:05 smithi110 ceph-mon[97633]: pgmap v17624: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2022-09-10T13:40:06.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:05.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:05.840421+0000) 2022-09-10T13:40:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:06.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:06.226901+0000) 2022-09-10T13:40:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:06 smithi110 ceph-mon[97633]: pgmap v17625: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T13:40:07.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:06.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:06.840587+0000) 2022-09-10T13:40:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:06 smithi104 ceph-mon[97044]: pgmap v17625: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 24 op/s 2022-09-10T13:40:07.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:07.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:07.227061+0000) 2022-09-10T13:40:08.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:07.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:07.840808+0000) 2022-09-10T13:40:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:08.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:08.227281+0000) 2022-09-10T13:40:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:08 smithi110 ceph-mon[97633]: osdmap e1182: 12 total, 12 up, 12 in 2022-09-10T13:40:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:08 smithi110 ceph-mon[97633]: pgmap v17627: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:40:09.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:08.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:08.841030+0000) 2022-09-10T13:40:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:08 smithi104 ceph-mon[97044]: osdmap e1182: 12 total, 12 up, 12 in 2022-09-10T13:40:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:08 smithi104 ceph-mon[97044]: pgmap v17627: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:40:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:09.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:09.227424+0000) 2022-09-10T13:40:10.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:09.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:09.841234+0000) 2022-09-10T13:40:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:10.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:10.227610+0000) 2022-09-10T13:40:11.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:10.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:10.841456+0000) 2022-09-10T13:40:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:11.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:11.227800+0000) 2022-09-10T13:40:11.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:11 smithi104 ceph-mon[97044]: pgmap v17628: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:40:11.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:11 smithi104 ceph-mon[97044]: osdmap e1183: 12 total, 12 up, 12 in 2022-09-10T13:40:11.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:11 smithi110 ceph-mon[97633]: pgmap v17628: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:40:11.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:11 smithi110 ceph-mon[97633]: osdmap e1183: 12 total, 12 up, 12 in 2022-09-10T13:40:12.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:11.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:11.841686+0000) 2022-09-10T13:40:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:12.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:12.227977+0000) 2022-09-10T13:40:13.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:12.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:12.841929+0000) 2022-09-10T13:40:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:13.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:13.228152+0000) 2022-09-10T13:40:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:13 smithi104 ceph-mon[97044]: pgmap v17630: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-09-10T13:40:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:13 smithi110 ceph-mon[97633]: pgmap v17630: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 12 op/s 2022-09-10T13:40:14.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:13.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:13.842156+0000) 2022-09-10T13:40:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:14.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:14.228288+0000) 2022-09-10T13:40:15.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:14.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:14.842370+0000) 2022-09-10T13:40:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:15.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:15.228418+0000) 2022-09-10T13:40:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:15 smithi104 ceph-mon[97044]: pgmap v17631: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T13:40:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:15 smithi110 ceph-mon[97633]: pgmap v17631: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T13:40:16.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:15.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:15.842647+0000) 2022-09-10T13:40:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:16.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:16.228647+0000) 2022-09-10T13:40:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:16 smithi110 ceph-mon[97633]: pgmap v17632: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:40:17.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:16.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:16.842864+0000) 2022-09-10T13:40:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:16 smithi104 ceph-mon[97044]: pgmap v17632: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:40:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:17.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:17.228882+0000) 2022-09-10T13:40:18.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:17.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:17.843137+0000) 2022-09-10T13:40:18.230 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:40:17.954+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:40:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:18.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:18.229179+0000) 2022-09-10T13:40:19.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:18.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:18.843306+0000) 2022-09-10T13:40:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:19.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:19.229386+0000) 2022-09-10T13:40:19.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:19 smithi104 ceph-mon[97044]: pgmap v17633: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:40:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:19 smithi110 ceph-mon[97633]: pgmap v17633: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:40:20.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:19.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:19.843488+0000) 2022-09-10T13:40:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:20.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:20.229576+0000) 2022-09-10T13:40:21.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:20.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:20.843738+0000) 2022-09-10T13:40:21.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:21.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:21.229752+0000) 2022-09-10T13:40:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:21 smithi104 ceph-mon[97044]: pgmap v17634: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:40:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:21 smithi110 ceph-mon[97633]: pgmap v17634: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:40:22.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:21.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:21.843971+0000) 2022-09-10T13:40:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:22.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:22.230039+0000) 2022-09-10T13:40:23.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:22.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:22.844189+0000) 2022-09-10T13:40:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:23.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:23.230152+0000) 2022-09-10T13:40:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:23 smithi104 ceph-mon[97044]: pgmap v17635: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:40:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:23 smithi110 ceph-mon[97633]: pgmap v17635: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:40:24.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:23.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:23.844438+0000) 2022-09-10T13:40:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:24.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:24.230368+0000) 2022-09-10T13:40:25.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:24.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:24.844648+0000) 2022-09-10T13:40:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:25.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:25.230627+0000) 2022-09-10T13:40:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:25 smithi104 ceph-mon[97044]: pgmap v17636: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:40:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:25 smithi110 ceph-mon[97633]: pgmap v17636: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:40:26.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:25.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:25.844880+0000) 2022-09-10T13:40:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:26.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:26.230857+0000) 2022-09-10T13:40:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:26 smithi110 ceph-mon[97633]: pgmap v17637: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:40:27.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:26.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:26.845061+0000) 2022-09-10T13:40:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:26 smithi104 ceph-mon[97044]: pgmap v17637: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:40:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:27.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:27.231081+0000) 2022-09-10T13:40:28.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:27.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:27.845266+0000) 2022-09-10T13:40:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:28.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:28.231301+0000) 2022-09-10T13:40:29.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:28.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:28.845520+0000) 2022-09-10T13:40:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:29.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:29.231541+0000) 2022-09-10T13:40:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:29 smithi104 ceph-mon[97044]: pgmap v17638: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:29 smithi110 ceph-mon[97633]: pgmap v17638: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:30.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:29.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:29.845751+0000) 2022-09-10T13:40:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:30.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:30.231807+0000) 2022-09-10T13:40:31.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:30.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:30.845994+0000) 2022-09-10T13:40:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:31.230+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:31.231973+0000) 2022-09-10T13:40:31.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:31 smithi104 ceph-mon[97044]: pgmap v17639: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:31 smithi110 ceph-mon[97633]: pgmap v17639: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:32.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:31.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:31.846211+0000) 2022-09-10T13:40:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:32.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:32.232134+0000) 2022-09-10T13:40:33.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:32.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:32.846471+0000) 2022-09-10T13:40:33.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:40:32.967+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:40:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:33.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:33.232351+0000) 2022-09-10T13:40:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:33 smithi104 ceph-mon[97044]: pgmap v17640: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:40:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:33 smithi110 ceph-mon[97633]: pgmap v17640: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:40:34.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:33.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:33.846672+0000) 2022-09-10T13:40:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:34.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:34.232554+0000) 2022-09-10T13:40:35.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:34.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:34.846831+0000) 2022-09-10T13:40:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:35.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:35.232708+0000) 2022-09-10T13:40:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:35 smithi104 ceph-mon[97044]: pgmap v17641: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:40:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:35 smithi110 ceph-mon[97633]: pgmap v17641: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:40:36.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:35.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:35.847059+0000) 2022-09-10T13:40:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:36.231+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:36.232876+0000) 2022-09-10T13:40:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:36 smithi110 ceph-mon[97633]: pgmap v17642: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:40:37.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:36.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:36.847345+0000) 2022-09-10T13:40:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:36 smithi104 ceph-mon[97044]: pgmap v17642: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:40:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:37.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:37.233167+0000) 2022-09-10T13:40:38.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:37.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:37.847583+0000) 2022-09-10T13:40:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:38.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:38.233437+0000) 2022-09-10T13:40:39.233 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:38.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:38.847844+0000) 2022-09-10T13:40:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:39.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:39.233603+0000) 2022-09-10T13:40:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:39 smithi104 ceph-mon[97044]: pgmap v17643: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:40:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:39 smithi110 ceph-mon[97633]: pgmap v17643: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:40:40.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:39.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:39.847975+0000) 2022-09-10T13:40:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:40.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:40.233834+0000) 2022-09-10T13:40:41.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:40.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:40.848192+0000) 2022-09-10T13:40:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:41.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:41.234075+0000) 2022-09-10T13:40:41.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:41 smithi104 ceph-mon[97044]: pgmap v17644: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:41 smithi110 ceph-mon[97633]: pgmap v17644: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:42.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:41.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:41.848431+0000) 2022-09-10T13:40:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:42.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:42.234260+0000) 2022-09-10T13:40:43.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:42.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:42.848675+0000) 2022-09-10T13:40:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:43.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:43.234446+0000) 2022-09-10T13:40:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:43 smithi104 ceph-mon[97044]: pgmap v17645: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:43 smithi110 ceph-mon[97633]: pgmap v17645: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:44.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:43.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:43.848878+0000) 2022-09-10T13:40:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:44.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:44.234603+0000) 2022-09-10T13:40:45.234 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:44.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:44.849054+0000) 2022-09-10T13:40:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:45.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:45.234848+0000) 2022-09-10T13:40:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:45 smithi104 ceph-mon[97044]: pgmap v17646: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:40:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:45 smithi110 ceph-mon[97633]: pgmap v17646: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:40:46.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:45.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:45.849292+0000) 2022-09-10T13:40:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:46.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:46.235076+0000) 2022-09-10T13:40:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:40:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:40:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:40:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:40:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:46 smithi110 ceph-mon[97633]: pgmap v17647: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:40:47.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:46.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:46.849503+0000) 2022-09-10T13:40:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:40:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:40:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:46 smithi104 ceph-mon[97044]: pgmap v17647: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:40:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:47.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:47.235376+0000) 2022-09-10T13:40:48.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:47.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:47.849719+0000) 2022-09-10T13:40:48.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:40:47.968+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:40:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:48.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:48.235604+0000) 2022-09-10T13:40:49.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:48.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:48.849967+0000) 2022-09-10T13:40:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:49.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:49.235846+0000) 2022-09-10T13:40:49.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:49 smithi104 ceph-mon[97044]: pgmap v17648: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:49 smithi110 ceph-mon[97633]: pgmap v17648: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:50.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:49.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:49.850200+0000) 2022-09-10T13:40:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:50.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:50.236179+0000) 2022-09-10T13:40:51.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:50.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:50.850359+0000) 2022-09-10T13:40:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:51.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:51.236319+0000) 2022-09-10T13:40:51.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:51 smithi104 ceph-mon[97044]: pgmap v17649: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:51 smithi110 ceph-mon[97633]: pgmap v17649: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:40:52.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:51.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:51.850589+0000) 2022-09-10T13:40:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:52.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:52.236513+0000) 2022-09-10T13:40:53.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:52.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:52.850802+0000) 2022-09-10T13:40:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:53.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:53.236744+0000) 2022-09-10T13:40:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:53 smithi104 ceph-mon[97044]: pgmap v17650: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:53 smithi110 ceph-mon[97633]: pgmap v17650: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:54.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:53.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:53.851070+0000) 2022-09-10T13:40:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:54.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:54.236962+0000) 2022-09-10T13:40:55.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:54.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:54.851357+0000) 2022-09-10T13:40:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:55.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:55.237207+0000) 2022-09-10T13:40:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:55 smithi104 ceph-mon[97044]: pgmap v17651: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:40:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:55 smithi110 ceph-mon[97633]: pgmap v17651: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:40:56.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:55.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:55.851603+0000) 2022-09-10T13:40:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:56.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:56.237407+0000) 2022-09-10T13:40:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:56 smithi110 ceph-mon[97633]: pgmap v17652: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:40:57.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:56.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:56.851837+0000) 2022-09-10T13:40:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:56 smithi104 ceph-mon[97044]: pgmap v17652: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:40:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:57.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:57.237663+0000) 2022-09-10T13:40:58.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:57.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:57.852119+0000) 2022-09-10T13:40:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:58.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:58.237861+0000) 2022-09-10T13:40:59.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:58.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:58.852407+0000) 2022-09-10T13:40:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:40:59.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:59.238092+0000) 2022-09-10T13:40:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:40:59 smithi104 ceph-mon[97044]: pgmap v17653: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:40:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:40:59 smithi110 ceph-mon[97633]: pgmap v17653: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:00.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:40:59.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:40:59.852661+0000) 2022-09-10T13:41:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:00.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:00.238310+0000) 2022-09-10T13:41:01.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:00.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:00.852912+0000) 2022-09-10T13:41:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:01.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:01.238535+0000) 2022-09-10T13:41:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:01 smithi104 ceph-mon[97044]: pgmap v17654: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:01 smithi110 ceph-mon[97633]: pgmap v17654: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:02.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:01.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:01.853177+0000) 2022-09-10T13:41:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:02.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:02.238802+0000) 2022-09-10T13:41:03.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:02.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:02.853395+0000) 2022-09-10T13:41:03.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:41:02.976+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:41:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:03.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:03.239079+0000) 2022-09-10T13:41:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:03 smithi104 ceph-mon[97044]: pgmap v17655: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:03 smithi110 ceph-mon[97633]: pgmap v17655: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:04.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:03.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:03.853574+0000) 2022-09-10T13:41:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:04.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:04.239341+0000) 2022-09-10T13:41:05.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:04.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:04.853811+0000) 2022-09-10T13:41:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:05.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:05.239571+0000) 2022-09-10T13:41:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:05 smithi104 ceph-mon[97044]: pgmap v17656: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T13:41:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:05 smithi110 ceph-mon[97633]: pgmap v17656: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T13:41:06.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:05.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:05.854061+0000) 2022-09-10T13:41:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:06.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:06.239823+0000) 2022-09-10T13:41:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:06 smithi110 ceph-mon[97633]: pgmap v17657: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:41:07.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:06.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:06.854304+0000) 2022-09-10T13:41:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:06 smithi104 ceph-mon[97044]: pgmap v17657: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:41:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:07.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:07.240055+0000) 2022-09-10T13:41:08.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:07.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:07.854473+0000) 2022-09-10T13:41:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:08.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:08.240278+0000) 2022-09-10T13:41:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:08 smithi110 ceph-mon[97633]: osdmap e1184: 12 total, 12 up, 12 in 2022-09-10T13:41:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:08 smithi110 ceph-mon[97633]: pgmap v17659: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:41:09.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:08.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:08.854703+0000) 2022-09-10T13:41:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:08 smithi104 ceph-mon[97044]: osdmap e1184: 12 total, 12 up, 12 in 2022-09-10T13:41:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:08 smithi104 ceph-mon[97044]: pgmap v17659: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:41:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:09.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:09.240499+0000) 2022-09-10T13:41:10.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:09.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:09.854908+0000) 2022-09-10T13:41:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:10.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:10.240742+0000) 2022-09-10T13:41:11.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:10.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:10.855208+0000) 2022-09-10T13:41:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:11.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:11.240979+0000) 2022-09-10T13:41:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:11 smithi104 ceph-mon[97044]: pgmap v17660: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:41:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:11 smithi104 ceph-mon[97044]: osdmap e1185: 12 total, 12 up, 12 in 2022-09-10T13:41:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:11 smithi110 ceph-mon[97633]: pgmap v17660: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:41:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:11 smithi110 ceph-mon[97633]: osdmap e1185: 12 total, 12 up, 12 in 2022-09-10T13:41:12.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:11.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:11.855418+0000) 2022-09-10T13:41:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:12.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:12.241203+0000) 2022-09-10T13:41:13.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:12.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:12.855596+0000) 2022-09-10T13:41:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:13.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:13.241476+0000) 2022-09-10T13:41:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:13 smithi104 ceph-mon[97044]: pgmap v17662: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 10 op/s 2022-09-10T13:41:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:13 smithi110 ceph-mon[97633]: pgmap v17662: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 10 op/s 2022-09-10T13:41:14.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:13.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:13.855806+0000) 2022-09-10T13:41:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:14.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:14.241706+0000) 2022-09-10T13:41:15.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:14.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:14.856015+0000) 2022-09-10T13:41:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:15.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:15.241937+0000) 2022-09-10T13:41:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:15 smithi104 ceph-mon[97044]: pgmap v17663: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T13:41:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:15 smithi110 ceph-mon[97633]: pgmap v17663: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T13:41:16.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:15.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:15.856267+0000) 2022-09-10T13:41:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:16.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:16.242165+0000) 2022-09-10T13:41:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:16 smithi110 ceph-mon[97633]: pgmap v17664: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:41:17.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:16.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:16.856473+0000) 2022-09-10T13:41:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:16 smithi104 ceph-mon[97044]: pgmap v17664: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:41:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:17.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:17.242374+0000) 2022-09-10T13:41:18.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:17.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:17.856662+0000) 2022-09-10T13:41:18.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:41:17.992+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:41:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:18.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:18.242538+0000) 2022-09-10T13:41:19.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:18.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:18.856850+0000) 2022-09-10T13:41:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:19.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:19.242793+0000) 2022-09-10T13:41:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:19 smithi104 ceph-mon[97044]: pgmap v17665: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:41:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:19 smithi110 ceph-mon[97633]: pgmap v17665: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:41:20.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:19.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:19.857012+0000) 2022-09-10T13:41:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:20.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:20.243018+0000) 2022-09-10T13:41:21.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:20.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:20.857203+0000) 2022-09-10T13:41:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:21.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:21.243286+0000) 2022-09-10T13:41:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:21 smithi104 ceph-mon[97044]: pgmap v17666: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:41:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:21 smithi110 ceph-mon[97633]: pgmap v17666: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:41:22.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:21.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:21.857398+0000) 2022-09-10T13:41:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:22.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:22.243476+0000) 2022-09-10T13:41:23.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:22.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:22.857618+0000) 2022-09-10T13:41:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:23.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:23.243706+0000) 2022-09-10T13:41:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:23 smithi104 ceph-mon[97044]: pgmap v17667: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:41:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:23 smithi110 ceph-mon[97633]: pgmap v17667: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:41:24.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:23.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:23.857795+0000) 2022-09-10T13:41:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:24.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:24.243951+0000) 2022-09-10T13:41:25.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:24.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:24.858007+0000) 2022-09-10T13:41:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:25.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:25.244125+0000) 2022-09-10T13:41:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:25 smithi104 ceph-mon[97044]: pgmap v17668: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:41:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:25 smithi110 ceph-mon[97633]: pgmap v17668: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:41:26.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:25.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:25.858258+0000) 2022-09-10T13:41:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:26.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:26.244346+0000) 2022-09-10T13:41:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:26 smithi110 ceph-mon[97633]: pgmap v17669: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:27.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:26.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:26.858526+0000) 2022-09-10T13:41:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:26 smithi104 ceph-mon[97044]: pgmap v17669: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:27.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:27.244608+0000) 2022-09-10T13:41:28.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:27.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:27.858687+0000) 2022-09-10T13:41:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:28.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:28.244723+0000) 2022-09-10T13:41:29.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:28.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:28.858938+0000) 2022-09-10T13:41:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:29.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:29.244919+0000) 2022-09-10T13:41:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:29 smithi104 ceph-mon[97044]: pgmap v17670: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:29 smithi110 ceph-mon[97633]: pgmap v17670: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:30.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:29.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:29.859133+0000) 2022-09-10T13:41:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:30.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:30.245171+0000) 2022-09-10T13:41:31.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:30.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:30.859392+0000) 2022-09-10T13:41:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:31.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:31.245411+0000) 2022-09-10T13:41:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:31 smithi104 ceph-mon[97044]: pgmap v17671: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:31 smithi110 ceph-mon[97633]: pgmap v17671: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:32.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:31.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:31.859587+0000) 2022-09-10T13:41:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:32.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:32.245666+0000) 2022-09-10T13:41:33.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:32.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:32.859812+0000) 2022-09-10T13:41:33.247 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:41:32.992+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:41:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:33.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:33.245842+0000) 2022-09-10T13:41:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:33 smithi104 ceph-mon[97044]: pgmap v17672: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:33 smithi110 ceph-mon[97633]: pgmap v17672: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:34.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:33.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:33.859989+0000) 2022-09-10T13:41:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:34.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:34.246090+0000) 2022-09-10T13:41:35.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:34.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:34.860228+0000) 2022-09-10T13:41:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:35.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:35.246293+0000) 2022-09-10T13:41:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:35 smithi104 ceph-mon[97044]: pgmap v17673: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:41:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:35 smithi110 ceph-mon[97633]: pgmap v17673: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:41:36.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:35.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:35.860465+0000) 2022-09-10T13:41:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:36.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:36.246439+0000) 2022-09-10T13:41:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:36 smithi110 ceph-mon[97633]: pgmap v17674: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:37.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:36.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:36.860705+0000) 2022-09-10T13:41:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:36 smithi104 ceph-mon[97044]: pgmap v17674: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:37.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:37.246578+0000) 2022-09-10T13:41:38.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:37.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:37.860945+0000) 2022-09-10T13:41:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:38.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:38.246821+0000) 2022-09-10T13:41:39.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:38.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:38.861175+0000) 2022-09-10T13:41:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:39.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:39.247054+0000) 2022-09-10T13:41:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:39 smithi104 ceph-mon[97044]: pgmap v17675: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:39 smithi110 ceph-mon[97633]: pgmap v17675: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:40.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:39.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:39.861387+0000) 2022-09-10T13:41:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:40.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:40.247324+0000) 2022-09-10T13:41:41.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:40.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:40.861563+0000) 2022-09-10T13:41:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:41.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:41.247599+0000) 2022-09-10T13:41:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:41 smithi104 ceph-mon[97044]: pgmap v17676: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:41.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:41 smithi110 ceph-mon[97633]: pgmap v17676: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:42.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:41.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:41.861750+0000) 2022-09-10T13:41:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:42.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:42.247783+0000) 2022-09-10T13:41:43.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:42.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:42.861960+0000) 2022-09-10T13:41:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:43.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:43.247955+0000) 2022-09-10T13:41:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:43 smithi104 ceph-mon[97044]: pgmap v17677: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:43 smithi110 ceph-mon[97633]: pgmap v17677: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:44.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:43.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:43.862174+0000) 2022-09-10T13:41:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:44.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:44.248206+0000) 2022-09-10T13:41:45.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:44.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:44.862347+0000) 2022-09-10T13:41:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:45.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:45.248373+0000) 2022-09-10T13:41:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:45 smithi104 ceph-mon[97044]: pgmap v17678: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:41:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:45 smithi110 ceph-mon[97633]: pgmap v17678: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:41:46.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:45.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:45.862537+0000) 2022-09-10T13:41:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:46.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:46.248634+0000) 2022-09-10T13:41:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:41:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:41:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:41:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:41:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:46 smithi110 ceph-mon[97633]: pgmap v17679: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:47.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:46.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:46.862759+0000) 2022-09-10T13:41:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:41:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:41:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:46 smithi104 ceph-mon[97044]: pgmap v17679: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:47.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:47.248821+0000) 2022-09-10T13:41:48.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:47.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:47.862971+0000) 2022-09-10T13:41:48.249 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:41:48.007+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:41:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:48.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:48.248938+0000) 2022-09-10T13:41:49.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:48.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:48.863208+0000) 2022-09-10T13:41:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:49.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:49.249131+0000) 2022-09-10T13:41:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:49 smithi104 ceph-mon[97044]: pgmap v17680: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:49.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:49 smithi110 ceph-mon[97633]: pgmap v17680: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:50.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:49.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:49.863438+0000) 2022-09-10T13:41:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:50.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:50.249375+0000) 2022-09-10T13:41:51.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:50.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:50.863676+0000) 2022-09-10T13:41:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:51.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:51.249560+0000) 2022-09-10T13:41:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:51 smithi104 ceph-mon[97044]: pgmap v17681: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:51 smithi110 ceph-mon[97633]: pgmap v17681: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:41:52.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:51.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:51.863917+0000) 2022-09-10T13:41:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:52.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:52.249794+0000) 2022-09-10T13:41:53.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:52.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:52.864171+0000) 2022-09-10T13:41:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:53.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:53.249943+0000) 2022-09-10T13:41:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:53 smithi104 ceph-mon[97044]: pgmap v17682: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:53 smithi110 ceph-mon[97633]: pgmap v17682: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:54.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:53.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:53.864360+0000) 2022-09-10T13:41:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:54.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:54.250186+0000) 2022-09-10T13:41:55.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:54.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:54.864596+0000) 2022-09-10T13:41:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:55.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:55.250434+0000) 2022-09-10T13:41:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:55 smithi104 ceph-mon[97044]: pgmap v17683: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:41:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:55 smithi110 ceph-mon[97633]: pgmap v17683: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:41:56.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:55.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:55.864855+0000) 2022-09-10T13:41:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:56.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:56.250619+0000) 2022-09-10T13:41:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:56 smithi110 ceph-mon[97633]: pgmap v17684: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:57.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:56.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:56.864985+0000) 2022-09-10T13:41:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:56 smithi104 ceph-mon[97044]: pgmap v17684: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:41:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:57.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:57.250841+0000) 2022-09-10T13:41:58.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:57.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:57.865190+0000) 2022-09-10T13:41:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:58.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:58.251083+0000) 2022-09-10T13:41:59.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:58.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:58.865425+0000) 2022-09-10T13:41:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:41:59.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:59.251322+0000) 2022-09-10T13:41:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:41:59 smithi104 ceph-mon[97044]: pgmap v17685: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:41:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:41:59 smithi110 ceph-mon[97633]: pgmap v17685: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:00.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:41:59.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:41:59.865629+0000) 2022-09-10T13:42:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:00.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:00.251536+0000) 2022-09-10T13:42:01.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:00.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:00.865745+0000) 2022-09-10T13:42:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:01.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:01.251647+0000) 2022-09-10T13:42:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:01 smithi104 ceph-mon[97044]: pgmap v17686: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:01.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:01 smithi110 ceph-mon[97633]: pgmap v17686: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:02.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:01.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:01.865968+0000) 2022-09-10T13:42:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:02.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:02.251813+0000) 2022-09-10T13:42:03.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:02.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:02.866233+0000) 2022-09-10T13:42:03.252 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:42:03.013+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:42:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:03.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:03.252023+0000) 2022-09-10T13:42:03.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:03 smithi104 ceph-mon[97044]: pgmap v17687: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:03 smithi110 ceph-mon[97633]: pgmap v17687: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:04.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:03.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:03.866450+0000) 2022-09-10T13:42:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:04.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:04.252268+0000) 2022-09-10T13:42:05.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:04.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:04.866690+0000) 2022-09-10T13:42:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:05.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:05.252507+0000) 2022-09-10T13:42:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:05 smithi104 ceph-mon[97044]: pgmap v17688: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T13:42:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:05 smithi110 ceph-mon[97633]: pgmap v17688: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T13:42:06.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:05.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:05.866931+0000) 2022-09-10T13:42:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:06.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:06.252766+0000) 2022-09-10T13:42:07.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:06 smithi110 ceph-mon[97633]: pgmap v17689: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:42:07.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:06.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:06.867187+0000) 2022-09-10T13:42:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:06 smithi104 ceph-mon[97044]: pgmap v17689: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:42:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:07.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:07.253031+0000) 2022-09-10T13:42:08.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:07.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:07.867358+0000) 2022-09-10T13:42:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:08.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:08.253272+0000) 2022-09-10T13:42:09.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:08 smithi110 ceph-mon[97633]: osdmap e1186: 12 total, 12 up, 12 in 2022-09-10T13:42:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:08 smithi110 ceph-mon[97633]: pgmap v17691: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:42:09.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:08.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:08.867584+0000) 2022-09-10T13:42:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:08 smithi104 ceph-mon[97044]: osdmap e1186: 12 total, 12 up, 12 in 2022-09-10T13:42:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:08 smithi104 ceph-mon[97044]: pgmap v17691: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:42:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:09.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:09.253533+0000) 2022-09-10T13:42:10.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:09.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:09.867801+0000) 2022-09-10T13:42:10.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:10.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:10.253773+0000) 2022-09-10T13:42:11.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:10.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:10.867974+0000) 2022-09-10T13:42:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:11.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:11.253973+0000) 2022-09-10T13:42:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:11 smithi104 ceph-mon[97044]: pgmap v17692: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:42:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:11 smithi104 ceph-mon[97044]: osdmap e1187: 12 total, 12 up, 12 in 2022-09-10T13:42:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:11 smithi110 ceph-mon[97633]: pgmap v17692: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:42:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:11 smithi110 ceph-mon[97633]: osdmap e1187: 12 total, 12 up, 12 in 2022-09-10T13:42:12.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:11.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:11.868174+0000) 2022-09-10T13:42:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:12.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:12.254181+0000) 2022-09-10T13:42:13.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:12.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:12.868358+0000) 2022-09-10T13:42:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:13.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:13.254390+0000) 2022-09-10T13:42:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:13 smithi104 ceph-mon[97044]: pgmap v17694: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 10 op/s 2022-09-10T13:42:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:13 smithi110 ceph-mon[97633]: pgmap v17694: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 10 op/s 2022-09-10T13:42:14.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:13.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:13.868551+0000) 2022-09-10T13:42:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:14.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:14.254601+0000) 2022-09-10T13:42:15.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:14.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:14.868781+0000) 2022-09-10T13:42:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:15.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:15.254871+0000) 2022-09-10T13:42:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:15 smithi104 ceph-mon[97044]: pgmap v17695: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T13:42:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:15 smithi110 ceph-mon[97633]: pgmap v17695: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 11 op/s 2022-09-10T13:42:16.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:15.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:15.869011+0000) 2022-09-10T13:42:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:16.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:16.255124+0000) 2022-09-10T13:42:17.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:16 smithi110 ceph-mon[97633]: pgmap v17696: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:42:17.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:16.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:16.869277+0000) 2022-09-10T13:42:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:16 smithi104 ceph-mon[97044]: pgmap v17696: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:42:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:17.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:17.255254+0000) 2022-09-10T13:42:18.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:17.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:17.869414+0000) 2022-09-10T13:42:18.256 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:42:18.029+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:42:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:18.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:18.255458+0000) 2022-09-10T13:42:19.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:18.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:18.869631+0000) 2022-09-10T13:42:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:19.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:19.255679+0000) 2022-09-10T13:42:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:19 smithi104 ceph-mon[97044]: pgmap v17697: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:42:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:19 smithi110 ceph-mon[97633]: pgmap v17697: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:42:20.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:19.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:19.869804+0000) 2022-09-10T13:42:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:20.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:20.255869+0000) 2022-09-10T13:42:21.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:20.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:20.869962+0000) 2022-09-10T13:42:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:21.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:21.256146+0000) 2022-09-10T13:42:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:21 smithi104 ceph-mon[97044]: pgmap v17698: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:42:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:21 smithi110 ceph-mon[97633]: pgmap v17698: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:42:22.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:21.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:21.870144+0000) 2022-09-10T13:42:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:22.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:22.256387+0000) 2022-09-10T13:42:23.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:22.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:22.870268+0000) 2022-09-10T13:42:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:23.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:23.256577+0000) 2022-09-10T13:42:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:23 smithi104 ceph-mon[97044]: pgmap v17699: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:42:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:23 smithi110 ceph-mon[97633]: pgmap v17699: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:42:24.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:23.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:23.870471+0000) 2022-09-10T13:42:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:24.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:24.256709+0000) 2022-09-10T13:42:25.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:24.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:24.870658+0000) 2022-09-10T13:42:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:25.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:25.256938+0000) 2022-09-10T13:42:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:25 smithi104 ceph-mon[97044]: pgmap v17700: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:42:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:25 smithi110 ceph-mon[97633]: pgmap v17700: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T13:42:26.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:25.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:25.870872+0000) 2022-09-10T13:42:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:26.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:26.257191+0000) 2022-09-10T13:42:27.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:26 smithi110 ceph-mon[97633]: pgmap v17701: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:27.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:26.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:26.871166+0000) 2022-09-10T13:42:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:26 smithi104 ceph-mon[97044]: pgmap v17701: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:27.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:27.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:27.257441+0000) 2022-09-10T13:42:28.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:27.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:27.871433+0000) 2022-09-10T13:42:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:28.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:28.257678+0000) 2022-09-10T13:42:29.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:28.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:28.871735+0000) 2022-09-10T13:42:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:29.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:29.257899+0000) 2022-09-10T13:42:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:29 smithi104 ceph-mon[97044]: pgmap v17702: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:29 smithi110 ceph-mon[97633]: pgmap v17702: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:30.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:29.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:29.871855+0000) 2022-09-10T13:42:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:30.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:30.258086+0000) 2022-09-10T13:42:31.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:30.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:30.872089+0000) 2022-09-10T13:42:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:31.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:31.258228+0000) 2022-09-10T13:42:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:31 smithi104 ceph-mon[97044]: pgmap v17703: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:31 smithi110 ceph-mon[97633]: pgmap v17703: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:32.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:31.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:31.872371+0000) 2022-09-10T13:42:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:32.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:32.258382+0000) 2022-09-10T13:42:33.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:32.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:32.872575+0000) 2022-09-10T13:42:33.259 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:42:33.031+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:42:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:33.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:33.258594+0000) 2022-09-10T13:42:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:33 smithi104 ceph-mon[97044]: pgmap v17704: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:33 smithi110 ceph-mon[97633]: pgmap v17704: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:34.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:33.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:33.872795+0000) 2022-09-10T13:42:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:34.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:34.258882+0000) 2022-09-10T13:42:35.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:34.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:34.872988+0000) 2022-09-10T13:42:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:35.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:35.259066+0000) 2022-09-10T13:42:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:35 smithi104 ceph-mon[97044]: pgmap v17705: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:42:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:35 smithi110 ceph-mon[97633]: pgmap v17705: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:42:36.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:35.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:35.873205+0000) 2022-09-10T13:42:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:36.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:36.259325+0000) 2022-09-10T13:42:37.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:36 smithi110 ceph-mon[97633]: pgmap v17706: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:37.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:36.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:36.873459+0000) 2022-09-10T13:42:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:36 smithi104 ceph-mon[97044]: pgmap v17706: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:37.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:37.259534+0000) 2022-09-10T13:42:38.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:37.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:37.873686+0000) 2022-09-10T13:42:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:38.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:38.259788+0000) 2022-09-10T13:42:39.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:38.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:38.873885+0000) 2022-09-10T13:42:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:39.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:39.259964+0000) 2022-09-10T13:42:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:39 smithi104 ceph-mon[97044]: pgmap v17707: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:39 smithi110 ceph-mon[97633]: pgmap v17707: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:40.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:39.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:39.874082+0000) 2022-09-10T13:42:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:40.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:40.260223+0000) 2022-09-10T13:42:41.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:40.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:40.874243+0000) 2022-09-10T13:42:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:41.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:41.260416+0000) 2022-09-10T13:42:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:41 smithi104 ceph-mon[97044]: pgmap v17708: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:41 smithi110 ceph-mon[97633]: pgmap v17708: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:42.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:41.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:41.874365+0000) 2022-09-10T13:42:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:42.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:42.260599+0000) 2022-09-10T13:42:43.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:42.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:42.874534+0000) 2022-09-10T13:42:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:43.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:43.260749+0000) 2022-09-10T13:42:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:43 smithi104 ceph-mon[97044]: pgmap v17709: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:43 smithi110 ceph-mon[97633]: pgmap v17709: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:44.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:43.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:43.874742+0000) 2022-09-10T13:42:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:44.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:44.260949+0000) 2022-09-10T13:42:45.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:44.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:44.874948+0000) 2022-09-10T13:42:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:45.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:45.261169+0000) 2022-09-10T13:42:45.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:45 smithi104 ceph-mon[97044]: pgmap v17710: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:42:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:45 smithi110 ceph-mon[97633]: pgmap v17710: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:42:46.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:45.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:45.875130+0000) 2022-09-10T13:42:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:46.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:46.261411+0000) 2022-09-10T13:42:47.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:42:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:42:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:42:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:42:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:46 smithi110 ceph-mon[97633]: pgmap v17711: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:47.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:46.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:46.875382+0000) 2022-09-10T13:42:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:42:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:42:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:42:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:42:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:46 smithi104 ceph-mon[97044]: pgmap v17711: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:47.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:47.261664+0000) 2022-09-10T13:42:48.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:47.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:47.875561+0000) 2022-09-10T13:42:48.262 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:42:48.043+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:42:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:48.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:48.261918+0000) 2022-09-10T13:42:49.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:48.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:48.875744+0000) 2022-09-10T13:42:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:49.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:49.262143+0000) 2022-09-10T13:42:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:49 smithi104 ceph-mon[97044]: pgmap v17712: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:49 smithi110 ceph-mon[97633]: pgmap v17712: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:50.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:49.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:49.875953+0000) 2022-09-10T13:42:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:50.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:50.262383+0000) 2022-09-10T13:42:51.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:50.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:50.876211+0000) 2022-09-10T13:42:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:51.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:51.262627+0000) 2022-09-10T13:42:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:51 smithi104 ceph-mon[97044]: pgmap v17713: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:51 smithi110 ceph-mon[97633]: pgmap v17713: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:42:52.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:51.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:51.876500+0000) 2022-09-10T13:42:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:52.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:52.262865+0000) 2022-09-10T13:42:53.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:52.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:52.876727+0000) 2022-09-10T13:42:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:53.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:53.263134+0000) 2022-09-10T13:42:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:53 smithi104 ceph-mon[97044]: pgmap v17714: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:53 smithi110 ceph-mon[97633]: pgmap v17714: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:54.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:53.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:53.876917+0000) 2022-09-10T13:42:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:54.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:54.263321+0000) 2022-09-10T13:42:55.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:54.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:54.877138+0000) 2022-09-10T13:42:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:55.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:55.263505+0000) 2022-09-10T13:42:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:55 smithi104 ceph-mon[97044]: pgmap v17715: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:42:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:55 smithi110 ceph-mon[97633]: pgmap v17715: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:42:56.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:55.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:55.877439+0000) 2022-09-10T13:42:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:56.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:56.263751+0000) 2022-09-10T13:42:57.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:56 smithi110 ceph-mon[97633]: pgmap v17716: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:57.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:56.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:56.877605+0000) 2022-09-10T13:42:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:56 smithi104 ceph-mon[97044]: pgmap v17716: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:42:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:57.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:57.264009+0000) 2022-09-10T13:42:58.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:57.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:57.877855+0000) 2022-09-10T13:42:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:58.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:58.264283+0000) 2022-09-10T13:42:59.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:58.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:58.878028+0000) 2022-09-10T13:42:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:42:59.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:59.264509+0000) 2022-09-10T13:42:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:42:59 smithi104 ceph-mon[97044]: pgmap v17717: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:42:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:42:59 smithi110 ceph-mon[97633]: pgmap v17717: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:00.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:42:59.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:42:59.878267+0000) 2022-09-10T13:43:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:00.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:00.264794+0000) 2022-09-10T13:43:01.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:00.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:00.878429+0000) 2022-09-10T13:43:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:01.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:01.264947+0000) 2022-09-10T13:43:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:01 smithi104 ceph-mon[97044]: pgmap v17718: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:01 smithi110 ceph-mon[97633]: pgmap v17718: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:02.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:01.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:01.878683+0000) 2022-09-10T13:43:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:02.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:02.265081+0000) 2022-09-10T13:43:03.265 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:43:03.057+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:43:03.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:02.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:02.878954+0000) 2022-09-10T13:43:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:03.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:03.265313+0000) 2022-09-10T13:43:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:03 smithi104 ceph-mon[97044]: pgmap v17719: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:03 smithi110 ceph-mon[97633]: pgmap v17719: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:04.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:03.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:03.879245+0000) 2022-09-10T13:43:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:04.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:04.265522+0000) 2022-09-10T13:43:05.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:04.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:04.879459+0000) 2022-09-10T13:43:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:05.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:05.265789+0000) 2022-09-10T13:43:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:05 smithi104 ceph-mon[97044]: pgmap v17720: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T13:43:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:05 smithi110 ceph-mon[97633]: pgmap v17720: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2022-09-10T13:43:06.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:05.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:05.879699+0000) 2022-09-10T13:43:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:06.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:06.266064+0000) 2022-09-10T13:43:07.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:06.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:06.879926+0000) 2022-09-10T13:43:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:06 smithi104 ceph-mon[97044]: pgmap v17721: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:43:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:06 smithi110 ceph-mon[97633]: pgmap v17721: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:43:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:07.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:07.266383+0000) 2022-09-10T13:43:08.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:07.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:07.880151+0000) 2022-09-10T13:43:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:08.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:08.266623+0000) 2022-09-10T13:43:09.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:08.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:08.880378+0000) 2022-09-10T13:43:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:08 smithi104 ceph-mon[97044]: osdmap e1188: 12 total, 12 up, 12 in 2022-09-10T13:43:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:08 smithi104 ceph-mon[97044]: pgmap v17723: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:43:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:08 smithi110 ceph-mon[97633]: osdmap e1188: 12 total, 12 up, 12 in 2022-09-10T13:43:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:08 smithi110 ceph-mon[97633]: pgmap v17723: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:43:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:09.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:09.266807+0000) 2022-09-10T13:43:10.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:09.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:09.880595+0000) 2022-09-10T13:43:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:10.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:10.266978+0000) 2022-09-10T13:43:11.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:10.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:10.880806+0000) 2022-09-10T13:43:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:11.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:11.267154+0000) 2022-09-10T13:43:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:11 smithi104 ceph-mon[97044]: pgmap v17724: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:43:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:11 smithi104 ceph-mon[97044]: osdmap e1189: 12 total, 12 up, 12 in 2022-09-10T13:43:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:11 smithi110 ceph-mon[97633]: pgmap v17724: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:43:11.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:11 smithi110 ceph-mon[97633]: osdmap e1189: 12 total, 12 up, 12 in 2022-09-10T13:43:12.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:11.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:11.881024+0000) 2022-09-10T13:43:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:12.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:12.267368+0000) 2022-09-10T13:43:13.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:12.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:12.881225+0000) 2022-09-10T13:43:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:13.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:13.267541+0000) 2022-09-10T13:43:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:13 smithi104 ceph-mon[97044]: pgmap v17726: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 10 op/s 2022-09-10T13:43:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:13 smithi110 ceph-mon[97633]: pgmap v17726: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 34 KiB/s wr, 10 op/s 2022-09-10T13:43:14.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:13.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:13.881446+0000) 2022-09-10T13:43:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:14.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:14.267731+0000) 2022-09-10T13:43:15.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:14.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:14.881644+0000) 2022-09-10T13:43:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:15.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:15.267942+0000) 2022-09-10T13:43:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:15 smithi104 ceph-mon[97044]: pgmap v17727: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 383 B/s wr, 11 op/s 2022-09-10T13:43:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:15 smithi110 ceph-mon[97633]: pgmap v17727: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 383 B/s wr, 11 op/s 2022-09-10T13:43:16.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:15.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:15.881882+0000) 2022-09-10T13:43:16.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:16.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:16.268180+0000) 2022-09-10T13:43:17.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:16.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:16.882088+0000) 2022-09-10T13:43:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:16 smithi104 ceph-mon[97044]: pgmap v17728: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 360 B/s wr, 12 op/s 2022-09-10T13:43:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:16 smithi110 ceph-mon[97633]: pgmap v17728: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 360 B/s wr, 12 op/s 2022-09-10T13:43:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:17.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:17.268424+0000) 2022-09-10T13:43:18.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:17.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:17.882325+0000) 2022-09-10T13:43:18.269 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:43:18.068+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:43:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:18.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:18.268651+0000) 2022-09-10T13:43:19.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:18.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:18.882552+0000) 2022-09-10T13:43:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:19.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:19.268858+0000) 2022-09-10T13:43:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:19 smithi104 ceph-mon[97044]: pgmap v17729: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 307 B/s wr, 10 op/s 2022-09-10T13:43:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:19 smithi110 ceph-mon[97633]: pgmap v17729: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 307 B/s wr, 10 op/s 2022-09-10T13:43:20.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:19.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:19.882748+0000) 2022-09-10T13:43:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:20.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:20.269016+0000) 2022-09-10T13:43:21.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:20.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:20.882990+0000) 2022-09-10T13:43:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:21.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:21.269224+0000) 2022-09-10T13:43:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:21 smithi104 ceph-mon[97044]: pgmap v17730: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:43:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:21 smithi110 ceph-mon[97633]: pgmap v17730: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:43:22.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:21.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:21.883233+0000) 2022-09-10T13:43:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:22.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:22.269419+0000) 2022-09-10T13:43:23.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:22.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:22.883480+0000) 2022-09-10T13:43:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:23.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:23.269672+0000) 2022-09-10T13:43:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:23 smithi104 ceph-mon[97044]: pgmap v17731: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 177 B/s wr, 8 op/s 2022-09-10T13:43:23.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:23 smithi110 ceph-mon[97633]: pgmap v17731: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 177 B/s wr, 8 op/s 2022-09-10T13:43:24.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:23.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:23.883748+0000) 2022-09-10T13:43:24.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:24.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:24.269912+0000) 2022-09-10T13:43:25.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:24.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:24.883995+0000) 2022-09-10T13:43:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:25.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:25.270177+0000) 2022-09-10T13:43:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:25 smithi104 ceph-mon[97044]: pgmap v17732: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2022-09-10T13:43:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:25 smithi110 ceph-mon[97633]: pgmap v17732: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 11 op/s 2022-09-10T13:43:26.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:25.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:25.884258+0000) 2022-09-10T13:43:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:26.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:26.270353+0000) 2022-09-10T13:43:27.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:26.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:26.884429+0000) 2022-09-10T13:43:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:26 smithi104 ceph-mon[97044]: pgmap v17733: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T13:43:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:26 smithi110 ceph-mon[97633]: pgmap v17733: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T13:43:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:27.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:27.270541+0000) 2022-09-10T13:43:28.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:27.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:27.884675+0000) 2022-09-10T13:43:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:28.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:28.270716+0000) 2022-09-10T13:43:29.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:28.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:28.884875+0000) 2022-09-10T13:43:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:29.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:29.270939+0000) 2022-09-10T13:43:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:29 smithi104 ceph-mon[97044]: pgmap v17734: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:29 smithi110 ceph-mon[97633]: pgmap v17734: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:30.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:29.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:29.885048+0000) 2022-09-10T13:43:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:30.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:30.271206+0000) 2022-09-10T13:43:31.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:30.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:30.885308+0000) 2022-09-10T13:43:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:31.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:31.271384+0000) 2022-09-10T13:43:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:31 smithi104 ceph-mon[97044]: pgmap v17735: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:31 smithi110 ceph-mon[97633]: pgmap v17735: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:32.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:31.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:31.885558+0000) 2022-09-10T13:43:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:32.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:32.271564+0000) 2022-09-10T13:43:33.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:32.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:32.885700+0000) 2022-09-10T13:43:33.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:43:33.082+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:43:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:33.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:33.271786+0000) 2022-09-10T13:43:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:33 smithi104 ceph-mon[97044]: pgmap v17736: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:33 smithi110 ceph-mon[97633]: pgmap v17736: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:34.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:33.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:33.885883+0000) 2022-09-10T13:43:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:34.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:34.272037+0000) 2022-09-10T13:43:35.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:34.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:34.886031+0000) 2022-09-10T13:43:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:35.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:35.272175+0000) 2022-09-10T13:43:35.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:35 smithi104 ceph-mon[97044]: pgmap v17737: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:43:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:35 smithi110 ceph-mon[97633]: pgmap v17737: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T13:43:36.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:35.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:35.886277+0000) 2022-09-10T13:43:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:36.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:36.272398+0000) 2022-09-10T13:43:37.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:36.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:36.886508+0000) 2022-09-10T13:43:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:36 smithi104 ceph-mon[97044]: pgmap v17738: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:43:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:36 smithi110 ceph-mon[97633]: pgmap v17738: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:43:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:37.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:37.272621+0000) 2022-09-10T13:43:38.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:37.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:37.886694+0000) 2022-09-10T13:43:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:38.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:38.272857+0000) 2022-09-10T13:43:39.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:38.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:38.886851+0000) 2022-09-10T13:43:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:39.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:39.273137+0000) 2022-09-10T13:43:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:39 smithi104 ceph-mon[97044]: pgmap v17739: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:39 smithi110 ceph-mon[97633]: pgmap v17739: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:40.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:39.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:39.887131+0000) 2022-09-10T13:43:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:40.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:40.273367+0000) 2022-09-10T13:43:41.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:40.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:40.887366+0000) 2022-09-10T13:43:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:41.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:41.273609+0000) 2022-09-10T13:43:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:41 smithi104 ceph-mon[97044]: pgmap v17740: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:41 smithi110 ceph-mon[97633]: pgmap v17740: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:42.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:41.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:41.887564+0000) 2022-09-10T13:43:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:42.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:42.273775+0000) 2022-09-10T13:43:43.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:42.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:42.887782+0000) 2022-09-10T13:43:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:43.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:43.273997+0000) 2022-09-10T13:43:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:43 smithi104 ceph-mon[97044]: pgmap v17741: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:43 smithi110 ceph-mon[97633]: pgmap v17741: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:44.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:43.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:43.888002+0000) 2022-09-10T13:43:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:44.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:44.274122+0000) 2022-09-10T13:43:45.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:44.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:44.888232+0000) 2022-09-10T13:43:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:45.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:45.274323+0000) 2022-09-10T13:43:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:45 smithi104 ceph-mon[97044]: pgmap v17742: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:45 smithi110 ceph-mon[97633]: pgmap v17742: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:46.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:45.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:45.888442+0000) 2022-09-10T13:43:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:46.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:46.274465+0000) 2022-09-10T13:43:47.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:46.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:46.888642+0000) 2022-09-10T13:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:43:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:43:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:43:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:46 smithi104 ceph-mon[97044]: pgmap v17743: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:43:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:43:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:43:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:43:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:43:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:46 smithi110 ceph-mon[97633]: pgmap v17743: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T13:43:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:47.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:47.274685+0000) 2022-09-10T13:43:48.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:47.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:47.888830+0000) 2022-09-10T13:43:48.276 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:43:48.090+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:43:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:48.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:48.274856+0000) 2022-09-10T13:43:49.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:48.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:48.889015+0000) 2022-09-10T13:43:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:49.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:49.275115+0000) 2022-09-10T13:43:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:49 smithi104 ceph-mon[97044]: pgmap v17744: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:49 smithi110 ceph-mon[97633]: pgmap v17744: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:43:50.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:49.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:49.889223+0000) 2022-09-10T13:43:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:50.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:50.275311+0000) 2022-09-10T13:43:51.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:50.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:50.889421+0000) 2022-09-10T13:43:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:51.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:51.275454+0000) 2022-09-10T13:43:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:51 smithi104 ceph-mon[97044]: pgmap v17745: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:51 smithi110 ceph-mon[97633]: pgmap v17745: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:52.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:51.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:51.889583+0000) 2022-09-10T13:43:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:52.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:52.275694+0000) 2022-09-10T13:43:53.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:52.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:52.889843+0000) 2022-09-10T13:43:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:53.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:53.275948+0000) 2022-09-10T13:43:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:53 smithi104 ceph-mon[97044]: pgmap v17746: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:53 smithi110 ceph-mon[97633]: pgmap v17746: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:54.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:53.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:53.890050+0000) 2022-09-10T13:43:54.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:54.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:54.276214+0000) 2022-09-10T13:43:55.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:54.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:54.890271+0000) 2022-09-10T13:43:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:55.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:55.276387+0000) 2022-09-10T13:43:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:55 smithi104 ceph-mon[97044]: pgmap v17747: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:55 smithi110 ceph-mon[97633]: pgmap v17747: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:43:56.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:55.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:55.890457+0000) 2022-09-10T13:43:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:56.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:56.276617+0000) 2022-09-10T13:43:57.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:56.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:56.890613+0000) 2022-09-10T13:43:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:56 smithi104 ceph-mon[97044]: pgmap v17748: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:56 smithi110 ceph-mon[97633]: pgmap v17748: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:57.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:57.276840+0000) 2022-09-10T13:43:58.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:57.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:57.890822+0000) 2022-09-10T13:43:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:58.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:58.277086+0000) 2022-09-10T13:43:59.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:58.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:58.891060+0000) 2022-09-10T13:43:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:43:59.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:59.277238+0000) 2022-09-10T13:43:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:43:59 smithi104 ceph-mon[97044]: pgmap v17749: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:43:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:43:59 smithi110 ceph-mon[97633]: pgmap v17749: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:44:00.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:43:59.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:43:59.891227+0000) 2022-09-10T13:44:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:00.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:00.277446+0000) 2022-09-10T13:44:01.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:00.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:00.891528+0000) 2022-09-10T13:44:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:01.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:01.277668+0000) 2022-09-10T13:44:01.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:01 smithi104 ceph-mon[97044]: pgmap v17750: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:01 smithi110 ceph-mon[97633]: pgmap v17750: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:02.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:01.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:01.891797+0000) 2022-09-10T13:44:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:02.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:02.277899+0000) 2022-09-10T13:44:03.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:02.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:02.892079+0000) 2022-09-10T13:44:03.279 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:44:03.104+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:44:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:03.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:03.278179+0000) 2022-09-10T13:44:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:03 smithi104 ceph-mon[97044]: pgmap v17751: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:03.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:03 smithi110 ceph-mon[97633]: pgmap v17751: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:04.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:03.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:03.892367+0000) 2022-09-10T13:44:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:04.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:04.278314+0000) 2022-09-10T13:44:05.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:04.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:04.892627+0000) 2022-09-10T13:44:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:05.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:05.278557+0000) 2022-09-10T13:44:05.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:05 smithi104 ceph-mon[97044]: pgmap v17752: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:44:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:05 smithi110 ceph-mon[97633]: pgmap v17752: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:44:06.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:05.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:05.892832+0000) 2022-09-10T13:44:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:06.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:06.278780+0000) 2022-09-10T13:44:07.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:06.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:06.893021+0000) 2022-09-10T13:44:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:06 smithi104 ceph-mon[97044]: pgmap v17753: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:44:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:06 smithi110 ceph-mon[97633]: pgmap v17753: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:44:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:07.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:07.279005+0000) 2022-09-10T13:44:08.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:07.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:07.893240+0000) 2022-09-10T13:44:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:08.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:08.279195+0000) 2022-09-10T13:44:09.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:08.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:08.893420+0000) 2022-09-10T13:44:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:08 smithi104 ceph-mon[97044]: osdmap e1190: 12 total, 12 up, 12 in 2022-09-10T13:44:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:08 smithi104 ceph-mon[97044]: pgmap v17755: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:44:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:08 smithi110 ceph-mon[97633]: osdmap e1190: 12 total, 12 up, 12 in 2022-09-10T13:44:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:08 smithi110 ceph-mon[97633]: pgmap v17755: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:44:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:09.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:09.279423+0000) 2022-09-10T13:44:10.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:09.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:09.893638+0000) 2022-09-10T13:44:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:10.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:10.279658+0000) 2022-09-10T13:44:11.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:10.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:10.893868+0000) 2022-09-10T13:44:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:11.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:11.279925+0000) 2022-09-10T13:44:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:11 smithi104 ceph-mon[97044]: pgmap v17756: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:44:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:11 smithi104 ceph-mon[97044]: osdmap e1191: 12 total, 12 up, 12 in 2022-09-10T13:44:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:11 smithi110 ceph-mon[97633]: pgmap v17756: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:44:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:11 smithi110 ceph-mon[97633]: osdmap e1191: 12 total, 12 up, 12 in 2022-09-10T13:44:12.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:11.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:11.894043+0000) 2022-09-10T13:44:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:12.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:12.280215+0000) 2022-09-10T13:44:13.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:12.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:12.894287+0000) 2022-09-10T13:44:13.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:13.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:13.280424+0000) 2022-09-10T13:44:13.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:13 smithi104 ceph-mon[97044]: pgmap v17758: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:44:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:13 smithi110 ceph-mon[97633]: pgmap v17758: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:44:14.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:13.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:13.894447+0000) 2022-09-10T13:44:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:14.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:14.280605+0000) 2022-09-10T13:44:15.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:14.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:14.894639+0000) 2022-09-10T13:44:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:15.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:15.280837+0000) 2022-09-10T13:44:15.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:15 smithi104 ceph-mon[97044]: pgmap v17759: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:44:15.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:15 smithi110 ceph-mon[97633]: pgmap v17759: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:44:16.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:15.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:15.894923+0000) 2022-09-10T13:44:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:16.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:16.281077+0000) 2022-09-10T13:44:17.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:16.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:16.895211+0000) 2022-09-10T13:44:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:16 smithi104 ceph-mon[97044]: pgmap v17760: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:44:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:16 smithi110 ceph-mon[97633]: pgmap v17760: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:44:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:17.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:17.281383+0000) 2022-09-10T13:44:18.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:44:18.110+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:44:18.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:17.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:17.895439+0000) 2022-09-10T13:44:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:18.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:18.281588+0000) 2022-09-10T13:44:19.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:18.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:18.895612+0000) 2022-09-10T13:44:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:19.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:19.281740+0000) 2022-09-10T13:44:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:19 smithi104 ceph-mon[97044]: pgmap v17761: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:44:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:19 smithi110 ceph-mon[97633]: pgmap v17761: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:44:20.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:19.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:19.895875+0000) 2022-09-10T13:44:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:20.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:20.281934+0000) 2022-09-10T13:44:21.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:20.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:20.896171+0000) 2022-09-10T13:44:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:21.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:21.282183+0000) 2022-09-10T13:44:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:21 smithi104 ceph-mon[97044]: pgmap v17762: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:44:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:21 smithi110 ceph-mon[97633]: pgmap v17762: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:44:22.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:21.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:21.896370+0000) 2022-09-10T13:44:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:22.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:22.282396+0000) 2022-09-10T13:44:23.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:22.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:22.896653+0000) 2022-09-10T13:44:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:23.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:23.282624+0000) 2022-09-10T13:44:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:23 smithi104 ceph-mon[97044]: pgmap v17763: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:44:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:23 smithi110 ceph-mon[97633]: pgmap v17763: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:44:24.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:23.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:23.896875+0000) 2022-09-10T13:44:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:24.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:24.282854+0000) 2022-09-10T13:44:25.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:24.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:24.897180+0000) 2022-09-10T13:44:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:25.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:25.283145+0000) 2022-09-10T13:44:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:25 smithi104 ceph-mon[97044]: pgmap v17764: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:44:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:25 smithi110 ceph-mon[97633]: pgmap v17764: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:44:26.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:25.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:25.897396+0000) 2022-09-10T13:44:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:26.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:26.283290+0000) 2022-09-10T13:44:27.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:26.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:26.897626+0000) 2022-09-10T13:44:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:26 smithi104 ceph-mon[97044]: pgmap v17765: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:26 smithi110 ceph-mon[97633]: pgmap v17765: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:27.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:27.283423+0000) 2022-09-10T13:44:28.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:27.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:27.897862+0000) 2022-09-10T13:44:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:28.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:28.283638+0000) 2022-09-10T13:44:29.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:28.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:28.898053+0000) 2022-09-10T13:44:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:29.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:29.283844+0000) 2022-09-10T13:44:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:29 smithi104 ceph-mon[97044]: pgmap v17766: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:29 smithi110 ceph-mon[97633]: pgmap v17766: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:30.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:29.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:29.898286+0000) 2022-09-10T13:44:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:30.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:30.284173+0000) 2022-09-10T13:44:31.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:30.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:30.898494+0000) 2022-09-10T13:44:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:31.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:31.284437+0000) 2022-09-10T13:44:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:31 smithi104 ceph-mon[97044]: pgmap v17767: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:31 smithi110 ceph-mon[97633]: pgmap v17767: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:32.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:31.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:31.898715+0000) 2022-09-10T13:44:32.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:32.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:32.284591+0000) 2022-09-10T13:44:33.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:32.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:32.898899+0000) 2022-09-10T13:44:33.286 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:44:33.126+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:44:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:33.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:33.284856+0000) 2022-09-10T13:44:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:33 smithi104 ceph-mon[97044]: pgmap v17768: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:33 smithi110 ceph-mon[97633]: pgmap v17768: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:34.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:33.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:33.899143+0000) 2022-09-10T13:44:34.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:34.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:34.285129+0000) 2022-09-10T13:44:35.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:34.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:34.899347+0000) 2022-09-10T13:44:35.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:35.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:35.285328+0000) 2022-09-10T13:44:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:35 smithi104 ceph-mon[97044]: pgmap v17769: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:35 smithi110 ceph-mon[97633]: pgmap v17769: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:36.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:35.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:35.899513+0000) 2022-09-10T13:44:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:36.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:36.285578+0000) 2022-09-10T13:44:37.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:36.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:36.899761+0000) 2022-09-10T13:44:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:36 smithi104 ceph-mon[97044]: pgmap v17770: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:36 smithi110 ceph-mon[97633]: pgmap v17770: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:37.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:37.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:37.285816+0000) 2022-09-10T13:44:38.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:37.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:37.900009+0000) 2022-09-10T13:44:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:38.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:38.286055+0000) 2022-09-10T13:44:39.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:38.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:38.900195+0000) 2022-09-10T13:44:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:39.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:39.286217+0000) 2022-09-10T13:44:39.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:39 smithi104 ceph-mon[97044]: pgmap v17771: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:39 smithi110 ceph-mon[97633]: pgmap v17771: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:40.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:39.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:39.900334+0000) 2022-09-10T13:44:40.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:40.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:40.286429+0000) 2022-09-10T13:44:41.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:40.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:40.900561+0000) 2022-09-10T13:44:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:41.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:41.286705+0000) 2022-09-10T13:44:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:41 smithi104 ceph-mon[97044]: pgmap v17772: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:41 smithi110 ceph-mon[97633]: pgmap v17772: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:42.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:41.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:41.900739+0000) 2022-09-10T13:44:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:42.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:42.286923+0000) 2022-09-10T13:44:43.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:42.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:42.900957+0000) 2022-09-10T13:44:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:43.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:43.287174+0000) 2022-09-10T13:44:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:43 smithi104 ceph-mon[97044]: pgmap v17773: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:43 smithi110 ceph-mon[97633]: pgmap v17773: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:44.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:43.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:43.901247+0000) 2022-09-10T13:44:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:44.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:44.287378+0000) 2022-09-10T13:44:45.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:44.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:44.901402+0000) 2022-09-10T13:44:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:45.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:45.287648+0000) 2022-09-10T13:44:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:45 smithi104 ceph-mon[97044]: pgmap v17774: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:45 smithi110 ceph-mon[97633]: pgmap v17774: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:46.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:45.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:45.901545+0000) 2022-09-10T13:44:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:46.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:46.287890+0000) 2022-09-10T13:44:47.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:46.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:46.901798+0000) 2022-09-10T13:44:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:44:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:46 smithi104 ceph-mon[97044]: pgmap v17775: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:44:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:46 smithi110 ceph-mon[97633]: pgmap v17775: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:47.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:47.288194+0000) 2022-09-10T13:44:48.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:47.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:47.902046+0000) 2022-09-10T13:44:48.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:44:48.136+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:44:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:48.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:48.288339+0000) 2022-09-10T13:44:49.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:48.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:48.902323+0000) 2022-09-10T13:44:49.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:49.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:49.288558+0000) 2022-09-10T13:44:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:49 smithi104 ceph-mon[97044]: pgmap v17776: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:49 smithi110 ceph-mon[97633]: pgmap v17776: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:50.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:49.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:49.902487+0000) 2022-09-10T13:44:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:50.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:50.288789+0000) 2022-09-10T13:44:51.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:50.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:50.902668+0000) 2022-09-10T13:44:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:51.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:51.289028+0000) 2022-09-10T13:44:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:51 smithi104 ceph-mon[97044]: pgmap v17777: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:51 smithi110 ceph-mon[97633]: pgmap v17777: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:52.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:51.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:51.902916+0000) 2022-09-10T13:44:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:52.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:52.289257+0000) 2022-09-10T13:44:53.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:52.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:52.903167+0000) 2022-09-10T13:44:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:53.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:53.289439+0000) 2022-09-10T13:44:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:53 smithi104 ceph-mon[97044]: pgmap v17778: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:53 smithi110 ceph-mon[97633]: pgmap v17778: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:54.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:53.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:53.903355+0000) 2022-09-10T13:44:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:54.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:54.289645+0000) 2022-09-10T13:44:55.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:54.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:54.903520+0000) 2022-09-10T13:44:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:55.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:55.289804+0000) 2022-09-10T13:44:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:55 smithi104 ceph-mon[97044]: pgmap v17779: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:55 smithi110 ceph-mon[97633]: pgmap v17779: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:44:56.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:55.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:55.903738+0000) 2022-09-10T13:44:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:56.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:56.290038+0000) 2022-09-10T13:44:57.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:56.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:56.903979+0000) 2022-09-10T13:44:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:56 smithi104 ceph-mon[97044]: pgmap v17780: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:56 smithi110 ceph-mon[97633]: pgmap v17780: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:57.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:57.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:57.290280+0000) 2022-09-10T13:44:58.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:57.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:57.904242+0000) 2022-09-10T13:44:58.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:58.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:58.290484+0000) 2022-09-10T13:44:59.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:58.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:58.904436+0000) 2022-09-10T13:44:59.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:44:59.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:59.290742+0000) 2022-09-10T13:44:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:44:59 smithi104 ceph-mon[97044]: pgmap v17781: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:44:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:44:59 smithi110 ceph-mon[97633]: pgmap v17781: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:00.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:44:59.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:44:59.904652+0000) 2022-09-10T13:45:00.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:00.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:00.290965+0000) 2022-09-10T13:45:01.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:00.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:00.904893+0000) 2022-09-10T13:45:01.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:01.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:01.291209+0000) 2022-09-10T13:45:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:01 smithi104 ceph-mon[97044]: pgmap v17782: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:01 smithi110 ceph-mon[97633]: pgmap v17782: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:02.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:01.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:01.905194+0000) 2022-09-10T13:45:02.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:02.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:02.291353+0000) 2022-09-10T13:45:03.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:02.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:02.905389+0000) 2022-09-10T13:45:03.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:45:03.147+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:45:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:03.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:03.291525+0000) 2022-09-10T13:45:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:03 smithi104 ceph-mon[97044]: pgmap v17783: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:03 smithi110 ceph-mon[97633]: pgmap v17783: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:04.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:03.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:03.905558+0000) 2022-09-10T13:45:04.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:04.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:04.291743+0000) 2022-09-10T13:45:05.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:04.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:04.905744+0000) 2022-09-10T13:45:05.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:05.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:05.291925+0000) 2022-09-10T13:45:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:05 smithi104 ceph-mon[97044]: pgmap v17784: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:45:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:05 smithi110 ceph-mon[97633]: pgmap v17784: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:45:06.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:05.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:05.905982+0000) 2022-09-10T13:45:06.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:06.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:06.292189+0000) 2022-09-10T13:45:07.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:06.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:06.906245+0000) 2022-09-10T13:45:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:06 smithi104 ceph-mon[97044]: pgmap v17785: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:45:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:06 smithi110 ceph-mon[97633]: pgmap v17785: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:45:07.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:07.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:07.292418+0000) 2022-09-10T13:45:08.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:07.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:07.906425+0000) 2022-09-10T13:45:08.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:08.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:08.292693+0000) 2022-09-10T13:45:09.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:08.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:08.906647+0000) 2022-09-10T13:45:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:08 smithi104 ceph-mon[97044]: osdmap e1192: 12 total, 12 up, 12 in 2022-09-10T13:45:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:08 smithi104 ceph-mon[97044]: pgmap v17787: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:45:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:08 smithi110 ceph-mon[97633]: osdmap e1192: 12 total, 12 up, 12 in 2022-09-10T13:45:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:08 smithi110 ceph-mon[97633]: pgmap v17787: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:45:09.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:09.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:09.292889+0000) 2022-09-10T13:45:10.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:09.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:09.906816+0000) 2022-09-10T13:45:10.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:10.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:10.293174+0000) 2022-09-10T13:45:11.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:10.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:10.906972+0000) 2022-09-10T13:45:11.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:11.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:11.293355+0000) 2022-09-10T13:45:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:11 smithi104 ceph-mon[97044]: pgmap v17788: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:45:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:11 smithi104 ceph-mon[97044]: osdmap e1193: 12 total, 12 up, 12 in 2022-09-10T13:45:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:11 smithi110 ceph-mon[97633]: pgmap v17788: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:45:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:11 smithi110 ceph-mon[97633]: osdmap e1193: 12 total, 12 up, 12 in 2022-09-10T13:45:12.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:11.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:11.907168+0000) 2022-09-10T13:45:12.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:12.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:12.293580+0000) 2022-09-10T13:45:13.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:12.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:12.907340+0000) 2022-09-10T13:45:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:13.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:13.293801+0000) 2022-09-10T13:45:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:13 smithi104 ceph-mon[97044]: pgmap v17790: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:45:13.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:13 smithi110 ceph-mon[97633]: pgmap v17790: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:45:14.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:13.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:13.907550+0000) 2022-09-10T13:45:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:14.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:14.294006+0000) 2022-09-10T13:45:15.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:14.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:14.907796+0000) 2022-09-10T13:45:15.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:15.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:15.294262+0000) 2022-09-10T13:45:15.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:15 smithi104 ceph-mon[97044]: pgmap v17791: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:45:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:15 smithi110 ceph-mon[97633]: pgmap v17791: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:45:16.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:15.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:15.908014+0000) 2022-09-10T13:45:16.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:16.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:16.294392+0000) 2022-09-10T13:45:17.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:16.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:16.908276+0000) 2022-09-10T13:45:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:16 smithi104 ceph-mon[97044]: pgmap v17792: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:45:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:16 smithi110 ceph-mon[97633]: pgmap v17792: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:45:17.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:17.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:17.294624+0000) 2022-09-10T13:45:18.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:17.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:17.908501+0000) 2022-09-10T13:45:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:18.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:18.294855+0000) 2022-09-10T13:45:18.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:45:18.160+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:45:19.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:18.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:18.908719+0000) 2022-09-10T13:45:19.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:19.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:19.295060+0000) 2022-09-10T13:45:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:19 smithi104 ceph-mon[97044]: pgmap v17793: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:45:19.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:19 smithi110 ceph-mon[97633]: pgmap v17793: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:45:20.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:19.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:19.908865+0000) 2022-09-10T13:45:20.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:20.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:20.295326+0000) 2022-09-10T13:45:21.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:20.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:20.909011+0000) 2022-09-10T13:45:21.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:21.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:21.295491+0000) 2022-09-10T13:45:21.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:21 smithi104 ceph-mon[97044]: pgmap v17794: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:45:21.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:21 smithi110 ceph-mon[97633]: pgmap v17794: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:45:22.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:21.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:21.909193+0000) 2022-09-10T13:45:22.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:22.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:22.295700+0000) 2022-09-10T13:45:23.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:22.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:22.909432+0000) 2022-09-10T13:45:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:23.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:23.295907+0000) 2022-09-10T13:45:23.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:23 smithi104 ceph-mon[97044]: pgmap v17795: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:45:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:23 smithi110 ceph-mon[97633]: pgmap v17795: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:45:24.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:23.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:23.909625+0000) 2022-09-10T13:45:24.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:24.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:24.296200+0000) 2022-09-10T13:45:25.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:24.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:24.909899+0000) 2022-09-10T13:45:25.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:25.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:25.296460+0000) 2022-09-10T13:45:25.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:25 smithi104 ceph-mon[97044]: pgmap v17796: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:45:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:25 smithi110 ceph-mon[97633]: pgmap v17796: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:45:26.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:25.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:25.910117+0000) 2022-09-10T13:45:26.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:26.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:26.296736+0000) 2022-09-10T13:45:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:26 smithi104 ceph-mon[97044]: pgmap v17797: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:27.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:26.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:26.910388+0000) 2022-09-10T13:45:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:26 smithi110 ceph-mon[97633]: pgmap v17797: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:27.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:27.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:27.297010+0000) 2022-09-10T13:45:28.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:27.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:27.910611+0000) 2022-09-10T13:45:28.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:28.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:28.297258+0000) 2022-09-10T13:45:29.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:28.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:28.910825+0000) 2022-09-10T13:45:29.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:29.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:29.297525+0000) 2022-09-10T13:45:29.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:29 smithi104 ceph-mon[97044]: pgmap v17798: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:29 smithi110 ceph-mon[97633]: pgmap v17798: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:30.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:29.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:29.911039+0000) 2022-09-10T13:45:30.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:30.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:30.297762+0000) 2022-09-10T13:45:31.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:30.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:30.911233+0000) 2022-09-10T13:45:31.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:31.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:31.298008+0000) 2022-09-10T13:45:31.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:31 smithi104 ceph-mon[97044]: pgmap v17799: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:31.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:31 smithi110 ceph-mon[97633]: pgmap v17799: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:32.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:31.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:31.911420+0000) 2022-09-10T13:45:32.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:32.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:32.298259+0000) 2022-09-10T13:45:33.176 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:32.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:32.911664+0000) 2022-09-10T13:45:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:33.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:33.298429+0000) 2022-09-10T13:45:33.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:45:33.175+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:45:33.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:33 smithi104 ceph-mon[97044]: pgmap v17800: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:33 smithi110 ceph-mon[97633]: pgmap v17800: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:34.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:33.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:33.911912+0000) 2022-09-10T13:45:34.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:34.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:34.298620+0000) 2022-09-10T13:45:35.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:34.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:34.912200+0000) 2022-09-10T13:45:35.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:35.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:35.298892+0000) 2022-09-10T13:45:35.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:35 smithi104 ceph-mon[97044]: pgmap v17801: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:35 smithi110 ceph-mon[97633]: pgmap v17801: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:36.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:35.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:35.912478+0000) 2022-09-10T13:45:36.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:36.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:36.299109+0000) 2022-09-10T13:45:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:36 smithi104 ceph-mon[97044]: pgmap v17802: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:37.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:36.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:36.912712+0000) 2022-09-10T13:45:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:36 smithi110 ceph-mon[97633]: pgmap v17802: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:37.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:37.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:37.299307+0000) 2022-09-10T13:45:38.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:37.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:37.912922+0000) 2022-09-10T13:45:38.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:38.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:38.299548+0000) 2022-09-10T13:45:39.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:38.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:38.913169+0000) 2022-09-10T13:45:39.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:39.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:39.299707+0000) 2022-09-10T13:45:39.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:39 smithi104 ceph-mon[97044]: pgmap v17803: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:39.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:39 smithi110 ceph-mon[97633]: pgmap v17803: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:40.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:39.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:39.913367+0000) 2022-09-10T13:45:40.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:40.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:40.299888+0000) 2022-09-10T13:45:41.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:40.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:40.913629+0000) 2022-09-10T13:45:41.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:41.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:41.300091+0000) 2022-09-10T13:45:41.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:41 smithi104 ceph-mon[97044]: pgmap v17804: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:41 smithi110 ceph-mon[97633]: pgmap v17804: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:42.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:41.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:41.913857+0000) 2022-09-10T13:45:42.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:42.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:42.300268+0000) 2022-09-10T13:45:43.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:42.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:42.914066+0000) 2022-09-10T13:45:43.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:43.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:43.300399+0000) 2022-09-10T13:45:43.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:43 smithi104 ceph-mon[97044]: pgmap v17805: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:43 smithi110 ceph-mon[97633]: pgmap v17805: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:44.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:43.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:43.914282+0000) 2022-09-10T13:45:44.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:44.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:44.300526+0000) 2022-09-10T13:45:45.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:44.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:44.914514+0000) 2022-09-10T13:45:45.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:45.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:45.300719+0000) 2022-09-10T13:45:45.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:45 smithi104 ceph-mon[97044]: pgmap v17806: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:45 smithi110 ceph-mon[97633]: pgmap v17806: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:46.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:45.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:45.914744+0000) 2022-09-10T13:45:46.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:46.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:46.300910+0000) 2022-09-10T13:45:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:45:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:45:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:46 smithi104 ceph-mon[97044]: pgmap v17807: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:47.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:46.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:46.914924+0000) 2022-09-10T13:45:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:45:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:45:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:45:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:45:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:46 smithi110 ceph-mon[97633]: pgmap v17807: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:47.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:47.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:47.301184+0000) 2022-09-10T13:45:48.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:47.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:47.915166+0000) 2022-09-10T13:45:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:48.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:48.301445+0000) 2022-09-10T13:45:48.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:45:48.191+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:45:49.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:48.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:48.915406+0000) 2022-09-10T13:45:49.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:49.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:49.301656+0000) 2022-09-10T13:45:49.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:49 smithi104 ceph-mon[97044]: pgmap v17808: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:49 smithi110 ceph-mon[97633]: pgmap v17808: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:50.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:49.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:49.915673+0000) 2022-09-10T13:45:50.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:50.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:50.301891+0000) 2022-09-10T13:45:51.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:50.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:50.915863+0000) 2022-09-10T13:45:51.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:51.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:51.302182+0000) 2022-09-10T13:45:51.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:51 smithi104 ceph-mon[97044]: pgmap v17809: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:51 smithi110 ceph-mon[97633]: pgmap v17809: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:52.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:51.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:51.916080+0000) 2022-09-10T13:45:52.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:52.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:52.302378+0000) 2022-09-10T13:45:53.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:52.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:52.916290+0000) 2022-09-10T13:45:53.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:53.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:53.302570+0000) 2022-09-10T13:45:53.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:53 smithi104 ceph-mon[97044]: pgmap v17810: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:53 smithi110 ceph-mon[97633]: pgmap v17810: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:54.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:53.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:53.916489+0000) 2022-09-10T13:45:54.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:54.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:54.302820+0000) 2022-09-10T13:45:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:54.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:54.916672+0000) 2022-09-10T13:45:55.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:55.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:55.302955+0000) 2022-09-10T13:45:55.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:55 smithi104 ceph-mon[97044]: pgmap v17811: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:55.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:55 smithi110 ceph-mon[97633]: pgmap v17811: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:45:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:55.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:55.916924+0000) 2022-09-10T13:45:56.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:56.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:56.303081+0000) 2022-09-10T13:45:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:56 smithi104 ceph-mon[97044]: pgmap v17812: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:56.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:56.917182+0000) 2022-09-10T13:45:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:56 smithi110 ceph-mon[97633]: pgmap v17812: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:57.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:57.303358+0000) 2022-09-10T13:45:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:57.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:57.917427+0000) 2022-09-10T13:45:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:58.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:58.303609+0000) 2022-09-10T13:45:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:58.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:58.917603+0000) 2022-09-10T13:45:59.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:45:59 smithi104 ceph-mon[97044]: pgmap v17813: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:45:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:45:59.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:59.303863+0000) 2022-09-10T13:45:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:45:59 smithi110 ceph-mon[97633]: pgmap v17813: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:45:59.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:45:59.917841+0000) 2022-09-10T13:46:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:00.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:00.304024+0000) 2022-09-10T13:46:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:00.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:00.918124+0000) 2022-09-10T13:46:01.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:01 smithi104 ceph-mon[97044]: pgmap v17814: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:01.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:01.304149+0000) 2022-09-10T13:46:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:01 smithi110 ceph-mon[97633]: pgmap v17814: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:01.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:01.918378+0000) 2022-09-10T13:46:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:02.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:02.304278+0000) 2022-09-10T13:46:03.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:02.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:02.918579+0000) 2022-09-10T13:46:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:03.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:03.304531+0000) 2022-09-10T13:46:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:46:03.198+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:46:03.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:03 smithi104 ceph-mon[97044]: pgmap v17815: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:03 smithi110 ceph-mon[97633]: pgmap v17815: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:03.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:03.918747+0000) 2022-09-10T13:46:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:04.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:04.304701+0000) 2022-09-10T13:46:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:04.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:04.918964+0000) 2022-09-10T13:46:05.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:05 smithi104 ceph-mon[97044]: pgmap v17816: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:46:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:05.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:05.304906+0000) 2022-09-10T13:46:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:05 smithi110 ceph-mon[97633]: pgmap v17816: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:46:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:05.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:05.919238+0000) 2022-09-10T13:46:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:06.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:06.305161+0000) 2022-09-10T13:46:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:06 smithi104 ceph-mon[97044]: pgmap v17817: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:46:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:06.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:06.919472+0000) 2022-09-10T13:46:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:06 smithi110 ceph-mon[97633]: pgmap v17817: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:46:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:07.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:07.305334+0000) 2022-09-10T13:46:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:07.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:07.919706+0000) 2022-09-10T13:46:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:08.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:08.305466+0000) 2022-09-10T13:46:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:08 smithi104 ceph-mon[97044]: osdmap e1194: 12 total, 12 up, 12 in 2022-09-10T13:46:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:08 smithi104 ceph-mon[97044]: pgmap v17819: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:46:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:08.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:08.919980+0000) 2022-09-10T13:46:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:08 smithi110 ceph-mon[97633]: osdmap e1194: 12 total, 12 up, 12 in 2022-09-10T13:46:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:08 smithi110 ceph-mon[97633]: pgmap v17819: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:46:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:09.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:09.305595+0000) 2022-09-10T13:46:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:09.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:09.920275+0000) 2022-09-10T13:46:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:10.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:10.305829+0000) 2022-09-10T13:46:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:10.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:10.920442+0000) 2022-09-10T13:46:11.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:11 smithi104 ceph-mon[97044]: pgmap v17820: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:46:11.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:11 smithi104 ceph-mon[97044]: osdmap e1195: 12 total, 12 up, 12 in 2022-09-10T13:46:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:11.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:11.306052+0000) 2022-09-10T13:46:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:11 smithi110 ceph-mon[97633]: pgmap v17820: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:46:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:11 smithi110 ceph-mon[97633]: osdmap e1195: 12 total, 12 up, 12 in 2022-09-10T13:46:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:11.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:11.920632+0000) 2022-09-10T13:46:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:12.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:12.306236+0000) 2022-09-10T13:46:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:12.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:12.920830+0000) 2022-09-10T13:46:13.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:13 smithi104 ceph-mon[97044]: pgmap v17822: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:46:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:13.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:13.306377+0000) 2022-09-10T13:46:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:13 smithi110 ceph-mon[97633]: pgmap v17822: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:46:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:13.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:13.921190+0000) 2022-09-10T13:46:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:14.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:14.306518+0000) 2022-09-10T13:46:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:14.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:14.921315+0000) 2022-09-10T13:46:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:15.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:15.306718+0000) 2022-09-10T13:46:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:15 smithi110 ceph-mon[97633]: pgmap v17823: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:46:16.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:15 smithi104 ceph-mon[97044]: pgmap v17823: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:46:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:15.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:15.921482+0000) 2022-09-10T13:46:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:16.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:16.306937+0000) 2022-09-10T13:46:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:16 smithi104 ceph-mon[97044]: pgmap v17824: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:46:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:16.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:16.921692+0000) 2022-09-10T13:46:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:16 smithi110 ceph-mon[97633]: pgmap v17824: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:46:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:17.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:17.307154+0000) 2022-09-10T13:46:18.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:17.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:17.921895+0000) 2022-09-10T13:46:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:18.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:18.307335+0000) 2022-09-10T13:46:18.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:46:18.202+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:46:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:18.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:18.922043+0000) 2022-09-10T13:46:19.795 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:19 smithi104 ceph-mon[97044]: pgmap v17825: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:46:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:19.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:19.307532+0000) 2022-09-10T13:46:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:19 smithi110 ceph-mon[97633]: pgmap v17825: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:46:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:19.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:19.922202+0000) 2022-09-10T13:46:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:20.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:20.307737+0000) 2022-09-10T13:46:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:20.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:20.922374+0000) 2022-09-10T13:46:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:21.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:21.307959+0000) 2022-09-10T13:46:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:21 smithi110 ceph-mon[97633]: pgmap v17826: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:46:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:21 smithi104 ceph-mon[97044]: pgmap v17826: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:46:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:21.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:21.922618+0000) 2022-09-10T13:46:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:22.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:22.308174+0000) 2022-09-10T13:46:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:22.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:22.922869+0000) 2022-09-10T13:46:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:23.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:23.308327+0000) 2022-09-10T13:46:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:23 smithi110 ceph-mon[97633]: pgmap v17827: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:46:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:23 smithi104 ceph-mon[97044]: pgmap v17827: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:46:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:23.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:23.923149+0000) 2022-09-10T13:46:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:24.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:24.308496+0000) 2022-09-10T13:46:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:24.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:24.923339+0000) 2022-09-10T13:46:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:25.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:25.308631+0000) 2022-09-10T13:46:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:25 smithi110 ceph-mon[97633]: pgmap v17828: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:46:26.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:25 smithi104 ceph-mon[97044]: pgmap v17828: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:46:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:25.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:25.923505+0000) 2022-09-10T13:46:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:26.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:26.308795+0000) 2022-09-10T13:46:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:26 smithi104 ceph-mon[97044]: pgmap v17829: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:26.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:26.923752+0000) 2022-09-10T13:46:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:26 smithi110 ceph-mon[97633]: pgmap v17829: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:27.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:27.308975+0000) 2022-09-10T13:46:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:27.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:27.924016+0000) 2022-09-10T13:46:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:28.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:28.309186+0000) 2022-09-10T13:46:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:28.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:28.924247+0000) 2022-09-10T13:46:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:29.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:29.309317+0000) 2022-09-10T13:46:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:29 smithi110 ceph-mon[97633]: pgmap v17830: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:29 smithi104 ceph-mon[97044]: pgmap v17830: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:29.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:29.924519+0000) 2022-09-10T13:46:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:30.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:30.309481+0000) 2022-09-10T13:46:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:30.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:30.924770+0000) 2022-09-10T13:46:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:31.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:31.309620+0000) 2022-09-10T13:46:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:31 smithi110 ceph-mon[97633]: pgmap v17831: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:31 smithi104 ceph-mon[97044]: pgmap v17831: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:31.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:31.925015+0000) 2022-09-10T13:46:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:32.308+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:32.309798+0000) 2022-09-10T13:46:33.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:32.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:32.925262+0000) 2022-09-10T13:46:33.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:46:33.216+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:46:33.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:33.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:33.309997+0000) 2022-09-10T13:46:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:33 smithi110 ceph-mon[97633]: pgmap v17832: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:33 smithi104 ceph-mon[97044]: pgmap v17832: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:33.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:33.925469+0000) 2022-09-10T13:46:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:34.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:34.310207+0000) 2022-09-10T13:46:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:34.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:34.925646+0000) 2022-09-10T13:46:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:35.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:35.310371+0000) 2022-09-10T13:46:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:35 smithi110 ceph-mon[97633]: pgmap v17833: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:36.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:35 smithi104 ceph-mon[97044]: pgmap v17833: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:35.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:35.925859+0000) 2022-09-10T13:46:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:36.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:36.310547+0000) 2022-09-10T13:46:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:36 smithi104 ceph-mon[97044]: pgmap v17834: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:36.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:36.926131+0000) 2022-09-10T13:46:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:36 smithi110 ceph-mon[97633]: pgmap v17834: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:37.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:37.310679+0000) 2022-09-10T13:46:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:37.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:37.926392+0000) 2022-09-10T13:46:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:38.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:38.310840+0000) 2022-09-10T13:46:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:38.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:38.926615+0000) 2022-09-10T13:46:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:39.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:39.311015+0000) 2022-09-10T13:46:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:39 smithi110 ceph-mon[97633]: pgmap v17835: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:39 smithi104 ceph-mon[97044]: pgmap v17835: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:39.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:39.926815+0000) 2022-09-10T13:46:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:40.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:40.311207+0000) 2022-09-10T13:46:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:40.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:40.927043+0000) 2022-09-10T13:46:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:41.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:41.311415+0000) 2022-09-10T13:46:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:41 smithi110 ceph-mon[97633]: pgmap v17836: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:41 smithi104 ceph-mon[97044]: pgmap v17836: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:41.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:41.927261+0000) 2022-09-10T13:46:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:42.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:42.311608+0000) 2022-09-10T13:46:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:42.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:42.927450+0000) 2022-09-10T13:46:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:43.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:43.311764+0000) 2022-09-10T13:46:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:43 smithi110 ceph-mon[97633]: pgmap v17837: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:43 smithi104 ceph-mon[97044]: pgmap v17837: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:43.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:43.927635+0000) 2022-09-10T13:46:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:44.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:44.311943+0000) 2022-09-10T13:46:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:44.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:44.927854+0000) 2022-09-10T13:46:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:45.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:45.312167+0000) 2022-09-10T13:46:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:45 smithi110 ceph-mon[97633]: pgmap v17838: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:46.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:45 smithi104 ceph-mon[97044]: pgmap v17838: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:45.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:45.928050+0000) 2022-09-10T13:46:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:46.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:46.312333+0000) 2022-09-10T13:46:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:46:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:46:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:46:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:46:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:46 smithi104 ceph-mon[97044]: pgmap v17839: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:46.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:46.928198+0000) 2022-09-10T13:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:46:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:46 smithi110 ceph-mon[97633]: pgmap v17839: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:47.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:47.312533+0000) 2022-09-10T13:46:48.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:47.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:47.928326+0000) 2022-09-10T13:46:48.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:48.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:48.312715+0000) 2022-09-10T13:46:48.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:46:48.227+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:46:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:48.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:48.928489+0000) 2022-09-10T13:46:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:49.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:49.312903+0000) 2022-09-10T13:46:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:49 smithi110 ceph-mon[97633]: pgmap v17840: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:49 smithi104 ceph-mon[97044]: pgmap v17840: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:49.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:49.928703+0000) 2022-09-10T13:46:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:50.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:50.313152+0000) 2022-09-10T13:46:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:50.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:50.928881+0000) 2022-09-10T13:46:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:51.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:51.313353+0000) 2022-09-10T13:46:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:51 smithi110 ceph-mon[97633]: pgmap v17841: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:51 smithi104 ceph-mon[97044]: pgmap v17841: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:51.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:51.929189+0000) 2022-09-10T13:46:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:52.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:52.313556+0000) 2022-09-10T13:46:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:52.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:52.929430+0000) 2022-09-10T13:46:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:53.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:53.313781+0000) 2022-09-10T13:46:53.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:53 smithi110 ceph-mon[97633]: pgmap v17842: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:53 smithi104 ceph-mon[97044]: pgmap v17842: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:46:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:53.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:53.929599+0000) 2022-09-10T13:46:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:54.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:54.313995+0000) 2022-09-10T13:46:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:54.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:54.929827+0000) 2022-09-10T13:46:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:55.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:55.314287+0000) 2022-09-10T13:46:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:55 smithi110 ceph-mon[97633]: pgmap v17843: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:56.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:55 smithi104 ceph-mon[97044]: pgmap v17843: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:46:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:55.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:55.930065+0000) 2022-09-10T13:46:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:56.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:56.314494+0000) 2022-09-10T13:46:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:56 smithi104 ceph-mon[97044]: pgmap v17844: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:46:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:56.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:56.930293+0000) 2022-09-10T13:46:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:56 smithi110 ceph-mon[97633]: pgmap v17844: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:46:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:57.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:57.314739+0000) 2022-09-10T13:46:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:57.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:57.930462+0000) 2022-09-10T13:46:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:58.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:58.314968+0000) 2022-09-10T13:46:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:58.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:58.930613+0000) 2022-09-10T13:46:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:46:59.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:59.315135+0000) 2022-09-10T13:46:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:46:59 smithi110 ceph-mon[97633]: pgmap v17845: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:47:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:46:59 smithi104 ceph-mon[97044]: pgmap v17845: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:47:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:46:59.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:46:59.930830+0000) 2022-09-10T13:47:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:00.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:00.315405+0000) 2022-09-10T13:47:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:00.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:00.931037+0000) 2022-09-10T13:47:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:01.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:01.315669+0000) 2022-09-10T13:47:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:01 smithi110 ceph-mon[97633]: pgmap v17846: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:01 smithi104 ceph-mon[97044]: pgmap v17846: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:01.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:01.931289+0000) 2022-09-10T13:47:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:02.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:02.315896+0000) 2022-09-10T13:47:03.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:02.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:02.931533+0000) 2022-09-10T13:47:03.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:03.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:03.316141+0000) 2022-09-10T13:47:03.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:47:03.242+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:47:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:03 smithi110 ceph-mon[97633]: pgmap v17847: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:47:04.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:03 smithi104 ceph-mon[97044]: pgmap v17847: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:47:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:03.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:03.931731+0000) 2022-09-10T13:47:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:04.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:04.316301+0000) 2022-09-10T13:47:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:04.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:04.931929+0000) 2022-09-10T13:47:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:05.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:05.316483+0000) 2022-09-10T13:47:05.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:05 smithi110 ceph-mon[97633]: pgmap v17848: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:47:06.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:05 smithi104 ceph-mon[97044]: pgmap v17848: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:47:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:05.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:05.932199+0000) 2022-09-10T13:47:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:06.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:06.316671+0000) 2022-09-10T13:47:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:06 smithi104 ceph-mon[97044]: pgmap v17849: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:47:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:06.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:06.932437+0000) 2022-09-10T13:47:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:06 smithi110 ceph-mon[97633]: pgmap v17849: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:47:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:07.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:07.316928+0000) 2022-09-10T13:47:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:07.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:07.932689+0000) 2022-09-10T13:47:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:08.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:08.317220+0000) 2022-09-10T13:47:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:08 smithi104 ceph-mon[97044]: osdmap e1196: 12 total, 12 up, 12 in 2022-09-10T13:47:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:08 smithi104 ceph-mon[97044]: pgmap v17851: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:47:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:08.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:08.932857+0000) 2022-09-10T13:47:09.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:08 smithi110 ceph-mon[97633]: osdmap e1196: 12 total, 12 up, 12 in 2022-09-10T13:47:09.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:08 smithi110 ceph-mon[97633]: pgmap v17851: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:47:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:09.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:09.317435+0000) 2022-09-10T13:47:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:09.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:09.933055+0000) 2022-09-10T13:47:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:10.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:10.317707+0000) 2022-09-10T13:47:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:10.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:10.933269+0000) 2022-09-10T13:47:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:11.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:11.317872+0000) 2022-09-10T13:47:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:11 smithi110 ceph-mon[97633]: pgmap v17852: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:47:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:11 smithi110 ceph-mon[97633]: osdmap e1197: 12 total, 12 up, 12 in 2022-09-10T13:47:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:11 smithi104 ceph-mon[97044]: pgmap v17852: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:47:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:11 smithi104 ceph-mon[97044]: osdmap e1197: 12 total, 12 up, 12 in 2022-09-10T13:47:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:11.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:11.933505+0000) 2022-09-10T13:47:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:12.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:12.318052+0000) 2022-09-10T13:47:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:12.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:12.933714+0000) 2022-09-10T13:47:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:13.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:13.318344+0000) 2022-09-10T13:47:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:13 smithi110 ceph-mon[97633]: pgmap v17854: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:47:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:13 smithi104 ceph-mon[97044]: pgmap v17854: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:47:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:13.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:13.933883+0000) 2022-09-10T13:47:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:14.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:14.318534+0000) 2022-09-10T13:47:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:14.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:14.934057+0000) 2022-09-10T13:47:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:15.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:15.318766+0000) 2022-09-10T13:47:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:15 smithi110 ceph-mon[97633]: pgmap v17855: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:47:16.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:15 smithi104 ceph-mon[97044]: pgmap v17855: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:47:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:15.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:15.934343+0000) 2022-09-10T13:47:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:16.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:16.318958+0000) 2022-09-10T13:47:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:16 smithi104 ceph-mon[97044]: pgmap v17856: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:47:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:16.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:16.934612+0000) 2022-09-10T13:47:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:16 smithi110 ceph-mon[97633]: pgmap v17856: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:47:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:17.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:17.319220+0000) 2022-09-10T13:47:18.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:17.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:17.934760+0000) 2022-09-10T13:47:18.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:47:18.255+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:47:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:18.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:18.319472+0000) 2022-09-10T13:47:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:18.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:18.934949+0000) 2022-09-10T13:47:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:19.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:19.319742+0000) 2022-09-10T13:47:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:19 smithi110 ceph-mon[97633]: pgmap v17857: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:47:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:19 smithi104 ceph-mon[97044]: pgmap v17857: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:47:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:19.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:19.935179+0000) 2022-09-10T13:47:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:20.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:20.319970+0000) 2022-09-10T13:47:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:20.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:20.935428+0000) 2022-09-10T13:47:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:21.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:21.320196+0000) 2022-09-10T13:47:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:21 smithi110 ceph-mon[97633]: pgmap v17858: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:47:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:21 smithi104 ceph-mon[97044]: pgmap v17858: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:47:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:21.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:21.935672+0000) 2022-09-10T13:47:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:22.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:22.320412+0000) 2022-09-10T13:47:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:22.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:22.935853+0000) 2022-09-10T13:47:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:23.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:23.320633+0000) 2022-09-10T13:47:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:23 smithi110 ceph-mon[97633]: pgmap v17859: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:47:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:23 smithi104 ceph-mon[97044]: pgmap v17859: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:47:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:23.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:23.936066+0000) 2022-09-10T13:47:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:24.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:24.320800+0000) 2022-09-10T13:47:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:24.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:24.936262+0000) 2022-09-10T13:47:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:25.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:25.320961+0000) 2022-09-10T13:47:25.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:25 smithi110 ceph-mon[97633]: pgmap v17860: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:47:26.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:25 smithi104 ceph-mon[97044]: pgmap v17860: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:47:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:25.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:25.936471+0000) 2022-09-10T13:47:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:26.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:26.321211+0000) 2022-09-10T13:47:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:26 smithi104 ceph-mon[97044]: pgmap v17861: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:26.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:26.936675+0000) 2022-09-10T13:47:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:26 smithi110 ceph-mon[97633]: pgmap v17861: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:27.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:27.321475+0000) 2022-09-10T13:47:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:27.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:27.936911+0000) 2022-09-10T13:47:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:28.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:28.321733+0000) 2022-09-10T13:47:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:28.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:28.937156+0000) 2022-09-10T13:47:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:29.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:29.321988+0000) 2022-09-10T13:47:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:29 smithi110 ceph-mon[97633]: pgmap v17862: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:29 smithi104 ceph-mon[97044]: pgmap v17862: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:29.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:29.937358+0000) 2022-09-10T13:47:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:30.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:30.322249+0000) 2022-09-10T13:47:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:30.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:30.937561+0000) 2022-09-10T13:47:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:31.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:31.322407+0000) 2022-09-10T13:47:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:31 smithi110 ceph-mon[97633]: pgmap v17863: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:31 smithi104 ceph-mon[97044]: pgmap v17863: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:31.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:31.937773+0000) 2022-09-10T13:47:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:32.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:32.322614+0000) 2022-09-10T13:47:33.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:32.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:32.938039+0000) 2022-09-10T13:47:33.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:33.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:33.322864+0000) 2022-09-10T13:47:33.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:47:33.266+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:47:33.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:33 smithi110 ceph-mon[97633]: pgmap v17864: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:33 smithi104 ceph-mon[97044]: pgmap v17864: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:33.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:33.938244+0000) 2022-09-10T13:47:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:34.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:34.323127+0000) 2022-09-10T13:47:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:34.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:34.938420+0000) 2022-09-10T13:47:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:35.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:35.323312+0000) 2022-09-10T13:47:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:35 smithi110 ceph-mon[97633]: pgmap v17865: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:36.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:35 smithi104 ceph-mon[97044]: pgmap v17865: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:35.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:35.938665+0000) 2022-09-10T13:47:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:36.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:36.323515+0000) 2022-09-10T13:47:37.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:36 smithi104 ceph-mon[97044]: pgmap v17866: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:36.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:36.938870+0000) 2022-09-10T13:47:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:36 smithi110 ceph-mon[97633]: pgmap v17866: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:37.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:37.323788+0000) 2022-09-10T13:47:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:37.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:37.939145+0000) 2022-09-10T13:47:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:38.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:38.323994+0000) 2022-09-10T13:47:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:38.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:38.939325+0000) 2022-09-10T13:47:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:39.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:39.324274+0000) 2022-09-10T13:47:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:39 smithi110 ceph-mon[97633]: pgmap v17867: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:39 smithi104 ceph-mon[97044]: pgmap v17867: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:39.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:39.939520+0000) 2022-09-10T13:47:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:40.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:40.324510+0000) 2022-09-10T13:47:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:40.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:40.939710+0000) 2022-09-10T13:47:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:41.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:41.324729+0000) 2022-09-10T13:47:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:41 smithi110 ceph-mon[97633]: pgmap v17868: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:41 smithi104 ceph-mon[97044]: pgmap v17868: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:42.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:41.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:41.939954+0000) 2022-09-10T13:47:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:42.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:42.324987+0000) 2022-09-10T13:47:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:42.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:42.940185+0000) 2022-09-10T13:47:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:43.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:43.325220+0000) 2022-09-10T13:47:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:43 smithi110 ceph-mon[97633]: pgmap v17869: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:43 smithi104 ceph-mon[97044]: pgmap v17869: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:43.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:43.940313+0000) 2022-09-10T13:47:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:44.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:44.325448+0000) 2022-09-10T13:47:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:44.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:44.940554+0000) 2022-09-10T13:47:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:45.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:45.325680+0000) 2022-09-10T13:47:45.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:45 smithi110 ceph-mon[97633]: pgmap v17870: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:46.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:45 smithi104 ceph-mon[97044]: pgmap v17870: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:45.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:45.940707+0000) 2022-09-10T13:47:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:46.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:46.325911+0000) 2022-09-10T13:47:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:47:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:47:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:47:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:47:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:46 smithi104 ceph-mon[97044]: pgmap v17871: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:46.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:46.940892+0000) 2022-09-10T13:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:47:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:47:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:46 smithi110 ceph-mon[97633]: pgmap v17871: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:47.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:47.326173+0000) 2022-09-10T13:47:48.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:47.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:47.941152+0000) 2022-09-10T13:47:48.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:47:48.282+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:47:48.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:48.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:48.326293+0000) 2022-09-10T13:47:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:48.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:48.941394+0000) 2022-09-10T13:47:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:49.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:49.326538+0000) 2022-09-10T13:47:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:49 smithi110 ceph-mon[97633]: pgmap v17872: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:49 smithi104 ceph-mon[97044]: pgmap v17872: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:49.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:49.941616+0000) 2022-09-10T13:47:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:50.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:50.326759+0000) 2022-09-10T13:47:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:50.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:50.941889+0000) 2022-09-10T13:47:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:51.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:51.326951+0000) 2022-09-10T13:47:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:51 smithi110 ceph-mon[97633]: pgmap v17873: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:52.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:51 smithi104 ceph-mon[97044]: pgmap v17873: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:51.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:51.942132+0000) 2022-09-10T13:47:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:52.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:52.327210+0000) 2022-09-10T13:47:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:52.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:52.942354+0000) 2022-09-10T13:47:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:53.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:53.327471+0000) 2022-09-10T13:47:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:53 smithi110 ceph-mon[97633]: pgmap v17874: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:53 smithi104 ceph-mon[97044]: pgmap v17874: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:53.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:53.942564+0000) 2022-09-10T13:47:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:54.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:54.327712+0000) 2022-09-10T13:47:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:54.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:54.942815+0000) 2022-09-10T13:47:55.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:55.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:55.327972+0000) 2022-09-10T13:47:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:55 smithi110 ceph-mon[97633]: pgmap v17875: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:56.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:55 smithi104 ceph-mon[97044]: pgmap v17875: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:47:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:55.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:55.943086+0000) 2022-09-10T13:47:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:56.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:56.328193+0000) 2022-09-10T13:47:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:56 smithi104 ceph-mon[97044]: pgmap v17876: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:56.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:56.943254+0000) 2022-09-10T13:47:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:56 smithi110 ceph-mon[97633]: pgmap v17876: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:47:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:57.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:57.328434+0000) 2022-09-10T13:47:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:57.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:57.943423+0000) 2022-09-10T13:47:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:58.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:58.328688+0000) 2022-09-10T13:47:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:58.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:58.943603+0000) 2022-09-10T13:47:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:47:59.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:59.328860+0000) 2022-09-10T13:47:59.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:47:59 smithi110 ceph-mon[97633]: pgmap v17877: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:47:59 smithi104 ceph-mon[97044]: pgmap v17877: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:47:59.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:47:59.943806+0000) 2022-09-10T13:48:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:00.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:00.329077+0000) 2022-09-10T13:48:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:00.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:00.944011+0000) 2022-09-10T13:48:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:01.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:01.329303+0000) 2022-09-10T13:48:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:01 smithi110 ceph-mon[97633]: pgmap v17878: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:01 smithi104 ceph-mon[97044]: pgmap v17878: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:01.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:01.944275+0000) 2022-09-10T13:48:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:02.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:02.329456+0000) 2022-09-10T13:48:03.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:02.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:02.944452+0000) 2022-09-10T13:48:03.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:48:03.290+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:48:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:03.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:03.329620+0000) 2022-09-10T13:48:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:03 smithi110 ceph-mon[97633]: pgmap v17879: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:03 smithi104 ceph-mon[97044]: pgmap v17879: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:03.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:03.944682+0000) 2022-09-10T13:48:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:04.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:04.329847+0000) 2022-09-10T13:48:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:04.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:04.944951+0000) 2022-09-10T13:48:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:05.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:05.330074+0000) 2022-09-10T13:48:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:05 smithi110 ceph-mon[97633]: pgmap v17880: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:48:06.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:05 smithi104 ceph-mon[97044]: pgmap v17880: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:48:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:05.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:05.945154+0000) 2022-09-10T13:48:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:06.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:06.330285+0000) 2022-09-10T13:48:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:06 smithi104 ceph-mon[97044]: pgmap v17881: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:48:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:06.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:06.945300+0000) 2022-09-10T13:48:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:06 smithi110 ceph-mon[97633]: pgmap v17881: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:48:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:07.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:07.330447+0000) 2022-09-10T13:48:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:07.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:07.945489+0000) 2022-09-10T13:48:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:08.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:08.330696+0000) 2022-09-10T13:48:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:08 smithi104 ceph-mon[97044]: osdmap e1198: 12 total, 12 up, 12 in 2022-09-10T13:48:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:08 smithi104 ceph-mon[97044]: pgmap v17883: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:48:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:08.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:08.945695+0000) 2022-09-10T13:48:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:08 smithi110 ceph-mon[97633]: osdmap e1198: 12 total, 12 up, 12 in 2022-09-10T13:48:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:08 smithi110 ceph-mon[97633]: pgmap v17883: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:48:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:09.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:09.330916+0000) 2022-09-10T13:48:10.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:09.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:09.945932+0000) 2022-09-10T13:48:10.491 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:10.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:10.331137+0000) 2022-09-10T13:48:11.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:10.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:10.946144+0000) 2022-09-10T13:48:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:11.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:11.331298+0000) 2022-09-10T13:48:11.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:11 smithi110 ceph-mon[97633]: pgmap v17884: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:48:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:11 smithi110 ceph-mon[97633]: osdmap e1199: 12 total, 12 up, 12 in 2022-09-10T13:48:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:11 smithi104 ceph-mon[97044]: pgmap v17884: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T13:48:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:11 smithi104 ceph-mon[97044]: osdmap e1199: 12 total, 12 up, 12 in 2022-09-10T13:48:12.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:11.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:11.946286+0000) 2022-09-10T13:48:12.485 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:12.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:12.331422+0000) 2022-09-10T13:48:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:12.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:12.946427+0000) 2022-09-10T13:48:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:13.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:13.331592+0000) 2022-09-10T13:48:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:13 smithi110 ceph-mon[97633]: pgmap v17886: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:48:14.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:13 smithi104 ceph-mon[97044]: pgmap v17886: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:48:14.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:13.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:13.946634+0000) 2022-09-10T13:48:14.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:14.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:14.331839+0000) 2022-09-10T13:48:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:14.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:14.946861+0000) 2022-09-10T13:48:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:15.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:15.332131+0000) 2022-09-10T13:48:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:15 smithi110 ceph-mon[97633]: pgmap v17887: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:48:16.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:15 smithi104 ceph-mon[97044]: pgmap v17887: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:48:16.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:15.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:15.947164+0000) 2022-09-10T13:48:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:16.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:16.332377+0000) 2022-09-10T13:48:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:16 smithi104 ceph-mon[97044]: pgmap v17888: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:48:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:16.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:16.947396+0000) 2022-09-10T13:48:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:16 smithi110 ceph-mon[97633]: pgmap v17888: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T13:48:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:17.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:17.332593+0000) 2022-09-10T13:48:18.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:17.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:17.947583+0000) 2022-09-10T13:48:18.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:18.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:18.332844+0000) 2022-09-10T13:48:18.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:48:18.299+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:48:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:18.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:18.947801+0000) 2022-09-10T13:48:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:19.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:19.333078+0000) 2022-09-10T13:48:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:19 smithi110 ceph-mon[97633]: pgmap v17889: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:48:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:19 smithi104 ceph-mon[97044]: pgmap v17889: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:48:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:19.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:19.948004+0000) 2022-09-10T13:48:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:20.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:20.333302+0000) 2022-09-10T13:48:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:20.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:20.948178+0000) 2022-09-10T13:48:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:21.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:21.333555+0000) 2022-09-10T13:48:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:21 smithi110 ceph-mon[97633]: pgmap v17890: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:48:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:21 smithi104 ceph-mon[97044]: pgmap v17890: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:48:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:21.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:21.948368+0000) 2022-09-10T13:48:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:22.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:22.333754+0000) 2022-09-10T13:48:23.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:22.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:22.948625+0000) 2022-09-10T13:48:23.553 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:23.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:23.333954+0000) 2022-09-10T13:48:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:23 smithi110 ceph-mon[97633]: pgmap v17891: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:48:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:23 smithi104 ceph-mon[97044]: pgmap v17891: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:48:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:23.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:23.948868+0000) 2022-09-10T13:48:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:24.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:24.334216+0000) 2022-09-10T13:48:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:24.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:24.949145+0000) 2022-09-10T13:48:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:25.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:25.334450+0000) 2022-09-10T13:48:25.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:25 smithi110 ceph-mon[97633]: pgmap v17892: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:48:26.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:25 smithi104 ceph-mon[97044]: pgmap v17892: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:48:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:25.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:25.949349+0000) 2022-09-10T13:48:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:26.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:26.334705+0000) 2022-09-10T13:48:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:26 smithi104 ceph-mon[97044]: pgmap v17893: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:48:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:26.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:26.949597+0000) 2022-09-10T13:48:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:26 smithi110 ceph-mon[97633]: pgmap v17893: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:48:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:27.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:27.334863+0000) 2022-09-10T13:48:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:27.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:27.949846+0000) 2022-09-10T13:48:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:28.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:28.334992+0000) 2022-09-10T13:48:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:28.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:28.950062+0000) 2022-09-10T13:48:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:29.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:29.335225+0000) 2022-09-10T13:48:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:29 smithi110 ceph-mon[97633]: pgmap v17894: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:48:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:29 smithi104 ceph-mon[97044]: pgmap v17894: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:48:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:29.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:29.950266+0000) 2022-09-10T13:48:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:30.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:30.335460+0000) 2022-09-10T13:48:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:30.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:30.950503+0000) 2022-09-10T13:48:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:31.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:31.335703+0000) 2022-09-10T13:48:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:31 smithi110 ceph-mon[97633]: pgmap v17895: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:32.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:31 smithi104 ceph-mon[97044]: pgmap v17895: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:31.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:31.950760+0000) 2022-09-10T13:48:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:32.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:32.335832+0000) 2022-09-10T13:48:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:32.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:32.950990+0000) 2022-09-10T13:48:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:33.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:33.335980+0000) 2022-09-10T13:48:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:48:33.315+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:48:33.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:33 smithi110 ceph-mon[97633]: pgmap v17896: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:34.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:33 smithi104 ceph-mon[97044]: pgmap v17896: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:33.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:33.951212+0000) 2022-09-10T13:48:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:34.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:34.336163+0000) 2022-09-10T13:48:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:34.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:34.951496+0000) 2022-09-10T13:48:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:35.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:35.336355+0000) 2022-09-10T13:48:35.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:35 smithi110 ceph-mon[97633]: pgmap v17897: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:36.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:35 smithi104 ceph-mon[97044]: pgmap v17897: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:35.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:35.951687+0000) 2022-09-10T13:48:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:36.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:36.336557+0000) 2022-09-10T13:48:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:36 smithi104 ceph-mon[97044]: pgmap v17898: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:36.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:36.951905+0000) 2022-09-10T13:48:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:36 smithi110 ceph-mon[97633]: pgmap v17898: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:37.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:37.336729+0000) 2022-09-10T13:48:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:37.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:37.952177+0000) 2022-09-10T13:48:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:38.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:38.336908+0000) 2022-09-10T13:48:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:38.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:38.952362+0000) 2022-09-10T13:48:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:39.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:39.337143+0000) 2022-09-10T13:48:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:39 smithi110 ceph-mon[97633]: pgmap v17899: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:39 smithi104 ceph-mon[97044]: pgmap v17899: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:39.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:39.952572+0000) 2022-09-10T13:48:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:40.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:40.337282+0000) 2022-09-10T13:48:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:40.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:40.952821+0000) 2022-09-10T13:48:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:41.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:41.337430+0000) 2022-09-10T13:48:41.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:41 smithi110 ceph-mon[97633]: pgmap v17900: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:42.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:41 smithi104 ceph-mon[97044]: pgmap v17900: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:41.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:41.953065+0000) 2022-09-10T13:48:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:42.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:42.337575+0000) 2022-09-10T13:48:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:42.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:42.953328+0000) 2022-09-10T13:48:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:43.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:43.337814+0000) 2022-09-10T13:48:43.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:43 smithi110 ceph-mon[97633]: pgmap v17901: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:43 smithi104 ceph-mon[97044]: pgmap v17901: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:43.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:43.953519+0000) 2022-09-10T13:48:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:44.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:44.338059+0000) 2022-09-10T13:48:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:44.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:44.953764+0000) 2022-09-10T13:48:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:45.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:45.338292+0000) 2022-09-10T13:48:45.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:45 smithi110 ceph-mon[97633]: pgmap v17902: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:46.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:45 smithi104 ceph-mon[97044]: pgmap v17902: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:45.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:45.954000+0000) 2022-09-10T13:48:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:46.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:46.338569+0000) 2022-09-10T13:48:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:48:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:48:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:48:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:48:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:46 smithi104 ceph-mon[97044]: pgmap v17903: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:46.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:46.954274+0000) 2022-09-10T13:48:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:48:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:48:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:48:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:46 smithi110 ceph-mon[97633]: pgmap v17903: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:47.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:47.338851+0000) 2022-09-10T13:48:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:47.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:47.954496+0000) 2022-09-10T13:48:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:48.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:48.339122+0000) 2022-09-10T13:48:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:48:48.320+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:48:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:48.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:48.954783+0000) 2022-09-10T13:48:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:49.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:49.339356+0000) 2022-09-10T13:48:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:49 smithi110 ceph-mon[97633]: pgmap v17904: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:49 smithi104 ceph-mon[97044]: pgmap v17904: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:49.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:49.954945+0000) 2022-09-10T13:48:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:50.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:50.339630+0000) 2022-09-10T13:48:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:50.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:50.955198+0000) 2022-09-10T13:48:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:51.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:51.339889+0000) 2022-09-10T13:48:51.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:51 smithi110 ceph-mon[97633]: pgmap v17905: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:52.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:51 smithi104 ceph-mon[97044]: pgmap v17905: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:51.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:51.955376+0000) 2022-09-10T13:48:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:52.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:52.340075+0000) 2022-09-10T13:48:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:52.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:52.955562+0000) 2022-09-10T13:48:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:53.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:53.340328+0000) 2022-09-10T13:48:53.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:53 smithi110 ceph-mon[97633]: pgmap v17906: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:54.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:53 smithi104 ceph-mon[97044]: pgmap v17906: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:53.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:53.955732+0000) 2022-09-10T13:48:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:54.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:54.340625+0000) 2022-09-10T13:48:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:54.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:54.955969+0000) 2022-09-10T13:48:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:55.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:55.340826+0000) 2022-09-10T13:48:55.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:55 smithi110 ceph-mon[97633]: pgmap v17907: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:56.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:55 smithi104 ceph-mon[97044]: pgmap v17907: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:48:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:55.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:55.956135+0000) 2022-09-10T13:48:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:56.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:56.341001+0000) 2022-09-10T13:48:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:57 smithi104 ceph-mon[97044]: pgmap v17908: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:56.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:56.956364+0000) 2022-09-10T13:48:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:57 smithi110 ceph-mon[97633]: pgmap v17908: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:48:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:57.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:57.341268+0000) 2022-09-10T13:48:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:57.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:57.956547+0000) 2022-09-10T13:48:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:58.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:58.341480+0000) 2022-09-10T13:48:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:58.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:58.956758+0000) 2022-09-10T13:48:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:48:59.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:59.341732+0000) 2022-09-10T13:48:59.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:48:59 smithi110 ceph-mon[97633]: pgmap v17909: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:00.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:48:59 smithi104 ceph-mon[97044]: pgmap v17909: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:48:59.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:48:59.956998+0000) 2022-09-10T13:49:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:00.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:00.342005+0000) 2022-09-10T13:49:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:00.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:00.957155+0000) 2022-09-10T13:49:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:01.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:01.342179+0000) 2022-09-10T13:49:01.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:01 smithi110 ceph-mon[97633]: pgmap v17910: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:02.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:01 smithi104 ceph-mon[97044]: pgmap v17910: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:01.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:01.957365+0000) 2022-09-10T13:49:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:02.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:02.342386+0000) 2022-09-10T13:49:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:02.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:02.957604+0000) 2022-09-10T13:49:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:03.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:03.342590+0000) 2022-09-10T13:49:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:49:03.335+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:49:03.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:03 smithi110 ceph-mon[97633]: pgmap v17911: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:04.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:03 smithi104 ceph-mon[97044]: pgmap v17911: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:03.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:03.957804+0000) 2022-09-10T13:49:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:04.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:04.342810+0000) 2022-09-10T13:49:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:04.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:04.957945+0000) 2022-09-10T13:49:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:05.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:05.343030+0000) 2022-09-10T13:49:05.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:05 smithi110 ceph-mon[97633]: pgmap v17912: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:49:06.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:05 smithi104 ceph-mon[97044]: pgmap v17912: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:49:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:05.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:05.958161+0000) 2022-09-10T13:49:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:06.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:06.343280+0000) 2022-09-10T13:49:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:06 smithi104 ceph-mon[97044]: pgmap v17913: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:49:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:06.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:06.958362+0000) 2022-09-10T13:49:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:06 smithi110 ceph-mon[97633]: pgmap v17913: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:49:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:07.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:07.343446+0000) 2022-09-10T13:49:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:07.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:07.958557+0000) 2022-09-10T13:49:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:08.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:08.343617+0000) 2022-09-10T13:49:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:09 smithi104 ceph-mon[97044]: osdmap e1200: 12 total, 12 up, 12 in 2022-09-10T13:49:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:09 smithi104 ceph-mon[97044]: pgmap v17915: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:49:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:08.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:08.958741+0000) 2022-09-10T13:49:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:09 smithi110 ceph-mon[97633]: osdmap e1200: 12 total, 12 up, 12 in 2022-09-10T13:49:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:09 smithi110 ceph-mon[97633]: pgmap v17915: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:49:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:09.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:09.343775+0000) 2022-09-10T13:49:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:09.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:09.958937+0000) 2022-09-10T13:49:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:10.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:10.343995+0000) 2022-09-10T13:49:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:10.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:10.959181+0000) 2022-09-10T13:49:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:11.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:11.344232+0000) 2022-09-10T13:49:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:11 smithi110 ceph-mon[97633]: pgmap v17916: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:49:11.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:11 smithi110 ceph-mon[97633]: osdmap e1201: 12 total, 12 up, 12 in 2022-09-10T13:49:12.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:11 smithi104 ceph-mon[97044]: pgmap v17916: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:49:12.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:11 smithi104 ceph-mon[97044]: osdmap e1201: 12 total, 12 up, 12 in 2022-09-10T13:49:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:11.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:11.959387+0000) 2022-09-10T13:49:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:12.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:12.344450+0000) 2022-09-10T13:49:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:12.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:12.959656+0000) 2022-09-10T13:49:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:13.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:13.344708+0000) 2022-09-10T13:49:13.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:13 smithi110 ceph-mon[97633]: pgmap v17918: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:49:14.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:13 smithi104 ceph-mon[97044]: pgmap v17918: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:49:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:13.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:13.959801+0000) 2022-09-10T13:49:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:14.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:14.344964+0000) 2022-09-10T13:49:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:14.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:14.960022+0000) 2022-09-10T13:49:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:15.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:15.345214+0000) 2022-09-10T13:49:15.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:15 smithi110 ceph-mon[97633]: pgmap v17919: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:49:16.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:15 smithi104 ceph-mon[97044]: pgmap v17919: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:49:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:15.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:15.960291+0000) 2022-09-10T13:49:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:16.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:16.345423+0000) 2022-09-10T13:49:17.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:17 smithi104 ceph-mon[97044]: pgmap v17920: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:49:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:16.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:16.960548+0000) 2022-09-10T13:49:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:17 smithi110 ceph-mon[97633]: pgmap v17920: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:49:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:17.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:17.345652+0000) 2022-09-10T13:49:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:17.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:17.960802+0000) 2022-09-10T13:49:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:18.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:18.345828+0000) 2022-09-10T13:49:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:49:18.351+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:49:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:18.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:18.960982+0000) 2022-09-10T13:49:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:19.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:19.346085+0000) 2022-09-10T13:49:19.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:19 smithi110 ceph-mon[97633]: pgmap v17921: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:49:20.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:19 smithi104 ceph-mon[97044]: pgmap v17921: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:49:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:19.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:19.961232+0000) 2022-09-10T13:49:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:20.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:20.346319+0000) 2022-09-10T13:49:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:20.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:20.961459+0000) 2022-09-10T13:49:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:21.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:21.346474+0000) 2022-09-10T13:49:21.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:21 smithi110 ceph-mon[97633]: pgmap v17922: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:49:22.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:21 smithi104 ceph-mon[97044]: pgmap v17922: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:49:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:21.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:21.961689+0000) 2022-09-10T13:49:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:22.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:22.346665+0000) 2022-09-10T13:49:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:22.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:22.961910+0000) 2022-09-10T13:49:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:23.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:23.346899+0000) 2022-09-10T13:49:23.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:23 smithi110 ceph-mon[97633]: pgmap v17923: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:49:24.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:23 smithi104 ceph-mon[97044]: pgmap v17923: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:49:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:23.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:23.962124+0000) 2022-09-10T13:49:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:24.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:24.347135+0000) 2022-09-10T13:49:24.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:24 smithi110 ceph-mon[97633]: pgmap v17924: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:49:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:24 smithi104 ceph-mon[97044]: pgmap v17924: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:49:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:24.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:24.962349+0000) 2022-09-10T13:49:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:25.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:25.347346+0000) 2022-09-10T13:49:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:25.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:25.962588+0000) 2022-09-10T13:49:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:26.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:26.347582+0000) 2022-09-10T13:49:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:27 smithi104 ceph-mon[97044]: pgmap v17925: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:49:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:26.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:26.962803+0000) 2022-09-10T13:49:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:27 smithi110 ceph-mon[97633]: pgmap v17925: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:49:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:27.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:27.347832+0000) 2022-09-10T13:49:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:27.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:27.962984+0000) 2022-09-10T13:49:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:28.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:28.348060+0000) 2022-09-10T13:49:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:28.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:28.963152+0000) 2022-09-10T13:49:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:29.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:29.348285+0000) 2022-09-10T13:49:29.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:29 smithi110 ceph-mon[97633]: pgmap v17926: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:49:30.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:29 smithi104 ceph-mon[97044]: pgmap v17926: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:49:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:29.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:29.963366+0000) 2022-09-10T13:49:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:30.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:30.348502+0000) 2022-09-10T13:49:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:30.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:30.963638+0000) 2022-09-10T13:49:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:31.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:31.348700+0000) 2022-09-10T13:49:31.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:31 smithi110 ceph-mon[97633]: pgmap v17927: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:32.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:31 smithi104 ceph-mon[97044]: pgmap v17927: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:31.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:31.963833+0000) 2022-09-10T13:49:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:32.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:32.348829+0000) 2022-09-10T13:49:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:32 smithi104 ceph-mon[97044]: pgmap v17928: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:49:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:32 smithi110 ceph-mon[97633]: pgmap v17928: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:49:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:32.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:32.964123+0000) 2022-09-10T13:49:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:33.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:33.349048+0000) 2022-09-10T13:49:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:49:33.367+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:49:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:33.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:33.964372+0000) 2022-09-10T13:49:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:34.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:34.349280+0000) 2022-09-10T13:49:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:34.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:34.964623+0000) 2022-09-10T13:49:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:35.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:35.349510+0000) 2022-09-10T13:49:35.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:35 smithi110 ceph-mon[97633]: pgmap v17929: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:36.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:35 smithi104 ceph-mon[97044]: pgmap v17929: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:35.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:35.964880+0000) 2022-09-10T13:49:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:36.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:36.349727+0000) 2022-09-10T13:49:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:37 smithi104 ceph-mon[97044]: pgmap v17930: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:36.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:36.965134+0000) 2022-09-10T13:49:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:37 smithi110 ceph-mon[97633]: pgmap v17930: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:37.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:37.349980+0000) 2022-09-10T13:49:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:37.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:37.965403+0000) 2022-09-10T13:49:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:38.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:38.350194+0000) 2022-09-10T13:49:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:38.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:38.965660+0000) 2022-09-10T13:49:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:39.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:39.350425+0000) 2022-09-10T13:49:39.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:39 smithi110 ceph-mon[97633]: pgmap v17931: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:40.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:39 smithi104 ceph-mon[97044]: pgmap v17931: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:39.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:39.965936+0000) 2022-09-10T13:49:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:40.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:40.350633+0000) 2022-09-10T13:49:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:40 smithi104 ceph-mon[97044]: pgmap v17932: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:40 smithi110 ceph-mon[97633]: pgmap v17932: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:40.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:40.966219+0000) 2022-09-10T13:49:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:41.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:41.350833+0000) 2022-09-10T13:49:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:41.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:41.966487+0000) 2022-09-10T13:49:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:42.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:42.350987+0000) 2022-09-10T13:49:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:42.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:42.966690+0000) 2022-09-10T13:49:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:43.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:43.351215+0000) 2022-09-10T13:49:43.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:43 smithi110 ceph-mon[97633]: pgmap v17933: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:44.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:43 smithi104 ceph-mon[97044]: pgmap v17933: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:43.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:43.966919+0000) 2022-09-10T13:49:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:44.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:44.351384+0000) 2022-09-10T13:49:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:44 smithi104 ceph-mon[97044]: pgmap v17934: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:44 smithi110 ceph-mon[97633]: pgmap v17934: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:44.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:44.967189+0000) 2022-09-10T13:49:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:45.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:45.351600+0000) 2022-09-10T13:49:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:45.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:45.967403+0000) 2022-09-10T13:49:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:46.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:46.351844+0000) 2022-09-10T13:49:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:49:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:49:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:49:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:49:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:47 smithi104 ceph-mon[97044]: pgmap v17935: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:46.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:46.967644+0000) 2022-09-10T13:49:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:49:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:49:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:49:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:49:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:47 smithi110 ceph-mon[97633]: pgmap v17935: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:47.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:47.352136+0000) 2022-09-10T13:49:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:47.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:47.967898+0000) 2022-09-10T13:49:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:48.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:48.352339+0000) 2022-09-10T13:49:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:49:48.370+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:49:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:48.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:48.968186+0000) 2022-09-10T13:49:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:49.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:49.352523+0000) 2022-09-10T13:49:49.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:49 smithi110 ceph-mon[97633]: pgmap v17936: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:50.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:49 smithi104 ceph-mon[97044]: pgmap v17936: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:49.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:49.968470+0000) 2022-09-10T13:49:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:50.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:50.352743+0000) 2022-09-10T13:49:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:50 smithi110 ceph-mon[97633]: pgmap v17937: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:50 smithi104 ceph-mon[97044]: pgmap v17937: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:50.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:50.968684+0000) 2022-09-10T13:49:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:51.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:51.352917+0000) 2022-09-10T13:49:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:51.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:51.968853+0000) 2022-09-10T13:49:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:52.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:52.353165+0000) 2022-09-10T13:49:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:52 smithi104 ceph-mon[97044]: pgmap v17938: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:52 smithi110 ceph-mon[97633]: pgmap v17938: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:52.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:52.969091+0000) 2022-09-10T13:49:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:53.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:53.353396+0000) 2022-09-10T13:49:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:53.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:53.969397+0000) 2022-09-10T13:49:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:54.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:54.353678+0000) 2022-09-10T13:49:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:54 smithi104 ceph-mon[97044]: pgmap v17939: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:54 smithi110 ceph-mon[97633]: pgmap v17939: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:49:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:54.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:54.969666+0000) 2022-09-10T13:49:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:55.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:55.353883+0000) 2022-09-10T13:49:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:55.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:55.969915+0000) 2022-09-10T13:49:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:56.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:56.354014+0000) 2022-09-10T13:49:57.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:57 smithi104 ceph-mon[97044]: pgmap v17940: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:56.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:56.970126+0000) 2022-09-10T13:49:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:57 smithi110 ceph-mon[97633]: pgmap v17940: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:57.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:57.354257+0000) 2022-09-10T13:49:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:57.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:57.970386+0000) 2022-09-10T13:49:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:58.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:58.354492+0000) 2022-09-10T13:49:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:49:58 smithi104 ceph-mon[97044]: pgmap v17941: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:49:58 smithi110 ceph-mon[97633]: pgmap v17941: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:49:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:58.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:58.970623+0000) 2022-09-10T13:49:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:49:59.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:59.354623+0000) 2022-09-10T13:50:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:49:59.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:49:59.970824+0000) 2022-09-10T13:50:00.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:50:00.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T13:50:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:00.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:00.354813+0000) 2022-09-10T13:50:01.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:00.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:00.971065+0000) 2022-09-10T13:50:01.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:01 smithi110 ceph-mon[97633]: pgmap v17942: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:01.466 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:01 smithi104 ceph-mon[97044]: pgmap v17942: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:01.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:01.355051+0000) 2022-09-10T13:50:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:01.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:01.971300+0000) 2022-09-10T13:50:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:02.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:02.355306+0000) 2022-09-10T13:50:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:02 smithi104 ceph-mon[97044]: pgmap v17943: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:02 smithi110 ceph-mon[97633]: pgmap v17943: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:02.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:02.971519+0000) 2022-09-10T13:50:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:03.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:03.355459+0000) 2022-09-10T13:50:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:50:03.378+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:50:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:03.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:03.971725+0000) 2022-09-10T13:50:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:04.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:04.355600+0000) 2022-09-10T13:50:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:04 smithi104 ceph-mon[97044]: pgmap v17944: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:50:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:04 smithi110 ceph-mon[97633]: pgmap v17944: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:50:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:04.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:04.972020+0000) 2022-09-10T13:50:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:05.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:05.355770+0000) 2022-09-10T13:50:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:05.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:05.972225+0000) 2022-09-10T13:50:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:06.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:06.356035+0000) 2022-09-10T13:50:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:07 smithi104 ceph-mon[97044]: pgmap v17945: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:50:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:06.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:06.972398+0000) 2022-09-10T13:50:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:07 smithi110 ceph-mon[97633]: pgmap v17945: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:50:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:07.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:07.356243+0000) 2022-09-10T13:50:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:07.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:07.972634+0000) 2022-09-10T13:50:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:08.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:08.356376+0000) 2022-09-10T13:50:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:09 smithi104 ceph-mon[97044]: osdmap e1202: 12 total, 12 up, 12 in 2022-09-10T13:50:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:09 smithi104 ceph-mon[97044]: pgmap v17947: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:50:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:08.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:08.972908+0000) 2022-09-10T13:50:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:09 smithi110 ceph-mon[97633]: osdmap e1202: 12 total, 12 up, 12 in 2022-09-10T13:50:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:09 smithi110 ceph-mon[97633]: pgmap v17947: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:50:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:09.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:09.356592+0000) 2022-09-10T13:50:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:09.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:09.973125+0000) 2022-09-10T13:50:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:10.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:10.356798+0000) 2022-09-10T13:50:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:10 smithi104 ceph-mon[97044]: pgmap v17948: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:50:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:10 smithi110 ceph-mon[97633]: pgmap v17948: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:50:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:10.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:10.973267+0000) 2022-09-10T13:50:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:11.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:11.357033+0000) 2022-09-10T13:50:12.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:12 smithi104 ceph-mon[97044]: osdmap e1203: 12 total, 12 up, 12 in 2022-09-10T13:50:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:11.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:11.973541+0000) 2022-09-10T13:50:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:12 smithi110 ceph-mon[97633]: osdmap e1203: 12 total, 12 up, 12 in 2022-09-10T13:50:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:12.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:12.357277+0000) 2022-09-10T13:50:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:13 smithi104 ceph-mon[97044]: pgmap v17950: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:50:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:12.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:12.973787+0000) 2022-09-10T13:50:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:13 smithi110 ceph-mon[97633]: pgmap v17950: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:50:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:13.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:13.357541+0000) 2022-09-10T13:50:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:13.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:13.974026+0000) 2022-09-10T13:50:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:14.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:14.357774+0000) 2022-09-10T13:50:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:14 smithi104 ceph-mon[97044]: pgmap v17951: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:50:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:14 smithi110 ceph-mon[97633]: pgmap v17951: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:50:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:14.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:14.974224+0000) 2022-09-10T13:50:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:15.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:15.357951+0000) 2022-09-10T13:50:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:15.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:15.974446+0000) 2022-09-10T13:50:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:16.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:16.358204+0000) 2022-09-10T13:50:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:17 smithi104 ceph-mon[97044]: pgmap v17952: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:50:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:16.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:16.974703+0000) 2022-09-10T13:50:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:17 smithi110 ceph-mon[97633]: pgmap v17952: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:50:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:17.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:17.358362+0000) 2022-09-10T13:50:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:17.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:17.974949+0000) 2022-09-10T13:50:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:18.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:18.358555+0000) 2022-09-10T13:50:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:50:18.391+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:50:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:18 smithi104 ceph-mon[97044]: pgmap v17953: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:50:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:18 smithi110 ceph-mon[97633]: pgmap v17953: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:50:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:18.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:18.975202+0000) 2022-09-10T13:50:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:19.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:19.358824+0000) 2022-09-10T13:50:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:19.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:19.975436+0000) 2022-09-10T13:50:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:20.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:20.359090+0000) 2022-09-10T13:50:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:20 smithi104 ceph-mon[97044]: pgmap v17954: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:50:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:20 smithi110 ceph-mon[97633]: pgmap v17954: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:50:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:20.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:20.975659+0000) 2022-09-10T13:50:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:21.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:21.359376+0000) 2022-09-10T13:50:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:21.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:21.975942+0000) 2022-09-10T13:50:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:22.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:22.359660+0000) 2022-09-10T13:50:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:22 smithi104 ceph-mon[97044]: pgmap v17955: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:50:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:22 smithi110 ceph-mon[97633]: pgmap v17955: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:50:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:22.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:22.976201+0000) 2022-09-10T13:50:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:23.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:23.359940+0000) 2022-09-10T13:50:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:23.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:23.976394+0000) 2022-09-10T13:50:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:24.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:24.360166+0000) 2022-09-10T13:50:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:24 smithi104 ceph-mon[97044]: pgmap v17956: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:50:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:24 smithi110 ceph-mon[97633]: pgmap v17956: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:50:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:24.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:24.976556+0000) 2022-09-10T13:50:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:25.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:25.360421+0000) 2022-09-10T13:50:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:25.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:25.976778+0000) 2022-09-10T13:50:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:26.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:26.360632+0000) 2022-09-10T13:50:27.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:27 smithi104 ceph-mon[97044]: pgmap v17957: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:26.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:26.976964+0000) 2022-09-10T13:50:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:27 smithi110 ceph-mon[97633]: pgmap v17957: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:27.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:27.360839+0000) 2022-09-10T13:50:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:27.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:27.977169+0000) 2022-09-10T13:50:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:28.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:28.361060+0000) 2022-09-10T13:50:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:28 smithi104 ceph-mon[97044]: pgmap v17958: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:28 smithi110 ceph-mon[97633]: pgmap v17958: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:28.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:28.977328+0000) 2022-09-10T13:50:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:29.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:29.361262+0000) 2022-09-10T13:50:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:29.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:29.977561+0000) 2022-09-10T13:50:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:30.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:30.361562+0000) 2022-09-10T13:50:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:30 smithi104 ceph-mon[97044]: pgmap v17959: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:30 smithi110 ceph-mon[97633]: pgmap v17959: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:30.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:30.977787+0000) 2022-09-10T13:50:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:31.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:31.361806+0000) 2022-09-10T13:50:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:31.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:31.978024+0000) 2022-09-10T13:50:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:32.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:32.361994+0000) 2022-09-10T13:50:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:32 smithi104 ceph-mon[97044]: pgmap v17960: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:32 smithi110 ceph-mon[97633]: pgmap v17960: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:32.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:32.978217+0000) 2022-09-10T13:50:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:33.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:33.362215+0000) 2022-09-10T13:50:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:50:33.397+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:50:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:33.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:33.978428+0000) 2022-09-10T13:50:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:34.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:34.362449+0000) 2022-09-10T13:50:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:34 smithi104 ceph-mon[97044]: pgmap v17961: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:34 smithi110 ceph-mon[97633]: pgmap v17961: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:34.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:34.978675+0000) 2022-09-10T13:50:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:35.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:35.362718+0000) 2022-09-10T13:50:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:35.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:35.978946+0000) 2022-09-10T13:50:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:36.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:36.362932+0000) 2022-09-10T13:50:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:37 smithi104 ceph-mon[97044]: pgmap v17962: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:36.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:36.979217+0000) 2022-09-10T13:50:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:37 smithi110 ceph-mon[97633]: pgmap v17962: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:37.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:37.363175+0000) 2022-09-10T13:50:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:37.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:37.979477+0000) 2022-09-10T13:50:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:38.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:38.363340+0000) 2022-09-10T13:50:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:38 smithi104 ceph-mon[97044]: pgmap v17963: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:38 smithi110 ceph-mon[97633]: pgmap v17963: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:38.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:38.979704+0000) 2022-09-10T13:50:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:39.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:39.363567+0000) 2022-09-10T13:50:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:39.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:39.979902+0000) 2022-09-10T13:50:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:40.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:40.363764+0000) 2022-09-10T13:50:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:40 smithi104 ceph-mon[97044]: pgmap v17964: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:40 smithi110 ceph-mon[97633]: pgmap v17964: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:40.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:40.980195+0000) 2022-09-10T13:50:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:41.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:41.363996+0000) 2022-09-10T13:50:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:41.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:41.980417+0000) 2022-09-10T13:50:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:42.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:42.364238+0000) 2022-09-10T13:50:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:42 smithi104 ceph-mon[97044]: pgmap v17965: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:42 smithi110 ceph-mon[97633]: pgmap v17965: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:42.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:42.980598+0000) 2022-09-10T13:50:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:43.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:43.364390+0000) 2022-09-10T13:50:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:43.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:43.980808+0000) 2022-09-10T13:50:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:44.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:44.364555+0000) 2022-09-10T13:50:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:44 smithi104 ceph-mon[97044]: pgmap v17966: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:44 smithi110 ceph-mon[97633]: pgmap v17966: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:44.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:44.981060+0000) 2022-09-10T13:50:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:45.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:45.364874+0000) 2022-09-10T13:50:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:45.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:45.981351+0000) 2022-09-10T13:50:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:46.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:46.365144+0000) 2022-09-10T13:50:47.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:50:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:50:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:50:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:50:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:47 smithi104 ceph-mon[97044]: pgmap v17967: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:46.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:46.981596+0000) 2022-09-10T13:50:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:50:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:50:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:50:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:50:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:47 smithi110 ceph-mon[97633]: pgmap v17967: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:47.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:47.365357+0000) 2022-09-10T13:50:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:47.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:47.981760+0000) 2022-09-10T13:50:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:48.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:48.365580+0000) 2022-09-10T13:50:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:50:48.412+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:50:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:48 smithi104 ceph-mon[97044]: pgmap v17968: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:48 smithi110 ceph-mon[97633]: pgmap v17968: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:48.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:48.982008+0000) 2022-09-10T13:50:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:49.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:49.365846+0000) 2022-09-10T13:50:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:49.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:49.982259+0000) 2022-09-10T13:50:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:50.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:50.366080+0000) 2022-09-10T13:50:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:50 smithi104 ceph-mon[97044]: pgmap v17969: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:50 smithi110 ceph-mon[97633]: pgmap v17969: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:50.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:50.982484+0000) 2022-09-10T13:50:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:51.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:51.366360+0000) 2022-09-10T13:50:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:51.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:51.982731+0000) 2022-09-10T13:50:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:52.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:52.366560+0000) 2022-09-10T13:50:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:52 smithi104 ceph-mon[97044]: pgmap v17970: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:53.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:52 smithi110 ceph-mon[97633]: pgmap v17970: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:52.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:52.982927+0000) 2022-09-10T13:50:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:53.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:53.366796+0000) 2022-09-10T13:50:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:53.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:53.983184+0000) 2022-09-10T13:50:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:54.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:54.367052+0000) 2022-09-10T13:50:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:54 smithi104 ceph-mon[97044]: pgmap v17971: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:54 smithi110 ceph-mon[97633]: pgmap v17971: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:50:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:54.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:54.983371+0000) 2022-09-10T13:50:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:55.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:55.367223+0000) 2022-09-10T13:50:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:55.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:55.983578+0000) 2022-09-10T13:50:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:56.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:56.367447+0000) 2022-09-10T13:50:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:57 smithi104 ceph-mon[97044]: pgmap v17972: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:56.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:56.983714+0000) 2022-09-10T13:50:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:57 smithi110 ceph-mon[97633]: pgmap v17972: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:57.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:57.367639+0000) 2022-09-10T13:50:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:57.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:57.983896+0000) 2022-09-10T13:50:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:58.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:58.367839+0000) 2022-09-10T13:50:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:50:58 smithi104 ceph-mon[97044]: pgmap v17973: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:50:58 smithi110 ceph-mon[97633]: pgmap v17973: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:50:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:58.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:58.984164+0000) 2022-09-10T13:50:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:50:59.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:59.368003+0000) 2022-09-10T13:51:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:50:59.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:50:59.984392+0000) 2022-09-10T13:51:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:00.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:00.368142+0000) 2022-09-10T13:51:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:00 smithi104 ceph-mon[97044]: pgmap v17974: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:00 smithi110 ceph-mon[97633]: pgmap v17974: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:00.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:00.984585+0000) 2022-09-10T13:51:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:01.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:01.368340+0000) 2022-09-10T13:51:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:01.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:01.984829+0000) 2022-09-10T13:51:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:02.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:02.368496+0000) 2022-09-10T13:51:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:02 smithi104 ceph-mon[97044]: pgmap v17975: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:02 smithi110 ceph-mon[97633]: pgmap v17975: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:02.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:02.985123+0000) 2022-09-10T13:51:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:03.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:03.368668+0000) 2022-09-10T13:51:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:51:03.428+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:51:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:03.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:03.985391+0000) 2022-09-10T13:51:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:04.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:04.368822+0000) 2022-09-10T13:51:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:04 smithi104 ceph-mon[97044]: pgmap v17976: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:51:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:04 smithi110 ceph-mon[97633]: pgmap v17976: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:51:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:04.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:04.985599+0000) 2022-09-10T13:51:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:05.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:05.369064+0000) 2022-09-10T13:51:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:05.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:05.985844+0000) 2022-09-10T13:51:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:06.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:06.369235+0000) 2022-09-10T13:51:07.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:07 smithi104 ceph-mon[97044]: pgmap v17977: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:51:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:06.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:06.986131+0000) 2022-09-10T13:51:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:07 smithi110 ceph-mon[97633]: pgmap v17977: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:51:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:07.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:07.369410+0000) 2022-09-10T13:51:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:07.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:07.986297+0000) 2022-09-10T13:51:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:08.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:08.369615+0000) 2022-09-10T13:51:09.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:09 smithi104 ceph-mon[97044]: osdmap e1204: 12 total, 12 up, 12 in 2022-09-10T13:51:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:09 smithi104 ceph-mon[97044]: pgmap v17979: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:51:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:08.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:08.986545+0000) 2022-09-10T13:51:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:09 smithi110 ceph-mon[97633]: osdmap e1204: 12 total, 12 up, 12 in 2022-09-10T13:51:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:09 smithi110 ceph-mon[97633]: pgmap v17979: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:51:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:09.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:09.369869+0000) 2022-09-10T13:51:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:09.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:09.986816+0000) 2022-09-10T13:51:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:10.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:10.370175+0000) 2022-09-10T13:51:11.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:10 smithi104 ceph-mon[97044]: pgmap v17980: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:51:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:10 smithi110 ceph-mon[97633]: pgmap v17980: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:51:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:10.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:10.986978+0000) 2022-09-10T13:51:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:11.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:11.370339+0000) 2022-09-10T13:51:12.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:12 smithi104 ceph-mon[97044]: osdmap e1205: 12 total, 12 up, 12 in 2022-09-10T13:51:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:11.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:11.987239+0000) 2022-09-10T13:51:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:12 smithi110 ceph-mon[97633]: osdmap e1205: 12 total, 12 up, 12 in 2022-09-10T13:51:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:12.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:12.370584+0000) 2022-09-10T13:51:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:13 smithi104 ceph-mon[97044]: pgmap v17982: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:51:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:12.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:12.987413+0000) 2022-09-10T13:51:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:13 smithi110 ceph-mon[97633]: pgmap v17982: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:51:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:13.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:13.370747+0000) 2022-09-10T13:51:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:13.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:13.987643+0000) 2022-09-10T13:51:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:14.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:14.370887+0000) 2022-09-10T13:51:15.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:14 smithi104 ceph-mon[97044]: pgmap v17983: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:51:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:14 smithi110 ceph-mon[97633]: pgmap v17983: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:51:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:14.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:14.987931+0000) 2022-09-10T13:51:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:15.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:15.371150+0000) 2022-09-10T13:51:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:15.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:15.988188+0000) 2022-09-10T13:51:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:16.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:16.371305+0000) 2022-09-10T13:51:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:17 smithi104 ceph-mon[97044]: pgmap v17984: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:51:17.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:16.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:16.988397+0000) 2022-09-10T13:51:17.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:17 smithi110 ceph-mon[97633]: pgmap v17984: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:51:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:17.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:17.371448+0000) 2022-09-10T13:51:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:17.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:17.988642+0000) 2022-09-10T13:51:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:18.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:18.371629+0000) 2022-09-10T13:51:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:51:18.433+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:51:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:18 smithi104 ceph-mon[97044]: pgmap v17985: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:51:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:18 smithi110 ceph-mon[97633]: pgmap v17985: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:51:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:18.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:18.988880+0000) 2022-09-10T13:51:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:19.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:19.371909+0000) 2022-09-10T13:51:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:19.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:19.989205+0000) 2022-09-10T13:51:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:20.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:20.372092+0000) 2022-09-10T13:51:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:20 smithi104 ceph-mon[97044]: pgmap v17986: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:51:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:20 smithi110 ceph-mon[97633]: pgmap v17986: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:51:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:20.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:20.989407+0000) 2022-09-10T13:51:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:21.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:21.372297+0000) 2022-09-10T13:51:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:21.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:21.989593+0000) 2022-09-10T13:51:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:22.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:22.372516+0000) 2022-09-10T13:51:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:22 smithi104 ceph-mon[97044]: pgmap v17987: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:51:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:22 smithi110 ceph-mon[97633]: pgmap v17987: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:51:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:22.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:22.989774+0000) 2022-09-10T13:51:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:23.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:23.372682+0000) 2022-09-10T13:51:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:23.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:23.990026+0000) 2022-09-10T13:51:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:24.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:24.372857+0000) 2022-09-10T13:51:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:24 smithi104 ceph-mon[97044]: pgmap v17988: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:51:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:24 smithi110 ceph-mon[97633]: pgmap v17988: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:51:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:24.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:24.990281+0000) 2022-09-10T13:51:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:25.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:25.373052+0000) 2022-09-10T13:51:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:25.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:25.990453+0000) 2022-09-10T13:51:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:26.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:26.373208+0000) 2022-09-10T13:51:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:27 smithi104 ceph-mon[97044]: pgmap v17989: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:26.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:26.990676+0000) 2022-09-10T13:51:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:27 smithi110 ceph-mon[97633]: pgmap v17989: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:27.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:27.373333+0000) 2022-09-10T13:51:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:27.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:27.990867+0000) 2022-09-10T13:51:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:28.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:28.373561+0000) 2022-09-10T13:51:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:28 smithi104 ceph-mon[97044]: pgmap v17990: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:28 smithi110 ceph-mon[97633]: pgmap v17990: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:28.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:28.991084+0000) 2022-09-10T13:51:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:29.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:29.373819+0000) 2022-09-10T13:51:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:29.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:29.991256+0000) 2022-09-10T13:51:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:30.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:30.374039+0000) 2022-09-10T13:51:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:30 smithi104 ceph-mon[97044]: pgmap v17991: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:30 smithi110 ceph-mon[97633]: pgmap v17991: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:30.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:30.991439+0000) 2022-09-10T13:51:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:31.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:31.374291+0000) 2022-09-10T13:51:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:31.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:31.991583+0000) 2022-09-10T13:51:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:32.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:32.374501+0000) 2022-09-10T13:51:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:32 smithi104 ceph-mon[97044]: pgmap v17992: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:32 smithi110 ceph-mon[97633]: pgmap v17992: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:32.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:32.991796+0000) 2022-09-10T13:51:33.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:33.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:33.374702+0000) 2022-09-10T13:51:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:51:33.449+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:51:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:33.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:33.991976+0000) 2022-09-10T13:51:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:34.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:34.374871+0000) 2022-09-10T13:51:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:34 smithi104 ceph-mon[97044]: pgmap v17993: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:34 smithi110 ceph-mon[97633]: pgmap v17993: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:34.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:34.992203+0000) 2022-09-10T13:51:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:35.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:35.375108+0000) 2022-09-10T13:51:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:35.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:35.992398+0000) 2022-09-10T13:51:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:36.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:36.375340+0000) 2022-09-10T13:51:37.295 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:37 smithi104 ceph-mon[97044]: pgmap v17994: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:36.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:36.992575+0000) 2022-09-10T13:51:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:37 smithi110 ceph-mon[97633]: pgmap v17994: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:37.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:37.375596+0000) 2022-09-10T13:51:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:37.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:37.992815+0000) 2022-09-10T13:51:38.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:38.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:38.375810+0000) 2022-09-10T13:51:39.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:38 smithi104 ceph-mon[97044]: pgmap v17995: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:38 smithi110 ceph-mon[97633]: pgmap v17995: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:38.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:38.993052+0000) 2022-09-10T13:51:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:39.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:39.376041+0000) 2022-09-10T13:51:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:39.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:39.993212+0000) 2022-09-10T13:51:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:40.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:40.376327+0000) 2022-09-10T13:51:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:40 smithi104 ceph-mon[97044]: pgmap v17996: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:41.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:40 smithi110 ceph-mon[97633]: pgmap v17996: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:40.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:40.993445+0000) 2022-09-10T13:51:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:41.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:41.376588+0000) 2022-09-10T13:51:42.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:41.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:41.993708+0000) 2022-09-10T13:51:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:42.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:42.376767+0000) 2022-09-10T13:51:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:42 smithi104 ceph-mon[97044]: pgmap v17997: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:42 smithi110 ceph-mon[97633]: pgmap v17997: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:42.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:42.993938+0000) 2022-09-10T13:51:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:43.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:43.376934+0000) 2022-09-10T13:51:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:43.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:43.994218+0000) 2022-09-10T13:51:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:44.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:44.377164+0000) 2022-09-10T13:51:45.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:44 smithi104 ceph-mon[97044]: pgmap v17998: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:44 smithi110 ceph-mon[97633]: pgmap v17998: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:44.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:44.994404+0000) 2022-09-10T13:51:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:45.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:45.377392+0000) 2022-09-10T13:51:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:45.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:45.994672+0000) 2022-09-10T13:51:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:46.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:46.377644+0000) 2022-09-10T13:51:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:46.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:46.994925+0000) 2022-09-10T13:51:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:51:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:51:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:51:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:51:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:47 smithi110 ceph-mon[97633]: pgmap v17999: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:51:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:51:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:51:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:51:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:47 smithi104 ceph-mon[97044]: pgmap v17999: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:47.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:47.377860+0000) 2022-09-10T13:51:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:47.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:47.995230+0000) 2022-09-10T13:51:48.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:51:48.460+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:51:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:48.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:48.377998+0000) 2022-09-10T13:51:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:48 smithi104 ceph-mon[97044]: pgmap v18000: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:48 smithi110 ceph-mon[97633]: pgmap v18000: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:48.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:48.995397+0000) 2022-09-10T13:51:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:49.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:49.378204+0000) 2022-09-10T13:51:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:49.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:49.995628+0000) 2022-09-10T13:51:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:50.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:50.378423+0000) 2022-09-10T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:50 smithi104 ceph-mon[97044]: pgmap v18001: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:50 smithi110 ceph-mon[97633]: pgmap v18001: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:50.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:50.995902+0000) 2022-09-10T13:51:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:51.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:51.378624+0000) 2022-09-10T13:51:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:51.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:51.996204+0000) 2022-09-10T13:51:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:52.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:52.378769+0000) 2022-09-10T13:51:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:52 smithi104 ceph-mon[97044]: pgmap v18002: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:52 smithi110 ceph-mon[97633]: pgmap v18002: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:52.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:52.996478+0000) 2022-09-10T13:51:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:53.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:53.379022+0000) 2022-09-10T13:51:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:53.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:53.996733+0000) 2022-09-10T13:51:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:54.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:54.379215+0000) 2022-09-10T13:51:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:54 smithi104 ceph-mon[97044]: pgmap v18003: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:54 smithi110 ceph-mon[97633]: pgmap v18003: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:51:55.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:54.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:54.996986+0000) 2022-09-10T13:51:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:55.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:55.379449+0000) 2022-09-10T13:51:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:55.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:55.997284+0000) 2022-09-10T13:51:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:56.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:56.379678+0000) 2022-09-10T13:51:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:56.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:56.997542+0000) 2022-09-10T13:51:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:57 smithi110 ceph-mon[97633]: pgmap v18004: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:57.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:57 smithi104 ceph-mon[97044]: pgmap v18004: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:57.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:57.379840+0000) 2022-09-10T13:51:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:57.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:57.997707+0000) 2022-09-10T13:51:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:58.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:58.380032+0000) 2022-09-10T13:51:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:51:58 smithi104 ceph-mon[97044]: pgmap v18005: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:51:58 smithi110 ceph-mon[97633]: pgmap v18005: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:51:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:58.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:58.997948+0000) 2022-09-10T13:51:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:51:59.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:59.380256+0000) 2022-09-10T13:52:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:51:59.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:51:59.998235+0000) 2022-09-10T13:52:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:00.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:00.380434+0000) 2022-09-10T13:52:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:00 smithi104 ceph-mon[97044]: pgmap v18006: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:00 smithi110 ceph-mon[97633]: pgmap v18006: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:00.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:00.998420+0000) 2022-09-10T13:52:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:01.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:01.380641+0000) 2022-09-10T13:52:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:01.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:01.998661+0000) 2022-09-10T13:52:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:02.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:02.380866+0000) 2022-09-10T13:52:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:02 smithi104 ceph-mon[97044]: pgmap v18007: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:02 smithi110 ceph-mon[97633]: pgmap v18007: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:02.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:02.998951+0000) 2022-09-10T13:52:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:03.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:03.381082+0000) 2022-09-10T13:52:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:52:03.469+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:52:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:03.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:03.999229+0000) 2022-09-10T13:52:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:04.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:04.381362+0000) 2022-09-10T13:52:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:04 smithi104 ceph-mon[97044]: pgmap v18008: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:52:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:04 smithi110 ceph-mon[97633]: pgmap v18008: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:52:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:04.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:04.999495+0000) 2022-09-10T13:52:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:05.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:05.381604+0000) 2022-09-10T13:52:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:05.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:05.999779+0000) 2022-09-10T13:52:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:06.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:06.381838+0000) 2022-09-10T13:52:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:06.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:07.000046+0000) 2022-09-10T13:52:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:07 smithi110 ceph-mon[97633]: pgmap v18009: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:52:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:07 smithi104 ceph-mon[97044]: pgmap v18009: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:52:07.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:07.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:07.382072+0000) 2022-09-10T13:52:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:07.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:08.000269+0000) 2022-09-10T13:52:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:08.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:08.382370+0000) 2022-09-10T13:52:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:08.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:09.000500+0000) 2022-09-10T13:52:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:09 smithi110 ceph-mon[97633]: osdmap e1206: 12 total, 12 up, 12 in 2022-09-10T13:52:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:09 smithi110 ceph-mon[97633]: pgmap v18011: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:52:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:09 smithi104 ceph-mon[97044]: osdmap e1206: 12 total, 12 up, 12 in 2022-09-10T13:52:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:09 smithi104 ceph-mon[97044]: pgmap v18011: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:52:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:09.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:09.382554+0000) 2022-09-10T13:52:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:09.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:10.000702+0000) 2022-09-10T13:52:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:10.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:10.382734+0000) 2022-09-10T13:52:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:10 smithi104 ceph-mon[97044]: pgmap v18012: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:52:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:10 smithi110 ceph-mon[97633]: pgmap v18012: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:52:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:10.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:11.000923+0000) 2022-09-10T13:52:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:11.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:11.382992+0000) 2022-09-10T13:52:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:12.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:12.001191+0000) 2022-09-10T13:52:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:12 smithi110 ceph-mon[97633]: osdmap e1207: 12 total, 12 up, 12 in 2022-09-10T13:52:12.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:12 smithi104 ceph-mon[97044]: osdmap e1207: 12 total, 12 up, 12 in 2022-09-10T13:52:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:12.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:12.383214+0000) 2022-09-10T13:52:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:13.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:13.001392+0000) 2022-09-10T13:52:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:13 smithi110 ceph-mon[97633]: pgmap v18014: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:52:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:13 smithi104 ceph-mon[97044]: pgmap v18014: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:52:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:13.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:13.383428+0000) 2022-09-10T13:52:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:14.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:14.001582+0000) 2022-09-10T13:52:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:14.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:14.383596+0000) 2022-09-10T13:52:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:14 smithi104 ceph-mon[97044]: pgmap v18015: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:52:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:14 smithi110 ceph-mon[97633]: pgmap v18015: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:52:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:15.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:15.001806+0000) 2022-09-10T13:52:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:15.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:15.383729+0000) 2022-09-10T13:52:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:16.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:16.002006+0000) 2022-09-10T13:52:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:16.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:16.383915+0000) 2022-09-10T13:52:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:17.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:17.002209+0000) 2022-09-10T13:52:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:17 smithi110 ceph-mon[97633]: pgmap v18016: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:52:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:17 smithi104 ceph-mon[97044]: pgmap v18016: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:52:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:17.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:17.384042+0000) 2022-09-10T13:52:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:18.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:18.002408+0000) 2022-09-10T13:52:18.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:52:18.476+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:52:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:18.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:18.384241+0000) 2022-09-10T13:52:19.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:18 smithi104 ceph-mon[97044]: pgmap v18017: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:52:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:18 smithi110 ceph-mon[97633]: pgmap v18017: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:52:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:19.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:19.002585+0000) 2022-09-10T13:52:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:19.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:19.384392+0000) 2022-09-10T13:52:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:20.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:20.002839+0000) 2022-09-10T13:52:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:20.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:20.384606+0000) 2022-09-10T13:52:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:20 smithi104 ceph-mon[97044]: pgmap v18018: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:52:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:20 smithi110 ceph-mon[97633]: pgmap v18018: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:52:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:21.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:21.003020+0000) 2022-09-10T13:52:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:21.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:21.384831+0000) 2022-09-10T13:52:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:22.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:22.003213+0000) 2022-09-10T13:52:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:22.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:22.385056+0000) 2022-09-10T13:52:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:22 smithi104 ceph-mon[97044]: pgmap v18019: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:52:23.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:22 smithi110 ceph-mon[97633]: pgmap v18019: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:52:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:23.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:23.003457+0000) 2022-09-10T13:52:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:23.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:23.385325+0000) 2022-09-10T13:52:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:24.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:24.003704+0000) 2022-09-10T13:52:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:24.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:24.385500+0000) 2022-09-10T13:52:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:24 smithi104 ceph-mon[97044]: pgmap v18020: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:52:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:24 smithi110 ceph-mon[97633]: pgmap v18020: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:52:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:25.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:25.003928+0000) 2022-09-10T13:52:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:25.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:25.385774+0000) 2022-09-10T13:52:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:26.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:26.004153+0000) 2022-09-10T13:52:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:26.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:26.386028+0000) 2022-09-10T13:52:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:27.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:27.004342+0000) 2022-09-10T13:52:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:27 smithi110 ceph-mon[97633]: pgmap v18021: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:27 smithi104 ceph-mon[97044]: pgmap v18021: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:27.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:27.386261+0000) 2022-09-10T13:52:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:28.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:28.004496+0000) 2022-09-10T13:52:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:28.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:28.386426+0000) 2022-09-10T13:52:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:28 smithi104 ceph-mon[97044]: pgmap v18022: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:28 smithi110 ceph-mon[97633]: pgmap v18022: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:29.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:29.004699+0000) 2022-09-10T13:52:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:29.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:29.386634+0000) 2022-09-10T13:52:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:30.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:30.004932+0000) 2022-09-10T13:52:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:30.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:30.386904+0000) 2022-09-10T13:52:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:30 smithi104 ceph-mon[97044]: pgmap v18023: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:30 smithi110 ceph-mon[97633]: pgmap v18023: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:31.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:31.005169+0000) 2022-09-10T13:52:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:31.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:31.387132+0000) 2022-09-10T13:52:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:32.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:32.005402+0000) 2022-09-10T13:52:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:32.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:32.387369+0000) 2022-09-10T13:52:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:32 smithi104 ceph-mon[97044]: pgmap v18024: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:32 smithi110 ceph-mon[97633]: pgmap v18024: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:33.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:33.005628+0000) 2022-09-10T13:52:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:33.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:33.387602+0000) 2022-09-10T13:52:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:52:33.488+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:52:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:34.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:34.005874+0000) 2022-09-10T13:52:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:34.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:34.387839+0000) 2022-09-10T13:52:35.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:34 smithi104 ceph-mon[97044]: pgmap v18025: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:35.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:34 smithi110 ceph-mon[97633]: pgmap v18025: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:35.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:35.006126+0000) 2022-09-10T13:52:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:35.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:35.388138+0000) 2022-09-10T13:52:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:36.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:36.006390+0000) 2022-09-10T13:52:36.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:36.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:36.388336+0000) 2022-09-10T13:52:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:37.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:37.006534+0000) 2022-09-10T13:52:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:37 smithi110 ceph-mon[97633]: pgmap v18026: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:37 smithi104 ceph-mon[97044]: pgmap v18026: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:37.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:37.388586+0000) 2022-09-10T13:52:38.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:38.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:38.006700+0000) 2022-09-10T13:52:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:38.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:38.388789+0000) 2022-09-10T13:52:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:38 smithi104 ceph-mon[97044]: pgmap v18027: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:39.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:38 smithi110 ceph-mon[97633]: pgmap v18027: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:39.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:39.006895+0000) 2022-09-10T13:52:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:39.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:39.389024+0000) 2022-09-10T13:52:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:40.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:40.007171+0000) 2022-09-10T13:52:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:40.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:40.389276+0000) 2022-09-10T13:52:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:40 smithi104 ceph-mon[97044]: pgmap v18028: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:40 smithi110 ceph-mon[97633]: pgmap v18028: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:41.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:41.007344+0000) 2022-09-10T13:52:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:41.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:41.389463+0000) 2022-09-10T13:52:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:42.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:42.007572+0000) 2022-09-10T13:52:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:42.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:42.389690+0000) 2022-09-10T13:52:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:42 smithi104 ceph-mon[97044]: pgmap v18029: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:42 smithi110 ceph-mon[97633]: pgmap v18029: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:43.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:43.007881+0000) 2022-09-10T13:52:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:43.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:43.389929+0000) 2022-09-10T13:52:44.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:44.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:44.008149+0000) 2022-09-10T13:52:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:44.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:44.390165+0000) 2022-09-10T13:52:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:44 smithi104 ceph-mon[97044]: pgmap v18030: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:44 smithi110 ceph-mon[97633]: pgmap v18030: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:45.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:45.008300+0000) 2022-09-10T13:52:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:45.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:45.390362+0000) 2022-09-10T13:52:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:46.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:46.008552+0000) 2022-09-10T13:52:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:46.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:46.390648+0000) 2022-09-10T13:52:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:47.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:47.008769+0000) 2022-09-10T13:52:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:52:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:52:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:52:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:52:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:47 smithi110 ceph-mon[97633]: pgmap v18031: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:47.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:52:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:52:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:52:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:52:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:47 smithi104 ceph-mon[97044]: pgmap v18031: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:47.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:47.390846+0000) 2022-09-10T13:52:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:48.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:48.009002+0000) 2022-09-10T13:52:48.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:48.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:48.391145+0000) 2022-09-10T13:52:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:52:48.501+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:52:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:48 smithi104 ceph-mon[97044]: pgmap v18032: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:48 smithi110 ceph-mon[97633]: pgmap v18032: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:49.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:49.009236+0000) 2022-09-10T13:52:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:49.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:49.391307+0000) 2022-09-10T13:52:50.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:50.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:50.009475+0000) 2022-09-10T13:52:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:50.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:50.391543+0000) 2022-09-10T13:52:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:50 smithi104 ceph-mon[97044]: pgmap v18033: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:50 smithi110 ceph-mon[97633]: pgmap v18033: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:51.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:51.009660+0000) 2022-09-10T13:52:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:51.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:51.391783+0000) 2022-09-10T13:52:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:52.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:52.009932+0000) 2022-09-10T13:52:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:52.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:52.391942+0000) 2022-09-10T13:52:53.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:52 smithi104 ceph-mon[97044]: pgmap v18034: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:52 smithi110 ceph-mon[97633]: pgmap v18034: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:53.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:53.010183+0000) 2022-09-10T13:52:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:53.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:53.392217+0000) 2022-09-10T13:52:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:54.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:54.010383+0000) 2022-09-10T13:52:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:54.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:54.392478+0000) 2022-09-10T13:52:55.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:54 smithi104 ceph-mon[97044]: pgmap v18035: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:54 smithi110 ceph-mon[97633]: pgmap v18035: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:52:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:55.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:55.010596+0000) 2022-09-10T13:52:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:55.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:55.392749+0000) 2022-09-10T13:52:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:56.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:56.010832+0000) 2022-09-10T13:52:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:56.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:56.392943+0000) 2022-09-10T13:52:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:57.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:57.011080+0000) 2022-09-10T13:52:57.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:57 smithi110 ceph-mon[97633]: pgmap v18036: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:57 smithi104 ceph-mon[97044]: pgmap v18036: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:57.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:57.393209+0000) 2022-09-10T13:52:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:58.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:58.011300+0000) 2022-09-10T13:52:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:58.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:58.393412+0000) 2022-09-10T13:52:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:52:58 smithi104 ceph-mon[97044]: pgmap v18037: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:52:58 smithi110 ceph-mon[97633]: pgmap v18037: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:52:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:52:59.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:59.011525+0000) 2022-09-10T13:52:59.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:52:59.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:52:59.393644+0000) 2022-09-10T13:53:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:00.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:00.011727+0000) 2022-09-10T13:53:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:00.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:00.393834+0000) 2022-09-10T13:53:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:00 smithi104 ceph-mon[97044]: pgmap v18038: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:00 smithi110 ceph-mon[97633]: pgmap v18038: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:01.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:01.011963+0000) 2022-09-10T13:53:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:01.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:01.394022+0000) 2022-09-10T13:53:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:02.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:02.012217+0000) 2022-09-10T13:53:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:02.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:02.394209+0000) 2022-09-10T13:53:03.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:02 smithi104 ceph-mon[97044]: pgmap v18039: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:02 smithi110 ceph-mon[97633]: pgmap v18039: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:03.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:03.012387+0000) 2022-09-10T13:53:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:03.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:03.394386+0000) 2022-09-10T13:53:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:53:03.515+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:53:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:04.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:04.012572+0000) 2022-09-10T13:53:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:04.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:04.394577+0000) 2022-09-10T13:53:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:04 smithi104 ceph-mon[97044]: pgmap v18040: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:53:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:04 smithi110 ceph-mon[97633]: pgmap v18040: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:53:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:05.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:05.012755+0000) 2022-09-10T13:53:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:05.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:05.394799+0000) 2022-09-10T13:53:06.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:06.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:06.012989+0000) 2022-09-10T13:53:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:06.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:06.395056+0000) 2022-09-10T13:53:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:07.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:07.013223+0000) 2022-09-10T13:53:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:07 smithi110 ceph-mon[97633]: pgmap v18041: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:53:07.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:07 smithi104 ceph-mon[97044]: pgmap v18041: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:53:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:07.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:07.395269+0000) 2022-09-10T13:53:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:08.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:08.013440+0000) 2022-09-10T13:53:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:08.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:08.395489+0000) 2022-09-10T13:53:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:09.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:09.013612+0000) 2022-09-10T13:53:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:09 smithi110 ceph-mon[97633]: osdmap e1208: 12 total, 12 up, 12 in 2022-09-10T13:53:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:09 smithi110 ceph-mon[97633]: pgmap v18043: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:53:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:09 smithi104 ceph-mon[97044]: osdmap e1208: 12 total, 12 up, 12 in 2022-09-10T13:53:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:09 smithi104 ceph-mon[97044]: pgmap v18043: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:53:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:09.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:09.395695+0000) 2022-09-10T13:53:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:10.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:10.013866+0000) 2022-09-10T13:53:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:10.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:10.395924+0000) 2022-09-10T13:53:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:10 smithi104 ceph-mon[97044]: pgmap v18044: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:53:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:10 smithi110 ceph-mon[97633]: pgmap v18044: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:53:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:11.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:11.014129+0000) 2022-09-10T13:53:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:11.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:11.396188+0000) 2022-09-10T13:53:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:12.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:12.014364+0000) 2022-09-10T13:53:12.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:12 smithi110 ceph-mon[97633]: osdmap e1209: 12 total, 12 up, 12 in 2022-09-10T13:53:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:12 smithi104 ceph-mon[97044]: osdmap e1209: 12 total, 12 up, 12 in 2022-09-10T13:53:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:12.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:12.396392+0000) 2022-09-10T13:53:13.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:13.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:13.014614+0000) 2022-09-10T13:53:13.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:13 smithi110 ceph-mon[97633]: pgmap v18046: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:53:13.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:13 smithi104 ceph-mon[97044]: pgmap v18046: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:53:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:13.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:13.396603+0000) 2022-09-10T13:53:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:14.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:14.014774+0000) 2022-09-10T13:53:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:14.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:14.396884+0000) 2022-09-10T13:53:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:14 smithi104 ceph-mon[97044]: pgmap v18047: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:53:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:14 smithi110 ceph-mon[97633]: pgmap v18047: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:53:15.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:15.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:15.015025+0000) 2022-09-10T13:53:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:15.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:15.397120+0000) 2022-09-10T13:53:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:16.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:16.015250+0000) 2022-09-10T13:53:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:16.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:16.397321+0000) 2022-09-10T13:53:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:17.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:17.015498+0000) 2022-09-10T13:53:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:17 smithi110 ceph-mon[97633]: pgmap v18048: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:53:17.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:17 smithi104 ceph-mon[97044]: pgmap v18048: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:53:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:17.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:17.397543+0000) 2022-09-10T13:53:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:18.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:18.015700+0000) 2022-09-10T13:53:18.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:18.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:18.397758+0000) 2022-09-10T13:53:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:53:18.526+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:53:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:18 smithi104 ceph-mon[97044]: pgmap v18049: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:53:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:18 smithi110 ceph-mon[97633]: pgmap v18049: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:53:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:19.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:19.015958+0000) 2022-09-10T13:53:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:19.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:19.397930+0000) 2022-09-10T13:53:20.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:20.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:20.016156+0000) 2022-09-10T13:53:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:20.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:20.398159+0000) 2022-09-10T13:53:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:20 smithi104 ceph-mon[97044]: pgmap v18050: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:53:21.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:20 smithi110 ceph-mon[97633]: pgmap v18050: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:53:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:21.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:21.016320+0000) 2022-09-10T13:53:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:21.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:21.398389+0000) 2022-09-10T13:53:22.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:22.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:22.016564+0000) 2022-09-10T13:53:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:22.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:22.398648+0000) 2022-09-10T13:53:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:22 smithi104 ceph-mon[97044]: pgmap v18051: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:53:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:22 smithi110 ceph-mon[97633]: pgmap v18051: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:53:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:23.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:23.016812+0000) 2022-09-10T13:53:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:23.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:23.398901+0000) 2022-09-10T13:53:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:24.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:24.017036+0000) 2022-09-10T13:53:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:24.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:24.399125+0000) 2022-09-10T13:53:25.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:24 smithi104 ceph-mon[97044]: pgmap v18052: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:53:25.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:24 smithi110 ceph-mon[97633]: pgmap v18052: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:53:25.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:25.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:25.017271+0000) 2022-09-10T13:53:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:25.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:25.399321+0000) 2022-09-10T13:53:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:26.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:26.017527+0000) 2022-09-10T13:53:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:26.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:26.399540+0000) 2022-09-10T13:53:27.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:27.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:27.017704+0000) 2022-09-10T13:53:27.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:27 smithi110 ceph-mon[97633]: pgmap v18053: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:27.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:27 smithi104 ceph-mon[97044]: pgmap v18053: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:27.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:27.399780+0000) 2022-09-10T13:53:28.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:28.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:28.017970+0000) 2022-09-10T13:53:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:28.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:28.400039+0000) 2022-09-10T13:53:29.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:28 smithi104 ceph-mon[97044]: pgmap v18054: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:28 smithi110 ceph-mon[97633]: pgmap v18054: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:29.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:29.018215+0000) 2022-09-10T13:53:29.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:29.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:29.400347+0000) 2022-09-10T13:53:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:30.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:30.018452+0000) 2022-09-10T13:53:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:30.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:30.400619+0000) 2022-09-10T13:53:31.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:30 smithi104 ceph-mon[97044]: pgmap v18055: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:31.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:30 smithi110 ceph-mon[97633]: pgmap v18055: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:31.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:31.018666+0000) 2022-09-10T13:53:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:31.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:31.400847+0000) 2022-09-10T13:53:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:32.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:32.018844+0000) 2022-09-10T13:53:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:32.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:32.401115+0000) 2022-09-10T13:53:33.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:32 smithi104 ceph-mon[97044]: pgmap v18056: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:33.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:32 smithi110 ceph-mon[97633]: pgmap v18056: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:33.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:33.019087+0000) 2022-09-10T13:53:33.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:53:33.540+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:53:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:33.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:33.401306+0000) 2022-09-10T13:53:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:34.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:34.019388+0000) 2022-09-10T13:53:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:34.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:34.401515+0000) 2022-09-10T13:53:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:34 smithi104 ceph-mon[97044]: pgmap v18057: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:34 smithi110 ceph-mon[97633]: pgmap v18057: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:35.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:35.019631+0000) 2022-09-10T13:53:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:35.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:35.401703+0000) 2022-09-10T13:53:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:36.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:36.019884+0000) 2022-09-10T13:53:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:36.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:36.401902+0000) 2022-09-10T13:53:37.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:37.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:37.020164+0000) 2022-09-10T13:53:37.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:37 smithi110 ceph-mon[97633]: pgmap v18058: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:37.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:37 smithi104 ceph-mon[97044]: pgmap v18058: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:37.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:37.402022+0000) 2022-09-10T13:53:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:38.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:38.020335+0000) 2022-09-10T13:53:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:38.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:38.402245+0000) 2022-09-10T13:53:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:38 smithi104 ceph-mon[97044]: pgmap v18059: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:38 smithi110 ceph-mon[97633]: pgmap v18059: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:39.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:39.020534+0000) 2022-09-10T13:53:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:39.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:39.402466+0000) 2022-09-10T13:53:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:40.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:40.020715+0000) 2022-09-10T13:53:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:40.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:40.402709+0000) 2022-09-10T13:53:41.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:40 smithi104 ceph-mon[97044]: pgmap v18060: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:40 smithi110 ceph-mon[97633]: pgmap v18060: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:41.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:41.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:41.020934+0000) 2022-09-10T13:53:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:41.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:41.402989+0000) 2022-09-10T13:53:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:42.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:42.021151+0000) 2022-09-10T13:53:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:42.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:42.403260+0000) 2022-09-10T13:53:43.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:42 smithi104 ceph-mon[97044]: pgmap v18061: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:42 smithi110 ceph-mon[97633]: pgmap v18061: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:43.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:43.021324+0000) 2022-09-10T13:53:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:43.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:43.403493+0000) 2022-09-10T13:53:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:44.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:44.021558+0000) 2022-09-10T13:53:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:44.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:44.403675+0000) 2022-09-10T13:53:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:44 smithi104 ceph-mon[97044]: pgmap v18062: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:45.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:44 smithi110 ceph-mon[97633]: pgmap v18062: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:45.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:45.021795+0000) 2022-09-10T13:53:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:45.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:45.403881+0000) 2022-09-10T13:53:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:46.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:46.022009+0000) 2022-09-10T13:53:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:46.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:46.404082+0000) 2022-09-10T13:53:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:47.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:47.022217+0000) 2022-09-10T13:53:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:53:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:53:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:53:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:53:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:47 smithi110 ceph-mon[97633]: pgmap v18063: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:53:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:53:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:53:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:47 smithi104 ceph-mon[97044]: pgmap v18063: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:47.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:47.404309+0000) 2022-09-10T13:53:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:48.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:48.022429+0000) 2022-09-10T13:53:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:48.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:48.404541+0000) 2022-09-10T13:53:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:53:48.552+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:53:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:48 smithi104 ceph-mon[97044]: pgmap v18064: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:49.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:48 smithi110 ceph-mon[97633]: pgmap v18064: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:49.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:49.022604+0000) 2022-09-10T13:53:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:49.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:49.404799+0000) 2022-09-10T13:53:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:50.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:50.022869+0000) 2022-09-10T13:53:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:50.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:50.405068+0000) 2022-09-10T13:53:51.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:50 smithi104 ceph-mon[97044]: pgmap v18065: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:50 smithi110 ceph-mon[97633]: pgmap v18065: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:51.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:51.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:51.023155+0000) 2022-09-10T13:53:51.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:51.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:51.405330+0000) 2022-09-10T13:53:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:52.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:52.023377+0000) 2022-09-10T13:53:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:52.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:52.405520+0000) 2022-09-10T13:53:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:52 smithi104 ceph-mon[97044]: pgmap v18066: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:52 smithi110 ceph-mon[97633]: pgmap v18066: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:53.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:53.023583+0000) 2022-09-10T13:53:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:53.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:53.405667+0000) 2022-09-10T13:53:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:54.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:54.023834+0000) 2022-09-10T13:53:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:54.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:54.405923+0000) 2022-09-10T13:53:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:54 smithi104 ceph-mon[97044]: pgmap v18067: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:55.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:54 smithi110 ceph-mon[97633]: pgmap v18067: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:53:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:55.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:55.024079+0000) 2022-09-10T13:53:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:55.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:55.406194+0000) 2022-09-10T13:53:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:56.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:56.024353+0000) 2022-09-10T13:53:56.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:56.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:56.406415+0000) 2022-09-10T13:53:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:57.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:57.024551+0000) 2022-09-10T13:53:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:57 smithi110 ceph-mon[97633]: pgmap v18068: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:57.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:57 smithi104 ceph-mon[97044]: pgmap v18068: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:57.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:57.406553+0000) 2022-09-10T13:53:58.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:58.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:58.024737+0000) 2022-09-10T13:53:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:58.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:58.406793+0000) 2022-09-10T13:53:59.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:53:58 smithi104 ceph-mon[97044]: pgmap v18069: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:59.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:53:58 smithi110 ceph-mon[97633]: pgmap v18069: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:53:59.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:53:59.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:59.024968+0000) 2022-09-10T13:53:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:53:59.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:53:59.407047+0000) 2022-09-10T13:54:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:00.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:00.025214+0000) 2022-09-10T13:54:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:00.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:00.407273+0000) 2022-09-10T13:54:01.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:00 smithi104 ceph-mon[97044]: pgmap v18070: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:01.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:00 smithi110 ceph-mon[97633]: pgmap v18070: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:01.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:01.025466+0000) 2022-09-10T13:54:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:01.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:01.407509+0000) 2022-09-10T13:54:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:02.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:02.025656+0000) 2022-09-10T13:54:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:02.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:02.407740+0000) 2022-09-10T13:54:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:02 smithi104 ceph-mon[97044]: pgmap v18071: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:03.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:02 smithi110 ceph-mon[97633]: pgmap v18071: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:03.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:03.025881+0000) 2022-09-10T13:54:03.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:03.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:03.407938+0000) 2022-09-10T13:54:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:54:03.553+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:54:04.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:04.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:04.026061+0000) 2022-09-10T13:54:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:04.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:04.408135+0000) 2022-09-10T13:54:05.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:04 smithi104 ceph-mon[97044]: pgmap v18072: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:54:05.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:04 smithi110 ceph-mon[97633]: pgmap v18072: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:54:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:05.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:05.026264+0000) 2022-09-10T13:54:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:05.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:05.408360+0000) 2022-09-10T13:54:06.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:06.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:06.026505+0000) 2022-09-10T13:54:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:06.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:06.408640+0000) 2022-09-10T13:54:07.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:07.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:07.026689+0000) 2022-09-10T13:54:07.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:07 smithi110 ceph-mon[97633]: pgmap v18073: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:54:07.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:07 smithi104 ceph-mon[97044]: pgmap v18073: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:54:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:07.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:07.408890+0000) 2022-09-10T13:54:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:08.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:08.026956+0000) 2022-09-10T13:54:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:08.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:08.409175+0000) 2022-09-10T13:54:09.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:09.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:09.027189+0000) 2022-09-10T13:54:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:09 smithi110 ceph-mon[97633]: osdmap e1210: 12 total, 12 up, 12 in 2022-09-10T13:54:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:09 smithi110 ceph-mon[97633]: pgmap v18075: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:54:09.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:09 smithi104 ceph-mon[97044]: osdmap e1210: 12 total, 12 up, 12 in 2022-09-10T13:54:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:09 smithi104 ceph-mon[97044]: pgmap v18075: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:54:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:09.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:09.409340+0000) 2022-09-10T13:54:10.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:10.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:10.027329+0000) 2022-09-10T13:54:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:10.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:10.409573+0000) 2022-09-10T13:54:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:10 smithi104 ceph-mon[97044]: pgmap v18076: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:54:11.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:10 smithi110 ceph-mon[97633]: pgmap v18076: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:54:11.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:11.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:11.027522+0000) 2022-09-10T13:54:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:11.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:11.409799+0000) 2022-09-10T13:54:12.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:12.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:12.027672+0000) 2022-09-10T13:54:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:12 smithi110 ceph-mon[97633]: osdmap e1211: 12 total, 12 up, 12 in 2022-09-10T13:54:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:12 smithi104 ceph-mon[97044]: osdmap e1211: 12 total, 12 up, 12 in 2022-09-10T13:54:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:12.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:12.410025+0000) 2022-09-10T13:54:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:13.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:13.027887+0000) 2022-09-10T13:54:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:13 smithi110 ceph-mon[97633]: pgmap v18078: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:54:13.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:13 smithi104 ceph-mon[97044]: pgmap v18078: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T13:54:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:13.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:13.410164+0000) 2022-09-10T13:54:14.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:14.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:14.028081+0000) 2022-09-10T13:54:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:14.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:14.410412+0000) 2022-09-10T13:54:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:14 smithi104 ceph-mon[97044]: pgmap v18079: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:54:15.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:14 smithi110 ceph-mon[97633]: pgmap v18079: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T13:54:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:15.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:15.028305+0000) 2022-09-10T13:54:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:15.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:15.410660+0000) 2022-09-10T13:54:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:16.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:16.028596+0000) 2022-09-10T13:54:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:16.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:16.410917+0000) 2022-09-10T13:54:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:17.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:17.028848+0000) 2022-09-10T13:54:17.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:17 smithi110 ceph-mon[97633]: pgmap v18080: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:54:17.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:17 smithi104 ceph-mon[97044]: pgmap v18080: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T13:54:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:17.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:17.411192+0000) 2022-09-10T13:54:18.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:18.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:18.029124+0000) 2022-09-10T13:54:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:54:18.569+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:54:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:18.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:18.411443+0000) 2022-09-10T13:54:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:18 smithi104 ceph-mon[97044]: pgmap v18081: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:54:19.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:18 smithi110 ceph-mon[97633]: pgmap v18081: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:54:19.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:19.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:19.029378+0000) 2022-09-10T13:54:19.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:19.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:19.411689+0000) 2022-09-10T13:54:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:20.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:20.029601+0000) 2022-09-10T13:54:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:20.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:20.411882+0000) 2022-09-10T13:54:21.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:20 smithi104 ceph-mon[97044]: pgmap v18082: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:54:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:20 smithi110 ceph-mon[97633]: pgmap v18082: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:54:21.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:21.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:21.029850+0000) 2022-09-10T13:54:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:21.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:21.412081+0000) 2022-09-10T13:54:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:22.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:22.030041+0000) 2022-09-10T13:54:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:22.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:22.412288+0000) 2022-09-10T13:54:23.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:22 smithi104 ceph-mon[97044]: pgmap v18083: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:54:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:22 smithi110 ceph-mon[97633]: pgmap v18083: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T13:54:23.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:23.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:23.030243+0000) 2022-09-10T13:54:23.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:23.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:23.412451+0000) 2022-09-10T13:54:24.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:24.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:24.030497+0000) 2022-09-10T13:54:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:24.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:24.412745+0000) 2022-09-10T13:54:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:24 smithi104 ceph-mon[97044]: pgmap v18084: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:54:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:24 smithi110 ceph-mon[97633]: pgmap v18084: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T13:54:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:25.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:25.030679+0000) 2022-09-10T13:54:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:25.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:25.412997+0000) 2022-09-10T13:54:26.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:26.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:26.030944+0000) 2022-09-10T13:54:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:26.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:26.413201+0000) 2022-09-10T13:54:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:27.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:27.031185+0000) 2022-09-10T13:54:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:27 smithi110 ceph-mon[97633]: pgmap v18085: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:27.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:27 smithi104 ceph-mon[97044]: pgmap v18085: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:27.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:27.413422+0000) 2022-09-10T13:54:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:28.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:28.031428+0000) 2022-09-10T13:54:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:28.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:28.413559+0000) 2022-09-10T13:54:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:28 smithi104 ceph-mon[97044]: pgmap v18086: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:29.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:28 smithi110 ceph-mon[97633]: pgmap v18086: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:29.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:29.031679+0000) 2022-09-10T13:54:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:29.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:29.413772+0000) 2022-09-10T13:54:30.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:30.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:30.031932+0000) 2022-09-10T13:54:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:30.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:30.414033+0000) 2022-09-10T13:54:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:30 smithi104 ceph-mon[97044]: pgmap v18087: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:30 smithi110 ceph-mon[97633]: pgmap v18087: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:31.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:31.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:31.032123+0000) 2022-09-10T13:54:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:31.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:31.414232+0000) 2022-09-10T13:54:32.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:32.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:32.032267+0000) 2022-09-10T13:54:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:32.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:32.414439+0000) 2022-09-10T13:54:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:32 smithi104 ceph-mon[97044]: pgmap v18088: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:32 smithi110 ceph-mon[97633]: pgmap v18088: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:33.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:33.032460+0000) 2022-09-10T13:54:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:33.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:33.414687+0000) 2022-09-10T13:54:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:54:33.575+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:54:34.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:34.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:34.032672+0000) 2022-09-10T13:54:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:34.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:34.414898+0000) 2022-09-10T13:54:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:34 smithi104 ceph-mon[97044]: pgmap v18089: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:34 smithi110 ceph-mon[97633]: pgmap v18089: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:35.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:35.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:35.032899+0000) 2022-09-10T13:54:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:35.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:35.415148+0000) 2022-09-10T13:54:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:36.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:36.033087+0000) 2022-09-10T13:54:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:36.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:36.415340+0000) 2022-09-10T13:54:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:37.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:37.033286+0000) 2022-09-10T13:54:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:37 smithi110 ceph-mon[97633]: pgmap v18090: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:37.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:37 smithi104 ceph-mon[97044]: pgmap v18090: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:37.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:37.415519+0000) 2022-09-10T13:54:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:38.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:38.033535+0000) 2022-09-10T13:54:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:38.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:38.415752+0000) 2022-09-10T13:54:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:38 smithi104 ceph-mon[97044]: pgmap v18091: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:38 smithi110 ceph-mon[97633]: pgmap v18091: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:39.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:39.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:39.033776+0000) 2022-09-10T13:54:39.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:39.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:39.415913+0000) 2022-09-10T13:54:40.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:40.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:40.034030+0000) 2022-09-10T13:54:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:40.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:40.416163+0000) 2022-09-10T13:54:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:40 smithi104 ceph-mon[97044]: pgmap v18092: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:41.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:40 smithi110 ceph-mon[97633]: pgmap v18092: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:41.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:41.034275+0000) 2022-09-10T13:54:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:41.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:41.416370+0000) 2022-09-10T13:54:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:42.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:42.034465+0000) 2022-09-10T13:54:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:42.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:42.416596+0000) 2022-09-10T13:54:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:42 smithi104 ceph-mon[97044]: pgmap v18093: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:43.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:42 smithi110 ceph-mon[97633]: pgmap v18093: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:43.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:43.034666+0000) 2022-09-10T13:54:43.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:43.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:43.416846+0000) 2022-09-10T13:54:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:44.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:44.034880+0000) 2022-09-10T13:54:44.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:44.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:44.417062+0000) 2022-09-10T13:54:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:44 smithi104 ceph-mon[97044]: pgmap v18094: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:44 smithi110 ceph-mon[97633]: pgmap v18094: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:45.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:45.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:45.035055+0000) 2022-09-10T13:54:45.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:45.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:45.417298+0000) 2022-09-10T13:54:46.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:46.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:46.035260+0000) 2022-09-10T13:54:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:46.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:46.417548+0000) 2022-09-10T13:54:47.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:47.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:47.035450+0000) 2022-09-10T13:54:47.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:54:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:54:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:54:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:54:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:47 smithi110 ceph-mon[97633]: pgmap v18095: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:54:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:54:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:54:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:54:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:47 smithi104 ceph-mon[97044]: pgmap v18095: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:47.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:47.417744+0000) 2022-09-10T13:54:48.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:48.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:48.035665+0000) 2022-09-10T13:54:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:48.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:48.417940+0000) 2022-09-10T13:54:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:54:48.588+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:54:49.045 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:48 smithi104 ceph-mon[97044]: pgmap v18096: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:48 smithi110 ceph-mon[97633]: pgmap v18096: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:54:49.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:49.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:49.035898+0000) 2022-09-10T13:54:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:49.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:49.418230+0000) 2022-09-10T13:54:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:50.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:50.036207+0000) 2022-09-10T13:54:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:50.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:50.418437+0000) 2022-09-10T13:54:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:50 smithi104 ceph-mon[97044]: pgmap v18097: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:51.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:50 smithi110 ceph-mon[97633]: pgmap v18097: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:51.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:51.036362+0000) 2022-09-10T13:54:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:51.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:51.418668+0000) 2022-09-10T13:54:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:52.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:52.036512+0000) 2022-09-10T13:54:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:52.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:52.418914+0000) 2022-09-10T13:54:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:52 smithi104 ceph-mon[97044]: pgmap v18098: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:54:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:52 smithi110 ceph-mon[97633]: pgmap v18098: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:54:53.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:53.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:53.036722+0000) 2022-09-10T13:54:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:53.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:53.419153+0000) 2022-09-10T13:54:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:54.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:54.036917+0000) 2022-09-10T13:54:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:54.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:54.419331+0000) 2022-09-10T13:54:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:55.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:55.037159+0000) 2022-09-10T13:54:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:55.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:55.419519+0000) 2022-09-10T13:54:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:55 smithi104 ceph-mon[97044]: pgmap v18099: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:56.156 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:55 smithi110 ceph-mon[97633]: pgmap v18099: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:54:56.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:56.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:56.037392+0000) 2022-09-10T13:54:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:56.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:56.419784+0000) 2022-09-10T13:54:57.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:57.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:57.037533+0000) 2022-09-10T13:54:57.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:57.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:57.419962+0000) 2022-09-10T13:54:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:58.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:58.037784+0000) 2022-09-10T13:54:58.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:54:58 smithi110 ceph-mon[97633]: pgmap v18100: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:54:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:54:58 smithi104 ceph-mon[97044]: pgmap v18100: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:54:58.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:58.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:58.420203+0000) 2022-09-10T13:54:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:54:59.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:59.038048+0000) 2022-09-10T13:54:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:54:59.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:54:59.420346+0000) 2022-09-10T13:55:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:00.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:00.038265+0000) 2022-09-10T13:55:00.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:00 smithi110 ceph-mon[97633]: pgmap v18101: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:55:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:00 smithi104 ceph-mon[97044]: pgmap v18101: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:55:00.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:00.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:00.420579+0000) 2022-09-10T13:55:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:01.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:01.038400+0000) 2022-09-10T13:55:01.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:01.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:01.420797+0000) 2022-09-10T13:55:02.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:02.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:02.038544+0000) 2022-09-10T13:55:02.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:02 smithi110 ceph-mon[97633]: pgmap v18102: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:02.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:02 smithi104 ceph-mon[97044]: pgmap v18102: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:02.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:02.421047+0000) 2022-09-10T13:55:03.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:03.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:03.038756+0000) 2022-09-10T13:55:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:55:03.604+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:55:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:03.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:03.421294+0000) 2022-09-10T13:55:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:04.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:04.039040+0000) 2022-09-10T13:55:04.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:04 smithi110 ceph-mon[97633]: pgmap v18103: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:04 smithi104 ceph-mon[97044]: pgmap v18103: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:04.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:04.421511+0000) 2022-09-10T13:55:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:05.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:05.039212+0000) 2022-09-10T13:55:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:05.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:05.421703+0000) 2022-09-10T13:55:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:06.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:06.039436+0000) 2022-09-10T13:55:06.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:06 smithi110 ceph-mon[97633]: pgmap v18104: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:55:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:06 smithi104 ceph-mon[97044]: pgmap v18104: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:55:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:06.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:06.421890+0000) 2022-09-10T13:55:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:07.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:07.039692+0000) 2022-09-10T13:55:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:07.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:07.422159+0000) 2022-09-10T13:55:08.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:08.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:08.039958+0000) 2022-09-10T13:55:08.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:08 smithi110 ceph-mon[97633]: pgmap v18105: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:55:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:08 smithi104 ceph-mon[97044]: pgmap v18105: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:55:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:08.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:08.422357+0000) 2022-09-10T13:55:09.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:09.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:09.040204+0000) 2022-09-10T13:55:09.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:09 smithi110 ceph-mon[97633]: osdmap e1212: 12 total, 12 up, 12 in 2022-09-10T13:55:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:09 smithi104 ceph-mon[97044]: osdmap e1212: 12 total, 12 up, 12 in 2022-09-10T13:55:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:09.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:09.422606+0000) 2022-09-10T13:55:10.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:10.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:10.040446+0000) 2022-09-10T13:55:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:10 smithi110 ceph-mon[97633]: pgmap v18107: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:55:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:10 smithi104 ceph-mon[97044]: pgmap v18107: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:55:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:10.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:10.422837+0000) 2022-09-10T13:55:11.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:11.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:11.040621+0000) 2022-09-10T13:55:11.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:11.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:11.423069+0000) 2022-09-10T13:55:12.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:12.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:12.040802+0000) 2022-09-10T13:55:12.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:12 smithi110 ceph-mon[97633]: pgmap v18108: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:55:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:12 smithi104 ceph-mon[97044]: pgmap v18108: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:55:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:12.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:12.423287+0000) 2022-09-10T13:55:13.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:13.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:13.040998+0000) 2022-09-10T13:55:13.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:13.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:13.423515+0000) 2022-09-10T13:55:14.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:14.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:14.041180+0000) 2022-09-10T13:55:14.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:14 smithi110 ceph-mon[97633]: pgmap v18109: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:55:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:14 smithi104 ceph-mon[97044]: pgmap v18109: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:55:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:14.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:14.423715+0000) 2022-09-10T13:55:15.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:15.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:15.041384+0000) 2022-09-10T13:55:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:15.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:15.423960+0000) 2022-09-10T13:55:16.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:16.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:16.041655+0000) 2022-09-10T13:55:16.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:16 smithi110 ceph-mon[97633]: pgmap v18110: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:55:16.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:16 smithi110 ceph-mon[97633]: osdmap e1213: 12 total, 12 up, 12 in 2022-09-10T13:55:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:16 smithi104 ceph-mon[97044]: pgmap v18110: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:55:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:16 smithi104 ceph-mon[97044]: osdmap e1213: 12 total, 12 up, 12 in 2022-09-10T13:55:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:16.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:16.424131+0000) 2022-09-10T13:55:17.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:17.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:17.041906+0000) 2022-09-10T13:55:17.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:17.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:17.424337+0000) 2022-09-10T13:55:18.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:18.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:18.042172+0000) 2022-09-10T13:55:18.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:18 smithi110 ceph-mon[97633]: pgmap v18112: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 239 B/s wr, 11 op/s 2022-09-10T13:55:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:18 smithi104 ceph-mon[97044]: pgmap v18112: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 239 B/s wr, 11 op/s 2022-09-10T13:55:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:18.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:18.424557+0000) 2022-09-10T13:55:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:55:18.615+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:55:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:19.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:19.042418+0000) 2022-09-10T13:55:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:19.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:19.424797+0000) 2022-09-10T13:55:20.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:20.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:20.042614+0000) 2022-09-10T13:55:20.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:20 smithi110 ceph-mon[97633]: pgmap v18113: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:55:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:20 smithi104 ceph-mon[97044]: pgmap v18113: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:55:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:20.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:20.425025+0000) 2022-09-10T13:55:21.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:21.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:21.042849+0000) 2022-09-10T13:55:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:21.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:21.425234+0000) 2022-09-10T13:55:22.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:22.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:22.043168+0000) 2022-09-10T13:55:22.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:22 smithi110 ceph-mon[97633]: pgmap v18114: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:55:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:22 smithi104 ceph-mon[97044]: pgmap v18114: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:55:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:22.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:22.425526+0000) 2022-09-10T13:55:23.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:23.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:23.043374+0000) 2022-09-10T13:55:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:23.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:23.425777+0000) 2022-09-10T13:55:24.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:24.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:24.043585+0000) 2022-09-10T13:55:24.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:24 smithi110 ceph-mon[97633]: pgmap v18115: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:55:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:24 smithi104 ceph-mon[97044]: pgmap v18115: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:55:24.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:24.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:24.426032+0000) 2022-09-10T13:55:25.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:25.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:25.043851+0000) 2022-09-10T13:55:25.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:25.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:25.426280+0000) 2022-09-10T13:55:26.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:26.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:26.044073+0000) 2022-09-10T13:55:26.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:26 smithi110 ceph-mon[97633]: pgmap v18116: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:55:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:26 smithi104 ceph-mon[97044]: pgmap v18116: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:55:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:26.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:26.426452+0000) 2022-09-10T13:55:27.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:27.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:27.044347+0000) 2022-09-10T13:55:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:27.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:27.426689+0000) 2022-09-10T13:55:28.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:28.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:28.044551+0000) 2022-09-10T13:55:28.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:28 smithi110 ceph-mon[97633]: pgmap v18117: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:55:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:28 smithi104 ceph-mon[97044]: pgmap v18117: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:55:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:28.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:28.426944+0000) 2022-09-10T13:55:29.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:29.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:29.044817+0000) 2022-09-10T13:55:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:29.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:29.427175+0000) 2022-09-10T13:55:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:30.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:30.045012+0000) 2022-09-10T13:55:30.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:30 smithi110 ceph-mon[97633]: pgmap v18118: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:30 smithi104 ceph-mon[97044]: pgmap v18118: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:30.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:30.427414+0000) 2022-09-10T13:55:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:31.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:31.045277+0000) 2022-09-10T13:55:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:31.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:31.427623+0000) 2022-09-10T13:55:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:32.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:32.045492+0000) 2022-09-10T13:55:32.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:32 smithi110 ceph-mon[97633]: pgmap v18119: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:32 smithi104 ceph-mon[97044]: pgmap v18119: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:32.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:32.427835+0000) 2022-09-10T13:55:33.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:33.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:33.045620+0000) 2022-09-10T13:55:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:55:33.625+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:55:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:33.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:33.428040+0000) 2022-09-10T13:55:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:34.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:34.045797+0000) 2022-09-10T13:55:34.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:34 smithi110 ceph-mon[97633]: pgmap v18120: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:34 smithi104 ceph-mon[97044]: pgmap v18120: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:34.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:34.428229+0000) 2022-09-10T13:55:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:35.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:35.045940+0000) 2022-09-10T13:55:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:35.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:35.428440+0000) 2022-09-10T13:55:36.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:36.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:36.046135+0000) 2022-09-10T13:55:36.406 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:36 smithi110 ceph-mon[97633]: pgmap v18121: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:36 smithi104 ceph-mon[97044]: pgmap v18121: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:36.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:36.428666+0000) 2022-09-10T13:55:37.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:37.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:37.046380+0000) 2022-09-10T13:55:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:37.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:37.428811+0000) 2022-09-10T13:55:38.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:38.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:38.046624+0000) 2022-09-10T13:55:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:38 smithi104 ceph-mon[97044]: pgmap v18122: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:38 smithi110 ceph-mon[97633]: pgmap v18122: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:38.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:38.428988+0000) 2022-09-10T13:55:39.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:39.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:39.046772+0000) 2022-09-10T13:55:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:39.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:39.429173+0000) 2022-09-10T13:55:40.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:40.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:40.046998+0000) 2022-09-10T13:55:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:40 smithi104 ceph-mon[97044]: pgmap v18123: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:40 smithi110 ceph-mon[97633]: pgmap v18123: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:40.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:40.429351+0000) 2022-09-10T13:55:41.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:41.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:41.047253+0000) 2022-09-10T13:55:41.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:41.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:41.429568+0000) 2022-09-10T13:55:42.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:42.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:42.047495+0000) 2022-09-10T13:55:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:42 smithi104 ceph-mon[97044]: pgmap v18124: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:42.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:42 smithi110 ceph-mon[97633]: pgmap v18124: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:42.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:42.429731+0000) 2022-09-10T13:55:43.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:43.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:43.047742+0000) 2022-09-10T13:55:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:43.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:43.429892+0000) 2022-09-10T13:55:44.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:44.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:44.048010+0000) 2022-09-10T13:55:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:44 smithi104 ceph-mon[97044]: pgmap v18125: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:44 smithi110 ceph-mon[97633]: pgmap v18125: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:44.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:44.430141+0000) 2022-09-10T13:55:45.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:45.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:45.048179+0000) 2022-09-10T13:55:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:45.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:45.430358+0000) 2022-09-10T13:55:46.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:46.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:46.048359+0000) 2022-09-10T13:55:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:46 smithi104 ceph-mon[97044]: pgmap v18126: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:46 smithi110 ceph-mon[97633]: pgmap v18126: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:46.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:46.430553+0000) 2022-09-10T13:55:47.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:47.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:47.048612+0000) 2022-09-10T13:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:55:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:55:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:55:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:55:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:55:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:55:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:55:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:47.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:47.430715+0000) 2022-09-10T13:55:48.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:48.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:48.048839+0000) 2022-09-10T13:55:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:48 smithi104 ceph-mon[97044]: pgmap v18127: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:48 smithi110 ceph-mon[97633]: pgmap v18127: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:48.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:48.430926+0000) 2022-09-10T13:55:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:55:48.634+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:55:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:49.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:49.049126+0000) 2022-09-10T13:55:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:49.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:49.431187+0000) 2022-09-10T13:55:50.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:50.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:50.049316+0000) 2022-09-10T13:55:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:50 smithi104 ceph-mon[97044]: pgmap v18128: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:50.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:50 smithi110 ceph-mon[97633]: pgmap v18128: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:50.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:50.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:50.431397+0000) 2022-09-10T13:55:51.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:51.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:51.049524+0000) 2022-09-10T13:55:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:51.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:51.431594+0000) 2022-09-10T13:55:52.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:52.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:52.049718+0000) 2022-09-10T13:55:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:52 smithi104 ceph-mon[97044]: pgmap v18129: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:52 smithi110 ceph-mon[97633]: pgmap v18129: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:52.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:52.431835+0000) 2022-09-10T13:55:53.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:53.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:53.049932+0000) 2022-09-10T13:55:53.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:53.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:53.432129+0000) 2022-09-10T13:55:54.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:54.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:54.050139+0000) 2022-09-10T13:55:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:54 smithi104 ceph-mon[97044]: pgmap v18130: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:54 smithi110 ceph-mon[97633]: pgmap v18130: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:54.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:54.432272+0000) 2022-09-10T13:55:55.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:55.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:55.050368+0000) 2022-09-10T13:55:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:55.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:55.432484+0000) 2022-09-10T13:55:56.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:56.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:56.050664+0000) 2022-09-10T13:55:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:56 smithi104 ceph-mon[97044]: pgmap v18131: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:56 smithi110 ceph-mon[97633]: pgmap v18131: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:55:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:56.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:56.432662+0000) 2022-09-10T13:55:57.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:57.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:57.050884+0000) 2022-09-10T13:55:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:57.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:57.432900+0000) 2022-09-10T13:55:58.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:58.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:58.051155+0000) 2022-09-10T13:55:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:55:58 smithi104 ceph-mon[97044]: pgmap v18132: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:55:58 smithi110 ceph-mon[97633]: pgmap v18132: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:55:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:58.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:58.433068+0000) 2022-09-10T13:55:59.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:55:59.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:59.051380+0000) 2022-09-10T13:55:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:55:59.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:55:59.433319+0000) 2022-09-10T13:56:00.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:00.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:00.051561+0000) 2022-09-10T13:56:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:00 smithi104 ceph-mon[97044]: pgmap v18133: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:00 smithi110 ceph-mon[97633]: pgmap v18133: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:00.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:00.433526+0000) 2022-09-10T13:56:01.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:01.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:01.051712+0000) 2022-09-10T13:56:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:01.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:01.433716+0000) 2022-09-10T13:56:02.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:02.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:02.051921+0000) 2022-09-10T13:56:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:02 smithi104 ceph-mon[97044]: pgmap v18134: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:02 smithi110 ceph-mon[97633]: pgmap v18134: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:02.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:02.433963+0000) 2022-09-10T13:56:03.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:03.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:03.052207+0000) 2022-09-10T13:56:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:03.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:03.434198+0000) 2022-09-10T13:56:03.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:56:03.645+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:56:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:04.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:04.052469+0000) 2022-09-10T13:56:04.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:04 smithi104 ceph-mon[97044]: pgmap v18135: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:04 smithi110 ceph-mon[97633]: pgmap v18135: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:04.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:04.434371+0000) 2022-09-10T13:56:05.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:05.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:05.052693+0000) 2022-09-10T13:56:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:05.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:05.434620+0000) 2022-09-10T13:56:06.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:06.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:06.052922+0000) 2022-09-10T13:56:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:06 smithi104 ceph-mon[97044]: pgmap v18136: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:56:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:06 smithi110 ceph-mon[97633]: pgmap v18136: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:56:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:06.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:06.434856+0000) 2022-09-10T13:56:07.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:07.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:07.053171+0000) 2022-09-10T13:56:07.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:07.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:07.435044+0000) 2022-09-10T13:56:08.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:08.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:08.053374+0000) 2022-09-10T13:56:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:08 smithi104 ceph-mon[97044]: pgmap v18137: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:56:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:08 smithi110 ceph-mon[97633]: pgmap v18137: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:56:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:08.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:08.435239+0000) 2022-09-10T13:56:09.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:09.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:09.053518+0000) 2022-09-10T13:56:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:09 smithi104 ceph-mon[97044]: osdmap e1214: 12 total, 12 up, 12 in 2022-09-10T13:56:09.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:09 smithi110 ceph-mon[97633]: osdmap e1214: 12 total, 12 up, 12 in 2022-09-10T13:56:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:09.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:09.435501+0000) 2022-09-10T13:56:10.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:10.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:10.053773+0000) 2022-09-10T13:56:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:10 smithi104 ceph-mon[97044]: pgmap v18139: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:56:10.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:10 smithi110 ceph-mon[97633]: pgmap v18139: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:56:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:10.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:10.435742+0000) 2022-09-10T13:56:11.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:11.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:11.054017+0000) 2022-09-10T13:56:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:11.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:11.435894+0000) 2022-09-10T13:56:12.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:12.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:12.054265+0000) 2022-09-10T13:56:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:12 smithi104 ceph-mon[97044]: pgmap v18140: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:56:12.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:12 smithi110 ceph-mon[97633]: pgmap v18140: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:56:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:12.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:12.436128+0000) 2022-09-10T13:56:13.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:13.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:13.054525+0000) 2022-09-10T13:56:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:13.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:13.436346+0000) 2022-09-10T13:56:14.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:14.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:14.054734+0000) 2022-09-10T13:56:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:14 smithi104 ceph-mon[97044]: pgmap v18141: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:56:14.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:14 smithi110 ceph-mon[97633]: pgmap v18141: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:56:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:14.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:14.436571+0000) 2022-09-10T13:56:15.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:15.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:15.054984+0000) 2022-09-10T13:56:15.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:15.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:15.436748+0000) 2022-09-10T13:56:16.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:16.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:16.055252+0000) 2022-09-10T13:56:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:16 smithi104 ceph-mon[97044]: pgmap v18142: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:56:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:16 smithi104 ceph-mon[97044]: osdmap e1215: 12 total, 12 up, 12 in 2022-09-10T13:56:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:16 smithi110 ceph-mon[97633]: pgmap v18142: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:56:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:16 smithi110 ceph-mon[97633]: osdmap e1215: 12 total, 12 up, 12 in 2022-09-10T13:56:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:16.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:16.436979+0000) 2022-09-10T13:56:17.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:17.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:17.055442+0000) 2022-09-10T13:56:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:17.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:17.437205+0000) 2022-09-10T13:56:18.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:18.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:18.055673+0000) 2022-09-10T13:56:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:18 smithi104 ceph-mon[97044]: pgmap v18144: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:56:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:18 smithi110 ceph-mon[97633]: pgmap v18144: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T13:56:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:18.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:18.437405+0000) 2022-09-10T13:56:18.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:56:18.659+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:56:19.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:19.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:19.055847+0000) 2022-09-10T13:56:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:19.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:19.437568+0000) 2022-09-10T13:56:20.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:20.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:20.056070+0000) 2022-09-10T13:56:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:20 smithi104 ceph-mon[97044]: pgmap v18145: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:56:20.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:20 smithi110 ceph-mon[97633]: pgmap v18145: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:56:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:20.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:20.437803+0000) 2022-09-10T13:56:21.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:21.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:21.056263+0000) 2022-09-10T13:56:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:21.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:21.438068+0000) 2022-09-10T13:56:22.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:22.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:22.056448+0000) 2022-09-10T13:56:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:22 smithi104 ceph-mon[97044]: pgmap v18146: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:56:22.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:22 smithi110 ceph-mon[97633]: pgmap v18146: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:56:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:22.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:22.438320+0000) 2022-09-10T13:56:23.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:23.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:23.056624+0000) 2022-09-10T13:56:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:23.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:23.438574+0000) 2022-09-10T13:56:24.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:24.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:24.056859+0000) 2022-09-10T13:56:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:24 smithi104 ceph-mon[97044]: pgmap v18147: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:56:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:24 smithi110 ceph-mon[97633]: pgmap v18147: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:56:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:24.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:24.438803+0000) 2022-09-10T13:56:25.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:25.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:25.057039+0000) 2022-09-10T13:56:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:25.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:25.438958+0000) 2022-09-10T13:56:26.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:26.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:26.057324+0000) 2022-09-10T13:56:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:26 smithi104 ceph-mon[97044]: pgmap v18148: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:56:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:26 smithi110 ceph-mon[97633]: pgmap v18148: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:56:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:26.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:26.439211+0000) 2022-09-10T13:56:27.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:27.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:27.057509+0000) 2022-09-10T13:56:27.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:27.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:27.439406+0000) 2022-09-10T13:56:28.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:28.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:28.057773+0000) 2022-09-10T13:56:28.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:28 smithi104 ceph-mon[97044]: pgmap v18149: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:56:28.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:28 smithi110 ceph-mon[97633]: pgmap v18149: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:56:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:28.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:28.439580+0000) 2022-09-10T13:56:29.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:29.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:29.057984+0000) 2022-09-10T13:56:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:29.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:29.439824+0000) 2022-09-10T13:56:30.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:30.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:30.058227+0000) 2022-09-10T13:56:30.545 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:30 smithi104 ceph-mon[97044]: pgmap v18150: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:30 smithi110 ceph-mon[97633]: pgmap v18150: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:30.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:30.439981+0000) 2022-09-10T13:56:31.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:31.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:31.058490+0000) 2022-09-10T13:56:31.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:31.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:31.440179+0000) 2022-09-10T13:56:32.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:32.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:32.058710+0000) 2022-09-10T13:56:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:32 smithi104 ceph-mon[97044]: pgmap v18151: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:32 smithi110 ceph-mon[97633]: pgmap v18151: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:32.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:32.440411+0000) 2022-09-10T13:56:33.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:33.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:33.058948+0000) 2022-09-10T13:56:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:33.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:33.440632+0000) 2022-09-10T13:56:33.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:56:33.672+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:56:34.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:34.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:34.059186+0000) 2022-09-10T13:56:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:34 smithi104 ceph-mon[97044]: pgmap v18152: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:34.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:34 smithi110 ceph-mon[97633]: pgmap v18152: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:34.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:34.440790+0000) 2022-09-10T13:56:35.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:35.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:35.059432+0000) 2022-09-10T13:56:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:35.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:35.441019+0000) 2022-09-10T13:56:36.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:36.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:36.059698+0000) 2022-09-10T13:56:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:36 smithi104 ceph-mon[97044]: pgmap v18153: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:36 smithi110 ceph-mon[97633]: pgmap v18153: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:36.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:36.441203+0000) 2022-09-10T13:56:37.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:37.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:37.059939+0000) 2022-09-10T13:56:37.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:37.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:37.441382+0000) 2022-09-10T13:56:38.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:38.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:38.060177+0000) 2022-09-10T13:56:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:38 smithi104 ceph-mon[97044]: pgmap v18154: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:38.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:38 smithi110 ceph-mon[97633]: pgmap v18154: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:38.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:38.441594+0000) 2022-09-10T13:56:39.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:39.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:39.060368+0000) 2022-09-10T13:56:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:39.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:39.441819+0000) 2022-09-10T13:56:40.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:40.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:40.060600+0000) 2022-09-10T13:56:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:40 smithi104 ceph-mon[97044]: pgmap v18155: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:40.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:40 smithi110 ceph-mon[97633]: pgmap v18155: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:40.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:40.442037+0000) 2022-09-10T13:56:41.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:41.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:41.060885+0000) 2022-09-10T13:56:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:41.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:41.442245+0000) 2022-09-10T13:56:42.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:42.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:42.061130+0000) 2022-09-10T13:56:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:42 smithi104 ceph-mon[97044]: pgmap v18156: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:42.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:42 smithi110 ceph-mon[97633]: pgmap v18156: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:42.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:42.442495+0000) 2022-09-10T13:56:43.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:43.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:43.061399+0000) 2022-09-10T13:56:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:43.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:43.442696+0000) 2022-09-10T13:56:44.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:44.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:44.061645+0000) 2022-09-10T13:56:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:44 smithi104 ceph-mon[97044]: pgmap v18157: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:44.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:44 smithi110 ceph-mon[97633]: pgmap v18157: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:44.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:44.442887+0000) 2022-09-10T13:56:45.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:45.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:45.061862+0000) 2022-09-10T13:56:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:45.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:45.443110+0000) 2022-09-10T13:56:46.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:46.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:46.062032+0000) 2022-09-10T13:56:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:46 smithi104 ceph-mon[97044]: pgmap v18158: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:46.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:46 smithi110 ceph-mon[97633]: pgmap v18158: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:46.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:46.443237+0000) 2022-09-10T13:56:47.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:47.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:47.062282+0000) 2022-09-10T13:56:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:56:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:56:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:56:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:56:47.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:56:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:56:47.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:47.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:47.443412+0000) 2022-09-10T13:56:48.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:48.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:48.062428+0000) 2022-09-10T13:56:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:48 smithi104 ceph-mon[97044]: pgmap v18159: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:48.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:48 smithi110 ceph-mon[97633]: pgmap v18159: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:48.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:48.443619+0000) 2022-09-10T13:56:48.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:56:48.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:56:49.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:49.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:49.062685+0000) 2022-09-10T13:56:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:49.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:49.443875+0000) 2022-09-10T13:56:50.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:50.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:50.062876+0000) 2022-09-10T13:56:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:50 smithi104 ceph-mon[97044]: pgmap v18160: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:50 smithi110 ceph-mon[97633]: pgmap v18160: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:56:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:50.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:50.444175+0000) 2022-09-10T13:56:51.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:51.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:51.063029+0000) 2022-09-10T13:56:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:51.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:51.444417+0000) 2022-09-10T13:56:52.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:52.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:52.063218+0000) 2022-09-10T13:56:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:52 smithi104 ceph-mon[97044]: pgmap v18161: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:52.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:52 smithi110 ceph-mon[97633]: pgmap v18161: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:52.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:52.444587+0000) 2022-09-10T13:56:53.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:53.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:53.063377+0000) 2022-09-10T13:56:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:53.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:53.444809+0000) 2022-09-10T13:56:54.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:54.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:54.063671+0000) 2022-09-10T13:56:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:54 smithi104 ceph-mon[97044]: pgmap v18162: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:56:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:54 smithi110 ceph-mon[97633]: pgmap v18162: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:56:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:54.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:54.444966+0000) 2022-09-10T13:56:55.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:55.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:55.063846+0000) 2022-09-10T13:56:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:55.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:55.445197+0000) 2022-09-10T13:56:56.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:56.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:56.064033+0000) 2022-09-10T13:56:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:56 smithi104 ceph-mon[97044]: pgmap v18163: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:56.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:56 smithi110 ceph-mon[97633]: pgmap v18163: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:56:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:56.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:56.445369+0000) 2022-09-10T13:56:57.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:57.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:57.064204+0000) 2022-09-10T13:56:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:57.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:57.445578+0000) 2022-09-10T13:56:58.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:58.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:58.064422+0000) 2022-09-10T13:56:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:56:58 smithi104 ceph-mon[97044]: pgmap v18164: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:56:58.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:56:58 smithi110 ceph-mon[97633]: pgmap v18164: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:56:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:58.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:58.445840+0000) 2022-09-10T13:56:59.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:56:59.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:59.064651+0000) 2022-09-10T13:56:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:56:59.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:56:59.446004+0000) 2022-09-10T13:57:00.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:00.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:00.064820+0000) 2022-09-10T13:57:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:00 smithi104 ceph-mon[97044]: pgmap v18165: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:57:00.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:00 smithi110 ceph-mon[97633]: pgmap v18165: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:57:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:00.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:00.446198+0000) 2022-09-10T13:57:01.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:01.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:01.065024+0000) 2022-09-10T13:57:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:01.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:01.446406+0000) 2022-09-10T13:57:02.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:02.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:02.065262+0000) 2022-09-10T13:57:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:02 smithi110 ceph-mon[97633]: pgmap v18166: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:02 smithi104 ceph-mon[97044]: pgmap v18166: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:02.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:02.446674+0000) 2022-09-10T13:57:03.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:03.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:03.065522+0000) 2022-09-10T13:57:03.701 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:03.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:03.446905+0000) 2022-09-10T13:57:04.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:57:03.699+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:57:04.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:04.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:04.065807+0000) 2022-09-10T13:57:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:04 smithi110 ceph-mon[97633]: pgmap v18167: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:04 smithi104 ceph-mon[97044]: pgmap v18167: 129 pgs: 129 active+clean; 9.7 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:04.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:04.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:04.447167+0000) 2022-09-10T13:57:05.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:05.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:05.066047+0000) 2022-09-10T13:57:05.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:05.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:05.447349+0000) 2022-09-10T13:57:06.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:06.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:06.066281+0000) 2022-09-10T13:57:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:06 smithi110 ceph-mon[97633]: pgmap v18168: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:57:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:06 smithi104 ceph-mon[97044]: pgmap v18168: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:57:06.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:06.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:06.447607+0000) 2022-09-10T13:57:07.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:07.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:07.066493+0000) 2022-09-10T13:57:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:07.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:07.447834+0000) 2022-09-10T13:57:08.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:08.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:08.066712+0000) 2022-09-10T13:57:08.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:08 smithi110 ceph-mon[97633]: pgmap v18169: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:57:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:08 smithi104 ceph-mon[97044]: pgmap v18169: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:57:08.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:08.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:08.448189+0000) 2022-09-10T13:57:09.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:09.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:09.066927+0000) 2022-09-10T13:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:09 smithi110 ceph-mon[97633]: osdmap e1216: 12 total, 12 up, 12 in 2022-09-10T13:57:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:09 smithi104 ceph-mon[97044]: osdmap e1216: 12 total, 12 up, 12 in 2022-09-10T13:57:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:09.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:09.448320+0000) 2022-09-10T13:57:10.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:10.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:10.067173+0000) 2022-09-10T13:57:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:10 smithi110 ceph-mon[97633]: pgmap v18171: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:57:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:10 smithi104 ceph-mon[97044]: pgmap v18171: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:57:10.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:10.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:10.448487+0000) 2022-09-10T13:57:11.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:11.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:11.067337+0000) 2022-09-10T13:57:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:11.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:11.448740+0000) 2022-09-10T13:57:12.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:12.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:12.067552+0000) 2022-09-10T13:57:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:12 smithi110 ceph-mon[97633]: pgmap v18172: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:57:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:12 smithi104 ceph-mon[97044]: pgmap v18172: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:57:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:12.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:12.448981+0000) 2022-09-10T13:57:13.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:13.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:13.067801+0000) 2022-09-10T13:57:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:13.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:13.449226+0000) 2022-09-10T13:57:14.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:14.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:14.068051+0000) 2022-09-10T13:57:14.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:14 smithi110 ceph-mon[97633]: pgmap v18173: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:57:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:14 smithi104 ceph-mon[97044]: pgmap v18173: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:57:14.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:14.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:14.449401+0000) 2022-09-10T13:57:15.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:15.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:15.068341+0000) 2022-09-10T13:57:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:15.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:15.449566+0000) 2022-09-10T13:57:16.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:16.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:16.068537+0000) 2022-09-10T13:57:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:16 smithi110 ceph-mon[97633]: pgmap v18174: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:57:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:16 smithi110 ceph-mon[97633]: osdmap e1217: 12 total, 12 up, 12 in 2022-09-10T13:57:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:16 smithi104 ceph-mon[97044]: pgmap v18174: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:57:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:16 smithi104 ceph-mon[97044]: osdmap e1217: 12 total, 12 up, 12 in 2022-09-10T13:57:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:16.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:16.449760+0000) 2022-09-10T13:57:17.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:17.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:17.068742+0000) 2022-09-10T13:57:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:17.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:17.449971+0000) 2022-09-10T13:57:18.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:18.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:18.068930+0000) 2022-09-10T13:57:18.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:18 smithi110 ceph-mon[97633]: pgmap v18176: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T13:57:18.712 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:18.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:18.450161+0000) 2022-09-10T13:57:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:18 smithi104 ceph-mon[97044]: pgmap v18176: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T13:57:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:57:18.710+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:57:19.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:19.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:19.069158+0000) 2022-09-10T13:57:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:19.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:19.450324+0000) 2022-09-10T13:57:20.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:20.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:20.069380+0000) 2022-09-10T13:57:20.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:20 smithi110 ceph-mon[97633]: pgmap v18177: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:57:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:20 smithi104 ceph-mon[97044]: pgmap v18177: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:57:20.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:20.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:20.450539+0000) 2022-09-10T13:57:21.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:21.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:21.069635+0000) 2022-09-10T13:57:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:21.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:21.450808+0000) 2022-09-10T13:57:22.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:22.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:22.069802+0000) 2022-09-10T13:57:22.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:22 smithi110 ceph-mon[97633]: pgmap v18178: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:57:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:22 smithi104 ceph-mon[97044]: pgmap v18178: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:57:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:22.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:22.451012+0000) 2022-09-10T13:57:23.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:23.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:23.070020+0000) 2022-09-10T13:57:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:23.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:23.451251+0000) 2022-09-10T13:57:24.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:24.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:24.070265+0000) 2022-09-10T13:57:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:24 smithi110 ceph-mon[97633]: pgmap v18179: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:57:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:24 smithi104 ceph-mon[97044]: pgmap v18179: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:57:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:24.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:24.451422+0000) 2022-09-10T13:57:25.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:25.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:25.070455+0000) 2022-09-10T13:57:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:25.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:25.451563+0000) 2022-09-10T13:57:26.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:26.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:26.070670+0000) 2022-09-10T13:57:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:26 smithi110 ceph-mon[97633]: pgmap v18180: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:57:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:26 smithi104 ceph-mon[97044]: pgmap v18180: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:57:26.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:26.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:26.451760+0000) 2022-09-10T13:57:27.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:27.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:27.070921+0000) 2022-09-10T13:57:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:27.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:27.451991+0000) 2022-09-10T13:57:28.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:28.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:28.071145+0000) 2022-09-10T13:57:28.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:28 smithi110 ceph-mon[97633]: pgmap v18181: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:57:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:28 smithi104 ceph-mon[97044]: pgmap v18181: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:57:28.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:28.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:28.452224+0000) 2022-09-10T13:57:29.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:29.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:29.071360+0000) 2022-09-10T13:57:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:29.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:29.452373+0000) 2022-09-10T13:57:30.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:30.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:30.071601+0000) 2022-09-10T13:57:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:30 smithi110 ceph-mon[97633]: pgmap v18182: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:30 smithi104 ceph-mon[97044]: pgmap v18182: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:30.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:30.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:30.452568+0000) 2022-09-10T13:57:31.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:31.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:31.071801+0000) 2022-09-10T13:57:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:31.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:31.452729+0000) 2022-09-10T13:57:32.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:32.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:32.071968+0000) 2022-09-10T13:57:32.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:32 smithi110 ceph-mon[97633]: pgmap v18183: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:32 smithi104 ceph-mon[97044]: pgmap v18183: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:32.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:32.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:32.452964+0000) 2022-09-10T13:57:33.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:33.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:33.072192+0000) 2022-09-10T13:57:33.727 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:33.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:33.453222+0000) 2022-09-10T13:57:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:57:33.725+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:57:34.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:34.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:34.072341+0000) 2022-09-10T13:57:34.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:34 smithi110 ceph-mon[97633]: pgmap v18184: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:34 smithi104 ceph-mon[97044]: pgmap v18184: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:34.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:34.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:34.453479+0000) 2022-09-10T13:57:35.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:35.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:35.072535+0000) 2022-09-10T13:57:35.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:35.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:35.453716+0000) 2022-09-10T13:57:36.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:36.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:36.072704+0000) 2022-09-10T13:57:36.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:36 smithi110 ceph-mon[97633]: pgmap v18185: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:36 smithi104 ceph-mon[97044]: pgmap v18185: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:36.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:36.453913+0000) 2022-09-10T13:57:37.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:37.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:37.072927+0000) 2022-09-10T13:57:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:37.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:37.454188+0000) 2022-09-10T13:57:38.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:38.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:38.073157+0000) 2022-09-10T13:57:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:38 smithi110 ceph-mon[97633]: pgmap v18186: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:38 smithi104 ceph-mon[97044]: pgmap v18186: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:38.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:38.454364+0000) 2022-09-10T13:57:39.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:39.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:39.073312+0000) 2022-09-10T13:57:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:39.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:39.454611+0000) 2022-09-10T13:57:40.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:40.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:40.073538+0000) 2022-09-10T13:57:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:40 smithi110 ceph-mon[97633]: pgmap v18187: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:40 smithi104 ceph-mon[97044]: pgmap v18187: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:40.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:40.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:40.454763+0000) 2022-09-10T13:57:41.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:41.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:41.073732+0000) 2022-09-10T13:57:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:41.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:41.454935+0000) 2022-09-10T13:57:42.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:42.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:42.073937+0000) 2022-09-10T13:57:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:42 smithi110 ceph-mon[97633]: pgmap v18188: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:42 smithi104 ceph-mon[97044]: pgmap v18188: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:42.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:42.455197+0000) 2022-09-10T13:57:43.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:43.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:43.074177+0000) 2022-09-10T13:57:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:43.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:43.455381+0000) 2022-09-10T13:57:44.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:44.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:44.074428+0000) 2022-09-10T13:57:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:44 smithi110 ceph-mon[97633]: pgmap v18189: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:44 smithi104 ceph-mon[97044]: pgmap v18189: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:44.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:44.455585+0000) 2022-09-10T13:57:45.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:45.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:45.074673+0000) 2022-09-10T13:57:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:45.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:45.455763+0000) 2022-09-10T13:57:46.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:46.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:46.074937+0000) 2022-09-10T13:57:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:46 smithi110 ceph-mon[97633]: pgmap v18190: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:46 smithi104 ceph-mon[97044]: pgmap v18190: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:46.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:46.455980+0000) 2022-09-10T13:57:47.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:47.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:47.075212+0000) 2022-09-10T13:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:57:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:57:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:57:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:57:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:57:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:57:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:47.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:47.456197+0000) 2022-09-10T13:57:48.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:48.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:48.075439+0000) 2022-09-10T13:57:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:48 smithi110 ceph-mon[97633]: pgmap v18191: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:48.742 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:48.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:48.456408+0000) 2022-09-10T13:57:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:48 smithi104 ceph-mon[97044]: pgmap v18191: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:57:48.740+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:57:49.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:49.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:49.075729+0000) 2022-09-10T13:57:49.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:49.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:49.456604+0000) 2022-09-10T13:57:50.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:50.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:50.075988+0000) 2022-09-10T13:57:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:50 smithi110 ceph-mon[97633]: pgmap v18192: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:50 smithi104 ceph-mon[97044]: pgmap v18192: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:50.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:50.456834+0000) 2022-09-10T13:57:51.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:51.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:51.076246+0000) 2022-09-10T13:57:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:51.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:51.457132+0000) 2022-09-10T13:57:52.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:52.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:52.076498+0000) 2022-09-10T13:57:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:52 smithi110 ceph-mon[97633]: pgmap v18193: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:52 smithi104 ceph-mon[97044]: pgmap v18193: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:52.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:52.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:52.457311+0000) 2022-09-10T13:57:53.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:53.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:53.076700+0000) 2022-09-10T13:57:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:53.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:53.457545+0000) 2022-09-10T13:57:54.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:54.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:54.076944+0000) 2022-09-10T13:57:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:54 smithi110 ceph-mon[97633]: pgmap v18194: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:54 smithi104 ceph-mon[97044]: pgmap v18194: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:54.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:54.457724+0000) 2022-09-10T13:57:55.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:55.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:55.077188+0000) 2022-09-10T13:57:55.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:55.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:55.457923+0000) 2022-09-10T13:57:56.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:56.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:56.077430+0000) 2022-09-10T13:57:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:56 smithi110 ceph-mon[97633]: pgmap v18195: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:56 smithi104 ceph-mon[97044]: pgmap v18195: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:57:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:56.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:56.458200+0000) 2022-09-10T13:57:57.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:57.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:57.077591+0000) 2022-09-10T13:57:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:57.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:57.458396+0000) 2022-09-10T13:57:58.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:58.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:58.077794+0000) 2022-09-10T13:57:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:57:58 smithi110 ceph-mon[97633]: pgmap v18196: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:57:58 smithi104 ceph-mon[97044]: pgmap v18196: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:57:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:58.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:58.458627+0000) 2022-09-10T13:57:59.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:57:59.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:59.077975+0000) 2022-09-10T13:57:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:57:59.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:57:59.458832+0000) 2022-09-10T13:58:00.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:00.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:00.078189+0000) 2022-09-10T13:58:00.656 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:00 smithi110 ceph-mon[97633]: pgmap v18197: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:00 smithi104 ceph-mon[97044]: pgmap v18197: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:00.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:00.459006+0000) 2022-09-10T13:58:01.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:01.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:01.078409+0000) 2022-09-10T13:58:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:01.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:01.459261+0000) 2022-09-10T13:58:02.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:02.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:02.078643+0000) 2022-09-10T13:58:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:02 smithi110 ceph-mon[97633]: pgmap v18198: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:02 smithi104 ceph-mon[97044]: pgmap v18198: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:02.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:02.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:02.459499+0000) 2022-09-10T13:58:03.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:03.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:03.078885+0000) 2022-09-10T13:58:03.750 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:03.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:03.459708+0000) 2022-09-10T13:58:04.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:58:03.748+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:58:04.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:04.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:04.079133+0000) 2022-09-10T13:58:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:04 smithi104 ceph-mon[97044]: pgmap v18199: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:04.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:04.459951+0000) 2022-09-10T13:58:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:04 smithi110 ceph-mon[97633]: pgmap v18199: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:05.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:05.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:05.079274+0000) 2022-09-10T13:58:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:05.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:05.460077+0000) 2022-09-10T13:58:06.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:06.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:06.079438+0000) 2022-09-10T13:58:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:06 smithi104 ceph-mon[97044]: pgmap v18200: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:58:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:06.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:06.460291+0000) 2022-09-10T13:58:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:06 smithi110 ceph-mon[97633]: pgmap v18200: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:58:07.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:07.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:07.079606+0000) 2022-09-10T13:58:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:07.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:07.460490+0000) 2022-09-10T13:58:08.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:08.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:08.079806+0000) 2022-09-10T13:58:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:08 smithi104 ceph-mon[97044]: pgmap v18201: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:58:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:08.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:08.460689+0000) 2022-09-10T13:58:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:08 smithi110 ceph-mon[97633]: pgmap v18201: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:58:09.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:09.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:09.080011+0000) 2022-09-10T13:58:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:09 smithi104 ceph-mon[97044]: osdmap e1218: 12 total, 12 up, 12 in 2022-09-10T13:58:09.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:09.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:09.460946+0000) 2022-09-10T13:58:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:09 smithi110 ceph-mon[97633]: osdmap e1218: 12 total, 12 up, 12 in 2022-09-10T13:58:10.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:10.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:10.080241+0000) 2022-09-10T13:58:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:10 smithi104 ceph-mon[97044]: pgmap v18203: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:58:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:10.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:10.461210+0000) 2022-09-10T13:58:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:10 smithi110 ceph-mon[97633]: pgmap v18203: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:58:11.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:11.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:11.080430+0000) 2022-09-10T13:58:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:11.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:11.461418+0000) 2022-09-10T13:58:12.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:12.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:12.080616+0000) 2022-09-10T13:58:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:12 smithi104 ceph-mon[97044]: pgmap v18204: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:58:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:12.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:12.461528+0000) 2022-09-10T13:58:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:12 smithi110 ceph-mon[97633]: pgmap v18204: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:58:13.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:13.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:13.080840+0000) 2022-09-10T13:58:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:13.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:13.461742+0000) 2022-09-10T13:58:14.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:14.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:14.081056+0000) 2022-09-10T13:58:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:14 smithi104 ceph-mon[97044]: pgmap v18205: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:58:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:14.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:14.461964+0000) 2022-09-10T13:58:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:14 smithi110 ceph-mon[97633]: pgmap v18205: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:58:15.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:15.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:15.081185+0000) 2022-09-10T13:58:15.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:15.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:15.462207+0000) 2022-09-10T13:58:16.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:16.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:16.081419+0000) 2022-09-10T13:58:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:16 smithi104 ceph-mon[97044]: pgmap v18206: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:58:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:16 smithi104 ceph-mon[97044]: osdmap e1219: 12 total, 12 up, 12 in 2022-09-10T13:58:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:16.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:16.462387+0000) 2022-09-10T13:58:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:16 smithi110 ceph-mon[97633]: pgmap v18206: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:58:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:16 smithi110 ceph-mon[97633]: osdmap e1219: 12 total, 12 up, 12 in 2022-09-10T13:58:17.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:17.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:17.081591+0000) 2022-09-10T13:58:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:17.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:17.462614+0000) 2022-09-10T13:58:18.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:18.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:18.081759+0000) 2022-09-10T13:58:18.763 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:18.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:18.462810+0000) 2022-09-10T13:58:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:18 smithi104 ceph-mon[97044]: pgmap v18208: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T13:58:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:18 smithi110 ceph-mon[97633]: pgmap v18208: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 246 B/s wr, 12 op/s 2022-09-10T13:58:19.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:58:18.761+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:58:19.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:19.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:19.081930+0000) 2022-09-10T13:58:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:19.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:19.463022+0000) 2022-09-10T13:58:20.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:20.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:20.082138+0000) 2022-09-10T13:58:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:20 smithi104 ceph-mon[97044]: pgmap v18209: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:58:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:20.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:20.463196+0000) 2022-09-10T13:58:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:20 smithi110 ceph-mon[97633]: pgmap v18209: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:58:21.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:21.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:21.082351+0000) 2022-09-10T13:58:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:21.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:21.463388+0000) 2022-09-10T13:58:22.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:22.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:22.082545+0000) 2022-09-10T13:58:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:22 smithi104 ceph-mon[97044]: pgmap v18210: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:58:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:22.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:22.463625+0000) 2022-09-10T13:58:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:22 smithi110 ceph-mon[97633]: pgmap v18210: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:58:23.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:23.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:23.082729+0000) 2022-09-10T13:58:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:23.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:23.463824+0000) 2022-09-10T13:58:24.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:24.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:24.082907+0000) 2022-09-10T13:58:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:24 smithi104 ceph-mon[97044]: pgmap v18211: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:58:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:24.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:24.464006+0000) 2022-09-10T13:58:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:24 smithi110 ceph-mon[97633]: pgmap v18211: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:58:25.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:25.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:25.083058+0000) 2022-09-10T13:58:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:25.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:25.464193+0000) 2022-09-10T13:58:26.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:26.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:26.083240+0000) 2022-09-10T13:58:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:26 smithi104 ceph-mon[97044]: pgmap v18212: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:58:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:26.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:26.464365+0000) 2022-09-10T13:58:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:26 smithi110 ceph-mon[97633]: pgmap v18212: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:58:27.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:27.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:27.083420+0000) 2022-09-10T13:58:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:27.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:27.464568+0000) 2022-09-10T13:58:28.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:28.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:28.083569+0000) 2022-09-10T13:58:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:28 smithi104 ceph-mon[97044]: pgmap v18213: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:58:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:28.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:28.464824+0000) 2022-09-10T13:58:28.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:28 smithi110 ceph-mon[97633]: pgmap v18213: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:58:29.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:29.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:29.083694+0000) 2022-09-10T13:58:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:29.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:29.465005+0000) 2022-09-10T13:58:30.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:30.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:30.083866+0000) 2022-09-10T13:58:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:30 smithi104 ceph-mon[97044]: pgmap v18214: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:30.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:30.465240+0000) 2022-09-10T13:58:30.915 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:30 smithi110 ceph-mon[97633]: pgmap v18214: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:31.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:31.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:31.084065+0000) 2022-09-10T13:58:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:31.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:31.465408+0000) 2022-09-10T13:58:32.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:32.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:32.084278+0000) 2022-09-10T13:58:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:32 smithi104 ceph-mon[97044]: pgmap v18215: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:32.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:32.465559+0000) 2022-09-10T13:58:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:32 smithi110 ceph-mon[97633]: pgmap v18215: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:33.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:33.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:33.084470+0000) 2022-09-10T13:58:33.778 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:33.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:33.465751+0000) 2022-09-10T13:58:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:58:33.777+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:58:34.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:34.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:34.084660+0000) 2022-09-10T13:58:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:34 smithi104 ceph-mon[97044]: pgmap v18216: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:34.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:34.465955+0000) 2022-09-10T13:58:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:34 smithi110 ceph-mon[97633]: pgmap v18216: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:35.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:35.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:35.084860+0000) 2022-09-10T13:58:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:35.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:35.466185+0000) 2022-09-10T13:58:36.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:36.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:36.085068+0000) 2022-09-10T13:58:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:36 smithi104 ceph-mon[97044]: pgmap v18217: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:36.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:36.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:36.466422+0000) 2022-09-10T13:58:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:36 smithi110 ceph-mon[97633]: pgmap v18217: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:37.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:37.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:37.085273+0000) 2022-09-10T13:58:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:37.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:37.466588+0000) 2022-09-10T13:58:38.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:38.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:38.085458+0000) 2022-09-10T13:58:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:38 smithi104 ceph-mon[97044]: pgmap v18218: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:38.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:38.466775+0000) 2022-09-10T13:58:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:38 smithi110 ceph-mon[97633]: pgmap v18218: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:39.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:39.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:39.085628+0000) 2022-09-10T13:58:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:39.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:39.466959+0000) 2022-09-10T13:58:40.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:40.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:40.085859+0000) 2022-09-10T13:58:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:40 smithi104 ceph-mon[97044]: pgmap v18219: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:40.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:40.467190+0000) 2022-09-10T13:58:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:40 smithi110 ceph-mon[97633]: pgmap v18219: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:41.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:41.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:41.086057+0000) 2022-09-10T13:58:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:41.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:41.467361+0000) 2022-09-10T13:58:42.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:42.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:42.086290+0000) 2022-09-10T13:58:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:42 smithi104 ceph-mon[97044]: pgmap v18220: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:42.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:42.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:42.467547+0000) 2022-09-10T13:58:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:42 smithi110 ceph-mon[97633]: pgmap v18220: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:43.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:43.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:43.086435+0000) 2022-09-10T13:58:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:43.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:43.467799+0000) 2022-09-10T13:58:44.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:44.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:44.086577+0000) 2022-09-10T13:58:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:44 smithi104 ceph-mon[97044]: pgmap v18221: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:58:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:44.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:44.467991+0000) 2022-09-10T13:58:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:44 smithi110 ceph-mon[97633]: pgmap v18221: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:58:45.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:45.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:45.086767+0000) 2022-09-10T13:58:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:45.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:45.468233+0000) 2022-09-10T13:58:46.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:46.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:46.086940+0000) 2022-09-10T13:58:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:46 smithi104 ceph-mon[97044]: pgmap v18222: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:58:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:58:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:58:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:58:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:58:46.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:46.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:46.468480+0000) 2022-09-10T13:58:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:46 smithi110 ceph-mon[97633]: pgmap v18222: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:58:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:58:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:58:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:58:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:58:47.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:47.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:47.087194+0000) 2022-09-10T13:58:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:47.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:47.468622+0000) 2022-09-10T13:58:48.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:48.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:48.087338+0000) 2022-09-10T13:58:48.792 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:48.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:48.468866+0000) 2022-09-10T13:58:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:48 smithi104 ceph-mon[97044]: pgmap v18223: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:58:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:48 smithi110 ceph-mon[97633]: pgmap v18223: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:58:49.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:58:48.791+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:58:49.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:49.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:49.087605+0000) 2022-09-10T13:58:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:49.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:49.469071+0000) 2022-09-10T13:58:50.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:50.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:50.087749+0000) 2022-09-10T13:58:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:50 smithi104 ceph-mon[97044]: pgmap v18224: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:58:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:50.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:50.469313+0000) 2022-09-10T13:58:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:50 smithi110 ceph-mon[97633]: pgmap v18224: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T13:58:51.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:51.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:51.087948+0000) 2022-09-10T13:58:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:51.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:51.469500+0000) 2022-09-10T13:58:52.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:52.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:52.088163+0000) 2022-09-10T13:58:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:52 smithi104 ceph-mon[97044]: pgmap v18225: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:58:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:52.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:52.469680+0000) 2022-09-10T13:58:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:52 smithi110 ceph-mon[97633]: pgmap v18225: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T13:58:53.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:53.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:53.088371+0000) 2022-09-10T13:58:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:53.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:53.469966+0000) 2022-09-10T13:58:54.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:54.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:54.088606+0000) 2022-09-10T13:58:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:54 smithi104 ceph-mon[97044]: pgmap v18226: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:58:54.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:54.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:54.470213+0000) 2022-09-10T13:58:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:54 smithi110 ceph-mon[97633]: pgmap v18226: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T13:58:55.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:55.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:55.088833+0000) 2022-09-10T13:58:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:55.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:55.470355+0000) 2022-09-10T13:58:56.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:56.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:56.089032+0000) 2022-09-10T13:58:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:56 smithi104 ceph-mon[97044]: pgmap v18227: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:56.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:56.470608+0000) 2022-09-10T13:58:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:56 smithi110 ceph-mon[97633]: pgmap v18227: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:58:57.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:57.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:57.089299+0000) 2022-09-10T13:58:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:57.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:57.470856+0000) 2022-09-10T13:58:58.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:58.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:58.089541+0000) 2022-09-10T13:58:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:58:58 smithi104 ceph-mon[97044]: pgmap v18228: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:58.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:58.471132+0000) 2022-09-10T13:58:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:58:58 smithi110 ceph-mon[97633]: pgmap v18228: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:58:59.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:58:59.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:59.089720+0000) 2022-09-10T13:58:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:58:59.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:58:59.471350+0000) 2022-09-10T13:59:00.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:00.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:00.089972+0000) 2022-09-10T13:59:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:00 smithi104 ceph-mon[97044]: pgmap v18229: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:00.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:00.471574+0000) 2022-09-10T13:59:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:00 smithi110 ceph-mon[97633]: pgmap v18229: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:01.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:01.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:01.090166+0000) 2022-09-10T13:59:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:01.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:01.471865+0000) 2022-09-10T13:59:02.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:02.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:02.090336+0000) 2022-09-10T13:59:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:02 smithi104 ceph-mon[97044]: pgmap v18230: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:02.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:02.472157+0000) 2022-09-10T13:59:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:02 smithi110 ceph-mon[97633]: pgmap v18230: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:03.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:03.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:03.090579+0000) 2022-09-10T13:59:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:03.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:03.472414+0000) 2022-09-10T13:59:04.091 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:59:03.803+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:59:04.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:04.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:04.090776+0000) 2022-09-10T13:59:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:04 smithi104 ceph-mon[97044]: pgmap v18231: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:04.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:04.472590+0000) 2022-09-10T13:59:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:04 smithi110 ceph-mon[97633]: pgmap v18231: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:05.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:05.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:05.090956+0000) 2022-09-10T13:59:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:05.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:05.472735+0000) 2022-09-10T13:59:06.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:06.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:06.091189+0000) 2022-09-10T13:59:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:06 smithi104 ceph-mon[97044]: pgmap v18232: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:59:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:06.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:06.472951+0000) 2022-09-10T13:59:06.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:06 smithi110 ceph-mon[97633]: pgmap v18232: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T13:59:07.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:07.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:07.091405+0000) 2022-09-10T13:59:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:07.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:07.473197+0000) 2022-09-10T13:59:08.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:08.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:08.091607+0000) 2022-09-10T13:59:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:08 smithi104 ceph-mon[97044]: pgmap v18233: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:59:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:08.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:08.473420+0000) 2022-09-10T13:59:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:08 smithi110 ceph-mon[97633]: pgmap v18233: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T13:59:09.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:09.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:09.091881+0000) 2022-09-10T13:59:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:09 smithi104 ceph-mon[97044]: osdmap e1220: 12 total, 12 up, 12 in 2022-09-10T13:59:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:09.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:09.473641+0000) 2022-09-10T13:59:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:09 smithi110 ceph-mon[97633]: osdmap e1220: 12 total, 12 up, 12 in 2022-09-10T13:59:10.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:10.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:10.092163+0000) 2022-09-10T13:59:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:10 smithi104 ceph-mon[97044]: pgmap v18235: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:59:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:10.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:10.473829+0000) 2022-09-10T13:59:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:10 smithi110 ceph-mon[97633]: pgmap v18235: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:59:11.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:11.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:11.092330+0000) 2022-09-10T13:59:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:11.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:11.474033+0000) 2022-09-10T13:59:12.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:12.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:12.092576+0000) 2022-09-10T13:59:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:12 smithi104 ceph-mon[97044]: pgmap v18236: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:59:12.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:12.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:12.474215+0000) 2022-09-10T13:59:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:12 smithi110 ceph-mon[97633]: pgmap v18236: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:59:13.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:13.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:13.092815+0000) 2022-09-10T13:59:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:13.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:13.474348+0000) 2022-09-10T13:59:14.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:14.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:14.093066+0000) 2022-09-10T13:59:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:14 smithi104 ceph-mon[97044]: pgmap v18237: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:59:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:14.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:14.474533+0000) 2022-09-10T13:59:14.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:14 smithi110 ceph-mon[97633]: pgmap v18237: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T13:59:15.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:15.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:15.093347+0000) 2022-09-10T13:59:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:15.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:15.474685+0000) 2022-09-10T13:59:16.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:16.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:16.093570+0000) 2022-09-10T13:59:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:16 smithi104 ceph-mon[97044]: pgmap v18238: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:59:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:16 smithi104 ceph-mon[97044]: osdmap e1221: 12 total, 12 up, 12 in 2022-09-10T13:59:16.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:16.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:16.474914+0000) 2022-09-10T13:59:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:16 smithi110 ceph-mon[97633]: pgmap v18238: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:59:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:16 smithi110 ceph-mon[97633]: osdmap e1221: 12 total, 12 up, 12 in 2022-09-10T13:59:17.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:17.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:17.093800+0000) 2022-09-10T13:59:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:17.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:17.475138+0000) 2022-09-10T13:59:18.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:18.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:18.094025+0000) 2022-09-10T13:59:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:18 smithi104 ceph-mon[97044]: pgmap v18240: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T13:59:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:18.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:18.475372+0000) 2022-09-10T13:59:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:18 smithi110 ceph-mon[97633]: pgmap v18240: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2022-09-10T13:59:19.094 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:59:18.818+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:59:19.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:19.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:19.094270+0000) 2022-09-10T13:59:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:19.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:19.475632+0000) 2022-09-10T13:59:20.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:20.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:20.094478+0000) 2022-09-10T13:59:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:20 smithi104 ceph-mon[97044]: pgmap v18241: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:59:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:20.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:20.475805+0000) 2022-09-10T13:59:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:20 smithi110 ceph-mon[97633]: pgmap v18241: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T13:59:21.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:21.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:21.094739+0000) 2022-09-10T13:59:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:21.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:21.476016+0000) 2022-09-10T13:59:22.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:22.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:22.094991+0000) 2022-09-10T13:59:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:22 smithi104 ceph-mon[97044]: pgmap v18242: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:59:22.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:22.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:22.476243+0000) 2022-09-10T13:59:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:22 smithi110 ceph-mon[97633]: pgmap v18242: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:59:23.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:23.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:23.095207+0000) 2022-09-10T13:59:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:23.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:23.476494+0000) 2022-09-10T13:59:24.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:24.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:24.095441+0000) 2022-09-10T13:59:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:24 smithi104 ceph-mon[97044]: pgmap v18243: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:59:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:24.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:24.476708+0000) 2022-09-10T13:59:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:24 smithi110 ceph-mon[97633]: pgmap v18243: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T13:59:25.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:25.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:25.095651+0000) 2022-09-10T13:59:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:25.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:25.476949+0000) 2022-09-10T13:59:26.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:26.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:26.095869+0000) 2022-09-10T13:59:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:26.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:26.477154+0000) 2022-09-10T13:59:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:26 smithi110 ceph-mon[97633]: pgmap v18244: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:59:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:26 smithi104 ceph-mon[97044]: pgmap v18244: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T13:59:27.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:27.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:27.096020+0000) 2022-09-10T13:59:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:27.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:27.477386+0000) 2022-09-10T13:59:28.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:28.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:28.096228+0000) 2022-09-10T13:59:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:28.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:28.477615+0000) 2022-09-10T13:59:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:28 smithi110 ceph-mon[97633]: pgmap v18245: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:59:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:28 smithi104 ceph-mon[97044]: pgmap v18245: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T13:59:29.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:29.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:29.096422+0000) 2022-09-10T13:59:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:29.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:29.477807+0000) 2022-09-10T13:59:30.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:30.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:30.096614+0000) 2022-09-10T13:59:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:30.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:30.478042+0000) 2022-09-10T13:59:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:30 smithi110 ceph-mon[97633]: pgmap v18246: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:30 smithi104 ceph-mon[97044]: pgmap v18246: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:31.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:31.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:31.096753+0000) 2022-09-10T13:59:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:31.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:31.478352+0000) 2022-09-10T13:59:32.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:32.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:32.096921+0000) 2022-09-10T13:59:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:32.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:32.478487+0000) 2022-09-10T13:59:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:32 smithi110 ceph-mon[97633]: pgmap v18247: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:32 smithi104 ceph-mon[97044]: pgmap v18247: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:33.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:33.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:33.097126+0000) 2022-09-10T13:59:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:33.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:33.478658+0000) 2022-09-10T13:59:34.098 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:59:33.830+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:59:34.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:34.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:34.097334+0000) 2022-09-10T13:59:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:34.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:34.478891+0000) 2022-09-10T13:59:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:34 smithi110 ceph-mon[97633]: pgmap v18248: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:34 smithi104 ceph-mon[97044]: pgmap v18248: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:35.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:35.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:35.097550+0000) 2022-09-10T13:59:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:35.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:35.479133+0000) 2022-09-10T13:59:36.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:36.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:36.097761+0000) 2022-09-10T13:59:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:36.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:36.479313+0000) 2022-09-10T13:59:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:36 smithi110 ceph-mon[97633]: pgmap v18249: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:36 smithi104 ceph-mon[97044]: pgmap v18249: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:37.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:37.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:37.098015+0000) 2022-09-10T13:59:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:37.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:37.479567+0000) 2022-09-10T13:59:38.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:38.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:38.098194+0000) 2022-09-10T13:59:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:38.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:38.479820+0000) 2022-09-10T13:59:38.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:38 smithi110 ceph-mon[97633]: pgmap v18250: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:38 smithi104 ceph-mon[97044]: pgmap v18250: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:39.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:39.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:39.098446+0000) 2022-09-10T13:59:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:39.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:39.480081+0000) 2022-09-10T13:59:40.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:40.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:40.098693+0000) 2022-09-10T13:59:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:40.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:40.480326+0000) 2022-09-10T13:59:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:40 smithi110 ceph-mon[97633]: pgmap v18251: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:40 smithi104 ceph-mon[97044]: pgmap v18251: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:41.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:41.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:41.098943+0000) 2022-09-10T13:59:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:41.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:41.480550+0000) 2022-09-10T13:59:42.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:42.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:42.099176+0000) 2022-09-10T13:59:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:42.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:42.480764+0000) 2022-09-10T13:59:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:42 smithi110 ceph-mon[97633]: pgmap v18252: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:42 smithi104 ceph-mon[97044]: pgmap v18252: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:43.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:43.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:43.099363+0000) 2022-09-10T13:59:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:43.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:43.480984+0000) 2022-09-10T13:59:44.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:44.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:44.099530+0000) 2022-09-10T13:59:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:44.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:44.481202+0000) 2022-09-10T13:59:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:44 smithi110 ceph-mon[97633]: pgmap v18253: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:44 smithi104 ceph-mon[97044]: pgmap v18253: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:45.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:45.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:45.099721+0000) 2022-09-10T13:59:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:45.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:45.481418+0000) 2022-09-10T13:59:46.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:46.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:46.099963+0000) 2022-09-10T13:59:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:46.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:46.481643+0000) 2022-09-10T13:59:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:46 smithi110 ceph-mon[97633]: pgmap v18254: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:59:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:59:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:59:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:59:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:46 smithi104 ceph-mon[97044]: pgmap v18254: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:59:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T13:59:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:59:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T13:59:47.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:47.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:47.100230+0000) 2022-09-10T13:59:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:47.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:47.481866+0000) 2022-09-10T13:59:48.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:48.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:48.100402+0000) 2022-09-10T13:59:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:48.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:48.482078+0000) 2022-09-10T13:59:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:48 smithi110 ceph-mon[97633]: pgmap v18255: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:48 smithi104 ceph-mon[97044]: pgmap v18255: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:49.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 13:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T13:59:48.845+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T13:59:49.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:49.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:49.100521+0000) 2022-09-10T13:59:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:49.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:49.482281+0000) 2022-09-10T13:59:50.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:50.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:50.100729+0000) 2022-09-10T13:59:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:50.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:50.482465+0000) 2022-09-10T13:59:50.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:50 smithi110 ceph-mon[97633]: pgmap v18256: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:50 smithi104 ceph-mon[97044]: pgmap v18256: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:51.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:51.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:51.100967+0000) 2022-09-10T13:59:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:51.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:51.482615+0000) 2022-09-10T13:59:52.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:52.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:52.101238+0000) 2022-09-10T13:59:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:52.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:52.482797+0000) 2022-09-10T13:59:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:52 smithi110 ceph-mon[97633]: pgmap v18257: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:52 smithi104 ceph-mon[97044]: pgmap v18257: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:53.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:53.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:53.101416+0000) 2022-09-10T13:59:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:53.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:53.483024+0000) 2022-09-10T13:59:54.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:54.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:54.101660+0000) 2022-09-10T13:59:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:54.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:54.483253+0000) 2022-09-10T13:59:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:54 smithi110 ceph-mon[97633]: pgmap v18258: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:54 smithi104 ceph-mon[97044]: pgmap v18258: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:55.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:55.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:55.101882+0000) 2022-09-10T13:59:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:55.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:55.483463+0000) 2022-09-10T13:59:56.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:56.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:56.102166+0000) 2022-09-10T13:59:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:56.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:56.483638+0000) 2022-09-10T13:59:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:56 smithi110 ceph-mon[97633]: pgmap v18259: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:56 smithi104 ceph-mon[97044]: pgmap v18259: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T13:59:57.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:57.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:57.102380+0000) 2022-09-10T13:59:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:57.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:57.483909+0000) 2022-09-10T13:59:58.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:58.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:58.102575+0000) 2022-09-10T13:59:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:58.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:58.484208+0000) 2022-09-10T13:59:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 13:59:58 smithi110 ceph-mon[97633]: pgmap v18260: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 13:59:58 smithi104 ceph-mon[97044]: pgmap v18260: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T13:59:59.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 13:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T13:59:59.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:59.102737+0000) 2022-09-10T13:59:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 13:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T13:59:59.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T12:59:59.484346+0000) 2022-09-10T14:00:00.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:00.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:00.102868+0000) 2022-09-10T14:00:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:00.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:00.484577+0000) 2022-09-10T14:00:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:00 smithi110 ceph-mon[97633]: pgmap v18261: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:00:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:00 smithi104 ceph-mon[97044]: pgmap v18261: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:01.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:00:01.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:01.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:01.103090+0000) 2022-09-10T14:00:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:01.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:01.484709+0000) 2022-09-10T14:00:02.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:02.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:02.103296+0000) 2022-09-10T14:00:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:02.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:02.484924+0000) 2022-09-10T14:00:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:02 smithi110 ceph-mon[97633]: pgmap v18262: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:02 smithi104 ceph-mon[97044]: pgmap v18262: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:03.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:03.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:03.103543+0000) 2022-09-10T14:00:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:03.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:03.485190+0000) 2022-09-10T14:00:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:04.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:04.103764+0000) 2022-09-10T14:00:04.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:00:03.860+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:00:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:04.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:04.485409+0000) 2022-09-10T14:00:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:04 smithi110 ceph-mon[97633]: pgmap v18263: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:04 smithi104 ceph-mon[97044]: pgmap v18263: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:05.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:05.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:05.103960+0000) 2022-09-10T14:00:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:05.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:05.485636+0000) 2022-09-10T14:00:06.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:06.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:06.104178+0000) 2022-09-10T14:00:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:06.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:06.485897+0000) 2022-09-10T14:00:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:06 smithi110 ceph-mon[97633]: pgmap v18264: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:00:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:06 smithi104 ceph-mon[97044]: pgmap v18264: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:00:07.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:07.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:07.104361+0000) 2022-09-10T14:00:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:07.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:07.486139+0000) 2022-09-10T14:00:08.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:08.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:08.104602+0000) 2022-09-10T14:00:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:08.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:08.486317+0000) 2022-09-10T14:00:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:08 smithi110 ceph-mon[97633]: pgmap v18265: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:00:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:08 smithi104 ceph-mon[97044]: pgmap v18265: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:00:09.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:09.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:09.104863+0000) 2022-09-10T14:00:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:09.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:09.486545+0000) 2022-09-10T14:00:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:09 smithi110 ceph-mon[97633]: osdmap e1222: 12 total, 12 up, 12 in 2022-09-10T14:00:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:09 smithi104 ceph-mon[97044]: osdmap e1222: 12 total, 12 up, 12 in 2022-09-10T14:00:10.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:10.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:10.105137+0000) 2022-09-10T14:00:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:10.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:10.486792+0000) 2022-09-10T14:00:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:10 smithi110 ceph-mon[97633]: pgmap v18267: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:00:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:10 smithi104 ceph-mon[97044]: pgmap v18267: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:00:11.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:11.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:11.105286+0000) 2022-09-10T14:00:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:11.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:11.487014+0000) 2022-09-10T14:00:12.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:12.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:12.105529+0000) 2022-09-10T14:00:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:12.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:12.487219+0000) 2022-09-10T14:00:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:12 smithi110 ceph-mon[97633]: pgmap v18268: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:00:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:12 smithi104 ceph-mon[97044]: pgmap v18268: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:00:13.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:13.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:13.105781+0000) 2022-09-10T14:00:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:13.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:13.487427+0000) 2022-09-10T14:00:14.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:14.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:14.105947+0000) 2022-09-10T14:00:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:14.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:14.487679+0000) 2022-09-10T14:00:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:14 smithi110 ceph-mon[97633]: pgmap v18269: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:00:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:14 smithi104 ceph-mon[97044]: pgmap v18269: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:00:15.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:15.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:15.106123+0000) 2022-09-10T14:00:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:15.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:15.487903+0000) 2022-09-10T14:00:16.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:16.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:16.106315+0000) 2022-09-10T14:00:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:16.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:16.488168+0000) 2022-09-10T14:00:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:16 smithi110 ceph-mon[97633]: pgmap v18270: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:00:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:16 smithi110 ceph-mon[97633]: osdmap e1223: 12 total, 12 up, 12 in 2022-09-10T14:00:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:16 smithi104 ceph-mon[97044]: pgmap v18270: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:00:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:16 smithi104 ceph-mon[97044]: osdmap e1223: 12 total, 12 up, 12 in 2022-09-10T14:00:17.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:17.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:17.106520+0000) 2022-09-10T14:00:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:17.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:17.488342+0000) 2022-09-10T14:00:18.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:18.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:18.106738+0000) 2022-09-10T14:00:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:18.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:18.488528+0000) 2022-09-10T14:00:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:18 smithi110 ceph-mon[97633]: pgmap v18272: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T14:00:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:18 smithi104 ceph-mon[97044]: pgmap v18272: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 251 B/s wr, 12 op/s 2022-09-10T14:00:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:19.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:19.107005+0000) 2022-09-10T14:00:19.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:00:18.868+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:00:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:19.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:19.488750+0000) 2022-09-10T14:00:20.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:20.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:20.107251+0000) 2022-09-10T14:00:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:20.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:20.488971+0000) 2022-09-10T14:00:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:20 smithi110 ceph-mon[97633]: pgmap v18273: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:00:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:20 smithi104 ceph-mon[97044]: pgmap v18273: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:00:21.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:21.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:21.107448+0000) 2022-09-10T14:00:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:21.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:21.489220+0000) 2022-09-10T14:00:22.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:22.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:22.107633+0000) 2022-09-10T14:00:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:22.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:22.489508+0000) 2022-09-10T14:00:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:22 smithi110 ceph-mon[97633]: pgmap v18274: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:00:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:22 smithi104 ceph-mon[97044]: pgmap v18274: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:00:23.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:23.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:23.107844+0000) 2022-09-10T14:00:23.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:23.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:23.489690+0000) 2022-09-10T14:00:24.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:24.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:24.108088+0000) 2022-09-10T14:00:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:24.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:24.489830+0000) 2022-09-10T14:00:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:24 smithi110 ceph-mon[97633]: pgmap v18275: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:00:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:24 smithi104 ceph-mon[97044]: pgmap v18275: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:00:25.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:25.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:25.108297+0000) 2022-09-10T14:00:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:25.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:25.490068+0000) 2022-09-10T14:00:26.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:26.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:26.108511+0000) 2022-09-10T14:00:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:26.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:26.490335+0000) 2022-09-10T14:00:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:26 smithi110 ceph-mon[97633]: pgmap v18276: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:00:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:26 smithi104 ceph-mon[97044]: pgmap v18276: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:00:27.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:27.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:27.108697+0000) 2022-09-10T14:00:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:27.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:27.490618+0000) 2022-09-10T14:00:28.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:28.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:28.108933+0000) 2022-09-10T14:00:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:28.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:28.490876+0000) 2022-09-10T14:00:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:28 smithi110 ceph-mon[97633]: pgmap v18277: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:00:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:28 smithi104 ceph-mon[97044]: pgmap v18277: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:00:29.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:29.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:29.109207+0000) 2022-09-10T14:00:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:29.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:29.491065+0000) 2022-09-10T14:00:30.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:30.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:30.109354+0000) 2022-09-10T14:00:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:30.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:30.491301+0000) 2022-09-10T14:00:30.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:30 smithi110 ceph-mon[97633]: pgmap v18278: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:30 smithi104 ceph-mon[97044]: pgmap v18278: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:31.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:31.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:31.109547+0000) 2022-09-10T14:00:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:31.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:31.491505+0000) 2022-09-10T14:00:32.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:32.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:32.109720+0000) 2022-09-10T14:00:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:32.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:32.491689+0000) 2022-09-10T14:00:32.906 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:32 smithi110 ceph-mon[97633]: pgmap v18279: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:32 smithi104 ceph-mon[97044]: pgmap v18279: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:33.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:33.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:33.109917+0000) 2022-09-10T14:00:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:33.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:33.491933+0000) 2022-09-10T14:00:34.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:00:33.883+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:00:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:34.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:34.110124+0000) 2022-09-10T14:00:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:34.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:34.492214+0000) 2022-09-10T14:00:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:34 smithi104 ceph-mon[97044]: pgmap v18280: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:34 smithi110 ceph-mon[97633]: pgmap v18280: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:35.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:35.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:35.110306+0000) 2022-09-10T14:00:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:35.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:35.492455+0000) 2022-09-10T14:00:36.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:36.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:36.110513+0000) 2022-09-10T14:00:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:36.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:36.492670+0000) 2022-09-10T14:00:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:36 smithi104 ceph-mon[97044]: pgmap v18281: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:36 smithi110 ceph-mon[97633]: pgmap v18281: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:37.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:37.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:37.110761+0000) 2022-09-10T14:00:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:37.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:37.492897+0000) 2022-09-10T14:00:38.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:38.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:38.110973+0000) 2022-09-10T14:00:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:38.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:38.493126+0000) 2022-09-10T14:00:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:38 smithi104 ceph-mon[97044]: pgmap v18282: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:38 smithi110 ceph-mon[97633]: pgmap v18282: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:39.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:39.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:39.111246+0000) 2022-09-10T14:00:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:39.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:39.493346+0000) 2022-09-10T14:00:40.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:40.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:40.111512+0000) 2022-09-10T14:00:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:40.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:40.493608+0000) 2022-09-10T14:00:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:40 smithi104 ceph-mon[97044]: pgmap v18283: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:40 smithi110 ceph-mon[97633]: pgmap v18283: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:41.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:41.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:41.111754+0000) 2022-09-10T14:00:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:41.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:41.493846+0000) 2022-09-10T14:00:42.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:42.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:42.112026+0000) 2022-09-10T14:00:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:42.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:42.494126+0000) 2022-09-10T14:00:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:42 smithi104 ceph-mon[97044]: pgmap v18284: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:42 smithi110 ceph-mon[97633]: pgmap v18284: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:43.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:43.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:43.112194+0000) 2022-09-10T14:00:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:43.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:43.494384+0000) 2022-09-10T14:00:44.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:44.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:44.112339+0000) 2022-09-10T14:00:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:44.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:44.494624+0000) 2022-09-10T14:00:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:44 smithi104 ceph-mon[97044]: pgmap v18285: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:44 smithi110 ceph-mon[97633]: pgmap v18285: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:45.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:45.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:45.112559+0000) 2022-09-10T14:00:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:45.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:45.494841+0000) 2022-09-10T14:00:46.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:46.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:46.112812+0000) 2022-09-10T14:00:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:46.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:46.495058+0000) 2022-09-10T14:00:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:46 smithi104 ceph-mon[97044]: pgmap v18286: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:00:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:00:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:00:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:00:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:46 smithi110 ceph-mon[97633]: pgmap v18286: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:00:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:00:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:00:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:00:47.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:47.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:47.113040+0000) 2022-09-10T14:00:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:47.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:47.495198+0000) 2022-09-10T14:00:48.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:48.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:48.113225+0000) 2022-09-10T14:00:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:48.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:48.495415+0000) 2022-09-10T14:00:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:48 smithi104 ceph-mon[97044]: pgmap v18287: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:48 smithi110 ceph-mon[97633]: pgmap v18287: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:49.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:00:48.897+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:00:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:49.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:49.113439+0000) 2022-09-10T14:00:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:49.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:49.495598+0000) 2022-09-10T14:00:50.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:50.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:50.113676+0000) 2022-09-10T14:00:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:50.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:50.495828+0000) 2022-09-10T14:00:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:50 smithi104 ceph-mon[97044]: pgmap v18288: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:50 smithi110 ceph-mon[97633]: pgmap v18288: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:51.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:51.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:51.113933+0000) 2022-09-10T14:00:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:51.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:51.496082+0000) 2022-09-10T14:00:52.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:52.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:52.114149+0000) 2022-09-10T14:00:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:52.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:52.496335+0000) 2022-09-10T14:00:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:52 smithi104 ceph-mon[97044]: pgmap v18289: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:52 smithi110 ceph-mon[97633]: pgmap v18289: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:53.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:53.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:53.114374+0000) 2022-09-10T14:00:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:53.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:53.496555+0000) 2022-09-10T14:00:54.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:54.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:54.114559+0000) 2022-09-10T14:00:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:54.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:54.496722+0000) 2022-09-10T14:00:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:54 smithi104 ceph-mon[97044]: pgmap v18290: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:54 smithi110 ceph-mon[97633]: pgmap v18290: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:55.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:55.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:55.114732+0000) 2022-09-10T14:00:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:55.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:55.496935+0000) 2022-09-10T14:00:56.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:56.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:56.114943+0000) 2022-09-10T14:00:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:56.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:56.497131+0000) 2022-09-10T14:00:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:56 smithi104 ceph-mon[97044]: pgmap v18291: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:56 smithi110 ceph-mon[97633]: pgmap v18291: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:00:57.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:57.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:57.115171+0000) 2022-09-10T14:00:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:57.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:57.497364+0000) 2022-09-10T14:00:58.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:58.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:58.115354+0000) 2022-09-10T14:00:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:58.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:58.497553+0000) 2022-09-10T14:00:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:00:58 smithi104 ceph-mon[97044]: pgmap v18292: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:00:58 smithi110 ceph-mon[97633]: pgmap v18292: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:00:59.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:00:59.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:59.115604+0000) 2022-09-10T14:00:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:00:59.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:00:59.497752+0000) 2022-09-10T14:01:00.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:00.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:00.115844+0000) 2022-09-10T14:01:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:00.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:00.497946+0000) 2022-09-10T14:01:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:00 smithi104 ceph-mon[97044]: pgmap v18293: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:00 smithi110 ceph-mon[97633]: pgmap v18293: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:01.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:01.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:01.115996+0000) 2022-09-10T14:01:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:01.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:01.498201+0000) 2022-09-10T14:01:02.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:02.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:02.116188+0000) 2022-09-10T14:01:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:02.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:02.498394+0000) 2022-09-10T14:01:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:02 smithi104 ceph-mon[97044]: pgmap v18294: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:02 smithi110 ceph-mon[97633]: pgmap v18294: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:03.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:03.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:03.116327+0000) 2022-09-10T14:01:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:03.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:03.498611+0000) 2022-09-10T14:01:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:04.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:04.116473+0000) 2022-09-10T14:01:04.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:01:03.901+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:01:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:04.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:04.498859+0000) 2022-09-10T14:01:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:04 smithi104 ceph-mon[97044]: pgmap v18295: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:04 smithi110 ceph-mon[97633]: pgmap v18295: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:05.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:05.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:05.116708+0000) 2022-09-10T14:01:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:05.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:05.499027+0000) 2022-09-10T14:01:06.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:06.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:06.116863+0000) 2022-09-10T14:01:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:06.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:06.499236+0000) 2022-09-10T14:01:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:06 smithi104 ceph-mon[97044]: pgmap v18296: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:01:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:06 smithi110 ceph-mon[97633]: pgmap v18296: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:01:07.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:07.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:07.117003+0000) 2022-09-10T14:01:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:07.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:07.499511+0000) 2022-09-10T14:01:08.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:08.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:08.117217+0000) 2022-09-10T14:01:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:08.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:08.499711+0000) 2022-09-10T14:01:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:08 smithi104 ceph-mon[97044]: pgmap v18297: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:01:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:08 smithi110 ceph-mon[97633]: pgmap v18297: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:01:09.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:09.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:09.117449+0000) 2022-09-10T14:01:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:09.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:09.499963+0000) 2022-09-10T14:01:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:09 smithi104 ceph-mon[97044]: osdmap e1224: 12 total, 12 up, 12 in 2022-09-10T14:01:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:09 smithi104 ceph-mon[97044]: pgmap v18299: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:01:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:09 smithi110 ceph-mon[97633]: osdmap e1224: 12 total, 12 up, 12 in 2022-09-10T14:01:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:09 smithi110 ceph-mon[97633]: pgmap v18299: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:01:10.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:10.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:10.117631+0000) 2022-09-10T14:01:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:10.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:10.500210+0000) 2022-09-10T14:01:11.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:11.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:11.117880+0000) 2022-09-10T14:01:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:11.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:11.500492+0000) 2022-09-10T14:01:12.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:12.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:12.118085+0000) 2022-09-10T14:01:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:12 smithi104 ceph-mon[97044]: pgmap v18300: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:01:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:12 smithi104 ceph-mon[97044]: osdmap e1225: 12 total, 12 up, 12 in 2022-09-10T14:01:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:12 smithi110 ceph-mon[97633]: pgmap v18300: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:01:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:12 smithi110 ceph-mon[97633]: osdmap e1225: 12 total, 12 up, 12 in 2022-09-10T14:01:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:12.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:12.500712+0000) 2022-09-10T14:01:13.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:13.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:13.118282+0000) 2022-09-10T14:01:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:13.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:13.500930+0000) 2022-09-10T14:01:14.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:14.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:14.118507+0000) 2022-09-10T14:01:14.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:14 smithi104 ceph-mon[97044]: pgmap v18302: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:01:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:14 smithi110 ceph-mon[97633]: pgmap v18302: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:01:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:14.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:14.501069+0000) 2022-09-10T14:01:15.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:15.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:15.118650+0000) 2022-09-10T14:01:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:15.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:15.501310+0000) 2022-09-10T14:01:16.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:16.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:16.118839+0000) 2022-09-10T14:01:16.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:16 smithi104 ceph-mon[97044]: pgmap v18303: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:01:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:16 smithi110 ceph-mon[97633]: pgmap v18303: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:01:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:16.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:16.501538+0000) 2022-09-10T14:01:17.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:17.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:17.119004+0000) 2022-09-10T14:01:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:17.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:17.501751+0000) 2022-09-10T14:01:18.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:18.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:18.119260+0000) 2022-09-10T14:01:18.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:18 smithi104 ceph-mon[97044]: pgmap v18304: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T14:01:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:18 smithi110 ceph-mon[97633]: pgmap v18304: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 254 B/s wr, 12 op/s 2022-09-10T14:01:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:18.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:18.501932+0000) 2022-09-10T14:01:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:19.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:19.119540+0000) 2022-09-10T14:01:19.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:01:18.910+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:01:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:19.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:19.502191+0000) 2022-09-10T14:01:20.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:20.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:20.119744+0000) 2022-09-10T14:01:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:20 smithi104 ceph-mon[97044]: pgmap v18305: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:01:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:20 smithi110 ceph-mon[97633]: pgmap v18305: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:01:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:20.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:20.502383+0000) 2022-09-10T14:01:21.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:21.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:21.119908+0000) 2022-09-10T14:01:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:21.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:21.502572+0000) 2022-09-10T14:01:22.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:22.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:22.120153+0000) 2022-09-10T14:01:22.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:22 smithi104 ceph-mon[97044]: pgmap v18306: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:01:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:22 smithi110 ceph-mon[97633]: pgmap v18306: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:01:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:22.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:22.502776+0000) 2022-09-10T14:01:23.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:23.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:23.120347+0000) 2022-09-10T14:01:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:23.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:23.503011+0000) 2022-09-10T14:01:24.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:24.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:24.120551+0000) 2022-09-10T14:01:24.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:24 smithi104 ceph-mon[97044]: pgmap v18307: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T14:01:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:24 smithi110 ceph-mon[97633]: pgmap v18307: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 88 B/s wr, 8 op/s 2022-09-10T14:01:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:24.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:24.503262+0000) 2022-09-10T14:01:25.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:25.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:25.120773+0000) 2022-09-10T14:01:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:25.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:25.503502+0000) 2022-09-10T14:01:26.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:26.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:26.121029+0000) 2022-09-10T14:01:26.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:26 smithi104 ceph-mon[97044]: pgmap v18308: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:01:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:26 smithi110 ceph-mon[97633]: pgmap v18308: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:01:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:26.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:26.503749+0000) 2022-09-10T14:01:27.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:27.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:27.121281+0000) 2022-09-10T14:01:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:27.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:27.504031+0000) 2022-09-10T14:01:28.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:28.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:28.121561+0000) 2022-09-10T14:01:28.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:28 smithi104 ceph-mon[97044]: pgmap v18309: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:28 smithi110 ceph-mon[97633]: pgmap v18309: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:28.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:28.504265+0000) 2022-09-10T14:01:29.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:29.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:29.121784+0000) 2022-09-10T14:01:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:29.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:29.504525+0000) 2022-09-10T14:01:30.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:30.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:30.121980+0000) 2022-09-10T14:01:30.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:30 smithi104 ceph-mon[97044]: pgmap v18310: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:30 smithi110 ceph-mon[97633]: pgmap v18310: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:30.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:30.504765+0000) 2022-09-10T14:01:31.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:31.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:31.122183+0000) 2022-09-10T14:01:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:31.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:31.505036+0000) 2022-09-10T14:01:32.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:32.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:32.122409+0000) 2022-09-10T14:01:32.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:32 smithi104 ceph-mon[97044]: pgmap v18311: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:32 smithi110 ceph-mon[97633]: pgmap v18311: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:32.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:32.505318+0000) 2022-09-10T14:01:33.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:33.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:33.122586+0000) 2022-09-10T14:01:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:33.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:33.505497+0000) 2022-09-10T14:01:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:34.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:34.122805+0000) 2022-09-10T14:01:34.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:01:33.923+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:01:34.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:34 smithi104 ceph-mon[97044]: pgmap v18312: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:34 smithi110 ceph-mon[97633]: pgmap v18312: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:34.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:34.505756+0000) 2022-09-10T14:01:35.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:35.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:35.123061+0000) 2022-09-10T14:01:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:35.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:35.506000+0000) 2022-09-10T14:01:36.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:36.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:36.123334+0000) 2022-09-10T14:01:36.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:36 smithi104 ceph-mon[97044]: pgmap v18313: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:36 smithi110 ceph-mon[97633]: pgmap v18313: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:36.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:36.506188+0000) 2022-09-10T14:01:37.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:37.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:37.123519+0000) 2022-09-10T14:01:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:37.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:37.506429+0000) 2022-09-10T14:01:38.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:38.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:38.123657+0000) 2022-09-10T14:01:38.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:38 smithi104 ceph-mon[97044]: pgmap v18314: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:38 smithi110 ceph-mon[97633]: pgmap v18314: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:38.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:38.506608+0000) 2022-09-10T14:01:39.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:39.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:39.123898+0000) 2022-09-10T14:01:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:39.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:39.506876+0000) 2022-09-10T14:01:40.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:40.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:40.124129+0000) 2022-09-10T14:01:40.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:40 smithi104 ceph-mon[97044]: pgmap v18315: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:40 smithi110 ceph-mon[97633]: pgmap v18315: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:40.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:40.507155+0000) 2022-09-10T14:01:41.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:41.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:41.124308+0000) 2022-09-10T14:01:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:41.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:41.507379+0000) 2022-09-10T14:01:42.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:42.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:42.124486+0000) 2022-09-10T14:01:42.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:42 smithi104 ceph-mon[97044]: pgmap v18316: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:42 smithi110 ceph-mon[97633]: pgmap v18316: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:42.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:42.507636+0000) 2022-09-10T14:01:43.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:43.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:43.124711+0000) 2022-09-10T14:01:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:43.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:43.507823+0000) 2022-09-10T14:01:44.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:44.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:44.124965+0000) 2022-09-10T14:01:44.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:44 smithi104 ceph-mon[97044]: pgmap v18317: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:44 smithi110 ceph-mon[97633]: pgmap v18317: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:44.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:44.508067+0000) 2022-09-10T14:01:45.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:45.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:45.125139+0000) 2022-09-10T14:01:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:45.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:45.508301+0000) 2022-09-10T14:01:46.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:46.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:46.125369+0000) 2022-09-10T14:01:46.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:46 smithi104 ceph-mon[97044]: pgmap v18318: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:46 smithi110 ceph-mon[97633]: pgmap v18318: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:46.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:46.508526+0000) 2022-09-10T14:01:47.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:47.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:47.125626+0000) 2022-09-10T14:01:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:01:47.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:01:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:01:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:01:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:01:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:01:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:01:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:01:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:47.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:47.508726+0000) 2022-09-10T14:01:48.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:48.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:48.125853+0000) 2022-09-10T14:01:48.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:48 smithi104 ceph-mon[97044]: pgmap v18319: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:48 smithi110 ceph-mon[97633]: pgmap v18319: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:48.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:48.508975+0000) 2022-09-10T14:01:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:49.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:49.126077+0000) 2022-09-10T14:01:49.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:01:48.937+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:01:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:49.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:49.509234+0000) 2022-09-10T14:01:50.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:50.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:50.126273+0000) 2022-09-10T14:01:50.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:50 smithi104 ceph-mon[97044]: pgmap v18320: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:50 smithi110 ceph-mon[97633]: pgmap v18320: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:50.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:50.509511+0000) 2022-09-10T14:01:51.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:51.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:51.126481+0000) 2022-09-10T14:01:51.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:51.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:51.509774+0000) 2022-09-10T14:01:52.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:52.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:52.126753+0000) 2022-09-10T14:01:52.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:52 smithi104 ceph-mon[97044]: pgmap v18321: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:52 smithi110 ceph-mon[97633]: pgmap v18321: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:52.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:52.509935+0000) 2022-09-10T14:01:53.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:53.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:53.126933+0000) 2022-09-10T14:01:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:53.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:53.510246+0000) 2022-09-10T14:01:54.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:54.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:54.127175+0000) 2022-09-10T14:01:54.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:54 smithi104 ceph-mon[97044]: pgmap v18322: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:54 smithi110 ceph-mon[97633]: pgmap v18322: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:54.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:54.510461+0000) 2022-09-10T14:01:55.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:55.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:55.127350+0000) 2022-09-10T14:01:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:55.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:55.510722+0000) 2022-09-10T14:01:56.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:56.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:56.127580+0000) 2022-09-10T14:01:56.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:56 smithi104 ceph-mon[97044]: pgmap v18323: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:56 smithi110 ceph-mon[97633]: pgmap v18323: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:01:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:56.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:56.510976+0000) 2022-09-10T14:01:57.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:57.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:57.127823+0000) 2022-09-10T14:01:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:57.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:57.511175+0000) 2022-09-10T14:01:58.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:58.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:58.128073+0000) 2022-09-10T14:01:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:01:58 smithi104 ceph-mon[97044]: pgmap v18324: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:01:58 smithi110 ceph-mon[97633]: pgmap v18324: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:01:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:58.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:58.511336+0000) 2022-09-10T14:01:59.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:01:59.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:59.128384+0000) 2022-09-10T14:01:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:01:59.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:01:59.511566+0000) 2022-09-10T14:02:00.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:00.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:00.128519+0000) 2022-09-10T14:02:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:00 smithi104 ceph-mon[97044]: pgmap v18325: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:00 smithi110 ceph-mon[97633]: pgmap v18325: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:00.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:00.511795+0000) 2022-09-10T14:02:01.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:01.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:01.128731+0000) 2022-09-10T14:02:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:01.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:01.512043+0000) 2022-09-10T14:02:02.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:02.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:02.128947+0000) 2022-09-10T14:02:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:02 smithi104 ceph-mon[97044]: pgmap v18326: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:02 smithi110 ceph-mon[97633]: pgmap v18326: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:02.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:02.512324+0000) 2022-09-10T14:02:03.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:03.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:03.129207+0000) 2022-09-10T14:02:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:03.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:03.512588+0000) 2022-09-10T14:02:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:04.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:04.129435+0000) 2022-09-10T14:02:04.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:02:03.953+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:02:04.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:04 smithi104 ceph-mon[97044]: pgmap v18327: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:04 smithi110 ceph-mon[97633]: pgmap v18327: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:04.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:04.512786+0000) 2022-09-10T14:02:05.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:05.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:05.129625+0000) 2022-09-10T14:02:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:05.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:05.513034+0000) 2022-09-10T14:02:06.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:06.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:06.129838+0000) 2022-09-10T14:02:06.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:06 smithi104 ceph-mon[97044]: pgmap v18328: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:02:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:06 smithi110 ceph-mon[97633]: pgmap v18328: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:02:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:06.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:06.513266+0000) 2022-09-10T14:02:07.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:07.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:07.130122+0000) 2022-09-10T14:02:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:07.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:07.513427+0000) 2022-09-10T14:02:08.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:08.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:08.130321+0000) 2022-09-10T14:02:08.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:08 smithi104 ceph-mon[97044]: pgmap v18329: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:02:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:08 smithi110 ceph-mon[97633]: pgmap v18329: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:02:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:08.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:08.513690+0000) 2022-09-10T14:02:09.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:09.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:09.130586+0000) 2022-09-10T14:02:09.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:09 smithi104 ceph-mon[97044]: osdmap e1226: 12 total, 12 up, 12 in 2022-09-10T14:02:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:09 smithi110 ceph-mon[97633]: osdmap e1226: 12 total, 12 up, 12 in 2022-09-10T14:02:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:09.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:09.513918+0000) 2022-09-10T14:02:10.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:10.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:10.130747+0000) 2022-09-10T14:02:10.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:10 smithi104 ceph-mon[97044]: pgmap v18331: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:02:10.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:10 smithi110 ceph-mon[97633]: pgmap v18331: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:02:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:10.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:10.514187+0000) 2022-09-10T14:02:11.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:11.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:11.130948+0000) 2022-09-10T14:02:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:11.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:11.514392+0000) 2022-09-10T14:02:12.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:12.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:12.131217+0000) 2022-09-10T14:02:12.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:12 smithi104 ceph-mon[97044]: pgmap v18332: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:02:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:12 smithi110 ceph-mon[97633]: pgmap v18332: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:02:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:12.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:12.514622+0000) 2022-09-10T14:02:13.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:13.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:13.131456+0000) 2022-09-10T14:02:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:13.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:13.514800+0000) 2022-09-10T14:02:14.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:14.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:14.131730+0000) 2022-09-10T14:02:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:14 smithi110 ceph-mon[97633]: pgmap v18333: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:02:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:14 smithi104 ceph-mon[97044]: pgmap v18333: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:02:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:14.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:14.515061+0000) 2022-09-10T14:02:15.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:15.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:15.132007+0000) 2022-09-10T14:02:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:15.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:15.515335+0000) 2022-09-10T14:02:16.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:16.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:16.132246+0000) 2022-09-10T14:02:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:16 smithi110 ceph-mon[97633]: pgmap v18334: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:02:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:16 smithi110 ceph-mon[97633]: osdmap e1227: 12 total, 12 up, 12 in 2022-09-10T14:02:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:16 smithi104 ceph-mon[97044]: pgmap v18334: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:02:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:16 smithi104 ceph-mon[97044]: osdmap e1227: 12 total, 12 up, 12 in 2022-09-10T14:02:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:16.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:16.515524+0000) 2022-09-10T14:02:17.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:17.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:17.132424+0000) 2022-09-10T14:02:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:17.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:17.515758+0000) 2022-09-10T14:02:18.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:18.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:18.132578+0000) 2022-09-10T14:02:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:18 smithi110 ceph-mon[97633]: pgmap v18336: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T14:02:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:18 smithi104 ceph-mon[97044]: pgmap v18336: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T14:02:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:18.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:18.515989+0000) 2022-09-10T14:02:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:19.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:19.132791+0000) 2022-09-10T14:02:19.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:02:18.959+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:02:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:19.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:19.516200+0000) 2022-09-10T14:02:20.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:20.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:20.133066+0000) 2022-09-10T14:02:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:20 smithi110 ceph-mon[97633]: pgmap v18337: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:02:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:20 smithi104 ceph-mon[97044]: pgmap v18337: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:02:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:20.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:20.516423+0000) 2022-09-10T14:02:21.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:21.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:21.133238+0000) 2022-09-10T14:02:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:21.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:21.516611+0000) 2022-09-10T14:02:22.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:22.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:22.133468+0000) 2022-09-10T14:02:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:22 smithi110 ceph-mon[97633]: pgmap v18338: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:02:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:22 smithi104 ceph-mon[97044]: pgmap v18338: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:02:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:22.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:22.516781+0000) 2022-09-10T14:02:23.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:23.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:23.133733+0000) 2022-09-10T14:02:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:23.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:23.517003+0000) 2022-09-10T14:02:24.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:24.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:24.133968+0000) 2022-09-10T14:02:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:24 smithi110 ceph-mon[97633]: pgmap v18339: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:02:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:24 smithi104 ceph-mon[97044]: pgmap v18339: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:02:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:24.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:24.517193+0000) 2022-09-10T14:02:25.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:25.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:25.134212+0000) 2022-09-10T14:02:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:25.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:25.517357+0000) 2022-09-10T14:02:26.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:26.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:26.134450+0000) 2022-09-10T14:02:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:26 smithi110 ceph-mon[97633]: pgmap v18340: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:02:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:26 smithi104 ceph-mon[97044]: pgmap v18340: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:02:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:26.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:26.517590+0000) 2022-09-10T14:02:27.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:27.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:27.134708+0000) 2022-09-10T14:02:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:27.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:27.517847+0000) 2022-09-10T14:02:28.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:28.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:28.134887+0000) 2022-09-10T14:02:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:28 smithi110 ceph-mon[97633]: pgmap v18341: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:02:28.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:28 smithi104 ceph-mon[97044]: pgmap v18341: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:02:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:28.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:28.518088+0000) 2022-09-10T14:02:29.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:29.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:29.135027+0000) 2022-09-10T14:02:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:29.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:29.518339+0000) 2022-09-10T14:02:30.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:30.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:30.135244+0000) 2022-09-10T14:02:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:30 smithi110 ceph-mon[97633]: pgmap v18342: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:30 smithi104 ceph-mon[97044]: pgmap v18342: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:30.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:30.518490+0000) 2022-09-10T14:02:31.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:31.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:31.135424+0000) 2022-09-10T14:02:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:31.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:31.518716+0000) 2022-09-10T14:02:32.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:32.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:32.135687+0000) 2022-09-10T14:02:32.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:32 smithi110 ceph-mon[97633]: pgmap v18343: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:32 smithi104 ceph-mon[97044]: pgmap v18343: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:32.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:32.518928+0000) 2022-09-10T14:02:33.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:33.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:33.135885+0000) 2022-09-10T14:02:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:33.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:33.519082+0000) 2022-09-10T14:02:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:34.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:34.136161+0000) 2022-09-10T14:02:34.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:02:33.961+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:02:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:34 smithi110 ceph-mon[97633]: pgmap v18344: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:34 smithi104 ceph-mon[97044]: pgmap v18344: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:34.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:34.519304+0000) 2022-09-10T14:02:35.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:35.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:35.136310+0000) 2022-09-10T14:02:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:35.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:35.519537+0000) 2022-09-10T14:02:36.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:36.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:36.136508+0000) 2022-09-10T14:02:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:36 smithi110 ceph-mon[97633]: pgmap v18345: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:36 smithi104 ceph-mon[97044]: pgmap v18345: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:36.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:36.519774+0000) 2022-09-10T14:02:37.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:37.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:37.136761+0000) 2022-09-10T14:02:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:37.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:37.520005+0000) 2022-09-10T14:02:38.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:38.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:38.137019+0000) 2022-09-10T14:02:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:38 smithi110 ceph-mon[97633]: pgmap v18346: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:38 smithi104 ceph-mon[97044]: pgmap v18346: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:38.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:38.520255+0000) 2022-09-10T14:02:39.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:39.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:39.137274+0000) 2022-09-10T14:02:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:39.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:39.520500+0000) 2022-09-10T14:02:40.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:40.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:40.137452+0000) 2022-09-10T14:02:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:40 smithi110 ceph-mon[97633]: pgmap v18347: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:40 smithi104 ceph-mon[97044]: pgmap v18347: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:40.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:40.520655+0000) 2022-09-10T14:02:41.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:41.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:41.137655+0000) 2022-09-10T14:02:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:41.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:41.520882+0000) 2022-09-10T14:02:42.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:42.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:42.137880+0000) 2022-09-10T14:02:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:42 smithi110 ceph-mon[97633]: pgmap v18348: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:42 smithi104 ceph-mon[97044]: pgmap v18348: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:42.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:42.521175+0000) 2022-09-10T14:02:43.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:43.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:43.138124+0000) 2022-09-10T14:02:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:43.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:43.521393+0000) 2022-09-10T14:02:44.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:44.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:44.138289+0000) 2022-09-10T14:02:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:44 smithi110 ceph-mon[97633]: pgmap v18349: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:44 smithi104 ceph-mon[97044]: pgmap v18349: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:44.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:44.521595+0000) 2022-09-10T14:02:45.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:45.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:45.138513+0000) 2022-09-10T14:02:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:45.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:45.521860+0000) 2022-09-10T14:02:46.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:46.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:46.138768+0000) 2022-09-10T14:02:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:46 smithi110 ceph-mon[97633]: pgmap v18350: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:46 smithi104 ceph-mon[97044]: pgmap v18350: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:46.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:46.522122+0000) 2022-09-10T14:02:47.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:47.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:47.138949+0000) 2022-09-10T14:02:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:02:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:02:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:02:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:02:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:02:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:02:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:47.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:47.522357+0000) 2022-09-10T14:02:48.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:48.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:48.139180+0000) 2022-09-10T14:02:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:48 smithi110 ceph-mon[97633]: pgmap v18351: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:48 smithi104 ceph-mon[97044]: pgmap v18351: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:48.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:48.522508+0000) 2022-09-10T14:02:49.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:49.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:49.139328+0000) 2022-09-10T14:02:49.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:02:48.970+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:02:49.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:49.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:49.522656+0000) 2022-09-10T14:02:50.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:50.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:50.139540+0000) 2022-09-10T14:02:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:50 smithi110 ceph-mon[97633]: pgmap v18352: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:50 smithi104 ceph-mon[97044]: pgmap v18352: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:50.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:50.522895+0000) 2022-09-10T14:02:51.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:51.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:51.139735+0000) 2022-09-10T14:02:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:51.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:51.523138+0000) 2022-09-10T14:02:52.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:52.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:52.139925+0000) 2022-09-10T14:02:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:52 smithi110 ceph-mon[97633]: pgmap v18353: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:52 smithi104 ceph-mon[97044]: pgmap v18353: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:52.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:52.523325+0000) 2022-09-10T14:02:53.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:53.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:53.140213+0000) 2022-09-10T14:02:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:53.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:53.523478+0000) 2022-09-10T14:02:54.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:54.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:54.140357+0000) 2022-09-10T14:02:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:54 smithi110 ceph-mon[97633]: pgmap v18354: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:54 smithi104 ceph-mon[97044]: pgmap v18354: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:54.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:54.523704+0000) 2022-09-10T14:02:55.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:55.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:55.140517+0000) 2022-09-10T14:02:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:55.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:55.523943+0000) 2022-09-10T14:02:56.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:56.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:56.140706+0000) 2022-09-10T14:02:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:56 smithi110 ceph-mon[97633]: pgmap v18355: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:56 smithi104 ceph-mon[97044]: pgmap v18355: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:02:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:56.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:56.524191+0000) 2022-09-10T14:02:57.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:57.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:57.140939+0000) 2022-09-10T14:02:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:57.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:57.524350+0000) 2022-09-10T14:02:58.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:58.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:58.141198+0000) 2022-09-10T14:02:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:02:58 smithi110 ceph-mon[97633]: pgmap v18356: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:02:58 smithi104 ceph-mon[97044]: pgmap v18356: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:02:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:58.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:58.524539+0000) 2022-09-10T14:02:59.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:02:59.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:59.141399+0000) 2022-09-10T14:02:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:02:59.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:02:59.524767+0000) 2022-09-10T14:03:00.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:00.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:00.141606+0000) 2022-09-10T14:03:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:00 smithi110 ceph-mon[97633]: pgmap v18357: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:00 smithi104 ceph-mon[97044]: pgmap v18357: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:00.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:00.524969+0000) 2022-09-10T14:03:01.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:01.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:01.141818+0000) 2022-09-10T14:03:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:01.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:01.525179+0000) 2022-09-10T14:03:02.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:02.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:02.142056+0000) 2022-09-10T14:03:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:02 smithi110 ceph-mon[97633]: pgmap v18358: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:02 smithi104 ceph-mon[97044]: pgmap v18358: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:02.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:02.525364+0000) 2022-09-10T14:03:03.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:03.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:03.142162+0000) 2022-09-10T14:03:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:03.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:03.525625+0000) 2022-09-10T14:03:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:04.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:04.142302+0000) 2022-09-10T14:03:04.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:03:03.983+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:03:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:04 smithi110 ceph-mon[97633]: pgmap v18359: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:04 smithi104 ceph-mon[97044]: pgmap v18359: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:04.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:04.525803+0000) 2022-09-10T14:03:05.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:05.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:05.142479+0000) 2022-09-10T14:03:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:05.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:05.526036+0000) 2022-09-10T14:03:06.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:06.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:06.142701+0000) 2022-09-10T14:03:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:06 smithi110 ceph-mon[97633]: pgmap v18360: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:03:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:06 smithi104 ceph-mon[97044]: pgmap v18360: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:03:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:06.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:06.526261+0000) 2022-09-10T14:03:07.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:07.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:07.142879+0000) 2022-09-10T14:03:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:07.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:07.526423+0000) 2022-09-10T14:03:08.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:08.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:08.143082+0000) 2022-09-10T14:03:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:08 smithi110 ceph-mon[97633]: pgmap v18361: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:03:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:08 smithi104 ceph-mon[97044]: pgmap v18361: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:03:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:08.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:08.526647+0000) 2022-09-10T14:03:09.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:09.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:09.143323+0000) 2022-09-10T14:03:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:09 smithi110 ceph-mon[97633]: osdmap e1228: 12 total, 12 up, 12 in 2022-09-10T14:03:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:09 smithi104 ceph-mon[97044]: osdmap e1228: 12 total, 12 up, 12 in 2022-09-10T14:03:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:09.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:09.526831+0000) 2022-09-10T14:03:10.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:10.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:10.143577+0000) 2022-09-10T14:03:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:10 smithi110 ceph-mon[97633]: pgmap v18363: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:03:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:10 smithi104 ceph-mon[97044]: pgmap v18363: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:03:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:10.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:10.527053+0000) 2022-09-10T14:03:11.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:11.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:11.143843+0000) 2022-09-10T14:03:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:11.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:11.527307+0000) 2022-09-10T14:03:12.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:12.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:12.144076+0000) 2022-09-10T14:03:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:12 smithi110 ceph-mon[97633]: pgmap v18364: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:03:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:12 smithi104 ceph-mon[97044]: pgmap v18364: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:03:12.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:12.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:12.527534+0000) 2022-09-10T14:03:13.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:13.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:13.144328+0000) 2022-09-10T14:03:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:13.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:13.527783+0000) 2022-09-10T14:03:14.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:14.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:14.144480+0000) 2022-09-10T14:03:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:14 smithi110 ceph-mon[97633]: pgmap v18365: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:03:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:14 smithi104 ceph-mon[97044]: pgmap v18365: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:03:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:14.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:14.528031+0000) 2022-09-10T14:03:15.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:15.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:15.144683+0000) 2022-09-10T14:03:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:15.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:15.528293+0000) 2022-09-10T14:03:16.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:16.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:16.144948+0000) 2022-09-10T14:03:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:16 smithi110 ceph-mon[97633]: pgmap v18366: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:03:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:16 smithi110 ceph-mon[97633]: osdmap e1229: 12 total, 12 up, 12 in 2022-09-10T14:03:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:16 smithi104 ceph-mon[97044]: pgmap v18366: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:03:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:16 smithi104 ceph-mon[97044]: osdmap e1229: 12 total, 12 up, 12 in 2022-09-10T14:03:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:16.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:16.528532+0000) 2022-09-10T14:03:17.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:17.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:17.145092+0000) 2022-09-10T14:03:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:17.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:17.528763+0000) 2022-09-10T14:03:18.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:18.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:18.145347+0000) 2022-09-10T14:03:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:18 smithi110 ceph-mon[97633]: pgmap v18368: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T14:03:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:18 smithi104 ceph-mon[97044]: pgmap v18368: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T14:03:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:18.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:18.529015+0000) 2022-09-10T14:03:19.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:03:18.992+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:03:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:19.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:19.145565+0000) 2022-09-10T14:03:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:19.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:19.529246+0000) 2022-09-10T14:03:20.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:20.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:20.145739+0000) 2022-09-10T14:03:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:20 smithi104 ceph-mon[97044]: pgmap v18369: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:03:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:20.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:20.529481+0000) 2022-09-10T14:03:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:20 smithi110 ceph-mon[97633]: pgmap v18369: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:03:21.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:21.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:21.145944+0000) 2022-09-10T14:03:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:21.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:21.529685+0000) 2022-09-10T14:03:22.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:22.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:22.146154+0000) 2022-09-10T14:03:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:22 smithi104 ceph-mon[97044]: pgmap v18370: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:03:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:22.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:22.529887+0000) 2022-09-10T14:03:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:22 smithi110 ceph-mon[97633]: pgmap v18370: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:03:23.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:23.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:23.146318+0000) 2022-09-10T14:03:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:23.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:23.530139+0000) 2022-09-10T14:03:24.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:24.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:24.146505+0000) 2022-09-10T14:03:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:24 smithi104 ceph-mon[97044]: pgmap v18371: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:03:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:24.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:24.530385+0000) 2022-09-10T14:03:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:24 smithi110 ceph-mon[97633]: pgmap v18371: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:03:25.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:25.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:25.146712+0000) 2022-09-10T14:03:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:25.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:25.530617+0000) 2022-09-10T14:03:26.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:26.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:26.146928+0000) 2022-09-10T14:03:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:26 smithi104 ceph-mon[97044]: pgmap v18372: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:03:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:26.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:26.530778+0000) 2022-09-10T14:03:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:26 smithi110 ceph-mon[97633]: pgmap v18372: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:03:27.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:27.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:27.147155+0000) 2022-09-10T14:03:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:27.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:27.530979+0000) 2022-09-10T14:03:28.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:28.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:28.147320+0000) 2022-09-10T14:03:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:28 smithi104 ceph-mon[97044]: pgmap v18373: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:03:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:28.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:28.531188+0000) 2022-09-10T14:03:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:28 smithi110 ceph-mon[97633]: pgmap v18373: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:03:29.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:29.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:29.147453+0000) 2022-09-10T14:03:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:29.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:29.531425+0000) 2022-09-10T14:03:30.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:30.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:30.147610+0000) 2022-09-10T14:03:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:30 smithi104 ceph-mon[97044]: pgmap v18374: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:30.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:30.531616+0000) 2022-09-10T14:03:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:30 smithi110 ceph-mon[97633]: pgmap v18374: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:31.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:31.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:31.147881+0000) 2022-09-10T14:03:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:31.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:31.531845+0000) 2022-09-10T14:03:32.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:32.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:32.148056+0000) 2022-09-10T14:03:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:32 smithi104 ceph-mon[97044]: pgmap v18375: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:32.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:32.532058+0000) 2022-09-10T14:03:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:32 smithi110 ceph-mon[97633]: pgmap v18375: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:33.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:33.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:33.148185+0000) 2022-09-10T14:03:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:33.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:33.532285+0000) 2022-09-10T14:03:34.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:03:34.004+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:03:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:34.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:34.148348+0000) 2022-09-10T14:03:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:34 smithi104 ceph-mon[97044]: pgmap v18376: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:34.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:34.532540+0000) 2022-09-10T14:03:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:34 smithi110 ceph-mon[97633]: pgmap v18376: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:35.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:35.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:35.148524+0000) 2022-09-10T14:03:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:35.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:35.532791+0000) 2022-09-10T14:03:36.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:36.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:36.148754+0000) 2022-09-10T14:03:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:36 smithi104 ceph-mon[97044]: pgmap v18377: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:36.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:36.533023+0000) 2022-09-10T14:03:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:36 smithi110 ceph-mon[97633]: pgmap v18377: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:37.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:37.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:37.148997+0000) 2022-09-10T14:03:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:37.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:37.533277+0000) 2022-09-10T14:03:38.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:38.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:38.149232+0000) 2022-09-10T14:03:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:38 smithi104 ceph-mon[97044]: pgmap v18378: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:38.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:38.533500+0000) 2022-09-10T14:03:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:38 smithi110 ceph-mon[97633]: pgmap v18378: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:39.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:39.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:39.149497+0000) 2022-09-10T14:03:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:39.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:39.533759+0000) 2022-09-10T14:03:40.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:40.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:40.149669+0000) 2022-09-10T14:03:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:40 smithi104 ceph-mon[97044]: pgmap v18379: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:40.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:40.533922+0000) 2022-09-10T14:03:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:40 smithi110 ceph-mon[97633]: pgmap v18379: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:41.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:41.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:41.149794+0000) 2022-09-10T14:03:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:41.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:41.534161+0000) 2022-09-10T14:03:42.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:42.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:42.150014+0000) 2022-09-10T14:03:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:42 smithi104 ceph-mon[97044]: pgmap v18380: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:42.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:42.534335+0000) 2022-09-10T14:03:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:42 smithi110 ceph-mon[97633]: pgmap v18380: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:43.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:43.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:43.150277+0000) 2022-09-10T14:03:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:43.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:43.534563+0000) 2022-09-10T14:03:44.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:44.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:44.150485+0000) 2022-09-10T14:03:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:44 smithi104 ceph-mon[97044]: pgmap v18381: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:44.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:44.534773+0000) 2022-09-10T14:03:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:44 smithi110 ceph-mon[97633]: pgmap v18381: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:45.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:45.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:45.150678+0000) 2022-09-10T14:03:45.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:45.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:45.534987+0000) 2022-09-10T14:03:46.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:46.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:46.150947+0000) 2022-09-10T14:03:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:46 smithi104 ceph-mon[97044]: pgmap v18382: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:46.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:46.535240+0000) 2022-09-10T14:03:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:46 smithi110 ceph-mon[97633]: pgmap v18382: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:47.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:47.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:47.151173+0000) 2022-09-10T14:03:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:03:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:03:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:03:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:03:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:47.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:47.535417+0000) 2022-09-10T14:03:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:03:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:03:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:03:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:03:48.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:48.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:48.151356+0000) 2022-09-10T14:03:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:48 smithi104 ceph-mon[97044]: pgmap v18383: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:48.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:48.535611+0000) 2022-09-10T14:03:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:48 smithi110 ceph-mon[97633]: pgmap v18383: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:49.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:03:49.019+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:03:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:49.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:49.151627+0000) 2022-09-10T14:03:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:49.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:49.535843+0000) 2022-09-10T14:03:50.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:50.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:50.151897+0000) 2022-09-10T14:03:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:50 smithi104 ceph-mon[97044]: pgmap v18384: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:50.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:50.536147+0000) 2022-09-10T14:03:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:50 smithi110 ceph-mon[97633]: pgmap v18384: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:51.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:51.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:51.152037+0000) 2022-09-10T14:03:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:51.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:51.536322+0000) 2022-09-10T14:03:52.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:52.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:52.152178+0000) 2022-09-10T14:03:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:52 smithi104 ceph-mon[97044]: pgmap v18385: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:52.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:52.536556+0000) 2022-09-10T14:03:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:52 smithi110 ceph-mon[97633]: pgmap v18385: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:53.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:53.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:53.152419+0000) 2022-09-10T14:03:53.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:53.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:53.536807+0000) 2022-09-10T14:03:54.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:54.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:54.152673+0000) 2022-09-10T14:03:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:54 smithi104 ceph-mon[97044]: pgmap v18386: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:54.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:54.537039+0000) 2022-09-10T14:03:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:54 smithi110 ceph-mon[97633]: pgmap v18386: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:55.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:55.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:55.152949+0000) 2022-09-10T14:03:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:55.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:55.537298+0000) 2022-09-10T14:03:56.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:56.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:56.153259+0000) 2022-09-10T14:03:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:56 smithi104 ceph-mon[97044]: pgmap v18387: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:56.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:56.537466+0000) 2022-09-10T14:03:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:56 smithi110 ceph-mon[97633]: pgmap v18387: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:03:57.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:57.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:57.153461+0000) 2022-09-10T14:03:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:57.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:57.537688+0000) 2022-09-10T14:03:58.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:58.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:58.153725+0000) 2022-09-10T14:03:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:03:58 smithi104 ceph-mon[97044]: pgmap v18388: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:58.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:58.537928+0000) 2022-09-10T14:03:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:03:58 smithi110 ceph-mon[97633]: pgmap v18388: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:03:59.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:03:59.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:59.153999+0000) 2022-09-10T14:03:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:03:59.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:03:59.538212+0000) 2022-09-10T14:04:00.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:00.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:00.154233+0000) 2022-09-10T14:04:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:00 smithi104 ceph-mon[97044]: pgmap v18389: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:00.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:00.538339+0000) 2022-09-10T14:04:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:00 smithi110 ceph-mon[97633]: pgmap v18389: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:01.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:01.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:01.154479+0000) 2022-09-10T14:04:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:01.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:01.538548+0000) 2022-09-10T14:04:02.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:02.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:02.154721+0000) 2022-09-10T14:04:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:02 smithi104 ceph-mon[97044]: pgmap v18390: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:02.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:02.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:02.538811+0000) 2022-09-10T14:04:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:02 smithi110 ceph-mon[97633]: pgmap v18390: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:03.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:03.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:03.154925+0000) 2022-09-10T14:04:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:03.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:03.539040+0000) 2022-09-10T14:04:04.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:04.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:04.155168+0000) 2022-09-10T14:04:04.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:04:04.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:04:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:04 smithi104 ceph-mon[97044]: pgmap v18391: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:04.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:04.539297+0000) 2022-09-10T14:04:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:04 smithi110 ceph-mon[97633]: pgmap v18391: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:05.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:05.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:05.155378+0000) 2022-09-10T14:04:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:05.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:05.539569+0000) 2022-09-10T14:04:06.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:06.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:06.155586+0000) 2022-09-10T14:04:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:06 smithi104 ceph-mon[97044]: pgmap v18392: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:04:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:06.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:06.539725+0000) 2022-09-10T14:04:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:06 smithi110 ceph-mon[97633]: pgmap v18392: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:04:07.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:07.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:07.155801+0000) 2022-09-10T14:04:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:07.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:07.539925+0000) 2022-09-10T14:04:08.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:08.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:08.156040+0000) 2022-09-10T14:04:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:08 smithi104 ceph-mon[97044]: pgmap v18393: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:04:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:08.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:08.540155+0000) 2022-09-10T14:04:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:08 smithi110 ceph-mon[97633]: pgmap v18393: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:04:09.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:09.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:09.156318+0000) 2022-09-10T14:04:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:09 smithi104 ceph-mon[97044]: osdmap e1230: 12 total, 12 up, 12 in 2022-09-10T14:04:09.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:09.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:09.540327+0000) 2022-09-10T14:04:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:09 smithi110 ceph-mon[97633]: osdmap e1230: 12 total, 12 up, 12 in 2022-09-10T14:04:10.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:10.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:10.156583+0000) 2022-09-10T14:04:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:10 smithi104 ceph-mon[97044]: pgmap v18395: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:04:10.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:10.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:10.540596+0000) 2022-09-10T14:04:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:10 smithi110 ceph-mon[97633]: pgmap v18395: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:04:11.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:11.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:11.156795+0000) 2022-09-10T14:04:11.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:11.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:11.540840+0000) 2022-09-10T14:04:12.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:12.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:12.156985+0000) 2022-09-10T14:04:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:12 smithi104 ceph-mon[97044]: pgmap v18396: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:04:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:12.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:12.541083+0000) 2022-09-10T14:04:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:12 smithi110 ceph-mon[97633]: pgmap v18396: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:04:13.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:13.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:13.157186+0000) 2022-09-10T14:04:13.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:13.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:13.541344+0000) 2022-09-10T14:04:14.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:14.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:14.157352+0000) 2022-09-10T14:04:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:14 smithi104 ceph-mon[97044]: pgmap v18397: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:04:14.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:14.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:14.541555+0000) 2022-09-10T14:04:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:14 smithi110 ceph-mon[97633]: pgmap v18397: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:04:15.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:15.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:15.157572+0000) 2022-09-10T14:04:15.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:15.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:15.541784+0000) 2022-09-10T14:04:16.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:16.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:16.157776+0000) 2022-09-10T14:04:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:16 smithi104 ceph-mon[97044]: pgmap v18398: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:04:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:16 smithi104 ceph-mon[97044]: osdmap e1231: 12 total, 12 up, 12 in 2022-09-10T14:04:16.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:16.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:16.542012+0000) 2022-09-10T14:04:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:16 smithi110 ceph-mon[97633]: pgmap v18398: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:04:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:16 smithi110 ceph-mon[97633]: osdmap e1231: 12 total, 12 up, 12 in 2022-09-10T14:04:17.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:17.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:17.157984+0000) 2022-09-10T14:04:17.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:17.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:17.542243+0000) 2022-09-10T14:04:18.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:18.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:18.158220+0000) 2022-09-10T14:04:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:18 smithi104 ceph-mon[97044]: pgmap v18400: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T14:04:18.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:18.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:18.542486+0000) 2022-09-10T14:04:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:18 smithi110 ceph-mon[97633]: pgmap v18400: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T14:04:19.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:19.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:19.158426+0000) 2022-09-10T14:04:19.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:04:19.048+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:04:19.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:19.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:19.542746+0000) 2022-09-10T14:04:20.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:20.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:20.158632+0000) 2022-09-10T14:04:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:20 smithi104 ceph-mon[97044]: pgmap v18401: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:04:20.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:20.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:20.542953+0000) 2022-09-10T14:04:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:20 smithi110 ceph-mon[97633]: pgmap v18401: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:04:21.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:21.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:21.158840+0000) 2022-09-10T14:04:21.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:21.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:21.543214+0000) 2022-09-10T14:04:22.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:22.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:22.159079+0000) 2022-09-10T14:04:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:22 smithi104 ceph-mon[97044]: pgmap v18402: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:04:22.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:22.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:22.543445+0000) 2022-09-10T14:04:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:22 smithi110 ceph-mon[97633]: pgmap v18402: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:04:23.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:23.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:23.159369+0000) 2022-09-10T14:04:23.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:23.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:23.543642+0000) 2022-09-10T14:04:24.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:24.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:24.159594+0000) 2022-09-10T14:04:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:24 smithi104 ceph-mon[97044]: pgmap v18403: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:04:24.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:24.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:24.543852+0000) 2022-09-10T14:04:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:24 smithi110 ceph-mon[97633]: pgmap v18403: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:04:25.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:25.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:25.159764+0000) 2022-09-10T14:04:25.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:25.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:25.544032+0000) 2022-09-10T14:04:26.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:26.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:26.159956+0000) 2022-09-10T14:04:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:26 smithi104 ceph-mon[97044]: pgmap v18404: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:04:26.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:26.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:26.544234+0000) 2022-09-10T14:04:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:26 smithi110 ceph-mon[97633]: pgmap v18404: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:04:27.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:27.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:27.160173+0000) 2022-09-10T14:04:27.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:27.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:27.544408+0000) 2022-09-10T14:04:28.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:28.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:28.160333+0000) 2022-09-10T14:04:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:28 smithi104 ceph-mon[97044]: pgmap v18405: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:04:28.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:28.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:28.544666+0000) 2022-09-10T14:04:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:28 smithi110 ceph-mon[97633]: pgmap v18405: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:04:29.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:29.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:29.160606+0000) 2022-09-10T14:04:29.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:29.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:29.544919+0000) 2022-09-10T14:04:30.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:30.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:30.160752+0000) 2022-09-10T14:04:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:30 smithi104 ceph-mon[97044]: pgmap v18406: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:30.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:30.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:30.545139+0000) 2022-09-10T14:04:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:30 smithi110 ceph-mon[97633]: pgmap v18406: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:31.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:31.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:31.160981+0000) 2022-09-10T14:04:31.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:31.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:31.545348+0000) 2022-09-10T14:04:32.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:32.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:32.161215+0000) 2022-09-10T14:04:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:32 smithi104 ceph-mon[97044]: pgmap v18407: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:32.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:32.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:32.545619+0000) 2022-09-10T14:04:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:32 smithi110 ceph-mon[97633]: pgmap v18407: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:33.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:33.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:33.161362+0000) 2022-09-10T14:04:33.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:33.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:33.545876+0000) 2022-09-10T14:04:34.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:34.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:34.161613+0000) 2022-09-10T14:04:34.547 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:04:34.060+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:04:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:34 smithi104 ceph-mon[97044]: pgmap v18408: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:34.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:34.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:34.546109+0000) 2022-09-10T14:04:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:34 smithi110 ceph-mon[97633]: pgmap v18408: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:35.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:35.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:35.161873+0000) 2022-09-10T14:04:35.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:35.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:35.546321+0000) 2022-09-10T14:04:36.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:36.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:36.162069+0000) 2022-09-10T14:04:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:36 smithi104 ceph-mon[97044]: pgmap v18409: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:36.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:36.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:36.546462+0000) 2022-09-10T14:04:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:36 smithi110 ceph-mon[97633]: pgmap v18409: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:37.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:37.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:37.162285+0000) 2022-09-10T14:04:37.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:37.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:37.546641+0000) 2022-09-10T14:04:38.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:38.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:38.162426+0000) 2022-09-10T14:04:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:38 smithi104 ceph-mon[97044]: pgmap v18410: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:38.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:38.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:38.546870+0000) 2022-09-10T14:04:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:38 smithi110 ceph-mon[97633]: pgmap v18410: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:39.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:39.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:39.162649+0000) 2022-09-10T14:04:39.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:39.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:39.547143+0000) 2022-09-10T14:04:40.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:40.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:40.162881+0000) 2022-09-10T14:04:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:40 smithi104 ceph-mon[97044]: pgmap v18411: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:40.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:40.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:40.547319+0000) 2022-09-10T14:04:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:40 smithi110 ceph-mon[97633]: pgmap v18411: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:41.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:41.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:41.163087+0000) 2022-09-10T14:04:41.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:41.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:41.547623+0000) 2022-09-10T14:04:42.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:42.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:42.163245+0000) 2022-09-10T14:04:42.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:42.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:42.547787+0000) 2022-09-10T14:04:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:42 smithi110 ceph-mon[97633]: pgmap v18412: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:42 smithi104 ceph-mon[97044]: pgmap v18412: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:43.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:43.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:43.163457+0000) 2022-09-10T14:04:43.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:43.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:43.547988+0000) 2022-09-10T14:04:44.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:44.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:44.163715+0000) 2022-09-10T14:04:44.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:44.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:44.548237+0000) 2022-09-10T14:04:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:44 smithi110 ceph-mon[97633]: pgmap v18413: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:44 smithi104 ceph-mon[97044]: pgmap v18413: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:04:45.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:45.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:45.163985+0000) 2022-09-10T14:04:45.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:45.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:45.548428+0000) 2022-09-10T14:04:46.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:46.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:46.164230+0000) 2022-09-10T14:04:46.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:46.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:46.548585+0000) 2022-09-10T14:04:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:46 smithi110 ceph-mon[97633]: pgmap v18414: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:04:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:04:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:04:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:04:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:46 smithi104 ceph-mon[97044]: pgmap v18414: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:04:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:04:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:04:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:04:47.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:47.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:47.164509+0000) 2022-09-10T14:04:47.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:47.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:47.548801+0000) 2022-09-10T14:04:48.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:48.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:48.164770+0000) 2022-09-10T14:04:48.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:48.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:48.548955+0000) 2022-09-10T14:04:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:48 smithi110 ceph-mon[97633]: pgmap v18415: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:48 smithi104 ceph-mon[97044]: pgmap v18415: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:49.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:49.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:49.164982+0000) 2022-09-10T14:04:49.551 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:04:49.062+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:04:49.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:49.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:49.549170+0000) 2022-09-10T14:04:50.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:50.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:50.165145+0000) 2022-09-10T14:04:50.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:50.548+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:50.549412+0000) 2022-09-10T14:04:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:50 smithi110 ceph-mon[97633]: pgmap v18416: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:50 smithi104 ceph-mon[97044]: pgmap v18416: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:51.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:51.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:51.165356+0000) 2022-09-10T14:04:51.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:51.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:51.549633+0000) 2022-09-10T14:04:52.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:52.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:52.165568+0000) 2022-09-10T14:04:52.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:52.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:52.549891+0000) 2022-09-10T14:04:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:52 smithi110 ceph-mon[97633]: pgmap v18417: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:52 smithi104 ceph-mon[97044]: pgmap v18417: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:04:53.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:53.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:53.165797+0000) 2022-09-10T14:04:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:53.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:53.550142+0000) 2022-09-10T14:04:54.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:54.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:54.166029+0000) 2022-09-10T14:04:54.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:54.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:54.550388+0000) 2022-09-10T14:04:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:54 smithi110 ceph-mon[97633]: pgmap v18418: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:54 smithi104 ceph-mon[97044]: pgmap v18418: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:55.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:55.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:55.166328+0000) 2022-09-10T14:04:55.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:55.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:55.550570+0000) 2022-09-10T14:04:56.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:56.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:56.166569+0000) 2022-09-10T14:04:56.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:56.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:56.550813+0000) 2022-09-10T14:04:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:56 smithi110 ceph-mon[97633]: pgmap v18419: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:04:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:56 smithi104 ceph-mon[97044]: pgmap v18419: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:04:57.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:57.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:57.166806+0000) 2022-09-10T14:04:57.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:57.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:57.550983+0000) 2022-09-10T14:04:58.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:58.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:58.167078+0000) 2022-09-10T14:04:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:58.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:58.551217+0000) 2022-09-10T14:04:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:04:58 smithi110 ceph-mon[97633]: pgmap v18420: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:04:58 smithi104 ceph-mon[97044]: pgmap v18420: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:04:59.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:04:59.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:59.167316+0000) 2022-09-10T14:04:59.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:04:59.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:04:59.551406+0000) 2022-09-10T14:05:00.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:00.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:00.167479+0000) 2022-09-10T14:05:00.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:00.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:00.551574+0000) 2022-09-10T14:05:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:00 smithi110 ceph-mon[97633]: pgmap v18421: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:00 smithi104 ceph-mon[97044]: pgmap v18421: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:01.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:01.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:01.167686+0000) 2022-09-10T14:05:01.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:01.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:01.551730+0000) 2022-09-10T14:05:02.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:02.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:02.167909+0000) 2022-09-10T14:05:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:02.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:02.551890+0000) 2022-09-10T14:05:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:02 smithi110 ceph-mon[97633]: pgmap v18422: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:02 smithi104 ceph-mon[97044]: pgmap v18422: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:03.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:03.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:03.168086+0000) 2022-09-10T14:05:03.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:03.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:03.552152+0000) 2022-09-10T14:05:04.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:04.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:04.168320+0000) 2022-09-10T14:05:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:05:04.078+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:05:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:04.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:04.552402+0000) 2022-09-10T14:05:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:04 smithi110 ceph-mon[97633]: pgmap v18423: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:04 smithi104 ceph-mon[97044]: pgmap v18423: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:05.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:05.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:05.168566+0000) 2022-09-10T14:05:05.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:05.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:05.552548+0000) 2022-09-10T14:05:06.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:06.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:06.168856+0000) 2022-09-10T14:05:06.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:06.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:06.552708+0000) 2022-09-10T14:05:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:06 smithi110 ceph-mon[97633]: pgmap v18424: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:05:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:06 smithi104 ceph-mon[97044]: pgmap v18424: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:05:07.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:07.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:07.169110+0000) 2022-09-10T14:05:07.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:07.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:07.552977+0000) 2022-09-10T14:05:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:08.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:08.169261+0000) 2022-09-10T14:05:08.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:08.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:08.553217+0000) 2022-09-10T14:05:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:08 smithi110 ceph-mon[97633]: pgmap v18425: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:05:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:08 smithi104 ceph-mon[97044]: pgmap v18425: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:05:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:09.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:09.169503+0000) 2022-09-10T14:05:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:09 smithi110 ceph-mon[97633]: osdmap e1232: 12 total, 12 up, 12 in 2022-09-10T14:05:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:09 smithi104 ceph-mon[97044]: osdmap e1232: 12 total, 12 up, 12 in 2022-09-10T14:05:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:09.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:09.553446+0000) 2022-09-10T14:05:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:10.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:10.169667+0000) 2022-09-10T14:05:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:10 smithi110 ceph-mon[97633]: pgmap v18427: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:05:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:10 smithi104 ceph-mon[97044]: pgmap v18427: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:05:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:10.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:10.553711+0000) 2022-09-10T14:05:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:11.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:11.169888+0000) 2022-09-10T14:05:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:11.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:11.553881+0000) 2022-09-10T14:05:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:12.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:12.170147+0000) 2022-09-10T14:05:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:12 smithi110 ceph-mon[97633]: pgmap v18428: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:05:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:12 smithi104 ceph-mon[97044]: pgmap v18428: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:05:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:12.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:12.554010+0000) 2022-09-10T14:05:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:13.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:13.170398+0000) 2022-09-10T14:05:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:13.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:13.554152+0000) 2022-09-10T14:05:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:14.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:14.170545+0000) 2022-09-10T14:05:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:14 smithi110 ceph-mon[97633]: pgmap v18429: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:05:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:14 smithi104 ceph-mon[97044]: pgmap v18429: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:05:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:14.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:14.554281+0000) 2022-09-10T14:05:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:15.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:15.170689+0000) 2022-09-10T14:05:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:15.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:15.554397+0000) 2022-09-10T14:05:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:16.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:16.170856+0000) 2022-09-10T14:05:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:16 smithi110 ceph-mon[97633]: pgmap v18430: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:05:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:16 smithi110 ceph-mon[97633]: osdmap e1233: 12 total, 12 up, 12 in 2022-09-10T14:05:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:16 smithi104 ceph-mon[97044]: pgmap v18430: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:05:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:16 smithi104 ceph-mon[97044]: osdmap e1233: 12 total, 12 up, 12 in 2022-09-10T14:05:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:16.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:16.554532+0000) 2022-09-10T14:05:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:17.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:17.171055+0000) 2022-09-10T14:05:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:17.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:17.554755+0000) 2022-09-10T14:05:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:18.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:18.171224+0000) 2022-09-10T14:05:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:18 smithi110 ceph-mon[97633]: pgmap v18432: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T14:05:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:18 smithi104 ceph-mon[97044]: pgmap v18432: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T14:05:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:18.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:18.554928+0000) 2022-09-10T14:05:19.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:05:19.088+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:05:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:19.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:19.171412+0000) 2022-09-10T14:05:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:19.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:19.555182+0000) 2022-09-10T14:05:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:20.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:20.171659+0000) 2022-09-10T14:05:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:20 smithi110 ceph-mon[97633]: pgmap v18433: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:05:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:20 smithi104 ceph-mon[97044]: pgmap v18433: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:05:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:20.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:20.555336+0000) 2022-09-10T14:05:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:21.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:21.171893+0000) 2022-09-10T14:05:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:21.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:21.555482+0000) 2022-09-10T14:05:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:22.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:22.172184+0000) 2022-09-10T14:05:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:22 smithi110 ceph-mon[97633]: pgmap v18434: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:05:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:22 smithi104 ceph-mon[97044]: pgmap v18434: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:05:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:22.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:22.555631+0000) 2022-09-10T14:05:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:23.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:23.172360+0000) 2022-09-10T14:05:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:23.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:23.555816+0000) 2022-09-10T14:05:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:24.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:24.172566+0000) 2022-09-10T14:05:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:24 smithi110 ceph-mon[97633]: pgmap v18435: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:05:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:24 smithi104 ceph-mon[97044]: pgmap v18435: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:05:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:24.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:24.556035+0000) 2022-09-10T14:05:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:25.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:25.172784+0000) 2022-09-10T14:05:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:25.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:25.556231+0000) 2022-09-10T14:05:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:26.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:26.173013+0000) 2022-09-10T14:05:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:26 smithi110 ceph-mon[97633]: pgmap v18436: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:05:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:26 smithi104 ceph-mon[97044]: pgmap v18436: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:05:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:26.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:26.556361+0000) 2022-09-10T14:05:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:27.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:27.173282+0000) 2022-09-10T14:05:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:27.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:27.556497+0000) 2022-09-10T14:05:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:28.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:28.173501+0000) 2022-09-10T14:05:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:28 smithi110 ceph-mon[97633]: pgmap v18437: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:05:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:28 smithi104 ceph-mon[97044]: pgmap v18437: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:05:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:28.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:28.556618+0000) 2022-09-10T14:05:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:29.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:29.173680+0000) 2022-09-10T14:05:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:29.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:29.556793+0000) 2022-09-10T14:05:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:30.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:30.173916+0000) 2022-09-10T14:05:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:30 smithi110 ceph-mon[97633]: pgmap v18438: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:30 smithi104 ceph-mon[97044]: pgmap v18438: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:30.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:30.557016+0000) 2022-09-10T14:05:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:31.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:31.174142+0000) 2022-09-10T14:05:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:31.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:31.557247+0000) 2022-09-10T14:05:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:32.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:32.174309+0000) 2022-09-10T14:05:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:32 smithi110 ceph-mon[97633]: pgmap v18439: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:32 smithi104 ceph-mon[97044]: pgmap v18439: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:32.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:32.557439+0000) 2022-09-10T14:05:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:33.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:33.174538+0000) 2022-09-10T14:05:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:33.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:33.557601+0000) 2022-09-10T14:05:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:05:34.103+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:05:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:34.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:34.174734+0000) 2022-09-10T14:05:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:34 smithi110 ceph-mon[97633]: pgmap v18440: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:34 smithi104 ceph-mon[97044]: pgmap v18440: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:34.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:34.557828+0000) 2022-09-10T14:05:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:35.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:35.174955+0000) 2022-09-10T14:05:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:35.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:35.558047+0000) 2022-09-10T14:05:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:36.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:36.175226+0000) 2022-09-10T14:05:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:36 smithi110 ceph-mon[97633]: pgmap v18441: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:36 smithi104 ceph-mon[97044]: pgmap v18441: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:36.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:36.558247+0000) 2022-09-10T14:05:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:37.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:37.175458+0000) 2022-09-10T14:05:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:37.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:37.558395+0000) 2022-09-10T14:05:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:38.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:38.175623+0000) 2022-09-10T14:05:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:38 smithi110 ceph-mon[97633]: pgmap v18442: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:38 smithi104 ceph-mon[97044]: pgmap v18442: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:38.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:38.558524+0000) 2022-09-10T14:05:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:39.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:39.175796+0000) 2022-09-10T14:05:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:39.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:39.558716+0000) 2022-09-10T14:05:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:40.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:40.175974+0000) 2022-09-10T14:05:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:40 smithi110 ceph-mon[97633]: pgmap v18443: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:40 smithi104 ceph-mon[97044]: pgmap v18443: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:40.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:40.558941+0000) 2022-09-10T14:05:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:41.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:41.176241+0000) 2022-09-10T14:05:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:41.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:41.559163+0000) 2022-09-10T14:05:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:42.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:42.176474+0000) 2022-09-10T14:05:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:42 smithi110 ceph-mon[97633]: pgmap v18444: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:42 smithi104 ceph-mon[97044]: pgmap v18444: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:42.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:42.559292+0000) 2022-09-10T14:05:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:43.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:43.176771+0000) 2022-09-10T14:05:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:43.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:43.559422+0000) 2022-09-10T14:05:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:44.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:44.177021+0000) 2022-09-10T14:05:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:44 smithi110 ceph-mon[97633]: pgmap v18445: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:44 smithi104 ceph-mon[97044]: pgmap v18445: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:44.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:44.559597+0000) 2022-09-10T14:05:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:45.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:45.177224+0000) 2022-09-10T14:05:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:45.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:45.559779+0000) 2022-09-10T14:05:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:46.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:46.177416+0000) 2022-09-10T14:05:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:46 smithi110 ceph-mon[97633]: pgmap v18446: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:05:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:05:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:05:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:05:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:46 smithi104 ceph-mon[97044]: pgmap v18446: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:05:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:05:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:05:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:05:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:46.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:46.560001+0000) 2022-09-10T14:05:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:47.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:47.177600+0000) 2022-09-10T14:05:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:47.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:47.560246+0000) 2022-09-10T14:05:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:48.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:48.177854+0000) 2022-09-10T14:05:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:48 smithi110 ceph-mon[97633]: pgmap v18447: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:48 smithi104 ceph-mon[97044]: pgmap v18447: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:48.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:48.560395+0000) 2022-09-10T14:05:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:05:49.119+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:05:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:49.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:49.178118+0000) 2022-09-10T14:05:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:49.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:49.560570+0000) 2022-09-10T14:05:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:50.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:50.178339+0000) 2022-09-10T14:05:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:50 smithi104 ceph-mon[97044]: pgmap v18448: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:50.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:50.560804+0000) 2022-09-10T14:05:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:50 smithi110 ceph-mon[97633]: pgmap v18448: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:51.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:51.178569+0000) 2022-09-10T14:05:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:51.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:51.560995+0000) 2022-09-10T14:05:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:52.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:52.178798+0000) 2022-09-10T14:05:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:52 smithi104 ceph-mon[97044]: pgmap v18449: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:52.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:52.561226+0000) 2022-09-10T14:05:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:52 smithi110 ceph-mon[97633]: pgmap v18449: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:53.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:53.179054+0000) 2022-09-10T14:05:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:53.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:53.561369+0000) 2022-09-10T14:05:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:54.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:54.179247+0000) 2022-09-10T14:05:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:54 smithi104 ceph-mon[97044]: pgmap v18450: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:54.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:54.561524+0000) 2022-09-10T14:05:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:54 smithi110 ceph-mon[97633]: pgmap v18450: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:05:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:55.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:55.179490+0000) 2022-09-10T14:05:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:55.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:55.561776+0000) 2022-09-10T14:05:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:56.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:56.179699+0000) 2022-09-10T14:05:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:56 smithi104 ceph-mon[97044]: pgmap v18451: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:56.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:56.562020+0000) 2022-09-10T14:05:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:56 smithi110 ceph-mon[97633]: pgmap v18451: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:05:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:57.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:57.179865+0000) 2022-09-10T14:05:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:57.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:57.562270+0000) 2022-09-10T14:05:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:58.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:58.180136+0000) 2022-09-10T14:05:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:05:58 smithi104 ceph-mon[97044]: pgmap v18452: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:58.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:58.562498+0000) 2022-09-10T14:05:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:05:58 smithi110 ceph-mon[97633]: pgmap v18452: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:05:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:05:59.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:59.180363+0000) 2022-09-10T14:06:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:05:59.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:05:59.562680+0000) 2022-09-10T14:06:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:00.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:00.180573+0000) 2022-09-10T14:06:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:00 smithi104 ceph-mon[97044]: pgmap v18453: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:00.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:00.562886+0000) 2022-09-10T14:06:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:00 smithi110 ceph-mon[97633]: pgmap v18453: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:01.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:01.180736+0000) 2022-09-10T14:06:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:01.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:01.563114+0000) 2022-09-10T14:06:02.565 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:02.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:02.180947+0000) 2022-09-10T14:06:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:02 smithi104 ceph-mon[97044]: pgmap v18454: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:02.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:02.563358+0000) 2022-09-10T14:06:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:02 smithi110 ceph-mon[97633]: pgmap v18454: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:03.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:03.181168+0000) 2022-09-10T14:06:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:03.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:03.563521+0000) 2022-09-10T14:06:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:06:04.134+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:06:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:04.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:04.181409+0000) 2022-09-10T14:06:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:04 smithi104 ceph-mon[97044]: pgmap v18455: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:04.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:04.563696+0000) 2022-09-10T14:06:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:04 smithi110 ceph-mon[97633]: pgmap v18455: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:05.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:05.181617+0000) 2022-09-10T14:06:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:05.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:05.563928+0000) 2022-09-10T14:06:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:06.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:06.181856+0000) 2022-09-10T14:06:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:06 smithi104 ceph-mon[97044]: pgmap v18456: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:06:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:06.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:06.564180+0000) 2022-09-10T14:06:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:06 smithi110 ceph-mon[97633]: pgmap v18456: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:06:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:07.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:07.182091+0000) 2022-09-10T14:06:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:07.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:07.564402+0000) 2022-09-10T14:06:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:08.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:08.182305+0000) 2022-09-10T14:06:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:08 smithi104 ceph-mon[97044]: pgmap v18457: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:06:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:08.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:08.564665+0000) 2022-09-10T14:06:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:08 smithi110 ceph-mon[97633]: pgmap v18457: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:06:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:09.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:09.182487+0000) 2022-09-10T14:06:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:09 smithi104 ceph-mon[97044]: osdmap e1234: 12 total, 12 up, 12 in 2022-09-10T14:06:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:09.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:09.564827+0000) 2022-09-10T14:06:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:09 smithi110 ceph-mon[97633]: osdmap e1234: 12 total, 12 up, 12 in 2022-09-10T14:06:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:10.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:10.182668+0000) 2022-09-10T14:06:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:10 smithi104 ceph-mon[97044]: pgmap v18459: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:06:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:10.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:10.565064+0000) 2022-09-10T14:06:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:10 smithi110 ceph-mon[97633]: pgmap v18459: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:06:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:11.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:11.182901+0000) 2022-09-10T14:06:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:11.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:11.565330+0000) 2022-09-10T14:06:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:12.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:12.183169+0000) 2022-09-10T14:06:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:12 smithi104 ceph-mon[97044]: pgmap v18460: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:06:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:12.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:12.565496+0000) 2022-09-10T14:06:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:12 smithi110 ceph-mon[97633]: pgmap v18460: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:06:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:13.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:13.183371+0000) 2022-09-10T14:06:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:13.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:13.565707+0000) 2022-09-10T14:06:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:14.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:14.183564+0000) 2022-09-10T14:06:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:14 smithi104 ceph-mon[97044]: pgmap v18461: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:06:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:14.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:14.565920+0000) 2022-09-10T14:06:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:14 smithi110 ceph-mon[97633]: pgmap v18461: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:06:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:15.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:15.183782+0000) 2022-09-10T14:06:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:15.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:15.566158+0000) 2022-09-10T14:06:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:16.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:16.183971+0000) 2022-09-10T14:06:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:16 smithi104 ceph-mon[97044]: pgmap v18462: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:06:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:16 smithi104 ceph-mon[97044]: osdmap e1235: 12 total, 12 up, 12 in 2022-09-10T14:06:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:16.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:16.566386+0000) 2022-09-10T14:06:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:16 smithi110 ceph-mon[97633]: pgmap v18462: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:06:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:16 smithi110 ceph-mon[97633]: osdmap e1235: 12 total, 12 up, 12 in 2022-09-10T14:06:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:17.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:17.184163+0000) 2022-09-10T14:06:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:17.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:17.566659+0000) 2022-09-10T14:06:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:18.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:18.184288+0000) 2022-09-10T14:06:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:18 smithi104 ceph-mon[97044]: pgmap v18464: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T14:06:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:18.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:18.566927+0000) 2022-09-10T14:06:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:18 smithi110 ceph-mon[97633]: pgmap v18464: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T14:06:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:19.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:19.184475+0000) 2022-09-10T14:06:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:06:19.146+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:06:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:19.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:19.567155+0000) 2022-09-10T14:06:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:20.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:20.184725+0000) 2022-09-10T14:06:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:20 smithi104 ceph-mon[97044]: pgmap v18465: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:06:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:20.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:20.567311+0000) 2022-09-10T14:06:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:20 smithi110 ceph-mon[97633]: pgmap v18465: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:06:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:21.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:21.184939+0000) 2022-09-10T14:06:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:21.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:21.567494+0000) 2022-09-10T14:06:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:22.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:22.185176+0000) 2022-09-10T14:06:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:22 smithi104 ceph-mon[97044]: pgmap v18466: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:06:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:22.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:22.567739+0000) 2022-09-10T14:06:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:22 smithi110 ceph-mon[97633]: pgmap v18466: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:06:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:23.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:23.185459+0000) 2022-09-10T14:06:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:23.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:23.568024+0000) 2022-09-10T14:06:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:24.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:24.185614+0000) 2022-09-10T14:06:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:24 smithi104 ceph-mon[97044]: pgmap v18467: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:06:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:24.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:24.568256+0000) 2022-09-10T14:06:25.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:24 smithi110 ceph-mon[97633]: pgmap v18467: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:06:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:25.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:25.185890+0000) 2022-09-10T14:06:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:25.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:25.568472+0000) 2022-09-10T14:06:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:26.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:26.186168+0000) 2022-09-10T14:06:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:26 smithi104 ceph-mon[97044]: pgmap v18468: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:06:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:26.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:26.568726+0000) 2022-09-10T14:06:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:26 smithi110 ceph-mon[97633]: pgmap v18468: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:06:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:27.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:27.186393+0000) 2022-09-10T14:06:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:27.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:27.568969+0000) 2022-09-10T14:06:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:28.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:28.186650+0000) 2022-09-10T14:06:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:28 smithi104 ceph-mon[97044]: pgmap v18469: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:06:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:28.568+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:28.569207+0000) 2022-09-10T14:06:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:28 smithi110 ceph-mon[97633]: pgmap v18469: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:06:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:29.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:29.186788+0000) 2022-09-10T14:06:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:29.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:29.569448+0000) 2022-09-10T14:06:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:30.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:30.187013+0000) 2022-09-10T14:06:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:30 smithi104 ceph-mon[97044]: pgmap v18470: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:30.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:30.569680+0000) 2022-09-10T14:06:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:30 smithi110 ceph-mon[97633]: pgmap v18470: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:31.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:31.187250+0000) 2022-09-10T14:06:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:31.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:31.569904+0000) 2022-09-10T14:06:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:32.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:32.187458+0000) 2022-09-10T14:06:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:32 smithi104 ceph-mon[97044]: pgmap v18471: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:32.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:32.570119+0000) 2022-09-10T14:06:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:32 smithi110 ceph-mon[97633]: pgmap v18471: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:33.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:33.187712+0000) 2022-09-10T14:06:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:33.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:33.570279+0000) 2022-09-10T14:06:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:34.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:34.187915+0000) 2022-09-10T14:06:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:06:34.150+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:06:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:34 smithi104 ceph-mon[97044]: pgmap v18472: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:34.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:34.570443+0000) 2022-09-10T14:06:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:34 smithi110 ceph-mon[97633]: pgmap v18472: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:35.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:35.188160+0000) 2022-09-10T14:06:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:35.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:35.570653+0000) 2022-09-10T14:06:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:36.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:36.188386+0000) 2022-09-10T14:06:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:36 smithi104 ceph-mon[97044]: pgmap v18473: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:36.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:36.570902+0000) 2022-09-10T14:06:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:36 smithi110 ceph-mon[97633]: pgmap v18473: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:37.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:37.188609+0000) 2022-09-10T14:06:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:37.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:37.571117+0000) 2022-09-10T14:06:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:38.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:38.188842+0000) 2022-09-10T14:06:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:38 smithi104 ceph-mon[97044]: pgmap v18474: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:38.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:38.571354+0000) 2022-09-10T14:06:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:38 smithi110 ceph-mon[97633]: pgmap v18474: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:39.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:39.189085+0000) 2022-09-10T14:06:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:39.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:39.571583+0000) 2022-09-10T14:06:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:40.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:40.189384+0000) 2022-09-10T14:06:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:40 smithi104 ceph-mon[97044]: pgmap v18475: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:40.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:40.571828+0000) 2022-09-10T14:06:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:40 smithi110 ceph-mon[97633]: pgmap v18475: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:41.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:41.189560+0000) 2022-09-10T14:06:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:41.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:41.572040+0000) 2022-09-10T14:06:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:42.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:42.189779+0000) 2022-09-10T14:06:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:42 smithi104 ceph-mon[97044]: pgmap v18476: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:42.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:42.572255+0000) 2022-09-10T14:06:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:42 smithi110 ceph-mon[97633]: pgmap v18476: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:43.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:43.190022+0000) 2022-09-10T14:06:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:43.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:43.572475+0000) 2022-09-10T14:06:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:44.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:44.190264+0000) 2022-09-10T14:06:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:44 smithi104 ceph-mon[97044]: pgmap v18477: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:44.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:44.572698+0000) 2022-09-10T14:06:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:44 smithi110 ceph-mon[97633]: pgmap v18477: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:45.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:45.190520+0000) 2022-09-10T14:06:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:45.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:45.572943+0000) 2022-09-10T14:06:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:46.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:46.190783+0000) 2022-09-10T14:06:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:46 smithi104 ceph-mon[97044]: pgmap v18478: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:06:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:06:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:06:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:06:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:46.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:46.573201+0000) 2022-09-10T14:06:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:46 smithi110 ceph-mon[97633]: pgmap v18478: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:06:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:06:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:06:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:47.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:47.191008+0000) 2022-09-10T14:06:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:47.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:47.573437+0000) 2022-09-10T14:06:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:48.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:48.191242+0000) 2022-09-10T14:06:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:48 smithi104 ceph-mon[97044]: pgmap v18479: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:48.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:48.573635+0000) 2022-09-10T14:06:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:48 smithi110 ceph-mon[97633]: pgmap v18479: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:06:49.166+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:06:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:49.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:49.191467+0000) 2022-09-10T14:06:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:49.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:49.573870+0000) 2022-09-10T14:06:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:50.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:50.191709+0000) 2022-09-10T14:06:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:50 smithi104 ceph-mon[97044]: pgmap v18480: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:50.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:50.574167+0000) 2022-09-10T14:06:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:50 smithi110 ceph-mon[97633]: pgmap v18480: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:51.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:51.191979+0000) 2022-09-10T14:06:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:51.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:51.574411+0000) 2022-09-10T14:06:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:52.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:52.192158+0000) 2022-09-10T14:06:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:52 smithi104 ceph-mon[97044]: pgmap v18481: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:52.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:52.574647+0000) 2022-09-10T14:06:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:52 smithi110 ceph-mon[97633]: pgmap v18481: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:53.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:53.192368+0000) 2022-09-10T14:06:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:53.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:53.574922+0000) 2022-09-10T14:06:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:54.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:54.192533+0000) 2022-09-10T14:06:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:54 smithi104 ceph-mon[97044]: pgmap v18482: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:54.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:54.575197+0000) 2022-09-10T14:06:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:54 smithi110 ceph-mon[97633]: pgmap v18482: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:55.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:55.192745+0000) 2022-09-10T14:06:56.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:55 smithi104 ceph-mon[97044]: pgmap v18483: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:55.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:55.575465+0000) 2022-09-10T14:06:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:55 smithi110 ceph-mon[97633]: pgmap v18483: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:06:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:56.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:56.192979+0000) 2022-09-10T14:06:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:56.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:56.575653+0000) 2022-09-10T14:06:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:57.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:57.193192+0000) 2022-09-10T14:06:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:57.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:57.575876+0000) 2022-09-10T14:06:58.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:06:58 smithi104 ceph-mon[97044]: pgmap v18484: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:58.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:58.193420+0000) 2022-09-10T14:06:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:06:58 smithi110 ceph-mon[97633]: pgmap v18484: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:06:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:58.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:58.576062+0000) 2022-09-10T14:06:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:06:59.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:59.193611+0000) 2022-09-10T14:07:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:06:59.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:06:59.576230+0000) 2022-09-10T14:07:00.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:00 smithi104 ceph-mon[97044]: pgmap v18485: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:00.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:00.193878+0000) 2022-09-10T14:07:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:00 smithi110 ceph-mon[97633]: pgmap v18485: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:01.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:00.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:00.576394+0000) 2022-09-10T14:07:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:01.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:01.194143+0000) 2022-09-10T14:07:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:01.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:01.576668+0000) 2022-09-10T14:07:02.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:02 smithi104 ceph-mon[97044]: pgmap v18486: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:02.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:02.194329+0000) 2022-09-10T14:07:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:02 smithi110 ceph-mon[97633]: pgmap v18486: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:02.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:02.576869+0000) 2022-09-10T14:07:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:03.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:03.194468+0000) 2022-09-10T14:07:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:03.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:03.577075+0000) 2022-09-10T14:07:04.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:07:04.181+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:07:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:04.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:04.194712+0000) 2022-09-10T14:07:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:04 smithi110 ceph-mon[97633]: pgmap v18487: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:04 smithi104 ceph-mon[97044]: pgmap v18487: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:04.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:04.577252+0000) 2022-09-10T14:07:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:05.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:05.194918+0000) 2022-09-10T14:07:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:05.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:05.577429+0000) 2022-09-10T14:07:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:06.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:06.195162+0000) 2022-09-10T14:07:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:06 smithi110 ceph-mon[97633]: pgmap v18488: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:07:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:06 smithi104 ceph-mon[97044]: pgmap v18488: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:07:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:06.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:06.577614+0000) 2022-09-10T14:07:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:07.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:07.195324+0000) 2022-09-10T14:07:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:07.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:07.577835+0000) 2022-09-10T14:07:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:08.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:08.195560+0000) 2022-09-10T14:07:08.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:08 smithi110 ceph-mon[97633]: pgmap v18489: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:07:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:08 smithi104 ceph-mon[97044]: pgmap v18489: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:07:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:08.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:08.578078+0000) 2022-09-10T14:07:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:09.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:09.195754+0000) 2022-09-10T14:07:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:09 smithi110 ceph-mon[97633]: osdmap e1236: 12 total, 12 up, 12 in 2022-09-10T14:07:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:09 smithi104 ceph-mon[97044]: osdmap e1236: 12 total, 12 up, 12 in 2022-09-10T14:07:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:09.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:09.578333+0000) 2022-09-10T14:07:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:10.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:10.195976+0000) 2022-09-10T14:07:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:10 smithi110 ceph-mon[97633]: pgmap v18491: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:07:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:10 smithi104 ceph-mon[97044]: pgmap v18491: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:07:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:10.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:10.578488+0000) 2022-09-10T14:07:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:11.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:11.196208+0000) 2022-09-10T14:07:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:11.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:11.578704+0000) 2022-09-10T14:07:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:12.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:12.196433+0000) 2022-09-10T14:07:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:12 smithi110 ceph-mon[97633]: pgmap v18492: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:07:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:12 smithi104 ceph-mon[97044]: pgmap v18492: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:07:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:12.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:12.578897+0000) 2022-09-10T14:07:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:13.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:13.196575+0000) 2022-09-10T14:07:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:13.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:13.579157+0000) 2022-09-10T14:07:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:14.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:14.196769+0000) 2022-09-10T14:07:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:14 smithi110 ceph-mon[97633]: pgmap v18493: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:07:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:14 smithi104 ceph-mon[97044]: pgmap v18493: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:07:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:14.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:14.579290+0000) 2022-09-10T14:07:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:15.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:15.196948+0000) 2022-09-10T14:07:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:15.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:15.579493+0000) 2022-09-10T14:07:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:16.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:16.197209+0000) 2022-09-10T14:07:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:16 smithi110 ceph-mon[97633]: pgmap v18494: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:07:16.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:16 smithi110 ceph-mon[97633]: osdmap e1237: 12 total, 12 up, 12 in 2022-09-10T14:07:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:16 smithi104 ceph-mon[97044]: pgmap v18494: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:07:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:16 smithi104 ceph-mon[97044]: osdmap e1237: 12 total, 12 up, 12 in 2022-09-10T14:07:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:16.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:16.579639+0000) 2022-09-10T14:07:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:17.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:17.197428+0000) 2022-09-10T14:07:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:17.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:17.579874+0000) 2022-09-10T14:07:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:18.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:18.197678+0000) 2022-09-10T14:07:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:18 smithi110 ceph-mon[97633]: pgmap v18496: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:07:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:18 smithi104 ceph-mon[97044]: pgmap v18496: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:07:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:18.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:18.580131+0000) 2022-09-10T14:07:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:19.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:19.197875+0000) 2022-09-10T14:07:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:07:19.188+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:07:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:19.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:19.580384+0000) 2022-09-10T14:07:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:20.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:20.198170+0000) 2022-09-10T14:07:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:20 smithi110 ceph-mon[97633]: pgmap v18497: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:07:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:20 smithi104 ceph-mon[97044]: pgmap v18497: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:07:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:20.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:20.580641+0000) 2022-09-10T14:07:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:21.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:21.198394+0000) 2022-09-10T14:07:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:21.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:21.580849+0000) 2022-09-10T14:07:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:22.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:22.198613+0000) 2022-09-10T14:07:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:22 smithi110 ceph-mon[97633]: pgmap v18498: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:07:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:22 smithi104 ceph-mon[97044]: pgmap v18498: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:07:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:22.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:22.581067+0000) 2022-09-10T14:07:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:23.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:23.198855+0000) 2022-09-10T14:07:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:23.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:23.581283+0000) 2022-09-10T14:07:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:24.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:24.199082+0000) 2022-09-10T14:07:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:24 smithi110 ceph-mon[97633]: pgmap v18499: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:07:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:24 smithi104 ceph-mon[97044]: pgmap v18499: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:07:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:24.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:24.581514+0000) 2022-09-10T14:07:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:25.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:25.199346+0000) 2022-09-10T14:07:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:25.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:25.581706+0000) 2022-09-10T14:07:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:26.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:26.199545+0000) 2022-09-10T14:07:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:26 smithi110 ceph-mon[97633]: pgmap v18500: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:07:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:26 smithi104 ceph-mon[97044]: pgmap v18500: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:07:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:26.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:26.581883+0000) 2022-09-10T14:07:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:27.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:27.199734+0000) 2022-09-10T14:07:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:27.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:27.582070+0000) 2022-09-10T14:07:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:28.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:28.199942+0000) 2022-09-10T14:07:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:28 smithi110 ceph-mon[97633]: pgmap v18501: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:07:28.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:28 smithi104 ceph-mon[97044]: pgmap v18501: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:07:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:28.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:28.582241+0000) 2022-09-10T14:07:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:29.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:29.200154+0000) 2022-09-10T14:07:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:29.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:29.582400+0000) 2022-09-10T14:07:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:30.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:30.200302+0000) 2022-09-10T14:07:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:30 smithi110 ceph-mon[97633]: pgmap v18502: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:30 smithi104 ceph-mon[97044]: pgmap v18502: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:30.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:30.582632+0000) 2022-09-10T14:07:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:31.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:31.200537+0000) 2022-09-10T14:07:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:31.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:31.582859+0000) 2022-09-10T14:07:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:32.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:32.200677+0000) 2022-09-10T14:07:32.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:32 smithi110 ceph-mon[97633]: pgmap v18503: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:32 smithi104 ceph-mon[97044]: pgmap v18503: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:32.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:32.583089+0000) 2022-09-10T14:07:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:33.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:33.200894+0000) 2022-09-10T14:07:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:33.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:33.583366+0000) 2022-09-10T14:07:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:07:34.203+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:07:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:34.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:34.201123+0000) 2022-09-10T14:07:34.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:34 smithi110 ceph-mon[97633]: pgmap v18504: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:34 smithi104 ceph-mon[97044]: pgmap v18504: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:34.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:34.583588+0000) 2022-09-10T14:07:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:35.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:35.201329+0000) 2022-09-10T14:07:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:35.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:35.583840+0000) 2022-09-10T14:07:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:36.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:36.201547+0000) 2022-09-10T14:07:36.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:36 smithi110 ceph-mon[97633]: pgmap v18505: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:36 smithi104 ceph-mon[97044]: pgmap v18505: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:36.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:36.583971+0000) 2022-09-10T14:07:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:37.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:37.201760+0000) 2022-09-10T14:07:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:37.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:37.584196+0000) 2022-09-10T14:07:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:38.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:38.201987+0000) 2022-09-10T14:07:38.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:38 smithi110 ceph-mon[97633]: pgmap v18506: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:38 smithi104 ceph-mon[97044]: pgmap v18506: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:38.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:38.584379+0000) 2022-09-10T14:07:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:39.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:39.202217+0000) 2022-09-10T14:07:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:39.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:39.584597+0000) 2022-09-10T14:07:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:40.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:40.202440+0000) 2022-09-10T14:07:40.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:40 smithi110 ceph-mon[97633]: pgmap v18507: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:40 smithi104 ceph-mon[97044]: pgmap v18507: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:40.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:40.584807+0000) 2022-09-10T14:07:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:41.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:41.202585+0000) 2022-09-10T14:07:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:41.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:41.585054+0000) 2022-09-10T14:07:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:42.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:42.202793+0000) 2022-09-10T14:07:42.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:42 smithi110 ceph-mon[97633]: pgmap v18508: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:42 smithi104 ceph-mon[97044]: pgmap v18508: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:42.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:42.585291+0000) 2022-09-10T14:07:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:43.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:43.203031+0000) 2022-09-10T14:07:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:43.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:43.585491+0000) 2022-09-10T14:07:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:44.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:44.203166+0000) 2022-09-10T14:07:44.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:44 smithi110 ceph-mon[97633]: pgmap v18509: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:44 smithi104 ceph-mon[97044]: pgmap v18509: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:44.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:44.585682+0000) 2022-09-10T14:07:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:45.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:45.203376+0000) 2022-09-10T14:07:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:45.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:45.585898+0000) 2022-09-10T14:07:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:46.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:46.203546+0000) 2022-09-10T14:07:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:46 smithi110 ceph-mon[97633]: pgmap v18510: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:46 smithi104 ceph-mon[97044]: pgmap v18510: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:46.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:46.586172+0000) 2022-09-10T14:07:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:47.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:47.203756+0000) 2022-09-10T14:07:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:07:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:07:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:07:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:07:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:07:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:07:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:07:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:07:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:47.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:47.586340+0000) 2022-09-10T14:07:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:48.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:48.204023+0000) 2022-09-10T14:07:48.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:48 smithi110 ceph-mon[97633]: pgmap v18511: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:48 smithi104 ceph-mon[97044]: pgmap v18511: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:48.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:48.586578+0000) 2022-09-10T14:07:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:49.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:49.204252+0000) 2022-09-10T14:07:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:07:49.216+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:07:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:49.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:49.586807+0000) 2022-09-10T14:07:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:50.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:50.204501+0000) 2022-09-10T14:07:50.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:50 smithi110 ceph-mon[97633]: pgmap v18512: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:50 smithi104 ceph-mon[97044]: pgmap v18512: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:07:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:50.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:50.587048+0000) 2022-09-10T14:07:51.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:51.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:51.204741+0000) 2022-09-10T14:07:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:51.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:51.587280+0000) 2022-09-10T14:07:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:52.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:52.204923+0000) 2022-09-10T14:07:52.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:52 smithi110 ceph-mon[97633]: pgmap v18513: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:52 smithi104 ceph-mon[97044]: pgmap v18513: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:52.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:52.587520+0000) 2022-09-10T14:07:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:53.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:53.205185+0000) 2022-09-10T14:07:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:53.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:53.587800+0000) 2022-09-10T14:07:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:54.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:54.205442+0000) 2022-09-10T14:07:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:54 smithi110 ceph-mon[97633]: pgmap v18514: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:07:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:54 smithi104 ceph-mon[97044]: pgmap v18514: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:07:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:54.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:54.588030+0000) 2022-09-10T14:07:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:55.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:55.205695+0000) 2022-09-10T14:07:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:55.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:55.588251+0000) 2022-09-10T14:07:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:56.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:56.205896+0000) 2022-09-10T14:07:56.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:56 smithi110 ceph-mon[97633]: pgmap v18515: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:56 smithi104 ceph-mon[97044]: pgmap v18515: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:07:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:56.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:56.588444+0000) 2022-09-10T14:07:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:57.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:57.206121+0000) 2022-09-10T14:07:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:57.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:57.588618+0000) 2022-09-10T14:07:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:58.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:58.206357+0000) 2022-09-10T14:07:58.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:07:58 smithi110 ceph-mon[97633]: pgmap v18516: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:07:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:07:58 smithi104 ceph-mon[97044]: pgmap v18516: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:07:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:58.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:58.588771+0000) 2022-09-10T14:07:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:07:59.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:59.206541+0000) 2022-09-10T14:08:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:07:59.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:07:59.588953+0000) 2022-09-10T14:08:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:00.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:00.206697+0000) 2022-09-10T14:08:00.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:00 smithi110 ceph-mon[97633]: pgmap v18517: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:08:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:00 smithi104 ceph-mon[97044]: pgmap v18517: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:08:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:00.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:00.589206+0000) 2022-09-10T14:08:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:01.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:01.206915+0000) 2022-09-10T14:08:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:01.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:01.589436+0000) 2022-09-10T14:08:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:02.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:02.207125+0000) 2022-09-10T14:08:02.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:02 smithi110 ceph-mon[97633]: pgmap v18518: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:02 smithi104 ceph-mon[97044]: pgmap v18518: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:02.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:02.589666+0000) 2022-09-10T14:08:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:03.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:03.207353+0000) 2022-09-10T14:08:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:03.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:03.589882+0000) 2022-09-10T14:08:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:04.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:04.207566+0000) 2022-09-10T14:08:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:08:04.232+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:08:04.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:04 smithi110 ceph-mon[97633]: pgmap v18519: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:04 smithi104 ceph-mon[97044]: pgmap v18519: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:04.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:04.590160+0000) 2022-09-10T14:08:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:05.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:05.207816+0000) 2022-09-10T14:08:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:05.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:05.590369+0000) 2022-09-10T14:08:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:06.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:06.207981+0000) 2022-09-10T14:08:06.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:06 smithi110 ceph-mon[97633]: pgmap v18520: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:08:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:06 smithi104 ceph-mon[97044]: pgmap v18520: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:08:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:06.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:06.590616+0000) 2022-09-10T14:08:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:07.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:07.208178+0000) 2022-09-10T14:08:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:07.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:07.590859+0000) 2022-09-10T14:08:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:08.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:08.208346+0000) 2022-09-10T14:08:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:08 smithi104 ceph-mon[97044]: pgmap v18521: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:08:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:08 smithi110 ceph-mon[97633]: pgmap v18521: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:08:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:08.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:08.591089+0000) 2022-09-10T14:08:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:09.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:09.208577+0000) 2022-09-10T14:08:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:09 smithi104 ceph-mon[97044]: osdmap e1238: 12 total, 12 up, 12 in 2022-09-10T14:08:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:09 smithi110 ceph-mon[97633]: osdmap e1238: 12 total, 12 up, 12 in 2022-09-10T14:08:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:09.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:09.591263+0000) 2022-09-10T14:08:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:10.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:10.208807+0000) 2022-09-10T14:08:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:10 smithi104 ceph-mon[97044]: pgmap v18523: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:08:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:10 smithi110 ceph-mon[97633]: pgmap v18523: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:08:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:10.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:10.591470+0000) 2022-09-10T14:08:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:11.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:11.208987+0000) 2022-09-10T14:08:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:11.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:11.591723+0000) 2022-09-10T14:08:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:12.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:12.209194+0000) 2022-09-10T14:08:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:12 smithi104 ceph-mon[97044]: pgmap v18524: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:08:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:12 smithi110 ceph-mon[97633]: pgmap v18524: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:08:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:12.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:12.591931+0000) 2022-09-10T14:08:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:13.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:13.209445+0000) 2022-09-10T14:08:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:13.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:13.592171+0000) 2022-09-10T14:08:14.564 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:14.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:14.209637+0000) 2022-09-10T14:08:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:14 smithi104 ceph-mon[97044]: pgmap v18525: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:08:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:14 smithi110 ceph-mon[97633]: pgmap v18525: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:08:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:14.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:14.592404+0000) 2022-09-10T14:08:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:15.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:15.209817+0000) 2022-09-10T14:08:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:15.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:15.592638+0000) 2022-09-10T14:08:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:16.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:16.210058+0000) 2022-09-10T14:08:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:16 smithi104 ceph-mon[97044]: pgmap v18526: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:08:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:16 smithi104 ceph-mon[97044]: osdmap e1239: 12 total, 12 up, 12 in 2022-09-10T14:08:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:16 smithi110 ceph-mon[97633]: pgmap v18526: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:08:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:16 smithi110 ceph-mon[97633]: osdmap e1239: 12 total, 12 up, 12 in 2022-09-10T14:08:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:16.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:16.592878+0000) 2022-09-10T14:08:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:17.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:17.210231+0000) 2022-09-10T14:08:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:17.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:17.593126+0000) 2022-09-10T14:08:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:18.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:18.210430+0000) 2022-09-10T14:08:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:18 smithi104 ceph-mon[97044]: pgmap v18528: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T14:08:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:18 smithi110 ceph-mon[97633]: pgmap v18528: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T14:08:19.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:18.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:18.593340+0000) 2022-09-10T14:08:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:19.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:19.210648+0000) 2022-09-10T14:08:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:08:19.243+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:08:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:19.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:19.593584+0000) 2022-09-10T14:08:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:20.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:20.210916+0000) 2022-09-10T14:08:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:20 smithi104 ceph-mon[97044]: pgmap v18529: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:08:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:20 smithi110 ceph-mon[97633]: pgmap v18529: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:08:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:20.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:20.593876+0000) 2022-09-10T14:08:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:21.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:21.211208+0000) 2022-09-10T14:08:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:21.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:21.594032+0000) 2022-09-10T14:08:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:22.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:22.211380+0000) 2022-09-10T14:08:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:22 smithi104 ceph-mon[97044]: pgmap v18530: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:08:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:22 smithi110 ceph-mon[97633]: pgmap v18530: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:08:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:22.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:22.594274+0000) 2022-09-10T14:08:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:23.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:23.211544+0000) 2022-09-10T14:08:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:23.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:23.594479+0000) 2022-09-10T14:08:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:24.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:24.211805+0000) 2022-09-10T14:08:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:24 smithi104 ceph-mon[97044]: pgmap v18531: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:08:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:24 smithi110 ceph-mon[97633]: pgmap v18531: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:08:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:24.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:24.594644+0000) 2022-09-10T14:08:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:25.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:25.212054+0000) 2022-09-10T14:08:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:25.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:25.594857+0000) 2022-09-10T14:08:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:26.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:26.212384+0000) 2022-09-10T14:08:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:26 smithi104 ceph-mon[97044]: pgmap v18532: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:08:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:26 smithi110 ceph-mon[97633]: pgmap v18532: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:08:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:26.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:26.595045+0000) 2022-09-10T14:08:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:27.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:27.212642+0000) 2022-09-10T14:08:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:27.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:27.595284+0000) 2022-09-10T14:08:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:28.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:28.212866+0000) 2022-09-10T14:08:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:28 smithi104 ceph-mon[97044]: pgmap v18533: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:08:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:28 smithi110 ceph-mon[97633]: pgmap v18533: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:08:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:28.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:28.595558+0000) 2022-09-10T14:08:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:29.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:29.213074+0000) 2022-09-10T14:08:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:29.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:29.595814+0000) 2022-09-10T14:08:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:30.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:30.213346+0000) 2022-09-10T14:08:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:30 smithi104 ceph-mon[97044]: pgmap v18534: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:30 smithi110 ceph-mon[97633]: pgmap v18534: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:30.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:30.596068+0000) 2022-09-10T14:08:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:31.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:31.213560+0000) 2022-09-10T14:08:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:31.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:31.596347+0000) 2022-09-10T14:08:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:32.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:32.213789+0000) 2022-09-10T14:08:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:32 smithi104 ceph-mon[97044]: pgmap v18535: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:32 smithi110 ceph-mon[97633]: pgmap v18535: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:32.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:32.596579+0000) 2022-09-10T14:08:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:33.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:33.213984+0000) 2022-09-10T14:08:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:33.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:33.596860+0000) 2022-09-10T14:08:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:34.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:34.214224+0000) 2022-09-10T14:08:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:08:34.254+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:08:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:34 smithi104 ceph-mon[97044]: pgmap v18536: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:34 smithi110 ceph-mon[97633]: pgmap v18536: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:34.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:34.597139+0000) 2022-09-10T14:08:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:35.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:35.214481+0000) 2022-09-10T14:08:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:35.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:35.597326+0000) 2022-09-10T14:08:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:36.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:36.214739+0000) 2022-09-10T14:08:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:36 smithi104 ceph-mon[97044]: pgmap v18537: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:36 smithi110 ceph-mon[97633]: pgmap v18537: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:36.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:36.597553+0000) 2022-09-10T14:08:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:37.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:37.214979+0000) 2022-09-10T14:08:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:37.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:37.597818+0000) 2022-09-10T14:08:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:38.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:38.215278+0000) 2022-09-10T14:08:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:38 smithi104 ceph-mon[97044]: pgmap v18538: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:38 smithi110 ceph-mon[97633]: pgmap v18538: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:38.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:38.598037+0000) 2022-09-10T14:08:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:39.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:39.215502+0000) 2022-09-10T14:08:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:39.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:39.598180+0000) 2022-09-10T14:08:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:40.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:40.215740+0000) 2022-09-10T14:08:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:40 smithi104 ceph-mon[97044]: pgmap v18539: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:40 smithi110 ceph-mon[97633]: pgmap v18539: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:40.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:40.598434+0000) 2022-09-10T14:08:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:41.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:41.216011+0000) 2022-09-10T14:08:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:41.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:41.598662+0000) 2022-09-10T14:08:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:42.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:42.216206+0000) 2022-09-10T14:08:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:42 smithi104 ceph-mon[97044]: pgmap v18540: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:42 smithi110 ceph-mon[97633]: pgmap v18540: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:42.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:42.598887+0000) 2022-09-10T14:08:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:43.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:43.216403+0000) 2022-09-10T14:08:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:43.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:43.599044+0000) 2022-09-10T14:08:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:44.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:44.216692+0000) 2022-09-10T14:08:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:44 smithi104 ceph-mon[97044]: pgmap v18541: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:44 smithi110 ceph-mon[97633]: pgmap v18541: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:44.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:44.599218+0000) 2022-09-10T14:08:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:45.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:45.216956+0000) 2022-09-10T14:08:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:45.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:45.599336+0000) 2022-09-10T14:08:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:46.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:46.217219+0000) 2022-09-10T14:08:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:46 smithi104 ceph-mon[97044]: pgmap v18542: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:46 smithi110 ceph-mon[97633]: pgmap v18542: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:46.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:46.599550+0000) 2022-09-10T14:08:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:47.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:47.217456+0000) 2022-09-10T14:08:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:08:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:08:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:08:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:08:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:08:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:08:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:08:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:08:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:47.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:47.599732+0000) 2022-09-10T14:08:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:48.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:48.217721+0000) 2022-09-10T14:08:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:48 smithi104 ceph-mon[97044]: pgmap v18543: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:48 smithi110 ceph-mon[97633]: pgmap v18543: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:48.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:48.599952+0000) 2022-09-10T14:08:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:49.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:49.217940+0000) 2022-09-10T14:08:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:08:49.269+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:08:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:49.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:49.600241+0000) 2022-09-10T14:08:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:50.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:50.218209+0000) 2022-09-10T14:08:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:50 smithi104 ceph-mon[97044]: pgmap v18544: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:50 smithi110 ceph-mon[97633]: pgmap v18544: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:50.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:50.600417+0000) 2022-09-10T14:08:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:51.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:51.218447+0000) 2022-09-10T14:08:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:51.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:51.600601+0000) 2022-09-10T14:08:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:52.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:52.218671+0000) 2022-09-10T14:08:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:52 smithi104 ceph-mon[97044]: pgmap v18545: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:52 smithi110 ceph-mon[97633]: pgmap v18545: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:52.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:52.600801+0000) 2022-09-10T14:08:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:53.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:53.218944+0000) 2022-09-10T14:08:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:53.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:53.601036+0000) 2022-09-10T14:08:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:54.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:54.219231+0000) 2022-09-10T14:08:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:54 smithi104 ceph-mon[97044]: pgmap v18546: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:54 smithi110 ceph-mon[97633]: pgmap v18546: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:54.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:54.601203+0000) 2022-09-10T14:08:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:55.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:55.219412+0000) 2022-09-10T14:08:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:55.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:55.601444+0000) 2022-09-10T14:08:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:56.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:56.219614+0000) 2022-09-10T14:08:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:56 smithi104 ceph-mon[97044]: pgmap v18547: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:56 smithi110 ceph-mon[97633]: pgmap v18547: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:08:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:56.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:56.601628+0000) 2022-09-10T14:08:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:57.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:57.219883+0000) 2022-09-10T14:08:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:57.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:57.601837+0000) 2022-09-10T14:08:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:58.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:58.220173+0000) 2022-09-10T14:08:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:08:58 smithi104 ceph-mon[97044]: pgmap v18548: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:08:58 smithi110 ceph-mon[97633]: pgmap v18548: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:08:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:58.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:58.602054+0000) 2022-09-10T14:08:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:08:59.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:59.220392+0000) 2022-09-10T14:09:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:08:59.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:08:59.602332+0000) 2022-09-10T14:09:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:00.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:00.220558+0000) 2022-09-10T14:09:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:00 smithi104 ceph-mon[97044]: pgmap v18549: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:00 smithi110 ceph-mon[97633]: pgmap v18549: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:00.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:00.602621+0000) 2022-09-10T14:09:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:01.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:01.220771+0000) 2022-09-10T14:09:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:01.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:01.602876+0000) 2022-09-10T14:09:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:02.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:02.221046+0000) 2022-09-10T14:09:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:02 smithi104 ceph-mon[97044]: pgmap v18550: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:02 smithi110 ceph-mon[97633]: pgmap v18550: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:02.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:02.603126+0000) 2022-09-10T14:09:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:03.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:03.221328+0000) 2022-09-10T14:09:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:03.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:03.603323+0000) 2022-09-10T14:09:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:04.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:04.221561+0000) 2022-09-10T14:09:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:09:04.283+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:09:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:04 smithi104 ceph-mon[97044]: pgmap v18551: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:04 smithi110 ceph-mon[97633]: pgmap v18551: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:04.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:04.603498+0000) 2022-09-10T14:09:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:05.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:05.221761+0000) 2022-09-10T14:09:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:05.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:05.603742+0000) 2022-09-10T14:09:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:06.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:06.221964+0000) 2022-09-10T14:09:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:06 smithi104 ceph-mon[97044]: pgmap v18552: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:09:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:06 smithi110 ceph-mon[97633]: pgmap v18552: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:09:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:06.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:06.603983+0000) 2022-09-10T14:09:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:07.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:07.222212+0000) 2022-09-10T14:09:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:07.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:07.604211+0000) 2022-09-10T14:09:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:08.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:08.222416+0000) 2022-09-10T14:09:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:08 smithi104 ceph-mon[97044]: pgmap v18553: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:09:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:08 smithi110 ceph-mon[97633]: pgmap v18553: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:09:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:08.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:08.604394+0000) 2022-09-10T14:09:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:09.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:09.222664+0000) 2022-09-10T14:09:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:09 smithi104 ceph-mon[97044]: osdmap e1240: 12 total, 12 up, 12 in 2022-09-10T14:09:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:09 smithi110 ceph-mon[97633]: osdmap e1240: 12 total, 12 up, 12 in 2022-09-10T14:09:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:09.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:09.604635+0000) 2022-09-10T14:09:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:10.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:10.222872+0000) 2022-09-10T14:09:10.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:10 smithi104 ceph-mon[97044]: pgmap v18555: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:09:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:10 smithi110 ceph-mon[97633]: pgmap v18555: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:09:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:10.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:10.604851+0000) 2022-09-10T14:09:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:11.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:11.223138+0000) 2022-09-10T14:09:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:11.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:11.605130+0000) 2022-09-10T14:09:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:12.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:12.223379+0000) 2022-09-10T14:09:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:12 smithi104 ceph-mon[97044]: pgmap v18556: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:09:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:12 smithi110 ceph-mon[97633]: pgmap v18556: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:09:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:12.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:12.605320+0000) 2022-09-10T14:09:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:13.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:13.223584+0000) 2022-09-10T14:09:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:13.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:13.605531+0000) 2022-09-10T14:09:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:14.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:14.223803+0000) 2022-09-10T14:09:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:14 smithi104 ceph-mon[97044]: pgmap v18557: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:09:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:14 smithi110 ceph-mon[97633]: pgmap v18557: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:09:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:14.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:14.605755+0000) 2022-09-10T14:09:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:15.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:15.224033+0000) 2022-09-10T14:09:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:15.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:15.606011+0000) 2022-09-10T14:09:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:16.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:16.224269+0000) 2022-09-10T14:09:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:16 smithi104 ceph-mon[97044]: pgmap v18558: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:09:16.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:16 smithi104 ceph-mon[97044]: osdmap e1241: 12 total, 12 up, 12 in 2022-09-10T14:09:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:16 smithi110 ceph-mon[97633]: pgmap v18558: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:09:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:16 smithi110 ceph-mon[97633]: osdmap e1241: 12 total, 12 up, 12 in 2022-09-10T14:09:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:16.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:16.606238+0000) 2022-09-10T14:09:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:17.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:17.224432+0000) 2022-09-10T14:09:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:17.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:17.606461+0000) 2022-09-10T14:09:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:18.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:18.224593+0000) 2022-09-10T14:09:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:18 smithi104 ceph-mon[97044]: pgmap v18560: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T14:09:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:18 smithi110 ceph-mon[97633]: pgmap v18560: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T14:09:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:18.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:18.606701+0000) 2022-09-10T14:09:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:19.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:19.224843+0000) 2022-09-10T14:09:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:09:19.289+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:09:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:19.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:19.606964+0000) 2022-09-10T14:09:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:20.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:20.225059+0000) 2022-09-10T14:09:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:20 smithi104 ceph-mon[97044]: pgmap v18561: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:09:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:20 smithi110 ceph-mon[97633]: pgmap v18561: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:09:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:20.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:20.607230+0000) 2022-09-10T14:09:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:21.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:21.225270+0000) 2022-09-10T14:09:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:21.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:21.607474+0000) 2022-09-10T14:09:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:22.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:22.225519+0000) 2022-09-10T14:09:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:22 smithi104 ceph-mon[97044]: pgmap v18562: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:09:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:22 smithi110 ceph-mon[97633]: pgmap v18562: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:09:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:22.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:22.607661+0000) 2022-09-10T14:09:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:23.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:23.225744+0000) 2022-09-10T14:09:24.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:23.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:23.607943+0000) 2022-09-10T14:09:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:24.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:24.225967+0000) 2022-09-10T14:09:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:24 smithi104 ceph-mon[97044]: pgmap v18563: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:09:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:24 smithi110 ceph-mon[97633]: pgmap v18563: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:09:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:24.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:24.608160+0000) 2022-09-10T14:09:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:25.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:25.226164+0000) 2022-09-10T14:09:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:25.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:25.608394+0000) 2022-09-10T14:09:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:26.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:26.226382+0000) 2022-09-10T14:09:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:26 smithi104 ceph-mon[97044]: pgmap v18564: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:09:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:26 smithi110 ceph-mon[97633]: pgmap v18564: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:09:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:26.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:26.608613+0000) 2022-09-10T14:09:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:27.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:27.226608+0000) 2022-09-10T14:09:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:27.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:27.608849+0000) 2022-09-10T14:09:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:28.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:28.226870+0000) 2022-09-10T14:09:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:28 smithi110 ceph-mon[97633]: pgmap v18565: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:09:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:28 smithi104 ceph-mon[97044]: pgmap v18565: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:09:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:28.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:28.609101+0000) 2022-09-10T14:09:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:29.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:29.227036+0000) 2022-09-10T14:09:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:29.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:29.609355+0000) 2022-09-10T14:09:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:30.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:30.227270+0000) 2022-09-10T14:09:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:30 smithi110 ceph-mon[97633]: pgmap v18566: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:31.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:30 smithi104 ceph-mon[97044]: pgmap v18566: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:30.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:30.609563+0000) 2022-09-10T14:09:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:31.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:31.227461+0000) 2022-09-10T14:09:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:31.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:31.609696+0000) 2022-09-10T14:09:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:32.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:32.227641+0000) 2022-09-10T14:09:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:32 smithi110 ceph-mon[97633]: pgmap v18567: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:32 smithi104 ceph-mon[97044]: pgmap v18567: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:32.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:32.609850+0000) 2022-09-10T14:09:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:33.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:33.227817+0000) 2022-09-10T14:09:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:33.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:33.610127+0000) 2022-09-10T14:09:34.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:34.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:34.228034+0000) 2022-09-10T14:09:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:09:34.305+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:09:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:34 smithi110 ceph-mon[97633]: pgmap v18568: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:34 smithi104 ceph-mon[97044]: pgmap v18568: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:34.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:34.610323+0000) 2022-09-10T14:09:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:35.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:35.228276+0000) 2022-09-10T14:09:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:35.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:35.610492+0000) 2022-09-10T14:09:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:36.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:36.228463+0000) 2022-09-10T14:09:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:36 smithi110 ceph-mon[97633]: pgmap v18569: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:36 smithi104 ceph-mon[97044]: pgmap v18569: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:36.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:36.610659+0000) 2022-09-10T14:09:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:37.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:37.228642+0000) 2022-09-10T14:09:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:37.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:37.610913+0000) 2022-09-10T14:09:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:38.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:38.228864+0000) 2022-09-10T14:09:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:38 smithi110 ceph-mon[97633]: pgmap v18570: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:38 smithi104 ceph-mon[97044]: pgmap v18570: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:38.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:38.611175+0000) 2022-09-10T14:09:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:39.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:39.229031+0000) 2022-09-10T14:09:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:39.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:39.611301+0000) 2022-09-10T14:09:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:40.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:40.229264+0000) 2022-09-10T14:09:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:40 smithi110 ceph-mon[97633]: pgmap v18571: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:40 smithi104 ceph-mon[97044]: pgmap v18571: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:40.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:40.611477+0000) 2022-09-10T14:09:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:41.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:41.229480+0000) 2022-09-10T14:09:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:41.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:41.611705+0000) 2022-09-10T14:09:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:42.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:42.229705+0000) 2022-09-10T14:09:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:42 smithi110 ceph-mon[97633]: pgmap v18572: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:42 smithi104 ceph-mon[97044]: pgmap v18572: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:42.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:42.611953+0000) 2022-09-10T14:09:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:43.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:43.229962+0000) 2022-09-10T14:09:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:43.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:43.612157+0000) 2022-09-10T14:09:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:44.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:44.230171+0000) 2022-09-10T14:09:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:44 smithi110 ceph-mon[97633]: pgmap v18573: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:44 smithi104 ceph-mon[97044]: pgmap v18573: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:44.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:44.612320+0000) 2022-09-10T14:09:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:45.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:45.230397+0000) 2022-09-10T14:09:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:45.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:45.612553+0000) 2022-09-10T14:09:46.563 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:46.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:46.230662+0000) 2022-09-10T14:09:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:46 smithi110 ceph-mon[97633]: pgmap v18574: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:09:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:09:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:09:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:09:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:46 smithi104 ceph-mon[97044]: pgmap v18574: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:09:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:09:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:09:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:09:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:46.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:46.612796+0000) 2022-09-10T14:09:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:47.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:47.230828+0000) 2022-09-10T14:09:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:47.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:47.612943+0000) 2022-09-10T14:09:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:48.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:48.231029+0000) 2022-09-10T14:09:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:48 smithi110 ceph-mon[97633]: pgmap v18575: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:48 smithi104 ceph-mon[97044]: pgmap v18575: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:48.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:48.613172+0000) 2022-09-10T14:09:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:09:49.318+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:09:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:49.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:49.231241+0000) 2022-09-10T14:09:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:49.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:49.613376+0000) 2022-09-10T14:09:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:50.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:50.231459+0000) 2022-09-10T14:09:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:50 smithi110 ceph-mon[97633]: pgmap v18576: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:50 smithi104 ceph-mon[97044]: pgmap v18576: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:50.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:50.613618+0000) 2022-09-10T14:09:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:51.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:51.231688+0000) 2022-09-10T14:09:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:51.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:51.613888+0000) 2022-09-10T14:09:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:52.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:52.231938+0000) 2022-09-10T14:09:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:52 smithi110 ceph-mon[97633]: pgmap v18577: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:52 smithi104 ceph-mon[97044]: pgmap v18577: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:52.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:52.614077+0000) 2022-09-10T14:09:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:53.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:53.232140+0000) 2022-09-10T14:09:54.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:53.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:53.614293+0000) 2022-09-10T14:09:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:54.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:54.232313+0000) 2022-09-10T14:09:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:54 smithi110 ceph-mon[97633]: pgmap v18578: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:54 smithi104 ceph-mon[97044]: pgmap v18578: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:54.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:54.614537+0000) 2022-09-10T14:09:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:55.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:55.232527+0000) 2022-09-10T14:09:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:55.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:55.614787+0000) 2022-09-10T14:09:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:56.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:56.232780+0000) 2022-09-10T14:09:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:56 smithi110 ceph-mon[97633]: pgmap v18579: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:56 smithi104 ceph-mon[97044]: pgmap v18579: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:09:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:56.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:56.615022+0000) 2022-09-10T14:09:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:57.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:57.233044+0000) 2022-09-10T14:09:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:57.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:57.615271+0000) 2022-09-10T14:09:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:58.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:58.233264+0000) 2022-09-10T14:09:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:09:58 smithi110 ceph-mon[97633]: pgmap v18580: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:09:58 smithi104 ceph-mon[97044]: pgmap v18580: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:09:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:58.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:58.615442+0000) 2022-09-10T14:09:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:09:59.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:59.233535+0000) 2022-09-10T14:10:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:09:59.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:09:59.615672+0000) 2022-09-10T14:10:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:00.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:00.233777+0000) 2022-09-10T14:10:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:00 smithi110 ceph-mon[97633]: pgmap v18581: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:10:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:00 smithi104 ceph-mon[97044]: pgmap v18581: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:10:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:00.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:00.615894+0000) 2022-09-10T14:10:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:01.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:01.234014+0000) 2022-09-10T14:10:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:01.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:01.616079+0000) 2022-09-10T14:10:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:02.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:02.234243+0000) 2022-09-10T14:10:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:02 smithi110 ceph-mon[97633]: pgmap v18582: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:02 smithi104 ceph-mon[97044]: pgmap v18582: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:02.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:02.616247+0000) 2022-09-10T14:10:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:03.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:03.234507+0000) 2022-09-10T14:10:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:03.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:03.616434+0000) 2022-09-10T14:10:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:04.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:04.234777+0000) 2022-09-10T14:10:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:10:04.329+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:10:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:04 smithi110 ceph-mon[97633]: pgmap v18583: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:04 smithi104 ceph-mon[97044]: pgmap v18583: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:04.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:04.616657+0000) 2022-09-10T14:10:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:05.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:05.235011+0000) 2022-09-10T14:10:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:05.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:05.616897+0000) 2022-09-10T14:10:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:06.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:06.235266+0000) 2022-09-10T14:10:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:06 smithi110 ceph-mon[97633]: pgmap v18584: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:10:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:06 smithi104 ceph-mon[97044]: pgmap v18584: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:10:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:06.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:06.617157+0000) 2022-09-10T14:10:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:07.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:07.235484+0000) 2022-09-10T14:10:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:07.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:07.617304+0000) 2022-09-10T14:10:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:08.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:08.235663+0000) 2022-09-10T14:10:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:08 smithi110 ceph-mon[97633]: pgmap v18585: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:10:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:08 smithi104 ceph-mon[97044]: pgmap v18585: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:10:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:08.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:08.617462+0000) 2022-09-10T14:10:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:09.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:09.235911+0000) 2022-09-10T14:10:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:09 smithi110 ceph-mon[97633]: osdmap e1242: 12 total, 12 up, 12 in 2022-09-10T14:10:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:09 smithi104 ceph-mon[97044]: osdmap e1242: 12 total, 12 up, 12 in 2022-09-10T14:10:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:09.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:09.617732+0000) 2022-09-10T14:10:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:10.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:10.236175+0000) 2022-09-10T14:10:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:10 smithi110 ceph-mon[97633]: pgmap v18587: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:10:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:10 smithi104 ceph-mon[97044]: pgmap v18587: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:10:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:10.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:10.618011+0000) 2022-09-10T14:10:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:11.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:11.236391+0000) 2022-09-10T14:10:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:11.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:11.618264+0000) 2022-09-10T14:10:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:12.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:12.236590+0000) 2022-09-10T14:10:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:12 smithi110 ceph-mon[97633]: pgmap v18588: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:10:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:12 smithi104 ceph-mon[97044]: pgmap v18588: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:10:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:12.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:12.618478+0000) 2022-09-10T14:10:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:13.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:13.236768+0000) 2022-09-10T14:10:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:13.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:13.618712+0000) 2022-09-10T14:10:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:14.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:14.237011+0000) 2022-09-10T14:10:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:14 smithi110 ceph-mon[97633]: pgmap v18589: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:10:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:14 smithi104 ceph-mon[97044]: pgmap v18589: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:10:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:14.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:14.618903+0000) 2022-09-10T14:10:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:15.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:15.237217+0000) 2022-09-10T14:10:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:15.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:15.619071+0000) 2022-09-10T14:10:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:16.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:16.237495+0000) 2022-09-10T14:10:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:16 smithi110 ceph-mon[97633]: pgmap v18590: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:10:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:16 smithi110 ceph-mon[97633]: osdmap e1243: 12 total, 12 up, 12 in 2022-09-10T14:10:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:16 smithi104 ceph-mon[97044]: pgmap v18590: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:10:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:16 smithi104 ceph-mon[97044]: osdmap e1243: 12 total, 12 up, 12 in 2022-09-10T14:10:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:16.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:16.619361+0000) 2022-09-10T14:10:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:17.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:17.237705+0000) 2022-09-10T14:10:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:17.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:17.619603+0000) 2022-09-10T14:10:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:18.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:18.237925+0000) 2022-09-10T14:10:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:18 smithi110 ceph-mon[97633]: pgmap v18592: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T14:10:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:18 smithi104 ceph-mon[97044]: pgmap v18592: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 245 B/s wr, 12 op/s 2022-09-10T14:10:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:18.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:18.619827+0000) 2022-09-10T14:10:19.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:10:19.343+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:10:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:19.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:19.238071+0000) 2022-09-10T14:10:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:19.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:19.620075+0000) 2022-09-10T14:10:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:20.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:20.238301+0000) 2022-09-10T14:10:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:20 smithi110 ceph-mon[97633]: pgmap v18593: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:10:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:20 smithi104 ceph-mon[97044]: pgmap v18593: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:10:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:20.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:20.620374+0000) 2022-09-10T14:10:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:21.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:21.238544+0000) 2022-09-10T14:10:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:21.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:21.620613+0000) 2022-09-10T14:10:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:22.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:22.238782+0000) 2022-09-10T14:10:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:22 smithi110 ceph-mon[97633]: pgmap v18594: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:10:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:22 smithi104 ceph-mon[97044]: pgmap v18594: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:10:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:22.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:22.620833+0000) 2022-09-10T14:10:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:23.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:23.239031+0000) 2022-09-10T14:10:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:23.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:23.621128+0000) 2022-09-10T14:10:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:24.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:24.239245+0000) 2022-09-10T14:10:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:24 smithi110 ceph-mon[97633]: pgmap v18595: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:10:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:24 smithi104 ceph-mon[97044]: pgmap v18595: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:10:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:24.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:24.621361+0000) 2022-09-10T14:10:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:25.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:25.239478+0000) 2022-09-10T14:10:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:25.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:25.621503+0000) 2022-09-10T14:10:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:26.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:26.239680+0000) 2022-09-10T14:10:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:26 smithi110 ceph-mon[97633]: pgmap v18596: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:10:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:26 smithi104 ceph-mon[97044]: pgmap v18596: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:10:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:26.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:26.621693+0000) 2022-09-10T14:10:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:27.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:27.239802+0000) 2022-09-10T14:10:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:27.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:27.621874+0000) 2022-09-10T14:10:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:28.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:28.240035+0000) 2022-09-10T14:10:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:28 smithi110 ceph-mon[97633]: pgmap v18597: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:10:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:28 smithi104 ceph-mon[97044]: pgmap v18597: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:10:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:28.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:28.622151+0000) 2022-09-10T14:10:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:29.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:29.240290+0000) 2022-09-10T14:10:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:29.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:29.622364+0000) 2022-09-10T14:10:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:30.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:30.240494+0000) 2022-09-10T14:10:30.915 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:30 smithi110 ceph-mon[97633]: pgmap v18598: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:30 smithi104 ceph-mon[97044]: pgmap v18598: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:30.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:30.622585+0000) 2022-09-10T14:10:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:31.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:31.240673+0000) 2022-09-10T14:10:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:31.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:31.622798+0000) 2022-09-10T14:10:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:32.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:32.240942+0000) 2022-09-10T14:10:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:32 smithi110 ceph-mon[97633]: pgmap v18599: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:32 smithi104 ceph-mon[97044]: pgmap v18599: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:32.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:32.623019+0000) 2022-09-10T14:10:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:33.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:33.241197+0000) 2022-09-10T14:10:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:33.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:33.623197+0000) 2022-09-10T14:10:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:10:34.355+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:10:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:34.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:34.241418+0000) 2022-09-10T14:10:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:34 smithi110 ceph-mon[97633]: pgmap v18600: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:34 smithi104 ceph-mon[97044]: pgmap v18600: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:34.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:34.623377+0000) 2022-09-10T14:10:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:35.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:35.241639+0000) 2022-09-10T14:10:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:35.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:35.623629+0000) 2022-09-10T14:10:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:36.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:36.241827+0000) 2022-09-10T14:10:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:36 smithi104 ceph-mon[97044]: pgmap v18601: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:36.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:36.623879+0000) 2022-09-10T14:10:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:36 smithi110 ceph-mon[97633]: pgmap v18601: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:37.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:37.242073+0000) 2022-09-10T14:10:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:37.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:37.624146+0000) 2022-09-10T14:10:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:38.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:38.242282+0000) 2022-09-10T14:10:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:38 smithi104 ceph-mon[97044]: pgmap v18602: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:38.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:38.624362+0000) 2022-09-10T14:10:39.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:38 smithi110 ceph-mon[97633]: pgmap v18602: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:39.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:39.242504+0000) 2022-09-10T14:10:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:39.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:39.624546+0000) 2022-09-10T14:10:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:40.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:40.242731+0000) 2022-09-10T14:10:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:40 smithi104 ceph-mon[97044]: pgmap v18603: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:40.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:40.624780+0000) 2022-09-10T14:10:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:40 smithi110 ceph-mon[97633]: pgmap v18603: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:41.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:41.242929+0000) 2022-09-10T14:10:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:41.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:41.624987+0000) 2022-09-10T14:10:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:42.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:42.243167+0000) 2022-09-10T14:10:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:42 smithi104 ceph-mon[97044]: pgmap v18604: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:42.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:42.625211+0000) 2022-09-10T14:10:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:42 smithi110 ceph-mon[97633]: pgmap v18604: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:43.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:43.243317+0000) 2022-09-10T14:10:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:43.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:43.625409+0000) 2022-09-10T14:10:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:44.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:44.243498+0000) 2022-09-10T14:10:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:44 smithi104 ceph-mon[97044]: pgmap v18605: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:44.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:44.625610+0000) 2022-09-10T14:10:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:44 smithi110 ceph-mon[97633]: pgmap v18605: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:45.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:45.243742+0000) 2022-09-10T14:10:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:45.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:45.625807+0000) 2022-09-10T14:10:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:46.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:46.243999+0000) 2022-09-10T14:10:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:46 smithi104 ceph-mon[97044]: pgmap v18606: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:10:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:10:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:10:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:10:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:46.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:46.626065+0000) 2022-09-10T14:10:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:46 smithi110 ceph-mon[97633]: pgmap v18606: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:10:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:10:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:10:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:10:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:47.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:47.244177+0000) 2022-09-10T14:10:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:47.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:47.626291+0000) 2022-09-10T14:10:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:48.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:48.244375+0000) 2022-09-10T14:10:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:48 smithi104 ceph-mon[97044]: pgmap v18607: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:48.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:48.626484+0000) 2022-09-10T14:10:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:48 smithi110 ceph-mon[97633]: pgmap v18607: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:49.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:49.244626+0000) 2022-09-10T14:10:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:10:49.368+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:10:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:49.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:49.626703+0000) 2022-09-10T14:10:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:50.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:50.244808+0000) 2022-09-10T14:10:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:50 smithi104 ceph-mon[97044]: pgmap v18608: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:50.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:50.626917+0000) 2022-09-10T14:10:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:50 smithi110 ceph-mon[97633]: pgmap v18608: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:51.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:51.245022+0000) 2022-09-10T14:10:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:51.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:51.627190+0000) 2022-09-10T14:10:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:52.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:52.245273+0000) 2022-09-10T14:10:53.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:52 smithi104 ceph-mon[97044]: pgmap v18609: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:52.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:52.627387+0000) 2022-09-10T14:10:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:52 smithi110 ceph-mon[97633]: pgmap v18609: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:53.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:53.245520+0000) 2022-09-10T14:10:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:53.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:53.627616+0000) 2022-09-10T14:10:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:54.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:54.245729+0000) 2022-09-10T14:10:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:54 smithi104 ceph-mon[97044]: pgmap v18610: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:54.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:54.627816+0000) 2022-09-10T14:10:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:54 smithi110 ceph-mon[97633]: pgmap v18610: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:55.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:55.245977+0000) 2022-09-10T14:10:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:55.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:55.628074+0000) 2022-09-10T14:10:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:56.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:56.246240+0000) 2022-09-10T14:10:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:56 smithi104 ceph-mon[97044]: pgmap v18611: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:56.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:56.628249+0000) 2022-09-10T14:10:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:56 smithi110 ceph-mon[97633]: pgmap v18611: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:10:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:57.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:57.246449+0000) 2022-09-10T14:10:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:57.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:57.628459+0000) 2022-09-10T14:10:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:58.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:58.246686+0000) 2022-09-10T14:10:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:10:58 smithi104 ceph-mon[97044]: pgmap v18612: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:58.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:58.628640+0000) 2022-09-10T14:10:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:10:58 smithi110 ceph-mon[97633]: pgmap v18612: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:10:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:10:59.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:59.246921+0000) 2022-09-10T14:11:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:10:59.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:10:59.628837+0000) 2022-09-10T14:11:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:00.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:00.247164+0000) 2022-09-10T14:11:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:00 smithi104 ceph-mon[97044]: pgmap v18613: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:00.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:00.628982+0000) 2022-09-10T14:11:01.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:00 smithi110 ceph-mon[97633]: pgmap v18613: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:01.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:01.247381+0000) 2022-09-10T14:11:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:01.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:01.629166+0000) 2022-09-10T14:11:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:02.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:02.247526+0000) 2022-09-10T14:11:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:02 smithi104 ceph-mon[97044]: pgmap v18614: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:02.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:02.629369+0000) 2022-09-10T14:11:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:02 smithi110 ceph-mon[97633]: pgmap v18614: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:03.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:03.247777+0000) 2022-09-10T14:11:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:03.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:03.629647+0000) 2022-09-10T14:11:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:04.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:04.248005+0000) 2022-09-10T14:11:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:11:04.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:11:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:04 smithi104 ceph-mon[97044]: pgmap v18615: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:04.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:04.629893+0000) 2022-09-10T14:11:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:04 smithi110 ceph-mon[97633]: pgmap v18615: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:05.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:05.248227+0000) 2022-09-10T14:11:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:05.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:05.630169+0000) 2022-09-10T14:11:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:06.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:06.248495+0000) 2022-09-10T14:11:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:06 smithi104 ceph-mon[97044]: pgmap v18616: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:11:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:06.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:06.630396+0000) 2022-09-10T14:11:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:06 smithi110 ceph-mon[97633]: pgmap v18616: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:11:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:07.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:07.248724+0000) 2022-09-10T14:11:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:07.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:07.630614+0000) 2022-09-10T14:11:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:08.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:08.248918+0000) 2022-09-10T14:11:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:08 smithi104 ceph-mon[97044]: pgmap v18617: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:11:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:08.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:08.630791+0000) 2022-09-10T14:11:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:08 smithi110 ceph-mon[97633]: pgmap v18617: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:11:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:09.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:09.249146+0000) 2022-09-10T14:11:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:09 smithi104 ceph-mon[97044]: osdmap e1244: 12 total, 12 up, 12 in 2022-09-10T14:11:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:09.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:09.631060+0000) 2022-09-10T14:11:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:09 smithi110 ceph-mon[97633]: osdmap e1244: 12 total, 12 up, 12 in 2022-09-10T14:11:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:10.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:10.249375+0000) 2022-09-10T14:11:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:10 smithi104 ceph-mon[97044]: pgmap v18619: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:11:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:10.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:10.631325+0000) 2022-09-10T14:11:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:10 smithi110 ceph-mon[97633]: pgmap v18619: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:11:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:11.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:11.249628+0000) 2022-09-10T14:11:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:11.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:11.631559+0000) 2022-09-10T14:11:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:12.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:12.249840+0000) 2022-09-10T14:11:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:12 smithi104 ceph-mon[97044]: pgmap v18620: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:11:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:12.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:12.631794+0000) 2022-09-10T14:11:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:12 smithi110 ceph-mon[97633]: pgmap v18620: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:11:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:13.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:13.250064+0000) 2022-09-10T14:11:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:13.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:13.632025+0000) 2022-09-10T14:11:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:14.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:14.250316+0000) 2022-09-10T14:11:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:14 smithi104 ceph-mon[97044]: pgmap v18621: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:11:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:14.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:14.632178+0000) 2022-09-10T14:11:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:14 smithi110 ceph-mon[97633]: pgmap v18621: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:11:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:15.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:15.250527+0000) 2022-09-10T14:11:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:15.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:15.632357+0000) 2022-09-10T14:11:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:16.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:16.250758+0000) 2022-09-10T14:11:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:16 smithi104 ceph-mon[97044]: pgmap v18622: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:11:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:16 smithi104 ceph-mon[97044]: osdmap e1245: 12 total, 12 up, 12 in 2022-09-10T14:11:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:16.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:16.632506+0000) 2022-09-10T14:11:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:16 smithi110 ceph-mon[97633]: pgmap v18622: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:11:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:16 smithi110 ceph-mon[97633]: osdmap e1245: 12 total, 12 up, 12 in 2022-09-10T14:11:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:17.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:17.250907+0000) 2022-09-10T14:11:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:17.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:17.632718+0000) 2022-09-10T14:11:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:18.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:18.251091+0000) 2022-09-10T14:11:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:18 smithi104 ceph-mon[97044]: pgmap v18624: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T14:11:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:18.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:18.632896+0000) 2022-09-10T14:11:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:18 smithi110 ceph-mon[97633]: pgmap v18624: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 248 B/s wr, 12 op/s 2022-09-10T14:11:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:19.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:19.251236+0000) 2022-09-10T14:11:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:11:19.390+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:11:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:19.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:19.633175+0000) 2022-09-10T14:11:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:20.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:20.251416+0000) 2022-09-10T14:11:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:20 smithi104 ceph-mon[97044]: pgmap v18625: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:11:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:20.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:20.633411+0000) 2022-09-10T14:11:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:20 smithi110 ceph-mon[97633]: pgmap v18625: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:11:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:21.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:21.251569+0000) 2022-09-10T14:11:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:21.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:21.633607+0000) 2022-09-10T14:11:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:22.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:22.251769+0000) 2022-09-10T14:11:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:22 smithi104 ceph-mon[97044]: pgmap v18626: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:11:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:22.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:22.633820+0000) 2022-09-10T14:11:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:22 smithi110 ceph-mon[97633]: pgmap v18626: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:11:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:23.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:23.251938+0000) 2022-09-10T14:11:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:23.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:23.634028+0000) 2022-09-10T14:11:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:24.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:24.252155+0000) 2022-09-10T14:11:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:24 smithi104 ceph-mon[97044]: pgmap v18627: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:11:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:24.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:24.634261+0000) 2022-09-10T14:11:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:24 smithi110 ceph-mon[97633]: pgmap v18627: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:11:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:25.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:25.252291+0000) 2022-09-10T14:11:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:25.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:25.634423+0000) 2022-09-10T14:11:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:26.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:26.252452+0000) 2022-09-10T14:11:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:26 smithi104 ceph-mon[97044]: pgmap v18628: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:11:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:26.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:26.634645+0000) 2022-09-10T14:11:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:26 smithi110 ceph-mon[97633]: pgmap v18628: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:11:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:27.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:27.252612+0000) 2022-09-10T14:11:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:27.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:27.634896+0000) 2022-09-10T14:11:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:28.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:28.252742+0000) 2022-09-10T14:11:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:28 smithi104 ceph-mon[97044]: pgmap v18629: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:11:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:28.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:28.635176+0000) 2022-09-10T14:11:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:28 smithi110 ceph-mon[97633]: pgmap v18629: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:11:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:29.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:29.252870+0000) 2022-09-10T14:11:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:29.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:29.635406+0000) 2022-09-10T14:11:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:30.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:30.253037+0000) 2022-09-10T14:11:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:30 smithi104 ceph-mon[97044]: pgmap v18630: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:30.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:30.635640+0000) 2022-09-10T14:11:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:30 smithi110 ceph-mon[97633]: pgmap v18630: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:31.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:31.253261+0000) 2022-09-10T14:11:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:31.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:31.635840+0000) 2022-09-10T14:11:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:32.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:32.253451+0000) 2022-09-10T14:11:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:32 smithi104 ceph-mon[97044]: pgmap v18631: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:32.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:32.636066+0000) 2022-09-10T14:11:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:32 smithi110 ceph-mon[97633]: pgmap v18631: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:33.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:33.253562+0000) 2022-09-10T14:11:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:33.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:33.636262+0000) 2022-09-10T14:11:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:11:34.403+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:11:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:34.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:34.253816+0000) 2022-09-10T14:11:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:34 smithi104 ceph-mon[97044]: pgmap v18632: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:34.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:34.636489+0000) 2022-09-10T14:11:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:34 smithi110 ceph-mon[97633]: pgmap v18632: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:35.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:35.253987+0000) 2022-09-10T14:11:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:35.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:35.636698+0000) 2022-09-10T14:11:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:36.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:36.254231+0000) 2022-09-10T14:11:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:36 smithi104 ceph-mon[97044]: pgmap v18633: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:36.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:36.636887+0000) 2022-09-10T14:11:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:36 smithi110 ceph-mon[97633]: pgmap v18633: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:37.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:37.254390+0000) 2022-09-10T14:11:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:37.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:37.637136+0000) 2022-09-10T14:11:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:38.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:38.254632+0000) 2022-09-10T14:11:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:38 smithi104 ceph-mon[97044]: pgmap v18634: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:38.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:38.637401+0000) 2022-09-10T14:11:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:38 smithi110 ceph-mon[97633]: pgmap v18634: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:39.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:39.254837+0000) 2022-09-10T14:11:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:39.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:39.637614+0000) 2022-09-10T14:11:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:40.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:40.255036+0000) 2022-09-10T14:11:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:40 smithi104 ceph-mon[97044]: pgmap v18635: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:40.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:40.637774+0000) 2022-09-10T14:11:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:40 smithi110 ceph-mon[97633]: pgmap v18635: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:41.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:41.255259+0000) 2022-09-10T14:11:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:41.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:41.637983+0000) 2022-09-10T14:11:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:42.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:42.255483+0000) 2022-09-10T14:11:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:42 smithi104 ceph-mon[97044]: pgmap v18636: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:42.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:42.638235+0000) 2022-09-10T14:11:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:42 smithi110 ceph-mon[97633]: pgmap v18636: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:43.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:43.255636+0000) 2022-09-10T14:11:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:43.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:43.638495+0000) 2022-09-10T14:11:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:44.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:44.255864+0000) 2022-09-10T14:11:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:44 smithi104 ceph-mon[97044]: pgmap v18637: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:44.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:44.638698+0000) 2022-09-10T14:11:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:44 smithi110 ceph-mon[97633]: pgmap v18637: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:45.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:45.256040+0000) 2022-09-10T14:11:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:45.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:45.638934+0000) 2022-09-10T14:11:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:46.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:46.256251+0000) 2022-09-10T14:11:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:46 smithi104 ceph-mon[97044]: pgmap v18638: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:11:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:11:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:11:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:11:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:46.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:46.639155+0000) 2022-09-10T14:11:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:46 smithi110 ceph-mon[97633]: pgmap v18638: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:11:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:11:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:11:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:11:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:47.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:47.256411+0000) 2022-09-10T14:11:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:47.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:47.639363+0000) 2022-09-10T14:11:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:48.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:48.256664+0000) 2022-09-10T14:11:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:48 smithi104 ceph-mon[97044]: pgmap v18639: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:48.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:48.639591+0000) 2022-09-10T14:11:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:48 smithi110 ceph-mon[97633]: pgmap v18639: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:49.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:11:49.404+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:11:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:49.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:49.256939+0000) 2022-09-10T14:11:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:49.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:49.639822+0000) 2022-09-10T14:11:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:50.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:50.257188+0000) 2022-09-10T14:11:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:50 smithi104 ceph-mon[97044]: pgmap v18640: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:50.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:50.640063+0000) 2022-09-10T14:11:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:50 smithi110 ceph-mon[97633]: pgmap v18640: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:51.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:51.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:51.257392+0000) 2022-09-10T14:11:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:51.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:51.640313+0000) 2022-09-10T14:11:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:52.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:52.257593+0000) 2022-09-10T14:11:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:52 smithi104 ceph-mon[97044]: pgmap v18641: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:52.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:52.640503+0000) 2022-09-10T14:11:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:52 smithi110 ceph-mon[97633]: pgmap v18641: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:53.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:53.257834+0000) 2022-09-10T14:11:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:53.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:53.640726+0000) 2022-09-10T14:11:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:54.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:54.258039+0000) 2022-09-10T14:11:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:54 smithi104 ceph-mon[97044]: pgmap v18642: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:54.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:54.640973+0000) 2022-09-10T14:11:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:54 smithi110 ceph-mon[97633]: pgmap v18642: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:55.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:55.258234+0000) 2022-09-10T14:11:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:55.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:55.641173+0000) 2022-09-10T14:11:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:56.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:56.258508+0000) 2022-09-10T14:11:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:56 smithi104 ceph-mon[97044]: pgmap v18643: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:56.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:56.641377+0000) 2022-09-10T14:11:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:56 smithi110 ceph-mon[97633]: pgmap v18643: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:11:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:57.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:57.258688+0000) 2022-09-10T14:11:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:57.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:57.641555+0000) 2022-09-10T14:11:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:58.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:58.258863+0000) 2022-09-10T14:11:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:11:58 smithi104 ceph-mon[97044]: pgmap v18644: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:58.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:58.641767+0000) 2022-09-10T14:11:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:11:58 smithi110 ceph-mon[97633]: pgmap v18644: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:11:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:11:59.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:59.259090+0000) 2022-09-10T14:12:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:11:59.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:11:59.641982+0000) 2022-09-10T14:12:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:00.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:00.259316+0000) 2022-09-10T14:12:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:00.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:00.642143+0000) 2022-09-10T14:12:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:00 smithi110 ceph-mon[97633]: pgmap v18645: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:00 smithi104 ceph-mon[97044]: pgmap v18645: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:01.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:01.259469+0000) 2022-09-10T14:12:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:01.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:01.642349+0000) 2022-09-10T14:12:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:02.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:02.259700+0000) 2022-09-10T14:12:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:02.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:02.642558+0000) 2022-09-10T14:12:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:02 smithi110 ceph-mon[97633]: pgmap v18646: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:02 smithi104 ceph-mon[97044]: pgmap v18646: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:03.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:03.259889+0000) 2022-09-10T14:12:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:03.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:03.642816+0000) 2022-09-10T14:12:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:04.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:04.260109+0000) 2022-09-10T14:12:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:12:04.417+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:12:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:04.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:04.643063+0000) 2022-09-10T14:12:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:04 smithi110 ceph-mon[97633]: pgmap v18647: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:04 smithi104 ceph-mon[97044]: pgmap v18647: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:05.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:05.260235+0000) 2022-09-10T14:12:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:05.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:05.643323+0000) 2022-09-10T14:12:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:06.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:06.260392+0000) 2022-09-10T14:12:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:06.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:06.643483+0000) 2022-09-10T14:12:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:06 smithi110 ceph-mon[97633]: pgmap v18648: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:12:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:06 smithi104 ceph-mon[97044]: pgmap v18648: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:12:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:07.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:07.260536+0000) 2022-09-10T14:12:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:07.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:07.643684+0000) 2022-09-10T14:12:08.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:07 smithi110 ceph-mon[97633]: pgmap v18649: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:12:08.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:07 smithi104 ceph-mon[97044]: pgmap v18649: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:12:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:08.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:08.260745+0000) 2022-09-10T14:12:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:08.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:08.643924+0000) 2022-09-10T14:12:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:09.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:09.260925+0000) 2022-09-10T14:12:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:09.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:09.644187+0000) 2022-09-10T14:12:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:09 smithi110 ceph-mon[97633]: osdmap e1246: 12 total, 12 up, 12 in 2022-09-10T14:12:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:09 smithi110 ceph-mon[97633]: pgmap v18651: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:12:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:09 smithi104 ceph-mon[97044]: osdmap e1246: 12 total, 12 up, 12 in 2022-09-10T14:12:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:09 smithi104 ceph-mon[97044]: pgmap v18651: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:12:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:10.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:10.261186+0000) 2022-09-10T14:12:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:10.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:10.644351+0000) 2022-09-10T14:12:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:11.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:11.261369+0000) 2022-09-10T14:12:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:11.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:11.644513+0000) 2022-09-10T14:12:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:12.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:12.261573+0000) 2022-09-10T14:12:12.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:12 smithi110 ceph-mon[97633]: pgmap v18652: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:12:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:12 smithi104 ceph-mon[97044]: pgmap v18652: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:12:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:12.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:12.644691+0000) 2022-09-10T14:12:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:13.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:13.261795+0000) 2022-09-10T14:12:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:13.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:13.644919+0000) 2022-09-10T14:12:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:14.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:14.262074+0000) 2022-09-10T14:12:14.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:14 smithi110 ceph-mon[97633]: pgmap v18653: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:12:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:14 smithi104 ceph-mon[97044]: pgmap v18653: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:12:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:14.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:14.645123+0000) 2022-09-10T14:12:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:15.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:15.262311+0000) 2022-09-10T14:12:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:15.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:15.645331+0000) 2022-09-10T14:12:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:16.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:16.262520+0000) 2022-09-10T14:12:16.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:16 smithi110 ceph-mon[97633]: pgmap v18654: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:12:16.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:16 smithi110 ceph-mon[97633]: osdmap e1247: 12 total, 12 up, 12 in 2022-09-10T14:12:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:16 smithi104 ceph-mon[97044]: pgmap v18654: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:12:16.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:16 smithi104 ceph-mon[97044]: osdmap e1247: 12 total, 12 up, 12 in 2022-09-10T14:12:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:16.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:16.645533+0000) 2022-09-10T14:12:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:17.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:17.262748+0000) 2022-09-10T14:12:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:17.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:17.645788+0000) 2022-09-10T14:12:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:18.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:18.262962+0000) 2022-09-10T14:12:18.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:18 smithi110 ceph-mon[97633]: pgmap v18656: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T14:12:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:18 smithi104 ceph-mon[97044]: pgmap v18656: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 250 B/s wr, 12 op/s 2022-09-10T14:12:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:18.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:18.645991+0000) 2022-09-10T14:12:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:12:19.422+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:12:19.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:19.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:19.263232+0000) 2022-09-10T14:12:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:19.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:19.646230+0000) 2022-09-10T14:12:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:20.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:20.263480+0000) 2022-09-10T14:12:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:20 smithi110 ceph-mon[97633]: pgmap v18657: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:12:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:20 smithi104 ceph-mon[97044]: pgmap v18657: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:12:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:20.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:20.646452+0000) 2022-09-10T14:12:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:21.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:21.263724+0000) 2022-09-10T14:12:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:21.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:21.646660+0000) 2022-09-10T14:12:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:22.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:22.263904+0000) 2022-09-10T14:12:22.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:22 smithi110 ceph-mon[97633]: pgmap v18658: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:12:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:22 smithi104 ceph-mon[97044]: pgmap v18658: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:12:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:22.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:22.646894+0000) 2022-09-10T14:12:23.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:23.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:23.264156+0000) 2022-09-10T14:12:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:23.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:23.647084+0000) 2022-09-10T14:12:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:24.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:24.264408+0000) 2022-09-10T14:12:24.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:24 smithi110 ceph-mon[97633]: pgmap v18659: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:12:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:24 smithi104 ceph-mon[97044]: pgmap v18659: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:12:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:24.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:24.647281+0000) 2022-09-10T14:12:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:25.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:25.264638+0000) 2022-09-10T14:12:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:25.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:25.647519+0000) 2022-09-10T14:12:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:26.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:26.264879+0000) 2022-09-10T14:12:26.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:26 smithi110 ceph-mon[97633]: pgmap v18660: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:12:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:26 smithi104 ceph-mon[97044]: pgmap v18660: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:12:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:26.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:26.647737+0000) 2022-09-10T14:12:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:27.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:27.265039+0000) 2022-09-10T14:12:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:27.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:27.647948+0000) 2022-09-10T14:12:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:28.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:28.265288+0000) 2022-09-10T14:12:28.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:28 smithi110 ceph-mon[97633]: pgmap v18661: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:12:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:28 smithi104 ceph-mon[97044]: pgmap v18661: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:12:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:28.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:28.648157+0000) 2022-09-10T14:12:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:29.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:29.265522+0000) 2022-09-10T14:12:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:29.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:29.648349+0000) 2022-09-10T14:12:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:30.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:30.265780+0000) 2022-09-10T14:12:30.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:30 smithi110 ceph-mon[97633]: pgmap v18662: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:30 smithi104 ceph-mon[97044]: pgmap v18662: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:30.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:30.648533+0000) 2022-09-10T14:12:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:31.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:31.266031+0000) 2022-09-10T14:12:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:31.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:31.648723+0000) 2022-09-10T14:12:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:32.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:32.266232+0000) 2022-09-10T14:12:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:32 smithi104 ceph-mon[97044]: pgmap v18663: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:32 smithi110 ceph-mon[97633]: pgmap v18663: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:32.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:32.648933+0000) 2022-09-10T14:12:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:33.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:33.266481+0000) 2022-09-10T14:12:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:33.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:33.649182+0000) 2022-09-10T14:12:34.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:12:34.437+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:12:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:34.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:34.266708+0000) 2022-09-10T14:12:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:34 smithi104 ceph-mon[97044]: pgmap v18664: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:34 smithi110 ceph-mon[97633]: pgmap v18664: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:34.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:34.649436+0000) 2022-09-10T14:12:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:35.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:35.266932+0000) 2022-09-10T14:12:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:35.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:35.649583+0000) 2022-09-10T14:12:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:36.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:36.267083+0000) 2022-09-10T14:12:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:36 smithi104 ceph-mon[97044]: pgmap v18665: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:36 smithi110 ceph-mon[97633]: pgmap v18665: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:36.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:36.649783+0000) 2022-09-10T14:12:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:37.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:37.267293+0000) 2022-09-10T14:12:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:37.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:37.650059+0000) 2022-09-10T14:12:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:38.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:38.267519+0000) 2022-09-10T14:12:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:38 smithi104 ceph-mon[97044]: pgmap v18666: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:38 smithi110 ceph-mon[97633]: pgmap v18666: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:38.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:38.650290+0000) 2022-09-10T14:12:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:39.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:39.267727+0000) 2022-09-10T14:12:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:39.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:39.650500+0000) 2022-09-10T14:12:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:40.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:40.267979+0000) 2022-09-10T14:12:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:40 smithi104 ceph-mon[97044]: pgmap v18667: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:40 smithi110 ceph-mon[97633]: pgmap v18667: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:40.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:40.650731+0000) 2022-09-10T14:12:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:41.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:41.268221+0000) 2022-09-10T14:12:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:41.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:41.650987+0000) 2022-09-10T14:12:42.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:42.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:42.268439+0000) 2022-09-10T14:12:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:42 smithi104 ceph-mon[97044]: pgmap v18668: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:42 smithi110 ceph-mon[97633]: pgmap v18668: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:42.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:42.651182+0000) 2022-09-10T14:12:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:43.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:43.268634+0000) 2022-09-10T14:12:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:43.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:43.651393+0000) 2022-09-10T14:12:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:44.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:44.268816+0000) 2022-09-10T14:12:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:44 smithi104 ceph-mon[97044]: pgmap v18669: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:44 smithi110 ceph-mon[97633]: pgmap v18669: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:44.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:44.651579+0000) 2022-09-10T14:12:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:45.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:45.269024+0000) 2022-09-10T14:12:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:45.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:45.651810+0000) 2022-09-10T14:12:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:46.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:46.269240+0000) 2022-09-10T14:12:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:46 smithi104 ceph-mon[97044]: pgmap v18670: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:46 smithi110 ceph-mon[97633]: pgmap v18670: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:46.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:46.652018+0000) 2022-09-10T14:12:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:47.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:47.269454+0000) 2022-09-10T14:12:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:12:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:12:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:12:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:12:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:12:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:12:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:12:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:12:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:47.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:47.652236+0000) 2022-09-10T14:12:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:48.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:48.269702+0000) 2022-09-10T14:12:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:48 smithi104 ceph-mon[97044]: pgmap v18671: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:48 smithi110 ceph-mon[97633]: pgmap v18671: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:48.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:48.652521+0000) 2022-09-10T14:12:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:49.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:49.269907+0000) 2022-09-10T14:12:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:12:49.452+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:12:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:49.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:49.652681+0000) 2022-09-10T14:12:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:50.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:50.270170+0000) 2022-09-10T14:12:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:50 smithi104 ceph-mon[97044]: pgmap v18672: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:50 smithi110 ceph-mon[97633]: pgmap v18672: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:50.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:50.652913+0000) 2022-09-10T14:12:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:51.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:51.270358+0000) 2022-09-10T14:12:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:51.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:51.653190+0000) 2022-09-10T14:12:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:52.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:52.270570+0000) 2022-09-10T14:12:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:52 smithi104 ceph-mon[97044]: pgmap v18673: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:52 smithi110 ceph-mon[97633]: pgmap v18673: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:52.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:52.653425+0000) 2022-09-10T14:12:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:53.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:53.270747+0000) 2022-09-10T14:12:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:53.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:53.653669+0000) 2022-09-10T14:12:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:54.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:54.270947+0000) 2022-09-10T14:12:54.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:54 smithi104 ceph-mon[97044]: pgmap v18674: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:54 smithi110 ceph-mon[97633]: pgmap v18674: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:54.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:54.653836+0000) 2022-09-10T14:12:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:55.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:55.271163+0000) 2022-09-10T14:12:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:55.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:55.654035+0000) 2022-09-10T14:12:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:56.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:56.271414+0000) 2022-09-10T14:12:56.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:56 smithi104 ceph-mon[97044]: pgmap v18675: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:56 smithi110 ceph-mon[97633]: pgmap v18675: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:12:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:56.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:56.654210+0000) 2022-09-10T14:12:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:57.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:57.271659+0000) 2022-09-10T14:12:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:57.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:57.654355+0000) 2022-09-10T14:12:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:58.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:58.271860+0000) 2022-09-10T14:12:58.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:12:58 smithi104 ceph-mon[97044]: pgmap v18676: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:12:58 smithi110 ceph-mon[97633]: pgmap v18676: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:12:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:58.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:58.654593+0000) 2022-09-10T14:12:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:12:59.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:59.272017+0000) 2022-09-10T14:13:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:12:59.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:12:59.654738+0000) 2022-09-10T14:13:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:00.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:00.272272+0000) 2022-09-10T14:13:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:00 smithi104 ceph-mon[97044]: pgmap v18677: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:00 smithi110 ceph-mon[97633]: pgmap v18677: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:00.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:00.654919+0000) 2022-09-10T14:13:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:01.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:01.272434+0000) 2022-09-10T14:13:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:01.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:01.655160+0000) 2022-09-10T14:13:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:02.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:02.272624+0000) 2022-09-10T14:13:02.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:02 smithi104 ceph-mon[97044]: pgmap v18678: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:02 smithi110 ceph-mon[97633]: pgmap v18678: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:02.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:02.655392+0000) 2022-09-10T14:13:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:03.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:03.272805+0000) 2022-09-10T14:13:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:03.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:03.655551+0000) 2022-09-10T14:13:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:04.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:04.273068+0000) 2022-09-10T14:13:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:13:04.468+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:13:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:04 smithi104 ceph-mon[97044]: pgmap v18679: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:04 smithi110 ceph-mon[97633]: pgmap v18679: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:04.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:04.655767+0000) 2022-09-10T14:13:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:05.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:05.273326+0000) 2022-09-10T14:13:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:05.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:05.655917+0000) 2022-09-10T14:13:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:06.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:06.273583+0000) 2022-09-10T14:13:06.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:06 smithi104 ceph-mon[97044]: pgmap v18680: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:13:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:06 smithi110 ceph-mon[97633]: pgmap v18680: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:13:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:06.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:06.656083+0000) 2022-09-10T14:13:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:07.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:07.273795+0000) 2022-09-10T14:13:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:07.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:07.656284+0000) 2022-09-10T14:13:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:08.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:08.274051+0000) 2022-09-10T14:13:08.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:08 smithi104 ceph-mon[97044]: pgmap v18681: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:13:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:08 smithi110 ceph-mon[97633]: pgmap v18681: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:13:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:08.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:08.656489+0000) 2022-09-10T14:13:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:09.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:09.274332+0000) 2022-09-10T14:13:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:09 smithi104 ceph-mon[97044]: osdmap e1248: 12 total, 12 up, 12 in 2022-09-10T14:13:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:09 smithi110 ceph-mon[97633]: osdmap e1248: 12 total, 12 up, 12 in 2022-09-10T14:13:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:09.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:09.656759+0000) 2022-09-10T14:13:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:10.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:10.274516+0000) 2022-09-10T14:13:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:10 smithi104 ceph-mon[97044]: pgmap v18683: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:13:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:10 smithi110 ceph-mon[97633]: pgmap v18683: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:13:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:10.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:10.657005+0000) 2022-09-10T14:13:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:11.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:11.274691+0000) 2022-09-10T14:13:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:11.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:11.657260+0000) 2022-09-10T14:13:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:12.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:12.274886+0000) 2022-09-10T14:13:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:12 smithi104 ceph-mon[97044]: pgmap v18684: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:13:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:12 smithi110 ceph-mon[97633]: pgmap v18684: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:13:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:12.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:12.657483+0000) 2022-09-10T14:13:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:13.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:13.275159+0000) 2022-09-10T14:13:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:13.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:13.657692+0000) 2022-09-10T14:13:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:14.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:14.275315+0000) 2022-09-10T14:13:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:14 smithi104 ceph-mon[97044]: pgmap v18685: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:13:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:14 smithi110 ceph-mon[97633]: pgmap v18685: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:13:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:14.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:14.657828+0000) 2022-09-10T14:13:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:15.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:15.275559+0000) 2022-09-10T14:13:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:15.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:15.658068+0000) 2022-09-10T14:13:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:16.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:16.275779+0000) 2022-09-10T14:13:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:16 smithi104 ceph-mon[97044]: pgmap v18686: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:13:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:16 smithi104 ceph-mon[97044]: osdmap e1249: 12 total, 12 up, 12 in 2022-09-10T14:13:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:16 smithi110 ceph-mon[97633]: pgmap v18686: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:13:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:16 smithi110 ceph-mon[97633]: osdmap e1249: 12 total, 12 up, 12 in 2022-09-10T14:13:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:16.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:16.658283+0000) 2022-09-10T14:13:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:17.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:17.275967+0000) 2022-09-10T14:13:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:17.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:17.658524+0000) 2022-09-10T14:13:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:18.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:18.276190+0000) 2022-09-10T14:13:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:18 smithi104 ceph-mon[97044]: pgmap v18688: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T14:13:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:18 smithi110 ceph-mon[97633]: pgmap v18688: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T14:13:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:18.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:18.658720+0000) 2022-09-10T14:13:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:19.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:19.276432+0000) 2022-09-10T14:13:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:13:19.484+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:13:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:19.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:19.658956+0000) 2022-09-10T14:13:20.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:20.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:20.276694+0000) 2022-09-10T14:13:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:20 smithi104 ceph-mon[97044]: pgmap v18689: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:13:20.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:20 smithi110 ceph-mon[97633]: pgmap v18689: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:13:21.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:20.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:20.659241+0000) 2022-09-10T14:13:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:21.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:21.276972+0000) 2022-09-10T14:13:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:21.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:21.659431+0000) 2022-09-10T14:13:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:22.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:22.277160+0000) 2022-09-10T14:13:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:22 smithi104 ceph-mon[97044]: pgmap v18690: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:13:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:22 smithi110 ceph-mon[97633]: pgmap v18690: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:13:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:22.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:22.659577+0000) 2022-09-10T14:13:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:23.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:23.277371+0000) 2022-09-10T14:13:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:23.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:23.659805+0000) 2022-09-10T14:13:24.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:24.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:24.277597+0000) 2022-09-10T14:13:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:24 smithi104 ceph-mon[97044]: pgmap v18691: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:13:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:24 smithi110 ceph-mon[97633]: pgmap v18691: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:13:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:24.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:24.660072+0000) 2022-09-10T14:13:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:25.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:25.277766+0000) 2022-09-10T14:13:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:25.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:25.660270+0000) 2022-09-10T14:13:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:26.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:26.278001+0000) 2022-09-10T14:13:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:26 smithi104 ceph-mon[97044]: pgmap v18692: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:13:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:26 smithi110 ceph-mon[97633]: pgmap v18692: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:13:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:26.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:26.660518+0000) 2022-09-10T14:13:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:27.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:27.278236+0000) 2022-09-10T14:13:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:27.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:27.660699+0000) 2022-09-10T14:13:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:28.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:28.278492+0000) 2022-09-10T14:13:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:28 smithi104 ceph-mon[97044]: pgmap v18693: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:13:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:28 smithi110 ceph-mon[97633]: pgmap v18693: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:13:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:28.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:28.660921+0000) 2022-09-10T14:13:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:29.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:29.278771+0000) 2022-09-10T14:13:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:29.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:29.661128+0000) 2022-09-10T14:13:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:30.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:30.278976+0000) 2022-09-10T14:13:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:30 smithi104 ceph-mon[97044]: pgmap v18694: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:30 smithi110 ceph-mon[97633]: pgmap v18694: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:30.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:30.661338+0000) 2022-09-10T14:13:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:31.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:31.279194+0000) 2022-09-10T14:13:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:31.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:31.661641+0000) 2022-09-10T14:13:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:32.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:32.279410+0000) 2022-09-10T14:13:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:32 smithi104 ceph-mon[97044]: pgmap v18695: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:32 smithi110 ceph-mon[97633]: pgmap v18695: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:32.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:32.661820+0000) 2022-09-10T14:13:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:33.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:33.279658+0000) 2022-09-10T14:13:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:33.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:33.661957+0000) 2022-09-10T14:13:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:13:34.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:13:34.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:34.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:34.279855+0000) 2022-09-10T14:13:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:34 smithi104 ceph-mon[97044]: pgmap v18696: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:34 smithi110 ceph-mon[97633]: pgmap v18696: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:34.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:34.662184+0000) 2022-09-10T14:13:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:35.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:35.280120+0000) 2022-09-10T14:13:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:35.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:35.662396+0000) 2022-09-10T14:13:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:36.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:36.280429+0000) 2022-09-10T14:13:36.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:36 smithi104 ceph-mon[97044]: pgmap v18697: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:36 smithi110 ceph-mon[97633]: pgmap v18697: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:36.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:36.662615+0000) 2022-09-10T14:13:37.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:37.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:37.280661+0000) 2022-09-10T14:13:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:37.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:37.662816+0000) 2022-09-10T14:13:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:38.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:38.280960+0000) 2022-09-10T14:13:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:38 smithi104 ceph-mon[97044]: pgmap v18698: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:38 smithi110 ceph-mon[97633]: pgmap v18698: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:38.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:38.663017+0000) 2022-09-10T14:13:39.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:39.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:39.281223+0000) 2022-09-10T14:13:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:39.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:39.663256+0000) 2022-09-10T14:13:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:40.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:40.281397+0000) 2022-09-10T14:13:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:40 smithi104 ceph-mon[97044]: pgmap v18699: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:40 smithi110 ceph-mon[97633]: pgmap v18699: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:40.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:40.663491+0000) 2022-09-10T14:13:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:41.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:41.281566+0000) 2022-09-10T14:13:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:41.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:41.663743+0000) 2022-09-10T14:13:42.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:42.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:42.281801+0000) 2022-09-10T14:13:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:42 smithi104 ceph-mon[97044]: pgmap v18700: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:42 smithi110 ceph-mon[97633]: pgmap v18700: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:42.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:42.664000+0000) 2022-09-10T14:13:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:43.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:43.282037+0000) 2022-09-10T14:13:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:43.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:43.664244+0000) 2022-09-10T14:13:44.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:44.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:44.282258+0000) 2022-09-10T14:13:44.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:44 smithi104 ceph-mon[97044]: pgmap v18701: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:44 smithi110 ceph-mon[97633]: pgmap v18701: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:44.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:44.664477+0000) 2022-09-10T14:13:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:45.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:45.282505+0000) 2022-09-10T14:13:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:45.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:45.664659+0000) 2022-09-10T14:13:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:46.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:46.282735+0000) 2022-09-10T14:13:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:46 smithi104 ceph-mon[97044]: pgmap v18702: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:46 smithi110 ceph-mon[97633]: pgmap v18702: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:46.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:46.664868+0000) 2022-09-10T14:13:47.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:47.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:47.282980+0000) 2022-09-10T14:13:47.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:13:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:13:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:13:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:13:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:13:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:13:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:13:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:13:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:47.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:47.665030+0000) 2022-09-10T14:13:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:48.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:48.283214+0000) 2022-09-10T14:13:48.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:48 smithi104 ceph-mon[97044]: pgmap v18703: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:48 smithi110 ceph-mon[97633]: pgmap v18703: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:48.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:48.665245+0000) 2022-09-10T14:13:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:49.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:49.283405+0000) 2022-09-10T14:13:49.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:13:49.499+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:13:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:49.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:49.665447+0000) 2022-09-10T14:13:50.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:50.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:50.283579+0000) 2022-09-10T14:13:50.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:50 smithi104 ceph-mon[97044]: pgmap v18704: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:50 smithi110 ceph-mon[97633]: pgmap v18704: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:50.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:50.665686+0000) 2022-09-10T14:13:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:51.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:51.283769+0000) 2022-09-10T14:13:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:51.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:51.665930+0000) 2022-09-10T14:13:52.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:52.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:52.284031+0000) 2022-09-10T14:13:52.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:52 smithi104 ceph-mon[97044]: pgmap v18705: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:52 smithi110 ceph-mon[97633]: pgmap v18705: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:52.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:52.666156+0000) 2022-09-10T14:13:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:53.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:53.284259+0000) 2022-09-10T14:13:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:53.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:53.666367+0000) 2022-09-10T14:13:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:54.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:54.284504+0000) 2022-09-10T14:13:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:54 smithi110 ceph-mon[97633]: pgmap v18706: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:54 smithi104 ceph-mon[97044]: pgmap v18706: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:54.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:54.666592+0000) 2022-09-10T14:13:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:55.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:55.284739+0000) 2022-09-10T14:13:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:55.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:55.666827+0000) 2022-09-10T14:13:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:56.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:56.284996+0000) 2022-09-10T14:13:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:56 smithi110 ceph-mon[97633]: pgmap v18707: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:56 smithi104 ceph-mon[97044]: pgmap v18707: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:13:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:56.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:56.667046+0000) 2022-09-10T14:13:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:57.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:57.285229+0000) 2022-09-10T14:13:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:57.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:57.667269+0000) 2022-09-10T14:13:58.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:58.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:58.285455+0000) 2022-09-10T14:13:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:13:58 smithi110 ceph-mon[97633]: pgmap v18708: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:13:58 smithi104 ceph-mon[97044]: pgmap v18708: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:13:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:58.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:58.667509+0000) 2022-09-10T14:13:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:13:59.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:59.285614+0000) 2022-09-10T14:14:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:13:59.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:13:59.667731+0000) 2022-09-10T14:14:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:00.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:00.285768+0000) 2022-09-10T14:14:00.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:00 smithi110 ceph-mon[97633]: pgmap v18709: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:00 smithi104 ceph-mon[97044]: pgmap v18709: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:00.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:00.667951+0000) 2022-09-10T14:14:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:01.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:01.285988+0000) 2022-09-10T14:14:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:01.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:01.668189+0000) 2022-09-10T14:14:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:02.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:02.286194+0000) 2022-09-10T14:14:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:02 smithi110 ceph-mon[97633]: pgmap v18710: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:02 smithi104 ceph-mon[97044]: pgmap v18710: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:02.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:02.668425+0000) 2022-09-10T14:14:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:03.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:03.286390+0000) 2022-09-10T14:14:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:03.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:03.668687+0000) 2022-09-10T14:14:04.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:14:04.515+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:14:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:04.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:04.286542+0000) 2022-09-10T14:14:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:04 smithi110 ceph-mon[97633]: pgmap v18711: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:04 smithi104 ceph-mon[97044]: pgmap v18711: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:05.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:04.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:04.668943+0000) 2022-09-10T14:14:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:05.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:05.286757+0000) 2022-09-10T14:14:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:05.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:05.669201+0000) 2022-09-10T14:14:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:06.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:06.286983+0000) 2022-09-10T14:14:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:06 smithi110 ceph-mon[97633]: pgmap v18712: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:14:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:06 smithi104 ceph-mon[97044]: pgmap v18712: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:14:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:06.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:06.669378+0000) 2022-09-10T14:14:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:07.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:07.287214+0000) 2022-09-10T14:14:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:07.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:07.669649+0000) 2022-09-10T14:14:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:08.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:08.287434+0000) 2022-09-10T14:14:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:08 smithi110 ceph-mon[97633]: pgmap v18713: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:14:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:08 smithi104 ceph-mon[97044]: pgmap v18713: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:14:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:08.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:08.669877+0000) 2022-09-10T14:14:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:09.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:09.287629+0000) 2022-09-10T14:14:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:09 smithi110 ceph-mon[97633]: osdmap e1250: 12 total, 12 up, 12 in 2022-09-10T14:14:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:09 smithi104 ceph-mon[97044]: osdmap e1250: 12 total, 12 up, 12 in 2022-09-10T14:14:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:09.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:09.670161+0000) 2022-09-10T14:14:10.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:10.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:10.287801+0000) 2022-09-10T14:14:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:10 smithi110 ceph-mon[97633]: pgmap v18715: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:14:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:10 smithi104 ceph-mon[97044]: pgmap v18715: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:14:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:10.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:10.670351+0000) 2022-09-10T14:14:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:11.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:11.288058+0000) 2022-09-10T14:14:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:11.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:11.670607+0000) 2022-09-10T14:14:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:12.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:12.288390+0000) 2022-09-10T14:14:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:12 smithi110 ceph-mon[97633]: pgmap v18716: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:14:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:12 smithi104 ceph-mon[97044]: pgmap v18716: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:14:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:12.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:12.670758+0000) 2022-09-10T14:14:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:13.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:13.288699+0000) 2022-09-10T14:14:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:13.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:13.670930+0000) 2022-09-10T14:14:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:14.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:14.288919+0000) 2022-09-10T14:14:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:14 smithi110 ceph-mon[97633]: pgmap v18717: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:14:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:14 smithi104 ceph-mon[97044]: pgmap v18717: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:14:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:14.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:14.671201+0000) 2022-09-10T14:14:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:15.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:15.289180+0000) 2022-09-10T14:14:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:15.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:15.671404+0000) 2022-09-10T14:14:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:16.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:16.289382+0000) 2022-09-10T14:14:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:16 smithi110 ceph-mon[97633]: pgmap v18718: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:14:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:16 smithi110 ceph-mon[97633]: osdmap e1251: 12 total, 12 up, 12 in 2022-09-10T14:14:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:16 smithi104 ceph-mon[97044]: pgmap v18718: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:14:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:16 smithi104 ceph-mon[97044]: osdmap e1251: 12 total, 12 up, 12 in 2022-09-10T14:14:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:16.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:16.671635+0000) 2022-09-10T14:14:17.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:17.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:17.289599+0000) 2022-09-10T14:14:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:17.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:17.671905+0000) 2022-09-10T14:14:18.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:18.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:18.289850+0000) 2022-09-10T14:14:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:18 smithi110 ceph-mon[97633]: pgmap v18720: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T14:14:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:18 smithi104 ceph-mon[97044]: pgmap v18720: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T14:14:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:18.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:18.672128+0000) 2022-09-10T14:14:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:19.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:19.290052+0000) 2022-09-10T14:14:19.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:14:19.526+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:14:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:19.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:19.672289+0000) 2022-09-10T14:14:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:20.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:20.290313+0000) 2022-09-10T14:14:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:20 smithi110 ceph-mon[97633]: pgmap v18721: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:14:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:20 smithi104 ceph-mon[97044]: pgmap v18721: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:14:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:20.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:20.672517+0000) 2022-09-10T14:14:21.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:21.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:21.290497+0000) 2022-09-10T14:14:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:21.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:21.672720+0000) 2022-09-10T14:14:22.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:22.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:22.290704+0000) 2022-09-10T14:14:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:22 smithi110 ceph-mon[97633]: pgmap v18722: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:14:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:22 smithi104 ceph-mon[97044]: pgmap v18722: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:14:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:22.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:22.672947+0000) 2022-09-10T14:14:23.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:23.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:23.290926+0000) 2022-09-10T14:14:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:23.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:23.673200+0000) 2022-09-10T14:14:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:24.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:24.291202+0000) 2022-09-10T14:14:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:24 smithi110 ceph-mon[97633]: pgmap v18723: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:14:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:24 smithi104 ceph-mon[97044]: pgmap v18723: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:14:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:24.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:24.673463+0000) 2022-09-10T14:14:25.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:25.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:25.291428+0000) 2022-09-10T14:14:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:25.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:25.673656+0000) 2022-09-10T14:14:26.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:26.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:26.291663+0000) 2022-09-10T14:14:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:26 smithi110 ceph-mon[97633]: pgmap v18724: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:14:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:26 smithi104 ceph-mon[97044]: pgmap v18724: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:14:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:26.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:26.673811+0000) 2022-09-10T14:14:27.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:27.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:27.291849+0000) 2022-09-10T14:14:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:27.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:27.673984+0000) 2022-09-10T14:14:28.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:28.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:28.292086+0000) 2022-09-10T14:14:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:28 smithi110 ceph-mon[97633]: pgmap v18725: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:14:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:28 smithi104 ceph-mon[97044]: pgmap v18725: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:14:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:28.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:28.674202+0000) 2022-09-10T14:14:29.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:29.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:29.292310+0000) 2022-09-10T14:14:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:29.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:29.674399+0000) 2022-09-10T14:14:30.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:30.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:30.292468+0000) 2022-09-10T14:14:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:30 smithi110 ceph-mon[97633]: pgmap v18726: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:30 smithi104 ceph-mon[97044]: pgmap v18726: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:30.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:30.674630+0000) 2022-09-10T14:14:31.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:31.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:31.292703+0000) 2022-09-10T14:14:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:31.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:31.674877+0000) 2022-09-10T14:14:32.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:32.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:32.292929+0000) 2022-09-10T14:14:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:32 smithi110 ceph-mon[97633]: pgmap v18727: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:32 smithi104 ceph-mon[97044]: pgmap v18727: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:32.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:32.675131+0000) 2022-09-10T14:14:33.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:33.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:33.293195+0000) 2022-09-10T14:14:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:33.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:33.675389+0000) 2022-09-10T14:14:34.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:14:34.527+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:14:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:34.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:34.293439+0000) 2022-09-10T14:14:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:34 smithi110 ceph-mon[97633]: pgmap v18728: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:34 smithi104 ceph-mon[97044]: pgmap v18728: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:34.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:34.675564+0000) 2022-09-10T14:14:35.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:35.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:35.293598+0000) 2022-09-10T14:14:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:35.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:35.675806+0000) 2022-09-10T14:14:36.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:36.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:36.293819+0000) 2022-09-10T14:14:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:36 smithi110 ceph-mon[97633]: pgmap v18729: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:36 smithi104 ceph-mon[97044]: pgmap v18729: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:36.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:36.676068+0000) 2022-09-10T14:14:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:37.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:37.294021+0000) 2022-09-10T14:14:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:37.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:37.676319+0000) 2022-09-10T14:14:38.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:38.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:38.294241+0000) 2022-09-10T14:14:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:38 smithi110 ceph-mon[97633]: pgmap v18730: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:38 smithi104 ceph-mon[97044]: pgmap v18730: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:38.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:38.676499+0000) 2022-09-10T14:14:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:39.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:39.294492+0000) 2022-09-10T14:14:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:39.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:39.676692+0000) 2022-09-10T14:14:40.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:40.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:40.294734+0000) 2022-09-10T14:14:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:40 smithi110 ceph-mon[97633]: pgmap v18731: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:40 smithi104 ceph-mon[97044]: pgmap v18731: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:40.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:40.676925+0000) 2022-09-10T14:14:41.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:41.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:41.294956+0000) 2022-09-10T14:14:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:41.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:41.677215+0000) 2022-09-10T14:14:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:42.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:42.295217+0000) 2022-09-10T14:14:42.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:42 smithi110 ceph-mon[97633]: pgmap v18732: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:42 smithi104 ceph-mon[97044]: pgmap v18732: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:42.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:42.677453+0000) 2022-09-10T14:14:43.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:43.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:43.295424+0000) 2022-09-10T14:14:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:43.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:43.677712+0000) 2022-09-10T14:14:44.566 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:44.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:44.295677+0000) 2022-09-10T14:14:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:44 smithi110 ceph-mon[97633]: pgmap v18733: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:44 smithi104 ceph-mon[97044]: pgmap v18733: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:44.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:44.678005+0000) 2022-09-10T14:14:45.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:45.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:45.295959+0000) 2022-09-10T14:14:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:45.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:45.678217+0000) 2022-09-10T14:14:46.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:46.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:46.296235+0000) 2022-09-10T14:14:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:46 smithi110 ceph-mon[97633]: pgmap v18734: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:14:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:14:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:14:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:14:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:46 smithi104 ceph-mon[97044]: pgmap v18734: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:14:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:14:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:14:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:14:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:46.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:46.678464+0000) 2022-09-10T14:14:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:47.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:47.296477+0000) 2022-09-10T14:14:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:47.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:47.678636+0000) 2022-09-10T14:14:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:48.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:48.296663+0000) 2022-09-10T14:14:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:48 smithi110 ceph-mon[97633]: pgmap v18735: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:48 smithi104 ceph-mon[97044]: pgmap v18735: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:48.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:48.678875+0000) 2022-09-10T14:14:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:49.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:49.296846+0000) 2022-09-10T14:14:49.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:14:49.539+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:14:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:49.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:49.679191+0000) 2022-09-10T14:14:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:50.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:50.297043+0000) 2022-09-10T14:14:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:50 smithi110 ceph-mon[97633]: pgmap v18736: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:50 smithi104 ceph-mon[97044]: pgmap v18736: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:50.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:50.679451+0000) 2022-09-10T14:14:51.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:51.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:51.297256+0000) 2022-09-10T14:14:52.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:51.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:51.679694+0000) 2022-09-10T14:14:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:52.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:52.297387+0000) 2022-09-10T14:14:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:52 smithi110 ceph-mon[97633]: pgmap v18737: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:53.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:52 smithi104 ceph-mon[97044]: pgmap v18737: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:52.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:52.679902+0000) 2022-09-10T14:14:53.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:53.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:53.297548+0000) 2022-09-10T14:14:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:53.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:53.680130+0000) 2022-09-10T14:14:54.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:54.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:54.297747+0000) 2022-09-10T14:14:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:54 smithi110 ceph-mon[97633]: pgmap v18738: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:55.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:54 smithi104 ceph-mon[97044]: pgmap v18738: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:54.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:54.680387+0000) 2022-09-10T14:14:55.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:55.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:55.297964+0000) 2022-09-10T14:14:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:55.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:55.680605+0000) 2022-09-10T14:14:56.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:56.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:56.298209+0000) 2022-09-10T14:14:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:56 smithi110 ceph-mon[97633]: pgmap v18739: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:56 smithi104 ceph-mon[97044]: pgmap v18739: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:14:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:56.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:56.680861+0000) 2022-09-10T14:14:57.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:57.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:57.298449+0000) 2022-09-10T14:14:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:57.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:57.681150+0000) 2022-09-10T14:14:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:58.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:58.298638+0000) 2022-09-10T14:14:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:14:58 smithi110 ceph-mon[97633]: pgmap v18740: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:14:58 smithi104 ceph-mon[97044]: pgmap v18740: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:14:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:58.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:58.681418+0000) 2022-09-10T14:14:59.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:14:59.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:59.298857+0000) 2022-09-10T14:15:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:14:59.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:14:59.681661+0000) 2022-09-10T14:15:00.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:00.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:00.299012+0000) 2022-09-10T14:15:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:00 smithi104 ceph-mon[97044]: pgmap v18741: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:00.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:00.681812+0000) 2022-09-10T14:15:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:00 smithi110 ceph-mon[97633]: pgmap v18741: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:01.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:01.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:01.299231+0000) 2022-09-10T14:15:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:01.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:01.681998+0000) 2022-09-10T14:15:02.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:02.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:02.299444+0000) 2022-09-10T14:15:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:02 smithi104 ceph-mon[97044]: pgmap v18742: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:02.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:02.682237+0000) 2022-09-10T14:15:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:02 smithi110 ceph-mon[97633]: pgmap v18742: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:03.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:03.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:03.299670+0000) 2022-09-10T14:15:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:03.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:03.682493+0000) 2022-09-10T14:15:04.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:04.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:04.299979+0000) 2022-09-10T14:15:04.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:04.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:04.682744+0000) 2022-09-10T14:15:04.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:15:04.552+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:15:05.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:04 smithi104 ceph-mon[97044]: pgmap v18743: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:04 smithi110 ceph-mon[97633]: pgmap v18743: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:05.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:05.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:05.300240+0000) 2022-09-10T14:15:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:05.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:05.682905+0000) 2022-09-10T14:15:06.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:06.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:06.300508+0000) 2022-09-10T14:15:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:06 smithi104 ceph-mon[97044]: pgmap v18744: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:15:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:06.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:06.683137+0000) 2022-09-10T14:15:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:06 smithi110 ceph-mon[97633]: pgmap v18744: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:15:07.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:07.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:07.300703+0000) 2022-09-10T14:15:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:07.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:07.683369+0000) 2022-09-10T14:15:08.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:08.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:08.300920+0000) 2022-09-10T14:15:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:08 smithi104 ceph-mon[97044]: pgmap v18745: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:15:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:08.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:08.683547+0000) 2022-09-10T14:15:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:08 smithi110 ceph-mon[97633]: pgmap v18745: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:15:09.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:09.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:09.301156+0000) 2022-09-10T14:15:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:09 smithi104 ceph-mon[97044]: osdmap e1252: 12 total, 12 up, 12 in 2022-09-10T14:15:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:09.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:09.683777+0000) 2022-09-10T14:15:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:09 smithi110 ceph-mon[97633]: osdmap e1252: 12 total, 12 up, 12 in 2022-09-10T14:15:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:10.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:10.301364+0000) 2022-09-10T14:15:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:10 smithi104 ceph-mon[97044]: pgmap v18747: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:15:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:10.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:10.683972+0000) 2022-09-10T14:15:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:10 smithi110 ceph-mon[97633]: pgmap v18747: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:15:11.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:11.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:11.301581+0000) 2022-09-10T14:15:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:11.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:11.684254+0000) 2022-09-10T14:15:12.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:12.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:12.301796+0000) 2022-09-10T14:15:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:12 smithi104 ceph-mon[97044]: pgmap v18748: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:15:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:12.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:12.684475+0000) 2022-09-10T14:15:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:12 smithi110 ceph-mon[97633]: pgmap v18748: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:15:13.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:13.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:13.302042+0000) 2022-09-10T14:15:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:13.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:13.684725+0000) 2022-09-10T14:15:14.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:14.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:14.302291+0000) 2022-09-10T14:15:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:14 smithi104 ceph-mon[97044]: pgmap v18749: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:15:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:14.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:14.684980+0000) 2022-09-10T14:15:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:14 smithi110 ceph-mon[97633]: pgmap v18749: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:15:15.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:15.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:15.302486+0000) 2022-09-10T14:15:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:15.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:15.685222+0000) 2022-09-10T14:15:16.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:16.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:16.302729+0000) 2022-09-10T14:15:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:16 smithi104 ceph-mon[97044]: pgmap v18750: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:15:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:16 smithi104 ceph-mon[97044]: osdmap e1253: 12 total, 12 up, 12 in 2022-09-10T14:15:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:16.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:16.685393+0000) 2022-09-10T14:15:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:16 smithi110 ceph-mon[97633]: pgmap v18750: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:15:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:16 smithi110 ceph-mon[97633]: osdmap e1253: 12 total, 12 up, 12 in 2022-09-10T14:15:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:17.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:17.302935+0000) 2022-09-10T14:15:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:17.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:17.685642+0000) 2022-09-10T14:15:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:18.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:18.303193+0000) 2022-09-10T14:15:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:18 smithi104 ceph-mon[97044]: pgmap v18752: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T14:15:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:18.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:18.685887+0000) 2022-09-10T14:15:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:18 smithi110 ceph-mon[97633]: pgmap v18752: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T14:15:19.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:19.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:19.303414+0000) 2022-09-10T14:15:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:15:19.567+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:15:20.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:19.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:19.686066+0000) 2022-09-10T14:15:20.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:20.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:20.303597+0000) 2022-09-10T14:15:21.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:20 smithi104 ceph-mon[97044]: pgmap v18753: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:15:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:20.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:20.686353+0000) 2022-09-10T14:15:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:20 smithi110 ceph-mon[97633]: pgmap v18753: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:15:21.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:21.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:21.303851+0000) 2022-09-10T14:15:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:21.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:21.686613+0000) 2022-09-10T14:15:22.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:22.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:22.304091+0000) 2022-09-10T14:15:23.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:22 smithi104 ceph-mon[97044]: pgmap v18754: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:15:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:22.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:22.686804+0000) 2022-09-10T14:15:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:22 smithi110 ceph-mon[97633]: pgmap v18754: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:15:23.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:23.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:23.304319+0000) 2022-09-10T14:15:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:23.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:23.686985+0000) 2022-09-10T14:15:24.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:24.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:24.304564+0000) 2022-09-10T14:15:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:24 smithi104 ceph-mon[97044]: pgmap v18755: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:15:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:24.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:24.687218+0000) 2022-09-10T14:15:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:24 smithi110 ceph-mon[97633]: pgmap v18755: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:15:25.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:25.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:25.304760+0000) 2022-09-10T14:15:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:25.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:25.687419+0000) 2022-09-10T14:15:26.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:26.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:26.304941+0000) 2022-09-10T14:15:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:26 smithi104 ceph-mon[97044]: pgmap v18756: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:15:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:26.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:26.687607+0000) 2022-09-10T14:15:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:26 smithi110 ceph-mon[97633]: pgmap v18756: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:15:27.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:27.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:27.305154+0000) 2022-09-10T14:15:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:27.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:27.687874+0000) 2022-09-10T14:15:28.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:28.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:28.305386+0000) 2022-09-10T14:15:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:28 smithi104 ceph-mon[97044]: pgmap v18757: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:15:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:28.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:28.688124+0000) 2022-09-10T14:15:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:28 smithi110 ceph-mon[97633]: pgmap v18757: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:15:29.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:29.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:29.305649+0000) 2022-09-10T14:15:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:29.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:29.688355+0000) 2022-09-10T14:15:30.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:30.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:30.305890+0000) 2022-09-10T14:15:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:30 smithi104 ceph-mon[97044]: pgmap v18758: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:30.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:30.688575+0000) 2022-09-10T14:15:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:30 smithi110 ceph-mon[97633]: pgmap v18758: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:31.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:31.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:31.306170+0000) 2022-09-10T14:15:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:31.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:31.688814+0000) 2022-09-10T14:15:32.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:32.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:32.306415+0000) 2022-09-10T14:15:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:32 smithi104 ceph-mon[97044]: pgmap v18759: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:32.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:32.689011+0000) 2022-09-10T14:15:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:32 smithi110 ceph-mon[97633]: pgmap v18759: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:33.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:33.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:33.306640+0000) 2022-09-10T14:15:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:33.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:33.689235+0000) 2022-09-10T14:15:34.581 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:34.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:34.306849+0000) 2022-09-10T14:15:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:34 smithi104 ceph-mon[97044]: pgmap v18760: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:35.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:15:34.577+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:15:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:34.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:34.689500+0000) 2022-09-10T14:15:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:34 smithi110 ceph-mon[97633]: pgmap v18760: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:35.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:35.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:35.307085+0000) 2022-09-10T14:15:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:35.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:35.689718+0000) 2022-09-10T14:15:36.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:36.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:36.307351+0000) 2022-09-10T14:15:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:36 smithi104 ceph-mon[97044]: pgmap v18761: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:36.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:36.689974+0000) 2022-09-10T14:15:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:36 smithi110 ceph-mon[97633]: pgmap v18761: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:37.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:37.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:37.307516+0000) 2022-09-10T14:15:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:37.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:37.690241+0000) 2022-09-10T14:15:38.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:38.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:38.307703+0000) 2022-09-10T14:15:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:38 smithi104 ceph-mon[97044]: pgmap v18762: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:38.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:38.690477+0000) 2022-09-10T14:15:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:38 smithi110 ceph-mon[97633]: pgmap v18762: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:39.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:39.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:39.307921+0000) 2022-09-10T14:15:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:39.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:39.690666+0000) 2022-09-10T14:15:40.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:40.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:40.308143+0000) 2022-09-10T14:15:41.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:40 smithi104 ceph-mon[97044]: pgmap v18763: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:40.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:40.690922+0000) 2022-09-10T14:15:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:40 smithi110 ceph-mon[97633]: pgmap v18763: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:41.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:41.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:41.308410+0000) 2022-09-10T14:15:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:41.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:41.691201+0000) 2022-09-10T14:15:42.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:42.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:42.308557+0000) 2022-09-10T14:15:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:42 smithi104 ceph-mon[97044]: pgmap v18764: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:42.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:42.691410+0000) 2022-09-10T14:15:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:42 smithi110 ceph-mon[97633]: pgmap v18764: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:43.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:43.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:43.308780+0000) 2022-09-10T14:15:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:43.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:43.691577+0000) 2022-09-10T14:15:44.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:44.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:44.309010+0000) 2022-09-10T14:15:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:44 smithi104 ceph-mon[97044]: pgmap v18765: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:44.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:44.691786+0000) 2022-09-10T14:15:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:44 smithi110 ceph-mon[97633]: pgmap v18765: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:45.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:45.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:45.309263+0000) 2022-09-10T14:15:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:45.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:45.691999+0000) 2022-09-10T14:15:46.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:46.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:46.309507+0000) 2022-09-10T14:15:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:46 smithi104 ceph-mon[97044]: pgmap v18766: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:15:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:15:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:15:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:15:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:46.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:46.692251+0000) 2022-09-10T14:15:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:46 smithi110 ceph-mon[97633]: pgmap v18766: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:15:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:15:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:15:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:15:47.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:47.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:47.309738+0000) 2022-09-10T14:15:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:47.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:47.692453+0000) 2022-09-10T14:15:48.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:48.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:48.310008+0000) 2022-09-10T14:15:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:48 smithi104 ceph-mon[97044]: pgmap v18767: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:15:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:48.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:48.692647+0000) 2022-09-10T14:15:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:48 smithi110 ceph-mon[97633]: pgmap v18767: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:15:49.584 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:49.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:49.310261+0000) 2022-09-10T14:15:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:49.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:49.692895+0000) 2022-09-10T14:15:50.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:15:49.583+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:15:50.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:50.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:50.310488+0000) 2022-09-10T14:15:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:50 smithi104 ceph-mon[97044]: pgmap v18768: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:15:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:50.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:50.693214+0000) 2022-09-10T14:15:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:50 smithi110 ceph-mon[97633]: pgmap v18768: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:15:51.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:51.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:51.310651+0000) 2022-09-10T14:15:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:51.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:51.693473+0000) 2022-09-10T14:15:52.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:52.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:52.310872+0000) 2022-09-10T14:15:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:52 smithi104 ceph-mon[97044]: pgmap v18769: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:52.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:52.693703+0000) 2022-09-10T14:15:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:52 smithi110 ceph-mon[97633]: pgmap v18769: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:53.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:53.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:53.311088+0000) 2022-09-10T14:15:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:53.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:53.693981+0000) 2022-09-10T14:15:54.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:54.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:54.311303+0000) 2022-09-10T14:15:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:54 smithi104 ceph-mon[97044]: pgmap v18770: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:15:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:54.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:54.694180+0000) 2022-09-10T14:15:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:54 smithi110 ceph-mon[97633]: pgmap v18770: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:15:55.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:55.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:55.311500+0000) 2022-09-10T14:15:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:55.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:55.694401+0000) 2022-09-10T14:15:56.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:56.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:56.311713+0000) 2022-09-10T14:15:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:56 smithi104 ceph-mon[97044]: pgmap v18771: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:56.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:56.694623+0000) 2022-09-10T14:15:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:56 smithi110 ceph-mon[97633]: pgmap v18771: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:15:57.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:57.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:57.311978+0000) 2022-09-10T14:15:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:57.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:57.694872+0000) 2022-09-10T14:15:58.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:58.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:58.312210+0000) 2022-09-10T14:15:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:15:58 smithi104 ceph-mon[97044]: pgmap v18772: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:58.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:58.695074+0000) 2022-09-10T14:15:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:15:58 smithi110 ceph-mon[97633]: pgmap v18772: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:15:59.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:15:59.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:59.312402+0000) 2022-09-10T14:16:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:15:59.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:15:59.695271+0000) 2022-09-10T14:16:00.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:00.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:00.312543+0000) 2022-09-10T14:16:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:00 smithi104 ceph-mon[97044]: pgmap v18773: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:00.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:00.695482+0000) 2022-09-10T14:16:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:00 smithi110 ceph-mon[97633]: pgmap v18773: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:01.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:01.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:01.312832+0000) 2022-09-10T14:16:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:01.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:01.695658+0000) 2022-09-10T14:16:02.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:02.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:02.312997+0000) 2022-09-10T14:16:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:02 smithi104 ceph-mon[97044]: pgmap v18774: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:02.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:02.695868+0000) 2022-09-10T14:16:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:02 smithi110 ceph-mon[97633]: pgmap v18774: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:03.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:03.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:03.313190+0000) 2022-09-10T14:16:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:03.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:03.696069+0000) 2022-09-10T14:16:04.595 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:04.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:04.313369+0000) 2022-09-10T14:16:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:04 smithi104 ceph-mon[97044]: pgmap v18775: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:05.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:16:04.593+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:16:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:04.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:04.696317+0000) 2022-09-10T14:16:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:04 smithi110 ceph-mon[97633]: pgmap v18775: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:05.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:05.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:05.313510+0000) 2022-09-10T14:16:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:05.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:05.696551+0000) 2022-09-10T14:16:06.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:06.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:06.313662+0000) 2022-09-10T14:16:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:06 smithi104 ceph-mon[97044]: pgmap v18776: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:16:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:06.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:06.696801+0000) 2022-09-10T14:16:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:06 smithi110 ceph-mon[97633]: pgmap v18776: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:16:07.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:07.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:07.313929+0000) 2022-09-10T14:16:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:07.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:07.697064+0000) 2022-09-10T14:16:08.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:08.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:08.314226+0000) 2022-09-10T14:16:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:08 smithi104 ceph-mon[97044]: pgmap v18777: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:16:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:08.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:08.697313+0000) 2022-09-10T14:16:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:08 smithi110 ceph-mon[97633]: pgmap v18777: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:16:09.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:09.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:09.314525+0000) 2022-09-10T14:16:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:09 smithi104 ceph-mon[97044]: osdmap e1254: 12 total, 12 up, 12 in 2022-09-10T14:16:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:09.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:09.697577+0000) 2022-09-10T14:16:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:09 smithi110 ceph-mon[97633]: osdmap e1254: 12 total, 12 up, 12 in 2022-09-10T14:16:10.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:10.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:10.314709+0000) 2022-09-10T14:16:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:10 smithi104 ceph-mon[97044]: pgmap v18779: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:16:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:10.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:10.697820+0000) 2022-09-10T14:16:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:10 smithi110 ceph-mon[97633]: pgmap v18779: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:16:11.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:11.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:11.314942+0000) 2022-09-10T14:16:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:11.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:11.697989+0000) 2022-09-10T14:16:12.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:12.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:12.315225+0000) 2022-09-10T14:16:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:12 smithi104 ceph-mon[97044]: pgmap v18780: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:16:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:12.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:12.698257+0000) 2022-09-10T14:16:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:12 smithi110 ceph-mon[97633]: pgmap v18780: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:16:13.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:13.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:13.315481+0000) 2022-09-10T14:16:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:13.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:13.698482+0000) 2022-09-10T14:16:14.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:14.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:14.315684+0000) 2022-09-10T14:16:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:14 smithi104 ceph-mon[97044]: pgmap v18781: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:16:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:14.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:14.698682+0000) 2022-09-10T14:16:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:14 smithi110 ceph-mon[97633]: pgmap v18781: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:16:15.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:15.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:15.315890+0000) 2022-09-10T14:16:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:15.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:15.698844+0000) 2022-09-10T14:16:16.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:16.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:16.316146+0000) 2022-09-10T14:16:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:16 smithi104 ceph-mon[97044]: pgmap v18782: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:16:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:16 smithi104 ceph-mon[97044]: osdmap e1255: 12 total, 12 up, 12 in 2022-09-10T14:16:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:16.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:16.699087+0000) 2022-09-10T14:16:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:16 smithi110 ceph-mon[97633]: pgmap v18782: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:16:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:16 smithi110 ceph-mon[97633]: osdmap e1255: 12 total, 12 up, 12 in 2022-09-10T14:16:17.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:17.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:17.316356+0000) 2022-09-10T14:16:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:17.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:17.699363+0000) 2022-09-10T14:16:18.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:18.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:18.316560+0000) 2022-09-10T14:16:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:18 smithi104 ceph-mon[97044]: pgmap v18784: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T14:16:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:18.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:18.699518+0000) 2022-09-10T14:16:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:18 smithi110 ceph-mon[97633]: pgmap v18784: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 247 B/s wr, 12 op/s 2022-09-10T14:16:19.603 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:19.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:19.316707+0000) 2022-09-10T14:16:20.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:19.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:19.699764+0000) 2022-09-10T14:16:20.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:16:19.601+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:16:20.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:20.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:20.316917+0000) 2022-09-10T14:16:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:20 smithi104 ceph-mon[97044]: pgmap v18785: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:16:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:20.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:20.700034+0000) 2022-09-10T14:16:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:20 smithi110 ceph-mon[97633]: pgmap v18785: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:16:21.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:21.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:21.317138+0000) 2022-09-10T14:16:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:21.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:21.700300+0000) 2022-09-10T14:16:22.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:22.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:22.317346+0000) 2022-09-10T14:16:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:22.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:22.700478+0000) 2022-09-10T14:16:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:22 smithi110 ceph-mon[97633]: pgmap v18786: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:16:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:22 smithi104 ceph-mon[97044]: pgmap v18786: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:16:23.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:23.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:23.317588+0000) 2022-09-10T14:16:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:23.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:23.700625+0000) 2022-09-10T14:16:24.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:24.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:24.317848+0000) 2022-09-10T14:16:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:24.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:24.700876+0000) 2022-09-10T14:16:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:24 smithi110 ceph-mon[97633]: pgmap v18787: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:16:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:24 smithi104 ceph-mon[97044]: pgmap v18787: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:16:25.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:25.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:25.318028+0000) 2022-09-10T14:16:26.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:25.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:25.701150+0000) 2022-09-10T14:16:26.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:26.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:26.318274+0000) 2022-09-10T14:16:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:26.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:26.701352+0000) 2022-09-10T14:16:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:26 smithi110 ceph-mon[97633]: pgmap v18788: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T14:16:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:26 smithi104 ceph-mon[97044]: pgmap v18788: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T14:16:27.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:27.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:27.318441+0000) 2022-09-10T14:16:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:27.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:27.701530+0000) 2022-09-10T14:16:28.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:28.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:28.318636+0000) 2022-09-10T14:16:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:28.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:28.701764+0000) 2022-09-10T14:16:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:28 smithi110 ceph-mon[97633]: pgmap v18789: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T14:16:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:28 smithi104 ceph-mon[97044]: pgmap v18789: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T14:16:29.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:29.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:29.318898+0000) 2022-09-10T14:16:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:29.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:29.701915+0000) 2022-09-10T14:16:30.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:30.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:30.319150+0000) 2022-09-10T14:16:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:30.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:30.702052+0000) 2022-09-10T14:16:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:30 smithi110 ceph-mon[97633]: pgmap v18790: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:16:31.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:30 smithi104 ceph-mon[97044]: pgmap v18790: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:16:31.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:31.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:31.319360+0000) 2022-09-10T14:16:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:31.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:31.702213+0000) 2022-09-10T14:16:32.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:32.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:32.319595+0000) 2022-09-10T14:16:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:32.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:32.702374+0000) 2022-09-10T14:16:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:32 smithi110 ceph-mon[97633]: pgmap v18791: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:32 smithi104 ceph-mon[97044]: pgmap v18791: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:33.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:33.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:33.319844+0000) 2022-09-10T14:16:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:33.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:33.702635+0000) 2022-09-10T14:16:34.616 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:34.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:34.319998+0000) 2022-09-10T14:16:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:16:34.614+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:16:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:34.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:34.702865+0000) 2022-09-10T14:16:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:34 smithi110 ceph-mon[97633]: pgmap v18792: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:16:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:34 smithi104 ceph-mon[97044]: pgmap v18792: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:16:35.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:35.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:35.320229+0000) 2022-09-10T14:16:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:35.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:35.703074+0000) 2022-09-10T14:16:36.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:36.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:36.320425+0000) 2022-09-10T14:16:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:36.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:36.703354+0000) 2022-09-10T14:16:37.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:36 smithi110 ceph-mon[97633]: pgmap v18793: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:36 smithi104 ceph-mon[97044]: pgmap v18793: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:37.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:37.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:37.320679+0000) 2022-09-10T14:16:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:37.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:37.703580+0000) 2022-09-10T14:16:38.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:38.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:38.320857+0000) 2022-09-10T14:16:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:38.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:38.703810+0000) 2022-09-10T14:16:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:38 smithi110 ceph-mon[97633]: pgmap v18794: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:38 smithi104 ceph-mon[97044]: pgmap v18794: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:39.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:39.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:39.321065+0000) 2022-09-10T14:16:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:39.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:39.704087+0000) 2022-09-10T14:16:40.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:40.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:40.321246+0000) 2022-09-10T14:16:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:40.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:40.704274+0000) 2022-09-10T14:16:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:40 smithi110 ceph-mon[97633]: pgmap v18795: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:40 smithi104 ceph-mon[97044]: pgmap v18795: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:41.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:41.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:41.321462+0000) 2022-09-10T14:16:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:41.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:41.704422+0000) 2022-09-10T14:16:42.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:42.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:42.321703+0000) 2022-09-10T14:16:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:42.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:42.704552+0000) 2022-09-10T14:16:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:42 smithi110 ceph-mon[97633]: pgmap v18796: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:42 smithi104 ceph-mon[97044]: pgmap v18796: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:43.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:43.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:43.321925+0000) 2022-09-10T14:16:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:43.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:43.704760+0000) 2022-09-10T14:16:44.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:44.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:44.322199+0000) 2022-09-10T14:16:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:44.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:44.705004+0000) 2022-09-10T14:16:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:44 smithi110 ceph-mon[97633]: pgmap v18797: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:44 smithi104 ceph-mon[97044]: pgmap v18797: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:45.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:45.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:45.322438+0000) 2022-09-10T14:16:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:45.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:45.705225+0000) 2022-09-10T14:16:46.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:46.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:46.322682+0000) 2022-09-10T14:16:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:46.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:46.705475+0000) 2022-09-10T14:16:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:46 smithi110 ceph-mon[97633]: pgmap v18798: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:16:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:16:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:16:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:16:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:46 smithi104 ceph-mon[97044]: pgmap v18798: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:16:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:16:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:16:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:16:47.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:47.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:47.322882+0000) 2022-09-10T14:16:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:47.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:47.705725+0000) 2022-09-10T14:16:48.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:48.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:48.323178+0000) 2022-09-10T14:16:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:48.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:48.705889+0000) 2022-09-10T14:16:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:48 smithi110 ceph-mon[97633]: pgmap v18799: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:48 smithi104 ceph-mon[97044]: pgmap v18799: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:49.631 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:49.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:49.323367+0000) 2022-09-10T14:16:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:49.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:49.706192+0000) 2022-09-10T14:16:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:16:49.629+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:16:50.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:50.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:50.323509+0000) 2022-09-10T14:16:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:50.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:50.706403+0000) 2022-09-10T14:16:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:50 smithi110 ceph-mon[97633]: pgmap v18800: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:50 smithi104 ceph-mon[97044]: pgmap v18800: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:51.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:51.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:51.323672+0000) 2022-09-10T14:16:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:51.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:51.706582+0000) 2022-09-10T14:16:52.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:52.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:52.323827+0000) 2022-09-10T14:16:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:52.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:52.706814+0000) 2022-09-10T14:16:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:52 smithi110 ceph-mon[97633]: pgmap v18801: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:52 smithi104 ceph-mon[97044]: pgmap v18801: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:53.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:53.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:53.324071+0000) 2022-09-10T14:16:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:53.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:53.706982+0000) 2022-09-10T14:16:54.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:54.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:54.324328+0000) 2022-09-10T14:16:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:54.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:54.707184+0000) 2022-09-10T14:16:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:54 smithi110 ceph-mon[97633]: pgmap v18802: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:54 smithi104 ceph-mon[97044]: pgmap v18802: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:55.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:55.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:55.324557+0000) 2022-09-10T14:16:56.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:55.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:55.707365+0000) 2022-09-10T14:16:56.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:55 smithi110 ceph-mon[97633]: pgmap v18803: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:56.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:55 smithi104 ceph-mon[97044]: pgmap v18803: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:16:56.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:56.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:56.324807+0000) 2022-09-10T14:16:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:56.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:56.707602+0000) 2022-09-10T14:16:57.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:57.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:57.325072+0000) 2022-09-10T14:16:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:57.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:57.707768+0000) 2022-09-10T14:16:58.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:58.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:58.325330+0000) 2022-09-10T14:16:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:16:58 smithi104 ceph-mon[97044]: pgmap v18804: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:16:58 smithi110 ceph-mon[97633]: pgmap v18804: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:16:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:58.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:58.707990+0000) 2022-09-10T14:16:59.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:16:59.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:59.325515+0000) 2022-09-10T14:17:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:16:59.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:16:59.708233+0000) 2022-09-10T14:17:00.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:00.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:00.325714+0000) 2022-09-10T14:17:00.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:00 smithi104 ceph-mon[97044]: pgmap v18805: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:00 smithi110 ceph-mon[97633]: pgmap v18805: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:00.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:00.708377+0000) 2022-09-10T14:17:01.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:01.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:01.325881+0000) 2022-09-10T14:17:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:01.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:01.708597+0000) 2022-09-10T14:17:02.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:02.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:02.326127+0000) 2022-09-10T14:17:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:02 smithi104 ceph-mon[97044]: pgmap v18806: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:02 smithi110 ceph-mon[97633]: pgmap v18806: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:02.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:02.708857+0000) 2022-09-10T14:17:03.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:03.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:03.326376+0000) 2022-09-10T14:17:04.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:03.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:03.709133+0000) 2022-09-10T14:17:04.647 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:04.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:04.326613+0000) 2022-09-10T14:17:04.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:04 smithi104 ceph-mon[97044]: pgmap v18807: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:04 smithi110 ceph-mon[97633]: pgmap v18807: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:04.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:04.709407+0000) 2022-09-10T14:17:05.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:17:04.645+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:17:05.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:05.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:05.326884+0000) 2022-09-10T14:17:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:05.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:05.709636+0000) 2022-09-10T14:17:06.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:06.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:06.327087+0000) 2022-09-10T14:17:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:06 smithi104 ceph-mon[97044]: pgmap v18808: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:17:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:06 smithi110 ceph-mon[97633]: pgmap v18808: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:17:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:06.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:06.709884+0000) 2022-09-10T14:17:07.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:07.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:07.327283+0000) 2022-09-10T14:17:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:07.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:07.710156+0000) 2022-09-10T14:17:08.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:08.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:08.327538+0000) 2022-09-10T14:17:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:08 smithi104 ceph-mon[97044]: pgmap v18809: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:17:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:08 smithi110 ceph-mon[97633]: pgmap v18809: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:17:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:08.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:08.710323+0000) 2022-09-10T14:17:09.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:09.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:09.327770+0000) 2022-09-10T14:17:09.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:09 smithi104 ceph-mon[97044]: osdmap e1256: 12 total, 12 up, 12 in 2022-09-10T14:17:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:09 smithi110 ceph-mon[97633]: osdmap e1256: 12 total, 12 up, 12 in 2022-09-10T14:17:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:09.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:09.710596+0000) 2022-09-10T14:17:10.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:10.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:10.327964+0000) 2022-09-10T14:17:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:10 smithi104 ceph-mon[97044]: pgmap v18811: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:17:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:10 smithi110 ceph-mon[97633]: pgmap v18811: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:17:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:10.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:10.710806+0000) 2022-09-10T14:17:11.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:11.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:11.328195+0000) 2022-09-10T14:17:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:11.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:11.711051+0000) 2022-09-10T14:17:12.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:12.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:12.328448+0000) 2022-09-10T14:17:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:12 smithi104 ceph-mon[97044]: pgmap v18812: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:17:12.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:12 smithi110 ceph-mon[97633]: pgmap v18812: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:17:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:12.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:12.711283+0000) 2022-09-10T14:17:13.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:13.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:13.328630+0000) 2022-09-10T14:17:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:13.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:13.711504+0000) 2022-09-10T14:17:14.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:14.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:14.328890+0000) 2022-09-10T14:17:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:14 smithi104 ceph-mon[97044]: pgmap v18813: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:17:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:14 smithi110 ceph-mon[97633]: pgmap v18813: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:17:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:14.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:14.711699+0000) 2022-09-10T14:17:15.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:15.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:15.329160+0000) 2022-09-10T14:17:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:15.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:15.711926+0000) 2022-09-10T14:17:16.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:16.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:16.329335+0000) 2022-09-10T14:17:16.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:16 smithi104 ceph-mon[97044]: pgmap v18814: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:17:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:16 smithi104 ceph-mon[97044]: osdmap e1257: 12 total, 12 up, 12 in 2022-09-10T14:17:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:16 smithi110 ceph-mon[97633]: pgmap v18814: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:17:16.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:16 smithi110 ceph-mon[97633]: osdmap e1257: 12 total, 12 up, 12 in 2022-09-10T14:17:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:16.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:16.712171+0000) 2022-09-10T14:17:17.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:17.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:17.329567+0000) 2022-09-10T14:17:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:17.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:17.712329+0000) 2022-09-10T14:17:18.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:18.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:18.329765+0000) 2022-09-10T14:17:18.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:18 smithi104 ceph-mon[97044]: pgmap v18816: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:17:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:18 smithi110 ceph-mon[97633]: pgmap v18816: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:17:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:18.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:18.712456+0000) 2022-09-10T14:17:19.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:19.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:19.329910+0000) 2022-09-10T14:17:20.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:17:19.660+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:17:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:19.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:19.712695+0000) 2022-09-10T14:17:20.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:20.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:20.330131+0000) 2022-09-10T14:17:20.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:20 smithi104 ceph-mon[97044]: pgmap v18817: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:17:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:20 smithi110 ceph-mon[97633]: pgmap v18817: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:17:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:20.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:20.712860+0000) 2022-09-10T14:17:21.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:21.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:21.330353+0000) 2022-09-10T14:17:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:21.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:21.713026+0000) 2022-09-10T14:17:22.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:22.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:22.330539+0000) 2022-09-10T14:17:22.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:22 smithi104 ceph-mon[97044]: pgmap v18818: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:17:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:22 smithi110 ceph-mon[97633]: pgmap v18818: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:17:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:22.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:22.713199+0000) 2022-09-10T14:17:23.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:23.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:23.330815+0000) 2022-09-10T14:17:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:23.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:23.713400+0000) 2022-09-10T14:17:24.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:24.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:24.331036+0000) 2022-09-10T14:17:24.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:24 smithi104 ceph-mon[97044]: pgmap v18819: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:17:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:24 smithi110 ceph-mon[97633]: pgmap v18819: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:17:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:24.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:24.713628+0000) 2022-09-10T14:17:25.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:25.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:25.331259+0000) 2022-09-10T14:17:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:25.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:25.713845+0000) 2022-09-10T14:17:26.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:26.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:26.331451+0000) 2022-09-10T14:17:26.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:26 smithi104 ceph-mon[97044]: pgmap v18820: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:17:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:26 smithi110 ceph-mon[97633]: pgmap v18820: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:17:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:26.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:26.714079+0000) 2022-09-10T14:17:27.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:27.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:27.331612+0000) 2022-09-10T14:17:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:27.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:27.714376+0000) 2022-09-10T14:17:28.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:28.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:28.331812+0000) 2022-09-10T14:17:28.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:28 smithi104 ceph-mon[97044]: pgmap v18821: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:17:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:28 smithi110 ceph-mon[97633]: pgmap v18821: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:17:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:28.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:28.714627+0000) 2022-09-10T14:17:29.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:29.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:29.332084+0000) 2022-09-10T14:17:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:29.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:29.714924+0000) 2022-09-10T14:17:30.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:30.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:30.332338+0000) 2022-09-10T14:17:30.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:30 smithi104 ceph-mon[97044]: pgmap v18822: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:30 smithi110 ceph-mon[97633]: pgmap v18822: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:30.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:30.715142+0000) 2022-09-10T14:17:31.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:31.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:31.332557+0000) 2022-09-10T14:17:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:31.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:31.715312+0000) 2022-09-10T14:17:32.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:32.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:32.332790+0000) 2022-09-10T14:17:32.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:32 smithi104 ceph-mon[97044]: pgmap v18823: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:32 smithi110 ceph-mon[97633]: pgmap v18823: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:32.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:32.715483+0000) 2022-09-10T14:17:33.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:33.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:33.333035+0000) 2022-09-10T14:17:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:33.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:33.715733+0000) 2022-09-10T14:17:34.666 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:34.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:34.333235+0000) 2022-09-10T14:17:34.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:34 smithi104 ceph-mon[97044]: pgmap v18824: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:34 smithi110 ceph-mon[97633]: pgmap v18824: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:34.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:34.716021+0000) 2022-09-10T14:17:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:17:34.664+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:17:35.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:35.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:35.333501+0000) 2022-09-10T14:17:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:35.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:35.716179+0000) 2022-09-10T14:17:36.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:36.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:36.333774+0000) 2022-09-10T14:17:36.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:36 smithi104 ceph-mon[97044]: pgmap v18825: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:36 smithi110 ceph-mon[97633]: pgmap v18825: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:36.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:36.716315+0000) 2022-09-10T14:17:37.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:37.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:37.334075+0000) 2022-09-10T14:17:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:37.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:37.716463+0000) 2022-09-10T14:17:38.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:38.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:38.334349+0000) 2022-09-10T14:17:38.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:38 smithi104 ceph-mon[97044]: pgmap v18826: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:38 smithi110 ceph-mon[97633]: pgmap v18826: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:38.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:38.716652+0000) 2022-09-10T14:17:39.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:39.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:39.334617+0000) 2022-09-10T14:17:40.063 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:39.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:39.716836+0000) 2022-09-10T14:17:40.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:40.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:40.334891+0000) 2022-09-10T14:17:40.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:40 smithi104 ceph-mon[97044]: pgmap v18827: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:40 smithi110 ceph-mon[97633]: pgmap v18827: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:40.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:40.717074+0000) 2022-09-10T14:17:41.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:41.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:41.335168+0000) 2022-09-10T14:17:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:41.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:41.717331+0000) 2022-09-10T14:17:42.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:42.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:42.335402+0000) 2022-09-10T14:17:42.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:42 smithi104 ceph-mon[97044]: pgmap v18828: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:42 smithi110 ceph-mon[97633]: pgmap v18828: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:42.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:42.717504+0000) 2022-09-10T14:17:43.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:43.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:43.335562+0000) 2022-09-10T14:17:44.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:43.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:43.717638+0000) 2022-09-10T14:17:44.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:44.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:44.335792+0000) 2022-09-10T14:17:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:44 smithi104 ceph-mon[97044]: pgmap v18829: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:44 smithi110 ceph-mon[97633]: pgmap v18829: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:44.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:44.717831+0000) 2022-09-10T14:17:45.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:45.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:45.336029+0000) 2022-09-10T14:17:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:45.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:45.717988+0000) 2022-09-10T14:17:46.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:46.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:46.336254+0000) 2022-09-10T14:17:46.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:46 smithi104 ceph-mon[97044]: pgmap v18830: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:46 smithi110 ceph-mon[97633]: pgmap v18830: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:47.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:46.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:46.718169+0000) 2022-09-10T14:17:47.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:47.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:47.336494+0000) 2022-09-10T14:17:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:17:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:17:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:17:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:17:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:17:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:17:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:17:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:17:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:47.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:47.718307+0000) 2022-09-10T14:17:48.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:48.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:48.336718+0000) 2022-09-10T14:17:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:48 smithi110 ceph-mon[97633]: pgmap v18831: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:48 smithi104 ceph-mon[97044]: pgmap v18831: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:48.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:48.718489+0000) 2022-09-10T14:17:49.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:49.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:49.336938+0000) 2022-09-10T14:17:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:49.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:49.718653+0000) 2022-09-10T14:17:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:17:49.679+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:17:50.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:50.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:50.337200+0000) 2022-09-10T14:17:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:50 smithi110 ceph-mon[97633]: pgmap v18832: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:50 smithi104 ceph-mon[97044]: pgmap v18832: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:50.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:50.718806+0000) 2022-09-10T14:17:51.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:51.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:51.337369+0000) 2022-09-10T14:17:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:51.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:51.718942+0000) 2022-09-10T14:17:52.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:52.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:52.337521+0000) 2022-09-10T14:17:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:52 smithi110 ceph-mon[97633]: pgmap v18833: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:52 smithi104 ceph-mon[97044]: pgmap v18833: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:52.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:52.719127+0000) 2022-09-10T14:17:53.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:53.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:53.337703+0000) 2022-09-10T14:17:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:53.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:53.719355+0000) 2022-09-10T14:17:54.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:54.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:54.337943+0000) 2022-09-10T14:17:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:54 smithi110 ceph-mon[97633]: pgmap v18834: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:54 smithi104 ceph-mon[97044]: pgmap v18834: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:54.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:54.719598+0000) 2022-09-10T14:17:55.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:55.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:55.338187+0000) 2022-09-10T14:17:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:55.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:55.719800+0000) 2022-09-10T14:17:56.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:56.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:56.338402+0000) 2022-09-10T14:17:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:56 smithi110 ceph-mon[97633]: pgmap v18835: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:56 smithi104 ceph-mon[97044]: pgmap v18835: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:17:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:56.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:56.720029+0000) 2022-09-10T14:17:57.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:57.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:57.338599+0000) 2022-09-10T14:17:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:57.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:57.720285+0000) 2022-09-10T14:17:58.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:58.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:58.338800+0000) 2022-09-10T14:17:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:17:58 smithi110 ceph-mon[97633]: pgmap v18836: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:17:58 smithi104 ceph-mon[97044]: pgmap v18836: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:17:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:58.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:58.720498+0000) 2022-09-10T14:17:59.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:17:59.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:59.338990+0000) 2022-09-10T14:18:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:17:59.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:17:59.720695+0000) 2022-09-10T14:18:00.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:00.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:00.339229+0000) 2022-09-10T14:18:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:00 smithi110 ceph-mon[97633]: pgmap v18837: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:00 smithi104 ceph-mon[97044]: pgmap v18837: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:00.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:00.720865+0000) 2022-09-10T14:18:01.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:01.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:01.339411+0000) 2022-09-10T14:18:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:01.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:01.721051+0000) 2022-09-10T14:18:02.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:02.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:02.339647+0000) 2022-09-10T14:18:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:02 smithi110 ceph-mon[97633]: pgmap v18838: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:02 smithi104 ceph-mon[97044]: pgmap v18838: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:02.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:02.721272+0000) 2022-09-10T14:18:03.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:03.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:03.339913+0000) 2022-09-10T14:18:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:03.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:03.721467+0000) 2022-09-10T14:18:04.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:04.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:04.340073+0000) 2022-09-10T14:18:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:04 smithi110 ceph-mon[97633]: pgmap v18839: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:05.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:04 smithi104 ceph-mon[97044]: pgmap v18839: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:04.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:04.721683+0000) 2022-09-10T14:18:05.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:18:04.685+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:18:05.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:05.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:05.340333+0000) 2022-09-10T14:18:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:05.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:05.721850+0000) 2022-09-10T14:18:06.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:06.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:06.340568+0000) 2022-09-10T14:18:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:06 smithi110 ceph-mon[97633]: pgmap v18840: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:18:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:06 smithi104 ceph-mon[97044]: pgmap v18840: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:18:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:06.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:06.722131+0000) 2022-09-10T14:18:07.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:07.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:07.340730+0000) 2022-09-10T14:18:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:07.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:07.722380+0000) 2022-09-10T14:18:08.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:08.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:08.340988+0000) 2022-09-10T14:18:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:08 smithi110 ceph-mon[97633]: pgmap v18841: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:18:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:08 smithi104 ceph-mon[97044]: pgmap v18841: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:18:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:08.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:08.722624+0000) 2022-09-10T14:18:09.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:09.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:09.341222+0000) 2022-09-10T14:18:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:09 smithi110 ceph-mon[97633]: osdmap e1258: 12 total, 12 up, 12 in 2022-09-10T14:18:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:09 smithi104 ceph-mon[97044]: osdmap e1258: 12 total, 12 up, 12 in 2022-09-10T14:18:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:09.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:09.722875+0000) 2022-09-10T14:18:10.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:10.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:10.341484+0000) 2022-09-10T14:18:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:10 smithi110 ceph-mon[97633]: pgmap v18843: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:18:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:10 smithi104 ceph-mon[97044]: pgmap v18843: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:18:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:10.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:10.723181+0000) 2022-09-10T14:18:11.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:11.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:11.341772+0000) 2022-09-10T14:18:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:11.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:11.723351+0000) 2022-09-10T14:18:12.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:12.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:12.342040+0000) 2022-09-10T14:18:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:12 smithi110 ceph-mon[97633]: pgmap v18844: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:18:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:12 smithi104 ceph-mon[97044]: pgmap v18844: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:18:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:12.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:12.723519+0000) 2022-09-10T14:18:13.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:13.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:13.342203+0000) 2022-09-10T14:18:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:13.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:13.723682+0000) 2022-09-10T14:18:14.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:14.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:14.342425+0000) 2022-09-10T14:18:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:14 smithi110 ceph-mon[97633]: pgmap v18845: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:18:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:14 smithi104 ceph-mon[97044]: pgmap v18845: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:18:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:14.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:14.723897+0000) 2022-09-10T14:18:15.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:15.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:15.342654+0000) 2022-09-10T14:18:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:15.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:15.724122+0000) 2022-09-10T14:18:16.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:16.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:16.342911+0000) 2022-09-10T14:18:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:16 smithi110 ceph-mon[97633]: pgmap v18846: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:18:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:16 smithi110 ceph-mon[97633]: osdmap e1259: 12 total, 12 up, 12 in 2022-09-10T14:18:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:16 smithi104 ceph-mon[97044]: pgmap v18846: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:18:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:16 smithi104 ceph-mon[97044]: osdmap e1259: 12 total, 12 up, 12 in 2022-09-10T14:18:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:16.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:16.724306+0000) 2022-09-10T14:18:17.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:17.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:17.343183+0000) 2022-09-10T14:18:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:17.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:17.724524+0000) 2022-09-10T14:18:18.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:18.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:18.343449+0000) 2022-09-10T14:18:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:18 smithi110 ceph-mon[97633]: pgmap v18848: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T14:18:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:18 smithi104 ceph-mon[97044]: pgmap v18848: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T14:18:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:18.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:18.724714+0000) 2022-09-10T14:18:19.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:19.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:19.343676+0000) 2022-09-10T14:18:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:19.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:19.724898+0000) 2022-09-10T14:18:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:18:19.701+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:18:20.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:20.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:20.343860+0000) 2022-09-10T14:18:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:20 smithi110 ceph-mon[97633]: pgmap v18849: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:18:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:20 smithi104 ceph-mon[97044]: pgmap v18849: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:18:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:20.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:20.725111+0000) 2022-09-10T14:18:21.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:21.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:21.344134+0000) 2022-09-10T14:18:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:21.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:21.725307+0000) 2022-09-10T14:18:22.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:22.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:22.344354+0000) 2022-09-10T14:18:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:22 smithi110 ceph-mon[97633]: pgmap v18850: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:18:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:22 smithi104 ceph-mon[97044]: pgmap v18850: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:18:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:22.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:22.725471+0000) 2022-09-10T14:18:23.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:23.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:23.344537+0000) 2022-09-10T14:18:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:23.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:23.725697+0000) 2022-09-10T14:18:24.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:24.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:24.344760+0000) 2022-09-10T14:18:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:24 smithi110 ceph-mon[97633]: pgmap v18851: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:18:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:24 smithi104 ceph-mon[97044]: pgmap v18851: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:18:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:24.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:24.725939+0000) 2022-09-10T14:18:25.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:25.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:25.345009+0000) 2022-09-10T14:18:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:25.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:25.726164+0000) 2022-09-10T14:18:26.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:26.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:26.345205+0000) 2022-09-10T14:18:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:26 smithi110 ceph-mon[97633]: pgmap v18852: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:18:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:26 smithi104 ceph-mon[97044]: pgmap v18852: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:18:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:26.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:26.726320+0000) 2022-09-10T14:18:27.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:27.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:27.345475+0000) 2022-09-10T14:18:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:27.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:27.726491+0000) 2022-09-10T14:18:28.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:28.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:28.345696+0000) 2022-09-10T14:18:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:28 smithi110 ceph-mon[97633]: pgmap v18853: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:18:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:28 smithi104 ceph-mon[97044]: pgmap v18853: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:18:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:28.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:28.726727+0000) 2022-09-10T14:18:29.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:29.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:29.345857+0000) 2022-09-10T14:18:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:29.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:29.726973+0000) 2022-09-10T14:18:30.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:30.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:30.346042+0000) 2022-09-10T14:18:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:30 smithi110 ceph-mon[97633]: pgmap v18854: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:30 smithi104 ceph-mon[97044]: pgmap v18854: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:30.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:30.727232+0000) 2022-09-10T14:18:31.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:31.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:31.346271+0000) 2022-09-10T14:18:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:31.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:31.727444+0000) 2022-09-10T14:18:32.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:32.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:32.346509+0000) 2022-09-10T14:18:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:32 smithi110 ceph-mon[97633]: pgmap v18855: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:32 smithi104 ceph-mon[97044]: pgmap v18855: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:32.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:32.727636+0000) 2022-09-10T14:18:33.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:33.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:33.346715+0000) 2022-09-10T14:18:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:33.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:33.727849+0000) 2022-09-10T14:18:34.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:34.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:34.346967+0000) 2022-09-10T14:18:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:34 smithi110 ceph-mon[97633]: pgmap v18856: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:34 smithi104 ceph-mon[97044]: pgmap v18856: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:34.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:34.728016+0000) 2022-09-10T14:18:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:18:34.711+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:18:35.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:35.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:35.347217+0000) 2022-09-10T14:18:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:35.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:35.728272+0000) 2022-09-10T14:18:36.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:36.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:36.347397+0000) 2022-09-10T14:18:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:36 smithi110 ceph-mon[97633]: pgmap v18857: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:36 smithi104 ceph-mon[97044]: pgmap v18857: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:36.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:36.728465+0000) 2022-09-10T14:18:37.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:37.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:37.347619+0000) 2022-09-10T14:18:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:37.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:37.728631+0000) 2022-09-10T14:18:38.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:38.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:38.347863+0000) 2022-09-10T14:18:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:38 smithi110 ceph-mon[97633]: pgmap v18858: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:38 smithi104 ceph-mon[97044]: pgmap v18858: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:38.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:38.728885+0000) 2022-09-10T14:18:39.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:39.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:39.348021+0000) 2022-09-10T14:18:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:39.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:39.729144+0000) 2022-09-10T14:18:40.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:40.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:40.348282+0000) 2022-09-10T14:18:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:40 smithi110 ceph-mon[97633]: pgmap v18859: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:40 smithi104 ceph-mon[97044]: pgmap v18859: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:40.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:40.729361+0000) 2022-09-10T14:18:41.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:41.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:41.348521+0000) 2022-09-10T14:18:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:41.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:41.729574+0000) 2022-09-10T14:18:42.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:42.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:42.348767+0000) 2022-09-10T14:18:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:42 smithi110 ceph-mon[97633]: pgmap v18860: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:42 smithi104 ceph-mon[97044]: pgmap v18860: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:42.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:42.729813+0000) 2022-09-10T14:18:43.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:43.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:43.349029+0000) 2022-09-10T14:18:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:43.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:43.730004+0000) 2022-09-10T14:18:44.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:44.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:44.349267+0000) 2022-09-10T14:18:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:44 smithi110 ceph-mon[97633]: pgmap v18861: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:44 smithi104 ceph-mon[97044]: pgmap v18861: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:44.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:44.730237+0000) 2022-09-10T14:18:45.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:45.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:45.349525+0000) 2022-09-10T14:18:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:45.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:45.730492+0000) 2022-09-10T14:18:46.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:46.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:46.349753+0000) 2022-09-10T14:18:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:46 smithi110 ceph-mon[97633]: pgmap v18862: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:18:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:18:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:18:46.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:18:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:46 smithi104 ceph-mon[97044]: pgmap v18862: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:18:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:18:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:18:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:18:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:46.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:46.730747+0000) 2022-09-10T14:18:47.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:47.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:47.349944+0000) 2022-09-10T14:18:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:47.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:47.731034+0000) 2022-09-10T14:18:48.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:48.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:48.350157+0000) 2022-09-10T14:18:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:48 smithi110 ceph-mon[97633]: pgmap v18863: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:48 smithi104 ceph-mon[97044]: pgmap v18863: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:48.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:48.731261+0000) 2022-09-10T14:18:49.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:49.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:49.350337+0000) 2022-09-10T14:18:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:49.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:49.731449+0000) 2022-09-10T14:18:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:18:49.727+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:18:50.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:50.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:50.350510+0000) 2022-09-10T14:18:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:50 smithi110 ceph-mon[97633]: pgmap v18864: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:50 smithi104 ceph-mon[97044]: pgmap v18864: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:50.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:50.731749+0000) 2022-09-10T14:18:51.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:51.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:51.350683+0000) 2022-09-10T14:18:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:51.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:51.731979+0000) 2022-09-10T14:18:52.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:52.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:52.350867+0000) 2022-09-10T14:18:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:52 smithi110 ceph-mon[97633]: pgmap v18865: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:52 smithi104 ceph-mon[97044]: pgmap v18865: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:52.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:52.732207+0000) 2022-09-10T14:18:53.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:53.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:53.351067+0000) 2022-09-10T14:18:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:53.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:53.732360+0000) 2022-09-10T14:18:54.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:54.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:54.351305+0000) 2022-09-10T14:18:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:54 smithi110 ceph-mon[97633]: pgmap v18866: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:54 smithi104 ceph-mon[97044]: pgmap v18866: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:54.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:54.732593+0000) 2022-09-10T14:18:55.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:55.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:55.351536+0000) 2022-09-10T14:18:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:55.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:55.732882+0000) 2022-09-10T14:18:56.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:56.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:56.351797+0000) 2022-09-10T14:18:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:56 smithi104 ceph-mon[97044]: pgmap v18867: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:56.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:56.733084+0000) 2022-09-10T14:18:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:56 smithi110 ceph-mon[97633]: pgmap v18867: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:18:57.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:57.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:57.352060+0000) 2022-09-10T14:18:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:57.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:57.733360+0000) 2022-09-10T14:18:58.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:58.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:58.352348+0000) 2022-09-10T14:18:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:18:58 smithi104 ceph-mon[97044]: pgmap v18868: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:58.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:58.733540+0000) 2022-09-10T14:18:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:18:58 smithi110 ceph-mon[97633]: pgmap v18868: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:18:59.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:18:59.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:59.352571+0000) 2022-09-10T14:19:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:18:59.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:18:59.733752+0000) 2022-09-10T14:19:00.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:00.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:00.352848+0000) 2022-09-10T14:19:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:00 smithi104 ceph-mon[97044]: pgmap v18869: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:00.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:00.733988+0000) 2022-09-10T14:19:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:00 smithi110 ceph-mon[97633]: pgmap v18869: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:01.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:01.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:01.353026+0000) 2022-09-10T14:19:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:01.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:01.734235+0000) 2022-09-10T14:19:02.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:02.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:02.353244+0000) 2022-09-10T14:19:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:02 smithi104 ceph-mon[97044]: pgmap v18870: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:02.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:02.734495+0000) 2022-09-10T14:19:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:02 smithi110 ceph-mon[97633]: pgmap v18870: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:03.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:03.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:03.353422+0000) 2022-09-10T14:19:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:03.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:03.734761+0000) 2022-09-10T14:19:04.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:04.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:04.353622+0000) 2022-09-10T14:19:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:04 smithi104 ceph-mon[97044]: pgmap v18871: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:05.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:04.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:04.735089+0000) 2022-09-10T14:19:05.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:19:04.743+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:19:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:04 smithi110 ceph-mon[97633]: pgmap v18871: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:05.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:05.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:05.353878+0000) 2022-09-10T14:19:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:05.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:05.735413+0000) 2022-09-10T14:19:06.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:06.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:06.354139+0000) 2022-09-10T14:19:07.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:06 smithi104 ceph-mon[97044]: pgmap v18872: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:19:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:06.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:06.735667+0000) 2022-09-10T14:19:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:06 smithi110 ceph-mon[97633]: pgmap v18872: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:19:07.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:07.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:07.354357+0000) 2022-09-10T14:19:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:07.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:07.735916+0000) 2022-09-10T14:19:08.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:08.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:08.354605+0000) 2022-09-10T14:19:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:08 smithi104 ceph-mon[97044]: pgmap v18873: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:19:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:08.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:08.736159+0000) 2022-09-10T14:19:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:08 smithi110 ceph-mon[97633]: pgmap v18873: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:19:09.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:09.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:09.354816+0000) 2022-09-10T14:19:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:09 smithi104 ceph-mon[97044]: osdmap e1260: 12 total, 12 up, 12 in 2022-09-10T14:19:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:09.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:09.736406+0000) 2022-09-10T14:19:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:09 smithi110 ceph-mon[97633]: osdmap e1260: 12 total, 12 up, 12 in 2022-09-10T14:19:10.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:10.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:10.355079+0000) 2022-09-10T14:19:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:10 smithi104 ceph-mon[97044]: pgmap v18875: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:19:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:10.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:10.736705+0000) 2022-09-10T14:19:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:10 smithi110 ceph-mon[97633]: pgmap v18875: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:19:11.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:11.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:11.355336+0000) 2022-09-10T14:19:12.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:11.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:11.736911+0000) 2022-09-10T14:19:12.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:12.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:12.355585+0000) 2022-09-10T14:19:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:12 smithi104 ceph-mon[97044]: pgmap v18876: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:19:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:12.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:12.737104+0000) 2022-09-10T14:19:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:12 smithi110 ceph-mon[97633]: pgmap v18876: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:19:13.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:13.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:13.355830+0000) 2022-09-10T14:19:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:13.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:13.737329+0000) 2022-09-10T14:19:14.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:14.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:14.356021+0000) 2022-09-10T14:19:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:14 smithi104 ceph-mon[97044]: pgmap v18877: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:19:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:14.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:14.737531+0000) 2022-09-10T14:19:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:14 smithi110 ceph-mon[97633]: pgmap v18877: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:19:15.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:15.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:15.356198+0000) 2022-09-10T14:19:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:15.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:15.737781+0000) 2022-09-10T14:19:16.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:16.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:16.356406+0000) 2022-09-10T14:19:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:16 smithi104 ceph-mon[97044]: pgmap v18878: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:19:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:16 smithi104 ceph-mon[97044]: osdmap e1261: 12 total, 12 up, 12 in 2022-09-10T14:19:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:16.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:16.738007+0000) 2022-09-10T14:19:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:16 smithi110 ceph-mon[97633]: pgmap v18878: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:19:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:16 smithi110 ceph-mon[97633]: osdmap e1261: 12 total, 12 up, 12 in 2022-09-10T14:19:17.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:17.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:17.356573+0000) 2022-09-10T14:19:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:17.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:17.738225+0000) 2022-09-10T14:19:18.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:18.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:18.356721+0000) 2022-09-10T14:19:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:18 smithi104 ceph-mon[97044]: pgmap v18880: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T14:19:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:18.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:18.738467+0000) 2022-09-10T14:19:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:18 smithi110 ceph-mon[97633]: pgmap v18880: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 242 B/s wr, 12 op/s 2022-09-10T14:19:19.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:19.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:19.356973+0000) 2022-09-10T14:19:20.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:19:19.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:19:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:19.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:19.738711+0000) 2022-09-10T14:19:20.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:20.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:20.357220+0000) 2022-09-10T14:19:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:20 smithi104 ceph-mon[97044]: pgmap v18881: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:19:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:20.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:20.738977+0000) 2022-09-10T14:19:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:20 smithi110 ceph-mon[97633]: pgmap v18881: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:19:21.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:21.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:21.357447+0000) 2022-09-10T14:19:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:21.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:21.739235+0000) 2022-09-10T14:19:22.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:22.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:22.357609+0000) 2022-09-10T14:19:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:22 smithi104 ceph-mon[97044]: pgmap v18882: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:19:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:22.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:22.739484+0000) 2022-09-10T14:19:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:22 smithi110 ceph-mon[97633]: pgmap v18882: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:19:23.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:23.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:23.357840+0000) 2022-09-10T14:19:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:23.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:23.739729+0000) 2022-09-10T14:19:24.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:24.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:24.358078+0000) 2022-09-10T14:19:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:24 smithi104 ceph-mon[97044]: pgmap v18883: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:19:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:24.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:24.739928+0000) 2022-09-10T14:19:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:24 smithi110 ceph-mon[97633]: pgmap v18883: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:19:25.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:25.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:25.358319+0000) 2022-09-10T14:19:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:25.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:25.740210+0000) 2022-09-10T14:19:26.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:26.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:26.358560+0000) 2022-09-10T14:19:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:26 smithi104 ceph-mon[97044]: pgmap v18884: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:19:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:26.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:26.740402+0000) 2022-09-10T14:19:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:26 smithi110 ceph-mon[97633]: pgmap v18884: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:19:27.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:27.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:27.358819+0000) 2022-09-10T14:19:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:27.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:27.740625+0000) 2022-09-10T14:19:28.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:28.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:28.359040+0000) 2022-09-10T14:19:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:28 smithi104 ceph-mon[97044]: pgmap v18885: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:19:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:28.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:28.740916+0000) 2022-09-10T14:19:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:28 smithi110 ceph-mon[97633]: pgmap v18885: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:19:29.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:29.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:29.359264+0000) 2022-09-10T14:19:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:29.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:29.741168+0000) 2022-09-10T14:19:30.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:30.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:30.359502+0000) 2022-09-10T14:19:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:30 smithi104 ceph-mon[97044]: pgmap v18886: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:30.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:30.741380+0000) 2022-09-10T14:19:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:30 smithi110 ceph-mon[97633]: pgmap v18886: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:31.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:31.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:31.359710+0000) 2022-09-10T14:19:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:31.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:31.741615+0000) 2022-09-10T14:19:32.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:32.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:32.359966+0000) 2022-09-10T14:19:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:32 smithi104 ceph-mon[97044]: pgmap v18887: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:32.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:32.741862+0000) 2022-09-10T14:19:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:32 smithi110 ceph-mon[97633]: pgmap v18887: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:33.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:33.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:33.360156+0000) 2022-09-10T14:19:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:33.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:33.742150+0000) 2022-09-10T14:19:34.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:34.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:34.360341+0000) 2022-09-10T14:19:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:34 smithi104 ceph-mon[97044]: pgmap v18888: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:34.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:34.742413+0000) 2022-09-10T14:19:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:19:34.764+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:19:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:34 smithi110 ceph-mon[97633]: pgmap v18888: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:35.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:35.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:35.360574+0000) 2022-09-10T14:19:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:35.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:35.742684+0000) 2022-09-10T14:19:36.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:36.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:36.360820+0000) 2022-09-10T14:19:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:36 smithi104 ceph-mon[97044]: pgmap v18889: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:36.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:36.742942+0000) 2022-09-10T14:19:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:36 smithi110 ceph-mon[97633]: pgmap v18889: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:37.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:37.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:37.361071+0000) 2022-09-10T14:19:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:37.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:37.743137+0000) 2022-09-10T14:19:38.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:38.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:38.361243+0000) 2022-09-10T14:19:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:38 smithi104 ceph-mon[97044]: pgmap v18890: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:38.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:38.743390+0000) 2022-09-10T14:19:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:38 smithi110 ceph-mon[97633]: pgmap v18890: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:39.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:39.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:39.361457+0000) 2022-09-10T14:19:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:39.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:39.743644+0000) 2022-09-10T14:19:40.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:40.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:40.361684+0000) 2022-09-10T14:19:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:40 smithi104 ceph-mon[97044]: pgmap v18891: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:40.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:40.743949+0000) 2022-09-10T14:19:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:40 smithi110 ceph-mon[97633]: pgmap v18891: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:41.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:41.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:41.361890+0000) 2022-09-10T14:19:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:41.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:41.744108+0000) 2022-09-10T14:19:42.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:42.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:42.362080+0000) 2022-09-10T14:19:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:42 smithi104 ceph-mon[97044]: pgmap v18892: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:42.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:42.744392+0000) 2022-09-10T14:19:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:42 smithi110 ceph-mon[97633]: pgmap v18892: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:43.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:43.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:43.362341+0000) 2022-09-10T14:19:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:43.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:43.744664+0000) 2022-09-10T14:19:44.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:44.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:44.362564+0000) 2022-09-10T14:19:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:44 smithi104 ceph-mon[97044]: pgmap v18893: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:44.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:44.744847+0000) 2022-09-10T14:19:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:44 smithi110 ceph-mon[97633]: pgmap v18893: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:45.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:45.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:45.362761+0000) 2022-09-10T14:19:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:45.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:45.744999+0000) 2022-09-10T14:19:46.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:46.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:46.362997+0000) 2022-09-10T14:19:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:46 smithi104 ceph-mon[97044]: pgmap v18894: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:19:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:19:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:19:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:19:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:46.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:46.745171+0000) 2022-09-10T14:19:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:46 smithi110 ceph-mon[97633]: pgmap v18894: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:19:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:19:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:19:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:19:47.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:47.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:47.363288+0000) 2022-09-10T14:19:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:47.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:47.745327+0000) 2022-09-10T14:19:48.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:48.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:48.363518+0000) 2022-09-10T14:19:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:48 smithi104 ceph-mon[97044]: pgmap v18895: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:48.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:48.745511+0000) 2022-09-10T14:19:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:48 smithi110 ceph-mon[97633]: pgmap v18895: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:49.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:49.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:49.363716+0000) 2022-09-10T14:19:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:49.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:49.745723+0000) 2022-09-10T14:19:50.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:19:49.777+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:19:50.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:50.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:50.363978+0000) 2022-09-10T14:19:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:50 smithi104 ceph-mon[97044]: pgmap v18896: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:50.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:50.745897+0000) 2022-09-10T14:19:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:50 smithi110 ceph-mon[97633]: pgmap v18896: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:51.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:51.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:51.364143+0000) 2022-09-10T14:19:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:51.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:51.746079+0000) 2022-09-10T14:19:52.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:52.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:52.364332+0000) 2022-09-10T14:19:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:52 smithi104 ceph-mon[97044]: pgmap v18897: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:52.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:52.746272+0000) 2022-09-10T14:19:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:52 smithi110 ceph-mon[97633]: pgmap v18897: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:53.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:53.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:53.364519+0000) 2022-09-10T14:19:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:53.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:53.746534+0000) 2022-09-10T14:19:54.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:54.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:54.364663+0000) 2022-09-10T14:19:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:54 smithi104 ceph-mon[97044]: pgmap v18898: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:54.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:54.746752+0000) 2022-09-10T14:19:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:54 smithi110 ceph-mon[97633]: pgmap v18898: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:55.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:55.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:55.364900+0000) 2022-09-10T14:19:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:55.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:55.746987+0000) 2022-09-10T14:19:56.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:56.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:56.365202+0000) 2022-09-10T14:19:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:56 smithi104 ceph-mon[97044]: pgmap v18899: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:56.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:56.747205+0000) 2022-09-10T14:19:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:56 smithi110 ceph-mon[97633]: pgmap v18899: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:19:57.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:57.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:57.365366+0000) 2022-09-10T14:19:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:57.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:57.747470+0000) 2022-09-10T14:19:58.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:58.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:58.365546+0000) 2022-09-10T14:19:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:19:58 smithi104 ceph-mon[97044]: pgmap v18900: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:58.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:58.747732+0000) 2022-09-10T14:19:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:19:58 smithi110 ceph-mon[97633]: pgmap v18900: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:19:59.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:19:59.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:59.365687+0000) 2022-09-10T14:20:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:19:59.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:19:59.747994+0000) 2022-09-10T14:20:00.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:00.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:00.365855+0000) 2022-09-10T14:20:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:00 smithi104 ceph-mon[97044]: pgmap v18901: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:20:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:00.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:00.748197+0000) 2022-09-10T14:20:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:00 smithi110 ceph-mon[97633]: pgmap v18901: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:20:01.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:01.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:01.366067+0000) 2022-09-10T14:20:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:01.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:01.748463+0000) 2022-09-10T14:20:02.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:02.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:02.366291+0000) 2022-09-10T14:20:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:02 smithi104 ceph-mon[97044]: pgmap v18902: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:02.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:02.748704+0000) 2022-09-10T14:20:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:02 smithi110 ceph-mon[97633]: pgmap v18902: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:03.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:03.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:03.366515+0000) 2022-09-10T14:20:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:03.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:03.748914+0000) 2022-09-10T14:20:04.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:04.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:04.366749+0000) 2022-09-10T14:20:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:04 smithi104 ceph-mon[97044]: pgmap v18903: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:04.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:04.749200+0000) 2022-09-10T14:20:05.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:20:04.780+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:20:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:04 smithi110 ceph-mon[97633]: pgmap v18903: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:05.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:05.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:05.366958+0000) 2022-09-10T14:20:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:05.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:05.749486+0000) 2022-09-10T14:20:06.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:06.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:06.367179+0000) 2022-09-10T14:20:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:06 smithi104 ceph-mon[97044]: pgmap v18904: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:20:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:06.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:06.749707+0000) 2022-09-10T14:20:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:06 smithi110 ceph-mon[97633]: pgmap v18904: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:20:07.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:07.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:07.367425+0000) 2022-09-10T14:20:08.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:07.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:07.749957+0000) 2022-09-10T14:20:08.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:08.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:08.367645+0000) 2022-09-10T14:20:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:08 smithi104 ceph-mon[97044]: pgmap v18905: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:20:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:08.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:08.750215+0000) 2022-09-10T14:20:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:08 smithi110 ceph-mon[97633]: pgmap v18905: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:20:09.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:09.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:09.367893+0000) 2022-09-10T14:20:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:09 smithi104 ceph-mon[97044]: osdmap e1262: 12 total, 12 up, 12 in 2022-09-10T14:20:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:09.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:09.750478+0000) 2022-09-10T14:20:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:09 smithi110 ceph-mon[97633]: osdmap e1262: 12 total, 12 up, 12 in 2022-09-10T14:20:10.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:10.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:10.368122+0000) 2022-09-10T14:20:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:10 smithi104 ceph-mon[97044]: pgmap v18907: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:20:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:10.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:10.750695+0000) 2022-09-10T14:20:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:10 smithi110 ceph-mon[97633]: pgmap v18907: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:20:11.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:11.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:11.368304+0000) 2022-09-10T14:20:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:11.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:11.750860+0000) 2022-09-10T14:20:12.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:12.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:12.368520+0000) 2022-09-10T14:20:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:12 smithi104 ceph-mon[97044]: pgmap v18908: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:20:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:12.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:12.751066+0000) 2022-09-10T14:20:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:12 smithi110 ceph-mon[97633]: pgmap v18908: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:20:13.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:13.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:13.368704+0000) 2022-09-10T14:20:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:13.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:13.751279+0000) 2022-09-10T14:20:14.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:14.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:14.368937+0000) 2022-09-10T14:20:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:14 smithi104 ceph-mon[97044]: pgmap v18909: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:20:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:14.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:14.751510+0000) 2022-09-10T14:20:15.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:14 smithi110 ceph-mon[97633]: pgmap v18909: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:20:15.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:15.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:15.369206+0000) 2022-09-10T14:20:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:15.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:15.751747+0000) 2022-09-10T14:20:16.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:16.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:16.369439+0000) 2022-09-10T14:20:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:16.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:16.751994+0000) 2022-09-10T14:20:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:16 smithi110 ceph-mon[97633]: pgmap v18910: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:20:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:16 smithi110 ceph-mon[97633]: osdmap e1263: 12 total, 12 up, 12 in 2022-09-10T14:20:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:16 smithi104 ceph-mon[97044]: pgmap v18910: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:20:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:16 smithi104 ceph-mon[97044]: osdmap e1263: 12 total, 12 up, 12 in 2022-09-10T14:20:17.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:17.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:17.369612+0000) 2022-09-10T14:20:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:17.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:17.752260+0000) 2022-09-10T14:20:18.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:18.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:18.369802+0000) 2022-09-10T14:20:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:18.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:18.752524+0000) 2022-09-10T14:20:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:18 smithi110 ceph-mon[97633]: pgmap v18912: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T14:20:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:18 smithi104 ceph-mon[97044]: pgmap v18912: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 244 B/s wr, 12 op/s 2022-09-10T14:20:19.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:19.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:19.370013+0000) 2022-09-10T14:20:20.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:19.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:19.752786+0000) 2022-09-10T14:20:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:20:19.795+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:20:20.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:20.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:20.370261+0000) 2022-09-10T14:20:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:20.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:20.753015+0000) 2022-09-10T14:20:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:20 smithi110 ceph-mon[97633]: pgmap v18913: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:20:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:20 smithi104 ceph-mon[97044]: pgmap v18913: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:20:21.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:21.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:21.370484+0000) 2022-09-10T14:20:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:21.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:21.753245+0000) 2022-09-10T14:20:22.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:22.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:22.370669+0000) 2022-09-10T14:20:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:22.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:22.753524+0000) 2022-09-10T14:20:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:22 smithi110 ceph-mon[97633]: pgmap v18914: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:20:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:22 smithi104 ceph-mon[97044]: pgmap v18914: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:20:23.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:23.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:23.370922+0000) 2022-09-10T14:20:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:23.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:23.753810+0000) 2022-09-10T14:20:24.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:24.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:24.371154+0000) 2022-09-10T14:20:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:24.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:24.754049+0000) 2022-09-10T14:20:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:24 smithi110 ceph-mon[97633]: pgmap v18915: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:20:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:24 smithi104 ceph-mon[97044]: pgmap v18915: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:20:25.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:25.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:25.371327+0000) 2022-09-10T14:20:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:25.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:25.754229+0000) 2022-09-10T14:20:26.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:26.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:26.371540+0000) 2022-09-10T14:20:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:26.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:26.754454+0000) 2022-09-10T14:20:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:26 smithi110 ceph-mon[97633]: pgmap v18916: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:20:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:26 smithi104 ceph-mon[97044]: pgmap v18916: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:20:27.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:27.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:27.371744+0000) 2022-09-10T14:20:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:27.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:27.754633+0000) 2022-09-10T14:20:28.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:28.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:28.371956+0000) 2022-09-10T14:20:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:28.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:28.754885+0000) 2022-09-10T14:20:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:28 smithi110 ceph-mon[97633]: pgmap v18917: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:20:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:28 smithi104 ceph-mon[97044]: pgmap v18917: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:20:29.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:29.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:29.372179+0000) 2022-09-10T14:20:30.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:29.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:29.755140+0000) 2022-09-10T14:20:30.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:30.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:30.372401+0000) 2022-09-10T14:20:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:30.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:30.755258+0000) 2022-09-10T14:20:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:30 smithi110 ceph-mon[97633]: pgmap v18918: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:30 smithi104 ceph-mon[97044]: pgmap v18918: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:31.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:31.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:31.372572+0000) 2022-09-10T14:20:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:31.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:31.755443+0000) 2022-09-10T14:20:32.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:32.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:32.372784+0000) 2022-09-10T14:20:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:32.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:32.755695+0000) 2022-09-10T14:20:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:32 smithi110 ceph-mon[97633]: pgmap v18919: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:32 smithi104 ceph-mon[97044]: pgmap v18919: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:33.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:33.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:33.372997+0000) 2022-09-10T14:20:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:33.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:33.755915+0000) 2022-09-10T14:20:34.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:34.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:34.373193+0000) 2022-09-10T14:20:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:34.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:34.756119+0000) 2022-09-10T14:20:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:20:34.805+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:20:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:34 smithi110 ceph-mon[97633]: pgmap v18920: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:34 smithi104 ceph-mon[97044]: pgmap v18920: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:35.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:35.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:35.373317+0000) 2022-09-10T14:20:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:35.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:35.756367+0000) 2022-09-10T14:20:36.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:36.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:36.373546+0000) 2022-09-10T14:20:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:36.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:36.756534+0000) 2022-09-10T14:20:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:36 smithi110 ceph-mon[97633]: pgmap v18921: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:36 smithi104 ceph-mon[97044]: pgmap v18921: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:37.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:37.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:37.373756+0000) 2022-09-10T14:20:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:37.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:37.756697+0000) 2022-09-10T14:20:38.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:38.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:38.373990+0000) 2022-09-10T14:20:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:38.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:38.756909+0000) 2022-09-10T14:20:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:38 smithi110 ceph-mon[97633]: pgmap v18922: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:38 smithi104 ceph-mon[97044]: pgmap v18922: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:39.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:39.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:39.374202+0000) 2022-09-10T14:20:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:39.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:39.757174+0000) 2022-09-10T14:20:40.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:40.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:40.374450+0000) 2022-09-10T14:20:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:40.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:40.757414+0000) 2022-09-10T14:20:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:40 smithi110 ceph-mon[97633]: pgmap v18923: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:40 smithi104 ceph-mon[97044]: pgmap v18923: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:41.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:41.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:41.374721+0000) 2022-09-10T14:20:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:41.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:41.757651+0000) 2022-09-10T14:20:42.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:42.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:42.374903+0000) 2022-09-10T14:20:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:42.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:42.757853+0000) 2022-09-10T14:20:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:42 smithi110 ceph-mon[97633]: pgmap v18924: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:42 smithi104 ceph-mon[97044]: pgmap v18924: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:43.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:43.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:43.375164+0000) 2022-09-10T14:20:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:43.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:43.758081+0000) 2022-09-10T14:20:44.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:44.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:44.375344+0000) 2022-09-10T14:20:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:44.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:44.758269+0000) 2022-09-10T14:20:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:44 smithi110 ceph-mon[97633]: pgmap v18925: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:44 smithi104 ceph-mon[97044]: pgmap v18925: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:45.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:45.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:45.375563+0000) 2022-09-10T14:20:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:45.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:45.758448+0000) 2022-09-10T14:20:46.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:46.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:46.375794+0000) 2022-09-10T14:20:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:46.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:46.758645+0000) 2022-09-10T14:20:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:46 smithi110 ceph-mon[97633]: pgmap v18926: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:20:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:20:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:20:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:20:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:46 smithi104 ceph-mon[97044]: pgmap v18926: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:20:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:20:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:20:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:20:47.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:47.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:47.376011+0000) 2022-09-10T14:20:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:47.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:47.758826+0000) 2022-09-10T14:20:48.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:48.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:48.376257+0000) 2022-09-10T14:20:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:48.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:48.759026+0000) 2022-09-10T14:20:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:48 smithi110 ceph-mon[97633]: pgmap v18927: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:48 smithi104 ceph-mon[97044]: pgmap v18927: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:49.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:49.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:49.376452+0000) 2022-09-10T14:20:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:49.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:49.759163+0000) 2022-09-10T14:20:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:20:49.819+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:20:50.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:50.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:50.376638+0000) 2022-09-10T14:20:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:50.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:50.759423+0000) 2022-09-10T14:20:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:50 smithi110 ceph-mon[97633]: pgmap v18928: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:50 smithi104 ceph-mon[97044]: pgmap v18928: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:51.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:51.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:51.376863+0000) 2022-09-10T14:20:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:51.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:51.759634+0000) 2022-09-10T14:20:52.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:52.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:52.377086+0000) 2022-09-10T14:20:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:52.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:52.759884+0000) 2022-09-10T14:20:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:52 smithi110 ceph-mon[97633]: pgmap v18929: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:52 smithi104 ceph-mon[97044]: pgmap v18929: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:53.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:53.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:53.377350+0000) 2022-09-10T14:20:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:53.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:53.760039+0000) 2022-09-10T14:20:54.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:54.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:54.377602+0000) 2022-09-10T14:20:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:54.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:54.760220+0000) 2022-09-10T14:20:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:54 smithi110 ceph-mon[97633]: pgmap v18930: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:54 smithi104 ceph-mon[97044]: pgmap v18930: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:55.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:55.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:55.377796+0000) 2022-09-10T14:20:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:55.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:55.760348+0000) 2022-09-10T14:20:56.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:56.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:56.378027+0000) 2022-09-10T14:20:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:56.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:56.760501+0000) 2022-09-10T14:20:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:56 smithi110 ceph-mon[97633]: pgmap v18931: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:56 smithi104 ceph-mon[97044]: pgmap v18931: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:20:57.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:57.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:57.378298+0000) 2022-09-10T14:20:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:57.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:57.760647+0000) 2022-09-10T14:20:58.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:58.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:58.378558+0000) 2022-09-10T14:20:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:58.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:58.760876+0000) 2022-09-10T14:20:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:20:58 smithi110 ceph-mon[97633]: pgmap v18932: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:20:58 smithi104 ceph-mon[97044]: pgmap v18932: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:20:59.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:20:59.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:59.378807+0000) 2022-09-10T14:21:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:20:59.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:20:59.761054+0000) 2022-09-10T14:21:00.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:00.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:00.378939+0000) 2022-09-10T14:21:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:00.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:00.761223+0000) 2022-09-10T14:21:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:00 smithi110 ceph-mon[97633]: pgmap v18933: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:00 smithi104 ceph-mon[97044]: pgmap v18933: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:01.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:01.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:01.379188+0000) 2022-09-10T14:21:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:01.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:01.761393+0000) 2022-09-10T14:21:02.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:02.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:02.379403+0000) 2022-09-10T14:21:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:02.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:02.761624+0000) 2022-09-10T14:21:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:02 smithi110 ceph-mon[97633]: pgmap v18934: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:02 smithi104 ceph-mon[97044]: pgmap v18934: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:03.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:03.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:03.379626+0000) 2022-09-10T14:21:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:03.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:03.761826+0000) 2022-09-10T14:21:04.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:04.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:04.379806+0000) 2022-09-10T14:21:05.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:04.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:04.762052+0000) 2022-09-10T14:21:05.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:21:04.830+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:21:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:04 smithi110 ceph-mon[97633]: pgmap v18935: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:04 smithi104 ceph-mon[97044]: pgmap v18935: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:05.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:05.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:05.380042+0000) 2022-09-10T14:21:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:05.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:05.762327+0000) 2022-09-10T14:21:06.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:06.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:06.380266+0000) 2022-09-10T14:21:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:06.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:06.762538+0000) 2022-09-10T14:21:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:06 smithi110 ceph-mon[97633]: pgmap v18936: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:21:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:06 smithi104 ceph-mon[97044]: pgmap v18936: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:21:07.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:07.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:07.380430+0000) 2022-09-10T14:21:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:07.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:07.762730+0000) 2022-09-10T14:21:08.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:08.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:08.380626+0000) 2022-09-10T14:21:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:08.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:08.762932+0000) 2022-09-10T14:21:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:08 smithi110 ceph-mon[97633]: pgmap v18937: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:21:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:08 smithi110 ceph-mon[97633]: osdmap e1264: 12 total, 12 up, 12 in 2022-09-10T14:21:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:08 smithi104 ceph-mon[97044]: pgmap v18937: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:21:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:08 smithi104 ceph-mon[97044]: osdmap e1264: 12 total, 12 up, 12 in 2022-09-10T14:21:09.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:09.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:09.380889+0000) 2022-09-10T14:21:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:09.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:09.763197+0000) 2022-09-10T14:21:10.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:10.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:10.381146+0000) 2022-09-10T14:21:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:10.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:10.763390+0000) 2022-09-10T14:21:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:10 smithi110 ceph-mon[97633]: pgmap v18939: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:21:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:10 smithi104 ceph-mon[97044]: pgmap v18939: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:21:11.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:11.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:11.381375+0000) 2022-09-10T14:21:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:11.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:11.763615+0000) 2022-09-10T14:21:12.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:12.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:12.381596+0000) 2022-09-10T14:21:12.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:12 smithi104 ceph-mon[97044]: pgmap v18940: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:21:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:12 smithi110 ceph-mon[97633]: pgmap v18940: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:21:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:12.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:12.763926+0000) 2022-09-10T14:21:13.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:13.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:13.381817+0000) 2022-09-10T14:21:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:13.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:13.764175+0000) 2022-09-10T14:21:14.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:14.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:14.381961+0000) 2022-09-10T14:21:14.796 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:14 smithi104 ceph-mon[97044]: pgmap v18941: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:21:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:14 smithi110 ceph-mon[97633]: pgmap v18941: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:21:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:14.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:14.764390+0000) 2022-09-10T14:21:15.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:15.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:15.382199+0000) 2022-09-10T14:21:16.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:15.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:15.764569+0000) 2022-09-10T14:21:16.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:16.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:16.382342+0000) 2022-09-10T14:21:16.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:16 smithi104 ceph-mon[97044]: pgmap v18942: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:21:16.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:16 smithi104 ceph-mon[97044]: osdmap e1265: 12 total, 12 up, 12 in 2022-09-10T14:21:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:16 smithi110 ceph-mon[97633]: pgmap v18942: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:21:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:16 smithi110 ceph-mon[97633]: osdmap e1265: 12 total, 12 up, 12 in 2022-09-10T14:21:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:16.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:16.764808+0000) 2022-09-10T14:21:17.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:17.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:17.382518+0000) 2022-09-10T14:21:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:17.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:17.765025+0000) 2022-09-10T14:21:18.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:18.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:18.382718+0000) 2022-09-10T14:21:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:18 smithi104 ceph-mon[97044]: pgmap v18944: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T14:21:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:18 smithi110 ceph-mon[97633]: pgmap v18944: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 220 B/s wr, 11 op/s 2022-09-10T14:21:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:18.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:18.765237+0000) 2022-09-10T14:21:19.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:19.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:19.382964+0000) 2022-09-10T14:21:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:19.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:19.765447+0000) 2022-09-10T14:21:20.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:21:19.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:21:20.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:20.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:20.383186+0000) 2022-09-10T14:21:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:20 smithi104 ceph-mon[97044]: pgmap v18945: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:21:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:20 smithi110 ceph-mon[97633]: pgmap v18945: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:21:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:20.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:20.765704+0000) 2022-09-10T14:21:21.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:21.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:21.383368+0000) 2022-09-10T14:21:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:21.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:21.765933+0000) 2022-09-10T14:21:22.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:22.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:22.383554+0000) 2022-09-10T14:21:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:22 smithi110 ceph-mon[97633]: pgmap v18946: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:21:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:22 smithi104 ceph-mon[97044]: pgmap v18946: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:21:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:22.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:22.766147+0000) 2022-09-10T14:21:23.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:23.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:23.383724+0000) 2022-09-10T14:21:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:23.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:23.766392+0000) 2022-09-10T14:21:24.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:24.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:24.383934+0000) 2022-09-10T14:21:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:24 smithi110 ceph-mon[97633]: pgmap v18947: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:21:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:24 smithi104 ceph-mon[97044]: pgmap v18947: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:21:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:24.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:24.766576+0000) 2022-09-10T14:21:25.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:25.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:25.384212+0000) 2022-09-10T14:21:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:25.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:25.766821+0000) 2022-09-10T14:21:26.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:26.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:26.384463+0000) 2022-09-10T14:21:26.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:26 smithi110 ceph-mon[97633]: pgmap v18948: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:21:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:26 smithi104 ceph-mon[97044]: pgmap v18948: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:21:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:26.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:26.766996+0000) 2022-09-10T14:21:27.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:27.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:27.384715+0000) 2022-09-10T14:21:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:27.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:27.767264+0000) 2022-09-10T14:21:28.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:28.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:28.384911+0000) 2022-09-10T14:21:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:28 smithi110 ceph-mon[97633]: pgmap v18949: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:21:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:28 smithi104 ceph-mon[97044]: pgmap v18949: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:21:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:28.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:28.767506+0000) 2022-09-10T14:21:29.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:29.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:29.385195+0000) 2022-09-10T14:21:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:29.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:29.767753+0000) 2022-09-10T14:21:30.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:30.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:30.385374+0000) 2022-09-10T14:21:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:30 smithi110 ceph-mon[97633]: pgmap v18950: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:30 smithi104 ceph-mon[97044]: pgmap v18950: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:30.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:30.767977+0000) 2022-09-10T14:21:31.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:31.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:31.385596+0000) 2022-09-10T14:21:32.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:31.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:31.768267+0000) 2022-09-10T14:21:32.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:32.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:32.385838+0000) 2022-09-10T14:21:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:32 smithi110 ceph-mon[97633]: pgmap v18951: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:32 smithi104 ceph-mon[97044]: pgmap v18951: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:32.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:32.768479+0000) 2022-09-10T14:21:33.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:33.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:33.386177+0000) 2022-09-10T14:21:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:33.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:33.768694+0000) 2022-09-10T14:21:34.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:34.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:34.386418+0000) 2022-09-10T14:21:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:34 smithi110 ceph-mon[97633]: pgmap v18952: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:34 smithi104 ceph-mon[97044]: pgmap v18952: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:34.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:34.768920+0000) 2022-09-10T14:21:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:21:34.850+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:21:35.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:35.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:35.386657+0000) 2022-09-10T14:21:36.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:35.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:35.769162+0000) 2022-09-10T14:21:36.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:36.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:36.386921+0000) 2022-09-10T14:21:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:36 smithi110 ceph-mon[97633]: pgmap v18953: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:36 smithi104 ceph-mon[97044]: pgmap v18953: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:36.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:36.769370+0000) 2022-09-10T14:21:37.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:37.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:37.387162+0000) 2022-09-10T14:21:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:37.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:37.769620+0000) 2022-09-10T14:21:38.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:38.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:38.387377+0000) 2022-09-10T14:21:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:38 smithi110 ceph-mon[97633]: pgmap v18954: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:38 smithi104 ceph-mon[97044]: pgmap v18954: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:38.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:38.769863+0000) 2022-09-10T14:21:39.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:39.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:39.387634+0000) 2022-09-10T14:21:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:39.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:39.770063+0000) 2022-09-10T14:21:40.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:40.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:40.387878+0000) 2022-09-10T14:21:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:40 smithi110 ceph-mon[97633]: pgmap v18955: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:41.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:40 smithi104 ceph-mon[97044]: pgmap v18955: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:40.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:40.770307+0000) 2022-09-10T14:21:41.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:41.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:41.388076+0000) 2022-09-10T14:21:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:41.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:41.770544+0000) 2022-09-10T14:21:42.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:42.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:42.388368+0000) 2022-09-10T14:21:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:42 smithi110 ceph-mon[97633]: pgmap v18956: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:42 smithi104 ceph-mon[97044]: pgmap v18956: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:42.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:42.770771+0000) 2022-09-10T14:21:43.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:43.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:43.388554+0000) 2022-09-10T14:21:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:43.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:43.770988+0000) 2022-09-10T14:21:44.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:44.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:44.388782+0000) 2022-09-10T14:21:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:44 smithi110 ceph-mon[97633]: pgmap v18957: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:44 smithi104 ceph-mon[97044]: pgmap v18957: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:45.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:44.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:44.771199+0000) 2022-09-10T14:21:45.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:45.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:45.389033+0000) 2022-09-10T14:21:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:45.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:45.771451+0000) 2022-09-10T14:21:46.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:46.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:46.389324+0000) 2022-09-10T14:21:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:46 smithi110 ceph-mon[97633]: pgmap v18958: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:46 smithi104 ceph-mon[97044]: pgmap v18958: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:46.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:46.771614+0000) 2022-09-10T14:21:47.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:47.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:47.389549+0000) 2022-09-10T14:21:47.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:21:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:21:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:21:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:21:48.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:21:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:21:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:21:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:21:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:47.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:47.771740+0000) 2022-09-10T14:21:48.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:48.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:48.389731+0000) 2022-09-10T14:21:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:48 smithi110 ceph-mon[97633]: pgmap v18959: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:48 smithi104 ceph-mon[97044]: pgmap v18959: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:48.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:48.771936+0000) 2022-09-10T14:21:49.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:49.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:49.389977+0000) 2022-09-10T14:21:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:21:49.862+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:21:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:49.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:49.772144+0000) 2022-09-10T14:21:50.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:50.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:50.390204+0000) 2022-09-10T14:21:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:50 smithi110 ceph-mon[97633]: pgmap v18960: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:50 smithi104 ceph-mon[97044]: pgmap v18960: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:50.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:50.772378+0000) 2022-09-10T14:21:51.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:51.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:51.390444+0000) 2022-09-10T14:21:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:51.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:51.772575+0000) 2022-09-10T14:21:52.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:52.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:52.390674+0000) 2022-09-10T14:21:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:52 smithi110 ceph-mon[97633]: pgmap v18961: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:52 smithi104 ceph-mon[97044]: pgmap v18961: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:52.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:52.772774+0000) 2022-09-10T14:21:53.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:53.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:53.390918+0000) 2022-09-10T14:21:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:53.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:53.772957+0000) 2022-09-10T14:21:54.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:54.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:54.391164+0000) 2022-09-10T14:21:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:54 smithi110 ceph-mon[97633]: pgmap v18962: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:54 smithi104 ceph-mon[97044]: pgmap v18962: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:54.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:54.773188+0000) 2022-09-10T14:21:55.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:55.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:55.391333+0000) 2022-09-10T14:21:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:55.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:55.773491+0000) 2022-09-10T14:21:56.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:56.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:56.391601+0000) 2022-09-10T14:21:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:56 smithi110 ceph-mon[97633]: pgmap v18963: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:56 smithi104 ceph-mon[97044]: pgmap v18963: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:21:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:56.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:56.773644+0000) 2022-09-10T14:21:57.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:57.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:57.391868+0000) 2022-09-10T14:21:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:57.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:57.773800+0000) 2022-09-10T14:21:58.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:58.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:58.392150+0000) 2022-09-10T14:21:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:21:58 smithi110 ceph-mon[97633]: pgmap v18964: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:21:58 smithi104 ceph-mon[97044]: pgmap v18964: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:21:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:58.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:58.774015+0000) 2022-09-10T14:21:59.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:21:59.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:59.392397+0000) 2022-09-10T14:22:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:21:59.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:21:59.774269+0000) 2022-09-10T14:22:00.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:00.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:00.392554+0000) 2022-09-10T14:22:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:00 smithi110 ceph-mon[97633]: pgmap v18965: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:00 smithi104 ceph-mon[97044]: pgmap v18965: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:00.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:00.774479+0000) 2022-09-10T14:22:01.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:01.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:01.392700+0000) 2022-09-10T14:22:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:01.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:01.774746+0000) 2022-09-10T14:22:02.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:02.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:02.392897+0000) 2022-09-10T14:22:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:02 smithi110 ceph-mon[97633]: pgmap v18966: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:02 smithi104 ceph-mon[97044]: pgmap v18966: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:02.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:02.774997+0000) 2022-09-10T14:22:03.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:03.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:03.393189+0000) 2022-09-10T14:22:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:03.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:03.775241+0000) 2022-09-10T14:22:04.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:04.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:04.393364+0000) 2022-09-10T14:22:04.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:04 smithi110 ceph-mon[97633]: pgmap v18967: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:04 smithi104 ceph-mon[97044]: pgmap v18967: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:05.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:22:04.865+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:22:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:04.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:04.775479+0000) 2022-09-10T14:22:05.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:05.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:05.393550+0000) 2022-09-10T14:22:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:05.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:05.775691+0000) 2022-09-10T14:22:06.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:06.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:06.393779+0000) 2022-09-10T14:22:06.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:06 smithi110 ceph-mon[97633]: pgmap v18968: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:22:07.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:06 smithi104 ceph-mon[97044]: pgmap v18968: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:22:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:06.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:06.775827+0000) 2022-09-10T14:22:07.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:07.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:07.394030+0000) 2022-09-10T14:22:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:07.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:07.775966+0000) 2022-09-10T14:22:08.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:08.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:08.394282+0000) 2022-09-10T14:22:08.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:08 smithi110 ceph-mon[97633]: pgmap v18969: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:22:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:08 smithi104 ceph-mon[97044]: pgmap v18969: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:22:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:08.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:08.776179+0000) 2022-09-10T14:22:09.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:09.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:09.394461+0000) 2022-09-10T14:22:09.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:09 smithi110 ceph-mon[97633]: osdmap e1266: 12 total, 12 up, 12 in 2022-09-10T14:22:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:09 smithi104 ceph-mon[97044]: osdmap e1266: 12 total, 12 up, 12 in 2022-09-10T14:22:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:09.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:09.776404+0000) 2022-09-10T14:22:10.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:10.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:10.394642+0000) 2022-09-10T14:22:10.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:10 smithi110 ceph-mon[97633]: pgmap v18971: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:22:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:10 smithi104 ceph-mon[97044]: pgmap v18971: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:22:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:10.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:10.776641+0000) 2022-09-10T14:22:11.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:11.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:11.394847+0000) 2022-09-10T14:22:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:11.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:11.776871+0000) 2022-09-10T14:22:12.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:12.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:12.395087+0000) 2022-09-10T14:22:12.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:12 smithi110 ceph-mon[97633]: pgmap v18972: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:22:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:12 smithi104 ceph-mon[97044]: pgmap v18972: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:22:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:12.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:12.777092+0000) 2022-09-10T14:22:13.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:13.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:13.395403+0000) 2022-09-10T14:22:14.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:13.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:13.777345+0000) 2022-09-10T14:22:14.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:14.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:14.395592+0000) 2022-09-10T14:22:14.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:14 smithi110 ceph-mon[97633]: pgmap v18973: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:22:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:14 smithi104 ceph-mon[97044]: pgmap v18973: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:22:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:14.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:14.777507+0000) 2022-09-10T14:22:15.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:15.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:15.395813+0000) 2022-09-10T14:22:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:15.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:15.777777+0000) 2022-09-10T14:22:16.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:16.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:16.396075+0000) 2022-09-10T14:22:16.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:16 smithi110 ceph-mon[97633]: pgmap v18974: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:22:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:16 smithi110 ceph-mon[97633]: osdmap e1267: 12 total, 12 up, 12 in 2022-09-10T14:22:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:16 smithi104 ceph-mon[97044]: pgmap v18974: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:22:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:16 smithi104 ceph-mon[97044]: osdmap e1267: 12 total, 12 up, 12 in 2022-09-10T14:22:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:16.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:16.778015+0000) 2022-09-10T14:22:17.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:17.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:17.396338+0000) 2022-09-10T14:22:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:17.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:17.778259+0000) 2022-09-10T14:22:18.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:18.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:18.396576+0000) 2022-09-10T14:22:18.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:18 smithi110 ceph-mon[97633]: pgmap v18976: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T14:22:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:18 smithi104 ceph-mon[97044]: pgmap v18976: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 239 B/s wr, 12 op/s 2022-09-10T14:22:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:18.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:18.778477+0000) 2022-09-10T14:22:19.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:19.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:19.396752+0000) 2022-09-10T14:22:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:19.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:19.778736+0000) 2022-09-10T14:22:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:22:19.881+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:22:20.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:20.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:20.396979+0000) 2022-09-10T14:22:20.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:20 smithi110 ceph-mon[97633]: pgmap v18977: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:22:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:20 smithi104 ceph-mon[97044]: pgmap v18977: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:22:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:20.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:20.778974+0000) 2022-09-10T14:22:21.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:21.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:21.397240+0000) 2022-09-10T14:22:22.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:21.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:21.779188+0000) 2022-09-10T14:22:22.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:22.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:22.397488+0000) 2022-09-10T14:22:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:22 smithi110 ceph-mon[97633]: pgmap v18978: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:22:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:22 smithi104 ceph-mon[97044]: pgmap v18978: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:22:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:22.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:22.779408+0000) 2022-09-10T14:22:23.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:23.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:23.397702+0000) 2022-09-10T14:22:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:23.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:23.779684+0000) 2022-09-10T14:22:24.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:24.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:24.397935+0000) 2022-09-10T14:22:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:24 smithi110 ceph-mon[97633]: pgmap v18979: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:22:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:24 smithi104 ceph-mon[97044]: pgmap v18979: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:22:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:24.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:24.779859+0000) 2022-09-10T14:22:25.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:25.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:25.398188+0000) 2022-09-10T14:22:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:25.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:25.780034+0000) 2022-09-10T14:22:26.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:26.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:26.398426+0000) 2022-09-10T14:22:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:26 smithi110 ceph-mon[97633]: pgmap v18980: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:22:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:26 smithi104 ceph-mon[97044]: pgmap v18980: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:22:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:26.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:26.780273+0000) 2022-09-10T14:22:27.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:27.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:27.398617+0000) 2022-09-10T14:22:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:27.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:27.780533+0000) 2022-09-10T14:22:28.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:28.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:28.398783+0000) 2022-09-10T14:22:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:28 smithi110 ceph-mon[97633]: pgmap v18981: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:22:29.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:28 smithi104 ceph-mon[97044]: pgmap v18981: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:22:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:28.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:28.780740+0000) 2022-09-10T14:22:29.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:29.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:29.398955+0000) 2022-09-10T14:22:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:29.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:29.780981+0000) 2022-09-10T14:22:30.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:30.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:30.399217+0000) 2022-09-10T14:22:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:30 smithi110 ceph-mon[97633]: pgmap v18982: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:30 smithi104 ceph-mon[97044]: pgmap v18982: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:30.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:30.781279+0000) 2022-09-10T14:22:31.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:31.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:31.399479+0000) 2022-09-10T14:22:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:31.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:31.781493+0000) 2022-09-10T14:22:32.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:32.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:32.399736+0000) 2022-09-10T14:22:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:32 smithi104 ceph-mon[97044]: pgmap v18983: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:32.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:32.781717+0000) 2022-09-10T14:22:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:32 smithi110 ceph-mon[97633]: pgmap v18983: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:33.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:33.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:33.399997+0000) 2022-09-10T14:22:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:33.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:33.781948+0000) 2022-09-10T14:22:34.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:34.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:34.400198+0000) 2022-09-10T14:22:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:34 smithi104 ceph-mon[97044]: pgmap v18984: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:35.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:34.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:34.782200+0000) 2022-09-10T14:22:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:22:34.896+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:22:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:34 smithi110 ceph-mon[97633]: pgmap v18984: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:35.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:35.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:35.400398+0000) 2022-09-10T14:22:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:35.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:35.782408+0000) 2022-09-10T14:22:36.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:36.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:36.400644+0000) 2022-09-10T14:22:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:36 smithi104 ceph-mon[97044]: pgmap v18985: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:36.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:36.782674+0000) 2022-09-10T14:22:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:36 smithi110 ceph-mon[97633]: pgmap v18985: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:37.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:37.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:37.400850+0000) 2022-09-10T14:22:38.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:37.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:37.782970+0000) 2022-09-10T14:22:38.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:38.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:38.401041+0000) 2022-09-10T14:22:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:38 smithi104 ceph-mon[97044]: pgmap v18986: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:38.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:38.783264+0000) 2022-09-10T14:22:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:38 smithi110 ceph-mon[97633]: pgmap v18986: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:39.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:39.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:39.401288+0000) 2022-09-10T14:22:40.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:39.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:39.783503+0000) 2022-09-10T14:22:40.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:40.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:40.401528+0000) 2022-09-10T14:22:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:40 smithi104 ceph-mon[97044]: pgmap v18987: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:40.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:40.783678+0000) 2022-09-10T14:22:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:40 smithi110 ceph-mon[97633]: pgmap v18987: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:41.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:41.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:41.401730+0000) 2022-09-10T14:22:42.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:41.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:41.783879+0000) 2022-09-10T14:22:42.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:42.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:42.401994+0000) 2022-09-10T14:22:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:42 smithi104 ceph-mon[97044]: pgmap v18988: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:42.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:42.784147+0000) 2022-09-10T14:22:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:42 smithi110 ceph-mon[97633]: pgmap v18988: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:43.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:43.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:43.402253+0000) 2022-09-10T14:22:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:43.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:43.784364+0000) 2022-09-10T14:22:44.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:44.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:44.402528+0000) 2022-09-10T14:22:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:44 smithi104 ceph-mon[97044]: pgmap v18989: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:44.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:44.784618+0000) 2022-09-10T14:22:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:44 smithi110 ceph-mon[97633]: pgmap v18989: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:45.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:45.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:45.402807+0000) 2022-09-10T14:22:46.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:45.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:45.784874+0000) 2022-09-10T14:22:46.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:46.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:46.403087+0000) 2022-09-10T14:22:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:46 smithi104 ceph-mon[97044]: pgmap v18990: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:22:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:22:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:22:47.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:22:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:46.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:46.785107+0000) 2022-09-10T14:22:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:46 smithi110 ceph-mon[97633]: pgmap v18990: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:22:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:22:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:22:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:22:47.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:47.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:47.403349+0000) 2022-09-10T14:22:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:47.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:47.785403+0000) 2022-09-10T14:22:48.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:48.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:48.403584+0000) 2022-09-10T14:22:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:48 smithi104 ceph-mon[97044]: pgmap v18991: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:48.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:48.785630+0000) 2022-09-10T14:22:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:48 smithi110 ceph-mon[97633]: pgmap v18991: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:49.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:49.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:49.403780+0000) 2022-09-10T14:22:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:49.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:49.785819+0000) 2022-09-10T14:22:50.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:22:49.912+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:22:50.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:50.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:50.403968+0000) 2022-09-10T14:22:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:50 smithi104 ceph-mon[97044]: pgmap v18992: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:50.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:50.786068+0000) 2022-09-10T14:22:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:50 smithi110 ceph-mon[97633]: pgmap v18992: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:51.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:51.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:51.404207+0000) 2022-09-10T14:22:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:51.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:51.786305+0000) 2022-09-10T14:22:52.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:52.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:52.404437+0000) 2022-09-10T14:22:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:52 smithi104 ceph-mon[97044]: pgmap v18993: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:52.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:52.786580+0000) 2022-09-10T14:22:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:52 smithi110 ceph-mon[97633]: pgmap v18993: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:53.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:53.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:53.404635+0000) 2022-09-10T14:22:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:53.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:53.786716+0000) 2022-09-10T14:22:54.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:54.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:54.404867+0000) 2022-09-10T14:22:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:54 smithi104 ceph-mon[97044]: pgmap v18994: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:55.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:54.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:54.786918+0000) 2022-09-10T14:22:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:54 smithi110 ceph-mon[97633]: pgmap v18994: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:55.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:55.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:55.405109+0000) 2022-09-10T14:22:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:55.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:55.787180+0000) 2022-09-10T14:22:56.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:56.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:56.405278+0000) 2022-09-10T14:22:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:56 smithi104 ceph-mon[97044]: pgmap v18995: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:56.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:56.787429+0000) 2022-09-10T14:22:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:56 smithi110 ceph-mon[97633]: pgmap v18995: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:22:57.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:57.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:57.405474+0000) 2022-09-10T14:22:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:57.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:57.787694+0000) 2022-09-10T14:22:58.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:58.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:58.405705+0000) 2022-09-10T14:22:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:22:58 smithi104 ceph-mon[97044]: pgmap v18996: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:58.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:58.787960+0000) 2022-09-10T14:22:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:22:58 smithi110 ceph-mon[97633]: pgmap v18996: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:22:59.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:22:59.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:59.405963+0000) 2022-09-10T14:23:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:22:59.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:22:59.788195+0000) 2022-09-10T14:23:00.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:00.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:00.406152+0000) 2022-09-10T14:23:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:00 smithi104 ceph-mon[97044]: pgmap v18997: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:00.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:00.788412+0000) 2022-09-10T14:23:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:00 smithi110 ceph-mon[97633]: pgmap v18997: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:01.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:01.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:01.406375+0000) 2022-09-10T14:23:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:01.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:01.788666+0000) 2022-09-10T14:23:02.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:02.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:02.406630+0000) 2022-09-10T14:23:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:02 smithi104 ceph-mon[97044]: pgmap v18998: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:02.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:02.788922+0000) 2022-09-10T14:23:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:02 smithi110 ceph-mon[97633]: pgmap v18998: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:03.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:03.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:03.406869+0000) 2022-09-10T14:23:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:03.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:03.789070+0000) 2022-09-10T14:23:04.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:04.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:04.407125+0000) 2022-09-10T14:23:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:04 smithi104 ceph-mon[97044]: pgmap v18999: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:05.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:04.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:04.789245+0000) 2022-09-10T14:23:05.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:23:04.919+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:23:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:04 smithi110 ceph-mon[97633]: pgmap v18999: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:05.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:05.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:05.407349+0000) 2022-09-10T14:23:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:05.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:05.789419+0000) 2022-09-10T14:23:06.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:06.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:06.407631+0000) 2022-09-10T14:23:07.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:06 smithi104 ceph-mon[97044]: pgmap v19000: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:23:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:06.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:06.789627+0000) 2022-09-10T14:23:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:06 smithi110 ceph-mon[97633]: pgmap v19000: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:23:07.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:07.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:07.407873+0000) 2022-09-10T14:23:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:07.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:07.789890+0000) 2022-09-10T14:23:08.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:08.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:08.408143+0000) 2022-09-10T14:23:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:08 smithi104 ceph-mon[97044]: pgmap v19001: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:23:09.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:08.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:08.790126+0000) 2022-09-10T14:23:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:08 smithi110 ceph-mon[97633]: pgmap v19001: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:23:09.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:09.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:09.408392+0000) 2022-09-10T14:23:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:09 smithi104 ceph-mon[97044]: osdmap e1268: 12 total, 12 up, 12 in 2022-09-10T14:23:10.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:09.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:09.790252+0000) 2022-09-10T14:23:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:09 smithi110 ceph-mon[97633]: osdmap e1268: 12 total, 12 up, 12 in 2022-09-10T14:23:10.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:10.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:10.408657+0000) 2022-09-10T14:23:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:10 smithi104 ceph-mon[97044]: pgmap v19003: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:23:11.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:10.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:10.790466+0000) 2022-09-10T14:23:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:10 smithi110 ceph-mon[97633]: pgmap v19003: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:23:11.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:11.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:11.408844+0000) 2022-09-10T14:23:12.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:11.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:11.790653+0000) 2022-09-10T14:23:12.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:12.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:12.409115+0000) 2022-09-10T14:23:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:12 smithi104 ceph-mon[97044]: pgmap v19004: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:23:13.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:12.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:12.790893+0000) 2022-09-10T14:23:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:12 smithi110 ceph-mon[97633]: pgmap v19004: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:23:13.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:13.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:13.409296+0000) 2022-09-10T14:23:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:13.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:13.791080+0000) 2022-09-10T14:23:14.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:14.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:14.409521+0000) 2022-09-10T14:23:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:14 smithi104 ceph-mon[97044]: pgmap v19005: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:23:15.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:14.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:14.791321+0000) 2022-09-10T14:23:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:14 smithi110 ceph-mon[97633]: pgmap v19005: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:23:15.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:15.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:15.409738+0000) 2022-09-10T14:23:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:15.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:15.791524+0000) 2022-09-10T14:23:16.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:16.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:16.409965+0000) 2022-09-10T14:23:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:16 smithi104 ceph-mon[97044]: pgmap v19006: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:23:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:16 smithi104 ceph-mon[97044]: osdmap e1269: 12 total, 12 up, 12 in 2022-09-10T14:23:17.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:16.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:16.791761+0000) 2022-09-10T14:23:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:16 smithi110 ceph-mon[97633]: pgmap v19006: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:23:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:16 smithi110 ceph-mon[97633]: osdmap e1269: 12 total, 12 up, 12 in 2022-09-10T14:23:17.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:17.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:17.410224+0000) 2022-09-10T14:23:18.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:17.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:17.791957+0000) 2022-09-10T14:23:18.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:18.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:18.410417+0000) 2022-09-10T14:23:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:18 smithi104 ceph-mon[97044]: pgmap v19008: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T14:23:19.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:18.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:18.792198+0000) 2022-09-10T14:23:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:18 smithi110 ceph-mon[97633]: pgmap v19008: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 241 B/s wr, 12 op/s 2022-09-10T14:23:19.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:19.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:19.410592+0000) 2022-09-10T14:23:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:19.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:19.792423+0000) 2022-09-10T14:23:20.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:23:19.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:23:20.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:20.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:20.410779+0000) 2022-09-10T14:23:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:20 smithi104 ceph-mon[97044]: pgmap v19009: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:23:21.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:20.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:20.792656+0000) 2022-09-10T14:23:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:20 smithi110 ceph-mon[97633]: pgmap v19009: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:23:21.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:21.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:21.411005+0000) 2022-09-10T14:23:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:21.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:21.792906+0000) 2022-09-10T14:23:22.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:22.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:22.411200+0000) 2022-09-10T14:23:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:22 smithi104 ceph-mon[97044]: pgmap v19010: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:23:23.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:22.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:22.793130+0000) 2022-09-10T14:23:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:22 smithi110 ceph-mon[97633]: pgmap v19010: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:23:23.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:23.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:23.411389+0000) 2022-09-10T14:23:24.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:23.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:23.793367+0000) 2022-09-10T14:23:24.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:24.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:24.411629+0000) 2022-09-10T14:23:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:24 smithi104 ceph-mon[97044]: pgmap v19011: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:23:25.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:24.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:24.793490+0000) 2022-09-10T14:23:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:24 smithi110 ceph-mon[97633]: pgmap v19011: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:23:25.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:25.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:25.411911+0000) 2022-09-10T14:23:26.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:25.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:25.793756+0000) 2022-09-10T14:23:26.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:26.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:26.412226+0000) 2022-09-10T14:23:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:26 smithi104 ceph-mon[97044]: pgmap v19012: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:23:27.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:26.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:26.793996+0000) 2022-09-10T14:23:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:26 smithi110 ceph-mon[97633]: pgmap v19012: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:23:27.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:27.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:27.412429+0000) 2022-09-10T14:23:28.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:27.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:27.794251+0000) 2022-09-10T14:23:28.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:28.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:28.412678+0000) 2022-09-10T14:23:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:28 smithi104 ceph-mon[97044]: pgmap v19013: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:23:29.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:28.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:28.794484+0000) 2022-09-10T14:23:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:28 smithi110 ceph-mon[97633]: pgmap v19013: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:23:29.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:29.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:29.412874+0000) 2022-09-10T14:23:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:29.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:29.794723+0000) 2022-09-10T14:23:30.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:30.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:30.413125+0000) 2022-09-10T14:23:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:30 smithi104 ceph-mon[97044]: pgmap v19014: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:31.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:30.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:30.794965+0000) 2022-09-10T14:23:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:30 smithi110 ceph-mon[97633]: pgmap v19014: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:31.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:31.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:31.413316+0000) 2022-09-10T14:23:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:31.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:31.795178+0000) 2022-09-10T14:23:32.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:32.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:32.413540+0000) 2022-09-10T14:23:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:32 smithi104 ceph-mon[97044]: pgmap v19015: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:33.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:32.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:32.795383+0000) 2022-09-10T14:23:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:32 smithi110 ceph-mon[97633]: pgmap v19015: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:33.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:33.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:33.413755+0000) 2022-09-10T14:23:34.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:33.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:33.795659+0000) 2022-09-10T14:23:34.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:34.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:34.413980+0000) 2022-09-10T14:23:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:34 smithi104 ceph-mon[97044]: pgmap v19016: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:34.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:34.795875+0000) 2022-09-10T14:23:35.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:23:34.942+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:23:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:34 smithi110 ceph-mon[97633]: pgmap v19016: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:35.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:35.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:35.414156+0000) 2022-09-10T14:23:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:35.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:35.796146+0000) 2022-09-10T14:23:36.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:36.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:36.414413+0000) 2022-09-10T14:23:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:36 smithi104 ceph-mon[97044]: pgmap v19017: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:37.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:36.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:36.796389+0000) 2022-09-10T14:23:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:36 smithi110 ceph-mon[97633]: pgmap v19017: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:37.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:37.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:37.414649+0000) 2022-09-10T14:23:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:37.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:37.796632+0000) 2022-09-10T14:23:38.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:38.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:38.414867+0000) 2022-09-10T14:23:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:38 smithi104 ceph-mon[97044]: pgmap v19018: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:39.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:38.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:38.796775+0000) 2022-09-10T14:23:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:38 smithi110 ceph-mon[97633]: pgmap v19018: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:39.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:39.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:39.415049+0000) 2022-09-10T14:23:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:39.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:39.797001+0000) 2022-09-10T14:23:40.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:40.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:40.415273+0000) 2022-09-10T14:23:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:40 smithi104 ceph-mon[97044]: pgmap v19019: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:41.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:40.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:40.797213+0000) 2022-09-10T14:23:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:40 smithi110 ceph-mon[97633]: pgmap v19019: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:41.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:41.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:41.415556+0000) 2022-09-10T14:23:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:41.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:41.797471+0000) 2022-09-10T14:23:42.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:42.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:42.415782+0000) 2022-09-10T14:23:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:42 smithi104 ceph-mon[97044]: pgmap v19020: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:43.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:42.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:42.797674+0000) 2022-09-10T14:23:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:42 smithi110 ceph-mon[97633]: pgmap v19020: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:43.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:43.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:43.416031+0000) 2022-09-10T14:23:44.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:43.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:43.797854+0000) 2022-09-10T14:23:44.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:44.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:44.416161+0000) 2022-09-10T14:23:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:44 smithi104 ceph-mon[97044]: pgmap v19021: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:45.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:44.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:44.798115+0000) 2022-09-10T14:23:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:44 smithi110 ceph-mon[97633]: pgmap v19021: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:45.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:45.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:45.416428+0000) 2022-09-10T14:23:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:45.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:45.798280+0000) 2022-09-10T14:23:46.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:46.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:46.416729+0000) 2022-09-10T14:23:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:46 smithi104 ceph-mon[97044]: pgmap v19022: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:23:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:23:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:23:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:23:47.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:46.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:46.798501+0000) 2022-09-10T14:23:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:46 smithi110 ceph-mon[97633]: pgmap v19022: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:23:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:23:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:23:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:23:47.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:47.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:47.416885+0000) 2022-09-10T14:23:48.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:47.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:47.798659+0000) 2022-09-10T14:23:48.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:48.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:48.417112+0000) 2022-09-10T14:23:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:48 smithi104 ceph-mon[97044]: pgmap v19023: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:49.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:48.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:48.798881+0000) 2022-09-10T14:23:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:48 smithi110 ceph-mon[97633]: pgmap v19023: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:49.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:49.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:49.417282+0000) 2022-09-10T14:23:50.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:49.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:49.799186+0000) 2022-09-10T14:23:50.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:23:49.953+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:23:50.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:50.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:50.417483+0000) 2022-09-10T14:23:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:50 smithi104 ceph-mon[97044]: pgmap v19024: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:51.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:50.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:50.799429+0000) 2022-09-10T14:23:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:50 smithi110 ceph-mon[97633]: pgmap v19024: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:51.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:51.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:51.417705+0000) 2022-09-10T14:23:52.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:51.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:51.799667+0000) 2022-09-10T14:23:52.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:52.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:52.417933+0000) 2022-09-10T14:23:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:52 smithi104 ceph-mon[97044]: pgmap v19025: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:53.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:52.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:52.799894+0000) 2022-09-10T14:23:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:52 smithi110 ceph-mon[97633]: pgmap v19025: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:53.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:53.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:53.418207+0000) 2022-09-10T14:23:54.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:53.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:53.800133+0000) 2022-09-10T14:23:54.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:54.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:54.418431+0000) 2022-09-10T14:23:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:54 smithi104 ceph-mon[97044]: pgmap v19026: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:55.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:54.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:54.800331+0000) 2022-09-10T14:23:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:54 smithi110 ceph-mon[97633]: pgmap v19026: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:55.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:55.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:55.418683+0000) 2022-09-10T14:23:56.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:55.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:55.800535+0000) 2022-09-10T14:23:56.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:56.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:56.418869+0000) 2022-09-10T14:23:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:56 smithi104 ceph-mon[97044]: pgmap v19027: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:57.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:56.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:56.800777+0000) 2022-09-10T14:23:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:56 smithi110 ceph-mon[97633]: pgmap v19027: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:23:57.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:57.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:57.419113+0000) 2022-09-10T14:23:58.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:57.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:57.801017+0000) 2022-09-10T14:23:58.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:58.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:58.419337+0000) 2022-09-10T14:23:59.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:58.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:58.801210+0000) 2022-09-10T14:23:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:23:58 smithi110 ceph-mon[97633]: pgmap v19028: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:23:58 smithi104 ceph-mon[97044]: pgmap v19028: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:23:59.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:23:59.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:59.419579+0000) 2022-09-10T14:24:00.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:23:59.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:23:59.801491+0000) 2022-09-10T14:24:00.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:00.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:00.419848+0000) 2022-09-10T14:24:01.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:00.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:00.801666+0000) 2022-09-10T14:24:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:00 smithi110 ceph-mon[97633]: pgmap v19029: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:00 smithi104 ceph-mon[97044]: pgmap v19029: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:01.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:01.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:01.420091+0000) 2022-09-10T14:24:02.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:01.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:01.801880+0000) 2022-09-10T14:24:02.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:02.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:02.420336+0000) 2022-09-10T14:24:03.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:02.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:02.802147+0000) 2022-09-10T14:24:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:02 smithi110 ceph-mon[97633]: pgmap v19030: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:02 smithi104 ceph-mon[97044]: pgmap v19030: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:03.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:03.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:03.420517+0000) 2022-09-10T14:24:04.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:03.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:03.802364+0000) 2022-09-10T14:24:04.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:04.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:04.420771+0000) 2022-09-10T14:24:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:04.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:04.802561+0000) 2022-09-10T14:24:05.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:24:04.966+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:24:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:04 smithi110 ceph-mon[97633]: pgmap v19031: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:04 smithi104 ceph-mon[97044]: pgmap v19031: 129 pgs: 129 active+clean; 9.8 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:05.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:05.420948+0000) 2022-09-10T14:24:06.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:05.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:05.802775+0000) 2022-09-10T14:24:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:06.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:06.421214+0000) 2022-09-10T14:24:07.054 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:06.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:06.803033+0000) 2022-09-10T14:24:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:06 smithi110 ceph-mon[97633]: pgmap v19032: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:24:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:06 smithi104 ceph-mon[97044]: pgmap v19032: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:24:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:07.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:07.421467+0000) 2022-09-10T14:24:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:07.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:07.803217+0000) 2022-09-10T14:24:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:08.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:08.421659+0000) 2022-09-10T14:24:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:08 smithi110 ceph-mon[97633]: pgmap v19033: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:24:09.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:08 smithi104 ceph-mon[97044]: pgmap v19033: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:24:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:08.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:08.803440+0000) 2022-09-10T14:24:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:09.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:09.421874+0000) 2022-09-10T14:24:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:09 smithi110 ceph-mon[97633]: osdmap e1270: 12 total, 12 up, 12 in 2022-09-10T14:24:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:09 smithi104 ceph-mon[97044]: osdmap e1270: 12 total, 12 up, 12 in 2022-09-10T14:24:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:09.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:09.803645+0000) 2022-09-10T14:24:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:10.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:10.422166+0000) 2022-09-10T14:24:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:10 smithi110 ceph-mon[97633]: pgmap v19035: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:24:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:10 smithi104 ceph-mon[97044]: pgmap v19035: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:24:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:10.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:10.803893+0000) 2022-09-10T14:24:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:11.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:11.422371+0000) 2022-09-10T14:24:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:11.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:11.804029+0000) 2022-09-10T14:24:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:12.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:12.422565+0000) 2022-09-10T14:24:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:12 smithi110 ceph-mon[97633]: pgmap v19036: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:24:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:12 smithi104 ceph-mon[97044]: pgmap v19036: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:24:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:12.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:12.804168+0000) 2022-09-10T14:24:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:13.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:13.422831+0000) 2022-09-10T14:24:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:13.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:13.804309+0000) 2022-09-10T14:24:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:14.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:14.423123+0000) 2022-09-10T14:24:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:14 smithi110 ceph-mon[97633]: pgmap v19037: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:24:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:14 smithi104 ceph-mon[97044]: pgmap v19037: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:24:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:14.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:14.804462+0000) 2022-09-10T14:24:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:15.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:15.423324+0000) 2022-09-10T14:24:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:15.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:15.804654+0000) 2022-09-10T14:24:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:16.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:16.423610+0000) 2022-09-10T14:24:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:16 smithi110 ceph-mon[97633]: pgmap v19038: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:24:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:16 smithi110 ceph-mon[97633]: osdmap e1271: 12 total, 12 up, 12 in 2022-09-10T14:24:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:16 smithi104 ceph-mon[97044]: pgmap v19038: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:24:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:16 smithi104 ceph-mon[97044]: osdmap e1271: 12 total, 12 up, 12 in 2022-09-10T14:24:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:16.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:16.804833+0000) 2022-09-10T14:24:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:17.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:17.423855+0000) 2022-09-10T14:24:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:17.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:17.805056+0000) 2022-09-10T14:24:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:18.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:18.424084+0000) 2022-09-10T14:24:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:18 smithi110 ceph-mon[97633]: pgmap v19040: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T14:24:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:18 smithi104 ceph-mon[97044]: pgmap v19040: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 243 B/s wr, 12 op/s 2022-09-10T14:24:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:18.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:18.805154+0000) 2022-09-10T14:24:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:19.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:19.424310+0000) 2022-09-10T14:24:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:19.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:19.805288+0000) 2022-09-10T14:24:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:24:19.979+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:24:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:20.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:20.424549+0000) 2022-09-10T14:24:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:20 smithi110 ceph-mon[97633]: pgmap v19041: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:24:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:20 smithi104 ceph-mon[97044]: pgmap v19041: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:24:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:20.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:20.805490+0000) 2022-09-10T14:24:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:21.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:21.424804+0000) 2022-09-10T14:24:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:21.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:21.805683+0000) 2022-09-10T14:24:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:22.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:22.425042+0000) 2022-09-10T14:24:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:22 smithi110 ceph-mon[97633]: pgmap v19042: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:24:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:22 smithi104 ceph-mon[97044]: pgmap v19042: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:24:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:22.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:22.805894+0000) 2022-09-10T14:24:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:23.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:23.425338+0000) 2022-09-10T14:24:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:23.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:23.806148+0000) 2022-09-10T14:24:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:24.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:24.425594+0000) 2022-09-10T14:24:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:24 smithi110 ceph-mon[97633]: pgmap v19043: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:24:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:24 smithi104 ceph-mon[97044]: pgmap v19043: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:24:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:24.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:24.806354+0000) 2022-09-10T14:24:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:25.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:25.425824+0000) 2022-09-10T14:24:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:25.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:25.806534+0000) 2022-09-10T14:24:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:26.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:26.426021+0000) 2022-09-10T14:24:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:26 smithi110 ceph-mon[97633]: pgmap v19044: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:24:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:26 smithi104 ceph-mon[97044]: pgmap v19044: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:24:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:26.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:26.806710+0000) 2022-09-10T14:24:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:27.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:27.426203+0000) 2022-09-10T14:24:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:27.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:27.806941+0000) 2022-09-10T14:24:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:28.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:28.426457+0000) 2022-09-10T14:24:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:28 smithi110 ceph-mon[97633]: pgmap v19045: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:24:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:28 smithi104 ceph-mon[97044]: pgmap v19045: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:24:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:28.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:28.807190+0000) 2022-09-10T14:24:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:29.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:29.426664+0000) 2022-09-10T14:24:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:29.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:29.807401+0000) 2022-09-10T14:24:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:30.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:30.426861+0000) 2022-09-10T14:24:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:30 smithi110 ceph-mon[97633]: pgmap v19046: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:30 smithi104 ceph-mon[97044]: pgmap v19046: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:30.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:30.807588+0000) 2022-09-10T14:24:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:31.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:31.427133+0000) 2022-09-10T14:24:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:31.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:31.807820+0000) 2022-09-10T14:24:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:32.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:32.427323+0000) 2022-09-10T14:24:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:32 smithi110 ceph-mon[97633]: pgmap v19047: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:32 smithi104 ceph-mon[97044]: pgmap v19047: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:32.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:32.808027+0000) 2022-09-10T14:24:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:33.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:33.427555+0000) 2022-09-10T14:24:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:33.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:33.808240+0000) 2022-09-10T14:24:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:34.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:34.427736+0000) 2022-09-10T14:24:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:34 smithi110 ceph-mon[97633]: pgmap v19048: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:34 smithi104 ceph-mon[97044]: pgmap v19048: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:34.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:34.808422+0000) 2022-09-10T14:24:35.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:24:34.994+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:24:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:35.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:35.427964+0000) 2022-09-10T14:24:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:35.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:35.808622+0000) 2022-09-10T14:24:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:36.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:36.428212+0000) 2022-09-10T14:24:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:36 smithi110 ceph-mon[97633]: pgmap v19049: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:36 smithi104 ceph-mon[97044]: pgmap v19049: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:36.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:36.808853+0000) 2022-09-10T14:24:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:37.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:37.428394+0000) 2022-09-10T14:24:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:37.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:37.809114+0000) 2022-09-10T14:24:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:38.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:38.428656+0000) 2022-09-10T14:24:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:38 smithi110 ceph-mon[97633]: pgmap v19050: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:38 smithi104 ceph-mon[97044]: pgmap v19050: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:38.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:38.809247+0000) 2022-09-10T14:24:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:39.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:39.428925+0000) 2022-09-10T14:24:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:39.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:39.809442+0000) 2022-09-10T14:24:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:40.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:40.429164+0000) 2022-09-10T14:24:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:40 smithi110 ceph-mon[97633]: pgmap v19051: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:40 smithi104 ceph-mon[97044]: pgmap v19051: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:40.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:40.809656+0000) 2022-09-10T14:24:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:41.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:41.429361+0000) 2022-09-10T14:24:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:41.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:41.809879+0000) 2022-09-10T14:24:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:42.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:42.429621+0000) 2022-09-10T14:24:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:42 smithi110 ceph-mon[97633]: pgmap v19052: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:42 smithi104 ceph-mon[97044]: pgmap v19052: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:42.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:42.810143+0000) 2022-09-10T14:24:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:43.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:43.429850+0000) 2022-09-10T14:24:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:43.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:43.810362+0000) 2022-09-10T14:24:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:44.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:44.430136+0000) 2022-09-10T14:24:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:44 smithi110 ceph-mon[97633]: pgmap v19053: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:44 smithi104 ceph-mon[97044]: pgmap v19053: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:44.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:44.810527+0000) 2022-09-10T14:24:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:45.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:45.430314+0000) 2022-09-10T14:24:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:45.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:45.810675+0000) 2022-09-10T14:24:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:46.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:46.430547+0000) 2022-09-10T14:24:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:46 smithi110 ceph-mon[97633]: pgmap v19054: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:24:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:24:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:24:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:24:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:46 smithi104 ceph-mon[97044]: pgmap v19054: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:24:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:24:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:24:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:24:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:46.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:46.810883+0000) 2022-09-10T14:24:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:47.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:47.430710+0000) 2022-09-10T14:24:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:47.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:47.811050+0000) 2022-09-10T14:24:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:48.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:48.430851+0000) 2022-09-10T14:24:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:48 smithi110 ceph-mon[97633]: pgmap v19055: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:48 smithi104 ceph-mon[97044]: pgmap v19055: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:48.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:48.811262+0000) 2022-09-10T14:24:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:49.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:49.431061+0000) 2022-09-10T14:24:50.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:24:49.998+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:24:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:49.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:49.811412+0000) 2022-09-10T14:24:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:50.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:50.431242+0000) 2022-09-10T14:24:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:50 smithi110 ceph-mon[97633]: pgmap v19056: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:50 smithi104 ceph-mon[97044]: pgmap v19056: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:50.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:50.811541+0000) 2022-09-10T14:24:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:51.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:51.431468+0000) 2022-09-10T14:24:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:51.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:51.811803+0000) 2022-09-10T14:24:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:52.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:52.431682+0000) 2022-09-10T14:24:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:52 smithi110 ceph-mon[97633]: pgmap v19057: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:52 smithi104 ceph-mon[97044]: pgmap v19057: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:52.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:52.812014+0000) 2022-09-10T14:24:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:53.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:53.431971+0000) 2022-09-10T14:24:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:53.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:53.812260+0000) 2022-09-10T14:24:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:54.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:54.432163+0000) 2022-09-10T14:24:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:54 smithi110 ceph-mon[97633]: pgmap v19058: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:54 smithi104 ceph-mon[97044]: pgmap v19058: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:54.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:54.812446+0000) 2022-09-10T14:24:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:55.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:55.432344+0000) 2022-09-10T14:24:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:55.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:55.812659+0000) 2022-09-10T14:24:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:56.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:56.432607+0000) 2022-09-10T14:24:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:56 smithi110 ceph-mon[97633]: pgmap v19059: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:56 smithi104 ceph-mon[97044]: pgmap v19059: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:24:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:56.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:56.812905+0000) 2022-09-10T14:24:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:57.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:57.432805+0000) 2022-09-10T14:24:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:57.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:57.813161+0000) 2022-09-10T14:24:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:58.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:58.433029+0000) 2022-09-10T14:24:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:24:58 smithi110 ceph-mon[97633]: pgmap v19060: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:24:58 smithi104 ceph-mon[97044]: pgmap v19060: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:24:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:58.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:58.813329+0000) 2022-09-10T14:24:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:24:59.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:59.433276+0000) 2022-09-10T14:25:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:24:59.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:24:59.813514+0000) 2022-09-10T14:25:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:00.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:00.433503+0000) 2022-09-10T14:25:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:00 smithi110 ceph-mon[97633]: pgmap v19061: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:00 smithi104 ceph-mon[97044]: pgmap v19061: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:00.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:00.813745+0000) 2022-09-10T14:25:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:01.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:01.433763+0000) 2022-09-10T14:25:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:01.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:01.814006+0000) 2022-09-10T14:25:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:02.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:02.433874+0000) 2022-09-10T14:25:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:02 smithi110 ceph-mon[97633]: pgmap v19062: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:02 smithi104 ceph-mon[97044]: pgmap v19062: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:02.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:02.814256+0000) 2022-09-10T14:25:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:03.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:03.434128+0000) 2022-09-10T14:25:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:03.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:03.814440+0000) 2022-09-10T14:25:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:04.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:04.434400+0000) 2022-09-10T14:25:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:04 smithi110 ceph-mon[97633]: pgmap v19063: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:04 smithi104 ceph-mon[97044]: pgmap v19063: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:04.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:04.814654+0000) 2022-09-10T14:25:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:25:05.011+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:25:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:05.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:05.434641+0000) 2022-09-10T14:25:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:05.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:05.814866+0000) 2022-09-10T14:25:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:06.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:06.434878+0000) 2022-09-10T14:25:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:06 smithi110 ceph-mon[97633]: pgmap v19064: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:25:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:06 smithi104 ceph-mon[97044]: pgmap v19064: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:25:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:06.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:06.815113+0000) 2022-09-10T14:25:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:07.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:07.435142+0000) 2022-09-10T14:25:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:07.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:07.815288+0000) 2022-09-10T14:25:08.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:08.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:08.435314+0000) 2022-09-10T14:25:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:08 smithi104 ceph-mon[97044]: pgmap v19065: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:25:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:08 smithi104 ceph-mon[97044]: osdmap e1272: 12 total, 12 up, 12 in 2022-09-10T14:25:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:08.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:08.815433+0000) 2022-09-10T14:25:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:08 smithi110 ceph-mon[97633]: pgmap v19065: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:25:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:08 smithi110 ceph-mon[97633]: osdmap e1272: 12 total, 12 up, 12 in 2022-09-10T14:25:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:09.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:09.435503+0000) 2022-09-10T14:25:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:09.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:09.815691+0000) 2022-09-10T14:25:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:10.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:10.435694+0000) 2022-09-10T14:25:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:10 smithi104 ceph-mon[97044]: pgmap v19067: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:25:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:10.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:10.815927+0000) 2022-09-10T14:25:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:10 smithi110 ceph-mon[97633]: pgmap v19067: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:25:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:11.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:11.435940+0000) 2022-09-10T14:25:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:11.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:11.816140+0000) 2022-09-10T14:25:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:12.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:12.436199+0000) 2022-09-10T14:25:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:12 smithi104 ceph-mon[97044]: pgmap v19068: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:25:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:12.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:12.816293+0000) 2022-09-10T14:25:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:12 smithi110 ceph-mon[97633]: pgmap v19068: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:25:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:13.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:13.436423+0000) 2022-09-10T14:25:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:13.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:13.816500+0000) 2022-09-10T14:25:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:14.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:14.436653+0000) 2022-09-10T14:25:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:14 smithi104 ceph-mon[97044]: pgmap v19069: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:25:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:14.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:14.816741+0000) 2022-09-10T14:25:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:14 smithi110 ceph-mon[97633]: pgmap v19069: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:25:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:15.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:15.436798+0000) 2022-09-10T14:25:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:15.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:15.817003+0000) 2022-09-10T14:25:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:16.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:16.437013+0000) 2022-09-10T14:25:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:16 smithi104 ceph-mon[97044]: pgmap v19070: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:25:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:16 smithi104 ceph-mon[97044]: osdmap e1273: 12 total, 12 up, 12 in 2022-09-10T14:25:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:16.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:16.817221+0000) 2022-09-10T14:25:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:16 smithi110 ceph-mon[97633]: pgmap v19070: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:25:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:16 smithi110 ceph-mon[97633]: osdmap e1273: 12 total, 12 up, 12 in 2022-09-10T14:25:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:17.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:17.437202+0000) 2022-09-10T14:25:18.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:17 smithi104 ceph-mon[97044]: pgmap v19072: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T14:25:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:17.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:17.817463+0000) 2022-09-10T14:25:18.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:17 smithi110 ceph-mon[97633]: pgmap v19072: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2022-09-10T14:25:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:18.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:18.437438+0000) 2022-09-10T14:25:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:18.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:18.817728+0000) 2022-09-10T14:25:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:19.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:19.437674+0000) 2022-09-10T14:25:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:19.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:19.817967+0000) 2022-09-10T14:25:20.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:25:20.015+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:25:20.546 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:20 smithi104 ceph-mon[97044]: pgmap v19073: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:25:20.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:20 smithi110 ceph-mon[97633]: pgmap v19073: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:25:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:20.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:20.437896+0000) 2022-09-10T14:25:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:20.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:20.818195+0000) 2022-09-10T14:25:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:21.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:21.438165+0000) 2022-09-10T14:25:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:21.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:21.818410+0000) 2022-09-10T14:25:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:22.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:22.438330+0000) 2022-09-10T14:25:22.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:22 smithi110 ceph-mon[97633]: pgmap v19074: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:25:23.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:22 smithi104 ceph-mon[97044]: pgmap v19074: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:25:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:22.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:22.818623+0000) 2022-09-10T14:25:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:23.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:23.438586+0000) 2022-09-10T14:25:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:23.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:23.818887+0000) 2022-09-10T14:25:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:24.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:24.438806+0000) 2022-09-10T14:25:24.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:24 smithi110 ceph-mon[97633]: pgmap v19075: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:25:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:24 smithi104 ceph-mon[97044]: pgmap v19075: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:25:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:24.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:24.819136+0000) 2022-09-10T14:25:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:25.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:25.439030+0000) 2022-09-10T14:25:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:25.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:25.819314+0000) 2022-09-10T14:25:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:26.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:26.439323+0000) 2022-09-10T14:25:26.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:26 smithi110 ceph-mon[97633]: pgmap v19076: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:25:27.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:26 smithi104 ceph-mon[97044]: pgmap v19076: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:25:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:26.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:26.819548+0000) 2022-09-10T14:25:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:27.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:27.439538+0000) 2022-09-10T14:25:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:27.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:27.819745+0000) 2022-09-10T14:25:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:28.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:28.439689+0000) 2022-09-10T14:25:28.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:28 smithi110 ceph-mon[97633]: pgmap v19077: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:25:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:28 smithi104 ceph-mon[97044]: pgmap v19077: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:25:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:28.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:28.819955+0000) 2022-09-10T14:25:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:29.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:29.439853+0000) 2022-09-10T14:25:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:29.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:29.820185+0000) 2022-09-10T14:25:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:30.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:30.440048+0000) 2022-09-10T14:25:30.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:30 smithi110 ceph-mon[97633]: pgmap v19078: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:30 smithi104 ceph-mon[97044]: pgmap v19078: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:30.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:30.820447+0000) 2022-09-10T14:25:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:31.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:31.440305+0000) 2022-09-10T14:25:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:31.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:31.820598+0000) 2022-09-10T14:25:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:32.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:32.440584+0000) 2022-09-10T14:25:32.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:32 smithi110 ceph-mon[97633]: pgmap v19079: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:32 smithi104 ceph-mon[97044]: pgmap v19079: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:32.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:32.820785+0000) 2022-09-10T14:25:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:33.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:33.440836+0000) 2022-09-10T14:25:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:33.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:33.820988+0000) 2022-09-10T14:25:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:34.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:34.441068+0000) 2022-09-10T14:25:34.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:34 smithi110 ceph-mon[97633]: pgmap v19080: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:34 smithi104 ceph-mon[97044]: pgmap v19080: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:25:35.029+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:25:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:34.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:34.821160+0000) 2022-09-10T14:25:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:35.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:35.441319+0000) 2022-09-10T14:25:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:35.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:35.821324+0000) 2022-09-10T14:25:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:36.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:36.441609+0000) 2022-09-10T14:25:36.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:36 smithi110 ceph-mon[97633]: pgmap v19081: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:36 smithi104 ceph-mon[97044]: pgmap v19081: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:36.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:36.821506+0000) 2022-09-10T14:25:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:37.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:37.441854+0000) 2022-09-10T14:25:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:37.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:37.821643+0000) 2022-09-10T14:25:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:38.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:38.442070+0000) 2022-09-10T14:25:38.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:38 smithi110 ceph-mon[97633]: pgmap v19082: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:38 smithi104 ceph-mon[97044]: pgmap v19082: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:38.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:38.821896+0000) 2022-09-10T14:25:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:39.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:39.442261+0000) 2022-09-10T14:25:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:39.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:39.822138+0000) 2022-09-10T14:25:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:40.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:40.442517+0000) 2022-09-10T14:25:40.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:40 smithi110 ceph-mon[97633]: pgmap v19083: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:40 smithi104 ceph-mon[97044]: pgmap v19083: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:40.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:40.822320+0000) 2022-09-10T14:25:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:41.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:41.442673+0000) 2022-09-10T14:25:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:41.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:41.822461+0000) 2022-09-10T14:25:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:42.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:42.442822+0000) 2022-09-10T14:25:42.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:42 smithi110 ceph-mon[97633]: pgmap v19084: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:42 smithi104 ceph-mon[97044]: pgmap v19084: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:42.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:42.822639+0000) 2022-09-10T14:25:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:43.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:43.443081+0000) 2022-09-10T14:25:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:43.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:43.822843+0000) 2022-09-10T14:25:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:44.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:44.443342+0000) 2022-09-10T14:25:44.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:44 smithi110 ceph-mon[97633]: pgmap v19085: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:44 smithi104 ceph-mon[97044]: pgmap v19085: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:44.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:44.823032+0000) 2022-09-10T14:25:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:45.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:45.443561+0000) 2022-09-10T14:25:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:45.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:45.823178+0000) 2022-09-10T14:25:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:46.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:46.443760+0000) 2022-09-10T14:25:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:46 smithi110 ceph-mon[97633]: pgmap v19086: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:25:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:46 smithi104 ceph-mon[97044]: pgmap v19086: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:25:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:25:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:25:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:25:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:46.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:46.823362+0000) 2022-09-10T14:25:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:47.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:47.443931+0000) 2022-09-10T14:25:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:47.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:47.823516+0000) 2022-09-10T14:25:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:48.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:48.444091+0000) 2022-09-10T14:25:48.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:48 smithi110 ceph-mon[97633]: pgmap v19087: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:48 smithi104 ceph-mon[97044]: pgmap v19087: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:48.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:48.823637+0000) 2022-09-10T14:25:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:49.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:49.444335+0000) 2022-09-10T14:25:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:49.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:49.823846+0000) 2022-09-10T14:25:50.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:25:50.045+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:25:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:50.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:50.444506+0000) 2022-09-10T14:25:50.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:50 smithi110 ceph-mon[97633]: pgmap v19088: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:50 smithi104 ceph-mon[97044]: pgmap v19088: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:50.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:50.824133+0000) 2022-09-10T14:25:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:51.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:51.444728+0000) 2022-09-10T14:25:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:51.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:51.824293+0000) 2022-09-10T14:25:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:52.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:52.444993+0000) 2022-09-10T14:25:52.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:52 smithi110 ceph-mon[97633]: pgmap v19089: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:52 smithi104 ceph-mon[97044]: pgmap v19089: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:52.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:52.824447+0000) 2022-09-10T14:25:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:53.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:53.445227+0000) 2022-09-10T14:25:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:53.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:53.824689+0000) 2022-09-10T14:25:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:54.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:54.445482+0000) 2022-09-10T14:25:54.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:54 smithi110 ceph-mon[97633]: pgmap v19090: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:54 smithi104 ceph-mon[97044]: pgmap v19090: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:54.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:54.824885+0000) 2022-09-10T14:25:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:55.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:55.445694+0000) 2022-09-10T14:25:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:55.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:55.825154+0000) 2022-09-10T14:25:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:56.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:56.445945+0000) 2022-09-10T14:25:56.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:56 smithi110 ceph-mon[97633]: pgmap v19091: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:56 smithi104 ceph-mon[97044]: pgmap v19091: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:25:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:56.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:56.825403+0000) 2022-09-10T14:25:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:57.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:57.446133+0000) 2022-09-10T14:25:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:57.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:57.825638+0000) 2022-09-10T14:25:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:58.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:58.446343+0000) 2022-09-10T14:25:58.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:25:58 smithi110 ceph-mon[97633]: pgmap v19092: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:25:58 smithi104 ceph-mon[97044]: pgmap v19092: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:25:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:58.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:58.825913+0000) 2022-09-10T14:25:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:25:59.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:59.446523+0000) 2022-09-10T14:26:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:25:59.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:25:59.826180+0000) 2022-09-10T14:26:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:00.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:00.446712+0000) 2022-09-10T14:26:00.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:00 smithi110 ceph-mon[97633]: pgmap v19093: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:00 smithi104 ceph-mon[97044]: pgmap v19093: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:00.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:00.826345+0000) 2022-09-10T14:26:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:01.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:01.446930+0000) 2022-09-10T14:26:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:01.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:01.826574+0000) 2022-09-10T14:26:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:02.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:02.447081+0000) 2022-09-10T14:26:02.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:02 smithi110 ceph-mon[97633]: pgmap v19094: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:02 smithi104 ceph-mon[97044]: pgmap v19094: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:02.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:02.826824+0000) 2022-09-10T14:26:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:03.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:03.447326+0000) 2022-09-10T14:26:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:03.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:03.827008+0000) 2022-09-10T14:26:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:04.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:04.447559+0000) 2022-09-10T14:26:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:04 smithi104 ceph-mon[97044]: pgmap v19095: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:04 smithi110 ceph-mon[97633]: pgmap v19095: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:05.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:26:05.047+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:26:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:04.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:04.827191+0000) 2022-09-10T14:26:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:05.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:05.447783+0000) 2022-09-10T14:26:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:05.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:05.827410+0000) 2022-09-10T14:26:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:06.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:06.448031+0000) 2022-09-10T14:26:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:06 smithi104 ceph-mon[97044]: pgmap v19096: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:26:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:06 smithi110 ceph-mon[97633]: pgmap v19096: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:26:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:06.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:06.827642+0000) 2022-09-10T14:26:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:07.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:07.448292+0000) 2022-09-10T14:26:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:07.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:07.827900+0000) 2022-09-10T14:26:08.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:08.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:08.448540+0000) 2022-09-10T14:26:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:08 smithi104 ceph-mon[97044]: pgmap v19097: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:26:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:08 smithi110 ceph-mon[97633]: pgmap v19097: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:26:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:08.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:08.828198+0000) 2022-09-10T14:26:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:09.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:09.448769+0000) 2022-09-10T14:26:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:09 smithi104 ceph-mon[97044]: osdmap e1274: 12 total, 12 up, 12 in 2022-09-10T14:26:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:09 smithi110 ceph-mon[97633]: osdmap e1274: 12 total, 12 up, 12 in 2022-09-10T14:26:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:09.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:09.828357+0000) 2022-09-10T14:26:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:10.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:10.448959+0000) 2022-09-10T14:26:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:10 smithi104 ceph-mon[97044]: pgmap v19099: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:26:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:10 smithi110 ceph-mon[97633]: pgmap v19099: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:26:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:10.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:10.828586+0000) 2022-09-10T14:26:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:11.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:11.449224+0000) 2022-09-10T14:26:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:11.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:11.828844+0000) 2022-09-10T14:26:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:12.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:12.449458+0000) 2022-09-10T14:26:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:12 smithi104 ceph-mon[97044]: pgmap v19100: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:26:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:12 smithi110 ceph-mon[97633]: pgmap v19100: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:26:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:12.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:12.829139+0000) 2022-09-10T14:26:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:13.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:13.449734+0000) 2022-09-10T14:26:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:13.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:13.829296+0000) 2022-09-10T14:26:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:14.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:14.449932+0000) 2022-09-10T14:26:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:14 smithi104 ceph-mon[97044]: pgmap v19101: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:26:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:14 smithi110 ceph-mon[97633]: pgmap v19101: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:26:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:14.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:14.829455+0000) 2022-09-10T14:26:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:15.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:15.450181+0000) 2022-09-10T14:26:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:15.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:15.829685+0000) 2022-09-10T14:26:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:16.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:16.450416+0000) 2022-09-10T14:26:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:16 smithi104 ceph-mon[97044]: pgmap v19102: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:26:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:16 smithi104 ceph-mon[97044]: osdmap e1275: 12 total, 12 up, 12 in 2022-09-10T14:26:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:16 smithi110 ceph-mon[97633]: pgmap v19102: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:26:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:16 smithi110 ceph-mon[97633]: osdmap e1275: 12 total, 12 up, 12 in 2022-09-10T14:26:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:16.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:16.829941+0000) 2022-09-10T14:26:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:17.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:17.450651+0000) 2022-09-10T14:26:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:17.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:17.830199+0000) 2022-09-10T14:26:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:18.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:18.450901+0000) 2022-09-10T14:26:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:18 smithi104 ceph-mon[97044]: pgmap v19104: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:26:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:18 smithi110 ceph-mon[97633]: pgmap v19104: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:26:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:18.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:18.830421+0000) 2022-09-10T14:26:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:19.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:19.451081+0000) 2022-09-10T14:26:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:19.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:19.830620+0000) 2022-09-10T14:26:20.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:26:20.051+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:26:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:20.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:20.451316+0000) 2022-09-10T14:26:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:20 smithi104 ceph-mon[97044]: pgmap v19105: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:26:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:20 smithi110 ceph-mon[97633]: pgmap v19105: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:26:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:20.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:20.830853+0000) 2022-09-10T14:26:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:21.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:21.451472+0000) 2022-09-10T14:26:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:21.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:21.831072+0000) 2022-09-10T14:26:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:22.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:22.451694+0000) 2022-09-10T14:26:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:22 smithi104 ceph-mon[97044]: pgmap v19106: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:26:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:22 smithi110 ceph-mon[97633]: pgmap v19106: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:26:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:22.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:22.831306+0000) 2022-09-10T14:26:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:23.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:23.451964+0000) 2022-09-10T14:26:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:23.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:23.831550+0000) 2022-09-10T14:26:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:24.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:24.452213+0000) 2022-09-10T14:26:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:24 smithi104 ceph-mon[97044]: pgmap v19107: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:26:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:24 smithi110 ceph-mon[97633]: pgmap v19107: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:26:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:24.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:24.831757+0000) 2022-09-10T14:26:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:25.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:25.452469+0000) 2022-09-10T14:26:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:25.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:25.832005+0000) 2022-09-10T14:26:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:26.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:26.452753+0000) 2022-09-10T14:26:27.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:26 smithi104 ceph-mon[97044]: pgmap v19108: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T14:26:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:26 smithi110 ceph-mon[97633]: pgmap v19108: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T14:26:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:26.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:26.832221+0000) 2022-09-10T14:26:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:27.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:27.453046+0000) 2022-09-10T14:26:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:27.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:27.832471+0000) 2022-09-10T14:26:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:28.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:28.453259+0000) 2022-09-10T14:26:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:28 smithi104 ceph-mon[97044]: pgmap v19109: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T14:26:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:28 smithi110 ceph-mon[97633]: pgmap v19109: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T14:26:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:28.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:28.832728+0000) 2022-09-10T14:26:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:29.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:29.453433+0000) 2022-09-10T14:26:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:29.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:29.832965+0000) 2022-09-10T14:26:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:30.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:30.453592+0000) 2022-09-10T14:26:31.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:30 smithi104 ceph-mon[97044]: pgmap v19110: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:26:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:30 smithi110 ceph-mon[97633]: pgmap v19110: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:26:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:30.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:30.833180+0000) 2022-09-10T14:26:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:31.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:31.453806+0000) 2022-09-10T14:26:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:31.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:31.833358+0000) 2022-09-10T14:26:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:32.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:32.453968+0000) 2022-09-10T14:26:33.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:32 smithi104 ceph-mon[97044]: pgmap v19111: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:32 smithi110 ceph-mon[97633]: pgmap v19111: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:32.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:32.833520+0000) 2022-09-10T14:26:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:33.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:33.454202+0000) 2022-09-10T14:26:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:33.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:33.833720+0000) 2022-09-10T14:26:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:34.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:34.454390+0000) 2022-09-10T14:26:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:34 smithi104 ceph-mon[97044]: pgmap v19112: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:34 smithi110 ceph-mon[97633]: pgmap v19112: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:35.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:26:35.067+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:26:35.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:34.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:34.833955+0000) 2022-09-10T14:26:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:35.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:35.454605+0000) 2022-09-10T14:26:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:35.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:35.834145+0000) 2022-09-10T14:26:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:36.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:36.454841+0000) 2022-09-10T14:26:37.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:36 smithi104 ceph-mon[97044]: pgmap v19113: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:36 smithi110 ceph-mon[97633]: pgmap v19113: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:36.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:36.834359+0000) 2022-09-10T14:26:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:37.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:37.455019+0000) 2022-09-10T14:26:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:37.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:37.834552+0000) 2022-09-10T14:26:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:38.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:38.455262+0000) 2022-09-10T14:26:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:38 smithi104 ceph-mon[97044]: pgmap v19114: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:38 smithi110 ceph-mon[97633]: pgmap v19114: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:38.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:38.834777+0000) 2022-09-10T14:26:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:39.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:39.455414+0000) 2022-09-10T14:26:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:39.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:39.835022+0000) 2022-09-10T14:26:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:40.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:40.455605+0000) 2022-09-10T14:26:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:40 smithi104 ceph-mon[97044]: pgmap v19115: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:40 smithi110 ceph-mon[97633]: pgmap v19115: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:40.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:40.835225+0000) 2022-09-10T14:26:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:41.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:41.455828+0000) 2022-09-10T14:26:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:41.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:41.835368+0000) 2022-09-10T14:26:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:42.454+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:42.456072+0000) 2022-09-10T14:26:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:42 smithi104 ceph-mon[97044]: pgmap v19116: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:42 smithi110 ceph-mon[97633]: pgmap v19116: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:42.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:42.835580+0000) 2022-09-10T14:26:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:43.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:43.456304+0000) 2022-09-10T14:26:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:43.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:43.835760+0000) 2022-09-10T14:26:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:44.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:44.456517+0000) 2022-09-10T14:26:45.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:44 smithi104 ceph-mon[97044]: pgmap v19117: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:44 smithi110 ceph-mon[97633]: pgmap v19117: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:44.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:44.835998+0000) 2022-09-10T14:26:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:45.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:45.456669+0000) 2022-09-10T14:26:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:45.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:45.836167+0000) 2022-09-10T14:26:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:46.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:46.456918+0000) 2022-09-10T14:26:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:46 smithi104 ceph-mon[97044]: pgmap v19118: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:26:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:26:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:26:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:26:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:46 smithi110 ceph-mon[97633]: pgmap v19118: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:26:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:26:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:26:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:26:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:46.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:46.836414+0000) 2022-09-10T14:26:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:47.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:47.457176+0000) 2022-09-10T14:26:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:47.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:47.836603+0000) 2022-09-10T14:26:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:48.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:48.457358+0000) 2022-09-10T14:26:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:48 smithi104 ceph-mon[97044]: pgmap v19119: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:48 smithi110 ceph-mon[97633]: pgmap v19119: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:48.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:48.836781+0000) 2022-09-10T14:26:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:49.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:49.457536+0000) 2022-09-10T14:26:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:49.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:49.837047+0000) 2022-09-10T14:26:50.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:26:50.082+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:26:50.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:50.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:50.457783+0000) 2022-09-10T14:26:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:50 smithi104 ceph-mon[97044]: pgmap v19120: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:50 smithi110 ceph-mon[97633]: pgmap v19120: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:50.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:50.837305+0000) 2022-09-10T14:26:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:51.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:51.458022+0000) 2022-09-10T14:26:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:51.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:51.837517+0000) 2022-09-10T14:26:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:52.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:52.458215+0000) 2022-09-10T14:26:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:52 smithi104 ceph-mon[97044]: pgmap v19121: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:52 smithi110 ceph-mon[97633]: pgmap v19121: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:52.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:52.837740+0000) 2022-09-10T14:26:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:53.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:53.458459+0000) 2022-09-10T14:26:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:53.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:53.837963+0000) 2022-09-10T14:26:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:54.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:54.458609+0000) 2022-09-10T14:26:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:54 smithi104 ceph-mon[97044]: pgmap v19122: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:54 smithi110 ceph-mon[97633]: pgmap v19122: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:54.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:54.838212+0000) 2022-09-10T14:26:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:55.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:55.458785+0000) 2022-09-10T14:26:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:55.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:55.838450+0000) 2022-09-10T14:26:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:56.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:56.459030+0000) 2022-09-10T14:26:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:56 smithi104 ceph-mon[97044]: pgmap v19123: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:56 smithi110 ceph-mon[97633]: pgmap v19123: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:26:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:56.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:56.838683+0000) 2022-09-10T14:26:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:57.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:57.459155+0000) 2022-09-10T14:26:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:57.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:57.838906+0000) 2022-09-10T14:26:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:58.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:58.459346+0000) 2022-09-10T14:26:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:26:58 smithi104 ceph-mon[97044]: pgmap v19124: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:26:58 smithi110 ceph-mon[97633]: pgmap v19124: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:26:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:58.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:58.839187+0000) 2022-09-10T14:26:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:26:59.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:59.459530+0000) 2022-09-10T14:27:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:26:59.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:26:59.839397+0000) 2022-09-10T14:27:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:00.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:00.459739+0000) 2022-09-10T14:27:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:00 smithi104 ceph-mon[97044]: pgmap v19125: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:00 smithi110 ceph-mon[97633]: pgmap v19125: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:00.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:00.839595+0000) 2022-09-10T14:27:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:01.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:01.459912+0000) 2022-09-10T14:27:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:01.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:01.839859+0000) 2022-09-10T14:27:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:02.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:02.460124+0000) 2022-09-10T14:27:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:02 smithi104 ceph-mon[97044]: pgmap v19126: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:02 smithi110 ceph-mon[97633]: pgmap v19126: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:02.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:02.840001+0000) 2022-09-10T14:27:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:03.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:03.460381+0000) 2022-09-10T14:27:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:03.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:03.840250+0000) 2022-09-10T14:27:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:04.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:04.460550+0000) 2022-09-10T14:27:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:04 smithi104 ceph-mon[97044]: pgmap v19127: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:04 smithi110 ceph-mon[97633]: pgmap v19127: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:04.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:04.840490+0000) 2022-09-10T14:27:05.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:27:05.088+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:27:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:05.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:05.460728+0000) 2022-09-10T14:27:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:05.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:05.840719+0000) 2022-09-10T14:27:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:06.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:06.460952+0000) 2022-09-10T14:27:07.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:06 smithi104 ceph-mon[97044]: pgmap v19128: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:27:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:06 smithi110 ceph-mon[97633]: pgmap v19128: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:27:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:06.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:06.840965+0000) 2022-09-10T14:27:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:07.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:07.461237+0000) 2022-09-10T14:27:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:07.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:07.841158+0000) 2022-09-10T14:27:08.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:08.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:08.461411+0000) 2022-09-10T14:27:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:08 smithi104 ceph-mon[97044]: pgmap v19129: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:27:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:08 smithi110 ceph-mon[97633]: pgmap v19129: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:27:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:08.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:08.841308+0000) 2022-09-10T14:27:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:09.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:09.461589+0000) 2022-09-10T14:27:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:09 smithi104 ceph-mon[97044]: osdmap e1276: 12 total, 12 up, 12 in 2022-09-10T14:27:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:09 smithi110 ceph-mon[97633]: osdmap e1276: 12 total, 12 up, 12 in 2022-09-10T14:27:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:09.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:09.841536+0000) 2022-09-10T14:27:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:10.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:10.461768+0000) 2022-09-10T14:27:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:10 smithi104 ceph-mon[97044]: pgmap v19131: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:27:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:10 smithi110 ceph-mon[97633]: pgmap v19131: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:27:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:10.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:10.841717+0000) 2022-09-10T14:27:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:11.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:11.462029+0000) 2022-09-10T14:27:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:11.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:11.841963+0000) 2022-09-10T14:27:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:12.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:12.462274+0000) 2022-09-10T14:27:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:12 smithi104 ceph-mon[97044]: pgmap v19132: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:27:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:12 smithi110 ceph-mon[97633]: pgmap v19132: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:27:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:12.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:12.842138+0000) 2022-09-10T14:27:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:13.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:13.462534+0000) 2022-09-10T14:27:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:13.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:13.842341+0000) 2022-09-10T14:27:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:14.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:14.462740+0000) 2022-09-10T14:27:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:14 smithi104 ceph-mon[97044]: pgmap v19133: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:27:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:14 smithi110 ceph-mon[97633]: pgmap v19133: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:27:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:14.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:14.842555+0000) 2022-09-10T14:27:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:15.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:15.462971+0000) 2022-09-10T14:27:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:15.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:15.842722+0000) 2022-09-10T14:27:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:16.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:16.463279+0000) 2022-09-10T14:27:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:16 smithi104 ceph-mon[97044]: pgmap v19134: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:27:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:16 smithi104 ceph-mon[97044]: osdmap e1277: 12 total, 12 up, 12 in 2022-09-10T14:27:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:16 smithi110 ceph-mon[97633]: pgmap v19134: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:27:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:16 smithi110 ceph-mon[97633]: osdmap e1277: 12 total, 12 up, 12 in 2022-09-10T14:27:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:16.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:16.842927+0000) 2022-09-10T14:27:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:17.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:17.463488+0000) 2022-09-10T14:27:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:17.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:17.843163+0000) 2022-09-10T14:27:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:18.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:18.463670+0000) 2022-09-10T14:27:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:18 smithi104 ceph-mon[97044]: pgmap v19136: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T14:27:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:18 smithi110 ceph-mon[97633]: pgmap v19136: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 240 B/s wr, 12 op/s 2022-09-10T14:27:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:18.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:18.843389+0000) 2022-09-10T14:27:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:19.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:19.463876+0000) 2022-09-10T14:27:20.104 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:19.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:19.843646+0000) 2022-09-10T14:27:20.466 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:27:20.102+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:27:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:20.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:20.464137+0000) 2022-09-10T14:27:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:20 smithi104 ceph-mon[97044]: pgmap v19137: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:27:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:20 smithi110 ceph-mon[97633]: pgmap v19137: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:27:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:20.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:20.843884+0000) 2022-09-10T14:27:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:21.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:21.464387+0000) 2022-09-10T14:27:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:21.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:21.844185+0000) 2022-09-10T14:27:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:22.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:22.464621+0000) 2022-09-10T14:27:23.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:22 smithi104 ceph-mon[97044]: pgmap v19138: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:27:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:22 smithi110 ceph-mon[97633]: pgmap v19138: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:27:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:22.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:22.844411+0000) 2022-09-10T14:27:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:23.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:23.464839+0000) 2022-09-10T14:27:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:23.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:23.844578+0000) 2022-09-10T14:27:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:24.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:24.465089+0000) 2022-09-10T14:27:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:24 smithi110 ceph-mon[97633]: pgmap v19139: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:27:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:24 smithi104 ceph-mon[97044]: pgmap v19139: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:27:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:24.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:24.844720+0000) 2022-09-10T14:27:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:25.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:25.465312+0000) 2022-09-10T14:27:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:25.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:25.844924+0000) 2022-09-10T14:27:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:26.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:26.465486+0000) 2022-09-10T14:27:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:26 smithi110 ceph-mon[97633]: pgmap v19140: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:27:27.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:26 smithi104 ceph-mon[97044]: pgmap v19140: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:27:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:26.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:26.845203+0000) 2022-09-10T14:27:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:27.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:27.465714+0000) 2022-09-10T14:27:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:27.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:27.845399+0000) 2022-09-10T14:27:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:28.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:28.465870+0000) 2022-09-10T14:27:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:28 smithi110 ceph-mon[97633]: pgmap v19141: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:27:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:28 smithi104 ceph-mon[97044]: pgmap v19141: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:27:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:28.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:28.845641+0000) 2022-09-10T14:27:29.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:29.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:29.466091+0000) 2022-09-10T14:27:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:29.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:29.845895+0000) 2022-09-10T14:27:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:30.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:30.466313+0000) 2022-09-10T14:27:31.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:30 smithi110 ceph-mon[97633]: pgmap v19142: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:30 smithi104 ceph-mon[97044]: pgmap v19142: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:30.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:30.846159+0000) 2022-09-10T14:27:31.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:31.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:31.466485+0000) 2022-09-10T14:27:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:31.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:31.846390+0000) 2022-09-10T14:27:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:32.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:32.466724+0000) 2022-09-10T14:27:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:32 smithi110 ceph-mon[97633]: pgmap v19143: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:32 smithi104 ceph-mon[97044]: pgmap v19143: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:32.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:32.846590+0000) 2022-09-10T14:27:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:33.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:33.466928+0000) 2022-09-10T14:27:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:33.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:33.846816+0000) 2022-09-10T14:27:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:34.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:34.467174+0000) 2022-09-10T14:27:35.120 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:34.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:34.847072+0000) 2022-09-10T14:27:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:34 smithi110 ceph-mon[97633]: pgmap v19144: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:34 smithi104 ceph-mon[97044]: pgmap v19144: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:35.468 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:27:35.118+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:27:35.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:35.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:35.467396+0000) 2022-09-10T14:27:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:35.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:35.847324+0000) 2022-09-10T14:27:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:36.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:36.467675+0000) 2022-09-10T14:27:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:36 smithi110 ceph-mon[97633]: pgmap v19145: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:36 smithi104 ceph-mon[97044]: pgmap v19145: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:36.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:36.847576+0000) 2022-09-10T14:27:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:37.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:37.467885+0000) 2022-09-10T14:27:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:37.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:37.847760+0000) 2022-09-10T14:27:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:38.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:38.468167+0000) 2022-09-10T14:27:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:38 smithi110 ceph-mon[97633]: pgmap v19146: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:27:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:38 smithi104 ceph-mon[97044]: pgmap v19146: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:27:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:38.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:38.847923+0000) 2022-09-10T14:27:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:39.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:39.468338+0000) 2022-09-10T14:27:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:39.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:39.848198+0000) 2022-09-10T14:27:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:40.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:40.468497+0000) 2022-09-10T14:27:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:40 smithi110 ceph-mon[97633]: pgmap v19147: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:27:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:40 smithi104 ceph-mon[97044]: pgmap v19147: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:27:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:40.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:40.848445+0000) 2022-09-10T14:27:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:41.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:41.468724+0000) 2022-09-10T14:27:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:41.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:41.848708+0000) 2022-09-10T14:27:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:42.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:42.468955+0000) 2022-09-10T14:27:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:42 smithi110 ceph-mon[97633]: pgmap v19148: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:42 smithi104 ceph-mon[97044]: pgmap v19148: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:42.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:42.848924+0000) 2022-09-10T14:27:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:43.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:43.469200+0000) 2022-09-10T14:27:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:43.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:43.849200+0000) 2022-09-10T14:27:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:44.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:44.469431+0000) 2022-09-10T14:27:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:44 smithi110 ceph-mon[97633]: pgmap v19149: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:27:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:44 smithi104 ceph-mon[97044]: pgmap v19149: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:27:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:44.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:44.849452+0000) 2022-09-10T14:27:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:45.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:45.469572+0000) 2022-09-10T14:27:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:45.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:45.849683+0000) 2022-09-10T14:27:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:46.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:46.469808+0000) 2022-09-10T14:27:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:46 smithi110 ceph-mon[97633]: pgmap v19150: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:27:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:27:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:27:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:46 smithi104 ceph-mon[97044]: pgmap v19150: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:27:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:27:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:27:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:27:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:46.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:46.849909+0000) 2022-09-10T14:27:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:47.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:47.470053+0000) 2022-09-10T14:27:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:47.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:47.850158+0000) 2022-09-10T14:27:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:48.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:48.470288+0000) 2022-09-10T14:27:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:48 smithi110 ceph-mon[97633]: pgmap v19151: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:48 smithi104 ceph-mon[97044]: pgmap v19151: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:48.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:48.850297+0000) 2022-09-10T14:27:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:49.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:49.470524+0000) 2022-09-10T14:27:50.134 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:49.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:49.850489+0000) 2022-09-10T14:27:50.472 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:27:50.132+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:27:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:50.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:50.470752+0000) 2022-09-10T14:27:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:50 smithi110 ceph-mon[97633]: pgmap v19152: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:50 smithi104 ceph-mon[97044]: pgmap v19152: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:50.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:50.850748+0000) 2022-09-10T14:27:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:51.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:51.471030+0000) 2022-09-10T14:27:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:51.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:51.851001+0000) 2022-09-10T14:27:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:52.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:52.471251+0000) 2022-09-10T14:27:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:52 smithi110 ceph-mon[97633]: pgmap v19153: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:52 smithi104 ceph-mon[97044]: pgmap v19153: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:53.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:52.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:52.851218+0000) 2022-09-10T14:27:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:53.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:53.471386+0000) 2022-09-10T14:27:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:53.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:53.851461+0000) 2022-09-10T14:27:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:54.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:54.471561+0000) 2022-09-10T14:27:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:54 smithi110 ceph-mon[97633]: pgmap v19154: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:54 smithi104 ceph-mon[97044]: pgmap v19154: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:54.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:54.851644+0000) 2022-09-10T14:27:55.816 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:55.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:55.471695+0000) 2022-09-10T14:27:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:55.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:55.851853+0000) 2022-09-10T14:27:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:56.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:56.471899+0000) 2022-09-10T14:27:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:56 smithi110 ceph-mon[97633]: pgmap v19155: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:56 smithi104 ceph-mon[97044]: pgmap v19155: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:27:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:56.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:56.852050+0000) 2022-09-10T14:27:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:57.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:57.472180+0000) 2022-09-10T14:27:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:57.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:57.852279+0000) 2022-09-10T14:27:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:58.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:58.472429+0000) 2022-09-10T14:27:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:27:58 smithi110 ceph-mon[97633]: pgmap v19156: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:27:58 smithi104 ceph-mon[97044]: pgmap v19156: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:27:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:58.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:58.852453+0000) 2022-09-10T14:27:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:27:59.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:59.472693+0000) 2022-09-10T14:28:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:27:59.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:27:59.852634+0000) 2022-09-10T14:28:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:00.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:00.472848+0000) 2022-09-10T14:28:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:00 smithi110 ceph-mon[97633]: pgmap v19157: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:00 smithi104 ceph-mon[97044]: pgmap v19157: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:00.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:00.852773+0000) 2022-09-10T14:28:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:01.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:01.473046+0000) 2022-09-10T14:28:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:01.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:01.853017+0000) 2022-09-10T14:28:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:02.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:02.473281+0000) 2022-09-10T14:28:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:02 smithi110 ceph-mon[97633]: pgmap v19158: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:02 smithi104 ceph-mon[97044]: pgmap v19158: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:02.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:02.853226+0000) 2022-09-10T14:28:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:03.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:03.473477+0000) 2022-09-10T14:28:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:03.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:03.853454+0000) 2022-09-10T14:28:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:04.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:04.473642+0000) 2022-09-10T14:28:05.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:04.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:04.853674+0000) 2022-09-10T14:28:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:04 smithi110 ceph-mon[97633]: pgmap v19159: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:04 smithi104 ceph-mon[97044]: pgmap v19159: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:05.475 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:28:05.148+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:28:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:05.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:05.473899+0000) 2022-09-10T14:28:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:05.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:05.853918+0000) 2022-09-10T14:28:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:06.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:06.474013+0000) 2022-09-10T14:28:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:06 smithi110 ceph-mon[97633]: pgmap v19160: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:28:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:06 smithi104 ceph-mon[97044]: pgmap v19160: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:28:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:06.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:06.854173+0000) 2022-09-10T14:28:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:07.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:07.474199+0000) 2022-09-10T14:28:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:07.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:07.854417+0000) 2022-09-10T14:28:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:08.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:08.474435+0000) 2022-09-10T14:28:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:08 smithi110 ceph-mon[97633]: pgmap v19161: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:28:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:08 smithi110 ceph-mon[97633]: osdmap e1278: 12 total, 12 up, 12 in 2022-09-10T14:28:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:08 smithi104 ceph-mon[97044]: pgmap v19161: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:28:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:08 smithi104 ceph-mon[97044]: osdmap e1278: 12 total, 12 up, 12 in 2022-09-10T14:28:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:08.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:08.854613+0000) 2022-09-10T14:28:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:09.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:09.474610+0000) 2022-09-10T14:28:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:09.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:09.854865+0000) 2022-09-10T14:28:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:10.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:10.474831+0000) 2022-09-10T14:28:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:10 smithi110 ceph-mon[97633]: pgmap v19163: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:28:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:10 smithi104 ceph-mon[97044]: pgmap v19163: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:28:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:10.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:10.855131+0000) 2022-09-10T14:28:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:11.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:11.475058+0000) 2022-09-10T14:28:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:11.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:11.855305+0000) 2022-09-10T14:28:12.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:12.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:12.475251+0000) 2022-09-10T14:28:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:12 smithi110 ceph-mon[97633]: pgmap v19164: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:28:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:12 smithi104 ceph-mon[97044]: pgmap v19164: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:28:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:12.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:12.855556+0000) 2022-09-10T14:28:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:13.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:13.475465+0000) 2022-09-10T14:28:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:13.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:13.855711+0000) 2022-09-10T14:28:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:14.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:14.475714+0000) 2022-09-10T14:28:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:14 smithi110 ceph-mon[97633]: pgmap v19165: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:28:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:14 smithi104 ceph-mon[97044]: pgmap v19165: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:28:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:14.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:14.855954+0000) 2022-09-10T14:28:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:15.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:15.475965+0000) 2022-09-10T14:28:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:15.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:15.856149+0000) 2022-09-10T14:28:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:16.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:16.476184+0000) 2022-09-10T14:28:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:16 smithi110 ceph-mon[97633]: pgmap v19166: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:28:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:16 smithi110 ceph-mon[97633]: osdmap e1279: 12 total, 12 up, 12 in 2022-09-10T14:28:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:16 smithi104 ceph-mon[97044]: pgmap v19166: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:28:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:16 smithi104 ceph-mon[97044]: osdmap e1279: 12 total, 12 up, 12 in 2022-09-10T14:28:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:16.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:16.856393+0000) 2022-09-10T14:28:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:17.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:17.476352+0000) 2022-09-10T14:28:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:17.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:17.856605+0000) 2022-09-10T14:28:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:18.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:18.476531+0000) 2022-09-10T14:28:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:18 smithi110 ceph-mon[97633]: pgmap v19168: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T14:28:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:18 smithi104 ceph-mon[97044]: pgmap v19168: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T14:28:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:18.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:18.856786+0000) 2022-09-10T14:28:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:19.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:19.476713+0000) 2022-09-10T14:28:20.166 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:19.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:19.857027+0000) 2022-09-10T14:28:20.478 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:28:20.163+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:28:20.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:20.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:20.476831+0000) 2022-09-10T14:28:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:20 smithi110 ceph-mon[97633]: pgmap v19169: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:28:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:20 smithi104 ceph-mon[97044]: pgmap v19169: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:28:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:20.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:20.857261+0000) 2022-09-10T14:28:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:21.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:21.477122+0000) 2022-09-10T14:28:22.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:21.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:21.857488+0000) 2022-09-10T14:28:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:22.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:22.477340+0000) 2022-09-10T14:28:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:22 smithi110 ceph-mon[97633]: pgmap v19170: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:28:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:22 smithi104 ceph-mon[97044]: pgmap v19170: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:28:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:22.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:22.857702+0000) 2022-09-10T14:28:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:23.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:23.477583+0000) 2022-09-10T14:28:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:23.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:23.857954+0000) 2022-09-10T14:28:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:24.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:24.477774+0000) 2022-09-10T14:28:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:24 smithi110 ceph-mon[97633]: pgmap v19171: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:28:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:24 smithi104 ceph-mon[97044]: pgmap v19171: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:28:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:24.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:24.858146+0000) 2022-09-10T14:28:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:25.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:25.478004+0000) 2022-09-10T14:28:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:25.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:25.858358+0000) 2022-09-10T14:28:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:26.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:26.478267+0000) 2022-09-10T14:28:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:26 smithi110 ceph-mon[97633]: pgmap v19172: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:28:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:26 smithi104 ceph-mon[97044]: pgmap v19172: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:28:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:26.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:26.858587+0000) 2022-09-10T14:28:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:27.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:27.478451+0000) 2022-09-10T14:28:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:27.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:27.858839+0000) 2022-09-10T14:28:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:28.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:28.478636+0000) 2022-09-10T14:28:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:28 smithi110 ceph-mon[97633]: pgmap v19173: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:28:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:28 smithi104 ceph-mon[97044]: pgmap v19173: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:28:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:28.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:28.859067+0000) 2022-09-10T14:28:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:29.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:29.478883+0000) 2022-09-10T14:28:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:29.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:29.859327+0000) 2022-09-10T14:28:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:30.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:30.479151+0000) 2022-09-10T14:28:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:30 smithi110 ceph-mon[97633]: pgmap v19174: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:30 smithi104 ceph-mon[97044]: pgmap v19174: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:30.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:30.859551+0000) 2022-09-10T14:28:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:31.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:31.479313+0000) 2022-09-10T14:28:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:31.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:31.859787+0000) 2022-09-10T14:28:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:32.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:32.479560+0000) 2022-09-10T14:28:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:32 smithi110 ceph-mon[97633]: pgmap v19175: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:32 smithi104 ceph-mon[97044]: pgmap v19175: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:32.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:32.860000+0000) 2022-09-10T14:28:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:33.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:33.479790+0000) 2022-09-10T14:28:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:33.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:33.860215+0000) 2022-09-10T14:28:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:34.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:34.480021+0000) 2022-09-10T14:28:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:34 smithi110 ceph-mon[97633]: pgmap v19176: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:35.175 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:34.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:34.860441+0000) 2022-09-10T14:28:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:34 smithi104 ceph-mon[97044]: pgmap v19176: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:35.481 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:28:35.172+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:28:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:35.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:35.480227+0000) 2022-09-10T14:28:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:35.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:35.860676+0000) 2022-09-10T14:28:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:36.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:36.480448+0000) 2022-09-10T14:28:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:36 smithi110 ceph-mon[97633]: pgmap v19177: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:36 smithi104 ceph-mon[97044]: pgmap v19177: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:36.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:36.860929+0000) 2022-09-10T14:28:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:37.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:37.480705+0000) 2022-09-10T14:28:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:37.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:37.861195+0000) 2022-09-10T14:28:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:38.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:38.480928+0000) 2022-09-10T14:28:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:38 smithi104 ceph-mon[97044]: pgmap v19178: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:38.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:38.861340+0000) 2022-09-10T14:28:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:38 smithi110 ceph-mon[97633]: pgmap v19178: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:39.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:39.481210+0000) 2022-09-10T14:28:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:39.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:39.861575+0000) 2022-09-10T14:28:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:40.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:40.481370+0000) 2022-09-10T14:28:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:40 smithi104 ceph-mon[97044]: pgmap v19179: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:40.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:40.861802+0000) 2022-09-10T14:28:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:40 smithi110 ceph-mon[97633]: pgmap v19179: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:41.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:41.481611+0000) 2022-09-10T14:28:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:41.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:41.862050+0000) 2022-09-10T14:28:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:42.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:42.481881+0000) 2022-09-10T14:28:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:42 smithi104 ceph-mon[97044]: pgmap v19180: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:42.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:42.862291+0000) 2022-09-10T14:28:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:42 smithi110 ceph-mon[97633]: pgmap v19180: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:43.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:43.482046+0000) 2022-09-10T14:28:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:43.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:43.862496+0000) 2022-09-10T14:28:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:44.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:44.482282+0000) 2022-09-10T14:28:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:44 smithi104 ceph-mon[97044]: pgmap v19181: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:44.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:44.862754+0000) 2022-09-10T14:28:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:44 smithi110 ceph-mon[97633]: pgmap v19181: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:45.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:45.482500+0000) 2022-09-10T14:28:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:45.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:45.862974+0000) 2022-09-10T14:28:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:46.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:46.482756+0000) 2022-09-10T14:28:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:46 smithi104 ceph-mon[97044]: pgmap v19182: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:28:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:28:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:28:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:28:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:46.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:46.863201+0000) 2022-09-10T14:28:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:46 smithi110 ceph-mon[97633]: pgmap v19182: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:28:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:28:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:28:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:28:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:47.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:47.483002+0000) 2022-09-10T14:28:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:47.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:47.863425+0000) 2022-09-10T14:28:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:48.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:48.483195+0000) 2022-09-10T14:28:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:48 smithi104 ceph-mon[97044]: pgmap v19183: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:48.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:48.863696+0000) 2022-09-10T14:28:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:48 smithi110 ceph-mon[97633]: pgmap v19183: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:49.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:49.483428+0000) 2022-09-10T14:28:50.186 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:49.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:49.863930+0000) 2022-09-10T14:28:50.484 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:28:50.184+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:28:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:50.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:50.483691+0000) 2022-09-10T14:28:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:50 smithi104 ceph-mon[97044]: pgmap v19184: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:28:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:50.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:50.864185+0000) 2022-09-10T14:28:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:50 smithi110 ceph-mon[97633]: pgmap v19184: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:28:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:51.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:51.483907+0000) 2022-09-10T14:28:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:51.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:51.864396+0000) 2022-09-10T14:28:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:52.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:52.484179+0000) 2022-09-10T14:28:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:52 smithi104 ceph-mon[97044]: pgmap v19185: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:52.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:52.864592+0000) 2022-09-10T14:28:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:52 smithi110 ceph-mon[97633]: pgmap v19185: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:53.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:53.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:53.484403+0000) 2022-09-10T14:28:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:53.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:53.864822+0000) 2022-09-10T14:28:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:54.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:54.484654+0000) 2022-09-10T14:28:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:54 smithi104 ceph-mon[97044]: pgmap v19186: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:54.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:54.865047+0000) 2022-09-10T14:28:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:54 smithi110 ceph-mon[97633]: pgmap v19186: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:55.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:55.484902+0000) 2022-09-10T14:28:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:55.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:55.865276+0000) 2022-09-10T14:28:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:56.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:56.485197+0000) 2022-09-10T14:28:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:56 smithi104 ceph-mon[97044]: pgmap v19187: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:56.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:56.865512+0000) 2022-09-10T14:28:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:56 smithi110 ceph-mon[97633]: pgmap v19187: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:28:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:57.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:57.485455+0000) 2022-09-10T14:28:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:57.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:57.865687+0000) 2022-09-10T14:28:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:58.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:58.485689+0000) 2022-09-10T14:28:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:28:58 smithi104 ceph-mon[97044]: pgmap v19188: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:58.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:58.865875+0000) 2022-09-10T14:28:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:28:58 smithi110 ceph-mon[97633]: pgmap v19188: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:28:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:28:59.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:59.485866+0000) 2022-09-10T14:29:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:28:59.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:28:59.866148+0000) 2022-09-10T14:29:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:00.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:00.486176+0000) 2022-09-10T14:29:01.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:00 smithi104 ceph-mon[97044]: pgmap v19189: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:00.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:00.866367+0000) 2022-09-10T14:29:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:00 smithi110 ceph-mon[97633]: pgmap v19189: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:01.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:01.486372+0000) 2022-09-10T14:29:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:01.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:01.866554+0000) 2022-09-10T14:29:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:02.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:02.486577+0000) 2022-09-10T14:29:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:02 smithi104 ceph-mon[97044]: pgmap v19190: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:02.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:02.866800+0000) 2022-09-10T14:29:03.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:02 smithi110 ceph-mon[97633]: pgmap v19190: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:03.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:03.486804+0000) 2022-09-10T14:29:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:03.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:03.867080+0000) 2022-09-10T14:29:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:04.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:04.487136+0000) 2022-09-10T14:29:05.192 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:04.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:04.867238+0000) 2022-09-10T14:29:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:04 smithi104 ceph-mon[97044]: pgmap v19191: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:04 smithi110 ceph-mon[97633]: pgmap v19191: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:05.488 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:29:05.190+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:29:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:05.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:05.487300+0000) 2022-09-10T14:29:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:05.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:05.867469+0000) 2022-09-10T14:29:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:06.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:06.487553+0000) 2022-09-10T14:29:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:06 smithi104 ceph-mon[97044]: pgmap v19192: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:29:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:06.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:06.867691+0000) 2022-09-10T14:29:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:06 smithi110 ceph-mon[97633]: pgmap v19192: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:29:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:07.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:07.487819+0000) 2022-09-10T14:29:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:07.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:07.867924+0000) 2022-09-10T14:29:08.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:08.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:08.488015+0000) 2022-09-10T14:29:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:08 smithi104 ceph-mon[97044]: pgmap v19193: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:29:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:08 smithi104 ceph-mon[97044]: osdmap e1280: 12 total, 12 up, 12 in 2022-09-10T14:29:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:08.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:08.868175+0000) 2022-09-10T14:29:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:08 smithi110 ceph-mon[97633]: pgmap v19193: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:29:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:08 smithi110 ceph-mon[97633]: osdmap e1280: 12 total, 12 up, 12 in 2022-09-10T14:29:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:09.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:09.488178+0000) 2022-09-10T14:29:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:09 smithi104 ceph-mon[97044]: pgmap v19195: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:29:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:09.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:09.868424+0000) 2022-09-10T14:29:10.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:09 smithi110 ceph-mon[97633]: pgmap v19195: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:29:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:10.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:10.488403+0000) 2022-09-10T14:29:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:10.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:10.868635+0000) 2022-09-10T14:29:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:11.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:11.488593+0000) 2022-09-10T14:29:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:11.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:11.868926+0000) 2022-09-10T14:29:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:12.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:12.488737+0000) 2022-09-10T14:29:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:12 smithi104 ceph-mon[97044]: pgmap v19196: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:29:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:12 smithi104 ceph-mon[97044]: osdmap e1281: 12 total, 12 up, 12 in 2022-09-10T14:29:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:12 smithi110 ceph-mon[97633]: pgmap v19196: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:29:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:12 smithi110 ceph-mon[97633]: osdmap e1281: 12 total, 12 up, 12 in 2022-09-10T14:29:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:12.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:12.869182+0000) 2022-09-10T14:29:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:13.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:13.488928+0000) 2022-09-10T14:29:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:13.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:13.869415+0000) 2022-09-10T14:29:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:14.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:14.489200+0000) 2022-09-10T14:29:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:14 smithi104 ceph-mon[97044]: pgmap v19198: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:29:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:14 smithi110 ceph-mon[97633]: pgmap v19198: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:29:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:14.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:14.869599+0000) 2022-09-10T14:29:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:15.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:15.489364+0000) 2022-09-10T14:29:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:15.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:15.869840+0000) 2022-09-10T14:29:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:16.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:16.489548+0000) 2022-09-10T14:29:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:16 smithi104 ceph-mon[97044]: pgmap v19199: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:29:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:16 smithi110 ceph-mon[97633]: pgmap v19199: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:29:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:16.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:16.870121+0000) 2022-09-10T14:29:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:17.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:17.489733+0000) 2022-09-10T14:29:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:17.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:17.870388+0000) 2022-09-10T14:29:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:18.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:18.489954+0000) 2022-09-10T14:29:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:18 smithi104 ceph-mon[97044]: pgmap v19200: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T14:29:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:18 smithi110 ceph-mon[97633]: pgmap v19200: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 218 B/s wr, 11 op/s 2022-09-10T14:29:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:18.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:18.870620+0000) 2022-09-10T14:29:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:19.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:19.490197+0000) 2022-09-10T14:29:20.205 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:19.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:19.870805+0000) 2022-09-10T14:29:20.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:29:20.203+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:29:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:20.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:20.490398+0000) 2022-09-10T14:29:21.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:20 smithi104 ceph-mon[97044]: pgmap v19201: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:29:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:20 smithi110 ceph-mon[97633]: pgmap v19201: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:29:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:20.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:20.871038+0000) 2022-09-10T14:29:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:21.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:21.490645+0000) 2022-09-10T14:29:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:21.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:21.871307+0000) 2022-09-10T14:29:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:22.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:22.490844+0000) 2022-09-10T14:29:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:22 smithi104 ceph-mon[97044]: pgmap v19202: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:29:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:22 smithi110 ceph-mon[97633]: pgmap v19202: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:29:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:22.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:22.871557+0000) 2022-09-10T14:29:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:23.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:23.491009+0000) 2022-09-10T14:29:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:23.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:23.871806+0000) 2022-09-10T14:29:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:24.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:24.491234+0000) 2022-09-10T14:29:25.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:24 smithi104 ceph-mon[97044]: pgmap v19203: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:29:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:24 smithi110 ceph-mon[97633]: pgmap v19203: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:29:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:24.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:24.871985+0000) 2022-09-10T14:29:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:25.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:25.491485+0000) 2022-09-10T14:29:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:25.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:25.872204+0000) 2022-09-10T14:29:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:26.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:26.491656+0000) 2022-09-10T14:29:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:26 smithi104 ceph-mon[97044]: pgmap v19204: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:29:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:26 smithi110 ceph-mon[97633]: pgmap v19204: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:29:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:26.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:26.872450+0000) 2022-09-10T14:29:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:27.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:27.491853+0000) 2022-09-10T14:29:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:27.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:27.872686+0000) 2022-09-10T14:29:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:28.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:28.492066+0000) 2022-09-10T14:29:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:28 smithi104 ceph-mon[97044]: pgmap v19205: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:28 smithi110 ceph-mon[97633]: pgmap v19205: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:28.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:28.872860+0000) 2022-09-10T14:29:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:29.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:29.492277+0000) 2022-09-10T14:29:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:29.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:29.873112+0000) 2022-09-10T14:29:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:30.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:30.492513+0000) 2022-09-10T14:29:31.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:30 smithi104 ceph-mon[97044]: pgmap v19206: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:30 smithi110 ceph-mon[97633]: pgmap v19206: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:30.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:30.873298+0000) 2022-09-10T14:29:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:31.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:31.492753+0000) 2022-09-10T14:29:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:31.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:31.873484+0000) 2022-09-10T14:29:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:32.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:32.492987+0000) 2022-09-10T14:29:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:32 smithi104 ceph-mon[97044]: pgmap v19207: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:32 smithi110 ceph-mon[97633]: pgmap v19207: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:32.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:32.873714+0000) 2022-09-10T14:29:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:33.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:33.493231+0000) 2022-09-10T14:29:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:33.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:33.873954+0000) 2022-09-10T14:29:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:34.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:34.493515+0000) 2022-09-10T14:29:35.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:34 smithi104 ceph-mon[97044]: pgmap v19208: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:34 smithi110 ceph-mon[97633]: pgmap v19208: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:35.221 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:34.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:34.874207+0000) 2022-09-10T14:29:35.495 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:29:35.218+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:29:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:35.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:35.493744+0000) 2022-09-10T14:29:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:35.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:35.874435+0000) 2022-09-10T14:29:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:36.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:36.494028+0000) 2022-09-10T14:29:37.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:36 smithi104 ceph-mon[97044]: pgmap v19209: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:29:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:36 smithi110 ceph-mon[97633]: pgmap v19209: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:29:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:36.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:36.874612+0000) 2022-09-10T14:29:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:37.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:37.494247+0000) 2022-09-10T14:29:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:37.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:37.874830+0000) 2022-09-10T14:29:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:38.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:38.494500+0000) 2022-09-10T14:29:39.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:38 smithi104 ceph-mon[97044]: pgmap v19210: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:38 smithi110 ceph-mon[97633]: pgmap v19210: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:38.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:38.874977+0000) 2022-09-10T14:29:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:39.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:39.494754+0000) 2022-09-10T14:29:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:39.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:39.875230+0000) 2022-09-10T14:29:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:40.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:40.495006+0000) 2022-09-10T14:29:41.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:40 smithi104 ceph-mon[97044]: pgmap v19211: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:40 smithi110 ceph-mon[97633]: pgmap v19211: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:40.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:40.875422+0000) 2022-09-10T14:29:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:41.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:41.495272+0000) 2022-09-10T14:29:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:41.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:41.875668+0000) 2022-09-10T14:29:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:42.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:42.495501+0000) 2022-09-10T14:29:43.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:42 smithi104 ceph-mon[97044]: pgmap v19212: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:42 smithi110 ceph-mon[97633]: pgmap v19212: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:42.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:42.875868+0000) 2022-09-10T14:29:43.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:43.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:43.495641+0000) 2022-09-10T14:29:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:43.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:43.876053+0000) 2022-09-10T14:29:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:44.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:44.495887+0000) 2022-09-10T14:29:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:44 smithi104 ceph-mon[97044]: pgmap v19213: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:44 smithi110 ceph-mon[97633]: pgmap v19213: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:44.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:44.876251+0000) 2022-09-10T14:29:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:45.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:45.496047+0000) 2022-09-10T14:29:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:45.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:45.876433+0000) 2022-09-10T14:29:46.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:46.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:46.496266+0000) 2022-09-10T14:29:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:46 smithi104 ceph-mon[97044]: pgmap v19214: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:29:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:29:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:29:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:29:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:29:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:46 smithi110 ceph-mon[97633]: pgmap v19214: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:29:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:29:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:29:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:29:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:29:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:46.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:46.876669+0000) 2022-09-10T14:29:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:47.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:47.496463+0000) 2022-09-10T14:29:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:47.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:47.876896+0000) 2022-09-10T14:29:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:48.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:48.496673+0000) 2022-09-10T14:29:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:48 smithi104 ceph-mon[97044]: pgmap v19215: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:48 smithi110 ceph-mon[97633]: pgmap v19215: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:48.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:48.877121+0000) 2022-09-10T14:29:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:49.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:49.496898+0000) 2022-09-10T14:29:50.237 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:49.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:49.877386+0000) 2022-09-10T14:29:50.498 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:29:50.234+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:29:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:50.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:50.497126+0000) 2022-09-10T14:29:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:50 smithi104 ceph-mon[97044]: pgmap v19216: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:50 smithi110 ceph-mon[97633]: pgmap v19216: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:29:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:50.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:50.877615+0000) 2022-09-10T14:29:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:51.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:51.497282+0000) 2022-09-10T14:29:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:51.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:51.877852+0000) 2022-09-10T14:29:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:52.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:52.497487+0000) 2022-09-10T14:29:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:52 smithi104 ceph-mon[97044]: pgmap v19217: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:52 smithi110 ceph-mon[97633]: pgmap v19217: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:29:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:52.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:52.878120+0000) 2022-09-10T14:29:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:53.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:53.497723+0000) 2022-09-10T14:29:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:53.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:53.878332+0000) 2022-09-10T14:29:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:54.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:54.497998+0000) 2022-09-10T14:29:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:54 smithi104 ceph-mon[97044]: pgmap v19218: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:54 smithi110 ceph-mon[97633]: pgmap v19218: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:54.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:54.878565+0000) 2022-09-10T14:29:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:55.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:55.498224+0000) 2022-09-10T14:29:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:55.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:55.878829+0000) 2022-09-10T14:29:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:56.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:56.498445+0000) 2022-09-10T14:29:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:56 smithi104 ceph-mon[97044]: pgmap v19219: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:29:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:56 smithi110 ceph-mon[97633]: pgmap v19219: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:29:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:56.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:56.879024+0000) 2022-09-10T14:29:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:57.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:57.498630+0000) 2022-09-10T14:29:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:57.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:57.879234+0000) 2022-09-10T14:29:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:58.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:58.498870+0000) 2022-09-10T14:29:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:29:58 smithi104 ceph-mon[97044]: pgmap v19220: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:29:58 smithi110 ceph-mon[97633]: pgmap v19220: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:29:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:58.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:58.879463+0000) 2022-09-10T14:29:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:29:59.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:59.499027+0000) 2022-09-10T14:30:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:29:59.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:29:59.879638+0000) 2022-09-10T14:30:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:00.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:00.499187+0000) 2022-09-10T14:30:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:00 smithi104 ceph-mon[97044]: pgmap v19221: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:30:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:00 smithi110 ceph-mon[97633]: pgmap v19221: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:01.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:30:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:00.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:00.879797+0000) 2022-09-10T14:30:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:01.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:01.499368+0000) 2022-09-10T14:30:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:01.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:01.880027+0000) 2022-09-10T14:30:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:02.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:02.499601+0000) 2022-09-10T14:30:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:02 smithi104 ceph-mon[97044]: pgmap v19222: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:02 smithi110 ceph-mon[97633]: pgmap v19222: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:02.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:02.880215+0000) 2022-09-10T14:30:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:03.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:03.499859+0000) 2022-09-10T14:30:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:03.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:03.880457+0000) 2022-09-10T14:30:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:04.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:04.500129+0000) 2022-09-10T14:30:05.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:04 smithi104 ceph-mon[97044]: pgmap v19223: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:04 smithi110 ceph-mon[97633]: pgmap v19223: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:05.242 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:04.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:04.880618+0000) 2022-09-10T14:30:05.502 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:30:05.239+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:30:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:05.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:05.500359+0000) 2022-09-10T14:30:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:05.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:05.880803+0000) 2022-09-10T14:30:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:06.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:06.500619+0000) 2022-09-10T14:30:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:06 smithi104 ceph-mon[97044]: pgmap v19224: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:30:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:06 smithi110 ceph-mon[97633]: pgmap v19224: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:30:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:06.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:06.881028+0000) 2022-09-10T14:30:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:07.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:07.500892+0000) 2022-09-10T14:30:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:07.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:07.881212+0000) 2022-09-10T14:30:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:08.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:08.501177+0000) 2022-09-10T14:30:09.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:08 smithi104 ceph-mon[97044]: pgmap v19225: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:30:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:08 smithi110 ceph-mon[97633]: pgmap v19225: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:30:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:08.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:08.881435+0000) 2022-09-10T14:30:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:09.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:09.501421+0000) 2022-09-10T14:30:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:09 smithi104 ceph-mon[97044]: osdmap e1282: 12 total, 12 up, 12 in 2022-09-10T14:30:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:09 smithi110 ceph-mon[97633]: osdmap e1282: 12 total, 12 up, 12 in 2022-09-10T14:30:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:09.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:09.881582+0000) 2022-09-10T14:30:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:10.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:10.501661+0000) 2022-09-10T14:30:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:10 smithi104 ceph-mon[97044]: pgmap v19227: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:30:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:10 smithi110 ceph-mon[97633]: pgmap v19227: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:30:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:10.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:10.881799+0000) 2022-09-10T14:30:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:11.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:11.501890+0000) 2022-09-10T14:30:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:11.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:11.882033+0000) 2022-09-10T14:30:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:12.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:12.502187+0000) 2022-09-10T14:30:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:12 smithi104 ceph-mon[97044]: pgmap v19228: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:30:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:12 smithi110 ceph-mon[97633]: pgmap v19228: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:30:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:12.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:12.882282+0000) 2022-09-10T14:30:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:13.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:13.502305+0000) 2022-09-10T14:30:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:13.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:13.882507+0000) 2022-09-10T14:30:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:14.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:14.502529+0000) 2022-09-10T14:30:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:14 smithi104 ceph-mon[97044]: pgmap v19229: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:30:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:14 smithi110 ceph-mon[97633]: pgmap v19229: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:30:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:14.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:14.882737+0000) 2022-09-10T14:30:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:15.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:15.502724+0000) 2022-09-10T14:30:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:15.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:15.882979+0000) 2022-09-10T14:30:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:16.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:16.502963+0000) 2022-09-10T14:30:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:16 smithi104 ceph-mon[97044]: pgmap v19230: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:30:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:16 smithi104 ceph-mon[97044]: osdmap e1283: 12 total, 12 up, 12 in 2022-09-10T14:30:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:16 smithi110 ceph-mon[97633]: pgmap v19230: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:30:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:16 smithi110 ceph-mon[97633]: osdmap e1283: 12 total, 12 up, 12 in 2022-09-10T14:30:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:16.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:16.883178+0000) 2022-09-10T14:30:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:17.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:17.503169+0000) 2022-09-10T14:30:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:17.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:17.883408+0000) 2022-09-10T14:30:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:18.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:18.503401+0000) 2022-09-10T14:30:19.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:18 smithi104 ceph-mon[97044]: pgmap v19232: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:30:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:18 smithi110 ceph-mon[97633]: pgmap v19232: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 238 B/s wr, 12 op/s 2022-09-10T14:30:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:18.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:18.883637+0000) 2022-09-10T14:30:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:19.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:19.503658+0000) 2022-09-10T14:30:20.257 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:19.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:19.883864+0000) 2022-09-10T14:30:20.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:20.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:20.503884+0000) 2022-09-10T14:30:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:30:20.255+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:30:21.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:20 smithi104 ceph-mon[97044]: pgmap v19233: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:30:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:20 smithi110 ceph-mon[97633]: pgmap v19233: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:30:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:20.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:20.884135+0000) 2022-09-10T14:30:21.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:21.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:21.504155+0000) 2022-09-10T14:30:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:21.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:21.884407+0000) 2022-09-10T14:30:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:22.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:22.504342+0000) 2022-09-10T14:30:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:22 smithi104 ceph-mon[97044]: pgmap v19234: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:30:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:22 smithi110 ceph-mon[97633]: pgmap v19234: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:30:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:22.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:22.884635+0000) 2022-09-10T14:30:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:23.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:23.504544+0000) 2022-09-10T14:30:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:23.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:23.884856+0000) 2022-09-10T14:30:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:24.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:24.504781+0000) 2022-09-10T14:30:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:24 smithi104 ceph-mon[97044]: pgmap v19235: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:30:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:24 smithi110 ceph-mon[97633]: pgmap v19235: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:30:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:24.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:24.885049+0000) 2022-09-10T14:30:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:25.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:25.504991+0000) 2022-09-10T14:30:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:25.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:25.885232+0000) 2022-09-10T14:30:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:26.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:26.505236+0000) 2022-09-10T14:30:27.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:26 smithi104 ceph-mon[97044]: pgmap v19236: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:30:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:26 smithi110 ceph-mon[97633]: pgmap v19236: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:30:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:26.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:26.885412+0000) 2022-09-10T14:30:27.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:27.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:27.505446+0000) 2022-09-10T14:30:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:27.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:27.885631+0000) 2022-09-10T14:30:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:28.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:28.505628+0000) 2022-09-10T14:30:29.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:28 smithi104 ceph-mon[97044]: pgmap v19237: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:30:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:28 smithi110 ceph-mon[97633]: pgmap v19237: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:30:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:28.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:28.885828+0000) 2022-09-10T14:30:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:29.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:29.505900+0000) 2022-09-10T14:30:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:29.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:29.886089+0000) 2022-09-10T14:30:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:30.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:30.506137+0000) 2022-09-10T14:30:31.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:30 smithi104 ceph-mon[97044]: pgmap v19238: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:30 smithi110 ceph-mon[97633]: pgmap v19238: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:30.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:30.886355+0000) 2022-09-10T14:30:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:31.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:31.506306+0000) 2022-09-10T14:30:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:31.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:31.886564+0000) 2022-09-10T14:30:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:32.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:32.506552+0000) 2022-09-10T14:30:33.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:32 smithi104 ceph-mon[97044]: pgmap v19239: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:32 smithi110 ceph-mon[97633]: pgmap v19239: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:32.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:32.886745+0000) 2022-09-10T14:30:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:33.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:33.506803+0000) 2022-09-10T14:30:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:33.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:33.886988+0000) 2022-09-10T14:30:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:34.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:34.507045+0000) 2022-09-10T14:30:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:34 smithi110 ceph-mon[97633]: pgmap v19240: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:35.270 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:34.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:34.887159+0000) 2022-09-10T14:30:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:34 smithi104 ceph-mon[97044]: pgmap v19240: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:35.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:35.507244+0000) 2022-09-10T14:30:35.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:30:35.267+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:30:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:35.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:35.887388+0000) 2022-09-10T14:30:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:36.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:36.507453+0000) 2022-09-10T14:30:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:36 smithi110 ceph-mon[97633]: pgmap v19241: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:30:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:36 smithi104 ceph-mon[97044]: pgmap v19241: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:30:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:36.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:36.887577+0000) 2022-09-10T14:30:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:37.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:37.507689+0000) 2022-09-10T14:30:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:37.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:37.887824+0000) 2022-09-10T14:30:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:38.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:38.507953+0000) 2022-09-10T14:30:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:38 smithi110 ceph-mon[97633]: pgmap v19242: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:38 smithi104 ceph-mon[97044]: pgmap v19242: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:38.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:38.888084+0000) 2022-09-10T14:30:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:39.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:39.508214+0000) 2022-09-10T14:30:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:39.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:39.888319+0000) 2022-09-10T14:30:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:40.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:40.508344+0000) 2022-09-10T14:30:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:40 smithi110 ceph-mon[97633]: pgmap v19243: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:40 smithi104 ceph-mon[97044]: pgmap v19243: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:40.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:40.888539+0000) 2022-09-10T14:30:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:41.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:41.508529+0000) 2022-09-10T14:30:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:41.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:41.888789+0000) 2022-09-10T14:30:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:42.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:42.508652+0000) 2022-09-10T14:30:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:42 smithi110 ceph-mon[97633]: pgmap v19244: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:42 smithi104 ceph-mon[97044]: pgmap v19244: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:42.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:42.889008+0000) 2022-09-10T14:30:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:43.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:43.508870+0000) 2022-09-10T14:30:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:43.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:43.889215+0000) 2022-09-10T14:30:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:44.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:44.509177+0000) 2022-09-10T14:30:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:44 smithi110 ceph-mon[97633]: pgmap v19245: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:44 smithi104 ceph-mon[97044]: pgmap v19245: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:44.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:44.889442+0000) 2022-09-10T14:30:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:45.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:45.509439+0000) 2022-09-10T14:30:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:45.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:45.889652+0000) 2022-09-10T14:30:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:46.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:46.509717+0000) 2022-09-10T14:30:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:46 smithi110 ceph-mon[97633]: pgmap v19246: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:30:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:30:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:30:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:30:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:30:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:46 smithi104 ceph-mon[97044]: pgmap v19246: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:30:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:30:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:30:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:30:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:30:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:46.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:46.889888+0000) 2022-09-10T14:30:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:47.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:47.509962+0000) 2022-09-10T14:30:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:47.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:47.890023+0000) 2022-09-10T14:30:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:48.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:48.510231+0000) 2022-09-10T14:30:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:48 smithi110 ceph-mon[97633]: pgmap v19247: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:49.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:48 smithi104 ceph-mon[97044]: pgmap v19247: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:48.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:48.890247+0000) 2022-09-10T14:30:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:49.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:49.510450+0000) 2022-09-10T14:30:50.277 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:49.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:49.890421+0000) 2022-09-10T14:30:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:50.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:50.510613+0000) 2022-09-10T14:30:50.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:30:50.275+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:30:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:50 smithi110 ceph-mon[97633]: pgmap v19248: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:50 smithi104 ceph-mon[97044]: pgmap v19248: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:30:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:50.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:50.890678+0000) 2022-09-10T14:30:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:51.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:51.510851+0000) 2022-09-10T14:30:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:51.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:51.890895+0000) 2022-09-10T14:30:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:52.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:52.511138+0000) 2022-09-10T14:30:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:52 smithi110 ceph-mon[97633]: pgmap v19249: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:52 smithi104 ceph-mon[97044]: pgmap v19249: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:30:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:52.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:52.891138+0000) 2022-09-10T14:30:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:53.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:53.511379+0000) 2022-09-10T14:30:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:53.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:53.891326+0000) 2022-09-10T14:30:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:54.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:54.511666+0000) 2022-09-10T14:30:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:54 smithi110 ceph-mon[97633]: pgmap v19250: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:54 smithi104 ceph-mon[97044]: pgmap v19250: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:54.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:54.891520+0000) 2022-09-10T14:30:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:55.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:55.511860+0000) 2022-09-10T14:30:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:55.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:55.891740+0000) 2022-09-10T14:30:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:56.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:56.512088+0000) 2022-09-10T14:30:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:56 smithi110 ceph-mon[97633]: pgmap v19251: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:30:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:56 smithi104 ceph-mon[97044]: pgmap v19251: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:30:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:56.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:56.891960+0000) 2022-09-10T14:30:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:57.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:57.512364+0000) 2022-09-10T14:30:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:57.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:57.892179+0000) 2022-09-10T14:30:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:58.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:58.512645+0000) 2022-09-10T14:30:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:30:58 smithi110 ceph-mon[97633]: pgmap v19252: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:30:58 smithi104 ceph-mon[97044]: pgmap v19252: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:30:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:58.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:58.892409+0000) 2022-09-10T14:30:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:30:59.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:59.512881+0000) 2022-09-10T14:31:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:30:59.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:30:59.892661+0000) 2022-09-10T14:31:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:00.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:00.513077+0000) 2022-09-10T14:31:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:00 smithi110 ceph-mon[97633]: pgmap v19253: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:00 smithi104 ceph-mon[97044]: pgmap v19253: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:00.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:00.892889+0000) 2022-09-10T14:31:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:01.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:01.513305+0000) 2022-09-10T14:31:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:01.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:01.893045+0000) 2022-09-10T14:31:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:02.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:02.513487+0000) 2022-09-10T14:31:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:02 smithi110 ceph-mon[97633]: pgmap v19254: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:02 smithi104 ceph-mon[97044]: pgmap v19254: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:02.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:02.893274+0000) 2022-09-10T14:31:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:03.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:03.513707+0000) 2022-09-10T14:31:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:03.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:03.893517+0000) 2022-09-10T14:31:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:04.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:04.513963+0000) 2022-09-10T14:31:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:04 smithi110 ceph-mon[97633]: pgmap v19255: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:05.278 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:04.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:04.893707+0000) 2022-09-10T14:31:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:04 smithi104 ceph-mon[97044]: pgmap v19255: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:05.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:31:05.276+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:31:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:05.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:05.514173+0000) 2022-09-10T14:31:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:05.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:05.893881+0000) 2022-09-10T14:31:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:06.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:06.514437+0000) 2022-09-10T14:31:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:06 smithi110 ceph-mon[97633]: pgmap v19256: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:31:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:06 smithi104 ceph-mon[97044]: pgmap v19256: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:31:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:06.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:06.894064+0000) 2022-09-10T14:31:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:07.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:07.514653+0000) 2022-09-10T14:31:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:07.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:07.894190+0000) 2022-09-10T14:31:08.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:08.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:08.514893+0000) 2022-09-10T14:31:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:08 smithi110 ceph-mon[97633]: pgmap v19257: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:31:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:08 smithi104 ceph-mon[97044]: pgmap v19257: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:31:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:08.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:08.894370+0000) 2022-09-10T14:31:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:09.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:09.515170+0000) 2022-09-10T14:31:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:09 smithi110 ceph-mon[97633]: osdmap e1284: 12 total, 12 up, 12 in 2022-09-10T14:31:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:09 smithi104 ceph-mon[97044]: osdmap e1284: 12 total, 12 up, 12 in 2022-09-10T14:31:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:09.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:09.894593+0000) 2022-09-10T14:31:10.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:10.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:10.515409+0000) 2022-09-10T14:31:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:10 smithi110 ceph-mon[97633]: pgmap v19259: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:31:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:10 smithi104 ceph-mon[97044]: pgmap v19259: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:31:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:10.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:10.894823+0000) 2022-09-10T14:31:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:11.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:11.515603+0000) 2022-09-10T14:31:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:11.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:11.894985+0000) 2022-09-10T14:31:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:12.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:12.515863+0000) 2022-09-10T14:31:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:12 smithi110 ceph-mon[97633]: pgmap v19260: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:31:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:12 smithi104 ceph-mon[97044]: pgmap v19260: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:31:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:12.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:12.895169+0000) 2022-09-10T14:31:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:13.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:13.516040+0000) 2022-09-10T14:31:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:13.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:13.895379+0000) 2022-09-10T14:31:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:14.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:14.516259+0000) 2022-09-10T14:31:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:14 smithi110 ceph-mon[97633]: pgmap v19261: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:31:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:14 smithi104 ceph-mon[97044]: pgmap v19261: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:31:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:14.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:14.895570+0000) 2022-09-10T14:31:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:15.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:15.516398+0000) 2022-09-10T14:31:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:15.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:15.895731+0000) 2022-09-10T14:31:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:16.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:16.516613+0000) 2022-09-10T14:31:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:16 smithi110 ceph-mon[97633]: pgmap v19262: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:31:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:16 smithi110 ceph-mon[97633]: osdmap e1285: 12 total, 12 up, 12 in 2022-09-10T14:31:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:16 smithi104 ceph-mon[97044]: pgmap v19262: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:31:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:16 smithi104 ceph-mon[97044]: osdmap e1285: 12 total, 12 up, 12 in 2022-09-10T14:31:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:16.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:16.895915+0000) 2022-09-10T14:31:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:17.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:17.516868+0000) 2022-09-10T14:31:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:17.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:17.896188+0000) 2022-09-10T14:31:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:18.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:18.517128+0000) 2022-09-10T14:31:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:18 smithi110 ceph-mon[97633]: pgmap v19264: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 240 B/s wr, 11 op/s 2022-09-10T14:31:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:18 smithi104 ceph-mon[97044]: pgmap v19264: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 240 B/s wr, 11 op/s 2022-09-10T14:31:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:18.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:18.896370+0000) 2022-09-10T14:31:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:19.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:19.517382+0000) 2022-09-10T14:31:20.291 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:19.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:19.896571+0000) 2022-09-10T14:31:20.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:31:20.288+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:31:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:20.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:20.517604+0000) 2022-09-10T14:31:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:20 smithi110 ceph-mon[97633]: pgmap v19265: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:31:21.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:20 smithi104 ceph-mon[97044]: pgmap v19265: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:31:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:20.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:20.896735+0000) 2022-09-10T14:31:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:21.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:21.517829+0000) 2022-09-10T14:31:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:21.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:21.896936+0000) 2022-09-10T14:31:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:22.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:22.518081+0000) 2022-09-10T14:31:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:22 smithi110 ceph-mon[97633]: pgmap v19266: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:31:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:22 smithi104 ceph-mon[97044]: pgmap v19266: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:31:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:22.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:22.897176+0000) 2022-09-10T14:31:23.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:23.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:23.518368+0000) 2022-09-10T14:31:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:23.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:23.897361+0000) 2022-09-10T14:31:24.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:24.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:24.518519+0000) 2022-09-10T14:31:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:24 smithi110 ceph-mon[97633]: pgmap v19267: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:31:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:24 smithi104 ceph-mon[97044]: pgmap v19267: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:31:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:24.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:24.897541+0000) 2022-09-10T14:31:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:25.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:25.518703+0000) 2022-09-10T14:31:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:25.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:25.897751+0000) 2022-09-10T14:31:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:26.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:26.518933+0000) 2022-09-10T14:31:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:26 smithi110 ceph-mon[97633]: pgmap v19268: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:31:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:26 smithi104 ceph-mon[97044]: pgmap v19268: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:31:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:26.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:26.897964+0000) 2022-09-10T14:31:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:27.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:27.519191+0000) 2022-09-10T14:31:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:27.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:27.898189+0000) 2022-09-10T14:31:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:28.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:28.519392+0000) 2022-09-10T14:31:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:28 smithi110 ceph-mon[97633]: pgmap v19269: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:31:29.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:28 smithi104 ceph-mon[97044]: pgmap v19269: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:31:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:28.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:28.898326+0000) 2022-09-10T14:31:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:29.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:29.519594+0000) 2022-09-10T14:31:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:29.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:29.898517+0000) 2022-09-10T14:31:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:30.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:30.519836+0000) 2022-09-10T14:31:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:30 smithi110 ceph-mon[97633]: pgmap v19270: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:30 smithi104 ceph-mon[97044]: pgmap v19270: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:30.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:30.898772+0000) 2022-09-10T14:31:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:31.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:31.520050+0000) 2022-09-10T14:31:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:31.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:31.899055+0000) 2022-09-10T14:31:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:32.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:32.520275+0000) 2022-09-10T14:31:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:32 smithi110 ceph-mon[97633]: pgmap v19271: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:32 smithi104 ceph-mon[97044]: pgmap v19271: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:32.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:32.899230+0000) 2022-09-10T14:31:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:33.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:33.520420+0000) 2022-09-10T14:31:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:33.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:33.899412+0000) 2022-09-10T14:31:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:34.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:34.520570+0000) 2022-09-10T14:31:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:34 smithi110 ceph-mon[97633]: pgmap v19272: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:35.292 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:34.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:34.899646+0000) 2022-09-10T14:31:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:34 smithi104 ceph-mon[97044]: pgmap v19272: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:35.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:35.520782+0000) 2022-09-10T14:31:35.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:31:35.291+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:31:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:35.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:35.899884+0000) 2022-09-10T14:31:36.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:36.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:36.521073+0000) 2022-09-10T14:31:37.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:36 smithi110 ceph-mon[97633]: pgmap v19273: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:31:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:36 smithi104 ceph-mon[97044]: pgmap v19273: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:31:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:36.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:36.900141+0000) 2022-09-10T14:31:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:37.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:37.521309+0000) 2022-09-10T14:31:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:37.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:37.900392+0000) 2022-09-10T14:31:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:38.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:38.521529+0000) 2022-09-10T14:31:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:38 smithi110 ceph-mon[97633]: pgmap v19274: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:38 smithi104 ceph-mon[97044]: pgmap v19274: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:38.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:38.900596+0000) 2022-09-10T14:31:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:39.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:39.521693+0000) 2022-09-10T14:31:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:39.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:39.900825+0000) 2022-09-10T14:31:40.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:40.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:40.521883+0000) 2022-09-10T14:31:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:40 smithi110 ceph-mon[97633]: pgmap v19275: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:40 smithi104 ceph-mon[97044]: pgmap v19275: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:40.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:40.901077+0000) 2022-09-10T14:31:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:41.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:41.522125+0000) 2022-09-10T14:31:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:41.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:41.901285+0000) 2022-09-10T14:31:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:42.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:42.522292+0000) 2022-09-10T14:31:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:42 smithi110 ceph-mon[97633]: pgmap v19276: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:42 smithi104 ceph-mon[97044]: pgmap v19276: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:42.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:42.901451+0000) 2022-09-10T14:31:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:43.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:43.522550+0000) 2022-09-10T14:31:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:43.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:43.901634+0000) 2022-09-10T14:31:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:44.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:44.522776+0000) 2022-09-10T14:31:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:44 smithi110 ceph-mon[97633]: pgmap v19277: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:44 smithi104 ceph-mon[97044]: pgmap v19277: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:44.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:44.901865+0000) 2022-09-10T14:31:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:45.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:45.523028+0000) 2022-09-10T14:31:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:45.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:45.902087+0000) 2022-09-10T14:31:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:46.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:46.523257+0000) 2022-09-10T14:31:47.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:46 smithi104 ceph-mon[97044]: pgmap v19278: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:31:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:31:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:31:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:31:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:31:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:46.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:46.902258+0000) 2022-09-10T14:31:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:46 smithi110 ceph-mon[97633]: pgmap v19278: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:31:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:31:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:31:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:31:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:31:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:47.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:47.523514+0000) 2022-09-10T14:31:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:47.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:47.902483+0000) 2022-09-10T14:31:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:48.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:48.523775+0000) 2022-09-10T14:31:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:48 smithi104 ceph-mon[97044]: pgmap v19279: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:48.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:48.902667+0000) 2022-09-10T14:31:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:48 smithi110 ceph-mon[97633]: pgmap v19279: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:49.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:49.524008+0000) 2022-09-10T14:31:50.303 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:49.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:49.902875+0000) 2022-09-10T14:31:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:50.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:50.524237+0000) 2022-09-10T14:31:50.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:31:50.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:31:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:50 smithi104 ceph-mon[97044]: pgmap v19280: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:50.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:50.903064+0000) 2022-09-10T14:31:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:50 smithi110 ceph-mon[97633]: pgmap v19280: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:31:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:51.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:51.524377+0000) 2022-09-10T14:31:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:51.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:51.903233+0000) 2022-09-10T14:31:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:52.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:52.524556+0000) 2022-09-10T14:31:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:52 smithi104 ceph-mon[97044]: pgmap v19281: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:52.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:52.903400+0000) 2022-09-10T14:31:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:52 smithi110 ceph-mon[97633]: pgmap v19281: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:31:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:53.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:53.524699+0000) 2022-09-10T14:31:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:53.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:53.903585+0000) 2022-09-10T14:31:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:54.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:54.524937+0000) 2022-09-10T14:31:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:54 smithi104 ceph-mon[97044]: pgmap v19282: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:54.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:54.903768+0000) 2022-09-10T14:31:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:54 smithi110 ceph-mon[97633]: pgmap v19282: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:55.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:55.525188+0000) 2022-09-10T14:31:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:55.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:55.903963+0000) 2022-09-10T14:31:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:56.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:56.525432+0000) 2022-09-10T14:31:57.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:56 smithi104 ceph-mon[97044]: pgmap v19283: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:31:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:56.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:56.904188+0000) 2022-09-10T14:31:57.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:56 smithi110 ceph-mon[97633]: pgmap v19283: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:31:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:57.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:57.525659+0000) 2022-09-10T14:31:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:57.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:57.904410+0000) 2022-09-10T14:31:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:58.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:58.525914+0000) 2022-09-10T14:31:59.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:31:58 smithi104 ceph-mon[97044]: pgmap v19284: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:58.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:58.904606+0000) 2022-09-10T14:31:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:31:58 smithi110 ceph-mon[97633]: pgmap v19284: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:31:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:31:59.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:59.526053+0000) 2022-09-10T14:32:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:31:59.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:31:59.904840+0000) 2022-09-10T14:32:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:00.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:00.526329+0000) 2022-09-10T14:32:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:00 smithi104 ceph-mon[97044]: pgmap v19285: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:00.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:00.905083+0000) 2022-09-10T14:32:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:00 smithi110 ceph-mon[97633]: pgmap v19285: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:01.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:01.526481+0000) 2022-09-10T14:32:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:01.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:01.905381+0000) 2022-09-10T14:32:02.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:02.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:02.526667+0000) 2022-09-10T14:32:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:02 smithi104 ceph-mon[97044]: pgmap v19286: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:02.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:02.905567+0000) 2022-09-10T14:32:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:02 smithi110 ceph-mon[97633]: pgmap v19286: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:03.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:03.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:03.526898+0000) 2022-09-10T14:32:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:03.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:03.905767+0000) 2022-09-10T14:32:04.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:04.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:04.527086+0000) 2022-09-10T14:32:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:04 smithi104 ceph-mon[97044]: pgmap v19287: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:04.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:04.905959+0000) 2022-09-10T14:32:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:04 smithi110 ceph-mon[97633]: pgmap v19287: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:05.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:05.527289+0000) 2022-09-10T14:32:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:32:05.312+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:32:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:05.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:05.906202+0000) 2022-09-10T14:32:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:06.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:06.527492+0000) 2022-09-10T14:32:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:06 smithi104 ceph-mon[97044]: pgmap v19288: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:32:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:06.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:06.906363+0000) 2022-09-10T14:32:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:06 smithi110 ceph-mon[97633]: pgmap v19288: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:32:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:07.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:07.527754+0000) 2022-09-10T14:32:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:07.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:07.906558+0000) 2022-09-10T14:32:08.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:08.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:08.527962+0000) 2022-09-10T14:32:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:08 smithi104 ceph-mon[97044]: pgmap v19289: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:32:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:08 smithi104 ceph-mon[97044]: osdmap e1286: 12 total, 12 up, 12 in 2022-09-10T14:32:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:08.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:08.906811+0000) 2022-09-10T14:32:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:08 smithi110 ceph-mon[97633]: pgmap v19289: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:32:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:08 smithi110 ceph-mon[97633]: osdmap e1286: 12 total, 12 up, 12 in 2022-09-10T14:32:09.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:09.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:09.528155+0000) 2022-09-10T14:32:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:09.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:09.907056+0000) 2022-09-10T14:32:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:10.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:10.528380+0000) 2022-09-10T14:32:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:10 smithi104 ceph-mon[97044]: pgmap v19291: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:32:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:10.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:10.907277+0000) 2022-09-10T14:32:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:10 smithi110 ceph-mon[97633]: pgmap v19291: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:32:11.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:11.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:11.528642+0000) 2022-09-10T14:32:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:11.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:11.907500+0000) 2022-09-10T14:32:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:12.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:12.528801+0000) 2022-09-10T14:32:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:12 smithi104 ceph-mon[97044]: pgmap v19292: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:32:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:12.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:12.907683+0000) 2022-09-10T14:32:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:12 smithi110 ceph-mon[97633]: pgmap v19292: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:32:13.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:13.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:13.528998+0000) 2022-09-10T14:32:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:13.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:13.907897+0000) 2022-09-10T14:32:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:14.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:14.529258+0000) 2022-09-10T14:32:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:14 smithi104 ceph-mon[97044]: pgmap v19293: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:32:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:14.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:14.908193+0000) 2022-09-10T14:32:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:14 smithi110 ceph-mon[97633]: pgmap v19293: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:32:15.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:15.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:15.529460+0000) 2022-09-10T14:32:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:15.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:15.908415+0000) 2022-09-10T14:32:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:16.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:16.529674+0000) 2022-09-10T14:32:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:16 smithi104 ceph-mon[97044]: pgmap v19294: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:32:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:16 smithi104 ceph-mon[97044]: osdmap e1287: 12 total, 12 up, 12 in 2022-09-10T14:32:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:16.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:16.908635+0000) 2022-09-10T14:32:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:16 smithi110 ceph-mon[97633]: pgmap v19294: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:32:17.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:16 smithi110 ceph-mon[97633]: osdmap e1287: 12 total, 12 up, 12 in 2022-09-10T14:32:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:17.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:17.529908+0000) 2022-09-10T14:32:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:17.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:17.908897+0000) 2022-09-10T14:32:18.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:18.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:18.530153+0000) 2022-09-10T14:32:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:18 smithi104 ceph-mon[97044]: pgmap v19296: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T14:32:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:18.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:18.909169+0000) 2022-09-10T14:32:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:18 smithi110 ceph-mon[97633]: pgmap v19296: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2022-09-10T14:32:19.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:19.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:19.530359+0000) 2022-09-10T14:32:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:19.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:19.909319+0000) 2022-09-10T14:32:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:32:20.322+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:32:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:20.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:20.530594+0000) 2022-09-10T14:32:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:20 smithi104 ceph-mon[97044]: pgmap v19297: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:32:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:20.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:20.909496+0000) 2022-09-10T14:32:21.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:20 smithi110 ceph-mon[97633]: pgmap v19297: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:32:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:21.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:21.530788+0000) 2022-09-10T14:32:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:21.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:21.909710+0000) 2022-09-10T14:32:22.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:22.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:22.531000+0000) 2022-09-10T14:32:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:22 smithi104 ceph-mon[97044]: pgmap v19298: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:32:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:22.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:22.909936+0000) 2022-09-10T14:32:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:22 smithi110 ceph-mon[97633]: pgmap v19298: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:32:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:23.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:23.531225+0000) 2022-09-10T14:32:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:23.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:23.910192+0000) 2022-09-10T14:32:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:24.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:24.531425+0000) 2022-09-10T14:32:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:24 smithi104 ceph-mon[97044]: pgmap v19299: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:32:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:24.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:24.910455+0000) 2022-09-10T14:32:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:24 smithi110 ceph-mon[97633]: pgmap v19299: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:32:25.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:25.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:25.531612+0000) 2022-09-10T14:32:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:25.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:25.910648+0000) 2022-09-10T14:32:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:26.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:26.531882+0000) 2022-09-10T14:32:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:26 smithi104 ceph-mon[97044]: pgmap v19300: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:32:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:26.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:26.910835+0000) 2022-09-10T14:32:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:26 smithi110 ceph-mon[97633]: pgmap v19300: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:32:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:27.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:27.532188+0000) 2022-09-10T14:32:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:27.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:27.911052+0000) 2022-09-10T14:32:28.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:28.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:28.532433+0000) 2022-09-10T14:32:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:28 smithi104 ceph-mon[97044]: pgmap v19301: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:32:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:28.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:28.911313+0000) 2022-09-10T14:32:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:28 smithi110 ceph-mon[97633]: pgmap v19301: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:32:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:29.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:29.532654+0000) 2022-09-10T14:32:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:29.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:29.911515+0000) 2022-09-10T14:32:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:30.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:30.532939+0000) 2022-09-10T14:32:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:30 smithi104 ceph-mon[97044]: pgmap v19302: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:30.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:30.911775+0000) 2022-09-10T14:32:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:30 smithi110 ceph-mon[97633]: pgmap v19302: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:31.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:31.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:31.533190+0000) 2022-09-10T14:32:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:31.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:31.912016+0000) 2022-09-10T14:32:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:32.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:32.533452+0000) 2022-09-10T14:32:33.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:32 smithi104 ceph-mon[97044]: pgmap v19303: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:32.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:32.912267+0000) 2022-09-10T14:32:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:32 smithi110 ceph-mon[97633]: pgmap v19303: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:33.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:33.533697+0000) 2022-09-10T14:32:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:33.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:33.912501+0000) 2022-09-10T14:32:34.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:34.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:34.533900+0000) 2022-09-10T14:32:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:34 smithi104 ceph-mon[97044]: pgmap v19304: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:34.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:34.912725+0000) 2022-09-10T14:32:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:34 smithi110 ceph-mon[97633]: pgmap v19304: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:35.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:35.534163+0000) 2022-09-10T14:32:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:32:35.334+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:32:36.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:35 smithi104 ceph-mon[97044]: pgmap v19305: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:32:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:35.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:35.912957+0000) 2022-09-10T14:32:36.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:35 smithi110 ceph-mon[97633]: pgmap v19305: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:32:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:36.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:36.534396+0000) 2022-09-10T14:32:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:36.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:36.913155+0000) 2022-09-10T14:32:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:37.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:37.534550+0000) 2022-09-10T14:32:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:37.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:37.913352+0000) 2022-09-10T14:32:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:38.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:38.534788+0000) 2022-09-10T14:32:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:38 smithi104 ceph-mon[97044]: pgmap v19306: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:38 smithi110 ceph-mon[97633]: pgmap v19306: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:38.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:38.913536+0000) 2022-09-10T14:32:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:39.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:39.535053+0000) 2022-09-10T14:32:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:39.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:39.913711+0000) 2022-09-10T14:32:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:40.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:40.535294+0000) 2022-09-10T14:32:41.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:40 smithi104 ceph-mon[97044]: pgmap v19307: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:40 smithi110 ceph-mon[97633]: pgmap v19307: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:40.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:40.913927+0000) 2022-09-10T14:32:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:41.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:41.535492+0000) 2022-09-10T14:32:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:41.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:41.914211+0000) 2022-09-10T14:32:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:42.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:42.535688+0000) 2022-09-10T14:32:43.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:42 smithi104 ceph-mon[97044]: pgmap v19308: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:42 smithi110 ceph-mon[97633]: pgmap v19308: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:42.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:42.914358+0000) 2022-09-10T14:32:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:43.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:43.535913+0000) 2022-09-10T14:32:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:43.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:43.914553+0000) 2022-09-10T14:32:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:44.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:44.536215+0000) 2022-09-10T14:32:45.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:44 smithi104 ceph-mon[97044]: pgmap v19309: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:44 smithi110 ceph-mon[97633]: pgmap v19309: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:44.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:44.914796+0000) 2022-09-10T14:32:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:45.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:45.536446+0000) 2022-09-10T14:32:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:45.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:45.914936+0000) 2022-09-10T14:32:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:46.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:46.536677+0000) 2022-09-10T14:32:47.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:46 smithi104 ceph-mon[97044]: pgmap v19310: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:32:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:32:47.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:32:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:32:47.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:32:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:46 smithi110 ceph-mon[97633]: pgmap v19310: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:32:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:32:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:32:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:32:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:32:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:46.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:46.915176+0000) 2022-09-10T14:32:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:47.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:47.536941+0000) 2022-09-10T14:32:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:47.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:47.915358+0000) 2022-09-10T14:32:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:48.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:48.537161+0000) 2022-09-10T14:32:49.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:48 smithi104 ceph-mon[97044]: pgmap v19311: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:48 smithi110 ceph-mon[97633]: pgmap v19311: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:48.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:48.915605+0000) 2022-09-10T14:32:49.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:49.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:49.537309+0000) 2022-09-10T14:32:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:49.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:49.915856+0000) 2022-09-10T14:32:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:32:50.342+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:32:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:50.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:50.537517+0000) 2022-09-10T14:32:51.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:50 smithi104 ceph-mon[97044]: pgmap v19312: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:50 smithi110 ceph-mon[97633]: pgmap v19312: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:32:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:50.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:50.916089+0000) 2022-09-10T14:32:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:51.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:51.537779+0000) 2022-09-10T14:32:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:51.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:51.916340+0000) 2022-09-10T14:32:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:52.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:52.538034+0000) 2022-09-10T14:32:53.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:52 smithi104 ceph-mon[97044]: pgmap v19313: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:52 smithi110 ceph-mon[97633]: pgmap v19313: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:32:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:52.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:52.916598+0000) 2022-09-10T14:32:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:53.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:53.538233+0000) 2022-09-10T14:32:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:53.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:53.916840+0000) 2022-09-10T14:32:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:54.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:54.538488+0000) 2022-09-10T14:32:55.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:54 smithi104 ceph-mon[97044]: pgmap v19314: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:54 smithi110 ceph-mon[97633]: pgmap v19314: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:32:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:54.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:54.917042+0000) 2022-09-10T14:32:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:55.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:55.538712+0000) 2022-09-10T14:32:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:55.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:55.917285+0000) 2022-09-10T14:32:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:56.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:56.538943+0000) 2022-09-10T14:32:57.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:56 smithi104 ceph-mon[97044]: pgmap v19315: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:32:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:56 smithi110 ceph-mon[97633]: pgmap v19315: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:32:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:56.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:56.917527+0000) 2022-09-10T14:32:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:57.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:57.539199+0000) 2022-09-10T14:32:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:57.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:57.917739+0000) 2022-09-10T14:32:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:58.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:58.539439+0000) 2022-09-10T14:32:59.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:32:58 smithi104 ceph-mon[97044]: pgmap v19316: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:32:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:32:58 smithi110 ceph-mon[97633]: pgmap v19316: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:32:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:58.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:58.917989+0000) 2022-09-10T14:32:59.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:32:59.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:59.539640+0000) 2022-09-10T14:33:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:32:59.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:32:59.918244+0000) 2022-09-10T14:33:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:00.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:00.539877+0000) 2022-09-10T14:33:01.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:00 smithi104 ceph-mon[97044]: pgmap v19317: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:00 smithi110 ceph-mon[97633]: pgmap v19317: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:00.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:00.918485+0000) 2022-09-10T14:33:01.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:01.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:01.540062+0000) 2022-09-10T14:33:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:01.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:01.918680+0000) 2022-09-10T14:33:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:02.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:02.540300+0000) 2022-09-10T14:33:03.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:02 smithi104 ceph-mon[97044]: pgmap v19318: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:33:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:02 smithi110 ceph-mon[97633]: pgmap v19318: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:33:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:02.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:02.918923+0000) 2022-09-10T14:33:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:03.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:03.540482+0000) 2022-09-10T14:33:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:03.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:03.919185+0000) 2022-09-10T14:33:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:04.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:04.540654+0000) 2022-09-10T14:33:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:04 smithi104 ceph-mon[97044]: pgmap v19319: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:33:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:04 smithi110 ceph-mon[97633]: pgmap v19319: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:33:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:04.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:04.919413+0000) 2022-09-10T14:33:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:05.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:05.540889+0000) 2022-09-10T14:33:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:33:05.358+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:33:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:05.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:05.919658+0000) 2022-09-10T14:33:06.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:06.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:06.541230+0000) 2022-09-10T14:33:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:06 smithi104 ceph-mon[97044]: pgmap v19320: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:33:07.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:06 smithi110 ceph-mon[97633]: pgmap v19320: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:33:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:06.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:06.919899+0000) 2022-09-10T14:33:07.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:07.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:07.541470+0000) 2022-09-10T14:33:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:07.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:07.920114+0000) 2022-09-10T14:33:08.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:08.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:08.541731+0000) 2022-09-10T14:33:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:08 smithi104 ceph-mon[97044]: pgmap v19321: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:33:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:08 smithi110 ceph-mon[97633]: pgmap v19321: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:33:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:08.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:08.920288+0000) 2022-09-10T14:33:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:09.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:09.542003+0000) 2022-09-10T14:33:10.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:09 smithi104 ceph-mon[97044]: osdmap e1288: 12 total, 12 up, 12 in 2022-09-10T14:33:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:09 smithi110 ceph-mon[97633]: osdmap e1288: 12 total, 12 up, 12 in 2022-09-10T14:33:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:09.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:09.920490+0000) 2022-09-10T14:33:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:10.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:10.542292+0000) 2022-09-10T14:33:11.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:10 smithi104 ceph-mon[97044]: pgmap v19323: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:33:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:10 smithi110 ceph-mon[97633]: pgmap v19323: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:33:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:10.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:10.920702+0000) 2022-09-10T14:33:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:11.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:11.542454+0000) 2022-09-10T14:33:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:11.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:11.920936+0000) 2022-09-10T14:33:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:12.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:12.542683+0000) 2022-09-10T14:33:13.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:12 smithi104 ceph-mon[97044]: pgmap v19324: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:33:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:12 smithi110 ceph-mon[97633]: pgmap v19324: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:33:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:12.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:12.921219+0000) 2022-09-10T14:33:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:13.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:13.542928+0000) 2022-09-10T14:33:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:13.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:13.921461+0000) 2022-09-10T14:33:14.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:14.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:14.543216+0000) 2022-09-10T14:33:15.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:14 smithi104 ceph-mon[97044]: pgmap v19325: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:33:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:14 smithi110 ceph-mon[97633]: pgmap v19325: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:33:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:14.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:14.921731+0000) 2022-09-10T14:33:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:15.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:15.543440+0000) 2022-09-10T14:33:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:15.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:15.921966+0000) 2022-09-10T14:33:16.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:16.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:16.543648+0000) 2022-09-10T14:33:17.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:16 smithi104 ceph-mon[97044]: pgmap v19326: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:33:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:16 smithi104 ceph-mon[97044]: osdmap e1289: 12 total, 12 up, 12 in 2022-09-10T14:33:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:16 smithi110 ceph-mon[97633]: pgmap v19326: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:33:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:16 smithi110 ceph-mon[97633]: osdmap e1289: 12 total, 12 up, 12 in 2022-09-10T14:33:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:16.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:16.922179+0000) 2022-09-10T14:33:17.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:17.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:17.543882+0000) 2022-09-10T14:33:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:17.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:17.922367+0000) 2022-09-10T14:33:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:18.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:18.544134+0000) 2022-09-10T14:33:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:18 smithi104 ceph-mon[97044]: pgmap v19328: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T14:33:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:18 smithi110 ceph-mon[97633]: pgmap v19328: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 237 B/s wr, 12 op/s 2022-09-10T14:33:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:18.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:18.922571+0000) 2022-09-10T14:33:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:19.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:19.544287+0000) 2022-09-10T14:33:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:19.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:19.922819+0000) 2022-09-10T14:33:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:20.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:20.544531+0000) 2022-09-10T14:33:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:33:20.374+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:33:21.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:20 smithi104 ceph-mon[97044]: pgmap v19329: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:33:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:20 smithi110 ceph-mon[97633]: pgmap v19329: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:33:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:20.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:20.923092+0000) 2022-09-10T14:33:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:21.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:21.544793+0000) 2022-09-10T14:33:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:21.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:21.923326+0000) 2022-09-10T14:33:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:22.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:22.545032+0000) 2022-09-10T14:33:23.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:22 smithi104 ceph-mon[97044]: pgmap v19330: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:33:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:22 smithi110 ceph-mon[97633]: pgmap v19330: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:33:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:22.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:22.923588+0000) 2022-09-10T14:33:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:23.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:23.545175+0000) 2022-09-10T14:33:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:23.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:23.923809+0000) 2022-09-10T14:33:24.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:24.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:24.545417+0000) 2022-09-10T14:33:25.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:24 smithi104 ceph-mon[97044]: pgmap v19331: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:33:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:24 smithi110 ceph-mon[97633]: pgmap v19331: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:33:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:24.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:24.924007+0000) 2022-09-10T14:33:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:25.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:25.545618+0000) 2022-09-10T14:33:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:25.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:25.924239+0000) 2022-09-10T14:33:26.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:26.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:26.545861+0000) 2022-09-10T14:33:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:26 smithi110 ceph-mon[97633]: pgmap v19332: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:33:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:26 smithi104 ceph-mon[97044]: pgmap v19332: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:33:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:26.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:26.924437+0000) 2022-09-10T14:33:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:27.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:27.546161+0000) 2022-09-10T14:33:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:27.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:27.924619+0000) 2022-09-10T14:33:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:28.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:28.546336+0000) 2022-09-10T14:33:29.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:28 smithi110 ceph-mon[97633]: pgmap v19333: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:33:29.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:28 smithi104 ceph-mon[97044]: pgmap v19333: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:33:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:28.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:28.924859+0000) 2022-09-10T14:33:29.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:29.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:29.546533+0000) 2022-09-10T14:33:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:29.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:29.925126+0000) 2022-09-10T14:33:30.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:30.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:30.546772+0000) 2022-09-10T14:33:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:30 smithi110 ceph-mon[97633]: pgmap v19334: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:31.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:30.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:30.925419+0000) 2022-09-10T14:33:31.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:30 smithi104 ceph-mon[97044]: pgmap v19334: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:31.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:31.547003+0000) 2022-09-10T14:33:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:31.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:31.925613+0000) 2022-09-10T14:33:32.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:32.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:32.547250+0000) 2022-09-10T14:33:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:32 smithi110 ceph-mon[97633]: pgmap v19335: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:33:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:32 smithi104 ceph-mon[97044]: pgmap v19335: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:33:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:32.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:32.925884+0000) 2022-09-10T14:33:33.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:33.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:33.547450+0000) 2022-09-10T14:33:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:33.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:33.926166+0000) 2022-09-10T14:33:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:34.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:34.547653+0000) 2022-09-10T14:33:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:34 smithi110 ceph-mon[97633]: pgmap v19336: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:34 smithi104 ceph-mon[97044]: pgmap v19336: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:34.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:34.926469+0000) 2022-09-10T14:33:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:35.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:35.547938+0000) 2022-09-10T14:33:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:33:35.382+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:33:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:35.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:35.926608+0000) 2022-09-10T14:33:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:36.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:36.548199+0000) 2022-09-10T14:33:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:36 smithi110 ceph-mon[97633]: pgmap v19337: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:33:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:36 smithi104 ceph-mon[97044]: pgmap v19337: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:33:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:36.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:36.926861+0000) 2022-09-10T14:33:37.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:37.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:37.548391+0000) 2022-09-10T14:33:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:37.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:37.927150+0000) 2022-09-10T14:33:38.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:38.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:38.548612+0000) 2022-09-10T14:33:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:38 smithi110 ceph-mon[97633]: pgmap v19338: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:38 smithi104 ceph-mon[97044]: pgmap v19338: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:38.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:38.927399+0000) 2022-09-10T14:33:39.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:39.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:39.548791+0000) 2022-09-10T14:33:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:39.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:39.927601+0000) 2022-09-10T14:33:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:40.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:40.549043+0000) 2022-09-10T14:33:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:40 smithi110 ceph-mon[97633]: pgmap v19339: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:40 smithi104 ceph-mon[97044]: pgmap v19339: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:40.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:40.927913+0000) 2022-09-10T14:33:41.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:41.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:41.549317+0000) 2022-09-10T14:33:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:41.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:41.928146+0000) 2022-09-10T14:33:42.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:42.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:42.549498+0000) 2022-09-10T14:33:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:42 smithi110 ceph-mon[97633]: pgmap v19340: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:33:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:42 smithi104 ceph-mon[97044]: pgmap v19340: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:33:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:42.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:42.928373+0000) 2022-09-10T14:33:43.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:43.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:43.549739+0000) 2022-09-10T14:33:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:43.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:43.928593+0000) 2022-09-10T14:33:44.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:44.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:44.549963+0000) 2022-09-10T14:33:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:44 smithi110 ceph-mon[97633]: pgmap v19341: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:44 smithi104 ceph-mon[97044]: pgmap v19341: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:44.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:44.928792+0000) 2022-09-10T14:33:45.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:45.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:45.550164+0000) 2022-09-10T14:33:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:45.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:45.929009+0000) 2022-09-10T14:33:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:46.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:46.550367+0000) 2022-09-10T14:33:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:46 smithi110 ceph-mon[97633]: pgmap v19342: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:33:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:33:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:33:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:33:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:33:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:46 smithi104 ceph-mon[97044]: pgmap v19342: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:33:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:33:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:33:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:33:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:33:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:46.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:46.929233+0000) 2022-09-10T14:33:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:47.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:47.550550+0000) 2022-09-10T14:33:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:47.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:47.929482+0000) 2022-09-10T14:33:48.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:48.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:48.550759+0000) 2022-09-10T14:33:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:48 smithi110 ceph-mon[97633]: pgmap v19343: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:48 smithi104 ceph-mon[97044]: pgmap v19343: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:48.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:48.929716+0000) 2022-09-10T14:33:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:49.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:49.550992+0000) 2022-09-10T14:33:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:49.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:49.929943+0000) 2022-09-10T14:33:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:50.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:50.551237+0000) 2022-09-10T14:33:50.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:33:50.389+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:33:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:50 smithi110 ceph-mon[97633]: pgmap v19344: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:50 smithi104 ceph-mon[97044]: pgmap v19344: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:33:51.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:50.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:50.930213+0000) 2022-09-10T14:33:51.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:51.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:51.551423+0000) 2022-09-10T14:33:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:51.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:51.930430+0000) 2022-09-10T14:33:52.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:52.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:52.551618+0000) 2022-09-10T14:33:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:52 smithi110 ceph-mon[97633]: pgmap v19345: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:33:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:52 smithi104 ceph-mon[97044]: pgmap v19345: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:33:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:52.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:52.930662+0000) 2022-09-10T14:33:53.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:53.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:53.551811+0000) 2022-09-10T14:33:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:53.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:53.930966+0000) 2022-09-10T14:33:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:54.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:54.552029+0000) 2022-09-10T14:33:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:54 smithi110 ceph-mon[97633]: pgmap v19346: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:54 smithi104 ceph-mon[97044]: pgmap v19346: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:54.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:54.931202+0000) 2022-09-10T14:33:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:55.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:55.552152+0000) 2022-09-10T14:33:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:55.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:55.931422+0000) 2022-09-10T14:33:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:56.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:56.552421+0000) 2022-09-10T14:33:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:56 smithi110 ceph-mon[97633]: pgmap v19347: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:33:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:56 smithi104 ceph-mon[97044]: pgmap v19347: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:33:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:56.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:56.931599+0000) 2022-09-10T14:33:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:57.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:57.552612+0000) 2022-09-10T14:33:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:57.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:57.931863+0000) 2022-09-10T14:33:58.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:58.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:58.552884+0000) 2022-09-10T14:33:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:33:58 smithi110 ceph-mon[97633]: pgmap v19348: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:33:58 smithi104 ceph-mon[97044]: pgmap v19348: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:33:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:58.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:58.932138+0000) 2022-09-10T14:33:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:33:59.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:59.553141+0000) 2022-09-10T14:34:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:33:59.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:33:59.932332+0000) 2022-09-10T14:34:00.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:00.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:00.553262+0000) 2022-09-10T14:34:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:00 smithi110 ceph-mon[97633]: pgmap v19349: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:00 smithi104 ceph-mon[97044]: pgmap v19349: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:00.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:00.932454+0000) 2022-09-10T14:34:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:01.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:01.553374+0000) 2022-09-10T14:34:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:01.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:01.932638+0000) 2022-09-10T14:34:02.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:02.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:02.553609+0000) 2022-09-10T14:34:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:02 smithi110 ceph-mon[97633]: pgmap v19350: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:34:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:02 smithi104 ceph-mon[97044]: pgmap v19350: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:34:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:02.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:02.932828+0000) 2022-09-10T14:34:03.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:03.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:03.553785+0000) 2022-09-10T14:34:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:03.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:03.933032+0000) 2022-09-10T14:34:04.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:04.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:04.553986+0000) 2022-09-10T14:34:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:04 smithi110 ceph-mon[97633]: pgmap v19351: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:04 smithi104 ceph-mon[97044]: pgmap v19351: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:04.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:04.933246+0000) 2022-09-10T14:34:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:05.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:05.554236+0000) 2022-09-10T14:34:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:34:05.391+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:34:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:05.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:05.933431+0000) 2022-09-10T14:34:06.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:06.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:06.554511+0000) 2022-09-10T14:34:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:06 smithi110 ceph-mon[97633]: pgmap v19352: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:34:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:06 smithi104 ceph-mon[97044]: pgmap v19352: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:34:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:06.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:06.933634+0000) 2022-09-10T14:34:07.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:07.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:07.554768+0000) 2022-09-10T14:34:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:07.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:07.933873+0000) 2022-09-10T14:34:08.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:08.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:08.554935+0000) 2022-09-10T14:34:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:08 smithi110 ceph-mon[97633]: pgmap v19353: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:34:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:08 smithi110 ceph-mon[97633]: osdmap e1290: 12 total, 12 up, 12 in 2022-09-10T14:34:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:08 smithi104 ceph-mon[97044]: pgmap v19353: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:34:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:08 smithi104 ceph-mon[97044]: osdmap e1290: 12 total, 12 up, 12 in 2022-09-10T14:34:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:08.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:08.934028+0000) 2022-09-10T14:34:09.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:09.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:09.555217+0000) 2022-09-10T14:34:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:09.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:09.934265+0000) 2022-09-10T14:34:10.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:10.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:10.555438+0000) 2022-09-10T14:34:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:10 smithi110 ceph-mon[97633]: pgmap v19355: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:34:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:10 smithi104 ceph-mon[97044]: pgmap v19355: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:34:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:10.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:10.934506+0000) 2022-09-10T14:34:11.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:11.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:11.555692+0000) 2022-09-10T14:34:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:11.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:11.934756+0000) 2022-09-10T14:34:12.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:12.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:12.555912+0000) 2022-09-10T14:34:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:12 smithi110 ceph-mon[97633]: pgmap v19356: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:34:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:12 smithi104 ceph-mon[97044]: pgmap v19356: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:34:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:12.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:12.934994+0000) 2022-09-10T14:34:13.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:13.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:13.556215+0000) 2022-09-10T14:34:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:13.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:13.935225+0000) 2022-09-10T14:34:14.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:14.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:14.556359+0000) 2022-09-10T14:34:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:14 smithi110 ceph-mon[97633]: pgmap v19357: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:34:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:14 smithi104 ceph-mon[97044]: pgmap v19357: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:34:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:14.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:14.935404+0000) 2022-09-10T14:34:15.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:15.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:15.556574+0000) 2022-09-10T14:34:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:15.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:15.935638+0000) 2022-09-10T14:34:16.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:16.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:16.556786+0000) 2022-09-10T14:34:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:16 smithi110 ceph-mon[97633]: pgmap v19358: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:34:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:16 smithi110 ceph-mon[97633]: osdmap e1291: 12 total, 12 up, 12 in 2022-09-10T14:34:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:16 smithi104 ceph-mon[97044]: pgmap v19358: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:34:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:16 smithi104 ceph-mon[97044]: osdmap e1291: 12 total, 12 up, 12 in 2022-09-10T14:34:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:16.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:16.935821+0000) 2022-09-10T14:34:17.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:17.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:17.557020+0000) 2022-09-10T14:34:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:17.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:17.936076+0000) 2022-09-10T14:34:18.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:18.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:18.557230+0000) 2022-09-10T14:34:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:18 smithi110 ceph-mon[97633]: pgmap v19360: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T14:34:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:18 smithi104 ceph-mon[97044]: pgmap v19360: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T14:34:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:18.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:18.936329+0000) 2022-09-10T14:34:19.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:19.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:19.557477+0000) 2022-09-10T14:34:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:19.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:19.936485+0000) 2022-09-10T14:34:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:34:20.401+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:34:20.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:20.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:20.557657+0000) 2022-09-10T14:34:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:20 smithi110 ceph-mon[97633]: pgmap v19361: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:34:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:20 smithi104 ceph-mon[97044]: pgmap v19361: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:34:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:20.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:20.936675+0000) 2022-09-10T14:34:21.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:21.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:21.557831+0000) 2022-09-10T14:34:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:21.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:21.936925+0000) 2022-09-10T14:34:22.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:22.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:22.558112+0000) 2022-09-10T14:34:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:22 smithi110 ceph-mon[97633]: pgmap v19362: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:34:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:22 smithi104 ceph-mon[97044]: pgmap v19362: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:34:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:22.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:22.937128+0000) 2022-09-10T14:34:23.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:23.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:23.558326+0000) 2022-09-10T14:34:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:23.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:23.937233+0000) 2022-09-10T14:34:24.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:24.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:24.558530+0000) 2022-09-10T14:34:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:24 smithi110 ceph-mon[97633]: pgmap v19363: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:34:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:24 smithi104 ceph-mon[97044]: pgmap v19363: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:34:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:24.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:24.937485+0000) 2022-09-10T14:34:25.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:25.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:25.558759+0000) 2022-09-10T14:34:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:25.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:25.937697+0000) 2022-09-10T14:34:26.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:26.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:26.559000+0000) 2022-09-10T14:34:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:26 smithi110 ceph-mon[97633]: pgmap v19364: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T14:34:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:26 smithi104 ceph-mon[97044]: pgmap v19364: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T14:34:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:26.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:26.937909+0000) 2022-09-10T14:34:27.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:27.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:27.559232+0000) 2022-09-10T14:34:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:27.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:27.938169+0000) 2022-09-10T14:34:28.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:28.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:28.559397+0000) 2022-09-10T14:34:29.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:28 smithi110 ceph-mon[97633]: pgmap v19365: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T14:34:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:28 smithi104 ceph-mon[97044]: pgmap v19365: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T14:34:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:28.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:28.938391+0000) 2022-09-10T14:34:29.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:29.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:29.559600+0000) 2022-09-10T14:34:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:29.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:29.938661+0000) 2022-09-10T14:34:30.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:30.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:30.559834+0000) 2022-09-10T14:34:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:30 smithi110 ceph-mon[97633]: pgmap v19366: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:31.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:30 smithi104 ceph-mon[97044]: pgmap v19366: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:30.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:30.938915+0000) 2022-09-10T14:34:31.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:31.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:31.560061+0000) 2022-09-10T14:34:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:31.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:31.939110+0000) 2022-09-10T14:34:32.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:32.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:32.560304+0000) 2022-09-10T14:34:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:32 smithi110 ceph-mon[97633]: pgmap v19367: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T14:34:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:32 smithi104 ceph-mon[97044]: pgmap v19367: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T14:34:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:32.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:32.939347+0000) 2022-09-10T14:34:33.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:33.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:33.560538+0000) 2022-09-10T14:34:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:33.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:33.939515+0000) 2022-09-10T14:34:34.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:34.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:34.560741+0000) 2022-09-10T14:34:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:34 smithi110 ceph-mon[97633]: pgmap v19368: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:34:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:34 smithi104 ceph-mon[97044]: pgmap v19368: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:34:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:34.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:34.939755+0000) 2022-09-10T14:34:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:34:35.410+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:34:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:35.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:35.560905+0000) 2022-09-10T14:34:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:35.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:35.939987+0000) 2022-09-10T14:34:36.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:36.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:36.561186+0000) 2022-09-10T14:34:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:36 smithi104 ceph-mon[97044]: pgmap v19369: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:34:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:36.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:36.940232+0000) 2022-09-10T14:34:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:36 smithi110 ceph-mon[97633]: pgmap v19369: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:34:37.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:37.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:37.561383+0000) 2022-09-10T14:34:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:37.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:37.940375+0000) 2022-09-10T14:34:38.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:38.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:38.561633+0000) 2022-09-10T14:34:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:38 smithi104 ceph-mon[97044]: pgmap v19370: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:38.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:38.940518+0000) 2022-09-10T14:34:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:38 smithi110 ceph-mon[97633]: pgmap v19370: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:39.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:39.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:39.561924+0000) 2022-09-10T14:34:40.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:39.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:39.940725+0000) 2022-09-10T14:34:40.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:40.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:40.562190+0000) 2022-09-10T14:34:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:40 smithi104 ceph-mon[97044]: pgmap v19371: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:40.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:40.940965+0000) 2022-09-10T14:34:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:40 smithi110 ceph-mon[97633]: pgmap v19371: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:41.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:41.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:41.562416+0000) 2022-09-10T14:34:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:41.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:41.941151+0000) 2022-09-10T14:34:42.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:42.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:42.562629+0000) 2022-09-10T14:34:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:42 smithi104 ceph-mon[97044]: pgmap v19372: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:34:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:42.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:42.941330+0000) 2022-09-10T14:34:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:42 smithi110 ceph-mon[97633]: pgmap v19372: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:34:43.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:43.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:43.562887+0000) 2022-09-10T14:34:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:43.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:43.941538+0000) 2022-09-10T14:34:44.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:44.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:44.563144+0000) 2022-09-10T14:34:45.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:44 smithi104 ceph-mon[97044]: pgmap v19373: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:44.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:44.941759+0000) 2022-09-10T14:34:45.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:44 smithi110 ceph-mon[97633]: pgmap v19373: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:45.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:45.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:45.563384+0000) 2022-09-10T14:34:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:45.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:45.941935+0000) 2022-09-10T14:34:46.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:46.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:46.563619+0000) 2022-09-10T14:34:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:46 smithi104 ceph-mon[97044]: pgmap v19374: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:34:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:34:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:34:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:34:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:34:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:46.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:46.942190+0000) 2022-09-10T14:34:47.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:46 smithi110 ceph-mon[97633]: pgmap v19374: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:34:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:34:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:34:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:34:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:34:47.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:47.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:47.563835+0000) 2022-09-10T14:34:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:47.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:47.942359+0000) 2022-09-10T14:34:48.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:48.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:48.564143+0000) 2022-09-10T14:34:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:48 smithi104 ceph-mon[97044]: pgmap v19375: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:48.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:48.942517+0000) 2022-09-10T14:34:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:48 smithi110 ceph-mon[97633]: pgmap v19375: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:49.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:49.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:49.564375+0000) 2022-09-10T14:34:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:49.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:49.942717+0000) 2022-09-10T14:34:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:50.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:50.564643+0000) 2022-09-10T14:34:50.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:34:50.424+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:34:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:50 smithi104 ceph-mon[97044]: pgmap v19376: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:50.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:50.942957+0000) 2022-09-10T14:34:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:50 smithi110 ceph-mon[97633]: pgmap v19376: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:34:51.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:51.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:51.564870+0000) 2022-09-10T14:34:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:51.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:51.943147+0000) 2022-09-10T14:34:52.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:52.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:52.565062+0000) 2022-09-10T14:34:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:52 smithi104 ceph-mon[97044]: pgmap v19377: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:34:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:52.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:52.943391+0000) 2022-09-10T14:34:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:52 smithi110 ceph-mon[97633]: pgmap v19377: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:34:53.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:53.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:53.565303+0000) 2022-09-10T14:34:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:53.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:53.943633+0000) 2022-09-10T14:34:54.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:54.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:54.565580+0000) 2022-09-10T14:34:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:54 smithi104 ceph-mon[97044]: pgmap v19378: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:54.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:54.943858+0000) 2022-09-10T14:34:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:54 smithi110 ceph-mon[97633]: pgmap v19378: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:55.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:55.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:55.565753+0000) 2022-09-10T14:34:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:55.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:55.944080+0000) 2022-09-10T14:34:56.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:56.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:56.566020+0000) 2022-09-10T14:34:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:56 smithi104 ceph-mon[97044]: pgmap v19379: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:34:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:56.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:56.944328+0000) 2022-09-10T14:34:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:56 smithi110 ceph-mon[97633]: pgmap v19379: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:34:57.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:57.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:57.566242+0000) 2022-09-10T14:34:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:57.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:57.944524+0000) 2022-09-10T14:34:58.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:58.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:58.566528+0000) 2022-09-10T14:34:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:34:58 smithi104 ceph-mon[97044]: pgmap v19380: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:58.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:58.944792+0000) 2022-09-10T14:34:59.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:34:58 smithi110 ceph-mon[97633]: pgmap v19380: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:34:59.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:34:59.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:59.566710+0000) 2022-09-10T14:35:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:34:59.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:34:59.945029+0000) 2022-09-10T14:35:00.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:00.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:00.566899+0000) 2022-09-10T14:35:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:00 smithi104 ceph-mon[97044]: pgmap v19381: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:00.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:00.945241+0000) 2022-09-10T14:35:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:00 smithi110 ceph-mon[97633]: pgmap v19381: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:01.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:01.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:01.567193+0000) 2022-09-10T14:35:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:01.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:01.945464+0000) 2022-09-10T14:35:02.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:02.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:02.567376+0000) 2022-09-10T14:35:03.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:02 smithi104 ceph-mon[97044]: pgmap v19382: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:02.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:02.945660+0000) 2022-09-10T14:35:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:02 smithi110 ceph-mon[97633]: pgmap v19382: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:03.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:03.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:03.567582+0000) 2022-09-10T14:35:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:03.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:03.945832+0000) 2022-09-10T14:35:04.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:04.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:04.567765+0000) 2022-09-10T14:35:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:04 smithi104 ceph-mon[97044]: pgmap v19383: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:04.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:04.946016+0000) 2022-09-10T14:35:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:04 smithi110 ceph-mon[97633]: pgmap v19383: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:05.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:05.567945+0000) 2022-09-10T14:35:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:35:05.436+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:35:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:05.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:05.946205+0000) 2022-09-10T14:35:06.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:06.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:06.568130+0000) 2022-09-10T14:35:07.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:06 smithi104 ceph-mon[97044]: pgmap v19384: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:35:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:06.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:06.946359+0000) 2022-09-10T14:35:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:06 smithi110 ceph-mon[97633]: pgmap v19384: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:35:07.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:07.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:07.568297+0000) 2022-09-10T14:35:08.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:07.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:07.946585+0000) 2022-09-10T14:35:08.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:08.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:08.568518+0000) 2022-09-10T14:35:09.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:08 smithi104 ceph-mon[97044]: pgmap v19385: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:35:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:08 smithi104 ceph-mon[97044]: osdmap e1292: 12 total, 12 up, 12 in 2022-09-10T14:35:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:08.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:08.946855+0000) 2022-09-10T14:35:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:08 smithi110 ceph-mon[97633]: pgmap v19385: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:35:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:08 smithi110 ceph-mon[97633]: osdmap e1292: 12 total, 12 up, 12 in 2022-09-10T14:35:09.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:09.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:09.568743+0000) 2022-09-10T14:35:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:09.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:09.947054+0000) 2022-09-10T14:35:10.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:10.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:10.569007+0000) 2022-09-10T14:35:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:10 smithi104 ceph-mon[97044]: pgmap v19387: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:35:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:10.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:10.947282+0000) 2022-09-10T14:35:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:10 smithi110 ceph-mon[97633]: pgmap v19387: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:35:11.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:11.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:11.569226+0000) 2022-09-10T14:35:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:11.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:11.947511+0000) 2022-09-10T14:35:12.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:12.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:12.569451+0000) 2022-09-10T14:35:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:12 smithi104 ceph-mon[97044]: pgmap v19388: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:35:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:12.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:12.947758+0000) 2022-09-10T14:35:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:12 smithi110 ceph-mon[97633]: pgmap v19388: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:35:13.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:13.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:13.569693+0000) 2022-09-10T14:35:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:13.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:13.947986+0000) 2022-09-10T14:35:14.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:14.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:14.569947+0000) 2022-09-10T14:35:15.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:14 smithi104 ceph-mon[97044]: pgmap v19389: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:35:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:14.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:14.948168+0000) 2022-09-10T14:35:15.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:14 smithi110 ceph-mon[97633]: pgmap v19389: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:35:15.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:15.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:15.570074+0000) 2022-09-10T14:35:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:15.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:15.948389+0000) 2022-09-10T14:35:16.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:16.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:16.570273+0000) 2022-09-10T14:35:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:16 smithi104 ceph-mon[97044]: pgmap v19390: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:35:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:16 smithi104 ceph-mon[97044]: osdmap e1293: 12 total, 12 up, 12 in 2022-09-10T14:35:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:16.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:16.948616+0000) 2022-09-10T14:35:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:16 smithi110 ceph-mon[97633]: pgmap v19390: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:35:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:16 smithi110 ceph-mon[97633]: osdmap e1293: 12 total, 12 up, 12 in 2022-09-10T14:35:17.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:17.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:17.570510+0000) 2022-09-10T14:35:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:17.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:17.948860+0000) 2022-09-10T14:35:18.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:18.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:18.570713+0000) 2022-09-10T14:35:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:18 smithi104 ceph-mon[97044]: pgmap v19392: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 216 B/s wr, 10 op/s 2022-09-10T14:35:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:18.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:18.949001+0000) 2022-09-10T14:35:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:18 smithi110 ceph-mon[97633]: pgmap v19392: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 216 B/s wr, 10 op/s 2022-09-10T14:35:19.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:19.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:19.570898+0000) 2022-09-10T14:35:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:19.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:19.949225+0000) 2022-09-10T14:35:20.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:35:20.448+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:35:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:20.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:20.571084+0000) 2022-09-10T14:35:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:20 smithi104 ceph-mon[97044]: pgmap v19393: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:35:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:20.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:20.949477+0000) 2022-09-10T14:35:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:20 smithi110 ceph-mon[97633]: pgmap v19393: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:35:21.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:21.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:21.571273+0000) 2022-09-10T14:35:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:21.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:21.949636+0000) 2022-09-10T14:35:22.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:22.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:22.571445+0000) 2022-09-10T14:35:23.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:22 smithi104 ceph-mon[97044]: pgmap v19394: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:35:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:22.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:22.949839+0000) 2022-09-10T14:35:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:22 smithi110 ceph-mon[97633]: pgmap v19394: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:35:23.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:23.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:23.571588+0000) 2022-09-10T14:35:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:23.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:23.950068+0000) 2022-09-10T14:35:24.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:24.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:24.571745+0000) 2022-09-10T14:35:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:24 smithi104 ceph-mon[97044]: pgmap v19395: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:35:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:24.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:24.950326+0000) 2022-09-10T14:35:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:24 smithi110 ceph-mon[97633]: pgmap v19395: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:35:25.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:25.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:25.571904+0000) 2022-09-10T14:35:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:25.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:25.950490+0000) 2022-09-10T14:35:26.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:26.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:26.572146+0000) 2022-09-10T14:35:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:26 smithi104 ceph-mon[97044]: pgmap v19396: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:35:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:26.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:26.950737+0000) 2022-09-10T14:35:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:26 smithi110 ceph-mon[97633]: pgmap v19396: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:35:27.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:27.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:27.572335+0000) 2022-09-10T14:35:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:27.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:27.950991+0000) 2022-09-10T14:35:28.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:28.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:28.572461+0000) 2022-09-10T14:35:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:28 smithi104 ceph-mon[97044]: pgmap v19397: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:35:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:28.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:28.951151+0000) 2022-09-10T14:35:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:28 smithi110 ceph-mon[97633]: pgmap v19397: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:35:29.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:29.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:29.572610+0000) 2022-09-10T14:35:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:29.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:29.951388+0000) 2022-09-10T14:35:30.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:30.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:30.572737+0000) 2022-09-10T14:35:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:30 smithi104 ceph-mon[97044]: pgmap v19398: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:30.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:30.951649+0000) 2022-09-10T14:35:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:30 smithi110 ceph-mon[97633]: pgmap v19398: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:31.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:31.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:31.572935+0000) 2022-09-10T14:35:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:31.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:31.951886+0000) 2022-09-10T14:35:32.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:32.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:32.573173+0000) 2022-09-10T14:35:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:32 smithi104 ceph-mon[97044]: pgmap v19399: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:32.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:32.952143+0000) 2022-09-10T14:35:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:32 smithi110 ceph-mon[97633]: pgmap v19399: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:33.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:33.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:33.573385+0000) 2022-09-10T14:35:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:33.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:33.952328+0000) 2022-09-10T14:35:34.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:34.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:34.573584+0000) 2022-09-10T14:35:35.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:34 smithi104 ceph-mon[97044]: pgmap v19400: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:34.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:34.952526+0000) 2022-09-10T14:35:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:34 smithi110 ceph-mon[97633]: pgmap v19400: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:35.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:35.573788+0000) 2022-09-10T14:35:35.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:35:35.463+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:35:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:35.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:35.952665+0000) 2022-09-10T14:35:36.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:36.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:36.573974+0000) 2022-09-10T14:35:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:37 smithi104 ceph-mon[97044]: pgmap v19401: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:35:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:36.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:36.952877+0000) 2022-09-10T14:35:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:37 smithi110 ceph-mon[97633]: pgmap v19401: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:35:37.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:37.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:37.574177+0000) 2022-09-10T14:35:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:37.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:37.953139+0000) 2022-09-10T14:35:38.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:38.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:38.574382+0000) 2022-09-10T14:35:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:39 smithi104 ceph-mon[97044]: pgmap v19402: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:38.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:38.953362+0000) 2022-09-10T14:35:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:39 smithi110 ceph-mon[97633]: pgmap v19402: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:39.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:39.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:39.574579+0000) 2022-09-10T14:35:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:39.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:39.953600+0000) 2022-09-10T14:35:40.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:40.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:40.574810+0000) 2022-09-10T14:35:41.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:41 smithi104 ceph-mon[97044]: pgmap v19403: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:40.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:40.953857+0000) 2022-09-10T14:35:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:41 smithi110 ceph-mon[97633]: pgmap v19403: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:41.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:41.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:41.575075+0000) 2022-09-10T14:35:42.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:41.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:41.954024+0000) 2022-09-10T14:35:42.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:42.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:42.575352+0000) 2022-09-10T14:35:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:43 smithi104 ceph-mon[97044]: pgmap v19404: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:42.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:42.954216+0000) 2022-09-10T14:35:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:43 smithi110 ceph-mon[97633]: pgmap v19404: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:43.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:43.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:43.575526+0000) 2022-09-10T14:35:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:43.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:43.954458+0000) 2022-09-10T14:35:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:44 smithi110 ceph-mon[97633]: pgmap v19405: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:44 smithi104 ceph-mon[97044]: pgmap v19405: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:44.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:44.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:44.575687+0000) 2022-09-10T14:35:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:44.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:44.954743+0000) 2022-09-10T14:35:45.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:45.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:45.575939+0000) 2022-09-10T14:35:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:45.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:45.954944+0000) 2022-09-10T14:35:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:46 smithi104 ceph-mon[97044]: pgmap v19406: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:35:46.907 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:46 smithi110 ceph-mon[97633]: pgmap v19406: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:35:46.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:46.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:46.576108+0000) 2022-09-10T14:35:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:46.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:46.955177+0000) 2022-09-10T14:35:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:35:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:35:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:35:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:35:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:35:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:35:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:35:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:35:47.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:47.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:47.576317+0000) 2022-09-10T14:35:48.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:47.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:47.955413+0000) 2022-09-10T14:35:48.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:48.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:48.576525+0000) 2022-09-10T14:35:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:48 smithi104 ceph-mon[97044]: pgmap v19407: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:48 smithi110 ceph-mon[97633]: pgmap v19407: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:48.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:48.955667+0000) 2022-09-10T14:35:49.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:49.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:49.576792+0000) 2022-09-10T14:35:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:49.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:49.955865+0000) 2022-09-10T14:35:50.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:35:50.476+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:35:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:50.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:50.577062+0000) 2022-09-10T14:35:51.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:50 smithi104 ceph-mon[97044]: pgmap v19408: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:50 smithi110 ceph-mon[97633]: pgmap v19408: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:35:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:50.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:50.956155+0000) 2022-09-10T14:35:51.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:51.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:51.577310+0000) 2022-09-10T14:35:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:51.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:51.956373+0000) 2022-09-10T14:35:52.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:52.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:52.577484+0000) 2022-09-10T14:35:53.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:52 smithi104 ceph-mon[97044]: pgmap v19409: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:52 smithi110 ceph-mon[97633]: pgmap v19409: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:35:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:52.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:52.956519+0000) 2022-09-10T14:35:53.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:53.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:53.577736+0000) 2022-09-10T14:35:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:53.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:53.956733+0000) 2022-09-10T14:35:54.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:54.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:54.577993+0000) 2022-09-10T14:35:55.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:54 smithi104 ceph-mon[97044]: pgmap v19410: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:54 smithi110 ceph-mon[97633]: pgmap v19410: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:54.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:54.956984+0000) 2022-09-10T14:35:55.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:55.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:55.578229+0000) 2022-09-10T14:35:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:55.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:55.957229+0000) 2022-09-10T14:35:56.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:56.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:56.578506+0000) 2022-09-10T14:35:57.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:56 smithi104 ceph-mon[97044]: pgmap v19411: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:35:57.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:56 smithi110 ceph-mon[97633]: pgmap v19411: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:35:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:56.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:56.957413+0000) 2022-09-10T14:35:57.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:57.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:57.578726+0000) 2022-09-10T14:35:58.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:57.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:57.957608+0000) 2022-09-10T14:35:58.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:58.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:58.578962+0000) 2022-09-10T14:35:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:35:58 smithi104 ceph-mon[97044]: pgmap v19412: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:35:58 smithi110 ceph-mon[97633]: pgmap v19412: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:35:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:58.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:58.957752+0000) 2022-09-10T14:35:59.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:35:59.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:59.579174+0000) 2022-09-10T14:36:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:35:59.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:35:59.957928+0000) 2022-09-10T14:36:00.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:00.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:00.579411+0000) 2022-09-10T14:36:01.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:00 smithi104 ceph-mon[97044]: pgmap v19413: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:36:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:00 smithi110 ceph-mon[97633]: pgmap v19413: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T14:36:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:00.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:00.958090+0000) 2022-09-10T14:36:01.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:01.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:01.579654+0000) 2022-09-10T14:36:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:01.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:01.958336+0000) 2022-09-10T14:36:02.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:02.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:02.579890+0000) 2022-09-10T14:36:03.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:02 smithi104 ceph-mon[97044]: pgmap v19414: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:36:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:02 smithi110 ceph-mon[97633]: pgmap v19414: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T14:36:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:02.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:02.958612+0000) 2022-09-10T14:36:03.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:03.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:03.580183+0000) 2022-09-10T14:36:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:03.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:03.958874+0000) 2022-09-10T14:36:04.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:04.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:04.580383+0000) 2022-09-10T14:36:05.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:04 smithi104 ceph-mon[97044]: pgmap v19415: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:04 smithi110 ceph-mon[97633]: pgmap v19415: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:04.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:04.959136+0000) 2022-09-10T14:36:05.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:05.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:05.580595+0000) 2022-09-10T14:36:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:36:05.490+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:36:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:05.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:05.959383+0000) 2022-09-10T14:36:06.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:06.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:06.580845+0000) 2022-09-10T14:36:07.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:06 smithi104 ceph-mon[97044]: pgmap v19416: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:36:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:06 smithi110 ceph-mon[97633]: pgmap v19416: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:36:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:06.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:06.959634+0000) 2022-09-10T14:36:07.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:07.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:07.581124+0000) 2022-09-10T14:36:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:07.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:07.959890+0000) 2022-09-10T14:36:08.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:08.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:08.581303+0000) 2022-09-10T14:36:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:08 smithi110 ceph-mon[97633]: pgmap v19417: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:36:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:08 smithi104 ceph-mon[97044]: pgmap v19417: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:36:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:08.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:08.960128+0000) 2022-09-10T14:36:09.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:09.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:09.581543+0000) 2022-09-10T14:36:10.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:09 smithi110 ceph-mon[97633]: osdmap e1294: 12 total, 12 up, 12 in 2022-09-10T14:36:10.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:09 smithi104 ceph-mon[97044]: osdmap e1294: 12 total, 12 up, 12 in 2022-09-10T14:36:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:09.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:09.960357+0000) 2022-09-10T14:36:10.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:10.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:10.581789+0000) 2022-09-10T14:36:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:10 smithi110 ceph-mon[97633]: pgmap v19419: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:36:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:10 smithi104 ceph-mon[97044]: pgmap v19419: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:36:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:10.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:10.960586+0000) 2022-09-10T14:36:11.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:11.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:11.582042+0000) 2022-09-10T14:36:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:11.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:11.960835+0000) 2022-09-10T14:36:12.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:12.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:12.582296+0000) 2022-09-10T14:36:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:12 smithi110 ceph-mon[97633]: pgmap v19420: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:36:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:12 smithi104 ceph-mon[97044]: pgmap v19420: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:36:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:12.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:12.961016+0000) 2022-09-10T14:36:13.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:13.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:13.582505+0000) 2022-09-10T14:36:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:13.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:13.961230+0000) 2022-09-10T14:36:14.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:14.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:14.582708+0000) 2022-09-10T14:36:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:14 smithi110 ceph-mon[97633]: pgmap v19421: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:36:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:14 smithi104 ceph-mon[97044]: pgmap v19421: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:36:15.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:14.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:14.961486+0000) 2022-09-10T14:36:15.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:15.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:15.582882+0000) 2022-09-10T14:36:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:15.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:15.961750+0000) 2022-09-10T14:36:16.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:16.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:16.583063+0000) 2022-09-10T14:36:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:16 smithi110 ceph-mon[97633]: pgmap v19422: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:36:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:16 smithi110 ceph-mon[97633]: osdmap e1295: 12 total, 12 up, 12 in 2022-09-10T14:36:17.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:16 smithi104 ceph-mon[97044]: pgmap v19422: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T14:36:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:16 smithi104 ceph-mon[97044]: osdmap e1295: 12 total, 12 up, 12 in 2022-09-10T14:36:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:16.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:16.962022+0000) 2022-09-10T14:36:17.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:17.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:17.583259+0000) 2022-09-10T14:36:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:17.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:17.962290+0000) 2022-09-10T14:36:18.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:18.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:18.583503+0000) 2022-09-10T14:36:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:18 smithi110 ceph-mon[97633]: pgmap v19424: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T14:36:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:18 smithi104 ceph-mon[97044]: pgmap v19424: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 236 B/s wr, 12 op/s 2022-09-10T14:36:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:18.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:18.962476+0000) 2022-09-10T14:36:19.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:19.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:19.583770+0000) 2022-09-10T14:36:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:19.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:19.962725+0000) 2022-09-10T14:36:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:36:20.505+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:36:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:20.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:20.584007+0000) 2022-09-10T14:36:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:20 smithi110 ceph-mon[97633]: pgmap v19425: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2022-09-10T14:36:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:20 smithi104 ceph-mon[97044]: pgmap v19425: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 11 op/s 2022-09-10T14:36:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:20.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:20.962997+0000) 2022-09-10T14:36:21.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:21.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:21.584242+0000) 2022-09-10T14:36:22.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:21.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:21.963166+0000) 2022-09-10T14:36:22.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:22.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:22.584480+0000) 2022-09-10T14:36:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:22 smithi110 ceph-mon[97633]: pgmap v19426: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:36:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:22 smithi104 ceph-mon[97044]: pgmap v19426: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:36:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:22.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:22.963351+0000) 2022-09-10T14:36:23.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:23.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:23.584717+0000) 2022-09-10T14:36:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:23.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:23.963585+0000) 2022-09-10T14:36:24.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:24.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:24.584976+0000) 2022-09-10T14:36:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:24 smithi110 ceph-mon[97633]: pgmap v19427: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:36:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:24 smithi104 ceph-mon[97044]: pgmap v19427: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:36:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:24.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:24.963816+0000) 2022-09-10T14:36:25.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:25.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:25.585124+0000) 2022-09-10T14:36:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:25.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:25.963993+0000) 2022-09-10T14:36:26.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:26.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:26.585267+0000) 2022-09-10T14:36:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:26 smithi110 ceph-mon[97633]: pgmap v19428: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:36:27.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:26 smithi104 ceph-mon[97044]: pgmap v19428: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T14:36:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:26.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:26.964221+0000) 2022-09-10T14:36:27.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:27.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:27.585491+0000) 2022-09-10T14:36:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:27.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:27.964468+0000) 2022-09-10T14:36:28.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:28.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:28.585651+0000) 2022-09-10T14:36:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:28 smithi110 ceph-mon[97633]: pgmap v19429: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:36:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:28 smithi104 ceph-mon[97044]: pgmap v19429: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2022-09-10T14:36:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:28.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:28.964702+0000) 2022-09-10T14:36:29.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:29.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:29.585831+0000) 2022-09-10T14:36:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:29.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:29.964880+0000) 2022-09-10T14:36:30.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:30.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:30.586019+0000) 2022-09-10T14:36:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:30 smithi110 ceph-mon[97633]: pgmap v19430: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:30 smithi104 ceph-mon[97044]: pgmap v19430: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:30.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:30.965142+0000) 2022-09-10T14:36:31.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:31.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:31.586233+0000) 2022-09-10T14:36:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:31.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:31.965305+0000) 2022-09-10T14:36:32.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:32.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:32.586452+0000) 2022-09-10T14:36:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:32 smithi110 ceph-mon[97633]: pgmap v19431: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:32 smithi104 ceph-mon[97044]: pgmap v19431: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:32.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:32.965532+0000) 2022-09-10T14:36:33.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:33.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:33.586616+0000) 2022-09-10T14:36:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:33.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:33.965678+0000) 2022-09-10T14:36:34.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:34.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:34.586831+0000) 2022-09-10T14:36:35.177 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:34 smithi110 ceph-mon[97633]: pgmap v19432: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:35.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:34 smithi104 ceph-mon[97044]: pgmap v19432: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:34.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:34.965908+0000) 2022-09-10T14:36:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:36:35.520+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:36:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:35.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:35.587068+0000) 2022-09-10T14:36:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:35.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:35.966166+0000) 2022-09-10T14:36:36.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:36.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:36.587277+0000) 2022-09-10T14:36:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:36 smithi110 ceph-mon[97633]: pgmap v19433: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:36 smithi104 ceph-mon[97044]: pgmap v19433: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:36.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:36.966343+0000) 2022-09-10T14:36:37.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:37.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:37.587515+0000) 2022-09-10T14:36:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:37.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:37.966563+0000) 2022-09-10T14:36:38.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:38.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:38.587747+0000) 2022-09-10T14:36:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:38 smithi110 ceph-mon[97633]: pgmap v19434: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:38 smithi104 ceph-mon[97044]: pgmap v19434: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:38.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:38.966825+0000) 2022-09-10T14:36:39.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:39.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:39.587931+0000) 2022-09-10T14:36:40.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:39.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:39.966968+0000) 2022-09-10T14:36:40.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:40.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:40.588173+0000) 2022-09-10T14:36:41.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:40 smithi110 ceph-mon[97633]: pgmap v19435: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:40 smithi104 ceph-mon[97044]: pgmap v19435: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:40.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:40.967204+0000) 2022-09-10T14:36:41.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:41.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:41.588345+0000) 2022-09-10T14:36:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:41.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:41.967456+0000) 2022-09-10T14:36:42.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:42.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:42.588580+0000) 2022-09-10T14:36:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:42 smithi110 ceph-mon[97633]: pgmap v19436: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:43.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:42 smithi104 ceph-mon[97044]: pgmap v19436: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:42.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:42.967632+0000) 2022-09-10T14:36:43.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:43.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:43.588831+0000) 2022-09-10T14:36:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:43.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:43.967861+0000) 2022-09-10T14:36:44.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:44.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:44.589021+0000) 2022-09-10T14:36:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:44 smithi110 ceph-mon[97633]: pgmap v19437: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:44 smithi104 ceph-mon[97044]: pgmap v19437: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:44.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:44.968134+0000) 2022-09-10T14:36:45.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:45.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:45.589269+0000) 2022-09-10T14:36:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:45.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:45.968322+0000) 2022-09-10T14:36:46.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:46.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:46.589493+0000) 2022-09-10T14:36:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:46 smithi110 ceph-mon[97633]: pgmap v19438: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:36:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:36:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:36:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:36:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:46 smithi104 ceph-mon[97044]: pgmap v19438: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:36:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:36:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:36:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:36:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:46.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:46.968527+0000) 2022-09-10T14:36:47.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:47.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:47.589720+0000) 2022-09-10T14:36:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:47.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:47.968781+0000) 2022-09-10T14:36:48.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:48.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:48.589965+0000) 2022-09-10T14:36:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:48 smithi110 ceph-mon[97633]: pgmap v19439: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:48 smithi104 ceph-mon[97044]: pgmap v19439: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:36:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:48.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:48.969032+0000) 2022-09-10T14:36:49.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:49.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:49.590202+0000) 2022-09-10T14:36:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:49.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:49.969290+0000) 2022-09-10T14:36:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:50.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:50.590413+0000) 2022-09-10T14:36:50.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:36:50.535+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:36:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:50 smithi110 ceph-mon[97633]: pgmap v19440: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:51.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:50 smithi104 ceph-mon[97044]: pgmap v19440: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:50.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:50.969459+0000) 2022-09-10T14:36:51.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:51.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:51.590672+0000) 2022-09-10T14:36:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:51.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:51.969609+0000) 2022-09-10T14:36:52.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:52.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:52.590867+0000) 2022-09-10T14:36:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:52 smithi110 ceph-mon[97633]: pgmap v19441: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:36:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:52 smithi104 ceph-mon[97044]: pgmap v19441: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:36:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:52.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:52.969823+0000) 2022-09-10T14:36:53.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:53.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:53.591147+0000) 2022-09-10T14:36:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:53.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:53.970003+0000) 2022-09-10T14:36:54.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:54.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:54.591353+0000) 2022-09-10T14:36:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:54 smithi110 ceph-mon[97633]: pgmap v19442: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:36:55.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:54 smithi104 ceph-mon[97044]: pgmap v19442: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:36:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:54.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:54.970191+0000) 2022-09-10T14:36:55.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:55.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:55.591563+0000) 2022-09-10T14:36:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:55.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:55.970383+0000) 2022-09-10T14:36:56.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:56.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:56.591799+0000) 2022-09-10T14:36:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:56 smithi110 ceph-mon[97633]: pgmap v19443: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:56 smithi104 ceph-mon[97044]: pgmap v19443: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:36:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:56.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:56.970607+0000) 2022-09-10T14:36:57.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:57.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:57.592024+0000) 2022-09-10T14:36:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:57.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:57.970752+0000) 2022-09-10T14:36:58.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:58.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:58.592211+0000) 2022-09-10T14:36:59.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:36:58 smithi110 ceph-mon[97633]: pgmap v19444: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:36:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:36:58 smithi104 ceph-mon[97044]: pgmap v19444: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:36:59.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:58.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:58.970912+0000) 2022-09-10T14:36:59.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:36:59.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:59.592408+0000) 2022-09-10T14:37:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:36:59.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:36:59.971194+0000) 2022-09-10T14:37:00.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:00.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:00.592632+0000) 2022-09-10T14:37:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:00 smithi110 ceph-mon[97633]: pgmap v19445: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:00 smithi104 ceph-mon[97044]: pgmap v19445: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:00.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:00.971369+0000) 2022-09-10T14:37:01.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:01.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:01.592776+0000) 2022-09-10T14:37:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:01.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:01.971575+0000) 2022-09-10T14:37:02.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:02.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:02.592934+0000) 2022-09-10T14:37:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:02 smithi110 ceph-mon[97633]: pgmap v19446: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:02 smithi104 ceph-mon[97044]: pgmap v19446: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:03.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:02.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:02.971825+0000) 2022-09-10T14:37:03.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:03.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:03.593127+0000) 2022-09-10T14:37:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:03.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:03.972080+0000) 2022-09-10T14:37:04.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:04.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:04.593328+0000) 2022-09-10T14:37:05.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:04 smithi110 ceph-mon[97633]: pgmap v19447: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:04 smithi104 ceph-mon[97044]: pgmap v19447: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:04.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:04.972354+0000) 2022-09-10T14:37:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:05.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:05.593505+0000) 2022-09-10T14:37:05.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:37:05.538+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:37:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:05.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:05.972617+0000) 2022-09-10T14:37:06.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:06.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:06.593721+0000) 2022-09-10T14:37:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:06 smithi110 ceph-mon[97633]: pgmap v19448: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:37:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:06 smithi104 ceph-mon[97044]: pgmap v19448: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:37:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:06.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:06.972855+0000) 2022-09-10T14:37:07.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:07.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:07.593895+0000) 2022-09-10T14:37:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:07.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:07.973149+0000) 2022-09-10T14:37:08.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:08.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:08.594108+0000) 2022-09-10T14:37:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:08 smithi110 ceph-mon[97633]: pgmap v19449: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:37:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:08 smithi110 ceph-mon[97633]: osdmap e1296: 12 total, 12 up, 12 in 2022-09-10T14:37:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:08 smithi104 ceph-mon[97044]: pgmap v19449: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:37:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:08 smithi104 ceph-mon[97044]: osdmap e1296: 12 total, 12 up, 12 in 2022-09-10T14:37:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:08.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:08.973366+0000) 2022-09-10T14:37:09.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:09.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:09.594296+0000) 2022-09-10T14:37:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:09.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:09.973594+0000) 2022-09-10T14:37:10.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:10.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:10.594480+0000) 2022-09-10T14:37:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:10 smithi110 ceph-mon[97633]: pgmap v19451: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:37:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:10 smithi104 ceph-mon[97044]: pgmap v19451: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:37:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:10.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:10.973835+0000) 2022-09-10T14:37:11.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:11.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:11.594748+0000) 2022-09-10T14:37:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:11.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:11.974062+0000) 2022-09-10T14:37:12.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:12.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:12.594880+0000) 2022-09-10T14:37:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:12 smithi104 ceph-mon[97044]: pgmap v19452: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:37:13.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:12 smithi104 ceph-mon[97044]: osdmap e1297: 12 total, 12 up, 12 in 2022-09-10T14:37:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:12 smithi110 ceph-mon[97633]: pgmap v19452: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:37:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:12 smithi110 ceph-mon[97633]: osdmap e1297: 12 total, 12 up, 12 in 2022-09-10T14:37:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:12.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:12.974280+0000) 2022-09-10T14:37:13.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:13.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:13.595145+0000) 2022-09-10T14:37:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:13.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:13.974462+0000) 2022-09-10T14:37:14.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:14.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:14.595376+0000) 2022-09-10T14:37:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:14 smithi104 ceph-mon[97044]: pgmap v19454: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:37:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:14 smithi110 ceph-mon[97633]: pgmap v19454: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:37:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:14.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:14.974672+0000) 2022-09-10T14:37:15.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:15.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:15.595545+0000) 2022-09-10T14:37:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:15.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:15.974897+0000) 2022-09-10T14:37:16.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:16.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:16.595776+0000) 2022-09-10T14:37:17.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:16 smithi104 ceph-mon[97044]: pgmap v19455: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:37:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:16 smithi110 ceph-mon[97633]: pgmap v19455: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:37:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:16.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:16.975153+0000) 2022-09-10T14:37:17.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:17.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:17.595997+0000) 2022-09-10T14:37:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:17.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:17.975322+0000) 2022-09-10T14:37:18.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:18.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:18.596245+0000) 2022-09-10T14:37:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:18 smithi110 ceph-mon[97633]: pgmap v19456: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T14:37:19.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:18 smithi104 ceph-mon[97044]: pgmap v19456: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 214 B/s wr, 10 op/s 2022-09-10T14:37:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:18.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:18.975533+0000) 2022-09-10T14:37:19.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:19.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:19.596440+0000) 2022-09-10T14:37:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:19.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:19.975736+0000) 2022-09-10T14:37:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:20.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:20.596648+0000) 2022-09-10T14:37:20.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:37:20.543+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:37:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:20 smithi110 ceph-mon[97633]: pgmap v19457: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:37:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:20 smithi104 ceph-mon[97044]: pgmap v19457: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:37:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:20.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:20.975975+0000) 2022-09-10T14:37:21.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:21.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:21.596894+0000) 2022-09-10T14:37:22.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:21.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:21.976234+0000) 2022-09-10T14:37:22.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:22.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:22.597125+0000) 2022-09-10T14:37:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:22 smithi110 ceph-mon[97633]: pgmap v19458: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:37:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:22 smithi104 ceph-mon[97044]: pgmap v19458: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:37:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:22.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:22.976439+0000) 2022-09-10T14:37:23.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:23.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:23.597350+0000) 2022-09-10T14:37:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:23.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:23.976699+0000) 2022-09-10T14:37:24.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:24.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:24.597596+0000) 2022-09-10T14:37:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:24 smithi110 ceph-mon[97633]: pgmap v19459: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:37:25.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:24 smithi104 ceph-mon[97044]: pgmap v19459: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:37:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:24.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:24.976948+0000) 2022-09-10T14:37:25.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:25.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:25.597820+0000) 2022-09-10T14:37:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:25.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:25.977214+0000) 2022-09-10T14:37:26.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:26.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:26.598067+0000) 2022-09-10T14:37:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:26 smithi110 ceph-mon[97633]: pgmap v19460: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:37:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:26 smithi104 ceph-mon[97044]: pgmap v19460: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:37:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:26.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:26.977422+0000) 2022-09-10T14:37:27.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:27.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:27.598319+0000) 2022-09-10T14:37:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:27.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:27.977722+0000) 2022-09-10T14:37:28.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:28.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:28.598500+0000) 2022-09-10T14:37:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:28 smithi110 ceph-mon[97633]: pgmap v19461: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:28 smithi104 ceph-mon[97044]: pgmap v19461: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:28.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:28.977959+0000) 2022-09-10T14:37:29.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:29.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:29.598702+0000) 2022-09-10T14:37:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:29.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:29.978166+0000) 2022-09-10T14:37:30.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:30.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:30.598911+0000) 2022-09-10T14:37:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:30 smithi110 ceph-mon[97633]: pgmap v19462: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:30 smithi104 ceph-mon[97044]: pgmap v19462: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:30.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:30.978366+0000) 2022-09-10T14:37:31.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:31.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:31.599173+0000) 2022-09-10T14:37:32.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:31.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:31.978547+0000) 2022-09-10T14:37:32.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:32.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:32.599387+0000) 2022-09-10T14:37:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:32 smithi110 ceph-mon[97633]: pgmap v19463: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:32 smithi104 ceph-mon[97044]: pgmap v19463: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:33.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:32.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:32.978766+0000) 2022-09-10T14:37:33.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:33.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:33.599608+0000) 2022-09-10T14:37:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:33.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:33.979016+0000) 2022-09-10T14:37:34.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:34.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:34.599874+0000) 2022-09-10T14:37:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:34 smithi110 ceph-mon[97633]: pgmap v19464: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:34 smithi104 ceph-mon[97044]: pgmap v19464: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:34.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:34.979232+0000) 2022-09-10T14:37:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:35.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:35.600177+0000) 2022-09-10T14:37:35.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:37:35.550+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:37:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:35.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:35.979475+0000) 2022-09-10T14:37:36.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:36.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:36.600393+0000) 2022-09-10T14:37:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:36 smithi110 ceph-mon[97633]: pgmap v19465: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:36 smithi104 ceph-mon[97044]: pgmap v19465: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:36.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:36.979686+0000) 2022-09-10T14:37:37.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:37.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:37.600612+0000) 2022-09-10T14:37:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:37.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:37.979929+0000) 2022-09-10T14:37:38.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:38.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:38.600880+0000) 2022-09-10T14:37:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:38 smithi110 ceph-mon[97633]: pgmap v19466: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:39.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:38 smithi104 ceph-mon[97044]: pgmap v19466: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:38.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:38.980211+0000) 2022-09-10T14:37:39.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:39.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:39.601171+0000) 2022-09-10T14:37:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:39.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:39.980451+0000) 2022-09-10T14:37:40.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:40.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:40.601336+0000) 2022-09-10T14:37:41.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:40 smithi110 ceph-mon[97633]: pgmap v19467: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:40 smithi104 ceph-mon[97044]: pgmap v19467: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:40.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:40.980658+0000) 2022-09-10T14:37:41.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:41.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:41.601546+0000) 2022-09-10T14:37:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:41.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:41.980889+0000) 2022-09-10T14:37:42.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:42.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:42.601716+0000) 2022-09-10T14:37:43.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:42 smithi110 ceph-mon[97633]: pgmap v19468: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:42 smithi104 ceph-mon[97044]: pgmap v19468: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:42.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:42.981156+0000) 2022-09-10T14:37:43.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:43.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:43.601931+0000) 2022-09-10T14:37:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:43.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:43.981345+0000) 2022-09-10T14:37:44.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:44.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:44.602151+0000) 2022-09-10T14:37:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:44 smithi110 ceph-mon[97633]: pgmap v19469: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:44 smithi104 ceph-mon[97044]: pgmap v19469: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:44.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:44.981586+0000) 2022-09-10T14:37:45.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:45.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:45.602384+0000) 2022-09-10T14:37:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:45.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:45.981836+0000) 2022-09-10T14:37:46.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:46.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:46.602586+0000) 2022-09-10T14:37:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:46 smithi110 ceph-mon[97633]: pgmap v19470: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:37:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:37:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:37:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:37:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:46 smithi104 ceph-mon[97044]: pgmap v19470: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:37:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:37:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:37:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:37:47.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:46.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:46.981991+0000) 2022-09-10T14:37:47.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:47.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:47.602846+0000) 2022-09-10T14:37:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:47.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:47.982222+0000) 2022-09-10T14:37:48.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:48.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:48.603119+0000) 2022-09-10T14:37:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:48 smithi110 ceph-mon[97633]: pgmap v19471: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:48 smithi104 ceph-mon[97044]: pgmap v19471: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:48.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:48.982432+0000) 2022-09-10T14:37:49.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:49.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:49.603316+0000) 2022-09-10T14:37:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:49.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:49.982652+0000) 2022-09-10T14:37:50.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:50.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:50.603482+0000) 2022-09-10T14:37:50.984 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:37:50.552+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:37:51.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:50 smithi110 ceph-mon[97633]: pgmap v19472: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:50 smithi104 ceph-mon[97044]: pgmap v19472: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:50.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:50.982903+0000) 2022-09-10T14:37:51.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:51.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:51.603688+0000) 2022-09-10T14:37:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:51.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:51.983157+0000) 2022-09-10T14:37:52.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:52.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:52.603869+0000) 2022-09-10T14:37:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:52 smithi110 ceph-mon[97633]: pgmap v19473: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:53.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:52 smithi104 ceph-mon[97044]: pgmap v19473: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:52.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:52.983311+0000) 2022-09-10T14:37:53.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:53.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:53.604062+0000) 2022-09-10T14:37:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:53.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:53.983558+0000) 2022-09-10T14:37:54.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:54.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:54.604329+0000) 2022-09-10T14:37:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:54 smithi110 ceph-mon[97633]: pgmap v19474: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:54 smithi104 ceph-mon[97044]: pgmap v19474: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:54.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:54.983796+0000) 2022-09-10T14:37:55.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:55.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:55.604574+0000) 2022-09-10T14:37:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:55.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:55.984063+0000) 2022-09-10T14:37:56.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:56.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:56.604805+0000) 2022-09-10T14:37:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:56 smithi110 ceph-mon[97633]: pgmap v19475: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:56 smithi104 ceph-mon[97044]: pgmap v19475: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:37:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:56.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:56.984295+0000) 2022-09-10T14:37:57.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:57.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:57.604947+0000) 2022-09-10T14:37:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:57.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:57.984454+0000) 2022-09-10T14:37:58.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:58.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:58.605166+0000) 2022-09-10T14:37:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:37:58 smithi110 ceph-mon[97633]: pgmap v19476: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:37:58 smithi104 ceph-mon[97044]: pgmap v19476: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:37:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:58.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:58.984611+0000) 2022-09-10T14:37:59.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:37:59.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:59.605331+0000) 2022-09-10T14:38:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:37:59.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:37:59.984791+0000) 2022-09-10T14:38:00.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:00.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:00.605531+0000) 2022-09-10T14:38:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:00 smithi110 ceph-mon[97633]: pgmap v19477: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:00 smithi104 ceph-mon[97044]: pgmap v19477: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:00.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:00.984990+0000) 2022-09-10T14:38:01.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:01.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:01.605748+0000) 2022-09-10T14:38:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:01.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:01.985214+0000) 2022-09-10T14:38:02.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:02.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:02.606020+0000) 2022-09-10T14:38:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:02 smithi110 ceph-mon[97633]: pgmap v19478: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:02 smithi104 ceph-mon[97044]: pgmap v19478: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:02.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:02.985506+0000) 2022-09-10T14:38:03.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:03.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:03.606285+0000) 2022-09-10T14:38:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:03.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:03.985667+0000) 2022-09-10T14:38:04.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:04.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:04.606553+0000) 2022-09-10T14:38:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:04 smithi110 ceph-mon[97633]: pgmap v19479: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:04 smithi104 ceph-mon[97044]: pgmap v19479: 129 pgs: 129 active+clean; 9.9 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:04.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:04.985832+0000) 2022-09-10T14:38:05.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:05.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:05.606764+0000) 2022-09-10T14:38:05.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:38:05.553+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:38:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:05.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:05.986009+0000) 2022-09-10T14:38:06.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:06.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:06.606937+0000) 2022-09-10T14:38:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:06 smithi110 ceph-mon[97633]: pgmap v19480: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:38:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:06 smithi104 ceph-mon[97044]: pgmap v19480: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:38:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:06.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:06.986259+0000) 2022-09-10T14:38:07.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:07.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:07.607115+0000) 2022-09-10T14:38:07.990 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:07.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:07.986514+0000) 2022-09-10T14:38:08.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:08.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:08.607230+0000) 2022-09-10T14:38:09.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:08 smithi110 ceph-mon[97633]: pgmap v19481: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:38:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:08 smithi110 ceph-mon[97633]: osdmap e1298: 12 total, 12 up, 12 in 2022-09-10T14:38:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:08 smithi104 ceph-mon[97044]: pgmap v19481: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:38:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:08 smithi104 ceph-mon[97044]: osdmap e1298: 12 total, 12 up, 12 in 2022-09-10T14:38:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:08.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:08.986680+0000) 2022-09-10T14:38:09.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:09.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:09.607447+0000) 2022-09-10T14:38:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:09.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:09.986859+0000) 2022-09-10T14:38:10.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:10.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:10.607691+0000) 2022-09-10T14:38:11.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:10 smithi110 ceph-mon[97633]: pgmap v19483: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:38:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:10 smithi104 ceph-mon[97044]: pgmap v19483: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:38:11.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:10.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:10.987125+0000) 2022-09-10T14:38:11.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:11.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:11.607946+0000) 2022-09-10T14:38:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:11.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:11.987275+0000) 2022-09-10T14:38:12.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:12.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:12.608194+0000) 2022-09-10T14:38:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:12 smithi110 ceph-mon[97633]: pgmap v19484: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:38:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:12 smithi110 ceph-mon[97633]: osdmap e1299: 12 total, 12 up, 12 in 2022-09-10T14:38:13.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:12 smithi104 ceph-mon[97044]: pgmap v19484: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:38:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:12 smithi104 ceph-mon[97044]: osdmap e1299: 12 total, 12 up, 12 in 2022-09-10T14:38:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:12.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:12.987501+0000) 2022-09-10T14:38:13.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:13.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:13.608350+0000) 2022-09-10T14:38:14.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:13.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:13.987733+0000) 2022-09-10T14:38:14.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:14.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:14.608558+0000) 2022-09-10T14:38:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:14 smithi110 ceph-mon[97633]: pgmap v19486: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:38:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:14 smithi104 ceph-mon[97044]: pgmap v19486: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:38:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:14.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:14.987929+0000) 2022-09-10T14:38:15.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:15.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:15.608753+0000) 2022-09-10T14:38:16.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:15.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:15.988170+0000) 2022-09-10T14:38:16.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:16.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:16.608915+0000) 2022-09-10T14:38:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:16 smithi110 ceph-mon[97633]: pgmap v19487: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:38:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:16 smithi104 ceph-mon[97044]: pgmap v19487: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:38:17.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:16.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:16.988410+0000) 2022-09-10T14:38:17.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:17.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:17.609181+0000) 2022-09-10T14:38:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:17.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:17.988577+0000) 2022-09-10T14:38:18.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:18.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:18.609395+0000) 2022-09-10T14:38:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:18 smithi110 ceph-mon[97633]: pgmap v19488: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:38:19.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:18 smithi104 ceph-mon[97044]: pgmap v19488: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:38:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:18.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:18.988787+0000) 2022-09-10T14:38:19.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:19.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:19.609654+0000) 2022-09-10T14:38:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:19.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:19.989008+0000) 2022-09-10T14:38:20.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:38:20.564+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:38:20.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:20.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:20.609791+0000) 2022-09-10T14:38:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:20 smithi110 ceph-mon[97633]: pgmap v19489: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:38:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:20 smithi104 ceph-mon[97044]: pgmap v19489: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:38:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:20.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:20.989261+0000) 2022-09-10T14:38:21.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:21.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:21.610024+0000) 2022-09-10T14:38:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:21.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:21.989511+0000) 2022-09-10T14:38:22.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:22.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:22.610274+0000) 2022-09-10T14:38:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:22 smithi110 ceph-mon[97633]: pgmap v19490: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:38:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:22 smithi104 ceph-mon[97044]: pgmap v19490: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:38:23.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:22.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:22.989758+0000) 2022-09-10T14:38:23.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:23.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:23.610486+0000) 2022-09-10T14:38:24.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:23.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:23.989961+0000) 2022-09-10T14:38:24.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:24.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:24.610666+0000) 2022-09-10T14:38:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:24 smithi110 ceph-mon[97633]: pgmap v19491: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:38:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:24 smithi104 ceph-mon[97044]: pgmap v19491: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:38:25.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:24.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:24.990211+0000) 2022-09-10T14:38:25.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:25.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:25.610901+0000) 2022-09-10T14:38:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:25.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:25.990459+0000) 2022-09-10T14:38:26.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:26.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:26.611173+0000) 2022-09-10T14:38:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:26 smithi110 ceph-mon[97633]: pgmap v19492: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:38:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:26 smithi104 ceph-mon[97044]: pgmap v19492: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:38:27.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:26.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:26.990707+0000) 2022-09-10T14:38:27.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:27.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:27.611341+0000) 2022-09-10T14:38:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:27.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:27.990905+0000) 2022-09-10T14:38:28.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:28.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:28.611532+0000) 2022-09-10T14:38:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:28 smithi110 ceph-mon[97633]: pgmap v19493: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:28 smithi104 ceph-mon[97044]: pgmap v19493: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:28.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:28.991171+0000) 2022-09-10T14:38:29.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:29.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:29.611718+0000) 2022-09-10T14:38:30.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:29.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:29.991418+0000) 2022-09-10T14:38:30.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:30.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:30.611954+0000) 2022-09-10T14:38:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:30 smithi110 ceph-mon[97633]: pgmap v19494: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:31.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:30 smithi104 ceph-mon[97044]: pgmap v19494: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:30.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:30.991605+0000) 2022-09-10T14:38:31.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:31.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:31.612214+0000) 2022-09-10T14:38:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:31.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:31.991777+0000) 2022-09-10T14:38:32.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:32.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:32.612459+0000) 2022-09-10T14:38:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:32 smithi110 ceph-mon[97633]: pgmap v19495: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:32 smithi104 ceph-mon[97044]: pgmap v19495: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:32.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:32.991994+0000) 2022-09-10T14:38:33.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:33.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:33.612665+0000) 2022-09-10T14:38:34.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:33.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:33.992229+0000) 2022-09-10T14:38:34.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:34.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:34.612900+0000) 2022-09-10T14:38:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:34 smithi110 ceph-mon[97633]: pgmap v19496: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:34 smithi104 ceph-mon[97044]: pgmap v19496: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:34.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:34.992465+0000) 2022-09-10T14:38:35.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:38:35.565+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:38:35.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:35.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:35.613157+0000) 2022-09-10T14:38:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:35.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:35.992694+0000) 2022-09-10T14:38:36.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:36.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:36.613355+0000) 2022-09-10T14:38:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:36 smithi110 ceph-mon[97633]: pgmap v19497: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:36 smithi104 ceph-mon[97044]: pgmap v19497: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:36.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:36.992927+0000) 2022-09-10T14:38:37.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:37.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:37.613527+0000) 2022-09-10T14:38:38.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:37.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:37.993163+0000) 2022-09-10T14:38:38.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:38.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:38.613736+0000) 2022-09-10T14:38:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:38 smithi110 ceph-mon[97633]: pgmap v19498: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:38 smithi104 ceph-mon[97044]: pgmap v19498: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:38.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:38.993345+0000) 2022-09-10T14:38:39.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:39.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:39.613965+0000) 2022-09-10T14:38:40.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:39.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:39.993490+0000) 2022-09-10T14:38:40.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:40.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:40.614239+0000) 2022-09-10T14:38:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:40 smithi110 ceph-mon[97633]: pgmap v19499: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:40 smithi104 ceph-mon[97044]: pgmap v19499: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:40.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:40.993674+0000) 2022-09-10T14:38:41.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:41.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:41.614495+0000) 2022-09-10T14:38:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:41.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:41.993906+0000) 2022-09-10T14:38:42.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:42.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:42.614748+0000) 2022-09-10T14:38:43.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:42 smithi110 ceph-mon[97633]: pgmap v19500: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:42 smithi104 ceph-mon[97044]: pgmap v19500: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:42.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:42.994148+0000) 2022-09-10T14:38:43.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:43.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:43.614993+0000) 2022-09-10T14:38:44.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:43.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:43.994378+0000) 2022-09-10T14:38:44.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:44.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:44.615163+0000) 2022-09-10T14:38:45.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:44 smithi110 ceph-mon[97633]: pgmap v19501: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:44 smithi104 ceph-mon[97044]: pgmap v19501: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:45.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:44.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:44.994520+0000) 2022-09-10T14:38:45.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:45.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:45.615408+0000) 2022-09-10T14:38:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:45.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:45.994695+0000) 2022-09-10T14:38:46.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:46.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:46.615655+0000) 2022-09-10T14:38:47.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:46 smithi110 ceph-mon[97633]: pgmap v19502: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:38:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:38:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:38:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:38:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:46 smithi104 ceph-mon[97044]: pgmap v19502: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:38:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:38:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:38:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:38:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:46.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:46.994847+0000) 2022-09-10T14:38:47.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:47.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:47.615905+0000) 2022-09-10T14:38:48.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:47.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:47.995025+0000) 2022-09-10T14:38:48.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:48.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:48.616146+0000) 2022-09-10T14:38:49.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:48 smithi110 ceph-mon[97633]: pgmap v19503: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:48 smithi104 ceph-mon[97044]: pgmap v19503: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:48.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:48.995275+0000) 2022-09-10T14:38:49.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:49.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:49.616334+0000) 2022-09-10T14:38:50.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:49.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:49.995506+0000) 2022-09-10T14:38:50.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:50.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:50.616515+0000) 2022-09-10T14:38:50.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:38:50.573+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:38:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:50 smithi110 ceph-mon[97633]: pgmap v19504: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:50 smithi104 ceph-mon[97044]: pgmap v19504: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:50.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:50.995747+0000) 2022-09-10T14:38:51.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:51.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:51.616758+0000) 2022-09-10T14:38:52.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:51.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:51.995927+0000) 2022-09-10T14:38:52.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:52.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:52.617024+0000) 2022-09-10T14:38:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:52 smithi110 ceph-mon[97633]: pgmap v19505: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:52 smithi104 ceph-mon[97044]: pgmap v19505: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:52.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:52.996126+0000) 2022-09-10T14:38:53.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:53.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:53.617283+0000) 2022-09-10T14:38:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:53.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:53.996368+0000) 2022-09-10T14:38:54.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:54.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:54.617441+0000) 2022-09-10T14:38:55.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:54 smithi110 ceph-mon[97633]: pgmap v19506: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:54 smithi104 ceph-mon[97044]: pgmap v19506: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:55.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:54.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:54.996596+0000) 2022-09-10T14:38:55.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:55.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:55.617634+0000) 2022-09-10T14:38:56.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:55.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:55.996841+0000) 2022-09-10T14:38:56.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:56.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:56.617888+0000) 2022-09-10T14:38:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:56 smithi110 ceph-mon[97633]: pgmap v19507: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:56 smithi104 ceph-mon[97044]: pgmap v19507: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:38:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:56.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:56.997038+0000) 2022-09-10T14:38:57.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:57.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:57.618130+0000) 2022-09-10T14:38:58.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:57.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:57.997279+0000) 2022-09-10T14:38:59.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:58.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:58.618265+0000) 2022-09-10T14:38:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:38:58 smithi110 ceph-mon[97633]: pgmap v19508: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:59.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:38:58 smithi104 ceph-mon[97044]: pgmap v19508: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:38:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:58.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:58.997555+0000) 2022-09-10T14:38:59.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:38:59.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:59.618488+0000) 2022-09-10T14:39:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:38:59.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:38:59.997798+0000) 2022-09-10T14:39:00.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:00.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:00.618664+0000) 2022-09-10T14:39:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:00 smithi110 ceph-mon[97633]: pgmap v19509: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:00 smithi104 ceph-mon[97044]: pgmap v19509: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:00.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:00.998035+0000) 2022-09-10T14:39:01.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:01.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:01.618914+0000) 2022-09-10T14:39:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:01.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:01.998257+0000) 2022-09-10T14:39:02.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:02.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:02.619172+0000) 2022-09-10T14:39:03.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:02 smithi110 ceph-mon[97633]: pgmap v19510: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:02 smithi104 ceph-mon[97044]: pgmap v19510: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:02.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:02.998476+0000) 2022-09-10T14:39:04.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:03.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:03.619332+0000) 2022-09-10T14:39:04.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:03.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:03.998713+0000) 2022-09-10T14:39:05.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:04.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:04.619548+0000) 2022-09-10T14:39:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:04 smithi110 ceph-mon[97633]: pgmap v19511: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:04 smithi104 ceph-mon[97044]: pgmap v19511: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:04.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:04.998907+0000) 2022-09-10T14:39:06.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:39:05.580+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:39:06.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:05.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:05.619805+0000) 2022-09-10T14:39:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:05.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:05.999143+0000) 2022-09-10T14:39:07.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:06.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:06.620070+0000) 2022-09-10T14:39:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:06 smithi110 ceph-mon[97633]: pgmap v19512: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:39:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:06 smithi104 ceph-mon[97044]: pgmap v19512: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:39:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:06.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:06.999320+0000) 2022-09-10T14:39:08.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:07.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:07.620321+0000) 2022-09-10T14:39:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:07.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:07.999484+0000) 2022-09-10T14:39:09.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:08.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:08.620573+0000) 2022-09-10T14:39:09.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:08 smithi110 ceph-mon[97633]: pgmap v19513: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:39:09.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:08 smithi110 ceph-mon[97633]: osdmap e1300: 12 total, 12 up, 12 in 2022-09-10T14:39:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:08 smithi104 ceph-mon[97044]: pgmap v19513: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:39:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:08 smithi104 ceph-mon[97044]: osdmap e1300: 12 total, 12 up, 12 in 2022-09-10T14:39:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:08.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:08.999687+0000) 2022-09-10T14:39:10.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:09.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:09.620798+0000) 2022-09-10T14:39:10.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:09.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:09.999914+0000) 2022-09-10T14:39:11.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:10.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:10.621035+0000) 2022-09-10T14:39:11.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:10 smithi110 ceph-mon[97633]: pgmap v19515: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:39:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:10 smithi104 ceph-mon[97044]: pgmap v19515: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:39:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:11.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:11.000132+0000) 2022-09-10T14:39:12.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:11.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:11.621168+0000) 2022-09-10T14:39:12.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:11.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:12.000269+0000) 2022-09-10T14:39:13.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:12.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:12.621371+0000) 2022-09-10T14:39:13.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:12 smithi110 ceph-mon[97633]: pgmap v19516: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:39:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:12 smithi110 ceph-mon[97633]: osdmap e1301: 12 total, 12 up, 12 in 2022-09-10T14:39:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:12 smithi104 ceph-mon[97044]: pgmap v19516: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:39:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:12 smithi104 ceph-mon[97044]: osdmap e1301: 12 total, 12 up, 12 in 2022-09-10T14:39:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:12.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:13.000491+0000) 2022-09-10T14:39:14.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:13.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:13.621558+0000) 2022-09-10T14:39:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:13.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:14.000657+0000) 2022-09-10T14:39:15.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:14.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:14.621778+0000) 2022-09-10T14:39:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:14 smithi110 ceph-mon[97633]: pgmap v19518: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:39:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:14 smithi104 ceph-mon[97044]: pgmap v19518: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:39:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:14.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:15.000800+0000) 2022-09-10T14:39:16.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:15.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:15.621987+0000) 2022-09-10T14:39:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:16.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:16.001062+0000) 2022-09-10T14:39:17.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:16.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:16.622177+0000) 2022-09-10T14:39:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:16 smithi110 ceph-mon[97633]: pgmap v19519: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:39:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:16 smithi104 ceph-mon[97044]: pgmap v19519: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:39:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:17.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:17.001329+0000) 2022-09-10T14:39:18.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:17.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:17.622374+0000) 2022-09-10T14:39:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:18.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:18.001508+0000) 2022-09-10T14:39:19.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:18.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:18.622534+0000) 2022-09-10T14:39:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:18 smithi110 ceph-mon[97633]: pgmap v19520: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:39:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:18 smithi104 ceph-mon[97044]: pgmap v19520: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:39:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:19.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:19.001662+0000) 2022-09-10T14:39:20.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:19.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:19.622711+0000) 2022-09-10T14:39:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:20.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:20.001896+0000) 2022-09-10T14:39:21.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:20.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:20.622908+0000) 2022-09-10T14:39:21.004 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:39:20.594+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:39:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:20 smithi110 ceph-mon[97633]: pgmap v19521: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:39:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:20 smithi104 ceph-mon[97044]: pgmap v19521: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:39:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:21.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:21.002082+0000) 2022-09-10T14:39:22.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:21.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:21.623176+0000) 2022-09-10T14:39:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:22.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:22.002393+0000) 2022-09-10T14:39:23.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:22.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:22.623302+0000) 2022-09-10T14:39:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:22 smithi110 ceph-mon[97633]: pgmap v19522: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:39:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:22 smithi104 ceph-mon[97044]: pgmap v19522: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:39:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:23.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:23.002639+0000) 2022-09-10T14:39:24.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:23.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:23.623535+0000) 2022-09-10T14:39:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:24.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:24.002906+0000) 2022-09-10T14:39:25.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:24.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:24.623816+0000) 2022-09-10T14:39:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:24 smithi110 ceph-mon[97633]: pgmap v19523: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:39:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:24 smithi104 ceph-mon[97044]: pgmap v19523: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:39:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:25.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:25.003081+0000) 2022-09-10T14:39:26.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:25.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:25.624050+0000) 2022-09-10T14:39:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:26.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:26.003285+0000) 2022-09-10T14:39:27.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:26.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:26.624197+0000) 2022-09-10T14:39:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:26 smithi110 ceph-mon[97633]: pgmap v19524: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:39:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:26 smithi104 ceph-mon[97044]: pgmap v19524: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:39:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:27.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:27.003519+0000) 2022-09-10T14:39:28.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:27.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:27.624408+0000) 2022-09-10T14:39:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:28.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:28.003727+0000) 2022-09-10T14:39:29.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:28.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:28.624661+0000) 2022-09-10T14:39:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:28 smithi110 ceph-mon[97633]: pgmap v19525: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:28 smithi104 ceph-mon[97044]: pgmap v19525: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:29.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:29.003964+0000) 2022-09-10T14:39:30.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:29.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:29.624892+0000) 2022-09-10T14:39:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:30.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:30.004161+0000) 2022-09-10T14:39:31.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:30.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:30.625140+0000) 2022-09-10T14:39:31.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:30 smithi110 ceph-mon[97633]: pgmap v19526: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:30 smithi104 ceph-mon[97044]: pgmap v19526: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:31.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:31.004426+0000) 2022-09-10T14:39:32.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:31.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:31.625339+0000) 2022-09-10T14:39:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:32.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:32.004627+0000) 2022-09-10T14:39:33.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:32.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:32.625546+0000) 2022-09-10T14:39:33.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:32 smithi110 ceph-mon[97633]: pgmap v19527: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:32 smithi104 ceph-mon[97044]: pgmap v19527: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:33.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:33.004864+0000) 2022-09-10T14:39:34.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:33.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:33.625765+0000) 2022-09-10T14:39:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:34.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:34.005078+0000) 2022-09-10T14:39:35.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:34.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:34.626011+0000) 2022-09-10T14:39:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:34 smithi110 ceph-mon[97633]: pgmap v19528: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:34 smithi104 ceph-mon[97044]: pgmap v19528: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:35.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:35.005252+0000) 2022-09-10T14:39:36.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:35.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:35.626228+0000) 2022-09-10T14:39:36.008 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:39:35.606+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:39:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:36.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:36.005461+0000) 2022-09-10T14:39:37.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:36.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:36.626472+0000) 2022-09-10T14:39:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:36 smithi110 ceph-mon[97633]: pgmap v19529: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:37.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:36 smithi104 ceph-mon[97044]: pgmap v19529: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:37.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:37.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:37.005639+0000) 2022-09-10T14:39:38.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:37.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:37.626667+0000) 2022-09-10T14:39:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:38.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:38.005858+0000) 2022-09-10T14:39:39.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:38.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:38.626831+0000) 2022-09-10T14:39:39.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:38 smithi110 ceph-mon[97633]: pgmap v19530: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:38 smithi104 ceph-mon[97044]: pgmap v19530: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:39.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:39.006054+0000) 2022-09-10T14:39:40.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:39.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:39.627071+0000) 2022-09-10T14:39:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:40.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:40.006231+0000) 2022-09-10T14:39:41.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:40.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:40.627301+0000) 2022-09-10T14:39:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:40 smithi110 ceph-mon[97633]: pgmap v19531: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:40 smithi104 ceph-mon[97044]: pgmap v19531: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:41.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:41.006443+0000) 2022-09-10T14:39:42.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:41.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:41.627523+0000) 2022-09-10T14:39:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:42.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:42.006666+0000) 2022-09-10T14:39:43.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:42.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:42.627784+0000) 2022-09-10T14:39:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:42 smithi110 ceph-mon[97633]: pgmap v19532: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:42 smithi104 ceph-mon[97044]: pgmap v19532: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:43.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:43.006893+0000) 2022-09-10T14:39:44.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:43.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:43.627982+0000) 2022-09-10T14:39:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:44.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:44.007137+0000) 2022-09-10T14:39:45.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:44.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:44.628253+0000) 2022-09-10T14:39:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:44 smithi110 ceph-mon[97633]: pgmap v19533: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:44 smithi104 ceph-mon[97044]: pgmap v19533: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:45.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:45.007346+0000) 2022-09-10T14:39:46.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:45.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:45.628585+0000) 2022-09-10T14:39:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:46.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:46.007519+0000) 2022-09-10T14:39:47.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:46.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:46.628831+0000) 2022-09-10T14:39:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:46 smithi110 ceph-mon[97633]: pgmap v19534: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:39:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:39:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:39:47.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:39:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:46 smithi104 ceph-mon[97044]: pgmap v19534: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:39:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:39:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:39:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:39:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:47.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:47.007690+0000) 2022-09-10T14:39:48.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:47.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:47.629053+0000) 2022-09-10T14:39:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:48.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:48.007846+0000) 2022-09-10T14:39:49.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:48.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:48.629279+0000) 2022-09-10T14:39:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:48 smithi110 ceph-mon[97633]: pgmap v19535: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:48 smithi104 ceph-mon[97044]: pgmap v19535: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:49.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:49.008063+0000) 2022-09-10T14:39:50.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:49.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:49.629501+0000) 2022-09-10T14:39:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:50.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:50.008334+0000) 2022-09-10T14:39:51.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:50.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:50.629664+0000) 2022-09-10T14:39:51.011 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:39:50.622+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:39:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:50 smithi110 ceph-mon[97633]: pgmap v19536: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:50 smithi104 ceph-mon[97044]: pgmap v19536: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:51.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:51.008507+0000) 2022-09-10T14:39:52.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:51.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:51.629876+0000) 2022-09-10T14:39:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:52.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:52.008700+0000) 2022-09-10T14:39:53.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:52.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:52.630074+0000) 2022-09-10T14:39:53.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:52 smithi110 ceph-mon[97633]: pgmap v19537: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:52 smithi104 ceph-mon[97044]: pgmap v19537: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:53.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:53.008987+0000) 2022-09-10T14:39:54.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:53.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:53.630449+0000) 2022-09-10T14:39:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:54.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:54.009188+0000) 2022-09-10T14:39:55.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:54.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:54.630634+0000) 2022-09-10T14:39:55.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:54 smithi110 ceph-mon[97633]: pgmap v19538: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:54 smithi104 ceph-mon[97044]: pgmap v19538: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:55.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:55.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:55.009430+0000) 2022-09-10T14:39:56.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:55.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:55.630875+0000) 2022-09-10T14:39:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:56.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:56.009697+0000) 2022-09-10T14:39:57.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:56.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:56.631156+0000) 2022-09-10T14:39:57.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:56 smithi110 ceph-mon[97633]: pgmap v19539: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:56 smithi104 ceph-mon[97044]: pgmap v19539: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:39:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:57.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:57.009943+0000) 2022-09-10T14:39:58.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:57.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:57.631391+0000) 2022-09-10T14:39:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:58.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:58.010210+0000) 2022-09-10T14:39:59.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:58.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:58.631584+0000) 2022-09-10T14:39:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:39:58 smithi110 ceph-mon[97633]: pgmap v19540: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:39:58 smithi104 ceph-mon[97044]: pgmap v19540: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:39:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:39:59.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:59.010485+0000) 2022-09-10T14:40:00.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:39:59.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:39:59.631806+0000) 2022-09-10T14:40:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:00.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:00.010742+0000) 2022-09-10T14:40:01.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:00.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:00.631972+0000) 2022-09-10T14:40:01.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:00 smithi110 ceph-mon[97633]: pgmap v19541: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:40:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:00 smithi104 ceph-mon[97044]: pgmap v19541: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:40:01.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:01.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:01.010981+0000) 2022-09-10T14:40:02.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:01.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:01.632232+0000) 2022-09-10T14:40:02.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:02.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:02.011157+0000) 2022-09-10T14:40:03.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:02.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:02.632371+0000) 2022-09-10T14:40:03.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:02 smithi110 ceph-mon[97633]: pgmap v19542: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:02 smithi104 ceph-mon[97044]: pgmap v19542: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:03.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:03.011366+0000) 2022-09-10T14:40:04.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:03.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:03.632539+0000) 2022-09-10T14:40:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:04.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:04.011600+0000) 2022-09-10T14:40:05.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:04.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:04.632691+0000) 2022-09-10T14:40:05.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:04 smithi110 ceph-mon[97633]: pgmap v19543: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:04 smithi104 ceph-mon[97044]: pgmap v19543: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:05.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:05.011808+0000) 2022-09-10T14:40:06.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:05.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:05.632811+0000) 2022-09-10T14:40:06.015 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:40:05.636+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:40:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:06.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:06.012018+0000) 2022-09-10T14:40:07.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:06.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:06.632954+0000) 2022-09-10T14:40:07.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:06 smithi110 ceph-mon[97633]: pgmap v19544: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:40:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:06 smithi104 ceph-mon[97044]: pgmap v19544: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:40:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:07.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:07.012257+0000) 2022-09-10T14:40:08.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:07.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:07.633225+0000) 2022-09-10T14:40:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:08.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:08.012408+0000) 2022-09-10T14:40:09.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:08.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:08.633447+0000) 2022-09-10T14:40:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:08 smithi104 ceph-mon[97044]: pgmap v19545: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:40:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:08 smithi104 ceph-mon[97044]: osdmap e1302: 12 total, 12 up, 12 in 2022-09-10T14:40:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:09.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:09.012665+0000) 2022-09-10T14:40:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:08 smithi110 ceph-mon[97633]: pgmap v19545: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:40:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:08 smithi110 ceph-mon[97633]: osdmap e1302: 12 total, 12 up, 12 in 2022-09-10T14:40:10.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:09.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:09.633569+0000) 2022-09-10T14:40:10.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:10.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:10.012874+0000) 2022-09-10T14:40:11.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:10.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:10.633812+0000) 2022-09-10T14:40:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:10 smithi104 ceph-mon[97044]: pgmap v19547: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:40:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:11.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:11.013134+0000) 2022-09-10T14:40:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:10 smithi110 ceph-mon[97633]: pgmap v19547: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:40:12.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:11.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:11.634022+0000) 2022-09-10T14:40:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:12.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:12.013320+0000) 2022-09-10T14:40:13.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:12.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:12.634254+0000) 2022-09-10T14:40:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:12 smithi110 ceph-mon[97633]: pgmap v19548: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:40:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:12 smithi110 ceph-mon[97633]: osdmap e1303: 12 total, 12 up, 12 in 2022-09-10T14:40:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:12 smithi104 ceph-mon[97044]: pgmap v19548: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:40:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:12 smithi104 ceph-mon[97044]: osdmap e1303: 12 total, 12 up, 12 in 2022-09-10T14:40:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:13.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:13.013463+0000) 2022-09-10T14:40:14.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:13.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:13.634486+0000) 2022-09-10T14:40:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:14.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:14.013650+0000) 2022-09-10T14:40:15.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:14.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:14.634718+0000) 2022-09-10T14:40:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:14 smithi110 ceph-mon[97633]: pgmap v19550: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:40:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:14 smithi104 ceph-mon[97044]: pgmap v19550: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:40:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:15.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:15.013898+0000) 2022-09-10T14:40:16.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:15.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:15.634979+0000) 2022-09-10T14:40:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:16.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:16.014195+0000) 2022-09-10T14:40:17.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:16.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:16.635164+0000) 2022-09-10T14:40:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:16 smithi110 ceph-mon[97633]: pgmap v19551: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:40:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:16 smithi104 ceph-mon[97044]: pgmap v19551: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:40:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:17.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:17.014362+0000) 2022-09-10T14:40:18.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:17.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:17.635282+0000) 2022-09-10T14:40:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:18.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:18.014523+0000) 2022-09-10T14:40:19.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:18.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:18.635466+0000) 2022-09-10T14:40:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:18 smithi110 ceph-mon[97633]: pgmap v19552: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:40:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:18 smithi104 ceph-mon[97044]: pgmap v19552: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:40:19.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:19.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:19.014733+0000) 2022-09-10T14:40:20.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:19.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:19.635717+0000) 2022-09-10T14:40:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:20.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:20.014984+0000) 2022-09-10T14:40:21.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:20.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:20.635975+0000) 2022-09-10T14:40:21.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:40:20.651+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:40:21.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:20 smithi110 ceph-mon[97633]: pgmap v19553: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:40:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:20 smithi104 ceph-mon[97044]: pgmap v19553: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:40:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:21.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:21.015245+0000) 2022-09-10T14:40:22.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:21.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:21.636206+0000) 2022-09-10T14:40:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:22.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:22.015483+0000) 2022-09-10T14:40:23.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:22.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:22.636381+0000) 2022-09-10T14:40:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:22 smithi110 ceph-mon[97633]: pgmap v19554: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:40:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:22 smithi104 ceph-mon[97044]: pgmap v19554: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T14:40:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:23.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:23.015634+0000) 2022-09-10T14:40:24.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:23.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:23.636664+0000) 2022-09-10T14:40:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:24.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:24.015843+0000) 2022-09-10T14:40:25.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:24.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:24.636849+0000) 2022-09-10T14:40:25.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:24 smithi110 ceph-mon[97633]: pgmap v19555: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:40:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:24 smithi104 ceph-mon[97044]: pgmap v19555: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:40:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:25.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:25.016074+0000) 2022-09-10T14:40:26.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:25.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:25.637044+0000) 2022-09-10T14:40:26.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:26.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:26.016305+0000) 2022-09-10T14:40:27.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:26.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:26.637282+0000) 2022-09-10T14:40:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:26 smithi110 ceph-mon[97633]: pgmap v19556: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:40:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:26 smithi104 ceph-mon[97044]: pgmap v19556: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:40:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:27.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:27.016517+0000) 2022-09-10T14:40:28.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:27.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:27.637547+0000) 2022-09-10T14:40:28.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:28.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:28.016715+0000) 2022-09-10T14:40:29.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:28.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:28.637813+0000) 2022-09-10T14:40:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:28 smithi110 ceph-mon[97633]: pgmap v19557: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:28 smithi104 ceph-mon[97044]: pgmap v19557: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:29.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:29.016936+0000) 2022-09-10T14:40:30.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:29.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:29.638131+0000) 2022-09-10T14:40:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:30.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:30.017204+0000) 2022-09-10T14:40:31.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:30.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:30.638353+0000) 2022-09-10T14:40:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:30 smithi110 ceph-mon[97633]: pgmap v19558: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:30 smithi104 ceph-mon[97044]: pgmap v19558: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:31.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:31.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:31.017435+0000) 2022-09-10T14:40:32.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:31.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:31.638479+0000) 2022-09-10T14:40:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:32.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:32.017671+0000) 2022-09-10T14:40:33.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:32.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:32.638671+0000) 2022-09-10T14:40:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:32 smithi110 ceph-mon[97633]: pgmap v19559: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:32 smithi104 ceph-mon[97044]: pgmap v19559: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:33.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:33.017907+0000) 2022-09-10T14:40:34.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:33.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:33.638928+0000) 2022-09-10T14:40:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:34.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:34.018142+0000) 2022-09-10T14:40:35.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:34.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:34.639176+0000) 2022-09-10T14:40:35.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:34 smithi110 ceph-mon[97633]: pgmap v19560: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:34 smithi104 ceph-mon[97044]: pgmap v19560: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:35.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:35.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:35.018330+0000) 2022-09-10T14:40:36.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:35.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:35.639386+0000) 2022-09-10T14:40:36.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:40:35.665+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:40:36.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:36.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:36.018453+0000) 2022-09-10T14:40:37.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:36.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:36.639678+0000) 2022-09-10T14:40:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:36 smithi110 ceph-mon[97633]: pgmap v19561: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:37.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:36 smithi104 ceph-mon[97044]: pgmap v19561: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:37.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:37.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:37.018682+0000) 2022-09-10T14:40:38.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:37.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:37.639928+0000) 2022-09-10T14:40:38.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:38.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:38.018911+0000) 2022-09-10T14:40:39.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:38.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:38.640143+0000) 2022-09-10T14:40:39.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:38 smithi110 ceph-mon[97633]: pgmap v19562: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:38 smithi104 ceph-mon[97044]: pgmap v19562: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:39.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:39.019192+0000) 2022-09-10T14:40:40.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:39.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:39.640304+0000) 2022-09-10T14:40:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:40.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:40.019370+0000) 2022-09-10T14:40:41.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:40.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:40.640464+0000) 2022-09-10T14:40:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:40 smithi110 ceph-mon[97633]: pgmap v19563: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:40 smithi104 ceph-mon[97044]: pgmap v19563: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:41.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:41.019532+0000) 2022-09-10T14:40:42.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:41.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:41.640622+0000) 2022-09-10T14:40:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:42.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:42.019741+0000) 2022-09-10T14:40:43.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:42.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:42.640863+0000) 2022-09-10T14:40:43.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:42 smithi110 ceph-mon[97633]: pgmap v19564: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:42 smithi104 ceph-mon[97044]: pgmap v19564: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:43.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:43.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:43.019986+0000) 2022-09-10T14:40:44.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:43.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:43.641156+0000) 2022-09-10T14:40:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:44.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:44.020208+0000) 2022-09-10T14:40:45.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:44.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:44.641425+0000) 2022-09-10T14:40:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:44 smithi110 ceph-mon[97633]: pgmap v19565: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:44 smithi104 ceph-mon[97044]: pgmap v19565: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:45.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:45.020451+0000) 2022-09-10T14:40:46.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:45.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:45.641626+0000) 2022-09-10T14:40:46.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:46.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:46.020662+0000) 2022-09-10T14:40:47.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:46.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:46.641901+0000) 2022-09-10T14:40:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:46 smithi110 ceph-mon[97633]: pgmap v19566: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:40:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:40:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:40:47.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:40:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:46 smithi104 ceph-mon[97044]: pgmap v19566: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:40:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:40:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:40:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:40:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:47.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:47.020905+0000) 2022-09-10T14:40:48.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:47.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:47.642152+0000) 2022-09-10T14:40:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:48.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:48.021161+0000) 2022-09-10T14:40:49.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:48.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:48.642408+0000) 2022-09-10T14:40:49.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:48 smithi110 ceph-mon[97633]: pgmap v19567: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:48 smithi104 ceph-mon[97044]: pgmap v19567: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:49.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:49.021380+0000) 2022-09-10T14:40:50.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:49.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:49.642663+0000) 2022-09-10T14:40:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:50.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:50.021607+0000) 2022-09-10T14:40:51.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:50.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:50.642912+0000) 2022-09-10T14:40:51.024 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:40:50.673+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:40:51.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:50 smithi110 ceph-mon[97633]: pgmap v19568: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:50 smithi104 ceph-mon[97044]: pgmap v19568: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:51.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:51.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:51.021854+0000) 2022-09-10T14:40:52.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:51.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:51.643170+0000) 2022-09-10T14:40:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:52.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:52.022037+0000) 2022-09-10T14:40:53.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:52.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:52.643447+0000) 2022-09-10T14:40:53.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:52 smithi110 ceph-mon[97633]: pgmap v19569: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:52 smithi104 ceph-mon[97044]: pgmap v19569: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:53.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:53.022201+0000) 2022-09-10T14:40:54.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:53.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:53.643691+0000) 2022-09-10T14:40:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:54.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:54.022383+0000) 2022-09-10T14:40:55.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:54.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:54.643935+0000) 2022-09-10T14:40:55.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:54 smithi110 ceph-mon[97633]: pgmap v19570: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:54 smithi104 ceph-mon[97044]: pgmap v19570: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:55.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:55.022530+0000) 2022-09-10T14:40:56.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:55.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:55.644214+0000) 2022-09-10T14:40:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:56.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:56.022725+0000) 2022-09-10T14:40:57.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:56.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:56.644491+0000) 2022-09-10T14:40:57.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:56 smithi110 ceph-mon[97633]: pgmap v19571: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:56 smithi104 ceph-mon[97044]: pgmap v19571: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:40:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:57.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:57.022982+0000) 2022-09-10T14:40:58.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:57.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:57.644760+0000) 2022-09-10T14:40:58.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:58.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:58.023251+0000) 2022-09-10T14:40:59.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:58.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:58.644924+0000) 2022-09-10T14:40:59.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:40:58 smithi110 ceph-mon[97633]: pgmap v19572: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:59.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:40:58 smithi104 ceph-mon[97044]: pgmap v19572: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:40:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:40:59.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:59.023453+0000) 2022-09-10T14:41:00.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:40:59.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:40:59.645198+0000) 2022-09-10T14:41:00.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:00.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:00.023642+0000) 2022-09-10T14:41:01.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:00.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:00.645355+0000) 2022-09-10T14:41:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:00 smithi104 ceph-mon[97044]: pgmap v19573: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:01.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:01.023922+0000) 2022-09-10T14:41:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:00 smithi110 ceph-mon[97633]: pgmap v19573: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:02.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:01.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:01.645496+0000) 2022-09-10T14:41:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:02.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:02.024184+0000) 2022-09-10T14:41:03.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:02.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:02.645756+0000) 2022-09-10T14:41:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:02 smithi104 ceph-mon[97044]: pgmap v19574: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:03.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:03.024364+0000) 2022-09-10T14:41:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:02 smithi110 ceph-mon[97633]: pgmap v19574: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:04.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:03.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:03.645979+0000) 2022-09-10T14:41:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:04.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:04.024588+0000) 2022-09-10T14:41:05.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:04.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:04.646255+0000) 2022-09-10T14:41:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:04 smithi104 ceph-mon[97044]: pgmap v19575: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:05.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:05.024836+0000) 2022-09-10T14:41:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:04 smithi110 ceph-mon[97633]: pgmap v19575: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:06.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:05.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:05.646454+0000) 2022-09-10T14:41:06.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:41:05.684+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:41:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:06.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:06.025054+0000) 2022-09-10T14:41:07.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:06.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:06.646691+0000) 2022-09-10T14:41:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:06 smithi104 ceph-mon[97044]: pgmap v19576: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:41:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:07.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:07.025285+0000) 2022-09-10T14:41:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:06 smithi110 ceph-mon[97633]: pgmap v19576: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:41:08.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:07.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:07.646903+0000) 2022-09-10T14:41:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:08.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:08.025546+0000) 2022-09-10T14:41:09.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:08.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:08.647173+0000) 2022-09-10T14:41:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:08 smithi104 ceph-mon[97044]: pgmap v19577: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:41:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:08 smithi104 ceph-mon[97044]: osdmap e1304: 12 total, 12 up, 12 in 2022-09-10T14:41:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:09.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:09.025806+0000) 2022-09-10T14:41:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:08 smithi110 ceph-mon[97633]: pgmap v19577: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:41:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:08 smithi110 ceph-mon[97633]: osdmap e1304: 12 total, 12 up, 12 in 2022-09-10T14:41:10.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:09.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:09.647424+0000) 2022-09-10T14:41:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:10.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:10.026030+0000) 2022-09-10T14:41:11.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:10.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:10.647654+0000) 2022-09-10T14:41:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:10 smithi104 ceph-mon[97044]: pgmap v19579: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:41:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:11.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:11.026262+0000) 2022-09-10T14:41:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:10 smithi110 ceph-mon[97633]: pgmap v19579: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:41:12.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:11.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:11.647838+0000) 2022-09-10T14:41:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:12.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:12.026507+0000) 2022-09-10T14:41:13.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:12.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:12.648073+0000) 2022-09-10T14:41:13.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:12 smithi110 ceph-mon[97633]: pgmap v19580: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:41:13.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:12 smithi110 ceph-mon[97633]: osdmap e1305: 12 total, 12 up, 12 in 2022-09-10T14:41:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:12 smithi104 ceph-mon[97044]: pgmap v19580: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:41:13.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:12 smithi104 ceph-mon[97044]: osdmap e1305: 12 total, 12 up, 12 in 2022-09-10T14:41:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:13.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:13.026747+0000) 2022-09-10T14:41:14.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:13.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:13.648233+0000) 2022-09-10T14:41:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:14.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:14.026985+0000) 2022-09-10T14:41:15.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:14.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:14.648466+0000) 2022-09-10T14:41:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:14 smithi110 ceph-mon[97633]: pgmap v19582: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:41:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:14 smithi104 ceph-mon[97044]: pgmap v19582: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:41:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:15.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:15.027228+0000) 2022-09-10T14:41:16.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:15.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:15.648674+0000) 2022-09-10T14:41:16.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:16.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:16.027480+0000) 2022-09-10T14:41:17.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:16.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:16.648956+0000) 2022-09-10T14:41:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:16 smithi110 ceph-mon[97633]: pgmap v19583: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:41:17.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:16 smithi104 ceph-mon[97044]: pgmap v19583: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:41:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:17.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:17.027724+0000) 2022-09-10T14:41:18.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:17.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:17.649189+0000) 2022-09-10T14:41:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:18.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:18.027928+0000) 2022-09-10T14:41:19.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:18.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:18.649421+0000) 2022-09-10T14:41:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:18 smithi110 ceph-mon[97633]: pgmap v19584: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:41:19.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:18 smithi104 ceph-mon[97044]: pgmap v19584: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:41:19.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:19.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:19.028181+0000) 2022-09-10T14:41:20.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:19.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:19.649641+0000) 2022-09-10T14:41:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:20.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:20.028413+0000) 2022-09-10T14:41:21.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:20.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:20.649855+0000) 2022-09-10T14:41:21.032 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:41:20.698+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:41:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:20 smithi110 ceph-mon[97633]: pgmap v19585: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:41:21.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:20 smithi104 ceph-mon[97044]: pgmap v19585: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:41:21.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:21.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:21.028672+0000) 2022-09-10T14:41:22.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:21.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:21.650042+0000) 2022-09-10T14:41:22.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:22.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:22.028904+0000) 2022-09-10T14:41:23.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:22.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:22.650217+0000) 2022-09-10T14:41:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:22 smithi110 ceph-mon[97633]: pgmap v19586: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:41:23.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:22 smithi104 ceph-mon[97044]: pgmap v19586: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:41:23.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:23.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:23.029169+0000) 2022-09-10T14:41:24.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:23.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:23.650416+0000) 2022-09-10T14:41:24.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:24.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:24.029395+0000) 2022-09-10T14:41:25.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:24.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:24.650672+0000) 2022-09-10T14:41:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:24 smithi110 ceph-mon[97633]: pgmap v19587: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:41:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:24 smithi104 ceph-mon[97044]: pgmap v19587: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:41:25.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:25.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:25.029621+0000) 2022-09-10T14:41:26.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:25.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:25.650872+0000) 2022-09-10T14:41:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:26.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:26.029851+0000) 2022-09-10T14:41:27.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:26.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:26.651139+0000) 2022-09-10T14:41:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:26 smithi110 ceph-mon[97633]: pgmap v19588: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:41:27.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:26 smithi104 ceph-mon[97044]: pgmap v19588: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:41:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:27.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:27.030067+0000) 2022-09-10T14:41:28.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:27.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:27.651365+0000) 2022-09-10T14:41:28.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:28.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:28.030340+0000) 2022-09-10T14:41:29.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:28.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:28.651563+0000) 2022-09-10T14:41:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:28 smithi110 ceph-mon[97633]: pgmap v19589: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:28 smithi104 ceph-mon[97044]: pgmap v19589: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:29.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:29.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:29.030547+0000) 2022-09-10T14:41:30.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:29.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:29.651789+0000) 2022-09-10T14:41:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:30.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:30.030741+0000) 2022-09-10T14:41:31.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:30.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:30.652056+0000) 2022-09-10T14:41:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:30 smithi110 ceph-mon[97633]: pgmap v19590: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:31.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:31.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:31.030988+0000) 2022-09-10T14:41:31.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:30 smithi104 ceph-mon[97044]: pgmap v19590: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:32.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:31.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:31.652357+0000) 2022-09-10T14:41:32.314 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:32.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:32.031249+0000) 2022-09-10T14:41:33.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:32.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:32.652594+0000) 2022-09-10T14:41:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:32 smithi110 ceph-mon[97633]: pgmap v19591: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:33.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:32 smithi104 ceph-mon[97044]: pgmap v19591: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:33.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:33.031478+0000) 2022-09-10T14:41:34.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:33.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:33.652772+0000) 2022-09-10T14:41:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:34.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:34.031702+0000) 2022-09-10T14:41:35.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:34.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:34.653046+0000) 2022-09-10T14:41:35.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:34 smithi110 ceph-mon[97633]: pgmap v19592: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:34 smithi104 ceph-mon[97044]: pgmap v19592: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:35.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:35.031890+0000) 2022-09-10T14:41:36.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:35.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:35.653237+0000) 2022-09-10T14:41:36.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:41:35.701+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:41:36.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:36.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:36.032091+0000) 2022-09-10T14:41:37.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:36.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:36.653471+0000) 2022-09-10T14:41:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:36 smithi110 ceph-mon[97633]: pgmap v19593: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:37.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:36 smithi104 ceph-mon[97044]: pgmap v19593: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:37.318 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:37.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:37.032367+0000) 2022-09-10T14:41:38.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:37.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:37.653654+0000) 2022-09-10T14:41:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:38.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:38.032595+0000) 2022-09-10T14:41:39.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:38.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:38.653882+0000) 2022-09-10T14:41:39.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:38 smithi110 ceph-mon[97633]: pgmap v19594: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:38 smithi104 ceph-mon[97044]: pgmap v19594: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:39.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:39.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:39.032832+0000) 2022-09-10T14:41:40.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:39.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:39.654091+0000) 2022-09-10T14:41:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:40.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:40.033004+0000) 2022-09-10T14:41:41.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:40.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:40.654282+0000) 2022-09-10T14:41:41.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:40 smithi110 ceph-mon[97633]: pgmap v19595: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:40 smithi104 ceph-mon[97044]: pgmap v19595: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:41.320 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:41.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:41.033174+0000) 2022-09-10T14:41:42.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:41.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:41.654515+0000) 2022-09-10T14:41:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:42.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:42.033320+0000) 2022-09-10T14:41:43.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:42.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:42.654792+0000) 2022-09-10T14:41:43.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:42 smithi110 ceph-mon[97633]: pgmap v19596: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:43.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:42 smithi104 ceph-mon[97044]: pgmap v19596: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:43.366 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:43.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:43.033551+0000) 2022-09-10T14:41:44.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:43.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:43.655018+0000) 2022-09-10T14:41:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:44.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:44.033790+0000) 2022-09-10T14:41:45.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:44.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:44.655218+0000) 2022-09-10T14:41:45.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:44 smithi110 ceph-mon[97633]: pgmap v19597: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:44 smithi104 ceph-mon[97044]: pgmap v19597: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:45.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:45.034031+0000) 2022-09-10T14:41:46.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:45.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:45.655419+0000) 2022-09-10T14:41:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:46.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:46.034236+0000) 2022-09-10T14:41:47.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:46.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:46.655682+0000) 2022-09-10T14:41:47.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:46 smithi110 ceph-mon[97633]: pgmap v19598: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:47.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:41:47.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:41:47.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:41:47.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:41:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:46 smithi104 ceph-mon[97044]: pgmap v19598: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:41:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:41:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:41:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:41:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:47.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:47.034472+0000) 2022-09-10T14:41:48.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:47.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:47.655869+0000) 2022-09-10T14:41:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:48.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:48.034706+0000) 2022-09-10T14:41:49.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:48.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:48.656132+0000) 2022-09-10T14:41:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:48 smithi104 ceph-mon[97044]: pgmap v19599: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:49.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:49.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:49.034944+0000) 2022-09-10T14:41:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:48 smithi110 ceph-mon[97633]: pgmap v19599: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:50.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:49.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:49.656339+0000) 2022-09-10T14:41:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:50.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:50.035162+0000) 2022-09-10T14:41:51.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:50.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:50.656604+0000) 2022-09-10T14:41:51.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:41:50.716+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:41:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:50 smithi104 ceph-mon[97044]: pgmap v19600: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:51.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:51.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:51.035380+0000) 2022-09-10T14:41:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:50 smithi110 ceph-mon[97633]: pgmap v19600: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:52.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:51.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:51.656839+0000) 2022-09-10T14:41:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:52.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:52.035685+0000) 2022-09-10T14:41:53.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:52.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:52.656984+0000) 2022-09-10T14:41:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:52 smithi104 ceph-mon[97044]: pgmap v19601: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:53.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:53.035929+0000) 2022-09-10T14:41:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:52 smithi110 ceph-mon[97633]: pgmap v19601: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:54.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:53.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:53.657154+0000) 2022-09-10T14:41:54.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:54.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:54.036223+0000) 2022-09-10T14:41:55.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:54.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:54.657357+0000) 2022-09-10T14:41:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:54 smithi104 ceph-mon[97044]: pgmap v19602: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:55.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:55.036453+0000) 2022-09-10T14:41:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:54 smithi110 ceph-mon[97633]: pgmap v19602: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:56.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:55.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:55.657524+0000) 2022-09-10T14:41:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:56.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:56.036618+0000) 2022-09-10T14:41:57.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:56.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:56.657711+0000) 2022-09-10T14:41:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:56 smithi104 ceph-mon[97044]: pgmap v19603: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:57.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:57.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:57.036857+0000) 2022-09-10T14:41:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:56 smithi110 ceph-mon[97633]: pgmap v19603: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:41:58.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:57.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:57.657874+0000) 2022-09-10T14:41:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:58.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:58.037019+0000) 2022-09-10T14:41:59.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:58.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:58.658170+0000) 2022-09-10T14:41:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:41:58 smithi104 ceph-mon[97044]: pgmap v19604: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:41:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:41:59.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:59.037163+0000) 2022-09-10T14:41:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:41:58 smithi110 ceph-mon[97633]: pgmap v19604: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:00.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:41:59.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:41:59.658398+0000) 2022-09-10T14:42:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:00.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:00.037393+0000) 2022-09-10T14:42:01.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:00.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:00.658628+0000) 2022-09-10T14:42:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:00 smithi104 ceph-mon[97044]: pgmap v19605: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:01.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:01.037652+0000) 2022-09-10T14:42:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:00 smithi110 ceph-mon[97633]: pgmap v19605: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:02.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:01.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:01.658876+0000) 2022-09-10T14:42:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:02.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:02.037818+0000) 2022-09-10T14:42:03.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:02.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:02.659115+0000) 2022-09-10T14:42:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:02 smithi104 ceph-mon[97044]: pgmap v19606: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:03.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:03.038068+0000) 2022-09-10T14:42:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:02 smithi110 ceph-mon[97633]: pgmap v19606: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:04.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:03.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:03.659304+0000) 2022-09-10T14:42:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:04.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:04.038291+0000) 2022-09-10T14:42:05.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:04.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:04.659565+0000) 2022-09-10T14:42:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:04 smithi104 ceph-mon[97044]: pgmap v19607: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:05.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:05.038509+0000) 2022-09-10T14:42:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:04 smithi110 ceph-mon[97633]: pgmap v19607: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:06.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:05.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:05.659795+0000) 2022-09-10T14:42:06.041 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:42:05.730+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:42:06.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:06.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:06.038674+0000) 2022-09-10T14:42:07.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:06.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:06.659988+0000) 2022-09-10T14:42:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:06 smithi104 ceph-mon[97044]: pgmap v19608: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:42:07.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:07.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:07.038895+0000) 2022-09-10T14:42:07.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:06 smithi110 ceph-mon[97633]: pgmap v19608: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:42:08.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:07.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:07.660214+0000) 2022-09-10T14:42:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:08.037+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:08.039073+0000) 2022-09-10T14:42:09.040 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:08.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:08.660395+0000) 2022-09-10T14:42:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:08 smithi104 ceph-mon[97044]: pgmap v19609: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:42:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:08 smithi104 ceph-mon[97044]: osdmap e1306: 12 total, 12 up, 12 in 2022-09-10T14:42:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:09.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:09.039260+0000) 2022-09-10T14:42:09.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:08 smithi110 ceph-mon[97633]: pgmap v19609: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:42:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:08 smithi110 ceph-mon[97633]: osdmap e1306: 12 total, 12 up, 12 in 2022-09-10T14:42:10.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:09.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:09.660617+0000) 2022-09-10T14:42:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:10.038+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:10.039502+0000) 2022-09-10T14:42:11.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:10.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:10.660839+0000) 2022-09-10T14:42:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:10 smithi104 ceph-mon[97044]: pgmap v19611: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:42:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:11.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:11.039776+0000) 2022-09-10T14:42:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:10 smithi110 ceph-mon[97633]: pgmap v19611: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:42:12.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:11.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:11.661071+0000) 2022-09-10T14:42:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:12.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:12.039969+0000) 2022-09-10T14:42:13.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:12.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:12.661355+0000) 2022-09-10T14:42:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:12 smithi110 ceph-mon[97633]: pgmap v19612: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:42:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:12 smithi110 ceph-mon[97633]: osdmap e1307: 12 total, 12 up, 12 in 2022-09-10T14:42:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:12 smithi104 ceph-mon[97044]: pgmap v19612: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:42:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:12 smithi104 ceph-mon[97044]: osdmap e1307: 12 total, 12 up, 12 in 2022-09-10T14:42:13.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:13.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:13.040197+0000) 2022-09-10T14:42:14.041 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:13.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:13.661518+0000) 2022-09-10T14:42:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:14.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:14.040446+0000) 2022-09-10T14:42:15.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:14.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:14.661732+0000) 2022-09-10T14:42:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:14 smithi110 ceph-mon[97633]: pgmap v19614: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:42:15.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:14 smithi104 ceph-mon[97044]: pgmap v19614: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:42:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:15.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:15.040701+0000) 2022-09-10T14:42:16.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:15.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:15.661990+0000) 2022-09-10T14:42:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:16.039+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:16.040994+0000) 2022-09-10T14:42:17.042 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:16.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:16.662186+0000) 2022-09-10T14:42:17.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:16 smithi110 ceph-mon[97633]: pgmap v19615: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:42:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:16 smithi104 ceph-mon[97044]: pgmap v19615: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:42:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:17.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:17.041205+0000) 2022-09-10T14:42:18.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:17.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:17.662420+0000) 2022-09-10T14:42:18.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:18.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:18.041370+0000) 2022-09-10T14:42:19.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:18.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:18.662576+0000) 2022-09-10T14:42:19.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:18 smithi110 ceph-mon[97633]: pgmap v19616: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:42:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:18 smithi104 ceph-mon[97044]: pgmap v19616: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:42:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:19.040+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:19.041552+0000) 2022-09-10T14:42:20.043 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:19.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:19.662765+0000) 2022-09-10T14:42:20.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:20.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:20.041727+0000) 2022-09-10T14:42:21.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:42:20.746+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:42:21.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:20.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:20.662935+0000) 2022-09-10T14:42:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:20 smithi110 ceph-mon[97633]: pgmap v19617: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:42:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:20 smithi104 ceph-mon[97044]: pgmap v19617: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:42:21.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:21.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:21.041987+0000) 2022-09-10T14:42:22.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:21.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:21.663144+0000) 2022-09-10T14:42:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:22.041+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:22.042260+0000) 2022-09-10T14:42:23.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:22.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:22.663376+0000) 2022-09-10T14:42:23.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:22 smithi110 ceph-mon[97633]: pgmap v19618: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:42:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:22 smithi104 ceph-mon[97044]: pgmap v19618: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:42:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:23.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:23.042523+0000) 2022-09-10T14:42:24.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:23.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:23.663587+0000) 2022-09-10T14:42:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:24.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:24.042784+0000) 2022-09-10T14:42:25.044 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:24.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:24.663833+0000) 2022-09-10T14:42:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:24 smithi110 ceph-mon[97633]: pgmap v19619: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:42:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:24 smithi104 ceph-mon[97044]: pgmap v19619: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:42:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:25.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:25.043007+0000) 2022-09-10T14:42:26.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:25.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:25.664067+0000) 2022-09-10T14:42:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:26.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:26.043245+0000) 2022-09-10T14:42:27.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:26.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:26.664272+0000) 2022-09-10T14:42:27.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:26 smithi110 ceph-mon[97633]: pgmap v19620: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:42:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:26 smithi104 ceph-mon[97044]: pgmap v19620: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:42:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:27.042+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:27.043441+0000) 2022-09-10T14:42:28.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:27.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:27.664461+0000) 2022-09-10T14:42:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:28.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:28.043725+0000) 2022-09-10T14:42:29.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:28.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:28.664713+0000) 2022-09-10T14:42:29.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:28 smithi110 ceph-mon[97633]: pgmap v19621: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:42:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:28 smithi104 ceph-mon[97044]: pgmap v19621: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:42:29.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:29.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:29.043927+0000) 2022-09-10T14:42:30.045 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:29.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:29.664908+0000) 2022-09-10T14:42:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:30.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:30.044192+0000) 2022-09-10T14:42:31.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:30.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:30.665179+0000) 2022-09-10T14:42:31.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:30 smithi110 ceph-mon[97633]: pgmap v19622: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:30 smithi104 ceph-mon[97044]: pgmap v19622: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:31.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:31.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:31.044446+0000) 2022-09-10T14:42:32.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:31.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:31.665375+0000) 2022-09-10T14:42:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:32.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:32.044578+0000) 2022-09-10T14:42:33.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:32.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:32.665616+0000) 2022-09-10T14:42:33.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:32 smithi110 ceph-mon[97633]: pgmap v19623: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:42:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:32 smithi104 ceph-mon[97044]: pgmap v19623: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:42:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:33.043+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:33.044842+0000) 2022-09-10T14:42:34.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:33.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:33.665891+0000) 2022-09-10T14:42:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:34.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:34.045128+0000) 2022-09-10T14:42:35.046 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:34.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:34.666190+0000) 2022-09-10T14:42:35.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:34 smithi110 ceph-mon[97633]: pgmap v19624: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:42:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:34 smithi104 ceph-mon[97044]: pgmap v19624: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:42:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:35.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:35.045393+0000) 2022-09-10T14:42:36.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:35.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:35.666439+0000) 2022-09-10T14:42:36.048 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:42:35.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:42:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:36.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:36.045630+0000) 2022-09-10T14:42:37.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:36.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:36.666644+0000) 2022-09-10T14:42:37.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:36 smithi110 ceph-mon[97633]: pgmap v19625: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:37.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:36 smithi104 ceph-mon[97044]: pgmap v19625: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:37.044+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:37.045884+0000) 2022-09-10T14:42:38.047 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:37.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:37.666860+0000) 2022-09-10T14:42:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:38.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:38.046147+0000) 2022-09-10T14:42:39.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:38.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:38.667138+0000) 2022-09-10T14:42:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:38 smithi104 ceph-mon[97044]: pgmap v19626: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:39.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:39.046388+0000) 2022-09-10T14:42:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:38 smithi110 ceph-mon[97633]: pgmap v19626: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:40.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:39.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:39.667343+0000) 2022-09-10T14:42:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:40.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:40.046570+0000) 2022-09-10T14:42:41.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:40.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:40.667532+0000) 2022-09-10T14:42:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:40 smithi104 ceph-mon[97044]: pgmap v19627: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:41.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:41.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:41.046797+0000) 2022-09-10T14:42:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:40 smithi110 ceph-mon[97633]: pgmap v19627: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:42.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:41.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:41.667774+0000) 2022-09-10T14:42:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:42.045+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:42.046986+0000) 2022-09-10T14:42:43.048 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:42.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:42.667975+0000) 2022-09-10T14:42:43.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:42 smithi104 ceph-mon[97044]: pgmap v19628: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:43.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:43.047201+0000) 2022-09-10T14:42:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:42 smithi110 ceph-mon[97633]: pgmap v19628: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:44.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:43.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:43.668171+0000) 2022-09-10T14:42:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:44.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:44.047483+0000) 2022-09-10T14:42:45.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:44.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:44.668303+0000) 2022-09-10T14:42:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:44 smithi104 ceph-mon[97044]: pgmap v19629: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:45.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:45.047748+0000) 2022-09-10T14:42:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:44 smithi110 ceph-mon[97633]: pgmap v19629: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:46.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:45.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:45.668448+0000) 2022-09-10T14:42:46.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:46.046+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:46.047971+0000) 2022-09-10T14:42:47.049 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:46.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:46.668636+0000) 2022-09-10T14:42:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:46 smithi104 ceph-mon[97044]: pgmap v19630: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:42:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:42:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:42:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:42:47.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:47.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:47.048198+0000) 2022-09-10T14:42:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:46 smithi110 ceph-mon[97633]: pgmap v19630: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:42:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:42:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:42:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:42:48.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:47.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:47.668818+0000) 2022-09-10T14:42:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:48.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:48.048357+0000) 2022-09-10T14:42:49.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:48.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:48.669008+0000) 2022-09-10T14:42:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:48 smithi104 ceph-mon[97044]: pgmap v19631: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:49.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:49.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:49.048542+0000) 2022-09-10T14:42:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:48 smithi110 ceph-mon[97633]: pgmap v19631: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:50.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:49.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:49.669199+0000) 2022-09-10T14:42:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:50.047+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:50.048680+0000) 2022-09-10T14:42:51.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:50.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:50.669407+0000) 2022-09-10T14:42:51.051 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:42:50.762+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:42:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:50 smithi104 ceph-mon[97044]: pgmap v19632: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:51.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:51.048855+0000) 2022-09-10T14:42:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:50 smithi110 ceph-mon[97633]: pgmap v19632: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:52.050 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:51.668+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:51.669655+0000) 2022-09-10T14:42:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:52.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:52.049124+0000) 2022-09-10T14:42:53.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:52.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:52.669905+0000) 2022-09-10T14:42:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:52 smithi104 ceph-mon[97044]: pgmap v19633: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:53.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:53.049414+0000) 2022-09-10T14:42:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:52 smithi110 ceph-mon[97633]: pgmap v19633: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:54.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:53.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:53.670148+0000) 2022-09-10T14:42:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:54.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:54.049630+0000) 2022-09-10T14:42:55.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:54.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:54.670374+0000) 2022-09-10T14:42:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:54 smithi104 ceph-mon[97044]: pgmap v19634: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:55.048+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:55.049844+0000) 2022-09-10T14:42:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:54 smithi110 ceph-mon[97633]: pgmap v19634: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:56.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:55.669+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:55.670539+0000) 2022-09-10T14:42:56.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:56.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:56.050130+0000) 2022-09-10T14:42:57.051 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:56.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:56.670752+0000) 2022-09-10T14:42:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:56 smithi104 ceph-mon[97044]: pgmap v19635: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:57.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:57.050349+0000) 2022-09-10T14:42:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:56 smithi110 ceph-mon[97633]: pgmap v19635: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:42:58.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:57.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:57.670970+0000) 2022-09-10T14:42:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:58.049+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:58.050524+0000) 2022-09-10T14:42:59.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:58.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:58.671245+0000) 2022-09-10T14:42:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:42:58 smithi104 ceph-mon[97044]: pgmap v19636: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:42:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:42:59.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:59.050742+0000) 2022-09-10T14:42:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:42:58 smithi110 ceph-mon[97633]: pgmap v19636: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:00.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:42:59.670+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:42:59.671492+0000) 2022-09-10T14:43:00.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:00.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:00.050990+0000) 2022-09-10T14:43:01.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:00.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:00.671667+0000) 2022-09-10T14:43:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:00 smithi104 ceph-mon[97044]: pgmap v19637: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:01.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:01.051163+0000) 2022-09-10T14:43:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:00 smithi110 ceph-mon[97633]: pgmap v19637: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:02.052 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:01.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:01.671870+0000) 2022-09-10T14:43:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:02.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:02.051356+0000) 2022-09-10T14:43:03.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:02.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:02.672068+0000) 2022-09-10T14:43:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:02 smithi104 ceph-mon[97044]: pgmap v19638: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:03.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:03.051620+0000) 2022-09-10T14:43:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:02 smithi110 ceph-mon[97633]: pgmap v19638: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:04.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:03.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:03.672283+0000) 2022-09-10T14:43:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:04.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:04.051844+0000) 2022-09-10T14:43:05.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:04.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:04.672520+0000) 2022-09-10T14:43:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:04 smithi104 ceph-mon[97044]: pgmap v19639: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:05.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:05.050+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:05.052054+0000) 2022-09-10T14:43:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:04 smithi110 ceph-mon[97633]: pgmap v19639: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:06.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:43:05.769+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:43:06.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:05.671+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:05.672755+0000) 2022-09-10T14:43:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:06.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:06.052302+0000) 2022-09-10T14:43:07.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:06.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:06.673036+0000) 2022-09-10T14:43:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:06 smithi104 ceph-mon[97044]: pgmap v19640: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:43:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:07.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:07.052480+0000) 2022-09-10T14:43:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:06 smithi110 ceph-mon[97633]: pgmap v19640: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:43:08.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:07.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:07.673261+0000) 2022-09-10T14:43:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:08.051+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:08.052698+0000) 2022-09-10T14:43:09.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:08.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:08.673469+0000) 2022-09-10T14:43:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:08 smithi104 ceph-mon[97044]: pgmap v19641: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:43:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:08 smithi104 ceph-mon[97044]: osdmap e1308: 12 total, 12 up, 12 in 2022-09-10T14:43:09.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:09.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:09.052941+0000) 2022-09-10T14:43:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:08 smithi110 ceph-mon[97633]: pgmap v19641: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:43:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:08 smithi110 ceph-mon[97633]: osdmap e1308: 12 total, 12 up, 12 in 2022-09-10T14:43:10.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:09.672+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:09.673698+0000) 2022-09-10T14:43:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:10.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:10.053159+0000) 2022-09-10T14:43:11.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:10.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:10.673895+0000) 2022-09-10T14:43:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:10 smithi104 ceph-mon[97044]: pgmap v19643: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:43:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:10 smithi110 ceph-mon[97633]: pgmap v19643: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:43:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:11.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:11.053384+0000) 2022-09-10T14:43:12.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:11.673+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:11.674152+0000) 2022-09-10T14:43:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:12.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:12.053530+0000) 2022-09-10T14:43:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:12.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:12.674321+0000) 2022-09-10T14:43:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:12 smithi110 ceph-mon[97633]: pgmap v19644: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:43:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:12 smithi110 ceph-mon[97633]: osdmap e1309: 12 total, 12 up, 12 in 2022-09-10T14:43:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:12 smithi104 ceph-mon[97044]: pgmap v19644: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:43:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:12 smithi104 ceph-mon[97044]: osdmap e1309: 12 total, 12 up, 12 in 2022-09-10T14:43:13.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:13.052+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:13.053763+0000) 2022-09-10T14:43:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:13.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:13.674584+0000) 2022-09-10T14:43:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:14.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:14.053977+0000) 2022-09-10T14:43:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:14.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:14.674842+0000) 2022-09-10T14:43:15.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:14 smithi110 ceph-mon[97633]: pgmap v19646: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:43:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:14 smithi104 ceph-mon[97044]: pgmap v19646: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:43:15.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:15.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:15.054122+0000) 2022-09-10T14:43:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:15.674+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:15.675074+0000) 2022-09-10T14:43:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:16.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:16.054246+0000) 2022-09-10T14:43:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:16.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:16.675333+0000) 2022-09-10T14:43:17.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:16 smithi110 ceph-mon[97633]: pgmap v19647: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:43:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:16 smithi104 ceph-mon[97044]: pgmap v19647: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:43:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:17.053+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:17.054362+0000) 2022-09-10T14:43:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:17.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:17.675583+0000) 2022-09-10T14:43:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:18.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:18.054484+0000) 2022-09-10T14:43:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:18.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:18.675832+0000) 2022-09-10T14:43:19.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:18 smithi110 ceph-mon[97633]: pgmap v19648: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:43:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:18 smithi104 ceph-mon[97044]: pgmap v19648: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:43:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:19.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:19.054620+0000) 2022-09-10T14:43:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:19.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:19.676033+0000) 2022-09-10T14:43:20.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:20.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:20.054783+0000) 2022-09-10T14:43:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:43:20.777+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:43:21.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:20.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:20.676269+0000) 2022-09-10T14:43:21.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:20 smithi110 ceph-mon[97633]: pgmap v19649: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:43:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:20 smithi104 ceph-mon[97044]: pgmap v19649: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:43:21.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:21.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:21.054958+0000) 2022-09-10T14:43:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:21.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:21.676491+0000) 2022-09-10T14:43:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:22.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:22.055205+0000) 2022-09-10T14:43:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:22.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:22.676661+0000) 2022-09-10T14:43:23.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:22 smithi110 ceph-mon[97633]: pgmap v19650: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:43:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:22 smithi104 ceph-mon[97044]: pgmap v19650: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:43:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:23.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:23.055367+0000) 2022-09-10T14:43:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:23.675+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:23.676909+0000) 2022-09-10T14:43:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:24.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:24.055557+0000) 2022-09-10T14:43:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:24.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:24.677204+0000) 2022-09-10T14:43:25.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:24 smithi110 ceph-mon[97633]: pgmap v19651: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:43:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:24 smithi104 ceph-mon[97044]: pgmap v19651: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:43:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:25.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:25.055775+0000) 2022-09-10T14:43:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:25.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:25.677418+0000) 2022-09-10T14:43:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:26.054+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:26.056023+0000) 2022-09-10T14:43:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:26.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:26.677618+0000) 2022-09-10T14:43:27.157 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:26 smithi110 ceph-mon[97633]: pgmap v19652: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:43:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:26 smithi104 ceph-mon[97044]: pgmap v19652: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:43:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:27.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:27.056217+0000) 2022-09-10T14:43:28.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:27.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:27.677812+0000) 2022-09-10T14:43:28.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:28.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:28.056355+0000) 2022-09-10T14:43:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:28.676+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:28.678036+0000) 2022-09-10T14:43:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:28 smithi104 ceph-mon[97044]: pgmap v19653: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:28 smithi110 ceph-mon[97633]: pgmap v19653: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:29.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:29.056545+0000) 2022-09-10T14:43:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:29.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:29.678266+0000) 2022-09-10T14:43:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:30.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:30.056756+0000) 2022-09-10T14:43:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:30.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:30.678498+0000) 2022-09-10T14:43:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:30 smithi104 ceph-mon[97044]: pgmap v19654: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:30 smithi110 ceph-mon[97633]: pgmap v19654: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:31.055+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:31.057028+0000) 2022-09-10T14:43:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:31.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:31.678683+0000) 2022-09-10T14:43:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:32.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:32.057255+0000) 2022-09-10T14:43:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:32.677+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:32.678925+0000) 2022-09-10T14:43:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:32 smithi104 ceph-mon[97044]: pgmap v19655: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:32 smithi110 ceph-mon[97633]: pgmap v19655: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:33.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:33.057451+0000) 2022-09-10T14:43:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:33.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:33.679165+0000) 2022-09-10T14:43:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:34.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:34.057660+0000) 2022-09-10T14:43:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:34.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:34.679351+0000) 2022-09-10T14:43:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:34 smithi104 ceph-mon[97044]: pgmap v19656: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:34 smithi110 ceph-mon[97633]: pgmap v19656: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:35.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:35.057872+0000) 2022-09-10T14:43:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:35.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:35.679527+0000) 2022-09-10T14:43:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:43:35.790+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:43:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:36.056+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:36.058086+0000) 2022-09-10T14:43:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:36.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:36.679733+0000) 2022-09-10T14:43:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:36 smithi104 ceph-mon[97044]: pgmap v19657: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:36 smithi110 ceph-mon[97633]: pgmap v19657: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:37.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:37.058289+0000) 2022-09-10T14:43:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:37.678+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:37.679898+0000) 2022-09-10T14:43:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:38.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:38.058473+0000) 2022-09-10T14:43:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:38.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:38.680130+0000) 2022-09-10T14:43:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:38 smithi104 ceph-mon[97044]: pgmap v19658: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:38 smithi110 ceph-mon[97633]: pgmap v19658: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:39.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:39.058645+0000) 2022-09-10T14:43:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:39.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:39.680378+0000) 2022-09-10T14:43:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:40.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:40.058789+0000) 2022-09-10T14:43:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:40.679+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:40.680604+0000) 2022-09-10T14:43:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:40 smithi104 ceph-mon[97044]: pgmap v19659: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:40 smithi110 ceph-mon[97633]: pgmap v19659: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:41.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:41.057+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:41.058923+0000) 2022-09-10T14:43:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:41.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:41.680806+0000) 2022-09-10T14:43:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:42.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:42.059120+0000) 2022-09-10T14:43:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:42.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:42.681055+0000) 2022-09-10T14:43:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:42 smithi104 ceph-mon[97044]: pgmap v19660: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:42 smithi110 ceph-mon[97633]: pgmap v19660: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:43.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:43.059306+0000) 2022-09-10T14:43:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:43.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:43.681273+0000) 2022-09-10T14:43:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:44.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:44.059478+0000) 2022-09-10T14:43:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:44.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:44.681483+0000) 2022-09-10T14:43:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:44 smithi104 ceph-mon[97044]: pgmap v19661: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:44 smithi110 ceph-mon[97633]: pgmap v19661: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:45.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:45.059660+0000) 2022-09-10T14:43:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:45.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:45.681613+0000) 2022-09-10T14:43:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:46.058+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:46.059878+0000) 2022-09-10T14:43:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:46.680+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:46.681787+0000) 2022-09-10T14:43:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:46 smithi104 ceph-mon[97044]: pgmap v19662: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:43:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:43:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:43:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:43:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:46 smithi110 ceph-mon[97633]: pgmap v19662: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:43:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:43:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:43:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:43:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:47.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:47.060112+0000) 2022-09-10T14:43:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:47.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:47.682057+0000) 2022-09-10T14:43:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:48.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:48.060335+0000) 2022-09-10T14:43:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:48.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:48.682267+0000) 2022-09-10T14:43:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:48 smithi104 ceph-mon[97044]: pgmap v19663: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:48 smithi110 ceph-mon[97633]: pgmap v19663: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:49.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:49.060492+0000) 2022-09-10T14:43:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:49.681+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:49.682430+0000) 2022-09-10T14:43:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:50.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:50.060749+0000) 2022-09-10T14:43:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:43:50.806+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:43:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:50.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:50.682699+0000) 2022-09-10T14:43:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:50 smithi104 ceph-mon[97044]: pgmap v19664: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:50 smithi110 ceph-mon[97633]: pgmap v19664: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:51.059+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:51.061013+0000) 2022-09-10T14:43:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:51.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:51.682955+0000) 2022-09-10T14:43:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:52.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:52.061242+0000) 2022-09-10T14:43:53.062 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:52.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:52.683203+0000) 2022-09-10T14:43:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:52 smithi104 ceph-mon[97044]: pgmap v19665: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:52 smithi110 ceph-mon[97633]: pgmap v19665: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:53.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:53.061460+0000) 2022-09-10T14:43:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:53.682+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:53.683439+0000) 2022-09-10T14:43:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:54.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:54.061671+0000) 2022-09-10T14:43:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:54.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:54.683691+0000) 2022-09-10T14:43:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:54 smithi104 ceph-mon[97044]: pgmap v19666: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:54 smithi110 ceph-mon[97633]: pgmap v19666: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:55.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:55.061842+0000) 2022-09-10T14:43:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:55.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:55.683942+0000) 2022-09-10T14:43:56.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:56.060+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:56.062038+0000) 2022-09-10T14:43:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:56.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:56.684200+0000) 2022-09-10T14:43:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:56 smithi104 ceph-mon[97044]: pgmap v19667: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:56 smithi110 ceph-mon[97633]: pgmap v19667: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:43:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:57.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:57.062281+0000) 2022-09-10T14:43:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:57.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:57.684373+0000) 2022-09-10T14:43:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:58.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:58.062442+0000) 2022-09-10T14:43:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:58.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:58.684608+0000) 2022-09-10T14:43:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:43:58 smithi104 ceph-mon[97044]: pgmap v19668: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:43:58 smithi110 ceph-mon[97633]: pgmap v19668: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:43:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:43:59.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:59.062639+0000) 2022-09-10T14:44:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:43:59.683+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:43:59.684854+0000) 2022-09-10T14:44:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:00.061+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:00.062854+0000) 2022-09-10T14:44:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:00.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:00.685091+0000) 2022-09-10T14:44:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:00 smithi104 ceph-mon[97044]: pgmap v19669: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:00 smithi110 ceph-mon[97633]: pgmap v19669: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:01.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:01.063102+0000) 2022-09-10T14:44:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:01.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:01.685229+0000) 2022-09-10T14:44:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:02.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:02.063262+0000) 2022-09-10T14:44:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:02.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:02.685470+0000) 2022-09-10T14:44:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:02 smithi104 ceph-mon[97044]: pgmap v19670: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:02 smithi110 ceph-mon[97633]: pgmap v19670: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:03.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:03.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:03.063441+0000) 2022-09-10T14:44:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:03.684+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:03.685670+0000) 2022-09-10T14:44:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:04.062+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:04.063628+0000) 2022-09-10T14:44:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:04.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:04.685921+0000) 2022-09-10T14:44:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:04 smithi104 ceph-mon[97044]: pgmap v19671: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:04 smithi110 ceph-mon[97633]: pgmap v19671: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:05.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:05.063872+0000) 2022-09-10T14:44:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:44:05.815+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:44:06.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:05.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:05.686173+0000) 2022-09-10T14:44:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:06.063+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:06.064111+0000) 2022-09-10T14:44:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:06.685+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:06.686361+0000) 2022-09-10T14:44:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:06 smithi104 ceph-mon[97044]: pgmap v19672: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:44:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:06 smithi110 ceph-mon[97633]: pgmap v19672: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:44:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:07.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:07.064273+0000) 2022-09-10T14:44:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:07.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:07.686507+0000) 2022-09-10T14:44:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:08.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:08.064438+0000) 2022-09-10T14:44:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:08.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:08.686730+0000) 2022-09-10T14:44:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:08 smithi104 ceph-mon[97044]: pgmap v19673: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:44:09.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:08 smithi104 ceph-mon[97044]: osdmap e1310: 12 total, 12 up, 12 in 2022-09-10T14:44:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:08 smithi110 ceph-mon[97633]: pgmap v19673: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:44:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:08 smithi110 ceph-mon[97633]: osdmap e1310: 12 total, 12 up, 12 in 2022-09-10T14:44:09.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:09.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:09.064619+0000) 2022-09-10T14:44:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:09.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:09.686958+0000) 2022-09-10T14:44:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:10.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:10.064814+0000) 2022-09-10T14:44:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:10.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:10.687206+0000) 2022-09-10T14:44:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:10 smithi104 ceph-mon[97044]: pgmap v19675: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:44:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:10 smithi110 ceph-mon[97633]: pgmap v19675: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:44:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:11.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:11.065024+0000) 2022-09-10T14:44:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:11.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:11.687474+0000) 2022-09-10T14:44:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:12.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:12.065256+0000) 2022-09-10T14:44:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:12.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:12.687649+0000) 2022-09-10T14:44:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:12 smithi110 ceph-mon[97633]: pgmap v19676: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:44:13.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:12 smithi110 ceph-mon[97633]: osdmap e1311: 12 total, 12 up, 12 in 2022-09-10T14:44:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:12 smithi104 ceph-mon[97044]: pgmap v19676: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:44:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:12 smithi104 ceph-mon[97044]: osdmap e1311: 12 total, 12 up, 12 in 2022-09-10T14:44:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:13.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:13.065441+0000) 2022-09-10T14:44:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:13.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:13.687859+0000) 2022-09-10T14:44:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:14.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:14.065664+0000) 2022-09-10T14:44:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:14.686+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:14.688054+0000) 2022-09-10T14:44:15.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:14 smithi110 ceph-mon[97633]: pgmap v19678: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:44:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:14 smithi104 ceph-mon[97044]: pgmap v19678: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:44:15.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:15.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:15.065878+0000) 2022-09-10T14:44:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:15.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:15.688276+0000) 2022-09-10T14:44:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:16.064+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:16.066056+0000) 2022-09-10T14:44:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:16.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:16.688540+0000) 2022-09-10T14:44:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:16 smithi104 ceph-mon[97044]: pgmap v19679: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:44:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:16 smithi110 ceph-mon[97633]: pgmap v19679: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:44:17.645 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:17.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:17.066273+0000) 2022-09-10T14:44:18.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:17.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:17.688755+0000) 2022-09-10T14:44:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:18.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:18.066414+0000) 2022-09-10T14:44:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:18.687+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:18.688967+0000) 2022-09-10T14:44:19.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:18 smithi104 ceph-mon[97044]: pgmap v19680: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:44:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:18 smithi110 ceph-mon[97633]: pgmap v19680: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:44:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:19.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:19.066617+0000) 2022-09-10T14:44:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:19.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:19.689241+0000) 2022-09-10T14:44:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:20.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:20.066810+0000) 2022-09-10T14:44:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:20.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:20.689430+0000) 2022-09-10T14:44:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:44:20.830+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:44:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:20 smithi104 ceph-mon[97044]: pgmap v19681: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:44:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:20 smithi110 ceph-mon[97633]: pgmap v19681: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:44:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:21.065+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:21.067039+0000) 2022-09-10T14:44:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:21.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:21.689637+0000) 2022-09-10T14:44:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:22.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:22.067263+0000) 2022-09-10T14:44:23.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:22.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:22.689884+0000) 2022-09-10T14:44:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:22 smithi104 ceph-mon[97044]: pgmap v19682: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:44:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:22 smithi110 ceph-mon[97633]: pgmap v19682: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:44:23.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:23.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:23.067487+0000) 2022-09-10T14:44:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:23.688+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:23.690035+0000) 2022-09-10T14:44:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:24.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:24.067745+0000) 2022-09-10T14:44:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:24.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:24.690248+0000) 2022-09-10T14:44:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:24 smithi104 ceph-mon[97044]: pgmap v19683: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:44:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:24 smithi110 ceph-mon[97633]: pgmap v19683: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:44:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:25.066+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:25.067882+0000) 2022-09-10T14:44:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:25.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:25.690480+0000) 2022-09-10T14:44:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:26.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:26.068134+0000) 2022-09-10T14:44:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:26.689+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:26.690717+0000) 2022-09-10T14:44:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:26 smithi104 ceph-mon[97044]: pgmap v19684: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:44:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:26 smithi110 ceph-mon[97633]: pgmap v19684: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:44:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:27.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:27.068321+0000) 2022-09-10T14:44:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:27.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:27.690961+0000) 2022-09-10T14:44:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:28.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:28.068479+0000) 2022-09-10T14:44:29.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:28.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:28.691176+0000) 2022-09-10T14:44:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:28 smithi104 ceph-mon[97044]: pgmap v19685: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:28 smithi110 ceph-mon[97633]: pgmap v19685: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:29.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:29.068759+0000) 2022-09-10T14:44:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:29.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:29.691387+0000) 2022-09-10T14:44:30.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:30.067+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:30.068925+0000) 2022-09-10T14:44:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:30.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:30.691620+0000) 2022-09-10T14:44:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:30 smithi104 ceph-mon[97044]: pgmap v19686: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:30 smithi110 ceph-mon[97633]: pgmap v19686: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:31.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:31.069192+0000) 2022-09-10T14:44:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:31.690+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:31.691806+0000) 2022-09-10T14:44:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:32.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:32.069386+0000) 2022-09-10T14:44:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:32.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:32.692040+0000) 2022-09-10T14:44:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:32 smithi104 ceph-mon[97044]: pgmap v19687: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:32 smithi110 ceph-mon[97633]: pgmap v19687: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:33.068+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:33.069673+0000) 2022-09-10T14:44:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:33.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:33.692256+0000) 2022-09-10T14:44:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:34.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:34.069941+0000) 2022-09-10T14:44:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:34.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:34.692465+0000) 2022-09-10T14:44:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:34 smithi104 ceph-mon[97044]: pgmap v19688: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:34 smithi110 ceph-mon[97633]: pgmap v19688: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:35.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:35.070152+0000) 2022-09-10T14:44:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:44:35.839+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:44:36.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:35.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:35.692719+0000) 2022-09-10T14:44:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:36.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:36.070427+0000) 2022-09-10T14:44:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:36.691+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:36.692972+0000) 2022-09-10T14:44:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:36 smithi104 ceph-mon[97044]: pgmap v19689: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:36 smithi110 ceph-mon[97633]: pgmap v19689: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:37.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:37.070689+0000) 2022-09-10T14:44:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:37.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:37.693171+0000) 2022-09-10T14:44:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:38.069+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:38.070919+0000) 2022-09-10T14:44:39.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:38.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:38.693380+0000) 2022-09-10T14:44:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:38 smithi104 ceph-mon[97044]: pgmap v19690: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:38 smithi110 ceph-mon[97633]: pgmap v19690: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:39.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:39.071149+0000) 2022-09-10T14:44:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:39.692+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:39.693577+0000) 2022-09-10T14:44:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:40.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:40.071307+0000) 2022-09-10T14:44:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:40.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:40.693788+0000) 2022-09-10T14:44:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:40 smithi104 ceph-mon[97044]: pgmap v19691: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:40 smithi110 ceph-mon[97633]: pgmap v19691: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:41.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:41.071512+0000) 2022-09-10T14:44:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:41.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:41.694023+0000) 2022-09-10T14:44:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:42.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:42.071678+0000) 2022-09-10T14:44:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:42.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:42.694256+0000) 2022-09-10T14:44:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:42 smithi104 ceph-mon[97044]: pgmap v19692: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:42 smithi110 ceph-mon[97633]: pgmap v19692: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:43.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:43.071862+0000) 2022-09-10T14:44:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:43.693+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:43.694462+0000) 2022-09-10T14:44:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:44.070+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:44.072065+0000) 2022-09-10T14:44:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:44.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:44.694695+0000) 2022-09-10T14:44:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:44 smithi104 ceph-mon[97044]: pgmap v19693: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:44 smithi110 ceph-mon[97633]: pgmap v19693: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:45.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:45.072198+0000) 2022-09-10T14:44:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:45.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:45.694854+0000) 2022-09-10T14:44:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:46.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:46.072400+0000) 2022-09-10T14:44:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:46.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:46.695031+0000) 2022-09-10T14:44:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:46 smithi104 ceph-mon[97044]: pgmap v19694: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:44:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:44:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:44:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:44:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:46 smithi110 ceph-mon[97633]: pgmap v19694: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:44:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:44:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:44:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:44:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:47.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:47.072616+0000) 2022-09-10T14:44:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:47.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:47.695225+0000) 2022-09-10T14:44:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:48.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:48.072794+0000) 2022-09-10T14:44:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:48.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:48.695446+0000) 2022-09-10T14:44:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:48 smithi104 ceph-mon[97044]: pgmap v19695: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:48 smithi110 ceph-mon[97633]: pgmap v19695: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:49.071+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:49.072985+0000) 2022-09-10T14:44:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:49.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:49.695615+0000) 2022-09-10T14:44:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:50.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:50.073229+0000) 2022-09-10T14:44:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:50.694+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:50.695830+0000) 2022-09-10T14:44:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:44:50.854+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:44:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:50 smithi104 ceph-mon[97044]: pgmap v19696: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:50 smithi110 ceph-mon[97633]: pgmap v19696: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:51.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:51.073477+0000) 2022-09-10T14:44:52.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:51.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:51.696045+0000) 2022-09-10T14:44:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:52.072+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:52.073643+0000) 2022-09-10T14:44:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:52.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:52.696290+0000) 2022-09-10T14:44:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:52 smithi104 ceph-mon[97044]: pgmap v19697: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:52 smithi110 ceph-mon[97633]: pgmap v19697: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:53.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:53.073800+0000) 2022-09-10T14:44:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:53.695+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:53.696422+0000) 2022-09-10T14:44:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:54.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:54.074015+0000) 2022-09-10T14:44:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:54.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:54.696620+0000) 2022-09-10T14:44:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:54 smithi104 ceph-mon[97044]: pgmap v19698: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:54 smithi110 ceph-mon[97633]: pgmap v19698: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:55.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:55.074160+0000) 2022-09-10T14:44:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:55.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:55.696840+0000) 2022-09-10T14:44:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:56.073+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:56.074384+0000) 2022-09-10T14:44:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:56.696+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:56.697077+0000) 2022-09-10T14:44:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:56 smithi104 ceph-mon[97044]: pgmap v19699: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:56 smithi110 ceph-mon[97633]: pgmap v19699: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:44:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:57.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:57.074598+0000) 2022-09-10T14:44:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:57.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:57.697289+0000) 2022-09-10T14:44:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:58.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:58.074784+0000) 2022-09-10T14:44:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:58.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:58.697477+0000) 2022-09-10T14:44:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:44:58 smithi104 ceph-mon[97044]: pgmap v19700: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:44:58 smithi110 ceph-mon[97633]: pgmap v19700: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:44:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:44:59.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:59.074994+0000) 2022-09-10T14:45:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:44:59.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:44:59.697700+0000) 2022-09-10T14:45:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:00.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:00.075167+0000) 2022-09-10T14:45:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:00.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:00.697904+0000) 2022-09-10T14:45:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:00 smithi104 ceph-mon[97044]: pgmap v19701: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:00 smithi110 ceph-mon[97633]: pgmap v19701: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:01.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:01.075339+0000) 2022-09-10T14:45:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:01.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:01.698177+0000) 2022-09-10T14:45:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:02.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:02.075533+0000) 2022-09-10T14:45:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:02.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:02.698440+0000) 2022-09-10T14:45:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:02 smithi104 ceph-mon[97044]: pgmap v19702: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:45:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:02 smithi110 ceph-mon[97633]: pgmap v19702: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:45:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:03.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:03.075676+0000) 2022-09-10T14:45:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:03.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:03.698614+0000) 2022-09-10T14:45:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:04.074+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:04.075904+0000) 2022-09-10T14:45:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:04.697+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:04.698864+0000) 2022-09-10T14:45:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:04 smithi104 ceph-mon[97044]: pgmap v19703: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:45:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:04 smithi110 ceph-mon[97633]: pgmap v19703: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:45:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:05.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:05.076118+0000) 2022-09-10T14:45:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:05.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:05.699125+0000) 2022-09-10T14:45:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:45:05.867+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:45:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:06.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:06.076250+0000) 2022-09-10T14:45:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:06.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:06.699357+0000) 2022-09-10T14:45:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:06 smithi104 ceph-mon[97044]: pgmap v19704: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:45:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:06 smithi110 ceph-mon[97633]: pgmap v19704: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:45:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:07.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:07.076441+0000) 2022-09-10T14:45:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:07.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:07.699559+0000) 2022-09-10T14:45:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:08.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:08.076594+0000) 2022-09-10T14:45:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:08.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:08.699745+0000) 2022-09-10T14:45:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:08 smithi104 ceph-mon[97044]: pgmap v19705: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:45:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:08 smithi104 ceph-mon[97044]: osdmap e1312: 12 total, 12 up, 12 in 2022-09-10T14:45:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:08 smithi110 ceph-mon[97633]: pgmap v19705: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:45:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:08 smithi110 ceph-mon[97633]: osdmap e1312: 12 total, 12 up, 12 in 2022-09-10T14:45:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:09.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:09.076827+0000) 2022-09-10T14:45:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:09.698+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:09.699961+0000) 2022-09-10T14:45:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:10.075+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:10.077070+0000) 2022-09-10T14:45:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:10.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:10.700236+0000) 2022-09-10T14:45:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:10 smithi104 ceph-mon[97044]: pgmap v19707: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:45:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:10 smithi110 ceph-mon[97633]: pgmap v19707: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:45:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:11.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:11.077296+0000) 2022-09-10T14:45:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:11.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:11.700435+0000) 2022-09-10T14:45:12.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:12.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:12.077506+0000) 2022-09-10T14:45:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:12.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:12.700613+0000) 2022-09-10T14:45:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:12 smithi104 ceph-mon[97044]: pgmap v19708: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:45:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:12 smithi104 ceph-mon[97044]: osdmap e1313: 12 total, 12 up, 12 in 2022-09-10T14:45:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:12 smithi110 ceph-mon[97633]: pgmap v19708: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:45:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:12 smithi110 ceph-mon[97633]: osdmap e1313: 12 total, 12 up, 12 in 2022-09-10T14:45:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:13.076+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:13.077661+0000) 2022-09-10T14:45:14.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:13.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:13.700796+0000) 2022-09-10T14:45:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:14.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:14.077894+0000) 2022-09-10T14:45:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:14.699+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:14.701017+0000) 2022-09-10T14:45:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:14 smithi104 ceph-mon[97044]: pgmap v19710: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:45:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:14 smithi110 ceph-mon[97633]: pgmap v19710: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:45:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:15.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:15.078190+0000) 2022-09-10T14:45:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:15.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:15.701191+0000) 2022-09-10T14:45:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:16.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:16.078429+0000) 2022-09-10T14:45:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:16.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:16.701407+0000) 2022-09-10T14:45:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:16 smithi104 ceph-mon[97044]: pgmap v19711: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:45:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:16 smithi110 ceph-mon[97633]: pgmap v19711: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:45:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:17.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:17.078640+0000) 2022-09-10T14:45:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:17.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:17.701614+0000) 2022-09-10T14:45:18.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:18.077+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:18.078882+0000) 2022-09-10T14:45:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:18.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:18.701830+0000) 2022-09-10T14:45:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:18 smithi104 ceph-mon[97044]: pgmap v19712: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:45:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:18 smithi110 ceph-mon[97633]: pgmap v19712: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:45:19.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:19.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:19.079073+0000) 2022-09-10T14:45:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:19.700+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:19.702031+0000) 2022-09-10T14:45:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:20.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:20.079336+0000) 2022-09-10T14:45:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:45:20.879+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:45:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:20.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:20.702280+0000) 2022-09-10T14:45:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:20 smithi104 ceph-mon[97044]: pgmap v19713: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:45:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:20 smithi110 ceph-mon[97633]: pgmap v19713: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:45:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:21.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:21.079605+0000) 2022-09-10T14:45:22.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:21.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:21.702499+0000) 2022-09-10T14:45:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:22.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:22.079777+0000) 2022-09-10T14:45:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:22.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:22.702756+0000) 2022-09-10T14:45:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:22 smithi104 ceph-mon[97044]: pgmap v19714: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:45:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:22 smithi110 ceph-mon[97633]: pgmap v19714: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:45:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:23.078+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:23.080023+0000) 2022-09-10T14:45:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:23.701+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:23.703016+0000) 2022-09-10T14:45:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:24.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:24.080264+0000) 2022-09-10T14:45:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:24.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:24.703258+0000) 2022-09-10T14:45:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:24 smithi104 ceph-mon[97044]: pgmap v19715: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:45:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:24 smithi110 ceph-mon[97633]: pgmap v19715: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:45:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:25.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:25.080430+0000) 2022-09-10T14:45:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:25.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:25.703468+0000) 2022-09-10T14:45:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:26.079+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:26.080650+0000) 2022-09-10T14:45:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:26.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:26.703649+0000) 2022-09-10T14:45:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:26 smithi104 ceph-mon[97044]: pgmap v19716: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:45:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:26 smithi110 ceph-mon[97633]: pgmap v19716: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:45:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:27.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:27.080883+0000) 2022-09-10T14:45:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:27.702+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:27.703879+0000) 2022-09-10T14:45:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:28.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:28.081145+0000) 2022-09-10T14:45:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:28.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:28.704071+0000) 2022-09-10T14:45:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:28 smithi104 ceph-mon[97044]: pgmap v19717: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:28 smithi110 ceph-mon[97633]: pgmap v19717: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:29.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:29.081314+0000) 2022-09-10T14:45:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:29.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:29.704237+0000) 2022-09-10T14:45:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:30.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:30.081471+0000) 2022-09-10T14:45:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:30.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:30.704444+0000) 2022-09-10T14:45:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:30 smithi104 ceph-mon[97044]: pgmap v19718: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:30 smithi110 ceph-mon[97633]: pgmap v19718: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:31.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:31.081698+0000) 2022-09-10T14:45:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:31.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:31.704637+0000) 2022-09-10T14:45:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:32.080+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:32.081973+0000) 2022-09-10T14:45:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:32.703+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:32.704828+0000) 2022-09-10T14:45:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:32 smithi104 ceph-mon[97044]: pgmap v19719: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:32 smithi110 ceph-mon[97633]: pgmap v19719: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:33.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:33.082176+0000) 2022-09-10T14:45:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:33.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:33.704993+0000) 2022-09-10T14:45:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:34.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:34.082399+0000) 2022-09-10T14:45:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:34.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:34.705161+0000) 2022-09-10T14:45:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:34 smithi104 ceph-mon[97044]: pgmap v19720: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:34 smithi110 ceph-mon[97633]: pgmap v19720: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:35.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:35.082626+0000) 2022-09-10T14:45:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:35.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:35.705328+0000) 2022-09-10T14:45:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:45:35.885+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:45:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:36.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:36.082801+0000) 2022-09-10T14:45:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:36.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:36.705530+0000) 2022-09-10T14:45:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:36 smithi104 ceph-mon[97044]: pgmap v19721: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:36 smithi110 ceph-mon[97633]: pgmap v19721: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:37.081+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:37.083026+0000) 2022-09-10T14:45:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:37.704+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:37.705764+0000) 2022-09-10T14:45:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:38.082+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:38.083251+0000) 2022-09-10T14:45:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:38.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:38.706023+0000) 2022-09-10T14:45:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:38 smithi104 ceph-mon[97044]: pgmap v19722: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:38 smithi110 ceph-mon[97633]: pgmap v19722: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:39.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:39.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:39.083500+0000) 2022-09-10T14:45:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:39.705+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:39.706260+0000) 2022-09-10T14:45:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:40.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:40.083761+0000) 2022-09-10T14:45:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:40.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:40.706433+0000) 2022-09-10T14:45:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:40 smithi104 ceph-mon[97044]: pgmap v19723: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:40 smithi110 ceph-mon[97633]: pgmap v19723: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:41.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:41.083953+0000) 2022-09-10T14:45:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:41.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:41.706663+0000) 2022-09-10T14:45:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:42.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:42.084150+0000) 2022-09-10T14:45:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:42.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:42.706909+0000) 2022-09-10T14:45:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:42 smithi104 ceph-mon[97044]: pgmap v19724: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:42 smithi110 ceph-mon[97633]: pgmap v19724: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:43.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:43.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:43.084352+0000) 2022-09-10T14:45:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:43.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:43.707075+0000) 2022-09-10T14:45:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:44.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:44.084566+0000) 2022-09-10T14:45:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:44.706+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:44.707254+0000) 2022-09-10T14:45:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:44 smithi104 ceph-mon[97044]: pgmap v19725: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:44 smithi110 ceph-mon[97633]: pgmap v19725: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:45.083+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:45.084706+0000) 2022-09-10T14:45:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:45.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:45.707480+0000) 2022-09-10T14:45:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:46.084+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:46.084911+0000) 2022-09-10T14:45:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:46.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:46.707667+0000) 2022-09-10T14:45:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:46 smithi104 ceph-mon[97044]: pgmap v19726: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:45:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:45:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:45:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:45:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:46 smithi110 ceph-mon[97633]: pgmap v19726: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:45:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:45:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:45:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:45:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:47.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:47.085165+0000) 2022-09-10T14:45:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:47.707+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:47.707915+0000) 2022-09-10T14:45:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:48.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:48.085352+0000) 2022-09-10T14:45:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:48.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:48.708188+0000) 2022-09-10T14:45:49.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:48 smithi104 ceph-mon[97044]: pgmap v19727: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:49.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:48 smithi110 ceph-mon[97633]: pgmap v19727: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:49.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:49.085579+0000) 2022-09-10T14:45:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:49.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:49.708394+0000) 2022-09-10T14:45:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:50.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:50.085826+0000) 2022-09-10T14:45:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:45:50.902+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:45:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:50.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:50.708672+0000) 2022-09-10T14:45:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:50 smithi104 ceph-mon[97044]: pgmap v19728: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:50 smithi110 ceph-mon[97633]: pgmap v19728: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:51.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:51.086060+0000) 2022-09-10T14:45:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:51.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:51.708872+0000) 2022-09-10T14:45:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:52.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:52.086314+0000) 2022-09-10T14:45:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:52.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:52.709130+0000) 2022-09-10T14:45:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:52 smithi104 ceph-mon[97044]: pgmap v19729: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:52 smithi110 ceph-mon[97633]: pgmap v19729: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:53.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:53.086548+0000) 2022-09-10T14:45:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:53.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:53.709314+0000) 2022-09-10T14:45:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:54.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:54.086730+0000) 2022-09-10T14:45:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:54.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:54.709520+0000) 2022-09-10T14:45:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:54 smithi104 ceph-mon[97044]: pgmap v19730: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:54 smithi110 ceph-mon[97633]: pgmap v19730: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:55.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:55.085+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:55.086939+0000) 2022-09-10T14:45:56.061 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:55.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:55.709684+0000) 2022-09-10T14:45:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:56.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:56.087185+0000) 2022-09-10T14:45:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:56.708+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:56.709908+0000) 2022-09-10T14:45:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:56 smithi104 ceph-mon[97044]: pgmap v19731: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:56 smithi110 ceph-mon[97633]: pgmap v19731: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:45:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:57.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:57.087367+0000) 2022-09-10T14:45:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:57.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:57.710166+0000) 2022-09-10T14:45:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:58.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:58.087620+0000) 2022-09-10T14:45:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:58.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:58.710374+0000) 2022-09-10T14:45:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:45:58 smithi104 ceph-mon[97044]: pgmap v19732: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:45:58 smithi110 ceph-mon[97633]: pgmap v19732: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:45:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:45:59.086+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:59.087870+0000) 2022-09-10T14:46:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:45:59.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:45:59.710617+0000) 2022-09-10T14:46:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:00.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:00.088074+0000) 2022-09-10T14:46:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:00.709+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:00.710829+0000) 2022-09-10T14:46:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:00 smithi104 ceph-mon[97044]: pgmap v19733: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:00 smithi110 ceph-mon[97633]: pgmap v19733: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:01.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:01.088354+0000) 2022-09-10T14:46:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:01.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:01.710989+0000) 2022-09-10T14:46:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:02.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:02.088530+0000) 2022-09-10T14:46:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:02.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:02.711195+0000) 2022-09-10T14:46:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:02 smithi104 ceph-mon[97044]: pgmap v19734: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:02 smithi110 ceph-mon[97633]: pgmap v19734: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:03.087+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:03.088764+0000) 2022-09-10T14:46:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:03.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:03.711445+0000) 2022-09-10T14:46:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:04.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:04.088995+0000) 2022-09-10T14:46:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:04.710+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:04.711652+0000) 2022-09-10T14:46:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:04 smithi104 ceph-mon[97044]: pgmap v19735: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:04 smithi110 ceph-mon[97633]: pgmap v19735: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:05.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:05.089162+0000) 2022-09-10T14:46:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:05.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:05.711885+0000) 2022-09-10T14:46:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:46:05.916+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:46:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:06.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:06.089406+0000) 2022-09-10T14:46:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:06.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:06.712202+0000) 2022-09-10T14:46:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:06 smithi104 ceph-mon[97044]: pgmap v19736: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:46:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:06 smithi110 ceph-mon[97633]: pgmap v19736: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:46:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:07.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:07.089614+0000) 2022-09-10T14:46:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:07.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:07.712405+0000) 2022-09-10T14:46:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:08.088+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:08.089837+0000) 2022-09-10T14:46:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:08.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:08.712639+0000) 2022-09-10T14:46:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:09 smithi104 ceph-mon[97044]: pgmap v19737: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:46:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:09 smithi104 ceph-mon[97044]: osdmap e1314: 12 total, 12 up, 12 in 2022-09-10T14:46:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:09 smithi110 ceph-mon[97633]: pgmap v19737: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:46:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:09 smithi110 ceph-mon[97633]: osdmap e1314: 12 total, 12 up, 12 in 2022-09-10T14:46:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:09.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:09.090082+0000) 2022-09-10T14:46:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:09.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:09.712811+0000) 2022-09-10T14:46:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:10.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:10.090325+0000) 2022-09-10T14:46:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:10.711+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:10.712989+0000) 2022-09-10T14:46:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:11 smithi104 ceph-mon[97044]: pgmap v19739: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:46:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:11 smithi110 ceph-mon[97633]: pgmap v19739: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:46:11.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:11.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:11.090549+0000) 2022-09-10T14:46:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:11.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:11.713267+0000) 2022-09-10T14:46:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:12.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:12.090832+0000) 2022-09-10T14:46:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:12.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:12.713439+0000) 2022-09-10T14:46:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:12 smithi104 ceph-mon[97044]: pgmap v19740: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:46:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:12 smithi104 ceph-mon[97044]: osdmap e1315: 12 total, 12 up, 12 in 2022-09-10T14:46:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:12 smithi110 ceph-mon[97633]: pgmap v19740: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:46:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:12 smithi110 ceph-mon[97633]: osdmap e1315: 12 total, 12 up, 12 in 2022-09-10T14:46:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:13.089+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:13.090992+0000) 2022-09-10T14:46:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:13.712+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:13.713577+0000) 2022-09-10T14:46:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:14.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:14.091214+0000) 2022-09-10T14:46:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:14.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:14.713807+0000) 2022-09-10T14:46:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:14 smithi104 ceph-mon[97044]: pgmap v19742: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:46:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:14 smithi110 ceph-mon[97633]: pgmap v19742: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:46:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:15.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:15.091480+0000) 2022-09-10T14:46:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:15.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:15.713990+0000) 2022-09-10T14:46:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:16.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:16.091649+0000) 2022-09-10T14:46:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:16.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:16.714254+0000) 2022-09-10T14:46:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:16 smithi104 ceph-mon[97044]: pgmap v19743: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:46:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:16 smithi110 ceph-mon[97633]: pgmap v19743: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:46:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:17.090+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:17.091875+0000) 2022-09-10T14:46:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:17.713+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:17.714450+0000) 2022-09-10T14:46:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:18.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:18.092137+0000) 2022-09-10T14:46:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:18.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:18.714696+0000) 2022-09-10T14:46:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:18 smithi104 ceph-mon[97044]: pgmap v19744: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:46:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:18 smithi110 ceph-mon[97633]: pgmap v19744: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:46:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:19.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:19.092355+0000) 2022-09-10T14:46:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:19.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:19.714937+0000) 2022-09-10T14:46:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:20.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:20.092529+0000) 2022-09-10T14:46:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:46:20.919+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:46:21.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:20.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:20.715189+0000) 2022-09-10T14:46:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:20 smithi104 ceph-mon[97044]: pgmap v19745: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:46:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:20 smithi110 ceph-mon[97633]: pgmap v19745: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:46:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:21.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:21.092761+0000) 2022-09-10T14:46:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:21.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:21.715408+0000) 2022-09-10T14:46:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:22.091+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:22.092983+0000) 2022-09-10T14:46:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:22.714+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:22.715642+0000) 2022-09-10T14:46:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:22 smithi104 ceph-mon[97044]: pgmap v19746: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:46:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:22 smithi110 ceph-mon[97633]: pgmap v19746: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:46:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:23.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:23.093153+0000) 2022-09-10T14:46:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:23.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:23.715845+0000) 2022-09-10T14:46:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:24.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:24.093327+0000) 2022-09-10T14:46:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:24.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:24.716044+0000) 2022-09-10T14:46:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:24 smithi104 ceph-mon[97044]: pgmap v19747: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:46:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:24 smithi110 ceph-mon[97633]: pgmap v19747: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:46:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:25.092+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:25.093466+0000) 2022-09-10T14:46:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:25.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:25.716238+0000) 2022-09-10T14:46:26.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:26.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:26.093629+0000) 2022-09-10T14:46:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:26.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:26.716465+0000) 2022-09-10T14:46:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:26 smithi104 ceph-mon[97044]: pgmap v19748: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:46:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:26 smithi110 ceph-mon[97633]: pgmap v19748: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:46:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:27.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:27.093849+0000) 2022-09-10T14:46:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:27.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:27.716661+0000) 2022-09-10T14:46:28.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:28.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:28.094029+0000) 2022-09-10T14:46:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:28.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:28.716828+0000) 2022-09-10T14:46:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:28 smithi104 ceph-mon[97044]: pgmap v19749: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:28 smithi110 ceph-mon[97633]: pgmap v19749: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:29.093+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:29.094256+0000) 2022-09-10T14:46:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:29.715+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:29.716998+0000) 2022-09-10T14:46:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:30.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:30.094537+0000) 2022-09-10T14:46:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:30.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:30.717204+0000) 2022-09-10T14:46:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:30 smithi104 ceph-mon[97044]: pgmap v19750: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:30 smithi110 ceph-mon[97633]: pgmap v19750: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:31.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:31.094710+0000) 2022-09-10T14:46:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:31.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:31.717406+0000) 2022-09-10T14:46:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:32.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:32.094888+0000) 2022-09-10T14:46:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:32.716+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:32.717629+0000) 2022-09-10T14:46:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:32 smithi104 ceph-mon[97044]: pgmap v19751: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:32 smithi110 ceph-mon[97633]: pgmap v19751: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:33.094+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:33.095155+0000) 2022-09-10T14:46:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:33.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:33.717868+0000) 2022-09-10T14:46:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:34.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:34.095424+0000) 2022-09-10T14:46:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:34.717+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:34.718123+0000) 2022-09-10T14:46:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:34 smithi104 ceph-mon[97044]: pgmap v19752: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:34 smithi110 ceph-mon[97633]: pgmap v19752: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:35.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:35.095671+0000) 2022-09-10T14:46:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:46:35.922+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:46:36.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:35.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:35.718370+0000) 2022-09-10T14:46:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:36.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:36.095836+0000) 2022-09-10T14:46:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:36.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:36.718630+0000) 2022-09-10T14:46:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:36 smithi104 ceph-mon[97044]: pgmap v19753: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:36 smithi110 ceph-mon[97633]: pgmap v19753: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:37.095+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:37.096041+0000) 2022-09-10T14:46:38.054 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:37.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:37.718849+0000) 2022-09-10T14:46:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:38.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:38.096269+0000) 2022-09-10T14:46:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:38.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:38.719121+0000) 2022-09-10T14:46:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:38 smithi104 ceph-mon[97044]: pgmap v19754: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:38 smithi110 ceph-mon[97633]: pgmap v19754: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:39.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:39.096498+0000) 2022-09-10T14:46:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:39.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:39.719349+0000) 2022-09-10T14:46:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:40.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:40.096741+0000) 2022-09-10T14:46:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:40.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:40.719574+0000) 2022-09-10T14:46:41.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:40 smithi104 ceph-mon[97044]: pgmap v19755: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:40 smithi110 ceph-mon[97633]: pgmap v19755: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:41.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:41.096954+0000) 2022-09-10T14:46:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:41.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:41.719803+0000) 2022-09-10T14:46:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:42.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:42.097201+0000) 2022-09-10T14:46:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:42.718+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:42.720068+0000) 2022-09-10T14:46:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:42 smithi104 ceph-mon[97044]: pgmap v19756: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:42 smithi110 ceph-mon[97633]: pgmap v19756: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:43.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:43.097436+0000) 2022-09-10T14:46:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:43.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:43.720247+0000) 2022-09-10T14:46:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:44.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:44.097642+0000) 2022-09-10T14:46:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:44.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:44.720477+0000) 2022-09-10T14:46:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:44 smithi104 ceph-mon[97044]: pgmap v19757: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:44 smithi110 ceph-mon[97633]: pgmap v19757: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:45.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:45.097790+0000) 2022-09-10T14:46:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:45.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:45.720648+0000) 2022-09-10T14:46:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:46.096+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:46.097990+0000) 2022-09-10T14:46:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:46.719+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:46.720807+0000) 2022-09-10T14:46:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:46 smithi104 ceph-mon[97044]: pgmap v19758: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:46:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:46:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:46:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:46 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:46 smithi110 ceph-mon[97633]: pgmap v19758: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:46:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:46:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:46 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:46:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:47.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:47.098241+0000) 2022-09-10T14:46:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:47.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:47.721021+0000) 2022-09-10T14:46:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:48.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:48.098531+0000) 2022-09-10T14:46:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:48.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:48.721226+0000) 2022-09-10T14:46:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:48 smithi104 ceph-mon[97044]: pgmap v19759: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:48 smithi110 ceph-mon[97633]: pgmap v19759: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:49.097+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:49.098720+0000) 2022-09-10T14:46:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:49.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:49.721419+0000) 2022-09-10T14:46:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:50.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:50.098998+0000) 2022-09-10T14:46:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:50.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:50.721664+0000) 2022-09-10T14:46:51.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:46:50.938+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:46:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:50 smithi104 ceph-mon[97044]: pgmap v19760: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:50 smithi110 ceph-mon[97633]: pgmap v19760: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:51.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:51.099169+0000) 2022-09-10T14:46:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:51.720+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:51.721905+0000) 2022-09-10T14:46:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:52.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:52.099407+0000) 2022-09-10T14:46:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:52.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:52.722179+0000) 2022-09-10T14:46:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:52 smithi104 ceph-mon[97044]: pgmap v19761: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:52 smithi110 ceph-mon[97633]: pgmap v19761: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:53.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:53.099572+0000) 2022-09-10T14:46:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:53.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:53.722397+0000) 2022-09-10T14:46:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:54.098+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:54.099801+0000) 2022-09-10T14:46:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:54.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:54.722554+0000) 2022-09-10T14:46:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:54 smithi104 ceph-mon[97044]: pgmap v19762: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:54 smithi110 ceph-mon[97633]: pgmap v19762: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:55.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:55.100020+0000) 2022-09-10T14:46:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:55.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:55.722727+0000) 2022-09-10T14:46:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:56.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:56.100197+0000) 2022-09-10T14:46:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:56.721+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:56.722986+0000) 2022-09-10T14:46:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:57 smithi104 ceph-mon[97044]: pgmap v19763: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:56 smithi110 ceph-mon[97633]: pgmap v19763: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:46:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:57.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:57.100371+0000) 2022-09-10T14:46:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:57.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:57.723172+0000) 2022-09-10T14:46:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:58.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:58.100585+0000) 2022-09-10T14:46:59.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:58.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:58.723411+0000) 2022-09-10T14:46:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:46:59 smithi104 ceph-mon[97044]: pgmap v19764: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:46:59 smithi110 ceph-mon[97633]: pgmap v19764: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:46:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:46:59.099+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:59.100844+0000) 2022-09-10T14:47:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:46:59.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:46:59.723596+0000) 2022-09-10T14:47:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:00.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:00.101134+0000) 2022-09-10T14:47:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:00.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:00.723735+0000) 2022-09-10T14:47:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:01 smithi104 ceph-mon[97044]: pgmap v19765: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:01 smithi110 ceph-mon[97633]: pgmap v19765: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:01.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:01.101439+0000) 2022-09-10T14:47:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:01.722+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:01.723929+0000) 2022-09-10T14:47:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:02.100+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:02.101679+0000) 2022-09-10T14:47:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:02.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:02.724186+0000) 2022-09-10T14:47:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:03 smithi104 ceph-mon[97044]: pgmap v19766: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:03 smithi110 ceph-mon[97633]: pgmap v19766: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:03.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:03.101943+0000) 2022-09-10T14:47:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:03.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:03.724439+0000) 2022-09-10T14:47:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:04.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:04.102181+0000) 2022-09-10T14:47:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:04.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:04.724644+0000) 2022-09-10T14:47:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:05 smithi104 ceph-mon[97044]: pgmap v19767: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:05 smithi110 ceph-mon[97633]: pgmap v19767: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:05.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:05.102357+0000) 2022-09-10T14:47:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:05.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:05.724801+0000) 2022-09-10T14:47:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:47:05.946+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:47:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:06.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:06.102551+0000) 2022-09-10T14:47:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:06.723+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:06.724953+0000) 2022-09-10T14:47:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:07 smithi104 ceph-mon[97044]: pgmap v19768: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:47:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:07 smithi110 ceph-mon[97633]: pgmap v19768: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:47:07.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:07.101+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:07.102758+0000) 2022-09-10T14:47:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:07.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:07.725190+0000) 2022-09-10T14:47:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:08.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:08.102974+0000) 2022-09-10T14:47:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:08.724+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:08.725442+0000) 2022-09-10T14:47:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:09 smithi104 ceph-mon[97044]: pgmap v19769: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:47:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:09 smithi104 ceph-mon[97044]: osdmap e1316: 12 total, 12 up, 12 in 2022-09-10T14:47:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:09 smithi110 ceph-mon[97633]: pgmap v19769: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:47:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:09 smithi110 ceph-mon[97633]: osdmap e1316: 12 total, 12 up, 12 in 2022-09-10T14:47:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:09.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:09.103218+0000) 2022-09-10T14:47:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:09.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:09.725654+0000) 2022-09-10T14:47:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:10.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:10.103392+0000) 2022-09-10T14:47:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:10.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:10.725892+0000) 2022-09-10T14:47:11.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:11 smithi104 ceph-mon[97044]: pgmap v19771: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:47:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:11 smithi110 ceph-mon[97633]: pgmap v19771: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:47:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:11.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:11.103577+0000) 2022-09-10T14:47:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:11.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:11.726140+0000) 2022-09-10T14:47:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:12.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:12.103771+0000) 2022-09-10T14:47:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:12.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:12.726283+0000) 2022-09-10T14:47:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:12 smithi104 ceph-mon[97044]: pgmap v19772: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:47:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:12 smithi104 ceph-mon[97044]: osdmap e1317: 12 total, 12 up, 12 in 2022-09-10T14:47:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:12 smithi110 ceph-mon[97633]: pgmap v19772: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:47:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:12 smithi110 ceph-mon[97633]: osdmap e1317: 12 total, 12 up, 12 in 2022-09-10T14:47:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:13.102+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:13.104048+0000) 2022-09-10T14:47:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:13.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:13.726467+0000) 2022-09-10T14:47:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:14.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:14.104301+0000) 2022-09-10T14:47:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:14.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:14.726678+0000) 2022-09-10T14:47:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:14 smithi104 ceph-mon[97044]: pgmap v19774: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:47:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:14 smithi110 ceph-mon[97633]: pgmap v19774: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:47:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:15.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:15.104499+0000) 2022-09-10T14:47:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:15.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:15.726855+0000) 2022-09-10T14:47:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:16.103+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:16.104706+0000) 2022-09-10T14:47:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:16.725+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:16.727031+0000) 2022-09-10T14:47:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:16 smithi104 ceph-mon[97044]: pgmap v19775: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:47:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:16 smithi110 ceph-mon[97633]: pgmap v19775: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:47:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:17.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:17.104895+0000) 2022-09-10T14:47:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:17.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:17.727252+0000) 2022-09-10T14:47:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:18.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:18.105068+0000) 2022-09-10T14:47:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:18.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:18.727473+0000) 2022-09-10T14:47:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:18 smithi104 ceph-mon[97044]: pgmap v19776: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:47:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:18 smithi110 ceph-mon[97633]: pgmap v19776: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:47:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:19.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:19.105305+0000) 2022-09-10T14:47:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:19.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:19.727616+0000) 2022-09-10T14:47:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:20.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:20.105531+0000) 2022-09-10T14:47:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:20.726+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:20.727865+0000) 2022-09-10T14:47:21.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:47:20.949+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:47:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:20 smithi104 ceph-mon[97044]: pgmap v19777: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:47:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:20 smithi110 ceph-mon[97633]: pgmap v19777: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:47:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:21.104+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:21.105791+0000) 2022-09-10T14:47:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:21.727+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:21.728150+0000) 2022-09-10T14:47:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:22.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:22.106055+0000) 2022-09-10T14:47:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:22.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:22.728386+0000) 2022-09-10T14:47:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:22 smithi104 ceph-mon[97044]: pgmap v19778: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:47:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:22 smithi110 ceph-mon[97633]: pgmap v19778: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:47:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:23.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:23.106249+0000) 2022-09-10T14:47:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:23.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:23.728617+0000) 2022-09-10T14:47:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:24.106+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:24.106491+0000) 2022-09-10T14:47:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:24.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:24.728793+0000) 2022-09-10T14:47:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:24 smithi104 ceph-mon[97044]: pgmap v19779: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:47:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:24 smithi110 ceph-mon[97633]: pgmap v19779: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:47:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:25.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:25.106719+0000) 2022-09-10T14:47:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:25.728+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:25.729010+0000) 2022-09-10T14:47:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:26.105+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:26.106994+0000) 2022-09-10T14:47:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:26.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:26.729229+0000) 2022-09-10T14:47:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:26 smithi104 ceph-mon[97044]: pgmap v19780: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:47:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:26 smithi110 ceph-mon[97633]: pgmap v19780: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:47:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:27.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:27.107214+0000) 2022-09-10T14:47:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:27.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:27.729489+0000) 2022-09-10T14:47:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:28.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:28.107441+0000) 2022-09-10T14:47:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:28.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:28.729735+0000) 2022-09-10T14:47:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:28 smithi104 ceph-mon[97044]: pgmap v19781: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:28 smithi110 ceph-mon[97633]: pgmap v19781: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:29.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:29.107717+0000) 2022-09-10T14:47:30.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:29.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:29.729949+0000) 2022-09-10T14:47:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:30.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:30.107940+0000) 2022-09-10T14:47:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:30.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:30.730210+0000) 2022-09-10T14:47:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:30 smithi104 ceph-mon[97044]: pgmap v19782: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:30 smithi110 ceph-mon[97633]: pgmap v19782: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:31.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:31.108212+0000) 2022-09-10T14:47:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:31.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:31.730398+0000) 2022-09-10T14:47:32.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:32.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:32.108413+0000) 2022-09-10T14:47:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:32.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:32.730600+0000) 2022-09-10T14:47:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:32 smithi104 ceph-mon[97044]: pgmap v19783: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:32 smithi110 ceph-mon[97633]: pgmap v19783: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:33.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:33.108676+0000) 2022-09-10T14:47:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:33.729+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:33.730863+0000) 2022-09-10T14:47:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:34.107+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:34.108945+0000) 2022-09-10T14:47:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:34.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:34.731128+0000) 2022-09-10T14:47:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:34 smithi104 ceph-mon[97044]: pgmap v19784: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:34 smithi110 ceph-mon[97633]: pgmap v19784: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:35.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:35.109226+0000) 2022-09-10T14:47:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:47:35.956+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:47:36.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:35.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:35.731329+0000) 2022-09-10T14:47:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:36.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:36.109470+0000) 2022-09-10T14:47:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:36.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:36.731575+0000) 2022-09-10T14:47:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:36 smithi104 ceph-mon[97044]: pgmap v19785: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:36 smithi110 ceph-mon[97633]: pgmap v19785: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:37.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:37.109733+0000) 2022-09-10T14:47:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:37.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:37.731763+0000) 2022-09-10T14:47:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:38.108+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:38.110000+0000) 2022-09-10T14:47:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:38.730+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:38.731985+0000) 2022-09-10T14:47:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:38 smithi104 ceph-mon[97044]: pgmap v19786: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:38 smithi110 ceph-mon[97633]: pgmap v19786: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:39.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:39.110268+0000) 2022-09-10T14:47:40.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:39.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:39.732234+0000) 2022-09-10T14:47:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:40.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:40.110529+0000) 2022-09-10T14:47:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:40.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:40.732437+0000) 2022-09-10T14:47:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:40 smithi104 ceph-mon[97044]: pgmap v19787: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:40 smithi110 ceph-mon[97633]: pgmap v19787: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:41.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:41.110803+0000) 2022-09-10T14:47:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:41.731+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:41.732674+0000) 2022-09-10T14:47:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:42.109+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:42.111024+0000) 2022-09-10T14:47:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:42.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:42.732926+0000) 2022-09-10T14:47:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:42 smithi104 ceph-mon[97044]: pgmap v19788: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:42 smithi110 ceph-mon[97633]: pgmap v19788: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:43.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:43.111262+0000) 2022-09-10T14:47:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:43.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:43.733173+0000) 2022-09-10T14:47:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:44.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:44.111467+0000) 2022-09-10T14:47:45.063 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:44.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:44.733318+0000) 2022-09-10T14:47:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:44 smithi104 ceph-mon[97044]: pgmap v19789: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:44 smithi110 ceph-mon[97633]: pgmap v19789: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:45.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:45.111756+0000) 2022-09-10T14:47:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:45.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:45.733532+0000) 2022-09-10T14:47:46.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:46.110+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:46.112026+0000) 2022-09-10T14:47:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:46.732+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:46.733788+0000) 2022-09-10T14:47:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:47 smithi104 ceph-mon[97044]: pgmap v19790: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:47:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:47:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:47:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:47:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:47 smithi110 ceph-mon[97633]: pgmap v19790: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:47:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:47:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:47:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:47:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:47.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:47.112313+0000) 2022-09-10T14:47:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:47.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:47.734023+0000) 2022-09-10T14:47:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:48.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:48.112597+0000) 2022-09-10T14:47:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:48.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:48.734221+0000) 2022-09-10T14:47:49.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:49 smithi104 ceph-mon[97044]: pgmap v19791: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:49 smithi110 ceph-mon[97633]: pgmap v19791: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:49.111+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:49.112849+0000) 2022-09-10T14:47:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:49.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:49.734437+0000) 2022-09-10T14:47:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:50.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:50.113156+0000) 2022-09-10T14:47:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:50.733+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:50.734687+0000) 2022-09-10T14:47:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:47:50.957+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:47:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:51 smithi104 ceph-mon[97044]: pgmap v19792: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:51.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:51 smithi110 ceph-mon[97633]: pgmap v19792: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:51.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:51.113423+0000) 2022-09-10T14:47:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:51.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:51.734916+0000) 2022-09-10T14:47:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:52.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:52.113613+0000) 2022-09-10T14:47:53.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:52.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:52.735206+0000) 2022-09-10T14:47:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:53 smithi104 ceph-mon[97044]: pgmap v19793: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:53 smithi110 ceph-mon[97633]: pgmap v19793: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:53.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:53.113787+0000) 2022-09-10T14:47:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:53.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:53.735431+0000) 2022-09-10T14:47:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:54.112+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:54.114045+0000) 2022-09-10T14:47:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:54.734+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:54.735598+0000) 2022-09-10T14:47:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:55 smithi104 ceph-mon[97044]: pgmap v19794: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:55 smithi110 ceph-mon[97633]: pgmap v19794: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:55.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:55.114212+0000) 2022-09-10T14:47:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:55.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:55.735822+0000) 2022-09-10T14:47:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:56.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:56.114352+0000) 2022-09-10T14:47:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:56.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:56.736071+0000) 2022-09-10T14:47:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:57 smithi104 ceph-mon[97044]: pgmap v19795: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:57 smithi110 ceph-mon[97633]: pgmap v19795: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:47:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:57.113+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:57.114595+0000) 2022-09-10T14:47:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:57.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:57.736288+0000) 2022-09-10T14:47:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:58.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:58.114860+0000) 2022-09-10T14:47:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:58.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:58.736453+0000) 2022-09-10T14:47:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:47:59 smithi104 ceph-mon[97044]: pgmap v19796: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:47:59 smithi110 ceph-mon[97633]: pgmap v19796: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:47:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:47:59.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:59.115175+0000) 2022-09-10T14:48:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:47:59.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:47:59.736675+0000) 2022-09-10T14:48:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:00.114+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:00.115376+0000) 2022-09-10T14:48:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:00.735+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:00.736859+0000) 2022-09-10T14:48:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:01 smithi104 ceph-mon[97044]: pgmap v19797: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:01 smithi110 ceph-mon[97633]: pgmap v19797: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:01.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:01.115598+0000) 2022-09-10T14:48:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:01.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:01.737092+0000) 2022-09-10T14:48:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:02.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:02.115816+0000) 2022-09-10T14:48:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:02.736+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:02.737360+0000) 2022-09-10T14:48:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:03 smithi104 ceph-mon[97044]: pgmap v19798: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:03 smithi110 ceph-mon[97633]: pgmap v19798: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:03.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:03.116059+0000) 2022-09-10T14:48:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:03.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:03.737584+0000) 2022-09-10T14:48:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:04.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:04.116343+0000) 2022-09-10T14:48:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:04.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:04.737830+0000) 2022-09-10T14:48:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:05 smithi104 ceph-mon[97044]: pgmap v19799: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:05 smithi110 ceph-mon[97633]: pgmap v19799: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:05.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:05.116560+0000) 2022-09-10T14:48:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:05.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:05.738082+0000) 2022-09-10T14:48:06.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:48:05.966+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:48:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:06.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:06.116788+0000) 2022-09-10T14:48:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:06.737+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:06.738287+0000) 2022-09-10T14:48:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:07 smithi104 ceph-mon[97044]: pgmap v19800: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:48:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:07 smithi110 ceph-mon[97633]: pgmap v19800: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:48:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:07.115+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:07.117031+0000) 2022-09-10T14:48:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:07.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:07.738498+0000) 2022-09-10T14:48:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:08.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:08.117210+0000) 2022-09-10T14:48:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:08.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:08.738735+0000) 2022-09-10T14:48:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:09 smithi104 ceph-mon[97044]: pgmap v19801: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:48:09.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:09 smithi104 ceph-mon[97044]: osdmap e1318: 12 total, 12 up, 12 in 2022-09-10T14:48:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:09 smithi110 ceph-mon[97633]: pgmap v19801: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:48:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:09 smithi110 ceph-mon[97633]: osdmap e1318: 12 total, 12 up, 12 in 2022-09-10T14:48:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:09.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:09.117341+0000) 2022-09-10T14:48:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:09.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:09.738929+0000) 2022-09-10T14:48:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:10.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:10.117501+0000) 2022-09-10T14:48:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:10.738+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:10.739175+0000) 2022-09-10T14:48:11.306 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:11 smithi104 ceph-mon[97044]: pgmap v19803: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:48:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:11 smithi110 ceph-mon[97633]: pgmap v19803: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:48:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:11.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:11.117677+0000) 2022-09-10T14:48:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:11.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:11.739402+0000) 2022-09-10T14:48:12.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:12.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:12.117876+0000) 2022-09-10T14:48:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:12.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:12.739656+0000) 2022-09-10T14:48:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:12 smithi104 ceph-mon[97044]: pgmap v19804: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:48:13.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:12 smithi104 ceph-mon[97044]: osdmap e1319: 12 total, 12 up, 12 in 2022-09-10T14:48:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:12 smithi110 ceph-mon[97633]: pgmap v19804: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:48:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:12 smithi110 ceph-mon[97633]: osdmap e1319: 12 total, 12 up, 12 in 2022-09-10T14:48:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:13.116+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:13.118073+0000) 2022-09-10T14:48:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:13.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:13.739872+0000) 2022-09-10T14:48:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:14.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:14.118165+0000) 2022-09-10T14:48:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:14.739+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:14.740054+0000) 2022-09-10T14:48:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:14 smithi104 ceph-mon[97044]: pgmap v19806: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:48:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:14 smithi110 ceph-mon[97633]: pgmap v19806: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:48:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:15.117+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:15.118299+0000) 2022-09-10T14:48:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:15.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:15.740233+0000) 2022-09-10T14:48:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:16.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:16.118421+0000) 2022-09-10T14:48:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:16.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:16.740462+0000) 2022-09-10T14:48:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:16 smithi104 ceph-mon[97044]: pgmap v19807: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:48:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:16 smithi110 ceph-mon[97633]: pgmap v19807: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:48:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:17.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:17.118610+0000) 2022-09-10T14:48:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:17.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:17.740662+0000) 2022-09-10T14:48:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:18.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:18.118735+0000) 2022-09-10T14:48:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:18.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:18.740930+0000) 2022-09-10T14:48:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:18 smithi104 ceph-mon[97044]: pgmap v19808: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:48:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:18 smithi110 ceph-mon[97633]: pgmap v19808: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:48:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:19.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:19.118899+0000) 2022-09-10T14:48:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:19.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:19.741165+0000) 2022-09-10T14:48:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:20.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:20.119023+0000) 2022-09-10T14:48:21.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:48:20.974+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:48:21.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:20.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:20.741318+0000) 2022-09-10T14:48:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:20 smithi104 ceph-mon[97044]: pgmap v19809: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:48:21.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:20 smithi110 ceph-mon[97633]: pgmap v19809: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:48:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:21.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:21.119163+0000) 2022-09-10T14:48:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:21.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:21.741527+0000) 2022-09-10T14:48:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:22.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:22.119292+0000) 2022-09-10T14:48:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:22.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:22.741760+0000) 2022-09-10T14:48:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:22 smithi104 ceph-mon[97044]: pgmap v19810: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:48:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:22 smithi110 ceph-mon[97633]: pgmap v19810: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:48:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:23.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:23.119418+0000) 2022-09-10T14:48:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:23.740+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:23.742018+0000) 2022-09-10T14:48:24.565 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:24.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:24.119532+0000) 2022-09-10T14:48:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:24.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:24.742261+0000) 2022-09-10T14:48:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:24 smithi104 ceph-mon[97044]: pgmap v19811: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:48:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:24 smithi110 ceph-mon[97633]: pgmap v19811: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:48:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:25.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:25.119658+0000) 2022-09-10T14:48:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:25.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:25.742513+0000) 2022-09-10T14:48:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:26.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:26.119792+0000) 2022-09-10T14:48:27.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:26.741+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:26.742802+0000) 2022-09-10T14:48:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:26 smithi104 ceph-mon[97044]: pgmap v19812: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:48:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:26 smithi110 ceph-mon[97633]: pgmap v19812: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:48:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:27.118+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:27.119925+0000) 2022-09-10T14:48:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:27.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:27.743138+0000) 2022-09-10T14:48:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:28.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:28.120123+0000) 2022-09-10T14:48:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:28.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:28.743291+0000) 2022-09-10T14:48:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:28 smithi104 ceph-mon[97044]: pgmap v19813: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:28 smithi110 ceph-mon[97633]: pgmap v19813: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:29.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:29.120268+0000) 2022-09-10T14:48:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:29.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:29.743458+0000) 2022-09-10T14:48:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:30.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:30.120396+0000) 2022-09-10T14:48:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:30.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:30.743730+0000) 2022-09-10T14:48:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:30 smithi104 ceph-mon[97044]: pgmap v19814: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:30 smithi110 ceph-mon[97633]: pgmap v19814: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:31.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:31.120528+0000) 2022-09-10T14:48:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:31.742+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:31.743875+0000) 2022-09-10T14:48:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:32.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:32.120648+0000) 2022-09-10T14:48:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:32.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:32.744135+0000) 2022-09-10T14:48:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:32 smithi104 ceph-mon[97044]: pgmap v19815: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:32 smithi110 ceph-mon[97633]: pgmap v19815: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:33.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:33.120844+0000) 2022-09-10T14:48:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:33.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:33.744371+0000) 2022-09-10T14:48:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:34.119+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:34.120996+0000) 2022-09-10T14:48:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:34.743+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:34.744685+0000) 2022-09-10T14:48:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:34 smithi104 ceph-mon[97044]: pgmap v19816: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:34 smithi110 ceph-mon[97633]: pgmap v19816: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:35.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:35.121174+0000) 2022-09-10T14:48:36.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:48:35.988+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:48:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:35.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:35.744950+0000) 2022-09-10T14:48:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:36.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:36.121296+0000) 2022-09-10T14:48:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:36.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:36.745166+0000) 2022-09-10T14:48:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:37 smithi104 ceph-mon[97044]: pgmap v19817: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:37 smithi110 ceph-mon[97633]: pgmap v19817: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:37.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:37.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:37.121430+0000) 2022-09-10T14:48:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:37.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:37.745318+0000) 2022-09-10T14:48:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:38.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:38.121563+0000) 2022-09-10T14:48:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:38.744+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:38.745545+0000) 2022-09-10T14:48:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:39 smithi104 ceph-mon[97044]: pgmap v19818: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:39 smithi110 ceph-mon[97633]: pgmap v19818: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:39.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:39.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:39.121691+0000) 2022-09-10T14:48:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:39.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:39.745774+0000) 2022-09-10T14:48:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:40.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:40.121810+0000) 2022-09-10T14:48:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:40.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:40.746012+0000) 2022-09-10T14:48:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:41 smithi104 ceph-mon[97044]: pgmap v19819: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:41 smithi110 ceph-mon[97633]: pgmap v19819: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:41.120+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:41.121943+0000) 2022-09-10T14:48:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:41.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:41.746210+0000) 2022-09-10T14:48:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:42.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:42.122107+0000) 2022-09-10T14:48:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:42.745+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:42.746406+0000) 2022-09-10T14:48:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:43 smithi104 ceph-mon[97044]: pgmap v19820: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:43 smithi110 ceph-mon[97633]: pgmap v19820: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:43.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:43.122239+0000) 2022-09-10T14:48:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:43.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:43.746667+0000) 2022-09-10T14:48:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:44.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:44.122357+0000) 2022-09-10T14:48:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:44.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:44.746905+0000) 2022-09-10T14:48:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:45 smithi104 ceph-mon[97044]: pgmap v19821: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:45 smithi110 ceph-mon[97633]: pgmap v19821: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:45.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:45.122557+0000) 2022-09-10T14:48:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:45.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:45.747148+0000) 2022-09-10T14:48:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:46.121+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:46.122691+0000) 2022-09-10T14:48:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:46.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:46.747377+0000) 2022-09-10T14:48:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:47 smithi104 ceph-mon[97044]: pgmap v19822: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:48:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:48:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:48:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:48:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:47 smithi110 ceph-mon[97633]: pgmap v19822: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:48:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:48:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:48:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:48:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:47.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:47.122823+0000) 2022-09-10T14:48:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:47.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:47.747606+0000) 2022-09-10T14:48:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:48.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:48.122997+0000) 2022-09-10T14:48:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:48.746+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:48.747872+0000) 2022-09-10T14:48:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:49 smithi104 ceph-mon[97044]: pgmap v19823: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:49 smithi110 ceph-mon[97633]: pgmap v19823: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:49.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:49.123219+0000) 2022-09-10T14:48:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:49.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:49.748092+0000) 2022-09-10T14:48:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:50.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:50.123458+0000) 2022-09-10T14:48:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:50.747+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:50.748311+0000) 2022-09-10T14:48:51.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:48:50.994+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:48:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:51 smithi104 ceph-mon[97044]: pgmap v19824: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:51 smithi110 ceph-mon[97633]: pgmap v19824: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:51.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:51.123598+0000) 2022-09-10T14:48:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:51.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:51.748538+0000) 2022-09-10T14:48:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:52.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:52.123769+0000) 2022-09-10T14:48:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:52.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:52.748783+0000) 2022-09-10T14:48:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:53 smithi104 ceph-mon[97044]: pgmap v19825: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:53 smithi110 ceph-mon[97633]: pgmap v19825: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:53.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:53.123912+0000) 2022-09-10T14:48:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:53.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:53.749018+0000) 2022-09-10T14:48:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:54.122+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:54.124035+0000) 2022-09-10T14:48:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:54.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:54.749258+0000) 2022-09-10T14:48:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:55 smithi104 ceph-mon[97044]: pgmap v19826: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:55 smithi110 ceph-mon[97633]: pgmap v19826: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:55.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:55.124191+0000) 2022-09-10T14:48:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:55.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:55.749487+0000) 2022-09-10T14:48:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:56.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:56.124311+0000) 2022-09-10T14:48:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:56.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:56.749769+0000) 2022-09-10T14:48:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:57 smithi104 ceph-mon[97044]: pgmap v19827: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:57 smithi110 ceph-mon[97633]: pgmap v19827: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:48:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:57.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:57.124435+0000) 2022-09-10T14:48:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:57.748+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:57.750014+0000) 2022-09-10T14:48:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:58.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:58.124558+0000) 2022-09-10T14:48:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:58.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:58.750273+0000) 2022-09-10T14:48:59.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:48:59 smithi104 ceph-mon[97044]: pgmap v19828: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:48:59 smithi110 ceph-mon[97633]: pgmap v19828: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:48:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:48:59.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:59.124679+0000) 2022-09-10T14:49:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:48:59.749+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:48:59.750497+0000) 2022-09-10T14:49:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:00.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:00.124793+0000) 2022-09-10T14:49:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:00.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:00.750707+0000) 2022-09-10T14:49:01.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:01 smithi104 ceph-mon[97044]: pgmap v19829: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:01 smithi110 ceph-mon[97633]: pgmap v19829: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:01.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:01.124915+0000) 2022-09-10T14:49:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:01.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:01.750928+0000) 2022-09-10T14:49:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:02.123+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:02.125069+0000) 2022-09-10T14:49:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:02.750+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:02.751108+0000) 2022-09-10T14:49:03.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:03 smithi104 ceph-mon[97044]: pgmap v19830: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:03 smithi110 ceph-mon[97633]: pgmap v19830: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:03.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:03.125235+0000) 2022-09-10T14:49:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:03.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:03.751357+0000) 2022-09-10T14:49:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:04.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:04.125368+0000) 2022-09-10T14:49:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:04.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:04.751617+0000) 2022-09-10T14:49:05.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:05 smithi104 ceph-mon[97044]: pgmap v19831: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:05 smithi110 ceph-mon[97633]: pgmap v19831: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:05.124+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:05.125500+0000) 2022-09-10T14:49:06.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:05.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:05.751784+0000) 2022-09-10T14:49:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:06.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:06.125673+0000) 2022-09-10T14:49:06.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:49:06.010+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:49:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:06.751+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:06.752010+0000) 2022-09-10T14:49:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:07 smithi110 ceph-mon[97633]: pgmap v19832: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:49:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:07 smithi104 ceph-mon[97044]: pgmap v19832: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:49:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:07.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:07.125837+0000) 2022-09-10T14:49:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:07.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:07.752213+0000) 2022-09-10T14:49:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:08.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:08.125977+0000) 2022-09-10T14:49:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:08.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:08.752379+0000) 2022-09-10T14:49:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:09 smithi110 ceph-mon[97633]: pgmap v19833: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:49:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:09 smithi110 ceph-mon[97633]: osdmap e1320: 12 total, 12 up, 12 in 2022-09-10T14:49:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:09 smithi104 ceph-mon[97044]: pgmap v19833: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:49:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:09 smithi104 ceph-mon[97044]: osdmap e1320: 12 total, 12 up, 12 in 2022-09-10T14:49:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:09.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:09.126145+0000) 2022-09-10T14:49:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:09.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:09.752609+0000) 2022-09-10T14:49:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:10.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:10.126299+0000) 2022-09-10T14:49:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:10.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:10.752879+0000) 2022-09-10T14:49:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:11 smithi110 ceph-mon[97633]: pgmap v19835: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:49:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:11 smithi104 ceph-mon[97044]: pgmap v19835: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:49:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:11.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:11.126519+0000) 2022-09-10T14:49:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:11.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:11.753117+0000) 2022-09-10T14:49:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:12.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:12.126664+0000) 2022-09-10T14:49:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:12.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:12.753336+0000) 2022-09-10T14:49:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:12 smithi104 ceph-mon[97044]: pgmap v19836: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:49:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:12 smithi104 ceph-mon[97044]: osdmap e1321: 12 total, 12 up, 12 in 2022-09-10T14:49:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:12 smithi110 ceph-mon[97633]: pgmap v19836: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:49:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:12 smithi110 ceph-mon[97633]: osdmap e1321: 12 total, 12 up, 12 in 2022-09-10T14:49:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:13.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:13.126793+0000) 2022-09-10T14:49:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:13.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:13.753498+0000) 2022-09-10T14:49:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:14.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:14.126932+0000) 2022-09-10T14:49:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:14.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:14.753714+0000) 2022-09-10T14:49:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:14 smithi104 ceph-mon[97044]: pgmap v19838: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:49:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:14 smithi110 ceph-mon[97633]: pgmap v19838: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:49:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:15.125+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:15.127063+0000) 2022-09-10T14:49:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:15.752+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:15.753935+0000) 2022-09-10T14:49:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:16.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:16.127190+0000) 2022-09-10T14:49:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:16.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:16.754187+0000) 2022-09-10T14:49:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:16 smithi104 ceph-mon[97044]: pgmap v19839: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:49:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:16 smithi110 ceph-mon[97633]: pgmap v19839: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:49:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:17.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:17.127390+0000) 2022-09-10T14:49:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:17.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:17.754470+0000) 2022-09-10T14:49:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:18.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:18.127540+0000) 2022-09-10T14:49:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:18.753+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:18.754780+0000) 2022-09-10T14:49:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:18 smithi104 ceph-mon[97044]: pgmap v19840: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:49:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:18 smithi110 ceph-mon[97633]: pgmap v19840: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:49:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:19.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:19.127707+0000) 2022-09-10T14:49:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:19.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:19.754976+0000) 2022-09-10T14:49:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:20.126+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:20.127890+0000) 2022-09-10T14:49:21.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:20.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:20.755221+0000) 2022-09-10T14:49:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:20 smithi104 ceph-mon[97044]: pgmap v19841: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:49:21.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:49:21.025+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:49:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:21.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:21.128113+0000) 2022-09-10T14:49:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:20 smithi110 ceph-mon[97633]: pgmap v19841: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:49:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:21.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:21.755457+0000) 2022-09-10T14:49:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:22.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:22.128330+0000) 2022-09-10T14:49:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:22.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:22.755688+0000) 2022-09-10T14:49:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:22 smithi104 ceph-mon[97044]: pgmap v19842: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:49:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:22 smithi110 ceph-mon[97633]: pgmap v19842: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:49:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:23.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:23.128546+0000) 2022-09-10T14:49:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:23.754+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:23.755928+0000) 2022-09-10T14:49:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:24.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:24.128764+0000) 2022-09-10T14:49:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:24.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:24.756140+0000) 2022-09-10T14:49:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:25 smithi104 ceph-mon[97044]: pgmap v19843: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:49:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:25 smithi110 ceph-mon[97633]: pgmap v19843: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:49:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:25.127+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:25.129023+0000) 2022-09-10T14:49:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:25.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:25.756302+0000) 2022-09-10T14:49:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:26.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:26.129220+0000) 2022-09-10T14:49:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:26.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:26.756501+0000) 2022-09-10T14:49:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:27 smithi104 ceph-mon[97044]: pgmap v19844: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:49:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:27 smithi110 ceph-mon[97633]: pgmap v19844: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:49:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:27.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:27.129450+0000) 2022-09-10T14:49:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:27.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:27.756746+0000) 2022-09-10T14:49:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:28.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:28.129626+0000) 2022-09-10T14:49:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:28.755+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:28.756971+0000) 2022-09-10T14:49:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:29 smithi104 ceph-mon[97044]: pgmap v19845: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:29 smithi110 ceph-mon[97633]: pgmap v19845: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:29.128+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:29.129852+0000) 2022-09-10T14:49:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:29.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:29.757151+0000) 2022-09-10T14:49:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:30.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:30.130105+0000) 2022-09-10T14:49:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:30.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:30.757358+0000) 2022-09-10T14:49:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:31 smithi104 ceph-mon[97044]: pgmap v19846: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:31 smithi110 ceph-mon[97633]: pgmap v19846: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:31.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:31.130293+0000) 2022-09-10T14:49:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:31.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:31.757580+0000) 2022-09-10T14:49:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:32.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:32.130428+0000) 2022-09-10T14:49:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:32.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:32.757829+0000) 2022-09-10T14:49:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:33 smithi104 ceph-mon[97044]: pgmap v19847: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:33 smithi110 ceph-mon[97633]: pgmap v19847: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:33.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:33.130585+0000) 2022-09-10T14:49:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:33.756+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:33.758067+0000) 2022-09-10T14:49:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:34.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:34.130841+0000) 2022-09-10T14:49:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:34.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:34.758286+0000) 2022-09-10T14:49:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:35 smithi104 ceph-mon[97044]: pgmap v19848: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:35 smithi110 ceph-mon[97633]: pgmap v19848: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:35.129+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:35.131079+0000) 2022-09-10T14:49:36.039 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:35.757+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:35.758566+0000) 2022-09-10T14:49:36.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:49:36.037+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:49:36.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:36.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:36.131357+0000) 2022-09-10T14:49:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:36.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:36.758825+0000) 2022-09-10T14:49:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:37 smithi104 ceph-mon[97044]: pgmap v19849: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:37 smithi110 ceph-mon[97633]: pgmap v19849: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:37.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:37.131603+0000) 2022-09-10T14:49:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:37.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:37.759088+0000) 2022-09-10T14:49:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:38.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:38.131813+0000) 2022-09-10T14:49:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:38.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:38.759371+0000) 2022-09-10T14:49:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:39 smithi104 ceph-mon[97044]: pgmap v19850: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:39 smithi110 ceph-mon[97633]: pgmap v19850: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:39.130+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:39.132028+0000) 2022-09-10T14:49:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:39.758+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:39.759611+0000) 2022-09-10T14:49:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:40.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:40.132258+0000) 2022-09-10T14:49:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:40.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:40.759868+0000) 2022-09-10T14:49:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:41 smithi104 ceph-mon[97044]: pgmap v19851: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:41 smithi110 ceph-mon[97633]: pgmap v19851: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:41.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:41.132427+0000) 2022-09-10T14:49:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:41.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:41.760069+0000) 2022-09-10T14:49:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:42.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:42.132601+0000) 2022-09-10T14:49:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:42.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:42.760273+0000) 2022-09-10T14:49:43.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:43 smithi104 ceph-mon[97044]: pgmap v19852: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:43 smithi110 ceph-mon[97633]: pgmap v19852: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:43.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:43.132827+0000) 2022-09-10T14:49:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:43.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:43.760454+0000) 2022-09-10T14:49:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:44.131+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:44.133002+0000) 2022-09-10T14:49:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:44.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:44.760603+0000) 2022-09-10T14:49:45.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:45 smithi104 ceph-mon[97044]: pgmap v19853: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:45 smithi110 ceph-mon[97633]: pgmap v19853: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:45.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:45.133203+0000) 2022-09-10T14:49:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:45.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:45.760812+0000) 2022-09-10T14:49:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:46.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:46.133427+0000) 2022-09-10T14:49:47.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:46.759+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:46.760948+0000) 2022-09-10T14:49:47.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:47 smithi104 ceph-mon[97044]: pgmap v19854: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:49:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:49:47.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:49:47.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:49:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:47 smithi110 ceph-mon[97633]: pgmap v19854: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:49:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:49:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:49:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:49:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:47.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:47.133656+0000) 2022-09-10T14:49:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:47.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:47.761133+0000) 2022-09-10T14:49:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:48.132+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:48.133897+0000) 2022-09-10T14:49:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:48.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:48.761302+0000) 2022-09-10T14:49:49.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:49 smithi104 ceph-mon[97044]: pgmap v19855: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:49 smithi110 ceph-mon[97633]: pgmap v19855: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:49.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:49.134189+0000) 2022-09-10T14:49:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:49.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:49.761537+0000) 2022-09-10T14:49:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:50.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:50.134451+0000) 2022-09-10T14:49:51.053 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:50.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:50.761789+0000) 2022-09-10T14:49:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:51 smithi104 ceph-mon[97044]: pgmap v19856: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:51.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:51.134723+0000) 2022-09-10T14:49:51.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:49:51.051+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:49:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:51 smithi110 ceph-mon[97633]: pgmap v19856: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:51.760+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:51.761948+0000) 2022-09-10T14:49:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:52.133+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:52.134988+0000) 2022-09-10T14:49:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:52.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:52.762200+0000) 2022-09-10T14:49:53.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:53 smithi104 ceph-mon[97044]: pgmap v19857: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:53 smithi110 ceph-mon[97633]: pgmap v19857: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:53.134+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:53.135278+0000) 2022-09-10T14:49:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:53.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:53.762423+0000) 2022-09-10T14:49:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:54.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:54.135524+0000) 2022-09-10T14:49:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:54.761+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:54.762644+0000) 2022-09-10T14:49:55.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:55 smithi104 ceph-mon[97044]: pgmap v19858: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:55 smithi110 ceph-mon[97633]: pgmap v19858: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:55.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:55.135749+0000) 2022-09-10T14:49:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:55.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:55.762914+0000) 2022-09-10T14:49:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:56.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:56.135936+0000) 2022-09-10T14:49:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:56.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:56.763157+0000) 2022-09-10T14:49:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:57 smithi110 ceph-mon[97633]: pgmap v19859: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:57 smithi104 ceph-mon[97044]: pgmap v19859: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:49:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:57.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:57.136198+0000) 2022-09-10T14:49:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:57.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:57.763392+0000) 2022-09-10T14:49:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:58.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:58.136429+0000) 2022-09-10T14:49:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:58.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:58.763646+0000) 2022-09-10T14:49:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:49:59 smithi110 ceph-mon[97633]: pgmap v19860: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:49:59 smithi104 ceph-mon[97044]: pgmap v19860: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:49:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:49:59.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:59.136607+0000) 2022-09-10T14:50:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:49:59.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:49:59.763843+0000) 2022-09-10T14:50:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:50:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T14:50:00.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:00.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:00.136790+0000) 2022-09-10T14:50:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:00.762+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:00.764019+0000) 2022-09-10T14:50:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:01 smithi110 ceph-mon[97633]: pgmap v19861: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:01.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:01 smithi104 ceph-mon[97044]: pgmap v19861: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:01.135+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:01.137012+0000) 2022-09-10T14:50:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:01.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:01.764191+0000) 2022-09-10T14:50:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:02.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:02.137243+0000) 2022-09-10T14:50:03.060 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:02.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:02.764379+0000) 2022-09-10T14:50:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:03 smithi110 ceph-mon[97633]: pgmap v19862: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:03 smithi104 ceph-mon[97044]: pgmap v19862: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:03.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:03.137404+0000) 2022-09-10T14:50:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:03.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:03.764636+0000) 2022-09-10T14:50:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:04.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:04.137561+0000) 2022-09-10T14:50:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:04.763+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:04.764863+0000) 2022-09-10T14:50:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:05 smithi110 ceph-mon[97633]: pgmap v19863: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:05 smithi104 ceph-mon[97044]: pgmap v19863: 129 pgs: 129 active+clean; 9.9 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:05.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:05.137807+0000) 2022-09-10T14:50:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:05.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:05.765089+0000) 2022-09-10T14:50:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:06.136+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:06.137965+0000) 2022-09-10T14:50:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:50:06.053+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:50:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:06.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:06.765334+0000) 2022-09-10T14:50:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:07 smithi110 ceph-mon[97633]: pgmap v19864: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:50:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:07 smithi104 ceph-mon[97044]: pgmap v19864: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:50:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:07.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:07.138156+0000) 2022-09-10T14:50:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:07.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:07.765569+0000) 2022-09-10T14:50:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:08.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:08.138314+0000) 2022-09-10T14:50:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:08.764+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:08.765819+0000) 2022-09-10T14:50:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:09 smithi110 ceph-mon[97633]: pgmap v19865: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:50:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:09 smithi110 ceph-mon[97633]: osdmap e1322: 12 total, 12 up, 12 in 2022-09-10T14:50:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:09 smithi104 ceph-mon[97044]: pgmap v19865: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:50:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:09 smithi104 ceph-mon[97044]: osdmap e1322: 12 total, 12 up, 12 in 2022-09-10T14:50:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:09.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:09.138479+0000) 2022-09-10T14:50:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:09.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:09.766043+0000) 2022-09-10T14:50:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:10.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:10.138759+0000) 2022-09-10T14:50:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:10.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:10.766246+0000) 2022-09-10T14:50:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:11 smithi110 ceph-mon[97633]: pgmap v19867: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:50:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:11 smithi104 ceph-mon[97044]: pgmap v19867: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:50:11.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:11.137+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:11.138932+0000) 2022-09-10T14:50:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:11.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:11.766525+0000) 2022-09-10T14:50:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:12.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:12.139211+0000) 2022-09-10T14:50:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:12.765+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:12.766772+0000) 2022-09-10T14:50:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:13 smithi104 ceph-mon[97044]: pgmap v19868: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:50:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:13 smithi104 ceph-mon[97044]: osdmap e1323: 12 total, 12 up, 12 in 2022-09-10T14:50:13.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:13 smithi110 ceph-mon[97633]: pgmap v19868: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:50:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:13 smithi110 ceph-mon[97633]: osdmap e1323: 12 total, 12 up, 12 in 2022-09-10T14:50:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:13.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:13.139430+0000) 2022-09-10T14:50:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:13.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:13.766957+0000) 2022-09-10T14:50:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:14.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:14.139565+0000) 2022-09-10T14:50:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:14.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:14.767169+0000) 2022-09-10T14:50:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:15 smithi104 ceph-mon[97044]: pgmap v19870: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:50:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:15 smithi110 ceph-mon[97633]: pgmap v19870: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:50:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:15.138+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:15.139703+0000) 2022-09-10T14:50:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:15.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:15.767380+0000) 2022-09-10T14:50:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:16.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:16.139886+0000) 2022-09-10T14:50:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:16.766+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:16.767626+0000) 2022-09-10T14:50:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:17 smithi104 ceph-mon[97044]: pgmap v19871: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:50:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:17 smithi110 ceph-mon[97633]: pgmap v19871: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:50:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:17.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:17.140082+0000) 2022-09-10T14:50:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:17.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:17.767849+0000) 2022-09-10T14:50:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:18.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:18.140286+0000) 2022-09-10T14:50:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:18.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:18.768146+0000) 2022-09-10T14:50:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:19 smithi104 ceph-mon[97044]: pgmap v19872: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:50:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:19 smithi110 ceph-mon[97633]: pgmap v19872: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:50:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:19.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:19.140540+0000) 2022-09-10T14:50:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:19.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:19.768370+0000) 2022-09-10T14:50:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:20.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:20.140786+0000) 2022-09-10T14:50:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:20.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:20.768608+0000) 2022-09-10T14:50:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:21 smithi104 ceph-mon[97044]: pgmap v19873: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:50:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:21 smithi110 ceph-mon[97633]: pgmap v19873: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:50:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:21.139+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:21.141008+0000) 2022-09-10T14:50:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:50:21.068+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:50:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:21.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:21.768798+0000) 2022-09-10T14:50:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:22.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:22.141216+0000) 2022-09-10T14:50:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:22.767+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:22.769060+0000) 2022-09-10T14:50:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:23 smithi104 ceph-mon[97044]: pgmap v19874: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:50:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:23 smithi110 ceph-mon[97633]: pgmap v19874: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:50:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:23.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:23.141448+0000) 2022-09-10T14:50:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:23.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:23.769318+0000) 2022-09-10T14:50:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:24.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:24.141732+0000) 2022-09-10T14:50:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:24.768+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:24.769526+0000) 2022-09-10T14:50:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:25 smithi104 ceph-mon[97044]: pgmap v19875: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:50:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:25 smithi110 ceph-mon[97633]: pgmap v19875: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:50:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:25.140+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:25.141954+0000) 2022-09-10T14:50:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:25.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:25.769744+0000) 2022-09-10T14:50:26.563 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:26.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:26.142217+0000) 2022-09-10T14:50:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:26.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:26.769916+0000) 2022-09-10T14:50:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:27 smithi104 ceph-mon[97044]: pgmap v19876: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:50:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:27 smithi110 ceph-mon[97633]: pgmap v19876: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:50:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:27.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:27.142447+0000) 2022-09-10T14:50:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:27.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:27.770122+0000) 2022-09-10T14:50:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:28.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:28.142708+0000) 2022-09-10T14:50:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:28.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:28.770215+0000) 2022-09-10T14:50:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:29 smithi104 ceph-mon[97044]: pgmap v19877: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:29 smithi110 ceph-mon[97633]: pgmap v19877: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:29.141+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:29.142842+0000) 2022-09-10T14:50:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:29.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:29.770439+0000) 2022-09-10T14:50:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:30.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:30.143080+0000) 2022-09-10T14:50:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:30.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:30.770640+0000) 2022-09-10T14:50:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:31 smithi104 ceph-mon[97044]: pgmap v19878: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:31 smithi110 ceph-mon[97633]: pgmap v19878: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:31.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:31.143387+0000) 2022-09-10T14:50:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:31.769+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:31.770884+0000) 2022-09-10T14:50:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:32.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:32.143588+0000) 2022-09-10T14:50:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:32.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:32.771127+0000) 2022-09-10T14:50:33.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:33 smithi104 ceph-mon[97044]: pgmap v19879: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:33 smithi110 ceph-mon[97633]: pgmap v19879: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:33.142+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:33.143819+0000) 2022-09-10T14:50:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:33.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:33.771311+0000) 2022-09-10T14:50:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:34.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:34.144051+0000) 2022-09-10T14:50:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:34.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:34.771482+0000) 2022-09-10T14:50:35.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:35 smithi104 ceph-mon[97044]: pgmap v19880: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:35 smithi110 ceph-mon[97633]: pgmap v19880: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:35.143+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:35.144277+0000) 2022-09-10T14:50:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:35.770+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:35.771718+0000) 2022-09-10T14:50:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:36.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:36.144536+0000) 2022-09-10T14:50:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:50:36.080+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:50:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:36.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:36.772006+0000) 2022-09-10T14:50:37.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:37 smithi104 ceph-mon[97044]: pgmap v19881: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:37 smithi110 ceph-mon[97633]: pgmap v19881: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:37.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:37.144763+0000) 2022-09-10T14:50:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:37.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:37.772282+0000) 2022-09-10T14:50:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:38.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:38.144994+0000) 2022-09-10T14:50:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:38.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:38.772508+0000) 2022-09-10T14:50:39.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:39 smithi104 ceph-mon[97044]: pgmap v19882: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:39 smithi110 ceph-mon[97633]: pgmap v19882: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:39.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:39.145196+0000) 2022-09-10T14:50:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:39.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:39.772703+0000) 2022-09-10T14:50:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:40.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:40.145372+0000) 2022-09-10T14:50:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:40.771+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:40.772925+0000) 2022-09-10T14:50:41.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:41 smithi104 ceph-mon[97044]: pgmap v19883: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:41 smithi110 ceph-mon[97633]: pgmap v19883: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:41.144+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:41.145556+0000) 2022-09-10T14:50:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:41.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:41.773183+0000) 2022-09-10T14:50:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:42.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:42.145732+0000) 2022-09-10T14:50:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:42.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:42.773373+0000) 2022-09-10T14:50:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:43 smithi110 ceph-mon[97633]: pgmap v19884: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:43 smithi104 ceph-mon[97044]: pgmap v19884: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:43.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:43.145967+0000) 2022-09-10T14:50:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:43.772+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:43.773547+0000) 2022-09-10T14:50:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:44.145+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:44.146203+0000) 2022-09-10T14:50:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:44.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:44.773806+0000) 2022-09-10T14:50:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:45 smithi110 ceph-mon[97633]: pgmap v19885: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:45 smithi104 ceph-mon[97044]: pgmap v19885: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:45.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:45.146457+0000) 2022-09-10T14:50:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:45.773+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:45.774057+0000) 2022-09-10T14:50:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:46.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:46.146739+0000) 2022-09-10T14:50:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:46.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:46.774279+0000) 2022-09-10T14:50:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:47 smithi110 ceph-mon[97633]: pgmap v19886: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:50:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:50:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:50:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:50:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:47 smithi104 ceph-mon[97044]: pgmap v19886: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:50:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:50:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:50:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:50:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:47.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:47.146966+0000) 2022-09-10T14:50:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:47.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:47.774481+0000) 2022-09-10T14:50:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:48.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:48.147211+0000) 2022-09-10T14:50:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:48.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:48.774669+0000) 2022-09-10T14:50:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:49 smithi110 ceph-mon[97633]: pgmap v19887: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:49 smithi104 ceph-mon[97044]: pgmap v19887: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:49.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:49.147374+0000) 2022-09-10T14:50:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:49.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:49.774892+0000) 2022-09-10T14:50:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:50.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:50.147528+0000) 2022-09-10T14:50:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:50.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:50.775187+0000) 2022-09-10T14:50:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:51 smithi110 ceph-mon[97633]: pgmap v19888: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:51 smithi104 ceph-mon[97044]: pgmap v19888: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:51.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:51.147711+0000) 2022-09-10T14:50:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:50:51.093+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:50:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:51.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:51.775419+0000) 2022-09-10T14:50:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:52.146+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:52.147842+0000) 2022-09-10T14:50:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:52.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:52.775619+0000) 2022-09-10T14:50:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:53 smithi110 ceph-mon[97633]: pgmap v19889: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:53 smithi104 ceph-mon[97044]: pgmap v19889: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:53.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:53.148042+0000) 2022-09-10T14:50:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:53.774+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:53.775939+0000) 2022-09-10T14:50:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:54.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:54.148239+0000) 2022-09-10T14:50:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:54.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:54.776184+0000) 2022-09-10T14:50:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:55 smithi110 ceph-mon[97633]: pgmap v19890: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:55 smithi104 ceph-mon[97044]: pgmap v19890: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:50:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:55.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:55.148447+0000) 2022-09-10T14:50:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:55.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:55.776377+0000) 2022-09-10T14:50:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:56.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:56.148693+0000) 2022-09-10T14:50:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:56.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:56.776633+0000) 2022-09-10T14:50:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:57 smithi110 ceph-mon[97633]: pgmap v19891: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:57 smithi104 ceph-mon[97044]: pgmap v19891: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:50:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:57.147+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:57.148943+0000) 2022-09-10T14:50:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:57.775+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:57.776852+0000) 2022-09-10T14:50:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:58.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:58.149195+0000) 2022-09-10T14:50:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:58.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:58.777167+0000) 2022-09-10T14:50:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:50:59 smithi110 ceph-mon[97633]: pgmap v19892: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:50:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:50:59 smithi104 ceph-mon[97044]: pgmap v19892: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:50:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:50:59.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:59.149449+0000) 2022-09-10T14:51:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:50:59.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:50:59.777346+0000) 2022-09-10T14:51:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:00.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:00.149653+0000) 2022-09-10T14:51:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:00.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:00.777517+0000) 2022-09-10T14:51:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:01 smithi110 ceph-mon[97633]: pgmap v19893: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:01 smithi104 ceph-mon[97044]: pgmap v19893: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:01.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:01.149905+0000) 2022-09-10T14:51:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:01.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:01.777732+0000) 2022-09-10T14:51:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:02.148+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:02.150019+0000) 2022-09-10T14:51:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:02.776+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:02.778009+0000) 2022-09-10T14:51:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:03 smithi110 ceph-mon[97633]: pgmap v19894: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:51:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:03 smithi104 ceph-mon[97044]: pgmap v19894: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:51:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:03.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:03.150246+0000) 2022-09-10T14:51:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:03.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:03.778206+0000) 2022-09-10T14:51:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:04.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:04.150490+0000) 2022-09-10T14:51:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:04.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:04.778377+0000) 2022-09-10T14:51:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:05 smithi110 ceph-mon[97633]: pgmap v19895: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:51:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:05 smithi104 ceph-mon[97044]: pgmap v19895: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:51:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:05.149+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:05.150680+0000) 2022-09-10T14:51:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:05.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:05.778525+0000) 2022-09-10T14:51:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:06.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:06.150865+0000) 2022-09-10T14:51:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:51:06.094+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:51:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:06.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:06.778709+0000) 2022-09-10T14:51:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:07 smithi110 ceph-mon[97633]: pgmap v19896: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:51:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:07 smithi104 ceph-mon[97044]: pgmap v19896: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:51:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:07.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:07.151056+0000) 2022-09-10T14:51:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:07.777+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:07.778931+0000) 2022-09-10T14:51:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:08.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:08.151256+0000) 2022-09-10T14:51:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:08.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:08.779195+0000) 2022-09-10T14:51:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:09 smithi110 ceph-mon[97633]: pgmap v19897: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:51:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:09 smithi110 ceph-mon[97633]: osdmap e1324: 12 total, 12 up, 12 in 2022-09-10T14:51:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:09 smithi104 ceph-mon[97044]: pgmap v19897: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:51:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:09 smithi104 ceph-mon[97044]: osdmap e1324: 12 total, 12 up, 12 in 2022-09-10T14:51:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:09.150+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:09.151471+0000) 2022-09-10T14:51:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:09.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:09.779348+0000) 2022-09-10T14:51:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:10.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:10.151724+0000) 2022-09-10T14:51:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:10.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:10.779566+0000) 2022-09-10T14:51:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:11 smithi110 ceph-mon[97633]: pgmap v19899: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:51:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:11 smithi104 ceph-mon[97044]: pgmap v19899: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:51:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:11.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:11.151967+0000) 2022-09-10T14:51:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:11.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:11.779785+0000) 2022-09-10T14:51:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:12.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:12.152180+0000) 2022-09-10T14:51:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:12.778+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:12.780048+0000) 2022-09-10T14:51:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:13 smithi104 ceph-mon[97044]: pgmap v19900: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:51:13.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:13 smithi104 ceph-mon[97044]: osdmap e1325: 12 total, 12 up, 12 in 2022-09-10T14:51:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:13 smithi110 ceph-mon[97633]: pgmap v19900: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:51:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:13 smithi110 ceph-mon[97633]: osdmap e1325: 12 total, 12 up, 12 in 2022-09-10T14:51:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:13.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:13.152404+0000) 2022-09-10T14:51:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:13.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:13.780243+0000) 2022-09-10T14:51:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:14.151+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:14.152596+0000) 2022-09-10T14:51:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:14.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:14.780399+0000) 2022-09-10T14:51:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:15 smithi104 ceph-mon[97044]: pgmap v19902: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:51:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:15 smithi110 ceph-mon[97633]: pgmap v19902: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:51:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:15.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:15.152838+0000) 2022-09-10T14:51:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:15.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:15.780560+0000) 2022-09-10T14:51:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:16.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:16.153083+0000) 2022-09-10T14:51:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:16.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:16.780791+0000) 2022-09-10T14:51:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:17 smithi104 ceph-mon[97044]: pgmap v19903: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:51:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:17 smithi110 ceph-mon[97633]: pgmap v19903: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:51:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:17.152+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:17.153352+0000) 2022-09-10T14:51:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:17.779+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:17.780961+0000) 2022-09-10T14:51:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:18.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:18.153586+0000) 2022-09-10T14:51:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:18.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:18.781199+0000) 2022-09-10T14:51:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:19 smithi104 ceph-mon[97044]: pgmap v19904: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:51:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:19 smithi110 ceph-mon[97633]: pgmap v19904: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:51:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:19.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:19.153836+0000) 2022-09-10T14:51:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:19.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:19.781393+0000) 2022-09-10T14:51:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:20.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:20.154070+0000) 2022-09-10T14:51:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:20.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:20.781651+0000) 2022-09-10T14:51:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:21 smithi104 ceph-mon[97044]: pgmap v19905: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:51:21.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:21 smithi110 ceph-mon[97633]: pgmap v19905: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:51:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:51:21.110+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:51:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:21.153+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:21.154364+0000) 2022-09-10T14:51:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:21.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:21.781844+0000) 2022-09-10T14:51:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:22.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:22.154591+0000) 2022-09-10T14:51:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:22.780+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:22.782039+0000) 2022-09-10T14:51:23.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:23 smithi104 ceph-mon[97044]: pgmap v19906: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:51:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:23 smithi110 ceph-mon[97633]: pgmap v19906: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:51:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:23.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:23.154813+0000) 2022-09-10T14:51:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:23.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:23.782203+0000) 2022-09-10T14:51:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:24.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:24.155023+0000) 2022-09-10T14:51:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:24.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:24.782335+0000) 2022-09-10T14:51:25.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:25 smithi104 ceph-mon[97044]: pgmap v19907: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:51:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:25 smithi110 ceph-mon[97633]: pgmap v19907: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:51:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:25.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:25.155248+0000) 2022-09-10T14:51:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:25.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:25.782483+0000) 2022-09-10T14:51:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:26.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:26.155440+0000) 2022-09-10T14:51:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:26.781+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:26.782679+0000) 2022-09-10T14:51:27.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:27 smithi104 ceph-mon[97044]: pgmap v19908: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:51:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:27 smithi110 ceph-mon[97633]: pgmap v19908: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:51:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:27.154+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:27.155663+0000) 2022-09-10T14:51:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:27.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:27.782887+0000) 2022-09-10T14:51:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:28.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:28.155877+0000) 2022-09-10T14:51:29.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:28.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:28.783159+0000) 2022-09-10T14:51:29.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:29 smithi104 ceph-mon[97044]: pgmap v19909: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:29 smithi110 ceph-mon[97633]: pgmap v19909: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:29.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:29.156132+0000) 2022-09-10T14:51:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:29.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:29.783317+0000) 2022-09-10T14:51:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:30.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:30.156354+0000) 2022-09-10T14:51:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:30.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:30.783585+0000) 2022-09-10T14:51:31.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:31 smithi104 ceph-mon[97044]: pgmap v19910: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:31 smithi110 ceph-mon[97633]: pgmap v19910: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:31.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:31.156597+0000) 2022-09-10T14:51:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:31.782+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:31.783829+0000) 2022-09-10T14:51:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:32.155+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:32.156792+0000) 2022-09-10T14:51:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:32.783+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:32.783989+0000) 2022-09-10T14:51:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:33 smithi110 ceph-mon[97633]: pgmap v19911: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:33 smithi104 ceph-mon[97044]: pgmap v19911: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:33.156+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:33.157045+0000) 2022-09-10T14:51:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:33.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:33.784193+0000) 2022-09-10T14:51:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:34.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:34.157283+0000) 2022-09-10T14:51:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:34.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:34.784423+0000) 2022-09-10T14:51:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:35 smithi110 ceph-mon[97633]: pgmap v19912: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:35 smithi104 ceph-mon[97044]: pgmap v19912: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:35.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:35.157511+0000) 2022-09-10T14:51:36.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:35.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:35.784642+0000) 2022-09-10T14:51:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:36.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:36.157746+0000) 2022-09-10T14:51:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:51:36.119+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:51:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:36.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:36.784888+0000) 2022-09-10T14:51:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:37 smithi110 ceph-mon[97633]: pgmap v19913: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:37 smithi104 ceph-mon[97044]: pgmap v19913: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:37.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:37.157928+0000) 2022-09-10T14:51:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:37.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:37.785177+0000) 2022-09-10T14:51:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:38.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:38.158195+0000) 2022-09-10T14:51:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:38.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:38.785439+0000) 2022-09-10T14:51:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:39 smithi110 ceph-mon[97633]: pgmap v19914: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:39 smithi104 ceph-mon[97044]: pgmap v19914: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:39.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:39.158437+0000) 2022-09-10T14:51:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:39.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:39.785624+0000) 2022-09-10T14:51:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:40.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:40.158673+0000) 2022-09-10T14:51:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:40.784+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:40.785866+0000) 2022-09-10T14:51:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:41 smithi110 ceph-mon[97633]: pgmap v19915: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:41 smithi104 ceph-mon[97044]: pgmap v19915: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:41.157+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:41.158880+0000) 2022-09-10T14:51:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:41.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:41.786149+0000) 2022-09-10T14:51:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:42.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:42.159144+0000) 2022-09-10T14:51:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:42.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:42.786319+0000) 2022-09-10T14:51:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:43 smithi110 ceph-mon[97633]: pgmap v19916: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:43 smithi104 ceph-mon[97044]: pgmap v19916: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:43.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:43.159336+0000) 2022-09-10T14:51:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:43.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:43.786527+0000) 2022-09-10T14:51:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:44.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:44.159467+0000) 2022-09-10T14:51:45.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:44.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:44.786808+0000) 2022-09-10T14:51:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:45 smithi110 ceph-mon[97633]: pgmap v19917: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:45 smithi104 ceph-mon[97044]: pgmap v19917: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:45.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:45.159709+0000) 2022-09-10T14:51:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:45.785+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:45.787011+0000) 2022-09-10T14:51:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:46.158+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:46.159928+0000) 2022-09-10T14:51:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:46.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:46.787274+0000) 2022-09-10T14:51:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:47 smithi110 ceph-mon[97633]: pgmap v19918: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:51:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:51:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:51:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:51:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:47 smithi104 ceph-mon[97044]: pgmap v19918: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:51:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:51:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:51:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:51:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:47.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:47.160178+0000) 2022-09-10T14:51:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:47.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:47.787477+0000) 2022-09-10T14:51:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:48.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:48.160343+0000) 2022-09-10T14:51:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:48.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:48.787682+0000) 2022-09-10T14:51:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:49 smithi110 ceph-mon[97633]: pgmap v19919: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:49 smithi104 ceph-mon[97044]: pgmap v19919: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:49.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:49.160579+0000) 2022-09-10T14:51:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:49.786+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:49.787863+0000) 2022-09-10T14:51:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:50.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:50.160807+0000) 2022-09-10T14:51:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:50.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:50.788074+0000) 2022-09-10T14:51:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:51 smithi110 ceph-mon[97633]: pgmap v19920: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:51 smithi104 ceph-mon[97044]: pgmap v19920: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:51.159+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:51.161059+0000) 2022-09-10T14:51:51.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:51:51.123+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:51:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:51.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:51.788218+0000) 2022-09-10T14:51:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:52.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:52.161253+0000) 2022-09-10T14:51:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:52.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:52.788395+0000) 2022-09-10T14:51:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:53 smithi110 ceph-mon[97633]: pgmap v19921: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:53 smithi104 ceph-mon[97044]: pgmap v19921: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:53.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:53.161431+0000) 2022-09-10T14:51:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:53.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:53.788707+0000) 2022-09-10T14:51:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:54.160+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:54.161713+0000) 2022-09-10T14:51:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:54.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:54.788865+0000) 2022-09-10T14:51:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:55 smithi110 ceph-mon[97633]: pgmap v19922: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:55 smithi104 ceph-mon[97044]: pgmap v19922: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:55.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:55.161889+0000) 2022-09-10T14:51:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:55.787+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:55.789054+0000) 2022-09-10T14:51:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:56.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:56.162144+0000) 2022-09-10T14:51:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:56.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:56.789346+0000) 2022-09-10T14:51:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:57 smithi110 ceph-mon[97633]: pgmap v19923: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:57 smithi104 ceph-mon[97044]: pgmap v19923: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:51:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:57.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:57.162321+0000) 2022-09-10T14:51:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:57.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:57.789597+0000) 2022-09-10T14:51:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:58.161+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:58.162591+0000) 2022-09-10T14:51:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:58.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:58.789829+0000) 2022-09-10T14:51:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:51:59 smithi110 ceph-mon[97633]: pgmap v19924: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:51:59 smithi104 ceph-mon[97044]: pgmap v19924: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:51:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:51:59.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:59.162842+0000) 2022-09-10T14:52:00.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:51:59.788+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:51:59.790006+0000) 2022-09-10T14:52:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:00.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:00.163058+0000) 2022-09-10T14:52:01.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:00.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:00.790226+0000) 2022-09-10T14:52:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:01 smithi110 ceph-mon[97633]: pgmap v19925: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:01 smithi104 ceph-mon[97044]: pgmap v19925: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:01.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:01.163355+0000) 2022-09-10T14:52:02.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:01.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:01.790387+0000) 2022-09-10T14:52:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:02.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:02.163553+0000) 2022-09-10T14:52:03.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:02.789+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:02.790640+0000) 2022-09-10T14:52:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:03 smithi110 ceph-mon[97633]: pgmap v19926: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:03 smithi104 ceph-mon[97044]: pgmap v19926: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:03.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:03.163786+0000) 2022-09-10T14:52:04.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:03.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:03.790893+0000) 2022-09-10T14:52:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:04.162+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:04.163996+0000) 2022-09-10T14:52:05.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:04.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:04.791144+0000) 2022-09-10T14:52:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:05 smithi110 ceph-mon[97633]: pgmap v19927: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:05 smithi104 ceph-mon[97044]: pgmap v19927: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:05.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:05.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:05.164231+0000) 2022-09-10T14:52:06.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:05.790+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:05.791359+0000) 2022-09-10T14:52:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:06.163+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:06.164453+0000) 2022-09-10T14:52:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:52:06.131+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:52:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:06.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:06.791659+0000) 2022-09-10T14:52:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:07 smithi110 ceph-mon[97633]: pgmap v19928: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:52:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:07 smithi104 ceph-mon[97044]: pgmap v19928: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:52:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:07.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:07.164708+0000) 2022-09-10T14:52:08.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:07.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:07.791919+0000) 2022-09-10T14:52:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:08.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:08.164958+0000) 2022-09-10T14:52:09.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:08.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:08.792202+0000) 2022-09-10T14:52:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:09 smithi110 ceph-mon[97633]: pgmap v19929: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:52:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:09 smithi110 ceph-mon[97633]: osdmap e1326: 12 total, 12 up, 12 in 2022-09-10T14:52:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:09 smithi104 ceph-mon[97044]: pgmap v19929: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:52:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:09 smithi104 ceph-mon[97044]: osdmap e1326: 12 total, 12 up, 12 in 2022-09-10T14:52:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:09.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:09.165229+0000) 2022-09-10T14:52:10.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:09.791+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:09.792369+0000) 2022-09-10T14:52:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:10.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:10.165496+0000) 2022-09-10T14:52:11.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:10.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:10.792620+0000) 2022-09-10T14:52:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:11 smithi110 ceph-mon[97633]: pgmap v19931: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:52:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:11 smithi104 ceph-mon[97044]: pgmap v19931: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:52:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:11.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:11.165704+0000) 2022-09-10T14:52:12.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:11.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:11.792828+0000) 2022-09-10T14:52:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:12.164+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:12.165913+0000) 2022-09-10T14:52:13.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:12.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:12.793007+0000) 2022-09-10T14:52:13.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:13 smithi104 ceph-mon[97044]: pgmap v19932: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:52:13.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:13 smithi104 ceph-mon[97044]: osdmap e1327: 12 total, 12 up, 12 in 2022-09-10T14:52:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:13 smithi110 ceph-mon[97633]: pgmap v19932: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:52:13.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:13 smithi110 ceph-mon[97633]: osdmap e1327: 12 total, 12 up, 12 in 2022-09-10T14:52:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:13.165+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:13.166188+0000) 2022-09-10T14:52:14.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:13.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:13.793230+0000) 2022-09-10T14:52:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:14.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:14.166414+0000) 2022-09-10T14:52:15.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:14.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:14.793429+0000) 2022-09-10T14:52:15.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:15 smithi104 ceph-mon[97044]: pgmap v19934: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:52:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:15 smithi110 ceph-mon[97633]: pgmap v19934: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:52:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:15.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:15.166658+0000) 2022-09-10T14:52:16.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:15.792+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:15.793596+0000) 2022-09-10T14:52:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:16.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:16.166898+0000) 2022-09-10T14:52:17.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:16.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:16.793803+0000) 2022-09-10T14:52:17.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:17 smithi104 ceph-mon[97044]: pgmap v19935: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:52:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:17 smithi110 ceph-mon[97633]: pgmap v19935: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:52:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:17.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:17.167173+0000) 2022-09-10T14:52:18.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:17.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:17.794045+0000) 2022-09-10T14:52:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:18.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:18.167409+0000) 2022-09-10T14:52:19.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:18.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:18.794321+0000) 2022-09-10T14:52:19.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:19 smithi104 ceph-mon[97044]: pgmap v19936: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:52:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:19 smithi110 ceph-mon[97633]: pgmap v19936: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 212 B/s wr, 10 op/s 2022-09-10T14:52:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:19.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:19.167608+0000) 2022-09-10T14:52:20.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:19.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:19.794565+0000) 2022-09-10T14:52:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:20.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:20.167817+0000) 2022-09-10T14:52:21.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:20.794+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:20.794762+0000) 2022-09-10T14:52:21.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:21 smithi104 ceph-mon[97044]: pgmap v19937: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:52:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:21 smithi110 ceph-mon[97633]: pgmap v19937: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:52:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:21.166+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:21.168040+0000) 2022-09-10T14:52:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:52:21.144+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:52:22.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:21.793+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:21.794985+0000) 2022-09-10T14:52:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:22.167+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:22.168152+0000) 2022-09-10T14:52:23.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:22.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:22.795261+0000) 2022-09-10T14:52:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:23 smithi110 ceph-mon[97633]: pgmap v19938: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:52:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:23 smithi104 ceph-mon[97044]: pgmap v19938: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:52:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:23.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:23.168311+0000) 2022-09-10T14:52:24.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:23.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:23.795490+0000) 2022-09-10T14:52:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:24.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:24.168502+0000) 2022-09-10T14:52:25.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:24.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:24.795675+0000) 2022-09-10T14:52:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:25 smithi110 ceph-mon[97633]: pgmap v19939: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:52:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:25 smithi104 ceph-mon[97044]: pgmap v19939: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:52:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:25.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:25.168709+0000) 2022-09-10T14:52:26.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:25.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:25.795937+0000) 2022-09-10T14:52:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:26.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:26.168897+0000) 2022-09-10T14:52:27.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:26.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:26.796225+0000) 2022-09-10T14:52:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:27 smithi110 ceph-mon[97633]: pgmap v19940: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:52:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:27 smithi104 ceph-mon[97044]: pgmap v19940: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:52:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:27.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:27.169174+0000) 2022-09-10T14:52:28.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:27.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:27.796362+0000) 2022-09-10T14:52:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:28.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:28.169355+0000) 2022-09-10T14:52:29.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:28.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:28.796594+0000) 2022-09-10T14:52:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:29 smithi110 ceph-mon[97633]: pgmap v19941: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:29 smithi104 ceph-mon[97044]: pgmap v19941: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:29.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:29.169590+0000) 2022-09-10T14:52:30.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:29.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:29.796839+0000) 2022-09-10T14:52:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:30.168+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:30.169841+0000) 2022-09-10T14:52:31.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:30.795+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:30.797071+0000) 2022-09-10T14:52:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:31 smithi110 ceph-mon[97633]: pgmap v19942: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:31.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:31 smithi104 ceph-mon[97044]: pgmap v19942: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:31.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:31.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:31.170135+0000) 2022-09-10T14:52:32.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:31.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:31.797264+0000) 2022-09-10T14:52:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:32.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:32.170397+0000) 2022-09-10T14:52:33.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:32.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:32.797471+0000) 2022-09-10T14:52:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:33 smithi110 ceph-mon[97633]: pgmap v19943: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:33 smithi104 ceph-mon[97044]: pgmap v19943: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:33.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:33.170656+0000) 2022-09-10T14:52:34.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:33.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:33.797624+0000) 2022-09-10T14:52:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:34.169+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:34.170902+0000) 2022-09-10T14:52:35.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:34.796+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:34.797836+0000) 2022-09-10T14:52:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:35 smithi110 ceph-mon[97633]: pgmap v19944: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:35 smithi104 ceph-mon[97044]: pgmap v19944: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:35.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:35.171133+0000) 2022-09-10T14:52:36.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:35.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:35.798036+0000) 2022-09-10T14:52:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:36.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:36.171327+0000) 2022-09-10T14:52:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:52:36.151+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:52:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:36.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:36.798312+0000) 2022-09-10T14:52:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:37 smithi110 ceph-mon[97633]: pgmap v19945: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:37 smithi104 ceph-mon[97044]: pgmap v19945: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:37.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:37.171557+0000) 2022-09-10T14:52:38.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:37.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:37.798509+0000) 2022-09-10T14:52:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:38.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:38.171831+0000) 2022-09-10T14:52:39.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:38.797+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:38.798787+0000) 2022-09-10T14:52:39.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:39 smithi110 ceph-mon[97633]: pgmap v19946: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:39 smithi104 ceph-mon[97044]: pgmap v19946: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:39.170+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:39.172064+0000) 2022-09-10T14:52:40.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:39.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:39.798988+0000) 2022-09-10T14:52:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:40.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:40.172287+0000) 2022-09-10T14:52:41.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:40.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:40.799206+0000) 2022-09-10T14:52:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:41 smithi110 ceph-mon[97633]: pgmap v19947: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:41 smithi104 ceph-mon[97044]: pgmap v19947: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:41.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:41.172454+0000) 2022-09-10T14:52:42.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:41.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:41.799414+0000) 2022-09-10T14:52:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:42.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:42.172641+0000) 2022-09-10T14:52:43.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:42.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:42.799651+0000) 2022-09-10T14:52:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:43 smithi110 ceph-mon[97633]: pgmap v19948: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:43 smithi104 ceph-mon[97044]: pgmap v19948: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:43.171+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:43.172864+0000) 2022-09-10T14:52:44.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:43.798+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:43.799885+0000) 2022-09-10T14:52:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:44.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:44.173103+0000) 2022-09-10T14:52:45.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:44.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:44.800139+0000) 2022-09-10T14:52:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:45 smithi110 ceph-mon[97633]: pgmap v19949: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:45 smithi104 ceph-mon[97044]: pgmap v19949: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:45.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:45.173335+0000) 2022-09-10T14:52:46.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:45.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:45.800365+0000) 2022-09-10T14:52:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:46.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:46.173497+0000) 2022-09-10T14:52:47.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:46.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:46.800587+0000) 2022-09-10T14:52:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:47 smithi110 ceph-mon[97633]: pgmap v19950: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:52:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:52:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:52:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:52:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:47 smithi104 ceph-mon[97044]: pgmap v19950: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:52:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:52:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:52:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:52:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:47.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:47.173675+0000) 2022-09-10T14:52:48.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:47.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:47.800809+0000) 2022-09-10T14:52:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:48.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:48.173873+0000) 2022-09-10T14:52:49.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:48.799+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:48.800947+0000) 2022-09-10T14:52:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:49 smithi110 ceph-mon[97633]: pgmap v19951: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:49 smithi104 ceph-mon[97044]: pgmap v19951: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:49.172+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:49.174015+0000) 2022-09-10T14:52:50.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:49.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:49.801165+0000) 2022-09-10T14:52:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:50.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:50.174220+0000) 2022-09-10T14:52:51.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:50.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:50.801368+0000) 2022-09-10T14:52:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:51 smithi110 ceph-mon[97633]: pgmap v19952: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:51 smithi104 ceph-mon[97044]: pgmap v19952: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:51.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:51.174402+0000) 2022-09-10T14:52:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:52:51.157+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:52:52.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:51.800+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:51.801587+0000) 2022-09-10T14:52:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:52.173+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:52.174617+0000) 2022-09-10T14:52:53.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:52.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:52.801841+0000) 2022-09-10T14:52:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:53 smithi110 ceph-mon[97633]: pgmap v19953: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:53 smithi104 ceph-mon[97044]: pgmap v19953: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:53.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:53.174850+0000) 2022-09-10T14:52:54.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:53.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:53.802119+0000) 2022-09-10T14:52:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:54.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:54.175046+0000) 2022-09-10T14:52:55.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:54.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:54.802343+0000) 2022-09-10T14:52:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:55 smithi110 ceph-mon[97633]: pgmap v19954: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:55 smithi104 ceph-mon[97044]: pgmap v19954: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:55.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:55.175209+0000) 2022-09-10T14:52:56.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:55.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:55.802589+0000) 2022-09-10T14:52:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:56.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:56.175418+0000) 2022-09-10T14:52:57.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:56.801+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:56.802862+0000) 2022-09-10T14:52:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:57 smithi110 ceph-mon[97633]: pgmap v19955: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:57 smithi104 ceph-mon[97044]: pgmap v19955: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:52:57.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:57.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:57.175670+0000) 2022-09-10T14:52:58.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:57.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:57.803024+0000) 2022-09-10T14:52:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:58.174+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:58.175915+0000) 2022-09-10T14:52:59.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:58.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:58.803254+0000) 2022-09-10T14:52:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:52:59 smithi110 ceph-mon[97633]: pgmap v19956: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:52:59 smithi104 ceph-mon[97044]: pgmap v19956: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:52:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:52:59.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:59.176128+0000) 2022-09-10T14:53:00.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:52:59.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:52:59.803528+0000) 2022-09-10T14:53:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:00.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:00.176308+0000) 2022-09-10T14:53:01.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:00.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:00.803671+0000) 2022-09-10T14:53:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:01 smithi110 ceph-mon[97633]: pgmap v19957: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:01 smithi104 ceph-mon[97044]: pgmap v19957: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:01.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:01.176535+0000) 2022-09-10T14:53:02.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:01.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:01.803862+0000) 2022-09-10T14:53:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:02.175+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:02.176711+0000) 2022-09-10T14:53:03.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:02.802+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:02.804034+0000) 2022-09-10T14:53:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:03 smithi110 ceph-mon[97633]: pgmap v19958: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:03 smithi104 ceph-mon[97044]: pgmap v19958: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:03.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:03.176888+0000) 2022-09-10T14:53:04.178 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:03.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:03.804263+0000) 2022-09-10T14:53:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:04.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:04.177134+0000) 2022-09-10T14:53:05.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:04.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:04.804472+0000) 2022-09-10T14:53:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:05 smithi110 ceph-mon[97633]: pgmap v19959: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:05 smithi104 ceph-mon[97044]: pgmap v19959: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:05.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:05.177373+0000) 2022-09-10T14:53:06.174 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:05.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:05.804697+0000) 2022-09-10T14:53:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:06.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:06.177539+0000) 2022-09-10T14:53:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:53:06.171+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:53:07.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:06.803+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:06.804916+0000) 2022-09-10T14:53:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:07 smithi110 ceph-mon[97633]: pgmap v19960: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:53:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:07 smithi104 ceph-mon[97044]: pgmap v19960: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:53:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:07.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:07.177718+0000) 2022-09-10T14:53:08.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:07.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:07.805066+0000) 2022-09-10T14:53:08.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:08.176+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:08.177911+0000) 2022-09-10T14:53:09.179 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:08.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:08.805263+0000) 2022-09-10T14:53:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:09 smithi110 ceph-mon[97633]: pgmap v19961: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:53:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:09 smithi110 ceph-mon[97633]: osdmap e1328: 12 total, 12 up, 12 in 2022-09-10T14:53:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:09 smithi104 ceph-mon[97044]: pgmap v19961: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:53:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:09 smithi104 ceph-mon[97044]: osdmap e1328: 12 total, 12 up, 12 in 2022-09-10T14:53:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:09.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:09.178166+0000) 2022-09-10T14:53:10.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:09.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:09.805397+0000) 2022-09-10T14:53:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:10.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:10.178388+0000) 2022-09-10T14:53:11.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:10.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:10.805594+0000) 2022-09-10T14:53:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:11 smithi110 ceph-mon[97633]: pgmap v19963: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:53:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:11 smithi104 ceph-mon[97044]: pgmap v19963: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:53:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:11.177+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:11.178613+0000) 2022-09-10T14:53:12.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:11.804+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:11.805818+0000) 2022-09-10T14:53:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:12.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:12.178811+0000) 2022-09-10T14:53:13.180 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:12.805+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:12.806009+0000) 2022-09-10T14:53:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:13 smithi110 ceph-mon[97633]: pgmap v19964: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:53:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:13 smithi110 ceph-mon[97633]: osdmap e1329: 12 total, 12 up, 12 in 2022-09-10T14:53:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:13 smithi104 ceph-mon[97044]: pgmap v19964: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:53:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:13 smithi104 ceph-mon[97044]: osdmap e1329: 12 total, 12 up, 12 in 2022-09-10T14:53:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:13.178+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:13.179080+0000) 2022-09-10T14:53:14.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:13.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:13.806157+0000) 2022-09-10T14:53:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:14.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:14.179373+0000) 2022-09-10T14:53:15.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:14.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:14.806347+0000) 2022-09-10T14:53:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:15 smithi110 ceph-mon[97633]: pgmap v19966: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:53:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:15 smithi104 ceph-mon[97044]: pgmap v19966: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:53:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:15.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:15.179643+0000) 2022-09-10T14:53:16.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:15.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:15.806565+0000) 2022-09-10T14:53:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:16.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:16.179850+0000) 2022-09-10T14:53:17.181 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:16.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:16.806772+0000) 2022-09-10T14:53:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:17 smithi110 ceph-mon[97633]: pgmap v19967: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:53:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:17 smithi104 ceph-mon[97044]: pgmap v19967: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:53:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:17.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:17.180130+0000) 2022-09-10T14:53:18.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:17.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:17.806987+0000) 2022-09-10T14:53:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:18.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:18.180376+0000) 2022-09-10T14:53:19.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:18.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:18.807233+0000) 2022-09-10T14:53:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:19 smithi110 ceph-mon[97633]: pgmap v19968: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:53:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:19 smithi104 ceph-mon[97044]: pgmap v19968: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:53:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:19.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:19.180607+0000) 2022-09-10T14:53:20.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:19.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:19.807374+0000) 2022-09-10T14:53:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:20.179+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:20.180854+0000) 2022-09-10T14:53:21.182 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:20.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:20.807553+0000) 2022-09-10T14:53:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:21 smithi110 ceph-mon[97633]: pgmap v19969: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:53:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:21 smithi104 ceph-mon[97044]: pgmap v19969: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:53:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:21.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:21.181136+0000) 2022-09-10T14:53:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:53:21.186+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:53:22.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:21.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:21.807754+0000) 2022-09-10T14:53:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:22.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:22.181394+0000) 2022-09-10T14:53:23.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:22.806+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:22.807992+0000) 2022-09-10T14:53:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:23 smithi110 ceph-mon[97633]: pgmap v19970: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:53:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:23 smithi104 ceph-mon[97044]: pgmap v19970: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:53:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:23.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:23.181563+0000) 2022-09-10T14:53:24.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:23.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:23.808221+0000) 2022-09-10T14:53:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:24.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:24.181803+0000) 2022-09-10T14:53:25.183 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:24.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:24.808475+0000) 2022-09-10T14:53:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:25 smithi110 ceph-mon[97633]: pgmap v19971: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:53:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:25 smithi104 ceph-mon[97044]: pgmap v19971: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:53:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:25.180+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:25.182021+0000) 2022-09-10T14:53:26.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:25.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:25.808668+0000) 2022-09-10T14:53:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:26.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:26.182248+0000) 2022-09-10T14:53:27.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:26.807+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:26.808855+0000) 2022-09-10T14:53:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:27 smithi110 ceph-mon[97633]: pgmap v19972: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:53:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:27 smithi104 ceph-mon[97044]: pgmap v19972: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:53:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:27.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:27.182481+0000) 2022-09-10T14:53:28.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:27.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:27.809076+0000) 2022-09-10T14:53:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:28.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:28.182666+0000) 2022-09-10T14:53:29.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:28.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:28.809300+0000) 2022-09-10T14:53:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:29 smithi110 ceph-mon[97633]: pgmap v19973: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:29 smithi104 ceph-mon[97044]: pgmap v19973: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:29.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:29.181+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:29.182898+0000) 2022-09-10T14:53:30.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:29.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:29.809506+0000) 2022-09-10T14:53:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:30.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:30.183123+0000) 2022-09-10T14:53:31.184 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:30.808+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:30.809732+0000) 2022-09-10T14:53:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:31 smithi110 ceph-mon[97633]: pgmap v19974: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:31 smithi104 ceph-mon[97044]: pgmap v19974: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:31.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:31.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:31.183274+0000) 2022-09-10T14:53:32.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:31.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:31.809983+0000) 2022-09-10T14:53:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:32.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:32.183473+0000) 2022-09-10T14:53:33.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:32.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:32.810168+0000) 2022-09-10T14:53:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:33 smithi110 ceph-mon[97633]: pgmap v19975: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:33 smithi104 ceph-mon[97044]: pgmap v19975: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:33.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:33.183731+0000) 2022-09-10T14:53:34.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:33.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:33.810370+0000) 2022-09-10T14:53:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:34.182+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:34.183957+0000) 2022-09-10T14:53:35.185 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:34.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:34.810616+0000) 2022-09-10T14:53:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:35 smithi110 ceph-mon[97633]: pgmap v19976: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:35 smithi104 ceph-mon[97044]: pgmap v19976: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:35.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:35.184200+0000) 2022-09-10T14:53:36.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:35.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:35.810799+0000) 2022-09-10T14:53:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:36.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:36.184359+0000) 2022-09-10T14:53:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:53:36.195+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:53:37.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:36.809+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:36.811039+0000) 2022-09-10T14:53:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:37 smithi110 ceph-mon[97633]: pgmap v19977: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:37 smithi104 ceph-mon[97044]: pgmap v19977: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:37.183+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:37.184600+0000) 2022-09-10T14:53:38.186 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:37.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:37.811245+0000) 2022-09-10T14:53:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:38.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:38.184802+0000) 2022-09-10T14:53:39.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:38.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:38.811494+0000) 2022-09-10T14:53:39.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:39 smithi110 ceph-mon[97633]: pgmap v19978: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:39 smithi104 ceph-mon[97044]: pgmap v19978: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:39.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:39.185036+0000) 2022-09-10T14:53:40.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:39.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:39.811705+0000) 2022-09-10T14:53:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:40.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:40.185254+0000) 2022-09-10T14:53:41.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:40.810+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:40.811887+0000) 2022-09-10T14:53:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:41 smithi110 ceph-mon[97633]: pgmap v19979: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:41 smithi104 ceph-mon[97044]: pgmap v19979: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:41.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:41.185409+0000) 2022-09-10T14:53:42.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:41.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:41.812109+0000) 2022-09-10T14:53:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:42.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:42.185629+0000) 2022-09-10T14:53:43.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:42.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:42.812350+0000) 2022-09-10T14:53:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:43 smithi110 ceph-mon[97633]: pgmap v19980: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:43 smithi104 ceph-mon[97044]: pgmap v19980: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:43.184+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:43.185851+0000) 2022-09-10T14:53:44.187 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:43.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:43.812607+0000) 2022-09-10T14:53:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:44.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:44.186043+0000) 2022-09-10T14:53:45.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:44.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:44.812801+0000) 2022-09-10T14:53:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:45 smithi110 ceph-mon[97633]: pgmap v19981: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:45 smithi104 ceph-mon[97044]: pgmap v19981: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:45.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:45.186278+0000) 2022-09-10T14:53:46.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:45.811+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:45.812947+0000) 2022-09-10T14:53:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:46.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:46.186503+0000) 2022-09-10T14:53:47.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:46.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:46.813190+0000) 2022-09-10T14:53:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:47 smithi110 ceph-mon[97633]: pgmap v19982: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:53:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:53:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:53:47.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:53:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:47 smithi104 ceph-mon[97044]: pgmap v19982: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:53:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:53:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:53:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:53:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:47.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:47.186734+0000) 2022-09-10T14:53:48.188 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:47.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:47.813419+0000) 2022-09-10T14:53:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:48.185+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:48.186977+0000) 2022-09-10T14:53:49.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:48.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:48.813694+0000) 2022-09-10T14:53:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:49 smithi110 ceph-mon[97633]: pgmap v19983: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:49 smithi104 ceph-mon[97044]: pgmap v19983: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:49.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:49.187254+0000) 2022-09-10T14:53:50.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:49.812+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:49.813922+0000) 2022-09-10T14:53:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:50.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:50.187478+0000) 2022-09-10T14:53:51.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:50.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:50.814199+0000) 2022-09-10T14:53:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:51 smithi110 ceph-mon[97633]: pgmap v19984: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:51 smithi104 ceph-mon[97044]: pgmap v19984: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:51.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:51.187691+0000) 2022-09-10T14:53:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:53:51.210+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:53:52.189 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:51.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:51.814443+0000) 2022-09-10T14:53:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:52.186+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:52.187929+0000) 2022-09-10T14:53:53.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:52.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:52.814611+0000) 2022-09-10T14:53:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:53 smithi110 ceph-mon[97633]: pgmap v19985: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:53 smithi104 ceph-mon[97044]: pgmap v19985: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:53.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:53.188092+0000) 2022-09-10T14:53:54.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:53.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:53.814792+0000) 2022-09-10T14:53:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:54.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:54.188285+0000) 2022-09-10T14:53:55.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:54.813+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:54.815019+0000) 2022-09-10T14:53:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:55 smithi110 ceph-mon[97633]: pgmap v19986: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:55 smithi104 ceph-mon[97044]: pgmap v19986: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:55.187+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:55.188512+0000) 2022-09-10T14:53:56.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:55.814+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:55.815219+0000) 2022-09-10T14:53:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:56.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:56.188677+0000) 2022-09-10T14:53:57.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:56.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:56.815484+0000) 2022-09-10T14:53:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:57 smithi110 ceph-mon[97633]: pgmap v19987: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:57 smithi104 ceph-mon[97044]: pgmap v19987: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:53:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:57.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:57.188899+0000) 2022-09-10T14:53:58.190 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:57.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:57.815758+0000) 2022-09-10T14:53:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:58.188+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:58.189150+0000) 2022-09-10T14:53:59.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:58.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:58.816006+0000) 2022-09-10T14:53:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:53:59 smithi110 ceph-mon[97633]: pgmap v19988: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:53:59 smithi104 ceph-mon[97044]: pgmap v19988: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:53:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:53:59.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:59.189374+0000) 2022-09-10T14:54:00.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:53:59.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:53:59.816201+0000) 2022-09-10T14:54:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:00.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:00.189604+0000) 2022-09-10T14:54:01.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:00.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:00.816391+0000) 2022-09-10T14:54:01.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:01 smithi110 ceph-mon[97633]: pgmap v19989: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:01 smithi104 ceph-mon[97044]: pgmap v19989: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:01.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:01.189844+0000) 2022-09-10T14:54:02.191 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:01.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:01.816555+0000) 2022-09-10T14:54:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:02.189+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:02.190091+0000) 2022-09-10T14:54:03.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:02.815+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:02.816688+0000) 2022-09-10T14:54:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:03 smithi110 ceph-mon[97633]: pgmap v19990: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:03 smithi104 ceph-mon[97044]: pgmap v19990: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:03.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:03.190364+0000) 2022-09-10T14:54:04.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:03.816+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:03.816893+0000) 2022-09-10T14:54:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:04.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:04.190541+0000) 2022-09-10T14:54:05.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:04.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:04.817131+0000) 2022-09-10T14:54:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:05 smithi110 ceph-mon[97633]: pgmap v19991: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:05 smithi104 ceph-mon[97044]: pgmap v19991: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:05.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:05.190688+0000) 2022-09-10T14:54:06.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:05.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:05.817335+0000) 2022-09-10T14:54:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:06.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:06.190913+0000) 2022-09-10T14:54:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:54:06.221+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:54:07.192 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:06.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:06.817659+0000) 2022-09-10T14:54:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:07 smithi110 ceph-mon[97633]: pgmap v19992: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:54:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:07 smithi104 ceph-mon[97044]: pgmap v19992: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:54:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:07.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:07.191159+0000) 2022-09-10T14:54:08.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:07.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:07.817886+0000) 2022-09-10T14:54:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:08.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:08.191344+0000) 2022-09-10T14:54:09.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:08.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:08.818172+0000) 2022-09-10T14:54:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:09 smithi110 ceph-mon[97633]: pgmap v19993: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:54:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:09 smithi110 ceph-mon[97633]: osdmap e1330: 12 total, 12 up, 12 in 2022-09-10T14:54:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:09 smithi104 ceph-mon[97044]: pgmap v19993: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:54:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:09 smithi104 ceph-mon[97044]: osdmap e1330: 12 total, 12 up, 12 in 2022-09-10T14:54:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:09.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:09.191543+0000) 2022-09-10T14:54:10.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:09.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:09.818403+0000) 2022-09-10T14:54:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:10.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:10.191714+0000) 2022-09-10T14:54:11.193 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:10.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:10.818647+0000) 2022-09-10T14:54:11.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:11 smithi110 ceph-mon[97633]: pgmap v19995: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:54:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:11 smithi104 ceph-mon[97044]: pgmap v19995: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:54:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:11.190+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:11.191995+0000) 2022-09-10T14:54:12.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:11.817+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:11.818908+0000) 2022-09-10T14:54:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:12.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:12.192224+0000) 2022-09-10T14:54:13.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:12.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:12.819195+0000) 2022-09-10T14:54:13.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:13 smithi110 ceph-mon[97633]: pgmap v19996: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:54:13.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:13 smithi110 ceph-mon[97633]: osdmap e1331: 12 total, 12 up, 12 in 2022-09-10T14:54:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:13 smithi104 ceph-mon[97044]: pgmap v19996: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T14:54:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:13 smithi104 ceph-mon[97044]: osdmap e1331: 12 total, 12 up, 12 in 2022-09-10T14:54:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:13.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:13.192469+0000) 2022-09-10T14:54:14.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:13.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:13.819446+0000) 2022-09-10T14:54:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:14.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:14.192654+0000) 2022-09-10T14:54:15.194 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:14.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:14.819718+0000) 2022-09-10T14:54:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:15 smithi110 ceph-mon[97633]: pgmap v19998: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:54:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:15 smithi104 ceph-mon[97044]: pgmap v19998: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:54:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:15.191+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:15.192873+0000) 2022-09-10T14:54:16.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:15.818+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:15.819946+0000) 2022-09-10T14:54:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:16.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:16.193137+0000) 2022-09-10T14:54:17.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:16.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:16.820223+0000) 2022-09-10T14:54:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:17 smithi110 ceph-mon[97633]: pgmap v19999: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:54:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:17 smithi104 ceph-mon[97044]: pgmap v19999: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:54:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:17.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:17.193283+0000) 2022-09-10T14:54:18.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:17.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:17.820396+0000) 2022-09-10T14:54:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:18.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:18.193493+0000) 2022-09-10T14:54:19.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:18.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:18.820634+0000) 2022-09-10T14:54:19.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:19 smithi110 ceph-mon[97633]: pgmap v20000: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:54:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:19 smithi104 ceph-mon[97044]: pgmap v20000: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:54:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:19.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:19.193679+0000) 2022-09-10T14:54:20.195 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:19.819+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:19.820842+0000) 2022-09-10T14:54:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:20.192+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:20.193904+0000) 2022-09-10T14:54:21.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:20.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:20.821150+0000) 2022-09-10T14:54:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:21 smithi110 ceph-mon[97633]: pgmap v20001: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:54:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:21 smithi104 ceph-mon[97044]: pgmap v20001: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:54:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:21.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:21.194130+0000) 2022-09-10T14:54:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:54:21.235+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:54:22.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:21.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:21.821398+0000) 2022-09-10T14:54:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:22.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:22.194359+0000) 2022-09-10T14:54:23.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:22.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:22.821658+0000) 2022-09-10T14:54:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:23 smithi110 ceph-mon[97633]: pgmap v20002: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:54:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:23 smithi104 ceph-mon[97044]: pgmap v20002: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:54:23.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:23.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:23.194508+0000) 2022-09-10T14:54:24.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:23.820+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:23.821817+0000) 2022-09-10T14:54:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:24.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:24.194735+0000) 2022-09-10T14:54:25.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:24.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:24.822042+0000) 2022-09-10T14:54:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:25 smithi110 ceph-mon[97633]: pgmap v20003: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:54:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:25 smithi104 ceph-mon[97044]: pgmap v20003: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:54:25.620 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:25.193+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:25.194916+0000) 2022-09-10T14:54:26.196 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:25.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:25.822268+0000) 2022-09-10T14:54:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:26.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:26.195156+0000) 2022-09-10T14:54:27.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:26.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:26.822518+0000) 2022-09-10T14:54:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:27 smithi110 ceph-mon[97633]: pgmap v20004: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:54:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:27 smithi104 ceph-mon[97044]: pgmap v20004: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:54:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:27.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:27.195369+0000) 2022-09-10T14:54:28.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:27.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:27.822751+0000) 2022-09-10T14:54:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:28.194+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:28.195618+0000) 2022-09-10T14:54:29.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:28.821+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:28.822946+0000) 2022-09-10T14:54:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:29 smithi110 ceph-mon[97633]: pgmap v20005: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:29 smithi104 ceph-mon[97044]: pgmap v20005: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:29.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:29.195881+0000) 2022-09-10T14:54:30.197 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:29.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:29.823167+0000) 2022-09-10T14:54:30.565 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:30.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:30.196141+0000) 2022-09-10T14:54:31.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:30.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:30.823384+0000) 2022-09-10T14:54:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:31 smithi110 ceph-mon[97633]: pgmap v20006: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:31 smithi104 ceph-mon[97044]: pgmap v20006: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:31.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:31.196407+0000) 2022-09-10T14:54:32.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:31.822+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:31.823597+0000) 2022-09-10T14:54:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:32.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:32.196548+0000) 2022-09-10T14:54:33.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:32.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:32.823798+0000) 2022-09-10T14:54:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:33 smithi110 ceph-mon[97633]: pgmap v20007: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:33 smithi104 ceph-mon[97044]: pgmap v20007: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:33.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:33.196744+0000) 2022-09-10T14:54:34.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:33.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:33.824001+0000) 2022-09-10T14:54:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:34.195+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:34.196995+0000) 2022-09-10T14:54:35.198 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:34.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:34.824224+0000) 2022-09-10T14:54:35.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:35 smithi110 ceph-mon[97633]: pgmap v20008: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:35 smithi104 ceph-mon[97044]: pgmap v20008: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:35.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:35.197187+0000) 2022-09-10T14:54:36.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:35.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:35.824425+0000) 2022-09-10T14:54:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:54:36.247+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:54:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:36.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:36.197311+0000) 2022-09-10T14:54:37.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:36.823+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:36.824609+0000) 2022-09-10T14:54:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:37 smithi110 ceph-mon[97633]: pgmap v20009: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:37 smithi104 ceph-mon[97044]: pgmap v20009: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:37.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:37.197444+0000) 2022-09-10T14:54:38.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:37.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:37.824836+0000) 2022-09-10T14:54:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:38.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:38.197661+0000) 2022-09-10T14:54:39.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:38.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:38.825068+0000) 2022-09-10T14:54:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:39 smithi110 ceph-mon[97633]: pgmap v20010: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:39 smithi104 ceph-mon[97044]: pgmap v20010: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:39.196+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:39.197848+0000) 2022-09-10T14:54:40.199 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:39.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:39.825236+0000) 2022-09-10T14:54:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:40.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:40.198004+0000) 2022-09-10T14:54:41.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:40.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:40.825442+0000) 2022-09-10T14:54:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:41 smithi110 ceph-mon[97633]: pgmap v20011: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:41 smithi104 ceph-mon[97044]: pgmap v20011: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:41.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:41.198340+0000) 2022-09-10T14:54:42.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:41.824+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:41.825561+0000) 2022-09-10T14:54:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:42.197+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:42.198531+0000) 2022-09-10T14:54:43.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:42.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:42.825779+0000) 2022-09-10T14:54:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:43 smithi110 ceph-mon[97633]: pgmap v20012: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:43 smithi104 ceph-mon[97044]: pgmap v20012: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:43.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:43.198755+0000) 2022-09-10T14:54:44.200 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:43.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:43.825998+0000) 2022-09-10T14:54:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:44.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:44.198971+0000) 2022-09-10T14:54:45.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:44.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:44.826241+0000) 2022-09-10T14:54:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:45 smithi110 ceph-mon[97633]: pgmap v20013: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:45 smithi104 ceph-mon[97044]: pgmap v20013: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:45.554 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:45.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:45.199211+0000) 2022-09-10T14:54:46.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:45.825+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:45.826422+0000) 2022-09-10T14:54:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:46.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:46.199439+0000) 2022-09-10T14:54:47.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:46.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:46.826729+0000) 2022-09-10T14:54:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:47 smithi110 ceph-mon[97633]: pgmap v20014: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:54:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:54:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:54:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:54:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:47 smithi104 ceph-mon[97044]: pgmap v20014: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:54:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:54:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:54:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:54:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:47.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:47.199664+0000) 2022-09-10T14:54:48.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:47.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:47.826970+0000) 2022-09-10T14:54:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:48.198+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:48.199926+0000) 2022-09-10T14:54:49.201 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:48.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:48.827276+0000) 2022-09-10T14:54:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:49 smithi110 ceph-mon[97633]: pgmap v20015: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:49 smithi104 ceph-mon[97044]: pgmap v20015: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:49.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:49.200197+0000) 2022-09-10T14:54:50.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:49.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:49.827458+0000) 2022-09-10T14:54:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:50.199+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:50.200374+0000) 2022-09-10T14:54:51.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:50.826+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:50.827703+0000) 2022-09-10T14:54:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:51 smithi110 ceph-mon[97633]: pgmap v20016: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:51 smithi104 ceph-mon[97044]: pgmap v20016: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:54:51.257+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:54:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:51.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:51.200599+0000) 2022-09-10T14:54:52.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:51.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:51.827892+0000) 2022-09-10T14:54:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:52.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:52.200802+0000) 2022-09-10T14:54:53.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:52.827+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:52.828037+0000) 2022-09-10T14:54:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:53 smithi110 ceph-mon[97633]: pgmap v20017: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:53 smithi104 ceph-mon[97044]: pgmap v20017: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:53.200+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:53.201076+0000) 2022-09-10T14:54:54.202 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:53.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:53.828240+0000) 2022-09-10T14:54:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:54.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:54.201297+0000) 2022-09-10T14:54:55.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:54.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:54.828473+0000) 2022-09-10T14:54:55.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:55 smithi110 ceph-mon[97633]: pgmap v20018: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:55 smithi104 ceph-mon[97044]: pgmap v20018: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:55.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:55.201534+0000) 2022-09-10T14:54:56.203 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:55.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:55.828706+0000) 2022-09-10T14:54:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:56.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:56.201700+0000) 2022-09-10T14:54:57.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:56.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:56.828990+0000) 2022-09-10T14:54:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:57 smithi110 ceph-mon[97633]: pgmap v20019: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:57 smithi104 ceph-mon[97044]: pgmap v20019: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:54:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:57.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:57.201924+0000) 2022-09-10T14:54:58.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:57.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:57.829239+0000) 2022-09-10T14:54:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:58.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:58.202205+0000) 2022-09-10T14:54:59.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:58.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:58.829501+0000) 2022-09-10T14:54:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:54:59 smithi110 ceph-mon[97633]: pgmap v20020: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:54:59 smithi104 ceph-mon[97044]: pgmap v20020: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:54:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:54:59.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:59.202435+0000) 2022-09-10T14:55:00.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:54:59.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:54:59.829718+0000) 2022-09-10T14:55:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:00.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:00.202639+0000) 2022-09-10T14:55:01.204 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:00.828+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:00.829904+0000) 2022-09-10T14:55:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:01 smithi110 ceph-mon[97633]: pgmap v20021: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:01 smithi104 ceph-mon[97044]: pgmap v20021: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:01.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:01.202839+0000) 2022-09-10T14:55:02.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:01.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:01.830131+0000) 2022-09-10T14:55:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:02.201+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:02.203059+0000) 2022-09-10T14:55:03.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:02.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:02.830344+0000) 2022-09-10T14:55:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:03 smithi110 ceph-mon[97633]: pgmap v20022: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:03 smithi104 ceph-mon[97044]: pgmap v20022: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:03.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:03.203228+0000) 2022-09-10T14:55:04.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:03.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:03.830518+0000) 2022-09-10T14:55:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:04.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:04.203473+0000) 2022-09-10T14:55:05.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:04.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:04.830706+0000) 2022-09-10T14:55:05.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:05 smithi110 ceph-mon[97633]: pgmap v20023: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:05 smithi104 ceph-mon[97044]: pgmap v20023: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:05.202+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:05.203720+0000) 2022-09-10T14:55:06.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:05.829+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:05.830944+0000) 2022-09-10T14:55:06.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:06.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:06.203916+0000) 2022-09-10T14:55:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:55:06.262+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:55:07.205 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:06.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:06.831189+0000) 2022-09-10T14:55:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:07 smithi110 ceph-mon[97633]: pgmap v20024: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:55:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:07 smithi104 ceph-mon[97044]: pgmap v20024: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:55:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:07.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:07.204173+0000) 2022-09-10T14:55:08.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:07.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:07.831383+0000) 2022-09-10T14:55:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:08.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:08.204400+0000) 2022-09-10T14:55:09.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:08.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:08.831601+0000) 2022-09-10T14:55:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:09 smithi110 ceph-mon[97633]: pgmap v20025: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:55:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:09 smithi110 ceph-mon[97633]: osdmap e1332: 12 total, 12 up, 12 in 2022-09-10T14:55:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:09 smithi104 ceph-mon[97044]: pgmap v20025: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:55:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:09 smithi104 ceph-mon[97044]: osdmap e1332: 12 total, 12 up, 12 in 2022-09-10T14:55:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:09.203+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:09.204633+0000) 2022-09-10T14:55:10.206 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:09.830+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:09.831823+0000) 2022-09-10T14:55:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:10.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:10.204877+0000) 2022-09-10T14:55:11.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:10.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:10.832089+0000) 2022-09-10T14:55:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:11 smithi110 ceph-mon[97633]: pgmap v20027: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:55:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:11 smithi104 ceph-mon[97044]: pgmap v20027: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:55:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:11.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:11.205193+0000) 2022-09-10T14:55:12.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:11.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:11.832295+0000) 2022-09-10T14:55:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:12.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:12.205355+0000) 2022-09-10T14:55:13.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:12.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:12.832503+0000) 2022-09-10T14:55:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:13 smithi110 ceph-mon[97633]: pgmap v20028: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:55:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:13 smithi110 ceph-mon[97633]: osdmap e1333: 12 total, 12 up, 12 in 2022-09-10T14:55:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:13 smithi104 ceph-mon[97044]: pgmap v20028: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:55:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:13 smithi104 ceph-mon[97044]: osdmap e1333: 12 total, 12 up, 12 in 2022-09-10T14:55:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:13.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:13.205617+0000) 2022-09-10T14:55:14.207 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:13.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:13.832694+0000) 2022-09-10T14:55:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:14.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:14.205850+0000) 2022-09-10T14:55:15.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:14.831+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:14.832918+0000) 2022-09-10T14:55:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:15 smithi110 ceph-mon[97633]: pgmap v20030: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:55:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:15 smithi104 ceph-mon[97044]: pgmap v20030: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:55:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:15.204+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:15.206072+0000) 2022-09-10T14:55:16.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:15.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:15.833218+0000) 2022-09-10T14:55:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:16.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:16.206343+0000) 2022-09-10T14:55:17.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:16.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:16.833427+0000) 2022-09-10T14:55:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:17 smithi110 ceph-mon[97633]: pgmap v20031: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:55:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:17 smithi104 ceph-mon[97044]: pgmap v20031: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:55:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:17.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:17.206541+0000) 2022-09-10T14:55:18.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:17.832+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:17.833644+0000) 2022-09-10T14:55:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:18.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:18.206732+0000) 2022-09-10T14:55:19.208 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:18.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:18.833897+0000) 2022-09-10T14:55:20.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:19 smithi104 ceph-mon[97044]: pgmap v20032: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:55:20.075 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:19 smithi110 ceph-mon[97633]: pgmap v20032: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:55:20.075 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:19.205+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:19.206927+0000) 2022-09-10T14:55:20.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:19.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:19.834153+0000) 2022-09-10T14:55:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:20.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:20.207172+0000) 2022-09-10T14:55:21.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:20.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:20.834375+0000) 2022-09-10T14:55:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:21 smithi110 ceph-mon[97633]: pgmap v20033: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:55:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:21 smithi104 ceph-mon[97044]: pgmap v20033: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:55:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:21.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:21.207376+0000) 2022-09-10T14:55:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:55:21.268+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:55:22.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:21.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:21.834559+0000) 2022-09-10T14:55:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:22.206+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:22.207575+0000) 2022-09-10T14:55:23.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:22.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:22.834763+0000) 2022-09-10T14:55:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:23 smithi110 ceph-mon[97633]: pgmap v20034: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:55:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:23 smithi104 ceph-mon[97044]: pgmap v20034: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:55:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:23.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:23.207796+0000) 2022-09-10T14:55:24.209 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:23.833+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:23.835009+0000) 2022-09-10T14:55:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:24.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:24.208031+0000) 2022-09-10T14:55:25.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:24.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:24.835248+0000) 2022-09-10T14:55:25.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:25 smithi110 ceph-mon[97633]: pgmap v20035: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:55:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:25 smithi104 ceph-mon[97044]: pgmap v20035: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:55:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:25.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:25.208271+0000) 2022-09-10T14:55:26.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:25.834+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:25.835409+0000) 2022-09-10T14:55:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:26.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:26.208493+0000) 2022-09-10T14:55:27.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:26.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:26.835677+0000) 2022-09-10T14:55:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:27 smithi110 ceph-mon[97633]: pgmap v20036: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:55:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:27 smithi104 ceph-mon[97044]: pgmap v20036: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:55:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:27.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:27.208679+0000) 2022-09-10T14:55:28.210 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:27.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:27.835927+0000) 2022-09-10T14:55:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:28.207+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:28.208861+0000) 2022-09-10T14:55:29.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:28.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:28.836148+0000) 2022-09-10T14:55:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:29 smithi110 ceph-mon[97633]: pgmap v20037: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:29 smithi104 ceph-mon[97044]: pgmap v20037: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:29.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:29.209186+0000) 2022-09-10T14:55:30.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:29.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:29.836314+0000) 2022-09-10T14:55:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:30.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:30.209350+0000) 2022-09-10T14:55:31.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:30.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:30.836507+0000) 2022-09-10T14:55:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:31 smithi110 ceph-mon[97633]: pgmap v20038: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:31 smithi104 ceph-mon[97044]: pgmap v20038: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:31.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:31.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:31.209600+0000) 2022-09-10T14:55:32.211 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:31.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:31.836726+0000) 2022-09-10T14:55:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:32.208+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:32.209860+0000) 2022-09-10T14:55:33.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:32.835+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:32.836959+0000) 2022-09-10T14:55:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:33 smithi110 ceph-mon[97633]: pgmap v20039: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:33 smithi104 ceph-mon[97044]: pgmap v20039: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:33.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:33.210147+0000) 2022-09-10T14:55:34.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:33.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:33.837184+0000) 2022-09-10T14:55:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:34.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:34.210373+0000) 2022-09-10T14:55:35.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:34.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:34.837382+0000) 2022-09-10T14:55:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:35 smithi110 ceph-mon[97633]: pgmap v20040: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:35.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:35 smithi104 ceph-mon[97044]: pgmap v20040: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:35.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:35.210560+0000) 2022-09-10T14:55:36.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:35.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:35.837610+0000) 2022-09-10T14:55:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:55:36.284+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:55:36.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:36.209+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:36.210735+0000) 2022-09-10T14:55:37.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:36.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:36.837872+0000) 2022-09-10T14:55:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:37 smithi110 ceph-mon[97633]: pgmap v20041: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:37 smithi104 ceph-mon[97044]: pgmap v20041: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:37.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:37.210962+0000) 2022-09-10T14:55:38.212 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:37.836+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:37.838021+0000) 2022-09-10T14:55:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:38.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:38.211135+0000) 2022-09-10T14:55:39.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:38.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:38.838241+0000) 2022-09-10T14:55:39.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:39 smithi110 ceph-mon[97633]: pgmap v20042: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:39 smithi104 ceph-mon[97044]: pgmap v20042: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:39.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:39.211341+0000) 2022-09-10T14:55:40.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:39.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:39.838397+0000) 2022-09-10T14:55:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:40.210+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:40.211521+0000) 2022-09-10T14:55:41.213 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:40.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:40.838615+0000) 2022-09-10T14:55:41.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:41 smithi110 ceph-mon[97633]: pgmap v20043: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:41 smithi104 ceph-mon[97044]: pgmap v20043: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:41.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:41.211749+0000) 2022-09-10T14:55:42.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:41.837+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:41.838855+0000) 2022-09-10T14:55:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:42.211+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:42.211962+0000) 2022-09-10T14:55:43.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:42.838+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:42.839136+0000) 2022-09-10T14:55:43.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:43 smithi110 ceph-mon[97633]: pgmap v20044: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:43 smithi104 ceph-mon[97044]: pgmap v20044: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:43.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:43.212179+0000) 2022-09-10T14:55:44.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:43.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:43.839368+0000) 2022-09-10T14:55:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:44.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:44.212342+0000) 2022-09-10T14:55:45.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:44.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:44.839561+0000) 2022-09-10T14:55:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:45 smithi110 ceph-mon[97633]: pgmap v20045: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:45 smithi104 ceph-mon[97044]: pgmap v20045: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:45.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:45.212558+0000) 2022-09-10T14:55:46.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:45.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:45.839753+0000) 2022-09-10T14:55:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:46.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:46.212765+0000) 2022-09-10T14:55:47.214 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:46.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:46.840005+0000) 2022-09-10T14:55:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:47 smithi110 ceph-mon[97633]: pgmap v20046: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:55:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:55:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:55:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:55:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:47 smithi104 ceph-mon[97044]: pgmap v20046: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:55:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:55:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:55:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:55:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:47.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:47.212984+0000) 2022-09-10T14:55:48.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:47.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:47.840220+0000) 2022-09-10T14:55:48.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:48.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:48.213243+0000) 2022-09-10T14:55:49.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:48.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:48.840503+0000) 2022-09-10T14:55:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:49 smithi110 ceph-mon[97633]: pgmap v20047: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:49 smithi104 ceph-mon[97044]: pgmap v20047: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:49.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:49.213449+0000) 2022-09-10T14:55:50.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:49.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:49.840751+0000) 2022-09-10T14:55:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:50.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:50.213669+0000) 2022-09-10T14:55:51.215 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:50.839+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:50.840986+0000) 2022-09-10T14:55:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:51 smithi110 ceph-mon[97633]: pgmap v20048: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:51 smithi104 ceph-mon[97044]: pgmap v20048: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:51.212+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:51.213944+0000) 2022-09-10T14:55:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:55:51.296+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:55:52.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:51.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:51.841238+0000) 2022-09-10T14:55:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:52.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:52.214215+0000) 2022-09-10T14:55:53.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:52.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:52.841432+0000) 2022-09-10T14:55:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:53 smithi110 ceph-mon[97633]: pgmap v20049: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:53 smithi104 ceph-mon[97044]: pgmap v20049: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:53.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:53.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:53.214469+0000) 2022-09-10T14:55:54.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:53.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:53.841660+0000) 2022-09-10T14:55:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:54.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:54.214726+0000) 2022-09-10T14:55:55.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:54.840+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:54.841852+0000) 2022-09-10T14:55:55.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:55 smithi110 ceph-mon[97633]: pgmap v20050: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:55 smithi104 ceph-mon[97044]: pgmap v20050: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:55:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:55.213+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:55.214922+0000) 2022-09-10T14:55:56.216 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:55.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:55.842082+0000) 2022-09-10T14:55:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:56.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:56.215152+0000) 2022-09-10T14:55:57.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:56.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:56.842341+0000) 2022-09-10T14:55:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:57 smithi110 ceph-mon[97633]: pgmap v20051: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:57 smithi104 ceph-mon[97044]: pgmap v20051: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:55:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:57.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:57.215317+0000) 2022-09-10T14:55:58.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:57.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:57.842542+0000) 2022-09-10T14:55:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:58.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:58.215555+0000) 2022-09-10T14:55:59.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:58.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:58.842759+0000) 2022-09-10T14:55:59.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:55:59 smithi110 ceph-mon[97633]: pgmap v20052: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:55:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:55:59 smithi104 ceph-mon[97044]: pgmap v20052: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:55:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:55:59.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:59.215777+0000) 2022-09-10T14:56:00.217 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:55:59.841+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:55:59.842985+0000) 2022-09-10T14:56:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:00.214+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:00.215916+0000) 2022-09-10T14:56:01.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:00.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:00.843164+0000) 2022-09-10T14:56:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:01 smithi110 ceph-mon[97633]: pgmap v20053: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:01 smithi104 ceph-mon[97044]: pgmap v20053: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:01.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:01.216137+0000) 2022-09-10T14:56:02.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:01.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:01.843334+0000) 2022-09-10T14:56:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:02.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:02.216414+0000) 2022-09-10T14:56:03.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:02.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:02.843493+0000) 2022-09-10T14:56:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:03 smithi110 ceph-mon[97633]: pgmap v20054: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:56:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:03 smithi104 ceph-mon[97044]: pgmap v20054: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:56:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:03.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:03.216662+0000) 2022-09-10T14:56:04.218 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:03.842+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:03.843731+0000) 2022-09-10T14:56:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:04.215+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:04.216912+0000) 2022-09-10T14:56:05.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:04.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:04.844002+0000) 2022-09-10T14:56:05.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:05 smithi110 ceph-mon[97633]: pgmap v20055: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:56:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:05 smithi104 ceph-mon[97044]: pgmap v20055: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:56:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:05.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:05.217184+0000) 2022-09-10T14:56:06.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:05.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:05.844220+0000) 2022-09-10T14:56:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:06.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:06.217386+0000) 2022-09-10T14:56:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:56:06.306+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:56:07.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:06.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:06.844395+0000) 2022-09-10T14:56:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:07 smithi104 ceph-mon[97044]: pgmap v20056: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:56:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:07.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:07.217540+0000) 2022-09-10T14:56:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:07 smithi110 ceph-mon[97633]: pgmap v20056: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:56:08.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:07.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:07.844615+0000) 2022-09-10T14:56:08.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:08.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:08.217726+0000) 2022-09-10T14:56:09.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:08.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:08.844841+0000) 2022-09-10T14:56:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:09 smithi104 ceph-mon[97044]: pgmap v20057: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:56:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:09 smithi104 ceph-mon[97044]: osdmap e1334: 12 total, 12 up, 12 in 2022-09-10T14:56:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:09.216+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:09.217957+0000) 2022-09-10T14:56:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:09 smithi110 ceph-mon[97633]: pgmap v20057: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:56:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:09 smithi110 ceph-mon[97633]: osdmap e1334: 12 total, 12 up, 12 in 2022-09-10T14:56:10.219 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:09.843+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:09.845049+0000) 2022-09-10T14:56:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:10.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:10.218205+0000) 2022-09-10T14:56:11.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:10.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:10.845304+0000) 2022-09-10T14:56:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:11 smithi104 ceph-mon[97044]: pgmap v20059: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:56:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:11.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:11.218467+0000) 2022-09-10T14:56:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:11 smithi110 ceph-mon[97633]: pgmap v20059: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:56:12.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:11.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:11.845516+0000) 2022-09-10T14:56:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:12.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:12.218692+0000) 2022-09-10T14:56:13.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:12.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:12.845700+0000) 2022-09-10T14:56:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:13 smithi110 ceph-mon[97633]: pgmap v20060: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:56:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:13 smithi110 ceph-mon[97633]: osdmap e1335: 12 total, 12 up, 12 in 2022-09-10T14:56:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:13 smithi104 ceph-mon[97044]: pgmap v20060: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:56:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:13 smithi104 ceph-mon[97044]: osdmap e1335: 12 total, 12 up, 12 in 2022-09-10T14:56:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:13.217+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:13.218869+0000) 2022-09-10T14:56:14.220 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:13.844+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:13.845960+0000) 2022-09-10T14:56:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:14.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:14.219090+0000) 2022-09-10T14:56:15.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:14.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:14.846193+0000) 2022-09-10T14:56:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:15 smithi110 ceph-mon[97633]: pgmap v20062: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:56:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:15 smithi104 ceph-mon[97044]: pgmap v20062: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:56:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:15.218+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:15.219345+0000) 2022-09-10T14:56:16.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:15.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:15.846375+0000) 2022-09-10T14:56:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:16.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:16.219545+0000) 2022-09-10T14:56:17.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:16.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:16.846651+0000) 2022-09-10T14:56:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:17 smithi110 ceph-mon[97633]: pgmap v20063: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:56:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:17 smithi104 ceph-mon[97044]: pgmap v20063: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:56:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:17.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:17.219754+0000) 2022-09-10T14:56:18.221 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:17.845+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:17.846878+0000) 2022-09-10T14:56:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:18.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:18.219979+0000) 2022-09-10T14:56:19.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:18.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:18.847188+0000) 2022-09-10T14:56:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:19 smithi110 ceph-mon[97633]: pgmap v20064: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:56:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:19 smithi104 ceph-mon[97044]: pgmap v20064: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:56:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:19.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:19.220226+0000) 2022-09-10T14:56:20.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:19.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:19.847382+0000) 2022-09-10T14:56:20.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:20.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:20.220389+0000) 2022-09-10T14:56:21.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:20.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:20.847623+0000) 2022-09-10T14:56:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:21 smithi110 ceph-mon[97633]: pgmap v20065: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:56:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:21 smithi104 ceph-mon[97044]: pgmap v20065: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:56:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:21.219+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:21.220576+0000) 2022-09-10T14:56:21.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:56:21.307+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:56:22.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:21.846+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:21.847877+0000) 2022-09-10T14:56:22.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:22.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:22.220770+0000) 2022-09-10T14:56:23.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:22.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:22.848075+0000) 2022-09-10T14:56:23.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:23 smithi110 ceph-mon[97633]: pgmap v20066: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:56:23.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:23.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:23.220963+0000) 2022-09-10T14:56:23.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:23 smithi104 ceph-mon[97044]: pgmap v20066: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:56:24.222 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:23.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:23.848234+0000) 2022-09-10T14:56:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:24.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:24.221176+0000) 2022-09-10T14:56:25.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:24.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:24.848464+0000) 2022-09-10T14:56:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:25 smithi110 ceph-mon[97633]: pgmap v20067: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:56:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:25 smithi104 ceph-mon[97044]: pgmap v20067: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:56:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:25.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:25.221374+0000) 2022-09-10T14:56:26.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:25.847+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:25.848633+0000) 2022-09-10T14:56:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:26.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:26.221520+0000) 2022-09-10T14:56:27.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:26.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:26.848838+0000) 2022-09-10T14:56:27.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:27 smithi110 ceph-mon[97633]: pgmap v20068: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:56:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:27 smithi104 ceph-mon[97044]: pgmap v20068: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:56:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:27.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:27.221736+0000) 2022-09-10T14:56:28.223 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:27.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:27.849031+0000) 2022-09-10T14:56:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:28.220+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:28.221865+0000) 2022-09-10T14:56:29.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:28.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:28.849293+0000) 2022-09-10T14:56:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:29 smithi110 ceph-mon[97633]: pgmap v20069: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:29 smithi104 ceph-mon[97044]: pgmap v20069: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:29.221+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:29.222157+0000) 2022-09-10T14:56:30.224 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:29.849+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:29.849479+0000) 2022-09-10T14:56:30.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:30.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:30.222324+0000) 2022-09-10T14:56:31.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:30.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:30.849706+0000) 2022-09-10T14:56:31.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:31 smithi110 ceph-mon[97633]: pgmap v20070: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:31 smithi104 ceph-mon[97044]: pgmap v20070: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:31.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:31.222520+0000) 2022-09-10T14:56:32.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:31.848+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:31.849990+0000) 2022-09-10T14:56:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:32.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:32.222718+0000) 2022-09-10T14:56:33.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:32.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:32.850240+0000) 2022-09-10T14:56:33.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:33 smithi110 ceph-mon[97633]: pgmap v20071: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:33 smithi104 ceph-mon[97044]: pgmap v20071: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:33.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:33.222942+0000) 2022-09-10T14:56:34.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:33.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:33.850455+0000) 2022-09-10T14:56:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:34.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:34.223153+0000) 2022-09-10T14:56:35.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:34.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:34.850636+0000) 2022-09-10T14:56:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:35 smithi110 ceph-mon[97633]: pgmap v20072: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:35 smithi104 ceph-mon[97044]: pgmap v20072: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:35.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:35.223355+0000) 2022-09-10T14:56:36.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:35.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:35.850835+0000) 2022-09-10T14:56:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:36.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:36.223601+0000) 2022-09-10T14:56:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:56:36.322+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:56:37.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:36.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:36.851081+0000) 2022-09-10T14:56:37.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:37 smithi110 ceph-mon[97633]: pgmap v20073: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:37 smithi104 ceph-mon[97044]: pgmap v20073: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:37.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:37.223808+0000) 2022-09-10T14:56:38.225 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:37.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:37.851348+0000) 2022-09-10T14:56:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:38.222+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:38.224015+0000) 2022-09-10T14:56:39.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:38.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:38.851606+0000) 2022-09-10T14:56:39.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:39 smithi110 ceph-mon[97633]: pgmap v20074: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:39 smithi104 ceph-mon[97044]: pgmap v20074: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:39.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:39.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:39.224246+0000) 2022-09-10T14:56:40.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:39.850+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:39.851827+0000) 2022-09-10T14:56:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:40.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:40.224393+0000) 2022-09-10T14:56:41.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:40.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:40.852049+0000) 2022-09-10T14:56:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:41 smithi110 ceph-mon[97633]: pgmap v20075: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:41 smithi104 ceph-mon[97044]: pgmap v20075: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:41.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:41.224642+0000) 2022-09-10T14:56:42.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:41.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:41.852301+0000) 2022-09-10T14:56:42.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:42.223+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:42.224917+0000) 2022-09-10T14:56:43.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:42.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:42.852445+0000) 2022-09-10T14:56:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:43 smithi110 ceph-mon[97633]: pgmap v20076: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:43 smithi104 ceph-mon[97044]: pgmap v20076: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:43.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:43.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:43.225087+0000) 2022-09-10T14:56:44.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:43.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:43.852692+0000) 2022-09-10T14:56:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:44.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:44.225320+0000) 2022-09-10T14:56:45.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:44.851+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:44.852918+0000) 2022-09-10T14:56:45.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:45 smithi110 ceph-mon[97633]: pgmap v20077: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:45 smithi104 ceph-mon[97044]: pgmap v20077: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:45.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:45.225499+0000) 2022-09-10T14:56:46.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:45.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:45.853104+0000) 2022-09-10T14:56:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:46.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:46.225650+0000) 2022-09-10T14:56:47.227 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:46.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:46.853269+0000) 2022-09-10T14:56:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:47 smithi110 ceph-mon[97633]: pgmap v20078: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:47.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:56:47.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:56:47.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:56:47.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:56:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:47 smithi104 ceph-mon[97044]: pgmap v20078: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:56:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:56:47.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:56:47.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:56:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:47.224+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:47.225861+0000) 2022-09-10T14:56:48.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:47.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:47.853433+0000) 2022-09-10T14:56:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:48.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:48.226141+0000) 2022-09-10T14:56:49.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:48.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:48.853633+0000) 2022-09-10T14:56:49.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:49 smithi110 ceph-mon[97633]: pgmap v20079: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:49 smithi104 ceph-mon[97044]: pgmap v20079: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:49.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:49.226296+0000) 2022-09-10T14:56:50.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:49.852+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:49.853876+0000) 2022-09-10T14:56:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:50.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:50.226505+0000) 2022-09-10T14:56:51.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:50.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:50.854089+0000) 2022-09-10T14:56:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:51 smithi110 ceph-mon[97633]: pgmap v20080: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:51 smithi104 ceph-mon[97044]: pgmap v20080: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:51.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:51.226681+0000) 2022-09-10T14:56:51.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:56:51.325+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:56:52.228 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:51.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:51.854325+0000) 2022-09-10T14:56:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:52.225+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:52.226868+0000) 2022-09-10T14:56:53.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:52.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:52.854533+0000) 2022-09-10T14:56:53.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:53 smithi110 ceph-mon[97633]: pgmap v20081: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:53 smithi104 ceph-mon[97044]: pgmap v20081: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:53.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:53.227071+0000) 2022-09-10T14:56:54.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:53.853+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:53.854768+0000) 2022-09-10T14:56:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:54.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:54.227350+0000) 2022-09-10T14:56:55.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:54.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:54.855049+0000) 2022-09-10T14:56:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:55 smithi104 ceph-mon[97044]: pgmap v20082: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:55.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:55.226+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:55.227595+0000) 2022-09-10T14:56:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:55 smithi110 ceph-mon[97633]: pgmap v20082: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:56.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:55.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:55.855248+0000) 2022-09-10T14:56:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:56.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:56.227796+0000) 2022-09-10T14:56:57.229 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:56.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:56.855505+0000) 2022-09-10T14:56:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:57 smithi104 ceph-mon[97044]: pgmap v20083: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:57.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:57.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:57.227984+0000) 2022-09-10T14:56:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:57 smithi110 ceph-mon[97633]: pgmap v20083: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:56:58.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:57.854+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:57.855677+0000) 2022-09-10T14:56:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:58.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:58.228232+0000) 2022-09-10T14:56:59.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:58.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:58.855898+0000) 2022-09-10T14:56:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:56:59 smithi104 ceph-mon[97044]: pgmap v20084: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:56:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:56:59.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:59.228437+0000) 2022-09-10T14:56:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:56:59 smithi110 ceph-mon[97633]: pgmap v20084: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:00.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:56:59.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:56:59.856125+0000) 2022-09-10T14:57:00.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:00.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:00.228566+0000) 2022-09-10T14:57:01.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:00.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:00.856322+0000) 2022-09-10T14:57:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:01 smithi104 ceph-mon[97044]: pgmap v20085: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:01.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:01.228793+0000) 2022-09-10T14:57:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:01 smithi110 ceph-mon[97633]: pgmap v20085: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:02.230 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:01.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:01.856516+0000) 2022-09-10T14:57:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:02.227+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:02.228994+0000) 2022-09-10T14:57:03.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:02.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:02.856766+0000) 2022-09-10T14:57:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:03 smithi104 ceph-mon[97044]: pgmap v20086: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:03.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:03.229255+0000) 2022-09-10T14:57:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:03 smithi110 ceph-mon[97633]: pgmap v20086: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:04.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:03.855+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:03.856987+0000) 2022-09-10T14:57:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:04.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:04.229489+0000) 2022-09-10T14:57:05.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:04.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:04.857199+0000) 2022-09-10T14:57:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:05 smithi104 ceph-mon[97044]: pgmap v20087: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:05.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:05.229715+0000) 2022-09-10T14:57:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:05 smithi110 ceph-mon[97633]: pgmap v20087: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:06.231 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:05.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:05.857353+0000) 2022-09-10T14:57:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:06.228+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:06.229979+0000) 2022-09-10T14:57:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:57:06.340+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:57:07.232 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:06.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:06.857569+0000) 2022-09-10T14:57:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:07 smithi104 ceph-mon[97044]: pgmap v20088: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:57:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:07.229+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:07.230240+0000) 2022-09-10T14:57:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:07 smithi110 ceph-mon[97633]: pgmap v20088: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:57:08.239 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:08.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:08.233734+0000) 2022-09-10T14:57:08.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:07.856+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:07.857781+0000) 2022-09-10T14:57:09.235 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:08.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:08.858001+0000) 2022-09-10T14:57:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:09 smithi104 ceph-mon[97044]: pgmap v20089: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:57:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:09 smithi104 ceph-mon[97044]: osdmap e1336: 12 total, 12 up, 12 in 2022-09-10T14:57:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:09.232+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:09.233973+0000) 2022-09-10T14:57:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:09 smithi110 ceph-mon[97633]: pgmap v20089: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:57:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:09 smithi110 ceph-mon[97633]: osdmap e1336: 12 total, 12 up, 12 in 2022-09-10T14:57:10.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:09.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:09.858246+0000) 2022-09-10T14:57:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:10.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:10.234164+0000) 2022-09-10T14:57:11.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:10.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:10.858413+0000) 2022-09-10T14:57:11.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:11 smithi104 ceph-mon[97044]: pgmap v20091: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:57:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:11.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:11.234420+0000) 2022-09-10T14:57:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:11 smithi110 ceph-mon[97633]: pgmap v20091: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:57:12.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:11.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:11.858592+0000) 2022-09-10T14:57:12.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:12.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:12.234716+0000) 2022-09-10T14:57:13.236 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:12.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:12.858765+0000) 2022-09-10T14:57:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:13 smithi110 ceph-mon[97633]: pgmap v20092: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:57:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:13 smithi110 ceph-mon[97633]: osdmap e1337: 12 total, 12 up, 12 in 2022-09-10T14:57:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:13 smithi104 ceph-mon[97044]: pgmap v20092: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:57:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:13 smithi104 ceph-mon[97044]: osdmap e1337: 12 total, 12 up, 12 in 2022-09-10T14:57:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:13.233+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:13.234878+0000) 2022-09-10T14:57:14.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:13.857+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:13.858927+0000) 2022-09-10T14:57:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:14.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:14.235129+0000) 2022-09-10T14:57:15.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:14.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:14.859054+0000) 2022-09-10T14:57:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:15 smithi110 ceph-mon[97633]: pgmap v20094: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:57:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:15 smithi104 ceph-mon[97044]: pgmap v20094: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:57:15.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:15.234+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:15.235313+0000) 2022-09-10T14:57:16.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:15.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:15.859252+0000) 2022-09-10T14:57:16.560 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:16.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:16.235452+0000) 2022-09-10T14:57:17.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:16.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:16.859472+0000) 2022-09-10T14:57:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:17 smithi110 ceph-mon[97633]: pgmap v20095: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:57:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:17 smithi104 ceph-mon[97044]: pgmap v20095: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:57:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:17.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:17.235619+0000) 2022-09-10T14:57:18.237 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:17.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:17.859648+0000) 2022-09-10T14:57:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:18.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:18.235855+0000) 2022-09-10T14:57:19.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:18.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:18.859867+0000) 2022-09-10T14:57:19.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:19 smithi110 ceph-mon[97633]: pgmap v20096: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:57:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:19 smithi104 ceph-mon[97044]: pgmap v20096: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:57:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:19.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:19.236137+0000) 2022-09-10T14:57:20.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:19.858+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:19.860009+0000) 2022-09-10T14:57:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:20.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:20.236350+0000) 2022-09-10T14:57:21.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:20.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:20.860247+0000) 2022-09-10T14:57:21.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:21 smithi110 ceph-mon[97633]: pgmap v20097: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:57:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:21 smithi104 ceph-mon[97044]: pgmap v20097: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:57:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:57:21.349+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:57:21.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:21.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:21.236616+0000) 2022-09-10T14:57:22.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:21.859+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:21.860410+0000) 2022-09-10T14:57:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:22.236+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:22.236844+0000) 2022-09-10T14:57:23.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:22.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:22.860626+0000) 2022-09-10T14:57:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:23 smithi110 ceph-mon[97633]: pgmap v20098: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:57:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:23 smithi104 ceph-mon[97044]: pgmap v20098: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:57:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:23.235+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:23.237031+0000) 2022-09-10T14:57:24.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:23.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:23.860813+0000) 2022-09-10T14:57:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:24.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:24.237258+0000) 2022-09-10T14:57:25.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:24.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:24.860960+0000) 2022-09-10T14:57:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:25 smithi110 ceph-mon[97633]: pgmap v20099: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:57:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:25 smithi104 ceph-mon[97044]: pgmap v20099: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:57:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:25.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:25.237434+0000) 2022-09-10T14:57:26.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:25.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:25.861140+0000) 2022-09-10T14:57:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:26.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:26.237650+0000) 2022-09-10T14:57:27.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:26.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:26.861344+0000) 2022-09-10T14:57:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:27 smithi110 ceph-mon[97633]: pgmap v20100: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:57:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:27 smithi104 ceph-mon[97044]: pgmap v20100: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:57:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:27.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:27.237910+0000) 2022-09-10T14:57:28.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:27.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:27.861533+0000) 2022-09-10T14:57:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:28.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:28.238068+0000) 2022-09-10T14:57:29.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:28.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:28.861711+0000) 2022-09-10T14:57:29.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:29 smithi110 ceph-mon[97633]: pgmap v20101: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:29 smithi104 ceph-mon[97044]: pgmap v20101: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:29.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:29.238233+0000) 2022-09-10T14:57:30.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:29.860+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:29.861959+0000) 2022-09-10T14:57:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:30.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:30.238483+0000) 2022-09-10T14:57:31.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:30.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:30.862196+0000) 2022-09-10T14:57:31.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:31 smithi110 ceph-mon[97633]: pgmap v20102: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:31 smithi104 ceph-mon[97044]: pgmap v20102: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:31.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:31.238727+0000) 2022-09-10T14:57:32.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:31.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:31.862470+0000) 2022-09-10T14:57:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:32.237+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:32.238939+0000) 2022-09-10T14:57:33.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:32.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:32.862711+0000) 2022-09-10T14:57:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:33 smithi110 ceph-mon[97633]: pgmap v20103: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:33 smithi104 ceph-mon[97044]: pgmap v20103: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:33.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:33.239191+0000) 2022-09-10T14:57:34.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:33.861+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:33.862909+0000) 2022-09-10T14:57:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:34.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:34.239350+0000) 2022-09-10T14:57:35.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:34.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:34.863159+0000) 2022-09-10T14:57:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:35 smithi110 ceph-mon[97633]: pgmap v20104: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:35 smithi104 ceph-mon[97044]: pgmap v20104: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:35.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:35.239488+0000) 2022-09-10T14:57:36.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:35.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:35.863323+0000) 2022-09-10T14:57:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:36.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:36.239701+0000) 2022-09-10T14:57:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:57:36.364+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:57:37.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:36.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:36.863594+0000) 2022-09-10T14:57:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:37 smithi110 ceph-mon[97633]: pgmap v20105: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:37 smithi104 ceph-mon[97044]: pgmap v20105: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:37.238+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:37.239927+0000) 2022-09-10T14:57:38.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:37.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:37.863790+0000) 2022-09-10T14:57:38.568 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:38.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:38.240199+0000) 2022-09-10T14:57:39.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:38.862+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:38.864035+0000) 2022-09-10T14:57:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:39 smithi104 ceph-mon[97044]: pgmap v20106: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:39.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:39.240352+0000) 2022-09-10T14:57:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:39 smithi110 ceph-mon[97633]: pgmap v20106: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:40.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:39.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:39.864236+0000) 2022-09-10T14:57:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:40.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:40.240549+0000) 2022-09-10T14:57:41.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:40.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:40.864462+0000) 2022-09-10T14:57:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:41 smithi104 ceph-mon[97044]: pgmap v20107: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:41.239+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:41.240824+0000) 2022-09-10T14:57:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:41 smithi110 ceph-mon[97633]: pgmap v20107: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:42.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:41.863+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:41.864724+0000) 2022-09-10T14:57:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:42.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:42.241068+0000) 2022-09-10T14:57:43.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:42.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:42.864955+0000) 2022-09-10T14:57:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:43 smithi104 ceph-mon[97044]: pgmap v20108: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:43.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:43.241329+0000) 2022-09-10T14:57:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:43 smithi110 ceph-mon[97633]: pgmap v20108: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:44.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:43.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:43.865220+0000) 2022-09-10T14:57:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:44.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:44.241497+0000) 2022-09-10T14:57:45.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:44.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:44.865344+0000) 2022-09-10T14:57:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:45 smithi104 ceph-mon[97044]: pgmap v20109: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:45.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:45.241683+0000) 2022-09-10T14:57:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:45 smithi110 ceph-mon[97633]: pgmap v20109: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:46.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:45.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:45.865489+0000) 2022-09-10T14:57:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:46.240+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:46.241870+0000) 2022-09-10T14:57:47.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:46.864+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:46.865617+0000) 2022-09-10T14:57:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:47 smithi104 ceph-mon[97044]: pgmap v20110: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:57:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:57:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:57:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:57:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:47.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:47.242141+0000) 2022-09-10T14:57:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:47 smithi110 ceph-mon[97633]: pgmap v20110: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:57:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:57:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:57:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:57:48.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:47.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:47.865832+0000) 2022-09-10T14:57:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:48.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:48.242351+0000) 2022-09-10T14:57:49.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:48.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:48.866035+0000) 2022-09-10T14:57:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:49 smithi104 ceph-mon[97044]: pgmap v20111: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:49.241+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:49.242484+0000) 2022-09-10T14:57:49.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:49 smithi110 ceph-mon[97633]: pgmap v20111: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:50.244 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:49.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:49.866254+0000) 2022-09-10T14:57:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:50.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:50.242677+0000) 2022-09-10T14:57:51.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:50.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:50.866499+0000) 2022-09-10T14:57:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:51 smithi104 ceph-mon[97044]: pgmap v20112: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:51.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:51.242930+0000) 2022-09-10T14:57:51.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:57:51.380+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:57:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:51 smithi110 ceph-mon[97633]: pgmap v20112: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:52.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:51.865+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:51.866733+0000) 2022-09-10T14:57:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:52.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:52.243191+0000) 2022-09-10T14:57:53.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:52.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:52.866985+0000) 2022-09-10T14:57:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:53 smithi104 ceph-mon[97044]: pgmap v20113: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:53.564 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:53.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:53.243372+0000) 2022-09-10T14:57:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:53 smithi110 ceph-mon[97633]: pgmap v20113: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:54.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:53.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:53.867160+0000) 2022-09-10T14:57:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:54.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:54.243615+0000) 2022-09-10T14:57:55.245 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:54.866+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:54.867395+0000) 2022-09-10T14:57:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:55 smithi104 ceph-mon[97044]: pgmap v20114: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:55.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:55.243830+0000) 2022-09-10T14:57:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:55 smithi110 ceph-mon[97633]: pgmap v20114: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:56.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:55.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:55.867619+0000) 2022-09-10T14:57:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:56.242+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:56.244015+0000) 2022-09-10T14:57:57.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:56.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:56.867896+0000) 2022-09-10T14:57:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:57 smithi104 ceph-mon[97044]: pgmap v20115: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:57.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:57.244245+0000) 2022-09-10T14:57:57.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:57 smithi110 ceph-mon[97633]: pgmap v20115: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:57:58.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:57.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:57.868113+0000) 2022-09-10T14:57:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:58.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:58.244405+0000) 2022-09-10T14:57:59.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:58.867+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:58.868328+0000) 2022-09-10T14:57:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:57:59 smithi104 ceph-mon[97044]: pgmap v20116: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:57:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:57:59.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:59.244572+0000) 2022-09-10T14:57:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:57:59 smithi110 ceph-mon[97633]: pgmap v20116: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:00.246 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:57:59.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:57:59.868577+0000) 2022-09-10T14:58:00.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:00.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:00.244775+0000) 2022-09-10T14:58:01.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:00.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:00.868743+0000) 2022-09-10T14:58:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:01 smithi104 ceph-mon[97044]: pgmap v20117: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:01.243+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:01.244985+0000) 2022-09-10T14:58:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:01 smithi110 ceph-mon[97633]: pgmap v20117: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:02.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:01.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:01.868958+0000) 2022-09-10T14:58:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:02.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:02.245197+0000) 2022-09-10T14:58:03.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:02.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:02.869193+0000) 2022-09-10T14:58:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:03 smithi104 ceph-mon[97044]: pgmap v20118: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:03.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:03.245414+0000) 2022-09-10T14:58:03.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:03 smithi110 ceph-mon[97633]: pgmap v20118: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:04.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:03.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:03.869440+0000) 2022-09-10T14:58:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:04.244+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:04.245621+0000) 2022-09-10T14:58:05.247 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:04.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:04.869656+0000) 2022-09-10T14:58:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:05 smithi104 ceph-mon[97044]: pgmap v20119: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:05.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:05.245936+0000) 2022-09-10T14:58:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:05 smithi110 ceph-mon[97633]: pgmap v20119: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:06.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:05.868+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:05.869825+0000) 2022-09-10T14:58:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:58:06.387+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:58:06.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:06.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:06.246181+0000) 2022-09-10T14:58:07.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:06.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:06.870042+0000) 2022-09-10T14:58:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:07 smithi104 ceph-mon[97044]: pgmap v20120: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:58:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:07.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:07.246417+0000) 2022-09-10T14:58:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:07 smithi110 ceph-mon[97633]: pgmap v20120: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:58:08.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:07.869+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:07.870249+0000) 2022-09-10T14:58:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:08.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:08.246560+0000) 2022-09-10T14:58:09.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:08.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:08.870437+0000) 2022-09-10T14:58:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:09 smithi104 ceph-mon[97044]: pgmap v20121: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:58:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:09 smithi104 ceph-mon[97044]: osdmap e1338: 12 total, 12 up, 12 in 2022-09-10T14:58:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:09.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:09.246718+0000) 2022-09-10T14:58:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:09 smithi110 ceph-mon[97633]: pgmap v20121: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:58:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:09 smithi110 ceph-mon[97633]: osdmap e1338: 12 total, 12 up, 12 in 2022-09-10T14:58:10.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:09.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:09.870680+0000) 2022-09-10T14:58:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:10 smithi104 ceph-mon[97044]: pgmap v20123: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:58:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:10.245+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:10.246880+0000) 2022-09-10T14:58:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:10 smithi110 ceph-mon[97633]: pgmap v20123: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:58:11.248 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:10.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:10.870855+0000) 2022-09-10T14:58:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:11.246+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:11.247007+0000) 2022-09-10T14:58:12.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:11.870+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:11.871172+0000) 2022-09-10T14:58:12.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:12.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:12.247249+0000) 2022-09-10T14:58:13.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:12.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:12.871434+0000) 2022-09-10T14:58:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:13 smithi110 ceph-mon[97633]: pgmap v20124: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:58:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:13 smithi110 ceph-mon[97633]: osdmap e1339: 12 total, 12 up, 12 in 2022-09-10T14:58:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:13 smithi104 ceph-mon[97044]: pgmap v20124: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:58:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:13 smithi104 ceph-mon[97044]: osdmap e1339: 12 total, 12 up, 12 in 2022-09-10T14:58:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:13.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:13.247449+0000) 2022-09-10T14:58:14.249 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:13.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:13.871614+0000) 2022-09-10T14:58:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:14.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:14.247675+0000) 2022-09-10T14:58:15.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:14.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:14.871876+0000) 2022-09-10T14:58:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:15 smithi110 ceph-mon[97633]: pgmap v20126: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:58:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:15 smithi104 ceph-mon[97044]: pgmap v20126: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:58:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:15.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:15.247894+0000) 2022-09-10T14:58:16.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:15.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:15.872136+0000) 2022-09-10T14:58:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:16.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:16.248078+0000) 2022-09-10T14:58:17.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:16.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:16.872294+0000) 2022-09-10T14:58:17.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:17 smithi110 ceph-mon[97633]: pgmap v20127: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:58:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:17 smithi104 ceph-mon[97044]: pgmap v20127: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:58:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:17.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:17.248311+0000) 2022-09-10T14:58:18.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:17.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:17.872464+0000) 2022-09-10T14:58:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:18.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:18.248479+0000) 2022-09-10T14:58:19.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:18.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:18.872673+0000) 2022-09-10T14:58:19.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:19 smithi110 ceph-mon[97633]: pgmap v20128: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:58:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:19 smithi104 ceph-mon[97044]: pgmap v20128: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:58:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:19.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:19.248719+0000) 2022-09-10T14:58:20.250 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:19.871+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:19.872942+0000) 2022-09-10T14:58:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:20.247+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:20.248918+0000) 2022-09-10T14:58:21.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:20.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:20.873178+0000) 2022-09-10T14:58:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:21 smithi110 ceph-mon[97633]: pgmap v20129: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:58:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:21 smithi104 ceph-mon[97044]: pgmap v20129: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:58:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:21.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:21.249088+0000) 2022-09-10T14:58:21.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:58:21.402+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:58:22.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:21.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:21.873422+0000) 2022-09-10T14:58:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:22.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:22.249279+0000) 2022-09-10T14:58:23.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:22.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:22.873674+0000) 2022-09-10T14:58:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:23 smithi110 ceph-mon[97633]: pgmap v20130: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:58:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:23 smithi104 ceph-mon[97044]: pgmap v20130: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:58:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:23.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:23.249490+0000) 2022-09-10T14:58:24.251 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:23.872+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:23.873948+0000) 2022-09-10T14:58:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:24.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:24.249709+0000) 2022-09-10T14:58:25.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:24.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:24.874179+0000) 2022-09-10T14:58:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:25 smithi110 ceph-mon[97633]: pgmap v20131: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:58:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:25 smithi104 ceph-mon[97044]: pgmap v20131: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:58:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:25.248+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:25.249915+0000) 2022-09-10T14:58:26.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:25.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:25.874341+0000) 2022-09-10T14:58:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:26.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:26.250145+0000) 2022-09-10T14:58:27.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:26.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:26.874579+0000) 2022-09-10T14:58:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:27 smithi110 ceph-mon[97633]: pgmap v20132: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:58:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:27 smithi104 ceph-mon[97044]: pgmap v20132: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:58:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:27.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:27.250370+0000) 2022-09-10T14:58:28.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:27.873+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:27.874798+0000) 2022-09-10T14:58:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:28.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:28.250546+0000) 2022-09-10T14:58:29.252 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:28.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:28.875003+0000) 2022-09-10T14:58:29.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:29 smithi110 ceph-mon[97633]: pgmap v20133: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:29 smithi104 ceph-mon[97044]: pgmap v20133: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:29.249+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:29.250792+0000) 2022-09-10T14:58:30.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:29.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:29.875216+0000) 2022-09-10T14:58:30.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:30.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:30.251008+0000) 2022-09-10T14:58:31.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:30.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:30.875407+0000) 2022-09-10T14:58:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:31 smithi110 ceph-mon[97633]: pgmap v20134: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:31 smithi104 ceph-mon[97044]: pgmap v20134: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:31.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:31.251306+0000) 2022-09-10T14:58:32.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:31.874+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:31.875638+0000) 2022-09-10T14:58:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:32.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:32.251457+0000) 2022-09-10T14:58:33.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:32.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:32.875911+0000) 2022-09-10T14:58:33.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:33 smithi104 ceph-mon[97044]: pgmap v20135: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:33.250+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:33.251679+0000) 2022-09-10T14:58:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:33 smithi110 ceph-mon[97633]: pgmap v20135: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:34.253 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:33.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:33.876125+0000) 2022-09-10T14:58:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:34.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:34.251876+0000) 2022-09-10T14:58:35.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:34.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:34.876292+0000) 2022-09-10T14:58:35.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:35 smithi104 ceph-mon[97044]: pgmap v20136: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:35.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:35.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:35.252128+0000) 2022-09-10T14:58:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:35 smithi110 ceph-mon[97633]: pgmap v20136: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:36.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:35.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:35.876467+0000) 2022-09-10T14:58:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:58:36.412+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:58:36.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:36.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:36.252330+0000) 2022-09-10T14:58:37.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:36.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:36.876686+0000) 2022-09-10T14:58:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:37 smithi104 ceph-mon[97044]: pgmap v20137: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:37.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:37.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:37.252478+0000) 2022-09-10T14:58:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:37 smithi110 ceph-mon[97633]: pgmap v20137: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:38.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:37.875+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:37.876937+0000) 2022-09-10T14:58:38.561 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:38.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:38.252680+0000) 2022-09-10T14:58:39.254 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:38.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:38.877212+0000) 2022-09-10T14:58:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:39 smithi104 ceph-mon[97044]: pgmap v20138: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:39.251+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:39.252846+0000) 2022-09-10T14:58:39.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:39 smithi110 ceph-mon[97633]: pgmap v20138: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:40.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:39.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:39.877349+0000) 2022-09-10T14:58:40.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:40.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:40.253027+0000) 2022-09-10T14:58:41.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:40.876+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:40.877598+0000) 2022-09-10T14:58:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:41 smithi104 ceph-mon[97044]: pgmap v20139: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:41.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:41.253233+0000) 2022-09-10T14:58:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:41 smithi110 ceph-mon[97633]: pgmap v20139: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:58:42.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:41.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:41.877775+0000) 2022-09-10T14:58:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:42.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:42.253397+0000) 2022-09-10T14:58:43.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:42.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:42.877999+0000) 2022-09-10T14:58:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:43 smithi104 ceph-mon[97044]: pgmap v20140: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:43.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:43.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:43.253580+0000) 2022-09-10T14:58:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:43 smithi110 ceph-mon[97633]: pgmap v20140: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:44.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:43.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:43.878259+0000) 2022-09-10T14:58:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:44.252+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:44.253734+0000) 2022-09-10T14:58:45.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:44.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:44.878494+0000) 2022-09-10T14:58:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:45 smithi104 ceph-mon[97044]: pgmap v20141: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:45.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:45.253963+0000) 2022-09-10T14:58:45.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:45 smithi110 ceph-mon[97633]: pgmap v20141: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:46.255 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:45.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:45.878703+0000) 2022-09-10T14:58:46.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:46.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:46.254198+0000) 2022-09-10T14:58:47.256 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:46.877+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:46.878886+0000) 2022-09-10T14:58:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:47 smithi104 ceph-mon[97044]: pgmap v20142: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:58:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:58:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:58:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:58:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:58:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:47.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:47.254505+0000) 2022-09-10T14:58:47.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:47 smithi110 ceph-mon[97633]: pgmap v20142: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:58:47.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:58:47.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:58:47.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:58:47.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:58:48.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:47.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:47.879145+0000) 2022-09-10T14:58:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:48.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:48.254744+0000) 2022-09-10T14:58:49.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:48.878+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:48.879350+0000) 2022-09-10T14:58:49.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:49 smithi104 ceph-mon[97044]: pgmap v20143: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T14:58:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:49.253+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:49.254928+0000) 2022-09-10T14:58:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:49 smithi110 ceph-mon[97633]: pgmap v20143: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T14:58:50.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:49.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:49.879575+0000) 2022-09-10T14:58:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:50.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:50.255152+0000) 2022-09-10T14:58:51.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:50.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:50.879858+0000) 2022-09-10T14:58:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:51 smithi104 ceph-mon[97044]: pgmap v20144: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:58:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:51.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:51.255332+0000) 2022-09-10T14:58:51.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:58:51.414+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:58:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:51 smithi110 ceph-mon[97633]: pgmap v20144: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:58:52.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:51.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:51.880122+0000) 2022-09-10T14:58:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:52.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:52.255543+0000) 2022-09-10T14:58:53.257 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:52.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:52.880357+0000) 2022-09-10T14:58:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:53 smithi104 ceph-mon[97044]: pgmap v20145: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T14:58:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:53.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:53.255781+0000) 2022-09-10T14:58:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:53 smithi110 ceph-mon[97633]: pgmap v20145: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T14:58:54.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:53.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:53.880543+0000) 2022-09-10T14:58:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:54.254+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:54.255932+0000) 2022-09-10T14:58:55.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:54.879+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:54.880785+0000) 2022-09-10T14:58:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:55 smithi104 ceph-mon[97044]: pgmap v20146: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T14:58:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:55.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:55.256192+0000) 2022-09-10T14:58:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:55 smithi110 ceph-mon[97633]: pgmap v20146: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T14:58:56.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:55.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:55.881027+0000) 2022-09-10T14:58:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:56.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:56.256373+0000) 2022-09-10T14:58:57.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:56.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:56.881298+0000) 2022-09-10T14:58:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:57 smithi104 ceph-mon[97044]: pgmap v20147: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:58:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:57.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:57.256608+0000) 2022-09-10T14:58:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:57 smithi110 ceph-mon[97633]: pgmap v20147: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T14:58:58.258 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:57.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:57.881519+0000) 2022-09-10T14:58:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:58.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:58.256799+0000) 2022-09-10T14:58:59.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:58.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:58.881726+0000) 2022-09-10T14:58:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:58:59 smithi104 ceph-mon[97044]: pgmap v20148: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:58:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:58:59.255+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:59.256961+0000) 2022-09-10T14:58:59.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:58:59 smithi110 ceph-mon[97633]: pgmap v20148: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:00.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:58:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:58:59.880+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:58:59.881994+0000) 2022-09-10T14:59:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:00.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:00.257164+0000) 2022-09-10T14:59:01.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:00.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:00.882154+0000) 2022-09-10T14:59:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:01 smithi104 ceph-mon[97044]: pgmap v20149: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:01.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:01.257295+0000) 2022-09-10T14:59:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:01 smithi110 ceph-mon[97633]: pgmap v20149: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:02.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:01.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:01.882341+0000) 2022-09-10T14:59:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:02.256+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:02.257461+0000) 2022-09-10T14:59:03.259 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:02.881+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:02.882482+0000) 2022-09-10T14:59:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:03 smithi104 ceph-mon[97044]: pgmap v20150: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:03.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:03.257713+0000) 2022-09-10T14:59:03.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:03 smithi110 ceph-mon[97633]: pgmap v20150: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:04.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:03.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:03.882690+0000) 2022-09-10T14:59:04.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:04.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:04.257866+0000) 2022-09-10T14:59:05.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:04.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:04.882918+0000) 2022-09-10T14:59:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:05 smithi104 ceph-mon[97044]: pgmap v20151: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:05.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:05.258063+0000) 2022-09-10T14:59:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:05 smithi110 ceph-mon[97633]: pgmap v20151: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:06.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:05.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:05.883134+0000) 2022-09-10T14:59:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:06.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:06.258259+0000) 2022-09-10T14:59:06.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:59:06.431+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:59:07.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:06.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:06.883368+0000) 2022-09-10T14:59:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:07 smithi104 ceph-mon[97044]: pgmap v20152: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:59:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:07.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:07.258477+0000) 2022-09-10T14:59:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:07 smithi110 ceph-mon[97633]: pgmap v20152: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T14:59:08.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:07.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:07.883536+0000) 2022-09-10T14:59:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:08.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:08.258680+0000) 2022-09-10T14:59:09.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:08.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:08.883752+0000) 2022-09-10T14:59:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:09 smithi104 ceph-mon[97044]: pgmap v20153: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:59:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:09 smithi104 ceph-mon[97044]: osdmap e1340: 12 total, 12 up, 12 in 2022-09-10T14:59:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:09.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:09.258869+0000) 2022-09-10T14:59:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:09 smithi110 ceph-mon[97633]: pgmap v20153: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T14:59:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:09 smithi110 ceph-mon[97633]: osdmap e1340: 12 total, 12 up, 12 in 2022-09-10T14:59:10.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:09.882+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:09.883909+0000) 2022-09-10T14:59:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:10 smithi104 ceph-mon[97044]: pgmap v20155: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:59:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:10.257+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:10.259022+0000) 2022-09-10T14:59:10.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:10 smithi110 ceph-mon[97633]: pgmap v20155: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:59:11.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:10.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:10.884170+0000) 2022-09-10T14:59:11.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:11.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:11.259204+0000) 2022-09-10T14:59:12.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:11.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:11.884402+0000) 2022-09-10T14:59:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:12.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:12.259437+0000) 2022-09-10T14:59:13.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:12.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:12.884592+0000) 2022-09-10T14:59:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:13 smithi110 ceph-mon[97633]: pgmap v20156: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:59:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:13 smithi110 ceph-mon[97633]: osdmap e1341: 12 total, 12 up, 12 in 2022-09-10T14:59:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:13 smithi104 ceph-mon[97044]: pgmap v20156: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T14:59:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:13 smithi104 ceph-mon[97044]: osdmap e1341: 12 total, 12 up, 12 in 2022-09-10T14:59:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:13.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:13.259604+0000) 2022-09-10T14:59:14.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:13.883+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:13.884821+0000) 2022-09-10T14:59:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:14.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:14.259759+0000) 2022-09-10T14:59:15.261 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:14.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:14.885092+0000) 2022-09-10T14:59:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:15 smithi110 ceph-mon[97633]: pgmap v20158: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:59:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:15 smithi104 ceph-mon[97044]: pgmap v20158: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T14:59:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:15.258+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:15.259938+0000) 2022-09-10T14:59:16.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:15.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:15.885306+0000) 2022-09-10T14:59:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:16.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:16.260149+0000) 2022-09-10T14:59:17.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:16.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:16.885514+0000) 2022-09-10T14:59:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:17 smithi110 ceph-mon[97633]: pgmap v20159: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:59:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:17 smithi104 ceph-mon[97044]: pgmap v20159: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T14:59:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:17.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:17.260279+0000) 2022-09-10T14:59:18.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:17.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:17.885673+0000) 2022-09-10T14:59:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:18.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:18.260480+0000) 2022-09-10T14:59:19.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:18.884+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:18.885916+0000) 2022-09-10T14:59:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:19 smithi104 ceph-mon[97044]: pgmap v20160: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:59:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:19.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:19.260644+0000) 2022-09-10T14:59:19.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:19 smithi110 ceph-mon[97633]: pgmap v20160: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 211 B/s wr, 10 op/s 2022-09-10T14:59:20.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:19.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:19.886130+0000) 2022-09-10T14:59:20.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:20.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:20.260800+0000) 2022-09-10T14:59:21.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:20.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:20.886310+0000) 2022-09-10T14:59:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:21 smithi104 ceph-mon[97044]: pgmap v20161: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:59:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:59:21.437+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:59:21.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:21.259+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:21.261005+0000) 2022-09-10T14:59:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:21 smithi110 ceph-mon[97633]: pgmap v20161: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:59:22.262 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:21.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:21.886552+0000) 2022-09-10T14:59:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:22.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:22.261236+0000) 2022-09-10T14:59:23.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:22.885+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:22.886752+0000) 2022-09-10T14:59:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:23 smithi104 ceph-mon[97044]: pgmap v20162: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:59:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:23.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:23.261466+0000) 2022-09-10T14:59:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:23 smithi110 ceph-mon[97633]: pgmap v20162: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T14:59:24.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:23.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:23.886928+0000) 2022-09-10T14:59:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:24.260+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:24.261627+0000) 2022-09-10T14:59:25.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:24.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:24.887069+0000) 2022-09-10T14:59:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:25 smithi104 ceph-mon[97044]: pgmap v20163: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:59:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:25.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:25.261859+0000) 2022-09-10T14:59:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:25 smithi110 ceph-mon[97633]: pgmap v20163: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T14:59:26.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:25.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:25.887298+0000) 2022-09-10T14:59:26.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:26.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:26.262038+0000) 2022-09-10T14:59:27.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:26.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:26.887445+0000) 2022-09-10T14:59:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:27 smithi104 ceph-mon[97044]: pgmap v20164: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:59:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:27.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:27.262282+0000) 2022-09-10T14:59:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:27 smithi110 ceph-mon[97633]: pgmap v20164: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T14:59:28.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:27.886+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:27.887639+0000) 2022-09-10T14:59:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:28.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:28.262536+0000) 2022-09-10T14:59:29.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:28.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:28.887880+0000) 2022-09-10T14:59:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:29 smithi104 ceph-mon[97044]: pgmap v20165: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:29.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:29.262693+0000) 2022-09-10T14:59:29.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:29 smithi110 ceph-mon[97633]: pgmap v20165: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:30.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:29.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:29.888141+0000) 2022-09-10T14:59:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:30.261+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:30.262885+0000) 2022-09-10T14:59:31.264 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:30.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:30.888305+0000) 2022-09-10T14:59:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:31 smithi104 ceph-mon[97044]: pgmap v20166: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:31.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:31.263049+0000) 2022-09-10T14:59:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:31 smithi110 ceph-mon[97633]: pgmap v20166: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:32.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:31.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:31.888528+0000) 2022-09-10T14:59:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:32.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:32.263273+0000) 2022-09-10T14:59:33.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:32.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:32.888765+0000) 2022-09-10T14:59:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:33 smithi104 ceph-mon[97044]: pgmap v20167: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:33.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:33.263485+0000) 2022-09-10T14:59:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:33 smithi110 ceph-mon[97633]: pgmap v20167: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:34.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:33.887+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:33.889014+0000) 2022-09-10T14:59:34.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:34.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:34.263665+0000) 2022-09-10T14:59:35.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:34.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:34.889285+0000) 2022-09-10T14:59:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:35 smithi104 ceph-mon[97044]: pgmap v20168: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:35.262+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:35.263912+0000) 2022-09-10T14:59:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:35 smithi110 ceph-mon[97633]: pgmap v20168: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:36.265 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:35.888+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:35.889500+0000) 2022-09-10T14:59:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:36.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:36.264176+0000) 2022-09-10T14:59:36.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:59:36.451+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:59:37.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:36.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:36.889698+0000) 2022-09-10T14:59:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:37 smithi104 ceph-mon[97044]: pgmap v20169: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:37.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:37.264357+0000) 2022-09-10T14:59:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:37 smithi110 ceph-mon[97633]: pgmap v20169: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:38.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:37.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:37.889901+0000) 2022-09-10T14:59:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:38.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:38.264548+0000) 2022-09-10T14:59:39.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:38.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:38.890185+0000) 2022-09-10T14:59:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:39 smithi104 ceph-mon[97044]: pgmap v20170: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:39.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:39.264708+0000) 2022-09-10T14:59:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:39 smithi110 ceph-mon[97633]: pgmap v20170: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:40.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:39.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:39.890369+0000) 2022-09-10T14:59:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:40.263+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:40.264891+0000) 2022-09-10T14:59:41.266 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:40.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:40.890538+0000) 2022-09-10T14:59:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:41 smithi104 ceph-mon[97044]: pgmap v20171: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:41.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:41.265127+0000) 2022-09-10T14:59:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:41 smithi110 ceph-mon[97633]: pgmap v20171: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:42.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:41.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:41.890767+0000) 2022-09-10T14:59:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:42.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:42.265347+0000) 2022-09-10T14:59:43.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:42.889+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:42.891019+0000) 2022-09-10T14:59:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:43 smithi104 ceph-mon[97044]: pgmap v20172: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:59:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:43.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:43.265532+0000) 2022-09-10T14:59:43.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:43 smithi110 ceph-mon[97633]: pgmap v20172: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:59:44.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:43.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:43.891244+0000) 2022-09-10T14:59:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:44.264+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:44.265786+0000) 2022-09-10T14:59:45.267 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:44.890+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:44.891444+0000) 2022-09-10T14:59:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:45 smithi104 ceph-mon[97044]: pgmap v20173: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:59:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:45.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:45.266046+0000) 2022-09-10T14:59:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:45 smithi110 ceph-mon[97633]: pgmap v20173: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:59:46.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:45.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:45.891666+0000) 2022-09-10T14:59:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:46.265+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:46.266150+0000) 2022-09-10T14:59:47.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:46.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:46.891871+0000) 2022-09-10T14:59:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:47 smithi104 ceph-mon[97044]: pgmap v20174: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:59:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:59:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:59:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:59:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:47.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:47.266390+0000) 2022-09-10T14:59:47.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:47 smithi110 ceph-mon[97633]: pgmap v20174: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:59:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:59:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T14:59:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T14:59:48.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:47.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:47.892177+0000) 2022-09-10T14:59:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:48.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:48.266606+0000) 2022-09-10T14:59:49.268 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:48.891+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:48.892395+0000) 2022-09-10T14:59:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:49 smithi104 ceph-mon[97044]: pgmap v20175: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:59:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:49.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:49.266808+0000) 2022-09-10T14:59:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:49 smithi110 ceph-mon[97633]: pgmap v20175: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T14:59:50.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:49.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:49.892603+0000) 2022-09-10T14:59:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:50.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:50.266985+0000) 2022-09-10T14:59:51.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:50.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:50.892830+0000) 2022-09-10T14:59:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:51 smithi104 ceph-mon[97044]: pgmap v20176: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:51.266+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:51.267197+0000) 2022-09-10T14:59:51.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 14:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T14:59:51.462+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T14:59:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:51 smithi110 ceph-mon[97633]: pgmap v20176: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:52.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:51.892+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:51.893087+0000) 2022-09-10T14:59:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:52.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:52.267359+0000) 2022-09-10T14:59:53.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:52.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:52.893271+0000) 2022-09-10T14:59:53.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:53 smithi104 ceph-mon[97044]: pgmap v20177: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:53.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:53.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:53.267541+0000) 2022-09-10T14:59:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:53 smithi110 ceph-mon[97633]: pgmap v20177: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:54.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:53.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:53.893467+0000) 2022-09-10T14:59:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:54.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:54.267725+0000) 2022-09-10T14:59:55.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:54.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:54.893709+0000) 2022-09-10T14:59:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:55 smithi104 ceph-mon[97044]: pgmap v20178: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:55.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:55.267888+0000) 2022-09-10T14:59:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:55 smithi110 ceph-mon[97633]: pgmap v20178: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:56.269 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:55.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:55.893959+0000) 2022-09-10T14:59:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:56.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:56.268049+0000) 2022-09-10T14:59:57.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:56.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:56.894216+0000) 2022-09-10T14:59:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:57 smithi104 ceph-mon[97044]: pgmap v20179: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:57.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:57.268253+0000) 2022-09-10T14:59:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:57 smithi110 ceph-mon[97633]: pgmap v20179: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T14:59:58.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:57.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:57.894402+0000) 2022-09-10T14:59:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:58.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:58.268435+0000) 2022-09-10T14:59:59.270 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:58.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:58.894628+0000) 2022-09-10T14:59:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 14:59:59 smithi104 ceph-mon[97044]: pgmap v20180: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T14:59:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 14:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T14:59:59.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:59.268704+0000) 2022-09-10T14:59:59.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 14:59:59 smithi110 ceph-mon[97633]: pgmap v20180: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:00.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 14:59:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T14:59:59.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T13:59:59.894870+0000) 2022-09-10T15:00:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:00:00.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:00.267+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:00.268872+0000) 2022-09-10T15:00:00.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:00:01.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:00.893+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:00.895012+0000) 2022-09-10T15:00:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:01 smithi104 ceph-mon[97044]: pgmap v20181: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:01.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:01.269007+0000) 2022-09-10T15:00:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:01 smithi110 ceph-mon[97633]: pgmap v20181: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:02.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:01.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:01.895209+0000) 2022-09-10T15:00:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:02.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:02.269227+0000) 2022-09-10T15:00:03.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:02.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:02.895454+0000) 2022-09-10T15:00:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:03 smithi104 ceph-mon[97044]: pgmap v20182: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:03.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:03.269437+0000) 2022-09-10T15:00:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:03 smithi110 ceph-mon[97633]: pgmap v20182: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:04.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:03.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:03.895732+0000) 2022-09-10T15:00:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:04.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:04.269600+0000) 2022-09-10T15:00:05.271 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:04.894+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:04.895977+0000) 2022-09-10T15:00:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:05 smithi104 ceph-mon[97044]: pgmap v20183: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:05.268+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:05.269840+0000) 2022-09-10T15:00:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:05 smithi110 ceph-mon[97633]: pgmap v20183: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:06.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:05.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:05.896231+0000) 2022-09-10T15:00:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:06.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:06.270082+0000) 2022-09-10T15:00:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:00:06.477+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:00:07.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:06.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:06.896424+0000) 2022-09-10T15:00:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:07 smithi104 ceph-mon[97044]: pgmap v20184: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:00:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:07.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:07.270301+0000) 2022-09-10T15:00:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:07 smithi110 ceph-mon[97633]: pgmap v20184: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:00:08.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:07.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:07.896586+0000) 2022-09-10T15:00:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:08.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:08.270435+0000) 2022-09-10T15:00:09.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:08.895+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:08.896816+0000) 2022-09-10T15:00:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:09 smithi104 ceph-mon[97044]: pgmap v20185: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:00:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:09 smithi104 ceph-mon[97044]: osdmap e1342: 12 total, 12 up, 12 in 2022-09-10T15:00:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:09.269+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:09.270694+0000) 2022-09-10T15:00:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:09 smithi110 ceph-mon[97633]: pgmap v20185: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:00:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:09 smithi110 ceph-mon[97633]: osdmap e1342: 12 total, 12 up, 12 in 2022-09-10T15:00:10.272 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:09.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:09.897029+0000) 2022-09-10T15:00:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:10.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:10.270955+0000) 2022-09-10T15:00:11.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:10.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:10.897303+0000) 2022-09-10T15:00:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:11 smithi104 ceph-mon[97044]: pgmap v20187: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:00:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:11.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:11.271207+0000) 2022-09-10T15:00:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:11 smithi110 ceph-mon[97633]: pgmap v20187: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:00:12.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:11.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:11.897533+0000) 2022-09-10T15:00:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:12.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:12.271381+0000) 2022-09-10T15:00:13.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:12.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:12.897790+0000) 2022-09-10T15:00:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:13 smithi104 ceph-mon[97044]: pgmap v20188: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:00:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:13 smithi104 ceph-mon[97044]: osdmap e1343: 12 total, 12 up, 12 in 2022-09-10T15:00:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:13.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:13.271635+0000) 2022-09-10T15:00:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:13 smithi110 ceph-mon[97633]: pgmap v20188: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:00:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:13 smithi110 ceph-mon[97633]: osdmap e1343: 12 total, 12 up, 12 in 2022-09-10T15:00:14.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:13.896+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:13.898026+0000) 2022-09-10T15:00:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:14.270+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:14.271809+0000) 2022-09-10T15:00:15.273 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:14.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:14.898234+0000) 2022-09-10T15:00:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:15 smithi104 ceph-mon[97044]: pgmap v20190: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:00:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:15.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:15.272009+0000) 2022-09-10T15:00:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:15 smithi110 ceph-mon[97633]: pgmap v20190: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:00:16.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:15.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:15.898389+0000) 2022-09-10T15:00:16.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:16.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:16.272230+0000) 2022-09-10T15:00:17.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:16.897+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:16.898615+0000) 2022-09-10T15:00:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:17 smithi104 ceph-mon[97044]: pgmap v20191: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:00:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:17.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:17.272430+0000) 2022-09-10T15:00:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:17 smithi110 ceph-mon[97633]: pgmap v20191: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:00:18.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:17.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:17.898848+0000) 2022-09-10T15:00:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:18.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:18.272607+0000) 2022-09-10T15:00:19.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:18.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:18.899014+0000) 2022-09-10T15:00:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:19 smithi104 ceph-mon[97044]: pgmap v20192: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:00:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:19.271+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:19.272765+0000) 2022-09-10T15:00:19.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:19 smithi110 ceph-mon[97633]: pgmap v20192: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:00:20.274 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:19.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:19.899187+0000) 2022-09-10T15:00:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:20.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:20.272938+0000) 2022-09-10T15:00:21.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:20.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:20.899399+0000) 2022-09-10T15:00:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:21 smithi104 ceph-mon[97044]: pgmap v20193: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:00:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:00:21.481+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:00:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:21.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:21.273194+0000) 2022-09-10T15:00:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:21 smithi110 ceph-mon[97633]: pgmap v20193: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:00:22.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:21.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:21.899658+0000) 2022-09-10T15:00:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:22.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:22.273390+0000) 2022-09-10T15:00:23.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:22.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:22.899863+0000) 2022-09-10T15:00:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:23 smithi104 ceph-mon[97044]: pgmap v20194: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:00:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:23.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:23.273630+0000) 2022-09-10T15:00:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:23 smithi110 ceph-mon[97633]: pgmap v20194: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:00:24.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:23.898+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:23.900053+0000) 2022-09-10T15:00:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:24.272+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:24.273823+0000) 2022-09-10T15:00:25.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:24.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:24.900267+0000) 2022-09-10T15:00:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:25 smithi104 ceph-mon[97044]: pgmap v20195: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:00:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:25.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:25.273967+0000) 2022-09-10T15:00:25.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:25 smithi110 ceph-mon[97633]: pgmap v20195: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:00:26.275 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:25.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:25.900414+0000) 2022-09-10T15:00:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:26.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:26.274160+0000) 2022-09-10T15:00:27.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:26.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:26.900627+0000) 2022-09-10T15:00:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:27 smithi104 ceph-mon[97044]: pgmap v20196: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:00:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:27.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:27.274338+0000) 2022-09-10T15:00:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:27 smithi110 ceph-mon[97633]: pgmap v20196: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:00:28.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:27.899+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:27.900859+0000) 2022-09-10T15:00:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:28.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:28.274602+0000) 2022-09-10T15:00:29.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:28.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:28.901049+0000) 2022-09-10T15:00:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:29 smithi104 ceph-mon[97044]: pgmap v20197: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:29.273+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:29.274760+0000) 2022-09-10T15:00:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:29 smithi110 ceph-mon[97633]: pgmap v20197: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:30.276 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:29.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:29.901266+0000) 2022-09-10T15:00:30.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:30.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:30.274999+0000) 2022-09-10T15:00:31.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:30.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:30.901507+0000) 2022-09-10T15:00:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:31 smithi104 ceph-mon[97044]: pgmap v20198: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:31.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:31.275239+0000) 2022-09-10T15:00:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:31 smithi110 ceph-mon[97633]: pgmap v20198: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:32.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:31.900+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:31.901763+0000) 2022-09-10T15:00:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:32.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:32.275457+0000) 2022-09-10T15:00:33.277 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:32.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:32.901976+0000) 2022-09-10T15:00:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:33 smithi104 ceph-mon[97044]: pgmap v20199: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:33.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:33.275642+0000) 2022-09-10T15:00:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:33 smithi110 ceph-mon[97633]: pgmap v20199: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:34.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:33.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:33.902221+0000) 2022-09-10T15:00:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:34.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:34.275869+0000) 2022-09-10T15:00:35.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:34.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:34.902473+0000) 2022-09-10T15:00:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:35 smithi104 ceph-mon[97044]: pgmap v20200: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:35.274+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:35.276028+0000) 2022-09-10T15:00:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:35 smithi110 ceph-mon[97633]: pgmap v20200: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:36.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:35.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:35.902715+0000) 2022-09-10T15:00:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:36.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:36.276264+0000) 2022-09-10T15:00:36.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:00:36.496+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:00:37.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:36.901+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:36.902984+0000) 2022-09-10T15:00:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:37 smithi104 ceph-mon[97044]: pgmap v20201: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:37.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:37.276535+0000) 2022-09-10T15:00:37.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:37 smithi110 ceph-mon[97633]: pgmap v20201: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:38.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:37.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:37.903212+0000) 2022-09-10T15:00:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:38.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:38.276699+0000) 2022-09-10T15:00:39.278 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:38.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:38.903391+0000) 2022-09-10T15:00:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:39 smithi104 ceph-mon[97044]: pgmap v20202: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:39.275+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:39.276951+0000) 2022-09-10T15:00:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:39 smithi110 ceph-mon[97633]: pgmap v20202: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:40.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:39.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:39.903608+0000) 2022-09-10T15:00:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:40.276+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:40.277117+0000) 2022-09-10T15:00:41.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:40.902+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:40.903803+0000) 2022-09-10T15:00:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:41 smithi104 ceph-mon[97044]: pgmap v20203: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:41.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:41.277282+0000) 2022-09-10T15:00:41.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:41 smithi110 ceph-mon[97633]: pgmap v20203: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:42.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:41.903+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:41.904054+0000) 2022-09-10T15:00:42.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:42.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:42.277489+0000) 2022-09-10T15:00:43.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:42.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:42.904287+0000) 2022-09-10T15:00:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:43 smithi104 ceph-mon[97044]: pgmap v20204: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:43.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:43.277729+0000) 2022-09-10T15:00:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:43 smithi110 ceph-mon[97633]: pgmap v20204: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:44.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:43.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:43.904503+0000) 2022-09-10T15:00:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:44.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:44.277975+0000) 2022-09-10T15:00:45.279 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:44.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:44.904726+0000) 2022-09-10T15:00:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:45 smithi104 ceph-mon[97044]: pgmap v20205: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:45.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:45.278209+0000) 2022-09-10T15:00:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:45 smithi110 ceph-mon[97633]: pgmap v20205: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:46.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:45.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:45.904913+0000) 2022-09-10T15:00:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:46.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:46.278448+0000) 2022-09-10T15:00:47.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:46.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:46.905178+0000) 2022-09-10T15:00:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:47 smithi104 ceph-mon[97044]: pgmap v20206: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:00:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:00:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:00:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:00:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:47.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:47.278734+0000) 2022-09-10T15:00:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:47 smithi110 ceph-mon[97633]: pgmap v20206: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:00:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:00:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:00:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:00:48.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:47.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:47.905405+0000) 2022-09-10T15:00:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:48.277+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:48.278959+0000) 2022-09-10T15:00:49.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:48.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:48.905542+0000) 2022-09-10T15:00:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:49 smithi104 ceph-mon[97044]: pgmap v20207: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:49.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:49.279138+0000) 2022-09-10T15:00:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:49 smithi110 ceph-mon[97633]: pgmap v20207: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:50.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:49.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:49.905786+0000) 2022-09-10T15:00:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:50.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:50.279367+0000) 2022-09-10T15:00:51.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:50.904+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:50.905991+0000) 2022-09-10T15:00:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:51 smithi104 ceph-mon[97044]: pgmap v20208: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:51.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:51.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:51.279601+0000) 2022-09-10T15:00:51.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:00:51.499+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:00:51.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:51 smithi110 ceph-mon[97633]: pgmap v20208: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:52.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:51.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:51.906192+0000) 2022-09-10T15:00:52.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:52.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:52.279802+0000) 2022-09-10T15:00:53.281 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:52.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:52.906422+0000) 2022-09-10T15:00:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:53 smithi104 ceph-mon[97044]: pgmap v20209: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:53.278+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:53.279975+0000) 2022-09-10T15:00:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:53 smithi110 ceph-mon[97633]: pgmap v20209: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:54.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:53.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:53.906677+0000) 2022-09-10T15:00:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:54 smithi104 ceph-mon[97044]: pgmap v20210: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:54.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:54.280131+0000) 2022-09-10T15:00:54.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:54 smithi110 ceph-mon[97633]: pgmap v20210: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:55.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:54.905+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:54.906865+0000) 2022-09-10T15:00:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:55.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:55.280353+0000) 2022-09-10T15:00:56.282 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:55.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:55.907163+0000) 2022-09-10T15:00:56.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:56.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:56.280578+0000) 2022-09-10T15:00:57.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:56.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:56.907378+0000) 2022-09-10T15:00:57.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:56 smithi104 ceph-mon[97044]: pgmap v20211: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:57.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:56 smithi110 ceph-mon[97633]: pgmap v20211: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:00:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:57.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:57.280830+0000) 2022-09-10T15:00:58.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:57.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:57.907548+0000) 2022-09-10T15:00:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:58.279+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:58.281044+0000) 2022-09-10T15:00:59.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:58.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:58.907736+0000) 2022-09-10T15:00:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:00:59 smithi104 ceph-mon[97044]: pgmap v20212: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:00:59.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:00:59.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:59.281226+0000) 2022-09-10T15:00:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:00:59 smithi110 ceph-mon[97633]: pgmap v20212: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:00.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:00:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:00:59.906+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:00:59.907945+0000) 2022-09-10T15:01:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:00.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:00.281442+0000) 2022-09-10T15:01:01.280 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:00.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:00.908171+0000) 2022-09-10T15:01:01.497 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:01 smithi110 ceph-mon[97633]: pgmap v20213: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:01 smithi104 ceph-mon[97044]: pgmap v20213: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:01.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:01.281582+0000) 2022-09-10T15:01:02.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:01.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:01.908361+0000) 2022-09-10T15:01:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:02.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:02.281794+0000) 2022-09-10T15:01:03.283 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:02.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:02.908561+0000) 2022-09-10T15:01:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:03 smithi104 ceph-mon[97044]: pgmap v20214: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:03.280+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:03.282014+0000) 2022-09-10T15:01:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:03 smithi110 ceph-mon[97633]: pgmap v20214: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:04.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:03.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:03.908782+0000) 2022-09-10T15:01:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:04.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:04.282215+0000) 2022-09-10T15:01:05.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:04.907+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:04.909046+0000) 2022-09-10T15:01:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:05 smithi104 ceph-mon[97044]: pgmap v20215: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:05.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:05.282506+0000) 2022-09-10T15:01:05.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:05 smithi110 ceph-mon[97633]: pgmap v20215: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:06.284 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:05.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:05.909246+0000) 2022-09-10T15:01:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:06.281+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:06.282696+0000) 2022-09-10T15:01:06.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:01:06.515+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:01:07.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:06.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:06.909375+0000) 2022-09-10T15:01:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:07 smithi104 ceph-mon[97044]: pgmap v20216: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:01:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:07.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:07.282952+0000) 2022-09-10T15:01:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:07 smithi110 ceph-mon[97633]: pgmap v20216: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:01:08.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:07.908+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:07.909574+0000) 2022-09-10T15:01:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:08.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:08.283226+0000) 2022-09-10T15:01:09.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:08.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:08.909805+0000) 2022-09-10T15:01:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:09 smithi104 ceph-mon[97044]: pgmap v20217: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:01:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:09 smithi104 ceph-mon[97044]: osdmap e1344: 12 total, 12 up, 12 in 2022-09-10T15:01:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:09.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:09.283425+0000) 2022-09-10T15:01:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:09 smithi110 ceph-mon[97633]: pgmap v20217: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:01:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:09 smithi110 ceph-mon[97633]: osdmap e1344: 12 total, 12 up, 12 in 2022-09-10T15:01:10.285 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:09.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:09.909996+0000) 2022-09-10T15:01:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:10.282+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:10.283613+0000) 2022-09-10T15:01:11.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:10.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:10.910228+0000) 2022-09-10T15:01:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:11 smithi104 ceph-mon[97044]: pgmap v20219: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:01:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:11.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:11.283864+0000) 2022-09-10T15:01:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:11 smithi110 ceph-mon[97633]: pgmap v20219: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:01:12.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:11.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:11.910436+0000) 2022-09-10T15:01:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:12.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:12.284147+0000) 2022-09-10T15:01:13.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:12.909+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:12.910627+0000) 2022-09-10T15:01:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:13 smithi104 ceph-mon[97044]: pgmap v20220: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:01:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:13 smithi104 ceph-mon[97044]: osdmap e1345: 12 total, 12 up, 12 in 2022-09-10T15:01:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:13.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:13.284296+0000) 2022-09-10T15:01:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:13 smithi110 ceph-mon[97633]: pgmap v20220: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:01:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:13 smithi110 ceph-mon[97633]: osdmap e1345: 12 total, 12 up, 12 in 2022-09-10T15:01:14.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:13.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:13.910791+0000) 2022-09-10T15:01:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:14.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:14.284418+0000) 2022-09-10T15:01:15.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:14.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:14.911018+0000) 2022-09-10T15:01:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:15 smithi104 ceph-mon[97044]: pgmap v20222: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:01:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:15.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:15.284659+0000) 2022-09-10T15:01:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:15 smithi110 ceph-mon[97633]: pgmap v20222: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:01:16.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:15.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:15.911168+0000) 2022-09-10T15:01:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:16.283+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:16.284873+0000) 2022-09-10T15:01:17.286 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:16.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:16.911361+0000) 2022-09-10T15:01:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:17 smithi104 ceph-mon[97044]: pgmap v20223: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:01:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:17.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:17.285113+0000) 2022-09-10T15:01:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:17 smithi110 ceph-mon[97633]: pgmap v20223: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:01:18.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:17.910+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:17.911539+0000) 2022-09-10T15:01:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:18.284+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:18.285278+0000) 2022-09-10T15:01:19.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:18.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:18.911793+0000) 2022-09-10T15:01:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:19 smithi104 ceph-mon[97044]: pgmap v20224: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:01:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:19.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:19.285457+0000) 2022-09-10T15:01:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:19 smithi110 ceph-mon[97633]: pgmap v20224: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:01:20.287 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:19.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:19.912058+0000) 2022-09-10T15:01:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:20.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:20.285659+0000) 2022-09-10T15:01:21.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:20.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:20.912240+0000) 2022-09-10T15:01:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:21 smithi104 ceph-mon[97044]: pgmap v20225: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:01:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:21.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:21.285865+0000) 2022-09-10T15:01:21.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:01:21.530+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:01:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:21 smithi110 ceph-mon[97633]: pgmap v20225: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:01:22.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:21.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:21.912480+0000) 2022-09-10T15:01:22.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:22.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:22.286108+0000) 2022-09-10T15:01:23.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:22.911+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:22.912727+0000) 2022-09-10T15:01:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:23 smithi104 ceph-mon[97044]: pgmap v20226: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:01:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:23.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:23.286327+0000) 2022-09-10T15:01:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:23 smithi110 ceph-mon[97633]: pgmap v20226: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:01:24.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:23.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:23.912894+0000) 2022-09-10T15:01:24.558 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:24.285+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:24.286494+0000) 2022-09-10T15:01:25.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:24.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:24.913158+0000) 2022-09-10T15:01:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:25 smithi104 ceph-mon[97044]: pgmap v20227: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:01:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:25.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:25.286722+0000) 2022-09-10T15:01:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:25 smithi110 ceph-mon[97633]: pgmap v20227: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:01:26.288 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:25.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:25.913360+0000) 2022-09-10T15:01:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:26.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:26.286863+0000) 2022-09-10T15:01:27.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:26.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:26.913552+0000) 2022-09-10T15:01:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:27 smithi104 ceph-mon[97044]: pgmap v20228: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:01:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:27.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:27.287090+0000) 2022-09-10T15:01:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:27 smithi110 ceph-mon[97633]: pgmap v20228: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:01:28.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:27.912+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:27.913772+0000) 2022-09-10T15:01:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:28.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:28.287339+0000) 2022-09-10T15:01:29.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:28.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:28.913973+0000) 2022-09-10T15:01:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:29 smithi104 ceph-mon[97044]: pgmap v20229: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:29.286+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:29.287548+0000) 2022-09-10T15:01:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:29 smithi110 ceph-mon[97633]: pgmap v20229: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:30.289 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:29.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:29.914232+0000) 2022-09-10T15:01:30.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:30.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:30.287774+0000) 2022-09-10T15:01:31.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:30.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:30.914459+0000) 2022-09-10T15:01:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:31 smithi104 ceph-mon[97044]: pgmap v20230: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:31.287+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:31.288039+0000) 2022-09-10T15:01:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:31 smithi110 ceph-mon[97633]: pgmap v20230: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:32.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:31.914+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:31.914634+0000) 2022-09-10T15:01:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:32.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:32.288259+0000) 2022-09-10T15:01:33.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:32.913+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:32.914857+0000) 2022-09-10T15:01:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:33 smithi104 ceph-mon[97044]: pgmap v20231: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:33.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:33.288501+0000) 2022-09-10T15:01:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:33 smithi110 ceph-mon[97633]: pgmap v20231: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:34.290 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:33.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:33.915141+0000) 2022-09-10T15:01:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:34.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:34.288666+0000) 2022-09-10T15:01:35.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:34.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:34.915349+0000) 2022-09-10T15:01:35.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:35 smithi104 ceph-mon[97044]: pgmap v20232: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:35.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:35.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:35.288884+0000) 2022-09-10T15:01:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:35 smithi110 ceph-mon[97633]: pgmap v20232: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:36.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:35.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:35.915565+0000) 2022-09-10T15:01:36.548 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:36.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:36.289089+0000) 2022-09-10T15:01:36.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:01:36.545+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:01:37.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:36.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:36.915807+0000) 2022-09-10T15:01:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:37 smithi104 ceph-mon[97044]: pgmap v20233: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:37.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:37.289279+0000) 2022-09-10T15:01:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:37 smithi110 ceph-mon[97633]: pgmap v20233: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:38.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:37.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:37.916038+0000) 2022-09-10T15:01:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:38.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:38.289427+0000) 2022-09-10T15:01:39.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:38.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:38.916263+0000) 2022-09-10T15:01:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:39 smithi104 ceph-mon[97044]: pgmap v20234: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:39.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:39.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:39.289624+0000) 2022-09-10T15:01:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:39 smithi110 ceph-mon[97633]: pgmap v20234: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:40.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:39.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:39.916474+0000) 2022-09-10T15:01:40.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:40.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:40.289786+0000) 2022-09-10T15:01:41.291 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:40.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:40.916667+0000) 2022-09-10T15:01:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:41 smithi104 ceph-mon[97044]: pgmap v20235: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:41.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:41.288+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:41.290014+0000) 2022-09-10T15:01:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:41 smithi110 ceph-mon[97633]: pgmap v20235: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:42.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:41.915+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:41.916899+0000) 2022-09-10T15:01:42.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:42.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:42.290284+0000) 2022-09-10T15:01:43.292 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:42.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:42.917205+0000) 2022-09-10T15:01:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:43 smithi104 ceph-mon[97044]: pgmap v20236: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:43.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:43.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:43.290504+0000) 2022-09-10T15:01:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:43 smithi110 ceph-mon[97633]: pgmap v20236: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:44.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:43.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:43.917378+0000) 2022-09-10T15:01:44.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:44.289+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:44.290766+0000) 2022-09-10T15:01:45.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:44.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:44.917584+0000) 2022-09-10T15:01:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:45 smithi104 ceph-mon[97044]: pgmap v20237: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:45.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:45.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:45.291027+0000) 2022-09-10T15:01:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:45 smithi110 ceph-mon[97633]: pgmap v20237: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:46.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:45.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:45.917782+0000) 2022-09-10T15:01:46.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:46.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:46.291268+0000) 2022-09-10T15:01:47.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:46.916+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:46.918033+0000) 2022-09-10T15:01:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:47 smithi104 ceph-mon[97044]: pgmap v20238: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:01:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:01:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:01:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:01:47.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:47.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:47.291510+0000) 2022-09-10T15:01:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:47 smithi110 ceph-mon[97633]: pgmap v20238: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:01:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:01:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:01:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:01:48.293 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:47.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:47.918210+0000) 2022-09-10T15:01:48.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:48.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:48.291744+0000) 2022-09-10T15:01:49.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:48.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:48.918431+0000) 2022-09-10T15:01:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:49 smithi104 ceph-mon[97044]: pgmap v20239: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:49.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:49.290+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:49.291930+0000) 2022-09-10T15:01:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:49 smithi110 ceph-mon[97633]: pgmap v20239: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:50.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:49.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:49.918715+0000) 2022-09-10T15:01:50.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:50.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:50.292184+0000) 2022-09-10T15:01:51.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:50.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:50.918877+0000) 2022-09-10T15:01:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:51 smithi104 ceph-mon[97044]: pgmap v20240: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:51.551 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:51.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:51.292425+0000) 2022-09-10T15:01:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:51 smithi110 ceph-mon[97633]: pgmap v20240: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:51.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:01:51.549+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:01:52.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:51.917+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:51.919084+0000) 2022-09-10T15:01:52.557 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:52.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:52.292667+0000) 2022-09-10T15:01:53.294 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:52.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:52.919302+0000) 2022-09-10T15:01:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:53 smithi104 ceph-mon[97044]: pgmap v20241: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:53.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:53.291+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:53.292943+0000) 2022-09-10T15:01:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:53 smithi110 ceph-mon[97633]: pgmap v20241: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:54.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:53.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:53.919503+0000) 2022-09-10T15:01:54.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:54.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:54.293157+0000) 2022-09-10T15:01:55.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:54.918+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:54.919743+0000) 2022-09-10T15:01:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:55 smithi104 ceph-mon[97044]: pgmap v20242: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:55.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:55.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:55.293319+0000) 2022-09-10T15:01:55.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:55 smithi110 ceph-mon[97633]: pgmap v20242: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:56.295 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:55.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:55.920003+0000) 2022-09-10T15:01:56.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:56.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:56.293497+0000) 2022-09-10T15:01:57.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:56.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:56.920218+0000) 2022-09-10T15:01:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:57 smithi104 ceph-mon[97044]: pgmap v20243: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:57.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:57.292+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:57.293724+0000) 2022-09-10T15:01:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:57 smithi110 ceph-mon[97633]: pgmap v20243: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:01:58.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:57.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:57.920365+0000) 2022-09-10T15:01:58.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:58.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:58.293965+0000) 2022-09-10T15:01:59.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:58.919+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:58.920621+0000) 2022-09-10T15:01:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:01:59 smithi104 ceph-mon[97044]: pgmap v20244: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:01:59.556 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:01:59.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:59.294205+0000) 2022-09-10T15:01:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:01:59 smithi110 ceph-mon[97633]: pgmap v20244: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:00.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:01:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:01:59.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:01:59.920822+0000) 2022-09-10T15:02:00.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:00.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:00.294339+0000) 2022-09-10T15:02:01.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:00.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:00.921054+0000) 2022-09-10T15:02:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:01 smithi104 ceph-mon[97044]: pgmap v20245: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:01.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:01.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:01.294494+0000) 2022-09-10T15:02:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:01 smithi110 ceph-mon[97633]: pgmap v20245: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:02.296 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:01.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:01.921235+0000) 2022-09-10T15:02:02.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:02.293+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:02.294777+0000) 2022-09-10T15:02:03.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:02.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:02.921367+0000) 2022-09-10T15:02:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:03 smithi104 ceph-mon[97044]: pgmap v20246: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:03.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:03.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:03.294982+0000) 2022-09-10T15:02:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:03 smithi110 ceph-mon[97633]: pgmap v20246: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:04.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:03.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:03.921581+0000) 2022-09-10T15:02:04.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:04.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:04.295204+0000) 2022-09-10T15:02:05.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:04.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:04.921821+0000) 2022-09-10T15:02:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:05 smithi104 ceph-mon[97044]: pgmap v20247: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:05.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:05.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:05.295434+0000) 2022-09-10T15:02:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:05 smithi110 ceph-mon[97633]: pgmap v20247: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:06.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:05.920+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:05.922067+0000) 2022-09-10T15:02:06.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:06.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:06.295670+0000) 2022-09-10T15:02:06.924 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:02:06.563+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:02:07.297 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:06.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:06.922383+0000) 2022-09-10T15:02:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:07 smithi104 ceph-mon[97044]: pgmap v20248: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:02:07.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:07.294+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:07.295914+0000) 2022-09-10T15:02:07.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:07 smithi110 ceph-mon[97633]: pgmap v20248: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:02:08.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:07.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:07.922545+0000) 2022-09-10T15:02:08.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:08.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:08.296190+0000) 2022-09-10T15:02:09.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:08.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:08.922788+0000) 2022-09-10T15:02:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:09 smithi104 ceph-mon[97044]: pgmap v20249: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:02:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:09 smithi104 ceph-mon[97044]: osdmap e1346: 12 total, 12 up, 12 in 2022-09-10T15:02:09.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:09.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:09.296370+0000) 2022-09-10T15:02:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:09 smithi110 ceph-mon[97633]: pgmap v20249: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:02:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:09 smithi110 ceph-mon[97633]: osdmap e1346: 12 total, 12 up, 12 in 2022-09-10T15:02:10.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:09.921+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:09.923011+0000) 2022-09-10T15:02:10.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:10.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:10.296614+0000) 2022-09-10T15:02:11.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:10.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:10.923211+0000) 2022-09-10T15:02:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:11 smithi104 ceph-mon[97044]: pgmap v20251: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:02:11.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:11.295+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:11.296806+0000) 2022-09-10T15:02:11.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:11 smithi110 ceph-mon[97633]: pgmap v20251: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:02:12.298 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:11.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:11.923429+0000) 2022-09-10T15:02:12.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:12.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:12.296999+0000) 2022-09-10T15:02:13.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:12.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:12.923616+0000) 2022-09-10T15:02:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:13 smithi104 ceph-mon[97044]: pgmap v20252: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:02:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:13 smithi104 ceph-mon[97044]: osdmap e1347: 12 total, 12 up, 12 in 2022-09-10T15:02:13.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:13.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:13.297239+0000) 2022-09-10T15:02:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:13 smithi110 ceph-mon[97633]: pgmap v20252: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:02:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:13 smithi110 ceph-mon[97633]: osdmap e1347: 12 total, 12 up, 12 in 2022-09-10T15:02:14.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:13.922+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:13.923747+0000) 2022-09-10T15:02:14.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:14.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:14.297410+0000) 2022-09-10T15:02:15.299 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:14.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:14.923927+0000) 2022-09-10T15:02:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:15 smithi104 ceph-mon[97044]: pgmap v20254: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:02:15.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:15.296+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:15.297632+0000) 2022-09-10T15:02:15.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:15 smithi110 ceph-mon[97633]: pgmap v20254: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:02:16.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:15.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:15.924131+0000) 2022-09-10T15:02:16.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:16.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:16.297844+0000) 2022-09-10T15:02:17.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:16.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:16.924339+0000) 2022-09-10T15:02:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:17 smithi104 ceph-mon[97044]: pgmap v20255: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:02:17.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:17.297+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:17.298145+0000) 2022-09-10T15:02:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:17 smithi110 ceph-mon[97633]: pgmap v20255: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:02:18.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:17.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:17.924556+0000) 2022-09-10T15:02:18.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:18.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:18.298374+0000) 2022-09-10T15:02:19.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:18.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:18.924776+0000) 2022-09-10T15:02:19.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:19 smithi104 ceph-mon[97044]: pgmap v20256: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:02:19.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:19.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:19.298638+0000) 2022-09-10T15:02:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:19 smithi110 ceph-mon[97633]: pgmap v20256: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:02:20.300 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:19.923+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:19.925018+0000) 2022-09-10T15:02:20.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:20.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:20.298888+0000) 2022-09-10T15:02:21.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:20.924+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:20.925210+0000) 2022-09-10T15:02:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:21 smithi104 ceph-mon[97044]: pgmap v20257: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:02:21.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:21.298+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:21.299175+0000) 2022-09-10T15:02:21.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:21 smithi110 ceph-mon[97633]: pgmap v20257: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:02:21.927 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:02:21.566+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:02:22.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:21.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:21.925398+0000) 2022-09-10T15:02:22.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:22.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:22.299386+0000) 2022-09-10T15:02:23.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:22.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:22.925676+0000) 2022-09-10T15:02:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:23 smithi104 ceph-mon[97044]: pgmap v20258: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:02:23.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:23.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:23.299638+0000) 2022-09-10T15:02:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:23 smithi110 ceph-mon[97633]: pgmap v20258: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:02:24.301 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:23.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:23.925885+0000) 2022-09-10T15:02:24.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:24.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:24.299868+0000) 2022-09-10T15:02:25.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:24.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:24.926012+0000) 2022-09-10T15:02:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:25 smithi104 ceph-mon[97044]: pgmap v20259: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:02:25.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:25.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:25.300168+0000) 2022-09-10T15:02:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:25 smithi110 ceph-mon[97633]: pgmap v20259: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:02:26.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:25.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:25.926248+0000) 2022-09-10T15:02:26.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:26.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:26.300375+0000) 2022-09-10T15:02:27.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:26.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:26.926455+0000) 2022-09-10T15:02:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:27 smithi104 ceph-mon[97044]: pgmap v20260: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:02:27.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:27.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:27.300555+0000) 2022-09-10T15:02:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:27 smithi110 ceph-mon[97633]: pgmap v20260: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:02:28.302 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:27.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:27.926647+0000) 2022-09-10T15:02:28.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:28.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:28.300770+0000) 2022-09-10T15:02:29.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:28.925+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:28.926911+0000) 2022-09-10T15:02:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:29 smithi104 ceph-mon[97044]: pgmap v20261: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:29.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:29.299+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:29.301040+0000) 2022-09-10T15:02:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:29 smithi110 ceph-mon[97633]: pgmap v20261: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:30.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:29.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:29.927131+0000) 2022-09-10T15:02:30.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:30.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:30.301305+0000) 2022-09-10T15:02:31.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:30.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:30.927361+0000) 2022-09-10T15:02:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:31 smithi104 ceph-mon[97044]: pgmap v20262: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:31.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:31.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:31.301534+0000) 2022-09-10T15:02:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:31 smithi110 ceph-mon[97633]: pgmap v20262: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:32.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:31.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:31.927590+0000) 2022-09-10T15:02:32.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:32.300+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:32.301736+0000) 2022-09-10T15:02:33.303 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:32.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:32.927724+0000) 2022-09-10T15:02:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:33 smithi104 ceph-mon[97044]: pgmap v20263: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:33.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:33.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:33.302016+0000) 2022-09-10T15:02:33.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:33 smithi110 ceph-mon[97633]: pgmap v20263: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:34.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:33.926+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:33.927979+0000) 2022-09-10T15:02:34.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:34.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:34.302240+0000) 2022-09-10T15:02:35.304 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:34.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:34.928150+0000) 2022-09-10T15:02:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:35 smithi104 ceph-mon[97044]: pgmap v20264: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:35.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:35.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:35.302449+0000) 2022-09-10T15:02:35.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:35 smithi110 ceph-mon[97633]: pgmap v20264: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:35.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:35.928393+0000) 2022-09-10T15:02:36.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:36.301+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:36.302699+0000) 2022-09-10T15:02:36.930 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:02:36.579+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:02:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:36.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:36.928618+0000) 2022-09-10T15:02:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:37 smithi104 ceph-mon[97044]: pgmap v20265: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:37.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:37.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:37.303052+0000) 2022-09-10T15:02:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:37 smithi110 ceph-mon[97633]: pgmap v20265: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:37.927+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:37.928885+0000) 2022-09-10T15:02:38.555 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:38.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:38.303209+0000) 2022-09-10T15:02:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:38.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:38.929209+0000) 2022-09-10T15:02:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:39 smithi104 ceph-mon[97044]: pgmap v20266: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:39 smithi110 ceph-mon[97633]: pgmap v20266: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:39.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:39.303403+0000) 2022-09-10T15:02:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:39.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:39.929426+0000) 2022-09-10T15:02:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:40.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:40.303621+0000) 2022-09-10T15:02:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:40.928+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:40.929672+0000) 2022-09-10T15:02:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:41 smithi104 ceph-mon[97044]: pgmap v20267: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:41 smithi110 ceph-mon[97633]: pgmap v20267: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:41.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:41.303842+0000) 2022-09-10T15:02:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:41.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:41.929962+0000) 2022-09-10T15:02:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:42.302+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:42.303981+0000) 2022-09-10T15:02:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:42.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:42.930165+0000) 2022-09-10T15:02:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:43 smithi104 ceph-mon[97044]: pgmap v20268: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:43 smithi110 ceph-mon[97633]: pgmap v20268: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:43.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:43.304151+0000) 2022-09-10T15:02:44.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:43.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:43.930417+0000) 2022-09-10T15:02:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:44.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:44.304277+0000) 2022-09-10T15:02:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:44.929+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:44.930629+0000) 2022-09-10T15:02:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:45 smithi104 ceph-mon[97044]: pgmap v20269: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:45 smithi110 ceph-mon[97633]: pgmap v20269: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:45.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:45.304453+0000) 2022-09-10T15:02:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:45.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:45.930893+0000) 2022-09-10T15:02:46.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:46 smithi110 ceph-mon[97633]: pgmap v20270: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:46 smithi104 ceph-mon[97044]: pgmap v20270: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:46.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:46.304627+0000) 2022-09-10T15:02:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:46.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:46.931215+0000) 2022-09-10T15:02:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:02:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:02:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:02:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:02:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:02:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:02:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:02:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:02:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:47.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:47.304784+0000) 2022-09-10T15:02:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:47.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:47.931343+0000) 2022-09-10T15:02:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:48 smithi110 ceph-mon[97633]: pgmap v20271: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:48 smithi104 ceph-mon[97044]: pgmap v20271: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:48.303+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:48.304940+0000) 2022-09-10T15:02:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:48.930+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:48.931562+0000) 2022-09-10T15:02:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:49.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:49.305158+0000) 2022-09-10T15:02:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:49.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:49.931807+0000) 2022-09-10T15:02:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:50.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:50.305354+0000) 2022-09-10T15:02:51.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:51 smithi104 ceph-mon[97044]: pgmap v20272: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:50.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:50.932037+0000) 2022-09-10T15:02:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:51 smithi110 ceph-mon[97633]: pgmap v20272: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:51.591 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:51.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:51.305530+0000) 2022-09-10T15:02:51.934 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:02:51.588+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:02:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:51.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:51.932224+0000) 2022-09-10T15:02:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:52.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:52.305667+0000) 2022-09-10T15:02:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:52.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:52.932469+0000) 2022-09-10T15:02:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:53 smithi104 ceph-mon[97044]: pgmap v20273: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:53 smithi110 ceph-mon[97633]: pgmap v20273: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:53.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:53.305813+0000) 2022-09-10T15:02:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:53.931+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:53.932670+0000) 2022-09-10T15:02:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:54.304+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:54.305983+0000) 2022-09-10T15:02:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:54.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:54.932925+0000) 2022-09-10T15:02:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:55 smithi104 ceph-mon[97044]: pgmap v20274: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:55 smithi110 ceph-mon[97633]: pgmap v20274: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:55.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:55.306193+0000) 2022-09-10T15:02:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:55.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:55.933154+0000) 2022-09-10T15:02:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:56.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:56.306371+0000) 2022-09-10T15:02:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:56.932+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:56.933386+0000) 2022-09-10T15:02:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:57 smithi104 ceph-mon[97044]: pgmap v20275: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:57 smithi110 ceph-mon[97633]: pgmap v20275: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:02:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:57.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:57.306491+0000) 2022-09-10T15:02:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:57.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:57.933614+0000) 2022-09-10T15:02:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:58.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:58.306670+0000) 2022-09-10T15:02:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:58.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:58.933882+0000) 2022-09-10T15:02:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:02:59 smithi104 ceph-mon[97044]: pgmap v20276: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:02:59 smithi110 ceph-mon[97633]: pgmap v20276: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:02:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:02:59.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:59.306843+0000) 2022-09-10T15:03:00.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:02:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:02:59.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:02:59.934151+0000) 2022-09-10T15:03:00.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:00.305+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:00.306986+0000) 2022-09-10T15:03:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:00.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:00.934366+0000) 2022-09-10T15:03:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:01 smithi104 ceph-mon[97044]: pgmap v20277: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:01 smithi110 ceph-mon[97633]: pgmap v20277: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:01.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:01.307167+0000) 2022-09-10T15:03:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:01.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:01.934512+0000) 2022-09-10T15:03:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:02.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:02.307306+0000) 2022-09-10T15:03:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:02.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:02.934698+0000) 2022-09-10T15:03:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:03 smithi104 ceph-mon[97044]: pgmap v20278: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:03 smithi110 ceph-mon[97633]: pgmap v20278: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:03.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:03.307464+0000) 2022-09-10T15:03:04.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:03.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:03.934911+0000) 2022-09-10T15:03:04.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:04.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:04.307629+0000) 2022-09-10T15:03:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:04.933+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:04.935043+0000) 2022-09-10T15:03:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:05 smithi104 ceph-mon[97044]: pgmap v20279: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:05 smithi110 ceph-mon[97633]: pgmap v20279: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:05.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:05.307806+0000) 2022-09-10T15:03:06.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:05.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:05.935297+0000) 2022-09-10T15:03:06.610 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:06.306+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:06.307981+0000) 2022-09-10T15:03:06.938 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:03:06.604+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:03:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:06.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:06.935582+0000) 2022-09-10T15:03:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:07 smithi104 ceph-mon[97044]: pgmap v20280: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:03:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:07 smithi110 ceph-mon[97633]: pgmap v20280: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:03:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:07.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:07.308189+0000) 2022-09-10T15:03:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:07.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:07.935799+0000) 2022-09-10T15:03:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:08.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:08.308352+0000) 2022-09-10T15:03:09.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:08.934+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:08.936054+0000) 2022-09-10T15:03:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:09 smithi104 ceph-mon[97044]: pgmap v20281: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:03:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:09 smithi104 ceph-mon[97044]: osdmap e1348: 12 total, 12 up, 12 in 2022-09-10T15:03:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:09 smithi110 ceph-mon[97633]: pgmap v20281: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:03:09.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:09 smithi110 ceph-mon[97633]: osdmap e1348: 12 total, 12 up, 12 in 2022-09-10T15:03:09.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:09.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:09.308474+0000) 2022-09-10T15:03:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:09.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:09.936350+0000) 2022-09-10T15:03:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:10.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:10.308615+0000) 2022-09-10T15:03:11.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:10.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:10.936607+0000) 2022-09-10T15:03:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:11 smithi104 ceph-mon[97044]: pgmap v20283: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:03:11.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:11 smithi110 ceph-mon[97633]: pgmap v20283: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:03:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:11.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:11.308810+0000) 2022-09-10T15:03:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:11.935+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:11.936821+0000) 2022-09-10T15:03:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:12.307+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:12.308973+0000) 2022-09-10T15:03:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:12.936+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:12.937072+0000) 2022-09-10T15:03:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:13 smithi104 ceph-mon[97044]: pgmap v20284: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:03:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:13 smithi104 ceph-mon[97044]: osdmap e1349: 12 total, 12 up, 12 in 2022-09-10T15:03:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:13 smithi110 ceph-mon[97633]: pgmap v20284: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:03:13.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:13 smithi110 ceph-mon[97633]: osdmap e1349: 12 total, 12 up, 12 in 2022-09-10T15:03:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:13.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:13.309132+0000) 2022-09-10T15:03:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:13.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:13.937315+0000) 2022-09-10T15:03:14.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:14.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:14.309299+0000) 2022-09-10T15:03:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:14.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:14.937490+0000) 2022-09-10T15:03:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:15 smithi104 ceph-mon[97044]: pgmap v20286: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:03:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:15 smithi110 ceph-mon[97633]: pgmap v20286: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:03:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:15.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:15.309459+0000) 2022-09-10T15:03:16.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:15.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:15.937729+0000) 2022-09-10T15:03:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:16.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:16.309626+0000) 2022-09-10T15:03:17.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:16.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:16.937919+0000) 2022-09-10T15:03:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:17 smithi104 ceph-mon[97044]: pgmap v20287: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:03:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:17 smithi110 ceph-mon[97633]: pgmap v20287: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:03:17.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:17.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:17.309861+0000) 2022-09-10T15:03:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:17.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:17.938129+0000) 2022-09-10T15:03:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:18.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:18.310036+0000) 2022-09-10T15:03:19.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:18.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:18.938329+0000) 2022-09-10T15:03:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:19 smithi104 ceph-mon[97044]: pgmap v20288: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:03:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:19 smithi110 ceph-mon[97633]: pgmap v20288: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:03:19.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:19.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:19.310243+0000) 2022-09-10T15:03:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:19.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:19.938465+0000) 2022-09-10T15:03:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:20.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:20.310372+0000) 2022-09-10T15:03:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:20.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:20.938610+0000) 2022-09-10T15:03:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:21 smithi104 ceph-mon[97044]: pgmap v20289: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:03:21.614 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:21.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:21.310521+0000) 2022-09-10T15:03:21.691 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:21 smithi110 ceph-mon[97633]: pgmap v20289: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:03:21.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:03:21.611+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:03:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:21.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:21.938863+0000) 2022-09-10T15:03:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:22.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:22.310768+0000) 2022-09-10T15:03:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:22.937+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:22.939035+0000) 2022-09-10T15:03:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:23 smithi104 ceph-mon[97044]: pgmap v20290: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:03:23.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:23 smithi110 ceph-mon[97633]: pgmap v20290: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:03:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:23.309+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:23.310928+0000) 2022-09-10T15:03:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:23.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:23.939208+0000) 2022-09-10T15:03:24.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:24.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:24.311115+0000) 2022-09-10T15:03:25.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:24.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:24.939377+0000) 2022-09-10T15:03:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:25 smithi104 ceph-mon[97044]: pgmap v20291: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:03:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:25 smithi110 ceph-mon[97633]: pgmap v20291: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:03:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:25.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:25.311337+0000) 2022-09-10T15:03:26.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:25.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:25.939554+0000) 2022-09-10T15:03:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:26.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:26.311467+0000) 2022-09-10T15:03:27.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:26.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:26.939805+0000) 2022-09-10T15:03:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:27 smithi104 ceph-mon[97044]: pgmap v20292: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:03:27.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:27 smithi110 ceph-mon[97633]: pgmap v20292: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:03:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:27.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:27.311708+0000) 2022-09-10T15:03:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:27.938+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:27.939965+0000) 2022-09-10T15:03:28.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:28.310+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:28.311935+0000) 2022-09-10T15:03:29.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:28.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:28.940222+0000) 2022-09-10T15:03:29.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:29 smithi104 ceph-mon[97044]: pgmap v20293: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:29 smithi110 ceph-mon[97633]: pgmap v20293: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:29.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:29.312195+0000) 2022-09-10T15:03:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:29.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:29.940412+0000) 2022-09-10T15:03:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:30.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:30.312391+0000) 2022-09-10T15:03:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:30.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:30.940643+0000) 2022-09-10T15:03:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:31 smithi104 ceph-mon[97044]: pgmap v20294: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:31.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:31 smithi110 ceph-mon[97633]: pgmap v20294: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:31.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:31.312589+0000) 2022-09-10T15:03:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:31.939+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:31.940909+0000) 2022-09-10T15:03:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:32.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:32.312799+0000) 2022-09-10T15:03:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:32.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:32.941092+0000) 2022-09-10T15:03:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:33 smithi104 ceph-mon[97044]: pgmap v20295: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:33.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:33 smithi110 ceph-mon[97633]: pgmap v20295: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:33.311+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:33.313010+0000) 2022-09-10T15:03:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:33.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:33.941269+0000) 2022-09-10T15:03:34.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:34.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:34.313184+0000) 2022-09-10T15:03:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:34.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:34.941490+0000) 2022-09-10T15:03:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:35 smithi104 ceph-mon[97044]: pgmap v20296: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:35.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:35 smithi110 ceph-mon[97633]: pgmap v20296: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:35.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:35.313312+0000) 2022-09-10T15:03:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:35.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:35.941675+0000) 2022-09-10T15:03:36.616 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:36.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:36.313464+0000) 2022-09-10T15:03:36.945 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:03:36.614+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:03:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:36.940+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:36.941877+0000) 2022-09-10T15:03:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:37 smithi104 ceph-mon[97044]: pgmap v20297: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:37 smithi110 ceph-mon[97633]: pgmap v20297: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:37.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:37.313641+0000) 2022-09-10T15:03:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:37.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:37.942152+0000) 2022-09-10T15:03:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:38.312+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:38.313838+0000) 2022-09-10T15:03:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:38.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:38.942404+0000) 2022-09-10T15:03:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:39 smithi104 ceph-mon[97044]: pgmap v20298: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:39 smithi110 ceph-mon[97633]: pgmap v20298: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:39.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:39.314026+0000) 2022-09-10T15:03:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:39.941+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:39.942648+0000) 2022-09-10T15:03:40.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:40 smithi110 ceph-mon[97633]: pgmap v20299: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:40 smithi104 ceph-mon[97044]: pgmap v20299: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:40.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:40.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:40.314231+0000) 2022-09-10T15:03:41.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:40.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:40.942901+0000) 2022-09-10T15:03:41.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:41.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:41.314503+0000) 2022-09-10T15:03:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:41.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:41.943137+0000) 2022-09-10T15:03:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:42.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:42.314765+0000) 2022-09-10T15:03:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:42.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:42.943330+0000) 2022-09-10T15:03:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:43 smithi104 ceph-mon[97044]: pgmap v20300: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:43 smithi110 ceph-mon[97633]: pgmap v20300: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:43.313+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:43.314963+0000) 2022-09-10T15:03:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:43.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:43.943578+0000) 2022-09-10T15:03:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:44.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:44.315187+0000) 2022-09-10T15:03:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:44.942+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:44.943798+0000) 2022-09-10T15:03:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:45 smithi104 ceph-mon[97044]: pgmap v20301: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:45 smithi110 ceph-mon[97633]: pgmap v20301: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:45.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:45.315427+0000) 2022-09-10T15:03:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:45.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:45.944059+0000) 2022-09-10T15:03:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:46.314+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:46.315624+0000) 2022-09-10T15:03:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:46.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:46.944363+0000) 2022-09-10T15:03:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:47 smithi104 ceph-mon[97044]: pgmap v20302: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:03:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:03:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:03:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:03:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:47 smithi110 ceph-mon[97633]: pgmap v20302: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:03:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:03:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:03:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:03:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:47.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:47.315867+0000) 2022-09-10T15:03:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:47.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:47.944525+0000) 2022-09-10T15:03:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:48.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:48.316111+0000) 2022-09-10T15:03:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:48.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:48.944721+0000) 2022-09-10T15:03:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:49 smithi104 ceph-mon[97044]: pgmap v20303: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:49.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:49 smithi110 ceph-mon[97633]: pgmap v20303: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:49.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:49.316309+0000) 2022-09-10T15:03:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:49.943+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:49.944946+0000) 2022-09-10T15:03:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:50.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:50.316504+0000) 2022-09-10T15:03:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:50.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:50.945199+0000) 2022-09-10T15:03:51.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:51 smithi104 ceph-mon[97044]: pgmap v20304: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:51.629 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:51.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:51.316718+0000) 2022-09-10T15:03:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:51 smithi110 ceph-mon[97633]: pgmap v20304: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:51.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:03:51.627+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:03:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:51.944+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:51.945424+0000) 2022-09-10T15:03:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:52.315+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:52.316997+0000) 2022-09-10T15:03:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:52.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:52.945682+0000) 2022-09-10T15:03:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:53 smithi104 ceph-mon[97044]: pgmap v20305: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:53.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:53 smithi110 ceph-mon[97633]: pgmap v20305: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:53.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:53.317211+0000) 2022-09-10T15:03:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:53.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:53.945834+0000) 2022-09-10T15:03:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:54.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:54.317397+0000) 2022-09-10T15:03:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:54.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:54.946045+0000) 2022-09-10T15:03:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:55 smithi104 ceph-mon[97044]: pgmap v20306: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:55 smithi110 ceph-mon[97633]: pgmap v20306: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:55.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:55.317611+0000) 2022-09-10T15:03:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:55.945+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:55.946325+0000) 2022-09-10T15:03:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:56.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:56.317789+0000) 2022-09-10T15:03:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:56.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:56.946548+0000) 2022-09-10T15:03:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:57 smithi104 ceph-mon[97044]: pgmap v20307: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:57 smithi110 ceph-mon[97633]: pgmap v20307: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:03:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:57.316+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:57.317996+0000) 2022-09-10T15:03:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:57.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:57.946764+0000) 2022-09-10T15:03:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:58.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:58.318253+0000) 2022-09-10T15:03:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:58.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:58.946938+0000) 2022-09-10T15:03:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:03:59 smithi104 ceph-mon[97044]: pgmap v20308: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:03:59 smithi110 ceph-mon[97633]: pgmap v20308: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:03:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:03:59.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:59.318467+0000) 2022-09-10T15:04:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:03:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:03:59.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:03:59.947145+0000) 2022-09-10T15:04:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:00.317+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:00.318647+0000) 2022-09-10T15:04:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:00.946+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:00.947299+0000) 2022-09-10T15:04:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:01 smithi104 ceph-mon[97044]: pgmap v20309: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:01.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:01 smithi110 ceph-mon[97633]: pgmap v20309: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:01.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:01.318817+0000) 2022-09-10T15:04:02.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:01.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:01.947514+0000) 2022-09-10T15:04:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:02.318+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:02.319057+0000) 2022-09-10T15:04:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:02.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:02.947761+0000) 2022-09-10T15:04:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:03 smithi104 ceph-mon[97044]: pgmap v20310: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:03 smithi110 ceph-mon[97633]: pgmap v20310: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:03.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:03.319366+0000) 2022-09-10T15:04:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:03.947+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:03.947922+0000) 2022-09-10T15:04:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:04 smithi110 ceph-mon[97633]: pgmap v20311: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:04 smithi104 ceph-mon[97044]: pgmap v20311: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:04.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:04.319508+0000) 2022-09-10T15:04:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:04.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:04.948145+0000) 2022-09-10T15:04:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:05.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:05.319728+0000) 2022-09-10T15:04:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:05.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:05.948384+0000) 2022-09-10T15:04:06.645 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:06.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:06.319917+0000) 2022-09-10T15:04:06.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:04:06.643+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:04:07.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:07 smithi104 ceph-mon[97044]: pgmap v20312: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:04:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:06.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:06.948647+0000) 2022-09-10T15:04:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:07 smithi110 ceph-mon[97633]: pgmap v20312: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:04:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:07.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:07.320170+0000) 2022-09-10T15:04:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:07.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:07.948849+0000) 2022-09-10T15:04:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:08.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:08.320417+0000) 2022-09-10T15:04:09.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:09 smithi104 ceph-mon[97044]: pgmap v20313: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:04:09.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:09 smithi104 ceph-mon[97044]: osdmap e1350: 12 total, 12 up, 12 in 2022-09-10T15:04:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:08.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:08.949149+0000) 2022-09-10T15:04:09.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:09 smithi110 ceph-mon[97633]: pgmap v20313: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:04:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:09 smithi110 ceph-mon[97633]: osdmap e1350: 12 total, 12 up, 12 in 2022-09-10T15:04:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:09.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:09.320620+0000) 2022-09-10T15:04:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:09.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:09.949415+0000) 2022-09-10T15:04:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:10.319+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:10.320858+0000) 2022-09-10T15:04:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:10.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:10.949675+0000) 2022-09-10T15:04:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:11 smithi104 ceph-mon[97044]: pgmap v20315: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:04:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:11 smithi110 ceph-mon[97633]: pgmap v20315: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:04:11.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:11.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:11.321140+0000) 2022-09-10T15:04:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:11.948+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:11.949900+0000) 2022-09-10T15:04:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:12.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:12.321338+0000) 2022-09-10T15:04:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:12.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:12.950181+0000) 2022-09-10T15:04:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:13 smithi104 ceph-mon[97044]: pgmap v20316: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:04:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:13 smithi104 ceph-mon[97044]: osdmap e1351: 12 total, 12 up, 12 in 2022-09-10T15:04:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:13 smithi110 ceph-mon[97633]: pgmap v20316: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:04:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:13 smithi110 ceph-mon[97633]: osdmap e1351: 12 total, 12 up, 12 in 2022-09-10T15:04:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:13.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:13.321574+0000) 2022-09-10T15:04:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:13.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:13.950410+0000) 2022-09-10T15:04:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:14.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:14.321833+0000) 2022-09-10T15:04:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:14.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:14.950666+0000) 2022-09-10T15:04:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:15 smithi104 ceph-mon[97044]: pgmap v20318: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:04:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:15 smithi110 ceph-mon[97633]: pgmap v20318: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:04:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:15.320+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:15.322049+0000) 2022-09-10T15:04:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:15.949+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:15.950898+0000) 2022-09-10T15:04:16.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:16.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:16.322319+0000) 2022-09-10T15:04:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:16.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:16.951187+0000) 2022-09-10T15:04:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:17 smithi104 ceph-mon[97044]: pgmap v20319: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:04:17.657 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:17 smithi110 ceph-mon[97633]: pgmap v20319: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:04:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:17.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:17.322621+0000) 2022-09-10T15:04:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:17.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:17.951436+0000) 2022-09-10T15:04:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:18.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:18.322828+0000) 2022-09-10T15:04:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:18.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:18.951667+0000) 2022-09-10T15:04:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:19 smithi104 ceph-mon[97044]: pgmap v20320: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:04:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:19 smithi110 ceph-mon[97633]: pgmap v20320: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:04:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:19.321+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:19.323026+0000) 2022-09-10T15:04:20.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:19.950+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:19.951838+0000) 2022-09-10T15:04:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:20.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:20.323250+0000) 2022-09-10T15:04:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:20.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:20.952079+0000) 2022-09-10T15:04:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:21 smithi104 ceph-mon[97044]: pgmap v20321: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:04:21.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:21 smithi110 ceph-mon[97633]: pgmap v20321: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:04:21.660 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:21.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:21.323494+0000) 2022-09-10T15:04:21.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:04:21.658+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:04:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:21.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:21.952288+0000) 2022-09-10T15:04:22.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:22.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:22.323726+0000) 2022-09-10T15:04:23.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:22.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:22.952531+0000) 2022-09-10T15:04:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:23 smithi104 ceph-mon[97044]: pgmap v20322: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:04:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:23 smithi110 ceph-mon[97633]: pgmap v20322: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:04:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:23.322+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:23.323964+0000) 2022-09-10T15:04:24.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:23.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:23.952776+0000) 2022-09-10T15:04:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:24.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:24.324170+0000) 2022-09-10T15:04:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:24.951+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:24.953013+0000) 2022-09-10T15:04:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:25 smithi104 ceph-mon[97044]: pgmap v20323: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:04:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:25 smithi110 ceph-mon[97633]: pgmap v20323: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:04:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:25.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:25.324353+0000) 2022-09-10T15:04:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:25.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:25.953258+0000) 2022-09-10T15:04:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:26.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:26.324544+0000) 2022-09-10T15:04:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:26.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:26.953531+0000) 2022-09-10T15:04:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:27 smithi104 ceph-mon[97044]: pgmap v20324: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:04:27.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:27 smithi110 ceph-mon[97633]: pgmap v20324: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:04:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:27.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:27.324778+0000) 2022-09-10T15:04:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:27.952+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:27.953759+0000) 2022-09-10T15:04:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:28.323+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:28.325033+0000) 2022-09-10T15:04:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:28.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:28.953999+0000) 2022-09-10T15:04:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:29 smithi104 ceph-mon[97044]: pgmap v20325: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:29 smithi110 ceph-mon[97633]: pgmap v20325: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:29.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:29.325264+0000) 2022-09-10T15:04:30.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:29.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:29.954193+0000) 2022-09-10T15:04:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:30.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:30.325546+0000) 2022-09-10T15:04:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:30.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:30.954354+0000) 2022-09-10T15:04:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:31 smithi104 ceph-mon[97044]: pgmap v20326: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:31 smithi110 ceph-mon[97633]: pgmap v20326: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:31.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:31.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:31.325778+0000) 2022-09-10T15:04:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:31.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:31.954511+0000) 2022-09-10T15:04:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:32.324+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:32.325946+0000) 2022-09-10T15:04:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:32.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:32.954738+0000) 2022-09-10T15:04:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:33 smithi104 ceph-mon[97044]: pgmap v20327: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:33 smithi110 ceph-mon[97633]: pgmap v20327: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:33.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:33.326171+0000) 2022-09-10T15:04:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:33.953+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:33.954943+0000) 2022-09-10T15:04:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:34.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:34.326320+0000) 2022-09-10T15:04:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:34.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:34.955170+0000) 2022-09-10T15:04:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:35 smithi104 ceph-mon[97044]: pgmap v20328: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:35 smithi110 ceph-mon[97633]: pgmap v20328: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:35.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:35.326533+0000) 2022-09-10T15:04:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:35.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:35.955398+0000) 2022-09-10T15:04:36.664 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:36.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:36.326702+0000) 2022-09-10T15:04:36.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:04:36.661+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:04:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:36.954+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:36.955655+0000) 2022-09-10T15:04:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:37 smithi104 ceph-mon[97044]: pgmap v20329: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:37 smithi110 ceph-mon[97633]: pgmap v20329: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:37.325+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:37.326928+0000) 2022-09-10T15:04:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:37.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:37.955849+0000) 2022-09-10T15:04:38.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:38 smithi104 ceph-mon[97044]: pgmap v20330: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:38 smithi110 ceph-mon[97633]: pgmap v20330: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:38.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:38.327178+0000) 2022-09-10T15:04:39.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:38.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:38.956080+0000) 2022-09-10T15:04:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:39.326+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:39.327422+0000) 2022-09-10T15:04:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:39.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:39.956311+0000) 2022-09-10T15:04:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:40.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:40.327600+0000) 2022-09-10T15:04:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:40.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:40.956474+0000) 2022-09-10T15:04:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:41 smithi110 ceph-mon[97633]: pgmap v20331: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:41 smithi104 ceph-mon[97044]: pgmap v20331: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:41.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:41.327832+0000) 2022-09-10T15:04:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:41.955+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:41.956651+0000) 2022-09-10T15:04:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:42.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:42.328077+0000) 2022-09-10T15:04:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:42.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:42.956917+0000) 2022-09-10T15:04:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:43 smithi104 ceph-mon[97044]: pgmap v20332: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:43 smithi110 ceph-mon[97633]: pgmap v20332: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:43.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:43.328285+0000) 2022-09-10T15:04:44.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:43.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:43.957148+0000) 2022-09-10T15:04:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:44.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:44.328505+0000) 2022-09-10T15:04:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:44.956+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:44.957356+0000) 2022-09-10T15:04:45.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:45 smithi104 ceph-mon[97044]: pgmap v20333: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:45 smithi110 ceph-mon[97633]: pgmap v20333: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:45.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:45.328778+0000) 2022-09-10T15:04:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:45.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:45.957563+0000) 2022-09-10T15:04:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:46.327+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:46.329007+0000) 2022-09-10T15:04:47.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:46.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:46.957837+0000) 2022-09-10T15:04:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:47 smithi104 ceph-mon[97044]: pgmap v20334: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:04:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:04:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:04:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:04:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:47 smithi110 ceph-mon[97633]: pgmap v20334: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:04:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:04:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:04:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:04:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:47.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:47.329217+0000) 2022-09-10T15:04:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:47.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:47.958089+0000) 2022-09-10T15:04:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:48.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:48.329410+0000) 2022-09-10T15:04:49.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:48.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:48.958372+0000) 2022-09-10T15:04:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:49 smithi104 ceph-mon[97044]: pgmap v20335: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:49.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:49 smithi110 ceph-mon[97633]: pgmap v20335: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:49.328+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:49.329575+0000) 2022-09-10T15:04:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:49.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:49.958586+0000) 2022-09-10T15:04:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:50.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:50.329788+0000) 2022-09-10T15:04:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:50.957+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:50.958798+0000) 2022-09-10T15:04:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:51 smithi104 ceph-mon[97044]: pgmap v20336: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:51.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:51 smithi110 ceph-mon[97633]: pgmap v20336: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:51.679 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:51.329+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:51.330041+0000) 2022-09-10T15:04:51.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:04:51.676+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:04:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:51.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:51.958989+0000) 2022-09-10T15:04:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:52.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:52.330241+0000) 2022-09-10T15:04:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:52.958+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:52.959207+0000) 2022-09-10T15:04:53.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:53 smithi104 ceph-mon[97044]: pgmap v20337: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:53.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:53 smithi110 ceph-mon[97633]: pgmap v20337: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:53.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:53.330468+0000) 2022-09-10T15:04:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:53.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:53.959480+0000) 2022-09-10T15:04:54.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:54.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:54.330706+0000) 2022-09-10T15:04:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:54.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:54.959702+0000) 2022-09-10T15:04:55.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:55 smithi104 ceph-mon[97044]: pgmap v20338: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:55 smithi110 ceph-mon[97633]: pgmap v20338: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:55.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:55.330952+0000) 2022-09-10T15:04:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:55.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:55.959888+0000) 2022-09-10T15:04:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:56.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:56.331090+0000) 2022-09-10T15:04:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:56.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:56.960062+0000) 2022-09-10T15:04:57.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:57 smithi104 ceph-mon[97044]: pgmap v20339: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:57 smithi110 ceph-mon[97633]: pgmap v20339: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:04:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:57.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:57.331288+0000) 2022-09-10T15:04:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:57.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:57.960316+0000) 2022-09-10T15:04:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:58.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:58.331484+0000) 2022-09-10T15:04:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:58.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:58.960546+0000) 2022-09-10T15:04:59.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:04:59 smithi104 ceph-mon[97044]: pgmap v20340: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:04:59 smithi110 ceph-mon[97633]: pgmap v20340: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:04:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:04:59.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:59.331696+0000) 2022-09-10T15:05:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:04:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:04:59.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:04:59.960757+0000) 2022-09-10T15:05:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:00.330+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:00.331945+0000) 2022-09-10T15:05:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:00.959+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:00.960967+0000) 2022-09-10T15:05:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:01 smithi104 ceph-mon[97044]: pgmap v20341: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:01 smithi110 ceph-mon[97633]: pgmap v20341: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:01.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:01.332195+0000) 2022-09-10T15:05:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:01.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:01.961219+0000) 2022-09-10T15:05:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:02.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:02.332386+0000) 2022-09-10T15:05:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:02.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:02.961399+0000) 2022-09-10T15:05:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:03 smithi104 ceph-mon[97044]: pgmap v20342: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:03 smithi110 ceph-mon[97633]: pgmap v20342: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:03.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:03.332641+0000) 2022-09-10T15:05:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:03.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:03.961574+0000) 2022-09-10T15:05:04.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:04.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:04.332821+0000) 2022-09-10T15:05:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:04.960+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:04.961809+0000) 2022-09-10T15:05:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:05 smithi104 ceph-mon[97044]: pgmap v20343: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:05 smithi110 ceph-mon[97633]: pgmap v20343: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:05.331+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:05.333029+0000) 2022-09-10T15:05:06.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:05.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:05.962063+0000) 2022-09-10T15:05:06.684 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:06.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:06.333249+0000) 2022-09-10T15:05:06.964 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:05:06.681+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:05:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:06.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:06.962271+0000) 2022-09-10T15:05:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:07 smithi104 ceph-mon[97044]: pgmap v20344: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:05:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:07 smithi110 ceph-mon[97633]: pgmap v20344: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:05:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:07.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:07.333415+0000) 2022-09-10T15:05:08.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:07.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:07.962426+0000) 2022-09-10T15:05:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:08.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:08.333588+0000) 2022-09-10T15:05:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:08.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:08.962609+0000) 2022-09-10T15:05:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:09 smithi104 ceph-mon[97044]: pgmap v20345: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:05:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:09 smithi104 ceph-mon[97044]: osdmap e1352: 12 total, 12 up, 12 in 2022-09-10T15:05:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:09 smithi110 ceph-mon[97633]: pgmap v20345: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:05:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:09 smithi110 ceph-mon[97633]: osdmap e1352: 12 total, 12 up, 12 in 2022-09-10T15:05:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:09.332+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:09.333849+0000) 2022-09-10T15:05:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:09.961+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:09.962860+0000) 2022-09-10T15:05:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:10.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:10.334086+0000) 2022-09-10T15:05:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:10.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:10.963082+0000) 2022-09-10T15:05:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:11 smithi104 ceph-mon[97044]: pgmap v20347: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:05:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:11 smithi110 ceph-mon[97633]: pgmap v20347: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:05:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:11.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:11.334413+0000) 2022-09-10T15:05:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:11.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:11.963338+0000) 2022-09-10T15:05:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:12.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:12.334588+0000) 2022-09-10T15:05:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:12.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:12.963538+0000) 2022-09-10T15:05:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:13 smithi104 ceph-mon[97044]: pgmap v20348: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:05:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:13 smithi104 ceph-mon[97044]: osdmap e1353: 12 total, 12 up, 12 in 2022-09-10T15:05:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:13 smithi110 ceph-mon[97633]: pgmap v20348: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:05:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:13 smithi110 ceph-mon[97633]: osdmap e1353: 12 total, 12 up, 12 in 2022-09-10T15:05:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:13.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:13.334841+0000) 2022-09-10T15:05:14.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:13.962+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:13.963803+0000) 2022-09-10T15:05:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:14.333+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:14.335057+0000) 2022-09-10T15:05:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:14.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:14.963989+0000) 2022-09-10T15:05:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:15 smithi104 ceph-mon[97044]: pgmap v20350: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:05:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:15 smithi110 ceph-mon[97633]: pgmap v20350: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:05:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:15.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:15.335289+0000) 2022-09-10T15:05:16.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:15.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:15.964155+0000) 2022-09-10T15:05:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:16.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:16.335487+0000) 2022-09-10T15:05:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:16.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:16.964316+0000) 2022-09-10T15:05:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:17 smithi104 ceph-mon[97044]: pgmap v20351: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:05:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:17 smithi110 ceph-mon[97633]: pgmap v20351: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:05:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:17.334+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:17.335675+0000) 2022-09-10T15:05:18.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:17.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:17.964536+0000) 2022-09-10T15:05:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:18.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:18.335869+0000) 2022-09-10T15:05:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:18.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:18.964849+0000) 2022-09-10T15:05:19.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:19 smithi104 ceph-mon[97044]: pgmap v20352: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:05:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:19 smithi110 ceph-mon[97633]: pgmap v20352: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:05:19.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:19.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:19.336118+0000) 2022-09-10T15:05:20.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:19.963+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:19.965039+0000) 2022-09-10T15:05:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:20.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:20.336289+0000) 2022-09-10T15:05:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:20.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:20.965178+0000) 2022-09-10T15:05:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:21 smithi104 ceph-mon[97044]: pgmap v20353: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:05:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:21 smithi110 ceph-mon[97633]: pgmap v20353: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:05:21.699 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:21.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:21.336566+0000) 2022-09-10T15:05:21.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:05:21.696+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:05:22.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:21.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:21.965335+0000) 2022-09-10T15:05:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:22.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:22.336767+0000) 2022-09-10T15:05:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:22.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:22.965575+0000) 2022-09-10T15:05:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:23 smithi104 ceph-mon[97044]: pgmap v20354: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:05:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:23 smithi110 ceph-mon[97633]: pgmap v20354: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:05:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:23.335+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:23.336961+0000) 2022-09-10T15:05:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:23.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:23.965740+0000) 2022-09-10T15:05:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:24.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:24.337216+0000) 2022-09-10T15:05:25.320 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:24.964+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:24.965978+0000) 2022-09-10T15:05:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:25 smithi104 ceph-mon[97044]: pgmap v20355: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:05:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:25 smithi110 ceph-mon[97633]: pgmap v20355: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 87 B/s wr, 8 op/s 2022-09-10T15:05:25.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:25.336+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:25.337504+0000) 2022-09-10T15:05:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:25.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:25.966202+0000) 2022-09-10T15:05:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:26.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:26.337703+0000) 2022-09-10T15:05:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:26.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:26.966421+0000) 2022-09-10T15:05:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:27 smithi104 ceph-mon[97044]: pgmap v20356: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:05:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:27 smithi110 ceph-mon[97633]: pgmap v20356: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:05:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:27.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:27.337945+0000) 2022-09-10T15:05:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:27.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:27.966646+0000) 2022-09-10T15:05:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:28.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:28.338153+0000) 2022-09-10T15:05:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:28.965+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:28.966891+0000) 2022-09-10T15:05:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:29 smithi104 ceph-mon[97044]: pgmap v20357: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:29 smithi110 ceph-mon[97633]: pgmap v20357: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:29.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:29.338383+0000) 2022-09-10T15:05:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:29.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:29.967174+0000) 2022-09-10T15:05:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:30 smithi104 ceph-mon[97044]: pgmap v20358: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:30 smithi110 ceph-mon[97633]: pgmap v20358: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:30.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:30.338608+0000) 2022-09-10T15:05:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:30.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:30.967347+0000) 2022-09-10T15:05:31.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:31.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:31.338867+0000) 2022-09-10T15:05:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:31.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:31.967581+0000) 2022-09-10T15:05:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:32.337+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:32.338981+0000) 2022-09-10T15:05:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:32.966+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:32.967781+0000) 2022-09-10T15:05:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:33 smithi104 ceph-mon[97044]: pgmap v20359: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:33 smithi110 ceph-mon[97633]: pgmap v20359: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:33.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:33.339196+0000) 2022-09-10T15:05:34.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:33.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:33.968079+0000) 2022-09-10T15:05:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:34.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:34.339451+0000) 2022-09-10T15:05:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:34.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:34.968323+0000) 2022-09-10T15:05:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:35 smithi104 ceph-mon[97044]: pgmap v20360: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:35 smithi110 ceph-mon[97633]: pgmap v20360: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:35.338+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:35.339667+0000) 2022-09-10T15:05:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:35.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:35.968557+0000) 2022-09-10T15:05:36.714 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:36.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:36.339895+0000) 2022-09-10T15:05:36.970 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:05:36.712+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:05:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:36.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:36.968850+0000) 2022-09-10T15:05:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:37 smithi104 ceph-mon[97044]: pgmap v20361: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:37 smithi110 ceph-mon[97633]: pgmap v20361: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:37.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:37.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:37.340070+0000) 2022-09-10T15:05:38.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:37.967+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:37.969033+0000) 2022-09-10T15:05:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:38.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:38.340250+0000) 2022-09-10T15:05:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:38.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:38.969271+0000) 2022-09-10T15:05:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:39 smithi104 ceph-mon[97044]: pgmap v20362: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:39 smithi110 ceph-mon[97633]: pgmap v20362: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:05:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:39.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:39.340462+0000) 2022-09-10T15:05:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:39.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:39.969393+0000) 2022-09-10T15:05:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:40.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:40.340587+0000) 2022-09-10T15:05:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:40.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:40.969595+0000) 2022-09-10T15:05:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:41 smithi104 ceph-mon[97044]: pgmap v20363: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:41 smithi110 ceph-mon[97633]: pgmap v20363: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:41.339+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:41.340848+0000) 2022-09-10T15:05:42.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:41.969+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:41.969768+0000) 2022-09-10T15:05:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:42.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:42.341131+0000) 2022-09-10T15:05:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:42.968+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:42.970039+0000) 2022-09-10T15:05:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:43 smithi104 ceph-mon[97044]: pgmap v20364: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:43 smithi110 ceph-mon[97633]: pgmap v20364: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:43.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:43.340+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:43.341292+0000) 2022-09-10T15:05:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:43.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:43.970244+0000) 2022-09-10T15:05:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:44.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:44.341538+0000) 2022-09-10T15:05:45.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:44.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:44.970397+0000) 2022-09-10T15:05:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:45 smithi104 ceph-mon[97044]: pgmap v20365: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:45 smithi110 ceph-mon[97633]: pgmap v20365: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:45.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:45.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:45.341783+0000) 2022-09-10T15:05:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:45.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:45.970520+0000) 2022-09-10T15:05:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:46.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:46.342045+0000) 2022-09-10T15:05:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:46.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:46.970683+0000) 2022-09-10T15:05:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:47 smithi104 ceph-mon[97044]: pgmap v20366: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:05:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:05:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:05:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:05:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:47 smithi110 ceph-mon[97633]: pgmap v20366: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:05:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:05:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:05:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:05:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:47.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:47.342273+0000) 2022-09-10T15:05:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:47.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:47.970925+0000) 2022-09-10T15:05:48.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:48 smithi104 ceph-mon[97044]: pgmap v20367: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:48 smithi110 ceph-mon[97633]: pgmap v20367: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:48.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:48.342461+0000) 2022-09-10T15:05:49.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:48.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:48.971172+0000) 2022-09-10T15:05:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:49.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:49.342647+0000) 2022-09-10T15:05:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:49.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:49.971355+0000) 2022-09-10T15:05:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:50.341+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:50.342860+0000) 2022-09-10T15:05:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:50.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:50.971522+0000) 2022-09-10T15:05:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:51 smithi110 ceph-mon[97633]: pgmap v20368: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:51.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:51 smithi104 ceph-mon[97044]: pgmap v20368: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:51.731 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:51.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:51.343142+0000) 2022-09-10T15:05:52.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:51.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:51.971716+0000) 2022-09-10T15:05:52.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:05:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:05:51.726+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:05:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:52.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:52.343341+0000) 2022-09-10T15:05:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:52.970+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:52.971884+0000) 2022-09-10T15:05:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:53 smithi104 ceph-mon[97044]: pgmap v20369: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:53 smithi110 ceph-mon[97633]: pgmap v20369: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:53.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:53.343592+0000) 2022-09-10T15:05:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:53.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:53.972083+0000) 2022-09-10T15:05:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:54.342+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:54.343790+0000) 2022-09-10T15:05:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:54.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:54.972351+0000) 2022-09-10T15:05:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:55 smithi104 ceph-mon[97044]: pgmap v20370: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:55 smithi110 ceph-mon[97633]: pgmap v20370: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:55.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:55.343975+0000) 2022-09-10T15:05:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:55.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:55.972533+0000) 2022-09-10T15:05:56.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:56.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:56.344216+0000) 2022-09-10T15:05:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:56.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:56.972788+0000) 2022-09-10T15:05:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:57 smithi104 ceph-mon[97044]: pgmap v20371: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:57 smithi110 ceph-mon[97633]: pgmap v20371: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:05:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:57.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:57.344456+0000) 2022-09-10T15:05:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:57.971+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:57.972951+0000) 2022-09-10T15:05:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:58.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:58.344695+0000) 2022-09-10T15:05:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:58.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:58.973103+0000) 2022-09-10T15:05:59.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:05:59 smithi104 ceph-mon[97044]: pgmap v20372: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:59.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:05:59 smithi110 ceph-mon[97633]: pgmap v20372: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:05:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:05:59.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:59.344872+0000) 2022-09-10T15:06:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:05:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:05:59.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:05:59.973291+0000) 2022-09-10T15:06:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:00.343+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:00.345057+0000) 2022-09-10T15:06:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:00.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:00.973486+0000) 2022-09-10T15:06:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:01 smithi104 ceph-mon[97044]: pgmap v20373: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:01 smithi110 ceph-mon[97633]: pgmap v20373: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:01.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:01.345241+0000) 2022-09-10T15:06:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:01.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:01.973688+0000) 2022-09-10T15:06:02.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:02.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:02.345423+0000) 2022-09-10T15:06:03.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:02.972+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:02.973931+0000) 2022-09-10T15:06:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:03 smithi104 ceph-mon[97044]: pgmap v20374: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:03 smithi110 ceph-mon[97633]: pgmap v20374: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:03.344+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:03.345677+0000) 2022-09-10T15:06:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:03.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:03.974177+0000) 2022-09-10T15:06:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:04.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:04.345901+0000) 2022-09-10T15:06:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:04.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:04.974367+0000) 2022-09-10T15:06:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:05 smithi104 ceph-mon[97044]: pgmap v20375: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:05 smithi110 ceph-mon[97633]: pgmap v20375: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:05.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:05.346176+0000) 2022-09-10T15:06:06.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:05.973+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:05.974589+0000) 2022-09-10T15:06:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:06 smithi104 ceph-mon[97044]: pgmap v20376: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:06:06.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:06 smithi110 ceph-mon[97633]: pgmap v20376: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:06:06.746 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:06.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:06.346393+0000) 2022-09-10T15:06:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:06.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:06.974868+0000) 2022-09-10T15:06:07.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:06:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:06:06.742+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:06:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:07.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:07.346582+0000) 2022-09-10T15:06:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:07.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:07.975155+0000) 2022-09-10T15:06:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:08.345+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:08.346796+0000) 2022-09-10T15:06:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:08.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:08.975381+0000) 2022-09-10T15:06:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:09 smithi104 ceph-mon[97044]: pgmap v20377: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:06:09.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:09 smithi104 ceph-mon[97044]: osdmap e1354: 12 total, 12 up, 12 in 2022-09-10T15:06:09.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:09 smithi110 ceph-mon[97633]: pgmap v20377: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:06:09.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:09 smithi110 ceph-mon[97633]: osdmap e1354: 12 total, 12 up, 12 in 2022-09-10T15:06:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:09.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:09.346996+0000) 2022-09-10T15:06:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:09.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:09.975653+0000) 2022-09-10T15:06:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:10.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:10.347212+0000) 2022-09-10T15:06:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:10.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:10.975860+0000) 2022-09-10T15:06:11.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:11 smithi104 ceph-mon[97044]: pgmap v20379: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:06:11.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:11 smithi110 ceph-mon[97633]: pgmap v20379: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:06:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:11.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:11.347365+0000) 2022-09-10T15:06:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:11.974+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:11.976032+0000) 2022-09-10T15:06:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:12.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:12.347511+0000) 2022-09-10T15:06:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:12.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:12.976292+0000) 2022-09-10T15:06:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:13 smithi104 ceph-mon[97044]: pgmap v20380: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:06:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:13 smithi104 ceph-mon[97044]: osdmap e1355: 12 total, 12 up, 12 in 2022-09-10T15:06:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:13 smithi110 ceph-mon[97633]: pgmap v20380: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:06:13.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:13 smithi110 ceph-mon[97633]: osdmap e1355: 12 total, 12 up, 12 in 2022-09-10T15:06:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:13.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:13.347673+0000) 2022-09-10T15:06:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:13.975+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:13.976550+0000) 2022-09-10T15:06:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:14.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:14.347846+0000) 2022-09-10T15:06:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:14.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:14.976822+0000) 2022-09-10T15:06:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:15 smithi104 ceph-mon[97044]: pgmap v20382: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:06:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:15 smithi110 ceph-mon[97633]: pgmap v20382: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:06:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:15.346+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:15.348074+0000) 2022-09-10T15:06:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:15.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:15.977024+0000) 2022-09-10T15:06:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:16.347+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:16.348339+0000) 2022-09-10T15:06:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:16.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:16.977263+0000) 2022-09-10T15:06:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:17 smithi104 ceph-mon[97044]: pgmap v20383: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:06:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:17 smithi110 ceph-mon[97633]: pgmap v20383: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:06:17.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:17.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:17.348534+0000) 2022-09-10T15:06:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:17.976+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:17.977513+0000) 2022-09-10T15:06:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:18.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:18.348801+0000) 2022-09-10T15:06:19.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:18.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:18.977748+0000) 2022-09-10T15:06:19.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:19 smithi104 ceph-mon[97044]: pgmap v20384: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:06:19.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:19 smithi110 ceph-mon[97633]: pgmap v20384: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:06:19.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:19.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:19.349052+0000) 2022-09-10T15:06:20.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:19.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:19.977995+0000) 2022-09-10T15:06:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:20.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:20.349254+0000) 2022-09-10T15:06:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:20.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:20.978226+0000) 2022-09-10T15:06:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:21 smithi104 ceph-mon[97044]: pgmap v20385: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:06:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:21 smithi110 ceph-mon[97633]: pgmap v20385: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:06:21.761 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:21.348+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:21.349481+0000) 2022-09-10T15:06:22.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:21.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:21.978449+0000) 2022-09-10T15:06:22.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:06:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:06:21.758+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:06:22.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:22.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:22.349702+0000) 2022-09-10T15:06:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:22.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:22.978706+0000) 2022-09-10T15:06:23.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:23 smithi104 ceph-mon[97044]: pgmap v20386: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:06:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:23 smithi110 ceph-mon[97633]: pgmap v20386: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:06:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:23.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:23.349885+0000) 2022-09-10T15:06:24.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:23.977+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:23.978907+0000) 2022-09-10T15:06:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:24 smithi104 ceph-mon[97044]: pgmap v20387: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:06:24.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:24 smithi110 ceph-mon[97633]: pgmap v20387: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:06:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:24.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:24.350077+0000) 2022-09-10T15:06:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:24.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:24.979090+0000) 2022-09-10T15:06:25.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:25.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:25.350321+0000) 2022-09-10T15:06:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:25.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:25.979313+0000) 2022-09-10T15:06:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:26.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:26.350516+0000) 2022-09-10T15:06:27.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:26.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:26.979490+0000) 2022-09-10T15:06:27.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:27 smithi110 ceph-mon[97633]: pgmap v20388: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:06:27.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:27 smithi104 ceph-mon[97044]: pgmap v20388: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:06:27.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:27.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:27.350692+0000) 2022-09-10T15:06:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:27.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:27.979740+0000) 2022-09-10T15:06:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:28.349+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:28.350873+0000) 2022-09-10T15:06:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:28.978+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:28.980010+0000) 2022-09-10T15:06:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:29 smithi104 ceph-mon[97044]: pgmap v20389: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:29.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:29 smithi110 ceph-mon[97633]: pgmap v20389: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:29.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:29.351178+0000) 2022-09-10T15:06:30.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:29.979+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:29.980213+0000) 2022-09-10T15:06:30.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:30.350+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:30.351309+0000) 2022-09-10T15:06:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:30.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:30.980488+0000) 2022-09-10T15:06:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:31 smithi104 ceph-mon[97044]: pgmap v20390: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:31 smithi110 ceph-mon[97633]: pgmap v20390: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:31.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:31.351526+0000) 2022-09-10T15:06:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:31.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:31.980738+0000) 2022-09-10T15:06:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:32.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:32.351698+0000) 2022-09-10T15:06:33.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:32.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:32.980932+0000) 2022-09-10T15:06:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:33 smithi104 ceph-mon[97044]: pgmap v20391: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:33 smithi110 ceph-mon[97633]: pgmap v20391: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:33.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:33.351943+0000) 2022-09-10T15:06:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:33.980+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:33.981183+0000) 2022-09-10T15:06:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:34.351+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:34.352137+0000) 2022-09-10T15:06:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:34.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:34.981380+0000) 2022-09-10T15:06:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:35 smithi104 ceph-mon[97044]: pgmap v20392: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:35 smithi110 ceph-mon[97633]: pgmap v20392: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:35.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:35.352299+0000) 2022-09-10T15:06:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:35.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:35.981615+0000) 2022-09-10T15:06:36.769 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:36.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:36.352503+0000) 2022-09-10T15:06:37.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:36.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:36.981901+0000) 2022-09-10T15:06:37.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:06:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:06:36.767+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:06:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:37 smithi104 ceph-mon[97044]: pgmap v20393: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:37 smithi110 ceph-mon[97633]: pgmap v20393: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:37.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:37.352668+0000) 2022-09-10T15:06:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:37.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:37.982118+0000) 2022-09-10T15:06:38.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:38 smithi104 ceph-mon[97044]: pgmap v20394: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:38 smithi110 ceph-mon[97633]: pgmap v20394: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:38.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:38.352926+0000) 2022-09-10T15:06:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:38.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:38.982351+0000) 2022-09-10T15:06:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:39.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:39.353196+0000) 2022-09-10T15:06:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:39.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:39.982579+0000) 2022-09-10T15:06:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:40.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:40.353397+0000) 2022-09-10T15:06:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:40.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:40.982756+0000) 2022-09-10T15:06:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:41 smithi110 ceph-mon[97633]: pgmap v20395: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:41 smithi104 ceph-mon[97044]: pgmap v20395: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:41.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:41.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:41.353633+0000) 2022-09-10T15:06:42.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:41.981+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:41.982967+0000) 2022-09-10T15:06:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:42.352+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:42.353810+0000) 2022-09-10T15:06:43.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:42.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:42.983244+0000) 2022-09-10T15:06:43.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:43 smithi104 ceph-mon[97044]: pgmap v20396: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:43.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:43 smithi110 ceph-mon[97633]: pgmap v20396: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:43.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:43.354133+0000) 2022-09-10T15:06:44.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:43.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:43.983491+0000) 2022-09-10T15:06:44.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:44.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:44.354353+0000) 2022-09-10T15:06:45.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:44.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:44.983750+0000) 2022-09-10T15:06:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:45 smithi104 ceph-mon[97044]: pgmap v20397: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:45.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:45 smithi110 ceph-mon[97633]: pgmap v20397: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:45.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:45.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:45.354536+0000) 2022-09-10T15:06:46.311 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:45.982+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:45.983961+0000) 2022-09-10T15:06:46.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:46 smithi110 ceph-mon[97633]: pgmap v20398: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:46 smithi104 ceph-mon[97044]: pgmap v20398: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:46.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:46.354741+0000) 2022-09-10T15:06:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:46.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:46.984228+0000) 2022-09-10T15:06:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:06:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:06:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:06:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:06:47.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:06:47.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:06:47.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:06:47.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:06:47.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:47.353+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:47.354928+0000) 2022-09-10T15:06:48.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:47.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:47.984430+0000) 2022-09-10T15:06:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:48 smithi110 ceph-mon[97633]: pgmap v20399: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:48.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:48 smithi104 ceph-mon[97044]: pgmap v20399: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:48.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:48.355186+0000) 2022-09-10T15:06:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:48.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:48.984652+0000) 2022-09-10T15:06:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:49.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:49.355355+0000) 2022-09-10T15:06:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:49.983+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:49.984834+0000) 2022-09-10T15:06:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:50.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:50.355514+0000) 2022-09-10T15:06:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:50.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:50.985016+0000) 2022-09-10T15:06:51.407 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:51 smithi110 ceph-mon[97633]: pgmap v20400: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:51 smithi104 ceph-mon[97044]: pgmap v20400: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:51.771 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:51.354+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:51.355752+0000) 2022-09-10T15:06:52.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:06:51.768+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:06:52.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:51.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:51.985177+0000) 2022-09-10T15:06:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:52.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:52.355913+0000) 2022-09-10T15:06:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:52.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:52.985368+0000) 2022-09-10T15:06:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:53 smithi104 ceph-mon[97044]: pgmap v20401: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:53 smithi110 ceph-mon[97633]: pgmap v20401: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:53.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:53.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:53.356191+0000) 2022-09-10T15:06:54.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:53.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:53.985548+0000) 2022-09-10T15:06:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:54.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:54.356349+0000) 2022-09-10T15:06:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:54.984+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:54.985761+0000) 2022-09-10T15:06:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:55 smithi104 ceph-mon[97044]: pgmap v20402: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:55 smithi110 ceph-mon[97633]: pgmap v20402: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:55.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:55.356549+0000) 2022-09-10T15:06:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:55.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:55.986008+0000) 2022-09-10T15:06:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:56.355+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:56.356700+0000) 2022-09-10T15:06:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:56.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:56.986288+0000) 2022-09-10T15:06:57.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:57 smithi104 ceph-mon[97044]: pgmap v20403: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:57 smithi110 ceph-mon[97633]: pgmap v20403: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:06:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:57.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:57.356910+0000) 2022-09-10T15:06:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:57.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:57.986452+0000) 2022-09-10T15:06:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:06:58 smithi110 ceph-mon[97633]: pgmap v20404: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:06:58 smithi104 ceph-mon[97044]: pgmap v20404: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:06:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:58.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:58.357169+0000) 2022-09-10T15:06:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:58.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:58.986613+0000) 2022-09-10T15:06:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:06:59.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:59.357307+0000) 2022-09-10T15:07:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:06:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:06:59.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:06:59.986832+0000) 2022-09-10T15:07:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:00.356+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:00.357515+0000) 2022-09-10T15:07:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:00.985+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:00.986953+0000) 2022-09-10T15:07:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:01 smithi110 ceph-mon[97633]: pgmap v20405: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:01 smithi104 ceph-mon[97044]: pgmap v20405: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:01.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:01.357743+0000) 2022-09-10T15:07:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:01.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:01.987188+0000) 2022-09-10T15:07:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:02.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:02.357968+0000) 2022-09-10T15:07:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:02.986+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:02.987445+0000) 2022-09-10T15:07:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:03 smithi104 ceph-mon[97044]: pgmap v20406: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:03 smithi110 ceph-mon[97633]: pgmap v20406: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:03.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:03.358187+0000) 2022-09-10T15:07:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:03.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:03.987670+0000) 2022-09-10T15:07:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:04.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:04.358367+0000) 2022-09-10T15:07:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:04.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:04.987920+0000) 2022-09-10T15:07:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:05 smithi104 ceph-mon[97044]: pgmap v20407: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:05 smithi110 ceph-mon[97633]: pgmap v20407: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:05.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:05.358589+0000) 2022-09-10T15:07:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:05.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:05.988169+0000) 2022-09-10T15:07:06.787 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:06.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:06.358775+0000) 2022-09-10T15:07:07.055 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:06.987+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:06.988391+0000) 2022-09-10T15:07:07.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:07:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:07:06.784+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:07:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:07 smithi104 ceph-mon[97044]: pgmap v20408: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:07:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:07 smithi110 ceph-mon[97633]: pgmap v20408: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:07:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:07.357+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:07.358930+0000) 2022-09-10T15:07:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:07.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:07.988638+0000) 2022-09-10T15:07:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:08 smithi110 ceph-mon[97633]: pgmap v20409: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:07:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:08 smithi110 ceph-mon[97633]: osdmap e1356: 12 total, 12 up, 12 in 2022-09-10T15:07:08.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:08 smithi104 ceph-mon[97044]: pgmap v20409: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:07:08.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:08 smithi104 ceph-mon[97044]: osdmap e1356: 12 total, 12 up, 12 in 2022-09-10T15:07:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:08.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:08.359189+0000) 2022-09-10T15:07:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:08.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:08.988844+0000) 2022-09-10T15:07:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:09.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:09.359364+0000) 2022-09-10T15:07:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:09.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:09.989069+0000) 2022-09-10T15:07:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:10.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:10.359585+0000) 2022-09-10T15:07:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:10.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:10.989345+0000) 2022-09-10T15:07:11.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:11 smithi110 ceph-mon[97633]: pgmap v20411: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:07:11.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:11 smithi104 ceph-mon[97044]: pgmap v20411: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:07:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:11.358+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:11.359807+0000) 2022-09-10T15:07:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:11.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:11.989594+0000) 2022-09-10T15:07:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:12.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:12.359990+0000) 2022-09-10T15:07:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:12.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:12.989770+0000) 2022-09-10T15:07:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:13 smithi104 ceph-mon[97044]: pgmap v20412: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:07:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:13 smithi104 ceph-mon[97044]: osdmap e1357: 12 total, 12 up, 12 in 2022-09-10T15:07:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:13 smithi110 ceph-mon[97633]: pgmap v20412: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:07:13.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:13 smithi110 ceph-mon[97633]: osdmap e1357: 12 total, 12 up, 12 in 2022-09-10T15:07:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:13.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:13.360180+0000) 2022-09-10T15:07:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:13.988+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:13.989971+0000) 2022-09-10T15:07:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:14.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:14.360387+0000) 2022-09-10T15:07:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:14.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:14.990243+0000) 2022-09-10T15:07:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:15 smithi104 ceph-mon[97044]: pgmap v20414: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:07:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:15 smithi110 ceph-mon[97633]: pgmap v20414: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:07:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:15.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:15.360602+0000) 2022-09-10T15:07:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:15.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:15.990469+0000) 2022-09-10T15:07:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:16.359+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:16.360810+0000) 2022-09-10T15:07:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:16.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:16.990701+0000) 2022-09-10T15:07:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:17 smithi104 ceph-mon[97044]: pgmap v20415: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:07:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:17 smithi110 ceph-mon[97633]: pgmap v20415: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:07:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:17.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:17.361079+0000) 2022-09-10T15:07:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:17.989+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:17.990973+0000) 2022-09-10T15:07:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:18 smithi104 ceph-mon[97044]: pgmap v20416: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:07:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:18 smithi110 ceph-mon[97633]: pgmap v20416: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:07:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:18.360+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:18.361269+0000) 2022-09-10T15:07:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:18.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:18.991174+0000) 2022-09-10T15:07:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:19.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:19.361539+0000) 2022-09-10T15:07:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:19.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:19.991386+0000) 2022-09-10T15:07:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:20.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:20.361773+0000) 2022-09-10T15:07:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:20.990+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:20.991585+0000) 2022-09-10T15:07:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:21 smithi110 ceph-mon[97633]: pgmap v20417: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:07:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:21 smithi104 ceph-mon[97044]: pgmap v20417: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:07:21.803 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:21.361+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:21.362034+0000) 2022-09-10T15:07:22.056 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:21.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:21.991782+0000) 2022-09-10T15:07:22.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:07:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:07:21.800+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:07:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:22.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:22.362213+0000) 2022-09-10T15:07:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:22.991+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:22.992054+0000) 2022-09-10T15:07:23.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:23 smithi104 ceph-mon[97044]: pgmap v20418: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:07:23.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:23 smithi110 ceph-mon[97633]: pgmap v20418: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:07:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:23.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:23.362362+0000) 2022-09-10T15:07:24.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:23.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:23.992250+0000) 2022-09-10T15:07:24.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:24 smithi110 ceph-mon[97633]: pgmap v20419: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:07:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:24 smithi104 ceph-mon[97044]: pgmap v20419: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:07:24.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:24.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:24.362579+0000) 2022-09-10T15:07:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:24.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:24.992462+0000) 2022-09-10T15:07:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:25.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:25.362790+0000) 2022-09-10T15:07:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:25.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:25.992688+0000) 2022-09-10T15:07:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:26.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:26.363048+0000) 2022-09-10T15:07:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:26.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:26.992965+0000) 2022-09-10T15:07:27.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:27 smithi110 ceph-mon[97633]: pgmap v20420: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:07:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:27 smithi104 ceph-mon[97044]: pgmap v20420: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:07:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:27.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:27.363298+0000) 2022-09-10T15:07:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:27.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:27.993215+0000) 2022-09-10T15:07:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:28.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:28.363514+0000) 2022-09-10T15:07:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:28.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:28.993423+0000) 2022-09-10T15:07:29.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:29 smithi104 ceph-mon[97044]: pgmap v20421: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:29.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:29 smithi110 ceph-mon[97633]: pgmap v20421: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:29.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:29.363792+0000) 2022-09-10T15:07:30.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:29.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:29.993667+0000) 2022-09-10T15:07:30.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:30 smithi110 ceph-mon[97633]: pgmap v20422: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:30 smithi104 ceph-mon[97044]: pgmap v20422: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:30.362+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:30.363988+0000) 2022-09-10T15:07:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:30.992+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:30.993875+0000) 2022-09-10T15:07:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:31.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:31.364194+0000) 2022-09-10T15:07:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:31.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:31.994161+0000) 2022-09-10T15:07:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:32.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:32.364355+0000) 2022-09-10T15:07:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:32.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:32.994387+0000) 2022-09-10T15:07:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:33 smithi104 ceph-mon[97044]: pgmap v20423: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:33 smithi110 ceph-mon[97633]: pgmap v20423: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:33.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:33.364553+0000) 2022-09-10T15:07:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:33.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:33.994549+0000) 2022-09-10T15:07:34.663 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:34 smithi110 ceph-mon[97633]: pgmap v20424: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:34 smithi104 ceph-mon[97044]: pgmap v20424: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:34.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:34.364675+0000) 2022-09-10T15:07:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:34.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:34.994808+0000) 2022-09-10T15:07:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:35.363+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:35.364932+0000) 2022-09-10T15:07:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:35.993+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:35.995009+0000) 2022-09-10T15:07:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:36.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:36.365184+0000) 2022-09-10T15:07:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:36.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:36.995299+0000) 2022-09-10T15:07:37.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:07:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:07:36.815+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:07:37.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:37 smithi110 ceph-mon[97633]: pgmap v20425: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:37 smithi104 ceph-mon[97044]: pgmap v20425: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:37.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:37.365391+0000) 2022-09-10T15:07:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:37.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:37.995519+0000) 2022-09-10T15:07:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:38.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:38.365608+0000) 2022-09-10T15:07:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:38.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:38.995718+0000) 2022-09-10T15:07:39.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:39 smithi104 ceph-mon[97044]: pgmap v20426: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:39.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:39 smithi110 ceph-mon[97633]: pgmap v20426: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:39.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:39.365837+0000) 2022-09-10T15:07:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:39.994+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:39.995938+0000) 2022-09-10T15:07:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:40 smithi104 ceph-mon[97044]: pgmap v20427: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:40.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:40 smithi110 ceph-mon[97633]: pgmap v20427: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:40.364+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:40.366061+0000) 2022-09-10T15:07:41.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:40.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:40.996208+0000) 2022-09-10T15:07:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:41.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:41.366371+0000) 2022-09-10T15:07:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:41.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:41.996374+0000) 2022-09-10T15:07:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:42.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:42.366610+0000) 2022-09-10T15:07:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:42.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:42.996513+0000) 2022-09-10T15:07:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:43 smithi104 ceph-mon[97044]: pgmap v20428: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:43 smithi110 ceph-mon[97633]: pgmap v20428: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:43.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:43.366803+0000) 2022-09-10T15:07:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:43.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:43.996740+0000) 2022-09-10T15:07:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:44 smithi104 ceph-mon[97044]: pgmap v20429: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:44.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:44 smithi110 ceph-mon[97633]: pgmap v20429: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:44.365+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:44.367049+0000) 2022-09-10T15:07:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:44.995+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:44.996915+0000) 2022-09-10T15:07:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:45.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:45.367304+0000) 2022-09-10T15:07:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:45.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:45.997168+0000) 2022-09-10T15:07:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:46.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:46.367508+0000) 2022-09-10T15:07:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:46.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:46.997350+0000) 2022-09-10T15:07:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:47 smithi110 ceph-mon[97633]: pgmap v20430: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:07:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:07:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:07:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:07:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:47 smithi104 ceph-mon[97044]: pgmap v20430: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:07:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:07:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:07:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:07:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:47.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:47.367708+0000) 2022-09-10T15:07:48.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:47.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:47.997599+0000) 2022-09-10T15:07:48.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:48.366+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:48.367878+0000) 2022-09-10T15:07:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:48.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:48.997846+0000) 2022-09-10T15:07:49.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:49 smithi104 ceph-mon[97044]: pgmap v20431: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:49.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:49 smithi110 ceph-mon[97633]: pgmap v20431: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:49.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:49.368132+0000) 2022-09-10T15:07:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:49.996+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:49.998035+0000) 2022-09-10T15:07:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:50 smithi104 ceph-mon[97044]: pgmap v20432: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:50.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:50 smithi110 ceph-mon[97633]: pgmap v20432: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:50.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:50.368376+0000) 2022-09-10T15:07:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:50.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:50.998167+0000) 2022-09-10T15:07:51.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:51.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:51.368642+0000) 2022-09-10T15:07:52.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:07:51.818+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:07:52.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:51.997+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:51.998377+0000) 2022-09-10T15:07:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:52.367+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:52.368796+0000) 2022-09-10T15:07:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:52.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:52.998600+0000) 2022-09-10T15:07:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:53 smithi104 ceph-mon[97044]: pgmap v20433: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:53 smithi110 ceph-mon[97633]: pgmap v20433: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:53.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:53.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:53.369008+0000) 2022-09-10T15:07:54.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:53.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:53.998823+0000) 2022-09-10T15:07:54.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:54 smithi104 ceph-mon[97044]: pgmap v20434: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:54.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:54 smithi110 ceph-mon[97633]: pgmap v20434: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:54.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:54.369281+0000) 2022-09-10T15:07:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:54.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:54.999055+0000) 2022-09-10T15:07:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:55.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:55.369549+0000) 2022-09-10T15:07:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:55.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:55.999298+0000) 2022-09-10T15:07:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:56.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:56.369777+0000) 2022-09-10T15:07:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:56.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:56.999523+0000) 2022-09-10T15:07:57.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:57 smithi110 ceph-mon[97633]: pgmap v20435: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:57 smithi104 ceph-mon[97044]: pgmap v20435: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:07:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:57.368+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:57.370062+0000) 2022-09-10T15:07:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:57.998+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:57.999689+0000) 2022-09-10T15:07:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:07:58 smithi110 ceph-mon[97633]: pgmap v20436: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:07:58 smithi104 ceph-mon[97044]: pgmap v20436: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:07:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:58.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:58.370226+0000) 2022-09-10T15:07:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:58.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:58.999934+0000) 2022-09-10T15:07:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:07:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:07:59.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:07:59.370457+0000) 2022-09-10T15:08:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:07:59.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:00.000121+0000) 2022-09-10T15:08:00.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:00.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:00.370577+0000) 2022-09-10T15:08:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:00.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:01.000320+0000) 2022-09-10T15:08:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:01 smithi110 ceph-mon[97633]: pgmap v20437: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:01.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:01 smithi104 ceph-mon[97044]: pgmap v20437: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:01.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:01.370750+0000) 2022-09-10T15:08:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:02.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:02.000506+0000) 2022-09-10T15:08:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:02.369+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:02.371011+0000) 2022-09-10T15:08:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:02.999+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:03.000681+0000) 2022-09-10T15:08:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:03 smithi104 ceph-mon[97044]: pgmap v20438: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:03 smithi110 ceph-mon[97633]: pgmap v20438: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:03.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:03.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:03.371245+0000) 2022-09-10T15:08:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:04.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:04.000938+0000) 2022-09-10T15:08:04.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:04 smithi110 ceph-mon[97633]: pgmap v20439: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:04 smithi104 ceph-mon[97044]: pgmap v20439: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:04.370+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:04.371441+0000) 2022-09-10T15:08:05.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:05.000+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:05.001220+0000) 2022-09-10T15:08:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:05.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:05.371660+0000) 2022-09-10T15:08:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:06.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:06.001471+0000) 2022-09-10T15:08:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:06.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:06.371855+0000) 2022-09-10T15:08:07.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:08:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:08:06.834+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:08:07.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:07.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:07.001734+0000) 2022-09-10T15:08:07.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:07 smithi110 ceph-mon[97633]: pgmap v20440: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:08:07.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:07 smithi104 ceph-mon[97044]: pgmap v20440: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:08:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:07.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:07.372132+0000) 2022-09-10T15:08:08.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:08.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:08.001963+0000) 2022-09-10T15:08:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:08 smithi110 ceph-mon[97633]: pgmap v20441: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:08:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:08 smithi104 ceph-mon[97044]: pgmap v20441: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:08:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:08.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:08.372316+0000) 2022-09-10T15:08:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:09.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:09.002181+0000) 2022-09-10T15:08:09.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:09 smithi110 ceph-mon[97633]: osdmap e1358: 12 total, 12 up, 12 in 2022-09-10T15:08:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:09 smithi104 ceph-mon[97044]: osdmap e1358: 12 total, 12 up, 12 in 2022-09-10T15:08:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:09.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:09.372485+0000) 2022-09-10T15:08:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:10.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:10.002378+0000) 2022-09-10T15:08:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:10 smithi110 ceph-mon[97633]: pgmap v20443: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:08:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:10 smithi104 ceph-mon[97044]: pgmap v20443: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:08:10.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:10.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:10.372724+0000) 2022-09-10T15:08:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:11.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:11.002661+0000) 2022-09-10T15:08:11.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:11.371+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:11.372978+0000) 2022-09-10T15:08:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:12.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:12.002839+0000) 2022-09-10T15:08:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:12.372+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:12.373183+0000) 2022-09-10T15:08:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:13.001+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:13.002987+0000) 2022-09-10T15:08:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:13 smithi110 ceph-mon[97633]: pgmap v20444: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:08:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:13 smithi110 ceph-mon[97633]: osdmap e1359: 12 total, 12 up, 12 in 2022-09-10T15:08:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:13 smithi104 ceph-mon[97044]: pgmap v20444: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:08:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:13 smithi104 ceph-mon[97044]: osdmap e1359: 12 total, 12 up, 12 in 2022-09-10T15:08:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:13.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:13.373431+0000) 2022-09-10T15:08:14.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:14.002+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:14.003201+0000) 2022-09-10T15:08:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:14.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:14.373673+0000) 2022-09-10T15:08:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:15.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:15.003413+0000) 2022-09-10T15:08:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:15 smithi110 ceph-mon[97633]: pgmap v20446: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:08:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:15 smithi104 ceph-mon[97044]: pgmap v20446: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:08:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:15.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:15.373837+0000) 2022-09-10T15:08:16.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:16.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:16.003606+0000) 2022-09-10T15:08:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:16.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:16.374009+0000) 2022-09-10T15:08:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:17.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:17.003803+0000) 2022-09-10T15:08:17.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:17 smithi110 ceph-mon[97633]: pgmap v20447: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:08:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:17 smithi104 ceph-mon[97044]: pgmap v20447: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:08:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:17.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:17.374241+0000) 2022-09-10T15:08:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:18.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:18.004021+0000) 2022-09-10T15:08:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:18 smithi110 ceph-mon[97633]: pgmap v20448: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:08:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:18 smithi104 ceph-mon[97044]: pgmap v20448: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:08:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:18.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:18.374463+0000) 2022-09-10T15:08:19.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:19.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:19.004269+0000) 2022-09-10T15:08:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:19.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:19.374714+0000) 2022-09-10T15:08:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:20.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:20.004531+0000) 2022-09-10T15:08:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:20.373+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:20.374933+0000) 2022-09-10T15:08:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:21.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:21.004770+0000) 2022-09-10T15:08:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:21 smithi110 ceph-mon[97633]: pgmap v20449: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:08:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:21 smithi104 ceph-mon[97044]: pgmap v20449: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:08:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:21.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:21.375172+0000) 2022-09-10T15:08:22.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:08:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:08:21.849+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:08:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:22.003+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:22.004924+0000) 2022-09-10T15:08:22.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:22.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:22.375377+0000) 2022-09-10T15:08:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:23.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:23.005183+0000) 2022-09-10T15:08:23.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:23 smithi110 ceph-mon[97633]: pgmap v20450: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:08:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:23 smithi104 ceph-mon[97044]: pgmap v20450: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:08:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:23.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:23.375542+0000) 2022-09-10T15:08:24.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:24.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:24.005418+0000) 2022-09-10T15:08:24.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:24.374+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:24.375808+0000) 2022-09-10T15:08:25.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:25.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:25.005607+0000) 2022-09-10T15:08:25.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:25 smithi110 ceph-mon[97633]: pgmap v20451: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:08:25.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:25 smithi104 ceph-mon[97044]: pgmap v20451: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:08:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:25.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:25.376060+0000) 2022-09-10T15:08:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:26.004+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:26.005818+0000) 2022-09-10T15:08:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:26.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:26.376260+0000) 2022-09-10T15:08:27.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:27.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:27.006059+0000) 2022-09-10T15:08:27.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:27 smithi110 ceph-mon[97633]: pgmap v20452: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:08:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:27 smithi104 ceph-mon[97044]: pgmap v20452: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:08:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:27.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:27.376482+0000) 2022-09-10T15:08:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:28.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:28.006305+0000) 2022-09-10T15:08:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:28 smithi110 ceph-mon[97633]: pgmap v20453: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:28.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:28 smithi104 ceph-mon[97044]: pgmap v20453: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:28.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:28.376686+0000) 2022-09-10T15:08:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:29.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:29.006500+0000) 2022-09-10T15:08:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:29.375+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:29.376854+0000) 2022-09-10T15:08:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:30.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:30.006718+0000) 2022-09-10T15:08:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:30.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:30.377130+0000) 2022-09-10T15:08:31.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:31.005+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:31.006990+0000) 2022-09-10T15:08:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:31 smithi110 ceph-mon[97633]: pgmap v20454: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:31 smithi104 ceph-mon[97044]: pgmap v20454: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:31.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:31.377352+0000) 2022-09-10T15:08:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:32.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:32.007210+0000) 2022-09-10T15:08:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:32.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:32.377595+0000) 2022-09-10T15:08:33.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:33.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:33.007441+0000) 2022-09-10T15:08:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:33 smithi110 ceph-mon[97633]: pgmap v20455: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:33 smithi104 ceph-mon[97044]: pgmap v20455: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:33.376+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:33.377812+0000) 2022-09-10T15:08:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:34.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:34.007601+0000) 2022-09-10T15:08:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:34.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:34.378007+0000) 2022-09-10T15:08:35.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:35.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:35.007826+0000) 2022-09-10T15:08:35.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:35 smithi110 ceph-mon[97633]: pgmap v20456: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:35.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:35 smithi104 ceph-mon[97044]: pgmap v20456: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:35.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:35.378258+0000) 2022-09-10T15:08:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:36.006+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:36.008072+0000) 2022-09-10T15:08:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:36.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:36.378497+0000) 2022-09-10T15:08:37.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:37.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:37.008353+0000) 2022-09-10T15:08:37.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:08:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:08:36.865+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:08:37.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:37 smithi110 ceph-mon[97633]: pgmap v20457: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:37 smithi104 ceph-mon[97044]: pgmap v20457: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:37.377+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:37.378762+0000) 2022-09-10T15:08:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:38.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:38.008541+0000) 2022-09-10T15:08:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:38 smithi110 ceph-mon[97633]: pgmap v20458: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:38 smithi104 ceph-mon[97044]: pgmap v20458: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:38.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:38.378983+0000) 2022-09-10T15:08:39.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:39.007+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:39.008769+0000) 2022-09-10T15:08:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:39.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:39.379275+0000) 2022-09-10T15:08:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:40.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:40.008928+0000) 2022-09-10T15:08:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:40.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:40.379477+0000) 2022-09-10T15:08:41.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:41.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:41.009207+0000) 2022-09-10T15:08:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:41 smithi110 ceph-mon[97633]: pgmap v20459: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:41 smithi104 ceph-mon[97044]: pgmap v20459: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:41.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:41.378+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:41.379770+0000) 2022-09-10T15:08:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:42.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:42.009409+0000) 2022-09-10T15:08:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:42.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:42.380001+0000) 2022-09-10T15:08:43.312 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:43.008+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:43.009654+0000) 2022-09-10T15:08:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:43 smithi110 ceph-mon[97633]: pgmap v20460: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:43.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:43 smithi104 ceph-mon[97044]: pgmap v20460: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:43.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:43.380266+0000) 2022-09-10T15:08:44.323 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:44.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:44.009899+0000) 2022-09-10T15:08:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:44.379+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:44.380524+0000) 2022-09-10T15:08:45.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:45.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:45.010218+0000) 2022-09-10T15:08:45.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:45 smithi110 ceph-mon[97633]: pgmap v20461: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:45 smithi104 ceph-mon[97044]: pgmap v20461: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:45.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:45.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:45.380769+0000) 2022-09-10T15:08:46.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:46.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:46.010464+0000) 2022-09-10T15:08:46.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:46.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:46.380996+0000) 2022-09-10T15:08:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:47.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:47.010744+0000) 2022-09-10T15:08:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:47 smithi110 ceph-mon[97633]: pgmap v20462: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:08:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:08:47.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:08:47.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:08:47.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:47 smithi104 ceph-mon[97044]: pgmap v20462: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:47.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:08:47.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:08:47.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:08:47.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:08:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:47.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:47.381237+0000) 2022-09-10T15:08:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:48.009+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:48.011013+0000) 2022-09-10T15:08:48.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:48 smithi110 ceph-mon[97633]: pgmap v20463: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:48.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:48 smithi104 ceph-mon[97044]: pgmap v20463: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:48.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:48.380+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:48.381423+0000) 2022-09-10T15:08:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:49.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:49.011169+0000) 2022-09-10T15:08:49.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:49.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:49.381640+0000) 2022-09-10T15:08:50.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:50.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:50.011390+0000) 2022-09-10T15:08:50.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:50.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:50.381867+0000) 2022-09-10T15:08:51.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:51.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:51.011609+0000) 2022-09-10T15:08:51.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:51 smithi110 ceph-mon[97633]: pgmap v20464: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:51.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:51 smithi104 ceph-mon[97044]: pgmap v20464: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:51.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:51.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:51.382159+0000) 2022-09-10T15:08:52.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:08:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:08:51.880+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:08:52.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:52.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:52.011774+0000) 2022-09-10T15:08:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:52.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:52.382384+0000) 2022-09-10T15:08:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:53.010+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:53.012018+0000) 2022-09-10T15:08:53.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:53 smithi110 ceph-mon[97633]: pgmap v20465: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:53 smithi104 ceph-mon[97044]: pgmap v20465: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:53.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:53.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:53.382588+0000) 2022-09-10T15:08:54.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:54.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:54.012287+0000) 2022-09-10T15:08:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:54.381+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:54.382809+0000) 2022-09-10T15:08:55.313 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:55.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:55.012503+0000) 2022-09-10T15:08:55.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:55 smithi110 ceph-mon[97633]: pgmap v20466: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:55.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:55 smithi104 ceph-mon[97044]: pgmap v20466: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:55.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:55.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:55.383059+0000) 2022-09-10T15:08:56.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:56.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:56.012710+0000) 2022-09-10T15:08:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:56.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:56.383328+0000) 2022-09-10T15:08:57.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:57.011+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:57.012947+0000) 2022-09-10T15:08:57.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:57 smithi110 ceph-mon[97633]: pgmap v20467: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:57.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:57 smithi104 ceph-mon[97044]: pgmap v20467: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:08:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:57.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:57.383511+0000) 2022-09-10T15:08:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:58.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:58.013187+0000) 2022-09-10T15:08:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:08:58 smithi110 ceph-mon[97633]: pgmap v20468: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:08:58 smithi104 ceph-mon[97044]: pgmap v20468: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:08:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:58.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:58.383682+0000) 2022-09-10T15:08:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:08:59.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:59.013436+0000) 2022-09-10T15:08:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:08:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:08:59.382+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:08:59.383942+0000) 2022-09-10T15:09:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:00.012+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:00.013641+0000) 2022-09-10T15:09:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:00.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:00.384088+0000) 2022-09-10T15:09:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:01.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:01.013881+0000) 2022-09-10T15:09:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:01 smithi110 ceph-mon[97633]: pgmap v20469: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:01.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:01 smithi104 ceph-mon[97044]: pgmap v20469: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:01.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:01.383+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:01.384296+0000) 2022-09-10T15:09:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:02.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:02.014156+0000) 2022-09-10T15:09:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:02.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:02.384518+0000) 2022-09-10T15:09:03.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:03.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:03.014332+0000) 2022-09-10T15:09:03.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:03 smithi110 ceph-mon[97633]: pgmap v20470: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:03 smithi104 ceph-mon[97044]: pgmap v20470: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:03.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:03.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:03.384772+0000) 2022-09-10T15:09:04.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:04.013+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:04.014492+0000) 2022-09-10T15:09:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:04.384+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:04.384964+0000) 2022-09-10T15:09:05.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:05.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:05.014728+0000) 2022-09-10T15:09:05.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:05 smithi110 ceph-mon[97633]: pgmap v20471: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:05 smithi104 ceph-mon[97044]: pgmap v20471: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:05.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:05.385183+0000) 2022-09-10T15:09:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:06.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:06.014977+0000) 2022-09-10T15:09:06.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:06.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:06.385384+0000) 2022-09-10T15:09:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:07.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:07.015206+0000) 2022-09-10T15:09:07.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:09:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:09:06.896+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:09:07.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:07 smithi110 ceph-mon[97633]: pgmap v20472: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:09:07.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:07 smithi104 ceph-mon[97044]: pgmap v20472: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:09:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:07.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:07.385566+0000) 2022-09-10T15:09:08.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:08.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:08.015446+0000) 2022-09-10T15:09:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:08 smithi110 ceph-mon[97633]: pgmap v20473: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:09:08.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:08 smithi104 ceph-mon[97044]: pgmap v20473: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:09:08.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:08.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:08.385787+0000) 2022-09-10T15:09:09.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:09.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:09.015628+0000) 2022-09-10T15:09:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:09 smithi110 ceph-mon[97633]: osdmap e1360: 12 total, 12 up, 12 in 2022-09-10T15:09:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:09 smithi104 ceph-mon[97044]: osdmap e1360: 12 total, 12 up, 12 in 2022-09-10T15:09:09.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:09.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:09.386015+0000) 2022-09-10T15:09:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:10.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:10.015823+0000) 2022-09-10T15:09:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:10 smithi110 ceph-mon[97633]: pgmap v20475: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:09:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:10 smithi104 ceph-mon[97044]: pgmap v20475: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:09:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:10.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:10.386241+0000) 2022-09-10T15:09:11.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:11.014+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:11.016042+0000) 2022-09-10T15:09:11.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:11.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:11.386477+0000) 2022-09-10T15:09:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:12.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:12.016281+0000) 2022-09-10T15:09:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:12.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:12.386698+0000) 2022-09-10T15:09:13.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:13.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:13.016520+0000) 2022-09-10T15:09:13.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:13 smithi110 ceph-mon[97633]: pgmap v20476: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:09:13.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:13 smithi110 ceph-mon[97633]: osdmap e1361: 12 total, 12 up, 12 in 2022-09-10T15:09:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:13 smithi104 ceph-mon[97044]: pgmap v20476: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:09:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:13 smithi104 ceph-mon[97044]: osdmap e1361: 12 total, 12 up, 12 in 2022-09-10T15:09:13.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:13.385+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:13.386927+0000) 2022-09-10T15:09:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:14.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:14.016658+0000) 2022-09-10T15:09:14.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:14.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:14.387172+0000) 2022-09-10T15:09:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:15.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:15.016881+0000) 2022-09-10T15:09:15.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:15 smithi110 ceph-mon[97633]: pgmap v20478: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:09:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:15 smithi104 ceph-mon[97044]: pgmap v20478: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:09:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:15.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:15.387417+0000) 2022-09-10T15:09:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:16.015+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:16.017061+0000) 2022-09-10T15:09:16.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:16.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:16.387665+0000) 2022-09-10T15:09:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:17.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:17.017312+0000) 2022-09-10T15:09:17.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:17 smithi110 ceph-mon[97633]: pgmap v20479: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:09:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:17 smithi104 ceph-mon[97044]: pgmap v20479: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:09:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:17.386+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:17.387870+0000) 2022-09-10T15:09:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:18.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:18.017459+0000) 2022-09-10T15:09:18.662 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:18 smithi110 ceph-mon[97633]: pgmap v20480: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:09:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:18 smithi104 ceph-mon[97044]: pgmap v20480: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:09:18.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:18.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:18.388084+0000) 2022-09-10T15:09:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:19.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:19.017660+0000) 2022-09-10T15:09:19.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:19.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:19.388306+0000) 2022-09-10T15:09:20.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:20.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:20.017867+0000) 2022-09-10T15:09:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:20.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:20.388465+0000) 2022-09-10T15:09:21.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:21.016+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:21.018047+0000) 2022-09-10T15:09:21.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:21 smithi110 ceph-mon[97633]: pgmap v20481: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:09:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:21 smithi104 ceph-mon[97044]: pgmap v20481: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:09:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:21.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:21.388664+0000) 2022-09-10T15:09:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:22.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:22.018203+0000) 2022-09-10T15:09:22.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:09:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:09:21.907+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:09:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:22.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:22.388823+0000) 2022-09-10T15:09:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:23.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:23.018417+0000) 2022-09-10T15:09:23.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:23 smithi110 ceph-mon[97633]: pgmap v20482: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:09:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:23 smithi104 ceph-mon[97044]: pgmap v20482: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:09:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:23.387+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:23.389018+0000) 2022-09-10T15:09:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:24.017+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:24.018645+0000) 2022-09-10T15:09:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:24.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:24.389229+0000) 2022-09-10T15:09:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:25.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:25.018896+0000) 2022-09-10T15:09:25.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:25 smithi110 ceph-mon[97633]: pgmap v20483: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:09:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:25 smithi104 ceph-mon[97044]: pgmap v20483: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:09:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:25.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:25.389427+0000) 2022-09-10T15:09:26.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:26.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:26.019158+0000) 2022-09-10T15:09:26.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:26.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:26.389586+0000) 2022-09-10T15:09:27.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:27.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:27.019398+0000) 2022-09-10T15:09:27.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:27 smithi110 ceph-mon[97633]: pgmap v20484: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:09:27.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:27 smithi104 ceph-mon[97044]: pgmap v20484: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:09:27.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:27.388+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:27.389755+0000) 2022-09-10T15:09:28.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:28.018+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:28.019624+0000) 2022-09-10T15:09:28.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:28 smithi110 ceph-mon[97633]: pgmap v20485: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:28 smithi104 ceph-mon[97044]: pgmap v20485: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:28.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:28.389998+0000) 2022-09-10T15:09:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:29.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:29.019880+0000) 2022-09-10T15:09:29.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:29.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:29.390251+0000) 2022-09-10T15:09:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:30.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:30.020056+0000) 2022-09-10T15:09:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:30.389+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:30.390478+0000) 2022-09-10T15:09:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:31.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:31.020269+0000) 2022-09-10T15:09:31.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:31 smithi110 ceph-mon[97633]: pgmap v20486: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:31.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:31 smithi104 ceph-mon[97044]: pgmap v20486: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:31.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:31.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:31.390740+0000) 2022-09-10T15:09:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:32.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:32.020448+0000) 2022-09-10T15:09:32.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:32.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:32.390977+0000) 2022-09-10T15:09:33.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:33.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:33.020621+0000) 2022-09-10T15:09:33.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:33 smithi110 ceph-mon[97633]: pgmap v20487: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:33 smithi104 ceph-mon[97044]: pgmap v20487: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:33.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:33.391246+0000) 2022-09-10T15:09:34.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:34.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:34.020892+0000) 2022-09-10T15:09:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:34.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:34.391380+0000) 2022-09-10T15:09:35.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:35.019+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:35.021034+0000) 2022-09-10T15:09:35.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:35 smithi110 ceph-mon[97633]: pgmap v20488: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:35.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:35 smithi104 ceph-mon[97044]: pgmap v20488: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:35.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:35.391580+0000) 2022-09-10T15:09:36.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:36.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:36.021247+0000) 2022-09-10T15:09:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:36.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:36.391760+0000) 2022-09-10T15:09:37.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:09:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:09:36.921+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:09:37.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:37.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:37.021408+0000) 2022-09-10T15:09:37.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:37 smithi104 ceph-mon[97044]: pgmap v20489: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:37 smithi110 ceph-mon[97633]: pgmap v20489: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:37.390+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:37.391953+0000) 2022-09-10T15:09:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:38.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:38.021576+0000) 2022-09-10T15:09:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:38 smithi110 ceph-mon[97633]: pgmap v20490: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:38 smithi104 ceph-mon[97044]: pgmap v20490: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:38.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:38.392174+0000) 2022-09-10T15:09:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:39.020+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:39.021753+0000) 2022-09-10T15:09:39.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:39.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:39.392350+0000) 2022-09-10T15:09:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:40.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:40.022023+0000) 2022-09-10T15:09:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:40.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:40.392569+0000) 2022-09-10T15:09:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:41.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:41.022272+0000) 2022-09-10T15:09:41.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:41 smithi110 ceph-mon[97633]: pgmap v20491: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:41 smithi104 ceph-mon[97044]: pgmap v20491: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:41.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:41.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:41.392840+0000) 2022-09-10T15:09:42.314 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:42.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:42.022475+0000) 2022-09-10T15:09:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:42.391+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:42.393081+0000) 2022-09-10T15:09:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:43.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:43.022734+0000) 2022-09-10T15:09:43.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:43 smithi110 ceph-mon[97633]: pgmap v20492: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:43.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:43 smithi104 ceph-mon[97044]: pgmap v20492: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:43.813 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:43.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:43.393320+0000) 2022-09-10T15:09:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:44.021+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:44.022928+0000) 2022-09-10T15:09:44.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:44.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:44.393506+0000) 2022-09-10T15:09:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:45.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:45.023207+0000) 2022-09-10T15:09:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:45 smithi104 ceph-mon[97044]: pgmap v20493: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:45 smithi110 ceph-mon[97633]: pgmap v20493: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:45.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:45.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:45.393728+0000) 2022-09-10T15:09:46.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:46.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:46.023385+0000) 2022-09-10T15:09:46.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:46.392+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:46.393937+0000) 2022-09-10T15:09:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:47.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:47.023575+0000) 2022-09-10T15:09:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:47 smithi104 ceph-mon[97044]: pgmap v20494: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:09:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:09:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:09:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:09:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:47 smithi110 ceph-mon[97633]: pgmap v20494: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:09:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:09:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:09:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:09:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:47.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:47.394192+0000) 2022-09-10T15:09:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:48.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:48.023789+0000) 2022-09-10T15:09:48.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:48 smithi110 ceph-mon[97633]: pgmap v20495: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:48.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:48 smithi104 ceph-mon[97044]: pgmap v20495: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:48.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:48.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:48.394426+0000) 2022-09-10T15:09:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:49.022+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:49.024067+0000) 2022-09-10T15:09:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:49.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:49.394664+0000) 2022-09-10T15:09:50.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:50.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:50.024302+0000) 2022-09-10T15:09:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:50.393+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:50.394883+0000) 2022-09-10T15:09:51.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:51.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:51.024563+0000) 2022-09-10T15:09:51.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:51 smithi110 ceph-mon[97633]: pgmap v20496: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:51 smithi104 ceph-mon[97044]: pgmap v20496: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:51.394+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:51.395091+0000) 2022-09-10T15:09:52.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:09:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:09:51.925+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:09:52.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:52.023+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:52.024821+0000) 2022-09-10T15:09:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:52.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:52.395260+0000) 2022-09-10T15:09:53.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:53.024+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:53.025063+0000) 2022-09-10T15:09:53.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:53 smithi104 ceph-mon[97044]: pgmap v20497: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:53 smithi110 ceph-mon[97633]: pgmap v20497: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:53.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:53.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:53.395491+0000) 2022-09-10T15:09:54.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:54.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:54.025361+0000) 2022-09-10T15:09:54.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:54.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:54.395751+0000) 2022-09-10T15:09:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:55.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:55.025614+0000) 2022-09-10T15:09:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:55 smithi104 ceph-mon[97044]: pgmap v20498: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:55 smithi110 ceph-mon[97633]: pgmap v20498: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:55.395+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:55.396004+0000) 2022-09-10T15:09:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:56.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:56.025845+0000) 2022-09-10T15:09:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:56.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:56.396232+0000) 2022-09-10T15:09:57.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:57.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:57.026045+0000) 2022-09-10T15:09:57.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:57 smithi104 ceph-mon[97044]: pgmap v20499: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:57 smithi110 ceph-mon[97633]: pgmap v20499: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:09:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:57.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:57.396478+0000) 2022-09-10T15:09:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:58.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:58.026299+0000) 2022-09-10T15:09:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:09:58 smithi110 ceph-mon[97633]: pgmap v20500: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:09:58 smithi104 ceph-mon[97044]: pgmap v20500: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:09:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:58.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:58.396727+0000) 2022-09-10T15:09:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:09:59.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:59.026550+0000) 2022-09-10T15:09:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:09:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:09:59.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:09:59.396911+0000) 2022-09-10T15:10:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:00.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:00.026705+0000) 2022-09-10T15:10:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:10:00.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:10:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:00.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:00.397146+0000) 2022-09-10T15:10:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:01.025+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:01.026880+0000) 2022-09-10T15:10:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:01 smithi110 ceph-mon[97633]: pgmap v20501: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:01.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:01 smithi104 ceph-mon[97044]: pgmap v20501: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:01.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:01.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:01.397387+0000) 2022-09-10T15:10:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:02.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:02.027171+0000) 2022-09-10T15:10:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:02.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:02.397598+0000) 2022-09-10T15:10:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:03.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:03.027387+0000) 2022-09-10T15:10:03.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:03 smithi104 ceph-mon[97044]: pgmap v20502: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:03.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:03 smithi110 ceph-mon[97633]: pgmap v20502: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:03.396+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:03.397836+0000) 2022-09-10T15:10:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:04.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:04.027573+0000) 2022-09-10T15:10:04.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:04.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:04.398139+0000) 2022-09-10T15:10:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:05.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:05.027777+0000) 2022-09-10T15:10:05.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:05 smithi104 ceph-mon[97044]: pgmap v20503: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:05.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:05 smithi110 ceph-mon[97633]: pgmap v20503: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:05.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:05.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:05.398346+0000) 2022-09-10T15:10:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:06.026+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:06.028005+0000) 2022-09-10T15:10:06.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:06.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:06.398553+0000) 2022-09-10T15:10:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:07.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:07.028223+0000) 2022-09-10T15:10:07.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:10:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:10:06.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:10:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:07 smithi104 ceph-mon[97044]: pgmap v20504: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:10:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:07 smithi110 ceph-mon[97633]: pgmap v20504: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:10:07.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:07.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:07.398772+0000) 2022-09-10T15:10:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:08.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:08.028444+0000) 2022-09-10T15:10:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:08 smithi110 ceph-mon[97633]: pgmap v20505: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:10:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:08 smithi104 ceph-mon[97044]: pgmap v20505: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:10:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:08.397+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:08.399024+0000) 2022-09-10T15:10:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:09.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:09.028683+0000) 2022-09-10T15:10:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:09 smithi110 ceph-mon[97633]: osdmap e1362: 12 total, 12 up, 12 in 2022-09-10T15:10:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:09 smithi104 ceph-mon[97044]: osdmap e1362: 12 total, 12 up, 12 in 2022-09-10T15:10:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:09.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:09.399216+0000) 2022-09-10T15:10:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:10.027+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:10.028851+0000) 2022-09-10T15:10:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:10 smithi110 ceph-mon[97633]: pgmap v20507: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:10:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:10 smithi104 ceph-mon[97044]: pgmap v20507: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:10:10.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:10.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:10.399449+0000) 2022-09-10T15:10:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:11.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:11.029070+0000) 2022-09-10T15:10:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:11.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:11.399638+0000) 2022-09-10T15:10:12.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:12.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:12.029339+0000) 2022-09-10T15:10:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:12.398+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:12.399865+0000) 2022-09-10T15:10:13.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:13.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:13.029542+0000) 2022-09-10T15:10:13.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:13 smithi104 ceph-mon[97044]: pgmap v20508: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:10:13.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:13 smithi104 ceph-mon[97044]: osdmap e1363: 12 total, 12 up, 12 in 2022-09-10T15:10:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:13 smithi110 ceph-mon[97633]: pgmap v20508: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:10:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:13 smithi110 ceph-mon[97633]: osdmap e1363: 12 total, 12 up, 12 in 2022-09-10T15:10:13.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:13.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:13.400081+0000) 2022-09-10T15:10:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:14.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:14.029755+0000) 2022-09-10T15:10:14.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:14.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:14.400342+0000) 2022-09-10T15:10:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:15.028+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:15.029999+0000) 2022-09-10T15:10:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:15 smithi104 ceph-mon[97044]: pgmap v20510: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:10:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:15 smithi110 ceph-mon[97633]: pgmap v20510: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:10:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:15.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:15.400561+0000) 2022-09-10T15:10:16.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:16.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:16.030210+0000) 2022-09-10T15:10:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:16.399+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:16.400785+0000) 2022-09-10T15:10:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:17.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:17.030472+0000) 2022-09-10T15:10:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:17 smithi104 ceph-mon[97044]: pgmap v20511: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:10:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:17 smithi110 ceph-mon[97633]: pgmap v20511: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:10:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:17.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:17.401007+0000) 2022-09-10T15:10:18.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:18.029+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:18.030682+0000) 2022-09-10T15:10:18.668 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:18 smithi110 ceph-mon[97633]: pgmap v20512: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:10:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:18 smithi104 ceph-mon[97044]: pgmap v20512: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:10:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:18.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:18.401233+0000) 2022-09-10T15:10:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:19.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:19.030903+0000) 2022-09-10T15:10:19.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:19.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:19.401445+0000) 2022-09-10T15:10:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:20.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:20.031146+0000) 2022-09-10T15:10:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:20.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:20.401596+0000) 2022-09-10T15:10:21.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:21.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:21.031329+0000) 2022-09-10T15:10:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:21 smithi104 ceph-mon[97044]: pgmap v20513: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:10:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:21 smithi110 ceph-mon[97633]: pgmap v20513: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:10:21.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:21.400+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:21.401851+0000) 2022-09-10T15:10:22.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:10:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:10:21.941+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:10:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:22.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:22.031554+0000) 2022-09-10T15:10:22.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:22.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:22.402081+0000) 2022-09-10T15:10:23.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:23.030+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:23.031715+0000) 2022-09-10T15:10:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:23 smithi104 ceph-mon[97044]: pgmap v20514: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:10:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:23 smithi110 ceph-mon[97633]: pgmap v20514: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:10:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:23.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:23.402328+0000) 2022-09-10T15:10:24.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:24.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:24.031936+0000) 2022-09-10T15:10:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:24.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:24.402578+0000) 2022-09-10T15:10:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:25.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:25.032193+0000) 2022-09-10T15:10:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:25 smithi104 ceph-mon[97044]: pgmap v20515: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:10:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:25 smithi110 ceph-mon[97633]: pgmap v20515: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:10:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:25.401+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:25.402827+0000) 2022-09-10T15:10:26.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:26.031+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:26.032410+0000) 2022-09-10T15:10:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:26.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:26.403018+0000) 2022-09-10T15:10:27.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:27.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:27.032680+0000) 2022-09-10T15:10:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:27 smithi104 ceph-mon[97044]: pgmap v20516: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:10:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:27 smithi110 ceph-mon[97633]: pgmap v20516: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:10:27.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:27.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:27.403211+0000) 2022-09-10T15:10:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:28.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:28.032930+0000) 2022-09-10T15:10:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:28 smithi110 ceph-mon[97633]: pgmap v20517: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:28.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:28 smithi104 ceph-mon[97044]: pgmap v20517: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:28.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:28.402+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:28.403399+0000) 2022-09-10T15:10:29.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:29.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:29.033166+0000) 2022-09-10T15:10:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:29.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:29.403618+0000) 2022-09-10T15:10:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:30.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:30.033343+0000) 2022-09-10T15:10:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:30.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:30.403847+0000) 2022-09-10T15:10:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:31.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:31.033558+0000) 2022-09-10T15:10:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:31 smithi104 ceph-mon[97044]: pgmap v20518: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:31 smithi110 ceph-mon[97633]: pgmap v20518: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:31.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:31.404060+0000) 2022-09-10T15:10:32.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:32.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:32.033737+0000) 2022-09-10T15:10:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:32.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:32.404246+0000) 2022-09-10T15:10:33.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:33.032+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:33.033941+0000) 2022-09-10T15:10:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:33 smithi104 ceph-mon[97044]: pgmap v20519: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:33 smithi110 ceph-mon[97633]: pgmap v20519: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:33.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:33.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:33.404370+0000) 2022-09-10T15:10:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:34.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:34.034145+0000) 2022-09-10T15:10:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:34.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:34.404564+0000) 2022-09-10T15:10:35.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:35.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:35.034366+0000) 2022-09-10T15:10:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:35 smithi104 ceph-mon[97044]: pgmap v20520: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:35 smithi110 ceph-mon[97633]: pgmap v20520: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:35.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:35.404694+0000) 2022-09-10T15:10:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:36.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:36.034595+0000) 2022-09-10T15:10:36.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:36.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:36.404857+0000) 2022-09-10T15:10:37.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:10:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:10:36.943+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:10:37.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:37.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:37.034857+0000) 2022-09-10T15:10:37.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:37 smithi104 ceph-mon[97044]: pgmap v20521: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:37 smithi110 ceph-mon[97633]: pgmap v20521: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:37.403+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:37.405066+0000) 2022-09-10T15:10:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:38.033+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:38.035059+0000) 2022-09-10T15:10:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:38 smithi110 ceph-mon[97633]: pgmap v20522: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:38 smithi104 ceph-mon[97044]: pgmap v20522: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:38.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:38.405250+0000) 2022-09-10T15:10:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:39.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:39.035292+0000) 2022-09-10T15:10:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:39.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:39.405434+0000) 2022-09-10T15:10:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:40.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:40.035492+0000) 2022-09-10T15:10:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:40.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:40.405563+0000) 2022-09-10T15:10:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:41.034+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:41.035679+0000) 2022-09-10T15:10:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:41 smithi104 ceph-mon[97044]: pgmap v20523: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:41 smithi110 ceph-mon[97633]: pgmap v20523: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:41.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:41.405710+0000) 2022-09-10T15:10:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:42.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:42.035908+0000) 2022-09-10T15:10:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:42.404+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:42.405944+0000) 2022-09-10T15:10:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:43.035+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:43.036119+0000) 2022-09-10T15:10:43.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:43 smithi104 ceph-mon[97044]: pgmap v20524: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:43 smithi110 ceph-mon[97633]: pgmap v20524: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:43.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:43.406168+0000) 2022-09-10T15:10:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:44.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:44.036397+0000) 2022-09-10T15:10:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:44.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:44.406412+0000) 2022-09-10T15:10:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:45.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:45.036637+0000) 2022-09-10T15:10:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:45 smithi104 ceph-mon[97044]: pgmap v20525: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:45 smithi110 ceph-mon[97633]: pgmap v20525: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:45.405+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:45.406549+0000) 2022-09-10T15:10:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:46.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:46.036891+0000) 2022-09-10T15:10:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:46.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:46.406730+0000) 2022-09-10T15:10:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:47.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:47.037138+0000) 2022-09-10T15:10:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:47 smithi104 ceph-mon[97044]: pgmap v20526: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:10:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:10:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:10:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:10:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:47 smithi110 ceph-mon[97633]: pgmap v20526: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:10:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:10:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:10:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:10:47.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:47.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:47.406937+0000) 2022-09-10T15:10:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:48.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:48.037316+0000) 2022-09-10T15:10:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:48 smithi110 ceph-mon[97633]: pgmap v20527: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:48.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:48 smithi104 ceph-mon[97044]: pgmap v20527: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:48.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:48.407179+0000) 2022-09-10T15:10:49.307 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:49.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:49.037539+0000) 2022-09-10T15:10:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:49.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:49.407373+0000) 2022-09-10T15:10:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:50.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:50.037778+0000) 2022-09-10T15:10:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:50.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:50.407542+0000) 2022-09-10T15:10:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:51.036+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:51.037998+0000) 2022-09-10T15:10:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:51 smithi104 ceph-mon[97044]: pgmap v20528: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:51 smithi110 ceph-mon[97633]: pgmap v20528: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:51.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:51.407748+0000) 2022-09-10T15:10:52.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:10:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:10:51.950+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:10:52.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:52.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:52.038231+0000) 2022-09-10T15:10:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:52.406+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:52.407879+0000) 2022-09-10T15:10:53.308 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:53.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:53.038445+0000) 2022-09-10T15:10:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:53 smithi104 ceph-mon[97044]: pgmap v20529: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:53.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:53 smithi110 ceph-mon[97633]: pgmap v20529: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:53.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:53.408127+0000) 2022-09-10T15:10:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:54.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:54.038677+0000) 2022-09-10T15:10:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:54.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:54.408360+0000) 2022-09-10T15:10:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:55.037+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:55.038902+0000) 2022-09-10T15:10:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:55 smithi104 ceph-mon[97044]: pgmap v20530: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:55 smithi110 ceph-mon[97633]: pgmap v20530: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:55.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:55.408574+0000) 2022-09-10T15:10:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:56.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:56.039170+0000) 2022-09-10T15:10:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:56.407+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:56.408833+0000) 2022-09-10T15:10:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:57.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:57.039365+0000) 2022-09-10T15:10:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:57 smithi104 ceph-mon[97044]: pgmap v20531: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:57.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:57 smithi110 ceph-mon[97633]: pgmap v20531: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:10:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:57.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:57.409073+0000) 2022-09-10T15:10:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:58.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:58.039558+0000) 2022-09-10T15:10:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:10:58 smithi110 ceph-mon[97633]: pgmap v20532: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:10:58 smithi104 ceph-mon[97044]: pgmap v20532: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:10:58.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:58.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:58.409241+0000) 2022-09-10T15:10:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:10:59.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:59.039818+0000) 2022-09-10T15:10:59.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:10:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:10:59.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:10:59.409474+0000) 2022-09-10T15:11:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:00.038+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:00.039985+0000) 2022-09-10T15:11:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:00.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:00.409657+0000) 2022-09-10T15:11:01.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:01.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:01.040206+0000) 2022-09-10T15:11:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:01 smithi104 ceph-mon[97044]: pgmap v20533: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:01 smithi110 ceph-mon[97633]: pgmap v20533: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:01.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:01.409848+0000) 2022-09-10T15:11:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:02.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:02.040416+0000) 2022-09-10T15:11:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:02.408+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:02.410052+0000) 2022-09-10T15:11:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:03.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:03.040678+0000) 2022-09-10T15:11:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:03 smithi104 ceph-mon[97044]: pgmap v20534: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:03 smithi110 ceph-mon[97633]: pgmap v20534: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:03.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:03.410269+0000) 2022-09-10T15:11:04.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:04.039+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:04.040910+0000) 2022-09-10T15:11:04.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:04.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:04.410460+0000) 2022-09-10T15:11:05.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:05.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:05.041091+0000) 2022-09-10T15:11:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:05 smithi104 ceph-mon[97044]: pgmap v20535: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:05 smithi110 ceph-mon[97633]: pgmap v20535: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:05.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:05.409+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:05.410665+0000) 2022-09-10T15:11:06.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:06.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:06.041299+0000) 2022-09-10T15:11:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:06.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:06.410889+0000) 2022-09-10T15:11:07.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:11:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:11:06.964+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:11:07.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:07.040+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:07.041546+0000) 2022-09-10T15:11:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:07 smithi104 ceph-mon[97044]: pgmap v20536: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:11:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:07 smithi110 ceph-mon[97633]: pgmap v20536: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:11:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:07.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:07.411176+0000) 2022-09-10T15:11:08.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:08.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:08.041801+0000) 2022-09-10T15:11:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:08 smithi110 ceph-mon[97633]: pgmap v20537: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:11:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:08 smithi104 ceph-mon[97044]: pgmap v20537: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:11:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:08.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:08.411358+0000) 2022-09-10T15:11:09.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:09.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:09.042000+0000) 2022-09-10T15:11:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:09 smithi110 ceph-mon[97633]: osdmap e1364: 12 total, 12 up, 12 in 2022-09-10T15:11:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:09 smithi104 ceph-mon[97044]: osdmap e1364: 12 total, 12 up, 12 in 2022-09-10T15:11:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:09.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:09.411539+0000) 2022-09-10T15:11:10.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:10.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:10.042209+0000) 2022-09-10T15:11:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:10 smithi110 ceph-mon[97633]: pgmap v20539: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:11:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:10 smithi104 ceph-mon[97044]: pgmap v20539: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:11:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:10.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:10.411707+0000) 2022-09-10T15:11:11.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:11.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:11.042445+0000) 2022-09-10T15:11:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:11.410+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:11.411975+0000) 2022-09-10T15:11:12.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:12.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:12.042657+0000) 2022-09-10T15:11:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:12.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:12.412200+0000) 2022-09-10T15:11:13.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:13.041+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:13.042871+0000) 2022-09-10T15:11:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:13 smithi104 ceph-mon[97044]: pgmap v20540: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:11:13.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:13 smithi104 ceph-mon[97044]: osdmap e1365: 12 total, 12 up, 12 in 2022-09-10T15:11:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:13 smithi110 ceph-mon[97633]: pgmap v20540: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:11:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:13 smithi110 ceph-mon[97633]: osdmap e1365: 12 total, 12 up, 12 in 2022-09-10T15:11:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:13.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:13.412368+0000) 2022-09-10T15:11:14.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:14.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:14.043046+0000) 2022-09-10T15:11:14.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:14.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:14.412521+0000) 2022-09-10T15:11:15.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:15.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:15.043273+0000) 2022-09-10T15:11:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:15 smithi104 ceph-mon[97044]: pgmap v20542: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:11:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:15 smithi110 ceph-mon[97633]: pgmap v20542: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:11:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:15.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:15.412743+0000) 2022-09-10T15:11:16.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:16.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:16.043551+0000) 2022-09-10T15:11:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:16.411+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:16.412941+0000) 2022-09-10T15:11:17.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:17.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:17.043800+0000) 2022-09-10T15:11:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:17 smithi104 ceph-mon[97044]: pgmap v20543: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:11:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:17 smithi110 ceph-mon[97633]: pgmap v20543: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:11:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:17.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:17.413199+0000) 2022-09-10T15:11:18.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:18.042+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:18.044007+0000) 2022-09-10T15:11:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:18 smithi110 ceph-mon[97633]: pgmap v20544: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:11:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:18 smithi104 ceph-mon[97044]: pgmap v20544: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:11:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:18.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:18.413377+0000) 2022-09-10T15:11:19.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:19.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:19.044242+0000) 2022-09-10T15:11:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:19.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:19.413528+0000) 2022-09-10T15:11:20.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:20.043+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:20.044466+0000) 2022-09-10T15:11:20.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:20.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:20.413738+0000) 2022-09-10T15:11:21.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:21.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:21.044711+0000) 2022-09-10T15:11:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:21 smithi104 ceph-mon[97044]: pgmap v20545: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:11:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:21 smithi110 ceph-mon[97633]: pgmap v20545: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:11:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:21.412+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:21.413957+0000) 2022-09-10T15:11:22.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:11:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:11:21.970+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:11:22.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:22.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:22.044951+0000) 2022-09-10T15:11:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:22.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:22.414145+0000) 2022-09-10T15:11:23.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:23.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:23.045198+0000) 2022-09-10T15:11:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:23 smithi104 ceph-mon[97044]: pgmap v20546: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:11:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:23 smithi110 ceph-mon[97633]: pgmap v20546: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:11:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:23.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:23.414316+0000) 2022-09-10T15:11:24.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:24.044+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:24.045433+0000) 2022-09-10T15:11:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:24.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:24.414476+0000) 2022-09-10T15:11:25.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:25.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:25.045673+0000) 2022-09-10T15:11:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:25 smithi104 ceph-mon[97044]: pgmap v20547: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:11:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:25 smithi110 ceph-mon[97633]: pgmap v20547: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:11:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:25.413+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:25.414654+0000) 2022-09-10T15:11:26.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:26.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:26.045888+0000) 2022-09-10T15:11:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:26.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:26.414821+0000) 2022-09-10T15:11:27.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:27.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:27.046150+0000) 2022-09-10T15:11:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:27 smithi104 ceph-mon[97044]: pgmap v20548: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:11:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:27 smithi110 ceph-mon[97633]: pgmap v20548: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:11:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:27.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:27.414979+0000) 2022-09-10T15:11:28.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:28.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:28.046352+0000) 2022-09-10T15:11:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:28 smithi110 ceph-mon[97633]: pgmap v20549: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:28 smithi104 ceph-mon[97044]: pgmap v20549: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:28.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:28.415214+0000) 2022-09-10T15:11:29.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:29.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:29.046531+0000) 2022-09-10T15:11:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:29.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:29.415488+0000) 2022-09-10T15:11:30.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:30.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:30.046685+0000) 2022-09-10T15:11:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:30.414+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:30.415684+0000) 2022-09-10T15:11:31.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:31.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:31.046823+0000) 2022-09-10T15:11:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:31 smithi104 ceph-mon[97044]: pgmap v20550: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:31 smithi110 ceph-mon[97633]: pgmap v20550: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:31.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:31.415901+0000) 2022-09-10T15:11:32.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:32.045+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:32.047043+0000) 2022-09-10T15:11:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:32.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:32.416139+0000) 2022-09-10T15:11:33.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:33.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:33.047210+0000) 2022-09-10T15:11:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:33 smithi104 ceph-mon[97044]: pgmap v20551: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:33 smithi110 ceph-mon[97633]: pgmap v20551: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:33.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:33.416308+0000) 2022-09-10T15:11:34.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:34.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:34.047355+0000) 2022-09-10T15:11:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:34.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:34.416534+0000) 2022-09-10T15:11:35.315 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:35.046+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:35.047521+0000) 2022-09-10T15:11:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:35 smithi104 ceph-mon[97044]: pgmap v20552: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:35 smithi110 ceph-mon[97633]: pgmap v20552: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:35.415+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:35.416789+0000) 2022-09-10T15:11:36.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:36.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:36.047688+0000) 2022-09-10T15:11:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:36.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:36.416945+0000) 2022-09-10T15:11:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:37.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:37.047917+0000) 2022-09-10T15:11:37.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:11:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:11:36.972+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:11:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:37 smithi104 ceph-mon[97044]: pgmap v20553: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:37 smithi110 ceph-mon[97633]: pgmap v20553: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:37.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:37.417204+0000) 2022-09-10T15:11:38.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:38.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:38.048114+0000) 2022-09-10T15:11:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:38 smithi110 ceph-mon[97633]: pgmap v20554: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:38 smithi104 ceph-mon[97044]: pgmap v20554: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:38.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:38.417426+0000) 2022-09-10T15:11:39.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:39.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:39.048316+0000) 2022-09-10T15:11:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:39.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:39.417642+0000) 2022-09-10T15:11:40.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:40.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:40.048511+0000) 2022-09-10T15:11:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:40.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:40.417771+0000) 2022-09-10T15:11:41.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:41.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:41.048709+0000) 2022-09-10T15:11:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:41 smithi104 ceph-mon[97044]: pgmap v20555: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:41 smithi110 ceph-mon[97633]: pgmap v20555: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:41.416+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:41.418009+0000) 2022-09-10T15:11:42.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:42.047+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:42.048907+0000) 2022-09-10T15:11:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:42.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:42.418235+0000) 2022-09-10T15:11:43.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:43.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:43.049067+0000) 2022-09-10T15:11:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:43 smithi104 ceph-mon[97044]: pgmap v20556: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:43 smithi110 ceph-mon[97633]: pgmap v20556: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:43.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:43.418417+0000) 2022-09-10T15:11:44.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:44.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:44.049306+0000) 2022-09-10T15:11:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:44.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:44.418682+0000) 2022-09-10T15:11:45.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:45.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:45.049503+0000) 2022-09-10T15:11:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:45 smithi104 ceph-mon[97044]: pgmap v20557: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:45 smithi110 ceph-mon[97633]: pgmap v20557: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:45.417+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:45.418903+0000) 2022-09-10T15:11:46.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:46.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:46.049661+0000) 2022-09-10T15:11:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:46.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:46.419083+0000) 2022-09-10T15:11:47.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:47.048+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:47.049868+0000) 2022-09-10T15:11:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:47 smithi104 ceph-mon[97044]: pgmap v20558: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:11:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:11:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:11:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:11:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:47 smithi110 ceph-mon[97633]: pgmap v20558: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:11:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:11:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:11:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:11:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:47.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:47.419351+0000) 2022-09-10T15:11:48.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:48.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:48.050137+0000) 2022-09-10T15:11:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:48 smithi110 ceph-mon[97633]: pgmap v20559: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:48 smithi104 ceph-mon[97044]: pgmap v20559: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:48.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:48.419587+0000) 2022-09-10T15:11:49.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:49.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:49.050379+0000) 2022-09-10T15:11:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:49.418+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:49.419866+0000) 2022-09-10T15:11:50.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:50.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:50.050594+0000) 2022-09-10T15:11:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:50.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:50.420163+0000) 2022-09-10T15:11:51.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:51.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:51.050823+0000) 2022-09-10T15:11:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:51 smithi104 ceph-mon[97044]: pgmap v20560: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:51 smithi110 ceph-mon[97633]: pgmap v20560: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:51.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:51.420377+0000) 2022-09-10T15:11:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:52.049+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:52.051013+0000) 2022-09-10T15:11:52.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:11:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:11:51.978+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:11:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:52.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:52.420592+0000) 2022-09-10T15:11:53.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:53.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:53.051240+0000) 2022-09-10T15:11:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:53 smithi104 ceph-mon[97044]: pgmap v20561: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:53 smithi110 ceph-mon[97633]: pgmap v20561: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:53.419+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:53.420845+0000) 2022-09-10T15:11:54.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:54.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:54.051460+0000) 2022-09-10T15:11:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:54.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:54.421037+0000) 2022-09-10T15:11:55.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:55.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:55.051670+0000) 2022-09-10T15:11:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:55 smithi104 ceph-mon[97044]: pgmap v20562: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:55 smithi110 ceph-mon[97633]: pgmap v20562: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:55.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:55.421233+0000) 2022-09-10T15:11:56.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:56.050+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:56.051921+0000) 2022-09-10T15:11:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:56.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:56.421404+0000) 2022-09-10T15:11:57.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:57.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:57.052197+0000) 2022-09-10T15:11:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:57 smithi104 ceph-mon[97044]: pgmap v20563: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:57 smithi110 ceph-mon[97633]: pgmap v20563: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:11:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:57.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:57.421638+0000) 2022-09-10T15:11:58.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:58.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:58.052406+0000) 2022-09-10T15:11:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:11:58 smithi110 ceph-mon[97633]: pgmap v20564: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:11:58 smithi104 ceph-mon[97044]: pgmap v20564: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:11:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:58.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:58.421818+0000) 2022-09-10T15:11:59.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:11:59.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:59.052567+0000) 2022-09-10T15:11:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:11:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:11:59.420+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:11:59.422037+0000) 2022-09-10T15:12:00.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:00.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:00.052817+0000) 2022-09-10T15:12:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:00.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:00.422267+0000) 2022-09-10T15:12:01.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:01.051+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:01.052982+0000) 2022-09-10T15:12:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:01 smithi104 ceph-mon[97044]: pgmap v20565: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:01 smithi110 ceph-mon[97633]: pgmap v20565: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:01.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:01.422505+0000) 2022-09-10T15:12:02.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:02.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:02.053227+0000) 2022-09-10T15:12:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:02.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:02.422662+0000) 2022-09-10T15:12:03.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:03.052+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:03.053430+0000) 2022-09-10T15:12:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:03 smithi104 ceph-mon[97044]: pgmap v20566: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:03 smithi110 ceph-mon[97633]: pgmap v20566: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:03.421+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:03.422902+0000) 2022-09-10T15:12:04.425 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:04.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:04.053605+0000) 2022-09-10T15:12:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:04.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:04.423173+0000) 2022-09-10T15:12:05.425 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:05.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:05.053835+0000) 2022-09-10T15:12:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:05 smithi104 ceph-mon[97044]: pgmap v20567: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:05 smithi110 ceph-mon[97633]: pgmap v20567: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:05.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:05.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:05.423413+0000) 2022-09-10T15:12:06.425 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:06.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:06.054051+0000) 2022-09-10T15:12:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:06.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:06.423589+0000) 2022-09-10T15:12:07.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:12:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:12:06.984+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:12:07.306 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:07.053+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:07.054371+0000) 2022-09-10T15:12:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:07 smithi104 ceph-mon[97044]: pgmap v20568: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:12:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:07 smithi110 ceph-mon[97633]: pgmap v20568: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:12:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:07.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:07.423836+0000) 2022-09-10T15:12:08.426 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:08.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:08.054591+0000) 2022-09-10T15:12:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:08 smithi110 ceph-mon[97633]: pgmap v20569: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:12:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:08 smithi104 ceph-mon[97044]: pgmap v20569: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:12:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:08.422+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:08.424061+0000) 2022-09-10T15:12:09.426 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:09.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:09.054822+0000) 2022-09-10T15:12:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:09 smithi110 ceph-mon[97633]: osdmap e1366: 12 total, 12 up, 12 in 2022-09-10T15:12:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:09 smithi104 ceph-mon[97044]: osdmap e1366: 12 total, 12 up, 12 in 2022-09-10T15:12:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:09.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:09.424287+0000) 2022-09-10T15:12:10.426 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:10.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:10.055057+0000) 2022-09-10T15:12:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:10 smithi110 ceph-mon[97633]: pgmap v20571: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:12:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:10 smithi104 ceph-mon[97044]: pgmap v20571: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:12:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:10.423+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:10.424475+0000) 2022-09-10T15:12:11.426 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:11.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:11.055304+0000) 2022-09-10T15:12:11.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:11.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:11.424737+0000) 2022-09-10T15:12:12.427 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:12.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:12.055525+0000) 2022-09-10T15:12:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:12.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:12.424978+0000) 2022-09-10T15:12:13.427 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:13.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:13.055686+0000) 2022-09-10T15:12:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:13 smithi104 ceph-mon[97044]: pgmap v20572: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:12:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:13 smithi104 ceph-mon[97044]: osdmap e1367: 12 total, 12 up, 12 in 2022-09-10T15:12:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:13 smithi110 ceph-mon[97633]: pgmap v20572: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:12:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:13 smithi110 ceph-mon[97633]: osdmap e1367: 12 total, 12 up, 12 in 2022-09-10T15:12:13.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:13.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:13.425155+0000) 2022-09-10T15:12:14.427 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:14.054+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:14.055921+0000) 2022-09-10T15:12:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:14.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:14.425343+0000) 2022-09-10T15:12:15.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:15.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:15.056172+0000) 2022-09-10T15:12:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:15 smithi104 ceph-mon[97044]: pgmap v20574: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:12:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:15 smithi110 ceph-mon[97633]: pgmap v20574: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:12:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:15.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:15.425576+0000) 2022-09-10T15:12:16.427 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:16.055+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:16.056370+0000) 2022-09-10T15:12:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:16.424+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:16.425759+0000) 2022-09-10T15:12:17.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:17.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:17.056619+0000) 2022-09-10T15:12:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:17 smithi104 ceph-mon[97044]: pgmap v20575: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:12:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:17 smithi110 ceph-mon[97633]: pgmap v20575: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:12:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:17.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:17.425978+0000) 2022-09-10T15:12:18.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:18.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:18.056798+0000) 2022-09-10T15:12:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:18 smithi110 ceph-mon[97633]: pgmap v20576: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:12:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:18 smithi104 ceph-mon[97044]: pgmap v20576: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:12:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:18.425+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:18.426222+0000) 2022-09-10T15:12:19.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:19.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:19.057047+0000) 2022-09-10T15:12:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:19.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:19.426413+0000) 2022-09-10T15:12:20.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:20.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:20.057244+0000) 2022-09-10T15:12:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:20.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:20.426614+0000) 2022-09-10T15:12:21.428 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:21.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:21.057498+0000) 2022-09-10T15:12:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:21 smithi104 ceph-mon[97044]: pgmap v20577: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:12:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:21 smithi110 ceph-mon[97633]: pgmap v20577: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:12:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:21.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:21.426876+0000) 2022-09-10T15:12:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:22.056+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:22.057690+0000) 2022-09-10T15:12:22.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:12:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:12:21.998+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:12:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:22.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:22.427116+0000) 2022-09-10T15:12:23.429 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:23.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:23.057933+0000) 2022-09-10T15:12:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:23 smithi104 ceph-mon[97044]: pgmap v20578: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:12:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:23 smithi110 ceph-mon[97633]: pgmap v20578: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:12:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:23.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:23.427334+0000) 2022-09-10T15:12:24.429 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:24.057+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:24.058201+0000) 2022-09-10T15:12:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:24.426+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:24.427486+0000) 2022-09-10T15:12:25.429 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:25.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:25.058379+0000) 2022-09-10T15:12:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:25 smithi104 ceph-mon[97044]: pgmap v20579: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:12:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:25 smithi110 ceph-mon[97633]: pgmap v20579: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:12:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:25.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:25.427717+0000) 2022-09-10T15:12:26.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:26.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:26.058580+0000) 2022-09-10T15:12:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:26.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:26.427944+0000) 2022-09-10T15:12:27.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:27.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:27.058790+0000) 2022-09-10T15:12:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:27 smithi104 ceph-mon[97044]: pgmap v20580: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:12:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:27 smithi110 ceph-mon[97633]: pgmap v20580: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:12:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:27.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:27.428158+0000) 2022-09-10T15:12:28.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:28.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:28.059014+0000) 2022-09-10T15:12:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:28 smithi110 ceph-mon[97633]: pgmap v20581: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:28 smithi104 ceph-mon[97044]: pgmap v20581: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:28.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:28.428344+0000) 2022-09-10T15:12:29.430 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:29.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:29.059283+0000) 2022-09-10T15:12:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:29.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:29.428575+0000) 2022-09-10T15:12:30.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:30.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:30.059527+0000) 2022-09-10T15:12:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:30.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:30.428812+0000) 2022-09-10T15:12:31.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:31.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:31.059801+0000) 2022-09-10T15:12:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:31 smithi104 ceph-mon[97044]: pgmap v20582: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:31 smithi110 ceph-mon[97633]: pgmap v20582: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:31.427+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:31.429042+0000) 2022-09-10T15:12:32.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:32.058+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:32.060037+0000) 2022-09-10T15:12:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:32.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:32.429220+0000) 2022-09-10T15:12:33.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:33.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:33.060286+0000) 2022-09-10T15:12:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:33 smithi104 ceph-mon[97044]: pgmap v20583: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:33 smithi110 ceph-mon[97633]: pgmap v20583: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:33.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:33.429434+0000) 2022-09-10T15:12:34.431 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:34.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:34.060491+0000) 2022-09-10T15:12:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:34.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:34.429667+0000) 2022-09-10T15:12:35.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:35.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:35.060746+0000) 2022-09-10T15:12:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:35 smithi104 ceph-mon[97044]: pgmap v20584: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:35 smithi110 ceph-mon[97633]: pgmap v20584: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:35.428+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:35.429922+0000) 2022-09-10T15:12:36.432 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:36.059+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:36.060941+0000) 2022-09-10T15:12:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:36.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:36.430202+0000) 2022-09-10T15:12:37.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:37.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:37.061155+0000) 2022-09-10T15:12:37.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:12:37.011+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:12:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:37 smithi104 ceph-mon[97044]: pgmap v20585: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:37 smithi110 ceph-mon[97633]: pgmap v20585: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:37.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:37.430473+0000) 2022-09-10T15:12:38.432 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:38.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:38.061334+0000) 2022-09-10T15:12:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:38 smithi110 ceph-mon[97633]: pgmap v20586: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:38 smithi104 ceph-mon[97044]: pgmap v20586: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:38.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:38.430694+0000) 2022-09-10T15:12:39.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:39.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:39.061474+0000) 2022-09-10T15:12:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:39.429+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:39.430929+0000) 2022-09-10T15:12:40.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:40.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:40.061612+0000) 2022-09-10T15:12:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:40.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:40.431178+0000) 2022-09-10T15:12:41.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:41.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:41.061782+0000) 2022-09-10T15:12:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:41 smithi104 ceph-mon[97044]: pgmap v20587: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:41 smithi110 ceph-mon[97633]: pgmap v20587: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:41.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:41.431406+0000) 2022-09-10T15:12:42.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:42.060+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:42.062014+0000) 2022-09-10T15:12:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:42.430+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:42.431659+0000) 2022-09-10T15:12:43.433 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:43.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:43.062271+0000) 2022-09-10T15:12:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:43 smithi104 ceph-mon[97044]: pgmap v20588: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:43 smithi110 ceph-mon[97633]: pgmap v20588: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:43.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:43.431896+0000) 2022-09-10T15:12:44.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:44.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:44.062527+0000) 2022-09-10T15:12:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:44.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:44.432122+0000) 2022-09-10T15:12:45.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:45.061+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:45.062800+0000) 2022-09-10T15:12:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:45 smithi104 ceph-mon[97044]: pgmap v20589: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:45 smithi110 ceph-mon[97633]: pgmap v20589: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:45.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:45.432341+0000) 2022-09-10T15:12:46.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:46.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:46.062997+0000) 2022-09-10T15:12:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:46.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:46.432622+0000) 2022-09-10T15:12:47.434 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:47.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:47.063238+0000) 2022-09-10T15:12:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:47 smithi104 ceph-mon[97044]: pgmap v20590: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:12:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:12:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:12:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:12:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:47 smithi110 ceph-mon[97633]: pgmap v20590: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:12:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:12:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:12:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:12:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:47.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:47.432826+0000) 2022-09-10T15:12:48.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:48.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:48.063448+0000) 2022-09-10T15:12:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:48 smithi110 ceph-mon[97633]: pgmap v20591: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:48 smithi104 ceph-mon[97044]: pgmap v20591: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:48.431+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:48.433046+0000) 2022-09-10T15:12:49.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:49.062+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:49.063670+0000) 2022-09-10T15:12:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:49.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:49.433282+0000) 2022-09-10T15:12:50.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:50.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:50.063886+0000) 2022-09-10T15:12:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:50.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:50.433461+0000) 2022-09-10T15:12:51.435 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:51.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:51.064155+0000) 2022-09-10T15:12:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:51 smithi104 ceph-mon[97044]: pgmap v20592: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:51 smithi110 ceph-mon[97633]: pgmap v20592: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:51.432+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:51.433690+0000) 2022-09-10T15:12:52.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:52.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:52.064358+0000) 2022-09-10T15:12:52.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:12:52.026+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:12:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:52.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:52.433924+0000) 2022-09-10T15:12:53.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:53.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:53.064595+0000) 2022-09-10T15:12:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:53 smithi104 ceph-mon[97044]: pgmap v20593: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:53 smithi110 ceph-mon[97633]: pgmap v20593: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:53.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:53.434192+0000) 2022-09-10T15:12:54.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:54.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:54.064810+0000) 2022-09-10T15:12:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:54.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:54.434444+0000) 2022-09-10T15:12:55.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:55.063+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:55.064934+0000) 2022-09-10T15:12:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:55 smithi104 ceph-mon[97044]: pgmap v20594: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:55 smithi110 ceph-mon[97633]: pgmap v20594: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:55.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:55.434646+0000) 2022-09-10T15:12:56.436 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:56.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:56.065211+0000) 2022-09-10T15:12:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:56.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:56.434787+0000) 2022-09-10T15:12:57.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:57.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:57.065472+0000) 2022-09-10T15:12:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:57 smithi104 ceph-mon[97044]: pgmap v20595: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:57 smithi110 ceph-mon[97633]: pgmap v20595: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:12:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:57.433+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:57.434968+0000) 2022-09-10T15:12:58.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:58.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:58.065698+0000) 2022-09-10T15:12:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:12:58 smithi110 ceph-mon[97633]: pgmap v20596: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:12:58 smithi104 ceph-mon[97044]: pgmap v20596: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:12:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:58.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:58.435181+0000) 2022-09-10T15:12:59.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:12:59.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:59.065888+0000) 2022-09-10T15:12:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:12:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:12:59.434+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:12:59.435427+0000) 2022-09-10T15:13:00.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:00.064+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:00.066020+0000) 2022-09-10T15:13:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:00.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:00.435603+0000) 2022-09-10T15:13:01.437 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:01.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:01.066215+0000) 2022-09-10T15:13:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:01 smithi104 ceph-mon[97044]: pgmap v20597: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:01 smithi110 ceph-mon[97633]: pgmap v20597: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:01.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:01.435848+0000) 2022-09-10T15:13:02.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:02.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:02.066395+0000) 2022-09-10T15:13:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:02.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:02.436157+0000) 2022-09-10T15:13:03.438 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:03.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:03.066596+0000) 2022-09-10T15:13:03.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:03 smithi104 ceph-mon[97044]: pgmap v20598: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:03 smithi110 ceph-mon[97633]: pgmap v20598: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:03.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:03.436414+0000) 2022-09-10T15:13:04.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:04.065+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:04.066750+0000) 2022-09-10T15:13:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:04.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:04.436648+0000) 2022-09-10T15:13:05.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:05.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:05.066981+0000) 2022-09-10T15:13:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:05 smithi104 ceph-mon[97044]: pgmap v20599: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:05 smithi110 ceph-mon[97633]: pgmap v20599: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:05.435+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:05.436909+0000) 2022-09-10T15:13:06.439 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:06.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:06.067193+0000) 2022-09-10T15:13:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:06.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:06.437201+0000) 2022-09-10T15:13:07.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:07.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:07.067422+0000) 2022-09-10T15:13:07.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:13:07.033+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:13:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:07 smithi104 ceph-mon[97044]: pgmap v20600: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:13:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:07 smithi110 ceph-mon[97633]: pgmap v20600: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:13:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:07.436+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:07.437581+0000) 2022-09-10T15:13:08.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:08.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:08.067663+0000) 2022-09-10T15:13:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:08 smithi110 ceph-mon[97633]: pgmap v20601: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:13:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:08 smithi104 ceph-mon[97044]: pgmap v20601: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:13:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:08.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:08.437814+0000) 2022-09-10T15:13:09.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:09.066+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:09.067932+0000) 2022-09-10T15:13:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:09 smithi110 ceph-mon[97633]: osdmap e1368: 12 total, 12 up, 12 in 2022-09-10T15:13:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:09 smithi104 ceph-mon[97044]: osdmap e1368: 12 total, 12 up, 12 in 2022-09-10T15:13:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:09.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:09.438078+0000) 2022-09-10T15:13:10.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:10.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:10.068186+0000) 2022-09-10T15:13:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:10 smithi110 ceph-mon[97633]: pgmap v20603: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:13:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:10 smithi104 ceph-mon[97044]: pgmap v20603: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:13:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:10.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:10.438245+0000) 2022-09-10T15:13:11.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:11.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:11.068392+0000) 2022-09-10T15:13:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:11.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:11.438467+0000) 2022-09-10T15:13:12.440 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:12.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:12.068642+0000) 2022-09-10T15:13:12.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:12.437+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:12.438669+0000) 2022-09-10T15:13:13.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:13.067+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:13.068841+0000) 2022-09-10T15:13:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:13 smithi104 ceph-mon[97044]: pgmap v20604: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:13:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:13 smithi104 ceph-mon[97044]: osdmap e1369: 12 total, 12 up, 12 in 2022-09-10T15:13:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:13 smithi110 ceph-mon[97633]: pgmap v20604: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:13:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:13 smithi110 ceph-mon[97633]: osdmap e1369: 12 total, 12 up, 12 in 2022-09-10T15:13:13.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:13.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:13.438890+0000) 2022-09-10T15:13:14.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:14.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:14.069060+0000) 2022-09-10T15:13:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:14.438+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:14.439162+0000) 2022-09-10T15:13:15.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:15.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:15.069258+0000) 2022-09-10T15:13:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:15 smithi104 ceph-mon[97044]: pgmap v20606: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:13:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:15 smithi110 ceph-mon[97633]: pgmap v20606: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:13:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:15.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:15.439290+0000) 2022-09-10T15:13:16.441 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:16.068+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:16.069425+0000) 2022-09-10T15:13:16.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:16.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:16.439479+0000) 2022-09-10T15:13:17.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:17.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:17.069666+0000) 2022-09-10T15:13:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:17 smithi104 ceph-mon[97044]: pgmap v20607: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:13:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:17 smithi110 ceph-mon[97633]: pgmap v20607: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:13:17.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:17.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:17.439662+0000) 2022-09-10T15:13:18.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:18.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:18.069873+0000) 2022-09-10T15:13:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:18 smithi110 ceph-mon[97633]: pgmap v20608: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:13:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:18 smithi104 ceph-mon[97044]: pgmap v20608: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:13:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:18.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:18.439893+0000) 2022-09-10T15:13:19.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:19.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:19.070169+0000) 2022-09-10T15:13:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:19.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:19.440148+0000) 2022-09-10T15:13:20.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:20.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:20.070320+0000) 2022-09-10T15:13:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:20.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:20.440359+0000) 2022-09-10T15:13:21.442 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:21.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:21.070477+0000) 2022-09-10T15:13:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:21 smithi104 ceph-mon[97044]: pgmap v20609: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:13:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:21 smithi110 ceph-mon[97633]: pgmap v20609: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:13:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:21.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:21.440598+0000) 2022-09-10T15:13:22.305 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:22.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:22.070709+0000) 2022-09-10T15:13:22.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:13:22.048+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:13:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:22.439+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:22.440805+0000) 2022-09-10T15:13:23.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:23.069+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:23.070986+0000) 2022-09-10T15:13:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:23 smithi104 ceph-mon[97044]: pgmap v20610: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:13:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:23 smithi110 ceph-mon[97633]: pgmap v20610: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:13:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:23.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:23.441058+0000) 2022-09-10T15:13:24.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:24.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:24.071232+0000) 2022-09-10T15:13:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:24.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:24.441327+0000) 2022-09-10T15:13:25.443 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:25.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:25.071530+0000) 2022-09-10T15:13:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:25 smithi104 ceph-mon[97044]: pgmap v20611: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:13:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:25 smithi110 ceph-mon[97633]: pgmap v20611: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:13:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:25.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:25.441529+0000) 2022-09-10T15:13:26.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:26.070+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:26.071763+0000) 2022-09-10T15:13:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:26.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:26.441719+0000) 2022-09-10T15:13:27.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:27.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:27.072034+0000) 2022-09-10T15:13:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:27 smithi104 ceph-mon[97044]: pgmap v20612: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:13:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:27 smithi110 ceph-mon[97633]: pgmap v20612: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:13:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:27.440+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:27.441943+0000) 2022-09-10T15:13:28.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:28.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:28.072227+0000) 2022-09-10T15:13:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:28 smithi110 ceph-mon[97633]: pgmap v20613: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:28 smithi104 ceph-mon[97044]: pgmap v20613: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:28.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:28.442183+0000) 2022-09-10T15:13:29.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:29.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:29.072449+0000) 2022-09-10T15:13:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:29.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:29.442372+0000) 2022-09-10T15:13:30.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:30.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:30.072654+0000) 2022-09-10T15:13:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:30.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:30.442601+0000) 2022-09-10T15:13:31.444 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:31.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:31.072855+0000) 2022-09-10T15:13:31.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:31 smithi104 ceph-mon[97044]: pgmap v20614: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:31 smithi110 ceph-mon[97633]: pgmap v20614: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:31.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:31.442822+0000) 2022-09-10T15:13:32.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:32.071+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:32.073065+0000) 2022-09-10T15:13:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:32.441+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:32.443007+0000) 2022-09-10T15:13:33.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:33.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:33.073360+0000) 2022-09-10T15:13:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:33 smithi104 ceph-mon[97044]: pgmap v20615: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:33 smithi110 ceph-mon[97633]: pgmap v20615: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:33.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:33.443266+0000) 2022-09-10T15:13:34.445 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:34.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:34.073545+0000) 2022-09-10T15:13:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:34.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:34.443524+0000) 2022-09-10T15:13:35.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:35.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:35.073830+0000) 2022-09-10T15:13:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:35 smithi104 ceph-mon[97044]: pgmap v20616: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:35 smithi110 ceph-mon[97633]: pgmap v20616: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:35.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:35.443706+0000) 2022-09-10T15:13:36.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:36.072+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:36.074078+0000) 2022-09-10T15:13:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:36.442+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:36.443900+0000) 2022-09-10T15:13:37.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:37.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:37.074236+0000) 2022-09-10T15:13:37.447 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:13:37.057+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:13:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:37 smithi104 ceph-mon[97044]: pgmap v20617: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:37 smithi110 ceph-mon[97633]: pgmap v20617: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:37.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:37.444160+0000) 2022-09-10T15:13:38.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:38.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:38.074460+0000) 2022-09-10T15:13:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:38 smithi110 ceph-mon[97633]: pgmap v20618: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:38 smithi104 ceph-mon[97044]: pgmap v20618: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:38.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:38.444340+0000) 2022-09-10T15:13:39.446 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:39.073+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:39.074738+0000) 2022-09-10T15:13:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:39.443+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:39.444574+0000) 2022-09-10T15:13:40.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:40.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:40.074981+0000) 2022-09-10T15:13:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:40.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:40.444811+0000) 2022-09-10T15:13:41.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:41.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:41.075188+0000) 2022-09-10T15:13:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:41 smithi104 ceph-mon[97044]: pgmap v20619: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:41 smithi110 ceph-mon[97633]: pgmap v20619: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:41.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:41.445082+0000) 2022-09-10T15:13:42.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:42.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:42.075365+0000) 2022-09-10T15:13:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:42.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:42.445287+0000) 2022-09-10T15:13:43.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:43.074+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:43.075588+0000) 2022-09-10T15:13:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:43 smithi104 ceph-mon[97044]: pgmap v20620: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:43 smithi110 ceph-mon[97633]: pgmap v20620: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:43.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:43.445411+0000) 2022-09-10T15:13:44.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:44.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:44.075780+0000) 2022-09-10T15:13:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:44.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:44.445590+0000) 2022-09-10T15:13:45.447 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:45.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:45.076028+0000) 2022-09-10T15:13:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:45 smithi104 ceph-mon[97044]: pgmap v20621: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:45 smithi110 ceph-mon[97633]: pgmap v20621: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:45.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:45.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:45.445770+0000) 2022-09-10T15:13:46.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:46.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:46.076258+0000) 2022-09-10T15:13:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:46.444+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:46.446025+0000) 2022-09-10T15:13:47.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:47.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:47.076422+0000) 2022-09-10T15:13:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:47 smithi104 ceph-mon[97044]: pgmap v20622: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:13:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:13:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:13:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:13:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:47 smithi110 ceph-mon[97633]: pgmap v20622: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:13:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:13:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:13:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:13:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:47.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:47.446285+0000) 2022-09-10T15:13:48.448 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:48.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:48.076623+0000) 2022-09-10T15:13:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:48 smithi110 ceph-mon[97633]: pgmap v20623: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:48 smithi104 ceph-mon[97044]: pgmap v20623: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:48.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:48.446493+0000) 2022-09-10T15:13:49.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:49.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:49.076794+0000) 2022-09-10T15:13:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:49.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:49.446668+0000) 2022-09-10T15:13:50.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:50.075+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:50.076957+0000) 2022-09-10T15:13:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:50.445+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:50.446834+0000) 2022-09-10T15:13:51.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:51.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:51.077205+0000) 2022-09-10T15:13:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:51 smithi104 ceph-mon[97044]: pgmap v20624: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:51 smithi110 ceph-mon[97633]: pgmap v20624: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:51.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:51.447090+0000) 2022-09-10T15:13:52.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:52.076+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:52.077425+0000) 2022-09-10T15:13:52.450 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:13:52.072+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:13:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:52.446+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:52.447348+0000) 2022-09-10T15:13:53.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:53.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:53.077671+0000) 2022-09-10T15:13:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:53 smithi104 ceph-mon[97044]: pgmap v20625: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:53 smithi110 ceph-mon[97633]: pgmap v20625: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:53.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:53.447609+0000) 2022-09-10T15:13:54.449 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:54.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:54.077903+0000) 2022-09-10T15:13:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:54.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:54.447820+0000) 2022-09-10T15:13:55.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:55.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:55.078160+0000) 2022-09-10T15:13:55.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:55 smithi104 ceph-mon[97044]: pgmap v20626: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:55 smithi110 ceph-mon[97633]: pgmap v20626: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:55.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:55.447988+0000) 2022-09-10T15:13:56.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:56.077+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:56.078292+0000) 2022-09-10T15:13:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:56.447+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:56.448226+0000) 2022-09-10T15:13:57.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:57.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:57.078495+0000) 2022-09-10T15:13:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:57 smithi104 ceph-mon[97044]: pgmap v20627: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:57 smithi110 ceph-mon[97633]: pgmap v20627: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:13:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:57.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:57.448459+0000) 2022-09-10T15:13:58.450 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:58.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:58.078713+0000) 2022-09-10T15:13:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:13:58 smithi110 ceph-mon[97633]: pgmap v20628: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:13:58 smithi104 ceph-mon[97044]: pgmap v20628: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:13:58.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:58.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:58.448708+0000) 2022-09-10T15:13:59.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:13:59.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:59.078930+0000) 2022-09-10T15:13:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:13:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:13:59.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:13:59.448957+0000) 2022-09-10T15:14:00.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:00.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:00.079154+0000) 2022-09-10T15:14:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:00.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:00.449164+0000) 2022-09-10T15:14:01.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:01.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:01.079372+0000) 2022-09-10T15:14:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:01 smithi104 ceph-mon[97044]: pgmap v20629: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:01 smithi110 ceph-mon[97633]: pgmap v20629: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:01.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:01.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:01.449410+0000) 2022-09-10T15:14:02.451 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:02.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:02.079601+0000) 2022-09-10T15:14:02.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:02.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:02.449597+0000) 2022-09-10T15:14:03.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:03.078+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:03.079779+0000) 2022-09-10T15:14:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:03 smithi104 ceph-mon[97044]: pgmap v20630: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:03 smithi110 ceph-mon[97633]: pgmap v20630: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:03.448+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:03.449808+0000) 2022-09-10T15:14:04.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:04.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:04.079982+0000) 2022-09-10T15:14:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:04.449+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:04.450041+0000) 2022-09-10T15:14:05.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:05.079+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:05.080209+0000) 2022-09-10T15:14:05.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:05 smithi104 ceph-mon[97044]: pgmap v20631: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:05 smithi110 ceph-mon[97633]: pgmap v20631: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:05.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:05.450291+0000) 2022-09-10T15:14:06.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:06.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:06.080426+0000) 2022-09-10T15:14:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:06.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:06.450498+0000) 2022-09-10T15:14:07.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:07.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:07.080706+0000) 2022-09-10T15:14:07.455 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:14:07.085+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:14:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:07 smithi104 ceph-mon[97044]: pgmap v20632: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:14:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:07 smithi110 ceph-mon[97633]: pgmap v20632: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:14:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:07.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:07.450676+0000) 2022-09-10T15:14:08.452 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:08.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:08.080944+0000) 2022-09-10T15:14:08.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:08 smithi110 ceph-mon[97633]: pgmap v20633: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:14:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:08 smithi104 ceph-mon[97044]: pgmap v20633: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:14:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:08.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:08.450829+0000) 2022-09-10T15:14:09.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:09.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:09.081180+0000) 2022-09-10T15:14:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:09 smithi110 ceph-mon[97633]: osdmap e1370: 12 total, 12 up, 12 in 2022-09-10T15:14:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:09 smithi104 ceph-mon[97044]: osdmap e1370: 12 total, 12 up, 12 in 2022-09-10T15:14:09.952 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:09.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:09.451084+0000) 2022-09-10T15:14:10.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:10.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:10.081352+0000) 2022-09-10T15:14:10.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:10 smithi110 ceph-mon[97633]: pgmap v20635: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:14:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:10 smithi104 ceph-mon[97044]: pgmap v20635: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:14:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:10.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:10.451291+0000) 2022-09-10T15:14:11.453 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:11.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:11.081593+0000) 2022-09-10T15:14:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:11.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:11.451556+0000) 2022-09-10T15:14:12.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:12.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:12.081790+0000) 2022-09-10T15:14:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:12.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:12.451775+0000) 2022-09-10T15:14:13.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:13.080+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:13.081975+0000) 2022-09-10T15:14:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:13 smithi104 ceph-mon[97044]: pgmap v20636: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:14:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:13 smithi104 ceph-mon[97044]: osdmap e1371: 12 total, 12 up, 12 in 2022-09-10T15:14:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:13 smithi110 ceph-mon[97633]: pgmap v20636: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:14:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:13 smithi110 ceph-mon[97633]: osdmap e1371: 12 total, 12 up, 12 in 2022-09-10T15:14:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:13.450+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:13.451985+0000) 2022-09-10T15:14:14.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:14.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:14.082144+0000) 2022-09-10T15:14:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:14.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:14.452230+0000) 2022-09-10T15:14:15.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:15.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:15.082357+0000) 2022-09-10T15:14:15.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:15 smithi104 ceph-mon[97044]: pgmap v20638: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:14:15.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:15 smithi110 ceph-mon[97633]: pgmap v20638: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:14:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:15.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:15.452505+0000) 2022-09-10T15:14:16.454 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:16.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:16.082598+0000) 2022-09-10T15:14:16.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:16.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:16.452718+0000) 2022-09-10T15:14:17.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:17.081+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:17.082783+0000) 2022-09-10T15:14:17.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:17 smithi104 ceph-mon[97044]: pgmap v20639: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:14:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:17 smithi110 ceph-mon[97633]: pgmap v20639: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:14:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:17.451+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:17.452958+0000) 2022-09-10T15:14:18.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:18.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:18.082997+0000) 2022-09-10T15:14:18.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:18 smithi110 ceph-mon[97633]: pgmap v20640: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:14:18.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:18 smithi104 ceph-mon[97044]: pgmap v20640: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:14:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:18.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:18.453216+0000) 2022-09-10T15:14:19.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:19.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:19.083255+0000) 2022-09-10T15:14:19.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:19.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:19.453460+0000) 2022-09-10T15:14:20.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:20.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:20.083473+0000) 2022-09-10T15:14:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:20.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:20.453689+0000) 2022-09-10T15:14:21.455 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:21.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:21.083682+0000) 2022-09-10T15:14:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:21 smithi104 ceph-mon[97044]: pgmap v20641: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:14:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:21 smithi110 ceph-mon[97633]: pgmap v20641: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:14:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:21.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:21.453908+0000) 2022-09-10T15:14:22.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:22.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:22.083845+0000) 2022-09-10T15:14:22.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:14:22.095+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:14:22.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:22.452+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:22.454043+0000) 2022-09-10T15:14:23.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:23.082+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:23.084073+0000) 2022-09-10T15:14:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:23 smithi104 ceph-mon[97044]: pgmap v20642: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:14:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:23 smithi110 ceph-mon[97633]: pgmap v20642: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:14:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:23.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:23.454251+0000) 2022-09-10T15:14:24.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:24.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:24.084338+0000) 2022-09-10T15:14:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:24.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:24.454456+0000) 2022-09-10T15:14:25.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:25.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:25.084546+0000) 2022-09-10T15:14:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:25 smithi104 ceph-mon[97044]: pgmap v20643: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:14:25.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:25 smithi110 ceph-mon[97633]: pgmap v20643: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:14:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:25.453+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:25.454615+0000) 2022-09-10T15:14:26.456 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:26.083+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:26.084773+0000) 2022-09-10T15:14:26.812 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:26.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:26.454842+0000) 2022-09-10T15:14:27.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:27.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:27.085078+0000) 2022-09-10T15:14:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:27 smithi104 ceph-mon[97044]: pgmap v20644: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:14:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:27 smithi110 ceph-mon[97633]: pgmap v20644: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:14:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:27.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:27.455084+0000) 2022-09-10T15:14:28.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:28.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:28.085322+0000) 2022-09-10T15:14:28.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:28 smithi110 ceph-mon[97633]: pgmap v20645: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:28 smithi104 ceph-mon[97044]: pgmap v20645: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:28.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:28.455326+0000) 2022-09-10T15:14:29.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:29.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:29.085521+0000) 2022-09-10T15:14:29.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:29.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:29.455489+0000) 2022-09-10T15:14:30.457 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:30.084+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:30.085736+0000) 2022-09-10T15:14:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:30.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:30.455650+0000) 2022-09-10T15:14:31.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:31.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:31.085995+0000) 2022-09-10T15:14:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:31 smithi104 ceph-mon[97044]: pgmap v20646: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:31 smithi110 ceph-mon[97633]: pgmap v20646: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:31.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:31.454+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:31.455923+0000) 2022-09-10T15:14:32.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:32.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:32.086167+0000) 2022-09-10T15:14:32.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:32.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:32.456145+0000) 2022-09-10T15:14:33.458 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:33.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:33.086393+0000) 2022-09-10T15:14:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:33 smithi104 ceph-mon[97044]: pgmap v20647: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:33.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:33 smithi110 ceph-mon[97633]: pgmap v20647: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:33.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:33.456365+0000) 2022-09-10T15:14:34.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:34.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:34.086579+0000) 2022-09-10T15:14:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:34.455+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:34.456568+0000) 2022-09-10T15:14:35.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:35.085+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:35.086841+0000) 2022-09-10T15:14:35.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:35 smithi104 ceph-mon[97044]: pgmap v20648: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:35.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:35 smithi110 ceph-mon[97633]: pgmap v20648: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:35.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:35.456786+0000) 2022-09-10T15:14:36.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:36.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:36.087060+0000) 2022-09-10T15:14:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:36.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:36.456944+0000) 2022-09-10T15:14:37.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:37.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:37.087220+0000) 2022-09-10T15:14:37.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:14:37.096+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:14:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:37 smithi104 ceph-mon[97044]: pgmap v20649: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:37.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:37 smithi110 ceph-mon[97633]: pgmap v20649: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:37.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:37.457136+0000) 2022-09-10T15:14:38.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:38.086+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:38.087401+0000) 2022-09-10T15:14:38.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:38 smithi110 ceph-mon[97633]: pgmap v20650: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:38 smithi104 ceph-mon[97044]: pgmap v20650: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:38.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:38.457285+0000) 2022-09-10T15:14:39.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:39.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:39.087651+0000) 2022-09-10T15:14:39.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:39.456+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:39.457515+0000) 2022-09-10T15:14:40.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:40.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:40.087895+0000) 2022-09-10T15:14:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:40.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:40.457765+0000) 2022-09-10T15:14:41.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:41.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:41.088156+0000) 2022-09-10T15:14:41.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:41 smithi104 ceph-mon[97044]: pgmap v20651: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:41 smithi110 ceph-mon[97633]: pgmap v20651: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:41.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:41.457995+0000) 2022-09-10T15:14:42.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:42.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:42.088392+0000) 2022-09-10T15:14:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:42.457+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:42.458252+0000) 2022-09-10T15:14:43.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:43.087+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:43.088670+0000) 2022-09-10T15:14:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:43 smithi104 ceph-mon[97044]: pgmap v20652: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:43 smithi110 ceph-mon[97633]: pgmap v20652: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:43.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:43.458523+0000) 2022-09-10T15:14:44.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:44.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:44.088881+0000) 2022-09-10T15:14:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:44.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:44.458740+0000) 2022-09-10T15:14:45.460 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:45.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:45.089065+0000) 2022-09-10T15:14:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:45 smithi104 ceph-mon[97044]: pgmap v20653: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:45 smithi110 ceph-mon[97633]: pgmap v20653: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:45.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:45.458958+0000) 2022-09-10T15:14:46.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:46.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:46.089342+0000) 2022-09-10T15:14:46.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:46.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:46.459084+0000) 2022-09-10T15:14:47.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:47.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:47.089631+0000) 2022-09-10T15:14:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:47 smithi104 ceph-mon[97044]: pgmap v20654: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:14:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:14:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:14:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:14:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:47 smithi110 ceph-mon[97633]: pgmap v20654: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:14:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:14:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:14:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:14:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:47.458+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:47.459243+0000) 2022-09-10T15:14:48.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:48.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:48.089830+0000) 2022-09-10T15:14:48.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:48 smithi110 ceph-mon[97633]: pgmap v20655: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:48 smithi104 ceph-mon[97044]: pgmap v20655: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:14:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:48.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:48.459486+0000) 2022-09-10T15:14:49.461 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:49.088+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:49.090032+0000) 2022-09-10T15:14:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:49.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:49.459781+0000) 2022-09-10T15:14:50.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:50.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:50.090253+0000) 2022-09-10T15:14:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:50.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:50.459946+0000) 2022-09-10T15:14:51.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:51.089+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:51.090478+0000) 2022-09-10T15:14:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:51 smithi104 ceph-mon[97044]: pgmap v20656: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:51 smithi110 ceph-mon[97633]: pgmap v20656: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:51.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:51.460240+0000) 2022-09-10T15:14:52.462 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:52.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:52.090731+0000) 2022-09-10T15:14:52.463 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:14:52.109+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:14:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:52.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:52.460468+0000) 2022-09-10T15:14:53.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:53.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:53.090948+0000) 2022-09-10T15:14:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:53 smithi104 ceph-mon[97044]: pgmap v20657: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:14:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:53 smithi110 ceph-mon[97633]: pgmap v20657: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:14:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:53.459+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:53.460740+0000) 2022-09-10T15:14:54.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:54.090+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:54.091183+0000) 2022-09-10T15:14:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:54.460+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:54.461009+0000) 2022-09-10T15:14:55.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:55.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:55.091479+0000) 2022-09-10T15:14:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:55 smithi104 ceph-mon[97044]: pgmap v20658: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:14:55.665 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:55 smithi110 ceph-mon[97633]: pgmap v20658: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:14:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:55.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:55.461201+0000) 2022-09-10T15:14:56.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:56.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:56.091671+0000) 2022-09-10T15:14:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:56.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:56.461419+0000) 2022-09-10T15:14:57.463 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:57.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:57.091937+0000) 2022-09-10T15:14:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:57 smithi104 ceph-mon[97044]: pgmap v20659: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:57 smithi110 ceph-mon[97633]: pgmap v20659: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:14:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:57.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:57.461675+0000) 2022-09-10T15:14:58.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:58.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:58.092131+0000) 2022-09-10T15:14:58.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:14:58 smithi110 ceph-mon[97633]: pgmap v20660: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:14:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:14:58 smithi104 ceph-mon[97044]: pgmap v20660: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:14:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:58.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:58.461926+0000) 2022-09-10T15:14:59.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:14:59.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:59.092301+0000) 2022-09-10T15:14:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:14:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:14:59.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:14:59.462085+0000) 2022-09-10T15:15:00.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:00.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:00.092534+0000) 2022-09-10T15:15:00.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:00.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:00.462245+0000) 2022-09-10T15:15:01.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:01.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:01.092734+0000) 2022-09-10T15:15:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:01 smithi104 ceph-mon[97044]: pgmap v20661: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:01 smithi110 ceph-mon[97633]: pgmap v20661: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:01.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:01.462460+0000) 2022-09-10T15:15:02.464 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:02.091+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:02.092959+0000) 2022-09-10T15:15:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:02.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:02.462615+0000) 2022-09-10T15:15:03.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:03.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:03.093236+0000) 2022-09-10T15:15:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:03 smithi104 ceph-mon[97044]: pgmap v20662: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:03 smithi110 ceph-mon[97633]: pgmap v20662: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:03.461+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:03.462855+0000) 2022-09-10T15:15:04.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:04.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:04.093465+0000) 2022-09-10T15:15:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:04.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:04.463129+0000) 2022-09-10T15:15:05.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:05.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:05.093707+0000) 2022-09-10T15:15:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:05 smithi104 ceph-mon[97044]: pgmap v20663: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:05 smithi110 ceph-mon[97633]: pgmap v20663: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:05.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:05.463383+0000) 2022-09-10T15:15:06.465 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:06.092+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:06.093921+0000) 2022-09-10T15:15:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:06.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:06.463583+0000) 2022-09-10T15:15:07.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:07.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:07.094195+0000) 2022-09-10T15:15:07.467 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:15:07.121+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:15:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:07 smithi104 ceph-mon[97044]: pgmap v20664: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:15:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:07 smithi110 ceph-mon[97633]: pgmap v20664: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:15:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:07.462+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:07.463748+0000) 2022-09-10T15:15:08.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:08.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:08.094426+0000) 2022-09-10T15:15:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:08 smithi110 ceph-mon[97633]: pgmap v20665: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:15:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:08 smithi104 ceph-mon[97044]: pgmap v20665: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:15:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:08.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:08.463989+0000) 2022-09-10T15:15:09.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:09.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:09.094627+0000) 2022-09-10T15:15:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:09 smithi110 ceph-mon[97633]: osdmap e1372: 12 total, 12 up, 12 in 2022-09-10T15:15:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:09 smithi104 ceph-mon[97044]: osdmap e1372: 12 total, 12 up, 12 in 2022-09-10T15:15:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:09.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:09.464247+0000) 2022-09-10T15:15:10.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:10.093+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:10.094852+0000) 2022-09-10T15:15:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:10 smithi104 ceph-mon[97044]: pgmap v20667: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:15:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:10.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:10.464438+0000) 2022-09-10T15:15:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:10 smithi110 ceph-mon[97633]: pgmap v20667: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:15:11.466 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:11.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:11.095089+0000) 2022-09-10T15:15:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:11.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:11.464691+0000) 2022-09-10T15:15:12.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:12.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:12.095302+0000) 2022-09-10T15:15:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:12.463+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:12.464871+0000) 2022-09-10T15:15:13.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:13.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:13.095565+0000) 2022-09-10T15:15:13.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:13 smithi104 ceph-mon[97044]: pgmap v20668: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:15:13.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:13 smithi104 ceph-mon[97044]: osdmap e1373: 12 total, 12 up, 12 in 2022-09-10T15:15:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:13 smithi110 ceph-mon[97633]: pgmap v20668: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:15:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:13 smithi110 ceph-mon[97633]: osdmap e1373: 12 total, 12 up, 12 in 2022-09-10T15:15:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:13.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:13.465153+0000) 2022-09-10T15:15:14.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:14.094+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:14.095800+0000) 2022-09-10T15:15:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:14.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:14.465366+0000) 2022-09-10T15:15:15.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:15.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:15.096031+0000) 2022-09-10T15:15:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:15 smithi104 ceph-mon[97044]: pgmap v20670: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:15:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:15 smithi110 ceph-mon[97633]: pgmap v20670: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:15:15.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:15.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:15.465565+0000) 2022-09-10T15:15:16.467 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:16.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:16.096271+0000) 2022-09-10T15:15:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:16.464+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:16.465785+0000) 2022-09-10T15:15:17.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:17.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:17.096541+0000) 2022-09-10T15:15:17.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:17 smithi104 ceph-mon[97044]: pgmap v20671: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:15:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:17 smithi110 ceph-mon[97633]: pgmap v20671: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:15:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:17.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:17.466025+0000) 2022-09-10T15:15:18.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:18.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:18.096743+0000) 2022-09-10T15:15:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:18 smithi104 ceph-mon[97044]: pgmap v20672: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:15:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:18.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:18.466255+0000) 2022-09-10T15:15:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:18 smithi110 ceph-mon[97633]: pgmap v20672: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:15:19.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:19.095+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:19.096959+0000) 2022-09-10T15:15:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:19.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:19.466462+0000) 2022-09-10T15:15:20.468 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:20.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:20.097195+0000) 2022-09-10T15:15:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:20.465+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:20.466685+0000) 2022-09-10T15:15:21.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:21.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:21.097424+0000) 2022-09-10T15:15:21.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:21 smithi104 ceph-mon[97044]: pgmap v20673: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:15:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:21 smithi110 ceph-mon[97633]: pgmap v20673: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:15:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:21.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:21.466961+0000) 2022-09-10T15:15:22.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:22.096+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:22.097673+0000) 2022-09-10T15:15:22.470 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:15:22.133+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:15:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:22.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:22.467233+0000) 2022-09-10T15:15:23.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:23.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:23.097936+0000) 2022-09-10T15:15:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:23 smithi104 ceph-mon[97044]: pgmap v20674: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:15:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:23 smithi110 ceph-mon[97633]: pgmap v20674: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:15:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:23.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:23.467415+0000) 2022-09-10T15:15:24.469 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:24.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:24.098138+0000) 2022-09-10T15:15:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:24.466+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:24.467588+0000) 2022-09-10T15:15:25.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:25.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:25.098375+0000) 2022-09-10T15:15:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:25 smithi104 ceph-mon[97044]: pgmap v20675: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:15:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:25 smithi110 ceph-mon[97633]: pgmap v20675: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:15:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:25.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:25.467859+0000) 2022-09-10T15:15:26.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:26.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:26.098567+0000) 2022-09-10T15:15:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:26.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:26.468082+0000) 2022-09-10T15:15:27.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:27.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:27.098780+0000) 2022-09-10T15:15:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:27 smithi104 ceph-mon[97044]: pgmap v20676: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:15:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:27 smithi110 ceph-mon[97633]: pgmap v20676: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:15:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:27.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:27.468319+0000) 2022-09-10T15:15:28.470 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:28.097+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:28.098950+0000) 2022-09-10T15:15:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:28 smithi104 ceph-mon[97044]: pgmap v20677: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:28.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:28.468518+0000) 2022-09-10T15:15:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:28 smithi110 ceph-mon[97633]: pgmap v20677: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:29.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:29.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:29.099159+0000) 2022-09-10T15:15:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:29.467+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:29.468712+0000) 2022-09-10T15:15:30.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:30.098+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:30.099391+0000) 2022-09-10T15:15:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:30.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:30.468928+0000) 2022-09-10T15:15:31.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:31.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:31.099621+0000) 2022-09-10T15:15:31.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:31 smithi104 ceph-mon[97044]: pgmap v20678: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:31 smithi110 ceph-mon[97633]: pgmap v20678: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:31.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:31.469184+0000) 2022-09-10T15:15:32.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:32.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:32.099849+0000) 2022-09-10T15:15:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:32.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:32.469455+0000) 2022-09-10T15:15:33.471 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:33.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:33.100145+0000) 2022-09-10T15:15:33.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:33 smithi104 ceph-mon[97044]: pgmap v20679: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:33 smithi110 ceph-mon[97633]: pgmap v20679: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:33.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:33.469696+0000) 2022-09-10T15:15:34.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:34.099+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:34.100349+0000) 2022-09-10T15:15:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:34.468+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:34.469937+0000) 2022-09-10T15:15:35.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:35.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:35.100574+0000) 2022-09-10T15:15:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:35 smithi104 ceph-mon[97044]: pgmap v20680: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:35 smithi110 ceph-mon[97633]: pgmap v20680: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:35.469+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:35.470200+0000) 2022-09-10T15:15:36.472 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:36.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:36.100822+0000) 2022-09-10T15:15:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:36.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:36.470404+0000) 2022-09-10T15:15:37.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:37.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:37.101027+0000) 2022-09-10T15:15:37.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:15:37.141+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:15:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:37 smithi104 ceph-mon[97044]: pgmap v20681: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:37 smithi110 ceph-mon[97633]: pgmap v20681: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:37.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:37.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:37.470665+0000) 2022-09-10T15:15:38.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:38.100+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:38.101241+0000) 2022-09-10T15:15:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:38 smithi104 ceph-mon[97044]: pgmap v20682: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:38.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:38.470867+0000) 2022-09-10T15:15:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:38 smithi110 ceph-mon[97633]: pgmap v20682: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:39.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:39.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:39.101463+0000) 2022-09-10T15:15:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:39.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:39.471069+0000) 2022-09-10T15:15:40.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:40.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:40.101710+0000) 2022-09-10T15:15:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:40.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:40.471296+0000) 2022-09-10T15:15:41.473 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:41.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:41.101889+0000) 2022-09-10T15:15:41.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:41 smithi104 ceph-mon[97044]: pgmap v20683: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:41 smithi110 ceph-mon[97633]: pgmap v20683: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:41.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:41.471537+0000) 2022-09-10T15:15:42.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:42.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:42.102077+0000) 2022-09-10T15:15:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:42.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:42.471737+0000) 2022-09-10T15:15:43.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:43.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:43.102297+0000) 2022-09-10T15:15:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:43 smithi104 ceph-mon[97044]: pgmap v20684: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:43 smithi110 ceph-mon[97633]: pgmap v20684: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:43.470+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:43.471946+0000) 2022-09-10T15:15:44.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:44.101+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:44.102461+0000) 2022-09-10T15:15:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:44.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:44.472185+0000) 2022-09-10T15:15:45.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:45.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:45.102640+0000) 2022-09-10T15:15:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:45 smithi104 ceph-mon[97044]: pgmap v20685: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:45 smithi110 ceph-mon[97633]: pgmap v20685: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:45.471+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:45.472425+0000) 2022-09-10T15:15:46.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:46.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:46.102881+0000) 2022-09-10T15:15:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:46.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:46.472674+0000) 2022-09-10T15:15:47.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:47.102+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:47.103130+0000) 2022-09-10T15:15:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:47 smithi104 ceph-mon[97044]: pgmap v20686: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:15:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:15:47.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:15:47.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:15:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:47 smithi110 ceph-mon[97633]: pgmap v20686: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:15:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:15:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:15:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:15:47.811 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:47.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:47.472903+0000) 2022-09-10T15:15:48.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:48.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:48.103361+0000) 2022-09-10T15:15:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:48 smithi104 ceph-mon[97044]: pgmap v20687: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:48.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:48.473172+0000) 2022-09-10T15:15:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:48 smithi110 ceph-mon[97633]: pgmap v20687: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:49.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:49.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:49.103572+0000) 2022-09-10T15:15:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:49.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:49.473395+0000) 2022-09-10T15:15:50.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:50.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:50.103826+0000) 2022-09-10T15:15:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:50.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:50.473513+0000) 2022-09-10T15:15:51.475 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:51.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:51.103968+0000) 2022-09-10T15:15:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:51 smithi104 ceph-mon[97044]: pgmap v20688: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:51 smithi110 ceph-mon[97633]: pgmap v20688: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:51.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:51.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:51.473744+0000) 2022-09-10T15:15:52.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:52.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:52.104157+0000) 2022-09-10T15:15:52.476 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:15:52.157+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:15:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:52.472+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:52.473954+0000) 2022-09-10T15:15:53.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:53.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:53.104352+0000) 2022-09-10T15:15:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:53 smithi104 ceph-mon[97044]: pgmap v20689: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:53 smithi110 ceph-mon[97633]: pgmap v20689: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:53.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:53.474220+0000) 2022-09-10T15:15:54.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:54.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:54.104582+0000) 2022-09-10T15:15:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:54.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:54.474472+0000) 2022-09-10T15:15:55.476 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:55.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:55.104849+0000) 2022-09-10T15:15:55.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:55 smithi104 ceph-mon[97044]: pgmap v20690: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:55 smithi110 ceph-mon[97633]: pgmap v20690: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:55.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:55.474627+0000) 2022-09-10T15:15:56.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:56.103+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:56.105048+0000) 2022-09-10T15:15:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:56.473+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:56.474833+0000) 2022-09-10T15:15:57.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:57.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:57.105260+0000) 2022-09-10T15:15:57.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:57 smithi104 ceph-mon[97044]: pgmap v20691: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:57 smithi110 ceph-mon[97633]: pgmap v20691: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:15:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:57.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:57.475063+0000) 2022-09-10T15:15:58.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:58.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:58.105440+0000) 2022-09-10T15:15:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:15:58 smithi104 ceph-mon[97044]: pgmap v20692: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:58.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:58.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:58.475223+0000) 2022-09-10T15:15:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:15:58 smithi110 ceph-mon[97633]: pgmap v20692: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:15:59.477 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:15:59.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:59.105690+0000) 2022-09-10T15:15:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:15:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:15:59.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:15:59.475450+0000) 2022-09-10T15:16:00.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:00.104+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:00.105947+0000) 2022-09-10T15:16:00.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:00.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:00.475669+0000) 2022-09-10T15:16:01.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:01.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:01.106162+0000) 2022-09-10T15:16:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:01 smithi104 ceph-mon[97044]: pgmap v20693: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:01 smithi110 ceph-mon[97633]: pgmap v20693: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:01.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:01.475857+0000) 2022-09-10T15:16:02.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:02.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:02.106383+0000) 2022-09-10T15:16:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:02.474+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:02.476042+0000) 2022-09-10T15:16:03.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:03.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:03.106650+0000) 2022-09-10T15:16:03.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:03 smithi104 ceph-mon[97044]: pgmap v20694: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:03 smithi110 ceph-mon[97633]: pgmap v20694: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:03.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:03.476258+0000) 2022-09-10T15:16:04.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:04.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:04.106816+0000) 2022-09-10T15:16:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:04.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:04.476508+0000) 2022-09-10T15:16:05.478 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:05.105+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:05.106988+0000) 2022-09-10T15:16:05.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:05 smithi104 ceph-mon[97044]: pgmap v20695: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:05 smithi110 ceph-mon[97633]: pgmap v20695: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:05.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:05.476718+0000) 2022-09-10T15:16:06.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:06.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:06.107202+0000) 2022-09-10T15:16:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:06.475+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:06.476975+0000) 2022-09-10T15:16:07.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:07.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:07.107483+0000) 2022-09-10T15:16:07.479 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:16:07.158+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:16:07.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:07 smithi104 ceph-mon[97044]: pgmap v20696: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:16:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:07 smithi110 ceph-mon[97633]: pgmap v20696: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:16:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:07.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:07.477185+0000) 2022-09-10T15:16:08.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:08.106+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:08.107662+0000) 2022-09-10T15:16:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:08 smithi104 ceph-mon[97044]: pgmap v20697: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:16:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:08.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:08.477338+0000) 2022-09-10T15:16:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:08 smithi110 ceph-mon[97633]: pgmap v20697: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:16:09.479 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:09.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:09.107910+0000) 2022-09-10T15:16:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:09 smithi104 ceph-mon[97044]: osdmap e1374: 12 total, 12 up, 12 in 2022-09-10T15:16:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:09.476+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:09.477555+0000) 2022-09-10T15:16:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:09 smithi110 ceph-mon[97633]: osdmap e1374: 12 total, 12 up, 12 in 2022-09-10T15:16:10.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:10.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:10.108167+0000) 2022-09-10T15:16:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:10 smithi104 ceph-mon[97044]: pgmap v20699: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:16:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:10.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:10.477825+0000) 2022-09-10T15:16:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:10 smithi110 ceph-mon[97633]: pgmap v20699: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:16:11.480 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:11.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:11.108388+0000) 2022-09-10T15:16:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:11.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:11.478082+0000) 2022-09-10T15:16:12.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:12.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:12.108569+0000) 2022-09-10T15:16:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:12.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:12.478291+0000) 2022-09-10T15:16:13.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:13.107+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:13.108744+0000) 2022-09-10T15:16:13.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:13 smithi104 ceph-mon[97044]: pgmap v20700: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:16:13.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:13 smithi104 ceph-mon[97044]: osdmap e1375: 12 total, 12 up, 12 in 2022-09-10T15:16:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:13 smithi110 ceph-mon[97633]: pgmap v20700: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:16:13.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:13 smithi110 ceph-mon[97633]: osdmap e1375: 12 total, 12 up, 12 in 2022-09-10T15:16:13.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:13.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:13.478452+0000) 2022-09-10T15:16:14.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:14.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:14.108939+0000) 2022-09-10T15:16:14.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:14.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:14.478671+0000) 2022-09-10T15:16:15.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:15.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:15.109078+0000) 2022-09-10T15:16:15.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:15 smithi104 ceph-mon[97044]: pgmap v20702: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:16:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:15 smithi110 ceph-mon[97633]: pgmap v20702: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:16:15.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:15.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:15.478900+0000) 2022-09-10T15:16:16.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:16.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:16.109298+0000) 2022-09-10T15:16:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:16.477+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:16.479046+0000) 2022-09-10T15:16:17.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:17.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:17.109470+0000) 2022-09-10T15:16:17.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:17 smithi104 ceph-mon[97044]: pgmap v20703: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:16:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:17 smithi110 ceph-mon[97633]: pgmap v20703: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:16:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:17.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:17.479275+0000) 2022-09-10T15:16:18.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:18.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:18.109676+0000) 2022-09-10T15:16:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:18 smithi104 ceph-mon[97044]: pgmap v20704: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:16:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:18.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:18.479497+0000) 2022-09-10T15:16:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:18 smithi110 ceph-mon[97633]: pgmap v20704: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:16:19.481 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:19.108+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:19.109918+0000) 2022-09-10T15:16:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:19.478+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:19.479727+0000) 2022-09-10T15:16:20.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:20.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:20.110172+0000) 2022-09-10T15:16:20.816 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:20.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:20.479964+0000) 2022-09-10T15:16:21.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:21.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:21.110398+0000) 2022-09-10T15:16:21.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:21 smithi104 ceph-mon[97044]: pgmap v20705: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:16:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:21 smithi110 ceph-mon[97633]: pgmap v20705: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:16:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:21.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:21.480246+0000) 2022-09-10T15:16:22.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:22.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:22.110642+0000) 2022-09-10T15:16:22.483 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:16:22.166+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:16:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:22.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:22.480493+0000) 2022-09-10T15:16:23.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:23.109+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:23.110857+0000) 2022-09-10T15:16:23.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:23 smithi104 ceph-mon[97044]: pgmap v20706: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:16:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:23 smithi110 ceph-mon[97633]: pgmap v20706: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:16:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:23.479+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:23.480651+0000) 2022-09-10T15:16:24.482 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:24.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:24.111173+0000) 2022-09-10T15:16:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:24.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:24.480866+0000) 2022-09-10T15:16:25.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:25.110+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:25.111363+0000) 2022-09-10T15:16:25.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:25 smithi104 ceph-mon[97044]: pgmap v20707: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:16:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:25 smithi110 ceph-mon[97633]: pgmap v20707: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:16:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:25.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:25.481074+0000) 2022-09-10T15:16:26.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:26.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:26.111537+0000) 2022-09-10T15:16:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:26.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:26.481338+0000) 2022-09-10T15:16:27.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:27.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:27.111773+0000) 2022-09-10T15:16:27.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:27 smithi104 ceph-mon[97044]: pgmap v20708: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:16:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:27 smithi110 ceph-mon[97633]: pgmap v20708: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:16:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:27.480+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:27.481504+0000) 2022-09-10T15:16:28.483 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:28.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:28.112026+0000) 2022-09-10T15:16:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:28 smithi104 ceph-mon[97044]: pgmap v20709: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:28.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:28.481728+0000) 2022-09-10T15:16:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:28 smithi110 ceph-mon[97633]: pgmap v20709: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:29.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:29.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:29.112266+0000) 2022-09-10T15:16:29.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:29.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:29.481993+0000) 2022-09-10T15:16:30.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:30.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:30.112466+0000) 2022-09-10T15:16:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:30.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:30.482223+0000) 2022-09-10T15:16:31.484 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:31.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:31.112744+0000) 2022-09-10T15:16:31.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:31 smithi104 ceph-mon[97044]: pgmap v20710: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:31 smithi110 ceph-mon[97633]: pgmap v20710: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:31.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:31.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:31.482400+0000) 2022-09-10T15:16:32.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:32.111+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:32.112923+0000) 2022-09-10T15:16:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:32.481+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:32.482562+0000) 2022-09-10T15:16:33.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:33.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:33.113140+0000) 2022-09-10T15:16:33.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:33 smithi104 ceph-mon[97044]: pgmap v20711: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:33 smithi110 ceph-mon[97633]: pgmap v20711: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:33.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:33.482817+0000) 2022-09-10T15:16:34.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:34.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:34.113333+0000) 2022-09-10T15:16:34.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:34.482+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:34.483135+0000) 2022-09-10T15:16:35.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:35.112+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:35.113555+0000) 2022-09-10T15:16:35.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:35 smithi104 ceph-mon[97044]: pgmap v20712: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:35 smithi110 ceph-mon[97633]: pgmap v20712: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:35.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:35.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:35.483334+0000) 2022-09-10T15:16:36.485 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:36.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:36.113788+0000) 2022-09-10T15:16:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:36.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:36.483547+0000) 2022-09-10T15:16:37.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:37.113+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:37.114084+0000) 2022-09-10T15:16:37.486 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:16:37.181+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:16:37.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:37 smithi104 ceph-mon[97044]: pgmap v20713: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:37 smithi110 ceph-mon[97633]: pgmap v20713: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:37.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:37.483860+0000) 2022-09-10T15:16:38.486 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:38.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:38.114267+0000) 2022-09-10T15:16:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:38 smithi104 ceph-mon[97044]: pgmap v20714: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:38.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:38.484128+0000) 2022-09-10T15:16:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:38 smithi110 ceph-mon[97633]: pgmap v20714: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:39.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:39.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:39.114457+0000) 2022-09-10T15:16:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:39.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:39.484341+0000) 2022-09-10T15:16:40.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:40.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:40.114647+0000) 2022-09-10T15:16:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:40.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:40.484511+0000) 2022-09-10T15:16:41.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:41.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:41.114882+0000) 2022-09-10T15:16:41.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:41 smithi104 ceph-mon[97044]: pgmap v20715: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:41.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:41 smithi110 ceph-mon[97633]: pgmap v20715: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:41.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:41.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:41.484782+0000) 2022-09-10T15:16:42.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:42.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:42.115075+0000) 2022-09-10T15:16:42.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:42.483+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:42.484987+0000) 2022-09-10T15:16:43.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:43.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:43.115338+0000) 2022-09-10T15:16:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:43 smithi104 ceph-mon[97044]: pgmap v20716: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:43 smithi110 ceph-mon[97633]: pgmap v20716: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:43.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:43.485228+0000) 2022-09-10T15:16:44.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:44.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:44.115504+0000) 2022-09-10T15:16:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:44.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:44.485490+0000) 2022-09-10T15:16:45.487 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:45.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:45.115710+0000) 2022-09-10T15:16:45.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:45 smithi104 ceph-mon[97044]: pgmap v20717: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:45 smithi110 ceph-mon[97633]: pgmap v20717: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:45.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:45.485780+0000) 2022-09-10T15:16:46.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:46.114+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:46.115907+0000) 2022-09-10T15:16:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:46.484+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:46.485996+0000) 2022-09-10T15:16:47.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:47.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:47.116192+0000) 2022-09-10T15:16:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:47 smithi110 ceph-mon[97633]: pgmap v20718: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:16:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:16:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:16:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:16:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:47 smithi104 ceph-mon[97044]: pgmap v20718: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:16:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:16:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:16:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:16:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:47.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:47.486189+0000) 2022-09-10T15:16:48.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:48.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:48.116404+0000) 2022-09-10T15:16:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:48 smithi104 ceph-mon[97044]: pgmap v20719: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:48.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:48.486313+0000) 2022-09-10T15:16:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:48 smithi110 ceph-mon[97633]: pgmap v20719: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:49.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:49.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:49.116582+0000) 2022-09-10T15:16:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:49.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:49.486511+0000) 2022-09-10T15:16:50.488 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:50.115+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:50.116792+0000) 2022-09-10T15:16:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:50.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:50.486729+0000) 2022-09-10T15:16:51.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:51.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:51.117009+0000) 2022-09-10T15:16:51.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:51 smithi104 ceph-mon[97044]: pgmap v20720: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:51 smithi110 ceph-mon[97633]: pgmap v20720: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:51.485+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:51.487007+0000) 2022-09-10T15:16:52.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:52.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:52.117214+0000) 2022-09-10T15:16:52.490 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:16:52.185+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:16:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:52.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:52.487252+0000) 2022-09-10T15:16:53.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:53.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:53.117446+0000) 2022-09-10T15:16:53.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:53 smithi104 ceph-mon[97044]: pgmap v20721: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:53 smithi110 ceph-mon[97633]: pgmap v20721: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:53.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:53.487407+0000) 2022-09-10T15:16:54.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:54.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:54.117690+0000) 2022-09-10T15:16:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:54.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:54.487629+0000) 2022-09-10T15:16:55.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:55.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:55.117936+0000) 2022-09-10T15:16:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:55 smithi110 ceph-mon[97633]: pgmap v20722: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:55 smithi104 ceph-mon[97044]: pgmap v20722: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:55.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:55.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:55.487857+0000) 2022-09-10T15:16:56.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:56.116+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:56.118078+0000) 2022-09-10T15:16:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:56.486+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:56.488040+0000) 2022-09-10T15:16:57.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:57.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:57.118296+0000) 2022-09-10T15:16:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:57 smithi110 ceph-mon[97633]: pgmap v20723: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:57 smithi104 ceph-mon[97044]: pgmap v20723: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:16:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:57.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:57.488232+0000) 2022-09-10T15:16:58.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:58.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:58.118495+0000) 2022-09-10T15:16:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:16:58 smithi104 ceph-mon[97044]: pgmap v20724: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:58.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:58.488419+0000) 2022-09-10T15:16:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:16:58 smithi110 ceph-mon[97633]: pgmap v20724: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:16:59.490 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:16:59.117+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:59.118709+0000) 2022-09-10T15:16:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:16:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:16:59.487+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:16:59.488637+0000) 2022-09-10T15:17:00.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:00.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:00.118939+0000) 2022-09-10T15:17:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:00.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:00.488873+0000) 2022-09-10T15:17:01.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:01.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:01.119190+0000) 2022-09-10T15:17:01.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:01 smithi104 ceph-mon[97044]: pgmap v20725: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:01 smithi110 ceph-mon[97633]: pgmap v20725: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:01.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:01.489232+0000) 2022-09-10T15:17:02.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:02.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:02.119357+0000) 2022-09-10T15:17:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:02.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:02.489401+0000) 2022-09-10T15:17:03.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:03.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:03.119560+0000) 2022-09-10T15:17:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:03 smithi110 ceph-mon[97633]: pgmap v20726: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:03 smithi104 ceph-mon[97044]: pgmap v20726: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:03.488+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:03.489593+0000) 2022-09-10T15:17:04.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:04.118+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:04.119748+0000) 2022-09-10T15:17:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:04.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:04.489809+0000) 2022-09-10T15:17:05.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:05.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:05.119991+0000) 2022-09-10T15:17:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:05 smithi110 ceph-mon[97633]: pgmap v20727: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:05 smithi104 ceph-mon[97044]: pgmap v20727: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:05.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:05.490059+0000) 2022-09-10T15:17:06.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:06.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:06.120224+0000) 2022-09-10T15:17:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:06.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:06.490315+0000) 2022-09-10T15:17:07.492 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:07.119+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:07.120441+0000) 2022-09-10T15:17:07.493 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:17:07.198+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:17:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:07 smithi110 ceph-mon[97633]: pgmap v20728: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:17:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:07 smithi104 ceph-mon[97044]: pgmap v20728: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:17:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:07.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:07.490521+0000) 2022-09-10T15:17:08.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:08.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:08.120672+0000) 2022-09-10T15:17:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:08 smithi104 ceph-mon[97044]: pgmap v20729: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:17:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:08.489+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:08.490715+0000) 2022-09-10T15:17:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:08 smithi110 ceph-mon[97633]: pgmap v20729: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:17:09.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:09.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:09.120924+0000) 2022-09-10T15:17:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:09 smithi104 ceph-mon[97044]: osdmap e1376: 12 total, 12 up, 12 in 2022-09-10T15:17:09.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:09.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:09.490941+0000) 2022-09-10T15:17:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:09 smithi110 ceph-mon[97633]: osdmap e1376: 12 total, 12 up, 12 in 2022-09-10T15:17:10.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:10.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:10.121205+0000) 2022-09-10T15:17:10.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:10 smithi104 ceph-mon[97044]: pgmap v20731: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:17:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:10.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:10.491206+0000) 2022-09-10T15:17:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:10 smithi110 ceph-mon[97633]: pgmap v20731: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:17:11.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:11.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:11.121375+0000) 2022-09-10T15:17:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:11.490+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:11.491388+0000) 2022-09-10T15:17:12.493 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:12.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:12.121635+0000) 2022-09-10T15:17:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:12.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:12.491566+0000) 2022-09-10T15:17:13.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:13.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:13.121810+0000) 2022-09-10T15:17:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:13 smithi110 ceph-mon[97633]: pgmap v20732: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:17:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:13 smithi110 ceph-mon[97633]: osdmap e1377: 12 total, 12 up, 12 in 2022-09-10T15:17:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:13 smithi104 ceph-mon[97044]: pgmap v20732: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:17:13.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:13 smithi104 ceph-mon[97044]: osdmap e1377: 12 total, 12 up, 12 in 2022-09-10T15:17:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:13.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:13.491795+0000) 2022-09-10T15:17:14.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:14.120+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:14.121937+0000) 2022-09-10T15:17:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:14.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:14.492044+0000) 2022-09-10T15:17:15.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:15.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:15.122125+0000) 2022-09-10T15:17:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:15 smithi110 ceph-mon[97633]: pgmap v20734: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:17:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:15 smithi104 ceph-mon[97044]: pgmap v20734: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:17:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:15.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:15.492207+0000) 2022-09-10T15:17:16.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:16.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:16.122328+0000) 2022-09-10T15:17:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:16.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:16.492419+0000) 2022-09-10T15:17:17.494 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:17.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:17.122585+0000) 2022-09-10T15:17:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:17 smithi110 ceph-mon[97633]: pgmap v20735: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:17:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:17 smithi104 ceph-mon[97044]: pgmap v20735: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:17:17.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:17.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:17.492666+0000) 2022-09-10T15:17:18.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:18.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:18.122799+0000) 2022-09-10T15:17:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:18 smithi104 ceph-mon[97044]: pgmap v20736: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:17:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:18.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:18.492842+0000) 2022-09-10T15:17:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:18 smithi110 ceph-mon[97633]: pgmap v20736: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:17:19.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:19.121+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:19.123021+0000) 2022-09-10T15:17:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:19.491+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:19.493043+0000) 2022-09-10T15:17:20.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:20.122+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:20.123232+0000) 2022-09-10T15:17:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:20.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:20.493238+0000) 2022-09-10T15:17:21.495 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:21.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:21.123483+0000) 2022-09-10T15:17:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:21 smithi110 ceph-mon[97633]: pgmap v20737: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:17:21.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:21 smithi104 ceph-mon[97044]: pgmap v20737: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:17:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:21.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:21.493442+0000) 2022-09-10T15:17:22.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:22.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:22.123704+0000) 2022-09-10T15:17:22.496 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:17:22.208+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:17:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:22.492+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:22.493638+0000) 2022-09-10T15:17:23.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:23.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:23.123939+0000) 2022-09-10T15:17:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:23 smithi110 ceph-mon[97633]: pgmap v20738: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:17:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:23 smithi104 ceph-mon[97044]: pgmap v20738: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:17:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:23.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:23.493927+0000) 2022-09-10T15:17:24.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:24.123+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:24.124209+0000) 2022-09-10T15:17:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:24.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:24.494186+0000) 2022-09-10T15:17:25.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:25.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:25.124455+0000) 2022-09-10T15:17:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:25 smithi110 ceph-mon[97633]: pgmap v20739: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:17:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:25 smithi104 ceph-mon[97044]: pgmap v20739: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:17:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:25.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:25.494360+0000) 2022-09-10T15:17:26.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:26.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:26.124678+0000) 2022-09-10T15:17:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:26.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:26.494587+0000) 2022-09-10T15:17:27.496 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:27.124+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:27.124934+0000) 2022-09-10T15:17:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:27 smithi110 ceph-mon[97633]: pgmap v20740: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:17:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:27 smithi104 ceph-mon[97044]: pgmap v20740: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:17:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:27.493+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:27.494804+0000) 2022-09-10T15:17:28.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:28.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:28.125163+0000) 2022-09-10T15:17:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:28 smithi104 ceph-mon[97044]: pgmap v20741: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:28.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:28.495030+0000) 2022-09-10T15:17:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:28 smithi110 ceph-mon[97633]: pgmap v20741: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:29.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:29.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:29.125396+0000) 2022-09-10T15:17:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:29.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:29.495236+0000) 2022-09-10T15:17:30.497 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:30.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:30.125580+0000) 2022-09-10T15:17:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:30.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:30.495439+0000) 2022-09-10T15:17:31.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:31.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:31.125781+0000) 2022-09-10T15:17:31.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:31 smithi110 ceph-mon[97633]: pgmap v20742: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:31 smithi104 ceph-mon[97044]: pgmap v20742: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:31.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:31.495690+0000) 2022-09-10T15:17:32.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:32.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:32.126034+0000) 2022-09-10T15:17:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:32.494+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:32.495906+0000) 2022-09-10T15:17:33.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:33.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:33.126317+0000) 2022-09-10T15:17:33.692 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:33 smithi110 ceph-mon[97633]: pgmap v20743: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:33 smithi104 ceph-mon[97044]: pgmap v20743: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:33.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:33.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:33.496130+0000) 2022-09-10T15:17:34.498 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:34.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:34.126514+0000) 2022-09-10T15:17:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:34.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:34.496352+0000) 2022-09-10T15:17:35.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:35.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:35.126707+0000) 2022-09-10T15:17:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:35 smithi110 ceph-mon[97633]: pgmap v20744: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:35 smithi104 ceph-mon[97044]: pgmap v20744: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:35.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:35.496576+0000) 2022-09-10T15:17:36.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:36.125+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:36.126910+0000) 2022-09-10T15:17:36.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:36.495+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:36.496814+0000) 2022-09-10T15:17:37.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:37.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:37.127186+0000) 2022-09-10T15:17:37.500 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:17:37.217+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:17:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:37 smithi110 ceph-mon[97633]: pgmap v20745: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:37 smithi104 ceph-mon[97044]: pgmap v20745: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:37.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:37.497063+0000) 2022-09-10T15:17:38.499 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:38.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:38.127424+0000) 2022-09-10T15:17:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:38 smithi104 ceph-mon[97044]: pgmap v20746: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:38.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:38.497316+0000) 2022-09-10T15:17:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:38 smithi110 ceph-mon[97633]: pgmap v20746: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:39.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:39.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:39.127684+0000) 2022-09-10T15:17:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:39.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:39.497497+0000) 2022-09-10T15:17:40.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:40.126+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:40.127858+0000) 2022-09-10T15:17:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:40.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:40.497732+0000) 2022-09-10T15:17:41.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:41.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:41.128126+0000) 2022-09-10T15:17:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:41 smithi110 ceph-mon[97633]: pgmap v20747: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:41 smithi104 ceph-mon[97044]: pgmap v20747: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:41.496+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:41.497976+0000) 2022-09-10T15:17:42.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:42.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:42.128339+0000) 2022-09-10T15:17:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:42.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:42.498214+0000) 2022-09-10T15:17:43.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:43.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:43.128514+0000) 2022-09-10T15:17:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:43 smithi110 ceph-mon[97633]: pgmap v20748: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:43 smithi104 ceph-mon[97044]: pgmap v20748: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:43.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:43.498359+0000) 2022-09-10T15:17:44.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:44.127+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:44.128736+0000) 2022-09-10T15:17:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:44.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:44.498559+0000) 2022-09-10T15:17:45.500 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:45.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:45.128934+0000) 2022-09-10T15:17:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:45 smithi110 ceph-mon[97633]: pgmap v20749: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:45 smithi104 ceph-mon[97044]: pgmap v20749: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:45.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:45.498814+0000) 2022-09-10T15:17:46.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:46.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:46.129143+0000) 2022-09-10T15:17:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:46.497+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:46.498991+0000) 2022-09-10T15:17:47.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:47.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:47.129393+0000) 2022-09-10T15:17:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:47 smithi110 ceph-mon[97633]: pgmap v20750: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:17:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:17:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:17:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:17:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:47 smithi104 ceph-mon[97044]: pgmap v20750: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:17:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:17:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:17:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:17:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:47.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:47.499252+0000) 2022-09-10T15:17:48.501 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:48.128+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:48.129621+0000) 2022-09-10T15:17:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:48 smithi104 ceph-mon[97044]: pgmap v20751: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:48.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:48.499473+0000) 2022-09-10T15:17:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:48 smithi110 ceph-mon[97633]: pgmap v20751: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:49.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:49.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:49.129880+0000) 2022-09-10T15:17:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:49.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:49.499766+0000) 2022-09-10T15:17:50.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:50.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:50.130161+0000) 2022-09-10T15:17:50.808 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:50.498+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:50.500016+0000) 2022-09-10T15:17:51.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:51.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:51.130341+0000) 2022-09-10T15:17:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:51 smithi110 ceph-mon[97633]: pgmap v20752: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:51 smithi104 ceph-mon[97044]: pgmap v20752: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:51.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:51.500159+0000) 2022-09-10T15:17:52.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:52.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:52.130544+0000) 2022-09-10T15:17:52.503 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:17:52.225+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:17:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:52.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:52.500386+0000) 2022-09-10T15:17:53.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:53.129+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:53.130725+0000) 2022-09-10T15:17:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:53 smithi110 ceph-mon[97633]: pgmap v20753: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:53 smithi104 ceph-mon[97044]: pgmap v20753: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:53.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:53.500668+0000) 2022-09-10T15:17:54.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:54.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:54.130937+0000) 2022-09-10T15:17:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:54.499+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:54.500882+0000) 2022-09-10T15:17:55.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:55.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:55.131190+0000) 2022-09-10T15:17:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:55 smithi110 ceph-mon[97633]: pgmap v20754: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:55 smithi104 ceph-mon[97044]: pgmap v20754: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:55.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:55.501128+0000) 2022-09-10T15:17:56.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:56.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:56.131387+0000) 2022-09-10T15:17:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:56.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:56.501349+0000) 2022-09-10T15:17:57.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:57.130+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:57.131589+0000) 2022-09-10T15:17:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:57 smithi110 ceph-mon[97633]: pgmap v20755: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:57.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:57 smithi104 ceph-mon[97044]: pgmap v20755: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:17:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:57.500+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:57.501556+0000) 2022-09-10T15:17:58.503 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:58.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:58.131789+0000) 2022-09-10T15:17:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:17:58 smithi104 ceph-mon[97044]: pgmap v20756: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:58.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:58.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:58.501758+0000) 2022-09-10T15:17:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:17:58 smithi110 ceph-mon[97633]: pgmap v20756: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:17:59.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:17:59.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:59.132002+0000) 2022-09-10T15:17:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:17:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:17:59.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:17:59.501973+0000) 2022-09-10T15:18:00.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:00.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:00.132260+0000) 2022-09-10T15:18:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:00.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:00.502215+0000) 2022-09-10T15:18:01.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:01.131+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:01.132449+0000) 2022-09-10T15:18:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:01 smithi110 ceph-mon[97633]: pgmap v20757: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:01 smithi104 ceph-mon[97044]: pgmap v20757: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:01.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:01.502425+0000) 2022-09-10T15:18:02.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:02.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:02.132599+0000) 2022-09-10T15:18:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:02.501+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:02.502608+0000) 2022-09-10T15:18:03.504 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:03.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:03.132861+0000) 2022-09-10T15:18:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:03 smithi110 ceph-mon[97633]: pgmap v20758: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:03 smithi104 ceph-mon[97044]: pgmap v20758: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:03.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:03.502858+0000) 2022-09-10T15:18:04.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:04.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:04.133119+0000) 2022-09-10T15:18:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:04.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:04.503116+0000) 2022-09-10T15:18:05.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:05.132+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:05.133296+0000) 2022-09-10T15:18:05.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:05 smithi110 ceph-mon[97633]: pgmap v20759: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:05 smithi104 ceph-mon[97044]: pgmap v20759: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:05.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:05.503338+0000) 2022-09-10T15:18:06.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:06.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:06.133543+0000) 2022-09-10T15:18:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:06.502+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:06.503568+0000) 2022-09-10T15:18:07.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:07.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:07.133820+0000) 2022-09-10T15:18:07.508 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:18:07.228+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:18:07.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:07 smithi110 ceph-mon[97633]: pgmap v20760: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:18:07.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:07 smithi104 ceph-mon[97044]: pgmap v20760: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:18:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:07.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:07.503820+0000) 2022-09-10T15:18:08.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:08.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:08.134064+0000) 2022-09-10T15:18:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:08 smithi104 ceph-mon[97044]: pgmap v20761: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:18:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:08.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:08.504015+0000) 2022-09-10T15:18:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:08 smithi110 ceph-mon[97633]: pgmap v20761: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:18:09.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:09.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:09.134280+0000) 2022-09-10T15:18:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:09 smithi104 ceph-mon[97044]: osdmap e1378: 12 total, 12 up, 12 in 2022-09-10T15:18:09.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:09.503+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:09.504296+0000) 2022-09-10T15:18:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:09 smithi110 ceph-mon[97633]: osdmap e1378: 12 total, 12 up, 12 in 2022-09-10T15:18:10.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:10.133+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:10.134445+0000) 2022-09-10T15:18:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:10 smithi104 ceph-mon[97044]: pgmap v20763: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:18:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:10.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:10.504558+0000) 2022-09-10T15:18:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:10 smithi110 ceph-mon[97633]: pgmap v20763: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:18:11.506 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:11.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:11.134654+0000) 2022-09-10T15:18:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:11.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:11.504801+0000) 2022-09-10T15:18:12.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:12.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:12.134843+0000) 2022-09-10T15:18:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:12.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:12.505050+0000) 2022-09-10T15:18:13.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:13.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:13.135076+0000) 2022-09-10T15:18:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:13 smithi110 ceph-mon[97633]: pgmap v20764: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:18:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:13 smithi110 ceph-mon[97633]: osdmap e1379: 12 total, 12 up, 12 in 2022-09-10T15:18:13.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:13 smithi104 ceph-mon[97044]: pgmap v20764: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:18:13.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:13 smithi104 ceph-mon[97044]: osdmap e1379: 12 total, 12 up, 12 in 2022-09-10T15:18:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:13.504+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:13.505284+0000) 2022-09-10T15:18:14.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:14.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:14.135240+0000) 2022-09-10T15:18:14.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:14.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:14.505506+0000) 2022-09-10T15:18:15.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:15.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:15.135475+0000) 2022-09-10T15:18:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:15 smithi110 ceph-mon[97633]: pgmap v20766: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:18:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:15 smithi104 ceph-mon[97044]: pgmap v20766: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:18:15.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:15.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:15.505775+0000) 2022-09-10T15:18:16.507 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:16.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:16.135673+0000) 2022-09-10T15:18:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:16.505+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:16.505969+0000) 2022-09-10T15:18:17.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:17.134+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:17.135839+0000) 2022-09-10T15:18:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:17 smithi110 ceph-mon[97633]: pgmap v20767: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:18:17.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:17 smithi104 ceph-mon[97044]: pgmap v20767: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:18:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:17.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:17.506198+0000) 2022-09-10T15:18:18.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:18.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:18.135977+0000) 2022-09-10T15:18:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:18 smithi104 ceph-mon[97044]: pgmap v20768: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:18:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:18.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:18.506448+0000) 2022-09-10T15:18:18.918 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:18 smithi110 ceph-mon[97633]: pgmap v20768: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:18:19.508 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:19.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:19.136201+0000) 2022-09-10T15:18:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:19.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:19.506639+0000) 2022-09-10T15:18:20.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:20.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:20.136393+0000) 2022-09-10T15:18:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:20.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:20.506883+0000) 2022-09-10T15:18:21.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:21.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:21.136627+0000) 2022-09-10T15:18:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:21 smithi110 ceph-mon[97633]: pgmap v20769: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:18:21.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:21 smithi104 ceph-mon[97044]: pgmap v20769: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:18:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:21.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:21.507172+0000) 2022-09-10T15:18:22.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:22.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:22.136839+0000) 2022-09-10T15:18:22.510 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:18:22.233+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:18:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:22.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:22.507409+0000) 2022-09-10T15:18:23.509 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:23.135+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:23.137060+0000) 2022-09-10T15:18:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:23 smithi110 ceph-mon[97633]: pgmap v20770: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:18:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:23 smithi104 ceph-mon[97044]: pgmap v20770: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:18:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:23.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:23.507549+0000) 2022-09-10T15:18:24.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:24.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:24.137321+0000) 2022-09-10T15:18:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:24.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:24.507769+0000) 2022-09-10T15:18:25.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:25.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:25.137497+0000) 2022-09-10T15:18:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:25 smithi110 ceph-mon[97633]: pgmap v20771: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:18:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:25 smithi104 ceph-mon[97044]: pgmap v20771: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:18:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:25.506+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:25.507996+0000) 2022-09-10T15:18:26.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:26.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:26.137725+0000) 2022-09-10T15:18:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:26.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:26.508192+0000) 2022-09-10T15:18:27.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:27.136+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:27.137852+0000) 2022-09-10T15:18:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:27 smithi110 ceph-mon[97633]: pgmap v20772: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:18:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:27 smithi104 ceph-mon[97044]: pgmap v20772: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:18:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:27.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:27.508427+0000) 2022-09-10T15:18:28.510 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:28.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:28.138086+0000) 2022-09-10T15:18:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:28 smithi104 ceph-mon[97044]: pgmap v20773: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:28.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:28.508650+0000) 2022-09-10T15:18:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:28 smithi110 ceph-mon[97633]: pgmap v20773: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:29.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:29.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:29.138358+0000) 2022-09-10T15:18:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:29.507+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:29.508914+0000) 2022-09-10T15:18:30.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:30.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:30.138614+0000) 2022-09-10T15:18:30.810 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:30.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:30.509092+0000) 2022-09-10T15:18:31.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:31.137+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:31.138812+0000) 2022-09-10T15:18:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:31 smithi110 ceph-mon[97633]: pgmap v20774: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:31.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:31 smithi104 ceph-mon[97044]: pgmap v20774: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:31.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:31.509342+0000) 2022-09-10T15:18:32.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:32.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:32.139013+0000) 2022-09-10T15:18:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:32.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:32.509554+0000) 2022-09-10T15:18:33.511 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:33.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:33.139170+0000) 2022-09-10T15:18:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:33 smithi110 ceph-mon[97633]: pgmap v20775: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:33 smithi104 ceph-mon[97044]: pgmap v20775: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:33.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:33.509781+0000) 2022-09-10T15:18:34.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:34.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:34.139327+0000) 2022-09-10T15:18:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:34.508+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:34.510031+0000) 2022-09-10T15:18:35.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:35.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:35.139581+0000) 2022-09-10T15:18:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:35 smithi110 ceph-mon[97633]: pgmap v20776: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:35 smithi104 ceph-mon[97044]: pgmap v20776: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:35.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:35.510254+0000) 2022-09-10T15:18:36.512 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:36.138+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:36.139854+0000) 2022-09-10T15:18:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:36.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:36.510477+0000) 2022-09-10T15:18:37.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:37.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:37.140123+0000) 2022-09-10T15:18:37.513 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:18:37.247+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:18:37.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:37 smithi110 ceph-mon[97633]: pgmap v20777: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:37.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:37 smithi104 ceph-mon[97044]: pgmap v20777: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:37.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:37.509+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:37.510652+0000) 2022-09-10T15:18:38.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:38.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:38.140319+0000) 2022-09-10T15:18:38.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:38 smithi104 ceph-mon[97044]: pgmap v20778: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:38.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:38.510886+0000) 2022-09-10T15:18:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:38 smithi110 ceph-mon[97633]: pgmap v20778: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:39.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:39.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:39.140443+0000) 2022-09-10T15:18:39.815 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:39.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:39.511144+0000) 2022-09-10T15:18:40.513 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:40.139+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:40.140655+0000) 2022-09-10T15:18:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:40.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:40.511375+0000) 2022-09-10T15:18:41.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:41.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:41.140869+0000) 2022-09-10T15:18:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:41 smithi110 ceph-mon[97633]: pgmap v20779: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:41 smithi104 ceph-mon[97044]: pgmap v20779: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:41.510+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:41.511639+0000) 2022-09-10T15:18:42.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:42.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:42.141124+0000) 2022-09-10T15:18:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:42.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:42.511880+0000) 2022-09-10T15:18:43.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:43.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:43.141353+0000) 2022-09-10T15:18:43.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:43 smithi104 ceph-mon[97044]: pgmap v20780: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:43 smithi110 ceph-mon[97633]: pgmap v20780: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:43.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:43.512064+0000) 2022-09-10T15:18:44.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:44.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:44.141614+0000) 2022-09-10T15:18:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:44.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:44.512281+0000) 2022-09-10T15:18:45.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:45.140+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:45.141852+0000) 2022-09-10T15:18:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:45 smithi110 ceph-mon[97633]: pgmap v20781: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:45 smithi104 ceph-mon[97044]: pgmap v20781: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:45.511+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:45.512485+0000) 2022-09-10T15:18:46.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:46.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:46.142049+0000) 2022-09-10T15:18:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:46.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:46.512671+0000) 2022-09-10T15:18:47.514 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:47.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:47.142313+0000) 2022-09-10T15:18:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:47 smithi110 ceph-mon[97633]: pgmap v20782: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:18:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:18:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:18:47.661 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:18:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:47 smithi104 ceph-mon[97044]: pgmap v20782: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:18:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:18:47.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:18:47.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:18:47.809 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:47.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:47.512885+0000) 2022-09-10T15:18:48.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:48.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:48.142493+0000) 2022-09-10T15:18:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:48 smithi104 ceph-mon[97044]: pgmap v20783: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:48.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:48.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:48.513135+0000) 2022-09-10T15:18:48.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:48 smithi110 ceph-mon[97633]: pgmap v20783: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:49.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:49.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:49.142694+0000) 2022-09-10T15:18:49.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:49.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:49.513345+0000) 2022-09-10T15:18:50.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:50.141+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:50.142929+0000) 2022-09-10T15:18:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:50.512+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:50.513576+0000) 2022-09-10T15:18:51.515 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:51.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:51.143169+0000) 2022-09-10T15:18:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:51 smithi110 ceph-mon[97633]: pgmap v20784: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:51 smithi104 ceph-mon[97044]: pgmap v20784: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:51.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:51.513766+0000) 2022-09-10T15:18:52.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:52.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:52.143419+0000) 2022-09-10T15:18:52.517 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:18:52.263+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:18:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:52.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:52.514054+0000) 2022-09-10T15:18:53.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:53.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:53.143668+0000) 2022-09-10T15:18:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:53 smithi110 ceph-mon[97633]: pgmap v20785: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:53 smithi104 ceph-mon[97044]: pgmap v20785: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:53.513+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:53.514230+0000) 2022-09-10T15:18:54.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:54.142+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:54.143913+0000) 2022-09-10T15:18:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:54.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:54.514476+0000) 2022-09-10T15:18:55.516 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:55.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:55.144076+0000) 2022-09-10T15:18:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:55 smithi110 ceph-mon[97633]: pgmap v20786: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:55 smithi104 ceph-mon[97044]: pgmap v20786: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:55.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:55.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:55.514729+0000) 2022-09-10T15:18:56.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:56.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:56.144282+0000) 2022-09-10T15:18:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:56.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:56.514937+0000) 2022-09-10T15:18:57.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:57.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:57.144412+0000) 2022-09-10T15:18:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:57 smithi110 ceph-mon[97633]: pgmap v20787: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:57 smithi104 ceph-mon[97044]: pgmap v20787: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:18:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:57.514+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:57.515197+0000) 2022-09-10T15:18:58.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:58.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:58.144610+0000) 2022-09-10T15:18:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:18:58 smithi104 ceph-mon[97044]: pgmap v20788: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:58.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:58.515432+0000) 2022-09-10T15:18:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:18:58 smithi110 ceph-mon[97633]: pgmap v20788: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:18:59.517 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:18:59.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:59.144858+0000) 2022-09-10T15:18:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:18:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:18:59.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:18:59.515649+0000) 2022-09-10T15:19:00.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:00.143+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:00.145011+0000) 2022-09-10T15:19:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:00.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:00.515873+0000) 2022-09-10T15:19:01.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:01.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:01.145195+0000) 2022-09-10T15:19:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:01 smithi110 ceph-mon[97633]: pgmap v20789: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:01 smithi104 ceph-mon[97044]: pgmap v20789: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:01.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:01.516159+0000) 2022-09-10T15:19:02.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:02.144+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:02.145329+0000) 2022-09-10T15:19:02.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:02.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:02.516376+0000) 2022-09-10T15:19:03.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:03.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:03.145565+0000) 2022-09-10T15:19:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:03 smithi110 ceph-mon[97633]: pgmap v20790: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:03 smithi104 ceph-mon[97044]: pgmap v20790: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:03.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:03.516553+0000) 2022-09-10T15:19:04.518 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:04.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:04.145726+0000) 2022-09-10T15:19:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:04.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:04.516731+0000) 2022-09-10T15:19:05.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:05.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:05.145947+0000) 2022-09-10T15:19:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:05 smithi110 ceph-mon[97633]: pgmap v20791: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:05 smithi104 ceph-mon[97044]: pgmap v20791: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:05.515+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:05.516948+0000) 2022-09-10T15:19:06.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:06.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:06.146210+0000) 2022-09-10T15:19:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:06.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:06.517137+0000) 2022-09-10T15:19:07.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:07.145+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:07.146391+0000) 2022-09-10T15:19:07.520 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:19:07.277+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:19:07.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:07 smithi110 ceph-mon[97633]: pgmap v20792: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:19:07.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:07 smithi104 ceph-mon[97044]: pgmap v20792: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:19:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:07.516+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:07.517286+0000) 2022-09-10T15:19:08.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:08.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:08.146535+0000) 2022-09-10T15:19:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:08 smithi104 ceph-mon[97044]: pgmap v20793: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:19:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:08.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:08.517473+0000) 2022-09-10T15:19:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:08 smithi110 ceph-mon[97633]: pgmap v20793: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:19:09.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:09.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:09.146717+0000) 2022-09-10T15:19:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:09 smithi104 ceph-mon[97044]: osdmap e1380: 12 total, 12 up, 12 in 2022-09-10T15:19:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:09.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:09.517677+0000) 2022-09-10T15:19:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:09 smithi110 ceph-mon[97633]: osdmap e1380: 12 total, 12 up, 12 in 2022-09-10T15:19:10.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:10.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:10.146945+0000) 2022-09-10T15:19:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:10 smithi104 ceph-mon[97044]: pgmap v20795: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:19:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:10.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:10.517876+0000) 2022-09-10T15:19:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:10 smithi110 ceph-mon[97633]: pgmap v20795: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:19:11.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:11.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:11.147185+0000) 2022-09-10T15:19:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:11.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:11.518149+0000) 2022-09-10T15:19:12.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:12.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:12.147350+0000) 2022-09-10T15:19:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:12.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:12.518387+0000) 2022-09-10T15:19:13.520 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:13.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:13.147525+0000) 2022-09-10T15:19:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:13 smithi110 ceph-mon[97633]: pgmap v20796: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:19:13.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:13 smithi110 ceph-mon[97633]: osdmap e1381: 12 total, 12 up, 12 in 2022-09-10T15:19:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:13 smithi104 ceph-mon[97044]: pgmap v20796: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:19:13.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:13 smithi104 ceph-mon[97044]: osdmap e1381: 12 total, 12 up, 12 in 2022-09-10T15:19:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:13.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:13.518621+0000) 2022-09-10T15:19:14.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:14.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:14.147769+0000) 2022-09-10T15:19:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:14.517+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:14.518864+0000) 2022-09-10T15:19:15.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:15.146+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:15.148064+0000) 2022-09-10T15:19:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:15 smithi110 ceph-mon[97633]: pgmap v20798: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:19:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:15 smithi104 ceph-mon[97044]: pgmap v20798: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:19:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:15.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:15.519088+0000) 2022-09-10T15:19:16.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:16.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:16.148243+0000) 2022-09-10T15:19:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:16.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:16.519297+0000) 2022-09-10T15:19:17.521 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:17.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:17.148442+0000) 2022-09-10T15:19:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:17 smithi110 ceph-mon[97633]: pgmap v20799: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:19:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:17 smithi104 ceph-mon[97044]: pgmap v20799: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:19:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:17.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:17.519478+0000) 2022-09-10T15:19:18.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:18.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:18.148613+0000) 2022-09-10T15:19:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:18 smithi104 ceph-mon[97044]: pgmap v20800: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:19:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:18.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:18.519643+0000) 2022-09-10T15:19:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:18 smithi110 ceph-mon[97633]: pgmap v20800: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:19:19.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:19.147+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:19.148861+0000) 2022-09-10T15:19:19.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:19.518+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:19.519850+0000) 2022-09-10T15:19:20.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:20.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:20.149127+0000) 2022-09-10T15:19:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:20.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:20.520083+0000) 2022-09-10T15:19:21.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:21.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:21.149292+0000) 2022-09-10T15:19:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:21 smithi110 ceph-mon[97633]: pgmap v20801: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:19:21.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:21 smithi104 ceph-mon[97044]: pgmap v20801: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:19:21.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:21.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:21.520285+0000) 2022-09-10T15:19:22.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:22.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:22.149464+0000) 2022-09-10T15:19:22.523 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:19:22.286+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:19:22.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:22.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:22.520421+0000) 2022-09-10T15:19:23.522 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:23.148+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:23.149665+0000) 2022-09-10T15:19:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:23 smithi110 ceph-mon[97633]: pgmap v20802: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:19:23.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:23 smithi104 ceph-mon[97044]: pgmap v20802: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:19:23.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:23.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:23.520617+0000) 2022-09-10T15:19:24.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:24.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:24.149908+0000) 2022-09-10T15:19:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:24.519+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:24.520873+0000) 2022-09-10T15:19:25.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:25.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:25.150160+0000) 2022-09-10T15:19:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:25 smithi110 ceph-mon[97633]: pgmap v20803: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:19:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:25 smithi104 ceph-mon[97044]: pgmap v20803: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:19:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:25.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:25.521117+0000) 2022-09-10T15:19:26.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:26.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:26.150383+0000) 2022-09-10T15:19:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:26.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:26.521315+0000) 2022-09-10T15:19:27.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:27.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:27.150637+0000) 2022-09-10T15:19:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:27 smithi110 ceph-mon[97633]: pgmap v20804: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:19:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:27 smithi104 ceph-mon[97044]: pgmap v20804: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:19:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:27.520+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:27.521587+0000) 2022-09-10T15:19:28.523 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:28.149+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:28.150880+0000) 2022-09-10T15:19:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:28 smithi104 ceph-mon[97044]: pgmap v20805: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:28.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:28.521773+0000) 2022-09-10T15:19:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:28 smithi110 ceph-mon[97633]: pgmap v20805: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:29.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:29.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:29.151131+0000) 2022-09-10T15:19:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:29.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:29.521895+0000) 2022-09-10T15:19:30.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:30.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:30.151314+0000) 2022-09-10T15:19:30.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:30.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:30.522085+0000) 2022-09-10T15:19:31.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:31.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:31.151534+0000) 2022-09-10T15:19:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:31 smithi110 ceph-mon[97633]: pgmap v20806: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:31 smithi104 ceph-mon[97044]: pgmap v20806: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:31.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:31.522252+0000) 2022-09-10T15:19:32.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:32.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:32.151783+0000) 2022-09-10T15:19:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:32.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:32.522488+0000) 2022-09-10T15:19:33.524 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:33.150+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:33.152026+0000) 2022-09-10T15:19:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:33 smithi110 ceph-mon[97633]: pgmap v20807: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:33 smithi104 ceph-mon[97044]: pgmap v20807: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:33.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:33.522754+0000) 2022-09-10T15:19:34.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:34.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:34.152183+0000) 2022-09-10T15:19:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:34.521+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:34.523037+0000) 2022-09-10T15:19:35.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:35.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:35.152373+0000) 2022-09-10T15:19:35.535 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:35.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:35.523307+0000) 2022-09-10T15:19:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:35 smithi110 ceph-mon[97633]: pgmap v20808: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:35 smithi104 ceph-mon[97044]: pgmap v20808: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:36.525 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:36.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:36.152567+0000) 2022-09-10T15:19:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:36.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:36.523506+0000) 2022-09-10T15:19:37.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:37.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:37.152781+0000) 2022-09-10T15:19:37.527 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:19:37.298+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:19:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:37 smithi110 ceph-mon[97633]: pgmap v20809: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:37.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:37 smithi104 ceph-mon[97044]: pgmap v20809: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:37.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:37.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:37.523734+0000) 2022-09-10T15:19:38.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:38.151+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:38.152939+0000) 2022-09-10T15:19:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:38 smithi104 ceph-mon[97044]: pgmap v20810: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:38.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:38.522+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:38.524012+0000) 2022-09-10T15:19:38.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:38 smithi110 ceph-mon[97633]: pgmap v20810: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:39.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:39.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:39.153151+0000) 2022-09-10T15:19:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:39.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:39.524259+0000) 2022-09-10T15:19:40.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:40.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:40.153359+0000) 2022-09-10T15:19:40.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:40.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:40.524467+0000) 2022-09-10T15:19:41.526 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:41.152+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:41.153546+0000) 2022-09-10T15:19:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:41 smithi110 ceph-mon[97633]: pgmap v20811: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:41 smithi104 ceph-mon[97044]: pgmap v20811: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:41.523+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:41.524613+0000) 2022-09-10T15:19:42.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:42.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:42.153701+0000) 2022-09-10T15:19:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:42.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:42.524838+0000) 2022-09-10T15:19:43.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:43.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:43.153887+0000) 2022-09-10T15:19:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:43 smithi110 ceph-mon[97633]: pgmap v20812: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:43 smithi104 ceph-mon[97044]: pgmap v20812: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:43.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:43.525076+0000) 2022-09-10T15:19:44.527 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:44.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:44.154175+0000) 2022-09-10T15:19:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:44.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:44.525300+0000) 2022-09-10T15:19:45.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:45.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:45.154376+0000) 2022-09-10T15:19:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:45 smithi110 ceph-mon[97633]: pgmap v20813: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:45 smithi104 ceph-mon[97044]: pgmap v20813: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:45.524+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:45.525464+0000) 2022-09-10T15:19:46.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:46.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:46.154593+0000) 2022-09-10T15:19:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:46.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:46.525695+0000) 2022-09-10T15:19:47.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:47.153+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:47.154788+0000) 2022-09-10T15:19:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:47 smithi110 ceph-mon[97633]: pgmap v20814: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:19:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:19:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:19:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:19:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:47 smithi104 ceph-mon[97044]: pgmap v20814: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:19:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:19:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:19:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:19:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:47.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:47.525949+0000) 2022-09-10T15:19:48.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:48.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:48.155051+0000) 2022-09-10T15:19:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:48 smithi104 ceph-mon[97044]: pgmap v20815: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:48.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:48.526216+0000) 2022-09-10T15:19:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:48 smithi110 ceph-mon[97633]: pgmap v20815: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:49.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:49.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:49.155302+0000) 2022-09-10T15:19:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:49.525+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:49.526366+0000) 2022-09-10T15:19:50.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:50.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:50.155521+0000) 2022-09-10T15:19:50.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:50.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:50.526601+0000) 2022-09-10T15:19:51.528 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:51.154+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:51.155684+0000) 2022-09-10T15:19:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:51 smithi110 ceph-mon[97633]: pgmap v20816: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:51 smithi104 ceph-mon[97044]: pgmap v20816: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:51.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:51.526771+0000) 2022-09-10T15:19:52.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:52.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:52.155922+0000) 2022-09-10T15:19:52.529 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:19:52.308+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:19:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:52.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:52.526980+0000) 2022-09-10T15:19:53.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:53.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:53.156166+0000) 2022-09-10T15:19:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:53 smithi110 ceph-mon[97633]: pgmap v20817: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:53.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:53 smithi104 ceph-mon[97044]: pgmap v20817: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:53.526+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:53.527235+0000) 2022-09-10T15:19:54.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:54.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:54.156418+0000) 2022-09-10T15:19:54.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:54.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:54.527480+0000) 2022-09-10T15:19:55.529 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:55.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:55.156650+0000) 2022-09-10T15:19:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:55 smithi110 ceph-mon[97633]: pgmap v20818: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:55 smithi104 ceph-mon[97044]: pgmap v20818: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:55.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:55.527681+0000) 2022-09-10T15:19:56.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:56.155+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:56.156929+0000) 2022-09-10T15:19:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:56.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:56.527889+0000) 2022-09-10T15:19:57.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:57.156+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:57.157150+0000) 2022-09-10T15:19:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:57 smithi110 ceph-mon[97633]: pgmap v20819: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:57 smithi104 ceph-mon[97044]: pgmap v20819: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:19:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:57.527+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:57.528040+0000) 2022-09-10T15:19:58.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:58.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:58.157394+0000) 2022-09-10T15:19:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:19:58 smithi104 ceph-mon[97044]: pgmap v20820: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:58.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:58.528217+0000) 2022-09-10T15:19:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:19:58 smithi110 ceph-mon[97633]: pgmap v20820: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:19:59.530 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:19:59.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:59.157562+0000) 2022-09-10T15:19:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:19:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:19:59.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:19:59.528442+0000) 2022-09-10T15:20:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:20:00.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:00.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:00.157778+0000) 2022-09-10T15:20:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:20:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:00.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:00.528620+0000) 2022-09-10T15:20:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:01 smithi110 ceph-mon[97633]: pgmap v20821: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:01.469 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:01 smithi104 ceph-mon[97044]: pgmap v20821: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:01.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:01.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:01.157961+0000) 2022-09-10T15:20:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:01.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:01.528884+0000) 2022-09-10T15:20:02.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:02.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:02.158178+0000) 2022-09-10T15:20:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:02.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:02.529064+0000) 2022-09-10T15:20:03.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:03.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:03.158408+0000) 2022-09-10T15:20:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:03 smithi110 ceph-mon[97633]: pgmap v20822: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:03 smithi104 ceph-mon[97044]: pgmap v20822: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:03.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:03.529207+0000) 2022-09-10T15:20:04.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:04.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:04.158617+0000) 2022-09-10T15:20:04.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:04.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:04.529418+0000) 2022-09-10T15:20:05.531 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:05.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:05.158849+0000) 2022-09-10T15:20:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:05 smithi110 ceph-mon[97633]: pgmap v20823: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:05 smithi104 ceph-mon[97044]: pgmap v20823: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:05.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:05.529645+0000) 2022-09-10T15:20:06.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:06.157+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:06.159041+0000) 2022-09-10T15:20:06.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:06.528+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:06.529878+0000) 2022-09-10T15:20:07.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:07.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:07.159232+0000) 2022-09-10T15:20:07.532 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:20:07.324+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:20:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:07 smithi110 ceph-mon[97633]: pgmap v20824: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:20:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:07 smithi104 ceph-mon[97044]: pgmap v20824: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:20:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:07.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:07.530145+0000) 2022-09-10T15:20:08.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:08.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:08.159533+0000) 2022-09-10T15:20:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:08 smithi104 ceph-mon[97044]: pgmap v20825: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:20:08.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:08.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:08.530331+0000) 2022-09-10T15:20:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:08 smithi110 ceph-mon[97633]: pgmap v20825: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:20:09.532 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:09.158+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:09.159803+0000) 2022-09-10T15:20:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:09 smithi104 ceph-mon[97044]: osdmap e1382: 12 total, 12 up, 12 in 2022-09-10T15:20:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:09.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:09.530498+0000) 2022-09-10T15:20:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:09 smithi110 ceph-mon[97633]: osdmap e1382: 12 total, 12 up, 12 in 2022-09-10T15:20:10.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:10.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:10.160071+0000) 2022-09-10T15:20:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:10 smithi104 ceph-mon[97044]: pgmap v20827: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:20:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:10.529+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:10.530752+0000) 2022-09-10T15:20:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:10 smithi110 ceph-mon[97633]: pgmap v20827: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:20:11.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:11.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:11.160308+0000) 2022-09-10T15:20:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:11.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:11.530990+0000) 2022-09-10T15:20:12.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:12.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:12.160520+0000) 2022-09-10T15:20:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:12.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:12.531185+0000) 2022-09-10T15:20:13.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:13.159+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:13.160706+0000) 2022-09-10T15:20:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:13 smithi110 ceph-mon[97633]: pgmap v20828: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:20:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:13 smithi110 ceph-mon[97633]: osdmap e1383: 12 total, 12 up, 12 in 2022-09-10T15:20:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:13 smithi104 ceph-mon[97044]: pgmap v20828: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:20:13.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:13 smithi104 ceph-mon[97044]: osdmap e1383: 12 total, 12 up, 12 in 2022-09-10T15:20:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:13.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:13.531347+0000) 2022-09-10T15:20:14.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:14.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:14.160926+0000) 2022-09-10T15:20:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:14.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:14.531577+0000) 2022-09-10T15:20:15.533 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:15.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:15.161145+0000) 2022-09-10T15:20:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:15 smithi110 ceph-mon[97633]: pgmap v20830: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:20:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:15 smithi104 ceph-mon[97044]: pgmap v20830: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:20:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:15.530+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:15.531757+0000) 2022-09-10T15:20:16.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:16.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:16.161376+0000) 2022-09-10T15:20:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:16.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:16.531957+0000) 2022-09-10T15:20:17.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:17.160+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:17.161641+0000) 2022-09-10T15:20:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:17 smithi110 ceph-mon[97633]: pgmap v20831: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:20:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:17 smithi104 ceph-mon[97044]: pgmap v20831: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:20:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:17.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:17.532175+0000) 2022-09-10T15:20:18.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:18.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:18.161886+0000) 2022-09-10T15:20:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:18 smithi104 ceph-mon[97044]: pgmap v20832: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:20:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:18.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:18.532394+0000) 2022-09-10T15:20:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:18 smithi110 ceph-mon[97633]: pgmap v20832: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:20:19.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:19.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:19.162198+0000) 2022-09-10T15:20:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:19.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:19.532672+0000) 2022-09-10T15:20:20.534 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:20.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:20.162424+0000) 2022-09-10T15:20:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:20.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:20.532858+0000) 2022-09-10T15:20:21.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:21.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:21.162613+0000) 2022-09-10T15:20:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:21 smithi110 ceph-mon[97633]: pgmap v20833: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:20:21.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:21 smithi104 ceph-mon[97044]: pgmap v20833: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:20:21.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:21.531+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:21.533076+0000) 2022-09-10T15:20:22.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:22.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:22.162825+0000) 2022-09-10T15:20:22.535 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:20:22.339+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:20:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:22.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:22.533239+0000) 2022-09-10T15:20:23.535 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:23.161+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:23.163044+0000) 2022-09-10T15:20:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:23 smithi110 ceph-mon[97633]: pgmap v20834: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:20:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:23 smithi104 ceph-mon[97044]: pgmap v20834: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:20:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:23.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:23.533505+0000) 2022-09-10T15:20:24.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:24.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:24.163256+0000) 2022-09-10T15:20:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:24.532+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:24.533721+0000) 2022-09-10T15:20:25.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:25.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:25.163460+0000) 2022-09-10T15:20:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:25 smithi110 ceph-mon[97633]: pgmap v20835: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:20:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:25 smithi104 ceph-mon[97044]: pgmap v20835: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:20:25.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:25.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:25.533945+0000) 2022-09-10T15:20:26.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:26.162+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:26.163651+0000) 2022-09-10T15:20:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:26.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:26.534198+0000) 2022-09-10T15:20:27.536 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:27.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:27.163861+0000) 2022-09-10T15:20:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:27 smithi110 ceph-mon[97633]: pgmap v20836: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:20:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:27 smithi104 ceph-mon[97044]: pgmap v20836: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:20:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:27.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:27.534414+0000) 2022-09-10T15:20:28.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:28.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:28.164072+0000) 2022-09-10T15:20:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:28 smithi104 ceph-mon[97044]: pgmap v20837: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:28.814 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:28.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:28.534615+0000) 2022-09-10T15:20:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:28 smithi110 ceph-mon[97633]: pgmap v20837: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:29.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:29.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:29.164340+0000) 2022-09-10T15:20:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:29.533+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:29.534773+0000) 2022-09-10T15:20:30.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:30.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:30.164543+0000) 2022-09-10T15:20:30.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:30.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:30.535005+0000) 2022-09-10T15:20:31.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:31.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:31.164774+0000) 2022-09-10T15:20:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:31 smithi110 ceph-mon[97633]: pgmap v20838: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:31 smithi104 ceph-mon[97044]: pgmap v20838: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:31.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:31.535202+0000) 2022-09-10T15:20:32.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:32.163+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:32.164995+0000) 2022-09-10T15:20:32.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:32.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:32.535429+0000) 2022-09-10T15:20:33.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:33.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:33.165295+0000) 2022-09-10T15:20:33.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:33 smithi110 ceph-mon[97633]: pgmap v20839: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:33 smithi104 ceph-mon[97044]: pgmap v20839: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:33.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:33.535668+0000) 2022-09-10T15:20:34.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:34.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:34.165532+0000) 2022-09-10T15:20:34.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:34.534+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:34.535929+0000) 2022-09-10T15:20:35.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:35.164+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:35.165769+0000) 2022-09-10T15:20:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:35 smithi110 ceph-mon[97633]: pgmap v20840: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:35 smithi104 ceph-mon[97044]: pgmap v20840: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:35.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:35.536203+0000) 2022-09-10T15:20:36.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:36.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:36.166026+0000) 2022-09-10T15:20:36.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:36.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:36.536416+0000) 2022-09-10T15:20:37.538 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:37.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:37.166265+0000) 2022-09-10T15:20:37.539 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:20:37.355+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:20:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:37 smithi110 ceph-mon[97633]: pgmap v20841: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:37 smithi104 ceph-mon[97044]: pgmap v20841: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:37.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:37.536645+0000) 2022-09-10T15:20:38.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:38.165+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:38.166443+0000) 2022-09-10T15:20:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:38 smithi104 ceph-mon[97044]: pgmap v20842: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:38.535+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:38.536916+0000) 2022-09-10T15:20:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:38 smithi110 ceph-mon[97633]: pgmap v20842: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:39.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:39.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:39.166681+0000) 2022-09-10T15:20:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:39.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:39.537200+0000) 2022-09-10T15:20:40.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:40.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:40.166900+0000) 2022-09-10T15:20:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:40.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:40.537361+0000) 2022-09-10T15:20:41.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:41.166+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:41.167187+0000) 2022-09-10T15:20:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:41 smithi110 ceph-mon[97633]: pgmap v20843: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:41 smithi104 ceph-mon[97044]: pgmap v20843: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:41.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:41.537543+0000) 2022-09-10T15:20:42.539 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:42.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:42.167385+0000) 2022-09-10T15:20:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:42.536+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:42.537741+0000) 2022-09-10T15:20:43.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:43.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:43.167591+0000) 2022-09-10T15:20:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:43 smithi110 ceph-mon[97633]: pgmap v20844: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:43 smithi104 ceph-mon[97044]: pgmap v20844: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:43.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:43.538005+0000) 2022-09-10T15:20:44.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:44.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:44.167791+0000) 2022-09-10T15:20:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:44.537+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:44.538215+0000) 2022-09-10T15:20:45.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:45.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:45.168054+0000) 2022-09-10T15:20:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:45 smithi110 ceph-mon[97633]: pgmap v20845: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:45 smithi104 ceph-mon[97044]: pgmap v20845: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:45.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:45.538456+0000) 2022-09-10T15:20:46.540 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:46.167+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:46.168246+0000) 2022-09-10T15:20:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:46.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:46.538698+0000) 2022-09-10T15:20:47.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:47.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:47.168421+0000) 2022-09-10T15:20:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:47 smithi110 ceph-mon[97633]: pgmap v20846: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:20:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:20:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:20:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:20:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:47 smithi104 ceph-mon[97044]: pgmap v20846: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:20:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:20:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:20:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:20:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:47.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:47.538926+0000) 2022-09-10T15:20:48.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:48.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:48.168565+0000) 2022-09-10T15:20:48.803 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:48 smithi104 ceph-mon[97044]: pgmap v20847: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:48.538+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:48.539172+0000) 2022-09-10T15:20:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:48 smithi110 ceph-mon[97633]: pgmap v20847: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:49.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:49.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:49.168721+0000) 2022-09-10T15:20:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:49.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:49.539314+0000) 2022-09-10T15:20:50.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:50.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:50.168924+0000) 2022-09-10T15:20:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:50.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:50.539488+0000) 2022-09-10T15:20:51.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:51.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:51.169195+0000) 2022-09-10T15:20:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:51 smithi110 ceph-mon[97633]: pgmap v20848: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:51 smithi104 ceph-mon[97044]: pgmap v20848: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:51.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:51.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:51.539664+0000) 2022-09-10T15:20:52.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:52.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:52.169421+0000) 2022-09-10T15:20:52.543 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:20:52.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:20:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:52.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:52.539887+0000) 2022-09-10T15:20:53.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:53.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:53.169623+0000) 2022-09-10T15:20:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:53 smithi110 ceph-mon[97633]: pgmap v20849: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:53 smithi104 ceph-mon[97044]: pgmap v20849: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:53.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:53.540079+0000) 2022-09-10T15:20:54.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:54.168+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:54.169851+0000) 2022-09-10T15:20:54.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:54.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:54.540324+0000) 2022-09-10T15:20:55.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:55.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:55.170080+0000) 2022-09-10T15:20:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:55 smithi110 ceph-mon[97633]: pgmap v20850: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:55 smithi104 ceph-mon[97044]: pgmap v20850: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:55.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:55.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:55.540536+0000) 2022-09-10T15:20:56.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:56.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:56.170325+0000) 2022-09-10T15:20:56.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:56.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:56.540694+0000) 2022-09-10T15:20:57.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:57.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:57.170612+0000) 2022-09-10T15:20:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:57 smithi110 ceph-mon[97633]: pgmap v20851: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:57 smithi104 ceph-mon[97044]: pgmap v20851: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:20:57.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:57.539+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:57.540866+0000) 2022-09-10T15:20:58.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:58.169+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:58.170857+0000) 2022-09-10T15:20:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:20:58 smithi104 ceph-mon[97044]: pgmap v20852: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:58.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:58.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:58.541121+0000) 2022-09-10T15:20:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:20:58 smithi110 ceph-mon[97633]: pgmap v20852: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:20:59.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:20:59.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:59.171134+0000) 2022-09-10T15:20:59.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:20:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:20:59.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:20:59.541326+0000) 2022-09-10T15:21:00.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:00.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:00.171337+0000) 2022-09-10T15:21:00.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:00.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:00.541486+0000) 2022-09-10T15:21:01.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:01.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:01.171477+0000) 2022-09-10T15:21:01.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:01 smithi110 ceph-mon[97633]: pgmap v20853: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:01 smithi104 ceph-mon[97044]: pgmap v20853: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:01.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:01.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:01.541719+0000) 2022-09-10T15:21:02.543 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:02.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:02.171641+0000) 2022-09-10T15:21:02.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:02.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:02.541894+0000) 2022-09-10T15:21:03.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:03.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:03.171825+0000) 2022-09-10T15:21:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:03 smithi110 ceph-mon[97633]: pgmap v20854: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:03 smithi104 ceph-mon[97044]: pgmap v20854: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:03.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:03.540+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:03.542064+0000) 2022-09-10T15:21:04.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:04.170+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:04.172041+0000) 2022-09-10T15:21:04.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:04.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:04.542270+0000) 2022-09-10T15:21:05.544 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:05.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:05.172201+0000) 2022-09-10T15:21:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:05 smithi110 ceph-mon[97633]: pgmap v20855: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:05 smithi104 ceph-mon[97044]: pgmap v20855: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:05.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:05.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:05.542536+0000) 2022-09-10T15:21:06.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:06.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:06.172410+0000) 2022-09-10T15:21:06.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:06.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:06.542808+0000) 2022-09-10T15:21:07.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:07.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:07.172649+0000) 2022-09-10T15:21:07.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:21:07.372+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:21:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:07 smithi110 ceph-mon[97633]: pgmap v20856: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:21:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:07 smithi104 ceph-mon[97044]: pgmap v20856: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:21:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:07.541+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:07.542981+0000) 2022-09-10T15:21:08.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:08.171+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:08.172840+0000) 2022-09-10T15:21:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:08 smithi104 ceph-mon[97044]: pgmap v20857: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:21:08.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:08.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:08.543162+0000) 2022-09-10T15:21:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:08 smithi110 ceph-mon[97633]: pgmap v20857: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:21:09.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:09.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:09.173055+0000) 2022-09-10T15:21:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:09 smithi104 ceph-mon[97044]: osdmap e1384: 12 total, 12 up, 12 in 2022-09-10T15:21:09.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:09.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:09.543334+0000) 2022-09-10T15:21:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:09 smithi110 ceph-mon[97633]: osdmap e1384: 12 total, 12 up, 12 in 2022-09-10T15:21:10.545 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:10.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:10.173324+0000) 2022-09-10T15:21:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:10 smithi104 ceph-mon[97044]: pgmap v20859: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:21:10.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:10.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:10.543581+0000) 2022-09-10T15:21:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:10 smithi110 ceph-mon[97633]: pgmap v20859: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:21:11.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:11.172+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:11.173603+0000) 2022-09-10T15:21:11.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:11.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:11.543800+0000) 2022-09-10T15:21:12.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:12.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:12.173868+0000) 2022-09-10T15:21:12.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:12.542+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:12.544008+0000) 2022-09-10T15:21:13.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:13.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:13.174156+0000) 2022-09-10T15:21:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:13 smithi110 ceph-mon[97633]: pgmap v20860: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:21:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:13 smithi104 ceph-mon[97044]: pgmap v20860: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:21:13.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:13.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:13.544255+0000) 2022-09-10T15:21:14.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:14.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:14.174300+0000) 2022-09-10T15:21:14.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:14.543+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:14.544458+0000) 2022-09-10T15:21:15.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:15.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:15.174420+0000) 2022-09-10T15:21:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:15 smithi110 ceph-mon[97633]: pgmap v20861: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:21:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:15 smithi104 ceph-mon[97044]: pgmap v20861: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:21:15.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:15.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:15.544703+0000) 2022-09-10T15:21:16.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:16.173+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:16.174653+0000) 2022-09-10T15:21:16.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:16.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:16.544941+0000) 2022-09-10T15:21:17.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:17.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:17.174902+0000) 2022-09-10T15:21:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:17 smithi110 ceph-mon[97633]: pgmap v20862: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:21:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:17 smithi104 ceph-mon[97044]: pgmap v20862: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:21:17.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:17.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:17.545197+0000) 2022-09-10T15:21:18.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:18.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:18.175137+0000) 2022-09-10T15:21:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:18 smithi104 ceph-mon[97044]: osdmap e1385: 12 total, 12 up, 12 in 2022-09-10T15:21:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:18 smithi104 ceph-mon[97044]: pgmap v20864: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:21:18.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:18.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:18.545368+0000) 2022-09-10T15:21:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:18 smithi110 ceph-mon[97633]: osdmap e1385: 12 total, 12 up, 12 in 2022-09-10T15:21:18.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:18 smithi110 ceph-mon[97633]: pgmap v20864: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:21:19.547 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:19.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:19.175376+0000) 2022-09-10T15:21:19.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:19.544+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:19.545577+0000) 2022-09-10T15:21:20.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:20.174+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:20.175589+0000) 2022-09-10T15:21:20.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:20.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:20.545814+0000) 2022-09-10T15:21:21.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:21.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:21.175801+0000) 2022-09-10T15:21:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:21 smithi110 ceph-mon[97633]: pgmap v20865: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:21:21.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:21 smithi104 ceph-mon[97044]: pgmap v20865: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:21:21.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:21.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:21.546111+0000) 2022-09-10T15:21:22.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:22.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:22.176013+0000) 2022-09-10T15:21:22.548 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:21:22.387+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:21:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:22.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:22.546294+0000) 2022-09-10T15:21:23.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:23.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:23.176214+0000) 2022-09-10T15:21:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:23 smithi110 ceph-mon[97633]: pgmap v20866: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:21:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:23 smithi104 ceph-mon[97044]: pgmap v20866: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:21:23.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:23.545+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:23.546444+0000) 2022-09-10T15:21:24.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:24.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:24.176458+0000) 2022-09-10T15:21:24.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:24.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:24.546615+0000) 2022-09-10T15:21:25.548 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:25.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:25.176721+0000) 2022-09-10T15:21:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:25 smithi110 ceph-mon[97633]: pgmap v20867: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:21:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:25 smithi104 ceph-mon[97044]: pgmap v20867: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:21:25.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:25.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:25.546843+0000) 2022-09-10T15:21:26.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:26.175+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:26.176911+0000) 2022-09-10T15:21:26.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:26.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:26.547079+0000) 2022-09-10T15:21:27.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:27.176+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:27.177193+0000) 2022-09-10T15:21:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:27 smithi110 ceph-mon[97633]: pgmap v20868: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:21:27.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:27 smithi104 ceph-mon[97044]: pgmap v20868: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:21:27.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:27.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:27.547282+0000) 2022-09-10T15:21:28.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:28.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:28.177441+0000) 2022-09-10T15:21:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:28 smithi104 ceph-mon[97044]: pgmap v20869: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:21:28.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:28.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:28.547493+0000) 2022-09-10T15:21:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:28 smithi110 ceph-mon[97633]: pgmap v20869: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:21:29.549 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:29.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:29.177717+0000) 2022-09-10T15:21:29.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:29.546+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:29.547638+0000) 2022-09-10T15:21:30.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:30.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:30.177971+0000) 2022-09-10T15:21:30.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:30.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:30.547871+0000) 2022-09-10T15:21:31.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:31.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:31.178220+0000) 2022-09-10T15:21:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:31 smithi110 ceph-mon[97633]: pgmap v20870: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:31 smithi104 ceph-mon[97044]: pgmap v20870: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:31.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:31.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:31.548183+0000) 2022-09-10T15:21:32.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:32.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:32.178464+0000) 2022-09-10T15:21:32.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:32.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:32.548348+0000) 2022-09-10T15:21:33.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:33.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:33.178696+0000) 2022-09-10T15:21:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:33 smithi110 ceph-mon[97633]: pgmap v20871: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:33 smithi104 ceph-mon[97044]: pgmap v20871: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:33.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:33.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:33.548616+0000) 2022-09-10T15:21:34.550 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:34.177+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:34.178949+0000) 2022-09-10T15:21:34.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:34.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:34.548821+0000) 2022-09-10T15:21:35.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:35.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:35.179192+0000) 2022-09-10T15:21:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:35 smithi110 ceph-mon[97633]: pgmap v20872: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:35 smithi104 ceph-mon[97044]: pgmap v20872: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:35.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:35.547+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:35.549031+0000) 2022-09-10T15:21:36.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:36.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:36.179377+0000) 2022-09-10T15:21:36.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:36.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:36.549272+0000) 2022-09-10T15:21:37.551 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:37.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:37.179595+0000) 2022-09-10T15:21:37.552 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:21:37.392+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:21:37.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:37 smithi110 ceph-mon[97633]: pgmap v20873: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:37 smithi104 ceph-mon[97044]: pgmap v20873: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:37.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:37.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:37.549505+0000) 2022-09-10T15:21:38.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:38.178+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:38.179782+0000) 2022-09-10T15:21:38.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:38 smithi104 ceph-mon[97044]: pgmap v20874: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:21:38.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:38.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:38.549707+0000) 2022-09-10T15:21:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:38 smithi110 ceph-mon[97633]: pgmap v20874: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:21:39.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:39.179+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:39.180009+0000) 2022-09-10T15:21:39.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:39.549+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:39.549904+0000) 2022-09-10T15:21:40.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:40.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:40.180228+0000) 2022-09-10T15:21:40.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:40.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:40.550173+0000) 2022-09-10T15:21:41.552 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:41.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:41.180487+0000) 2022-09-10T15:21:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:41 smithi110 ceph-mon[97633]: pgmap v20875: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:41 smithi104 ceph-mon[97044]: pgmap v20875: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:41.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:41.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:41.550465+0000) 2022-09-10T15:21:42.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:42.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:42.180702+0000) 2022-09-10T15:21:42.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:42.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:42.550700+0000) 2022-09-10T15:21:43.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:43.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:43.180932+0000) 2022-09-10T15:21:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:43 smithi110 ceph-mon[97633]: pgmap v20876: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:21:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:43 smithi104 ceph-mon[97044]: pgmap v20876: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:21:43.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:43.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:43.550929+0000) 2022-09-10T15:21:44.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:44.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:44.181218+0000) 2022-09-10T15:21:44.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:44.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:44.551175+0000) 2022-09-10T15:21:45.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:45.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:45.181344+0000) 2022-09-10T15:21:45.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:45 smithi110 ceph-mon[97633]: pgmap v20877: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:21:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:45 smithi104 ceph-mon[97044]: pgmap v20877: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:21:45.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:45.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:45.551382+0000) 2022-09-10T15:21:46.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:46.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:46.181527+0000) 2022-09-10T15:21:46.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:46.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:46.551583+0000) 2022-09-10T15:21:47.553 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:47.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:47.181768+0000) 2022-09-10T15:21:47.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:47 smithi110 ceph-mon[97633]: pgmap v20878: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:21:47.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:21:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:21:47.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:21:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:47 smithi104 ceph-mon[97044]: pgmap v20878: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:21:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:21:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:21:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:21:47.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:47.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:47.551764+0000) 2022-09-10T15:21:48.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:48.180+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:48.181959+0000) 2022-09-10T15:21:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:48 smithi104 ceph-mon[97044]: pgmap v20879: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:48.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:48.550+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:48.551988+0000) 2022-09-10T15:21:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:48 smithi110 ceph-mon[97633]: pgmap v20879: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:49.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:49.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:49.182156+0000) 2022-09-10T15:21:49.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:49.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:49.552152+0000) 2022-09-10T15:21:50.554 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:50.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:50.182352+0000) 2022-09-10T15:21:50.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:50.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:50.552327+0000) 2022-09-10T15:21:51.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:51.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:51.182596+0000) 2022-09-10T15:21:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:51 smithi110 ceph-mon[97633]: pgmap v20880: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:51 smithi104 ceph-mon[97044]: pgmap v20880: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:51.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:51.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:51.552531+0000) 2022-09-10T15:21:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:52.181+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:52.182779+0000) 2022-09-10T15:21:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:21:52.398+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:21:52.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:52.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:52.552778+0000) 2022-09-10T15:21:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:53.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:53.182995+0000) 2022-09-10T15:21:53.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:53 smithi110 ceph-mon[97633]: pgmap v20881: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:53 smithi104 ceph-mon[97044]: pgmap v20881: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:53.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:53.551+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:53.553061+0000) 2022-09-10T15:21:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:54.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:54.183233+0000) 2022-09-10T15:21:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:54.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:54.553319+0000) 2022-09-10T15:21:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:55.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:55.183430+0000) 2022-09-10T15:21:55.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:55 smithi110 ceph-mon[97633]: pgmap v20882: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:55.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:55 smithi104 ceph-mon[97044]: pgmap v20882: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:55.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:55.553546+0000) 2022-09-10T15:21:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:56.182+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:56.183682+0000) 2022-09-10T15:21:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:56.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:56.553772+0000) 2022-09-10T15:21:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:57.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:57.183950+0000) 2022-09-10T15:21:57.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:57 smithi110 ceph-mon[97633]: pgmap v20883: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:57 smithi104 ceph-mon[97044]: pgmap v20883: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:21:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:57.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:57.553927+0000) 2022-09-10T15:21:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:58.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:58.184203+0000) 2022-09-10T15:21:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:21:58 smithi104 ceph-mon[97044]: pgmap v20884: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:21:58 smithi110 ceph-mon[97633]: pgmap v20884: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:21:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:58.552+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:58.554054+0000) 2022-09-10T15:21:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:21:59.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:59.184481+0000) 2022-09-10T15:22:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:21:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:21:59.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:21:59.554176+0000) 2022-09-10T15:22:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:00.183+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:00.184696+0000) 2022-09-10T15:22:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:00.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:00.554293+0000) 2022-09-10T15:22:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:01.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:01.184974+0000) 2022-09-10T15:22:01.660 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:01 smithi110 ceph-mon[97633]: pgmap v20885: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:01.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:01 smithi104 ceph-mon[97044]: pgmap v20885: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:01.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:01.554440+0000) 2022-09-10T15:22:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:02.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:02.185203+0000) 2022-09-10T15:22:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:02.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:02.554638+0000) 2022-09-10T15:22:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:03.184+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:03.185471+0000) 2022-09-10T15:22:03.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:03 smithi110 ceph-mon[97633]: pgmap v20886: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:03 smithi104 ceph-mon[97044]: pgmap v20886: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:03.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:03.554840+0000) 2022-09-10T15:22:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:04.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:04.185720+0000) 2022-09-10T15:22:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:04.553+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:04.555059+0000) 2022-09-10T15:22:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:05.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:05.185977+0000) 2022-09-10T15:22:05.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:05 smithi110 ceph-mon[97633]: pgmap v20887: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:05 smithi104 ceph-mon[97044]: pgmap v20887: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:05.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:05.555240+0000) 2022-09-10T15:22:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:06.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:06.186218+0000) 2022-09-10T15:22:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:06.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:06.555430+0000) 2022-09-10T15:22:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:07.185+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:07.186484+0000) 2022-09-10T15:22:07.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:22:07.404+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:22:07.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:07 smithi110 ceph-mon[97633]: pgmap v20888: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:22:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:07 smithi104 ceph-mon[97044]: pgmap v20888: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:22:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:07.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:07.555565+0000) 2022-09-10T15:22:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:08.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:08.186688+0000) 2022-09-10T15:22:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:08 smithi104 ceph-mon[97044]: pgmap v20889: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:22:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:08 smithi110 ceph-mon[97633]: pgmap v20889: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:22:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:08.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:08.555757+0000) 2022-09-10T15:22:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:09.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:09.186966+0000) 2022-09-10T15:22:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:09 smithi104 ceph-mon[97044]: osdmap e1386: 12 total, 12 up, 12 in 2022-09-10T15:22:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:09 smithi110 ceph-mon[97633]: osdmap e1386: 12 total, 12 up, 12 in 2022-09-10T15:22:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:09.554+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:09.555979+0000) 2022-09-10T15:22:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:10.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:10.187166+0000) 2022-09-10T15:22:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:10 smithi104 ceph-mon[97044]: pgmap v20891: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:22:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:10 smithi110 ceph-mon[97633]: pgmap v20891: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:22:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:10.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:10.556212+0000) 2022-09-10T15:22:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:11.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:11.187293+0000) 2022-09-10T15:22:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:11.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:11.556412+0000) 2022-09-10T15:22:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:12.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:12.187539+0000) 2022-09-10T15:22:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:12.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:12.556534+0000) 2022-09-10T15:22:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:13.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:13.187739+0000) 2022-09-10T15:22:13.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:13 smithi110 ceph-mon[97633]: pgmap v20892: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:22:13.659 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:13 smithi110 ceph-mon[97633]: osdmap e1387: 12 total, 12 up, 12 in 2022-09-10T15:22:13.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:13 smithi104 ceph-mon[97044]: pgmap v20892: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:22:13.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:13 smithi104 ceph-mon[97044]: osdmap e1387: 12 total, 12 up, 12 in 2022-09-10T15:22:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:13.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:13.556678+0000) 2022-09-10T15:22:14.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:14.186+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:14.188033+0000) 2022-09-10T15:22:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:14.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:14.556880+0000) 2022-09-10T15:22:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:15.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:15.188258+0000) 2022-09-10T15:22:15.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:15 smithi110 ceph-mon[97633]: pgmap v20894: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:22:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:15 smithi104 ceph-mon[97044]: pgmap v20894: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2022-09-10T15:22:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:15.555+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:15.557055+0000) 2022-09-10T15:22:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:16.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:16.188467+0000) 2022-09-10T15:22:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:16.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:16.557241+0000) 2022-09-10T15:22:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:17.187+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:17.188620+0000) 2022-09-10T15:22:17.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:17 smithi110 ceph-mon[97633]: pgmap v20895: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:22:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:17 smithi104 ceph-mon[97044]: pgmap v20895: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:22:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:17.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:17.557447+0000) 2022-09-10T15:22:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:18.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:18.188831+0000) 2022-09-10T15:22:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:18 smithi104 ceph-mon[97044]: pgmap v20896: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:22:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:18 smithi110 ceph-mon[97633]: pgmap v20896: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:22:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:18.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:18.557631+0000) 2022-09-10T15:22:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:19.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:19.189020+0000) 2022-09-10T15:22:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:19.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:19.557819+0000) 2022-09-10T15:22:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:20.188+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:20.189204+0000) 2022-09-10T15:22:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:20.556+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:20.557970+0000) 2022-09-10T15:22:21.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:21.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:21.189457+0000) 2022-09-10T15:22:21.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:21 smithi110 ceph-mon[97633]: pgmap v20897: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:22:21.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:21 smithi104 ceph-mon[97044]: pgmap v20897: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:22:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:21.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:21.558217+0000) 2022-09-10T15:22:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:22.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:22.189688+0000) 2022-09-10T15:22:22.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:22:22.407+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:22:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:22.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:22.558379+0000) 2022-09-10T15:22:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:23.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:23.189946+0000) 2022-09-10T15:22:23.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:23 smithi110 ceph-mon[97633]: pgmap v20898: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:22:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:23 smithi104 ceph-mon[97044]: pgmap v20898: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:22:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:23.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:23.558505+0000) 2022-09-10T15:22:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:24.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:24.190232+0000) 2022-09-10T15:22:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:24.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:24.558674+0000) 2022-09-10T15:22:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:25.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:25.190414+0000) 2022-09-10T15:22:25.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:25 smithi110 ceph-mon[97633]: pgmap v20899: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:22:25.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:25 smithi104 ceph-mon[97044]: pgmap v20899: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 86 B/s wr, 8 op/s 2022-09-10T15:22:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:25.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:25.558850+0000) 2022-09-10T15:22:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:26.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:26.190593+0000) 2022-09-10T15:22:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:26.557+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:26.559074+0000) 2022-09-10T15:22:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:27.189+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:27.190859+0000) 2022-09-10T15:22:27.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:27 smithi110 ceph-mon[97633]: pgmap v20900: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:22:27.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:27 smithi104 ceph-mon[97044]: pgmap v20900: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:22:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:27.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:27.559283+0000) 2022-09-10T15:22:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:28.190+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:28.191055+0000) 2022-09-10T15:22:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:28 smithi104 ceph-mon[97044]: pgmap v20901: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:28 smithi110 ceph-mon[97633]: pgmap v20901: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:28.558+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:28.559415+0000) 2022-09-10T15:22:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:29.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:29.191312+0000) 2022-09-10T15:22:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:29.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:29.559586+0000) 2022-09-10T15:22:30.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:30.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:30.191449+0000) 2022-09-10T15:22:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:30.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:30.559798+0000) 2022-09-10T15:22:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:31.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:31.191642+0000) 2022-09-10T15:22:31.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:31 smithi110 ceph-mon[97633]: pgmap v20902: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:31 smithi104 ceph-mon[97044]: pgmap v20902: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:31.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:31.560031+0000) 2022-09-10T15:22:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:32.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:32.191837+0000) 2022-09-10T15:22:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:32.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:32.560187+0000) 2022-09-10T15:22:33.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:33.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:33.192071+0000) 2022-09-10T15:22:33.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:33 smithi110 ceph-mon[97633]: pgmap v20903: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:33 smithi104 ceph-mon[97044]: pgmap v20903: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:33.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:33.560351+0000) 2022-09-10T15:22:34.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:34.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:34.192324+0000) 2022-09-10T15:22:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:34.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:34.560518+0000) 2022-09-10T15:22:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:35.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:35.192677+0000) 2022-09-10T15:22:35.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:35 smithi110 ceph-mon[97633]: pgmap v20904: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:35.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:35 smithi104 ceph-mon[97044]: pgmap v20904: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:35.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:35.560724+0000) 2022-09-10T15:22:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:36.191+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:36.192900+0000) 2022-09-10T15:22:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:36.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:36.560891+0000) 2022-09-10T15:22:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:37.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:37.193092+0000) 2022-09-10T15:22:37.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:22:37.418+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:22:37.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:37 smithi104 ceph-mon[97044]: pgmap v20905: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:37 smithi110 ceph-mon[97633]: pgmap v20905: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:37.559+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:37.561053+0000) 2022-09-10T15:22:38.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:38.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:38.193244+0000) 2022-09-10T15:22:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:38 smithi104 ceph-mon[97044]: pgmap v20906: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:38 smithi110 ceph-mon[97633]: pgmap v20906: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:38.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:38.561207+0000) 2022-09-10T15:22:39.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:39.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:39.193412+0000) 2022-09-10T15:22:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:39.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:39.561352+0000) 2022-09-10T15:22:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:40.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:40.193635+0000) 2022-09-10T15:22:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:40.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:40.561512+0000) 2022-09-10T15:22:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:41.192+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:41.193888+0000) 2022-09-10T15:22:41.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:41 smithi110 ceph-mon[97633]: pgmap v20907: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:41 smithi104 ceph-mon[97044]: pgmap v20907: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:42.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:41.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:41.561765+0000) 2022-09-10T15:22:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:42.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:42.194119+0000) 2022-09-10T15:22:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:42.560+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:42.561949+0000) 2022-09-10T15:22:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:43.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:43.194261+0000) 2022-09-10T15:22:43.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:43 smithi110 ceph-mon[97633]: pgmap v20908: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:43.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:43 smithi104 ceph-mon[97044]: pgmap v20908: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:43.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:43.562221+0000) 2022-09-10T15:22:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:44.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:44.194486+0000) 2022-09-10T15:22:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:44.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:44.562412+0000) 2022-09-10T15:22:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:45.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:45.194709+0000) 2022-09-10T15:22:45.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:45 smithi104 ceph-mon[97044]: pgmap v20909: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:45 smithi110 ceph-mon[97633]: pgmap v20909: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:45.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:45.562561+0000) 2022-09-10T15:22:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:46.193+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:46.194923+0000) 2022-09-10T15:22:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:46.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:46.562752+0000) 2022-09-10T15:22:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:47.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:47.195120+0000) 2022-09-10T15:22:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:47 smithi104 ceph-mon[97044]: pgmap v20910: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:22:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:22:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:22:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:22:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:47 smithi110 ceph-mon[97633]: pgmap v20910: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:22:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:22:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:22:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:22:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:47.561+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:47.563001+0000) 2022-09-10T15:22:48.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:48.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:48.195335+0000) 2022-09-10T15:22:48.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:48 smithi104 ceph-mon[97044]: pgmap v20911: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:48 smithi110 ceph-mon[97633]: pgmap v20911: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:49.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:48.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:48.563232+0000) 2022-09-10T15:22:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:49.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:49.195516+0000) 2022-09-10T15:22:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:49.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:49.563409+0000) 2022-09-10T15:22:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:50.194+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:50.195721+0000) 2022-09-10T15:22:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:50.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:50.563657+0000) 2022-09-10T15:22:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:51.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:51.195953+0000) 2022-09-10T15:22:51.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:51 smithi110 ceph-mon[97633]: pgmap v20912: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:51 smithi104 ceph-mon[97044]: pgmap v20912: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:51.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:51.563871+0000) 2022-09-10T15:22:52.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:22:52.429+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:22:52.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:52.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:52.196197+0000) 2022-09-10T15:22:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:52.562+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:52.564050+0000) 2022-09-10T15:22:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:53.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:53.196409+0000) 2022-09-10T15:22:53.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:53 smithi104 ceph-mon[97044]: pgmap v20913: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:53.911 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:53 smithi110 ceph-mon[97633]: pgmap v20913: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:54.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:53.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:53.564271+0000) 2022-09-10T15:22:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:54.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:54.196618+0000) 2022-09-10T15:22:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:54.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:54.564488+0000) 2022-09-10T15:22:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:55.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:55.196829+0000) 2022-09-10T15:22:55.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:55 smithi104 ceph-mon[97044]: pgmap v20914: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:55.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:55 smithi110 ceph-mon[97633]: pgmap v20914: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:55.563+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:55.564717+0000) 2022-09-10T15:22:56.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:56.195+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:56.197048+0000) 2022-09-10T15:22:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:56.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:56.564949+0000) 2022-09-10T15:22:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:57.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:57.197273+0000) 2022-09-10T15:22:57.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:57 smithi104 ceph-mon[97044]: pgmap v20915: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:57 smithi110 ceph-mon[97633]: pgmap v20915: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:22:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:57.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:57.565205+0000) 2022-09-10T15:22:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:58.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:58.197446+0000) 2022-09-10T15:22:58.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:22:58 smithi104 ceph-mon[97044]: pgmap v20916: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:22:58 smithi110 ceph-mon[97633]: pgmap v20916: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:22:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:58.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:58.565444+0000) 2022-09-10T15:22:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:22:59.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:59.197647+0000) 2022-09-10T15:23:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:22:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:22:59.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:22:59.565656+0000) 2022-09-10T15:23:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:00.196+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:00.197895+0000) 2022-09-10T15:23:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:00.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:00.565800+0000) 2022-09-10T15:23:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:01.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:01.198185+0000) 2022-09-10T15:23:01.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:01 smithi104 ceph-mon[97044]: pgmap v20917: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:01 smithi110 ceph-mon[97633]: pgmap v20917: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:01.564+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:01.566031+0000) 2022-09-10T15:23:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:02.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:02.198369+0000) 2022-09-10T15:23:03.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:02.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:02.566242+0000) 2022-09-10T15:23:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:03.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:03.198584+0000) 2022-09-10T15:23:03.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:03 smithi104 ceph-mon[97044]: pgmap v20918: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:03.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:03 smithi110 ceph-mon[97633]: pgmap v20918: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:03.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:03.566395+0000) 2022-09-10T15:23:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:04.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:04.198845+0000) 2022-09-10T15:23:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:04.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:04.566594+0000) 2022-09-10T15:23:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:05.197+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:05.198999+0000) 2022-09-10T15:23:05.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:05 smithi104 ceph-mon[97044]: pgmap v20919: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:23:05.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:05 smithi110 ceph-mon[97633]: pgmap v20919: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:23:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:05.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:05.566816+0000) 2022-09-10T15:23:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:06.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:06.199242+0000) 2022-09-10T15:23:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:06.565+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:06.567034+0000) 2022-09-10T15:23:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:07.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:07.199434+0000) 2022-09-10T15:23:07.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:23:07.444+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:23:07.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:07 smithi104 ceph-mon[97044]: pgmap v20920: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:23:07.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:07 smithi110 ceph-mon[97633]: pgmap v20920: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:23:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:07.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:07.567203+0000) 2022-09-10T15:23:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:08.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:08.199606+0000) 2022-09-10T15:23:08.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:08 smithi104 ceph-mon[97044]: pgmap v20921: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:23:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:08 smithi110 ceph-mon[97633]: pgmap v20921: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:23:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:08.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:08.567340+0000) 2022-09-10T15:23:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:09.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:09.199822+0000) 2022-09-10T15:23:09.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:09 smithi104 ceph-mon[97044]: osdmap e1388: 12 total, 12 up, 12 in 2022-09-10T15:23:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:09 smithi110 ceph-mon[97633]: osdmap e1388: 12 total, 12 up, 12 in 2022-09-10T15:23:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:09.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:09.567545+0000) 2022-09-10T15:23:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:10.198+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:10.200059+0000) 2022-09-10T15:23:10.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:10 smithi104 ceph-mon[97044]: pgmap v20923: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:23:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:10 smithi110 ceph-mon[97633]: pgmap v20923: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:23:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:10.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:10.567784+0000) 2022-09-10T15:23:11.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:11.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:11.200343+0000) 2022-09-10T15:23:12.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:11.566+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:11.568063+0000) 2022-09-10T15:23:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:12.199+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:12.200516+0000) 2022-09-10T15:23:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:12.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:12.568224+0000) 2022-09-10T15:23:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:13.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:13.200763+0000) 2022-09-10T15:23:13.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:13 smithi104 ceph-mon[97044]: pgmap v20924: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:23:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:13 smithi110 ceph-mon[97633]: pgmap v20924: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:23:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:13.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:13.568365+0000) 2022-09-10T15:23:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:14.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:14.200989+0000) 2022-09-10T15:23:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:14.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:14.568529+0000) 2022-09-10T15:23:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:15.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:15.201159+0000) 2022-09-10T15:23:15.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:15 smithi104 ceph-mon[97044]: pgmap v20925: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 25 op/s 2022-09-10T15:23:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:15 smithi110 ceph-mon[97633]: pgmap v20925: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 25 op/s 2022-09-10T15:23:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:15.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:15.568721+0000) 2022-09-10T15:23:16.568 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:16.200+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:16.201359+0000) 2022-09-10T15:23:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:16.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:16.568913+0000) 2022-09-10T15:23:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:17.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:17.201632+0000) 2022-09-10T15:23:17.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:17 smithi104 ceph-mon[97044]: pgmap v20926: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:23:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:17 smithi110 ceph-mon[97633]: pgmap v20926: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:23:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:17.567+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:17.569040+0000) 2022-09-10T15:23:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:18.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:18.201885+0000) 2022-09-10T15:23:18.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:18 smithi104 ceph-mon[97044]: osdmap e1389: 12 total, 12 up, 12 in 2022-09-10T15:23:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:18 smithi104 ceph-mon[97044]: pgmap v20928: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:23:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:18 smithi110 ceph-mon[97633]: osdmap e1389: 12 total, 12 up, 12 in 2022-09-10T15:23:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:18 smithi110 ceph-mon[97633]: pgmap v20928: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:23:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:18.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:18.569263+0000) 2022-09-10T15:23:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:19.201+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:19.202058+0000) 2022-09-10T15:23:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:19.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:19.569437+0000) 2022-09-10T15:23:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:20.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:20.202296+0000) 2022-09-10T15:23:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:20.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:20.569628+0000) 2022-09-10T15:23:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:21.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:21.202579+0000) 2022-09-10T15:23:21.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:21 smithi104 ceph-mon[97044]: pgmap v20929: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:23:21.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:21 smithi110 ceph-mon[97633]: pgmap v20929: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:23:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:21.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:21.569882+0000) 2022-09-10T15:23:22.459 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:22.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:22.202757+0000) 2022-09-10T15:23:22.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:22.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:22.570144+0000) 2022-09-10T15:23:22.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:23:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:23:22.457+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:23:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:23.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:23.202924+0000) 2022-09-10T15:23:23.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:23 smithi104 ceph-mon[97044]: pgmap v20930: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:23:23.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:23 smithi110 ceph-mon[97633]: pgmap v20930: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:23:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:23.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:23.570378+0000) 2022-09-10T15:23:24.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:24.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:24.203188+0000) 2022-09-10T15:23:25.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:24.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:24.570620+0000) 2022-09-10T15:23:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:25.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:25.203407+0000) 2022-09-10T15:23:25.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:25 smithi104 ceph-mon[97044]: pgmap v20931: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:23:25.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:25 smithi110 ceph-mon[97633]: pgmap v20931: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:23:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:25.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:25.570775+0000) 2022-09-10T15:23:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:26.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:26.203627+0000) 2022-09-10T15:23:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:26 smithi104 ceph-mon[97044]: pgmap v20932: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:23:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:26 smithi110 ceph-mon[97633]: pgmap v20932: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:23:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:26.569+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:26.571008+0000) 2022-09-10T15:23:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:27.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:27.203824+0000) 2022-09-10T15:23:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:27.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:27.571271+0000) 2022-09-10T15:23:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:28.202+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:28.203992+0000) 2022-09-10T15:23:28.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:28 smithi104 ceph-mon[97044]: pgmap v20933: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:23:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:28 smithi110 ceph-mon[97633]: pgmap v20933: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:23:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:28.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:28.571441+0000) 2022-09-10T15:23:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:29.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:29.204250+0000) 2022-09-10T15:23:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:29.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:29.571635+0000) 2022-09-10T15:23:30.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:30.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:30.204454+0000) 2022-09-10T15:23:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:30.570+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:30.571882+0000) 2022-09-10T15:23:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:31.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:31.204593+0000) 2022-09-10T15:23:31.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:31 smithi104 ceph-mon[97044]: pgmap v20934: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:31 smithi110 ceph-mon[97633]: pgmap v20934: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:31.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:31.572140+0000) 2022-09-10T15:23:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:32.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:32.204799+0000) 2022-09-10T15:23:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:32.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:32.572336+0000) 2022-09-10T15:23:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:33.203+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:33.205018+0000) 2022-09-10T15:23:33.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:33 smithi104 ceph-mon[97044]: pgmap v20935: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:33.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:33 smithi110 ceph-mon[97633]: pgmap v20935: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:33.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:33.572579+0000) 2022-09-10T15:23:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:34.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:34.205224+0000) 2022-09-10T15:23:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:34 smithi104 ceph-mon[97044]: pgmap v20936: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:23:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:34 smithi110 ceph-mon[97633]: pgmap v20936: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:23:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:34.571+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:34.572840+0000) 2022-09-10T15:23:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:35.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:35.205470+0000) 2022-09-10T15:23:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:35.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:35.573064+0000) 2022-09-10T15:23:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:36.204+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:36.205650+0000) 2022-09-10T15:23:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:36.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:36.573346+0000) 2022-09-10T15:23:37.474 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:37.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:37.205891+0000) 2022-09-10T15:23:37.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:37 smithi104 ceph-mon[97044]: pgmap v20937: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:23:37.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:23:37.471+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:23:37.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:37.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:37.573575+0000) 2022-09-10T15:23:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:37 smithi110 ceph-mon[97633]: pgmap v20937: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:23:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:38.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:38.206175+0000) 2022-09-10T15:23:38.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:38 smithi104 ceph-mon[97044]: pgmap v20938: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:38 smithi110 ceph-mon[97633]: pgmap v20938: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:38.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:38.573810+0000) 2022-09-10T15:23:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:39.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:39.206434+0000) 2022-09-10T15:23:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:39.572+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:39.574000+0000) 2022-09-10T15:23:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:40.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:40.206600+0000) 2022-09-10T15:23:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:40.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:40.574153+0000) 2022-09-10T15:23:41.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:41.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:41.206803+0000) 2022-09-10T15:23:41.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:41 smithi104 ceph-mon[97044]: pgmap v20939: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:41.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:41 smithi110 ceph-mon[97633]: pgmap v20939: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:41.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:41.574300+0000) 2022-09-10T15:23:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:42.205+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:42.207015+0000) 2022-09-10T15:23:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:42 smithi104 ceph-mon[97044]: pgmap v20940: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:42.925 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:42 smithi110 ceph-mon[97633]: pgmap v20940: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:42.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:42.574509+0000) 2022-09-10T15:23:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:43.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:43.207277+0000) 2022-09-10T15:23:44.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:43.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:43.574713+0000) 2022-09-10T15:23:44.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:44.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:44.207449+0000) 2022-09-10T15:23:45.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:44.573+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:44.574934+0000) 2022-09-10T15:23:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:45.206+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:45.207658+0000) 2022-09-10T15:23:45.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:45 smithi104 ceph-mon[97044]: pgmap v20941: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:23:45.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:45 smithi110 ceph-mon[97633]: pgmap v20941: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:23:46.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:45.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:45.575089+0000) 2022-09-10T15:23:46.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:46.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:46.207852+0000) 2022-09-10T15:23:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:46 smithi104 ceph-mon[97044]: pgmap v20942: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:23:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:46 smithi110 ceph-mon[97633]: pgmap v20942: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:23:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:46.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:46.575295+0000) 2022-09-10T15:23:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:47.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:47.208128+0000) 2022-09-10T15:23:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:23:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:23:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:23:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:23:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:23:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:23:47.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:23:47.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:23:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:47.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:47.575470+0000) 2022-09-10T15:23:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:48.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:48.208328+0000) 2022-09-10T15:23:48.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:48 smithi104 ceph-mon[97044]: pgmap v20943: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:48 smithi110 ceph-mon[97633]: pgmap v20943: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:48.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:48.575602+0000) 2022-09-10T15:23:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:49.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:49.208569+0000) 2022-09-10T15:23:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:49.574+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:49.575777+0000) 2022-09-10T15:23:50.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:50.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:50.208727+0000) 2022-09-10T15:23:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:50.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:50.576004+0000) 2022-09-10T15:23:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:51.207+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:51.208894+0000) 2022-09-10T15:23:51.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:51 smithi104 ceph-mon[97044]: pgmap v20944: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:51.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:51 smithi110 ceph-mon[97633]: pgmap v20944: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:23:52.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:51.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:51.576241+0000) 2022-09-10T15:23:52.489 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:52.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:52.209126+0000) 2022-09-10T15:23:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:52 smithi104 ceph-mon[97044]: pgmap v20945: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:52.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:52.575+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:52.576481+0000) 2022-09-10T15:23:52.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:23:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:23:52.486+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:23:52.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:52 smithi110 ceph-mon[97633]: pgmap v20945: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:23:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:53.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:53.209365+0000) 2022-09-10T15:23:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:53.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:53.576726+0000) 2022-09-10T15:23:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:54.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:54.209588+0000) 2022-09-10T15:23:54.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:54 smithi104 ceph-mon[97044]: pgmap v20946: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:23:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:54 smithi110 ceph-mon[97633]: pgmap v20946: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:23:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:54.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:54.576972+0000) 2022-09-10T15:23:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:55.208+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:55.209772+0000) 2022-09-10T15:23:56.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:55.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:55.577147+0000) 2022-09-10T15:23:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:56.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:56.210021+0000) 2022-09-10T15:23:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:56 smithi104 ceph-mon[97044]: pgmap v20947: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:23:56.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:56 smithi110 ceph-mon[97633]: pgmap v20947: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:23:57.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:56.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:56.577367+0000) 2022-09-10T15:23:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:57.209+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:57.210352+0000) 2022-09-10T15:23:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:57.576+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:57.577593+0000) 2022-09-10T15:23:58.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:58.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:58.210544+0000) 2022-09-10T15:23:58.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:23:58 smithi104 ceph-mon[97044]: pgmap v20948: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:23:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:23:58 smithi110 ceph-mon[97633]: pgmap v20948: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:23:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:58.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:58.577841+0000) 2022-09-10T15:23:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:23:59.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:59.210751+0000) 2022-09-10T15:24:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:23:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:23:59.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:23:59.578063+0000) 2022-09-10T15:24:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:00.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:00.210997+0000) 2022-09-10T15:24:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:00 smithi104 ceph-mon[97044]: pgmap v20949: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:00 smithi110 ceph-mon[97633]: pgmap v20949: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:00.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:00.578277+0000) 2022-09-10T15:24:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:01.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:01.211189+0000) 2022-09-10T15:24:02.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:01.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:01.578534+0000) 2022-09-10T15:24:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:02.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:02.211397+0000) 2022-09-10T15:24:02.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:02 smithi104 ceph-mon[97044]: pgmap v20950: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:24:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:02 smithi110 ceph-mon[97633]: pgmap v20950: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:24:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:02.577+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:02.578782+0000) 2022-09-10T15:24:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:03.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:03.211585+0000) 2022-09-10T15:24:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:03.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:03.579038+0000) 2022-09-10T15:24:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:04.210+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:04.211770+0000) 2022-09-10T15:24:04.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:04 smithi104 ceph-mon[97044]: pgmap v20951: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:24:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:04 smithi110 ceph-mon[97633]: pgmap v20951: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:24:05.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:04.578+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:04.579257+0000) 2022-09-10T15:24:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:05.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:05.212014+0000) 2022-09-10T15:24:06.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:05.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:05.579451+0000) 2022-09-10T15:24:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:06.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:06.212313+0000) 2022-09-10T15:24:06.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:06 smithi104 ceph-mon[97044]: pgmap v20952: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T15:24:06.914 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:06 smithi110 ceph-mon[97633]: pgmap v20952: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2022-09-10T15:24:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:06.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:06.579663+0000) 2022-09-10T15:24:07.491 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:07.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:07.212556+0000) 2022-09-10T15:24:07.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:24:07.487+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:24:07.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:07.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:07.579873+0000) 2022-09-10T15:24:08.561 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:08.212+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:08.212785+0000) 2022-09-10T15:24:08.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:08 smithi104 ceph-mon[97044]: pgmap v20953: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:24:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:08 smithi110 ceph-mon[97633]: pgmap v20953: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:24:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:08.579+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:08.580142+0000) 2022-09-10T15:24:09.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:09.211+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:09.212957+0000) 2022-09-10T15:24:09.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:09 smithi104 ceph-mon[97044]: osdmap e1390: 12 total, 12 up, 12 in 2022-09-10T15:24:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:09 smithi110 ceph-mon[97633]: osdmap e1390: 12 total, 12 up, 12 in 2022-09-10T15:24:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:09.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:09.580338+0000) 2022-09-10T15:24:10.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:10.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:10.213236+0000) 2022-09-10T15:24:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:10 smithi110 ceph-mon[97633]: pgmap v20955: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:24:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:10 smithi104 ceph-mon[97044]: pgmap v20955: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:24:11.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:10.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:10.580504+0000) 2022-09-10T15:24:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:11.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:11.213484+0000) 2022-09-10T15:24:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:11.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:11.580722+0000) 2022-09-10T15:24:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:12.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:12.213714+0000) 2022-09-10T15:24:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:12 smithi104 ceph-mon[97044]: pgmap v20956: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:24:12.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:12 smithi110 ceph-mon[97633]: pgmap v20956: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:24:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:12.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:12.580917+0000) 2022-09-10T15:24:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:13.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:13.213947+0000) 2022-09-10T15:24:13.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:13 smithi104 ceph-mon[97044]: osdmap e1391: 12 total, 12 up, 12 in 2022-09-10T15:24:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:13 smithi110 ceph-mon[97633]: osdmap e1391: 12 total, 12 up, 12 in 2022-09-10T15:24:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:13.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:13.581125+0000) 2022-09-10T15:24:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:14.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:14.214195+0000) 2022-09-10T15:24:14.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:14 smithi104 ceph-mon[97044]: pgmap v20958: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:24:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:14 smithi110 ceph-mon[97633]: pgmap v20958: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:24:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:14.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:14.581346+0000) 2022-09-10T15:24:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:15.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:15.214418+0000) 2022-09-10T15:24:16.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:15.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:15.581601+0000) 2022-09-10T15:24:16.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:16.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:16.214640+0000) 2022-09-10T15:24:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:16 smithi104 ceph-mon[97044]: pgmap v20959: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:24:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:16 smithi110 ceph-mon[97633]: pgmap v20959: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:24:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:16.580+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:16.581876+0000) 2022-09-10T15:24:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:17.213+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:17.214859+0000) 2022-09-10T15:24:18.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:17.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:17.582135+0000) 2022-09-10T15:24:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:18.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:18.215134+0000) 2022-09-10T15:24:18.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:18 smithi104 ceph-mon[97044]: pgmap v20960: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:24:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:18 smithi110 ceph-mon[97633]: pgmap v20960: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:24:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:18.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:18.582370+0000) 2022-09-10T15:24:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:19.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:19.215300+0000) 2022-09-10T15:24:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:19.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:19.582575+0000) 2022-09-10T15:24:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:20.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:20.215458+0000) 2022-09-10T15:24:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:20 smithi104 ceph-mon[97044]: pgmap v20961: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:24:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:20 smithi110 ceph-mon[97633]: pgmap v20961: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:24:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:20.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:20.582825+0000) 2022-09-10T15:24:21.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:21.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:21.215603+0000) 2022-09-10T15:24:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:21.581+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:21.582977+0000) 2022-09-10T15:24:22.502 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:22.214+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:22.215812+0000) 2022-09-10T15:24:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:22 smithi104 ceph-mon[97044]: pgmap v20962: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:24:22.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:24:22.499+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:24:22.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:22.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:22.583223+0000) 2022-09-10T15:24:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:22 smithi110 ceph-mon[97633]: pgmap v20962: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:24:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:23.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:23.216013+0000) 2022-09-10T15:24:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:23.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:23.583448+0000) 2022-09-10T15:24:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:24.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:24.216227+0000) 2022-09-10T15:24:24.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:24 smithi104 ceph-mon[97044]: pgmap v20963: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:24:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:24 smithi110 ceph-mon[97633]: pgmap v20963: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:24:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:24.582+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:24.583679+0000) 2022-09-10T15:24:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:25.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:25.216483+0000) 2022-09-10T15:24:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:25.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:25.583915+0000) 2022-09-10T15:24:26.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:26.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:26.216690+0000) 2022-09-10T15:24:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:26 smithi104 ceph-mon[97044]: pgmap v20964: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:24:26.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:26 smithi110 ceph-mon[97633]: pgmap v20964: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:24:27.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:26.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:26.584090+0000) 2022-09-10T15:24:27.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:27.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:27.216855+0000) 2022-09-10T15:24:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:27.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:27.584338+0000) 2022-09-10T15:24:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:28.215+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:28.217038+0000) 2022-09-10T15:24:28.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:28 smithi104 ceph-mon[97044]: pgmap v20965: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:28 smithi110 ceph-mon[97633]: pgmap v20965: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:28.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:28.584578+0000) 2022-09-10T15:24:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:29.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:29.217302+0000) 2022-09-10T15:24:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:29.583+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:29.584779+0000) 2022-09-10T15:24:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:30.216+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:30.217585+0000) 2022-09-10T15:24:30.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:30 smithi104 ceph-mon[97044]: pgmap v20966: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:30 smithi110 ceph-mon[97633]: pgmap v20966: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:30.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:30.585021+0000) 2022-09-10T15:24:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:31.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:31.217870+0000) 2022-09-10T15:24:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:31.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:31.585267+0000) 2022-09-10T15:24:32.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:32.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:32.218125+0000) 2022-09-10T15:24:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:32 smithi104 ceph-mon[97044]: pgmap v20967: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:32 smithi110 ceph-mon[97633]: pgmap v20967: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:32.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:32.585460+0000) 2022-09-10T15:24:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:33.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:33.218287+0000) 2022-09-10T15:24:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:33.584+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:33.585659+0000) 2022-09-10T15:24:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:34.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:34.218417+0000) 2022-09-10T15:24:34.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:34 smithi104 ceph-mon[97044]: pgmap v20968: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:24:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:34 smithi110 ceph-mon[97633]: pgmap v20968: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:24:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:34.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:34.585863+0000) 2022-09-10T15:24:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:35.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:35.218672+0000) 2022-09-10T15:24:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:35.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:35.586125+0000) 2022-09-10T15:24:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:36.217+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:36.218904+0000) 2022-09-10T15:24:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:36 smithi104 ceph-mon[97044]: pgmap v20969: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:24:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:36 smithi110 ceph-mon[97633]: pgmap v20969: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:24:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:36.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:36.586400+0000) 2022-09-10T15:24:37.505 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:37.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:37.219191+0000) 2022-09-10T15:24:37.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:24:37.502+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:24:37.807 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:37.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:37.586570+0000) 2022-09-10T15:24:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:38.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:38.219399+0000) 2022-09-10T15:24:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:38 smithi110 ceph-mon[97633]: pgmap v20970: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:39.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:38 smithi104 ceph-mon[97044]: pgmap v20970: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:38.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:38.586785+0000) 2022-09-10T15:24:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:39.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:39.219533+0000) 2022-09-10T15:24:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:39.585+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:39.587037+0000) 2022-09-10T15:24:40.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:40.218+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:40.219687+0000) 2022-09-10T15:24:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:40 smithi104 ceph-mon[97044]: pgmap v20971: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:40 smithi110 ceph-mon[97633]: pgmap v20971: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:41.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:40.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:40.587229+0000) 2022-09-10T15:24:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:41.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:41.219872+0000) 2022-09-10T15:24:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:41.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:41.587437+0000) 2022-09-10T15:24:42.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:42.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:42.220008+0000) 2022-09-10T15:24:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:42 smithi104 ceph-mon[97044]: pgmap v20972: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:42.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:42 smithi110 ceph-mon[97633]: pgmap v20972: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:42.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:42.587620+0000) 2022-09-10T15:24:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:43.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:43.220240+0000) 2022-09-10T15:24:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:43.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:43.587831+0000) 2022-09-10T15:24:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:44.219+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:44.220498+0000) 2022-09-10T15:24:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:44 smithi104 ceph-mon[97044]: pgmap v20973: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:24:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:44 smithi110 ceph-mon[97633]: pgmap v20973: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:24:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:44.586+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:44.588023+0000) 2022-09-10T15:24:45.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:45.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:45.220739+0000) 2022-09-10T15:24:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:45.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:45.588205+0000) 2022-09-10T15:24:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:46.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:46.220947+0000) 2022-09-10T15:24:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:46 smithi104 ceph-mon[97044]: pgmap v20974: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:24:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:46 smithi110 ceph-mon[97633]: pgmap v20974: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:24:47.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:46.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:46.588424+0000) 2022-09-10T15:24:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:47.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:47.221244+0000) 2022-09-10T15:24:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:24:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:24:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:24:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:24:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:24:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:24:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:24:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:24:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:47.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:47.588668+0000) 2022-09-10T15:24:48.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:48.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:48.221456+0000) 2022-09-10T15:24:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:48 smithi110 ceph-mon[97633]: pgmap v20975: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:48 smithi104 ceph-mon[97044]: pgmap v20975: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:48.587+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:48.588925+0000) 2022-09-10T15:24:49.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:49.220+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:49.221679+0000) 2022-09-10T15:24:50.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:49.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:49.589218+0000) 2022-09-10T15:24:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:50.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:50.221922+0000) 2022-09-10T15:24:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:50 smithi104 ceph-mon[97044]: pgmap v20976: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:50 smithi110 ceph-mon[97633]: pgmap v20976: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:24:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:50.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:50.589445+0000) 2022-09-10T15:24:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:51.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:51.222153+0000) 2022-09-10T15:24:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:51.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:51.589713+0000) 2022-09-10T15:24:52.519 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:52.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:52.222363+0000) 2022-09-10T15:24:52.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:52 smithi104 ceph-mon[97044]: pgmap v20977: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:52.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:52.588+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:52.589960+0000) 2022-09-10T15:24:52.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:24:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:24:52.516+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:24:52.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:52 smithi110 ceph-mon[97633]: pgmap v20977: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:53.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:53.222645+0000) 2022-09-10T15:24:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:53.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:53.590213+0000) 2022-09-10T15:24:54.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:54.221+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:54.222777+0000) 2022-09-10T15:24:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:54 smithi104 ceph-mon[97044]: pgmap v20978: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:24:54.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:54 smithi110 ceph-mon[97633]: pgmap v20978: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:24:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:54.589+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:54.590373+0000) 2022-09-10T15:24:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:55.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:55.222970+0000) 2022-09-10T15:24:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:55.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:55.590565+0000) 2022-09-10T15:24:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:56.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:56.223215+0000) 2022-09-10T15:24:56.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:56 smithi104 ceph-mon[97044]: pgmap v20979: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:24:56.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:56 smithi110 ceph-mon[97633]: pgmap v20979: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:24:57.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:56.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:56.590792+0000) 2022-09-10T15:24:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:57.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:57.223361+0000) 2022-09-10T15:24:58.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:57.590+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:57.591030+0000) 2022-09-10T15:24:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:58.222+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:58.223521+0000) 2022-09-10T15:24:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:24:58 smithi110 ceph-mon[97633]: pgmap v20980: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:24:58 smithi104 ceph-mon[97044]: pgmap v20980: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:24:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:58.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:58.591231+0000) 2022-09-10T15:24:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:24:59.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:59.223690+0000) 2022-09-10T15:25:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:24:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:24:59.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:24:59.591369+0000) 2022-09-10T15:25:00.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:00.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:00.223780+0000) 2022-09-10T15:25:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:00 smithi104 ceph-mon[97044]: pgmap v20981: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:00 smithi110 ceph-mon[97633]: pgmap v20981: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:00.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:00.591593+0000) 2022-09-10T15:25:01.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:01.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:01.224001+0000) 2022-09-10T15:25:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:01.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:01.591784+0000) 2022-09-10T15:25:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:02.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:02.224206+0000) 2022-09-10T15:25:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:02 smithi104 ceph-mon[97044]: pgmap v20982: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:02.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:02 smithi110 ceph-mon[97633]: pgmap v20982: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:02.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:02.592003+0000) 2022-09-10T15:25:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:03.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:03.224366+0000) 2022-09-10T15:25:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:03.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:03.592260+0000) 2022-09-10T15:25:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:04.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:04.224569+0000) 2022-09-10T15:25:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:04 smithi104 ceph-mon[97044]: pgmap v20983: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:25:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:04 smithi110 ceph-mon[97633]: pgmap v20983: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:25:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:04.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:04.592502+0000) 2022-09-10T15:25:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:05.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:05.224780+0000) 2022-09-10T15:25:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:05.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:05.592699+0000) 2022-09-10T15:25:06.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:06.223+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:06.225015+0000) 2022-09-10T15:25:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:06 smithi104 ceph-mon[97044]: pgmap v20984: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:25:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:06 smithi110 ceph-mon[97633]: pgmap v20984: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:25:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:06.591+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:06.592979+0000) 2022-09-10T15:25:07.537 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:07.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:07.225270+0000) 2022-09-10T15:25:07.805 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:07.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:07.593298+0000) 2022-09-10T15:25:07.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:25:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:25:07.531+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:25:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:08.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:08.225500+0000) 2022-09-10T15:25:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:08 smithi110 ceph-mon[97633]: pgmap v20985: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:25:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:08 smithi104 ceph-mon[97044]: pgmap v20985: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:25:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:08.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:08.593505+0000) 2022-09-10T15:25:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:09.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:09.225686+0000) 2022-09-10T15:25:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:09 smithi110 ceph-mon[97633]: osdmap e1392: 12 total, 12 up, 12 in 2022-09-10T15:25:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:09 smithi104 ceph-mon[97044]: osdmap e1392: 12 total, 12 up, 12 in 2022-09-10T15:25:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:09.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:09.593753+0000) 2022-09-10T15:25:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:10.224+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:10.225930+0000) 2022-09-10T15:25:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:10 smithi110 ceph-mon[97633]: pgmap v20987: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:25:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:10 smithi104 ceph-mon[97044]: pgmap v20987: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:25:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:10.592+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:10.593981+0000) 2022-09-10T15:25:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:11.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:11.226156+0000) 2022-09-10T15:25:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:11.593+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:11.594231+0000) 2022-09-10T15:25:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:12.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:12.226366+0000) 2022-09-10T15:25:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:12 smithi104 ceph-mon[97044]: pgmap v20988: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:25:12.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:12 smithi110 ceph-mon[97633]: pgmap v20988: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:25:13.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:12.594+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:12.595924+0000) 2022-09-10T15:25:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:13.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:13.226586+0000) 2022-09-10T15:25:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:13 smithi110 ceph-mon[97633]: osdmap e1393: 12 total, 12 up, 12 in 2022-09-10T15:25:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:13 smithi104 ceph-mon[97044]: osdmap e1393: 12 total, 12 up, 12 in 2022-09-10T15:25:14.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:13.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:13.596228+0000) 2022-09-10T15:25:14.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:14.225+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:14.226819+0000) 2022-09-10T15:25:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:14 smithi110 ceph-mon[97633]: pgmap v20990: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:25:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:14 smithi104 ceph-mon[97044]: pgmap v20990: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:25:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:14.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:14.596486+0000) 2022-09-10T15:25:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:15.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:15.227017+0000) 2022-09-10T15:25:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:15.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:15.596704+0000) 2022-09-10T15:25:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:16.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:16.227266+0000) 2022-09-10T15:25:16.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:16 smithi104 ceph-mon[97044]: pgmap v20991: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:25:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:16 smithi110 ceph-mon[97633]: pgmap v20991: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:25:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:16.595+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:16.596981+0000) 2022-09-10T15:25:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:17.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:17.227498+0000) 2022-09-10T15:25:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:17.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:17.597194+0000) 2022-09-10T15:25:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:18.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:18.227733+0000) 2022-09-10T15:25:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:18 smithi110 ceph-mon[97633]: pgmap v20992: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:25:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:18 smithi104 ceph-mon[97044]: pgmap v20992: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:25:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:18.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:18.597343+0000) 2022-09-10T15:25:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:19.226+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:19.227917+0000) 2022-09-10T15:25:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:19.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:19.597587+0000) 2022-09-10T15:25:20.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:20.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:20.228143+0000) 2022-09-10T15:25:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:20 smithi104 ceph-mon[97044]: pgmap v20993: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:25:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:20 smithi110 ceph-mon[97633]: pgmap v20993: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:25:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:20.596+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:20.597764+0000) 2022-09-10T15:25:21.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:21.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:21.228323+0000) 2022-09-10T15:25:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:21.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:21.598053+0000) 2022-09-10T15:25:22.546 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:22.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:22.228514+0000) 2022-09-10T15:25:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:22 smithi104 ceph-mon[97044]: pgmap v20994: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:25:22.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:25:22.543+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:25:22.806 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:22.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:22.598262+0000) 2022-09-10T15:25:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:22 smithi110 ceph-mon[97633]: pgmap v20994: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:25:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:23.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:23.228754+0000) 2022-09-10T15:25:24.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:23.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:23.598454+0000) 2022-09-10T15:25:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:24.227+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:24.228940+0000) 2022-09-10T15:25:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:24 smithi104 ceph-mon[97044]: pgmap v20995: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:25:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:24 smithi110 ceph-mon[97633]: pgmap v20995: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:25:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:24.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:24.598697+0000) 2022-09-10T15:25:25.562 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:25.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:25.229121+0000) 2022-09-10T15:25:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:25.597+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:25.598912+0000) 2022-09-10T15:25:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:26.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:26.229364+0000) 2022-09-10T15:25:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:26 smithi104 ceph-mon[97044]: pgmap v20996: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:25:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:26 smithi110 ceph-mon[97633]: pgmap v20996: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:25:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:26.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:26.599188+0000) 2022-09-10T15:25:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:27.228+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:27.229645+0000) 2022-09-10T15:25:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:27.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:27.599406+0000) 2022-09-10T15:25:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:28.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:28.229859+0000) 2022-09-10T15:25:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:28 smithi110 ceph-mon[97633]: pgmap v20997: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:29.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:28 smithi104 ceph-mon[97044]: pgmap v20997: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:28.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:28.599589+0000) 2022-09-10T15:25:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:29.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:29.230106+0000) 2022-09-10T15:25:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:29.598+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:29.599846+0000) 2022-09-10T15:25:30.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:30.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:30.230314+0000) 2022-09-10T15:25:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:30 smithi104 ceph-mon[97044]: pgmap v20998: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:30 smithi110 ceph-mon[97633]: pgmap v20998: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:30.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:30.600146+0000) 2022-09-10T15:25:31.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:31.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:31.230559+0000) 2022-09-10T15:25:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:31.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:31.600408+0000) 2022-09-10T15:25:32.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:32.229+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:32.230864+0000) 2022-09-10T15:25:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:32 smithi104 ceph-mon[97044]: pgmap v20999: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:32 smithi110 ceph-mon[97633]: pgmap v20999: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:32.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:32.600594+0000) 2022-09-10T15:25:33.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:33.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:33.231057+0000) 2022-09-10T15:25:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:33.599+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:33.600862+0000) 2022-09-10T15:25:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:34.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:34.231293+0000) 2022-09-10T15:25:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:34 smithi104 ceph-mon[97044]: pgmap v21000: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:25:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:34 smithi110 ceph-mon[97633]: pgmap v21000: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:25:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:34.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:34.601172+0000) 2022-09-10T15:25:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:35.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:35.231511+0000) 2022-09-10T15:25:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:35.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:35.601437+0000) 2022-09-10T15:25:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:36.230+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:36.231652+0000) 2022-09-10T15:25:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:36 smithi104 ceph-mon[97044]: pgmap v21001: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:25:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:36 smithi110 ceph-mon[97633]: pgmap v21001: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:25:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:36.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:36.601698+0000) 2022-09-10T15:25:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:37.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:37.231911+0000) 2022-09-10T15:25:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:25:37.553+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:25:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:37.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:37.601897+0000) 2022-09-10T15:25:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:38.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:38.232143+0000) 2022-09-10T15:25:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:38 smithi110 ceph-mon[97633]: pgmap v21002: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:38 smithi104 ceph-mon[97044]: pgmap v21002: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:39.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:38.600+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:38.602064+0000) 2022-09-10T15:25:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:39.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:39.232324+0000) 2022-09-10T15:25:40.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:39.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:39.602238+0000) 2022-09-10T15:25:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:40.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:40.232460+0000) 2022-09-10T15:25:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:40 smithi104 ceph-mon[97044]: pgmap v21003: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:40 smithi110 ceph-mon[97633]: pgmap v21003: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:40.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:40.602469+0000) 2022-09-10T15:25:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:41.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:41.232636+0000) 2022-09-10T15:25:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:41.601+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:41.602763+0000) 2022-09-10T15:25:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:42.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:42.232905+0000) 2022-09-10T15:25:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:42 smithi104 ceph-mon[97044]: pgmap v21004: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:42.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:42 smithi110 ceph-mon[97633]: pgmap v21004: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:42.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:42.603045+0000) 2022-09-10T15:25:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:43.231+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:43.233061+0000) 2022-09-10T15:25:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:43.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:43.603302+0000) 2022-09-10T15:25:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:44.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:44.233249+0000) 2022-09-10T15:25:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:44 smithi104 ceph-mon[97044]: pgmap v21005: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:25:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:44 smithi110 ceph-mon[97633]: pgmap v21005: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:25:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:44.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:44.603533+0000) 2022-09-10T15:25:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:45.232+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:45.233388+0000) 2022-09-10T15:25:46.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:45.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:45.603784+0000) 2022-09-10T15:25:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:46.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:46.233549+0000) 2022-09-10T15:25:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:46 smithi104 ceph-mon[97044]: pgmap v21006: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:25:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:46 smithi110 ceph-mon[97633]: pgmap v21006: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:25:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:46.602+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:46.604073+0000) 2022-09-10T15:25:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:47.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:47.233797+0000) 2022-09-10T15:25:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:25:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:25:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:25:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:25:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:25:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:25:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:25:47.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:25:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:47.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:47.604350+0000) 2022-09-10T15:25:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:48.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:48.234028+0000) 2022-09-10T15:25:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:48 smithi110 ceph-mon[97633]: pgmap v21007: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:49.050 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:48 smithi104 ceph-mon[97044]: pgmap v21007: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:49.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:48.603+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:48.604590+0000) 2022-09-10T15:25:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:49.233+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:49.234191+0000) 2022-09-10T15:25:50.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:49.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:49.604785+0000) 2022-09-10T15:25:50.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:50.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:50.234479+0000) 2022-09-10T15:25:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:50 smithi104 ceph-mon[97044]: pgmap v21008: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:50 smithi110 ceph-mon[97633]: pgmap v21008: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:25:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:50.604+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:50.604976+0000) 2022-09-10T15:25:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:51.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:51.234757+0000) 2022-09-10T15:25:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:51.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:51.605277+0000) 2022-09-10T15:25:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:52.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:52.234925+0000) 2022-09-10T15:25:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:52 smithi104 ceph-mon[97044]: pgmap v21009: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:52.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:52 smithi110 ceph-mon[97633]: pgmap v21009: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:52.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:52.605531+0000) 2022-09-10T15:25:53.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:25:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:25:52.566+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:25:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:53.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:53.235139+0000) 2022-09-10T15:25:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:53.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:53.605751+0000) 2022-09-10T15:25:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:54.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:54.235299+0000) 2022-09-10T15:25:54.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:54 smithi104 ceph-mon[97044]: pgmap v21010: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:25:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:54 smithi110 ceph-mon[97633]: pgmap v21010: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:25:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:54.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:54.605995+0000) 2022-09-10T15:25:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:55.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:55.235526+0000) 2022-09-10T15:25:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:55.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:55.606244+0000) 2022-09-10T15:25:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:56.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:56.235749+0000) 2022-09-10T15:25:56.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:56 smithi104 ceph-mon[97044]: pgmap v21011: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:25:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:56 smithi110 ceph-mon[97633]: pgmap v21011: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:25:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:56.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:56.606431+0000) 2022-09-10T15:25:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:57.234+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:57.235957+0000) 2022-09-10T15:25:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:57.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:57.606648+0000) 2022-09-10T15:25:58.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:58.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:58.236200+0000) 2022-09-10T15:25:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:25:58 smithi110 ceph-mon[97633]: pgmap v21012: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:25:58 smithi104 ceph-mon[97044]: pgmap v21012: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:25:59.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:58.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:58.606858+0000) 2022-09-10T15:25:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:25:59.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:59.236379+0000) 2022-09-10T15:26:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:25:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:25:59.605+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:25:59.607061+0000) 2022-09-10T15:26:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:00.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:00.236570+0000) 2022-09-10T15:26:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:00 smithi104 ceph-mon[97044]: pgmap v21013: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:00 smithi110 ceph-mon[97633]: pgmap v21013: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:00.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:00.607318+0000) 2022-09-10T15:26:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:01.235+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:01.236770+0000) 2022-09-10T15:26:02.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:01.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:01.607489+0000) 2022-09-10T15:26:02.559 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:02.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:02.237012+0000) 2022-09-10T15:26:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:02 smithi104 ceph-mon[97044]: pgmap v21014: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:02 smithi110 ceph-mon[97633]: pgmap v21014: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:02.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:02.607738+0000) 2022-09-10T15:26:03.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:03.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:03.237245+0000) 2022-09-10T15:26:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:03.606+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:03.607984+0000) 2022-09-10T15:26:04.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:04.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:04.237437+0000) 2022-09-10T15:26:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:04 smithi104 ceph-mon[97044]: pgmap v21015: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:26:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:04 smithi110 ceph-mon[97633]: pgmap v21015: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:26:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:04.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:04.608167+0000) 2022-09-10T15:26:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:05.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:05.237688+0000) 2022-09-10T15:26:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:05.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:05.608377+0000) 2022-09-10T15:26:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:06.236+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:06.237874+0000) 2022-09-10T15:26:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:06 smithi104 ceph-mon[97044]: pgmap v21016: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:26:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:06 smithi110 ceph-mon[97633]: pgmap v21016: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:26:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:06.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:06.608548+0000) 2022-09-10T15:26:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:07.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:07.238056+0000) 2022-09-10T15:26:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:07.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:07.608742+0000) 2022-09-10T15:26:08.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:26:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:26:07.570+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:26:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:08.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:08.238285+0000) 2022-09-10T15:26:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:08 smithi110 ceph-mon[97633]: pgmap v21017: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:26:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:08 smithi104 ceph-mon[97044]: pgmap v21017: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:26:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:08.607+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:08.608985+0000) 2022-09-10T15:26:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:09.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:09.238467+0000) 2022-09-10T15:26:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:09 smithi110 ceph-mon[97633]: osdmap e1394: 12 total, 12 up, 12 in 2022-09-10T15:26:10.054 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:09 smithi104 ceph-mon[97044]: osdmap e1394: 12 total, 12 up, 12 in 2022-09-10T15:26:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:09.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:09.609205+0000) 2022-09-10T15:26:10.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:10.237+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:10.238659+0000) 2022-09-10T15:26:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:10 smithi110 ceph-mon[97633]: pgmap v21019: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:26:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:10 smithi104 ceph-mon[97044]: pgmap v21019: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:26:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:10.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:10.609417+0000) 2022-09-10T15:26:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:11.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:11.238883+0000) 2022-09-10T15:26:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:11.608+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:11.609668+0000) 2022-09-10T15:26:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:12.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:12.239198+0000) 2022-09-10T15:26:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:12 smithi104 ceph-mon[97044]: pgmap v21020: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:26:12.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:12 smithi110 ceph-mon[97633]: pgmap v21020: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:26:13.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:12.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:12.609888+0000) 2022-09-10T15:26:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:13.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:13.239366+0000) 2022-09-10T15:26:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:13 smithi110 ceph-mon[97633]: osdmap e1395: 12 total, 12 up, 12 in 2022-09-10T15:26:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:13 smithi104 ceph-mon[97044]: osdmap e1395: 12 total, 12 up, 12 in 2022-09-10T15:26:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:13.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:13.610161+0000) 2022-09-10T15:26:14.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:14.238+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:14.239615+0000) 2022-09-10T15:26:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:14 smithi110 ceph-mon[97633]: pgmap v21022: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:26:15.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:14 smithi104 ceph-mon[97044]: pgmap v21022: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:26:15.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:14.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:14.610407+0000) 2022-09-10T15:26:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:15.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:15.239853+0000) 2022-09-10T15:26:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:15.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:15.610599+0000) 2022-09-10T15:26:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:16.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:16.240083+0000) 2022-09-10T15:26:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:16 smithi104 ceph-mon[97044]: pgmap v21023: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:26:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:16 smithi110 ceph-mon[97633]: pgmap v21023: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:26:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:16.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:16.610804+0000) 2022-09-10T15:26:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:17.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:17.240318+0000) 2022-09-10T15:26:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:17.609+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:17.611068+0000) 2022-09-10T15:26:18.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:18.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:18.240517+0000) 2022-09-10T15:26:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:18 smithi110 ceph-mon[97633]: pgmap v21024: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:26:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:18 smithi104 ceph-mon[97044]: pgmap v21024: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:26:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:18.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:18.611281+0000) 2022-09-10T15:26:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:19.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:19.240760+0000) 2022-09-10T15:26:20.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:19.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:19.611495+0000) 2022-09-10T15:26:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:20.239+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:20.240940+0000) 2022-09-10T15:26:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:20 smithi104 ceph-mon[97044]: pgmap v21025: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:26:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:20 smithi110 ceph-mon[97633]: pgmap v21025: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:26:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:20.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:20.611723+0000) 2022-09-10T15:26:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:21.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:21.241190+0000) 2022-09-10T15:26:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:21.610+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:21.611914+0000) 2022-09-10T15:26:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:22.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:22.241381+0000) 2022-09-10T15:26:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:22 smithi104 ceph-mon[97044]: pgmap v21026: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:26:22.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:22 smithi110 ceph-mon[97633]: pgmap v21026: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:26:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:26:22.581+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:26:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:22.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:22.612167+0000) 2022-09-10T15:26:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:23.240+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:23.241599+0000) 2022-09-10T15:26:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:23.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:23.612319+0000) 2022-09-10T15:26:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:24.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:24.241809+0000) 2022-09-10T15:26:24.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:24 smithi104 ceph-mon[97044]: pgmap v21027: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:26:25.423 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:24.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:24.612503+0000) 2022-09-10T15:26:25.423 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:24 smithi110 ceph-mon[97633]: pgmap v21027: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:26:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:25.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:25.242018+0000) 2022-09-10T15:26:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:25.611+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:25.612690+0000) 2022-09-10T15:26:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:26.241+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:26.242251+0000) 2022-09-10T15:26:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:26 smithi104 ceph-mon[97044]: pgmap v21028: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:26:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:26 smithi110 ceph-mon[97633]: pgmap v21028: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:26:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:26.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:26.612928+0000) 2022-09-10T15:26:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:27.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:27.242516+0000) 2022-09-10T15:26:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:27.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:27.613120+0000) 2022-09-10T15:26:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:28.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:28.242712+0000) 2022-09-10T15:26:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:28 smithi110 ceph-mon[97633]: pgmap v21029: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:29.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:28 smithi104 ceph-mon[97044]: pgmap v21029: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:28.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:28.613321+0000) 2022-09-10T15:26:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:29.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:29.242945+0000) 2022-09-10T15:26:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:29.612+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:29.613547+0000) 2022-09-10T15:26:30.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:30.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:30.243176+0000) 2022-09-10T15:26:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:30 smithi104 ceph-mon[97044]: pgmap v21030: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:30 smithi110 ceph-mon[97633]: pgmap v21030: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:31.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:30.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:30.613814+0000) 2022-09-10T15:26:31.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:31.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:31.243403+0000) 2022-09-10T15:26:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:31.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:31.614063+0000) 2022-09-10T15:26:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:32.242+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:32.243567+0000) 2022-09-10T15:26:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:32 smithi104 ceph-mon[97044]: pgmap v21031: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:26:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:32 smithi110 ceph-mon[97633]: pgmap v21031: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:26:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:32.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:32.614342+0000) 2022-09-10T15:26:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:33.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:33.243808+0000) 2022-09-10T15:26:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:33.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:33.614555+0000) 2022-09-10T15:26:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:34.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:34.244062+0000) 2022-09-10T15:26:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:34 smithi104 ceph-mon[97044]: pgmap v21032: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T15:26:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:34 smithi110 ceph-mon[97633]: pgmap v21032: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2022-09-10T15:26:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:34.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:34.614704+0000) 2022-09-10T15:26:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:35.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:35.244265+0000) 2022-09-10T15:26:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:35.613+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:35.614914+0000) 2022-09-10T15:26:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:36.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:36.244470+0000) 2022-09-10T15:26:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:36 smithi104 ceph-mon[97044]: pgmap v21033: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:26:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:36 smithi110 ceph-mon[97633]: pgmap v21033: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:26:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:36.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:36.615155+0000) 2022-09-10T15:26:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:37.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:37.244699+0000) 2022-09-10T15:26:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:37.614+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:37.615295+0000) 2022-09-10T15:26:38.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:26:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:26:37.594+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:26:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:38.243+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:38.244899+0000) 2022-09-10T15:26:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:38 smithi110 ceph-mon[97633]: pgmap v21034: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:26:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:38 smithi104 ceph-mon[97044]: pgmap v21034: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:26:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:38.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:38.615513+0000) 2022-09-10T15:26:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:39.244+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:39.245128+0000) 2022-09-10T15:26:40.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:39.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:39.615721+0000) 2022-09-10T15:26:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:40.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:40.245302+0000) 2022-09-10T15:26:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:40 smithi104 ceph-mon[97044]: pgmap v21035: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:40 smithi110 ceph-mon[97633]: pgmap v21035: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:40.615+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:40.615940+0000) 2022-09-10T15:26:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:41.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:41.245554+0000) 2022-09-10T15:26:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:41.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:41.616195+0000) 2022-09-10T15:26:42.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:42.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:42.245764+0000) 2022-09-10T15:26:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:42 smithi104 ceph-mon[97044]: pgmap v21036: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:42.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:42 smithi110 ceph-mon[97633]: pgmap v21036: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:42.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:42.616419+0000) 2022-09-10T15:26:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:43.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:43.245958+0000) 2022-09-10T15:26:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:43.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:43.616649+0000) 2022-09-10T15:26:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:44.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:44.246179+0000) 2022-09-10T15:26:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:44 smithi104 ceph-mon[97044]: pgmap v21037: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:26:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:44 smithi110 ceph-mon[97633]: pgmap v21037: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:26:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:44.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:44.616896+0000) 2022-09-10T15:26:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:45.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:45.246418+0000) 2022-09-10T15:26:46.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:45.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:45.617151+0000) 2022-09-10T15:26:46.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:46.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:46.246606+0000) 2022-09-10T15:26:46.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:46 smithi104 ceph-mon[97044]: pgmap v21038: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:26:46.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:46 smithi110 ceph-mon[97633]: pgmap v21038: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:26:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:46.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:46.617375+0000) 2022-09-10T15:26:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:47.245+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:47.246899+0000) 2022-09-10T15:26:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:26:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:26:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:26:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:26:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:26:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:26:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:26:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:26:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:47.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:47.617566+0000) 2022-09-10T15:26:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:48.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:48.247178+0000) 2022-09-10T15:26:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:48 smithi110 ceph-mon[97633]: pgmap v21039: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:48 smithi104 ceph-mon[97044]: pgmap v21039: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:48.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:48.617753+0000) 2022-09-10T15:26:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:49.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:49.247406+0000) 2022-09-10T15:26:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:49.616+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:49.617972+0000) 2022-09-10T15:26:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:50.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:50.247652+0000) 2022-09-10T15:26:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:50 smithi104 ceph-mon[97044]: pgmap v21040: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:50 smithi110 ceph-mon[97633]: pgmap v21040: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:26:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:50.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:50.618245+0000) 2022-09-10T15:26:51.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:51.246+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:51.247902+0000) 2022-09-10T15:26:52.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:51.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:51.618507+0000) 2022-09-10T15:26:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:52.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:52.248168+0000) 2022-09-10T15:26:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:52 smithi104 ceph-mon[97044]: pgmap v21041: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:52.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:52 smithi110 ceph-mon[97633]: pgmap v21041: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:26:52.610+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:26:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:52.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:52.618703+0000) 2022-09-10T15:26:53.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:53.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:53.248395+0000) 2022-09-10T15:26:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:53.617+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:53.618939+0000) 2022-09-10T15:26:54.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:54.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:54.248636+0000) 2022-09-10T15:26:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:54 smithi104 ceph-mon[97044]: pgmap v21042: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:26:54.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:54 smithi110 ceph-mon[97633]: pgmap v21042: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:26:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:54.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:54.619236+0000) 2022-09-10T15:26:55.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:55.247+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:55.248912+0000) 2022-09-10T15:26:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:55.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:55.619429+0000) 2022-09-10T15:26:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:56.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:56.249188+0000) 2022-09-10T15:26:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:56 smithi104 ceph-mon[97044]: pgmap v21043: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:26:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:56 smithi110 ceph-mon[97633]: pgmap v21043: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:26:57.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:56.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:56.619638+0000) 2022-09-10T15:26:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:57.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:57.249420+0000) 2022-09-10T15:26:58.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:57.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:57.619810+0000) 2022-09-10T15:26:58.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:58.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:58.249568+0000) 2022-09-10T15:26:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:26:58 smithi110 ceph-mon[97633]: pgmap v21044: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:59.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:26:58 smithi104 ceph-mon[97044]: pgmap v21044: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:26:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:58.618+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:58.620034+0000) 2022-09-10T15:26:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:26:59.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:59.249794+0000) 2022-09-10T15:27:00.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:26:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:26:59.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:26:59.620264+0000) 2022-09-10T15:27:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:00.248+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:00.250025+0000) 2022-09-10T15:27:00.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:00 smithi104 ceph-mon[97044]: pgmap v21045: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:00.913 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:00 smithi110 ceph-mon[97633]: pgmap v21045: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:00.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:00.620412+0000) 2022-09-10T15:27:01.565 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:01.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:01.250268+0000) 2022-09-10T15:27:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:01.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:01.620640+0000) 2022-09-10T15:27:02.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:02.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:02.250462+0000) 2022-09-10T15:27:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:02 smithi104 ceph-mon[97044]: pgmap v21046: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:02 smithi110 ceph-mon[97633]: pgmap v21046: 129 pgs: 129 active+clean; 10 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:02.619+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:02.620840+0000) 2022-09-10T15:27:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:03.249+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:03.250630+0000) 2022-09-10T15:27:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:03.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:03.621069+0000) 2022-09-10T15:27:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:04.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:04.250848+0000) 2022-09-10T15:27:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:04 smithi104 ceph-mon[97044]: pgmap v21047: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:27:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:04 smithi110 ceph-mon[97633]: pgmap v21047: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:27:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:04.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:04.621342+0000) 2022-09-10T15:27:05.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:05.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:05.251073+0000) 2022-09-10T15:27:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:05.620+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:05.621567+0000) 2022-09-10T15:27:06.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:06.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:06.251359+0000) 2022-09-10T15:27:06.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:06 smithi104 ceph-mon[97044]: pgmap v21048: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:27:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:06 smithi110 ceph-mon[97633]: pgmap v21048: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:27:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:06.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:06.621776+0000) 2022-09-10T15:27:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:07.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:07.251615+0000) 2022-09-10T15:27:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:07.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:07.621973+0000) 2022-09-10T15:27:08.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:27:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:27:07.621+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:27:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:08.250+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:08.251815+0000) 2022-09-10T15:27:08.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:08 smithi110 ceph-mon[97633]: pgmap v21049: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:27:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:08 smithi104 ceph-mon[97044]: pgmap v21049: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:27:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:08.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:08.622148+0000) 2022-09-10T15:27:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:09.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:09.252090+0000) 2022-09-10T15:27:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:09 smithi110 ceph-mon[97633]: osdmap e1396: 12 total, 12 up, 12 in 2022-09-10T15:27:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:09 smithi104 ceph-mon[97044]: osdmap e1396: 12 total, 12 up, 12 in 2022-09-10T15:27:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:09.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:09.622317+0000) 2022-09-10T15:27:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:10.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:10.252337+0000) 2022-09-10T15:27:10.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:10 smithi110 ceph-mon[97633]: pgmap v21051: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:27:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:10 smithi104 ceph-mon[97044]: pgmap v21051: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:27:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:10.621+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:10.622560+0000) 2022-09-10T15:27:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:11.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:11.252580+0000) 2022-09-10T15:27:12.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:11.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:11.622783+0000) 2022-09-10T15:27:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:12.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:12.252815+0000) 2022-09-10T15:27:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:12 smithi104 ceph-mon[97044]: pgmap v21052: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:27:12.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:12 smithi110 ceph-mon[97633]: pgmap v21052: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:27:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:12.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:12.622993+0000) 2022-09-10T15:27:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:13.251+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:13.253066+0000) 2022-09-10T15:27:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:13 smithi110 ceph-mon[97633]: osdmap e1397: 12 total, 12 up, 12 in 2022-09-10T15:27:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:13 smithi104 ceph-mon[97044]: osdmap e1397: 12 total, 12 up, 12 in 2022-09-10T15:27:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:13.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:13.623179+0000) 2022-09-10T15:27:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:14.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:14.253285+0000) 2022-09-10T15:27:14.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:14 smithi110 ceph-mon[97633]: pgmap v21054: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:27:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:14 smithi104 ceph-mon[97044]: pgmap v21054: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:27:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:14.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:14.623359+0000) 2022-09-10T15:27:15.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:15.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:15.253513+0000) 2022-09-10T15:27:16.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:15.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:15.623579+0000) 2022-09-10T15:27:16.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:16.252+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:16.253756+0000) 2022-09-10T15:27:16.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:16 smithi104 ceph-mon[97044]: pgmap v21055: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:27:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:16 smithi110 ceph-mon[97633]: pgmap v21055: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:27:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:16.622+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:16.623767+0000) 2022-09-10T15:27:17.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:17.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:17.253948+0000) 2022-09-10T15:27:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:17.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:17.623979+0000) 2022-09-10T15:27:18.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:18.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:18.254199+0000) 2022-09-10T15:27:18.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:18 smithi110 ceph-mon[97633]: pgmap v21056: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:27:19.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:18 smithi104 ceph-mon[97044]: pgmap v21056: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:27:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:18.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:18.624202+0000) 2022-09-10T15:27:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:19.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:19.254421+0000) 2022-09-10T15:27:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:19.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:19.624449+0000) 2022-09-10T15:27:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:20.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:20.254597+0000) 2022-09-10T15:27:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:20 smithi104 ceph-mon[97044]: pgmap v21057: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:27:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:20 smithi110 ceph-mon[97633]: pgmap v21057: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:27:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:20.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:20.624664+0000) 2022-09-10T15:27:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:21.253+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:21.254821+0000) 2022-09-10T15:27:22.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:21.623+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:21.624912+0000) 2022-09-10T15:27:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:22.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:22.255090+0000) 2022-09-10T15:27:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:22 smithi104 ceph-mon[97044]: pgmap v21058: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:27:22.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:22 smithi110 ceph-mon[97633]: pgmap v21058: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:27:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:22.624+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:22.625089+0000) 2022-09-10T15:27:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:27:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:27:22.637+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:27:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:23.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:23.255345+0000) 2022-09-10T15:27:24.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:23.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:23.625359+0000) 2022-09-10T15:27:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:24.254+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:24.255555+0000) 2022-09-10T15:27:24.801 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:24 smithi104 ceph-mon[97044]: pgmap v21059: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:27:24.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:24 smithi110 ceph-mon[97633]: pgmap v21059: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:27:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:24.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:24.625628+0000) 2022-09-10T15:27:25.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:25.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:25.255799+0000) 2022-09-10T15:27:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:25.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:25.625857+0000) 2022-09-10T15:27:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:26.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:26.255988+0000) 2022-09-10T15:27:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:26 smithi104 ceph-mon[97044]: pgmap v21060: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:27:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:26 smithi110 ceph-mon[97633]: pgmap v21060: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:27:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:26.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:26.626126+0000) 2022-09-10T15:27:27.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:27.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:27.256222+0000) 2022-09-10T15:27:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:27.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:27.626293+0000) 2022-09-10T15:27:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:28.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:28.256401+0000) 2022-09-10T15:27:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:28 smithi110 ceph-mon[97633]: pgmap v21061: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:29.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:28 smithi104 ceph-mon[97044]: pgmap v21061: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:29.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:28.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:28.626440+0000) 2022-09-10T15:27:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:29.255+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:29.256625+0000) 2022-09-10T15:27:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:29.626+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:29.626678+0000) 2022-09-10T15:27:30.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:30.256+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:30.256836+0000) 2022-09-10T15:27:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:30 smithi104 ceph-mon[97044]: pgmap v21062: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:30 smithi110 ceph-mon[97633]: pgmap v21062: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:31.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:30.625+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:30.626878+0000) 2022-09-10T15:27:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:31.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:31.257078+0000) 2022-09-10T15:27:32.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:31.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:31.627150+0000) 2022-09-10T15:27:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:32.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:32.257335+0000) 2022-09-10T15:27:32.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:32 smithi104 ceph-mon[97044]: pgmap v21063: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:32.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:32 smithi110 ceph-mon[97633]: pgmap v21063: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:33.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:32.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:32.627332+0000) 2022-09-10T15:27:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:33.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:33.257616+0000) 2022-09-10T15:27:34.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:33.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:33.627505+0000) 2022-09-10T15:27:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:34.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:34.257865+0000) 2022-09-10T15:27:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:34 smithi104 ceph-mon[97044]: pgmap v21064: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:27:34.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:34 smithi110 ceph-mon[97633]: pgmap v21064: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:27:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:34.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:34.627752+0000) 2022-09-10T15:27:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:35.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:35.258137+0000) 2022-09-10T15:27:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:35.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:35.627977+0000) 2022-09-10T15:27:36.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:36.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:36.258360+0000) 2022-09-10T15:27:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:36 smithi104 ceph-mon[97044]: pgmap v21065: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:27:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:36 smithi110 ceph-mon[97633]: pgmap v21065: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:27:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:36.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:36.628219+0000) 2022-09-10T15:27:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:37.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:37.258617+0000) 2022-09-10T15:27:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:27:37.650+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:27:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:37.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:37.628470+0000) 2022-09-10T15:27:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:38.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:38.258788+0000) 2022-09-10T15:27:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:38 smithi110 ceph-mon[97633]: pgmap v21066: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:38 smithi104 ceph-mon[97044]: pgmap v21066: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:38.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:38.628663+0000) 2022-09-10T15:27:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:39.257+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:39.258963+0000) 2022-09-10T15:27:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:39.627+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:39.628867+0000) 2022-09-10T15:27:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:40.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:40.259194+0000) 2022-09-10T15:27:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:40 smithi104 ceph-mon[97044]: pgmap v21067: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:40 smithi110 ceph-mon[97633]: pgmap v21067: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:40.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:40.629048+0000) 2022-09-10T15:27:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:41.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:41.259384+0000) 2022-09-10T15:27:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:41.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:41.629226+0000) 2022-09-10T15:27:42.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:42.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:42.259632+0000) 2022-09-10T15:27:42.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:42 smithi104 ceph-mon[97044]: pgmap v21068: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:42.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:42 smithi110 ceph-mon[97633]: pgmap v21068: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:42.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:42.629472+0000) 2022-09-10T15:27:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:43.258+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:43.259904+0000) 2022-09-10T15:27:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:43.628+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:43.629715+0000) 2022-09-10T15:27:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:44.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:44.260087+0000) 2022-09-10T15:27:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:44 smithi104 ceph-mon[97044]: pgmap v21069: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:27:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:44 smithi110 ceph-mon[97633]: pgmap v21069: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:27:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:44.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:44.629943+0000) 2022-09-10T15:27:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:45.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:45.260338+0000) 2022-09-10T15:27:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:45.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:45.630169+0000) 2022-09-10T15:27:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:46.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:46.260576+0000) 2022-09-10T15:27:46.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:46 smithi104 ceph-mon[97044]: pgmap v21070: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:27:46.917 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:46 smithi110 ceph-mon[97633]: pgmap v21070: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:27:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:46.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:46.630408+0000) 2022-09-10T15:27:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:47.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:47.260754+0000) 2022-09-10T15:27:47.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:27:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:27:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:27:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:27:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:27:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:27:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:27:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:27:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:47.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:47.630659+0000) 2022-09-10T15:27:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:48.259+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:48.260999+0000) 2022-09-10T15:27:48.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:48 smithi110 ceph-mon[97633]: pgmap v21071: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:49.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:48 smithi104 ceph-mon[97044]: pgmap v21071: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:49.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:48.629+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:48.630887+0000) 2022-09-10T15:27:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:49.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:49.261253+0000) 2022-09-10T15:27:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:49.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:49.631130+0000) 2022-09-10T15:27:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:50.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:50.261459+0000) 2022-09-10T15:27:50.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:50 smithi104 ceph-mon[97044]: pgmap v21072: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:50.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:50 smithi110 ceph-mon[97633]: pgmap v21072: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:27:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:50.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:50.631351+0000) 2022-09-10T15:27:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:51.260+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:51.261701+0000) 2022-09-10T15:27:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:51.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:51.631565+0000) 2022-09-10T15:27:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:52.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:52.261985+0000) 2022-09-10T15:27:52.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:52 smithi104 ceph-mon[97044]: pgmap v21073: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:52.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:52 smithi110 ceph-mon[97633]: pgmap v21073: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:52.630+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:52.631761+0000) 2022-09-10T15:27:53.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:27:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:27:52.666+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:27:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:53.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:53.262182+0000) 2022-09-10T15:27:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:53.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:53.631956+0000) 2022-09-10T15:27:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:54.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:54.262371+0000) 2022-09-10T15:27:54.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:54 smithi104 ceph-mon[97044]: pgmap v21074: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:27:54.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:54 smithi110 ceph-mon[97633]: pgmap v21074: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:27:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:54.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:54.632205+0000) 2022-09-10T15:27:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:55.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:55.262584+0000) 2022-09-10T15:27:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:55.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:55.632420+0000) 2022-09-10T15:27:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:56.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:56.262819+0000) 2022-09-10T15:27:56.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:56 smithi104 ceph-mon[97044]: pgmap v21075: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:27:56.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:56 smithi110 ceph-mon[97633]: pgmap v21075: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:27:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:56.631+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:56.632657+0000) 2022-09-10T15:27:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:57.261+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:57.263005+0000) 2022-09-10T15:27:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:57.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:57.632903+0000) 2022-09-10T15:27:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:58.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:58.263204+0000) 2022-09-10T15:27:58.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:27:58 smithi110 ceph-mon[97633]: pgmap v21076: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:59.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:27:58 smithi104 ceph-mon[97044]: pgmap v21076: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:27:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:58.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:58.633137+0000) 2022-09-10T15:27:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:27:59.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:59.263481+0000) 2022-09-10T15:28:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:27:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:27:59.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:27:59.633382+0000) 2022-09-10T15:28:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:00.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:00.263713+0000) 2022-09-10T15:28:00.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:00 smithi104 ceph-mon[97044]: pgmap v21077: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:00.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:00 smithi110 ceph-mon[97633]: pgmap v21077: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:00.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:00.633551+0000) 2022-09-10T15:28:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:01.262+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:01.263883+0000) 2022-09-10T15:28:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:01.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:01.633765+0000) 2022-09-10T15:28:02.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:02.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:02.264136+0000) 2022-09-10T15:28:02.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:02 smithi104 ceph-mon[97044]: pgmap v21078: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:02.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:02 smithi110 ceph-mon[97633]: pgmap v21078: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:02.632+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:02.633988+0000) 2022-09-10T15:28:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:03.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:03.264413+0000) 2022-09-10T15:28:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:03.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:03.634176+0000) 2022-09-10T15:28:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:04.263+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:04.264583+0000) 2022-09-10T15:28:04.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:04 smithi104 ceph-mon[97044]: pgmap v21079: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:28:04.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:04 smithi110 ceph-mon[97633]: pgmap v21079: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 22 KiB/s wr, 11 op/s 2022-09-10T15:28:05.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:04.633+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:04.634391+0000) 2022-09-10T15:28:05.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:05.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:05.264853+0000) 2022-09-10T15:28:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:05.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:05.634644+0000) 2022-09-10T15:28:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:06.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:06.265044+0000) 2022-09-10T15:28:06.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:06 smithi104 ceph-mon[97044]: pgmap v21080: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:28:06.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:06 smithi110 ceph-mon[97633]: pgmap v21080: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 26 op/s 2022-09-10T15:28:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:06.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:06.634829+0000) 2022-09-10T15:28:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:07.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:07.265283+0000) 2022-09-10T15:28:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:07.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:07.635047+0000) 2022-09-10T15:28:08.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:28:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:28:07.677+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:28:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:08.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:08.265495+0000) 2022-09-10T15:28:08.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:08 smithi110 ceph-mon[97633]: pgmap v21081: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:28:09.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:08 smithi104 ceph-mon[97044]: pgmap v21081: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:28:09.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:08.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:08.635217+0000) 2022-09-10T15:28:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:09.264+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:09.265718+0000) 2022-09-10T15:28:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:09 smithi110 ceph-mon[97633]: osdmap e1398: 12 total, 12 up, 12 in 2022-09-10T15:28:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:09 smithi104 ceph-mon[97044]: osdmap e1398: 12 total, 12 up, 12 in 2022-09-10T15:28:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:09.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:09.635451+0000) 2022-09-10T15:28:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:10.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:10.265946+0000) 2022-09-10T15:28:10.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:10 smithi110 ceph-mon[97633]: pgmap v21083: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:28:11.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:10 smithi104 ceph-mon[97044]: pgmap v21083: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:28:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:10.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:10.635679+0000) 2022-09-10T15:28:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:11.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:11.266188+0000) 2022-09-10T15:28:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:11.634+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:11.635966+0000) 2022-09-10T15:28:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:12.265+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:12.266350+0000) 2022-09-10T15:28:12.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:12 smithi104 ceph-mon[97044]: pgmap v21084: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:28:12.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:12 smithi110 ceph-mon[97633]: pgmap v21084: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:28:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:12.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:12.636248+0000) 2022-09-10T15:28:13.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:13.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:13.266609+0000) 2022-09-10T15:28:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:13 smithi110 ceph-mon[97633]: osdmap e1399: 12 total, 12 up, 12 in 2022-09-10T15:28:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:13 smithi104 ceph-mon[97044]: osdmap e1399: 12 total, 12 up, 12 in 2022-09-10T15:28:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:13.635+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:13.636457+0000) 2022-09-10T15:28:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:14.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:14.266779+0000) 2022-09-10T15:28:14.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:14 smithi110 ceph-mon[97633]: pgmap v21086: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:28:15.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:14 smithi104 ceph-mon[97044]: pgmap v21086: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 8 op/s 2022-09-10T15:28:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:14.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:14.636691+0000) 2022-09-10T15:28:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:15.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:15.267004+0000) 2022-09-10T15:28:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:15.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:15.636963+0000) 2022-09-10T15:28:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:16.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:16.267241+0000) 2022-09-10T15:28:16.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:16 smithi104 ceph-mon[97044]: pgmap v21087: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:28:16.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:16 smithi110 ceph-mon[97633]: pgmap v21087: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:28:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:16.636+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:16.637196+0000) 2022-09-10T15:28:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:17.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:17.267456+0000) 2022-09-10T15:28:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:17.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:17.637454+0000) 2022-09-10T15:28:18.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:18.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:18.267610+0000) 2022-09-10T15:28:18.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:18 smithi110 ceph-mon[97633]: pgmap v21088: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:28:19.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:18 smithi104 ceph-mon[97044]: pgmap v21088: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:28:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:18.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:18.637651+0000) 2022-09-10T15:28:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:19.266+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:19.267874+0000) 2022-09-10T15:28:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:19.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:19.637851+0000) 2022-09-10T15:28:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:20.267+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:20.268165+0000) 2022-09-10T15:28:20.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:20 smithi104 ceph-mon[97044]: pgmap v21089: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:28:20.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:20 smithi110 ceph-mon[97633]: pgmap v21089: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:28:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:20.637+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:20.638071+0000) 2022-09-10T15:28:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:21.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:21.268380+0000) 2022-09-10T15:28:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:21.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:21.638295+0000) 2022-09-10T15:28:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:22.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:22.268607+0000) 2022-09-10T15:28:22.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:22 smithi104 ceph-mon[97044]: pgmap v21090: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:28:22.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:22 smithi110 ceph-mon[97633]: pgmap v21090: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:28:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:22.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:22.638464+0000) 2022-09-10T15:28:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:28:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:28:22.692+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:28:23.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:23.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:23.268821+0000) 2022-09-10T15:28:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:23.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:23.638639+0000) 2022-09-10T15:28:24.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:24.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:24.269006+0000) 2022-09-10T15:28:24.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:24 smithi104 ceph-mon[97044]: pgmap v21091: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:28:24.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:24 smithi110 ceph-mon[97633]: pgmap v21091: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 86 B/s wr, 9 op/s 2022-09-10T15:28:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:24.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:24.638889+0000) 2022-09-10T15:28:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:25.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:25.269268+0000) 2022-09-10T15:28:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:25.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:25.639188+0000) 2022-09-10T15:28:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:26.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:26.269398+0000) 2022-09-10T15:28:26.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:26 smithi104 ceph-mon[97044]: pgmap v21092: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:28:26.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:26 smithi110 ceph-mon[97633]: pgmap v21092: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 11 op/s 2022-09-10T15:28:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:26.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:26.639415+0000) 2022-09-10T15:28:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:27.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:27.269592+0000) 2022-09-10T15:28:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:27.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:27.639668+0000) 2022-09-10T15:28:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:28.268+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:28.269791+0000) 2022-09-10T15:28:28.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:28 smithi110 ceph-mon[97633]: pgmap v21093: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:29.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:28 smithi104 ceph-mon[97044]: pgmap v21093: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:28.638+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:28.639853+0000) 2022-09-10T15:28:29.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:29.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:29.270003+0000) 2022-09-10T15:28:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:29.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:29.640014+0000) 2022-09-10T15:28:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:30.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:30.270263+0000) 2022-09-10T15:28:30.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:30 smithi104 ceph-mon[97044]: pgmap v21094: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:30.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:30 smithi110 ceph-mon[97633]: pgmap v21094: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:30.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:30.640197+0000) 2022-09-10T15:28:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:31.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:31.270470+0000) 2022-09-10T15:28:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:31.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:31.640430+0000) 2022-09-10T15:28:32.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:32.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:32.270699+0000) 2022-09-10T15:28:32.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:32 smithi104 ceph-mon[97044]: pgmap v21095: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:32.910 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:32 smithi110 ceph-mon[97633]: pgmap v21095: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:32.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:32.640651+0000) 2022-09-10T15:28:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:33.269+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:33.270962+0000) 2022-09-10T15:28:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:33.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:33.640844+0000) 2022-09-10T15:28:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:34.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:34.271155+0000) 2022-09-10T15:28:34.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:34 smithi104 ceph-mon[97044]: pgmap v21096: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:28:34.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:34 smithi110 ceph-mon[97633]: pgmap v21096: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2022-09-10T15:28:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:34.639+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:34.640989+0000) 2022-09-10T15:28:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:35.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:35.271400+0000) 2022-09-10T15:28:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:35.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:35.641199+0000) 2022-09-10T15:28:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:36.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:36.271563+0000) 2022-09-10T15:28:36.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:36 smithi104 ceph-mon[97044]: pgmap v21097: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:28:36.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:36 smithi110 ceph-mon[97633]: pgmap v21097: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:28:37.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:36.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:36.641455+0000) 2022-09-10T15:28:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:37.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:37.271761+0000) 2022-09-10T15:28:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:28:37.697+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:28:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:37.640+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:37.641713+0000) 2022-09-10T15:28:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:38.270+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:38.271978+0000) 2022-09-10T15:28:38.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:38 smithi110 ceph-mon[97633]: pgmap v21098: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:39.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:38 smithi104 ceph-mon[97044]: pgmap v21098: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:38.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:38.641944+0000) 2022-09-10T15:28:39.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:39.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:39.272228+0000) 2022-09-10T15:28:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:39.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:39.642141+0000) 2022-09-10T15:28:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:40.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:40.272488+0000) 2022-09-10T15:28:40.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:40 smithi104 ceph-mon[97044]: pgmap v21099: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:40.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:40 smithi110 ceph-mon[97633]: pgmap v21099: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:40.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:40.642349+0000) 2022-09-10T15:28:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:41.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:41.272733+0000) 2022-09-10T15:28:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:41.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:41.642551+0000) 2022-09-10T15:28:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:42.271+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:42.272996+0000) 2022-09-10T15:28:42.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:42 smithi104 ceph-mon[97044]: pgmap v21100: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:42.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:42 smithi110 ceph-mon[97633]: pgmap v21100: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:42.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:42.642725+0000) 2022-09-10T15:28:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:43.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:43.273233+0000) 2022-09-10T15:28:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:43.641+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:43.642911+0000) 2022-09-10T15:28:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:44.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:44.273398+0000) 2022-09-10T15:28:44.797 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:44 smithi104 ceph-mon[97044]: pgmap v21101: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:44.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:44 smithi110 ceph-mon[97633]: pgmap v21101: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:44.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:44.643147+0000) 2022-09-10T15:28:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:45.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:45.273615+0000) 2022-09-10T15:28:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:45.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:45.643354+0000) 2022-09-10T15:28:46.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:46.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:46.273810+0000) 2022-09-10T15:28:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:46.642+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:46.643541+0000) 2022-09-10T15:28:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:47.272+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:47.274049+0000) 2022-09-10T15:28:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:47 smithi104 ceph-mon[97044]: pgmap v21102: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:28:47.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:28:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:28:47.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:28:47.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:28:47.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:47 smithi110 ceph-mon[97633]: pgmap v21102: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2022-09-10T15:28:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:28:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:28:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:28:47.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:28:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:47.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:47.643773+0000) 2022-09-10T15:28:48.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:48.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:48.274243+0000) 2022-09-10T15:28:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:48.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:48.643985+0000) 2022-09-10T15:28:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:49.273+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:49.274492+0000) 2022-09-10T15:28:49.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:49 smithi110 ceph-mon[97633]: pgmap v21103: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:28:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:49 smithi104 ceph-mon[97044]: pgmap v21103: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:28:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:49.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:49.644257+0000) 2022-09-10T15:28:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:50.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:50.274723+0000) 2022-09-10T15:28:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:50.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:50.644445+0000) 2022-09-10T15:28:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:51.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:51.274958+0000) 2022-09-10T15:28:51.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:51 smithi110 ceph-mon[97633]: pgmap v21104: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:51 smithi104 ceph-mon[97044]: pgmap v21104: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:51.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:51.644675+0000) 2022-09-10T15:28:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:52.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:52.275197+0000) 2022-09-10T15:28:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:52.643+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:52.644851+0000) 2022-09-10T15:28:53.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:28:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:28:52.707+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:28:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:53.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:53.275375+0000) 2022-09-10T15:28:53.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:53 smithi110 ceph-mon[97633]: pgmap v21105: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:28:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:53 smithi104 ceph-mon[97044]: pgmap v21105: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:28:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:53.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:53.645157+0000) 2022-09-10T15:28:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:54.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:54.275615+0000) 2022-09-10T15:28:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:54.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:54.645392+0000) 2022-09-10T15:28:55.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:55.274+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:55.275844+0000) 2022-09-10T15:28:55.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:55 smithi110 ceph-mon[97633]: pgmap v21106: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:56.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:55 smithi104 ceph-mon[97044]: pgmap v21106: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:28:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:55.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:55.645582+0000) 2022-09-10T15:28:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:56.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:56.276065+0000) 2022-09-10T15:28:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:56.644+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:56.645784+0000) 2022-09-10T15:28:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:57.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:57.276249+0000) 2022-09-10T15:28:57.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:57 smithi110 ceph-mon[97633]: pgmap v21107: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:58.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:57 smithi104 ceph-mon[97044]: pgmap v21107: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:28:58.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:57.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:57.645964+0000) 2022-09-10T15:28:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:58.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:58.276367+0000) 2022-09-10T15:28:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:58.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:58.646194+0000) 2022-09-10T15:28:59.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:28:59.275+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:59.276588+0000) 2022-09-10T15:28:59.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:28:59 smithi110 ceph-mon[97633]: pgmap v21108: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:28:59 smithi104 ceph-mon[97044]: pgmap v21108: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:28:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:28:59.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:28:59.646450+0000) 2022-09-10T15:29:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:00.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:00.276841+0000) 2022-09-10T15:29:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:00.645+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:00.646668+0000) 2022-09-10T15:29:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:01.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:01.277064+0000) 2022-09-10T15:29:01.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:01 smithi110 ceph-mon[97633]: pgmap v21109: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:01 smithi104 ceph-mon[97044]: pgmap v21109: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:01.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:01.646869+0000) 2022-09-10T15:29:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:02.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:02.277273+0000) 2022-09-10T15:29:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:02.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:02.647086+0000) 2022-09-10T15:29:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:03.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:03.277503+0000) 2022-09-10T15:29:03.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:03 smithi110 ceph-mon[97633]: pgmap v21110: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:03 smithi104 ceph-mon[97044]: pgmap v21110: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:03.646+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:03.647358+0000) 2022-09-10T15:29:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:04.276+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:04.277695+0000) 2022-09-10T15:29:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:04.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:04.647596+0000) 2022-09-10T15:29:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:05.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:05.277935+0000) 2022-09-10T15:29:05.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:05 smithi110 ceph-mon[97633]: pgmap v21111: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:29:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:05 smithi104 ceph-mon[97044]: pgmap v21111: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:29:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:05.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:05.647861+0000) 2022-09-10T15:29:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:06.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:06.278174+0000) 2022-09-10T15:29:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:06.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:06.648058+0000) 2022-09-10T15:29:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:07.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:07.278344+0000) 2022-09-10T15:29:07.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:07 smithi110 ceph-mon[97633]: pgmap v21112: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:29:08.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:07 smithi104 ceph-mon[97044]: pgmap v21112: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:29:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:07.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:07.648247+0000) 2022-09-10T15:29:08.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:29:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:29:07.713+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:29:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:08.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:08.278485+0000) 2022-09-10T15:29:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:08.647+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:08.648376+0000) 2022-09-10T15:29:09.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:09.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:09.278696+0000) 2022-09-10T15:29:09.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:09 smithi110 ceph-mon[97633]: pgmap v21113: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:29:09.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:09 smithi110 ceph-mon[97633]: osdmap e1400: 12 total, 12 up, 12 in 2022-09-10T15:29:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:09 smithi104 ceph-mon[97044]: pgmap v21113: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:29:10.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:09 smithi104 ceph-mon[97044]: osdmap e1400: 12 total, 12 up, 12 in 2022-09-10T15:29:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:09.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:09.648567+0000) 2022-09-10T15:29:10.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:10.277+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:10.278836+0000) 2022-09-10T15:29:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:10.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:10.648796+0000) 2022-09-10T15:29:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:11.278+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:11.279069+0000) 2022-09-10T15:29:11.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:11 smithi110 ceph-mon[97633]: pgmap v21115: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:29:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:11 smithi104 ceph-mon[97044]: pgmap v21115: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:29:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:11.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:11.649014+0000) 2022-09-10T15:29:12.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:12.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:12.279299+0000) 2022-09-10T15:29:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:12.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:12.649257+0000) 2022-09-10T15:29:13.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:13.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:13.279498+0000) 2022-09-10T15:29:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:13 smithi110 ceph-mon[97633]: pgmap v21116: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:29:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:13 smithi110 ceph-mon[97633]: osdmap e1401: 12 total, 12 up, 12 in 2022-09-10T15:29:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:13 smithi104 ceph-mon[97044]: pgmap v21116: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:29:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:13 smithi104 ceph-mon[97044]: osdmap e1401: 12 total, 12 up, 12 in 2022-09-10T15:29:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:13.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:13.649487+0000) 2022-09-10T15:29:14.560 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:14.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:14.279722+0000) 2022-09-10T15:29:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:14.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:14.649689+0000) 2022-09-10T15:29:15.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:15.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:15.279978+0000) 2022-09-10T15:29:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:15 smithi110 ceph-mon[97633]: pgmap v21118: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:29:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:15 smithi104 ceph-mon[97044]: pgmap v21118: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:29:16.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:15.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:15.649812+0000) 2022-09-10T15:29:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:16.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:16.280171+0000) 2022-09-10T15:29:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:16.648+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:16.650000+0000) 2022-09-10T15:29:17.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:17.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:17.280365+0000) 2022-09-10T15:29:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:17 smithi110 ceph-mon[97633]: pgmap v21119: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:29:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:17 smithi104 ceph-mon[97044]: pgmap v21119: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:29:18.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:17.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:17.650216+0000) 2022-09-10T15:29:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:18.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:18.280574+0000) 2022-09-10T15:29:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:18.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:18.650418+0000) 2022-09-10T15:29:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:19.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:19.280727+0000) 2022-09-10T15:29:19.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:19 smithi110 ceph-mon[97633]: pgmap v21120: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:29:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:19 smithi104 ceph-mon[97044]: pgmap v21120: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:29:20.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:19.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:19.650641+0000) 2022-09-10T15:29:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:20.279+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:20.280961+0000) 2022-09-10T15:29:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:20.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:20.650797+0000) 2022-09-10T15:29:21.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:21.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:21.281190+0000) 2022-09-10T15:29:21.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:21 smithi110 ceph-mon[97633]: pgmap v21121: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:29:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:21 smithi104 ceph-mon[97044]: pgmap v21121: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:29:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:21.649+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:21.651016+0000) 2022-09-10T15:29:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:22.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:22.281403+0000) 2022-09-10T15:29:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:29:22.717+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:29:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:22.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:22.651264+0000) 2022-09-10T15:29:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:23.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:23.281580+0000) 2022-09-10T15:29:23.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:23 smithi110 ceph-mon[97633]: pgmap v21122: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:29:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:23 smithi104 ceph-mon[97044]: pgmap v21122: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:29:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:23.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:23.651523+0000) 2022-09-10T15:29:24.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:24.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:24.281718+0000) 2022-09-10T15:29:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:24.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:24.651804+0000) 2022-09-10T15:29:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:25.280+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:25.281864+0000) 2022-09-10T15:29:25.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:25 smithi110 ceph-mon[97633]: pgmap v21123: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:29:26.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:25 smithi104 ceph-mon[97044]: pgmap v21123: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:29:26.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:25.650+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:25.652013+0000) 2022-09-10T15:29:26.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:26.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:26.282029+0000) 2022-09-10T15:29:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:26.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:26.652237+0000) 2022-09-10T15:29:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:27.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:27.282275+0000) 2022-09-10T15:29:27.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:27 smithi110 ceph-mon[97633]: pgmap v21124: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:28.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:27 smithi104 ceph-mon[97044]: pgmap v21124: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:27.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:27.652460+0000) 2022-09-10T15:29:28.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:28.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:28.282506+0000) 2022-09-10T15:29:29.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:28.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:28.652632+0000) 2022-09-10T15:29:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:29.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:29.282721+0000) 2022-09-10T15:29:29.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:29 smithi110 ceph-mon[97633]: pgmap v21125: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:29 smithi104 ceph-mon[97044]: pgmap v21125: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:30.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:29.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:29.652799+0000) 2022-09-10T15:29:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:30.281+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:30.282891+0000) 2022-09-10T15:29:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:30.651+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:30.653068+0000) 2022-09-10T15:29:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:31.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:31.283110+0000) 2022-09-10T15:29:31.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:31 smithi110 ceph-mon[97633]: pgmap v21126: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:31 smithi104 ceph-mon[97044]: pgmap v21126: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:31.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:31.653309+0000) 2022-09-10T15:29:32.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:32.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:32.283332+0000) 2022-09-10T15:29:33.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:32.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:32.653447+0000) 2022-09-10T15:29:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:33.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:33.283548+0000) 2022-09-10T15:29:33.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:33 smithi110 ceph-mon[97633]: pgmap v21127: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:34.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:33 smithi104 ceph-mon[97044]: pgmap v21127: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:34.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:33.652+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:33.653688+0000) 2022-09-10T15:29:34.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:34.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:34.283717+0000) 2022-09-10T15:29:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:34.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:34.653897+0000) 2022-09-10T15:29:35.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:35.282+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:35.283897+0000) 2022-09-10T15:29:35.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:35 smithi110 ceph-mon[97633]: pgmap v21128: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:36.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:35 smithi104 ceph-mon[97044]: pgmap v21128: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:35.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:35.654087+0000) 2022-09-10T15:29:36.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:36.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:36.284144+0000) 2022-09-10T15:29:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:36.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:36.654331+0000) 2022-09-10T15:29:37.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:37.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:37.284350+0000) 2022-09-10T15:29:37.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:37 smithi110 ceph-mon[97633]: pgmap v21129: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:37 smithi104 ceph-mon[97044]: pgmap v21129: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:29:37.733+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:29:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:37.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:37.654539+0000) 2022-09-10T15:29:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:38.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:38.284604+0000) 2022-09-10T15:29:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:38.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:38.654744+0000) 2022-09-10T15:29:39.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:39.283+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:39.284847+0000) 2022-09-10T15:29:39.912 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:39 smithi110 ceph-mon[97633]: pgmap v21130: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:39 smithi104 ceph-mon[97044]: pgmap v21130: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:39.653+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:39.654974+0000) 2022-09-10T15:29:40.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:40.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:40.285128+0000) 2022-09-10T15:29:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:40.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:40.655269+0000) 2022-09-10T15:29:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:41.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:41.285374+0000) 2022-09-10T15:29:41.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:41 smithi110 ceph-mon[97633]: pgmap v21131: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:41 smithi104 ceph-mon[97044]: pgmap v21131: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:41.654+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:41.655475+0000) 2022-09-10T15:29:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:42.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:42.285642+0000) 2022-09-10T15:29:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:42.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:42.655707+0000) 2022-09-10T15:29:43.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:43.284+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:43.285906+0000) 2022-09-10T15:29:44.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:43 smithi104 ceph-mon[97044]: pgmap v21132: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:43.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:43.655936+0000) 2022-09-10T15:29:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:43 smithi110 ceph-mon[97633]: pgmap v21132: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:44.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:44.286202+0000) 2022-09-10T15:29:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:44.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:44.656156+0000) 2022-09-10T15:29:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:45.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:45.286330+0000) 2022-09-10T15:29:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:45 smithi104 ceph-mon[97044]: pgmap v21133: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:45.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:45.656404+0000) 2022-09-10T15:29:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:45 smithi110 ceph-mon[97633]: pgmap v21133: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:46.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:46.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:46.286484+0000) 2022-09-10T15:29:47.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:46.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:46.656675+0000) 2022-09-10T15:29:47.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:47.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:47.286743+0000) 2022-09-10T15:29:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:47 smithi104 ceph-mon[97044]: pgmap v21134: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:29:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:29:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:29:48.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:29:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:47.655+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:47.656944+0000) 2022-09-10T15:29:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:47 smithi110 ceph-mon[97633]: pgmap v21134: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:29:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:29:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:29:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:29:48.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:48.285+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:48.286936+0000) 2022-09-10T15:29:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:48.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:48.657191+0000) 2022-09-10T15:29:49.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:49.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:49.287207+0000) 2022-09-10T15:29:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:49 smithi104 ceph-mon[97044]: pgmap v21135: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:49.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:49.657433+0000) 2022-09-10T15:29:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:49 smithi110 ceph-mon[97633]: pgmap v21135: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:50.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:50.287360+0000) 2022-09-10T15:29:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:50.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:50.657635+0000) 2022-09-10T15:29:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:51.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:51.287558+0000) 2022-09-10T15:29:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:51 smithi104 ceph-mon[97044]: pgmap v21136: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:51.656+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:51.657828+0000) 2022-09-10T15:29:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:51 smithi110 ceph-mon[97633]: pgmap v21136: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:52.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:52.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:52.287736+0000) 2022-09-10T15:29:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:29:52.743+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:29:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:52.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:52.658051+0000) 2022-09-10T15:29:53.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:53.286+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:53.287918+0000) 2022-09-10T15:29:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:53 smithi104 ceph-mon[97044]: pgmap v21137: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:53.657+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:53.658244+0000) 2022-09-10T15:29:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:53 smithi110 ceph-mon[97633]: pgmap v21137: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:54.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:54.288139+0000) 2022-09-10T15:29:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:54.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:54.658463+0000) 2022-09-10T15:29:55.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:55.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:55.288308+0000) 2022-09-10T15:29:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:55 smithi104 ceph-mon[97044]: pgmap v21138: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:55.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:55.658698+0000) 2022-09-10T15:29:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:55 smithi110 ceph-mon[97633]: pgmap v21138: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:29:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:56.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:56.288532+0000) 2022-09-10T15:29:57.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:56.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:56.658849+0000) 2022-09-10T15:29:57.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:57.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:57.288770+0000) 2022-09-10T15:29:58.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:57 smithi104 ceph-mon[97044]: pgmap v21139: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:57.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:57.659026+0000) 2022-09-10T15:29:58.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:57 smithi110 ceph-mon[97633]: pgmap v21139: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:29:58.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:58.287+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:58.288947+0000) 2022-09-10T15:29:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:58.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:58.659241+0000) 2022-09-10T15:29:59.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:29:59.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:59.289101+0000) 2022-09-10T15:30:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:29:59 smithi104 ceph-mon[97044]: pgmap v21140: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:00.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:29:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:29:59.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:29:59.659417+0000) 2022-09-10T15:30:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:29:59 smithi110 ceph-mon[97633]: pgmap v21140: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:00.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:00.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:00.289239+0000) 2022-09-10T15:30:01.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:30:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:00.658+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:00.659529+0000) 2022-09-10T15:30:01.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:30:01.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:01.288+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:01.289441+0000) 2022-09-10T15:30:01.798 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:01 smithi104 ceph-mon[97044]: pgmap v21141: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:01.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:01.659680+0000) 2022-09-10T15:30:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:01 smithi110 ceph-mon[97633]: pgmap v21141: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:02.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:02.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:02.289579+0000) 2022-09-10T15:30:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:02.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:02.659854+0000) 2022-09-10T15:30:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:03.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:03.289777+0000) 2022-09-10T15:30:04.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:03 smithi104 ceph-mon[97044]: pgmap v21142: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:03.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:03.660062+0000) 2022-09-10T15:30:04.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:03 smithi110 ceph-mon[97633]: pgmap v21142: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:04.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:04.289946+0000) 2022-09-10T15:30:05.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:04.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:04.660321+0000) 2022-09-10T15:30:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:05.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:05.290130+0000) 2022-09-10T15:30:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:05 smithi104 ceph-mon[97044]: pgmap v21143: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:30:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:05.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:05.660500+0000) 2022-09-10T15:30:06.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:05 smithi110 ceph-mon[97633]: pgmap v21143: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:30:06.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:06.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:06.290335+0000) 2022-09-10T15:30:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:06.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:06.660685+0000) 2022-09-10T15:30:07.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:07.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:07.290594+0000) 2022-09-10T15:30:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:07 smithi104 ceph-mon[97044]: pgmap v21144: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:30:08.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:30:07.755+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:30:08.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:07.659+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:07.660895+0000) 2022-09-10T15:30:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:07 smithi110 ceph-mon[97633]: pgmap v21144: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:30:08.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:08.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:08.290874+0000) 2022-09-10T15:30:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:08.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:08.661131+0000) 2022-09-10T15:30:09.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:09.289+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:09.291067+0000) 2022-09-10T15:30:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:09 smithi104 ceph-mon[97044]: pgmap v21145: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:30:10.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:09 smithi104 ceph-mon[97044]: osdmap e1402: 12 total, 12 up, 12 in 2022-09-10T15:30:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:09.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:09.661374+0000) 2022-09-10T15:30:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:09 smithi110 ceph-mon[97633]: pgmap v21145: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:30:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:09 smithi110 ceph-mon[97633]: osdmap e1402: 12 total, 12 up, 12 in 2022-09-10T15:30:10.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:10.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:10.291341+0000) 2022-09-10T15:30:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:10.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:10.661604+0000) 2022-09-10T15:30:11.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:11.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:11.291592+0000) 2022-09-10T15:30:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:11 smithi104 ceph-mon[97044]: pgmap v21147: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:30:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:11.660+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:11.661867+0000) 2022-09-10T15:30:12.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:11 smithi110 ceph-mon[97633]: pgmap v21147: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:30:12.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:12.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:12.291813+0000) 2022-09-10T15:30:13.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:12.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:12.662126+0000) 2022-09-10T15:30:13.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:13.290+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:13.292013+0000) 2022-09-10T15:30:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:13 smithi110 ceph-mon[97633]: pgmap v21148: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:30:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:13 smithi110 ceph-mon[97633]: osdmap e1403: 12 total, 12 up, 12 in 2022-09-10T15:30:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:13 smithi104 ceph-mon[97044]: pgmap v21148: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:30:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:13 smithi104 ceph-mon[97044]: osdmap e1403: 12 total, 12 up, 12 in 2022-09-10T15:30:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:13.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:13.662339+0000) 2022-09-10T15:30:14.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:14.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:14.292232+0000) 2022-09-10T15:30:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:14.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:14.662584+0000) 2022-09-10T15:30:15.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:15.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:15.292403+0000) 2022-09-10T15:30:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:15 smithi110 ceph-mon[97633]: pgmap v21150: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:30:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:15 smithi104 ceph-mon[97044]: pgmap v21150: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:30:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:15.661+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:15.662829+0000) 2022-09-10T15:30:16.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:16.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:16.292530+0000) 2022-09-10T15:30:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:16.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:16.663031+0000) 2022-09-10T15:30:17.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:17.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:17.292764+0000) 2022-09-10T15:30:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:17 smithi110 ceph-mon[97633]: pgmap v21151: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:30:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:17 smithi104 ceph-mon[97044]: pgmap v21151: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:30:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:17.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:17.663279+0000) 2022-09-10T15:30:18.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:18.291+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:18.293026+0000) 2022-09-10T15:30:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:18.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:18.663539+0000) 2022-09-10T15:30:19.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:19.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:19.293274+0000) 2022-09-10T15:30:19.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:19 smithi110 ceph-mon[97633]: pgmap v21152: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:30:20.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:19 smithi104 ceph-mon[97044]: pgmap v21152: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:30:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:19.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:19.663767+0000) 2022-09-10T15:30:20.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:20.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:20.293526+0000) 2022-09-10T15:30:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:20.662+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:20.663969+0000) 2022-09-10T15:30:21.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:21.292+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:21.293737+0000) 2022-09-10T15:30:21.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:21 smithi110 ceph-mon[97633]: pgmap v21153: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:30:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:21 smithi104 ceph-mon[97044]: pgmap v21153: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:30:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:21.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:21.664228+0000) 2022-09-10T15:30:22.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:22.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:22.293950+0000) 2022-09-10T15:30:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:22.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:22.664477+0000) 2022-09-10T15:30:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:30:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:30:22.771+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:30:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:23.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:23.294143+0000) 2022-09-10T15:30:23.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:23 smithi110 ceph-mon[97633]: pgmap v21154: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:30:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:23 smithi104 ceph-mon[97044]: pgmap v21154: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:30:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:23.663+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:23.664726+0000) 2022-09-10T15:30:24.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:24.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:24.294281+0000) 2022-09-10T15:30:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:24.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:24.664977+0000) 2022-09-10T15:30:25.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:25.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:25.294462+0000) 2022-09-10T15:30:25.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:25 smithi110 ceph-mon[97633]: pgmap v21155: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:30:26.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:25 smithi104 ceph-mon[97044]: pgmap v21155: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:30:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:25.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:25.665232+0000) 2022-09-10T15:30:26.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:26.293+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:26.294677+0000) 2022-09-10T15:30:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:26.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:26.665469+0000) 2022-09-10T15:30:27.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:27.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:27.294913+0000) 2022-09-10T15:30:27.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:27 smithi110 ceph-mon[97633]: pgmap v21156: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:27 smithi104 ceph-mon[97044]: pgmap v21156: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:27.664+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:27.665676+0000) 2022-09-10T15:30:28.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:28.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:28.295185+0000) 2022-09-10T15:30:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:28.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:28.665900+0000) 2022-09-10T15:30:29.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:29.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:29.295422+0000) 2022-09-10T15:30:29.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:29 smithi110 ceph-mon[97633]: pgmap v21157: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:29 smithi104 ceph-mon[97044]: pgmap v21157: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:29.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:29.666153+0000) 2022-09-10T15:30:30.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:30.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:30.295624+0000) 2022-09-10T15:30:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:30.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:30.666366+0000) 2022-09-10T15:30:31.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:31.294+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:31.295866+0000) 2022-09-10T15:30:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:31 smithi104 ceph-mon[97044]: pgmap v21158: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:31.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:31.666573+0000) 2022-09-10T15:30:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:31 smithi110 ceph-mon[97633]: pgmap v21158: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:32.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:32.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:32.296072+0000) 2022-09-10T15:30:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:32.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:32.666751+0000) 2022-09-10T15:30:33.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:33.295+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:33.296338+0000) 2022-09-10T15:30:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:33 smithi104 ceph-mon[97044]: pgmap v21159: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:33.665+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:33.666969+0000) 2022-09-10T15:30:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:33 smithi110 ceph-mon[97633]: pgmap v21159: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:34.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:34.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:34.296604+0000) 2022-09-10T15:30:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:34.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:34.667246+0000) 2022-09-10T15:30:35.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:35.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:35.296858+0000) 2022-09-10T15:30:36.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:35 smithi104 ceph-mon[97044]: pgmap v21160: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:35.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:35.667420+0000) 2022-09-10T15:30:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:35 smithi110 ceph-mon[97633]: pgmap v21160: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:36.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:36.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:36.297139+0000) 2022-09-10T15:30:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:36.666+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:36.667542+0000) 2022-09-10T15:30:37.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:37.296+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:37.297301+0000) 2022-09-10T15:30:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:37 smithi104 ceph-mon[97044]: pgmap v21161: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:37.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:37.667757+0000) 2022-09-10T15:30:38.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:30:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:30:37.786+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:30:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:37 smithi110 ceph-mon[97633]: pgmap v21161: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:38.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:38.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:38.297512+0000) 2022-09-10T15:30:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:38.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:38.667963+0000) 2022-09-10T15:30:39.557 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:39.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:39.297743+0000) 2022-09-10T15:30:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:39 smithi104 ceph-mon[97044]: pgmap v21162: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:39.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:39.668181+0000) 2022-09-10T15:30:40.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:39 smithi110 ceph-mon[97633]: pgmap v21162: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:40.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:40.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:40.297981+0000) 2022-09-10T15:30:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:40.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:40.668433+0000) 2022-09-10T15:30:41.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:41.297+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:41.298211+0000) 2022-09-10T15:30:42.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:41 smithi104 ceph-mon[97044]: pgmap v21163: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:41.667+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:41.668642+0000) 2022-09-10T15:30:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:41 smithi110 ceph-mon[97633]: pgmap v21163: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:42.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:42.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:42.298415+0000) 2022-09-10T15:30:43.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:42.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:42.668870+0000) 2022-09-10T15:30:43.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:43.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:43.298644+0000) 2022-09-10T15:30:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:43 smithi104 ceph-mon[97044]: pgmap v21164: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:43.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:43.669124+0000) 2022-09-10T15:30:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:43 smithi110 ceph-mon[97633]: pgmap v21164: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:44.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:44.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:44.298892+0000) 2022-09-10T15:30:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:44.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:44.669375+0000) 2022-09-10T15:30:45.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:45.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:45.299032+0000) 2022-09-10T15:30:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:45 smithi104 ceph-mon[97044]: pgmap v21165: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:45.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:45.669632+0000) 2022-09-10T15:30:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:45 smithi110 ceph-mon[97633]: pgmap v21165: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:46.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:46.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:46.299267+0000) 2022-09-10T15:30:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:46.668+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:46.669810+0000) 2022-09-10T15:30:47.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:47.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:47.299527+0000) 2022-09-10T15:30:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:47 smithi104 ceph-mon[97044]: pgmap v21166: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:30:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:30:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:30:48.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:30:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:47.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:47.670060+0000) 2022-09-10T15:30:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:47 smithi110 ceph-mon[97633]: pgmap v21166: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:30:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:30:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:30:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:30:48.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:48.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:48.299743+0000) 2022-09-10T15:30:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:48.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:48.670332+0000) 2022-09-10T15:30:49.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:49.298+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:49.299900+0000) 2022-09-10T15:30:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:49 smithi104 ceph-mon[97044]: pgmap v21167: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:49.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:49.670536+0000) 2022-09-10T15:30:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:49 smithi110 ceph-mon[97633]: pgmap v21167: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:50.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:50.299+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:50.300141+0000) 2022-09-10T15:30:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:50.670+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:50.670764+0000) 2022-09-10T15:30:51.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:51.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:51.300393+0000) 2022-09-10T15:30:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:51 smithi104 ceph-mon[97044]: pgmap v21168: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:52.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:51.669+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:51.671005+0000) 2022-09-10T15:30:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:51 smithi110 ceph-mon[97633]: pgmap v21168: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:52.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:52.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:52.300608+0000) 2022-09-10T15:30:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:52.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:52.671215+0000) 2022-09-10T15:30:53.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:30:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:30:52.803+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:30:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:53.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:53.300838+0000) 2022-09-10T15:30:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:53 smithi104 ceph-mon[97044]: pgmap v21169: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:53.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:53.671405+0000) 2022-09-10T15:30:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:53 smithi110 ceph-mon[97633]: pgmap v21169: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:54.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:54.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:54.301149+0000) 2022-09-10T15:30:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:54.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:54.671632+0000) 2022-09-10T15:30:55.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:55.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:55.301368+0000) 2022-09-10T15:30:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:55 smithi104 ceph-mon[97044]: pgmap v21170: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:55.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:55.671884+0000) 2022-09-10T15:30:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:55 smithi110 ceph-mon[97633]: pgmap v21170: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:30:56.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:56.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:56.301553+0000) 2022-09-10T15:30:57.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:56.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:56.672082+0000) 2022-09-10T15:30:57.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:57.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:57.301747+0000) 2022-09-10T15:30:58.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:57 smithi104 ceph-mon[97044]: pgmap v21171: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:57.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:57.672281+0000) 2022-09-10T15:30:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:57 smithi110 ceph-mon[97633]: pgmap v21171: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:30:58.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:58.300+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:58.301999+0000) 2022-09-10T15:30:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:58.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:58.672457+0000) 2022-09-10T15:30:59.558 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:30:59.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:59.302156+0000) 2022-09-10T15:31:00.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:30:59 smithi104 ceph-mon[97044]: pgmap v21172: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:30:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:30:59.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:30:59.672659+0000) 2022-09-10T15:31:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:30:59 smithi110 ceph-mon[97633]: pgmap v21172: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:00.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:00.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:00.302292+0000) 2022-09-10T15:31:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:00.671+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:00.672884+0000) 2022-09-10T15:31:01.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:01.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:01.302461+0000) 2022-09-10T15:31:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:01 smithi104 ceph-mon[97044]: pgmap v21173: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:02.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:01.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:01.673138+0000) 2022-09-10T15:31:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:01 smithi110 ceph-mon[97633]: pgmap v21173: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:02.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:02.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:02.302632+0000) 2022-09-10T15:31:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:02.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:02.673366+0000) 2022-09-10T15:31:03.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:03.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:03.302882+0000) 2022-09-10T15:31:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:03 smithi104 ceph-mon[97044]: pgmap v21174: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:31:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:03.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:03.673565+0000) 2022-09-10T15:31:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:03 smithi110 ceph-mon[97633]: pgmap v21174: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:31:04.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:04.301+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:04.303054+0000) 2022-09-10T15:31:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:04.672+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:04.673844+0000) 2022-09-10T15:31:05.555 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:05.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:05.303274+0000) 2022-09-10T15:31:06.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:05 smithi104 ceph-mon[97044]: pgmap v21175: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:31:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:05.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:05.674123+0000) 2022-09-10T15:31:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:05 smithi110 ceph-mon[97633]: pgmap v21175: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:31:06.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:06.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:06.303534+0000) 2022-09-10T15:31:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:06.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:06.674281+0000) 2022-09-10T15:31:07.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:07.302+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:07.303819+0000) 2022-09-10T15:31:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:07 smithi104 ceph-mon[97044]: pgmap v21176: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:31:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:31:07.807+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:31:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:07.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:07.674494+0000) 2022-09-10T15:31:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:07 smithi110 ceph-mon[97633]: pgmap v21176: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:31:08.676 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:08.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:08.304134+0000) 2022-09-10T15:31:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:08.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:08.674717+0000) 2022-09-10T15:31:09.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:09.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:09.304301+0000) 2022-09-10T15:31:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:09 smithi104 ceph-mon[97044]: pgmap v21177: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:31:10.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:09 smithi104 ceph-mon[97044]: osdmap e1404: 12 total, 12 up, 12 in 2022-09-10T15:31:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:09.673+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:09.674931+0000) 2022-09-10T15:31:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:09 smithi110 ceph-mon[97633]: pgmap v21177: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:31:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:09 smithi110 ceph-mon[97633]: osdmap e1404: 12 total, 12 up, 12 in 2022-09-10T15:31:10.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:10.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:10.304535+0000) 2022-09-10T15:31:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:10.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:10.675233+0000) 2022-09-10T15:31:11.677 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:11.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:11.304779+0000) 2022-09-10T15:31:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:11 smithi104 ceph-mon[97044]: pgmap v21179: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:31:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:11.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:11.675426+0000) 2022-09-10T15:31:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:11 smithi110 ceph-mon[97633]: pgmap v21179: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:31:12.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:12.303+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:12.304953+0000) 2022-09-10T15:31:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:12.674+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:12.675666+0000) 2022-09-10T15:31:13.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:13.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:13.305223+0000) 2022-09-10T15:31:13.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:13 smithi110 ceph-mon[97633]: pgmap v21180: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:31:13.909 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:13 smithi110 ceph-mon[97633]: osdmap e1405: 12 total, 12 up, 12 in 2022-09-10T15:31:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:13 smithi104 ceph-mon[97044]: pgmap v21180: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:31:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:13 smithi104 ceph-mon[97044]: osdmap e1405: 12 total, 12 up, 12 in 2022-09-10T15:31:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:13.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:13.675863+0000) 2022-09-10T15:31:14.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:14.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:14.305420+0000) 2022-09-10T15:31:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:14.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:14.676120+0000) 2022-09-10T15:31:15.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:15.304+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:15.305608+0000) 2022-09-10T15:31:15.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:15 smithi110 ceph-mon[97633]: pgmap v21182: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:31:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:15 smithi104 ceph-mon[97044]: pgmap v21182: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:31:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:15.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:15.676319+0000) 2022-09-10T15:31:16.678 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:16.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:16.305868+0000) 2022-09-10T15:31:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:16.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:16.676543+0000) 2022-09-10T15:31:17.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:17.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:17.306090+0000) 2022-09-10T15:31:17.908 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:17 smithi110 ceph-mon[97633]: pgmap v21183: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:31:18.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:17 smithi104 ceph-mon[97044]: pgmap v21183: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:31:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:17.675+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:17.676751+0000) 2022-09-10T15:31:18.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:18.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:18.306340+0000) 2022-09-10T15:31:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:18.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:18.676949+0000) 2022-09-10T15:31:19.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:19.305+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:19.306535+0000) 2022-09-10T15:31:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:19 smithi104 ceph-mon[97044]: pgmap v21184: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:31:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:19.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:19.677226+0000) 2022-09-10T15:31:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:19 smithi110 ceph-mon[97633]: pgmap v21184: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:31:20.679 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:20.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:20.306698+0000) 2022-09-10T15:31:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:20.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:20.677464+0000) 2022-09-10T15:31:21.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:21.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:21.306906+0000) 2022-09-10T15:31:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:21 smithi104 ceph-mon[97044]: pgmap v21185: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:31:22.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:21.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:21.677695+0000) 2022-09-10T15:31:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:21 smithi110 ceph-mon[97633]: pgmap v21185: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:31:22.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:22.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:22.307105+0000) 2022-09-10T15:31:23.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:22.676+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:22.677917+0000) 2022-09-10T15:31:23.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:31:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:31:22.811+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:31:23.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:23.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:23.307343+0000) 2022-09-10T15:31:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:23 smithi104 ceph-mon[97044]: pgmap v21186: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:31:24.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:23.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:23.678189+0000) 2022-09-10T15:31:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:23 smithi110 ceph-mon[97633]: pgmap v21186: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:31:24.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:24.306+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:24.307520+0000) 2022-09-10T15:31:25.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:24.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:24.678443+0000) 2022-09-10T15:31:25.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:25.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:25.307771+0000) 2022-09-10T15:31:26.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:25 smithi104 ceph-mon[97044]: pgmap v21187: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:31:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:25.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:25.678586+0000) 2022-09-10T15:31:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:25 smithi110 ceph-mon[97633]: pgmap v21187: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:31:26.680 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:26.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:26.308032+0000) 2022-09-10T15:31:27.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:26.677+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:26.678783+0000) 2022-09-10T15:31:27.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:27.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:27.308269+0000) 2022-09-10T15:31:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:27 smithi104 ceph-mon[97044]: pgmap v21188: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:27.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:27.678997+0000) 2022-09-10T15:31:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:27 smithi110 ceph-mon[97633]: pgmap v21188: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:28.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:28.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:28.308520+0000) 2022-09-10T15:31:29.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:28.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:28.679171+0000) 2022-09-10T15:31:29.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:29.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:29.308655+0000) 2022-09-10T15:31:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:29 smithi104 ceph-mon[97044]: pgmap v21189: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:29.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:29.679313+0000) 2022-09-10T15:31:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:29 smithi110 ceph-mon[97633]: pgmap v21189: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:30.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:30.307+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:30.308867+0000) 2022-09-10T15:31:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:30.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:30.679494+0000) 2022-09-10T15:31:31.681 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:31.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:31.309008+0000) 2022-09-10T15:31:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:31 smithi104 ceph-mon[97044]: pgmap v21190: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:31.678+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:31.679722+0000) 2022-09-10T15:31:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:31 smithi110 ceph-mon[97633]: pgmap v21190: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:32.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:32.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:32.309183+0000) 2022-09-10T15:31:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:32.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:32.679922+0000) 2022-09-10T15:31:33.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:33.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:33.309366+0000) 2022-09-10T15:31:34.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:33 smithi104 ceph-mon[97044]: pgmap v21191: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:33.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:33.680210+0000) 2022-09-10T15:31:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:33 smithi110 ceph-mon[97633]: pgmap v21191: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:34.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:34.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:34.309611+0000) 2022-09-10T15:31:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:34.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:34.680425+0000) 2022-09-10T15:31:35.682 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:35.308+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:35.309831+0000) 2022-09-10T15:31:36.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:35 smithi104 ceph-mon[97044]: pgmap v21192: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:35.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:35.680649+0000) 2022-09-10T15:31:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:35 smithi110 ceph-mon[97633]: pgmap v21192: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:36.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:36.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:36.310053+0000) 2022-09-10T15:31:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:36.680+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:36.680816+0000) 2022-09-10T15:31:37.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:37.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:37.310312+0000) 2022-09-10T15:31:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:37 smithi104 ceph-mon[97044]: pgmap v21193: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:38.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:31:37.826+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:31:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:37.679+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:37.681043+0000) 2022-09-10T15:31:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:37 smithi110 ceph-mon[97633]: pgmap v21193: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:38.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:38.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:38.310563+0000) 2022-09-10T15:31:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:38.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:38.681290+0000) 2022-09-10T15:31:39.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:39.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:39.310790+0000) 2022-09-10T15:31:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:39 smithi104 ceph-mon[97044]: pgmap v21194: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:39.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:39.681512+0000) 2022-09-10T15:31:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:39 smithi110 ceph-mon[97633]: pgmap v21194: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:40.683 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:40.309+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:40.311032+0000) 2022-09-10T15:31:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:40.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:40.681705+0000) 2022-09-10T15:31:41.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:41.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:41.311244+0000) 2022-09-10T15:31:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:41 smithi104 ceph-mon[97044]: pgmap v21195: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:41.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:41.681966+0000) 2022-09-10T15:31:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:41 smithi110 ceph-mon[97633]: pgmap v21195: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:42.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:42.310+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:42.311424+0000) 2022-09-10T15:31:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:42.681+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:42.682190+0000) 2022-09-10T15:31:43.684 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:43.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:43.311651+0000) 2022-09-10T15:31:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:43 smithi104 ceph-mon[97044]: pgmap v21196: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:43.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:43.682427+0000) 2022-09-10T15:31:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:43 smithi110 ceph-mon[97633]: pgmap v21196: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:44.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:44.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:44.311891+0000) 2022-09-10T15:31:45.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:44.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:44.682679+0000) 2022-09-10T15:31:45.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:45.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:45.312166+0000) 2022-09-10T15:31:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:45 smithi104 ceph-mon[97044]: pgmap v21197: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:45.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:45.682889+0000) 2022-09-10T15:31:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:45 smithi110 ceph-mon[97633]: pgmap v21197: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:46.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:46.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:46.312394+0000) 2022-09-10T15:31:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:46.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:46.683048+0000) 2022-09-10T15:31:47.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:47.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:47.312597+0000) 2022-09-10T15:31:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:47 smithi104 ceph-mon[97044]: pgmap v21198: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:31:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:31:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:31:48.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:31:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:47.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:47.683219+0000) 2022-09-10T15:31:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:47 smithi110 ceph-mon[97633]: pgmap v21198: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:31:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:31:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:31:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:31:48.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:48.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:48.312829+0000) 2022-09-10T15:31:49.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:48.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:48.683371+0000) 2022-09-10T15:31:49.685 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:49.311+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:49.312997+0000) 2022-09-10T15:31:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:49 smithi104 ceph-mon[97044]: pgmap v21199: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:49.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:49.683580+0000) 2022-09-10T15:31:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:49 smithi110 ceph-mon[97633]: pgmap v21199: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:50.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:50.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:50.313249+0000) 2022-09-10T15:31:51.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:50.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:50.683797+0000) 2022-09-10T15:31:51.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:51.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:51.313508+0000) 2022-09-10T15:31:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:51 smithi104 ceph-mon[97044]: pgmap v21200: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:51.682+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:51.684025+0000) 2022-09-10T15:31:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:51 smithi110 ceph-mon[97633]: pgmap v21200: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:52.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:52.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:52.313754+0000) 2022-09-10T15:31:53.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:31:52.828+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:31:53.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:52.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:52.684246+0000) 2022-09-10T15:31:53.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:53.312+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:53.313926+0000) 2022-09-10T15:31:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:53 smithi104 ceph-mon[97044]: pgmap v21201: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:53.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:53.684457+0000) 2022-09-10T15:31:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:53 smithi110 ceph-mon[97633]: pgmap v21201: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:54.686 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:54.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:54.314202+0000) 2022-09-10T15:31:55.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:54.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:54.684667+0000) 2022-09-10T15:31:55.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:55.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:55.314424+0000) 2022-09-10T15:31:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:55 smithi104 ceph-mon[97044]: pgmap v21202: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:55.683+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:55.684912+0000) 2022-09-10T15:31:56.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:55 smithi110 ceph-mon[97633]: pgmap v21202: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:31:56.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:56.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:56.314675+0000) 2022-09-10T15:31:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:56.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:56.685180+0000) 2022-09-10T15:31:57.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:57.313+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:57.314935+0000) 2022-09-10T15:31:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:57 smithi104 ceph-mon[97044]: pgmap v21203: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:57.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:57.685313+0000) 2022-09-10T15:31:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:57 smithi110 ceph-mon[97633]: pgmap v21203: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:31:58.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:58.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:58.315182+0000) 2022-09-10T15:31:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:58.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:58.685517+0000) 2022-09-10T15:31:59.687 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:31:59.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:59.315413+0000) 2022-09-10T15:32:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:31:59 smithi104 ceph-mon[97044]: pgmap v21204: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:31:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:31:59.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:31:59.685696+0000) 2022-09-10T15:32:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:31:59 smithi110 ceph-mon[97633]: pgmap v21204: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:00.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:00.314+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:00.315659+0000) 2022-09-10T15:32:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:00.684+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:00.685845+0000) 2022-09-10T15:32:01.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:01.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:01.315838+0000) 2022-09-10T15:32:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:01 smithi104 ceph-mon[97044]: pgmap v21205: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:01.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:01.686037+0000) 2022-09-10T15:32:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:01 smithi110 ceph-mon[97633]: pgmap v21205: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:02.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:02.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:02.316042+0000) 2022-09-10T15:32:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:02.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:02.686246+0000) 2022-09-10T15:32:03.688 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:03.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:03.316364+0000) 2022-09-10T15:32:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:03 smithi104 ceph-mon[97044]: pgmap v21206: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:03.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:03.686518+0000) 2022-09-10T15:32:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:03 smithi110 ceph-mon[97633]: pgmap v21206: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:04.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:04.315+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:04.316552+0000) 2022-09-10T15:32:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:04.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:04.686753+0000) 2022-09-10T15:32:05.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:05.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:05.316731+0000) 2022-09-10T15:32:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:05 smithi104 ceph-mon[97044]: pgmap v21207: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:32:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:05.685+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:05.686998+0000) 2022-09-10T15:32:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:05 smithi110 ceph-mon[97633]: pgmap v21207: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:32:06.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:06.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:06.316952+0000) 2022-09-10T15:32:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:06.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:06.687241+0000) 2022-09-10T15:32:07.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:07.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:07.317222+0000) 2022-09-10T15:32:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:07 smithi104 ceph-mon[97044]: pgmap v21208: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:32:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:07.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:07.687432+0000) 2022-09-10T15:32:08.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:32:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:32:07.842+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:32:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:07 smithi110 ceph-mon[97633]: pgmap v21208: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:32:08.689 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:08.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:08.317471+0000) 2022-09-10T15:32:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:08.686+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:08.687642+0000) 2022-09-10T15:32:09.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:09.316+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:09.317610+0000) 2022-09-10T15:32:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:09 smithi104 ceph-mon[97044]: pgmap v21209: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:32:10.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:09 smithi104 ceph-mon[97044]: osdmap e1406: 12 total, 12 up, 12 in 2022-09-10T15:32:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:09.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:09.687851+0000) 2022-09-10T15:32:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:09 smithi110 ceph-mon[97633]: pgmap v21209: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:32:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:09 smithi110 ceph-mon[97633]: osdmap e1406: 12 total, 12 up, 12 in 2022-09-10T15:32:10.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:10.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:10.317836+0000) 2022-09-10T15:32:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:10.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:10.688056+0000) 2022-09-10T15:32:11.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:11.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:11.318084+0000) 2022-09-10T15:32:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:11 smithi104 ceph-mon[97044]: pgmap v21211: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:32:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:11.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:11.688307+0000) 2022-09-10T15:32:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:11 smithi110 ceph-mon[97633]: pgmap v21211: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:32:12.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:12.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:12.318339+0000) 2022-09-10T15:32:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:12.687+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:12.688502+0000) 2022-09-10T15:32:13.690 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:13.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:13.318515+0000) 2022-09-10T15:32:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:13 smithi104 ceph-mon[97044]: pgmap v21212: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:32:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:13 smithi104 ceph-mon[97044]: osdmap e1407: 12 total, 12 up, 12 in 2022-09-10T15:32:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:13.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:13.688757+0000) 2022-09-10T15:32:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:13 smithi110 ceph-mon[97633]: pgmap v21212: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:32:14.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:13 smithi110 ceph-mon[97633]: osdmap e1407: 12 total, 12 up, 12 in 2022-09-10T15:32:14.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:14.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:14.318660+0000) 2022-09-10T15:32:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:14.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:14.689006+0000) 2022-09-10T15:32:15.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:15.317+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:15.318908+0000) 2022-09-10T15:32:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:15 smithi104 ceph-mon[97044]: pgmap v21214: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:32:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:15.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:15.689230+0000) 2022-09-10T15:32:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:15 smithi110 ceph-mon[97633]: pgmap v21214: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:32:16.691 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:16.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:16.319170+0000) 2022-09-10T15:32:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:16.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:16.689497+0000) 2022-09-10T15:32:17.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:17.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:17.319370+0000) 2022-09-10T15:32:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:17 smithi104 ceph-mon[97044]: pgmap v21215: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:32:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:17.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:17.689780+0000) 2022-09-10T15:32:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:17 smithi110 ceph-mon[97633]: pgmap v21215: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:32:18.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:18.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:18.319575+0000) 2022-09-10T15:32:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:18.688+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:18.689993+0000) 2022-09-10T15:32:19.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:19.318+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:19.319768+0000) 2022-09-10T15:32:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:19 smithi104 ceph-mon[97044]: pgmap v21216: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:32:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:19.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:19.690239+0000) 2022-09-10T15:32:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:19 smithi110 ceph-mon[97633]: pgmap v21216: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 209 B/s wr, 10 op/s 2022-09-10T15:32:20.692 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:20.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:20.319978+0000) 2022-09-10T15:32:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:20.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:20.690450+0000) 2022-09-10T15:32:21.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:21.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:21.320212+0000) 2022-09-10T15:32:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:21 smithi104 ceph-mon[97044]: pgmap v21217: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:32:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:21.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:21.690642+0000) 2022-09-10T15:32:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:21 smithi110 ceph-mon[97633]: pgmap v21217: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:32:22.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:22.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:22.320429+0000) 2022-09-10T15:32:23.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:32:22.858+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:32:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:22.689+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:22.690901+0000) 2022-09-10T15:32:23.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:23.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:23.320659+0000) 2022-09-10T15:32:24.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:23 smithi104 ceph-mon[97044]: pgmap v21218: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:32:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:23.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:23.691184+0000) 2022-09-10T15:32:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:23 smithi110 ceph-mon[97633]: pgmap v21218: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:32:24.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:24.319+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:24.320882+0000) 2022-09-10T15:32:25.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:24.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:24.691356+0000) 2022-09-10T15:32:25.693 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:25.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:25.321063+0000) 2022-09-10T15:32:26.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:25 smithi104 ceph-mon[97044]: pgmap v21219: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:32:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:25.690+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:25.691523+0000) 2022-09-10T15:32:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:25 smithi110 ceph-mon[97633]: pgmap v21219: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:32:26.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:26.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:26.321261+0000) 2022-09-10T15:32:27.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:26.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:26.691763+0000) 2022-09-10T15:32:27.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:27.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:27.321472+0000) 2022-09-10T15:32:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:27 smithi104 ceph-mon[97044]: pgmap v21220: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:27.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:27.691990+0000) 2022-09-10T15:32:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:27 smithi110 ceph-mon[97633]: pgmap v21220: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:28.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:28.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:28.321723+0000) 2022-09-10T15:32:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:28.691+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:28.692256+0000) 2022-09-10T15:32:29.694 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:29.320+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:29.321861+0000) 2022-09-10T15:32:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:29 smithi104 ceph-mon[97044]: pgmap v21221: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:29.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:29.692483+0000) 2022-09-10T15:32:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:29 smithi110 ceph-mon[97633]: pgmap v21221: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:30.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:30.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:30.322006+0000) 2022-09-10T15:32:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:30.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:30.692731+0000) 2022-09-10T15:32:31.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:31.321+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:31.322221+0000) 2022-09-10T15:32:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:31 smithi104 ceph-mon[97044]: pgmap v21222: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:31.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:31.692977+0000) 2022-09-10T15:32:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:31 smithi110 ceph-mon[97633]: pgmap v21222: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:32.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:32.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:32.322445+0000) 2022-09-10T15:32:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:32.692+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:32.693175+0000) 2022-09-10T15:32:33.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:33.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:33.322722+0000) 2022-09-10T15:32:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:33 smithi104 ceph-mon[97044]: pgmap v21223: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:33.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:33.693386+0000) 2022-09-10T15:32:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:33 smithi110 ceph-mon[97633]: pgmap v21223: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:34.695 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:34.322+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:34.322944+0000) 2022-09-10T15:32:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:34.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:34.693634+0000) 2022-09-10T15:32:35.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:35.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:35.323191+0000) 2022-09-10T15:32:36.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:35 smithi104 ceph-mon[97044]: pgmap v21224: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:35.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:35.693863+0000) 2022-09-10T15:32:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:35 smithi110 ceph-mon[97633]: pgmap v21224: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:36.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:36.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:36.323338+0000) 2022-09-10T15:32:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:36.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:36.694050+0000) 2022-09-10T15:32:37.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:37.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:37.323554+0000) 2022-09-10T15:32:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:37 smithi104 ceph-mon[97044]: pgmap v21225: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:37.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:37.694313+0000) 2022-09-10T15:32:38.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:32:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:32:37.867+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:32:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:37 smithi110 ceph-mon[97633]: pgmap v21225: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:38.696 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:38.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:38.323818+0000) 2022-09-10T15:32:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:38.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:38.694569+0000) 2022-09-10T15:32:39.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:39.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:39.324081+0000) 2022-09-10T15:32:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:39 smithi104 ceph-mon[97044]: pgmap v21226: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:39.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:39.694767+0000) 2022-09-10T15:32:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:39 smithi110 ceph-mon[97633]: pgmap v21226: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:40.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:40.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:40.324302+0000) 2022-09-10T15:32:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:40.693+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:40.695002+0000) 2022-09-10T15:32:41.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:41.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:41.324552+0000) 2022-09-10T15:32:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:41 smithi104 ceph-mon[97044]: pgmap v21227: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:41.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:41.695232+0000) 2022-09-10T15:32:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:41 smithi110 ceph-mon[97633]: pgmap v21227: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:42.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:42.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:42.324722+0000) 2022-09-10T15:32:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:42.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:42.695461+0000) 2022-09-10T15:32:43.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:43.323+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:43.324950+0000) 2022-09-10T15:32:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:43 smithi104 ceph-mon[97044]: pgmap v21228: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:43.694+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:43.695707+0000) 2022-09-10T15:32:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:43 smithi110 ceph-mon[97633]: pgmap v21228: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:44.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:44.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:44.325151+0000) 2022-09-10T15:32:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:44.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:44.695944+0000) 2022-09-10T15:32:45.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:45.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:45.325351+0000) 2022-09-10T15:32:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:45 smithi104 ceph-mon[97044]: pgmap v21229: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:45.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:45.696229+0000) 2022-09-10T15:32:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:45 smithi110 ceph-mon[97633]: pgmap v21229: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:46.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:46.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:46.325537+0000) 2022-09-10T15:32:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:46.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:46.696406+0000) 2022-09-10T15:32:47.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:47.324+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:47.325738+0000) 2022-09-10T15:32:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:47 smithi104 ceph-mon[97044]: pgmap v21230: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:32:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:32:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:32:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:32:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:47.695+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:47.696675+0000) 2022-09-10T15:32:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:47 smithi110 ceph-mon[97633]: pgmap v21230: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:32:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:32:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:32:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:32:48.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:48.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:48.325921+0000) 2022-09-10T15:32:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:48.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:48.696914+0000) 2022-09-10T15:32:49.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:49.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:49.326201+0000) 2022-09-10T15:32:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:49 smithi104 ceph-mon[97044]: pgmap v21231: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:49.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:49.697175+0000) 2022-09-10T15:32:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:49 smithi110 ceph-mon[97633]: pgmap v21231: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:50.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:50.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:50.326373+0000) 2022-09-10T15:32:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:50.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:50.697411+0000) 2022-09-10T15:32:51.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:51.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:51.326616+0000) 2022-09-10T15:32:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:51 smithi104 ceph-mon[97044]: pgmap v21232: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:51.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:51.697646+0000) 2022-09-10T15:32:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:51 smithi110 ceph-mon[97633]: pgmap v21232: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:52.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:52.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:52.326826+0000) 2022-09-10T15:32:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:32:52.881+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:32:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:52.696+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:52.697802+0000) 2022-09-10T15:32:53.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:53.325+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:53.327049+0000) 2022-09-10T15:32:54.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:53 smithi104 ceph-mon[97044]: pgmap v21233: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:53.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:53.698032+0000) 2022-09-10T15:32:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:53 smithi110 ceph-mon[97633]: pgmap v21233: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:54.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:54.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:54.327278+0000) 2022-09-10T15:32:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:54.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:54.698265+0000) 2022-09-10T15:32:55.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:55.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:55.327511+0000) 2022-09-10T15:32:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:55 smithi104 ceph-mon[97044]: pgmap v21234: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:55.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:55.698496+0000) 2022-09-10T15:32:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:55 smithi110 ceph-mon[97633]: pgmap v21234: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:32:56.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:56.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:56.327758+0000) 2022-09-10T15:32:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:56.697+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:56.698747+0000) 2022-09-10T15:32:57.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:57.326+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:57.327989+0000) 2022-09-10T15:32:58.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:57 smithi104 ceph-mon[97044]: pgmap v21235: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:57.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:57.698971+0000) 2022-09-10T15:32:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:57 smithi110 ceph-mon[97633]: pgmap v21235: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:32:58.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:58.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:58.328209+0000) 2022-09-10T15:32:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:58.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:58.699136+0000) 2022-09-10T15:32:59.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:32:59.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:59.328380+0000) 2022-09-10T15:33:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:32:59 smithi104 ceph-mon[97044]: pgmap v21236: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:32:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:32:59.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:32:59.699334+0000) 2022-09-10T15:33:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:32:59 smithi110 ceph-mon[97633]: pgmap v21236: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:00.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:00.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:00.328613+0000) 2022-09-10T15:33:01.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:00.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:00.699507+0000) 2022-09-10T15:33:01.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:01.327+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:01.328820+0000) 2022-09-10T15:33:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:01 smithi104 ceph-mon[97044]: pgmap v21237: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:01.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:01.699700+0000) 2022-09-10T15:33:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:01 smithi110 ceph-mon[97633]: pgmap v21237: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:02.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:02.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:02.328991+0000) 2022-09-10T15:33:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:02.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:02.699889+0000) 2022-09-10T15:33:03.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:03.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:03.329285+0000) 2022-09-10T15:33:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:03 smithi104 ceph-mon[97044]: pgmap v21238: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:03.698+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:03.700047+0000) 2022-09-10T15:33:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:03 smithi110 ceph-mon[97633]: pgmap v21238: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:04.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:04.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:04.329420+0000) 2022-09-10T15:33:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:04.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:04.700244+0000) 2022-09-10T15:33:05.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:05.328+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:05.329617+0000) 2022-09-10T15:33:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:05 smithi104 ceph-mon[97044]: pgmap v21239: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:33:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:05.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:05.700476+0000) 2022-09-10T15:33:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:05 smithi110 ceph-mon[97633]: pgmap v21239: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:33:06.702 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:06.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:06.329850+0000) 2022-09-10T15:33:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:06.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:06.700706+0000) 2022-09-10T15:33:07.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:07.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:07.330067+0000) 2022-09-10T15:33:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:07 smithi104 ceph-mon[97044]: pgmap v21240: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:33:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:33:07.889+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:33:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:07.699+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:07.700960+0000) 2022-09-10T15:33:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:07 smithi110 ceph-mon[97633]: pgmap v21240: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:33:08.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:08.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:08.330275+0000) 2022-09-10T15:33:09.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:08.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:08.701214+0000) 2022-09-10T15:33:09.703 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:09.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:09.330460+0000) 2022-09-10T15:33:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:09 smithi104 ceph-mon[97044]: pgmap v21241: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:33:10.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:09 smithi104 ceph-mon[97044]: osdmap e1408: 12 total, 12 up, 12 in 2022-09-10T15:33:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:09.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:09.701441+0000) 2022-09-10T15:33:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:09 smithi110 ceph-mon[97633]: pgmap v21241: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:33:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:09 smithi110 ceph-mon[97633]: osdmap e1408: 12 total, 12 up, 12 in 2022-09-10T15:33:10.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:10.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:10.330690+0000) 2022-09-10T15:33:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:10.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:10.701703+0000) 2022-09-10T15:33:11.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:11.329+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:11.330937+0000) 2022-09-10T15:33:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:11 smithi104 ceph-mon[97044]: pgmap v21243: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:33:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:11.700+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:11.701902+0000) 2022-09-10T15:33:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:11 smithi110 ceph-mon[97633]: pgmap v21243: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:33:12.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:12.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:12.331172+0000) 2022-09-10T15:33:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:12.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:12.702085+0000) 2022-09-10T15:33:13.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:13.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:13.331415+0000) 2022-09-10T15:33:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:13 smithi104 ceph-mon[97044]: pgmap v21244: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:33:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:13 smithi104 ceph-mon[97044]: osdmap e1409: 12 total, 12 up, 12 in 2022-09-10T15:33:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:13.701+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:13.702312+0000) 2022-09-10T15:33:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:13 smithi110 ceph-mon[97633]: pgmap v21244: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:33:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:13 smithi110 ceph-mon[97633]: osdmap e1409: 12 total, 12 up, 12 in 2022-09-10T15:33:14.704 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:14.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:14.331608+0000) 2022-09-10T15:33:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:14.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:14.702559+0000) 2022-09-10T15:33:15.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:15.330+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:15.331818+0000) 2022-09-10T15:33:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:15 smithi104 ceph-mon[97044]: pgmap v21246: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:33:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:15.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:15.702779+0000) 2022-09-10T15:33:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:15 smithi110 ceph-mon[97633]: pgmap v21246: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:33:16.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:16.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:16.332049+0000) 2022-09-10T15:33:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:16.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:16.703011+0000) 2022-09-10T15:33:17.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:17.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:17.332302+0000) 2022-09-10T15:33:18.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:17 smithi104 ceph-mon[97044]: pgmap v21247: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:33:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:17.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:17.703235+0000) 2022-09-10T15:33:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:17 smithi110 ceph-mon[97633]: pgmap v21247: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:33:18.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:18.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:18.332548+0000) 2022-09-10T15:33:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:18.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:18.703468+0000) 2022-09-10T15:33:19.705 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:19.331+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:19.332731+0000) 2022-09-10T15:33:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:19 smithi104 ceph-mon[97044]: pgmap v21248: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:33:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:19.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:19.703673+0000) 2022-09-10T15:33:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:19 smithi110 ceph-mon[97633]: pgmap v21248: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:33:20.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:20.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:20.332942+0000) 2022-09-10T15:33:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:20.702+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:20.703858+0000) 2022-09-10T15:33:21.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:21.332+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:21.333210+0000) 2022-09-10T15:33:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:21 smithi104 ceph-mon[97044]: pgmap v21249: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:33:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:21.703+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:21.704084+0000) 2022-09-10T15:33:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:21 smithi110 ceph-mon[97633]: pgmap v21249: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:33:22.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:22.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:22.333441+0000) 2022-09-10T15:33:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:22.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:22.704343+0000) 2022-09-10T15:33:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:33:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:33:22.890+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:33:23.706 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:23.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:23.333603+0000) 2022-09-10T15:33:24.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:23 smithi104 ceph-mon[97044]: pgmap v21250: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:33:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:23.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:23.704507+0000) 2022-09-10T15:33:24.164 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:23 smithi110 ceph-mon[97633]: pgmap v21250: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:33:24.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:24.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:24.333801+0000) 2022-09-10T15:33:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:24.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:24.704712+0000) 2022-09-10T15:33:25.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:25.333+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:25.334007+0000) 2022-09-10T15:33:26.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:25 smithi104 ceph-mon[97044]: pgmap v21251: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:33:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:25.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:25.704959+0000) 2022-09-10T15:33:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:25 smithi110 ceph-mon[97633]: pgmap v21251: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:33:26.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:26.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:26.334258+0000) 2022-09-10T15:33:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:26.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:26.705210+0000) 2022-09-10T15:33:27.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:27.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:27.334523+0000) 2022-09-10T15:33:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:27 smithi104 ceph-mon[97044]: pgmap v21252: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:27.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:27.705398+0000) 2022-09-10T15:33:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:27 smithi110 ceph-mon[97633]: pgmap v21252: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:28.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:28.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:28.334769+0000) 2022-09-10T15:33:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:28.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:28.705569+0000) 2022-09-10T15:33:29.707 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:29.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:29.335001+0000) 2022-09-10T15:33:30.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:29 smithi104 ceph-mon[97044]: pgmap v21253: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:29.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:29.705692+0000) 2022-09-10T15:33:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:29 smithi110 ceph-mon[97633]: pgmap v21253: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:30.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:30.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:30.335167+0000) 2022-09-10T15:33:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:30.704+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:30.705941+0000) 2022-09-10T15:33:31.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:31.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:31.335379+0000) 2022-09-10T15:33:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:31 smithi104 ceph-mon[97044]: pgmap v21254: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:31.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:31.706171+0000) 2022-09-10T15:33:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:31 smithi110 ceph-mon[97633]: pgmap v21254: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:32.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:32.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:32.335559+0000) 2022-09-10T15:33:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:32.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:32.706382+0000) 2022-09-10T15:33:33.708 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:33.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:33.335809+0000) 2022-09-10T15:33:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:33 smithi104 ceph-mon[97044]: pgmap v21255: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:33.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:33.706587+0000) 2022-09-10T15:33:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:33 smithi110 ceph-mon[97633]: pgmap v21255: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:34.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:34.334+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:34.336014+0000) 2022-09-10T15:33:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:34.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:34.706828+0000) 2022-09-10T15:33:35.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:35.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:35.336242+0000) 2022-09-10T15:33:36.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:35 smithi104 ceph-mon[97044]: pgmap v21256: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:35.705+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:35.707060+0000) 2022-09-10T15:33:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:35 smithi110 ceph-mon[97633]: pgmap v21256: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:36.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:36.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:36.336461+0000) 2022-09-10T15:33:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:36.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:36.707289+0000) 2022-09-10T15:33:37.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:37.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:37.336681+0000) 2022-09-10T15:33:38.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:37 smithi104 ceph-mon[97044]: pgmap v21257: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:38.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:33:37.903+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:33:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:37.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:37.707463+0000) 2022-09-10T15:33:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:37 smithi110 ceph-mon[97633]: pgmap v21257: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:38.709 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:38.335+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:38.336926+0000) 2022-09-10T15:33:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:38.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:38.707667+0000) 2022-09-10T15:33:39.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:39.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:39.337169+0000) 2022-09-10T15:33:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:39 smithi104 ceph-mon[97044]: pgmap v21258: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:39.706+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:39.707885+0000) 2022-09-10T15:33:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:39 smithi110 ceph-mon[97633]: pgmap v21258: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:40.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:40.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:40.337392+0000) 2022-09-10T15:33:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:40.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:40.708081+0000) 2022-09-10T15:33:41.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:41.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:41.337596+0000) 2022-09-10T15:33:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:41 smithi104 ceph-mon[97044]: pgmap v21259: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:41.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:41.708265+0000) 2022-09-10T15:33:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:41 smithi110 ceph-mon[97633]: pgmap v21259: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:42.710 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:42.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:42.337810+0000) 2022-09-10T15:33:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:42.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:42.708447+0000) 2022-09-10T15:33:43.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:43.336+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:43.338040+0000) 2022-09-10T15:33:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:43 smithi104 ceph-mon[97044]: pgmap v21260: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:43.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:43.708721+0000) 2022-09-10T15:33:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:43 smithi110 ceph-mon[97633]: pgmap v21260: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:44.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:44.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:44.338251+0000) 2022-09-10T15:33:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:44.707+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:44.708945+0000) 2022-09-10T15:33:45.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:45.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:45.338454+0000) 2022-09-10T15:33:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:45 smithi104 ceph-mon[97044]: pgmap v21261: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:45.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:45.709189+0000) 2022-09-10T15:33:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:45 smithi110 ceph-mon[97633]: pgmap v21261: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:46.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:46.337+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:46.338656+0000) 2022-09-10T15:33:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:46.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:46.709418+0000) 2022-09-10T15:33:47.711 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:47.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:47.338910+0000) 2022-09-10T15:33:48.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:47 smithi104 ceph-mon[97044]: pgmap v21262: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:33:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:33:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:33:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:33:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:47.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:47.709649+0000) 2022-09-10T15:33:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:47 smithi110 ceph-mon[97633]: pgmap v21262: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:33:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:33:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:33:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:33:48.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:48.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:48.339075+0000) 2022-09-10T15:33:49.061 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:48.708+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:48.709815+0000) 2022-09-10T15:33:49.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:49.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:49.339285+0000) 2022-09-10T15:33:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:49 smithi104 ceph-mon[97044]: pgmap v21263: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:49.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:49.710057+0000) 2022-09-10T15:33:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:49 smithi110 ceph-mon[97633]: pgmap v21263: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:50.712 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:50.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:50.339487+0000) 2022-09-10T15:33:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:50.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:50.710253+0000) 2022-09-10T15:33:51.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:51.338+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:51.339724+0000) 2022-09-10T15:33:52.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:51 smithi104 ceph-mon[97044]: pgmap v21264: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:51.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:51.710489+0000) 2022-09-10T15:33:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:51 smithi110 ceph-mon[97633]: pgmap v21264: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:52.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:52.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:52.339949+0000) 2022-09-10T15:33:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:52.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:52.710719+0000) 2022-09-10T15:33:53.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:33:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:33:52.914+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:33:53.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:53.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:53.340209+0000) 2022-09-10T15:33:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:53 smithi104 ceph-mon[97044]: pgmap v21265: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:53.709+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:53.710878+0000) 2022-09-10T15:33:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:53 smithi110 ceph-mon[97633]: pgmap v21265: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:54.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:54.339+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:54.340446+0000) 2022-09-10T15:33:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:54.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:54.711172+0000) 2022-09-10T15:33:55.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:55.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:55.340628+0000) 2022-09-10T15:33:56.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:55 smithi104 ceph-mon[97044]: pgmap v21266: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:55.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:55.711365+0000) 2022-09-10T15:33:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:55 smithi110 ceph-mon[97633]: pgmap v21266: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:33:56.713 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:56.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:56.340857+0000) 2022-09-10T15:33:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:56.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:56.711599+0000) 2022-09-10T15:33:57.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:57.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:57.341156+0000) 2022-09-10T15:33:58.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:57 smithi104 ceph-mon[97044]: pgmap v21267: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:57.710+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:57.711874+0000) 2022-09-10T15:33:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:57 smithi110 ceph-mon[97633]: pgmap v21267: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:33:58.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:58.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:58.341362+0000) 2022-09-10T15:33:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:58.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:58.712076+0000) 2022-09-10T15:33:59.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:33:59.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:59.341560+0000) 2022-09-10T15:34:00.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:33:59 smithi104 ceph-mon[97044]: pgmap v21268: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:33:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:33:59.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:33:59.712244+0000) 2022-09-10T15:34:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:33:59 smithi110 ceph-mon[97633]: pgmap v21268: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:00.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:00.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:00.341791+0000) 2022-09-10T15:34:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:00.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:00.712426+0000) 2022-09-10T15:34:01.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:01.340+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:01.341924+0000) 2022-09-10T15:34:02.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:01 smithi104 ceph-mon[97044]: pgmap v21269: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:01.711+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:01.712577+0000) 2022-09-10T15:34:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:01 smithi110 ceph-mon[97633]: pgmap v21269: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:02.714 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:02.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:02.342185+0000) 2022-09-10T15:34:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:02.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:02.712720+0000) 2022-09-10T15:34:03.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:03.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:03.342392+0000) 2022-09-10T15:34:04.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:03 smithi104 ceph-mon[97044]: pgmap v21270: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:03.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:03.712916+0000) 2022-09-10T15:34:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:03 smithi110 ceph-mon[97633]: pgmap v21270: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:04.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:04.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:04.342617+0000) 2022-09-10T15:34:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:04.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:04.713090+0000) 2022-09-10T15:34:05.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:05.341+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:05.342871+0000) 2022-09-10T15:34:06.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:05 smithi104 ceph-mon[97044]: pgmap v21271: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:34:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:05.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:05.713294+0000) 2022-09-10T15:34:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:05 smithi110 ceph-mon[97633]: pgmap v21271: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:34:06.715 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:06.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:06.343070+0000) 2022-09-10T15:34:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:06.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:06.713483+0000) 2022-09-10T15:34:07.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:07.342+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:07.343295+0000) 2022-09-10T15:34:08.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:07 smithi104 ceph-mon[97044]: pgmap v21272: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:34:08.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:34:07.928+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:34:08.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:07.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:07.713722+0000) 2022-09-10T15:34:08.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:07 smithi110 ceph-mon[97633]: pgmap v21272: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:34:08.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:08.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:08.343492+0000) 2022-09-10T15:34:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:08.712+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:08.713924+0000) 2022-09-10T15:34:09.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:09.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:09.343729+0000) 2022-09-10T15:34:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:09 smithi104 ceph-mon[97044]: pgmap v21273: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:34:10.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:09 smithi104 ceph-mon[97044]: osdmap e1410: 12 total, 12 up, 12 in 2022-09-10T15:34:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:09.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:09.714198+0000) 2022-09-10T15:34:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:09 smithi110 ceph-mon[97633]: pgmap v21273: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:34:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:09 smithi110 ceph-mon[97633]: osdmap e1410: 12 total, 12 up, 12 in 2022-09-10T15:34:10.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:10.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:10.343901+0000) 2022-09-10T15:34:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:10.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:10.714363+0000) 2022-09-10T15:34:11.716 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:11.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:11.344185+0000) 2022-09-10T15:34:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:11 smithi104 ceph-mon[97044]: pgmap v21275: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:34:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:11.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:11.714618+0000) 2022-09-10T15:34:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:11 smithi110 ceph-mon[97633]: pgmap v21275: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:34:12.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:12.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:12.344399+0000) 2022-09-10T15:34:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:12.713+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:12.714787+0000) 2022-09-10T15:34:13.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:13.344+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:13.344596+0000) 2022-09-10T15:34:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:13 smithi104 ceph-mon[97044]: pgmap v21276: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:34:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:13.714+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:13.715047+0000) 2022-09-10T15:34:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:13 smithi110 ceph-mon[97633]: pgmap v21276: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:34:14.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:14.343+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:14.344833+0000) 2022-09-10T15:34:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:14.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:14.715280+0000) 2022-09-10T15:34:15.717 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:15.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:15.345089+0000) 2022-09-10T15:34:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:15 smithi104 ceph-mon[97044]: pgmap v21277: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:34:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:15.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:15.715469+0000) 2022-09-10T15:34:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:15 smithi110 ceph-mon[97633]: pgmap v21277: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:34:16.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:16.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:16.345369+0000) 2022-09-10T15:34:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:16.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:16.715640+0000) 2022-09-10T15:34:17.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:17.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:17.345645+0000) 2022-09-10T15:34:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:17 smithi104 ceph-mon[97044]: pgmap v21278: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:34:18.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:17 smithi104 ceph-mon[97044]: osdmap e1411: 12 total, 12 up, 12 in 2022-09-10T15:34:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:17.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:17.715901+0000) 2022-09-10T15:34:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:17 smithi110 ceph-mon[97633]: pgmap v21278: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:34:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:17 smithi110 ceph-mon[97633]: osdmap e1411: 12 total, 12 up, 12 in 2022-09-10T15:34:18.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:18.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:18.345887+0000) 2022-09-10T15:34:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:18.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:18.716083+0000) 2022-09-10T15:34:19.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:19.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:19.346152+0000) 2022-09-10T15:34:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:19 smithi104 ceph-mon[97044]: pgmap v21280: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:34:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:19.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:19.716298+0000) 2022-09-10T15:34:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:19 smithi110 ceph-mon[97633]: pgmap v21280: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:34:20.718 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:20.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:20.346387+0000) 2022-09-10T15:34:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:20.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:20.716470+0000) 2022-09-10T15:34:21.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:21.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:21.346526+0000) 2022-09-10T15:34:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:21 smithi104 ceph-mon[97044]: pgmap v21281: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:34:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:21.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:21.716737+0000) 2022-09-10T15:34:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:21 smithi110 ceph-mon[97633]: pgmap v21281: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:34:22.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:22.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:22.346681+0000) 2022-09-10T15:34:23.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:34:22.933+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:34:23.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:22.715+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:22.716992+0000) 2022-09-10T15:34:23.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:23.345+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:23.346928+0000) 2022-09-10T15:34:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:23.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:23.717208+0000) 2022-09-10T15:34:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:23 smithi110 ceph-mon[97633]: pgmap v21282: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:34:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:23 smithi104 ceph-mon[97044]: pgmap v21282: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:34:24.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:24.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:24.347177+0000) 2022-09-10T15:34:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:24.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:24.717433+0000) 2022-09-10T15:34:25.719 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:25.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:25.347318+0000) 2022-09-10T15:34:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:25.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:25.717675+0000) 2022-09-10T15:34:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:25 smithi110 ceph-mon[97633]: pgmap v21283: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:34:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:25 smithi104 ceph-mon[97044]: pgmap v21283: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:34:26.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:26.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:26.347515+0000) 2022-09-10T15:34:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:26.716+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:26.717870+0000) 2022-09-10T15:34:27.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:27.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:27.347690+0000) 2022-09-10T15:34:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:27.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:27.718070+0000) 2022-09-10T15:34:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:27 smithi110 ceph-mon[97633]: pgmap v21284: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:34:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:27 smithi104 ceph-mon[97044]: pgmap v21284: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:34:28.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:28.346+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:28.347852+0000) 2022-09-10T15:34:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:28.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:28.718326+0000) 2022-09-10T15:34:29.720 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:29.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:29.348147+0000) 2022-09-10T15:34:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:29.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:29.718557+0000) 2022-09-10T15:34:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:29 smithi110 ceph-mon[97633]: pgmap v21285: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:34:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:29 smithi104 ceph-mon[97044]: pgmap v21285: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:34:30.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:30.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:30.348426+0000) 2022-09-10T15:34:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:30.717+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:30.718805+0000) 2022-09-10T15:34:31.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:31.347+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:31.348682+0000) 2022-09-10T15:34:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:31.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:31.719037+0000) 2022-09-10T15:34:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:31 smithi110 ceph-mon[97633]: pgmap v21286: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:31 smithi104 ceph-mon[97044]: pgmap v21286: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:32.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:32.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:32.348913+0000) 2022-09-10T15:34:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:32.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:32.719179+0000) 2022-09-10T15:34:33.721 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:33.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:33.349186+0000) 2022-09-10T15:34:34.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:33.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:33.719380+0000) 2022-09-10T15:34:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:33 smithi110 ceph-mon[97633]: pgmap v21287: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:34:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:33 smithi104 ceph-mon[97044]: pgmap v21287: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:34:34.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:34.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:34.349407+0000) 2022-09-10T15:34:35.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:34.718+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:34.719620+0000) 2022-09-10T15:34:35.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:35.348+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:35.349628+0000) 2022-09-10T15:34:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:35.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:35.719828+0000) 2022-09-10T15:34:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:35 smithi110 ceph-mon[97633]: pgmap v21288: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:35 smithi104 ceph-mon[97044]: pgmap v21288: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:36.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:36.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:36.349886+0000) 2022-09-10T15:34:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:36.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:36.720012+0000) 2022-09-10T15:34:37.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:37.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:37.350191+0000) 2022-09-10T15:34:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:37.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:37.720315+0000) 2022-09-10T15:34:38.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:34:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:34:37.937+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:34:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:37 smithi110 ceph-mon[97633]: pgmap v21289: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:34:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:37 smithi104 ceph-mon[97044]: pgmap v21289: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:34:38.722 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:38.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:38.350407+0000) 2022-09-10T15:34:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:38.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:38.720563+0000) 2022-09-10T15:34:39.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:39.349+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:39.350663+0000) 2022-09-10T15:34:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:39.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:39.720760+0000) 2022-09-10T15:34:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:39 smithi110 ceph-mon[97633]: pgmap v21290: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:34:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:39 smithi104 ceph-mon[97044]: pgmap v21290: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:34:40.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:40.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:40.350890+0000) 2022-09-10T15:34:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:40.719+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:40.720957+0000) 2022-09-10T15:34:41.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:41.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:41.351168+0000) 2022-09-10T15:34:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:41.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:41.721199+0000) 2022-09-10T15:34:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:41 smithi110 ceph-mon[97633]: pgmap v21291: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:41 smithi104 ceph-mon[97044]: pgmap v21291: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:42.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:42.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:42.351326+0000) 2022-09-10T15:34:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:42.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:42.721415+0000) 2022-09-10T15:34:43.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:43.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:43.351504+0000) 2022-09-10T15:34:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:43.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:43.721616+0000) 2022-09-10T15:34:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:43 smithi110 ceph-mon[97633]: pgmap v21292: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:43 smithi104 ceph-mon[97044]: pgmap v21292: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:44.723 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:44.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:44.351678+0000) 2022-09-10T15:34:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:44.720+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:44.721819+0000) 2022-09-10T15:34:45.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:45.350+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:45.351887+0000) 2022-09-10T15:34:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:45.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:45.722037+0000) 2022-09-10T15:34:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:45 smithi110 ceph-mon[97633]: pgmap v21293: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:45 smithi104 ceph-mon[97044]: pgmap v21293: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:46.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:46.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:46.352137+0000) 2022-09-10T15:34:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:46.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:46.722326+0000) 2022-09-10T15:34:47.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:47.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:47.352337+0000) 2022-09-10T15:34:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:47.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:47.722487+0000) 2022-09-10T15:34:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:47 smithi110 ceph-mon[97633]: pgmap v21294: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:34:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:34:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:34:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:34:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:47 smithi104 ceph-mon[97044]: pgmap v21294: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:34:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:34:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:34:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:34:48.724 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:48.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:48.352513+0000) 2022-09-10T15:34:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:48.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:48.722672+0000) 2022-09-10T15:34:49.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:49.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:49.352731+0000) 2022-09-10T15:34:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:49.721+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:49.722840+0000) 2022-09-10T15:34:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:49 smithi110 ceph-mon[97633]: pgmap v21295: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:49 smithi104 ceph-mon[97044]: pgmap v21295: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:50.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:50.351+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:50.352877+0000) 2022-09-10T15:34:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:50.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:50.723050+0000) 2022-09-10T15:34:51.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:51.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:51.353129+0000) 2022-09-10T15:34:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:51.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:51.723227+0000) 2022-09-10T15:34:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:51 smithi110 ceph-mon[97633]: pgmap v21296: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:51 smithi104 ceph-mon[97044]: pgmap v21296: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:52.725 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:52.352+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:52.353374+0000) 2022-09-10T15:34:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:34:52.939+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:34:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:52.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:52.723457+0000) 2022-09-10T15:34:53.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:53.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:53.353637+0000) 2022-09-10T15:34:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:53.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:53.723735+0000) 2022-09-10T15:34:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:53 smithi110 ceph-mon[97633]: pgmap v21297: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:53 smithi104 ceph-mon[97044]: pgmap v21297: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:54.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:54.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:54.353857+0000) 2022-09-10T15:34:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:54.722+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:54.723928+0000) 2022-09-10T15:34:55.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:55.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:55.354129+0000) 2022-09-10T15:34:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:55.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:55.724200+0000) 2022-09-10T15:34:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:55 smithi110 ceph-mon[97633]: pgmap v21298: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:55 smithi104 ceph-mon[97044]: pgmap v21298: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:34:56.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:56.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:56.354378+0000) 2022-09-10T15:34:57.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:56.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:56.724391+0000) 2022-09-10T15:34:57.726 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:57.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:57.354655+0000) 2022-09-10T15:34:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:57.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:57.724581+0000) 2022-09-10T15:34:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:57 smithi110 ceph-mon[97633]: pgmap v21299: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:57 smithi104 ceph-mon[97044]: pgmap v21299: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:34:58.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:58.353+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:58.354854+0000) 2022-09-10T15:34:59.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:58.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:58.724781+0000) 2022-09-10T15:34:59.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:34:59.354+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:59.355135+0000) 2022-09-10T15:35:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:34:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:34:59.723+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:34:59.724976+0000) 2022-09-10T15:35:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:34:59 smithi110 ceph-mon[97633]: pgmap v21300: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:34:59 smithi104 ceph-mon[97044]: pgmap v21300: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:00.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:00.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:00.355366+0000) 2022-09-10T15:35:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:00.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:00.725154+0000) 2022-09-10T15:35:01.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:01.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:01.355626+0000) 2022-09-10T15:35:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:01.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:01.725344+0000) 2022-09-10T15:35:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:01 smithi110 ceph-mon[97633]: pgmap v21301: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:01 smithi104 ceph-mon[97044]: pgmap v21301: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:02.727 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:02.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:02.355756+0000) 2022-09-10T15:35:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:02.724+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:02.725560+0000) 2022-09-10T15:35:03.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:03.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:03.355891+0000) 2022-09-10T15:35:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:03.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:03.725798+0000) 2022-09-10T15:35:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:03 smithi110 ceph-mon[97633]: pgmap v21302: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:03 smithi104 ceph-mon[97044]: pgmap v21302: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:04.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:04.355+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:04.356133+0000) 2022-09-10T15:35:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:04.725+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:04.725988+0000) 2022-09-10T15:35:05.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:05.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:05.356309+0000) 2022-09-10T15:35:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:05.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:05.726235+0000) 2022-09-10T15:35:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:05 smithi110 ceph-mon[97633]: pgmap v21303: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:35:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:05 smithi104 ceph-mon[97044]: pgmap v21303: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:35:06.728 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:06.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:06.356560+0000) 2022-09-10T15:35:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:06.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:06.726399+0000) 2022-09-10T15:35:07.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:07.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:07.356794+0000) 2022-09-10T15:35:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:07.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:07.726648+0000) 2022-09-10T15:35:08.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:35:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:35:07.949+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:35:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:07 smithi110 ceph-mon[97633]: pgmap v21304: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:35:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:07 smithi104 ceph-mon[97044]: pgmap v21304: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:35:08.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:08.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:08.357024+0000) 2022-09-10T15:35:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:08.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:08.726911+0000) 2022-09-10T15:35:09.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:09.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:09.357259+0000) 2022-09-10T15:35:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:09.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:09.727176+0000) 2022-09-10T15:35:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:09 smithi110 ceph-mon[97633]: pgmap v21305: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:35:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:09 smithi110 ceph-mon[97633]: osdmap e1412: 12 total, 12 up, 12 in 2022-09-10T15:35:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:09 smithi104 ceph-mon[97044]: pgmap v21305: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:35:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:09 smithi104 ceph-mon[97044]: osdmap e1412: 12 total, 12 up, 12 in 2022-09-10T15:35:10.729 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:10.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:10.357500+0000) 2022-09-10T15:35:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:10.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:10.727420+0000) 2022-09-10T15:35:11.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:11.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:11.357760+0000) 2022-09-10T15:35:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:11.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:11.727673+0000) 2022-09-10T15:35:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:11 smithi110 ceph-mon[97633]: pgmap v21307: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:35:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:11 smithi104 ceph-mon[97044]: pgmap v21307: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:35:12.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:12.356+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:12.357999+0000) 2022-09-10T15:35:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:12.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:12.727913+0000) 2022-09-10T15:35:13.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:13.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:13.358227+0000) 2022-09-10T15:35:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:13.726+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:13.728080+0000) 2022-09-10T15:35:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:13 smithi110 ceph-mon[97633]: pgmap v21308: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:35:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:13 smithi104 ceph-mon[97044]: pgmap v21308: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:35:14.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:14.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:14.358444+0000) 2022-09-10T15:35:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:14.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:14.728335+0000) 2022-09-10T15:35:15.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:15.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:15.358658+0000) 2022-09-10T15:35:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:15.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:15.728552+0000) 2022-09-10T15:35:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:15 smithi110 ceph-mon[97633]: pgmap v21309: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:35:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:15 smithi104 ceph-mon[97044]: pgmap v21309: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:35:16.730 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:16.357+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:16.358898+0000) 2022-09-10T15:35:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:16.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:16.728795+0000) 2022-09-10T15:35:17.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:17.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:17.359166+0000) 2022-09-10T15:35:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:17.727+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:17.729003+0000) 2022-09-10T15:35:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:17 smithi110 ceph-mon[97633]: pgmap v21310: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:35:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:17 smithi110 ceph-mon[97633]: osdmap e1413: 12 total, 12 up, 12 in 2022-09-10T15:35:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:17 smithi104 ceph-mon[97044]: pgmap v21310: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:35:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:17 smithi104 ceph-mon[97044]: osdmap e1413: 12 total, 12 up, 12 in 2022-09-10T15:35:18.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:18.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:18.359445+0000) 2022-09-10T15:35:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:18.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:18.729253+0000) 2022-09-10T15:35:19.731 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:19.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:19.359668+0000) 2022-09-10T15:35:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:19.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:19.729469+0000) 2022-09-10T15:35:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:19 smithi110 ceph-mon[97633]: pgmap v21312: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:35:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:19 smithi104 ceph-mon[97044]: pgmap v21312: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:35:20.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:20.358+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:20.359905+0000) 2022-09-10T15:35:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:20.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:20.729670+0000) 2022-09-10T15:35:21.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:21.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:21.360206+0000) 2022-09-10T15:35:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:21.728+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:21.729842+0000) 2022-09-10T15:35:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:21 smithi110 ceph-mon[97633]: pgmap v21313: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:35:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:21 smithi104 ceph-mon[97044]: pgmap v21313: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:35:22.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:22.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:22.360405+0000) 2022-09-10T15:35:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:22.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:22.730062+0000) 2022-09-10T15:35:23.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:35:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:35:22.953+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:35:23.732 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:23.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:23.360633+0000) 2022-09-10T15:35:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:23.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:23.730348+0000) 2022-09-10T15:35:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:23 smithi110 ceph-mon[97633]: pgmap v21314: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:35:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:23 smithi104 ceph-mon[97044]: pgmap v21314: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:35:24.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:24.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:24.360864+0000) 2022-09-10T15:35:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:24.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:24.730544+0000) 2022-09-10T15:35:25.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:25.359+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:25.361049+0000) 2022-09-10T15:35:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:25.729+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:25.730790+0000) 2022-09-10T15:35:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:25 smithi110 ceph-mon[97633]: pgmap v21315: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:35:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:25 smithi104 ceph-mon[97044]: pgmap v21315: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:35:26.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:26.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:26.361221+0000) 2022-09-10T15:35:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:26.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:26.731031+0000) 2022-09-10T15:35:27.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:27.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:27.361456+0000) 2022-09-10T15:35:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:27.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:27.731267+0000) 2022-09-10T15:35:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:27 smithi110 ceph-mon[97633]: pgmap v21316: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:35:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:27 smithi104 ceph-mon[97044]: pgmap v21316: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:35:28.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:28.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:28.361658+0000) 2022-09-10T15:35:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:28.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:28.731487+0000) 2022-09-10T15:35:29.733 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:29.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:29.361859+0000) 2022-09-10T15:35:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:29.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:29.731663+0000) 2022-09-10T15:35:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:29 smithi110 ceph-mon[97633]: pgmap v21317: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:35:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:29 smithi104 ceph-mon[97044]: pgmap v21317: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:35:30.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:30.360+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:30.362051+0000) 2022-09-10T15:35:31.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:30.730+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:30.731824+0000) 2022-09-10T15:35:31.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:31.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:31.362261+0000) 2022-09-10T15:35:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:31.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:31.732036+0000) 2022-09-10T15:35:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:31 smithi110 ceph-mon[97633]: pgmap v21318: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:31 smithi104 ceph-mon[97044]: pgmap v21318: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:32.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:32.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:32.362443+0000) 2022-09-10T15:35:33.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:32.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:32.732231+0000) 2022-09-10T15:35:33.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:33.361+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:33.362647+0000) 2022-09-10T15:35:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:33.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:33.732407+0000) 2022-09-10T15:35:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:33 smithi110 ceph-mon[97633]: pgmap v21319: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:33 smithi104 ceph-mon[97044]: pgmap v21319: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:34.734 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:34.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:34.362874+0000) 2022-09-10T15:35:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:34.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:34.732591+0000) 2022-09-10T15:35:35.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:35.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:35.363148+0000) 2022-09-10T15:35:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:35.731+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:35.732834+0000) 2022-09-10T15:35:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:35 smithi104 ceph-mon[97044]: pgmap v21320: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:35 smithi110 ceph-mon[97633]: pgmap v21320: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:36.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:36.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:36.363361+0000) 2022-09-10T15:35:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:36.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:36.733067+0000) 2022-09-10T15:35:37.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:37.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:37.363598+0000) 2022-09-10T15:35:38.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:37.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:37.733246+0000) 2022-09-10T15:35:38.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:35:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:35:37.965+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:35:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:37 smithi104 ceph-mon[97044]: pgmap v21321: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:37 smithi110 ceph-mon[97633]: pgmap v21321: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:38.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:38.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:38.363776+0000) 2022-09-10T15:35:39.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:38.732+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:38.733468+0000) 2022-09-10T15:35:39.735 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:39.362+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:39.363955+0000) 2022-09-10T15:35:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:39.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:39.733668+0000) 2022-09-10T15:35:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:39 smithi104 ceph-mon[97044]: pgmap v21322: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:39 smithi110 ceph-mon[97633]: pgmap v21322: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:40.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:40.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:40.364145+0000) 2022-09-10T15:35:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:40.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:40.733869+0000) 2022-09-10T15:35:41.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:41.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:41.364376+0000) 2022-09-10T15:35:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:41.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:41.734125+0000) 2022-09-10T15:35:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:41 smithi104 ceph-mon[97044]: pgmap v21323: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:41 smithi110 ceph-mon[97633]: pgmap v21323: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:42.736 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:42.363+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:42.364517+0000) 2022-09-10T15:35:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:42.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:42.734318+0000) 2022-09-10T15:35:43.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:43.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:43.364748+0000) 2022-09-10T15:35:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:43.733+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:43.734542+0000) 2022-09-10T15:35:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:43 smithi104 ceph-mon[97044]: pgmap v21324: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:43 smithi110 ceph-mon[97633]: pgmap v21324: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:44.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:44.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:44.364999+0000) 2022-09-10T15:35:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:44.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:44.734767+0000) 2022-09-10T15:35:45.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:45.364+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:45.365290+0000) 2022-09-10T15:35:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:45.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:45.735011+0000) 2022-09-10T15:35:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:45 smithi104 ceph-mon[97044]: pgmap v21325: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:45 smithi110 ceph-mon[97633]: pgmap v21325: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:46.737 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:46.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:46.365509+0000) 2022-09-10T15:35:47.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:46.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:46.735266+0000) 2022-09-10T15:35:47.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:47.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:47.365771+0000) 2022-09-10T15:35:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:47.734+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:47.735500+0000) 2022-09-10T15:35:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:47 smithi104 ceph-mon[97044]: pgmap v21326: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:35:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:35:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:35:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:35:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:47 smithi110 ceph-mon[97633]: pgmap v21326: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:35:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:35:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:35:48.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:35:48.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:48.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:48.366026+0000) 2022-09-10T15:35:49.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:48.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:48.735760+0000) 2022-09-10T15:35:49.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:49.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:49.366225+0000) 2022-09-10T15:35:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:49.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:49.735939+0000) 2022-09-10T15:35:50.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:49 smithi104 ceph-mon[97044]: pgmap v21327: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:49 smithi110 ceph-mon[97633]: pgmap v21327: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:50.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:50.365+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:50.366368+0000) 2022-09-10T15:35:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:50.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:50.736190+0000) 2022-09-10T15:35:51.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:51.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:51.366593+0000) 2022-09-10T15:35:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:51.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:51.736391+0000) 2022-09-10T15:35:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:51 smithi104 ceph-mon[97044]: pgmap v21328: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:51 smithi110 ceph-mon[97633]: pgmap v21328: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:52.738 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:52.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:52.366840+0000) 2022-09-10T15:35:53.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:35:52.974+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:35:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:52.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:52.736536+0000) 2022-09-10T15:35:53.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:53.366+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:53.367045+0000) 2022-09-10T15:35:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:53.735+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:53.736667+0000) 2022-09-10T15:35:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:53 smithi104 ceph-mon[97044]: pgmap v21329: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:53 smithi110 ceph-mon[97633]: pgmap v21329: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:54.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:54.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:54.367267+0000) 2022-09-10T15:35:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:54.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:54.736925+0000) 2022-09-10T15:35:55.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:55.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:55.367515+0000) 2022-09-10T15:35:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:55.736+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:55.737206+0000) 2022-09-10T15:35:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:55 smithi104 ceph-mon[97044]: pgmap v21330: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:55 smithi110 ceph-mon[97633]: pgmap v21330: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:35:56.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:56.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:56.367751+0000) 2022-09-10T15:35:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:56.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:56.737360+0000) 2022-09-10T15:35:57.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:57.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:57.367977+0000) 2022-09-10T15:35:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:57.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:57.737583+0000) 2022-09-10T15:35:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:57 smithi104 ceph-mon[97044]: pgmap v21331: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:57 smithi110 ceph-mon[97633]: pgmap v21331: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:35:58.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:58.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:58.368212+0000) 2022-09-10T15:35:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:58.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:58.737746+0000) 2022-09-10T15:35:59.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:35:59.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:59.368469+0000) 2022-09-10T15:36:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:35:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:35:59.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:35:59.737931+0000) 2022-09-10T15:36:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:35:59 smithi104 ceph-mon[97044]: pgmap v21332: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:35:59 smithi110 ceph-mon[97633]: pgmap v21332: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:00.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:00.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:00.368639+0000) 2022-09-10T15:36:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:00.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:00.738123+0000) 2022-09-10T15:36:01.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:01.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:01.368861+0000) 2022-09-10T15:36:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:01.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:01.738373+0000) 2022-09-10T15:36:02.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:01 smithi104 ceph-mon[97044]: pgmap v21333: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:01 smithi110 ceph-mon[97633]: pgmap v21333: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:02.740 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:02.367+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:02.369059+0000) 2022-09-10T15:36:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:02.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:02.738511+0000) 2022-09-10T15:36:03.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:03.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:03.369294+0000) 2022-09-10T15:36:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:03.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:03.738735+0000) 2022-09-10T15:36:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:03 smithi104 ceph-mon[97044]: pgmap v21334: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:03 smithi110 ceph-mon[97633]: pgmap v21334: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:04.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:04.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:04.369432+0000) 2022-09-10T15:36:05.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:04.737+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:04.738953+0000) 2022-09-10T15:36:05.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:05.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:05.369555+0000) 2022-09-10T15:36:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:05.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:05.739197+0000) 2022-09-10T15:36:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:05 smithi104 ceph-mon[97044]: pgmap v21335: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:36:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:05 smithi110 ceph-mon[97633]: pgmap v21335: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:36:06.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:06.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:06.369776+0000) 2022-09-10T15:36:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:06.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:06.739430+0000) 2022-09-10T15:36:07.741 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:07.368+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:07.369983+0000) 2022-09-10T15:36:07.993 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:07.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:07.739658+0000) 2022-09-10T15:36:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:07 smithi104 ceph-mon[97044]: pgmap v21336: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:36:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:36:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:36:07.990+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:36:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:07 smithi110 ceph-mon[97633]: pgmap v21336: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:36:08.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:08.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:08.370245+0000) 2022-09-10T15:36:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:08.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:08.739859+0000) 2022-09-10T15:36:09.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:09.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:09.370403+0000) 2022-09-10T15:36:10.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:09 smithi104 ceph-mon[97044]: pgmap v21337: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:36:10.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:09 smithi104 ceph-mon[97044]: osdmap e1414: 12 total, 12 up, 12 in 2022-09-10T15:36:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:09.738+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:09.740047+0000) 2022-09-10T15:36:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:09 smithi110 ceph-mon[97633]: pgmap v21337: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:36:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:09 smithi110 ceph-mon[97633]: osdmap e1414: 12 total, 12 up, 12 in 2022-09-10T15:36:10.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:10.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:10.370585+0000) 2022-09-10T15:36:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:10.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:10.740316+0000) 2022-09-10T15:36:11.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:11.369+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:11.370818+0000) 2022-09-10T15:36:12.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:11 smithi104 ceph-mon[97044]: pgmap v21339: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:36:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:11.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:11.740523+0000) 2022-09-10T15:36:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:11 smithi110 ceph-mon[97633]: pgmap v21339: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:36:12.742 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:12.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:12.371078+0000) 2022-09-10T15:36:13.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:12.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:12.740653+0000) 2022-09-10T15:36:13.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:13.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:13.371293+0000) 2022-09-10T15:36:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:13 smithi104 ceph-mon[97044]: pgmap v21340: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:36:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:13 smithi104 ceph-mon[97044]: osdmap e1415: 12 total, 12 up, 12 in 2022-09-10T15:36:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:13.739+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:13.740870+0000) 2022-09-10T15:36:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:13 smithi110 ceph-mon[97633]: pgmap v21340: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:36:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:13 smithi110 ceph-mon[97633]: osdmap e1415: 12 total, 12 up, 12 in 2022-09-10T15:36:14.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:14.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:14.371501+0000) 2022-09-10T15:36:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:14.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:14.741126+0000) 2022-09-10T15:36:15.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:15.370+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:15.371646+0000) 2022-09-10T15:36:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:15 smithi104 ceph-mon[97044]: pgmap v21342: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:36:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:15.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:15.741267+0000) 2022-09-10T15:36:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:15 smithi110 ceph-mon[97633]: pgmap v21342: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:36:16.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:16.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:16.371892+0000) 2022-09-10T15:36:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:16.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:16.741458+0000) 2022-09-10T15:36:17.743 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:17.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:17.372154+0000) 2022-09-10T15:36:18.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:17 smithi104 ceph-mon[97044]: pgmap v21343: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:36:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:17.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:17.741673+0000) 2022-09-10T15:36:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:17 smithi110 ceph-mon[97633]: pgmap v21343: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:36:18.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:18.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:18.372289+0000) 2022-09-10T15:36:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:18.740+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:18.741936+0000) 2022-09-10T15:36:19.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:19.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:19.372550+0000) 2022-09-10T15:36:20.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:19 smithi104 ceph-mon[97044]: pgmap v21344: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:36:20.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:19.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:19.742138+0000) 2022-09-10T15:36:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:19 smithi110 ceph-mon[97633]: pgmap v21344: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:36:20.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:20.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:20.372786+0000) 2022-09-10T15:36:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:20.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:20.742271+0000) 2022-09-10T15:36:21.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:21.371+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:21.373027+0000) 2022-09-10T15:36:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:21 smithi104 ceph-mon[97044]: pgmap v21345: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:36:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:21.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:21.742465+0000) 2022-09-10T15:36:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:21 smithi110 ceph-mon[97633]: pgmap v21345: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:36:22.744 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:22.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:22.373266+0000) 2022-09-10T15:36:23.006 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:22.741+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:22.742630+0000) 2022-09-10T15:36:23.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:36:23.003+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:36:23.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:23.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:23.373499+0000) 2022-09-10T15:36:24.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:23 smithi104 ceph-mon[97044]: pgmap v21346: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:36:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:23.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:23.742891+0000) 2022-09-10T15:36:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:23 smithi110 ceph-mon[97633]: pgmap v21346: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:36:24.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:24.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:24.373765+0000) 2022-09-10T15:36:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:24.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:24.743179+0000) 2022-09-10T15:36:25.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:25.372+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:25.373905+0000) 2022-09-10T15:36:26.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:25 smithi104 ceph-mon[97044]: pgmap v21347: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:36:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:25.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:25.743346+0000) 2022-09-10T15:36:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:25 smithi110 ceph-mon[97633]: pgmap v21347: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:36:26.745 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:26.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:26.374124+0000) 2022-09-10T15:36:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:26.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:26.743519+0000) 2022-09-10T15:36:27.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:27.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:27.374281+0000) 2022-09-10T15:36:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:27 smithi104 ceph-mon[97044]: pgmap v21348: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:27.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:27.743706+0000) 2022-09-10T15:36:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:27 smithi110 ceph-mon[97633]: pgmap v21348: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:28.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:28.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:28.374446+0000) 2022-09-10T15:36:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:28.742+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:28.743838+0000) 2022-09-10T15:36:29.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:29.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:29.374643+0000) 2022-09-10T15:36:30.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:29 smithi104 ceph-mon[97044]: pgmap v21349: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:30.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:29.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:29.744077+0000) 2022-09-10T15:36:30.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:29 smithi110 ceph-mon[97633]: pgmap v21349: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:30.746 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:30.373+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:30.374858+0000) 2022-09-10T15:36:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:30.743+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:30.744375+0000) 2022-09-10T15:36:31.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:31.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:31.375144+0000) 2022-09-10T15:36:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:31 smithi104 ceph-mon[97044]: pgmap v21350: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:31.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:31.744679+0000) 2022-09-10T15:36:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:31 smithi110 ceph-mon[97633]: pgmap v21350: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:32.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:32.374+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:32.375309+0000) 2022-09-10T15:36:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:32.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:32.744866+0000) 2022-09-10T15:36:33.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:33.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:33.375535+0000) 2022-09-10T15:36:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:33 smithi104 ceph-mon[97044]: pgmap v21351: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:33.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:33.745159+0000) 2022-09-10T15:36:34.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:33 smithi110 ceph-mon[97633]: pgmap v21351: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:34.747 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:34.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:34.375772+0000) 2022-09-10T15:36:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:34.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:34.745419+0000) 2022-09-10T15:36:35.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:35.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:35.376047+0000) 2022-09-10T15:36:36.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:35 smithi104 ceph-mon[97044]: pgmap v21352: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:35.744+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:35.745639+0000) 2022-09-10T15:36:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:35 smithi110 ceph-mon[97633]: pgmap v21352: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:36.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:36.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:36.376327+0000) 2022-09-10T15:36:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:36.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:36.745894+0000) 2022-09-10T15:36:37.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:37.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:37.376603+0000) 2022-09-10T15:36:38.021 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:37.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:37.746109+0000) 2022-09-10T15:36:38.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:37 smithi104 ceph-mon[97044]: pgmap v21353: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:37 smithi110 ceph-mon[97633]: pgmap v21353: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:38.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:36:38.018+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:36:38.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:38.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:38.376800+0000) 2022-09-10T15:36:39.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:38.745+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:38.746396+0000) 2022-09-10T15:36:39.748 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:39.375+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:39.376998+0000) 2022-09-10T15:36:40.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:39 smithi104 ceph-mon[97044]: pgmap v21354: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:40.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:39.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:39.746630+0000) 2022-09-10T15:36:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:39 smithi110 ceph-mon[97633]: pgmap v21354: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:40.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:40.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:40.377221+0000) 2022-09-10T15:36:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:40.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:40.746819+0000) 2022-09-10T15:36:41.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:41.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:41.377400+0000) 2022-09-10T15:36:42.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:41 smithi104 ceph-mon[97044]: pgmap v21355: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:41.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:41.747089+0000) 2022-09-10T15:36:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:41 smithi110 ceph-mon[97633]: pgmap v21355: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:42.749 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:42.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:42.377555+0000) 2022-09-10T15:36:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:42.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:42.747322+0000) 2022-09-10T15:36:43.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:43.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:43.377811+0000) 2022-09-10T15:36:44.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:43 smithi104 ceph-mon[97044]: pgmap v21356: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:44.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:43.746+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:43.747519+0000) 2022-09-10T15:36:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:43 smithi110 ceph-mon[97633]: pgmap v21356: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:44.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:44.376+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:44.377973+0000) 2022-09-10T15:36:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:44.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:44.747790+0000) 2022-09-10T15:36:45.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:45.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:45.378239+0000) 2022-09-10T15:36:46.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:45 smithi104 ceph-mon[97044]: pgmap v21357: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:45.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:45.748031+0000) 2022-09-10T15:36:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:45 smithi110 ceph-mon[97633]: pgmap v21357: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:46.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:46.377+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:46.378413+0000) 2022-09-10T15:36:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:46.747+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:46.748267+0000) 2022-09-10T15:36:47.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:47.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:47.378660+0000) 2022-09-10T15:36:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:47 smithi104 ceph-mon[97044]: pgmap v21358: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:48.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:36:48.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:36:48.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:36:48.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:36:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:47.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:47.748555+0000) 2022-09-10T15:36:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:47 smithi110 ceph-mon[97633]: pgmap v21358: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:36:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:36:48.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:36:48.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:36:48.750 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:48.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:48.378838+0000) 2022-09-10T15:36:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:48.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:48.748718+0000) 2022-09-10T15:36:49.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:49.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:49.379061+0000) 2022-09-10T15:36:50.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:49 smithi104 ceph-mon[97044]: pgmap v21359: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:50.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:49.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:49.748924+0000) 2022-09-10T15:36:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:49 smithi110 ceph-mon[97633]: pgmap v21359: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:50.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:50.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:50.379266+0000) 2022-09-10T15:36:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:50.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:50.749207+0000) 2022-09-10T15:36:51.751 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:51.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:51.379452+0000) 2022-09-10T15:36:52.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:51 smithi104 ceph-mon[97044]: pgmap v21360: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:52.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:51.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:51.749428+0000) 2022-09-10T15:36:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:51 smithi110 ceph-mon[97633]: pgmap v21360: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:52.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:52.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:52.379697+0000) 2022-09-10T15:36:53.037 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:52.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:52.749678+0000) 2022-09-10T15:36:53.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:36:53.035+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:36:53.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:53.378+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:53.379978+0000) 2022-09-10T15:36:54.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:53 smithi104 ceph-mon[97044]: pgmap v21361: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:54.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:53.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:53.749857+0000) 2022-09-10T15:36:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:53 smithi110 ceph-mon[97633]: pgmap v21361: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:36:54.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:54.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:54.380276+0000) 2022-09-10T15:36:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:54.748+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:54.750060+0000) 2022-09-10T15:36:55.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:55.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:55.380488+0000) 2022-09-10T15:36:56.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:55 smithi104 ceph-mon[97044]: pgmap v21362: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:55.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:55.750285+0000) 2022-09-10T15:36:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:55 smithi110 ceph-mon[97633]: pgmap v21362: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:36:56.752 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:56.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:56.380677+0000) 2022-09-10T15:36:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:56.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:56.750475+0000) 2022-09-10T15:36:57.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:57.379+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:57.380919+0000) 2022-09-10T15:36:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:57.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:57.750659+0000) 2022-09-10T15:36:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:57 smithi110 ceph-mon[97633]: pgmap v21363: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:36:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:57 smithi104 ceph-mon[97044]: pgmap v21363: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:36:58.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:58.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:58.381121+0000) 2022-09-10T15:36:59.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:58.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:58.750865+0000) 2022-09-10T15:36:59.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:36:59.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:59.381351+0000) 2022-09-10T15:37:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:36:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:36:59.749+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:36:59.751029+0000) 2022-09-10T15:37:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:36:59 smithi110 ceph-mon[97633]: pgmap v21364: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:37:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:36:59 smithi104 ceph-mon[97044]: pgmap v21364: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:37:00.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:00.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:00.381507+0000) 2022-09-10T15:37:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:00.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:00.751228+0000) 2022-09-10T15:37:01.753 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:01.380+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:01.381737+0000) 2022-09-10T15:37:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:01.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:01.751428+0000) 2022-09-10T15:37:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:01 smithi110 ceph-mon[97633]: pgmap v21365: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:01 smithi104 ceph-mon[97044]: pgmap v21365: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:02.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:02.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:02.381928+0000) 2022-09-10T15:37:03.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:02.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:02.751604+0000) 2022-09-10T15:37:03.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:03.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:03.382187+0000) 2022-09-10T15:37:04.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:03.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:03.751757+0000) 2022-09-10T15:37:04.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:03 smithi110 ceph-mon[97633]: pgmap v21366: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:37:04.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:03 smithi104 ceph-mon[97044]: pgmap v21366: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:37:04.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:04.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:04.382355+0000) 2022-09-10T15:37:05.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:04.750+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:04.751963+0000) 2022-09-10T15:37:05.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:05.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:05.382538+0000) 2022-09-10T15:37:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:05.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:05.752169+0000) 2022-09-10T15:37:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:05 smithi110 ceph-mon[97633]: pgmap v21367: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:37:06.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:05 smithi104 ceph-mon[97044]: pgmap v21367: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:37:06.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:06.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:06.382692+0000) 2022-09-10T15:37:07.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:06.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:06.752376+0000) 2022-09-10T15:37:07.754 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:07.381+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:07.382945+0000) 2022-09-10T15:37:08.051 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:07.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:07.752557+0000) 2022-09-10T15:37:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:07 smithi110 ceph-mon[97633]: pgmap v21368: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:37:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:07 smithi104 ceph-mon[97044]: pgmap v21368: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:37:08.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:37:08.043+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:37:08.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:08.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:08.383207+0000) 2022-09-10T15:37:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:08.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:08.752771+0000) 2022-09-10T15:37:09.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:09.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:09.383375+0000) 2022-09-10T15:37:10.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:09.751+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:09.752967+0000) 2022-09-10T15:37:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:09 smithi110 ceph-mon[97633]: pgmap v21369: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:37:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:09 smithi110 ceph-mon[97633]: osdmap e1416: 12 total, 12 up, 12 in 2022-09-10T15:37:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:09 smithi104 ceph-mon[97044]: pgmap v21369: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:37:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:09 smithi104 ceph-mon[97044]: osdmap e1416: 12 total, 12 up, 12 in 2022-09-10T15:37:10.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:10.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:10.383584+0000) 2022-09-10T15:37:11.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:10.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:10.753195+0000) 2022-09-10T15:37:11.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:11.382+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:11.383807+0000) 2022-09-10T15:37:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:11.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:11.753395+0000) 2022-09-10T15:37:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:11 smithi110 ceph-mon[97633]: pgmap v21371: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:37:12.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:11 smithi104 ceph-mon[97044]: pgmap v21371: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:37:12.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:12.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:12.384084+0000) 2022-09-10T15:37:13.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:12.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:12.753578+0000) 2022-09-10T15:37:13.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:13.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:13.384348+0000) 2022-09-10T15:37:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:13.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:13.753777+0000) 2022-09-10T15:37:14.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:13 smithi110 ceph-mon[97633]: pgmap v21372: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:37:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:13 smithi104 ceph-mon[97044]: pgmap v21372: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:37:14.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:14.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:14.384540+0000) 2022-09-10T15:37:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:14.752+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:14.754033+0000) 2022-09-10T15:37:15.756 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:15.383+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:15.384719+0000) 2022-09-10T15:37:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:15.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:15.754244+0000) 2022-09-10T15:37:16.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:15 smithi110 ceph-mon[97633]: pgmap v21373: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:37:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:15 smithi104 ceph-mon[97044]: pgmap v21373: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:37:16.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:16.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:16.384966+0000) 2022-09-10T15:37:17.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:16.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:16.754443+0000) 2022-09-10T15:37:17.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:17.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:17.385157+0000) 2022-09-10T15:37:18.064 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:17.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:17.754588+0000) 2022-09-10T15:37:18.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:17 smithi110 ceph-mon[97633]: pgmap v21374: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:37:18.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:17 smithi110 ceph-mon[97633]: osdmap e1417: 12 total, 12 up, 12 in 2022-09-10T15:37:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:17 smithi104 ceph-mon[97044]: pgmap v21374: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:37:18.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:17 smithi104 ceph-mon[97044]: osdmap e1417: 12 total, 12 up, 12 in 2022-09-10T15:37:18.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:18.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:18.385358+0000) 2022-09-10T15:37:19.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:18.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:18.754798+0000) 2022-09-10T15:37:19.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:19.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:19.385564+0000) 2022-09-10T15:37:20.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:19.753+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:19.755039+0000) 2022-09-10T15:37:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:19 smithi110 ceph-mon[97633]: pgmap v21376: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:37:20.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:19 smithi104 ceph-mon[97044]: pgmap v21376: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:37:20.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:20.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:20.385783+0000) 2022-09-10T15:37:21.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:20.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:20.755243+0000) 2022-09-10T15:37:21.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:21.384+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:21.386046+0000) 2022-09-10T15:37:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:21.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:21.755424+0000) 2022-09-10T15:37:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:21 smithi110 ceph-mon[97633]: pgmap v21377: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:37:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:21 smithi104 ceph-mon[97044]: pgmap v21377: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:37:22.757 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:22.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:22.386286+0000) 2022-09-10T15:37:23.048 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:22.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:22.755560+0000) 2022-09-10T15:37:23.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:37:23.045+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:37:23.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:23.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:23.386488+0000) 2022-09-10T15:37:24.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:23.754+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:23.755817+0000) 2022-09-10T15:37:24.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:23 smithi110 ceph-mon[97633]: pgmap v21378: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:37:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:23 smithi104 ceph-mon[97044]: pgmap v21378: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:37:24.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:24.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:24.386691+0000) 2022-09-10T15:37:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:24.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:24.756011+0000) 2022-09-10T15:37:25.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:25.385+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:25.386901+0000) 2022-09-10T15:37:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:25.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:25.756185+0000) 2022-09-10T15:37:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:25 smithi110 ceph-mon[97633]: pgmap v21379: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:37:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:25 smithi104 ceph-mon[97044]: pgmap v21379: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:37:26.758 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:26.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:26.387154+0000) 2022-09-10T15:37:27.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:26.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:26.756380+0000) 2022-09-10T15:37:27.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:27.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:27.387287+0000) 2022-09-10T15:37:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:27.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:27.756543+0000) 2022-09-10T15:37:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:27 smithi110 ceph-mon[97633]: pgmap v21380: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:37:28.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:27 smithi104 ceph-mon[97044]: pgmap v21380: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:37:28.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:28.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:28.387560+0000) 2022-09-10T15:37:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:28.755+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:28.756800+0000) 2022-09-10T15:37:29.759 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:29.386+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:29.387717+0000) 2022-09-10T15:37:30.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:29.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:29.757031+0000) 2022-09-10T15:37:30.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:29 smithi110 ceph-mon[97633]: pgmap v21381: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:37:30.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:29 smithi104 ceph-mon[97044]: pgmap v21381: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:37:30.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:30.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:30.387949+0000) 2022-09-10T15:37:31.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:30.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:30.757280+0000) 2022-09-10T15:37:31.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:31.387+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:31.388169+0000) 2022-09-10T15:37:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:31.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:31.757475+0000) 2022-09-10T15:37:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:31 smithi110 ceph-mon[97633]: pgmap v21382: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:31 smithi104 ceph-mon[97044]: pgmap v21382: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:32.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:32.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:32.388393+0000) 2022-09-10T15:37:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:32.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:32.757655+0000) 2022-09-10T15:37:33.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:33.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:33.388558+0000) 2022-09-10T15:37:34.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:33.756+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:33.757918+0000) 2022-09-10T15:37:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:33 smithi110 ceph-mon[97633]: pgmap v21383: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:33 smithi104 ceph-mon[97044]: pgmap v21383: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:34.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:34.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:34.388783+0000) 2022-09-10T15:37:35.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:34.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:34.758158+0000) 2022-09-10T15:37:35.760 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:35.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:35.388977+0000) 2022-09-10T15:37:36.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:35.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:35.758377+0000) 2022-09-10T15:37:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:35 smithi110 ceph-mon[97633]: pgmap v21384: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:36.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:35 smithi104 ceph-mon[97044]: pgmap v21384: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:36.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:36.388+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:36.389183+0000) 2022-09-10T15:37:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:36.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:36.758582+0000) 2022-09-10T15:37:37.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:37.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:37.389351+0000) 2022-09-10T15:37:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:37.758+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:37.758720+0000) 2022-09-10T15:37:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:37 smithi110 ceph-mon[97633]: pgmap v21385: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:37 smithi104 ceph-mon[97044]: pgmap v21385: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:38.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:37:38.061+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:37:38.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:38.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:38.389577+0000) 2022-09-10T15:37:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:38.757+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:38.758951+0000) 2022-09-10T15:37:39.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:39.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:39.389813+0000) 2022-09-10T15:37:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:39.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:39.759164+0000) 2022-09-10T15:37:40.168 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:39 smithi110 ceph-mon[97633]: pgmap v21386: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:40.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:39 smithi104 ceph-mon[97044]: pgmap v21386: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:40.761 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:40.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:40.389954+0000) 2022-09-10T15:37:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:40.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:40.759423+0000) 2022-09-10T15:37:41.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:41.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:41.390239+0000) 2022-09-10T15:37:42.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:41.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:41.759658+0000) 2022-09-10T15:37:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:41 smithi110 ceph-mon[97633]: pgmap v21387: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:41 smithi104 ceph-mon[97044]: pgmap v21387: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:42.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:42.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:42.390413+0000) 2022-09-10T15:37:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:42.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:42.759902+0000) 2022-09-10T15:37:43.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:43.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:43.390634+0000) 2022-09-10T15:37:44.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:43.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:43.760138+0000) 2022-09-10T15:37:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:43 smithi110 ceph-mon[97633]: pgmap v21388: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:43 smithi104 ceph-mon[97044]: pgmap v21388: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:44.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:44.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:44.390810+0000) 2022-09-10T15:37:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:44.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:44.760310+0000) 2022-09-10T15:37:45.762 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:45.389+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:45.391032+0000) 2022-09-10T15:37:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:45.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:45.760528+0000) 2022-09-10T15:37:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:45 smithi110 ceph-mon[97633]: pgmap v21389: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:45 smithi104 ceph-mon[97044]: pgmap v21389: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:46.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:46.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:46.391251+0000) 2022-09-10T15:37:47.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:46.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:46.760734+0000) 2022-09-10T15:37:47.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:47.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:47.391467+0000) 2022-09-10T15:37:48.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:47.759+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:47.760985+0000) 2022-09-10T15:37:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:47 smithi110 ceph-mon[97633]: pgmap v21390: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:37:48.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:37:48.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:37:48.161 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:37:48.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:47 smithi104 ceph-mon[97044]: pgmap v21390: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:48.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:37:48.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:37:48.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:37:48.302 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:37:48.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:48.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:48.391631+0000) 2022-09-10T15:37:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:48.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:48.761204+0000) 2022-09-10T15:37:49.763 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:49.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:49.391863+0000) 2022-09-10T15:37:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:49.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:49.761457+0000) 2022-09-10T15:37:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:49 smithi110 ceph-mon[97633]: pgmap v21391: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:50.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:49 smithi104 ceph-mon[97044]: pgmap v21391: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:50.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:50.390+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:50.392030+0000) 2022-09-10T15:37:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:50.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:50.761673+0000) 2022-09-10T15:37:51.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:51.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:51.392251+0000) 2022-09-10T15:37:52.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:51.760+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:51.761891+0000) 2022-09-10T15:37:52.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:51 smithi110 ceph-mon[97633]: pgmap v21392: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:52.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:51 smithi104 ceph-mon[97044]: pgmap v21392: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:52.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:52.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:52.392431+0000) 2022-09-10T15:37:53.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:52.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:52.762060+0000) 2022-09-10T15:37:53.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:37:53.069+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:37:53.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:53.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:53.392624+0000) 2022-09-10T15:37:54.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:53.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:53.762313+0000) 2022-09-10T15:37:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:53 smithi110 ceph-mon[97633]: pgmap v21393: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:53 smithi104 ceph-mon[97044]: pgmap v21393: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:54.764 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:54.391+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:54.392837+0000) 2022-09-10T15:37:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:54.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:54.762556+0000) 2022-09-10T15:37:55.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:55.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:55.393011+0000) 2022-09-10T15:37:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:55.761+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:55.762795+0000) 2022-09-10T15:37:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:55 smithi110 ceph-mon[97633]: pgmap v21394: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:55 smithi104 ceph-mon[97044]: pgmap v21394: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:37:56.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:56.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:56.393207+0000) 2022-09-10T15:37:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:56.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:56.762969+0000) 2022-09-10T15:37:57.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:57.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:57.393340+0000) 2022-09-10T15:37:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:57.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:57.763214+0000) 2022-09-10T15:37:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:57 smithi110 ceph-mon[97633]: pgmap v21395: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:57 smithi104 ceph-mon[97044]: pgmap v21395: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:37:58.765 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:58.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:58.393546+0000) 2022-09-10T15:37:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:58.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:58.763465+0000) 2022-09-10T15:37:59.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:37:59.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:59.393728+0000) 2022-09-10T15:38:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:37:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:37:59.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:37:59.763733+0000) 2022-09-10T15:38:00.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:37:59 smithi110 ceph-mon[97633]: pgmap v21396: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:37:59 smithi104 ceph-mon[97044]: pgmap v21396: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:00.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:00.392+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:00.393916+0000) 2022-09-10T15:38:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:00.762+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:00.764002+0000) 2022-09-10T15:38:01.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:01.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:01.394181+0000) 2022-09-10T15:38:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:01.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:01.764195+0000) 2022-09-10T15:38:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:01 smithi110 ceph-mon[97633]: pgmap v21397: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:01 smithi104 ceph-mon[97044]: pgmap v21397: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:02.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:02.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:02.394344+0000) 2022-09-10T15:38:03.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:02.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:02.764414+0000) 2022-09-10T15:38:03.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:03.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:03.394565+0000) 2022-09-10T15:38:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:03.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:03.764589+0000) 2022-09-10T15:38:04.167 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:03 smithi110 ceph-mon[97633]: pgmap v21398: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:03 smithi104 ceph-mon[97044]: pgmap v21398: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:04.766 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:04.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:04.394762+0000) 2022-09-10T15:38:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:04.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:04.764813+0000) 2022-09-10T15:38:05.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:05.393+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:05.395006+0000) 2022-09-10T15:38:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:05.763+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:05.765032+0000) 2022-09-10T15:38:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:05 smithi110 ceph-mon[97633]: pgmap v21399: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:38:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:05 smithi104 ceph-mon[97044]: pgmap v21399: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:38:06.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:06.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:06.395255+0000) 2022-09-10T15:38:07.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:06.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:06.765257+0000) 2022-09-10T15:38:07.767 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:07.394+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:07.395489+0000) 2022-09-10T15:38:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:07.764+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:07.765484+0000) 2022-09-10T15:38:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:07 smithi110 ceph-mon[97633]: pgmap v21400: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:38:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:07 smithi104 ceph-mon[97044]: pgmap v21400: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:38:08.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:38:08.081+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:38:08.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:08.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:08.395741+0000) 2022-09-10T15:38:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:08.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:08.765715+0000) 2022-09-10T15:38:09.768 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:09.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:09.395958+0000) 2022-09-10T15:38:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:09.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:09.766050+0000) 2022-09-10T15:38:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:09 smithi104 ceph-mon[97044]: pgmap v21401: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:38:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:09 smithi104 ceph-mon[97044]: osdmap e1418: 12 total, 12 up, 12 in 2022-09-10T15:38:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:09 smithi110 ceph-mon[97633]: pgmap v21401: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:38:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:09 smithi110 ceph-mon[97633]: osdmap e1418: 12 total, 12 up, 12 in 2022-09-10T15:38:10.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:10.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:10.396143+0000) 2022-09-10T15:38:11.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:10.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:10.766304+0000) 2022-09-10T15:38:11.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:11.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:11.396316+0000) 2022-09-10T15:38:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:11.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:11.766564+0000) 2022-09-10T15:38:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:11 smithi104 ceph-mon[97044]: pgmap v21403: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:38:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:11 smithi110 ceph-mon[97633]: pgmap v21403: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:38:12.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:12.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:12.396525+0000) 2022-09-10T15:38:13.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:12.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:12.766719+0000) 2022-09-10T15:38:13.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:13.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:13.396806+0000) 2022-09-10T15:38:14.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:13 smithi104 ceph-mon[97044]: pgmap v21404: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:38:14.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:13 smithi104 ceph-mon[97044]: osdmap e1419: 12 total, 12 up, 12 in 2022-09-10T15:38:14.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:13.765+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:13.766871+0000) 2022-09-10T15:38:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:13 smithi110 ceph-mon[97633]: pgmap v21404: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:38:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:13 smithi110 ceph-mon[97633]: osdmap e1419: 12 total, 12 up, 12 in 2022-09-10T15:38:14.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:14.395+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:14.396995+0000) 2022-09-10T15:38:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:14.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:14.767087+0000) 2022-09-10T15:38:15.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:15.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:15.397254+0000) 2022-09-10T15:38:16.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:15 smithi104 ceph-mon[97044]: pgmap v21406: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:38:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:15.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:15.767330+0000) 2022-09-10T15:38:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:15 smithi110 ceph-mon[97633]: pgmap v21406: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:38:16.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:16.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:16.397480+0000) 2022-09-10T15:38:17.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:16.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:16.767536+0000) 2022-09-10T15:38:17.769 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:17.396+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:17.397752+0000) 2022-09-10T15:38:18.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:17 smithi104 ceph-mon[97044]: pgmap v21407: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:38:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:17.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:17.767727+0000) 2022-09-10T15:38:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:17 smithi110 ceph-mon[97633]: pgmap v21407: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:38:18.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:18.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:18.397957+0000) 2022-09-10T15:38:19.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:18.766+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:18.767959+0000) 2022-09-10T15:38:19.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:19.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:19.398177+0000) 2022-09-10T15:38:20.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:19 smithi104 ceph-mon[97044]: pgmap v21408: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:38:20.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:19.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:19.768185+0000) 2022-09-10T15:38:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:19 smithi110 ceph-mon[97633]: pgmap v21408: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:38:20.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:20.397+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:20.398341+0000) 2022-09-10T15:38:21.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:20.767+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:20.768371+0000) 2022-09-10T15:38:21.770 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:21.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:21.398580+0000) 2022-09-10T15:38:22.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:21 smithi104 ceph-mon[97044]: pgmap v21409: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:38:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:21.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:21.768649+0000) 2022-09-10T15:38:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:21 smithi110 ceph-mon[97633]: pgmap v21409: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:38:22.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:22.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:22.398852+0000) 2022-09-10T15:38:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:22.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:22.768926+0000) 2022-09-10T15:38:23.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:38:23.095+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:38:23.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:23.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:23.399040+0000) 2022-09-10T15:38:24.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:23 smithi104 ceph-mon[97044]: pgmap v21410: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:38:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:23.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:23.769092+0000) 2022-09-10T15:38:24.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:23 smithi110 ceph-mon[97633]: pgmap v21410: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:38:24.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:24.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:24.399175+0000) 2022-09-10T15:38:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:24.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:24.769313+0000) 2022-09-10T15:38:25.771 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:25.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:25.399350+0000) 2022-09-10T15:38:26.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:25 smithi104 ceph-mon[97044]: pgmap v21411: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:38:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:25.768+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:25.769476+0000) 2022-09-10T15:38:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:25 smithi110 ceph-mon[97633]: pgmap v21411: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:38:26.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:26.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:26.399560+0000) 2022-09-10T15:38:27.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:26.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:26.769724+0000) 2022-09-10T15:38:27.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:27.398+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:27.399821+0000) 2022-09-10T15:38:28.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:27 smithi104 ceph-mon[97044]: pgmap v21412: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:27.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:27.769992+0000) 2022-09-10T15:38:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:27 smithi110 ceph-mon[97633]: pgmap v21412: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:28.772 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:28.399+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:28.400110+0000) 2022-09-10T15:38:29.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:28.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:28.770253+0000) 2022-09-10T15:38:29.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:29.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:29.400349+0000) 2022-09-10T15:38:30.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:29 smithi104 ceph-mon[97044]: pgmap v21413: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:29.769+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:29.770455+0000) 2022-09-10T15:38:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:29 smithi110 ceph-mon[97633]: pgmap v21413: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:30.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:30.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:30.400562+0000) 2022-09-10T15:38:31.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:30.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:30.770630+0000) 2022-09-10T15:38:31.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:31.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:31.400812+0000) 2022-09-10T15:38:32.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:31 smithi104 ceph-mon[97044]: pgmap v21414: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:31.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:31.770866+0000) 2022-09-10T15:38:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:31 smithi110 ceph-mon[97633]: pgmap v21414: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:32.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:32.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:32.401078+0000) 2022-09-10T15:38:33.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:32.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:32.771158+0000) 2022-09-10T15:38:33.773 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:33.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:33.401320+0000) 2022-09-10T15:38:34.047 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:33 smithi104 ceph-mon[97044]: pgmap v21415: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:34.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:33.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:33.771371+0000) 2022-09-10T15:38:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:33 smithi110 ceph-mon[97633]: pgmap v21415: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:34.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:34.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:34.401585+0000) 2022-09-10T15:38:35.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:34.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:34.771613+0000) 2022-09-10T15:38:35.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:35.400+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:35.401867+0000) 2022-09-10T15:38:36.048 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:35 smithi104 ceph-mon[97044]: pgmap v21416: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:36.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:35.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:35.771819+0000) 2022-09-10T15:38:36.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:35 smithi110 ceph-mon[97633]: pgmap v21416: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:36.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:36.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:36.402085+0000) 2022-09-10T15:38:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:36.770+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:36.771999+0000) 2022-09-10T15:38:37.774 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:37.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:37.402297+0000) 2022-09-10T15:38:38.065 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:37.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:37.772136+0000) 2022-09-10T15:38:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:37 smithi110 ceph-mon[97633]: pgmap v21417: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:38.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:37 smithi104 ceph-mon[97044]: pgmap v21417: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:38.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:38:38.102+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:38:38.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:38.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:38.402447+0000) 2022-09-10T15:38:39.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:38.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:38.772345+0000) 2022-09-10T15:38:39.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:39.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:39.402697+0000) 2022-09-10T15:38:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:39.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:39.772605+0000) 2022-09-10T15:38:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:39 smithi110 ceph-mon[97633]: pgmap v21418: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:39 smithi104 ceph-mon[97044]: pgmap v21418: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:40.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:40.401+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:40.402957+0000) 2022-09-10T15:38:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:40.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:40.772808+0000) 2022-09-10T15:38:41.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:41.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:41.403228+0000) 2022-09-10T15:38:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:41.771+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:41.772997+0000) 2022-09-10T15:38:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:41 smithi110 ceph-mon[97633]: pgmap v21419: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:42.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:41 smithi104 ceph-mon[97044]: pgmap v21419: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:42.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:42.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:42.403453+0000) 2022-09-10T15:38:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:42.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:42.773209+0000) 2022-09-10T15:38:43.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:43.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:43.403691+0000) 2022-09-10T15:38:44.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:43.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:43.773382+0000) 2022-09-10T15:38:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:43 smithi110 ceph-mon[97633]: pgmap v21420: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:44.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:43 smithi104 ceph-mon[97044]: pgmap v21420: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:44.775 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:44.402+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:44.403948+0000) 2022-09-10T15:38:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:44.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:44.773557+0000) 2022-09-10T15:38:45.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:45.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:45.404166+0000) 2022-09-10T15:38:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:45.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:45.773737+0000) 2022-09-10T15:38:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:45 smithi110 ceph-mon[97633]: pgmap v21421: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:45 smithi104 ceph-mon[97044]: pgmap v21421: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:46.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:46.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:46.404400+0000) 2022-09-10T15:38:47.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:46.772+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:46.773984+0000) 2022-09-10T15:38:47.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:47.403+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:47.404649+0000) 2022-09-10T15:38:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:47.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:47.774255+0000) 2022-09-10T15:38:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:47 smithi110 ceph-mon[97633]: pgmap v21422: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:38:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:38:48.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:38:48.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:38:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:47 smithi104 ceph-mon[97044]: pgmap v21422: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:38:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:38:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:38:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:38:48.776 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:48.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:48.404910+0000) 2022-09-10T15:38:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:48.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:48.774462+0000) 2022-09-10T15:38:49.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:49.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:49.405121+0000) 2022-09-10T15:38:50.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:49.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:49.774820+0000) 2022-09-10T15:38:50.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:49 smithi110 ceph-mon[97633]: pgmap v21423: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:50.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:49 smithi104 ceph-mon[97044]: pgmap v21423: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:50.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:50.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:50.405371+0000) 2022-09-10T15:38:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:50.773+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:50.775012+0000) 2022-09-10T15:38:51.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:51.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:51.405543+0000) 2022-09-10T15:38:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:51.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:51.775265+0000) 2022-09-10T15:38:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:51 smithi110 ceph-mon[97633]: pgmap v21424: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:51 smithi104 ceph-mon[97044]: pgmap v21424: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:52.777 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:52.404+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:52.405794+0000) 2022-09-10T15:38:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:52.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:52.775467+0000) 2022-09-10T15:38:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:38:53.118+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:38:53.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:53.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:53.406052+0000) 2022-09-10T15:38:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:53.774+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:53.775647+0000) 2022-09-10T15:38:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:53 smithi110 ceph-mon[97633]: pgmap v21425: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:53 smithi104 ceph-mon[97044]: pgmap v21425: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:54.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:54.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:54.406339+0000) 2022-09-10T15:38:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:54.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:54.775908+0000) 2022-09-10T15:38:55.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:55.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:55.406594+0000) 2022-09-10T15:38:56.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:55.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:55.776220+0000) 2022-09-10T15:38:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:55 smithi110 ceph-mon[97633]: pgmap v21426: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:56.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:55 smithi104 ceph-mon[97044]: pgmap v21426: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:38:56.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:56.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:56.406798+0000) 2022-09-10T15:38:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:56.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:56.776370+0000) 2022-09-10T15:38:57.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:57.405+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:57.407034+0000) 2022-09-10T15:38:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:57.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:57.776541+0000) 2022-09-10T15:38:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:57 smithi110 ceph-mon[97633]: pgmap v21427: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:57 smithi104 ceph-mon[97044]: pgmap v21427: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:38:58.778 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:58.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:58.407269+0000) 2022-09-10T15:38:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:58.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:58.776737+0000) 2022-09-10T15:38:59.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:38:59.406+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:59.407466+0000) 2022-09-10T15:39:00.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:38:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:38:59.775+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:38:59.776952+0000) 2022-09-10T15:39:00.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:38:59 smithi110 ceph-mon[97633]: pgmap v21428: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:38:59 smithi104 ceph-mon[97044]: pgmap v21428: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:00.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:00.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:00.407642+0000) 2022-09-10T15:39:01.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:00.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:00.777118+0000) 2022-09-10T15:39:01.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:01.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:01.407804+0000) 2022-09-10T15:39:02.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:01.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:01.777376+0000) 2022-09-10T15:39:02.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:01 smithi110 ceph-mon[97633]: pgmap v21429: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:01 smithi104 ceph-mon[97044]: pgmap v21429: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:02.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:02.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:02.408042+0000) 2022-09-10T15:39:03.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:02.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:02.777625+0000) 2022-09-10T15:39:03.779 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:03.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:03.408307+0000) 2022-09-10T15:39:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:03.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:03.777812+0000) 2022-09-10T15:39:04.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:03 smithi110 ceph-mon[97633]: pgmap v21430: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:04.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:03 smithi104 ceph-mon[97044]: pgmap v21430: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:04.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:04.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:04.408520+0000) 2022-09-10T15:39:05.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:04.776+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:04.777968+0000) 2022-09-10T15:39:05.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:05.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:05.408785+0000) 2022-09-10T15:39:06.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:05.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:05.778144+0000) 2022-09-10T15:39:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:05 smithi110 ceph-mon[97633]: pgmap v21431: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:39:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:05 smithi104 ceph-mon[97044]: pgmap v21431: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:39:06.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:06.407+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:06.409073+0000) 2022-09-10T15:39:07.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:06.777+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:06.778351+0000) 2022-09-10T15:39:07.780 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:07.408+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:07.409289+0000) 2022-09-10T15:39:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:07.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:07.778571+0000) 2022-09-10T15:39:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:07 smithi110 ceph-mon[97633]: pgmap v21432: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:39:08.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:07 smithi104 ceph-mon[97044]: pgmap v21432: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:39:08.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:39:08.130+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:39:08.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:08.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:08.409537+0000) 2022-09-10T15:39:09.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:08.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:08.778806+0000) 2022-09-10T15:39:09.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:09.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:09.409742+0000) 2022-09-10T15:39:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:09.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:09.779069+0000) 2022-09-10T15:39:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:09 smithi110 ceph-mon[97633]: pgmap v21433: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:39:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:09 smithi110 ceph-mon[97633]: osdmap e1420: 12 total, 12 up, 12 in 2022-09-10T15:39:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:09 smithi104 ceph-mon[97044]: pgmap v21433: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:39:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:09 smithi104 ceph-mon[97044]: osdmap e1420: 12 total, 12 up, 12 in 2022-09-10T15:39:10.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:10.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:10.409873+0000) 2022-09-10T15:39:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:10.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:10.779342+0000) 2022-09-10T15:39:11.781 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:11.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:11.410121+0000) 2022-09-10T15:39:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:11.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:11.779625+0000) 2022-09-10T15:39:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:11 smithi110 ceph-mon[97633]: pgmap v21435: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:39:12.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:11 smithi104 ceph-mon[97044]: pgmap v21435: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:39:12.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:12.409+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:12.410371+0000) 2022-09-10T15:39:13.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:12.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:12.779821+0000) 2022-09-10T15:39:13.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:13.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:13.410583+0000) 2022-09-10T15:39:14.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:13.778+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:13.779994+0000) 2022-09-10T15:39:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:13 smithi110 ceph-mon[97633]: pgmap v21436: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:39:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:13 smithi104 ceph-mon[97044]: pgmap v21436: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:39:14.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:14.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:14.410762+0000) 2022-09-10T15:39:15.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:14.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:14.780208+0000) 2022-09-10T15:39:15.782 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:15.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:15.410974+0000) 2022-09-10T15:39:16.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:15.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:15.780360+0000) 2022-09-10T15:39:16.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:15 smithi110 ceph-mon[97633]: pgmap v21437: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:39:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:15 smithi104 ceph-mon[97044]: pgmap v21437: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:39:16.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:16.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:16.411184+0000) 2022-09-10T15:39:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:16.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:16.780534+0000) 2022-09-10T15:39:17.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:17.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:17.411396+0000) 2022-09-10T15:39:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:17.779+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:17.780715+0000) 2022-09-10T15:39:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:17 smithi110 ceph-mon[97633]: pgmap v21438: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:39:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:17 smithi110 ceph-mon[97633]: osdmap e1421: 12 total, 12 up, 12 in 2022-09-10T15:39:18.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:17 smithi104 ceph-mon[97044]: pgmap v21438: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:39:18.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:17 smithi104 ceph-mon[97044]: osdmap e1421: 12 total, 12 up, 12 in 2022-09-10T15:39:18.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:18.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:18.411678+0000) 2022-09-10T15:39:19.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:18.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:18.780886+0000) 2022-09-10T15:39:19.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:19.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:19.411878+0000) 2022-09-10T15:39:20.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:19.780+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:19.781191+0000) 2022-09-10T15:39:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:19 smithi110 ceph-mon[97633]: pgmap v21440: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:39:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:19 smithi104 ceph-mon[97044]: pgmap v21440: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:39:20.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:20.410+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:20.412055+0000) 2022-09-10T15:39:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:20.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:20.781412+0000) 2022-09-10T15:39:21.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:21.411+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:21.412180+0000) 2022-09-10T15:39:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:21.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:21.781692+0000) 2022-09-10T15:39:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:21 smithi110 ceph-mon[97633]: pgmap v21441: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:39:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:21 smithi104 ceph-mon[97044]: pgmap v21441: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:39:22.783 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:22.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:22.412339+0000) 2022-09-10T15:39:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:22.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:22.781836+0000) 2022-09-10T15:39:23.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:39:23.136+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:39:23.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:23.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:23.412579+0000) 2022-09-10T15:39:24.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:23.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:23.782069+0000) 2022-09-10T15:39:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:23 smithi110 ceph-mon[97633]: pgmap v21442: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:39:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:23 smithi104 ceph-mon[97044]: pgmap v21442: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:39:24.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:24.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:24.412811+0000) 2022-09-10T15:39:25.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:24.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:24.782378+0000) 2022-09-10T15:39:25.784 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:25.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:25.412953+0000) 2022-09-10T15:39:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:25.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:25.782534+0000) 2022-09-10T15:39:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:25 smithi110 ceph-mon[97633]: pgmap v21443: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:39:26.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:25 smithi104 ceph-mon[97044]: pgmap v21443: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:39:26.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:26.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:26.413212+0000) 2022-09-10T15:39:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:26.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:26.782747+0000) 2022-09-10T15:39:27.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:27.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:27.413417+0000) 2022-09-10T15:39:28.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:27.781+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:27.782948+0000) 2022-09-10T15:39:28.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:27 smithi110 ceph-mon[97633]: pgmap v21444: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:39:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:27 smithi104 ceph-mon[97044]: pgmap v21444: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:39:28.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:28.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:28.413656+0000) 2022-09-10T15:39:29.062 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:28.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:28.783170+0000) 2022-09-10T15:39:29.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:29.412+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:29.413934+0000) 2022-09-10T15:39:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:29.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:29.783324+0000) 2022-09-10T15:39:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:29 smithi110 ceph-mon[97633]: pgmap v21445: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:39:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:29 smithi104 ceph-mon[97044]: pgmap v21445: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:39:30.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:30.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:30.414088+0000) 2022-09-10T15:39:31.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:30.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:30.783559+0000) 2022-09-10T15:39:31.785 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:31.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:31.414364+0000) 2022-09-10T15:39:32.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:31.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:31.783760+0000) 2022-09-10T15:39:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:31 smithi110 ceph-mon[97633]: pgmap v21446: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:31 smithi104 ceph-mon[97044]: pgmap v21446: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:32.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:32.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:32.414558+0000) 2022-09-10T15:39:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:32.782+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:32.783924+0000) 2022-09-10T15:39:33.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:33.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:33.414742+0000) 2022-09-10T15:39:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:33.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:33.784168+0000) 2022-09-10T15:39:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:33 smithi110 ceph-mon[97633]: pgmap v21447: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:33 smithi104 ceph-mon[97044]: pgmap v21447: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:34.786 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:34.413+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:34.414937+0000) 2022-09-10T15:39:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:34.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:34.784413+0000) 2022-09-10T15:39:35.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:35.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:35.415214+0000) 2022-09-10T15:39:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:35.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:35.784666+0000) 2022-09-10T15:39:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:35 smithi110 ceph-mon[97633]: pgmap v21448: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:35 smithi104 ceph-mon[97044]: pgmap v21448: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:36.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:36.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:36.415435+0000) 2022-09-10T15:39:37.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:36.783+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:36.784927+0000) 2022-09-10T15:39:37.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:37.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:37.415699+0000) 2022-09-10T15:39:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:37.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:37.785186+0000) 2022-09-10T15:39:38.165 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:37 smithi110 ceph-mon[97633]: pgmap v21449: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:38.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:37 smithi104 ceph-mon[97044]: pgmap v21449: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:38.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:39:38.141+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:39:38.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:38.414+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:38.415877+0000) 2022-09-10T15:39:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:38.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:38.785364+0000) 2022-09-10T15:39:39.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:39.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:39.416136+0000) 2022-09-10T15:39:40.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:39.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:39.785585+0000) 2022-09-10T15:39:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:39 smithi110 ceph-mon[97633]: pgmap v21450: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:39 smithi104 ceph-mon[97044]: pgmap v21450: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:40.787 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:40.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:40.416359+0000) 2022-09-10T15:39:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:40.784+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:40.785731+0000) 2022-09-10T15:39:41.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:41.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:41.416619+0000) 2022-09-10T15:39:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:41.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:41.785985+0000) 2022-09-10T15:39:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:41 smithi110 ceph-mon[97633]: pgmap v21451: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:41 smithi104 ceph-mon[97044]: pgmap v21451: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:42.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:42.415+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:42.416783+0000) 2022-09-10T15:39:43.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:42.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:42.786236+0000) 2022-09-10T15:39:43.788 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:43.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:43.416956+0000) 2022-09-10T15:39:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:43.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:43.786438+0000) 2022-09-10T15:39:44.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:43 smithi110 ceph-mon[97633]: pgmap v21452: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:43 smithi104 ceph-mon[97044]: pgmap v21452: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:44.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:44.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:44.417212+0000) 2022-09-10T15:39:45.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:44.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:44.786681+0000) 2022-09-10T15:39:45.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:45.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:45.417394+0000) 2022-09-10T15:39:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:45.785+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:45.786900+0000) 2022-09-10T15:39:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:45 smithi110 ceph-mon[97633]: pgmap v21453: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:45 smithi104 ceph-mon[97044]: pgmap v21453: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:46.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:46.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:46.417593+0000) 2022-09-10T15:39:47.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:46.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:46.787119+0000) 2022-09-10T15:39:47.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:47.416+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:47.417783+0000) 2022-09-10T15:39:48.059 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:47.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:47.787261+0000) 2022-09-10T15:39:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:47 smithi110 ceph-mon[97633]: pgmap v21454: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:39:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:39:48.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:39:48.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:39:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:47 smithi104 ceph-mon[97044]: pgmap v21454: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:39:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:39:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:39:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:39:48.789 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:48.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:48.418008+0000) 2022-09-10T15:39:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:48.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:48.787375+0000) 2022-09-10T15:39:49.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:49.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:49.418272+0000) 2022-09-10T15:39:50.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:49.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:49.787536+0000) 2022-09-10T15:39:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:49 smithi110 ceph-mon[97633]: pgmap v21455: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:50.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:49 smithi104 ceph-mon[97044]: pgmap v21455: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:50.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:50.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:50.418470+0000) 2022-09-10T15:39:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:50.786+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:50.787674+0000) 2022-09-10T15:39:51.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:51.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:51.418682+0000) 2022-09-10T15:39:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:51.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:51.787926+0000) 2022-09-10T15:39:52.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:51 smithi104 ceph-mon[97044]: pgmap v21456: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:51 smithi110 ceph-mon[97633]: pgmap v21456: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:52.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:52.417+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:52.418822+0000) 2022-09-10T15:39:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:52.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:52.788169+0000) 2022-09-10T15:39:53.423 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:39:53.155+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:39:53.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:53.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:53.419035+0000) 2022-09-10T15:39:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:53.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:53.788398+0000) 2022-09-10T15:39:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:53 smithi104 ceph-mon[97044]: pgmap v21457: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:53 smithi110 ceph-mon[97633]: pgmap v21457: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:54.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:54.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:54.419218+0000) 2022-09-10T15:39:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:54.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:54.788604+0000) 2022-09-10T15:39:55.790 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:55.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:55.419398+0000) 2022-09-10T15:39:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:55.787+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:55.788813+0000) 2022-09-10T15:39:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:55 smithi104 ceph-mon[97044]: pgmap v21458: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:56.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:55 smithi110 ceph-mon[97633]: pgmap v21458: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:39:56.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:56.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:56.419588+0000) 2022-09-10T15:39:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:56.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:56.789053+0000) 2022-09-10T15:39:57.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:57.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:57.419817+0000) 2022-09-10T15:39:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:57.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:57.789350+0000) 2022-09-10T15:39:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:57 smithi104 ceph-mon[97044]: pgmap v21459: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:57 smithi110 ceph-mon[97633]: pgmap v21459: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:39:58.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:58.418+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:58.420069+0000) 2022-09-10T15:39:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:58.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:58.789546+0000) 2022-09-10T15:39:59.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:39:59.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:59.420313+0000) 2022-09-10T15:40:00.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:39:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:39:59.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:39:59.789718+0000) 2022-09-10T15:40:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:39:59 smithi104 ceph-mon[97044]: pgmap v21460: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:39:59 smithi110 ceph-mon[97633]: pgmap v21460: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:00.791 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:00.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:00.420479+0000) 2022-09-10T15:40:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:00.788+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:00.789878+0000) 2022-09-10T15:40:01.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:40:01.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:40:01.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:01.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:01.420746+0000) 2022-09-10T15:40:02.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:01.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:01.790123+0000) 2022-09-10T15:40:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:01 smithi104 ceph-mon[97044]: pgmap v21461: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:01 smithi110 ceph-mon[97633]: pgmap v21461: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:02.792 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:02.419+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:02.420964+0000) 2022-09-10T15:40:03.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:02.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:02.790364+0000) 2022-09-10T15:40:03.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:03.420+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:03.421215+0000) 2022-09-10T15:40:04.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:03.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:03.790610+0000) 2022-09-10T15:40:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:03 smithi104 ceph-mon[97044]: pgmap v21462: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:03 smithi110 ceph-mon[97633]: pgmap v21462: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:04.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:04.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:04.421472+0000) 2022-09-10T15:40:05.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:04.789+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:04.790852+0000) 2022-09-10T15:40:05.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:05.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:05.421671+0000) 2022-09-10T15:40:06.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:05.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:05.791065+0000) 2022-09-10T15:40:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:05 smithi104 ceph-mon[97044]: pgmap v21463: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:40:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:05 smithi110 ceph-mon[97633]: pgmap v21463: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:40:06.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:06.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:06.421848+0000) 2022-09-10T15:40:07.060 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:06.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:06.791210+0000) 2022-09-10T15:40:07.793 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:07.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:07.422092+0000) 2022-09-10T15:40:08.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:07.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:07.791347+0000) 2022-09-10T15:40:08.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:07 smithi104 ceph-mon[97044]: pgmap v21464: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:40:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:07 smithi110 ceph-mon[97633]: pgmap v21464: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:40:08.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:40:08.171+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:40:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:08.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:08.422298+0000) 2022-09-10T15:40:09.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:08.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:08.791581+0000) 2022-09-10T15:40:09.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:09.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:09.422539+0000) 2022-09-10T15:40:10.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:09.790+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:09.791834+0000) 2022-09-10T15:40:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:09 smithi104 ceph-mon[97044]: pgmap v21465: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:40:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:09 smithi104 ceph-mon[97044]: osdmap e1422: 12 total, 12 up, 12 in 2022-09-10T15:40:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:09 smithi110 ceph-mon[97633]: pgmap v21465: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:40:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:09 smithi110 ceph-mon[97633]: osdmap e1422: 12 total, 12 up, 12 in 2022-09-10T15:40:10.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:10.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:10.422763+0000) 2022-09-10T15:40:11.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:10.791+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:10.792073+0000) 2022-09-10T15:40:11.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:11.421+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:11.422963+0000) 2022-09-10T15:40:12.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:11.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:11.792364+0000) 2022-09-10T15:40:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:11 smithi104 ceph-mon[97044]: pgmap v21467: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:40:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:11 smithi110 ceph-mon[97633]: pgmap v21467: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:40:12.794 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:12.422+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:12.423215+0000) 2022-09-10T15:40:13.058 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:12.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:12.792548+0000) 2022-09-10T15:40:13.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:13.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:13.423414+0000) 2022-09-10T15:40:14.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:13.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:13.792780+0000) 2022-09-10T15:40:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:13 smithi104 ceph-mon[97044]: pgmap v21468: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:40:14.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:13 smithi110 ceph-mon[97633]: pgmap v21468: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:40:14.795 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:14.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:14.423625+0000) 2022-09-10T15:40:15.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:14.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:14.793049+0000) 2022-09-10T15:40:15.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:15.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:15.423827+0000) 2022-09-10T15:40:16.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:15.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:15.793294+0000) 2022-09-10T15:40:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:15 smithi104 ceph-mon[97044]: pgmap v21469: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:40:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:15 smithi110 ceph-mon[97633]: pgmap v21469: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:40:16.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:16.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:16.424071+0000) 2022-09-10T15:40:17.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:16.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:16.793518+0000) 2022-09-10T15:40:17.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:17.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:17.424309+0000) 2022-09-10T15:40:18.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:17.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:17.793752+0000) 2022-09-10T15:40:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:17 smithi104 ceph-mon[97044]: pgmap v21470: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:40:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:17 smithi104 ceph-mon[97044]: osdmap e1423: 12 total, 12 up, 12 in 2022-09-10T15:40:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:17 smithi110 ceph-mon[97633]: pgmap v21470: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:40:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:17 smithi110 ceph-mon[97633]: osdmap e1423: 12 total, 12 up, 12 in 2022-09-10T15:40:18.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:18.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:18.424506+0000) 2022-09-10T15:40:19.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:18.792+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:18.794024+0000) 2022-09-10T15:40:19.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:19.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:19.424686+0000) 2022-09-10T15:40:20.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:19.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:19.794276+0000) 2022-09-10T15:40:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:19 smithi104 ceph-mon[97044]: pgmap v21472: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:40:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:19 smithi110 ceph-mon[97633]: pgmap v21472: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 210 B/s wr, 10 op/s 2022-09-10T15:40:20.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:20.423+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:20.424876+0000) 2022-09-10T15:40:21.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:20.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:20.794482+0000) 2022-09-10T15:40:21.796 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:21.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:21.425123+0000) 2022-09-10T15:40:22.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:21.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:21.794631+0000) 2022-09-10T15:40:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:21 smithi104 ceph-mon[97044]: pgmap v21473: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:40:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:21 smithi110 ceph-mon[97633]: pgmap v21473: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:40:22.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:22.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:22.425342+0000) 2022-09-10T15:40:23.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:22.793+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:22.794847+0000) 2022-09-10T15:40:23.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:40:23.186+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:40:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:23.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:23.425549+0000) 2022-09-10T15:40:24.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:23.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:23.795133+0000) 2022-09-10T15:40:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:23 smithi104 ceph-mon[97044]: pgmap v21474: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:40:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:23 smithi110 ceph-mon[97633]: pgmap v21474: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:40:24.797 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:24.424+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:24.425748+0000) 2022-09-10T15:40:25.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:24.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:24.795318+0000) 2022-09-10T15:40:25.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:25.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:25.425988+0000) 2022-09-10T15:40:26.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:25.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:25.795523+0000) 2022-09-10T15:40:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:25 smithi104 ceph-mon[97044]: pgmap v21475: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:40:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:25 smithi110 ceph-mon[97633]: pgmap v21475: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:40:26.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:26.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:26.426157+0000) 2022-09-10T15:40:27.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:26.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:26.795747+0000) 2022-09-10T15:40:27.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:27.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:27.426328+0000) 2022-09-10T15:40:28.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:27.794+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:27.795973+0000) 2022-09-10T15:40:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:27 smithi104 ceph-mon[97044]: pgmap v21476: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:40:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:27 smithi110 ceph-mon[97633]: pgmap v21476: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:40:28.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:28.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:28.426526+0000) 2022-09-10T15:40:29.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:28.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:28.796190+0000) 2022-09-10T15:40:29.798 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:29.425+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:29.426726+0000) 2022-09-10T15:40:30.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:29.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:29.796373+0000) 2022-09-10T15:40:30.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:29 smithi104 ceph-mon[97044]: pgmap v21477: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:40:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:29 smithi110 ceph-mon[97633]: pgmap v21477: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:40:30.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:30.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:30.426925+0000) 2022-09-10T15:40:31.057 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:30.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:30.796630+0000) 2022-09-10T15:40:31.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:31.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:31.427188+0000) 2022-09-10T15:40:32.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:31.795+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:31.796842+0000) 2022-09-10T15:40:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:31 smithi104 ceph-mon[97044]: pgmap v21478: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:31 smithi110 ceph-mon[97633]: pgmap v21478: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:32.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:32.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:32.427326+0000) 2022-09-10T15:40:33.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:32.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:32.797064+0000) 2022-09-10T15:40:33.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:33.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:33.427508+0000) 2022-09-10T15:40:34.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:33.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:33.797293+0000) 2022-09-10T15:40:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:33 smithi104 ceph-mon[97044]: pgmap v21479: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:33 smithi110 ceph-mon[97633]: pgmap v21479: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:34.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:34.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:34.427696+0000) 2022-09-10T15:40:35.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:34.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:34.797484+0000) 2022-09-10T15:40:35.799 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:35.426+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:35.427899+0000) 2022-09-10T15:40:36.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:35.796+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:35.797704+0000) 2022-09-10T15:40:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:35 smithi104 ceph-mon[97044]: pgmap v21480: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:35 smithi110 ceph-mon[97633]: pgmap v21480: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:36.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:36.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:36.428167+0000) 2022-09-10T15:40:37.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:36.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:36.797909+0000) 2022-09-10T15:40:37.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:37.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:37.428405+0000) 2022-09-10T15:40:38.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:37.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:37.798105+0000) 2022-09-10T15:40:38.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:37 smithi104 ceph-mon[97044]: pgmap v21481: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:37 smithi110 ceph-mon[97633]: pgmap v21481: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:38.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:40:38.202+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:40:38.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:38.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:38.428555+0000) 2022-09-10T15:40:39.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:38.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:38.798317+0000) 2022-09-10T15:40:39.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:39.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:39.428704+0000) 2022-09-10T15:40:40.056 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:39.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:39.798527+0000) 2022-09-10T15:40:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:39 smithi104 ceph-mon[97044]: pgmap v21482: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:39 smithi110 ceph-mon[97633]: pgmap v21482: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:40.800 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:40.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:40.428878+0000) 2022-09-10T15:40:41.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:40.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:40.798676+0000) 2022-09-10T15:40:41.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:41.427+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:41.429082+0000) 2022-09-10T15:40:42.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:41.797+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:41.798859+0000) 2022-09-10T15:40:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:41 smithi104 ceph-mon[97044]: pgmap v21483: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:41 smithi110 ceph-mon[97633]: pgmap v21483: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:42.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:42.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:42.429269+0000) 2022-09-10T15:40:43.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:42.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:42.799154+0000) 2022-09-10T15:40:43.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:43.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:43.429518+0000) 2022-09-10T15:40:44.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:43.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:43.799381+0000) 2022-09-10T15:40:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:43 smithi104 ceph-mon[97044]: pgmap v21484: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:43 smithi110 ceph-mon[97633]: pgmap v21484: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:44.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:44.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:44.429682+0000) 2022-09-10T15:40:45.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:44.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:44.799642+0000) 2022-09-10T15:40:45.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:45.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:45.429861+0000) 2022-09-10T15:40:46.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:45.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:45.799843+0000) 2022-09-10T15:40:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:46 smithi104 ceph-mon[97044]: pgmap v21485: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:46 smithi110 ceph-mon[97633]: pgmap v21485: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:46.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:46.428+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:46.430022+0000) 2022-09-10T15:40:47.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:46.798+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:46.800034+0000) 2022-09-10T15:40:47.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:47.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:47.430300+0000) 2022-09-10T15:40:48.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:47.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:47.800245+0000) 2022-09-10T15:40:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:48 smithi104 ceph-mon[97044]: pgmap v21486: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:40:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:40:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:40:48.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:40:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:48 smithi110 ceph-mon[97633]: pgmap v21486: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:40:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:40:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:40:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:40:48.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:48.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:48.430477+0000) 2022-09-10T15:40:49.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:48.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:48.800447+0000) 2022-09-10T15:40:49.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:49.429+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:49.430625+0000) 2022-09-10T15:40:50.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:49.799+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:49.800623+0000) 2022-09-10T15:40:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:50 smithi104 ceph-mon[97044]: pgmap v21487: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:50 smithi110 ceph-mon[97633]: pgmap v21487: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:50.802 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:50.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:50.430858+0000) 2022-09-10T15:40:51.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:50.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:50.800839+0000) 2022-09-10T15:40:51.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:51.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:51.431061+0000) 2022-09-10T15:40:52.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:51.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:51.801066+0000) 2022-09-10T15:40:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:52 smithi104 ceph-mon[97044]: pgmap v21488: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:52 smithi110 ceph-mon[97633]: pgmap v21488: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:52.803 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:52.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:52.431343+0000) 2022-09-10T15:40:53.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:52.800+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:52.801325+0000) 2022-09-10T15:40:53.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:40:53.205+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:40:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:53.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:53.431600+0000) 2022-09-10T15:40:54.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:53.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:53.801543+0000) 2022-09-10T15:40:54.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:54 smithi104 ceph-mon[97044]: pgmap v21489: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:54 smithi110 ceph-mon[97633]: pgmap v21489: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:54.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:54.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:54.431788+0000) 2022-09-10T15:40:55.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:54.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:54.801762+0000) 2022-09-10T15:40:55.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:55.430+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:55.432025+0000) 2022-09-10T15:40:56.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:55.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:55.801991+0000) 2022-09-10T15:40:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:56 smithi104 ceph-mon[97044]: pgmap v21490: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:56 smithi110 ceph-mon[97633]: pgmap v21490: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:40:56.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:56.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:56.432247+0000) 2022-09-10T15:40:57.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:56.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:56.802239+0000) 2022-09-10T15:40:57.804 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:57.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:57.432434+0000) 2022-09-10T15:40:58.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:57.801+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:57.802481+0000) 2022-09-10T15:40:58.306 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:40:58 smithi104 ceph-mon[97044]: pgmap v21491: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:40:58 smithi110 ceph-mon[97633]: pgmap v21491: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:40:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:58.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:58.432623+0000) 2022-09-10T15:40:59.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:58.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:58.802731+0000) 2022-09-10T15:40:59.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:40:59.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:59.432839+0000) 2022-09-10T15:41:00.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:40:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:40:59.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:40:59.802986+0000) 2022-09-10T15:41:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:00 smithi104 ceph-mon[97044]: pgmap v21492: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:00 smithi110 ceph-mon[97633]: pgmap v21492: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:00.431+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:00.433022+0000) 2022-09-10T15:41:01.055 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:00.802+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:00.803221+0000) 2022-09-10T15:41:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:01.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:01.433212+0000) 2022-09-10T15:41:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:02 smithi104 ceph-mon[97044]: pgmap v21493: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:01.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:01.803428+0000) 2022-09-10T15:41:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:02 smithi110 ceph-mon[97633]: pgmap v21493: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:02.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:02.433434+0000) 2022-09-10T15:41:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:02.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:02.803645+0000) 2022-09-10T15:41:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:03.432+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:03.433566+0000) 2022-09-10T15:41:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:04 smithi104 ceph-mon[97044]: pgmap v21494: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:03.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:03.803988+0000) 2022-09-10T15:41:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:04 smithi110 ceph-mon[97633]: pgmap v21494: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:04.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:04.433743+0000) 2022-09-10T15:41:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:04.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:04.804165+0000) 2022-09-10T15:41:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:05.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:05.433992+0000) 2022-09-10T15:41:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:06 smithi104 ceph-mon[97044]: pgmap v21495: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:41:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:05.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:05.804305+0000) 2022-09-10T15:41:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:06 smithi110 ceph-mon[97633]: pgmap v21495: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:41:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:06.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:06.434199+0000) 2022-09-10T15:41:07.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:06.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:06.804445+0000) 2022-09-10T15:41:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:07.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:07.434388+0000) 2022-09-10T15:41:08.224 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:07.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:07.804578+0000) 2022-09-10T15:41:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:08 smithi104 ceph-mon[97044]: pgmap v21496: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:41:08.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:08 smithi110 ceph-mon[97633]: pgmap v21496: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:41:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:08.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:08.434547+0000) 2022-09-10T15:41:08.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:41:08.222+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:41:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:08.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:08.804749+0000) 2022-09-10T15:41:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:09.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:09.434752+0000) 2022-09-10T15:41:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:09 smithi110 ceph-mon[97633]: pgmap v21497: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:41:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:09 smithi110 ceph-mon[97633]: osdmap e1424: 12 total, 12 up, 12 in 2022-09-10T15:41:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:09 smithi104 ceph-mon[97044]: pgmap v21497: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:41:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:09 smithi104 ceph-mon[97044]: osdmap e1424: 12 total, 12 up, 12 in 2022-09-10T15:41:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:09.803+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:09.804935+0000) 2022-09-10T15:41:10.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:10.433+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:10.434954+0000) 2022-09-10T15:41:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:10.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:10.805166+0000) 2022-09-10T15:41:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:11.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:11.435190+0000) 2022-09-10T15:41:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:11 smithi110 ceph-mon[97633]: pgmap v21499: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:41:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:11 smithi104 ceph-mon[97044]: pgmap v21499: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:41:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:11.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:11.805349+0000) 2022-09-10T15:41:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:12.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:12.435424+0000) 2022-09-10T15:41:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:12.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:12.805539+0000) 2022-09-10T15:41:13.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:13.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:13.435640+0000) 2022-09-10T15:41:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:13 smithi110 ceph-mon[97633]: pgmap v21500: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:41:14.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:13 smithi110 ceph-mon[97633]: osdmap e1425: 12 total, 12 up, 12 in 2022-09-10T15:41:14.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:13 smithi104 ceph-mon[97044]: pgmap v21500: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:41:14.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:13 smithi104 ceph-mon[97044]: osdmap e1425: 12 total, 12 up, 12 in 2022-09-10T15:41:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:13.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:13.805697+0000) 2022-09-10T15:41:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:14.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:14.435877+0000) 2022-09-10T15:41:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:14.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:14.805872+0000) 2022-09-10T15:41:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:15.434+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:15.436069+0000) 2022-09-10T15:41:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:15 smithi110 ceph-mon[97633]: pgmap v21502: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:41:16.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:15 smithi104 ceph-mon[97044]: pgmap v21502: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:41:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:15.804+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:15.806057+0000) 2022-09-10T15:41:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:16.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:16.436218+0000) 2022-09-10T15:41:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:16.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:16.806230+0000) 2022-09-10T15:41:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:17.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:17.436433+0000) 2022-09-10T15:41:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:17 smithi110 ceph-mon[97633]: pgmap v21503: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:41:18.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:17 smithi104 ceph-mon[97044]: pgmap v21503: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:41:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:17.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:17.806417+0000) 2022-09-10T15:41:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:18.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:18.436602+0000) 2022-09-10T15:41:19.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:18.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:18.806626+0000) 2022-09-10T15:41:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:19.435+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:19.436825+0000) 2022-09-10T15:41:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:19 smithi110 ceph-mon[97633]: pgmap v21504: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:41:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:19 smithi104 ceph-mon[97044]: pgmap v21504: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 207 B/s wr, 10 op/s 2022-09-10T15:41:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:19.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:19.806828+0000) 2022-09-10T15:41:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:20.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:20.437057+0000) 2022-09-10T15:41:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:20.805+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:20.806953+0000) 2022-09-10T15:41:21.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:21.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:21.437313+0000) 2022-09-10T15:41:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:21 smithi110 ceph-mon[97633]: pgmap v21505: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:41:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:21 smithi104 ceph-mon[97044]: pgmap v21505: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:41:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:21.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:21.807134+0000) 2022-09-10T15:41:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:22.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:22.437489+0000) 2022-09-10T15:41:23.229 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:22.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:22.807291+0000) 2022-09-10T15:41:23.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:41:23.226+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:41:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:23.436+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:23.437676+0000) 2022-09-10T15:41:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:23 smithi110 ceph-mon[97633]: pgmap v21506: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:41:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:23 smithi104 ceph-mon[97044]: pgmap v21506: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:41:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:23.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:23.807488+0000) 2022-09-10T15:41:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:24.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:24.437892+0000) 2022-09-10T15:41:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:24.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:24.807692+0000) 2022-09-10T15:41:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:25.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:25.438077+0000) 2022-09-10T15:41:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:25 smithi110 ceph-mon[97633]: pgmap v21507: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:41:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:25 smithi104 ceph-mon[97044]: pgmap v21507: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:41:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:25.806+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:25.807901+0000) 2022-09-10T15:41:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:26.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:26.438321+0000) 2022-09-10T15:41:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:26.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:26.808141+0000) 2022-09-10T15:41:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:27.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:27.438554+0000) 2022-09-10T15:41:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:27 smithi110 ceph-mon[97633]: pgmap v21508: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:27 smithi104 ceph-mon[97044]: pgmap v21508: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:27.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:27.808356+0000) 2022-09-10T15:41:28.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:28.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:28.438825+0000) 2022-09-10T15:41:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:28.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:28.808512+0000) 2022-09-10T15:41:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:29.437+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:29.439035+0000) 2022-09-10T15:41:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:29 smithi110 ceph-mon[97633]: pgmap v21509: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:30.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:29 smithi104 ceph-mon[97044]: pgmap v21509: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:30.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:29.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:29.808736+0000) 2022-09-10T15:41:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:30.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:30.439155+0000) 2022-09-10T15:41:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:30.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:30.808897+0000) 2022-09-10T15:41:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:31.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:31.439296+0000) 2022-09-10T15:41:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:31 smithi110 ceph-mon[97633]: pgmap v21510: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:31 smithi104 ceph-mon[97044]: pgmap v21510: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:31.807+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:31.809036+0000) 2022-09-10T15:41:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:32.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:32.439511+0000) 2022-09-10T15:41:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:32.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:32.809210+0000) 2022-09-10T15:41:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:33.438+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:33.439668+0000) 2022-09-10T15:41:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:33 smithi110 ceph-mon[97633]: pgmap v21511: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:33 smithi104 ceph-mon[97044]: pgmap v21511: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:33.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:33.809333+0000) 2022-09-10T15:41:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:34.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:34.439881+0000) 2022-09-10T15:41:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:34.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:34.809524+0000) 2022-09-10T15:41:35.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:35.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:35.440179+0000) 2022-09-10T15:41:36.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:35 smithi110 ceph-mon[97633]: pgmap v21512: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:35 smithi104 ceph-mon[97044]: pgmap v21512: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:36.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:35.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:35.809681+0000) 2022-09-10T15:41:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:36.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:36.440392+0000) 2022-09-10T15:41:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:36.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:36.809810+0000) 2022-09-10T15:41:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:37.439+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:37.440615+0000) 2022-09-10T15:41:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:37 smithi110 ceph-mon[97633]: pgmap v21513: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:38.239 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:37.808+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:37.810018+0000) 2022-09-10T15:41:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:37 smithi104 ceph-mon[97044]: pgmap v21513: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:38.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:41:38.236+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:41:38.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:38.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:38.440804+0000) 2022-09-10T15:41:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:38.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:38.810240+0000) 2022-09-10T15:41:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:39.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:39.441009+0000) 2022-09-10T15:41:40.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:39 smithi110 ceph-mon[97633]: pgmap v21514: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:39 smithi104 ceph-mon[97044]: pgmap v21514: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:39.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:39.810431+0000) 2022-09-10T15:41:40.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:40.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:40.441189+0000) 2022-09-10T15:41:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:40.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:40.810616+0000) 2022-09-10T15:41:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:41.440+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:41.441424+0000) 2022-09-10T15:41:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:41 smithi110 ceph-mon[97633]: pgmap v21515: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:41 smithi104 ceph-mon[97044]: pgmap v21515: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:41.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:41.810773+0000) 2022-09-10T15:41:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:42.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:42.441632+0000) 2022-09-10T15:41:43.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:42.809+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:42.810961+0000) 2022-09-10T15:41:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:43.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:43.441877+0000) 2022-09-10T15:41:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:43 smithi110 ceph-mon[97633]: pgmap v21516: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:43 smithi104 ceph-mon[97044]: pgmap v21516: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:43.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:43.811197+0000) 2022-09-10T15:41:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:44.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:44.442088+0000) 2022-09-10T15:41:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:44.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:44.811452+0000) 2022-09-10T15:41:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:45.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:45.442323+0000) 2022-09-10T15:41:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:45 smithi110 ceph-mon[97633]: pgmap v21517: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:45 smithi104 ceph-mon[97044]: pgmap v21517: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:45.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:45.811611+0000) 2022-09-10T15:41:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:46.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:46.442525+0000) 2022-09-10T15:41:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:46.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:46.811808+0000) 2022-09-10T15:41:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:47.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:47.442735+0000) 2022-09-10T15:41:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:47 smithi110 ceph-mon[97633]: pgmap v21518: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:41:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:41:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:41:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:41:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:47 smithi104 ceph-mon[97044]: pgmap v21518: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:41:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:41:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:41:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:41:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:47.810+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:47.811964+0000) 2022-09-10T15:41:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:48.441+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:48.442916+0000) 2022-09-10T15:41:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:48.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:48.812210+0000) 2022-09-10T15:41:49.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:49.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:49.443154+0000) 2022-09-10T15:41:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:49 smithi110 ceph-mon[97633]: pgmap v21519: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:49 smithi104 ceph-mon[97044]: pgmap v21519: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:49.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:49.812382+0000) 2022-09-10T15:41:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:50.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:50.443327+0000) 2022-09-10T15:41:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:50.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:50.812603+0000) 2022-09-10T15:41:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:51.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:51.443571+0000) 2022-09-10T15:41:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:51 smithi110 ceph-mon[97633]: pgmap v21520: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:51 smithi104 ceph-mon[97044]: pgmap v21520: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:51.811+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:51.812859+0000) 2022-09-10T15:41:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:52.442+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:52.443770+0000) 2022-09-10T15:41:53.247 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:52.812+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:52.813146+0000) 2022-09-10T15:41:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:41:53.244+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:41:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:53.443+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:53.443990+0000) 2022-09-10T15:41:54.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:53 smithi110 ceph-mon[97633]: pgmap v21521: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:53 smithi104 ceph-mon[97044]: pgmap v21521: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:53.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:53.813395+0000) 2022-09-10T15:41:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:54.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:54.444184+0000) 2022-09-10T15:41:55.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:54.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:54.813633+0000) 2022-09-10T15:41:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:55.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:55.444407+0000) 2022-09-10T15:41:56.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:55 smithi110 ceph-mon[97633]: pgmap v21522: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:56.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:55 smithi104 ceph-mon[97044]: pgmap v21522: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:41:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:55.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:55.813900+0000) 2022-09-10T15:41:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:56.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:56.444644+0000) 2022-09-10T15:41:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:56.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:56.814161+0000) 2022-09-10T15:41:57.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:57.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:57.444875+0000) 2022-09-10T15:41:58.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:57 smithi110 ceph-mon[97633]: pgmap v21523: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:58.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:57 smithi104 ceph-mon[97044]: pgmap v21523: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:41:58.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:57.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:57.814329+0000) 2022-09-10T15:41:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:58.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:58.445187+0000) 2022-09-10T15:41:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:58.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:58.814516+0000) 2022-09-10T15:41:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:41:59.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:59.445391+0000) 2022-09-10T15:42:00.163 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:41:59 smithi110 ceph-mon[97633]: pgmap v21524: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:00.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:41:59 smithi104 ceph-mon[97044]: pgmap v21524: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:00.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:41:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:41:59.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:41:59.814724+0000) 2022-09-10T15:42:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:00.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:00.445581+0000) 2022-09-10T15:42:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:00.813+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:00.814912+0000) 2022-09-10T15:42:01.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:01.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:01.445762+0000) 2022-09-10T15:42:02.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:01 smithi110 ceph-mon[97633]: pgmap v21525: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:01 smithi104 ceph-mon[97044]: pgmap v21525: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:02.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:01.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:01.815140+0000) 2022-09-10T15:42:02.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:02.444+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:02.445949+0000) 2022-09-10T15:42:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:02.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:02.815313+0000) 2022-09-10T15:42:03.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:03.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:03.446154+0000) 2022-09-10T15:42:04.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:03 smithi110 ceph-mon[97633]: pgmap v21526: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:04.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:03 smithi104 ceph-mon[97044]: pgmap v21526: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:04.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:03.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:03.815497+0000) 2022-09-10T15:42:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:04.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:04.446375+0000) 2022-09-10T15:42:05.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:04.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:04.815735+0000) 2022-09-10T15:42:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:05.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:05.446588+0000) 2022-09-10T15:42:06.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:05 smithi110 ceph-mon[97633]: pgmap v21527: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:42:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:05 smithi104 ceph-mon[97044]: pgmap v21527: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:42:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:05.814+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:05.815903+0000) 2022-09-10T15:42:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:06.445+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:06.446786+0000) 2022-09-10T15:42:07.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:06.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:06.816110+0000) 2022-09-10T15:42:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:07.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:07.446999+0000) 2022-09-10T15:42:08.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:07 smithi110 ceph-mon[97633]: pgmap v21528: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:42:08.261 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:07.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:07.816378+0000) 2022-09-10T15:42:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:07 smithi104 ceph-mon[97044]: pgmap v21528: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:42:08.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:42:08.258+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:42:08.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:08.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:08.447249+0000) 2022-09-10T15:42:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:08.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:08.816639+0000) 2022-09-10T15:42:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:09.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:09.447466+0000) 2022-09-10T15:42:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:09 smithi110 ceph-mon[97633]: pgmap v21529: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:42:10.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:09 smithi110 ceph-mon[97633]: osdmap e1426: 12 total, 12 up, 12 in 2022-09-10T15:42:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:09 smithi104 ceph-mon[97044]: pgmap v21529: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2022-09-10T15:42:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:09 smithi104 ceph-mon[97044]: osdmap e1426: 12 total, 12 up, 12 in 2022-09-10T15:42:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:09.815+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:09.816851+0000) 2022-09-10T15:42:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:10.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:10.447646+0000) 2022-09-10T15:42:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:10.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:10.817155+0000) 2022-09-10T15:42:11.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:11.446+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:11.447885+0000) 2022-09-10T15:42:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:11 smithi110 ceph-mon[97633]: pgmap v21531: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:42:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:11 smithi104 ceph-mon[97044]: pgmap v21531: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:42:12.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:11.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:11.817344+0000) 2022-09-10T15:42:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:12.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:12.448140+0000) 2022-09-10T15:42:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:12.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:12.817611+0000) 2022-09-10T15:42:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:13.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:13.448264+0000) 2022-09-10T15:42:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:13 smithi110 ceph-mon[97633]: pgmap v21532: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:42:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:13 smithi104 ceph-mon[97044]: pgmap v21532: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:42:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:13.816+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:13.817839+0000) 2022-09-10T15:42:14.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:14.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:14.448485+0000) 2022-09-10T15:42:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:14.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:14.818031+0000) 2022-09-10T15:42:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:15.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:15.448651+0000) 2022-09-10T15:42:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:15 smithi110 ceph-mon[97633]: pgmap v21533: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:42:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:15 smithi104 ceph-mon[97044]: pgmap v21533: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:42:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:15.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:15.818177+0000) 2022-09-10T15:42:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:16.447+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:16.448846+0000) 2022-09-10T15:42:17.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:16.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:16.818358+0000) 2022-09-10T15:42:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:17.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:17.449046+0000) 2022-09-10T15:42:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:17 smithi104 ceph-mon[97044]: pgmap v21534: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:42:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:17 smithi104 ceph-mon[97044]: osdmap e1427: 12 total, 12 up, 12 in 2022-09-10T15:42:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:17.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:17.818554+0000) 2022-09-10T15:42:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:17 smithi110 ceph-mon[97633]: pgmap v21534: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:42:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:17 smithi110 ceph-mon[97633]: osdmap e1427: 12 total, 12 up, 12 in 2022-09-10T15:42:18.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:18.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:18.449194+0000) 2022-09-10T15:42:19.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:18.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:18.818783+0000) 2022-09-10T15:42:19.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:19.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:19.449386+0000) 2022-09-10T15:42:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:19 smithi104 ceph-mon[97044]: pgmap v21536: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:42:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:19.817+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:19.818942+0000) 2022-09-10T15:42:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:19 smithi110 ceph-mon[97633]: pgmap v21536: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:42:20.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:20.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:20.449620+0000) 2022-09-10T15:42:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:20.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:20.819200+0000) 2022-09-10T15:42:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:21.448+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:21.449820+0000) 2022-09-10T15:42:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:21 smithi104 ceph-mon[97044]: pgmap v21537: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:42:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:21.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:21.819477+0000) 2022-09-10T15:42:22.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:21 smithi110 ceph-mon[97633]: pgmap v21537: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:42:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:22.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:22.450076+0000) 2022-09-10T15:42:23.273 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:22.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:22.819687+0000) 2022-09-10T15:42:23.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:42:23.270+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:42:23.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:23.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:23.450294+0000) 2022-09-10T15:42:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:23 smithi104 ceph-mon[97044]: pgmap v21538: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:42:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:23.818+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:23.819918+0000) 2022-09-10T15:42:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:23 smithi110 ceph-mon[97633]: pgmap v21538: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:42:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:24.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:24.450489+0000) 2022-09-10T15:42:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:24.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:24.820208+0000) 2022-09-10T15:42:25.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:25.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:25.450751+0000) 2022-09-10T15:42:26.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:25 smithi104 ceph-mon[97044]: pgmap v21539: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:42:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:25.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:25.820466+0000) 2022-09-10T15:42:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:25 smithi110 ceph-mon[97633]: pgmap v21539: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:42:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:26.449+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:26.450975+0000) 2022-09-10T15:42:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:26.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:26.820755+0000) 2022-09-10T15:42:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:27.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:27.451206+0000) 2022-09-10T15:42:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:27 smithi104 ceph-mon[97044]: pgmap v21540: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:42:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:27.819+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:27.820994+0000) 2022-09-10T15:42:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:27 smithi110 ceph-mon[97633]: pgmap v21540: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:42:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:28.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:28.451365+0000) 2022-09-10T15:42:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:28.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:28.821237+0000) 2022-09-10T15:42:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:29.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:29.451529+0000) 2022-09-10T15:42:30.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:29 smithi104 ceph-mon[97044]: pgmap v21541: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:42:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:29.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:29.821406+0000) 2022-09-10T15:42:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:29 smithi110 ceph-mon[97633]: pgmap v21541: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:42:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:30.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:30.451769+0000) 2022-09-10T15:42:31.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:30.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:30.821639+0000) 2022-09-10T15:42:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:31.450+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:31.452022+0000) 2022-09-10T15:42:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:31 smithi104 ceph-mon[97044]: pgmap v21542: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:31.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:31.821847+0000) 2022-09-10T15:42:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:31 smithi110 ceph-mon[97633]: pgmap v21542: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:32.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:32.452244+0000) 2022-09-10T15:42:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:32.820+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:32.821995+0000) 2022-09-10T15:42:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:33.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:33.452447+0000) 2022-09-10T15:42:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:33 smithi104 ceph-mon[97044]: pgmap v21543: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:33.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:33.822238+0000) 2022-09-10T15:42:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:33 smithi110 ceph-mon[97633]: pgmap v21543: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:34.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:34.452625+0000) 2022-09-10T15:42:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:34.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:34.822468+0000) 2022-09-10T15:42:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:35.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:35.452869+0000) 2022-09-10T15:42:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:35 smithi104 ceph-mon[97044]: pgmap v21544: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:35.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:35.822631+0000) 2022-09-10T15:42:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:35 smithi110 ceph-mon[97633]: pgmap v21544: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:36.451+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:36.453047+0000) 2022-09-10T15:42:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:36.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:36.822786+0000) 2022-09-10T15:42:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:37.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:37.453298+0000) 2022-09-10T15:42:38.289 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:37.821+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:37.823032+0000) 2022-09-10T15:42:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:37 smithi104 ceph-mon[97044]: pgmap v21545: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:37 smithi110 ceph-mon[97633]: pgmap v21545: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:38.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:38.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:38.453506+0000) 2022-09-10T15:42:38.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:42:38.286+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:42:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:38.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:38.823256+0000) 2022-09-10T15:42:39.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:39.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:39.453702+0000) 2022-09-10T15:42:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:39 smithi104 ceph-mon[97044]: pgmap v21546: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:39.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:39.823491+0000) 2022-09-10T15:42:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:39 smithi110 ceph-mon[97633]: pgmap v21546: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:40.452+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:40.453911+0000) 2022-09-10T15:42:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:40.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:40.823694+0000) 2022-09-10T15:42:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:41.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:41.454201+0000) 2022-09-10T15:42:42.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:41 smithi104 ceph-mon[97044]: pgmap v21547: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:42.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:41.822+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:41.823891+0000) 2022-09-10T15:42:42.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:41 smithi110 ceph-mon[97633]: pgmap v21547: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:42.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:42.454412+0000) 2022-09-10T15:42:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:42.823+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:42.824165+0000) 2022-09-10T15:42:43.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:43.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:43.454599+0000) 2022-09-10T15:42:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:43 smithi104 ceph-mon[97044]: pgmap v21548: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:43.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:43.824367+0000) 2022-09-10T15:42:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:43 smithi110 ceph-mon[97633]: pgmap v21548: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:44.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:44.454767+0000) 2022-09-10T15:42:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:44.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:44.824569+0000) 2022-09-10T15:42:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:45.453+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:45.454981+0000) 2022-09-10T15:42:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:45 smithi104 ceph-mon[97044]: pgmap v21549: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:45.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:45.824729+0000) 2022-09-10T15:42:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:45 smithi110 ceph-mon[97633]: pgmap v21549: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:46.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:46.455155+0000) 2022-09-10T15:42:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:46.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:46.824908+0000) 2022-09-10T15:42:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:47.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:47.455377+0000) 2022-09-10T15:42:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:47 smithi104 ceph-mon[97044]: pgmap v21550: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:42:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:42:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:42:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:42:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:47.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:47.825188+0000) 2022-09-10T15:42:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:47 smithi110 ceph-mon[97633]: pgmap v21550: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:42:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:42:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:42:48.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:42:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:48.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:48.455590+0000) 2022-09-10T15:42:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:48.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:48.825337+0000) 2022-09-10T15:42:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:49.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:49.455789+0000) 2022-09-10T15:42:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:49 smithi104 ceph-mon[97044]: pgmap v21551: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:49.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:49.825516+0000) 2022-09-10T15:42:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:49 smithi110 ceph-mon[97633]: pgmap v21551: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:50.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:50.455999+0000) 2022-09-10T15:42:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:50.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:50.825780+0000) 2022-09-10T15:42:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:51.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:51.456195+0000) 2022-09-10T15:42:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:51 smithi104 ceph-mon[97044]: pgmap v21552: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:51.824+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:51.825998+0000) 2022-09-10T15:42:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:51 smithi110 ceph-mon[97633]: pgmap v21552: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:52.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:52.456379+0000) 2022-09-10T15:42:53.304 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:52.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:52.826174+0000) 2022-09-10T15:42:53.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:42:53.300+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:42:53.556 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:53.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:53.456541+0000) 2022-09-10T15:42:54.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:53 smithi104 ceph-mon[97044]: pgmap v21553: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:53.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:53.826387+0000) 2022-09-10T15:42:54.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:53 smithi110 ceph-mon[97633]: pgmap v21553: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:54.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:54.456760+0000) 2022-09-10T15:42:55.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:54.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:54.826580+0000) 2022-09-10T15:42:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:55.455+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:55.457007+0000) 2022-09-10T15:42:56.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:55 smithi104 ceph-mon[97044]: pgmap v21554: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:55.825+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:55.826821+0000) 2022-09-10T15:42:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:55 smithi110 ceph-mon[97633]: pgmap v21554: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:42:56.810 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:56.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:56.457257+0000) 2022-09-10T15:42:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:56.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:56.827073+0000) 2022-09-10T15:42:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:57.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:57.457551+0000) 2022-09-10T15:42:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:57 smithi104 ceph-mon[97044]: pgmap v21555: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:57.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:57.827280+0000) 2022-09-10T15:42:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:57 smithi110 ceph-mon[97633]: pgmap v21555: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:42:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:58.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:58.457771+0000) 2022-09-10T15:42:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:58.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:58.827507+0000) 2022-09-10T15:42:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:42:59.456+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:59.457958+0000) 2022-09-10T15:43:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:42:59 smithi104 ceph-mon[97044]: pgmap v21556: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:00.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:42:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:42:59.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:42:59.827798+0000) 2022-09-10T15:43:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:42:59 smithi110 ceph-mon[97633]: pgmap v21556: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:00.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:00.458090+0000) 2022-09-10T15:43:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:00.826+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:00.827988+0000) 2022-09-10T15:43:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:01.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:01.458314+0000) 2022-09-10T15:43:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:01 smithi104 ceph-mon[97044]: pgmap v21557: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:01.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:01.828279+0000) 2022-09-10T15:43:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:01 smithi110 ceph-mon[97633]: pgmap v21557: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:02.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:02.458569+0000) 2022-09-10T15:43:03.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:02.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:02.828510+0000) 2022-09-10T15:43:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:03.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:03.458818+0000) 2022-09-10T15:43:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:03 smithi104 ceph-mon[97044]: pgmap v21558: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:03.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:03.828705+0000) 2022-09-10T15:43:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:03 smithi110 ceph-mon[97633]: pgmap v21558: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:04.457+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:04.459052+0000) 2022-09-10T15:43:05.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:04.827+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:04.828899+0000) 2022-09-10T15:43:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:05.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:05.459312+0000) 2022-09-10T15:43:06.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:05 smithi104 ceph-mon[97044]: pgmap v21559: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:43:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:05.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:05.829132+0000) 2022-09-10T15:43:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:05 smithi110 ceph-mon[97633]: pgmap v21559: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:43:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:06.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:06.459465+0000) 2022-09-10T15:43:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:06.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:06.829359+0000) 2022-09-10T15:43:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:07.458+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:07.459642+0000) 2022-09-10T15:43:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:07 smithi104 ceph-mon[97044]: pgmap v21560: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:43:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:07.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:07.829537+0000) 2022-09-10T15:43:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:07 smithi110 ceph-mon[97633]: pgmap v21560: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:43:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:08.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:08.459868+0000) 2022-09-10T15:43:08.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:43:08.311+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:43:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:08.828+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:08.829695+0000) 2022-09-10T15:43:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:09.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:09.460128+0000) 2022-09-10T15:43:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:09 smithi110 ceph-mon[97633]: pgmap v21561: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:43:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:09 smithi110 ceph-mon[97633]: osdmap e1428: 12 total, 12 up, 12 in 2022-09-10T15:43:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:09 smithi104 ceph-mon[97044]: pgmap v21561: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:43:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:09 smithi104 ceph-mon[97044]: osdmap e1428: 12 total, 12 up, 12 in 2022-09-10T15:43:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:09.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:09.829945+0000) 2022-09-10T15:43:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:10.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:10.460344+0000) 2022-09-10T15:43:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:10.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:10.830163+0000) 2022-09-10T15:43:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:11.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:11.460530+0000) 2022-09-10T15:43:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:11 smithi110 ceph-mon[97633]: pgmap v21563: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:43:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:11 smithi104 ceph-mon[97044]: pgmap v21563: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:43:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:11.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:11.830354+0000) 2022-09-10T15:43:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:12.459+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:12.460746+0000) 2022-09-10T15:43:13.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:12.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:12.830548+0000) 2022-09-10T15:43:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:13.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:13.460992+0000) 2022-09-10T15:43:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:13 smithi110 ceph-mon[97633]: pgmap v21564: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:43:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:13 smithi110 ceph-mon[97633]: osdmap e1429: 12 total, 12 up, 12 in 2022-09-10T15:43:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:13 smithi104 ceph-mon[97044]: pgmap v21564: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:43:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:13 smithi104 ceph-mon[97044]: osdmap e1429: 12 total, 12 up, 12 in 2022-09-10T15:43:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:13.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:13.830750+0000) 2022-09-10T15:43:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:14.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:14.461181+0000) 2022-09-10T15:43:15.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:14.829+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:14.830956+0000) 2022-09-10T15:43:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:15.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:15.461382+0000) 2022-09-10T15:43:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:15 smithi110 ceph-mon[97633]: pgmap v21566: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:43:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:15 smithi104 ceph-mon[97044]: pgmap v21566: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:43:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:15.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:15.831156+0000) 2022-09-10T15:43:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:16.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:16.461604+0000) 2022-09-10T15:43:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:16.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:16.831395+0000) 2022-09-10T15:43:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:17.460+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:17.461914+0000) 2022-09-10T15:43:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:17 smithi110 ceph-mon[97633]: pgmap v21567: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:43:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:17 smithi104 ceph-mon[97044]: pgmap v21567: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:43:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:17.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:17.831621+0000) 2022-09-10T15:43:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:18.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:18.462086+0000) 2022-09-10T15:43:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:18.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:18.831831+0000) 2022-09-10T15:43:19.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:19.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:19.462265+0000) 2022-09-10T15:43:20.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:19 smithi110 ceph-mon[97633]: pgmap v21568: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:43:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:19 smithi104 ceph-mon[97044]: pgmap v21568: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:43:20.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:19.830+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:19.832060+0000) 2022-09-10T15:43:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:20.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:20.462489+0000) 2022-09-10T15:43:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:20.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:20.832259+0000) 2022-09-10T15:43:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:21.461+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:21.462692+0000) 2022-09-10T15:43:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:21 smithi110 ceph-mon[97633]: pgmap v21569: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:43:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:21 smithi104 ceph-mon[97044]: pgmap v21569: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:43:22.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:21.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:21.832441+0000) 2022-09-10T15:43:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:22.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:22.462870+0000) 2022-09-10T15:43:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:22.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:22.832622+0000) 2022-09-10T15:43:23.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:43:23.322+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:43:23.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:23.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:23.463076+0000) 2022-09-10T15:43:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:23 smithi110 ceph-mon[97633]: pgmap v21570: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:43:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:23 smithi104 ceph-mon[97044]: pgmap v21570: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:43:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:23.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:23.832801+0000) 2022-09-10T15:43:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:24.462+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:24.463326+0000) 2022-09-10T15:43:25.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:24.831+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:24.832963+0000) 2022-09-10T15:43:25.814 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:25.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:25.463541+0000) 2022-09-10T15:43:26.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:25 smithi110 ceph-mon[97633]: pgmap v21571: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:43:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:25 smithi104 ceph-mon[97044]: pgmap v21571: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 86 B/s wr, 12 op/s 2022-09-10T15:43:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:25.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:25.833164+0000) 2022-09-10T15:43:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:26.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:26.463775+0000) 2022-09-10T15:43:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:26.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:26.833419+0000) 2022-09-10T15:43:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:27.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:27.464023+0000) 2022-09-10T15:43:28.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:27 smithi110 ceph-mon[97633]: pgmap v21572: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:27 smithi104 ceph-mon[97044]: pgmap v21572: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:28.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:27.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:27.833615+0000) 2022-09-10T15:43:28.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:28.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:28.464236+0000) 2022-09-10T15:43:29.316 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:28.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:28.833761+0000) 2022-09-10T15:43:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:29.463+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:29.464413+0000) 2022-09-10T15:43:30.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:29 smithi110 ceph-mon[97633]: pgmap v21573: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:30.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:29 smithi104 ceph-mon[97044]: pgmap v21573: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:29.832+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:29.833942+0000) 2022-09-10T15:43:30.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:30.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:30.464651+0000) 2022-09-10T15:43:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:30.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:30.834083+0000) 2022-09-10T15:43:31.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:31.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:31.464827+0000) 2022-09-10T15:43:32.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:31 smithi110 ceph-mon[97633]: pgmap v21574: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:31 smithi104 ceph-mon[97044]: pgmap v21574: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:32.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:31.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:31.834289+0000) 2022-09-10T15:43:32.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:32.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:32.464992+0000) 2022-09-10T15:43:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:32.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:32.834501+0000) 2022-09-10T15:43:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:33.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:33.465271+0000) 2022-09-10T15:43:34.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:33 smithi110 ceph-mon[97633]: pgmap v21575: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:33 smithi104 ceph-mon[97044]: pgmap v21575: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:33.833+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:33.834682+0000) 2022-09-10T15:43:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:34.464+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:34.465416+0000) 2022-09-10T15:43:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:34.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:34.834936+0000) 2022-09-10T15:43:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:35.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:35.465661+0000) 2022-09-10T15:43:36.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:35 smithi110 ceph-mon[97633]: pgmap v21576: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:35 smithi104 ceph-mon[97044]: pgmap v21576: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:35.834+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:35.835199+0000) 2022-09-10T15:43:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:36.465+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:36.465901+0000) 2022-09-10T15:43:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:36.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:36.835365+0000) 2022-09-10T15:43:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:37.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:37.466158+0000) 2022-09-10T15:43:38.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:37 smithi110 ceph-mon[97633]: pgmap v21577: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:37 smithi104 ceph-mon[97044]: pgmap v21577: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:37.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:37.835591+0000) 2022-09-10T15:43:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:43:38.338+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:43:38.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:38.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:38.466305+0000) 2022-09-10T15:43:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:38.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:38.835766+0000) 2022-09-10T15:43:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:39.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:39.466464+0000) 2022-09-10T15:43:40.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:39 smithi110 ceph-mon[97633]: pgmap v21578: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:39 smithi104 ceph-mon[97044]: pgmap v21578: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:40.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:39.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:39.835966+0000) 2022-09-10T15:43:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:40.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:40.466683+0000) 2022-09-10T15:43:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:40.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:40.836227+0000) 2022-09-10T15:43:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:41.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:41.466956+0000) 2022-09-10T15:43:42.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:41 smithi110 ceph-mon[97633]: pgmap v21579: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:41 smithi104 ceph-mon[97044]: pgmap v21579: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:41.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:41.836498+0000) 2022-09-10T15:43:42.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:42.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:42.467216+0000) 2022-09-10T15:43:43.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:42.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:42.836679+0000) 2022-09-10T15:43:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:43.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:43.467388+0000) 2022-09-10T15:43:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:43 smithi110 ceph-mon[97633]: pgmap v21580: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:43 smithi104 ceph-mon[97044]: pgmap v21580: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:43.835+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:43.836920+0000) 2022-09-10T15:43:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:44.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:44.467608+0000) 2022-09-10T15:43:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:44.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:44.837162+0000) 2022-09-10T15:43:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:45.466+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:45.467874+0000) 2022-09-10T15:43:46.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:45 smithi110 ceph-mon[97633]: pgmap v21581: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:45 smithi104 ceph-mon[97044]: pgmap v21581: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:45.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:45.837306+0000) 2022-09-10T15:43:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:46.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:46.468134+0000) 2022-09-10T15:43:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:46.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:46.837559+0000) 2022-09-10T15:43:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:47.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:47.468278+0000) 2022-09-10T15:43:48.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:47 smithi110 ceph-mon[97633]: pgmap v21582: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:43:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:43:48.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:43:48.160 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:43:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:47 smithi104 ceph-mon[97044]: pgmap v21582: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:43:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:43:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:43:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:43:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:47.836+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:47.837808+0000) 2022-09-10T15:43:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:48.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:48.468494+0000) 2022-09-10T15:43:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:48.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:48.837968+0000) 2022-09-10T15:43:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:49.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:49.468754+0000) 2022-09-10T15:43:50.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:49 smithi110 ceph-mon[97633]: pgmap v21583: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:49 smithi104 ceph-mon[97044]: pgmap v21583: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:50.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:49.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:49.838234+0000) 2022-09-10T15:43:50.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:50.467+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:50.468987+0000) 2022-09-10T15:43:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:50.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:50.838475+0000) 2022-09-10T15:43:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:51.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:51.469197+0000) 2022-09-10T15:43:52.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:51 smithi110 ceph-mon[97633]: pgmap v21584: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:51 smithi104 ceph-mon[97044]: pgmap v21584: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:51.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:51.838714+0000) 2022-09-10T15:43:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:52.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:52.469412+0000) 2022-09-10T15:43:53.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:52.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:52.838879+0000) 2022-09-10T15:43:53.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:43:53.351+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:43:53.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:53.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:53.469652+0000) 2022-09-10T15:43:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:53 smithi104 ceph-mon[97044]: pgmap v21585: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:53.837+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:53.839017+0000) 2022-09-10T15:43:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:53 smithi110 ceph-mon[97633]: pgmap v21585: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:54.468+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:54.469834+0000) 2022-09-10T15:43:55.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:54.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:54.839232+0000) 2022-09-10T15:43:55.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:55.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:55.470065+0000) 2022-09-10T15:43:56.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:55 smithi104 ceph-mon[97044]: pgmap v21586: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:55.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:55.839495+0000) 2022-09-10T15:43:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:55 smithi110 ceph-mon[97633]: pgmap v21586: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:43:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:56.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:56.470340+0000) 2022-09-10T15:43:57.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:56.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:56.839694+0000) 2022-09-10T15:43:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:57.469+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:57.470631+0000) 2022-09-10T15:43:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:57 smithi104 ceph-mon[97044]: pgmap v21587: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:58.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:57.838+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:57.839858+0000) 2022-09-10T15:43:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:57 smithi110 ceph-mon[97633]: pgmap v21587: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:43:58.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:58.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:58.470889+0000) 2022-09-10T15:43:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:58.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:58.840085+0000) 2022-09-10T15:43:59.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:43:59.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:59.471135+0000) 2022-09-10T15:44:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:43:59 smithi104 ceph-mon[97044]: pgmap v21588: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:43:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:43:59.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:43:59.840279+0000) 2022-09-10T15:44:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:43:59 smithi110 ceph-mon[97633]: pgmap v21588: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:00.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:00.471379+0000) 2022-09-10T15:44:01.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:00.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:00.840468+0000) 2022-09-10T15:44:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:01.470+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:01.471588+0000) 2022-09-10T15:44:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:01 smithi104 ceph-mon[97044]: pgmap v21589: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:01.839+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:01.840618+0000) 2022-09-10T15:44:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:01 smithi110 ceph-mon[97633]: pgmap v21589: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:02.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:02.471816+0000) 2022-09-10T15:44:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:02.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:02.840845+0000) 2022-09-10T15:44:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:03.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:03.472079+0000) 2022-09-10T15:44:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:03 smithi104 ceph-mon[97044]: pgmap v21590: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:03.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:03.841089+0000) 2022-09-10T15:44:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:03 smithi110 ceph-mon[97633]: pgmap v21590: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:04.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:04.472318+0000) 2022-09-10T15:44:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:04.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:04.841294+0000) 2022-09-10T15:44:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:05.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:05.472530+0000) 2022-09-10T15:44:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:05 smithi104 ceph-mon[97044]: pgmap v21591: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:44:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:05.840+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:05.841477+0000) 2022-09-10T15:44:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:05 smithi110 ceph-mon[97633]: pgmap v21591: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:44:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:06.471+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:06.472774+0000) 2022-09-10T15:44:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:06.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:06.841683+0000) 2022-09-10T15:44:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:07.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:07.472974+0000) 2022-09-10T15:44:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:07 smithi104 ceph-mon[97044]: pgmap v21592: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:44:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:07.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:07.841909+0000) 2022-09-10T15:44:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:07 smithi110 ceph-mon[97633]: pgmap v21592: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:44:08.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:44:08.365+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:44:08.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:08.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:08.473201+0000) 2022-09-10T15:44:09.314 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:08.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:08.842143+0000) 2022-09-10T15:44:09.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:09.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:09.473445+0000) 2022-09-10T15:44:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:09 smithi104 ceph-mon[97044]: pgmap v21593: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:44:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:09 smithi104 ceph-mon[97044]: osdmap e1430: 12 total, 12 up, 12 in 2022-09-10T15:44:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:09.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:09.842329+0000) 2022-09-10T15:44:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:09 smithi110 ceph-mon[97633]: pgmap v21593: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:44:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:09 smithi110 ceph-mon[97633]: osdmap e1430: 12 total, 12 up, 12 in 2022-09-10T15:44:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:10.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:10.473585+0000) 2022-09-10T15:44:11.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:10.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:10.842549+0000) 2022-09-10T15:44:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:11.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:11.473789+0000) 2022-09-10T15:44:12.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:11 smithi104 ceph-mon[97044]: pgmap v21595: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:44:12.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:11.841+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:11.842824+0000) 2022-09-10T15:44:12.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:11 smithi110 ceph-mon[97633]: pgmap v21595: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:44:12.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:12.472+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:12.474025+0000) 2022-09-10T15:44:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:12.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:12.843035+0000) 2022-09-10T15:44:13.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:13.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:13.474197+0000) 2022-09-10T15:44:14.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:13 smithi104 ceph-mon[97044]: pgmap v21596: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:44:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:13.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:13.843254+0000) 2022-09-10T15:44:14.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:13 smithi110 ceph-mon[97633]: pgmap v21596: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:44:14.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:14.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:14.474399+0000) 2022-09-10T15:44:15.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:14.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:14.843487+0000) 2022-09-10T15:44:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:15.473+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:15.474576+0000) 2022-09-10T15:44:16.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:15 smithi104 ceph-mon[97044]: pgmap v21597: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:44:16.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:15.842+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:15.843700+0000) 2022-09-10T15:44:16.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:15 smithi110 ceph-mon[97633]: pgmap v21597: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:44:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:16.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:16.474819+0000) 2022-09-10T15:44:17.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:16.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:16.843885+0000) 2022-09-10T15:44:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:17.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:17.475091+0000) 2022-09-10T15:44:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:17 smithi104 ceph-mon[97044]: pgmap v21598: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:44:18.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:17 smithi104 ceph-mon[97044]: osdmap e1431: 12 total, 12 up, 12 in 2022-09-10T15:44:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:17.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:17.844131+0000) 2022-09-10T15:44:18.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:17 smithi110 ceph-mon[97633]: pgmap v21598: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:44:18.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:17 smithi110 ceph-mon[97633]: osdmap e1431: 12 total, 12 up, 12 in 2022-09-10T15:44:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:18.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:18.475365+0000) 2022-09-10T15:44:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:18.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:18.844354+0000) 2022-09-10T15:44:19.815 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:19.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:19.475556+0000) 2022-09-10T15:44:20.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:19.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:19.844579+0000) 2022-09-10T15:44:20.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:19 smithi104 ceph-mon[97044]: pgmap v21600: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:44:20.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:19 smithi110 ceph-mon[97633]: pgmap v21600: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 208 B/s wr, 10 op/s 2022-09-10T15:44:20.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:20.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:20.475770+0000) 2022-09-10T15:44:21.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:20.843+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:20.844767+0000) 2022-09-10T15:44:21.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:21.474+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:21.475998+0000) 2022-09-10T15:44:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:21 smithi104 ceph-mon[97044]: pgmap v21601: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:44:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:21.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:21.844933+0000) 2022-09-10T15:44:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:21 smithi110 ceph-mon[97633]: pgmap v21601: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:44:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:22.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:22.476214+0000) 2022-09-10T15:44:23.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:22.844+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:22.845183+0000) 2022-09-10T15:44:23.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:23.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:23.476400+0000) 2022-09-10T15:44:23.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:44:23.373+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:44:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:23 smithi104 ceph-mon[97044]: pgmap v21602: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:44:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:23.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:23.845420+0000) 2022-09-10T15:44:24.418 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:23 smithi110 ceph-mon[97633]: pgmap v21602: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:44:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:24.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:24.476550+0000) 2022-09-10T15:44:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:24.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:24.845646+0000) 2022-09-10T15:44:25.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:25.475+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:25.476695+0000) 2022-09-10T15:44:26.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:25 smithi104 ceph-mon[97044]: pgmap v21603: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T15:44:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:25.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:25.845891+0000) 2022-09-10T15:44:26.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:25 smithi110 ceph-mon[97633]: pgmap v21603: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T15:44:26.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:26.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:26.476917+0000) 2022-09-10T15:44:27.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:26.845+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:26.846155+0000) 2022-09-10T15:44:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:27.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:27.477153+0000) 2022-09-10T15:44:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:27 smithi104 ceph-mon[97044]: pgmap v21604: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T15:44:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:27.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:27.846394+0000) 2022-09-10T15:44:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:27 smithi110 ceph-mon[97633]: pgmap v21604: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2022-09-10T15:44:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:28.476+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:28.477319+0000) 2022-09-10T15:44:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:28.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:28.846661+0000) 2022-09-10T15:44:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:29.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:29.477541+0000) 2022-09-10T15:44:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:29 smithi104 ceph-mon[97044]: pgmap v21605: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-09-10T15:44:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:29.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:29.846900+0000) 2022-09-10T15:44:30.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:29 smithi110 ceph-mon[97633]: pgmap v21605: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2022-09-10T15:44:30.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:30.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:30.477773+0000) 2022-09-10T15:44:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:30.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:30.847158+0000) 2022-09-10T15:44:31.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:31.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:31.478004+0000) 2022-09-10T15:44:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:31 smithi104 ceph-mon[97044]: pgmap v21606: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:31.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:31.847369+0000) 2022-09-10T15:44:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:31 smithi110 ceph-mon[97633]: pgmap v21606: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:32.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:32.478155+0000) 2022-09-10T15:44:33.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:32.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:32.847572+0000) 2022-09-10T15:44:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:33.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:33.478312+0000) 2022-09-10T15:44:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:33 smithi104 ceph-mon[97044]: pgmap v21607: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:44:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:33.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:33.847757+0000) 2022-09-10T15:44:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:33 smithi110 ceph-mon[97633]: pgmap v21607: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:44:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:34.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:34.478523+0000) 2022-09-10T15:44:35.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:34.846+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:34.847914+0000) 2022-09-10T15:44:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:35.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:35.478735+0000) 2022-09-10T15:44:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:35 smithi104 ceph-mon[97044]: pgmap v21608: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:35.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:35.848163+0000) 2022-09-10T15:44:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:35 smithi110 ceph-mon[97633]: pgmap v21608: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:36.477+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:36.478950+0000) 2022-09-10T15:44:37.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:36.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:36.848339+0000) 2022-09-10T15:44:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:37.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:37.479216+0000) 2022-09-10T15:44:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:37 smithi104 ceph-mon[97044]: pgmap v21609: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:37.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:37.848512+0000) 2022-09-10T15:44:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:37 smithi110 ceph-mon[97633]: pgmap v21609: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:38.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:38.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:38.479393+0000) 2022-09-10T15:44:38.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:44:38.385+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:44:39.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:38.847+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:38.848790+0000) 2022-09-10T15:44:39.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:39.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:39.479566+0000) 2022-09-10T15:44:40.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:39 smithi104 ceph-mon[97044]: pgmap v21610: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:39.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:39.849053+0000) 2022-09-10T15:44:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:39 smithi110 ceph-mon[97633]: pgmap v21610: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:40.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:40.479790+0000) 2022-09-10T15:44:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:40.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:40.849259+0000) 2022-09-10T15:44:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:41.478+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:41.480012+0000) 2022-09-10T15:44:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:41 smithi104 ceph-mon[97044]: pgmap v21611: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:41.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:41.849456+0000) 2022-09-10T15:44:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:41 smithi110 ceph-mon[97633]: pgmap v21611: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:42.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:42.480226+0000) 2022-09-10T15:44:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:42.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:42.849653+0000) 2022-09-10T15:44:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:43.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:43.480465+0000) 2022-09-10T15:44:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:43 smithi104 ceph-mon[97044]: pgmap v21612: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:43.848+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:43.849820+0000) 2022-09-10T15:44:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:43 smithi110 ceph-mon[97633]: pgmap v21612: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:44.479+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:44.480674+0000) 2022-09-10T15:44:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:44.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:44.850082+0000) 2022-09-10T15:44:45.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:45.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:45.480905+0000) 2022-09-10T15:44:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:45 smithi104 ceph-mon[97044]: pgmap v21613: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:45.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:45.850268+0000) 2022-09-10T15:44:46.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:45 smithi110 ceph-mon[97633]: pgmap v21613: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:46.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:46.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:46.481125+0000) 2022-09-10T15:44:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:46.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:46.850485+0000) 2022-09-10T15:44:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:47.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:47.481379+0000) 2022-09-10T15:44:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:47 smithi104 ceph-mon[97044]: pgmap v21614: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:44:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:44:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:44:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:44:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:47.849+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:47.850683+0000) 2022-09-10T15:44:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:47 smithi110 ceph-mon[97633]: pgmap v21614: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:44:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:44:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:44:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:44:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:48.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:48.481593+0000) 2022-09-10T15:44:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:48.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:48.850901+0000) 2022-09-10T15:44:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:49.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:49.481829+0000) 2022-09-10T15:44:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:49 smithi104 ceph-mon[97044]: pgmap v21615: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:49.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:49.851215+0000) 2022-09-10T15:44:50.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:49 smithi110 ceph-mon[97633]: pgmap v21615: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:50.480+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:50.482001+0000) 2022-09-10T15:44:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:50.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:50.851452+0000) 2022-09-10T15:44:51.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:51.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:51.482184+0000) 2022-09-10T15:44:52.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:52 smithi104 ceph-mon[97044]: pgmap v21616: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:52.344 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:51.850+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:51.851686+0000) 2022-09-10T15:44:52.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:52 smithi110 ceph-mon[97633]: pgmap v21616: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:52.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:52.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:52.482416+0000) 2022-09-10T15:44:53.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:52.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:52.851894+0000) 2022-09-10T15:44:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:53.481+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:53.482633+0000) 2022-09-10T15:44:53.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:44:53.401+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:44:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:54 smithi104 ceph-mon[97044]: pgmap v21617: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:53.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:53.852163+0000) 2022-09-10T15:44:54.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:54 smithi110 ceph-mon[97633]: pgmap v21617: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:54.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:54.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:54.482834+0000) 2022-09-10T15:44:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:54.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:54.852380+0000) 2022-09-10T15:44:55.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:55.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:55.483081+0000) 2022-09-10T15:44:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:56 smithi104 ceph-mon[97044]: pgmap v21618: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:55.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:55.852562+0000) 2022-09-10T15:44:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:56 smithi110 ceph-mon[97633]: pgmap v21618: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:44:56.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:56.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:56.483344+0000) 2022-09-10T15:44:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:56.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:56.852805+0000) 2022-09-10T15:44:57.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:57.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:57.483556+0000) 2022-09-10T15:44:58.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:44:58 smithi104 ceph-mon[97044]: pgmap v21619: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:57.851+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:57.852973+0000) 2022-09-10T15:44:58.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:44:58 smithi110 ceph-mon[97633]: pgmap v21619: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:44:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:58.482+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:58.483756+0000) 2022-09-10T15:44:59.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:58.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:58.853185+0000) 2022-09-10T15:44:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:44:59.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:59.484012+0000) 2022-09-10T15:45:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:00 smithi104 ceph-mon[97044]: pgmap v21620: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:00.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:44:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:44:59.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:44:59.853420+0000) 2022-09-10T15:45:00.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:00 smithi110 ceph-mon[97633]: pgmap v21620: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:00.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:00.484269+0000) 2022-09-10T15:45:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:00.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:00.853595+0000) 2022-09-10T15:45:01.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:01.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:01.484494+0000) 2022-09-10T15:45:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:02 smithi104 ceph-mon[97044]: pgmap v21621: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:01.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:01.853773+0000) 2022-09-10T15:45:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:02 smithi110 ceph-mon[97633]: pgmap v21621: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:02.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:02.484669+0000) 2022-09-10T15:45:03.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:02.852+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:02.854026+0000) 2022-09-10T15:45:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:03.483+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:03.484875+0000) 2022-09-10T15:45:04.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:04 smithi104 ceph-mon[97044]: pgmap v21622: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:04.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:03.853+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:03.854276+0000) 2022-09-10T15:45:04.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:04 smithi110 ceph-mon[97633]: pgmap v21622: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:04.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:04.485122+0000) 2022-09-10T15:45:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:04.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:04.854493+0000) 2022-09-10T15:45:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:05.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:05.485273+0000) 2022-09-10T15:45:06.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:06 smithi104 ceph-mon[97044]: pgmap v21623: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:45:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:05.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:05.854732+0000) 2022-09-10T15:45:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:06 smithi110 ceph-mon[97633]: pgmap v21623: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:45:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:06.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:06.485505+0000) 2022-09-10T15:45:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:06.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:06.854923+0000) 2022-09-10T15:45:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:07.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:07.485739+0000) 2022-09-10T15:45:08.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:08 smithi104 ceph-mon[97044]: pgmap v21624: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:45:08.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:07.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:07.855088+0000) 2022-09-10T15:45:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:08 smithi110 ceph-mon[97633]: pgmap v21624: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:45:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:08.484+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:08.485930+0000) 2022-09-10T15:45:08.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:45:08.415+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:45:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:08.854+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:08.855377+0000) 2022-09-10T15:45:09.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:09.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:09.486187+0000) 2022-09-10T15:45:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:09 smithi110 ceph-mon[97633]: pgmap v21625: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:45:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:09 smithi110 ceph-mon[97633]: osdmap e1432: 12 total, 12 up, 12 in 2022-09-10T15:45:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:09 smithi104 ceph-mon[97044]: pgmap v21625: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:45:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:09 smithi104 ceph-mon[97044]: osdmap e1432: 12 total, 12 up, 12 in 2022-09-10T15:45:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:09.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:09.855628+0000) 2022-09-10T15:45:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:10.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:10.486396+0000) 2022-09-10T15:45:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:10.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:10.855882+0000) 2022-09-10T15:45:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:11.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:11.486670+0000) 2022-09-10T15:45:12.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:11 smithi110 ceph-mon[97633]: pgmap v21627: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:45:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:11 smithi104 ceph-mon[97044]: pgmap v21627: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:45:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:11.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:11.856174+0000) 2022-09-10T15:45:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:12.485+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:12.486833+0000) 2022-09-10T15:45:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:12.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:12.856327+0000) 2022-09-10T15:45:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:13.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:13.487090+0000) 2022-09-10T15:45:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:13 smithi110 ceph-mon[97633]: pgmap v21628: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:45:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:13 smithi104 ceph-mon[97044]: pgmap v21628: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:45:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:13.855+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:13.856507+0000) 2022-09-10T15:45:14.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:14.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:14.487340+0000) 2022-09-10T15:45:15.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:14.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:14.856745+0000) 2022-09-10T15:45:15.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:15.486+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:15.487507+0000) 2022-09-10T15:45:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:15 smithi110 ceph-mon[97633]: pgmap v21629: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:45:16.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:15 smithi104 ceph-mon[97044]: pgmap v21629: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:45:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:15.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:15.856953+0000) 2022-09-10T15:45:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:16.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:16.487715+0000) 2022-09-10T15:45:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:16.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:16.857196+0000) 2022-09-10T15:45:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:17.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:17.487915+0000) 2022-09-10T15:45:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:17 smithi110 ceph-mon[97633]: pgmap v21630: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:45:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:17 smithi110 ceph-mon[97633]: osdmap e1433: 12 total, 12 up, 12 in 2022-09-10T15:45:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:17 smithi104 ceph-mon[97044]: pgmap v21630: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:45:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:17 smithi104 ceph-mon[97044]: osdmap e1433: 12 total, 12 up, 12 in 2022-09-10T15:45:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:17.856+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:17.857337+0000) 2022-09-10T15:45:18.817 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:18.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:18.488131+0000) 2022-09-10T15:45:19.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:18.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:18.857560+0000) 2022-09-10T15:45:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:19.487+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:19.488319+0000) 2022-09-10T15:45:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:19 smithi110 ceph-mon[97633]: pgmap v21632: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:45:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:19 smithi104 ceph-mon[97044]: pgmap v21632: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:45:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:19.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:19.857797+0000) 2022-09-10T15:45:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:20.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:20.488551+0000) 2022-09-10T15:45:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:20.857+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:20.858042+0000) 2022-09-10T15:45:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:21.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:21.488773+0000) 2022-09-10T15:45:22.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:21 smithi110 ceph-mon[97633]: pgmap v21633: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:45:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:21 smithi104 ceph-mon[97044]: pgmap v21633: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:45:22.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:21.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:21.858393+0000) 2022-09-10T15:45:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:22.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:22.488983+0000) 2022-09-10T15:45:23.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:22.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:22.858640+0000) 2022-09-10T15:45:23.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:23.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:23.489131+0000) 2022-09-10T15:45:23.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:45:23.427+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:45:24.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:23 smithi110 ceph-mon[97633]: pgmap v21634: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:45:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:23 smithi104 ceph-mon[97044]: pgmap v21634: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:45:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:23.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:23.858892+0000) 2022-09-10T15:45:24.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:24.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:24.489347+0000) 2022-09-10T15:45:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:24.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:24.859143+0000) 2022-09-10T15:45:25.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:25.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:25.489618+0000) 2022-09-10T15:45:26.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:25 smithi110 ceph-mon[97633]: pgmap v21635: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:45:26.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:25 smithi104 ceph-mon[97044]: pgmap v21635: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:45:26.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:25.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:25.859327+0000) 2022-09-10T15:45:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:26.488+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:26.489832+0000) 2022-09-10T15:45:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:26.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:26.859523+0000) 2022-09-10T15:45:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:27.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:27.490089+0000) 2022-09-10T15:45:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:27 smithi104 ceph-mon[97044]: pgmap v21636: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:45:28.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:27.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:27.859702+0000) 2022-09-10T15:45:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:27 smithi110 ceph-mon[97633]: pgmap v21636: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:45:28.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:28.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:28.490336+0000) 2022-09-10T15:45:29.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:28.858+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:28.859947+0000) 2022-09-10T15:45:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:29.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:29.490623+0000) 2022-09-10T15:45:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:29 smithi104 ceph-mon[97044]: pgmap v21637: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:45:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:29.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:29.860177+0000) 2022-09-10T15:45:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:29 smithi110 ceph-mon[97633]: pgmap v21637: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:45:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:30.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:30.490841+0000) 2022-09-10T15:45:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:30.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:30.860382+0000) 2022-09-10T15:45:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:31.489+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:31.490977+0000) 2022-09-10T15:45:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:31 smithi104 ceph-mon[97044]: pgmap v21638: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:31.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:31.860587+0000) 2022-09-10T15:45:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:31 smithi110 ceph-mon[97633]: pgmap v21638: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:32.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:32.491224+0000) 2022-09-10T15:45:33.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:32.859+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:32.860785+0000) 2022-09-10T15:45:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:33.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:33.491507+0000) 2022-09-10T15:45:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:33 smithi104 ceph-mon[97044]: pgmap v21639: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:34.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:33.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:33.861041+0000) 2022-09-10T15:45:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:33 smithi110 ceph-mon[97633]: pgmap v21639: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:34.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:34.491772+0000) 2022-09-10T15:45:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:34.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:34.861331+0000) 2022-09-10T15:45:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:35.490+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:35.491969+0000) 2022-09-10T15:45:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:35 smithi104 ceph-mon[97044]: pgmap v21640: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:35.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:35.861606+0000) 2022-09-10T15:45:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:35 smithi110 ceph-mon[97633]: pgmap v21640: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:36.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:36.492227+0000) 2022-09-10T15:45:37.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:36.860+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:36.861781+0000) 2022-09-10T15:45:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:37.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:37.492479+0000) 2022-09-10T15:45:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:37 smithi104 ceph-mon[97044]: pgmap v21641: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:37.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:37.861991+0000) 2022-09-10T15:45:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:37 smithi110 ceph-mon[97633]: pgmap v21641: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:45:38.442+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:45:38.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:38.491+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:38.492716+0000) 2022-09-10T15:45:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:38.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:38.862234+0000) 2022-09-10T15:45:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:39.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:39.492944+0000) 2022-09-10T15:45:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:39 smithi104 ceph-mon[97044]: pgmap v21642: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:39.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:39.862472+0000) 2022-09-10T15:45:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:39 smithi110 ceph-mon[97633]: pgmap v21642: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:40.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:40.493152+0000) 2022-09-10T15:45:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:40.861+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:40.862664+0000) 2022-09-10T15:45:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:41.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:41.493398+0000) 2022-09-10T15:45:42.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:41 smithi104 ceph-mon[97044]: pgmap v21643: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:41.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:41.862937+0000) 2022-09-10T15:45:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:41 smithi110 ceph-mon[97633]: pgmap v21643: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:42.492+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:42.493616+0000) 2022-09-10T15:45:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:42.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:42.863171+0000) 2022-09-10T15:45:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:43.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:43.493855+0000) 2022-09-10T15:45:44.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:43 smithi104 ceph-mon[97044]: pgmap v21644: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:43.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:43.863332+0000) 2022-09-10T15:45:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:43 smithi110 ceph-mon[97633]: pgmap v21644: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:44.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:44.494165+0000) 2022-09-10T15:45:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:44.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:44.863606+0000) 2022-09-10T15:45:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:45.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:45.494331+0000) 2022-09-10T15:45:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:45 smithi104 ceph-mon[97044]: pgmap v21645: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:45.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:45.863815+0000) 2022-09-10T15:45:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:45 smithi110 ceph-mon[97633]: pgmap v21645: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:46.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:46.494572+0000) 2022-09-10T15:45:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:46.862+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:46.863986+0000) 2022-09-10T15:45:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:47.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:47.494822+0000) 2022-09-10T15:45:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:47 smithi104 ceph-mon[97044]: pgmap v21646: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:45:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:45:48.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:45:48.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:45:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:47.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:47.864245+0000) 2022-09-10T15:45:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:47 smithi110 ceph-mon[97633]: pgmap v21646: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:45:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:45:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:45:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:45:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:48.493+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:48.495075+0000) 2022-09-10T15:45:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:48.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:48.864490+0000) 2022-09-10T15:45:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:49.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:49.495327+0000) 2022-09-10T15:45:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:49 smithi104 ceph-mon[97044]: pgmap v21647: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:50.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:49.863+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:49.864677+0000) 2022-09-10T15:45:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:49 smithi110 ceph-mon[97633]: pgmap v21647: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:50.494+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:50.495521+0000) 2022-09-10T15:45:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:50.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:50.864917+0000) 2022-09-10T15:45:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:51.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:51.495755+0000) 2022-09-10T15:45:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:51 smithi104 ceph-mon[97044]: pgmap v21648: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:51.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:51.865184+0000) 2022-09-10T15:45:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:51 smithi110 ceph-mon[97633]: pgmap v21648: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:52.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:52.495953+0000) 2022-09-10T15:45:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:52.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:52.865416+0000) 2022-09-10T15:45:53.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:45:53.457+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:45:53.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:53.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:53.496219+0000) 2022-09-10T15:45:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:53 smithi104 ceph-mon[97044]: pgmap v21649: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:53.864+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:53.865615+0000) 2022-09-10T15:45:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:53 smithi110 ceph-mon[97633]: pgmap v21649: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:54.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:54.496493+0000) 2022-09-10T15:45:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:54.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:54.865857+0000) 2022-09-10T15:45:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:55.495+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:55.496658+0000) 2022-09-10T15:45:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:55 smithi104 ceph-mon[97044]: pgmap v21650: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:55.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:55.866134+0000) 2022-09-10T15:45:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:55 smithi110 ceph-mon[97633]: pgmap v21650: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:45:56.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:56.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:56.496896+0000) 2022-09-10T15:45:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:56.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:56.866292+0000) 2022-09-10T15:45:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:57.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:57.497142+0000) 2022-09-10T15:45:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:57 smithi104 ceph-mon[97044]: pgmap v21651: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:57.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:57.866462+0000) 2022-09-10T15:45:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:57 smithi110 ceph-mon[97633]: pgmap v21651: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:45:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:58.496+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:58.497388+0000) 2022-09-10T15:45:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:58.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:58.866613+0000) 2022-09-10T15:45:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:45:59.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:59.497660+0000) 2022-09-10T15:46:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:45:59 smithi104 ceph-mon[97044]: pgmap v21652: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:45:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:45:59.865+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:45:59.866835+0000) 2022-09-10T15:46:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:45:59 smithi110 ceph-mon[97633]: pgmap v21652: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:00.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:00.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:00.497959+0000) 2022-09-10T15:46:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:00.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:00.867045+0000) 2022-09-10T15:46:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:01.497+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:01.498128+0000) 2022-09-10T15:46:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:01 smithi104 ceph-mon[97044]: pgmap v21653: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:01.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:01.867288+0000) 2022-09-10T15:46:02.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:01 smithi110 ceph-mon[97633]: pgmap v21653: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:02.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:02.498400+0000) 2022-09-10T15:46:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:02.866+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:02.867472+0000) 2022-09-10T15:46:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:03.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:03.498605+0000) 2022-09-10T15:46:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:03 smithi104 ceph-mon[97044]: pgmap v21654: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:03.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:03.867714+0000) 2022-09-10T15:46:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:03 smithi110 ceph-mon[97633]: pgmap v21654: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:04.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:04.498864+0000) 2022-09-10T15:46:05.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:04.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:04.867950+0000) 2022-09-10T15:46:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:05.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:05.499144+0000) 2022-09-10T15:46:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:05 smithi104 ceph-mon[97044]: pgmap v21655: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:46:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:05.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:05.868200+0000) 2022-09-10T15:46:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:05 smithi110 ceph-mon[97633]: pgmap v21655: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:46:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:06.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:06.499333+0000) 2022-09-10T15:46:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:06.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:06.868418+0000) 2022-09-10T15:46:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:07.499+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:07.499569+0000) 2022-09-10T15:46:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:07 smithi104 ceph-mon[97044]: pgmap v21656: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:46:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:07.867+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:07.868598+0000) 2022-09-10T15:46:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:07 smithi110 ceph-mon[97633]: pgmap v21656: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:46:08.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:46:08.473+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:46:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:08.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:08.499756+0000) 2022-09-10T15:46:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:08.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:08.868828+0000) 2022-09-10T15:46:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:09.498+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:09.500005+0000) 2022-09-10T15:46:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:09 smithi110 ceph-mon[97633]: pgmap v21657: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:46:10.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:09 smithi110 ceph-mon[97633]: osdmap e1434: 12 total, 12 up, 12 in 2022-09-10T15:46:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:09 smithi104 ceph-mon[97044]: pgmap v21657: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:46:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:09 smithi104 ceph-mon[97044]: osdmap e1434: 12 total, 12 up, 12 in 2022-09-10T15:46:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:09.868+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:09.869028+0000) 2022-09-10T15:46:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:10.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:10.500179+0000) 2022-09-10T15:46:11.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:10.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:10.869265+0000) 2022-09-10T15:46:11.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:11.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:11.500374+0000) 2022-09-10T15:46:12.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:11 smithi110 ceph-mon[97633]: pgmap v21659: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:46:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:11 smithi104 ceph-mon[97044]: pgmap v21659: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:46:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:11.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:11.869491+0000) 2022-09-10T15:46:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:12.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:12.500607+0000) 2022-09-10T15:46:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:12.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:12.869638+0000) 2022-09-10T15:46:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:13.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:13.500789+0000) 2022-09-10T15:46:14.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:13 smithi110 ceph-mon[97633]: pgmap v21660: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:46:14.159 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:13 smithi110 ceph-mon[97633]: osdmap e1435: 12 total, 12 up, 12 in 2022-09-10T15:46:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:13 smithi104 ceph-mon[97044]: pgmap v21660: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:46:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:13 smithi104 ceph-mon[97044]: osdmap e1435: 12 total, 12 up, 12 in 2022-09-10T15:46:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:13.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:13.869783+0000) 2022-09-10T15:46:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:14.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:14.501014+0000) 2022-09-10T15:46:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:14.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:14.870019+0000) 2022-09-10T15:46:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:15.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:15.501271+0000) 2022-09-10T15:46:16.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:15 smithi110 ceph-mon[97633]: pgmap v21662: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:46:16.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:15 smithi104 ceph-mon[97044]: pgmap v21662: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:46:16.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:15.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:15.870241+0000) 2022-09-10T15:46:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:16.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:16.501485+0000) 2022-09-10T15:46:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:16.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:16.870466+0000) 2022-09-10T15:46:17.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:17.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:17.501698+0000) 2022-09-10T15:46:18.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:17 smithi110 ceph-mon[97633]: pgmap v21663: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:46:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:17 smithi104 ceph-mon[97044]: pgmap v21663: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:46:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:17.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:17.870671+0000) 2022-09-10T15:46:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:18.500+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:18.501884+0000) 2022-09-10T15:46:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:18.869+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:18.870898+0000) 2022-09-10T15:46:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:19.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:19.502127+0000) 2022-09-10T15:46:20.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:19 smithi110 ceph-mon[97633]: pgmap v21664: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:46:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:19 smithi104 ceph-mon[97044]: pgmap v21664: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:46:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:19.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:19.871174+0000) 2022-09-10T15:46:20.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:20.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:20.502336+0000) 2022-09-10T15:46:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:20.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:20.871316+0000) 2022-09-10T15:46:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:21.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:21.502533+0000) 2022-09-10T15:46:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:21 smithi104 ceph-mon[97044]: pgmap v21665: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:46:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:21.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:21.871533+0000) 2022-09-10T15:46:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:21 smithi110 ceph-mon[97633]: pgmap v21665: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:46:22.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:22.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:22.502775+0000) 2022-09-10T15:46:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:22.870+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:22.871780+0000) 2022-09-10T15:46:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:46:23.487+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:46:23.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:23.501+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:23.503057+0000) 2022-09-10T15:46:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:23 smithi104 ceph-mon[97044]: pgmap v21666: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:46:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:23.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:23.871995+0000) 2022-09-10T15:46:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:23 smithi110 ceph-mon[97633]: pgmap v21666: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:46:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:24.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:24.503308+0000) 2022-09-10T15:46:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:24.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:24.872225+0000) 2022-09-10T15:46:25.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:25.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:25.503543+0000) 2022-09-10T15:46:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:25 smithi104 ceph-mon[97044]: pgmap v21667: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:46:26.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:25.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:25.872397+0000) 2022-09-10T15:46:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:25 smithi110 ceph-mon[97633]: pgmap v21667: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:46:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:26.502+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:26.503767+0000) 2022-09-10T15:46:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:26.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:26.872610+0000) 2022-09-10T15:46:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:27.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:27.503998+0000) 2022-09-10T15:46:28.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:27 smithi104 ceph-mon[97044]: pgmap v21668: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:27.871+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:27.872865+0000) 2022-09-10T15:46:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:27 smithi110 ceph-mon[97633]: pgmap v21668: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:28.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:28.504177+0000) 2022-09-10T15:46:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:28.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:28.873129+0000) 2022-09-10T15:46:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:29.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:29.504389+0000) 2022-09-10T15:46:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:29 smithi104 ceph-mon[97044]: pgmap v21669: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:29.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:29.873404+0000) 2022-09-10T15:46:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:29 smithi110 ceph-mon[97633]: pgmap v21669: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:30.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:30.504547+0000) 2022-09-10T15:46:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:30.872+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:30.873623+0000) 2022-09-10T15:46:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:31.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:31.504717+0000) 2022-09-10T15:46:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:31 smithi104 ceph-mon[97044]: pgmap v21670: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:32.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:31.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:31.873855+0000) 2022-09-10T15:46:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:31 smithi110 ceph-mon[97633]: pgmap v21670: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:32.503+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:32.504906+0000) 2022-09-10T15:46:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:32.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:32.874058+0000) 2022-09-10T15:46:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:33.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:33.505146+0000) 2022-09-10T15:46:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:33 smithi104 ceph-mon[97044]: pgmap v21671: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:33.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:33.874336+0000) 2022-09-10T15:46:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:33 smithi110 ceph-mon[97633]: pgmap v21671: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:34.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:34.505308+0000) 2022-09-10T15:46:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:34.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:34.874491+0000) 2022-09-10T15:46:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:35.504+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:35.505506+0000) 2022-09-10T15:46:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:35 smithi104 ceph-mon[97044]: pgmap v21672: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:35.873+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:35.874736+0000) 2022-09-10T15:46:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:35 smithi110 ceph-mon[97633]: pgmap v21672: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:36.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:36.505749+0000) 2022-09-10T15:46:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:36.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:36.874932+0000) 2022-09-10T15:46:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:37.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:37.506006+0000) 2022-09-10T15:46:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:37 smithi104 ceph-mon[97044]: pgmap v21673: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:37.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:37.875170+0000) 2022-09-10T15:46:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:37 smithi110 ceph-mon[97633]: pgmap v21673: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:46:38.490+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:46:38.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:38.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:38.506272+0000) 2022-09-10T15:46:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:38.874+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:38.875378+0000) 2022-09-10T15:46:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:39.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:39.506466+0000) 2022-09-10T15:46:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:39 smithi104 ceph-mon[97044]: pgmap v21674: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:39.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:39.875656+0000) 2022-09-10T15:46:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:39 smithi110 ceph-mon[97633]: pgmap v21674: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:40.505+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:40.506659+0000) 2022-09-10T15:46:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:40.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:40.875922+0000) 2022-09-10T15:46:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:41.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:41.506895+0000) 2022-09-10T15:46:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:41 smithi104 ceph-mon[97044]: pgmap v21675: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:41.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:41.876182+0000) 2022-09-10T15:46:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:41 smithi110 ceph-mon[97633]: pgmap v21675: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:42.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:42.507129+0000) 2022-09-10T15:46:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:42.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:42.876429+0000) 2022-09-10T15:46:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:43.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:43.507330+0000) 2022-09-10T15:46:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:43 smithi104 ceph-mon[97044]: pgmap v21676: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:43.875+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:43.876682+0000) 2022-09-10T15:46:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:43 smithi110 ceph-mon[97633]: pgmap v21676: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:44.506+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:44.507568+0000) 2022-09-10T15:46:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:44.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:44.876939+0000) 2022-09-10T15:46:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:45.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:45.507762+0000) 2022-09-10T15:46:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:45 smithi104 ceph-mon[97044]: pgmap v21677: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:45.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:45.877157+0000) 2022-09-10T15:46:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:45 smithi110 ceph-mon[97633]: pgmap v21677: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:46.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:46.507947+0000) 2022-09-10T15:46:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:46.876+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:46.877379+0000) 2022-09-10T15:46:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:47.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:47.508210+0000) 2022-09-10T15:46:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:47 smithi104 ceph-mon[97044]: pgmap v21678: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:46:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:46:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:46:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:46:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:47.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:47.877586+0000) 2022-09-10T15:46:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:47 smithi110 ceph-mon[97633]: pgmap v21678: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:46:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:46:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:46:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:46:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:48.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:48.508402+0000) 2022-09-10T15:46:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:48.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:48.877827+0000) 2022-09-10T15:46:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:49.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:49.508663+0000) 2022-09-10T15:46:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:49 smithi104 ceph-mon[97044]: pgmap v21679: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:49.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:49.878050+0000) 2022-09-10T15:46:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:49 smithi110 ceph-mon[97633]: pgmap v21679: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:46:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:50.507+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:50.508840+0000) 2022-09-10T15:46:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:50.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:50.878324+0000) 2022-09-10T15:46:51.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:51.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:51.509144+0000) 2022-09-10T15:46:52.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:51 smithi104 ceph-mon[97044]: pgmap v21680: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:51.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:51.878556+0000) 2022-09-10T15:46:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:51 smithi110 ceph-mon[97633]: pgmap v21680: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:52.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:52.509372+0000) 2022-09-10T15:46:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:52.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:52.878754+0000) 2022-09-10T15:46:53.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:53.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:53.509581+0000) 2022-09-10T15:46:53.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:46:53.494+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:46:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:53 smithi104 ceph-mon[97044]: pgmap v21681: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:46:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:53.877+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:53.878991+0000) 2022-09-10T15:46:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:53 smithi110 ceph-mon[97633]: pgmap v21681: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:46:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:54.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:54.509760+0000) 2022-09-10T15:46:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:54.878+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:54.879256+0000) 2022-09-10T15:46:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:55.508+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:55.509905+0000) 2022-09-10T15:46:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:55 smithi104 ceph-mon[97044]: pgmap v21682: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:55.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:55.879522+0000) 2022-09-10T15:46:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:55 smithi110 ceph-mon[97633]: pgmap v21682: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:46:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:56.509+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:56.510159+0000) 2022-09-10T15:46:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:56.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:56.879749+0000) 2022-09-10T15:46:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:57.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:57.510383+0000) 2022-09-10T15:46:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:57 smithi104 ceph-mon[97044]: pgmap v21683: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:46:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:57.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:57.879994+0000) 2022-09-10T15:46:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:57 smithi110 ceph-mon[97633]: pgmap v21683: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:46:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:58.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:58.510575+0000) 2022-09-10T15:46:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:58.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:58.880192+0000) 2022-09-10T15:46:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:46:59.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:59.510858+0000) 2022-09-10T15:47:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:46:59 smithi104 ceph-mon[97044]: pgmap v21684: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:47:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:46:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:46:59.879+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:46:59.880387+0000) 2022-09-10T15:47:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:46:59 smithi110 ceph-mon[97633]: pgmap v21684: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:47:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:00.510+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:00.511087+0000) 2022-09-10T15:47:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:00.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:00.880645+0000) 2022-09-10T15:47:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:01.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:01.511336+0000) 2022-09-10T15:47:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:01 smithi104 ceph-mon[97044]: pgmap v21685: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:01.880+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:01.880928+0000) 2022-09-10T15:47:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:01 smithi110 ceph-mon[97633]: pgmap v21685: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:02.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:02.511529+0000) 2022-09-10T15:47:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:02.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:02.881173+0000) 2022-09-10T15:47:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:03.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:03.511815+0000) 2022-09-10T15:47:04.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:03 smithi104 ceph-mon[97044]: pgmap v21686: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:03.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:03.881389+0000) 2022-09-10T15:47:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:03 smithi110 ceph-mon[97633]: pgmap v21686: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:04.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:04.512035+0000) 2022-09-10T15:47:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:04.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:04.881602+0000) 2022-09-10T15:47:05.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:05.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:05.512255+0000) 2022-09-10T15:47:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:05 smithi104 ceph-mon[97044]: pgmap v21687: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:47:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:05.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:05.881825+0000) 2022-09-10T15:47:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:05 smithi110 ceph-mon[97633]: pgmap v21687: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:47:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:06.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:06.512483+0000) 2022-09-10T15:47:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:06.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:06.882059+0000) 2022-09-10T15:47:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:07.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:07.512723+0000) 2022-09-10T15:47:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:07 smithi104 ceph-mon[97044]: pgmap v21688: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:47:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:07.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:07.882249+0000) 2022-09-10T15:47:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:07 smithi110 ceph-mon[97633]: pgmap v21688: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:47:08.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:47:08.510+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:47:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:08.511+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:08.512979+0000) 2022-09-10T15:47:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:08.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:08.882460+0000) 2022-09-10T15:47:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:09.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:09.513250+0000) 2022-09-10T15:47:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:09 smithi104 ceph-mon[97044]: pgmap v21689: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:47:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:09 smithi104 ceph-mon[97044]: osdmap e1436: 12 total, 12 up, 12 in 2022-09-10T15:47:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:09.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:09.882683+0000) 2022-09-10T15:47:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:09 smithi110 ceph-mon[97633]: pgmap v21689: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:47:10.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:09 smithi110 ceph-mon[97633]: osdmap e1436: 12 total, 12 up, 12 in 2022-09-10T15:47:10.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:10.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:10.513451+0000) 2022-09-10T15:47:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:10.881+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:10.882932+0000) 2022-09-10T15:47:11.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:11.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:11.513683+0000) 2022-09-10T15:47:12.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:11 smithi104 ceph-mon[97044]: pgmap v21691: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:47:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:11.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:11.883230+0000) 2022-09-10T15:47:12.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:11 smithi110 ceph-mon[97633]: pgmap v21691: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:47:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:12.512+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:12.513932+0000) 2022-09-10T15:47:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:12.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:12.883401+0000) 2022-09-10T15:47:13.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:13.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:13.514213+0000) 2022-09-10T15:47:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:13 smithi104 ceph-mon[97044]: pgmap v21692: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:47:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:13.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:13.883604+0000) 2022-09-10T15:47:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:13 smithi110 ceph-mon[97633]: pgmap v21692: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:47:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:14.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:14.514482+0000) 2022-09-10T15:47:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:14.882+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:14.883844+0000) 2022-09-10T15:47:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:15.513+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:15.514696+0000) 2022-09-10T15:47:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:15 smithi104 ceph-mon[97044]: pgmap v21693: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:47:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:15.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:15.884127+0000) 2022-09-10T15:47:16.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:15 smithi110 ceph-mon[97633]: pgmap v21693: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:47:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:16.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:16.514942+0000) 2022-09-10T15:47:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:16.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:16.884301+0000) 2022-09-10T15:47:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:17.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:17.515209+0000) 2022-09-10T15:47:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:17 smithi104 ceph-mon[97044]: pgmap v21694: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:47:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:17 smithi104 ceph-mon[97044]: osdmap e1437: 12 total, 12 up, 12 in 2022-09-10T15:47:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:17.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:17.884473+0000) 2022-09-10T15:47:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:17 smithi110 ceph-mon[97633]: pgmap v21694: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:47:18.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:17 smithi110 ceph-mon[97633]: osdmap e1437: 12 total, 12 up, 12 in 2022-09-10T15:47:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:18.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:18.515450+0000) 2022-09-10T15:47:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:18.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:18.884664+0000) 2022-09-10T15:47:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:19.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:19.515681+0000) 2022-09-10T15:47:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:19 smithi104 ceph-mon[97044]: pgmap v21696: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:47:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:19.883+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:19.884873+0000) 2022-09-10T15:47:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:19 smithi110 ceph-mon[97633]: pgmap v21696: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:47:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:20.514+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:20.515882+0000) 2022-09-10T15:47:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:20.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:20.885125+0000) 2022-09-10T15:47:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:21.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:21.516089+0000) 2022-09-10T15:47:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:21 smithi104 ceph-mon[97044]: pgmap v21697: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:47:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:21.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:21.885324+0000) 2022-09-10T15:47:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:21 smithi110 ceph-mon[97633]: pgmap v21697: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:47:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:22.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:22.516322+0000) 2022-09-10T15:47:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:22.884+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:22.885563+0000) 2022-09-10T15:47:23.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:23.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:23.516487+0000) 2022-09-10T15:47:23.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:47:23.525+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:47:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:23 smithi104 ceph-mon[97044]: pgmap v21698: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:47:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:23.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:23.885786+0000) 2022-09-10T15:47:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:23 smithi110 ceph-mon[97633]: pgmap v21698: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:47:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:24.515+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:24.516686+0000) 2022-09-10T15:47:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:24.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:24.885976+0000) 2022-09-10T15:47:25.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:25.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:25.516883+0000) 2022-09-10T15:47:26.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:25 smithi104 ceph-mon[97044]: pgmap v21699: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:47:26.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:25.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:25.886206+0000) 2022-09-10T15:47:26.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:25 smithi110 ceph-mon[97633]: pgmap v21699: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:47:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:26.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:26.517079+0000) 2022-09-10T15:47:27.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:26.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:26.886377+0000) 2022-09-10T15:47:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:27.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:27.517368+0000) 2022-09-10T15:47:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:27 smithi104 ceph-mon[97044]: pgmap v21700: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:47:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:27.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:27.886596+0000) 2022-09-10T15:47:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:27 smithi110 ceph-mon[97633]: pgmap v21700: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:47:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:28.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:28.517561+0000) 2022-09-10T15:47:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:28.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:28.886824+0000) 2022-09-10T15:47:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:29.516+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:29.517752+0000) 2022-09-10T15:47:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:29 smithi104 ceph-mon[97044]: pgmap v21701: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:47:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:29.885+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:29.886991+0000) 2022-09-10T15:47:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:29 smithi110 ceph-mon[97633]: pgmap v21701: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:47:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:30.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:30.517896+0000) 2022-09-10T15:47:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:30.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:30.887226+0000) 2022-09-10T15:47:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:31.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:31.518086+0000) 2022-09-10T15:47:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:31 smithi104 ceph-mon[97044]: pgmap v21702: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:31.886+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:31.887435+0000) 2022-09-10T15:47:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:31 smithi110 ceph-mon[97633]: pgmap v21702: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:32.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:32.518322+0000) 2022-09-10T15:47:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:32.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:32.887668+0000) 2022-09-10T15:47:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:33.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:33.518576+0000) 2022-09-10T15:47:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:33 smithi104 ceph-mon[97044]: pgmap v21703: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:33.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:33.887898+0000) 2022-09-10T15:47:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:33 smithi110 ceph-mon[97633]: pgmap v21703: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:34.517+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:34.518816+0000) 2022-09-10T15:47:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:34.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:34.888176+0000) 2022-09-10T15:47:35.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:35.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:35.519059+0000) 2022-09-10T15:47:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:35 smithi104 ceph-mon[97044]: pgmap v21704: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:35.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:35.888333+0000) 2022-09-10T15:47:36.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:35 smithi110 ceph-mon[97633]: pgmap v21704: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:36.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:36.518+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:36.519348+0000) 2022-09-10T15:47:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:36.887+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:36.888559+0000) 2022-09-10T15:47:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:37.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:37.519570+0000) 2022-09-10T15:47:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:37 smithi104 ceph-mon[97044]: pgmap v21705: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:37.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:37.888813+0000) 2022-09-10T15:47:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:37 smithi110 ceph-mon[97633]: pgmap v21705: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:47:38.540+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:47:38.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:38.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:38.519806+0000) 2022-09-10T15:47:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:38.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:38.889034+0000) 2022-09-10T15:47:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:39.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:39.519989+0000) 2022-09-10T15:47:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:39 smithi104 ceph-mon[97044]: pgmap v21706: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:39.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:39.889258+0000) 2022-09-10T15:47:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:39 smithi110 ceph-mon[97633]: pgmap v21706: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:40.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:40.520171+0000) 2022-09-10T15:47:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:40.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:40.889476+0000) 2022-09-10T15:47:41.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:41.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:41.520433+0000) 2022-09-10T15:47:42.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:41 smithi104 ceph-mon[97044]: pgmap v21707: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:41.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:41.889726+0000) 2022-09-10T15:47:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:41 smithi110 ceph-mon[97633]: pgmap v21707: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:42.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:42.520658+0000) 2022-09-10T15:47:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:42.888+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:42.889959+0000) 2022-09-10T15:47:43.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:43.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:43.520856+0000) 2022-09-10T15:47:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:43 smithi104 ceph-mon[97044]: pgmap v21708: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:43.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:43.890215+0000) 2022-09-10T15:47:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:43 smithi110 ceph-mon[97633]: pgmap v21708: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:44.519+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:44.521038+0000) 2022-09-10T15:47:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:44.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:44.890376+0000) 2022-09-10T15:47:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:45.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:45.521198+0000) 2022-09-10T15:47:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:45 smithi104 ceph-mon[97044]: pgmap v21709: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:45.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:45.890616+0000) 2022-09-10T15:47:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:45 smithi110 ceph-mon[97633]: pgmap v21709: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:46.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:46.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:46.521438+0000) 2022-09-10T15:47:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:46.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:46.890840+0000) 2022-09-10T15:47:47.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:47.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:47.521699+0000) 2022-09-10T15:47:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:47 smithi104 ceph-mon[97044]: pgmap v21710: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:47:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:47:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:47:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:47:48.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:47.889+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:47.891058+0000) 2022-09-10T15:47:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:47 smithi110 ceph-mon[97633]: pgmap v21710: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:47:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:47:48.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:47:48.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:47:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:48.520+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:48.521882+0000) 2022-09-10T15:47:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:48.890+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:48.891170+0000) 2022-09-10T15:47:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:49.521+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:49.522136+0000) 2022-09-10T15:47:50.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:49 smithi104 ceph-mon[97044]: pgmap v21711: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:49.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:49.891380+0000) 2022-09-10T15:47:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:49 smithi110 ceph-mon[97633]: pgmap v21711: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:50.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:50.522337+0000) 2022-09-10T15:47:51.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:50.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:50.891620+0000) 2022-09-10T15:47:51.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:51.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:51.522614+0000) 2022-09-10T15:47:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:51 smithi104 ceph-mon[97044]: pgmap v21712: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:51.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:51.891927+0000) 2022-09-10T15:47:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:51 smithi110 ceph-mon[97633]: pgmap v21712: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:52.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:52.522+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:52.522846+0000) 2022-09-10T15:47:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:52.891+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:52.892145+0000) 2022-09-10T15:47:53.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:47:53.556+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:47:53.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:53.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:53.523080+0000) 2022-09-10T15:47:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:53 smithi104 ceph-mon[97044]: pgmap v21713: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:53.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:53.892375+0000) 2022-09-10T15:47:54.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:53 smithi110 ceph-mon[97633]: pgmap v21713: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:54.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:54.523289+0000) 2022-09-10T15:47:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:54.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:54.892544+0000) 2022-09-10T15:47:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:55.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:55.523522+0000) 2022-09-10T15:47:56.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:55 smithi104 ceph-mon[97044]: pgmap v21714: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:56.485 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:55 smithi110 ceph-mon[97633]: pgmap v21714: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:47:56.485 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:55.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:55.892785+0000) 2022-09-10T15:47:57.309 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:56.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:56.523782+0000) 2022-09-10T15:47:57.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:56.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:56.893006+0000) 2022-09-10T15:47:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:57.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:57.524036+0000) 2022-09-10T15:47:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:57 smithi104 ceph-mon[97044]: pgmap v21715: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:57.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:57.893241+0000) 2022-09-10T15:47:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:57 smithi110 ceph-mon[97633]: pgmap v21715: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:47:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:58.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:58.524187+0000) 2022-09-10T15:47:59.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:58.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:58.893470+0000) 2022-09-10T15:47:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:47:59.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:59.524396+0000) 2022-09-10T15:48:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:47:59 smithi104 ceph-mon[97044]: pgmap v21716: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:47:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:47:59.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:47:59.893699+0000) 2022-09-10T15:48:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:47:59 smithi110 ceph-mon[97633]: pgmap v21716: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:00.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:00.524568+0000) 2022-09-10T15:48:01.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:00.892+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:00.893883+0000) 2022-09-10T15:48:01.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:01.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:01.524715+0000) 2022-09-10T15:48:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:01 smithi104 ceph-mon[97044]: pgmap v21717: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:01.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:01.894141+0000) 2022-09-10T15:48:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:01 smithi110 ceph-mon[97633]: pgmap v21717: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:02.523+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:02.524859+0000) 2022-09-10T15:48:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:02.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:02.894391+0000) 2022-09-10T15:48:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:03.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:03.525135+0000) 2022-09-10T15:48:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:03 smithi104 ceph-mon[97044]: pgmap v21718: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:03.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:03.894590+0000) 2022-09-10T15:48:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:03 smithi110 ceph-mon[97633]: pgmap v21718: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:04.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:04.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:04.525405+0000) 2022-09-10T15:48:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:04.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:04.894773+0000) 2022-09-10T15:48:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:05.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:05.525610+0000) 2022-09-10T15:48:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:06 smithi104 ceph-mon[97044]: pgmap v21719: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:48:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:05.893+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:05.894998+0000) 2022-09-10T15:48:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:06 smithi110 ceph-mon[97633]: pgmap v21719: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:48:06.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:06.524+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:06.525793+0000) 2022-09-10T15:48:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:06.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:06.895207+0000) 2022-09-10T15:48:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:07.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:07.526033+0000) 2022-09-10T15:48:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:08 smithi104 ceph-mon[97044]: pgmap v21720: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:48:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:07.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:07.895401+0000) 2022-09-10T15:48:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:08 smithi110 ceph-mon[97633]: pgmap v21720: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:48:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:08.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:08.526200+0000) 2022-09-10T15:48:08.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:48:08.562+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:48:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:08.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:08.895647+0000) 2022-09-10T15:48:09.580 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:09.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:09.526326+0000) 2022-09-10T15:48:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:09 smithi104 ceph-mon[97044]: pgmap v21721: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:48:10.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:09 smithi104 ceph-mon[97044]: osdmap e1438: 12 total, 12 up, 12 in 2022-09-10T15:48:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:09.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:09.895836+0000) 2022-09-10T15:48:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:09 smithi110 ceph-mon[97633]: pgmap v21721: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:48:10.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:09 smithi110 ceph-mon[97633]: osdmap e1438: 12 total, 12 up, 12 in 2022-09-10T15:48:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:10.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:10.526472+0000) 2022-09-10T15:48:11.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:10.894+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:10.895980+0000) 2022-09-10T15:48:11.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:11.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:11.526642+0000) 2022-09-10T15:48:12.099 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:11.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:11.896150+0000) 2022-09-10T15:48:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:11 smithi104 ceph-mon[97044]: pgmap v21723: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:48:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:11 smithi110 ceph-mon[97633]: pgmap v21723: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:48:12.634 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:12.525+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:12.526834+0000) 2022-09-10T15:48:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:12.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:12.896313+0000) 2022-09-10T15:48:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:13.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:13.527040+0000) 2022-09-10T15:48:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:13 smithi104 ceph-mon[97044]: pgmap v21724: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:48:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:13 smithi104 ceph-mon[97044]: osdmap e1439: 12 total, 12 up, 12 in 2022-09-10T15:48:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:13.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:13.896440+0000) 2022-09-10T15:48:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:13 smithi110 ceph-mon[97633]: pgmap v21724: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:48:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:13 smithi110 ceph-mon[97633]: osdmap e1439: 12 total, 12 up, 12 in 2022-09-10T15:48:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:14.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:14.527308+0000) 2022-09-10T15:48:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:14.895+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:14.896651+0000) 2022-09-10T15:48:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:15.526+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:15.527613+0000) 2022-09-10T15:48:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:15 smithi104 ceph-mon[97044]: pgmap v21726: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:48:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:15.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:15.896875+0000) 2022-09-10T15:48:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:15 smithi110 ceph-mon[97633]: pgmap v21726: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:48:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:16.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:16.527891+0000) 2022-09-10T15:48:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:16.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:16.897061+0000) 2022-09-10T15:48:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:17.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:17.528186+0000) 2022-09-10T15:48:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:17 smithi104 ceph-mon[97044]: pgmap v21727: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:48:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:17.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:17.897236+0000) 2022-09-10T15:48:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:17 smithi110 ceph-mon[97633]: pgmap v21727: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:48:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:18.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:18.528344+0000) 2022-09-10T15:48:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:18.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:18.897396+0000) 2022-09-10T15:48:19.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:19.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:19.528584+0000) 2022-09-10T15:48:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:19 smithi104 ceph-mon[97044]: pgmap v21728: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:48:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:19.896+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:19.897583+0000) 2022-09-10T15:48:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:19 smithi110 ceph-mon[97633]: pgmap v21728: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:48:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:20.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:20.528851+0000) 2022-09-10T15:48:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:20.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:20.897812+0000) 2022-09-10T15:48:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:21.527+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:21.529064+0000) 2022-09-10T15:48:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:21 smithi104 ceph-mon[97044]: pgmap v21729: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:48:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:21.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:21.898030+0000) 2022-09-10T15:48:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:21 smithi110 ceph-mon[97633]: pgmap v21729: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:48:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:22.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:22.529290+0000) 2022-09-10T15:48:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:22.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:22.898241+0000) 2022-09-10T15:48:23.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:23.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:23.529492+0000) 2022-09-10T15:48:23.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:48:23.568+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:48:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:23 smithi104 ceph-mon[97044]: pgmap v21730: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:48:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:23.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:23.898484+0000) 2022-09-10T15:48:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:23 smithi110 ceph-mon[97633]: pgmap v21730: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:48:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:24.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:24.529689+0000) 2022-09-10T15:48:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:24.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:24.898682+0000) 2022-09-10T15:48:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:25.528+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:25.529864+0000) 2022-09-10T15:48:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:25 smithi104 ceph-mon[97044]: pgmap v21731: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:48:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:25.897+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:25.898892+0000) 2022-09-10T15:48:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:25 smithi110 ceph-mon[97633]: pgmap v21731: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:48:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:26.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:26.530144+0000) 2022-09-10T15:48:27.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:26.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:26.899173+0000) 2022-09-10T15:48:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:27.529+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:27.530441+0000) 2022-09-10T15:48:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:27 smithi104 ceph-mon[97044]: pgmap v21732: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:48:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:27.898+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:27.899419+0000) 2022-09-10T15:48:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:27 smithi110 ceph-mon[97633]: pgmap v21732: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:48:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:28.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:28.530721+0000) 2022-09-10T15:48:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:28.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:28.899663+0000) 2022-09-10T15:48:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:29.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:29.530991+0000) 2022-09-10T15:48:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:29 smithi104 ceph-mon[97044]: pgmap v21733: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:48:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:29.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:29.899808+0000) 2022-09-10T15:48:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:29 smithi110 ceph-mon[97633]: pgmap v21733: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:48:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:30.530+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:30.531243+0000) 2022-09-10T15:48:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:30.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:30.900029+0000) 2022-09-10T15:48:31.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:31.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:31.531515+0000) 2022-09-10T15:48:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:31 smithi104 ceph-mon[97044]: pgmap v21734: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:31.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:31.900234+0000) 2022-09-10T15:48:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:31 smithi110 ceph-mon[97633]: pgmap v21734: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:32.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:32.531676+0000) 2022-09-10T15:48:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:32.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:32.900442+0000) 2022-09-10T15:48:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:33.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:33.531857+0000) 2022-09-10T15:48:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:33 smithi104 ceph-mon[97044]: pgmap v21735: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:33.899+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:33.900686+0000) 2022-09-10T15:48:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:33 smithi110 ceph-mon[97633]: pgmap v21735: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:34.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:34.532125+0000) 2022-09-10T15:48:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:34.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:34.900911+0000) 2022-09-10T15:48:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:35.531+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:35.532337+0000) 2022-09-10T15:48:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:35 smithi104 ceph-mon[97044]: pgmap v21736: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:35.900+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:35.901235+0000) 2022-09-10T15:48:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:35 smithi110 ceph-mon[97633]: pgmap v21736: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:36.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:36.532567+0000) 2022-09-10T15:48:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:36.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:36.901426+0000) 2022-09-10T15:48:37.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:37.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:37.532835+0000) 2022-09-10T15:48:38.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:37 smithi104 ceph-mon[97044]: pgmap v21737: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:38.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:37.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:37.901666+0000) 2022-09-10T15:48:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:37 smithi110 ceph-mon[97633]: pgmap v21737: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:38.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:48:38.577+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:48:38.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:38.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:38.533067+0000) 2022-09-10T15:48:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:38.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:38.901852+0000) 2022-09-10T15:48:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:39.532+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:39.533234+0000) 2022-09-10T15:48:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:39 smithi104 ceph-mon[97044]: pgmap v21738: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:39.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:39.902074+0000) 2022-09-10T15:48:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:39 smithi110 ceph-mon[97633]: pgmap v21738: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:40.808 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:40.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:40.533496+0000) 2022-09-10T15:48:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:40.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:40.902313+0000) 2022-09-10T15:48:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:41.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:41.533699+0000) 2022-09-10T15:48:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:41 smithi104 ceph-mon[97044]: pgmap v21739: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:41.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:41.902531+0000) 2022-09-10T15:48:42.360 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:41 smithi110 ceph-mon[97633]: pgmap v21739: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:42.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:42.533948+0000) 2022-09-10T15:48:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:42.901+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:42.902724+0000) 2022-09-10T15:48:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:43.533+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:43.534201+0000) 2022-09-10T15:48:44.158 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:43 smithi110 ceph-mon[97633]: pgmap v21740: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:43 smithi104 ceph-mon[97044]: pgmap v21740: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:43.902+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:43.902993+0000) 2022-09-10T15:48:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:44.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:44.534369+0000) 2022-09-10T15:48:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:44.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:44.903257+0000) 2022-09-10T15:48:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:45.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:45.534541+0000) 2022-09-10T15:48:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:45 smithi104 ceph-mon[97044]: pgmap v21741: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:45.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:45.903423+0000) 2022-09-10T15:48:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:45 smithi110 ceph-mon[97633]: pgmap v21741: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:46.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:46.534805+0000) 2022-09-10T15:48:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:46.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:46.903612+0000) 2022-09-10T15:48:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:47.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:47.535137+0000) 2022-09-10T15:48:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:47 smithi104 ceph-mon[97044]: pgmap v21742: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:48:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:48:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:48:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:48:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:47.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:47.903791+0000) 2022-09-10T15:48:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:47 smithi110 ceph-mon[97633]: pgmap v21742: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:48:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:48:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:48:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:48:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:48.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:48.535401+0000) 2022-09-10T15:48:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:48.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:48.904046+0000) 2022-09-10T15:48:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:49.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:49.535670+0000) 2022-09-10T15:48:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:49 smithi104 ceph-mon[97044]: pgmap v21743: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:49.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:49.904312+0000) 2022-09-10T15:48:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:49 smithi110 ceph-mon[97633]: pgmap v21743: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:50.534+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:50.535962+0000) 2022-09-10T15:48:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:50.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:50.904491+0000) 2022-09-10T15:48:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:51.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:51.536229+0000) 2022-09-10T15:48:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:51 smithi104 ceph-mon[97044]: pgmap v21744: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:51.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:51.904745+0000) 2022-09-10T15:48:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:51 smithi110 ceph-mon[97633]: pgmap v21744: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:52.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:52.536487+0000) 2022-09-10T15:48:53.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:52.903+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:52.904990+0000) 2022-09-10T15:48:53.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:53.535+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:53.536765+0000) 2022-09-10T15:48:53.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:48:53.588+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:48:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:53 smithi104 ceph-mon[97044]: pgmap v21745: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:53.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:53.905199+0000) 2022-09-10T15:48:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:53 smithi110 ceph-mon[97633]: pgmap v21745: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:54.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:54.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:54.537032+0000) 2022-09-10T15:48:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:54.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:54.905419+0000) 2022-09-10T15:48:55.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:55.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:55.537232+0000) 2022-09-10T15:48:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:55 smithi104 ceph-mon[97044]: pgmap v21746: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:55.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:55.905615+0000) 2022-09-10T15:48:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:55 smithi110 ceph-mon[97633]: pgmap v21746: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:48:56.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:56.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:56.537429+0000) 2022-09-10T15:48:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:56.904+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:56.905821+0000) 2022-09-10T15:48:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:57.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:57.537623+0000) 2022-09-10T15:48:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:57 smithi104 ceph-mon[97044]: pgmap v21747: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:57.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:57.906065+0000) 2022-09-10T15:48:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:57 smithi110 ceph-mon[97633]: pgmap v21747: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:48:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:58.536+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:58.537866+0000) 2022-09-10T15:48:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:58.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:58.906226+0000) 2022-09-10T15:48:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:48:59.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:59.538153+0000) 2022-09-10T15:49:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:48:59 smithi104 ceph-mon[97044]: pgmap v21748: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:00.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:48:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:48:59.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:48:59.906444+0000) 2022-09-10T15:49:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:48:59 smithi110 ceph-mon[97633]: pgmap v21748: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:00.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:00.538313+0000) 2022-09-10T15:49:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:00.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:00.906627+0000) 2022-09-10T15:49:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:01.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:01.538469+0000) 2022-09-10T15:49:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:01 smithi104 ceph-mon[97044]: pgmap v21749: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:01.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:01.906846+0000) 2022-09-10T15:49:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:01 smithi110 ceph-mon[97633]: pgmap v21749: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:02.537+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:02.538685+0000) 2022-09-10T15:49:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:02.905+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:02.907060+0000) 2022-09-10T15:49:03.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:03.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:03.538907+0000) 2022-09-10T15:49:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:04 smithi104 ceph-mon[97044]: pgmap v21750: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:03.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:03.907291+0000) 2022-09-10T15:49:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:04 smithi110 ceph-mon[97633]: pgmap v21750: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:04.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:04.539159+0000) 2022-09-10T15:49:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:04.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:04.907493+0000) 2022-09-10T15:49:05.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:05.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:05.539366+0000) 2022-09-10T15:49:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:06 smithi104 ceph-mon[97044]: pgmap v21751: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:49:06.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:05.906+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:05.907723+0000) 2022-09-10T15:49:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:06 smithi110 ceph-mon[97633]: pgmap v21751: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:49:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:06.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:06.539542+0000) 2022-09-10T15:49:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:06.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:06.907920+0000) 2022-09-10T15:49:07.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:07.538+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:07.539769+0000) 2022-09-10T15:49:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:08 smithi104 ceph-mon[97044]: pgmap v21752: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:49:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:07.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:07.908172+0000) 2022-09-10T15:49:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:08 smithi110 ceph-mon[97633]: pgmap v21752: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:49:08.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:49:08.602+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:49:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:08.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:08.540015+0000) 2022-09-10T15:49:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:08.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:08.908392+0000) 2022-09-10T15:49:09.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:09.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:09.540278+0000) 2022-09-10T15:49:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:09 smithi104 ceph-mon[97044]: pgmap v21753: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:49:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:09 smithi104 ceph-mon[97044]: osdmap e1440: 12 total, 12 up, 12 in 2022-09-10T15:49:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:09.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:09.908618+0000) 2022-09-10T15:49:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:09 smithi110 ceph-mon[97633]: pgmap v21753: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:49:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:09 smithi110 ceph-mon[97633]: osdmap e1440: 12 total, 12 up, 12 in 2022-09-10T15:49:10.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:10.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:10.540427+0000) 2022-09-10T15:49:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:10.907+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:10.908854+0000) 2022-09-10T15:49:11.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:11.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:11.540663+0000) 2022-09-10T15:49:12.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:11 smithi104 ceph-mon[97044]: pgmap v21755: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:49:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:11.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:11.909125+0000) 2022-09-10T15:49:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:11 smithi110 ceph-mon[97633]: pgmap v21755: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:49:12.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:12.539+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:12.540903+0000) 2022-09-10T15:49:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:12.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:12.909324+0000) 2022-09-10T15:49:13.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:13.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:13.541131+0000) 2022-09-10T15:49:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:13 smithi104 ceph-mon[97044]: pgmap v21756: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:49:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:13.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:13.909562+0000) 2022-09-10T15:49:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:13 smithi110 ceph-mon[97633]: pgmap v21756: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:49:14.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:14.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:14.541362+0000) 2022-09-10T15:49:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:14.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:14.909787+0000) 2022-09-10T15:49:15.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:15.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:15.541591+0000) 2022-09-10T15:49:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:15 smithi104 ceph-mon[97044]: pgmap v21757: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:49:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:15.908+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:15.909968+0000) 2022-09-10T15:49:16.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:15 smithi110 ceph-mon[97633]: pgmap v21757: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:49:16.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:16.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:16.541841+0000) 2022-09-10T15:49:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:16.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:16.910228+0000) 2022-09-10T15:49:17.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:17.540+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:17.542055+0000) 2022-09-10T15:49:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:17 smithi104 ceph-mon[97044]: pgmap v21758: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:49:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:17 smithi104 ceph-mon[97044]: osdmap e1441: 12 total, 12 up, 12 in 2022-09-10T15:49:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:17.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:17.910405+0000) 2022-09-10T15:49:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:17 smithi110 ceph-mon[97633]: pgmap v21758: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:49:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:17 smithi110 ceph-mon[97633]: osdmap e1441: 12 total, 12 up, 12 in 2022-09-10T15:49:18.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:18.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:18.542271+0000) 2022-09-10T15:49:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:18.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:18.910627+0000) 2022-09-10T15:49:19.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:19.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:19.542510+0000) 2022-09-10T15:49:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:19 smithi104 ceph-mon[97044]: pgmap v21760: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:49:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:19.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:19.910813+0000) 2022-09-10T15:49:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:19 smithi110 ceph-mon[97633]: pgmap v21760: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 206 B/s wr, 10 op/s 2022-09-10T15:49:20.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:20.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:20.542731+0000) 2022-09-10T15:49:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:20.909+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:20.911036+0000) 2022-09-10T15:49:21.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:21.541+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:21.542988+0000) 2022-09-10T15:49:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:21 smithi104 ceph-mon[97044]: pgmap v21761: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:49:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:21.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:21.911291+0000) 2022-09-10T15:49:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:21 smithi110 ceph-mon[97633]: pgmap v21761: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:49:22.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:22.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:22.543225+0000) 2022-09-10T15:49:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:22.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:22.911492+0000) 2022-09-10T15:49:23.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:49:23.612+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:49:23.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:23.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:23.543390+0000) 2022-09-10T15:49:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:23 smithi104 ceph-mon[97044]: pgmap v21762: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:49:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:23.910+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:23.911670+0000) 2022-09-10T15:49:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:23 smithi110 ceph-mon[97633]: pgmap v21762: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:49:24.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:24.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:24.543616+0000) 2022-09-10T15:49:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:24.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:24.911885+0000) 2022-09-10T15:49:25.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:25.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:25.543841+0000) 2022-09-10T15:49:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:25 smithi104 ceph-mon[97044]: pgmap v21763: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:49:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:25.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:25.912137+0000) 2022-09-10T15:49:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:25 smithi110 ceph-mon[97633]: pgmap v21763: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:49:26.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:26.542+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:26.543995+0000) 2022-09-10T15:49:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:26.911+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:26.912324+0000) 2022-09-10T15:49:27.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:27.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:27.544235+0000) 2022-09-10T15:49:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:27 smithi104 ceph-mon[97044]: pgmap v21764: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:49:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:27.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:27.912523+0000) 2022-09-10T15:49:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:27 smithi110 ceph-mon[97633]: pgmap v21764: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2022-09-10T15:49:28.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:28.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:28.544567+0000) 2022-09-10T15:49:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:28.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:28.912744+0000) 2022-09-10T15:49:29.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:29.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:29.544762+0000) 2022-09-10T15:49:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:29 smithi104 ceph-mon[97044]: pgmap v21765: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:49:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:29.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:29.912996+0000) 2022-09-10T15:49:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:29 smithi110 ceph-mon[97633]: pgmap v21765: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2022-09-10T15:49:30.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:30.543+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:30.544956+0000) 2022-09-10T15:49:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:30.912+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:30.913244+0000) 2022-09-10T15:49:31.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:31.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:31.545193+0000) 2022-09-10T15:49:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:31 smithi104 ceph-mon[97044]: pgmap v21766: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:31.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:31.913534+0000) 2022-09-10T15:49:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:31 smithi110 ceph-mon[97633]: pgmap v21766: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:32.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:32.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:32.545325+0000) 2022-09-10T15:49:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:32.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:32.913799+0000) 2022-09-10T15:49:33.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:33.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:33.545459+0000) 2022-09-10T15:49:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:33 smithi104 ceph-mon[97044]: pgmap v21767: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:33.913+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:33.914036+0000) 2022-09-10T15:49:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:33 smithi110 ceph-mon[97633]: pgmap v21767: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:34.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:34.544+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:34.545596+0000) 2022-09-10T15:49:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:34.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:34.914260+0000) 2022-09-10T15:49:35.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:35.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:35.545760+0000) 2022-09-10T15:49:36.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:35.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:35.914497+0000) 2022-09-10T15:49:36.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:35 smithi104 ceph-mon[97044]: pgmap v21768: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:35 smithi110 ceph-mon[97633]: pgmap v21768: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:36.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:36.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:36.545954+0000) 2022-09-10T15:49:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:36.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:36.914670+0000) 2022-09-10T15:49:37.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:37.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:37.546215+0000) 2022-09-10T15:49:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:37 smithi104 ceph-mon[97044]: pgmap v21769: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:37.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:37.914891+0000) 2022-09-10T15:49:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:37 smithi110 ceph-mon[97633]: pgmap v21769: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:38.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:38.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:38.546470+0000) 2022-09-10T15:49:38.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:49:38.620+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:49:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:38.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:38.915044+0000) 2022-09-10T15:49:39.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:39.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:39.546681+0000) 2022-09-10T15:49:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:39 smithi104 ceph-mon[97044]: pgmap v21770: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:39.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:39.915213+0000) 2022-09-10T15:49:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:39 smithi110 ceph-mon[97633]: pgmap v21770: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:40.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:40.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:40.546830+0000) 2022-09-10T15:49:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:40.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:40.915406+0000) 2022-09-10T15:49:41.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:41.545+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:41.547054+0000) 2022-09-10T15:49:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:41 smithi104 ceph-mon[97044]: pgmap v21771: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:41.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:41.915674+0000) 2022-09-10T15:49:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:41 smithi110 ceph-mon[97633]: pgmap v21771: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:42.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:42.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:42.547162+0000) 2022-09-10T15:49:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:42.914+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:42.915936+0000) 2022-09-10T15:49:43.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:43.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:43.547350+0000) 2022-09-10T15:49:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:43 smithi104 ceph-mon[97044]: pgmap v21772: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:43.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:43.916167+0000) 2022-09-10T15:49:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:43 smithi110 ceph-mon[97633]: pgmap v21772: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:44.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:44.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:44.547622+0000) 2022-09-10T15:49:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:44.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:44.916426+0000) 2022-09-10T15:49:45.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:45.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:45.547835+0000) 2022-09-10T15:49:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:46 smithi104 ceph-mon[97044]: pgmap v21773: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:45.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:45.916624+0000) 2022-09-10T15:49:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:45 smithi110 ceph-mon[97633]: pgmap v21773: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:46.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:46.546+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:46.547993+0000) 2022-09-10T15:49:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:46.915+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:46.916873+0000) 2022-09-10T15:49:47.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:47.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:47.548231+0000) 2022-09-10T15:49:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:48 smithi104 ceph-mon[97044]: pgmap v21774: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:49:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:49:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:49:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:49:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:47.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:47.917133+0000) 2022-09-10T15:49:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:48 smithi110 ceph-mon[97633]: pgmap v21774: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:49:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:49:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:49:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:49:48.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:48.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:48.548433+0000) 2022-09-10T15:49:49.311 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:48.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:48.917270+0000) 2022-09-10T15:49:49.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:49.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:49.548664+0000) 2022-09-10T15:49:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:50 smithi104 ceph-mon[97044]: pgmap v21775: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:49.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:49.917471+0000) 2022-09-10T15:49:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:50 smithi110 ceph-mon[97633]: pgmap v21775: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:50.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:50.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:50.548842+0000) 2022-09-10T15:49:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:50.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:50.917655+0000) 2022-09-10T15:49:51.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:51.547+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:51.549043+0000) 2022-09-10T15:49:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:52 smithi104 ceph-mon[97044]: pgmap v21776: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:51.916+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:51.917869+0000) 2022-09-10T15:49:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:52 smithi110 ceph-mon[97633]: pgmap v21776: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:52.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:52.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:52.549159+0000) 2022-09-10T15:49:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:52.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:52.918057+0000) 2022-09-10T15:49:53.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:53.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:53.549369+0000) 2022-09-10T15:49:53.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:49:53.621+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:49:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:54 smithi104 ceph-mon[97044]: pgmap v21777: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:53.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:53.918294+0000) 2022-09-10T15:49:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:54 smithi110 ceph-mon[97633]: pgmap v21777: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:54.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:54.548+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:54.549608+0000) 2022-09-10T15:49:55.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:54.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:54.918561+0000) 2022-09-10T15:49:55.807 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:55.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:55.549876+0000) 2022-09-10T15:49:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:56 smithi104 ceph-mon[97044]: pgmap v21778: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:55.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:55.918805+0000) 2022-09-10T15:49:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:56 smithi110 ceph-mon[97633]: pgmap v21778: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:49:56.811 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:56.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:56.550128+0000) 2022-09-10T15:49:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:56.917+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:56.918998+0000) 2022-09-10T15:49:57.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:57.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:57.550367+0000) 2022-09-10T15:49:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:49:58 smithi104 ceph-mon[97044]: pgmap v21779: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:57.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:57.919251+0000) 2022-09-10T15:49:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:49:58 smithi110 ceph-mon[97633]: pgmap v21779: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:49:58.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:58.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:58.550626+0000) 2022-09-10T15:49:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:58.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:58.919510+0000) 2022-09-10T15:49:59.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:49:59.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:59.550808+0000) 2022-09-10T15:50:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:00 smithi104 ceph-mon[97044]: pgmap v21780: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:00.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:00 smithi104 ceph-mon[97044]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:50:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:49:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:49:59.918+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:49:59.919749+0000) 2022-09-10T15:50:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:00 smithi110 ceph-mon[97633]: pgmap v21780: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:00 smithi110 ceph-mon[97633]: overall HEALTH_WARN 1/3 mons down, quorum c,b 2022-09-10T15:50:00.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:00.549+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:00.550995+0000) 2022-09-10T15:50:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:00.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:00.919929+0000) 2022-09-10T15:50:01.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:01.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:01.551241+0000) 2022-09-10T15:50:02.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:02 smithi104 ceph-mon[97044]: pgmap v21781: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:02.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:01.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:01.920199+0000) 2022-09-10T15:50:02.417 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:02 smithi110 ceph-mon[97633]: pgmap v21781: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:02.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:02.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:02.551452+0000) 2022-09-10T15:50:03.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:02.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:02.920386+0000) 2022-09-10T15:50:03.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:03.550+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:03.551577+0000) 2022-09-10T15:50:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:04 smithi104 ceph-mon[97044]: pgmap v21782: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:03.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:03.920622+0000) 2022-09-10T15:50:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:04 smithi110 ceph-mon[97633]: pgmap v21782: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:04.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:04.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:04.551757+0000) 2022-09-10T15:50:05.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:04.919+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:04.920752+0000) 2022-09-10T15:50:05.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:05.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:05.552003+0000) 2022-09-10T15:50:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:06 smithi104 ceph-mon[97044]: pgmap v21783: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:50:06.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:05.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:05.920969+0000) 2022-09-10T15:50:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:06 smithi110 ceph-mon[97633]: pgmap v21783: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:50:06.805 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:06.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:06.552181+0000) 2022-09-10T15:50:07.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:06.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:06.921185+0000) 2022-09-10T15:50:07.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:07.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:07.552358+0000) 2022-09-10T15:50:08.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:08 smithi104 ceph-mon[97044]: pgmap v21784: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:50:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:07.920+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:07.921419+0000) 2022-09-10T15:50:08.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:08 smithi110 ceph-mon[97633]: pgmap v21784: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:50:08.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:50:08.637+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:50:08.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:08.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:08.552539+0000) 2022-09-10T15:50:09.303 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:09 smithi104 ceph-mon[97044]: osdmap e1442: 12 total, 12 up, 12 in 2022-09-10T15:50:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:08.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:08.921670+0000) 2022-09-10T15:50:09.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:09 smithi110 ceph-mon[97633]: osdmap e1442: 12 total, 12 up, 12 in 2022-09-10T15:50:09.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:09.551+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:09.552806+0000) 2022-09-10T15:50:10.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:10 smithi104 ceph-mon[97044]: pgmap v21786: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:50:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:09.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:09.921894+0000) 2022-09-10T15:50:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:10 smithi110 ceph-mon[97633]: pgmap v21786: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:50:10.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:10.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:10.552977+0000) 2022-09-10T15:50:11.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:10.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:10.922163+0000) 2022-09-10T15:50:11.809 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:11.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:11.553199+0000) 2022-09-10T15:50:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:11.921+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:11.922440+0000) 2022-09-10T15:50:12.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:12 smithi110 ceph-mon[97633]: pgmap v21787: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:50:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:12 smithi104 ceph-mon[97044]: pgmap v21787: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:50:12.806 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:12.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:12.553384+0000) 2022-09-10T15:50:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:12.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:12.922658+0000) 2022-09-10T15:50:13.924 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:13.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:13.553519+0000) 2022-09-10T15:50:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:13 smithi104 ceph-mon[97044]: pgmap v21788: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:50:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:13 smithi104 ceph-mon[97044]: osdmap e1443: 12 total, 12 up, 12 in 2022-09-10T15:50:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:13.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:13.922884+0000) 2022-09-10T15:50:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:13 smithi110 ceph-mon[97633]: pgmap v21788: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:50:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:13 smithi110 ceph-mon[97633]: osdmap e1443: 12 total, 12 up, 12 in 2022-09-10T15:50:14.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:14.552+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:14.553740+0000) 2022-09-10T15:50:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:14.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:14.923172+0000) 2022-09-10T15:50:15.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:15.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:15.553957+0000) 2022-09-10T15:50:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:15 smithi104 ceph-mon[97044]: pgmap v21790: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:50:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:15.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:15.923315+0000) 2022-09-10T15:50:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:15 smithi110 ceph-mon[97633]: pgmap v21790: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:50:16.925 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:16.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:16.554210+0000) 2022-09-10T15:50:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:16.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:16.923537+0000) 2022-09-10T15:50:17.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:17.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:17.554391+0000) 2022-09-10T15:50:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:17 smithi104 ceph-mon[97044]: pgmap v21791: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:50:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:17.922+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:17.923807+0000) 2022-09-10T15:50:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:17 smithi110 ceph-mon[97633]: pgmap v21791: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:50:18.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:18.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:18.554591+0000) 2022-09-10T15:50:19.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:18.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:18.923982+0000) 2022-09-10T15:50:19.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:19.553+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:19.554770+0000) 2022-09-10T15:50:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:19 smithi104 ceph-mon[97044]: pgmap v21792: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:50:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:19.923+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:19.924268+0000) 2022-09-10T15:50:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:19 smithi110 ceph-mon[97633]: pgmap v21792: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:50:20.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:20.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:20.554934+0000) 2022-09-10T15:50:21.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:20.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:20.924485+0000) 2022-09-10T15:50:21.926 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:21.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:21.555081+0000) 2022-09-10T15:50:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:21 smithi104 ceph-mon[97044]: pgmap v21793: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:50:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:21.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:21.924717+0000) 2022-09-10T15:50:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:21 smithi110 ceph-mon[97633]: pgmap v21793: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:50:22.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:22.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:22.555286+0000) 2022-09-10T15:50:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:22.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:22.924988+0000) 2022-09-10T15:50:23.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:23.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:23.555509+0000) 2022-09-10T15:50:23.930 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:50:23.651+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:50:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:23 smithi104 ceph-mon[97044]: pgmap v21794: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:50:24.315 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:23.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:23.925232+0000) 2022-09-10T15:50:24.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:23 smithi110 ceph-mon[97633]: pgmap v21794: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:50:24.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:24.554+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:24.555774+0000) 2022-09-10T15:50:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:24.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:24.925421+0000) 2022-09-10T15:50:25.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:25.555+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:25.556002+0000) 2022-09-10T15:50:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:25 smithi104 ceph-mon[97044]: pgmap v21795: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:50:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:25.924+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:25.925666+0000) 2022-09-10T15:50:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:25 smithi110 ceph-mon[97633]: pgmap v21795: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:50:26.927 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:26.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:26.556235+0000) 2022-09-10T15:50:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:26.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:26.925875+0000) 2022-09-10T15:50:27.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:27.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:27.556467+0000) 2022-09-10T15:50:28.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:27 smithi104 ceph-mon[97044]: pgmap v21796: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:27.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:27.926063+0000) 2022-09-10T15:50:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:27 smithi110 ceph-mon[97633]: pgmap v21796: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:28.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:28.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:28.556703+0000) 2022-09-10T15:50:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:28.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:28.926240+0000) 2022-09-10T15:50:29.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:29.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:29.556892+0000) 2022-09-10T15:50:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:29 smithi104 ceph-mon[97044]: pgmap v21797: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:29.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:29.926458+0000) 2022-09-10T15:50:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:29 smithi110 ceph-mon[97633]: pgmap v21797: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:30.928 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:30.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:30.557016+0000) 2022-09-10T15:50:31.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:30.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:30.926692+0000) 2022-09-10T15:50:31.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:31.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:31.557183+0000) 2022-09-10T15:50:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:31 smithi104 ceph-mon[97044]: pgmap v21798: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:31.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:31.926869+0000) 2022-09-10T15:50:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:31 smithi110 ceph-mon[97633]: pgmap v21798: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:32.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:32.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:32.557362+0000) 2022-09-10T15:50:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:32.925+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:32.927066+0000) 2022-09-10T15:50:33.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:33.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:33.557572+0000) 2022-09-10T15:50:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:33 smithi104 ceph-mon[97044]: pgmap v21799: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:33.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:33.927290+0000) 2022-09-10T15:50:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:33 smithi110 ceph-mon[97633]: pgmap v21799: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:34.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:34.556+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:34.557830+0000) 2022-09-10T15:50:35.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:34.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:34.927509+0000) 2022-09-10T15:50:35.929 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:35.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:35.558073+0000) 2022-09-10T15:50:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:35 smithi104 ceph-mon[97044]: pgmap v21800: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:36.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:35.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:35.927711+0000) 2022-09-10T15:50:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:35 smithi110 ceph-mon[97633]: pgmap v21800: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:36.930 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:36.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:36.558273+0000) 2022-09-10T15:50:37.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:36.926+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:36.927955+0000) 2022-09-10T15:50:37.930 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:37.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:37.558533+0000) 2022-09-10T15:50:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:37 smithi104 ceph-mon[97044]: pgmap v21801: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:37.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:37.928221+0000) 2022-09-10T15:50:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:37 smithi110 ceph-mon[97633]: pgmap v21801: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:38.931 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:50:38.654+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:50:38.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:38.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:38.558781+0000) 2022-09-10T15:50:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:38.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:38.928446+0000) 2022-09-10T15:50:39.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:39.557+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:39.558958+0000) 2022-09-10T15:50:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:40 smithi104 ceph-mon[97044]: pgmap v21802: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:39.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:39.928715+0000) 2022-09-10T15:50:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:39 smithi110 ceph-mon[97633]: pgmap v21802: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:40.884 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:40.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:40.559173+0000) 2022-09-10T15:50:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:40.927+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:40.928979+0000) 2022-09-10T15:50:41.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:41.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:41.559367+0000) 2022-09-10T15:50:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:42 smithi104 ceph-mon[97044]: pgmap v21803: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:42.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:41.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:41.929253+0000) 2022-09-10T15:50:42.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:42 smithi110 ceph-mon[97633]: pgmap v21803: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:42.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:42.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:42.559571+0000) 2022-09-10T15:50:43.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:42.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:42.929501+0000) 2022-09-10T15:50:43.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:43.558+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:43.559768+0000) 2022-09-10T15:50:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:44 smithi104 ceph-mon[97044]: pgmap v21804: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:43.928+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:43.929689+0000) 2022-09-10T15:50:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:44 smithi110 ceph-mon[97633]: pgmap v21804: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:44.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:44.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:44.559981+0000) 2022-09-10T15:50:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:44.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:44.929898+0000) 2022-09-10T15:50:45.931 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:45.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:45.560161+0000) 2022-09-10T15:50:46.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:46 smithi104 ceph-mon[97044]: pgmap v21805: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:45.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:45.930075+0000) 2022-09-10T15:50:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:46 smithi110 ceph-mon[97633]: pgmap v21805: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:46.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:46.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:46.560342+0000) 2022-09-10T15:50:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:46.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:46.930311+0000) 2022-09-10T15:50:47.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:47.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:47.560604+0000) 2022-09-10T15:50:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:48 smithi104 ceph-mon[97044]: pgmap v21806: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:50:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:50:48.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:50:48.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:50:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:47.929+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:47.930522+0000) 2022-09-10T15:50:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:48 smithi110 ceph-mon[97633]: pgmap v21806: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:50:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:50:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:50:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:50:48.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:48.559+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:48.560862+0000) 2022-09-10T15:50:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:48.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:48.930798+0000) 2022-09-10T15:50:49.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:49.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:49.561083+0000) 2022-09-10T15:50:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:50 smithi104 ceph-mon[97044]: pgmap v21807: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:49.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:49.931032+0000) 2022-09-10T15:50:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:50 smithi110 ceph-mon[97633]: pgmap v21807: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:50.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:50.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:50.561323+0000) 2022-09-10T15:50:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:50.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:50.931177+0000) 2022-09-10T15:50:51.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:51.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:51.561488+0000) 2022-09-10T15:50:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:52 smithi104 ceph-mon[97044]: pgmap v21808: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:51.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:51.931393+0000) 2022-09-10T15:50:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:52 smithi110 ceph-mon[97633]: pgmap v21808: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:52.933 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:52.560+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:52.561635+0000) 2022-09-10T15:50:53.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:52.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:52.931588+0000) 2022-09-10T15:50:53.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:53.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:53.561838+0000) 2022-09-10T15:50:53.935 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:50:53.655+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:50:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:54 smithi104 ceph-mon[97044]: pgmap v21809: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:53.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:53.931796+0000) 2022-09-10T15:50:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:54 smithi110 ceph-mon[97633]: pgmap v21809: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:54.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:54.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:54.562040+0000) 2022-09-10T15:50:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:54.930+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:54.932020+0000) 2022-09-10T15:50:55.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:55.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:55.562230+0000) 2022-09-10T15:50:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:56 smithi104 ceph-mon[97044]: pgmap v21810: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:55.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:55.932247+0000) 2022-09-10T15:50:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:56 smithi110 ceph-mon[97633]: pgmap v21810: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:50:56.934 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:56.561+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:56.562391+0000) 2022-09-10T15:50:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:56.931+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:56.932493+0000) 2022-09-10T15:50:57.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:57.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:57.562594+0000) 2022-09-10T15:50:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:50:58 smithi104 ceph-mon[97044]: pgmap v21811: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:57.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:57.932776+0000) 2022-09-10T15:50:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:50:58 smithi110 ceph-mon[97633]: pgmap v21811: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:50:58.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:58.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:58.562802+0000) 2022-09-10T15:50:59.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:58.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:58.933014+0000) 2022-09-10T15:50:59.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:50:59.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:59.563022+0000) 2022-09-10T15:51:00.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:00 smithi104 ceph-mon[97044]: pgmap v21812: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:50:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:50:59.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:50:59.933242+0000) 2022-09-10T15:51:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:00 smithi110 ceph-mon[97633]: pgmap v21812: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:00.935 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:00.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:00.563263+0000) 2022-09-10T15:51:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:00.932+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:00.933447+0000) 2022-09-10T15:51:01.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:01.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:01.563493+0000) 2022-09-10T15:51:02.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:02 smithi104 ceph-mon[97044]: pgmap v21813: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:01.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:01.933697+0000) 2022-09-10T15:51:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:02 smithi110 ceph-mon[97633]: pgmap v21813: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:02.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:02.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:02.563703+0000) 2022-09-10T15:51:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:02.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:02.933903+0000) 2022-09-10T15:51:03.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:03.562+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:03.563947+0000) 2022-09-10T15:51:04.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:04 smithi104 ceph-mon[97044]: pgmap v21814: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:03.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:03.934162+0000) 2022-09-10T15:51:04.422 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:04 smithi110 ceph-mon[97633]: pgmap v21814: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:04.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:04.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:04.564253+0000) 2022-09-10T15:51:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:04.933+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:04.934334+0000) 2022-09-10T15:51:05.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:05.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:05.564486+0000) 2022-09-10T15:51:06.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:06 smithi104 ceph-mon[97044]: pgmap v21815: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:51:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:05.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:05.934566+0000) 2022-09-10T15:51:06.415 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:06 smithi110 ceph-mon[97633]: pgmap v21815: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:51:06.936 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:06.563+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:06.564661+0000) 2022-09-10T15:51:07.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:06.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:06.934733+0000) 2022-09-10T15:51:07.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:07.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:07.564876+0000) 2022-09-10T15:51:08.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:07.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:07.934973+0000) 2022-09-10T15:51:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:08 smithi110 ceph-mon[97633]: pgmap v21816: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:51:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:08 smithi104 ceph-mon[97044]: pgmap v21816: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:51:08.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:08.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:08.565072+0000) 2022-09-10T15:51:08.938 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:51:08.658+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:51:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:08.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:08.935166+0000) 2022-09-10T15:51:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:09 smithi110 ceph-mon[97633]: osdmap e1444: 12 total, 12 up, 12 in 2022-09-10T15:51:09.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:09 smithi104 ceph-mon[97044]: osdmap e1444: 12 total, 12 up, 12 in 2022-09-10T15:51:09.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:09.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:09.565307+0000) 2022-09-10T15:51:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:09.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:09.935395+0000) 2022-09-10T15:51:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:10 smithi110 ceph-mon[97633]: pgmap v21818: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:51:10.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:10 smithi104 ceph-mon[97044]: pgmap v21818: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:51:10.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:10.564+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:10.565481+0000) 2022-09-10T15:51:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:10.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:10.935648+0000) 2022-09-10T15:51:11.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:11.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:11.565742+0000) 2022-09-10T15:51:12.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:11.934+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:11.935934+0000) 2022-09-10T15:51:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:12 smithi110 ceph-mon[97633]: pgmap v21819: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:51:12.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:12 smithi104 ceph-mon[97044]: pgmap v21819: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:51:12.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:12.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:12.565960+0000) 2022-09-10T15:51:13.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:12.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:12.936161+0000) 2022-09-10T15:51:13.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:13.565+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:13.566184+0000) 2022-09-10T15:51:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:13 smithi104 ceph-mon[97044]: pgmap v21820: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:51:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:13 smithi104 ceph-mon[97044]: osdmap e1445: 12 total, 12 up, 12 in 2022-09-10T15:51:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:13.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:13.936370+0000) 2022-09-10T15:51:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:13 smithi110 ceph-mon[97633]: pgmap v21820: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:51:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:13 smithi110 ceph-mon[97633]: osdmap e1445: 12 total, 12 up, 12 in 2022-09-10T15:51:14.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:14.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:14.566431+0000) 2022-09-10T15:51:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:14.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:14.936559+0000) 2022-09-10T15:51:15.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:15.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:15.566642+0000) 2022-09-10T15:51:16.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:15 smithi104 ceph-mon[97044]: pgmap v21822: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:51:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:15.936+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:15.936758+0000) 2022-09-10T15:51:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:15 smithi110 ceph-mon[97633]: pgmap v21822: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:51:16.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:16.566+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:16.566917+0000) 2022-09-10T15:51:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:16.935+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:16.936976+0000) 2022-09-10T15:51:17.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:17.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:17.567193+0000) 2022-09-10T15:51:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:17 smithi104 ceph-mon[97044]: pgmap v21823: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:51:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:17.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:17.937205+0000) 2022-09-10T15:51:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:17 smithi110 ceph-mon[97633]: pgmap v21823: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:51:18.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:18.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:18.567436+0000) 2022-09-10T15:51:19.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:18.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:18.937362+0000) 2022-09-10T15:51:19.939 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:19.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:19.567615+0000) 2022-09-10T15:51:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:19 smithi104 ceph-mon[97044]: pgmap v21824: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:51:20.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:19.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:19.937575+0000) 2022-09-10T15:51:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:19 smithi110 ceph-mon[97633]: pgmap v21824: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:51:20.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:20.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:20.567784+0000) 2022-09-10T15:51:21.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:20.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:20.937795+0000) 2022-09-10T15:51:21.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:21.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:21.568019+0000) 2022-09-10T15:51:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:21 smithi104 ceph-mon[97044]: pgmap v21825: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:51:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:21.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:21.938056+0000) 2022-09-10T15:51:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:21 smithi110 ceph-mon[97633]: pgmap v21825: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:51:22.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:22.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:22.568241+0000) 2022-09-10T15:51:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:22.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:22.938332+0000) 2022-09-10T15:51:23.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:23.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:23.568442+0000) 2022-09-10T15:51:23.942 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:51:23.662+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:51:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:23 smithi104 ceph-mon[97044]: pgmap v21826: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:51:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:23.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:23.938553+0000) 2022-09-10T15:51:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:23 smithi110 ceph-mon[97633]: pgmap v21826: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:51:24.940 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:24.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:24.568622+0000) 2022-09-10T15:51:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:24.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:24.938781+0000) 2022-09-10T15:51:25.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:25.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:25.568841+0000) 2022-09-10T15:51:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:25 smithi104 ceph-mon[97044]: pgmap v21827: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:51:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:25.937+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:25.938980+0000) 2022-09-10T15:51:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:25 smithi110 ceph-mon[97633]: pgmap v21827: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:51:26.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:26.567+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:26.569045+0000) 2022-09-10T15:51:27.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:26.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:26.939205+0000) 2022-09-10T15:51:27.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:27.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:27.569246+0000) 2022-09-10T15:51:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:27 smithi104 ceph-mon[97044]: pgmap v21828: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:27.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:27.939366+0000) 2022-09-10T15:51:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:27 smithi110 ceph-mon[97633]: pgmap v21828: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:28.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:28.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:28.569458+0000) 2022-09-10T15:51:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:28.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:28.939600+0000) 2022-09-10T15:51:29.941 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:29.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:29.569697+0000) 2022-09-10T15:51:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:30 smithi104 ceph-mon[97044]: pgmap v21829: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:29.938+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:29.939802+0000) 2022-09-10T15:51:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:30 smithi110 ceph-mon[97633]: pgmap v21829: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:30.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:30.568+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:30.569865+0000) 2022-09-10T15:51:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:30.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:30.940047+0000) 2022-09-10T15:51:31.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:31.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:31.570121+0000) 2022-09-10T15:51:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:32 smithi104 ceph-mon[97044]: pgmap v21830: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:31.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:31.940359+0000) 2022-09-10T15:51:32.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:32 smithi110 ceph-mon[97633]: pgmap v21830: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:32.942 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:32.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:32.570382+0000) 2022-09-10T15:51:33.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:32.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:32.940591+0000) 2022-09-10T15:51:33.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:33.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:33.570583+0000) 2022-09-10T15:51:34.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:34 smithi104 ceph-mon[97044]: pgmap v21831: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:33.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:33.940853+0000) 2022-09-10T15:51:34.419 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:34 smithi110 ceph-mon[97633]: pgmap v21831: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:34.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:34.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:34.570849+0000) 2022-09-10T15:51:35.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:34.939+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:34.941063+0000) 2022-09-10T15:51:35.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:35.569+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:35.571046+0000) 2022-09-10T15:51:36.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:36 smithi104 ceph-mon[97044]: pgmap v21832: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:35.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:35.941286+0000) 2022-09-10T15:51:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:36 smithi110 ceph-mon[97633]: pgmap v21832: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:36.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:36.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:36.571235+0000) 2022-09-10T15:51:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:36.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:36.941500+0000) 2022-09-10T15:51:37.943 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:37.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:37.571507+0000) 2022-09-10T15:51:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:38 smithi104 ceph-mon[97044]: pgmap v21833: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:37.940+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:37.941678+0000) 2022-09-10T15:51:38.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:38 smithi110 ceph-mon[97633]: pgmap v21833: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:38.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:38.570+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:38.571777+0000) 2022-09-10T15:51:38.944 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:51:38.677+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:51:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:38.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:38.941934+0000) 2022-09-10T15:51:39.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:39.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:39.571995+0000) 2022-09-10T15:51:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:40 smithi104 ceph-mon[97044]: pgmap v21834: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:39.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:39.942183+0000) 2022-09-10T15:51:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:40 smithi110 ceph-mon[97633]: pgmap v21834: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:40.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:40.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:40.572164+0000) 2022-09-10T15:51:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:40.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:40.942410+0000) 2022-09-10T15:51:41.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:41.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:41.572417+0000) 2022-09-10T15:51:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:42 smithi104 ceph-mon[97044]: pgmap v21835: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:41.941+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:41.942647+0000) 2022-09-10T15:51:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:42 smithi110 ceph-mon[97633]: pgmap v21835: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:42.944 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:42.571+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:42.572657+0000) 2022-09-10T15:51:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:42.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:42.942873+0000) 2022-09-10T15:51:43.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:43.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:43.572912+0000) 2022-09-10T15:51:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:44 smithi104 ceph-mon[97044]: pgmap v21836: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:43.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:43.943126+0000) 2022-09-10T15:51:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:44 smithi110 ceph-mon[97633]: pgmap v21836: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:44.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:44.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:44.573147+0000) 2022-09-10T15:51:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:44.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:44.943312+0000) 2022-09-10T15:51:45.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:45.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:45.573369+0000) 2022-09-10T15:51:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:46 smithi104 ceph-mon[97044]: pgmap v21837: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:45.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:45.943544+0000) 2022-09-10T15:51:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:46 smithi110 ceph-mon[97633]: pgmap v21837: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:46.945 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:46.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:46.573582+0000) 2022-09-10T15:51:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:46.942+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:46.943703+0000) 2022-09-10T15:51:47.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:47.572+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:47.573899+0000) 2022-09-10T15:51:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:48 smithi104 ceph-mon[97044]: pgmap v21838: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:51:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:51:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:51:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:51:48.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:47.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:47.943888+0000) 2022-09-10T15:51:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:48 smithi110 ceph-mon[97633]: pgmap v21838: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:51:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:51:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:51:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:51:48.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:48.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:48.574144+0000) 2022-09-10T15:51:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:48.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:48.944076+0000) 2022-09-10T15:51:49.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:49.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:49.574378+0000) 2022-09-10T15:51:50.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:50 smithi104 ceph-mon[97044]: pgmap v21839: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:49.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:49.944212+0000) 2022-09-10T15:51:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:50 smithi110 ceph-mon[97633]: pgmap v21839: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:50.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:50.573+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:50.574655+0000) 2022-09-10T15:51:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:50.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:50.944395+0000) 2022-09-10T15:51:51.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:51.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:51.574917+0000) 2022-09-10T15:51:52.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:52 smithi104 ceph-mon[97044]: pgmap v21840: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:51.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:51.944612+0000) 2022-09-10T15:51:52.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:52 smithi110 ceph-mon[97633]: pgmap v21840: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:52.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:52.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:52.575185+0000) 2022-09-10T15:51:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:52.943+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:52.944876+0000) 2022-09-10T15:51:53.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:53.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:53.575407+0000) 2022-09-10T15:51:53.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:51:53.687+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:51:54.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:54 smithi104 ceph-mon[97044]: pgmap v21841: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:53.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:53.945072+0000) 2022-09-10T15:51:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:54 smithi110 ceph-mon[97633]: pgmap v21841: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:54.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:54.574+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:54.575636+0000) 2022-09-10T15:51:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:54.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:54.945279+0000) 2022-09-10T15:51:55.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:55.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:55.575822+0000) 2022-09-10T15:51:56.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:56 smithi104 ceph-mon[97044]: pgmap v21842: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:55.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:55.945416+0000) 2022-09-10T15:51:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:56 smithi110 ceph-mon[97633]: pgmap v21842: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:51:56.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:56.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:56.576014+0000) 2022-09-10T15:51:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:56.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:56.945599+0000) 2022-09-10T15:51:57.947 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:57.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:57.576295+0000) 2022-09-10T15:51:58.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:51:58 smithi104 ceph-mon[97044]: pgmap v21843: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:57.944+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:57.945770+0000) 2022-09-10T15:51:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:51:58 smithi110 ceph-mon[97633]: pgmap v21843: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:51:58.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:58.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:58.576474+0000) 2022-09-10T15:51:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:58.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:58.946000+0000) 2022-09-10T15:51:59.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:51:59.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:59.576670+0000) 2022-09-10T15:52:00.304 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:00 smithi104 ceph-mon[97044]: pgmap v21844: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:51:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:51:59.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:51:59.946226+0000) 2022-09-10T15:52:00.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:00 smithi110 ceph-mon[97633]: pgmap v21844: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:00.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:00.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:00.576855+0000) 2022-09-10T15:52:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:00.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:00.946510+0000) 2022-09-10T15:52:01.948 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:01.575+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:01.577077+0000) 2022-09-10T15:52:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:01.945+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:01.946789+0000) 2022-09-10T15:52:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:02 smithi110 ceph-mon[97633]: pgmap v21845: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:02 smithi104 ceph-mon[97044]: pgmap v21845: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:02.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:02.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:02.577264+0000) 2022-09-10T15:52:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:02.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:02.947013+0000) 2022-09-10T15:52:03.949 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:03.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:03.577474+0000) 2022-09-10T15:52:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:03.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:03.947288+0000) 2022-09-10T15:52:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:04 smithi110 ceph-mon[97633]: pgmap v21846: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:04.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:04 smithi104 ceph-mon[97044]: pgmap v21846: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:04.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:04.576+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:04.577727+0000) 2022-09-10T15:52:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:04.946+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:04.947490+0000) 2022-09-10T15:52:05.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:05.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:05.577970+0000) 2022-09-10T15:52:06.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:05.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:05.947713+0000) 2022-09-10T15:52:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:06 smithi110 ceph-mon[97633]: pgmap v21847: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:52:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:06 smithi104 ceph-mon[97044]: pgmap v21847: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:52:06.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:06.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:06.578190+0000) 2022-09-10T15:52:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:06.947+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:06.947964+0000) 2022-09-10T15:52:07.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:07.577+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:07.578354+0000) 2022-09-10T15:52:08.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:07.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:07.948237+0000) 2022-09-10T15:52:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:08 smithi110 ceph-mon[97633]: pgmap v21848: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:52:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:08 smithi104 ceph-mon[97044]: pgmap v21848: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:52:08.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:08.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:08.578603+0000) 2022-09-10T15:52:09.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:52:08.703+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:52:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:08.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:08.948392+0000) 2022-09-10T15:52:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:09 smithi110 ceph-mon[97633]: osdmap e1446: 12 total, 12 up, 12 in 2022-09-10T15:52:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:09 smithi104 ceph-mon[97044]: osdmap e1446: 12 total, 12 up, 12 in 2022-09-10T15:52:09.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:09.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:09.578827+0000) 2022-09-10T15:52:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:09.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:09.948610+0000) 2022-09-10T15:52:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:10 smithi110 ceph-mon[97633]: pgmap v21850: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:52:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:10 smithi104 ceph-mon[97044]: pgmap v21850: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:52:10.950 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:10.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:10.579020+0000) 2022-09-10T15:52:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:10.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:10.948813+0000) 2022-09-10T15:52:11.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:11.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:11.579197+0000) 2022-09-10T15:52:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:11.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:11.949070+0000) 2022-09-10T15:52:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:12 smithi110 ceph-mon[97633]: pgmap v21851: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:52:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:12 smithi104 ceph-mon[97044]: pgmap v21851: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:52:12.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:12.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:12.579344+0000) 2022-09-10T15:52:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:12.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:12.949273+0000) 2022-09-10T15:52:13.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:13.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:13.579540+0000) 2022-09-10T15:52:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:13 smithi104 ceph-mon[97044]: pgmap v21852: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:52:14.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:13 smithi104 ceph-mon[97044]: osdmap e1447: 12 total, 12 up, 12 in 2022-09-10T15:52:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:13.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:13.949521+0000) 2022-09-10T15:52:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:13 smithi110 ceph-mon[97633]: pgmap v21852: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:52:14.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:13 smithi110 ceph-mon[97633]: osdmap e1447: 12 total, 12 up, 12 in 2022-09-10T15:52:14.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:14.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:14.579722+0000) 2022-09-10T15:52:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:14.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:14.949718+0000) 2022-09-10T15:52:15.951 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:15.578+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:15.579902+0000) 2022-09-10T15:52:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:15 smithi104 ceph-mon[97044]: pgmap v21854: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:52:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:15.948+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:15.949964+0000) 2022-09-10T15:52:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:15 smithi110 ceph-mon[97633]: pgmap v21854: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:52:16.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:16.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:16.580178+0000) 2022-09-10T15:52:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:16.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:16.950217+0000) 2022-09-10T15:52:17.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:17.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:17.580392+0000) 2022-09-10T15:52:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:18 smithi104 ceph-mon[97044]: pgmap v21855: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:52:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:17.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:17.950395+0000) 2022-09-10T15:52:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:18 smithi110 ceph-mon[97633]: pgmap v21855: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:52:18.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:18.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:18.580593+0000) 2022-09-10T15:52:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:18.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:18.950572+0000) 2022-09-10T15:52:19.952 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:19.579+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:19.580845+0000) 2022-09-10T15:52:20.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:20 smithi104 ceph-mon[97044]: pgmap v21856: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:52:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:19.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:19.950788+0000) 2022-09-10T15:52:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:20 smithi110 ceph-mon[97633]: pgmap v21856: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:52:20.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:20.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:20.581082+0000) 2022-09-10T15:52:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:20.949+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:20.951025+0000) 2022-09-10T15:52:21.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:21.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:21.581286+0000) 2022-09-10T15:52:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:22 smithi104 ceph-mon[97044]: pgmap v21857: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:52:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:21.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:21.951288+0000) 2022-09-10T15:52:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:22 smithi110 ceph-mon[97633]: pgmap v21857: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:52:22.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:22.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:22.581536+0000) 2022-09-10T15:52:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:22.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:22.951422+0000) 2022-09-10T15:52:23.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:23.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:23.581724+0000) 2022-09-10T15:52:23.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:52:23.717+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:52:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:24 smithi104 ceph-mon[97044]: pgmap v21858: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:52:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:23.950+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:23.951676+0000) 2022-09-10T15:52:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:24 smithi110 ceph-mon[97633]: pgmap v21858: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:52:24.953 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:24.580+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:24.581964+0000) 2022-09-10T15:52:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:24.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:24.951919+0000) 2022-09-10T15:52:25.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:25.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:25.582170+0000) 2022-09-10T15:52:26.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:26 smithi104 ceph-mon[97044]: pgmap v21859: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:52:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:25.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:25.952142+0000) 2022-09-10T15:52:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:26 smithi110 ceph-mon[97633]: pgmap v21859: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:52:26.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:26.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:26.582339+0000) 2022-09-10T15:52:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:26.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:26.952361+0000) 2022-09-10T15:52:27.954 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:27.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:27.582588+0000) 2022-09-10T15:52:28.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:28 smithi104 ceph-mon[97044]: pgmap v21860: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T15:52:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:27.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:27.952533+0000) 2022-09-10T15:52:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:28 smithi110 ceph-mon[97633]: pgmap v21860: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T15:52:28.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:28.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:28.582805+0000) 2022-09-10T15:52:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:28.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:28.952781+0000) 2022-09-10T15:52:29.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:29.581+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:29.583049+0000) 2022-09-10T15:52:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:30 smithi104 ceph-mon[97044]: pgmap v21861: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T15:52:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:29.951+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:29.953059+0000) 2022-09-10T15:52:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:30 smithi110 ceph-mon[97633]: pgmap v21861: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2022-09-10T15:52:30.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:30.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:30.583324+0000) 2022-09-10T15:52:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:30.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:30.953239+0000) 2022-09-10T15:52:31.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:31.582+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:31.583561+0000) 2022-09-10T15:52:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:32 smithi104 ceph-mon[97044]: pgmap v21862: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T15:52:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:31.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:31.953447+0000) 2022-09-10T15:52:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:32 smithi110 ceph-mon[97633]: pgmap v21862: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 12 op/s 2022-09-10T15:52:32.955 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:32.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:32.583787+0000) 2022-09-10T15:52:33.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:32.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:32.953613+0000) 2022-09-10T15:52:33.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:33.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:33.584046+0000) 2022-09-10T15:52:34.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:34 smithi104 ceph-mon[97044]: pgmap v21863: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:52:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:33.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:33.953842+0000) 2022-09-10T15:52:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:34 smithi110 ceph-mon[97633]: pgmap v21863: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:52:34.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:34.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:34.584272+0000) 2022-09-10T15:52:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:34.952+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:34.954053+0000) 2022-09-10T15:52:35.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:35.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:35.584493+0000) 2022-09-10T15:52:36.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:36 smithi104 ceph-mon[97044]: pgmap v21864: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:35.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:35.954342+0000) 2022-09-10T15:52:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:36 smithi110 ceph-mon[97633]: pgmap v21864: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:36.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:36.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:36.584690+0000) 2022-09-10T15:52:37.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:36.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:36.954556+0000) 2022-09-10T15:52:37.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:37.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:37.584858+0000) 2022-09-10T15:52:38.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:38 smithi104 ceph-mon[97044]: pgmap v21865: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:37.953+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:37.954696+0000) 2022-09-10T15:52:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:38 smithi110 ceph-mon[97633]: pgmap v21865: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:38.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:38.583+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:38.585043+0000) 2022-09-10T15:52:38.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:52:38.728+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:52:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:38.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:38.954927+0000) 2022-09-10T15:52:39.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:39.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:39.585247+0000) 2022-09-10T15:52:40.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:40 smithi104 ceph-mon[97044]: pgmap v21866: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:39.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:39.955135+0000) 2022-09-10T15:52:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:40 smithi110 ceph-mon[97633]: pgmap v21866: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:40.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:40.584+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:40.585461+0000) 2022-09-10T15:52:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:40.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:40.955295+0000) 2022-09-10T15:52:41.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:41.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:41.585711+0000) 2022-09-10T15:52:42.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:42 smithi104 ceph-mon[97044]: pgmap v21867: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:41.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:41.955570+0000) 2022-09-10T15:52:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:42 smithi110 ceph-mon[97633]: pgmap v21867: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:42.957 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:42.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:42.585968+0000) 2022-09-10T15:52:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:42.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:42.955710+0000) 2022-09-10T15:52:43.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:43.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:43.586162+0000) 2022-09-10T15:52:44.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:44 smithi104 ceph-mon[97044]: pgmap v21868: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:43.954+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:43.955936+0000) 2022-09-10T15:52:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:44 smithi110 ceph-mon[97633]: pgmap v21868: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:44.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:44.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:44.586373+0000) 2022-09-10T15:52:45.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:44.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:44.956172+0000) 2022-09-10T15:52:45.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:45.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:45.586639+0000) 2022-09-10T15:52:46.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:46 smithi104 ceph-mon[97044]: pgmap v21869: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:46.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:45.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:45.956381+0000) 2022-09-10T15:52:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:46 smithi110 ceph-mon[97633]: pgmap v21869: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:46.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:46.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:46.586852+0000) 2022-09-10T15:52:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:46.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:46.956619+0000) 2022-09-10T15:52:47.958 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:47.585+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:47.587010+0000) 2022-09-10T15:52:48.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:48 smithi104 ceph-mon[97044]: pgmap v21870: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:52:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:52:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:52:48.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:52:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:47.955+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:47.956777+0000) 2022-09-10T15:52:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:48 smithi110 ceph-mon[97633]: pgmap v21870: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:52:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:52:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:52:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:52:48.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:48.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:48.587225+0000) 2022-09-10T15:52:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:48.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:48.957000+0000) 2022-09-10T15:52:49.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:49.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:49.587471+0000) 2022-09-10T15:52:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:49.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:49.957256+0000) 2022-09-10T15:52:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:50 smithi110 ceph-mon[97633]: pgmap v21871: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:50 smithi104 ceph-mon[97044]: pgmap v21871: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:50.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:50.586+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:50.587647+0000) 2022-09-10T15:52:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:50.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:50.957491+0000) 2022-09-10T15:52:51.959 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:51.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:51.587867+0000) 2022-09-10T15:52:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:51.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:51.957777+0000) 2022-09-10T15:52:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:52 smithi110 ceph-mon[97633]: pgmap v21872: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:52 smithi104 ceph-mon[97044]: pgmap v21872: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:52.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:52.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:52.588077+0000) 2022-09-10T15:52:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:52.956+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:52.958005+0000) 2022-09-10T15:52:53.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:53.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:53.588261+0000) 2022-09-10T15:52:53.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:52:53.733+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:52:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:53.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:53.958253+0000) 2022-09-10T15:52:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:54 smithi110 ceph-mon[97633]: pgmap v21873: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:54.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:54 smithi104 ceph-mon[97044]: pgmap v21873: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:54.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:54.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:54.588457+0000) 2022-09-10T15:52:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:54.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:54.958466+0000) 2022-09-10T15:52:55.960 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:55.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:55.588611+0000) 2022-09-10T15:52:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:55.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:55.958690+0000) 2022-09-10T15:52:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:56 smithi110 ceph-mon[97633]: pgmap v21874: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:56 smithi104 ceph-mon[97044]: pgmap v21874: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:52:56.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:56.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:56.588734+0000) 2022-09-10T15:52:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:56.957+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:56.958943+0000) 2022-09-10T15:52:57.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:57.587+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:57.588906+0000) 2022-09-10T15:52:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:57.958+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:57.959165+0000) 2022-09-10T15:52:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:52:58 smithi110 ceph-mon[97633]: pgmap v21875: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:52:58 smithi104 ceph-mon[97044]: pgmap v21875: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:52:58.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:58.588+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:58.589125+0000) 2022-09-10T15:52:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:58.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:58.959417+0000) 2022-09-10T15:52:59.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:52:59.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:59.589384+0000) 2022-09-10T15:53:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:52:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:52:59.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:52:59.959599+0000) 2022-09-10T15:53:00.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:00 smithi110 ceph-mon[97633]: pgmap v21876: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:00 smithi104 ceph-mon[97044]: pgmap v21876: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:00.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:00.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:00.589570+0000) 2022-09-10T15:53:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:00.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:00.959778+0000) 2022-09-10T15:53:01.961 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:01.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:01.589759+0000) 2022-09-10T15:53:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:01.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:01.960001+0000) 2022-09-10T15:53:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:02 smithi110 ceph-mon[97633]: pgmap v21877: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:02 smithi104 ceph-mon[97044]: pgmap v21877: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:02.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:02.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:02.589982+0000) 2022-09-10T15:53:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:02.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:02.960194+0000) 2022-09-10T15:53:03.962 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:03.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:03.590171+0000) 2022-09-10T15:53:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:03.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:03.960411+0000) 2022-09-10T15:53:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:04 smithi110 ceph-mon[97633]: pgmap v21878: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:04 smithi104 ceph-mon[97044]: pgmap v21878: 129 pgs: 129 active+clean; 10 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:04.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:04.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:04.590376+0000) 2022-09-10T15:53:05.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:04.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:04.960576+0000) 2022-09-10T15:53:05.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:05.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:05.590575+0000) 2022-09-10T15:53:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:05.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:05.960789+0000) 2022-09-10T15:53:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:06 smithi110 ceph-mon[97633]: pgmap v21879: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:53:06.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:06 smithi104 ceph-mon[97044]: pgmap v21879: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:53:06.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:06.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:06.590769+0000) 2022-09-10T15:53:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:06.959+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:06.961034+0000) 2022-09-10T15:53:07.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:07.589+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:07.591009+0000) 2022-09-10T15:53:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:07.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:07.961226+0000) 2022-09-10T15:53:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:08 smithi110 ceph-mon[97633]: pgmap v21880: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:53:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:08 smithi104 ceph-mon[97044]: pgmap v21880: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:53:08.963 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:08.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:08.591277+0000) 2022-09-10T15:53:08.964 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:53:08.744+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:53:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:08.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:08.961453+0000) 2022-09-10T15:53:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:09 smithi110 ceph-mon[97633]: osdmap e1448: 12 total, 12 up, 12 in 2022-09-10T15:53:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:09 smithi104 ceph-mon[97044]: osdmap e1448: 12 total, 12 up, 12 in 2022-09-10T15:53:09.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:09.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:09.591528+0000) 2022-09-10T15:53:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:09.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:09.961703+0000) 2022-09-10T15:53:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:10 smithi110 ceph-mon[97633]: pgmap v21882: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:53:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:10 smithi104 ceph-mon[97044]: pgmap v21882: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:53:10.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:10.590+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:10.591751+0000) 2022-09-10T15:53:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:10.960+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:10.961934+0000) 2022-09-10T15:53:11.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:11.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:11.592001+0000) 2022-09-10T15:53:12.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:11.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:11.962187+0000) 2022-09-10T15:53:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:12 smithi110 ceph-mon[97633]: pgmap v21883: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:53:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:12 smithi104 ceph-mon[97044]: pgmap v21883: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:53:12.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:12.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:12.592199+0000) 2022-09-10T15:53:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:12.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:12.962411+0000) 2022-09-10T15:53:13.964 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:13.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:13.592385+0000) 2022-09-10T15:53:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:14 smithi104 ceph-mon[97044]: pgmap v21884: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:53:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:14 smithi104 ceph-mon[97044]: osdmap e1449: 12 total, 12 up, 12 in 2022-09-10T15:53:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:13.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:13.962607+0000) 2022-09-10T15:53:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:14 smithi110 ceph-mon[97633]: pgmap v21884: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:53:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:14 smithi110 ceph-mon[97633]: osdmap e1449: 12 total, 12 up, 12 in 2022-09-10T15:53:14.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:14.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:14.592555+0000) 2022-09-10T15:53:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:14.961+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:14.962876+0000) 2022-09-10T15:53:15.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:15.591+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:15.592742+0000) 2022-09-10T15:53:16.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:16 smithi104 ceph-mon[97044]: pgmap v21886: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:53:16.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:15.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:15.963180+0000) 2022-09-10T15:53:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:16 smithi110 ceph-mon[97633]: pgmap v21886: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:53:16.965 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:16.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:16.592994+0000) 2022-09-10T15:53:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:16.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:16.963413+0000) 2022-09-10T15:53:17.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:17.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:17.593177+0000) 2022-09-10T15:53:18.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:18 smithi104 ceph-mon[97044]: pgmap v21887: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:53:18.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:17.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:17.963602+0000) 2022-09-10T15:53:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:18 smithi110 ceph-mon[97633]: pgmap v21887: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:53:18.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:18.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:18.593343+0000) 2022-09-10T15:53:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:18.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:18.963831+0000) 2022-09-10T15:53:19.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:19.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:19.593556+0000) 2022-09-10T15:53:20.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:20 smithi104 ceph-mon[97044]: pgmap v21888: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:53:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:19.962+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:19.964066+0000) 2022-09-10T15:53:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:20 smithi110 ceph-mon[97633]: pgmap v21888: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:53:20.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:20.592+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:20.593729+0000) 2022-09-10T15:53:21.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:20.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:20.964322+0000) 2022-09-10T15:53:21.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:21.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:21.593953+0000) 2022-09-10T15:53:22.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:22 smithi104 ceph-mon[97044]: pgmap v21889: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:53:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:21.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:21.964508+0000) 2022-09-10T15:53:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:22 smithi110 ceph-mon[97633]: pgmap v21889: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:53:22.966 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:22.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:22.594195+0000) 2022-09-10T15:53:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:22.963+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:22.964724+0000) 2022-09-10T15:53:23.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:23.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:23.594441+0000) 2022-09-10T15:53:23.968 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:53:23.757+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:53:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:24 smithi104 ceph-mon[97044]: pgmap v21890: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:53:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:23.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:23.964945+0000) 2022-09-10T15:53:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:24 smithi110 ceph-mon[97633]: pgmap v21890: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:53:24.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:24.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:24.594699+0000) 2022-09-10T15:53:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:24.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:24.965201+0000) 2022-09-10T15:53:25.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:25.593+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:25.594865+0000) 2022-09-10T15:53:26.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:26 smithi104 ceph-mon[97044]: pgmap v21891: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:53:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:25.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:25.965388+0000) 2022-09-10T15:53:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:26 smithi110 ceph-mon[97633]: pgmap v21891: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:53:26.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:26.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:26.595082+0000) 2022-09-10T15:53:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:26.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:26.965546+0000) 2022-09-10T15:53:27.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:27.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:27.595290+0000) 2022-09-10T15:53:28.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:28 smithi104 ceph-mon[97044]: pgmap v21892: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:27.964+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:27.965767+0000) 2022-09-10T15:53:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:28 smithi110 ceph-mon[97633]: pgmap v21892: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:28.967 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:28.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:28.595430+0000) 2022-09-10T15:53:29.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:28.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:28.965951+0000) 2022-09-10T15:53:29.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:29.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:29.595650+0000) 2022-09-10T15:53:30.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:30 smithi104 ceph-mon[97044]: pgmap v21893: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:29.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:29.966162+0000) 2022-09-10T15:53:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:30 smithi110 ceph-mon[97633]: pgmap v21893: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:30.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:30.594+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:30.595907+0000) 2022-09-10T15:53:31.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:30.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:30.966409+0000) 2022-09-10T15:53:31.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:31.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:31.596122+0000) 2022-09-10T15:53:32.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:32 smithi104 ceph-mon[97044]: pgmap v21894: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:32.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:31.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:31.966725+0000) 2022-09-10T15:53:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:32 smithi110 ceph-mon[97633]: pgmap v21894: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:32.968 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:32.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:32.596330+0000) 2022-09-10T15:53:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:32.965+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:32.966950+0000) 2022-09-10T15:53:33.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:33.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:33.596526+0000) 2022-09-10T15:53:34.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:34 smithi104 ceph-mon[97044]: pgmap v21895: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:33.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:33.967157+0000) 2022-09-10T15:53:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:34 smithi110 ceph-mon[97633]: pgmap v21895: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:34.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:34.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:34.596775+0000) 2022-09-10T15:53:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:34.966+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:34.967359+0000) 2022-09-10T15:53:35.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:35.595+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:35.596934+0000) 2022-09-10T15:53:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:35.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:35.967578+0000) 2022-09-10T15:53:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:36 smithi110 ceph-mon[97633]: pgmap v21896: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:36 smithi104 ceph-mon[97044]: pgmap v21896: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:36.969 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:36.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:36.597140+0000) 2022-09-10T15:53:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:36.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:36.967746+0000) 2022-09-10T15:53:37.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:37.596+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:37.597366+0000) 2022-09-10T15:53:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:37.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:37.967916+0000) 2022-09-10T15:53:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:38 smithi110 ceph-mon[97633]: pgmap v21897: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:38 smithi104 ceph-mon[97044]: pgmap v21897: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:38.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:38.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:38.597612+0000) 2022-09-10T15:53:38.971 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:53:38.771+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:53:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:38.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:38.968187+0000) 2022-09-10T15:53:39.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:39.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:39.597842+0000) 2022-09-10T15:53:40.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:39.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:39.968408+0000) 2022-09-10T15:53:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:40 smithi110 ceph-mon[97633]: pgmap v21898: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:40 smithi104 ceph-mon[97044]: pgmap v21898: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:40.970 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:40.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:40.598083+0000) 2022-09-10T15:53:41.312 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:40.967+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:40.968530+0000) 2022-09-10T15:53:41.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:41.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:41.598291+0000) 2022-09-10T15:53:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:41.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:41.968759+0000) 2022-09-10T15:53:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:42 smithi110 ceph-mon[97633]: pgmap v21899: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:42 smithi104 ceph-mon[97044]: pgmap v21899: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:42.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:42.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:42.598418+0000) 2022-09-10T15:53:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:42.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:42.969012+0000) 2022-09-10T15:53:43.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:43.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:43.598650+0000) 2022-09-10T15:53:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:43.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:43.969264+0000) 2022-09-10T15:53:44.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:44 smithi110 ceph-mon[97633]: pgmap v21900: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:44 smithi104 ceph-mon[97044]: pgmap v21900: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:44.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:44.597+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:44.598877+0000) 2022-09-10T15:53:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:44.968+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:44.969510+0000) 2022-09-10T15:53:45.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:45.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:45.599022+0000) 2022-09-10T15:53:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:45.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:45.969746+0000) 2022-09-10T15:53:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:46 smithi110 ceph-mon[97633]: pgmap v21901: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:46.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:46 smithi104 ceph-mon[97044]: pgmap v21901: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:46.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:46.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:46.599207+0000) 2022-09-10T15:53:47.317 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:46.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:46.969959+0000) 2022-09-10T15:53:47.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:47.599+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:47.599473+0000) 2022-09-10T15:53:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:47.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:47.970196+0000) 2022-09-10T15:53:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:48 smithi110 ceph-mon[97633]: pgmap v21902: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:53:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:53:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:53:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:53:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:48 smithi104 ceph-mon[97044]: pgmap v21902: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:53:48.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:53:48.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:53:48.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:53:48.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:48.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:48.599678+0000) 2022-09-10T15:53:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:48.969+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:48.970352+0000) 2022-09-10T15:53:49.972 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:49.598+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:49.599860+0000) 2022-09-10T15:53:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:49.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:49.970528+0000) 2022-09-10T15:53:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:50 smithi110 ceph-mon[97633]: pgmap v21903: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:50.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:50 smithi104 ceph-mon[97044]: pgmap v21903: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:50.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:50.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:50.600127+0000) 2022-09-10T15:53:51.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:50.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:50.970763+0000) 2022-09-10T15:53:51.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:51.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:51.600331+0000) 2022-09-10T15:53:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:51.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:51.971025+0000) 2022-09-10T15:53:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:52 smithi110 ceph-mon[97633]: pgmap v21904: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:52 smithi104 ceph-mon[97044]: pgmap v21904: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:52.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:52.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:52.600532+0000) 2022-09-10T15:53:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:52.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:52.971248+0000) 2022-09-10T15:53:53.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:53.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:53.600698+0000) 2022-09-10T15:53:53.974 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:53:53.773+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:53:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:53.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:53.971470+0000) 2022-09-10T15:53:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:54 smithi110 ceph-mon[97633]: pgmap v21905: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:54 smithi104 ceph-mon[97044]: pgmap v21905: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:54.973 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:54.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:54.600909+0000) 2022-09-10T15:53:55.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:54.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:54.971668+0000) 2022-09-10T15:53:55.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:55.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:55.601175+0000) 2022-09-10T15:53:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:55.970+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:55.971899+0000) 2022-09-10T15:53:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:56 smithi110 ceph-mon[97633]: pgmap v21906: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:56 smithi104 ceph-mon[97044]: pgmap v21906: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:53:56.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:56.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:56.601434+0000) 2022-09-10T15:53:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:56.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:56.972185+0000) 2022-09-10T15:53:57.974 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:57.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:57.601633+0000) 2022-09-10T15:53:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:57.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:57.972369+0000) 2022-09-10T15:53:58.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:53:58 smithi110 ceph-mon[97633]: pgmap v21907: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:53:58 smithi104 ceph-mon[97044]: pgmap v21907: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:53:58.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:58.600+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:58.601810+0000) 2022-09-10T15:53:59.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:58.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:58.972576+0000) 2022-09-10T15:53:59.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:53:59.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:59.602041+0000) 2022-09-10T15:54:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:53:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:53:59.971+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:53:59.972827+0000) 2022-09-10T15:54:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:00 smithi110 ceph-mon[97633]: pgmap v21908: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:00 smithi104 ceph-mon[97044]: pgmap v21908: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:00.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:00.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:00.602168+0000) 2022-09-10T15:54:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:00.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:00.973037+0000) 2022-09-10T15:54:01.976 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:01.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:01.602335+0000) 2022-09-10T15:54:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:01.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:01.973252+0000) 2022-09-10T15:54:02.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:02 smithi110 ceph-mon[97633]: pgmap v21909: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:02.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:02 smithi104 ceph-mon[97044]: pgmap v21909: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:02.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:02.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:02.602611+0000) 2022-09-10T15:54:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:02.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:02.973444+0000) 2022-09-10T15:54:03.975 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:03.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:03.602757+0000) 2022-09-10T15:54:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:03.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:03.973565+0000) 2022-09-10T15:54:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:04 smithi110 ceph-mon[97633]: pgmap v21910: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:04.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:04 smithi104 ceph-mon[97044]: pgmap v21910: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:04.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:04.601+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:04.603013+0000) 2022-09-10T15:54:05.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:04.972+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:04.973794+0000) 2022-09-10T15:54:05.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:05.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:05.603217+0000) 2022-09-10T15:54:06.314 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:05.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:05.974022+0000) 2022-09-10T15:54:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:06 smithi110 ceph-mon[97633]: pgmap v21911: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:54:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:06 smithi104 ceph-mon[97044]: pgmap v21911: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:54:06.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:06.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:06.603402+0000) 2022-09-10T15:54:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:06.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:06.974287+0000) 2022-09-10T15:54:07.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:07.602+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:07.603641+0000) 2022-09-10T15:54:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:07.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:07.974512+0000) 2022-09-10T15:54:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:08 smithi110 ceph-mon[97633]: pgmap v21912: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:54:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:08 smithi104 ceph-mon[97044]: pgmap v21912: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:54:08.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:08.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:08.603851+0000) 2022-09-10T15:54:08.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:54:08.787+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:54:09.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:08.973+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:08.974774+0000) 2022-09-10T15:54:09.420 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:09 smithi110 ceph-mon[97633]: osdmap e1450: 12 total, 12 up, 12 in 2022-09-10T15:54:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:09 smithi104 ceph-mon[97044]: osdmap e1450: 12 total, 12 up, 12 in 2022-09-10T15:54:09.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:09.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:09.604087+0000) 2022-09-10T15:54:10.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:09.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:09.974971+0000) 2022-09-10T15:54:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:10 smithi110 ceph-mon[97633]: pgmap v21914: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:54:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:10 smithi104 ceph-mon[97044]: pgmap v21914: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:54:10.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:10.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:10.604293+0000) 2022-09-10T15:54:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:10.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:10.975215+0000) 2022-09-10T15:54:11.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:11.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:11.604517+0000) 2022-09-10T15:54:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:11.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:11.975433+0000) 2022-09-10T15:54:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:12 smithi110 ceph-mon[97633]: pgmap v21915: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:54:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:12 smithi104 ceph-mon[97044]: pgmap v21915: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:54:12.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:12.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:12.604707+0000) 2022-09-10T15:54:13.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:12.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:12.975630+0000) 2022-09-10T15:54:13.977 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:13.603+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:13.604907+0000) 2022-09-10T15:54:14.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:14 smithi104 ceph-mon[97044]: pgmap v21916: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:54:14.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:14 smithi104 ceph-mon[97044]: osdmap e1451: 12 total, 12 up, 12 in 2022-09-10T15:54:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:13.974+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:13.975826+0000) 2022-09-10T15:54:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:14 smithi110 ceph-mon[97633]: pgmap v21916: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:54:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:14 smithi110 ceph-mon[97633]: osdmap e1451: 12 total, 12 up, 12 in 2022-09-10T15:54:14.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:14.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:14.605174+0000) 2022-09-10T15:54:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:14.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:14.976046+0000) 2022-09-10T15:54:15.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:15.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:15.605434+0000) 2022-09-10T15:54:16.301 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:16 smithi104 ceph-mon[97044]: pgmap v21918: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:54:16.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:15.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:15.976314+0000) 2022-09-10T15:54:16.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:16 smithi110 ceph-mon[97633]: pgmap v21918: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:54:16.978 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:16.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:16.605631+0000) 2022-09-10T15:54:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:16.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:16.976565+0000) 2022-09-10T15:54:17.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:17.604+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:17.605871+0000) 2022-09-10T15:54:18.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:18 smithi104 ceph-mon[97044]: pgmap v21919: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:54:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:17.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:17.976773+0000) 2022-09-10T15:54:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:18 smithi110 ceph-mon[97633]: pgmap v21919: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:54:18.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:18.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:18.606055+0000) 2022-09-10T15:54:19.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:18.975+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:18.977001+0000) 2022-09-10T15:54:19.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:19.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:19.606336+0000) 2022-09-10T15:54:20.993 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:20 smithi110 ceph-mon[97633]: pgmap v21920: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:54:20.993 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:20 smithi104 ceph-mon[97044]: pgmap v21920: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:54:20.994 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:19.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:19.977203+0000) 2022-09-10T15:54:20.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:20.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:20.606495+0000) 2022-09-10T15:54:21.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:20.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:20.977377+0000) 2022-09-10T15:54:21.979 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:21.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:21.606734+0000) 2022-09-10T15:54:22.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:22 smithi104 ceph-mon[97044]: pgmap v21921: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:54:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:21.976+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:21.977585+0000) 2022-09-10T15:54:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:22 smithi110 ceph-mon[97633]: pgmap v21921: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:54:22.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:22.605+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:22.607003+0000) 2022-09-10T15:54:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:22.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:22.977835+0000) 2022-09-10T15:54:23.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:23.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:23.607253+0000) 2022-09-10T15:54:23.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:54:23.802+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:54:24.298 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:24 smithi104 ceph-mon[97044]: pgmap v21922: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:54:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:23.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:23.978066+0000) 2022-09-10T15:54:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:24 smithi110 ceph-mon[97633]: pgmap v21922: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:54:24.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:24.606+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:24.607531+0000) 2022-09-10T15:54:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:24.977+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:24.978322+0000) 2022-09-10T15:54:25.980 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:25.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:25.607731+0000) 2022-09-10T15:54:26.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:25.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:25.978534+0000) 2022-09-10T15:54:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:26 smithi110 ceph-mon[97633]: pgmap v21923: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:54:26.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:26 smithi104 ceph-mon[97044]: pgmap v21923: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:54:26.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:26.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:26.607984+0000) 2022-09-10T15:54:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:26.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:26.978789+0000) 2022-09-10T15:54:27.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:27.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:27.608226+0000) 2022-09-10T15:54:28.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:27.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:27.978969+0000) 2022-09-10T15:54:28.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:28 smithi110 ceph-mon[97633]: pgmap v21924: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:28 smithi104 ceph-mon[97044]: pgmap v21924: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:28.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:28.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:28.608457+0000) 2022-09-10T15:54:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:28.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:28.979108+0000) 2022-09-10T15:54:29.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:29.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:29.608648+0000) 2022-09-10T15:54:30.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:29.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:29.979292+0000) 2022-09-10T15:54:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:30 smithi110 ceph-mon[97633]: pgmap v21925: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:30 smithi104 ceph-mon[97044]: pgmap v21925: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:30.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:30.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:30.608888+0000) 2022-09-10T15:54:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:30.978+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:30.979489+0000) 2022-09-10T15:54:31.981 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:31.607+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:31.609022+0000) 2022-09-10T15:54:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:31.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:31.979774+0000) 2022-09-10T15:54:32.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:32 smithi110 ceph-mon[97633]: pgmap v21926: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:32.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:32 smithi104 ceph-mon[97044]: pgmap v21926: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:32.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:32.608+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:32.609220+0000) 2022-09-10T15:54:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:32.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:32.980036+0000) 2022-09-10T15:54:33.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:33.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:33.609459+0000) 2022-09-10T15:54:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:33.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:33.980265+0000) 2022-09-10T15:54:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:34 smithi110 ceph-mon[97633]: pgmap v21927: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:34 smithi104 ceph-mon[97044]: pgmap v21927: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:34.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:34.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:34.609651+0000) 2022-09-10T15:54:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:34.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:34.980460+0000) 2022-09-10T15:54:35.982 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:35.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:35.609788+0000) 2022-09-10T15:54:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:35.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:35.980653+0000) 2022-09-10T15:54:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:36 smithi110 ceph-mon[97633]: pgmap v21928: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:36 smithi104 ceph-mon[97044]: pgmap v21928: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:36.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:36.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:36.610042+0000) 2022-09-10T15:54:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:36.979+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:36.980834+0000) 2022-09-10T15:54:37.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:37.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:37.610311+0000) 2022-09-10T15:54:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:37.980+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:37.981073+0000) 2022-09-10T15:54:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:38 smithi110 ceph-mon[97633]: pgmap v21929: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:38 smithi104 ceph-mon[97044]: pgmap v21929: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:38.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:38.609+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:38.610484+0000) 2022-09-10T15:54:38.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:54:38.813+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:54:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:38.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:38.981306+0000) 2022-09-10T15:54:39.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:39.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:39.610748+0000) 2022-09-10T15:54:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:39.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:39.981494+0000) 2022-09-10T15:54:40.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:40 smithi110 ceph-mon[97633]: pgmap v21930: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:40 smithi104 ceph-mon[97044]: pgmap v21930: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:40.983 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:40.610+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:40.610976+0000) 2022-09-10T15:54:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:40.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:40.981664+0000) 2022-09-10T15:54:41.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:41.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:41.611214+0000) 2022-09-10T15:54:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:41.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:41.981929+0000) 2022-09-10T15:54:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:42 smithi110 ceph-mon[97633]: pgmap v21931: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:42 smithi104 ceph-mon[97044]: pgmap v21931: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:42.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:42.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:42.611452+0000) 2022-09-10T15:54:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:42.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:42.982200+0000) 2022-09-10T15:54:43.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:43.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:43.611701+0000) 2022-09-10T15:54:44.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:43.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:43.982404+0000) 2022-09-10T15:54:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:44 smithi110 ceph-mon[97633]: pgmap v21932: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:44 smithi104 ceph-mon[97044]: pgmap v21932: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:44.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:44.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:44.611957+0000) 2022-09-10T15:54:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:44.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:44.982649+0000) 2022-09-10T15:54:45.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:45.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:45.612238+0000) 2022-09-10T15:54:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:45.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:45.982830+0000) 2022-09-10T15:54:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:46 smithi110 ceph-mon[97633]: pgmap v21933: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:46.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:46 smithi104 ceph-mon[97044]: pgmap v21933: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:46.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:46.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:46.612458+0000) 2022-09-10T15:54:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:46.981+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:46.983033+0000) 2022-09-10T15:54:47.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:47.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:47.612750+0000) 2022-09-10T15:54:48.320 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:47.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:47.983220+0000) 2022-09-10T15:54:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:48 smithi110 ceph-mon[97633]: pgmap v21934: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:54:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:54:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:54:48.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:54:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:48 smithi104 ceph-mon[97044]: pgmap v21934: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:54:48.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:54:48.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:54:48.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:54:48.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:48.611+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:48.613011+0000) 2022-09-10T15:54:49.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:48.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:48.983448+0000) 2022-09-10T15:54:49.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:49.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:49.613280+0000) 2022-09-10T15:54:50.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:49.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:49.983595+0000) 2022-09-10T15:54:50.413 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:50 smithi110 ceph-mon[97633]: pgmap v21935: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:50.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:50 smithi104 ceph-mon[97044]: pgmap v21935: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:50.985 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:50.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:50.613536+0000) 2022-09-10T15:54:51.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:50.982+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:50.983819+0000) 2022-09-10T15:54:51.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:51.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:51.613755+0000) 2022-09-10T15:54:52.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:51.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:51.984083+0000) 2022-09-10T15:54:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:52 smithi110 ceph-mon[97633]: pgmap v21936: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:52 smithi104 ceph-mon[97044]: pgmap v21936: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:52.986 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:52.612+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:52.613995+0000) 2022-09-10T15:54:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:52.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:52.984364+0000) 2022-09-10T15:54:53.986 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:54:53.827+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:54:53.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:53.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:53.614259+0000) 2022-09-10T15:54:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:53.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:53.984571+0000) 2022-09-10T15:54:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:54 smithi110 ceph-mon[97633]: pgmap v21937: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:54.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:54 smithi104 ceph-mon[97044]: pgmap v21937: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:54.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:54.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:54.614523+0000) 2022-09-10T15:54:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:54.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:54.984797+0000) 2022-09-10T15:54:55.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:55.613+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:55.614798+0000) 2022-09-10T15:54:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:55.983+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:55.984936+0000) 2022-09-10T15:54:56.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:56 smithi110 ceph-mon[97633]: pgmap v21938: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:56 smithi104 ceph-mon[97044]: pgmap v21938: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:54:56.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:56.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:56.614997+0000) 2022-09-10T15:54:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:56.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:56.985141+0000) 2022-09-10T15:54:57.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:57.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:57.615274+0000) 2022-09-10T15:54:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:57.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:57.985354+0000) 2022-09-10T15:54:58.414 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:54:58 smithi110 ceph-mon[97633]: pgmap v21939: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:58.551 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:54:58 smithi104 ceph-mon[97044]: pgmap v21939: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:54:58.987 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:58.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:58.615431+0000) 2022-09-10T15:54:59.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:58.984+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:58.985613+0000) 2022-09-10T15:54:59.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:54:59.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:59.615675+0000) 2022-09-10T15:55:00.308 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:54:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:54:59.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:54:59.985846+0000) 2022-09-10T15:55:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:00 smithi110 ceph-mon[97633]: pgmap v21940: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:00 smithi104 ceph-mon[97044]: pgmap v21940: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:00.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:00.614+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:00.615835+0000) 2022-09-10T15:55:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:00.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:00.986056+0000) 2022-09-10T15:55:01.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:01.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:01.616072+0000) 2022-09-10T15:55:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:01.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:01.986268+0000) 2022-09-10T15:55:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:02 smithi110 ceph-mon[97633]: pgmap v21941: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:02 smithi104 ceph-mon[97044]: pgmap v21941: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:02.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:02.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:02.616339+0000) 2022-09-10T15:55:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:02.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:02.986499+0000) 2022-09-10T15:55:03.988 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:03.615+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:03.616530+0000) 2022-09-10T15:55:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:03.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:03.986773+0000) 2022-09-10T15:55:04.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:04 smithi110 ceph-mon[97633]: pgmap v21942: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:04 smithi104 ceph-mon[97044]: pgmap v21942: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:04.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:04.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:04.616757+0000) 2022-09-10T15:55:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:04.985+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:04.986985+0000) 2022-09-10T15:55:05.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:05.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:05.617025+0000) 2022-09-10T15:55:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:05.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:05.987162+0000) 2022-09-10T15:55:06.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:06 smithi110 ceph-mon[97633]: pgmap v21943: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:55:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:06 smithi104 ceph-mon[97044]: pgmap v21943: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:55:06.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:06.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:06.617214+0000) 2022-09-10T15:55:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:06.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:06.987302+0000) 2022-09-10T15:55:07.989 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:07.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:07.617464+0000) 2022-09-10T15:55:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:07.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:07.987481+0000) 2022-09-10T15:55:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:08 smithi110 ceph-mon[97633]: pgmap v21944: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:55:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:08 smithi104 ceph-mon[97044]: pgmap v21944: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:55:08.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:08.616+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:08.617667+0000) 2022-09-10T15:55:08.991 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:55:08.837+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:55:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:08.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:08.987694+0000) 2022-09-10T15:55:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:09 smithi110 ceph-mon[97633]: osdmap e1452: 12 total, 12 up, 12 in 2022-09-10T15:55:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:09 smithi104 ceph-mon[97044]: osdmap e1452: 12 total, 12 up, 12 in 2022-09-10T15:55:09.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:09.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:09.617936+0000) 2022-09-10T15:55:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:09.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:09.987870+0000) 2022-09-10T15:55:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:10 smithi110 ceph-mon[97633]: pgmap v21946: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:55:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:10 smithi104 ceph-mon[97044]: pgmap v21946: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:55:10.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:10.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:10.618219+0000) 2022-09-10T15:55:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:10.986+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:10.988042+0000) 2022-09-10T15:55:11.990 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:11.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:11.618412+0000) 2022-09-10T15:55:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:11.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:11.988339+0000) 2022-09-10T15:55:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:12 smithi110 ceph-mon[97633]: pgmap v21947: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:55:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:12 smithi104 ceph-mon[97044]: pgmap v21947: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:55:12.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:12.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:12.618643+0000) 2022-09-10T15:55:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:12.987+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:12.988526+0000) 2022-09-10T15:55:14.096 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:13.617+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:13.618838+0000) 2022-09-10T15:55:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:14 smithi104 ceph-mon[97044]: pgmap v21948: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:55:14.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:14 smithi104 ceph-mon[97044]: osdmap e1453: 12 total, 12 up, 12 in 2022-09-10T15:55:14.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:13.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:13.988780+0000) 2022-09-10T15:55:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:14 smithi110 ceph-mon[97633]: pgmap v21948: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:55:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:14 smithi110 ceph-mon[97633]: osdmap e1453: 12 total, 12 up, 12 in 2022-09-10T15:55:14.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:14.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:14.619061+0000) 2022-09-10T15:55:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:14.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:14.988981+0000) 2022-09-10T15:55:15.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:15.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:15.619259+0000) 2022-09-10T15:55:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:15.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:15.989231+0000) 2022-09-10T15:55:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:16 smithi110 ceph-mon[97633]: pgmap v21950: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:55:16.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:16 smithi104 ceph-mon[97044]: pgmap v21950: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:55:16.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:16.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:16.619497+0000) 2022-09-10T15:55:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:16.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:16.989439+0000) 2022-09-10T15:55:17.991 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:17.618+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:17.619738+0000) 2022-09-10T15:55:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:17.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:17.989690+0000) 2022-09-10T15:55:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:18 smithi110 ceph-mon[97633]: pgmap v21951: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:55:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:18 smithi104 ceph-mon[97044]: pgmap v21951: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:55:18.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:18.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:18.620008+0000) 2022-09-10T15:55:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:18.988+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:18.989971+0000) 2022-09-10T15:55:19.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:19.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:19.620292+0000) 2022-09-10T15:55:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:19.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:19.990130+0000) 2022-09-10T15:55:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:20 smithi110 ceph-mon[97633]: pgmap v21952: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:55:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:20 smithi104 ceph-mon[97044]: pgmap v21952: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:55:20.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:20.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:20.620491+0000) 2022-09-10T15:55:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:20.989+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:20.990277+0000) 2022-09-10T15:55:21.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:21.619+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:21.620773+0000) 2022-09-10T15:55:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:21.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:21.990485+0000) 2022-09-10T15:55:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:22 smithi110 ceph-mon[97633]: pgmap v21953: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:55:22.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:22 smithi104 ceph-mon[97044]: pgmap v21953: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:55:22.992 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:22.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:22.620982+0000) 2022-09-10T15:55:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:22.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:22.990724+0000) 2022-09-10T15:55:23.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:23.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:23.621214+0000) 2022-09-10T15:55:23.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:55:23.848+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:55:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:23.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:23.990997+0000) 2022-09-10T15:55:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:24 smithi110 ceph-mon[97633]: pgmap v21954: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:55:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:24 smithi104 ceph-mon[97044]: pgmap v21954: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:55:24.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:24.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:24.621453+0000) 2022-09-10T15:55:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:24.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:24.991229+0000) 2022-09-10T15:55:25.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:25.620+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:25.621640+0000) 2022-09-10T15:55:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:25.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:25.991362+0000) 2022-09-10T15:55:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:26 smithi110 ceph-mon[97633]: pgmap v21955: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:55:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:26 smithi104 ceph-mon[97044]: pgmap v21955: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:55:26.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:26.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:26.621899+0000) 2022-09-10T15:55:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:26.990+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:26.991567+0000) 2022-09-10T15:55:27.993 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:27.621+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:27.622157+0000) 2022-09-10T15:55:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:27.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:27.991820+0000) 2022-09-10T15:55:28.418 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:28 smithi110 ceph-mon[97633]: pgmap v21956: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:28.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:28 smithi104 ceph-mon[97044]: pgmap v21956: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:28.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:28.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:28.622345+0000) 2022-09-10T15:55:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:28.991+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:28.992078+0000) 2022-09-10T15:55:29.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:29.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:29.622578+0000) 2022-09-10T15:55:30.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:29.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:29.992302+0000) 2022-09-10T15:55:30.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:30 smithi110 ceph-mon[97633]: pgmap v21957: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:30 smithi104 ceph-mon[97044]: pgmap v21957: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:30.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:30.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:30.622797+0000) 2022-09-10T15:55:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:30.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:30.992530+0000) 2022-09-10T15:55:31.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:31.622+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:31.623015+0000) 2022-09-10T15:55:32.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:31.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:31.992806+0000) 2022-09-10T15:55:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:32 smithi110 ceph-mon[97633]: pgmap v21958: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:32.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:32 smithi104 ceph-mon[97044]: pgmap v21958: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:32.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:32.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:32.623261+0000) 2022-09-10T15:55:33.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:32.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:32.993047+0000) 2022-09-10T15:55:33.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:33.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:33.623543+0000) 2022-09-10T15:55:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:33.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:33.993329+0000) 2022-09-10T15:55:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:34 smithi110 ceph-mon[97633]: pgmap v21959: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:34 smithi104 ceph-mon[97044]: pgmap v21959: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:34.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:34.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:34.623794+0000) 2022-09-10T15:55:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:34.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:34.993504+0000) 2022-09-10T15:55:35.995 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:35.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:35.623949+0000) 2022-09-10T15:55:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:35.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:35.993715+0000) 2022-09-10T15:55:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:36 smithi110 ceph-mon[97633]: pgmap v21960: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:36 smithi104 ceph-mon[97044]: pgmap v21960: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:36.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:36.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:36.624158+0000) 2022-09-10T15:55:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:36.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:36.993926+0000) 2022-09-10T15:55:37.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:37.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:37.624381+0000) 2022-09-10T15:55:38.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:37.992+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:37.994084+0000) 2022-09-10T15:55:38.412 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:38 smithi110 ceph-mon[97633]: pgmap v21961: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:38 smithi104 ceph-mon[97044]: pgmap v21961: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:38.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:55:38.854+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:55:38.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:38.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:38.624571+0000) 2022-09-10T15:55:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:38.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:38.994295+0000) 2022-09-10T15:55:39.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:39.623+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:39.624866+0000) 2022-09-10T15:55:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:39.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:39.994517+0000) 2022-09-10T15:55:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:40 smithi110 ceph-mon[97633]: pgmap v21962: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:40 smithi104 ceph-mon[97044]: pgmap v21962: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:40.996 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:40.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:40.625126+0000) 2022-09-10T15:55:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:40.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:40.994709+0000) 2022-09-10T15:55:41.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:41.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:41.625340+0000) 2022-09-10T15:55:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:41.993+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:41.994978+0000) 2022-09-10T15:55:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:42 smithi110 ceph-mon[97633]: pgmap v21963: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:42.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:42 smithi104 ceph-mon[97044]: pgmap v21963: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:42.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:42.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:42.625560+0000) 2022-09-10T15:55:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:42.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:42.995241+0000) 2022-09-10T15:55:43.997 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:43.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:43.625734+0000) 2022-09-10T15:55:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:43.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:43.995472+0000) 2022-09-10T15:55:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:44 smithi110 ceph-mon[97633]: pgmap v21964: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:44 smithi104 ceph-mon[97044]: pgmap v21964: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:44.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:44.624+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:44.625880+0000) 2022-09-10T15:55:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:44.994+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:44.995669+0000) 2022-09-10T15:55:45.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:45.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:45.626090+0000) 2022-09-10T15:55:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:45.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:45.995920+0000) 2022-09-10T15:55:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:46 smithi110 ceph-mon[97633]: pgmap v21965: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:46 smithi104 ceph-mon[97044]: pgmap v21965: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:46.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:46.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:46.626346+0000) 2022-09-10T15:55:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:46.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:46.996156+0000) 2022-09-10T15:55:47.998 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:47.625+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:47.626640+0000) 2022-09-10T15:55:48.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:47.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:47.996373+0000) 2022-09-10T15:55:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:48 smithi110 ceph-mon[97633]: pgmap v21966: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:55:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:55:48.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:55:48.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:55:48.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:55:48.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:48 smithi104 ceph-mon[97044]: pgmap v21966: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:55:48.554 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:55:48.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:55:48.555 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:55:48.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:55:48.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:48.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:48.626912+0000) 2022-09-10T15:55:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:48.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:48.996564+0000) 2022-09-10T15:55:49.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:49.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:49.627185+0000) 2022-09-10T15:55:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:49.995+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:49.996784+0000) 2022-09-10T15:55:50.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:50 smithi110 ceph-mon[97633]: pgmap v21967: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:55:50.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:50 smithi104 ceph-mon[97044]: pgmap v21967: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:55:50.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:50.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:50.627442+0000) 2022-09-10T15:55:51.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:50.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:50.996994+0000) 2022-09-10T15:55:52.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:51.626+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:51.627689+0000) 2022-09-10T15:55:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:51.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:51.997291+0000) 2022-09-10T15:55:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:52 smithi110 ceph-mon[97633]: pgmap v21968: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:52.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:52 smithi104 ceph-mon[97044]: pgmap v21968: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:52.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:52.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:52.627928+0000) 2022-09-10T15:55:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:52.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:52.997428+0000) 2022-09-10T15:55:53.999 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:53.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:53.628229+0000) 2022-09-10T15:55:54.000 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:55:53.860+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:55:54.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:53.996+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:53.997593+0000) 2022-09-10T15:55:54.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:54 smithi110 ceph-mon[97633]: pgmap v21969: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:55:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:54 smithi104 ceph-mon[97044]: pgmap v21969: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:55:55.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:54.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:54.628451+0000) 2022-09-10T15:55:55.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:54.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:54.997832+0000) 2022-09-10T15:55:56.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:55.627+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:55.628684+0000) 2022-09-10T15:55:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:55.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:55.998120+0000) 2022-09-10T15:55:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:56 smithi110 ceph-mon[97633]: pgmap v21970: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:56 smithi104 ceph-mon[97044]: pgmap v21970: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:55:57.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:56.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:56.628953+0000) 2022-09-10T15:55:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:56.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:56.998315+0000) 2022-09-10T15:55:58.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:57.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:57.629165+0000) 2022-09-10T15:55:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:57.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:57.998533+0000) 2022-09-10T15:55:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:55:58 smithi110 ceph-mon[97633]: pgmap v21971: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:55:58 smithi104 ceph-mon[97044]: pgmap v21971: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:55:59.000 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:58.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:58.629415+0000) 2022-09-10T15:55:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:58.997+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:58.998729+0000) 2022-09-10T15:56:00.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:55:59.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:59.629683+0000) 2022-09-10T15:56:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:55:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:55:59.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:55:59.998903+0000) 2022-09-10T15:56:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:00 smithi110 ceph-mon[97633]: pgmap v21972: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:00 smithi104 ceph-mon[97044]: pgmap v21972: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:01.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:00.628+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:00.629947+0000) 2022-09-10T15:56:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:00.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:00.999112+0000) 2022-09-10T15:56:02.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:01.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:01.630220+0000) 2022-09-10T15:56:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:01.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:01.999361+0000) 2022-09-10T15:56:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:02 smithi110 ceph-mon[97633]: pgmap v21973: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:02 smithi104 ceph-mon[97044]: pgmap v21973: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:03.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:02.629+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:02.630492+0000) 2022-09-10T15:56:03.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:02.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:02.999603+0000) 2022-09-10T15:56:04.001 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:03.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:03.630747+0000) 2022-09-10T15:56:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:03.998+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:03.999871+0000) 2022-09-10T15:56:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:04 smithi110 ceph-mon[97633]: pgmap v21974: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:04 smithi104 ceph-mon[97044]: pgmap v21974: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:05.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:04.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:04.630994+0000) 2022-09-10T15:56:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:04.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:05.000173+0000) 2022-09-10T15:56:06.002 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:05.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:05.631193+0000) 2022-09-10T15:56:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:05.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:06.000344+0000) 2022-09-10T15:56:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:06 smithi110 ceph-mon[97633]: pgmap v21975: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:56:06.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:06 smithi104 ceph-mon[97044]: pgmap v21975: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:56:07.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:06.630+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:06.631432+0000) 2022-09-10T15:56:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:07.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:07.000602+0000) 2022-09-10T15:56:08.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:07.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:07.631679+0000) 2022-09-10T15:56:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:07.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:08.000787+0000) 2022-09-10T15:56:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:08 smithi110 ceph-mon[97633]: pgmap v21976: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:56:08.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:08 smithi104 ceph-mon[97044]: pgmap v21976: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:56:09.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:08.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:08.631813+0000) 2022-09-10T15:56:09.003 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:56:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:56:08.875+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:56:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:08.999+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:09.000957+0000) 2022-09-10T15:56:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:09 smithi110 ceph-mon[97633]: osdmap e1454: 12 total, 12 up, 12 in 2022-09-10T15:56:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:09 smithi104 ceph-mon[97044]: osdmap e1454: 12 total, 12 up, 12 in 2022-09-10T15:56:10.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:09.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:09.632065+0000) 2022-09-10T15:56:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:10.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:10.001213+0000) 2022-09-10T15:56:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:10 smithi110 ceph-mon[97633]: pgmap v21978: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:56:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:10 smithi104 ceph-mon[97044]: pgmap v21978: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:56:11.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:10.631+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:10.632321+0000) 2022-09-10T15:56:11.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:11.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:11.001376+0000) 2022-09-10T15:56:12.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:11.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:11.632524+0000) 2022-09-10T15:56:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:12.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:12.001566+0000) 2022-09-10T15:56:12.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:12 smithi110 ceph-mon[97633]: pgmap v21979: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:56:12.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:12 smithi104 ceph-mon[97044]: pgmap v21979: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:56:13.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:12.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:12.632771+0000) 2022-09-10T15:56:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:13.000+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:13.001745+0000) 2022-09-10T15:56:14.003 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:13.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:13.632986+0000) 2022-09-10T15:56:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:14.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:14.002015+0000) 2022-09-10T15:56:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:14 smithi110 ceph-mon[97633]: pgmap v21980: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:56:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:14 smithi110 ceph-mon[97633]: osdmap e1455: 12 total, 12 up, 12 in 2022-09-10T15:56:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:14 smithi104 ceph-mon[97044]: pgmap v21980: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 28 op/s 2022-09-10T15:56:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:14 smithi104 ceph-mon[97044]: osdmap e1455: 12 total, 12 up, 12 in 2022-09-10T15:56:15.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:14.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:14.633209+0000) 2022-09-10T15:56:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:15.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:15.002265+0000) 2022-09-10T15:56:16.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:15.632+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:15.633420+0000) 2022-09-10T15:56:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:16.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:16.002444+0000) 2022-09-10T15:56:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:16 smithi110 ceph-mon[97633]: pgmap v21982: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:56:16.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:16 smithi104 ceph-mon[97044]: pgmap v21982: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:56:17.004 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:16.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:16.633633+0000) 2022-09-10T15:56:17.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:17.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:17.002689+0000) 2022-09-10T15:56:18.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:17.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:17.633897+0000) 2022-09-10T15:56:18.309 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:18.001+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:18.002948+0000) 2022-09-10T15:56:18.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:18 smithi110 ceph-mon[97633]: pgmap v21983: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T15:56:18.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:18 smithi104 ceph-mon[97044]: pgmap v21983: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T15:56:19.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:18.633+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:18.634171+0000) 2022-09-10T15:56:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:19.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:19.003171+0000) 2022-09-10T15:56:20.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:19.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:19.634418+0000) 2022-09-10T15:56:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:20.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:20.003339+0000) 2022-09-10T15:56:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:20 smithi110 ceph-mon[97633]: pgmap v21984: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:56:20.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:20 smithi104 ceph-mon[97044]: pgmap v21984: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:56:21.005 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:20.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:20.634672+0000) 2022-09-10T15:56:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:21.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:21.003582+0000) 2022-09-10T15:56:22.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:21.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:21.634946+0000) 2022-09-10T15:56:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:22.002+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:22.003811+0000) 2022-09-10T15:56:22.410 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:22 smithi110 ceph-mon[97633]: pgmap v21985: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:56:22.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:22 smithi104 ceph-mon[97044]: pgmap v21985: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:56:23.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:22.634+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:22.635181+0000) 2022-09-10T15:56:23.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:23.003+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:23.004063+0000) 2022-09-10T15:56:23.893 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:23.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:23.635408+0000) 2022-09-10T15:56:24.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:56:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:56:23.890+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:56:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:24.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:24.004348+0000) 2022-09-10T15:56:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:24 smithi110 ceph-mon[97633]: pgmap v21986: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:56:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:24 smithi104 ceph-mon[97044]: pgmap v21986: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:56:25.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:24.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:24.635606+0000) 2022-09-10T15:56:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:25.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:25.004563+0000) 2022-09-10T15:56:26.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:25.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:25.635770+0000) 2022-09-10T15:56:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:26.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:26.004794+0000) 2022-09-10T15:56:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:26 smithi110 ceph-mon[97633]: pgmap v21987: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:56:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:26 smithi104 ceph-mon[97044]: pgmap v21987: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:56:27.006 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:26.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:26.635958+0000) 2022-09-10T15:56:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:27.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:27.004999+0000) 2022-09-10T15:56:28.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:27.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:27.636191+0000) 2022-09-10T15:56:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:28.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:28.005242+0000) 2022-09-10T15:56:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:28 smithi110 ceph-mon[97633]: pgmap v21988: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:56:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:28 smithi104 ceph-mon[97044]: pgmap v21988: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:56:29.007 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:28.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:28.636383+0000) 2022-09-10T15:56:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:29.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:29.005442+0000) 2022-09-10T15:56:30.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:29.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:29.636606+0000) 2022-09-10T15:56:30.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:30.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:30.005720+0000) 2022-09-10T15:56:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:30 smithi110 ceph-mon[97633]: pgmap v21989: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:56:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:30 smithi104 ceph-mon[97044]: pgmap v21989: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:56:31.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:30.635+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:30.636847+0000) 2022-09-10T15:56:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:31.004+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:31.005946+0000) 2022-09-10T15:56:32.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:31.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:31.637139+0000) 2022-09-10T15:56:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:32.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:32.006221+0000) 2022-09-10T15:56:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:32 smithi110 ceph-mon[97633]: pgmap v21990: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:32 smithi104 ceph-mon[97044]: pgmap v21990: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:33.091 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:32.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:32.637361+0000) 2022-09-10T15:56:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:33.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:33.006383+0000) 2022-09-10T15:56:34.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:33.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:33.637631+0000) 2022-09-10T15:56:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:34.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:34.006507+0000) 2022-09-10T15:56:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:34 smithi110 ceph-mon[97633]: pgmap v21991: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:56:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:34 smithi104 ceph-mon[97044]: pgmap v21991: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2022-09-10T15:56:35.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:34.636+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:34.637863+0000) 2022-09-10T15:56:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:35.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:35.006765+0000) 2022-09-10T15:56:36.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:35.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:35.638070+0000) 2022-09-10T15:56:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:36.005+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:36.006995+0000) 2022-09-10T15:56:36.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:36 smithi110 ceph-mon[97633]: pgmap v21992: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:36 smithi104 ceph-mon[97044]: pgmap v21992: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:37.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:36.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:36.638342+0000) 2022-09-10T15:56:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:37.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:37.007174+0000) 2022-09-10T15:56:38.009 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:37.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:37.638589+0000) 2022-09-10T15:56:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:38.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:38.007389+0000) 2022-09-10T15:56:38.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:38 smithi110 ceph-mon[97633]: pgmap v21993: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:38.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:38 smithi104 ceph-mon[97044]: pgmap v21993: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:38.905 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:38.637+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:38.638817+0000) 2022-09-10T15:56:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:39.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:39.007587+0000) 2022-09-10T15:56:39.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:56:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:56:38.902+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:56:40.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:39.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:39.639148+0000) 2022-09-10T15:56:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:40.006+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:40.007835+0000) 2022-09-10T15:56:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:40 smithi110 ceph-mon[97633]: pgmap v21994: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:40 smithi104 ceph-mon[97044]: pgmap v21994: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:41.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:40.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:40.639409+0000) 2022-09-10T15:56:41.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:41.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:41.008131+0000) 2022-09-10T15:56:42.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:41.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:41.639624+0000) 2022-09-10T15:56:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:42.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:42.008342+0000) 2022-09-10T15:56:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:42 smithi110 ceph-mon[97633]: pgmap v21995: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:42.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:42 smithi104 ceph-mon[97044]: pgmap v21995: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:43.010 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:42.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:42.639821+0000) 2022-09-10T15:56:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:43.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:43.008591+0000) 2022-09-10T15:56:44.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:43.638+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:43.640063+0000) 2022-09-10T15:56:44.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:44.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:44.008850+0000) 2022-09-10T15:56:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:44 smithi110 ceph-mon[97633]: pgmap v21996: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:44 smithi104 ceph-mon[97044]: pgmap v21996: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:45.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:44.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:44.640201+0000) 2022-09-10T15:56:45.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:45.007+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:45.009074+0000) 2022-09-10T15:56:46.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:45.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:45.640410+0000) 2022-09-10T15:56:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:46.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:46.009276+0000) 2022-09-10T15:56:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:46 smithi110 ceph-mon[97633]: pgmap v21997: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:46 smithi104 ceph-mon[97044]: pgmap v21997: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:47.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:46.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:46.640584+0000) 2022-09-10T15:56:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:47.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:47.009525+0000) 2022-09-10T15:56:48.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:47.639+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:47.640826+0000) 2022-09-10T15:56:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:48.008+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:48.009719+0000) 2022-09-10T15:56:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:48 smithi110 ceph-mon[97633]: pgmap v21998: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:56:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:56:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:56:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:56:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:48 smithi104 ceph-mon[97044]: pgmap v21998: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:56:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:56:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:56:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:56:49.011 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:48.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:48.640997+0000) 2022-09-10T15:56:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:49.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:49.009914+0000) 2022-09-10T15:56:50.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:49.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:49.641261+0000) 2022-09-10T15:56:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:50.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:50.010171+0000) 2022-09-10T15:56:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:50 smithi110 ceph-mon[97633]: pgmap v21999: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:50 smithi104 ceph-mon[97044]: pgmap v21999: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:51.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:50.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:50.641496+0000) 2022-09-10T15:56:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:51.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:51.010383+0000) 2022-09-10T15:56:52.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:51.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:51.641729+0000) 2022-09-10T15:56:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:52.009+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:52.010588+0000) 2022-09-10T15:56:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:52 smithi110 ceph-mon[97633]: pgmap v22000: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:52 smithi104 ceph-mon[97044]: pgmap v22000: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:53.012 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:52.640+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:52.641924+0000) 2022-09-10T15:56:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:53.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:53.010818+0000) 2022-09-10T15:56:53.918 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:53.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:53.642085+0000) 2022-09-10T15:56:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:56:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:56:53.915+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:56:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:54.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:54.011039+0000) 2022-09-10T15:56:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:54 smithi110 ceph-mon[97633]: pgmap v22001: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:54 smithi104 ceph-mon[97044]: pgmap v22001: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:55.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:54.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:54.642301+0000) 2022-09-10T15:56:55.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:55.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:55.011228+0000) 2022-09-10T15:56:56.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:55.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:55.642480+0000) 2022-09-10T15:56:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:56.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:56.011494+0000) 2022-09-10T15:56:56.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:56 smithi110 ceph-mon[97633]: pgmap v22002: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:56 smithi104 ceph-mon[97044]: pgmap v22002: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:56:57.013 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:56.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:56.642701+0000) 2022-09-10T15:56:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:57.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:57.011725+0000) 2022-09-10T15:56:58.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:57.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:57.642897+0000) 2022-09-10T15:56:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:58.010+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:58.011966+0000) 2022-09-10T15:56:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:56:58 smithi110 ceph-mon[97633]: pgmap v22003: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:58.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:56:58 smithi104 ceph-mon[97044]: pgmap v22003: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:56:59.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:58.641+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:58.643065+0000) 2022-09-10T15:56:59.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:56:59.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:59.012190+0000) 2022-09-10T15:57:00.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:56:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:56:59.642+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:56:59.643336+0000) 2022-09-10T15:57:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:00.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:00.012384+0000) 2022-09-10T15:57:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:00 smithi110 ceph-mon[97633]: pgmap v22004: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:00 smithi104 ceph-mon[97044]: pgmap v22004: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:01.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:00.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:00.643588+0000) 2022-09-10T15:57:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:01.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:01.012563+0000) 2022-09-10T15:57:02.014 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:01.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:01.643832+0000) 2022-09-10T15:57:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:02.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:02.012752+0000) 2022-09-10T15:57:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:02 smithi110 ceph-mon[97633]: pgmap v22005: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:02 smithi104 ceph-mon[97044]: pgmap v22005: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:03.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:02.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:02.644130+0000) 2022-09-10T15:57:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:03.011+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:03.012981+0000) 2022-09-10T15:57:04.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:03.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:03.644371+0000) 2022-09-10T15:57:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:04.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:04.013206+0000) 2022-09-10T15:57:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:04 smithi110 ceph-mon[97633]: pgmap v22006: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:04 smithi104 ceph-mon[97044]: pgmap v22006: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:05.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:04.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:04.644567+0000) 2022-09-10T15:57:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:05.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:05.013464+0000) 2022-09-10T15:57:06.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:05.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:05.644762+0000) 2022-09-10T15:57:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:06.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:06.013691+0000) 2022-09-10T15:57:06.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:06 smithi110 ceph-mon[97633]: pgmap v22007: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:57:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:06 smithi104 ceph-mon[97044]: pgmap v22007: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:57:07.015 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:06.643+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:06.644944+0000) 2022-09-10T15:57:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:07.012+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:07.013937+0000) 2022-09-10T15:57:08.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:07.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:07.645235+0000) 2022-09-10T15:57:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:08.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:08.014208+0000) 2022-09-10T15:57:08.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:08 smithi110 ceph-mon[97633]: pgmap v22008: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:57:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:08 smithi104 ceph-mon[97044]: pgmap v22008: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:57:08.920 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:08.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:08.645441+0000) 2022-09-10T15:57:09.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:57:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:57:08.917+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:57:09.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:09.013+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:09.014350+0000) 2022-09-10T15:57:09.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:09 smithi110 ceph-mon[97633]: osdmap e1456: 12 total, 12 up, 12 in 2022-09-10T15:57:09.550 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:09 smithi104 ceph-mon[97044]: osdmap e1456: 12 total, 12 up, 12 in 2022-09-10T15:57:10.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:09.645+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:09.645615+0000) 2022-09-10T15:57:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:10.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:10.014554+0000) 2022-09-10T15:57:10.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:10 smithi110 ceph-mon[97633]: pgmap v22010: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:57:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:10 smithi104 ceph-mon[97044]: pgmap v22010: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:57:11.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:10.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:10.645861+0000) 2022-09-10T15:57:11.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:11.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:11.014754+0000) 2022-09-10T15:57:12.016 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:11.644+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:11.646049+0000) 2022-09-10T15:57:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:12.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:12.014969+0000) 2022-09-10T15:57:12.427 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:12 smithi110 ceph-mon[97633]: pgmap v22011: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:57:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:12 smithi104 ceph-mon[97044]: pgmap v22011: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:57:13.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:12.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:12.646280+0000) 2022-09-10T15:57:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:13.014+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:13.015210+0000) 2022-09-10T15:57:14.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:13.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:13.646495+0000) 2022-09-10T15:57:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:14.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:14.015446+0000) 2022-09-10T15:57:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:14 smithi110 ceph-mon[97633]: pgmap v22012: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:57:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:14 smithi110 ceph-mon[97633]: osdmap e1457: 12 total, 12 up, 12 in 2022-09-10T15:57:14.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:14 smithi104 ceph-mon[97044]: pgmap v22012: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:57:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:14 smithi104 ceph-mon[97044]: osdmap e1457: 12 total, 12 up, 12 in 2022-09-10T15:57:15.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:14.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:14.646697+0000) 2022-09-10T15:57:15.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:15.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:15.015669+0000) 2022-09-10T15:57:16.017 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:15.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:15.646915+0000) 2022-09-10T15:57:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:16.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:16.015946+0000) 2022-09-10T15:57:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:16 smithi110 ceph-mon[97633]: pgmap v22014: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:57:16.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:16 smithi104 ceph-mon[97044]: pgmap v22014: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:57:17.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:16.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:16.647190+0000) 2022-09-10T15:57:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:17.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:17.016208+0000) 2022-09-10T15:57:18.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:17.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:17.647332+0000) 2022-09-10T15:57:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:18.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:18.016484+0000) 2022-09-10T15:57:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:18 smithi110 ceph-mon[97633]: pgmap v22015: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:57:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:18 smithi104 ceph-mon[97044]: pgmap v22015: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 231 B/s wr, 11 op/s 2022-09-10T15:57:19.018 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:18.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:18.647549+0000) 2022-09-10T15:57:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:19.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:19.016747+0000) 2022-09-10T15:57:20.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:19.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:19.647711+0000) 2022-09-10T15:57:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:20.015+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:20.017013+0000) 2022-09-10T15:57:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:20 smithi110 ceph-mon[97633]: pgmap v22016: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:57:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:20 smithi104 ceph-mon[97044]: pgmap v22016: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:57:21.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:20.646+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:20.647974+0000) 2022-09-10T15:57:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:21.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:21.017241+0000) 2022-09-10T15:57:22.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:21.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:21.648193+0000) 2022-09-10T15:57:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:22.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:22.017498+0000) 2022-09-10T15:57:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:22 smithi110 ceph-mon[97633]: pgmap v22017: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:57:22.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:22 smithi104 ceph-mon[97044]: pgmap v22017: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:57:23.019 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:22.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:22.648454+0000) 2022-09-10T15:57:23.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:23.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:23.017696+0000) 2022-09-10T15:57:23.932 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:23.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:23.648641+0000) 2022-09-10T15:57:24.297 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:24 smithi104 ceph-mon[97044]: pgmap v22018: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:57:24.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:24.016+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:24.017889+0000) 2022-09-10T15:57:24.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:57:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:57:23.929+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:57:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:24 smithi110 ceph-mon[97633]: pgmap v22018: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2022-09-10T15:57:25.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:24.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:24.648867+0000) 2022-09-10T15:57:25.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:25.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:25.018144+0000) 2022-09-10T15:57:26.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:25.647+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:25.649043+0000) 2022-09-10T15:57:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:26.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:26.018371+0000) 2022-09-10T15:57:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:26 smithi110 ceph-mon[97633]: pgmap v22019: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:57:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:26 smithi104 ceph-mon[97044]: pgmap v22019: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:57:27.020 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:26.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:26.649249+0000) 2022-09-10T15:57:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:27.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:27.018545+0000) 2022-09-10T15:57:28.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:27.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:27.649475+0000) 2022-09-10T15:57:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:28.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:28.018785+0000) 2022-09-10T15:57:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:28 smithi110 ceph-mon[97633]: pgmap v22020: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:28 smithi104 ceph-mon[97044]: pgmap v22020: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:29.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:28.648+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:28.649684+0000) 2022-09-10T15:57:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:29.017+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:29.019023+0000) 2022-09-10T15:57:30.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:29.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:29.649924+0000) 2022-09-10T15:57:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:30.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:30.019285+0000) 2022-09-10T15:57:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:30 smithi110 ceph-mon[97633]: pgmap v22021: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:30 smithi104 ceph-mon[97044]: pgmap v22021: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:31.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:30.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:30.650168+0000) 2022-09-10T15:57:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:31.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:31.019517+0000) 2022-09-10T15:57:32.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:31.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:31.650376+0000) 2022-09-10T15:57:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:32.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:32.019752+0000) 2022-09-10T15:57:32.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:32 smithi110 ceph-mon[97633]: pgmap v22022: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:32 smithi104 ceph-mon[97044]: pgmap v22022: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:33.021 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:32.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:32.650645+0000) 2022-09-10T15:57:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:33.018+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:33.019956+0000) 2022-09-10T15:57:34.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:33.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:33.650801+0000) 2022-09-10T15:57:34.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:34.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:34.020187+0000) 2022-09-10T15:57:34.411 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:34 smithi110 ceph-mon[97633]: pgmap v22023: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:34 smithi104 ceph-mon[97044]: pgmap v22023: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:35.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:34.649+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:34.650977+0000) 2022-09-10T15:57:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:35.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:35.020434+0000) 2022-09-10T15:57:36.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:35.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:35.651241+0000) 2022-09-10T15:57:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:36.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:36.020677+0000) 2022-09-10T15:57:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:36 smithi110 ceph-mon[97633]: pgmap v22024: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:36 smithi104 ceph-mon[97044]: pgmap v22024: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:37.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:36.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:36.651437+0000) 2022-09-10T15:57:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:37.019+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:37.020838+0000) 2022-09-10T15:57:38.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:37.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:37.651678+0000) 2022-09-10T15:57:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:38.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:38.021071+0000) 2022-09-10T15:57:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:38 smithi110 ceph-mon[97633]: pgmap v22025: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:38 smithi104 ceph-mon[97044]: pgmap v22025: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:38.946 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:38.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:38.651821+0000) 2022-09-10T15:57:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:39.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:39.021360+0000) 2022-09-10T15:57:39.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:57:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:57:38.943+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:57:40.023 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:39.650+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:39.652033+0000) 2022-09-10T15:57:40.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:40.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:40.021596+0000) 2022-09-10T15:57:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:40 smithi110 ceph-mon[97633]: pgmap v22026: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:40.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:40 smithi104 ceph-mon[97044]: pgmap v22026: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:41.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:40.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:40.652207+0000) 2022-09-10T15:57:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:41.020+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:41.021765+0000) 2022-09-10T15:57:42.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:41.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:41.652335+0000) 2022-09-10T15:57:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:42.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:42.021942+0000) 2022-09-10T15:57:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:42 smithi110 ceph-mon[97633]: pgmap v22027: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:42 smithi104 ceph-mon[97044]: pgmap v22027: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:43.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:42.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:42.652548+0000) 2022-09-10T15:57:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:43.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:43.022080+0000) 2022-09-10T15:57:44.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:43.651+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:43.652828+0000) 2022-09-10T15:57:44.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:44.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:44.022281+0000) 2022-09-10T15:57:44.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:44 smithi110 ceph-mon[97633]: pgmap v22028: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:44.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:44 smithi104 ceph-mon[97044]: pgmap v22028: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:45.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:44.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:44.653056+0000) 2022-09-10T15:57:45.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:45.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:45.022490+0000) 2022-09-10T15:57:46.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:45.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:45.653201+0000) 2022-09-10T15:57:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:46.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:46.022667+0000) 2022-09-10T15:57:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:46 smithi110 ceph-mon[97633]: pgmap v22029: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:46 smithi104 ceph-mon[97044]: pgmap v22029: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:47.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:46.652+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:46.653376+0000) 2022-09-10T15:57:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:47.021+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:47.022806+0000) 2022-09-10T15:57:48.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:47.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:47.653612+0000) 2022-09-10T15:57:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:48.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:48.022957+0000) 2022-09-10T15:57:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:48 smithi110 ceph-mon[97633]: pgmap v22030: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:57:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:48 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:57:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:57:48.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:48 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:57:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:48 smithi104 ceph-mon[97044]: pgmap v22030: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:57:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:48 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:57:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:57:48.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:48 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:57:49.024 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:48.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:48.653867+0000) 2022-09-10T15:57:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:49.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:49.023120+0000) 2022-09-10T15:57:50.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:49.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:49.654169+0000) 2022-09-10T15:57:50.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:50.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:50.023361+0000) 2022-09-10T15:57:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:50 smithi110 ceph-mon[97633]: pgmap v22031: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:50.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:50 smithi104 ceph-mon[97044]: pgmap v22031: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:51.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:50.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:50.654382+0000) 2022-09-10T15:57:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:51.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:51.023540+0000) 2022-09-10T15:57:52.025 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:51.653+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:51.654532+0000) 2022-09-10T15:57:52.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:52.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:52.023733+0000) 2022-09-10T15:57:52.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:52 smithi110 ceph-mon[97633]: pgmap v22032: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:52.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:52 smithi104 ceph-mon[97044]: pgmap v22032: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:53.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:52 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:52.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:52.654769+0000) 2022-09-10T15:57:53.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:53.022+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:53.023959+0000) 2022-09-10T15:57:53.956 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:53.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:53.654995+0000) 2022-09-10T15:57:54.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:57:53 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:57:53.953+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:57:54.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:54.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:54.024181+0000) 2022-09-10T15:57:54.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:54 smithi110 ceph-mon[97633]: pgmap v22033: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:54.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:54 smithi104 ceph-mon[97044]: pgmap v22033: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:55.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:54 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:54.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:54.655253+0000) 2022-09-10T15:57:55.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:55.023+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:55.024418+0000) 2022-09-10T15:57:56.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:55 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:55.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:55.655556+0000) 2022-09-10T15:57:56.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:56.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:56.024640+0000) 2022-09-10T15:57:56.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:56 smithi110 ceph-mon[97633]: pgmap v22034: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:56.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:56 smithi104 ceph-mon[97044]: pgmap v22034: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:57:57.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:56 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:56.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:56.655781+0000) 2022-09-10T15:57:57.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:57.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:57.024823+0000) 2022-09-10T15:57:58.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:57 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:57.654+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:57.655978+0000) 2022-09-10T15:57:58.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:58.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:58.025060+0000) 2022-09-10T15:57:58.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:57:58 smithi110 ceph-mon[97633]: pgmap v22035: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:58.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:57:58 smithi104 ceph-mon[97044]: pgmap v22035: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:57:59.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:58 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:58.655+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:58.656213+0000) 2022-09-10T15:57:59.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:57:59.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:59.025281+0000) 2022-09-10T15:58:00.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:57:59 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:57:59.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:57:59.656483+0000) 2022-09-10T15:58:00.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:00.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:00.025431+0000) 2022-09-10T15:58:00.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:00 smithi110 ceph-mon[97633]: pgmap v22036: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:00.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:00 smithi104 ceph-mon[97044]: pgmap v22036: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:01.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:00 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:00.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:00.656749+0000) 2022-09-10T15:58:01.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:01.024+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:01.025575+0000) 2022-09-10T15:58:02.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:01 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:01.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:01.656999+0000) 2022-09-10T15:58:02.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:02.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:02.025772+0000) 2022-09-10T15:58:02.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:02 smithi110 ceph-mon[97633]: pgmap v22037: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:02.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:02 smithi104 ceph-mon[97044]: pgmap v22037: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:03.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:02 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:02.656+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:02.657258+0000) 2022-09-10T15:58:03.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:03.025+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:03.025986+0000) 2022-09-10T15:58:04.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:03 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:03.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:03.657526+0000) 2022-09-10T15:58:04.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:04.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:04.026233+0000) 2022-09-10T15:58:04.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:04 smithi110 ceph-mon[97633]: pgmap v22038: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:04.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:04 smithi104 ceph-mon[97044]: pgmap v22038: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:05.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:04 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:04.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:04.657765+0000) 2022-09-10T15:58:05.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:05.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:05.026396+0000) 2022-09-10T15:58:06.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:05 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:05.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:05.658037+0000) 2022-09-10T15:58:06.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:06.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:06.026605+0000) 2022-09-10T15:58:06.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:06 smithi110 ceph-mon[97633]: pgmap v22039: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:58:06.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:06 smithi104 ceph-mon[97044]: pgmap v22039: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 27 op/s 2022-09-10T15:58:07.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:06 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:06.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:06.658291+0000) 2022-09-10T15:58:07.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:07.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:07.026734+0000) 2022-09-10T15:58:08.028 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:07 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:07.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:07.658465+0000) 2022-09-10T15:58:08.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:08.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:08.026954+0000) 2022-09-10T15:58:08.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:08 smithi104 ceph-mon[97044]: pgmap v22040: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:58:08.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:08 smithi110 ceph-mon[97633]: pgmap v22040: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2022-09-10T15:58:08.971 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:08.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:08.658717+0000) 2022-09-10T15:58:09.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:09.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:09.027213+0000) 2022-09-10T15:58:09.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:58:08 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:58:08.969+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:58:09.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:09 smithi104 ceph-mon[97044]: osdmap e1458: 12 total, 12 up, 12 in 2022-09-10T15:58:09.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:09 smithi110 ceph-mon[97633]: osdmap e1458: 12 total, 12 up, 12 in 2022-09-10T15:58:10.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:09 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:09.657+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:09.658968+0000) 2022-09-10T15:58:10.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:10.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:10.027404+0000) 2022-09-10T15:58:10.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:10 smithi104 ceph-mon[97044]: pgmap v22042: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:58:10.809 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:10 smithi110 ceph-mon[97633]: pgmap v22042: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:58:11.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:10 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:10.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:10.659163+0000) 2022-09-10T15:58:11.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:11.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:11.027604+0000) 2022-09-10T15:58:12.029 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:11 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:11.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:11.659389+0000) 2022-09-10T15:58:12.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:12.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:12.027809+0000) 2022-09-10T15:58:12.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:12 smithi104 ceph-mon[97044]: pgmap v22043: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:58:12.658 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:12 smithi110 ceph-mon[97633]: pgmap v22043: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:58:13.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:12 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:12.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:12.659532+0000) 2022-09-10T15:58:13.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:13.026+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:13.028015+0000) 2022-09-10T15:58:14.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:13 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:13.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:13.659674+0000) 2022-09-10T15:58:14.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:14.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:14.028265+0000) 2022-09-10T15:58:14.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:14 smithi110 ceph-mon[97633]: pgmap v22044: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:58:14.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:14 smithi110 ceph-mon[97633]: osdmap e1459: 12 total, 12 up, 12 in 2022-09-10T15:58:14.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:14 smithi104 ceph-mon[97044]: pgmap v22044: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2022-09-10T15:58:14.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:14 smithi104 ceph-mon[97044]: osdmap e1459: 12 total, 12 up, 12 in 2022-09-10T15:58:15.030 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:14 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:14.658+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:14.659926+0000) 2022-09-10T15:58:15.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:15.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:15.028481+0000) 2022-09-10T15:58:16.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:15 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:15.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:15.660146+0000) 2022-09-10T15:58:16.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:16.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:16.028668+0000) 2022-09-10T15:58:16.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:16 smithi110 ceph-mon[97633]: pgmap v22046: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:58:16.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:16 smithi104 ceph-mon[97044]: pgmap v22046: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2022-09-10T15:58:17.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:16 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:16.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:16.660319+0000) 2022-09-10T15:58:17.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:17.027+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:17.028853+0000) 2022-09-10T15:58:18.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:17 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:17.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:17.660591+0000) 2022-09-10T15:58:18.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:18.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:18.029121+0000) 2022-09-10T15:58:18.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:18 smithi110 ceph-mon[97633]: pgmap v22047: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T15:58:18.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:18 smithi104 ceph-mon[97044]: pgmap v22047: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 230 B/s wr, 11 op/s 2022-09-10T15:58:19.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:18 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:18.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:18.660780+0000) 2022-09-10T15:58:19.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:19.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:19.029313+0000) 2022-09-10T15:58:20.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:19 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:19.659+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:19.661029+0000) 2022-09-10T15:58:20.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:20.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:20.029537+0000) 2022-09-10T15:58:20.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:20 smithi110 ceph-mon[97633]: pgmap v22048: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:58:20.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:20 smithi104 ceph-mon[97044]: pgmap v22048: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2022-09-10T15:58:21.031 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:20 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:20.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:20.661262+0000) 2022-09-10T15:58:21.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:21.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:21.029798+0000) 2022-09-10T15:58:22.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:21 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:21.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:21.661499+0000) 2022-09-10T15:58:22.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:22.028+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:22.030037+0000) 2022-09-10T15:58:22.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:22 smithi110 ceph-mon[97633]: pgmap v22049: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:58:22.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:22 smithi104 ceph-mon[97044]: pgmap v22049: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:58:23.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:22 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:22.660+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:22.661662+0000) 2022-09-10T15:58:23.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:23.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:23.030213+0000) 2022-09-10T15:58:23.984 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:23.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:23.661897+0000) 2022-09-10T15:58:24.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:58:23 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:58:23.981+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:58:24.306 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:24.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:24.030435+0000) 2022-09-10T15:58:24.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:24 smithi110 ceph-mon[97633]: pgmap v22050: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:58:24.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:24 smithi104 ceph-mon[97044]: pgmap v22050: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2022-09-10T15:58:25.032 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:24 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:24.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:24.662183+0000) 2022-09-10T15:58:25.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:25.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:25.030600+0000) 2022-09-10T15:58:26.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:25 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:25.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:25.662407+0000) 2022-09-10T15:58:26.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:26.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:26.030874+0000) 2022-09-10T15:58:26.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:26 smithi110 ceph-mon[97633]: pgmap v22051: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:58:26.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:26 smithi104 ceph-mon[97044]: pgmap v22051: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 85 B/s wr, 12 op/s 2022-09-10T15:58:27.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:26 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:26.661+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:26.662638+0000) 2022-09-10T15:58:27.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:27.029+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:27.031027+0000) 2022-09-10T15:58:28.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:27 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:27.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:27.662940+0000) 2022-09-10T15:58:28.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:28.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:28.031215+0000) 2022-09-10T15:58:28.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:28 smithi110 ceph-mon[97633]: pgmap v22052: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:28.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:28 smithi104 ceph-mon[97044]: pgmap v22052: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:29.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:28 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:28.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:28.663235+0000) 2022-09-10T15:58:29.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:29.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:29.031427+0000) 2022-09-10T15:58:30.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:29 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:29.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:29.663493+0000) 2022-09-10T15:58:30.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:30.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:30.031604+0000) 2022-09-10T15:58:30.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:30 smithi110 ceph-mon[97633]: pgmap v22053: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:30.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:30 smithi104 ceph-mon[97044]: pgmap v22053: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:31.033 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:30 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:30.662+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:30.663730+0000) 2022-09-10T15:58:31.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:31.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:31.031775+0000) 2022-09-10T15:58:32.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:31 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:31.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:31.663964+0000) 2022-09-10T15:58:32.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:32.030+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:32.031964+0000) 2022-09-10T15:58:32.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:32 smithi110 ceph-mon[97633]: pgmap v22054: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:32.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:32 smithi104 ceph-mon[97044]: pgmap v22054: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:33.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:32 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:32.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:32.664163+0000) 2022-09-10T15:58:33.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:33.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:33.032236+0000) 2022-09-10T15:58:34.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:33 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:33.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:33.664312+0000) 2022-09-10T15:58:34.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:34.031+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:34.032480+0000) 2022-09-10T15:58:34.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:34 smithi110 ceph-mon[97633]: pgmap v22055: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:34.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:34 smithi104 ceph-mon[97044]: pgmap v22055: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:35.034 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:34 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:34.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:34.664488+0000) 2022-09-10T15:58:35.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:35.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:35.032719+0000) 2022-09-10T15:58:36.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:35 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:35.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:35.664690+0000) 2022-09-10T15:58:36.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:36.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:36.032933+0000) 2022-09-10T15:58:36.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:36 smithi110 ceph-mon[97633]: pgmap v22056: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:36.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:36 smithi104 ceph-mon[97044]: pgmap v22056: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:37.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:36 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:36.663+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:36.664967+0000) 2022-09-10T15:58:37.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:37.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:37.033183+0000) 2022-09-10T15:58:38.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:37 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:37.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:37.665170+0000) 2022-09-10T15:58:38.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:38.032+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:38.033405+0000) 2022-09-10T15:58:38.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:38 smithi110 ceph-mon[97633]: pgmap v22057: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:38.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:38 smithi104 ceph-mon[97044]: pgmap v22057: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:38.994 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:38.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:38.665371+0000) 2022-09-10T15:58:39.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:39.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:39.033647+0000) 2022-09-10T15:58:39.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Sep 10 15:58:38 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mon-a[90109]: 2022-09-10T15:58:38.991+0000 7f2f211cc700 -1 mon.a@0(leader) e3 handle_auth_bad_method hmm, they didn't like 2 result (13) Permission denied 2022-09-10T15:58:40.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:39 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:39.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:39.665638+0000) 2022-09-10T15:58:40.307 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:40.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:40.033849+0000) 2022-09-10T15:58:40.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:40 smithi110 ceph-mon[97633]: pgmap v22058: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:40.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:40 smithi104 ceph-mon[97044]: pgmap v22058: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:41.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:40 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:40.664+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:40.665861+0000) 2022-09-10T15:58:41.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:41.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:41.034073+0000) 2022-09-10T15:58:42.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:41 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:41.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:41.666141+0000) 2022-09-10T15:58:42.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:42.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:42.034335+0000) 2022-09-10T15:58:42.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:42 smithi110 ceph-mon[97633]: pgmap v22059: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:42.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:42 smithi104 ceph-mon[97044]: pgmap v22059: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:43.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:42 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:42.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:42.666408+0000) 2022-09-10T15:58:43.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:43.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:43.034534+0000) 2022-09-10T15:58:44.036 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:43 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:43.665+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:43.666628+0000) 2022-09-10T15:58:44.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:44.033+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:44.034785+0000) 2022-09-10T15:58:44.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:44 smithi110 ceph-mon[97633]: pgmap v22060: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:44.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:44 smithi104 ceph-mon[97044]: pgmap v22060: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:45.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:44 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:44.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:44.666852+0000) 2022-09-10T15:58:45.310 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:45.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:45.035004+0000) 2022-09-10T15:58:46.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:45 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:45.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:45.667057+0000) 2022-09-10T15:58:46.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:46.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:46.035278+0000) 2022-09-10T15:58:46.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:46 smithi110 ceph-mon[97633]: pgmap v22061: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:46.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:46 smithi104 ceph-mon[97044]: pgmap v22061: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2022-09-10T15:58:47.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:46 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:46.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:46.667256+0000) 2022-09-10T15:58:47.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:47.034+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:47.035485+0000) 2022-09-10T15:58:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:58:47.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:58:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:58:47.409 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:47 smithi110 ceph-mon[97633]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:58:47.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:58:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 172.21.15.81:0/544058993' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:58:47.548 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2022-09-10T15:58:47.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:47 smithi104 ceph-mon[97044]: from='mgr.25379 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2022-09-10T15:58:48.037 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:47 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:47.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:47.667489+0000) 2022-09-10T15:58:48.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:48.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:48.035709+0000) 2022-09-10T15:58:48.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:48 smithi110 ceph-mon[97633]: pgmap v22062: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:48.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:48 smithi104 ceph-mon[97044]: pgmap v22062: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:49.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:48 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:48.666+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:48.667626+0000) 2022-09-10T15:58:49.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:49.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:49.035964+0000) 2022-09-10T15:58:50.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:49 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:49.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:49.667869+0000) 2022-09-10T15:58:50.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:50.035+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:50.036259+0000) 2022-09-10T15:58:50.408 INFO:journalctl@ceph.mon.c.smithi110.stdout:Sep 10 15:58:50 smithi110 ceph-mon[97633]: pgmap v22063: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:50.547 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 10 15:58:50 smithi104 ceph-mon[97044]: pgmap v22063: 129 pgs: 129 active+clean; 10 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2022-09-10T15:58:51.038 INFO:journalctl@ceph.mds.d.smithi081.stdout:Sep 10 15:58:50 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-d[124325]: 2022-09-10T15:58:50.667+0000 7f6e5ef36700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:50.668124+0000) 2022-09-10T15:58:51.305 INFO:journalctl@ceph.mds.g.smithi081.stdout:Sep 10 15:58:51 smithi081 ceph-70cf895c-30bf-11ed-8431-001a4aab830c-mds-g[124945]: 2022-09-10T15:58:51.036+0000 7f1640515700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2022-09-10T14:58:51.036501+0000) 2022-09-10T15:58:52.021 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-09-10T15:58:52.023 DEBUG:teuthology.task.console_log:Killing console logger for smithi081 2022-09-10T15:58:52.025 DEBUG:teuthology.task.console_log:Killing console logger for smithi104 2022-09-10T15:58:52.026 DEBUG:teuthology.task.console_log:Killing console logger for smithi110 2022-09-10T15:58:52.027 DEBUG:teuthology.exit:Finished running handlers